2024-03-31T14:48:22.266 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-31T14:48:22.266 DEBUG:teuthology.run:Teuthology command: teuthology --archive /home/teuthworker/archive/vshankar-2024-03-30_23:12:08-fs-wip-vshankar-testing-20240330.170739-testing-default-smithi/7631609 --verbose --name vshankar-2024-03-30_23:12:08-fs-wip-vshankar-testing-20240330.170739-testing-default-smithi --owner scheduled_vshankar@teuthology --description fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v1 tasks/forward-scrub} -- /home/teuthworker/archive/vshankar-2024-03-30_23:12:08-fs-wip-vshankar-testing-20240330.170739-testing-default-smithi/7631609/orig.config.yaml 2024-03-31T14:48:22.313 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-31T14:48:22.394 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/vshankar-2024-03-30_23:12:08-fs-wip-vshankar-testing-20240330.170739-testing-default-smithi/7631609 branch: wip-vshankar-testing-20240330.170739 description: fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v1 tasks/forward-scrub} email: vshankar@redhat.com first_in_suite: false job_id: '7631609' kernel: flavor: default kdb: true sha1: 1df0c7e8a2354c42d738e5454ba83ea7332deac2 ktype: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi mgrmodules: sequential: - print: Enabling mgr modules name: vshankar-2024-03-30_23:12:08-fs-wip-vshankar-testing-20240330.170739-testing-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: ubuntu os_version: '22.04' overrides: admin_socket: branch: wip-vshankar-testing-20240330.170739 ceph: conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 pid file: '' rados mon op timeout: 900 rados osd op timeout: 900 mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug client: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon down mkfs grace: 300 mon op complaint time: 120 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - filesystem is online with fewer MDS than max_mds - POOL_APP_NOT_ENABLED - overall HEALTH_ - Replacing daemon - deprecated feature inline_data - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - PG_AVAILABILITY - PG_DEGRADED - Reduced data availability - inode wrongly marked free - bad backtrace on inode - inode table repaired for inode - Scrub error on inode - Scrub error on dir - Metadata damage detected sha1: d2271579d5e735cfd6d436b8349c004f06527cb0 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: d2271579d5e735cfd6d436b8349c004f06527cb0 subvolume_version: 1 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-vshankar-testing-20240330.170739 sha1: d2271579d5e735cfd6d436b8349c004f06527cb0 owner: scheduled_vshankar@teuthology priority: 100 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.y - mds.a - mds.c - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - client.2 - - mon.b - mon.c - mgr.x - mds.b - mds.d - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - client.3 seed: 6891 sha1: d2271579d5e735cfd6d436b8349c004f06527cb0 sleep_before_teardown: 0 subset: 932/1024 suite: fs suite_branch: wip-vshankar-testing-20240330.170739 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_d2271579d5e735cfd6d436b8349c004f06527cb0/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: d2271579d5e735cfd6d436b8349c004f06527cb0 syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi016.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGrNGRw477WUQu17A6o65D4kDQ6WSLg/SUYnTZXG9vNI9kj8M4+jdSsqhTA1AVsIdEZoduIq19dWV/GhMsuzXqM= smithi089.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBANsxlXyxrksTMZg1EXGMkYGT/ASySvp6CS9Cas0w9gzWiSepEnTbAeyQC0DgQosrPfrSbAwG8nwNKSj4IuxKLE= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - ceph: create_rbd_pool: false - sequential: - mgrmodules - ceph-fuse: null - cephfs_test_runner: modules: - tasks.cephfs.test_forward_scrub teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return false end - local function is_kdistro() return false end - local function is_fuse() return true end - local function syntax_version() return '' end teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-30_23:12:08 tube: smithi user: vshankar verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-31T14:48:22.394 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_d2271579d5e735cfd6d436b8349c004f06527cb0/qa; will attempt to use it 2024-03-31T14:48:22.395 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_d2271579d5e735cfd6d436b8349c004f06527cb0/qa/tasks 2024-03-31T14:48:22.395 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-31T14:48:22.397 INFO:teuthology.task.internal:Checking packages... 2024-03-31T14:48:22.421 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash 'd2271579d5e735cfd6d436b8349c004f06527cb0' 2024-03-31T14:48:22.421 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-31T14:48:22.421 INFO:teuthology.packaging:ref: None 2024-03-31T14:48:22.421 INFO:teuthology.packaging:tag: None 2024-03-31T14:48:22.421 INFO:teuthology.packaging:branch: wip-vshankar-testing-20240330.170739 2024-03-31T14:48:22.421 INFO:teuthology.packaging:sha1: d2271579d5e735cfd6d436b8349c004f06527cb0 2024-03-31T14:48:22.421 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&ref=wip-vshankar-testing-20240330.170739 2024-03-31T14:48:22.640 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2574-gd2271579-1jammy 2024-03-31T14:48:22.642 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-31T14:48:22.648 INFO:teuthology.task.internal:no buildpackages task found 2024-03-31T14:48:22.648 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-31T14:48:22.653 INFO:teuthology.task.internal:Saving configuration 2024-03-31T14:48:22.670 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-31T14:48:22.677 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-31T14:48:22.704 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi016.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/vshankar-2024-03-30_23:12:08-fs-wip-vshankar-testing-20240330.170739-testing-default-smithi/7631609', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-31 14:42:50.434724', 'locked_by': 'scheduled_vshankar@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIJUNlcbI7+y4QFn7G0hDzVIafoPXVPgVkxYS1qWelaxa'} 2024-03-31T14:48:22.726 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi089.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/vshankar-2024-03-30_23:12:08-fs-wip-vshankar-testing-20240330.170739-testing-default-smithi/7631609', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-31 14:42:50.436123', 'locked_by': 'scheduled_vshankar@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIMdJB4tvvvODh3kkst1aui7wNEjYhhT9RZmnNp2RGf4O'} 2024-03-31T14:48:22.726 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-31T14:48:22.732 INFO:teuthology.task.internal:roles: ubuntu@smithi016.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2'] 2024-03-31T14:48:22.732 INFO:teuthology.task.internal:roles: ubuntu@smithi089.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3'] 2024-03-31T14:48:22.732 INFO:teuthology.run_tasks:Running task console_log... 2024-03-31T14:48:22.807 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7ff20188a4c0>, signals=[15]) 2024-03-31T14:48:22.807 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-31T14:48:22.814 INFO:teuthology.task.internal:Opening connections... 2024-03-31T14:48:22.814 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi016.front.sepia.ceph.com 2024-03-31T14:48:22.816 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi016.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-31T14:48:22.899 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi089.front.sepia.ceph.com 2024-03-31T14:48:22.900 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi089.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-31T14:48:22.994 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-31T14:48:23.001 DEBUG:teuthology.orchestra.run.smithi016:> uname -m 2024-03-31T14:48:23.008 INFO:teuthology.orchestra.run.smithi016.stdout:x86_64 2024-03-31T14:48:23.009 DEBUG:teuthology.orchestra.run.smithi016:> cat /etc/os-release 2024-03-31T14:48:23.057 INFO:teuthology.orchestra.run.smithi016.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2024-03-31T14:48:23.057 INFO:teuthology.orchestra.run.smithi016.stdout:NAME="Ubuntu" 2024-03-31T14:48:23.057 INFO:teuthology.orchestra.run.smithi016.stdout:VERSION_ID="22.04" 2024-03-31T14:48:23.057 INFO:teuthology.orchestra.run.smithi016.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2024-03-31T14:48:23.058 INFO:teuthology.orchestra.run.smithi016.stdout:VERSION_CODENAME=jammy 2024-03-31T14:48:23.058 INFO:teuthology.orchestra.run.smithi016.stdout:ID=ubuntu 2024-03-31T14:48:23.058 INFO:teuthology.orchestra.run.smithi016.stdout:ID_LIKE=debian 2024-03-31T14:48:23.058 INFO:teuthology.orchestra.run.smithi016.stdout:HOME_URL="https://www.ubuntu.com/" 2024-03-31T14:48:23.058 INFO:teuthology.orchestra.run.smithi016.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-03-31T14:48:23.059 INFO:teuthology.orchestra.run.smithi016.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-03-31T14:48:23.059 INFO:teuthology.orchestra.run.smithi016.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-03-31T14:48:23.059 INFO:teuthology.orchestra.run.smithi016.stdout:UBUNTU_CODENAME=jammy 2024-03-31T14:48:23.060 INFO:teuthology.lock.ops:Updating smithi016.front.sepia.ceph.com on lock server 2024-03-31T14:48:23.084 DEBUG:teuthology.orchestra.run.smithi089:> uname -m 2024-03-31T14:48:23.090 INFO:teuthology.orchestra.run.smithi089.stdout:x86_64 2024-03-31T14:48:23.090 DEBUG:teuthology.orchestra.run.smithi089:> cat /etc/os-release 2024-03-31T14:48:23.136 INFO:teuthology.orchestra.run.smithi089.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2024-03-31T14:48:23.136 INFO:teuthology.orchestra.run.smithi089.stdout:NAME="Ubuntu" 2024-03-31T14:48:23.136 INFO:teuthology.orchestra.run.smithi089.stdout:VERSION_ID="22.04" 2024-03-31T14:48:23.137 INFO:teuthology.orchestra.run.smithi089.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2024-03-31T14:48:23.137 INFO:teuthology.orchestra.run.smithi089.stdout:VERSION_CODENAME=jammy 2024-03-31T14:48:23.137 INFO:teuthology.orchestra.run.smithi089.stdout:ID=ubuntu 2024-03-31T14:48:23.137 INFO:teuthology.orchestra.run.smithi089.stdout:ID_LIKE=debian 2024-03-31T14:48:23.137 INFO:teuthology.orchestra.run.smithi089.stdout:HOME_URL="https://www.ubuntu.com/" 2024-03-31T14:48:23.137 INFO:teuthology.orchestra.run.smithi089.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-03-31T14:48:23.137 INFO:teuthology.orchestra.run.smithi089.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-03-31T14:48:23.138 INFO:teuthology.orchestra.run.smithi089.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-03-31T14:48:23.138 INFO:teuthology.orchestra.run.smithi089.stdout:UBUNTU_CODENAME=jammy 2024-03-31T14:48:23.138 INFO:teuthology.lock.ops:Updating smithi089.front.sepia.ceph.com on lock server 2024-03-31T14:48:23.159 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-31T14:48:23.172 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-31T14:48:23.180 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-31T14:48:23.180 DEBUG:teuthology.orchestra.run.smithi016:> test '!' -e /home/ubuntu/cephtest 2024-03-31T14:48:23.183 DEBUG:teuthology.orchestra.run.smithi089:> test '!' -e /home/ubuntu/cephtest 2024-03-31T14:48:23.188 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-31T14:48:23.193 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-31T14:48:23.193 DEBUG:teuthology.orchestra.run.smithi016:> test -z $(ls -A /var/lib/ceph) 2024-03-31T14:48:23.229 DEBUG:teuthology.orchestra.run.smithi089:> test -z $(ls -A /var/lib/ceph) 2024-03-31T14:48:23.323 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-31T14:48:23.383 INFO:teuthology.run_tasks:Running task kernel... 2024-03-31T14:48:23.397 INFO:teuthology.task.kernel:normalize config orig: {'flavor': 'default', 'kdb': True, 'sha1': '1df0c7e8a2354c42d738e5454ba83ea7332deac2'} 2024-03-31T14:48:23.397 INFO:teuthology.task.kernel:config {'mon.a': {'flavor': 'default', 'kdb': True, 'sha1': '1df0c7e8a2354c42d738e5454ba83ea7332deac2'}, 'mon.b': {'flavor': 'default', 'kdb': True, 'sha1': '1df0c7e8a2354c42d738e5454ba83ea7332deac2'}}, timeout 300 2024-03-31T14:48:23.398 DEBUG:teuthology.orchestra.run.smithi016:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-31T14:48:23.398 DEBUG:teuthology.orchestra.run.smithi089:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-31T14:48:23.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-31T14:48:23.403 DEBUG:teuthology.task.kernel:sha1 for mon.b is 1df0c7e8a2354c42d738e5454ba83ea7332deac2 2024-03-31T14:48:23.404 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "1df0c7e8a2354c42d738e5454ba83ea7332deac2"... 2024-03-31T14:48:23.404 DEBUG:teuthology.orchestra.run.smithi089:> uname -r 2024-03-31T14:48:23.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-31T14:48:23.405 DEBUG:teuthology.task.kernel:sha1 for mon.a is 1df0c7e8a2354c42d738e5454ba83ea7332deac2 2024-03-31T14:48:23.405 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "1df0c7e8a2354c42d738e5454ba83ea7332deac2"... 2024-03-31T14:48:23.405 DEBUG:teuthology.orchestra.run.smithi016:> uname -r 2024-03-31T14:48:23.448 INFO:teuthology.orchestra.run.smithi089.stdout:5.15.0-56-generic 2024-03-31T14:48:23.449 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-56-generic vs 1df0c7e8a2354c42d738e5454ba83ea7332deac2 2024-03-31T14:48:23.449 DEBUG:teuthology.task.kernel:failed to parse current kernel version 2024-03-31T14:48:23.449 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=1df0c7e8a2354c42d738e5454ba83ea7332deac2 2024-03-31T14:48:23.455 INFO:teuthology.orchestra.run.smithi016.stdout:5.15.0-56-generic 2024-03-31T14:48:23.455 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-56-generic vs 1df0c7e8a2354c42d738e5454ba83ea7332deac2 2024-03-31T14:48:23.455 DEBUG:teuthology.task.kernel:failed to parse current kernel version 2024-03-31T14:48:23.456 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=1df0c7e8a2354c42d738e5454ba83ea7332deac2 2024-03-31T14:48:23.682 INFO:teuthology.task.kernel:Installing linux-firmware on mon.a... 2024-03-31T14:48:23.682 DEBUG:teuthology.orchestra.run.smithi016:> cd /lib/firmware/updates && test -d .git && test '!' -s .git/HEAD && sudo rm -rf .git ; sudo install -d -m0755 /lib/firmware/updates && cd /lib/firmware/updates && sudo git init 2024-03-31T14:48:23.684 INFO:teuthology.task.kernel:Installing linux-firmware on mon.b... 2024-03-31T14:48:23.684 DEBUG:teuthology.orchestra.run.smithi089:> cd /lib/firmware/updates && test -d .git && test '!' -s .git/HEAD && sudo rm -rf .git ; sudo install -d -m0755 /lib/firmware/updates && cd /lib/firmware/updates && sudo git init 2024-03-31T14:48:23.692 INFO:teuthology.orchestra.run.smithi089.stderr:bash: line 1: cd: /lib/firmware/updates: No such file or directory 2024-03-31T14:48:23.694 INFO:teuthology.orchestra.run.smithi016.stderr:bash: line 1: cd: /lib/firmware/updates: No such file or directory 2024-03-31T14:48:23.828 INFO:teuthology.orchestra.run.smithi089.stderr:hint: Using 'master' as the name for the initial branch. This default branch name 2024-03-31T14:48:23.828 INFO:teuthology.orchestra.run.smithi089.stderr:hint: is subject to change. To configure the initial branch name to use in all 2024-03-31T14:48:23.828 INFO:teuthology.orchestra.run.smithi089.stderr:hint: of your new repositories, which will suppress this warning, call: 2024-03-31T14:48:23.829 INFO:teuthology.orchestra.run.smithi089.stderr:hint: 2024-03-31T14:48:23.829 INFO:teuthology.orchestra.run.smithi089.stderr:hint: git config --global init.defaultBranch 2024-03-31T14:48:23.829 INFO:teuthology.orchestra.run.smithi089.stderr:hint: 2024-03-31T14:48:23.829 INFO:teuthology.orchestra.run.smithi089.stderr:hint: Names commonly chosen instead of 'master' are 'main', 'trunk' and 2024-03-31T14:48:23.829 INFO:teuthology.orchestra.run.smithi089.stderr:hint: 'development'. The just-created branch can be renamed via this command: 2024-03-31T14:48:23.829 INFO:teuthology.orchestra.run.smithi089.stderr:hint: 2024-03-31T14:48:23.829 INFO:teuthology.orchestra.run.smithi089.stderr:hint: git branch -m 2024-03-31T14:48:23.830 INFO:teuthology.orchestra.run.smithi089.stdout:Initialized empty Git repository in /usr/lib/firmware/updates/.git/ 2024-03-31T14:48:23.832 DEBUG:teuthology.orchestra.run.smithi089:> sudo git --git-dir=/lib/firmware/updates/.git config --get remote.origin.url >/dev/null || sudo git --git-dir=/lib/firmware/updates/.git remote add origin git://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git 2024-03-31T14:48:23.832 INFO:teuthology.orchestra.run.smithi016.stderr:hint: Using 'master' as the name for the initial branch. This default branch name 2024-03-31T14:48:23.832 INFO:teuthology.orchestra.run.smithi016.stderr:hint: is subject to change. To configure the initial branch name to use in all 2024-03-31T14:48:23.832 INFO:teuthology.orchestra.run.smithi016.stderr:hint: of your new repositories, which will suppress this warning, call: 2024-03-31T14:48:23.832 INFO:teuthology.orchestra.run.smithi016.stderr:hint: 2024-03-31T14:48:23.833 INFO:teuthology.orchestra.run.smithi016.stderr:hint: git config --global init.defaultBranch 2024-03-31T14:48:23.833 INFO:teuthology.orchestra.run.smithi016.stderr:hint: 2024-03-31T14:48:23.833 INFO:teuthology.orchestra.run.smithi016.stderr:hint: Names commonly chosen instead of 'master' are 'main', 'trunk' and 2024-03-31T14:48:23.833 INFO:teuthology.orchestra.run.smithi016.stderr:hint: 'development'. The just-created branch can be renamed via this command: 2024-03-31T14:48:23.833 INFO:teuthology.orchestra.run.smithi016.stderr:hint: 2024-03-31T14:48:23.833 INFO:teuthology.orchestra.run.smithi016.stderr:hint: git branch -m 2024-03-31T14:48:23.833 INFO:teuthology.orchestra.run.smithi016.stdout:Initialized empty Git repository in /usr/lib/firmware/updates/.git/ 2024-03-31T14:48:23.835 DEBUG:teuthology.orchestra.run.smithi016:> sudo git --git-dir=/lib/firmware/updates/.git config --get remote.origin.url >/dev/null || sudo git --git-dir=/lib/firmware/updates/.git remote add origin git://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git 2024-03-31T14:48:23.898 DEBUG:teuthology.orchestra.run.smithi089:> sudo git --git-dir=/lib/firmware/updates/.git remote set-url origin git://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git >/dev/null 2024-03-31T14:48:23.907 DEBUG:teuthology.orchestra.run.smithi016:> sudo git --git-dir=/lib/firmware/updates/.git remote set-url origin git://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git >/dev/null 2024-03-31T14:48:23.952 DEBUG:teuthology.orchestra.run.smithi089:> cd /lib/firmware/updates && sudo git fetch origin && sudo git reset --hard origin/main 2024-03-31T14:48:23.962 DEBUG:teuthology.orchestra.run.smithi016:> cd /lib/firmware/updates && sudo git fetch origin && sudo git reset --hard origin/main 2024-03-31T14:49:07.423 INFO:teuthology.orchestra.run.smithi016.stderr:From git://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware 2024-03-31T14:49:07.423 INFO:teuthology.orchestra.run.smithi016.stderr: * [new branch] main -> origin/main 2024-03-31T14:49:07.457 INFO:teuthology.orchestra.run.smithi016.stderr: * [new tag] 20190312 -> 20190312 2024-03-31T14:49:07.457 INFO:teuthology.orchestra.run.smithi016.stderr: * [new tag] 20190416 -> 20190416 2024-03-31T14:49:07.457 INFO:teuthology.orchestra.run.smithi016.stderr: * [new tag] 20190514 -> 20190514 2024-03-31T14:49:07.457 INFO:teuthology.orchestra.run.smithi016.stderr: * [new tag] 20190618 -> 20190618 2024-03-31T14:49:07.457 INFO:teuthology.orchestra.run.smithi016.stderr: * [new tag] 20190717 -> 20190717 2024-03-31T14:49:07.457 INFO:teuthology.orchestra.run.smithi016.stderr: * [new tag] 20190815 -> 20190815 2024-03-31T14:49:07.457 INFO:teuthology.orchestra.run.smithi016.stderr: * [new tag] 20190923 -> 20190923 2024-03-31T14:49:07.457 INFO:teuthology.orchestra.run.smithi016.stderr: * [new tag] 20191022 -> 20191022 2024-03-31T14:49:07.459 INFO:teuthology.orchestra.run.smithi016.stderr: * [new tag] 20191215 -> 20191215 2024-03-31T14:49:07.459 INFO:teuthology.orchestra.run.smithi016.stderr: * [new tag] 20200122 -> 20200122 2024-03-31T14:49:07.459 INFO:teuthology.orchestra.run.smithi016.stderr: * [new tag] 20200316 -> 20200316 2024-03-31T14:49:07.459 INFO:teuthology.orchestra.run.smithi016.stderr: * [new tag] 20200421 -> 20200421 2024-03-31T14:49:07.459 INFO:teuthology.orchestra.run.smithi016.stderr: * [new tag] 20200519 -> 20200519 2024-03-31T14:49:07.460 INFO:teuthology.orchestra.run.smithi016.stderr: * [new tag] 20200619 -> 20200619 2024-03-31T14:49:07.460 INFO:teuthology.orchestra.run.smithi016.stderr: * [new tag] 20200721 -> 20200721 2024-03-31T14:49:07.460 INFO:teuthology.orchestra.run.smithi016.stderr: * [new tag] 20200817 -> 20200817 2024-03-31T14:49:07.460 INFO:teuthology.orchestra.run.smithi016.stderr: * [new tag] 20200918 -> 20200918 2024-03-31T14:49:07.460 INFO:teuthology.orchestra.run.smithi016.stderr: * [new tag] 20201022 -> 20201022 2024-03-31T14:49:07.460 INFO:teuthology.orchestra.run.smithi016.stderr: * [new tag] 20201118 -> 20201118 2024-03-31T14:49:07.460 INFO:teuthology.orchestra.run.smithi016.stderr: * [new tag] 20201218 -> 20201218 2024-03-31T14:49:07.460 INFO:teuthology.orchestra.run.smithi016.stderr: * [new tag] 20210208 -> 20210208 2024-03-31T14:49:07.460 INFO:teuthology.orchestra.run.smithi016.stderr: * [new tag] 20210315 -> 20210315 2024-03-31T14:49:07.460 INFO:teuthology.orchestra.run.smithi016.stderr: * [new tag] 20210511 -> 20210511 2024-03-31T14:49:07.460 INFO:teuthology.orchestra.run.smithi016.stderr: * [new tag] 20210716 -> 20210716 2024-03-31T14:49:07.460 INFO:teuthology.orchestra.run.smithi016.stderr: * [new tag] 20210818 -> 20210818 2024-03-31T14:49:07.463 INFO:teuthology.orchestra.run.smithi016.stderr: * [new tag] 20210919 -> 20210919 2024-03-31T14:49:07.463 INFO:teuthology.orchestra.run.smithi016.stderr: * [new tag] 20211027 -> 20211027 2024-03-31T14:49:07.463 INFO:teuthology.orchestra.run.smithi016.stderr: * [new tag] 20211216 -> 20211216 2024-03-31T14:49:07.463 INFO:teuthology.orchestra.run.smithi016.stderr: * [new tag] 20220209 -> 20220209 2024-03-31T14:49:07.463 INFO:teuthology.orchestra.run.smithi016.stderr: * [new tag] 20220310 -> 20220310 2024-03-31T14:49:07.463 INFO:teuthology.orchestra.run.smithi016.stderr: * [new tag] 20220411 -> 20220411 2024-03-31T14:49:07.463 INFO:teuthology.orchestra.run.smithi016.stderr: * [new tag] 20220509 -> 20220509 2024-03-31T14:49:07.463 INFO:teuthology.orchestra.run.smithi016.stderr: * [new tag] 20220610 -> 20220610 2024-03-31T14:49:07.463 INFO:teuthology.orchestra.run.smithi016.stderr: * [new tag] 20220708 -> 20220708 2024-03-31T14:49:07.463 INFO:teuthology.orchestra.run.smithi016.stderr: * [new tag] 20220815 -> 20220815 2024-03-31T14:49:07.463 INFO:teuthology.orchestra.run.smithi016.stderr: * [new tag] 20220913 -> 20220913 2024-03-31T14:49:07.463 INFO:teuthology.orchestra.run.smithi016.stderr: * [new tag] 20221012 -> 20221012 2024-03-31T14:49:07.464 INFO:teuthology.orchestra.run.smithi016.stderr: * [new tag] 20221109 -> 20221109 2024-03-31T14:49:07.464 INFO:teuthology.orchestra.run.smithi016.stderr: * [new tag] 20221214 -> 20221214 2024-03-31T14:49:07.464 INFO:teuthology.orchestra.run.smithi016.stderr: * [new tag] 20230117 -> 20230117 2024-03-31T14:49:07.464 INFO:teuthology.orchestra.run.smithi016.stderr: * [new tag] 20230210 -> 20230210 2024-03-31T14:49:07.464 INFO:teuthology.orchestra.run.smithi016.stderr: * [new tag] 20230310 -> 20230310 2024-03-31T14:49:07.464 INFO:teuthology.orchestra.run.smithi016.stderr: * [new tag] 20230404 -> 20230404 2024-03-31T14:49:07.464 INFO:teuthology.orchestra.run.smithi016.stderr: * [new tag] 20230515 -> 20230515 2024-03-31T14:49:07.464 INFO:teuthology.orchestra.run.smithi016.stderr: * [new tag] 20230625 -> 20230625 2024-03-31T14:49:07.464 INFO:teuthology.orchestra.run.smithi016.stderr: * [new tag] 20230804 -> 20230804 2024-03-31T14:49:07.464 INFO:teuthology.orchestra.run.smithi016.stderr: * [new tag] 20230919 -> 20230919 2024-03-31T14:49:07.464 INFO:teuthology.orchestra.run.smithi016.stderr: * [new tag] 20231030 -> 20231030 2024-03-31T14:49:07.464 INFO:teuthology.orchestra.run.smithi016.stderr: * [new tag] 20231111 -> 20231111 2024-03-31T14:49:07.465 INFO:teuthology.orchestra.run.smithi016.stderr: * [new tag] 20231211 -> 20231211 2024-03-31T14:49:07.465 INFO:teuthology.orchestra.run.smithi016.stderr: * [new tag] 20240115 -> 20240115 2024-03-31T14:49:07.465 INFO:teuthology.orchestra.run.smithi016.stderr: * [new tag] 20240220 -> 20240220 2024-03-31T14:49:07.465 INFO:teuthology.orchestra.run.smithi016.stderr: * [new tag] 20240312 -> 20240312 2024-03-31T14:49:09.883 INFO:teuthology.orchestra.run.smithi089.stderr:From git://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware 2024-03-31T14:49:09.883 INFO:teuthology.orchestra.run.smithi089.stderr: * [new branch] main -> origin/main 2024-03-31T14:49:09.911 INFO:teuthology.orchestra.run.smithi089.stderr: * [new tag] 20190312 -> 20190312 2024-03-31T14:49:09.911 INFO:teuthology.orchestra.run.smithi089.stderr: * [new tag] 20190416 -> 20190416 2024-03-31T14:49:09.911 INFO:teuthology.orchestra.run.smithi089.stderr: * [new tag] 20190514 -> 20190514 2024-03-31T14:49:09.911 INFO:teuthology.orchestra.run.smithi089.stderr: * [new tag] 20190618 -> 20190618 2024-03-31T14:49:09.911 INFO:teuthology.orchestra.run.smithi089.stderr: * [new tag] 20190717 -> 20190717 2024-03-31T14:49:09.911 INFO:teuthology.orchestra.run.smithi089.stderr: * [new tag] 20190815 -> 20190815 2024-03-31T14:49:09.911 INFO:teuthology.orchestra.run.smithi089.stderr: * [new tag] 20190923 -> 20190923 2024-03-31T14:49:09.911 INFO:teuthology.orchestra.run.smithi089.stderr: * [new tag] 20191022 -> 20191022 2024-03-31T14:49:09.912 INFO:teuthology.orchestra.run.smithi089.stderr: * [new tag] 20191215 -> 20191215 2024-03-31T14:49:09.914 INFO:teuthology.orchestra.run.smithi089.stderr: * [new tag] 20200122 -> 20200122 2024-03-31T14:49:09.915 INFO:teuthology.orchestra.run.smithi089.stderr: * [new tag] 20200316 -> 20200316 2024-03-31T14:49:09.915 INFO:teuthology.orchestra.run.smithi089.stderr: * [new tag] 20200421 -> 20200421 2024-03-31T14:49:09.915 INFO:teuthology.orchestra.run.smithi089.stderr: * [new tag] 20200519 -> 20200519 2024-03-31T14:49:09.915 INFO:teuthology.orchestra.run.smithi089.stderr: * [new tag] 20200619 -> 20200619 2024-03-31T14:49:09.915 INFO:teuthology.orchestra.run.smithi089.stderr: * [new tag] 20200721 -> 20200721 2024-03-31T14:49:09.915 INFO:teuthology.orchestra.run.smithi089.stderr: * [new tag] 20200817 -> 20200817 2024-03-31T14:49:09.915 INFO:teuthology.orchestra.run.smithi089.stderr: * [new tag] 20200918 -> 20200918 2024-03-31T14:49:09.915 INFO:teuthology.orchestra.run.smithi089.stderr: * [new tag] 20201022 -> 20201022 2024-03-31T14:49:09.915 INFO:teuthology.orchestra.run.smithi089.stderr: * [new tag] 20201118 -> 20201118 2024-03-31T14:49:09.915 INFO:teuthology.orchestra.run.smithi089.stderr: * [new tag] 20201218 -> 20201218 2024-03-31T14:49:09.915 INFO:teuthology.orchestra.run.smithi089.stderr: * [new tag] 20210208 -> 20210208 2024-03-31T14:49:09.915 INFO:teuthology.orchestra.run.smithi089.stderr: * [new tag] 20210315 -> 20210315 2024-03-31T14:49:09.915 INFO:teuthology.orchestra.run.smithi089.stderr: * [new tag] 20210511 -> 20210511 2024-03-31T14:49:09.915 INFO:teuthology.orchestra.run.smithi089.stderr: * [new tag] 20210716 -> 20210716 2024-03-31T14:49:09.915 INFO:teuthology.orchestra.run.smithi089.stderr: * [new tag] 20210818 -> 20210818 2024-03-31T14:49:09.916 INFO:teuthology.orchestra.run.smithi089.stderr: * [new tag] 20210919 -> 20210919 2024-03-31T14:49:09.916 INFO:teuthology.orchestra.run.smithi089.stderr: * [new tag] 20211027 -> 20211027 2024-03-31T14:49:09.916 INFO:teuthology.orchestra.run.smithi089.stderr: * [new tag] 20211216 -> 20211216 2024-03-31T14:49:09.916 INFO:teuthology.orchestra.run.smithi089.stderr: * [new tag] 20220209 -> 20220209 2024-03-31T14:49:09.916 INFO:teuthology.orchestra.run.smithi089.stderr: * [new tag] 20220310 -> 20220310 2024-03-31T14:49:09.916 INFO:teuthology.orchestra.run.smithi089.stderr: * [new tag] 20220411 -> 20220411 2024-03-31T14:49:09.916 INFO:teuthology.orchestra.run.smithi089.stderr: * [new tag] 20220509 -> 20220509 2024-03-31T14:49:09.916 INFO:teuthology.orchestra.run.smithi089.stderr: * [new tag] 20220610 -> 20220610 2024-03-31T14:49:09.916 INFO:teuthology.orchestra.run.smithi089.stderr: * [new tag] 20220708 -> 20220708 2024-03-31T14:49:09.918 INFO:teuthology.orchestra.run.smithi089.stderr: * [new tag] 20220815 -> 20220815 2024-03-31T14:49:09.918 INFO:teuthology.orchestra.run.smithi089.stderr: * [new tag] 20220913 -> 20220913 2024-03-31T14:49:09.918 INFO:teuthology.orchestra.run.smithi089.stderr: * [new tag] 20221012 -> 20221012 2024-03-31T14:49:09.919 INFO:teuthology.orchestra.run.smithi089.stderr: * [new tag] 20221109 -> 20221109 2024-03-31T14:49:09.919 INFO:teuthology.orchestra.run.smithi089.stderr: * [new tag] 20221214 -> 20221214 2024-03-31T14:49:09.919 INFO:teuthology.orchestra.run.smithi089.stderr: * [new tag] 20230117 -> 20230117 2024-03-31T14:49:09.919 INFO:teuthology.orchestra.run.smithi089.stderr: * [new tag] 20230210 -> 20230210 2024-03-31T14:49:09.919 INFO:teuthology.orchestra.run.smithi089.stderr: * [new tag] 20230310 -> 20230310 2024-03-31T14:49:09.919 INFO:teuthology.orchestra.run.smithi089.stderr: * [new tag] 20230404 -> 20230404 2024-03-31T14:49:09.919 INFO:teuthology.orchestra.run.smithi089.stderr: * [new tag] 20230515 -> 20230515 2024-03-31T14:49:09.919 INFO:teuthology.orchestra.run.smithi089.stderr: * [new tag] 20230625 -> 20230625 2024-03-31T14:49:09.919 INFO:teuthology.orchestra.run.smithi089.stderr: * [new tag] 20230804 -> 20230804 2024-03-31T14:49:09.919 INFO:teuthology.orchestra.run.smithi089.stderr: * [new tag] 20230919 -> 20230919 2024-03-31T14:49:09.919 INFO:teuthology.orchestra.run.smithi089.stderr: * [new tag] 20231030 -> 20231030 2024-03-31T14:49:09.919 INFO:teuthology.orchestra.run.smithi089.stderr: * [new tag] 20231111 -> 20231111 2024-03-31T14:49:09.920 INFO:teuthology.orchestra.run.smithi089.stderr: * [new tag] 20231211 -> 20231211 2024-03-31T14:49:09.920 INFO:teuthology.orchestra.run.smithi089.stderr: * [new tag] 20240115 -> 20240115 2024-03-31T14:49:09.920 INFO:teuthology.orchestra.run.smithi089.stderr: * [new tag] 20240220 -> 20240220 2024-03-31T14:49:09.920 INFO:teuthology.orchestra.run.smithi089.stderr: * [new tag] 20240312 -> 20240312 2024-03-31T14:49:11.879 INFO:teuthology.orchestra.run.smithi016.stderr:Updating files: 48% (1756/3598) Updating files: 49% (1764/3598) Updating files: 50% (1799/3598) Updating files: 51% (1835/3598) Updating files: 52% (1871/3598) Updating files: 53% (1907/3598) Updating files: 54% (1943/3598) Updating files: 55% (1979/3598) Updating files: 56% (2015/3598) Updating files: 57% (2051/3598) Updating files: 58% (2087/3598) Updating files: 59% (2123/3598) Updating files: 60% (2159/3598) Updating files: 61% (2195/3598) Updating files: 61% (2224/3598) Updating files: 62% (2231/3598) Updating files: 63% (2267/3598) Updating files: 64% (2303/3598) Updating files: 65% (2339/3598) Updating files: 66% (2375/3598) Updating files: 67% (2411/3598) Updating files: 68% (2447/3598) Updating files: 69% (2483/3598) Updating files: 70% (2519/3598) Updating files: 71% (2555/3598) Updating files: 71% (2580/3598) Updating files: 72% (2591/3598) Updating files: 73% (2627/3598) Updating files: 74% (2663/3598) Updating files: 75% (2699/3598) Updating files: 76% (2735/3598) Updating files: 77% (2771/3598) Updating files: 78% (2807/3598) Updating files: 79% (2843/3598) Updating files: 80% (2879/3598) Updating files: 81% (2915/3598) Updating files: 82% (2951/3598) Updating files: 83% (2987/3598) Updating files: 83% (3008/3598) Updating files: 84% (3023/3598) Updating files: 85% (3059/3598) Updating files: 86% (3095/3598) Updating files: 87% (3131/3598) Updating files: 88% (3167/3598) Updating files: 89% (3203/3598) Updating files: 90% (3239/3598) Updating files: 91% (3275/3598) Updating files: 92% (3311/3598) Updating files: 93% (3347/3598) Updating files: 94% (3383/3598) Updating files: 95% (3419/3598) Updating files: 96% (3455/3598) Updating files: 97% (3491/3598) Updating files: 98% (3527/3598) Updating files: 99% (3563/3598) Updating files: 100% (3598/3598) Updating files: 100% (3598/3598), done. 2024-03-31T14:49:11.885 INFO:teuthology.orchestra.run.smithi016.stdout:HEAD is now at e068ccf0 Merge branch 'robot/pr-0-1711100280' into 'main' 2024-03-31T14:49:11.907 INFO:teuthology.task.kernel:Downloading kernel 1df0c7e8a2354c42d738e5454ba83ea7332deac2 on mon.a... 2024-03-31T14:49:11.907 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=1df0c7e8a2354c42d738e5454ba83ea7332deac2 2024-03-31T14:49:12.199 INFO:teuthology.task.kernel:fetching, builder baseurl is https://4.chacra.ceph.com/r/kernel/testing/1df0c7e8a2354c42d738e5454ba83ea7332deac2/ubuntu/jammy/flavors/default/pool/main/l/linux-6.8.0-g1df0c7e8a235/ 2024-03-31T14:49:12.199 DEBUG:teuthology.orchestra.run.smithi016:> rm -f /tmp/linux-image.deb && echo linux-image-6.8.0-g1df0c7e8a235_6.8.0-g1df0c7e8a235-1_amd64.deb | wget -nv -O /tmp/linux-image.deb --base=https://4.chacra.ceph.com/r/kernel/testing/1df0c7e8a2354c42d738e5454ba83ea7332deac2/ubuntu/jammy/flavors/default/pool/main/l/linux-6.8.0-g1df0c7e8a235/ --input-file=- 2024-03-31T14:49:12.202 DEBUG:teuthology.task.kernel:Waiting for download/copy to ubuntu@smithi016.front.sepia.ceph.com to complete... 2024-03-31T14:49:13.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31 14:49:14 URL:https://4.chacra.ceph.com/r/kernel/testing/1df0c7e8a2354c42d738e5454ba83ea7332deac2/ubuntu/jammy/flavors/default/pool/main/l/linux-6.8.0-g1df0c7e8a235/linux-image-6.8.0-g1df0c7e8a235_6.8.0-g1df0c7e8a235-1_amd64.deb [39278344/39278344] -> "/tmp/linux-image.deb" [1] 2024-03-31T14:49:13.995 INFO:teuthology.orchestra.run.smithi016.stderr:FINISHED --2024-03-31 14:49:14-- 2024-03-31T14:49:13.995 INFO:teuthology.orchestra.run.smithi016.stderr:Total wall clock time: 1.8s 2024-03-31T14:49:13.995 INFO:teuthology.orchestra.run.smithi016.stderr:Downloaded: 1 files, 37M in 1.6s (22.8 MB/s) 2024-03-31T14:49:14.000 INFO:teuthology.task.kernel:Installing kernel 1df0c7e8a2354c42d738e5454ba83ea7332deac2 on mon.a... 2024-03-31T14:49:14.000 DEBUG:teuthology.orchestra.run.smithi016:> sudo dpkg -i /tmp/linux-image.deb 2024-03-31T14:49:14.108 INFO:teuthology.orchestra.run.smithi089.stderr:Updating files: 51% (1850/3598) Updating files: 52% (1871/3598) Updating files: 53% (1907/3598) Updating files: 54% (1943/3598) Updating files: 55% (1979/3598) Updating files: 56% (2015/3598) Updating files: 57% (2051/3598) Updating files: 58% (2087/3598) Updating files: 59% (2123/3598) Updating files: 60% (2159/3598) Updating files: 61% (2195/3598) Updating files: 62% (2231/3598) Updating files: 63% (2267/3598) Updating files: 64% (2303/3598) Updating files: 64% (2327/3598) Updating files: 65% (2339/3598) Updating files: 66% (2375/3598) Updating files: 67% (2411/3598) Updating files: 68% (2447/3598) Updating files: 69% (2483/3598) Updating files: 70% (2519/3598) Updating files: 71% (2555/3598) Updating files: 72% (2591/3598) Updating files: 73% (2627/3598) Updating files: 74% (2663/3598) Updating files: 75% (2699/3598) Updating files: 76% (2735/3598) Updating files: 77% (2771/3598) Updating files: 77% (2785/3598) Updating files: 78% (2807/3598) Updating files: 79% (2843/3598) Updating files: 80% (2879/3598) Updating files: 81% (2915/3598) Updating files: 82% (2951/3598) Updating files: 83% (2987/3598) Updating files: 84% (3023/3598) Updating files: 85% (3059/3598) Updating files: 86% (3095/3598) Updating files: 87% (3131/3598) Updating files: 88% (3167/3598) Updating files: 89% (3203/3598) Updating files: 89% (3217/3598) Updating files: 90% (3239/3598) Updating files: 91% (3275/3598) Updating files: 92% (3311/3598) Updating files: 93% (3347/3598) Updating files: 94% (3383/3598) Updating files: 95% (3419/3598) Updating files: 96% (3455/3598) Updating files: 97% (3491/3598) Updating files: 98% (3527/3598) Updating files: 99% (3563/3598) Updating files: 100% (3598/3598) Updating files: 100% (3598/3598), done. 2024-03-31T14:49:14.113 INFO:teuthology.orchestra.run.smithi089.stdout:HEAD is now at e068ccf0 Merge branch 'robot/pr-0-1711100280' into 'main' 2024-03-31T14:49:14.134 INFO:teuthology.task.kernel:Downloading kernel 1df0c7e8a2354c42d738e5454ba83ea7332deac2 on mon.b... 2024-03-31T14:49:14.134 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=1df0c7e8a2354c42d738e5454ba83ea7332deac2 2024-03-31T14:49:14.255 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package linux-image-6.8.0-g1df0c7e8a235. 2024-03-31T14:49:14.287 INFO:teuthology.task.kernel:fetching, builder baseurl is https://4.chacra.ceph.com/r/kernel/testing/1df0c7e8a2354c42d738e5454ba83ea7332deac2/ubuntu/jammy/flavors/default/pool/main/l/linux-6.8.0-g1df0c7e8a235/ 2024-03-31T14:49:14.288 DEBUG:teuthology.orchestra.run.smithi089:> rm -f /tmp/linux-image.deb && echo linux-image-6.8.0-g1df0c7e8a235_6.8.0-g1df0c7e8a235-1_amd64.deb | wget -nv -O /tmp/linux-image.deb --base=https://4.chacra.ceph.com/r/kernel/testing/1df0c7e8a2354c42d738e5454ba83ea7332deac2/ubuntu/jammy/flavors/default/pool/main/l/linux-6.8.0-g1df0c7e8a235/ --input-file=- 2024-03-31T14:49:14.290 DEBUG:teuthology.task.kernel:Waiting for download/copy to ubuntu@smithi089.front.sepia.ceph.com to complete... 2024-03-31T14:49:15.946 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31 14:49:15 URL:https://4.chacra.ceph.com/r/kernel/testing/1df0c7e8a2354c42d738e5454ba83ea7332deac2/ubuntu/jammy/flavors/default/pool/main/l/linux-6.8.0-g1df0c7e8a235/linux-image-6.8.0-g1df0c7e8a235_6.8.0-g1df0c7e8a235-1_amd64.deb [39278344/39278344] -> "/tmp/linux-image.deb" [1] 2024-03-31T14:49:15.946 INFO:teuthology.orchestra.run.smithi089.stderr:FINISHED --2024-03-31 14:49:15-- 2024-03-31T14:49:15.947 INFO:teuthology.orchestra.run.smithi089.stderr:Total wall clock time: 1.6s 2024-03-31T14:49:15.947 INFO:teuthology.orchestra.run.smithi089.stderr:Downloaded: 1 files, 37M in 1.4s (26.0 MB/s) 2024-03-31T14:49:15.951 INFO:teuthology.task.kernel:Installing kernel 1df0c7e8a2354c42d738e5454ba83ea7332deac2 on mon.b... 2024-03-31T14:49:15.951 DEBUG:teuthology.orchestra.run.smithi089:> sudo dpkg -i /tmp/linux-image.deb 2024-03-31T14:49:16.185 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package linux-image-6.8.0-g1df0c7e8a235. 2024-03-31T14:49:16.277 INFO:teuthology.orchestra.run.smithi016.stdout:(Reading database ... 129542 files and directories currently installed.) 2024-03-31T14:49:16.282 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack /tmp/linux-image.deb ... 2024-03-31T14:49:16.500 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking linux-image-6.8.0-g1df0c7e8a235 (6.8.0-g1df0c7e8a235-1) ... 2024-03-31T14:49:18.154 INFO:teuthology.orchestra.run.smithi089.stdout:(Reading database ... 129542 files and directories currently installed.) 2024-03-31T14:49:18.161 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack /tmp/linux-image.deb ... 2024-03-31T14:49:18.357 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking linux-image-6.8.0-g1df0c7e8a235 (6.8.0-g1df0c7e8a235-1) ... 2024-03-31T14:49:19.697 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up linux-image-6.8.0-g1df0c7e8a235 (6.8.0-g1df0c7e8a235-1) ... 2024-03-31T14:49:19.808 INFO:teuthology.orchestra.run.smithi016.stderr:update-initramfs: Generating /boot/initrd.img-6.8.0-g1df0c7e8a235 2024-03-31T14:49:21.337 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up linux-image-6.8.0-g1df0c7e8a235 (6.8.0-g1df0c7e8a235-1) ... 2024-03-31T14:49:21.444 INFO:teuthology.orchestra.run.smithi089.stderr:update-initramfs: Generating /boot/initrd.img-6.8.0-g1df0c7e8a235 2024-03-31T14:49:42.385 INFO:teuthology.orchestra.run.smithi089.stderr:Sourcing file `/etc/default/grub' 2024-03-31T14:49:42.390 INFO:teuthology.orchestra.run.smithi016.stderr:Sourcing file `/etc/default/grub' 2024-03-31T14:49:42.400 INFO:teuthology.orchestra.run.smithi089.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-03-31T14:49:42.404 INFO:teuthology.orchestra.run.smithi016.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-03-31T14:49:42.419 INFO:teuthology.orchestra.run.smithi016.stderr:Generating grub configuration file ... 2024-03-31T14:49:42.420 INFO:teuthology.orchestra.run.smithi089.stderr:Generating grub configuration file ... 2024-03-31T14:49:42.716 INFO:teuthology.orchestra.run.smithi089.stderr:Found linux image: /boot/vmlinuz-6.8.0-g1df0c7e8a235 2024-03-31T14:49:42.730 INFO:teuthology.orchestra.run.smithi016.stderr:Found linux image: /boot/vmlinuz-6.8.0-g1df0c7e8a235 2024-03-31T14:49:42.732 INFO:teuthology.orchestra.run.smithi089.stderr:Found initrd image: /boot/initrd.img-6.8.0-g1df0c7e8a235 2024-03-31T14:49:42.746 INFO:teuthology.orchestra.run.smithi016.stderr:Found initrd image: /boot/initrd.img-6.8.0-g1df0c7e8a235 2024-03-31T14:49:43.002 INFO:teuthology.orchestra.run.smithi089.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-03-31T14:49:43.017 INFO:teuthology.orchestra.run.smithi089.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-03-31T14:49:43.020 INFO:teuthology.orchestra.run.smithi016.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-03-31T14:49:43.030 INFO:teuthology.orchestra.run.smithi016.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-03-31T14:49:43.078 INFO:teuthology.orchestra.run.smithi089.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-03-31T14:49:43.084 INFO:teuthology.orchestra.run.smithi089.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-03-31T14:49:43.094 INFO:teuthology.orchestra.run.smithi016.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-03-31T14:49:43.100 INFO:teuthology.orchestra.run.smithi016.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-03-31T14:49:43.240 INFO:teuthology.orchestra.run.smithi089.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2024-03-31T14:49:43.240 INFO:teuthology.orchestra.run.smithi089.stderr:Systems on them will not be added to the GRUB boot configuration. 2024-03-31T14:49:43.240 INFO:teuthology.orchestra.run.smithi089.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-03-31T14:49:43.245 INFO:teuthology.orchestra.run.smithi016.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2024-03-31T14:49:43.245 INFO:teuthology.orchestra.run.smithi016.stderr:Systems on them will not be added to the GRUB boot configuration. 2024-03-31T14:49:43.245 INFO:teuthology.orchestra.run.smithi016.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-03-31T14:49:43.276 INFO:teuthology.orchestra.run.smithi016.stderr:done 2024-03-31T14:49:43.280 INFO:teuthology.orchestra.run.smithi089.stderr:done 2024-03-31T14:49:43.433 DEBUG:teuthology.orchestra.run.smithi016:> dpkg-deb -c /tmp/linux-image.deb 2024-03-31T14:49:43.436 DEBUG:teuthology.orchestra.run.smithi089:> dpkg-deb -c /tmp/linux-image.deb 2024-03-31T14:49:43.440 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./ 2024-03-31T14:49:43.440 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./boot/ 2024-03-31T14:49:43.440 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 4474334 2024-03-25 03:29 ./boot/System.map-6.8.0-g1df0c7e8a235 2024-03-31T14:49:43.444 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./ 2024-03-31T14:49:43.444 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./boot/ 2024-03-31T14:49:43.444 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 4474334 2024-03-25 03:29 ./boot/System.map-6.8.0-g1df0c7e8a235 2024-03-31T14:49:43.454 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 170899 2024-03-25 03:29 ./boot/config-6.8.0-g1df0c7e8a235 2024-03-31T14:49:43.454 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 11551744 2024-03-25 03:29 ./boot/vmlinuz-6.8.0-g1df0c7e8a235 2024-03-31T14:49:43.456 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 170899 2024-03-25 03:29 ./boot/config-6.8.0-g1df0c7e8a235 2024-03-31T14:49:43.457 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 11551744 2024-03-25 03:29 ./boot/vmlinuz-6.8.0-g1df0c7e8a235 2024-03-31T14:49:43.475 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./etc/ 2024-03-31T14:49:43.475 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./etc/kernel/ 2024-03-31T14:49:43.476 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./etc/kernel/postinst.d/ 2024-03-31T14:49:43.476 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./etc/kernel/postrm.d/ 2024-03-31T14:49:43.476 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./etc/kernel/preinst.d/ 2024-03-31T14:49:43.476 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./etc/kernel/prerm.d/ 2024-03-31T14:49:43.476 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/ 2024-03-31T14:49:43.476 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/ 2024-03-31T14:49:43.476 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/ 2024-03-31T14:49:43.476 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/ 2024-03-31T14:49:43.476 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/ 2024-03-31T14:49:43.476 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/ 2024-03-31T14:49:43.476 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/ 2024-03-31T14:49:43.476 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 490441 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/aesni-intel.ko 2024-03-31T14:49:43.476 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 25289 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/blowfish-x86_64.ko 2024-03-31T14:49:43.477 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 29409 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/camellia-aesni-avx-x86_64.ko 2024-03-31T14:49:43.477 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 30201 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/camellia-aesni-avx2.ko 2024-03-31T14:49:43.477 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 93073 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/camellia-x86_64.ko 2024-03-31T14:49:43.477 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 106689 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/cast5-avx-x86_64.ko 2024-03-31T14:49:43.477 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 152137 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/cast6-avx-x86_64.ko 2024-03-31T14:49:43.477 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 12449 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/crc32-pclmul.ko 2024-03-31T14:49:43.477 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 13673 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/crct10dif-pclmul.ko 2024-03-31T14:49:43.477 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 83849 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/des3_ede-x86_64.ko 2024-03-31T14:49:43.477 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 18641 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/ghash-clmulni-intel.ko 2024-03-31T14:49:43.477 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 47177 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/serpent-avx-x86_64.ko 2024-03-31T14:49:43.481 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 47169 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/serpent-avx2.ko 2024-03-31T14:49:43.481 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 51689 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/serpent-sse2-x86_64.ko 2024-03-31T14:49:43.481 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 39593 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/sha1-ssse3.ko 2024-03-31T14:49:43.481 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 42529 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/sha256-ssse3.ko 2024-03-31T14:49:43.481 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 58481 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/sha512-ssse3.ko 2024-03-31T14:49:43.481 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 50385 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/twofish-avx-x86_64.ko 2024-03-31T14:49:43.481 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 31953 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/twofish-x86_64-3way.ko 2024-03-31T14:49:43.481 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 13521 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/twofish-x86_64.ko 2024-03-31T14:49:43.481 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/kernel/ 2024-03-31T14:49:43.481 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/kernel/cpu/ 2024-03-31T14:49:43.481 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/kernel/cpu/mce/ 2024-03-31T14:49:43.481 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 41425 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/kernel/cpu/mce/mce-inject.ko 2024-03-31T14:49:43.482 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 13585 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/kernel/cpuid.ko 2024-03-31T14:49:43.482 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 17841 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/kernel/msr.ko 2024-03-31T14:49:43.482 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/kvm/ 2024-03-31T14:49:43.482 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 354225 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/kvm/kvm-amd.ko 2024-03-31T14:49:43.482 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 886129 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/kvm/kvm-intel.ko 2024-03-31T14:49:43.482 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 2294793 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/kvm/kvm.ko 2024-03-31T14:49:43.483 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./etc/ 2024-03-31T14:49:43.483 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./etc/kernel/ 2024-03-31T14:49:43.484 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./etc/kernel/postinst.d/ 2024-03-31T14:49:43.484 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./etc/kernel/postrm.d/ 2024-03-31T14:49:43.484 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./etc/kernel/preinst.d/ 2024-03-31T14:49:43.484 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./etc/kernel/prerm.d/ 2024-03-31T14:49:43.484 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/ 2024-03-31T14:49:43.484 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/ 2024-03-31T14:49:43.484 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/ 2024-03-31T14:49:43.484 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/ 2024-03-31T14:49:43.484 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/ 2024-03-31T14:49:43.484 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/ 2024-03-31T14:49:43.484 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/ 2024-03-31T14:49:43.484 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 490441 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/aesni-intel.ko 2024-03-31T14:49:43.484 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 25289 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/blowfish-x86_64.ko 2024-03-31T14:49:43.484 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 29409 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/camellia-aesni-avx-x86_64.ko 2024-03-31T14:49:43.485 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 30201 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/camellia-aesni-avx2.ko 2024-03-31T14:49:43.485 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 93073 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/camellia-x86_64.ko 2024-03-31T14:49:43.485 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 106689 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/cast5-avx-x86_64.ko 2024-03-31T14:49:43.485 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 152137 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/cast6-avx-x86_64.ko 2024-03-31T14:49:43.485 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 12449 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/crc32-pclmul.ko 2024-03-31T14:49:43.485 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 13673 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/crct10dif-pclmul.ko 2024-03-31T14:49:43.485 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 83849 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/des3_ede-x86_64.ko 2024-03-31T14:49:43.485 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 18641 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/ghash-clmulni-intel.ko 2024-03-31T14:49:43.485 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 47177 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/serpent-avx-x86_64.ko 2024-03-31T14:49:43.485 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 47169 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/serpent-avx2.ko 2024-03-31T14:49:43.485 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 51689 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/serpent-sse2-x86_64.ko 2024-03-31T14:49:43.485 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 39593 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/sha1-ssse3.ko 2024-03-31T14:49:43.485 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 42529 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/sha256-ssse3.ko 2024-03-31T14:49:43.485 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 58481 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/sha512-ssse3.ko 2024-03-31T14:49:43.486 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 50385 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/twofish-avx-x86_64.ko 2024-03-31T14:49:43.486 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 31953 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/twofish-x86_64-3way.ko 2024-03-31T14:49:43.486 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 13521 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/twofish-x86_64.ko 2024-03-31T14:49:43.486 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/kernel/ 2024-03-31T14:49:43.486 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/kernel/cpu/ 2024-03-31T14:49:43.486 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/kernel/cpu/mce/ 2024-03-31T14:49:43.486 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 41425 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/kernel/cpu/mce/mce-inject.ko 2024-03-31T14:49:43.486 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 13585 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/kernel/cpuid.ko 2024-03-31T14:49:43.486 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 17841 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/kernel/msr.ko 2024-03-31T14:49:43.486 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/kvm/ 2024-03-31T14:49:43.486 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 354225 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/kvm/kvm-amd.ko 2024-03-31T14:49:43.486 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 886129 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/kvm/kvm-intel.ko 2024-03-31T14:49:43.486 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 2294793 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/kvm/kvm.ko 2024-03-31T14:49:43.828 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/ 2024-03-31T14:49:43.828 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 53833 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/af_alg.ko 2024-03-31T14:49:43.828 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 23433 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/algif_hash.ko 2024-03-31T14:49:43.828 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 21585 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/algif_skcipher.ko 2024-03-31T14:49:43.828 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 17105 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/ansi_cprng.ko 2024-03-31T14:49:43.828 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 19545 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/anubis.ko 2024-03-31T14:49:43.828 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 9577 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/arc4.ko 2024-03-31T14:49:43.828 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/asymmetric_keys/ 2024-03-31T14:49:43.828 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 10081 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/asymmetric_keys/pkcs7_test_key.ko 2024-03-31T14:49:43.828 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/async_tx/ 2024-03-31T14:49:43.829 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 11145 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/async_tx/async_memcpy.ko 2024-03-31T14:49:43.829 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 19377 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/async_tx/async_pq.ko 2024-03-31T14:49:43.829 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 21449 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/async_tx/async_raid6_recov.ko 2024-03-31T14:49:43.829 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 13729 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/async_tx/async_tx.ko 2024-03-31T14:49:43.829 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 17649 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/async_tx/async_xor.ko 2024-03-31T14:49:43.829 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 17817 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/async_tx/raid6test.ko 2024-03-31T14:49:43.829 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 20329 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/authenc.ko 2024-03-31T14:49:43.829 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 22345 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/authencesn.ko 2024-03-31T14:49:43.829 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 21721 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/blake2b_generic.ko 2024-03-31T14:49:43.829 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 12625 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/blowfish_common.ko 2024-03-31T14:49:43.829 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 10433 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/blowfish_generic.ko 2024-03-31T14:49:43.829 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 38705 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/camellia_generic.ko 2024-03-31T14:49:43.829 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 27225 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/cast5_generic.ko 2024-03-31T14:49:43.830 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 17649 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/cast6_generic.ko 2024-03-31T14:49:43.830 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 10433 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/cast_common.ko 2024-03-31T14:49:43.830 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 35177 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/ccm.ko 2024-03-31T14:49:43.830 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 14561 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/cmac.ko 2024-03-31T14:49:43.830 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 9481 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/crc32_generic.ko 2024-03-31T14:49:43.830 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 47673 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/cryptd.ko 2024-03-31T14:49:43.830 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 39465 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/crypto_engine.ko 2024-03-31T14:49:43.830 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 26913 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/crypto_simd.ko 2024-03-31T14:49:43.830 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 24937 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/crypto_user.ko 2024-03-31T14:49:43.830 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 17641 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/cts.ko 2024-03-31T14:49:43.830 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 10809 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/des_generic.ko 2024-03-31T14:49:43.830 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 15577 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/dh_generic.ko 2024-03-31T14:49:43.830 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 10905 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/echainiv.ko 2024-03-31T14:49:43.830 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 25233 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/essiv.ko 2024-03-31T14:49:43.831 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 17897 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/fcrypt.ko 2024-03-31T14:49:43.831 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 26801 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/khazad.ko 2024-03-31T14:49:43.831 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 15561 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/lrw.ko 2024-03-31T14:49:43.831 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 11417 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/lz4.ko 2024-03-31T14:49:43.831 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 11433 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/lz4hc.ko 2024-03-31T14:49:43.831 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 11601 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/md4.ko 2024-03-31T14:49:43.831 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 9625 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/michael_mic.ko 2024-03-31T14:49:43.831 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 10577 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/pcbc.ko 2024-03-31T14:49:43.831 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 18721 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/pcrypt.ko 2024-03-31T14:49:43.831 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 15953 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/rmd160.ko 2024-03-31T14:49:43.831 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 29297 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/seed.ko 2024-03-31T14:49:43.831 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 26985 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/serpent_generic.ko 2024-03-31T14:49:43.831 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 143025 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/tcrypt.ko 2024-03-31T14:49:43.831 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 10785 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/tea.ko 2024-03-31T14:49:43.832 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 28673 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/twofish_common.ko 2024-03-31T14:49:43.832 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 13065 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/twofish_generic.ko 2024-03-31T14:49:43.832 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 17305 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/vmac.ko 2024-03-31T14:49:43.832 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 36089 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/wp512.ko 2024-03-31T14:49:43.832 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 13945 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/xcbc.ko 2024-03-31T14:49:43.832 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 26569 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/xor.ko 2024-03-31T14:49:43.832 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 18929 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/xts.ko 2024-03-31T14:49:43.832 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 9377 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/xxhash_generic.ko 2024-03-31T14:49:43.832 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ 2024-03-31T14:49:43.832 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/acpi/ 2024-03-31T14:49:43.832 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 18465 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/acpi/acpi_extlog.ko 2024-03-31T14:49:43.832 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 26881 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/acpi/acpi_ipmi.ko 2024-03-31T14:49:43.832 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 28145 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/acpi/acpi_pad.ko 2024-03-31T14:49:43.832 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/acpi/apei/ 2024-03-31T14:49:43.833 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 31961 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/acpi/apei/einj.ko 2024-03-31T14:49:43.833 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 12705 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/acpi/ec_sys.ko 2024-03-31T14:49:43.833 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 14833 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/acpi/platform_profile.ko 2024-03-31T14:49:43.833 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 24305 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/acpi/sbs.ko 2024-03-31T14:49:43.833 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 19425 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/acpi/sbshc.ko 2024-03-31T14:49:43.833 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 104417 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/acpi/video.ko 2024-03-31T14:49:43.833 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/ 2024-03-31T14:49:43.833 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 21497 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/acard-ahci.ko 2024-03-31T14:49:43.833 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 89929 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/ahci.ko 2024-03-31T14:49:43.833 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 14625 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/ahci_platform.ko 2024-03-31T14:49:43.833 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 88305 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/libahci.ko 2024-03-31T14:49:43.833 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 34881 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/libahci_platform.ko 2024-03-31T14:49:43.833 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 17033 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_acpi.ko 2024-03-31T14:49:43.834 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 30337 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_ali.ko 2024-03-31T14:49:43.834 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 33409 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_amd.ko 2024-03-31T14:49:43.834 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 21161 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_artop.ko 2024-03-31T14:49:43.834 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 21017 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_atiixp.ko 2024-03-31T14:49:43.834 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 24841 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_atp867x.ko 2024-03-31T14:49:43.834 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 17553 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_cmd640.ko 2024-03-31T14:49:43.834 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 25081 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_cmd64x.ko 2024-03-31T14:49:43.834 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 14945 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_cypress.ko 2024-03-31T14:49:43.834 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 17745 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_efar.ko 2024-03-31T14:49:43.834 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 21841 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_hpt366.ko 2024-03-31T14:49:43.834 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 33873 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_hpt37x.ko 2024-03-31T14:49:43.834 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 23025 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_hpt3x2n.ko 2024-03-31T14:49:43.834 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 16817 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_hpt3x3.ko 2024-03-31T14:49:43.835 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 16889 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_it8213.ko 2024-03-31T14:49:43.835 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 35169 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_it821x.ko 2024-03-31T14:49:43.835 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 12785 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_jmicron.ko 2024-03-31T14:49:43.835 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 52169 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_legacy.ko 2024-03-31T14:49:43.835 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 15081 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_marvell.ko 2024-03-31T14:49:43.835 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 16177 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_mpiix.ko 2024-03-31T14:49:43.835 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 12729 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_netcell.ko 2024-03-31T14:49:43.835 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 16713 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_ninja32.ko 2024-03-31T14:49:43.835 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 15001 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_ns87410.ko 2024-03-31T14:49:43.835 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 17257 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_ns87415.ko 2024-03-31T14:49:43.835 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 16465 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_oldpiix.ko 2024-03-31T14:49:43.835 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 15273 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_opti.ko 2024-03-31T14:49:43.835 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 20577 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_optidma.ko 2024-03-31T14:49:43.835 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 32481 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_pcmcia.ko 2024-03-31T14:49:43.836 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 37689 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_pdc2027x.ko 2024-03-31T14:49:43.836 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 20449 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_pdc202xx_old.ko 2024-03-31T14:49:43.836 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 14121 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_piccolo.ko 2024-03-31T14:49:43.836 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 15345 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_radisys.ko 2024-03-31T14:49:43.836 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 19425 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_rdc.ko 2024-03-31T14:49:43.836 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 15225 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_rz1000.ko 2024-03-31T14:49:43.836 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 13777 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_sch.ko 2024-03-31T14:49:43.836 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 22993 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_serverworks.ko 2024-03-31T14:49:43.836 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 23353 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_sil680.ko 2024-03-31T14:49:43.836 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 19593 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_sl82c105.ko 2024-03-31T14:49:43.836 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 15905 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_triflex.ko 2024-03-31T14:49:43.836 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 30337 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_via.ko 2024-03-31T14:49:43.836 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 22697 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pdc_adma.ko 2024-03-31T14:49:43.836 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 29017 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_inic162x.ko 2024-03-31T14:49:43.837 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 83249 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_mv.ko 2024-03-31T14:49:43.837 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 70297 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_nv.ko 2024-03-31T14:49:43.837 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 37273 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_promise.ko 2024-03-31T14:49:43.837 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 22937 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_qstor.ko 2024-03-31T14:49:43.837 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 29177 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_sil.ko 2024-03-31T14:49:43.837 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 38169 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_sil24.ko 2024-03-31T14:49:43.837 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 17849 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_sis.ko 2024-03-31T14:49:43.837 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 20345 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_svw.ko 2024-03-31T14:49:43.837 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 44697 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_sx4.ko 2024-03-31T14:49:43.837 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 15441 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_uli.ko 2024-03-31T14:49:43.837 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 34969 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_via.ko 2024-03-31T14:49:43.837 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 19969 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_vsc.ko 2024-03-31T14:49:43.837 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/auxdisplay/ 2024-03-31T14:49:43.837 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 16193 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/auxdisplay/ks0108.ko 2024-03-31T14:49:43.838 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/bcma/ 2024-03-31T14:49:43.838 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 124385 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/bcma/bcma.ko 2024-03-31T14:49:43.838 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/ 2024-03-31T14:49:43.838 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/aoe/ 2024-03-31T14:49:43.838 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 113801 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/aoe/aoe.ko 2024-03-31T14:49:43.838 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/drbd/ 2024-03-31T14:49:43.838 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 964745 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/drbd/drbd.ko 2024-03-31T14:49:43.838 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 169697 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/floppy.ko 2024-03-31T14:49:43.838 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/mtip32xx/ 2024-03-31T14:49:43.838 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 94209 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/mtip32xx/mtip32xx.ko 2024-03-31T14:49:43.838 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 116329 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/nbd.ko 2024-03-31T14:49:43.838 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/null_blk/ 2024-03-31T14:49:43.838 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 110321 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/null_blk/null_blk.ko 2024-03-31T14:49:43.839 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 98185 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/pktcdvd.ko 2024-03-31T14:49:43.839 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 263633 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/rbd.ko 2024-03-31T14:49:43.839 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/xen-blkback/ 2024-03-31T14:49:43.839 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 98105 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/xen-blkback/xen-blkback.ko 2024-03-31T14:49:43.839 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/zram/ 2024-03-31T14:49:43.839 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 58761 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/zram/zram.ko 2024-03-31T14:49:43.839 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/ 2024-03-31T14:49:43.839 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 30609 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/applicom.ko 2024-03-31T14:49:43.839 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 11905 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/hangcheck-timer.ko 2024-03-31T14:49:43.839 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/hw_random/ 2024-03-31T14:49:43.839 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 13793 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/hw_random/amd-rng.ko 2024-03-31T14:49:43.839 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 18705 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/hw_random/intel-rng.ko 2024-03-31T14:49:43.839 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 13065 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/hw_random/timeriomem-rng.ko 2024-03-31T14:49:43.839 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 11065 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/hw_random/via-rng.ko 2024-03-31T14:49:43.840 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 15529 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/hw_random/virtio-rng.ko 2024-03-31T14:49:43.840 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/ipmi/ 2024-03-31T14:49:43.840 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 30609 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/ipmi/ipmi_devintf.ko 2024-03-31T14:49:43.840 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 148513 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/ipmi/ipmi_msghandler.ko 2024-03-31T14:49:43.840 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 26401 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/ipmi/ipmi_poweroff.ko 2024-03-31T14:49:43.840 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 139305 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/ipmi/ipmi_si.ko 2024-03-31T14:49:43.840 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 70865 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/ipmi/ipmi_ssif.ko 2024-03-31T14:49:43.840 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 53409 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/ipmi/ipmi_watchdog.ko 2024-03-31T14:49:43.840 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 36225 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/lp.ko 2024-03-31T14:49:43.840 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/mwave/ 2024-03-31T14:49:43.840 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 84001 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/mwave/mwave.ko 2024-03-31T14:49:43.840 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 25777 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/nvram.ko 2024-03-31T14:49:43.840 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 32073 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/ppdev.ko 2024-03-31T14:49:43.840 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 51017 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/tlclk.ko 2024-03-31T14:49:43.841 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/tpm/ 2024-03-31T14:49:43.841 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 16297 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/tpm/tpm_atmel.ko 2024-03-31T14:49:43.841 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 15777 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/tpm/tpm_i2c_atmel.ko 2024-03-31T14:49:43.841 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 23369 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/tpm/tpm_i2c_infineon.ko 2024-03-31T14:49:43.841 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 32761 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/tpm/tpm_i2c_nuvoton.ko 2024-03-31T14:49:43.841 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 30017 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/tpm/tpm_infineon.ko 2024-03-31T14:49:43.841 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 22601 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/tpm/tpm_nsc.ko 2024-03-31T14:49:43.841 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 23457 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/tpm/xen-tpmfront.ko 2024-03-31T14:49:43.841 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/xillybus/ 2024-03-31T14:49:43.841 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 16897 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/xillybus/xillybus_class.ko 2024-03-31T14:49:43.841 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 50841 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/xillybus/xillybus_core.ko 2024-03-31T14:49:43.841 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 12225 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/xillybus/xillybus_pcie.ko 2024-03-31T14:49:43.841 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/clk/ 2024-03-31T14:49:43.841 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 12385 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/clk/clk-palmas.ko 2024-03-31T14:49:43.842 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 47721 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/clk/clk-si5351.ko 2024-03-31T14:49:43.842 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 11353 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/clk/clk-twl6040.ko 2024-03-31T14:49:43.842 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 19145 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/clk/clk-wm831x.ko 2024-03-31T14:49:43.842 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/cpufreq/ 2024-03-31T14:49:43.842 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 10369 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/cpufreq/amd_freq_sensitivity.ko 2024-03-31T14:49:43.842 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 16729 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/cpufreq/p4-clockmod.ko 2024-03-31T14:49:43.842 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 25785 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/cpufreq/speedstep-lib.ko 2024-03-31T14:49:43.842 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/ 2024-03-31T14:49:43.842 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/ccp/ 2024-03-31T14:49:43.842 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 70425 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/ccp/ccp-crypto.ko 2024-03-31T14:49:43.842 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 246777 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/ccp/ccp.ko 2024-03-31T14:49:43.842 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/intel/ 2024-03-31T14:49:43.842 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/intel/qat/ 2024-03-31T14:49:43.843 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/intel/qat/qat_common/ 2024-03-31T14:49:43.843 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 570217 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/intel/qat/qat_common/intel_qat.ko 2024-03-31T14:49:43.843 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/intel/qat/qat_dh895xcc/ 2024-03-31T14:49:43.843 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 24033 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/intel/qat/qat_dh895xcc/qat_dh895xcc.ko 2024-03-31T14:49:43.843 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 19897 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/padlock-aes.ko 2024-03-31T14:49:43.843 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 21737 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/padlock-sha.ko 2024-03-31T14:49:43.843 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/virtio/ 2024-03-31T14:49:43.843 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 70409 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/virtio/virtio_crypto.ko 2024-03-31T14:49:43.843 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/dca/ 2024-03-31T14:49:43.843 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 29689 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/dca/dca.ko 2024-03-31T14:49:43.843 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/dma/ 2024-03-31T14:49:43.843 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/dma/dw/ 2024-03-31T14:49:43.843 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 15761 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/dma/dw/dw_dmac.ko 2024-03-31T14:49:43.843 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/dma/ioat/ 2024-03-31T14:49:43.843 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 122793 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/dma/ioat/ioatdma.ko 2024-03-31T14:49:43.844 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/ 2024-03-31T14:49:43.844 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 80297 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/amd64_edac.ko 2024-03-31T14:49:43.844 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 35665 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/e752x_edac.ko 2024-03-31T14:49:43.844 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 41633 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/edac_mce_amd.ko 2024-03-31T14:49:43.844 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 18001 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/i3000_edac.ko 2024-03-31T14:49:43.844 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 17961 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/i3200_edac.ko 2024-03-31T14:49:43.844 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 22433 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/i5100_edac.ko 2024-03-31T14:49:43.844 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 25489 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/i5400_edac.ko 2024-03-31T14:49:43.844 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 24809 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/i7300_edac.ko 2024-03-31T14:49:43.844 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 51937 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/i7core_edac.ko 2024-03-31T14:49:43.844 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 18265 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/i82975x_edac.ko 2024-03-31T14:49:43.844 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 19921 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/ie31200_edac.ko 2024-03-31T14:49:43.844 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 56729 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/sb_edac.ko 2024-03-31T14:49:43.844 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 17433 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/x38_edac.ko 2024-03-31T14:49:43.845 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/extcon/ 2024-03-31T14:49:43.845 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 12569 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/extcon/extcon-gpio.ko 2024-03-31T14:49:43.845 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 27081 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/extcon/extcon-max14577.ko 2024-03-31T14:49:43.845 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 30209 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/extcon/extcon-max77693.ko 2024-03-31T14:49:43.845 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 24641 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/extcon/extcon-max8997.ko 2024-03-31T14:49:43.845 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 26433 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/extcon/extcon-palmas.ko 2024-03-31T14:49:43.845 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 23609 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/extcon/extcon-rt8973a.ko 2024-03-31T14:49:43.845 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 27681 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/extcon/extcon-sm5502.ko 2024-03-31T14:49:43.845 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/firmware/ 2024-03-31T14:49:43.845 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 34873 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/firmware/dmi-sysfs.ko 2024-03-31T14:49:43.845 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/firmware/efi/ 2024-03-31T14:49:43.845 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 15417 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/firmware/efi/efi-pstore.ko 2024-03-31T14:49:43.845 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 20625 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/firmware/iscsi_ibft.ko 2024-03-31T14:49:43.845 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/gpio/ 2024-03-31T14:49:43.846 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 12209 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/gpio/gpio-adp5520.ko 2024-03-31T14:49:43.846 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 20049 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/gpio/gpio-amd8111.ko 2024-03-31T14:49:43.846 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 14889 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/gpio/gpio-arizona.ko 2024-03-31T14:49:43.846 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 13737 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/gpio/gpio-da9052.ko 2024-03-31T14:49:43.846 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 11825 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/gpio/gpio-da9055.ko 2024-03-31T14:49:43.846 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 21961 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/gpio/gpio-dln2.ko 2024-03-31T14:49:43.846 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 77745 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/gpio/gpio-f7188x.ko 2024-03-31T14:49:43.846 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 31297 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/gpio/gpio-generic.ko 2024-03-31T14:49:43.846 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 19473 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/gpio/gpio-ich.ko 2024-03-31T14:49:43.846 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 11809 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/gpio/gpio-janz-ttl.ko 2024-03-31T14:49:43.846 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 14009 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/gpio/gpio-kempld.ko 2024-03-31T14:49:43.846 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 12441 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/gpio/gpio-lp3943.ko 2024-03-31T14:49:43.846 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 9409 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/gpio/gpio-max7300.ko 2024-03-31T14:49:43.846 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 11801 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/gpio/gpio-max730x.ko 2024-03-31T14:49:43.847 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 21969 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/gpio/gpio-max732x.ko 2024-03-31T14:49:43.847 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 25041 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/gpio/gpio-ml-ioh.ko 2024-03-31T14:49:43.847 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 47441 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/gpio/gpio-pca953x.ko 2024-03-31T14:49:43.847 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 26721 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/gpio/gpio-pcf857x.ko 2024-03-31T14:49:43.847 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 14273 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/gpio/gpio-rdc321x.ko 2024-03-31T14:49:43.847 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 20497 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/gpio/gpio-sch.ko 2024-03-31T14:49:43.847 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 19401 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/gpio/gpio-sch311x.ko 2024-03-31T14:49:43.847 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 12049 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/gpio/gpio-tps65912.ko 2024-03-31T14:49:43.847 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 20961 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/gpio/gpio-twl4030.ko 2024-03-31T14:49:43.847 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 11321 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/gpio/gpio-twl6040.ko 2024-03-31T14:49:43.847 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 20497 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/gpio/gpio-viperboard.ko 2024-03-31T14:49:43.847 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 15273 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/gpio/gpio-vx855.ko 2024-03-31T14:49:43.847 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 16481 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/gpio/gpio-wm831x.ko 2024-03-31T14:49:43.847 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 11521 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/gpio/gpio-wm8350.ko 2024-03-31T14:49:43.848 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 16945 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/gpio/gpio-wm8994.ko 2024-03-31T14:49:43.848 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/ 2024-03-31T14:49:43.848 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 11473 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-a4tech.ko 2024-03-31T14:49:43.848 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 39409 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-apple.ko 2024-03-31T14:49:43.848 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 14385 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-appleir.ko 2024-03-31T14:49:43.848 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 8249 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-aureal.ko 2024-03-31T14:49:43.848 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 15073 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-axff.ko 2024-03-31T14:49:43.848 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 10945 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-belkin.ko 2024-03-31T14:49:43.848 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 10209 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-cherry.ko 2024-03-31T14:49:43.848 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 15937 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-chicony.ko 2024-03-31T14:49:43.848 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 60657 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-cp2112.ko 2024-03-31T14:49:43.848 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 10945 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-cypress.ko 2024-03-31T14:49:43.848 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 16289 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-dr.ko 2024-03-31T14:49:43.849 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 10593 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-elecom.ko 2024-03-31T14:49:43.849 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 17569 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-elo.ko 2024-03-31T14:49:43.849 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 13353 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-emsff.ko 2024-03-31T14:49:43.849 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 10001 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-ezkey.ko 2024-03-31T14:49:43.849 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 14209 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-gaff.ko 2024-03-31T14:49:43.849 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 9129 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-generic.ko 2024-03-31T14:49:43.849 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 17025 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-gt683r.ko 2024-03-31T14:49:43.849 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 12577 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-gyration.ko 2024-03-31T14:49:43.849 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 10065 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-holtek-kbd.ko 2024-03-31T14:49:43.849 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 10361 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-holtek-mouse.ko 2024-03-31T14:49:43.849 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 14345 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-holtekff.ko 2024-03-31T14:49:43.849 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 19001 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-hyperv.ko 2024-03-31T14:49:43.849 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 10385 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-icade.ko 2024-03-31T14:49:43.849 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 9137 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-kensington.ko 2024-03-31T14:49:43.850 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 8441 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-keytouch.ko 2024-03-31T14:49:43.850 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 17105 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-kye.ko 2024-03-31T14:49:43.850 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 10705 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-lcpower.ko 2024-03-31T14:49:43.850 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 20401 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-led.ko 2024-03-31T14:49:43.850 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 42841 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-lenovo.ko 2024-03-31T14:49:43.850 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 30593 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-lg-g15.ko 2024-03-31T14:49:43.850 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 57969 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-logitech-dj.ko 2024-03-31T14:49:43.850 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 101537 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-logitech-hidpp.ko 2024-03-31T14:49:43.850 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 80897 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-logitech.ko 2024-03-31T14:49:43.850 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 30697 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-magicmouse.ko 2024-03-31T14:49:43.850 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 21865 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-microsoft.ko 2024-03-31T14:49:43.850 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 10257 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-monterey.ko 2024-03-31T14:49:43.850 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 50273 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-multitouch.ko 2024-03-31T14:49:43.851 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 32489 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-ntrig.ko 2024-03-31T14:49:43.851 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 8753 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-ortek.ko 2024-03-31T14:49:43.851 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 9017 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-penmount.ko 2024-03-31T14:49:43.851 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 12185 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-petalynx.ko 2024-03-31T14:49:43.851 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 72841 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-picolcd.ko 2024-03-31T14:49:43.851 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 14129 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-pl.ko 2024-03-31T14:49:43.851 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 15001 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-plantronics.ko 2024-03-31T14:49:43.851 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 8321 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-primax.ko 2024-03-31T14:49:43.851 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 35521 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-rmi.ko 2024-03-31T14:49:43.851 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 20729 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-arvo.ko 2024-03-31T14:49:43.851 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 13953 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-common.ko 2024-03-31T14:49:43.851 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 31321 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-isku.ko 2024-03-31T14:49:43.851 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 29625 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-kone.ko 2024-03-31T14:49:43.851 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 31017 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-koneplus.ko 2024-03-31T14:49:43.851 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 21025 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-konepure.ko 2024-03-31T14:49:43.852 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 29377 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-kovaplus.ko 2024-03-31T14:49:43.852 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 13201 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-lua.ko 2024-03-31T14:49:43.852 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 29209 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-pyra.ko 2024-03-31T14:49:43.852 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 27833 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-ryos.ko 2024-03-31T14:49:43.852 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 19417 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-savu.ko 2024-03-31T14:49:43.852 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 23233 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat.ko 2024-03-31T14:49:43.852 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 13577 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-saitek.ko 2024-03-31T14:49:43.852 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 15089 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-samsung.ko 2024-03-31T14:49:43.852 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 36713 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-sensor-hub.ko 2024-03-31T14:49:43.852 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 13505 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-sjoy.ko 2024-03-31T14:49:43.852 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 51153 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-sony.ko 2024-03-31T14:49:43.852 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 8217 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-speedlink.ko 2024-03-31T14:49:43.852 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 25193 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-steelseries.ko 2024-03-31T14:49:43.852 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 9713 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-sunplus.ko 2024-03-31T14:49:43.853 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 16889 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-thrustmaster.ko 2024-03-31T14:49:43.853 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 10441 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-tivo.ko 2024-03-31T14:49:43.853 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 14993 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-tmff.ko 2024-03-31T14:49:43.853 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 14281 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-topseed.ko 2024-03-31T14:49:43.853 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 16369 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-twinhan.ko 2024-03-31T14:49:43.853 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 70977 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-uclogic.ko 2024-03-31T14:49:43.853 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 10825 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-waltop.ko 2024-03-31T14:49:43.853 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 143377 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-wiimote.ko 2024-03-31T14:49:43.853 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 8057 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-xinmo.ko 2024-03-31T14:49:43.853 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 13225 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-zpff.ko 2024-03-31T14:49:43.853 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 16561 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-zydacron.ko 2024-03-31T14:49:43.853 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 269209 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid.ko 2024-03-31T14:49:43.853 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 32225 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/uhid.ko 2024-03-31T14:49:43.853 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/usbhid/ 2024-03-31T14:49:43.854 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 125625 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/usbhid/usbhid.ko 2024-03-31T14:49:43.854 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 19041 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/usbhid/usbkbd.ko 2024-03-31T14:49:43.854 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 14857 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/usbhid/usbmouse.ko 2024-03-31T14:49:43.854 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 218865 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/wacom.ko 2024-03-31T14:49:43.854 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hv/ 2024-03-31T14:49:43.854 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 64297 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hv/hv_balloon.ko 2024-03-31T14:49:43.854 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 90089 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hv/hv_utils.ko 2024-03-31T14:49:43.854 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 304913 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hv/hv_vmbus.ko 2024-03-31T14:49:43.854 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ 2024-03-31T14:49:43.854 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 66793 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/abituguru.ko 2024-03-31T14:49:43.854 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 61809 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/abituguru3.ko 2024-03-31T14:49:43.854 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 34569 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/acpi_power_meter.ko 2024-03-31T14:49:43.854 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 17409 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ad7414.ko 2024-03-31T14:49:43.855 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 19217 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ad7418.ko 2024-03-31T14:49:43.855 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 25153 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adc128d818.ko 2024-03-31T14:49:43.855 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 31633 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adm1025.ko 2024-03-31T14:49:43.855 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 97881 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adm1026.ko 2024-03-31T14:49:43.855 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 19649 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adm1029.ko 2024-03-31T14:49:43.855 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 45457 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adm1031.ko 2024-03-31T14:49:43.855 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 26193 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adm9240.ko 2024-03-31T14:49:43.855 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 14673 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ads7828.ko 2024-03-31T14:49:43.855 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 10673 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adt7410.ko 2024-03-31T14:49:43.855 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 21953 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adt7411.ko 2024-03-31T14:49:43.855 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 74313 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adt7462.ko 2024-03-31T14:49:43.855 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 39489 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adt7470.ko 2024-03-31T14:49:43.855 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 69305 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adt7475.ko 2024-03-31T14:49:43.855 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 19537 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adt7x10.ko 2024-03-31T14:49:43.856 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 38977 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/amc6821.ko 2024-03-31T14:49:43.856 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 55105 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/applesmc.ko 2024-03-31T14:49:43.856 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 49817 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/asb100.ko 2024-03-31T14:49:43.856 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 58521 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/asc7621.ko 2024-03-31T14:49:43.856 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 40721 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/asus_atk0110.ko 2024-03-31T14:49:43.856 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 17129 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/atxp1.ko 2024-03-31T14:49:43.856 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 28129 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/coretemp.ko 2024-03-31T14:49:43.856 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 24489 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/da9052-hwmon.ko 2024-03-31T14:49:43.856 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 17161 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/da9055-hwmon.ko 2024-03-31T14:49:43.856 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 49233 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/dell-smm-hwmon.ko 2024-03-31T14:49:43.856 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 99505 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/dme1737.ko 2024-03-31T14:49:43.856 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 19393 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ds1621.ko 2024-03-31T14:49:43.856 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 15225 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ds620.ko 2024-03-31T14:49:43.856 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 29833 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/emc1403.ko 2024-03-31T14:49:43.857 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 33089 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/emc2103.ko 2024-03-31T14:49:43.857 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 30225 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/emc6w201.ko 2024-03-31T14:49:43.857 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 66889 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/f71805f.ko 2024-03-31T14:49:43.857 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 94593 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/f71882fg.ko 2024-03-31T14:49:43.857 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 41305 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/f75375s.ko 2024-03-31T14:49:43.857 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 21673 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/fam15h_power.ko 2024-03-31T14:49:43.857 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 55353 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/fschmd.ko 2024-03-31T14:49:43.857 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 14385 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/g760a.ko 2024-03-31T14:49:43.857 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 23921 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/g762.ko 2024-03-31T14:49:43.857 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 46705 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/gl518sm.ko 2024-03-31T14:49:43.857 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 45433 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/gl520sm.ko 2024-03-31T14:49:43.857 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 13177 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/hih6130.ko 2024-03-31T14:49:43.857 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 10017 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/hwmon-vid.ko 2024-03-31T14:49:43.857 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 11385 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/i5500_temp.ko 2024-03-31T14:49:43.858 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 22505 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/i5k_amb.ko 2024-03-31T14:49:43.858 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 34433 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ibmaem.ko 2024-03-31T14:49:43.858 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 21161 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ibmpex.ko 2024-03-31T14:49:43.858 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 24777 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ina209.ko 2024-03-31T14:49:43.858 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 13305 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ina2xx.ko 2024-03-31T14:49:43.858 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 140129 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/it87.ko 2024-03-31T14:49:43.858 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 18745 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/jc42.ko 2024-03-31T14:49:43.858 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 24553 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/k10temp.ko 2024-03-31T14:49:43.858 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 13785 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/k8temp.ko 2024-03-31T14:49:43.858 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 18825 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lineage-pem.ko 2024-03-31T14:49:43.858 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 48505 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm63.ko 2024-03-31T14:49:43.858 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 16601 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm73.ko 2024-03-31T14:49:43.858 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 38841 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm75.ko 2024-03-31T14:49:43.858 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 19441 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm77.ko 2024-03-31T14:49:43.859 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 34497 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm78.ko 2024-03-31T14:49:43.859 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 31649 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm80.ko 2024-03-31T14:49:43.859 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 16793 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm83.ko 2024-03-31T14:49:43.859 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 72737 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm85.ko 2024-03-31T14:49:43.859 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 46057 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm87.ko 2024-03-31T14:49:43.859 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 37817 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm90.ko 2024-03-31T14:49:43.859 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 19649 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm92.ko 2024-03-31T14:49:43.859 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 108745 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm93.ko 2024-03-31T14:49:43.859 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 33297 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm95234.ko 2024-03-31T14:49:43.859 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 16105 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm95241.ko 2024-03-31T14:49:43.859 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 18017 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm95245.ko 2024-03-31T14:49:43.859 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 23929 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ltc2945.ko 2024-03-31T14:49:43.859 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 13697 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ltc4151.ko 2024-03-31T14:49:43.859 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 15097 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ltc4215.ko 2024-03-31T14:49:43.860 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 14793 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ltc4222.ko 2024-03-31T14:49:43.860 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 14729 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ltc4245.ko 2024-03-31T14:49:43.860 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 13153 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ltc4260.ko 2024-03-31T14:49:43.860 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 14897 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ltc4261.ko 2024-03-31T14:49:43.860 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 33393 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/max16065.ko 2024-03-31T14:49:43.860 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 18337 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/max1619.ko 2024-03-31T14:49:43.860 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 24257 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/max1668.ko 2024-03-31T14:49:43.860 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 19873 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/max197.ko 2024-03-31T14:49:43.860 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 27033 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/max6639.ko 2024-03-31T14:49:43.860 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 23209 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/max6650.ko 2024-03-31T14:49:43.860 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 35681 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/max6697.ko 2024-03-31T14:49:43.861 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 18481 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/mc13783-adc.ko 2024-03-31T14:49:43.861 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 10489 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/mcp3021.ko 2024-03-31T14:49:43.861 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 15833 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/menf21bmc_hwmon.ko 2024-03-31T14:49:43.861 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 47785 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/nct6683.ko 2024-03-31T14:49:43.861 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 119841 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/nct6775-core.ko 2024-03-31T14:49:43.861 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 61433 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/nct6775.ko 2024-03-31T14:49:43.861 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 59641 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/nct7802.ko 2024-03-31T14:49:43.861 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 82713 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pc87360.ko 2024-03-31T14:49:43.861 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 54337 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pc87427.ko 2024-03-31T14:49:43.861 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 18377 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pcf8591.ko 2024-03-31T14:49:43.861 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/ 2024-03-31T14:49:43.861 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 21257 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/adm1275.ko 2024-03-31T14:49:43.861 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 16961 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/lm25066.ko 2024-03-31T14:49:43.861 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 28233 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/ltc2978.ko 2024-03-31T14:49:43.862 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 9489 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/max16064.ko 2024-03-31T14:49:43.862 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 17561 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/max34440.ko 2024-03-31T14:49:43.862 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 10081 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/max8688.ko 2024-03-31T14:49:43.862 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 15409 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/pmbus.ko 2024-03-31T14:49:43.862 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 82473 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/pmbus_core.ko 2024-03-31T14:49:43.862 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 8641 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/tps40422.ko 2024-03-31T14:49:43.862 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 30633 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/ucd9000.ko 2024-03-31T14:49:43.862 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 14977 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/ucd9200.ko 2024-03-31T14:49:43.862 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 20585 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/zl6100.ko 2024-03-31T14:49:43.862 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 12921 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/powr1220.ko 2024-03-31T14:49:43.862 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 21673 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/sch5627.ko 2024-03-31T14:49:43.862 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 28721 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/sch5636.ko 2024-03-31T14:49:43.862 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 29145 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/sch56xx-common.ko 2024-03-31T14:49:43.862 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 32041 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/sht15.ko 2024-03-31T14:49:43.863 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 13409 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/sht21.ko 2024-03-31T14:49:43.863 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 16737 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/shtc1.ko 2024-03-31T14:49:43.863 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 40481 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/sis5595.ko 2024-03-31T14:49:43.863 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 19801 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/smsc47b397.ko 2024-03-31T14:49:43.863 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 40353 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/smsc47m1.ko 2024-03-31T14:49:43.863 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 37281 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/smsc47m192.ko 2024-03-31T14:49:43.863 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 26457 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/thmc50.ko 2024-03-31T14:49:43.863 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 16961 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/tmp102.ko 2024-03-31T14:49:43.863 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 14001 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/tmp103.ko 2024-03-31T14:49:43.863 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 21161 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/tmp401.ko 2024-03-31T14:49:43.863 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 19441 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/tmp421.ko 2024-03-31T14:49:43.863 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 19617 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/via-cputemp.ko 2024-03-31T14:49:43.863 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 43441 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/via686a.ko 2024-03-31T14:49:43.863 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 52361 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/vt1211.ko 2024-03-31T14:49:43.864 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 53201 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/vt8231.ko 2024-03-31T14:49:43.864 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 58257 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/w83627ehf.ko 2024-03-31T14:49:43.864 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 79833 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/w83627hf.ko 2024-03-31T14:49:43.864 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 71249 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/w83781d.ko 2024-03-31T14:49:43.864 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 63129 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/w83791d.ko 2024-03-31T14:49:43.864 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 71225 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/w83792d.ko 2024-03-31T14:49:43.864 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 101377 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/w83793.ko 2024-03-31T14:49:43.864 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 119409 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/w83795.ko 2024-03-31T14:49:43.864 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 17209 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/w83l785ts.ko 2024-03-31T14:49:43.864 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 35521 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/w83l786ng.ko 2024-03-31T14:49:43.864 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 13721 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/wm831x-hwmon.ko 2024-03-31T14:49:43.864 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 10521 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/wm8350-hwmon.ko 2024-03-31T14:49:43.864 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/ 2024-03-31T14:49:43.864 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/algos/ 2024-03-31T14:49:43.865 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 19945 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/algos/i2c-algo-bit.ko 2024-03-31T14:49:43.865 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 19241 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/algos/i2c-algo-pca.ko 2024-03-31T14:49:43.865 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/ 2024-03-31T14:49:43.865 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 30857 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-ali1535.ko 2024-03-31T14:49:43.865 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 32473 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-ali1563.ko 2024-03-31T14:49:43.865 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 30385 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-ali15x3.ko 2024-03-31T14:49:43.865 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 17833 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-amd756-s4882.ko 2024-03-31T14:49:43.865 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 28073 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-amd756.ko 2024-03-31T14:49:43.865 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 16033 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-amd8111.ko 2024-03-31T14:49:43.865 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 15049 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-cbus-gpio.ko 2024-03-31T14:49:43.865 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 6825 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-ccgx-ucsi.ko 2024-03-31T14:49:43.865 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 13849 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-cros-ec-tunnel.ko 2024-03-31T14:49:43.865 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 50289 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-designware-core.ko 2024-03-31T14:49:43.865 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 23793 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-designware-pci.ko 2024-03-31T14:49:43.866 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 19889 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-designware-platform.ko 2024-03-31T14:49:43.866 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 17233 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-diolan-u2c.ko 2024-03-31T14:49:43.866 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 12401 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-dln2.ko 2024-03-31T14:49:43.866 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 17489 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-gpio.ko 2024-03-31T14:49:43.866 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 67681 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-i801.ko 2024-03-31T14:49:43.866 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 22337 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-isch.ko 2024-03-31T14:49:43.866 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 47793 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-ismt.ko 2024-03-31T14:49:43.866 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 17617 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-kempld.ko 2024-03-31T14:49:43.866 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 17769 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-nforce2-s4985.ko 2024-03-31T14:49:43.866 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 23537 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-nforce2.ko 2024-03-31T14:49:43.866 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 26161 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-ocores.ko 2024-03-31T14:49:43.866 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 25041 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-parport.ko 2024-03-31T14:49:43.866 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 16393 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-pca-platform.ko 2024-03-31T14:49:43.866 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 50969 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-piix4.ko 2024-03-31T14:49:43.867 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 15177 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-robotfuzz-osif.ko 2024-03-31T14:49:43.867 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 18553 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-scmi.ko 2024-03-31T14:49:43.867 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 11713 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-simtec.ko 2024-03-31T14:49:43.867 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 22209 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-sis5595.ko 2024-03-31T14:49:43.867 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 29257 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-sis630.ko 2024-03-31T14:49:43.867 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 20225 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-sis96x.ko 2024-03-31T14:49:43.867 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 19137 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-taos-evm.ko 2024-03-31T14:49:43.867 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 19705 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-tiny-usb.ko 2024-03-31T14:49:43.867 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 13705 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-via.ko 2024-03-31T14:49:43.867 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 27617 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-viapro.ko 2024-03-31T14:49:43.867 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 18937 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-viperboard.ko 2024-03-31T14:49:43.867 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 47561 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-xiic.ko 2024-03-31T14:49:43.867 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 22353 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/i2c-mux.ko 2024-03-31T14:49:43.867 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 18721 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/i2c-smbus.ko 2024-03-31T14:49:43.868 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 26977 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/i2c-stub.ko 2024-03-31T14:49:43.868 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/muxes/ 2024-03-31T14:49:43.868 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 15089 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/muxes/i2c-mux-gpio.ko 2024-03-31T14:49:43.868 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 13353 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/muxes/i2c-mux-pca9541.ko 2024-03-31T14:49:43.868 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 29633 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/muxes/i2c-mux-pca954x.ko 2024-03-31T14:49:43.868 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ 2024-03-31T14:49:43.868 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/core/ 2024-03-31T14:49:43.868 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 273513 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/core/ib_cm.ko 2024-03-31T14:49:43.868 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 883777 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/core/ib_core.ko 2024-03-31T14:49:43.868 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 69177 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/core/ib_umad.ko 2024-03-31T14:49:43.868 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 324233 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/core/ib_uverbs.ko 2024-03-31T14:49:43.868 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 97825 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/core/iw_cm.ko 2024-03-31T14:49:43.868 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 282329 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/core/rdma_cm.ko 2024-03-31T14:49:43.868 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 66049 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/core/rdma_ucm.ko 2024-03-31T14:49:43.869 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/hw/ 2024-03-31T14:49:43.869 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/hw/cxgb4/ 2024-03-31T14:49:43.869 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 493129 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/hw/cxgb4/iw_cxgb4.ko 2024-03-31T14:49:43.869 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/hw/mlx4/ 2024-03-31T14:49:43.869 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 420529 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/hw/mlx4/mlx4_ib.ko 2024-03-31T14:49:43.869 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/hw/mlx5/ 2024-03-31T14:49:43.869 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 741609 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko 2024-03-31T14:49:43.869 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/hw/mthca/ 2024-03-31T14:49:43.869 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 296785 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/hw/mthca/ib_mthca.ko 2024-03-31T14:49:43.869 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/hw/ocrdma/ 2024-03-31T14:49:43.869 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 153705 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/hw/ocrdma/ocrdma.ko 2024-03-31T14:49:43.869 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/hw/usnic/ 2024-03-31T14:49:43.869 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 169089 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/hw/usnic/usnic_verbs.ko 2024-03-31T14:49:43.869 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp/ 2024-03-31T14:49:43.870 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp/ipoib/ 2024-03-31T14:49:43.870 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 266529 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp/ipoib/ib_ipoib.ko 2024-03-31T14:49:43.870 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp/iser/ 2024-03-31T14:49:43.870 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 100257 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp/iser/ib_iser.ko 2024-03-31T14:49:43.870 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp/isert/ 2024-03-31T14:49:43.870 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 111329 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp/isert/ib_isert.ko 2024-03-31T14:49:43.870 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp/srp/ 2024-03-31T14:49:43.870 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 128657 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp/srp/ib_srp.ko 2024-03-31T14:49:43.870 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp/srpt/ 2024-03-31T14:49:43.870 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 129793 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp/srpt/ib_srpt.ko 2024-03-31T14:49:43.870 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/ 2024-03-31T14:49:43.870 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 10649 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/evbug.ko 2024-03-31T14:49:43.870 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 24025 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/ff-memless.ko 2024-03-31T14:49:43.870 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/gameport/ 2024-03-31T14:49:43.871 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 11777 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/gameport/emu10k1-gp.ko 2024-03-31T14:49:43.871 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 11985 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/gameport/fm801-gp.ko 2024-03-31T14:49:43.871 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 41449 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/gameport/gameport.ko 2024-03-31T14:49:43.871 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 41705 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/joydev.ko 2024-03-31T14:49:43.871 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/keyboard/ 2024-03-31T14:49:43.871 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 13209 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/keyboard/adp5520-keys.ko 2024-03-31T14:49:43.871 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 12601 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/keyboard/adp5588-keys.ko 2024-03-31T14:49:43.871 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 26737 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/keyboard/adp5589-keys.ko 2024-03-31T14:49:43.871 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 24473 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/keyboard/cros_ec_keyb.ko 2024-03-31T14:49:43.871 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 37273 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/keyboard/gpio_keys.ko 2024-03-31T14:49:43.871 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 18777 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/keyboard/gpio_keys_polled.ko 2024-03-31T14:49:43.871 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 21161 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/keyboard/lkkbd.ko 2024-03-31T14:49:43.871 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 26377 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/keyboard/lm8323.ko 2024-03-31T14:49:43.872 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 14313 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/keyboard/lm8333.ko 2024-03-31T14:49:43.872 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 24225 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/keyboard/matrix_keypad.ko 2024-03-31T14:49:43.872 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 18113 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/keyboard/max7359_keypad.ko 2024-03-31T14:49:43.872 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 15825 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/keyboard/mcs_touchkey.ko 2024-03-31T14:49:43.872 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 16385 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/keyboard/mpr121_touchkey.ko 2024-03-31T14:49:43.872 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 11489 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/keyboard/newtonkbd.ko 2024-03-31T14:49:43.872 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 10569 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/keyboard/opencores-kbd.ko 2024-03-31T14:49:43.872 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 14065 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/keyboard/qt1070.ko 2024-03-31T14:49:43.872 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 17833 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/keyboard/qt2160.ko 2024-03-31T14:49:43.872 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 23713 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/keyboard/samsung-keypad.ko 2024-03-31T14:49:43.872 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 11393 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/keyboard/stowaway.ko 2024-03-31T14:49:43.872 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 18193 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/keyboard/sunkbd.ko 2024-03-31T14:49:43.872 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 17321 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/keyboard/tca6416-keypad.ko 2024-03-31T14:49:43.872 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 16073 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/keyboard/tca8418_keypad.ko 2024-03-31T14:49:43.872 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 16697 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/keyboard/twl4030_keypad.ko 2024-03-31T14:49:43.873 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 11657 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/keyboard/xtkbd.ko 2024-03-31T14:49:43.873 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 11361 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/matrix-keymap.ko 2024-03-31T14:49:43.873 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/mouse/ 2024-03-31T14:49:43.873 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 34225 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/mouse/appletouch.ko 2024-03-31T14:49:43.873 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 34081 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/mouse/bcm5974.ko 2024-03-31T14:49:43.873 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 123177 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/mouse/cyapatp.ko 2024-03-31T14:49:43.873 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 89377 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/mouse/elan_i2c.ko 2024-03-31T14:49:43.873 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 13857 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/mouse/gpio_mouse.ko 2024-03-31T14:49:43.873 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 335857 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/mouse/psmouse.ko 2024-03-31T14:49:43.873 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 14585 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/mouse/sermouse.ko 2024-03-31T14:49:43.873 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 23777 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/mouse/synaptics_i2c.ko 2024-03-31T14:49:43.873 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 22465 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/mouse/synaptics_usb.ko 2024-03-31T14:49:43.873 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 15569 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/mouse/vsxxxaa.ko 2024-03-31T14:49:43.873 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/rmi4/ 2024-03-31T14:49:43.874 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 109553 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/rmi4/rmi_core.ko 2024-03-31T14:49:43.874 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/serio/ 2024-03-31T14:49:43.874 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 11569 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/serio/altera_ps2.ko 2024-03-31T14:49:43.874 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 16193 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/serio/arc_ps2.ko 2024-03-31T14:49:43.874 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 17681 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/serio/ct82c710.ko 2024-03-31T14:49:43.874 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 16849 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/serio/hyperv-keyboard.ko 2024-03-31T14:49:43.874 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 15737 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/serio/parkbd.ko 2024-03-31T14:49:43.874 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 13129 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/serio/pcips2.ko 2024-03-31T14:49:43.874 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 20065 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/serio/ps2mult.ko 2024-03-31T14:49:43.874 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 24049 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/serio/serio_raw.ko 2024-03-31T14:49:43.874 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 17881 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/serio/serport.ko 2024-03-31T14:49:43.874 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 13233 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/sparse-keymap.ko 2024-03-31T14:49:43.874 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/ 2024-03-31T14:49:43.874 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 13713 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/leds-88pm860x.ko 2024-03-31T14:49:43.875 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 11281 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/leds-adp5520.ko 2024-03-31T14:49:43.875 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 59649 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/leds-bd2802.ko 2024-03-31T14:49:43.875 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 24681 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/leds-blinkm.ko 2024-03-31T14:49:43.875 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 10369 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/leds-da903x.ko 2024-03-31T14:49:43.875 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 12137 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/leds-da9052.ko 2024-03-31T14:49:43.875 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 16681 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/leds-gpio.ko 2024-03-31T14:49:43.875 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 15689 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/leds-lm3530.ko 2024-03-31T14:49:43.875 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 30401 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/leds-lm3533.ko 2024-03-31T14:49:43.875 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 19249 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/leds-lm355x.ko 2024-03-31T14:49:43.875 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 19857 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/leds-lm3642.ko 2024-03-31T14:49:43.875 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 16137 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/leds-lp3944.ko 2024-03-31T14:49:43.875 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 11137 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/leds-lp8788.ko 2024-03-31T14:49:43.875 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 14825 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/leds-max8997.ko 2024-03-31T14:49:43.875 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 12441 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/leds-mc13783.ko 2024-03-31T14:49:43.876 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 13281 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/leds-menf21bmc.ko 2024-03-31T14:49:43.876 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 20625 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/leds-pca9532.ko 2024-03-31T14:49:43.876 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 20393 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/leds-pca955x.ko 2024-03-31T14:49:43.876 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 17753 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/leds-pca963x.ko 2024-03-31T14:49:43.876 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 34329 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/leds-ss4200.ko 2024-03-31T14:49:43.876 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 23897 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/leds-tca6507.ko 2024-03-31T14:49:43.876 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 15553 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/leds-wm831x-status.ko 2024-03-31T14:49:43.876 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 11745 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/leds-wm8350.ko 2024-03-31T14:49:43.876 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/trigger/ 2024-03-31T14:49:43.876 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 9881 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/trigger/ledtrig-audio.ko 2024-03-31T14:49:43.876 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 10761 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/trigger/ledtrig-backlight.ko 2024-03-31T14:49:43.876 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 8985 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/trigger/ledtrig-camera.ko 2024-03-31T14:49:43.876 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 7513 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/trigger/ledtrig-default-on.ko 2024-03-31T14:49:43.876 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 12809 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/trigger/ledtrig-gpio.ko 2024-03-31T14:49:43.877 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 14137 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/trigger/ledtrig-heartbeat.ko 2024-03-31T14:49:43.877 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 13969 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/trigger/ledtrig-oneshot.ko 2024-03-31T14:49:43.877 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 11785 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/trigger/ledtrig-timer.ko 2024-03-31T14:49:43.877 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 13961 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/trigger/ledtrig-transient.ko 2024-03-31T14:49:43.877 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/ 2024-03-31T14:49:43.877 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/bcache/ 2024-03-31T14:49:43.877 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 647033 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/bcache/bcache.ko 2024-03-31T14:49:43.877 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 40473 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-bio-prison.ko 2024-03-31T14:49:43.877 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 93313 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-bufio.ko 2024-03-31T14:49:43.877 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 47433 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-cache-smq.ko 2024-03-31T14:49:43.877 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 155801 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-cache.ko 2024-03-31T14:49:43.877 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 107929 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-crypt.ko 2024-03-31T14:49:43.877 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 23945 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-delay.ko 2024-03-31T14:49:43.877 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 53761 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-era.ko 2024-03-31T14:49:43.878 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 28209 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-flakey.ko 2024-03-31T14:49:43.878 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 41425 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-log-userspace.ko 2024-03-31T14:49:43.878 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 32641 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-log.ko 2024-03-31T14:49:43.878 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 49441 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-mirror.ko 2024-03-31T14:49:43.878 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 77865 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-multipath.ko 2024-03-31T14:49:43.878 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 16369 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-queue-length.ko 2024-03-31T14:49:43.878 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 70977 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-raid.ko 2024-03-31T14:49:43.878 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 36193 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-region-hash.ko 2024-03-31T14:49:43.878 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 15769 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-round-robin.ko 2024-03-31T14:49:43.878 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 16697 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-service-time.ko 2024-03-31T14:49:43.878 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 109433 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-snapshot.ko 2024-03-31T14:49:43.878 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 19257 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-switch.ko 2024-03-31T14:49:43.878 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 181761 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-thin-pool.ko 2024-03-31T14:49:43.878 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 53729 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-verity.ko 2024-03-31T14:49:43.878 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 8473 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-zero.ko 2024-03-31T14:49:43.879 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/persistent-data/ 2024-03-31T14:49:43.879 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 199825 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/persistent-data/dm-persistent-data.ko 2024-03-31T14:49:43.879 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 37313 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/raid0.ko 2024-03-31T14:49:43.879 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 100833 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/raid1.ko 2024-03-31T14:49:43.879 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 126361 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/raid10.ko 2024-03-31T14:49:43.879 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 368481 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/raid456.ko 2024-03-31T14:49:43.879 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/message/ 2024-03-31T14:49:43.879 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/message/fusion/ 2024-03-31T14:49:43.879 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 178041 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/message/fusion/mptbase.ko 2024-03-31T14:49:43.879 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 63233 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/message/fusion/mptctl.ko 2024-03-31T14:49:43.879 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 46241 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/message/fusion/mptfc.ko 2024-03-31T14:49:43.879 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 37113 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/message/fusion/mptlan.ko 2024-03-31T14:49:43.879 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 123225 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/message/fusion/mptsas.ko 2024-03-31T14:49:43.880 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 80913 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/message/fusion/mptscsih.ko 2024-03-31T14:49:43.880 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 52857 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/message/fusion/mptspi.ko 2024-03-31T14:49:43.880 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/ 2024-03-31T14:49:43.880 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 17209 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/88pm800.ko 2024-03-31T14:49:43.880 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 12665 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/88pm805.ko 2024-03-31T14:49:43.880 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 11097 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/88pm80x.ko 2024-03-31T14:49:43.880 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 11561 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/arizona-i2c.ko 2024-03-31T14:49:43.880 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 123625 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/arizona.ko 2024-03-31T14:49:43.880 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 11865 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/bcm590xx.ko 2024-03-31T14:49:43.880 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 17361 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/cros_ec_dev.ko 2024-03-31T14:49:43.880 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 34841 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/dln2.ko 2024-03-31T14:49:43.880 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 16585 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/janz-cmodio.ko 2024-03-31T14:49:43.880 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 53481 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/kempld-core.ko 2024-03-31T14:49:43.880 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 27425 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/lm3533-core.ko 2024-03-31T14:49:43.880 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 14889 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/lm3533-ctrlbank.ko 2024-03-31T14:49:43.881 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 12913 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/lp3943.ko 2024-03-31T14:49:43.881 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 56961 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/lpc_ich.ko 2024-03-31T14:49:43.881 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 12017 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/lpc_sch.ko 2024-03-31T14:49:43.881 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 17313 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/max8907.ko 2024-03-31T14:49:43.881 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 29753 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/mc13xxx-core.ko 2024-03-31T14:49:43.881 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 11697 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/mc13xxx-i2c.ko 2024-03-31T14:49:43.881 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 11489 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/menf21bmc.ko 2024-03-31T14:49:43.881 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 17225 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/pcf50633-adc.ko 2024-03-31T14:49:43.881 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 9049 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/pcf50633-gpio.ko 2024-03-31T14:49:43.881 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 34777 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/pcf50633.ko 2024-03-31T14:49:43.881 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 9993 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/rdc321x-southbridge.ko 2024-03-31T14:49:43.881 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 20193 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/retu-mfd.ko 2024-03-31T14:49:43.881 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 80329 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/si476x-core.ko 2024-03-31T14:49:43.882 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 48361 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/sm501.ko 2024-03-31T14:49:43.882 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 51201 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/tps65010.ko 2024-03-31T14:49:43.882 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 10257 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/tps6507x.ko 2024-03-31T14:49:43.882 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 13225 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/viperboard.ko 2024-03-31T14:49:43.882 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 10441 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/vx855.ko 2024-03-31T14:49:43.882 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 17849 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/wl1273-core.ko 2024-03-31T14:49:43.882 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 44249 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/wm8994.ko 2024-03-31T14:49:43.882 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/ 2024-03-31T14:49:43.882 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 13729 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/ad525x_dpot-i2c.ko 2024-03-31T14:49:43.882 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 48385 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/ad525x_dpot.ko 2024-03-31T14:49:43.882 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/altera-stapl/ 2024-03-31T14:49:43.882 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 46489 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/altera-stapl/altera-stapl.ko 2024-03-31T14:49:43.882 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 18289 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/apds9802als.ko 2024-03-31T14:49:43.882 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 36633 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/apds990x.ko 2024-03-31T14:49:43.883 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 45489 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/bh1770glc.ko 2024-03-31T14:49:43.883 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/c2port/ 2024-03-31T14:49:43.883 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 11041 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/c2port/c2port-duramar2150.ko 2024-03-31T14:49:43.883 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 35489 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/c2port/core.ko 2024-03-31T14:49:43.883 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/cb710/ 2024-03-31T14:49:43.883 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 25065 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/cb710/cb710.ko 2024-03-31T14:49:43.883 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 17705 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/ds1682.ko 2024-03-31T14:49:43.883 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 8793 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/dummy-irq.ko 2024-03-31T14:49:43.883 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/echo/ 2024-03-31T14:49:43.883 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 13561 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/echo/echo.ko 2024-03-31T14:49:43.883 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/eeprom/ 2024-03-31T14:49:43.883 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 39017 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/eeprom/at24.ko 2024-03-31T14:49:43.883 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 14665 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/eeprom/eeprom_93cx6.ko 2024-03-31T14:49:43.883 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 13441 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/eeprom/max6875.ko 2024-03-31T14:49:43.883 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 32905 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/enclosure.ko 2024-03-31T14:49:43.884 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/genwqe/ 2024-03-31T14:49:43.884 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 152041 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/genwqe/genwqe_card.ko 2024-03-31T14:49:43.884 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 14033 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/hmc6352.ko 2024-03-31T14:49:43.884 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 31161 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/hpilo.ko 2024-03-31T14:49:43.884 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/ibmasm/ 2024-03-31T14:49:43.884 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 72505 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/ibmasm/ibmasm.ko 2024-03-31T14:49:43.884 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 19681 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/ics932s401.ko 2024-03-31T14:49:43.884 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 21585 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/isl29003.ko 2024-03-31T14:49:43.884 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 15657 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/isl29020.ko 2024-03-31T14:49:43.884 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/lis3lv02d/ 2024-03-31T14:49:43.884 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 42409 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/lis3lv02d/lis3lv02d.ko 2024-03-31T14:49:43.884 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 14201 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/lis3lv02d/lis3lv02d_i2c.ko 2024-03-31T14:49:43.884 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/mei/ 2024-03-31T14:49:43.884 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 126137 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/mei/mei-me.ko 2024-03-31T14:49:43.885 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 56361 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/mei/mei-txe.ko 2024-03-31T14:49:43.885 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 313545 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/mei/mei.ko 2024-03-31T14:49:43.885 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 31393 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/phantom.ko 2024-03-31T14:49:43.885 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/ti-st/ 2024-03-31T14:49:43.885 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 74289 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/ti-st/st_drv.ko 2024-03-31T14:49:43.885 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 24201 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/tifm_7xx1.ko 2024-03-31T14:49:43.885 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 28681 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/tifm_core.ko 2024-03-31T14:49:43.885 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 18633 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/tsl2550.ko 2024-03-31T14:49:43.885 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 48337 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/vmw_balloon.ko 2024-03-31T14:49:43.885 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/vmw_vmci/ 2024-03-31T14:49:43.885 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 183513 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/vmw_vmci/vmw_vmci.ko 2024-03-31T14:49:43.885 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ 2024-03-31T14:49:43.885 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/bonding/ 2024-03-31T14:49:43.885 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 491561 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/bonding/bonding.ko 2024-03-31T14:49:43.885 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/dsa/ 2024-03-31T14:49:43.886 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/dsa/b53/ 2024-03-31T14:49:43.886 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 105929 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/dsa/b53/b53_common.ko 2024-03-31T14:49:43.886 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 40209 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/dsa/bcm-sf2.ko 2024-03-31T14:49:43.886 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 15937 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/dsa/mv88e6060.ko 2024-03-31T14:49:43.886 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/dsa/mv88e6xxx/ 2024-03-31T14:49:43.886 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 458905 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/dsa/mv88e6xxx/mv88e6xxx.ko 2024-03-31T14:49:43.886 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 16945 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/dummy.ko 2024-03-31T14:49:43.886 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 17985 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/eql.ko 2024-03-31T14:49:43.886 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/ 2024-03-31T14:49:43.886 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/3com/ 2024-03-31T14:49:43.886 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 45953 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/3com/3c574_cs.ko 2024-03-31T14:49:43.886 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 42185 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/3com/3c589_cs.ko 2024-03-31T14:49:43.886 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 130657 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/3com/3c59x.ko 2024-03-31T14:49:43.886 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 68777 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/3com/typhoon.ko 2024-03-31T14:49:43.887 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/8390/ 2024-03-31T14:49:43.887 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 38041 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/8390/8390.ko 2024-03-31T14:49:43.887 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 64609 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/8390/axnet_cs.ko 2024-03-31T14:49:43.887 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 27081 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/8390/ne2k-pci.ko 2024-03-31T14:49:43.887 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 101609 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/8390/pcnet_cs.ko 2024-03-31T14:49:43.887 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/adaptec/ 2024-03-31T14:49:43.887 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 54617 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/adaptec/starfire.ko 2024-03-31T14:49:43.887 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/agere/ 2024-03-31T14:49:43.887 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 69161 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/agere/et131x.ko 2024-03-31T14:49:43.887 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/alteon/ 2024-03-31T14:49:43.887 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 59985 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/alteon/acenic.ko 2024-03-31T14:49:43.887 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/altera/ 2024-03-31T14:49:43.887 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 41385 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/altera/altera_tse.ko 2024-03-31T14:49:43.887 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/amd/ 2024-03-31T14:49:43.887 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 54609 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/amd/amd8111e.ko 2024-03-31T14:49:43.888 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 49185 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/amd/nmclan_cs.ko 2024-03-31T14:49:43.888 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 87449 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/amd/pcnet32.ko 2024-03-31T14:49:43.888 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/atheros/ 2024-03-31T14:49:43.888 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/atheros/alx/ 2024-03-31T14:49:43.888 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 97185 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/atheros/alx/alx.ko 2024-03-31T14:49:43.888 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/atheros/atl1c/ 2024-03-31T14:49:43.888 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 102481 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/atheros/atl1c/atl1c.ko 2024-03-31T14:49:43.888 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/atheros/atl1e/ 2024-03-31T14:49:43.888 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 90185 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/atheros/atl1e/atl1e.ko 2024-03-31T14:49:43.888 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/atheros/atlx/ 2024-03-31T14:49:43.888 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 81313 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/atheros/atlx/atl1.ko 2024-03-31T14:49:43.888 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 69537 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/atheros/atlx/atl2.ko 2024-03-31T14:49:43.888 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/broadcom/ 2024-03-31T14:49:43.889 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 78377 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/broadcom/b44.ko 2024-03-31T14:49:43.889 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 176689 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/broadcom/bnx2.ko 2024-03-31T14:49:43.889 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/broadcom/bnx2x/ 2024-03-31T14:49:43.889 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 1434449 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/broadcom/bnx2x/bnx2x.ko 2024-03-31T14:49:43.889 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 128361 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/broadcom/cnic.ko 2024-03-31T14:49:43.889 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/broadcom/genet/ 2024-03-31T14:49:43.889 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 131265 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/broadcom/genet/genet.ko 2024-03-31T14:49:43.889 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 314321 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/broadcom/tg3.ko 2024-03-31T14:49:43.889 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/brocade/ 2024-03-31T14:49:43.889 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/brocade/bna/ 2024-03-31T14:49:43.889 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 323681 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/brocade/bna/bna.ko 2024-03-31T14:49:43.889 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio/ 2024-03-31T14:49:43.889 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio/cxgb/ 2024-03-31T14:49:43.889 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 193537 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio/cxgb/cxgb.ko 2024-03-31T14:49:43.889 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio/cxgb3/ 2024-03-31T14:49:43.890 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 347161 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio/cxgb3/cxgb3.ko 2024-03-31T14:49:43.890 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio/cxgb4/ 2024-03-31T14:49:43.890 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 1045321 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio/cxgb4/cxgb4.ko 2024-03-31T14:49:43.890 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio/cxgb4vf/ 2024-03-31T14:49:43.890 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 166129 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf.ko 2024-03-31T14:49:43.890 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio/libcxgb/ 2024-03-31T14:49:43.890 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 34849 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio/libcxgb/libcxgb.ko 2024-03-31T14:49:43.890 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/cisco/ 2024-03-31T14:49:43.890 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/cisco/enic/ 2024-03-31T14:49:43.890 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 204185 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/cisco/enic/enic.ko 2024-03-31T14:49:43.890 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dec/ 2024-03-31T14:49:43.890 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dec/tulip/ 2024-03-31T14:49:43.890 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 69425 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dec/tulip/de2104x.ko 2024-03-31T14:49:43.890 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 61417 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dec/tulip/dmfe.ko 2024-03-31T14:49:43.891 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 141153 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dec/tulip/tulip.ko 2024-03-31T14:49:43.891 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 51361 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dec/tulip/uli526x.ko 2024-03-31T14:49:43.891 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 62825 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dec/tulip/winbond-840.ko 2024-03-31T14:49:43.891 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 31201 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dec/tulip/xircom_cb.ko 2024-03-31T14:49:43.891 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dlink/ 2024-03-31T14:49:43.891 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 60665 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dlink/dl2k.ko 2024-03-31T14:49:43.891 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 62825 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dlink/sundance.ko 2024-03-31T14:49:43.891 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 45481 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dnet.ko 2024-03-31T14:49:43.891 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 25089 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/ec_bhf.ko 2024-03-31T14:49:43.891 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/emulex/ 2024-03-31T14:49:43.891 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/emulex/benet/ 2024-03-31T14:49:43.891 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 281193 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/emulex/benet/be2net.ko 2024-03-31T14:49:43.891 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 49449 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/ethoc.ko 2024-03-31T14:49:43.891 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 51777 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/fealnx.ko 2024-03-31T14:49:43.892 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/fujitsu/ 2024-03-31T14:49:43.892 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 45545 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/fujitsu/fmvj18x_cs.ko 2024-03-31T14:49:43.892 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/ 2024-03-31T14:49:43.892 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 90297 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/e100.ko 2024-03-31T14:49:43.892 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/e1000/ 2024-03-31T14:49:43.892 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 295673 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/e1000/e1000.ko 2024-03-31T14:49:43.892 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/e1000e/ 2024-03-31T14:49:43.892 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 584297 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/e1000e/e1000e.ko 2024-03-31T14:49:43.892 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/fm10k/ 2024-03-31T14:49:43.892 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 245977 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/fm10k/fm10k.ko 2024-03-31T14:49:43.892 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/i40e/ 2024-03-31T14:49:43.892 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 944449 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/i40e/i40e.ko 2024-03-31T14:49:43.892 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/iavf/ 2024-03-31T14:49:43.892 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 288481 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/iavf/iavf.ko 2024-03-31T14:49:43.893 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/igb/ 2024-03-31T14:49:43.893 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 514337 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/igb/igb.ko 2024-03-31T14:49:43.893 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/igbvf/ 2024-03-31T14:49:43.893 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 93465 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/igbvf/igbvf.ko 2024-03-31T14:49:43.893 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/ixgbe/ 2024-03-31T14:49:43.893 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 735129 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko 2024-03-31T14:49:43.893 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/ixgbevf/ 2024-03-31T14:49:43.893 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 151649 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/ixgbevf/ixgbevf.ko 2024-03-31T14:49:43.893 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 86985 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/jme.ko 2024-03-31T14:49:43.893 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/marvell/ 2024-03-31T14:49:43.893 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 21233 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/marvell/mvmdio.ko 2024-03-31T14:49:43.893 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 102433 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/marvell/skge.ko 2024-03-31T14:49:43.893 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 122601 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/marvell/sky2.ko 2024-03-31T14:49:43.893 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/mellanox/ 2024-03-31T14:49:43.893 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/mellanox/mlx4/ 2024-03-31T14:49:43.894 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 727377 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_core.ko 2024-03-31T14:49:43.894 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 278825 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_en.ko 2024-03-31T14:49:43.894 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/mellanox/mlx5/ 2024-03-31T14:49:43.894 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/mellanox/mlx5/core/ 2024-03-31T14:49:43.894 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 1122065 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.ko 2024-03-31T14:49:43.894 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/micrel/ 2024-03-31T14:49:43.894 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 52977 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/micrel/ks8842.ko 2024-03-31T14:49:43.894 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 40873 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/micrel/ks8851_common.ko 2024-03-31T14:49:43.894 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 19385 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/micrel/ks8851_par.ko 2024-03-31T14:49:43.894 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 95569 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/micrel/ksz884x.ko 2024-03-31T14:49:43.894 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/myricom/ 2024-03-31T14:49:43.894 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/myricom/myri10ge/ 2024-03-31T14:49:43.894 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 108401 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/myricom/myri10ge/myri10ge.ko 2024-03-31T14:49:43.894 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/natsemi/ 2024-03-31T14:49:43.895 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 73609 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/natsemi/natsemi.ko 2024-03-31T14:49:43.895 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 47793 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/natsemi/ns83820.ko 2024-03-31T14:49:43.895 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/neterion/ 2024-03-31T14:49:43.895 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 151193 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/neterion/s2io.ko 2024-03-31T14:49:43.895 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/nvidia/ 2024-03-31T14:49:43.895 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 136793 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/nvidia/forcedeth.ko 2024-03-31T14:49:43.895 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/packetengines/ 2024-03-31T14:49:43.895 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 50425 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/packetengines/hamachi.ko 2024-03-31T14:49:43.895 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 42977 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/packetengines/yellowfin.ko 2024-03-31T14:49:43.895 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/qlogic/ 2024-03-31T14:49:43.895 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/qlogic/netxen/ 2024-03-31T14:49:43.895 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 227841 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/qlogic/netxen/netxen_nic.ko 2024-03-31T14:49:43.895 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 87009 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/qlogic/qla3xxx.ko 2024-03-31T14:49:43.895 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/qlogic/qlcnic/ 2024-03-31T14:49:43.896 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 671985 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/qlogic/qlcnic/qlcnic.ko 2024-03-31T14:49:43.896 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/rdc/ 2024-03-31T14:49:43.896 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 42929 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/rdc/r6040.ko 2024-03-31T14:49:43.896 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/realtek/ 2024-03-31T14:49:43.896 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 67273 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/realtek/8139cp.ko 2024-03-31T14:49:43.896 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 83801 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/realtek/8139too.ko 2024-03-31T14:49:43.896 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 29153 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/realtek/atp.ko 2024-03-31T14:49:43.896 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 194057 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/realtek/r8169.ko 2024-03-31T14:49:43.896 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/samsung/ 2024-03-31T14:49:43.896 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/samsung/sxgbe/ 2024-03-31T14:49:43.896 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 104001 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/samsung/sxgbe/samsung-sxgbe.ko 2024-03-31T14:49:43.896 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/sfc/ 2024-03-31T14:49:43.896 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 1066817 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/sfc/sfc.ko 2024-03-31T14:49:43.896 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/silan/ 2024-03-31T14:49:43.896 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 47001 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/silan/sc92031.ko 2024-03-31T14:49:43.897 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/sis/ 2024-03-31T14:49:43.897 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 54721 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/sis/sis190.ko 2024-03-31T14:49:43.897 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 63193 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/sis/sis900.ko 2024-03-31T14:49:43.897 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/smsc/ 2024-03-31T14:49:43.897 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 61417 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/smsc/epic100.ko 2024-03-31T14:49:43.897 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 67289 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/smsc/smc91c92_cs.ko 2024-03-31T14:49:43.897 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 73249 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/smsc/smsc911x.ko 2024-03-31T14:49:43.897 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 68041 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/smsc/smsc9420.ko 2024-03-31T14:49:43.897 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/stmicro/ 2024-03-31T14:49:43.897 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/stmicro/stmmac/ 2024-03-31T14:49:43.897 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 13025 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-generic.ko 2024-03-31T14:49:43.897 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 39377 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-intel.ko 2024-03-31T14:49:43.897 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 18449 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac-platform.ko 2024-03-31T14:49:43.897 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 495641 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac.ko 2024-03-31T14:49:43.898 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/sun/ 2024-03-31T14:49:43.898 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 103177 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/sun/cassini.ko 2024-03-31T14:49:43.898 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 166041 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/sun/niu.ko 2024-03-31T14:49:43.898 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 66265 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/sun/sungem.ko 2024-03-31T14:49:43.898 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 78713 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/sun/sunhme.ko 2024-03-31T14:49:43.898 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/tehuti/ 2024-03-31T14:49:43.898 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 50009 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/tehuti/tehuti.ko 2024-03-31T14:49:43.898 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/ti/ 2024-03-31T14:49:43.898 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 92337 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/ti/tlan.ko 2024-03-31T14:49:43.898 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/via/ 2024-03-31T14:49:43.898 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 79929 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/via/via-rhine.ko 2024-03-31T14:49:43.898 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 83713 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/via/via-velocity.ko 2024-03-31T14:49:43.898 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/wiznet/ 2024-03-31T14:49:43.898 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 46305 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/wiznet/w5100.ko 2024-03-31T14:49:43.899 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 30817 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/wiznet/w5300.ko 2024-03-31T14:49:43.899 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/xircom/ 2024-03-31T14:49:43.899 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 59977 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/xircom/xirc2ps_cs.ko 2024-03-31T14:49:43.899 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/fddi/ 2024-03-31T14:49:43.899 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 41177 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/fddi/defxx.ko 2024-03-31T14:49:43.899 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/fddi/skfp/ 2024-03-31T14:49:43.899 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 217025 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/fddi/skfp/skfp.ko 2024-03-31T14:49:43.899 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 74985 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/geneve.ko 2024-03-31T14:49:43.899 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/hyperv/ 2024-03-31T14:49:43.899 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 237241 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/hyperv/hv_netvsc.ko 2024-03-31T14:49:43.899 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ieee802154/ 2024-03-31T14:49:43.899 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 18297 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ieee802154/fakelb.ko 2024-03-31T14:49:43.899 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 25273 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ifb.ko 2024-03-31T14:49:43.899 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ipvlan/ 2024-03-31T14:49:43.899 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 85841 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ipvlan/ipvlan.ko 2024-03-31T14:49:43.900 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 62889 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/macvlan.ko 2024-03-31T14:49:43.900 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 16153 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/macvtap.ko 2024-03-31T14:49:43.900 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/mdio/ 2024-03-31T14:49:43.900 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 21241 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/mdio/mdio-bcm-unimac.ko 2024-03-31T14:49:43.900 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 14129 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/mdio/mdio-bitbang.ko 2024-03-31T14:49:43.900 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 13721 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/mdio/mdio-gpio.ko 2024-03-31T14:49:43.900 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 8721 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/mdio/mdio-regmap.ko 2024-03-31T14:49:43.900 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 15929 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/mdio.ko 2024-03-31T14:49:43.900 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 20417 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/mii.ko 2024-03-31T14:49:43.900 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 45049 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/netconsole.ko 2024-03-31T14:49:43.900 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 12985 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/nlmon.ko 2024-03-31T14:49:43.900 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/pcs/ 2024-03-31T14:49:43.900 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 17889 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/pcs/pcs-lynx.ko 2024-03-31T14:49:43.901 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 39121 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/pcs/pcs_xpcs.ko 2024-03-31T14:49:43.901 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/ 2024-03-31T14:49:43.901 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 10361 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/amd.ko 2024-03-31T14:49:43.901 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 49929 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/bcm-phy-lib.ko 2024-03-31T14:49:43.901 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 44625 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/bcm7xxx.ko 2024-03-31T14:49:43.901 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 11553 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/bcm87xx.ko 2024-03-31T14:49:43.901 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 44817 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/broadcom.ko 2024-03-31T14:49:43.901 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 10537 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/cicada.ko 2024-03-31T14:49:43.901 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 12937 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/davicom.ko 2024-03-31T14:49:43.901 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 9361 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/et1011c.ko 2024-03-31T14:49:43.901 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 24873 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/icplus.ko 2024-03-31T14:49:43.901 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 15761 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/lxt.ko 2024-03-31T14:49:43.901 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 81953 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/marvell.ko 2024-03-31T14:49:43.901 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 76561 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/micrel.ko 2024-03-31T14:49:43.901 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 11937 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/national.ko 2024-03-31T14:49:43.902 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 99001 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/phylink.ko 2024-03-31T14:49:43.902 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 9977 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/qsemi.ko 2024-03-31T14:49:43.902 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 49969 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/realtek.ko 2024-03-31T14:49:43.902 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 30089 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/smsc.ko 2024-03-31T14:49:43.902 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 10769 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/ste10Xp.ko 2024-03-31T14:49:43.902 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 24201 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/vitesse.ko 2024-03-31T14:49:43.902 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/plip/ 2024-03-31T14:49:43.902 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 43761 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/plip/plip.ko 2024-03-31T14:49:43.902 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ppp/ 2024-03-31T14:49:43.902 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 15881 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ppp/bsd_comp.ko 2024-03-31T14:49:43.902 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 29721 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ppp/ppp_async.ko 2024-03-31T14:49:43.902 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 17673 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ppp/ppp_deflate.ko 2024-03-31T14:49:43.902 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 18529 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ppp/ppp_mppe.ko 2024-03-31T14:49:43.903 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 28745 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ppp/ppp_synctty.ko 2024-03-31T14:49:43.903 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 42665 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ppp/pppoe.ko 2024-03-31T14:49:43.903 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 13345 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ppp/pppox.ko 2024-03-31T14:49:43.903 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 32665 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ppp/pptp.ko 2024-03-31T14:49:43.903 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 28353 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/sb1000.ko 2024-03-31T14:49:43.903 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/slip/ 2024-03-31T14:49:43.903 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 42385 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/slip/slip.ko 2024-03-31T14:49:43.903 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 26945 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/sungem_phy.ko 2024-03-31T14:49:43.903 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 64553 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/tap.ko 2024-03-31T14:49:43.903 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/team/ 2024-03-31T14:49:43.903 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 107225 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/team/team.ko 2024-03-31T14:49:43.903 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 13449 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/team/team_mode_activebackup.ko 2024-03-31T14:49:43.903 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 10209 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/team/team_mode_broadcast.ko 2024-03-31T14:49:43.903 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 28833 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/team/team_mode_loadbalance.ko 2024-03-31T14:49:43.904 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 9609 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/team/team_mode_random.ko 2024-03-31T14:49:43.904 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 9273 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/team/team_mode_roundrobin.ko 2024-03-31T14:49:43.904 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 86817 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/veth.ko 2024-03-31T14:49:43.904 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/vmxnet3/ 2024-03-31T14:49:43.904 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 144345 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/vmxnet3/vmxnet3.ko 2024-03-31T14:49:43.904 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/vxlan/ 2024-03-31T14:49:43.904 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 307169 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/vxlan/vxlan.ko 2024-03-31T14:49:43.904 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/xen-netback/ 2024-03-31T14:49:43.904 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 143993 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/xen-netback/xen-netback.ko 2024-03-31T14:49:43.904 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ntb/ 2024-03-31T14:49:43.904 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 20873 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ntb/ntb.ko 2024-03-31T14:49:43.904 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/nvme/ 2024-03-31T14:49:43.904 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/nvme/host/ 2024-03-31T14:49:43.904 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 267721 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/nvme/host/nvme-core.ko 2024-03-31T14:49:43.905 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 103353 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/nvme/host/nvme.ko 2024-03-31T14:49:43.905 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/parport/ 2024-03-31T14:49:43.905 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 131297 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/parport/parport.ko 2024-03-31T14:49:43.905 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 14457 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/parport/parport_cs.ko 2024-03-31T14:49:43.905 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 85209 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/parport/parport_pc.ko 2024-03-31T14:49:43.905 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 27561 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/parport/parport_serial.ko 2024-03-31T14:49:43.905 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pci/ 2024-03-31T14:49:43.905 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pci/hotplug/ 2024-03-31T14:49:43.905 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 24905 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pci/hotplug/acpiphp_ibm.ko 2024-03-31T14:49:43.905 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 18073 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pci/hotplug/cpcihp_generic.ko 2024-03-31T14:49:43.905 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 20289 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pci/hotplug/cpcihp_zt5550.ko 2024-03-31T14:49:43.905 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 11697 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pci/pci-stub.ko 2024-03-31T14:49:43.905 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 46361 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pci/xen-pcifront.ko 2024-03-31T14:49:43.905 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pcmcia/ 2024-03-31T14:49:43.906 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 26697 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pcmcia/i82092.ko 2024-03-31T14:49:43.906 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 146129 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pcmcia/pcmcia.ko 2024-03-31T14:49:43.906 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 62065 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pcmcia/pcmcia_core.ko 2024-03-31T14:49:43.906 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 34241 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pcmcia/pcmcia_rsrc.ko 2024-03-31T14:49:43.906 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 34169 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pcmcia/pd6729.ko 2024-03-31T14:49:43.906 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 112785 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pcmcia/yenta_socket.ko 2024-03-31T14:49:43.906 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/phy/ 2024-03-31T14:49:43.906 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/phy/broadcom/ 2024-03-31T14:49:43.906 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 10729 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/phy/broadcom/phy-bcm-kona-usb2.ko 2024-03-31T14:49:43.906 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/phy/samsung/ 2024-03-31T14:49:43.906 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 8337 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/phy/samsung/phy-exynos-usb2.ko 2024-03-31T14:49:43.906 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pinctrl/ 2024-03-31T14:49:43.906 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pinctrl/intel/ 2024-03-31T14:49:43.906 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 69025 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pinctrl/intel/pinctrl-cherryview.ko 2024-03-31T14:49:43.907 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/ 2024-03-31T14:49:43.907 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/chrome/ 2024-03-31T14:49:43.907 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 51217 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/chrome/cros-ec-sensorhub.ko 2024-03-31T14:49:43.907 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 24577 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/chrome/cros_ec.ko 2024-03-31T14:49:43.907 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 21345 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/chrome/cros_ec_chardev.ko 2024-03-31T14:49:43.907 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 25369 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/chrome/cros_ec_debugfs.ko 2024-03-31T14:49:43.907 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 26097 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/chrome/cros_ec_lightbar.ko 2024-03-31T14:49:43.907 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 18209 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/chrome/cros_ec_sysfs.ko 2024-03-31T14:49:43.907 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 17249 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/chrome/cros_usbpd_notify.ko 2024-03-31T14:49:43.907 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/ 2024-03-31T14:49:43.907 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 73593 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/acer-wmi.ko 2024-03-31T14:49:43.907 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 39049 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/acerhdf.ko 2024-03-31T14:49:43.907 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 36921 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/apple-gmux.ko 2024-03-31T14:49:43.907 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 59561 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/asus-laptop.ko 2024-03-31T14:49:43.908 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 31377 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/asus-nb-wmi.ko 2024-03-31T14:49:43.908 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 131937 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/asus-wmi.ko 2024-03-31T14:49:43.908 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 36745 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/classmate-laptop.ko 2024-03-31T14:49:43.908 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 49017 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/eeepc-laptop.ko 2024-03-31T14:49:43.908 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 13857 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/eeepc-wmi.ko 2024-03-31T14:49:43.908 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 40161 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/fujitsu-laptop.ko 2024-03-31T14:49:43.908 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 22417 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/fujitsu-tablet.ko 2024-03-31T14:49:43.908 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 36497 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/hdaps.ko 2024-03-31T14:49:43.908 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 19353 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/ibm_rtl.ko 2024-03-31T14:49:43.908 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/intel/ 2024-03-31T14:49:43.908 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 11977 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/intel/intel-rst.ko 2024-03-31T14:49:43.908 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 8873 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/intel/intel-smartconnect.ko 2024-03-31T14:49:43.908 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 44529 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/intel_ips.ko 2024-03-31T14:49:43.908 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 20177 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/msi-wmi.ko 2024-03-31T14:49:43.909 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 10281 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/mxm-wmi.ko 2024-03-31T14:49:43.909 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 34697 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/panasonic-laptop.ko 2024-03-31T14:49:43.909 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 42225 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/samsung-laptop.ko 2024-03-31T14:49:43.909 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 14065 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/samsung-q10.ko 2024-03-31T14:49:43.909 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 17217 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/topstar-laptop.ko 2024-03-31T14:49:43.909 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 13609 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/toshiba_bluetooth.ko 2024-03-31T14:49:43.909 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 16977 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/toshiba_haps.ko 2024-03-31T14:49:43.909 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 10041 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/wmi-bmof.ko 2024-03-31T14:49:43.909 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 48473 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/wmi.ko 2024-03-31T14:49:43.909 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/power/ 2024-03-31T14:49:43.909 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/power/supply/ 2024-03-31T14:49:43.909 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 30129 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/power/supply/88pm860x_battery.ko 2024-03-31T14:49:43.909 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 33881 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/power/supply/88pm860x_charger.ko 2024-03-31T14:49:43.909 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 54953 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/power/supply/bq2415x_charger.ko 2024-03-31T14:49:43.909 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 59497 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/power/supply/bq24190_charger.ko 2024-03-31T14:49:43.910 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 21449 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/power/supply/bq24735-charger.ko 2024-03-31T14:49:43.910 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 17225 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/power/supply/cros_peripheral_charger.ko 2024-03-31T14:49:43.910 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 19505 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/power/supply/da9030_battery.ko 2024-03-31T14:49:43.910 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 19737 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/power/supply/da9052-battery.ko 2024-03-31T14:49:43.910 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 29177 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/power/supply/ds2760_battery.ko 2024-03-31T14:49:43.910 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 23113 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/power/supply/ds2780_battery.ko 2024-03-31T14:49:43.910 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 23249 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/power/supply/ds2781_battery.ko 2024-03-31T14:49:43.910 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 23545 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/power/supply/ds2782_battery.ko 2024-03-31T14:49:43.910 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 20249 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/power/supply/gpio-charger.ko 2024-03-31T14:49:43.910 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 17985 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/power/supply/isp1704_charger.ko 2024-03-31T14:49:43.910 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 20169 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/power/supply/lp8727_charger.ko 2024-03-31T14:49:43.910 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 12473 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/power/supply/max14577_charger.ko 2024-03-31T14:49:43.910 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 30185 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/power/supply/max17042_battery.ko 2024-03-31T14:49:43.911 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 18481 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/power/supply/max8903_charger.ko 2024-03-31T14:49:43.911 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 23625 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/power/supply/max8925_power.ko 2024-03-31T14:49:43.911 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 21553 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/power/supply/pcf50633-charger.ko 2024-03-31T14:49:43.911 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 37833 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/power/supply/sbs-battery.ko 2024-03-31T14:49:43.911 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 28281 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/power/supply/test_power.ko 2024-03-31T14:49:43.911 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 18761 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/power/supply/tps65090-charger.ko 2024-03-31T14:49:43.911 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 11865 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/power/supply/wm831x_backup.ko 2024-03-31T14:49:43.911 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 24353 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/power/supply/wm831x_power.ko 2024-03-31T14:49:43.911 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 22313 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/power/supply/wm8350_power.ko 2024-03-31T14:49:43.911 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/powercap/ 2024-03-31T14:49:43.911 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 58097 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/powercap/intel_rapl_common.ko 2024-03-31T14:49:43.911 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 17617 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/powercap/intel_rapl_msr.ko 2024-03-31T14:49:43.911 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pps/ 2024-03-31T14:49:43.911 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pps/clients/ 2024-03-31T14:49:43.911 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 15425 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pps/clients/pps-gpio.ko 2024-03-31T14:49:43.912 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 13145 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pps/clients/pps-ldisc.ko 2024-03-31T14:49:43.912 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 16065 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pps/clients/pps_parport.ko 2024-03-31T14:49:43.912 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 41785 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pps/pps_core.ko 2024-03-31T14:49:43.912 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ptp/ 2024-03-31T14:49:43.912 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 70849 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ptp/ptp.ko 2024-03-31T14:49:43.912 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 16361 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ptp/ptp_kvm.ko 2024-03-31T14:49:43.912 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/ 2024-03-31T14:49:43.912 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 23393 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-88pm80x.ko 2024-03-31T14:49:43.912 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 24417 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-88pm860x.ko 2024-03-31T14:49:43.912 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 15289 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-bq32k.ko 2024-03-31T14:49:43.912 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 20001 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-da9052.ko 2024-03-31T14:49:43.912 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 18401 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-da9055.ko 2024-03-31T14:49:43.912 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 20737 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-da9063.ko 2024-03-31T14:49:43.912 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 16353 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-ds1286.ko 2024-03-31T14:49:43.913 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 68473 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-ds1307.ko 2024-03-31T14:49:43.913 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 20713 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-ds1374.ko 2024-03-31T14:49:43.913 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 18113 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-ds1511.ko 2024-03-31T14:49:43.913 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 16633 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-ds1553.ko 2024-03-31T14:49:43.913 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 13849 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-ds1672.ko 2024-03-31T14:49:43.913 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 13265 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-ds1742.ko 2024-03-31T14:49:43.913 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 14593 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-ds2404.ko 2024-03-31T14:49:43.913 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 23601 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-ds3232.ko 2024-03-31T14:49:43.913 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 10977 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-em3027.ko 2024-03-31T14:49:43.913 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 24225 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-fm3130.ko 2024-03-31T14:49:43.913 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 20625 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-isl12022.ko 2024-03-31T14:49:43.913 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 35865 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-isl1208.ko 2024-03-31T14:49:43.913 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 14569 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-lp8788.ko 2024-03-31T14:49:43.914 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 37993 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-m41t80.ko 2024-03-31T14:49:43.914 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 11985 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-m48t35.ko 2024-03-31T14:49:43.914 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 25841 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-m48t59.ko 2024-03-31T14:49:43.914 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 14737 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-m48t86.ko 2024-03-31T14:49:43.914 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 11529 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-max6900.ko 2024-03-31T14:49:43.914 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 14569 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-max8907.ko 2024-03-31T14:49:43.914 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 15241 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-max8925.ko 2024-03-31T14:49:43.914 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 25705 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-max8997.ko 2024-03-31T14:49:43.914 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 16577 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-max8998.ko 2024-03-31T14:49:43.914 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 18409 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-mc13xxx.ko 2024-03-31T14:49:43.914 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 11305 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-msm6242.ko 2024-03-31T14:49:43.914 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 19521 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-palmas.ko 2024-03-31T14:49:43.914 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 46529 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-pcf2127.ko 2024-03-31T14:49:43.914 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 17873 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-pcf50633.ko 2024-03-31T14:49:43.915 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 26889 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-pcf85063.ko 2024-03-31T14:49:43.915 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 21721 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-pcf8523.ko 2024-03-31T14:49:43.915 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 29713 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-pcf8563.ko 2024-03-31T14:49:43.915 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 12337 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-pcf8583.ko 2024-03-31T14:49:43.915 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 17929 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-rc5t583.ko 2024-03-31T14:49:43.915 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 12945 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-rp5c01.ko 2024-03-31T14:49:43.915 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 33649 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-rs5c372.ko 2024-03-31T14:49:43.915 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 27729 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-rv3029c2.ko 2024-03-31T14:49:43.915 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 25153 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-rx8025.ko 2024-03-31T14:49:43.915 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 20009 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-rx8581.ko 2024-03-31T14:49:43.915 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 21513 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-s35390a.ko 2024-03-31T14:49:43.915 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 16721 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-stk17ta8.ko 2024-03-31T14:49:43.915 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 17785 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-tps6586x.ko 2024-03-31T14:49:43.916 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 21233 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-tps65910.ko 2024-03-31T14:49:43.916 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 20441 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-wm831x.ko 2024-03-31T14:49:43.916 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 20985 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-wm8350.ko 2024-03-31T14:49:43.916 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 27769 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-x1205.ko 2024-03-31T14:49:43.916 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/ 2024-03-31T14:49:43.916 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 77521 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/3w-9xxx.ko 2024-03-31T14:49:43.916 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 57713 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/3w-sas.ko 2024-03-31T14:49:43.916 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 55009 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/3w-xxxx.ko 2024-03-31T14:49:43.916 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 135249 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/BusLogic.ko 2024-03-31T14:49:43.916 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 26561 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/a100u2w.ko 2024-03-31T14:49:43.916 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/aacraid/ 2024-03-31T14:49:43.916 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 271385 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/aacraid/aacraid.ko 2024-03-31T14:49:43.916 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 100529 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/advansys.ko 2024-03-31T14:49:43.916 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/aic7xxx/ 2024-03-31T14:49:43.917 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 356833 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/aic7xxx/aic79xx.ko 2024-03-31T14:49:43.917 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 295889 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/aic7xxx/aic7xxx.ko 2024-03-31T14:49:43.917 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/aic94xx/ 2024-03-31T14:49:43.917 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 189089 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/aic94xx/aic94xx.ko 2024-03-31T14:49:43.917 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 23977 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/am53c974.ko 2024-03-31T14:49:43.917 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/arcmsr/ 2024-03-31T14:49:43.917 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 108193 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/arcmsr/arcmsr.ko 2024-03-31T14:49:43.917 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 38089 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/atp870u.ko 2024-03-31T14:49:43.917 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/be2iscsi/ 2024-03-31T14:49:43.917 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 246913 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/be2iscsi/be2iscsi.ko 2024-03-31T14:49:43.917 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/bfa/ 2024-03-31T14:49:43.917 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 1171857 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/bfa/bfa.ko 2024-03-31T14:49:43.917 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/bnx2fc/ 2024-03-31T14:49:43.917 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 238369 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/bnx2fc/bnx2fc.ko 2024-03-31T14:49:43.918 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/bnx2i/ 2024-03-31T14:49:43.918 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 115305 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/bnx2i/bnx2i.ko 2024-03-31T14:49:43.918 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 40777 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/ch.ko 2024-03-31T14:49:43.918 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/csiostor/ 2024-03-31T14:49:43.918 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 308905 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/csiostor/csiostor.ko 2024-03-31T14:49:43.918 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/cxgbi/ 2024-03-31T14:49:43.918 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/cxgbi/cxgb3i/ 2024-03-31T14:49:43.918 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 76033 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/cxgbi/cxgb3i/cxgb3i.ko 2024-03-31T14:49:43.918 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/cxgbi/cxgb4i/ 2024-03-31T14:49:43.918 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 101417 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/cxgbi/cxgb4i/cxgb4i.ko 2024-03-31T14:49:43.918 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 146745 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/cxgbi/libcxgbi.ko 2024-03-31T14:49:43.918 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 75969 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/dc395x.ko 2024-03-31T14:49:43.918 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 38201 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/dmx3191d.ko 2024-03-31T14:49:43.918 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/esas2r/ 2024-03-31T14:49:43.919 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 212329 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/esas2r/esas2r.ko 2024-03-31T14:49:43.919 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 60497 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/esp_scsi.ko 2024-03-31T14:49:43.919 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/fcoe/ 2024-03-31T14:49:43.919 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 78217 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/fcoe/fcoe.ko 2024-03-31T14:49:43.919 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 157713 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/fcoe/libfcoe.ko 2024-03-31T14:49:43.919 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 18849 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/fdomain.ko 2024-03-31T14:49:43.919 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/fnic/ 2024-03-31T14:49:43.919 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 294881 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/fnic/fnic.ko 2024-03-31T14:49:43.919 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 201721 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/hpsa.ko 2024-03-31T14:49:43.919 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 51113 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/hptiop.ko 2024-03-31T14:49:43.919 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 48937 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/hv_storvsc.ko 2024-03-31T14:49:43.919 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 36121 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/imm.ko 2024-03-31T14:49:43.919 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 43969 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/initio.ko 2024-03-31T14:49:43.919 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 265937 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/ipr.ko 2024-03-31T14:49:43.919 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 101873 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/ips.ko 2024-03-31T14:49:43.920 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/isci/ 2024-03-31T14:49:43.920 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 317129 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/isci/isci.ko 2024-03-31T14:49:43.920 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 26281 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/iscsi_boot_sysfs.ko 2024-03-31T14:49:43.920 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 52857 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/iscsi_tcp.ko 2024-03-31T14:49:43.920 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/libfc/ 2024-03-31T14:49:43.920 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 315273 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/libfc/libfc.ko 2024-03-31T14:49:43.920 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 145489 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/libiscsi.ko 2024-03-31T14:49:43.920 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 53801 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/libiscsi_tcp.ko 2024-03-31T14:49:43.920 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/libsas/ 2024-03-31T14:49:43.920 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 216345 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/libsas/libsas.ko 2024-03-31T14:49:43.920 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/lpfc/ 2024-03-31T14:49:43.920 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 2193953 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/lpfc/lpfc.ko 2024-03-31T14:49:43.920 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/megaraid/ 2024-03-31T14:49:43.920 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 73465 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/megaraid/megaraid_mbox.ko 2024-03-31T14:49:43.921 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 33425 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/megaraid/megaraid_mm.ko 2024-03-31T14:49:43.921 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 310665 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/megaraid/megaraid_sas.ko 2024-03-31T14:49:43.921 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 85513 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/megaraid.ko 2024-03-31T14:49:43.921 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/mpt3sas/ 2024-03-31T14:49:43.921 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 659585 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/mpt3sas/mpt3sas.ko 2024-03-31T14:49:43.921 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/mvsas/ 2024-03-31T14:49:43.921 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 119465 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/mvsas/mvsas.ko 2024-03-31T14:49:43.921 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 78033 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/mvumi.ko 2024-03-31T14:49:43.921 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/pcmcia/ 2024-03-31T14:49:43.921 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 81545 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/pcmcia/aha152x_cs.ko 2024-03-31T14:49:43.921 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 11609 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/pcmcia/fdomain_cs.ko 2024-03-31T14:49:43.921 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 20761 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/pcmcia/qlogic_cs.ko 2024-03-31T14:49:43.921 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 23753 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/pcmcia/sym53c500_cs.ko 2024-03-31T14:49:43.921 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/pm8001/ 2024-03-31T14:49:43.921 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 366305 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/pm8001/pm80xx.ko 2024-03-31T14:49:43.922 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 124593 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/pmcraid.ko 2024-03-31T14:49:43.922 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 34417 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/ppa.ko 2024-03-31T14:49:43.922 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 55329 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/qla1280.ko 2024-03-31T14:49:43.922 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/qla2xxx/ 2024-03-31T14:49:43.922 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 1876657 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/qla2xxx/qla2xxx.ko 2024-03-31T14:49:43.922 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 85185 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/qla2xxx/tcm_qla2xxx.ko 2024-03-31T14:49:43.922 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/qla4xxx/ 2024-03-31T14:49:43.922 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 558897 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/qla4xxx/qla4xxx.ko 2024-03-31T14:49:43.922 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 18921 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/qlogicfas408.ko 2024-03-31T14:49:43.922 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 15889 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/raid_class.ko 2024-03-31T14:49:43.922 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 261073 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/scsi_debug.ko 2024-03-31T14:49:43.922 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 173921 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/scsi_transport_fc.ko 2024-03-31T14:49:43.922 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 280105 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/scsi_transport_iscsi.ko 2024-03-31T14:49:43.922 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 97617 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/scsi_transport_sas.ko 2024-03-31T14:49:43.923 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 73857 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/scsi_transport_spi.ko 2024-03-31T14:49:43.923 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 44881 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/scsi_transport_srp.ko 2024-03-31T14:49:43.923 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 26153 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/ses.ko 2024-03-31T14:49:43.923 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 115641 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/st.ko 2024-03-31T14:49:43.923 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 46729 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/stex.ko 2024-03-31T14:49:43.923 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/sym53c8xx_2/ 2024-03-31T14:49:43.923 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 164865 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/sym53c8xx_2/sym53c8xx.ko 2024-03-31T14:49:43.923 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 44849 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/virtio_scsi.ko 2024-03-31T14:49:43.923 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 50225 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/vmw_pvscsi.ko 2024-03-31T14:49:43.923 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 39873 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/wd719x.ko 2024-03-31T14:49:43.924 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 49921 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/xen-scsifront.ko 2024-03-31T14:49:43.924 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ssb/ 2024-03-31T14:49:43.924 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 129345 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ssb/ssb.ko 2024-03-31T14:49:43.924 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/ 2024-03-31T14:49:43.924 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/iscsi/ 2024-03-31T14:49:43.924 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 736665 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/iscsi/iscsi_target_mod.ko 2024-03-31T14:49:43.924 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/loopback/ 2024-03-31T14:49:43.924 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 49873 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/loopback/tcm_loop.ko 2024-03-31T14:49:43.924 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 39609 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/target_core_file.ko 2024-03-31T14:49:43.924 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 42801 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/target_core_iblock.ko 2024-03-31T14:49:43.924 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 931433 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/target_core_mod.ko 2024-03-31T14:49:43.924 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 44145 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/target_core_pscsi.ko 2024-03-31T14:49:43.924 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 122481 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/target_core_user.ko 2024-03-31T14:49:43.924 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/tcm_fc/ 2024-03-31T14:49:43.925 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 75497 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/tcm_fc/tcm_fc.ko 2024-03-31T14:49:43.925 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/ 2024-03-31T14:49:43.925 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/ 2024-03-31T14:49:43.925 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/ 2024-03-31T14:49:43.925 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 23377 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/acpi_thermal_rel.ko 2024-03-31T14:49:43.925 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 29217 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/int3400_thermal.ko 2024-03-31T14:49:43.925 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 9849 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/int3401_thermal.ko 2024-03-31T14:49:43.925 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 10161 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/int3402_thermal.ko 2024-03-31T14:49:43.925 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 14865 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/int3403_thermal.ko 2024-03-31T14:49:43.925 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 16249 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/int340x_thermal_zone.ko 2024-03-31T14:49:43.925 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 30417 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device.ko 2024-03-31T14:49:43.925 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 22625 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci.ko 2024-03-31T14:49:43.925 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 14737 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci_legacy.ko 2024-03-31T14:49:43.925 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 10825 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_mbox.ko 2024-03-31T14:49:43.926 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 10745 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_power_floor.ko 2024-03-31T14:49:43.926 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 12889 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rapl.ko 2024-03-31T14:49:43.926 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 36953 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rfim.ko 2024-03-31T14:49:43.926 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 14897 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_hint.ko 2024-03-31T14:49:43.926 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 11457 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_req.ko 2024-03-31T14:49:43.926 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 30801 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/intel_powerclamp.ko 2024-03-31T14:49:43.926 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 18825 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/intel_soc_dts_iosf.ko 2024-03-31T14:49:43.926 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 11409 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/intel_soc_dts_thermal.ko 2024-03-31T14:49:43.926 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 26985 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/x86_pkg_temp_thermal.ko 2024-03-31T14:49:43.926 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/ 2024-03-31T14:49:43.926 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/ipwireless/ 2024-03-31T14:49:43.926 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 78329 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/ipwireless/ipwireless.ko 2024-03-31T14:49:43.926 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 57769 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/moxa.ko 2024-03-31T14:49:43.927 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 50857 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/mxser.ko 2024-03-31T14:49:43.927 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 108481 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/n_gsm.ko 2024-03-31T14:49:43.927 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 31865 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/n_hdlc.ko 2024-03-31T14:49:43.927 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 67833 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/nozomi.ko 2024-03-31T14:49:43.927 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/serial/ 2024-03-31T14:49:43.927 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/serial/8250/ 2024-03-31T14:49:43.927 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 34785 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/serial/8250/8250_dw.ko 2024-03-31T14:49:43.927 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 59393 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/serial/8250/serial_cs.ko 2024-03-31T14:49:43.927 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 17969 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/serial/altera_jtaguart.ko 2024-03-31T14:49:43.927 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 22473 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/serial/altera_uart.ko 2024-03-31T14:49:43.927 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 9329 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/serial/arc_uart.ko 2024-03-31T14:49:43.927 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 72545 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/serial/fsl_lpuart.ko 2024-03-31T14:49:43.927 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/serial/jsm/ 2024-03-31T14:49:43.927 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 101969 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/serial/jsm/jsm.ko 2024-03-31T14:49:43.927 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 28689 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/serial/rp2.ko 2024-03-31T14:49:43.928 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 53169 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/serial/sc16is7xx.ko 2024-03-31T14:49:43.928 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ufs/ 2024-03-31T14:49:43.928 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ufs/core/ 2024-03-31T14:49:43.928 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 545377 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ufs/core/ufshcd-core.ko 2024-03-31T14:49:43.928 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ufs/host/ 2024-03-31T14:49:43.928 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 28825 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ufs/host/ufshcd-pci.ko 2024-03-31T14:49:43.928 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 14505 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ufs/host/ufshcd-pltfrm.ko 2024-03-31T14:49:43.928 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/uio/ 2024-03-31T14:49:43.928 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 44465 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/uio/uio.ko 2024-03-31T14:49:43.928 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 12145 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/uio/uio_aec.ko 2024-03-31T14:49:43.928 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 11193 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/uio/uio_cif.ko 2024-03-31T14:49:43.928 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 18225 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/uio/uio_dmem_genirq.ko 2024-03-31T14:49:43.928 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 12833 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/uio/uio_mf624.ko 2024-03-31T14:49:43.929 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 12081 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/uio/uio_netx.ko 2024-03-31T14:49:43.929 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 10497 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/uio/uio_pci_generic.ko 2024-03-31T14:49:43.929 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 15953 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/uio/uio_pdrv_genirq.ko 2024-03-31T14:49:43.929 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 13537 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/uio/uio_sercos3.ko 2024-03-31T14:49:43.929 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/ 2024-03-31T14:49:43.929 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/c67x00/ 2024-03-31T14:49:43.929 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 73201 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/c67x00/c67x00.ko 2024-03-31T14:49:43.929 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/chipidea/ 2024-03-31T14:49:43.929 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 145753 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/chipidea/ci_hdrc.ko 2024-03-31T14:49:43.929 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 18401 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/chipidea/ci_hdrc_msm.ko 2024-03-31T14:49:43.929 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 13073 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/chipidea/ci_hdrc_npcm.ko 2024-03-31T14:49:43.929 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 12609 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/chipidea/ci_hdrc_pci.ko 2024-03-31T14:49:43.929 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 13113 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/chipidea/ci_hdrc_usb2.ko 2024-03-31T14:49:43.929 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/class/ 2024-03-31T14:49:43.930 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 87001 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/class/cdc-acm.ko 2024-03-31T14:49:43.930 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 51249 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/class/cdc-wdm.ko 2024-03-31T14:49:43.930 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 47209 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/class/usblp.ko 2024-03-31T14:49:43.930 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 89761 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/class/usbtmc.ko 2024-03-31T14:49:43.930 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/common/ 2024-03-31T14:49:43.930 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 22857 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/common/ulpi.ko 2024-03-31T14:49:43.930 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/dwc2/ 2024-03-31T14:49:43.930 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 11433 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/dwc2/dwc2_pci.ko 2024-03-31T14:49:43.930 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/dwc3/ 2024-03-31T14:49:43.930 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 11985 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/dwc3/dwc3-haps.ko 2024-03-31T14:49:43.930 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 30209 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/dwc3/dwc3-pci.ko 2024-03-31T14:49:43.930 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 452785 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/dwc3/dwc3.ko 2024-03-31T14:49:43.930 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/gadget/ 2024-03-31T14:49:43.930 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/gadget/function/ 2024-03-31T14:49:43.931 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 60321 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/gadget/function/u_ether.ko 2024-03-31T14:49:43.931 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 51785 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/gadget/function/u_serial.ko 2024-03-31T14:49:43.931 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 28737 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/gadget/function/usb_f_acm.ko 2024-03-31T14:49:43.931 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 42345 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/gadget/function/usb_f_ecm.ko 2024-03-31T14:49:43.931 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 27849 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/gadget/function/usb_f_ecm_subset.ko 2024-03-31T14:49:43.931 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 36409 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/gadget/function/usb_f_eem.ko 2024-03-31T14:49:43.931 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 92401 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/gadget/function/usb_f_fs.ko 2024-03-31T14:49:43.931 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 50937 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/gadget/function/usb_f_hid.ko 2024-03-31T14:49:43.931 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 122241 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/gadget/function/usb_f_mass_storage.ko 2024-03-31T14:49:43.931 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 59857 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/gadget/function/usb_f_ncm.ko 2024-03-31T14:49:43.931 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 22081 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/gadget/function/usb_f_obex.ko 2024-03-31T14:49:43.931 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 31161 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/gadget/function/usb_f_phonet.ko 2024-03-31T14:49:43.931 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 57625 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/gadget/function/usb_f_printer.ko 2024-03-31T14:49:43.931 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 85505 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/gadget/function/usb_f_rndis.ko 2024-03-31T14:49:43.931 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 19169 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/gadget/function/usb_f_serial.ko 2024-03-31T14:49:43.932 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 58009 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/gadget/function/usb_f_ss_lb.ko 2024-03-31T14:49:43.932 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 67345 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/gadget/function/usb_f_tcm.ko 2024-03-31T14:49:43.932 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/gadget/legacy/ 2024-03-31T14:49:43.932 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 21169 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/gadget/legacy/g_acm_ms.ko 2024-03-31T14:49:43.932 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 17961 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/gadget/legacy/g_cdc.ko 2024-03-31T14:49:43.932 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 18369 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/gadget/legacy/g_dbgp.ko 2024-03-31T14:49:43.932 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 21561 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/gadget/legacy/g_ether.ko 2024-03-31T14:49:43.932 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 28305 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/gadget/legacy/g_ffs.ko 2024-03-31T14:49:43.932 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 18913 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/gadget/legacy/g_hid.ko 2024-03-31T14:49:43.932 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 20305 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/gadget/legacy/g_mass_storage.ko 2024-03-31T14:49:43.932 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 16881 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/gadget/legacy/g_ncm.ko 2024-03-31T14:49:43.932 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 31977 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/gadget/legacy/g_nokia.ko 2024-03-31T14:49:43.932 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 16689 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/gadget/legacy/g_printer.ko 2024-03-31T14:49:43.932 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 19297 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/gadget/legacy/g_serial.ko 2024-03-31T14:49:43.933 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 27105 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/gadget/legacy/g_zero.ko 2024-03-31T14:49:43.933 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 62361 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/gadget/legacy/gadgetfs.ko 2024-03-31T14:49:43.933 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 15097 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/gadget/legacy/tcm_usb_gadget.ko 2024-03-31T14:49:43.933 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 161193 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/gadget/libcomposite.ko 2024-03-31T14:49:43.933 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/gadget/udc/ 2024-03-31T14:49:43.933 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 17601 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/gadget/udc/amd5536udc_pci.ko 2024-03-31T14:49:43.933 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/gadget/udc/bdc/ 2024-03-31T14:49:43.933 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 157593 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/gadget/udc/bdc/bdc.ko 2024-03-31T14:49:43.933 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 35993 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/gadget/udc/goku_udc.ko 2024-03-31T14:49:43.933 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 52921 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/gadget/udc/gr_udc.ko 2024-03-31T14:49:43.933 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 69609 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/gadget/udc/mv_u3d_core.ko 2024-03-31T14:49:43.933 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 55945 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/gadget/udc/mv_udc.ko 2024-03-31T14:49:43.933 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 75961 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/gadget/udc/net2272.ko 2024-03-31T14:49:43.933 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 74977 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/gadget/udc/net2280.ko 2024-03-31T14:49:43.934 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 73009 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/gadget/udc/pch_udc.ko 2024-03-31T14:49:43.934 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 65873 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/gadget/udc/pxa27x_udc.ko 2024-03-31T14:49:43.934 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 65993 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/gadget/udc/r8a66597-udc.ko 2024-03-31T14:49:43.934 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 81401 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/gadget/udc/snps_udc_core.ko 2024-03-31T14:49:43.934 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 156321 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/gadget/udc/udc-core.ko 2024-03-31T14:49:43.934 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/host/ 2024-03-31T14:49:43.934 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 11497 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/host/bcma-hcd.ko 2024-03-31T14:49:43.934 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 80009 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/host/isp116x-hcd.ko 2024-03-31T14:49:43.934 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 87249 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/host/oxu210hp-hcd.ko 2024-03-31T14:49:43.934 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 78769 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/host/r8a66597-hcd.ko 2024-03-31T14:49:43.934 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 51385 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/host/sl811-hcd.ko 2024-03-31T14:49:43.934 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 15497 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/host/sl811_cs.ko 2024-03-31T14:49:43.934 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 13217 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/host/ssb-hcd.ko 2024-03-31T14:49:43.934 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 26617 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/host/xhci-plat-hcd.ko 2024-03-31T14:49:43.935 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/image/ 2024-03-31T14:49:43.935 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 34489 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/image/mdc800.ko 2024-03-31T14:49:43.935 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 20985 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/image/microtek.ko 2024-03-31T14:49:43.935 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/ 2024-03-31T14:49:43.935 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 41625 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/adutux.ko 2024-03-31T14:49:43.935 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 20609 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/appledisplay.ko 2024-03-31T14:49:43.935 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 17617 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/cypress_cy7c63.ko 2024-03-31T14:49:43.935 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 22953 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/cytherm.ko 2024-03-31T14:49:43.935 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 13633 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/ehset.ko 2024-03-31T14:49:43.935 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 14961 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/emi26.ko 2024-03-31T14:49:43.935 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 15889 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/emi62.ko 2024-03-31T14:49:43.935 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 11337 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/ezusb.ko 2024-03-31T14:49:43.935 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 19137 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/idmouse.ko 2024-03-31T14:49:43.935 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 36449 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/iowarrior.ko 2024-03-31T14:49:43.936 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 11489 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/isight_firmware.ko 2024-03-31T14:49:43.936 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 36433 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/ldusb.ko 2024-03-31T14:49:43.936 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 36097 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/legousbtower.ko 2024-03-31T14:49:43.936 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 27697 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/lvstest.ko 2024-03-31T14:49:43.936 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/sisusbvga/ 2024-03-31T14:49:43.936 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 64737 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/sisusbvga/sisusbvga.ko 2024-03-31T14:49:43.936 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 12833 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/trancevibrator.ko 2024-03-31T14:49:43.936 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 20377 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/usb3503.ko 2024-03-31T14:49:43.936 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 25049 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/usblcd.ko 2024-03-31T14:49:43.936 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 22649 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/usbsevseg.ko 2024-03-31T14:49:43.936 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 70713 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/usbtest.ko 2024-03-31T14:49:43.936 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 41809 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/uss720.ko 2024-03-31T14:49:43.936 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 27217 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/yurex.ko 2024-03-31T14:49:43.936 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/mon/ 2024-03-31T14:49:43.936 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 72801 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/mon/usbmon.ko 2024-03-31T14:49:43.936 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/musb/ 2024-03-31T14:49:43.937 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 301593 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/musb/musb_hdrc.ko 2024-03-31T14:49:43.937 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/phy/ 2024-03-31T14:49:43.937 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 20953 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/phy/phy-generic.ko 2024-03-31T14:49:43.937 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 20065 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/phy/phy-gpio-vbus-usb.ko 2024-03-31T14:49:43.937 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 12153 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/phy/phy-isp1301.ko 2024-03-31T14:49:43.937 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 21641 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/phy/phy-tahvo.ko 2024-03-31T14:49:43.937 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/ 2024-03-31T14:49:43.937 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 13113 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/aircable.ko 2024-03-31T14:49:43.937 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 33233 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/ark3116.ko 2024-03-31T14:49:43.937 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 26529 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/belkin_sa.ko 2024-03-31T14:49:43.937 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 38009 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/ch341.ko 2024-03-31T14:49:43.937 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 70273 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/cp210x.ko 2024-03-31T14:49:43.937 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 30953 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/cyberjack.ko 2024-03-31T14:49:43.937 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 58113 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/cypress_m8.ko 2024-03-31T14:49:43.938 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 49713 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/digi_acceleport.ko 2024-03-31T14:49:43.938 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 9521 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/empeg.ko 2024-03-31T14:49:43.938 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 44177 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/f81232.ko 2024-03-31T14:49:43.938 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 175097 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/ftdi_sio.ko 2024-03-31T14:49:43.938 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 53009 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/garmin_gps.ko 2024-03-31T14:49:43.938 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 136977 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/io_edgeport.ko 2024-03-31T14:49:43.938 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 124737 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/io_ti.ko 2024-03-31T14:49:43.938 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 67673 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/ipaq.ko 2024-03-31T14:49:43.938 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 19065 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/ipw.ko 2024-03-31T14:49:43.938 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 26801 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/ir-usb.ko 2024-03-31T14:49:43.938 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 58129 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/iuu_phoenix.ko 2024-03-31T14:49:43.938 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 107425 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/keyspan.ko 2024-03-31T14:49:43.938 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 35513 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/keyspan_pda.ko 2024-03-31T14:49:43.939 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 25561 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/kl5kusb105.ko 2024-03-31T14:49:43.939 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 33585 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/kobil_sct.ko 2024-03-31T14:49:43.939 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 32857 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/mct_u232.ko 2024-03-31T14:49:43.939 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 21929 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/metro-usb.ko 2024-03-31T14:49:43.939 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 66353 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/mos7720.ko 2024-03-31T14:49:43.939 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 78121 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/mos7840.ko 2024-03-31T14:49:43.939 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 56241 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/mxuport.ko 2024-03-31T14:49:43.939 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 13929 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/navman.ko 2024-03-31T14:49:43.939 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 11913 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/omninet.ko 2024-03-31T14:49:43.939 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 24537 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/opticon.ko 2024-03-31T14:49:43.939 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 191817 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/option.ko 2024-03-31T14:49:43.939 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 38977 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/oti6858.ko 2024-03-31T14:49:43.939 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 54257 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/pl2303.ko 2024-03-31T14:49:43.939 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 9641 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/qcaux.ko 2024-03-31T14:49:43.940 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 38017 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/qcserial.ko 2024-03-31T14:49:43.940 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 34225 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/quatech2.ko 2024-03-31T14:49:43.940 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 15289 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/safe_serial.ko 2024-03-31T14:49:43.940 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 51529 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/sierra.ko 2024-03-31T14:49:43.940 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 21577 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/spcp8x5.ko 2024-03-31T14:49:43.940 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 29897 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/ssu100.ko 2024-03-31T14:49:43.940 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 17161 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/symbolserial.ko 2024-03-31T14:49:43.940 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 61385 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/ti_usb_3410_5052.ko 2024-03-31T14:49:43.940 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 23617 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/usb-serial-simple.ko 2024-03-31T14:49:43.940 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 10137 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/usb_debug.ko 2024-03-31T14:49:43.940 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 33105 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/usb_wwan.ko 2024-03-31T14:49:43.940 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 125513 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/usbserial.ko 2024-03-31T14:49:43.940 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 28657 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/visor.ko 2024-03-31T14:49:43.940 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 37305 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/whiteheat.ko 2024-03-31T14:49:43.940 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 10433 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/wishbone-serial.ko 2024-03-31T14:49:43.941 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 9193 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/xsens_mt.ko 2024-03-31T14:49:43.941 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ 2024-03-31T14:49:43.941 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 52753 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/uas.ko 2024-03-31T14:49:43.941 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 25849 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-alauda.ko 2024-03-31T14:49:43.941 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 12985 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-cypress.ko 2024-03-31T14:49:43.941 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 21881 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-datafab.ko 2024-03-31T14:49:43.941 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 33393 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-eneub6250.ko 2024-03-31T14:49:43.941 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 13649 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-freecom.ko 2024-03-31T14:49:43.941 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 19513 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-isd200.ko 2024-03-31T14:49:43.941 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 19201 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-jumpshot.ko 2024-03-31T14:49:43.941 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 13553 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-karma.ko 2024-03-31T14:49:43.941 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 17737 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-onetouch.ko 2024-03-31T14:49:43.941 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 28337 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-realtek.ko 2024-03-31T14:49:43.941 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 28801 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-sddr09.ko 2024-03-31T14:49:43.941 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 19417 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-sddr55.ko 2024-03-31T14:49:43.942 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 35241 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-usbat.ko 2024-03-31T14:49:43.942 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 180281 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/usb-storage.ko 2024-03-31T14:49:43.942 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/usbip/ 2024-03-31T14:49:43.942 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 67385 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/usbip/usbip-core.ko 2024-03-31T14:49:43.942 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 71001 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/usbip/usbip-host.ko 2024-03-31T14:49:43.942 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 103977 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/usbip/vhci-hcd.ko 2024-03-31T14:49:43.942 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vfio/ 2024-03-31T14:49:43.942 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vfio/pci/ 2024-03-31T14:49:43.942 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 162841 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vfio/pci/vfio-pci-core.ko 2024-03-31T14:49:43.942 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 29377 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vfio/pci/vfio-pci.ko 2024-03-31T14:49:43.942 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 103649 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vfio/vfio.ko 2024-03-31T14:49:43.942 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 76433 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vfio/vfio_iommu_type1.ko 2024-03-31T14:49:43.942 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vhost/ 2024-03-31T14:49:43.943 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 113657 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vhost/vhost.ko 2024-03-31T14:49:43.943 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 16905 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vhost/vhost_iotlb.ko 2024-03-31T14:49:43.943 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 57993 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vhost/vhost_net.ko 2024-03-31T14:49:43.943 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 89049 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vhost/vhost_scsi.ko 2024-03-31T14:49:43.943 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/video/ 2024-03-31T14:49:43.943 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/video/backlight/ 2024-03-31T14:49:43.943 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 15313 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/video/backlight/88pm860x_bl.ko 2024-03-31T14:49:43.943 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 12289 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/video/backlight/aat2870_bl.ko 2024-03-31T14:49:43.943 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 23201 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/video/backlight/adp5520_bl.ko 2024-03-31T14:49:43.943 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 32505 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/video/backlight/adp8860_bl.ko 2024-03-31T14:49:43.943 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 39889 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/video/backlight/adp8870_bl.ko 2024-03-31T14:49:43.943 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 16825 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/video/backlight/apple_bl.ko 2024-03-31T14:49:43.943 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 16673 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/video/backlight/as3711_bl.ko 2024-03-31T14:49:43.943 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 11937 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/video/backlight/bd6107.ko 2024-03-31T14:49:43.944 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 11305 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/video/backlight/da903x_bl.ko 2024-03-31T14:49:43.944 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 11169 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/video/backlight/da9052_bl.ko 2024-03-31T14:49:43.944 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 11649 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/video/backlight/gpio_backlight.ko 2024-03-31T14:49:43.944 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 14665 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/video/backlight/kb3886_bl.ko 2024-03-31T14:49:43.944 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 21921 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/video/backlight/lcd.ko 2024-03-31T14:49:43.944 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 23553 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/video/backlight/lm3533_bl.ko 2024-03-31T14:49:43.944 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 19233 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/video/backlight/lm3639_bl.ko 2024-03-31T14:49:43.944 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 11329 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/video/backlight/lv5207lp.ko 2024-03-31T14:49:43.944 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 14241 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/video/backlight/max8925_bl.ko 2024-03-31T14:49:43.944 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 11209 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/video/backlight/pandora_bl.ko 2024-03-31T14:49:43.944 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 10753 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/video/backlight/pcf50633-backlight.ko 2024-03-31T14:49:43.944 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 10793 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/video/backlight/platform_lcd.ko 2024-03-31T14:49:43.944 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 12617 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/video/backlight/wm831x_bl.ko 2024-03-31T14:49:43.944 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/ 2024-03-31T14:49:43.944 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/masters/ 2024-03-31T14:49:43.945 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 18689 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/masters/ds2482.ko 2024-03-31T14:49:43.945 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 39113 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/masters/ds2490.ko 2024-03-31T14:49:43.945 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 11161 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/masters/matrox_w1.ko 2024-03-31T14:49:43.945 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 11265 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/masters/w1-gpio.ko 2024-03-31T14:49:43.945 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/ 2024-03-31T14:49:43.945 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 12001 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_ds2406.ko 2024-03-31T14:49:43.945 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 22641 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_ds2408.ko 2024-03-31T14:49:43.945 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 15009 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_ds2413.ko 2024-03-31T14:49:43.945 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 10721 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_ds2423.ko 2024-03-31T14:49:43.945 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 14049 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_ds2431.ko 2024-03-31T14:49:43.945 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 13809 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_ds2433.ko 2024-03-31T14:49:43.945 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 13945 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_ds2780.ko 2024-03-31T14:49:43.945 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 13945 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_ds2781.ko 2024-03-31T14:49:43.945 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 17817 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_ds28e04.ko 2024-03-31T14:49:43.946 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 7289 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_smem.ko 2024-03-31T14:49:43.946 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 56329 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_therm.ko 2024-03-31T14:49:43.946 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 103025 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/wire.ko 2024-03-31T14:49:43.946 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/ 2024-03-31T14:49:43.946 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 19305 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/acquirewdt.ko 2024-03-31T14:49:43.946 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 20537 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/advantechwdt.ko 2024-03-31T14:49:43.946 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 18457 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/alim1535_wdt.ko 2024-03-31T14:49:43.946 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 22865 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/alim7101_wdt.ko 2024-03-31T14:49:43.946 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 19017 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/cpu5wdt.ko 2024-03-31T14:49:43.946 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 13513 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/da9052_wdt.ko 2024-03-31T14:49:43.946 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 12673 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/da9055_wdt.ko 2024-03-31T14:49:43.946 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 17305 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/da9063_wdt.ko 2024-03-31T14:49:43.946 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 20073 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/dw_wdt.ko 2024-03-31T14:49:43.946 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 20161 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/eurotechwdt.ko 2024-03-31T14:49:43.947 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 23761 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/f71808e_wdt.ko 2024-03-31T14:49:43.947 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 27385 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/hpwdt.ko 2024-03-31T14:49:43.947 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 15481 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/i6300esb.ko 2024-03-31T14:49:43.947 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 10721 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/iTCO_vendor_support.ko 2024-03-31T14:49:43.947 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 21817 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/iTCO_wdt.ko 2024-03-31T14:49:43.947 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 19633 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/ib700wdt.ko 2024-03-31T14:49:43.947 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 18593 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/ibmasr.ko 2024-03-31T14:49:43.947 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 19609 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/ie6xx_wdt.ko 2024-03-31T14:49:43.947 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 21121 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/it8712f_wdt.ko 2024-03-31T14:49:43.947 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 13993 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/it87_wdt.ko 2024-03-31T14:49:43.947 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 21385 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/kempld_wdt.ko 2024-03-31T14:49:43.947 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 18521 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/machzwd.ko 2024-03-31T14:49:43.947 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 14617 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/mena21_wdt.ko 2024-03-31T14:49:43.947 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 13113 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/menf21bmc_wdt.ko 2024-03-31T14:49:43.947 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 24577 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/nv_tco.ko 2024-03-31T14:49:43.948 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 17217 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/of_xilinx_wdt.ko 2024-03-31T14:49:43.948 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 23825 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/pc87413_wdt.ko 2024-03-31T14:49:43.948 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 39913 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/pcwd_pci.ko 2024-03-31T14:49:43.948 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 31729 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/pcwd_usb.ko 2024-03-31T14:49:43.948 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 12969 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/retu_wdt.ko 2024-03-31T14:49:43.948 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 20977 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/sbc60xxwdt.ko 2024-03-31T14:49:43.948 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 14025 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/sbc_epx_c3.ko 2024-03-31T14:49:43.948 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 17217 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/sbc_fitpc2_wdt.ko 2024-03-31T14:49:43.948 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 24625 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/sc1200wdt.ko 2024-03-31T14:49:43.948 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 23217 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/sch311x_wdt.ko 2024-03-31T14:49:43.948 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 21473 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/smsc37b787_wdt.ko 2024-03-31T14:49:43.948 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 17177 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/softdog.ko 2024-03-31T14:49:43.948 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 22169 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/sp5100_tco.ko 2024-03-31T14:49:43.948 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 12713 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/twl4030_wdt.ko 2024-03-31T14:49:43.949 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 17553 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/via_wdt.ko 2024-03-31T14:49:43.949 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 20233 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/w83627hf_wdt.ko 2024-03-31T14:49:43.949 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 21193 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/w83877f_wdt.ko 2024-03-31T14:49:43.949 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 28561 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/w83977f_wdt.ko 2024-03-31T14:49:43.949 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 19505 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/wafer5823wdt.ko 2024-03-31T14:49:43.949 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 27993 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/wdt_pci.ko 2024-03-31T14:49:43.949 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 16153 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/wm831x_wdt.ko 2024-03-31T14:49:43.949 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 13681 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/wm8350_wdt.ko 2024-03-31T14:49:43.949 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 15601 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/xen_wdt.ko 2024-03-31T14:49:43.949 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/xen/ 2024-03-31T14:49:43.949 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 24625 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/xen/xen-evtchn.ko 2024-03-31T14:49:43.949 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 26665 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/xen/xen-gntalloc.ko 2024-03-31T14:49:43.949 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 47761 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/xen/xen-gntdev.ko 2024-03-31T14:49:43.949 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/xen/xen-pciback/ 2024-03-31T14:49:43.949 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 176337 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/xen/xen-pciback/xen-pciback.ko 2024-03-31T14:49:43.950 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 40081 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/xen/xen-privcmd.ko 2024-03-31T14:49:43.950 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 71145 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/xen/xen-scsiback.ko 2024-03-31T14:49:43.950 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/xen/xenfs/ 2024-03-31T14:49:43.950 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 19329 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/xen/xenfs/xenfs.ko 2024-03-31T14:49:43.950 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ 2024-03-31T14:49:43.950 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/9p/ 2024-03-31T14:49:43.950 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 158217 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/9p/9p.ko 2024-03-31T14:49:43.950 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/adfs/ 2024-03-31T14:49:43.950 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 61681 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/adfs/adfs.ko 2024-03-31T14:49:43.950 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/affs/ 2024-03-31T14:49:43.950 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 163369 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/affs/affs.ko 2024-03-31T14:49:43.950 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/afs/ 2024-03-31T14:49:43.950 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 1019049 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/afs/kafs.ko 2024-03-31T14:49:43.951 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/autofs/ 2024-03-31T14:49:43.951 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 103185 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/autofs/autofs4.ko 2024-03-31T14:49:43.951 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/befs/ 2024-03-31T14:49:43.951 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 64585 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/befs/befs.ko 2024-03-31T14:49:43.951 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/bfs/ 2024-03-31T14:49:43.951 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 41001 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/bfs/bfs.ko 2024-03-31T14:49:43.951 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 38641 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/binfmt_misc.ko 2024-03-31T14:49:43.951 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/btrfs/ 2024-03-31T14:49:43.951 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 3487249 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/btrfs/btrfs.ko 2024-03-31T14:49:43.951 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/cachefiles/ 2024-03-31T14:49:43.951 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 305841 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/cachefiles/cachefiles.ko 2024-03-31T14:49:43.951 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ceph/ 2024-03-31T14:49:43.951 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 1335577 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ceph/ceph.ko 2024-03-31T14:49:43.951 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/coda/ 2024-03-31T14:49:43.951 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 97713 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/coda/coda.ko 2024-03-31T14:49:43.952 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/configfs/ 2024-03-31T14:49:43.952 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 108545 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/configfs/configfs.ko 2024-03-31T14:49:43.952 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/cramfs/ 2024-03-31T14:49:43.952 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 29577 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/cramfs/cramfs.ko 2024-03-31T14:49:43.952 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/dlm/ 2024-03-31T14:49:43.952 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 644161 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/dlm/dlm.ko 2024-03-31T14:49:43.952 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/efs/ 2024-03-31T14:49:43.952 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 36697 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/efs/efs.ko 2024-03-31T14:49:43.952 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/f2fs/ 2024-03-31T14:49:43.952 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 1527561 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/f2fs/f2fs.ko 2024-03-31T14:49:43.952 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/fat/ 2024-03-31T14:49:43.952 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 27833 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/fat/msdos.ko 2024-03-31T14:49:43.952 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/freevxfs/ 2024-03-31T14:49:43.952 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 42513 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/freevxfs/freevxfs.ko 2024-03-31T14:49:43.953 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/fuse/ 2024-03-31T14:49:43.953 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 24289 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/fuse/cuse.ko 2024-03-31T14:49:43.953 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/gfs2/ 2024-03-31T14:49:43.953 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 1231233 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/gfs2/gfs2.ko 2024-03-31T14:49:43.953 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/hfs/ 2024-03-31T14:49:43.953 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 141625 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/hfs/hfs.ko 2024-03-31T14:49:43.953 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/hfsplus/ 2024-03-31T14:49:43.953 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 223177 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/hfsplus/hfsplus.ko 2024-03-31T14:49:43.953 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/hpfs/ 2024-03-31T14:49:43.953 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 193481 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/hpfs/hpfs.ko 2024-03-31T14:49:43.953 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/isofs/ 2024-03-31T14:49:43.953 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 74689 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/isofs/isofs.ko 2024-03-31T14:49:43.953 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/jfs/ 2024-03-31T14:49:43.953 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 420329 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/jfs/jfs.ko 2024-03-31T14:49:43.953 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/lockd/ 2024-03-31T14:49:43.954 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 281617 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/lockd/lockd.ko 2024-03-31T14:49:43.954 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/minix/ 2024-03-31T14:49:43.954 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 88897 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/minix/minix.ko 2024-03-31T14:49:43.954 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/netfs/ 2024-03-31T14:49:43.954 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 490409 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/netfs/netfs.ko 2024-03-31T14:49:43.954 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs/ 2024-03-31T14:49:43.954 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs/blocklayout/ 2024-03-31T14:49:43.954 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 81433 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs/blocklayout/blocklayoutdriver.ko 2024-03-31T14:49:43.954 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs/filelayout/ 2024-03-31T14:49:43.954 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 59337 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs/filelayout/nfs_layout_nfsv41_files.ko 2024-03-31T14:49:43.954 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs/flexfilelayout/ 2024-03-31T14:49:43.954 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 117361 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs/flexfilelayout/nfs_layout_flexfiles.ko 2024-03-31T14:49:43.954 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 1058273 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs/nfs.ko 2024-03-31T14:49:43.954 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 66345 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs/nfsv2.ko 2024-03-31T14:49:43.955 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 120169 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs/nfsv3.ko 2024-03-31T14:49:43.955 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 2041705 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs/nfsv4.ko 2024-03-31T14:49:43.955 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs_common/ 2024-03-31T14:49:43.955 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 16833 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs_common/grace.ko 2024-03-31T14:49:43.955 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 16049 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs_common/nfs_acl.ko 2024-03-31T14:49:43.955 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfsd/ 2024-03-31T14:49:43.955 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 1511641 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfsd/nfsd.ko 2024-03-31T14:49:43.955 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nilfs2/ 2024-03-31T14:49:43.955 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 582409 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nilfs2/nilfs2.ko 2024-03-31T14:49:43.955 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/ 2024-03-31T14:49:43.955 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 13689 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-celtic.ko 2024-03-31T14:49:43.955 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 13065 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-centeuro.ko 2024-03-31T14:49:43.955 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 13689 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-croatian.ko 2024-03-31T14:49:43.955 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 12761 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-cyrillic.ko 2024-03-31T14:49:43.956 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 13377 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-gaelic.ko 2024-03-31T14:49:43.956 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 12753 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-greek.ko 2024-03-31T14:49:43.956 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 13689 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-iceland.ko 2024-03-31T14:49:43.956 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 13065 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-inuit.ko 2024-03-31T14:49:43.956 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 13993 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-roman.ko 2024-03-31T14:49:43.956 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 13689 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-romanian.ko 2024-03-31T14:49:43.956 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 13689 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-turkish.ko 2024-03-31T14:49:43.956 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 11193 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_ascii.ko 2024-03-31T14:49:43.956 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 12433 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp1250.ko 2024-03-31T14:49:43.956 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 12129 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp1251.ko 2024-03-31T14:49:43.956 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 12817 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp1255.ko 2024-03-31T14:49:43.956 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 12433 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp737.ko 2024-03-31T14:49:43.956 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 12433 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp775.ko 2024-03-31T14:49:43.956 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 12121 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp850.ko 2024-03-31T14:49:43.957 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 12121 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp852.ko 2024-03-31T14:49:43.957 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 12121 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp855.ko 2024-03-31T14:49:43.957 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 11809 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp857.ko 2024-03-31T14:49:43.957 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 12745 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp860.ko 2024-03-31T14:49:43.957 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 13057 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp861.ko 2024-03-31T14:49:43.957 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 13369 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp862.ko 2024-03-31T14:49:43.957 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 13057 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp863.ko 2024-03-31T14:49:43.957 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 12745 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp864.ko 2024-03-31T14:49:43.957 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 13057 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp865.ko 2024-03-31T14:49:43.957 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 12433 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp866.ko 2024-03-31T14:49:43.957 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 12121 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp869.ko 2024-03-31T14:49:43.957 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 11881 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp874.ko 2024-03-31T14:49:43.957 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 94033 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp932.ko 2024-03-31T14:49:43.957 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 142569 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp936.ko 2024-03-31T14:49:43.957 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 165641 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp949.ko 2024-03-31T14:49:43.958 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 118017 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp950.ko 2024-03-31T14:49:43.958 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 12361 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_euc-jp.ko 2024-03-31T14:49:43.958 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 11201 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-1.ko 2024-03-31T14:49:43.958 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 11825 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-13.ko 2024-03-31T14:49:43.958 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 11825 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-14.ko 2024-03-31T14:49:43.958 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 11825 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-15.ko 2024-03-31T14:49:43.958 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 11817 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-2.ko 2024-03-31T14:49:43.958 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 11817 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-3.ko 2024-03-31T14:49:43.958 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 11817 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-4.ko 2024-03-31T14:49:43.958 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 11817 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-5.ko 2024-03-31T14:49:43.958 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 11513 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-6.ko 2024-03-31T14:49:43.958 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 12129 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-7.ko 2024-03-31T14:49:43.958 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 11513 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-9.ko 2024-03-31T14:49:43.958 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 12433 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_koi8-r.ko 2024-03-31T14:49:43.959 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 7921 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_koi8-ru.ko 2024-03-31T14:49:43.959 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 12433 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_koi8-u.ko 2024-03-31T14:49:43.959 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 7465 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_ucs2_utils.ko 2024-03-31T14:49:43.959 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 8249 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_utf8.ko 2024-03-31T14:49:43.959 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ntfs/ 2024-03-31T14:49:43.959 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 220457 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ntfs/ntfs.ko 2024-03-31T14:49:43.959 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2/ 2024-03-31T14:49:43.959 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2/cluster/ 2024-03-31T14:49:43.959 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 244633 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2/cluster/ocfs2_nodemanager.ko 2024-03-31T14:49:43.959 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2/dlm/ 2024-03-31T14:49:43.959 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 440633 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2/dlm/ocfs2_dlm.ko 2024-03-31T14:49:43.959 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2/dlmfs/ 2024-03-31T14:49:43.959 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 54161 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2/dlmfs/ocfs2_dlmfs.ko 2024-03-31T14:49:43.959 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 2590681 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2/ocfs2.ko 2024-03-31T14:49:43.960 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 17809 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2/ocfs2_stack_o2cb.ko 2024-03-31T14:49:43.960 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 31625 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2/ocfs2_stack_user.ko 2024-03-31T14:49:43.960 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 35729 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2/ocfs2_stackglue.ko 2024-03-31T14:49:43.960 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/omfs/ 2024-03-31T14:49:43.960 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 53681 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/omfs/omfs.ko 2024-03-31T14:49:43.960 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/overlayfs/ 2024-03-31T14:49:43.960 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 360897 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/overlayfs/overlay.ko 2024-03-31T14:49:43.960 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/pstore/ 2024-03-31T14:49:43.960 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 56929 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/pstore/ramoops.ko 2024-03-31T14:49:43.960 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/qnx4/ 2024-03-31T14:49:43.960 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 26889 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/qnx4/qnx4.ko 2024-03-31T14:49:43.961 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/qnx6/ 2024-03-31T14:49:43.961 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 46601 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/qnx6/qnx6.ko 2024-03-31T14:49:43.961 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/quota/ 2024-03-31T14:49:43.961 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 30777 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/quota/quota_tree.ko 2024-03-31T14:49:43.961 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 14201 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/quota/quota_v1.ko 2024-03-31T14:49:43.961 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 21921 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/quota/quota_v2.ko 2024-03-31T14:49:43.961 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/reiserfs/ 2024-03-31T14:49:43.961 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 532161 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/reiserfs/reiserfs.ko 2024-03-31T14:49:43.961 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/romfs/ 2024-03-31T14:49:43.961 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 28889 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/romfs/romfs.ko 2024-03-31T14:49:43.961 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/smb/ 2024-03-31T14:49:43.961 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/smb/client/ 2024-03-31T14:49:43.961 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 2712385 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/smb/client/cifs.ko 2024-03-31T14:49:43.962 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/smb/common/ 2024-03-31T14:49:43.962 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 7057 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/smb/common/cifs_arc4.ko 2024-03-31T14:49:43.962 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 11161 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/smb/common/cifs_md4.ko 2024-03-31T14:49:43.962 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/squashfs/ 2024-03-31T14:49:43.962 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 134705 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/squashfs/squashfs.ko 2024-03-31T14:49:43.962 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/sysv/ 2024-03-31T14:49:43.962 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 91281 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/sysv/sysv.ko 2024-03-31T14:49:43.962 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/udf/ 2024-03-31T14:49:43.962 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 248329 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/udf/udf.ko 2024-03-31T14:49:43.962 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ufs/ 2024-03-31T14:49:43.962 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 162737 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ufs/ufs.ko 2024-03-31T14:49:43.962 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/xfs/ 2024-03-31T14:49:43.962 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 4035417 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/xfs/xfs.ko 2024-03-31T14:49:43.963 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/kernel/ 2024-03-31T14:49:43.963 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/kernel/locking/ 2024-03-31T14:49:43.963 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 80481 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/kernel/locking/locktorture.ko 2024-03-31T14:49:43.963 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/kernel/time/ 2024-03-31T14:49:43.963 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 13633 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/kernel/time/test_udelay.ko 2024-03-31T14:49:43.963 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 60257 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/kernel/torture.ko 2024-03-31T14:49:43.963 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/ 2024-03-31T14:49:43.963 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 6729 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/crc-itu-t.ko 2024-03-31T14:49:43.963 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 6433 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/crc7.ko 2024-03-31T14:49:43.963 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 6929 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/crc8.ko 2024-03-31T14:49:43.963 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/crypto/ 2024-03-31T14:49:43.963 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 7001 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/crypto/libarc4.ko 2024-03-31T14:49:43.963 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 36985 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/crypto/libdes.ko 2024-03-31T14:49:43.964 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 13201 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/interval_tree_test.ko 2024-03-31T14:49:43.964 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 8321 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/libcrc32c.ko 2024-03-31T14:49:43.964 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 24849 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/lru_cache.ko 2024-03-31T14:49:43.964 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/lz4/ 2024-03-31T14:49:43.964 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 33697 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/lz4/lz4_compress.ko 2024-03-31T14:49:43.964 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 22529 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/lz4/lz4hc_compress.ko 2024-03-31T14:49:43.964 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/math/ 2024-03-31T14:49:43.964 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 6521 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/math/cordic.ko 2024-03-31T14:49:43.964 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 8113 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/memory-notifier-error-inject.ko 2024-03-31T14:49:43.964 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 10753 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/notifier-error-inject.ko 2024-03-31T14:49:43.964 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 13985 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/percpu_test.ko 2024-03-31T14:49:43.964 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 8193 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/pm-notifier-error-inject.ko 2024-03-31T14:49:43.964 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/raid6/ 2024-03-31T14:49:43.964 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 143257 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/raid6/raid6_pq.ko 2024-03-31T14:49:43.965 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 21009 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/rbtree_test.ko 2024-03-31T14:49:43.965 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/reed_solomon/ 2024-03-31T14:49:43.965 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 18649 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/reed_solomon/reed_solomon.ko 2024-03-31T14:49:43.965 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 38737 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/test-kstrtox.ko 2024-03-31T14:49:43.965 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 26209 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/test-string_helpers.ko 2024-03-31T14:49:43.965 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 4863217 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/test_bpf.ko 2024-03-31T14:49:43.965 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 48985 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/test_firmware.ko 2024-03-31T14:49:43.965 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 6273 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/test_module.ko 2024-03-31T14:49:43.965 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 20705 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/test_user_copy.ko 2024-03-31T14:49:43.965 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 9489 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/ts_bm.ko 2024-03-31T14:49:43.965 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 11025 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/ts_fsm.ko 2024-03-31T14:49:43.965 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 9529 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/ts_kmp.ko 2024-03-31T14:49:43.965 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/xz/ 2024-03-31T14:49:43.965 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 13441 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/xz/xz_dec_test.ko 2024-03-31T14:49:43.966 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/mm/ 2024-03-31T14:49:43.966 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 18121 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/mm/hwpoison-inject.ko 2024-03-31T14:49:43.966 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ 2024-03-31T14:49:43.966 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/6lowpan/ 2024-03-31T14:49:43.966 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 55753 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/6lowpan/6lowpan.ko 2024-03-31T14:49:43.966 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 6953 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/6lowpan/nhc_dest.ko 2024-03-31T14:49:43.966 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 6961 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/6lowpan/nhc_fragment.ko 2024-03-31T14:49:43.966 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 6953 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/6lowpan/nhc_hop.ko 2024-03-31T14:49:43.966 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 6889 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/6lowpan/nhc_ipv6.ko 2024-03-31T14:49:43.966 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 6961 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/6lowpan/nhc_mobility.ko 2024-03-31T14:49:43.966 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 6897 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/6lowpan/nhc_routing.ko 2024-03-31T14:49:43.966 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 15233 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/6lowpan/nhc_udp.ko 2024-03-31T14:49:43.966 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/802/ 2024-03-31T14:49:43.966 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 30401 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/802/garp.ko 2024-03-31T14:49:43.967 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 34745 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/802/mrp.ko 2024-03-31T14:49:43.967 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 8561 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/802/p8022.ko 2024-03-31T14:49:43.967 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 16297 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/802/psnap.ko 2024-03-31T14:49:43.967 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 11553 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/802/stp.ko 2024-03-31T14:49:43.967 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/8021q/ 2024-03-31T14:49:43.967 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 82865 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/8021q/8021q.ko 2024-03-31T14:49:43.967 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/9p/ 2024-03-31T14:49:43.967 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 198809 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/9p/9pnet.ko 2024-03-31T14:49:43.967 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 36137 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/9p/9pnet_fd.ko 2024-03-31T14:49:43.967 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 26241 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/9p/9pnet_rdma.ko 2024-03-31T14:49:43.967 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 32241 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/9p/9pnet_virtio.ko 2024-03-31T14:49:43.967 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/batman-adv/ 2024-03-31T14:49:43.967 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 667353 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/batman-adv/batman-adv.ko 2024-03-31T14:49:43.967 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/ 2024-03-31T14:49:43.968 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 67753 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/br_netfilter.ko 2024-03-31T14:49:43.968 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 790545 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/bridge.ko 2024-03-31T14:49:43.968 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ 2024-03-31T14:49:43.968 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 7553 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_802_3.ko 2024-03-31T14:49:43.968 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 12281 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_among.ko 2024-03-31T14:49:43.968 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 9073 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_arp.ko 2024-03-31T14:49:43.968 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 8969 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_arpreply.ko 2024-03-31T14:49:43.968 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 9529 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_dnat.ko 2024-03-31T14:49:43.968 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 8641 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_ip.ko 2024-03-31T14:49:43.968 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 9049 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_ip6.ko 2024-03-31T14:49:43.968 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 9721 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_limit.ko 2024-03-31T14:49:43.968 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 13185 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_log.ko 2024-03-31T14:49:43.968 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 7489 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_mark.ko 2024-03-31T14:49:43.968 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 7489 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_mark_m.ko 2024-03-31T14:49:43.969 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 8281 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_nflog.ko 2024-03-31T14:49:43.969 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 7433 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_pkttype.ko 2024-03-31T14:49:43.969 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 9489 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_redirect.ko 2024-03-31T14:49:43.969 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 8553 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_snat.ko 2024-03-31T14:49:43.969 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 8729 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_stp.ko 2024-03-31T14:49:43.969 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 12305 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_vlan.ko 2024-03-31T14:49:43.969 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 11537 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebtable_broute.ko 2024-03-31T14:49:43.969 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 9833 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebtable_filter.ko 2024-03-31T14:49:43.969 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 9809 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebtable_nat.ko 2024-03-31T14:49:43.969 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 55937 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebtables.ko 2024-03-31T14:49:43.969 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 13937 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/nft_meta_bridge.ko 2024-03-31T14:49:43.969 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 13841 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/nft_reject_bridge.ko 2024-03-31T14:49:43.969 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ceph/ 2024-03-31T14:49:43.969 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 1046689 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ceph/libceph.ko 2024-03-31T14:49:43.970 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/core/ 2024-03-31T14:49:43.970 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 125697 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/core/pktgen.ko 2024-03-31T14:49:43.970 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dccp/ 2024-03-31T14:49:43.970 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 194441 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dccp/dccp.ko 2024-03-31T14:49:43.970 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 8561 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dccp/dccp_diag.ko 2024-03-31T14:49:43.970 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 58737 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dccp/dccp_ipv4.ko 2024-03-31T14:49:43.970 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 61393 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dccp/dccp_ipv6.ko 2024-03-31T14:49:43.970 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dsa/ 2024-03-31T14:49:43.970 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 390033 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dsa/dsa_core.ko 2024-03-31T14:49:43.970 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 13353 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dsa/tag_brcm.ko 2024-03-31T14:49:43.970 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 12593 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dsa/tag_dsa.ko 2024-03-31T14:49:43.970 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 7737 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dsa/tag_none.ko 2024-03-31T14:49:43.970 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 8969 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dsa/tag_trailer.ko 2024-03-31T14:49:43.970 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/hsr/ 2024-03-31T14:49:43.971 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 102033 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/hsr/hsr.ko 2024-03-31T14:49:43.971 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ieee802154/ 2024-03-31T14:49:43.971 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ieee802154/6lowpan/ 2024-03-31T14:49:43.971 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 45481 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ieee802154/6lowpan/ieee802154_6lowpan.ko 2024-03-31T14:49:43.971 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 284665 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ieee802154/ieee802154.ko 2024-03-31T14:49:43.971 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 47873 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ieee802154/ieee802154_socket.ko 2024-03-31T14:49:43.971 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/ 2024-03-31T14:49:43.971 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 24777 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/ah4.ko 2024-03-31T14:49:43.971 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 39089 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/esp4.ko 2024-03-31T14:49:43.971 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 48233 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/fou.ko 2024-03-31T14:49:43.971 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 18025 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/gre.ko 2024-03-31T14:49:43.971 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 51209 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/inet_diag.ko 2024-03-31T14:49:43.971 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 61129 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/ip_gre.ko 2024-03-31T14:49:43.972 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 61505 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/ip_tunnel.ko 2024-03-31T14:49:43.972 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 34921 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/ip_vti.ko 2024-03-31T14:49:43.972 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 14129 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/ipcomp.ko 2024-03-31T14:49:43.972 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 29073 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/ipip.ko 2024-03-31T14:49:43.972 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/ 2024-03-31T14:49:43.972 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 38001 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/arp_tables.ko 2024-03-31T14:49:43.972 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 8769 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/arpt_mangle.ko 2024-03-31T14:49:43.972 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 10313 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/arptable_filter.ko 2024-03-31T14:49:43.972 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 42049 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/ip_tables.ko 2024-03-31T14:49:43.972 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 9961 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/ipt_ECN.ko 2024-03-31T14:49:43.972 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 9577 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/ipt_REJECT.ko 2024-03-31T14:49:43.972 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 14017 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/ipt_SYNPROXY.ko 2024-03-31T14:49:43.972 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 11081 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/ipt_ah.ko 2024-03-31T14:49:43.972 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 14777 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/ipt_rpfilter.ko 2024-03-31T14:49:43.973 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 11201 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/iptable_filter.ko 2024-03-31T14:49:43.973 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 11553 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/iptable_mangle.ko 2024-03-31T14:49:43.973 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 14921 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/iptable_nat.ko 2024-03-31T14:49:43.973 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 11553 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/iptable_raw.ko 2024-03-31T14:49:43.973 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 10313 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/iptable_security.ko 2024-03-31T14:49:43.973 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 12441 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/nf_defrag_ipv4.ko 2024-03-31T14:49:43.973 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 8713 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/nf_dup_ipv4.ko 2024-03-31T14:49:43.973 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 27521 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/nf_nat_h323.ko 2024-03-31T14:49:43.973 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 18193 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/nf_nat_pptp.ko 2024-03-31T14:49:43.973 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 13225 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/nf_nat_snmp_basic.ko 2024-03-31T14:49:43.973 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 21033 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/nf_reject_ipv4.ko 2024-03-31T14:49:43.973 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 9225 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/nf_socket_ipv4.ko 2024-03-31T14:49:43.973 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 15073 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/nf_tproxy_ipv4.ko 2024-03-31T14:49:43.973 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 9857 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/nft_dup_ipv4.ko 2024-03-31T14:49:43.973 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 17009 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/nft_fib_ipv4.ko 2024-03-31T14:49:43.974 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 8905 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/nft_reject_ipv4.ko 2024-03-31T14:49:43.974 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 12977 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_bic.ko 2024-03-31T14:49:43.974 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 13641 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_dctcp.ko 2024-03-31T14:49:43.974 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 15961 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_diag.ko 2024-03-31T14:49:43.974 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 9433 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_highspeed.ko 2024-03-31T14:49:43.974 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 12305 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_htcp.ko 2024-03-31T14:49:43.974 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 10345 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_hybla.ko 2024-03-31T14:49:43.974 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 11369 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_illinois.ko 2024-03-31T14:49:43.974 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 10337 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_lp.ko 2024-03-31T14:49:43.974 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 8225 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_scalable.ko 2024-03-31T14:49:43.974 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 14321 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_vegas.ko 2024-03-31T14:49:43.974 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 9993 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_veno.ko 2024-03-31T14:49:43.974 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 10849 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_westwood.ko 2024-03-31T14:49:43.974 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 10089 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_yeah.ko 2024-03-31T14:49:43.974 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 22625 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tunnel4.ko 2024-03-31T14:49:43.975 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 17761 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/udp_diag.ko 2024-03-31T14:49:43.975 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 45041 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/udp_tunnel.ko 2024-03-31T14:49:43.975 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 11377 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/xfrm4_tunnel.ko 2024-03-31T14:49:43.975 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/ 2024-03-31T14:49:43.975 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 26433 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/ah6.ko 2024-03-31T14:49:43.975 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 39721 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/esp6.ko 2024-03-31T14:49:43.975 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 14337 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/fou6.ko 2024-03-31T14:49:43.975 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 79921 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/ip6_gre.ko 2024-03-31T14:49:43.975 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 94161 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/ip6_tunnel.ko 2024-03-31T14:49:43.975 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 14249 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/ip6_udp_tunnel.ko 2024-03-31T14:49:43.975 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 54265 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/ip6_vti.ko 2024-03-31T14:49:43.975 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 13873 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/ipcomp6.ko 2024-03-31T14:49:43.975 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 19201 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/mip6.ko 2024-03-31T14:49:43.976 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ 2024-03-31T14:49:43.976 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 42545 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6_tables.ko 2024-03-31T14:49:43.976 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 11657 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6t_NPT.ko 2024-03-31T14:49:43.976 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 9649 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6t_REJECT.ko 2024-03-31T14:49:43.976 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 14033 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6t_SYNPROXY.ko 2024-03-31T14:49:43.976 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 14177 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6t_ah.ko 2024-03-31T14:49:43.976 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 7697 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6t_eui64.ko 2024-03-31T14:49:43.976 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 16105 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6t_frag.ko 2024-03-31T14:49:43.976 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 15937 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6t_hbh.ko 2024-03-31T14:49:43.976 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 8809 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6t_ipv6header.ko 2024-03-31T14:49:43.976 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 9801 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6t_mh.ko 2024-03-31T14:49:43.976 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 10833 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6t_rpfilter.ko 2024-03-31T14:49:43.976 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 10801 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6t_rt.ko 2024-03-31T14:49:43.976 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 11217 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6table_filter.ko 2024-03-31T14:49:43.977 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 11025 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6table_mangle.ko 2024-03-31T14:49:43.977 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 15017 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6table_nat.ko 2024-03-31T14:49:43.977 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 11569 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6table_raw.ko 2024-03-31T14:49:43.977 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 10329 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6table_security.ko 2024-03-31T14:49:43.977 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 39121 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/nf_defrag_ipv6.ko 2024-03-31T14:49:43.977 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 8345 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/nf_dup_ipv6.ko 2024-03-31T14:49:43.977 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 25257 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/nf_reject_ipv6.ko 2024-03-31T14:49:43.977 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 9697 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/nf_socket_ipv6.ko 2024-03-31T14:49:43.977 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 15289 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/nf_tproxy_ipv6.ko 2024-03-31T14:49:43.977 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 9529 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/nft_dup_ipv6.ko 2024-03-31T14:49:43.977 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 15921 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/nft_fib_ipv6.ko 2024-03-31T14:49:43.977 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 8905 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/nft_reject_ipv6.ko 2024-03-31T14:49:43.977 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 74873 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/sit.ko 2024-03-31T14:49:43.978 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 24145 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/tunnel6.ko 2024-03-31T14:49:43.978 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 23289 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/xfrm6_tunnel.ko 2024-03-31T14:49:43.978 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/key/ 2024-03-31T14:49:43.978 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 84393 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/key/af_key.ko 2024-03-31T14:49:43.978 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/l2tp/ 2024-03-31T14:49:43.978 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 125201 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/l2tp/l2tp_core.ko 2024-03-31T14:49:43.978 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 20073 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/l2tp/l2tp_debugfs.ko 2024-03-31T14:49:43.978 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 25809 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/l2tp/l2tp_eth.ko 2024-03-31T14:49:43.978 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 36489 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/l2tp/l2tp_ip.ko 2024-03-31T14:49:43.978 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 37369 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/l2tp/l2tp_ip6.ko 2024-03-31T14:49:43.978 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 33441 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/l2tp/l2tp_netlink.ko 2024-03-31T14:49:43.978 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 53225 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/l2tp/l2tp_ppp.ko 2024-03-31T14:49:43.978 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/lapb/ 2024-03-31T14:49:43.979 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 48097 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/lapb/lapb.ko 2024-03-31T14:49:43.979 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/llc/ 2024-03-31T14:49:43.979 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 22817 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/llc/llc.ko 2024-03-31T14:49:43.979 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 268449 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/llc/llc2.ko 2024-03-31T14:49:43.979 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/mac802154/ 2024-03-31T14:49:43.979 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 309361 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/mac802154/mac802154.ko 2024-03-31T14:49:43.979 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/mpls/ 2024-03-31T14:49:43.979 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 9297 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/mpls/mpls_gso.ko 2024-03-31T14:49:43.979 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ 2024-03-31T14:49:43.979 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ 2024-03-31T14:49:43.979 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 119689 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set.ko 2024-03-31T14:49:43.979 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 28609 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_bitmap_ip.ko 2024-03-31T14:49:43.979 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 27297 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_bitmap_ipmac.ko 2024-03-31T14:49:43.979 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 26721 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_bitmap_port.ko 2024-03-31T14:49:43.980 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 99985 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_ip.ko 2024-03-31T14:49:43.980 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 100273 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_ipmark.ko 2024-03-31T14:49:43.980 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 102545 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_ipport.ko 2024-03-31T14:49:43.980 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 102337 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_ipportip.ko 2024-03-31T14:49:43.980 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 111665 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_ipportnet.ko 2024-03-31T14:49:43.980 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 59185 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_mac.ko 2024-03-31T14:49:43.980 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 108337 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_net.ko 2024-03-31T14:49:43.980 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 112137 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_netiface.ko 2024-03-31T14:49:43.980 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 112401 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_netnet.ko 2024-03-31T14:49:43.980 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 110329 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_netport.ko 2024-03-31T14:49:43.980 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 113281 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_netportnet.ko 2024-03-31T14:49:43.980 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 30297 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_list_set.ko 2024-03-31T14:49:43.980 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ 2024-03-31T14:49:43.980 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 412433 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs.ko 2024-03-31T14:49:43.981 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 11985 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_dh.ko 2024-03-31T14:49:43.981 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 7649 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_fo.ko 2024-03-31T14:49:43.981 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 17673 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_ftp.ko 2024-03-31T14:49:43.981 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 17929 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_lblc.ko 2024-03-31T14:49:43.981 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 20761 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_lblcr.ko 2024-03-31T14:49:43.981 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 7649 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_lc.ko 2024-03-31T14:49:43.981 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 7713 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_nq.ko 2024-03-31T14:49:43.981 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 10633 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_pe_sip.ko 2024-03-31T14:49:43.981 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 9257 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_rr.ko 2024-03-31T14:49:43.981 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 7713 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_sed.ko 2024-03-31T14:49:43.981 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 13537 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_sh.ko 2024-03-31T14:49:43.981 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 7713 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_wlc.ko 2024-03-31T14:49:43.981 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 11001 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_wrr.ko 2024-03-31T14:49:43.981 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 49161 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conncount.ko 2024-03-31T14:49:43.982 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 411665 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack.ko 2024-03-31T14:49:43.982 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 15873 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_amanda.ko 2024-03-31T14:49:43.982 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 10289 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_broadcast.ko 2024-03-31T14:49:43.982 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 33657 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_ftp.ko 2024-03-31T14:49:43.982 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 118145 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_h323.ko 2024-03-31T14:49:43.982 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 22081 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_irc.ko 2024-03-31T14:49:43.982 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 8705 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_netbios_ns.ko 2024-03-31T14:49:43.982 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 108305 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_netlink.ko 2024-03-31T14:49:43.982 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 33569 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_pptp.ko 2024-03-31T14:49:43.982 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 17289 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_sane.ko 2024-03-31T14:49:43.982 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 66529 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_sip.ko 2024-03-31T14:49:43.982 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 11129 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_snmp.ko 2024-03-31T14:49:43.982 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 19073 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_tftp.ko 2024-03-31T14:49:43.983 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 10009 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_dup_netdev.ko 2024-03-31T14:49:43.983 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 92217 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_flow_table.ko 2024-03-31T14:49:43.983 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 9545 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_flow_table_inet.ko 2024-03-31T14:49:43.983 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 38073 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_log_syslog.ko 2024-03-31T14:49:43.983 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 96489 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_nat.ko 2024-03-31T14:49:43.983 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 10025 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_nat_amanda.ko 2024-03-31T14:49:43.983 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 12521 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_nat_ftp.ko 2024-03-31T14:49:43.983 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 11753 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_nat_irc.ko 2024-03-31T14:49:43.983 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 28825 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_nat_sip.ko 2024-03-31T14:49:43.983 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 8705 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_nat_tftp.ko 2024-03-31T14:49:43.983 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 44129 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_synproxy_core.ko 2024-03-31T14:49:43.983 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 623337 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_tables.ko 2024-03-31T14:49:43.983 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 42505 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nfnetlink.ko 2024-03-31T14:49:43.983 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 30153 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nfnetlink_acct.ko 2024-03-31T14:49:43.984 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 30345 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nfnetlink_cttimeout.ko 2024-03-31T14:49:43.984 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 39553 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nfnetlink_log.ko 2024-03-31T14:49:43.985 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 22929 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nfnetlink_osf.ko 2024-03-31T14:49:43.985 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 49641 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nfnetlink_queue.ko 2024-03-31T14:49:43.985 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 10465 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_chain_nat.ko 2024-03-31T14:49:43.985 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 33553 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_compat.ko 2024-03-31T14:49:43.985 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 40801 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_ct.ko 2024-03-31T14:49:43.985 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 9497 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_dup_netdev.ko 2024-03-31T14:49:43.985 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 11417 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_fib.ko 2024-03-31T14:49:43.986 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 13761 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_fwd_netdev.ko 2024-03-31T14:49:43.986 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 14089 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_hash.ko 2024-03-31T14:49:43.986 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 20025 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_limit.ko 2024-03-31T14:49:43.986 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 15305 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_log.ko 2024-03-31T14:49:43.986 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 14705 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_masq.ko 2024-03-31T14:49:43.986 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 14321 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_nat.ko 2024-03-31T14:49:43.986 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 13569 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_numgen.ko 2024-03-31T14:49:43.986 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 11745 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_osf.ko 2024-03-31T14:49:43.986 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 13969 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_queue.ko 2024-03-31T14:49:43.986 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 14089 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_redir.ko 2024-03-31T14:49:43.986 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 10393 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_reject.ko 2024-03-31T14:49:43.986 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 9977 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_reject_inet.ko 2024-03-31T14:49:43.986 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 12833 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_socket.ko 2024-03-31T14:49:43.986 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 20529 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_synproxy.ko 2024-03-31T14:49:43.987 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 14809 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_tproxy.ko 2024-03-31T14:49:43.987 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 89225 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/x_tables.ko 2024-03-31T14:49:43.987 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 12001 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_AUDIT.ko 2024-03-31T14:49:43.987 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 9529 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_CHECKSUM.ko 2024-03-31T14:49:43.987 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 7625 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_CLASSIFY.ko 2024-03-31T14:49:43.987 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 11721 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_CONNSECMARK.ko 2024-03-31T14:49:43.987 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 18073 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_CT.ko 2024-03-31T14:49:43.987 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 10401 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_DSCP.ko 2024-03-31T14:49:43.987 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 9369 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_HL.ko 2024-03-31T14:49:43.987 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 13273 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_HMARK.ko 2024-03-31T14:49:43.987 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 38769 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_IDLETIMER.ko 2024-03-31T14:49:43.987 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 13793 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_LED.ko 2024-03-31T14:49:43.987 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 11393 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_LOG.ko 2024-03-31T14:49:43.987 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 11649 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_MASQUERADE.ko 2024-03-31T14:49:43.988 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 11553 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_NETMAP.ko 2024-03-31T14:49:43.988 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 9377 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_NFLOG.ko 2024-03-31T14:49:43.988 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 13185 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_NFQUEUE.ko 2024-03-31T14:49:43.988 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 19561 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_RATEEST.ko 2024-03-31T14:49:43.988 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 10969 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_REDIRECT.ko 2024-03-31T14:49:43.988 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 14241 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_SECMARK.ko 2024-03-31T14:49:43.988 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 15401 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_TCPMSS.ko 2024-03-31T14:49:43.988 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 10329 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_TCPOPTSTRIP.ko 2024-03-31T14:49:43.988 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 17401 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_TEE.ko 2024-03-31T14:49:43.988 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 16689 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_TPROXY.ko 2024-03-31T14:49:43.988 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 8225 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_TRACE.ko 2024-03-31T14:49:43.988 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 13089 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_addrtype.ko 2024-03-31T14:49:43.988 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 13801 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_bpf.ko 2024-03-31T14:49:43.988 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 16929 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_cgroup.ko 2024-03-31T14:49:43.989 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 11113 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_cluster.ko 2024-03-31T14:49:43.989 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 7401 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_comment.ko 2024-03-31T14:49:43.989 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 10361 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_connbytes.ko 2024-03-31T14:49:43.989 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 11249 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_connlabel.ko 2024-03-31T14:49:43.989 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 9297 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_connlimit.ko 2024-03-31T14:49:43.989 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 13345 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_connmark.ko 2024-03-31T14:49:43.989 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 12297 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_conntrack.ko 2024-03-31T14:49:43.989 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 7721 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_cpu.ko 2024-03-31T14:49:43.989 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 10881 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_dccp.ko 2024-03-31T14:49:43.989 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 7761 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_devgroup.ko 2024-03-31T14:49:43.989 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 8705 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_dscp.ko 2024-03-31T14:49:43.989 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 10561 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_ecn.ko 2024-03-31T14:49:43.989 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 11377 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_esp.ko 2024-03-31T14:49:43.989 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 43193 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_hashlimit.ko 2024-03-31T14:49:43.990 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 11209 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_helper.ko 2024-03-31T14:49:43.990 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 7769 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_hl.ko 2024-03-31T14:49:43.990 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 11817 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_ipcomp.ko 2024-03-31T14:49:43.990 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 11369 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_iprange.ko 2024-03-31T14:49:43.990 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 11689 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_ipvs.ko 2024-03-31T14:49:43.990 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 15945 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_l2tp.ko 2024-03-31T14:49:43.990 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 7897 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_length.ko 2024-03-31T14:49:43.990 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 10113 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_limit.ko 2024-03-31T14:49:43.990 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 7449 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_mac.ko 2024-03-31T14:49:43.990 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 8481 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_mark.ko 2024-03-31T14:49:43.990 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 11721 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_multiport.ko 2024-03-31T14:49:43.990 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 13729 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_nat.ko 2024-03-31T14:49:43.990 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 10001 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_nfacct.ko 2024-03-31T14:49:43.991 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 8209 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_osf.ko 2024-03-31T14:49:43.991 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 9969 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_owner.ko 2024-03-31T14:49:43.991 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 10009 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_physdev.ko 2024-03-31T14:49:43.991 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 7529 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_pkttype.ko 2024-03-31T14:49:43.991 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 11889 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_policy.ko 2024-03-31T14:49:43.991 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 9377 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_quota.ko 2024-03-31T14:49:43.991 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 10305 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_rateest.ko 2024-03-31T14:49:43.991 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 8161 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_realm.ko 2024-03-31T14:49:43.991 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 36353 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_recent.ko 2024-03-31T14:49:43.991 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 13233 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_sctp.ko 2024-03-31T14:49:43.991 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 29993 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_set.ko 2024-03-31T14:49:43.991 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 16809 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_socket.ko 2024-03-31T14:49:43.991 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 9409 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_state.ko 2024-03-31T14:49:43.991 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 9369 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_statistic.ko 2024-03-31T14:49:43.991 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 8705 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_string.ko 2024-03-31T14:49:43.992 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 8449 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_tcpmss.ko 2024-03-31T14:49:43.992 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 15673 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_tcpudp.ko 2024-03-31T14:49:43.992 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 10969 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_time.ko 2024-03-31T14:49:43.992 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 9305 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_u32.ko 2024-03-31T14:49:43.992 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netlink/ 2024-03-31T14:49:43.992 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 14433 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netlink/netlink_diag.ko 2024-03-31T14:49:43.992 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/nsh/ 2024-03-31T14:49:43.992 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 12329 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/nsh/nsh.ko 2024-03-31T14:49:43.992 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/openvswitch/ 2024-03-31T14:49:43.992 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 384137 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/openvswitch/openvswitch.ko 2024-03-31T14:49:43.992 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 10361 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/openvswitch/vport-geneve.ko 2024-03-31T14:49:43.992 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 9481 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/openvswitch/vport-gre.ko 2024-03-31T14:49:43.992 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 11825 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/openvswitch/vport-vxlan.ko 2024-03-31T14:49:43.992 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/packet/ 2024-03-31T14:49:43.993 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 13465 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/packet/af_packet_diag.ko 2024-03-31T14:49:43.993 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/phonet/ 2024-03-31T14:49:43.993 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 112521 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/phonet/phonet.ko 2024-03-31T14:49:43.993 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 55041 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/phonet/pn_pep.ko 2024-03-31T14:49:43.993 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/rds/ 2024-03-31T14:49:43.993 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 253881 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/rds/rds.ko 2024-03-31T14:49:43.993 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 165049 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/rds/rds_rdma.ko 2024-03-31T14:49:43.993 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 68065 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/rds/rds_tcp.ko 2024-03-31T14:49:43.993 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/rxrpc/ 2024-03-31T14:49:43.993 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 828145 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/rxrpc/rxrpc.ko 2024-03-31T14:49:43.993 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/ 2024-03-31T14:49:43.993 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 32145 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/act_csum.ko 2024-03-31T14:49:43.993 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 23481 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/act_gact.ko 2024-03-31T14:49:43.993 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 37081 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/act_mirred.ko 2024-03-31T14:49:43.994 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 23369 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/act_nat.ko 2024-03-31T14:49:43.994 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 43345 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/act_pedit.ko 2024-03-31T14:49:43.994 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 31921 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/act_police.ko 2024-03-31T14:49:43.994 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 20185 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/act_simple.ko 2024-03-31T14:49:43.994 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 35193 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/act_skbedit.ko 2024-03-31T14:49:43.994 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 36569 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/act_vlan.ko 2024-03-31T14:49:43.994 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 26177 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/cls_basic.ko 2024-03-31T14:49:43.994 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 33865 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/cls_bpf.ko 2024-03-31T14:49:43.994 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 19177 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/cls_cgroup.ko 2024-03-31T14:49:43.994 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 31401 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/cls_flow.ko 2024-03-31T14:49:43.994 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 31857 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/cls_fw.ko 2024-03-31T14:49:43.994 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 39513 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/cls_route.ko 2024-03-31T14:49:43.994 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 62649 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/cls_u32.ko 2024-03-31T14:49:43.994 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 7457 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/em_cmp.ko 2024-03-31T14:49:43.995 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 13313 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/em_ipset.ko 2024-03-31T14:49:43.995 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 33417 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/em_meta.ko 2024-03-31T14:49:43.995 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 7913 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/em_nbyte.ko 2024-03-31T14:49:43.995 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 10921 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/em_text.ko 2024-03-31T14:49:43.995 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 7265 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/em_u32.ko 2024-03-31T14:49:43.995 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 21665 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_choke.ko 2024-03-31T14:49:43.995 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 20857 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_codel.ko 2024-03-31T14:49:43.995 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 34585 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_drr.ko 2024-03-31T14:49:43.995 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 35785 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_fq.ko 2024-03-31T14:49:43.995 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 31513 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_fq_codel.ko 2024-03-31T14:49:43.995 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 35449 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_gred.ko 2024-03-31T14:49:43.995 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 49841 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_hfsc.ko 2024-03-31T14:49:43.996 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 21969 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_hhf.ko 2024-03-31T14:49:43.996 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 66441 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_htb.ko 2024-03-31T14:49:43.996 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 26081 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_ingress.ko 2024-03-31T14:49:43.996 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 30217 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_mqprio.ko 2024-03-31T14:49:43.996 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 10801 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_mqprio_lib.ko 2024-03-31T14:49:43.996 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 26081 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_multiq.ko 2024-03-31T14:49:43.996 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 36145 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_netem.ko 2024-03-31T14:49:43.996 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 27401 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_pie.ko 2024-03-31T14:49:43.996 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 10657 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_plug.ko 2024-03-31T14:49:43.996 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 24769 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_prio.ko 2024-03-31T14:49:43.996 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 48529 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_qfq.ko 2024-03-31T14:49:43.996 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 32809 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_red.ko 2024-03-31T14:49:43.996 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 26785 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_sfb.ko 2024-03-31T14:49:43.996 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 31313 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_sfq.ko 2024-03-31T14:49:43.996 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 28937 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_tbf.ko 2024-03-31T14:49:43.997 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 28265 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_teql.ko 2024-03-31T14:49:43.997 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sctp/ 2024-03-31T14:49:43.997 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 932785 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sctp/sctp.ko 2024-03-31T14:49:43.997 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 20241 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sctp/sctp_diag.ko 2024-03-31T14:49:43.997 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sunrpc/ 2024-03-31T14:49:43.997 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sunrpc/auth_gss/ 2024-03-31T14:49:43.997 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 336969 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sunrpc/auth_gss/auth_rpcgss.ko 2024-03-31T14:49:43.997 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 55009 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sunrpc/auth_gss/rpcsec_gss_krb5.ko 2024-03-31T14:49:43.997 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 1513769 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sunrpc/sunrpc.ko 2024-03-31T14:49:43.997 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sunrpc/xprtrdma/ 2024-03-31T14:49:43.997 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 704697 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sunrpc/xprtrdma/rpcrdma.ko 2024-03-31T14:49:43.997 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/tipc/ 2024-03-31T14:49:43.997 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 10393 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/tipc/diag.ko 2024-03-31T14:49:43.997 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 867121 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/tipc/tipc.ko 2024-03-31T14:49:43.998 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/unix/ 2024-03-31T14:49:43.998 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 15929 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/unix/unix_diag.ko 2024-03-31T14:49:43.998 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/vmw_vsock/ 2024-03-31T14:49:43.998 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 87737 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/vmw_vsock/vmw_vsock_virtio_transport_common.ko 2024-03-31T14:49:43.998 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 67937 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/vmw_vsock/vmw_vsock_vmci_transport.ko 2024-03-31T14:49:43.998 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 98089 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/vmw_vsock/vsock.ko 2024-03-31T14:49:43.998 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 11073 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/vmw_vsock/vsock_diag.ko 2024-03-31T14:49:43.998 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 17809 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/vmw_vsock/vsock_loopback.ko 2024-03-31T14:49:43.998 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/x25/ 2024-03-31T14:49:43.998 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 131209 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/x25/x25.ko 2024-03-31T14:49:43.998 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/xfrm/ 2024-03-31T14:49:43.998 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 23057 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/xfrm/xfrm_algo.ko 2024-03-31T14:49:43.998 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 20545 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/xfrm/xfrm_ipcomp.ko 2024-03-31T14:49:43.998 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 105209 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/xfrm/xfrm_user.ko 2024-03-31T14:49:43.998 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/virt/ 2024-03-31T14:49:43.999 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/virt/lib/ 2024-03-31T14:49:43.999 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 15433 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/virt/lib/irqbypass.ko 2024-03-31T14:49:43.999 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 642383 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/modules.alias 2024-03-31T14:49:43.999 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 625850 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/modules.alias.bin 2024-03-31T14:49:43.999 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 7677 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/modules.builtin 2024-03-31T14:49:43.999 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 23498 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/modules.builtin.alias.bin 2024-03-31T14:49:43.999 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 9916 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/modules.builtin.bin 2024-03-31T14:49:43.999 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 70732 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/modules.builtin.modinfo 2024-03-31T14:49:43.999 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 147214 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/modules.dep 2024-03-31T14:49:43.999 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 233213 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/modules.dep.bin 2024-03-31T14:49:43.999 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 210 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/modules.devname 2024-03-31T14:49:43.999 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 72614 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/modules.order 2024-03-31T14:49:43.999 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 879 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/modules.softdep 2024-03-31T14:49:43.999 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 257955 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/modules.symbols 2024-03-31T14:49:43.999 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 312782 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/modules.symbols.bin 2024-03-31T14:49:44.000 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./usr/ 2024-03-31T14:49:44.000 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./usr/share/ 2024-03-31T14:49:44.000 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./usr/share/doc/ 2024-03-31T14:49:44.000 INFO:teuthology.orchestra.run.smithi016.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./usr/share/doc/linux-image-6.8.0-g1df0c7e8a235/ 2024-03-31T14:49:44.000 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 169 2024-03-25 03:29 ./usr/share/doc/linux-image-6.8.0-g1df0c7e8a235/changelog.Debian.gz 2024-03-31T14:49:44.000 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r-- root/root 691 2024-03-25 03:29 ./usr/share/doc/linux-image-6.8.0-g1df0c7e8a235/copyright 2024-03-31T14:49:44.005 DEBUG:teuthology.task.kernel:get_image_version: 6.8.0-g1df0c7e8a235 2024-03-31T14:49:44.005 INFO:teuthology.task.kernel:searching for kernel 6.8.0-g1df0c7e8a235 2024-03-31T14:49:44.006 DEBUG:teuthology.orchestra.run.smithi016:> egrep '(submenu|menuentry.*6.8.0-g1df0c7e8a235).*{' /boot/grub/grub.cfg 2024-03-31T14:49:44.015 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/ 2024-03-31T14:49:44.015 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 53833 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/af_alg.ko 2024-03-31T14:49:44.015 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 23433 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/algif_hash.ko 2024-03-31T14:49:44.015 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 21585 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/algif_skcipher.ko 2024-03-31T14:49:44.015 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 17105 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/ansi_cprng.ko 2024-03-31T14:49:44.015 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 19545 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/anubis.ko 2024-03-31T14:49:44.015 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 9577 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/arc4.ko 2024-03-31T14:49:44.015 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/asymmetric_keys/ 2024-03-31T14:49:44.016 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 10081 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/asymmetric_keys/pkcs7_test_key.ko 2024-03-31T14:49:44.016 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/async_tx/ 2024-03-31T14:49:44.016 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 11145 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/async_tx/async_memcpy.ko 2024-03-31T14:49:44.016 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 19377 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/async_tx/async_pq.ko 2024-03-31T14:49:44.016 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 21449 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/async_tx/async_raid6_recov.ko 2024-03-31T14:49:44.016 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 13729 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/async_tx/async_tx.ko 2024-03-31T14:49:44.016 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 17649 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/async_tx/async_xor.ko 2024-03-31T14:49:44.016 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 17817 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/async_tx/raid6test.ko 2024-03-31T14:49:44.016 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 20329 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/authenc.ko 2024-03-31T14:49:44.016 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 22345 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/authencesn.ko 2024-03-31T14:49:44.016 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 21721 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/blake2b_generic.ko 2024-03-31T14:49:44.016 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 12625 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/blowfish_common.ko 2024-03-31T14:49:44.016 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 10433 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/blowfish_generic.ko 2024-03-31T14:49:44.016 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 38705 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/camellia_generic.ko 2024-03-31T14:49:44.017 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 27225 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/cast5_generic.ko 2024-03-31T14:49:44.017 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 17649 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/cast6_generic.ko 2024-03-31T14:49:44.017 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 10433 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/cast_common.ko 2024-03-31T14:49:44.017 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 35177 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/ccm.ko 2024-03-31T14:49:44.017 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 14561 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/cmac.ko 2024-03-31T14:49:44.017 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 9481 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/crc32_generic.ko 2024-03-31T14:49:44.017 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 47673 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/cryptd.ko 2024-03-31T14:49:44.017 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 39465 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/crypto_engine.ko 2024-03-31T14:49:44.017 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 26913 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/crypto_simd.ko 2024-03-31T14:49:44.017 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 24937 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/crypto_user.ko 2024-03-31T14:49:44.017 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 17641 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/cts.ko 2024-03-31T14:49:44.017 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 10809 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/des_generic.ko 2024-03-31T14:49:44.017 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 15577 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/dh_generic.ko 2024-03-31T14:49:44.017 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 10905 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/echainiv.ko 2024-03-31T14:49:44.018 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 25233 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/essiv.ko 2024-03-31T14:49:44.018 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 17897 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/fcrypt.ko 2024-03-31T14:49:44.018 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 26801 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/khazad.ko 2024-03-31T14:49:44.018 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 15561 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/lrw.ko 2024-03-31T14:49:44.018 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 11417 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/lz4.ko 2024-03-31T14:49:44.018 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 11433 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/lz4hc.ko 2024-03-31T14:49:44.018 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 11601 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/md4.ko 2024-03-31T14:49:44.018 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 9625 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/michael_mic.ko 2024-03-31T14:49:44.018 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 10577 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/pcbc.ko 2024-03-31T14:49:44.018 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 18721 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/pcrypt.ko 2024-03-31T14:49:44.018 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 15953 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/rmd160.ko 2024-03-31T14:49:44.018 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 29297 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/seed.ko 2024-03-31T14:49:44.018 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 26985 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/serpent_generic.ko 2024-03-31T14:49:44.018 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 143025 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/tcrypt.ko 2024-03-31T14:49:44.019 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 10785 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/tea.ko 2024-03-31T14:49:44.019 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 28673 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/twofish_common.ko 2024-03-31T14:49:44.019 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 13065 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/twofish_generic.ko 2024-03-31T14:49:44.019 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 17305 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/vmac.ko 2024-03-31T14:49:44.019 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 36089 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/wp512.ko 2024-03-31T14:49:44.019 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 13945 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/xcbc.ko 2024-03-31T14:49:44.019 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 26569 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/xor.ko 2024-03-31T14:49:44.019 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 18929 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/xts.ko 2024-03-31T14:49:44.019 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 9377 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/xxhash_generic.ko 2024-03-31T14:49:44.019 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ 2024-03-31T14:49:44.019 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/acpi/ 2024-03-31T14:49:44.019 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 18465 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/acpi/acpi_extlog.ko 2024-03-31T14:49:44.019 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 26881 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/acpi/acpi_ipmi.ko 2024-03-31T14:49:44.019 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 28145 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/acpi/acpi_pad.ko 2024-03-31T14:49:44.020 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/acpi/apei/ 2024-03-31T14:49:44.020 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 31961 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/acpi/apei/einj.ko 2024-03-31T14:49:44.020 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 12705 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/acpi/ec_sys.ko 2024-03-31T14:49:44.020 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 14833 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/acpi/platform_profile.ko 2024-03-31T14:49:44.020 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 24305 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/acpi/sbs.ko 2024-03-31T14:49:44.020 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 19425 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/acpi/sbshc.ko 2024-03-31T14:49:44.020 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 104417 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/acpi/video.ko 2024-03-31T14:49:44.020 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/ 2024-03-31T14:49:44.020 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 21497 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/acard-ahci.ko 2024-03-31T14:49:44.020 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 89929 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/ahci.ko 2024-03-31T14:49:44.020 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 14625 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/ahci_platform.ko 2024-03-31T14:49:44.020 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 88305 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/libahci.ko 2024-03-31T14:49:44.020 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 34881 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/libahci_platform.ko 2024-03-31T14:49:44.021 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 17033 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_acpi.ko 2024-03-31T14:49:44.021 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 30337 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_ali.ko 2024-03-31T14:49:44.021 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 33409 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_amd.ko 2024-03-31T14:49:44.021 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 21161 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_artop.ko 2024-03-31T14:49:44.021 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 21017 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_atiixp.ko 2024-03-31T14:49:44.021 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 24841 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_atp867x.ko 2024-03-31T14:49:44.021 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 17553 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_cmd640.ko 2024-03-31T14:49:44.021 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 25081 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_cmd64x.ko 2024-03-31T14:49:44.021 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 14945 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_cypress.ko 2024-03-31T14:49:44.021 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 17745 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_efar.ko 2024-03-31T14:49:44.021 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 21841 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_hpt366.ko 2024-03-31T14:49:44.021 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 33873 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_hpt37x.ko 2024-03-31T14:49:44.021 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 23025 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_hpt3x2n.ko 2024-03-31T14:49:44.021 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 16817 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_hpt3x3.ko 2024-03-31T14:49:44.022 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 16889 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_it8213.ko 2024-03-31T14:49:44.022 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 35169 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_it821x.ko 2024-03-31T14:49:44.022 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 12785 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_jmicron.ko 2024-03-31T14:49:44.022 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 52169 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_legacy.ko 2024-03-31T14:49:44.022 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 15081 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_marvell.ko 2024-03-31T14:49:44.022 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 16177 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_mpiix.ko 2024-03-31T14:49:44.022 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 12729 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_netcell.ko 2024-03-31T14:49:44.022 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 16713 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_ninja32.ko 2024-03-31T14:49:44.022 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 15001 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_ns87410.ko 2024-03-31T14:49:44.022 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 17257 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_ns87415.ko 2024-03-31T14:49:44.022 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 16465 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_oldpiix.ko 2024-03-31T14:49:44.022 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 15273 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_opti.ko 2024-03-31T14:49:44.022 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 20577 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_optidma.ko 2024-03-31T14:49:44.022 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 32481 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_pcmcia.ko 2024-03-31T14:49:44.023 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 37689 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_pdc2027x.ko 2024-03-31T14:49:44.023 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 20449 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_pdc202xx_old.ko 2024-03-31T14:49:44.023 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 14121 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_piccolo.ko 2024-03-31T14:49:44.023 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 15345 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_radisys.ko 2024-03-31T14:49:44.023 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 19425 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_rdc.ko 2024-03-31T14:49:44.023 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 15225 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_rz1000.ko 2024-03-31T14:49:44.023 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 13777 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_sch.ko 2024-03-31T14:49:44.023 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 22993 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_serverworks.ko 2024-03-31T14:49:44.023 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 23353 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_sil680.ko 2024-03-31T14:49:44.023 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 19593 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_sl82c105.ko 2024-03-31T14:49:44.023 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 15905 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_triflex.ko 2024-03-31T14:49:44.023 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 30337 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_via.ko 2024-03-31T14:49:44.023 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 22697 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pdc_adma.ko 2024-03-31T14:49:44.023 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 29017 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_inic162x.ko 2024-03-31T14:49:44.023 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 83249 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_mv.ko 2024-03-31T14:49:44.024 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 70297 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_nv.ko 2024-03-31T14:49:44.024 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 37273 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_promise.ko 2024-03-31T14:49:44.024 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 22937 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_qstor.ko 2024-03-31T14:49:44.024 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 29177 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_sil.ko 2024-03-31T14:49:44.024 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 38169 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_sil24.ko 2024-03-31T14:49:44.024 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 17849 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_sis.ko 2024-03-31T14:49:44.024 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 20345 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_svw.ko 2024-03-31T14:49:44.024 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 44697 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_sx4.ko 2024-03-31T14:49:44.024 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 15441 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_uli.ko 2024-03-31T14:49:44.024 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 34969 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_via.ko 2024-03-31T14:49:44.024 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 19969 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_vsc.ko 2024-03-31T14:49:44.024 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/auxdisplay/ 2024-03-31T14:49:44.024 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 16193 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/auxdisplay/ks0108.ko 2024-03-31T14:49:44.025 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/bcma/ 2024-03-31T14:49:44.025 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 124385 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/bcma/bcma.ko 2024-03-31T14:49:44.025 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/ 2024-03-31T14:49:44.025 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/aoe/ 2024-03-31T14:49:44.025 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 113801 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/aoe/aoe.ko 2024-03-31T14:49:44.025 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/drbd/ 2024-03-31T14:49:44.025 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 964745 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/drbd/drbd.ko 2024-03-31T14:49:44.025 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 169697 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/floppy.ko 2024-03-31T14:49:44.025 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/mtip32xx/ 2024-03-31T14:49:44.025 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 94209 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/mtip32xx/mtip32xx.ko 2024-03-31T14:49:44.025 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 116329 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/nbd.ko 2024-03-31T14:49:44.025 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/null_blk/ 2024-03-31T14:49:44.025 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 110321 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/null_blk/null_blk.ko 2024-03-31T14:49:44.025 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 98185 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/pktcdvd.ko 2024-03-31T14:49:44.026 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 263633 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/rbd.ko 2024-03-31T14:49:44.026 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/xen-blkback/ 2024-03-31T14:49:44.026 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 98105 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/xen-blkback/xen-blkback.ko 2024-03-31T14:49:44.026 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/zram/ 2024-03-31T14:49:44.026 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 58761 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/zram/zram.ko 2024-03-31T14:49:44.026 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/ 2024-03-31T14:49:44.026 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 30609 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/applicom.ko 2024-03-31T14:49:44.026 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 11905 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/hangcheck-timer.ko 2024-03-31T14:49:44.026 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/hw_random/ 2024-03-31T14:49:44.026 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 13793 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/hw_random/amd-rng.ko 2024-03-31T14:49:44.026 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 18705 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/hw_random/intel-rng.ko 2024-03-31T14:49:44.026 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 13065 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/hw_random/timeriomem-rng.ko 2024-03-31T14:49:44.026 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 11065 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/hw_random/via-rng.ko 2024-03-31T14:49:44.026 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 15529 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/hw_random/virtio-rng.ko 2024-03-31T14:49:44.027 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/ipmi/ 2024-03-31T14:49:44.027 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 30609 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/ipmi/ipmi_devintf.ko 2024-03-31T14:49:44.027 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 148513 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/ipmi/ipmi_msghandler.ko 2024-03-31T14:49:44.027 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 26401 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/ipmi/ipmi_poweroff.ko 2024-03-31T14:49:44.027 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 139305 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/ipmi/ipmi_si.ko 2024-03-31T14:49:44.027 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 70865 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/ipmi/ipmi_ssif.ko 2024-03-31T14:49:44.027 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 53409 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/ipmi/ipmi_watchdog.ko 2024-03-31T14:49:44.027 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 36225 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/lp.ko 2024-03-31T14:49:44.027 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/mwave/ 2024-03-31T14:49:44.027 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 84001 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/mwave/mwave.ko 2024-03-31T14:49:44.027 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 25777 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/nvram.ko 2024-03-31T14:49:44.027 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 32073 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/ppdev.ko 2024-03-31T14:49:44.027 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 51017 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/tlclk.ko 2024-03-31T14:49:44.027 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/tpm/ 2024-03-31T14:49:44.027 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 16297 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/tpm/tpm_atmel.ko 2024-03-31T14:49:44.028 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 15777 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/tpm/tpm_i2c_atmel.ko 2024-03-31T14:49:44.028 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 23369 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/tpm/tpm_i2c_infineon.ko 2024-03-31T14:49:44.028 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 32761 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/tpm/tpm_i2c_nuvoton.ko 2024-03-31T14:49:44.028 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 30017 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/tpm/tpm_infineon.ko 2024-03-31T14:49:44.028 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 22601 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/tpm/tpm_nsc.ko 2024-03-31T14:49:44.028 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 23457 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/tpm/xen-tpmfront.ko 2024-03-31T14:49:44.028 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/xillybus/ 2024-03-31T14:49:44.028 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 16897 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/xillybus/xillybus_class.ko 2024-03-31T14:49:44.028 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 50841 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/xillybus/xillybus_core.ko 2024-03-31T14:49:44.028 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 12225 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/xillybus/xillybus_pcie.ko 2024-03-31T14:49:44.028 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/clk/ 2024-03-31T14:49:44.028 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 12385 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/clk/clk-palmas.ko 2024-03-31T14:49:44.028 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 47721 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/clk/clk-si5351.ko 2024-03-31T14:49:44.028 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 11353 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/clk/clk-twl6040.ko 2024-03-31T14:49:44.029 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 19145 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/clk/clk-wm831x.ko 2024-03-31T14:49:44.029 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/cpufreq/ 2024-03-31T14:49:44.029 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 10369 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/cpufreq/amd_freq_sensitivity.ko 2024-03-31T14:49:44.029 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 16729 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/cpufreq/p4-clockmod.ko 2024-03-31T14:49:44.029 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 25785 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/cpufreq/speedstep-lib.ko 2024-03-31T14:49:44.029 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/ 2024-03-31T14:49:44.029 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/ccp/ 2024-03-31T14:49:44.029 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 70425 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/ccp/ccp-crypto.ko 2024-03-31T14:49:44.029 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 246777 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/ccp/ccp.ko 2024-03-31T14:49:44.029 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/intel/ 2024-03-31T14:49:44.029 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/intel/qat/ 2024-03-31T14:49:44.029 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/intel/qat/qat_common/ 2024-03-31T14:49:44.029 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 570217 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/intel/qat/qat_common/intel_qat.ko 2024-03-31T14:49:44.029 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/intel/qat/qat_dh895xcc/ 2024-03-31T14:49:44.030 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 24033 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/intel/qat/qat_dh895xcc/qat_dh895xcc.ko 2024-03-31T14:49:44.030 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 19897 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/padlock-aes.ko 2024-03-31T14:49:44.030 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 21737 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/padlock-sha.ko 2024-03-31T14:49:44.030 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/virtio/ 2024-03-31T14:49:44.030 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 70409 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/virtio/virtio_crypto.ko 2024-03-31T14:49:44.030 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/dca/ 2024-03-31T14:49:44.030 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 29689 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/dca/dca.ko 2024-03-31T14:49:44.030 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/dma/ 2024-03-31T14:49:44.030 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/dma/dw/ 2024-03-31T14:49:44.030 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 15761 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/dma/dw/dw_dmac.ko 2024-03-31T14:49:44.030 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/dma/ioat/ 2024-03-31T14:49:44.030 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 122793 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/dma/ioat/ioatdma.ko 2024-03-31T14:49:44.030 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/ 2024-03-31T14:49:44.031 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 80297 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/amd64_edac.ko 2024-03-31T14:49:44.031 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 35665 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/e752x_edac.ko 2024-03-31T14:49:44.031 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 41633 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/edac_mce_amd.ko 2024-03-31T14:49:44.031 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 18001 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/i3000_edac.ko 2024-03-31T14:49:44.031 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 17961 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/i3200_edac.ko 2024-03-31T14:49:44.031 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 22433 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/i5100_edac.ko 2024-03-31T14:49:44.031 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 25489 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/i5400_edac.ko 2024-03-31T14:49:44.031 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 24809 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/i7300_edac.ko 2024-03-31T14:49:44.031 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 51937 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/i7core_edac.ko 2024-03-31T14:49:44.031 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 18265 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/i82975x_edac.ko 2024-03-31T14:49:44.031 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 19921 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/ie31200_edac.ko 2024-03-31T14:49:44.031 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 56729 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/sb_edac.ko 2024-03-31T14:49:44.031 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 17433 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/x38_edac.ko 2024-03-31T14:49:44.031 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/extcon/ 2024-03-31T14:49:44.032 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 12569 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/extcon/extcon-gpio.ko 2024-03-31T14:49:44.032 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 27081 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/extcon/extcon-max14577.ko 2024-03-31T14:49:44.032 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 30209 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/extcon/extcon-max77693.ko 2024-03-31T14:49:44.032 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 24641 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/extcon/extcon-max8997.ko 2024-03-31T14:49:44.032 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 26433 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/extcon/extcon-palmas.ko 2024-03-31T14:49:44.032 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 23609 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/extcon/extcon-rt8973a.ko 2024-03-31T14:49:44.032 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 27681 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/extcon/extcon-sm5502.ko 2024-03-31T14:49:44.032 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/firmware/ 2024-03-31T14:49:44.032 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 34873 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/firmware/dmi-sysfs.ko 2024-03-31T14:49:44.032 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/firmware/efi/ 2024-03-31T14:49:44.033 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 15417 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/firmware/efi/efi-pstore.ko 2024-03-31T14:49:44.033 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 20625 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/firmware/iscsi_ibft.ko 2024-03-31T14:49:44.033 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/gpio/ 2024-03-31T14:49:44.033 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 12209 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/gpio/gpio-adp5520.ko 2024-03-31T14:49:44.033 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 20049 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/gpio/gpio-amd8111.ko 2024-03-31T14:49:44.033 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 14889 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/gpio/gpio-arizona.ko 2024-03-31T14:49:44.033 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 13737 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/gpio/gpio-da9052.ko 2024-03-31T14:49:44.033 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 11825 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/gpio/gpio-da9055.ko 2024-03-31T14:49:44.033 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 21961 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/gpio/gpio-dln2.ko 2024-03-31T14:49:44.033 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 77745 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/gpio/gpio-f7188x.ko 2024-03-31T14:49:44.033 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 31297 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/gpio/gpio-generic.ko 2024-03-31T14:49:44.033 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 19473 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/gpio/gpio-ich.ko 2024-03-31T14:49:44.033 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 11809 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/gpio/gpio-janz-ttl.ko 2024-03-31T14:49:44.034 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 14009 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/gpio/gpio-kempld.ko 2024-03-31T14:49:44.034 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 12441 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/gpio/gpio-lp3943.ko 2024-03-31T14:49:44.034 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 9409 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/gpio/gpio-max7300.ko 2024-03-31T14:49:44.034 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 11801 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/gpio/gpio-max730x.ko 2024-03-31T14:49:44.034 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 21969 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/gpio/gpio-max732x.ko 2024-03-31T14:49:44.034 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 25041 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/gpio/gpio-ml-ioh.ko 2024-03-31T14:49:44.034 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 47441 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/gpio/gpio-pca953x.ko 2024-03-31T14:49:44.034 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 26721 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/gpio/gpio-pcf857x.ko 2024-03-31T14:49:44.034 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 14273 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/gpio/gpio-rdc321x.ko 2024-03-31T14:49:44.034 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 20497 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/gpio/gpio-sch.ko 2024-03-31T14:49:44.034 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 19401 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/gpio/gpio-sch311x.ko 2024-03-31T14:49:44.034 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 12049 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/gpio/gpio-tps65912.ko 2024-03-31T14:49:44.034 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 20961 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/gpio/gpio-twl4030.ko 2024-03-31T14:49:44.035 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 11321 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/gpio/gpio-twl6040.ko 2024-03-31T14:49:44.035 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 20497 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/gpio/gpio-viperboard.ko 2024-03-31T14:49:44.035 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 15273 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/gpio/gpio-vx855.ko 2024-03-31T14:49:44.035 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 16481 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/gpio/gpio-wm831x.ko 2024-03-31T14:49:44.035 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 11521 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/gpio/gpio-wm8350.ko 2024-03-31T14:49:44.035 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 16945 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/gpio/gpio-wm8994.ko 2024-03-31T14:49:44.035 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/ 2024-03-31T14:49:44.035 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 11473 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-a4tech.ko 2024-03-31T14:49:44.035 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 39409 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-apple.ko 2024-03-31T14:49:44.035 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 14385 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-appleir.ko 2024-03-31T14:49:44.035 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 8249 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-aureal.ko 2024-03-31T14:49:44.035 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 15073 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-axff.ko 2024-03-31T14:49:44.035 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 10945 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-belkin.ko 2024-03-31T14:49:44.035 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 10209 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-cherry.ko 2024-03-31T14:49:44.035 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 15937 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-chicony.ko 2024-03-31T14:49:44.036 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 60657 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-cp2112.ko 2024-03-31T14:49:44.036 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 10945 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-cypress.ko 2024-03-31T14:49:44.036 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 16289 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-dr.ko 2024-03-31T14:49:44.036 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 10593 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-elecom.ko 2024-03-31T14:49:44.036 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 17569 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-elo.ko 2024-03-31T14:49:44.036 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 13353 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-emsff.ko 2024-03-31T14:49:44.036 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 10001 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-ezkey.ko 2024-03-31T14:49:44.036 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 14209 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-gaff.ko 2024-03-31T14:49:44.036 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 9129 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-generic.ko 2024-03-31T14:49:44.036 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 17025 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-gt683r.ko 2024-03-31T14:49:44.036 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 12577 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-gyration.ko 2024-03-31T14:49:44.036 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 10065 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-holtek-kbd.ko 2024-03-31T14:49:44.036 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 10361 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-holtek-mouse.ko 2024-03-31T14:49:44.036 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 14345 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-holtekff.ko 2024-03-31T14:49:44.037 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 19001 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-hyperv.ko 2024-03-31T14:49:44.037 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 10385 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-icade.ko 2024-03-31T14:49:44.037 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 9137 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-kensington.ko 2024-03-31T14:49:44.037 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 8441 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-keytouch.ko 2024-03-31T14:49:44.037 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 17105 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-kye.ko 2024-03-31T14:49:44.037 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 10705 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-lcpower.ko 2024-03-31T14:49:44.037 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 20401 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-led.ko 2024-03-31T14:49:44.037 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 42841 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-lenovo.ko 2024-03-31T14:49:44.037 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 30593 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-lg-g15.ko 2024-03-31T14:49:44.037 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 57969 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-logitech-dj.ko 2024-03-31T14:49:44.037 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 101537 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-logitech-hidpp.ko 2024-03-31T14:49:44.037 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 80897 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-logitech.ko 2024-03-31T14:49:44.037 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 30697 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-magicmouse.ko 2024-03-31T14:49:44.037 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 21865 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-microsoft.ko 2024-03-31T14:49:44.038 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 10257 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-monterey.ko 2024-03-31T14:49:44.038 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 50273 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-multitouch.ko 2024-03-31T14:49:44.038 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 32489 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-ntrig.ko 2024-03-31T14:49:44.038 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 8753 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-ortek.ko 2024-03-31T14:49:44.038 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 9017 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-penmount.ko 2024-03-31T14:49:44.038 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 12185 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-petalynx.ko 2024-03-31T14:49:44.038 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 72841 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-picolcd.ko 2024-03-31T14:49:44.038 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 14129 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-pl.ko 2024-03-31T14:49:44.038 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 15001 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-plantronics.ko 2024-03-31T14:49:44.038 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 8321 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-primax.ko 2024-03-31T14:49:44.038 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 35521 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-rmi.ko 2024-03-31T14:49:44.038 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 20729 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-arvo.ko 2024-03-31T14:49:44.038 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 13953 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-common.ko 2024-03-31T14:49:44.038 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 31321 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-isku.ko 2024-03-31T14:49:44.038 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 29625 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-kone.ko 2024-03-31T14:49:44.039 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 31017 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-koneplus.ko 2024-03-31T14:49:44.039 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 21025 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-konepure.ko 2024-03-31T14:49:44.039 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 29377 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-kovaplus.ko 2024-03-31T14:49:44.039 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 13201 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-lua.ko 2024-03-31T14:49:44.039 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 29209 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-pyra.ko 2024-03-31T14:49:44.039 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 27833 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-ryos.ko 2024-03-31T14:49:44.039 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 19417 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-savu.ko 2024-03-31T14:49:44.039 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 23233 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat.ko 2024-03-31T14:49:44.039 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 13577 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-saitek.ko 2024-03-31T14:49:44.039 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 15089 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-samsung.ko 2024-03-31T14:49:44.039 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 36713 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-sensor-hub.ko 2024-03-31T14:49:44.039 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 13505 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-sjoy.ko 2024-03-31T14:49:44.039 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 51153 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-sony.ko 2024-03-31T14:49:44.039 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 8217 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-speedlink.ko 2024-03-31T14:49:44.040 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 25193 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-steelseries.ko 2024-03-31T14:49:44.040 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 9713 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-sunplus.ko 2024-03-31T14:49:44.040 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 16889 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-thrustmaster.ko 2024-03-31T14:49:44.040 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 10441 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-tivo.ko 2024-03-31T14:49:44.040 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 14993 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-tmff.ko 2024-03-31T14:49:44.040 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 14281 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-topseed.ko 2024-03-31T14:49:44.040 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 16369 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-twinhan.ko 2024-03-31T14:49:44.040 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 70977 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-uclogic.ko 2024-03-31T14:49:44.040 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 10825 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-waltop.ko 2024-03-31T14:49:44.040 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 143377 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-wiimote.ko 2024-03-31T14:49:44.040 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 8057 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-xinmo.ko 2024-03-31T14:49:44.040 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 13225 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-zpff.ko 2024-03-31T14:49:44.040 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 16561 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-zydacron.ko 2024-03-31T14:49:44.040 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 269209 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid.ko 2024-03-31T14:49:44.040 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 32225 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/uhid.ko 2024-03-31T14:49:44.041 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/usbhid/ 2024-03-31T14:49:44.041 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 125625 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/usbhid/usbhid.ko 2024-03-31T14:49:44.041 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 19041 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/usbhid/usbkbd.ko 2024-03-31T14:49:44.041 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 14857 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/usbhid/usbmouse.ko 2024-03-31T14:49:44.041 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 218865 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/wacom.ko 2024-03-31T14:49:44.041 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hv/ 2024-03-31T14:49:44.041 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 64297 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hv/hv_balloon.ko 2024-03-31T14:49:44.041 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 90089 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hv/hv_utils.ko 2024-03-31T14:49:44.041 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 304913 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hv/hv_vmbus.ko 2024-03-31T14:49:44.041 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ 2024-03-31T14:49:44.041 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 66793 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/abituguru.ko 2024-03-31T14:49:44.041 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 61809 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/abituguru3.ko 2024-03-31T14:49:44.041 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 34569 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/acpi_power_meter.ko 2024-03-31T14:49:44.041 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 17409 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ad7414.ko 2024-03-31T14:49:44.042 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 19217 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ad7418.ko 2024-03-31T14:49:44.042 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 25153 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adc128d818.ko 2024-03-31T14:49:44.042 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 31633 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adm1025.ko 2024-03-31T14:49:44.042 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 97881 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adm1026.ko 2024-03-31T14:49:44.042 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 19649 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adm1029.ko 2024-03-31T14:49:44.042 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 45457 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adm1031.ko 2024-03-31T14:49:44.042 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 26193 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adm9240.ko 2024-03-31T14:49:44.042 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 14673 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ads7828.ko 2024-03-31T14:49:44.042 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 10673 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adt7410.ko 2024-03-31T14:49:44.042 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 21953 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adt7411.ko 2024-03-31T14:49:44.042 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 74313 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adt7462.ko 2024-03-31T14:49:44.042 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 39489 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adt7470.ko 2024-03-31T14:49:44.042 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 69305 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adt7475.ko 2024-03-31T14:49:44.042 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 19537 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adt7x10.ko 2024-03-31T14:49:44.043 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 38977 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/amc6821.ko 2024-03-31T14:49:44.043 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 55105 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/applesmc.ko 2024-03-31T14:49:44.043 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 49817 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/asb100.ko 2024-03-31T14:49:44.043 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 58521 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/asc7621.ko 2024-03-31T14:49:44.043 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 40721 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/asus_atk0110.ko 2024-03-31T14:49:44.043 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 17129 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/atxp1.ko 2024-03-31T14:49:44.043 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 28129 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/coretemp.ko 2024-03-31T14:49:44.043 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 24489 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/da9052-hwmon.ko 2024-03-31T14:49:44.043 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 17161 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/da9055-hwmon.ko 2024-03-31T14:49:44.043 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 49233 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/dell-smm-hwmon.ko 2024-03-31T14:49:44.043 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 99505 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/dme1737.ko 2024-03-31T14:49:44.043 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 19393 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ds1621.ko 2024-03-31T14:49:44.043 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 15225 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ds620.ko 2024-03-31T14:49:44.043 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 29833 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/emc1403.ko 2024-03-31T14:49:44.043 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 33089 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/emc2103.ko 2024-03-31T14:49:44.044 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 30225 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/emc6w201.ko 2024-03-31T14:49:44.044 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 66889 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/f71805f.ko 2024-03-31T14:49:44.044 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 94593 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/f71882fg.ko 2024-03-31T14:49:44.044 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 41305 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/f75375s.ko 2024-03-31T14:49:44.044 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 21673 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/fam15h_power.ko 2024-03-31T14:49:44.044 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 55353 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/fschmd.ko 2024-03-31T14:49:44.044 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 14385 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/g760a.ko 2024-03-31T14:49:44.044 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 23921 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/g762.ko 2024-03-31T14:49:44.044 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 46705 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/gl518sm.ko 2024-03-31T14:49:44.044 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 45433 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/gl520sm.ko 2024-03-31T14:49:44.044 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 13177 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/hih6130.ko 2024-03-31T14:49:44.044 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 10017 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/hwmon-vid.ko 2024-03-31T14:49:44.044 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 11385 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/i5500_temp.ko 2024-03-31T14:49:44.044 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 22505 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/i5k_amb.ko 2024-03-31T14:49:44.045 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 34433 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ibmaem.ko 2024-03-31T14:49:44.045 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 21161 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ibmpex.ko 2024-03-31T14:49:44.045 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 24777 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ina209.ko 2024-03-31T14:49:44.045 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 13305 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ina2xx.ko 2024-03-31T14:49:44.045 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 140129 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/it87.ko 2024-03-31T14:49:44.045 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 18745 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/jc42.ko 2024-03-31T14:49:44.045 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 24553 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/k10temp.ko 2024-03-31T14:49:44.045 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 13785 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/k8temp.ko 2024-03-31T14:49:44.045 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 18825 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lineage-pem.ko 2024-03-31T14:49:44.045 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 48505 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm63.ko 2024-03-31T14:49:44.045 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 16601 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm73.ko 2024-03-31T14:49:44.045 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 38841 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm75.ko 2024-03-31T14:49:44.045 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 19441 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm77.ko 2024-03-31T14:49:44.045 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 34497 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm78.ko 2024-03-31T14:49:44.045 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 31649 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm80.ko 2024-03-31T14:49:44.046 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 16793 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm83.ko 2024-03-31T14:49:44.046 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 72737 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm85.ko 2024-03-31T14:49:44.046 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 46057 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm87.ko 2024-03-31T14:49:44.046 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 37817 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm90.ko 2024-03-31T14:49:44.046 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 19649 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm92.ko 2024-03-31T14:49:44.046 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 108745 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm93.ko 2024-03-31T14:49:44.046 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 33297 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm95234.ko 2024-03-31T14:49:44.046 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 16105 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm95241.ko 2024-03-31T14:49:44.046 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 18017 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm95245.ko 2024-03-31T14:49:44.046 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 23929 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ltc2945.ko 2024-03-31T14:49:44.046 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 13697 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ltc4151.ko 2024-03-31T14:49:44.046 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 15097 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ltc4215.ko 2024-03-31T14:49:44.047 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 14793 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ltc4222.ko 2024-03-31T14:49:44.047 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 14729 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ltc4245.ko 2024-03-31T14:49:44.047 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 13153 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ltc4260.ko 2024-03-31T14:49:44.047 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 14897 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ltc4261.ko 2024-03-31T14:49:44.047 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 33393 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/max16065.ko 2024-03-31T14:49:44.047 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 18337 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/max1619.ko 2024-03-31T14:49:44.047 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 24257 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/max1668.ko 2024-03-31T14:49:44.047 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 19873 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/max197.ko 2024-03-31T14:49:44.047 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 27033 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/max6639.ko 2024-03-31T14:49:44.047 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 23209 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/max6650.ko 2024-03-31T14:49:44.047 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 35681 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/max6697.ko 2024-03-31T14:49:44.047 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 18481 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/mc13783-adc.ko 2024-03-31T14:49:44.047 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 10489 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/mcp3021.ko 2024-03-31T14:49:44.047 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 15833 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/menf21bmc_hwmon.ko 2024-03-31T14:49:44.047 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 47785 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/nct6683.ko 2024-03-31T14:49:44.048 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 119841 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/nct6775-core.ko 2024-03-31T14:49:44.048 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 61433 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/nct6775.ko 2024-03-31T14:49:44.048 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 59641 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/nct7802.ko 2024-03-31T14:49:44.048 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 82713 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pc87360.ko 2024-03-31T14:49:44.048 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 54337 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pc87427.ko 2024-03-31T14:49:44.048 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 18377 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pcf8591.ko 2024-03-31T14:49:44.048 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/ 2024-03-31T14:49:44.048 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 21257 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/adm1275.ko 2024-03-31T14:49:44.048 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 16961 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/lm25066.ko 2024-03-31T14:49:44.048 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 28233 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/ltc2978.ko 2024-03-31T14:49:44.048 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 9489 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/max16064.ko 2024-03-31T14:49:44.048 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 17561 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/max34440.ko 2024-03-31T14:49:44.048 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 10081 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/max8688.ko 2024-03-31T14:49:44.048 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 15409 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/pmbus.ko 2024-03-31T14:49:44.049 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 82473 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/pmbus_core.ko 2024-03-31T14:49:44.049 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 8641 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/tps40422.ko 2024-03-31T14:49:44.049 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 30633 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/ucd9000.ko 2024-03-31T14:49:44.049 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 14977 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/ucd9200.ko 2024-03-31T14:49:44.049 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 20585 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/zl6100.ko 2024-03-31T14:49:44.049 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 12921 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/powr1220.ko 2024-03-31T14:49:44.049 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 21673 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/sch5627.ko 2024-03-31T14:49:44.049 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 28721 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/sch5636.ko 2024-03-31T14:49:44.049 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 29145 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/sch56xx-common.ko 2024-03-31T14:49:44.049 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 32041 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/sht15.ko 2024-03-31T14:49:44.049 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 13409 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/sht21.ko 2024-03-31T14:49:44.049 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 16737 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/shtc1.ko 2024-03-31T14:49:44.049 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 40481 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/sis5595.ko 2024-03-31T14:49:44.049 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 19801 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/smsc47b397.ko 2024-03-31T14:49:44.050 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 40353 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/smsc47m1.ko 2024-03-31T14:49:44.050 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 37281 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/smsc47m192.ko 2024-03-31T14:49:44.050 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 26457 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/thmc50.ko 2024-03-31T14:49:44.050 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 16961 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/tmp102.ko 2024-03-31T14:49:44.050 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 14001 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/tmp103.ko 2024-03-31T14:49:44.050 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 21161 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/tmp401.ko 2024-03-31T14:49:44.050 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 19441 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/tmp421.ko 2024-03-31T14:49:44.050 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 19617 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/via-cputemp.ko 2024-03-31T14:49:44.050 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 43441 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/via686a.ko 2024-03-31T14:49:44.050 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 52361 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/vt1211.ko 2024-03-31T14:49:44.050 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 53201 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/vt8231.ko 2024-03-31T14:49:44.050 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 58257 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/w83627ehf.ko 2024-03-31T14:49:44.050 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 79833 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/w83627hf.ko 2024-03-31T14:49:44.050 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 71249 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/w83781d.ko 2024-03-31T14:49:44.051 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 63129 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/w83791d.ko 2024-03-31T14:49:44.051 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 71225 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/w83792d.ko 2024-03-31T14:49:44.051 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 101377 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/w83793.ko 2024-03-31T14:49:44.051 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 119409 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/w83795.ko 2024-03-31T14:49:44.051 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 17209 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/w83l785ts.ko 2024-03-31T14:49:44.051 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 35521 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/w83l786ng.ko 2024-03-31T14:49:44.051 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 13721 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/wm831x-hwmon.ko 2024-03-31T14:49:44.051 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 10521 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/wm8350-hwmon.ko 2024-03-31T14:49:44.051 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/ 2024-03-31T14:49:44.051 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/algos/ 2024-03-31T14:49:44.051 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 19945 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/algos/i2c-algo-bit.ko 2024-03-31T14:49:44.051 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 19241 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/algos/i2c-algo-pca.ko 2024-03-31T14:49:44.051 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/ 2024-03-31T14:49:44.051 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 30857 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-ali1535.ko 2024-03-31T14:49:44.052 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 32473 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-ali1563.ko 2024-03-31T14:49:44.052 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 30385 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-ali15x3.ko 2024-03-31T14:49:44.052 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 17833 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-amd756-s4882.ko 2024-03-31T14:49:44.052 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 28073 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-amd756.ko 2024-03-31T14:49:44.052 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 16033 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-amd8111.ko 2024-03-31T14:49:44.052 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 15049 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-cbus-gpio.ko 2024-03-31T14:49:44.052 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 6825 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-ccgx-ucsi.ko 2024-03-31T14:49:44.052 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 13849 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-cros-ec-tunnel.ko 2024-03-31T14:49:44.052 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 50289 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-designware-core.ko 2024-03-31T14:49:44.052 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 23793 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-designware-pci.ko 2024-03-31T14:49:44.052 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 19889 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-designware-platform.ko 2024-03-31T14:49:44.052 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 17233 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-diolan-u2c.ko 2024-03-31T14:49:44.052 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 12401 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-dln2.ko 2024-03-31T14:49:44.052 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 17489 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-gpio.ko 2024-03-31T14:49:44.053 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 67681 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-i801.ko 2024-03-31T14:49:44.053 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 22337 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-isch.ko 2024-03-31T14:49:44.053 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 47793 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-ismt.ko 2024-03-31T14:49:44.053 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 17617 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-kempld.ko 2024-03-31T14:49:44.053 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 17769 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-nforce2-s4985.ko 2024-03-31T14:49:44.053 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 23537 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-nforce2.ko 2024-03-31T14:49:44.053 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 26161 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-ocores.ko 2024-03-31T14:49:44.053 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 25041 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-parport.ko 2024-03-31T14:49:44.053 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 16393 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-pca-platform.ko 2024-03-31T14:49:44.053 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 50969 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-piix4.ko 2024-03-31T14:49:44.053 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 15177 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-robotfuzz-osif.ko 2024-03-31T14:49:44.053 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 18553 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-scmi.ko 2024-03-31T14:49:44.053 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 11713 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-simtec.ko 2024-03-31T14:49:44.053 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 22209 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-sis5595.ko 2024-03-31T14:49:44.053 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 29257 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-sis630.ko 2024-03-31T14:49:44.054 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 20225 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-sis96x.ko 2024-03-31T14:49:44.054 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 19137 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-taos-evm.ko 2024-03-31T14:49:44.054 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 19705 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-tiny-usb.ko 2024-03-31T14:49:44.054 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 13705 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-via.ko 2024-03-31T14:49:44.054 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 27617 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-viapro.ko 2024-03-31T14:49:44.054 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 18937 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-viperboard.ko 2024-03-31T14:49:44.054 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 47561 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-xiic.ko 2024-03-31T14:49:44.054 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 22353 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/i2c-mux.ko 2024-03-31T14:49:44.054 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 18721 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/i2c-smbus.ko 2024-03-31T14:49:44.054 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 26977 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/i2c-stub.ko 2024-03-31T14:49:44.054 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/muxes/ 2024-03-31T14:49:44.054 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 15089 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/muxes/i2c-mux-gpio.ko 2024-03-31T14:49:44.054 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 13353 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/muxes/i2c-mux-pca9541.ko 2024-03-31T14:49:44.054 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 29633 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/muxes/i2c-mux-pca954x.ko 2024-03-31T14:49:44.055 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ 2024-03-31T14:49:44.055 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/core/ 2024-03-31T14:49:44.055 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 273513 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/core/ib_cm.ko 2024-03-31T14:49:44.055 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 883777 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/core/ib_core.ko 2024-03-31T14:49:44.055 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 69177 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/core/ib_umad.ko 2024-03-31T14:49:44.055 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 324233 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/core/ib_uverbs.ko 2024-03-31T14:49:44.055 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 97825 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/core/iw_cm.ko 2024-03-31T14:49:44.055 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 282329 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/core/rdma_cm.ko 2024-03-31T14:49:44.055 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 66049 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/core/rdma_ucm.ko 2024-03-31T14:49:44.055 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/hw/ 2024-03-31T14:49:44.055 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/hw/cxgb4/ 2024-03-31T14:49:44.055 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 493129 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/hw/cxgb4/iw_cxgb4.ko 2024-03-31T14:49:44.055 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/hw/mlx4/ 2024-03-31T14:49:44.055 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 420529 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/hw/mlx4/mlx4_ib.ko 2024-03-31T14:49:44.056 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/hw/mlx5/ 2024-03-31T14:49:44.056 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 741609 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko 2024-03-31T14:49:44.056 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/hw/mthca/ 2024-03-31T14:49:44.056 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 296785 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/hw/mthca/ib_mthca.ko 2024-03-31T14:49:44.056 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/hw/ocrdma/ 2024-03-31T14:49:44.056 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 153705 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/hw/ocrdma/ocrdma.ko 2024-03-31T14:49:44.056 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/hw/usnic/ 2024-03-31T14:49:44.056 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 169089 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/hw/usnic/usnic_verbs.ko 2024-03-31T14:49:44.056 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp/ 2024-03-31T14:49:44.056 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp/ipoib/ 2024-03-31T14:49:44.056 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 266529 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp/ipoib/ib_ipoib.ko 2024-03-31T14:49:44.056 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp/iser/ 2024-03-31T14:49:44.056 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 100257 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp/iser/ib_iser.ko 2024-03-31T14:49:44.056 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp/isert/ 2024-03-31T14:49:44.057 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 111329 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp/isert/ib_isert.ko 2024-03-31T14:49:44.057 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp/srp/ 2024-03-31T14:49:44.057 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 128657 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp/srp/ib_srp.ko 2024-03-31T14:49:44.057 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp/srpt/ 2024-03-31T14:49:44.057 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 129793 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp/srpt/ib_srpt.ko 2024-03-31T14:49:44.057 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/ 2024-03-31T14:49:44.057 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 10649 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/evbug.ko 2024-03-31T14:49:44.057 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 24025 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/ff-memless.ko 2024-03-31T14:49:44.057 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/gameport/ 2024-03-31T14:49:44.057 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 11777 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/gameport/emu10k1-gp.ko 2024-03-31T14:49:44.057 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 11985 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/gameport/fm801-gp.ko 2024-03-31T14:49:44.057 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 41449 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/gameport/gameport.ko 2024-03-31T14:49:44.057 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 41705 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/joydev.ko 2024-03-31T14:49:44.057 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/keyboard/ 2024-03-31T14:49:44.058 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 13209 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/keyboard/adp5520-keys.ko 2024-03-31T14:49:44.058 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 12601 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/keyboard/adp5588-keys.ko 2024-03-31T14:49:44.058 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 26737 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/keyboard/adp5589-keys.ko 2024-03-31T14:49:44.058 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 24473 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/keyboard/cros_ec_keyb.ko 2024-03-31T14:49:44.058 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 37273 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/keyboard/gpio_keys.ko 2024-03-31T14:49:44.058 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 18777 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/keyboard/gpio_keys_polled.ko 2024-03-31T14:49:44.058 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 21161 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/keyboard/lkkbd.ko 2024-03-31T14:49:44.058 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 26377 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/keyboard/lm8323.ko 2024-03-31T14:49:44.058 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 14313 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/keyboard/lm8333.ko 2024-03-31T14:49:44.058 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 24225 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/keyboard/matrix_keypad.ko 2024-03-31T14:49:44.058 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 18113 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/keyboard/max7359_keypad.ko 2024-03-31T14:49:44.058 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 15825 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/keyboard/mcs_touchkey.ko 2024-03-31T14:49:44.058 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 16385 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/keyboard/mpr121_touchkey.ko 2024-03-31T14:49:44.059 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 11489 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/keyboard/newtonkbd.ko 2024-03-31T14:49:44.059 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 10569 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/keyboard/opencores-kbd.ko 2024-03-31T14:49:44.059 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 14065 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/keyboard/qt1070.ko 2024-03-31T14:49:44.059 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 17833 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/keyboard/qt2160.ko 2024-03-31T14:49:44.059 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 23713 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/keyboard/samsung-keypad.ko 2024-03-31T14:49:44.059 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 11393 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/keyboard/stowaway.ko 2024-03-31T14:49:44.059 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 18193 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/keyboard/sunkbd.ko 2024-03-31T14:49:44.059 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 17321 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/keyboard/tca6416-keypad.ko 2024-03-31T14:49:44.059 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 16073 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/keyboard/tca8418_keypad.ko 2024-03-31T14:49:44.059 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 16697 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/keyboard/twl4030_keypad.ko 2024-03-31T14:49:44.059 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 11657 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/keyboard/xtkbd.ko 2024-03-31T14:49:44.059 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 11361 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/matrix-keymap.ko 2024-03-31T14:49:44.059 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/mouse/ 2024-03-31T14:49:44.059 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 34225 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/mouse/appletouch.ko 2024-03-31T14:49:44.059 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 34081 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/mouse/bcm5974.ko 2024-03-31T14:49:44.060 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 123177 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/mouse/cyapatp.ko 2024-03-31T14:49:44.060 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 89377 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/mouse/elan_i2c.ko 2024-03-31T14:49:44.060 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 13857 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/mouse/gpio_mouse.ko 2024-03-31T14:49:44.060 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 335857 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/mouse/psmouse.ko 2024-03-31T14:49:44.060 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 14585 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/mouse/sermouse.ko 2024-03-31T14:49:44.060 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 23777 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/mouse/synaptics_i2c.ko 2024-03-31T14:49:44.060 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 22465 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/mouse/synaptics_usb.ko 2024-03-31T14:49:44.060 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 15569 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/mouse/vsxxxaa.ko 2024-03-31T14:49:44.060 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/rmi4/ 2024-03-31T14:49:44.060 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 109553 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/rmi4/rmi_core.ko 2024-03-31T14:49:44.060 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/serio/ 2024-03-31T14:49:44.060 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 11569 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/serio/altera_ps2.ko 2024-03-31T14:49:44.061 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 16193 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/serio/arc_ps2.ko 2024-03-31T14:49:44.061 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 17681 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/serio/ct82c710.ko 2024-03-31T14:49:44.061 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 16849 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/serio/hyperv-keyboard.ko 2024-03-31T14:49:44.061 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 15737 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/serio/parkbd.ko 2024-03-31T14:49:44.061 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 13129 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/serio/pcips2.ko 2024-03-31T14:49:44.061 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 20065 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/serio/ps2mult.ko 2024-03-31T14:49:44.061 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 24049 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/serio/serio_raw.ko 2024-03-31T14:49:44.061 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 17881 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/serio/serport.ko 2024-03-31T14:49:44.061 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 13233 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/sparse-keymap.ko 2024-03-31T14:49:44.061 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/ 2024-03-31T14:49:44.061 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 13713 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/leds-88pm860x.ko 2024-03-31T14:49:44.061 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 11281 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/leds-adp5520.ko 2024-03-31T14:49:44.061 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 59649 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/leds-bd2802.ko 2024-03-31T14:49:44.061 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 24681 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/leds-blinkm.ko 2024-03-31T14:49:44.062 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 10369 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/leds-da903x.ko 2024-03-31T14:49:44.062 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 12137 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/leds-da9052.ko 2024-03-31T14:49:44.062 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 16681 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/leds-gpio.ko 2024-03-31T14:49:44.062 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 15689 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/leds-lm3530.ko 2024-03-31T14:49:44.062 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 30401 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/leds-lm3533.ko 2024-03-31T14:49:44.062 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 19249 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/leds-lm355x.ko 2024-03-31T14:49:44.062 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 19857 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/leds-lm3642.ko 2024-03-31T14:49:44.062 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 16137 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/leds-lp3944.ko 2024-03-31T14:49:44.062 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 11137 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/leds-lp8788.ko 2024-03-31T14:49:44.062 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 14825 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/leds-max8997.ko 2024-03-31T14:49:44.062 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 12441 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/leds-mc13783.ko 2024-03-31T14:49:44.062 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 13281 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/leds-menf21bmc.ko 2024-03-31T14:49:44.062 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 20625 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/leds-pca9532.ko 2024-03-31T14:49:44.062 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 20393 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/leds-pca955x.ko 2024-03-31T14:49:44.063 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 17753 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/leds-pca963x.ko 2024-03-31T14:49:44.063 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 34329 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/leds-ss4200.ko 2024-03-31T14:49:44.063 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 23897 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/leds-tca6507.ko 2024-03-31T14:49:44.063 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 15553 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/leds-wm831x-status.ko 2024-03-31T14:49:44.063 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 11745 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/leds-wm8350.ko 2024-03-31T14:49:44.063 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/trigger/ 2024-03-31T14:49:44.063 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 9881 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/trigger/ledtrig-audio.ko 2024-03-31T14:49:44.063 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 10761 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/trigger/ledtrig-backlight.ko 2024-03-31T14:49:44.063 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 8985 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/trigger/ledtrig-camera.ko 2024-03-31T14:49:44.063 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 7513 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/trigger/ledtrig-default-on.ko 2024-03-31T14:49:44.063 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 12809 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/trigger/ledtrig-gpio.ko 2024-03-31T14:49:44.063 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 14137 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/trigger/ledtrig-heartbeat.ko 2024-03-31T14:49:44.063 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 13969 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/trigger/ledtrig-oneshot.ko 2024-03-31T14:49:44.063 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 11785 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/trigger/ledtrig-timer.ko 2024-03-31T14:49:44.064 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 13961 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/trigger/ledtrig-transient.ko 2024-03-31T14:49:44.064 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/ 2024-03-31T14:49:44.064 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/bcache/ 2024-03-31T14:49:44.064 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 647033 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/bcache/bcache.ko 2024-03-31T14:49:44.064 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 40473 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-bio-prison.ko 2024-03-31T14:49:44.064 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 93313 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-bufio.ko 2024-03-31T14:49:44.064 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 47433 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-cache-smq.ko 2024-03-31T14:49:44.064 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 155801 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-cache.ko 2024-03-31T14:49:44.064 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 107929 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-crypt.ko 2024-03-31T14:49:44.064 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 23945 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-delay.ko 2024-03-31T14:49:44.064 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 53761 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-era.ko 2024-03-31T14:49:44.064 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 28209 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-flakey.ko 2024-03-31T14:49:44.064 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 41425 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-log-userspace.ko 2024-03-31T14:49:44.065 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 32641 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-log.ko 2024-03-31T14:49:44.065 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 49441 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-mirror.ko 2024-03-31T14:49:44.065 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 77865 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-multipath.ko 2024-03-31T14:49:44.065 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 16369 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-queue-length.ko 2024-03-31T14:49:44.065 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 70977 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-raid.ko 2024-03-31T14:49:44.065 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 36193 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-region-hash.ko 2024-03-31T14:49:44.065 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 15769 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-round-robin.ko 2024-03-31T14:49:44.065 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 16697 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-service-time.ko 2024-03-31T14:49:44.065 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 109433 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-snapshot.ko 2024-03-31T14:49:44.065 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 19257 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-switch.ko 2024-03-31T14:49:44.065 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 181761 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-thin-pool.ko 2024-03-31T14:49:44.065 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 53729 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-verity.ko 2024-03-31T14:49:44.065 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 8473 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-zero.ko 2024-03-31T14:49:44.066 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/persistent-data/ 2024-03-31T14:49:44.066 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 199825 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/persistent-data/dm-persistent-data.ko 2024-03-31T14:49:44.066 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 37313 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/raid0.ko 2024-03-31T14:49:44.066 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 100833 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/raid1.ko 2024-03-31T14:49:44.066 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 126361 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/raid10.ko 2024-03-31T14:49:44.066 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 368481 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/raid456.ko 2024-03-31T14:49:44.066 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/message/ 2024-03-31T14:49:44.066 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/message/fusion/ 2024-03-31T14:49:44.066 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 178041 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/message/fusion/mptbase.ko 2024-03-31T14:49:44.066 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 63233 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/message/fusion/mptctl.ko 2024-03-31T14:49:44.066 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 46241 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/message/fusion/mptfc.ko 2024-03-31T14:49:44.066 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 37113 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/message/fusion/mptlan.ko 2024-03-31T14:49:44.067 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 123225 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/message/fusion/mptsas.ko 2024-03-31T14:49:44.067 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 80913 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/message/fusion/mptscsih.ko 2024-03-31T14:49:44.067 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 52857 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/message/fusion/mptspi.ko 2024-03-31T14:49:44.067 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/ 2024-03-31T14:49:44.067 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 17209 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/88pm800.ko 2024-03-31T14:49:44.067 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 12665 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/88pm805.ko 2024-03-31T14:49:44.067 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 11097 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/88pm80x.ko 2024-03-31T14:49:44.067 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 11561 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/arizona-i2c.ko 2024-03-31T14:49:44.067 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 123625 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/arizona.ko 2024-03-31T14:49:44.067 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 11865 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/bcm590xx.ko 2024-03-31T14:49:44.067 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 17361 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/cros_ec_dev.ko 2024-03-31T14:49:44.067 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 34841 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/dln2.ko 2024-03-31T14:49:44.067 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 16585 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/janz-cmodio.ko 2024-03-31T14:49:44.068 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 53481 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/kempld-core.ko 2024-03-31T14:49:44.068 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 27425 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/lm3533-core.ko 2024-03-31T14:49:44.068 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 14889 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/lm3533-ctrlbank.ko 2024-03-31T14:49:44.068 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 12913 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/lp3943.ko 2024-03-31T14:49:44.068 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 56961 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/lpc_ich.ko 2024-03-31T14:49:44.068 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 12017 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/lpc_sch.ko 2024-03-31T14:49:44.068 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 17313 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/max8907.ko 2024-03-31T14:49:44.068 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 29753 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/mc13xxx-core.ko 2024-03-31T14:49:44.068 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 11697 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/mc13xxx-i2c.ko 2024-03-31T14:49:44.068 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 11489 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/menf21bmc.ko 2024-03-31T14:49:44.068 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 17225 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/pcf50633-adc.ko 2024-03-31T14:49:44.068 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 9049 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/pcf50633-gpio.ko 2024-03-31T14:49:44.068 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 34777 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/pcf50633.ko 2024-03-31T14:49:44.068 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 9993 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/rdc321x-southbridge.ko 2024-03-31T14:49:44.069 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 20193 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/retu-mfd.ko 2024-03-31T14:49:44.069 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 80329 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/si476x-core.ko 2024-03-31T14:49:44.069 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 48361 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/sm501.ko 2024-03-31T14:49:44.069 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 51201 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/tps65010.ko 2024-03-31T14:49:44.069 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 10257 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/tps6507x.ko 2024-03-31T14:49:44.069 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 13225 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/viperboard.ko 2024-03-31T14:49:44.069 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 10441 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/vx855.ko 2024-03-31T14:49:44.069 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 17849 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/wl1273-core.ko 2024-03-31T14:49:44.069 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 44249 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/wm8994.ko 2024-03-31T14:49:44.069 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/ 2024-03-31T14:49:44.069 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 13729 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/ad525x_dpot-i2c.ko 2024-03-31T14:49:44.069 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 48385 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/ad525x_dpot.ko 2024-03-31T14:49:44.069 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/altera-stapl/ 2024-03-31T14:49:44.069 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 46489 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/altera-stapl/altera-stapl.ko 2024-03-31T14:49:44.070 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 18289 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/apds9802als.ko 2024-03-31T14:49:44.070 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 36633 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/apds990x.ko 2024-03-31T14:49:44.070 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 45489 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/bh1770glc.ko 2024-03-31T14:49:44.070 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/c2port/ 2024-03-31T14:49:44.070 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 11041 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/c2port/c2port-duramar2150.ko 2024-03-31T14:49:44.070 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 35489 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/c2port/core.ko 2024-03-31T14:49:44.070 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/cb710/ 2024-03-31T14:49:44.070 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 25065 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/cb710/cb710.ko 2024-03-31T14:49:44.070 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 17705 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/ds1682.ko 2024-03-31T14:49:44.070 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 8793 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/dummy-irq.ko 2024-03-31T14:49:44.070 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/echo/ 2024-03-31T14:49:44.070 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 13561 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/echo/echo.ko 2024-03-31T14:49:44.070 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/eeprom/ 2024-03-31T14:49:44.070 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 39017 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/eeprom/at24.ko 2024-03-31T14:49:44.071 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 14665 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/eeprom/eeprom_93cx6.ko 2024-03-31T14:49:44.071 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 13441 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/eeprom/max6875.ko 2024-03-31T14:49:44.071 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 32905 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/enclosure.ko 2024-03-31T14:49:44.071 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/genwqe/ 2024-03-31T14:49:44.071 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 152041 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/genwqe/genwqe_card.ko 2024-03-31T14:49:44.071 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 14033 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/hmc6352.ko 2024-03-31T14:49:44.071 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 31161 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/hpilo.ko 2024-03-31T14:49:44.071 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/ibmasm/ 2024-03-31T14:49:44.071 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 72505 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/ibmasm/ibmasm.ko 2024-03-31T14:49:44.071 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 19681 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/ics932s401.ko 2024-03-31T14:49:44.071 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 21585 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/isl29003.ko 2024-03-31T14:49:44.071 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 15657 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/isl29020.ko 2024-03-31T14:49:44.071 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/lis3lv02d/ 2024-03-31T14:49:44.072 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 42409 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/lis3lv02d/lis3lv02d.ko 2024-03-31T14:49:44.072 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 14201 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/lis3lv02d/lis3lv02d_i2c.ko 2024-03-31T14:49:44.072 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/mei/ 2024-03-31T14:49:44.072 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 126137 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/mei/mei-me.ko 2024-03-31T14:49:44.072 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 56361 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/mei/mei-txe.ko 2024-03-31T14:49:44.072 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 313545 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/mei/mei.ko 2024-03-31T14:49:44.072 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 31393 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/phantom.ko 2024-03-31T14:49:44.072 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/ti-st/ 2024-03-31T14:49:44.072 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 74289 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/ti-st/st_drv.ko 2024-03-31T14:49:44.072 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 24201 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/tifm_7xx1.ko 2024-03-31T14:49:44.072 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 28681 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/tifm_core.ko 2024-03-31T14:49:44.072 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 18633 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/tsl2550.ko 2024-03-31T14:49:44.072 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 48337 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/vmw_balloon.ko 2024-03-31T14:49:44.072 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/vmw_vmci/ 2024-03-31T14:49:44.072 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 183513 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/vmw_vmci/vmw_vmci.ko 2024-03-31T14:49:44.073 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ 2024-03-31T14:49:44.073 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/bonding/ 2024-03-31T14:49:44.073 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 491561 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/bonding/bonding.ko 2024-03-31T14:49:44.073 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/dsa/ 2024-03-31T14:49:44.073 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/dsa/b53/ 2024-03-31T14:49:44.073 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 105929 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/dsa/b53/b53_common.ko 2024-03-31T14:49:44.073 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 40209 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/dsa/bcm-sf2.ko 2024-03-31T14:49:44.073 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 15937 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/dsa/mv88e6060.ko 2024-03-31T14:49:44.073 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/dsa/mv88e6xxx/ 2024-03-31T14:49:44.073 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 458905 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/dsa/mv88e6xxx/mv88e6xxx.ko 2024-03-31T14:49:44.073 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 16945 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/dummy.ko 2024-03-31T14:49:44.073 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 17985 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/eql.ko 2024-03-31T14:49:44.073 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/ 2024-03-31T14:49:44.074 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/3com/ 2024-03-31T14:49:44.074 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 45953 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/3com/3c574_cs.ko 2024-03-31T14:49:44.074 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 42185 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/3com/3c589_cs.ko 2024-03-31T14:49:44.074 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 130657 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/3com/3c59x.ko 2024-03-31T14:49:44.074 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 68777 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/3com/typhoon.ko 2024-03-31T14:49:44.074 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/8390/ 2024-03-31T14:49:44.074 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 38041 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/8390/8390.ko 2024-03-31T14:49:44.074 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 64609 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/8390/axnet_cs.ko 2024-03-31T14:49:44.074 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 27081 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/8390/ne2k-pci.ko 2024-03-31T14:49:44.074 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 101609 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/8390/pcnet_cs.ko 2024-03-31T14:49:44.074 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/adaptec/ 2024-03-31T14:49:44.074 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 54617 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/adaptec/starfire.ko 2024-03-31T14:49:44.074 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/agere/ 2024-03-31T14:49:44.074 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 69161 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/agere/et131x.ko 2024-03-31T14:49:44.075 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/alteon/ 2024-03-31T14:49:44.075 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 59985 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/alteon/acenic.ko 2024-03-31T14:49:44.075 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/altera/ 2024-03-31T14:49:44.075 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 41385 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/altera/altera_tse.ko 2024-03-31T14:49:44.075 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/amd/ 2024-03-31T14:49:44.075 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 54609 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/amd/amd8111e.ko 2024-03-31T14:49:44.075 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 49185 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/amd/nmclan_cs.ko 2024-03-31T14:49:44.075 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 87449 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/amd/pcnet32.ko 2024-03-31T14:49:44.075 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/atheros/ 2024-03-31T14:49:44.075 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/atheros/alx/ 2024-03-31T14:49:44.075 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 97185 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/atheros/alx/alx.ko 2024-03-31T14:49:44.075 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/atheros/atl1c/ 2024-03-31T14:49:44.075 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 102481 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/atheros/atl1c/atl1c.ko 2024-03-31T14:49:44.075 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/atheros/atl1e/ 2024-03-31T14:49:44.075 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 90185 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/atheros/atl1e/atl1e.ko 2024-03-31T14:49:44.076 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/atheros/atlx/ 2024-03-31T14:49:44.076 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 81313 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/atheros/atlx/atl1.ko 2024-03-31T14:49:44.076 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 69537 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/atheros/atlx/atl2.ko 2024-03-31T14:49:44.076 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/broadcom/ 2024-03-31T14:49:44.076 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 78377 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/broadcom/b44.ko 2024-03-31T14:49:44.076 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 176689 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/broadcom/bnx2.ko 2024-03-31T14:49:44.076 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/broadcom/bnx2x/ 2024-03-31T14:49:44.076 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 1434449 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/broadcom/bnx2x/bnx2x.ko 2024-03-31T14:49:44.076 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 128361 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/broadcom/cnic.ko 2024-03-31T14:49:44.076 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/broadcom/genet/ 2024-03-31T14:49:44.076 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 131265 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/broadcom/genet/genet.ko 2024-03-31T14:49:44.076 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 314321 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/broadcom/tg3.ko 2024-03-31T14:49:44.076 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/brocade/ 2024-03-31T14:49:44.077 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/brocade/bna/ 2024-03-31T14:49:44.077 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 323681 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/brocade/bna/bna.ko 2024-03-31T14:49:44.077 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio/ 2024-03-31T14:49:44.077 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio/cxgb/ 2024-03-31T14:49:44.077 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 193537 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio/cxgb/cxgb.ko 2024-03-31T14:49:44.077 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio/cxgb3/ 2024-03-31T14:49:44.077 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 347161 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio/cxgb3/cxgb3.ko 2024-03-31T14:49:44.077 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio/cxgb4/ 2024-03-31T14:49:44.077 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 1045321 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio/cxgb4/cxgb4.ko 2024-03-31T14:49:44.077 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio/cxgb4vf/ 2024-03-31T14:49:44.077 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 166129 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf.ko 2024-03-31T14:49:44.077 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio/libcxgb/ 2024-03-31T14:49:44.077 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 34849 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio/libcxgb/libcxgb.ko 2024-03-31T14:49:44.078 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/cisco/ 2024-03-31T14:49:44.078 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/cisco/enic/ 2024-03-31T14:49:44.078 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 204185 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/cisco/enic/enic.ko 2024-03-31T14:49:44.078 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dec/ 2024-03-31T14:49:44.078 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dec/tulip/ 2024-03-31T14:49:44.078 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 69425 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dec/tulip/de2104x.ko 2024-03-31T14:49:44.078 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 61417 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dec/tulip/dmfe.ko 2024-03-31T14:49:44.078 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 141153 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dec/tulip/tulip.ko 2024-03-31T14:49:44.078 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 51361 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dec/tulip/uli526x.ko 2024-03-31T14:49:44.078 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 62825 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dec/tulip/winbond-840.ko 2024-03-31T14:49:44.078 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 31201 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dec/tulip/xircom_cb.ko 2024-03-31T14:49:44.078 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dlink/ 2024-03-31T14:49:44.078 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 60665 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dlink/dl2k.ko 2024-03-31T14:49:44.078 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 62825 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dlink/sundance.ko 2024-03-31T14:49:44.079 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 45481 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dnet.ko 2024-03-31T14:49:44.079 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 25089 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/ec_bhf.ko 2024-03-31T14:49:44.079 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/emulex/ 2024-03-31T14:49:44.079 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/emulex/benet/ 2024-03-31T14:49:44.079 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 281193 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/emulex/benet/be2net.ko 2024-03-31T14:49:44.079 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 49449 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/ethoc.ko 2024-03-31T14:49:44.079 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 51777 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/fealnx.ko 2024-03-31T14:49:44.079 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/fujitsu/ 2024-03-31T14:49:44.079 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 45545 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/fujitsu/fmvj18x_cs.ko 2024-03-31T14:49:44.079 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/ 2024-03-31T14:49:44.079 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 90297 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/e100.ko 2024-03-31T14:49:44.079 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/e1000/ 2024-03-31T14:49:44.079 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 295673 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/e1000/e1000.ko 2024-03-31T14:49:44.080 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/e1000e/ 2024-03-31T14:49:44.080 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 584297 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/e1000e/e1000e.ko 2024-03-31T14:49:44.080 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/fm10k/ 2024-03-31T14:49:44.080 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 245977 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/fm10k/fm10k.ko 2024-03-31T14:49:44.080 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/i40e/ 2024-03-31T14:49:44.080 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 944449 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/i40e/i40e.ko 2024-03-31T14:49:44.080 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/iavf/ 2024-03-31T14:49:44.080 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 288481 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/iavf/iavf.ko 2024-03-31T14:49:44.080 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/igb/ 2024-03-31T14:49:44.080 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 514337 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/igb/igb.ko 2024-03-31T14:49:44.080 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/igbvf/ 2024-03-31T14:49:44.080 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 93465 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/igbvf/igbvf.ko 2024-03-31T14:49:44.080 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/ixgbe/ 2024-03-31T14:49:44.080 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 735129 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko 2024-03-31T14:49:44.080 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/ixgbevf/ 2024-03-31T14:49:44.081 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 151649 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/ixgbevf/ixgbevf.ko 2024-03-31T14:49:44.081 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 86985 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/jme.ko 2024-03-31T14:49:44.081 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/marvell/ 2024-03-31T14:49:44.081 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 21233 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/marvell/mvmdio.ko 2024-03-31T14:49:44.081 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 102433 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/marvell/skge.ko 2024-03-31T14:49:44.081 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 122601 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/marvell/sky2.ko 2024-03-31T14:49:44.081 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/mellanox/ 2024-03-31T14:49:44.081 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/mellanox/mlx4/ 2024-03-31T14:49:44.081 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 727377 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_core.ko 2024-03-31T14:49:44.081 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 278825 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_en.ko 2024-03-31T14:49:44.081 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/mellanox/mlx5/ 2024-03-31T14:49:44.081 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/mellanox/mlx5/core/ 2024-03-31T14:49:44.082 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 1122065 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.ko 2024-03-31T14:49:44.082 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/micrel/ 2024-03-31T14:49:44.082 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 52977 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/micrel/ks8842.ko 2024-03-31T14:49:44.082 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 40873 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/micrel/ks8851_common.ko 2024-03-31T14:49:44.082 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 19385 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/micrel/ks8851_par.ko 2024-03-31T14:49:44.082 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 95569 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/micrel/ksz884x.ko 2024-03-31T14:49:44.082 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/myricom/ 2024-03-31T14:49:44.082 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/myricom/myri10ge/ 2024-03-31T14:49:44.082 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 108401 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/myricom/myri10ge/myri10ge.ko 2024-03-31T14:49:44.082 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/natsemi/ 2024-03-31T14:49:44.082 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 73609 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/natsemi/natsemi.ko 2024-03-31T14:49:44.082 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 47793 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/natsemi/ns83820.ko 2024-03-31T14:49:44.082 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/neterion/ 2024-03-31T14:49:44.082 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 151193 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/neterion/s2io.ko 2024-03-31T14:49:44.083 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/nvidia/ 2024-03-31T14:49:44.083 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 136793 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/nvidia/forcedeth.ko 2024-03-31T14:49:44.083 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/packetengines/ 2024-03-31T14:49:44.083 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 50425 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/packetengines/hamachi.ko 2024-03-31T14:49:44.083 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 42977 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/packetengines/yellowfin.ko 2024-03-31T14:49:44.083 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/qlogic/ 2024-03-31T14:49:44.083 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/qlogic/netxen/ 2024-03-31T14:49:44.083 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 227841 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/qlogic/netxen/netxen_nic.ko 2024-03-31T14:49:44.083 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 87009 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/qlogic/qla3xxx.ko 2024-03-31T14:49:44.083 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/qlogic/qlcnic/ 2024-03-31T14:49:44.083 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 671985 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/qlogic/qlcnic/qlcnic.ko 2024-03-31T14:49:44.083 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/rdc/ 2024-03-31T14:49:44.083 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 42929 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/rdc/r6040.ko 2024-03-31T14:49:44.083 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/realtek/ 2024-03-31T14:49:44.084 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 67273 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/realtek/8139cp.ko 2024-03-31T14:49:44.084 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 83801 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/realtek/8139too.ko 2024-03-31T14:49:44.084 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 29153 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/realtek/atp.ko 2024-03-31T14:49:44.084 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 194057 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/realtek/r8169.ko 2024-03-31T14:49:44.084 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/samsung/ 2024-03-31T14:49:44.084 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/samsung/sxgbe/ 2024-03-31T14:49:44.084 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 104001 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/samsung/sxgbe/samsung-sxgbe.ko 2024-03-31T14:49:44.084 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/sfc/ 2024-03-31T14:49:44.084 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 1066817 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/sfc/sfc.ko 2024-03-31T14:49:44.084 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/silan/ 2024-03-31T14:49:44.084 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 47001 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/silan/sc92031.ko 2024-03-31T14:49:44.084 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/sis/ 2024-03-31T14:49:44.084 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 54721 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/sis/sis190.ko 2024-03-31T14:49:44.084 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 63193 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/sis/sis900.ko 2024-03-31T14:49:44.085 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/smsc/ 2024-03-31T14:49:44.085 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 61417 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/smsc/epic100.ko 2024-03-31T14:49:44.085 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 67289 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/smsc/smc91c92_cs.ko 2024-03-31T14:49:44.085 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 73249 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/smsc/smsc911x.ko 2024-03-31T14:49:44.085 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 68041 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/smsc/smsc9420.ko 2024-03-31T14:49:44.085 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/stmicro/ 2024-03-31T14:49:44.085 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/stmicro/stmmac/ 2024-03-31T14:49:44.085 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 13025 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-generic.ko 2024-03-31T14:49:44.085 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 39377 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-intel.ko 2024-03-31T14:49:44.085 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 18449 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac-platform.ko 2024-03-31T14:49:44.085 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 495641 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac.ko 2024-03-31T14:49:44.085 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/sun/ 2024-03-31T14:49:44.085 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 103177 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/sun/cassini.ko 2024-03-31T14:49:44.085 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 166041 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/sun/niu.ko 2024-03-31T14:49:44.086 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 66265 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/sun/sungem.ko 2024-03-31T14:49:44.086 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 78713 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/sun/sunhme.ko 2024-03-31T14:49:44.086 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/tehuti/ 2024-03-31T14:49:44.086 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 50009 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/tehuti/tehuti.ko 2024-03-31T14:49:44.086 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/ti/ 2024-03-31T14:49:44.086 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 92337 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/ti/tlan.ko 2024-03-31T14:49:44.086 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/via/ 2024-03-31T14:49:44.086 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 79929 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/via/via-rhine.ko 2024-03-31T14:49:44.086 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 83713 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/via/via-velocity.ko 2024-03-31T14:49:44.086 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/wiznet/ 2024-03-31T14:49:44.086 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 46305 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/wiznet/w5100.ko 2024-03-31T14:49:44.086 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 30817 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/wiznet/w5300.ko 2024-03-31T14:49:44.086 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/xircom/ 2024-03-31T14:49:44.087 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 59977 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/xircom/xirc2ps_cs.ko 2024-03-31T14:49:44.087 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/fddi/ 2024-03-31T14:49:44.087 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 41177 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/fddi/defxx.ko 2024-03-31T14:49:44.087 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/fddi/skfp/ 2024-03-31T14:49:44.087 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 217025 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/fddi/skfp/skfp.ko 2024-03-31T14:49:44.087 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 74985 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/geneve.ko 2024-03-31T14:49:44.087 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/hyperv/ 2024-03-31T14:49:44.087 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 237241 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/hyperv/hv_netvsc.ko 2024-03-31T14:49:44.087 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ieee802154/ 2024-03-31T14:49:44.087 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 18297 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ieee802154/fakelb.ko 2024-03-31T14:49:44.087 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 25273 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ifb.ko 2024-03-31T14:49:44.087 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ipvlan/ 2024-03-31T14:49:44.087 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 85841 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ipvlan/ipvlan.ko 2024-03-31T14:49:44.088 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 62889 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/macvlan.ko 2024-03-31T14:49:44.088 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 16153 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/macvtap.ko 2024-03-31T14:49:44.088 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/mdio/ 2024-03-31T14:49:44.088 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 21241 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/mdio/mdio-bcm-unimac.ko 2024-03-31T14:49:44.088 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 14129 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/mdio/mdio-bitbang.ko 2024-03-31T14:49:44.088 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 13721 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/mdio/mdio-gpio.ko 2024-03-31T14:49:44.088 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 8721 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/mdio/mdio-regmap.ko 2024-03-31T14:49:44.088 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 15929 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/mdio.ko 2024-03-31T14:49:44.088 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 20417 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/mii.ko 2024-03-31T14:49:44.088 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 45049 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/netconsole.ko 2024-03-31T14:49:44.088 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 12985 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/nlmon.ko 2024-03-31T14:49:44.088 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/pcs/ 2024-03-31T14:49:44.088 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 17889 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/pcs/pcs-lynx.ko 2024-03-31T14:49:44.088 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 39121 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/pcs/pcs_xpcs.ko 2024-03-31T14:49:44.089 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/ 2024-03-31T14:49:44.089 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 10361 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/amd.ko 2024-03-31T14:49:44.089 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 49929 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/bcm-phy-lib.ko 2024-03-31T14:49:44.089 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 44625 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/bcm7xxx.ko 2024-03-31T14:49:44.089 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 11553 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/bcm87xx.ko 2024-03-31T14:49:44.089 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 44817 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/broadcom.ko 2024-03-31T14:49:44.089 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 10537 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/cicada.ko 2024-03-31T14:49:44.089 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 12937 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/davicom.ko 2024-03-31T14:49:44.089 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 9361 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/et1011c.ko 2024-03-31T14:49:44.089 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 24873 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/icplus.ko 2024-03-31T14:49:44.089 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 15761 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/lxt.ko 2024-03-31T14:49:44.089 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 81953 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/marvell.ko 2024-03-31T14:49:44.089 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 76561 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/micrel.ko 2024-03-31T14:49:44.090 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 11937 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/national.ko 2024-03-31T14:49:44.090 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 99001 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/phylink.ko 2024-03-31T14:49:44.090 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 9977 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/qsemi.ko 2024-03-31T14:49:44.090 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 49969 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/realtek.ko 2024-03-31T14:49:44.090 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 30089 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/smsc.ko 2024-03-31T14:49:44.090 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 10769 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/ste10Xp.ko 2024-03-31T14:49:44.090 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 24201 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/vitesse.ko 2024-03-31T14:49:44.090 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/plip/ 2024-03-31T14:49:44.090 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 43761 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/plip/plip.ko 2024-03-31T14:49:44.090 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ppp/ 2024-03-31T14:49:44.090 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 15881 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ppp/bsd_comp.ko 2024-03-31T14:49:44.090 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 29721 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ppp/ppp_async.ko 2024-03-31T14:49:44.090 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 17673 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ppp/ppp_deflate.ko 2024-03-31T14:49:44.090 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 18529 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ppp/ppp_mppe.ko 2024-03-31T14:49:44.091 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 28745 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ppp/ppp_synctty.ko 2024-03-31T14:49:44.091 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 42665 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ppp/pppoe.ko 2024-03-31T14:49:44.091 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 13345 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ppp/pppox.ko 2024-03-31T14:49:44.091 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 32665 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ppp/pptp.ko 2024-03-31T14:49:44.091 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 28353 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/sb1000.ko 2024-03-31T14:49:44.091 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/slip/ 2024-03-31T14:49:44.091 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 42385 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/slip/slip.ko 2024-03-31T14:49:44.091 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 26945 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/sungem_phy.ko 2024-03-31T14:49:44.091 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 64553 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/tap.ko 2024-03-31T14:49:44.091 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/team/ 2024-03-31T14:49:44.091 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 107225 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/team/team.ko 2024-03-31T14:49:44.091 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 13449 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/team/team_mode_activebackup.ko 2024-03-31T14:49:44.091 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 10209 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/team/team_mode_broadcast.ko 2024-03-31T14:49:44.091 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 28833 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/team/team_mode_loadbalance.ko 2024-03-31T14:49:44.092 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 9609 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/team/team_mode_random.ko 2024-03-31T14:49:44.092 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 9273 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/team/team_mode_roundrobin.ko 2024-03-31T14:49:44.092 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 86817 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/veth.ko 2024-03-31T14:49:44.092 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/vmxnet3/ 2024-03-31T14:49:44.092 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 144345 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/vmxnet3/vmxnet3.ko 2024-03-31T14:49:44.092 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/vxlan/ 2024-03-31T14:49:44.092 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 307169 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/vxlan/vxlan.ko 2024-03-31T14:49:44.092 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/xen-netback/ 2024-03-31T14:49:44.092 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 143993 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/xen-netback/xen-netback.ko 2024-03-31T14:49:44.092 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ntb/ 2024-03-31T14:49:44.092 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 20873 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ntb/ntb.ko 2024-03-31T14:49:44.092 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/nvme/ 2024-03-31T14:49:44.092 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/nvme/host/ 2024-03-31T14:49:44.092 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 267721 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/nvme/host/nvme-core.ko 2024-03-31T14:49:44.092 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 103353 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/nvme/host/nvme.ko 2024-03-31T14:49:44.093 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/parport/ 2024-03-31T14:49:44.093 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 131297 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/parport/parport.ko 2024-03-31T14:49:44.093 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 14457 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/parport/parport_cs.ko 2024-03-31T14:49:44.093 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 85209 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/parport/parport_pc.ko 2024-03-31T14:49:44.093 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 27561 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/parport/parport_serial.ko 2024-03-31T14:49:44.093 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pci/ 2024-03-31T14:49:44.093 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pci/hotplug/ 2024-03-31T14:49:44.093 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 24905 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pci/hotplug/acpiphp_ibm.ko 2024-03-31T14:49:44.093 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 18073 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pci/hotplug/cpcihp_generic.ko 2024-03-31T14:49:44.093 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 20289 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pci/hotplug/cpcihp_zt5550.ko 2024-03-31T14:49:44.093 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 11697 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pci/pci-stub.ko 2024-03-31T14:49:44.093 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 46361 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pci/xen-pcifront.ko 2024-03-31T14:49:44.093 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pcmcia/ 2024-03-31T14:49:44.093 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 26697 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pcmcia/i82092.ko 2024-03-31T14:49:44.094 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 146129 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pcmcia/pcmcia.ko 2024-03-31T14:49:44.094 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 62065 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pcmcia/pcmcia_core.ko 2024-03-31T14:49:44.094 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 34241 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pcmcia/pcmcia_rsrc.ko 2024-03-31T14:49:44.094 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 34169 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pcmcia/pd6729.ko 2024-03-31T14:49:44.094 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 112785 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pcmcia/yenta_socket.ko 2024-03-31T14:49:44.094 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/phy/ 2024-03-31T14:49:44.094 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/phy/broadcom/ 2024-03-31T14:49:44.094 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 10729 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/phy/broadcom/phy-bcm-kona-usb2.ko 2024-03-31T14:49:44.094 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/phy/samsung/ 2024-03-31T14:49:44.094 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 8337 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/phy/samsung/phy-exynos-usb2.ko 2024-03-31T14:49:44.094 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pinctrl/ 2024-03-31T14:49:44.094 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pinctrl/intel/ 2024-03-31T14:49:44.094 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 69025 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pinctrl/intel/pinctrl-cherryview.ko 2024-03-31T14:49:44.094 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/ 2024-03-31T14:49:44.095 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/chrome/ 2024-03-31T14:49:44.095 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 51217 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/chrome/cros-ec-sensorhub.ko 2024-03-31T14:49:44.095 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 24577 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/chrome/cros_ec.ko 2024-03-31T14:49:44.095 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 21345 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/chrome/cros_ec_chardev.ko 2024-03-31T14:49:44.095 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 25369 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/chrome/cros_ec_debugfs.ko 2024-03-31T14:49:44.095 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 26097 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/chrome/cros_ec_lightbar.ko 2024-03-31T14:49:44.095 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 18209 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/chrome/cros_ec_sysfs.ko 2024-03-31T14:49:44.095 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 17249 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/chrome/cros_usbpd_notify.ko 2024-03-31T14:49:44.095 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/ 2024-03-31T14:49:44.095 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 73593 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/acer-wmi.ko 2024-03-31T14:49:44.095 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 39049 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/acerhdf.ko 2024-03-31T14:49:44.095 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 36921 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/apple-gmux.ko 2024-03-31T14:49:44.095 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 59561 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/asus-laptop.ko 2024-03-31T14:49:44.095 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 31377 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/asus-nb-wmi.ko 2024-03-31T14:49:44.096 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 131937 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/asus-wmi.ko 2024-03-31T14:49:44.096 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 36745 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/classmate-laptop.ko 2024-03-31T14:49:44.096 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 49017 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/eeepc-laptop.ko 2024-03-31T14:49:44.096 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 13857 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/eeepc-wmi.ko 2024-03-31T14:49:44.096 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 40161 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/fujitsu-laptop.ko 2024-03-31T14:49:44.096 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 22417 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/fujitsu-tablet.ko 2024-03-31T14:49:44.096 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 36497 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/hdaps.ko 2024-03-31T14:49:44.096 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 19353 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/ibm_rtl.ko 2024-03-31T14:49:44.096 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/intel/ 2024-03-31T14:49:44.096 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 11977 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/intel/intel-rst.ko 2024-03-31T14:49:44.096 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 8873 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/intel/intel-smartconnect.ko 2024-03-31T14:49:44.096 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 44529 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/intel_ips.ko 2024-03-31T14:49:44.096 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 20177 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/msi-wmi.ko 2024-03-31T14:49:44.096 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 10281 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/mxm-wmi.ko 2024-03-31T14:49:44.096 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 34697 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/panasonic-laptop.ko 2024-03-31T14:49:44.097 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 42225 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/samsung-laptop.ko 2024-03-31T14:49:44.097 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 14065 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/samsung-q10.ko 2024-03-31T14:49:44.097 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 17217 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/topstar-laptop.ko 2024-03-31T14:49:44.097 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 13609 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/toshiba_bluetooth.ko 2024-03-31T14:49:44.097 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 16977 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/toshiba_haps.ko 2024-03-31T14:49:44.097 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 10041 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/wmi-bmof.ko 2024-03-31T14:49:44.097 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 48473 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/wmi.ko 2024-03-31T14:49:44.097 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/power/ 2024-03-31T14:49:44.097 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/power/supply/ 2024-03-31T14:49:44.097 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 30129 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/power/supply/88pm860x_battery.ko 2024-03-31T14:49:44.097 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 33881 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/power/supply/88pm860x_charger.ko 2024-03-31T14:49:44.097 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 54953 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/power/supply/bq2415x_charger.ko 2024-03-31T14:49:44.097 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 59497 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/power/supply/bq24190_charger.ko 2024-03-31T14:49:44.097 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 21449 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/power/supply/bq24735-charger.ko 2024-03-31T14:49:44.098 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 17225 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/power/supply/cros_peripheral_charger.ko 2024-03-31T14:49:44.098 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 19505 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/power/supply/da9030_battery.ko 2024-03-31T14:49:44.098 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 19737 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/power/supply/da9052-battery.ko 2024-03-31T14:49:44.098 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 29177 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/power/supply/ds2760_battery.ko 2024-03-31T14:49:44.098 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 23113 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/power/supply/ds2780_battery.ko 2024-03-31T14:49:44.098 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 23249 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/power/supply/ds2781_battery.ko 2024-03-31T14:49:44.098 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 23545 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/power/supply/ds2782_battery.ko 2024-03-31T14:49:44.098 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 20249 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/power/supply/gpio-charger.ko 2024-03-31T14:49:44.098 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 17985 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/power/supply/isp1704_charger.ko 2024-03-31T14:49:44.098 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 20169 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/power/supply/lp8727_charger.ko 2024-03-31T14:49:44.098 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 12473 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/power/supply/max14577_charger.ko 2024-03-31T14:49:44.098 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 30185 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/power/supply/max17042_battery.ko 2024-03-31T14:49:44.098 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 18481 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/power/supply/max8903_charger.ko 2024-03-31T14:49:44.099 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 23625 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/power/supply/max8925_power.ko 2024-03-31T14:49:44.099 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 21553 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/power/supply/pcf50633-charger.ko 2024-03-31T14:49:44.099 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 37833 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/power/supply/sbs-battery.ko 2024-03-31T14:49:44.099 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 28281 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/power/supply/test_power.ko 2024-03-31T14:49:44.099 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 18761 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/power/supply/tps65090-charger.ko 2024-03-31T14:49:44.099 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 11865 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/power/supply/wm831x_backup.ko 2024-03-31T14:49:44.099 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 24353 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/power/supply/wm831x_power.ko 2024-03-31T14:49:44.099 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 22313 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/power/supply/wm8350_power.ko 2024-03-31T14:49:44.099 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/powercap/ 2024-03-31T14:49:44.099 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 58097 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/powercap/intel_rapl_common.ko 2024-03-31T14:49:44.099 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 17617 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/powercap/intel_rapl_msr.ko 2024-03-31T14:49:44.099 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pps/ 2024-03-31T14:49:44.099 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pps/clients/ 2024-03-31T14:49:44.099 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 15425 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pps/clients/pps-gpio.ko 2024-03-31T14:49:44.100 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 13145 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pps/clients/pps-ldisc.ko 2024-03-31T14:49:44.100 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 16065 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pps/clients/pps_parport.ko 2024-03-31T14:49:44.100 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 41785 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pps/pps_core.ko 2024-03-31T14:49:44.100 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ptp/ 2024-03-31T14:49:44.100 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 70849 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ptp/ptp.ko 2024-03-31T14:49:44.100 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 16361 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ptp/ptp_kvm.ko 2024-03-31T14:49:44.100 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/ 2024-03-31T14:49:44.100 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 23393 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-88pm80x.ko 2024-03-31T14:49:44.100 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 24417 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-88pm860x.ko 2024-03-31T14:49:44.100 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 15289 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-bq32k.ko 2024-03-31T14:49:44.100 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 20001 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-da9052.ko 2024-03-31T14:49:44.100 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 18401 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-da9055.ko 2024-03-31T14:49:44.100 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 20737 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-da9063.ko 2024-03-31T14:49:44.101 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 16353 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-ds1286.ko 2024-03-31T14:49:44.101 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 68473 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-ds1307.ko 2024-03-31T14:49:44.101 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 20713 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-ds1374.ko 2024-03-31T14:49:44.101 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 18113 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-ds1511.ko 2024-03-31T14:49:44.101 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 16633 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-ds1553.ko 2024-03-31T14:49:44.101 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 13849 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-ds1672.ko 2024-03-31T14:49:44.101 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 13265 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-ds1742.ko 2024-03-31T14:49:44.101 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 14593 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-ds2404.ko 2024-03-31T14:49:44.101 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 23601 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-ds3232.ko 2024-03-31T14:49:44.101 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 10977 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-em3027.ko 2024-03-31T14:49:44.101 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 24225 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-fm3130.ko 2024-03-31T14:49:44.101 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 20625 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-isl12022.ko 2024-03-31T14:49:44.101 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 35865 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-isl1208.ko 2024-03-31T14:49:44.101 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 14569 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-lp8788.ko 2024-03-31T14:49:44.102 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 37993 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-m41t80.ko 2024-03-31T14:49:44.102 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 11985 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-m48t35.ko 2024-03-31T14:49:44.102 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 25841 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-m48t59.ko 2024-03-31T14:49:44.102 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 14737 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-m48t86.ko 2024-03-31T14:49:44.102 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 11529 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-max6900.ko 2024-03-31T14:49:44.102 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 14569 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-max8907.ko 2024-03-31T14:49:44.102 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 15241 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-max8925.ko 2024-03-31T14:49:44.102 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 25705 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-max8997.ko 2024-03-31T14:49:44.102 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 16577 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-max8998.ko 2024-03-31T14:49:44.102 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 18409 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-mc13xxx.ko 2024-03-31T14:49:44.102 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 11305 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-msm6242.ko 2024-03-31T14:49:44.102 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 19521 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-palmas.ko 2024-03-31T14:49:44.102 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 46529 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-pcf2127.ko 2024-03-31T14:49:44.102 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 17873 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-pcf50633.ko 2024-03-31T14:49:44.103 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 26889 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-pcf85063.ko 2024-03-31T14:49:44.103 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 21721 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-pcf8523.ko 2024-03-31T14:49:44.103 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 29713 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-pcf8563.ko 2024-03-31T14:49:44.103 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 12337 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-pcf8583.ko 2024-03-31T14:49:44.103 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 17929 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-rc5t583.ko 2024-03-31T14:49:44.103 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 12945 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-rp5c01.ko 2024-03-31T14:49:44.103 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 33649 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-rs5c372.ko 2024-03-31T14:49:44.103 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 27729 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-rv3029c2.ko 2024-03-31T14:49:44.103 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 25153 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-rx8025.ko 2024-03-31T14:49:44.103 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 20009 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-rx8581.ko 2024-03-31T14:49:44.103 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 21513 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-s35390a.ko 2024-03-31T14:49:44.103 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 16721 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-stk17ta8.ko 2024-03-31T14:49:44.103 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 17785 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-tps6586x.ko 2024-03-31T14:49:44.104 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 21233 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-tps65910.ko 2024-03-31T14:49:44.104 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 20441 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-wm831x.ko 2024-03-31T14:49:44.104 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 20985 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-wm8350.ko 2024-03-31T14:49:44.104 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 27769 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-x1205.ko 2024-03-31T14:49:44.104 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/ 2024-03-31T14:49:44.104 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 77521 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/3w-9xxx.ko 2024-03-31T14:49:44.104 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 57713 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/3w-sas.ko 2024-03-31T14:49:44.104 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 55009 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/3w-xxxx.ko 2024-03-31T14:49:44.104 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 135249 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/BusLogic.ko 2024-03-31T14:49:44.104 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 26561 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/a100u2w.ko 2024-03-31T14:49:44.104 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/aacraid/ 2024-03-31T14:49:44.104 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 271385 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/aacraid/aacraid.ko 2024-03-31T14:49:44.104 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 100529 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/advansys.ko 2024-03-31T14:49:44.104 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/aic7xxx/ 2024-03-31T14:49:44.104 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 356833 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/aic7xxx/aic79xx.ko 2024-03-31T14:49:44.105 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 295889 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/aic7xxx/aic7xxx.ko 2024-03-31T14:49:44.105 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/aic94xx/ 2024-03-31T14:49:44.105 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 189089 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/aic94xx/aic94xx.ko 2024-03-31T14:49:44.105 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 23977 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/am53c974.ko 2024-03-31T14:49:44.105 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/arcmsr/ 2024-03-31T14:49:44.105 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 108193 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/arcmsr/arcmsr.ko 2024-03-31T14:49:44.105 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 38089 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/atp870u.ko 2024-03-31T14:49:44.105 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/be2iscsi/ 2024-03-31T14:49:44.105 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 246913 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/be2iscsi/be2iscsi.ko 2024-03-31T14:49:44.105 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/bfa/ 2024-03-31T14:49:44.105 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 1171857 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/bfa/bfa.ko 2024-03-31T14:49:44.105 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/bnx2fc/ 2024-03-31T14:49:44.105 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 238369 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/bnx2fc/bnx2fc.ko 2024-03-31T14:49:44.105 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/bnx2i/ 2024-03-31T14:49:44.106 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 115305 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/bnx2i/bnx2i.ko 2024-03-31T14:49:44.106 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 40777 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/ch.ko 2024-03-31T14:49:44.106 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/csiostor/ 2024-03-31T14:49:44.106 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 308905 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/csiostor/csiostor.ko 2024-03-31T14:49:44.106 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/cxgbi/ 2024-03-31T14:49:44.106 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/cxgbi/cxgb3i/ 2024-03-31T14:49:44.106 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 76033 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/cxgbi/cxgb3i/cxgb3i.ko 2024-03-31T14:49:44.106 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/cxgbi/cxgb4i/ 2024-03-31T14:49:44.106 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 101417 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/cxgbi/cxgb4i/cxgb4i.ko 2024-03-31T14:49:44.106 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 146745 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/cxgbi/libcxgbi.ko 2024-03-31T14:49:44.106 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 75969 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/dc395x.ko 2024-03-31T14:49:44.106 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 38201 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/dmx3191d.ko 2024-03-31T14:49:44.106 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/esas2r/ 2024-03-31T14:49:44.107 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 212329 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/esas2r/esas2r.ko 2024-03-31T14:49:44.107 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 60497 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/esp_scsi.ko 2024-03-31T14:49:44.107 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/fcoe/ 2024-03-31T14:49:44.107 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 78217 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/fcoe/fcoe.ko 2024-03-31T14:49:44.107 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 157713 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/fcoe/libfcoe.ko 2024-03-31T14:49:44.107 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 18849 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/fdomain.ko 2024-03-31T14:49:44.107 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/fnic/ 2024-03-31T14:49:44.107 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 294881 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/fnic/fnic.ko 2024-03-31T14:49:44.107 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 201721 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/hpsa.ko 2024-03-31T14:49:44.107 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 51113 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/hptiop.ko 2024-03-31T14:49:44.107 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 48937 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/hv_storvsc.ko 2024-03-31T14:49:44.107 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 36121 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/imm.ko 2024-03-31T14:49:44.107 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 43969 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/initio.ko 2024-03-31T14:49:44.108 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 265937 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/ipr.ko 2024-03-31T14:49:44.108 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 101873 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/ips.ko 2024-03-31T14:49:44.108 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/isci/ 2024-03-31T14:49:44.108 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 317129 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/isci/isci.ko 2024-03-31T14:49:44.108 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 26281 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/iscsi_boot_sysfs.ko 2024-03-31T14:49:44.108 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 52857 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/iscsi_tcp.ko 2024-03-31T14:49:44.108 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/libfc/ 2024-03-31T14:49:44.108 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 315273 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/libfc/libfc.ko 2024-03-31T14:49:44.108 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 145489 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/libiscsi.ko 2024-03-31T14:49:44.108 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 53801 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/libiscsi_tcp.ko 2024-03-31T14:49:44.108 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/libsas/ 2024-03-31T14:49:44.108 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 216345 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/libsas/libsas.ko 2024-03-31T14:49:44.108 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/lpfc/ 2024-03-31T14:49:44.109 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 2193953 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/lpfc/lpfc.ko 2024-03-31T14:49:44.109 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/megaraid/ 2024-03-31T14:49:44.109 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 73465 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/megaraid/megaraid_mbox.ko 2024-03-31T14:49:44.109 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 33425 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/megaraid/megaraid_mm.ko 2024-03-31T14:49:44.109 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 310665 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/megaraid/megaraid_sas.ko 2024-03-31T14:49:44.109 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 85513 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/megaraid.ko 2024-03-31T14:49:44.109 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/mpt3sas/ 2024-03-31T14:49:44.109 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 659585 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/mpt3sas/mpt3sas.ko 2024-03-31T14:49:44.109 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/mvsas/ 2024-03-31T14:49:44.109 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 119465 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/mvsas/mvsas.ko 2024-03-31T14:49:44.109 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 78033 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/mvumi.ko 2024-03-31T14:49:44.109 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/pcmcia/ 2024-03-31T14:49:44.109 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 81545 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/pcmcia/aha152x_cs.ko 2024-03-31T14:49:44.110 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 11609 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/pcmcia/fdomain_cs.ko 2024-03-31T14:49:44.110 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 20761 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/pcmcia/qlogic_cs.ko 2024-03-31T14:49:44.110 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 23753 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/pcmcia/sym53c500_cs.ko 2024-03-31T14:49:44.110 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/pm8001/ 2024-03-31T14:49:44.110 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 366305 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/pm8001/pm80xx.ko 2024-03-31T14:49:44.110 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 124593 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/pmcraid.ko 2024-03-31T14:49:44.110 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 34417 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/ppa.ko 2024-03-31T14:49:44.110 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 55329 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/qla1280.ko 2024-03-31T14:49:44.110 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/qla2xxx/ 2024-03-31T14:49:44.110 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 1876657 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/qla2xxx/qla2xxx.ko 2024-03-31T14:49:44.110 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 85185 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/qla2xxx/tcm_qla2xxx.ko 2024-03-31T14:49:44.110 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/qla4xxx/ 2024-03-31T14:49:44.110 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 558897 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/qla4xxx/qla4xxx.ko 2024-03-31T14:49:44.110 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 18921 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/qlogicfas408.ko 2024-03-31T14:49:44.111 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 15889 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/raid_class.ko 2024-03-31T14:49:44.111 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 261073 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/scsi_debug.ko 2024-03-31T14:49:44.111 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 173921 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/scsi_transport_fc.ko 2024-03-31T14:49:44.111 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 280105 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/scsi_transport_iscsi.ko 2024-03-31T14:49:44.111 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 97617 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/scsi_transport_sas.ko 2024-03-31T14:49:44.111 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 73857 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/scsi_transport_spi.ko 2024-03-31T14:49:44.111 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 44881 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/scsi_transport_srp.ko 2024-03-31T14:49:44.111 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 26153 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/ses.ko 2024-03-31T14:49:44.111 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 115641 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/st.ko 2024-03-31T14:49:44.111 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 46729 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/stex.ko 2024-03-31T14:49:44.111 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/sym53c8xx_2/ 2024-03-31T14:49:44.111 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 164865 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/sym53c8xx_2/sym53c8xx.ko 2024-03-31T14:49:44.111 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 44849 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/virtio_scsi.ko 2024-03-31T14:49:44.111 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 50225 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/vmw_pvscsi.ko 2024-03-31T14:49:44.112 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 39873 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/wd719x.ko 2024-03-31T14:49:44.112 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 49921 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/xen-scsifront.ko 2024-03-31T14:49:44.112 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ssb/ 2024-03-31T14:49:44.112 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 129345 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ssb/ssb.ko 2024-03-31T14:49:44.112 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/ 2024-03-31T14:49:44.112 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/iscsi/ 2024-03-31T14:49:44.112 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 736665 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/iscsi/iscsi_target_mod.ko 2024-03-31T14:49:44.112 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/loopback/ 2024-03-31T14:49:44.112 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 49873 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/loopback/tcm_loop.ko 2024-03-31T14:49:44.112 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 39609 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/target_core_file.ko 2024-03-31T14:49:44.112 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 42801 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/target_core_iblock.ko 2024-03-31T14:49:44.112 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 931433 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/target_core_mod.ko 2024-03-31T14:49:44.112 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 44145 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/target_core_pscsi.ko 2024-03-31T14:49:44.112 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 122481 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/target_core_user.ko 2024-03-31T14:49:44.112 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/tcm_fc/ 2024-03-31T14:49:44.113 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 75497 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/tcm_fc/tcm_fc.ko 2024-03-31T14:49:44.113 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/ 2024-03-31T14:49:44.113 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/ 2024-03-31T14:49:44.113 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/ 2024-03-31T14:49:44.113 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 23377 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/acpi_thermal_rel.ko 2024-03-31T14:49:44.113 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 29217 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/int3400_thermal.ko 2024-03-31T14:49:44.113 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 9849 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/int3401_thermal.ko 2024-03-31T14:49:44.113 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 10161 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/int3402_thermal.ko 2024-03-31T14:49:44.113 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 14865 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/int3403_thermal.ko 2024-03-31T14:49:44.113 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 16249 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/int340x_thermal_zone.ko 2024-03-31T14:49:44.113 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 30417 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device.ko 2024-03-31T14:49:44.113 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 22625 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci.ko 2024-03-31T14:49:44.113 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 14737 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci_legacy.ko 2024-03-31T14:49:44.113 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 10825 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_mbox.ko 2024-03-31T14:49:44.114 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 10745 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_power_floor.ko 2024-03-31T14:49:44.114 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 12889 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rapl.ko 2024-03-31T14:49:44.114 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 36953 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rfim.ko 2024-03-31T14:49:44.114 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 14897 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_hint.ko 2024-03-31T14:49:44.114 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 11457 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_req.ko 2024-03-31T14:49:44.114 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 30801 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/intel_powerclamp.ko 2024-03-31T14:49:44.114 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 18825 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/intel_soc_dts_iosf.ko 2024-03-31T14:49:44.114 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 11409 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/intel_soc_dts_thermal.ko 2024-03-31T14:49:44.114 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 26985 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/x86_pkg_temp_thermal.ko 2024-03-31T14:49:44.114 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/ 2024-03-31T14:49:44.114 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/ipwireless/ 2024-03-31T14:49:44.114 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 78329 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/ipwireless/ipwireless.ko 2024-03-31T14:49:44.114 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 57769 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/moxa.ko 2024-03-31T14:49:44.114 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 50857 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/mxser.ko 2024-03-31T14:49:44.115 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 108481 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/n_gsm.ko 2024-03-31T14:49:44.115 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 31865 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/n_hdlc.ko 2024-03-31T14:49:44.115 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 67833 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/nozomi.ko 2024-03-31T14:49:44.115 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/serial/ 2024-03-31T14:49:44.115 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/serial/8250/ 2024-03-31T14:49:44.115 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 34785 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/serial/8250/8250_dw.ko 2024-03-31T14:49:44.115 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 59393 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/serial/8250/serial_cs.ko 2024-03-31T14:49:44.115 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 17969 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/serial/altera_jtaguart.ko 2024-03-31T14:49:44.115 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 22473 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/serial/altera_uart.ko 2024-03-31T14:49:44.115 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 9329 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/serial/arc_uart.ko 2024-03-31T14:49:44.115 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 72545 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/serial/fsl_lpuart.ko 2024-03-31T14:49:44.115 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/serial/jsm/ 2024-03-31T14:49:44.115 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 101969 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/serial/jsm/jsm.ko 2024-03-31T14:49:44.115 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 28689 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/serial/rp2.ko 2024-03-31T14:49:44.116 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 53169 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/serial/sc16is7xx.ko 2024-03-31T14:49:44.116 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ufs/ 2024-03-31T14:49:44.116 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ufs/core/ 2024-03-31T14:49:44.116 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 545377 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ufs/core/ufshcd-core.ko 2024-03-31T14:49:44.116 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ufs/host/ 2024-03-31T14:49:44.116 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 28825 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ufs/host/ufshcd-pci.ko 2024-03-31T14:49:44.116 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 14505 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ufs/host/ufshcd-pltfrm.ko 2024-03-31T14:49:44.116 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/uio/ 2024-03-31T14:49:44.116 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 44465 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/uio/uio.ko 2024-03-31T14:49:44.116 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 12145 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/uio/uio_aec.ko 2024-03-31T14:49:44.116 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 11193 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/uio/uio_cif.ko 2024-03-31T14:49:44.116 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 18225 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/uio/uio_dmem_genirq.ko 2024-03-31T14:49:44.116 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 12833 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/uio/uio_mf624.ko 2024-03-31T14:49:44.116 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 12081 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/uio/uio_netx.ko 2024-03-31T14:49:44.116 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 10497 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/uio/uio_pci_generic.ko 2024-03-31T14:49:44.117 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 15953 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/uio/uio_pdrv_genirq.ko 2024-03-31T14:49:44.117 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 13537 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/uio/uio_sercos3.ko 2024-03-31T14:49:44.117 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/ 2024-03-31T14:49:44.117 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/c67x00/ 2024-03-31T14:49:44.117 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 73201 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/c67x00/c67x00.ko 2024-03-31T14:49:44.117 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/chipidea/ 2024-03-31T14:49:44.117 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 145753 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/chipidea/ci_hdrc.ko 2024-03-31T14:49:44.117 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 18401 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/chipidea/ci_hdrc_msm.ko 2024-03-31T14:49:44.117 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 13073 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/chipidea/ci_hdrc_npcm.ko 2024-03-31T14:49:44.117 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 12609 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/chipidea/ci_hdrc_pci.ko 2024-03-31T14:49:44.117 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 13113 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/chipidea/ci_hdrc_usb2.ko 2024-03-31T14:49:44.117 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/class/ 2024-03-31T14:49:44.117 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 87001 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/class/cdc-acm.ko 2024-03-31T14:49:44.117 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 51249 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/class/cdc-wdm.ko 2024-03-31T14:49:44.118 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 47209 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/class/usblp.ko 2024-03-31T14:49:44.118 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 89761 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/class/usbtmc.ko 2024-03-31T14:49:44.118 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/common/ 2024-03-31T14:49:44.118 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 22857 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/common/ulpi.ko 2024-03-31T14:49:44.118 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/dwc2/ 2024-03-31T14:49:44.118 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 11433 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/dwc2/dwc2_pci.ko 2024-03-31T14:49:44.118 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/dwc3/ 2024-03-31T14:49:44.118 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 11985 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/dwc3/dwc3-haps.ko 2024-03-31T14:49:44.118 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 30209 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/dwc3/dwc3-pci.ko 2024-03-31T14:49:44.118 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 452785 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/dwc3/dwc3.ko 2024-03-31T14:49:44.118 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/gadget/ 2024-03-31T14:49:44.118 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/gadget/function/ 2024-03-31T14:49:44.118 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 60321 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/gadget/function/u_ether.ko 2024-03-31T14:49:44.118 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 51785 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/gadget/function/u_serial.ko 2024-03-31T14:49:44.119 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 28737 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/gadget/function/usb_f_acm.ko 2024-03-31T14:49:44.119 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 42345 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/gadget/function/usb_f_ecm.ko 2024-03-31T14:49:44.119 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 27849 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/gadget/function/usb_f_ecm_subset.ko 2024-03-31T14:49:44.119 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 36409 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/gadget/function/usb_f_eem.ko 2024-03-31T14:49:44.119 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 92401 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/gadget/function/usb_f_fs.ko 2024-03-31T14:49:44.119 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 50937 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/gadget/function/usb_f_hid.ko 2024-03-31T14:49:44.119 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 122241 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/gadget/function/usb_f_mass_storage.ko 2024-03-31T14:49:44.119 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 59857 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/gadget/function/usb_f_ncm.ko 2024-03-31T14:49:44.119 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 22081 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/gadget/function/usb_f_obex.ko 2024-03-31T14:49:44.119 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 31161 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/gadget/function/usb_f_phonet.ko 2024-03-31T14:49:44.119 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 57625 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/gadget/function/usb_f_printer.ko 2024-03-31T14:49:44.119 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 85505 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/gadget/function/usb_f_rndis.ko 2024-03-31T14:49:44.119 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 19169 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/gadget/function/usb_f_serial.ko 2024-03-31T14:49:44.119 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 58009 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/gadget/function/usb_f_ss_lb.ko 2024-03-31T14:49:44.119 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 67345 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/gadget/function/usb_f_tcm.ko 2024-03-31T14:49:44.120 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/gadget/legacy/ 2024-03-31T14:49:44.120 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 21169 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/gadget/legacy/g_acm_ms.ko 2024-03-31T14:49:44.120 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 17961 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/gadget/legacy/g_cdc.ko 2024-03-31T14:49:44.120 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 18369 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/gadget/legacy/g_dbgp.ko 2024-03-31T14:49:44.120 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 21561 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/gadget/legacy/g_ether.ko 2024-03-31T14:49:44.120 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 28305 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/gadget/legacy/g_ffs.ko 2024-03-31T14:49:44.120 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 18913 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/gadget/legacy/g_hid.ko 2024-03-31T14:49:44.120 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 20305 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/gadget/legacy/g_mass_storage.ko 2024-03-31T14:49:44.120 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 16881 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/gadget/legacy/g_ncm.ko 2024-03-31T14:49:44.120 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 31977 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/gadget/legacy/g_nokia.ko 2024-03-31T14:49:44.120 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 16689 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/gadget/legacy/g_printer.ko 2024-03-31T14:49:44.120 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 19297 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/gadget/legacy/g_serial.ko 2024-03-31T14:49:44.120 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 27105 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/gadget/legacy/g_zero.ko 2024-03-31T14:49:44.120 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 62361 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/gadget/legacy/gadgetfs.ko 2024-03-31T14:49:44.120 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 15097 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/gadget/legacy/tcm_usb_gadget.ko 2024-03-31T14:49:44.121 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 161193 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/gadget/libcomposite.ko 2024-03-31T14:49:44.121 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/gadget/udc/ 2024-03-31T14:49:44.121 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 17601 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/gadget/udc/amd5536udc_pci.ko 2024-03-31T14:49:44.121 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/gadget/udc/bdc/ 2024-03-31T14:49:44.121 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 157593 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/gadget/udc/bdc/bdc.ko 2024-03-31T14:49:44.121 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 35993 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/gadget/udc/goku_udc.ko 2024-03-31T14:49:44.121 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 52921 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/gadget/udc/gr_udc.ko 2024-03-31T14:49:44.121 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 69609 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/gadget/udc/mv_u3d_core.ko 2024-03-31T14:49:44.121 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 55945 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/gadget/udc/mv_udc.ko 2024-03-31T14:49:44.121 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 75961 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/gadget/udc/net2272.ko 2024-03-31T14:49:44.121 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 74977 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/gadget/udc/net2280.ko 2024-03-31T14:49:44.121 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 73009 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/gadget/udc/pch_udc.ko 2024-03-31T14:49:44.121 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 65873 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/gadget/udc/pxa27x_udc.ko 2024-03-31T14:49:44.121 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 65993 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/gadget/udc/r8a66597-udc.ko 2024-03-31T14:49:44.122 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 81401 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/gadget/udc/snps_udc_core.ko 2024-03-31T14:49:44.122 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 156321 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/gadget/udc/udc-core.ko 2024-03-31T14:49:44.122 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/host/ 2024-03-31T14:49:44.122 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 11497 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/host/bcma-hcd.ko 2024-03-31T14:49:44.122 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 80009 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/host/isp116x-hcd.ko 2024-03-31T14:49:44.122 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 87249 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/host/oxu210hp-hcd.ko 2024-03-31T14:49:44.122 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 78769 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/host/r8a66597-hcd.ko 2024-03-31T14:49:44.122 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 51385 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/host/sl811-hcd.ko 2024-03-31T14:49:44.122 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 15497 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/host/sl811_cs.ko 2024-03-31T14:49:44.122 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 13217 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/host/ssb-hcd.ko 2024-03-31T14:49:44.122 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 26617 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/host/xhci-plat-hcd.ko 2024-03-31T14:49:44.122 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/image/ 2024-03-31T14:49:44.122 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 34489 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/image/mdc800.ko 2024-03-31T14:49:44.122 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 20985 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/image/microtek.ko 2024-03-31T14:49:44.122 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/ 2024-03-31T14:49:44.123 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 41625 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/adutux.ko 2024-03-31T14:49:44.123 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 20609 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/appledisplay.ko 2024-03-31T14:49:44.123 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 17617 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/cypress_cy7c63.ko 2024-03-31T14:49:44.123 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 22953 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/cytherm.ko 2024-03-31T14:49:44.123 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 13633 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/ehset.ko 2024-03-31T14:49:44.123 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 14961 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/emi26.ko 2024-03-31T14:49:44.123 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 15889 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/emi62.ko 2024-03-31T14:49:44.123 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 11337 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/ezusb.ko 2024-03-31T14:49:44.123 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 19137 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/idmouse.ko 2024-03-31T14:49:44.123 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 36449 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/iowarrior.ko 2024-03-31T14:49:44.123 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 11489 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/isight_firmware.ko 2024-03-31T14:49:44.123 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 36433 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/ldusb.ko 2024-03-31T14:49:44.123 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 36097 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/legousbtower.ko 2024-03-31T14:49:44.123 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 27697 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/lvstest.ko 2024-03-31T14:49:44.124 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/sisusbvga/ 2024-03-31T14:49:44.124 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 64737 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/sisusbvga/sisusbvga.ko 2024-03-31T14:49:44.124 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 12833 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/trancevibrator.ko 2024-03-31T14:49:44.124 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 20377 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/usb3503.ko 2024-03-31T14:49:44.124 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 25049 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/usblcd.ko 2024-03-31T14:49:44.124 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 22649 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/usbsevseg.ko 2024-03-31T14:49:44.124 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 70713 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/usbtest.ko 2024-03-31T14:49:44.124 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 41809 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/uss720.ko 2024-03-31T14:49:44.124 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 27217 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/yurex.ko 2024-03-31T14:49:44.124 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/mon/ 2024-03-31T14:49:44.124 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 72801 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/mon/usbmon.ko 2024-03-31T14:49:44.124 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/musb/ 2024-03-31T14:49:44.124 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 301593 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/musb/musb_hdrc.ko 2024-03-31T14:49:44.124 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/phy/ 2024-03-31T14:49:44.125 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 20953 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/phy/phy-generic.ko 2024-03-31T14:49:44.125 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 20065 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/phy/phy-gpio-vbus-usb.ko 2024-03-31T14:49:44.125 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 12153 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/phy/phy-isp1301.ko 2024-03-31T14:49:44.125 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 21641 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/phy/phy-tahvo.ko 2024-03-31T14:49:44.125 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/ 2024-03-31T14:49:44.125 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 13113 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/aircable.ko 2024-03-31T14:49:44.125 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 33233 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/ark3116.ko 2024-03-31T14:49:44.125 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 26529 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/belkin_sa.ko 2024-03-31T14:49:44.125 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 38009 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/ch341.ko 2024-03-31T14:49:44.125 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 70273 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/cp210x.ko 2024-03-31T14:49:44.125 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 30953 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/cyberjack.ko 2024-03-31T14:49:44.125 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 58113 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/cypress_m8.ko 2024-03-31T14:49:44.125 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 49713 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/digi_acceleport.ko 2024-03-31T14:49:44.125 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 9521 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/empeg.ko 2024-03-31T14:49:44.126 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 44177 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/f81232.ko 2024-03-31T14:49:44.126 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 175097 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/ftdi_sio.ko 2024-03-31T14:49:44.126 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 53009 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/garmin_gps.ko 2024-03-31T14:49:44.126 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 136977 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/io_edgeport.ko 2024-03-31T14:49:44.126 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 124737 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/io_ti.ko 2024-03-31T14:49:44.126 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 67673 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/ipaq.ko 2024-03-31T14:49:44.126 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 19065 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/ipw.ko 2024-03-31T14:49:44.126 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 26801 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/ir-usb.ko 2024-03-31T14:49:44.126 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 58129 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/iuu_phoenix.ko 2024-03-31T14:49:44.126 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 107425 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/keyspan.ko 2024-03-31T14:49:44.126 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 35513 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/keyspan_pda.ko 2024-03-31T14:49:44.126 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 25561 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/kl5kusb105.ko 2024-03-31T14:49:44.126 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 33585 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/kobil_sct.ko 2024-03-31T14:49:44.126 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 32857 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/mct_u232.ko 2024-03-31T14:49:44.127 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 21929 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/metro-usb.ko 2024-03-31T14:49:44.127 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 66353 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/mos7720.ko 2024-03-31T14:49:44.127 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 78121 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/mos7840.ko 2024-03-31T14:49:44.127 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 56241 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/mxuport.ko 2024-03-31T14:49:44.127 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 13929 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/navman.ko 2024-03-31T14:49:44.127 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 11913 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/omninet.ko 2024-03-31T14:49:44.127 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 24537 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/opticon.ko 2024-03-31T14:49:44.127 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 191817 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/option.ko 2024-03-31T14:49:44.127 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 38977 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/oti6858.ko 2024-03-31T14:49:44.127 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 54257 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/pl2303.ko 2024-03-31T14:49:44.127 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 9641 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/qcaux.ko 2024-03-31T14:49:44.127 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 38017 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/qcserial.ko 2024-03-31T14:49:44.127 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 34225 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/quatech2.ko 2024-03-31T14:49:44.127 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 15289 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/safe_serial.ko 2024-03-31T14:49:44.128 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 51529 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/sierra.ko 2024-03-31T14:49:44.128 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 21577 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/spcp8x5.ko 2024-03-31T14:49:44.128 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 29897 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/ssu100.ko 2024-03-31T14:49:44.128 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 17161 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/symbolserial.ko 2024-03-31T14:49:44.128 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 61385 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/ti_usb_3410_5052.ko 2024-03-31T14:49:44.128 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 23617 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/usb-serial-simple.ko 2024-03-31T14:49:44.128 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 10137 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/usb_debug.ko 2024-03-31T14:49:44.128 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 33105 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/usb_wwan.ko 2024-03-31T14:49:44.128 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 125513 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/usbserial.ko 2024-03-31T14:49:44.128 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 28657 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/visor.ko 2024-03-31T14:49:44.128 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 37305 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/whiteheat.ko 2024-03-31T14:49:44.128 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 10433 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/wishbone-serial.ko 2024-03-31T14:49:44.128 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 9193 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/xsens_mt.ko 2024-03-31T14:49:44.128 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ 2024-03-31T14:49:44.128 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 52753 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/uas.ko 2024-03-31T14:49:44.129 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 25849 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-alauda.ko 2024-03-31T14:49:44.129 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 12985 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-cypress.ko 2024-03-31T14:49:44.129 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 21881 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-datafab.ko 2024-03-31T14:49:44.129 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 33393 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-eneub6250.ko 2024-03-31T14:49:44.129 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 13649 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-freecom.ko 2024-03-31T14:49:44.129 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 19513 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-isd200.ko 2024-03-31T14:49:44.129 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 19201 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-jumpshot.ko 2024-03-31T14:49:44.129 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 13553 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-karma.ko 2024-03-31T14:49:44.129 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 17737 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-onetouch.ko 2024-03-31T14:49:44.129 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 28337 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-realtek.ko 2024-03-31T14:49:44.129 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 28801 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-sddr09.ko 2024-03-31T14:49:44.129 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 19417 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-sddr55.ko 2024-03-31T14:49:44.129 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 35241 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-usbat.ko 2024-03-31T14:49:44.129 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 180281 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/usb-storage.ko 2024-03-31T14:49:44.129 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/usbip/ 2024-03-31T14:49:44.130 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 67385 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/usbip/usbip-core.ko 2024-03-31T14:49:44.130 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 71001 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/usbip/usbip-host.ko 2024-03-31T14:49:44.130 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 103977 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/usbip/vhci-hcd.ko 2024-03-31T14:49:44.130 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vfio/ 2024-03-31T14:49:44.130 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vfio/pci/ 2024-03-31T14:49:44.130 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 162841 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vfio/pci/vfio-pci-core.ko 2024-03-31T14:49:44.130 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 29377 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vfio/pci/vfio-pci.ko 2024-03-31T14:49:44.130 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 103649 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vfio/vfio.ko 2024-03-31T14:49:44.130 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 76433 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vfio/vfio_iommu_type1.ko 2024-03-31T14:49:44.130 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vhost/ 2024-03-31T14:49:44.130 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 113657 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vhost/vhost.ko 2024-03-31T14:49:44.130 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 16905 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vhost/vhost_iotlb.ko 2024-03-31T14:49:44.130 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 57993 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vhost/vhost_net.ko 2024-03-31T14:49:44.130 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 89049 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vhost/vhost_scsi.ko 2024-03-31T14:49:44.131 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/video/ 2024-03-31T14:49:44.131 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/video/backlight/ 2024-03-31T14:49:44.131 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 15313 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/video/backlight/88pm860x_bl.ko 2024-03-31T14:49:44.131 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 12289 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/video/backlight/aat2870_bl.ko 2024-03-31T14:49:44.131 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 23201 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/video/backlight/adp5520_bl.ko 2024-03-31T14:49:44.131 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 32505 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/video/backlight/adp8860_bl.ko 2024-03-31T14:49:44.131 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 39889 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/video/backlight/adp8870_bl.ko 2024-03-31T14:49:44.131 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 16825 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/video/backlight/apple_bl.ko 2024-03-31T14:49:44.131 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 16673 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/video/backlight/as3711_bl.ko 2024-03-31T14:49:44.131 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 11937 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/video/backlight/bd6107.ko 2024-03-31T14:49:44.131 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 11305 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/video/backlight/da903x_bl.ko 2024-03-31T14:49:44.131 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 11169 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/video/backlight/da9052_bl.ko 2024-03-31T14:49:44.131 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 11649 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/video/backlight/gpio_backlight.ko 2024-03-31T14:49:44.131 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 14665 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/video/backlight/kb3886_bl.ko 2024-03-31T14:49:44.132 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 21921 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/video/backlight/lcd.ko 2024-03-31T14:49:44.132 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 23553 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/video/backlight/lm3533_bl.ko 2024-03-31T14:49:44.132 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 19233 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/video/backlight/lm3639_bl.ko 2024-03-31T14:49:44.132 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 11329 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/video/backlight/lv5207lp.ko 2024-03-31T14:49:44.132 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 14241 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/video/backlight/max8925_bl.ko 2024-03-31T14:49:44.132 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 11209 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/video/backlight/pandora_bl.ko 2024-03-31T14:49:44.132 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 10753 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/video/backlight/pcf50633-backlight.ko 2024-03-31T14:49:44.132 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 10793 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/video/backlight/platform_lcd.ko 2024-03-31T14:49:44.132 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 12617 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/video/backlight/wm831x_bl.ko 2024-03-31T14:49:44.132 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/ 2024-03-31T14:49:44.132 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/masters/ 2024-03-31T14:49:44.132 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 18689 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/masters/ds2482.ko 2024-03-31T14:49:44.132 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 39113 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/masters/ds2490.ko 2024-03-31T14:49:44.132 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 11161 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/masters/matrox_w1.ko 2024-03-31T14:49:44.132 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 11265 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/masters/w1-gpio.ko 2024-03-31T14:49:44.133 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/ 2024-03-31T14:49:44.133 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 12001 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_ds2406.ko 2024-03-31T14:49:44.133 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 22641 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_ds2408.ko 2024-03-31T14:49:44.133 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 15009 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_ds2413.ko 2024-03-31T14:49:44.133 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 10721 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_ds2423.ko 2024-03-31T14:49:44.133 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 14049 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_ds2431.ko 2024-03-31T14:49:44.133 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 13809 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_ds2433.ko 2024-03-31T14:49:44.133 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 13945 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_ds2780.ko 2024-03-31T14:49:44.133 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 13945 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_ds2781.ko 2024-03-31T14:49:44.133 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 17817 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_ds28e04.ko 2024-03-31T14:49:44.133 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 7289 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_smem.ko 2024-03-31T14:49:44.133 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 56329 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_therm.ko 2024-03-31T14:49:44.133 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 103025 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/wire.ko 2024-03-31T14:49:44.133 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/ 2024-03-31T14:49:44.134 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 19305 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/acquirewdt.ko 2024-03-31T14:49:44.134 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 20537 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/advantechwdt.ko 2024-03-31T14:49:44.134 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 18457 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/alim1535_wdt.ko 2024-03-31T14:49:44.134 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 22865 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/alim7101_wdt.ko 2024-03-31T14:49:44.134 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 19017 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/cpu5wdt.ko 2024-03-31T14:49:44.134 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 13513 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/da9052_wdt.ko 2024-03-31T14:49:44.134 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 12673 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/da9055_wdt.ko 2024-03-31T14:49:44.134 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 17305 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/da9063_wdt.ko 2024-03-31T14:49:44.134 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 20073 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/dw_wdt.ko 2024-03-31T14:49:44.134 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 20161 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/eurotechwdt.ko 2024-03-31T14:49:44.134 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 23761 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/f71808e_wdt.ko 2024-03-31T14:49:44.134 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 27385 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/hpwdt.ko 2024-03-31T14:49:44.134 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 15481 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/i6300esb.ko 2024-03-31T14:49:44.134 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 10721 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/iTCO_vendor_support.ko 2024-03-31T14:49:44.135 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 21817 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/iTCO_wdt.ko 2024-03-31T14:49:44.135 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 19633 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/ib700wdt.ko 2024-03-31T14:49:44.135 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 18593 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/ibmasr.ko 2024-03-31T14:49:44.135 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 19609 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/ie6xx_wdt.ko 2024-03-31T14:49:44.135 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 21121 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/it8712f_wdt.ko 2024-03-31T14:49:44.135 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 13993 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/it87_wdt.ko 2024-03-31T14:49:44.135 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 21385 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/kempld_wdt.ko 2024-03-31T14:49:44.135 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 18521 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/machzwd.ko 2024-03-31T14:49:44.135 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 14617 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/mena21_wdt.ko 2024-03-31T14:49:44.135 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 13113 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/menf21bmc_wdt.ko 2024-03-31T14:49:44.135 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 24577 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/nv_tco.ko 2024-03-31T14:49:44.135 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 17217 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/of_xilinx_wdt.ko 2024-03-31T14:49:44.135 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 23825 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/pc87413_wdt.ko 2024-03-31T14:49:44.135 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 39913 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/pcwd_pci.ko 2024-03-31T14:49:44.135 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 31729 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/pcwd_usb.ko 2024-03-31T14:49:44.136 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 12969 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/retu_wdt.ko 2024-03-31T14:49:44.136 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 20977 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/sbc60xxwdt.ko 2024-03-31T14:49:44.136 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 14025 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/sbc_epx_c3.ko 2024-03-31T14:49:44.136 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 17217 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/sbc_fitpc2_wdt.ko 2024-03-31T14:49:44.136 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 24625 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/sc1200wdt.ko 2024-03-31T14:49:44.136 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 23217 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/sch311x_wdt.ko 2024-03-31T14:49:44.136 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 21473 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/smsc37b787_wdt.ko 2024-03-31T14:49:44.136 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 17177 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/softdog.ko 2024-03-31T14:49:44.136 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 22169 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/sp5100_tco.ko 2024-03-31T14:49:44.136 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 12713 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/twl4030_wdt.ko 2024-03-31T14:49:44.136 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 17553 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/via_wdt.ko 2024-03-31T14:49:44.136 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 20233 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/w83627hf_wdt.ko 2024-03-31T14:49:44.136 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 21193 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/w83877f_wdt.ko 2024-03-31T14:49:44.136 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 28561 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/w83977f_wdt.ko 2024-03-31T14:49:44.136 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 19505 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/wafer5823wdt.ko 2024-03-31T14:49:44.137 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 27993 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/wdt_pci.ko 2024-03-31T14:49:44.137 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 16153 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/wm831x_wdt.ko 2024-03-31T14:49:44.137 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 13681 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/wm8350_wdt.ko 2024-03-31T14:49:44.137 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 15601 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/xen_wdt.ko 2024-03-31T14:49:44.137 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/xen/ 2024-03-31T14:49:44.137 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 24625 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/xen/xen-evtchn.ko 2024-03-31T14:49:44.137 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 26665 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/xen/xen-gntalloc.ko 2024-03-31T14:49:44.137 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 47761 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/xen/xen-gntdev.ko 2024-03-31T14:49:44.137 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/xen/xen-pciback/ 2024-03-31T14:49:44.137 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 176337 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/xen/xen-pciback/xen-pciback.ko 2024-03-31T14:49:44.137 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 40081 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/xen/xen-privcmd.ko 2024-03-31T14:49:44.137 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 71145 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/xen/xen-scsiback.ko 2024-03-31T14:49:44.137 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/xen/xenfs/ 2024-03-31T14:49:44.137 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 19329 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/xen/xenfs/xenfs.ko 2024-03-31T14:49:44.138 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ 2024-03-31T14:49:44.138 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/9p/ 2024-03-31T14:49:44.138 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 158217 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/9p/9p.ko 2024-03-31T14:49:44.138 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/adfs/ 2024-03-31T14:49:44.138 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 61681 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/adfs/adfs.ko 2024-03-31T14:49:44.138 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/affs/ 2024-03-31T14:49:44.138 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 163369 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/affs/affs.ko 2024-03-31T14:49:44.138 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/afs/ 2024-03-31T14:49:44.138 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 1019049 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/afs/kafs.ko 2024-03-31T14:49:44.138 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/autofs/ 2024-03-31T14:49:44.138 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 103185 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/autofs/autofs4.ko 2024-03-31T14:49:44.138 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/befs/ 2024-03-31T14:49:44.138 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 64585 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/befs/befs.ko 2024-03-31T14:49:44.138 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/bfs/ 2024-03-31T14:49:44.139 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 41001 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/bfs/bfs.ko 2024-03-31T14:49:44.139 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 38641 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/binfmt_misc.ko 2024-03-31T14:49:44.139 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/btrfs/ 2024-03-31T14:49:44.139 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 3487249 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/btrfs/btrfs.ko 2024-03-31T14:49:44.139 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/cachefiles/ 2024-03-31T14:49:44.139 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 305841 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/cachefiles/cachefiles.ko 2024-03-31T14:49:44.139 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ceph/ 2024-03-31T14:49:44.139 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 1335577 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ceph/ceph.ko 2024-03-31T14:49:44.139 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/coda/ 2024-03-31T14:49:44.139 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 97713 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/coda/coda.ko 2024-03-31T14:49:44.139 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/configfs/ 2024-03-31T14:49:44.139 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 108545 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/configfs/configfs.ko 2024-03-31T14:49:44.139 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/cramfs/ 2024-03-31T14:49:44.139 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 29577 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/cramfs/cramfs.ko 2024-03-31T14:49:44.139 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/dlm/ 2024-03-31T14:49:44.140 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 644161 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/dlm/dlm.ko 2024-03-31T14:49:44.140 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/efs/ 2024-03-31T14:49:44.140 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 36697 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/efs/efs.ko 2024-03-31T14:49:44.140 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/f2fs/ 2024-03-31T14:49:44.140 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 1527561 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/f2fs/f2fs.ko 2024-03-31T14:49:44.140 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/fat/ 2024-03-31T14:49:44.140 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 27833 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/fat/msdos.ko 2024-03-31T14:49:44.140 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/freevxfs/ 2024-03-31T14:49:44.140 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 42513 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/freevxfs/freevxfs.ko 2024-03-31T14:49:44.140 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/fuse/ 2024-03-31T14:49:44.140 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 24289 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/fuse/cuse.ko 2024-03-31T14:49:44.140 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/gfs2/ 2024-03-31T14:49:44.140 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 1231233 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/gfs2/gfs2.ko 2024-03-31T14:49:44.140 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/hfs/ 2024-03-31T14:49:44.141 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 141625 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/hfs/hfs.ko 2024-03-31T14:49:44.141 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/hfsplus/ 2024-03-31T14:49:44.141 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 223177 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/hfsplus/hfsplus.ko 2024-03-31T14:49:44.141 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/hpfs/ 2024-03-31T14:49:44.141 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 193481 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/hpfs/hpfs.ko 2024-03-31T14:49:44.141 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/isofs/ 2024-03-31T14:49:44.141 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 74689 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/isofs/isofs.ko 2024-03-31T14:49:44.141 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/jfs/ 2024-03-31T14:49:44.141 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 420329 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/jfs/jfs.ko 2024-03-31T14:49:44.141 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/lockd/ 2024-03-31T14:49:44.141 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 281617 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/lockd/lockd.ko 2024-03-31T14:49:44.141 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/minix/ 2024-03-31T14:49:44.141 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 88897 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/minix/minix.ko 2024-03-31T14:49:44.141 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/netfs/ 2024-03-31T14:49:44.142 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 490409 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/netfs/netfs.ko 2024-03-31T14:49:44.142 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs/ 2024-03-31T14:49:44.142 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs/blocklayout/ 2024-03-31T14:49:44.142 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 81433 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs/blocklayout/blocklayoutdriver.ko 2024-03-31T14:49:44.142 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs/filelayout/ 2024-03-31T14:49:44.142 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 59337 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs/filelayout/nfs_layout_nfsv41_files.ko 2024-03-31T14:49:44.142 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs/flexfilelayout/ 2024-03-31T14:49:44.142 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 117361 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs/flexfilelayout/nfs_layout_flexfiles.ko 2024-03-31T14:49:44.142 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 1058273 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs/nfs.ko 2024-03-31T14:49:44.142 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 66345 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs/nfsv2.ko 2024-03-31T14:49:44.142 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 120169 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs/nfsv3.ko 2024-03-31T14:49:44.142 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 2041705 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs/nfsv4.ko 2024-03-31T14:49:44.142 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs_common/ 2024-03-31T14:49:44.142 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 16833 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs_common/grace.ko 2024-03-31T14:49:44.143 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 16049 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs_common/nfs_acl.ko 2024-03-31T14:49:44.143 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfsd/ 2024-03-31T14:49:44.143 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 1511641 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfsd/nfsd.ko 2024-03-31T14:49:44.143 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nilfs2/ 2024-03-31T14:49:44.143 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 582409 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nilfs2/nilfs2.ko 2024-03-31T14:49:44.143 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/ 2024-03-31T14:49:44.143 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 13689 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-celtic.ko 2024-03-31T14:49:44.143 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 13065 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-centeuro.ko 2024-03-31T14:49:44.143 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 13689 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-croatian.ko 2024-03-31T14:49:44.143 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 12761 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-cyrillic.ko 2024-03-31T14:49:44.143 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 13377 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-gaelic.ko 2024-03-31T14:49:44.143 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 12753 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-greek.ko 2024-03-31T14:49:44.143 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 13689 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-iceland.ko 2024-03-31T14:49:44.143 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 13065 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-inuit.ko 2024-03-31T14:49:44.144 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 13993 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-roman.ko 2024-03-31T14:49:44.144 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 13689 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-romanian.ko 2024-03-31T14:49:44.144 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 13689 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-turkish.ko 2024-03-31T14:49:44.144 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 11193 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_ascii.ko 2024-03-31T14:49:44.144 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 12433 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp1250.ko 2024-03-31T14:49:44.144 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 12129 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp1251.ko 2024-03-31T14:49:44.144 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 12817 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp1255.ko 2024-03-31T14:49:44.144 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 12433 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp737.ko 2024-03-31T14:49:44.144 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 12433 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp775.ko 2024-03-31T14:49:44.144 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 12121 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp850.ko 2024-03-31T14:49:44.144 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 12121 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp852.ko 2024-03-31T14:49:44.144 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 12121 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp855.ko 2024-03-31T14:49:44.144 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 11809 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp857.ko 2024-03-31T14:49:44.144 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 12745 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp860.ko 2024-03-31T14:49:44.144 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 13057 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp861.ko 2024-03-31T14:49:44.145 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 13369 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp862.ko 2024-03-31T14:49:44.145 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 13057 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp863.ko 2024-03-31T14:49:44.145 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 12745 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp864.ko 2024-03-31T14:49:44.145 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 13057 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp865.ko 2024-03-31T14:49:44.145 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 12433 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp866.ko 2024-03-31T14:49:44.145 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 12121 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp869.ko 2024-03-31T14:49:44.145 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 11881 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp874.ko 2024-03-31T14:49:44.145 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 94033 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp932.ko 2024-03-31T14:49:44.145 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 142569 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp936.ko 2024-03-31T14:49:44.145 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 165641 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp949.ko 2024-03-31T14:49:44.145 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 118017 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp950.ko 2024-03-31T14:49:44.145 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 12361 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_euc-jp.ko 2024-03-31T14:49:44.145 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 11201 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-1.ko 2024-03-31T14:49:44.145 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 11825 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-13.ko 2024-03-31T14:49:44.146 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 11825 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-14.ko 2024-03-31T14:49:44.146 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 11825 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-15.ko 2024-03-31T14:49:44.146 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 11817 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-2.ko 2024-03-31T14:49:44.146 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 11817 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-3.ko 2024-03-31T14:49:44.146 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 11817 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-4.ko 2024-03-31T14:49:44.146 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 11817 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-5.ko 2024-03-31T14:49:44.146 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 11513 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-6.ko 2024-03-31T14:49:44.146 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 12129 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-7.ko 2024-03-31T14:49:44.146 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 11513 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-9.ko 2024-03-31T14:49:44.146 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 12433 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_koi8-r.ko 2024-03-31T14:49:44.146 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 7921 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_koi8-ru.ko 2024-03-31T14:49:44.146 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 12433 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_koi8-u.ko 2024-03-31T14:49:44.146 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 7465 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_ucs2_utils.ko 2024-03-31T14:49:44.146 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 8249 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_utf8.ko 2024-03-31T14:49:44.147 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ntfs/ 2024-03-31T14:49:44.147 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 220457 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ntfs/ntfs.ko 2024-03-31T14:49:44.147 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2/ 2024-03-31T14:49:44.147 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2/cluster/ 2024-03-31T14:49:44.147 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 244633 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2/cluster/ocfs2_nodemanager.ko 2024-03-31T14:49:44.147 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2/dlm/ 2024-03-31T14:49:44.147 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 440633 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2/dlm/ocfs2_dlm.ko 2024-03-31T14:49:44.147 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2/dlmfs/ 2024-03-31T14:49:44.147 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 54161 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2/dlmfs/ocfs2_dlmfs.ko 2024-03-31T14:49:44.147 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 2590681 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2/ocfs2.ko 2024-03-31T14:49:44.147 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 17809 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2/ocfs2_stack_o2cb.ko 2024-03-31T14:49:44.147 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 31625 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2/ocfs2_stack_user.ko 2024-03-31T14:49:44.147 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 35729 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2/ocfs2_stackglue.ko 2024-03-31T14:49:44.147 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/omfs/ 2024-03-31T14:49:44.148 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 53681 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/omfs/omfs.ko 2024-03-31T14:49:44.148 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/overlayfs/ 2024-03-31T14:49:44.148 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 360897 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/overlayfs/overlay.ko 2024-03-31T14:49:44.148 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/pstore/ 2024-03-31T14:49:44.148 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 56929 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/pstore/ramoops.ko 2024-03-31T14:49:44.148 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/qnx4/ 2024-03-31T14:49:44.148 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 26889 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/qnx4/qnx4.ko 2024-03-31T14:49:44.148 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/qnx6/ 2024-03-31T14:49:44.148 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 46601 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/qnx6/qnx6.ko 2024-03-31T14:49:44.148 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/quota/ 2024-03-31T14:49:44.148 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 30777 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/quota/quota_tree.ko 2024-03-31T14:49:44.148 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 14201 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/quota/quota_v1.ko 2024-03-31T14:49:44.148 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 21921 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/quota/quota_v2.ko 2024-03-31T14:49:44.148 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/reiserfs/ 2024-03-31T14:49:44.148 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 532161 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/reiserfs/reiserfs.ko 2024-03-31T14:49:44.149 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/romfs/ 2024-03-31T14:49:44.149 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 28889 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/romfs/romfs.ko 2024-03-31T14:49:44.149 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/smb/ 2024-03-31T14:49:44.149 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/smb/client/ 2024-03-31T14:49:44.149 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 2712385 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/smb/client/cifs.ko 2024-03-31T14:49:44.149 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/smb/common/ 2024-03-31T14:49:44.149 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 7057 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/smb/common/cifs_arc4.ko 2024-03-31T14:49:44.149 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 11161 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/smb/common/cifs_md4.ko 2024-03-31T14:49:44.149 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/squashfs/ 2024-03-31T14:49:44.149 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 134705 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/squashfs/squashfs.ko 2024-03-31T14:49:44.149 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/sysv/ 2024-03-31T14:49:44.149 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 91281 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/sysv/sysv.ko 2024-03-31T14:49:44.149 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/udf/ 2024-03-31T14:49:44.149 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 248329 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/udf/udf.ko 2024-03-31T14:49:44.150 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ufs/ 2024-03-31T14:49:44.150 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 162737 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ufs/ufs.ko 2024-03-31T14:49:44.150 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/xfs/ 2024-03-31T14:49:44.150 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 4035417 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/xfs/xfs.ko 2024-03-31T14:49:44.150 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/kernel/ 2024-03-31T14:49:44.150 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/kernel/locking/ 2024-03-31T14:49:44.150 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 80481 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/kernel/locking/locktorture.ko 2024-03-31T14:49:44.150 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/kernel/time/ 2024-03-31T14:49:44.150 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 13633 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/kernel/time/test_udelay.ko 2024-03-31T14:49:44.150 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 60257 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/kernel/torture.ko 2024-03-31T14:49:44.150 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/ 2024-03-31T14:49:44.150 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 6729 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/crc-itu-t.ko 2024-03-31T14:49:44.150 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 6433 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/crc7.ko 2024-03-31T14:49:44.151 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 6929 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/crc8.ko 2024-03-31T14:49:44.151 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/crypto/ 2024-03-31T14:49:44.151 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 7001 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/crypto/libarc4.ko 2024-03-31T14:49:44.151 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 36985 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/crypto/libdes.ko 2024-03-31T14:49:44.151 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 13201 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/interval_tree_test.ko 2024-03-31T14:49:44.151 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 8321 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/libcrc32c.ko 2024-03-31T14:49:44.151 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 24849 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/lru_cache.ko 2024-03-31T14:49:44.151 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/lz4/ 2024-03-31T14:49:44.151 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 33697 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/lz4/lz4_compress.ko 2024-03-31T14:49:44.151 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 22529 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/lz4/lz4hc_compress.ko 2024-03-31T14:49:44.151 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/math/ 2024-03-31T14:49:44.151 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 6521 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/math/cordic.ko 2024-03-31T14:49:44.151 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 8113 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/memory-notifier-error-inject.ko 2024-03-31T14:49:44.151 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 10753 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/notifier-error-inject.ko 2024-03-31T14:49:44.151 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 13985 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/percpu_test.ko 2024-03-31T14:49:44.152 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 8193 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/pm-notifier-error-inject.ko 2024-03-31T14:49:44.152 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/raid6/ 2024-03-31T14:49:44.152 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 143257 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/raid6/raid6_pq.ko 2024-03-31T14:49:44.152 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 21009 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/rbtree_test.ko 2024-03-31T14:49:44.152 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/reed_solomon/ 2024-03-31T14:49:44.152 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 18649 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/reed_solomon/reed_solomon.ko 2024-03-31T14:49:44.152 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 38737 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/test-kstrtox.ko 2024-03-31T14:49:44.152 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 26209 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/test-string_helpers.ko 2024-03-31T14:49:44.152 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 4863217 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/test_bpf.ko 2024-03-31T14:49:44.152 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 48985 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/test_firmware.ko 2024-03-31T14:49:44.152 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 6273 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/test_module.ko 2024-03-31T14:49:44.152 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 20705 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/test_user_copy.ko 2024-03-31T14:49:44.152 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 9489 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/ts_bm.ko 2024-03-31T14:49:44.152 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 11025 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/ts_fsm.ko 2024-03-31T14:49:44.153 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 9529 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/ts_kmp.ko 2024-03-31T14:49:44.153 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/xz/ 2024-03-31T14:49:44.153 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 13441 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/xz/xz_dec_test.ko 2024-03-31T14:49:44.153 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/mm/ 2024-03-31T14:49:44.153 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 18121 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/mm/hwpoison-inject.ko 2024-03-31T14:49:44.153 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ 2024-03-31T14:49:44.153 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/6lowpan/ 2024-03-31T14:49:44.153 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 55753 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/6lowpan/6lowpan.ko 2024-03-31T14:49:44.153 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 6953 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/6lowpan/nhc_dest.ko 2024-03-31T14:49:44.153 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 6961 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/6lowpan/nhc_fragment.ko 2024-03-31T14:49:44.153 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 6953 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/6lowpan/nhc_hop.ko 2024-03-31T14:49:44.153 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 6889 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/6lowpan/nhc_ipv6.ko 2024-03-31T14:49:44.153 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 6961 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/6lowpan/nhc_mobility.ko 2024-03-31T14:49:44.153 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 6897 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/6lowpan/nhc_routing.ko 2024-03-31T14:49:44.154 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 15233 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/6lowpan/nhc_udp.ko 2024-03-31T14:49:44.154 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/802/ 2024-03-31T14:49:44.154 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 30401 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/802/garp.ko 2024-03-31T14:49:44.154 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 34745 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/802/mrp.ko 2024-03-31T14:49:44.154 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 8561 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/802/p8022.ko 2024-03-31T14:49:44.154 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 16297 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/802/psnap.ko 2024-03-31T14:49:44.154 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 11553 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/802/stp.ko 2024-03-31T14:49:44.154 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/8021q/ 2024-03-31T14:49:44.154 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 82865 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/8021q/8021q.ko 2024-03-31T14:49:44.154 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/9p/ 2024-03-31T14:49:44.154 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 198809 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/9p/9pnet.ko 2024-03-31T14:49:44.154 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 36137 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/9p/9pnet_fd.ko 2024-03-31T14:49:44.154 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 26241 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/9p/9pnet_rdma.ko 2024-03-31T14:49:44.154 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 32241 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/9p/9pnet_virtio.ko 2024-03-31T14:49:44.154 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/batman-adv/ 2024-03-31T14:49:44.155 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 667353 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/batman-adv/batman-adv.ko 2024-03-31T14:49:44.155 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/ 2024-03-31T14:49:44.155 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 67753 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/br_netfilter.ko 2024-03-31T14:49:44.155 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 790545 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/bridge.ko 2024-03-31T14:49:44.155 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ 2024-03-31T14:49:44.155 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 7553 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_802_3.ko 2024-03-31T14:49:44.155 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 12281 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_among.ko 2024-03-31T14:49:44.155 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 9073 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_arp.ko 2024-03-31T14:49:44.155 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 8969 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_arpreply.ko 2024-03-31T14:49:44.155 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 9529 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_dnat.ko 2024-03-31T14:49:44.155 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 8641 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_ip.ko 2024-03-31T14:49:44.155 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 9049 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_ip6.ko 2024-03-31T14:49:44.155 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 9721 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_limit.ko 2024-03-31T14:49:44.155 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 13185 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_log.ko 2024-03-31T14:49:44.156 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 7489 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_mark.ko 2024-03-31T14:49:44.156 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 7489 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_mark_m.ko 2024-03-31T14:49:44.156 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 8281 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_nflog.ko 2024-03-31T14:49:44.156 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 7433 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_pkttype.ko 2024-03-31T14:49:44.156 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 9489 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_redirect.ko 2024-03-31T14:49:44.156 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 8553 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_snat.ko 2024-03-31T14:49:44.156 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 8729 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_stp.ko 2024-03-31T14:49:44.156 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 12305 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_vlan.ko 2024-03-31T14:49:44.156 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 11537 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebtable_broute.ko 2024-03-31T14:49:44.156 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 9833 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebtable_filter.ko 2024-03-31T14:49:44.156 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 9809 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebtable_nat.ko 2024-03-31T14:49:44.156 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 55937 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebtables.ko 2024-03-31T14:49:44.156 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 13937 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/nft_meta_bridge.ko 2024-03-31T14:49:44.156 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 13841 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/nft_reject_bridge.ko 2024-03-31T14:49:44.157 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ceph/ 2024-03-31T14:49:44.157 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 1046689 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ceph/libceph.ko 2024-03-31T14:49:44.157 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/core/ 2024-03-31T14:49:44.157 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 125697 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/core/pktgen.ko 2024-03-31T14:49:44.157 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dccp/ 2024-03-31T14:49:44.157 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 194441 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dccp/dccp.ko 2024-03-31T14:49:44.157 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 8561 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dccp/dccp_diag.ko 2024-03-31T14:49:44.157 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 58737 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dccp/dccp_ipv4.ko 2024-03-31T14:49:44.157 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 61393 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dccp/dccp_ipv6.ko 2024-03-31T14:49:44.157 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dsa/ 2024-03-31T14:49:44.157 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 390033 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dsa/dsa_core.ko 2024-03-31T14:49:44.157 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 13353 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dsa/tag_brcm.ko 2024-03-31T14:49:44.157 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 12593 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dsa/tag_dsa.ko 2024-03-31T14:49:44.157 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 7737 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dsa/tag_none.ko 2024-03-31T14:49:44.157 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 8969 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dsa/tag_trailer.ko 2024-03-31T14:49:44.158 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/hsr/ 2024-03-31T14:49:44.158 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 102033 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/hsr/hsr.ko 2024-03-31T14:49:44.158 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ieee802154/ 2024-03-31T14:49:44.158 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ieee802154/6lowpan/ 2024-03-31T14:49:44.158 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 45481 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ieee802154/6lowpan/ieee802154_6lowpan.ko 2024-03-31T14:49:44.158 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 284665 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ieee802154/ieee802154.ko 2024-03-31T14:49:44.158 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 47873 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ieee802154/ieee802154_socket.ko 2024-03-31T14:49:44.158 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/ 2024-03-31T14:49:44.158 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 24777 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/ah4.ko 2024-03-31T14:49:44.158 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 39089 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/esp4.ko 2024-03-31T14:49:44.158 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 48233 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/fou.ko 2024-03-31T14:49:44.158 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 18025 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/gre.ko 2024-03-31T14:49:44.158 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 51209 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/inet_diag.ko 2024-03-31T14:49:44.158 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 61129 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/ip_gre.ko 2024-03-31T14:49:44.158 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 61505 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/ip_tunnel.ko 2024-03-31T14:49:44.159 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 34921 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/ip_vti.ko 2024-03-31T14:49:44.159 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 14129 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/ipcomp.ko 2024-03-31T14:49:44.159 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 29073 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/ipip.ko 2024-03-31T14:49:44.159 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/ 2024-03-31T14:49:44.159 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 38001 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/arp_tables.ko 2024-03-31T14:49:44.159 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 8769 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/arpt_mangle.ko 2024-03-31T14:49:44.159 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 10313 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/arptable_filter.ko 2024-03-31T14:49:44.159 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 42049 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/ip_tables.ko 2024-03-31T14:49:44.159 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 9961 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/ipt_ECN.ko 2024-03-31T14:49:44.159 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 9577 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/ipt_REJECT.ko 2024-03-31T14:49:44.159 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 14017 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/ipt_SYNPROXY.ko 2024-03-31T14:49:44.159 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 11081 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/ipt_ah.ko 2024-03-31T14:49:44.159 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 14777 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/ipt_rpfilter.ko 2024-03-31T14:49:44.159 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 11201 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/iptable_filter.ko 2024-03-31T14:49:44.160 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 11553 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/iptable_mangle.ko 2024-03-31T14:49:44.160 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 14921 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/iptable_nat.ko 2024-03-31T14:49:44.160 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 11553 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/iptable_raw.ko 2024-03-31T14:49:44.160 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 10313 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/iptable_security.ko 2024-03-31T14:49:44.160 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 12441 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/nf_defrag_ipv4.ko 2024-03-31T14:49:44.160 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 8713 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/nf_dup_ipv4.ko 2024-03-31T14:49:44.160 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 27521 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/nf_nat_h323.ko 2024-03-31T14:49:44.160 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 18193 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/nf_nat_pptp.ko 2024-03-31T14:49:44.160 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 13225 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/nf_nat_snmp_basic.ko 2024-03-31T14:49:44.160 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 21033 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/nf_reject_ipv4.ko 2024-03-31T14:49:44.160 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 9225 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/nf_socket_ipv4.ko 2024-03-31T14:49:44.160 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 15073 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/nf_tproxy_ipv4.ko 2024-03-31T14:49:44.160 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 9857 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/nft_dup_ipv4.ko 2024-03-31T14:49:44.160 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 17009 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/nft_fib_ipv4.ko 2024-03-31T14:49:44.161 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 8905 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/nft_reject_ipv4.ko 2024-03-31T14:49:44.161 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 12977 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_bic.ko 2024-03-31T14:49:44.161 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 13641 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_dctcp.ko 2024-03-31T14:49:44.161 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 15961 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_diag.ko 2024-03-31T14:49:44.161 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 9433 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_highspeed.ko 2024-03-31T14:49:44.161 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 12305 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_htcp.ko 2024-03-31T14:49:44.161 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 10345 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_hybla.ko 2024-03-31T14:49:44.161 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 11369 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_illinois.ko 2024-03-31T14:49:44.161 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 10337 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_lp.ko 2024-03-31T14:49:44.161 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 8225 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_scalable.ko 2024-03-31T14:49:44.161 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 14321 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_vegas.ko 2024-03-31T14:49:44.161 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 9993 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_veno.ko 2024-03-31T14:49:44.161 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 10849 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_westwood.ko 2024-03-31T14:49:44.161 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 10089 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_yeah.ko 2024-03-31T14:49:44.162 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 22625 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tunnel4.ko 2024-03-31T14:49:44.162 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 17761 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/udp_diag.ko 2024-03-31T14:49:44.162 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 45041 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/udp_tunnel.ko 2024-03-31T14:49:44.162 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 11377 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/xfrm4_tunnel.ko 2024-03-31T14:49:44.162 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/ 2024-03-31T14:49:44.162 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 26433 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/ah6.ko 2024-03-31T14:49:44.162 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 39721 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/esp6.ko 2024-03-31T14:49:44.162 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 14337 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/fou6.ko 2024-03-31T14:49:44.162 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 79921 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/ip6_gre.ko 2024-03-31T14:49:44.162 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 94161 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/ip6_tunnel.ko 2024-03-31T14:49:44.162 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 14249 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/ip6_udp_tunnel.ko 2024-03-31T14:49:44.162 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 54265 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/ip6_vti.ko 2024-03-31T14:49:44.162 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 13873 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/ipcomp6.ko 2024-03-31T14:49:44.163 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 19201 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/mip6.ko 2024-03-31T14:49:44.163 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ 2024-03-31T14:49:44.163 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 42545 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6_tables.ko 2024-03-31T14:49:44.163 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 11657 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6t_NPT.ko 2024-03-31T14:49:44.163 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 9649 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6t_REJECT.ko 2024-03-31T14:49:44.163 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 14033 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6t_SYNPROXY.ko 2024-03-31T14:49:44.163 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 14177 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6t_ah.ko 2024-03-31T14:49:44.163 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 7697 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6t_eui64.ko 2024-03-31T14:49:44.163 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 16105 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6t_frag.ko 2024-03-31T14:49:44.163 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 15937 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6t_hbh.ko 2024-03-31T14:49:44.163 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 8809 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6t_ipv6header.ko 2024-03-31T14:49:44.163 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 9801 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6t_mh.ko 2024-03-31T14:49:44.163 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 10833 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6t_rpfilter.ko 2024-03-31T14:49:44.163 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 10801 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6t_rt.ko 2024-03-31T14:49:44.164 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 11217 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6table_filter.ko 2024-03-31T14:49:44.164 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 11025 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6table_mangle.ko 2024-03-31T14:49:44.164 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 15017 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6table_nat.ko 2024-03-31T14:49:44.164 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 11569 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6table_raw.ko 2024-03-31T14:49:44.164 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 10329 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6table_security.ko 2024-03-31T14:49:44.164 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 39121 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/nf_defrag_ipv6.ko 2024-03-31T14:49:44.164 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 8345 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/nf_dup_ipv6.ko 2024-03-31T14:49:44.164 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 25257 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/nf_reject_ipv6.ko 2024-03-31T14:49:44.164 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 9697 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/nf_socket_ipv6.ko 2024-03-31T14:49:44.164 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 15289 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/nf_tproxy_ipv6.ko 2024-03-31T14:49:44.164 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 9529 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/nft_dup_ipv6.ko 2024-03-31T14:49:44.164 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 15921 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/nft_fib_ipv6.ko 2024-03-31T14:49:44.164 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 8905 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/nft_reject_ipv6.ko 2024-03-31T14:49:44.164 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 74873 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/sit.ko 2024-03-31T14:49:44.165 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 24145 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/tunnel6.ko 2024-03-31T14:49:44.165 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 23289 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/xfrm6_tunnel.ko 2024-03-31T14:49:44.165 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/key/ 2024-03-31T14:49:44.165 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 84393 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/key/af_key.ko 2024-03-31T14:49:44.165 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/l2tp/ 2024-03-31T14:49:44.165 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 125201 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/l2tp/l2tp_core.ko 2024-03-31T14:49:44.165 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 20073 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/l2tp/l2tp_debugfs.ko 2024-03-31T14:49:44.165 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 25809 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/l2tp/l2tp_eth.ko 2024-03-31T14:49:44.165 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 36489 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/l2tp/l2tp_ip.ko 2024-03-31T14:49:44.165 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 37369 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/l2tp/l2tp_ip6.ko 2024-03-31T14:49:44.165 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 33441 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/l2tp/l2tp_netlink.ko 2024-03-31T14:49:44.165 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 53225 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/l2tp/l2tp_ppp.ko 2024-03-31T14:49:44.165 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/lapb/ 2024-03-31T14:49:44.165 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 48097 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/lapb/lapb.ko 2024-03-31T14:49:44.166 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/llc/ 2024-03-31T14:49:44.166 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 22817 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/llc/llc.ko 2024-03-31T14:49:44.166 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 268449 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/llc/llc2.ko 2024-03-31T14:49:44.166 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/mac802154/ 2024-03-31T14:49:44.166 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 309361 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/mac802154/mac802154.ko 2024-03-31T14:49:44.166 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/mpls/ 2024-03-31T14:49:44.166 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 9297 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/mpls/mpls_gso.ko 2024-03-31T14:49:44.166 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ 2024-03-31T14:49:44.166 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ 2024-03-31T14:49:44.166 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 119689 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set.ko 2024-03-31T14:49:44.166 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 28609 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_bitmap_ip.ko 2024-03-31T14:49:44.166 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 27297 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_bitmap_ipmac.ko 2024-03-31T14:49:44.166 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 26721 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_bitmap_port.ko 2024-03-31T14:49:44.167 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 99985 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_ip.ko 2024-03-31T14:49:44.167 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 100273 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_ipmark.ko 2024-03-31T14:49:44.167 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 102545 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_ipport.ko 2024-03-31T14:49:44.167 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 102337 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_ipportip.ko 2024-03-31T14:49:44.167 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 111665 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_ipportnet.ko 2024-03-31T14:49:44.167 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 59185 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_mac.ko 2024-03-31T14:49:44.167 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 108337 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_net.ko 2024-03-31T14:49:44.167 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 112137 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_netiface.ko 2024-03-31T14:49:44.167 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 112401 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_netnet.ko 2024-03-31T14:49:44.167 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 110329 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_netport.ko 2024-03-31T14:49:44.167 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 113281 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_netportnet.ko 2024-03-31T14:49:44.167 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 30297 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_list_set.ko 2024-03-31T14:49:44.167 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ 2024-03-31T14:49:44.167 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 412433 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs.ko 2024-03-31T14:49:44.168 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 11985 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_dh.ko 2024-03-31T14:49:44.168 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 7649 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_fo.ko 2024-03-31T14:49:44.168 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 17673 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_ftp.ko 2024-03-31T14:49:44.168 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 17929 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_lblc.ko 2024-03-31T14:49:44.168 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 20761 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_lblcr.ko 2024-03-31T14:49:44.168 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 7649 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_lc.ko 2024-03-31T14:49:44.168 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 7713 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_nq.ko 2024-03-31T14:49:44.168 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 10633 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_pe_sip.ko 2024-03-31T14:49:44.168 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 9257 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_rr.ko 2024-03-31T14:49:44.168 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 7713 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_sed.ko 2024-03-31T14:49:44.168 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 13537 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_sh.ko 2024-03-31T14:49:44.168 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 7713 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_wlc.ko 2024-03-31T14:49:44.168 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 11001 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_wrr.ko 2024-03-31T14:49:44.169 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 49161 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conncount.ko 2024-03-31T14:49:44.169 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 411665 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack.ko 2024-03-31T14:49:44.169 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 15873 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_amanda.ko 2024-03-31T14:49:44.169 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 10289 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_broadcast.ko 2024-03-31T14:49:44.169 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 33657 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_ftp.ko 2024-03-31T14:49:44.169 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 118145 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_h323.ko 2024-03-31T14:49:44.169 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 22081 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_irc.ko 2024-03-31T14:49:44.169 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 8705 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_netbios_ns.ko 2024-03-31T14:49:44.169 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 108305 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_netlink.ko 2024-03-31T14:49:44.169 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 33569 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_pptp.ko 2024-03-31T14:49:44.169 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 17289 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_sane.ko 2024-03-31T14:49:44.169 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 66529 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_sip.ko 2024-03-31T14:49:44.169 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 11129 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_snmp.ko 2024-03-31T14:49:44.169 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 19073 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_tftp.ko 2024-03-31T14:49:44.169 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 10009 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_dup_netdev.ko 2024-03-31T14:49:44.170 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 92217 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_flow_table.ko 2024-03-31T14:49:44.170 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 9545 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_flow_table_inet.ko 2024-03-31T14:49:44.170 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 38073 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_log_syslog.ko 2024-03-31T14:49:44.170 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 96489 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_nat.ko 2024-03-31T14:49:44.170 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 10025 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_nat_amanda.ko 2024-03-31T14:49:44.170 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 12521 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_nat_ftp.ko 2024-03-31T14:49:44.170 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 11753 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_nat_irc.ko 2024-03-31T14:49:44.170 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 28825 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_nat_sip.ko 2024-03-31T14:49:44.170 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 8705 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_nat_tftp.ko 2024-03-31T14:49:44.170 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 44129 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_synproxy_core.ko 2024-03-31T14:49:44.170 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 623337 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_tables.ko 2024-03-31T14:49:44.170 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 42505 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nfnetlink.ko 2024-03-31T14:49:44.170 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 30153 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nfnetlink_acct.ko 2024-03-31T14:49:44.171 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 30345 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nfnetlink_cttimeout.ko 2024-03-31T14:49:44.171 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 39553 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nfnetlink_log.ko 2024-03-31T14:49:44.171 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 22929 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nfnetlink_osf.ko 2024-03-31T14:49:44.171 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 49641 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nfnetlink_queue.ko 2024-03-31T14:49:44.171 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 10465 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_chain_nat.ko 2024-03-31T14:49:44.171 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 33553 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_compat.ko 2024-03-31T14:49:44.171 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 40801 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_ct.ko 2024-03-31T14:49:44.171 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 9497 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_dup_netdev.ko 2024-03-31T14:49:44.171 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 11417 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_fib.ko 2024-03-31T14:49:44.171 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 13761 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_fwd_netdev.ko 2024-03-31T14:49:44.171 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 14089 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_hash.ko 2024-03-31T14:49:44.171 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 20025 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_limit.ko 2024-03-31T14:49:44.171 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 15305 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_log.ko 2024-03-31T14:49:44.171 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 14705 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_masq.ko 2024-03-31T14:49:44.172 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 14321 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_nat.ko 2024-03-31T14:49:44.172 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 13569 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_numgen.ko 2024-03-31T14:49:44.172 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 11745 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_osf.ko 2024-03-31T14:49:44.172 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 13969 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_queue.ko 2024-03-31T14:49:44.172 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 14089 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_redir.ko 2024-03-31T14:49:44.172 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 10393 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_reject.ko 2024-03-31T14:49:44.172 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 9977 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_reject_inet.ko 2024-03-31T14:49:44.172 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 12833 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_socket.ko 2024-03-31T14:49:44.172 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 20529 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_synproxy.ko 2024-03-31T14:49:44.172 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 14809 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_tproxy.ko 2024-03-31T14:49:44.172 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 89225 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/x_tables.ko 2024-03-31T14:49:44.172 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 12001 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_AUDIT.ko 2024-03-31T14:49:44.172 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 9529 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_CHECKSUM.ko 2024-03-31T14:49:44.172 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 7625 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_CLASSIFY.ko 2024-03-31T14:49:44.173 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 11721 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_CONNSECMARK.ko 2024-03-31T14:49:44.173 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 18073 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_CT.ko 2024-03-31T14:49:44.173 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 10401 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_DSCP.ko 2024-03-31T14:49:44.173 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 9369 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_HL.ko 2024-03-31T14:49:44.173 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 13273 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_HMARK.ko 2024-03-31T14:49:44.173 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 38769 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_IDLETIMER.ko 2024-03-31T14:49:44.173 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 13793 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_LED.ko 2024-03-31T14:49:44.173 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 11393 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_LOG.ko 2024-03-31T14:49:44.173 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 11649 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_MASQUERADE.ko 2024-03-31T14:49:44.173 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 11553 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_NETMAP.ko 2024-03-31T14:49:44.173 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 9377 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_NFLOG.ko 2024-03-31T14:49:44.173 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 13185 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_NFQUEUE.ko 2024-03-31T14:49:44.173 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 19561 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_RATEEST.ko 2024-03-31T14:49:44.173 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 10969 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_REDIRECT.ko 2024-03-31T14:49:44.174 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 14241 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_SECMARK.ko 2024-03-31T14:49:44.174 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 15401 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_TCPMSS.ko 2024-03-31T14:49:44.174 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 10329 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_TCPOPTSTRIP.ko 2024-03-31T14:49:44.174 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 17401 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_TEE.ko 2024-03-31T14:49:44.174 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 16689 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_TPROXY.ko 2024-03-31T14:49:44.174 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 8225 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_TRACE.ko 2024-03-31T14:49:44.174 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 13089 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_addrtype.ko 2024-03-31T14:49:44.174 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 13801 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_bpf.ko 2024-03-31T14:49:44.174 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 16929 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_cgroup.ko 2024-03-31T14:49:44.174 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 11113 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_cluster.ko 2024-03-31T14:49:44.174 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 7401 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_comment.ko 2024-03-31T14:49:44.174 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 10361 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_connbytes.ko 2024-03-31T14:49:44.174 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 11249 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_connlabel.ko 2024-03-31T14:49:44.174 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 9297 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_connlimit.ko 2024-03-31T14:49:44.175 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 13345 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_connmark.ko 2024-03-31T14:49:44.175 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 12297 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_conntrack.ko 2024-03-31T14:49:44.175 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 7721 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_cpu.ko 2024-03-31T14:49:44.175 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 10881 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_dccp.ko 2024-03-31T14:49:44.175 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 7761 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_devgroup.ko 2024-03-31T14:49:44.175 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 8705 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_dscp.ko 2024-03-31T14:49:44.175 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 10561 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_ecn.ko 2024-03-31T14:49:44.175 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 11377 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_esp.ko 2024-03-31T14:49:44.175 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 43193 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_hashlimit.ko 2024-03-31T14:49:44.175 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 11209 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_helper.ko 2024-03-31T14:49:44.175 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 7769 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_hl.ko 2024-03-31T14:49:44.175 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 11817 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_ipcomp.ko 2024-03-31T14:49:44.175 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 11369 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_iprange.ko 2024-03-31T14:49:44.175 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 11689 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_ipvs.ko 2024-03-31T14:49:44.176 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 15945 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_l2tp.ko 2024-03-31T14:49:44.176 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 7897 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_length.ko 2024-03-31T14:49:44.176 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 10113 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_limit.ko 2024-03-31T14:49:44.176 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 7449 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_mac.ko 2024-03-31T14:49:44.176 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 8481 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_mark.ko 2024-03-31T14:49:44.176 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 11721 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_multiport.ko 2024-03-31T14:49:44.176 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 13729 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_nat.ko 2024-03-31T14:49:44.176 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 10001 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_nfacct.ko 2024-03-31T14:49:44.176 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 8209 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_osf.ko 2024-03-31T14:49:44.176 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 9969 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_owner.ko 2024-03-31T14:49:44.176 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 10009 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_physdev.ko 2024-03-31T14:49:44.176 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 7529 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_pkttype.ko 2024-03-31T14:49:44.176 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 11889 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_policy.ko 2024-03-31T14:49:44.176 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 9377 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_quota.ko 2024-03-31T14:49:44.176 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 10305 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_rateest.ko 2024-03-31T14:49:44.177 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 8161 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_realm.ko 2024-03-31T14:49:44.177 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 36353 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_recent.ko 2024-03-31T14:49:44.177 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 13233 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_sctp.ko 2024-03-31T14:49:44.177 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 29993 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_set.ko 2024-03-31T14:49:44.177 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 16809 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_socket.ko 2024-03-31T14:49:44.177 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 9409 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_state.ko 2024-03-31T14:49:44.177 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 9369 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_statistic.ko 2024-03-31T14:49:44.177 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 8705 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_string.ko 2024-03-31T14:49:44.177 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 8449 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_tcpmss.ko 2024-03-31T14:49:44.177 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 15673 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_tcpudp.ko 2024-03-31T14:49:44.177 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 10969 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_time.ko 2024-03-31T14:49:44.177 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 9305 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_u32.ko 2024-03-31T14:49:44.177 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netlink/ 2024-03-31T14:49:44.177 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 14433 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netlink/netlink_diag.ko 2024-03-31T14:49:44.177 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/nsh/ 2024-03-31T14:49:44.178 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 12329 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/nsh/nsh.ko 2024-03-31T14:49:44.178 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/openvswitch/ 2024-03-31T14:49:44.178 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 384137 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/openvswitch/openvswitch.ko 2024-03-31T14:49:44.178 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 10361 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/openvswitch/vport-geneve.ko 2024-03-31T14:49:44.178 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 9481 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/openvswitch/vport-gre.ko 2024-03-31T14:49:44.178 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 11825 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/openvswitch/vport-vxlan.ko 2024-03-31T14:49:44.178 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/packet/ 2024-03-31T14:49:44.178 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 13465 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/packet/af_packet_diag.ko 2024-03-31T14:49:44.178 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/phonet/ 2024-03-31T14:49:44.178 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 112521 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/phonet/phonet.ko 2024-03-31T14:49:44.178 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 55041 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/phonet/pn_pep.ko 2024-03-31T14:49:44.178 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/rds/ 2024-03-31T14:49:44.178 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 253881 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/rds/rds.ko 2024-03-31T14:49:44.179 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 165049 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/rds/rds_rdma.ko 2024-03-31T14:49:44.179 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 68065 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/rds/rds_tcp.ko 2024-03-31T14:49:44.179 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/rxrpc/ 2024-03-31T14:49:44.179 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 828145 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/rxrpc/rxrpc.ko 2024-03-31T14:49:44.179 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/ 2024-03-31T14:49:44.179 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 32145 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/act_csum.ko 2024-03-31T14:49:44.179 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 23481 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/act_gact.ko 2024-03-31T14:49:44.179 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 37081 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/act_mirred.ko 2024-03-31T14:49:44.179 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 23369 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/act_nat.ko 2024-03-31T14:49:44.179 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 43345 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/act_pedit.ko 2024-03-31T14:49:44.179 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 31921 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/act_police.ko 2024-03-31T14:49:44.179 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 20185 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/act_simple.ko 2024-03-31T14:49:44.179 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 35193 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/act_skbedit.ko 2024-03-31T14:49:44.179 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 36569 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/act_vlan.ko 2024-03-31T14:49:44.180 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 26177 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/cls_basic.ko 2024-03-31T14:49:44.180 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 33865 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/cls_bpf.ko 2024-03-31T14:49:44.180 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 19177 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/cls_cgroup.ko 2024-03-31T14:49:44.180 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 31401 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/cls_flow.ko 2024-03-31T14:49:44.180 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 31857 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/cls_fw.ko 2024-03-31T14:49:44.180 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 39513 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/cls_route.ko 2024-03-31T14:49:44.180 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 62649 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/cls_u32.ko 2024-03-31T14:49:44.180 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 7457 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/em_cmp.ko 2024-03-31T14:49:44.180 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 13313 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/em_ipset.ko 2024-03-31T14:49:44.180 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 33417 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/em_meta.ko 2024-03-31T14:49:44.180 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 7913 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/em_nbyte.ko 2024-03-31T14:49:44.180 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 10921 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/em_text.ko 2024-03-31T14:49:44.180 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 7265 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/em_u32.ko 2024-03-31T14:49:44.180 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 21665 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_choke.ko 2024-03-31T14:49:44.180 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 20857 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_codel.ko 2024-03-31T14:49:44.181 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 34585 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_drr.ko 2024-03-31T14:49:44.181 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 35785 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_fq.ko 2024-03-31T14:49:44.181 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 31513 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_fq_codel.ko 2024-03-31T14:49:44.181 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 35449 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_gred.ko 2024-03-31T14:49:44.181 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 49841 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_hfsc.ko 2024-03-31T14:49:44.181 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 21969 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_hhf.ko 2024-03-31T14:49:44.181 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 66441 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_htb.ko 2024-03-31T14:49:44.181 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 26081 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_ingress.ko 2024-03-31T14:49:44.181 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 30217 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_mqprio.ko 2024-03-31T14:49:44.181 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 10801 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_mqprio_lib.ko 2024-03-31T14:49:44.181 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 26081 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_multiq.ko 2024-03-31T14:49:44.181 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 36145 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_netem.ko 2024-03-31T14:49:44.181 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 27401 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_pie.ko 2024-03-31T14:49:44.181 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 10657 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_plug.ko 2024-03-31T14:49:44.182 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 24769 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_prio.ko 2024-03-31T14:49:44.182 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 48529 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_qfq.ko 2024-03-31T14:49:44.182 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 32809 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_red.ko 2024-03-31T14:49:44.182 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 26785 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_sfb.ko 2024-03-31T14:49:44.182 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 31313 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_sfq.ko 2024-03-31T14:49:44.182 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 28937 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_tbf.ko 2024-03-31T14:49:44.182 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 28265 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_teql.ko 2024-03-31T14:49:44.182 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sctp/ 2024-03-31T14:49:44.182 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 932785 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sctp/sctp.ko 2024-03-31T14:49:44.182 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 20241 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sctp/sctp_diag.ko 2024-03-31T14:49:44.182 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sunrpc/ 2024-03-31T14:49:44.182 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sunrpc/auth_gss/ 2024-03-31T14:49:44.182 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 336969 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sunrpc/auth_gss/auth_rpcgss.ko 2024-03-31T14:49:44.182 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 55009 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sunrpc/auth_gss/rpcsec_gss_krb5.ko 2024-03-31T14:49:44.183 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 1513769 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sunrpc/sunrpc.ko 2024-03-31T14:49:44.183 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sunrpc/xprtrdma/ 2024-03-31T14:49:44.183 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 704697 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sunrpc/xprtrdma/rpcrdma.ko 2024-03-31T14:49:44.183 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/tipc/ 2024-03-31T14:49:44.183 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 10393 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/tipc/diag.ko 2024-03-31T14:49:44.183 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 867121 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/tipc/tipc.ko 2024-03-31T14:49:44.183 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/unix/ 2024-03-31T14:49:44.183 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 15929 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/unix/unix_diag.ko 2024-03-31T14:49:44.183 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/vmw_vsock/ 2024-03-31T14:49:44.183 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 87737 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/vmw_vsock/vmw_vsock_virtio_transport_common.ko 2024-03-31T14:49:44.183 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 67937 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/vmw_vsock/vmw_vsock_vmci_transport.ko 2024-03-31T14:49:44.183 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 98089 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/vmw_vsock/vsock.ko 2024-03-31T14:49:44.183 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 11073 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/vmw_vsock/vsock_diag.ko 2024-03-31T14:49:44.183 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 17809 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/vmw_vsock/vsock_loopback.ko 2024-03-31T14:49:44.183 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/x25/ 2024-03-31T14:49:44.184 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 131209 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/x25/x25.ko 2024-03-31T14:49:44.184 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/xfrm/ 2024-03-31T14:49:44.184 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 23057 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/xfrm/xfrm_algo.ko 2024-03-31T14:49:44.184 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 20545 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/xfrm/xfrm_ipcomp.ko 2024-03-31T14:49:44.184 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 105209 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/xfrm/xfrm_user.ko 2024-03-31T14:49:44.184 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/virt/ 2024-03-31T14:49:44.184 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/virt/lib/ 2024-03-31T14:49:44.184 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 15433 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/virt/lib/irqbypass.ko 2024-03-31T14:49:44.184 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 642383 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/modules.alias 2024-03-31T14:49:44.184 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 625850 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/modules.alias.bin 2024-03-31T14:49:44.184 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 7677 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/modules.builtin 2024-03-31T14:49:44.184 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 23498 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/modules.builtin.alias.bin 2024-03-31T14:49:44.184 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 9916 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/modules.builtin.bin 2024-03-31T14:49:44.184 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 70732 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/modules.builtin.modinfo 2024-03-31T14:49:44.184 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 147214 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/modules.dep 2024-03-31T14:49:44.185 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 233213 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/modules.dep.bin 2024-03-31T14:49:44.185 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 210 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/modules.devname 2024-03-31T14:49:44.185 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 72614 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/modules.order 2024-03-31T14:49:44.185 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 879 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/modules.softdep 2024-03-31T14:49:44.185 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 257955 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/modules.symbols 2024-03-31T14:49:44.185 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 312782 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/modules.symbols.bin 2024-03-31T14:49:44.185 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./usr/ 2024-03-31T14:49:44.185 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./usr/share/ 2024-03-31T14:49:44.185 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./usr/share/doc/ 2024-03-31T14:49:44.185 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./usr/share/doc/linux-image-6.8.0-g1df0c7e8a235/ 2024-03-31T14:49:44.185 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 169 2024-03-25 03:29 ./usr/share/doc/linux-image-6.8.0-g1df0c7e8a235/changelog.Debian.gz 2024-03-31T14:49:44.185 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 691 2024-03-25 03:29 ./usr/share/doc/linux-image-6.8.0-g1df0c7e8a235/copyright 2024-03-31T14:49:44.190 DEBUG:teuthology.task.kernel:get_image_version: 6.8.0-g1df0c7e8a235 2024-03-31T14:49:44.190 INFO:teuthology.task.kernel:searching for kernel 6.8.0-g1df0c7e8a235 2024-03-31T14:49:44.190 DEBUG:teuthology.orchestra.run.smithi089:> egrep '(submenu|menuentry.*6.8.0-g1df0c7e8a235).*{' /boot/grub/grub.cfg 2024-03-31T14:49:44.192 INFO:teuthology.orchestra.run.smithi016.stdout:submenu 'Advanced options for Ubuntu' $menuentry_id_option 'gnulinux-advanced-7e9e2103-8bb7-4452-b35a-3b1230743254' { 2024-03-31T14:49:44.193 INFO:teuthology.orchestra.run.smithi016.stdout: menuentry 'Ubuntu, with Linux 6.8.0-g1df0c7e8a235' --class ubuntu --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-6.8.0-g1df0c7e8a235-advanced-7e9e2103-8bb7-4452-b35a-3b1230743254' { 2024-03-31T14:49:44.193 INFO:teuthology.orchestra.run.smithi016.stdout: menuentry 'Ubuntu, with Linux 6.8.0-g1df0c7e8a235 (recovery mode)' --class ubuntu --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-6.8.0-g1df0c7e8a235-recovery-7e9e2103-8bb7-4452-b35a-3b1230743254' { 2024-03-31T14:49:44.193 INFO:teuthology.task.kernel:submenu_title:Advanced options for Ubuntu> 2024-03-31T14:49:44.193 INFO:teuthology.task.kernel:default_title:Ubuntu, with Linux 6.8.0-g1df0c7e8a235 2024-03-31T14:49:44.193 DEBUG:teuthology.orchestra.run.smithi016:> /bin/echo -e 'cat </dev/null && sudo chmod a+x -- '/etc/grub.d/01_ceph_kernel.tmp~' && sudo mv -- '/etc/grub.d/01_ceph_kernel.tmp~' /etc/grub.d/01_ceph_kernel && sudo update-grub && rm /tmp/linux-image.deb && ( sleep 1 && sudo shutdown -r now & ) 2024-03-31T14:49:44.197 INFO:teuthology.orchestra.run.smithi089.stdout:submenu 'Advanced options for Ubuntu' $menuentry_id_option 'gnulinux-advanced-7e9e2103-8bb7-4452-b35a-3b1230743254' { 2024-03-31T14:49:44.198 INFO:teuthology.orchestra.run.smithi089.stdout: menuentry 'Ubuntu, with Linux 6.8.0-g1df0c7e8a235' --class ubuntu --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-6.8.0-g1df0c7e8a235-advanced-7e9e2103-8bb7-4452-b35a-3b1230743254' { 2024-03-31T14:49:44.198 INFO:teuthology.orchestra.run.smithi089.stdout: menuentry 'Ubuntu, with Linux 6.8.0-g1df0c7e8a235 (recovery mode)' --class ubuntu --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-6.8.0-g1df0c7e8a235-recovery-7e9e2103-8bb7-4452-b35a-3b1230743254' { 2024-03-31T14:49:44.198 INFO:teuthology.task.kernel:submenu_title:Advanced options for Ubuntu> 2024-03-31T14:49:44.198 INFO:teuthology.task.kernel:default_title:Ubuntu, with Linux 6.8.0-g1df0c7e8a235 2024-03-31T14:49:44.198 DEBUG:teuthology.orchestra.run.smithi089:> /bin/echo -e 'cat </dev/null && sudo chmod a+x -- '/etc/grub.d/01_ceph_kernel.tmp~' && sudo mv -- '/etc/grub.d/01_ceph_kernel.tmp~' /etc/grub.d/01_ceph_kernel && sudo update-grub && rm /tmp/linux-image.deb && ( sleep 1 && sudo shutdown -r now & ) 2024-03-31T14:49:44.237 DEBUG:teuthology.task.kernel:Waiting for install on ubuntu@smithi016.front.sepia.ceph.com to complete... 2024-03-31T14:49:44.240 DEBUG:teuthology.task.kernel:Waiting for install on ubuntu@smithi089.front.sepia.ceph.com to complete... 2024-03-31T14:49:44.602 INFO:teuthology.orchestra.run.smithi089.stderr:Sourcing file `/etc/default/grub' 2024-03-31T14:49:44.604 INFO:teuthology.orchestra.run.smithi089.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-03-31T14:49:44.634 INFO:teuthology.orchestra.run.smithi089.stderr:Generating grub configuration file ... 2024-03-31T14:49:44.646 INFO:teuthology.orchestra.run.smithi016.stderr:Sourcing file `/etc/default/grub' 2024-03-31T14:49:44.647 INFO:teuthology.orchestra.run.smithi016.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-03-31T14:49:44.658 INFO:teuthology.orchestra.run.smithi016.stderr:Generating grub configuration file ... 2024-03-31T14:49:44.815 INFO:teuthology.orchestra.run.smithi089.stderr:Found linux image: /boot/vmlinuz-6.8.0-g1df0c7e8a235 2024-03-31T14:49:44.821 INFO:teuthology.orchestra.run.smithi089.stderr:Found initrd image: /boot/initrd.img-6.8.0-g1df0c7e8a235 2024-03-31T14:49:44.869 INFO:teuthology.orchestra.run.smithi016.stderr:Found linux image: /boot/vmlinuz-6.8.0-g1df0c7e8a235 2024-03-31T14:49:44.875 INFO:teuthology.orchestra.run.smithi016.stderr:Found initrd image: /boot/initrd.img-6.8.0-g1df0c7e8a235 2024-03-31T14:49:45.087 INFO:teuthology.orchestra.run.smithi089.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-03-31T14:49:45.092 INFO:teuthology.orchestra.run.smithi089.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-03-31T14:49:45.116 INFO:teuthology.orchestra.run.smithi016.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-03-31T14:49:45.121 INFO:teuthology.orchestra.run.smithi016.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-03-31T14:49:45.132 INFO:teuthology.orchestra.run.smithi089.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-03-31T14:49:45.137 INFO:teuthology.orchestra.run.smithi089.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-03-31T14:49:45.167 INFO:teuthology.orchestra.run.smithi016.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-03-31T14:49:45.172 INFO:teuthology.orchestra.run.smithi016.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-03-31T14:49:45.256 INFO:teuthology.orchestra.run.smithi089.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2024-03-31T14:49:45.257 INFO:teuthology.orchestra.run.smithi089.stderr:Systems on them will not be added to the GRUB boot configuration. 2024-03-31T14:49:45.257 INFO:teuthology.orchestra.run.smithi089.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-03-31T14:49:45.266 INFO:teuthology.orchestra.run.smithi089.stderr:done 2024-03-31T14:49:45.331 INFO:teuthology.orchestra.run.smithi016.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2024-03-31T14:49:45.332 INFO:teuthology.orchestra.run.smithi016.stderr:Systems on them will not be added to the GRUB boot configuration. 2024-03-31T14:49:45.332 INFO:teuthology.orchestra.run.smithi016.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-03-31T14:49:45.343 INFO:teuthology.orchestra.run.smithi016.stderr:done 2024-03-31T14:50:16.388 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-03-31T14:50:16.389 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi089.front.sepia.ceph.com' 2024-03-31T14:50:16.390 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi089.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-31T14:50:16.447 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-03-31T14:50:16.447 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi016.front.sepia.ceph.com' 2024-03-31T14:50:16.448 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi016.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-31T14:50:34.911 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.16 2024-03-31T14:50:43.920 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi016.front.sepia.ceph.com' 2024-03-31T14:50:43.921 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi016.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-31T14:51:16.399 DEBUG:teuthology.orchestra.remote:timed out 2024-03-31T14:51:25.408 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi089.front.sepia.ceph.com' 2024-03-31T14:51:25.409 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi089.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-31T14:51:25.788 DEBUG:teuthology.orchestra.run.smithi089:> true 2024-03-31T14:51:26.840 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi089.front.sepia.ceph.com' 2024-03-31T14:51:26.840 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "6.8.0-g1df0c7e8a235"... 2024-03-31T14:51:26.841 DEBUG:teuthology.orchestra.run.smithi089:> uname -r 2024-03-31T14:51:26.890 INFO:teuthology.orchestra.run.smithi089.stdout:6.8.0-g1df0c7e8a235 2024-03-31T14:51:26.891 DEBUG:teuthology.task.kernel:current kernel version is 6.8.0-g1df0c7e8a235 vs 6.8.0-g1df0c7e8a235 2024-03-31T14:51:26.891 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-31T14:51:26.891 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-03-31T14:51:27.891 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-03-31T14:51:27.891 DEBUG:teuthology.orchestra.run.smithi089:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-31T14:51:28.021 INFO:teuthology.orchestra.run.smithi089.stdout:ttyS1 2024-03-31T14:51:28.092 DEBUG:teuthology.parallel:result is None 2024-03-31T14:51:43.924 DEBUG:teuthology.orchestra.remote:timed out 2024-03-31T14:51:55.925 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi016.front.sepia.ceph.com' 2024-03-31T14:51:55.927 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi016.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-31T14:51:56.152 DEBUG:teuthology.orchestra.run.smithi016:> true 2024-03-31T14:51:56.948 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi016.front.sepia.ceph.com' 2024-03-31T14:51:56.948 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "6.8.0-g1df0c7e8a235"... 2024-03-31T14:51:56.948 DEBUG:teuthology.orchestra.run.smithi016:> uname -r 2024-03-31T14:51:56.998 INFO:teuthology.orchestra.run.smithi016.stdout:6.8.0-g1df0c7e8a235 2024-03-31T14:51:56.998 DEBUG:teuthology.task.kernel:current kernel version is 6.8.0-g1df0c7e8a235 vs 6.8.0-g1df0c7e8a235 2024-03-31T14:51:56.998 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-31T14:51:56.999 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-03-31T14:51:58.000 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-03-31T14:51:58.000 DEBUG:teuthology.orchestra.run.smithi016:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-31T14:51:58.070 INFO:teuthology.orchestra.run.smithi016.stdout:ttyS1 2024-03-31T14:51:58.113 DEBUG:teuthology.parallel:result is None 2024-03-31T14:51:58.114 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-31T14:51:58.123 INFO:teuthology.task.internal:Creating test directory... 2024-03-31T14:51:58.123 DEBUG:teuthology.orchestra.run.smithi016:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-31T14:51:58.126 DEBUG:teuthology.orchestra.run.smithi089:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-31T14:51:58.136 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-31T14:51:58.146 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-31T14:51:58.160 INFO:teuthology.task.internal:Creating archive directory... 2024-03-31T14:51:58.160 DEBUG:teuthology.orchestra.run.smithi016:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-31T14:51:58.177 DEBUG:teuthology.orchestra.run.smithi089:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-31T14:51:58.197 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-31T14:51:58.209 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-31T14:51:58.210 DEBUG:teuthology.orchestra.run.smithi016:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-31T14:51:58.233 DEBUG:teuthology.orchestra.run.smithi089:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-31T14:51:58.255 INFO:teuthology.orchestra.run.smithi016.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-31T14:51:58.268 INFO:teuthology.orchestra.run.smithi089.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-31T14:51:58.269 INFO:teuthology.orchestra.run.smithi016.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-31T14:51:58.282 INFO:teuthology.orchestra.run.smithi089.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-31T14:51:58.285 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-31T14:51:58.292 INFO:teuthology.task.internal:Configuring sudo... 2024-03-31T14:51:58.293 DEBUG:teuthology.orchestra.run.smithi016:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-31T14:51:58.317 DEBUG:teuthology.orchestra.run.smithi089:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-31T14:51:58.351 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-31T14:51:58.362 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-31T14:51:58.362 DEBUG:teuthology.orchestra.run.smithi016:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-31T14:51:58.381 DEBUG:teuthology.orchestra.run.smithi089:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-31T14:51:58.406 DEBUG:teuthology.orchestra.run.smithi016:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-31T14:51:58.439 DEBUG:teuthology.orchestra.run.smithi016:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-31T14:51:58.491 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-03-31T14:51:58.491 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-31T14:51:58.561 DEBUG:teuthology.orchestra.run.smithi089:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-31T14:51:58.570 DEBUG:teuthology.orchestra.run.smithi089:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-31T14:51:58.619 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-03-31T14:51:58.619 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-31T14:51:58.698 DEBUG:teuthology.orchestra.run.smithi016:> sudo service rsyslog restart 2024-03-31T14:51:58.702 DEBUG:teuthology.orchestra.run.smithi089:> sudo service rsyslog restart 2024-03-31T14:51:58.820 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-31T14:51:58.828 INFO:teuthology.task.internal:Starting timer... 2024-03-31T14:51:58.828 INFO:teuthology.run_tasks:Running task pcp... 2024-03-31T14:51:58.882 INFO:teuthology.run_tasks:Running task selinux... 2024-03-31T14:51:58.899 DEBUG:teuthology.task.selinux:Excluding smithi016: OS 'ubuntu' does not support SELinux 2024-03-31T14:51:58.899 DEBUG:teuthology.task.selinux:Excluding smithi089: OS 'ubuntu' does not support SELinux 2024-03-31T14:51:58.899 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-31T14:51:58.899 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2024-03-31T14:51:58.899 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-31T14:51:58.899 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-31T14:51:58.910 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-03-31T14:51:58.916 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-03-31T14:51:59.076 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-31T14:51:59.095 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-31T14:51:59.096 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi016.front.sepia.ceph.com,smithi089.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-31T14:59:38.243 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi016.front.sepia.ceph.com'), Remote(name='ubuntu@smithi089.front.sepia.ceph.com')] 2024-03-31T14:59:38.244 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi016.front.sepia.ceph.com' 2024-03-31T14:59:38.245 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi016.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-31T14:59:38.334 DEBUG:teuthology.orchestra.run.smithi016:> true 2024-03-31T14:59:38.468 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi016.front.sepia.ceph.com' 2024-03-31T14:59:38.468 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi089.front.sepia.ceph.com' 2024-03-31T14:59:38.469 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi089.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-31T14:59:38.553 DEBUG:teuthology.orchestra.run.smithi089:> true 2024-03-31T14:59:38.673 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi089.front.sepia.ceph.com' 2024-03-31T14:59:38.674 INFO:teuthology.run_tasks:Running task clock... 2024-03-31T14:59:38.703 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-31T14:59:38.703 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-31T14:59:38.704 DEBUG:teuthology.orchestra.run.smithi016:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-31T14:59:38.708 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-31T14:59:38.708 DEBUG:teuthology.orchestra.run.smithi089:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-31T14:59:38.746 INFO:teuthology.orchestra.run.smithi016.stdout:31 Mar 14:59:38 ntpd[17052]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2024-03-31T14:59:38.746 INFO:teuthology.orchestra.run.smithi016.stdout:31 Mar 14:59:38 ntpd[17052]: Command line: ntpd -gq 2024-03-31T14:59:38.747 INFO:teuthology.orchestra.run.smithi016.stdout:31 Mar 14:59:38 ntpd[17052]: ---------------------------------------------------- 2024-03-31T14:59:38.747 INFO:teuthology.orchestra.run.smithi016.stdout:31 Mar 14:59:38 ntpd[17052]: ntp-4 is maintained by Network Time Foundation, 2024-03-31T14:59:38.747 INFO:teuthology.orchestra.run.smithi016.stdout:31 Mar 14:59:38 ntpd[17052]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2024-03-31T14:59:38.747 INFO:teuthology.orchestra.run.smithi016.stdout:31 Mar 14:59:38 ntpd[17052]: corporation. Support and training for ntp-4 are 2024-03-31T14:59:38.747 INFO:teuthology.orchestra.run.smithi016.stdout:31 Mar 14:59:38 ntpd[17052]: available at https://www.nwtime.org/support 2024-03-31T14:59:38.747 INFO:teuthology.orchestra.run.smithi016.stdout:31 Mar 14:59:38 ntpd[17052]: ---------------------------------------------------- 2024-03-31T14:59:38.748 INFO:teuthology.orchestra.run.smithi016.stdout:31 Mar 14:59:38 ntpd[17052]: proto: precision = 0.043 usec (-24) 2024-03-31T14:59:38.748 INFO:teuthology.orchestra.run.smithi016.stdout:31 Mar 14:59:38 ntpd[17052]: basedate set to 2022-02-04 2024-03-31T14:59:38.748 INFO:teuthology.orchestra.run.smithi016.stdout:31 Mar 14:59:38 ntpd[17052]: gps base set to 2022-02-06 (week 2196) 2024-03-31T14:59:38.748 INFO:teuthology.orchestra.run.smithi016.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-03-31T14:59:38.748 INFO:teuthology.orchestra.run.smithi016.stderr:restrict ::: KOD does nothing without LIMITED. 2024-03-31T14:59:38.748 INFO:teuthology.orchestra.run.smithi016.stdout:31 Mar 14:59:38 ntpd[17052]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-03-31T14:59:38.748 INFO:teuthology.orchestra.run.smithi016.stdout:31 Mar 14:59:38 ntpd[17052]: restrict ::: KOD does nothing without LIMITED. 2024-03-31T14:59:38.750 INFO:teuthology.orchestra.run.smithi016.stdout:31 Mar 14:59:38 ntpd[17052]: Listen and drop on 0 v6wildcard [::]:123 2024-03-31T14:59:38.750 INFO:teuthology.orchestra.run.smithi016.stdout:31 Mar 14:59:38 ntpd[17052]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-03-31T14:59:38.750 INFO:teuthology.orchestra.run.smithi016.stdout:31 Mar 14:59:38 ntpd[17052]: Listen normally on 2 lo 127.0.0.1:123 2024-03-31T14:59:38.750 INFO:teuthology.orchestra.run.smithi016.stdout:31 Mar 14:59:38 ntpd[17052]: Listen normally on 3 ens1f0 172.21.15.16:123 2024-03-31T14:59:38.750 INFO:teuthology.orchestra.run.smithi016.stdout:31 Mar 14:59:38 ntpd[17052]: Listen normally on 4 lo [::1]:123 2024-03-31T14:59:38.750 INFO:teuthology.orchestra.run.smithi016.stdout:31 Mar 14:59:38 ntpd[17052]: Listen normally on 5 ens1f0 [fe80::ec4:7aff:febd:1564%4]:123 2024-03-31T14:59:38.750 INFO:teuthology.orchestra.run.smithi016.stdout:31 Mar 14:59:38 ntpd[17052]: Listening on routing socket on fd #22 for interface updates 2024-03-31T14:59:38.753 INFO:teuthology.orchestra.run.smithi089.stdout:31 Mar 14:59:38 ntpd[17171]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2024-03-31T14:59:38.753 INFO:teuthology.orchestra.run.smithi089.stdout:31 Mar 14:59:38 ntpd[17171]: Command line: ntpd -gq 2024-03-31T14:59:38.753 INFO:teuthology.orchestra.run.smithi089.stdout:31 Mar 14:59:38 ntpd[17171]: ---------------------------------------------------- 2024-03-31T14:59:38.753 INFO:teuthology.orchestra.run.smithi089.stdout:31 Mar 14:59:38 ntpd[17171]: ntp-4 is maintained by Network Time Foundation, 2024-03-31T14:59:38.754 INFO:teuthology.orchestra.run.smithi089.stdout:31 Mar 14:59:38 ntpd[17171]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2024-03-31T14:59:38.754 INFO:teuthology.orchestra.run.smithi089.stdout:31 Mar 14:59:38 ntpd[17171]: corporation. Support and training for ntp-4 are 2024-03-31T14:59:38.754 INFO:teuthology.orchestra.run.smithi089.stdout:31 Mar 14:59:38 ntpd[17171]: available at https://www.nwtime.org/support 2024-03-31T14:59:38.754 INFO:teuthology.orchestra.run.smithi089.stdout:31 Mar 14:59:38 ntpd[17171]: ---------------------------------------------------- 2024-03-31T14:59:38.754 INFO:teuthology.orchestra.run.smithi089.stdout:31 Mar 14:59:38 ntpd[17171]: proto: precision = 0.041 usec (-24) 2024-03-31T14:59:38.754 INFO:teuthology.orchestra.run.smithi089.stdout:31 Mar 14:59:38 ntpd[17171]: basedate set to 2022-02-04 2024-03-31T14:59:38.755 INFO:teuthology.orchestra.run.smithi089.stdout:31 Mar 14:59:38 ntpd[17171]: gps base set to 2022-02-06 (week 2196) 2024-03-31T14:59:38.755 INFO:teuthology.orchestra.run.smithi089.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-03-31T14:59:38.755 INFO:teuthology.orchestra.run.smithi089.stdout:31 Mar 14:59:38 ntpd[17171]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-03-31T14:59:38.755 INFO:teuthology.orchestra.run.smithi089.stderr:restrict ::: KOD does nothing without LIMITED. 2024-03-31T14:59:38.756 INFO:teuthology.orchestra.run.smithi089.stdout:31 Mar 14:59:38 ntpd[17171]: restrict ::: KOD does nothing without LIMITED. 2024-03-31T14:59:38.757 INFO:teuthology.orchestra.run.smithi089.stdout:31 Mar 14:59:38 ntpd[17171]: Listen and drop on 0 v6wildcard [::]:123 2024-03-31T14:59:38.757 INFO:teuthology.orchestra.run.smithi089.stdout:31 Mar 14:59:38 ntpd[17171]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-03-31T14:59:38.757 INFO:teuthology.orchestra.run.smithi089.stdout:31 Mar 14:59:38 ntpd[17171]: Listen normally on 2 lo 127.0.0.1:123 2024-03-31T14:59:38.757 INFO:teuthology.orchestra.run.smithi089.stdout:31 Mar 14:59:38 ntpd[17171]: Listen normally on 3 enp3s0f1 172.21.15.89:123 2024-03-31T14:59:38.757 INFO:teuthology.orchestra.run.smithi089.stdout:31 Mar 14:59:38 ntpd[17171]: Listen normally on 4 lo [::1]:123 2024-03-31T14:59:38.757 INFO:teuthology.orchestra.run.smithi089.stdout:31 Mar 14:59:38 ntpd[17171]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe8f:d501%5]:123 2024-03-31T14:59:38.757 INFO:teuthology.orchestra.run.smithi089.stdout:31 Mar 14:59:38 ntpd[17171]: Listening on routing socket on fd #22 for interface updates 2024-03-31T14:59:39.748 INFO:teuthology.orchestra.run.smithi016.stderr:31 Mar 14:59:39 ntpd[17052]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-31T14:59:39.748 INFO:teuthology.orchestra.run.smithi016.stderr:31 Mar 14:59:39 ntpd[17052]: can't open /var/log/ntpstats/rawstats.20240331: Permission denied 2024-03-31T14:59:39.748 INFO:teuthology.orchestra.run.smithi016.stderr:31 Mar 14:59:39 ntpd[17052]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-31T14:59:39.748 INFO:teuthology.orchestra.run.smithi016.stderr:31 Mar 14:59:39 ntpd[17052]: can't open /var/log/ntpstats/rawstats.20240331: Permission denied 2024-03-31T14:59:39.748 INFO:teuthology.orchestra.run.smithi016.stderr:31 Mar 14:59:39 ntpd[17052]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-31T14:59:39.748 INFO:teuthology.orchestra.run.smithi016.stderr:31 Mar 14:59:39 ntpd[17052]: can't open /var/log/ntpstats/rawstats.20240331: Permission denied 2024-03-31T14:59:39.749 INFO:teuthology.orchestra.run.smithi016.stderr:31 Mar 14:59:39 ntpd[17052]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-31T14:59:39.749 INFO:teuthology.orchestra.run.smithi016.stderr:31 Mar 14:59:39 ntpd[17052]: can't open /var/log/ntpstats/peerstats.20240331: Permission denied 2024-03-31T14:59:39.749 INFO:teuthology.orchestra.run.smithi016.stderr:31 Mar 14:59:39 ntpd[17052]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-31T14:59:39.749 INFO:teuthology.orchestra.run.smithi016.stderr:31 Mar 14:59:39 ntpd[17052]: can't open /var/log/ntpstats/rawstats.20240331: Permission denied 2024-03-31T14:59:39.749 INFO:teuthology.orchestra.run.smithi016.stderr:31 Mar 14:59:39 ntpd[17052]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-31T14:59:39.749 INFO:teuthology.orchestra.run.smithi016.stderr:31 Mar 14:59:39 ntpd[17052]: can't open /var/log/ntpstats/peerstats.20240331: Permission denied 2024-03-31T14:59:40.747 INFO:teuthology.orchestra.run.smithi016.stderr:31 Mar 14:59:40 ntpd[17052]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-31T14:59:40.747 INFO:teuthology.orchestra.run.smithi016.stderr:31 Mar 14:59:40 ntpd[17052]: can't open /var/log/ntpstats/rawstats.20240331: Permission denied 2024-03-31T14:59:40.747 INFO:teuthology.orchestra.run.smithi016.stderr:31 Mar 14:59:40 ntpd[17052]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-31T14:59:40.747 INFO:teuthology.orchestra.run.smithi016.stderr:31 Mar 14:59:40 ntpd[17052]: can't open /var/log/ntpstats/rawstats.20240331: Permission denied 2024-03-31T14:59:40.748 INFO:teuthology.orchestra.run.smithi016.stderr:31 Mar 14:59:40 ntpd[17052]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-31T14:59:40.748 INFO:teuthology.orchestra.run.smithi016.stderr:31 Mar 14:59:40 ntpd[17052]: can't open /var/log/ntpstats/peerstats.20240331: Permission denied 2024-03-31T14:59:40.754 INFO:teuthology.orchestra.run.smithi089.stderr:31 Mar 14:59:40 ntpd[17171]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-31T14:59:40.754 INFO:teuthology.orchestra.run.smithi089.stderr:31 Mar 14:59:40 ntpd[17171]: can't open /var/log/ntpstats/rawstats.20240331: Permission denied 2024-03-31T14:59:41.748 INFO:teuthology.orchestra.run.smithi016.stderr:31 Mar 14:59:41 ntpd[17052]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-31T14:59:41.748 INFO:teuthology.orchestra.run.smithi016.stderr:31 Mar 14:59:41 ntpd[17052]: can't open /var/log/ntpstats/rawstats.20240331: Permission denied 2024-03-31T14:59:41.748 INFO:teuthology.orchestra.run.smithi016.stderr:31 Mar 14:59:41 ntpd[17052]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-31T14:59:41.748 INFO:teuthology.orchestra.run.smithi016.stderr:31 Mar 14:59:41 ntpd[17052]: can't open /var/log/ntpstats/rawstats.20240331: Permission denied 2024-03-31T14:59:41.748 INFO:teuthology.orchestra.run.smithi016.stderr:31 Mar 14:59:41 ntpd[17052]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-31T14:59:41.748 INFO:teuthology.orchestra.run.smithi016.stderr:31 Mar 14:59:41 ntpd[17052]: can't open /var/log/ntpstats/rawstats.20240331: Permission denied 2024-03-31T14:59:41.748 INFO:teuthology.orchestra.run.smithi016.stderr:31 Mar 14:59:41 ntpd[17052]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-31T14:59:41.749 INFO:teuthology.orchestra.run.smithi016.stderr:31 Mar 14:59:41 ntpd[17052]: can't open /var/log/ntpstats/peerstats.20240331: Permission denied 2024-03-31T14:59:41.749 INFO:teuthology.orchestra.run.smithi016.stderr:31 Mar 14:59:41 ntpd[17052]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-31T14:59:41.749 INFO:teuthology.orchestra.run.smithi016.stderr:31 Mar 14:59:41 ntpd[17052]: can't open /var/log/ntpstats/rawstats.20240331: Permission denied 2024-03-31T14:59:41.749 INFO:teuthology.orchestra.run.smithi016.stderr:31 Mar 14:59:41 ntpd[17052]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-31T14:59:41.749 INFO:teuthology.orchestra.run.smithi016.stderr:31 Mar 14:59:41 ntpd[17052]: can't open /var/log/ntpstats/peerstats.20240331: Permission denied 2024-03-31T14:59:42.747 INFO:teuthology.orchestra.run.smithi016.stderr:31 Mar 14:59:42 ntpd[17052]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-31T14:59:42.747 INFO:teuthology.orchestra.run.smithi016.stderr:31 Mar 14:59:42 ntpd[17052]: can't open /var/log/ntpstats/rawstats.20240331: Permission denied 2024-03-31T14:59:42.747 INFO:teuthology.orchestra.run.smithi016.stderr:31 Mar 14:59:42 ntpd[17052]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-31T14:59:42.747 INFO:teuthology.orchestra.run.smithi016.stderr:31 Mar 14:59:42 ntpd[17052]: can't open /var/log/ntpstats/rawstats.20240331: Permission denied 2024-03-31T14:59:42.747 INFO:teuthology.orchestra.run.smithi016.stderr:31 Mar 14:59:42 ntpd[17052]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-31T14:59:42.747 INFO:teuthology.orchestra.run.smithi016.stderr:31 Mar 14:59:42 ntpd[17052]: can't open /var/log/ntpstats/peerstats.20240331: Permission denied 2024-03-31T14:59:42.755 INFO:teuthology.orchestra.run.smithi089.stderr:31 Mar 14:59:42 ntpd[17171]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-31T14:59:42.755 INFO:teuthology.orchestra.run.smithi089.stderr:31 Mar 14:59:42 ntpd[17171]: can't open /var/log/ntpstats/rawstats.20240331: Permission denied 2024-03-31T14:59:42.755 INFO:teuthology.orchestra.run.smithi089.stderr:31 Mar 14:59:42 ntpd[17171]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-31T14:59:42.755 INFO:teuthology.orchestra.run.smithi089.stderr:31 Mar 14:59:42 ntpd[17171]: can't open /var/log/ntpstats/rawstats.20240331: Permission denied 2024-03-31T14:59:42.755 INFO:teuthology.orchestra.run.smithi089.stderr:31 Mar 14:59:42 ntpd[17171]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-31T14:59:42.755 INFO:teuthology.orchestra.run.smithi089.stderr:31 Mar 14:59:42 ntpd[17171]: can't open /var/log/ntpstats/rawstats.20240331: Permission denied 2024-03-31T14:59:42.755 INFO:teuthology.orchestra.run.smithi089.stderr:31 Mar 14:59:42 ntpd[17171]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-31T14:59:42.755 INFO:teuthology.orchestra.run.smithi089.stderr:31 Mar 14:59:42 ntpd[17171]: can't open /var/log/ntpstats/peerstats.20240331: Permission denied 2024-03-31T14:59:42.755 INFO:teuthology.orchestra.run.smithi089.stderr:31 Mar 14:59:42 ntpd[17171]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-31T14:59:42.755 INFO:teuthology.orchestra.run.smithi089.stderr:31 Mar 14:59:42 ntpd[17171]: can't open /var/log/ntpstats/rawstats.20240331: Permission denied 2024-03-31T14:59:42.755 INFO:teuthology.orchestra.run.smithi089.stderr:31 Mar 14:59:42 ntpd[17171]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-31T14:59:42.756 INFO:teuthology.orchestra.run.smithi089.stderr:31 Mar 14:59:42 ntpd[17171]: can't open /var/log/ntpstats/peerstats.20240331: Permission denied 2024-03-31T14:59:43.748 INFO:teuthology.orchestra.run.smithi016.stderr:31 Mar 14:59:43 ntpd[17052]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-31T14:59:43.748 INFO:teuthology.orchestra.run.smithi016.stderr:31 Mar 14:59:43 ntpd[17052]: can't open /var/log/ntpstats/rawstats.20240331: Permission denied 2024-03-31T14:59:43.748 INFO:teuthology.orchestra.run.smithi016.stderr:31 Mar 14:59:43 ntpd[17052]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-31T14:59:43.748 INFO:teuthology.orchestra.run.smithi016.stderr:31 Mar 14:59:43 ntpd[17052]: can't open /var/log/ntpstats/rawstats.20240331: Permission denied 2024-03-31T14:59:43.748 INFO:teuthology.orchestra.run.smithi016.stderr:31 Mar 14:59:43 ntpd[17052]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-31T14:59:43.749 INFO:teuthology.orchestra.run.smithi016.stderr:31 Mar 14:59:43 ntpd[17052]: can't open /var/log/ntpstats/rawstats.20240331: Permission denied 2024-03-31T14:59:43.749 INFO:teuthology.orchestra.run.smithi016.stderr:31 Mar 14:59:43 ntpd[17052]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-31T14:59:43.749 INFO:teuthology.orchestra.run.smithi016.stderr:31 Mar 14:59:43 ntpd[17052]: can't open /var/log/ntpstats/rawstats.20240331: Permission denied 2024-03-31T14:59:43.749 INFO:teuthology.orchestra.run.smithi016.stderr:31 Mar 14:59:43 ntpd[17052]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-31T14:59:43.749 INFO:teuthology.orchestra.run.smithi016.stderr:31 Mar 14:59:43 ntpd[17052]: can't open /var/log/ntpstats/peerstats.20240331: Permission denied 2024-03-31T14:59:43.749 INFO:teuthology.orchestra.run.smithi016.stderr:31 Mar 14:59:43 ntpd[17052]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-31T14:59:43.749 INFO:teuthology.orchestra.run.smithi016.stderr:31 Mar 14:59:43 ntpd[17052]: can't open /var/log/ntpstats/rawstats.20240331: Permission denied 2024-03-31T14:59:43.749 INFO:teuthology.orchestra.run.smithi016.stderr:31 Mar 14:59:43 ntpd[17052]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-31T14:59:43.749 INFO:teuthology.orchestra.run.smithi016.stderr:31 Mar 14:59:43 ntpd[17052]: can't open /var/log/ntpstats/peerstats.20240331: Permission denied 2024-03-31T14:59:43.754 INFO:teuthology.orchestra.run.smithi089.stderr:31 Mar 14:59:43 ntpd[17171]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-31T14:59:43.754 INFO:teuthology.orchestra.run.smithi089.stderr:31 Mar 14:59:43 ntpd[17171]: can't open /var/log/ntpstats/rawstats.20240331: Permission denied 2024-03-31T14:59:43.754 INFO:teuthology.orchestra.run.smithi089.stderr:31 Mar 14:59:43 ntpd[17171]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-31T14:59:43.754 INFO:teuthology.orchestra.run.smithi089.stderr:31 Mar 14:59:43 ntpd[17171]: can't open /var/log/ntpstats/rawstats.20240331: Permission denied 2024-03-31T14:59:43.754 INFO:teuthology.orchestra.run.smithi089.stderr:31 Mar 14:59:43 ntpd[17171]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-31T14:59:43.754 INFO:teuthology.orchestra.run.smithi089.stderr:31 Mar 14:59:43 ntpd[17171]: can't open /var/log/ntpstats/peerstats.20240331: Permission denied 2024-03-31T14:59:44.747 INFO:teuthology.orchestra.run.smithi016.stderr:31 Mar 14:59:44 ntpd[17052]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-31T14:59:44.747 INFO:teuthology.orchestra.run.smithi016.stderr:31 Mar 14:59:44 ntpd[17052]: can't open /var/log/ntpstats/rawstats.20240331: Permission denied 2024-03-31T14:59:44.747 INFO:teuthology.orchestra.run.smithi016.stderr:31 Mar 14:59:44 ntpd[17052]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-31T14:59:44.747 INFO:teuthology.orchestra.run.smithi016.stderr:31 Mar 14:59:44 ntpd[17052]: can't open /var/log/ntpstats/rawstats.20240331: Permission denied 2024-03-31T14:59:44.747 INFO:teuthology.orchestra.run.smithi016.stderr:31 Mar 14:59:44 ntpd[17052]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-31T14:59:44.747 INFO:teuthology.orchestra.run.smithi016.stderr:31 Mar 14:59:44 ntpd[17052]: can't open /var/log/ntpstats/peerstats.20240331: Permission denied 2024-03-31T14:59:44.755 INFO:teuthology.orchestra.run.smithi089.stderr:31 Mar 14:59:44 ntpd[17171]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-31T14:59:44.755 INFO:teuthology.orchestra.run.smithi089.stderr:31 Mar 14:59:44 ntpd[17171]: can't open /var/log/ntpstats/rawstats.20240331: Permission denied 2024-03-31T14:59:44.755 INFO:teuthology.orchestra.run.smithi089.stderr:31 Mar 14:59:44 ntpd[17171]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-31T14:59:44.755 INFO:teuthology.orchestra.run.smithi089.stderr:31 Mar 14:59:44 ntpd[17171]: can't open /var/log/ntpstats/rawstats.20240331: Permission denied 2024-03-31T14:59:44.755 INFO:teuthology.orchestra.run.smithi089.stderr:31 Mar 14:59:44 ntpd[17171]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-31T14:59:44.755 INFO:teuthology.orchestra.run.smithi089.stderr:31 Mar 14:59:44 ntpd[17171]: can't open /var/log/ntpstats/rawstats.20240331: Permission denied 2024-03-31T14:59:44.755 INFO:teuthology.orchestra.run.smithi089.stderr:31 Mar 14:59:44 ntpd[17171]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-31T14:59:44.755 INFO:teuthology.orchestra.run.smithi089.stderr:31 Mar 14:59:44 ntpd[17171]: can't open /var/log/ntpstats/peerstats.20240331: Permission denied 2024-03-31T14:59:44.755 INFO:teuthology.orchestra.run.smithi089.stderr:31 Mar 14:59:44 ntpd[17171]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-31T14:59:44.755 INFO:teuthology.orchestra.run.smithi089.stderr:31 Mar 14:59:44 ntpd[17171]: can't open /var/log/ntpstats/rawstats.20240331: Permission denied 2024-03-31T14:59:44.755 INFO:teuthology.orchestra.run.smithi089.stderr:31 Mar 14:59:44 ntpd[17171]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-31T14:59:44.755 INFO:teuthology.orchestra.run.smithi089.stderr:31 Mar 14:59:44 ntpd[17171]: can't open /var/log/ntpstats/peerstats.20240331: Permission denied 2024-03-31T14:59:45.747 INFO:teuthology.orchestra.run.smithi016.stderr:31 Mar 14:59:45 ntpd[17052]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-31T14:59:45.747 INFO:teuthology.orchestra.run.smithi016.stderr:31 Mar 14:59:45 ntpd[17052]: can't open /var/log/ntpstats/rawstats.20240331: Permission denied 2024-03-31T14:59:45.747 INFO:teuthology.orchestra.run.smithi016.stderr:31 Mar 14:59:45 ntpd[17052]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-31T14:59:45.747 INFO:teuthology.orchestra.run.smithi016.stderr:31 Mar 14:59:45 ntpd[17052]: can't open /var/log/ntpstats/rawstats.20240331: Permission denied 2024-03-31T14:59:45.747 INFO:teuthology.orchestra.run.smithi016.stderr:31 Mar 14:59:45 ntpd[17052]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-31T14:59:45.748 INFO:teuthology.orchestra.run.smithi016.stderr:31 Mar 14:59:45 ntpd[17052]: can't open /var/log/ntpstats/rawstats.20240331: Permission denied 2024-03-31T14:59:45.748 INFO:teuthology.orchestra.run.smithi016.stderr:31 Mar 14:59:45 ntpd[17052]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-31T14:59:45.748 INFO:teuthology.orchestra.run.smithi016.stderr:31 Mar 14:59:45 ntpd[17052]: can't open /var/log/ntpstats/peerstats.20240331: Permission denied 2024-03-31T14:59:45.748 INFO:teuthology.orchestra.run.smithi016.stdout:31 Mar 14:59:45 ntpd[17052]: ntpd: time slew +0.000078 s 2024-03-31T14:59:45.748 INFO:teuthology.orchestra.run.smithi016.stdout:ntpd: time slew +0.000078s 2024-03-31T14:59:45.749 INFO:teuthology.orchestra.run.smithi016.stderr:31 Mar 14:59:45 ntpd[17052]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-03-31T14:59:45.749 INFO:teuthology.orchestra.run.smithi016.stderr:31 Mar 14:59:45 ntpd[17052]: can't open /var/log/ntpstats/loopstats.20240331: Permission denied 2024-03-31T14:59:45.754 INFO:teuthology.orchestra.run.smithi089.stderr:31 Mar 14:59:45 ntpd[17171]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-31T14:59:45.754 INFO:teuthology.orchestra.run.smithi089.stderr:31 Mar 14:59:45 ntpd[17171]: can't open /var/log/ntpstats/rawstats.20240331: Permission denied 2024-03-31T14:59:45.754 INFO:teuthology.orchestra.run.smithi089.stderr:31 Mar 14:59:45 ntpd[17171]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-31T14:59:45.754 INFO:teuthology.orchestra.run.smithi089.stderr:31 Mar 14:59:45 ntpd[17171]: can't open /var/log/ntpstats/rawstats.20240331: Permission denied 2024-03-31T14:59:45.754 INFO:teuthology.orchestra.run.smithi089.stderr:31 Mar 14:59:45 ntpd[17171]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-31T14:59:45.754 INFO:teuthology.orchestra.run.smithi089.stderr:31 Mar 14:59:45 ntpd[17171]: can't open /var/log/ntpstats/peerstats.20240331: Permission denied 2024-03-31T14:59:45.840 INFO:teuthology.orchestra.run.smithi016.stdout: remote refid st t when poll reach delay offset jitter 2024-03-31T14:59:45.840 INFO:teuthology.orchestra.run.smithi016.stdout:============================================================================== 2024-03-31T14:59:45.840 INFO:teuthology.orchestra.run.smithi016.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-03-31T14:59:45.840 INFO:teuthology.orchestra.run.smithi016.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-03-31T14:59:45.840 INFO:teuthology.orchestra.run.smithi016.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-03-31T14:59:45.840 INFO:teuthology.orchestra.run.smithi016.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-03-31T14:59:46.755 INFO:teuthology.orchestra.run.smithi089.stderr:31 Mar 14:59:46 ntpd[17171]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-31T14:59:46.755 INFO:teuthology.orchestra.run.smithi089.stderr:31 Mar 14:59:46 ntpd[17171]: can't open /var/log/ntpstats/rawstats.20240331: Permission denied 2024-03-31T14:59:46.755 INFO:teuthology.orchestra.run.smithi089.stderr:31 Mar 14:59:46 ntpd[17171]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-31T14:59:46.755 INFO:teuthology.orchestra.run.smithi089.stderr:31 Mar 14:59:46 ntpd[17171]: can't open /var/log/ntpstats/rawstats.20240331: Permission denied 2024-03-31T14:59:46.755 INFO:teuthology.orchestra.run.smithi089.stderr:31 Mar 14:59:46 ntpd[17171]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-31T14:59:46.755 INFO:teuthology.orchestra.run.smithi089.stderr:31 Mar 14:59:46 ntpd[17171]: can't open /var/log/ntpstats/rawstats.20240331: Permission denied 2024-03-31T14:59:46.755 INFO:teuthology.orchestra.run.smithi089.stderr:31 Mar 14:59:46 ntpd[17171]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-31T14:59:46.755 INFO:teuthology.orchestra.run.smithi089.stderr:31 Mar 14:59:46 ntpd[17171]: can't open /var/log/ntpstats/peerstats.20240331: Permission denied 2024-03-31T14:59:46.755 INFO:teuthology.orchestra.run.smithi089.stderr:31 Mar 14:59:46 ntpd[17171]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-31T14:59:46.756 INFO:teuthology.orchestra.run.smithi089.stderr:31 Mar 14:59:46 ntpd[17171]: can't open /var/log/ntpstats/rawstats.20240331: Permission denied 2024-03-31T14:59:46.756 INFO:teuthology.orchestra.run.smithi089.stderr:31 Mar 14:59:46 ntpd[17171]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-31T14:59:46.756 INFO:teuthology.orchestra.run.smithi089.stderr:31 Mar 14:59:46 ntpd[17171]: can't open /var/log/ntpstats/peerstats.20240331: Permission denied 2024-03-31T14:59:47.754 INFO:teuthology.orchestra.run.smithi089.stderr:31 Mar 14:59:47 ntpd[17171]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-31T14:59:47.754 INFO:teuthology.orchestra.run.smithi089.stderr:31 Mar 14:59:47 ntpd[17171]: can't open /var/log/ntpstats/rawstats.20240331: Permission denied 2024-03-31T14:59:47.754 INFO:teuthology.orchestra.run.smithi089.stderr:31 Mar 14:59:47 ntpd[17171]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-31T14:59:47.754 INFO:teuthology.orchestra.run.smithi089.stderr:31 Mar 14:59:47 ntpd[17171]: can't open /var/log/ntpstats/rawstats.20240331: Permission denied 2024-03-31T14:59:47.755 INFO:teuthology.orchestra.run.smithi089.stderr:31 Mar 14:59:47 ntpd[17171]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-31T14:59:47.755 INFO:teuthology.orchestra.run.smithi089.stderr:31 Mar 14:59:47 ntpd[17171]: can't open /var/log/ntpstats/peerstats.20240331: Permission denied 2024-03-31T14:59:48.755 INFO:teuthology.orchestra.run.smithi089.stderr:31 Mar 14:59:48 ntpd[17171]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-31T14:59:48.755 INFO:teuthology.orchestra.run.smithi089.stderr:31 Mar 14:59:48 ntpd[17171]: can't open /var/log/ntpstats/rawstats.20240331: Permission denied 2024-03-31T14:59:48.755 INFO:teuthology.orchestra.run.smithi089.stderr:31 Mar 14:59:48 ntpd[17171]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-31T14:59:48.755 INFO:teuthology.orchestra.run.smithi089.stderr:31 Mar 14:59:48 ntpd[17171]: can't open /var/log/ntpstats/rawstats.20240331: Permission denied 2024-03-31T14:59:48.755 INFO:teuthology.orchestra.run.smithi089.stderr:31 Mar 14:59:48 ntpd[17171]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-31T14:59:48.755 INFO:teuthology.orchestra.run.smithi089.stderr:31 Mar 14:59:48 ntpd[17171]: can't open /var/log/ntpstats/rawstats.20240331: Permission denied 2024-03-31T14:59:48.755 INFO:teuthology.orchestra.run.smithi089.stderr:31 Mar 14:59:48 ntpd[17171]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-31T14:59:48.755 INFO:teuthology.orchestra.run.smithi089.stderr:31 Mar 14:59:48 ntpd[17171]: can't open /var/log/ntpstats/peerstats.20240331: Permission denied 2024-03-31T14:59:48.756 INFO:teuthology.orchestra.run.smithi089.stdout:31 Mar 14:59:48 ntpd[17171]: ntpd: time slew +0.000595 s 2024-03-31T14:59:48.756 INFO:teuthology.orchestra.run.smithi089.stdout:ntpd: time slew +0.000595s 2024-03-31T14:59:48.756 INFO:teuthology.orchestra.run.smithi089.stderr:31 Mar 14:59:48 ntpd[17171]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-03-31T14:59:48.756 INFO:teuthology.orchestra.run.smithi089.stderr:31 Mar 14:59:48 ntpd[17171]: can't open /var/log/ntpstats/loopstats.20240331: Permission denied 2024-03-31T14:59:48.862 INFO:teuthology.orchestra.run.smithi089.stdout: remote refid st t when poll reach delay offset jitter 2024-03-31T14:59:48.862 INFO:teuthology.orchestra.run.smithi089.stdout:============================================================================== 2024-03-31T14:59:48.862 INFO:teuthology.orchestra.run.smithi089.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-03-31T14:59:48.862 INFO:teuthology.orchestra.run.smithi089.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-03-31T14:59:48.862 INFO:teuthology.orchestra.run.smithi089.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-03-31T14:59:48.862 INFO:teuthology.orchestra.run.smithi089.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-03-31T14:59:48.864 INFO:teuthology.run_tasks:Running task install... 2024-03-31T14:59:48.888 DEBUG:teuthology.task.install:project ceph 2024-03-31T14:59:48.888 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'd2271579d5e735cfd6d436b8349c004f06527cb0'}} 2024-03-31T14:59:48.888 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': 'd2271579d5e735cfd6d436b8349c004f06527cb0'} 2024-03-31T14:59:48.889 INFO:teuthology.task.install:Using flavor: default 2024-03-31T14:59:48.895 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-03-31T14:59:48.895 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']} 2024-03-31T14:59:48.895 DEBUG:teuthology.orchestra.run.smithi016:> sudo apt-key list | grep Ceph 2024-03-31T14:59:48.896 DEBUG:teuthology.orchestra.run.smithi089:> sudo apt-key list | grep Ceph 2024-03-31T14:59:49.004 INFO:teuthology.orchestra.run.smithi016.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2024-03-31T14:59:49.014 INFO:teuthology.orchestra.run.smithi089.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2024-03-31T14:59:49.057 INFO:teuthology.orchestra.run.smithi016.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-03-31T14:59:49.057 INFO:teuthology.orchestra.run.smithi016.stdout:uid [ unknown] Ceph.com (release key) 2024-03-31T14:59:49.058 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror on remote deb x86_64 2024-03-31T14:59:49.058 INFO:teuthology.task.install.deb:Installing system (non-project) packages: bison, flex, libelf-dev, libssl-dev, network-manager, iproute2, util-linux, dump, indent, libaio-dev, libtool-bin, uuid-dev, xfslibs-dev, postgresql, postgresql-client, postgresql-common, postgresql-contrib on remote deb x86_64 2024-03-31T14:59:49.058 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=d2271579d5e735cfd6d436b8349c004f06527cb0 2024-03-31T14:59:49.064 INFO:teuthology.orchestra.run.smithi089.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-03-31T14:59:49.064 INFO:teuthology.orchestra.run.smithi089.stdout:uid [ unknown] Ceph.com (release key) 2024-03-31T14:59:49.065 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror on remote deb x86_64 2024-03-31T14:59:49.065 INFO:teuthology.task.install.deb:Installing system (non-project) packages: bison, flex, libelf-dev, libssl-dev, network-manager, iproute2, util-linux, dump, indent, libaio-dev, libtool-bin, uuid-dev, xfslibs-dev, postgresql, postgresql-client, postgresql-common, postgresql-contrib on remote deb x86_64 2024-03-31T14:59:49.065 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=d2271579d5e735cfd6d436b8349c004f06527cb0 2024-03-31T14:59:49.287 INFO:teuthology.task.install.deb:Pulling from https://3.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240330.170739/d2271579d5e735cfd6d436b8349c004f06527cb0/ubuntu/jammy/flavors/default/ 2024-03-31T14:59:49.287 INFO:teuthology.task.install.deb:Package version is 19.0.0-2574-gd2271579-1jammy 2024-03-31T14:59:49.291 INFO:teuthology.task.install.deb:Pulling from https://3.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240330.170739/d2271579d5e735cfd6d436b8349c004f06527cb0/ubuntu/jammy/flavors/default/ 2024-03-31T14:59:49.292 INFO:teuthology.task.install.deb:Package version is 19.0.0-2574-gd2271579-1jammy 2024-03-31T14:59:49.406 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-03-31T14:59:49.406 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-03-31T14:59:49.426 DEBUG:teuthology.orchestra.run.smithi016:> sudo apt-get update 2024-03-31T14:59:49.469 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-03-31T14:59:49.469 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-03-31T14:59:49.489 DEBUG:teuthology.orchestra.run.smithi089:> sudo apt-get update 2024-03-31T14:59:49.624 INFO:teuthology.orchestra.run.smithi016.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-03-31T14:59:49.624 INFO:teuthology.orchestra.run.smithi016.stdout:Hit:2 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-03-31T14:59:49.634 INFO:teuthology.orchestra.run.smithi016.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-03-31T14:59:49.655 INFO:teuthology.orchestra.run.smithi016.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-03-31T14:59:49.668 INFO:teuthology.orchestra.run.smithi016.stdout:Ign:5 https://3.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240330.170739/d2271579d5e735cfd6d436b8349c004f06527cb0/ubuntu/jammy/flavors/default jammy InRelease 2024-03-31T14:59:49.680 INFO:teuthology.orchestra.run.smithi089.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-03-31T14:59:49.680 INFO:teuthology.orchestra.run.smithi089.stdout:Hit:2 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-03-31T14:59:49.690 INFO:teuthology.orchestra.run.smithi016.stdout:Get:6 https://3.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240330.170739/d2271579d5e735cfd6d436b8349c004f06527cb0/ubuntu/jammy/flavors/default jammy Release [7,662 B] 2024-03-31T14:59:49.691 INFO:teuthology.orchestra.run.smithi089.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-03-31T14:59:49.710 INFO:teuthology.orchestra.run.smithi089.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-03-31T14:59:49.711 INFO:teuthology.orchestra.run.smithi016.stdout:Ign:7 https://3.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240330.170739/d2271579d5e735cfd6d436b8349c004f06527cb0/ubuntu/jammy/flavors/default jammy Release.gpg 2024-03-31T14:59:49.724 INFO:teuthology.orchestra.run.smithi089.stdout:Ign:5 https://3.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240330.170739/d2271579d5e735cfd6d436b8349c004f06527cb0/ubuntu/jammy/flavors/default jammy InRelease 2024-03-31T14:59:49.734 INFO:teuthology.orchestra.run.smithi016.stdout:Get:8 https://3.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240330.170739/d2271579d5e735cfd6d436b8349c004f06527cb0/ubuntu/jammy/flavors/default jammy/main amd64 Packages [18.1 kB] 2024-03-31T14:59:49.746 INFO:teuthology.orchestra.run.smithi089.stdout:Get:6 https://3.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240330.170739/d2271579d5e735cfd6d436b8349c004f06527cb0/ubuntu/jammy/flavors/default jammy Release [7,662 B] 2024-03-31T14:59:49.768 INFO:teuthology.orchestra.run.smithi089.stdout:Ign:7 https://3.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240330.170739/d2271579d5e735cfd6d436b8349c004f06527cb0/ubuntu/jammy/flavors/default jammy Release.gpg 2024-03-31T14:59:49.787 INFO:teuthology.orchestra.run.smithi016.stdout:Get:9 https://3.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240330.170739/d2271579d5e735cfd6d436b8349c004f06527cb0/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,418 B] 2024-03-31T14:59:49.789 INFO:teuthology.orchestra.run.smithi089.stdout:Get:8 https://3.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240330.170739/d2271579d5e735cfd6d436b8349c004f06527cb0/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,418 B] 2024-03-31T14:59:49.822 INFO:teuthology.orchestra.run.smithi089.stdout:Get:9 https://3.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240330.170739/d2271579d5e735cfd6d436b8349c004f06527cb0/ubuntu/jammy/flavors/default jammy/main amd64 Packages [18.1 kB] 2024-03-31T14:59:50.229 INFO:teuthology.orchestra.run.smithi016.stdout:Fetched 29.2 kB in 1s (51.1 kB/s) 2024-03-31T14:59:50.265 INFO:teuthology.orchestra.run.smithi089.stdout:Fetched 29.2 kB in 1s (52.8 kB/s) 2024-03-31T14:59:51.220 INFO:teuthology.orchestra.run.smithi089.stdout:Reading package lists... 2024-03-31T14:59:51.230 INFO:teuthology.orchestra.run.smithi016.stdout:Reading package lists... 2024-03-31T14:59:51.234 DEBUG:teuthology.orchestra.run.smithi089:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=19.0.0-2574-gd2271579-1jammy cephadm=19.0.0-2574-gd2271579-1jammy ceph-mds=19.0.0-2574-gd2271579-1jammy ceph-mgr=19.0.0-2574-gd2271579-1jammy ceph-common=19.0.0-2574-gd2271579-1jammy ceph-fuse=19.0.0-2574-gd2271579-1jammy ceph-test=19.0.0-2574-gd2271579-1jammy ceph-volume=19.0.0-2574-gd2271579-1jammy radosgw=19.0.0-2574-gd2271579-1jammy python3-rados=19.0.0-2574-gd2271579-1jammy python3-rgw=19.0.0-2574-gd2271579-1jammy python3-cephfs=19.0.0-2574-gd2271579-1jammy python3-rbd=19.0.0-2574-gd2271579-1jammy libcephfs2=19.0.0-2574-gd2271579-1jammy libcephfs-dev=19.0.0-2574-gd2271579-1jammy librados2=19.0.0-2574-gd2271579-1jammy librbd1=19.0.0-2574-gd2271579-1jammy rbd-fuse=19.0.0-2574-gd2271579-1jammy python3-cephfs=19.0.0-2574-gd2271579-1jammy cephfs-shell=19.0.0-2574-gd2271579-1jammy cephfs-top=19.0.0-2574-gd2271579-1jammy cephfs-mirror=19.0.0-2574-gd2271579-1jammy 2024-03-31T14:59:51.244 DEBUG:teuthology.orchestra.run.smithi016:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=19.0.0-2574-gd2271579-1jammy cephadm=19.0.0-2574-gd2271579-1jammy ceph-mds=19.0.0-2574-gd2271579-1jammy ceph-mgr=19.0.0-2574-gd2271579-1jammy ceph-common=19.0.0-2574-gd2271579-1jammy ceph-fuse=19.0.0-2574-gd2271579-1jammy ceph-test=19.0.0-2574-gd2271579-1jammy ceph-volume=19.0.0-2574-gd2271579-1jammy radosgw=19.0.0-2574-gd2271579-1jammy python3-rados=19.0.0-2574-gd2271579-1jammy python3-rgw=19.0.0-2574-gd2271579-1jammy python3-cephfs=19.0.0-2574-gd2271579-1jammy python3-rbd=19.0.0-2574-gd2271579-1jammy libcephfs2=19.0.0-2574-gd2271579-1jammy libcephfs-dev=19.0.0-2574-gd2271579-1jammy librados2=19.0.0-2574-gd2271579-1jammy librbd1=19.0.0-2574-gd2271579-1jammy rbd-fuse=19.0.0-2574-gd2271579-1jammy python3-cephfs=19.0.0-2574-gd2271579-1jammy cephfs-shell=19.0.0-2574-gd2271579-1jammy cephfs-top=19.0.0-2574-gd2271579-1jammy cephfs-mirror=19.0.0-2574-gd2271579-1jammy 2024-03-31T14:59:51.307 INFO:teuthology.orchestra.run.smithi089.stdout:Reading package lists... 2024-03-31T14:59:51.321 INFO:teuthology.orchestra.run.smithi016.stdout:Reading package lists... 2024-03-31T14:59:51.477 INFO:teuthology.orchestra.run.smithi089.stdout:Building dependency tree... 2024-03-31T14:59:51.478 INFO:teuthology.orchestra.run.smithi089.stdout:Reading state information... 2024-03-31T14:59:51.498 INFO:teuthology.orchestra.run.smithi016.stdout:Building dependency tree... 2024-03-31T14:59:51.498 INFO:teuthology.orchestra.run.smithi016.stdout:Reading state information... 2024-03-31T14:59:51.645 INFO:teuthology.orchestra.run.smithi089.stdout:The following packages were automatically installed and are no longer required: 2024-03-31T14:59:51.646 INFO:teuthology.orchestra.run.smithi089.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-03-31T14:59:51.646 INFO:teuthology.orchestra.run.smithi089.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-03-31T14:59:51.646 INFO:teuthology.orchestra.run.smithi089.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-03-31T14:59:51.646 INFO:teuthology.orchestra.run.smithi089.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-03-31T14:59:51.646 INFO:teuthology.orchestra.run.smithi089.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-03-31T14:59:51.646 INFO:teuthology.orchestra.run.smithi089.stdout:Use 'sudo apt autoremove' to remove them. 2024-03-31T14:59:51.648 INFO:teuthology.orchestra.run.smithi089.stdout:The following additional packages will be installed: 2024-03-31T14:59:51.648 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-03-31T14:59:51.649 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2024-03-31T14:59:51.649 INFO:teuthology.orchestra.run.smithi089.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev liboath0 libonig5 2024-03-31T14:59:51.649 INFO:teuthology.orchestra.run.smithi089.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 libradosstriper1 2024-03-31T14:59:51.649 INFO:teuthology.orchestra.run.smithi089.stdout: librdkafka1 libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 2024-03-31T14:59:51.649 INFO:teuthology.orchestra.run.smithi089.stdout: lua-any lua-sec lua-socket lua5.1 luarocks nvme-cli pkg-config 2024-03-31T14:59:51.649 INFO:teuthology.orchestra.run.smithi089.stdout: python-asyncssh-doc python-babel-localedata python-pastedeploy-tpl 2024-03-31T14:59:51.649 INFO:teuthology.orchestra.run.smithi089.stdout: python3-asyncssh python3-babel python3-bcrypt python3-cachetools 2024-03-31T14:59:51.649 INFO:teuthology.orchestra.run.smithi089.stdout: python3-ceph-argparse python3-ceph-common python3-certifi python3-cheroot 2024-03-31T14:59:51.649 INFO:teuthology.orchestra.run.smithi089.stdout: python3-cherrypy3 python3-cmd2 python3-colorama python3-google-auth 2024-03-31T14:59:51.649 INFO:teuthology.orchestra.run.smithi089.stdout: python3-idna python3-iniconfig python3-jaraco.classes 2024-03-31T14:59:51.650 INFO:teuthology.orchestra.run.smithi089.stdout: python3-jaraco.collections python3-jaraco.functools python3-jaraco.text 2024-03-31T14:59:51.650 INFO:teuthology.orchestra.run.smithi089.stdout: python3-jinja2 python3-joblib python3-kubernetes python3-logutils 2024-03-31T14:59:51.650 INFO:teuthology.orchestra.run.smithi089.stdout: python3-mako python3-markupsafe python3-natsort python3-openssl 2024-03-31T14:59:51.650 INFO:teuthology.orchestra.run.smithi089.stdout: python3-paste python3-pastedeploy python3-pastescript python3-pecan 2024-03-31T14:59:51.650 INFO:teuthology.orchestra.run.smithi089.stdout: python3-pluggy python3-portend python3-prettytable python3-psutil python3-py 2024-03-31T14:59:51.650 INFO:teuthology.orchestra.run.smithi089.stdout: python3-pyasn1 python3-pyasn1-modules python3-pygments python3-pyinotify 2024-03-31T14:59:51.650 INFO:teuthology.orchestra.run.smithi089.stdout: python3-pyperclip python3-pytest python3-repoze.lru python3-requests 2024-03-31T14:59:51.650 INFO:teuthology.orchestra.run.smithi089.stdout: python3-requests-oauthlib python3-routes python3-rsa python3-simplegeneric 2024-03-31T14:59:51.650 INFO:teuthology.orchestra.run.smithi089.stdout: python3-simplejson python3-singledispatch python3-sklearn 2024-03-31T14:59:51.650 INFO:teuthology.orchestra.run.smithi089.stdout: python3-sklearn-lib python3-tempita python3-tempora python3-threadpoolctl 2024-03-31T14:59:51.650 INFO:teuthology.orchestra.run.smithi089.stdout: python3-toml python3-urllib3 python3-waitress python3-wcwidth python3-webob 2024-03-31T14:59:51.650 INFO:teuthology.orchestra.run.smithi089.stdout: python3-websocket python3-webtest python3-werkzeug python3-zc.lockfile 2024-03-31T14:59:51.650 INFO:teuthology.orchestra.run.smithi089.stdout: qttranslations5-l10n socat unzip xmlstarlet zip 2024-03-31T14:59:51.650 INFO:teuthology.orchestra.run.smithi089.stdout:Suggested packages: 2024-03-31T14:59:51.651 INFO:teuthology.orchestra.run.smithi089.stdout: python3-influxdb readline-doc python-jinja2-doc python3-beaker 2024-03-31T14:59:51.651 INFO:teuthology.orchestra.run.smithi089.stdout: python-mako-doc python-natsort-doc python-openssl-doc python3-openssl-dbg 2024-03-31T14:59:51.651 INFO:teuthology.orchestra.run.smithi089.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2024-03-31T14:59:51.651 INFO:teuthology.orchestra.run.smithi089.stdout: python-pecan-doc python-psutil-doc subversion python-pygments-doc 2024-03-31T14:59:51.651 INFO:teuthology.orchestra.run.smithi089.stdout: python-pyinotify-doc python3-socks python-requests-doc python3-dap 2024-03-31T14:59:51.651 INFO:teuthology.orchestra.run.smithi089.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2024-03-31T14:59:51.651 INFO:teuthology.orchestra.run.smithi089.stdout: python-webtest-doc python-werkzeug-doc python3-watchdog 2024-03-31T14:59:51.651 INFO:teuthology.orchestra.run.smithi089.stdout:Recommended packages: 2024-03-31T14:59:51.651 INFO:teuthology.orchestra.run.smithi089.stdout: btrfs-tools 2024-03-31T14:59:51.676 INFO:teuthology.orchestra.run.smithi016.stdout:The following packages were automatically installed and are no longer required: 2024-03-31T14:59:51.676 INFO:teuthology.orchestra.run.smithi016.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-03-31T14:59:51.676 INFO:teuthology.orchestra.run.smithi016.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-03-31T14:59:51.676 INFO:teuthology.orchestra.run.smithi016.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-03-31T14:59:51.677 INFO:teuthology.orchestra.run.smithi016.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-03-31T14:59:51.677 INFO:teuthology.orchestra.run.smithi016.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-03-31T14:59:51.677 INFO:teuthology.orchestra.run.smithi016.stdout:Use 'sudo apt autoremove' to remove them. 2024-03-31T14:59:51.679 INFO:teuthology.orchestra.run.smithi016.stdout:The following additional packages will be installed: 2024-03-31T14:59:51.679 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-03-31T14:59:51.679 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2024-03-31T14:59:51.679 INFO:teuthology.orchestra.run.smithi016.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev liboath0 libonig5 2024-03-31T14:59:51.679 INFO:teuthology.orchestra.run.smithi016.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 libradosstriper1 2024-03-31T14:59:51.679 INFO:teuthology.orchestra.run.smithi016.stdout: librdkafka1 libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 2024-03-31T14:59:51.679 INFO:teuthology.orchestra.run.smithi016.stdout: lua-any lua-sec lua-socket lua5.1 luarocks nvme-cli pkg-config 2024-03-31T14:59:51.680 INFO:teuthology.orchestra.run.smithi016.stdout: python-asyncssh-doc python-babel-localedata python-pastedeploy-tpl 2024-03-31T14:59:51.680 INFO:teuthology.orchestra.run.smithi016.stdout: python3-asyncssh python3-babel python3-bcrypt python3-cachetools 2024-03-31T14:59:51.680 INFO:teuthology.orchestra.run.smithi016.stdout: python3-ceph-argparse python3-ceph-common python3-certifi python3-cheroot 2024-03-31T14:59:51.680 INFO:teuthology.orchestra.run.smithi016.stdout: python3-cherrypy3 python3-cmd2 python3-colorama python3-google-auth 2024-03-31T14:59:51.680 INFO:teuthology.orchestra.run.smithi016.stdout: python3-idna python3-iniconfig python3-jaraco.classes 2024-03-31T14:59:51.680 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jaraco.collections python3-jaraco.functools python3-jaraco.text 2024-03-31T14:59:51.680 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jinja2 python3-joblib python3-kubernetes python3-logutils 2024-03-31T14:59:51.680 INFO:teuthology.orchestra.run.smithi016.stdout: python3-mako python3-markupsafe python3-natsort python3-openssl 2024-03-31T14:59:51.680 INFO:teuthology.orchestra.run.smithi016.stdout: python3-paste python3-pastedeploy python3-pastescript python3-pecan 2024-03-31T14:59:51.680 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pluggy python3-portend python3-prettytable python3-psutil python3-py 2024-03-31T14:59:51.680 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pyasn1 python3-pyasn1-modules python3-pygments python3-pyinotify 2024-03-31T14:59:51.680 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pyperclip python3-pytest python3-repoze.lru python3-requests 2024-03-31T14:59:51.680 INFO:teuthology.orchestra.run.smithi016.stdout: python3-requests-oauthlib python3-routes python3-rsa python3-simplegeneric 2024-03-31T14:59:51.680 INFO:teuthology.orchestra.run.smithi016.stdout: python3-simplejson python3-singledispatch python3-sklearn 2024-03-31T14:59:51.680 INFO:teuthology.orchestra.run.smithi016.stdout: python3-sklearn-lib python3-tempita python3-tempora python3-threadpoolctl 2024-03-31T14:59:51.681 INFO:teuthology.orchestra.run.smithi016.stdout: python3-toml python3-urllib3 python3-waitress python3-wcwidth python3-webob 2024-03-31T14:59:51.681 INFO:teuthology.orchestra.run.smithi016.stdout: python3-websocket python3-webtest python3-werkzeug python3-zc.lockfile 2024-03-31T14:59:51.681 INFO:teuthology.orchestra.run.smithi016.stdout: qttranslations5-l10n socat unzip xmlstarlet zip 2024-03-31T14:59:51.681 INFO:teuthology.orchestra.run.smithi016.stdout:Suggested packages: 2024-03-31T14:59:51.681 INFO:teuthology.orchestra.run.smithi016.stdout: python3-influxdb readline-doc python-jinja2-doc python3-beaker 2024-03-31T14:59:51.681 INFO:teuthology.orchestra.run.smithi016.stdout: python-mako-doc python-natsort-doc python-openssl-doc python3-openssl-dbg 2024-03-31T14:59:51.681 INFO:teuthology.orchestra.run.smithi016.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2024-03-31T14:59:51.681 INFO:teuthology.orchestra.run.smithi016.stdout: python-pecan-doc python-psutil-doc subversion python-pygments-doc 2024-03-31T14:59:51.681 INFO:teuthology.orchestra.run.smithi016.stdout: python-pyinotify-doc python3-socks python-requests-doc python3-dap 2024-03-31T14:59:51.682 INFO:teuthology.orchestra.run.smithi016.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2024-03-31T14:59:51.682 INFO:teuthology.orchestra.run.smithi016.stdout: python-webtest-doc python-werkzeug-doc python3-watchdog 2024-03-31T14:59:51.682 INFO:teuthology.orchestra.run.smithi016.stdout:Recommended packages: 2024-03-31T14:59:51.682 INFO:teuthology.orchestra.run.smithi016.stdout: btrfs-tools 2024-03-31T14:59:51.799 INFO:teuthology.orchestra.run.smithi089.stdout:The following NEW packages will be installed: 2024-03-31T14:59:51.800 INFO:teuthology.orchestra.run.smithi089.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-03-31T14:59:51.800 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-03-31T14:59:51.800 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm 2024-03-31T14:59:51.800 INFO:teuthology.orchestra.run.smithi089.stdout: cephfs-mirror cephfs-shell cephfs-top jq libcephfs-dev libcephfs2 2024-03-31T14:59:51.800 INFO:teuthology.orchestra.run.smithi089.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev liboath0 libonig5 2024-03-31T14:59:51.800 INFO:teuthology.orchestra.run.smithi089.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 librados2 2024-03-31T14:59:51.800 INFO:teuthology.orchestra.run.smithi089.stdout: libradosstriper1 librbd1 librdkafka1 libreadline-dev librgw2 2024-03-31T14:59:51.800 INFO:teuthology.orchestra.run.smithi089.stdout: libsqlite3-mod-ceph libthrift-0.16.0 lua-any lua-sec lua-socket lua5.1 2024-03-31T14:59:51.800 INFO:teuthology.orchestra.run.smithi089.stdout: luarocks nvme-cli pkg-config python-asyncssh-doc python-babel-localedata 2024-03-31T14:59:51.800 INFO:teuthology.orchestra.run.smithi089.stdout: python-pastedeploy-tpl python3-asyncssh python3-babel python3-bcrypt 2024-03-31T14:59:51.801 INFO:teuthology.orchestra.run.smithi089.stdout: python3-cachetools python3-ceph-argparse python3-ceph-common python3-cephfs 2024-03-31T14:59:51.801 INFO:teuthology.orchestra.run.smithi089.stdout: python3-certifi python3-cheroot python3-cherrypy3 python3-cmd2 2024-03-31T14:59:51.801 INFO:teuthology.orchestra.run.smithi089.stdout: python3-colorama python3-google-auth python3-idna python3-iniconfig 2024-03-31T14:59:51.801 INFO:teuthology.orchestra.run.smithi089.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-03-31T14:59:51.801 INFO:teuthology.orchestra.run.smithi089.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-03-31T14:59:51.801 INFO:teuthology.orchestra.run.smithi089.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-03-31T14:59:51.801 INFO:teuthology.orchestra.run.smithi089.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-03-31T14:59:51.801 INFO:teuthology.orchestra.run.smithi089.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-03-31T14:59:51.801 INFO:teuthology.orchestra.run.smithi089.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-03-31T14:59:51.801 INFO:teuthology.orchestra.run.smithi089.stdout: python3-pygments python3-pyinotify python3-pyperclip python3-pytest 2024-03-31T14:59:51.801 INFO:teuthology.orchestra.run.smithi089.stdout: python3-rados python3-rbd python3-repoze.lru python3-requests 2024-03-31T14:59:51.801 INFO:teuthology.orchestra.run.smithi089.stdout: python3-requests-oauthlib python3-rgw python3-routes python3-rsa 2024-03-31T14:59:51.802 INFO:teuthology.orchestra.run.smithi089.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2024-03-31T14:59:51.802 INFO:teuthology.orchestra.run.smithi089.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2024-03-31T14:59:51.802 INFO:teuthology.orchestra.run.smithi089.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2024-03-31T14:59:51.802 INFO:teuthology.orchestra.run.smithi089.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2024-03-31T14:59:51.802 INFO:teuthology.orchestra.run.smithi089.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n radosgw rbd-fuse 2024-03-31T14:59:51.802 INFO:teuthology.orchestra.run.smithi089.stdout: socat unzip xmlstarlet zip 2024-03-31T14:59:51.838 INFO:teuthology.orchestra.run.smithi016.stdout:The following NEW packages will be installed: 2024-03-31T14:59:51.838 INFO:teuthology.orchestra.run.smithi016.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-03-31T14:59:51.838 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-03-31T14:59:51.838 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm 2024-03-31T14:59:51.838 INFO:teuthology.orchestra.run.smithi016.stdout: cephfs-mirror cephfs-shell cephfs-top jq libcephfs-dev libcephfs2 2024-03-31T14:59:51.838 INFO:teuthology.orchestra.run.smithi016.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev liboath0 libonig5 2024-03-31T14:59:51.838 INFO:teuthology.orchestra.run.smithi016.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 librados2 2024-03-31T14:59:51.838 INFO:teuthology.orchestra.run.smithi016.stdout: libradosstriper1 librbd1 librdkafka1 libreadline-dev librgw2 2024-03-31T14:59:51.838 INFO:teuthology.orchestra.run.smithi016.stdout: libsqlite3-mod-ceph libthrift-0.16.0 lua-any lua-sec lua-socket lua5.1 2024-03-31T14:59:51.838 INFO:teuthology.orchestra.run.smithi016.stdout: luarocks nvme-cli pkg-config python-asyncssh-doc python-babel-localedata 2024-03-31T14:59:51.838 INFO:teuthology.orchestra.run.smithi016.stdout: python-pastedeploy-tpl python3-asyncssh python3-babel python3-bcrypt 2024-03-31T14:59:51.839 INFO:teuthology.orchestra.run.smithi016.stdout: python3-cachetools python3-ceph-argparse python3-ceph-common python3-cephfs 2024-03-31T14:59:51.839 INFO:teuthology.orchestra.run.smithi016.stdout: python3-certifi python3-cheroot python3-cherrypy3 python3-cmd2 2024-03-31T14:59:51.839 INFO:teuthology.orchestra.run.smithi016.stdout: python3-colorama python3-google-auth python3-idna python3-iniconfig 2024-03-31T14:59:51.839 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-03-31T14:59:51.839 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-03-31T14:59:51.839 INFO:teuthology.orchestra.run.smithi016.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-03-31T14:59:51.839 INFO:teuthology.orchestra.run.smithi016.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-03-31T14:59:51.839 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-03-31T14:59:51.839 INFO:teuthology.orchestra.run.smithi016.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-03-31T14:59:51.839 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pygments python3-pyinotify python3-pyperclip python3-pytest 2024-03-31T14:59:51.840 INFO:teuthology.orchestra.run.smithi016.stdout: python3-rados python3-rbd python3-repoze.lru python3-requests 2024-03-31T14:59:51.840 INFO:teuthology.orchestra.run.smithi016.stdout: python3-requests-oauthlib python3-rgw python3-routes python3-rsa 2024-03-31T14:59:51.840 INFO:teuthology.orchestra.run.smithi016.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2024-03-31T14:59:51.840 INFO:teuthology.orchestra.run.smithi016.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2024-03-31T14:59:51.840 INFO:teuthology.orchestra.run.smithi016.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2024-03-31T14:59:51.840 INFO:teuthology.orchestra.run.smithi016.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2024-03-31T14:59:51.840 INFO:teuthology.orchestra.run.smithi016.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n radosgw rbd-fuse 2024-03-31T14:59:51.840 INFO:teuthology.orchestra.run.smithi016.stdout: socat unzip xmlstarlet zip 2024-03-31T14:59:51.871 INFO:teuthology.orchestra.run.smithi089.stdout:0 upgraded, 124 newly installed, 0 to remove and 301 not upgraded. 2024-03-31T14:59:51.872 INFO:teuthology.orchestra.run.smithi089.stdout:Need to get 179 MB of archives. 2024-03-31T14:59:51.872 INFO:teuthology.orchestra.run.smithi089.stdout:After this operation, 821 MB of additional disk space will be used. 2024-03-31T14:59:51.872 INFO:teuthology.orchestra.run.smithi089.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyperclip all 1.8.2-2 [11.5 kB] 2024-03-31T14:59:51.892 INFO:teuthology.orchestra.run.smithi089.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2024-03-31T14:59:51.903 INFO:teuthology.orchestra.run.smithi089.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cmd2 all 2.3.3-1 [124 kB] 2024-03-31T14:59:51.951 INFO:teuthology.orchestra.run.smithi089.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2024-03-31T14:59:51.975 INFO:teuthology.orchestra.run.smithi089.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2024-03-31T14:59:51.977 INFO:teuthology.orchestra.run.smithi089.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2024-03-31T14:59:51.986 INFO:teuthology.orchestra.run.smithi016.stdout:0 upgraded, 124 newly installed, 0 to remove and 301 not upgraded. 2024-03-31T14:59:51.986 INFO:teuthology.orchestra.run.smithi016.stdout:Need to get 179 MB of archives. 2024-03-31T14:59:51.987 INFO:teuthology.orchestra.run.smithi016.stdout:After this operation, 821 MB of additional disk space will be used. 2024-03-31T14:59:51.987 INFO:teuthology.orchestra.run.smithi016.stdout:Get:1 https://3.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240330.170739/d2271579d5e735cfd6d436b8349c004f06527cb0/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 19.0.0-2574-gd2271579-1jammy [3,553 kB] 2024-03-31T14:59:51.991 INFO:teuthology.orchestra.run.smithi089.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2024-03-31T14:59:52.022 INFO:teuthology.orchestra.run.smithi089.stdout:Get:8 https://3.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240330.170739/d2271579d5e735cfd6d436b8349c004f06527cb0/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 19.0.0-2574-gd2271579-1jammy [3,553 kB] 2024-03-31T14:59:52.034 INFO:teuthology.orchestra.run.smithi016.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyperclip all 1.8.2-2 [11.5 kB] 2024-03-31T14:59:52.035 INFO:teuthology.orchestra.run.smithi089.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2024-03-31T14:59:52.037 INFO:teuthology.orchestra.run.smithi089.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2024-03-31T14:59:52.045 INFO:teuthology.orchestra.run.smithi089.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2024-03-31T14:59:52.048 INFO:teuthology.orchestra.run.smithi089.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2024-03-31T14:59:52.049 INFO:teuthology.orchestra.run.smithi089.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2024-03-31T14:59:52.056 INFO:teuthology.orchestra.run.smithi089.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy/main amd64 libreadline-dev amd64 8.1.2-1 [166 kB] 2024-03-31T14:59:52.058 INFO:teuthology.orchestra.run.smithi089.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblua5.3-dev amd64 5.3.6-1build1 [167 kB] 2024-03-31T14:59:52.060 INFO:teuthology.orchestra.run.smithi089.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua5.1 amd64 5.1.5-8.1build4 [94.6 kB] 2024-03-31T14:59:52.061 INFO:teuthology.orchestra.run.smithi089.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-any all 27ubuntu1 [5,034 B] 2024-03-31T14:59:52.061 INFO:teuthology.orchestra.run.smithi089.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy/main amd64 zip amd64 3.0-12build2 [176 kB] 2024-03-31T14:59:52.076 INFO:teuthology.orchestra.run.smithi089.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 unzip amd64 6.0-26ubuntu3.2 [175 kB] 2024-03-31T14:59:52.078 INFO:teuthology.orchestra.run.smithi089.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/universe amd64 luarocks all 3.8.0+dfsg1-1 [140 kB] 2024-03-31T14:59:52.080 INFO:teuthology.orchestra.run.smithi089.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2024-03-31T14:59:52.081 INFO:teuthology.orchestra.run.smithi089.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2024-03-31T14:59:52.096 INFO:teuthology.orchestra.run.smithi089.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2024-03-31T14:59:52.097 INFO:teuthology.orchestra.run.smithi089.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2024-03-31T14:59:52.098 INFO:teuthology.orchestra.run.smithi089.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2024-03-31T14:59:52.098 INFO:teuthology.orchestra.run.smithi089.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2024-03-31T14:59:52.099 INFO:teuthology.orchestra.run.smithi089.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2024-03-31T14:59:52.099 INFO:teuthology.orchestra.run.smithi089.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2024-03-31T14:59:52.116 INFO:teuthology.orchestra.run.smithi089.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2024-03-31T14:59:52.116 INFO:teuthology.orchestra.run.smithi089.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2024-03-31T14:59:52.117 INFO:teuthology.orchestra.run.smithi016.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2024-03-31T14:59:52.118 INFO:teuthology.orchestra.run.smithi089.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2024-03-31T14:59:52.119 INFO:teuthology.orchestra.run.smithi089.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2024-03-31T14:59:52.135 INFO:teuthology.orchestra.run.smithi089.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2024-03-31T14:59:52.136 INFO:teuthology.orchestra.run.smithi089.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2024-03-31T14:59:52.137 INFO:teuthology.orchestra.run.smithi089.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-03-31T14:59:52.138 INFO:teuthology.orchestra.run.smithi089.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2024-03-31T14:59:52.138 INFO:teuthology.orchestra.run.smithi089.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webob all 1:1.8.6-1.1 [85.2 kB] 2024-03-31T14:59:52.139 INFO:teuthology.orchestra.run.smithi089.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2024-03-31T14:59:52.155 INFO:teuthology.orchestra.run.smithi089.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2024-03-31T14:59:52.156 INFO:teuthology.orchestra.run.smithi089.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2024-03-31T14:59:52.160 INFO:teuthology.orchestra.run.smithi089.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2024-03-31T14:59:52.161 INFO:teuthology.orchestra.run.smithi089.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2024-03-31T14:59:52.162 INFO:teuthology.orchestra.run.smithi016.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cmd2 all 2.3.3-1 [124 kB] 2024-03-31T14:59:52.175 INFO:teuthology.orchestra.run.smithi089.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2024-03-31T14:59:52.176 INFO:teuthology.orchestra.run.smithi089.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2024-03-31T14:59:52.177 INFO:teuthology.orchestra.run.smithi089.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2024-03-31T14:59:52.179 INFO:teuthology.orchestra.run.smithi089.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-idna all 3.3-1 [49.3 kB] 2024-03-31T14:59:52.180 INFO:teuthology.orchestra.run.smithi089.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-urllib3 all 1.26.5-1~exp1ubuntu0.1 [98.2 kB] 2024-03-31T14:59:52.181 INFO:teuthology.orchestra.run.smithi089.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2024-03-31T14:59:52.194 INFO:teuthology.orchestra.run.smithi089.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.1 [181 kB] 2024-03-31T14:59:52.197 INFO:teuthology.orchestra.run.smithi089.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2024-03-31T14:59:52.198 INFO:teuthology.orchestra.run.smithi089.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2024-03-31T14:59:52.199 INFO:teuthology.orchestra.run.smithi089.stdout:Get:52 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2024-03-31T14:59:52.214 INFO:teuthology.orchestra.run.smithi089.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2024-03-31T14:59:52.238 INFO:teuthology.orchestra.run.smithi016.stdout:Get:5 https://3.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240330.170739/d2271579d5e735cfd6d436b8349c004f06527cb0/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 19.0.0-2574-gd2271579-1jammy [3,224 kB] 2024-03-31T14:59:52.265 INFO:teuthology.orchestra.run.smithi089.stdout:Get:54 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2024-03-31T14:59:52.266 INFO:teuthology.orchestra.run.smithi089.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-jinja2 all 3.0.3-1ubuntu0.1 [108 kB] 2024-03-31T14:59:52.267 INFO:teuthology.orchestra.run.smithi089.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-03-31T14:59:52.268 INFO:teuthology.orchestra.run.smithi089.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2024-03-31T14:59:52.269 INFO:teuthology.orchestra.run.smithi089.stdout:Get:58 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2024-03-31T14:59:52.290 INFO:teuthology.orchestra.run.smithi089.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2024-03-31T14:59:52.294 INFO:teuthology.orchestra.run.smithi089.stdout:Get:60 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2024-03-31T14:59:52.294 INFO:teuthology.orchestra.run.smithi089.stdout:Get:61 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2024-03-31T14:59:52.313 INFO:teuthology.orchestra.run.smithi089.stdout:Get:62 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2024-03-31T14:59:52.313 INFO:teuthology.orchestra.run.smithi089.stdout:Get:63 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2024-03-31T14:59:52.314 INFO:teuthology.orchestra.run.smithi089.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2024-03-31T14:59:52.315 INFO:teuthology.orchestra.run.smithi089.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2024-03-31T14:59:52.316 INFO:teuthology.orchestra.run.smithi089.stdout:Get:66 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2024-03-31T14:59:52.317 INFO:teuthology.orchestra.run.smithi089.stdout:Get:67 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2024-03-31T14:59:52.317 INFO:teuthology.orchestra.run.smithi089.stdout:Get:68 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2024-03-31T14:59:52.334 INFO:teuthology.orchestra.run.smithi089.stdout:Get:69 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2024-03-31T14:59:52.338 INFO:teuthology.orchestra.run.smithi089.stdout:Get:70 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2024-03-31T14:59:52.340 INFO:teuthology.orchestra.run.smithi089.stdout:Get:71 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2024-03-31T14:59:52.342 INFO:teuthology.orchestra.run.smithi089.stdout:Get:72 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2024-03-31T14:59:52.343 INFO:teuthology.orchestra.run.smithi089.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2024-03-31T14:59:52.354 INFO:teuthology.orchestra.run.smithi089.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2024-03-31T14:59:52.356 INFO:teuthology.orchestra.run.smithi016.stdout:Get:6 https://3.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240330.170739/d2271579d5e735cfd6d436b8349c004f06527cb0/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 19.0.0-2574-gd2271579-1jammy [712 kB] 2024-03-31T14:59:52.357 INFO:teuthology.orchestra.run.smithi089.stdout:Get:75 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-colorama all 0.4.4-1 [24.5 kB] 2024-03-31T14:59:52.358 INFO:teuthology.orchestra.run.smithi089.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-socket amd64 3.0~rc1+git+ac3201d-6 [78.9 kB] 2024-03-31T14:59:52.359 INFO:teuthology.orchestra.run.smithi089.stdout:Get:77 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-sec amd64 1.0.2-1 [37.6 kB] 2024-03-31T14:59:52.360 INFO:teuthology.orchestra.run.smithi089.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.1 [474 kB] 2024-03-31T14:59:52.364 INFO:teuthology.orchestra.run.smithi016.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2024-03-31T14:59:52.373 INFO:teuthology.orchestra.run.smithi089.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy/main amd64 pkg-config amd64 0.29.2-1ubuntu3 [48.2 kB] 2024-03-31T14:59:52.374 INFO:teuthology.orchestra.run.smithi089.stdout:Get:80 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2024-03-31T14:59:52.378 INFO:teuthology.orchestra.run.smithi089.stdout:Get:81 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2024-03-31T14:59:52.379 INFO:teuthology.orchestra.run.smithi089.stdout:Get:82 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-03-31T14:59:52.379 INFO:teuthology.orchestra.run.smithi089.stdout:Get:83 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2024-03-31T14:59:52.385 INFO:teuthology.orchestra.run.smithi016.stdout:Get:8 https://3.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240330.170739/d2271579d5e735cfd6d436b8349c004f06527cb0/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 19.0.0-2574-gd2271579-1jammy [356 kB] 2024-03-31T14:59:52.393 INFO:teuthology.orchestra.run.smithi089.stdout:Get:84 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2024-03-31T14:59:52.397 INFO:teuthology.orchestra.run.smithi089.stdout:Get:85 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2024-03-31T14:59:52.397 INFO:teuthology.orchestra.run.smithi089.stdout:Get:86 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2024-03-31T14:59:52.398 INFO:teuthology.orchestra.run.smithi016.stdout:Get:9 https://3.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240330.170739/d2271579d5e735cfd6d436b8349c004f06527cb0/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 19.0.0-2574-gd2271579-1jammy [32.8 kB] 2024-03-31T14:59:52.398 INFO:teuthology.orchestra.run.smithi016.stdout:Get:10 https://3.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240330.170739/d2271579d5e735cfd6d436b8349c004f06527cb0/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 19.0.0-2574-gd2271579-1jammy [180 kB] 2024-03-31T14:59:52.404 INFO:teuthology.orchestra.run.smithi089.stdout:Get:87 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2024-03-31T14:59:52.404 INFO:teuthology.orchestra.run.smithi016.stdout:Get:11 https://3.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240330.170739/d2271579d5e735cfd6d436b8349c004f06527cb0/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 19.0.0-2574-gd2271579-1jammy [66.7 kB] 2024-03-31T14:59:52.405 INFO:teuthology.orchestra.run.smithi089.stdout:Get:88 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2024-03-31T14:59:52.407 INFO:teuthology.orchestra.run.smithi016.stdout:Get:12 https://3.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240330.170739/d2271579d5e735cfd6d436b8349c004f06527cb0/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 19.0.0-2574-gd2271579-1jammy [330 kB] 2024-03-31T14:59:52.413 INFO:teuthology.orchestra.run.smithi089.stdout:Get:89 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2024-03-31T14:59:52.415 INFO:teuthology.orchestra.run.smithi089.stdout:Get:90 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2024-03-31T14:59:52.416 INFO:teuthology.orchestra.run.smithi089.stdout:Get:91 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2024-03-31T14:59:52.427 INFO:teuthology.orchestra.run.smithi016.stdout:Get:13 https://3.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240330.170739/d2271579d5e735cfd6d436b8349c004f06527cb0/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 19.0.0-2574-gd2271579-1jammy [6,660 kB] 2024-03-31T14:59:52.458 INFO:teuthology.orchestra.run.smithi016.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2024-03-31T14:59:52.471 INFO:teuthology.orchestra.run.smithi016.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2024-03-31T14:59:52.522 INFO:teuthology.orchestra.run.smithi016.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2024-03-31T14:59:52.690 INFO:teuthology.orchestra.run.smithi016.stdout:Get:17 https://3.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240330.170739/d2271579d5e735cfd6d436b8349c004f06527cb0/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 19.0.0-2574-gd2271579-1jammy [112 kB] 2024-03-31T14:59:52.691 INFO:teuthology.orchestra.run.smithi016.stdout:Get:18 https://3.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240330.170739/d2271579d5e735cfd6d436b8349c004f06527cb0/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 19.0.0-2574-gd2271579-1jammy [466 kB] 2024-03-31T14:59:52.696 INFO:teuthology.orchestra.run.smithi016.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2024-03-31T14:59:52.706 INFO:teuthology.orchestra.run.smithi016.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2024-03-31T14:59:52.718 INFO:teuthology.orchestra.run.smithi089.stdout:Get:92 https://3.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240330.170739/d2271579d5e735cfd6d436b8349c004f06527cb0/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 19.0.0-2574-gd2271579-1jammy [3,224 kB] 2024-03-31T14:59:52.723 INFO:teuthology.orchestra.run.smithi016.stdout:Get:21 https://3.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240330.170739/d2271579d5e735cfd6d436b8349c004f06527cb0/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 19.0.0-2574-gd2271579-1jammy [25.3 MB] 2024-03-31T14:59:52.733 INFO:teuthology.orchestra.run.smithi016.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2024-03-31T14:59:52.742 INFO:teuthology.orchestra.run.smithi016.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2024-03-31T14:59:52.906 INFO:teuthology.orchestra.run.smithi016.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2024-03-31T14:59:53.115 INFO:teuthology.orchestra.run.smithi089.stdout:Get:93 https://3.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240330.170739/d2271579d5e735cfd6d436b8349c004f06527cb0/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 19.0.0-2574-gd2271579-1jammy [712 kB] 2024-03-31T14:59:53.184 INFO:teuthology.orchestra.run.smithi089.stdout:Get:94 https://3.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240330.170739/d2271579d5e735cfd6d436b8349c004f06527cb0/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 19.0.0-2574-gd2271579-1jammy [356 kB] 2024-03-31T14:59:53.230 INFO:teuthology.orchestra.run.smithi089.stdout:Get:95 https://3.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240330.170739/d2271579d5e735cfd6d436b8349c004f06527cb0/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 19.0.0-2574-gd2271579-1jammy [32.8 kB] 2024-03-31T14:59:53.231 INFO:teuthology.orchestra.run.smithi089.stdout:Get:96 https://3.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240330.170739/d2271579d5e735cfd6d436b8349c004f06527cb0/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 19.0.0-2574-gd2271579-1jammy [180 kB] 2024-03-31T14:59:53.243 INFO:teuthology.orchestra.run.smithi089.stdout:Get:97 https://3.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240330.170739/d2271579d5e735cfd6d436b8349c004f06527cb0/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 19.0.0-2574-gd2271579-1jammy [66.7 kB] 2024-03-31T14:59:53.249 INFO:teuthology.orchestra.run.smithi089.stdout:Get:98 https://3.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240330.170739/d2271579d5e735cfd6d436b8349c004f06527cb0/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 19.0.0-2574-gd2271579-1jammy [330 kB] 2024-03-31T14:59:53.282 INFO:teuthology.orchestra.run.smithi089.stdout:Get:99 https://3.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240330.170739/d2271579d5e735cfd6d436b8349c004f06527cb0/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 19.0.0-2574-gd2271579-1jammy [6,660 kB] 2024-03-31T14:59:53.497 INFO:teuthology.orchestra.run.smithi016.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/main amd64 libreadline-dev amd64 8.1.2-1 [166 kB] 2024-03-31T14:59:53.513 INFO:teuthology.orchestra.run.smithi016.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblua5.3-dev amd64 5.3.6-1build1 [167 kB] 2024-03-31T14:59:53.529 INFO:teuthology.orchestra.run.smithi016.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua5.1 amd64 5.1.5-8.1build4 [94.6 kB] 2024-03-31T14:59:53.538 INFO:teuthology.orchestra.run.smithi016.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-any all 27ubuntu1 [5,034 B] 2024-03-31T14:59:53.538 INFO:teuthology.orchestra.run.smithi016.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/main amd64 zip amd64 3.0-12build2 [176 kB] 2024-03-31T14:59:53.555 INFO:teuthology.orchestra.run.smithi016.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 unzip amd64 6.0-26ubuntu3.2 [175 kB] 2024-03-31T14:59:53.592 INFO:teuthology.orchestra.run.smithi016.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/universe amd64 luarocks all 3.8.0+dfsg1-1 [140 kB] 2024-03-31T14:59:53.605 INFO:teuthology.orchestra.run.smithi016.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2024-03-31T14:59:53.609 INFO:teuthology.orchestra.run.smithi016.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2024-03-31T14:59:53.610 INFO:teuthology.orchestra.run.smithi016.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2024-03-31T14:59:53.780 INFO:teuthology.orchestra.run.smithi016.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2024-03-31T14:59:53.862 INFO:teuthology.orchestra.run.smithi016.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2024-03-31T14:59:53.870 INFO:teuthology.orchestra.run.smithi089.stdout:Get:100 https://3.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240330.170739/d2271579d5e735cfd6d436b8349c004f06527cb0/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 19.0.0-2574-gd2271579-1jammy [112 kB] 2024-03-31T14:59:53.876 INFO:teuthology.orchestra.run.smithi089.stdout:Get:101 https://3.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240330.170739/d2271579d5e735cfd6d436b8349c004f06527cb0/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 19.0.0-2574-gd2271579-1jammy [466 kB] 2024-03-31T14:59:53.884 INFO:teuthology.orchestra.run.smithi016.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2024-03-31T14:59:53.919 INFO:teuthology.orchestra.run.smithi089.stdout:Get:102 https://3.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240330.170739/d2271579d5e735cfd6d436b8349c004f06527cb0/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 19.0.0-2574-gd2271579-1jammy [25.3 MB] 2024-03-31T14:59:53.943 INFO:teuthology.orchestra.run.smithi016.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2024-03-31T14:59:53.969 INFO:teuthology.orchestra.run.smithi016.stdout:Get:39 https://3.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240330.170739/d2271579d5e735cfd6d436b8349c004f06527cb0/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 19.0.0-2574-gd2271579-1jammy [5,086 kB] 2024-03-31T14:59:53.987 INFO:teuthology.orchestra.run.smithi016.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2024-03-31T14:59:54.009 INFO:teuthology.orchestra.run.smithi016.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2024-03-31T14:59:54.031 INFO:teuthology.orchestra.run.smithi016.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2024-03-31T14:59:54.202 INFO:teuthology.orchestra.run.smithi016.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2024-03-31T14:59:54.218 INFO:teuthology.orchestra.run.smithi016.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2024-03-31T14:59:54.225 INFO:teuthology.orchestra.run.smithi016.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2024-03-31T14:59:54.230 INFO:teuthology.orchestra.run.smithi016.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2024-03-31T14:59:54.315 INFO:teuthology.orchestra.run.smithi016.stdout:Get:47 https://3.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240330.170739/d2271579d5e735cfd6d436b8349c004f06527cb0/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 19.0.0-2574-gd2271579-1jammy [244 kB] 2024-03-31T14:59:54.317 INFO:teuthology.orchestra.run.smithi016.stdout:Get:48 https://3.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240330.170739/d2271579d5e735cfd6d436b8349c004f06527cb0/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 19.0.0-2574-gd2271579-1jammy [124 kB] 2024-03-31T14:59:54.318 INFO:teuthology.orchestra.run.smithi016.stdout:Get:49 https://3.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240330.170739/d2271579d5e735cfd6d436b8349c004f06527cb0/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 19.0.0-2574-gd2271579-1jammy [1,526 kB] 2024-03-31T14:59:54.369 INFO:teuthology.orchestra.run.smithi016.stdout:Get:50 https://3.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240330.170739/d2271579d5e735cfd6d436b8349c004f06527cb0/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 19.0.0-2574-gd2271579-1jammy [6,259 kB] 2024-03-31T14:59:54.415 INFO:teuthology.orchestra.run.smithi016.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-03-31T14:59:54.497 INFO:teuthology.orchestra.run.smithi016.stdout:Get:52 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2024-03-31T14:59:54.513 INFO:teuthology.orchestra.run.smithi016.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webob all 1:1.8.6-1.1 [85.2 kB] 2024-03-31T14:59:54.698 INFO:teuthology.orchestra.run.smithi016.stdout:Get:54 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2024-03-31T14:59:54.734 INFO:teuthology.orchestra.run.smithi016.stdout:Get:55 https://3.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240330.170739/d2271579d5e735cfd6d436b8349c004f06527cb0/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 19.0.0-2574-gd2271579-1jammy [22.8 MB] 2024-03-31T14:59:54.749 INFO:teuthology.orchestra.run.smithi016.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2024-03-31T14:59:54.762 INFO:teuthology.orchestra.run.smithi016.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2024-03-31T14:59:54.920 INFO:teuthology.orchestra.run.smithi016.stdout:Get:58 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2024-03-31T14:59:54.921 INFO:teuthology.orchestra.run.smithi016.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2024-03-31T14:59:54.926 INFO:teuthology.orchestra.run.smithi016.stdout:Get:60 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2024-03-31T14:59:54.930 INFO:teuthology.orchestra.run.smithi016.stdout:Get:61 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2024-03-31T14:59:54.945 INFO:teuthology.orchestra.run.smithi016.stdout:Get:62 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2024-03-31T14:59:55.129 INFO:teuthology.orchestra.run.smithi016.stdout:Get:63 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-idna all 3.3-1 [49.3 kB] 2024-03-31T14:59:55.392 INFO:teuthology.orchestra.run.smithi016.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-urllib3 all 1.26.5-1~exp1ubuntu0.1 [98.2 kB] 2024-03-31T14:59:55.520 INFO:teuthology.orchestra.run.smithi016.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2024-03-31T14:59:55.562 INFO:teuthology.orchestra.run.smithi016.stdout:Get:66 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.1 [181 kB] 2024-03-31T14:59:55.650 INFO:teuthology.orchestra.run.smithi016.stdout:Get:67 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2024-03-31T14:59:55.660 INFO:teuthology.orchestra.run.smithi016.stdout:Get:68 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2024-03-31T14:59:55.673 INFO:teuthology.orchestra.run.smithi016.stdout:Get:69 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2024-03-31T14:59:55.714 INFO:teuthology.orchestra.run.smithi016.stdout:Get:70 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2024-03-31T14:59:55.797 INFO:teuthology.orchestra.run.smithi089.stdout:Get:103 https://3.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240330.170739/d2271579d5e735cfd6d436b8349c004f06527cb0/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 19.0.0-2574-gd2271579-1jammy [5,086 kB] 2024-03-31T14:59:55.975 INFO:teuthology.orchestra.run.smithi016.stdout:Get:71 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2024-03-31T14:59:55.976 INFO:teuthology.orchestra.run.smithi016.stdout:Get:72 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-jinja2 all 3.0.3-1ubuntu0.1 [108 kB] 2024-03-31T14:59:55.977 INFO:teuthology.orchestra.run.smithi016.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-03-31T14:59:56.103 INFO:teuthology.orchestra.run.smithi016.stdout:Get:74 https://3.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240330.170739/d2271579d5e735cfd6d436b8349c004f06527cb0/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 19.0.0-2574-gd2271579-1jammy [14.0 kB] 2024-03-31T14:59:56.104 INFO:teuthology.orchestra.run.smithi016.stdout:Get:75 https://3.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240330.170739/d2271579d5e735cfd6d436b8349c004f06527cb0/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 19.0.0-2574-gd2271579-1jammy [870 kB] 2024-03-31T14:59:56.142 INFO:teuthology.orchestra.run.smithi016.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2024-03-31T14:59:56.148 INFO:teuthology.orchestra.run.smithi016.stdout:Get:77 https://3.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240330.170739/d2271579d5e735cfd6d436b8349c004f06527cb0/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 19.0.0-2574-gd2271579-1jammy [2,435 kB] 2024-03-31T14:59:56.152 INFO:teuthology.orchestra.run.smithi089.stdout:Get:104 https://3.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240330.170739/d2271579d5e735cfd6d436b8349c004f06527cb0/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 19.0.0-2574-gd2271579-1jammy [244 kB] 2024-03-31T14:59:56.171 INFO:teuthology.orchestra.run.smithi089.stdout:Get:105 https://3.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240330.170739/d2271579d5e735cfd6d436b8349c004f06527cb0/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 19.0.0-2574-gd2271579-1jammy [124 kB] 2024-03-31T14:59:56.175 INFO:teuthology.orchestra.run.smithi089.stdout:Get:106 https://3.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240330.170739/d2271579d5e735cfd6d436b8349c004f06527cb0/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 19.0.0-2574-gd2271579-1jammy [1,526 kB] 2024-03-31T14:59:56.286 INFO:teuthology.orchestra.run.smithi089.stdout:Get:107 https://3.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240330.170739/d2271579d5e735cfd6d436b8349c004f06527cb0/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 19.0.0-2574-gd2271579-1jammy [6,259 kB] 2024-03-31T14:59:56.297 INFO:teuthology.orchestra.run.smithi016.stdout:Get:78 https://3.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240330.170739/d2271579d5e735cfd6d436b8349c004f06527cb0/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 19.0.0-2574-gd2271579-1jammy [791 kB] 2024-03-31T14:59:56.346 INFO:teuthology.orchestra.run.smithi016.stdout:Get:79 https://3.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240330.170739/d2271579d5e735cfd6d436b8349c004f06527cb0/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 19.0.0-2574-gd2271579-1jammy [147 kB] 2024-03-31T14:59:56.353 INFO:teuthology.orchestra.run.smithi016.stdout:Get:80 https://3.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240330.170739/d2271579d5e735cfd6d436b8349c004f06527cb0/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 19.0.0-2574-gd2271579-1jammy [2,313 kB] 2024-03-31T14:59:56.486 INFO:teuthology.orchestra.run.smithi016.stdout:Get:81 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2024-03-31T14:59:56.496 INFO:teuthology.orchestra.run.smithi016.stdout:Get:82 https://3.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240330.170739/d2271579d5e735cfd6d436b8349c004f06527cb0/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 19.0.0-2574-gd2271579-1jammy [8,625 kB] 2024-03-31T14:59:56.729 INFO:teuthology.orchestra.run.smithi089.stdout:Get:108 https://3.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240330.170739/d2271579d5e735cfd6d436b8349c004f06527cb0/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 19.0.0-2574-gd2271579-1jammy [22.8 MB] 2024-03-31T14:59:56.887 INFO:teuthology.orchestra.run.smithi016.stdout:Get:83 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2024-03-31T14:59:56.897 INFO:teuthology.orchestra.run.smithi016.stdout:Get:84 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2024-03-31T14:59:56.898 INFO:teuthology.orchestra.run.smithi016.stdout:Get:85 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2024-03-31T14:59:56.967 INFO:teuthology.orchestra.run.smithi016.stdout:Get:86 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2024-03-31T14:59:56.967 INFO:teuthology.orchestra.run.smithi016.stdout:Get:87 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2024-03-31T14:59:56.968 INFO:teuthology.orchestra.run.smithi016.stdout:Get:88 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2024-03-31T14:59:56.969 INFO:teuthology.orchestra.run.smithi016.stdout:Get:89 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2024-03-31T14:59:56.970 INFO:teuthology.orchestra.run.smithi016.stdout:Get:90 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2024-03-31T14:59:56.971 INFO:teuthology.orchestra.run.smithi016.stdout:Get:91 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2024-03-31T14:59:57.030 INFO:teuthology.orchestra.run.smithi016.stdout:Get:92 https://3.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240330.170739/d2271579d5e735cfd6d436b8349c004f06527cb0/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 19.0.0-2574-gd2271579-1jammy [14.2 kB] 2024-03-31T14:59:57.031 INFO:teuthology.orchestra.run.smithi016.stdout:Get:93 https://3.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240330.170739/d2271579d5e735cfd6d436b8349c004f06527cb0/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 19.0.0-2574-gd2271579-1jammy [50.3 MB] 2024-03-31T14:59:57.136 INFO:teuthology.orchestra.run.smithi016.stdout:Get:94 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2024-03-31T14:59:57.340 INFO:teuthology.orchestra.run.smithi016.stdout:Get:95 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2024-03-31T14:59:57.663 INFO:teuthology.orchestra.run.smithi016.stdout:Get:96 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2024-03-31T14:59:57.709 INFO:teuthology.orchestra.run.smithi016.stdout:Get:97 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2024-03-31T14:59:57.737 INFO:teuthology.orchestra.run.smithi016.stdout:Get:98 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2024-03-31T14:59:57.745 INFO:teuthology.orchestra.run.smithi016.stdout:Get:99 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2024-03-31T14:59:57.791 INFO:teuthology.orchestra.run.smithi016.stdout:Get:100 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2024-03-31T14:59:57.817 INFO:teuthology.orchestra.run.smithi016.stdout:Get:101 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-colorama all 0.4.4-1 [24.5 kB] 2024-03-31T14:59:57.818 INFO:teuthology.orchestra.run.smithi016.stdout:Get:102 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-socket amd64 3.0~rc1+git+ac3201d-6 [78.9 kB] 2024-03-31T14:59:57.826 INFO:teuthology.orchestra.run.smithi016.stdout:Get:103 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-sec amd64 1.0.2-1 [37.6 kB] 2024-03-31T14:59:57.829 INFO:teuthology.orchestra.run.smithi016.stdout:Get:104 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.1 [474 kB] 2024-03-31T14:59:58.025 INFO:teuthology.orchestra.run.smithi016.stdout:Get:105 http://archive.ubuntu.com/ubuntu jammy/main amd64 pkg-config amd64 0.29.2-1ubuntu3 [48.2 kB] 2024-03-31T14:59:58.284 INFO:teuthology.orchestra.run.smithi016.stdout:Get:106 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2024-03-31T14:59:58.313 INFO:teuthology.orchestra.run.smithi089.stdout:Get:109 https://3.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240330.170739/d2271579d5e735cfd6d436b8349c004f06527cb0/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 19.0.0-2574-gd2271579-1jammy [14.0 kB] 2024-03-31T14:59:58.314 INFO:teuthology.orchestra.run.smithi089.stdout:Get:110 https://3.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240330.170739/d2271579d5e735cfd6d436b8349c004f06527cb0/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 19.0.0-2574-gd2271579-1jammy [870 kB] 2024-03-31T14:59:58.366 INFO:teuthology.orchestra.run.smithi089.stdout:Get:111 https://3.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240330.170739/d2271579d5e735cfd6d436b8349c004f06527cb0/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 19.0.0-2574-gd2271579-1jammy [2,435 kB] 2024-03-31T14:59:58.547 INFO:teuthology.orchestra.run.smithi016.stdout:Get:107 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2024-03-31T14:59:58.549 INFO:teuthology.orchestra.run.smithi016.stdout:Get:108 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-03-31T14:59:58.555 INFO:teuthology.orchestra.run.smithi089.stdout:Get:112 https://3.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240330.170739/d2271579d5e735cfd6d436b8349c004f06527cb0/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 19.0.0-2574-gd2271579-1jammy [791 kB] 2024-03-31T14:59:58.565 INFO:teuthology.orchestra.run.smithi016.stdout:Get:109 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2024-03-31T14:59:58.570 INFO:teuthology.orchestra.run.smithi016.stdout:Get:110 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2024-03-31T14:59:58.597 INFO:teuthology.orchestra.run.smithi089.stdout:Get:113 https://3.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240330.170739/d2271579d5e735cfd6d436b8349c004f06527cb0/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 19.0.0-2574-gd2271579-1jammy [147 kB] 2024-03-31T14:59:58.608 INFO:teuthology.orchestra.run.smithi016.stdout:Get:111 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2024-03-31T14:59:58.609 INFO:teuthology.orchestra.run.smithi089.stdout:Get:114 https://3.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240330.170739/d2271579d5e735cfd6d436b8349c004f06527cb0/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 19.0.0-2574-gd2271579-1jammy [2,313 kB] 2024-03-31T14:59:58.622 INFO:teuthology.orchestra.run.smithi016.stdout:Get:112 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2024-03-31T14:59:58.714 INFO:teuthology.orchestra.run.smithi016.stdout:Get:113 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2024-03-31T14:59:58.715 INFO:teuthology.orchestra.run.smithi016.stdout:Get:114 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2024-03-31T14:59:58.717 INFO:teuthology.orchestra.run.smithi016.stdout:Get:115 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2024-03-31T14:59:58.806 INFO:teuthology.orchestra.run.smithi089.stdout:Get:115 https://3.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240330.170739/d2271579d5e735cfd6d436b8349c004f06527cb0/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 19.0.0-2574-gd2271579-1jammy [8,625 kB] 2024-03-31T14:59:58.897 INFO:teuthology.orchestra.run.smithi016.stdout:Get:116 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2024-03-31T14:59:59.162 INFO:teuthology.orchestra.run.smithi016.stdout:Get:117 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2024-03-31T14:59:59.525 INFO:teuthology.orchestra.run.smithi089.stdout:Get:116 https://3.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240330.170739/d2271579d5e735cfd6d436b8349c004f06527cb0/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 19.0.0-2574-gd2271579-1jammy [14.2 kB] 2024-03-31T14:59:59.525 INFO:teuthology.orchestra.run.smithi089.stdout:Get:117 https://3.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240330.170739/d2271579d5e735cfd6d436b8349c004f06527cb0/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 19.0.0-2574-gd2271579-1jammy [50.3 MB] 2024-03-31T14:59:59.978 INFO:teuthology.orchestra.run.smithi016.stdout:Get:118 https://3.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240330.170739/d2271579d5e735cfd6d436b8349c004f06527cb0/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 19.0.0-2574-gd2271579-1jammy [121 kB] 2024-03-31T14:59:59.978 INFO:teuthology.orchestra.run.smithi016.stdout:Get:119 https://3.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240330.170739/d2271579d5e735cfd6d436b8349c004f06527cb0/ubuntu/jammy/flavors/default jammy/main amd64 cephfs-mirror amd64 19.0.0-2574-gd2271579-1jammy [232 kB] 2024-03-31T14:59:59.981 INFO:teuthology.orchestra.run.smithi016.stdout:Get:120 https://3.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240330.170739/d2271579d5e735cfd6d436b8349c004f06527cb0/ubuntu/jammy/flavors/default jammy/main amd64 cephfs-shell all 19.0.0-2574-gd2271579-1jammy [34.0 kB] 2024-03-31T14:59:59.982 INFO:teuthology.orchestra.run.smithi016.stdout:Get:121 https://3.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240330.170739/d2271579d5e735cfd6d436b8349c004f06527cb0/ubuntu/jammy/flavors/default jammy/main amd64 cephfs-top all 19.0.0-2574-gd2271579-1jammy [23.4 kB] 2024-03-31T14:59:59.982 INFO:teuthology.orchestra.run.smithi016.stdout:Get:122 https://3.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240330.170739/d2271579d5e735cfd6d436b8349c004f06527cb0/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 19.0.0-2574-gd2271579-1jammy [38.9 kB] 2024-03-31T14:59:59.984 INFO:teuthology.orchestra.run.smithi016.stdout:Get:123 https://3.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240330.170739/d2271579d5e735cfd6d436b8349c004f06527cb0/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 19.0.0-2574-gd2271579-1jammy [13.2 MB] 2024-03-31T15:00:00.726 INFO:teuthology.orchestra.run.smithi016.stdout:Get:124 https://3.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240330.170739/d2271579d5e735cfd6d436b8349c004f06527cb0/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 19.0.0-2574-gd2271579-1jammy [91.2 kB] 2024-03-31T15:00:01.398 INFO:teuthology.orchestra.run.smithi016.stdout:Fetched 179 MB in 9s (20.2 MB/s) 2024-03-31T15:00:01.623 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-pyperclip. 2024-03-31T15:00:01.691 INFO:teuthology.orchestra.run.smithi016.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 132923 files and directories currently installed.) 2024-03-31T15:00:01.694 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../000-python3-pyperclip_1.8.2-2_all.deb ... 2024-03-31T15:00:01.730 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-pyperclip (1.8.2-2) ... 2024-03-31T15:00:01.970 INFO:teuthology.orchestra.run.smithi089.stdout:Get:118 https://3.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240330.170739/d2271579d5e735cfd6d436b8349c004f06527cb0/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 19.0.0-2574-gd2271579-1jammy [121 kB] 2024-03-31T15:00:01.970 INFO:teuthology.orchestra.run.smithi089.stdout:Get:119 https://3.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240330.170739/d2271579d5e735cfd6d436b8349c004f06527cb0/ubuntu/jammy/flavors/default jammy/main amd64 cephfs-mirror amd64 19.0.0-2574-gd2271579-1jammy [232 kB] 2024-03-31T15:00:01.973 INFO:teuthology.orchestra.run.smithi089.stdout:Get:120 https://3.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240330.170739/d2271579d5e735cfd6d436b8349c004f06527cb0/ubuntu/jammy/flavors/default jammy/main amd64 cephfs-shell all 19.0.0-2574-gd2271579-1jammy [34.0 kB] 2024-03-31T15:00:01.973 INFO:teuthology.orchestra.run.smithi089.stdout:Get:121 https://3.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240330.170739/d2271579d5e735cfd6d436b8349c004f06527cb0/ubuntu/jammy/flavors/default jammy/main amd64 cephfs-top all 19.0.0-2574-gd2271579-1jammy [23.4 kB] 2024-03-31T15:00:01.973 INFO:teuthology.orchestra.run.smithi089.stdout:Get:122 https://3.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240330.170739/d2271579d5e735cfd6d436b8349c004f06527cb0/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 19.0.0-2574-gd2271579-1jammy [38.9 kB] 2024-03-31T15:00:01.974 INFO:teuthology.orchestra.run.smithi089.stdout:Get:123 https://3.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240330.170739/d2271579d5e735cfd6d436b8349c004f06527cb0/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 19.0.0-2574-gd2271579-1jammy [13.2 MB] 2024-03-31T15:00:02.057 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-wcwidth. 2024-03-31T15:00:02.072 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../001-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2024-03-31T15:00:02.108 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2024-03-31T15:00:02.428 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-cmd2. 2024-03-31T15:00:02.438 INFO:teuthology.orchestra.run.smithi089.stdout:Get:124 https://3.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240330.170739/d2271579d5e735cfd6d436b8349c004f06527cb0/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 19.0.0-2574-gd2271579-1jammy [91.2 kB] 2024-03-31T15:00:02.443 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../002-python3-cmd2_2.3.3-1_all.deb ... 2024-03-31T15:00:02.486 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-cmd2 (2.3.3-1) ... 2024-03-31T15:00:02.847 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2024-03-31T15:00:02.862 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../003-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2024-03-31T15:00:02.947 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-03-31T15:00:03.077 INFO:teuthology.orchestra.run.smithi089.stdout:Fetched 179 MB in 11s (16.8 MB/s) 2024-03-31T15:00:03.231 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package python3-pyperclip. 2024-03-31T15:00:03.259 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-03-31T15:00:03.275 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../004-libdouble-conversion3_3.1.7-4_amd64.deb ... 2024-03-31T15:00:03.296 INFO:teuthology.orchestra.run.smithi089.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 132923 files and directories currently installed.) 2024-03-31T15:00:03.300 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../000-python3-pyperclip_1.8.2-2_all.deb ... 2024-03-31T15:00:03.317 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2024-03-31T15:00:03.346 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking python3-pyperclip (1.8.2-2) ... 2024-03-31T15:00:03.623 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package python3-wcwidth. 2024-03-31T15:00:03.993 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../001-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2024-03-31T15:00:03.993 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2024-03-31T15:00:03.993 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-03-31T15:00:03.994 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../005-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2024-03-31T15:00:03.994 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-03-31T15:00:04.018 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package python3-cmd2. 2024-03-31T15:00:04.031 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../002-python3-cmd2_2.3.3-1_all.deb ... 2024-03-31T15:00:04.076 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking python3-cmd2 (2.3.3-1) ... 2024-03-31T15:00:04.132 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-03-31T15:00:04.148 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../006-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-03-31T15:00:04.191 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-03-31T15:00:04.429 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2024-03-31T15:00:04.443 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../003-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2024-03-31T15:00:04.530 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-03-31T15:00:04.594 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-03-31T15:00:04.610 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../007-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-03-31T15:00:04.652 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-03-31T15:00:05.014 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-03-31T15:00:05.029 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../008-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-03-31T15:00:05.072 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-03-31T15:00:05.208 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-03-31T15:00:05.223 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../004-libdouble-conversion3_3.1.7-4_amd64.deb ... 2024-03-31T15:00:05.266 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2024-03-31T15:00:05.367 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2024-03-31T15:00:05.382 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../009-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2024-03-31T15:00:05.417 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-03-31T15:00:05.619 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-03-31T15:00:05.634 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../005-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2024-03-31T15:00:05.669 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-03-31T15:00:05.762 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package librados2. 2024-03-31T15:00:05.777 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../010-librados2_19.0.0-2574-gd2271579-1jammy_amd64.deb ... 2024-03-31T15:00:05.820 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking librados2 (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:00:06.089 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-03-31T15:00:06.104 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../006-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-03-31T15:00:06.156 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-03-31T15:00:06.240 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package librbd1. 2024-03-31T15:00:06.255 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../011-librbd1_19.0.0-2574-gd2271579-1jammy_amd64.deb ... 2024-03-31T15:00:06.298 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking librbd1 (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:00:06.576 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-03-31T15:00:06.590 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../007-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-03-31T15:00:06.634 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-03-31T15:00:06.884 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package libcephfs2. 2024-03-31T15:00:06.899 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../012-libcephfs2_19.0.0-2574-gd2271579-1jammy_amd64.deb ... 2024-03-31T15:00:06.951 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking libcephfs2 (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:00:06.996 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-03-31T15:00:07.011 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../008-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-03-31T15:00:07.046 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-03-31T15:00:07.212 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-rados. 2024-03-31T15:00:07.227 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../013-python3-rados_19.0.0-2574-gd2271579-1jammy_amd64.deb ... 2024-03-31T15:00:07.270 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-rados (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:00:07.366 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2024-03-31T15:00:07.380 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../009-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2024-03-31T15:00:07.424 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-03-31T15:00:07.590 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-03-31T15:00:07.605 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../014-python3-ceph-argparse_19.0.0-2574-gd2271579-1jammy_all.deb ... 2024-03-31T15:00:07.665 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-ceph-argparse (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:00:07.752 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package librados2. 2024-03-31T15:00:07.767 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../010-librados2_19.0.0-2574-gd2271579-1jammy_amd64.deb ... 2024-03-31T15:00:07.810 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking librados2 (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:00:07.900 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-cephfs. 2024-03-31T15:00:07.916 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../015-python3-cephfs_19.0.0-2574-gd2271579-1jammy_amd64.deb ... 2024-03-31T15:00:07.958 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-cephfs (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:00:08.220 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-ceph-common. 2024-03-31T15:00:08.235 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../016-python3-ceph-common_19.0.0-2574-gd2271579-1jammy_all.deb ... 2024-03-31T15:00:08.254 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package librbd1. 2024-03-31T15:00:08.269 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../011-librbd1_19.0.0-2574-gd2271579-1jammy_amd64.deb ... 2024-03-31T15:00:08.270 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-ceph-common (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:00:08.313 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking librbd1 (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:00:08.598 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-prettytable. 2024-03-31T15:00:08.614 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../017-python3-prettytable_2.5.0-2_all.deb ... 2024-03-31T15:00:08.656 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2024-03-31T15:00:08.749 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package libcephfs2. 2024-03-31T15:00:08.763 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../012-libcephfs2_19.0.0-2574-gd2271579-1jammy_amd64.deb ... 2024-03-31T15:00:08.807 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking libcephfs2 (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:00:08.892 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-rbd. 2024-03-31T15:00:08.908 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../018-python3-rbd_19.0.0-2574-gd2271579-1jammy_amd64.deb ... 2024-03-31T15:00:08.967 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-rbd (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:00:09.060 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package python3-rados. 2024-03-31T15:00:09.074 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../013-python3-rados_19.0.0-2574-gd2271579-1jammy_amd64.deb ... 2024-03-31T15:00:09.126 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking python3-rados (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:00:09.287 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-03-31T15:00:09.302 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../019-librdkafka1_1.8.0-1build1_amd64.deb ... 2024-03-31T15:00:09.337 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2024-03-31T15:00:09.430 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-03-31T15:00:09.443 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../014-python3-ceph-argparse_19.0.0-2574-gd2271579-1jammy_all.deb ... 2024-03-31T15:00:09.488 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking python3-ceph-argparse (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:00:09.599 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package libreadline-dev:amd64. 2024-03-31T15:00:09.613 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../020-libreadline-dev_8.1.2-1_amd64.deb ... 2024-03-31T15:00:09.656 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking libreadline-dev:amd64 (8.1.2-1) ... 2024-03-31T15:00:09.740 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package python3-cephfs. 2024-03-31T15:00:09.754 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../015-python3-cephfs_19.0.0-2574-gd2271579-1jammy_amd64.deb ... 2024-03-31T15:00:09.823 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking python3-cephfs (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:00:09.984 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package liblua5.3-dev:amd64. 2024-03-31T15:00:10.000 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../021-liblua5.3-dev_5.3.6-1build1_amd64.deb ... 2024-03-31T15:00:10.042 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-03-31T15:00:10.134 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package python3-ceph-common. 2024-03-31T15:00:10.149 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../016-python3-ceph-common_19.0.0-2574-gd2271579-1jammy_all.deb ... 2024-03-31T15:00:10.334 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking python3-ceph-common (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:00:10.346 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package lua5.1. 2024-03-31T15:00:10.362 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../022-lua5.1_5.1.5-8.1build4_amd64.deb ... 2024-03-31T15:00:10.404 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking lua5.1 (5.1.5-8.1build4) ... 2024-03-31T15:00:10.679 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package python3-prettytable. 2024-03-31T15:00:10.691 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package lua-any. 2024-03-31T15:00:10.693 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../017-python3-prettytable_2.5.0-2_all.deb ... 2024-03-31T15:00:10.706 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../023-lua-any_27ubuntu1_all.deb ... 2024-03-31T15:00:10.737 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2024-03-31T15:00:10.740 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking lua-any (27ubuntu1) ... 2024-03-31T15:00:10.968 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package zip. 2024-03-31T15:00:10.982 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package python3-rbd. 2024-03-31T15:00:10.984 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../024-zip_3.0-12build2_amd64.deb ... 2024-03-31T15:00:10.996 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../018-python3-rbd_19.0.0-2574-gd2271579-1jammy_amd64.deb ... 2024-03-31T15:00:11.026 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking zip (3.0-12build2) ... 2024-03-31T15:00:11.032 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking python3-rbd (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:00:11.360 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-03-31T15:00:11.372 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package unzip. 2024-03-31T15:00:11.374 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../019-librdkafka1_1.8.0-1build1_amd64.deb ... 2024-03-31T15:00:11.387 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../025-unzip_6.0-26ubuntu3.2_amd64.deb ... 2024-03-31T15:00:11.418 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2024-03-31T15:00:11.430 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking unzip (6.0-26ubuntu3.2) ... 2024-03-31T15:00:11.704 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package libreadline-dev:amd64. 2024-03-31T15:00:11.718 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../020-libreadline-dev_8.1.2-1_amd64.deb ... 2024-03-31T15:00:11.762 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking libreadline-dev:amd64 (8.1.2-1) ... 2024-03-31T15:00:11.767 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package luarocks. 2024-03-31T15:00:11.783 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../026-luarocks_3.8.0+dfsg1-1_all.deb ... 2024-03-31T15:00:11.817 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking luarocks (3.8.0+dfsg1-1) ... 2024-03-31T15:00:12.099 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package liblua5.3-dev:amd64. 2024-03-31T15:00:12.113 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../021-liblua5.3-dev_5.3.6-1build1_amd64.deb ... 2024-03-31T15:00:12.157 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-03-31T15:00:12.217 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package librgw2. 2024-03-31T15:00:12.231 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../027-librgw2_19.0.0-2574-gd2271579-1jammy_amd64.deb ... 2024-03-31T15:00:12.275 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking librgw2 (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:00:12.452 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package lua5.1. 2024-03-31T15:00:12.467 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../022-lua5.1_5.1.5-8.1build4_amd64.deb ... 2024-03-31T15:00:12.501 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking lua5.1 (5.1.5-8.1build4) ... 2024-03-31T15:00:12.744 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-rgw. 2024-03-31T15:00:12.759 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../028-python3-rgw_19.0.0-2574-gd2271579-1jammy_amd64.deb ... 2024-03-31T15:00:12.794 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-rgw (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:00:12.838 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package lua-any. 2024-03-31T15:00:12.853 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../023-lua-any_27ubuntu1_all.deb ... 2024-03-31T15:00:12.896 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking lua-any (27ubuntu1) ... 2024-03-31T15:00:13.097 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package liboath0:amd64. 2024-03-31T15:00:13.113 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../029-liboath0_2.6.7-3build1_amd64.deb ... 2024-03-31T15:00:13.149 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package zip. 2024-03-31T15:00:13.155 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2024-03-31T15:00:13.163 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../024-zip_3.0-12build2_amd64.deb ... 2024-03-31T15:00:13.207 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking zip (3.0-12build2) ... 2024-03-31T15:00:13.494 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package libradosstriper1. 2024-03-31T15:00:13.507 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../030-libradosstriper1_19.0.0-2574-gd2271579-1jammy_amd64.deb ... 2024-03-31T15:00:13.510 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package unzip. 2024-03-31T15:00:13.525 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../025-unzip_6.0-26ubuntu3.2_amd64.deb ... 2024-03-31T15:00:13.550 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking libradosstriper1 (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:00:13.560 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking unzip (6.0-26ubuntu3.2) ... 2024-03-31T15:00:13.921 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package ceph-common. 2024-03-31T15:00:13.936 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../031-ceph-common_19.0.0-2574-gd2271579-1jammy_amd64.deb ... 2024-03-31T15:00:13.948 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package luarocks. 2024-03-31T15:00:13.963 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../026-luarocks_3.8.0+dfsg1-1_all.deb ... 2024-03-31T15:00:13.970 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking ceph-common (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:00:14.006 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking luarocks (3.8.0+dfsg1-1) ... 2024-03-31T15:00:14.397 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package librgw2. 2024-03-31T15:00:14.411 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../027-librgw2_19.0.0-2574-gd2271579-1jammy_amd64.deb ... 2024-03-31T15:00:14.455 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking librgw2 (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:00:14.966 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package python3-rgw. 2024-03-31T15:00:14.980 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../028-python3-rgw_19.0.0-2574-gd2271579-1jammy_amd64.deb ... 2024-03-31T15:00:15.016 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking python3-rgw (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:00:15.067 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package ceph-base. 2024-03-31T15:00:15.083 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../032-ceph-base_19.0.0-2574-gd2271579-1jammy_amd64.deb ... 2024-03-31T15:00:15.135 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking ceph-base (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:00:15.319 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package liboath0:amd64. 2024-03-31T15:00:15.333 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../029-liboath0_2.6.7-3build1_amd64.deb ... 2024-03-31T15:00:15.377 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2024-03-31T15:00:15.681 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-jaraco.functools. 2024-03-31T15:00:15.695 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../033-python3-jaraco.functools_3.4.0-2_all.deb ... 2024-03-31T15:00:15.714 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package libradosstriper1. 2024-03-31T15:00:15.728 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../030-libradosstriper1_19.0.0-2574-gd2271579-1jammy_amd64.deb ... 2024-03-31T15:00:15.730 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2024-03-31T15:00:15.772 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking libradosstriper1 (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:00:16.000 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-cheroot. 2024-03-31T15:00:16.016 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../034-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2024-03-31T15:00:16.058 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-03-31T15:00:16.208 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package ceph-common. 2024-03-31T15:00:16.223 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../031-ceph-common_19.0.0-2574-gd2271579-1jammy_amd64.deb ... 2024-03-31T15:00:16.258 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking ceph-common (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:00:16.420 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-jaraco.classes. 2024-03-31T15:00:16.435 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../035-python3-jaraco.classes_3.2.1-3_all.deb ... 2024-03-31T15:00:16.478 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2024-03-31T15:00:16.781 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-jaraco.text. 2024-03-31T15:00:16.797 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../036-python3-jaraco.text_3.6.0-2_all.deb ... 2024-03-31T15:00:16.839 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2024-03-31T15:00:17.205 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-jaraco.collections. 2024-03-31T15:00:17.222 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../037-python3-jaraco.collections_3.4.0-2_all.deb ... 2024-03-31T15:00:17.317 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2024-03-31T15:00:17.339 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package ceph-base. 2024-03-31T15:00:17.354 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../032-ceph-base_19.0.0-2574-gd2271579-1jammy_amd64.deb ... 2024-03-31T15:00:17.406 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking ceph-base (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:00:17.620 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-tempora. 2024-03-31T15:00:17.636 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../038-python3-tempora_4.1.2-1_all.deb ... 2024-03-31T15:00:17.679 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-tempora (4.1.2-1) ... 2024-03-31T15:00:17.968 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package python3-jaraco.functools. 2024-03-31T15:00:17.982 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../033-python3-jaraco.functools_3.4.0-2_all.deb ... 2024-03-31T15:00:17.982 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-portend. 2024-03-31T15:00:17.997 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../039-python3-portend_3.0.0-1_all.deb ... 2024-03-31T15:00:18.026 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2024-03-31T15:00:18.056 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-portend (3.0.0-1) ... 2024-03-31T15:00:18.345 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package python3-cheroot. 2024-03-31T15:00:18.351 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-zc.lockfile. 2024-03-31T15:00:18.359 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../034-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2024-03-31T15:00:18.366 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../040-python3-zc.lockfile_2.0-1_all.deb ... 2024-03-31T15:00:18.401 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2024-03-31T15:00:18.437 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-03-31T15:00:18.662 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-cherrypy3. 2024-03-31T15:00:18.678 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../041-python3-cherrypy3_18.6.1-4_all.deb ... 2024-03-31T15:00:18.721 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2024-03-31T15:00:18.749 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package python3-jaraco.classes. 2024-03-31T15:00:18.763 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../035-python3-jaraco.classes_3.2.1-3_all.deb ... 2024-03-31T15:00:18.807 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2024-03-31T15:00:19.091 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-natsort. 2024-03-31T15:00:19.107 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../042-python3-natsort_8.0.2-1_all.deb ... 2024-03-31T15:00:19.127 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package python3-jaraco.text. 2024-03-31T15:00:19.141 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../036-python3-jaraco.text_3.6.0-2_all.deb ... 2024-03-31T15:00:19.149 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-natsort (8.0.2-1) ... 2024-03-31T15:00:19.185 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2024-03-31T15:00:19.419 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-logutils. 2024-03-31T15:00:19.434 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../043-python3-logutils_0.3.3-8_all.deb ... 2024-03-31T15:00:19.446 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package python3-jaraco.collections. 2024-03-31T15:00:19.461 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../037-python3-jaraco.collections_3.4.0-2_all.deb ... 2024-03-31T15:00:19.469 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-logutils (0.3.3-8) ... 2024-03-31T15:00:19.504 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2024-03-31T15:00:19.822 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-markupsafe. 2024-03-31T15:00:19.838 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../044-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2024-03-31T15:00:19.841 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package python3-tempora. 2024-03-31T15:00:19.855 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../038-python3-tempora_4.1.2-1_all.deb ... 2024-03-31T15:00:19.880 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2024-03-31T15:00:19.899 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking python3-tempora (4.1.2-1) ... 2024-03-31T15:00:20.167 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-mako. 2024-03-31T15:00:20.183 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../045-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2024-03-31T15:00:20.185 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package python3-portend. 2024-03-31T15:00:20.200 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../039-python3-portend_3.0.0-1_all.deb ... 2024-03-31T15:00:20.217 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-03-31T15:00:20.243 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking python3-portend (3.0.0-1) ... 2024-03-31T15:00:20.564 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-simplegeneric. 2024-03-31T15:00:20.579 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../046-python3-simplegeneric_0.8.1-3_all.deb ... 2024-03-31T15:00:20.622 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-03-31T15:00:20.738 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package python3-zc.lockfile. 2024-03-31T15:00:20.753 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../040-python3-zc.lockfile_2.0-1_all.deb ... 2024-03-31T15:00:20.796 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2024-03-31T15:00:20.925 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-singledispatch. 2024-03-31T15:00:20.941 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../047-python3-singledispatch_3.4.0.3-3_all.deb ... 2024-03-31T15:00:20.983 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2024-03-31T15:00:21.099 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package python3-cherrypy3. 2024-03-31T15:00:21.114 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../041-python3-cherrypy3_18.6.1-4_all.deb ... 2024-03-31T15:00:21.157 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2024-03-31T15:00:21.261 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-webob. 2024-03-31T15:00:21.277 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../048-python3-webob_1%3a1.8.6-1.1_all.deb ... 2024-03-31T15:00:21.311 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-webob (1:1.8.6-1.1) ... 2024-03-31T15:00:21.519 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package python3-natsort. 2024-03-31T15:00:21.533 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../042-python3-natsort_8.0.2-1_all.deb ... 2024-03-31T15:00:21.569 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking python3-natsort (8.0.2-1) ... 2024-03-31T15:00:21.647 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-waitress. 2024-03-31T15:00:21.662 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../049-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2024-03-31T15:00:21.708 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2024-03-31T15:00:21.864 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package python3-logutils. 2024-03-31T15:00:21.878 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../043-python3-logutils_0.3.3-8_all.deb ... 2024-03-31T15:00:21.922 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking python3-logutils (0.3.3-8) ... 2024-03-31T15:00:22.225 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package python3-markupsafe. 2024-03-31T15:00:22.240 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../044-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2024-03-31T15:00:22.283 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2024-03-31T15:00:22.586 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package python3-mako. 2024-03-31T15:00:22.601 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../045-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2024-03-31T15:00:22.644 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-03-31T15:00:22.658 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-tempita. 2024-03-31T15:00:22.674 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../050-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2024-03-31T15:00:22.717 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2024-03-31T15:00:22.967 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package python3-simplegeneric. 2024-03-31T15:00:22.981 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../046-python3-simplegeneric_0.8.1-3_all.deb ... 2024-03-31T15:00:23.024 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-03-31T15:00:23.061 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-paste. 2024-03-31T15:00:23.077 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../051-python3-paste_3.5.0+dfsg1-1_all.deb ... 2024-03-31T15:00:23.119 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2024-03-31T15:00:23.328 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package python3-singledispatch. 2024-03-31T15:00:23.342 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../047-python3-singledispatch_3.4.0.3-3_all.deb ... 2024-03-31T15:00:23.386 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2024-03-31T15:00:23.456 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-03-31T15:00:23.472 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../052-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2024-03-31T15:00:23.514 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2024-03-31T15:00:23.705 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package python3-webob. 2024-03-31T15:00:23.720 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../048-python3-webob_1%3a1.8.6-1.1_all.deb ... 2024-03-31T15:00:23.763 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking python3-webob (1:1.8.6-1.1) ... 2024-03-31T15:00:23.842 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-pastedeploy. 2024-03-31T15:00:23.857 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../053-python3-pastedeploy_2.1.1-1_all.deb ... 2024-03-31T15:00:23.900 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2024-03-31T15:00:24.092 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package python3-waitress. 2024-03-31T15:00:24.106 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../049-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2024-03-31T15:00:24.153 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2024-03-31T15:00:24.153 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-webtest. 2024-03-31T15:00:24.171 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../054-python3-webtest_2.0.35-1_all.deb ... 2024-03-31T15:00:24.203 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-webtest (2.0.35-1) ... 2024-03-31T15:00:24.486 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package python3-tempita. 2024-03-31T15:00:24.498 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-pecan. 2024-03-31T15:00:24.501 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../050-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2024-03-31T15:00:24.513 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../055-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2024-03-31T15:00:24.536 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2024-03-31T15:00:24.548 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2024-03-31T15:00:24.805 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package python3-paste. 2024-03-31T15:00:24.820 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../051-python3-paste_3.5.0+dfsg1-1_all.deb ... 2024-03-31T15:00:24.861 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-certifi. 2024-03-31T15:00:24.863 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2024-03-31T15:00:24.876 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../056-python3-certifi_2020.6.20-1_all.deb ... 2024-03-31T15:00:24.910 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2024-03-31T15:00:25.175 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-03-31T15:00:25.190 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../052-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2024-03-31T15:00:25.233 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2024-03-31T15:00:25.238 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-idna. 2024-03-31T15:00:25.254 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../057-python3-idna_3.3-1_all.deb ... 2024-03-31T15:00:25.297 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-idna (3.3-1) ... 2024-03-31T15:00:25.530 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package python3-pastedeploy. 2024-03-31T15:00:25.543 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../053-python3-pastedeploy_2.1.1-1_all.deb ... 2024-03-31T15:00:25.608 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-urllib3. 2024-03-31T15:00:25.624 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../058-python3-urllib3_1.26.5-1~exp1ubuntu0.1_all.deb ... 2024-03-31T15:00:25.666 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-03-31T15:00:26.020 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-requests. 2024-03-31T15:00:26.035 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../059-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2024-03-31T15:00:26.078 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-03-31T15:00:26.253 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2024-03-31T15:00:26.356 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-werkzeug. 2024-03-31T15:00:26.371 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../060-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.1_all.deb ... 2024-03-31T15:00:26.406 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-03-31T15:00:26.606 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package python3-webtest. 2024-03-31T15:00:26.621 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../054-python3-webtest_2.0.35-1_all.deb ... 2024-03-31T15:00:26.673 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking python3-webtest (2.0.35-1) ... 2024-03-31T15:00:26.742 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-03-31T15:00:26.758 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../061-ceph-mgr-modules-core_19.0.0-2574-gd2271579-1jammy_all.deb ... 2024-03-31T15:00:26.800 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking ceph-mgr-modules-core (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:00:26.984 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package python3-pecan. 2024-03-31T15:00:26.999 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../055-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2024-03-31T15:00:27.042 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2024-03-31T15:00:27.121 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-03-31T15:00:27.137 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../062-libsqlite3-mod-ceph_19.0.0-2574-gd2271579-1jammy_amd64.deb ... 2024-03-31T15:00:27.196 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking libsqlite3-mod-ceph (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:00:27.406 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package python3-certifi. 2024-03-31T15:00:27.420 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../056-python3-certifi_2020.6.20-1_all.deb ... 2024-03-31T15:00:27.464 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2024-03-31T15:00:27.541 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-bcrypt. 2024-03-31T15:00:27.557 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../063-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2024-03-31T15:00:27.599 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2024-03-31T15:00:27.825 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package python3-idna. 2024-03-31T15:00:27.840 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../057-python3-idna_3.3-1_all.deb ... 2024-03-31T15:00:27.883 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking python3-idna (3.3-1) ... 2024-03-31T15:00:27.902 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-openssl. 2024-03-31T15:00:27.918 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../064-python3-openssl_21.0.0-1_all.deb ... 2024-03-31T15:00:27.960 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-openssl (21.0.0-1) ... 2024-03-31T15:00:28.186 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package python3-urllib3. 2024-03-31T15:00:28.201 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../058-python3-urllib3_1.26.5-1~exp1ubuntu0.1_all.deb ... 2024-03-31T15:00:28.236 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-03-31T15:00:28.288 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package ceph-mgr. 2024-03-31T15:00:28.304 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../065-ceph-mgr_19.0.0-2574-gd2271579-1jammy_amd64.deb ... 2024-03-31T15:00:28.338 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking ceph-mgr (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:00:28.573 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package python3-requests. 2024-03-31T15:00:28.590 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../059-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2024-03-31T15:00:28.622 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-03-31T15:00:28.692 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package ceph-mon. 2024-03-31T15:00:28.707 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../066-ceph-mon_19.0.0-2574-gd2271579-1jammy_amd64.deb ... 2024-03-31T15:00:28.750 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking ceph-mon (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:00:28.909 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package python3-werkzeug. 2024-03-31T15:00:28.924 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../060-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.1_all.deb ... 2024-03-31T15:00:28.967 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-03-31T15:00:29.279 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-03-31T15:00:29.294 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../061-ceph-mgr-modules-core_19.0.0-2574-gd2271579-1jammy_all.deb ... 2024-03-31T15:00:29.320 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package ceph-osd. 2024-03-31T15:00:29.328 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking ceph-mgr-modules-core (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:00:29.335 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../067-ceph-osd_19.0.0-2574-gd2271579-1jammy_amd64.deb ... 2024-03-31T15:00:29.378 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking ceph-osd (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:00:29.700 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-03-31T15:00:29.714 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../062-libsqlite3-mod-ceph_19.0.0-2574-gd2271579-1jammy_amd64.deb ... 2024-03-31T15:00:29.758 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking libsqlite3-mod-ceph (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:00:30.086 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package python3-bcrypt. 2024-03-31T15:00:30.101 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../063-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2024-03-31T15:00:30.106 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package ceph. 2024-03-31T15:00:30.121 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../068-ceph_19.0.0-2574-gd2271579-1jammy_amd64.deb ... 2024-03-31T15:00:30.144 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2024-03-31T15:00:30.156 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking ceph (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:00:30.414 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package python3-openssl. 2024-03-31T15:00:30.429 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../064-python3-openssl_21.0.0-1_all.deb ... 2024-03-31T15:00:30.451 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package ceph-fuse. 2024-03-31T15:00:30.464 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking python3-openssl (21.0.0-1) ... 2024-03-31T15:00:30.466 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../069-ceph-fuse_19.0.0-2574-gd2271579-1jammy_amd64.deb ... 2024-03-31T15:00:30.517 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking ceph-fuse (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:00:30.870 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package ceph-mds. 2024-03-31T15:00:30.886 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../070-ceph-mds_19.0.0-2574-gd2271579-1jammy_amd64.deb ... 2024-03-31T15:00:30.962 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking ceph-mds (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:00:30.992 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package ceph-mgr. 2024-03-31T15:00:31.007 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../065-ceph-mgr_19.0.0-2574-gd2271579-1jammy_amd64.deb ... 2024-03-31T15:00:31.050 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking ceph-mgr (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:00:31.348 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package cephadm. 2024-03-31T15:00:31.364 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../071-cephadm_19.0.0-2574-gd2271579-1jammy_amd64.deb ... 2024-03-31T15:00:31.406 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking cephadm (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:00:31.470 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package ceph-mon. 2024-03-31T15:00:31.485 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../066-ceph-mon_19.0.0-2574-gd2271579-1jammy_amd64.deb ... 2024-03-31T15:00:31.528 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking ceph-mon (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:00:31.701 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-asyncssh. 2024-03-31T15:00:31.716 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../072-python3-asyncssh_2.5.0-1_all.deb ... 2024-03-31T15:00:31.750 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2024-03-31T15:00:32.037 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python-babel-localedata. 2024-03-31T15:00:32.053 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../073-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2024-03-31T15:00:32.090 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package ceph-osd. 2024-03-31T15:00:32.095 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-03-31T15:00:32.104 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../067-ceph-osd_19.0.0-2574-gd2271579-1jammy_amd64.deb ... 2024-03-31T15:00:32.140 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking ceph-osd (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:00:32.935 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package ceph. 2024-03-31T15:00:32.949 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../068-ceph_19.0.0-2574-gd2271579-1jammy_amd64.deb ... 2024-03-31T15:00:32.993 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking ceph (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:00:33.225 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-babel. 2024-03-31T15:00:33.239 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../074-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2024-03-31T15:00:33.274 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2024-03-31T15:00:33.312 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package ceph-fuse. 2024-03-31T15:00:33.327 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../069-ceph-fuse_19.0.0-2574-gd2271579-1jammy_amd64.deb ... 2024-03-31T15:00:33.370 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking ceph-fuse (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:00:33.569 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-jinja2. 2024-03-31T15:00:33.584 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../075-python3-jinja2_3.0.3-1ubuntu0.1_all.deb ... 2024-03-31T15:00:33.619 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-03-31T15:00:33.749 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package ceph-mds. 2024-03-31T15:00:33.763 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../070-ceph-mds_19.0.0-2574-gd2271579-1jammy_amd64.deb ... 2024-03-31T15:00:33.815 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking ceph-mds (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:00:33.964 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-03-31T15:00:33.972 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../076-ceph-mgr-cephadm_19.0.0-2574-gd2271579-1jammy_all.deb ... 2024-03-31T15:00:34.014 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking ceph-mgr-cephadm (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:00:34.202 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package cephadm. 2024-03-31T15:00:34.216 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../071-cephadm_19.0.0-2574-gd2271579-1jammy_amd64.deb ... 2024-03-31T15:00:34.260 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking cephadm (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:00:34.359 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-repoze.lru. 2024-03-31T15:00:34.367 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../077-python3-repoze.lru_0.7-2_all.deb ... 2024-03-31T15:00:34.409 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-03-31T15:00:34.571 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package python3-asyncssh. 2024-03-31T15:00:34.586 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../072-python3-asyncssh_2.5.0-1_all.deb ... 2024-03-31T15:00:34.629 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2024-03-31T15:00:34.712 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-routes. 2024-03-31T15:00:34.723 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../078-python3-routes_2.5.1-1ubuntu1_all.deb ... 2024-03-31T15:00:34.753 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2024-03-31T15:00:34.990 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package python-babel-localedata. 2024-03-31T15:00:35.004 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../073-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2024-03-31T15:00:35.048 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-03-31T15:00:35.056 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-03-31T15:00:35.065 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../079-ceph-mgr-dashboard_19.0.0-2574-gd2271579-1jammy_all.deb ... 2024-03-31T15:00:35.115 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking ceph-mgr-dashboard (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:00:36.137 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package python3-babel. 2024-03-31T15:00:36.150 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../074-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2024-03-31T15:00:36.195 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2024-03-31T15:00:36.345 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2024-03-31T15:00:36.353 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../080-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2024-03-31T15:00:36.386 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-03-31T15:00:36.548 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package python3-jinja2. 2024-03-31T15:00:36.562 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../075-python3-jinja2_3.0.3-1ubuntu0.1_all.deb ... 2024-03-31T15:00:36.606 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-03-31T15:00:36.915 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-joblib. 2024-03-31T15:00:36.924 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../081-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2024-03-31T15:00:36.959 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-03-31T15:00:36.965 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2024-03-31T15:00:36.974 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../076-ceph-mgr-cephadm_19.0.0-2574-gd2271579-1jammy_all.deb ... 2024-03-31T15:00:37.042 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking ceph-mgr-cephadm (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:00:37.285 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-threadpoolctl. 2024-03-31T15:00:37.294 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../082-python3-threadpoolctl_3.1.0-1_all.deb ... 2024-03-31T15:00:37.379 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package python3-repoze.lru. 2024-03-31T15:00:37.394 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../077-python3-repoze.lru_0.7-2_all.deb ... 2024-03-31T15:00:37.404 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2024-03-31T15:00:37.437 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-03-31T15:00:37.715 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-sklearn. 2024-03-31T15:00:37.727 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../083-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2024-03-31T15:00:37.773 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2024-03-31T15:00:37.774 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package python3-routes. 2024-03-31T15:00:37.788 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../078-python3-routes_2.5.1-1ubuntu1_all.deb ... 2024-03-31T15:00:37.831 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2024-03-31T15:00:38.169 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-03-31T15:00:38.184 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../079-ceph-mgr-dashboard_19.0.0-2574-gd2271579-1jammy_all.deb ... 2024-03-31T15:00:38.226 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking ceph-mgr-dashboard (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:00:38.474 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-03-31T15:00:38.483 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../084-ceph-mgr-diskprediction-local_19.0.0-2574-gd2271579-1jammy_all.deb ... 2024-03-31T15:00:38.524 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking ceph-mgr-diskprediction-local (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:00:39.431 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2024-03-31T15:00:39.445 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../080-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2024-03-31T15:00:39.452 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-cachetools. 2024-03-31T15:00:39.462 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../085-python3-cachetools_5.0.0-1_all.deb ... 2024-03-31T15:00:39.490 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-03-31T15:00:39.502 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2024-03-31T15:00:39.830 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-pyasn1. 2024-03-31T15:00:39.840 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../086-python3-pyasn1_0.4.8-1_all.deb ... 2024-03-31T15:00:39.880 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2024-03-31T15:00:39.960 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package python3-joblib. 2024-03-31T15:00:39.974 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../081-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2024-03-31T15:00:40.010 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2024-03-31T15:00:40.217 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-03-31T15:00:40.230 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../087-python3-pyasn1-modules_0.2.1-1_all.deb ... 2024-03-31T15:00:40.266 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2024-03-31T15:00:40.372 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package python3-threadpoolctl. 2024-03-31T15:00:40.385 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../082-python3-threadpoolctl_3.1.0-1_all.deb ... 2024-03-31T15:00:40.515 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2024-03-31T15:00:40.603 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-rsa. 2024-03-31T15:00:40.614 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../088-python3-rsa_4.8-1_all.deb ... 2024-03-31T15:00:40.652 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-rsa (4.8-1) ... 2024-03-31T15:00:40.810 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package python3-sklearn. 2024-03-31T15:00:40.824 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../083-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2024-03-31T15:00:40.860 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2024-03-31T15:00:40.923 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-google-auth. 2024-03-31T15:00:40.934 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../089-python3-google-auth_1.5.1-3_all.deb ... 2024-03-31T15:00:40.973 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2024-03-31T15:00:41.326 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-03-31T15:00:41.337 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../090-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2024-03-31T15:00:41.376 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-03-31T15:00:41.494 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-03-31T15:00:41.508 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../084-ceph-mgr-diskprediction-local_19.0.0-2574-gd2271579-1jammy_all.deb ... 2024-03-31T15:00:41.552 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking ceph-mgr-diskprediction-local (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:00:41.688 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-websocket. 2024-03-31T15:00:41.699 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../091-python3-websocket_1.2.3-1_all.deb ... 2024-03-31T15:00:41.738 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-websocket (1.2.3-1) ... 2024-03-31T15:00:42.066 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-kubernetes. 2024-03-31T15:00:42.077 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../092-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2024-03-31T15:00:42.115 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2024-03-31T15:00:42.539 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package python3-cachetools. 2024-03-31T15:00:42.554 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../085-python3-cachetools_5.0.0-1_all.deb ... 2024-03-31T15:00:42.597 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2024-03-31T15:00:42.771 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-03-31T15:00:42.781 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../093-ceph-mgr-k8sevents_19.0.0-2574-gd2271579-1jammy_all.deb ... 2024-03-31T15:00:42.813 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking ceph-mgr-k8sevents (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:00:42.917 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package python3-pyasn1. 2024-03-31T15:00:42.932 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../086-python3-pyasn1_0.4.8-1_all.deb ... 2024-03-31T15:00:43.092 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2024-03-31T15:00:43.412 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-03-31T15:00:43.424 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package libonig5:amd64. 2024-03-31T15:00:43.427 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../087-python3-pyasn1-modules_0.2.1-1_all.deb ... 2024-03-31T15:00:43.435 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../094-libonig5_6.9.7.1-2build1_amd64.deb ... 2024-03-31T15:00:43.474 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2024-03-31T15:00:43.489 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2024-03-31T15:00:43.802 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package libjq1:amd64. 2024-03-31T15:00:43.813 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../095-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2024-03-31T15:00:43.815 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package python3-rsa. 2024-03-31T15:00:43.828 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../088-python3-rsa_4.8-1_all.deb ... 2024-03-31T15:00:43.851 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-03-31T15:00:43.873 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking python3-rsa (4.8-1) ... 2024-03-31T15:00:44.138 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package jq. 2024-03-31T15:00:44.150 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../096-jq_1.6-2.1ubuntu3_amd64.deb ... 2024-03-31T15:00:44.188 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2024-03-31T15:00:44.218 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package python3-google-auth. 2024-03-31T15:00:44.233 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../089-python3-google-auth_1.5.1-3_all.deb ... 2024-03-31T15:00:44.293 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2024-03-31T15:00:44.399 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package socat. 2024-03-31T15:00:44.412 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../097-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2024-03-31T15:00:44.457 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2024-03-31T15:00:44.613 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-03-31T15:00:44.627 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../090-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2024-03-31T15:00:44.671 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-03-31T15:00:44.777 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package xmlstarlet. 2024-03-31T15:00:44.788 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../098-xmlstarlet_1.6.1-2.1_amd64.deb ... 2024-03-31T15:00:44.826 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2024-03-31T15:00:44.983 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package python3-websocket. 2024-03-31T15:00:44.996 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../091-python3-websocket_1.2.3-1_all.deb ... 2024-03-31T15:00:45.041 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking python3-websocket (1.2.3-1) ... 2024-03-31T15:00:45.148 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package ceph-test. 2024-03-31T15:00:45.158 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../099-ceph-test_19.0.0-2574-gd2271579-1jammy_amd64.deb ... 2024-03-31T15:00:45.189 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking ceph-test (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:00:45.344 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package python3-kubernetes. 2024-03-31T15:00:45.358 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../092-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2024-03-31T15:00:45.402 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2024-03-31T15:00:46.084 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-03-31T15:00:46.096 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../093-ceph-mgr-k8sevents_19.0.0-2574-gd2271579-1jammy_all.deb ... 2024-03-31T15:00:46.133 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking ceph-mgr-k8sevents (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:00:46.477 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package libonig5:amd64. 2024-03-31T15:00:46.493 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../094-libonig5_6.9.7.1-2build1_amd64.deb ... 2024-03-31T15:00:46.536 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2024-03-31T15:00:46.880 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package libjq1:amd64. 2024-03-31T15:00:46.895 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../095-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2024-03-31T15:00:46.938 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-03-31T15:00:47.119 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package ceph-volume. 2024-03-31T15:00:47.130 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../100-ceph-volume_19.0.0-2574-gd2271579-1jammy_all.deb ... 2024-03-31T15:00:47.168 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking ceph-volume (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:00:47.225 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package jq. 2024-03-31T15:00:47.239 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../096-jq_1.6-2.1ubuntu3_amd64.deb ... 2024-03-31T15:00:47.283 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2024-03-31T15:00:47.489 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package cephfs-mirror. 2024-03-31T15:00:47.500 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../101-cephfs-mirror_19.0.0-2574-gd2271579-1jammy_amd64.deb ... 2024-03-31T15:00:47.510 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package socat. 2024-03-31T15:00:47.525 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../097-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2024-03-31T15:00:47.539 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking cephfs-mirror (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:00:47.569 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2024-03-31T15:00:47.850 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-colorama. 2024-03-31T15:00:47.861 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../102-python3-colorama_0.4.4-1_all.deb ... 2024-03-31T15:00:47.897 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package xmlstarlet. 2024-03-31T15:00:47.900 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-colorama (0.4.4-1) ... 2024-03-31T15:00:47.912 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../098-xmlstarlet_1.6.1-2.1_amd64.deb ... 2024-03-31T15:00:47.955 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2024-03-31T15:00:48.144 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package cephfs-shell. 2024-03-31T15:00:48.156 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../103-cephfs-shell_19.0.0-2574-gd2271579-1jammy_all.deb ... 2024-03-31T15:00:48.194 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking cephfs-shell (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:00:48.268 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package ceph-test. 2024-03-31T15:00:48.281 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../099-ceph-test_19.0.0-2574-gd2271579-1jammy_amd64.deb ... 2024-03-31T15:00:48.326 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking ceph-test (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:00:48.480 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package cephfs-top. 2024-03-31T15:00:48.492 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../104-cephfs-top_19.0.0-2574-gd2271579-1jammy_all.deb ... 2024-03-31T15:00:48.530 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking cephfs-top (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:00:48.783 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package libcephfs-dev. 2024-03-31T15:00:48.794 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../105-libcephfs-dev_19.0.0-2574-gd2271579-1jammy_amd64.deb ... 2024-03-31T15:00:48.832 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking libcephfs-dev (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:00:49.169 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package lua-socket:amd64. 2024-03-31T15:00:49.181 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../106-lua-socket_3.0~rc1+git+ac3201d-6_amd64.deb ... 2024-03-31T15:00:49.219 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-03-31T15:00:49.557 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package lua-sec:amd64. 2024-03-31T15:00:49.568 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../107-lua-sec_1.0.2-1_amd64.deb ... 2024-03-31T15:00:49.606 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking lua-sec:amd64 (1.0.2-1) ... 2024-03-31T15:00:49.944 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package nvme-cli. 2024-03-31T15:00:49.955 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../108-nvme-cli_1.16-3ubuntu0.1_amd64.deb ... 2024-03-31T15:00:49.985 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking nvme-cli (1.16-3ubuntu0.1) ... 2024-03-31T15:00:50.288 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package ceph-volume. 2024-03-31T15:00:50.303 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../100-ceph-volume_19.0.0-2574-gd2271579-1jammy_all.deb ... 2024-03-31T15:00:50.332 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package pkg-config. 2024-03-31T15:00:50.342 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../109-pkg-config_0.29.2-1ubuntu3_amd64.deb ... 2024-03-31T15:00:50.346 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking ceph-volume (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:00:50.381 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking pkg-config (0.29.2-1ubuntu3) ... 2024-03-31T15:00:50.652 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python-asyncssh-doc. 2024-03-31T15:00:50.663 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../110-python-asyncssh-doc_2.5.0-1_all.deb ... 2024-03-31T15:00:50.701 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2024-03-31T15:00:50.721 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package cephfs-mirror. 2024-03-31T15:00:50.736 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../101-cephfs-mirror_19.0.0-2574-gd2271579-1jammy_amd64.deb ... 2024-03-31T15:00:50.776 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking cephfs-mirror (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:00:51.087 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package python3-colorama. 2024-03-31T15:00:51.101 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../102-python3-colorama_0.4.4-1_all.deb ... 2024-03-31T15:00:51.121 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-iniconfig. 2024-03-31T15:00:51.132 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../111-python3-iniconfig_1.1.1-2_all.deb ... 2024-03-31T15:00:51.168 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking python3-colorama (0.4.4-1) ... 2024-03-31T15:00:51.171 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2024-03-31T15:00:51.448 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package cephfs-shell. 2024-03-31T15:00:51.463 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../103-cephfs-shell_19.0.0-2574-gd2271579-1jammy_all.deb ... 2024-03-31T15:00:51.490 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-pastescript. 2024-03-31T15:00:51.502 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../112-python3-pastescript_2.0.2-4_all.deb ... 2024-03-31T15:00:51.506 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking cephfs-shell (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:00:51.540 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-03-31T15:00:51.742 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package cephfs-top. 2024-03-31T15:00:51.757 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../104-cephfs-top_19.0.0-2574-gd2271579-1jammy_all.deb ... 2024-03-31T15:00:51.792 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking cephfs-top (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:00:51.860 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-pluggy. 2024-03-31T15:00:51.872 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../113-python3-pluggy_0.13.0-7.1_all.deb ... 2024-03-31T15:00:51.910 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2024-03-31T15:00:52.003 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package libcephfs-dev. 2024-03-31T15:00:52.017 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../105-libcephfs-dev_19.0.0-2574-gd2271579-1jammy_amd64.deb ... 2024-03-31T15:00:52.061 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking libcephfs-dev (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:00:52.213 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-psutil. 2024-03-31T15:00:52.225 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../114-python3-psutil_5.9.0-1build1_amd64.deb ... 2024-03-31T15:00:52.254 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2024-03-31T15:00:52.414 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package lua-socket:amd64. 2024-03-31T15:00:52.428 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../106-lua-socket_3.0~rc1+git+ac3201d-6_amd64.deb ... 2024-03-31T15:00:52.472 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-03-31T15:00:52.591 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-py. 2024-03-31T15:00:52.602 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../115-python3-py_1.10.0-1_all.deb ... 2024-03-31T15:00:52.640 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-py (1.10.0-1) ... 2024-03-31T15:00:52.801 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package lua-sec:amd64. 2024-03-31T15:00:52.815 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../107-lua-sec_1.0.2-1_amd64.deb ... 2024-03-31T15:00:52.884 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking lua-sec:amd64 (1.0.2-1) ... 2024-03-31T15:00:52.994 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-pygments. 2024-03-31T15:00:53.005 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../116-python3-pygments_2.11.2+dfsg-2_all.deb ... 2024-03-31T15:00:53.044 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2024-03-31T15:00:53.222 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package nvme-cli. 2024-03-31T15:00:53.236 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../108-nvme-cli_1.16-3ubuntu0.1_amd64.deb ... 2024-03-31T15:00:53.271 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking nvme-cli (1.16-3ubuntu0.1) ... 2024-03-31T15:00:53.523 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-pyinotify. 2024-03-31T15:00:53.533 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../117-python3-pyinotify_0.9.6-1.3_all.deb ... 2024-03-31T15:00:53.572 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2024-03-31T15:00:53.668 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package pkg-config. 2024-03-31T15:00:53.683 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../109-pkg-config_0.29.2-1ubuntu3_amd64.deb ... 2024-03-31T15:00:53.727 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking pkg-config (0.29.2-1ubuntu3) ... 2024-03-31T15:00:53.892 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-toml. 2024-03-31T15:00:53.904 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../118-python3-toml_0.10.2-1_all.deb ... 2024-03-31T15:00:53.934 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-toml (0.10.2-1) ... 2024-03-31T15:00:53.988 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package python-asyncssh-doc. 2024-03-31T15:00:54.000 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../110-python-asyncssh-doc_2.5.0-1_all.deb ... 2024-03-31T15:00:54.037 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2024-03-31T15:00:54.220 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-pytest. 2024-03-31T15:00:54.232 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../119-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2024-03-31T15:00:54.270 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2024-03-31T15:00:54.466 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package python3-iniconfig. 2024-03-31T15:00:54.479 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../111-python3-iniconfig_1.1.1-2_all.deb ... 2024-03-31T15:00:54.515 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2024-03-31T15:00:54.607 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-simplejson. 2024-03-31T15:00:54.618 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../120-python3-simplejson_3.17.6-1build1_amd64.deb ... 2024-03-31T15:00:54.656 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2024-03-31T15:00:54.802 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package python3-pastescript. 2024-03-31T15:00:54.813 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../112-python3-pastescript_2.0.2-4_all.deb ... 2024-03-31T15:00:54.852 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-03-31T15:00:54.951 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-03-31T15:00:54.962 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../121-qttranslations5-l10n_5.15.3-1_all.deb ... 2024-03-31T15:00:54.993 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2024-03-31T15:00:55.197 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package python3-pluggy. 2024-03-31T15:00:55.209 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../113-python3-pluggy_0.13.0-7.1_all.deb ... 2024-03-31T15:00:55.246 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2024-03-31T15:00:55.558 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package python3-psutil. 2024-03-31T15:00:55.573 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../114-python3-psutil_5.9.0-1build1_amd64.deb ... 2024-03-31T15:00:55.616 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2024-03-31T15:00:55.696 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package radosgw. 2024-03-31T15:00:55.706 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../122-radosgw_19.0.0-2574-gd2271579-1jammy_amd64.deb ... 2024-03-31T15:00:55.746 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking radosgw (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:00:55.986 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package python3-py. 2024-03-31T15:00:56.000 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../115-python3-py_1.10.0-1_all.deb ... 2024-03-31T15:00:56.044 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking python3-py (1.10.0-1) ... 2024-03-31T15:00:56.372 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package python3-pygments. 2024-03-31T15:00:56.386 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../116-python3-pygments_2.11.2+dfsg-2_all.deb ... 2024-03-31T15:00:56.424 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package rbd-fuse. 2024-03-31T15:00:56.436 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../123-rbd-fuse_19.0.0-2574-gd2271579-1jammy_amd64.deb ... 2024-03-31T15:00:56.455 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2024-03-31T15:00:56.474 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking rbd-fuse (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:00:56.734 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2024-03-31T15:00:56.918 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package python3-pyinotify. 2024-03-31T15:00:56.930 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../117-python3-pyinotify_0.9.6-1.3_all.deb ... 2024-03-31T15:00:56.968 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2024-03-31T15:00:57.034 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2024-03-31T15:00:57.155 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up nvme-cli (1.16-3ubuntu0.1) ... 2024-03-31T15:00:57.287 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package python3-toml. 2024-03-31T15:00:57.300 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../118-python3-toml_0.10.2-1_all.deb ... 2024-03-31T15:00:57.345 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking python3-toml (0.10.2-1) ... 2024-03-31T15:00:57.396 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2024-03-31T15:00:57.623 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package python3-pytest. 2024-03-31T15:00:57.637 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../119-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2024-03-31T15:00:57.681 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2024-03-31T15:00:57.806 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2024-03-31T15:00:58.043 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package python3-simplejson. 2024-03-31T15:00:58.056 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../120-python3-simplejson_3.17.6-1build1_amd64.deb ... 2024-03-31T15:00:58.093 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2024-03-31T15:00:58.371 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-03-31T15:00:58.384 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../121-qttranslations5-l10n_5.15.3-1_all.deb ... 2024-03-31T15:00:58.420 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2024-03-31T15:00:58.456 INFO:teuthology.orchestra.run.smithi016.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2024-03-31T15:00:58.490 INFO:teuthology.orchestra.run.smithi016.stdout:Could not execute systemctl: at /usr/bin/deb-systemd-invoke line 142. 2024-03-31T15:00:58.532 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up cephadm (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:00:59.091 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package radosgw. 2024-03-31T15:00:59.104 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../122-radosgw_19.0.0-2574-gd2271579-1jammy_amd64.deb ... 2024-03-31T15:00:59.141 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking radosgw (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:00:59.344 INFO:teuthology.orchestra.run.smithi016.stdout:Adding system user cephadm....done 2024-03-31T15:00:59.489 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2024-03-31T15:00:59.744 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package rbd-fuse. 2024-03-31T15:00:59.757 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2024-03-31T15:00:59.758 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../123-rbd-fuse_19.0.0-2574-gd2271579-1jammy_amd64.deb ... 2024-03-31T15:00:59.803 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking rbd-fuse (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:01:00.035 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2024-03-31T15:01:00.120 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2024-03-31T15:01:00.161 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2024-03-31T15:01:00.391 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2024-03-31T15:01:00.429 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-03-31T15:01:00.517 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up nvme-cli (1.16-3ubuntu0.1) ... 2024-03-31T15:01:00.705 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2024-03-31T15:01:00.802 INFO:teuthology.orchestra.run.smithi089.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2024-03-31T15:01:00.806 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-py (1.10.0-1) ... 2024-03-31T15:01:01.118 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2024-03-31T15:01:01.188 INFO:teuthology.orchestra.run.smithi089.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2024-03-31T15:01:01.493 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-cachetools (5.0.0-1) ... 2024-03-31T15:01:01.778 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-colorama (0.4.4-1) ... 2024-03-31T15:01:01.865 INFO:teuthology.orchestra.run.smithi089.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2024-03-31T15:01:01.878 INFO:teuthology.orchestra.run.smithi089.stdout:Could not execute systemctl: at /usr/bin/deb-systemd-invoke line 142. 2024-03-31T15:01:01.928 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up cephadm (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:01:02.062 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-openssl (21.0.0-1) ... 2024-03-31T15:01:02.346 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up unzip (6.0-26ubuntu3.2) ... 2024-03-31T15:01:02.429 INFO:teuthology.orchestra.run.smithi089.stdout:Adding system user cephadm....done 2024-03-31T15:01:02.583 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2024-03-31T15:01:02.706 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2024-03-31T15:01:02.870 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2024-03-31T15:01:02.999 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2024-03-31T15:01:03.129 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2024-03-31T15:01:03.255 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2024-03-31T15:01:03.275 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2024-03-31T15:01:03.515 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-03-31T15:01:03.543 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-ceph-argparse (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:01:03.783 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2024-03-31T15:01:03.900 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up python3-py (1.10.0-1) ... 2024-03-31T15:01:04.019 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-03-31T15:01:04.180 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-03-31T15:01:04.227 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2024-03-31T15:01:04.313 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-pyperclip (1.8.2-2) ... 2024-03-31T15:01:04.589 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up libreadline-dev:amd64 (8.1.2-1) ... 2024-03-31T15:01:04.596 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up python3-cachetools (5.0.0-1) ... 2024-03-31T15:01:04.690 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2024-03-31T15:01:04.855 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up python3-colorama (0.4.4-1) ... 2024-03-31T15:01:04.941 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up lua5.1 (5.1.5-8.1build4) ... 2024-03-31T15:01:05.132 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up python3-openssl (21.0.0-1) ... 2024-03-31T15:01:05.325 INFO:teuthology.orchestra.run.smithi016.stdout:update-alternatives: using /usr/bin/lua5.1 to provide /usr/bin/lua (lua-interpreter) in auto mode 2024-03-31T15:01:05.356 INFO:teuthology.orchestra.run.smithi016.stdout:update-alternatives: using /usr/bin/luac5.1 to provide /usr/bin/luac (lua-compiler) in auto mode 2024-03-31T15:01:05.416 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up unzip (6.0-26ubuntu3.2) ... 2024-03-31T15:01:05.419 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-03-31T15:01:05.546 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2024-03-31T15:01:05.710 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2024-03-31T15:01:05.947 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-03-31T15:01:05.969 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2024-03-31T15:01:06.048 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-natsort (8.0.2-1) ... 2024-03-31T15:01:06.316 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2024-03-31T15:01:06.362 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2024-03-31T15:01:06.592 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2024-03-31T15:01:06.879 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up python3-ceph-argparse (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:01:06.918 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up zip (3.0-12build2) ... 2024-03-31T15:01:07.069 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2024-03-31T15:01:07.131 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-03-31T15:01:07.256 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-03-31T15:01:07.382 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up python3-pyperclip (1.8.2-2) ... 2024-03-31T15:01:07.642 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up libreadline-dev:amd64 (8.1.2-1) ... 2024-03-31T15:01:07.793 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2024-03-31T15:01:07.804 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2024-03-31T15:01:08.053 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up lua5.1 (5.1.5-8.1build4) ... 2024-03-31T15:01:08.072 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2024-03-31T15:01:08.173 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2024-03-31T15:01:08.274 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2024-03-31T15:01:08.448 INFO:teuthology.orchestra.run.smithi089.stdout:update-alternatives: using /usr/bin/lua5.1 to provide /usr/bin/lua (lua-interpreter) in auto mode 2024-03-31T15:01:08.508 INFO:teuthology.orchestra.run.smithi089.stdout:update-alternatives: using /usr/bin/luac5.1 to provide /usr/bin/luac (lua-compiler) in auto mode 2024-03-31T15:01:08.581 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-03-31T15:01:08.618 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2024-03-31T15:01:08.707 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2024-03-31T15:01:09.036 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-certifi (2020.6.20-1) ... 2024-03-31T15:01:09.074 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-03-31T15:01:09.226 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up python3-natsort (8.0.2-1) ... 2024-03-31T15:01:09.337 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2024-03-31T15:01:09.502 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2024-03-31T15:01:09.738 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-03-31T15:01:09.770 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2024-03-31T15:01:10.031 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-cmd2 (2.3.3-1) ... 2024-03-31T15:01:10.104 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up zip (3.0-12build2) ... 2024-03-31T15:01:10.205 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2024-03-31T15:01:10.341 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-03-31T15:01:10.700 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2024-03-31T15:01:10.882 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2024-03-31T15:01:10.977 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2024-03-31T15:01:11.103 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-ceph-common (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:01:11.142 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2024-03-31T15:01:11.276 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2024-03-31T15:01:11.386 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2024-03-31T15:01:11.420 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-idna (3.3-1) ... 2024-03-31T15:01:11.695 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2024-03-31T15:01:11.772 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2024-03-31T15:01:12.088 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up python3-certifi (2020.6.20-1) ... 2024-03-31T15:01:12.339 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2024-03-31T15:01:12.724 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-03-31T15:01:13.025 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up python3-cmd2 (2.3.3-1) ... 2024-03-31T15:01:13.053 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up pkg-config (0.29.2-1ubuntu3) ... 2024-03-31T15:01:13.321 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-03-31T15:01:13.368 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-03-31T15:01:13.472 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-toml (0.10.2-1) ... 2024-03-31T15:01:13.728 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2024-03-31T15:01:13.773 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2024-03-31T15:01:13.874 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2024-03-31T15:01:13.984 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-03-31T15:01:13.987 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2024-03-31T15:01:14.138 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up python3-ceph-common (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:01:14.319 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2024-03-31T15:01:14.440 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up python3-idna (3.3-1) ... 2024-03-31T15:01:14.595 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2024-03-31T15:01:14.790 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2024-03-31T15:01:14.863 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-03-31T15:01:14.989 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2024-03-31T15:01:15.282 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2024-03-31T15:01:15.541 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-logutils (0.3.3-8) ... 2024-03-31T15:01:15.801 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-tempora (4.1.2-1) ... 2024-03-31T15:01:16.038 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up pkg-config (0.29.2-1ubuntu3) ... 2024-03-31T15:01:16.077 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-03-31T15:01:16.314 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-03-31T15:01:16.370 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-prettytable (2.5.0-2) ... 2024-03-31T15:01:16.449 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up python3-toml (0.10.2-1) ... 2024-03-31T15:01:16.654 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-03-31T15:01:16.725 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2024-03-31T15:01:16.781 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-websocket (1.2.3-1) ... 2024-03-31T15:01:16.851 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2024-03-31T15:01:17.011 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-03-31T15:01:17.065 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2024-03-31T15:01:17.166 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-03-31T15:01:17.312 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2024-03-31T15:01:17.450 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-webob (1:1.8.6-1.1) ... 2024-03-31T15:01:17.571 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2024-03-31T15:01:17.776 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2024-03-31T15:01:17.847 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-03-31T15:01:17.973 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2024-03-31T15:01:18.053 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-03-31T15:01:18.179 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up lua-sec:amd64 (1.0.2-1) ... 2024-03-31T15:01:18.306 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2024-03-31T15:01:18.338 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2024-03-31T15:01:18.427 INFO:teuthology.orchestra.run.smithi016.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2024-03-31T15:01:18.576 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up python3-logutils (0.3.3-8) ... 2024-03-31T15:01:18.665 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-03-31T15:01:18.766 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2024-03-31T15:01:18.844 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up python3-tempora (4.1.2-1) ... 2024-03-31T15:01:19.112 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-03-31T15:01:19.159 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2024-03-31T15:01:19.354 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up python3-prettytable (2.5.0-2) ... 2024-03-31T15:01:19.452 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up lua-any (27ubuntu1) ... 2024-03-31T15:01:19.578 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-portend (3.0.0-1) ... 2024-03-31T15:01:19.614 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-03-31T15:01:19.773 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up python3-websocket (1.2.3-1) ... 2024-03-31T15:01:19.846 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-03-31T15:01:19.972 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-03-31T15:01:20.066 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2024-03-31T15:01:20.192 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-03-31T15:01:20.298 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2024-03-31T15:01:20.476 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up python3-webob (1:1.8.6-1.1) ... 2024-03-31T15:01:20.616 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-03-31T15:01:20.786 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2024-03-31T15:01:20.892 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2024-03-31T15:01:21.045 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-03-31T15:01:21.112 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-rsa (4.8-1) ... 2024-03-31T15:01:21.180 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up lua-sec:amd64 (1.0.2-1) ... 2024-03-31T15:01:21.281 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2024-03-31T15:01:21.369 INFO:teuthology.orchestra.run.smithi089.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2024-03-31T15:01:21.405 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-webtest (2.0.35-1) ... 2024-03-31T15:01:21.607 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-03-31T15:01:21.673 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-03-31T15:01:21.717 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2024-03-31T15:01:21.940 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2024-03-31T15:01:22.110 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2024-03-31T15:01:22.334 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-03-31T15:01:22.378 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up lua-any (27ubuntu1) ... 2024-03-31T15:01:22.526 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up python3-portend (3.0.0-1) ... 2024-03-31T15:01:22.636 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2024-03-31T15:01:22.780 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-03-31T15:01:22.906 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-03-31T15:01:22.996 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-03-31T15:01:23.097 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up librados2 (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:01:23.198 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up libsqlite3-mod-ceph (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:01:23.249 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2024-03-31T15:01:23.349 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up luarocks (3.8.0+dfsg1-1) ... 2024-03-31T15:01:23.558 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-03-31T15:01:23.686 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up libcephfs2 (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:01:23.812 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up libradosstriper1 (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:01:23.851 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2024-03-31T15:01:23.938 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-google-auth (1.5.1-3) ... 2024-03-31T15:01:24.087 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up python3-rsa (4.8-1) ... 2024-03-31T15:01:24.256 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up librbd1 (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:01:24.347 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up python3-webtest (2.0.35-1) ... 2024-03-31T15:01:24.357 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up ceph-mgr-modules-core (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:01:24.466 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up ceph-fuse (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:01:24.651 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-03-31T15:01:24.675 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-03-31T15:01:24.675 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-03-31T15:01:24.916 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2024-03-31T15:01:25.301 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-03-31T15:01:25.343 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up libcephfs-dev (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:01:25.503 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-rados (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:01:25.620 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2024-03-31T15:01:25.628 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up librgw2 (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:01:25.754 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up cephfs-top (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:01:25.863 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2024-03-31T15:01:25.955 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-03-31T15:01:26.098 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up librados2 (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:01:26.224 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up libsqlite3-mod-ceph (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:01:26.350 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up luarocks (3.8.0+dfsg1-1) ... 2024-03-31T15:01:26.662 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up libcephfs2 (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:01:26.821 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up libradosstriper1 (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:01:26.947 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up python3-google-auth (1.5.1-3) ... 2024-03-31T15:01:27.192 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-rbd (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:01:27.240 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up librbd1 (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:01:27.318 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up rbd-fuse (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:01:27.366 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up ceph-mgr-modules-core (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:01:27.461 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-rgw (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:01:27.525 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up ceph-fuse (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:01:27.612 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-cephfs (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:01:27.739 INFO:teuthology.orchestra.run.smithi089.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-03-31T15:01:27.739 INFO:teuthology.orchestra.run.smithi089.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-03-31T15:01:27.755 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up cephfs-shell (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:01:27.881 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up ceph-common (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:01:28.384 INFO:teuthology.orchestra.run.smithi016.stdout:Adding group ceph....done 2024-03-31T15:01:28.394 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up libcephfs-dev (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:01:28.536 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up python3-rados (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:01:28.637 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up librgw2 (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:01:28.727 INFO:teuthology.orchestra.run.smithi016.stdout:Adding system user ceph....done 2024-03-31T15:01:28.738 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up cephfs-top (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:01:28.871 INFO:teuthology.orchestra.run.smithi016.stdout:Setting system user ceph properties....done 2024-03-31T15:01:28.872 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2024-03-31T15:01:28.892 INFO:teuthology.orchestra.run.smithi016.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-03-31T15:01:29.047 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-03-31T15:01:29.454 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-03-31T15:01:30.157 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up cephfs-mirror (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:01:30.159 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up python3-rbd (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:01:30.310 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up rbd-fuse (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:01:30.374 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target -> /lib/systemd/system/cephfs-mirror.target. 2024-03-31T15:01:30.374 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target -> /lib/systemd/system/cephfs-mirror.target. 2024-03-31T15:01:30.437 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up python3-rgw (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:01:30.562 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up python3-cephfs (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:01:30.688 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up cephfs-shell (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:01:30.814 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up ceph-common (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:01:31.034 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up ceph-test (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:01:31.193 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up radosgw (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:01:31.351 INFO:teuthology.orchestra.run.smithi089.stdout:Adding group ceph....done 2024-03-31T15:01:31.760 INFO:teuthology.orchestra.run.smithi089.stdout:Adding system user ceph....done 2024-03-31T15:01:31.764 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-03-31T15:01:31.764 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-03-31T15:01:31.913 INFO:teuthology.orchestra.run.smithi089.stdout:Setting system user ceph properties....done 2024-03-31T15:01:31.931 INFO:teuthology.orchestra.run.smithi089.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-03-31T15:01:32.072 INFO:teuthology.orchestra.run.smithi089.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-03-31T15:01:32.435 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up ceph-base (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:01:32.481 INFO:teuthology.orchestra.run.smithi089.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-03-31T15:01:32.952 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-03-31T15:01:33.182 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up cephfs-mirror (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:01:33.391 INFO:teuthology.orchestra.run.smithi089.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target -> /lib/systemd/system/cephfs-mirror.target. 2024-03-31T15:01:33.392 INFO:teuthology.orchestra.run.smithi089.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target -> /lib/systemd/system/cephfs-mirror.target. 2024-03-31T15:01:33.623 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up ceph-mds (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:01:33.860 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-03-31T15:01:33.861 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-03-31T15:01:34.050 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up ceph-test (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:01:34.226 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up radosgw (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:01:34.466 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up ceph-mgr (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:01:34.694 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-03-31T15:01:34.694 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-03-31T15:01:34.807 INFO:teuthology.orchestra.run.smithi089.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-03-31T15:01:34.808 INFO:teuthology.orchestra.run.smithi089.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-03-31T15:01:35.363 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up ceph-osd (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:01:35.454 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up ceph-base (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:01:35.703 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-03-31T15:01:35.703 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-03-31T15:01:35.789 INFO:teuthology.orchestra.run.smithi089.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-03-31T15:01:36.353 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up ceph-mgr-k8sevents (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:01:36.454 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up ceph-mgr-diskprediction-local (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:01:36.464 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up ceph-mds (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:01:36.589 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up ceph-mon (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:01:36.687 INFO:teuthology.orchestra.run.smithi089.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-03-31T15:01:36.687 INFO:teuthology.orchestra.run.smithi089.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-03-31T15:01:36.804 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-03-31T15:01:36.804 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-03-31T15:01:37.325 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up ceph-mgr (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:01:37.490 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up ceph-mgr-cephadm (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:01:37.571 INFO:teuthology.orchestra.run.smithi089.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-03-31T15:01:37.571 INFO:teuthology.orchestra.run.smithi089.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-03-31T15:01:37.632 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up ceph (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:01:37.758 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up ceph-mgr-dashboard (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:01:37.909 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up ceph-volume (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:01:38.201 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up ceph-osd (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:01:38.295 INFO:teuthology.orchestra.run.smithi016.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2024-03-31T15:01:38.396 INFO:teuthology.orchestra.run.smithi016.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-03-31T15:01:38.540 INFO:teuthology.orchestra.run.smithi016.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-03-31T15:01:38.635 INFO:teuthology.orchestra.run.smithi089.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-03-31T15:01:38.635 INFO:teuthology.orchestra.run.smithi089.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-03-31T15:01:39.270 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up ceph-mgr-k8sevents (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:01:39.396 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up ceph-mgr-diskprediction-local (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:01:39.532 INFO:teuthology.orchestra.run.smithi016.stdout:Processing triggers for install-info (6.8-4build1) ... 2024-03-31T15:01:39.563 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up ceph-mon (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:01:39.772 INFO:teuthology.orchestra.run.smithi089.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-03-31T15:01:39.772 INFO:teuthology.orchestra.run.smithi089.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-03-31T15:01:40.346 INFO:teuthology.orchestra.run.smithi016.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-03-31T15:01:40.351 DEBUG:teuthology.orchestra.run.smithi016:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install bison flex libelf-dev libssl-dev network-manager iproute2 util-linux dump indent libaio-dev libtool-bin uuid-dev xfslibs-dev postgresql postgresql-client postgresql-common postgresql-contrib 2024-03-31T15:01:40.406 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up ceph-mgr-cephadm (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:01:40.427 INFO:teuthology.orchestra.run.smithi016.stdout:Reading package lists... 2024-03-31T15:01:40.549 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up ceph (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:01:40.604 INFO:teuthology.orchestra.run.smithi016.stdout:Building dependency tree... 2024-03-31T15:01:40.605 INFO:teuthology.orchestra.run.smithi016.stdout:Reading state information... 2024-03-31T15:01:40.675 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up ceph-mgr-dashboard (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:01:40.778 INFO:teuthology.orchestra.run.smithi016.stdout:iproute2 is already the newest version (5.15.0-1ubuntu2). 2024-03-31T15:01:40.779 INFO:teuthology.orchestra.run.smithi016.stdout:libtool-bin is already the newest version (2.4.6-15build2). 2024-03-31T15:01:40.779 INFO:teuthology.orchestra.run.smithi016.stdout:xfslibs-dev is already the newest version (5.13.0-1ubuntu2). 2024-03-31T15:01:40.779 INFO:teuthology.orchestra.run.smithi016.stdout:The following packages were automatically installed and are no longer required: 2024-03-31T15:01:40.779 INFO:teuthology.orchestra.run.smithi016.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-03-31T15:01:40.779 INFO:teuthology.orchestra.run.smithi016.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-03-31T15:01:40.779 INFO:teuthology.orchestra.run.smithi016.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-03-31T15:01:40.779 INFO:teuthology.orchestra.run.smithi016.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-03-31T15:01:40.780 INFO:teuthology.orchestra.run.smithi016.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-03-31T15:01:40.781 INFO:teuthology.orchestra.run.smithi016.stdout:Use 'sudo apt autoremove' to remove them. 2024-03-31T15:01:40.781 INFO:teuthology.orchestra.run.smithi016.stdout:The following additional packages will be installed: 2024-03-31T15:01:40.781 INFO:teuthology.orchestra.run.smithi016.stdout: libbluetooth3 libfl-dev libfl2 libjson-perl libllvm14 liblzo2-2 2024-03-31T15:01:40.781 INFO:teuthology.orchestra.run.smithi016.stdout: libmbim-glib4 libmbim-proxy libmm-glib0 libndp0 libnm0 libpq5 libqmi-glib5 2024-03-31T15:01:40.781 INFO:teuthology.orchestra.run.smithi016.stdout: libqmi-proxy libssl3 libteamdctl0 libuuid1 modemmanager network-manager-pptp 2024-03-31T15:01:40.781 INFO:teuthology.orchestra.run.smithi016.stdout: postgresql-14 postgresql-client-14 postgresql-client-common ppp pptp-linux 2024-03-31T15:01:40.781 INFO:teuthology.orchestra.run.smithi016.stdout: tcl tcl8.6 usb-modeswitch usb-modeswitch-data wpasupplicant 2024-03-31T15:01:40.782 INFO:teuthology.orchestra.run.smithi016.stdout:Suggested packages: 2024-03-31T15:01:40.782 INFO:teuthology.orchestra.run.smithi016.stdout: bison-doc flex-doc indent-doc libssl-doc avahi-autoipd libteam-utils 2024-03-31T15:01:40.782 INFO:teuthology.orchestra.run.smithi016.stdout: postgresql-doc postgresql-doc-14 tcl-tclreadline comgt wvdial 2024-03-31T15:01:40.782 INFO:teuthology.orchestra.run.smithi016.stdout: util-linux-locales wpagui libengine-pkcs11-openssl 2024-03-31T15:01:40.809 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up ceph-volume (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:01:40.837 INFO:teuthology.orchestra.run.smithi016.stdout:The following NEW packages will be installed: 2024-03-31T15:01:40.838 INFO:teuthology.orchestra.run.smithi016.stdout: bison dump flex indent libaio-dev libbluetooth3 libelf-dev libfl-dev libfl2 2024-03-31T15:01:40.838 INFO:teuthology.orchestra.run.smithi016.stdout: libjson-perl libllvm14 liblzo2-2 libmbim-glib4 libmbim-proxy libmm-glib0 2024-03-31T15:01:40.838 INFO:teuthology.orchestra.run.smithi016.stdout: libndp0 libnm0 libpq5 libqmi-glib5 libqmi-proxy libssl-dev libteamdctl0 2024-03-31T15:01:40.838 INFO:teuthology.orchestra.run.smithi016.stdout: modemmanager network-manager network-manager-pptp postgresql postgresql-14 2024-03-31T15:01:40.838 INFO:teuthology.orchestra.run.smithi016.stdout: postgresql-client postgresql-client-14 postgresql-client-common 2024-03-31T15:01:40.839 INFO:teuthology.orchestra.run.smithi016.stdout: postgresql-common postgresql-contrib ppp pptp-linux tcl tcl8.6 2024-03-31T15:01:40.839 INFO:teuthology.orchestra.run.smithi016.stdout: usb-modeswitch usb-modeswitch-data wpasupplicant 2024-03-31T15:01:40.840 INFO:teuthology.orchestra.run.smithi016.stdout:The following packages will be upgraded: 2024-03-31T15:01:40.841 INFO:teuthology.orchestra.run.smithi016.stdout: libssl3 libuuid1 util-linux uuid-dev 2024-03-31T15:01:40.924 INFO:teuthology.orchestra.run.smithi016.stdout:4 upgraded, 39 newly installed, 0 to remove and 297 not upgraded. 2024-03-31T15:01:40.924 INFO:teuthology.orchestra.run.smithi016.stdout:Need to get 55.7 MB of archives. 2024-03-31T15:01:40.924 INFO:teuthology.orchestra.run.smithi016.stdout:After this operation, 202 MB of additional disk space will be used. 2024-03-31T15:01:40.924 INFO:teuthology.orchestra.run.smithi016.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 util-linux amd64 2.37.2-4ubuntu3.3 [1,063 kB] 2024-03-31T15:01:41.082 INFO:teuthology.orchestra.run.smithi016.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy/main amd64 flex amd64 2.6.4-8build2 [307 kB] 2024-03-31T15:01:41.087 INFO:teuthology.orchestra.run.smithi016.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libssl3 amd64 3.0.2-0ubuntu1.15 [1,905 kB] 2024-03-31T15:01:41.118 INFO:teuthology.orchestra.run.smithi016.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 uuid-dev amd64 2.37.2-4ubuntu3.3 [33.1 kB] 2024-03-31T15:01:41.118 INFO:teuthology.orchestra.run.smithi016.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libuuid1 amd64 2.37.2-4ubuntu3.3 [24.4 kB] 2024-03-31T15:01:41.119 INFO:teuthology.orchestra.run.smithi016.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy/main amd64 bison amd64 2:3.8.2+dfsg-1build1 [748 kB] 2024-03-31T15:01:41.126 INFO:teuthology.orchestra.run.smithi016.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblzo2-2 amd64 2.10-2build3 [53.7 kB] 2024-03-31T15:01:41.126 INFO:teuthology.orchestra.run.smithi016.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy/universe amd64 dump amd64 0.4b47-3 [133 kB] 2024-03-31T15:01:41.128 INFO:teuthology.orchestra.run.smithi016.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 indent amd64 2.2.12-1ubuntu0.22.04.1 [90.8 kB] 2024-03-31T15:01:41.129 INFO:teuthology.orchestra.run.smithi016.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy/main amd64 libaio-dev amd64 0.3.112-13build1 [21.2 kB] 2024-03-31T15:01:41.130 INFO:teuthology.orchestra.run.smithi016.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libbluetooth3 amd64 5.64-0ubuntu1.1 [87.0 kB] 2024-03-31T15:01:41.131 INFO:teuthology.orchestra.run.smithi016.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/main amd64 libelf-dev amd64 0.186-1build1 [64.4 kB] 2024-03-31T15:01:41.142 INFO:teuthology.orchestra.run.smithi016.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy/main amd64 libfl2 amd64 2.6.4-8build2 [10.7 kB] 2024-03-31T15:01:41.143 INFO:teuthology.orchestra.run.smithi016.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy/main amd64 libfl-dev amd64 2.6.4-8build2 [6,236 B] 2024-03-31T15:01:41.143 INFO:teuthology.orchestra.run.smithi016.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjson-perl all 4.04000-1 [81.8 kB] 2024-03-31T15:01:41.154 INFO:teuthology.orchestra.run.smithi089.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2024-03-31T15:01:41.162 INFO:teuthology.orchestra.run.smithi016.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libllvm14 amd64 1:14.0.0-1ubuntu1.1 [24.0 MB] 2024-03-31T15:01:41.246 INFO:teuthology.orchestra.run.smithi089.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-03-31T15:01:41.390 INFO:teuthology.orchestra.run.smithi089.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-03-31T15:01:41.395 INFO:teuthology.orchestra.run.smithi016.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libmbim-glib4 amd64 1.28.0-1~ubuntu20.04.1 [191 kB] 2024-03-31T15:01:41.397 INFO:teuthology.orchestra.run.smithi016.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libmbim-proxy amd64 1.28.0-1~ubuntu20.04.1 [6,130 B] 2024-03-31T15:01:41.398 INFO:teuthology.orchestra.run.smithi016.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libmm-glib0 amd64 1.20.0-1~ubuntu22.04.3 [263 kB] 2024-03-31T15:01:41.401 INFO:teuthology.orchestra.run.smithi016.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/main amd64 libndp0 amd64 1.8-0ubuntu3 [12.3 kB] 2024-03-31T15:01:41.401 INFO:teuthology.orchestra.run.smithi016.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libnm0 amd64 1.36.6-0ubuntu2 [456 kB] 2024-03-31T15:01:41.405 INFO:teuthology.orchestra.run.smithi016.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpq5 amd64 14.11-0ubuntu0.22.04.1 [144 kB] 2024-03-31T15:01:41.407 INFO:teuthology.orchestra.run.smithi016.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libqmi-glib5 amd64 1.32.0-1ubuntu0.22.04.1 [772 kB] 2024-03-31T15:01:41.414 INFO:teuthology.orchestra.run.smithi016.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libqmi-proxy amd64 1.32.0-1ubuntu0.22.04.1 [6,072 B] 2024-03-31T15:01:41.415 INFO:teuthology.orchestra.run.smithi016.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libssl-dev amd64 3.0.2-0ubuntu1.15 [2,376 kB] 2024-03-31T15:01:41.437 INFO:teuthology.orchestra.run.smithi016.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/main amd64 libteamdctl0 amd64 1.31-1build2 [12.6 kB] 2024-03-31T15:01:41.437 INFO:teuthology.orchestra.run.smithi016.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 modemmanager amd64 1.20.0-1~ubuntu22.04.3 [1,094 kB] 2024-03-31T15:01:41.447 INFO:teuthology.orchestra.run.smithi016.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 network-manager amd64 1.36.6-0ubuntu2 [2,203 kB] 2024-03-31T15:01:41.470 INFO:teuthology.orchestra.run.smithi016.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/main amd64 ppp amd64 2.4.9-1+1ubuntu3 [363 kB] 2024-03-31T15:01:41.474 INFO:teuthology.orchestra.run.smithi016.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/main amd64 pptp-linux amd64 1.10.0-1build3 [42.5 kB] 2024-03-31T15:01:41.474 INFO:teuthology.orchestra.run.smithi016.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 network-manager-pptp amd64 1.2.10-1 [30.9 kB] 2024-03-31T15:01:41.475 INFO:teuthology.orchestra.run.smithi016.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-client-common all 238 [29.6 kB] 2024-03-31T15:01:41.476 INFO:teuthology.orchestra.run.smithi016.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 postgresql-client-14 amd64 14.11-0ubuntu0.22.04.1 [1,222 kB] 2024-03-31T15:01:41.487 INFO:teuthology.orchestra.run.smithi016.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-common all 238 [169 kB] 2024-03-31T15:01:41.489 INFO:teuthology.orchestra.run.smithi016.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 postgresql-14 amd64 14.11-0ubuntu0.22.04.1 [16.2 MB] 2024-03-31T15:01:41.645 INFO:teuthology.orchestra.run.smithi016.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql all 14+238 [3,288 B] 2024-03-31T15:01:41.646 INFO:teuthology.orchestra.run.smithi016.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-client all 14+238 [3,292 B] 2024-03-31T15:01:41.646 INFO:teuthology.orchestra.run.smithi016.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-contrib all 14+238 [3,292 B] 2024-03-31T15:01:41.646 INFO:teuthology.orchestra.run.smithi016.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 tcl8.6 amd64 8.6.12+dfsg-1build1 [15.0 kB] 2024-03-31T15:01:41.646 INFO:teuthology.orchestra.run.smithi016.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 tcl amd64 8.6.11+1build2 [4,678 B] 2024-03-31T15:01:41.646 INFO:teuthology.orchestra.run.smithi016.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 wpasupplicant amd64 2:2.10-6ubuntu2 [1,481 kB] 2024-03-31T15:01:41.660 INFO:teuthology.orchestra.run.smithi016.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 usb-modeswitch-data all 20191128-4 [33.2 kB] 2024-03-31T15:01:41.660 INFO:teuthology.orchestra.run.smithi016.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/main amd64 usb-modeswitch amd64 2.6.1-3ubuntu2 [46.0 kB] 2024-03-31T15:01:42.052 INFO:teuthology.orchestra.run.smithi016.stdout:Preconfiguring packages ... 2024-03-31T15:01:42.197 INFO:teuthology.orchestra.run.smithi016.stdout:Fetched 55.7 MB in 1s (68.7 MB/s) 2024-03-31T15:01:42.439 INFO:teuthology.orchestra.run.smithi016.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 140915 files and directories currently installed.) 2024-03-31T15:01:42.442 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../util-linux_2.37.2-4ubuntu3.3_amd64.deb ... 2024-03-31T15:01:42.597 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking util-linux (2.37.2-4ubuntu3.3) over (2.37.2-4ubuntu3) ... 2024-03-31T15:01:42.841 INFO:teuthology.orchestra.run.smithi089.stdout:Processing triggers for install-info (6.8-4build1) ... 2024-03-31T15:01:43.240 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up util-linux (2.37.2-4ubuntu3.3) ... 2024-03-31T15:01:43.558 INFO:teuthology.orchestra.run.smithi089.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-03-31T15:01:43.563 DEBUG:teuthology.orchestra.run.smithi089:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install bison flex libelf-dev libssl-dev network-manager iproute2 util-linux dump indent libaio-dev libtool-bin uuid-dev xfslibs-dev postgresql postgresql-client postgresql-common postgresql-contrib 2024-03-31T15:01:43.636 INFO:teuthology.orchestra.run.smithi089.stdout:Reading package lists... 2024-03-31T15:01:43.808 INFO:teuthology.orchestra.run.smithi089.stdout:Building dependency tree... 2024-03-31T15:01:43.809 INFO:teuthology.orchestra.run.smithi089.stdout:Reading state information... 2024-03-31T15:01:43.975 INFO:teuthology.orchestra.run.smithi089.stdout:iproute2 is already the newest version (5.15.0-1ubuntu2). 2024-03-31T15:01:43.975 INFO:teuthology.orchestra.run.smithi089.stdout:libtool-bin is already the newest version (2.4.6-15build2). 2024-03-31T15:01:43.975 INFO:teuthology.orchestra.run.smithi089.stdout:xfslibs-dev is already the newest version (5.13.0-1ubuntu2). 2024-03-31T15:01:43.975 INFO:teuthology.orchestra.run.smithi089.stdout:The following packages were automatically installed and are no longer required: 2024-03-31T15:01:43.975 INFO:teuthology.orchestra.run.smithi089.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-03-31T15:01:43.975 INFO:teuthology.orchestra.run.smithi089.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-03-31T15:01:43.975 INFO:teuthology.orchestra.run.smithi089.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-03-31T15:01:43.976 INFO:teuthology.orchestra.run.smithi089.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-03-31T15:01:43.976 INFO:teuthology.orchestra.run.smithi089.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-03-31T15:01:43.976 INFO:teuthology.orchestra.run.smithi089.stdout:Use 'sudo apt autoremove' to remove them. 2024-03-31T15:01:43.977 INFO:teuthology.orchestra.run.smithi089.stdout:The following additional packages will be installed: 2024-03-31T15:01:43.977 INFO:teuthology.orchestra.run.smithi089.stdout: libbluetooth3 libfl-dev libfl2 libjson-perl libllvm14 liblzo2-2 2024-03-31T15:01:43.977 INFO:teuthology.orchestra.run.smithi089.stdout: libmbim-glib4 libmbim-proxy libmm-glib0 libndp0 libnm0 libpq5 libqmi-glib5 2024-03-31T15:01:43.977 INFO:teuthology.orchestra.run.smithi089.stdout: libqmi-proxy libssl3 libteamdctl0 libuuid1 modemmanager network-manager-pptp 2024-03-31T15:01:43.977 INFO:teuthology.orchestra.run.smithi089.stdout: postgresql-14 postgresql-client-14 postgresql-client-common ppp pptp-linux 2024-03-31T15:01:43.977 INFO:teuthology.orchestra.run.smithi089.stdout: tcl tcl8.6 usb-modeswitch usb-modeswitch-data wpasupplicant 2024-03-31T15:01:43.978 INFO:teuthology.orchestra.run.smithi089.stdout:Suggested packages: 2024-03-31T15:01:43.978 INFO:teuthology.orchestra.run.smithi089.stdout: bison-doc flex-doc indent-doc libssl-doc avahi-autoipd libteam-utils 2024-03-31T15:01:43.978 INFO:teuthology.orchestra.run.smithi089.stdout: postgresql-doc postgresql-doc-14 tcl-tclreadline comgt wvdial 2024-03-31T15:01:43.978 INFO:teuthology.orchestra.run.smithi089.stdout: util-linux-locales wpagui libengine-pkcs11-openssl 2024-03-31T15:01:44.032 INFO:teuthology.orchestra.run.smithi089.stdout:The following NEW packages will be installed: 2024-03-31T15:01:44.033 INFO:teuthology.orchestra.run.smithi089.stdout: bison dump flex indent libaio-dev libbluetooth3 libelf-dev libfl-dev libfl2 2024-03-31T15:01:44.033 INFO:teuthology.orchestra.run.smithi089.stdout: libjson-perl libllvm14 liblzo2-2 libmbim-glib4 libmbim-proxy libmm-glib0 2024-03-31T15:01:44.034 INFO:teuthology.orchestra.run.smithi089.stdout: libndp0 libnm0 libpq5 libqmi-glib5 libqmi-proxy libssl-dev libteamdctl0 2024-03-31T15:01:44.034 INFO:teuthology.orchestra.run.smithi089.stdout: modemmanager network-manager network-manager-pptp postgresql postgresql-14 2024-03-31T15:01:44.034 INFO:teuthology.orchestra.run.smithi089.stdout: postgresql-client postgresql-client-14 postgresql-client-common 2024-03-31T15:01:44.034 INFO:teuthology.orchestra.run.smithi089.stdout: postgresql-common postgresql-contrib ppp pptp-linux tcl tcl8.6 2024-03-31T15:01:44.034 INFO:teuthology.orchestra.run.smithi089.stdout: usb-modeswitch usb-modeswitch-data wpasupplicant 2024-03-31T15:01:44.035 INFO:teuthology.orchestra.run.smithi089.stdout:The following packages will be upgraded: 2024-03-31T15:01:44.036 INFO:teuthology.orchestra.run.smithi089.stdout: libssl3 libuuid1 util-linux uuid-dev 2024-03-31T15:01:44.220 INFO:teuthology.orchestra.run.smithi089.stdout:4 upgraded, 39 newly installed, 0 to remove and 297 not upgraded. 2024-03-31T15:01:44.220 INFO:teuthology.orchestra.run.smithi089.stdout:Need to get 55.7 MB of archives. 2024-03-31T15:01:44.221 INFO:teuthology.orchestra.run.smithi089.stdout:After this operation, 202 MB of additional disk space will be used. 2024-03-31T15:01:44.221 INFO:teuthology.orchestra.run.smithi089.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 util-linux amd64 2.37.2-4ubuntu3.3 [1,063 kB] 2024-03-31T15:01:44.298 INFO:teuthology.orchestra.run.smithi016.stdout:fstrim.service is a disabled or a static unit not running, not starting it. 2024-03-31T15:01:44.560 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package flex. 2024-03-31T15:01:44.634 INFO:teuthology.orchestra.run.smithi016.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 140915 files and directories currently installed.) 2024-03-31T15:01:44.637 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../flex_2.6.4-8build2_amd64.deb ... 2024-03-31T15:01:44.684 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking flex (2.6.4-8build2) ... 2024-03-31T15:01:44.866 INFO:teuthology.orchestra.run.smithi089.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy/main amd64 flex amd64 2.6.4-8build2 [307 kB] 2024-03-31T15:01:44.882 INFO:teuthology.orchestra.run.smithi089.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libssl3 amd64 3.0.2-0ubuntu1.15 [1,905 kB] 2024-03-31T15:01:45.009 INFO:teuthology.orchestra.run.smithi089.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 uuid-dev amd64 2.37.2-4ubuntu3.3 [33.1 kB] 2024-03-31T15:01:45.010 INFO:teuthology.orchestra.run.smithi089.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libuuid1 amd64 2.37.2-4ubuntu3.3 [24.4 kB] 2024-03-31T15:01:45.010 INFO:teuthology.orchestra.run.smithi089.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy/main amd64 bison amd64 2:3.8.2+dfsg-1build1 [748 kB] 2024-03-31T15:01:45.039 INFO:teuthology.orchestra.run.smithi089.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblzo2-2 amd64 2.10-2build3 [53.7 kB] 2024-03-31T15:01:45.040 INFO:teuthology.orchestra.run.smithi089.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy/universe amd64 dump amd64 0.4b47-3 [133 kB] 2024-03-31T15:01:45.044 INFO:teuthology.orchestra.run.smithi089.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 indent amd64 2.2.12-1ubuntu0.22.04.1 [90.8 kB] 2024-03-31T15:01:45.046 INFO:teuthology.orchestra.run.smithi089.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy/main amd64 libaio-dev amd64 0.3.112-13build1 [21.2 kB] 2024-03-31T15:01:45.047 INFO:teuthology.orchestra.run.smithi089.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libbluetooth3 amd64 5.64-0ubuntu1.1 [87.0 kB] 2024-03-31T15:01:45.213 INFO:teuthology.orchestra.run.smithi089.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/main amd64 libelf-dev amd64 0.186-1build1 [64.4 kB] 2024-03-31T15:01:45.255 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../libssl3_3.0.2-0ubuntu1.15_amd64.deb ... 2024-03-31T15:01:45.425 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking libssl3:amd64 (3.0.2-0ubuntu1.15) over (3.0.2-0ubuntu1.7) ... 2024-03-31T15:01:45.494 INFO:teuthology.orchestra.run.smithi089.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy/main amd64 libfl2 amd64 2.6.4-8build2 [10.7 kB] 2024-03-31T15:01:45.504 INFO:teuthology.orchestra.run.smithi089.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy/main amd64 libfl-dev amd64 2.6.4-8build2 [6,236 B] 2024-03-31T15:01:45.510 INFO:teuthology.orchestra.run.smithi089.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjson-perl all 4.04000-1 [81.8 kB] 2024-03-31T15:01:45.598 INFO:teuthology.orchestra.run.smithi089.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libllvm14 amd64 1:14.0.0-1ubuntu1.1 [24.0 MB] 2024-03-31T15:01:45.849 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up libssl3:amd64 (3.0.2-0ubuntu1.15) ... 2024-03-31T15:01:46.371 INFO:teuthology.orchestra.run.smithi016.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 140988 files and directories currently installed.) 2024-03-31T15:01:46.375 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../uuid-dev_2.37.2-4ubuntu3.3_amd64.deb ... 2024-03-31T15:01:46.499 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking uuid-dev:amd64 (2.37.2-4ubuntu3.3) over (2.37.2-4ubuntu3) ... 2024-03-31T15:01:46.568 INFO:teuthology.orchestra.run.smithi089.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libmbim-glib4 amd64 1.28.0-1~ubuntu20.04.1 [191 kB] 2024-03-31T15:01:46.569 INFO:teuthology.orchestra.run.smithi089.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libmbim-proxy amd64 1.28.0-1~ubuntu20.04.1 [6,130 B] 2024-03-31T15:01:46.570 INFO:teuthology.orchestra.run.smithi089.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libmm-glib0 amd64 1.20.0-1~ubuntu22.04.3 [263 kB] 2024-03-31T15:01:46.597 INFO:teuthology.orchestra.run.smithi089.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/main amd64 libndp0 amd64 1.8-0ubuntu3 [12.3 kB] 2024-03-31T15:01:46.597 INFO:teuthology.orchestra.run.smithi089.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libnm0 amd64 1.36.6-0ubuntu2 [456 kB] 2024-03-31T15:01:46.602 INFO:teuthology.orchestra.run.smithi089.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpq5 amd64 14.11-0ubuntu0.22.04.1 [144 kB] 2024-03-31T15:01:46.768 INFO:teuthology.orchestra.run.smithi089.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libqmi-glib5 amd64 1.32.0-1ubuntu0.22.04.1 [772 kB] 2024-03-31T15:01:46.888 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../libuuid1_2.37.2-4ubuntu3.3_amd64.deb ... 2024-03-31T15:01:47.022 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking libuuid1:amd64 (2.37.2-4ubuntu3.3) over (2.37.2-4ubuntu3) ... 2024-03-31T15:01:47.331 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up libuuid1:amd64 (2.37.2-4ubuntu3.3) ... 2024-03-31T15:01:47.375 INFO:teuthology.orchestra.run.smithi089.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libqmi-proxy amd64 1.32.0-1ubuntu0.22.04.1 [6,072 B] 2024-03-31T15:01:47.375 INFO:teuthology.orchestra.run.smithi089.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libssl-dev amd64 3.0.2-0ubuntu1.15 [2,376 kB] 2024-03-31T15:01:47.555 INFO:teuthology.orchestra.run.smithi089.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/main amd64 libteamdctl0 amd64 1.31-1build2 [12.6 kB] 2024-03-31T15:01:47.557 INFO:teuthology.orchestra.run.smithi089.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 modemmanager amd64 1.20.0-1~ubuntu22.04.3 [1,094 kB] 2024-03-31T15:01:47.593 INFO:teuthology.orchestra.run.smithi089.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 network-manager amd64 1.36.6-0ubuntu2 [2,203 kB] 2024-03-31T15:01:47.655 INFO:teuthology.orchestra.run.smithi089.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/main amd64 ppp amd64 2.4.9-1+1ubuntu3 [363 kB] 2024-03-31T15:01:47.660 INFO:teuthology.orchestra.run.smithi089.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/main amd64 pptp-linux amd64 1.10.0-1build3 [42.5 kB] 2024-03-31T15:01:47.661 INFO:teuthology.orchestra.run.smithi089.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 network-manager-pptp amd64 1.2.10-1 [30.9 kB] 2024-03-31T15:01:47.661 INFO:teuthology.orchestra.run.smithi089.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-client-common all 238 [29.6 kB] 2024-03-31T15:01:47.662 INFO:teuthology.orchestra.run.smithi089.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 postgresql-client-14 amd64 14.11-0ubuntu0.22.04.1 [1,222 kB] 2024-03-31T15:01:47.715 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package bison. 2024-03-31T15:01:47.792 INFO:teuthology.orchestra.run.smithi016.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 140988 files and directories currently installed.) 2024-03-31T15:01:47.796 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../00-bison_2%3a3.8.2+dfsg-1build1_amd64.deb ... 2024-03-31T15:01:47.834 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking bison (2:3.8.2+dfsg-1build1) ... 2024-03-31T15:01:47.870 INFO:teuthology.orchestra.run.smithi089.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-common all 238 [169 kB] 2024-03-31T15:01:48.251 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package liblzo2-2:amd64. 2024-03-31T15:01:48.267 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../01-liblzo2-2_2.10-2build3_amd64.deb ... 2024-03-31T15:01:48.291 INFO:teuthology.orchestra.run.smithi089.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 postgresql-14 amd64 14.11-0ubuntu0.22.04.1 [16.2 MB] 2024-03-31T15:01:48.301 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking liblzo2-2:amd64 (2.10-2build3) ... 2024-03-31T15:01:48.579 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package dump. 2024-03-31T15:01:48.596 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../02-dump_0.4b47-3_amd64.deb ... 2024-03-31T15:01:48.638 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking dump (0.4b47-3) ... 2024-03-31T15:01:48.941 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package indent. 2024-03-31T15:01:48.958 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../03-indent_2.2.12-1ubuntu0.22.04.1_amd64.deb ... 2024-03-31T15:01:48.999 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking indent (2.2.12-1ubuntu0.22.04.1) ... 2024-03-31T15:01:49.044 INFO:teuthology.orchestra.run.smithi089.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql all 14+238 [3,288 B] 2024-03-31T15:01:49.045 INFO:teuthology.orchestra.run.smithi089.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-client all 14+238 [3,292 B] 2024-03-31T15:01:49.045 INFO:teuthology.orchestra.run.smithi089.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-contrib all 14+238 [3,292 B] 2024-03-31T15:01:49.045 INFO:teuthology.orchestra.run.smithi089.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 tcl8.6 amd64 8.6.12+dfsg-1build1 [15.0 kB] 2024-03-31T15:01:49.045 INFO:teuthology.orchestra.run.smithi089.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 tcl amd64 8.6.11+1build2 [4,678 B] 2024-03-31T15:01:49.045 INFO:teuthology.orchestra.run.smithi089.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 wpasupplicant amd64 2:2.10-6ubuntu2 [1,481 kB] 2024-03-31T15:01:49.080 INFO:teuthology.orchestra.run.smithi089.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 usb-modeswitch-data all 20191128-4 [33.2 kB] 2024-03-31T15:01:49.081 INFO:teuthology.orchestra.run.smithi089.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/main amd64 usb-modeswitch amd64 2.6.1-3ubuntu2 [46.0 kB] 2024-03-31T15:01:49.252 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package libaio-dev:amd64. 2024-03-31T15:01:49.269 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../04-libaio-dev_0.3.112-13build1_amd64.deb ... 2024-03-31T15:01:49.310 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking libaio-dev:amd64 (0.3.112-13build1) ... 2024-03-31T15:01:49.458 INFO:teuthology.orchestra.run.smithi089.stdout:Preconfiguring packages ... 2024-03-31T15:01:49.598 INFO:teuthology.orchestra.run.smithi089.stdout:Fetched 55.7 MB in 5s (11.1 MB/s) 2024-03-31T15:01:49.683 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package libbluetooth3:amd64. 2024-03-31T15:01:49.700 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../05-libbluetooth3_5.64-0ubuntu1.1_amd64.deb ... 2024-03-31T15:01:49.741 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking libbluetooth3:amd64 (5.64-0ubuntu1.1) ... 2024-03-31T15:01:49.887 INFO:teuthology.orchestra.run.smithi089.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 140915 files and directories currently installed.) 2024-03-31T15:01:49.891 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../util-linux_2.37.2-4ubuntu3.3_amd64.deb ... 2024-03-31T15:01:50.011 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package libelf-dev:amd64. 2024-03-31T15:01:50.028 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../06-libelf-dev_0.186-1build1_amd64.deb ... 2024-03-31T15:01:50.032 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking util-linux (2.37.2-4ubuntu3.3) over (2.37.2-4ubuntu3) ... 2024-03-31T15:01:50.120 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking libelf-dev:amd64 (0.186-1build1) ... 2024-03-31T15:01:50.482 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package libfl2:amd64. 2024-03-31T15:01:50.498 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../07-libfl2_2.6.4-8build2_amd64.deb ... 2024-03-31T15:01:50.540 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking libfl2:amd64 (2.6.4-8build2) ... 2024-03-31T15:01:50.690 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up util-linux (2.37.2-4ubuntu3.3) ... 2024-03-31T15:01:50.792 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package libfl-dev:amd64. 2024-03-31T15:01:50.809 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../08-libfl-dev_2.6.4-8build2_amd64.deb ... 2024-03-31T15:01:50.851 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking libfl-dev:amd64 (2.6.4-8build2) ... 2024-03-31T15:01:51.104 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package libjson-perl. 2024-03-31T15:01:51.120 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../09-libjson-perl_4.04000-1_all.deb ... 2024-03-31T15:01:51.153 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking libjson-perl (4.04000-1) ... 2024-03-31T15:01:51.465 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package libllvm14:amd64. 2024-03-31T15:01:51.481 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../10-libllvm14_1%3a14.0.0-1ubuntu1.1_amd64.deb ... 2024-03-31T15:01:51.524 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking libllvm14:amd64 (1:14.0.0-1ubuntu1.1) ... 2024-03-31T15:01:51.805 INFO:teuthology.orchestra.run.smithi089.stdout:fstrim.service is a disabled or a static unit not running, not starting it. 2024-03-31T15:01:52.049 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package flex. 2024-03-31T15:01:52.119 INFO:teuthology.orchestra.run.smithi089.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 140915 files and directories currently installed.) 2024-03-31T15:01:52.123 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../flex_2.6.4-8build2_amd64.deb ... 2024-03-31T15:01:52.161 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking flex (2.6.4-8build2) ... 2024-03-31T15:01:52.715 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../libssl3_3.0.2-0ubuntu1.15_amd64.deb ... 2024-03-31T15:01:52.901 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package libmbim-glib4:amd64. 2024-03-31T15:01:52.918 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../11-libmbim-glib4_1.28.0-1~ubuntu20.04.1_amd64.deb ... 2024-03-31T15:01:52.918 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking libssl3:amd64 (3.0.2-0ubuntu1.15) over (3.0.2-0ubuntu1.7) ... 2024-03-31T15:01:52.960 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking libmbim-glib4:amd64 (1.28.0-1~ubuntu20.04.1) ... 2024-03-31T15:01:53.221 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package libmbim-proxy. 2024-03-31T15:01:53.237 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../12-libmbim-proxy_1.28.0-1~ubuntu20.04.1_amd64.deb ... 2024-03-31T15:01:53.299 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up libssl3:amd64 (3.0.2-0ubuntu1.15) ... 2024-03-31T15:01:53.312 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking libmbim-proxy (1.28.0-1~ubuntu20.04.1) ... 2024-03-31T15:01:53.615 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package libmm-glib0:amd64. 2024-03-31T15:01:53.632 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../13-libmm-glib0_1.20.0-1~ubuntu22.04.3_amd64.deb ... 2024-03-31T15:01:53.673 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking libmm-glib0:amd64 (1.20.0-1~ubuntu22.04.3) ... 2024-03-31T15:01:53.871 INFO:teuthology.orchestra.run.smithi089.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 140988 files and directories currently installed.) 2024-03-31T15:01:53.875 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../uuid-dev_2.37.2-4ubuntu3.3_amd64.deb ... 2024-03-31T15:01:54.001 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking uuid-dev:amd64 (2.37.2-4ubuntu3.3) over (2.37.2-4ubuntu3) ... 2024-03-31T15:01:54.026 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package libndp0:amd64. 2024-03-31T15:01:54.043 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../14-libndp0_1.8-0ubuntu3_amd64.deb ... 2024-03-31T15:01:54.085 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking libndp0:amd64 (1.8-0ubuntu3) ... 2024-03-31T15:01:54.413 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package libnm0:amd64. 2024-03-31T15:01:54.429 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../15-libnm0_1.36.6-0ubuntu2_amd64.deb ... 2024-03-31T15:01:54.455 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../libuuid1_2.37.2-4ubuntu3.3_amd64.deb ... 2024-03-31T15:01:54.470 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking libnm0:amd64 (1.36.6-0ubuntu2) ... 2024-03-31T15:01:54.557 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking libuuid1:amd64 (2.37.2-4ubuntu3.3) over (2.37.2-4ubuntu3) ... 2024-03-31T15:01:54.849 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package libpq5:amd64. 2024-03-31T15:01:54.866 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../16-libpq5_14.11-0ubuntu0.22.04.1_amd64.deb ... 2024-03-31T15:01:54.898 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up libuuid1:amd64 (2.37.2-4ubuntu3.3) ... 2024-03-31T15:01:54.907 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking libpq5:amd64 (14.11-0ubuntu0.22.04.1) ... 2024-03-31T15:01:55.278 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package libqmi-glib5:amd64. 2024-03-31T15:01:55.295 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../17-libqmi-glib5_1.32.0-1ubuntu0.22.04.1_amd64.deb ... 2024-03-31T15:01:55.308 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package bison. 2024-03-31T15:01:55.336 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking libqmi-glib5:amd64 (1.32.0-1ubuntu0.22.04.1) ... 2024-03-31T15:01:55.384 INFO:teuthology.orchestra.run.smithi089.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 140988 files and directories currently installed.) 2024-03-31T15:01:55.388 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../00-bison_2%3a3.8.2+dfsg-1build1_amd64.deb ... 2024-03-31T15:01:55.435 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking bison (2:3.8.2+dfsg-1build1) ... 2024-03-31T15:01:55.681 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package libqmi-proxy. 2024-03-31T15:01:55.697 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../18-libqmi-proxy_1.32.0-1ubuntu0.22.04.1_amd64.deb ... 2024-03-31T15:01:55.730 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking libqmi-proxy (1.32.0-1ubuntu0.22.04.1) ... 2024-03-31T15:01:55.853 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package liblzo2-2:amd64. 2024-03-31T15:01:55.868 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../01-liblzo2-2_2.10-2build3_amd64.deb ... 2024-03-31T15:01:55.911 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking liblzo2-2:amd64 (2.10-2build3) ... 2024-03-31T15:01:55.967 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package libssl-dev:amd64. 2024-03-31T15:01:55.983 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../19-libssl-dev_3.0.2-0ubuntu1.15_amd64.deb ... 2024-03-31T15:01:56.025 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking libssl-dev:amd64 (3.0.2-0ubuntu1.15) ... 2024-03-31T15:01:56.206 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package dump. 2024-03-31T15:01:56.221 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../02-dump_0.4b47-3_amd64.deb ... 2024-03-31T15:01:56.256 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking dump (0.4b47-3) ... 2024-03-31T15:01:56.562 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package libteamdctl0:amd64. 2024-03-31T15:01:56.568 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package indent. 2024-03-31T15:01:56.578 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../20-libteamdctl0_1.31-1build2_amd64.deb ... 2024-03-31T15:01:56.584 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../03-indent_2.2.12-1ubuntu0.22.04.1_amd64.deb ... 2024-03-31T15:01:56.620 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking libteamdctl0:amd64 (1.31-1build2) ... 2024-03-31T15:01:56.635 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking indent (2.2.12-1ubuntu0.22.04.1) ... 2024-03-31T15:01:56.896 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package libaio-dev:amd64. 2024-03-31T15:01:56.911 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../04-libaio-dev_0.3.112-13build1_amd64.deb ... 2024-03-31T15:01:56.954 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking libaio-dev:amd64 (0.3.112-13build1) ... 2024-03-31T15:01:57.015 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package modemmanager. 2024-03-31T15:01:57.032 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../21-modemmanager_1.20.0-1~ubuntu22.04.3_amd64.deb ... 2024-03-31T15:01:57.073 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking modemmanager (1.20.0-1~ubuntu22.04.3) ... 2024-03-31T15:01:57.283 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package libbluetooth3:amd64. 2024-03-31T15:01:57.298 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../05-libbluetooth3_5.64-0ubuntu1.1_amd64.deb ... 2024-03-31T15:01:57.333 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking libbluetooth3:amd64 (5.64-0ubuntu1.1) ... 2024-03-31T15:01:57.637 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package libelf-dev:amd64. 2024-03-31T15:01:57.653 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../06-libelf-dev_0.186-1build1_amd64.deb ... 2024-03-31T15:01:57.695 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking libelf-dev:amd64 (0.186-1build1) ... 2024-03-31T15:01:57.697 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package network-manager. 2024-03-31T15:01:57.713 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../22-network-manager_1.36.6-0ubuntu2_amd64.deb ... 2024-03-31T15:01:57.764 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking network-manager (1.36.6-0ubuntu2) ... 2024-03-31T15:01:58.040 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package libfl2:amd64. 2024-03-31T15:01:58.056 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../07-libfl2_2.6.4-8build2_amd64.deb ... 2024-03-31T15:01:58.098 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking libfl2:amd64 (2.6.4-8build2) ... 2024-03-31T15:01:58.326 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package libfl-dev:amd64. 2024-03-31T15:01:58.338 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package ppp. 2024-03-31T15:01:58.342 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../08-libfl-dev_2.6.4-8build2_amd64.deb ... 2024-03-31T15:01:58.354 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../23-ppp_2.4.9-1+1ubuntu3_amd64.deb ... 2024-03-31T15:01:58.376 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking libfl-dev:amd64 (2.6.4-8build2) ... 2024-03-31T15:01:58.637 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package libjson-perl. 2024-03-31T15:01:58.653 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../09-libjson-perl_4.04000-1_all.deb ... 2024-03-31T15:01:58.696 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking libjson-perl (4.04000-1) ... 2024-03-31T15:01:58.709 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking ppp (2.4.9-1+1ubuntu3) ... 2024-03-31T15:01:58.982 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package libllvm14:amd64. 2024-03-31T15:01:58.998 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../10-libllvm14_1%3a14.0.0-1ubuntu1.1_amd64.deb ... 2024-03-31T15:01:59.026 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package pptp-linux. 2024-03-31T15:01:59.040 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking libllvm14:amd64 (1:14.0.0-1ubuntu1.1) ... 2024-03-31T15:01:59.041 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../24-pptp-linux_1.10.0-1build3_amd64.deb ... 2024-03-31T15:01:59.109 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking pptp-linux (1.10.0-1build3) ... 2024-03-31T15:01:59.362 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package network-manager-pptp. 2024-03-31T15:01:59.378 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../25-network-manager-pptp_1.2.10-1_amd64.deb ... 2024-03-31T15:01:59.420 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking network-manager-pptp (1.2.10-1) ... 2024-03-31T15:01:59.723 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package postgresql-client-common. 2024-03-31T15:01:59.739 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../26-postgresql-client-common_238_all.deb ... 2024-03-31T15:01:59.873 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking postgresql-client-common (238) ... 2024-03-31T15:02:00.136 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package postgresql-client-14. 2024-03-31T15:02:00.152 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../27-postgresql-client-14_14.11-0ubuntu0.22.04.1_amd64.deb ... 2024-03-31T15:02:00.193 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking postgresql-client-14 (14.11-0ubuntu0.22.04.1) ... 2024-03-31T15:02:00.394 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package libmbim-glib4:amd64. 2024-03-31T15:02:00.409 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../11-libmbim-glib4_1.28.0-1~ubuntu20.04.1_amd64.deb ... 2024-03-31T15:02:00.454 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking libmbim-glib4:amd64 (1.28.0-1~ubuntu20.04.1) ... 2024-03-31T15:02:00.666 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package libmbim-proxy. 2024-03-31T15:02:00.680 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../12-libmbim-proxy_1.28.0-1~ubuntu20.04.1_amd64.deb ... 2024-03-31T15:02:00.723 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking libmbim-proxy (1.28.0-1~ubuntu20.04.1) ... 2024-03-31T15:02:00.983 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package postgresql-common. 2024-03-31T15:02:01.000 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../28-postgresql-common_238_all.deb ... 2024-03-31T15:02:01.060 INFO:teuthology.orchestra.run.smithi016.stdout:Adding 'diversion of /usr/bin/pg_config to /usr/bin/pg_config.libpq-dev by postgresql-common' 2024-03-31T15:02:01.076 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package libmm-glib0:amd64. 2024-03-31T15:02:01.090 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking postgresql-common (238) ... 2024-03-31T15:02:01.091 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../13-libmm-glib0_1.20.0-1~ubuntu22.04.3_amd64.deb ... 2024-03-31T15:02:01.134 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking libmm-glib0:amd64 (1.20.0-1~ubuntu22.04.3) ... 2024-03-31T15:02:01.470 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package libndp0:amd64. 2024-03-31T15:02:01.486 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../14-libndp0_1.8-0ubuntu3_amd64.deb ... 2024-03-31T15:02:01.549 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package postgresql-14. 2024-03-31T15:02:01.554 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking libndp0:amd64 (1.8-0ubuntu3) ... 2024-03-31T15:02:01.565 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../29-postgresql-14_14.11-0ubuntu0.22.04.1_amd64.deb ... 2024-03-31T15:02:01.602 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking postgresql-14 (14.11-0ubuntu0.22.04.1) ... 2024-03-31T15:02:01.840 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package libnm0:amd64. 2024-03-31T15:02:01.855 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../15-libnm0_1.36.6-0ubuntu2_amd64.deb ... 2024-03-31T15:02:01.890 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking libnm0:amd64 (1.36.6-0ubuntu2) ... 2024-03-31T15:02:02.268 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package libpq5:amd64. 2024-03-31T15:02:02.285 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../16-libpq5_14.11-0ubuntu0.22.04.1_amd64.deb ... 2024-03-31T15:02:02.326 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking libpq5:amd64 (14.11-0ubuntu0.22.04.1) ... 2024-03-31T15:02:02.751 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package postgresql. 2024-03-31T15:02:02.767 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../30-postgresql_14+238_all.deb ... 2024-03-31T15:02:02.806 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package libqmi-glib5:amd64. 2024-03-31T15:02:02.809 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking postgresql (14+238) ... 2024-03-31T15:02:02.821 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../17-libqmi-glib5_1.32.0-1ubuntu0.22.04.1_amd64.deb ... 2024-03-31T15:02:02.864 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking libqmi-glib5:amd64 (1.32.0-1ubuntu0.22.04.1) ... 2024-03-31T15:02:03.037 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package postgresql-client. 2024-03-31T15:02:03.052 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../31-postgresql-client_14+238_all.deb ... 2024-03-31T15:02:03.087 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking postgresql-client (14+238) ... 2024-03-31T15:02:03.184 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package libqmi-proxy. 2024-03-31T15:02:03.198 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../18-libqmi-proxy_1.32.0-1ubuntu0.22.04.1_amd64.deb ... 2024-03-31T15:02:03.242 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking libqmi-proxy (1.32.0-1ubuntu0.22.04.1) ... 2024-03-31T15:02:03.322 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package postgresql-contrib. 2024-03-31T15:02:03.339 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../32-postgresql-contrib_14+238_all.deb ... 2024-03-31T15:02:03.380 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking postgresql-contrib (14+238) ... 2024-03-31T15:02:03.478 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package libssl-dev:amd64. 2024-03-31T15:02:03.493 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../19-libssl-dev_3.0.2-0ubuntu1.15_amd64.deb ... 2024-03-31T15:02:03.536 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking libssl-dev:amd64 (3.0.2-0ubuntu1.15) ... 2024-03-31T15:02:03.733 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package tcl8.6. 2024-03-31T15:02:03.750 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../33-tcl8.6_8.6.12+dfsg-1build1_amd64.deb ... 2024-03-31T15:02:03.791 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking tcl8.6 (8.6.12+dfsg-1build1) ... 2024-03-31T15:02:04.015 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package libteamdctl0:amd64. 2024-03-31T15:02:04.029 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../20-libteamdctl0_1.31-1build2_amd64.deb ... 2024-03-31T15:02:04.069 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package tcl. 2024-03-31T15:02:04.073 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking libteamdctl0:amd64 (1.31-1build2) ... 2024-03-31T15:02:04.086 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../34-tcl_8.6.11+1build2_amd64.deb ... 2024-03-31T15:02:04.132 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking tcl (8.6.11+1build2) ... 2024-03-31T15:02:04.439 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package wpasupplicant. 2024-03-31T15:02:04.443 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package modemmanager. 2024-03-31T15:02:04.455 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../35-wpasupplicant_2%3a2.10-6ubuntu2_amd64.deb ... 2024-03-31T15:02:04.459 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../21-modemmanager_1.20.0-1~ubuntu22.04.3_amd64.deb ... 2024-03-31T15:02:04.497 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking wpasupplicant (2:2.10-6ubuntu2) ... 2024-03-31T15:02:04.501 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking modemmanager (1.20.0-1~ubuntu22.04.3) ... 2024-03-31T15:02:04.834 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package usb-modeswitch-data. 2024-03-31T15:02:04.851 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../36-usb-modeswitch-data_20191128-4_all.deb ... 2024-03-31T15:02:04.984 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking usb-modeswitch-data (20191128-4) ... 2024-03-31T15:02:05.075 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package network-manager. 2024-03-31T15:02:05.091 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../22-network-manager_1.36.6-0ubuntu2_amd64.deb ... 2024-03-31T15:02:05.166 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking network-manager (1.36.6-0ubuntu2) ... 2024-03-31T15:02:05.362 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package usb-modeswitch. 2024-03-31T15:02:05.379 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../37-usb-modeswitch_2.6.1-3ubuntu2_amd64.deb ... 2024-03-31T15:02:05.782 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package ppp. 2024-03-31T15:02:05.799 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../23-ppp_2.4.9-1+1ubuntu3_amd64.deb ... 2024-03-31T15:02:05.848 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking usb-modeswitch (2.6.1-3ubuntu2) ... 2024-03-31T15:02:06.141 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up postgresql-client-common (238) ... 2024-03-31T15:02:06.170 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking ppp (2.4.9-1+1ubuntu3) ... 2024-03-31T15:02:06.352 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up flex (2.6.4-8build2) ... 2024-03-31T15:02:06.478 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up tcl8.6 (8.6.12+dfsg-1build1) ... 2024-03-31T15:02:06.479 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package pptp-linux. 2024-03-31T15:02:06.493 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../24-pptp-linux_1.10.0-1build3_amd64.deb ... 2024-03-31T15:02:06.554 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking pptp-linux (1.10.0-1build3) ... 2024-03-31T15:02:06.620 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up libpq5:amd64 (14.11-0ubuntu0.22.04.1) ... 2024-03-31T15:02:06.747 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up liblzo2-2:amd64 (2.10-2build3) ... 2024-03-31T15:02:06.782 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package network-manager-pptp. 2024-03-31T15:02:06.796 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../25-network-manager-pptp_1.2.10-1_amd64.deb ... 2024-03-31T15:02:06.840 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking network-manager-pptp (1.2.10-1) ... 2024-03-31T15:02:06.864 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up libaio-dev:amd64 (0.3.112-13build1) ... 2024-03-31T15:02:06.982 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up postgresql-client-14 (14.11-0ubuntu0.22.04.1) ... 2024-03-31T15:02:07.143 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package postgresql-client-common. 2024-03-31T15:02:07.158 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../26-postgresql-client-common_238_all.deb ... 2024-03-31T15:02:07.201 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking postgresql-client-common (238) ... 2024-03-31T15:02:07.530 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package postgresql-client-14. 2024-03-31T15:02:07.545 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../27-postgresql-client-14_14.11-0ubuntu0.22.04.1_amd64.deb ... 2024-03-31T15:02:07.588 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking postgresql-client-14 (14.11-0ubuntu0.22.04.1) ... 2024-03-31T15:02:07.950 INFO:teuthology.orchestra.run.smithi016.stdout:update-alternatives: using /usr/share/postgresql/14/man/man1/psql.1.gz to provide /usr/share/man/man1/psql.1.gz (psql.1.gz) in auto mode 2024-03-31T15:02:08.054 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up wpasupplicant (2:2.10-6ubuntu2) ... 2024-03-31T15:02:08.152 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package postgresql-common. 2024-03-31T15:02:08.166 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../28-postgresql-common_238_all.deb ... 2024-03-31T15:02:08.219 INFO:teuthology.orchestra.run.smithi089.stdout:Adding 'diversion of /usr/bin/pg_config to /usr/bin/pg_config.libpq-dev by postgresql-common' 2024-03-31T15:02:08.250 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking postgresql-common (238) ... 2024-03-31T15:02:08.475 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/dbus-fi.w1.wpa_supplicant1.service -> /lib/systemd/system/wpa_supplicant.service. 2024-03-31T15:02:08.476 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/wpa_supplicant.service -> /lib/systemd/system/wpa_supplicant.service. 2024-03-31T15:02:08.751 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package postgresql-14. 2024-03-31T15:02:08.765 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../29-postgresql-14_14.11-0ubuntu0.22.04.1_amd64.deb ... 2024-03-31T15:02:08.812 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking postgresql-14 (14.11-0ubuntu0.22.04.1) ... 2024-03-31T15:02:09.116 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up uuid-dev:amd64 (2.37.2-4ubuntu3.3) ... 2024-03-31T15:02:09.242 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up libteamdctl0:amd64 (1.31-1build2) ... 2024-03-31T15:02:09.401 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up libssl-dev:amd64 (3.0.2-0ubuntu1.15) ... 2024-03-31T15:02:09.527 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up usb-modeswitch-data (20191128-4) ... 2024-03-31T15:02:09.653 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up libelf-dev:amd64 (0.186-1build1) ... 2024-03-31T15:02:09.779 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up libfl2:amd64 (2.6.4-8build2) ... 2024-03-31T15:02:09.905 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up libnm0:amd64 (1.36.6-0ubuntu2) ... 2024-03-31T15:02:09.962 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package postgresql. 2024-03-31T15:02:09.977 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../30-postgresql_14+238_all.deb ... 2024-03-31T15:02:10.012 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking postgresql (14+238) ... 2024-03-31T15:02:10.014 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up ppp (2.4.9-1+1ubuntu3) ... 2024-03-31T15:02:10.248 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package postgresql-client. 2024-03-31T15:02:10.262 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../31-postgresql-client_14+238_all.deb ... 2024-03-31T15:02:10.306 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking postgresql-client (14+238) ... 2024-03-31T15:02:10.550 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package postgresql-contrib. 2024-03-31T15:02:10.564 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../32-postgresql-contrib_14+238_all.deb ... 2024-03-31T15:02:10.608 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking postgresql-contrib (14+238) ... 2024-03-31T15:02:10.686 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up libllvm14:amd64 (1:14.0.0-1ubuntu1.1) ... 2024-03-31T15:02:10.877 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package tcl8.6. 2024-03-31T15:02:10.892 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../33-tcl8.6_8.6.12+dfsg-1build1_amd64.deb ... 2024-03-31T15:02:10.952 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking tcl8.6 (8.6.12+dfsg-1build1) ... 2024-03-31T15:02:11.171 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up dump (0.4b47-3) ... 2024-03-31T15:02:11.221 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package tcl. 2024-03-31T15:02:11.236 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../34-tcl_8.6.11+1build2_amd64.deb ... 2024-03-31T15:02:11.266 INFO:teuthology.orchestra.run.smithi016.stdout:update-alternatives: using /usr/sbin/rmt-dump to provide /usr/sbin/rmt (rmt) in auto mode 2024-03-31T15:02:11.293 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking tcl (8.6.11+1build2) ... 2024-03-31T15:02:11.323 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up libmm-glib0:amd64 (1.20.0-1~ubuntu22.04.3) ... 2024-03-31T15:02:11.424 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up libbluetooth3:amd64 (5.64-0ubuntu1.1) ... 2024-03-31T15:02:11.575 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up tcl (8.6.11+1build2) ... 2024-03-31T15:02:11.666 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package wpasupplicant. 2024-03-31T15:02:11.682 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../35-wpasupplicant_2%3a2.10-6ubuntu2_amd64.deb ... 2024-03-31T15:02:11.701 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up libjson-perl (4.04000-1) ... 2024-03-31T15:02:11.724 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking wpasupplicant (2:2.10-6ubuntu2) ... 2024-03-31T15:02:11.827 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up bison (2:3.8.2+dfsg-1build1) ... 2024-03-31T15:02:11.916 INFO:teuthology.orchestra.run.smithi016.stdout:update-alternatives: using /usr/bin/bison.yacc to provide /usr/bin/yacc (yacc) in auto mode 2024-03-31T15:02:11.986 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up indent (2.2.12-1ubuntu0.22.04.1) ... 2024-03-31T15:02:12.037 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package usb-modeswitch-data. 2024-03-31T15:02:12.052 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../36-usb-modeswitch-data_20191128-4_all.deb ... 2024-03-31T15:02:12.086 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking usb-modeswitch-data (20191128-4) ... 2024-03-31T15:02:12.112 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up usb-modeswitch (2.6.1-3ubuntu2) ... 2024-03-31T15:02:12.322 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up libndp0:amd64 (1.8-0ubuntu3) ... 2024-03-31T15:02:12.440 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up libmbim-glib4:amd64 (1.28.0-1~ubuntu20.04.1) ... 2024-03-31T15:02:12.465 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package usb-modeswitch. 2024-03-31T15:02:12.480 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../37-usb-modeswitch_2.6.1-3ubuntu2_amd64.deb ... 2024-03-31T15:02:12.531 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking usb-modeswitch (2.6.1-3ubuntu2) ... 2024-03-31T15:02:12.550 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up pptp-linux (1.10.0-1build3) ... 2024-03-31T15:02:12.718 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up postgresql-client (14+238) ... 2024-03-31T15:02:12.808 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up postgresql-client-common (238) ... 2024-03-31T15:02:12.843 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up network-manager (1.36.6-0ubuntu2) ... 2024-03-31T15:02:13.371 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up flex (2.6.4-8build2) ... 2024-03-31T15:02:13.466 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:02:13.466 INFO:teuthology.orchestra.run.smithi016.stdout:The following network interfaces were found in /etc/network/interfaces 2024-03-31T15:02:13.466 INFO:teuthology.orchestra.run.smithi016.stdout:which means they are currently configured by ifupdown: 2024-03-31T15:02:13.466 INFO:teuthology.orchestra.run.smithi016.stdout:- ens1f0 2024-03-31T15:02:13.467 INFO:teuthology.orchestra.run.smithi016.stdout:If you want to manage those interfaces with NetworkManager instead 2024-03-31T15:02:13.467 INFO:teuthology.orchestra.run.smithi016.stdout:remove their configuration from /etc/network/interfaces. 2024-03-31T15:02:13.467 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:02:13.472 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up tcl8.6 (8.6.12+dfsg-1build1) ... 2024-03-31T15:02:13.598 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up libpq5:amd64 (14.11-0ubuntu0.22.04.1) ... 2024-03-31T15:02:13.605 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/dbus-org.freedesktop.nm-dispatcher.service -> /lib/systemd/system/NetworkManager-dispatcher.service. 2024-03-31T15:02:13.724 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up liblzo2-2:amd64 (2.10-2build3) ... 2024-03-31T15:02:13.850 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up libaio-dev:amd64 (0.3.112-13build1) ... 2024-03-31T15:02:14.001 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up postgresql-client-14 (14.11-0ubuntu0.22.04.1) ... 2024-03-31T15:02:14.019 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/network-online.target.wants/NetworkManager-wait-online.service -> /lib/systemd/system/NetworkManager-wait-online.service. 2024-03-31T15:02:14.434 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/NetworkManager.service -> /lib/systemd/system/NetworkManager.service. 2024-03-31T15:02:14.951 INFO:teuthology.orchestra.run.smithi089.stdout:update-alternatives: using /usr/share/postgresql/14/man/man1/psql.1.gz to provide /usr/share/man/man1/psql.1.gz (psql.1.gz) in auto mode 2024-03-31T15:02:15.040 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up wpasupplicant (2:2.10-6ubuntu2) ... 2024-03-31T15:02:15.129 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up libmbim-proxy (1.28.0-1~ubuntu20.04.1) ... 2024-03-31T15:02:15.297 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up libfl-dev:amd64 (2.6.4-8build2) ... 2024-03-31T15:02:15.418 INFO:teuthology.orchestra.run.smithi089.stdout:Created symlink /etc/systemd/system/dbus-fi.w1.wpa_supplicant1.service -> /lib/systemd/system/wpa_supplicant.service. 2024-03-31T15:02:15.418 INFO:teuthology.orchestra.run.smithi089.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/wpa_supplicant.service -> /lib/systemd/system/wpa_supplicant.service. 2024-03-31T15:02:15.457 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up network-manager-pptp (1.2.10-1) ... 2024-03-31T15:02:15.608 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up postgresql-common (238) ... 2024-03-31T15:02:16.135 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up uuid-dev:amd64 (2.37.2-4ubuntu3.3) ... 2024-03-31T15:02:16.261 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up libteamdctl0:amd64 (1.31-1build2) ... 2024-03-31T15:02:16.387 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up libssl-dev:amd64 (3.0.2-0ubuntu1.15) ... 2024-03-31T15:02:16.513 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up usb-modeswitch-data (20191128-4) ... 2024-03-31T15:02:16.631 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up libelf-dev:amd64 (0.186-1build1) ... 2024-03-31T15:02:16.644 INFO:teuthology.orchestra.run.smithi016.stdout:Adding user postgres to group ssl-cert 2024-03-31T15:02:16.782 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up libfl2:amd64 (2.6.4-8build2) ... 2024-03-31T15:02:16.902 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:02:16.902 INFO:teuthology.orchestra.run.smithi016.stdout:Creating config file /etc/postgresql-common/createcluster.conf with new version 2024-03-31T15:02:16.954 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up libnm0:amd64 (1.36.6-0ubuntu2) ... 2024-03-31T15:02:17.125 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up ppp (2.4.9-1+1ubuntu3) ... 2024-03-31T15:02:17.161 INFO:teuthology.orchestra.run.smithi016.stdout:Building PostgreSQL dictionaries from installed myspell/hunspell packages... 2024-03-31T15:02:17.161 INFO:teuthology.orchestra.run.smithi016.stdout:Removing obsolete dictionary files: 2024-03-31T15:02:17.797 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up libllvm14:amd64 (1:14.0.0-1ubuntu1.1) ... 2024-03-31T15:02:17.923 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up dump (0.4b47-3) ... 2024-03-31T15:02:17.926 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/postgresql.service -> /lib/systemd/system/postgresql.service. 2024-03-31T15:02:18.393 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up postgresql-14 (14.11-0ubuntu0.22.04.1) ... 2024-03-31T15:02:18.394 INFO:teuthology.orchestra.run.smithi089.stdout:update-alternatives: using /usr/sbin/rmt-dump to provide /usr/sbin/rmt (rmt) in auto mode 2024-03-31T15:02:18.458 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up libmm-glib0:amd64 (1.20.0-1~ubuntu22.04.3) ... 2024-03-31T15:02:18.585 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up libbluetooth3:amd64 (5.64-0ubuntu1.1) ... 2024-03-31T15:02:18.710 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up tcl (8.6.11+1build2) ... 2024-03-31T15:02:18.740 INFO:teuthology.orchestra.run.smithi016.stdout:Creating new PostgreSQL cluster 14/main ... 2024-03-31T15:02:18.801 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/lib/postgresql/14/bin/initdb -D /var/lib/postgresql/14/main --auth-local peer --auth-host scram-sha-256 --no-instructions 2024-03-31T15:02:18.836 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up libjson-perl (4.04000-1) ... 2024-03-31T15:02:18.859 INFO:teuthology.orchestra.run.smithi016.stdout:The files belonging to this database system will be owned by user "postgres". 2024-03-31T15:02:18.859 INFO:teuthology.orchestra.run.smithi016.stdout:This user must also own the server process. 2024-03-31T15:02:18.859 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:02:18.859 INFO:teuthology.orchestra.run.smithi016.stdout:The database cluster will be initialized with locale "en_US". 2024-03-31T15:02:18.859 INFO:teuthology.orchestra.run.smithi016.stdout:The default database encoding has accordingly been set to "LATIN1". 2024-03-31T15:02:18.859 INFO:teuthology.orchestra.run.smithi016.stdout:The default text search configuration will be set to "english". 2024-03-31T15:02:18.859 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:02:18.859 INFO:teuthology.orchestra.run.smithi016.stdout:Data page checksums are disabled. 2024-03-31T15:02:18.859 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:02:18.859 INFO:teuthology.orchestra.run.smithi016.stdout:fixing permissions on existing directory /var/lib/postgresql/14/main ... ok 2024-03-31T15:02:18.860 INFO:teuthology.orchestra.run.smithi016.stdout:creating subdirectories ... ok 2024-03-31T15:02:18.860 INFO:teuthology.orchestra.run.smithi016.stdout:selecting dynamic shared memory implementation ... posix 2024-03-31T15:02:18.885 INFO:teuthology.orchestra.run.smithi016.stdout:selecting default max_connections ... 100 2024-03-31T15:02:18.920 INFO:teuthology.orchestra.run.smithi016.stdout:selecting default shared_buffers ... 128MB 2024-03-31T15:02:18.934 INFO:teuthology.orchestra.run.smithi016.stdout:selecting default time zone ... Etc/UTC 2024-03-31T15:02:18.935 INFO:teuthology.orchestra.run.smithi016.stdout:creating configuration files ... ok 2024-03-31T15:02:18.996 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up bison (2:3.8.2+dfsg-1build1) ... 2024-03-31T15:02:19.077 INFO:teuthology.orchestra.run.smithi089.stdout:update-alternatives: using /usr/bin/bison.yacc to provide /usr/bin/yacc (yacc) in auto mode 2024-03-31T15:02:19.107 INFO:teuthology.orchestra.run.smithi016.stdout:running bootstrap script ... ok 2024-03-31T15:02:19.131 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up indent (2.2.12-1ubuntu0.22.04.1) ... 2024-03-31T15:02:19.257 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up usb-modeswitch (2.6.1-3ubuntu2) ... 2024-03-31T15:02:19.433 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up libndp0:amd64 (1.8-0ubuntu3) ... 2024-03-31T15:02:19.559 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up libmbim-glib4:amd64 (1.28.0-1~ubuntu20.04.1) ... 2024-03-31T15:02:19.710 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up pptp-linux (1.10.0-1build3) ... 2024-03-31T15:02:19.752 INFO:teuthology.orchestra.run.smithi016.stdout:performing post-bootstrap initialization ... ok 2024-03-31T15:02:19.878 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up postgresql-client (14+238) ... 2024-03-31T15:02:20.004 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up network-manager (1.36.6-0ubuntu2) ... 2024-03-31T15:02:20.312 INFO:teuthology.orchestra.run.smithi016.stdout:syncing data to disk ... ok 2024-03-31T15:02:20.587 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-31T15:02:20.587 INFO:teuthology.orchestra.run.smithi089.stdout:The following network interfaces were found in /etc/network/interfaces 2024-03-31T15:02:20.587 INFO:teuthology.orchestra.run.smithi089.stdout:which means they are currently configured by ifupdown: 2024-03-31T15:02:20.587 INFO:teuthology.orchestra.run.smithi089.stdout:- enp3s0f1 2024-03-31T15:02:20.587 INFO:teuthology.orchestra.run.smithi089.stdout:If you want to manage those interfaces with NetworkManager instead 2024-03-31T15:02:20.587 INFO:teuthology.orchestra.run.smithi089.stdout:remove their configuration from /etc/network/interfaces. 2024-03-31T15:02:20.587 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-31T15:02:20.719 INFO:teuthology.orchestra.run.smithi089.stdout:Created symlink /etc/systemd/system/dbus-org.freedesktop.nm-dispatcher.service -> /lib/systemd/system/NetworkManager-dispatcher.service. 2024-03-31T15:02:20.814 INFO:teuthology.orchestra.run.smithi016.stdout:update-alternatives: using /usr/share/postgresql/14/man/man1/postmaster.1.gz to provide /usr/share/man/man1/postmaster.1.gz (postmaster.1.gz) in auto mode 2024-03-31T15:02:21.114 INFO:teuthology.orchestra.run.smithi089.stdout:Created symlink /etc/systemd/system/network-online.target.wants/NetworkManager-wait-online.service -> /lib/systemd/system/NetworkManager-wait-online.service. 2024-03-31T15:02:21.531 INFO:teuthology.orchestra.run.smithi089.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/NetworkManager.service -> /lib/systemd/system/NetworkManager.service. 2024-03-31T15:02:22.232 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up libmbim-proxy (1.28.0-1~ubuntu20.04.1) ... 2024-03-31T15:02:22.383 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up libfl-dev:amd64 (2.6.4-8build2) ... 2024-03-31T15:02:22.532 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up network-manager-pptp (1.2.10-1) ... 2024-03-31T15:02:22.660 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up postgresql-common (238) ... 2024-03-31T15:02:23.613 INFO:teuthology.orchestra.run.smithi089.stdout:Adding user postgres to group ssl-cert 2024-03-31T15:02:23.817 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up libqmi-glib5:amd64 (1.32.0-1ubuntu0.22.04.1) ... 2024-03-31T15:02:23.848 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-31T15:02:23.849 INFO:teuthology.orchestra.run.smithi089.stdout:Creating config file /etc/postgresql-common/createcluster.conf with new version 2024-03-31T15:02:23.968 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up postgresql-contrib (14+238) ... 2024-03-31T15:02:24.103 INFO:teuthology.orchestra.run.smithi089.stdout:Building PostgreSQL dictionaries from installed myspell/hunspell packages... 2024-03-31T15:02:24.104 INFO:teuthology.orchestra.run.smithi089.stdout:Removing obsolete dictionary files: 2024-03-31T15:02:24.119 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up postgresql (14+238) ... 2024-03-31T15:02:24.253 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up libqmi-proxy (1.32.0-1ubuntu0.22.04.1) ... 2024-03-31T15:02:24.421 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up modemmanager (1.20.0-1~ubuntu22.04.3) ... 2024-03-31T15:02:24.715 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/dbus-org.freedesktop.ModemManager1.service -> /lib/systemd/system/ModemManager.service. 2024-03-31T15:02:24.715 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ModemManager.service -> /lib/systemd/system/ModemManager.service. 2024-03-31T15:02:24.834 INFO:teuthology.orchestra.run.smithi089.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/postgresql.service -> /lib/systemd/system/postgresql.service. 2024-03-31T15:02:25.303 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up postgresql-14 (14.11-0ubuntu0.22.04.1) ... 2024-03-31T15:02:25.516 INFO:teuthology.orchestra.run.smithi016.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2024-03-31T15:02:25.608 INFO:teuthology.orchestra.run.smithi016.stdout:Processing triggers for hicolor-icon-theme (0.17-2) ... 2024-03-31T15:02:25.795 INFO:teuthology.orchestra.run.smithi089.stdout:Creating new PostgreSQL cluster 14/main ... 2024-03-31T15:02:25.814 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/lib/postgresql/14/bin/initdb -D /var/lib/postgresql/14/main --auth-local peer --auth-host scram-sha-256 --no-instructions 2024-03-31T15:02:25.864 INFO:teuthology.orchestra.run.smithi089.stdout:The files belonging to this database system will be owned by user "postgres". 2024-03-31T15:02:25.865 INFO:teuthology.orchestra.run.smithi089.stdout:This user must also own the server process. 2024-03-31T15:02:25.865 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-31T15:02:25.865 INFO:teuthology.orchestra.run.smithi089.stdout:The database cluster will be initialized with locale "en_US". 2024-03-31T15:02:25.865 INFO:teuthology.orchestra.run.smithi089.stdout:The default database encoding has accordingly been set to "LATIN1". 2024-03-31T15:02:25.865 INFO:teuthology.orchestra.run.smithi089.stdout:The default text search configuration will be set to "english". 2024-03-31T15:02:25.865 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-31T15:02:25.865 INFO:teuthology.orchestra.run.smithi089.stdout:Data page checksums are disabled. 2024-03-31T15:02:25.865 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-31T15:02:25.865 INFO:teuthology.orchestra.run.smithi089.stdout:fixing permissions on existing directory /var/lib/postgresql/14/main ... ok 2024-03-31T15:02:25.866 INFO:teuthology.orchestra.run.smithi089.stdout:creating subdirectories ... ok 2024-03-31T15:02:25.866 INFO:teuthology.orchestra.run.smithi089.stdout:selecting dynamic shared memory implementation ... posix 2024-03-31T15:02:25.889 INFO:teuthology.orchestra.run.smithi089.stdout:selecting default max_connections ... 100 2024-03-31T15:02:25.923 INFO:teuthology.orchestra.run.smithi089.stdout:selecting default shared_buffers ... 128MB 2024-03-31T15:02:25.936 INFO:teuthology.orchestra.run.smithi089.stdout:selecting default time zone ... Etc/UTC 2024-03-31T15:02:25.937 INFO:teuthology.orchestra.run.smithi089.stdout:creating configuration files ... ok 2024-03-31T15:02:25.968 INFO:teuthology.orchestra.run.smithi016.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-03-31T15:02:26.099 INFO:teuthology.orchestra.run.smithi089.stdout:running bootstrap script ... ok 2024-03-31T15:02:26.128 INFO:teuthology.orchestra.run.smithi016.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-03-31T15:02:26.707 INFO:teuthology.orchestra.run.smithi089.stdout:performing post-bootstrap initialization ... ok 2024-03-31T15:02:27.304 INFO:teuthology.orchestra.run.smithi089.stdout:syncing data to disk ... ok 2024-03-31T15:02:27.581 INFO:teuthology.orchestra.run.smithi016.stdout:Processing triggers for dbus (1.12.20-2ubuntu4.1) ... 2024-03-31T15:02:27.681 INFO:teuthology.orchestra.run.smithi016.stdout:Processing triggers for install-info (6.8-4build1) ... 2024-03-31T15:02:27.825 INFO:teuthology.orchestra.run.smithi089.stdout:update-alternatives: using /usr/share/postgresql/14/man/man1/postmaster.1.gz to provide /usr/share/man/man1/postmaster.1.gz (postmaster.1.gz) in auto mode 2024-03-31T15:02:28.269 INFO:teuthology.orchestra.run.smithi016.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-03-31T15:02:28.278 DEBUG:teuthology.parallel:result is None 2024-03-31T15:02:30.794 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up libqmi-glib5:amd64 (1.32.0-1ubuntu0.22.04.1) ... 2024-03-31T15:02:30.920 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up postgresql-contrib (14+238) ... 2024-03-31T15:02:31.071 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up postgresql (14+238) ... 2024-03-31T15:02:31.172 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up libqmi-proxy (1.32.0-1ubuntu0.22.04.1) ... 2024-03-31T15:02:31.298 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up modemmanager (1.20.0-1~ubuntu22.04.3) ... 2024-03-31T15:02:31.559 INFO:teuthology.orchestra.run.smithi089.stdout:Created symlink /etc/systemd/system/dbus-org.freedesktop.ModemManager1.service -> /lib/systemd/system/ModemManager.service. 2024-03-31T15:02:31.559 INFO:teuthology.orchestra.run.smithi089.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ModemManager.service -> /lib/systemd/system/ModemManager.service. 2024-03-31T15:02:32.317 INFO:teuthology.orchestra.run.smithi089.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2024-03-31T15:02:32.410 INFO:teuthology.orchestra.run.smithi089.stdout:Processing triggers for hicolor-icon-theme (0.17-2) ... 2024-03-31T15:02:32.737 INFO:teuthology.orchestra.run.smithi089.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-03-31T15:02:32.880 INFO:teuthology.orchestra.run.smithi089.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-03-31T15:02:34.292 INFO:teuthology.orchestra.run.smithi089.stdout:Processing triggers for dbus (1.12.20-2ubuntu4.1) ... 2024-03-31T15:02:34.375 INFO:teuthology.orchestra.run.smithi089.stdout:Processing triggers for install-info (6.8-4build1) ... 2024-03-31T15:02:34.930 INFO:teuthology.orchestra.run.smithi089.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-03-31T15:02:34.936 DEBUG:teuthology.parallel:result is None 2024-03-31T15:02:34.937 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=d2271579d5e735cfd6d436b8349c004f06527cb0 2024-03-31T15:02:35.213 DEBUG:teuthology.orchestra.run.smithi016:> dpkg-query -W -f '${Version}' ceph 2024-03-31T15:02:35.237 INFO:teuthology.orchestra.run.smithi016.stdout:19.0.0-2574-gd2271579-1jammy 2024-03-31T15:02:35.238 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2574-gd2271579-1jammy 2024-03-31T15:02:35.238 INFO:teuthology.task.install:The correct ceph version 19.0.0-2574-gd2271579-1jammy is installed. 2024-03-31T15:02:35.240 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=d2271579d5e735cfd6d436b8349c004f06527cb0 2024-03-31T15:02:35.381 DEBUG:teuthology.orchestra.run.smithi089:> dpkg-query -W -f '${Version}' ceph 2024-03-31T15:02:35.403 INFO:teuthology.orchestra.run.smithi089.stdout:19.0.0-2574-gd2271579-1jammy 2024-03-31T15:02:35.403 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2574-gd2271579-1jammy 2024-03-31T15:02:35.404 INFO:teuthology.task.install:The correct ceph version 19.0.0-2574-gd2271579-1jammy is installed. 2024-03-31T15:02:35.405 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-31T15:02:35.406 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-03-31T15:02:35.406 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-31T15:02:35.429 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-03-31T15:02:35.429 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-31T15:02:35.465 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-31T15:02:35.465 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-03-31T15:02:35.466 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd of=/usr/bin/daemon-helper 2024-03-31T15:02:35.491 DEBUG:teuthology.orchestra.run.smithi016:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-31T15:02:35.556 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-03-31T15:02:35.556 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd of=/usr/bin/daemon-helper 2024-03-31T15:02:35.575 DEBUG:teuthology.orchestra.run.smithi089:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-31T15:02:35.635 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-31T15:02:35.635 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-03-31T15:02:35.635 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-31T15:02:35.657 DEBUG:teuthology.orchestra.run.smithi016:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-31T15:02:35.720 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-03-31T15:02:35.720 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-31T15:02:35.739 DEBUG:teuthology.orchestra.run.smithi089:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-31T15:02:35.799 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-31T15:02:35.799 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-03-31T15:02:35.799 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd of=/usr/bin/stdin-killer 2024-03-31T15:02:35.820 DEBUG:teuthology.orchestra.run.smithi016:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-31T15:02:35.883 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-03-31T15:02:35.883 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd of=/usr/bin/stdin-killer 2024-03-31T15:02:35.902 DEBUG:teuthology.orchestra.run.smithi089:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-31T15:02:35.963 INFO:teuthology.run_tasks:Running task ceph... 2024-03-31T15:02:36.044 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-03-31T15:02:36.045 DEBUG:teuthology.orchestra.run.smithi016:> sudo chmod 777 /var/log/ceph 2024-03-31T15:02:36.048 DEBUG:teuthology.orchestra.run.smithi089:> sudo chmod 777 /var/log/ceph 2024-03-31T15:02:36.067 INFO:tasks.ceph:Disabling ceph logrotate... 2024-03-31T15:02:36.067 DEBUG:teuthology.orchestra.run.smithi016:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-31T15:02:36.110 DEBUG:teuthology.orchestra.run.smithi089:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-31T15:02:36.128 INFO:tasks.ceph:Creating extra log directories... 2024-03-31T15:02:36.128 DEBUG:teuthology.orchestra.run.smithi016:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-31T15:02:36.174 DEBUG:teuthology.orchestra.run.smithi089:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-31T15:02:36.199 INFO:tasks.ceph:Setting up log rotation with {'ceph-mds': '10G', 'ceph-osd': '10G'} 2024-03-31T15:02:36.200 INFO:tasks.ceph:writing logrotate stanza for ceph-mds 2024-03-31T15:02:36.200 INFO:tasks.ceph:writing logrotate stanza for ceph-osd 2024-03-31T15:02:36.200 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-03-31T15:02:36.200 DEBUG:teuthology.orchestra.run.smithi016:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2024-03-31T15:02:36.243 DEBUG:teuthology.orchestra.run.smithi016:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2024-03-31T15:02:36.337 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-03-31T15:02:36.337 DEBUG:teuthology.orchestra.run.smithi089:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2024-03-31T15:02:36.345 DEBUG:teuthology.orchestra.run.smithi089:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2024-03-31T15:02:36.440 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-03-31T15:02:36.440 INFO:tasks.ceph:config {'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability', 'inode wrongly marked free', 'bad backtrace on inode', 'inode table repaired for inode', 'Scrub error on inode', 'Scrub error on dir', 'Metadata damage detected'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-03-31T15:02:36.440 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/vshankar-2024-03-30_23:12:08-fs-wip-vshankar-testing-20240330.170739-testing-default-smithi/7631609', 'branch': 'wip-vshankar-testing-20240330.170739', 'description': 'fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v1 tasks/forward-scrub}', 'email': 'vshankar@redhat.com', 'first_in_suite': False, 'job_id': '7631609', 'kernel': {'flavor': 'default', 'kdb': True, 'sha1': '1df0c7e8a2354c42d738e5454ba83ea7332deac2'}, 'ktype': 'distro', 'last_in_suite': False, 'log-rotate': {'ceph-mds': '10G', 'ceph-osd': '10G'}, 'machine_type': 'smithi', 'mgrmodules': {'sequential': [{'print': 'Enabling mgr modules'}]}, 'name': 'vshankar-2024-03-30_23:12:08-fs-wip-vshankar-testing-20240330.170739-testing-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 20}}, {'machine': {'disk': 200}}], 'os_type': 'ubuntu', 'os_version': '22.04', 'overrides': {'admin_socket': {'branch': 'wip-vshankar-testing-20240330.170739'}, 'ceph': {'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability', 'inode wrongly marked free', 'bad backtrace on inode', 'inode table repaired for inode', 'Scrub error on inode', 'Scrub error on dir', 'Metadata damage detected'], 'sha1': 'd2271579d5e735cfd6d436b8349c004f06527cb0'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {'osd default pool size': 2}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'flavor': 'default', 'sha1': 'd2271579d5e735cfd6d436b8349c004f06527cb0'}}, 'subvolume_version': 1, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-vshankar-testing-20240330.170739', 'sha1': 'd2271579d5e735cfd6d436b8349c004f06527cb0'}}, 'owner': 'scheduled_vshankar@teuthology', 'priority': 100, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2'], ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3']], 'seed': 6891, 'sha1': 'd2271579d5e735cfd6d436b8349c004f06527cb0', 'sleep_before_teardown': 0, 'subset': '932/1024', 'suite': 'fs', 'suite_branch': 'wip-vshankar-testing-20240330.170739', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_d2271579d5e735cfd6d436b8349c004f06527cb0/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': 'd2271579d5e735cfd6d436b8349c004f06527cb0', 'syslog': {'ignorelist': ['WARNING*.*check_session_state', 'WARNING*.*__ceph_remove_cap']}, 'targets': {'smithi016.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGrNGRw477WUQu17A6o65D4kDQ6WSLg/SUYnTZXG9vNI9kj8M4+jdSsqhTA1AVsIdEZoduIq19dWV/GhMsuzXqM=', 'smithi089.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBANsxlXyxrksTMZg1EXGMkYGT/ASySvp6CS9Cas0w9gzWiSepEnTbAeyQC0DgQosrPfrSbAwG8nwNKSj4IuxKLE='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'flavor': 'default', 'kdb': True, 'sha1': '1df0c7e8a2354c42d738e5454ba83ea7332deac2'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': 'd2271579d5e735cfd6d436b8349c004f06527cb0'}}, {'ceph': {'create_rbd_pool': False, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability', 'inode wrongly marked free', 'bad backtrace on inode', 'inode table repaired for inode', 'Scrub error on inode', 'Scrub error on dir', 'Metadata damage detected'], 'sha1': 'd2271579d5e735cfd6d436b8349c004f06527cb0', 'cluster': 'ceph'}}, {'sequential': ['mgrmodules']}, {'ceph-fuse': None}, {'cephfs_test_runner': {'modules': ['tasks.cephfs.test_forward_scrub']}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': ['local function is_kupstream() return false end', 'local function is_kdistro() return false end', 'local function is_fuse() return true end', "local function syntax_version() return '' end"]}, 'teuthology_branch': 'main', 'teuthology_sha1': 'e691533f9cbb33d85b2187bba20d7102f098636d', 'timestamp': '2024-03-30_23:12:08', 'tube': 'smithi', 'user': 'vshankar', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-03-31T15:02:36.441 DEBUG:teuthology.orchestra.run.smithi016:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-31T15:02:36.444 DEBUG:teuthology.orchestra.run.smithi089:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-31T15:02:36.453 DEBUG:teuthology.orchestra.run.smithi016:> sudo install -d -m0777 -- /var/run/ceph 2024-03-31T15:02:36.494 DEBUG:teuthology.orchestra.run.smithi089:> sudo install -d -m0777 -- /var/run/ceph 2024-03-31T15:02:36.515 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-03-31T15:02:36.515 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/scratch_devs of=/dev/stdout 2024-03-31T15:02:36.564 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-31T15:02:36.564 DEBUG:teuthology.orchestra.run.smithi016:> stat /dev/vg_nvme/lv_1 2024-03-31T15:02:36.617 INFO:teuthology.orchestra.run.smithi016.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-31T15:02:36.617 INFO:teuthology.orchestra.run.smithi016.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-31T15:02:36.617 INFO:teuthology.orchestra.run.smithi016.stdout:Device: 5h/5d Inode: 900 Links: 1 2024-03-31T15:02:36.617 INFO:teuthology.orchestra.run.smithi016.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-31T15:02:36.617 INFO:teuthology.orchestra.run.smithi016.stdout:Access: 2024-03-31 14:56:08.649013457 +0000 2024-03-31T15:02:36.618 INFO:teuthology.orchestra.run.smithi016.stdout:Modify: 2024-03-31 14:56:08.645013493 +0000 2024-03-31T15:02:36.618 INFO:teuthology.orchestra.run.smithi016.stdout:Change: 2024-03-31 14:56:08.645013493 +0000 2024-03-31T15:02:36.618 INFO:teuthology.orchestra.run.smithi016.stdout: Birth: 2024-03-31 14:56:08.645013493 +0000 2024-03-31T15:02:36.619 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-31T15:02:36.679 INFO:teuthology.orchestra.run.smithi016.stderr:1+0 records in 2024-03-31T15:02:36.679 INFO:teuthology.orchestra.run.smithi016.stderr:1+0 records out 2024-03-31T15:02:36.679 INFO:teuthology.orchestra.run.smithi016.stderr:512 bytes copied, 0.000277707 s, 1.8 MB/s 2024-03-31T15:02:36.681 DEBUG:teuthology.orchestra.run.smithi016:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-31T15:02:36.734 DEBUG:teuthology.orchestra.run.smithi016:> stat /dev/vg_nvme/lv_2 2024-03-31T15:02:36.784 INFO:teuthology.orchestra.run.smithi016.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-31T15:02:36.784 INFO:teuthology.orchestra.run.smithi016.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-31T15:02:36.784 INFO:teuthology.orchestra.run.smithi016.stdout:Device: 5h/5d Inode: 915 Links: 1 2024-03-31T15:02:36.784 INFO:teuthology.orchestra.run.smithi016.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-31T15:02:36.785 INFO:teuthology.orchestra.run.smithi016.stdout:Access: 2024-03-31 14:56:09.149008991 +0000 2024-03-31T15:02:36.785 INFO:teuthology.orchestra.run.smithi016.stdout:Modify: 2024-03-31 14:56:09.145009026 +0000 2024-03-31T15:02:36.785 INFO:teuthology.orchestra.run.smithi016.stdout:Change: 2024-03-31 14:56:09.145009026 +0000 2024-03-31T15:02:36.785 INFO:teuthology.orchestra.run.smithi016.stdout: Birth: 2024-03-31 14:56:09.145009026 +0000 2024-03-31T15:02:36.785 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-31T15:02:36.846 INFO:teuthology.orchestra.run.smithi016.stderr:1+0 records in 2024-03-31T15:02:36.846 INFO:teuthology.orchestra.run.smithi016.stderr:1+0 records out 2024-03-31T15:02:36.846 INFO:teuthology.orchestra.run.smithi016.stderr:512 bytes copied, 0.000264306 s, 1.9 MB/s 2024-03-31T15:02:36.848 DEBUG:teuthology.orchestra.run.smithi016:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-31T15:02:36.901 DEBUG:teuthology.orchestra.run.smithi016:> stat /dev/vg_nvme/lv_3 2024-03-31T15:02:36.956 INFO:teuthology.orchestra.run.smithi016.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-31T15:02:36.956 INFO:teuthology.orchestra.run.smithi016.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-31T15:02:36.956 INFO:teuthology.orchestra.run.smithi016.stdout:Device: 5h/5d Inode: 935 Links: 1 2024-03-31T15:02:36.956 INFO:teuthology.orchestra.run.smithi016.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-31T15:02:36.957 INFO:teuthology.orchestra.run.smithi016.stdout:Access: 2024-03-31 14:56:09.929002024 +0000 2024-03-31T15:02:36.957 INFO:teuthology.orchestra.run.smithi016.stdout:Modify: 2024-03-31 14:56:09.665004382 +0000 2024-03-31T15:02:36.957 INFO:teuthology.orchestra.run.smithi016.stdout:Change: 2024-03-31 14:56:09.665004382 +0000 2024-03-31T15:02:36.957 INFO:teuthology.orchestra.run.smithi016.stdout: Birth: 2024-03-31 14:56:09.665004382 +0000 2024-03-31T15:02:36.957 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-31T15:02:37.018 INFO:teuthology.orchestra.run.smithi016.stderr:1+0 records in 2024-03-31T15:02:37.018 INFO:teuthology.orchestra.run.smithi016.stderr:1+0 records out 2024-03-31T15:02:37.018 INFO:teuthology.orchestra.run.smithi016.stderr:512 bytes copied, 0.000253578 s, 2.0 MB/s 2024-03-31T15:02:37.020 DEBUG:teuthology.orchestra.run.smithi016:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-31T15:02:37.078 DEBUG:teuthology.orchestra.run.smithi016:> stat /dev/vg_nvme/lv_4 2024-03-31T15:02:37.128 INFO:teuthology.orchestra.run.smithi016.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-31T15:02:37.128 INFO:teuthology.orchestra.run.smithi016.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-31T15:02:37.128 INFO:teuthology.orchestra.run.smithi016.stdout:Device: 5h/5d Inode: 952 Links: 1 2024-03-31T15:02:37.128 INFO:teuthology.orchestra.run.smithi016.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-31T15:02:37.128 INFO:teuthology.orchestra.run.smithi016.stdout:Access: 2024-03-31 14:56:10.448997380 +0000 2024-03-31T15:02:37.128 INFO:teuthology.orchestra.run.smithi016.stdout:Modify: 2024-03-31 14:56:10.180999773 +0000 2024-03-31T15:02:37.128 INFO:teuthology.orchestra.run.smithi016.stdout:Change: 2024-03-31 14:56:10.180999773 +0000 2024-03-31T15:02:37.128 INFO:teuthology.orchestra.run.smithi016.stdout: Birth: 2024-03-31 14:56:10.180999773 +0000 2024-03-31T15:02:37.129 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-31T15:02:37.190 INFO:teuthology.orchestra.run.smithi016.stderr:1+0 records in 2024-03-31T15:02:37.190 INFO:teuthology.orchestra.run.smithi016.stderr:1+0 records out 2024-03-31T15:02:37.190 INFO:teuthology.orchestra.run.smithi016.stderr:512 bytes copied, 0.000257701 s, 2.0 MB/s 2024-03-31T15:02:37.192 DEBUG:teuthology.orchestra.run.smithi016:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-31T15:02:37.249 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-31T15:02:37.250 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-03-31T15:02:37.250 DEBUG:teuthology.orchestra.run.smithi089:> dd if=/scratch_devs of=/dev/stdout 2024-03-31T15:02:37.259 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-31T15:02:37.259 DEBUG:teuthology.orchestra.run.smithi089:> stat /dev/vg_nvme/lv_1 2024-03-31T15:02:37.309 INFO:teuthology.orchestra.run.smithi089.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-31T15:02:37.309 INFO:teuthology.orchestra.run.smithi089.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-31T15:02:37.309 INFO:teuthology.orchestra.run.smithi089.stdout:Device: 5h/5d Inode: 891 Links: 1 2024-03-31T15:02:37.309 INFO:teuthology.orchestra.run.smithi089.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-31T15:02:37.309 INFO:teuthology.orchestra.run.smithi089.stdout:Access: 2024-03-31 14:56:07.211436290 +0000 2024-03-31T15:02:37.309 INFO:teuthology.orchestra.run.smithi089.stdout:Modify: 2024-03-31 14:56:06.907439004 +0000 2024-03-31T15:02:37.309 INFO:teuthology.orchestra.run.smithi089.stdout:Change: 2024-03-31 14:56:06.907439004 +0000 2024-03-31T15:02:37.309 INFO:teuthology.orchestra.run.smithi089.stdout: Birth: 2024-03-31 14:56:06.907439004 +0000 2024-03-31T15:02:37.309 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-31T15:02:37.371 INFO:teuthology.orchestra.run.smithi089.stderr:1+0 records in 2024-03-31T15:02:37.371 INFO:teuthology.orchestra.run.smithi089.stderr:1+0 records out 2024-03-31T15:02:37.371 INFO:teuthology.orchestra.run.smithi089.stderr:512 bytes copied, 0.00040181 s, 1.3 MB/s 2024-03-31T15:02:37.373 DEBUG:teuthology.orchestra.run.smithi089:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-31T15:02:37.431 DEBUG:teuthology.orchestra.run.smithi089:> stat /dev/vg_nvme/lv_2 2024-03-31T15:02:37.481 INFO:teuthology.orchestra.run.smithi089.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-31T15:02:37.481 INFO:teuthology.orchestra.run.smithi089.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-31T15:02:37.481 INFO:teuthology.orchestra.run.smithi089.stdout:Device: 5h/5d Inode: 907 Links: 1 2024-03-31T15:02:37.481 INFO:teuthology.orchestra.run.smithi089.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-31T15:02:37.481 INFO:teuthology.orchestra.run.smithi089.stdout:Access: 2024-03-31 14:56:07.731431644 +0000 2024-03-31T15:02:37.481 INFO:teuthology.orchestra.run.smithi089.stdout:Modify: 2024-03-31 14:56:07.463434039 +0000 2024-03-31T15:02:37.481 INFO:teuthology.orchestra.run.smithi089.stdout:Change: 2024-03-31 14:56:07.463434039 +0000 2024-03-31T15:02:37.481 INFO:teuthology.orchestra.run.smithi089.stdout: Birth: 2024-03-31 14:56:07.463434039 +0000 2024-03-31T15:02:37.482 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-31T15:02:37.543 INFO:teuthology.orchestra.run.smithi089.stderr:1+0 records in 2024-03-31T15:02:37.543 INFO:teuthology.orchestra.run.smithi089.stderr:1+0 records out 2024-03-31T15:02:37.543 INFO:teuthology.orchestra.run.smithi089.stderr:512 bytes copied, 0.000400294 s, 1.3 MB/s 2024-03-31T15:02:37.545 DEBUG:teuthology.orchestra.run.smithi089:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-31T15:02:37.599 DEBUG:teuthology.orchestra.run.smithi089:> stat /dev/vg_nvme/lv_3 2024-03-31T15:02:37.649 INFO:teuthology.orchestra.run.smithi089.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-31T15:02:37.649 INFO:teuthology.orchestra.run.smithi089.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-31T15:02:37.649 INFO:teuthology.orchestra.run.smithi089.stdout:Device: 5h/5d Inode: 926 Links: 1 2024-03-31T15:02:37.649 INFO:teuthology.orchestra.run.smithi089.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-31T15:02:37.649 INFO:teuthology.orchestra.run.smithi089.stdout:Access: 2024-03-31 14:56:08.295426607 +0000 2024-03-31T15:02:37.649 INFO:teuthology.orchestra.run.smithi089.stdout:Modify: 2024-03-31 14:56:08.023429036 +0000 2024-03-31T15:02:37.649 INFO:teuthology.orchestra.run.smithi089.stdout:Change: 2024-03-31 14:56:08.023429036 +0000 2024-03-31T15:02:37.649 INFO:teuthology.orchestra.run.smithi089.stdout: Birth: 2024-03-31 14:56:08.023429036 +0000 2024-03-31T15:02:37.650 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-31T15:02:37.710 INFO:teuthology.orchestra.run.smithi089.stderr:1+0 records in 2024-03-31T15:02:37.710 INFO:teuthology.orchestra.run.smithi089.stderr:1+0 records out 2024-03-31T15:02:37.710 INFO:teuthology.orchestra.run.smithi089.stderr:512 bytes copied, 0.000652157 s, 785 kB/s 2024-03-31T15:02:37.712 DEBUG:teuthology.orchestra.run.smithi089:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-31T15:02:37.762 DEBUG:teuthology.orchestra.run.smithi089:> stat /dev/vg_nvme/lv_4 2024-03-31T15:02:37.817 INFO:teuthology.orchestra.run.smithi089.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-31T15:02:37.817 INFO:teuthology.orchestra.run.smithi089.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-31T15:02:37.817 INFO:teuthology.orchestra.run.smithi089.stdout:Device: 5h/5d Inode: 941 Links: 1 2024-03-31T15:02:37.817 INFO:teuthology.orchestra.run.smithi089.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-31T15:02:37.817 INFO:teuthology.orchestra.run.smithi089.stdout:Access: 2024-03-31 14:56:08.815421962 +0000 2024-03-31T15:02:37.817 INFO:teuthology.orchestra.run.smithi089.stdout:Modify: 2024-03-31 14:56:08.551424320 +0000 2024-03-31T15:02:37.817 INFO:teuthology.orchestra.run.smithi089.stdout:Change: 2024-03-31 14:56:08.551424320 +0000 2024-03-31T15:02:37.817 INFO:teuthology.orchestra.run.smithi089.stdout: Birth: 2024-03-31 14:56:08.551424320 +0000 2024-03-31T15:02:37.817 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-31T15:02:37.878 INFO:teuthology.orchestra.run.smithi089.stderr:1+0 records in 2024-03-31T15:02:37.878 INFO:teuthology.orchestra.run.smithi089.stderr:1+0 records out 2024-03-31T15:02:37.878 INFO:teuthology.orchestra.run.smithi089.stderr:512 bytes copied, 0.000368221 s, 1.4 MB/s 2024-03-31T15:02:37.879 DEBUG:teuthology.orchestra.run.smithi089:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-31T15:02:37.930 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-31T15:02:37.930 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi016.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi089.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-31T15:02:37.931 INFO:tasks.ceph:Generating config... 2024-03-31T15:02:37.932 INFO:tasks.ceph:[client] client mount timeout = 600 2024-03-31T15:02:37.932 INFO:tasks.ceph:[client] debug client = 20 2024-03-31T15:02:37.932 INFO:tasks.ceph:[client] debug ms = 1 2024-03-31T15:02:37.932 INFO:tasks.ceph:[client] pid file = 2024-03-31T15:02:37.932 INFO:tasks.ceph:[client] rados mon op timeout = 900 2024-03-31T15:02:37.932 INFO:tasks.ceph:[client] rados osd op timeout = 900 2024-03-31T15:02:37.932 INFO:tasks.ceph:[mds] debug mds = 20 2024-03-31T15:02:37.932 INFO:tasks.ceph:[mds] debug mds balancer = 20 2024-03-31T15:02:37.932 INFO:tasks.ceph:[mds] debug ms = 1 2024-03-31T15:02:37.932 INFO:tasks.ceph:[mds] mds debug frag = True 2024-03-31T15:02:37.932 INFO:tasks.ceph:[mds] mds debug scatterstat = True 2024-03-31T15:02:37.933 INFO:tasks.ceph:[mds] mds op complaint time = 180 2024-03-31T15:02:37.933 INFO:tasks.ceph:[mds] mds verify scatter = True 2024-03-31T15:02:37.933 INFO:tasks.ceph:[mds] osd op complaint time = 180 2024-03-31T15:02:37.933 INFO:tasks.ceph:[mds] rados mon op timeout = 900 2024-03-31T15:02:37.933 INFO:tasks.ceph:[mds] rados osd op timeout = 900 2024-03-31T15:02:37.933 INFO:tasks.ceph:[mgr] debug client = 20 2024-03-31T15:02:37.933 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-03-31T15:02:37.933 INFO:tasks.ceph:[mgr] debug ms = 1 2024-03-31T15:02:37.933 INFO:tasks.ceph:[mon] debug mon = 20 2024-03-31T15:02:37.933 INFO:tasks.ceph:[mon] debug ms = 1 2024-03-31T15:02:37.933 INFO:tasks.ceph:[mon] debug paxos = 20 2024-03-31T15:02:37.933 INFO:tasks.ceph:[mon] mon down mkfs grace = 300 2024-03-31T15:02:37.933 INFO:tasks.ceph:[mon] mon op complaint time = 120 2024-03-31T15:02:37.933 INFO:tasks.ceph:[osd] bdev async discard = True 2024-03-31T15:02:37.933 INFO:tasks.ceph:[osd] bdev enable discard = True 2024-03-31T15:02:37.934 INFO:tasks.ceph:[osd] bluestore allocator = bitmap 2024-03-31T15:02:37.934 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-03-31T15:02:37.934 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-03-31T15:02:37.934 INFO:tasks.ceph:[osd] debug bluefs = 1/20 2024-03-31T15:02:37.934 INFO:tasks.ceph:[osd] debug bluestore = 1/20 2024-03-31T15:02:37.934 INFO:tasks.ceph:[osd] debug ms = 1 2024-03-31T15:02:37.934 INFO:tasks.ceph:[osd] debug osd = 20 2024-03-31T15:02:37.934 INFO:tasks.ceph:[osd] debug rocksdb = 4/10 2024-03-31T15:02:37.934 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-03-31T15:02:37.934 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-03-31T15:02:37.934 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-03-31T15:02:37.934 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-03-31T15:02:37.934 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-03-31T15:02:37.934 INFO:tasks.ceph:[osd] osd op complaint time = 180 2024-03-31T15:02:37.935 INFO:tasks.ceph:Setting up mon.a... 2024-03-31T15:02:37.935 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-03-31T15:02:37.980 INFO:teuthology.orchestra.run.smithi016.stdout:creating /etc/ceph/ceph.keyring 2024-03-31T15:02:37.993 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-03-31T15:02:38.042 DEBUG:teuthology.orchestra.run.smithi016:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-31T15:02:38.062 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.16'), ('mon.b', '172.21.15.89'), ('mon.c', '[v2:172.21.15.89:3301,v1:172.21.15.89:6790]')] 2024-03-31T15:02:38.062 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.16,172.21.15.89,[v2:172.21.15.89:3301,v1:172.21.15.89:6790]'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false', 'debug client': 20}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon down mkfs grace': 300, 'mon op complaint time': 120}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mon.a': {}, 'mon.b': {}, 'mon.c': {}, 'mds.a': {}, 'mds.c': {}, 'mds.b': {}, 'mds.d': {}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}} 2024-03-31T15:02:38.063 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-03-31T15:02:38.063 DEBUG:teuthology.orchestra.run.smithi016:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-03-31T15:02:38.111 DEBUG:teuthology.orchestra.run.smithi016:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.16 --add b 172.21.15.89 --addv c '[v2:172.21.15.89:3301,v1:172.21.15.89:6790]' --print /home/ubuntu/cephtest/ceph.monmap 2024-03-31T15:02:38.202 INFO:teuthology.orchestra.run.smithi016.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-03-31T15:02:38.202 INFO:teuthology.orchestra.run.smithi016.stdout:monmaptool: generated fsid 286eb5da-fb47-4550-b290-d9519c13b9a5 2024-03-31T15:02:38.202 INFO:teuthology.orchestra.run.smithi016.stdout:setting min_mon_release = quincy 2024-03-31T15:02:38.202 INFO:teuthology.orchestra.run.smithi016.stdout:epoch 0 2024-03-31T15:02:38.202 INFO:teuthology.orchestra.run.smithi016.stdout:fsid 286eb5da-fb47-4550-b290-d9519c13b9a5 2024-03-31T15:02:38.202 INFO:teuthology.orchestra.run.smithi016.stdout:last_changed 2024-03-31T15:02:38.212032+0000 2024-03-31T15:02:38.202 INFO:teuthology.orchestra.run.smithi016.stdout:created 2024-03-31T15:02:38.212032+0000 2024-03-31T15:02:38.203 INFO:teuthology.orchestra.run.smithi016.stdout:min_mon_release 17 (quincy) 2024-03-31T15:02:38.203 INFO:teuthology.orchestra.run.smithi016.stdout:election_strategy: 1 2024-03-31T15:02:38.203 INFO:teuthology.orchestra.run.smithi016.stdout:0: [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] mon.a 2024-03-31T15:02:38.203 INFO:teuthology.orchestra.run.smithi016.stdout:1: [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] mon.b 2024-03-31T15:02:38.203 INFO:teuthology.orchestra.run.smithi016.stdout:2: [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] mon.c 2024-03-31T15:02:38.203 INFO:teuthology.orchestra.run.smithi016.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-03-31T15:02:38.205 DEBUG:teuthology.orchestra.run.smithi016:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-03-31T15:02:38.213 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 286eb5da-fb47-4550-b290-d9519c13b9a5... 2024-03-31T15:02:38.214 DEBUG:teuthology.orchestra.run.smithi016:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-31T15:02:38.257 DEBUG:teuthology.orchestra.run.smithi089:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-31T15:02:38.298 INFO:teuthology.orchestra.run.smithi016.stdout:[global] 2024-03-31T15:02:38.298 INFO:teuthology.orchestra.run.smithi016.stdout: chdir = "" 2024-03-31T15:02:38.298 INFO:teuthology.orchestra.run.smithi016.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-31T15:02:38.298 INFO:teuthology.orchestra.run.smithi016.stdout: auth supported = cephx 2024-03-31T15:02:38.298 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:02:38.298 INFO:teuthology.orchestra.run.smithi016.stdout: filestore xattr use omap = true 2024-03-31T15:02:38.298 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:02:38.298 INFO:teuthology.orchestra.run.smithi016.stdout: mon clock drift allowed = 1.000 2024-03-31T15:02:38.298 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:02:38.298 INFO:teuthology.orchestra.run.smithi016.stdout: osd crush chooseleaf type = 0 2024-03-31T15:02:38.298 INFO:teuthology.orchestra.run.smithi016.stdout: auth debug = true 2024-03-31T15:02:38.299 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:02:38.299 INFO:teuthology.orchestra.run.smithi016.stdout: ms die on old message = true 2024-03-31T15:02:38.299 INFO:teuthology.orchestra.run.smithi016.stdout: ms die on bug = true 2024-03-31T15:02:38.299 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:02:38.299 INFO:teuthology.orchestra.run.smithi016.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-31T15:02:38.299 INFO:teuthology.orchestra.run.smithi016.stdout: mon pg warn max object skew = 0 2024-03-31T15:02:38.299 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:02:38.299 INFO:teuthology.orchestra.run.smithi016.stdout: # disable pg_autoscaler by default for new pools 2024-03-31T15:02:38.299 INFO:teuthology.orchestra.run.smithi016.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-31T15:02:38.299 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:02:38.300 INFO:teuthology.orchestra.run.smithi016.stdout: osd pool default size = 2 2024-03-31T15:02:38.300 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:02:38.300 INFO:teuthology.orchestra.run.smithi016.stdout: mon osd allow primary affinity = true 2024-03-31T15:02:38.300 INFO:teuthology.orchestra.run.smithi016.stdout: mon osd allow pg remap = true 2024-03-31T15:02:38.300 INFO:teuthology.orchestra.run.smithi016.stdout: mon warn on legacy crush tunables = false 2024-03-31T15:02:38.300 INFO:teuthology.orchestra.run.smithi016.stdout: mon warn on crush straw calc version zero = false 2024-03-31T15:02:38.300 INFO:teuthology.orchestra.run.smithi016.stdout: mon warn on no sortbitwise = false 2024-03-31T15:02:38.300 INFO:teuthology.orchestra.run.smithi016.stdout: mon warn on osd down out interval zero = false 2024-03-31T15:02:38.300 INFO:teuthology.orchestra.run.smithi016.stdout: mon warn on too few osds = false 2024-03-31T15:02:38.300 INFO:teuthology.orchestra.run.smithi016.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-31T15:02:38.300 INFO:teuthology.orchestra.run.smithi016.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-31T15:02:38.300 INFO:teuthology.orchestra.run.smithi016.stdout: mon_allow_pool_size_one = true 2024-03-31T15:02:38.300 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:02:38.301 INFO:teuthology.orchestra.run.smithi016.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-31T15:02:38.301 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:02:38.301 INFO:teuthology.orchestra.run.smithi016.stdout: osd default data pool replay window = 5 2024-03-31T15:02:38.301 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:02:38.301 INFO:teuthology.orchestra.run.smithi016.stdout: mon allow pool delete = true 2024-03-31T15:02:38.301 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:02:38.301 INFO:teuthology.orchestra.run.smithi016.stdout: mon cluster log file level = debug 2024-03-31T15:02:38.301 INFO:teuthology.orchestra.run.smithi016.stdout: debug asserts on shutdown = true 2024-03-31T15:02:38.301 INFO:teuthology.orchestra.run.smithi016.stdout: mon health detail to clog = false 2024-03-31T15:02:38.301 INFO:teuthology.orchestra.run.smithi016.stdout: mon host = "172.21.15.16,172.21.15.89,[v2:172.21.15.89:3301,v1:172.21.15.89:6790]" 2024-03-31T15:02:38.301 INFO:teuthology.orchestra.run.smithi016.stdout: fsid = 286eb5da-fb47-4550-b290-d9519c13b9a5 2024-03-31T15:02:38.301 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:02:38.301 INFO:teuthology.orchestra.run.smithi016.stdout:[osd] 2024-03-31T15:02:38.301 INFO:teuthology.orchestra.run.smithi016.stdout: osd journal size = 100 2024-03-31T15:02:38.301 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:02:38.302 INFO:teuthology.orchestra.run.smithi016.stdout: osd scrub load threshold = 5.0 2024-03-31T15:02:38.302 INFO:teuthology.orchestra.run.smithi016.stdout: osd scrub max interval = 600 2024-03-31T15:02:38.302 INFO:teuthology.orchestra.run.smithi016.stdout: osd mclock profile = high_recovery_ops 2024-03-31T15:02:38.302 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:02:38.302 INFO:teuthology.orchestra.run.smithi016.stdout: osd recover clone overlap = true 2024-03-31T15:02:38.302 INFO:teuthology.orchestra.run.smithi016.stdout: osd recovery max chunk = 1048576 2024-03-31T15:02:38.302 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:02:38.302 INFO:teuthology.orchestra.run.smithi016.stdout: osd debug shutdown = true 2024-03-31T15:02:38.302 INFO:teuthology.orchestra.run.smithi016.stdout: osd debug op order = true 2024-03-31T15:02:38.302 INFO:teuthology.orchestra.run.smithi016.stdout: osd debug verify stray on activate = true 2024-03-31T15:02:38.302 INFO:teuthology.orchestra.run.smithi016.stdout: osd debug trim objects = true 2024-03-31T15:02:38.302 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:02:38.302 INFO:teuthology.orchestra.run.smithi016.stdout: osd open classes on start = true 2024-03-31T15:02:38.303 INFO:teuthology.orchestra.run.smithi016.stdout: osd debug pg log writeout = true 2024-03-31T15:02:38.303 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:02:38.303 INFO:teuthology.orchestra.run.smithi016.stdout: osd deep scrub update digest min age = 30 2024-03-31T15:02:38.303 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:02:38.303 INFO:teuthology.orchestra.run.smithi016.stdout: osd map max advance = 10 2024-03-31T15:02:38.303 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:02:38.303 INFO:teuthology.orchestra.run.smithi016.stdout: journal zero on create = true 2024-03-31T15:02:38.303 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:02:38.303 INFO:teuthology.orchestra.run.smithi016.stdout: filestore ondisk finisher threads = 3 2024-03-31T15:02:38.303 INFO:teuthology.orchestra.run.smithi016.stdout: filestore apply finisher threads = 3 2024-03-31T15:02:38.303 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:02:38.303 INFO:teuthology.orchestra.run.smithi016.stdout: bdev debug aio = true 2024-03-31T15:02:38.303 INFO:teuthology.orchestra.run.smithi016.stdout: osd debug misdirected ops = true 2024-03-31T15:02:38.303 INFO:teuthology.orchestra.run.smithi016.stdout: bdev async discard = True 2024-03-31T15:02:38.304 INFO:teuthology.orchestra.run.smithi016.stdout: bdev enable discard = True 2024-03-31T15:02:38.304 INFO:teuthology.orchestra.run.smithi016.stdout: bluestore allocator = bitmap 2024-03-31T15:02:38.304 INFO:teuthology.orchestra.run.smithi016.stdout: bluestore block size = 96636764160 2024-03-31T15:02:38.304 INFO:teuthology.orchestra.run.smithi016.stdout: bluestore fsck on mount = True 2024-03-31T15:02:38.304 INFO:teuthology.orchestra.run.smithi016.stdout: debug bluefs = 1/20 2024-03-31T15:02:38.304 INFO:teuthology.orchestra.run.smithi016.stdout: debug bluestore = 1/20 2024-03-31T15:02:38.304 INFO:teuthology.orchestra.run.smithi016.stdout: debug ms = 1 2024-03-31T15:02:38.304 INFO:teuthology.orchestra.run.smithi016.stdout: debug osd = 20 2024-03-31T15:02:38.304 INFO:teuthology.orchestra.run.smithi016.stdout: debug rocksdb = 4/10 2024-03-31T15:02:38.304 INFO:teuthology.orchestra.run.smithi016.stdout: mon osd backfillfull_ratio = 0.85 2024-03-31T15:02:38.304 INFO:teuthology.orchestra.run.smithi016.stdout: mon osd full ratio = 0.9 2024-03-31T15:02:38.304 INFO:teuthology.orchestra.run.smithi016.stdout: mon osd nearfull ratio = 0.8 2024-03-31T15:02:38.304 INFO:teuthology.orchestra.run.smithi016.stdout: osd failsafe full ratio = 0.95 2024-03-31T15:02:38.304 INFO:teuthology.orchestra.run.smithi016.stdout: osd objectstore = bluestore 2024-03-31T15:02:38.305 INFO:teuthology.orchestra.run.smithi016.stdout: osd op complaint time = 180 2024-03-31T15:02:38.305 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:02:38.305 INFO:teuthology.orchestra.run.smithi016.stdout:[mgr] 2024-03-31T15:02:38.305 INFO:teuthology.orchestra.run.smithi016.stdout: debug ms = 1 2024-03-31T15:02:38.305 INFO:teuthology.orchestra.run.smithi016.stdout: debug mgr = 20 2024-03-31T15:02:38.305 INFO:teuthology.orchestra.run.smithi016.stdout: debug mon = 20 2024-03-31T15:02:38.305 INFO:teuthology.orchestra.run.smithi016.stdout: debug auth = 20 2024-03-31T15:02:38.305 INFO:teuthology.orchestra.run.smithi016.stdout: mon reweight min pgs per osd = 4 2024-03-31T15:02:38.305 INFO:teuthology.orchestra.run.smithi016.stdout: mon reweight min bytes per osd = 10 2024-03-31T15:02:38.305 INFO:teuthology.orchestra.run.smithi016.stdout: mgr/telemetry/nag = false 2024-03-31T15:02:38.305 INFO:teuthology.orchestra.run.smithi016.stdout: debug client = 20 2024-03-31T15:02:38.305 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:02:38.305 INFO:teuthology.orchestra.run.smithi016.stdout:[mon] 2024-03-31T15:02:38.305 INFO:teuthology.orchestra.run.smithi016.stdout: debug ms = 1 2024-03-31T15:02:38.306 INFO:teuthology.orchestra.run.smithi016.stdout: debug mon = 20 2024-03-31T15:02:38.306 INFO:teuthology.orchestra.run.smithi016.stdout: debug paxos = 20 2024-03-31T15:02:38.306 INFO:teuthology.orchestra.run.smithi016.stdout: debug auth = 20 2024-03-31T15:02:38.306 INFO:teuthology.orchestra.run.smithi016.stdout: mon data avail warn = 5 2024-03-31T15:02:38.306 INFO:teuthology.orchestra.run.smithi016.stdout: mon mgr mkfs grace = 240 2024-03-31T15:02:38.306 INFO:teuthology.orchestra.run.smithi016.stdout: mon reweight min pgs per osd = 4 2024-03-31T15:02:38.306 INFO:teuthology.orchestra.run.smithi016.stdout: mon osd reporter subtree level = osd 2024-03-31T15:02:38.306 INFO:teuthology.orchestra.run.smithi016.stdout: mon osd prime pg temp = true 2024-03-31T15:02:38.306 INFO:teuthology.orchestra.run.smithi016.stdout: mon reweight min bytes per osd = 10 2024-03-31T15:02:38.306 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:02:38.306 INFO:teuthology.orchestra.run.smithi016.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-31T15:02:38.306 INFO:teuthology.orchestra.run.smithi016.stdout: auth mon ticket ttl = 660 # 11m 2024-03-31T15:02:38.306 INFO:teuthology.orchestra.run.smithi016.stdout: auth service ticket ttl = 240 # 4m 2024-03-31T15:02:38.306 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:02:38.307 INFO:teuthology.orchestra.run.smithi016.stdout: # don't complain about insecure global_id in the test suite 2024-03-31T15:02:38.307 INFO:teuthology.orchestra.run.smithi016.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-31T15:02:38.307 INFO:teuthology.orchestra.run.smithi016.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-31T15:02:38.307 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:02:38.307 INFO:teuthology.orchestra.run.smithi016.stdout: # 1m isn't quite enough 2024-03-31T15:02:38.307 INFO:teuthology.orchestra.run.smithi016.stdout: mon_down_mkfs_grace = 2m 2024-03-31T15:02:38.307 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:02:38.307 INFO:teuthology.orchestra.run.smithi016.stdout: mon_warn_on_filestore_osds = false 2024-03-31T15:02:38.307 INFO:teuthology.orchestra.run.smithi016.stdout: mon down mkfs grace = 300 2024-03-31T15:02:38.307 INFO:teuthology.orchestra.run.smithi016.stdout: mon op complaint time = 120 2024-03-31T15:02:38.307 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:02:38.308 INFO:teuthology.orchestra.run.smithi016.stdout:[client] 2024-03-31T15:02:38.308 INFO:teuthology.orchestra.run.smithi016.stdout: rgw cache enabled = true 2024-03-31T15:02:38.308 INFO:teuthology.orchestra.run.smithi016.stdout: rgw enable ops log = true 2024-03-31T15:02:38.308 INFO:teuthology.orchestra.run.smithi016.stdout: rgw enable usage log = true 2024-03-31T15:02:38.308 INFO:teuthology.orchestra.run.smithi016.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-31T15:02:38.308 INFO:teuthology.orchestra.run.smithi016.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-31T15:02:38.308 INFO:teuthology.orchestra.run.smithi016.stdout: client mount timeout = 600 2024-03-31T15:02:38.308 INFO:teuthology.orchestra.run.smithi016.stdout: debug client = 20 2024-03-31T15:02:38.308 INFO:teuthology.orchestra.run.smithi016.stdout: debug ms = 1 2024-03-31T15:02:38.308 INFO:teuthology.orchestra.run.smithi016.stdout: pid file = "" 2024-03-31T15:02:38.308 INFO:teuthology.orchestra.run.smithi016.stdout: rados mon op timeout = 900 2024-03-31T15:02:38.308 INFO:teuthology.orchestra.run.smithi016.stdout: rados osd op timeout = 900 2024-03-31T15:02:38.308 INFO:teuthology.orchestra.run.smithi016.stdout:[mon.a] 2024-03-31T15:02:38.308 INFO:teuthology.orchestra.run.smithi016.stdout:[mon.b] 2024-03-31T15:02:38.308 INFO:teuthology.orchestra.run.smithi016.stdout:[mon.c] 2024-03-31T15:02:38.309 INFO:teuthology.orchestra.run.smithi016.stdout:[mds.a] 2024-03-31T15:02:38.309 INFO:teuthology.orchestra.run.smithi016.stdout:[mds.c] 2024-03-31T15:02:38.309 INFO:teuthology.orchestra.run.smithi016.stdout:[mds.b] 2024-03-31T15:02:38.309 INFO:teuthology.orchestra.run.smithi016.stdout:[mds.d] 2024-03-31T15:02:38.309 INFO:teuthology.orchestra.run.smithi016.stdout:[mds] 2024-03-31T15:02:38.309 INFO:teuthology.orchestra.run.smithi016.stdout: debug mds = 20 2024-03-31T15:02:38.309 INFO:teuthology.orchestra.run.smithi016.stdout: debug mds balancer = 20 2024-03-31T15:02:38.309 INFO:teuthology.orchestra.run.smithi016.stdout: debug ms = 1 2024-03-31T15:02:38.309 INFO:teuthology.orchestra.run.smithi016.stdout: mds debug frag = True 2024-03-31T15:02:38.309 INFO:teuthology.orchestra.run.smithi016.stdout: mds debug scatterstat = True 2024-03-31T15:02:38.309 INFO:teuthology.orchestra.run.smithi016.stdout: mds op complaint time = 180 2024-03-31T15:02:38.309 INFO:teuthology.orchestra.run.smithi016.stdout: mds verify scatter = True 2024-03-31T15:02:38.309 INFO:teuthology.orchestra.run.smithi016.stdout: osd op complaint time = 180 2024-03-31T15:02:38.309 INFO:teuthology.orchestra.run.smithi016.stdout: rados mon op timeout = 900 2024-03-31T15:02:38.309 INFO:teuthology.orchestra.run.smithi016.stdout: rados osd op timeout = 900 2024-03-31T15:02:38.310 INFO:teuthology.orchestra.run.smithi089.stdout:[global] 2024-03-31T15:02:38.310 INFO:teuthology.orchestra.run.smithi089.stdout: chdir = "" 2024-03-31T15:02:38.310 INFO:teuthology.orchestra.run.smithi089.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-31T15:02:38.310 INFO:teuthology.orchestra.run.smithi089.stdout: auth supported = cephx 2024-03-31T15:02:38.311 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-31T15:02:38.311 INFO:teuthology.orchestra.run.smithi089.stdout: filestore xattr use omap = true 2024-03-31T15:02:38.311 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-31T15:02:38.311 INFO:teuthology.orchestra.run.smithi089.stdout: mon clock drift allowed = 1.000 2024-03-31T15:02:38.311 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-31T15:02:38.311 INFO:teuthology.orchestra.run.smithi089.stdout: osd crush chooseleaf type = 0 2024-03-31T15:02:38.311 INFO:teuthology.orchestra.run.smithi089.stdout: auth debug = true 2024-03-31T15:02:38.311 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-31T15:02:38.311 INFO:teuthology.orchestra.run.smithi089.stdout: ms die on old message = true 2024-03-31T15:02:38.311 INFO:teuthology.orchestra.run.smithi089.stdout: ms die on bug = true 2024-03-31T15:02:38.311 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-31T15:02:38.311 INFO:teuthology.orchestra.run.smithi089.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-31T15:02:38.311 INFO:teuthology.orchestra.run.smithi089.stdout: mon pg warn max object skew = 0 2024-03-31T15:02:38.311 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-31T15:02:38.311 INFO:teuthology.orchestra.run.smithi089.stdout: # disable pg_autoscaler by default for new pools 2024-03-31T15:02:38.312 INFO:teuthology.orchestra.run.smithi089.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-31T15:02:38.312 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-31T15:02:38.312 INFO:teuthology.orchestra.run.smithi089.stdout: osd pool default size = 2 2024-03-31T15:02:38.312 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-31T15:02:38.312 INFO:teuthology.orchestra.run.smithi089.stdout: mon osd allow primary affinity = true 2024-03-31T15:02:38.312 INFO:teuthology.orchestra.run.smithi089.stdout: mon osd allow pg remap = true 2024-03-31T15:02:38.312 INFO:teuthology.orchestra.run.smithi089.stdout: mon warn on legacy crush tunables = false 2024-03-31T15:02:38.312 INFO:teuthology.orchestra.run.smithi089.stdout: mon warn on crush straw calc version zero = false 2024-03-31T15:02:38.312 INFO:teuthology.orchestra.run.smithi089.stdout: mon warn on no sortbitwise = false 2024-03-31T15:02:38.312 INFO:teuthology.orchestra.run.smithi089.stdout: mon warn on osd down out interval zero = false 2024-03-31T15:02:38.312 INFO:teuthology.orchestra.run.smithi089.stdout: mon warn on too few osds = false 2024-03-31T15:02:38.312 INFO:teuthology.orchestra.run.smithi089.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-31T15:02:38.312 INFO:teuthology.orchestra.run.smithi089.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-31T15:02:38.312 INFO:teuthology.orchestra.run.smithi089.stdout: mon_allow_pool_size_one = true 2024-03-31T15:02:38.313 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-31T15:02:38.313 INFO:teuthology.orchestra.run.smithi089.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-31T15:02:38.313 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-31T15:02:38.313 INFO:teuthology.orchestra.run.smithi089.stdout: osd default data pool replay window = 5 2024-03-31T15:02:38.313 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-31T15:02:38.313 INFO:teuthology.orchestra.run.smithi089.stdout: mon allow pool delete = true 2024-03-31T15:02:38.313 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-31T15:02:38.313 INFO:teuthology.orchestra.run.smithi089.stdout: mon cluster log file level = debug 2024-03-31T15:02:38.313 INFO:teuthology.orchestra.run.smithi089.stdout: debug asserts on shutdown = true 2024-03-31T15:02:38.313 INFO:teuthology.orchestra.run.smithi089.stdout: mon health detail to clog = false 2024-03-31T15:02:38.313 INFO:teuthology.orchestra.run.smithi089.stdout: mon host = "172.21.15.16,172.21.15.89,[v2:172.21.15.89:3301,v1:172.21.15.89:6790]" 2024-03-31T15:02:38.313 INFO:teuthology.orchestra.run.smithi089.stdout: fsid = 286eb5da-fb47-4550-b290-d9519c13b9a5 2024-03-31T15:02:38.313 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-31T15:02:38.313 INFO:teuthology.orchestra.run.smithi089.stdout:[osd] 2024-03-31T15:02:38.313 INFO:teuthology.orchestra.run.smithi089.stdout: osd journal size = 100 2024-03-31T15:02:38.313 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-31T15:02:38.314 INFO:teuthology.orchestra.run.smithi089.stdout: osd scrub load threshold = 5.0 2024-03-31T15:02:38.314 INFO:teuthology.orchestra.run.smithi089.stdout: osd scrub max interval = 600 2024-03-31T15:02:38.314 INFO:teuthology.orchestra.run.smithi089.stdout: osd mclock profile = high_recovery_ops 2024-03-31T15:02:38.314 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-31T15:02:38.314 INFO:teuthology.orchestra.run.smithi089.stdout: osd recover clone overlap = true 2024-03-31T15:02:38.314 INFO:teuthology.orchestra.run.smithi089.stdout: osd recovery max chunk = 1048576 2024-03-31T15:02:38.314 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-31T15:02:38.314 INFO:teuthology.orchestra.run.smithi089.stdout: osd debug shutdown = true 2024-03-31T15:02:38.314 INFO:teuthology.orchestra.run.smithi089.stdout: osd debug op order = true 2024-03-31T15:02:38.314 INFO:teuthology.orchestra.run.smithi089.stdout: osd debug verify stray on activate = true 2024-03-31T15:02:38.314 INFO:teuthology.orchestra.run.smithi089.stdout: osd debug trim objects = true 2024-03-31T15:02:38.314 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-31T15:02:38.314 INFO:teuthology.orchestra.run.smithi089.stdout: osd open classes on start = true 2024-03-31T15:02:38.314 INFO:teuthology.orchestra.run.smithi089.stdout: osd debug pg log writeout = true 2024-03-31T15:02:38.315 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-31T15:02:38.315 INFO:teuthology.orchestra.run.smithi089.stdout: osd deep scrub update digest min age = 30 2024-03-31T15:02:38.315 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-31T15:02:38.315 INFO:teuthology.orchestra.run.smithi089.stdout: osd map max advance = 10 2024-03-31T15:02:38.315 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-31T15:02:38.315 INFO:teuthology.orchestra.run.smithi089.stdout: journal zero on create = true 2024-03-31T15:02:38.315 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-31T15:02:38.315 INFO:teuthology.orchestra.run.smithi089.stdout: filestore ondisk finisher threads = 3 2024-03-31T15:02:38.315 INFO:teuthology.orchestra.run.smithi089.stdout: filestore apply finisher threads = 3 2024-03-31T15:02:38.315 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-31T15:02:38.315 INFO:teuthology.orchestra.run.smithi089.stdout: bdev debug aio = true 2024-03-31T15:02:38.315 INFO:teuthology.orchestra.run.smithi089.stdout: osd debug misdirected ops = true 2024-03-31T15:02:38.315 INFO:teuthology.orchestra.run.smithi089.stdout: bdev async discard = True 2024-03-31T15:02:38.315 INFO:teuthology.orchestra.run.smithi089.stdout: bdev enable discard = True 2024-03-31T15:02:38.315 INFO:teuthology.orchestra.run.smithi089.stdout: bluestore allocator = bitmap 2024-03-31T15:02:38.316 INFO:teuthology.orchestra.run.smithi089.stdout: bluestore block size = 96636764160 2024-03-31T15:02:38.316 INFO:teuthology.orchestra.run.smithi089.stdout: bluestore fsck on mount = True 2024-03-31T15:02:38.316 INFO:teuthology.orchestra.run.smithi089.stdout: debug bluefs = 1/20 2024-03-31T15:02:38.316 INFO:teuthology.orchestra.run.smithi089.stdout: debug bluestore = 1/20 2024-03-31T15:02:38.316 INFO:teuthology.orchestra.run.smithi089.stdout: debug ms = 1 2024-03-31T15:02:38.316 INFO:teuthology.orchestra.run.smithi089.stdout: debug osd = 20 2024-03-31T15:02:38.316 INFO:teuthology.orchestra.run.smithi089.stdout: debug rocksdb = 4/10 2024-03-31T15:02:38.316 INFO:teuthology.orchestra.run.smithi089.stdout: mon osd backfillfull_ratio = 0.85 2024-03-31T15:02:38.316 INFO:teuthology.orchestra.run.smithi089.stdout: mon osd full ratio = 0.9 2024-03-31T15:02:38.316 INFO:teuthology.orchestra.run.smithi089.stdout: mon osd nearfull ratio = 0.8 2024-03-31T15:02:38.316 INFO:teuthology.orchestra.run.smithi089.stdout: osd failsafe full ratio = 0.95 2024-03-31T15:02:38.316 INFO:teuthology.orchestra.run.smithi089.stdout: osd objectstore = bluestore 2024-03-31T15:02:38.316 INFO:teuthology.orchestra.run.smithi089.stdout: osd op complaint time = 180 2024-03-31T15:02:38.316 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-31T15:02:38.317 INFO:teuthology.orchestra.run.smithi089.stdout:[mgr] 2024-03-31T15:02:38.317 INFO:teuthology.orchestra.run.smithi089.stdout: debug ms = 1 2024-03-31T15:02:38.317 INFO:teuthology.orchestra.run.smithi089.stdout: debug mgr = 20 2024-03-31T15:02:38.317 INFO:teuthology.orchestra.run.smithi089.stdout: debug mon = 20 2024-03-31T15:02:38.317 INFO:teuthology.orchestra.run.smithi089.stdout: debug auth = 20 2024-03-31T15:02:38.317 INFO:teuthology.orchestra.run.smithi089.stdout: mon reweight min pgs per osd = 4 2024-03-31T15:02:38.317 INFO:teuthology.orchestra.run.smithi089.stdout: mon reweight min bytes per osd = 10 2024-03-31T15:02:38.317 INFO:teuthology.orchestra.run.smithi089.stdout: mgr/telemetry/nag = false 2024-03-31T15:02:38.317 INFO:teuthology.orchestra.run.smithi089.stdout: debug client = 20 2024-03-31T15:02:38.317 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-31T15:02:38.317 INFO:teuthology.orchestra.run.smithi089.stdout:[mon] 2024-03-31T15:02:38.317 INFO:teuthology.orchestra.run.smithi089.stdout: debug ms = 1 2024-03-31T15:02:38.317 INFO:teuthology.orchestra.run.smithi089.stdout: debug mon = 20 2024-03-31T15:02:38.317 INFO:teuthology.orchestra.run.smithi089.stdout: debug paxos = 20 2024-03-31T15:02:38.318 INFO:teuthology.orchestra.run.smithi089.stdout: debug auth = 20 2024-03-31T15:02:38.318 INFO:teuthology.orchestra.run.smithi089.stdout: mon data avail warn = 5 2024-03-31T15:02:38.318 INFO:teuthology.orchestra.run.smithi089.stdout: mon mgr mkfs grace = 240 2024-03-31T15:02:38.319 INFO:teuthology.orchestra.run.smithi089.stdout: mon reweight min pgs per osd = 4 2024-03-31T15:02:38.319 INFO:teuthology.orchestra.run.smithi089.stdout: mon osd reporter subtree level = osd 2024-03-31T15:02:38.319 INFO:teuthology.orchestra.run.smithi089.stdout: mon osd prime pg temp = true 2024-03-31T15:02:38.319 INFO:teuthology.orchestra.run.smithi089.stdout: mon reweight min bytes per osd = 10 2024-03-31T15:02:38.319 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-31T15:02:38.319 INFO:teuthology.orchestra.run.smithi089.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-31T15:02:38.319 INFO:teuthology.orchestra.run.smithi089.stdout: auth mon ticket ttl = 660 # 11m 2024-03-31T15:02:38.319 INFO:teuthology.orchestra.run.smithi089.stdout: auth service ticket ttl = 240 # 4m 2024-03-31T15:02:38.319 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-31T15:02:38.319 INFO:teuthology.orchestra.run.smithi089.stdout: # don't complain about insecure global_id in the test suite 2024-03-31T15:02:38.319 INFO:teuthology.orchestra.run.smithi089.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-31T15:02:38.319 INFO:teuthology.orchestra.run.smithi089.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-31T15:02:38.320 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-31T15:02:38.320 INFO:teuthology.orchestra.run.smithi089.stdout: # 1m isn't quite enough 2024-03-31T15:02:38.320 INFO:teuthology.orchestra.run.smithi089.stdout: mon_down_mkfs_grace = 2m 2024-03-31T15:02:38.320 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-31T15:02:38.320 INFO:teuthology.orchestra.run.smithi089.stdout: mon_warn_on_filestore_osds = false 2024-03-31T15:02:38.320 INFO:teuthology.orchestra.run.smithi089.stdout: mon down mkfs grace = 300 2024-03-31T15:02:38.320 INFO:teuthology.orchestra.run.smithi089.stdout: mon op complaint time = 120 2024-03-31T15:02:38.320 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-31T15:02:38.320 INFO:teuthology.orchestra.run.smithi089.stdout:[client] 2024-03-31T15:02:38.320 INFO:teuthology.orchestra.run.smithi089.stdout: rgw cache enabled = true 2024-03-31T15:02:38.320 INFO:teuthology.orchestra.run.smithi089.stdout: rgw enable ops log = true 2024-03-31T15:02:38.320 INFO:teuthology.orchestra.run.smithi089.stdout: rgw enable usage log = true 2024-03-31T15:02:38.320 INFO:teuthology.orchestra.run.smithi089.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-31T15:02:38.321 INFO:teuthology.orchestra.run.smithi089.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-31T15:02:38.321 INFO:teuthology.orchestra.run.smithi089.stdout: client mount timeout = 600 2024-03-31T15:02:38.321 INFO:teuthology.orchestra.run.smithi089.stdout: debug client = 20 2024-03-31T15:02:38.321 INFO:teuthology.orchestra.run.smithi089.stdout: debug ms = 1 2024-03-31T15:02:38.321 INFO:teuthology.orchestra.run.smithi089.stdout: pid file = "" 2024-03-31T15:02:38.321 INFO:teuthology.orchestra.run.smithi089.stdout: rados mon op timeout = 900 2024-03-31T15:02:38.321 INFO:teuthology.orchestra.run.smithi089.stdout: rados osd op timeout = 900 2024-03-31T15:02:38.321 INFO:teuthology.orchestra.run.smithi089.stdout:[mon.a] 2024-03-31T15:02:38.321 INFO:teuthology.orchestra.run.smithi089.stdout:[mon.b] 2024-03-31T15:02:38.321 INFO:teuthology.orchestra.run.smithi089.stdout:[mon.c] 2024-03-31T15:02:38.321 INFO:teuthology.orchestra.run.smithi089.stdout:[mds.a] 2024-03-31T15:02:38.321 INFO:teuthology.orchestra.run.smithi089.stdout:[mds.c] 2024-03-31T15:02:38.321 INFO:teuthology.orchestra.run.smithi089.stdout:[mds.b] 2024-03-31T15:02:38.321 INFO:teuthology.orchestra.run.smithi089.stdout:[mds.d] 2024-03-31T15:02:38.322 INFO:teuthology.orchestra.run.smithi089.stdout:[mds] 2024-03-31T15:02:38.322 INFO:teuthology.orchestra.run.smithi089.stdout: debug mds = 20 2024-03-31T15:02:38.322 INFO:teuthology.orchestra.run.smithi089.stdout: debug mds balancer = 20 2024-03-31T15:02:38.322 INFO:teuthology.orchestra.run.smithi089.stdout: debug ms = 1 2024-03-31T15:02:38.322 INFO:teuthology.orchestra.run.smithi089.stdout: mds debug frag = True 2024-03-31T15:02:38.322 INFO:teuthology.orchestra.run.smithi089.stdout: mds debug scatterstat = True 2024-03-31T15:02:38.322 INFO:teuthology.orchestra.run.smithi089.stdout: mds op complaint time = 180 2024-03-31T15:02:38.322 INFO:teuthology.orchestra.run.smithi089.stdout: mds verify scatter = True 2024-03-31T15:02:38.322 INFO:teuthology.orchestra.run.smithi089.stdout: osd op complaint time = 180 2024-03-31T15:02:38.322 INFO:teuthology.orchestra.run.smithi089.stdout: rados mon op timeout = 900 2024-03-31T15:02:38.322 INFO:teuthology.orchestra.run.smithi089.stdout: rados osd op timeout = 900 2024-03-31T15:02:38.324 INFO:tasks.ceph:Creating admin key on mon.a... 2024-03-31T15:02:38.324 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-03-31T15:02:38.427 INFO:tasks.ceph:Copying monmap to all nodes... 2024-03-31T15:02:38.428 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-03-31T15:02:38.428 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-03-31T15:02:38.436 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-03-31T15:02:38.436 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-03-31T15:02:38.486 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi016.front.sepia.ceph.com 2024-03-31T15:02:38.487 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-03-31T15:02:38.487 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-31T15:02:38.487 DEBUG:teuthology.orchestra.run.smithi016:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-31T15:02:38.558 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-03-31T15:02:38.559 DEBUG:teuthology.orchestra.run.smithi016:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-31T15:02:38.607 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi089.front.sepia.ceph.com 2024-03-31T15:02:38.607 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-03-31T15:02:38.607 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-31T15:02:38.607 DEBUG:teuthology.orchestra.run.smithi089:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-31T15:02:38.641 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-03-31T15:02:38.641 DEBUG:teuthology.orchestra.run.smithi089:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-31T15:02:38.693 INFO:tasks.ceph:Setting up mon nodes... 2024-03-31T15:02:38.693 INFO:tasks.ceph:Setting up mgr nodes... 2024-03-31T15:02:38.693 DEBUG:teuthology.orchestra.run.smithi016:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-03-31T15:02:38.747 INFO:teuthology.orchestra.run.smithi016.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-03-31T15:02:38.752 DEBUG:teuthology.orchestra.run.smithi089:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-03-31T15:02:38.811 INFO:teuthology.orchestra.run.smithi089.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-03-31T15:02:38.827 INFO:tasks.ceph:Setting up mds nodes... 2024-03-31T15:02:38.827 DEBUG:teuthology.orchestra.run.smithi016:> sudo mkdir -p /var/lib/ceph/mds/ceph-a && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.a /var/lib/ceph/mds/ceph-a/keyring 2024-03-31T15:02:38.883 INFO:teuthology.orchestra.run.smithi016.stdout:creating /var/lib/ceph/mds/ceph-a/keyring 2024-03-31T15:02:38.900 DEBUG:teuthology.orchestra.run.smithi016:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-a 2024-03-31T15:02:38.920 DEBUG:teuthology.orchestra.run.smithi016:> sudo mkdir -p /var/lib/ceph/mds/ceph-c && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.c /var/lib/ceph/mds/ceph-c/keyring 2024-03-31T15:02:39.017 INFO:teuthology.orchestra.run.smithi016.stdout:creating /var/lib/ceph/mds/ceph-c/keyring 2024-03-31T15:02:39.021 DEBUG:teuthology.orchestra.run.smithi016:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-c 2024-03-31T15:02:39.041 DEBUG:teuthology.orchestra.run.smithi089:> sudo mkdir -p /var/lib/ceph/mds/ceph-b && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.b /var/lib/ceph/mds/ceph-b/keyring 2024-03-31T15:02:39.100 INFO:teuthology.orchestra.run.smithi089.stdout:creating /var/lib/ceph/mds/ceph-b/keyring 2024-03-31T15:02:39.104 DEBUG:teuthology.orchestra.run.smithi089:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-b 2024-03-31T15:02:39.125 DEBUG:teuthology.orchestra.run.smithi089:> sudo mkdir -p /var/lib/ceph/mds/ceph-d && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.d /var/lib/ceph/mds/ceph-d/keyring 2024-03-31T15:02:39.224 INFO:teuthology.orchestra.run.smithi089.stdout:creating /var/lib/ceph/mds/ceph-d/keyring 2024-03-31T15:02:39.228 DEBUG:teuthology.orchestra.run.smithi089:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-d 2024-03-31T15:02:39.248 INFO:tasks.ceph_client:Setting up client nodes... 2024-03-31T15:02:39.248 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-31T15:02:39.292 INFO:teuthology.orchestra.run.smithi016.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-03-31T15:02:39.309 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.2 /etc/ceph/ceph.client.2.keyring && sudo chmod 0644 /etc/ceph/ceph.client.2.keyring 2024-03-31T15:02:39.397 INFO:teuthology.orchestra.run.smithi016.stdout:creating /etc/ceph/ceph.client.2.keyring 2024-03-31T15:02:39.414 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-03-31T15:02:39.456 INFO:teuthology.orchestra.run.smithi089.stdout:creating /etc/ceph/ceph.client.1.keyring 2024-03-31T15:02:39.473 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.3 /etc/ceph/ceph.client.3.keyring && sudo chmod 0644 /etc/ceph/ceph.client.3.keyring 2024-03-31T15:02:39.557 INFO:teuthology.orchestra.run.smithi089.stdout:creating /etc/ceph/ceph.client.3.keyring 2024-03-31T15:02:39.573 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-03-31T15:02:39.573 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi016.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi089.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-31T15:02:39.574 DEBUG:teuthology.orchestra.run.smithi016:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-03-31T15:02:39.593 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-31T15:02:39.593 INFO:tasks.ceph:role: osd.0 2024-03-31T15:02:39.593 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi016.front.sepia.ceph.com 2024-03-31T15:02:39.593 DEBUG:teuthology.orchestra.run.smithi016:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-31T15:02:39.653 INFO:teuthology.orchestra.run.smithi016.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-31T15:02:39.654 INFO:teuthology.orchestra.run.smithi016.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-31T15:02:39.654 INFO:teuthology.orchestra.run.smithi016.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-31T15:02:39.654 INFO:teuthology.orchestra.run.smithi016.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-03-31T15:02:39.654 INFO:teuthology.orchestra.run.smithi016.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-31T15:02:39.654 INFO:teuthology.orchestra.run.smithi016.stdout: = sunit=0 swidth=0 blks 2024-03-31T15:02:39.654 INFO:teuthology.orchestra.run.smithi016.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-31T15:02:39.654 INFO:teuthology.orchestra.run.smithi016.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-03-31T15:02:39.654 INFO:teuthology.orchestra.run.smithi016.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-31T15:02:39.654 INFO:teuthology.orchestra.run.smithi016.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-31T15:02:39.679 INFO:teuthology.orchestra.run.smithi016.stdout:Discarding blocks...Done. 2024-03-31T15:02:39.681 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi016.front.sepia.ceph.com -o noatime 2024-03-31T15:02:39.681 DEBUG:teuthology.orchestra.run.smithi016:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-03-31T15:02:39.747 DEBUG:teuthology.orchestra.run.smithi016:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-03-31T15:02:39.807 INFO:teuthology.orchestra.run.smithi016.stderr:sudo: /sbin/restorecon: command not found 2024-03-31T15:02:39.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-31T15:02:39.808 DEBUG:teuthology.orchestra.run.smithi016:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-03-31T15:02:39.870 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-31T15:02:39.870 INFO:tasks.ceph:role: osd.1 2024-03-31T15:02:39.870 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi016.front.sepia.ceph.com 2024-03-31T15:02:39.870 DEBUG:teuthology.orchestra.run.smithi016:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-31T15:02:39.929 INFO:teuthology.orchestra.run.smithi016.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-31T15:02:39.930 INFO:teuthology.orchestra.run.smithi016.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-31T15:02:39.930 INFO:teuthology.orchestra.run.smithi016.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-31T15:02:39.930 INFO:teuthology.orchestra.run.smithi016.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-03-31T15:02:39.930 INFO:teuthology.orchestra.run.smithi016.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-31T15:02:39.930 INFO:teuthology.orchestra.run.smithi016.stdout: = sunit=0 swidth=0 blks 2024-03-31T15:02:39.930 INFO:teuthology.orchestra.run.smithi016.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-31T15:02:39.931 INFO:teuthology.orchestra.run.smithi016.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-03-31T15:02:39.931 INFO:teuthology.orchestra.run.smithi016.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-31T15:02:39.931 INFO:teuthology.orchestra.run.smithi016.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-31T15:02:39.954 INFO:teuthology.orchestra.run.smithi016.stdout:Discarding blocks...Done. 2024-03-31T15:02:39.957 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi016.front.sepia.ceph.com -o noatime 2024-03-31T15:02:39.957 DEBUG:teuthology.orchestra.run.smithi016:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-03-31T15:02:40.024 DEBUG:teuthology.orchestra.run.smithi016:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-03-31T15:02:40.082 INFO:teuthology.orchestra.run.smithi016.stderr:sudo: /sbin/restorecon: command not found 2024-03-31T15:02:40.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-31T15:02:40.083 DEBUG:teuthology.orchestra.run.smithi016:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-03-31T15:02:40.146 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-31T15:02:40.146 INFO:tasks.ceph:role: osd.2 2024-03-31T15:02:40.146 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi016.front.sepia.ceph.com 2024-03-31T15:02:40.146 DEBUG:teuthology.orchestra.run.smithi016:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-31T15:02:40.205 INFO:teuthology.orchestra.run.smithi016.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-31T15:02:40.205 INFO:teuthology.orchestra.run.smithi016.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-31T15:02:40.205 INFO:teuthology.orchestra.run.smithi016.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-31T15:02:40.205 INFO:teuthology.orchestra.run.smithi016.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-03-31T15:02:40.205 INFO:teuthology.orchestra.run.smithi016.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-31T15:02:40.206 INFO:teuthology.orchestra.run.smithi016.stdout: = sunit=0 swidth=0 blks 2024-03-31T15:02:40.206 INFO:teuthology.orchestra.run.smithi016.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-31T15:02:40.206 INFO:teuthology.orchestra.run.smithi016.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-03-31T15:02:40.206 INFO:teuthology.orchestra.run.smithi016.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-31T15:02:40.206 INFO:teuthology.orchestra.run.smithi016.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-31T15:02:40.229 INFO:teuthology.orchestra.run.smithi016.stdout:Discarding blocks...Done. 2024-03-31T15:02:40.232 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi016.front.sepia.ceph.com -o noatime 2024-03-31T15:02:40.232 DEBUG:teuthology.orchestra.run.smithi016:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-03-31T15:02:40.299 DEBUG:teuthology.orchestra.run.smithi016:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-03-31T15:02:40.358 INFO:teuthology.orchestra.run.smithi016.stderr:sudo: /sbin/restorecon: command not found 2024-03-31T15:02:40.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-31T15:02:40.359 DEBUG:teuthology.orchestra.run.smithi016:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-03-31T15:02:40.422 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-31T15:02:40.422 INFO:tasks.ceph:role: osd.3 2024-03-31T15:02:40.422 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi016.front.sepia.ceph.com 2024-03-31T15:02:40.422 DEBUG:teuthology.orchestra.run.smithi016:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-31T15:02:40.486 INFO:teuthology.orchestra.run.smithi016.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-31T15:02:40.486 INFO:teuthology.orchestra.run.smithi016.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-31T15:02:40.486 INFO:teuthology.orchestra.run.smithi016.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-31T15:02:40.486 INFO:teuthology.orchestra.run.smithi016.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-03-31T15:02:40.486 INFO:teuthology.orchestra.run.smithi016.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-31T15:02:40.486 INFO:teuthology.orchestra.run.smithi016.stdout: = sunit=0 swidth=0 blks 2024-03-31T15:02:40.486 INFO:teuthology.orchestra.run.smithi016.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-31T15:02:40.487 INFO:teuthology.orchestra.run.smithi016.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-03-31T15:02:40.487 INFO:teuthology.orchestra.run.smithi016.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-31T15:02:40.487 INFO:teuthology.orchestra.run.smithi016.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-31T15:02:40.513 INFO:teuthology.orchestra.run.smithi016.stdout:Discarding blocks...Done. 2024-03-31T15:02:40.516 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi016.front.sepia.ceph.com -o noatime 2024-03-31T15:02:40.516 DEBUG:teuthology.orchestra.run.smithi016:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-03-31T15:02:40.584 DEBUG:teuthology.orchestra.run.smithi016:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-03-31T15:02:40.643 INFO:teuthology.orchestra.run.smithi016.stderr:sudo: /sbin/restorecon: command not found 2024-03-31T15:02:40.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-31T15:02:40.644 DEBUG:teuthology.orchestra.run.smithi016:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-31T15:02:40.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:02:40.779+0000 7efeff5a17c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-03-31T15:02:40.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:02:40.779+0000 7efeff5a17c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-03-31T15:02:40.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:02:40.779+0000 7efeff5a17c0 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-31T15:02:40.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:02:40.779+0000 7efeff5a17c0 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-31T15:02:40.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:02:40.779+0000 7efeff5a17c0 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-31T15:02:40.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:02:40.779+0000 7efeff5a17c0 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-03-31T15:02:43.775 DEBUG:teuthology.orchestra.run.smithi016:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-03-31T15:02:43.797 DEBUG:teuthology.orchestra.run.smithi016:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-31T15:02:43.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:02:43.903+0000 7feef91fd7c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-03-31T15:02:43.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:02:43.903+0000 7feef91fd7c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-03-31T15:02:43.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:02:43.903+0000 7feef91fd7c0 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-31T15:02:43.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:02:43.903+0000 7feef91fd7c0 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-31T15:02:43.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:02:43.903+0000 7feef91fd7c0 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-31T15:02:43.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:02:43.907+0000 7feef91fd7c0 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-03-31T15:02:46.886 DEBUG:teuthology.orchestra.run.smithi016:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-03-31T15:02:46.908 DEBUG:teuthology.orchestra.run.smithi016:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-31T15:02:47.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:02:47.015+0000 7f010e9e87c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-03-31T15:02:47.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:02:47.015+0000 7f010e9e87c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-03-31T15:02:47.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:02:47.015+0000 7f010e9e87c0 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-31T15:02:47.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:02:47.015+0000 7f010e9e87c0 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-31T15:02:47.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:02:47.015+0000 7f010e9e87c0 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-31T15:02:47.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:02:47.015+0000 7f010e9e87c0 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-03-31T15:02:49.978 DEBUG:teuthology.orchestra.run.smithi016:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-03-31T15:02:49.999 DEBUG:teuthology.orchestra.run.smithi016:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-31T15:02:50.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:02:50.103+0000 7fc9f6c577c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-03-31T15:02:50.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:02:50.103+0000 7fc9f6c577c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-03-31T15:02:50.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:02:50.103+0000 7fc9f6c577c0 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-31T15:02:50.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:02:50.103+0000 7fc9f6c577c0 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-31T15:02:50.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:02:50.103+0000 7fc9f6c577c0 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-31T15:02:50.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:02:50.107+0000 7fc9f6c577c0 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-03-31T15:02:53.065 DEBUG:teuthology.orchestra.run.smithi016:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-03-31T15:02:53.086 DEBUG:teuthology.orchestra.run.smithi089:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-03-31T15:02:53.108 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-31T15:02:53.108 INFO:tasks.ceph:role: osd.4 2024-03-31T15:02:53.108 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi089.front.sepia.ceph.com 2024-03-31T15:02:53.108 DEBUG:teuthology.orchestra.run.smithi089:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-31T15:02:53.181 INFO:teuthology.orchestra.run.smithi089.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-31T15:02:53.181 INFO:teuthology.orchestra.run.smithi089.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-31T15:02:53.181 INFO:teuthology.orchestra.run.smithi089.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-31T15:02:53.181 INFO:teuthology.orchestra.run.smithi089.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-03-31T15:02:53.181 INFO:teuthology.orchestra.run.smithi089.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-31T15:02:53.181 INFO:teuthology.orchestra.run.smithi089.stdout: = sunit=0 swidth=0 blks 2024-03-31T15:02:53.181 INFO:teuthology.orchestra.run.smithi089.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-31T15:02:53.181 INFO:teuthology.orchestra.run.smithi089.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-03-31T15:02:53.182 INFO:teuthology.orchestra.run.smithi089.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-31T15:02:53.182 INFO:teuthology.orchestra.run.smithi089.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-31T15:02:53.636 INFO:teuthology.orchestra.run.smithi089.stdout:Discarding blocks...Done. 2024-03-31T15:02:53.639 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi089.front.sepia.ceph.com -o noatime 2024-03-31T15:02:53.639 DEBUG:teuthology.orchestra.run.smithi089:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2024-03-31T15:02:53.751 DEBUG:teuthology.orchestra.run.smithi089:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-03-31T15:02:53.768 INFO:teuthology.orchestra.run.smithi089.stderr:sudo: /sbin/restorecon: command not found 2024-03-31T15:02:53.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-31T15:02:53.770 DEBUG:teuthology.orchestra.run.smithi089:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-03-31T15:02:53.832 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-31T15:02:53.832 INFO:tasks.ceph:role: osd.5 2024-03-31T15:02:53.832 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi089.front.sepia.ceph.com 2024-03-31T15:02:53.833 DEBUG:teuthology.orchestra.run.smithi089:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-31T15:02:53.904 INFO:teuthology.orchestra.run.smithi089.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-31T15:02:53.904 INFO:teuthology.orchestra.run.smithi089.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-31T15:02:53.904 INFO:teuthology.orchestra.run.smithi089.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-31T15:02:53.905 INFO:teuthology.orchestra.run.smithi089.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-03-31T15:02:53.905 INFO:teuthology.orchestra.run.smithi089.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-31T15:02:53.905 INFO:teuthology.orchestra.run.smithi089.stdout: = sunit=0 swidth=0 blks 2024-03-31T15:02:53.905 INFO:teuthology.orchestra.run.smithi089.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-31T15:02:53.905 INFO:teuthology.orchestra.run.smithi089.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-03-31T15:02:53.905 INFO:teuthology.orchestra.run.smithi089.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-31T15:02:53.905 INFO:teuthology.orchestra.run.smithi089.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-31T15:02:54.386 INFO:teuthology.orchestra.run.smithi089.stdout:Discarding blocks...Done. 2024-03-31T15:02:54.388 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi089.front.sepia.ceph.com -o noatime 2024-03-31T15:02:54.389 DEBUG:teuthology.orchestra.run.smithi089:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-03-31T15:02:54.558 DEBUG:teuthology.orchestra.run.smithi089:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-03-31T15:02:54.575 INFO:teuthology.orchestra.run.smithi089.stderr:sudo: /sbin/restorecon: command not found 2024-03-31T15:02:54.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-31T15:02:54.576 DEBUG:teuthology.orchestra.run.smithi089:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-03-31T15:02:54.645 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-31T15:02:54.645 INFO:tasks.ceph:role: osd.6 2024-03-31T15:02:54.645 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi089.front.sepia.ceph.com 2024-03-31T15:02:54.645 DEBUG:teuthology.orchestra.run.smithi089:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-31T15:02:54.716 INFO:teuthology.orchestra.run.smithi089.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-31T15:02:54.716 INFO:teuthology.orchestra.run.smithi089.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-31T15:02:54.716 INFO:teuthology.orchestra.run.smithi089.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-31T15:02:54.716 INFO:teuthology.orchestra.run.smithi089.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-03-31T15:02:54.716 INFO:teuthology.orchestra.run.smithi089.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-31T15:02:54.716 INFO:teuthology.orchestra.run.smithi089.stdout: = sunit=0 swidth=0 blks 2024-03-31T15:02:54.716 INFO:teuthology.orchestra.run.smithi089.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-31T15:02:54.716 INFO:teuthology.orchestra.run.smithi089.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-03-31T15:02:54.717 INFO:teuthology.orchestra.run.smithi089.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-31T15:02:54.717 INFO:teuthology.orchestra.run.smithi089.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-31T15:02:55.198 INFO:teuthology.orchestra.run.smithi089.stdout:Discarding blocks...Done. 2024-03-31T15:02:55.201 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi089.front.sepia.ceph.com -o noatime 2024-03-31T15:02:55.201 DEBUG:teuthology.orchestra.run.smithi089:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2024-03-31T15:02:55.321 DEBUG:teuthology.orchestra.run.smithi089:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-03-31T15:02:55.338 INFO:teuthology.orchestra.run.smithi089.stderr:sudo: /sbin/restorecon: command not found 2024-03-31T15:02:55.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-31T15:02:55.339 DEBUG:teuthology.orchestra.run.smithi089:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-03-31T15:02:55.401 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-31T15:02:55.401 INFO:tasks.ceph:role: osd.7 2024-03-31T15:02:55.401 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi089.front.sepia.ceph.com 2024-03-31T15:02:55.401 DEBUG:teuthology.orchestra.run.smithi089:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-31T15:02:55.472 INFO:teuthology.orchestra.run.smithi089.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-31T15:02:55.472 INFO:teuthology.orchestra.run.smithi089.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-31T15:02:55.472 INFO:teuthology.orchestra.run.smithi089.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-31T15:02:55.473 INFO:teuthology.orchestra.run.smithi089.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-03-31T15:02:55.473 INFO:teuthology.orchestra.run.smithi089.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-31T15:02:55.473 INFO:teuthology.orchestra.run.smithi089.stdout: = sunit=0 swidth=0 blks 2024-03-31T15:02:55.473 INFO:teuthology.orchestra.run.smithi089.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-31T15:02:55.473 INFO:teuthology.orchestra.run.smithi089.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-03-31T15:02:55.473 INFO:teuthology.orchestra.run.smithi089.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-31T15:02:55.473 INFO:teuthology.orchestra.run.smithi089.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-31T15:02:55.918 INFO:teuthology.orchestra.run.smithi089.stdout:Discarding blocks...Done. 2024-03-31T15:02:55.921 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi089.front.sepia.ceph.com -o noatime 2024-03-31T15:02:55.921 DEBUG:teuthology.orchestra.run.smithi089:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2024-03-31T15:02:56.030 DEBUG:teuthology.orchestra.run.smithi089:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-03-31T15:02:56.047 INFO:teuthology.orchestra.run.smithi089.stderr:sudo: /sbin/restorecon: command not found 2024-03-31T15:02:56.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-31T15:02:56.048 DEBUG:teuthology.orchestra.run.smithi089:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-31T15:02:56.176 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:02:56.184+0000 7f23d1c8c7c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-03-31T15:02:56.179 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:02:56.188+0000 7f23d1c8c7c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-03-31T15:02:56.179 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:02:56.188+0000 7f23d1c8c7c0 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-31T15:02:56.179 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:02:56.188+0000 7f23d1c8c7c0 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-31T15:02:56.179 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:02:56.188+0000 7f23d1c8c7c0 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-31T15:02:56.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:02:56.196+0000 7f23d1c8c7c0 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-03-31T15:02:59.194 DEBUG:teuthology.orchestra.run.smithi089:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-03-31T15:02:59.215 DEBUG:teuthology.orchestra.run.smithi089:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-31T15:02:59.310 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:02:59.320+0000 7fb1bafd67c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-03-31T15:02:59.311 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:02:59.320+0000 7fb1bafd67c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-03-31T15:02:59.311 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:02:59.320+0000 7fb1bafd67c0 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-31T15:02:59.311 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:02:59.320+0000 7fb1bafd67c0 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-31T15:02:59.311 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:02:59.320+0000 7fb1bafd67c0 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-31T15:02:59.312 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:02:59.320+0000 7fb1bafd67c0 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-03-31T15:03:02.279 DEBUG:teuthology.orchestra.run.smithi089:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-03-31T15:03:02.299 DEBUG:teuthology.orchestra.run.smithi089:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-31T15:03:02.392 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:02.399+0000 7f55b8c727c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-03-31T15:03:02.392 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:02.399+0000 7f55b8c727c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-03-31T15:03:02.392 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:02.399+0000 7f55b8c727c0 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-31T15:03:02.392 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:02.399+0000 7f55b8c727c0 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-31T15:03:02.393 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:02.399+0000 7f55b8c727c0 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-31T15:03:02.394 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:02.403+0000 7f55b8c727c0 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-03-31T15:03:05.363 DEBUG:teuthology.orchestra.run.smithi089:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-03-31T15:03:05.385 DEBUG:teuthology.orchestra.run.smithi089:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-31T15:03:05.483 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:05.491+0000 7f4d7fbda7c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-03-31T15:03:05.484 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:05.491+0000 7f4d7fbda7c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-03-31T15:03:05.484 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:05.491+0000 7f4d7fbda7c0 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-31T15:03:05.485 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:05.491+0000 7f4d7fbda7c0 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-31T15:03:05.485 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:05.491+0000 7f4d7fbda7c0 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-31T15:03:05.486 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:05.495+0000 7f4d7fbda7c0 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-03-31T15:03:06.442 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-31T15:03:06.446 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-31T15:03:08.454 DEBUG:teuthology.orchestra.run.smithi089:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-03-31T15:03:08.474 INFO:tasks.ceph:Reading keys from all nodes... 2024-03-31T15:03:08.474 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-03-31T15:03:08.474 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-03-31T15:03:08.496 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-03-31T15:03:08.496 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd if=/var/lib/ceph/mds/ceph-a/keyring of=/dev/stdout 2024-03-31T15:03:08.559 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-03-31T15:03:08.560 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd if=/var/lib/ceph/mds/ceph-c/keyring of=/dev/stdout 2024-03-31T15:03:08.623 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-03-31T15:03:08.624 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-03-31T15:03:08.687 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-03-31T15:03:08.688 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-03-31T15:03:08.751 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-03-31T15:03:08.751 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-03-31T15:03:08.816 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-03-31T15:03:08.816 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-03-31T15:03:08.883 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-03-31T15:03:08.883 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-03-31T15:03:08.904 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-03-31T15:03:08.904 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd if=/var/lib/ceph/mds/ceph-b/keyring of=/dev/stdout 2024-03-31T15:03:08.969 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-03-31T15:03:08.969 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd if=/var/lib/ceph/mds/ceph-d/keyring of=/dev/stdout 2024-03-31T15:03:09.033 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-03-31T15:03:09.033 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-03-31T15:03:09.101 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-03-31T15:03:09.101 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-03-31T15:03:09.164 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-03-31T15:03:09.164 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-03-31T15:03:09.229 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-03-31T15:03:09.229 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-03-31T15:03:09.292 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-03-31T15:03:09.293 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-03-31T15:03:09.302 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-03-31T15:03:09.302 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/etc/ceph/ceph.client.2.keyring of=/dev/stdout 2024-03-31T15:03:09.351 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-03-31T15:03:09.351 DEBUG:teuthology.orchestra.run.smithi089:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2024-03-31T15:03:09.360 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-03-31T15:03:09.360 DEBUG:teuthology.orchestra.run.smithi089:> dd if=/etc/ceph/ceph.client.3.keyring of=/dev/stdout 2024-03-31T15:03:09.413 INFO:tasks.ceph:Adding keys to all mons... 2024-03-31T15:03:09.413 DEBUG:teuthology.orchestra.run.smithi016:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-31T15:03:09.416 DEBUG:teuthology.orchestra.run.smithi089:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-31T15:03:09.460 INFO:teuthology.orchestra.run.smithi016.stdout:[mgr.y] 2024-03-31T15:03:09.461 INFO:teuthology.orchestra.run.smithi016.stdout: key = AQAOewlmq0cpLRAAKTgh04tRaHrpMhxdwgvXYA== 2024-03-31T15:03:09.461 INFO:teuthology.orchestra.run.smithi016.stdout:[mds.a] 2024-03-31T15:03:09.461 INFO:teuthology.orchestra.run.smithi016.stdout: key = AQAOewlmhp1ANRAAYS9zo+p4tLU1YK0ySOZE/Q== 2024-03-31T15:03:09.461 INFO:teuthology.orchestra.run.smithi016.stdout:[mds.c] 2024-03-31T15:03:09.461 INFO:teuthology.orchestra.run.smithi016.stdout: key = AQAPewlmjaieARAA32teg02vE/Q8RtqLtnArMw== 2024-03-31T15:03:09.461 INFO:teuthology.orchestra.run.smithi016.stdout:[osd.0] 2024-03-31T15:03:09.461 INFO:teuthology.orchestra.run.smithi016.stdout: key = AQAQewlmVP6kLhAAwFr4jwQgz1N3oIUZuJvmzw== 2024-03-31T15:03:09.461 INFO:teuthology.orchestra.run.smithi016.stdout:[osd.1] 2024-03-31T15:03:09.461 INFO:teuthology.orchestra.run.smithi016.stdout: key = AQATewlmBh0gNhAAR6LiuFypwA7d3SwKlWuI/Q== 2024-03-31T15:03:09.461 INFO:teuthology.orchestra.run.smithi016.stdout:[osd.2] 2024-03-31T15:03:09.461 INFO:teuthology.orchestra.run.smithi016.stdout: key = AQAXewlmtQwhARAACbzc7Woo9BoVFPfK3Pns+g== 2024-03-31T15:03:09.462 INFO:teuthology.orchestra.run.smithi016.stdout:[osd.3] 2024-03-31T15:03:09.462 INFO:teuthology.orchestra.run.smithi016.stdout: key = AQAaewlmmjWABhAA/hwvARNb51nd5vsnYVD/sg== 2024-03-31T15:03:09.462 INFO:teuthology.orchestra.run.smithi016.stdout:[mgr.x] 2024-03-31T15:03:09.462 INFO:teuthology.orchestra.run.smithi016.stdout: key = AQAOewlm7UkKMRAA8hqgeV86mTCkRcxdXNdWmw== 2024-03-31T15:03:09.462 INFO:teuthology.orchestra.run.smithi016.stdout:[mds.b] 2024-03-31T15:03:09.462 INFO:teuthology.orchestra.run.smithi016.stdout: key = AQAPewlmXxupBhAAtpkt93aJOFqhc7GXLIlkBg== 2024-03-31T15:03:09.462 INFO:teuthology.orchestra.run.smithi016.stdout:[mds.d] 2024-03-31T15:03:09.462 INFO:teuthology.orchestra.run.smithi016.stdout: key = AQAPewlmo+4PDhAAmJD0siZB3LjWkvdMq1d8aw== 2024-03-31T15:03:09.462 INFO:teuthology.orchestra.run.smithi016.stdout:[osd.4] 2024-03-31T15:03:09.462 INFO:teuthology.orchestra.run.smithi016.stdout: key = AQAgewlmby07CxAAChxE/qPuPboIlg3fDxSOeA== 2024-03-31T15:03:09.462 INFO:teuthology.orchestra.run.smithi016.stdout:[osd.5] 2024-03-31T15:03:09.462 INFO:teuthology.orchestra.run.smithi016.stdout: key = AQAjewlmlD8yExAA9pgP0heBywrgqc39iU8rpQ== 2024-03-31T15:03:09.463 INFO:teuthology.orchestra.run.smithi016.stdout:[osd.6] 2024-03-31T15:03:09.463 INFO:teuthology.orchestra.run.smithi016.stdout: key = AQAmewlmzAsTGBAALMjVcQ08sIiTDFaBcq8L6A== 2024-03-31T15:03:09.463 INFO:teuthology.orchestra.run.smithi016.stdout:[osd.7] 2024-03-31T15:03:09.463 INFO:teuthology.orchestra.run.smithi016.stdout: key = AQApewlmb+CEHRAAXE+iBumDsb2c4zH3C8f6YA== 2024-03-31T15:03:09.463 INFO:teuthology.orchestra.run.smithi016.stdout:[client.0] 2024-03-31T15:03:09.463 INFO:teuthology.orchestra.run.smithi016.stdout: key = AQAPewlmlMIDEhAAF0Ll1HcrWHmH6xDKq2Qq9A== 2024-03-31T15:03:09.463 INFO:teuthology.orchestra.run.smithi016.stdout:[client.2] 2024-03-31T15:03:09.463 INFO:teuthology.orchestra.run.smithi016.stdout: key = AQAPewlmQapPGBAAbwXSlCi+zyQ9118WyAJmrQ== 2024-03-31T15:03:09.463 INFO:teuthology.orchestra.run.smithi016.stdout:[client.1] 2024-03-31T15:03:09.463 INFO:teuthology.orchestra.run.smithi016.stdout: key = AQAPewlmdsvqGxAAaVdwWwri0IE/2sg7psilWw== 2024-03-31T15:03:09.463 INFO:teuthology.orchestra.run.smithi016.stdout:[client.3] 2024-03-31T15:03:09.463 INFO:teuthology.orchestra.run.smithi016.stdout: key = AQAPewlmCw7qIRAAEeIjF85JbyfJ6d3FtMOVhQ== 2024-03-31T15:03:09.474 INFO:teuthology.orchestra.run.smithi089.stdout:[mgr.y] 2024-03-31T15:03:09.474 INFO:teuthology.orchestra.run.smithi089.stdout: key = AQAOewlmq0cpLRAAKTgh04tRaHrpMhxdwgvXYA== 2024-03-31T15:03:09.475 INFO:teuthology.orchestra.run.smithi089.stdout:[mds.a] 2024-03-31T15:03:09.475 INFO:teuthology.orchestra.run.smithi089.stdout: key = AQAOewlmhp1ANRAAYS9zo+p4tLU1YK0ySOZE/Q== 2024-03-31T15:03:09.475 INFO:teuthology.orchestra.run.smithi089.stdout:[mds.c] 2024-03-31T15:03:09.475 INFO:teuthology.orchestra.run.smithi089.stdout: key = AQAPewlmjaieARAA32teg02vE/Q8RtqLtnArMw== 2024-03-31T15:03:09.475 INFO:teuthology.orchestra.run.smithi089.stdout:[osd.0] 2024-03-31T15:03:09.475 INFO:teuthology.orchestra.run.smithi089.stdout: key = AQAQewlmVP6kLhAAwFr4jwQgz1N3oIUZuJvmzw== 2024-03-31T15:03:09.475 INFO:teuthology.orchestra.run.smithi089.stdout:[osd.1] 2024-03-31T15:03:09.475 INFO:teuthology.orchestra.run.smithi089.stdout: key = AQATewlmBh0gNhAAR6LiuFypwA7d3SwKlWuI/Q== 2024-03-31T15:03:09.475 INFO:teuthology.orchestra.run.smithi089.stdout:[osd.2] 2024-03-31T15:03:09.475 INFO:teuthology.orchestra.run.smithi089.stdout: key = AQAXewlmtQwhARAACbzc7Woo9BoVFPfK3Pns+g== 2024-03-31T15:03:09.475 INFO:teuthology.orchestra.run.smithi089.stdout:[osd.3] 2024-03-31T15:03:09.475 INFO:teuthology.orchestra.run.smithi089.stdout: key = AQAaewlmmjWABhAA/hwvARNb51nd5vsnYVD/sg== 2024-03-31T15:03:09.475 INFO:teuthology.orchestra.run.smithi089.stdout:[mgr.x] 2024-03-31T15:03:09.475 INFO:teuthology.orchestra.run.smithi089.stdout: key = AQAOewlm7UkKMRAA8hqgeV86mTCkRcxdXNdWmw== 2024-03-31T15:03:09.475 INFO:teuthology.orchestra.run.smithi089.stdout:[mds.b] 2024-03-31T15:03:09.476 INFO:teuthology.orchestra.run.smithi089.stdout: key = AQAPewlmXxupBhAAtpkt93aJOFqhc7GXLIlkBg== 2024-03-31T15:03:09.476 INFO:teuthology.orchestra.run.smithi089.stdout:[mds.d] 2024-03-31T15:03:09.476 INFO:teuthology.orchestra.run.smithi089.stdout: key = AQAPewlmo+4PDhAAmJD0siZB3LjWkvdMq1d8aw== 2024-03-31T15:03:09.476 INFO:teuthology.orchestra.run.smithi089.stdout:[osd.4] 2024-03-31T15:03:09.476 INFO:teuthology.orchestra.run.smithi089.stdout: key = AQAgewlmby07CxAAChxE/qPuPboIlg3fDxSOeA== 2024-03-31T15:03:09.476 INFO:teuthology.orchestra.run.smithi089.stdout:[osd.5] 2024-03-31T15:03:09.476 INFO:teuthology.orchestra.run.smithi089.stdout: key = AQAjewlmlD8yExAA9pgP0heBywrgqc39iU8rpQ== 2024-03-31T15:03:09.476 INFO:teuthology.orchestra.run.smithi089.stdout:[osd.6] 2024-03-31T15:03:09.476 INFO:teuthology.orchestra.run.smithi089.stdout: key = AQAmewlmzAsTGBAALMjVcQ08sIiTDFaBcq8L6A== 2024-03-31T15:03:09.476 INFO:teuthology.orchestra.run.smithi089.stdout:[osd.7] 2024-03-31T15:03:09.476 INFO:teuthology.orchestra.run.smithi089.stdout: key = AQApewlmb+CEHRAAXE+iBumDsb2c4zH3C8f6YA== 2024-03-31T15:03:09.476 INFO:teuthology.orchestra.run.smithi089.stdout:[client.0] 2024-03-31T15:03:09.476 INFO:teuthology.orchestra.run.smithi089.stdout: key = AQAPewlmlMIDEhAAF0Ll1HcrWHmH6xDKq2Qq9A== 2024-03-31T15:03:09.476 INFO:teuthology.orchestra.run.smithi089.stdout:[client.2] 2024-03-31T15:03:09.477 INFO:teuthology.orchestra.run.smithi089.stdout: key = AQAPewlmQapPGBAAbwXSlCi+zyQ9118WyAJmrQ== 2024-03-31T15:03:09.477 INFO:teuthology.orchestra.run.smithi089.stdout:[client.1] 2024-03-31T15:03:09.477 INFO:teuthology.orchestra.run.smithi089.stdout: key = AQAPewlmdsvqGxAAaVdwWwri0IE/2sg7psilWw== 2024-03-31T15:03:09.477 INFO:teuthology.orchestra.run.smithi089.stdout:[client.3] 2024-03-31T15:03:09.477 INFO:teuthology.orchestra.run.smithi089.stdout: key = AQAPewlmCw7qIRAAEeIjF85JbyfJ6d3FtMOVhQ== 2024-03-31T15:03:09.478 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-31T15:03:09.512 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-31T15:03:09.577 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-31T15:03:09.581 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-31T15:03:09.640 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-31T15:03:09.643 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-31T15:03:09.700 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-31T15:03:09.703 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-31T15:03:09.756 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-31T15:03:09.759 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-31T15:03:09.813 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-31T15:03:09.816 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-31T15:03:09.870 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-31T15:03:09.874 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-31T15:03:09.927 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-31T15:03:09.931 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-31T15:03:09.983 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-31T15:03:09.987 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-31T15:03:10.043 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-31T15:03:10.048 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-31T15:03:10.104 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-31T15:03:10.108 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-31T15:03:10.162 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-31T15:03:10.166 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-31T15:03:10.219 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-31T15:03:10.223 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-31T15:03:10.278 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-31T15:03:10.284 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-31T15:03:10.336 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-31T15:03:10.339 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-31T15:03:10.395 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-31T15:03:10.398 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-31T15:03:10.453 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-31T15:03:10.457 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-31T15:03:10.513 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-31T15:03:10.516 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-31T15:03:10.571 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-03-31T15:03:10.571 DEBUG:teuthology.orchestra.run.smithi016:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-03-31T15:03:10.590 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-31T15:03:10.699 DEBUG:teuthology.orchestra.run.smithi016:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-03-31T15:03:10.719 DEBUG:teuthology.orchestra.run.smithi089:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-03-31T15:03:10.738 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-31T15:03:10.851 DEBUG:teuthology.orchestra.run.smithi089:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-03-31T15:03:10.870 DEBUG:teuthology.orchestra.run.smithi089:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-03-31T15:03:10.930 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-31T15:03:11.037 DEBUG:teuthology.orchestra.run.smithi089:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-03-31T15:03:11.056 DEBUG:teuthology.orchestra.run.smithi016:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-31T15:03:11.059 DEBUG:teuthology.orchestra.run.smithi089:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-31T15:03:11.103 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-03-31T15:03:11.104 INFO:tasks.ceph.mon.a:Restarting daemon 2024-03-31T15:03:11.104 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-03-31T15:03:11.113 INFO:tasks.ceph.mon.a:Started 2024-03-31T15:03:11.113 INFO:tasks.ceph.mon.b:Restarting daemon 2024-03-31T15:03:11.113 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-03-31T15:03:11.147 INFO:tasks.ceph.mon.b:Started 2024-03-31T15:03:11.147 INFO:tasks.ceph.mon.c:Restarting daemon 2024-03-31T15:03:11.147 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-03-31T15:03:11.150 INFO:tasks.ceph.mon.c:Started 2024-03-31T15:03:11.150 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-03-31T15:03:11.150 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-03-31T15:03:11.151 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2024-03-31T15:03:11.154 INFO:tasks.ceph.mgr.y:Started 2024-03-31T15:03:11.154 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-03-31T15:03:11.154 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-03-31T15:03:11.157 INFO:tasks.ceph.mgr.x:Started 2024-03-31T15:03:11.157 INFO:tasks.ceph:Setting crush tunables to default 2024-03-31T15:03:11.157 DEBUG:teuthology.orchestra.run.smithi016:> sudo ceph --cluster ceph osd crush tunables default 2024-03-31T15:03:11.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:11.302+0000 7f7631c00640 1 Processor -- start 2024-03-31T15:03:11.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:11.302+0000 7f7631c00640 1 -- start start 2024-03-31T15:03:11.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:11.306+0000 7f7631c00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f762c0cdc30 0x7f762c0ce010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:11.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:11.306+0000 7f7631c00640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f762c0ce550 con 0x7f762c0d0880 2024-03-31T15:03:11.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:11.306+0000 7f7631c00640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f762c070520 con 0x7f762c15c3d0 2024-03-31T15:03:11.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:11.306+0000 7f7631c00640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f762c0cfda0 con 0x7f762c0cdc30 2024-03-31T15:03:11.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:11.306+0000 7f762aa00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f762c0cdc30 0x7f762c0ce010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:11.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:11.306+0000 7f762aa00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f762c0cdc30 0x7f762c0ce010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.16:34776/0 (socket says 172.21.15.16:34776) 2024-03-31T15:03:11.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:11.306+0000 7f762aa00640 1 -- 172.21.15.16:0/2874246232 learned_addr learned my addr 172.21.15.16:0/2874246232 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:03:11.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:11.306+0000 7f762aa00640 1 -- 172.21.15.16:0/2874246232 >> v2:172.21.15.89:3300/0 conn(0x7f762c0cdc30 msgr2=0x7f762c0ce010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-03-31T15:03:11.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:11.306+0000 7f762aa00640 1 -- 172.21.15.16:0/2874246232 >> v2:172.21.15.89:3300/0 conn(0x7f762c0cdc30 msgr2=0x7f762c0ce010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-31T15:03:11.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:11.306+0000 7f762aa00640 1 --2- 172.21.15.16:0/2874246232 >> v2:172.21.15.89:3300/0 conn(0x7f762c0cdc30 0x7f762c0ce010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-31T15:03:11.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:11.306+0000 7f762aa00640 1 --2- 172.21.15.16:0/2874246232 >> v2:172.21.15.89:3300/0 conn(0x7f762c0cdc30 0x7f762c0ce010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-31T15:03:11.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:11.450+0000 7f762a000640 1 -- 172.21.15.16:0/2874246232 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 469554952 0 0) 0x7f762c0ce550 con 0x7f762c0d0880 2024-03-31T15:03:11.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:11.450+0000 7f762a000640 1 -- 172.21.15.16:0/2874246232 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7610005740 con 0x7f762c0d0880 2024-03-31T15:03:11.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:11.454+0000 7f762a000640 1 -- 172.21.15.16:0/2874246232 <== mon.1 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 322982293 0 0) 0x7f762c070520 con 0x7f762c15c3d0 2024-03-31T15:03:11.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:11.454+0000 7f762a000640 1 -- 172.21.15.16:0/2874246232 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f762c0ce550 con 0x7f762c15c3d0 2024-03-31T15:03:11.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:11.454+0000 7f762a000640 1 -- 172.21.15.16:0/2874246232 <== mon.1 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 887872982 0 0) 0x7f762c0ce550 con 0x7f762c15c3d0 2024-03-31T15:03:11.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:11.454+0000 7f762a000640 1 -- 172.21.15.16:0/2874246232 >> v2:172.21.15.89:3300/0 conn(0x7f762c0cdc30 msgr2=0x7f762c0ce010 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-31T15:03:11.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:11.454+0000 7f762a000640 1 --2- 172.21.15.16:0/2874246232 >> v2:172.21.15.89:3300/0 conn(0x7f762c0cdc30 0x7f762c0ce010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:11.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:11.454+0000 7f762a000640 1 -- 172.21.15.16:0/2874246232 >> v1:172.21.15.16:6789/0 conn(0x7f762c0d0880 legacy=0x7f762c0b4250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:11.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:11.454+0000 7f762a000640 1 -- 172.21.15.16:0/2874246232 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f762c0d0080 con 0x7f762c15c3d0 2024-03-31T15:03:11.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:11.454+0000 7f762a000640 1 -- 172.21.15.16:0/2874246232 <== mon.1 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 214807623 0 0) 0x7f7614001680 con 0x7f762c15c3d0 2024-03-31T15:03:11.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:11.454+0000 7f762a000640 1 -- 172.21.15.16:0/2874246232 >> v1:172.21.15.89:6789/0 conn(0x7f762c15c3d0 legacy=0x7f762c06fc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:11.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:11.454+0000 7f762a000640 1 --2- 172.21.15.16:0/2874246232 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f7610005d60 0x7f7610006160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:11.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:11.454+0000 7f762a000640 1 --2- 172.21.15.16:0/2874246232 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7610006a20 0x7f7610006ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:11.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:11.454+0000 7f762a000640 1 --2- 172.21.15.16:0/2874246232 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f76100073e0 0x7f761000dbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:11.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:11.454+0000 7f762a000640 1 -- 172.21.15.16:0/2874246232 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f762c0cfda0 con 0x7f7610006a20 2024-03-31T15:03:11.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:11.454+0000 7f762a000640 1 -- 172.21.15.16:0/2874246232 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f761000e2e0 con 0x7f76100073e0 2024-03-31T15:03:11.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:11.454+0000 7f762a000640 1 -- 172.21.15.16:0/2874246232 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f761000e4e0 con 0x7f7610005d60 2024-03-31T15:03:11.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:11.454+0000 7f762b400640 1 --2- 172.21.15.16:0/2874246232 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f76100073e0 0x7f761000dbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:11.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:11.454+0000 7f762aa00640 1 --2- 172.21.15.16:0/2874246232 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f7610005d60 0x7f7610006160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:11.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:11.454+0000 7f7622000640 1 --2- 172.21.15.16:0/2874246232 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7610006a20 0x7f7610006ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:11.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:11.454+0000 7f762b400640 1 -- 172.21.15.16:0/2874246232 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f7610005d60 msgr2=0x7f7610006160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:03:11.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:11.454+0000 7f762b400640 1 --2- 172.21.15.16:0/2874246232 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f7610005d60 0x7f7610006160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:11.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:11.454+0000 7f762b400640 1 -- 172.21.15.16:0/2874246232 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7610006a20 msgr2=0x7f7610006ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:03:11.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:11.454+0000 7f762b400640 1 --2- 172.21.15.16:0/2874246232 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7610006a20 0x7f7610006ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:11.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:11.454+0000 7f762b400640 1 -- 172.21.15.16:0/2874246232 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f762c0d0080 con 0x7f76100073e0 2024-03-31T15:03:11.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:11.454+0000 7f7622000640 1 --2- 172.21.15.16:0/2874246232 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7610006a20 0x7f7610006ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:03:11.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:11.458+0000 7f762b400640 1 -- 172.21.15.16:0/2874246232 reap_dead start 2024-03-31T15:03:11.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:11.458+0000 7f762b400640 1 --2- 172.21.15.16:0/2874246232 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f76100073e0 0x7f761000dbe0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f7624002020 tx=0x7f762401c920 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d39fcc1ff1a8cad0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:03:11.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:11.458+0000 7f762a000640 1 -- 172.21.15.16:0/2874246232 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7614007ce0 con 0x7f76100073e0 2024-03-31T15:03:11.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:11.458+0000 7f762a000640 1 -- 172.21.15.16:0/2874246232 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7624003b80 con 0x7f76100073e0 2024-03-31T15:03:11.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:11.458+0000 7f762a000640 1 -- 172.21.15.16:0/2874246232 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7624001c40 con 0x7f76100073e0 2024-03-31T15:03:11.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:11.458+0000 7f7631c00640 1 -- 172.21.15.16:0/2874246232 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f76100073e0 msgr2=0x7f761000dbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:11.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:11.458+0000 7f7631c00640 1 --2- 172.21.15.16:0/2874246232 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f76100073e0 0x7f761000dbe0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f7624002020 tx=0x7f762401c920 comp rx=0 tx=0).stop 2024-03-31T15:03:11.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:11.458+0000 7f7631c00640 1 -- 172.21.15.16:0/2874246232 shutdown_connections 2024-03-31T15:03:11.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:11.458+0000 7f7631c00640 1 --2- 172.21.15.16:0/2874246232 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f76100073e0 0x7f761000dbe0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:11.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:11.458+0000 7f7631c00640 1 -- 172.21.15.16:0/2874246232 >> 172.21.15.16:0/2874246232 conn(0x7f762c06e670 msgr2=0x7f762c06ea50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:03:11.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:11.458+0000 7f7631c00640 1 -- 172.21.15.16:0/2874246232 shutdown_connections 2024-03-31T15:03:11.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:11.458+0000 7f7631c00640 1 -- 172.21.15.16:0/2874246232 wait complete. 2024-03-31T15:03:11.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:11.462+0000 7f7631c00640 1 Processor -- start 2024-03-31T15:03:11.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:11.462+0000 7f7631c00640 1 -- start start 2024-03-31T15:03:11.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:11.462+0000 7f7631c00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f76100073e0 0x7f762c1efee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:11.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:11.462+0000 7f7631c00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f762c1555e0 0x7f762c0b4db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:11.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:11.462+0000 7f7631c00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f762c1584d0 0x7f762c0b52f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:11.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:11.462+0000 7f7631c00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f762c158f80 con 0x7f762c1584d0 2024-03-31T15:03:11.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:11.462+0000 7f7631c00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f762c070800 con 0x7f762c1555e0 2024-03-31T15:03:11.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:11.462+0000 7f7631c00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f762c159100 con 0x7f76100073e0 2024-03-31T15:03:11.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:11.462+0000 7f762aa00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f76100073e0 0x7f762c1efee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:11.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:11.462+0000 7f762b400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f762c1584d0 0x7f762c0b52f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:11.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:11.462+0000 7f762aa00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f76100073e0 0x7f762c1efee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3301/0 says I am v2:172.21.15.16:49802/0 (socket says 172.21.15.16:49802) 2024-03-31T15:03:11.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:11.462+0000 7f762aa00640 1 -- 172.21.15.16:0/1855727362 learned_addr learned my addr 172.21.15.16:0/1855727362 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:03:11.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:11.462+0000 7f7622000640 1 --2- 172.21.15.16:0/1855727362 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f762c1555e0 0x7f762c0b4db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:11.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:11.462+0000 7f762aa00640 1 -- 172.21.15.16:0/1855727362 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f762c1555e0 msgr2=0x7f762c0b4db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:11.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:11.462+0000 7f762aa00640 1 --2- 172.21.15.16:0/1855727362 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f762c1555e0 0x7f762c0b4db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:11.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:11.462+0000 7f762aa00640 1 -- 172.21.15.16:0/1855727362 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f762c1584d0 msgr2=0x7f762c0b52f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:11.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:11.462+0000 7f762aa00640 1 --2- 172.21.15.16:0/1855727362 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f762c1584d0 0x7f762c0b52f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:11.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:11.462+0000 7f762aa00640 1 -- 172.21.15.16:0/1855727362 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f762c0c6fb0 con 0x7f76100073e0 2024-03-31T15:03:11.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:11.462+0000 7f762aa00640 1 --2- 172.21.15.16:0/1855727362 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f76100073e0 0x7f762c1efee0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f761c00f4c0 tx=0x7f761c00f8c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:03:11.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:11.466+0000 7f7623400640 1 -- 172.21.15.16:0/1855727362 <== mon.2 v2:172.21.15.89:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f761c018030 con 0x7f76100073e0 2024-03-31T15:03:11.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:11.466+0000 7f7623400640 1 -- 172.21.15.16:0/1855727362 <== mon.2 v2:172.21.15.89:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f761c014690 con 0x7f76100073e0 2024-03-31T15:03:11.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:11.466+0000 7f7623400640 1 -- 172.21.15.16:0/1855727362 <== mon.2 v2:172.21.15.89:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f761c014a80 con 0x7f76100073e0 2024-03-31T15:03:11.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:11.466+0000 7f7631c00640 1 -- 172.21.15.16:0/1855727362 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f762c154820 con 0x7f76100073e0 2024-03-31T15:03:11.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:11.466+0000 7f7631c00640 1 -- 172.21.15.16:0/1855727362 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f762c154de0 con 0x7f76100073e0 2024-03-31T15:03:11.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:11.466+0000 7f7631c00640 1 -- 172.21.15.16:0/1855727362 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f762c069950 con 0x7f76100073e0 2024-03-31T15:03:11.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:11.470+0000 7f7623400640 1 -- 172.21.15.16:0/1855727362 <== mon.2 v2:172.21.15.89:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f761c01b070 con 0x7f76100073e0 2024-03-31T15:03:11.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:11.470+0000 7f7623400640 1 -- 172.21.15.16:0/1855727362 <== mon.2 v2:172.21.15.89:3301/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f761c01d750 con 0x7f76100073e0 2024-03-31T15:03:11.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:11.490+0000 7f7623400640 1 -- 172.21.15.16:0/1855727362 <== mon.2 v2:172.21.15.89:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f761c027d80 con 0x7f76100073e0 2024-03-31T15:03:11.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:11.550+0000 7f7631c00640 1 -- 172.21.15.16:0/1855727362 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) -- 0x7f762c070c90 con 0x7f76100073e0 2024-03-31T15:03:11.554 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:11.558+0000 7f7623400640 1 -- 172.21.15.16:0/1855727362 <== mon.2 v2:172.21.15.89:3301/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) ==== 124+0+0 (secure 0 0 0) 0x7f761c014c20 con 0x7f76100073e0 2024-03-31T15:03:11.555 INFO:teuthology.orchestra.run.smithi016.stderr:adjusted tunables profile to default 2024-03-31T15:03:11.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:11.562+0000 7f7631c00640 1 -- 172.21.15.16:0/1855727362 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f76100073e0 msgr2=0x7f762c1efee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:11.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:11.562+0000 7f7631c00640 1 --2- 172.21.15.16:0/1855727362 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f76100073e0 0x7f762c1efee0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f761c00f4c0 tx=0x7f761c00f8c0 comp rx=0 tx=0).stop 2024-03-31T15:03:11.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:11.562+0000 7f7631c00640 1 -- 172.21.15.16:0/1855727362 shutdown_connections 2024-03-31T15:03:11.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:11.562+0000 7f7631c00640 1 --2- 172.21.15.16:0/1855727362 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f762c1584d0 0x7f762c0b52f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:11.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:11.562+0000 7f7631c00640 1 --2- 172.21.15.16:0/1855727362 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f762c1555e0 0x7f762c0b4db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:11.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:11.562+0000 7f7631c00640 1 --2- 172.21.15.16:0/1855727362 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f76100073e0 0x7f762c1efee0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:11.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:11.562+0000 7f7631c00640 1 -- 172.21.15.16:0/1855727362 >> 172.21.15.16:0/1855727362 conn(0x7f762c06e670 msgr2=0x7f762c1592e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:03:11.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:11.562+0000 7f7631c00640 1 -- 172.21.15.16:0/1855727362 shutdown_connections 2024-03-31T15:03:11.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:11.562+0000 7f7631c00640 1 -- 172.21.15.16:0/1855727362 wait complete. 2024-03-31T15:03:11.574 INFO:tasks.ceph:check_enable_crimson: False 2024-03-31T15:03:11.574 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-03-31T15:03:11.574 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-03-31T15:03:11.575 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-03-31T15:03:11.595 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-03-31T15:03:11.595 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-03-31T15:03:11.659 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-03-31T15:03:11.660 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-03-31T15:03:11.686 INFO:tasks.ceph.mgr.y.smithi016.stderr:2024-03-31T15:03:11.690+0000 7f2c2622a180 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-31T15:03:11.707 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-03-31T15:03:11.707 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-03-31T15:03:11.714 INFO:tasks.ceph.mgr.x.smithi089.stderr:2024-03-31T15:03:11.723+0000 7f078c909180 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-31T15:03:11.771 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-03-31T15:03:11.771 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-03-31T15:03:11.792 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-03-31T15:03:11.792 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-03-31T15:03:11.851 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-03-31T15:03:11.851 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-03-31T15:03:11.917 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-03-31T15:03:11.917 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-03-31T15:03:11.979 DEBUG:teuthology.orchestra.run.smithi089:> sudo ceph --cluster ceph osd new 92a66450-ef19-40d8-ac3d-d760b23b0bda 0 2024-03-31T15:03:12.155 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.163+0000 7f178ea00640 1 Processor -- start 2024-03-31T15:03:12.156 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.163+0000 7f178ea00640 1 -- start start 2024-03-31T15:03:12.157 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.163+0000 7f178ea00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f1788070030 0x7f1788143c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:12.157 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.163+0000 7f178ea00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f1788141590 0x7f1788144170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:12.158 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.163+0000 7f178ea00640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f17881446b0 con 0x7f17881411b0 2024-03-31T15:03:12.158 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.163+0000 7f178ea00640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f178815f910 con 0x7f1788070030 2024-03-31T15:03:12.158 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.163+0000 7f178ea00640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f1788162910 con 0x7f1788141590 2024-03-31T15:03:12.158 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.167+0000 7f178ca00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f1788141590 0x7f1788144170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:12.158 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.167+0000 7f178ca00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f1788141590 0x7f1788144170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:46038/0 (socket says 172.21.15.89:46038) 2024-03-31T15:03:12.158 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.167+0000 7f178ca00640 1 -- 172.21.15.89:0/3112701727 learned_addr learned my addr 172.21.15.89:0/3112701727 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-03-31T15:03:12.158 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.167+0000 7f1787e00640 1 --1- 172.21.15.89:0/3112701727 >> v1:172.21.15.89:6789/0 conn(0x7f17881411b0 0x7f178806f920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:56068/0 (socket says 172.21.15.89:56068) 2024-03-31T15:03:12.158 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.167+0000 7f177fe00640 1 --2- 172.21.15.89:0/3112701727 >> v2:172.21.15.16:3300/0 conn(0x7f1788070030 0x7f1788143c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:12.158 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.167+0000 7f178ca00640 1 -- 172.21.15.89:0/3112701727 >> v2:172.21.15.16:3300/0 conn(0x7f1788070030 msgr2=0x7f1788143c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:03:12.158 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.167+0000 7f178ca00640 1 --2- 172.21.15.89:0/3112701727 >> v2:172.21.15.16:3300/0 conn(0x7f1788070030 0x7f1788143c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:12.158 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.167+0000 7f178ca00640 1 -- 172.21.15.89:0/3112701727 >> v1:172.21.15.89:6789/0 conn(0x7f17881411b0 legacy=0x7f178806f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:12.158 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.167+0000 7f178ca00640 1 -- 172.21.15.89:0/3112701727 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1788144980 con 0x7f1788141590 2024-03-31T15:03:12.160 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.167+0000 7f178ca00640 1 --2- 172.21.15.89:0/3112701727 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1788141590 0x7f1788144170 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f1780009e50 tx=0x7f178001c9c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=739de90639e56e38 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:03:12.160 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.167+0000 7f1787400640 1 -- 172.21.15.89:0/3112701727 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17800a5880 con 0x7f1788141590 2024-03-31T15:03:12.161 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.167+0000 7f1787400640 1 -- 172.21.15.89:0/3112701727 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f17800a5a20 con 0x7f1788141590 2024-03-31T15:03:12.161 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.167+0000 7f1787400640 1 -- 172.21.15.89:0/3112701727 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17800a5df0 con 0x7f1788141590 2024-03-31T15:03:12.162 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.171+0000 7f178ea00640 1 -- 172.21.15.89:0/3112701727 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1788141590 msgr2=0x7f1788144170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:12.162 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.171+0000 7f178ea00640 1 --2- 172.21.15.89:0/3112701727 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1788141590 0x7f1788144170 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f1780009e50 tx=0x7f178001c9c0 comp rx=0 tx=0).stop 2024-03-31T15:03:12.162 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.171+0000 7f178ea00640 1 -- 172.21.15.89:0/3112701727 shutdown_connections 2024-03-31T15:03:12.162 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.171+0000 7f178ea00640 1 --2- 172.21.15.89:0/3112701727 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1788141590 0x7f1788144170 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:12.162 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.171+0000 7f178ea00640 1 --2- 172.21.15.89:0/3112701727 >> v2:172.21.15.16:3300/0 conn(0x7f1788070030 0x7f1788143c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:12.162 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.171+0000 7f178ea00640 1 -- 172.21.15.89:0/3112701727 >> 172.21.15.89:0/3112701727 conn(0x7f1788067930 msgr2=0x7f17880b3ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:03:12.163 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.171+0000 7f178ea00640 1 -- 172.21.15.89:0/3112701727 shutdown_connections 2024-03-31T15:03:12.163 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.171+0000 7f178ea00640 1 -- 172.21.15.89:0/3112701727 wait complete. 2024-03-31T15:03:12.163 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.171+0000 7f178ea00640 1 Processor -- start 2024-03-31T15:03:12.163 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.171+0000 7f178ea00640 1 -- start start 2024-03-31T15:03:12.167 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.171+0000 7f178ea00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1788070030 0x7f178813f9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:12.167 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.171+0000 7f178ea00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f17881411b0 0x7f17881599c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:12.167 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.171+0000 7f178ea00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1788141590 0x7f17881437c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:12.167 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.171+0000 7f178ea00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f178815f110 con 0x7f1788141590 2024-03-31T15:03:12.167 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.171+0000 7f178ea00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f1788162d80 con 0x7f1788070030 2024-03-31T15:03:12.167 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.171+0000 7f178ea00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f178815f290 con 0x7f17881411b0 2024-03-31T15:03:12.168 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.171+0000 7f1787e00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1788070030 0x7f178813f9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:12.168 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.171+0000 7f1787e00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1788070030 0x7f178813f9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:46054/0 (socket says 172.21.15.89:46054) 2024-03-31T15:03:12.168 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.171+0000 7f177fe00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f17881411b0 0x7f17881599c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:12.168 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.171+0000 7f1787e00640 1 -- 172.21.15.89:0/2126802490 learned_addr learned my addr 172.21.15.89:0/2126802490 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-03-31T15:03:12.168 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.171+0000 7f178ca00640 1 --2- 172.21.15.89:0/2126802490 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1788141590 0x7f17881437c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:12.168 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.171+0000 7f1787e00640 1 -- 172.21.15.89:0/2126802490 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f17881411b0 msgr2=0x7f17881599c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:12.169 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.171+0000 7f1787e00640 1 --2- 172.21.15.89:0/2126802490 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f17881411b0 0x7f17881599c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:12.169 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.171+0000 7f1787e00640 1 -- 172.21.15.89:0/2126802490 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1788141590 msgr2=0x7f17881437c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:12.169 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.171+0000 7f1787e00640 1 --2- 172.21.15.89:0/2126802490 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1788141590 0x7f17881437c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:12.169 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.175+0000 7f1787e00640 1 -- 172.21.15.89:0/2126802490 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1788156880 con 0x7f1788070030 2024-03-31T15:03:12.169 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.175+0000 7f1787e00640 1 --2- 172.21.15.89:0/2126802490 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1788070030 0x7f178813f9a0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f177800c450 tx=0x7f177800c910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:03:12.169 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.175+0000 7f1784c00640 1 -- 172.21.15.89:0/2126802490 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1778018070 con 0x7f1788070030 2024-03-31T15:03:12.169 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.175+0000 7f1784c00640 1 -- 172.21.15.89:0/2126802490 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1778007480 con 0x7f1788070030 2024-03-31T15:03:12.169 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.175+0000 7f178ea00640 1 -- 172.21.15.89:0/2126802490 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1788156b60 con 0x7f1788070030 2024-03-31T15:03:12.169 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.175+0000 7f1784c00640 1 -- 172.21.15.89:0/2126802490 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17780047f0 con 0x7f1788070030 2024-03-31T15:03:12.169 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.175+0000 7f178ea00640 1 -- 172.21.15.89:0/2126802490 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1788157400 con 0x7f1788070030 2024-03-31T15:03:12.169 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.175+0000 7f1784c00640 1 -- 172.21.15.89:0/2126802490 <== mon.1 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f177801b070 con 0x7f1788070030 2024-03-31T15:03:12.169 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.175+0000 7f1784c00640 1 -- 172.21.15.89:0/2126802490 <== mon.1 v2:172.21.15.89:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 725+0+0 (secure 0 0 0) 0x7f1778012440 con 0x7f1788070030 2024-03-31T15:03:12.169 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.175+0000 7f178ea00640 1 -- 172.21.15.89:0/2126802490 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f178813ff40 con 0x7f1788070030 2024-03-31T15:03:12.170 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.179+0000 7f1784c00640 1 -- 172.21.15.89:0/2126802490 <== mon.1 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f177801d440 con 0x7f1788070030 2024-03-31T15:03:12.211 INFO:tasks.ceph.mgr.y.smithi016.stderr:2024-03-31T15:03:12.214+0000 7f2c2622a180 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-31T15:03:12.235 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.243+0000 7f178ea00640 1 -- 172.21.15.89:0/2126802490 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd new", "uuid": "92a66450-ef19-40d8-ac3d-d760b23b0bda", "id": 0} v 0) -- 0x7f17880b4a20 con 0x7f1788070030 2024-03-31T15:03:12.242 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.251+0000 7f1784c00640 1 -- 172.21.15.89:0/2126802490 <== mon.1 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "92a66450-ef19-40d8-ac3d-d760b23b0bda", "id": 0}]=0 v3) ==== 112+0+2 (secure 0 0 0) 0x7f177801d6f0 con 0x7f1788070030 2024-03-31T15:03:12.242 INFO:teuthology.orchestra.run.smithi089.stdout:0 2024-03-31T15:03:12.243 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.251+0000 7f178ea00640 1 -- 172.21.15.89:0/2126802490 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1788070030 msgr2=0x7f178813f9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:12.243 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.251+0000 7f178ea00640 1 --2- 172.21.15.89:0/2126802490 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1788070030 0x7f178813f9a0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f177800c450 tx=0x7f177800c910 comp rx=0 tx=0).stop 2024-03-31T15:03:12.243 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.251+0000 7f178ea00640 1 -- 172.21.15.89:0/2126802490 shutdown_connections 2024-03-31T15:03:12.244 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.251+0000 7f178ea00640 1 --2- 172.21.15.89:0/2126802490 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1788141590 0x7f17881437c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:12.244 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.251+0000 7f178ea00640 1 --2- 172.21.15.89:0/2126802490 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f17881411b0 0x7f17881599c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:12.244 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.251+0000 7f178ea00640 1 --2- 172.21.15.89:0/2126802490 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1788070030 0x7f178813f9a0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:12.244 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.251+0000 7f178ea00640 1 -- 172.21.15.89:0/2126802490 >> 172.21.15.89:0/2126802490 conn(0x7f1788067930 msgr2=0x7f17881612f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:03:12.244 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.251+0000 7f178ea00640 1 -- 172.21.15.89:0/2126802490 shutdown_connections 2024-03-31T15:03:12.244 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.251+0000 7f178ea00640 1 -- 172.21.15.89:0/2126802490 wait complete. 2024-03-31T15:03:12.251 INFO:tasks.ceph.mgr.x.smithi089.stderr:2024-03-31T15:03:12.259+0000 7f078c909180 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-31T15:03:12.261 DEBUG:teuthology.orchestra.run.smithi089:> sudo ceph --cluster ceph osd new da9d1f2b-1968-4959-b843-faa231988e92 1 2024-03-31T15:03:12.395 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.403+0000 7f793e200640 1 Processor -- start 2024-03-31T15:03:12.395 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.403+0000 7f793e200640 1 -- start start 2024-03-31T15:03:12.396 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.403+0000 7f793e200640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f7938156a70 0x7f793806fdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:12.396 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.403+0000 7f793e200640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f793815c8b0 con 0x7f7938156690 2024-03-31T15:03:12.396 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.403+0000 7f793e200640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f793815cb90 con 0x7f7938070300 2024-03-31T15:03:12.397 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.403+0000 7f793e200640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f7938070740 con 0x7f7938156a70 2024-03-31T15:03:12.397 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.403+0000 7f792ea00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f7938156a70 0x7f793806fdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:12.397 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.403+0000 7f7937e00640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f7938070300 0x7f79380cf2c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:56082/0 (socket says 172.21.15.89:56082) 2024-03-31T15:03:12.397 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.403+0000 7f792ea00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f7938156a70 0x7f793806fdc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:46056/0 (socket says 172.21.15.89:46056) 2024-03-31T15:03:12.397 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.403+0000 7f7937e00640 1 -- 172.21.15.89:0/736419534 learned_addr learned my addr 172.21.15.89:0/736419534 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-03-31T15:03:12.397 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.403+0000 7f792ea00640 1 -- 172.21.15.89:0/736419534 >> v1:172.21.15.89:6789/0 conn(0x7f7938070300 legacy=0x7f79380cf2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:12.397 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.403+0000 7f792ea00640 1 -- 172.21.15.89:0/736419534 >> v1:172.21.15.16:6789/0 conn(0x7f7938156690 legacy=0x7f793806c840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:12.397 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.403+0000 7f792ea00640 1 -- 172.21.15.89:0/736419534 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f793815ce70 con 0x7f7938156a70 2024-03-31T15:03:12.400 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.407+0000 7f792ea00640 1 --2- 172.21.15.89:0/736419534 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7938156a70 0x7f793806fdc0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f79200097c0 tx=0x7f792001c4a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1ae08330a8085c63 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:03:12.400 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.407+0000 7f7936a00640 1 -- 172.21.15.89:0/736419534 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79200b1070 con 0x7f7938156a70 2024-03-31T15:03:12.400 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.407+0000 7f7936a00640 1 -- 172.21.15.89:0/736419534 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f79200ac410 con 0x7f7938156a70 2024-03-31T15:03:12.400 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.407+0000 7f7936a00640 1 -- 172.21.15.89:0/736419534 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79200ac7e0 con 0x7f7938156a70 2024-03-31T15:03:12.401 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.407+0000 7f793e200640 1 -- 172.21.15.89:0/736419534 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7938156a70 msgr2=0x7f793806fdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:12.401 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.407+0000 7f793e200640 1 --2- 172.21.15.89:0/736419534 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7938156a70 0x7f793806fdc0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f79200097c0 tx=0x7f792001c4a0 comp rx=0 tx=0).stop 2024-03-31T15:03:12.401 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.407+0000 7f793e200640 1 -- 172.21.15.89:0/736419534 shutdown_connections 2024-03-31T15:03:12.401 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.407+0000 7f793e200640 1 --2- 172.21.15.89:0/736419534 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7938156a70 0x7f793806fdc0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:12.401 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.407+0000 7f793e200640 1 -- 172.21.15.89:0/736419534 >> 172.21.15.89:0/736419534 conn(0x7f7938071e30 msgr2=0x7f7938072240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:03:12.401 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.407+0000 7f793e200640 1 -- 172.21.15.89:0/736419534 shutdown_connections 2024-03-31T15:03:12.401 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.407+0000 7f793e200640 1 -- 172.21.15.89:0/736419534 wait complete. 2024-03-31T15:03:12.402 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.411+0000 7f793e200640 1 Processor -- start 2024-03-31T15:03:12.402 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.411+0000 7f793e200640 1 -- start start 2024-03-31T15:03:12.403 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.411+0000 7f793e200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7938070300 0x7f7938159a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:12.403 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.411+0000 7f793e200640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f7938156690 0x7f793815b800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:12.403 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.411+0000 7f793e200640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7938145970 0x7f793815bd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:12.404 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.411+0000 7f793e200640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f79380d0360 con 0x7f7938070300 2024-03-31T15:03:12.404 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.411+0000 7f793e200640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f79380d01e0 con 0x7f7938145970 2024-03-31T15:03:12.404 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.411+0000 7f793e200640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f79380d04e0 con 0x7f7938156690 2024-03-31T15:03:12.404 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.411+0000 7f792ea00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f7938156690 0x7f793815b800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:12.405 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.411+0000 7f7937e00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7938145970 0x7f793815bd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:12.405 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.411+0000 7f792ea00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f7938156690 0x7f793815b800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3301/0 says I am v2:172.21.15.89:56076/0 (socket says 172.21.15.89:56076) 2024-03-31T15:03:12.405 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.411+0000 7f792ea00640 1 -- 172.21.15.89:0/1460958285 learned_addr learned my addr 172.21.15.89:0/1460958285 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-03-31T15:03:12.405 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.411+0000 7f792ea00640 1 -- 172.21.15.89:0/1460958285 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7938145970 msgr2=0x7f793815bd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:12.405 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.411+0000 7f792ea00640 1 --2- 172.21.15.89:0/1460958285 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7938145970 0x7f793815bd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:12.405 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.411+0000 7f792ea00640 1 -- 172.21.15.89:0/1460958285 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7938070300 msgr2=0x7f7938159a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:12.406 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.411+0000 7f792ea00640 1 --2- 172.21.15.89:0/1460958285 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7938070300 0x7f7938159a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:12.406 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.411+0000 7f792ea00640 1 -- 172.21.15.89:0/1460958285 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f793814ee60 con 0x7f7938156690 2024-03-31T15:03:12.406 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.411+0000 7f792ea00640 1 --2- 172.21.15.89:0/1460958285 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f7938156690 0x7f793815b800 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f79200af040 tx=0x7f7920019240 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:03:12.406 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.411+0000 7f792fe00640 1 -- 172.21.15.89:0/1460958285 <== mon.2 v2:172.21.15.89:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79200b9020 con 0x7f7938156690 2024-03-31T15:03:12.406 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.411+0000 7f792fe00640 1 -- 172.21.15.89:0/1460958285 <== mon.2 v2:172.21.15.89:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f79200ace00 con 0x7f7938156690 2024-03-31T15:03:12.406 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.411+0000 7f792fe00640 1 -- 172.21.15.89:0/1460958285 <== mon.2 v2:172.21.15.89:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79200b1040 con 0x7f7938156690 2024-03-31T15:03:12.406 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.411+0000 7f793e200640 1 -- 172.21.15.89:0/1460958285 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7938134b40 con 0x7f7938156690 2024-03-31T15:03:12.406 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.411+0000 7f793e200640 1 -- 172.21.15.89:0/1460958285 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f7938159270 con 0x7f7938156690 2024-03-31T15:03:12.406 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.415+0000 7f792fe00640 1 -- 172.21.15.89:0/1460958285 <== mon.2 v2:172.21.15.89:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f79200bc020 con 0x7f7938156690 2024-03-31T15:03:12.406 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.415+0000 7f792fe00640 1 -- 172.21.15.89:0/1460958285 <== mon.2 v2:172.21.15.89:3301/0 5 ==== osd_map(3..3 src has 1..3) ==== 840+0+0 (secure 0 0 0) 0x7f79200bebc0 con 0x7f7938156690 2024-03-31T15:03:12.407 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.415+0000 7f793e200640 1 -- 172.21.15.89:0/1460958285 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7938069950 con 0x7f7938156690 2024-03-31T15:03:12.408 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.415+0000 7f792fe00640 1 -- 172.21.15.89:0/1460958285 <== mon.2 v2:172.21.15.89:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f79200c3030 con 0x7f7938156690 2024-03-31T15:03:12.451 INFO:tasks.ceph.mgr.x.smithi089.stderr:2024-03-31T15:03:12.459+0000 7f078c909180 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-31T15:03:12.452 INFO:tasks.ceph.mgr.y.smithi016.stderr:2024-03-31T15:03:12.458+0000 7f2c2622a180 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-31T15:03:12.477 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.483+0000 7f793e200640 1 -- 172.21.15.89:0/1460958285 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_command({"prefix": "osd new", "uuid": "da9d1f2b-1968-4959-b843-faa231988e92", "id": 1} v 0) -- 0x7f79380b3850 con 0x7f7938156690 2024-03-31T15:03:12.480 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.487+0000 7f792fe00640 1 -- 172.21.15.89:0/1460958285 <== mon.2 v2:172.21.15.89:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "da9d1f2b-1968-4959-b843-faa231988e92", "id": 1}]=0 v4) ==== 112+0+2 (secure 0 0 0) 0x7f7920016090 con 0x7f7938156690 2024-03-31T15:03:12.481 INFO:teuthology.orchestra.run.smithi089.stdout:1 2024-03-31T15:03:12.482 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.491+0000 7f793e200640 1 -- 172.21.15.89:0/1460958285 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f7938156690 msgr2=0x7f793815b800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:12.482 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.491+0000 7f793e200640 1 --2- 172.21.15.89:0/1460958285 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f7938156690 0x7f793815b800 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f79200af040 tx=0x7f7920019240 comp rx=0 tx=0).stop 2024-03-31T15:03:12.482 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.491+0000 7f793e200640 1 -- 172.21.15.89:0/1460958285 shutdown_connections 2024-03-31T15:03:12.482 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.491+0000 7f793e200640 1 --2- 172.21.15.89:0/1460958285 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7938145970 0x7f793815bd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:12.482 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.491+0000 7f793e200640 1 --2- 172.21.15.89:0/1460958285 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f7938156690 0x7f793815b800 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:12.482 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.491+0000 7f793e200640 1 --2- 172.21.15.89:0/1460958285 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7938070300 0x7f7938159a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:12.482 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.491+0000 7f793e200640 1 -- 172.21.15.89:0/1460958285 >> 172.21.15.89:0/1460958285 conn(0x7f7938071e30 msgr2=0x7f793815d9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:03:12.482 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.491+0000 7f793e200640 1 -- 172.21.15.89:0/1460958285 shutdown_connections 2024-03-31T15:03:12.482 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.491+0000 7f793e200640 1 -- 172.21.15.89:0/1460958285 wait complete. 2024-03-31T15:03:12.506 DEBUG:teuthology.orchestra.run.smithi089:> sudo ceph --cluster ceph osd new 01a73662-9a21-4b1f-bcbb-42b9636a4638 2 2024-03-31T15:03:12.642 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.651+0000 7f3dc7200640 1 Processor -- start 2024-03-31T15:03:12.642 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.651+0000 7f3dc7200640 1 -- start start 2024-03-31T15:03:12.644 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.651+0000 7f3dc7200640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f3dc0070350 0x7f3dc0160860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:12.644 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.651+0000 7f3dc7200640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3dc0070a10 con 0x7f3dc015c890 2024-03-31T15:03:12.645 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.651+0000 7f3dc7200640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3dc00cd9b0 con 0x7f3dc0145d50 2024-03-31T15:03:12.645 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.651+0000 7f3dc7200640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f3dc00ce5c0 con 0x7f3dc0070350 2024-03-31T15:03:12.645 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.651+0000 7f3dbfe00640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f3dc0145d50 0x7f3dc006fc40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:56088/0 (socket says 172.21.15.89:56088) 2024-03-31T15:03:12.645 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.651+0000 7f3dbfe00640 1 -- 172.21.15.89:0/165929362 learned_addr learned my addr 172.21.15.89:0/165929362 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-03-31T15:03:12.645 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.651+0000 7f3db7e00640 1 --2- 172.21.15.89:0/165929362 >> v2:172.21.15.16:3300/0 conn(0x7f3dc0070350 0x7f3dc0160860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:12.645 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.651+0000 7f3dbf400640 1 -- 172.21.15.89:0/165929362 <== mon.1 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1669216136 0 0) 0x7f3dc00cd9b0 con 0x7f3dc0145d50 2024-03-31T15:03:12.645 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.651+0000 7f3dbf400640 1 -- 172.21.15.89:0/165929362 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3da4003530 con 0x7f3dc0145d50 2024-03-31T15:03:12.645 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.651+0000 7f3dbf400640 1 -- 172.21.15.89:0/165929362 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1919646365 0 0) 0x7f3dc0070a10 con 0x7f3dc015c890 2024-03-31T15:03:12.646 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.651+0000 7f3dbf400640 1 -- 172.21.15.89:0/165929362 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3dc00cd9b0 con 0x7f3dc015c890 2024-03-31T15:03:12.646 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.651+0000 7f3db7e00640 1 -- 172.21.15.89:0/165929362 >> v1:172.21.15.89:6789/0 conn(0x7f3dc0145d50 legacy=0x7f3dc006fc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:12.646 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.651+0000 7f3db7e00640 1 -- 172.21.15.89:0/165929362 >> v1:172.21.15.16:6789/0 conn(0x7f3dc015c890 legacy=0x7f3dc00cea50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:12.646 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.651+0000 7f3db7e00640 1 -- 172.21.15.89:0/165929362 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3dc00cdc90 con 0x7f3dc0070350 2024-03-31T15:03:12.647 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.655+0000 7f3db7e00640 1 --2- 172.21.15.89:0/165929362 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3dc0070350 0x7f3dc0160860 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f3da80097c0 tx=0x7f3da801c470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72d640e255246703 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:03:12.647 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.655+0000 7f3dbf400640 1 -- 172.21.15.89:0/165929362 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3da80a7070 con 0x7f3dc0070350 2024-03-31T15:03:12.647 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.655+0000 7f3dbf400640 1 -- 172.21.15.89:0/165929362 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3da80b0070 con 0x7f3dc0070350 2024-03-31T15:03:12.647 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.655+0000 7f3dbf400640 1 -- 172.21.15.89:0/165929362 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3da80ab890 con 0x7f3dc0070350 2024-03-31T15:03:12.648 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.655+0000 7f3dc7200640 1 -- 172.21.15.89:0/165929362 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3dc0070350 msgr2=0x7f3dc0160860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:12.648 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.655+0000 7f3dc7200640 1 --2- 172.21.15.89:0/165929362 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3dc0070350 0x7f3dc0160860 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f3da80097c0 tx=0x7f3da801c470 comp rx=0 tx=0).stop 2024-03-31T15:03:12.648 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.655+0000 7f3dc7200640 1 -- 172.21.15.89:0/165929362 shutdown_connections 2024-03-31T15:03:12.648 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.655+0000 7f3dc7200640 1 --2- 172.21.15.89:0/165929362 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3dc0070350 0x7f3dc0160860 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:12.648 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.655+0000 7f3dc7200640 1 -- 172.21.15.89:0/165929362 >> 172.21.15.89:0/165929362 conn(0x7f3dc006e670 msgr2=0x7f3dc006ea50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:03:12.648 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.655+0000 7f3dc7200640 1 -- 172.21.15.89:0/165929362 shutdown_connections 2024-03-31T15:03:12.648 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.655+0000 7f3dc7200640 1 -- 172.21.15.89:0/165929362 wait complete. 2024-03-31T15:03:12.650 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.659+0000 7f3dc7200640 1 Processor -- start 2024-03-31T15:03:12.650 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.659+0000 7f3dc7200640 1 -- start start 2024-03-31T15:03:12.652 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.659+0000 7f3dc7200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3dc0070350 0x7f3dc0167750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:12.652 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.659+0000 7f3dc7200640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f3dc0145d50 0x7f3dc01448d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:12.653 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.659+0000 7f3dc7200640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3dc015c890 0x7f3dc013d970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:12.653 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.659+0000 7f3dc7200640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f3dc016a530 con 0x7f3dc0070350 2024-03-31T15:03:12.653 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.659+0000 7f3dc7200640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f3dc016a3b0 con 0x7f3dc015c890 2024-03-31T15:03:12.653 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.659+0000 7f3dc7200640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f3dc016a6b0 con 0x7f3dc0145d50 2024-03-31T15:03:12.653 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.659+0000 7f3dc5200640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3dc015c890 0x7f3dc013d970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:12.653 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.659+0000 7f3dbfe00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3dc0070350 0x7f3dc0167750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:12.653 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.659+0000 7f3dc5200640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3dc015c890 0x7f3dc013d970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:46082/0 (socket says 172.21.15.89:46082) 2024-03-31T15:03:12.653 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.659+0000 7f3dbfe00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3dc0070350 0x7f3dc0167750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.89:45288/0 (socket says 172.21.15.89:45288) 2024-03-31T15:03:12.653 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.659+0000 7f3dc5200640 1 -- 172.21.15.89:0/3177358998 learned_addr learned my addr 172.21.15.89:0/3177358998 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-03-31T15:03:12.653 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.659+0000 7f3db7e00640 1 --2- 172.21.15.89:0/3177358998 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f3dc0145d50 0x7f3dc01448d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:12.653 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.659+0000 7f3dc5200640 1 -- 172.21.15.89:0/3177358998 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f3dc0145d50 msgr2=0x7f3dc01448d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:12.653 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.659+0000 7f3dc5200640 1 --2- 172.21.15.89:0/3177358998 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f3dc0145d50 0x7f3dc01448d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:12.653 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.659+0000 7f3dc5200640 1 -- 172.21.15.89:0/3177358998 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3dc0070350 msgr2=0x7f3dc0167750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:12.653 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.659+0000 7f3dc5200640 1 --2- 172.21.15.89:0/3177358998 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3dc0070350 0x7f3dc0167750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:12.654 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.659+0000 7f3dc5200640 1 -- 172.21.15.89:0/3177358998 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3dc014b230 con 0x7f3dc015c890 2024-03-31T15:03:12.654 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.659+0000 7f3db7e00640 1 --2- 172.21.15.89:0/3177358998 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f3dc0145d50 0x7f3dc01448d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:03:12.654 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.659+0000 7f3dc5200640 1 --2- 172.21.15.89:0/3177358998 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3dc015c890 0x7f3dc013d970 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f3db800c760 tx=0x7f3db800cc20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:03:12.654 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.659+0000 7f3dbcc00640 1 -- 172.21.15.89:0/3177358998 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3db8003b80 con 0x7f3dc015c890 2024-03-31T15:03:12.654 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.659+0000 7f3dbcc00640 1 -- 172.21.15.89:0/3177358998 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3db8003d20 con 0x7f3dc015c890 2024-03-31T15:03:12.654 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.659+0000 7f3dc7200640 1 -- 172.21.15.89:0/3177358998 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3dc014b510 con 0x7f3dc015c890 2024-03-31T15:03:12.654 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.659+0000 7f3dbcc00640 1 -- 172.21.15.89:0/3177358998 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3db800aa30 con 0x7f3dc015c890 2024-03-31T15:03:12.654 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.659+0000 7f3dc7200640 1 -- 172.21.15.89:0/3177358998 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3dc014bad0 con 0x7f3dc015c890 2024-03-31T15:03:12.654 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.659+0000 7f3dbcc00640 1 -- 172.21.15.89:0/3177358998 <== mon.1 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f3db800abd0 con 0x7f3dc015c890 2024-03-31T15:03:12.654 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.659+0000 7f3dbcc00640 1 -- 172.21.15.89:0/3177358998 <== mon.1 v2:172.21.15.89:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 955+0+0 (secure 0 0 0) 0x7f3db80054f0 con 0x7f3dc015c890 2024-03-31T15:03:12.655 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.663+0000 7f3dc7200640 1 -- 172.21.15.89:0/3177358998 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3dc0069950 con 0x7f3dc015c890 2024-03-31T15:03:12.655 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.663+0000 7f3dbcc00640 1 -- 172.21.15.89:0/3177358998 <== mon.1 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f3db80050d0 con 0x7f3dc015c890 2024-03-31T15:03:12.721 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.731+0000 7f3dc7200640 1 -- 172.21.15.89:0/3177358998 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd new", "uuid": "01a73662-9a21-4b1f-bcbb-42b9636a4638", "id": 2} v 0) -- 0x7f3dc006eaf0 con 0x7f3dc015c890 2024-03-31T15:03:12.725 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.735+0000 7f3dbcc00640 1 -- 172.21.15.89:0/3177358998 <== mon.1 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "01a73662-9a21-4b1f-bcbb-42b9636a4638", "id": 2}]=0 v5) ==== 112+0+2 (secure 0 0 0) 0x7f3db8012510 con 0x7f3dc015c890 2024-03-31T15:03:12.725 INFO:teuthology.orchestra.run.smithi089.stdout:2 2024-03-31T15:03:12.726 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.735+0000 7f3dc7200640 1 -- 172.21.15.89:0/3177358998 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3dc015c890 msgr2=0x7f3dc013d970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:12.727 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.735+0000 7f3dc7200640 1 --2- 172.21.15.89:0/3177358998 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3dc015c890 0x7f3dc013d970 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f3db800c760 tx=0x7f3db800cc20 comp rx=0 tx=0).stop 2024-03-31T15:03:12.727 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.735+0000 7f3dc7200640 1 -- 172.21.15.89:0/3177358998 shutdown_connections 2024-03-31T15:03:12.727 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.735+0000 7f3dc7200640 1 --2- 172.21.15.89:0/3177358998 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3dc015c890 0x7f3dc013d970 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:12.727 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.735+0000 7f3dc7200640 1 --2- 172.21.15.89:0/3177358998 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f3dc0145d50 0x7f3dc01448d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:12.727 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.735+0000 7f3dc7200640 1 --2- 172.21.15.89:0/3177358998 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3dc0070350 0x7f3dc0167750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:12.727 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.735+0000 7f3dc7200640 1 -- 172.21.15.89:0/3177358998 >> 172.21.15.89:0/3177358998 conn(0x7f3dc006e670 msgr2=0x7f3dc015cd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:03:12.727 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.735+0000 7f3dc7200640 1 -- 172.21.15.89:0/3177358998 shutdown_connections 2024-03-31T15:03:12.727 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.735+0000 7f3dc7200640 1 -- 172.21.15.89:0/3177358998 wait complete. 2024-03-31T15:03:12.744 DEBUG:teuthology.orchestra.run.smithi089:> sudo ceph --cluster ceph osd new c7771fd3-e18e-4500-ac38-23cf10ebcd75 3 2024-03-31T15:03:12.878 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.887+0000 7f4c25200640 1 Processor -- start 2024-03-31T15:03:12.878 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.887+0000 7f4c25200640 1 -- start start 2024-03-31T15:03:12.880 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.887+0000 7f4c25200640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f4c200d0c00 0x7f4c20070350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:12.880 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.887+0000 7f4c25200640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4c2006f540 con 0x7f4c2006b620 2024-03-31T15:03:12.880 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.887+0000 7f4c25200640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4c200b43f0 con 0x7f4c200b60a0 2024-03-31T15:03:12.880 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.887+0000 7f4c25200640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f4c2006f820 con 0x7f4c200d0c00 2024-03-31T15:03:12.880 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.887+0000 7f4c1e800640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f4c200d0c00 0x7f4c20070350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:12.880 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.887+0000 7f4c17e00640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f4c200b60a0 0x7f4c2006fc40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:56094/0 (socket says 172.21.15.89:56094) 2024-03-31T15:03:12.880 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.887+0000 7f4c17e00640 1 -- 172.21.15.89:0/4186661805 learned_addr learned my addr 172.21.15.89:0/4186661805 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-03-31T15:03:12.880 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.887+0000 7f4c1d400640 1 -- 172.21.15.89:0/4186661805 <== mon.1 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3653944135 0 0) 0x7f4c200b43f0 con 0x7f4c200b60a0 2024-03-31T15:03:12.880 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.887+0000 7f4c1d400640 1 -- 172.21.15.89:0/4186661805 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4c04003530 con 0x7f4c200b60a0 2024-03-31T15:03:12.880 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.887+0000 7f4c1e800640 1 -- 172.21.15.89:0/4186661805 >> v1:172.21.15.89:6789/0 conn(0x7f4c200b60a0 legacy=0x7f4c2006fc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:12.881 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.887+0000 7f4c1e800640 1 -- 172.21.15.89:0/4186661805 >> v1:172.21.15.16:6789/0 conn(0x7f4c2006b620 legacy=0x7f4c2006ba00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:12.881 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.887+0000 7f4c1d400640 1 -- 172.21.15.89:0/4186661805 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1545311012 0 0) 0x7f4c2006f540 con 0x7f4c2006b620 2024-03-31T15:03:12.881 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.887+0000 7f4c1e800640 1 -- 172.21.15.89:0/4186661805 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c200b46d0 con 0x7f4c200d0c00 2024-03-31T15:03:12.884 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.891+0000 7f4c1e800640 1 --2- 172.21.15.89:0/4186661805 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f4c200d0c00 0x7f4c20070350 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f4c18009dc0 tx=0x7f4c1801c900 comp rx=0 tx=0).ready entity=mon.2 client_cookie=96b24d74cd2fa8f1 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:03:12.885 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.891+0000 7f4c1d400640 1 -- 172.21.15.89:0/4186661805 <== mon.2 v2:172.21.15.89:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c180a66c0 con 0x7f4c200d0c00 2024-03-31T15:03:12.885 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.891+0000 7f4c1d400640 1 -- 172.21.15.89:0/4186661805 <== mon.2 v2:172.21.15.89:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4c180a6860 con 0x7f4c200d0c00 2024-03-31T15:03:12.885 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.891+0000 7f4c1d400640 1 -- 172.21.15.89:0/4186661805 <== mon.2 v2:172.21.15.89:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c180a6c30 con 0x7f4c200d0c00 2024-03-31T15:03:12.885 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.895+0000 7f4c25200640 1 -- 172.21.15.89:0/4186661805 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f4c200d0c00 msgr2=0x7f4c20070350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:12.886 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.895+0000 7f4c25200640 1 --2- 172.21.15.89:0/4186661805 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f4c200d0c00 0x7f4c20070350 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f4c18009dc0 tx=0x7f4c1801c900 comp rx=0 tx=0).stop 2024-03-31T15:03:12.886 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.895+0000 7f4c25200640 1 -- 172.21.15.89:0/4186661805 shutdown_connections 2024-03-31T15:03:12.886 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.895+0000 7f4c25200640 1 --2- 172.21.15.89:0/4186661805 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f4c200d0c00 0x7f4c20070350 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:12.886 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.895+0000 7f4c25200640 1 -- 172.21.15.89:0/4186661805 >> 172.21.15.89:0/4186661805 conn(0x7f4c2006e670 msgr2=0x7f4c2006ea50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:03:12.886 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.895+0000 7f4c25200640 1 -- 172.21.15.89:0/4186661805 shutdown_connections 2024-03-31T15:03:12.886 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.895+0000 7f4c25200640 1 -- 172.21.15.89:0/4186661805 wait complete. 2024-03-31T15:03:12.887 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.895+0000 7f4c25200640 1 Processor -- start 2024-03-31T15:03:12.887 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.895+0000 7f4c25200640 1 -- start start 2024-03-31T15:03:12.889 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.895+0000 7f4c25200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4c2006b620 0x7f4c20148bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:12.889 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.895+0000 7f4c25200640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f4c200b60a0 0x7f4c2014ed60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:12.889 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.895+0000 7f4c25200640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4c200d0c00 0x7f4c2014f2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:12.889 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.895+0000 7f4c25200640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f4c2016cd10 con 0x7f4c2006b620 2024-03-31T15:03:12.889 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.895+0000 7f4c25200640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f4c2016cb90 con 0x7f4c200d0c00 2024-03-31T15:03:12.890 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.895+0000 7f4c25200640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f4c2016ce90 con 0x7f4c200b60a0 2024-03-31T15:03:12.890 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.895+0000 7f4c1e800640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4c200d0c00 0x7f4c2014f2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:12.890 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.895+0000 7f4c17e00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f4c200b60a0 0x7f4c2014ed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:12.890 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.895+0000 7f4c1e800640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4c200d0c00 0x7f4c2014f2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:46098/0 (socket says 172.21.15.89:46098) 2024-03-31T15:03:12.890 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.895+0000 7f4c1e800640 1 -- 172.21.15.89:0/2145291545 learned_addr learned my addr 172.21.15.89:0/2145291545 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-03-31T15:03:12.890 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.895+0000 7f4c1de00640 1 --2- 172.21.15.89:0/2145291545 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4c2006b620 0x7f4c20148bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:12.890 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.895+0000 7f4c1e800640 1 -- 172.21.15.89:0/2145291545 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f4c200b60a0 msgr2=0x7f4c2014ed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:12.890 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.895+0000 7f4c1e800640 1 --2- 172.21.15.89:0/2145291545 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f4c200b60a0 0x7f4c2014ed60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:12.890 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.895+0000 7f4c1e800640 1 -- 172.21.15.89:0/2145291545 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4c2006b620 msgr2=0x7f4c20148bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:12.890 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.895+0000 7f4c1e800640 1 --2- 172.21.15.89:0/2145291545 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4c2006b620 0x7f4c20148bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:12.890 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.899+0000 7f4c1e800640 1 -- 172.21.15.89:0/2145291545 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c201561c0 con 0x7f4c200d0c00 2024-03-31T15:03:12.890 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.899+0000 7f4c17e00640 1 --2- 172.21.15.89:0/2145291545 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f4c200b60a0 0x7f4c2014ed60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:03:12.890 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.899+0000 7f4c1e800640 1 --2- 172.21.15.89:0/2145291545 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4c200d0c00 0x7f4c2014f2a0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f4c1801cde0 tx=0x7f4c180196a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:03:12.891 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.899+0000 7f4c16000640 1 -- 172.21.15.89:0/2145291545 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c180b8630 con 0x7f4c200d0c00 2024-03-31T15:03:12.892 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.899+0000 7f4c16000640 1 -- 172.21.15.89:0/2145291545 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4c180c1070 con 0x7f4c200d0c00 2024-03-31T15:03:12.892 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.899+0000 7f4c16000640 1 -- 172.21.15.89:0/2145291545 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c180b89d0 con 0x7f4c200d0c00 2024-03-31T15:03:12.892 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.899+0000 7f4c25200640 1 -- 172.21.15.89:0/2145291545 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4c2014a440 con 0x7f4c200d0c00 2024-03-31T15:03:12.892 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.899+0000 7f4c25200640 1 -- 172.21.15.89:0/2145291545 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4c2014aa00 con 0x7f4c200d0c00 2024-03-31T15:03:12.892 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.899+0000 7f4c16000640 1 -- 172.21.15.89:0/2145291545 <== mon.1 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f4c180ae420 con 0x7f4c200d0c00 2024-03-31T15:03:12.892 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.899+0000 7f4c16000640 1 -- 172.21.15.89:0/2145291545 <== mon.1 v2:172.21.15.89:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1070+0+0 (secure 0 0 0) 0x7f4c180aeba0 con 0x7f4c200d0c00 2024-03-31T15:03:12.892 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.899+0000 7f4c25200640 1 -- 172.21.15.89:0/2145291545 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4c20069950 con 0x7f4c200d0c00 2024-03-31T15:03:12.892 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.899+0000 7f4c16000640 1 -- 172.21.15.89:0/2145291545 <== mon.1 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f4c180ca020 con 0x7f4c200d0c00 2024-03-31T15:03:12.959 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.967+0000 7f4c25200640 1 -- 172.21.15.89:0/2145291545 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd new", "uuid": "c7771fd3-e18e-4500-ac38-23cf10ebcd75", "id": 3} v 0) -- 0x7f4c2006c220 con 0x7f4c200d0c00 2024-03-31T15:03:12.963 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.971+0000 7f4c16000640 1 -- 172.21.15.89:0/2145291545 <== mon.1 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "c7771fd3-e18e-4500-ac38-23cf10ebcd75", "id": 3}]=0 v6) ==== 112+0+2 (secure 0 0 0) 0x7f4c18016580 con 0x7f4c200d0c00 2024-03-31T15:03:12.963 INFO:teuthology.orchestra.run.smithi089.stdout:3 2024-03-31T15:03:12.965 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.971+0000 7f4c25200640 1 -- 172.21.15.89:0/2145291545 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4c200d0c00 msgr2=0x7f4c2014f2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:12.965 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.971+0000 7f4c25200640 1 --2- 172.21.15.89:0/2145291545 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4c200d0c00 0x7f4c2014f2a0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f4c1801cde0 tx=0x7f4c180196a0 comp rx=0 tx=0).stop 2024-03-31T15:03:12.965 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.971+0000 7f4c25200640 1 -- 172.21.15.89:0/2145291545 shutdown_connections 2024-03-31T15:03:12.965 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.971+0000 7f4c25200640 1 --2- 172.21.15.89:0/2145291545 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4c200d0c00 0x7f4c2014f2a0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:12.965 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.971+0000 7f4c25200640 1 --2- 172.21.15.89:0/2145291545 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f4c200b60a0 0x7f4c2014ed60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:12.965 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.971+0000 7f4c25200640 1 --2- 172.21.15.89:0/2145291545 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4c2006b620 0x7f4c20148bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:12.965 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.971+0000 7f4c25200640 1 -- 172.21.15.89:0/2145291545 >> 172.21.15.89:0/2145291545 conn(0x7f4c2006e670 msgr2=0x7f4c20152390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:03:12.965 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.971+0000 7f4c25200640 1 -- 172.21.15.89:0/2145291545 shutdown_connections 2024-03-31T15:03:12.965 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:12.971+0000 7f4c25200640 1 -- 172.21.15.89:0/2145291545 wait complete. 2024-03-31T15:03:12.983 DEBUG:teuthology.orchestra.run.smithi089:> sudo ceph --cluster ceph osd new 88260730-1482-4fbd-a40f-cb1124e9df4a 4 2024-03-31T15:03:13.100 INFO:tasks.ceph.mgr.x.smithi089.stderr:2024-03-31T15:03:13.107+0000 7f078c909180 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-31T15:03:13.102 INFO:tasks.ceph.mgr.y.smithi016.stderr:2024-03-31T15:03:13.106+0000 7f2c2622a180 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-31T15:03:13.118 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.127+0000 7faaa2800640 1 Processor -- start 2024-03-31T15:03:13.118 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.127+0000 7faaa2800640 1 -- start start 2024-03-31T15:03:13.119 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.127+0000 7faaa2800640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7faa9c0cf150 0x7faa9c1670a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:13.119 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.127+0000 7faaa2800640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faa9c15ca70 con 0x7faa9c0d0a60 2024-03-31T15:03:13.119 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.127+0000 7faaa2800640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faa9c15cd50 con 0x7faa9c06f920 2024-03-31T15:03:13.119 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.127+0000 7faaa2800640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7faa9c13f670 con 0x7faa9c0cf150 2024-03-31T15:03:13.119 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.127+0000 7faa9be00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7faa9c0cf150 0x7faa9c1670a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:13.119 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.127+0000 7faa9b400640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7faa9c06f920 0x7faa9c06fd00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:56106/0 (socket says 172.21.15.89:56106) 2024-03-31T15:03:13.119 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.127+0000 7faa9b400640 1 -- 172.21.15.89:0/301158810 learned_addr learned my addr 172.21.15.89:0/301158810 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-03-31T15:03:13.120 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.127+0000 7faa9be00640 1 -- 172.21.15.89:0/301158810 >> v1:172.21.15.89:6789/0 conn(0x7faa9c06f920 legacy=0x7faa9c06fd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:13.120 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.127+0000 7faa9a000640 1 -- 172.21.15.89:0/301158810 <== mon.1 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 626484424 0 0) 0x7faa9c15cd50 con 0x7faa9c06f920 2024-03-31T15:03:13.120 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.127+0000 7faa9be00640 1 -- 172.21.15.89:0/301158810 >> v1:172.21.15.16:6789/0 conn(0x7faa9c0d0a60 legacy=0x7faa9c0cea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:13.120 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.127+0000 7faa9be00640 1 -- 172.21.15.89:0/301158810 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa9c15d030 con 0x7faa9c0cf150 2024-03-31T15:03:13.123 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.131+0000 7faa9be00640 1 --2- 172.21.15.89:0/301158810 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7faa9c0cf150 0x7faa9c1670a0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7faa94009dc0 tx=0x7faa9401c900 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1966d129f4c14fd9 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:03:13.123 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.131+0000 7faa9a000640 1 -- 172.21.15.89:0/301158810 <== mon.2 v2:172.21.15.89:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa940b2070 con 0x7faa9c0cf150 2024-03-31T15:03:13.124 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.131+0000 7faa9a000640 1 -- 172.21.15.89:0/301158810 <== mon.2 v2:172.21.15.89:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faa940a57c0 con 0x7faa9c0cf150 2024-03-31T15:03:13.124 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.131+0000 7faa9a000640 1 -- 172.21.15.89:0/301158810 <== mon.2 v2:172.21.15.89:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa940a5b90 con 0x7faa9c0cf150 2024-03-31T15:03:13.124 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.131+0000 7faaa2800640 1 -- 172.21.15.89:0/301158810 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7faa9c0cf150 msgr2=0x7faa9c1670a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:13.124 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.131+0000 7faaa2800640 1 --2- 172.21.15.89:0/301158810 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7faa9c0cf150 0x7faa9c1670a0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7faa94009dc0 tx=0x7faa9401c900 comp rx=0 tx=0).stop 2024-03-31T15:03:13.124 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.131+0000 7faaa2800640 1 -- 172.21.15.89:0/301158810 shutdown_connections 2024-03-31T15:03:13.124 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.131+0000 7faaa2800640 1 --2- 172.21.15.89:0/301158810 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7faa9c0cf150 0x7faa9c1670a0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:13.124 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.131+0000 7faaa2800640 1 -- 172.21.15.89:0/301158810 >> 172.21.15.89:0/301158810 conn(0x7faa9c067930 msgr2=0x7faa9c0b3ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:03:13.124 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.131+0000 7faaa2800640 1 -- 172.21.15.89:0/301158810 shutdown_connections 2024-03-31T15:03:13.124 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.131+0000 7faaa2800640 1 -- 172.21.15.89:0/301158810 wait complete. 2024-03-31T15:03:13.126 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.135+0000 7faaa2800640 1 Processor -- start 2024-03-31T15:03:13.126 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.135+0000 7faaa2800640 1 -- start start 2024-03-31T15:03:13.128 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.135+0000 7faaa2800640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7faa9c06f920 0x7faa9c0a94c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:13.128 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.135+0000 7faaa2800640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faa9c0cf150 0x7faa9c0b2690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:13.128 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.135+0000 7faaa2800640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faa9c0d0a60 0x7faa9c0b3420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:13.128 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.135+0000 7faaa2800640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7faa9c16a190 con 0x7faa9c0d0a60 2024-03-31T15:03:13.128 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.135+0000 7faaa2800640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7faa9c16a010 con 0x7faa9c0cf150 2024-03-31T15:03:13.128 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.135+0000 7faaa2800640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7faa9c16a310 con 0x7faa9c06f920 2024-03-31T15:03:13.128 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.135+0000 7faa9b400640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7faa9c06f920 0x7faa9c0a94c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:13.128 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.135+0000 7faa9b400640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7faa9c06f920 0x7faa9c0a94c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3301/0 says I am v2:172.21.15.89:56126/0 (socket says 172.21.15.89:56126) 2024-03-31T15:03:13.128 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.135+0000 7faa9b400640 1 -- 172.21.15.89:0/1850003259 learned_addr learned my addr 172.21.15.89:0/1850003259 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-03-31T15:03:13.129 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.135+0000 7faa9aa00640 1 --2- 172.21.15.89:0/1850003259 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faa9c0cf150 0x7faa9c0b2690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:13.129 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.135+0000 7faa9be00640 1 --2- 172.21.15.89:0/1850003259 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faa9c0d0a60 0x7faa9c0b3420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:13.129 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.135+0000 7faa9b400640 1 -- 172.21.15.89:0/1850003259 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faa9c0cf150 msgr2=0x7faa9c0b2690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:13.129 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.135+0000 7faa9b400640 1 --2- 172.21.15.89:0/1850003259 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faa9c0cf150 0x7faa9c0b2690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:13.129 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.135+0000 7faa9b400640 1 -- 172.21.15.89:0/1850003259 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faa9c0d0a60 msgr2=0x7faa9c0b3420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:13.129 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.135+0000 7faa9b400640 1 --2- 172.21.15.89:0/1850003259 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faa9c0d0a60 0x7faa9c0b3420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:13.129 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.135+0000 7faa9b400640 1 -- 172.21.15.89:0/1850003259 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa9c0cfe10 con 0x7faa9c06f920 2024-03-31T15:03:13.129 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.135+0000 7faa9b400640 1 --2- 172.21.15.89:0/1850003259 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7faa9c06f920 0x7faa9c0a94c0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7faa8c00c410 tx=0x7faa8c00c8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:03:13.129 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.135+0000 7faa93400640 1 -- 172.21.15.89:0/1850003259 <== mon.2 v2:172.21.15.89:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa8c0074e0 con 0x7faa9c06f920 2024-03-31T15:03:13.129 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.135+0000 7faa93400640 1 -- 172.21.15.89:0/1850003259 <== mon.2 v2:172.21.15.89:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faa8c0047f0 con 0x7faa9c06f920 2024-03-31T15:03:13.129 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.135+0000 7faaa2800640 1 -- 172.21.15.89:0/1850003259 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faa9c0b3960 con 0x7faa9c06f920 2024-03-31T15:03:13.129 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.135+0000 7faa93400640 1 -- 172.21.15.89:0/1850003259 <== mon.2 v2:172.21.15.89:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa8c007680 con 0x7faa9c06f920 2024-03-31T15:03:13.129 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.135+0000 7faaa2800640 1 -- 172.21.15.89:0/1850003259 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7faa9c0b1bb0 con 0x7faa9c06f920 2024-03-31T15:03:13.129 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.135+0000 7faa93400640 1 -- 172.21.15.89:0/1850003259 <== mon.2 v2:172.21.15.89:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7faa8c011070 con 0x7faa9c06f920 2024-03-31T15:03:13.130 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.135+0000 7faa93400640 1 -- 172.21.15.89:0/1850003259 <== mon.2 v2:172.21.15.89:3301/0 5 ==== osd_map(6..6 src has 1..6) ==== 1185+0+0 (secure 0 0 0) 0x7faa8c0138d0 con 0x7faa9c06f920 2024-03-31T15:03:13.130 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.135+0000 7faaa2800640 1 -- 172.21.15.89:0/1850003259 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faa9c069950 con 0x7faa9c06f920 2024-03-31T15:03:13.131 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.139+0000 7faa93400640 1 -- 172.21.15.89:0/1850003259 <== mon.2 v2:172.21.15.89:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7faa8c004d70 con 0x7faa9c06f920 2024-03-31T15:03:13.165 INFO:tasks.ceph.mgr.x.smithi089.stderr:2024-03-31T15:03:13.171+0000 7f078c909180 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-31T15:03:13.167 INFO:tasks.ceph.mgr.y.smithi016.stderr:2024-03-31T15:03:13.174+0000 7f2c2622a180 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-31T15:03:13.197 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.203+0000 7faaa2800640 1 -- 172.21.15.89:0/1850003259 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_command({"prefix": "osd new", "uuid": "88260730-1482-4fbd-a40f-cb1124e9df4a", "id": 4} v 0) -- 0x7faa9c0b50e0 con 0x7faa9c06f920 2024-03-31T15:03:13.201 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.207+0000 7faa93400640 1 -- 172.21.15.89:0/1850003259 <== mon.2 v2:172.21.15.89:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "88260730-1482-4fbd-a40f-cb1124e9df4a", "id": 4}]=0 v7) ==== 112+0+2 (secure 0 0 0) 0x7faa8c013ba0 con 0x7faa9c06f920 2024-03-31T15:03:13.201 INFO:teuthology.orchestra.run.smithi089.stdout:4 2024-03-31T15:03:13.202 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.211+0000 7faaa2800640 1 -- 172.21.15.89:0/1850003259 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7faa9c06f920 msgr2=0x7faa9c0a94c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:13.202 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.211+0000 7faaa2800640 1 --2- 172.21.15.89:0/1850003259 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7faa9c06f920 0x7faa9c0a94c0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7faa8c00c410 tx=0x7faa8c00c8d0 comp rx=0 tx=0).stop 2024-03-31T15:03:13.202 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.211+0000 7faaa2800640 1 -- 172.21.15.89:0/1850003259 shutdown_connections 2024-03-31T15:03:13.202 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.211+0000 7faaa2800640 1 --2- 172.21.15.89:0/1850003259 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faa9c0d0a60 0x7faa9c0b3420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:13.202 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.211+0000 7faaa2800640 1 --2- 172.21.15.89:0/1850003259 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faa9c0cf150 0x7faa9c0b2690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:13.202 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.211+0000 7faaa2800640 1 --2- 172.21.15.89:0/1850003259 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7faa9c06f920 0x7faa9c0a94c0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:13.202 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.211+0000 7faaa2800640 1 -- 172.21.15.89:0/1850003259 >> 172.21.15.89:0/1850003259 conn(0x7faa9c067930 msgr2=0x7faa9c06c730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:03:13.202 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.211+0000 7faaa2800640 1 -- 172.21.15.89:0/1850003259 shutdown_connections 2024-03-31T15:03:13.202 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.211+0000 7faaa2800640 1 -- 172.21.15.89:0/1850003259 wait complete. 2024-03-31T15:03:13.219 DEBUG:teuthology.orchestra.run.smithi089:> sudo ceph --cluster ceph osd new 4cb2c00c-fae0-4dd8-9d27-ec4f0093ccf7 5 2024-03-31T15:03:13.229 INFO:tasks.ceph.mgr.x.smithi089.stderr:2024-03-31T15:03:13.239+0000 7f078c909180 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-31T15:03:13.231 INFO:tasks.ceph.mgr.y.smithi016.stderr:2024-03-31T15:03:13.234+0000 7f2c2622a180 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-31T15:03:13.319 INFO:tasks.ceph.mgr.x.smithi089.stderr:2024-03-31T15:03:13.327+0000 7f078c909180 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-31T15:03:13.323 INFO:tasks.ceph.mgr.y.smithi016.stderr:2024-03-31T15:03:13.326+0000 7f2c2622a180 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-31T15:03:13.354 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.363+0000 7f725c200640 1 Processor -- start 2024-03-31T15:03:13.354 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.363+0000 7f725c200640 1 -- start start 2024-03-31T15:03:13.356 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.363+0000 7f725c200640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f72540d0880 0x7f72540b4250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:13.356 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.363+0000 7f725c200640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7254070520 con 0x7f72540cdc30 2024-03-31T15:03:13.356 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.363+0000 7f725c200640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f72540cfda0 con 0x7f725415c3d0 2024-03-31T15:03:13.356 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.363+0000 7f725c200640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f72540d0080 con 0x7f72540d0880 2024-03-31T15:03:13.356 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.363+0000 7f725a200640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f72540d0880 0x7f72540b4250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:13.356 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.363+0000 7f7258e00640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f725415c3d0 0x7f725406fc40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:56114/0 (socket says 172.21.15.89:56114) 2024-03-31T15:03:13.356 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.363+0000 7f7258e00640 1 -- 172.21.15.89:0/3053904214 learned_addr learned my addr 172.21.15.89:0/3053904214 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-03-31T15:03:13.356 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.363+0000 7f7253e00640 1 -- 172.21.15.89:0/3053904214 <== mon.1 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 349264068 0 0) 0x7f72540cfda0 con 0x7f725415c3d0 2024-03-31T15:03:13.357 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.363+0000 7f725a200640 1 -- 172.21.15.89:0/3053904214 >> v1:172.21.15.89:6789/0 conn(0x7f725415c3d0 legacy=0x7f725406fc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:13.357 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.363+0000 7f725a200640 1 -- 172.21.15.89:0/3053904214 >> v1:172.21.15.16:6789/0 conn(0x7f72540cdc30 legacy=0x7f72540ce010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:13.357 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.363+0000 7f725a200640 1 -- 172.21.15.89:0/3053904214 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72540d0360 con 0x7f72540d0880 2024-03-31T15:03:13.361 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.367+0000 7f725a200640 1 --2- 172.21.15.89:0/3053904214 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f72540d0880 0x7f72540b4250 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f724c009dc0 tx=0x7f724c01c990 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f95547b7b65db514 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:03:13.361 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.371+0000 7f7253e00640 1 -- 172.21.15.89:0/3053904214 <== mon.2 v2:172.21.15.89:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f724c0b2070 con 0x7f72540d0880 2024-03-31T15:03:13.361 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.371+0000 7f7253e00640 1 -- 172.21.15.89:0/3053904214 <== mon.2 v2:172.21.15.89:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f724c0a5960 con 0x7f72540d0880 2024-03-31T15:03:13.361 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.371+0000 7f7253e00640 1 -- 172.21.15.89:0/3053904214 <== mon.2 v2:172.21.15.89:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f724c0a5d30 con 0x7f72540d0880 2024-03-31T15:03:13.362 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.371+0000 7f725c200640 1 -- 172.21.15.89:0/3053904214 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f72540d0880 msgr2=0x7f72540b4250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:13.362 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.371+0000 7f725c200640 1 --2- 172.21.15.89:0/3053904214 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f72540d0880 0x7f72540b4250 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f724c009dc0 tx=0x7f724c01c990 comp rx=0 tx=0).stop 2024-03-31T15:03:13.362 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.371+0000 7f725c200640 1 -- 172.21.15.89:0/3053904214 shutdown_connections 2024-03-31T15:03:13.362 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.371+0000 7f725c200640 1 --2- 172.21.15.89:0/3053904214 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f72540d0880 0x7f72540b4250 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:13.362 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.371+0000 7f725c200640 1 -- 172.21.15.89:0/3053904214 >> 172.21.15.89:0/3053904214 conn(0x7f725406e670 msgr2=0x7f725406ea50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:03:13.362 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.371+0000 7f725c200640 1 -- 172.21.15.89:0/3053904214 shutdown_connections 2024-03-31T15:03:13.362 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.371+0000 7f725c200640 1 -- 172.21.15.89:0/3053904214 wait complete. 2024-03-31T15:03:13.363 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.371+0000 7f725c200640 1 Processor -- start 2024-03-31T15:03:13.363 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.371+0000 7f725c200640 1 -- start start 2024-03-31T15:03:13.365 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.371+0000 7f725c200640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f72540cdc30 0x7f725414cee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:13.365 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.371+0000 7f725c200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f72540d0880 0x7f725414fb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:13.365 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.371+0000 7f725c200640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f725415c3d0 0x7f72541500b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:13.366 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.371+0000 7f725c200640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f7254169800 con 0x7f72540d0880 2024-03-31T15:03:13.366 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.371+0000 7f725c200640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f7254169680 con 0x7f725415c3d0 2024-03-31T15:03:13.366 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.371+0000 7f725c200640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f7254169980 con 0x7f72540cdc30 2024-03-31T15:03:13.366 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.371+0000 7f7259800640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f72540cdc30 0x7f725414cee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:13.366 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.371+0000 7f7259800640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f72540cdc30 0x7f725414cee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3301/0 says I am v2:172.21.15.89:56142/0 (socket says 172.21.15.89:56142) 2024-03-31T15:03:13.366 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.371+0000 7f7259800640 1 -- 172.21.15.89:0/2658928087 learned_addr learned my addr 172.21.15.89:0/2658928087 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-03-31T15:03:13.366 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.371+0000 7f725a200640 1 --2- 172.21.15.89:0/2658928087 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f725415c3d0 0x7f72541500b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:13.366 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.375+0000 7f7258e00640 1 --2- 172.21.15.89:0/2658928087 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f72540d0880 0x7f725414fb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:13.366 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.375+0000 7f7259800640 1 -- 172.21.15.89:0/2658928087 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f725415c3d0 msgr2=0x7f72541500b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:13.366 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.375+0000 7f7259800640 1 --2- 172.21.15.89:0/2658928087 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f725415c3d0 0x7f72541500b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:13.366 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.375+0000 7f7259800640 1 -- 172.21.15.89:0/2658928087 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f72540d0880 msgr2=0x7f725414fb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:13.366 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.375+0000 7f7259800640 1 --2- 172.21.15.89:0/2658928087 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f72540d0880 0x7f725414fb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:13.366 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.375+0000 7f7259800640 1 -- 172.21.15.89:0/2658928087 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72540b1cb0 con 0x7f72540cdc30 2024-03-31T15:03:13.366 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.375+0000 7f7259800640 1 --2- 172.21.15.89:0/2658928087 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f72540cdc30 0x7f725414cee0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f724800c530 tx=0x7f724800c9f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:03:13.367 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.375+0000 7f7251600640 1 -- 172.21.15.89:0/2658928087 <== mon.2 v2:172.21.15.89:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7248016020 con 0x7f72540cdc30 2024-03-31T15:03:13.367 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.375+0000 7f7251600640 1 -- 172.21.15.89:0/2658928087 <== mon.2 v2:172.21.15.89:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7248005220 con 0x7f72540cdc30 2024-03-31T15:03:13.368 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.375+0000 7f7251600640 1 -- 172.21.15.89:0/2658928087 <== mon.2 v2:172.21.15.89:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7248007030 con 0x7f72540cdc30 2024-03-31T15:03:13.368 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.375+0000 7f725c200640 1 -- 172.21.15.89:0/2658928087 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f72540b1f90 con 0x7f72540cdc30 2024-03-31T15:03:13.368 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.375+0000 7f725c200640 1 -- 172.21.15.89:0/2658928087 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f72540b2550 con 0x7f72540cdc30 2024-03-31T15:03:13.368 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.375+0000 7f7251600640 1 -- 172.21.15.89:0/2658928087 <== mon.2 v2:172.21.15.89:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f7248007560 con 0x7f72540cdc30 2024-03-31T15:03:13.368 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.375+0000 7f7251600640 1 -- 172.21.15.89:0/2658928087 <== mon.2 v2:172.21.15.89:3301/0 5 ==== osd_map(7..7 src has 1..7) ==== 1300+0+0 (secure 0 0 0) 0x7f7248006d60 con 0x7f72540cdc30 2024-03-31T15:03:13.368 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.375+0000 7f725c200640 1 -- 172.21.15.89:0/2658928087 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7254069950 con 0x7f72540cdc30 2024-03-31T15:03:13.369 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.379+0000 7f7251600640 1 -- 172.21.15.89:0/2658928087 <== mon.2 v2:172.21.15.89:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f7248020020 con 0x7f72540cdc30 2024-03-31T15:03:13.384 INFO:tasks.ceph.mgr.x.smithi089.stderr:2024-03-31T15:03:13.391+0000 7f078c909180 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-31T15:03:13.386 INFO:tasks.ceph.mgr.y.smithi016.stderr:2024-03-31T15:03:13.390+0000 7f2c2622a180 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-31T15:03:13.435 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.443+0000 7f725c200640 1 -- 172.21.15.89:0/2658928087 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_command({"prefix": "osd new", "uuid": "4cb2c00c-fae0-4dd8-9d27-ec4f0093ccf7", "id": 5} v 0) -- 0x7f725415e6c0 con 0x7f72540cdc30 2024-03-31T15:03:13.439 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.447+0000 7f7251600640 1 -- 172.21.15.89:0/2658928087 <== mon.2 v2:172.21.15.89:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "4cb2c00c-fae0-4dd8-9d27-ec4f0093ccf7", "id": 5}]=0 v8) ==== 112+0+2 (secure 0 0 0) 0x7f7248012760 con 0x7f72540cdc30 2024-03-31T15:03:13.439 INFO:teuthology.orchestra.run.smithi089.stdout:5 2024-03-31T15:03:13.440 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.447+0000 7f725c200640 1 -- 172.21.15.89:0/2658928087 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f72540cdc30 msgr2=0x7f725414cee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:13.441 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.447+0000 7f725c200640 1 --2- 172.21.15.89:0/2658928087 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f72540cdc30 0x7f725414cee0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f724800c530 tx=0x7f724800c9f0 comp rx=0 tx=0).stop 2024-03-31T15:03:13.441 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.447+0000 7f725c200640 1 -- 172.21.15.89:0/2658928087 shutdown_connections 2024-03-31T15:03:13.441 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.447+0000 7f725c200640 1 --2- 172.21.15.89:0/2658928087 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f725415c3d0 0x7f72541500b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:13.441 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.447+0000 7f725c200640 1 --2- 172.21.15.89:0/2658928087 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f72540d0880 0x7f725414fb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:13.441 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.447+0000 7f725c200640 1 --2- 172.21.15.89:0/2658928087 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f72540cdc30 0x7f725414cee0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:13.441 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.447+0000 7f725c200640 1 -- 172.21.15.89:0/2658928087 >> 172.21.15.89:0/2658928087 conn(0x7f725406e670 msgr2=0x7f725406ea50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:03:13.441 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.447+0000 7f725c200640 1 -- 172.21.15.89:0/2658928087 shutdown_connections 2024-03-31T15:03:13.441 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.447+0000 7f725c200640 1 -- 172.21.15.89:0/2658928087 wait complete. 2024-03-31T15:03:13.459 DEBUG:teuthology.orchestra.run.smithi089:> sudo ceph --cluster ceph osd new 81ce0c19-2554-4cae-acbe-66a05e9e9b9b 6 2024-03-31T15:03:13.476 INFO:tasks.ceph.mgr.x.smithi089.stderr:2024-03-31T15:03:13.483+0000 7f078c909180 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-31T15:03:13.480 INFO:tasks.ceph.mgr.y.smithi016.stderr:2024-03-31T15:03:13.486+0000 7f2c2622a180 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-31T15:03:13.541 INFO:tasks.ceph.mgr.x.smithi089.stderr:2024-03-31T15:03:13.551+0000 7f078c909180 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-31T15:03:13.544 INFO:tasks.ceph.mgr.y.smithi016.stderr:2024-03-31T15:03:13.550+0000 7f2c2622a180 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-31T15:03:13.610 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.619+0000 7f2569000640 1 Processor -- start 2024-03-31T15:03:13.610 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.619+0000 7f2569000640 1 -- start start 2024-03-31T15:03:13.611 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.619+0000 7f2569000640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f25640b60a0 0x7f256406fc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:13.612 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.619+0000 7f2569000640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f25640d0c00 0x7f2564070180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:13.612 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.619+0000 7f2569000640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f256406f540 con 0x7f256406b620 2024-03-31T15:03:13.612 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.619+0000 7f2569000640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f256406f820 con 0x7f25640d0c00 2024-03-31T15:03:13.612 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.619+0000 7f2569000640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f25640b4a40 con 0x7f25640b60a0 2024-03-31T15:03:13.612 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.619+0000 7f2561e00640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f256406b620 0x7f256406ba00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:56130/0 (socket says 172.21.15.89:56130) 2024-03-31T15:03:13.612 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.619+0000 7f255be00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f25640b60a0 0x7f256406fc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:13.612 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.619+0000 7f2561e00640 1 -- 172.21.15.89:0/3039204735 learned_addr learned my addr 172.21.15.89:0/3039204735 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-03-31T15:03:13.612 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.619+0000 7f2562800640 1 --2- 172.21.15.89:0/3039204735 >> v2:172.21.15.16:3300/0 conn(0x7f25640d0c00 0x7f2564070180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:13.613 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.619+0000 7f2561400640 1 -- 172.21.15.89:0/3039204735 <== mon.1 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3245149688 0 0) 0x7f256406f540 con 0x7f256406b620 2024-03-31T15:03:13.613 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.619+0000 7f2561400640 1 -- 172.21.15.89:0/3039204735 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2548003530 con 0x7f256406b620 2024-03-31T15:03:13.613 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.619+0000 7f255be00640 1 -- 172.21.15.89:0/3039204735 >> v2:172.21.15.16:3300/0 conn(0x7f25640d0c00 msgr2=0x7f2564070180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:03:13.613 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.619+0000 7f255be00640 1 --2- 172.21.15.89:0/3039204735 >> v2:172.21.15.16:3300/0 conn(0x7f25640d0c00 0x7f2564070180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:13.613 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.619+0000 7f255be00640 1 -- 172.21.15.89:0/3039204735 >> v1:172.21.15.89:6789/0 conn(0x7f256406b620 legacy=0x7f256406ba00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:13.613 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.619+0000 7f255be00640 1 -- 172.21.15.89:0/3039204735 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25640b43f0 con 0x7f25640b60a0 2024-03-31T15:03:13.617 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.623+0000 7f255be00640 1 --2- 172.21.15.89:0/3039204735 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f25640b60a0 0x7f256406fc40 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f254c0097c0 tx=0x7f254c01c3e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=47890ce9cf3a87f8 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:03:13.617 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.627+0000 7f2561400640 1 -- 172.21.15.89:0/3039204735 <== mon.2 v2:172.21.15.89:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f254c0ab600 con 0x7f25640b60a0 2024-03-31T15:03:13.617 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.627+0000 7f2561400640 1 -- 172.21.15.89:0/3039204735 <== mon.2 v2:172.21.15.89:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f254c0b0070 con 0x7f25640b60a0 2024-03-31T15:03:13.617 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.627+0000 7f2561400640 1 -- 172.21.15.89:0/3039204735 <== mon.2 v2:172.21.15.89:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f254c0ab9c0 con 0x7f25640b60a0 2024-03-31T15:03:13.618 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.627+0000 7f2569000640 1 -- 172.21.15.89:0/3039204735 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f25640b60a0 msgr2=0x7f256406fc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:13.618 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.627+0000 7f2569000640 1 --2- 172.21.15.89:0/3039204735 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f25640b60a0 0x7f256406fc40 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f254c0097c0 tx=0x7f254c01c3e0 comp rx=0 tx=0).stop 2024-03-31T15:03:13.618 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.627+0000 7f2569000640 1 -- 172.21.15.89:0/3039204735 shutdown_connections 2024-03-31T15:03:13.618 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.627+0000 7f2569000640 1 --2- 172.21.15.89:0/3039204735 >> v2:172.21.15.16:3300/0 conn(0x7f25640d0c00 0x7f2564070180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:13.618 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.627+0000 7f2569000640 1 --2- 172.21.15.89:0/3039204735 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f25640b60a0 0x7f256406fc40 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:13.618 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.627+0000 7f2569000640 1 -- 172.21.15.89:0/3039204735 >> 172.21.15.89:0/3039204735 conn(0x7f256406e670 msgr2=0x7f256406ea50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:03:13.618 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.627+0000 7f2569000640 1 -- 172.21.15.89:0/3039204735 shutdown_connections 2024-03-31T15:03:13.618 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.627+0000 7f2569000640 1 -- 172.21.15.89:0/3039204735 wait complete. 2024-03-31T15:03:13.619 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.627+0000 7f2569000640 1 Processor -- start 2024-03-31T15:03:13.619 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.627+0000 7f2569000640 1 -- start start 2024-03-31T15:03:13.621 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.627+0000 7f2569000640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f256406b620 0x7f256413ab10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:13.621 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.627+0000 7f2569000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f25640b60a0 0x7f256414c420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:13.621 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.627+0000 7f2569000640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f25640d0c00 0x7f256414b120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:13.621 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.627+0000 7f2569000640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f256416bd00 con 0x7f25640b60a0 2024-03-31T15:03:13.621 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.627+0000 7f2569000640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f256416bb80 con 0x7f256406b620 2024-03-31T15:03:13.621 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.627+0000 7f2569000640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f256416be80 con 0x7f25640d0c00 2024-03-31T15:03:13.621 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.627+0000 7f2561e00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f256406b620 0x7f256413ab10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:13.622 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.627+0000 7f2562800640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f25640d0c00 0x7f256414b120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:13.622 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.627+0000 7f2562800640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f25640d0c00 0x7f256414b120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3301/0 says I am v2:172.21.15.89:56156/0 (socket says 172.21.15.89:56156) 2024-03-31T15:03:13.622 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.627+0000 7f2562800640 1 -- 172.21.15.89:0/1588127755 learned_addr learned my addr 172.21.15.89:0/1588127755 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-03-31T15:03:13.622 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.627+0000 7f255be00640 1 --2- 172.21.15.89:0/1588127755 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f25640b60a0 0x7f256414c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:13.622 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.631+0000 7f2562800640 1 -- 172.21.15.89:0/1588127755 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f256406b620 msgr2=0x7f256413ab10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:13.622 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.631+0000 7f2562800640 1 --2- 172.21.15.89:0/1588127755 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f256406b620 0x7f256413ab10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:13.622 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.631+0000 7f2562800640 1 -- 172.21.15.89:0/1588127755 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f25640b60a0 msgr2=0x7f256414c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:13.622 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.631+0000 7f2562800640 1 --2- 172.21.15.89:0/1588127755 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f25640b60a0 0x7f256414c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:13.622 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.631+0000 7f2562800640 1 -- 172.21.15.89:0/1588127755 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f256414d2e0 con 0x7f25640d0c00 2024-03-31T15:03:13.622 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.631+0000 7f2561e00640 1 --2- 172.21.15.89:0/1588127755 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f256406b620 0x7f256413ab10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:03:13.622 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.631+0000 7f2562800640 1 --2- 172.21.15.89:0/1588127755 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f25640d0c00 0x7f256414b120 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f255c009d30 tx=0x7f255c00c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:03:13.623 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.631+0000 7f255a000640 1 -- 172.21.15.89:0/1588127755 <== mon.2 v2:172.21.15.89:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f255c018020 con 0x7f25640d0c00 2024-03-31T15:03:13.623 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.631+0000 7f255a000640 1 -- 172.21.15.89:0/1588127755 <== mon.2 v2:172.21.15.89:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f255c014480 con 0x7f25640d0c00 2024-03-31T15:03:13.623 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.631+0000 7f2569000640 1 -- 172.21.15.89:0/1588127755 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f256414cd20 con 0x7f25640d0c00 2024-03-31T15:03:13.624 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.631+0000 7f255a000640 1 -- 172.21.15.89:0/1588127755 <== mon.2 v2:172.21.15.89:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f255c014870 con 0x7f25640d0c00 2024-03-31T15:03:13.624 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.631+0000 7f2569000640 1 -- 172.21.15.89:0/1588127755 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f256414a920 con 0x7f25640d0c00 2024-03-31T15:03:13.624 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.631+0000 7f255a000640 1 -- 172.21.15.89:0/1588127755 <== mon.2 v2:172.21.15.89:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f255c01b020 con 0x7f25640d0c00 2024-03-31T15:03:13.624 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.631+0000 7f255a000640 1 -- 172.21.15.89:0/1588127755 <== mon.2 v2:172.21.15.89:3301/0 5 ==== osd_map(8..8 src has 1..8) ==== 1415+0+0 (secure 0 0 0) 0x7f255c01d9f0 con 0x7f25640d0c00 2024-03-31T15:03:13.624 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.631+0000 7f2569000640 1 -- 172.21.15.89:0/1588127755 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2564069950 con 0x7f25640d0c00 2024-03-31T15:03:13.625 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.635+0000 7f255a000640 1 -- 172.21.15.89:0/1588127755 <== mon.2 v2:172.21.15.89:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f255c022070 con 0x7f25640d0c00 2024-03-31T15:03:13.691 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.699+0000 7f2569000640 1 -- 172.21.15.89:0/1588127755 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_command({"prefix": "osd new", "uuid": "81ce0c19-2554-4cae-acbe-66a05e9e9b9b", "id": 6} v 0) -- 0x7f2564169140 con 0x7f25640d0c00 2024-03-31T15:03:13.697 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.707+0000 7f255a000640 1 -- 172.21.15.89:0/1588127755 <== mon.2 v2:172.21.15.89:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "81ce0c19-2554-4cae-acbe-66a05e9e9b9b", "id": 6}]=0 v9) ==== 112+0+2 (secure 0 0 0) 0x7f255c010040 con 0x7f25640d0c00 2024-03-31T15:03:13.698 INFO:teuthology.orchestra.run.smithi089.stdout:6 2024-03-31T15:03:13.699 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.707+0000 7f2569000640 1 -- 172.21.15.89:0/1588127755 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f25640d0c00 msgr2=0x7f256414b120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:13.699 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.707+0000 7f2569000640 1 --2- 172.21.15.89:0/1588127755 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f25640d0c00 0x7f256414b120 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f255c009d30 tx=0x7f255c00c490 comp rx=0 tx=0).stop 2024-03-31T15:03:13.699 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.707+0000 7f2569000640 1 -- 172.21.15.89:0/1588127755 shutdown_connections 2024-03-31T15:03:13.699 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.707+0000 7f2569000640 1 --2- 172.21.15.89:0/1588127755 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f25640d0c00 0x7f256414b120 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:13.699 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.707+0000 7f2569000640 1 --2- 172.21.15.89:0/1588127755 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f25640b60a0 0x7f256414c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:13.699 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.707+0000 7f2569000640 1 --2- 172.21.15.89:0/1588127755 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f256406b620 0x7f256413ab10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:13.699 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.707+0000 7f2569000640 1 -- 172.21.15.89:0/1588127755 >> 172.21.15.89:0/1588127755 conn(0x7f256406e670 msgr2=0x7f256415d0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:03:13.699 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.707+0000 7f2569000640 1 -- 172.21.15.89:0/1588127755 shutdown_connections 2024-03-31T15:03:13.699 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.707+0000 7f2569000640 1 -- 172.21.15.89:0/1588127755 wait complete. 2024-03-31T15:03:13.714 INFO:tasks.ceph.mgr.x.smithi089.stderr:2024-03-31T15:03:13.723+0000 7f078c909180 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-31T15:03:13.716 DEBUG:teuthology.orchestra.run.smithi089:> sudo ceph --cluster ceph osd new 85e4313b-d37d-442e-8042-153bf20a847d 7 2024-03-31T15:03:13.718 INFO:tasks.ceph.mgr.y.smithi016.stderr:2024-03-31T15:03:13.722+0000 7f2c2622a180 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-31T15:03:13.787 INFO:tasks.ceph.mgr.x.smithi089.stderr:2024-03-31T15:03:13.795+0000 7f078c909180 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-31T15:03:13.792 INFO:tasks.ceph.mgr.y.smithi016.stderr:2024-03-31T15:03:13.798+0000 7f2c2622a180 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-31T15:03:13.851 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.859+0000 7f3c76200640 1 Processor -- start 2024-03-31T15:03:13.851 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.859+0000 7f3c76200640 1 -- start start 2024-03-31T15:03:13.852 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.859+0000 7f3c76200640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f3c70159d10 0x7f3c700675d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:13.853 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.859+0000 7f3c76200640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f3c700b4490 0x7f3c7006b620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:13.853 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.859+0000 7f3c76200640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3c7015d770 con 0x7f3c7015d310 2024-03-31T15:03:13.853 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.859+0000 7f3c76200640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f3c700d00f0 con 0x7f3c700b4490 2024-03-31T15:03:13.853 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.859+0000 7f3c76200640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f3c700cf4e0 con 0x7f3c70159d10 2024-03-31T15:03:13.853 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.859+0000 7f3c6f400640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f3c70159d10 0x7f3c700675d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:13.853 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.859+0000 7f3c6fe00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f3c700b4490 0x7f3c7006b620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:13.853 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.859+0000 7f3c6ea00640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f3c7015d310 0x7f3c700b3d80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:56144/0 (socket says 172.21.15.89:56144) 2024-03-31T15:03:13.853 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.859+0000 7f3c6ea00640 1 -- 172.21.15.89:0/417518533 learned_addr learned my addr 172.21.15.89:0/417518533 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-03-31T15:03:13.854 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.863+0000 7f3c6fe00640 1 -- 172.21.15.89:0/417518533 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f3c70159d10 msgr2=0x7f3c700675d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:03:13.854 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.863+0000 7f3c6fe00640 1 --2- 172.21.15.89:0/417518533 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f3c70159d10 0x7f3c700675d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:13.854 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.863+0000 7f3c6fe00640 1 -- 172.21.15.89:0/417518533 >> v1:172.21.15.89:6789/0 conn(0x7f3c7015d310 legacy=0x7f3c700b3d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:13.854 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.863+0000 7f3c6fe00640 1 -- 172.21.15.89:0/417518533 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c700cf7c0 con 0x7f3c700b4490 2024-03-31T15:03:13.854 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.863+0000 7f3c6e000640 1 -- 172.21.15.89:0/417518533 <== mon.1 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 688885759 0 0) 0x7f3c7015d770 con 0x7f3c7015d310 2024-03-31T15:03:13.854 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.863+0000 7f3c6f400640 1 --2- 172.21.15.89:0/417518533 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f3c70159d10 0x7f3c700675d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:03:13.856 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.863+0000 7f3c6fe00640 1 --2- 172.21.15.89:0/417518533 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3c700b4490 0x7f3c7006b620 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f3c68009d90 tx=0x7f3c6801c970 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3dad28b56fe36787 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:03:13.856 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.863+0000 7f3c6e000640 1 -- 172.21.15.89:0/417518533 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c680b2070 con 0x7f3c700b4490 2024-03-31T15:03:13.856 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.863+0000 7f3c6e000640 1 -- 172.21.15.89:0/417518533 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3c680a5920 con 0x7f3c700b4490 2024-03-31T15:03:13.856 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.863+0000 7f3c6e000640 1 -- 172.21.15.89:0/417518533 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c680a5cf0 con 0x7f3c700b4490 2024-03-31T15:03:13.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.863+0000 7f3c76200640 1 -- 172.21.15.89:0/417518533 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3c700b4490 msgr2=0x7f3c7006b620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:13.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.863+0000 7f3c76200640 1 --2- 172.21.15.89:0/417518533 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3c700b4490 0x7f3c7006b620 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f3c68009d90 tx=0x7f3c6801c970 comp rx=0 tx=0).stop 2024-03-31T15:03:13.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.867+0000 7f3c76200640 1 -- 172.21.15.89:0/417518533 shutdown_connections 2024-03-31T15:03:13.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.867+0000 7f3c76200640 1 --2- 172.21.15.89:0/417518533 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3c700b4490 0x7f3c7006b620 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:13.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.867+0000 7f3c76200640 1 --2- 172.21.15.89:0/417518533 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f3c70159d10 0x7f3c700675d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:13.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.867+0000 7f3c76200640 1 -- 172.21.15.89:0/417518533 >> 172.21.15.89:0/417518533 conn(0x7f3c700d07e0 msgr2=0x7f3c700d0bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:03:13.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.867+0000 7f3c76200640 1 -- 172.21.15.89:0/417518533 shutdown_connections 2024-03-31T15:03:13.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.867+0000 7f3c76200640 1 -- 172.21.15.89:0/417518533 wait complete. 2024-03-31T15:03:13.858 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.867+0000 7f3c76200640 1 Processor -- start 2024-03-31T15:03:13.859 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.867+0000 7f3c76200640 1 -- start start 2024-03-31T15:03:13.860 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.867+0000 7f3c76200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3c700b4490 0x7f3c701532a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:13.860 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.867+0000 7f3c76200640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f3c70159d10 0x7f3c7014ab90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:13.860 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.867+0000 7f3c76200640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3c7015d310 0x7f3c700b0d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:13.861 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.867+0000 7f3c76200640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f3c7006c4f0 con 0x7f3c700b4490 2024-03-31T15:03:13.861 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.867+0000 7f3c76200640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f3c7006c370 con 0x7f3c7015d310 2024-03-31T15:03:13.861 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.867+0000 7f3c76200640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f3c7006c670 con 0x7f3c70159d10 2024-03-31T15:03:13.861 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.867+0000 7f3c6ea00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f3c70159d10 0x7f3c7014ab90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:13.861 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.867+0000 7f3c6ea00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f3c70159d10 0x7f3c7014ab90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3301/0 says I am v2:172.21.15.89:56178/0 (socket says 172.21.15.89:56178) 2024-03-31T15:03:13.861 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.867+0000 7f3c6ea00640 1 -- 172.21.15.89:0/2921920721 learned_addr learned my addr 172.21.15.89:0/2921920721 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-03-31T15:03:13.861 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.867+0000 7f3c6fe00640 1 --2- 172.21.15.89:0/2921920721 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3c7015d310 0x7f3c700b0d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:13.861 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.867+0000 7f3c6f400640 1 --2- 172.21.15.89:0/2921920721 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3c700b4490 0x7f3c701532a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:13.861 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.867+0000 7f3c6fe00640 1 -- 172.21.15.89:0/2921920721 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f3c70159d10 msgr2=0x7f3c7014ab90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:13.861 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.867+0000 7f3c6fe00640 1 --2- 172.21.15.89:0/2921920721 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f3c70159d10 0x7f3c7014ab90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:13.861 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.867+0000 7f3c6fe00640 1 -- 172.21.15.89:0/2921920721 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3c700b4490 msgr2=0x7f3c701532a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:13.861 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.867+0000 7f3c6fe00640 1 --2- 172.21.15.89:0/2921920721 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3c700b4490 0x7f3c701532a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:13.861 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.867+0000 7f3c6fe00640 1 -- 172.21.15.89:0/2921920721 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c700b1670 con 0x7f3c7015d310 2024-03-31T15:03:13.861 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.867+0000 7f3c6ea00640 1 --2- 172.21.15.89:0/2921920721 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f3c70159d10 0x7f3c7014ab90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:03:13.863 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.867+0000 7f3c6fe00640 1 --2- 172.21.15.89:0/2921920721 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3c7015d310 0x7f3c700b0d90 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f3c680ac000 tx=0x7f3c68019710 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:03:13.863 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.871+0000 7f3c67400640 1 -- 172.21.15.89:0/2921920721 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c680b2050 con 0x7f3c7015d310 2024-03-31T15:03:13.863 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.871+0000 7f3c67400640 1 -- 172.21.15.89:0/2921920721 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3c680b3040 con 0x7f3c7015d310 2024-03-31T15:03:13.863 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.871+0000 7f3c76200640 1 -- 172.21.15.89:0/2921920721 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3c700b1950 con 0x7f3c7015d310 2024-03-31T15:03:13.863 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.871+0000 7f3c67400640 1 -- 172.21.15.89:0/2921920721 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c68003da0 con 0x7f3c7015d310 2024-03-31T15:03:13.863 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.871+0000 7f3c76200640 1 -- 172.21.15.89:0/2921920721 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3c700b1f10 con 0x7f3c7015d310 2024-03-31T15:03:13.863 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.871+0000 7f3c67400640 1 -- 172.21.15.89:0/2921920721 <== mon.1 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f3c680a57e0 con 0x7f3c7015d310 2024-03-31T15:03:13.863 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.871+0000 7f3c67400640 1 -- 172.21.15.89:0/2921920721 <== mon.1 v2:172.21.15.89:3300/0 5 ==== osd_map(9..9 src has 1..9) ==== 1530+0+0 (secure 0 0 0) 0x7f3c68004780 con 0x7f3c7015d310 2024-03-31T15:03:13.863 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.871+0000 7f3c76200640 1 -- 172.21.15.89:0/2921920721 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3c7015a5b0 con 0x7f3c7015d310 2024-03-31T15:03:13.864 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.871+0000 7f3c67400640 1 -- 172.21.15.89:0/2921920721 <== mon.1 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f3c680c0060 con 0x7f3c7015d310 2024-03-31T15:03:13.930 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.939+0000 7f3c76200640 1 -- 172.21.15.89:0/2921920721 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd new", "uuid": "85e4313b-d37d-442e-8042-153bf20a847d", "id": 7} v 0) -- 0x7f3c700b2250 con 0x7f3c7015d310 2024-03-31T15:03:13.938 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.947+0000 7f3c67400640 1 -- 172.21.15.89:0/2921920721 <== mon.1 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "85e4313b-d37d-442e-8042-153bf20a847d", "id": 7}]=0 v10) ==== 112+0+2 (secure 0 0 0) 0x7f3c700b2250 con 0x7f3c7015d310 2024-03-31T15:03:13.939 INFO:teuthology.orchestra.run.smithi089.stdout:7 2024-03-31T15:03:13.940 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.947+0000 7f3c76200640 1 -- 172.21.15.89:0/2921920721 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3c7015d310 msgr2=0x7f3c700b0d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:13.940 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.947+0000 7f3c76200640 1 --2- 172.21.15.89:0/2921920721 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3c7015d310 0x7f3c700b0d90 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f3c680ac000 tx=0x7f3c68019710 comp rx=0 tx=0).stop 2024-03-31T15:03:13.940 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.947+0000 7f3c76200640 1 -- 172.21.15.89:0/2921920721 shutdown_connections 2024-03-31T15:03:13.940 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.947+0000 7f3c76200640 1 --2- 172.21.15.89:0/2921920721 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3c7015d310 0x7f3c700b0d90 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:13.940 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.947+0000 7f3c76200640 1 --2- 172.21.15.89:0/2921920721 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f3c70159d10 0x7f3c7014ab90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:13.940 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.947+0000 7f3c76200640 1 --2- 172.21.15.89:0/2921920721 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3c700b4490 0x7f3c701532a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:13.940 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.947+0000 7f3c76200640 1 -- 172.21.15.89:0/2921920721 >> 172.21.15.89:0/2921920721 conn(0x7f3c700d07e0 msgr2=0x7f3c70160750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:03:13.940 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.947+0000 7f3c76200640 1 -- 172.21.15.89:0/2921920721 shutdown_connections 2024-03-31T15:03:13.940 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:13.947+0000 7f3c76200640 1 -- 172.21.15.89:0/2921920721 wait complete. 2024-03-31T15:03:13.957 INFO:tasks.ceph.osd.0:Restarting daemon 2024-03-31T15:03:13.957 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-03-31T15:03:13.960 INFO:tasks.ceph.osd.0:Started 2024-03-31T15:03:13.961 INFO:tasks.ceph.osd.1:Restarting daemon 2024-03-31T15:03:13.961 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-03-31T15:03:13.964 INFO:tasks.ceph.osd.1:Started 2024-03-31T15:03:13.964 INFO:tasks.ceph.osd.2:Restarting daemon 2024-03-31T15:03:13.964 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-03-31T15:03:13.967 INFO:tasks.ceph.osd.2:Started 2024-03-31T15:03:13.967 INFO:tasks.ceph.osd.3:Restarting daemon 2024-03-31T15:03:13.968 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-03-31T15:03:13.970 INFO:tasks.ceph.osd.3:Started 2024-03-31T15:03:13.971 INFO:tasks.ceph.osd.4:Restarting daemon 2024-03-31T15:03:13.971 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-03-31T15:03:13.974 INFO:tasks.ceph.osd.4:Started 2024-03-31T15:03:13.974 INFO:tasks.ceph.osd.5:Restarting daemon 2024-03-31T15:03:13.974 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-03-31T15:03:13.977 INFO:tasks.ceph.osd.5:Started 2024-03-31T15:03:13.977 INFO:tasks.ceph.osd.6:Restarting daemon 2024-03-31T15:03:13.977 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2024-03-31T15:03:13.980 INFO:tasks.ceph.osd.6:Started 2024-03-31T15:03:13.980 INFO:tasks.ceph.osd.7:Restarting daemon 2024-03-31T15:03:13.980 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2024-03-31T15:03:13.984 INFO:tasks.ceph.osd.7:Started 2024-03-31T15:03:13.984 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-31T15:03:14.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:14.198+0000 7fbf1b800640 1 Processor -- start 2024-03-31T15:03:14.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:14.198+0000 7fbf1b800640 1 -- start start 2024-03-31T15:03:14.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:14.198+0000 7fbf1b800640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fbf0c0e3070 0x7fbf0c057290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:14.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:14.198+0000 7fbf18e00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fbf0c0e3070 0x7fbf0c057290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:14.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:14.202+0000 7fbf18e00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fbf0c0e3070 0x7fbf0c057290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55964/0 (socket says 172.21.15.16:55964) 2024-03-31T15:03:14.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:14.202+0000 7fbf18e00640 1 -- 172.21.15.16:0/2385366050 learned_addr learned my addr 172.21.15.16:0/2385366050 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:03:14.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:14.198+0000 7fbf1b800640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbf0c058d10 con 0x7fbf0c0577d0 2024-03-31T15:03:14.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:14.202+0000 7fbf1b800640 1 -- 172.21.15.16:0/2385366050 --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbf0c057c70 con 0x7fbf0c053e70 2024-03-31T15:03:14.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:14.202+0000 7fbf1b800640 1 -- 172.21.15.16:0/2385366050 --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7fbf0c057f50 con 0x7fbf0c0e3070 2024-03-31T15:03:14.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:14.202+0000 7fbf13400640 1 -- 172.21.15.16:0/2385366050 <== mon.1 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4269189677 0 0) 0x7fbf0c057c70 con 0x7fbf0c053e70 2024-03-31T15:03:14.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:14.202+0000 7fbf13400640 1 -- 172.21.15.16:0/2385366050 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbefc003530 con 0x7fbf0c053e70 2024-03-31T15:03:14.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:14.202+0000 7fbf18e00640 1 -- 172.21.15.16:0/2385366050 >> v1:172.21.15.89:6789/0 conn(0x7fbf0c053e70 legacy=0x7fbf0c03c0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:14.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:14.202+0000 7fbf18e00640 1 -- 172.21.15.16:0/2385366050 >> v1:172.21.15.16:6789/0 conn(0x7fbf0c0577d0 legacy=0x7fbf0c053760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:14.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:14.202+0000 7fbf18e00640 1 -- 172.21.15.16:0/2385366050 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf0c058230 con 0x7fbf0c0e3070 2024-03-31T15:03:14.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:14.206+0000 7fbf18e00640 1 --2- 172.21.15.16:0/2385366050 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbf0c0e3070 0x7fbf0c057290 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fbf080098a0 tx=0x7fbf0801ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db32a183c54cad7d server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:03:14.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:14.206+0000 7fbf13400640 1 -- 172.21.15.16:0/2385366050 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf080b5070 con 0x7fbf0c0e3070 2024-03-31T15:03:14.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:14.206+0000 7fbf13400640 1 -- 172.21.15.16:0/2385366050 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbf080a89f0 con 0x7fbf0c0e3070 2024-03-31T15:03:14.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:14.206+0000 7fbf13400640 1 -- 172.21.15.16:0/2385366050 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf080a8dc0 con 0x7fbf0c0e3070 2024-03-31T15:03:14.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:14.206+0000 7fbf1b800640 1 -- 172.21.15.16:0/2385366050 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbf0c0e3070 msgr2=0x7fbf0c057290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:14.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:14.206+0000 7fbf1b800640 1 --2- 172.21.15.16:0/2385366050 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbf0c0e3070 0x7fbf0c057290 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fbf080098a0 tx=0x7fbf0801ecf0 comp rx=0 tx=0).stop 2024-03-31T15:03:14.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:14.206+0000 7fbf1b800640 1 -- 172.21.15.16:0/2385366050 shutdown_connections 2024-03-31T15:03:14.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:14.206+0000 7fbf1b800640 1 --2- 172.21.15.16:0/2385366050 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbf0c0e3070 0x7fbf0c057290 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:14.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:14.206+0000 7fbf1b800640 1 -- 172.21.15.16:0/2385366050 >> 172.21.15.16:0/2385366050 conn(0x7fbf0c058f00 msgr2=0x7fbf0c059310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:03:14.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:14.206+0000 7fbf1b800640 1 -- 172.21.15.16:0/2385366050 shutdown_connections 2024-03-31T15:03:14.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:14.206+0000 7fbf1b800640 1 -- 172.21.15.16:0/2385366050 wait complete. 2024-03-31T15:03:14.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:14.206+0000 7fbf1b800640 1 Processor -- start 2024-03-31T15:03:14.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:14.206+0000 7fbf1b800640 1 -- start start 2024-03-31T15:03:14.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:14.210+0000 7fbf1b800640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fbf0c053e70 0x7fbf0c0d53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:14.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:14.210+0000 7fbf1b800640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbf0c0577d0 0x7fbf0c050b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:14.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:14.210+0000 7fbf1b800640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbf0c0e3070 0x7fbf0c0e55c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:14.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:14.210+0000 7fbf1b800640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fbf0c0543b0 con 0x7fbf0c0577d0 2024-03-31T15:03:14.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:14.210+0000 7fbf1b800640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fbf0c0587f0 con 0x7fbf0c0e3070 2024-03-31T15:03:14.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:14.210+0000 7fbf1b800640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fbf0c054530 con 0x7fbf0c053e70 2024-03-31T15:03:14.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:14.210+0000 7fbf13e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbf0c0577d0 0x7fbf0c050b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:14.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:14.210+0000 7fbf13e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbf0c0577d0 0x7fbf0c050b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55970/0 (socket says 172.21.15.16:55970) 2024-03-31T15:03:14.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:14.210+0000 7fbf13e00640 1 -- 172.21.15.16:0/3069320875 learned_addr learned my addr 172.21.15.16:0/3069320875 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:03:14.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:14.210+0000 7fbf18e00640 1 --2- 172.21.15.16:0/3069320875 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fbf0c053e70 0x7fbf0c0d53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:14.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:14.210+0000 7fbf19800640 1 --2- 172.21.15.16:0/3069320875 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbf0c0e3070 0x7fbf0c0e55c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:14.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:14.210+0000 7fbf13e00640 1 -- 172.21.15.16:0/3069320875 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fbf0c053e70 msgr2=0x7fbf0c0d53e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:14.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:14.210+0000 7fbf13e00640 1 --2- 172.21.15.16:0/3069320875 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fbf0c053e70 0x7fbf0c0d53e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:14.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:14.210+0000 7fbf13e00640 1 -- 172.21.15.16:0/3069320875 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbf0c0e3070 msgr2=0x7fbf0c0e55c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:14.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:14.210+0000 7fbf13e00640 1 --2- 172.21.15.16:0/3069320875 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbf0c0e3070 0x7fbf0c0e55c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:14.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:14.210+0000 7fbf13e00640 1 -- 172.21.15.16:0/3069320875 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf0c0e4800 con 0x7fbf0c0577d0 2024-03-31T15:03:14.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:14.210+0000 7fbf19800640 1 --2- 172.21.15.16:0/3069320875 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbf0c0e3070 0x7fbf0c0e55c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:03:14.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:14.210+0000 7fbf13e00640 1 --2- 172.21.15.16:0/3069320875 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbf0c0577d0 0x7fbf0c050b50 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fbf0400c460 tx=0x7fbf0400c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:03:14.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:14.210+0000 7fbf10c00640 1 -- 172.21.15.16:0/3069320875 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf04016020 con 0x7fbf0c0577d0 2024-03-31T15:03:14.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:14.210+0000 7fbf10c00640 1 -- 172.21.15.16:0/3069320875 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbf040052d0 con 0x7fbf0c0577d0 2024-03-31T15:03:14.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:14.210+0000 7fbf1b800640 1 -- 172.21.15.16:0/3069320875 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbf0c0e4ae0 con 0x7fbf0c0577d0 2024-03-31T15:03:14.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:14.210+0000 7fbf10c00640 1 -- 172.21.15.16:0/3069320875 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf04007260 con 0x7fbf0c0577d0 2024-03-31T15:03:14.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:14.210+0000 7fbf1b800640 1 -- 172.21.15.16:0/3069320875 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbf0c0d5920 con 0x7fbf0c0577d0 2024-03-31T15:03:14.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:14.210+0000 7fbf10c00640 1 -- 172.21.15.16:0/3069320875 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fbf040067e0 con 0x7fbf0c0577d0 2024-03-31T15:03:14.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:14.210+0000 7fbf10c00640 1 -- 172.21.15.16:0/3069320875 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(10..10 src has 1..10) ==== 1645+0+0 (secure 0 0 0) 0x7fbf04012d40 con 0x7fbf0c0577d0 2024-03-31T15:03:14.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:14.210+0000 7fbf1b800640 1 -- 172.21.15.16:0/3069320875 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbf0c005720 con 0x7fbf0c0577d0 2024-03-31T15:03:14.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:14.214+0000 7fbf10c00640 1 -- 172.21.15.16:0/3069320875 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7fbf04020020 con 0x7fbf0c0577d0 2024-03-31T15:03:14.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:14.274+0000 7fbf1b800640 1 -- 172.21.15.16:0/3069320875 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fbf0c038100 con 0x7fbf0c0577d0 2024-03-31T15:03:14.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:14.274+0000 7fbf10c00640 1 -- 172.21.15.16:0/3069320875 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) ==== 74+0+7058 (secure 0 0 0) 0x7fbf04005480 con 0x7fbf0c0577d0 2024-03-31T15:03:14.268 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:03:14.269 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":10,"fsid":"286eb5da-fb47-4550-b290-d9519c13b9a5","created":"2024-03-31T15:03:11.438952+0000","modified":"2024-03-31T15:03:13.941667+0000","last_up_change":"0.000000","last_in_change":"2024-03-31T15:03:13.941667+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"92a66450-ef19-40d8-ac3d-d760b23b0bda","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"da9d1f2b-1968-4959-b843-faa231988e92","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"01a73662-9a21-4b1f-bcbb-42b9636a4638","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c7771fd3-e18e-4500-ac38-23cf10ebcd75","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"88260730-1482-4fbd-a40f-cb1124e9df4a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"4cb2c00c-fae0-4dd8-9d27-ec4f0093ccf7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"81ce0c19-2554-4cae-acbe-66a05e9e9b9b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"85e4313b-d37d-442e-8042-153bf20a847d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-31T15:03:14.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:14.274+0000 7fbf1b800640 1 -- 172.21.15.16:0/3069320875 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbf0c0577d0 msgr2=0x7fbf0c050b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:14.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:14.274+0000 7fbf1b800640 1 --2- 172.21.15.16:0/3069320875 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbf0c0577d0 0x7fbf0c050b50 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fbf0400c460 tx=0x7fbf0400c920 comp rx=0 tx=0).stop 2024-03-31T15:03:14.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:14.274+0000 7fbf1b800640 1 -- 172.21.15.16:0/3069320875 shutdown_connections 2024-03-31T15:03:14.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:14.274+0000 7fbf1b800640 1 --2- 172.21.15.16:0/3069320875 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbf0c0e3070 0x7fbf0c0e55c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:14.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:14.274+0000 7fbf1b800640 1 --2- 172.21.15.16:0/3069320875 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbf0c0577d0 0x7fbf0c050b50 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:14.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:14.274+0000 7fbf1b800640 1 --2- 172.21.15.16:0/3069320875 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fbf0c053e70 0x7fbf0c0d53e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:14.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:14.274+0000 7fbf1b800640 1 -- 172.21.15.16:0/3069320875 >> 172.21.15.16:0/3069320875 conn(0x7fbf0c058f00 msgr2=0x7fbf0c03e0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:03:14.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:14.274+0000 7fbf1b800640 1 -- 172.21.15.16:0/3069320875 shutdown_connections 2024-03-31T15:03:14.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:14.274+0000 7fbf1b800640 1 -- 172.21.15.16:0/3069320875 wait complete. 2024-03-31T15:03:14.287 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-03-31T15:03:14.287 INFO:tasks.ceph:Waiting for OSDs to come up 2024-03-31T15:03:14.437 INFO:tasks.ceph.mgr.x.smithi089.stderr:/usr/lib/python3/dist-packages/scipy/__init__.py:67: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2024-03-31T15:03:14.437 INFO:tasks.ceph.mgr.x.smithi089.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2024-03-31T15:03:14.437 INFO:tasks.ceph.mgr.x.smithi089.stderr: from numpy import show_config as show_numpy_config 2024-03-31T15:03:14.489 INFO:tasks.ceph.mgr.x.smithi089.stderr:2024-03-31T15:03:14.495+0000 7f078c909180 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-31T15:03:14.502 INFO:tasks.ceph.mgr.y.smithi016.stderr:/usr/lib/python3/dist-packages/scipy/__init__.py:67: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2024-03-31T15:03:14.502 INFO:tasks.ceph.mgr.y.smithi016.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2024-03-31T15:03:14.502 INFO:tasks.ceph.mgr.y.smithi016.stderr: from numpy import show_config as show_numpy_config 2024-03-31T15:03:14.550 INFO:tasks.ceph.mgr.y.smithi016.stderr:2024-03-31T15:03:14.554+0000 7f2c2622a180 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-31T15:03:14.614 INFO:tasks.ceph.osd.4.smithi089.stderr:2024-03-31T15:03:14.623+0000 7fae012de7c0 -1 Falling back to public interface 2024-03-31T15:03:14.623 INFO:tasks.ceph.osd.7.smithi089.stderr:2024-03-31T15:03:14.631+0000 7f78dc09e7c0 -1 Falling back to public interface 2024-03-31T15:03:14.623 INFO:tasks.ceph.osd.5.smithi089.stderr:2024-03-31T15:03:14.631+0000 7f348892d7c0 -1 Falling back to public interface 2024-03-31T15:03:14.623 INFO:tasks.ceph.osd.6.smithi089.stderr:2024-03-31T15:03:14.631+0000 7f5a667887c0 -1 Falling back to public interface 2024-03-31T15:03:14.625 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-03-31T15:03:14.630+0000 7fae5342b7c0 -1 Falling back to public interface 2024-03-31T15:03:14.630 INFO:tasks.ceph.osd.6.smithi089.stderr:2024-03-31T15:03:14.639+0000 7f5a65200640 -1 PosixStack listen unable to listen on v2:0.0.0.0:6826/0: (98) Address already in use 2024-03-31T15:03:14.636 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-03-31T15:03:14.642+0000 7fa5098937c0 -1 Falling back to public interface 2024-03-31T15:03:14.637 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-03-31T15:03:14.642+0000 7f5f211577c0 -1 Falling back to public interface 2024-03-31T15:03:14.640 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-03-31T15:03:14.646+0000 7fe16e02b7c0 -1 Falling back to public interface 2024-03-31T15:03:14.642 INFO:tasks.ceph.mgr.x.smithi089.stderr:2024-03-31T15:03:14.651+0000 7f078c909180 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-31T15:03:14.714 INFO:tasks.ceph.mgr.y.smithi016.stderr:2024-03-31T15:03:14.718+0000 7f2c2622a180 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-31T15:03:14.809 INFO:tasks.ceph.mgr.x.smithi089.stderr:2024-03-31T15:03:14.819+0000 7f078c909180 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-31T15:03:14.887 INFO:tasks.ceph.mgr.x.smithi089.stderr:2024-03-31T15:03:14.895+0000 7f078c909180 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-31T15:03:14.900 INFO:tasks.ceph.mgr.y.smithi016.stderr:2024-03-31T15:03:14.906+0000 7f2c2622a180 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-31T15:03:14.977 INFO:tasks.ceph.mgr.y.smithi016.stderr:2024-03-31T15:03:14.982+0000 7f2c2622a180 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-31T15:03:15.036 INFO:tasks.ceph.mgr.x.smithi089.stderr:2024-03-31T15:03:15.043+0000 7f078c909180 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-31T15:03:15.092 DEBUG:teuthology.orchestra.run.smithi016:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-31T15:03:15.134 INFO:tasks.ceph.mgr.y.smithi016.stderr:2024-03-31T15:03:15.138+0000 7f2c2622a180 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-31T15:03:15.223 INFO:teuthology.misc.health.smithi016.stderr:2024-03-31T15:03:15.226+0000 7fbe4be00640 1 Processor -- start 2024-03-31T15:03:15.223 INFO:teuthology.misc.health.smithi016.stderr:2024-03-31T15:03:15.226+0000 7fbe4be00640 1 -- start start 2024-03-31T15:03:15.225 INFO:teuthology.misc.health.smithi016.stderr:2024-03-31T15:03:15.230+0000 7fbe4be00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fbe4415c270 0x7fbe44070270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:15.225 INFO:teuthology.misc.health.smithi016.stderr:2024-03-31T15:03:15.230+0000 7fbe4be00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fbe440cc7e0 0x7fbe440ce3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:15.225 INFO:teuthology.misc.health.smithi016.stderr:2024-03-31T15:03:15.230+0000 7fbe4be00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fbe440d13d0 0x7fbe440d1830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:15.225 INFO:teuthology.misc.health.smithi016.stderr:2024-03-31T15:03:15.230+0000 7fbe4be00640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7fbe440ce900 con 0x7fbe4415c270 2024-03-31T15:03:15.225 INFO:teuthology.misc.health.smithi016.stderr:2024-03-31T15:03:15.230+0000 7fbe4be00640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7fbe440d1d70 con 0x7fbe440d13d0 2024-03-31T15:03:15.225 INFO:teuthology.misc.health.smithi016.stderr:2024-03-31T15:03:15.230+0000 7fbe4be00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fbe44071970 con 0x7fbe440cc7e0 2024-03-31T15:03:15.225 INFO:teuthology.misc.health.smithi016.stderr:2024-03-31T15:03:15.230+0000 7fbe49400640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fbe4415c270 0x7fbe44070270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:15.225 INFO:teuthology.misc.health.smithi016.stderr:2024-03-31T15:03:15.230+0000 7fbe49400640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fbe4415c270 0x7fbe44070270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55986/0 (socket says 172.21.15.16:55986) 2024-03-31T15:03:15.225 INFO:teuthology.misc.health.smithi016.stderr:2024-03-31T15:03:15.230+0000 7fbe49400640 1 -- 172.21.15.16:0/255659800 learned_addr learned my addr 172.21.15.16:0/255659800 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:03:15.225 INFO:teuthology.misc.health.smithi016.stderr:2024-03-31T15:03:15.230+0000 7fbe48a00640 1 --2- 172.21.15.16:0/255659800 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fbe440cc7e0 0x7fbe440ce3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:15.225 INFO:teuthology.misc.health.smithi016.stderr:2024-03-31T15:03:15.230+0000 7fbe49e00640 1 --2- 172.21.15.16:0/255659800 >> v2:172.21.15.89:3300/0 conn(0x7fbe440d13d0 0x7fbe440d1830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:15.225 INFO:teuthology.misc.health.smithi016.stderr:2024-03-31T15:03:15.230+0000 7fbe49400640 1 -- 172.21.15.16:0/255659800 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fbe440cc7e0 msgr2=0x7fbe440ce3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:03:15.226 INFO:teuthology.misc.health.smithi016.stderr:2024-03-31T15:03:15.230+0000 7fbe49400640 1 --2- 172.21.15.16:0/255659800 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fbe440cc7e0 0x7fbe440ce3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:15.226 INFO:teuthology.misc.health.smithi016.stderr:2024-03-31T15:03:15.230+0000 7fbe49400640 1 -- 172.21.15.16:0/255659800 >> v2:172.21.15.89:3300/0 conn(0x7fbe440d13d0 msgr2=0x7fbe440d1830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:03:15.226 INFO:teuthology.misc.health.smithi016.stderr:2024-03-31T15:03:15.230+0000 7fbe49400640 1 --2- 172.21.15.16:0/255659800 >> v2:172.21.15.89:3300/0 conn(0x7fbe440d13d0 0x7fbe440d1830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:15.226 INFO:teuthology.misc.health.smithi016.stderr:2024-03-31T15:03:15.230+0000 7fbe49400640 1 -- 172.21.15.16:0/255659800 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe4415ca00 con 0x7fbe4415c270 2024-03-31T15:03:15.226 INFO:teuthology.misc.health.smithi016.stderr:2024-03-31T15:03:15.230+0000 7fbe48a00640 1 --2- 172.21.15.16:0/255659800 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fbe440cc7e0 0x7fbe440ce3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:03:15.226 INFO:teuthology.misc.health.smithi016.stderr:2024-03-31T15:03:15.230+0000 7fbe49e00640 1 --2- 172.21.15.16:0/255659800 >> v2:172.21.15.89:3300/0 conn(0x7fbe440d13d0 0x7fbe440d1830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:03:15.228 INFO:teuthology.misc.health.smithi016.stderr:2024-03-31T15:03:15.234+0000 7fbe49400640 1 --2- 172.21.15.16:0/255659800 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbe4415c270 0x7fbe44070270 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fbe440b8220 tx=0x7fbe3801c460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2319106078a0c4b2 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:03:15.228 INFO:teuthology.misc.health.smithi016.stderr:2024-03-31T15:03:15.234+0000 7fbe43e00640 1 -- 172.21.15.16:0/255659800 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe380ab620 con 0x7fbe4415c270 2024-03-31T15:03:15.228 INFO:teuthology.misc.health.smithi016.stderr:2024-03-31T15:03:15.234+0000 7fbe43e00640 1 -- 172.21.15.16:0/255659800 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbe380b0070 con 0x7fbe4415c270 2024-03-31T15:03:15.228 INFO:teuthology.misc.health.smithi016.stderr:2024-03-31T15:03:15.234+0000 7fbe43e00640 1 -- 172.21.15.16:0/255659800 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe380ab9e0 con 0x7fbe4415c270 2024-03-31T15:03:15.229 INFO:teuthology.misc.health.smithi016.stderr:2024-03-31T15:03:15.234+0000 7fbe4be00640 1 -- 172.21.15.16:0/255659800 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbe4415c270 msgr2=0x7fbe44070270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:15.229 INFO:teuthology.misc.health.smithi016.stderr:2024-03-31T15:03:15.234+0000 7fbe4be00640 1 --2- 172.21.15.16:0/255659800 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbe4415c270 0x7fbe44070270 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fbe440b8220 tx=0x7fbe3801c460 comp rx=0 tx=0).stop 2024-03-31T15:03:15.229 INFO:teuthology.misc.health.smithi016.stderr:2024-03-31T15:03:15.234+0000 7fbe4be00640 1 -- 172.21.15.16:0/255659800 shutdown_connections 2024-03-31T15:03:15.229 INFO:teuthology.misc.health.smithi016.stderr:2024-03-31T15:03:15.234+0000 7fbe4be00640 1 --2- 172.21.15.16:0/255659800 >> v2:172.21.15.89:3300/0 conn(0x7fbe440d13d0 0x7fbe440d1830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:15.230 INFO:teuthology.misc.health.smithi016.stderr:2024-03-31T15:03:15.234+0000 7fbe4be00640 1 --2- 172.21.15.16:0/255659800 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fbe440cc7e0 0x7fbe440ce3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:15.230 INFO:teuthology.misc.health.smithi016.stderr:2024-03-31T15:03:15.234+0000 7fbe4be00640 1 --2- 172.21.15.16:0/255659800 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbe4415c270 0x7fbe44070270 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:15.230 INFO:teuthology.misc.health.smithi016.stderr:2024-03-31T15:03:15.234+0000 7fbe4be00640 1 -- 172.21.15.16:0/255659800 >> 172.21.15.16:0/255659800 conn(0x7fbe440b3ca0 msgr2=0x7fbe440b40b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:03:15.230 INFO:teuthology.misc.health.smithi016.stderr:2024-03-31T15:03:15.234+0000 7fbe4be00640 1 -- 172.21.15.16:0/255659800 shutdown_connections 2024-03-31T15:03:15.230 INFO:teuthology.misc.health.smithi016.stderr:2024-03-31T15:03:15.234+0000 7fbe4be00640 1 -- 172.21.15.16:0/255659800 wait complete. 2024-03-31T15:03:15.231 INFO:teuthology.misc.health.smithi016.stderr:2024-03-31T15:03:15.234+0000 7fbe4be00640 1 Processor -- start 2024-03-31T15:03:15.231 INFO:teuthology.misc.health.smithi016.stderr:2024-03-31T15:03:15.234+0000 7fbe4be00640 1 -- start start 2024-03-31T15:03:15.232 INFO:teuthology.misc.health.smithi016.stderr:2024-03-31T15:03:15.238+0000 7fbe4be00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fbe440cc7e0 0x7fbe440cbc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:15.232 INFO:teuthology.misc.health.smithi016.stderr:2024-03-31T15:03:15.238+0000 7fbe4be00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbe440d13d0 0x7fbe44158210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:15.232 INFO:teuthology.misc.health.smithi016.stderr:2024-03-31T15:03:15.238+0000 7fbe4be00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbe4415c270 0x7fbe440c7d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:15.232 INFO:teuthology.misc.health.smithi016.stderr:2024-03-31T15:03:15.238+0000 7fbe4be00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fbe4415c650 con 0x7fbe440d13d0 2024-03-31T15:03:15.232 INFO:teuthology.misc.health.smithi016.stderr:2024-03-31T15:03:15.238+0000 7fbe4be00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fbe4415d580 con 0x7fbe4415c270 2024-03-31T15:03:15.232 INFO:teuthology.misc.health.smithi016.stderr:2024-03-31T15:03:15.238+0000 7fbe4be00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fbe440ca130 con 0x7fbe440cc7e0 2024-03-31T15:03:15.232 INFO:teuthology.misc.health.smithi016.stderr:2024-03-31T15:03:15.238+0000 7fbe48a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbe440d13d0 0x7fbe44158210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:15.233 INFO:teuthology.misc.health.smithi016.stderr:2024-03-31T15:03:15.238+0000 7fbe49400640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fbe440cc7e0 0x7fbe440cbc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:15.233 INFO:teuthology.misc.health.smithi016.stderr:2024-03-31T15:03:15.238+0000 7fbe48a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbe440d13d0 0x7fbe44158210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55992/0 (socket says 172.21.15.16:55992) 2024-03-31T15:03:15.233 INFO:teuthology.misc.health.smithi016.stderr:2024-03-31T15:03:15.238+0000 7fbe48a00640 1 -- 172.21.15.16:0/2233613028 learned_addr learned my addr 172.21.15.16:0/2233613028 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:03:15.233 INFO:teuthology.misc.health.smithi016.stderr:2024-03-31T15:03:15.238+0000 7fbe49e00640 1 --2- 172.21.15.16:0/2233613028 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbe4415c270 0x7fbe440c7d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:15.234 INFO:teuthology.misc.health.smithi016.stderr:2024-03-31T15:03:15.238+0000 7fbe48a00640 1 -- 172.21.15.16:0/2233613028 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fbe440cc7e0 msgr2=0x7fbe440cbc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:15.234 INFO:teuthology.misc.health.smithi016.stderr:2024-03-31T15:03:15.238+0000 7fbe48a00640 1 --2- 172.21.15.16:0/2233613028 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fbe440cc7e0 0x7fbe440cbc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:15.234 INFO:teuthology.misc.health.smithi016.stderr:2024-03-31T15:03:15.238+0000 7fbe48a00640 1 -- 172.21.15.16:0/2233613028 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbe4415c270 msgr2=0x7fbe440c7d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:15.234 INFO:teuthology.misc.health.smithi016.stderr:2024-03-31T15:03:15.238+0000 7fbe48a00640 1 --2- 172.21.15.16:0/2233613028 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbe4415c270 0x7fbe440c7d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:15.235 INFO:teuthology.misc.health.smithi016.stderr:2024-03-31T15:03:15.238+0000 7fbe48a00640 1 -- 172.21.15.16:0/2233613028 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe440ca410 con 0x7fbe440d13d0 2024-03-31T15:03:15.235 INFO:teuthology.misc.health.smithi016.stderr:2024-03-31T15:03:15.238+0000 7fbe49400640 1 --2- 172.21.15.16:0/2233613028 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fbe440cc7e0 0x7fbe440cbc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:03:15.235 INFO:teuthology.misc.health.smithi016.stderr:2024-03-31T15:03:15.238+0000 7fbe48a00640 1 --2- 172.21.15.16:0/2233613028 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbe440d13d0 0x7fbe44158210 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fbe30009a50 tx=0x7fbe30009f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:03:15.235 INFO:teuthology.misc.health.smithi016.stderr:2024-03-31T15:03:15.238+0000 7fbe41600640 1 -- 172.21.15.16:0/2233613028 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe3000f020 con 0x7fbe440d13d0 2024-03-31T15:03:15.235 INFO:teuthology.misc.health.smithi016.stderr:2024-03-31T15:03:15.238+0000 7fbe41600640 1 -- 172.21.15.16:0/2233613028 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbe3000f1e0 con 0x7fbe440d13d0 2024-03-31T15:03:15.235 INFO:teuthology.misc.health.smithi016.stderr:2024-03-31T15:03:15.238+0000 7fbe41600640 1 -- 172.21.15.16:0/2233613028 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe3000f390 con 0x7fbe440d13d0 2024-03-31T15:03:15.235 INFO:teuthology.misc.health.smithi016.stderr:2024-03-31T15:03:15.238+0000 7fbe4be00640 1 -- 172.21.15.16:0/2233613028 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbe440ca6f0 con 0x7fbe440d13d0 2024-03-31T15:03:15.235 INFO:teuthology.misc.health.smithi016.stderr:2024-03-31T15:03:15.238+0000 7fbe4be00640 1 -- 172.21.15.16:0/2233613028 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbe440cc160 con 0x7fbe440d13d0 2024-03-31T15:03:15.235 INFO:teuthology.misc.health.smithi016.stderr:2024-03-31T15:03:15.238+0000 7fbe41600640 1 -- 172.21.15.16:0/2233613028 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fbe30004270 con 0x7fbe440d13d0 2024-03-31T15:03:15.236 INFO:teuthology.misc.health.smithi016.stderr:2024-03-31T15:03:15.238+0000 7fbe41600640 1 -- 172.21.15.16:0/2233613028 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(10..10 src has 1..10) ==== 1645+0+0 (secure 0 0 0) 0x7fbe30011d40 con 0x7fbe440d13d0 2024-03-31T15:03:15.236 INFO:teuthology.misc.health.smithi016.stderr:2024-03-31T15:03:15.238+0000 7fbe4be00640 1 -- 172.21.15.16:0/2233613028 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbe440cac10 con 0x7fbe440d13d0 2024-03-31T15:03:15.236 INFO:teuthology.misc.health.smithi016.stderr:2024-03-31T15:03:15.242+0000 7fbe41600640 1 -- 172.21.15.16:0/2233613028 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7fbe440cac10 con 0x7fbe440d13d0 2024-03-31T15:03:15.296 INFO:teuthology.misc.health.smithi016.stderr:2024-03-31T15:03:15.302+0000 7fbe4be00640 1 -- 172.21.15.16:0/2233613028 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fbe44070cb0 con 0x7fbe440d13d0 2024-03-31T15:03:15.297 INFO:teuthology.misc.health.smithi016.stderr:2024-03-31T15:03:15.302+0000 7fbe41600640 1 -- 172.21.15.16:0/2233613028 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) ==== 74+0+7058 (secure 0 0 0) 0x7fbe30003990 con 0x7fbe440d13d0 2024-03-31T15:03:15.297 INFO:teuthology.misc.health.smithi016.stdout: 2024-03-31T15:03:15.297 INFO:teuthology.misc.health.smithi016.stdout:{"epoch":10,"fsid":"286eb5da-fb47-4550-b290-d9519c13b9a5","created":"2024-03-31T15:03:11.438952+0000","modified":"2024-03-31T15:03:13.941667+0000","last_up_change":"0.000000","last_in_change":"2024-03-31T15:03:13.941667+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"92a66450-ef19-40d8-ac3d-d760b23b0bda","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"da9d1f2b-1968-4959-b843-faa231988e92","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"01a73662-9a21-4b1f-bcbb-42b9636a4638","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c7771fd3-e18e-4500-ac38-23cf10ebcd75","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"88260730-1482-4fbd-a40f-cb1124e9df4a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"4cb2c00c-fae0-4dd8-9d27-ec4f0093ccf7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"81ce0c19-2554-4cae-acbe-66a05e9e9b9b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"85e4313b-d37d-442e-8042-153bf20a847d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-31T15:03:15.298 INFO:teuthology.misc.health.smithi016.stderr:2024-03-31T15:03:15.302+0000 7fbe4be00640 1 -- 172.21.15.16:0/2233613028 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbe440d13d0 msgr2=0x7fbe44158210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:15.298 INFO:teuthology.misc.health.smithi016.stderr:2024-03-31T15:03:15.302+0000 7fbe4be00640 1 --2- 172.21.15.16:0/2233613028 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbe440d13d0 0x7fbe44158210 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fbe30009a50 tx=0x7fbe30009f10 comp rx=0 tx=0).stop 2024-03-31T15:03:15.298 INFO:teuthology.misc.health.smithi016.stderr:2024-03-31T15:03:15.302+0000 7fbe4be00640 1 -- 172.21.15.16:0/2233613028 shutdown_connections 2024-03-31T15:03:15.298 INFO:teuthology.misc.health.smithi016.stderr:2024-03-31T15:03:15.302+0000 7fbe4be00640 1 --2- 172.21.15.16:0/2233613028 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbe4415c270 0x7fbe440c7d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:15.298 INFO:teuthology.misc.health.smithi016.stderr:2024-03-31T15:03:15.302+0000 7fbe4be00640 1 --2- 172.21.15.16:0/2233613028 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbe440d13d0 0x7fbe44158210 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:15.299 INFO:teuthology.misc.health.smithi016.stderr:2024-03-31T15:03:15.302+0000 7fbe4be00640 1 --2- 172.21.15.16:0/2233613028 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fbe440cc7e0 0x7fbe440cbc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:15.299 INFO:teuthology.misc.health.smithi016.stderr:2024-03-31T15:03:15.302+0000 7fbe4be00640 1 -- 172.21.15.16:0/2233613028 >> 172.21.15.16:0/2233613028 conn(0x7fbe440b3ca0 msgr2=0x7fbe440b4080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:03:15.299 INFO:teuthology.misc.health.smithi016.stderr:2024-03-31T15:03:15.302+0000 7fbe4be00640 1 -- 172.21.15.16:0/2233613028 shutdown_connections 2024-03-31T15:03:15.299 INFO:teuthology.misc.health.smithi016.stderr:2024-03-31T15:03:15.302+0000 7fbe4be00640 1 -- 172.21.15.16:0/2233613028 wait complete. 2024-03-31T15:03:15.315 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-31T15:03:15.736 INFO:tasks.ceph.mgr.x.smithi089.stderr:2024-03-31T15:03:15.743+0000 7f078c909180 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-31T15:03:15.822 INFO:tasks.ceph.mgr.x.smithi089.stderr:2024-03-31T15:03:15.831+0000 7f078c909180 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-31T15:03:15.853 INFO:tasks.ceph.mgr.y.smithi016.stderr:2024-03-31T15:03:15.858+0000 7f2c2622a180 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-31T15:03:15.943 INFO:tasks.ceph.mgr.y.smithi016.stderr:2024-03-31T15:03:15.946+0000 7f2c2622a180 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-31T15:03:16.022 INFO:tasks.ceph.mgr.x.smithi089.stderr:2024-03-31T15:03:16.031+0000 7f078c909180 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-31T15:03:16.106 INFO:tasks.ceph.mgr.x.smithi089.stderr:2024-03-31T15:03:16.115+0000 7f078c909180 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-31T15:03:16.155 INFO:tasks.ceph.mgr.y.smithi016.stderr:2024-03-31T15:03:16.158+0000 7f2c2622a180 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-31T15:03:16.243 INFO:tasks.ceph.mgr.y.smithi016.stderr:2024-03-31T15:03:16.246+0000 7f2c2622a180 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-31T15:03:16.430 INFO:tasks.ceph.mgr.x.smithi089.stderr:2024-03-31T15:03:16.439+0000 7f078c909180 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-31T15:03:16.501 INFO:tasks.ceph.mgr.x.smithi089.stderr:2024-03-31T15:03:16.506+0000 7f078c909180 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-31T15:03:16.592 INFO:tasks.ceph.mgr.y.smithi016.stderr:2024-03-31T15:03:16.598+0000 7f2c2622a180 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-31T15:03:16.665 INFO:tasks.ceph.mgr.y.smithi016.stderr:2024-03-31T15:03:16.670+0000 7f2c2622a180 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-31T15:03:17.524 INFO:tasks.ceph.mgr.x.smithi089.stderr:2024-03-31T15:03:17.530+0000 7f0732a00640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-03-31T15:03:17.524 INFO:tasks.ceph.mgr.x.smithi089.stderr:2024-03-31T15:03:17.530+0000 7f0732a00640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-03-31T15:03:18.410 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-03-31T15:03:18.414+0000 7fae5342b7c0 -1 osd.0 0 log_to_monitors true 2024-03-31T15:03:18.437 INFO:tasks.ceph.osd.6.smithi089.stderr:2024-03-31T15:03:18.442+0000 7f5a667887c0 -1 osd.6 0 log_to_monitors true 2024-03-31T15:03:18.504 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-03-31T15:03:18.510+0000 7fa5098937c0 -1 osd.1 0 log_to_monitors true 2024-03-31T15:03:18.529 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-03-31T15:03:18.534+0000 7f5f211577c0 -1 osd.3 0 log_to_monitors true 2024-03-31T15:03:18.557 INFO:tasks.ceph.osd.5.smithi089.stderr:2024-03-31T15:03:18.566+0000 7f348892d7c0 -1 osd.5 0 log_to_monitors true 2024-03-31T15:03:18.559 INFO:tasks.ceph.osd.4.smithi089.stderr:2024-03-31T15:03:18.566+0000 7fae012de7c0 -1 osd.4 0 log_to_monitors true 2024-03-31T15:03:18.578 INFO:tasks.ceph.osd.7.smithi089.stderr:2024-03-31T15:03:18.586+0000 7f78dc09e7c0 -1 osd.7 0 log_to_monitors true 2024-03-31T15:03:18.624 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-03-31T15:03:18.630+0000 7fe16e02b7c0 -1 osd.2 0 log_to_monitors true 2024-03-31T15:03:20.586 INFO:tasks.ceph.osd.6.smithi089.stderr:2024-03-31T15:03:20.594+0000 7f5a61600640 -1 osd.6 0 waiting for initial osdmap 2024-03-31T15:03:20.756 INFO:tasks.ceph.osd.6.smithi089.stderr:2024-03-31T15:03:20.762+0000 7f5a57600640 -1 osd.6 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-31T15:03:21.034 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-03-31T15:03:21.038+0000 7f5f1c000640 -1 osd.3 0 waiting for initial osdmap 2024-03-31T15:03:21.064 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-03-31T15:03:21.066+0000 7fae4e200640 -1 osd.0 0 waiting for initial osdmap 2024-03-31T15:03:21.065 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-03-31T15:03:21.070+0000 7f5f12000640 -1 osd.3 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-31T15:03:21.085 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-03-31T15:03:21.090+0000 7fa504800640 -1 osd.1 0 waiting for initial osdmap 2024-03-31T15:03:21.097 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-03-31T15:03:21.102+0000 7fae43800640 -1 osd.0 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-31T15:03:21.121 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-03-31T15:03:21.126+0000 7fa4fa800640 -1 osd.1 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-31T15:03:21.873 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-03-31T15:03:21.878+0000 7fe168e00640 -1 osd.2 0 waiting for initial osdmap 2024-03-31T15:03:21.893 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-03-31T15:03:21.898+0000 7fe15e400640 -1 osd.2 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-31T15:03:22.121 DEBUG:teuthology.orchestra.run.smithi016:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-31T15:03:22.251 INFO:teuthology.misc.health.smithi016.stderr:2024-03-31T15:03:22.254+0000 7ffa03200640 1 Processor -- start 2024-03-31T15:03:22.252 INFO:teuthology.misc.health.smithi016.stderr:2024-03-31T15:03:22.254+0000 7ffa03200640 1 -- start start 2024-03-31T15:03:22.253 INFO:teuthology.misc.health.smithi016.stderr:2024-03-31T15:03:22.258+0000 7ffa03200640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff9fc073620 0x7ff9fc0cb270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:22.253 INFO:teuthology.misc.health.smithi016.stderr:2024-03-31T15:03:22.258+0000 7ffa03200640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7ff9fc06ef40 0x7ff9fc06d500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:22.254 INFO:teuthology.misc.health.smithi016.stderr:2024-03-31T15:03:22.258+0000 7ffa03200640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff9fc0ca8d0 con 0x7ff9fc06dff0 2024-03-31T15:03:22.254 INFO:teuthology.misc.health.smithi016.stderr:2024-03-31T15:03:22.258+0000 7ffa03200640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7ff9fc0cabb0 con 0x7ff9fc073620 2024-03-31T15:03:22.254 INFO:teuthology.misc.health.smithi016.stderr:2024-03-31T15:03:22.258+0000 7ffa03200640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7ff9fc0cf920 con 0x7ff9fc06ef40 2024-03-31T15:03:22.254 INFO:teuthology.misc.health.smithi016.stderr:2024-03-31T15:03:22.258+0000 7ff9fbe00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff9fc073620 0x7ff9fc0cb270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:22.254 INFO:teuthology.misc.health.smithi016.stderr:2024-03-31T15:03:22.258+0000 7ffa01200640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7ff9fc06dff0 0x7ff9fc072470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:38822/0 (socket says 172.21.15.16:38822) 2024-03-31T15:03:22.254 INFO:teuthology.misc.health.smithi016.stderr:2024-03-31T15:03:22.258+0000 7ffa01200640 1 -- 172.21.15.16:0/1522559854 learned_addr learned my addr 172.21.15.16:0/1522559854 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:03:22.254 INFO:teuthology.misc.health.smithi016.stderr:2024-03-31T15:03:22.258+0000 7ff9f3400640 1 --2- 172.21.15.16:0/1522559854 >> v2:172.21.15.89:3300/0 conn(0x7ff9fc06ef40 0x7ff9fc06d500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:22.254 INFO:teuthology.misc.health.smithi016.stderr:2024-03-31T15:03:22.258+0000 7ff9fb400640 1 -- 172.21.15.16:0/1522559854 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3124719851 0 0) 0x7ff9fc0ca8d0 con 0x7ff9fc06dff0 2024-03-31T15:03:22.254 INFO:teuthology.misc.health.smithi016.stderr:2024-03-31T15:03:22.258+0000 7ff9fb400640 1 -- 172.21.15.16:0/1522559854 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff9e0005740 con 0x7ff9fc06dff0 2024-03-31T15:03:22.254 INFO:teuthology.misc.health.smithi016.stderr:2024-03-31T15:03:22.258+0000 7ff9fbe00640 1 -- 172.21.15.16:0/1522559854 >> v2:172.21.15.89:3300/0 conn(0x7ff9fc06ef40 msgr2=0x7ff9fc06d500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:03:22.254 INFO:teuthology.misc.health.smithi016.stderr:2024-03-31T15:03:22.258+0000 7ff9fbe00640 1 --2- 172.21.15.16:0/1522559854 >> v2:172.21.15.89:3300/0 conn(0x7ff9fc06ef40 0x7ff9fc06d500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:22.254 INFO:teuthology.misc.health.smithi016.stderr:2024-03-31T15:03:22.258+0000 7ff9fbe00640 1 -- 172.21.15.16:0/1522559854 >> v1:172.21.15.16:6789/0 conn(0x7ff9fc06dff0 legacy=0x7ff9fc072470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:22.254 INFO:teuthology.misc.health.smithi016.stderr:2024-03-31T15:03:22.258+0000 7ff9fbe00640 1 -- 172.21.15.16:0/1522559854 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9fc0cfc00 con 0x7ff9fc073620 2024-03-31T15:03:22.256 INFO:teuthology.misc.health.smithi016.stderr:2024-03-31T15:03:22.262+0000 7ff9fbe00640 1 --2- 172.21.15.16:0/1522559854 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff9fc073620 0x7ff9fc0cb270 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7ff9fc06f640 tx=0x7ff9ec01c930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a0e47053e785e4dc server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:03:22.256 INFO:teuthology.misc.health.smithi016.stderr:2024-03-31T15:03:22.262+0000 7ff9fb400640 1 -- 172.21.15.16:0/1522559854 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9ec0b2070 con 0x7ff9fc073620 2024-03-31T15:03:22.257 INFO:teuthology.misc.health.smithi016.stderr:2024-03-31T15:03:22.262+0000 7ff9fb400640 1 -- 172.21.15.16:0/1522559854 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff9ec0a5a10 con 0x7ff9fc073620 2024-03-31T15:03:22.257 INFO:teuthology.misc.health.smithi016.stderr:2024-03-31T15:03:22.262+0000 7ff9fb400640 1 -- 172.21.15.16:0/1522559854 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9ec0a5de0 con 0x7ff9fc073620 2024-03-31T15:03:22.257 INFO:teuthology.misc.health.smithi016.stderr:2024-03-31T15:03:22.262+0000 7ffa03200640 1 -- 172.21.15.16:0/1522559854 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff9fc073620 msgr2=0x7ff9fc0cb270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:22.257 INFO:teuthology.misc.health.smithi016.stderr:2024-03-31T15:03:22.262+0000 7ffa03200640 1 --2- 172.21.15.16:0/1522559854 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff9fc073620 0x7ff9fc0cb270 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7ff9fc06f640 tx=0x7ff9ec01c930 comp rx=0 tx=0).stop 2024-03-31T15:03:22.257 INFO:teuthology.misc.health.smithi016.stderr:2024-03-31T15:03:22.262+0000 7ffa03200640 1 -- 172.21.15.16:0/1522559854 shutdown_connections 2024-03-31T15:03:22.258 INFO:teuthology.misc.health.smithi016.stderr:2024-03-31T15:03:22.262+0000 7ffa03200640 1 --2- 172.21.15.16:0/1522559854 >> v2:172.21.15.89:3300/0 conn(0x7ff9fc06ef40 0x7ff9fc06d500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:22.258 INFO:teuthology.misc.health.smithi016.stderr:2024-03-31T15:03:22.262+0000 7ffa03200640 1 --2- 172.21.15.16:0/1522559854 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff9fc073620 0x7ff9fc0cb270 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:22.258 INFO:teuthology.misc.health.smithi016.stderr:2024-03-31T15:03:22.262+0000 7ffa03200640 1 -- 172.21.15.16:0/1522559854 >> 172.21.15.16:0/1522559854 conn(0x7ff9fc0cffd0 msgr2=0x7ff9fc0d03e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:03:22.258 INFO:teuthology.misc.health.smithi016.stderr:2024-03-31T15:03:22.262+0000 7ffa03200640 1 -- 172.21.15.16:0/1522559854 shutdown_connections 2024-03-31T15:03:22.258 INFO:teuthology.misc.health.smithi016.stderr:2024-03-31T15:03:22.262+0000 7ffa03200640 1 -- 172.21.15.16:0/1522559854 wait complete. 2024-03-31T15:03:22.259 INFO:teuthology.misc.health.smithi016.stderr:2024-03-31T15:03:22.262+0000 7ffa03200640 1 Processor -- start 2024-03-31T15:03:22.259 INFO:teuthology.misc.health.smithi016.stderr:2024-03-31T15:03:22.262+0000 7ffa03200640 1 -- start start 2024-03-31T15:03:22.260 INFO:teuthology.misc.health.smithi016.stderr:2024-03-31T15:03:22.266+0000 7ffa03200640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff9fc06dff0 0x7ff9fc151170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:22.261 INFO:teuthology.misc.health.smithi016.stderr:2024-03-31T15:03:22.266+0000 7ffa03200640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff9fc06ef40 0x7ff9fc0c9d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:22.261 INFO:teuthology.misc.health.smithi016.stderr:2024-03-31T15:03:22.266+0000 7ffa03200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff9fc073620 0x7ff9fc0b2f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:22.261 INFO:teuthology.misc.health.smithi016.stderr:2024-03-31T15:03:22.266+0000 7ffa03200640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7ff9fc09e000 con 0x7ff9fc073620 2024-03-31T15:03:22.261 INFO:teuthology.misc.health.smithi016.stderr:2024-03-31T15:03:22.266+0000 7ffa03200640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7ff9fc09de80 con 0x7ff9fc06dff0 2024-03-31T15:03:22.261 INFO:teuthology.misc.health.smithi016.stderr:2024-03-31T15:03:22.266+0000 7ffa03200640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7ff9fc15f480 con 0x7ff9fc06ef40 2024-03-31T15:03:22.261 INFO:teuthology.misc.health.smithi016.stderr:2024-03-31T15:03:22.266+0000 7ffa01200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff9fc073620 0x7ff9fc0b2f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:22.261 INFO:teuthology.misc.health.smithi016.stderr:2024-03-31T15:03:22.266+0000 7ff9fbe00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff9fc06dff0 0x7ff9fc151170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:22.261 INFO:teuthology.misc.health.smithi016.stderr:2024-03-31T15:03:22.266+0000 7ffa01200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff9fc073620 0x7ff9fc0b2f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40746/0 (socket says 172.21.15.16:40746) 2024-03-31T15:03:22.261 INFO:teuthology.misc.health.smithi016.stderr:2024-03-31T15:03:22.266+0000 7ffa01200640 1 -- 172.21.15.16:0/3192963587 learned_addr learned my addr 172.21.15.16:0/3192963587 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:03:22.261 INFO:teuthology.misc.health.smithi016.stderr:2024-03-31T15:03:22.266+0000 7ff9f3400640 1 --2- 172.21.15.16:0/3192963587 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff9fc06ef40 0x7ff9fc0c9d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:22.261 INFO:teuthology.misc.health.smithi016.stderr:2024-03-31T15:03:22.266+0000 7ffa01200640 1 -- 172.21.15.16:0/3192963587 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff9fc06ef40 msgr2=0x7ff9fc0c9d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:22.261 INFO:teuthology.misc.health.smithi016.stderr:2024-03-31T15:03:22.266+0000 7ffa01200640 1 --2- 172.21.15.16:0/3192963587 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff9fc06ef40 0x7ff9fc0c9d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:22.261 INFO:teuthology.misc.health.smithi016.stderr:2024-03-31T15:03:22.266+0000 7ffa01200640 1 -- 172.21.15.16:0/3192963587 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff9fc06dff0 msgr2=0x7ff9fc151170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:22.261 INFO:teuthology.misc.health.smithi016.stderr:2024-03-31T15:03:22.266+0000 7ffa01200640 1 --2- 172.21.15.16:0/3192963587 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff9fc06dff0 0x7ff9fc151170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:22.261 INFO:teuthology.misc.health.smithi016.stderr:2024-03-31T15:03:22.266+0000 7ffa01200640 1 -- 172.21.15.16:0/3192963587 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9fc0b4e30 con 0x7ff9fc073620 2024-03-31T15:03:22.262 INFO:teuthology.misc.health.smithi016.stderr:2024-03-31T15:03:22.266+0000 7ffa01200640 1 --2- 172.21.15.16:0/3192963587 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff9fc073620 0x7ff9fc0b2f10 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7ff9f400c450 tx=0x7ff9f400c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:03:22.262 INFO:teuthology.misc.health.smithi016.stderr:2024-03-31T15:03:22.266+0000 7ff9f8c00640 1 -- 172.21.15.16:0/3192963587 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9f4017070 con 0x7ff9fc073620 2024-03-31T15:03:22.262 INFO:teuthology.misc.health.smithi016.stderr:2024-03-31T15:03:22.266+0000 7ff9f8c00640 1 -- 172.21.15.16:0/3192963587 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff9f4005cf0 con 0x7ff9fc073620 2024-03-31T15:03:22.263 INFO:teuthology.misc.health.smithi016.stderr:2024-03-31T15:03:22.266+0000 7ffa03200640 1 -- 172.21.15.16:0/3192963587 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff9fc0b5110 con 0x7ff9fc073620 2024-03-31T15:03:22.263 INFO:teuthology.misc.health.smithi016.stderr:2024-03-31T15:03:22.266+0000 7ff9f8c00640 1 -- 172.21.15.16:0/3192963587 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9f40072b0 con 0x7ff9fc073620 2024-03-31T15:03:22.263 INFO:teuthology.misc.health.smithi016.stderr:2024-03-31T15:03:22.266+0000 7ffa03200640 1 -- 172.21.15.16:0/3192963587 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff9fc0b56d0 con 0x7ff9fc073620 2024-03-31T15:03:22.263 INFO:teuthology.misc.health.smithi016.stderr:2024-03-31T15:03:22.266+0000 7ffa03200640 1 -- 172.21.15.16:0/3192963587 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff9fc068b20 con 0x7ff9fc073620 2024-03-31T15:03:22.263 INFO:teuthology.misc.health.smithi016.stderr:2024-03-31T15:03:22.266+0000 7ff9f8c00640 1 -- 172.21.15.16:0/3192963587 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 4) ==== 87931+0+0 (secure 0 0 0) 0x7ff9f401a070 con 0x7ff9fc073620 2024-03-31T15:03:22.264 INFO:teuthology.misc.health.smithi016.stderr:2024-03-31T15:03:22.270+0000 7ff9f8c00640 1 --2- 172.21.15.16:0/3192963587 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7ff9d80682f0 0x7ff9d806a7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:22.264 INFO:teuthology.misc.health.smithi016.stderr:2024-03-31T15:03:22.270+0000 7ff9f8c00640 1 -- 172.21.15.16:0/3192963587 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(14..14 src has 1..14) ==== 3351+0+0 (secure 0 0 0) 0x7ff9f4085290 con 0x7ff9fc073620 2024-03-31T15:03:22.264 INFO:teuthology.misc.health.smithi016.stderr:2024-03-31T15:03:22.270+0000 7ff9fbe00640 1 --2- 172.21.15.16:0/3192963587 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7ff9d80682f0 0x7ff9d806a7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:22.265 INFO:teuthology.misc.health.smithi016.stderr:2024-03-31T15:03:22.270+0000 7ff9fbe00640 1 --2- 172.21.15.16:0/3192963587 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7ff9d80682f0 0x7ff9d806a7b0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7ff9ec0af000 tx=0x7ff9ec0a7070 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:03:22.282 INFO:teuthology.misc.health.smithi016.stderr:2024-03-31T15:03:22.286+0000 7ff9f8c00640 1 -- 172.21.15.16:0/3192963587 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7ff9f4055c00 con 0x7ff9fc073620 2024-03-31T15:03:22.362 INFO:tasks.ceph.osd.4.smithi089.stderr:2024-03-31T15:03:22.370+0000 7fadfcc00640 -1 osd.4 0 waiting for initial osdmap 2024-03-31T15:03:22.388 INFO:tasks.ceph.osd.4.smithi089.stderr:2024-03-31T15:03:22.394+0000 7fadf2200640 -1 osd.4 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-31T15:03:22.398 INFO:tasks.ceph.osd.7.smithi089.stderr:2024-03-31T15:03:22.406+0000 7f78d7000640 -1 osd.7 0 waiting for initial osdmap 2024-03-31T15:03:22.402 INFO:tasks.ceph.osd.5.smithi089.stderr:2024-03-31T15:03:22.410+0000 7f3483800640 -1 osd.5 0 waiting for initial osdmap 2024-03-31T15:03:22.435 INFO:tasks.ceph.osd.5.smithi089.stderr:2024-03-31T15:03:22.442+0000 7f3479800640 -1 osd.5 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-31T15:03:22.435 INFO:tasks.ceph.osd.7.smithi089.stderr:2024-03-31T15:03:22.442+0000 7f78cd000640 -1 osd.7 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-31T15:03:22.452 INFO:teuthology.misc.health.smithi016.stderr:2024-03-31T15:03:22.458+0000 7ffa03200640 1 -- 172.21.15.16:0/3192963587 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ff9fc09e220 con 0x7ff9fc073620 2024-03-31T15:03:22.452 INFO:teuthology.misc.health.smithi016.stderr:2024-03-31T15:03:22.458+0000 7ff9f8c00640 1 -- 172.21.15.16:0/3192963587 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v14) ==== 74+0+8965 (secure 0 0 0) 0x7ff9f4059f00 con 0x7ff9fc073620 2024-03-31T15:03:22.452 INFO:teuthology.misc.health.smithi016.stdout: 2024-03-31T15:03:22.453 INFO:teuthology.misc.health.smithi016.stdout:{"epoch":14,"fsid":"286eb5da-fb47-4550-b290-d9519c13b9a5","created":"2024-03-31T15:03:11.438952+0000","modified":"2024-03-31T15:03:21.678703+0000","last_up_change":"2024-03-31T15:03:21.678703+0000","last_in_change":"2024-03-31T15:03:13.941667+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"92a66450-ef19-40d8-ac3d-d760b23b0bda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6801","nonce":932712244}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6803","nonce":932712244}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6807","nonce":932712244}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6805","nonce":932712244}]},"public_addr":"172.21.15.16:6801/932712244","cluster_addr":"172.21.15.16:6803/932712244","heartbeat_back_addr":"172.21.15.16:6807/932712244","heartbeat_front_addr":"172.21.15.16:6805/932712244","state":["exists","up"]},{"osd":1,"uuid":"da9d1f2b-1968-4959-b843-faa231988e92","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6810","nonce":1398737533}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6811","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6813","nonce":1398737533}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6821","nonce":1398737533}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6815","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6816","nonce":1398737533}]},"public_addr":"172.21.15.16:6810/1398737533","cluster_addr":"172.21.15.16:6813/1398737533","heartbeat_back_addr":"172.21.15.16:6821/1398737533","heartbeat_front_addr":"172.21.15.16:6816/1398737533","state":["exists","up"]},{"osd":2,"uuid":"01a73662-9a21-4b1f-bcbb-42b9636a4638","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c7771fd3-e18e-4500-ac38-23cf10ebcd75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6809","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6812","nonce":3360352974}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6817","nonce":3360352974}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6828","nonce":3360352974}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6823","nonce":3360352974}]},"public_addr":"172.21.15.16:6812/3360352974","cluster_addr":"172.21.15.16:6817/3360352974","heartbeat_back_addr":"172.21.15.16:6828/3360352974","heartbeat_front_addr":"172.21.15.16:6823/3360352974","state":["exists","up"]},{"osd":4,"uuid":"88260730-1482-4fbd-a40f-cb1124e9df4a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"4cb2c00c-fae0-4dd8-9d27-ec4f0093ccf7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"81ce0c19-2554-4cae-acbe-66a05e9e9b9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6809","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6812","nonce":3470471752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6815","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6818","nonce":3470471752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6827","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6829","nonce":3470471752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6821","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6824","nonce":3470471752}]},"public_addr":"172.21.15.89:6812/3470471752","cluster_addr":"172.21.15.89:6818/3470471752","heartbeat_back_addr":"172.21.15.89:6829/3470471752","heartbeat_front_addr":"172.21.15.89:6824/3470471752","state":["exists","up"]},{"osd":7,"uuid":"85e4313b-d37d-442e-8042-153bf20a847d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-31T15:03:22.455 INFO:teuthology.misc.health.smithi016.stderr:2024-03-31T15:03:22.458+0000 7ffa03200640 1 -- 172.21.15.16:0/3192963587 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7ff9d80682f0 msgr2=0x7ff9d806a7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:22.455 INFO:teuthology.misc.health.smithi016.stderr:2024-03-31T15:03:22.458+0000 7ffa03200640 1 --2- 172.21.15.16:0/3192963587 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7ff9d80682f0 0x7ff9d806a7b0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7ff9ec0af000 tx=0x7ff9ec0a7070 comp rx=0 tx=0).stop 2024-03-31T15:03:22.455 INFO:teuthology.misc.health.smithi016.stderr:2024-03-31T15:03:22.458+0000 7ffa03200640 1 -- 172.21.15.16:0/3192963587 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff9fc073620 msgr2=0x7ff9fc0b2f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:22.455 INFO:teuthology.misc.health.smithi016.stderr:2024-03-31T15:03:22.458+0000 7ffa03200640 1 --2- 172.21.15.16:0/3192963587 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff9fc073620 0x7ff9fc0b2f10 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7ff9f400c450 tx=0x7ff9f400c910 comp rx=0 tx=0).stop 2024-03-31T15:03:22.455 INFO:teuthology.misc.health.smithi016.stderr:2024-03-31T15:03:22.458+0000 7ffa03200640 1 -- 172.21.15.16:0/3192963587 shutdown_connections 2024-03-31T15:03:22.455 INFO:teuthology.misc.health.smithi016.stderr:2024-03-31T15:03:22.458+0000 7ffa03200640 1 --2- 172.21.15.16:0/3192963587 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7ff9d80682f0 0x7ff9d806a7b0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:22.455 INFO:teuthology.misc.health.smithi016.stderr:2024-03-31T15:03:22.458+0000 7ffa03200640 1 --2- 172.21.15.16:0/3192963587 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff9fc073620 0x7ff9fc0b2f10 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:22.455 INFO:teuthology.misc.health.smithi016.stderr:2024-03-31T15:03:22.458+0000 7ffa03200640 1 --2- 172.21.15.16:0/3192963587 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff9fc06ef40 0x7ff9fc0c9d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:22.455 INFO:teuthology.misc.health.smithi016.stderr:2024-03-31T15:03:22.458+0000 7ffa03200640 1 --2- 172.21.15.16:0/3192963587 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff9fc06dff0 0x7ff9fc151170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:22.455 INFO:teuthology.misc.health.smithi016.stderr:2024-03-31T15:03:22.458+0000 7ffa03200640 1 -- 172.21.15.16:0/3192963587 >> 172.21.15.16:0/3192963587 conn(0x7ff9fc0cffd0 msgr2=0x7ff9fc0ad8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:03:22.455 INFO:teuthology.misc.health.smithi016.stderr:2024-03-31T15:03:22.458+0000 7ffa03200640 1 -- 172.21.15.16:0/3192963587 shutdown_connections 2024-03-31T15:03:22.455 INFO:teuthology.misc.health.smithi016.stderr:2024-03-31T15:03:22.458+0000 7ffa03200640 1 -- 172.21.15.16:0/3192963587 wait complete. 2024-03-31T15:03:22.472 DEBUG:teuthology.misc:4 of 8 OSDs are up 2024-03-31T15:03:29.279 DEBUG:teuthology.orchestra.run.smithi016:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-31T15:03:29.421 INFO:teuthology.misc.health.smithi016.stderr:2024-03-31T15:03:29.426+0000 7f3977a00640 1 Processor -- start 2024-03-31T15:03:29.421 INFO:teuthology.misc.health.smithi016.stderr:2024-03-31T15:03:29.426+0000 7f3977a00640 1 -- start start 2024-03-31T15:03:29.422 INFO:teuthology.misc.health.smithi016.stderr:2024-03-31T15:03:29.426+0000 7f3977a00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f3970162d50 0x7f397015f210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:29.422 INFO:teuthology.misc.health.smithi016.stderr:2024-03-31T15:03:29.426+0000 7f3977a00640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f397015f810 con 0x7f39700cf0a0 2024-03-31T15:03:29.422 INFO:teuthology.misc.health.smithi016.stderr:2024-03-31T15:03:29.426+0000 7f3977a00640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3970162270 con 0x7f397006ff70 2024-03-31T15:03:29.423 INFO:teuthology.misc.health.smithi016.stderr:2024-03-31T15:03:29.426+0000 7f3977a00640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f3970162550 con 0x7f3970162d50 2024-03-31T15:03:29.423 INFO:teuthology.misc.health.smithi016.stderr:2024-03-31T15:03:29.426+0000 7f3975000640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f39700cf0a0 0x7f397006f860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:40578/0 (socket says 172.21.15.16:40578) 2024-03-31T15:03:29.423 INFO:teuthology.misc.health.smithi016.stderr:2024-03-31T15:03:29.426+0000 7f3975000640 1 -- 172.21.15.16:0/2933835356 learned_addr learned my addr 172.21.15.16:0/2933835356 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:03:29.423 INFO:teuthology.misc.health.smithi016.stderr:2024-03-31T15:03:29.426+0000 7f3975a00640 1 --2- 172.21.15.16:0/2933835356 >> v2:172.21.15.16:3300/0 conn(0x7f3970162d50 0x7f397015f210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:29.423 INFO:teuthology.misc.health.smithi016.stderr:2024-03-31T15:03:29.426+0000 7f396f400640 1 -- 172.21.15.16:0/2933835356 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3577479186 0 0) 0x7f397015f810 con 0x7f39700cf0a0 2024-03-31T15:03:29.424 INFO:teuthology.misc.health.smithi016.stderr:2024-03-31T15:03:29.426+0000 7f396f400640 1 -- 172.21.15.16:0/2933835356 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3958003530 con 0x7f39700cf0a0 2024-03-31T15:03:29.424 INFO:teuthology.misc.health.smithi016.stderr:2024-03-31T15:03:29.426+0000 7f3975a00640 1 -- 172.21.15.16:0/2933835356 >> v1:172.21.15.89:6789/0 conn(0x7f397006ff70 legacy=0x7f397015eb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:29.424 INFO:teuthology.misc.health.smithi016.stderr:2024-03-31T15:03:29.426+0000 7f3975a00640 1 -- 172.21.15.16:0/2933835356 >> v1:172.21.15.16:6789/0 conn(0x7f39700cf0a0 legacy=0x7f397006f860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:29.424 INFO:teuthology.misc.health.smithi016.stderr:2024-03-31T15:03:29.426+0000 7f3975a00640 1 -- 172.21.15.16:0/2933835356 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39700cf480 con 0x7f3970162d50 2024-03-31T15:03:29.426 INFO:teuthology.misc.health.smithi016.stderr:2024-03-31T15:03:29.430+0000 7f3975a00640 1 --2- 172.21.15.16:0/2933835356 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3970162d50 0x7f397015f210 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f3968009d90 tx=0x7f396801c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b74e3be40545c51f server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:03:29.426 INFO:teuthology.misc.health.smithi016.stderr:2024-03-31T15:03:29.430+0000 7f396f400640 1 -- 172.21.15.16:0/2933835356 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39680b2070 con 0x7f3970162d50 2024-03-31T15:03:29.426 INFO:teuthology.misc.health.smithi016.stderr:2024-03-31T15:03:29.430+0000 7f396f400640 1 -- 172.21.15.16:0/2933835356 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f39680a5830 con 0x7f3970162d50 2024-03-31T15:03:29.426 INFO:teuthology.misc.health.smithi016.stderr:2024-03-31T15:03:29.430+0000 7f396f400640 1 -- 172.21.15.16:0/2933835356 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39680a5be0 con 0x7f3970162d50 2024-03-31T15:03:29.427 INFO:teuthology.misc.health.smithi016.stderr:2024-03-31T15:03:29.430+0000 7f3977a00640 1 -- 172.21.15.16:0/2933835356 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3970162d50 msgr2=0x7f397015f210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:29.427 INFO:teuthology.misc.health.smithi016.stderr:2024-03-31T15:03:29.430+0000 7f3977a00640 1 --2- 172.21.15.16:0/2933835356 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3970162d50 0x7f397015f210 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f3968009d90 tx=0x7f396801c970 comp rx=0 tx=0).stop 2024-03-31T15:03:29.427 INFO:teuthology.misc.health.smithi016.stderr:2024-03-31T15:03:29.430+0000 7f3977a00640 1 -- 172.21.15.16:0/2933835356 shutdown_connections 2024-03-31T15:03:29.427 INFO:teuthology.misc.health.smithi016.stderr:2024-03-31T15:03:29.430+0000 7f3977a00640 1 --2- 172.21.15.16:0/2933835356 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3970162d50 0x7f397015f210 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:29.428 INFO:teuthology.misc.health.smithi016.stderr:2024-03-31T15:03:29.430+0000 7f3977a00640 1 -- 172.21.15.16:0/2933835356 >> 172.21.15.16:0/2933835356 conn(0x7f39700b3ca0 msgr2=0x7f39700b40b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:03:29.428 INFO:teuthology.misc.health.smithi016.stderr:2024-03-31T15:03:29.430+0000 7f3977a00640 1 -- 172.21.15.16:0/2933835356 shutdown_connections 2024-03-31T15:03:29.428 INFO:teuthology.misc.health.smithi016.stderr:2024-03-31T15:03:29.430+0000 7f3977a00640 1 -- 172.21.15.16:0/2933835356 wait complete. 2024-03-31T15:03:29.428 INFO:teuthology.misc.health.smithi016.stderr:2024-03-31T15:03:29.434+0000 7f3977a00640 1 Processor -- start 2024-03-31T15:03:29.429 INFO:teuthology.misc.health.smithi016.stderr:2024-03-31T15:03:29.434+0000 7f3977a00640 1 -- start start 2024-03-31T15:03:29.432 INFO:teuthology.misc.health.smithi016.stderr:2024-03-31T15:03:29.434+0000 7f3977a00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f397006ff70 0x7f39700c8c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:29.432 INFO:teuthology.misc.health.smithi016.stderr:2024-03-31T15:03:29.434+0000 7f3977a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f39700cf0a0 0x7f39700c9c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:29.432 INFO:teuthology.misc.health.smithi016.stderr:2024-03-31T15:03:29.434+0000 7f3977a00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f3970162d50 0x7f39700ce1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:29.432 INFO:teuthology.misc.health.smithi016.stderr:2024-03-31T15:03:29.434+0000 7f3977a00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f39701429a0 con 0x7f39700cf0a0 2024-03-31T15:03:29.433 INFO:teuthology.misc.health.smithi016.stderr:2024-03-31T15:03:29.434+0000 7f3977a00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f3970142820 con 0x7f397006ff70 2024-03-31T15:03:29.433 INFO:teuthology.misc.health.smithi016.stderr:2024-03-31T15:03:29.434+0000 7f3977a00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f3970142b20 con 0x7f3970162d50 2024-03-31T15:03:29.433 INFO:teuthology.misc.health.smithi016.stderr:2024-03-31T15:03:29.434+0000 7f396fe00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f39700cf0a0 0x7f39700c9c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:29.433 INFO:teuthology.misc.health.smithi016.stderr:2024-03-31T15:03:29.434+0000 7f396fe00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f39700cf0a0 0x7f39700c9c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59480/0 (socket says 172.21.15.16:59480) 2024-03-31T15:03:29.433 INFO:teuthology.misc.health.smithi016.stderr:2024-03-31T15:03:29.434+0000 7f396fe00640 1 -- 172.21.15.16:0/2616443455 learned_addr learned my addr 172.21.15.16:0/2616443455 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:03:29.433 INFO:teuthology.misc.health.smithi016.stderr:2024-03-31T15:03:29.434+0000 7f3975000640 1 --2- 172.21.15.16:0/2616443455 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f397006ff70 0x7f39700c8c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:29.433 INFO:teuthology.misc.health.smithi016.stderr:2024-03-31T15:03:29.434+0000 7f3975a00640 1 --2- 172.21.15.16:0/2616443455 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f3970162d50 0x7f39700ce1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:29.433 INFO:teuthology.misc.health.smithi016.stderr:2024-03-31T15:03:29.434+0000 7f396fe00640 1 -- 172.21.15.16:0/2616443455 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f3970162d50 msgr2=0x7f39700ce1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:29.433 INFO:teuthology.misc.health.smithi016.stderr:2024-03-31T15:03:29.434+0000 7f396fe00640 1 --2- 172.21.15.16:0/2616443455 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f3970162d50 0x7f39700ce1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:29.433 INFO:teuthology.misc.health.smithi016.stderr:2024-03-31T15:03:29.434+0000 7f396fe00640 1 -- 172.21.15.16:0/2616443455 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f397006ff70 msgr2=0x7f39700c8c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:29.433 INFO:teuthology.misc.health.smithi016.stderr:2024-03-31T15:03:29.434+0000 7f396fe00640 1 --2- 172.21.15.16:0/2616443455 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f397006ff70 0x7f39700c8c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:29.433 INFO:teuthology.misc.health.smithi016.stderr:2024-03-31T15:03:29.434+0000 7f396fe00640 1 -- 172.21.15.16:0/2616443455 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3970146650 con 0x7f39700cf0a0 2024-03-31T15:03:29.433 INFO:teuthology.misc.health.smithi016.stderr:2024-03-31T15:03:29.434+0000 7f396fe00640 1 --2- 172.21.15.16:0/2616443455 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f39700cf0a0 0x7f39700c9c40 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f395c00c3a0 tx=0x7f395c00c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:03:29.434 INFO:teuthology.misc.health.smithi016.stderr:2024-03-31T15:03:29.434+0000 7f396cc00640 1 -- 172.21.15.16:0/2616443455 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f395c0052d0 con 0x7f39700cf0a0 2024-03-31T15:03:29.434 INFO:teuthology.misc.health.smithi016.stderr:2024-03-31T15:03:29.434+0000 7f396cc00640 1 -- 172.21.15.16:0/2616443455 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f395c005470 con 0x7f39700cf0a0 2024-03-31T15:03:29.434 INFO:teuthology.misc.health.smithi016.stderr:2024-03-31T15:03:29.434+0000 7f396cc00640 1 -- 172.21.15.16:0/2616443455 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f395c007260 con 0x7f39700cf0a0 2024-03-31T15:03:29.434 INFO:teuthology.misc.health.smithi016.stderr:2024-03-31T15:03:29.434+0000 7f3977a00640 1 -- 172.21.15.16:0/2616443455 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3970146930 con 0x7f39700cf0a0 2024-03-31T15:03:29.434 INFO:teuthology.misc.health.smithi016.stderr:2024-03-31T15:03:29.434+0000 7f3977a00640 1 -- 172.21.15.16:0/2616443455 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3970146ef0 con 0x7f39700cf0a0 2024-03-31T15:03:29.434 INFO:teuthology.misc.health.smithi016.stderr:2024-03-31T15:03:29.438+0000 7f3977a00640 1 -- 172.21.15.16:0/2616443455 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3970070cb0 con 0x7f39700cf0a0 2024-03-31T15:03:29.435 INFO:teuthology.misc.health.smithi016.stderr:2024-03-31T15:03:29.438+0000 7f396cc00640 1 -- 172.21.15.16:0/2616443455 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f395c007400 con 0x7f39700cf0a0 2024-03-31T15:03:29.435 INFO:teuthology.misc.health.smithi016.stderr:2024-03-31T15:03:29.438+0000 7f396cc00640 1 --2- 172.21.15.16:0/2616443455 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f39500683b0 0x7f395006a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:29.435 INFO:teuthology.misc.health.smithi016.stderr:2024-03-31T15:03:29.438+0000 7f396cc00640 1 -- 172.21.15.16:0/2616443455 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 4857+0+0 (secure 0 0 0) 0x7f395c085a20 con 0x7f39700cf0a0 2024-03-31T15:03:29.435 INFO:teuthology.misc.health.smithi016.stderr:2024-03-31T15:03:29.438+0000 7f3975000640 1 --2- 172.21.15.16:0/2616443455 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f39500683b0 0x7f395006a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:29.435 INFO:teuthology.misc.health.smithi016.stderr:2024-03-31T15:03:29.438+0000 7f3975000640 1 --2- 172.21.15.16:0/2616443455 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f39500683b0 0x7f395006a870 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f39640011f0 tx=0x7f3964007d40 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:03:29.439 INFO:teuthology.misc.health.smithi016.stderr:2024-03-31T15:03:29.442+0000 7f396cc00640 1 -- 172.21.15.16:0/2616443455 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f395c055cd0 con 0x7f39700cf0a0 2024-03-31T15:03:29.605 INFO:teuthology.misc.health.smithi016.stderr:2024-03-31T15:03:29.610+0000 7f3977a00640 1 -- 172.21.15.16:0/2616443455 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f397004f9a0 con 0x7f39700cf0a0 2024-03-31T15:03:29.606 INFO:teuthology.misc.health.smithi016.stderr:2024-03-31T15:03:29.610+0000 7f396cc00640 1 -- 172.21.15.16:0/2616443455 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v17) ==== 74+0+12812 (secure 0 0 0) 0x7f395c059fd0 con 0x7f39700cf0a0 2024-03-31T15:03:29.607 INFO:teuthology.misc.health.smithi016.stdout: 2024-03-31T15:03:29.607 INFO:teuthology.misc.health.smithi016.stdout:{"epoch":17,"fsid":"286eb5da-fb47-4550-b290-d9519c13b9a5","created":"2024-03-31T15:03:11.438952+0000","modified":"2024-03-31T15:03:24.897591+0000","last_up_change":"2024-03-31T15:03:22.885535+0000","last_in_change":"2024-03-31T15:03:13.941667+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-31T15:03:22.550787+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"92a66450-ef19-40d8-ac3d-d760b23b0bda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6801","nonce":932712244}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6803","nonce":932712244}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6807","nonce":932712244}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6805","nonce":932712244}]},"public_addr":"172.21.15.16:6801/932712244","cluster_addr":"172.21.15.16:6803/932712244","heartbeat_back_addr":"172.21.15.16:6807/932712244","heartbeat_front_addr":"172.21.15.16:6805/932712244","state":["exists","up"]},{"osd":1,"uuid":"da9d1f2b-1968-4959-b843-faa231988e92","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6810","nonce":1398737533}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6811","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6813","nonce":1398737533}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6821","nonce":1398737533}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6815","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6816","nonce":1398737533}]},"public_addr":"172.21.15.16:6810/1398737533","cluster_addr":"172.21.15.16:6813/1398737533","heartbeat_back_addr":"172.21.15.16:6821/1398737533","heartbeat_front_addr":"172.21.15.16:6816/1398737533","state":["exists","up"]},{"osd":2,"uuid":"01a73662-9a21-4b1f-bcbb-42b9636a4638","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6819","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6822","nonce":3018526646}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6825","nonce":3018526646}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6831","nonce":3018526646}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6827","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6829","nonce":3018526646}]},"public_addr":"172.21.15.16:6822/3018526646","cluster_addr":"172.21.15.16:6825/3018526646","heartbeat_back_addr":"172.21.15.16:6831/3018526646","heartbeat_front_addr":"172.21.15.16:6829/3018526646","state":["exists","up"]},{"osd":3,"uuid":"c7771fd3-e18e-4500-ac38-23cf10ebcd75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6809","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6812","nonce":3360352974}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6817","nonce":3360352974}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6828","nonce":3360352974}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6823","nonce":3360352974}]},"public_addr":"172.21.15.16:6812/3360352974","cluster_addr":"172.21.15.16:6817/3360352974","heartbeat_back_addr":"172.21.15.16:6828/3360352974","heartbeat_front_addr":"172.21.15.16:6823/3360352974","state":["exists","up"]},{"osd":4,"uuid":"88260730-1482-4fbd-a40f-cb1124e9df4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6801","nonce":1856844856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6803","nonce":1856844856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6807","nonce":1856844856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6805","nonce":1856844856}]},"public_addr":"172.21.15.89:6801/1856844856","cluster_addr":"172.21.15.89:6803/1856844856","heartbeat_back_addr":"172.21.15.89:6807/1856844856","heartbeat_front_addr":"172.21.15.89:6805/1856844856","state":["exists","up"]},{"osd":5,"uuid":"4cb2c00c-fae0-4dd8-9d27-ec4f0093ccf7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6814","nonce":50395527}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6817","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6820","nonce":50395527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6831","nonce":50395527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6823","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6826","nonce":50395527}]},"public_addr":"172.21.15.89:6814/50395527","cluster_addr":"172.21.15.89:6820/50395527","heartbeat_back_addr":"172.21.15.89:6831/50395527","heartbeat_front_addr":"172.21.15.89:6826/50395527","state":["exists","up"]},{"osd":6,"uuid":"81ce0c19-2554-4cae-acbe-66a05e9e9b9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6809","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6812","nonce":3470471752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6815","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6818","nonce":3470471752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6827","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6829","nonce":3470471752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6821","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6824","nonce":3470471752}]},"public_addr":"172.21.15.89:6812/3470471752","cluster_addr":"172.21.15.89:6818/3470471752","heartbeat_back_addr":"172.21.15.89:6829/3470471752","heartbeat_front_addr":"172.21.15.89:6824/3470471752","state":["exists","up"]},{"osd":7,"uuid":"85e4313b-d37d-442e-8042-153bf20a847d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6811","nonce":1796778630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6813","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6816","nonce":1796778630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6825","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6828","nonce":1796778630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6819","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6822","nonce":1796778630}]},"public_addr":"172.21.15.89:6811/1796778630","cluster_addr":"172.21.15.89:6816/1796778630","heartbeat_back_addr":"172.21.15.89:6828/1796778630","heartbeat_front_addr":"172.21.15.89:6822/1796778630","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.439088+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.529970+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.680512+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:21.044161+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.582111+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.536558+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.409522+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.590229+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-31T15:03:29.608 INFO:teuthology.misc.health.smithi016.stderr:2024-03-31T15:03:29.610+0000 7f3977a00640 1 -- 172.21.15.16:0/2616443455 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f39500683b0 msgr2=0x7f395006a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:29.608 INFO:teuthology.misc.health.smithi016.stderr:2024-03-31T15:03:29.610+0000 7f3977a00640 1 --2- 172.21.15.16:0/2616443455 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f39500683b0 0x7f395006a870 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f39640011f0 tx=0x7f3964007d40 comp rx=0 tx=0).stop 2024-03-31T15:03:29.608 INFO:teuthology.misc.health.smithi016.stderr:2024-03-31T15:03:29.610+0000 7f3977a00640 1 -- 172.21.15.16:0/2616443455 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f39700cf0a0 msgr2=0x7f39700c9c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:29.608 INFO:teuthology.misc.health.smithi016.stderr:2024-03-31T15:03:29.610+0000 7f3977a00640 1 --2- 172.21.15.16:0/2616443455 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f39700cf0a0 0x7f39700c9c40 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f395c00c3a0 tx=0x7f395c00c860 comp rx=0 tx=0).stop 2024-03-31T15:03:29.608 INFO:teuthology.misc.health.smithi016.stderr:2024-03-31T15:03:29.610+0000 7f3977a00640 1 -- 172.21.15.16:0/2616443455 shutdown_connections 2024-03-31T15:03:29.608 INFO:teuthology.misc.health.smithi016.stderr:2024-03-31T15:03:29.610+0000 7f3977a00640 1 --2- 172.21.15.16:0/2616443455 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f39500683b0 0x7f395006a870 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:29.608 INFO:teuthology.misc.health.smithi016.stderr:2024-03-31T15:03:29.610+0000 7f3977a00640 1 --2- 172.21.15.16:0/2616443455 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f3970162d50 0x7f39700ce1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:29.608 INFO:teuthology.misc.health.smithi016.stderr:2024-03-31T15:03:29.610+0000 7f3977a00640 1 --2- 172.21.15.16:0/2616443455 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f39700cf0a0 0x7f39700c9c40 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:29.608 INFO:teuthology.misc.health.smithi016.stderr:2024-03-31T15:03:29.610+0000 7f3977a00640 1 --2- 172.21.15.16:0/2616443455 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f397006ff70 0x7f39700c8c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:29.608 INFO:teuthology.misc.health.smithi016.stderr:2024-03-31T15:03:29.610+0000 7f3977a00640 1 -- 172.21.15.16:0/2616443455 >> 172.21.15.16:0/2616443455 conn(0x7f39700b3ca0 msgr2=0x7f397015de50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:03:29.609 INFO:teuthology.misc.health.smithi016.stderr:2024-03-31T15:03:29.610+0000 7f3977a00640 1 -- 172.21.15.16:0/2616443455 shutdown_connections 2024-03-31T15:03:29.609 INFO:teuthology.misc.health.smithi016.stderr:2024-03-31T15:03:29.610+0000 7f3977a00640 1 -- 172.21.15.16:0/2616443455 wait complete. 2024-03-31T15:03:29.625 DEBUG:teuthology.misc:8 of 8 OSDs are up 2024-03-31T15:03:29.625 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-03-31T15:03:29.625 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-31T15:03:29.626 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-31T15:03:29.629 INFO:tasks.ceph.mds.a:Started 2024-03-31T15:03:29.629 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-31T15:03:29.629 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-31T15:03:29.632 INFO:tasks.ceph.mds.c:Started 2024-03-31T15:03:29.632 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-31T15:03:29.632 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-31T15:03:29.635 INFO:tasks.ceph.mds.b:Started 2024-03-31T15:03:29.635 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-31T15:03:29.635 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-31T15:03:29.638 INFO:tasks.ceph.mds.d:Started 2024-03-31T15:03:29.639 INFO:tasks.ceph:Setting up CephFS filesystem(s)... 2024-03-31T15:03:29.639 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-31T15:03:29.725 INFO:tasks.ceph.mds.a.smithi016.stdout:starting mds.a at 2024-03-31T15:03:29.726 INFO:tasks.ceph.mds.c.smithi016.stdout:starting mds.c at 2024-03-31T15:03:29.744 INFO:tasks.ceph.mds.d.smithi089.stdout:starting mds.d at 2024-03-31T15:03:29.747 INFO:tasks.ceph.mds.b.smithi089.stdout:starting mds.b at 2024-03-31T15:03:29.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:29.790+0000 7fb69f000640 1 Processor -- start 2024-03-31T15:03:29.787 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:29.790+0000 7fb69f000640 1 -- start start 2024-03-31T15:03:29.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:29.790+0000 7fb69f000640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb6980728b0 0x7fb6980b1cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:29.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:29.790+0000 7fb69f000640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb6980b2200 0x7fb69815cd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:29.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:29.790+0000 7fb69f000640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb698160e70 con 0x7fb698145d10 2024-03-31T15:03:29.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:29.790+0000 7fb69f000640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7fb6981602f0 con 0x7fb6980b2200 2024-03-31T15:03:29.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:29.790+0000 7fb69f000640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fb6981605d0 con 0x7fb6980728b0 2024-03-31T15:03:29.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:29.794+0000 7fb697e00640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fb698145d10 0x7fb6980cf790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:40616/0 (socket says 172.21.15.16:40616) 2024-03-31T15:03:29.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:29.794+0000 7fb697e00640 1 -- 172.21.15.16:0/2853198849 learned_addr learned my addr 172.21.15.16:0/2853198849 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:03:29.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:29.794+0000 7fb69d000640 1 --2- 172.21.15.16:0/2853198849 >> v2:172.21.15.16:3300/0 conn(0x7fb6980b2200 0x7fb69815cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:29.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:29.794+0000 7fb68f400640 1 --2- 172.21.15.16:0/2853198849 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb6980728b0 0x7fb6980b1cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:29.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:29.794+0000 7fb697400640 1 -- 172.21.15.16:0/2853198849 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 988639036 0 0) 0x7fb698160e70 con 0x7fb698145d10 2024-03-31T15:03:29.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:29.794+0000 7fb69d000640 1 -- 172.21.15.16:0/2853198849 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb6980728b0 msgr2=0x7fb6980b1cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:03:29.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:29.794+0000 7fb69d000640 1 --2- 172.21.15.16:0/2853198849 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb6980728b0 0x7fb6980b1cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:29.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:29.794+0000 7fb69d000640 1 -- 172.21.15.16:0/2853198849 >> v1:172.21.15.16:6789/0 conn(0x7fb698145d10 legacy=0x7fb6980cf790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:29.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:29.794+0000 7fb69d000640 1 -- 172.21.15.16:0/2853198849 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6981608b0 con 0x7fb6980b2200 2024-03-31T15:03:29.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:29.794+0000 7fb69d000640 1 --2- 172.21.15.16:0/2853198849 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb6980b2200 0x7fb69815cd50 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fb690009d90 tx=0x7fb69001c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4110f39a7bb889a server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:03:29.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:29.794+0000 7fb697400640 1 -- 172.21.15.16:0/2853198849 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6900b2070 con 0x7fb6980b2200 2024-03-31T15:03:29.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:29.794+0000 7fb697400640 1 -- 172.21.15.16:0/2853198849 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb6900a5830 con 0x7fb6980b2200 2024-03-31T15:03:29.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:29.794+0000 7fb697400640 1 -- 172.21.15.16:0/2853198849 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6900a5c00 con 0x7fb6980b2200 2024-03-31T15:03:29.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:29.798+0000 7fb69f000640 1 -- 172.21.15.16:0/2853198849 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb6980b2200 msgr2=0x7fb69815cd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:29.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:29.798+0000 7fb69f000640 1 --2- 172.21.15.16:0/2853198849 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb6980b2200 0x7fb69815cd50 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fb690009d90 tx=0x7fb69001c970 comp rx=0 tx=0).stop 2024-03-31T15:03:29.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:29.798+0000 7fb69f000640 1 -- 172.21.15.16:0/2853198849 shutdown_connections 2024-03-31T15:03:29.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:29.798+0000 7fb69f000640 1 --2- 172.21.15.16:0/2853198849 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb6980b2200 0x7fb69815cd50 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:29.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:29.798+0000 7fb69f000640 1 --2- 172.21.15.16:0/2853198849 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb6980728b0 0x7fb6980b1cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:29.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:29.798+0000 7fb69f000640 1 -- 172.21.15.16:0/2853198849 >> 172.21.15.16:0/2853198849 conn(0x7fb69806b620 msgr2=0x7fb69806ba30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:03:29.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:29.798+0000 7fb69f000640 1 -- 172.21.15.16:0/2853198849 shutdown_connections 2024-03-31T15:03:29.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:29.798+0000 7fb69f000640 1 -- 172.21.15.16:0/2853198849 wait complete. 2024-03-31T15:03:29.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:29.798+0000 7fb69f000640 1 Processor -- start 2024-03-31T15:03:29.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:29.798+0000 7fb69f000640 1 -- start start 2024-03-31T15:03:29.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:29.798+0000 7fb69f000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb6980728b0 0x7fb698141310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:29.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:29.798+0000 7fb69f000640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb6980b2200 0x7fb698157560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:29.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:29.798+0000 7fb69f000640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb698145d10 0x7fb698141dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:29.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:29.798+0000 7fb69f000640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fb698072e10 con 0x7fb6980728b0 2024-03-31T15:03:29.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:29.798+0000 7fb69f000640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fb698072c90 con 0x7fb698145d10 2024-03-31T15:03:29.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:29.798+0000 7fb69f000640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fb6981690f0 con 0x7fb6980b2200 2024-03-31T15:03:29.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:29.798+0000 7fb697e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb6980728b0 0x7fb698141310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:29.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:29.798+0000 7fb697e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb6980728b0 0x7fb698141310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59526/0 (socket says 172.21.15.16:59526) 2024-03-31T15:03:29.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:29.798+0000 7fb697e00640 1 -- 172.21.15.16:0/2209953384 learned_addr learned my addr 172.21.15.16:0/2209953384 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:03:29.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:29.798+0000 7fb69d000640 1 --2- 172.21.15.16:0/2209953384 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb698145d10 0x7fb698141dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:29.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:29.798+0000 7fb68f400640 1 --2- 172.21.15.16:0/2209953384 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb6980b2200 0x7fb698157560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:29.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:29.798+0000 7fb697e00640 1 -- 172.21.15.16:0/2209953384 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb6980b2200 msgr2=0x7fb698157560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:29.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:29.798+0000 7fb697e00640 1 --2- 172.21.15.16:0/2209953384 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb6980b2200 0x7fb698157560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:29.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:29.798+0000 7fb697e00640 1 -- 172.21.15.16:0/2209953384 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb698145d10 msgr2=0x7fb698141dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:29.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:29.798+0000 7fb697e00640 1 --2- 172.21.15.16:0/2209953384 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb698145d10 0x7fb698141dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:29.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:29.798+0000 7fb697e00640 1 -- 172.21.15.16:0/2209953384 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb698140870 con 0x7fb6980728b0 2024-03-31T15:03:29.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:29.798+0000 7fb697e00640 1 --2- 172.21.15.16:0/2209953384 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb6980728b0 0x7fb698141310 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fb68800c480 tx=0x7fb68800c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:03:29.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:29.802+0000 7fb694c00640 1 -- 172.21.15.16:0/2209953384 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb688018070 con 0x7fb6980728b0 2024-03-31T15:03:29.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:29.802+0000 7fb694c00640 1 -- 172.21.15.16:0/2209953384 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb688007500 con 0x7fb6980728b0 2024-03-31T15:03:29.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:29.802+0000 7fb694c00640 1 -- 172.21.15.16:0/2209953384 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb688004a30 con 0x7fb6980728b0 2024-03-31T15:03:29.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:29.802+0000 7fb69f000640 1 -- 172.21.15.16:0/2209953384 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb698139430 con 0x7fb6980728b0 2024-03-31T15:03:29.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:29.802+0000 7fb69f000640 1 -- 172.21.15.16:0/2209953384 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb698140b10 con 0x7fb6980728b0 2024-03-31T15:03:29.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:29.802+0000 7fb69f000640 1 -- 172.21.15.16:0/2209953384 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb698140f40 con 0x7fb6980728b0 2024-03-31T15:03:29.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:29.802+0000 7fb694c00640 1 -- 172.21.15.16:0/2209953384 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fb688004bd0 con 0x7fb6980728b0 2024-03-31T15:03:29.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:29.802+0000 7fb694c00640 1 --2- 172.21.15.16:0/2209953384 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fb674068380 0x7fb67406a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:29.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:29.802+0000 7fb694c00640 1 -- 172.21.15.16:0/2209953384 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 4857+0+0 (secure 0 0 0) 0x7fb6880858d0 con 0x7fb6980728b0 2024-03-31T15:03:29.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:29.802+0000 7fb68f400640 1 --2- 172.21.15.16:0/2209953384 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fb674068380 0x7fb67406a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:29.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:29.806+0000 7fb68f400640 1 --2- 172.21.15.16:0/2209953384 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fb674068380 0x7fb67406a840 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fb69815fad0 tx=0x7fb680008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:03:29.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:29.810+0000 7fb694c00640 1 -- 172.21.15.16:0/2209953384 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fb6880561d0 con 0x7fb6980728b0 2024-03-31T15:03:29.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:29.974+0000 7fb69f000640 1 -- 172.21.15.16:0/2209953384 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fb698069950 con 0x7fb6980728b0 2024-03-31T15:03:29.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:29.974+0000 7fb694c00640 1 -- 172.21.15.16:0/2209953384 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v17) ==== 74+0+12812 (secure 0 0 0) 0x7fb688055ad0 con 0x7fb6980728b0 2024-03-31T15:03:29.971 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:03:29.971 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":17,"fsid":"286eb5da-fb47-4550-b290-d9519c13b9a5","created":"2024-03-31T15:03:11.438952+0000","modified":"2024-03-31T15:03:24.897591+0000","last_up_change":"2024-03-31T15:03:22.885535+0000","last_in_change":"2024-03-31T15:03:13.941667+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-31T15:03:22.550787+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"92a66450-ef19-40d8-ac3d-d760b23b0bda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6801","nonce":932712244}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6803","nonce":932712244}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6807","nonce":932712244}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6805","nonce":932712244}]},"public_addr":"172.21.15.16:6801/932712244","cluster_addr":"172.21.15.16:6803/932712244","heartbeat_back_addr":"172.21.15.16:6807/932712244","heartbeat_front_addr":"172.21.15.16:6805/932712244","state":["exists","up"]},{"osd":1,"uuid":"da9d1f2b-1968-4959-b843-faa231988e92","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6810","nonce":1398737533}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6811","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6813","nonce":1398737533}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6821","nonce":1398737533}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6815","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6816","nonce":1398737533}]},"public_addr":"172.21.15.16:6810/1398737533","cluster_addr":"172.21.15.16:6813/1398737533","heartbeat_back_addr":"172.21.15.16:6821/1398737533","heartbeat_front_addr":"172.21.15.16:6816/1398737533","state":["exists","up"]},{"osd":2,"uuid":"01a73662-9a21-4b1f-bcbb-42b9636a4638","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6819","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6822","nonce":3018526646}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6825","nonce":3018526646}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6831","nonce":3018526646}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6827","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6829","nonce":3018526646}]},"public_addr":"172.21.15.16:6822/3018526646","cluster_addr":"172.21.15.16:6825/3018526646","heartbeat_back_addr":"172.21.15.16:6831/3018526646","heartbeat_front_addr":"172.21.15.16:6829/3018526646","state":["exists","up"]},{"osd":3,"uuid":"c7771fd3-e18e-4500-ac38-23cf10ebcd75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6809","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6812","nonce":3360352974}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6817","nonce":3360352974}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6828","nonce":3360352974}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6823","nonce":3360352974}]},"public_addr":"172.21.15.16:6812/3360352974","cluster_addr":"172.21.15.16:6817/3360352974","heartbeat_back_addr":"172.21.15.16:6828/3360352974","heartbeat_front_addr":"172.21.15.16:6823/3360352974","state":["exists","up"]},{"osd":4,"uuid":"88260730-1482-4fbd-a40f-cb1124e9df4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6801","nonce":1856844856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6803","nonce":1856844856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6807","nonce":1856844856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6805","nonce":1856844856}]},"public_addr":"172.21.15.89:6801/1856844856","cluster_addr":"172.21.15.89:6803/1856844856","heartbeat_back_addr":"172.21.15.89:6807/1856844856","heartbeat_front_addr":"172.21.15.89:6805/1856844856","state":["exists","up"]},{"osd":5,"uuid":"4cb2c00c-fae0-4dd8-9d27-ec4f0093ccf7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6814","nonce":50395527}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6817","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6820","nonce":50395527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6831","nonce":50395527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6823","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6826","nonce":50395527}]},"public_addr":"172.21.15.89:6814/50395527","cluster_addr":"172.21.15.89:6820/50395527","heartbeat_back_addr":"172.21.15.89:6831/50395527","heartbeat_front_addr":"172.21.15.89:6826/50395527","state":["exists","up"]},{"osd":6,"uuid":"81ce0c19-2554-4cae-acbe-66a05e9e9b9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6809","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6812","nonce":3470471752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6815","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6818","nonce":3470471752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6827","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6829","nonce":3470471752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6821","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6824","nonce":3470471752}]},"public_addr":"172.21.15.89:6812/3470471752","cluster_addr":"172.21.15.89:6818/3470471752","heartbeat_back_addr":"172.21.15.89:6829/3470471752","heartbeat_front_addr":"172.21.15.89:6824/3470471752","state":["exists","up"]},{"osd":7,"uuid":"85e4313b-d37d-442e-8042-153bf20a847d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6811","nonce":1796778630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6813","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6816","nonce":1796778630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6825","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6828","nonce":1796778630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6819","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6822","nonce":1796778630}]},"public_addr":"172.21.15.89:6811/1796778630","cluster_addr":"172.21.15.89:6816/1796778630","heartbeat_back_addr":"172.21.15.89:6828/1796778630","heartbeat_front_addr":"172.21.15.89:6822/1796778630","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.439088+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.529970+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.680512+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:21.044161+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.582111+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.536558+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.409522+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.590229+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-31T15:03:29.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:29.978+0000 7fb69f000640 1 -- 172.21.15.16:0/2209953384 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fb674068380 msgr2=0x7fb67406a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:29.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:29.978+0000 7fb69f000640 1 --2- 172.21.15.16:0/2209953384 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fb674068380 0x7fb67406a840 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fb69815fad0 tx=0x7fb680008040 comp rx=0 tx=0).stop 2024-03-31T15:03:29.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:29.978+0000 7fb69f000640 1 -- 172.21.15.16:0/2209953384 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb6980728b0 msgr2=0x7fb698141310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:29.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:29.978+0000 7fb69f000640 1 --2- 172.21.15.16:0/2209953384 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb6980728b0 0x7fb698141310 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fb68800c480 tx=0x7fb68800c940 comp rx=0 tx=0).stop 2024-03-31T15:03:29.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:29.978+0000 7fb69f000640 1 -- 172.21.15.16:0/2209953384 shutdown_connections 2024-03-31T15:03:29.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:29.978+0000 7fb69f000640 1 --2- 172.21.15.16:0/2209953384 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fb674068380 0x7fb67406a840 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:29.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:29.978+0000 7fb69f000640 1 --2- 172.21.15.16:0/2209953384 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb698145d10 0x7fb698141dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:29.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:29.978+0000 7fb69f000640 1 --2- 172.21.15.16:0/2209953384 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb6980b2200 0x7fb698157560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:29.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:29.978+0000 7fb69f000640 1 --2- 172.21.15.16:0/2209953384 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb6980728b0 0x7fb698141310 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:29.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:29.978+0000 7fb69f000640 1 -- 172.21.15.16:0/2209953384 >> 172.21.15.16:0/2209953384 conn(0x7fb69806b620 msgr2=0x7fb6980b4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:03:29.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:29.978+0000 7fb69f000640 1 -- 172.21.15.16:0/2209953384 shutdown_connections 2024-03-31T15:03:29.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:29.978+0000 7fb69f000640 1 -- 172.21.15.16:0/2209953384 wait complete. 2024-03-31T15:03:29.991 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-31T15:03:22.550787+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-31T15:03:29.992 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-31T15:03:30.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:30.138+0000 7f17a1a00640 1 Processor -- start 2024-03-31T15:03:30.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:30.138+0000 7f17a1a00640 1 -- start start 2024-03-31T15:03:30.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:30.138+0000 7f17a1a00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f179c06f920 0x7f179c06fd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:30.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:30.138+0000 7f17a1a00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f179c0d0a60 0x7f179c0cea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:30.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:30.138+0000 7f17a1a00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f179c0cef80 0x7f179c070e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:30.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:30.138+0000 7f17a1a00640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f179c13f670 con 0x7f179c0cef80 2024-03-31T15:03:30.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:30.138+0000 7f17a1a00640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f179c070240 con 0x7f179c06f920 2024-03-31T15:03:30.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:30.138+0000 7f17a1a00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f179c15ca70 con 0x7f179c0d0a60 2024-03-31T15:03:30.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:30.138+0000 7f179b400640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f179c0cef80 0x7f179c070e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:30.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:30.138+0000 7f179b400640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f179c0cef80 0x7f179c070e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59540/0 (socket says 172.21.15.16:59540) 2024-03-31T15:03:30.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:30.138+0000 7f179b400640 1 -- 172.21.15.16:0/2268955516 learned_addr learned my addr 172.21.15.16:0/2268955516 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:03:30.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:30.142+0000 7f1792000640 1 --2- 172.21.15.16:0/2268955516 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f179c0d0a60 0x7f179c0cea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:30.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:30.142+0000 7f179b400640 1 -- 172.21.15.16:0/2268955516 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f179c0d0a60 msgr2=0x7f179c0cea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:03:30.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:30.142+0000 7f179aa00640 1 --2- 172.21.15.16:0/2268955516 >> v2:172.21.15.89:3300/0 conn(0x7f179c06f920 0x7f179c06fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:30.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:30.142+0000 7f179b400640 1 --2- 172.21.15.16:0/2268955516 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f179c0d0a60 0x7f179c0cea40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:30.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:30.142+0000 7f179b400640 1 -- 172.21.15.16:0/2268955516 >> v2:172.21.15.89:3300/0 conn(0x7f179c06f920 msgr2=0x7f179c06fd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:03:30.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:30.142+0000 7f179b400640 1 --2- 172.21.15.16:0/2268955516 >> v2:172.21.15.89:3300/0 conn(0x7f179c06f920 0x7f179c06fd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:30.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:30.142+0000 7f179b400640 1 -- 172.21.15.16:0/2268955516 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f179c15cd50 con 0x7f179c0cef80 2024-03-31T15:03:30.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:30.142+0000 7f1792000640 1 --2- 172.21.15.16:0/2268955516 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f179c0d0a60 0x7f179c0cea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:03:30.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:30.142+0000 7f179aa00640 1 --2- 172.21.15.16:0/2268955516 >> v2:172.21.15.89:3300/0 conn(0x7f179c06f920 0x7f179c06fd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:03:30.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:30.142+0000 7f179b400640 1 --2- 172.21.15.16:0/2268955516 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f179c0cef80 0x7f179c070e60 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f179c0708e0 tx=0x7f179401c260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f48c17606095a90 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:03:30.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:30.142+0000 7f179a000640 1 -- 172.21.15.16:0/2268955516 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17940b0070 con 0x7f179c0cef80 2024-03-31T15:03:30.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:30.142+0000 7f179a000640 1 -- 172.21.15.16:0/2268955516 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f17940ab600 con 0x7f179c0cef80 2024-03-31T15:03:30.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:30.142+0000 7f179a000640 1 -- 172.21.15.16:0/2268955516 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17940ab9d0 con 0x7f179c0cef80 2024-03-31T15:03:30.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:30.142+0000 7f17a1a00640 1 -- 172.21.15.16:0/2268955516 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f179c0cef80 msgr2=0x7f179c070e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:30.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:30.142+0000 7f17a1a00640 1 --2- 172.21.15.16:0/2268955516 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f179c0cef80 0x7f179c070e60 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f179c0708e0 tx=0x7f179401c260 comp rx=0 tx=0).stop 2024-03-31T15:03:30.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:30.146+0000 7f17a1a00640 1 -- 172.21.15.16:0/2268955516 shutdown_connections 2024-03-31T15:03:30.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:30.146+0000 7f17a1a00640 1 --2- 172.21.15.16:0/2268955516 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f179c0cef80 0x7f179c070e60 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:30.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:30.146+0000 7f17a1a00640 1 --2- 172.21.15.16:0/2268955516 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f179c0d0a60 0x7f179c0cea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:30.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:30.146+0000 7f17a1a00640 1 --2- 172.21.15.16:0/2268955516 >> v2:172.21.15.89:3300/0 conn(0x7f179c06f920 0x7f179c06fd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:30.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:30.146+0000 7f17a1a00640 1 -- 172.21.15.16:0/2268955516 >> 172.21.15.16:0/2268955516 conn(0x7f179c067930 msgr2=0x7f179c0b3ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:03:30.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:30.146+0000 7f17a1a00640 1 -- 172.21.15.16:0/2268955516 shutdown_connections 2024-03-31T15:03:30.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:30.146+0000 7f17a1a00640 1 -- 172.21.15.16:0/2268955516 wait complete. 2024-03-31T15:03:30.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:30.146+0000 7f17a1a00640 1 Processor -- start 2024-03-31T15:03:30.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:30.146+0000 7f17a1a00640 1 -- start start 2024-03-31T15:03:30.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:30.146+0000 7f17a1a00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f179c06f920 0x7f179c13ec40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:30.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:30.146+0000 7f17a1a00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f179c0cef80 0x7f179c0b1c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:30.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:30.146+0000 7f17a1a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f179c0d0a60 0x7f179c158800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:30.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:30.146+0000 7f17a1a00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f179c1419c0 con 0x7f179c0d0a60 2024-03-31T15:03:30.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:30.146+0000 7f17a1a00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f179c15c720 con 0x7f179c0cef80 2024-03-31T15:03:30.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:30.146+0000 7f17a1a00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f179c141b40 con 0x7f179c06f920 2024-03-31T15:03:30.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:30.146+0000 7f179b400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f179c0d0a60 0x7f179c158800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:30.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:30.146+0000 7f179b400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f179c0d0a60 0x7f179c158800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59542/0 (socket says 172.21.15.16:59542) 2024-03-31T15:03:30.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:30.146+0000 7f179b400640 1 -- 172.21.15.16:0/377299222 learned_addr learned my addr 172.21.15.16:0/377299222 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:03:30.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:30.146+0000 7f179aa00640 1 --2- 172.21.15.16:0/377299222 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f179c06f920 0x7f179c13ec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:30.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:30.146+0000 7f179b400640 1 -- 172.21.15.16:0/377299222 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f179c06f920 msgr2=0x7f179c13ec40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:30.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:30.146+0000 7f1792000640 1 --2- 172.21.15.16:0/377299222 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f179c0cef80 0x7f179c0b1c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:30.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:30.146+0000 7f179b400640 1 --2- 172.21.15.16:0/377299222 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f179c06f920 0x7f179c13ec40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:30.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:30.146+0000 7f179b400640 1 -- 172.21.15.16:0/377299222 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f179c0cef80 msgr2=0x7f179c0b1c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:30.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:30.146+0000 7f179b400640 1 --2- 172.21.15.16:0/377299222 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f179c0cef80 0x7f179c0b1c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:30.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:30.146+0000 7f179b400640 1 -- 172.21.15.16:0/377299222 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f179c0b2550 con 0x7f179c0d0a60 2024-03-31T15:03:30.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:30.146+0000 7f179aa00640 1 --2- 172.21.15.16:0/377299222 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f179c06f920 0x7f179c13ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:03:30.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:30.146+0000 7f1792000640 1 --2- 172.21.15.16:0/377299222 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f179c0cef80 0x7f179c0b1c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:03:30.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:30.146+0000 7f179b400640 1 --2- 172.21.15.16:0/377299222 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f179c0d0a60 0x7f179c158800 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f179401cb00 tx=0x7f1794019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:03:30.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:30.146+0000 7f1793400640 1 -- 172.21.15.16:0/377299222 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17940ba070 con 0x7f179c0d0a60 2024-03-31T15:03:30.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:30.150+0000 7f1793400640 1 -- 172.21.15.16:0/377299222 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f17940b4650 con 0x7f179c0d0a60 2024-03-31T15:03:30.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:30.150+0000 7f17a1a00640 1 -- 172.21.15.16:0/377299222 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f179c0b2830 con 0x7f179c0d0a60 2024-03-31T15:03:30.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:30.150+0000 7f1793400640 1 -- 172.21.15.16:0/377299222 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17940b0040 con 0x7f179c0d0a60 2024-03-31T15:03:30.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:30.150+0000 7f17a1a00640 1 -- 172.21.15.16:0/377299222 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f179c0b2df0 con 0x7f179c0d0a60 2024-03-31T15:03:30.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:30.150+0000 7f17a1a00640 1 -- 172.21.15.16:0/377299222 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f179c0b2ff0 con 0x7f179c0d0a60 2024-03-31T15:03:30.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:30.150+0000 7f1793400640 1 -- 172.21.15.16:0/377299222 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f17940bd070 con 0x7f179c0d0a60 2024-03-31T15:03:30.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:30.150+0000 7f1793400640 1 --2- 172.21.15.16:0/377299222 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f1778068380 0x7f177806a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:30.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:30.150+0000 7f1793400640 1 -- 172.21.15.16:0/377299222 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 4857+0+0 (secure 0 0 0) 0x7f179411f810 con 0x7f179c0d0a60 2024-03-31T15:03:30.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:30.150+0000 7f179aa00640 1 --2- 172.21.15.16:0/377299222 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f1778068380 0x7f177806a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:30.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:30.150+0000 7f179aa00640 1 --2- 172.21.15.16:0/377299222 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f1778068380 0x7f177806a840 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f178c00d430 tx=0x7f178c007530 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:03:30.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:30.154+0000 7f1793400640 1 -- 172.21.15.16:0/377299222 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f1794016110 con 0x7f179c0d0a60 2024-03-31T15:03:30.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:30.318+0000 7f17a1a00640 1 -- 172.21.15.16:0/377299222 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f179c0b50e0 con 0x7f179c0d0a60 2024-03-31T15:03:30.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:30.318+0000 7f1793400640 1 -- 172.21.15.16:0/377299222 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v17) ==== 93+0+10 (secure 0 0 0) 0x7f17940efa30 con 0x7f179c0d0a60 2024-03-31T15:03:30.316 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2024-03-31T15:03:30.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:30.322+0000 7f17a1a00640 1 -- 172.21.15.16:0/377299222 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f1778068380 msgr2=0x7f177806a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:30.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:30.322+0000 7f17a1a00640 1 --2- 172.21.15.16:0/377299222 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f1778068380 0x7f177806a840 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f178c00d430 tx=0x7f178c007530 comp rx=0 tx=0).stop 2024-03-31T15:03:30.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:30.322+0000 7f17a1a00640 1 -- 172.21.15.16:0/377299222 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f179c0d0a60 msgr2=0x7f179c158800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:30.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:30.322+0000 7f17a1a00640 1 --2- 172.21.15.16:0/377299222 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f179c0d0a60 0x7f179c158800 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f179401cb00 tx=0x7f1794019000 comp rx=0 tx=0).stop 2024-03-31T15:03:30.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:30.322+0000 7f17a1a00640 1 -- 172.21.15.16:0/377299222 shutdown_connections 2024-03-31T15:03:30.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:30.322+0000 7f17a1a00640 1 --2- 172.21.15.16:0/377299222 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f1778068380 0x7f177806a840 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:30.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:30.322+0000 7f17a1a00640 1 --2- 172.21.15.16:0/377299222 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f179c0d0a60 0x7f179c158800 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:30.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:30.322+0000 7f17a1a00640 1 --2- 172.21.15.16:0/377299222 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f179c0cef80 0x7f179c0b1c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:30.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:30.322+0000 7f17a1a00640 1 --2- 172.21.15.16:0/377299222 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f179c06f920 0x7f179c13ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:30.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:30.322+0000 7f17a1a00640 1 -- 172.21.15.16:0/377299222 >> 172.21.15.16:0/377299222 conn(0x7f179c067930 msgr2=0x7f179c165cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:03:30.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:30.322+0000 7f17a1a00640 1 -- 172.21.15.16:0/377299222 shutdown_connections 2024-03-31T15:03:30.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:30.322+0000 7f17a1a00640 1 -- 172.21.15.16:0/377299222 wait complete. 2024-03-31T15:03:30.336 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-31T15:03:30.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:30.478+0000 7f46fd600640 1 Processor -- start 2024-03-31T15:03:30.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:30.482+0000 7f46fd600640 1 -- start start 2024-03-31T15:03:30.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:30.482+0000 7f46fd600640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f46f8145e40 0x7f46f80cc740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:30.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:30.482+0000 7f46fd600640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f46f80ccc80 0x7f46f806f920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:30.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:30.482+0000 7f46fd600640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f46f806fe60 0x7f46f815fdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:30.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:30.482+0000 7f46fd600640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f46f80d06d0 con 0x7f46f8145e40 2024-03-31T15:03:30.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:30.482+0000 7f46fd600640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f46f80d0920 con 0x7f46f80ccc80 2024-03-31T15:03:30.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:30.482+0000 7f46fd600640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f46f80d0c00 con 0x7f46f806fe60 2024-03-31T15:03:30.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:30.482+0000 7f46f6a00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f46f8145e40 0x7f46f80cc740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:30.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:30.482+0000 7f46f6a00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f46f8145e40 0x7f46f80cc740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59548/0 (socket says 172.21.15.16:59548) 2024-03-31T15:03:30.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:30.482+0000 7f46f6a00640 1 -- 172.21.15.16:0/600051669 learned_addr learned my addr 172.21.15.16:0/600051669 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:03:30.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:30.482+0000 7f46ee000640 1 --2- 172.21.15.16:0/600051669 >> v2:172.21.15.89:3300/0 conn(0x7f46f80ccc80 0x7f46f806f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:30.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:30.482+0000 7f46f7400640 1 --2- 172.21.15.16:0/600051669 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f46f806fe60 0x7f46f815fdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:30.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:30.482+0000 7f46f6a00640 1 -- 172.21.15.16:0/600051669 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f46f806fe60 msgr2=0x7f46f815fdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:03:30.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:30.482+0000 7f46f6a00640 1 --2- 172.21.15.16:0/600051669 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f46f806fe60 0x7f46f815fdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:30.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:30.482+0000 7f46f6a00640 1 -- 172.21.15.16:0/600051669 >> v2:172.21.15.89:3300/0 conn(0x7f46f80ccc80 msgr2=0x7f46f806f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:03:30.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:30.482+0000 7f46f6a00640 1 --2- 172.21.15.16:0/600051669 >> v2:172.21.15.89:3300/0 conn(0x7f46f80ccc80 0x7f46f806f920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:30.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:30.482+0000 7f46f6a00640 1 -- 172.21.15.16:0/600051669 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46f80d1810 con 0x7f46f8145e40 2024-03-31T15:03:30.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:30.482+0000 7f46f7400640 1 --2- 172.21.15.16:0/600051669 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f46f806fe60 0x7f46f815fdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:03:30.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:30.482+0000 7f46ee000640 1 --2- 172.21.15.16:0/600051669 >> v2:172.21.15.89:3300/0 conn(0x7f46f80ccc80 0x7f46f806f920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:03:30.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:30.486+0000 7f46f6a00640 1 --2- 172.21.15.16:0/600051669 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f46f8145e40 0x7f46f80cc740 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f46f8073cf0 tx=0x7f46e801c260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5302b2325ff8b5d8 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:03:30.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:30.486+0000 7f46f6000640 1 -- 172.21.15.16:0/600051669 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46e80b0070 con 0x7f46f8145e40 2024-03-31T15:03:30.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:30.486+0000 7f46f6000640 1 -- 172.21.15.16:0/600051669 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f46e80ab600 con 0x7f46f8145e40 2024-03-31T15:03:30.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:30.486+0000 7f46f6000640 1 -- 172.21.15.16:0/600051669 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46e80ab9d0 con 0x7f46f8145e40 2024-03-31T15:03:30.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:30.486+0000 7f46fd600640 1 -- 172.21.15.16:0/600051669 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f46f8145e40 msgr2=0x7f46f80cc740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:30.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:30.486+0000 7f46fd600640 1 --2- 172.21.15.16:0/600051669 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f46f8145e40 0x7f46f80cc740 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f46f8073cf0 tx=0x7f46e801c260 comp rx=0 tx=0).stop 2024-03-31T15:03:30.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:30.486+0000 7f46fd600640 1 -- 172.21.15.16:0/600051669 shutdown_connections 2024-03-31T15:03:30.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:30.486+0000 7f46fd600640 1 --2- 172.21.15.16:0/600051669 >> v2:172.21.15.89:3300/0 conn(0x7f46f80ccc80 0x7f46f806f920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:30.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:30.486+0000 7f46fd600640 1 --2- 172.21.15.16:0/600051669 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f46f806fe60 0x7f46f815fdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:30.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:30.486+0000 7f46fd600640 1 --2- 172.21.15.16:0/600051669 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f46f8145e40 0x7f46f80cc740 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:30.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:30.486+0000 7f46fd600640 1 -- 172.21.15.16:0/600051669 >> 172.21.15.16:0/600051669 conn(0x7f46f8067330 msgr2=0x7f46f8067740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:03:30.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:30.486+0000 7f46fd600640 1 -- 172.21.15.16:0/600051669 shutdown_connections 2024-03-31T15:03:30.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:30.486+0000 7f46fd600640 1 -- 172.21.15.16:0/600051669 wait complete. 2024-03-31T15:03:30.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:30.486+0000 7f46fd600640 1 Processor -- start 2024-03-31T15:03:30.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:30.486+0000 7f46fd600640 1 -- start start 2024-03-31T15:03:30.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:30.490+0000 7f46fd600640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f46f806fe60 0x7f46f80b6630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:30.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:30.490+0000 7f46fd600640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f46f80ccc80 0x7f46f81377a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:30.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:30.490+0000 7f46fd600640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f46f8145e40 0x7f46f815d730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:30.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:30.490+0000 7f46fd600640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f46f80b5240 con 0x7f46f806fe60 2024-03-31T15:03:30.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:30.490+0000 7f46fd600640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f46f80b50c0 con 0x7f46f80ccc80 2024-03-31T15:03:30.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:30.490+0000 7f46fd600640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f46f80d2010 con 0x7f46f8145e40 2024-03-31T15:03:30.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:30.490+0000 7f46f6a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f46f806fe60 0x7f46f80b6630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:30.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:30.490+0000 7f46f6a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f46f806fe60 0x7f46f80b6630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59564/0 (socket says 172.21.15.16:59564) 2024-03-31T15:03:30.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:30.490+0000 7f46f6a00640 1 -- 172.21.15.16:0/2479872242 learned_addr learned my addr 172.21.15.16:0/2479872242 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:03:30.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:30.490+0000 7f46ee000640 1 --2- 172.21.15.16:0/2479872242 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f46f80ccc80 0x7f46f81377a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:30.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:30.490+0000 7f46f7400640 1 --2- 172.21.15.16:0/2479872242 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f46f8145e40 0x7f46f815d730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:30.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:30.490+0000 7f46f6a00640 1 -- 172.21.15.16:0/2479872242 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f46f8145e40 msgr2=0x7f46f815d730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:30.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:30.490+0000 7f46f6a00640 1 --2- 172.21.15.16:0/2479872242 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f46f8145e40 0x7f46f815d730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:30.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:30.490+0000 7f46f6a00640 1 -- 172.21.15.16:0/2479872242 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f46f80ccc80 msgr2=0x7f46f81377a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:30.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:30.490+0000 7f46f6a00640 1 --2- 172.21.15.16:0/2479872242 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f46f80ccc80 0x7f46f81377a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:30.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:30.490+0000 7f46f6a00640 1 -- 172.21.15.16:0/2479872242 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46f815c690 con 0x7f46f806fe60 2024-03-31T15:03:30.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:30.490+0000 7f46f6a00640 1 --2- 172.21.15.16:0/2479872242 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f46f806fe60 0x7f46f80b6630 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f46e80ae040 tx=0x7f46e801aaa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:03:30.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:30.490+0000 7f46ef400640 1 -- 172.21.15.16:0/2479872242 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46e80ba020 con 0x7f46f806fe60 2024-03-31T15:03:30.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:30.490+0000 7f46ef400640 1 -- 172.21.15.16:0/2479872242 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f46e80b6640 con 0x7f46f806fe60 2024-03-31T15:03:30.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:30.490+0000 7f46ef400640 1 -- 172.21.15.16:0/2479872242 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46e80b0040 con 0x7f46f806fe60 2024-03-31T15:03:30.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:30.490+0000 7f46fd600640 1 -- 172.21.15.16:0/2479872242 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f46f815c970 con 0x7f46f806fe60 2024-03-31T15:03:30.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:30.490+0000 7f46fd600640 1 -- 172.21.15.16:0/2479872242 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f46f815cf30 con 0x7f46f806fe60 2024-03-31T15:03:30.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:30.490+0000 7f46fd600640 1 -- 172.21.15.16:0/2479872242 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f46f8069950 con 0x7f46f806fe60 2024-03-31T15:03:30.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:30.490+0000 7f46ef400640 1 -- 172.21.15.16:0/2479872242 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f46e80bd020 con 0x7f46f806fe60 2024-03-31T15:03:30.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:30.494+0000 7f46ef400640 1 --2- 172.21.15.16:0/2479872242 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f46d4068380 0x7f46d406a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:30.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:30.494+0000 7f46ef400640 1 -- 172.21.15.16:0/2479872242 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 4857+0+0 (secure 0 0 0) 0x7f46e8120a90 con 0x7f46f806fe60 2024-03-31T15:03:30.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:30.494+0000 7f46ee000640 1 --2- 172.21.15.16:0/2479872242 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f46d4068380 0x7f46d406a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:30.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:30.494+0000 7f46ee000640 1 --2- 172.21.15.16:0/2479872242 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f46d4068380 0x7f46d406a840 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f46f8138160 tx=0x7f46e0007990 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:03:30.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:30.498+0000 7f46ef400640 1 -- 172.21.15.16:0/2479872242 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f46e80c4030 con 0x7f46f806fe60 2024-03-31T15:03:30.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:30.698+0000 7f46fd600640 1 -- 172.21.15.16:0/2479872242 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f46f80cf3c0 con 0x7f46f806fe60 2024-03-31T15:03:30.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:30.698+0000 7f46ef400640 1 -- 172.21.15.16:0/2479872242 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) ==== 93+0+1389 (secure 0 0 0) 0x7f46e8016110 con 0x7f46f806fe60 2024-03-31T15:03:30.694 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:03:30.694 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4278,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6834/153294668","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":153294668},{"type":"v1","addr":"172.21.15.16:6834","nonce":153294668}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-03-31T15:03:30.694 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 2 2024-03-31T15:03:30.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:30.698+0000 7f46fd600640 1 -- 172.21.15.16:0/2479872242 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f46d4068380 msgr2=0x7f46d406a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:30.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:30.698+0000 7f46fd600640 1 --2- 172.21.15.16:0/2479872242 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f46d4068380 0x7f46d406a840 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f46f8138160 tx=0x7f46e0007990 comp rx=0 tx=0).stop 2024-03-31T15:03:30.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:30.698+0000 7f46fd600640 1 -- 172.21.15.16:0/2479872242 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f46f806fe60 msgr2=0x7f46f80b6630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:30.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:30.698+0000 7f46fd600640 1 --2- 172.21.15.16:0/2479872242 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f46f806fe60 0x7f46f80b6630 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f46e80ae040 tx=0x7f46e801aaa0 comp rx=0 tx=0).stop 2024-03-31T15:03:30.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:30.698+0000 7f46fd600640 1 -- 172.21.15.16:0/2479872242 shutdown_connections 2024-03-31T15:03:30.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:30.698+0000 7f46fd600640 1 --2- 172.21.15.16:0/2479872242 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f46d4068380 0x7f46d406a840 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:30.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:30.698+0000 7f46fd600640 1 --2- 172.21.15.16:0/2479872242 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f46f8145e40 0x7f46f815d730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:30.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:30.698+0000 7f46fd600640 1 --2- 172.21.15.16:0/2479872242 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f46f80ccc80 0x7f46f81377a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:30.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:30.698+0000 7f46fd600640 1 --2- 172.21.15.16:0/2479872242 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f46f806fe60 0x7f46f80b6630 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:30.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:30.698+0000 7f46fd600640 1 -- 172.21.15.16:0/2479872242 >> 172.21.15.16:0/2479872242 conn(0x7f46f8067330 msgr2=0x7f46f80b1c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:03:30.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:30.698+0000 7f46fd600640 1 -- 172.21.15.16:0/2479872242 shutdown_connections 2024-03-31T15:03:30.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:30.698+0000 7f46fd600640 1 -- 172.21.15.16:0/2479872242 wait complete. 2024-03-31T15:03:32.715 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-31T15:03:32.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:32.869+0000 7f904ee00640 1 Processor -- start 2024-03-31T15:03:32.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:32.869+0000 7f904ee00640 1 -- start start 2024-03-31T15:03:32.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:32.873+0000 7f904ee00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f90480ce210 0x7f904806f5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:32.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:32.873+0000 7f904ee00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f904806fb10 0x7f904806ffc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:32.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:32.873+0000 7f904ee00640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f90480cd170 con 0x7f90480b1280 2024-03-31T15:03:32.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:32.873+0000 7f904ee00640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f9048070500 con 0x7f904806fb10 2024-03-31T15:03:32.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:32.873+0000 7f904ee00640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f90480cdd80 con 0x7f90480ce210 2024-03-31T15:03:32.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:32.873+0000 7f904e400640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f904806fb10 0x7f904806ffc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:32.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:32.873+0000 7f904e400640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f904806fb10 0x7f904806ffc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59578/0 (socket says 172.21.15.16:59578) 2024-03-31T15:03:32.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:32.873+0000 7f904da00640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f90480b1280 0x7f90480cc900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:40632/0 (socket says 172.21.15.16:40632) 2024-03-31T15:03:32.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:32.873+0000 7f904e400640 1 -- 172.21.15.16:0/121477136 learned_addr learned my addr 172.21.15.16:0/121477136 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:03:32.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:32.873+0000 7f904d000640 1 --2- 172.21.15.16:0/121477136 >> v2:172.21.15.89:3300/0 conn(0x7f90480ce210 0x7f904806f5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:32.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:32.873+0000 7f904e400640 1 -- 172.21.15.16:0/121477136 >> v2:172.21.15.89:3300/0 conn(0x7f90480ce210 msgr2=0x7f904806f5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:03:32.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:32.873+0000 7f904e400640 1 --2- 172.21.15.16:0/121477136 >> v2:172.21.15.89:3300/0 conn(0x7f90480ce210 0x7f904806f5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:32.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:32.873+0000 7f904e400640 1 -- 172.21.15.16:0/121477136 >> v1:172.21.15.16:6789/0 conn(0x7f90480b1280 legacy=0x7f90480cc900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:32.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:32.873+0000 7f904e400640 1 -- 172.21.15.16:0/121477136 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f90480cd450 con 0x7f904806fb10 2024-03-31T15:03:32.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:32.873+0000 7f9047e00640 1 -- 172.21.15.16:0/121477136 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4026006534 0 0) 0x7f90480cd170 con 0x7f90480b1280 2024-03-31T15:03:32.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:32.877+0000 7f904e400640 1 --2- 172.21.15.16:0/121477136 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f904806fb10 0x7f904806ffc0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f9048072ae0 tx=0x7f904001c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5e63c83458844d2 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:03:32.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:32.877+0000 7f9047e00640 1 -- 172.21.15.16:0/121477136 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90400a57a0 con 0x7f904806fb10 2024-03-31T15:03:32.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:32.877+0000 7f9047e00640 1 -- 172.21.15.16:0/121477136 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f90400a5940 con 0x7f904806fb10 2024-03-31T15:03:32.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:32.877+0000 7f9047e00640 1 -- 172.21.15.16:0/121477136 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90400a5d10 con 0x7f904806fb10 2024-03-31T15:03:32.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:32.877+0000 7f904ee00640 1 -- 172.21.15.16:0/121477136 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f904806fb10 msgr2=0x7f904806ffc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:32.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:32.877+0000 7f904ee00640 1 --2- 172.21.15.16:0/121477136 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f904806fb10 0x7f904806ffc0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f9048072ae0 tx=0x7f904001c9f0 comp rx=0 tx=0).stop 2024-03-31T15:03:32.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:32.877+0000 7f904ee00640 1 -- 172.21.15.16:0/121477136 shutdown_connections 2024-03-31T15:03:32.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:32.877+0000 7f904ee00640 1 --2- 172.21.15.16:0/121477136 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f904806fb10 0x7f904806ffc0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:32.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:32.877+0000 7f904ee00640 1 --2- 172.21.15.16:0/121477136 >> v2:172.21.15.89:3300/0 conn(0x7f90480ce210 0x7f904806f5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:32.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:32.877+0000 7f904ee00640 1 -- 172.21.15.16:0/121477136 >> 172.21.15.16:0/121477136 conn(0x7f90480b39c0 msgr2=0x7f90480b3dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:03:32.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:32.877+0000 7f904ee00640 1 -- 172.21.15.16:0/121477136 shutdown_connections 2024-03-31T15:03:32.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:32.877+0000 7f904ee00640 1 -- 172.21.15.16:0/121477136 wait complete. 2024-03-31T15:03:32.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:32.877+0000 7f904ee00640 1 Processor -- start 2024-03-31T15:03:32.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:32.877+0000 7f904ee00640 1 -- start start 2024-03-31T15:03:32.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:32.881+0000 7f904ee00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f904806fb10 0x7f904815b8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:32.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:32.881+0000 7f904ee00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f90480b1280 0x7f90480cb940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:32.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:32.881+0000 7f904ee00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f90480ce210 0x7f90481f8430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:32.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:32.881+0000 7f904ee00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f90481608b0 con 0x7f90480b1280 2024-03-31T15:03:32.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:32.881+0000 7f904ee00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f9048160730 con 0x7f90480ce210 2024-03-31T15:03:32.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:32.881+0000 7f904ee00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f9048160a30 con 0x7f904806fb10 2024-03-31T15:03:32.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:32.881+0000 7f904d000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f90480b1280 0x7f90480cb940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:32.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:32.881+0000 7f904d000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f90480b1280 0x7f90480cb940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59590/0 (socket says 172.21.15.16:59590) 2024-03-31T15:03:32.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:32.881+0000 7f904d000640 1 -- 172.21.15.16:0/2094653892 learned_addr learned my addr 172.21.15.16:0/2094653892 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:03:32.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:32.881+0000 7f904e400640 1 --2- 172.21.15.16:0/2094653892 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f90480ce210 0x7f90481f8430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:32.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:32.881+0000 7f904da00640 1 --2- 172.21.15.16:0/2094653892 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f904806fb10 0x7f904815b8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:32.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:32.881+0000 7f904d000640 1 -- 172.21.15.16:0/2094653892 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f904806fb10 msgr2=0x7f904815b8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:32.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:32.881+0000 7f904d000640 1 --2- 172.21.15.16:0/2094653892 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f904806fb10 0x7f904815b8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:32.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:32.881+0000 7f904d000640 1 -- 172.21.15.16:0/2094653892 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f90480ce210 msgr2=0x7f90481f8430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:32.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:32.881+0000 7f904d000640 1 --2- 172.21.15.16:0/2094653892 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f90480ce210 0x7f90481f8430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:32.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:32.881+0000 7f904d000640 1 -- 172.21.15.16:0/2094653892 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f90480b6c10 con 0x7f90480b1280 2024-03-31T15:03:32.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:32.881+0000 7f904d000640 1 --2- 172.21.15.16:0/2094653892 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f90480b1280 0x7f90480cb940 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f903400d620 tx=0x7f903400dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:03:32.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:32.881+0000 7f9045600640 1 -- 172.21.15.16:0/2094653892 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f903400b620 con 0x7f90480b1280 2024-03-31T15:03:32.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:32.881+0000 7f9045600640 1 -- 172.21.15.16:0/2094653892 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9034019070 con 0x7f90480b1280 2024-03-31T15:03:32.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:32.881+0000 7f9045600640 1 -- 172.21.15.16:0/2094653892 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f903400b9e0 con 0x7f90480b1280 2024-03-31T15:03:32.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:32.881+0000 7f904ee00640 1 -- 172.21.15.16:0/2094653892 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f90480b6ef0 con 0x7f90480b1280 2024-03-31T15:03:32.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:32.881+0000 7f904ee00640 1 -- 172.21.15.16:0/2094653892 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f90480b74b0 con 0x7f90480b1280 2024-03-31T15:03:32.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:32.881+0000 7f904ee00640 1 -- 172.21.15.16:0/2094653892 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f904804f8c0 con 0x7f90480b1280 2024-03-31T15:03:32.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:32.881+0000 7f9045600640 1 -- 172.21.15.16:0/2094653892 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f903400bb80 con 0x7f90480b1280 2024-03-31T15:03:32.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:32.885+0000 7f9045600640 1 --2- 172.21.15.16:0/2094653892 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f9028068380 0x7f902806a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:32.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:32.885+0000 7f9045600640 1 -- 172.21.15.16:0/2094653892 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 4857+0+0 (secure 0 0 0) 0x7f9034086b20 con 0x7f90480b1280 2024-03-31T15:03:32.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:32.885+0000 7f904da00640 1 --2- 172.21.15.16:0/2094653892 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f9028068380 0x7f902806a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:32.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:32.885+0000 7f904da00640 1 --2- 172.21.15.16:0/2094653892 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f9028068380 0x7f902806a840 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f903c000f70 tx=0x7f903c007b60 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:03:32.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:32.889+0000 7f9045600640 1 -- 172.21.15.16:0/2094653892 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f9034057420 con 0x7f90480b1280 2024-03-31T15:03:33.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:33.089+0000 7f904ee00640 1 -- 172.21.15.16:0/2094653892 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f90480b0e00 con 0x7f90480b1280 2024-03-31T15:03:33.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:33.089+0000 7f9045600640 1 -- 172.21.15.16:0/2094653892 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) ==== 93+0+3861 (secure 0 0 0) 0x7f9034056d20 con 0x7f90480b1280 2024-03-31T15:03:33.086 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:03:33.086 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4276,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/2832752118","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6833","nonce":2832752118},{"type":"v1","addr":"172.21.15.16:6835","nonce":2832752118}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4278,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6834/153294668","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":153294668},{"type":"v1","addr":"172.21.15.16:6834","nonce":153294668}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4288,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/1653319357","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":1653319357},{"type":"v1","addr":"172.21.15.89:6835","nonce":1653319357}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4293,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/2583453131","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":2583453131},{"type":"v1","addr":"172.21.15.89:6837","nonce":2583453131}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-03-31T15:03:33.086 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 3 2024-03-31T15:03:33.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:33.089+0000 7f904ee00640 1 -- 172.21.15.16:0/2094653892 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f9028068380 msgr2=0x7f902806a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:33.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:33.089+0000 7f904ee00640 1 --2- 172.21.15.16:0/2094653892 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f9028068380 0x7f902806a840 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f903c000f70 tx=0x7f903c007b60 comp rx=0 tx=0).stop 2024-03-31T15:03:33.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:33.089+0000 7f904ee00640 1 -- 172.21.15.16:0/2094653892 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f90480b1280 msgr2=0x7f90480cb940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:33.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:33.089+0000 7f904ee00640 1 --2- 172.21.15.16:0/2094653892 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f90480b1280 0x7f90480cb940 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f903400d620 tx=0x7f903400dae0 comp rx=0 tx=0).stop 2024-03-31T15:03:33.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:33.089+0000 7f904ee00640 1 -- 172.21.15.16:0/2094653892 shutdown_connections 2024-03-31T15:03:33.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:33.089+0000 7f904ee00640 1 --2- 172.21.15.16:0/2094653892 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f9028068380 0x7f902806a840 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:33.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:33.089+0000 7f904ee00640 1 --2- 172.21.15.16:0/2094653892 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f90480ce210 0x7f90481f8430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:33.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:33.089+0000 7f904ee00640 1 --2- 172.21.15.16:0/2094653892 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f90480b1280 0x7f90480cb940 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:33.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:33.089+0000 7f904ee00640 1 --2- 172.21.15.16:0/2094653892 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f904806fb10 0x7f904815b8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:33.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:33.089+0000 7f904ee00640 1 -- 172.21.15.16:0/2094653892 >> 172.21.15.16:0/2094653892 conn(0x7f90480b39c0 msgr2=0x7f904815f310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:03:33.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:33.089+0000 7f904ee00640 1 -- 172.21.15.16:0/2094653892 shutdown_connections 2024-03-31T15:03:33.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:33.089+0000 7f904ee00640 1 -- 172.21.15.16:0/2094653892 wait complete. 2024-03-31T15:03:33.108 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-31T15:03:33.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:33.249+0000 7eff6bc00640 1 Processor -- start 2024-03-31T15:03:33.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:33.249+0000 7eff6bc00640 1 -- start start 2024-03-31T15:03:33.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:33.253+0000 7eff6bc00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7eff6415b3d0 0x7eff640b1cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:33.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:33.253+0000 7eff6bc00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7eff6415eee0 0x7eff640b2200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:33.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:33.253+0000 7eff6bc00640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7eff6415f3c0 con 0x7eff640ce2f0 2024-03-31T15:03:33.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:33.253+0000 7eff6bc00640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7eff6415b7b0 con 0x7eff6415b3d0 2024-03-31T15:03:33.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:33.253+0000 7eff6bc00640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7eff6415f6a0 con 0x7eff6415eee0 2024-03-31T15:03:33.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:33.253+0000 7eff69200640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7eff6415b3d0 0x7eff640b1cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:33.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:33.253+0000 7eff69200640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7eff6415b3d0 0x7eff640b1cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59606/0 (socket says 172.21.15.16:59606) 2024-03-31T15:03:33.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:33.253+0000 7eff69200640 1 -- 172.21.15.16:0/2010685470 learned_addr learned my addr 172.21.15.16:0/2010685470 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:03:33.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:33.253+0000 7eff5be00640 1 --2- 172.21.15.16:0/2010685470 >> v2:172.21.15.89:3300/0 conn(0x7eff6415eee0 0x7eff640b2200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:33.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:33.253+0000 7eff69200640 1 -- 172.21.15.16:0/2010685470 >> v2:172.21.15.89:3300/0 conn(0x7eff6415eee0 msgr2=0x7eff640b2200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:03:33.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:33.253+0000 7eff69200640 1 --2- 172.21.15.16:0/2010685470 >> v2:172.21.15.89:3300/0 conn(0x7eff6415eee0 0x7eff640b2200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:33.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:33.253+0000 7eff69200640 1 -- 172.21.15.16:0/2010685470 >> v1:172.21.15.16:6789/0 conn(0x7eff640ce2f0 legacy=0x7eff640cc9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:33.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:33.253+0000 7eff69200640 1 -- 172.21.15.16:0/2010685470 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff6415f980 con 0x7eff6415b3d0 2024-03-31T15:03:33.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:33.257+0000 7eff69200640 1 --2- 172.21.15.16:0/2010685470 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7eff6415b3d0 0x7eff640b1cc0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7eff640cdae0 tx=0x7eff5401c4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b31fe2db186c9e9 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:03:33.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:33.257+0000 7eff63e00640 1 -- 172.21.15.16:0/2010685470 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff540aa610 con 0x7eff6415b3d0 2024-03-31T15:03:33.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:33.257+0000 7eff63e00640 1 -- 172.21.15.16:0/2010685470 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7eff540af070 con 0x7eff6415b3d0 2024-03-31T15:03:33.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:33.257+0000 7eff63e00640 1 -- 172.21.15.16:0/2010685470 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff540aa9d0 con 0x7eff6415b3d0 2024-03-31T15:03:33.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:33.257+0000 7eff6bc00640 1 -- 172.21.15.16:0/2010685470 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7eff6415b3d0 msgr2=0x7eff640b1cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:33.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:33.257+0000 7eff6bc00640 1 --2- 172.21.15.16:0/2010685470 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7eff6415b3d0 0x7eff640b1cc0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7eff640cdae0 tx=0x7eff5401c4f0 comp rx=0 tx=0).stop 2024-03-31T15:03:33.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:33.257+0000 7eff6bc00640 1 -- 172.21.15.16:0/2010685470 shutdown_connections 2024-03-31T15:03:33.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:33.257+0000 7eff6bc00640 1 --2- 172.21.15.16:0/2010685470 >> v2:172.21.15.89:3300/0 conn(0x7eff6415eee0 0x7eff640b2200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:33.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:33.257+0000 7eff6bc00640 1 --2- 172.21.15.16:0/2010685470 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7eff6415b3d0 0x7eff640b1cc0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:33.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:33.257+0000 7eff6bc00640 1 -- 172.21.15.16:0/2010685470 >> 172.21.15.16:0/2010685470 conn(0x7eff6406b620 msgr2=0x7eff6406ba30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:03:33.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:33.257+0000 7eff6bc00640 1 -- 172.21.15.16:0/2010685470 shutdown_connections 2024-03-31T15:03:33.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:33.257+0000 7eff6bc00640 1 -- 172.21.15.16:0/2010685470 wait complete. 2024-03-31T15:03:33.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:33.257+0000 7eff6bc00640 1 Processor -- start 2024-03-31T15:03:33.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:33.257+0000 7eff6bc00640 1 -- start start 2024-03-31T15:03:33.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:33.257+0000 7eff6bc00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7eff640ce2f0 0x7eff641523d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:33.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:33.257+0000 7eff6bc00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7eff6415b3d0 0x7eff6415b930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:33.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:33.257+0000 7eff6bc00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7eff6415eee0 0x7eff6414d2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:33.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:33.257+0000 7eff6bc00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7eff64166600 con 0x7eff6415eee0 2024-03-31T15:03:33.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:33.257+0000 7eff6bc00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7eff64166480 con 0x7eff640ce2f0 2024-03-31T15:03:33.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:33.257+0000 7eff6bc00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7eff64166780 con 0x7eff6415b3d0 2024-03-31T15:03:33.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:33.261+0000 7eff69c00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7eff6415eee0 0x7eff6414d2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:33.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:33.261+0000 7eff69c00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7eff6415eee0 0x7eff6414d2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59608/0 (socket says 172.21.15.16:59608) 2024-03-31T15:03:33.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:33.261+0000 7eff69c00640 1 -- 172.21.15.16:0/3991291384 learned_addr learned my addr 172.21.15.16:0/3991291384 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:03:33.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:33.261+0000 7eff5be00640 1 --2- 172.21.15.16:0/3991291384 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7eff6415b3d0 0x7eff6415b930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:33.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:33.261+0000 7eff69200640 1 --2- 172.21.15.16:0/3991291384 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7eff640ce2f0 0x7eff641523d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:33.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:33.261+0000 7eff69c00640 1 -- 172.21.15.16:0/3991291384 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7eff6415b3d0 msgr2=0x7eff6415b930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:33.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:33.261+0000 7eff69c00640 1 --2- 172.21.15.16:0/3991291384 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7eff6415b3d0 0x7eff6415b930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:33.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:33.261+0000 7eff69c00640 1 -- 172.21.15.16:0/3991291384 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7eff640ce2f0 msgr2=0x7eff641523d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:33.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:33.261+0000 7eff69c00640 1 --2- 172.21.15.16:0/3991291384 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7eff640ce2f0 0x7eff641523d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:33.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:33.261+0000 7eff69c00640 1 -- 172.21.15.16:0/3991291384 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff64168820 con 0x7eff6415eee0 2024-03-31T15:03:33.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:33.261+0000 7eff5be00640 1 --2- 172.21.15.16:0/3991291384 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7eff6415b3d0 0x7eff6415b930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:03:33.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:33.261+0000 7eff69c00640 1 --2- 172.21.15.16:0/3991291384 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7eff6415eee0 0x7eff6414d2d0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7eff5c00c770 tx=0x7eff5c00cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:03:33.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:33.261+0000 7eff61600640 1 -- 172.21.15.16:0/3991291384 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff5c003b80 con 0x7eff6415eee0 2024-03-31T15:03:33.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:33.261+0000 7eff61600640 1 -- 172.21.15.16:0/3991291384 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7eff5c003d20 con 0x7eff6415eee0 2024-03-31T15:03:33.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:33.261+0000 7eff6bc00640 1 -- 172.21.15.16:0/3991291384 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7eff64168b00 con 0x7eff6415eee0 2024-03-31T15:03:33.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:33.261+0000 7eff61600640 1 -- 172.21.15.16:0/3991291384 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff5c00aa90 con 0x7eff6415eee0 2024-03-31T15:03:33.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:33.261+0000 7eff6bc00640 1 -- 172.21.15.16:0/3991291384 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7eff64150760 con 0x7eff6415eee0 2024-03-31T15:03:33.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:33.261+0000 7eff6bc00640 1 -- 172.21.15.16:0/3991291384 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7eff64069950 con 0x7eff6415eee0 2024-03-31T15:03:33.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:33.261+0000 7eff61600640 1 -- 172.21.15.16:0/3991291384 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7eff5c00ac30 con 0x7eff6415eee0 2024-03-31T15:03:33.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:33.261+0000 7eff61600640 1 --2- 172.21.15.16:0/3991291384 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7eff400683b0 0x7eff4006a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:33.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:33.265+0000 7eff61600640 1 -- 172.21.15.16:0/3991291384 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 4857+0+0 (secure 0 0 0) 0x7eff5c0864b0 con 0x7eff6415eee0 2024-03-31T15:03:33.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:33.265+0000 7eff69200640 1 --2- 172.21.15.16:0/3991291384 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7eff400683b0 0x7eff4006a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:33.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:33.265+0000 7eff69200640 1 --2- 172.21.15.16:0/3991291384 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7eff400683b0 0x7eff4006a870 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7eff540ae000 tx=0x7eff540a7000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:03:33.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:33.269+0000 7eff61600640 1 -- 172.21.15.16:0/3991291384 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7eff5c056db0 con 0x7eff6415eee0 2024-03-31T15:03:33.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:33.433+0000 7eff6bc00640 1 -- 172.21.15.16:0/3991291384 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7eff64169270 con 0x7eff6415eee0 2024-03-31T15:03:33.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:33.437+0000 7eff61600640 1 -- 172.21.15.16:0/3991291384 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v17) ==== 74+0+12812 (secure 0 0 0) 0x7eff5c0566b0 con 0x7eff6415eee0 2024-03-31T15:03:33.433 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:03:33.433 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":17,"fsid":"286eb5da-fb47-4550-b290-d9519c13b9a5","created":"2024-03-31T15:03:11.438952+0000","modified":"2024-03-31T15:03:24.897591+0000","last_up_change":"2024-03-31T15:03:22.885535+0000","last_in_change":"2024-03-31T15:03:13.941667+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-31T15:03:22.550787+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"92a66450-ef19-40d8-ac3d-d760b23b0bda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6801","nonce":932712244}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6803","nonce":932712244}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6807","nonce":932712244}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6805","nonce":932712244}]},"public_addr":"172.21.15.16:6801/932712244","cluster_addr":"172.21.15.16:6803/932712244","heartbeat_back_addr":"172.21.15.16:6807/932712244","heartbeat_front_addr":"172.21.15.16:6805/932712244","state":["exists","up"]},{"osd":1,"uuid":"da9d1f2b-1968-4959-b843-faa231988e92","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6810","nonce":1398737533}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6811","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6813","nonce":1398737533}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6821","nonce":1398737533}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6815","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6816","nonce":1398737533}]},"public_addr":"172.21.15.16:6810/1398737533","cluster_addr":"172.21.15.16:6813/1398737533","heartbeat_back_addr":"172.21.15.16:6821/1398737533","heartbeat_front_addr":"172.21.15.16:6816/1398737533","state":["exists","up"]},{"osd":2,"uuid":"01a73662-9a21-4b1f-bcbb-42b9636a4638","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6819","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6822","nonce":3018526646}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6825","nonce":3018526646}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6831","nonce":3018526646}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6827","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6829","nonce":3018526646}]},"public_addr":"172.21.15.16:6822/3018526646","cluster_addr":"172.21.15.16:6825/3018526646","heartbeat_back_addr":"172.21.15.16:6831/3018526646","heartbeat_front_addr":"172.21.15.16:6829/3018526646","state":["exists","up"]},{"osd":3,"uuid":"c7771fd3-e18e-4500-ac38-23cf10ebcd75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6809","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6812","nonce":3360352974}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6817","nonce":3360352974}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6828","nonce":3360352974}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6823","nonce":3360352974}]},"public_addr":"172.21.15.16:6812/3360352974","cluster_addr":"172.21.15.16:6817/3360352974","heartbeat_back_addr":"172.21.15.16:6828/3360352974","heartbeat_front_addr":"172.21.15.16:6823/3360352974","state":["exists","up"]},{"osd":4,"uuid":"88260730-1482-4fbd-a40f-cb1124e9df4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6801","nonce":1856844856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6803","nonce":1856844856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6807","nonce":1856844856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6805","nonce":1856844856}]},"public_addr":"172.21.15.89:6801/1856844856","cluster_addr":"172.21.15.89:6803/1856844856","heartbeat_back_addr":"172.21.15.89:6807/1856844856","heartbeat_front_addr":"172.21.15.89:6805/1856844856","state":["exists","up"]},{"osd":5,"uuid":"4cb2c00c-fae0-4dd8-9d27-ec4f0093ccf7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6814","nonce":50395527}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6817","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6820","nonce":50395527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6831","nonce":50395527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6823","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6826","nonce":50395527}]},"public_addr":"172.21.15.89:6814/50395527","cluster_addr":"172.21.15.89:6820/50395527","heartbeat_back_addr":"172.21.15.89:6831/50395527","heartbeat_front_addr":"172.21.15.89:6826/50395527","state":["exists","up"]},{"osd":6,"uuid":"81ce0c19-2554-4cae-acbe-66a05e9e9b9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6809","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6812","nonce":3470471752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6815","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6818","nonce":3470471752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6827","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6829","nonce":3470471752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6821","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6824","nonce":3470471752}]},"public_addr":"172.21.15.89:6812/3470471752","cluster_addr":"172.21.15.89:6818/3470471752","heartbeat_back_addr":"172.21.15.89:6829/3470471752","heartbeat_front_addr":"172.21.15.89:6824/3470471752","state":["exists","up"]},{"osd":7,"uuid":"85e4313b-d37d-442e-8042-153bf20a847d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6811","nonce":1796778630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6813","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6816","nonce":1796778630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6825","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6828","nonce":1796778630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6819","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6822","nonce":1796778630}]},"public_addr":"172.21.15.89:6811/1796778630","cluster_addr":"172.21.15.89:6816/1796778630","heartbeat_back_addr":"172.21.15.89:6828/1796778630","heartbeat_front_addr":"172.21.15.89:6822/1796778630","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.439088+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.529970+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.680512+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:21.044161+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.582111+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.536558+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.409522+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.590229+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-31T15:03:33.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:33.437+0000 7eff6bc00640 1 -- 172.21.15.16:0/3991291384 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7eff400683b0 msgr2=0x7eff4006a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:33.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:33.437+0000 7eff6bc00640 1 --2- 172.21.15.16:0/3991291384 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7eff400683b0 0x7eff4006a870 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7eff540ae000 tx=0x7eff540a7000 comp rx=0 tx=0).stop 2024-03-31T15:03:33.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:33.437+0000 7eff6bc00640 1 -- 172.21.15.16:0/3991291384 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7eff6415eee0 msgr2=0x7eff6414d2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:33.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:33.437+0000 7eff6bc00640 1 --2- 172.21.15.16:0/3991291384 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7eff6415eee0 0x7eff6414d2d0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7eff5c00c770 tx=0x7eff5c00cc30 comp rx=0 tx=0).stop 2024-03-31T15:03:33.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:33.437+0000 7eff6bc00640 1 -- 172.21.15.16:0/3991291384 shutdown_connections 2024-03-31T15:03:33.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:33.437+0000 7eff6bc00640 1 --2- 172.21.15.16:0/3991291384 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7eff400683b0 0x7eff4006a870 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:33.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:33.437+0000 7eff6bc00640 1 --2- 172.21.15.16:0/3991291384 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7eff6415eee0 0x7eff6414d2d0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:33.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:33.437+0000 7eff6bc00640 1 --2- 172.21.15.16:0/3991291384 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7eff6415b3d0 0x7eff6415b930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:33.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:33.437+0000 7eff6bc00640 1 --2- 172.21.15.16:0/3991291384 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7eff640ce2f0 0x7eff641523d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:33.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:33.437+0000 7eff6bc00640 1 -- 172.21.15.16:0/3991291384 >> 172.21.15.16:0/3991291384 conn(0x7eff6406b620 msgr2=0x7eff640ce840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:03:33.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:33.437+0000 7eff6bc00640 1 -- 172.21.15.16:0/3991291384 shutdown_connections 2024-03-31T15:03:33.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:33.437+0000 7eff6bc00640 1 -- 172.21.15.16:0/3991291384 wait complete. 2024-03-31T15:03:33.454 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-31T15:03:22.550787+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-31T15:03:33.454 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-31T15:03:33.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:33.601+0000 7f4afe000640 1 Processor -- start 2024-03-31T15:03:33.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:33.601+0000 7f4afe000640 1 -- start start 2024-03-31T15:03:33.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:33.601+0000 7f4afe000640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f4af81411b0 0x7f4af806f920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:33.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:33.601+0000 7f4afe000640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f4af8141590 0x7f4af806fe60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:33.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:33.601+0000 7f4afe000640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4af8144980 con 0x7f4af8143c30 2024-03-31T15:03:33.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:33.601+0000 7f4afe000640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f4af815f910 con 0x7f4af81411b0 2024-03-31T15:03:33.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:33.601+0000 7f4afe000640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f4af8162910 con 0x7f4af8141590 2024-03-31T15:03:33.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:33.601+0000 7f4af7400640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f4af81411b0 0x7f4af806f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:33.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:33.601+0000 7f4af7400640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f4af81411b0 0x7f4af806f920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59618/0 (socket says 172.21.15.16:59618) 2024-03-31T15:03:33.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:33.601+0000 7f4af7400640 1 -- 172.21.15.16:0/3107259121 learned_addr learned my addr 172.21.15.16:0/3107259121 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:03:33.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:33.601+0000 7f4af7e00640 1 --1- 172.21.15.16:0/3107259121 >> v1:172.21.15.16:6789/0 conn(0x7f4af8143c30 0x7f4af81440a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:40642/0 (socket says 172.21.15.16:40642) 2024-03-31T15:03:33.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:33.601+0000 7f4aeea00640 1 --2- 172.21.15.16:0/3107259121 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f4af8141590 0x7f4af806fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:33.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:33.601+0000 7f4af7400640 1 -- 172.21.15.16:0/3107259121 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f4af8141590 msgr2=0x7f4af806fe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:03:33.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:33.601+0000 7f4af7400640 1 --2- 172.21.15.16:0/3107259121 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f4af8141590 0x7f4af806fe60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:33.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:33.601+0000 7f4af7400640 1 -- 172.21.15.16:0/3107259121 >> v1:172.21.15.16:6789/0 conn(0x7f4af8143c30 legacy=0x7f4af81440a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:33.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:33.601+0000 7f4af7400640 1 -- 172.21.15.16:0/3107259121 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4af8161fe0 con 0x7f4af81411b0 2024-03-31T15:03:33.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:33.601+0000 7f4aeea00640 1 --2- 172.21.15.16:0/3107259121 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f4af8141590 0x7f4af806fe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:03:33.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:33.605+0000 7f4af7400640 1 --2- 172.21.15.16:0/3107259121 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4af81411b0 0x7f4af806f920 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f4ae8009e50 tx=0x7f4ae801ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb54401543295aa7 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:03:33.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:33.605+0000 7f4af6a00640 1 -- 172.21.15.16:0/3107259121 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ae80a57f0 con 0x7f4af81411b0 2024-03-31T15:03:33.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:33.605+0000 7f4af6a00640 1 -- 172.21.15.16:0/3107259121 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4ae80a5990 con 0x7f4af81411b0 2024-03-31T15:03:33.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:33.605+0000 7f4af6a00640 1 -- 172.21.15.16:0/3107259121 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ae80a5d60 con 0x7f4af81411b0 2024-03-31T15:03:33.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:33.605+0000 7f4afe000640 1 -- 172.21.15.16:0/3107259121 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4af81411b0 msgr2=0x7f4af806f920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:33.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:33.605+0000 7f4afe000640 1 --2- 172.21.15.16:0/3107259121 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4af81411b0 0x7f4af806f920 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f4ae8009e50 tx=0x7f4ae801ca30 comp rx=0 tx=0).stop 2024-03-31T15:03:33.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:33.605+0000 7f4afe000640 1 -- 172.21.15.16:0/3107259121 shutdown_connections 2024-03-31T15:03:33.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:33.605+0000 7f4afe000640 1 --2- 172.21.15.16:0/3107259121 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f4af8141590 0x7f4af806fe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:33.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:33.605+0000 7f4afe000640 1 --2- 172.21.15.16:0/3107259121 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4af81411b0 0x7f4af806f920 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:33.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:33.605+0000 7f4afe000640 1 -- 172.21.15.16:0/3107259121 >> 172.21.15.16:0/3107259121 conn(0x7f4af8067930 msgr2=0x7f4af80b3ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:03:33.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:33.605+0000 7f4afe000640 1 -- 172.21.15.16:0/3107259121 shutdown_connections 2024-03-31T15:03:33.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:33.605+0000 7f4afe000640 1 -- 172.21.15.16:0/3107259121 wait complete. 2024-03-31T15:03:33.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:33.609+0000 7f4afe000640 1 Processor -- start 2024-03-31T15:03:33.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:33.609+0000 7f4afe000640 1 -- start start 2024-03-31T15:03:33.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:33.609+0000 7f4afe000640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4af81411b0 0x7f4af813dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:33.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:33.609+0000 7f4afe000640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f4af8141590 0x7f4af8156370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:33.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:33.609+0000 7f4afe000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4af8143c30 0x7f4af80b4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:33.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:33.609+0000 7f4afe000640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f4af815f290 con 0x7f4af8143c30 2024-03-31T15:03:33.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:33.609+0000 7f4afe000640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f4af815f110 con 0x7f4af81411b0 2024-03-31T15:03:33.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:33.609+0000 7f4afe000640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f4af815f410 con 0x7f4af8141590 2024-03-31T15:03:33.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:33.609+0000 7f4af7e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4af8143c30 0x7f4af80b4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:33.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:33.609+0000 7f4af7e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4af8143c30 0x7f4af80b4d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59630/0 (socket says 172.21.15.16:59630) 2024-03-31T15:03:33.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:33.609+0000 7f4af7e00640 1 -- 172.21.15.16:0/4198335241 learned_addr learned my addr 172.21.15.16:0/4198335241 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:03:33.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:33.609+0000 7f4af7400640 1 --2- 172.21.15.16:0/4198335241 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4af81411b0 0x7f4af813dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:33.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:33.609+0000 7f4aeea00640 1 --2- 172.21.15.16:0/4198335241 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f4af8141590 0x7f4af8156370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:33.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:33.609+0000 7f4af7e00640 1 -- 172.21.15.16:0/4198335241 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f4af8141590 msgr2=0x7f4af8156370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:33.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:33.609+0000 7f4af7e00640 1 --2- 172.21.15.16:0/4198335241 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f4af8141590 0x7f4af8156370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:33.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:33.609+0000 7f4af7e00640 1 -- 172.21.15.16:0/4198335241 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4af81411b0 msgr2=0x7f4af813dd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:33.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:33.609+0000 7f4af7e00640 1 --2- 172.21.15.16:0/4198335241 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4af81411b0 0x7f4af813dd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:33.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:33.609+0000 7f4af7e00640 1 -- 172.21.15.16:0/4198335241 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4af813d7e0 con 0x7f4af8143c30 2024-03-31T15:03:33.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:33.609+0000 7f4af7e00640 1 --2- 172.21.15.16:0/4198335241 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4af8143c30 0x7f4af80b4d90 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f4af000c3c0 tx=0x7f4af000c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:03:33.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:33.609+0000 7f4aefe00640 1 -- 172.21.15.16:0/4198335241 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4af00074e0 con 0x7f4af8143c30 2024-03-31T15:03:33.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:33.609+0000 7f4aefe00640 1 -- 172.21.15.16:0/4198335241 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4af0007680 con 0x7f4af8143c30 2024-03-31T15:03:33.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:33.609+0000 7f4afe000640 1 -- 172.21.15.16:0/4198335241 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4af8158090 con 0x7f4af8143c30 2024-03-31T15:03:33.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:33.609+0000 7f4aefe00640 1 -- 172.21.15.16:0/4198335241 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4af0004a10 con 0x7f4af8143c30 2024-03-31T15:03:33.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:33.609+0000 7f4afe000640 1 -- 172.21.15.16:0/4198335241 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4af813e240 con 0x7f4af8143c30 2024-03-31T15:03:33.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:33.609+0000 7f4afe000640 1 -- 172.21.15.16:0/4198335241 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4af8069950 con 0x7f4af8143c30 2024-03-31T15:03:33.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:33.613+0000 7f4aefe00640 1 -- 172.21.15.16:0/4198335241 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f4af001a020 con 0x7f4af8143c30 2024-03-31T15:03:33.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:33.613+0000 7f4aefe00640 1 --2- 172.21.15.16:0/4198335241 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f4ad4068380 0x7f4ad406a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:33.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:33.613+0000 7f4aefe00640 1 -- 172.21.15.16:0/4198335241 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 4857+0+0 (secure 0 0 0) 0x7f4af0085e80 con 0x7f4af8143c30 2024-03-31T15:03:33.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:33.613+0000 7f4af7400640 1 --2- 172.21.15.16:0/4198335241 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f4ad4068380 0x7f4ad406a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:33.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:33.613+0000 7f4af7400640 1 --2- 172.21.15.16:0/4198335241 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f4ad4068380 0x7f4ad406a840 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f4ae80ae000 tx=0x7f4ae80a7070 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:03:33.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:33.617+0000 7f4aefe00640 1 -- 172.21.15.16:0/4198335241 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f4af00566f0 con 0x7f4af8143c30 2024-03-31T15:03:33.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:33.781+0000 7f4afe000640 1 -- 172.21.15.16:0/4198335241 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f4af806e260 con 0x7f4af8143c30 2024-03-31T15:03:33.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:33.781+0000 7f4aefe00640 1 -- 172.21.15.16:0/4198335241 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v17) ==== 93+0+10 (secure 0 0 0) 0x7f4af0055ff0 con 0x7f4af8143c30 2024-03-31T15:03:33.781 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2024-03-31T15:03:33.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:33.785+0000 7f4afe000640 1 -- 172.21.15.16:0/4198335241 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f4ad4068380 msgr2=0x7f4ad406a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:33.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:33.785+0000 7f4afe000640 1 --2- 172.21.15.16:0/4198335241 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f4ad4068380 0x7f4ad406a840 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f4ae80ae000 tx=0x7f4ae80a7070 comp rx=0 tx=0).stop 2024-03-31T15:03:33.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:33.785+0000 7f4afe000640 1 -- 172.21.15.16:0/4198335241 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4af8143c30 msgr2=0x7f4af80b4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:33.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:33.785+0000 7f4afe000640 1 --2- 172.21.15.16:0/4198335241 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4af8143c30 0x7f4af80b4d90 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f4af000c3c0 tx=0x7f4af000c880 comp rx=0 tx=0).stop 2024-03-31T15:03:33.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:33.785+0000 7f4afe000640 1 -- 172.21.15.16:0/4198335241 shutdown_connections 2024-03-31T15:03:33.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:33.785+0000 7f4afe000640 1 --2- 172.21.15.16:0/4198335241 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f4ad4068380 0x7f4ad406a840 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:33.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:33.785+0000 7f4afe000640 1 --2- 172.21.15.16:0/4198335241 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4af8143c30 0x7f4af80b4d90 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:33.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:33.785+0000 7f4afe000640 1 --2- 172.21.15.16:0/4198335241 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f4af8141590 0x7f4af8156370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:33.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:33.785+0000 7f4afe000640 1 --2- 172.21.15.16:0/4198335241 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4af81411b0 0x7f4af813dd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:33.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:33.785+0000 7f4afe000640 1 -- 172.21.15.16:0/4198335241 >> 172.21.15.16:0/4198335241 conn(0x7f4af8067930 msgr2=0x7f4af80704e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:03:33.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:33.785+0000 7f4afe000640 1 -- 172.21.15.16:0/4198335241 shutdown_connections 2024-03-31T15:03:33.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:33.785+0000 7f4afe000640 1 -- 172.21.15.16:0/4198335241 wait complete. 2024-03-31T15:03:33.801 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-31T15:03:33.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:33.945+0000 7f11d7400640 1 Processor -- start 2024-03-31T15:03:33.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:33.945+0000 7f11d7400640 1 -- start start 2024-03-31T15:03:33.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:33.945+0000 7f11d7400640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f11d015f650 0x7f11d006f920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:33.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:33.945+0000 7f11d7400640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f11d006fe60 0x7f11d0162180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:33.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:33.945+0000 7f11d7400640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f11d00cbe90 con 0x7f11d006dbc0 2024-03-31T15:03:33.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:33.945+0000 7f11d7400640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f11d00702d0 con 0x7f11d006fe60 2024-03-31T15:03:33.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:33.945+0000 7f11d7400640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f11d015f1c0 con 0x7f11d015f650 2024-03-31T15:03:33.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:33.945+0000 7f11d5400640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f11d006fe60 0x7f11d0162180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:33.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:33.945+0000 7f11d5400640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f11d006fe60 0x7f11d0162180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59632/0 (socket says 172.21.15.16:59632) 2024-03-31T15:03:33.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:33.945+0000 7f11d5400640 1 -- 172.21.15.16:0/3673127808 learned_addr learned my addr 172.21.15.16:0/3673127808 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:03:33.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:33.945+0000 7f11cfe00640 1 --2- 172.21.15.16:0/3673127808 >> v2:172.21.15.89:3300/0 conn(0x7f11d015f650 0x7f11d006f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:33.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:33.945+0000 7f11d5400640 1 -- 172.21.15.16:0/3673127808 >> v2:172.21.15.89:3300/0 conn(0x7f11d015f650 msgr2=0x7f11d006f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:03:33.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:33.945+0000 7f11d5400640 1 --2- 172.21.15.16:0/3673127808 >> v2:172.21.15.89:3300/0 conn(0x7f11d015f650 0x7f11d006f920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:33.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:33.945+0000 7f11d5400640 1 -- 172.21.15.16:0/3673127808 >> v1:172.21.15.89:6789/0 conn(0x7f11d006dbc0 legacy=0x7f11d00cb440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:33.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:33.945+0000 7f11d5400640 1 -- 172.21.15.16:0/3673127808 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11d00cc170 con 0x7f11d006fe60 2024-03-31T15:03:33.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:33.945+0000 7f11cfe00640 1 --2- 172.21.15.16:0/3673127808 >> v2:172.21.15.89:3300/0 conn(0x7f11d015f650 0x7f11d006f920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:03:33.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:33.949+0000 7f11d5400640 1 --2- 172.21.15.16:0/3673127808 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f11d006fe60 0x7f11d0162180 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f11d00cf0b0 tx=0x7f11c801c930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=558ac3e2bc5b26f6 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:03:33.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:33.949+0000 7f11cf400640 1 -- 172.21.15.16:0/3673127808 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11c80a57b0 con 0x7f11d006fe60 2024-03-31T15:03:33.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:33.949+0000 7f11cf400640 1 -- 172.21.15.16:0/3673127808 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f11c80a5950 con 0x7f11d006fe60 2024-03-31T15:03:33.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:33.949+0000 7f11cf400640 1 -- 172.21.15.16:0/3673127808 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11c80a5d20 con 0x7f11d006fe60 2024-03-31T15:03:33.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:33.949+0000 7f11d7400640 1 -- 172.21.15.16:0/3673127808 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f11d006fe60 msgr2=0x7f11d0162180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:33.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:33.949+0000 7f11d7400640 1 --2- 172.21.15.16:0/3673127808 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f11d006fe60 0x7f11d0162180 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f11d00cf0b0 tx=0x7f11c801c930 comp rx=0 tx=0).stop 2024-03-31T15:03:33.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:33.949+0000 7f11d7400640 1 -- 172.21.15.16:0/3673127808 shutdown_connections 2024-03-31T15:03:33.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:33.949+0000 7f11d7400640 1 --2- 172.21.15.16:0/3673127808 >> v2:172.21.15.89:3300/0 conn(0x7f11d015f650 0x7f11d006f920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:33.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:33.949+0000 7f11d7400640 1 --2- 172.21.15.16:0/3673127808 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f11d006fe60 0x7f11d0162180 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:33.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:33.949+0000 7f11d7400640 1 -- 172.21.15.16:0/3673127808 >> 172.21.15.16:0/3673127808 conn(0x7f11d0067330 msgr2=0x7f11d0067740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:03:33.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:33.949+0000 7f11d7400640 1 -- 172.21.15.16:0/3673127808 shutdown_connections 2024-03-31T15:03:33.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:33.949+0000 7f11d7400640 1 -- 172.21.15.16:0/3673127808 wait complete. 2024-03-31T15:03:33.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:33.949+0000 7f11d7400640 1 Processor -- start 2024-03-31T15:03:33.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:33.953+0000 7f11d7400640 1 -- start start 2024-03-31T15:03:33.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:33.953+0000 7f11d7400640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f11d006dbc0 0x7f11d0142640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:33.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:33.953+0000 7f11d7400640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f11d006fe60 0x7f11d01409c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:33.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:33.953+0000 7f11d7400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f11d015f650 0x7f11d015b1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:33.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:33.953+0000 7f11d7400640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f11d015fa30 con 0x7f11d015f650 2024-03-31T15:03:33.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:33.953+0000 7f11d7400640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f11d006dfa0 con 0x7f11d006fe60 2024-03-31T15:03:33.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:33.953+0000 7f11d7400640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f11d015fbb0 con 0x7f11d006dbc0 2024-03-31T15:03:33.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:33.953+0000 7f11d5400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f11d015f650 0x7f11d015b1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:33.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:33.953+0000 7f11d5400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f11d015f650 0x7f11d015b1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59636/0 (socket says 172.21.15.16:59636) 2024-03-31T15:03:33.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:33.953+0000 7f11d5400640 1 -- 172.21.15.16:0/3254978771 learned_addr learned my addr 172.21.15.16:0/3254978771 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:03:33.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:33.953+0000 7f11cfe00640 1 --2- 172.21.15.16:0/3254978771 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f11d006fe60 0x7f11d01409c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:33.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:33.953+0000 7f11d4a00640 1 --2- 172.21.15.16:0/3254978771 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f11d006dbc0 0x7f11d0142640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:33.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:33.953+0000 7f11d5400640 1 -- 172.21.15.16:0/3254978771 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f11d006dbc0 msgr2=0x7f11d0142640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:33.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:33.953+0000 7f11d5400640 1 --2- 172.21.15.16:0/3254978771 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f11d006dbc0 0x7f11d0142640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:33.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:33.953+0000 7f11d5400640 1 -- 172.21.15.16:0/3254978771 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f11d006fe60 msgr2=0x7f11d01409c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:33.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:33.953+0000 7f11d5400640 1 --2- 172.21.15.16:0/3254978771 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f11d006fe60 0x7f11d01409c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:33.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:33.953+0000 7f11d5400640 1 -- 172.21.15.16:0/3254978771 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11d0140f00 con 0x7f11d015f650 2024-03-31T15:03:33.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:33.953+0000 7f11d5400640 1 --2- 172.21.15.16:0/3254978771 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f11d015f650 0x7f11d015b1a0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f11c80af040 tx=0x7f11c80196d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:03:33.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:33.953+0000 7f11ccc00640 1 -- 172.21.15.16:0/3254978771 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11c80b8670 con 0x7f11d015f650 2024-03-31T15:03:33.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:33.953+0000 7f11ccc00640 1 -- 172.21.15.16:0/3254978771 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f11c80be070 con 0x7f11d015f650 2024-03-31T15:03:33.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:33.953+0000 7f11ccc00640 1 -- 172.21.15.16:0/3254978771 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11c80b8a70 con 0x7f11d015f650 2024-03-31T15:03:33.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:33.953+0000 7f11d7400640 1 -- 172.21.15.16:0/3254978771 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f11d015a7e0 con 0x7f11d015f650 2024-03-31T15:03:33.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:33.953+0000 7f11d7400640 1 -- 172.21.15.16:0/3254978771 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f11d015bd70 con 0x7f11d015f650 2024-03-31T15:03:33.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:33.953+0000 7f11d7400640 1 -- 172.21.15.16:0/3254978771 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f11d0069950 con 0x7f11d015f650 2024-03-31T15:03:33.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:33.957+0000 7f11ccc00640 1 -- 172.21.15.16:0/3254978771 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f11c80b8c10 con 0x7f11d015f650 2024-03-31T15:03:33.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:33.957+0000 7f11ccc00640 1 --2- 172.21.15.16:0/3254978771 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f11b0068380 0x7f11b006a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:33.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:33.957+0000 7f11ccc00640 1 -- 172.21.15.16:0/3254978771 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 4857+0+0 (secure 0 0 0) 0x7f11c8121720 con 0x7f11d015f650 2024-03-31T15:03:33.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:33.957+0000 7f11d4a00640 1 --2- 172.21.15.16:0/3254978771 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f11b0068380 0x7f11b006a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:33.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:33.957+0000 7f11d4a00640 1 --2- 172.21.15.16:0/3254978771 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f11b0068380 0x7f11b006a840 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f11d015ae20 tx=0x7f11c4009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:03:33.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:33.961+0000 7f11ccc00640 1 -- 172.21.15.16:0/3254978771 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f11d0140f00 con 0x7f11d015f650 2024-03-31T15:03:34.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:34.125+0000 7f11d7400640 1 -- 172.21.15.16:0/3254978771 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7f11d0162140 con 0x7f11d015f650 2024-03-31T15:03:34.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:34.129+0000 7f11ccc00640 1 -- 172.21.15.16:0/3254978771 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v17) ==== 84+0+65 (secure 0 0 0) 0x7f11c80165e0 con 0x7f11d015f650 2024-03-31T15:03:34.124 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:03:34.124 INFO:teuthology.orchestra.run.smithi016.stdout:[ 2024-03-31T15:03:34.124 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-31T15:03:34.125 INFO:teuthology.orchestra.run.smithi016.stdout: "poolnum": 1, 2024-03-31T15:03:34.125 INFO:teuthology.orchestra.run.smithi016.stdout: "poolname": ".mgr" 2024-03-31T15:03:34.125 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-31T15:03:34.125 INFO:teuthology.orchestra.run.smithi016.stdout:] 2024-03-31T15:03:34.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:34.129+0000 7f11d7400640 1 -- 172.21.15.16:0/3254978771 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f11b0068380 msgr2=0x7f11b006a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:34.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:34.129+0000 7f11d7400640 1 --2- 172.21.15.16:0/3254978771 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f11b0068380 0x7f11b006a840 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f11d015ae20 tx=0x7f11c4009040 comp rx=0 tx=0).stop 2024-03-31T15:03:34.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:34.129+0000 7f11d7400640 1 -- 172.21.15.16:0/3254978771 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f11d015f650 msgr2=0x7f11d015b1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:34.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:34.129+0000 7f11d7400640 1 --2- 172.21.15.16:0/3254978771 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f11d015f650 0x7f11d015b1a0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f11c80af040 tx=0x7f11c80196d0 comp rx=0 tx=0).stop 2024-03-31T15:03:34.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:34.129+0000 7f11d7400640 1 -- 172.21.15.16:0/3254978771 shutdown_connections 2024-03-31T15:03:34.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:34.129+0000 7f11d7400640 1 --2- 172.21.15.16:0/3254978771 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f11b0068380 0x7f11b006a840 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:34.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:34.129+0000 7f11d7400640 1 --2- 172.21.15.16:0/3254978771 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f11d015f650 0x7f11d015b1a0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:34.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:34.129+0000 7f11d7400640 1 --2- 172.21.15.16:0/3254978771 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f11d006fe60 0x7f11d01409c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:34.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:34.129+0000 7f11d7400640 1 --2- 172.21.15.16:0/3254978771 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f11d006dbc0 0x7f11d0142640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:34.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:34.129+0000 7f11d7400640 1 -- 172.21.15.16:0/3254978771 >> 172.21.15.16:0/3254978771 conn(0x7f11d0067330 msgr2=0x7f11d015c820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:03:34.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:34.129+0000 7f11d7400640 1 -- 172.21.15.16:0/3254978771 shutdown_connections 2024-03-31T15:03:34.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:34.129+0000 7f11d7400640 1 -- 172.21.15.16:0/3254978771 wait complete. 2024-03-31T15:03:34.144 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-31T15:03:34.145 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-31T15:03:34.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:34.293+0000 7f8008c00640 1 Processor -- start 2024-03-31T15:03:34.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:34.293+0000 7f8008c00640 1 -- start start 2024-03-31T15:03:34.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:34.293+0000 7f8008c00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f8004147230 0x7f80040b3dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:34.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:34.293+0000 7f8008c00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f80040b4300 0x7f80040d0930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:34.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:34.293+0000 7f8008c00640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f80040d2020 con 0x7f80040d0e70 2024-03-31T15:03:34.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:34.293+0000 7f8008c00640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f80040b4760 con 0x7f80040b4300 2024-03-31T15:03:34.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:34.293+0000 7f8008c00640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f80040d2c30 con 0x7f8004147230 2024-03-31T15:03:34.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:34.293+0000 7f7ffbe00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f80040b4300 0x7f80040d0930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:34.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:34.293+0000 7f7ffbe00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f80040b4300 0x7f80040d0930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59642/0 (socket says 172.21.15.16:59642) 2024-03-31T15:03:34.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:34.293+0000 7f7ffbe00640 1 -- 172.21.15.16:0/1177831820 learned_addr learned my addr 172.21.15.16:0/1177831820 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:03:34.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:34.293+0000 7f8001e00640 1 --2- 172.21.15.16:0/1177831820 >> v2:172.21.15.89:3300/0 conn(0x7f8004147230 0x7f80040b3dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:34.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:34.293+0000 7f7ffbe00640 1 -- 172.21.15.16:0/1177831820 >> v2:172.21.15.89:3300/0 conn(0x7f8004147230 msgr2=0x7f80040b3dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:03:34.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:34.293+0000 7f7ffbe00640 1 --2- 172.21.15.16:0/1177831820 >> v2:172.21.15.89:3300/0 conn(0x7f8004147230 0x7f80040b3dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:34.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:34.293+0000 7f7ffbe00640 1 -- 172.21.15.16:0/1177831820 >> v1:172.21.15.89:6789/0 conn(0x7f80040d0e70 legacy=0x7f8004162110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:34.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:34.293+0000 7f7ffbe00640 1 -- 172.21.15.16:0/1177831820 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80040d2300 con 0x7f80040b4300 2024-03-31T15:03:34.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:34.297+0000 7f7ffbe00640 1 --2- 172.21.15.16:0/1177831820 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f80040b4300 0x7f80040d0930 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f8004067e00 tx=0x7f7fec01c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ed6e9094363214 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:03:34.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:34.297+0000 7f8001400640 1 -- 172.21.15.16:0/1177831820 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fec0ab610 con 0x7f80040b4300 2024-03-31T15:03:34.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:34.297+0000 7f8001400640 1 -- 172.21.15.16:0/1177831820 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7fec0b0070 con 0x7f80040b4300 2024-03-31T15:03:34.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:34.297+0000 7f8001400640 1 -- 172.21.15.16:0/1177831820 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fec0ab9d0 con 0x7f80040b4300 2024-03-31T15:03:34.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:34.297+0000 7f8008c00640 1 -- 172.21.15.16:0/1177831820 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f80040b4300 msgr2=0x7f80040d0930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:34.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:34.297+0000 7f8008c00640 1 --2- 172.21.15.16:0/1177831820 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f80040b4300 0x7f80040d0930 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f8004067e00 tx=0x7f7fec01c520 comp rx=0 tx=0).stop 2024-03-31T15:03:34.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:34.297+0000 7f8008c00640 1 -- 172.21.15.16:0/1177831820 shutdown_connections 2024-03-31T15:03:34.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:34.297+0000 7f8008c00640 1 --2- 172.21.15.16:0/1177831820 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f80040b4300 0x7f80040d0930 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:34.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:34.297+0000 7f8008c00640 1 --2- 172.21.15.16:0/1177831820 >> v2:172.21.15.89:3300/0 conn(0x7f8004147230 0x7f80040b3dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:34.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:34.297+0000 7f8008c00640 1 -- 172.21.15.16:0/1177831820 >> 172.21.15.16:0/1177831820 conn(0x7f800406b7f0 msgr2=0x7f8004067330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:03:34.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:34.297+0000 7f8008c00640 1 -- 172.21.15.16:0/1177831820 shutdown_connections 2024-03-31T15:03:34.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:34.297+0000 7f8008c00640 1 -- 172.21.15.16:0/1177831820 wait complete. 2024-03-31T15:03:34.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:34.301+0000 7f8008c00640 1 Processor -- start 2024-03-31T15:03:34.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:34.301+0000 7f8008c00640 1 -- start start 2024-03-31T15:03:34.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:34.301+0000 7f8008c00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f80040b4300 0x7f80041614f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:34.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:34.301+0000 7f8008c00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f80040d0e70 0x7f800413e460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:34.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:34.301+0000 7f8008c00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f8004147230 0x7f800413e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:34.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:34.301+0000 7f8008c00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f800416b280 con 0x7f80040b4300 2024-03-31T15:03:34.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:34.301+0000 7f8008c00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f80040cc4c0 con 0x7f80040d0e70 2024-03-31T15:03:34.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:34.301+0000 7f8008c00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f800416b400 con 0x7f8004147230 2024-03-31T15:03:34.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:34.301+0000 7f8001e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f80040b4300 0x7f80041614f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:34.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:34.301+0000 7f8001e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f80040b4300 0x7f80041614f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59656/0 (socket says 172.21.15.16:59656) 2024-03-31T15:03:34.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:34.301+0000 7f8001e00640 1 -- 172.21.15.16:0/3335955062 learned_addr learned my addr 172.21.15.16:0/3335955062 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:03:34.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:34.301+0000 7f7ffbe00640 1 --2- 172.21.15.16:0/3335955062 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f80040d0e70 0x7f800413e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:34.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:34.301+0000 7f8002800640 1 --2- 172.21.15.16:0/3335955062 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f8004147230 0x7f800413e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:34.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:34.301+0000 7f8001e00640 1 -- 172.21.15.16:0/3335955062 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f8004147230 msgr2=0x7f800413e9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:34.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:34.301+0000 7f8001e00640 1 --2- 172.21.15.16:0/3335955062 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f8004147230 0x7f800413e9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:34.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:34.301+0000 7f8001e00640 1 -- 172.21.15.16:0/3335955062 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f80040d0e70 msgr2=0x7f800413e460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:34.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:34.301+0000 7f8001e00640 1 --2- 172.21.15.16:0/3335955062 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f80040d0e70 0x7f800413e460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:34.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:34.301+0000 7f8001e00640 1 -- 172.21.15.16:0/3335955062 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f800406c2a0 con 0x7f80040b4300 2024-03-31T15:03:34.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:34.301+0000 7f8002800640 1 --2- 172.21.15.16:0/3335955062 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f8004147230 0x7f800413e9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:03:34.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:34.301+0000 7f8001e00640 1 --2- 172.21.15.16:0/3335955062 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f80040b4300 0x7f80041614f0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f7ff400d620 tx=0x7f7ff400dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:03:34.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:34.301+0000 7f7ffa000640 1 -- 172.21.15.16:0/3335955062 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ff400b620 con 0x7f80040b4300 2024-03-31T15:03:34.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:34.301+0000 7f7ffa000640 1 -- 172.21.15.16:0/3335955062 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7ff4019070 con 0x7f80040b4300 2024-03-31T15:03:34.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:34.301+0000 7f7ffa000640 1 -- 172.21.15.16:0/3335955062 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ff400b9e0 con 0x7f80040b4300 2024-03-31T15:03:34.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:34.301+0000 7f8008c00640 1 -- 172.21.15.16:0/3335955062 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8004145970 con 0x7f80040b4300 2024-03-31T15:03:34.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:34.301+0000 7f8008c00640 1 -- 172.21.15.16:0/3335955062 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f800415ff80 con 0x7f80040b4300 2024-03-31T15:03:34.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:34.305+0000 7f8008c00640 1 -- 172.21.15.16:0/3335955062 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8004069950 con 0x7f80040b4300 2024-03-31T15:03:34.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:34.305+0000 7f7ffa000640 1 -- 172.21.15.16:0/3335955062 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f7ff400bb80 con 0x7f80040b4300 2024-03-31T15:03:34.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:34.305+0000 7f7ffa000640 1 --2- 172.21.15.16:0/3335955062 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f7fe0068380 0x7f7fe006a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:34.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:34.305+0000 7f7ffa000640 1 -- 172.21.15.16:0/3335955062 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 4857+0+0 (secure 0 0 0) 0x7f7ff4086a30 con 0x7f80040b4300 2024-03-31T15:03:34.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:34.305+0000 7f7ffbe00640 1 --2- 172.21.15.16:0/3335955062 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f7fe0068380 0x7f7fe006a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:34.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:34.305+0000 7f7ffbe00640 1 --2- 172.21.15.16:0/3335955062 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f7fe0068380 0x7f7fe006a840 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f7fec0af000 tx=0x7f7fec0a7000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:03:34.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:34.309+0000 7f7ffa000640 1 -- 172.21.15.16:0/3335955062 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f7ff4057330 con 0x7f80040b4300 2024-03-31T15:03:34.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:34.473+0000 7f8008c00640 1 -- 172.21.15.16:0/3335955062 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7f800406d4c0 con 0x7f80040b4300 2024-03-31T15:03:34.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:34.925+0000 7f7ffa000640 1 -- 172.21.15.16:0/3335955062 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v18) ==== 138+0+0 (secure 0 0 0) 0x7f7ff4056c30 con 0x7f80040b4300 2024-03-31T15:03:34.922 INFO:teuthology.orchestra.run.smithi016.stderr:pool 'cephfs_metadata' created 2024-03-31T15:03:34.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:34.929+0000 7f8008c00640 1 -- 172.21.15.16:0/3335955062 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f7fe0068380 msgr2=0x7f7fe006a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:34.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:34.929+0000 7f8008c00640 1 --2- 172.21.15.16:0/3335955062 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f7fe0068380 0x7f7fe006a840 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f7fec0af000 tx=0x7f7fec0a7000 comp rx=0 tx=0).stop 2024-03-31T15:03:34.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:34.929+0000 7f8008c00640 1 -- 172.21.15.16:0/3335955062 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f80040b4300 msgr2=0x7f80041614f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:34.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:34.929+0000 7f8008c00640 1 --2- 172.21.15.16:0/3335955062 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f80040b4300 0x7f80041614f0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f7ff400d620 tx=0x7f7ff400dae0 comp rx=0 tx=0).stop 2024-03-31T15:03:34.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:34.929+0000 7f8008c00640 1 -- 172.21.15.16:0/3335955062 shutdown_connections 2024-03-31T15:03:34.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:34.929+0000 7f8008c00640 1 --2- 172.21.15.16:0/3335955062 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f7fe0068380 0x7f7fe006a840 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:34.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:34.929+0000 7f8008c00640 1 --2- 172.21.15.16:0/3335955062 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f8004147230 0x7f800413e9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:34.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:34.929+0000 7f8008c00640 1 --2- 172.21.15.16:0/3335955062 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f80040d0e70 0x7f800413e460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:34.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:34.929+0000 7f8008c00640 1 --2- 172.21.15.16:0/3335955062 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f80040b4300 0x7f80041614f0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:34.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:34.929+0000 7f8008c00640 1 -- 172.21.15.16:0/3335955062 >> 172.21.15.16:0/3335955062 conn(0x7f800406b7f0 msgr2=0x7f8004067330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:03:34.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:34.929+0000 7f8008c00640 1 -- 172.21.15.16:0/3335955062 shutdown_connections 2024-03-31T15:03:34.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:34.929+0000 7f8008c00640 1 -- 172.21.15.16:0/3335955062 wait complete. 2024-03-31T15:03:34.951 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-31T15:03:35.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:35.101+0000 7f0d54400640 1 Processor -- start 2024-03-31T15:03:35.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:35.101+0000 7f0d54400640 1 -- start start 2024-03-31T15:03:35.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:35.101+0000 7f0d54400640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f0d4c06d4d0 0x7f0d4c06d930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:35.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:35.101+0000 7f0d54400640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0d4c06de70 con 0x7f0d4c1567e0 2024-03-31T15:03:35.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:35.101+0000 7f0d54400640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0d4c15c7e0 con 0x7f0d4c071550 2024-03-31T15:03:35.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:35.101+0000 7f0d54400640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f0d4c0736b0 con 0x7f0d4c06d4d0 2024-03-31T15:03:35.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:35.101+0000 7f0d51a00640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f0d4c1567e0 0x7f0d4c072c80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:40648/0 (socket says 172.21.15.16:40648) 2024-03-31T15:03:35.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:35.101+0000 7f0d51a00640 1 -- 172.21.15.16:0/1495886969 learned_addr learned my addr 172.21.15.16:0/1495886969 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:03:35.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:35.101+0000 7f0d51000640 1 --2- 172.21.15.16:0/1495886969 >> v2:172.21.15.16:3300/0 conn(0x7f0d4c06d4d0 0x7f0d4c06d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:35.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:35.101+0000 7f0d4be00640 1 -- 172.21.15.16:0/1495886969 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1166200433 0 0) 0x7f0d4c06de70 con 0x7f0d4c1567e0 2024-03-31T15:03:35.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:35.105+0000 7f0d4be00640 1 -- 172.21.15.16:0/1495886969 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0d34003530 con 0x7f0d4c1567e0 2024-03-31T15:03:35.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:35.105+0000 7f0d51000640 1 -- 172.21.15.16:0/1495886969 >> v1:172.21.15.89:6789/0 conn(0x7f0d4c071550 legacy=0x7f0d4c071960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:35.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:35.105+0000 7f0d51000640 1 -- 172.21.15.16:0/1495886969 >> v1:172.21.15.16:6789/0 conn(0x7f0d4c1567e0 legacy=0x7f0d4c072c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:35.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:35.105+0000 7f0d51000640 1 -- 172.21.15.16:0/1495886969 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d4c15caa0 con 0x7f0d4c06d4d0 2024-03-31T15:03:35.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:35.105+0000 7f0d51000640 1 --2- 172.21.15.16:0/1495886969 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0d4c06d4d0 0x7f0d4c06d930 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f0d380097c0 tx=0x7f0d3801c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9fea68502d25c859 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:03:35.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:35.105+0000 7f0d4be00640 1 -- 172.21.15.16:0/1495886969 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d380b0070 con 0x7f0d4c06d4d0 2024-03-31T15:03:35.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:35.105+0000 7f0d4be00640 1 -- 172.21.15.16:0/1495886969 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0d380ab600 con 0x7f0d4c06d4d0 2024-03-31T15:03:35.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:35.105+0000 7f0d4be00640 1 -- 172.21.15.16:0/1495886969 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d380ab9d0 con 0x7f0d4c06d4d0 2024-03-31T15:03:35.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:35.105+0000 7f0d54400640 1 -- 172.21.15.16:0/1495886969 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0d4c06d4d0 msgr2=0x7f0d4c06d930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:35.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:35.105+0000 7f0d54400640 1 --2- 172.21.15.16:0/1495886969 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0d4c06d4d0 0x7f0d4c06d930 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f0d380097c0 tx=0x7f0d3801c4a0 comp rx=0 tx=0).stop 2024-03-31T15:03:35.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:35.105+0000 7f0d54400640 1 -- 172.21.15.16:0/1495886969 shutdown_connections 2024-03-31T15:03:35.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:35.105+0000 7f0d54400640 1 --2- 172.21.15.16:0/1495886969 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0d4c06d4d0 0x7f0d4c06d930 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:35.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:35.105+0000 7f0d54400640 1 -- 172.21.15.16:0/1495886969 >> 172.21.15.16:0/1495886969 conn(0x7f0d4c06ee40 msgr2=0x7f0d4c06b620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:03:35.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:35.105+0000 7f0d54400640 1 -- 172.21.15.16:0/1495886969 shutdown_connections 2024-03-31T15:03:35.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:35.105+0000 7f0d54400640 1 -- 172.21.15.16:0/1495886969 wait complete. 2024-03-31T15:03:35.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:35.109+0000 7f0d54400640 1 Processor -- start 2024-03-31T15:03:35.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:35.109+0000 7f0d54400640 1 -- start start 2024-03-31T15:03:35.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:35.109+0000 7f0d54400640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0d4c06d4d0 0x7f0d4c148ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:35.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:35.109+0000 7f0d54400640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f0d4c071550 0x7f0d4c137980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:35.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:35.109+0000 7f0d54400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0d4c1567e0 0x7f0d4c14e3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:35.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:35.109+0000 7f0d54400640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f0d4c0afbb0 con 0x7f0d4c1567e0 2024-03-31T15:03:35.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:35.109+0000 7f0d54400640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f0d4c0afa30 con 0x7f0d4c06d4d0 2024-03-31T15:03:35.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:35.109+0000 7f0d54400640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f0d4c15d560 con 0x7f0d4c071550 2024-03-31T15:03:35.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:35.109+0000 7f0d52400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0d4c1567e0 0x7f0d4c14e3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:35.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:35.109+0000 7f0d52400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0d4c1567e0 0x7f0d4c14e3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59672/0 (socket says 172.21.15.16:59672) 2024-03-31T15:03:35.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:35.109+0000 7f0d52400640 1 -- 172.21.15.16:0/3494644906 learned_addr learned my addr 172.21.15.16:0/3494644906 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:03:35.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:35.109+0000 7f0d51a00640 1 --2- 172.21.15.16:0/3494644906 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0d4c06d4d0 0x7f0d4c148ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:35.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:35.109+0000 7f0d51000640 1 --2- 172.21.15.16:0/3494644906 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f0d4c071550 0x7f0d4c137980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:35.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:35.109+0000 7f0d52400640 1 -- 172.21.15.16:0/3494644906 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f0d4c071550 msgr2=0x7f0d4c137980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:35.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:35.109+0000 7f0d52400640 1 --2- 172.21.15.16:0/3494644906 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f0d4c071550 0x7f0d4c137980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:35.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:35.109+0000 7f0d52400640 1 -- 172.21.15.16:0/3494644906 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0d4c06d4d0 msgr2=0x7f0d4c148ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:35.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:35.109+0000 7f0d52400640 1 --2- 172.21.15.16:0/3494644906 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0d4c06d4d0 0x7f0d4c148ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:35.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:35.109+0000 7f0d52400640 1 -- 172.21.15.16:0/3494644906 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d4c136ee0 con 0x7f0d4c1567e0 2024-03-31T15:03:35.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:35.109+0000 7f0d52400640 1 --2- 172.21.15.16:0/3494644906 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0d4c1567e0 0x7f0d4c14e3d0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f0d4400c7a0 tx=0x7f0d4400cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:03:35.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:35.109+0000 7f0d49600640 1 -- 172.21.15.16:0/3494644906 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d44003b80 con 0x7f0d4c1567e0 2024-03-31T15:03:35.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:35.109+0000 7f0d49600640 1 -- 172.21.15.16:0/3494644906 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0d44003d20 con 0x7f0d4c1567e0 2024-03-31T15:03:35.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:35.109+0000 7f0d49600640 1 -- 172.21.15.16:0/3494644906 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d4400aa90 con 0x7f0d4c1567e0 2024-03-31T15:03:35.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:35.109+0000 7f0d54400640 1 -- 172.21.15.16:0/3494644906 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0d4c14f5a0 con 0x7f0d4c1567e0 2024-03-31T15:03:35.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:35.109+0000 7f0d54400640 1 -- 172.21.15.16:0/3494644906 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0d4c137460 con 0x7f0d4c1567e0 2024-03-31T15:03:35.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:35.113+0000 7f0d54400640 1 -- 172.21.15.16:0/3494644906 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0d4c069950 con 0x7f0d4c1567e0 2024-03-31T15:03:35.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:35.113+0000 7f0d49600640 1 -- 172.21.15.16:0/3494644906 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f0d4400ac30 con 0x7f0d4c1567e0 2024-03-31T15:03:35.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:35.113+0000 7f0d49600640 1 --2- 172.21.15.16:0/3494644906 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f0d2c068380 0x7f0d2c06a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:35.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:35.113+0000 7f0d49600640 1 -- 172.21.15.16:0/3494644906 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(18..18 src has 1..18) ==== 5227+0+0 (secure 0 0 0) 0x7f0d44086620 con 0x7f0d4c1567e0 2024-03-31T15:03:35.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:35.113+0000 7f0d51a00640 1 --2- 172.21.15.16:0/3494644906 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f0d2c068380 0x7f0d2c06a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:35.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:35.117+0000 7f0d51a00640 1 --2- 172.21.15.16:0/3494644906 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f0d2c068380 0x7f0d2c06a840 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f0d400011f0 tx=0x7f0d40007d40 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:03:35.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:35.117+0000 7f0d49600640 1 -- 172.21.15.16:0/3494644906 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f0d440566d0 con 0x7f0d4c1567e0 2024-03-31T15:03:35.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:35.285+0000 7f0d54400640 1 -- 172.21.15.16:0/3494644906 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7f0d4c06e470 con 0x7f0d4c1567e0 2024-03-31T15:03:35.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:35.929+0000 7f0d49600640 1 -- 172.21.15.16:0/3494644906 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v19) ==== 170+0+0 (secure 0 0 0) 0x7f0d4405a9d0 con 0x7f0d4c1567e0 2024-03-31T15:03:35.926 INFO:teuthology.orchestra.run.smithi016.stderr:pool 'cephfs_data' created 2024-03-31T15:03:35.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:35.933+0000 7f0d54400640 1 -- 172.21.15.16:0/3494644906 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f0d2c068380 msgr2=0x7f0d2c06a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:35.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:35.933+0000 7f0d54400640 1 --2- 172.21.15.16:0/3494644906 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f0d2c068380 0x7f0d2c06a840 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f0d400011f0 tx=0x7f0d40007d40 comp rx=0 tx=0).stop 2024-03-31T15:03:35.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:35.933+0000 7f0d54400640 1 -- 172.21.15.16:0/3494644906 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0d4c1567e0 msgr2=0x7f0d4c14e3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:35.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:35.933+0000 7f0d54400640 1 --2- 172.21.15.16:0/3494644906 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0d4c1567e0 0x7f0d4c14e3d0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f0d4400c7a0 tx=0x7f0d4400cc60 comp rx=0 tx=0).stop 2024-03-31T15:03:35.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:35.933+0000 7f0d54400640 1 -- 172.21.15.16:0/3494644906 shutdown_connections 2024-03-31T15:03:35.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:35.933+0000 7f0d54400640 1 --2- 172.21.15.16:0/3494644906 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f0d2c068380 0x7f0d2c06a840 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:35.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:35.933+0000 7f0d54400640 1 --2- 172.21.15.16:0/3494644906 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0d4c1567e0 0x7f0d4c14e3d0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:35.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:35.933+0000 7f0d54400640 1 --2- 172.21.15.16:0/3494644906 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f0d4c071550 0x7f0d4c137980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:35.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:35.933+0000 7f0d54400640 1 --2- 172.21.15.16:0/3494644906 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0d4c06d4d0 0x7f0d4c148ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:35.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:35.933+0000 7f0d54400640 1 -- 172.21.15.16:0/3494644906 >> 172.21.15.16:0/3494644906 conn(0x7f0d4c06ee40 msgr2=0x7f0d4c0b0220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:03:35.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:35.933+0000 7f0d54400640 1 -- 172.21.15.16:0/3494644906 shutdown_connections 2024-03-31T15:03:35.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:35.933+0000 7f0d54400640 1 -- 172.21.15.16:0/3494644906 wait complete. 2024-03-31T15:03:35.971 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-31T15:03:36.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:36.121+0000 7f0bc2e00640 1 Processor -- start 2024-03-31T15:03:36.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:36.121+0000 7f0bc2e00640 1 -- start start 2024-03-31T15:03:36.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:36.121+0000 7f0bc2e00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f0bbc147d60 0x7f0bbc0b3130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:36.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:36.121+0000 7f0bc2e00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f0bbc06dbe0 0x7f0bbc0ada40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:36.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:36.121+0000 7f0bc2e00640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0bbc0737c0 con 0x7f0bbc073310 2024-03-31T15:03:36.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:36.121+0000 7f0bc2e00640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f0bbc148140 con 0x7f0bbc06dbe0 2024-03-31T15:03:36.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:36.121+0000 7f0bc2e00640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f0bbc0b37c0 con 0x7f0bbc147d60 2024-03-31T15:03:36.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:36.121+0000 7f0bc0e00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f0bbc06dbe0 0x7f0bbc0ada40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:36.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:36.121+0000 7f0bc0e00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f0bbc06dbe0 0x7f0bbc0ada40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59678/0 (socket says 172.21.15.16:59678) 2024-03-31T15:03:36.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:36.121+0000 7f0bc0e00640 1 -- 172.21.15.16:0/2435142831 learned_addr learned my addr 172.21.15.16:0/2435142831 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:03:36.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:36.121+0000 7f0bbbe00640 1 --2- 172.21.15.16:0/2435142831 >> v2:172.21.15.89:3300/0 conn(0x7f0bbc147d60 0x7f0bbc0b3130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:36.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:36.121+0000 7f0bc0e00640 1 -- 172.21.15.16:0/2435142831 >> v2:172.21.15.89:3300/0 conn(0x7f0bbc147d60 msgr2=0x7f0bbc0b3130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:03:36.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:36.121+0000 7f0bc0e00640 1 --2- 172.21.15.16:0/2435142831 >> v2:172.21.15.89:3300/0 conn(0x7f0bbc147d60 0x7f0bbc0b3130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:36.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:36.121+0000 7f0bc0e00640 1 -- 172.21.15.16:0/2435142831 >> v1:172.21.15.89:6789/0 conn(0x7f0bbc073310 legacy=0x7f0bbc06d4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:36.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:36.121+0000 7f0bc0e00640 1 -- 172.21.15.16:0/2435142831 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0bbc0b3aa0 con 0x7f0bbc06dbe0 2024-03-31T15:03:36.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:36.125+0000 7f0bc0e00640 1 --2- 172.21.15.16:0/2435142831 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0bbc06dbe0 0x7f0bbc0ada40 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f0bbc0b5590 tx=0x7f0bb401c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c2f07bade8d0e119 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:03:36.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:36.125+0000 7f0bbb400640 1 -- 172.21.15.16:0/2435142831 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0bb40a5880 con 0x7f0bbc06dbe0 2024-03-31T15:03:36.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:36.125+0000 7f0bbb400640 1 -- 172.21.15.16:0/2435142831 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0bb40a5a20 con 0x7f0bbc06dbe0 2024-03-31T15:03:36.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:36.125+0000 7f0bbb400640 1 -- 172.21.15.16:0/2435142831 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0bb40a5df0 con 0x7f0bbc06dbe0 2024-03-31T15:03:36.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:36.125+0000 7f0bc2e00640 1 -- 172.21.15.16:0/2435142831 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0bbc06dbe0 msgr2=0x7f0bbc0ada40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:36.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:36.125+0000 7f0bc2e00640 1 --2- 172.21.15.16:0/2435142831 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0bbc06dbe0 0x7f0bbc0ada40 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f0bbc0b5590 tx=0x7f0bb401c9c0 comp rx=0 tx=0).stop 2024-03-31T15:03:36.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:36.125+0000 7f0bc2e00640 1 -- 172.21.15.16:0/2435142831 shutdown_connections 2024-03-31T15:03:36.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:36.125+0000 7f0bc2e00640 1 --2- 172.21.15.16:0/2435142831 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0bbc06dbe0 0x7f0bbc0ada40 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:36.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:36.125+0000 7f0bc2e00640 1 --2- 172.21.15.16:0/2435142831 >> v2:172.21.15.89:3300/0 conn(0x7f0bbc147d60 0x7f0bbc0b3130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:36.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:36.125+0000 7f0bc2e00640 1 -- 172.21.15.16:0/2435142831 >> 172.21.15.16:0/2435142831 conn(0x7f0bbc06ee40 msgr2=0x7f0bbc06b620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:03:36.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:36.125+0000 7f0bc2e00640 1 -- 172.21.15.16:0/2435142831 shutdown_connections 2024-03-31T15:03:36.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:36.125+0000 7f0bc2e00640 1 -- 172.21.15.16:0/2435142831 wait complete. 2024-03-31T15:03:36.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:36.129+0000 7f0bc2e00640 1 Processor -- start 2024-03-31T15:03:36.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:36.129+0000 7f0bc2e00640 1 -- start start 2024-03-31T15:03:36.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:36.129+0000 7f0bc2e00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f0bbc06dbe0 0x7f0bbc15ef90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:36.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:36.129+0000 7f0bc2e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0bbc073310 0x7f0bbc168ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:36.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:36.129+0000 7f0bc2e00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0bbc147d60 0x7f0bbc0b1e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:36.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:36.129+0000 7f0bc2e00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f0bbc0ae910 con 0x7f0bbc073310 2024-03-31T15:03:36.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:36.129+0000 7f0bc2e00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f0bbc0ae790 con 0x7f0bbc147d60 2024-03-31T15:03:36.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:36.129+0000 7f0bc2e00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f0bbc0aea90 con 0x7f0bbc06dbe0 2024-03-31T15:03:36.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:36.129+0000 7f0bb3e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0bbc073310 0x7f0bbc168ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:36.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:36.129+0000 7f0bb3e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0bbc073310 0x7f0bbc168ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59684/0 (socket says 172.21.15.16:59684) 2024-03-31T15:03:36.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:36.129+0000 7f0bb3e00640 1 -- 172.21.15.16:0/1782853471 learned_addr learned my addr 172.21.15.16:0/1782853471 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:03:36.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:36.129+0000 7f0bbbe00640 1 --2- 172.21.15.16:0/1782853471 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f0bbc06dbe0 0x7f0bbc15ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:36.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:36.129+0000 7f0bc0e00640 1 --2- 172.21.15.16:0/1782853471 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0bbc147d60 0x7f0bbc0b1e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:36.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:36.129+0000 7f0bb3e00640 1 -- 172.21.15.16:0/1782853471 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f0bbc06dbe0 msgr2=0x7f0bbc15ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:36.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:36.129+0000 7f0bb3e00640 1 --2- 172.21.15.16:0/1782853471 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f0bbc06dbe0 0x7f0bbc15ef90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:36.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:36.129+0000 7f0bb3e00640 1 -- 172.21.15.16:0/1782853471 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0bbc147d60 msgr2=0x7f0bbc0b1e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:36.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:36.129+0000 7f0bb3e00640 1 --2- 172.21.15.16:0/1782853471 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0bbc147d60 0x7f0bbc0b1e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:36.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:36.129+0000 7f0bb3e00640 1 -- 172.21.15.16:0/1782853471 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0bbc1444d0 con 0x7f0bbc073310 2024-03-31T15:03:36.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:36.129+0000 7f0bb3e00640 1 --2- 172.21.15.16:0/1782853471 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0bbc073310 0x7f0bbc168ab0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f0ba400c420 tx=0x7f0ba400c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:03:36.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:36.129+0000 7f0bb8c00640 1 -- 172.21.15.16:0/1782853471 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ba40052d0 con 0x7f0bbc073310 2024-03-31T15:03:36.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:36.129+0000 7f0bb8c00640 1 -- 172.21.15.16:0/1782853471 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0ba4005470 con 0x7f0bbc073310 2024-03-31T15:03:36.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:36.129+0000 7f0bc2e00640 1 -- 172.21.15.16:0/1782853471 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0bbc1682b0 con 0x7f0bbc073310 2024-03-31T15:03:36.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:36.129+0000 7f0bb8c00640 1 -- 172.21.15.16:0/1782853471 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ba4007260 con 0x7f0bbc073310 2024-03-31T15:03:36.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:36.129+0000 7f0bc2e00640 1 -- 172.21.15.16:0/1782853471 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0bbc168ff0 con 0x7f0bbc073310 2024-03-31T15:03:36.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:36.129+0000 7f0bc2e00640 1 -- 172.21.15.16:0/1782853471 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0bbc1446a0 con 0x7f0bbc073310 2024-03-31T15:03:36.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:36.133+0000 7f0bb8c00640 1 -- 172.21.15.16:0/1782853471 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f0ba4007400 con 0x7f0bbc073310 2024-03-31T15:03:36.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:36.133+0000 7f0bb8c00640 1 --2- 172.21.15.16:0/1782853471 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f0b98068380 0x7f0b9806a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:36.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:36.133+0000 7f0bb8c00640 1 -- 172.21.15.16:0/1782853471 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(19..19 src has 1..19) ==== 5609+0+0 (secure 0 0 0) 0x7f0ba4086550 con 0x7f0bbc073310 2024-03-31T15:03:36.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:36.133+0000 7f0bbbe00640 1 --2- 172.21.15.16:0/1782853471 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f0b98068380 0x7f0b9806a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:36.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:36.133+0000 7f0bbbe00640 1 --2- 172.21.15.16:0/1782853471 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f0b98068380 0x7f0b9806a840 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f0bbc140fb0 tx=0x7f0bac008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:03:36.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:36.137+0000 7f0bb8c00640 1 -- 172.21.15.16:0/1782853471 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f0ba4056600 con 0x7f0bbc073310 2024-03-31T15:03:36.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:36.341+0000 7f0bc2e00640 1 -- 172.21.15.16:0/1782853471 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7f0bbc069950 con 0x7f0bbc073310 2024-03-31T15:03:36.521 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-31T15:03:36.525 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-31T15:03:36.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:36.953+0000 7f0bb8c00640 1 -- 172.21.15.16:0/1782853471 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v4) ==== 172+0+0 (secure 0 0 0) 0x7f0ba405a900 con 0x7f0bbc073310 2024-03-31T15:03:36.953 INFO:teuthology.orchestra.run.smithi016.stderr:new fs with metadata pool 2 and data pool 3 2024-03-31T15:03:36.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:36.961+0000 7f0bc2e00640 1 -- 172.21.15.16:0/1782853471 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f0b98068380 msgr2=0x7f0b9806a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:36.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:36.961+0000 7f0bc2e00640 1 --2- 172.21.15.16:0/1782853471 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f0b98068380 0x7f0b9806a840 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f0bbc140fb0 tx=0x7f0bac008040 comp rx=0 tx=0).stop 2024-03-31T15:03:36.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:36.961+0000 7f0bc2e00640 1 -- 172.21.15.16:0/1782853471 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0bbc073310 msgr2=0x7f0bbc168ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:36.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:36.961+0000 7f0bc2e00640 1 --2- 172.21.15.16:0/1782853471 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0bbc073310 0x7f0bbc168ab0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f0ba400c420 tx=0x7f0ba400c8e0 comp rx=0 tx=0).stop 2024-03-31T15:03:36.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:36.961+0000 7f0bc2e00640 1 -- 172.21.15.16:0/1782853471 shutdown_connections 2024-03-31T15:03:36.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:36.961+0000 7f0bc2e00640 1 --2- 172.21.15.16:0/1782853471 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f0b98068380 0x7f0b9806a840 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:36.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:36.961+0000 7f0bc2e00640 1 --2- 172.21.15.16:0/1782853471 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0bbc147d60 0x7f0bbc0b1e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:36.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:36.961+0000 7f0bc2e00640 1 --2- 172.21.15.16:0/1782853471 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0bbc073310 0x7f0bbc168ab0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:36.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:36.961+0000 7f0bc2e00640 1 --2- 172.21.15.16:0/1782853471 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f0bbc06dbe0 0x7f0bbc15ef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:36.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:36.961+0000 7f0bc2e00640 1 -- 172.21.15.16:0/1782853471 >> 172.21.15.16:0/1782853471 conn(0x7f0bbc06ee40 msgr2=0x7f0bbc06d3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:03:36.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:36.961+0000 7f0bc2e00640 1 -- 172.21.15.16:0/1782853471 shutdown_connections 2024-03-31T15:03:36.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:36.961+0000 7f0bc2e00640 1 -- 172.21.15.16:0/1782853471 wait complete. 2024-03-31T15:03:36.998 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-31T15:03:37.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:37.141+0000 7f861e400640 1 Processor -- start 2024-03-31T15:03:37.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:37.141+0000 7f861e400640 1 -- start start 2024-03-31T15:03:37.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:37.145+0000 7f861e400640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f861806dbc0 0x7f861815f650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:37.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:37.145+0000 7f861e400640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f86180cb440 0x7f861806f920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:37.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:37.145+0000 7f861e400640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f86180cbe90 con 0x7f86180cb8c0 2024-03-31T15:03:37.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:37.145+0000 7f861e400640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f861815fb90 con 0x7f861806dbc0 2024-03-31T15:03:37.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:37.145+0000 7f861e400640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f861815f1c0 con 0x7f86180cb440 2024-03-31T15:03:37.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:37.145+0000 7f8617400640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f861806dbc0 0x7f861815f650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:37.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:37.145+0000 7f8617400640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f861806dbc0 0x7f861815f650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38786/0 (socket says 172.21.15.16:38786) 2024-03-31T15:03:37.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:37.145+0000 7f8617400640 1 -- 172.21.15.16:0/3266316251 learned_addr learned my addr 172.21.15.16:0/3266316251 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:03:37.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:37.145+0000 7f860ea00640 1 --2- 172.21.15.16:0/3266316251 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f86180cb440 0x7f861806f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:37.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:37.145+0000 7f8617400640 1 -- 172.21.15.16:0/3266316251 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f86180cb440 msgr2=0x7f861806f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:03:37.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:37.145+0000 7f8617400640 1 --2- 172.21.15.16:0/3266316251 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f86180cb440 0x7f861806f920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:37.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:37.145+0000 7f8617400640 1 -- 172.21.15.16:0/3266316251 >> v1:172.21.15.16:6789/0 conn(0x7f86180cb8c0 legacy=0x7f8618162180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:37.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:37.145+0000 7f8617400640 1 -- 172.21.15.16:0/3266316251 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86180cc170 con 0x7f861806dbc0 2024-03-31T15:03:37.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:37.145+0000 7f8616a00640 1 -- 172.21.15.16:0/3266316251 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1965545702 0 0) 0x7f86180cbe90 con 0x7f86180cb8c0 2024-03-31T15:03:37.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:37.145+0000 7f8617400640 1 --2- 172.21.15.16:0/3266316251 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f861806dbc0 0x7f861815f650 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f8608009870 tx=0x7f860801c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf6d93f36865306f server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:03:37.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:37.149+0000 7f8616a00640 1 -- 172.21.15.16:0/3266316251 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86080b1070 con 0x7f861806dbc0 2024-03-31T15:03:37.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:37.149+0000 7f8616a00640 1 -- 172.21.15.16:0/3266316251 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f86080ac410 con 0x7f861806dbc0 2024-03-31T15:03:37.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:37.149+0000 7f8616a00640 1 -- 172.21.15.16:0/3266316251 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86080ac7e0 con 0x7f861806dbc0 2024-03-31T15:03:37.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:37.149+0000 7f861e400640 1 -- 172.21.15.16:0/3266316251 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f861806dbc0 msgr2=0x7f861815f650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:37.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:37.149+0000 7f861e400640 1 --2- 172.21.15.16:0/3266316251 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f861806dbc0 0x7f861815f650 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f8608009870 tx=0x7f860801c4a0 comp rx=0 tx=0).stop 2024-03-31T15:03:37.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:37.149+0000 7f861e400640 1 -- 172.21.15.16:0/3266316251 shutdown_connections 2024-03-31T15:03:37.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:37.149+0000 7f861e400640 1 --2- 172.21.15.16:0/3266316251 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f86180cb440 0x7f861806f920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:37.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:37.149+0000 7f861e400640 1 --2- 172.21.15.16:0/3266316251 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f861806dbc0 0x7f861815f650 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:37.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:37.149+0000 7f861e400640 1 -- 172.21.15.16:0/3266316251 >> 172.21.15.16:0/3266316251 conn(0x7f8618067330 msgr2=0x7f8618067740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:03:37.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:37.149+0000 7f861e400640 1 -- 172.21.15.16:0/3266316251 shutdown_connections 2024-03-31T15:03:37.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:37.149+0000 7f861e400640 1 -- 172.21.15.16:0/3266316251 wait complete. 2024-03-31T15:03:37.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:37.149+0000 7f861e400640 1 Processor -- start 2024-03-31T15:03:37.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:37.149+0000 7f861e400640 1 -- start start 2024-03-31T15:03:37.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:37.149+0000 7f861e400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f861806dbc0 0x7f86181444a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:37.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:37.149+0000 7f861e400640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f86180cb440 0x7f86180cd910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:37.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:37.149+0000 7f861e400640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f86180cb8c0 0x7f8618159a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:37.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:37.149+0000 7f861e400640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f861806c950 con 0x7f861806dbc0 2024-03-31T15:03:37.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:37.149+0000 7f861e400640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f8618162ca0 con 0x7f86180cb440 2024-03-31T15:03:37.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:37.149+0000 7f861e400640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f861806cad0 con 0x7f86180cb8c0 2024-03-31T15:03:37.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:37.149+0000 7f8617400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f861806dbc0 0x7f86181444a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:37.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:37.149+0000 7f8617400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f861806dbc0 0x7f86181444a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38788/0 (socket says 172.21.15.16:38788) 2024-03-31T15:03:37.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:37.149+0000 7f8617400640 1 -- 172.21.15.16:0/3638043649 learned_addr learned my addr 172.21.15.16:0/3638043649 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:03:37.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:37.149+0000 7f8617e00640 1 --2- 172.21.15.16:0/3638043649 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f86180cb8c0 0x7f8618159a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:37.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:37.149+0000 7f860ea00640 1 --2- 172.21.15.16:0/3638043649 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f86180cb440 0x7f86180cd910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:37.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:37.153+0000 7f8617400640 1 -- 172.21.15.16:0/3638043649 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f86180cb8c0 msgr2=0x7f8618159a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:37.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:37.153+0000 7f8617400640 1 --2- 172.21.15.16:0/3638043649 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f86180cb8c0 0x7f8618159a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:37.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:37.153+0000 7f8617400640 1 -- 172.21.15.16:0/3638043649 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f86180cb440 msgr2=0x7f86180cd910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:37.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:37.153+0000 7f8617400640 1 --2- 172.21.15.16:0/3638043649 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f86180cb440 0x7f86180cd910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:37.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:37.153+0000 7f8617400640 1 -- 172.21.15.16:0/3638043649 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f861815be90 con 0x7f861806dbc0 2024-03-31T15:03:37.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:37.153+0000 7f860ea00640 1 --2- 172.21.15.16:0/3638043649 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f86180cb440 0x7f86180cd910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:03:37.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:37.153+0000 7f8617e00640 1 --2- 172.21.15.16:0/3638043649 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f86180cb8c0 0x7f8618159a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:03:37.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:37.153+0000 7f8617400640 1 --2- 172.21.15.16:0/3638043649 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f861806dbc0 0x7f86181444a0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f86080af040 tx=0x7f8608019240 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:03:37.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:37.153+0000 7f860fe00640 1 -- 172.21.15.16:0/3638043649 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86080b1040 con 0x7f861806dbc0 2024-03-31T15:03:37.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:37.153+0000 7f860fe00640 1 -- 172.21.15.16:0/3638043649 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f86080ba070 con 0x7f861806dbc0 2024-03-31T15:03:37.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:37.153+0000 7f860fe00640 1 -- 172.21.15.16:0/3638043649 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86080b5440 con 0x7f861806dbc0 2024-03-31T15:03:37.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:37.153+0000 7f861e400640 1 -- 172.21.15.16:0/3638043649 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f861815c170 con 0x7f861806dbc0 2024-03-31T15:03:37.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:37.153+0000 7f861e400640 1 -- 172.21.15.16:0/3638043649 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8618159230 con 0x7f861806dbc0 2024-03-31T15:03:37.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:37.153+0000 7f861e400640 1 -- 172.21.15.16:0/3638043649 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8618069950 con 0x7f861806dbc0 2024-03-31T15:03:37.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:37.153+0000 7f860fe00640 1 -- 172.21.15.16:0/3638043649 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f86080b55e0 con 0x7f861806dbc0 2024-03-31T15:03:37.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:37.153+0000 7f860fe00640 1 --2- 172.21.15.16:0/3638043649 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f85f4068380 0x7f85f406a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:37.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:37.153+0000 7f860fe00640 1 -- 172.21.15.16:0/3638043649 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f86081205b0 con 0x7f861806dbc0 2024-03-31T15:03:37.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:37.153+0000 7f860ea00640 1 --2- 172.21.15.16:0/3638043649 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f85f4068380 0x7f85f406a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:37.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:37.157+0000 7f860ea00640 1 --2- 172.21.15.16:0/3638043649 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f85f4068380 0x7f85f406a840 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f861815e4e0 tx=0x7f8600008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:03:37.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:37.161+0000 7f860fe00640 1 -- 172.21.15.16:0/3638043649 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f8608016150 con 0x7f861806dbc0 2024-03-31T15:03:37.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:37.325+0000 7f861e400640 1 -- 172.21.15.16:0/3638043649 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f861806faa0 con 0x7f861806dbc0 2024-03-31T15:03:37.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:37.325+0000 7f860fe00640 1 -- 172.21.15.16:0/3638043649 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) ==== 74+0+16546 (secure 0 0 0) 0x7f86080f0ad0 con 0x7f861806dbc0 2024-03-31T15:03:37.324 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:03:37.324 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":20,"fsid":"286eb5da-fb47-4550-b290-d9519c13b9a5","created":"2024-03-31T15:03:11.438952+0000","modified":"2024-03-31T15:03:36.349335+0000","last_up_change":"2024-03-31T15:03:22.885535+0000","last_in_change":"2024-03-31T15:03:13.941667+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-31T15:03:22.550787+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-31T15:03:34.488425+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-31T15:03:35.299116+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"92a66450-ef19-40d8-ac3d-d760b23b0bda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6801","nonce":932712244}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6803","nonce":932712244}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6807","nonce":932712244}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6805","nonce":932712244}]},"public_addr":"172.21.15.16:6801/932712244","cluster_addr":"172.21.15.16:6803/932712244","heartbeat_back_addr":"172.21.15.16:6807/932712244","heartbeat_front_addr":"172.21.15.16:6805/932712244","state":["exists","up"]},{"osd":1,"uuid":"da9d1f2b-1968-4959-b843-faa231988e92","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6810","nonce":1398737533}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6811","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6813","nonce":1398737533}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6821","nonce":1398737533}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6815","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6816","nonce":1398737533}]},"public_addr":"172.21.15.16:6810/1398737533","cluster_addr":"172.21.15.16:6813/1398737533","heartbeat_back_addr":"172.21.15.16:6821/1398737533","heartbeat_front_addr":"172.21.15.16:6816/1398737533","state":["exists","up"]},{"osd":2,"uuid":"01a73662-9a21-4b1f-bcbb-42b9636a4638","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6819","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6822","nonce":3018526646}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6825","nonce":3018526646}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6831","nonce":3018526646}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6827","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6829","nonce":3018526646}]},"public_addr":"172.21.15.16:6822/3018526646","cluster_addr":"172.21.15.16:6825/3018526646","heartbeat_back_addr":"172.21.15.16:6831/3018526646","heartbeat_front_addr":"172.21.15.16:6829/3018526646","state":["exists","up"]},{"osd":3,"uuid":"c7771fd3-e18e-4500-ac38-23cf10ebcd75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6809","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6812","nonce":3360352974}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6817","nonce":3360352974}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6828","nonce":3360352974}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6823","nonce":3360352974}]},"public_addr":"172.21.15.16:6812/3360352974","cluster_addr":"172.21.15.16:6817/3360352974","heartbeat_back_addr":"172.21.15.16:6828/3360352974","heartbeat_front_addr":"172.21.15.16:6823/3360352974","state":["exists","up"]},{"osd":4,"uuid":"88260730-1482-4fbd-a40f-cb1124e9df4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6801","nonce":1856844856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6803","nonce":1856844856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6807","nonce":1856844856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6805","nonce":1856844856}]},"public_addr":"172.21.15.89:6801/1856844856","cluster_addr":"172.21.15.89:6803/1856844856","heartbeat_back_addr":"172.21.15.89:6807/1856844856","heartbeat_front_addr":"172.21.15.89:6805/1856844856","state":["exists","up"]},{"osd":5,"uuid":"4cb2c00c-fae0-4dd8-9d27-ec4f0093ccf7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6814","nonce":50395527}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6817","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6820","nonce":50395527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6831","nonce":50395527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6823","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6826","nonce":50395527}]},"public_addr":"172.21.15.89:6814/50395527","cluster_addr":"172.21.15.89:6820/50395527","heartbeat_back_addr":"172.21.15.89:6831/50395527","heartbeat_front_addr":"172.21.15.89:6826/50395527","state":["exists","up"]},{"osd":6,"uuid":"81ce0c19-2554-4cae-acbe-66a05e9e9b9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6809","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6812","nonce":3470471752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6815","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6818","nonce":3470471752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6827","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6829","nonce":3470471752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6821","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6824","nonce":3470471752}]},"public_addr":"172.21.15.89:6812/3470471752","cluster_addr":"172.21.15.89:6818/3470471752","heartbeat_back_addr":"172.21.15.89:6829/3470471752","heartbeat_front_addr":"172.21.15.89:6824/3470471752","state":["exists","up"]},{"osd":7,"uuid":"85e4313b-d37d-442e-8042-153bf20a847d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6811","nonce":1796778630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6813","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6816","nonce":1796778630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6825","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6828","nonce":1796778630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6819","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6822","nonce":1796778630}]},"public_addr":"172.21.15.89:6811/1796778630","cluster_addr":"172.21.15.89:6816/1796778630","heartbeat_back_addr":"172.21.15.89:6828/1796778630","heartbeat_front_addr":"172.21.15.89:6822/1796778630","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.439088+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.529970+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.680512+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:21.044161+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.582111+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.536558+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.409522+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.590229+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-31T15:03:37.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:37.329+0000 7f861e400640 1 -- 172.21.15.16:0/3638043649 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f85f4068380 msgr2=0x7f85f406a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:37.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:37.329+0000 7f861e400640 1 --2- 172.21.15.16:0/3638043649 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f85f4068380 0x7f85f406a840 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f861815e4e0 tx=0x7f8600008040 comp rx=0 tx=0).stop 2024-03-31T15:03:37.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:37.329+0000 7f861e400640 1 -- 172.21.15.16:0/3638043649 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f861806dbc0 msgr2=0x7f86181444a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:37.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:37.329+0000 7f861e400640 1 --2- 172.21.15.16:0/3638043649 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f861806dbc0 0x7f86181444a0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f86080af040 tx=0x7f8608019240 comp rx=0 tx=0).stop 2024-03-31T15:03:37.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:37.329+0000 7f861e400640 1 -- 172.21.15.16:0/3638043649 shutdown_connections 2024-03-31T15:03:37.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:37.329+0000 7f861e400640 1 --2- 172.21.15.16:0/3638043649 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f85f4068380 0x7f85f406a840 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:37.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:37.329+0000 7f861e400640 1 --2- 172.21.15.16:0/3638043649 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f86180cb8c0 0x7f8618159a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:37.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:37.329+0000 7f861e400640 1 --2- 172.21.15.16:0/3638043649 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f86180cb440 0x7f86180cd910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:37.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:37.329+0000 7f861e400640 1 --2- 172.21.15.16:0/3638043649 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f861806dbc0 0x7f86181444a0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:37.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:37.329+0000 7f861e400640 1 -- 172.21.15.16:0/3638043649 >> 172.21.15.16:0/3638043649 conn(0x7f8618067330 msgr2=0x7f861815ca50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:03:37.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:37.329+0000 7f861e400640 1 -- 172.21.15.16:0/3638043649 shutdown_connections 2024-03-31T15:03:37.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:37.329+0000 7f861e400640 1 -- 172.21.15.16:0/3638043649 wait complete. 2024-03-31T15:03:37.345 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-31T15:03:37.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:37.489+0000 7f6fd2e00640 1 Processor -- start 2024-03-31T15:03:37.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:37.489+0000 7f6fd2e00640 1 -- start start 2024-03-31T15:03:37.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:37.489+0000 7f6fd2e00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f6fcc15c1d0 0x7f6fcc0b44e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:37.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:37.489+0000 7f6fd2e00640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6fcc0cd980 con 0x7f6fcc0cd4f0 2024-03-31T15:03:37.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:37.489+0000 7f6fd2e00640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6fcc0cdc60 con 0x7f6fcc15fd40 2024-03-31T15:03:37.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:37.489+0000 7f6fd2e00640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f6fcc0cfaa0 con 0x7f6fcc15c1d0 2024-03-31T15:03:37.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:37.489+0000 7f6fd1a00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f6fcc15c1d0 0x7f6fcc0b44e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:37.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:37.489+0000 7f6fd1000640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f6fcc0cd4f0 0x7f6fcc06f5d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:33920/0 (socket says 172.21.15.16:33920) 2024-03-31T15:03:37.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:37.489+0000 7f6fd1000640 1 -- 172.21.15.16:0/3863506050 learned_addr learned my addr 172.21.15.16:0/3863506050 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:03:37.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:37.489+0000 7f6fd1a00640 1 -- 172.21.15.16:0/3863506050 >> v1:172.21.15.89:6789/0 conn(0x7f6fcc15fd40 legacy=0x7f6fcc160150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:37.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:37.489+0000 7f6fd1a00640 1 -- 172.21.15.16:0/3863506050 >> v1:172.21.15.16:6789/0 conn(0x7f6fcc0cd4f0 legacy=0x7f6fcc06f5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:37.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:37.489+0000 7f6fd1a00640 1 -- 172.21.15.16:0/3863506050 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6fcc0cfcf0 con 0x7f6fcc15c1d0 2024-03-31T15:03:37.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:37.489+0000 7f6fcbe00640 1 -- 172.21.15.16:0/3863506050 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 967538964 0 0) 0x7f6fcc0cd980 con 0x7f6fcc0cd4f0 2024-03-31T15:03:37.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:37.493+0000 7f6fd1a00640 1 --2- 172.21.15.16:0/3863506050 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6fcc15c1d0 0x7f6fcc0b44e0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f6fc00097c0 tx=0x7f6fc001c470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=efb2f668ddd6bd5c server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:03:37.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:37.493+0000 7f6fcbe00640 1 -- 172.21.15.16:0/3863506050 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fc00ab610 con 0x7f6fcc15c1d0 2024-03-31T15:03:37.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:37.493+0000 7f6fcbe00640 1 -- 172.21.15.16:0/3863506050 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6fc00b0070 con 0x7f6fcc15c1d0 2024-03-31T15:03:37.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:37.493+0000 7f6fcbe00640 1 -- 172.21.15.16:0/3863506050 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fc00ab9d0 con 0x7f6fcc15c1d0 2024-03-31T15:03:37.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:37.493+0000 7f6fd2e00640 1 -- 172.21.15.16:0/3863506050 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6fcc15c1d0 msgr2=0x7f6fcc0b44e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:37.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:37.493+0000 7f6fd2e00640 1 --2- 172.21.15.16:0/3863506050 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6fcc15c1d0 0x7f6fcc0b44e0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f6fc00097c0 tx=0x7f6fc001c470 comp rx=0 tx=0).stop 2024-03-31T15:03:37.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:37.493+0000 7f6fd2e00640 1 -- 172.21.15.16:0/3863506050 shutdown_connections 2024-03-31T15:03:37.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:37.493+0000 7f6fd2e00640 1 --2- 172.21.15.16:0/3863506050 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6fcc15c1d0 0x7f6fcc0b44e0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:37.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:37.493+0000 7f6fd2e00640 1 -- 172.21.15.16:0/3863506050 >> 172.21.15.16:0/3863506050 conn(0x7f6fcc0b39c0 msgr2=0x7f6fcc0b3dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:03:37.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:37.493+0000 7f6fd2e00640 1 -- 172.21.15.16:0/3863506050 shutdown_connections 2024-03-31T15:03:37.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:37.493+0000 7f6fd2e00640 1 -- 172.21.15.16:0/3863506050 wait complete. 2024-03-31T15:03:37.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:37.497+0000 7f6fd2e00640 1 Processor -- start 2024-03-31T15:03:37.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:37.497+0000 7f6fd2e00640 1 -- start start 2024-03-31T15:03:37.494 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:37.497+0000 7f6fd2e00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6fcc15c1d0 0x7f6fcc15c840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:37.494 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:37.497+0000 7f6fd2e00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f6fcc15fd40 0x7f6fcc14f980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:37.494 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:37.497+0000 7f6fd2e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6fcc156560 0x7f6fcc156a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:37.494 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:37.497+0000 7f6fd2e00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f6fcc06fce0 con 0x7f6fcc156560 2024-03-31T15:03:37.494 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:37.497+0000 7f6fd2e00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f6fcc16a090 con 0x7f6fcc15c1d0 2024-03-31T15:03:37.494 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:37.497+0000 7f6fd2e00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f6fcc14fec0 con 0x7f6fcc15fd40 2024-03-31T15:03:37.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:37.497+0000 7f6fd2400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6fcc156560 0x7f6fcc156a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:37.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:37.497+0000 7f6fd2400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6fcc156560 0x7f6fcc156a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38800/0 (socket says 172.21.15.16:38800) 2024-03-31T15:03:37.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:37.497+0000 7f6fd2400640 1 -- 172.21.15.16:0/3629867710 learned_addr learned my addr 172.21.15.16:0/3629867710 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:03:37.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:37.497+0000 7f6fd1a00640 1 --2- 172.21.15.16:0/3629867710 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6fcc15c1d0 0x7f6fcc15c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:37.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:37.497+0000 7f6fd1000640 1 --2- 172.21.15.16:0/3629867710 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f6fcc15fd40 0x7f6fcc14f980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:37.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:37.497+0000 7f6fd2400640 1 -- 172.21.15.16:0/3629867710 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f6fcc15fd40 msgr2=0x7f6fcc14f980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:37.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:37.497+0000 7f6fd2400640 1 --2- 172.21.15.16:0/3629867710 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f6fcc15fd40 0x7f6fcc14f980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:37.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:37.497+0000 7f6fd2400640 1 -- 172.21.15.16:0/3629867710 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6fcc15c1d0 msgr2=0x7f6fcc15c840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:37.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:37.497+0000 7f6fd2400640 1 --2- 172.21.15.16:0/3629867710 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6fcc15c1d0 0x7f6fcc15c840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:37.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:37.497+0000 7f6fd2400640 1 -- 172.21.15.16:0/3629867710 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6fcc157fc0 con 0x7f6fcc156560 2024-03-31T15:03:37.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:37.497+0000 7f6fd2400640 1 --2- 172.21.15.16:0/3629867710 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6fcc156560 0x7f6fcc156a10 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f6fc400c730 tx=0x7f6fc400cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:03:37.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:37.497+0000 7f6fc9600640 1 -- 172.21.15.16:0/3629867710 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fc4003b80 con 0x7f6fcc156560 2024-03-31T15:03:37.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:37.497+0000 7f6fc9600640 1 -- 172.21.15.16:0/3629867710 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6fc4003d20 con 0x7f6fcc156560 2024-03-31T15:03:37.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:37.497+0000 7f6fd2e00640 1 -- 172.21.15.16:0/3629867710 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6fcc1582a0 con 0x7f6fcc156560 2024-03-31T15:03:37.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:37.497+0000 7f6fc9600640 1 -- 172.21.15.16:0/3629867710 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fc400aa30 con 0x7f6fcc156560 2024-03-31T15:03:37.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:37.497+0000 7f6fd2e00640 1 -- 172.21.15.16:0/3629867710 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6fcc156f50 con 0x7f6fcc156560 2024-03-31T15:03:37.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:37.501+0000 7f6fd2e00640 1 -- 172.21.15.16:0/3629867710 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6fcc04f8c0 con 0x7f6fcc156560 2024-03-31T15:03:37.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:37.501+0000 7f6fc9600640 1 -- 172.21.15.16:0/3629867710 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f6fc400abd0 con 0x7f6fcc156560 2024-03-31T15:03:37.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:37.501+0000 7f6fc9600640 1 --2- 172.21.15.16:0/3629867710 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f6fac068380 0x7f6fac06a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:37.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:37.501+0000 7f6fc9600640 1 -- 172.21.15.16:0/3629867710 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f6fc4086730 con 0x7f6fcc156560 2024-03-31T15:03:37.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:37.501+0000 7f6fd1a00640 1 --2- 172.21.15.16:0/3629867710 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f6fac068380 0x7f6fac06a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:37.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:37.501+0000 7f6fd1a00640 1 --2- 172.21.15.16:0/3629867710 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f6fac068380 0x7f6fac06a840 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f6fc00af000 tx=0x7f6fc00a7000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:03:37.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:37.505+0000 7f6fc9600640 1 -- 172.21.15.16:0/3629867710 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f6fc4056690 con 0x7f6fcc156560 2024-03-31T15:03:37.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:37.669+0000 7f6fd2e00640 1 -- 172.21.15.16:0/3629867710 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f6fcc068ae0 con 0x7f6fcc156560 2024-03-31T15:03:37.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:37.669+0000 7f6fc9600640 1 -- 172.21.15.16:0/3629867710 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) ==== 74+0+16546 (secure 0 0 0) 0x7f6fc405a990 con 0x7f6fcc156560 2024-03-31T15:03:37.668 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:03:37.668 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":20,"fsid":"286eb5da-fb47-4550-b290-d9519c13b9a5","created":"2024-03-31T15:03:11.438952+0000","modified":"2024-03-31T15:03:36.349335+0000","last_up_change":"2024-03-31T15:03:22.885535+0000","last_in_change":"2024-03-31T15:03:13.941667+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-31T15:03:22.550787+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-31T15:03:34.488425+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-31T15:03:35.299116+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"92a66450-ef19-40d8-ac3d-d760b23b0bda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6801","nonce":932712244}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6803","nonce":932712244}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6807","nonce":932712244}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6805","nonce":932712244}]},"public_addr":"172.21.15.16:6801/932712244","cluster_addr":"172.21.15.16:6803/932712244","heartbeat_back_addr":"172.21.15.16:6807/932712244","heartbeat_front_addr":"172.21.15.16:6805/932712244","state":["exists","up"]},{"osd":1,"uuid":"da9d1f2b-1968-4959-b843-faa231988e92","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6810","nonce":1398737533}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6811","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6813","nonce":1398737533}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6821","nonce":1398737533}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6815","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6816","nonce":1398737533}]},"public_addr":"172.21.15.16:6810/1398737533","cluster_addr":"172.21.15.16:6813/1398737533","heartbeat_back_addr":"172.21.15.16:6821/1398737533","heartbeat_front_addr":"172.21.15.16:6816/1398737533","state":["exists","up"]},{"osd":2,"uuid":"01a73662-9a21-4b1f-bcbb-42b9636a4638","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6819","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6822","nonce":3018526646}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6825","nonce":3018526646}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6831","nonce":3018526646}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6827","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6829","nonce":3018526646}]},"public_addr":"172.21.15.16:6822/3018526646","cluster_addr":"172.21.15.16:6825/3018526646","heartbeat_back_addr":"172.21.15.16:6831/3018526646","heartbeat_front_addr":"172.21.15.16:6829/3018526646","state":["exists","up"]},{"osd":3,"uuid":"c7771fd3-e18e-4500-ac38-23cf10ebcd75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6809","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6812","nonce":3360352974}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6817","nonce":3360352974}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6828","nonce":3360352974}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6823","nonce":3360352974}]},"public_addr":"172.21.15.16:6812/3360352974","cluster_addr":"172.21.15.16:6817/3360352974","heartbeat_back_addr":"172.21.15.16:6828/3360352974","heartbeat_front_addr":"172.21.15.16:6823/3360352974","state":["exists","up"]},{"osd":4,"uuid":"88260730-1482-4fbd-a40f-cb1124e9df4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6801","nonce":1856844856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6803","nonce":1856844856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6807","nonce":1856844856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6805","nonce":1856844856}]},"public_addr":"172.21.15.89:6801/1856844856","cluster_addr":"172.21.15.89:6803/1856844856","heartbeat_back_addr":"172.21.15.89:6807/1856844856","heartbeat_front_addr":"172.21.15.89:6805/1856844856","state":["exists","up"]},{"osd":5,"uuid":"4cb2c00c-fae0-4dd8-9d27-ec4f0093ccf7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6814","nonce":50395527}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6817","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6820","nonce":50395527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6831","nonce":50395527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6823","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6826","nonce":50395527}]},"public_addr":"172.21.15.89:6814/50395527","cluster_addr":"172.21.15.89:6820/50395527","heartbeat_back_addr":"172.21.15.89:6831/50395527","heartbeat_front_addr":"172.21.15.89:6826/50395527","state":["exists","up"]},{"osd":6,"uuid":"81ce0c19-2554-4cae-acbe-66a05e9e9b9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6809","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6812","nonce":3470471752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6815","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6818","nonce":3470471752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6827","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6829","nonce":3470471752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6821","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6824","nonce":3470471752}]},"public_addr":"172.21.15.89:6812/3470471752","cluster_addr":"172.21.15.89:6818/3470471752","heartbeat_back_addr":"172.21.15.89:6829/3470471752","heartbeat_front_addr":"172.21.15.89:6824/3470471752","state":["exists","up"]},{"osd":7,"uuid":"85e4313b-d37d-442e-8042-153bf20a847d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6811","nonce":1796778630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6813","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6816","nonce":1796778630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6825","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6828","nonce":1796778630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6819","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6822","nonce":1796778630}]},"public_addr":"172.21.15.89:6811/1796778630","cluster_addr":"172.21.15.89:6816/1796778630","heartbeat_back_addr":"172.21.15.89:6828/1796778630","heartbeat_front_addr":"172.21.15.89:6822/1796778630","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.439088+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.529970+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.680512+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:21.044161+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.582111+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.536558+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.409522+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.590229+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-31T15:03:37.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:37.673+0000 7f6fd2e00640 1 -- 172.21.15.16:0/3629867710 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f6fac068380 msgr2=0x7f6fac06a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:37.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:37.673+0000 7f6fd2e00640 1 --2- 172.21.15.16:0/3629867710 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f6fac068380 0x7f6fac06a840 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f6fc00af000 tx=0x7f6fc00a7000 comp rx=0 tx=0).stop 2024-03-31T15:03:37.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:37.673+0000 7f6fd2e00640 1 -- 172.21.15.16:0/3629867710 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6fcc156560 msgr2=0x7f6fcc156a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:37.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:37.673+0000 7f6fd2e00640 1 --2- 172.21.15.16:0/3629867710 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6fcc156560 0x7f6fcc156a10 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f6fc400c730 tx=0x7f6fc400cbf0 comp rx=0 tx=0).stop 2024-03-31T15:03:37.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:37.673+0000 7f6fd2e00640 1 -- 172.21.15.16:0/3629867710 shutdown_connections 2024-03-31T15:03:37.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:37.673+0000 7f6fd2e00640 1 --2- 172.21.15.16:0/3629867710 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f6fac068380 0x7f6fac06a840 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:37.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:37.673+0000 7f6fd2e00640 1 --2- 172.21.15.16:0/3629867710 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6fcc156560 0x7f6fcc156a10 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:37.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:37.673+0000 7f6fd2e00640 1 --2- 172.21.15.16:0/3629867710 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f6fcc15fd40 0x7f6fcc14f980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:37.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:37.673+0000 7f6fd2e00640 1 --2- 172.21.15.16:0/3629867710 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6fcc15c1d0 0x7f6fcc15c840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:37.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:37.673+0000 7f6fd2e00640 1 -- 172.21.15.16:0/3629867710 >> 172.21.15.16:0/3629867710 conn(0x7f6fcc0b39c0 msgr2=0x7f6fcc0aedf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:03:37.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:37.673+0000 7f6fd2e00640 1 -- 172.21.15.16:0/3629867710 shutdown_connections 2024-03-31T15:03:37.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:37.673+0000 7f6fd2e00640 1 -- 172.21.15.16:0/3629867710 wait complete. 2024-03-31T15:03:37.689 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-31T15:03:37.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:37.837+0000 7f340ca00640 1 Processor -- start 2024-03-31T15:03:37.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:37.837+0000 7f340ca00640 1 -- start start 2024-03-31T15:03:37.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:37.837+0000 7f340ca00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f34080d0a60 0x7f34080cea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:37.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:37.837+0000 7f340ca00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f34080cef80 0x7f34081419c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:37.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:37.837+0000 7f340ca00640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f340815ca70 con 0x7f340806f920 2024-03-31T15:03:37.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:37.837+0000 7f340ca00640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f340813f670 con 0x7f34080d0a60 2024-03-31T15:03:37.835 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:37.837+0000 7f340ca00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f340815cd50 con 0x7f34080cef80 2024-03-31T15:03:37.835 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:37.837+0000 7f3405400640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f34080d0a60 0x7f34080cea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:37.835 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:37.837+0000 7f3406800640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f34080cef80 0x7f34081419c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:37.835 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:37.837+0000 7f3406800640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f34080cef80 0x7f34081419c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3301/0 says I am v2:172.21.15.16:41350/0 (socket says 172.21.15.16:41350) 2024-03-31T15:03:37.835 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:37.837+0000 7f3406800640 1 -- 172.21.15.16:0/2641766917 learned_addr learned my addr 172.21.15.16:0/2641766917 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:03:37.835 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:37.837+0000 7f3405e00640 1 --1- 172.21.15.16:0/2641766917 >> v1:172.21.15.89:6789/0 conn(0x7f340806f920 0x7f340806fd00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.16:49104/0 (socket says 172.21.15.16:49104) 2024-03-31T15:03:37.836 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:37.837+0000 7f3405400640 1 -- 172.21.15.16:0/2641766917 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f34080cef80 msgr2=0x7f34081419c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:03:37.836 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:37.837+0000 7f3405400640 1 --2- 172.21.15.16:0/2641766917 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f34080cef80 0x7f34081419c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:37.836 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:37.837+0000 7f3405400640 1 -- 172.21.15.16:0/2641766917 >> v1:172.21.15.89:6789/0 conn(0x7f340806f920 legacy=0x7f340806fd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:37.836 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:37.837+0000 7f3405400640 1 -- 172.21.15.16:0/2641766917 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f340815d030 con 0x7f34080d0a60 2024-03-31T15:03:37.836 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:37.837+0000 7f3404a00640 1 -- 172.21.15.16:0/2641766917 <== mon.1 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 85449008 0 0) 0x7f340815ca70 con 0x7f340806f920 2024-03-31T15:03:37.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:37.841+0000 7f3405400640 1 --2- 172.21.15.16:0/2641766917 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f34080d0a60 0x7f34080cea40 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f33f40097c0 tx=0x7f33f401c4a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2a68cd786370bd22 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:03:37.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:37.841+0000 7f3404a00640 1 -- 172.21.15.16:0/2641766917 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33f40b0070 con 0x7f34080d0a60 2024-03-31T15:03:37.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:37.841+0000 7f3404a00640 1 -- 172.21.15.16:0/2641766917 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f33f40ab600 con 0x7f34080d0a60 2024-03-31T15:03:37.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:37.841+0000 7f3404a00640 1 -- 172.21.15.16:0/2641766917 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33f40ab9d0 con 0x7f34080d0a60 2024-03-31T15:03:37.839 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:37.841+0000 7f340ca00640 1 -- 172.21.15.16:0/2641766917 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f34080d0a60 msgr2=0x7f34080cea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:37.839 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:37.841+0000 7f340ca00640 1 --2- 172.21.15.16:0/2641766917 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f34080d0a60 0x7f34080cea40 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f33f40097c0 tx=0x7f33f401c4a0 comp rx=0 tx=0).stop 2024-03-31T15:03:37.839 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:37.841+0000 7f340ca00640 1 -- 172.21.15.16:0/2641766917 shutdown_connections 2024-03-31T15:03:37.839 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:37.841+0000 7f340ca00640 1 --2- 172.21.15.16:0/2641766917 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f34080d0a60 0x7f34080cea40 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:37.839 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:37.841+0000 7f340ca00640 1 --2- 172.21.15.16:0/2641766917 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f34080cef80 0x7f34081419c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:37.839 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:37.841+0000 7f340ca00640 1 -- 172.21.15.16:0/2641766917 >> 172.21.15.16:0/2641766917 conn(0x7f3408067930 msgr2=0x7f34080b3ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:03:37.839 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:37.841+0000 7f340ca00640 1 -- 172.21.15.16:0/2641766917 shutdown_connections 2024-03-31T15:03:37.839 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:37.841+0000 7f340ca00640 1 -- 172.21.15.16:0/2641766917 wait complete. 2024-03-31T15:03:37.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:37.845+0000 7f340ca00640 1 Processor -- start 2024-03-31T15:03:37.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:37.845+0000 7f340ca00640 1 -- start start 2024-03-31T15:03:37.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:37.845+0000 7f340ca00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f340806f920 0x7f340815e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:37.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:37.845+0000 7f340ca00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f34080cef80 0x7f340813e7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:37.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:37.845+0000 7f340ca00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f34080d0a60 0x7f3408140880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:37.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:37.845+0000 7f340ca00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f3408169230 con 0x7f34080d0a60 2024-03-31T15:03:37.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:37.845+0000 7f340ca00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f34081690b0 con 0x7f34080cef80 2024-03-31T15:03:37.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:37.845+0000 7f340ca00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f34081693b0 con 0x7f340806f920 2024-03-31T15:03:37.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:37.845+0000 7f3406800640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f34080d0a60 0x7f3408140880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:37.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:37.845+0000 7f3405400640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f34080cef80 0x7f340813e7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:37.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:37.845+0000 7f3406800640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f34080d0a60 0x7f3408140880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38816/0 (socket says 172.21.15.16:38816) 2024-03-31T15:03:37.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:37.845+0000 7f3405400640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f34080cef80 0x7f340813e7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.16:35838/0 (socket says 172.21.15.16:35838) 2024-03-31T15:03:37.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:37.845+0000 7f3405e00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f340806f920 0x7f340815e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:37.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:37.845+0000 7f3406800640 1 -- 172.21.15.16:0/894858029 learned_addr learned my addr 172.21.15.16:0/894858029 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:03:37.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:37.845+0000 7f3406800640 1 -- 172.21.15.16:0/894858029 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f340806f920 msgr2=0x7f340815e8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:37.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:37.845+0000 7f3406800640 1 --2- 172.21.15.16:0/894858029 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f340806f920 0x7f340815e8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:37.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:37.845+0000 7f3406800640 1 -- 172.21.15.16:0/894858029 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f34080cef80 msgr2=0x7f340813e7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:37.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:37.845+0000 7f3406800640 1 --2- 172.21.15.16:0/894858029 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f34080cef80 0x7f340813e7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:37.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:37.845+0000 7f3406800640 1 -- 172.21.15.16:0/894858029 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f340815d850 con 0x7f34080d0a60 2024-03-31T15:03:37.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:37.845+0000 7f3405400640 1 --2- 172.21.15.16:0/894858029 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f34080cef80 0x7f340813e7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:03:37.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:37.845+0000 7f3406800640 1 --2- 172.21.15.16:0/894858029 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f34080d0a60 0x7f3408140880 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f3400009d90 tx=0x7f340000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:03:37.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:37.845+0000 7f33fe000640 1 -- 172.21.15.16:0/894858029 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f340000cd20 con 0x7f34080d0a60 2024-03-31T15:03:37.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:37.845+0000 7f33fe000640 1 -- 172.21.15.16:0/894858029 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f34000039a0 con 0x7f34080d0a60 2024-03-31T15:03:37.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:37.845+0000 7f340ca00640 1 -- 172.21.15.16:0/894858029 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f340815db30 con 0x7f34080d0a60 2024-03-31T15:03:37.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:37.845+0000 7f33fe000640 1 -- 172.21.15.16:0/894858029 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3400003d70 con 0x7f34080d0a60 2024-03-31T15:03:37.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:37.845+0000 7f340ca00640 1 -- 172.21.15.16:0/894858029 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f340815e0f0 con 0x7f34080d0a60 2024-03-31T15:03:37.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:37.845+0000 7f340ca00640 1 -- 172.21.15.16:0/894858029 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3408069950 con 0x7f34080d0a60 2024-03-31T15:03:37.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:37.849+0000 7f33fe000640 1 -- 172.21.15.16:0/894858029 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f3400012050 con 0x7f34080d0a60 2024-03-31T15:03:37.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:37.849+0000 7f33fe000640 1 --2- 172.21.15.16:0/894858029 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f33e8068380 0x7f33e806a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:37.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:37.849+0000 7f33fe000640 1 -- 172.21.15.16:0/894858029 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f3400086dd0 con 0x7f34080d0a60 2024-03-31T15:03:37.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:37.849+0000 7f3405e00640 1 --2- 172.21.15.16:0/894858029 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f33e8068380 0x7f33e806a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:37.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:37.849+0000 7f3405e00640 1 --2- 172.21.15.16:0/894858029 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f33e8068380 0x7f33e806a840 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f33f8007fd0 tx=0x7f33f8007d60 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:03:37.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:37.853+0000 7f33fe000640 1 -- 172.21.15.16:0/894858029 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f3400010040 con 0x7f34080d0a60 2024-03-31T15:03:38.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:38.053+0000 7f340ca00640 1 -- 172.21.15.16:0/894858029 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f34080b4a20 con 0x7f34080d0a60 2024-03-31T15:03:38.969 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:38.973+0000 7f33fe000640 1 -- 172.21.15.16:0/894858029 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v7) ==== 118+0+0 (secure 0 0 0) 0x7f3400056d30 con 0x7f34080d0a60 2024-03-31T15:03:38.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:38.973+0000 7f340ca00640 1 -- 172.21.15.16:0/894858029 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f33e8068380 msgr2=0x7f33e806a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:38.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:38.973+0000 7f340ca00640 1 --2- 172.21.15.16:0/894858029 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f33e8068380 0x7f33e806a840 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f33f8007fd0 tx=0x7f33f8007d60 comp rx=0 tx=0).stop 2024-03-31T15:03:38.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:38.973+0000 7f340ca00640 1 -- 172.21.15.16:0/894858029 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f34080d0a60 msgr2=0x7f3408140880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:38.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:38.973+0000 7f340ca00640 1 --2- 172.21.15.16:0/894858029 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f34080d0a60 0x7f3408140880 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f3400009d90 tx=0x7f340000c490 comp rx=0 tx=0).stop 2024-03-31T15:03:38.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:38.973+0000 7f340ca00640 1 -- 172.21.15.16:0/894858029 shutdown_connections 2024-03-31T15:03:38.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:38.973+0000 7f340ca00640 1 --2- 172.21.15.16:0/894858029 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f33e8068380 0x7f33e806a840 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:38.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:38.973+0000 7f340ca00640 1 --2- 172.21.15.16:0/894858029 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f34080d0a60 0x7f3408140880 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:38.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:38.973+0000 7f340ca00640 1 --2- 172.21.15.16:0/894858029 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f34080cef80 0x7f340813e7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:38.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:38.973+0000 7f340ca00640 1 --2- 172.21.15.16:0/894858029 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f340806f920 0x7f340815e8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:38.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:38.973+0000 7f340ca00640 1 -- 172.21.15.16:0/894858029 >> 172.21.15.16:0/894858029 conn(0x7f3408067930 msgr2=0x7f340815ac50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:03:38.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:38.973+0000 7f340ca00640 1 -- 172.21.15.16:0/894858029 shutdown_connections 2024-03-31T15:03:38.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:38.973+0000 7f340ca00640 1 -- 172.21.15.16:0/894858029 wait complete. 2024-03-31T15:03:38.995 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-03-31T15:03:38.995 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-31T15:03:39.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:39.145+0000 7f07e0200640 1 Processor -- start 2024-03-31T15:03:39.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:39.145+0000 7f07e0200640 1 -- start start 2024-03-31T15:03:39.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:39.149+0000 7f07e0200640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f07d806d4d0 0x7f07d806d930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:39.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:39.149+0000 7f07e0200640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f07d8071550 0x7f07d8071980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:39.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:39.149+0000 7f07e0200640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f07d806de70 con 0x7f07d81567e0 2024-03-31T15:03:39.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:39.149+0000 7f07e0200640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f07d815c7e0 con 0x7f07d806d4d0 2024-03-31T15:03:39.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:39.149+0000 7f07e0200640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f07d80736b0 con 0x7f07d8071550 2024-03-31T15:03:39.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:39.149+0000 7f07dce00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f07d806d4d0 0x7f07d806d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:39.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:39.149+0000 7f07dce00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f07d806d4d0 0x7f07d806d930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38826/0 (socket says 172.21.15.16:38826) 2024-03-31T15:03:39.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:39.149+0000 7f07dce00640 1 -- 172.21.15.16:0/378674668 learned_addr learned my addr 172.21.15.16:0/378674668 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:03:39.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:39.149+0000 7f07dd800640 1 --1- 172.21.15.16:0/378674668 >> v1:172.21.15.89:6789/0 conn(0x7f07d81567e0 0x7f07d8072c80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.16:49120/0 (socket says 172.21.15.16:49120) 2024-03-31T15:03:39.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:39.149+0000 7f07de200640 1 --2- 172.21.15.16:0/378674668 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f07d8071550 0x7f07d8071980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:39.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:39.149+0000 7f07dce00640 1 -- 172.21.15.16:0/378674668 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f07d8071550 msgr2=0x7f07d8071980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:03:39.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:39.149+0000 7f07dce00640 1 --2- 172.21.15.16:0/378674668 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f07d8071550 0x7f07d8071980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:39.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:39.149+0000 7f07dce00640 1 -- 172.21.15.16:0/378674668 >> v1:172.21.15.89:6789/0 conn(0x7f07d81567e0 legacy=0x7f07d8072c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:39.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:39.149+0000 7f07dce00640 1 -- 172.21.15.16:0/378674668 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07d815caa0 con 0x7f07d806d4d0 2024-03-31T15:03:39.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:39.149+0000 7f07d7e00640 1 -- 172.21.15.16:0/378674668 <== mon.1 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1909406306 0 0) 0x7f07d806de70 con 0x7f07d81567e0 2024-03-31T15:03:39.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:39.153+0000 7f07dce00640 1 --2- 172.21.15.16:0/378674668 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f07d806d4d0 0x7f07d806d930 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f07c4009870 tx=0x7f07c401cfc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c904ff97aa62659 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:03:39.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:39.153+0000 7f07d7e00640 1 -- 172.21.15.16:0/378674668 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07c40a7070 con 0x7f07d806d4d0 2024-03-31T15:03:39.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:39.153+0000 7f07d7e00640 1 -- 172.21.15.16:0/378674668 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f07c40b0070 con 0x7f07d806d4d0 2024-03-31T15:03:39.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:39.153+0000 7f07d7e00640 1 -- 172.21.15.16:0/378674668 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07c40ab890 con 0x7f07d806d4d0 2024-03-31T15:03:39.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:39.153+0000 7f07e0200640 1 -- 172.21.15.16:0/378674668 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f07d806d4d0 msgr2=0x7f07d806d930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:39.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:39.153+0000 7f07e0200640 1 --2- 172.21.15.16:0/378674668 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f07d806d4d0 0x7f07d806d930 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f07c4009870 tx=0x7f07c401cfc0 comp rx=0 tx=0).stop 2024-03-31T15:03:39.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:39.153+0000 7f07e0200640 1 -- 172.21.15.16:0/378674668 shutdown_connections 2024-03-31T15:03:39.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:39.153+0000 7f07e0200640 1 --2- 172.21.15.16:0/378674668 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f07d8071550 0x7f07d8071980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:39.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:39.153+0000 7f07e0200640 1 --2- 172.21.15.16:0/378674668 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f07d806d4d0 0x7f07d806d930 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:39.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:39.153+0000 7f07e0200640 1 -- 172.21.15.16:0/378674668 >> 172.21.15.16:0/378674668 conn(0x7f07d806ee40 msgr2=0x7f07d806b620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:03:39.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:39.153+0000 7f07e0200640 1 -- 172.21.15.16:0/378674668 shutdown_connections 2024-03-31T15:03:39.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:39.153+0000 7f07e0200640 1 -- 172.21.15.16:0/378674668 wait complete. 2024-03-31T15:03:39.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:39.153+0000 7f07e0200640 1 Processor -- start 2024-03-31T15:03:39.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:39.153+0000 7f07e0200640 1 -- start start 2024-03-31T15:03:39.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:39.153+0000 7f07e0200640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f07d806d4d0 0x7f07d8144f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:39.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:39.153+0000 7f07e0200640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f07d8071550 0x7f07d8153290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:39.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:39.157+0000 7f07e0200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f07d81567e0 0x7f07d815e3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:39.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:39.157+0000 7f07e0200640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f07d80afbb0 con 0x7f07d81567e0 2024-03-31T15:03:39.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:39.157+0000 7f07e0200640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f07d80afa30 con 0x7f07d8071550 2024-03-31T15:03:39.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:39.157+0000 7f07e0200640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f07d80722d0 con 0x7f07d806d4d0 2024-03-31T15:03:39.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:39.157+0000 7f07de200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f07d81567e0 0x7f07d815e3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:39.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:39.157+0000 7f07de200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f07d81567e0 0x7f07d815e3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38838/0 (socket says 172.21.15.16:38838) 2024-03-31T15:03:39.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:39.157+0000 7f07dd800640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f07d806d4d0 0x7f07d8144f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:39.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:39.157+0000 7f07de200640 1 -- 172.21.15.16:0/4268349902 learned_addr learned my addr 172.21.15.16:0/4268349902 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:03:39.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:39.157+0000 7f07dce00640 1 --2- 172.21.15.16:0/4268349902 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f07d8071550 0x7f07d8153290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:39.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:39.157+0000 7f07de200640 1 -- 172.21.15.16:0/4268349902 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f07d806d4d0 msgr2=0x7f07d8144f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:39.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:39.157+0000 7f07de200640 1 --2- 172.21.15.16:0/4268349902 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f07d806d4d0 0x7f07d8144f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:39.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:39.157+0000 7f07de200640 1 -- 172.21.15.16:0/4268349902 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f07d8071550 msgr2=0x7f07d8153290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:39.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:39.157+0000 7f07de200640 1 --2- 172.21.15.16:0/4268349902 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f07d8071550 0x7f07d8153290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:39.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:39.157+0000 7f07de200640 1 -- 172.21.15.16:0/4268349902 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07d80b4e00 con 0x7f07d81567e0 2024-03-31T15:03:39.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:39.157+0000 7f07de200640 1 --2- 172.21.15.16:0/4268349902 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f07d81567e0 0x7f07d815e3f0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f07d0009d30 tx=0x7f07d000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:03:39.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:39.157+0000 7f07d5600640 1 -- 172.21.15.16:0/4268349902 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07d0018020 con 0x7f07d81567e0 2024-03-31T15:03:39.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:39.157+0000 7f07d5600640 1 -- 172.21.15.16:0/4268349902 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f07d0014480 con 0x7f07d81567e0 2024-03-31T15:03:39.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:39.157+0000 7f07e0200640 1 -- 172.21.15.16:0/4268349902 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f07d815f210 con 0x7f07d81567e0 2024-03-31T15:03:39.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:39.157+0000 7f07d5600640 1 -- 172.21.15.16:0/4268349902 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07d0014870 con 0x7f07d81567e0 2024-03-31T15:03:39.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:39.157+0000 7f07e0200640 1 -- 172.21.15.16:0/4268349902 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f07d80b50e0 con 0x7f07d81567e0 2024-03-31T15:03:39.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:39.157+0000 7f07e0200640 1 -- 172.21.15.16:0/4268349902 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f07d8069950 con 0x7f07d81567e0 2024-03-31T15:03:39.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:39.157+0000 7f07d5600640 1 -- 172.21.15.16:0/4268349902 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f07d001b020 con 0x7f07d81567e0 2024-03-31T15:03:39.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:39.157+0000 7f07d5600640 1 --2- 172.21.15.16:0/4268349902 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f07b8068380 0x7f07b806a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:39.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:39.161+0000 7f07d5600640 1 -- 172.21.15.16:0/4268349902 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f07d0087320 con 0x7f07d81567e0 2024-03-31T15:03:39.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:39.161+0000 7f07dd800640 1 --2- 172.21.15.16:0/4268349902 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f07b8068380 0x7f07b806a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:39.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:39.161+0000 7f07dd800640 1 --2- 172.21.15.16:0/4268349902 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f07b8068380 0x7f07b806a840 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f07cc007fd0 tx=0x7f07cc007d60 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:03:39.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:39.165+0000 7f07d5600640 1 -- 172.21.15.16:0/4268349902 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f07d0022070 con 0x7f07d81567e0 2024-03-31T15:03:39.368 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:39.373+0000 7f07e0200640 1 -- 172.21.15.16:0/4268349902 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7f07d806e470 con 0x7f07d81567e0 2024-03-31T15:03:39.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:39.973+0000 7f07d5600640 1 -- 172.21.15.16:0/4268349902 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v8) ==== 122+0+0 (secure 0 0 0) 0x7f07d0010040 con 0x7f07d81567e0 2024-03-31T15:03:39.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:39.977+0000 7f07e0200640 1 -- 172.21.15.16:0/4268349902 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f07b8068380 msgr2=0x7f07b806a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:39.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:39.977+0000 7f07e0200640 1 --2- 172.21.15.16:0/4268349902 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f07b8068380 0x7f07b806a840 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f07cc007fd0 tx=0x7f07cc007d60 comp rx=0 tx=0).stop 2024-03-31T15:03:39.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:39.977+0000 7f07e0200640 1 -- 172.21.15.16:0/4268349902 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f07d81567e0 msgr2=0x7f07d815e3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:39.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:39.977+0000 7f07e0200640 1 --2- 172.21.15.16:0/4268349902 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f07d81567e0 0x7f07d815e3f0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f07d0009d30 tx=0x7f07d000c490 comp rx=0 tx=0).stop 2024-03-31T15:03:39.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:39.977+0000 7f07e0200640 1 -- 172.21.15.16:0/4268349902 shutdown_connections 2024-03-31T15:03:39.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:39.977+0000 7f07e0200640 1 --2- 172.21.15.16:0/4268349902 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f07b8068380 0x7f07b806a840 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:39.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:39.977+0000 7f07e0200640 1 --2- 172.21.15.16:0/4268349902 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f07d81567e0 0x7f07d815e3f0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:39.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:39.977+0000 7f07e0200640 1 --2- 172.21.15.16:0/4268349902 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f07d8071550 0x7f07d8153290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:39.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:39.977+0000 7f07e0200640 1 --2- 172.21.15.16:0/4268349902 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f07d806d4d0 0x7f07d8144f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:39.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:39.977+0000 7f07e0200640 1 -- 172.21.15.16:0/4268349902 >> 172.21.15.16:0/4268349902 conn(0x7f07d806ee40 msgr2=0x7f07d80b1550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:03:39.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:39.977+0000 7f07e0200640 1 -- 172.21.15.16:0/4268349902 shutdown_connections 2024-03-31T15:03:39.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:39.977+0000 7f07e0200640 1 -- 172.21.15.16:0/4268349902 wait complete. 2024-03-31T15:03:39.997 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-31T15:03:39.997 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-31T15:03:40.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:40.145+0000 7f9850200640 1 Processor -- start 2024-03-31T15:03:40.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:40.145+0000 7f9850200640 1 -- start start 2024-03-31T15:03:40.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:40.145+0000 7f9850200640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f98480d0260 0x7f9848072380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:40.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:40.145+0000 7f9850200640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f984806faa0 0x7f984806feb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:40.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:40.145+0000 7f9850200640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f984806f000 con 0x7f9848145cd0 2024-03-31T15:03:40.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:40.145+0000 7f9850200640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f98480d1250 con 0x7f984806faa0 2024-03-31T15:03:40.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:40.145+0000 7f9850200640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f984806f610 con 0x7f98480d0260 2024-03-31T15:03:40.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:40.145+0000 7f984d800640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f9848145cd0 0x7f98480b3340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:33930/0 (socket says 172.21.15.16:33930) 2024-03-31T15:03:40.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:40.145+0000 7f984d800640 1 -- 172.21.15.16:0/3892661883 learned_addr learned my addr 172.21.15.16:0/3892661883 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:03:40.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:40.145+0000 7f984e200640 1 --2- 172.21.15.16:0/3892661883 >> v2:172.21.15.16:3300/0 conn(0x7f984806faa0 0x7f984806feb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:40.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:40.145+0000 7f984ce00640 1 --2- 172.21.15.16:0/3892661883 >> v2:172.21.15.89:3300/0 conn(0x7f98480d0260 0x7f9848072380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:40.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:40.145+0000 7f984e200640 1 -- 172.21.15.16:0/3892661883 >> v2:172.21.15.89:3300/0 conn(0x7f98480d0260 msgr2=0x7f9848072380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:03:40.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:40.145+0000 7f984e200640 1 --2- 172.21.15.16:0/3892661883 >> v2:172.21.15.89:3300/0 conn(0x7f98480d0260 0x7f9848072380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:40.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:40.145+0000 7f984e200640 1 -- 172.21.15.16:0/3892661883 >> v1:172.21.15.16:6789/0 conn(0x7f9848145cd0 legacy=0x7f98480b3340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:40.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:40.145+0000 7f984e200640 1 -- 172.21.15.16:0/3892661883 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f98480d0720 con 0x7f984806faa0 2024-03-31T15:03:40.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:40.145+0000 7f9847e00640 1 -- 172.21.15.16:0/3892661883 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3254625670 0 0) 0x7f984806f000 con 0x7f9848145cd0 2024-03-31T15:03:40.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:40.149+0000 7f984e200640 1 --2- 172.21.15.16:0/3892661883 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f984806faa0 0x7f984806feb0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f984806c650 tx=0x7f984001c930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=931a5526ff31f4b5 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:03:40.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:40.149+0000 7f9847e00640 1 -- 172.21.15.16:0/3892661883 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98400b2070 con 0x7f984806faa0 2024-03-31T15:03:40.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:40.149+0000 7f9847e00640 1 -- 172.21.15.16:0/3892661883 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f98400a57f0 con 0x7f984806faa0 2024-03-31T15:03:40.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:40.149+0000 7f9847e00640 1 -- 172.21.15.16:0/3892661883 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98400a5bc0 con 0x7f984806faa0 2024-03-31T15:03:40.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:40.149+0000 7f9850200640 1 -- 172.21.15.16:0/3892661883 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f984806faa0 msgr2=0x7f984806feb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:40.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:40.149+0000 7f9850200640 1 --2- 172.21.15.16:0/3892661883 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f984806faa0 0x7f984806feb0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f984806c650 tx=0x7f984001c930 comp rx=0 tx=0).stop 2024-03-31T15:03:40.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:40.149+0000 7f9850200640 1 -- 172.21.15.16:0/3892661883 shutdown_connections 2024-03-31T15:03:40.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:40.149+0000 7f9850200640 1 --2- 172.21.15.16:0/3892661883 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f984806faa0 0x7f984806feb0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:40.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:40.149+0000 7f9850200640 1 --2- 172.21.15.16:0/3892661883 >> v2:172.21.15.89:3300/0 conn(0x7f98480d0260 0x7f9848072380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:40.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:40.149+0000 7f9850200640 1 -- 172.21.15.16:0/3892661883 >> 172.21.15.16:0/3892661883 conn(0x7f98480b2040 msgr2=0x7f98480b2450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:03:40.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:40.149+0000 7f9850200640 1 -- 172.21.15.16:0/3892661883 shutdown_connections 2024-03-31T15:03:40.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:40.149+0000 7f9850200640 1 -- 172.21.15.16:0/3892661883 wait complete. 2024-03-31T15:03:40.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:40.153+0000 7f9850200640 1 Processor -- start 2024-03-31T15:03:40.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:40.153+0000 7f9850200640 1 -- start start 2024-03-31T15:03:40.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:40.153+0000 7f9850200640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f984806faa0 0x7f984815f2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:40.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:40.153+0000 7f9850200640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f98480d0260 0x7f984815e6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:40.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:40.153+0000 7f9850200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9848145cd0 0x7f984813fb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:40.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:40.153+0000 7f9850200640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f98481460b0 con 0x7f9848145cd0 2024-03-31T15:03:40.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:40.153+0000 7f9850200640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f98480728c0 con 0x7f984806faa0 2024-03-31T15:03:40.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:40.153+0000 7f9850200640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f9848146230 con 0x7f98480d0260 2024-03-31T15:03:40.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:40.153+0000 7f984e200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9848145cd0 0x7f984813fb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:40.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:40.153+0000 7f984e200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9848145cd0 0x7f984813fb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38854/0 (socket says 172.21.15.16:38854) 2024-03-31T15:03:40.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:40.153+0000 7f984e200640 1 -- 172.21.15.16:0/452806777 learned_addr learned my addr 172.21.15.16:0/452806777 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:03:40.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:40.153+0000 7f984d800640 1 --2- 172.21.15.16:0/452806777 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f984806faa0 0x7f984815f2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:40.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:40.153+0000 7f984ce00640 1 --2- 172.21.15.16:0/452806777 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f98480d0260 0x7f984815e6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:40.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:40.153+0000 7f984e200640 1 -- 172.21.15.16:0/452806777 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f98480d0260 msgr2=0x7f984815e6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:40.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:40.153+0000 7f984e200640 1 --2- 172.21.15.16:0/452806777 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f98480d0260 0x7f984815e6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:40.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:40.153+0000 7f984e200640 1 -- 172.21.15.16:0/452806777 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f984806faa0 msgr2=0x7f984815f2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:40.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:40.153+0000 7f984e200640 1 --2- 172.21.15.16:0/452806777 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f984806faa0 0x7f984815f2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:40.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:40.153+0000 7f984e200640 1 -- 172.21.15.16:0/452806777 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f98481402d0 con 0x7f9848145cd0 2024-03-31T15:03:40.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:40.153+0000 7f984e200640 1 --2- 172.21.15.16:0/452806777 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9848145cd0 0x7f984813fb60 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f9840019540 tx=0x7f98400196d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:03:40.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:40.153+0000 7f9845600640 1 -- 172.21.15.16:0/452806777 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98400b2050 con 0x7f9848145cd0 2024-03-31T15:03:40.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:40.153+0000 7f9845600640 1 -- 172.21.15.16:0/452806777 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f98400b3040 con 0x7f9848145cd0 2024-03-31T15:03:40.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:40.153+0000 7f9845600640 1 -- 172.21.15.16:0/452806777 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98400ad650 con 0x7f9848145cd0 2024-03-31T15:03:40.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:40.153+0000 7f9850200640 1 -- 172.21.15.16:0/452806777 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f98481405b0 con 0x7f9848145cd0 2024-03-31T15:03:40.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:40.153+0000 7f9850200640 1 -- 172.21.15.16:0/452806777 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9848140b70 con 0x7f9848145cd0 2024-03-31T15:03:40.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:40.153+0000 7f9850200640 1 -- 172.21.15.16:0/452806777 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9848069950 con 0x7f9848145cd0 2024-03-31T15:03:40.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:40.157+0000 7f9845600640 1 -- 172.21.15.16:0/452806777 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f98400c0020 con 0x7f9848145cd0 2024-03-31T15:03:40.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:40.157+0000 7f9845600640 1 --2- 172.21.15.16:0/452806777 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f98280683d0 0x7f982806a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:40.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:40.157+0000 7f9845600640 1 -- 172.21.15.16:0/452806777 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f9840121090 con 0x7f9848145cd0 2024-03-31T15:03:40.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:40.157+0000 7f984d800640 1 --2- 172.21.15.16:0/452806777 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f98280683d0 0x7f982806a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:40.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:40.157+0000 7f984d800640 1 --2- 172.21.15.16:0/452806777 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f98280683d0 0x7f982806a890 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f983c000f70 tx=0x7f983c009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:03:40.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:40.161+0000 7f9845600640 1 -- 172.21.15.16:0/452806777 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f9840016520 con 0x7f9848145cd0 2024-03-31T15:03:40.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:40.361+0000 7f9850200640 1 -- 172.21.15.16:0/452806777 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9848072af0 con 0x7f9848145cd0 2024-03-31T15:03:40.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:40.361+0000 7f9845600640 1 -- 172.21.15.16:0/452806777 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+5172 (secure 0 0 0) 0x7f98400f0fe0 con 0x7f9848145cd0 2024-03-31T15:03:40.360 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:03:40.360 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4276,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/2832752118","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6833","nonce":2832752118},{"type":"v1","addr":"172.21.15.16:6835","nonce":2832752118}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4278,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6834/153294668","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":153294668},{"type":"v1","addr":"172.21.15.16:6834","nonce":153294668}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4288,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/1653319357","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":1653319357},{"type":"v1","addr":"172.21.15.89:6835","nonce":1653319357}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-31T15:03:36.349865+0000","modified":"2024-03-31T15:03:39.378181+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4293},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.89:6837/2583453131","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":2583453131},{"type":"v1","addr":"172.21.15.89:6837","nonce":2583453131}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4293,"qdb_cluster":[4293]},"id":1}]} 2024-03-31T15:03:40.360 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 8 2024-03-31T15:03:40.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:40.365+0000 7f9850200640 1 -- 172.21.15.16:0/452806777 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f98280683d0 msgr2=0x7f982806a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:40.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:40.365+0000 7f9850200640 1 --2- 172.21.15.16:0/452806777 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f98280683d0 0x7f982806a890 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f983c000f70 tx=0x7f983c009040 comp rx=0 tx=0).stop 2024-03-31T15:03:40.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:40.365+0000 7f9850200640 1 -- 172.21.15.16:0/452806777 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9848145cd0 msgr2=0x7f984813fb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:40.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:40.365+0000 7f9850200640 1 --2- 172.21.15.16:0/452806777 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9848145cd0 0x7f984813fb60 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f9840019540 tx=0x7f98400196d0 comp rx=0 tx=0).stop 2024-03-31T15:03:40.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:40.365+0000 7f9850200640 1 -- 172.21.15.16:0/452806777 shutdown_connections 2024-03-31T15:03:40.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:40.365+0000 7f9850200640 1 --2- 172.21.15.16:0/452806777 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f98280683d0 0x7f982806a890 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:40.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:40.365+0000 7f9850200640 1 --2- 172.21.15.16:0/452806777 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9848145cd0 0x7f984813fb60 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:40.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:40.365+0000 7f9850200640 1 --2- 172.21.15.16:0/452806777 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f98480d0260 0x7f984815e6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:40.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:40.365+0000 7f9850200640 1 --2- 172.21.15.16:0/452806777 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f984806faa0 0x7f984815f2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:40.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:40.365+0000 7f9850200640 1 -- 172.21.15.16:0/452806777 >> 172.21.15.16:0/452806777 conn(0x7f98480b2040 msgr2=0x7f98480a94c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:03:40.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:40.365+0000 7f9850200640 1 -- 172.21.15.16:0/452806777 shutdown_connections 2024-03-31T15:03:40.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:40.365+0000 7f9850200640 1 -- 172.21.15.16:0/452806777 wait complete. 2024-03-31T15:03:40.380 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-31T15:03:40.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:40.525+0000 7f044b000640 1 Processor -- start 2024-03-31T15:03:40.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:40.525+0000 7f044b000640 1 -- start start 2024-03-31T15:03:40.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:40.525+0000 7f044b000640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f0444145e40 0x7f04440cc740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:40.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:40.525+0000 7f044b000640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f04440ccc80 0x7f044406f920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:40.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:40.525+0000 7f044b000640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f04440d0920 con 0x7f044406fe60 2024-03-31T15:03:40.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:40.525+0000 7f044b000640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f04440d06d0 con 0x7f04440ccc80 2024-03-31T15:03:40.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:40.525+0000 7f044b000640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f04440d0c00 con 0x7f0444145e40 2024-03-31T15:03:40.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:40.525+0000 7f043b400640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f04440ccc80 0x7f044406f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:40.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:40.525+0000 7f043b400640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f04440ccc80 0x7f044406f920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38862/0 (socket says 172.21.15.16:38862) 2024-03-31T15:03:40.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:40.525+0000 7f0443e00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f0444145e40 0x7f04440cc740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:40.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:40.525+0000 7f043b400640 1 -- 172.21.15.16:0/494209730 learned_addr learned my addr 172.21.15.16:0/494209730 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:03:40.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:40.525+0000 7f043b400640 1 -- 172.21.15.16:0/494209730 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f0444145e40 msgr2=0x7f04440cc740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:03:40.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:40.525+0000 7f043b400640 1 --2- 172.21.15.16:0/494209730 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f0444145e40 0x7f04440cc740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:40.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:40.525+0000 7f043b400640 1 -- 172.21.15.16:0/494209730 >> v1:172.21.15.89:6789/0 conn(0x7f044406fe60 legacy=0x7f044415fdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:40.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:40.525+0000 7f043b400640 1 -- 172.21.15.16:0/494209730 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04440d1810 con 0x7f04440ccc80 2024-03-31T15:03:40.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:40.525+0000 7f0443400640 1 -- 172.21.15.16:0/494209730 <== mon.1 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 278284223 0 0) 0x7f04440d0920 con 0x7f044406fe60 2024-03-31T15:03:40.525 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:40.529+0000 7f043b400640 1 --2- 172.21.15.16:0/494209730 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f04440ccc80 0x7f044406f920 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f042c009870 tx=0x7f042c01c560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a15eb4831bc1e995 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:03:40.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:40.529+0000 7f0443400640 1 -- 172.21.15.16:0/494209730 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f042c0b0070 con 0x7f04440ccc80 2024-03-31T15:03:40.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:40.529+0000 7f0443400640 1 -- 172.21.15.16:0/494209730 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f042c0ab600 con 0x7f04440ccc80 2024-03-31T15:03:40.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:40.529+0000 7f0443400640 1 -- 172.21.15.16:0/494209730 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f042c0ab9d0 con 0x7f04440ccc80 2024-03-31T15:03:40.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:40.529+0000 7f044b000640 1 -- 172.21.15.16:0/494209730 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f04440ccc80 msgr2=0x7f044406f920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:40.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:40.529+0000 7f044b000640 1 --2- 172.21.15.16:0/494209730 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f04440ccc80 0x7f044406f920 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f042c009870 tx=0x7f042c01c560 comp rx=0 tx=0).stop 2024-03-31T15:03:40.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:40.529+0000 7f044b000640 1 -- 172.21.15.16:0/494209730 shutdown_connections 2024-03-31T15:03:40.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:40.529+0000 7f044b000640 1 --2- 172.21.15.16:0/494209730 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f04440ccc80 0x7f044406f920 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:40.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:40.529+0000 7f044b000640 1 --2- 172.21.15.16:0/494209730 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f0444145e40 0x7f04440cc740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:40.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:40.529+0000 7f044b000640 1 -- 172.21.15.16:0/494209730 >> 172.21.15.16:0/494209730 conn(0x7f0444067330 msgr2=0x7f0444067740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:03:40.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:40.529+0000 7f044b000640 1 -- 172.21.15.16:0/494209730 shutdown_connections 2024-03-31T15:03:40.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:40.529+0000 7f044b000640 1 -- 172.21.15.16:0/494209730 wait complete. 2024-03-31T15:03:40.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:40.533+0000 7f044b000640 1 Processor -- start 2024-03-31T15:03:40.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:40.533+0000 7f044b000640 1 -- start start 2024-03-31T15:03:40.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:40.533+0000 7f044b000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f044406fe60 0x7f0444142d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:40.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:40.533+0000 7f044b000640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f04440ccc80 0x7f04440b1c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:40.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:40.533+0000 7f044b000640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f0444145e40 0x7f04440b2170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:40.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:40.533+0000 7f044b000640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f04440b5240 con 0x7f044406fe60 2024-03-31T15:03:40.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:40.533+0000 7f044b000640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f04440b50c0 con 0x7f04440ccc80 2024-03-31T15:03:40.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:40.533+0000 7f044b000640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f0444169190 con 0x7f0444145e40 2024-03-31T15:03:40.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:40.533+0000 7f0443e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f044406fe60 0x7f0444142d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:40.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:40.533+0000 7f0443e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f044406fe60 0x7f0444142d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38866/0 (socket says 172.21.15.16:38866) 2024-03-31T15:03:40.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:40.533+0000 7f0443e00640 1 -- 172.21.15.16:0/1481969602 learned_addr learned my addr 172.21.15.16:0/1481969602 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:03:40.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:40.533+0000 7f043b400640 1 --2- 172.21.15.16:0/1481969602 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f04440ccc80 0x7f04440b1c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:40.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:40.533+0000 7f0449000640 1 --2- 172.21.15.16:0/1481969602 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f0444145e40 0x7f04440b2170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:40.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:40.533+0000 7f0443e00640 1 -- 172.21.15.16:0/1481969602 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f0444145e40 msgr2=0x7f04440b2170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:40.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:40.533+0000 7f0443e00640 1 --2- 172.21.15.16:0/1481969602 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f0444145e40 0x7f04440b2170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:40.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:40.533+0000 7f0443e00640 1 -- 172.21.15.16:0/1481969602 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f04440ccc80 msgr2=0x7f04440b1c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:40.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:40.533+0000 7f0443e00640 1 --2- 172.21.15.16:0/1481969602 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f04440ccc80 0x7f04440b1c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:40.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:40.533+0000 7f0443e00640 1 -- 172.21.15.16:0/1481969602 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04440b7540 con 0x7f044406fe60 2024-03-31T15:03:40.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:40.533+0000 7f0443e00640 1 --2- 172.21.15.16:0/1481969602 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f044406fe60 0x7f0444142d20 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f043400efc0 tx=0x7f043400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:03:40.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:40.533+0000 7f0440c00640 1 -- 172.21.15.16:0/1481969602 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0434018020 con 0x7f044406fe60 2024-03-31T15:03:40.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:40.533+0000 7f0440c00640 1 -- 172.21.15.16:0/1481969602 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0434014480 con 0x7f044406fe60 2024-03-31T15:03:40.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:40.533+0000 7f0440c00640 1 -- 172.21.15.16:0/1481969602 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0434014870 con 0x7f044406fe60 2024-03-31T15:03:40.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:40.533+0000 7f044b000640 1 -- 172.21.15.16:0/1481969602 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f04440c7930 con 0x7f044406fe60 2024-03-31T15:03:40.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:40.533+0000 7f044b000640 1 -- 172.21.15.16:0/1481969602 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f044415f500 con 0x7f044406fe60 2024-03-31T15:03:40.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:40.533+0000 7f044b000640 1 -- 172.21.15.16:0/1481969602 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0444069950 con 0x7f044406fe60 2024-03-31T15:03:40.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:40.537+0000 7f0440c00640 1 -- 172.21.15.16:0/1481969602 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f043401b020 con 0x7f044406fe60 2024-03-31T15:03:40.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:40.537+0000 7f0440c00640 1 --2- 172.21.15.16:0/1481969602 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f0420068380 0x7f042006a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:40.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:40.537+0000 7f0440c00640 1 -- 172.21.15.16:0/1481969602 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f0434087760 con 0x7f044406fe60 2024-03-31T15:03:40.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:40.537+0000 7f043b400640 1 --2- 172.21.15.16:0/1481969602 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f0420068380 0x7f042006a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:40.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:40.537+0000 7f043b400640 1 --2- 172.21.15.16:0/1481969602 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f0420068380 0x7f042006a840 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f042c0ad000 tx=0x7f042c0a6000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:03:40.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:40.541+0000 7f0440c00640 1 -- 172.21.15.16:0/1481969602 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f0434022030 con 0x7f044406fe60 2024-03-31T15:03:40.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:40.709+0000 7f044b000640 1 -- 172.21.15.16:0/1481969602 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f04440cf3c0 con 0x7f044406fe60 2024-03-31T15:03:40.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:40.709+0000 7f0440c00640 1 -- 172.21.15.16:0/1481969602 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) ==== 74+0+16533 (secure 0 0 0) 0x7f0434010040 con 0x7f044406fe60 2024-03-31T15:03:40.707 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:03:40.707 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":21,"fsid":"286eb5da-fb47-4550-b290-d9519c13b9a5","created":"2024-03-31T15:03:11.438952+0000","modified":"2024-03-31T15:03:37.957829+0000","last_up_change":"2024-03-31T15:03:22.885535+0000","last_in_change":"2024-03-31T15:03:13.941667+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-31T15:03:22.550787+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-31T15:03:34.488425+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-31T15:03:35.299116+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"92a66450-ef19-40d8-ac3d-d760b23b0bda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6801","nonce":932712244}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6803","nonce":932712244}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6807","nonce":932712244}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6805","nonce":932712244}]},"public_addr":"172.21.15.16:6801/932712244","cluster_addr":"172.21.15.16:6803/932712244","heartbeat_back_addr":"172.21.15.16:6807/932712244","heartbeat_front_addr":"172.21.15.16:6805/932712244","state":["exists","up"]},{"osd":1,"uuid":"da9d1f2b-1968-4959-b843-faa231988e92","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6810","nonce":1398737533}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6811","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6813","nonce":1398737533}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6821","nonce":1398737533}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6815","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6816","nonce":1398737533}]},"public_addr":"172.21.15.16:6810/1398737533","cluster_addr":"172.21.15.16:6813/1398737533","heartbeat_back_addr":"172.21.15.16:6821/1398737533","heartbeat_front_addr":"172.21.15.16:6816/1398737533","state":["exists","up"]},{"osd":2,"uuid":"01a73662-9a21-4b1f-bcbb-42b9636a4638","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6819","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6822","nonce":3018526646}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6825","nonce":3018526646}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6831","nonce":3018526646}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6827","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6829","nonce":3018526646}]},"public_addr":"172.21.15.16:6822/3018526646","cluster_addr":"172.21.15.16:6825/3018526646","heartbeat_back_addr":"172.21.15.16:6831/3018526646","heartbeat_front_addr":"172.21.15.16:6829/3018526646","state":["exists","up"]},{"osd":3,"uuid":"c7771fd3-e18e-4500-ac38-23cf10ebcd75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6809","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6812","nonce":3360352974}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6817","nonce":3360352974}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6828","nonce":3360352974}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6823","nonce":3360352974}]},"public_addr":"172.21.15.16:6812/3360352974","cluster_addr":"172.21.15.16:6817/3360352974","heartbeat_back_addr":"172.21.15.16:6828/3360352974","heartbeat_front_addr":"172.21.15.16:6823/3360352974","state":["exists","up"]},{"osd":4,"uuid":"88260730-1482-4fbd-a40f-cb1124e9df4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6801","nonce":1856844856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6803","nonce":1856844856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6807","nonce":1856844856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6805","nonce":1856844856}]},"public_addr":"172.21.15.89:6801/1856844856","cluster_addr":"172.21.15.89:6803/1856844856","heartbeat_back_addr":"172.21.15.89:6807/1856844856","heartbeat_front_addr":"172.21.15.89:6805/1856844856","state":["exists","up"]},{"osd":5,"uuid":"4cb2c00c-fae0-4dd8-9d27-ec4f0093ccf7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6814","nonce":50395527}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6817","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6820","nonce":50395527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6831","nonce":50395527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6823","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6826","nonce":50395527}]},"public_addr":"172.21.15.89:6814/50395527","cluster_addr":"172.21.15.89:6820/50395527","heartbeat_back_addr":"172.21.15.89:6831/50395527","heartbeat_front_addr":"172.21.15.89:6826/50395527","state":["exists","up"]},{"osd":6,"uuid":"81ce0c19-2554-4cae-acbe-66a05e9e9b9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6809","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6812","nonce":3470471752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6815","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6818","nonce":3470471752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6827","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6829","nonce":3470471752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6821","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6824","nonce":3470471752}]},"public_addr":"172.21.15.89:6812/3470471752","cluster_addr":"172.21.15.89:6818/3470471752","heartbeat_back_addr":"172.21.15.89:6829/3470471752","heartbeat_front_addr":"172.21.15.89:6824/3470471752","state":["exists","up"]},{"osd":7,"uuid":"85e4313b-d37d-442e-8042-153bf20a847d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6811","nonce":1796778630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6813","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6816","nonce":1796778630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6825","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6828","nonce":1796778630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6819","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6822","nonce":1796778630}]},"public_addr":"172.21.15.89:6811/1796778630","cluster_addr":"172.21.15.89:6816/1796778630","heartbeat_back_addr":"172.21.15.89:6828/1796778630","heartbeat_front_addr":"172.21.15.89:6822/1796778630","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.439088+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.529970+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.680512+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:21.044161+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.582111+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.536558+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.409522+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.590229+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-31T15:03:40.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:40.709+0000 7f044b000640 1 -- 172.21.15.16:0/1481969602 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f0420068380 msgr2=0x7f042006a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:40.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:40.709+0000 7f044b000640 1 --2- 172.21.15.16:0/1481969602 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f0420068380 0x7f042006a840 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f042c0ad000 tx=0x7f042c0a6000 comp rx=0 tx=0).stop 2024-03-31T15:03:40.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:40.709+0000 7f044b000640 1 -- 172.21.15.16:0/1481969602 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f044406fe60 msgr2=0x7f0444142d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:40.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:40.709+0000 7f044b000640 1 --2- 172.21.15.16:0/1481969602 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f044406fe60 0x7f0444142d20 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f043400efc0 tx=0x7f043400c490 comp rx=0 tx=0).stop 2024-03-31T15:03:40.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:40.713+0000 7f044b000640 1 -- 172.21.15.16:0/1481969602 shutdown_connections 2024-03-31T15:03:40.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:40.713+0000 7f044b000640 1 --2- 172.21.15.16:0/1481969602 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f0420068380 0x7f042006a840 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:40.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:40.713+0000 7f044b000640 1 --2- 172.21.15.16:0/1481969602 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f0444145e40 0x7f04440b2170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:40.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:40.713+0000 7f044b000640 1 --2- 172.21.15.16:0/1481969602 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f04440ccc80 0x7f04440b1c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:40.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:40.713+0000 7f044b000640 1 --2- 172.21.15.16:0/1481969602 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f044406fe60 0x7f0444142d20 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:40.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:40.713+0000 7f044b000640 1 -- 172.21.15.16:0/1481969602 >> 172.21.15.16:0/1481969602 conn(0x7f0444067330 msgr2=0x7f04441434b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:03:40.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:40.713+0000 7f044b000640 1 -- 172.21.15.16:0/1481969602 shutdown_connections 2024-03-31T15:03:40.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:40.713+0000 7f044b000640 1 -- 172.21.15.16:0/1481969602 wait complete. 2024-03-31T15:03:40.728 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-31T15:03:40.729 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-31T15:03:40.743 INFO:tasks.ceph.mds.b.smithi089.stderr:2024-03-31T15:03:40.749+0000 7f0735c00640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-31T15:03:40.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:40.873+0000 7f4a12400640 1 Processor -- start 2024-03-31T15:03:40.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:40.873+0000 7f4a12400640 1 -- start start 2024-03-31T15:03:40.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:40.873+0000 7f4a12400640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f4a0c145e40 0x7f4a0c0cc740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:40.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:40.873+0000 7f4a12400640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4a0c0d0920 con 0x7f4a0c070030 2024-03-31T15:03:40.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:40.873+0000 7f4a12400640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4a0c0d0c00 con 0x7f4a0c0ccc80 2024-03-31T15:03:40.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:40.873+0000 7f4a12400640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f4a0c0d06d0 con 0x7f4a0c145e40 2024-03-31T15:03:40.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:40.873+0000 7f4a0b400640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f4a0c145e40 0x7f4a0c0cc740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:40.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:40.873+0000 7f4a0b400640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f4a0c145e40 0x7f4a0c0cc740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38876/0 (socket says 172.21.15.16:38876) 2024-03-31T15:03:40.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:40.873+0000 7f4a0b400640 1 -- 172.21.15.16:0/2933915723 learned_addr learned my addr 172.21.15.16:0/2933915723 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:03:40.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:40.873+0000 7f4a0be00640 1 --1- 172.21.15.16:0/2933915723 >> v1:172.21.15.16:6789/0 conn(0x7f4a0c070030 0x7f4a0c0d2010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:33940/0 (socket says 172.21.15.16:33940) 2024-03-31T15:03:40.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:40.873+0000 7f4a0b400640 1 -- 172.21.15.16:0/2933915723 >> v1:172.21.15.89:6789/0 conn(0x7f4a0c0ccc80 legacy=0x7f4a0c06f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:40.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:40.873+0000 7f4a0b400640 1 -- 172.21.15.16:0/2933915723 >> v1:172.21.15.16:6789/0 conn(0x7f4a0c070030 legacy=0x7f4a0c0d2010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:40.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:40.873+0000 7f4a0aa00640 1 -- 172.21.15.16:0/2933915723 <== mon.1 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3058832322 0 0) 0x7f4a0c0d0c00 con 0x7f4a0c0ccc80 2024-03-31T15:03:40.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:40.873+0000 7f4a0b400640 1 -- 172.21.15.16:0/2933915723 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a0c0d1810 con 0x7f4a0c145e40 2024-03-31T15:03:40.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:40.877+0000 7f4a0b400640 1 --2- 172.21.15.16:0/2933915723 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4a0c145e40 0x7f4a0c0cc740 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f49fc009870 tx=0x7f49fc01c560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d597026503e3cbc6 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:03:40.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:40.877+0000 7f4a0aa00640 1 -- 172.21.15.16:0/2933915723 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49fc0a7070 con 0x7f4a0c145e40 2024-03-31T15:03:40.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:40.877+0000 7f4a0aa00640 1 -- 172.21.15.16:0/2933915723 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f49fc0b0070 con 0x7f4a0c145e40 2024-03-31T15:03:40.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:40.877+0000 7f4a0aa00640 1 -- 172.21.15.16:0/2933915723 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49fc0ab890 con 0x7f4a0c145e40 2024-03-31T15:03:40.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:40.877+0000 7f4a12400640 1 -- 172.21.15.16:0/2933915723 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4a0c145e40 msgr2=0x7f4a0c0cc740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:40.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:40.877+0000 7f4a12400640 1 --2- 172.21.15.16:0/2933915723 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4a0c145e40 0x7f4a0c0cc740 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f49fc009870 tx=0x7f49fc01c560 comp rx=0 tx=0).stop 2024-03-31T15:03:40.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:40.877+0000 7f4a12400640 1 -- 172.21.15.16:0/2933915723 shutdown_connections 2024-03-31T15:03:40.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:40.877+0000 7f4a12400640 1 --2- 172.21.15.16:0/2933915723 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4a0c145e40 0x7f4a0c0cc740 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:40.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:40.877+0000 7f4a12400640 1 -- 172.21.15.16:0/2933915723 >> 172.21.15.16:0/2933915723 conn(0x7f4a0c067330 msgr2=0x7f4a0c067740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:03:40.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:40.877+0000 7f4a12400640 1 -- 172.21.15.16:0/2933915723 shutdown_connections 2024-03-31T15:03:40.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:40.877+0000 7f4a12400640 1 -- 172.21.15.16:0/2933915723 wait complete. 2024-03-31T15:03:40.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:40.881+0000 7f4a12400640 1 Processor -- start 2024-03-31T15:03:40.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:40.881+0000 7f4a12400640 1 -- start start 2024-03-31T15:03:40.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:40.881+0000 7f4a12400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4a0c070030 0x7f4a0c0b6ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:40.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:40.881+0000 7f4a12400640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f4a0c0ccc80 0x7f4a0c06c970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:40.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:40.881+0000 7f4a12400640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4a0c145e40 0x7f4a0c1444f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:40.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:40.881+0000 7f4a12400640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f4a0c0b5240 con 0x7f4a0c070030 2024-03-31T15:03:40.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:40.881+0000 7f4a12400640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f4a0c0b50c0 con 0x7f4a0c145e40 2024-03-31T15:03:40.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:40.881+0000 7f4a12400640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f4a0c16a1a0 con 0x7f4a0c0ccc80 2024-03-31T15:03:40.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:40.881+0000 7f4a0b400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4a0c070030 0x7f4a0c0b6ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:40.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:40.881+0000 7f4a0b400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4a0c070030 0x7f4a0c0b6ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38882/0 (socket says 172.21.15.16:38882) 2024-03-31T15:03:40.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:40.881+0000 7f4a0b400640 1 -- 172.21.15.16:0/180594335 learned_addr learned my addr 172.21.15.16:0/180594335 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:03:40.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:40.881+0000 7f4a0be00640 1 --2- 172.21.15.16:0/180594335 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4a0c145e40 0x7f4a0c1444f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:40.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:40.881+0000 7f4a03e00640 1 --2- 172.21.15.16:0/180594335 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f4a0c0ccc80 0x7f4a0c06c970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:40.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:40.881+0000 7f4a0b400640 1 -- 172.21.15.16:0/180594335 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f4a0c0ccc80 msgr2=0x7f4a0c06c970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:40.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:40.881+0000 7f4a0b400640 1 --2- 172.21.15.16:0/180594335 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f4a0c0ccc80 0x7f4a0c06c970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:40.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:40.881+0000 7f4a0b400640 1 -- 172.21.15.16:0/180594335 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4a0c145e40 msgr2=0x7f4a0c1444f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:40.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:40.881+0000 7f4a0b400640 1 --2- 172.21.15.16:0/180594335 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4a0c145e40 0x7f4a0c1444f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:40.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:40.881+0000 7f4a0b400640 1 -- 172.21.15.16:0/180594335 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a0c159310 con 0x7f4a0c070030 2024-03-31T15:03:40.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:40.881+0000 7f4a03e00640 1 --2- 172.21.15.16:0/180594335 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f4a0c0ccc80 0x7f4a0c06c970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:03:40.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:40.881+0000 7f4a0b400640 1 --2- 172.21.15.16:0/180594335 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4a0c070030 0x7f4a0c0b6ee0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f49fc019170 tx=0x7f49fc01ada0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:03:40.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:40.881+0000 7f4a03400640 1 -- 172.21.15.16:0/180594335 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49fc0abe60 con 0x7f4a0c070030 2024-03-31T15:03:40.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:40.881+0000 7f4a03400640 1 -- 172.21.15.16:0/180594335 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f49fc0b4800 con 0x7f4a0c070030 2024-03-31T15:03:40.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:40.881+0000 7f4a03400640 1 -- 172.21.15.16:0/180594335 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49fc0b4bf0 con 0x7f4a0c070030 2024-03-31T15:03:40.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:40.881+0000 7f4a12400640 1 -- 172.21.15.16:0/180594335 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4a0c159580 con 0x7f4a0c070030 2024-03-31T15:03:40.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:40.881+0000 7f4a12400640 1 -- 172.21.15.16:0/180594335 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4a0c06ceb0 con 0x7f4a0c070030 2024-03-31T15:03:40.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:40.881+0000 7f4a12400640 1 -- 172.21.15.16:0/180594335 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4a0c069950 con 0x7f4a0c070030 2024-03-31T15:03:40.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:40.885+0000 7f4a03400640 1 -- 172.21.15.16:0/180594335 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f49fc0b3380 con 0x7f4a0c070030 2024-03-31T15:03:40.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:40.885+0000 7f4a03400640 1 --2- 172.21.15.16:0/180594335 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f49e8068380 0x7f49e806a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:40.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:40.885+0000 7f4a03400640 1 -- 172.21.15.16:0/180594335 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f49fc120ef0 con 0x7f4a0c070030 2024-03-31T15:03:40.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:40.885+0000 7f4a03e00640 1 --2- 172.21.15.16:0/180594335 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f49e8068380 0x7f49e806a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:40.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:40.885+0000 7f4a03e00640 1 --2- 172.21.15.16:0/180594335 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f49e8068380 0x7f49e806a840 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f4a0c06e5d0 tx=0x7f49f4009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:03:40.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:40.889+0000 7f4a03400640 1 -- 172.21.15.16:0/180594335 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f49fc016150 con 0x7f4a0c070030 2024-03-31T15:03:41.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:41.053+0000 7f4a12400640 1 -- 172.21.15.16:0/180594335 --> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f4a0c160ec0 con 0x7f49e8068380 2024-03-31T15:03:41.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:41.061+0000 7f4a03400640 1 -- 172.21.15.16:0/180594335 <== mgr.4100 v2:172.21.15.89:6832/1549407273 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251885 (secure 0 0 0) 0x7f49f4001460 con 0x7f49e8068380 2024-03-31T15:03:41.059 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:03:41.061 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-03-31T15:03:41.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:41.061+0000 7f4a12400640 1 -- 172.21.15.16:0/180594335 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f49e8068380 msgr2=0x7f49e806a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:41.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:41.061+0000 7f4a12400640 1 --2- 172.21.15.16:0/180594335 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f49e8068380 0x7f49e806a840 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f4a0c06e5d0 tx=0x7f49f4009040 comp rx=0 tx=0).stop 2024-03-31T15:03:41.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:41.061+0000 7f4a12400640 1 -- 172.21.15.16:0/180594335 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4a0c070030 msgr2=0x7f4a0c0b6ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:41.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:41.061+0000 7f4a12400640 1 --2- 172.21.15.16:0/180594335 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4a0c070030 0x7f4a0c0b6ee0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f49fc019170 tx=0x7f49fc01ada0 comp rx=0 tx=0).stop 2024-03-31T15:03:41.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:41.065+0000 7f4a12400640 1 -- 172.21.15.16:0/180594335 shutdown_connections 2024-03-31T15:03:41.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:41.065+0000 7f4a12400640 1 --2- 172.21.15.16:0/180594335 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f49e8068380 0x7f49e806a840 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:41.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:41.065+0000 7f4a12400640 1 --2- 172.21.15.16:0/180594335 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4a0c145e40 0x7f4a0c1444f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:41.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:41.065+0000 7f4a12400640 1 --2- 172.21.15.16:0/180594335 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f4a0c0ccc80 0x7f4a0c06c970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:41.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:41.065+0000 7f4a12400640 1 --2- 172.21.15.16:0/180594335 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4a0c070030 0x7f4a0c0b6ee0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:41.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:41.065+0000 7f4a12400640 1 -- 172.21.15.16:0/180594335 >> 172.21.15.16:0/180594335 conn(0x7f4a0c067330 msgr2=0x7f4a0c0b4620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:03:41.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:41.065+0000 7f4a12400640 1 -- 172.21.15.16:0/180594335 shutdown_connections 2024-03-31T15:03:41.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:41.065+0000 7f4a12400640 1 -- 172.21.15.16:0/180594335 wait complete. 2024-03-31T15:03:41.079 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2024-03-31T15:03:40.530461+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":103,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":235548,"kb_used_data":2724,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754739172,"statfs":{"total":773094113280,"available":772852912128,"internally_reserved":0,"allocated":2789376,"data_stored":1707842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":238276144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":37,"apply_latency_ms":37,"commit_latency_ns":37000000,"apply_latency_ns":37000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.596594"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-31T15:03:36.993363+0000","last_change":"2024-03-31T15:03:36.984154+0000","last_active":"2024-03-31T15:03:36.993363+0000","last_peered":"2024-03-31T15:03:36.993363+0000","last_clean":"2024-03-31T15:03:36.993363+0000","last_became_active":"2024-03-31T15:03:36.983712+0000","last_became_peered":"2024-03-31T15:03:36.983712+0000","last_unstale":"2024-03-31T15:03:36.993363+0000","last_undegraded":"2024-03-31T15:03:36.993363+0000","last_fullsized":"2024-03-31T15:03:36.993363+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T16:25:42.516431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-31T15:03:36.966594+0000","last_change":"2024-03-31T15:03:36.962443+0000","last_active":"2024-03-31T15:03:36.966594+0000","last_peered":"2024-03-31T15:03:36.966594+0000","last_clean":"2024-03-31T15:03:36.966594+0000","last_became_active":"2024-03-31T15:03:36.962140+0000","last_became_peered":"2024-03-31T15:03:36.962140+0000","last_unstale":"2024-03-31T15:03:36.966594+0000","last_undegraded":"2024-03-31T15:03:36.966594+0000","last_fullsized":"2024-03-31T15:03:36.966594+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:51:11.582045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-31T15:03:36.986728+0000","last_change":"2024-03-31T15:03:36.985502+0000","last_active":"2024-03-31T15:03:36.986728+0000","last_peered":"2024-03-31T15:03:36.986728+0000","last_clean":"2024-03-31T15:03:36.986728+0000","last_became_active":"2024-03-31T15:03:36.985263+0000","last_became_peered":"2024-03-31T15:03:36.985263+0000","last_unstale":"2024-03-31T15:03:36.986728+0000","last_undegraded":"2024-03-31T15:03:36.986728+0000","last_fullsized":"2024-03-31T15:03:36.986728+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T16:00:46.569324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-31T15:03:36.988475+0000","last_change":"2024-03-31T15:03:36.981333+0000","last_active":"2024-03-31T15:03:36.988475+0000","last_peered":"2024-03-31T15:03:36.988475+0000","last_clean":"2024-03-31T15:03:36.988475+0000","last_became_active":"2024-03-31T15:03:36.980937+0000","last_became_peered":"2024-03-31T15:03:36.980937+0000","last_unstale":"2024-03-31T15:03:36.988475+0000","last_undegraded":"2024-03-31T15:03:36.988475+0000","last_fullsized":"2024-03-31T15:03:36.988475+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:05:28.499146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-31T15:03:36.994632+0000","last_change":"2024-03-31T15:03:36.989117+0000","last_active":"2024-03-31T15:03:36.994632+0000","last_peered":"2024-03-31T15:03:36.994632+0000","last_clean":"2024-03-31T15:03:36.994632+0000","last_became_active":"2024-03-31T15:03:36.986002+0000","last_became_peered":"2024-03-31T15:03:36.986002+0000","last_unstale":"2024-03-31T15:03:36.994632+0000","last_undegraded":"2024-03-31T15:03:36.994632+0000","last_fullsized":"2024-03-31T15:03:36.994632+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T19:58:22.015705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-31T15:03:36.988429+0000","last_change":"2024-03-31T15:03:36.981121+0000","last_active":"2024-03-31T15:03:36.988429+0000","last_peered":"2024-03-31T15:03:36.988429+0000","last_clean":"2024-03-31T15:03:36.988429+0000","last_became_active":"2024-03-31T15:03:36.980669+0000","last_became_peered":"2024-03-31T15:03:36.980669+0000","last_unstale":"2024-03-31T15:03:36.988429+0000","last_undegraded":"2024-03-31T15:03:36.988429+0000","last_fullsized":"2024-03-31T15:03:36.988429+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:07:25.174977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-31T15:03:36.976931+0000","last_change":"2024-03-31T15:03:36.974140+0000","last_active":"2024-03-31T15:03:36.976931+0000","last_peered":"2024-03-31T15:03:36.976931+0000","last_clean":"2024-03-31T15:03:36.976931+0000","last_became_active":"2024-03-31T15:03:36.973867+0000","last_became_peered":"2024-03-31T15:03:36.973867+0000","last_unstale":"2024-03-31T15:03:36.976931+0000","last_undegraded":"2024-03-31T15:03:36.976931+0000","last_fullsized":"2024-03-31T15:03:36.976931+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:02:20.135763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-31T15:03:36.974633+0000","last_change":"2024-03-31T15:03:36.969620+0000","last_active":"2024-03-31T15:03:36.974633+0000","last_peered":"2024-03-31T15:03:36.974633+0000","last_clean":"2024-03-31T15:03:36.974633+0000","last_became_active":"2024-03-31T15:03:36.969317+0000","last_became_peered":"2024-03-31T15:03:36.969317+0000","last_unstale":"2024-03-31T15:03:36.974633+0000","last_undegraded":"2024-03-31T15:03:36.974633+0000","last_fullsized":"2024-03-31T15:03:36.974633+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:29:47.722244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-31T15:03:36.994657+0000","last_change":"2024-03-31T15:03:36.990208+0000","last_active":"2024-03-31T15:03:36.994657+0000","last_peered":"2024-03-31T15:03:36.994657+0000","last_clean":"2024-03-31T15:03:36.994657+0000","last_became_active":"2024-03-31T15:03:36.989706+0000","last_became_peered":"2024-03-31T15:03:36.989706+0000","last_unstale":"2024-03-31T15:03:36.994657+0000","last_undegraded":"2024-03-31T15:03:36.994657+0000","last_fullsized":"2024-03-31T15:03:36.994657+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:26:15.339683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-31T15:03:36.986235+0000","last_change":"2024-03-31T15:03:36.982132+0000","last_active":"2024-03-31T15:03:36.986235+0000","last_peered":"2024-03-31T15:03:36.986235+0000","last_clean":"2024-03-31T15:03:36.986235+0000","last_became_active":"2024-03-31T15:03:36.981697+0000","last_became_peered":"2024-03-31T15:03:36.981697+0000","last_unstale":"2024-03-31T15:03:36.986235+0000","last_undegraded":"2024-03-31T15:03:36.986235+0000","last_fullsized":"2024-03-31T15:03:36.986235+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:37:26.685300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-31T15:03:36.986172+0000","last_change":"2024-03-31T15:03:36.980849+0000","last_active":"2024-03-31T15:03:36.986172+0000","last_peered":"2024-03-31T15:03:36.986172+0000","last_clean":"2024-03-31T15:03:36.986172+0000","last_became_active":"2024-03-31T15:03:36.980535+0000","last_became_peered":"2024-03-31T15:03:36.980535+0000","last_unstale":"2024-03-31T15:03:36.986172+0000","last_undegraded":"2024-03-31T15:03:36.986172+0000","last_fullsized":"2024-03-31T15:03:36.986172+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:29:57.089580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-31T15:03:37.966263+0000","last_change":"2024-03-31T15:03:36.962572+0000","last_active":"2024-03-31T15:03:37.966263+0000","last_peered":"2024-03-31T15:03:37.966263+0000","last_clean":"2024-03-31T15:03:37.966263+0000","last_became_active":"2024-03-31T15:03:36.960772+0000","last_became_peered":"2024-03-31T15:03:36.960772+0000","last_unstale":"2024-03-31T15:03:37.966263+0000","last_undegraded":"2024-03-31T15:03:37.966263+0000","last_fullsized":"2024-03-31T15:03:37.966263+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T15:58:19.498104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-31T15:03:36.965403+0000","last_change":"2024-03-31T15:03:36.960134+0000","last_active":"2024-03-31T15:03:36.965403+0000","last_peered":"2024-03-31T15:03:36.965403+0000","last_clean":"2024-03-31T15:03:36.965403+0000","last_became_active":"2024-03-31T15:03:36.959747+0000","last_became_peered":"2024-03-31T15:03:36.959747+0000","last_unstale":"2024-03-31T15:03:36.965403+0000","last_undegraded":"2024-03-31T15:03:36.965403+0000","last_fullsized":"2024-03-31T15:03:36.965403+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T16:00:46.569324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-31T15:03:36.971078+0000","last_change":"2024-03-31T15:03:36.964930+0000","last_active":"2024-03-31T15:03:36.971078+0000","last_peered":"2024-03-31T15:03:36.971078+0000","last_clean":"2024-03-31T15:03:36.971078+0000","last_became_active":"2024-03-31T15:03:36.964006+0000","last_became_peered":"2024-03-31T15:03:36.964006+0000","last_unstale":"2024-03-31T15:03:36.971078+0000","last_undegraded":"2024-03-31T15:03:36.971078+0000","last_fullsized":"2024-03-31T15:03:36.971078+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T15:38:43.103277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-31T15:03:36.983707+0000","last_change":"2024-03-31T15:03:36.981622+0000","last_active":"2024-03-31T15:03:36.983707+0000","last_peered":"2024-03-31T15:03:36.983707+0000","last_clean":"2024-03-31T15:03:36.983707+0000","last_became_active":"2024-03-31T15:03:36.981267+0000","last_became_peered":"2024-03-31T15:03:36.981267+0000","last_unstale":"2024-03-31T15:03:36.983707+0000","last_undegraded":"2024-03-31T15:03:36.983707+0000","last_fullsized":"2024-03-31T15:03:36.983707+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:29:57.089580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-31T15:03:36.984560+0000","last_change":"2024-03-31T15:03:36.977834+0000","last_active":"2024-03-31T15:03:36.984560+0000","last_peered":"2024-03-31T15:03:36.984560+0000","last_clean":"2024-03-31T15:03:36.984560+0000","last_became_active":"2024-03-31T15:03:36.977469+0000","last_became_peered":"2024-03-31T15:03:36.977469+0000","last_unstale":"2024-03-31T15:03:36.984560+0000","last_undegraded":"2024-03-31T15:03:36.984560+0000","last_fullsized":"2024-03-31T15:03:36.984560+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:19:24.339229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-31T15:03:36.988855+0000","last_change":"2024-03-31T15:03:36.981952+0000","last_active":"2024-03-31T15:03:36.988855+0000","last_peered":"2024-03-31T15:03:36.988855+0000","last_clean":"2024-03-31T15:03:36.988855+0000","last_became_active":"2024-03-31T15:03:36.981615+0000","last_became_peered":"2024-03-31T15:03:36.981615+0000","last_unstale":"2024-03-31T15:03:36.988855+0000","last_undegraded":"2024-03-31T15:03:36.988855+0000","last_fullsized":"2024-03-31T15:03:36.988855+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:11:06.260618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-31T15:03:36.984568+0000","last_change":"2024-03-31T15:03:36.981172+0000","last_active":"2024-03-31T15:03:36.984568+0000","last_peered":"2024-03-31T15:03:36.984568+0000","last_clean":"2024-03-31T15:03:36.984568+0000","last_became_active":"2024-03-31T15:03:36.980303+0000","last_became_peered":"2024-03-31T15:03:36.980303+0000","last_unstale":"2024-03-31T15:03:36.984568+0000","last_undegraded":"2024-03-31T15:03:36.984568+0000","last_fullsized":"2024-03-31T15:03:36.984568+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:26:54.177359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-31T15:03:36.994700+0000","last_change":"2024-03-31T15:03:36.991712+0000","last_active":"2024-03-31T15:03:36.994700+0000","last_peered":"2024-03-31T15:03:36.994700+0000","last_clean":"2024-03-31T15:03:36.994700+0000","last_became_active":"2024-03-31T15:03:36.988177+0000","last_became_peered":"2024-03-31T15:03:36.988177+0000","last_unstale":"2024-03-31T15:03:36.994700+0000","last_undegraded":"2024-03-31T15:03:36.994700+0000","last_fullsized":"2024-03-31T15:03:36.994700+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:05:30.818052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-31T15:03:36.983269+0000","last_change":"2024-03-31T15:03:36.968423+0000","last_active":"2024-03-31T15:03:36.983269+0000","last_peered":"2024-03-31T15:03:36.983269+0000","last_clean":"2024-03-31T15:03:36.983269+0000","last_became_active":"2024-03-31T15:03:36.968014+0000","last_became_peered":"2024-03-31T15:03:36.968014+0000","last_unstale":"2024-03-31T15:03:36.983269+0000","last_undegraded":"2024-03-31T15:03:36.983269+0000","last_fullsized":"2024-03-31T15:03:36.983269+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T16:00:46.569324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-31T15:03:36.986393+0000","last_change":"2024-03-31T15:03:36.975734+0000","last_active":"2024-03-31T15:03:36.986393+0000","last_peered":"2024-03-31T15:03:36.986393+0000","last_clean":"2024-03-31T15:03:36.986393+0000","last_became_active":"2024-03-31T15:03:36.970929+0000","last_became_peered":"2024-03-31T15:03:36.970929+0000","last_unstale":"2024-03-31T15:03:36.986393+0000","last_undegraded":"2024-03-31T15:03:36.986393+0000","last_fullsized":"2024-03-31T15:03:36.986393+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T15:07:58.072065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-31T15:03:36.989089+0000","last_change":"2024-03-31T15:03:36.982234+0000","last_active":"2024-03-31T15:03:36.989089+0000","last_peered":"2024-03-31T15:03:36.989089+0000","last_clean":"2024-03-31T15:03:36.989089+0000","last_became_active":"2024-03-31T15:03:36.981359+0000","last_became_peered":"2024-03-31T15:03:36.981359+0000","last_unstale":"2024-03-31T15:03:36.989089+0000","last_undegraded":"2024-03-31T15:03:36.989089+0000","last_fullsized":"2024-03-31T15:03:36.989089+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:16:22.583413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-31T15:03:36.974354+0000","last_change":"2024-03-31T15:03:36.965603+0000","last_active":"2024-03-31T15:03:36.974354+0000","last_peered":"2024-03-31T15:03:36.974354+0000","last_clean":"2024-03-31T15:03:36.974354+0000","last_became_active":"2024-03-31T15:03:36.965339+0000","last_became_peered":"2024-03-31T15:03:36.965339+0000","last_unstale":"2024-03-31T15:03:36.974354+0000","last_undegraded":"2024-03-31T15:03:36.974354+0000","last_fullsized":"2024-03-31T15:03:36.974354+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:11:51.069316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-31T15:03:36.984535+0000","last_change":"2024-03-31T15:03:36.977130+0000","last_active":"2024-03-31T15:03:36.984535+0000","last_peered":"2024-03-31T15:03:36.984535+0000","last_clean":"2024-03-31T15:03:36.984535+0000","last_became_active":"2024-03-31T15:03:36.973022+0000","last_became_peered":"2024-03-31T15:03:36.973022+0000","last_unstale":"2024-03-31T15:03:36.984535+0000","last_undegraded":"2024-03-31T15:03:36.984535+0000","last_fullsized":"2024-03-31T15:03:36.984535+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:48:54.968379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-31T15:03:36.975534+0000","last_change":"2024-03-31T15:03:36.967074+0000","last_active":"2024-03-31T15:03:36.975534+0000","last_peered":"2024-03-31T15:03:36.975534+0000","last_clean":"2024-03-31T15:03:36.975534+0000","last_became_active":"2024-03-31T15:03:36.966802+0000","last_became_peered":"2024-03-31T15:03:36.966802+0000","last_unstale":"2024-03-31T15:03:36.975534+0000","last_undegraded":"2024-03-31T15:03:36.975534+0000","last_fullsized":"2024-03-31T15:03:36.975534+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:27:39.263207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-31T15:03:36.986212+0000","last_change":"2024-03-31T15:03:36.977099+0000","last_active":"2024-03-31T15:03:36.986212+0000","last_peered":"2024-03-31T15:03:36.986212+0000","last_clean":"2024-03-31T15:03:36.986212+0000","last_became_active":"2024-03-31T15:03:36.976814+0000","last_became_peered":"2024-03-31T15:03:36.976814+0000","last_unstale":"2024-03-31T15:03:36.986212+0000","last_undegraded":"2024-03-31T15:03:36.986212+0000","last_fullsized":"2024-03-31T15:03:36.986212+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:27:39.263207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-31T15:03:36.973410+0000","last_change":"2024-03-31T15:03:36.968708+0000","last_active":"2024-03-31T15:03:36.973410+0000","last_peered":"2024-03-31T15:03:36.973410+0000","last_clean":"2024-03-31T15:03:36.973410+0000","last_became_active":"2024-03-31T15:03:36.968292+0000","last_became_peered":"2024-03-31T15:03:36.968292+0000","last_unstale":"2024-03-31T15:03:36.973410+0000","last_undegraded":"2024-03-31T15:03:36.973410+0000","last_fullsized":"2024-03-31T15:03:36.973410+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:11:51.069316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-31T15:03:36.977752+0000","last_change":"2024-03-31T15:03:36.970468+0000","last_active":"2024-03-31T15:03:36.977752+0000","last_peered":"2024-03-31T15:03:36.977752+0000","last_clean":"2024-03-31T15:03:36.977752+0000","last_became_active":"2024-03-31T15:03:36.970233+0000","last_became_peered":"2024-03-31T15:03:36.970233+0000","last_unstale":"2024-03-31T15:03:36.977752+0000","last_undegraded":"2024-03-31T15:03:36.977752+0000","last_fullsized":"2024-03-31T15:03:36.977752+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:29:57.089580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-31T15:03:36.975978+0000","last_change":"2024-03-31T15:03:36.962951+0000","last_active":"2024-03-31T15:03:36.975978+0000","last_peered":"2024-03-31T15:03:36.975978+0000","last_clean":"2024-03-31T15:03:36.975978+0000","last_became_active":"2024-03-31T15:03:36.961876+0000","last_became_peered":"2024-03-31T15:03:36.961876+0000","last_unstale":"2024-03-31T15:03:36.975978+0000","last_undegraded":"2024-03-31T15:03:36.975978+0000","last_fullsized":"2024-03-31T15:03:36.975978+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:18:04.322040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-31T15:03:36.988414+0000","last_change":"2024-03-31T15:03:36.979570+0000","last_active":"2024-03-31T15:03:36.988414+0000","last_peered":"2024-03-31T15:03:36.988414+0000","last_clean":"2024-03-31T15:03:36.988414+0000","last_became_active":"2024-03-31T15:03:36.979287+0000","last_became_peered":"2024-03-31T15:03:36.979287+0000","last_unstale":"2024-03-31T15:03:36.988414+0000","last_undegraded":"2024-03-31T15:03:36.988414+0000","last_fullsized":"2024-03-31T15:03:36.988414+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:55:58.515045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-31T15:03:36.989573+0000","last_change":"2024-03-31T15:03:36.988981+0000","last_active":"2024-03-31T15:03:36.989573+0000","last_peered":"2024-03-31T15:03:36.989573+0000","last_clean":"2024-03-31T15:03:36.989573+0000","last_became_active":"2024-03-31T15:03:36.988582+0000","last_became_peered":"2024-03-31T15:03:36.988582+0000","last_unstale":"2024-03-31T15:03:36.989573+0000","last_undegraded":"2024-03-31T15:03:36.989573+0000","last_fullsized":"2024-03-31T15:03:36.989573+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T15:58:19.498104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-31T15:03:36.984471+0000","last_change":"2024-03-31T15:03:36.975267+0000","last_active":"2024-03-31T15:03:36.984471+0000","last_peered":"2024-03-31T15:03:36.984471+0000","last_clean":"2024-03-31T15:03:36.984471+0000","last_became_active":"2024-03-31T15:03:36.972549+0000","last_became_peered":"2024-03-31T15:03:36.972549+0000","last_unstale":"2024-03-31T15:03:36.984471+0000","last_undegraded":"2024-03-31T15:03:36.984471+0000","last_fullsized":"2024-03-31T15:03:36.984471+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:11:51.069316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-31T15:03:37.966007+0000","last_change":"2024-03-31T15:03:36.964214+0000","last_active":"2024-03-31T15:03:37.966007+0000","last_peered":"2024-03-31T15:03:37.966007+0000","last_clean":"2024-03-31T15:03:37.966007+0000","last_became_active":"2024-03-31T15:03:36.963926+0000","last_became_peered":"2024-03-31T15:03:36.963926+0000","last_unstale":"2024-03-31T15:03:37.966007+0000","last_undegraded":"2024-03-31T15:03:37.966007+0000","last_fullsized":"2024-03-31T15:03:37.966007+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:26:54.177359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"20'1","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-31T15:03:36.990553+0000","last_change":"2024-03-31T15:03:35.966649+0000","last_active":"2024-03-31T15:03:36.990553+0000","last_peered":"2024-03-31T15:03:36.990553+0000","last_clean":"2024-03-31T15:03:36.990553+0000","last_became_active":"2024-03-31T15:03:35.964612+0000","last_became_peered":"2024-03-31T15:03:35.964612+0000","last_unstale":"2024-03-31T15:03:36.990553+0000","last_undegraded":"2024-03-31T15:03:36.990553+0000","last_fullsized":"2024-03-31T15:03:36.990553+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_clean_scrub_stamp":"2024-03-31T15:03:34.926145+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T15:38:42.097818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-31T15:03:36.962983+0000","last_change":"2024-03-31T15:03:36.961519+0000","last_active":"2024-03-31T15:03:36.962983+0000","last_peered":"2024-03-31T15:03:36.962983+0000","last_clean":"2024-03-31T15:03:36.962983+0000","last_became_active":"2024-03-31T15:03:36.961032+0000","last_became_peered":"2024-03-31T15:03:36.961032+0000","last_unstale":"2024-03-31T15:03:36.962983+0000","last_undegraded":"2024-03-31T15:03:36.962983+0000","last_fullsized":"2024-03-31T15:03:36.962983+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T15:15:06.651032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"20'1","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-31T15:03:36.992073+0000","last_change":"2024-03-31T15:03:35.961058+0000","last_active":"2024-03-31T15:03:36.992073+0000","last_peered":"2024-03-31T15:03:36.992073+0000","last_clean":"2024-03-31T15:03:36.992073+0000","last_became_active":"2024-03-31T15:03:35.960684+0000","last_became_peered":"2024-03-31T15:03:35.960684+0000","last_unstale":"2024-03-31T15:03:36.992073+0000","last_undegraded":"2024-03-31T15:03:36.992073+0000","last_fullsized":"2024-03-31T15:03:36.992073+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_clean_scrub_stamp":"2024-03-31T15:03:34.926145+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:07:06.210288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-31T15:03:36.954825+0000","last_change":"2024-03-31T15:03:35.963374+0000","last_active":"2024-03-31T15:03:36.954825+0000","last_peered":"2024-03-31T15:03:36.954825+0000","last_clean":"2024-03-31T15:03:36.954825+0000","last_became_active":"2024-03-31T15:03:35.963009+0000","last_became_peered":"2024-03-31T15:03:35.963009+0000","last_unstale":"2024-03-31T15:03:36.954825+0000","last_undegraded":"2024-03-31T15:03:36.954825+0000","last_fullsized":"2024-03-31T15:03:36.954825+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_clean_scrub_stamp":"2024-03-31T15:03:34.926145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:36:14.576891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-31T15:03:36.978932+0000","last_change":"2024-03-31T15:03:36.972478+0000","last_active":"2024-03-31T15:03:36.978932+0000","last_peered":"2024-03-31T15:03:36.978932+0000","last_clean":"2024-03-31T15:03:36.978932+0000","last_became_active":"2024-03-31T15:03:36.972297+0000","last_became_peered":"2024-03-31T15:03:36.972297+0000","last_unstale":"2024-03-31T15:03:36.978932+0000","last_undegraded":"2024-03-31T15:03:36.978932+0000","last_fullsized":"2024-03-31T15:03:36.978932+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:48:13.691269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-31T15:03:36.961621+0000","last_change":"2024-03-31T15:03:35.963562+0000","last_active":"2024-03-31T15:03:36.961621+0000","last_peered":"2024-03-31T15:03:36.961621+0000","last_clean":"2024-03-31T15:03:36.961621+0000","last_became_active":"2024-03-31T15:03:35.961838+0000","last_became_peered":"2024-03-31T15:03:35.961838+0000","last_unstale":"2024-03-31T15:03:36.961621+0000","last_undegraded":"2024-03-31T15:03:36.961621+0000","last_fullsized":"2024-03-31T15:03:36.961621+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_clean_scrub_stamp":"2024-03-31T15:03:34.926145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:27:19.823151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-31T15:03:36.974024+0000","last_change":"2024-03-31T15:03:36.970365+0000","last_active":"2024-03-31T15:03:36.974024+0000","last_peered":"2024-03-31T15:03:36.974024+0000","last_clean":"2024-03-31T15:03:36.974024+0000","last_became_active":"2024-03-31T15:03:36.970060+0000","last_became_peered":"2024-03-31T15:03:36.970060+0000","last_unstale":"2024-03-31T15:03:36.974024+0000","last_undegraded":"2024-03-31T15:03:36.974024+0000","last_fullsized":"2024-03-31T15:03:36.974024+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:27:39.263207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-31T15:03:36.955163+0000","last_change":"2024-03-31T15:03:35.956977+0000","last_active":"2024-03-31T15:03:36.955163+0000","last_peered":"2024-03-31T15:03:36.955163+0000","last_clean":"2024-03-31T15:03:36.955163+0000","last_became_active":"2024-03-31T15:03:35.954617+0000","last_became_peered":"2024-03-31T15:03:35.954617+0000","last_unstale":"2024-03-31T15:03:36.955163+0000","last_undegraded":"2024-03-31T15:03:36.955163+0000","last_fullsized":"2024-03-31T15:03:36.955163+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_clean_scrub_stamp":"2024-03-31T15:03:34.926145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T17:31:20.984335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-31T15:03:36.988694+0000","last_change":"2024-03-31T15:03:36.981437+0000","last_active":"2024-03-31T15:03:36.988694+0000","last_peered":"2024-03-31T15:03:36.988694+0000","last_clean":"2024-03-31T15:03:36.988694+0000","last_became_active":"2024-03-31T15:03:36.981050+0000","last_became_peered":"2024-03-31T15:03:36.981050+0000","last_unstale":"2024-03-31T15:03:36.988694+0000","last_undegraded":"2024-03-31T15:03:36.988694+0000","last_fullsized":"2024-03-31T15:03:36.988694+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T19:58:22.015705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-31T15:03:36.965706+0000","last_change":"2024-03-31T15:03:35.954630+0000","last_active":"2024-03-31T15:03:36.965706+0000","last_peered":"2024-03-31T15:03:36.965706+0000","last_clean":"2024-03-31T15:03:36.965706+0000","last_became_active":"2024-03-31T15:03:35.954149+0000","last_became_peered":"2024-03-31T15:03:35.954149+0000","last_unstale":"2024-03-31T15:03:36.965706+0000","last_undegraded":"2024-03-31T15:03:36.965706+0000","last_fullsized":"2024-03-31T15:03:36.965706+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_clean_scrub_stamp":"2024-03-31T15:03:34.926145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:23:12.843244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-31T15:03:36.989433+0000","last_change":"2024-03-31T15:03:36.988008+0000","last_active":"2024-03-31T15:03:36.989433+0000","last_peered":"2024-03-31T15:03:36.989433+0000","last_clean":"2024-03-31T15:03:36.989433+0000","last_became_active":"2024-03-31T15:03:36.987732+0000","last_became_peered":"2024-03-31T15:03:36.987732+0000","last_unstale":"2024-03-31T15:03:36.989433+0000","last_undegraded":"2024-03-31T15:03:36.989433+0000","last_fullsized":"2024-03-31T15:03:36.989433+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:51:11.582045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"20'3","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-31T15:03:36.991796+0000","last_change":"2024-03-31T15:03:35.959827+0000","last_active":"2024-03-31T15:03:36.991796+0000","last_peered":"2024-03-31T15:03:36.991796+0000","last_clean":"2024-03-31T15:03:36.991796+0000","last_became_active":"2024-03-31T15:03:35.959564+0000","last_became_peered":"2024-03-31T15:03:35.959564+0000","last_unstale":"2024-03-31T15:03:36.991796+0000","last_undegraded":"2024-03-31T15:03:36.991796+0000","last_fullsized":"2024-03-31T15:03:36.991796+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_clean_scrub_stamp":"2024-03-31T15:03:34.926145+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:36:14.576891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-31T15:03:37.966441+0000","last_change":"2024-03-31T15:03:36.982726+0000","last_active":"2024-03-31T15:03:37.966441+0000","last_peered":"2024-03-31T15:03:37.966441+0000","last_clean":"2024-03-31T15:03:37.966441+0000","last_became_active":"2024-03-31T15:03:36.981893+0000","last_became_peered":"2024-03-31T15:03:36.981893+0000","last_unstale":"2024-03-31T15:03:37.966441+0000","last_undegraded":"2024-03-31T15:03:37.966441+0000","last_fullsized":"2024-03-31T15:03:37.966441+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:19:24.339229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"20'6","reported_seq":18,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-31T15:03:36.994023+0000","last_change":"2024-03-31T15:03:35.970154+0000","last_active":"2024-03-31T15:03:36.994023+0000","last_peered":"2024-03-31T15:03:36.994023+0000","last_clean":"2024-03-31T15:03:36.994023+0000","last_became_active":"2024-03-31T15:03:35.969877+0000","last_became_peered":"2024-03-31T15:03:35.969877+0000","last_unstale":"2024-03-31T15:03:36.994023+0000","last_undegraded":"2024-03-31T15:03:36.994023+0000","last_fullsized":"2024-03-31T15:03:36.994023+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_clean_scrub_stamp":"2024-03-31T15:03:34.926145+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:36:14.576891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-31T15:03:36.986539+0000","last_change":"2024-03-31T15:03:36.980541+0000","last_active":"2024-03-31T15:03:36.986539+0000","last_peered":"2024-03-31T15:03:36.986539+0000","last_clean":"2024-03-31T15:03:36.986539+0000","last_became_active":"2024-03-31T15:03:36.980276+0000","last_became_peered":"2024-03-31T15:03:36.980276+0000","last_unstale":"2024-03-31T15:03:36.986539+0000","last_undegraded":"2024-03-31T15:03:36.986539+0000","last_fullsized":"2024-03-31T15:03:36.986539+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:53:00.086224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"20'2","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-31T15:03:36.992565+0000","last_change":"2024-03-31T15:03:35.960798+0000","last_active":"2024-03-31T15:03:36.992565+0000","last_peered":"2024-03-31T15:03:36.992565+0000","last_clean":"2024-03-31T15:03:36.992565+0000","last_became_active":"2024-03-31T15:03:35.960522+0000","last_became_peered":"2024-03-31T15:03:35.960522+0000","last_unstale":"2024-03-31T15:03:36.992565+0000","last_undegraded":"2024-03-31T15:03:36.992565+0000","last_fullsized":"2024-03-31T15:03:36.992565+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_clean_scrub_stamp":"2024-03-31T15:03:34.926145+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T16:07:33.984402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-31T15:03:36.975599+0000","last_change":"2024-03-31T15:03:36.972506+0000","last_active":"2024-03-31T15:03:36.975599+0000","last_peered":"2024-03-31T15:03:36.975599+0000","last_clean":"2024-03-31T15:03:36.975599+0000","last_became_active":"2024-03-31T15:03:36.972194+0000","last_became_peered":"2024-03-31T15:03:36.972194+0000","last_unstale":"2024-03-31T15:03:36.975599+0000","last_undegraded":"2024-03-31T15:03:36.975599+0000","last_fullsized":"2024-03-31T15:03:36.975599+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T17:31:21.989794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-31T15:03:36.967356+0000","last_change":"2024-03-31T15:03:35.966260+0000","last_active":"2024-03-31T15:03:36.967356+0000","last_peered":"2024-03-31T15:03:36.967356+0000","last_clean":"2024-03-31T15:03:36.967356+0000","last_became_active":"2024-03-31T15:03:35.965676+0000","last_became_peered":"2024-03-31T15:03:35.965676+0000","last_unstale":"2024-03-31T15:03:36.967356+0000","last_undegraded":"2024-03-31T15:03:36.967356+0000","last_fullsized":"2024-03-31T15:03:36.967356+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_clean_scrub_stamp":"2024-03-31T15:03:34.926145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:26:53.171901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-31T15:03:36.994700+0000","last_change":"2024-03-31T15:03:36.988567+0000","last_active":"2024-03-31T15:03:36.994700+0000","last_peered":"2024-03-31T15:03:36.994700+0000","last_clean":"2024-03-31T15:03:36.994700+0000","last_became_active":"2024-03-31T15:03:36.985668+0000","last_became_peered":"2024-03-31T15:03:36.985668+0000","last_unstale":"2024-03-31T15:03:36.994700+0000","last_undegraded":"2024-03-31T15:03:36.994700+0000","last_fullsized":"2024-03-31T15:03:36.994700+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:16:22.583413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-31T15:03:36.954637+0000","last_change":"2024-03-31T15:03:35.961562+0000","last_active":"2024-03-31T15:03:36.954637+0000","last_peered":"2024-03-31T15:03:36.954637+0000","last_clean":"2024-03-31T15:03:36.954637+0000","last_became_active":"2024-03-31T15:03:35.961280+0000","last_became_peered":"2024-03-31T15:03:35.961280+0000","last_unstale":"2024-03-31T15:03:36.954637+0000","last_undegraded":"2024-03-31T15:03:36.954637+0000","last_fullsized":"2024-03-31T15:03:36.954637+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_clean_scrub_stamp":"2024-03-31T15:03:34.926145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T15:38:42.097818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-31T15:03:36.986126+0000","last_change":"2024-03-31T15:03:36.979417+0000","last_active":"2024-03-31T15:03:36.986126+0000","last_peered":"2024-03-31T15:03:36.986126+0000","last_clean":"2024-03-31T15:03:36.986126+0000","last_became_active":"2024-03-31T15:03:36.973021+0000","last_became_peered":"2024-03-31T15:03:36.973021+0000","last_unstale":"2024-03-31T15:03:36.986126+0000","last_undegraded":"2024-03-31T15:03:36.986126+0000","last_fullsized":"2024-03-31T15:03:36.986126+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:29:47.722244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"20'8","reported_seq":20,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-31T15:03:36.988230+0000","last_change":"2024-03-31T15:03:35.955608+0000","last_active":"2024-03-31T15:03:36.988230+0000","last_peered":"2024-03-31T15:03:36.988230+0000","last_clean":"2024-03-31T15:03:36.988230+0000","last_became_active":"2024-03-31T15:03:35.955295+0000","last_became_peered":"2024-03-31T15:03:35.955295+0000","last_unstale":"2024-03-31T15:03:36.988230+0000","last_undegraded":"2024-03-31T15:03:36.988230+0000","last_fullsized":"2024-03-31T15:03:36.988230+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_clean_scrub_stamp":"2024-03-31T15:03:34.926145+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:23:12.843244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-31T15:03:36.994685+0000","last_change":"2024-03-31T15:03:36.990806+0000","last_active":"2024-03-31T15:03:36.994685+0000","last_peered":"2024-03-31T15:03:36.994685+0000","last_clean":"2024-03-31T15:03:36.994685+0000","last_became_active":"2024-03-31T15:03:36.990542+0000","last_became_peered":"2024-03-31T15:03:36.990542+0000","last_unstale":"2024-03-31T15:03:36.994685+0000","last_undegraded":"2024-03-31T15:03:36.994685+0000","last_fullsized":"2024-03-31T15:03:36.994685+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:11:06.260618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-31T15:03:37.966654+0000","last_change":"2024-03-31T15:03:35.968824+0000","last_active":"2024-03-31T15:03:37.966654+0000","last_peered":"2024-03-31T15:03:37.966654+0000","last_clean":"2024-03-31T15:03:37.966654+0000","last_became_active":"2024-03-31T15:03:35.968431+0000","last_became_peered":"2024-03-31T15:03:35.968431+0000","last_unstale":"2024-03-31T15:03:37.966654+0000","last_undegraded":"2024-03-31T15:03:37.966654+0000","last_fullsized":"2024-03-31T15:03:37.966654+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_clean_scrub_stamp":"2024-03-31T15:03:34.926145+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:30:58.752050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-31T15:03:36.998693+0000","last_change":"2024-03-31T15:03:36.993283+0000","last_active":"2024-03-31T15:03:36.998693+0000","last_peered":"2024-03-31T15:03:36.998693+0000","last_clean":"2024-03-31T15:03:36.998693+0000","last_became_active":"2024-03-31T15:03:36.992977+0000","last_became_peered":"2024-03-31T15:03:36.992977+0000","last_unstale":"2024-03-31T15:03:36.998693+0000","last_undegraded":"2024-03-31T15:03:36.998693+0000","last_fullsized":"2024-03-31T15:03:36.998693+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:34:49.982234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"20'2","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-31T15:03:36.983190+0000","last_change":"2024-03-31T15:03:35.969251+0000","last_active":"2024-03-31T15:03:36.983190+0000","last_peered":"2024-03-31T15:03:36.983190+0000","last_clean":"2024-03-31T15:03:36.983190+0000","last_became_active":"2024-03-31T15:03:35.968876+0000","last_became_peered":"2024-03-31T15:03:35.968876+0000","last_unstale":"2024-03-31T15:03:36.983190+0000","last_undegraded":"2024-03-31T15:03:36.983190+0000","last_fullsized":"2024-03-31T15:03:36.983190+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_clean_scrub_stamp":"2024-03-31T15:03:34.926145+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:27:19.823151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-31T15:03:36.993343+0000","last_change":"2024-03-31T15:03:36.986107+0000","last_active":"2024-03-31T15:03:36.993343+0000","last_peered":"2024-03-31T15:03:36.993343+0000","last_clean":"2024-03-31T15:03:36.993343+0000","last_became_active":"2024-03-31T15:03:36.985749+0000","last_became_peered":"2024-03-31T15:03:36.985749+0000","last_unstale":"2024-03-31T15:03:36.993343+0000","last_undegraded":"2024-03-31T15:03:36.993343+0000","last_fullsized":"2024-03-31T15:03:36.993343+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:05:28.499146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"20'1","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-31T15:03:36.993222+0000","last_change":"2024-03-31T15:03:35.965507+0000","last_active":"2024-03-31T15:03:36.993222+0000","last_peered":"2024-03-31T15:03:36.993222+0000","last_clean":"2024-03-31T15:03:36.993222+0000","last_became_active":"2024-03-31T15:03:35.964843+0000","last_became_peered":"2024-03-31T15:03:35.964843+0000","last_unstale":"2024-03-31T15:03:36.993222+0000","last_undegraded":"2024-03-31T15:03:36.993222+0000","last_fullsized":"2024-03-31T15:03:36.993222+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_clean_scrub_stamp":"2024-03-31T15:03:34.926145+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T17:31:20.984335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-31T15:03:36.988456+0000","last_change":"2024-03-31T15:03:36.979781+0000","last_active":"2024-03-31T15:03:36.988456+0000","last_peered":"2024-03-31T15:03:36.988456+0000","last_clean":"2024-03-31T15:03:36.988456+0000","last_became_active":"2024-03-31T15:03:36.979517+0000","last_became_peered":"2024-03-31T15:03:36.979517+0000","last_unstale":"2024-03-31T15:03:36.988456+0000","last_undegraded":"2024-03-31T15:03:36.988456+0000","last_fullsized":"2024-03-31T15:03:36.988456+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:02:20.135763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-31T15:03:36.966397+0000","last_change":"2024-03-31T15:03:35.960305+0000","last_active":"2024-03-31T15:03:36.966397+0000","last_peered":"2024-03-31T15:03:36.966397+0000","last_clean":"2024-03-31T15:03:36.966397+0000","last_became_active":"2024-03-31T15:03:35.960006+0000","last_became_peered":"2024-03-31T15:03:35.960006+0000","last_unstale":"2024-03-31T15:03:36.966397+0000","last_undegraded":"2024-03-31T15:03:36.966397+0000","last_fullsized":"2024-03-31T15:03:36.966397+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_clean_scrub_stamp":"2024-03-31T15:03:34.926145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:11:41.679652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-31T15:03:36.991744+0000","last_change":"2024-03-31T15:03:36.985221+0000","last_active":"2024-03-31T15:03:36.991744+0000","last_peered":"2024-03-31T15:03:36.991744+0000","last_clean":"2024-03-31T15:03:36.991744+0000","last_became_active":"2024-03-31T15:03:36.984963+0000","last_became_peered":"2024-03-31T15:03:36.984963+0000","last_unstale":"2024-03-31T15:03:36.991744+0000","last_undegraded":"2024-03-31T15:03:36.991744+0000","last_fullsized":"2024-03-31T15:03:36.991744+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:07:25.174977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":168,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-31T15:03:36.962575+0000","last_change":"2024-03-31T15:03:24.329890+0000","last_active":"2024-03-31T15:03:36.962575+0000","last_peered":"2024-03-31T15:03:36.962575+0000","last_clean":"2024-03-31T15:03:36.962575+0000","last_became_active":"2024-03-31T15:03:24.328882+0000","last_became_peered":"2024-03-31T15:03:24.328882+0000","last_unstale":"2024-03-31T15:03:36.962575+0000","last_undegraded":"2024-03-31T15:03:36.962575+0000","last_fullsized":"2024-03-31T15:03:36.962575+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:22.885535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:22.885535+0000","last_clean_scrub_stamp":"2024-03-31T15:03:22.885535+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:18:29.004296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-31T15:03:36.967122+0000","last_change":"2024-03-31T15:03:35.952977+0000","last_active":"2024-03-31T15:03:36.967122+0000","last_peered":"2024-03-31T15:03:36.967122+0000","last_clean":"2024-03-31T15:03:36.967122+0000","last_became_active":"2024-03-31T15:03:35.951288+0000","last_became_peered":"2024-03-31T15:03:35.951288+0000","last_unstale":"2024-03-31T15:03:36.967122+0000","last_undegraded":"2024-03-31T15:03:36.967122+0000","last_fullsized":"2024-03-31T15:03:36.967122+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_clean_scrub_stamp":"2024-03-31T15:03:34.926145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:11:41.679652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-31T15:03:36.985608+0000","last_change":"2024-03-31T15:03:36.980592+0000","last_active":"2024-03-31T15:03:36.985608+0000","last_peered":"2024-03-31T15:03:36.985608+0000","last_clean":"2024-03-31T15:03:36.985608+0000","last_became_active":"2024-03-31T15:03:36.980285+0000","last_became_peered":"2024-03-31T15:03:36.980285+0000","last_unstale":"2024-03-31T15:03:36.985608+0000","last_undegraded":"2024-03-31T15:03:36.985608+0000","last_fullsized":"2024-03-31T15:03:36.985608+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T17:15:21.722119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-31T15:03:36.962515+0000","last_change":"2024-03-31T15:03:35.968364+0000","last_active":"2024-03-31T15:03:36.962515+0000","last_peered":"2024-03-31T15:03:36.962515+0000","last_clean":"2024-03-31T15:03:36.962515+0000","last_became_active":"2024-03-31T15:03:35.968047+0000","last_became_peered":"2024-03-31T15:03:35.968047+0000","last_unstale":"2024-03-31T15:03:36.962515+0000","last_undegraded":"2024-03-31T15:03:36.962515+0000","last_fullsized":"2024-03-31T15:03:36.962515+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_clean_scrub_stamp":"2024-03-31T15:03:34.926145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:26:53.171901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-31T15:03:36.971139+0000","last_change":"2024-03-31T15:03:36.963584+0000","last_active":"2024-03-31T15:03:36.971139+0000","last_peered":"2024-03-31T15:03:36.971139+0000","last_clean":"2024-03-31T15:03:36.971139+0000","last_became_active":"2024-03-31T15:03:36.963216+0000","last_became_peered":"2024-03-31T15:03:36.963216+0000","last_unstale":"2024-03-31T15:03:36.971139+0000","last_undegraded":"2024-03-31T15:03:36.971139+0000","last_fullsized":"2024-03-31T15:03:36.971139+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:07:07.215747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-31T15:03:36.964813+0000","last_change":"2024-03-31T15:03:35.963196+0000","last_active":"2024-03-31T15:03:36.964813+0000","last_peered":"2024-03-31T15:03:36.964813+0000","last_clean":"2024-03-31T15:03:36.964813+0000","last_became_active":"2024-03-31T15:03:35.962746+0000","last_became_peered":"2024-03-31T15:03:35.962746+0000","last_unstale":"2024-03-31T15:03:36.964813+0000","last_undegraded":"2024-03-31T15:03:36.964813+0000","last_fullsized":"2024-03-31T15:03:36.964813+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_clean_scrub_stamp":"2024-03-31T15:03:34.926145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:00:32.840818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-31T15:03:36.973993+0000","last_change":"2024-03-31T15:03:36.972190+0000","last_active":"2024-03-31T15:03:36.973993+0000","last_peered":"2024-03-31T15:03:36.973993+0000","last_clean":"2024-03-31T15:03:36.973993+0000","last_became_active":"2024-03-31T15:03:36.971866+0000","last_became_peered":"2024-03-31T15:03:36.971866+0000","last_unstale":"2024-03-31T15:03:36.973993+0000","last_undegraded":"2024-03-31T15:03:36.973993+0000","last_fullsized":"2024-03-31T15:03:36.973993+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:29:47.722244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-31T15:03:36.967713+0000","last_change":"2024-03-31T15:03:35.953803+0000","last_active":"2024-03-31T15:03:36.967713+0000","last_peered":"2024-03-31T15:03:36.967713+0000","last_clean":"2024-03-31T15:03:36.967713+0000","last_became_active":"2024-03-31T15:03:35.953459+0000","last_became_peered":"2024-03-31T15:03:35.953459+0000","last_unstale":"2024-03-31T15:03:36.967713+0000","last_undegraded":"2024-03-31T15:03:36.967713+0000","last_fullsized":"2024-03-31T15:03:36.967713+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_clean_scrub_stamp":"2024-03-31T15:03:34.926145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T15:58:18.492645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-31T15:03:37.966372+0000","last_change":"2024-03-31T15:03:36.966599+0000","last_active":"2024-03-31T15:03:37.966372+0000","last_peered":"2024-03-31T15:03:37.966372+0000","last_clean":"2024-03-31T15:03:37.966372+0000","last_became_active":"2024-03-31T15:03:36.964867+0000","last_became_peered":"2024-03-31T15:03:36.964867+0000","last_unstale":"2024-03-31T15:03:37.966372+0000","last_undegraded":"2024-03-31T15:03:37.966372+0000","last_fullsized":"2024-03-31T15:03:37.966372+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:48:54.968379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-31T15:03:36.956046+0000","last_change":"2024-03-31T15:03:35.958174+0000","last_active":"2024-03-31T15:03:36.956046+0000","last_peered":"2024-03-31T15:03:36.956046+0000","last_clean":"2024-03-31T15:03:36.956046+0000","last_became_active":"2024-03-31T15:03:35.957870+0000","last_became_peered":"2024-03-31T15:03:35.957870+0000","last_unstale":"2024-03-31T15:03:36.956046+0000","last_undegraded":"2024-03-31T15:03:36.956046+0000","last_fullsized":"2024-03-31T15:03:36.956046+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_clean_scrub_stamp":"2024-03-31T15:03:34.926145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T15:38:42.097818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-31T15:03:36.989207+0000","last_change":"2024-03-31T15:03:36.980337+0000","last_active":"2024-03-31T15:03:36.989207+0000","last_peered":"2024-03-31T15:03:36.989207+0000","last_clean":"2024-03-31T15:03:36.989207+0000","last_became_active":"2024-03-31T15:03:36.980082+0000","last_became_peered":"2024-03-31T15:03:36.980082+0000","last_unstale":"2024-03-31T15:03:36.989207+0000","last_undegraded":"2024-03-31T15:03:36.989207+0000","last_fullsized":"2024-03-31T15:03:36.989207+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:53:00.086224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-31T15:03:36.964126+0000","last_change":"2024-03-31T15:03:35.969598+0000","last_active":"2024-03-31T15:03:36.964126+0000","last_peered":"2024-03-31T15:03:36.964126+0000","last_clean":"2024-03-31T15:03:36.964126+0000","last_became_active":"2024-03-31T15:03:35.969253+0000","last_became_peered":"2024-03-31T15:03:35.969253+0000","last_unstale":"2024-03-31T15:03:36.964126+0000","last_undegraded":"2024-03-31T15:03:36.964126+0000","last_fullsized":"2024-03-31T15:03:36.964126+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_clean_scrub_stamp":"2024-03-31T15:03:34.926145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:55:57.509587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-31T15:03:36.987586+0000","last_change":"2024-03-31T15:03:36.983933+0000","last_active":"2024-03-31T15:03:36.987586+0000","last_peered":"2024-03-31T15:03:36.987586+0000","last_clean":"2024-03-31T15:03:36.987586+0000","last_became_active":"2024-03-31T15:03:36.983650+0000","last_became_peered":"2024-03-31T15:03:36.983650+0000","last_unstale":"2024-03-31T15:03:36.987586+0000","last_undegraded":"2024-03-31T15:03:36.987586+0000","last_fullsized":"2024-03-31T15:03:36.987586+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:37:26.685300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-31T15:03:36.958497+0000","last_change":"2024-03-31T15:03:35.959254+0000","last_active":"2024-03-31T15:03:36.958497+0000","last_peered":"2024-03-31T15:03:36.958497+0000","last_clean":"2024-03-31T15:03:36.958497+0000","last_became_active":"2024-03-31T15:03:35.958903+0000","last_became_peered":"2024-03-31T15:03:35.958903+0000","last_unstale":"2024-03-31T15:03:36.958497+0000","last_undegraded":"2024-03-31T15:03:36.958497+0000","last_fullsized":"2024-03-31T15:03:36.958497+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_clean_scrub_stamp":"2024-03-31T15:03:34.926145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T15:07:57.066606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-31T15:03:36.978714+0000","last_change":"2024-03-31T15:03:36.973216+0000","last_active":"2024-03-31T15:03:36.978714+0000","last_peered":"2024-03-31T15:03:36.978714+0000","last_clean":"2024-03-31T15:03:36.978714+0000","last_became_active":"2024-03-31T15:03:36.972719+0000","last_became_peered":"2024-03-31T15:03:36.972719+0000","last_unstale":"2024-03-31T15:03:36.978714+0000","last_undegraded":"2024-03-31T15:03:36.978714+0000","last_fullsized":"2024-03-31T15:03:36.978714+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T19:12:45.358470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"20'2","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-31T15:03:36.991987+0000","last_change":"2024-03-31T15:03:35.964289+0000","last_active":"2024-03-31T15:03:36.991987+0000","last_peered":"2024-03-31T15:03:36.991987+0000","last_clean":"2024-03-31T15:03:36.991987+0000","last_became_active":"2024-03-31T15:03:35.963897+0000","last_became_peered":"2024-03-31T15:03:35.963897+0000","last_unstale":"2024-03-31T15:03:36.991987+0000","last_undegraded":"2024-03-31T15:03:36.991987+0000","last_fullsized":"2024-03-31T15:03:36.991987+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_clean_scrub_stamp":"2024-03-31T15:03:34.926145+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T15:07:57.066606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-31T15:03:36.971022+0000","last_change":"2024-03-31T15:03:36.967603+0000","last_active":"2024-03-31T15:03:36.971022+0000","last_peered":"2024-03-31T15:03:36.971022+0000","last_clean":"2024-03-31T15:03:36.971022+0000","last_became_active":"2024-03-31T15:03:36.967206+0000","last_became_peered":"2024-03-31T15:03:36.967206+0000","last_unstale":"2024-03-31T15:03:36.971022+0000","last_undegraded":"2024-03-31T15:03:36.971022+0000","last_fullsized":"2024-03-31T15:03:36.971022+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:36:15.582349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-31T15:03:36.962355+0000","last_change":"2024-03-31T15:03:35.967340+0000","last_active":"2024-03-31T15:03:36.962355+0000","last_peered":"2024-03-31T15:03:36.962355+0000","last_clean":"2024-03-31T15:03:36.962355+0000","last_became_active":"2024-03-31T15:03:35.967046+0000","last_became_peered":"2024-03-31T15:03:35.967046+0000","last_unstale":"2024-03-31T15:03:36.962355+0000","last_undegraded":"2024-03-31T15:03:36.962355+0000","last_fullsized":"2024-03-31T15:03:36.962355+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_clean_scrub_stamp":"2024-03-31T15:03:34.926145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:11:41.679652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-31T15:03:37.965991+0000","last_change":"2024-03-31T15:03:36.991196+0000","last_active":"2024-03-31T15:03:37.965991+0000","last_peered":"2024-03-31T15:03:37.965991+0000","last_clean":"2024-03-31T15:03:37.965991+0000","last_became_active":"2024-03-31T15:03:36.990914+0000","last_became_peered":"2024-03-31T15:03:36.990914+0000","last_unstale":"2024-03-31T15:03:37.965991+0000","last_undegraded":"2024-03-31T15:03:37.965991+0000","last_fullsized":"2024-03-31T15:03:37.965991+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:11:42.685111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-31T15:03:36.955854+0000","last_change":"2024-03-31T15:03:35.969671+0000","last_active":"2024-03-31T15:03:36.955854+0000","last_peered":"2024-03-31T15:03:36.955854+0000","last_clean":"2024-03-31T15:03:36.955854+0000","last_became_active":"2024-03-31T15:03:35.969412+0000","last_became_peered":"2024-03-31T15:03:35.969412+0000","last_unstale":"2024-03-31T15:03:36.955854+0000","last_undegraded":"2024-03-31T15:03:36.955854+0000","last_fullsized":"2024-03-31T15:03:36.955854+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_clean_scrub_stamp":"2024-03-31T15:03:34.926145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:07:06.210288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-31T15:03:36.965438+0000","last_change":"2024-03-31T15:03:36.959849+0000","last_active":"2024-03-31T15:03:36.965438+0000","last_peered":"2024-03-31T15:03:36.965438+0000","last_clean":"2024-03-31T15:03:36.965438+0000","last_became_active":"2024-03-31T15:03:36.959611+0000","last_became_peered":"2024-03-31T15:03:36.959611+0000","last_unstale":"2024-03-31T15:03:36.965438+0000","last_undegraded":"2024-03-31T15:03:36.965438+0000","last_fullsized":"2024-03-31T15:03:36.965438+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T15:07:58.072065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-31T15:03:36.958718+0000","last_change":"2024-03-31T15:03:35.963705+0000","last_active":"2024-03-31T15:03:36.958718+0000","last_peered":"2024-03-31T15:03:36.958718+0000","last_clean":"2024-03-31T15:03:36.958718+0000","last_became_active":"2024-03-31T15:03:35.963241+0000","last_became_peered":"2024-03-31T15:03:35.963241+0000","last_unstale":"2024-03-31T15:03:36.958718+0000","last_undegraded":"2024-03-31T15:03:36.958718+0000","last_fullsized":"2024-03-31T15:03:36.958718+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_clean_scrub_stamp":"2024-03-31T15:03:34.926145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T16:00:45.563865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-31T15:03:36.979092+0000","last_change":"2024-03-31T15:03:36.970959+0000","last_active":"2024-03-31T15:03:36.979092+0000","last_peered":"2024-03-31T15:03:36.979092+0000","last_clean":"2024-03-31T15:03:36.979092+0000","last_became_active":"2024-03-31T15:03:36.970431+0000","last_became_peered":"2024-03-31T15:03:36.970431+0000","last_unstale":"2024-03-31T15:03:36.979092+0000","last_undegraded":"2024-03-31T15:03:36.979092+0000","last_fullsized":"2024-03-31T15:03:36.979092+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:37:26.685300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"20'2","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-31T15:03:36.991659+0000","last_change":"2024-03-31T15:03:35.964624+0000","last_active":"2024-03-31T15:03:36.991659+0000","last_peered":"2024-03-31T15:03:36.991659+0000","last_clean":"2024-03-31T15:03:36.991659+0000","last_became_active":"2024-03-31T15:03:35.964307+0000","last_became_peered":"2024-03-31T15:03:35.964307+0000","last_unstale":"2024-03-31T15:03:36.991659+0000","last_undegraded":"2024-03-31T15:03:36.991659+0000","last_fullsized":"2024-03-31T15:03:36.991659+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_clean_scrub_stamp":"2024-03-31T15:03:34.926145+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T15:58:18.492645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-31T15:03:36.980813+0000","last_change":"2024-03-31T15:03:36.969447+0000","last_active":"2024-03-31T15:03:36.980813+0000","last_peered":"2024-03-31T15:03:36.980813+0000","last_clean":"2024-03-31T15:03:36.980813+0000","last_became_active":"2024-03-31T15:03:36.967372+0000","last_became_peered":"2024-03-31T15:03:36.967372+0000","last_unstale":"2024-03-31T15:03:36.980813+0000","last_undegraded":"2024-03-31T15:03:36.980813+0000","last_fullsized":"2024-03-31T15:03:36.980813+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:51:11.582045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-31T15:03:36.989229+0000","last_change":"2024-03-31T15:03:36.980538+0000","last_active":"2024-03-31T15:03:36.989229+0000","last_peered":"2024-03-31T15:03:36.989229+0000","last_clean":"2024-03-31T15:03:36.989229+0000","last_became_active":"2024-03-31T15:03:36.980255+0000","last_became_peered":"2024-03-31T15:03:36.980255+0000","last_unstale":"2024-03-31T15:03:36.989229+0000","last_undegraded":"2024-03-31T15:03:36.989229+0000","last_fullsized":"2024-03-31T15:03:36.989229+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T16:25:42.516431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"20'2","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-31T15:03:36.985931+0000","last_change":"2024-03-31T15:03:35.962539+0000","last_active":"2024-03-31T15:03:36.985931+0000","last_peered":"2024-03-31T15:03:36.985931+0000","last_clean":"2024-03-31T15:03:36.985931+0000","last_became_active":"2024-03-31T15:03:35.961927+0000","last_became_peered":"2024-03-31T15:03:35.961927+0000","last_unstale":"2024-03-31T15:03:36.985931+0000","last_undegraded":"2024-03-31T15:03:36.985931+0000","last_fullsized":"2024-03-31T15:03:36.985931+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_clean_scrub_stamp":"2024-03-31T15:03:34.926145+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:48:53.962920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-31T15:03:36.989057+0000","last_change":"2024-03-31T15:03:36.981810+0000","last_active":"2024-03-31T15:03:36.989057+0000","last_peered":"2024-03-31T15:03:36.989057+0000","last_clean":"2024-03-31T15:03:36.989057+0000","last_became_active":"2024-03-31T15:03:36.981486+0000","last_became_peered":"2024-03-31T15:03:36.981486+0000","last_unstale":"2024-03-31T15:03:36.989057+0000","last_undegraded":"2024-03-31T15:03:36.989057+0000","last_fullsized":"2024-03-31T15:03:36.989057+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:26:15.339683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"20'2","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-31T15:03:36.991675+0000","last_change":"2024-03-31T15:03:35.972149+0000","last_active":"2024-03-31T15:03:36.991675+0000","last_peered":"2024-03-31T15:03:36.991675+0000","last_clean":"2024-03-31T15:03:36.991675+0000","last_became_active":"2024-03-31T15:03:35.971844+0000","last_became_peered":"2024-03-31T15:03:35.971844+0000","last_unstale":"2024-03-31T15:03:36.991675+0000","last_undegraded":"2024-03-31T15:03:36.991675+0000","last_fullsized":"2024-03-31T15:03:36.991675+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_clean_scrub_stamp":"2024-03-31T15:03:34.926145+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:19:23.333770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-31T15:03:36.989455+0000","last_change":"2024-03-31T15:03:36.988093+0000","last_active":"2024-03-31T15:03:36.989455+0000","last_peered":"2024-03-31T15:03:36.989455+0000","last_clean":"2024-03-31T15:03:36.989455+0000","last_became_active":"2024-03-31T15:03:36.987794+0000","last_became_peered":"2024-03-31T15:03:36.987794+0000","last_unstale":"2024-03-31T15:03:36.989455+0000","last_undegraded":"2024-03-31T15:03:36.989455+0000","last_fullsized":"2024-03-31T15:03:36.989455+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:11:51.069316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"20'1","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-31T15:03:36.986305+0000","last_change":"2024-03-31T15:03:35.969099+0000","last_active":"2024-03-31T15:03:36.986305+0000","last_peered":"2024-03-31T15:03:36.986305+0000","last_clean":"2024-03-31T15:03:36.986305+0000","last_became_active":"2024-03-31T15:03:35.968350+0000","last_became_peered":"2024-03-31T15:03:35.968350+0000","last_unstale":"2024-03-31T15:03:36.986305+0000","last_undegraded":"2024-03-31T15:03:36.986305+0000","last_fullsized":"2024-03-31T15:03:36.986305+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_clean_scrub_stamp":"2024-03-31T15:03:34.926145+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T17:31:20.984335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-31T15:03:36.965360+0000","last_change":"2024-03-31T15:03:36.960526+0000","last_active":"2024-03-31T15:03:36.965360+0000","last_peered":"2024-03-31T15:03:36.965360+0000","last_clean":"2024-03-31T15:03:36.965360+0000","last_became_active":"2024-03-31T15:03:36.960245+0000","last_became_peered":"2024-03-31T15:03:36.960245+0000","last_unstale":"2024-03-31T15:03:36.965360+0000","last_undegraded":"2024-03-31T15:03:36.965360+0000","last_fullsized":"2024-03-31T15:03:36.965360+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:51:11.582045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"20'4","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-31T15:03:36.987133+0000","last_change":"2024-03-31T15:03:35.965202+0000","last_active":"2024-03-31T15:03:36.987133+0000","last_peered":"2024-03-31T15:03:36.987133+0000","last_clean":"2024-03-31T15:03:36.987133+0000","last_became_active":"2024-03-31T15:03:35.964867+0000","last_became_peered":"2024-03-31T15:03:35.964867+0000","last_unstale":"2024-03-31T15:03:36.987133+0000","last_undegraded":"2024-03-31T15:03:36.987133+0000","last_fullsized":"2024-03-31T15:03:36.987133+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_clean_scrub_stamp":"2024-03-31T15:03:34.926145+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T16:35:11.030977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509447,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27804,"kb_used_data":776,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344036,"statfs":{"total":96636764160,"available":96608292864,"internally_reserved":0,"allocated":794624,"data_stored":657545,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509446,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27216,"kb_used_data":196,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344624,"statfs":{"total":96636764160,"available":96608894976,"internally_reserved":0,"allocated":200704,"data_stored":66685,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":4,"up_from":15,"seq":64424509445,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":221184,"data_stored":73021,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509444,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31372,"kb_used_data":192,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340468,"statfs":{"total":96636764160,"available":96604639232,"internally_reserved":0,"allocated":196608,"data_stored":66595,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542150,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27200,"kb_used_data":180,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344640,"statfs":{"total":96636764160,"available":96608911360,"internally_reserved":0,"allocated":184320,"data_stored":59820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542150,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31360,"kb_used_data":180,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340480,"statfs":{"total":96636764160,"available":96604651520,"internally_reserved":0,"allocated":184320,"data_stored":59820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542150,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31384,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340456,"statfs":{"total":96636764160,"available":96604626944,"internally_reserved":0,"allocated":204800,"data_stored":66755,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542150,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31972,"kb_used_data":784,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339868,"statfs":{"total":96636764160,"available":96604024832,"internally_reserved":0,"allocated":802816,"data_stored":657601,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-31T15:03:41.084 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-31T15:03:41.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:41.265+0000 7fcf85400640 1 Processor -- start 2024-03-31T15:03:41.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:41.265+0000 7fcf85400640 1 -- start start 2024-03-31T15:03:41.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:41.265+0000 7fcf85400640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fcf80158700 0x7fcf800706d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:41.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:41.265+0000 7fcf85400640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcf8006e970 con 0x7fcf8006e4c0 2024-03-31T15:03:41.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:41.265+0000 7fcf85400640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcf8006ec30 con 0x7fcf8006b620 2024-03-31T15:03:41.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:41.265+0000 7fcf85400640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7fcf8015ec90 con 0x7fcf80158700 2024-03-31T15:03:41.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:41.265+0000 7fcf7fe00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fcf80158700 0x7fcf800706d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:41.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:41.265+0000 7fcf7f400640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fcf8006e4c0 0x7fcf8015f120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:33946/0 (socket says 172.21.15.16:33946) 2024-03-31T15:03:41.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:41.265+0000 7fcf7f400640 1 -- 172.21.15.16:0/2840161834 learned_addr learned my addr 172.21.15.16:0/2840161834 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:03:41.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:41.265+0000 7fcf7fe00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fcf80158700 0x7fcf800706d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38884/0 (socket says 172.21.15.16:38884) 2024-03-31T15:03:41.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:41.265+0000 7fcf7fe00640 1 -- 172.21.15.16:0/2840161834 >> v1:172.21.15.89:6789/0 conn(0x7fcf8006b620 legacy=0x7fcf8006ba30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:41.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:41.265+0000 7fcf7fe00640 1 -- 172.21.15.16:0/2840161834 >> v1:172.21.15.16:6789/0 conn(0x7fcf8006e4c0 legacy=0x7fcf8015f120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:41.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:41.265+0000 7fcf7fe00640 1 -- 172.21.15.16:0/2840161834 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcf8006ef10 con 0x7fcf80158700 2024-03-31T15:03:41.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:41.269+0000 7fcf7fe00640 1 --2- 172.21.15.16:0/2840161834 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcf80158700 0x7fcf800706d0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fcf700097c0 tx=0x7fcf7001c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9c0839014fb72f0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:03:41.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:41.269+0000 7fcf7ea00640 1 -- 172.21.15.16:0/2840161834 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf700b0070 con 0x7fcf80158700 2024-03-31T15:03:41.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:41.269+0000 7fcf7ea00640 1 -- 172.21.15.16:0/2840161834 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcf700ab600 con 0x7fcf80158700 2024-03-31T15:03:41.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:41.269+0000 7fcf7ea00640 1 -- 172.21.15.16:0/2840161834 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf700ab9d0 con 0x7fcf80158700 2024-03-31T15:03:41.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:41.269+0000 7fcf85400640 1 -- 172.21.15.16:0/2840161834 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcf80158700 msgr2=0x7fcf800706d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:41.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:41.269+0000 7fcf85400640 1 --2- 172.21.15.16:0/2840161834 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcf80158700 0x7fcf800706d0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fcf700097c0 tx=0x7fcf7001c4a0 comp rx=0 tx=0).stop 2024-03-31T15:03:41.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:41.269+0000 7fcf85400640 1 -- 172.21.15.16:0/2840161834 shutdown_connections 2024-03-31T15:03:41.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:41.269+0000 7fcf85400640 1 --2- 172.21.15.16:0/2840161834 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcf80158700 0x7fcf800706d0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:41.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:41.269+0000 7fcf85400640 1 -- 172.21.15.16:0/2840161834 >> 172.21.15.16:0/2840161834 conn(0x7fcf800b2b30 msgr2=0x7fcf800b2f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:03:41.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:41.269+0000 7fcf85400640 1 -- 172.21.15.16:0/2840161834 shutdown_connections 2024-03-31T15:03:41.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:41.269+0000 7fcf85400640 1 -- 172.21.15.16:0/2840161834 wait complete. 2024-03-31T15:03:41.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:41.269+0000 7fcf85400640 1 Processor -- start 2024-03-31T15:03:41.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:41.269+0000 7fcf85400640 1 -- start start 2024-03-31T15:03:41.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:41.273+0000 7fcf85400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcf8006b620 0x7fcf8014ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:41.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:41.273+0000 7fcf85400640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fcf8006e4c0 0x7fcf80146aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:41.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:41.273+0000 7fcf85400640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcf80158700 0x7fcf800b1380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:41.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:41.273+0000 7fcf85400640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fcf80072460 con 0x7fcf8006b620 2024-03-31T15:03:41.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:41.273+0000 7fcf85400640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fcf800d0c10 con 0x7fcf80158700 2024-03-31T15:03:41.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:41.273+0000 7fcf85400640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fcf800725e0 con 0x7fcf8006e4c0 2024-03-31T15:03:41.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:41.273+0000 7fcf7fe00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcf8006b620 0x7fcf8014ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:41.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:41.273+0000 7fcf7fe00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcf8006b620 0x7fcf8014ec80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38892/0 (socket says 172.21.15.16:38892) 2024-03-31T15:03:41.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:41.273+0000 7fcf7fe00640 1 -- 172.21.15.16:0/2382747471 learned_addr learned my addr 172.21.15.16:0/2382747471 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:03:41.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:41.273+0000 7fcf7f400640 1 --2- 172.21.15.16:0/2382747471 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fcf8006e4c0 0x7fcf80146aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:41.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:41.273+0000 7fcf84a00640 1 --2- 172.21.15.16:0/2382747471 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcf80158700 0x7fcf800b1380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:41.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:41.273+0000 7fcf7fe00640 1 -- 172.21.15.16:0/2382747471 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fcf8006e4c0 msgr2=0x7fcf80146aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:41.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:41.273+0000 7fcf7fe00640 1 --2- 172.21.15.16:0/2382747471 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fcf8006e4c0 0x7fcf80146aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:41.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:41.273+0000 7fcf7fe00640 1 -- 172.21.15.16:0/2382747471 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcf80158700 msgr2=0x7fcf800b1380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:41.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:41.273+0000 7fcf7fe00640 1 --2- 172.21.15.16:0/2382747471 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcf80158700 0x7fcf800b1380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:41.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:41.273+0000 7fcf7fe00640 1 -- 172.21.15.16:0/2382747471 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcf8014f1c0 con 0x7fcf8006b620 2024-03-31T15:03:41.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:41.273+0000 7fcf7fe00640 1 --2- 172.21.15.16:0/2382747471 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcf8006b620 0x7fcf8014ec80 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fcf700ae040 tx=0x7fcf70019240 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:03:41.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:41.273+0000 7fcf77e00640 1 -- 172.21.15.16:0/2382747471 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf700b0040 con 0x7fcf8006b620 2024-03-31T15:03:41.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:41.273+0000 7fcf77e00640 1 -- 172.21.15.16:0/2382747471 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcf700b4410 con 0x7fcf8006b620 2024-03-31T15:03:41.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:41.273+0000 7fcf77e00640 1 -- 172.21.15.16:0/2382747471 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf700b4820 con 0x7fcf8006b620 2024-03-31T15:03:41.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:41.273+0000 7fcf85400640 1 -- 172.21.15.16:0/2382747471 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcf800b2220 con 0x7fcf8006b620 2024-03-31T15:03:41.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:41.273+0000 7fcf85400640 1 -- 172.21.15.16:0/2382747471 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcf801619f0 con 0x7fcf8006b620 2024-03-31T15:03:41.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:41.273+0000 7fcf85400640 1 -- 172.21.15.16:0/2382747471 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcf801617e0 con 0x7fcf8006b620 2024-03-31T15:03:41.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:41.273+0000 7fcf77e00640 1 -- 172.21.15.16:0/2382747471 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fcf700bc020 con 0x7fcf8006b620 2024-03-31T15:03:41.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:41.277+0000 7fcf77e00640 1 --2- 172.21.15.16:0/2382747471 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fcf60068380 0x7fcf6006a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:41.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:41.277+0000 7fcf77e00640 1 -- 172.21.15.16:0/2382747471 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fcf7011fd40 con 0x7fcf8006b620 2024-03-31T15:03:41.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:41.277+0000 7fcf7f400640 1 --2- 172.21.15.16:0/2382747471 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fcf60068380 0x7fcf6006a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:41.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:41.277+0000 7fcf7f400640 1 --2- 172.21.15.16:0/2382747471 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fcf60068380 0x7fcf6006a840 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fcf8015e2c0 tx=0x7fcf6c009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:03:41.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:41.281+0000 7fcf77e00640 1 -- 172.21.15.16:0/2382747471 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fcf700c3020 con 0x7fcf8006b620 2024-03-31T15:03:41.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:41.445+0000 7fcf85400640 1 -- 172.21.15.16:0/2382747471 --> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fcf8004f8c0 con 0x7fcf60068380 2024-03-31T15:03:41.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:41.449+0000 7fcf77e00640 1 -- 172.21.15.16:0/2382747471 <== mgr.4100 v2:172.21.15.89:6832/1549407273 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251885 (secure 0 0 0) 0x7fcf8004f8c0 con 0x7fcf60068380 2024-03-31T15:03:41.450 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:03:41.452 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-03-31T15:03:41.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:41.453+0000 7fcf85400640 1 -- 172.21.15.16:0/2382747471 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fcf60068380 msgr2=0x7fcf6006a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:41.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:41.453+0000 7fcf85400640 1 --2- 172.21.15.16:0/2382747471 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fcf60068380 0x7fcf6006a840 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fcf8015e2c0 tx=0x7fcf6c009040 comp rx=0 tx=0).stop 2024-03-31T15:03:41.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:41.453+0000 7fcf85400640 1 -- 172.21.15.16:0/2382747471 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcf8006b620 msgr2=0x7fcf8014ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:41.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:41.453+0000 7fcf85400640 1 --2- 172.21.15.16:0/2382747471 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcf8006b620 0x7fcf8014ec80 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fcf700ae040 tx=0x7fcf70019240 comp rx=0 tx=0).stop 2024-03-31T15:03:41.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:41.453+0000 7fcf85400640 1 -- 172.21.15.16:0/2382747471 shutdown_connections 2024-03-31T15:03:41.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:41.453+0000 7fcf85400640 1 --2- 172.21.15.16:0/2382747471 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fcf60068380 0x7fcf6006a840 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:41.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:41.453+0000 7fcf85400640 1 --2- 172.21.15.16:0/2382747471 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcf80158700 0x7fcf800b1380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:41.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:41.453+0000 7fcf85400640 1 --2- 172.21.15.16:0/2382747471 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fcf8006e4c0 0x7fcf80146aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:41.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:41.453+0000 7fcf85400640 1 --2- 172.21.15.16:0/2382747471 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcf8006b620 0x7fcf8014ec80 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:41.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:41.453+0000 7fcf85400640 1 -- 172.21.15.16:0/2382747471 >> 172.21.15.16:0/2382747471 conn(0x7fcf800b2b30 msgr2=0x7fcf8006ba00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:03:41.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:41.453+0000 7fcf85400640 1 -- 172.21.15.16:0/2382747471 shutdown_connections 2024-03-31T15:03:41.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:41.453+0000 7fcf85400640 1 -- 172.21.15.16:0/2382747471 wait complete. 2024-03-31T15:03:41.470 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2024-03-31T15:03:40.530461+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":103,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":235548,"kb_used_data":2724,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754739172,"statfs":{"total":773094113280,"available":772852912128,"internally_reserved":0,"allocated":2789376,"data_stored":1707842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":238276144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":37,"apply_latency_ms":37,"commit_latency_ns":37000000,"apply_latency_ns":37000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.596594"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-31T15:03:36.993363+0000","last_change":"2024-03-31T15:03:36.984154+0000","last_active":"2024-03-31T15:03:36.993363+0000","last_peered":"2024-03-31T15:03:36.993363+0000","last_clean":"2024-03-31T15:03:36.993363+0000","last_became_active":"2024-03-31T15:03:36.983712+0000","last_became_peered":"2024-03-31T15:03:36.983712+0000","last_unstale":"2024-03-31T15:03:36.993363+0000","last_undegraded":"2024-03-31T15:03:36.993363+0000","last_fullsized":"2024-03-31T15:03:36.993363+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T16:25:42.516431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-31T15:03:36.966594+0000","last_change":"2024-03-31T15:03:36.962443+0000","last_active":"2024-03-31T15:03:36.966594+0000","last_peered":"2024-03-31T15:03:36.966594+0000","last_clean":"2024-03-31T15:03:36.966594+0000","last_became_active":"2024-03-31T15:03:36.962140+0000","last_became_peered":"2024-03-31T15:03:36.962140+0000","last_unstale":"2024-03-31T15:03:36.966594+0000","last_undegraded":"2024-03-31T15:03:36.966594+0000","last_fullsized":"2024-03-31T15:03:36.966594+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:51:11.582045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-31T15:03:36.986728+0000","last_change":"2024-03-31T15:03:36.985502+0000","last_active":"2024-03-31T15:03:36.986728+0000","last_peered":"2024-03-31T15:03:36.986728+0000","last_clean":"2024-03-31T15:03:36.986728+0000","last_became_active":"2024-03-31T15:03:36.985263+0000","last_became_peered":"2024-03-31T15:03:36.985263+0000","last_unstale":"2024-03-31T15:03:36.986728+0000","last_undegraded":"2024-03-31T15:03:36.986728+0000","last_fullsized":"2024-03-31T15:03:36.986728+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T16:00:46.569324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-31T15:03:36.988475+0000","last_change":"2024-03-31T15:03:36.981333+0000","last_active":"2024-03-31T15:03:36.988475+0000","last_peered":"2024-03-31T15:03:36.988475+0000","last_clean":"2024-03-31T15:03:36.988475+0000","last_became_active":"2024-03-31T15:03:36.980937+0000","last_became_peered":"2024-03-31T15:03:36.980937+0000","last_unstale":"2024-03-31T15:03:36.988475+0000","last_undegraded":"2024-03-31T15:03:36.988475+0000","last_fullsized":"2024-03-31T15:03:36.988475+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:05:28.499146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-31T15:03:36.994632+0000","last_change":"2024-03-31T15:03:36.989117+0000","last_active":"2024-03-31T15:03:36.994632+0000","last_peered":"2024-03-31T15:03:36.994632+0000","last_clean":"2024-03-31T15:03:36.994632+0000","last_became_active":"2024-03-31T15:03:36.986002+0000","last_became_peered":"2024-03-31T15:03:36.986002+0000","last_unstale":"2024-03-31T15:03:36.994632+0000","last_undegraded":"2024-03-31T15:03:36.994632+0000","last_fullsized":"2024-03-31T15:03:36.994632+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T19:58:22.015705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-31T15:03:36.988429+0000","last_change":"2024-03-31T15:03:36.981121+0000","last_active":"2024-03-31T15:03:36.988429+0000","last_peered":"2024-03-31T15:03:36.988429+0000","last_clean":"2024-03-31T15:03:36.988429+0000","last_became_active":"2024-03-31T15:03:36.980669+0000","last_became_peered":"2024-03-31T15:03:36.980669+0000","last_unstale":"2024-03-31T15:03:36.988429+0000","last_undegraded":"2024-03-31T15:03:36.988429+0000","last_fullsized":"2024-03-31T15:03:36.988429+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:07:25.174977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-31T15:03:36.976931+0000","last_change":"2024-03-31T15:03:36.974140+0000","last_active":"2024-03-31T15:03:36.976931+0000","last_peered":"2024-03-31T15:03:36.976931+0000","last_clean":"2024-03-31T15:03:36.976931+0000","last_became_active":"2024-03-31T15:03:36.973867+0000","last_became_peered":"2024-03-31T15:03:36.973867+0000","last_unstale":"2024-03-31T15:03:36.976931+0000","last_undegraded":"2024-03-31T15:03:36.976931+0000","last_fullsized":"2024-03-31T15:03:36.976931+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:02:20.135763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-31T15:03:36.974633+0000","last_change":"2024-03-31T15:03:36.969620+0000","last_active":"2024-03-31T15:03:36.974633+0000","last_peered":"2024-03-31T15:03:36.974633+0000","last_clean":"2024-03-31T15:03:36.974633+0000","last_became_active":"2024-03-31T15:03:36.969317+0000","last_became_peered":"2024-03-31T15:03:36.969317+0000","last_unstale":"2024-03-31T15:03:36.974633+0000","last_undegraded":"2024-03-31T15:03:36.974633+0000","last_fullsized":"2024-03-31T15:03:36.974633+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:29:47.722244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-31T15:03:36.994657+0000","last_change":"2024-03-31T15:03:36.990208+0000","last_active":"2024-03-31T15:03:36.994657+0000","last_peered":"2024-03-31T15:03:36.994657+0000","last_clean":"2024-03-31T15:03:36.994657+0000","last_became_active":"2024-03-31T15:03:36.989706+0000","last_became_peered":"2024-03-31T15:03:36.989706+0000","last_unstale":"2024-03-31T15:03:36.994657+0000","last_undegraded":"2024-03-31T15:03:36.994657+0000","last_fullsized":"2024-03-31T15:03:36.994657+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:26:15.339683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-31T15:03:36.986235+0000","last_change":"2024-03-31T15:03:36.982132+0000","last_active":"2024-03-31T15:03:36.986235+0000","last_peered":"2024-03-31T15:03:36.986235+0000","last_clean":"2024-03-31T15:03:36.986235+0000","last_became_active":"2024-03-31T15:03:36.981697+0000","last_became_peered":"2024-03-31T15:03:36.981697+0000","last_unstale":"2024-03-31T15:03:36.986235+0000","last_undegraded":"2024-03-31T15:03:36.986235+0000","last_fullsized":"2024-03-31T15:03:36.986235+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:37:26.685300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-31T15:03:36.986172+0000","last_change":"2024-03-31T15:03:36.980849+0000","last_active":"2024-03-31T15:03:36.986172+0000","last_peered":"2024-03-31T15:03:36.986172+0000","last_clean":"2024-03-31T15:03:36.986172+0000","last_became_active":"2024-03-31T15:03:36.980535+0000","last_became_peered":"2024-03-31T15:03:36.980535+0000","last_unstale":"2024-03-31T15:03:36.986172+0000","last_undegraded":"2024-03-31T15:03:36.986172+0000","last_fullsized":"2024-03-31T15:03:36.986172+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:29:57.089580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-31T15:03:37.966263+0000","last_change":"2024-03-31T15:03:36.962572+0000","last_active":"2024-03-31T15:03:37.966263+0000","last_peered":"2024-03-31T15:03:37.966263+0000","last_clean":"2024-03-31T15:03:37.966263+0000","last_became_active":"2024-03-31T15:03:36.960772+0000","last_became_peered":"2024-03-31T15:03:36.960772+0000","last_unstale":"2024-03-31T15:03:37.966263+0000","last_undegraded":"2024-03-31T15:03:37.966263+0000","last_fullsized":"2024-03-31T15:03:37.966263+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T15:58:19.498104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-31T15:03:36.965403+0000","last_change":"2024-03-31T15:03:36.960134+0000","last_active":"2024-03-31T15:03:36.965403+0000","last_peered":"2024-03-31T15:03:36.965403+0000","last_clean":"2024-03-31T15:03:36.965403+0000","last_became_active":"2024-03-31T15:03:36.959747+0000","last_became_peered":"2024-03-31T15:03:36.959747+0000","last_unstale":"2024-03-31T15:03:36.965403+0000","last_undegraded":"2024-03-31T15:03:36.965403+0000","last_fullsized":"2024-03-31T15:03:36.965403+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T16:00:46.569324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-31T15:03:36.971078+0000","last_change":"2024-03-31T15:03:36.964930+0000","last_active":"2024-03-31T15:03:36.971078+0000","last_peered":"2024-03-31T15:03:36.971078+0000","last_clean":"2024-03-31T15:03:36.971078+0000","last_became_active":"2024-03-31T15:03:36.964006+0000","last_became_peered":"2024-03-31T15:03:36.964006+0000","last_unstale":"2024-03-31T15:03:36.971078+0000","last_undegraded":"2024-03-31T15:03:36.971078+0000","last_fullsized":"2024-03-31T15:03:36.971078+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T15:38:43.103277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-31T15:03:36.983707+0000","last_change":"2024-03-31T15:03:36.981622+0000","last_active":"2024-03-31T15:03:36.983707+0000","last_peered":"2024-03-31T15:03:36.983707+0000","last_clean":"2024-03-31T15:03:36.983707+0000","last_became_active":"2024-03-31T15:03:36.981267+0000","last_became_peered":"2024-03-31T15:03:36.981267+0000","last_unstale":"2024-03-31T15:03:36.983707+0000","last_undegraded":"2024-03-31T15:03:36.983707+0000","last_fullsized":"2024-03-31T15:03:36.983707+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:29:57.089580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-31T15:03:36.984560+0000","last_change":"2024-03-31T15:03:36.977834+0000","last_active":"2024-03-31T15:03:36.984560+0000","last_peered":"2024-03-31T15:03:36.984560+0000","last_clean":"2024-03-31T15:03:36.984560+0000","last_became_active":"2024-03-31T15:03:36.977469+0000","last_became_peered":"2024-03-31T15:03:36.977469+0000","last_unstale":"2024-03-31T15:03:36.984560+0000","last_undegraded":"2024-03-31T15:03:36.984560+0000","last_fullsized":"2024-03-31T15:03:36.984560+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:19:24.339229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-31T15:03:36.988855+0000","last_change":"2024-03-31T15:03:36.981952+0000","last_active":"2024-03-31T15:03:36.988855+0000","last_peered":"2024-03-31T15:03:36.988855+0000","last_clean":"2024-03-31T15:03:36.988855+0000","last_became_active":"2024-03-31T15:03:36.981615+0000","last_became_peered":"2024-03-31T15:03:36.981615+0000","last_unstale":"2024-03-31T15:03:36.988855+0000","last_undegraded":"2024-03-31T15:03:36.988855+0000","last_fullsized":"2024-03-31T15:03:36.988855+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:11:06.260618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-31T15:03:36.984568+0000","last_change":"2024-03-31T15:03:36.981172+0000","last_active":"2024-03-31T15:03:36.984568+0000","last_peered":"2024-03-31T15:03:36.984568+0000","last_clean":"2024-03-31T15:03:36.984568+0000","last_became_active":"2024-03-31T15:03:36.980303+0000","last_became_peered":"2024-03-31T15:03:36.980303+0000","last_unstale":"2024-03-31T15:03:36.984568+0000","last_undegraded":"2024-03-31T15:03:36.984568+0000","last_fullsized":"2024-03-31T15:03:36.984568+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:26:54.177359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-31T15:03:36.994700+0000","last_change":"2024-03-31T15:03:36.991712+0000","last_active":"2024-03-31T15:03:36.994700+0000","last_peered":"2024-03-31T15:03:36.994700+0000","last_clean":"2024-03-31T15:03:36.994700+0000","last_became_active":"2024-03-31T15:03:36.988177+0000","last_became_peered":"2024-03-31T15:03:36.988177+0000","last_unstale":"2024-03-31T15:03:36.994700+0000","last_undegraded":"2024-03-31T15:03:36.994700+0000","last_fullsized":"2024-03-31T15:03:36.994700+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:05:30.818052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-31T15:03:36.983269+0000","last_change":"2024-03-31T15:03:36.968423+0000","last_active":"2024-03-31T15:03:36.983269+0000","last_peered":"2024-03-31T15:03:36.983269+0000","last_clean":"2024-03-31T15:03:36.983269+0000","last_became_active":"2024-03-31T15:03:36.968014+0000","last_became_peered":"2024-03-31T15:03:36.968014+0000","last_unstale":"2024-03-31T15:03:36.983269+0000","last_undegraded":"2024-03-31T15:03:36.983269+0000","last_fullsized":"2024-03-31T15:03:36.983269+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T16:00:46.569324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-31T15:03:36.986393+0000","last_change":"2024-03-31T15:03:36.975734+0000","last_active":"2024-03-31T15:03:36.986393+0000","last_peered":"2024-03-31T15:03:36.986393+0000","last_clean":"2024-03-31T15:03:36.986393+0000","last_became_active":"2024-03-31T15:03:36.970929+0000","last_became_peered":"2024-03-31T15:03:36.970929+0000","last_unstale":"2024-03-31T15:03:36.986393+0000","last_undegraded":"2024-03-31T15:03:36.986393+0000","last_fullsized":"2024-03-31T15:03:36.986393+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T15:07:58.072065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-31T15:03:36.989089+0000","last_change":"2024-03-31T15:03:36.982234+0000","last_active":"2024-03-31T15:03:36.989089+0000","last_peered":"2024-03-31T15:03:36.989089+0000","last_clean":"2024-03-31T15:03:36.989089+0000","last_became_active":"2024-03-31T15:03:36.981359+0000","last_became_peered":"2024-03-31T15:03:36.981359+0000","last_unstale":"2024-03-31T15:03:36.989089+0000","last_undegraded":"2024-03-31T15:03:36.989089+0000","last_fullsized":"2024-03-31T15:03:36.989089+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:16:22.583413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-31T15:03:36.974354+0000","last_change":"2024-03-31T15:03:36.965603+0000","last_active":"2024-03-31T15:03:36.974354+0000","last_peered":"2024-03-31T15:03:36.974354+0000","last_clean":"2024-03-31T15:03:36.974354+0000","last_became_active":"2024-03-31T15:03:36.965339+0000","last_became_peered":"2024-03-31T15:03:36.965339+0000","last_unstale":"2024-03-31T15:03:36.974354+0000","last_undegraded":"2024-03-31T15:03:36.974354+0000","last_fullsized":"2024-03-31T15:03:36.974354+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:11:51.069316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-31T15:03:36.984535+0000","last_change":"2024-03-31T15:03:36.977130+0000","last_active":"2024-03-31T15:03:36.984535+0000","last_peered":"2024-03-31T15:03:36.984535+0000","last_clean":"2024-03-31T15:03:36.984535+0000","last_became_active":"2024-03-31T15:03:36.973022+0000","last_became_peered":"2024-03-31T15:03:36.973022+0000","last_unstale":"2024-03-31T15:03:36.984535+0000","last_undegraded":"2024-03-31T15:03:36.984535+0000","last_fullsized":"2024-03-31T15:03:36.984535+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:48:54.968379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-31T15:03:36.975534+0000","last_change":"2024-03-31T15:03:36.967074+0000","last_active":"2024-03-31T15:03:36.975534+0000","last_peered":"2024-03-31T15:03:36.975534+0000","last_clean":"2024-03-31T15:03:36.975534+0000","last_became_active":"2024-03-31T15:03:36.966802+0000","last_became_peered":"2024-03-31T15:03:36.966802+0000","last_unstale":"2024-03-31T15:03:36.975534+0000","last_undegraded":"2024-03-31T15:03:36.975534+0000","last_fullsized":"2024-03-31T15:03:36.975534+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:27:39.263207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-31T15:03:36.986212+0000","last_change":"2024-03-31T15:03:36.977099+0000","last_active":"2024-03-31T15:03:36.986212+0000","last_peered":"2024-03-31T15:03:36.986212+0000","last_clean":"2024-03-31T15:03:36.986212+0000","last_became_active":"2024-03-31T15:03:36.976814+0000","last_became_peered":"2024-03-31T15:03:36.976814+0000","last_unstale":"2024-03-31T15:03:36.986212+0000","last_undegraded":"2024-03-31T15:03:36.986212+0000","last_fullsized":"2024-03-31T15:03:36.986212+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:27:39.263207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-31T15:03:36.973410+0000","last_change":"2024-03-31T15:03:36.968708+0000","last_active":"2024-03-31T15:03:36.973410+0000","last_peered":"2024-03-31T15:03:36.973410+0000","last_clean":"2024-03-31T15:03:36.973410+0000","last_became_active":"2024-03-31T15:03:36.968292+0000","last_became_peered":"2024-03-31T15:03:36.968292+0000","last_unstale":"2024-03-31T15:03:36.973410+0000","last_undegraded":"2024-03-31T15:03:36.973410+0000","last_fullsized":"2024-03-31T15:03:36.973410+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:11:51.069316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-31T15:03:36.977752+0000","last_change":"2024-03-31T15:03:36.970468+0000","last_active":"2024-03-31T15:03:36.977752+0000","last_peered":"2024-03-31T15:03:36.977752+0000","last_clean":"2024-03-31T15:03:36.977752+0000","last_became_active":"2024-03-31T15:03:36.970233+0000","last_became_peered":"2024-03-31T15:03:36.970233+0000","last_unstale":"2024-03-31T15:03:36.977752+0000","last_undegraded":"2024-03-31T15:03:36.977752+0000","last_fullsized":"2024-03-31T15:03:36.977752+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:29:57.089580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-31T15:03:36.975978+0000","last_change":"2024-03-31T15:03:36.962951+0000","last_active":"2024-03-31T15:03:36.975978+0000","last_peered":"2024-03-31T15:03:36.975978+0000","last_clean":"2024-03-31T15:03:36.975978+0000","last_became_active":"2024-03-31T15:03:36.961876+0000","last_became_peered":"2024-03-31T15:03:36.961876+0000","last_unstale":"2024-03-31T15:03:36.975978+0000","last_undegraded":"2024-03-31T15:03:36.975978+0000","last_fullsized":"2024-03-31T15:03:36.975978+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:18:04.322040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-31T15:03:36.988414+0000","last_change":"2024-03-31T15:03:36.979570+0000","last_active":"2024-03-31T15:03:36.988414+0000","last_peered":"2024-03-31T15:03:36.988414+0000","last_clean":"2024-03-31T15:03:36.988414+0000","last_became_active":"2024-03-31T15:03:36.979287+0000","last_became_peered":"2024-03-31T15:03:36.979287+0000","last_unstale":"2024-03-31T15:03:36.988414+0000","last_undegraded":"2024-03-31T15:03:36.988414+0000","last_fullsized":"2024-03-31T15:03:36.988414+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:55:58.515045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-31T15:03:36.989573+0000","last_change":"2024-03-31T15:03:36.988981+0000","last_active":"2024-03-31T15:03:36.989573+0000","last_peered":"2024-03-31T15:03:36.989573+0000","last_clean":"2024-03-31T15:03:36.989573+0000","last_became_active":"2024-03-31T15:03:36.988582+0000","last_became_peered":"2024-03-31T15:03:36.988582+0000","last_unstale":"2024-03-31T15:03:36.989573+0000","last_undegraded":"2024-03-31T15:03:36.989573+0000","last_fullsized":"2024-03-31T15:03:36.989573+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T15:58:19.498104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-31T15:03:36.984471+0000","last_change":"2024-03-31T15:03:36.975267+0000","last_active":"2024-03-31T15:03:36.984471+0000","last_peered":"2024-03-31T15:03:36.984471+0000","last_clean":"2024-03-31T15:03:36.984471+0000","last_became_active":"2024-03-31T15:03:36.972549+0000","last_became_peered":"2024-03-31T15:03:36.972549+0000","last_unstale":"2024-03-31T15:03:36.984471+0000","last_undegraded":"2024-03-31T15:03:36.984471+0000","last_fullsized":"2024-03-31T15:03:36.984471+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:11:51.069316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-31T15:03:37.966007+0000","last_change":"2024-03-31T15:03:36.964214+0000","last_active":"2024-03-31T15:03:37.966007+0000","last_peered":"2024-03-31T15:03:37.966007+0000","last_clean":"2024-03-31T15:03:37.966007+0000","last_became_active":"2024-03-31T15:03:36.963926+0000","last_became_peered":"2024-03-31T15:03:36.963926+0000","last_unstale":"2024-03-31T15:03:37.966007+0000","last_undegraded":"2024-03-31T15:03:37.966007+0000","last_fullsized":"2024-03-31T15:03:37.966007+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:26:54.177359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"20'1","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-31T15:03:36.990553+0000","last_change":"2024-03-31T15:03:35.966649+0000","last_active":"2024-03-31T15:03:36.990553+0000","last_peered":"2024-03-31T15:03:36.990553+0000","last_clean":"2024-03-31T15:03:36.990553+0000","last_became_active":"2024-03-31T15:03:35.964612+0000","last_became_peered":"2024-03-31T15:03:35.964612+0000","last_unstale":"2024-03-31T15:03:36.990553+0000","last_undegraded":"2024-03-31T15:03:36.990553+0000","last_fullsized":"2024-03-31T15:03:36.990553+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_clean_scrub_stamp":"2024-03-31T15:03:34.926145+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T15:38:42.097818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-31T15:03:36.962983+0000","last_change":"2024-03-31T15:03:36.961519+0000","last_active":"2024-03-31T15:03:36.962983+0000","last_peered":"2024-03-31T15:03:36.962983+0000","last_clean":"2024-03-31T15:03:36.962983+0000","last_became_active":"2024-03-31T15:03:36.961032+0000","last_became_peered":"2024-03-31T15:03:36.961032+0000","last_unstale":"2024-03-31T15:03:36.962983+0000","last_undegraded":"2024-03-31T15:03:36.962983+0000","last_fullsized":"2024-03-31T15:03:36.962983+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T15:15:06.651032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"20'1","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-31T15:03:36.992073+0000","last_change":"2024-03-31T15:03:35.961058+0000","last_active":"2024-03-31T15:03:36.992073+0000","last_peered":"2024-03-31T15:03:36.992073+0000","last_clean":"2024-03-31T15:03:36.992073+0000","last_became_active":"2024-03-31T15:03:35.960684+0000","last_became_peered":"2024-03-31T15:03:35.960684+0000","last_unstale":"2024-03-31T15:03:36.992073+0000","last_undegraded":"2024-03-31T15:03:36.992073+0000","last_fullsized":"2024-03-31T15:03:36.992073+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_clean_scrub_stamp":"2024-03-31T15:03:34.926145+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:07:06.210288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-31T15:03:36.954825+0000","last_change":"2024-03-31T15:03:35.963374+0000","last_active":"2024-03-31T15:03:36.954825+0000","last_peered":"2024-03-31T15:03:36.954825+0000","last_clean":"2024-03-31T15:03:36.954825+0000","last_became_active":"2024-03-31T15:03:35.963009+0000","last_became_peered":"2024-03-31T15:03:35.963009+0000","last_unstale":"2024-03-31T15:03:36.954825+0000","last_undegraded":"2024-03-31T15:03:36.954825+0000","last_fullsized":"2024-03-31T15:03:36.954825+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_clean_scrub_stamp":"2024-03-31T15:03:34.926145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:36:14.576891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-31T15:03:36.978932+0000","last_change":"2024-03-31T15:03:36.972478+0000","last_active":"2024-03-31T15:03:36.978932+0000","last_peered":"2024-03-31T15:03:36.978932+0000","last_clean":"2024-03-31T15:03:36.978932+0000","last_became_active":"2024-03-31T15:03:36.972297+0000","last_became_peered":"2024-03-31T15:03:36.972297+0000","last_unstale":"2024-03-31T15:03:36.978932+0000","last_undegraded":"2024-03-31T15:03:36.978932+0000","last_fullsized":"2024-03-31T15:03:36.978932+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:48:13.691269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-31T15:03:36.961621+0000","last_change":"2024-03-31T15:03:35.963562+0000","last_active":"2024-03-31T15:03:36.961621+0000","last_peered":"2024-03-31T15:03:36.961621+0000","last_clean":"2024-03-31T15:03:36.961621+0000","last_became_active":"2024-03-31T15:03:35.961838+0000","last_became_peered":"2024-03-31T15:03:35.961838+0000","last_unstale":"2024-03-31T15:03:36.961621+0000","last_undegraded":"2024-03-31T15:03:36.961621+0000","last_fullsized":"2024-03-31T15:03:36.961621+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_clean_scrub_stamp":"2024-03-31T15:03:34.926145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:27:19.823151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-31T15:03:36.974024+0000","last_change":"2024-03-31T15:03:36.970365+0000","last_active":"2024-03-31T15:03:36.974024+0000","last_peered":"2024-03-31T15:03:36.974024+0000","last_clean":"2024-03-31T15:03:36.974024+0000","last_became_active":"2024-03-31T15:03:36.970060+0000","last_became_peered":"2024-03-31T15:03:36.970060+0000","last_unstale":"2024-03-31T15:03:36.974024+0000","last_undegraded":"2024-03-31T15:03:36.974024+0000","last_fullsized":"2024-03-31T15:03:36.974024+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:27:39.263207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-31T15:03:36.955163+0000","last_change":"2024-03-31T15:03:35.956977+0000","last_active":"2024-03-31T15:03:36.955163+0000","last_peered":"2024-03-31T15:03:36.955163+0000","last_clean":"2024-03-31T15:03:36.955163+0000","last_became_active":"2024-03-31T15:03:35.954617+0000","last_became_peered":"2024-03-31T15:03:35.954617+0000","last_unstale":"2024-03-31T15:03:36.955163+0000","last_undegraded":"2024-03-31T15:03:36.955163+0000","last_fullsized":"2024-03-31T15:03:36.955163+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_clean_scrub_stamp":"2024-03-31T15:03:34.926145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T17:31:20.984335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-31T15:03:36.988694+0000","last_change":"2024-03-31T15:03:36.981437+0000","last_active":"2024-03-31T15:03:36.988694+0000","last_peered":"2024-03-31T15:03:36.988694+0000","last_clean":"2024-03-31T15:03:36.988694+0000","last_became_active":"2024-03-31T15:03:36.981050+0000","last_became_peered":"2024-03-31T15:03:36.981050+0000","last_unstale":"2024-03-31T15:03:36.988694+0000","last_undegraded":"2024-03-31T15:03:36.988694+0000","last_fullsized":"2024-03-31T15:03:36.988694+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T19:58:22.015705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-31T15:03:36.965706+0000","last_change":"2024-03-31T15:03:35.954630+0000","last_active":"2024-03-31T15:03:36.965706+0000","last_peered":"2024-03-31T15:03:36.965706+0000","last_clean":"2024-03-31T15:03:36.965706+0000","last_became_active":"2024-03-31T15:03:35.954149+0000","last_became_peered":"2024-03-31T15:03:35.954149+0000","last_unstale":"2024-03-31T15:03:36.965706+0000","last_undegraded":"2024-03-31T15:03:36.965706+0000","last_fullsized":"2024-03-31T15:03:36.965706+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_clean_scrub_stamp":"2024-03-31T15:03:34.926145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:23:12.843244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-31T15:03:36.989433+0000","last_change":"2024-03-31T15:03:36.988008+0000","last_active":"2024-03-31T15:03:36.989433+0000","last_peered":"2024-03-31T15:03:36.989433+0000","last_clean":"2024-03-31T15:03:36.989433+0000","last_became_active":"2024-03-31T15:03:36.987732+0000","last_became_peered":"2024-03-31T15:03:36.987732+0000","last_unstale":"2024-03-31T15:03:36.989433+0000","last_undegraded":"2024-03-31T15:03:36.989433+0000","last_fullsized":"2024-03-31T15:03:36.989433+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:51:11.582045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"20'3","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-31T15:03:36.991796+0000","last_change":"2024-03-31T15:03:35.959827+0000","last_active":"2024-03-31T15:03:36.991796+0000","last_peered":"2024-03-31T15:03:36.991796+0000","last_clean":"2024-03-31T15:03:36.991796+0000","last_became_active":"2024-03-31T15:03:35.959564+0000","last_became_peered":"2024-03-31T15:03:35.959564+0000","last_unstale":"2024-03-31T15:03:36.991796+0000","last_undegraded":"2024-03-31T15:03:36.991796+0000","last_fullsized":"2024-03-31T15:03:36.991796+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_clean_scrub_stamp":"2024-03-31T15:03:34.926145+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:36:14.576891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-31T15:03:37.966441+0000","last_change":"2024-03-31T15:03:36.982726+0000","last_active":"2024-03-31T15:03:37.966441+0000","last_peered":"2024-03-31T15:03:37.966441+0000","last_clean":"2024-03-31T15:03:37.966441+0000","last_became_active":"2024-03-31T15:03:36.981893+0000","last_became_peered":"2024-03-31T15:03:36.981893+0000","last_unstale":"2024-03-31T15:03:37.966441+0000","last_undegraded":"2024-03-31T15:03:37.966441+0000","last_fullsized":"2024-03-31T15:03:37.966441+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:19:24.339229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"20'6","reported_seq":18,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-31T15:03:36.994023+0000","last_change":"2024-03-31T15:03:35.970154+0000","last_active":"2024-03-31T15:03:36.994023+0000","last_peered":"2024-03-31T15:03:36.994023+0000","last_clean":"2024-03-31T15:03:36.994023+0000","last_became_active":"2024-03-31T15:03:35.969877+0000","last_became_peered":"2024-03-31T15:03:35.969877+0000","last_unstale":"2024-03-31T15:03:36.994023+0000","last_undegraded":"2024-03-31T15:03:36.994023+0000","last_fullsized":"2024-03-31T15:03:36.994023+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_clean_scrub_stamp":"2024-03-31T15:03:34.926145+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:36:14.576891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-31T15:03:36.986539+0000","last_change":"2024-03-31T15:03:36.980541+0000","last_active":"2024-03-31T15:03:36.986539+0000","last_peered":"2024-03-31T15:03:36.986539+0000","last_clean":"2024-03-31T15:03:36.986539+0000","last_became_active":"2024-03-31T15:03:36.980276+0000","last_became_peered":"2024-03-31T15:03:36.980276+0000","last_unstale":"2024-03-31T15:03:36.986539+0000","last_undegraded":"2024-03-31T15:03:36.986539+0000","last_fullsized":"2024-03-31T15:03:36.986539+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:53:00.086224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"20'2","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-31T15:03:36.992565+0000","last_change":"2024-03-31T15:03:35.960798+0000","last_active":"2024-03-31T15:03:36.992565+0000","last_peered":"2024-03-31T15:03:36.992565+0000","last_clean":"2024-03-31T15:03:36.992565+0000","last_became_active":"2024-03-31T15:03:35.960522+0000","last_became_peered":"2024-03-31T15:03:35.960522+0000","last_unstale":"2024-03-31T15:03:36.992565+0000","last_undegraded":"2024-03-31T15:03:36.992565+0000","last_fullsized":"2024-03-31T15:03:36.992565+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_clean_scrub_stamp":"2024-03-31T15:03:34.926145+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T16:07:33.984402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-31T15:03:36.975599+0000","last_change":"2024-03-31T15:03:36.972506+0000","last_active":"2024-03-31T15:03:36.975599+0000","last_peered":"2024-03-31T15:03:36.975599+0000","last_clean":"2024-03-31T15:03:36.975599+0000","last_became_active":"2024-03-31T15:03:36.972194+0000","last_became_peered":"2024-03-31T15:03:36.972194+0000","last_unstale":"2024-03-31T15:03:36.975599+0000","last_undegraded":"2024-03-31T15:03:36.975599+0000","last_fullsized":"2024-03-31T15:03:36.975599+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T17:31:21.989794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-31T15:03:36.967356+0000","last_change":"2024-03-31T15:03:35.966260+0000","last_active":"2024-03-31T15:03:36.967356+0000","last_peered":"2024-03-31T15:03:36.967356+0000","last_clean":"2024-03-31T15:03:36.967356+0000","last_became_active":"2024-03-31T15:03:35.965676+0000","last_became_peered":"2024-03-31T15:03:35.965676+0000","last_unstale":"2024-03-31T15:03:36.967356+0000","last_undegraded":"2024-03-31T15:03:36.967356+0000","last_fullsized":"2024-03-31T15:03:36.967356+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_clean_scrub_stamp":"2024-03-31T15:03:34.926145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:26:53.171901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-31T15:03:36.994700+0000","last_change":"2024-03-31T15:03:36.988567+0000","last_active":"2024-03-31T15:03:36.994700+0000","last_peered":"2024-03-31T15:03:36.994700+0000","last_clean":"2024-03-31T15:03:36.994700+0000","last_became_active":"2024-03-31T15:03:36.985668+0000","last_became_peered":"2024-03-31T15:03:36.985668+0000","last_unstale":"2024-03-31T15:03:36.994700+0000","last_undegraded":"2024-03-31T15:03:36.994700+0000","last_fullsized":"2024-03-31T15:03:36.994700+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:16:22.583413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-31T15:03:36.954637+0000","last_change":"2024-03-31T15:03:35.961562+0000","last_active":"2024-03-31T15:03:36.954637+0000","last_peered":"2024-03-31T15:03:36.954637+0000","last_clean":"2024-03-31T15:03:36.954637+0000","last_became_active":"2024-03-31T15:03:35.961280+0000","last_became_peered":"2024-03-31T15:03:35.961280+0000","last_unstale":"2024-03-31T15:03:36.954637+0000","last_undegraded":"2024-03-31T15:03:36.954637+0000","last_fullsized":"2024-03-31T15:03:36.954637+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_clean_scrub_stamp":"2024-03-31T15:03:34.926145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T15:38:42.097818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-31T15:03:36.986126+0000","last_change":"2024-03-31T15:03:36.979417+0000","last_active":"2024-03-31T15:03:36.986126+0000","last_peered":"2024-03-31T15:03:36.986126+0000","last_clean":"2024-03-31T15:03:36.986126+0000","last_became_active":"2024-03-31T15:03:36.973021+0000","last_became_peered":"2024-03-31T15:03:36.973021+0000","last_unstale":"2024-03-31T15:03:36.986126+0000","last_undegraded":"2024-03-31T15:03:36.986126+0000","last_fullsized":"2024-03-31T15:03:36.986126+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:29:47.722244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"20'8","reported_seq":20,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-31T15:03:36.988230+0000","last_change":"2024-03-31T15:03:35.955608+0000","last_active":"2024-03-31T15:03:36.988230+0000","last_peered":"2024-03-31T15:03:36.988230+0000","last_clean":"2024-03-31T15:03:36.988230+0000","last_became_active":"2024-03-31T15:03:35.955295+0000","last_became_peered":"2024-03-31T15:03:35.955295+0000","last_unstale":"2024-03-31T15:03:36.988230+0000","last_undegraded":"2024-03-31T15:03:36.988230+0000","last_fullsized":"2024-03-31T15:03:36.988230+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_clean_scrub_stamp":"2024-03-31T15:03:34.926145+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:23:12.843244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-31T15:03:36.994685+0000","last_change":"2024-03-31T15:03:36.990806+0000","last_active":"2024-03-31T15:03:36.994685+0000","last_peered":"2024-03-31T15:03:36.994685+0000","last_clean":"2024-03-31T15:03:36.994685+0000","last_became_active":"2024-03-31T15:03:36.990542+0000","last_became_peered":"2024-03-31T15:03:36.990542+0000","last_unstale":"2024-03-31T15:03:36.994685+0000","last_undegraded":"2024-03-31T15:03:36.994685+0000","last_fullsized":"2024-03-31T15:03:36.994685+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:11:06.260618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-31T15:03:37.966654+0000","last_change":"2024-03-31T15:03:35.968824+0000","last_active":"2024-03-31T15:03:37.966654+0000","last_peered":"2024-03-31T15:03:37.966654+0000","last_clean":"2024-03-31T15:03:37.966654+0000","last_became_active":"2024-03-31T15:03:35.968431+0000","last_became_peered":"2024-03-31T15:03:35.968431+0000","last_unstale":"2024-03-31T15:03:37.966654+0000","last_undegraded":"2024-03-31T15:03:37.966654+0000","last_fullsized":"2024-03-31T15:03:37.966654+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_clean_scrub_stamp":"2024-03-31T15:03:34.926145+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:30:58.752050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-31T15:03:36.998693+0000","last_change":"2024-03-31T15:03:36.993283+0000","last_active":"2024-03-31T15:03:36.998693+0000","last_peered":"2024-03-31T15:03:36.998693+0000","last_clean":"2024-03-31T15:03:36.998693+0000","last_became_active":"2024-03-31T15:03:36.992977+0000","last_became_peered":"2024-03-31T15:03:36.992977+0000","last_unstale":"2024-03-31T15:03:36.998693+0000","last_undegraded":"2024-03-31T15:03:36.998693+0000","last_fullsized":"2024-03-31T15:03:36.998693+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:34:49.982234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"20'2","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-31T15:03:36.983190+0000","last_change":"2024-03-31T15:03:35.969251+0000","last_active":"2024-03-31T15:03:36.983190+0000","last_peered":"2024-03-31T15:03:36.983190+0000","last_clean":"2024-03-31T15:03:36.983190+0000","last_became_active":"2024-03-31T15:03:35.968876+0000","last_became_peered":"2024-03-31T15:03:35.968876+0000","last_unstale":"2024-03-31T15:03:36.983190+0000","last_undegraded":"2024-03-31T15:03:36.983190+0000","last_fullsized":"2024-03-31T15:03:36.983190+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_clean_scrub_stamp":"2024-03-31T15:03:34.926145+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:27:19.823151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-31T15:03:36.993343+0000","last_change":"2024-03-31T15:03:36.986107+0000","last_active":"2024-03-31T15:03:36.993343+0000","last_peered":"2024-03-31T15:03:36.993343+0000","last_clean":"2024-03-31T15:03:36.993343+0000","last_became_active":"2024-03-31T15:03:36.985749+0000","last_became_peered":"2024-03-31T15:03:36.985749+0000","last_unstale":"2024-03-31T15:03:36.993343+0000","last_undegraded":"2024-03-31T15:03:36.993343+0000","last_fullsized":"2024-03-31T15:03:36.993343+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:05:28.499146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"20'1","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-31T15:03:36.993222+0000","last_change":"2024-03-31T15:03:35.965507+0000","last_active":"2024-03-31T15:03:36.993222+0000","last_peered":"2024-03-31T15:03:36.993222+0000","last_clean":"2024-03-31T15:03:36.993222+0000","last_became_active":"2024-03-31T15:03:35.964843+0000","last_became_peered":"2024-03-31T15:03:35.964843+0000","last_unstale":"2024-03-31T15:03:36.993222+0000","last_undegraded":"2024-03-31T15:03:36.993222+0000","last_fullsized":"2024-03-31T15:03:36.993222+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_clean_scrub_stamp":"2024-03-31T15:03:34.926145+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T17:31:20.984335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-31T15:03:36.988456+0000","last_change":"2024-03-31T15:03:36.979781+0000","last_active":"2024-03-31T15:03:36.988456+0000","last_peered":"2024-03-31T15:03:36.988456+0000","last_clean":"2024-03-31T15:03:36.988456+0000","last_became_active":"2024-03-31T15:03:36.979517+0000","last_became_peered":"2024-03-31T15:03:36.979517+0000","last_unstale":"2024-03-31T15:03:36.988456+0000","last_undegraded":"2024-03-31T15:03:36.988456+0000","last_fullsized":"2024-03-31T15:03:36.988456+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:02:20.135763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-31T15:03:36.966397+0000","last_change":"2024-03-31T15:03:35.960305+0000","last_active":"2024-03-31T15:03:36.966397+0000","last_peered":"2024-03-31T15:03:36.966397+0000","last_clean":"2024-03-31T15:03:36.966397+0000","last_became_active":"2024-03-31T15:03:35.960006+0000","last_became_peered":"2024-03-31T15:03:35.960006+0000","last_unstale":"2024-03-31T15:03:36.966397+0000","last_undegraded":"2024-03-31T15:03:36.966397+0000","last_fullsized":"2024-03-31T15:03:36.966397+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_clean_scrub_stamp":"2024-03-31T15:03:34.926145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:11:41.679652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-31T15:03:36.991744+0000","last_change":"2024-03-31T15:03:36.985221+0000","last_active":"2024-03-31T15:03:36.991744+0000","last_peered":"2024-03-31T15:03:36.991744+0000","last_clean":"2024-03-31T15:03:36.991744+0000","last_became_active":"2024-03-31T15:03:36.984963+0000","last_became_peered":"2024-03-31T15:03:36.984963+0000","last_unstale":"2024-03-31T15:03:36.991744+0000","last_undegraded":"2024-03-31T15:03:36.991744+0000","last_fullsized":"2024-03-31T15:03:36.991744+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:07:25.174977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":168,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-31T15:03:36.962575+0000","last_change":"2024-03-31T15:03:24.329890+0000","last_active":"2024-03-31T15:03:36.962575+0000","last_peered":"2024-03-31T15:03:36.962575+0000","last_clean":"2024-03-31T15:03:36.962575+0000","last_became_active":"2024-03-31T15:03:24.328882+0000","last_became_peered":"2024-03-31T15:03:24.328882+0000","last_unstale":"2024-03-31T15:03:36.962575+0000","last_undegraded":"2024-03-31T15:03:36.962575+0000","last_fullsized":"2024-03-31T15:03:36.962575+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:22.885535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:22.885535+0000","last_clean_scrub_stamp":"2024-03-31T15:03:22.885535+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:18:29.004296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-31T15:03:36.967122+0000","last_change":"2024-03-31T15:03:35.952977+0000","last_active":"2024-03-31T15:03:36.967122+0000","last_peered":"2024-03-31T15:03:36.967122+0000","last_clean":"2024-03-31T15:03:36.967122+0000","last_became_active":"2024-03-31T15:03:35.951288+0000","last_became_peered":"2024-03-31T15:03:35.951288+0000","last_unstale":"2024-03-31T15:03:36.967122+0000","last_undegraded":"2024-03-31T15:03:36.967122+0000","last_fullsized":"2024-03-31T15:03:36.967122+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_clean_scrub_stamp":"2024-03-31T15:03:34.926145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:11:41.679652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-31T15:03:36.985608+0000","last_change":"2024-03-31T15:03:36.980592+0000","last_active":"2024-03-31T15:03:36.985608+0000","last_peered":"2024-03-31T15:03:36.985608+0000","last_clean":"2024-03-31T15:03:36.985608+0000","last_became_active":"2024-03-31T15:03:36.980285+0000","last_became_peered":"2024-03-31T15:03:36.980285+0000","last_unstale":"2024-03-31T15:03:36.985608+0000","last_undegraded":"2024-03-31T15:03:36.985608+0000","last_fullsized":"2024-03-31T15:03:36.985608+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T17:15:21.722119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-31T15:03:36.962515+0000","last_change":"2024-03-31T15:03:35.968364+0000","last_active":"2024-03-31T15:03:36.962515+0000","last_peered":"2024-03-31T15:03:36.962515+0000","last_clean":"2024-03-31T15:03:36.962515+0000","last_became_active":"2024-03-31T15:03:35.968047+0000","last_became_peered":"2024-03-31T15:03:35.968047+0000","last_unstale":"2024-03-31T15:03:36.962515+0000","last_undegraded":"2024-03-31T15:03:36.962515+0000","last_fullsized":"2024-03-31T15:03:36.962515+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_clean_scrub_stamp":"2024-03-31T15:03:34.926145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:26:53.171901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-31T15:03:36.971139+0000","last_change":"2024-03-31T15:03:36.963584+0000","last_active":"2024-03-31T15:03:36.971139+0000","last_peered":"2024-03-31T15:03:36.971139+0000","last_clean":"2024-03-31T15:03:36.971139+0000","last_became_active":"2024-03-31T15:03:36.963216+0000","last_became_peered":"2024-03-31T15:03:36.963216+0000","last_unstale":"2024-03-31T15:03:36.971139+0000","last_undegraded":"2024-03-31T15:03:36.971139+0000","last_fullsized":"2024-03-31T15:03:36.971139+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:07:07.215747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-31T15:03:36.964813+0000","last_change":"2024-03-31T15:03:35.963196+0000","last_active":"2024-03-31T15:03:36.964813+0000","last_peered":"2024-03-31T15:03:36.964813+0000","last_clean":"2024-03-31T15:03:36.964813+0000","last_became_active":"2024-03-31T15:03:35.962746+0000","last_became_peered":"2024-03-31T15:03:35.962746+0000","last_unstale":"2024-03-31T15:03:36.964813+0000","last_undegraded":"2024-03-31T15:03:36.964813+0000","last_fullsized":"2024-03-31T15:03:36.964813+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_clean_scrub_stamp":"2024-03-31T15:03:34.926145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:00:32.840818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-31T15:03:36.973993+0000","last_change":"2024-03-31T15:03:36.972190+0000","last_active":"2024-03-31T15:03:36.973993+0000","last_peered":"2024-03-31T15:03:36.973993+0000","last_clean":"2024-03-31T15:03:36.973993+0000","last_became_active":"2024-03-31T15:03:36.971866+0000","last_became_peered":"2024-03-31T15:03:36.971866+0000","last_unstale":"2024-03-31T15:03:36.973993+0000","last_undegraded":"2024-03-31T15:03:36.973993+0000","last_fullsized":"2024-03-31T15:03:36.973993+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:29:47.722244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-31T15:03:36.967713+0000","last_change":"2024-03-31T15:03:35.953803+0000","last_active":"2024-03-31T15:03:36.967713+0000","last_peered":"2024-03-31T15:03:36.967713+0000","last_clean":"2024-03-31T15:03:36.967713+0000","last_became_active":"2024-03-31T15:03:35.953459+0000","last_became_peered":"2024-03-31T15:03:35.953459+0000","last_unstale":"2024-03-31T15:03:36.967713+0000","last_undegraded":"2024-03-31T15:03:36.967713+0000","last_fullsized":"2024-03-31T15:03:36.967713+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_clean_scrub_stamp":"2024-03-31T15:03:34.926145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T15:58:18.492645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-31T15:03:37.966372+0000","last_change":"2024-03-31T15:03:36.966599+0000","last_active":"2024-03-31T15:03:37.966372+0000","last_peered":"2024-03-31T15:03:37.966372+0000","last_clean":"2024-03-31T15:03:37.966372+0000","last_became_active":"2024-03-31T15:03:36.964867+0000","last_became_peered":"2024-03-31T15:03:36.964867+0000","last_unstale":"2024-03-31T15:03:37.966372+0000","last_undegraded":"2024-03-31T15:03:37.966372+0000","last_fullsized":"2024-03-31T15:03:37.966372+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:48:54.968379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-31T15:03:36.956046+0000","last_change":"2024-03-31T15:03:35.958174+0000","last_active":"2024-03-31T15:03:36.956046+0000","last_peered":"2024-03-31T15:03:36.956046+0000","last_clean":"2024-03-31T15:03:36.956046+0000","last_became_active":"2024-03-31T15:03:35.957870+0000","last_became_peered":"2024-03-31T15:03:35.957870+0000","last_unstale":"2024-03-31T15:03:36.956046+0000","last_undegraded":"2024-03-31T15:03:36.956046+0000","last_fullsized":"2024-03-31T15:03:36.956046+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_clean_scrub_stamp":"2024-03-31T15:03:34.926145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T15:38:42.097818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-31T15:03:36.989207+0000","last_change":"2024-03-31T15:03:36.980337+0000","last_active":"2024-03-31T15:03:36.989207+0000","last_peered":"2024-03-31T15:03:36.989207+0000","last_clean":"2024-03-31T15:03:36.989207+0000","last_became_active":"2024-03-31T15:03:36.980082+0000","last_became_peered":"2024-03-31T15:03:36.980082+0000","last_unstale":"2024-03-31T15:03:36.989207+0000","last_undegraded":"2024-03-31T15:03:36.989207+0000","last_fullsized":"2024-03-31T15:03:36.989207+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:53:00.086224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-31T15:03:36.964126+0000","last_change":"2024-03-31T15:03:35.969598+0000","last_active":"2024-03-31T15:03:36.964126+0000","last_peered":"2024-03-31T15:03:36.964126+0000","last_clean":"2024-03-31T15:03:36.964126+0000","last_became_active":"2024-03-31T15:03:35.969253+0000","last_became_peered":"2024-03-31T15:03:35.969253+0000","last_unstale":"2024-03-31T15:03:36.964126+0000","last_undegraded":"2024-03-31T15:03:36.964126+0000","last_fullsized":"2024-03-31T15:03:36.964126+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_clean_scrub_stamp":"2024-03-31T15:03:34.926145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:55:57.509587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-31T15:03:36.987586+0000","last_change":"2024-03-31T15:03:36.983933+0000","last_active":"2024-03-31T15:03:36.987586+0000","last_peered":"2024-03-31T15:03:36.987586+0000","last_clean":"2024-03-31T15:03:36.987586+0000","last_became_active":"2024-03-31T15:03:36.983650+0000","last_became_peered":"2024-03-31T15:03:36.983650+0000","last_unstale":"2024-03-31T15:03:36.987586+0000","last_undegraded":"2024-03-31T15:03:36.987586+0000","last_fullsized":"2024-03-31T15:03:36.987586+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:37:26.685300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-31T15:03:36.958497+0000","last_change":"2024-03-31T15:03:35.959254+0000","last_active":"2024-03-31T15:03:36.958497+0000","last_peered":"2024-03-31T15:03:36.958497+0000","last_clean":"2024-03-31T15:03:36.958497+0000","last_became_active":"2024-03-31T15:03:35.958903+0000","last_became_peered":"2024-03-31T15:03:35.958903+0000","last_unstale":"2024-03-31T15:03:36.958497+0000","last_undegraded":"2024-03-31T15:03:36.958497+0000","last_fullsized":"2024-03-31T15:03:36.958497+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_clean_scrub_stamp":"2024-03-31T15:03:34.926145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T15:07:57.066606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-31T15:03:36.978714+0000","last_change":"2024-03-31T15:03:36.973216+0000","last_active":"2024-03-31T15:03:36.978714+0000","last_peered":"2024-03-31T15:03:36.978714+0000","last_clean":"2024-03-31T15:03:36.978714+0000","last_became_active":"2024-03-31T15:03:36.972719+0000","last_became_peered":"2024-03-31T15:03:36.972719+0000","last_unstale":"2024-03-31T15:03:36.978714+0000","last_undegraded":"2024-03-31T15:03:36.978714+0000","last_fullsized":"2024-03-31T15:03:36.978714+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T19:12:45.358470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"20'2","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-31T15:03:36.991987+0000","last_change":"2024-03-31T15:03:35.964289+0000","last_active":"2024-03-31T15:03:36.991987+0000","last_peered":"2024-03-31T15:03:36.991987+0000","last_clean":"2024-03-31T15:03:36.991987+0000","last_became_active":"2024-03-31T15:03:35.963897+0000","last_became_peered":"2024-03-31T15:03:35.963897+0000","last_unstale":"2024-03-31T15:03:36.991987+0000","last_undegraded":"2024-03-31T15:03:36.991987+0000","last_fullsized":"2024-03-31T15:03:36.991987+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_clean_scrub_stamp":"2024-03-31T15:03:34.926145+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T15:07:57.066606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-31T15:03:36.971022+0000","last_change":"2024-03-31T15:03:36.967603+0000","last_active":"2024-03-31T15:03:36.971022+0000","last_peered":"2024-03-31T15:03:36.971022+0000","last_clean":"2024-03-31T15:03:36.971022+0000","last_became_active":"2024-03-31T15:03:36.967206+0000","last_became_peered":"2024-03-31T15:03:36.967206+0000","last_unstale":"2024-03-31T15:03:36.971022+0000","last_undegraded":"2024-03-31T15:03:36.971022+0000","last_fullsized":"2024-03-31T15:03:36.971022+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:36:15.582349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-31T15:03:36.962355+0000","last_change":"2024-03-31T15:03:35.967340+0000","last_active":"2024-03-31T15:03:36.962355+0000","last_peered":"2024-03-31T15:03:36.962355+0000","last_clean":"2024-03-31T15:03:36.962355+0000","last_became_active":"2024-03-31T15:03:35.967046+0000","last_became_peered":"2024-03-31T15:03:35.967046+0000","last_unstale":"2024-03-31T15:03:36.962355+0000","last_undegraded":"2024-03-31T15:03:36.962355+0000","last_fullsized":"2024-03-31T15:03:36.962355+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_clean_scrub_stamp":"2024-03-31T15:03:34.926145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:11:41.679652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-31T15:03:37.965991+0000","last_change":"2024-03-31T15:03:36.991196+0000","last_active":"2024-03-31T15:03:37.965991+0000","last_peered":"2024-03-31T15:03:37.965991+0000","last_clean":"2024-03-31T15:03:37.965991+0000","last_became_active":"2024-03-31T15:03:36.990914+0000","last_became_peered":"2024-03-31T15:03:36.990914+0000","last_unstale":"2024-03-31T15:03:37.965991+0000","last_undegraded":"2024-03-31T15:03:37.965991+0000","last_fullsized":"2024-03-31T15:03:37.965991+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:11:42.685111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-31T15:03:36.955854+0000","last_change":"2024-03-31T15:03:35.969671+0000","last_active":"2024-03-31T15:03:36.955854+0000","last_peered":"2024-03-31T15:03:36.955854+0000","last_clean":"2024-03-31T15:03:36.955854+0000","last_became_active":"2024-03-31T15:03:35.969412+0000","last_became_peered":"2024-03-31T15:03:35.969412+0000","last_unstale":"2024-03-31T15:03:36.955854+0000","last_undegraded":"2024-03-31T15:03:36.955854+0000","last_fullsized":"2024-03-31T15:03:36.955854+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_clean_scrub_stamp":"2024-03-31T15:03:34.926145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:07:06.210288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-31T15:03:36.965438+0000","last_change":"2024-03-31T15:03:36.959849+0000","last_active":"2024-03-31T15:03:36.965438+0000","last_peered":"2024-03-31T15:03:36.965438+0000","last_clean":"2024-03-31T15:03:36.965438+0000","last_became_active":"2024-03-31T15:03:36.959611+0000","last_became_peered":"2024-03-31T15:03:36.959611+0000","last_unstale":"2024-03-31T15:03:36.965438+0000","last_undegraded":"2024-03-31T15:03:36.965438+0000","last_fullsized":"2024-03-31T15:03:36.965438+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T15:07:58.072065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-31T15:03:36.958718+0000","last_change":"2024-03-31T15:03:35.963705+0000","last_active":"2024-03-31T15:03:36.958718+0000","last_peered":"2024-03-31T15:03:36.958718+0000","last_clean":"2024-03-31T15:03:36.958718+0000","last_became_active":"2024-03-31T15:03:35.963241+0000","last_became_peered":"2024-03-31T15:03:35.963241+0000","last_unstale":"2024-03-31T15:03:36.958718+0000","last_undegraded":"2024-03-31T15:03:36.958718+0000","last_fullsized":"2024-03-31T15:03:36.958718+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_clean_scrub_stamp":"2024-03-31T15:03:34.926145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T16:00:45.563865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-31T15:03:36.979092+0000","last_change":"2024-03-31T15:03:36.970959+0000","last_active":"2024-03-31T15:03:36.979092+0000","last_peered":"2024-03-31T15:03:36.979092+0000","last_clean":"2024-03-31T15:03:36.979092+0000","last_became_active":"2024-03-31T15:03:36.970431+0000","last_became_peered":"2024-03-31T15:03:36.970431+0000","last_unstale":"2024-03-31T15:03:36.979092+0000","last_undegraded":"2024-03-31T15:03:36.979092+0000","last_fullsized":"2024-03-31T15:03:36.979092+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:37:26.685300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"20'2","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-31T15:03:36.991659+0000","last_change":"2024-03-31T15:03:35.964624+0000","last_active":"2024-03-31T15:03:36.991659+0000","last_peered":"2024-03-31T15:03:36.991659+0000","last_clean":"2024-03-31T15:03:36.991659+0000","last_became_active":"2024-03-31T15:03:35.964307+0000","last_became_peered":"2024-03-31T15:03:35.964307+0000","last_unstale":"2024-03-31T15:03:36.991659+0000","last_undegraded":"2024-03-31T15:03:36.991659+0000","last_fullsized":"2024-03-31T15:03:36.991659+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_clean_scrub_stamp":"2024-03-31T15:03:34.926145+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T15:58:18.492645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-31T15:03:36.980813+0000","last_change":"2024-03-31T15:03:36.969447+0000","last_active":"2024-03-31T15:03:36.980813+0000","last_peered":"2024-03-31T15:03:36.980813+0000","last_clean":"2024-03-31T15:03:36.980813+0000","last_became_active":"2024-03-31T15:03:36.967372+0000","last_became_peered":"2024-03-31T15:03:36.967372+0000","last_unstale":"2024-03-31T15:03:36.980813+0000","last_undegraded":"2024-03-31T15:03:36.980813+0000","last_fullsized":"2024-03-31T15:03:36.980813+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:51:11.582045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-31T15:03:36.989229+0000","last_change":"2024-03-31T15:03:36.980538+0000","last_active":"2024-03-31T15:03:36.989229+0000","last_peered":"2024-03-31T15:03:36.989229+0000","last_clean":"2024-03-31T15:03:36.989229+0000","last_became_active":"2024-03-31T15:03:36.980255+0000","last_became_peered":"2024-03-31T15:03:36.980255+0000","last_unstale":"2024-03-31T15:03:36.989229+0000","last_undegraded":"2024-03-31T15:03:36.989229+0000","last_fullsized":"2024-03-31T15:03:36.989229+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T16:25:42.516431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"20'2","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-31T15:03:36.985931+0000","last_change":"2024-03-31T15:03:35.962539+0000","last_active":"2024-03-31T15:03:36.985931+0000","last_peered":"2024-03-31T15:03:36.985931+0000","last_clean":"2024-03-31T15:03:36.985931+0000","last_became_active":"2024-03-31T15:03:35.961927+0000","last_became_peered":"2024-03-31T15:03:35.961927+0000","last_unstale":"2024-03-31T15:03:36.985931+0000","last_undegraded":"2024-03-31T15:03:36.985931+0000","last_fullsized":"2024-03-31T15:03:36.985931+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_clean_scrub_stamp":"2024-03-31T15:03:34.926145+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:48:53.962920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-31T15:03:36.989057+0000","last_change":"2024-03-31T15:03:36.981810+0000","last_active":"2024-03-31T15:03:36.989057+0000","last_peered":"2024-03-31T15:03:36.989057+0000","last_clean":"2024-03-31T15:03:36.989057+0000","last_became_active":"2024-03-31T15:03:36.981486+0000","last_became_peered":"2024-03-31T15:03:36.981486+0000","last_unstale":"2024-03-31T15:03:36.989057+0000","last_undegraded":"2024-03-31T15:03:36.989057+0000","last_fullsized":"2024-03-31T15:03:36.989057+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:26:15.339683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"20'2","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-31T15:03:36.991675+0000","last_change":"2024-03-31T15:03:35.972149+0000","last_active":"2024-03-31T15:03:36.991675+0000","last_peered":"2024-03-31T15:03:36.991675+0000","last_clean":"2024-03-31T15:03:36.991675+0000","last_became_active":"2024-03-31T15:03:35.971844+0000","last_became_peered":"2024-03-31T15:03:35.971844+0000","last_unstale":"2024-03-31T15:03:36.991675+0000","last_undegraded":"2024-03-31T15:03:36.991675+0000","last_fullsized":"2024-03-31T15:03:36.991675+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_clean_scrub_stamp":"2024-03-31T15:03:34.926145+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:19:23.333770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-31T15:03:36.989455+0000","last_change":"2024-03-31T15:03:36.988093+0000","last_active":"2024-03-31T15:03:36.989455+0000","last_peered":"2024-03-31T15:03:36.989455+0000","last_clean":"2024-03-31T15:03:36.989455+0000","last_became_active":"2024-03-31T15:03:36.987794+0000","last_became_peered":"2024-03-31T15:03:36.987794+0000","last_unstale":"2024-03-31T15:03:36.989455+0000","last_undegraded":"2024-03-31T15:03:36.989455+0000","last_fullsized":"2024-03-31T15:03:36.989455+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:11:51.069316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"20'1","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-31T15:03:36.986305+0000","last_change":"2024-03-31T15:03:35.969099+0000","last_active":"2024-03-31T15:03:36.986305+0000","last_peered":"2024-03-31T15:03:36.986305+0000","last_clean":"2024-03-31T15:03:36.986305+0000","last_became_active":"2024-03-31T15:03:35.968350+0000","last_became_peered":"2024-03-31T15:03:35.968350+0000","last_unstale":"2024-03-31T15:03:36.986305+0000","last_undegraded":"2024-03-31T15:03:36.986305+0000","last_fullsized":"2024-03-31T15:03:36.986305+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_clean_scrub_stamp":"2024-03-31T15:03:34.926145+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T17:31:20.984335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-31T15:03:36.965360+0000","last_change":"2024-03-31T15:03:36.960526+0000","last_active":"2024-03-31T15:03:36.965360+0000","last_peered":"2024-03-31T15:03:36.965360+0000","last_clean":"2024-03-31T15:03:36.965360+0000","last_became_active":"2024-03-31T15:03:36.960245+0000","last_became_peered":"2024-03-31T15:03:36.960245+0000","last_unstale":"2024-03-31T15:03:36.965360+0000","last_undegraded":"2024-03-31T15:03:36.965360+0000","last_fullsized":"2024-03-31T15:03:36.965360+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:51:11.582045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"20'4","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-31T15:03:36.987133+0000","last_change":"2024-03-31T15:03:35.965202+0000","last_active":"2024-03-31T15:03:36.987133+0000","last_peered":"2024-03-31T15:03:36.987133+0000","last_clean":"2024-03-31T15:03:36.987133+0000","last_became_active":"2024-03-31T15:03:35.964867+0000","last_became_peered":"2024-03-31T15:03:35.964867+0000","last_unstale":"2024-03-31T15:03:36.987133+0000","last_undegraded":"2024-03-31T15:03:36.987133+0000","last_fullsized":"2024-03-31T15:03:36.987133+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_clean_scrub_stamp":"2024-03-31T15:03:34.926145+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T16:35:11.030977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509447,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27804,"kb_used_data":776,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344036,"statfs":{"total":96636764160,"available":96608292864,"internally_reserved":0,"allocated":794624,"data_stored":657545,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509446,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27216,"kb_used_data":196,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344624,"statfs":{"total":96636764160,"available":96608894976,"internally_reserved":0,"allocated":200704,"data_stored":66685,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":4,"up_from":15,"seq":64424509445,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":221184,"data_stored":73021,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509444,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31372,"kb_used_data":192,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340468,"statfs":{"total":96636764160,"available":96604639232,"internally_reserved":0,"allocated":196608,"data_stored":66595,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542150,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27200,"kb_used_data":180,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344640,"statfs":{"total":96636764160,"available":96608911360,"internally_reserved":0,"allocated":184320,"data_stored":59820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542150,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31360,"kb_used_data":180,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340480,"statfs":{"total":96636764160,"available":96604651520,"internally_reserved":0,"allocated":184320,"data_stored":59820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542150,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31384,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340456,"statfs":{"total":96636764160,"available":96604626944,"internally_reserved":0,"allocated":204800,"data_stored":66755,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542150,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31972,"kb_used_data":784,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339868,"statfs":{"total":96636764160,"available":96604024832,"internally_reserved":0,"allocated":802816,"data_stored":657601,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-31T15:03:41.474 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-31T15:03:41.474 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-03-31T15:03:41.474 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-03-31T15:03:41.474 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-03-31T15:03:41.475 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-03-31T15:03:41.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:41.653+0000 7f5c11a00640 1 Processor -- start 2024-03-31T15:03:41.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:41.653+0000 7f5c11a00640 1 -- start start 2024-03-31T15:03:41.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:41.657+0000 7f5c11a00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f5c0c0b23d0 0x7f5c0c15cd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:41.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:41.657+0000 7f5c11a00640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5c0c160e70 con 0x7f5c0c0728b0 2024-03-31T15:03:41.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:41.657+0000 7f5c11a00640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5c0c1602f0 con 0x7f5c0c145d10 2024-03-31T15:03:41.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:41.657+0000 7f5c11a00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f5c0c1605d0 con 0x7f5c0c0b23d0 2024-03-31T15:03:41.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:41.657+0000 7f5c0a000640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f5c0c0728b0 0x7f5c0c0b1cc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:33950/0 (socket says 172.21.15.16:33950) 2024-03-31T15:03:41.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:41.657+0000 7f5c0a000640 1 -- 172.21.15.16:0/911515289 learned_addr learned my addr 172.21.15.16:0/911515289 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:03:41.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:41.657+0000 7f5c0b400640 1 --2- 172.21.15.16:0/911515289 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f5c0c0b23d0 0x7f5c0c15cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:41.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:41.657+0000 7f5c09600640 1 -- 172.21.15.16:0/911515289 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 869140911 0 0) 0x7f5c0c160e70 con 0x7f5c0c0728b0 2024-03-31T15:03:41.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:41.657+0000 7f5c09600640 1 -- 172.21.15.16:0/911515289 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5bf0003530 con 0x7f5c0c0728b0 2024-03-31T15:03:41.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:41.657+0000 7f5c09600640 1 -- 172.21.15.16:0/911515289 <== mon.1 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2931309266 0 0) 0x7f5c0c1602f0 con 0x7f5c0c145d10 2024-03-31T15:03:41.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:41.657+0000 7f5c09600640 1 -- 172.21.15.16:0/911515289 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5c0c160e70 con 0x7f5c0c145d10 2024-03-31T15:03:41.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:41.657+0000 7f5c0b400640 1 -- 172.21.15.16:0/911515289 >> v1:172.21.15.89:6789/0 conn(0x7f5c0c145d10 legacy=0x7f5c0c0cf790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:41.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:41.657+0000 7f5c0b400640 1 -- 172.21.15.16:0/911515289 >> v1:172.21.15.16:6789/0 conn(0x7f5c0c0728b0 legacy=0x7f5c0c0b1cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:41.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:41.657+0000 7f5c0b400640 1 -- 172.21.15.16:0/911515289 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c0c1608b0 con 0x7f5c0c0b23d0 2024-03-31T15:03:41.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:41.661+0000 7f5c0b400640 1 --2- 172.21.15.16:0/911515289 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f5c0c0b23d0 0x7f5c0c15cd50 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f5c04009dc0 tx=0x7f5c0401c990 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8dbb8c4d710493d4 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:03:41.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:41.661+0000 7f5c09600640 1 -- 172.21.15.16:0/911515289 <== mon.2 v2:172.21.15.89:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c040b2070 con 0x7f5c0c0b23d0 2024-03-31T15:03:41.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:41.661+0000 7f5c09600640 1 -- 172.21.15.16:0/911515289 <== mon.2 v2:172.21.15.89:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5c040a5850 con 0x7f5c0c0b23d0 2024-03-31T15:03:41.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:41.661+0000 7f5c09600640 1 -- 172.21.15.16:0/911515289 <== mon.2 v2:172.21.15.89:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c040a5c00 con 0x7f5c0c0b23d0 2024-03-31T15:03:41.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:41.661+0000 7f5c11a00640 1 -- 172.21.15.16:0/911515289 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f5c0c0b23d0 msgr2=0x7f5c0c15cd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:41.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:41.661+0000 7f5c11a00640 1 --2- 172.21.15.16:0/911515289 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f5c0c0b23d0 0x7f5c0c15cd50 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f5c04009dc0 tx=0x7f5c0401c990 comp rx=0 tx=0).stop 2024-03-31T15:03:41.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:41.661+0000 7f5c11a00640 1 -- 172.21.15.16:0/911515289 shutdown_connections 2024-03-31T15:03:41.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:41.661+0000 7f5c11a00640 1 --2- 172.21.15.16:0/911515289 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f5c0c0b23d0 0x7f5c0c15cd50 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:41.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:41.661+0000 7f5c11a00640 1 -- 172.21.15.16:0/911515289 >> 172.21.15.16:0/911515289 conn(0x7f5c0c06b620 msgr2=0x7f5c0c06ba30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:03:41.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:41.661+0000 7f5c11a00640 1 -- 172.21.15.16:0/911515289 shutdown_connections 2024-03-31T15:03:41.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:41.661+0000 7f5c11a00640 1 -- 172.21.15.16:0/911515289 wait complete. 2024-03-31T15:03:41.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:41.661+0000 7f5c11a00640 1 Processor -- start 2024-03-31T15:03:41.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:41.661+0000 7f5c11a00640 1 -- start start 2024-03-31T15:03:41.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:41.661+0000 7f5c11a00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5c0c0728b0 0x7f5c0c13efe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:41.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:41.661+0000 7f5c11a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5c0c0b23d0 0x7f5c0c13d1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:41.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:41.661+0000 7f5c11a00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f5c0c145d10 0x7f5c0c139290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:41.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:41.661+0000 7f5c11a00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f5c0c072e40 con 0x7f5c0c0b23d0 2024-03-31T15:03:41.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:41.661+0000 7f5c11a00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f5c0c072cc0 con 0x7f5c0c0728b0 2024-03-31T15:03:41.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:41.661+0000 7f5c11a00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f5c0c16a100 con 0x7f5c0c145d10 2024-03-31T15:03:41.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:41.665+0000 7f5c0a000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5c0c0b23d0 0x7f5c0c13d1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:41.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:41.665+0000 7f5c0aa00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5c0c0728b0 0x7f5c0c13efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:41.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:41.665+0000 7f5c0a000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5c0c0b23d0 0x7f5c0c13d1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38906/0 (socket says 172.21.15.16:38906) 2024-03-31T15:03:41.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:41.665+0000 7f5c0b400640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f5c0c145d10 0x7f5c0c139290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:41.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:41.665+0000 7f5c0a000640 1 -- 172.21.15.16:0/3825062075 learned_addr learned my addr 172.21.15.16:0/3825062075 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:03:41.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:41.665+0000 7f5c0a000640 1 -- 172.21.15.16:0/3825062075 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f5c0c145d10 msgr2=0x7f5c0c139290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:41.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:41.665+0000 7f5c0a000640 1 --2- 172.21.15.16:0/3825062075 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f5c0c145d10 0x7f5c0c139290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:41.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:41.665+0000 7f5c0a000640 1 -- 172.21.15.16:0/3825062075 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5c0c0728b0 msgr2=0x7f5c0c13efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:41.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:41.665+0000 7f5c0a000640 1 --2- 172.21.15.16:0/3825062075 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5c0c0728b0 0x7f5c0c13efe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:41.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:41.665+0000 7f5c0a000640 1 -- 172.21.15.16:0/3825062075 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c0c167b10 con 0x7f5c0c0b23d0 2024-03-31T15:03:41.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:41.665+0000 7f5c0a000640 1 --2- 172.21.15.16:0/3825062075 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5c0c0b23d0 0x7f5c0c13d1d0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f5bf800c4d0 tx=0x7f5bf800c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:03:41.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:41.665+0000 7f5c02a00640 1 -- 172.21.15.16:0/3825062075 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bf8017070 con 0x7f5c0c0b23d0 2024-03-31T15:03:41.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:41.665+0000 7f5c02a00640 1 -- 172.21.15.16:0/3825062075 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5bf8005cf0 con 0x7f5c0c0b23d0 2024-03-31T15:03:41.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:41.665+0000 7f5c02a00640 1 -- 172.21.15.16:0/3825062075 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bf8007270 con 0x7f5c0c0b23d0 2024-03-31T15:03:41.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:41.665+0000 7f5c11a00640 1 -- 172.21.15.16:0/3825062075 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5c0c0b0620 con 0x7f5c0c0b23d0 2024-03-31T15:03:41.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:41.665+0000 7f5c11a00640 1 -- 172.21.15.16:0/3825062075 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5c0c0b0e60 con 0x7f5c0c0b23d0 2024-03-31T15:03:41.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:41.665+0000 7f5c11a00640 1 -- 172.21.15.16:0/3825062075 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5c0c069950 con 0x7f5c0c0b23d0 2024-03-31T15:03:41.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:41.665+0000 7f5c02a00640 1 -- 172.21.15.16:0/3825062075 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f5bf801a070 con 0x7f5c0c0b23d0 2024-03-31T15:03:41.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:41.669+0000 7f5c02a00640 1 --2- 172.21.15.16:0/3825062075 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f5bec0683b0 0x7f5bec06a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:41.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:41.669+0000 7f5c02a00640 1 -- 172.21.15.16:0/3825062075 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f5bf8085ce0 con 0x7f5c0c0b23d0 2024-03-31T15:03:41.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:41.669+0000 7f5c0aa00640 1 --2- 172.21.15.16:0/3825062075 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f5bec0683b0 0x7f5bec06a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:41.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:41.669+0000 7f5c0aa00640 1 --2- 172.21.15.16:0/3825062075 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f5bec0683b0 0x7f5bec06a870 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f5bfc007fd0 tx=0x7f5bfc009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:03:41.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:41.673+0000 7f5c02a00640 1 -- 172.21.15.16:0/3825062075 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f5c0c0b0620 con 0x7f5c0c0b23d0 2024-03-31T15:03:41.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:41.877+0000 7f5c11a00640 1 -- 172.21.15.16:0/3825062075 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) -- 0x7f5c0c06c630 con 0x7f5c0c0b23d0 2024-03-31T15:03:41.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:41.877+0000 7f5c02a00640 1 -- 172.21.15.16:0/3825062075 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v5) ==== 74+0+168085 (secure 0 0 0) 0x7f5bf8055bb0 con 0x7f5c0c0b23d0 2024-03-31T15:03:41.877 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:03:41.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:41.881+0000 7f5c11a00640 1 -- 172.21.15.16:0/3825062075 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f5bec0683b0 msgr2=0x7f5bec06a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:41.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:41.881+0000 7f5c11a00640 1 --2- 172.21.15.16:0/3825062075 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f5bec0683b0 0x7f5bec06a870 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f5bfc007fd0 tx=0x7f5bfc009040 comp rx=0 tx=0).stop 2024-03-31T15:03:41.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:41.881+0000 7f5c11a00640 1 -- 172.21.15.16:0/3825062075 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5c0c0b23d0 msgr2=0x7f5c0c13d1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:41.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:41.881+0000 7f5c11a00640 1 --2- 172.21.15.16:0/3825062075 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5c0c0b23d0 0x7f5c0c13d1d0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f5bf800c4d0 tx=0x7f5bf800c990 comp rx=0 tx=0).stop 2024-03-31T15:03:41.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:41.881+0000 7f5c11a00640 1 -- 172.21.15.16:0/3825062075 shutdown_connections 2024-03-31T15:03:41.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:41.881+0000 7f5c11a00640 1 --2- 172.21.15.16:0/3825062075 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f5bec0683b0 0x7f5bec06a870 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:41.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:41.881+0000 7f5c11a00640 1 --2- 172.21.15.16:0/3825062075 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f5c0c145d10 0x7f5c0c139290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:41.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:41.881+0000 7f5c11a00640 1 --2- 172.21.15.16:0/3825062075 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5c0c0b23d0 0x7f5c0c13d1d0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:41.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:41.881+0000 7f5c11a00640 1 --2- 172.21.15.16:0/3825062075 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5c0c0728b0 0x7f5c0c13efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:41.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:41.881+0000 7f5c11a00640 1 -- 172.21.15.16:0/3825062075 >> 172.21.15.16:0/3825062075 conn(0x7f5c0c06b620 msgr2=0x7f5c0c073050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:03:41.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:41.881+0000 7f5c11a00640 1 -- 172.21.15.16:0/3825062075 shutdown_connections 2024-03-31T15:03:41.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:41.881+0000 7f5c11a00640 1 -- 172.21.15.16:0/3825062075 wait complete. 2024-03-31T15:03:41.897 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":5,"flags":0,"active_gid":4100,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":1549407273},{"type":"v1","addr":"172.21.15.89:6833","nonce":1549407273}]},"active_addr":"172.21.15.89:6833/1549407273","active_change":"2024-03-31T15:03:16.513756+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":4107,"name":"y","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.89:0","nonce":2118314676}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.89:0","nonce":2693354765}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.89:0","nonce":2689087320}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.89:0","nonce":3193214790}]}]} 2024-03-31T15:03:41.902 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-03-31T15:03:41.902 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-03-31T15:03:41.903 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-31T15:03:42.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.045+0000 7f8c61a00640 1 Processor -- start 2024-03-31T15:03:42.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.045+0000 7f8c61a00640 1 -- start start 2024-03-31T15:03:42.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.049+0000 7f8c61a00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f8c5c145e40 0x7f8c5c0cc740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:42.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.049+0000 7f8c61a00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f8c5c070030 0x7f8c5c0d2010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:42.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.049+0000 7f8c61a00640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8c5c0d0920 con 0x7f8c5c0ccc80 2024-03-31T15:03:42.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.049+0000 7f8c61a00640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f8c5c0d06d0 con 0x7f8c5c070030 2024-03-31T15:03:42.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.049+0000 7f8c61a00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f8c5c0d0c00 con 0x7f8c5c145e40 2024-03-31T15:03:42.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.049+0000 7f8c5b400640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f8c5c070030 0x7f8c5c0d2010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:42.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.049+0000 7f8c5b400640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f8c5c070030 0x7f8c5c0d2010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38914/0 (socket says 172.21.15.16:38914) 2024-03-31T15:03:42.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.049+0000 7f8c5b400640 1 -- 172.21.15.16:0/2449445303 learned_addr learned my addr 172.21.15.16:0/2449445303 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:03:42.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.049+0000 7f8c5aa00640 1 --2- 172.21.15.16:0/2449445303 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f8c5c145e40 0x7f8c5c0cc740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:42.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.049+0000 7f8c5b400640 1 -- 172.21.15.16:0/2449445303 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f8c5c145e40 msgr2=0x7f8c5c0cc740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:03:42.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.049+0000 7f8c5b400640 1 --2- 172.21.15.16:0/2449445303 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f8c5c145e40 0x7f8c5c0cc740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:42.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.049+0000 7f8c5b400640 1 -- 172.21.15.16:0/2449445303 >> v1:172.21.15.89:6789/0 conn(0x7f8c5c0ccc80 legacy=0x7f8c5c06f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:42.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.049+0000 7f8c5b400640 1 -- 172.21.15.16:0/2449445303 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c5c0d1810 con 0x7f8c5c070030 2024-03-31T15:03:42.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.053+0000 7f8c5b400640 1 --2- 172.21.15.16:0/2449445303 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8c5c070030 0x7f8c5c0d2010 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f8c54009870 tx=0x7f8c5401c560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55fb1843d8012e73 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:03:42.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.053+0000 7f8c5a000640 1 -- 172.21.15.16:0/2449445303 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c540a7070 con 0x7f8c5c070030 2024-03-31T15:03:42.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.053+0000 7f8c5a000640 1 -- 172.21.15.16:0/2449445303 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8c540b0070 con 0x7f8c5c070030 2024-03-31T15:03:42.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.053+0000 7f8c5a000640 1 -- 172.21.15.16:0/2449445303 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c540ab890 con 0x7f8c5c070030 2024-03-31T15:03:42.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.053+0000 7f8c61a00640 1 -- 172.21.15.16:0/2449445303 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8c5c070030 msgr2=0x7f8c5c0d2010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:42.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.053+0000 7f8c61a00640 1 --2- 172.21.15.16:0/2449445303 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8c5c070030 0x7f8c5c0d2010 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f8c54009870 tx=0x7f8c5401c560 comp rx=0 tx=0).stop 2024-03-31T15:03:42.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.053+0000 7f8c61a00640 1 -- 172.21.15.16:0/2449445303 shutdown_connections 2024-03-31T15:03:42.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.053+0000 7f8c61a00640 1 --2- 172.21.15.16:0/2449445303 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8c5c070030 0x7f8c5c0d2010 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:42.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.053+0000 7f8c61a00640 1 --2- 172.21.15.16:0/2449445303 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f8c5c145e40 0x7f8c5c0cc740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:42.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.053+0000 7f8c61a00640 1 -- 172.21.15.16:0/2449445303 >> 172.21.15.16:0/2449445303 conn(0x7f8c5c067330 msgr2=0x7f8c5c067740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:03:42.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.053+0000 7f8c61a00640 1 -- 172.21.15.16:0/2449445303 shutdown_connections 2024-03-31T15:03:42.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.053+0000 7f8c61a00640 1 -- 172.21.15.16:0/2449445303 wait complete. 2024-03-31T15:03:42.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.053+0000 7f8c61a00640 1 Processor -- start 2024-03-31T15:03:42.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.053+0000 7f8c61a00640 1 -- start start 2024-03-31T15:03:42.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.053+0000 7f8c61a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8c5c070030 0x7f8c5c13cb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:42.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.053+0000 7f8c61a00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f8c5c0ccc80 0x7f8c5c15e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:42.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.053+0000 7f8c61a00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8c5c145e40 0x7f8c5c13c3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:42.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.053+0000 7f8c61a00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f8c5c0b5240 con 0x7f8c5c070030 2024-03-31T15:03:42.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.053+0000 7f8c61a00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f8c5c0b50c0 con 0x7f8c5c145e40 2024-03-31T15:03:42.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.053+0000 7f8c61a00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f8c5c169190 con 0x7f8c5c0ccc80 2024-03-31T15:03:42.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.057+0000 7f8c5aa00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8c5c070030 0x7f8c5c13cb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:42.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.057+0000 7f8c5aa00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8c5c070030 0x7f8c5c13cb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38916/0 (socket says 172.21.15.16:38916) 2024-03-31T15:03:42.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.057+0000 7f8c5aa00640 1 -- 172.21.15.16:0/82347313 learned_addr learned my addr 172.21.15.16:0/82347313 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:03:42.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.057+0000 7f8c52000640 1 --2- 172.21.15.16:0/82347313 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f8c5c0ccc80 0x7f8c5c15e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:42.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.057+0000 7f8c5b400640 1 --2- 172.21.15.16:0/82347313 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8c5c145e40 0x7f8c5c13c3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:42.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.057+0000 7f8c5aa00640 1 -- 172.21.15.16:0/82347313 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f8c5c0ccc80 msgr2=0x7f8c5c15e9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:42.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.057+0000 7f8c5aa00640 1 --2- 172.21.15.16:0/82347313 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f8c5c0ccc80 0x7f8c5c15e9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:42.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.057+0000 7f8c5aa00640 1 -- 172.21.15.16:0/82347313 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8c5c145e40 msgr2=0x7f8c5c13c3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:42.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.057+0000 7f8c5aa00640 1 --2- 172.21.15.16:0/82347313 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8c5c145e40 0x7f8c5c13c3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:42.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.057+0000 7f8c5aa00640 1 -- 172.21.15.16:0/82347313 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c5c13d190 con 0x7f8c5c070030 2024-03-31T15:03:42.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.057+0000 7f8c52000640 1 --2- 172.21.15.16:0/82347313 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f8c5c0ccc80 0x7f8c5c15e9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:03:42.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.057+0000 7f8c5aa00640 1 --2- 172.21.15.16:0/82347313 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8c5c070030 0x7f8c5c13cb00 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f8c4c009d10 tx=0x7f8c4c00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:03:42.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.057+0000 7f8c53400640 1 -- 172.21.15.16:0/82347313 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c4c018020 con 0x7f8c5c070030 2024-03-31T15:03:42.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.057+0000 7f8c53400640 1 -- 172.21.15.16:0/82347313 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8c4c014480 con 0x7f8c5c070030 2024-03-31T15:03:42.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.057+0000 7f8c53400640 1 -- 172.21.15.16:0/82347313 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c4c014830 con 0x7f8c5c070030 2024-03-31T15:03:42.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.057+0000 7f8c61a00640 1 -- 172.21.15.16:0/82347313 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8c5c13d470 con 0x7f8c5c070030 2024-03-31T15:03:42.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.057+0000 7f8c61a00640 1 -- 172.21.15.16:0/82347313 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8c5c13da30 con 0x7f8c5c070030 2024-03-31T15:03:42.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.057+0000 7f8c61a00640 1 -- 172.21.15.16:0/82347313 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8c5c069950 con 0x7f8c5c070030 2024-03-31T15:03:42.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.057+0000 7f8c53400640 1 -- 172.21.15.16:0/82347313 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f8c4c01b020 con 0x7f8c5c070030 2024-03-31T15:03:42.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.057+0000 7f8c53400640 1 --2- 172.21.15.16:0/82347313 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f8c38068380 0x7f8c3806a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:42.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.061+0000 7f8c53400640 1 -- 172.21.15.16:0/82347313 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f8c4c087450 con 0x7f8c5c070030 2024-03-31T15:03:42.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.061+0000 7f8c52000640 1 --2- 172.21.15.16:0/82347313 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f8c38068380 0x7f8c3806a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:42.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.061+0000 7f8c52000640 1 --2- 172.21.15.16:0/82347313 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f8c38068380 0x7f8c3806a840 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f8c5c139530 tx=0x7f8c44009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:03:42.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.065+0000 7f8c53400640 1 -- 172.21.15.16:0/82347313 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f8c4c022030 con 0x7f8c5c070030 2024-03-31T15:03:42.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.229+0000 7f8c61a00640 1 -- 172.21.15.16:0/82347313 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f8c5c0cf3c0 con 0x7f8c5c070030 2024-03-31T15:03:42.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.229+0000 7f8c53400640 1 -- 172.21.15.16:0/82347313 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) ==== 74+0+16533 (secure 0 0 0) 0x7f8c4c010040 con 0x7f8c5c070030 2024-03-31T15:03:42.226 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:03:42.226 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":21,"fsid":"286eb5da-fb47-4550-b290-d9519c13b9a5","created":"2024-03-31T15:03:11.438952+0000","modified":"2024-03-31T15:03:37.957829+0000","last_up_change":"2024-03-31T15:03:22.885535+0000","last_in_change":"2024-03-31T15:03:13.941667+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-31T15:03:22.550787+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-31T15:03:34.488425+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-31T15:03:35.299116+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"92a66450-ef19-40d8-ac3d-d760b23b0bda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6801","nonce":932712244}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6803","nonce":932712244}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6807","nonce":932712244}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6805","nonce":932712244}]},"public_addr":"172.21.15.16:6801/932712244","cluster_addr":"172.21.15.16:6803/932712244","heartbeat_back_addr":"172.21.15.16:6807/932712244","heartbeat_front_addr":"172.21.15.16:6805/932712244","state":["exists","up"]},{"osd":1,"uuid":"da9d1f2b-1968-4959-b843-faa231988e92","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6810","nonce":1398737533}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6811","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6813","nonce":1398737533}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6821","nonce":1398737533}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6815","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6816","nonce":1398737533}]},"public_addr":"172.21.15.16:6810/1398737533","cluster_addr":"172.21.15.16:6813/1398737533","heartbeat_back_addr":"172.21.15.16:6821/1398737533","heartbeat_front_addr":"172.21.15.16:6816/1398737533","state":["exists","up"]},{"osd":2,"uuid":"01a73662-9a21-4b1f-bcbb-42b9636a4638","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6819","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6822","nonce":3018526646}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6825","nonce":3018526646}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6831","nonce":3018526646}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6827","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6829","nonce":3018526646}]},"public_addr":"172.21.15.16:6822/3018526646","cluster_addr":"172.21.15.16:6825/3018526646","heartbeat_back_addr":"172.21.15.16:6831/3018526646","heartbeat_front_addr":"172.21.15.16:6829/3018526646","state":["exists","up"]},{"osd":3,"uuid":"c7771fd3-e18e-4500-ac38-23cf10ebcd75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6809","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6812","nonce":3360352974}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6817","nonce":3360352974}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6828","nonce":3360352974}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6823","nonce":3360352974}]},"public_addr":"172.21.15.16:6812/3360352974","cluster_addr":"172.21.15.16:6817/3360352974","heartbeat_back_addr":"172.21.15.16:6828/3360352974","heartbeat_front_addr":"172.21.15.16:6823/3360352974","state":["exists","up"]},{"osd":4,"uuid":"88260730-1482-4fbd-a40f-cb1124e9df4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6801","nonce":1856844856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6803","nonce":1856844856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6807","nonce":1856844856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6805","nonce":1856844856}]},"public_addr":"172.21.15.89:6801/1856844856","cluster_addr":"172.21.15.89:6803/1856844856","heartbeat_back_addr":"172.21.15.89:6807/1856844856","heartbeat_front_addr":"172.21.15.89:6805/1856844856","state":["exists","up"]},{"osd":5,"uuid":"4cb2c00c-fae0-4dd8-9d27-ec4f0093ccf7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6814","nonce":50395527}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6817","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6820","nonce":50395527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6831","nonce":50395527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6823","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6826","nonce":50395527}]},"public_addr":"172.21.15.89:6814/50395527","cluster_addr":"172.21.15.89:6820/50395527","heartbeat_back_addr":"172.21.15.89:6831/50395527","heartbeat_front_addr":"172.21.15.89:6826/50395527","state":["exists","up"]},{"osd":6,"uuid":"81ce0c19-2554-4cae-acbe-66a05e9e9b9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6809","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6812","nonce":3470471752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6815","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6818","nonce":3470471752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6827","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6829","nonce":3470471752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6821","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6824","nonce":3470471752}]},"public_addr":"172.21.15.89:6812/3470471752","cluster_addr":"172.21.15.89:6818/3470471752","heartbeat_back_addr":"172.21.15.89:6829/3470471752","heartbeat_front_addr":"172.21.15.89:6824/3470471752","state":["exists","up"]},{"osd":7,"uuid":"85e4313b-d37d-442e-8042-153bf20a847d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6811","nonce":1796778630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6813","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6816","nonce":1796778630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6825","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6828","nonce":1796778630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6819","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6822","nonce":1796778630}]},"public_addr":"172.21.15.89:6811/1796778630","cluster_addr":"172.21.15.89:6816/1796778630","heartbeat_back_addr":"172.21.15.89:6828/1796778630","heartbeat_front_addr":"172.21.15.89:6822/1796778630","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.439088+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.529970+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.680512+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:21.044161+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.582111+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.536558+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.409522+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.590229+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-31T15:03:42.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.229+0000 7f8c61a00640 1 -- 172.21.15.16:0/82347313 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f8c38068380 msgr2=0x7f8c3806a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:42.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.229+0000 7f8c61a00640 1 --2- 172.21.15.16:0/82347313 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f8c38068380 0x7f8c3806a840 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f8c5c139530 tx=0x7f8c44009040 comp rx=0 tx=0).stop 2024-03-31T15:03:42.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.229+0000 7f8c61a00640 1 -- 172.21.15.16:0/82347313 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8c5c070030 msgr2=0x7f8c5c13cb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:42.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.229+0000 7f8c61a00640 1 --2- 172.21.15.16:0/82347313 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8c5c070030 0x7f8c5c13cb00 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f8c4c009d10 tx=0x7f8c4c00c490 comp rx=0 tx=0).stop 2024-03-31T15:03:42.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.229+0000 7f8c61a00640 1 -- 172.21.15.16:0/82347313 shutdown_connections 2024-03-31T15:03:42.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.229+0000 7f8c61a00640 1 --2- 172.21.15.16:0/82347313 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f8c38068380 0x7f8c3806a840 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:42.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.229+0000 7f8c61a00640 1 --2- 172.21.15.16:0/82347313 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8c5c145e40 0x7f8c5c13c3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:42.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.229+0000 7f8c61a00640 1 --2- 172.21.15.16:0/82347313 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f8c5c0ccc80 0x7f8c5c15e9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:42.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.229+0000 7f8c61a00640 1 --2- 172.21.15.16:0/82347313 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8c5c070030 0x7f8c5c13cb00 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:42.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.229+0000 7f8c61a00640 1 -- 172.21.15.16:0/82347313 >> 172.21.15.16:0/82347313 conn(0x7f8c5c067330 msgr2=0x7f8c5c0b2f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:03:42.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.229+0000 7f8c61a00640 1 -- 172.21.15.16:0/82347313 shutdown_connections 2024-03-31T15:03:42.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.229+0000 7f8c61a00640 1 -- 172.21.15.16:0/82347313 wait complete. 2024-03-31T15:03:42.246 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-03-31T15:03:42.247 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-31T15:03:42.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.389+0000 7fb852800640 1 Processor -- start 2024-03-31T15:03:42.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.389+0000 7fb852800640 1 -- start start 2024-03-31T15:03:42.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.389+0000 7fb852800640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fb84c145d10 0x7fb84c0cf790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:42.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.393+0000 7fb852800640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb84c160e70 con 0x7fb84c0728b0 2024-03-31T15:03:42.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.393+0000 7fb852800640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb84c0cfcd0 con 0x7fb84c0b23d0 2024-03-31T15:03:42.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.393+0000 7fb852800640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7fb84c1602f0 con 0x7fb84c145d10 2024-03-31T15:03:42.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.393+0000 7fb842a00640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fb84c0728b0 0x7fb84c0b1cc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:33954/0 (socket says 172.21.15.16:33954) 2024-03-31T15:03:42.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.393+0000 7fb842a00640 1 -- 172.21.15.16:0/2680696527 learned_addr learned my addr 172.21.15.16:0/2680696527 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:03:42.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.393+0000 7fb84b400640 1 --2- 172.21.15.16:0/2680696527 >> v2:172.21.15.89:3300/0 conn(0x7fb84c145d10 0x7fb84c0cf790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:42.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.393+0000 7fb84aa00640 1 -- 172.21.15.16:0/2680696527 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3626605905 0 0) 0x7fb84c160e70 con 0x7fb84c0728b0 2024-03-31T15:03:42.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.393+0000 7fb84aa00640 1 -- 172.21.15.16:0/2680696527 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb830003530 con 0x7fb84c0728b0 2024-03-31T15:03:42.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.393+0000 7fb84aa00640 1 -- 172.21.15.16:0/2680696527 <== mon.1 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1293751896 0 0) 0x7fb84c0cfcd0 con 0x7fb84c0b23d0 2024-03-31T15:03:42.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.393+0000 7fb84aa00640 1 -- 172.21.15.16:0/2680696527 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb84c160e70 con 0x7fb84c0b23d0 2024-03-31T15:03:42.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.393+0000 7fb84b400640 1 -- 172.21.15.16:0/2680696527 >> v1:172.21.15.89:6789/0 conn(0x7fb84c0b23d0 legacy=0x7fb84c15cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:42.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.393+0000 7fb84b400640 1 -- 172.21.15.16:0/2680696527 >> v1:172.21.15.16:6789/0 conn(0x7fb84c0728b0 legacy=0x7fb84c0b1cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:42.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.393+0000 7fb84b400640 1 -- 172.21.15.16:0/2680696527 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb84c1605d0 con 0x7fb84c145d10 2024-03-31T15:03:42.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.393+0000 7fb84b400640 1 --2- 172.21.15.16:0/2680696527 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb84c145d10 0x7fb84c0cf790 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fb83c0097c0 tx=0x7fb83c01c4a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=50eecc45978ff634 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:03:42.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.397+0000 7fb84aa00640 1 -- 172.21.15.16:0/2680696527 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb83c0b0070 con 0x7fb84c145d10 2024-03-31T15:03:42.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.397+0000 7fb84aa00640 1 -- 172.21.15.16:0/2680696527 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb83c0ab600 con 0x7fb84c145d10 2024-03-31T15:03:42.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.397+0000 7fb84aa00640 1 -- 172.21.15.16:0/2680696527 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb83c0ab9d0 con 0x7fb84c145d10 2024-03-31T15:03:42.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.397+0000 7fb852800640 1 -- 172.21.15.16:0/2680696527 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb84c145d10 msgr2=0x7fb84c0cf790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:42.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.397+0000 7fb852800640 1 --2- 172.21.15.16:0/2680696527 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb84c145d10 0x7fb84c0cf790 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fb83c0097c0 tx=0x7fb83c01c4a0 comp rx=0 tx=0).stop 2024-03-31T15:03:42.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.397+0000 7fb852800640 1 -- 172.21.15.16:0/2680696527 shutdown_connections 2024-03-31T15:03:42.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.397+0000 7fb852800640 1 --2- 172.21.15.16:0/2680696527 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb84c145d10 0x7fb84c0cf790 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:42.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.397+0000 7fb852800640 1 -- 172.21.15.16:0/2680696527 >> 172.21.15.16:0/2680696527 conn(0x7fb84c06b620 msgr2=0x7fb84c06ba30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:03:42.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.397+0000 7fb852800640 1 -- 172.21.15.16:0/2680696527 shutdown_connections 2024-03-31T15:03:42.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.397+0000 7fb852800640 1 -- 172.21.15.16:0/2680696527 wait complete. 2024-03-31T15:03:42.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.397+0000 7fb852800640 1 Processor -- start 2024-03-31T15:03:42.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.397+0000 7fb852800640 1 -- start start 2024-03-31T15:03:42.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.397+0000 7fb852800640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb84c0728b0 0x7fb84c14daa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:42.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.397+0000 7fb852800640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb84c0b23d0 0x7fb84c164ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:42.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.397+0000 7fb852800640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb84c145d10 0x7fb84c162b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:42.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.397+0000 7fb852800640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fb84c0cf350 con 0x7fb84c0b23d0 2024-03-31T15:03:42.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.397+0000 7fb852800640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fb84c15d870 con 0x7fb84c0728b0 2024-03-31T15:03:42.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.397+0000 7fb852800640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fb84c0cf4d0 con 0x7fb84c145d10 2024-03-31T15:03:42.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.397+0000 7fb842a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb84c0b23d0 0x7fb84c164ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:42.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.397+0000 7fb842a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb84c0b23d0 0x7fb84c164ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38932/0 (socket says 172.21.15.16:38932) 2024-03-31T15:03:42.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.397+0000 7fb842a00640 1 -- 172.21.15.16:0/2031732157 learned_addr learned my addr 172.21.15.16:0/2031732157 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:03:42.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.397+0000 7fb84be00640 1 --2- 172.21.15.16:0/2031732157 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb84c145d10 0x7fb84c162b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:42.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.397+0000 7fb84b400640 1 --2- 172.21.15.16:0/2031732157 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb84c0728b0 0x7fb84c14daa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:42.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.397+0000 7fb842a00640 1 -- 172.21.15.16:0/2031732157 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb84c145d10 msgr2=0x7fb84c162b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:42.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.397+0000 7fb842a00640 1 --2- 172.21.15.16:0/2031732157 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb84c145d10 0x7fb84c162b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:42.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.397+0000 7fb842a00640 1 -- 172.21.15.16:0/2031732157 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb84c0728b0 msgr2=0x7fb84c14daa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:42.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.397+0000 7fb842a00640 1 --2- 172.21.15.16:0/2031732157 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb84c0728b0 0x7fb84c14daa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:42.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.397+0000 7fb842a00640 1 -- 172.21.15.16:0/2031732157 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb84c141910 con 0x7fb84c0b23d0 2024-03-31T15:03:42.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.401+0000 7fb842a00640 1 --2- 172.21.15.16:0/2031732157 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb84c0b23d0 0x7fb84c164ba0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fb83400c500 tx=0x7fb83400c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:03:42.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.401+0000 7fb843e00640 1 -- 172.21.15.16:0/2031732157 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb834017070 con 0x7fb84c0b23d0 2024-03-31T15:03:42.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.401+0000 7fb843e00640 1 -- 172.21.15.16:0/2031732157 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb834005cf0 con 0x7fb84c0b23d0 2024-03-31T15:03:42.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.401+0000 7fb843e00640 1 -- 172.21.15.16:0/2031732157 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb834007330 con 0x7fb84c0b23d0 2024-03-31T15:03:42.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.401+0000 7fb852800640 1 -- 172.21.15.16:0/2031732157 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb84c141bf0 con 0x7fb84c0b23d0 2024-03-31T15:03:42.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.401+0000 7fb852800640 1 -- 172.21.15.16:0/2031732157 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb84c1421b0 con 0x7fb84c0b23d0 2024-03-31T15:03:42.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.401+0000 7fb852800640 1 -- 172.21.15.16:0/2031732157 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb84c069950 con 0x7fb84c0b23d0 2024-03-31T15:03:42.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.401+0000 7fb843e00640 1 -- 172.21.15.16:0/2031732157 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fb83401a070 con 0x7fb84c0b23d0 2024-03-31T15:03:42.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.401+0000 7fb843e00640 1 --2- 172.21.15.16:0/2031732157 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fb828068380 0x7fb82806a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:42.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.401+0000 7fb843e00640 1 -- 172.21.15.16:0/2031732157 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fb834085db0 con 0x7fb84c0b23d0 2024-03-31T15:03:42.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.405+0000 7fb84b400640 1 --2- 172.21.15.16:0/2031732157 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fb828068380 0x7fb82806a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:42.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.405+0000 7fb84b400640 1 --2- 172.21.15.16:0/2031732157 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fb828068380 0x7fb82806a840 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fb83c0ad000 tx=0x7fb83c008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:03:42.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.409+0000 7fb843e00640 1 -- 172.21.15.16:0/2031732157 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fb83408a070 con 0x7fb84c0b23d0 2024-03-31T15:03:42.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.573+0000 7fb852800640 1 -- 172.21.15.16:0/2031732157 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fb84c0ce040 con 0x7fb84c0b23d0 2024-03-31T15:03:42.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.573+0000 7fb843e00640 1 -- 172.21.15.16:0/2031732157 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) ==== 74+0+16533 (secure 0 0 0) 0x7fb834055d10 con 0x7fb84c0b23d0 2024-03-31T15:03:42.572 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:03:42.572 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":21,"fsid":"286eb5da-fb47-4550-b290-d9519c13b9a5","created":"2024-03-31T15:03:11.438952+0000","modified":"2024-03-31T15:03:37.957829+0000","last_up_change":"2024-03-31T15:03:22.885535+0000","last_in_change":"2024-03-31T15:03:13.941667+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-31T15:03:22.550787+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-31T15:03:34.488425+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-31T15:03:35.299116+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"92a66450-ef19-40d8-ac3d-d760b23b0bda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6801","nonce":932712244}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6803","nonce":932712244}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6807","nonce":932712244}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6805","nonce":932712244}]},"public_addr":"172.21.15.16:6801/932712244","cluster_addr":"172.21.15.16:6803/932712244","heartbeat_back_addr":"172.21.15.16:6807/932712244","heartbeat_front_addr":"172.21.15.16:6805/932712244","state":["exists","up"]},{"osd":1,"uuid":"da9d1f2b-1968-4959-b843-faa231988e92","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6810","nonce":1398737533}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6811","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6813","nonce":1398737533}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6821","nonce":1398737533}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6815","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6816","nonce":1398737533}]},"public_addr":"172.21.15.16:6810/1398737533","cluster_addr":"172.21.15.16:6813/1398737533","heartbeat_back_addr":"172.21.15.16:6821/1398737533","heartbeat_front_addr":"172.21.15.16:6816/1398737533","state":["exists","up"]},{"osd":2,"uuid":"01a73662-9a21-4b1f-bcbb-42b9636a4638","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6819","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6822","nonce":3018526646}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6825","nonce":3018526646}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6831","nonce":3018526646}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6827","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6829","nonce":3018526646}]},"public_addr":"172.21.15.16:6822/3018526646","cluster_addr":"172.21.15.16:6825/3018526646","heartbeat_back_addr":"172.21.15.16:6831/3018526646","heartbeat_front_addr":"172.21.15.16:6829/3018526646","state":["exists","up"]},{"osd":3,"uuid":"c7771fd3-e18e-4500-ac38-23cf10ebcd75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6809","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6812","nonce":3360352974}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6817","nonce":3360352974}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6828","nonce":3360352974}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6823","nonce":3360352974}]},"public_addr":"172.21.15.16:6812/3360352974","cluster_addr":"172.21.15.16:6817/3360352974","heartbeat_back_addr":"172.21.15.16:6828/3360352974","heartbeat_front_addr":"172.21.15.16:6823/3360352974","state":["exists","up"]},{"osd":4,"uuid":"88260730-1482-4fbd-a40f-cb1124e9df4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6801","nonce":1856844856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6803","nonce":1856844856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6807","nonce":1856844856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6805","nonce":1856844856}]},"public_addr":"172.21.15.89:6801/1856844856","cluster_addr":"172.21.15.89:6803/1856844856","heartbeat_back_addr":"172.21.15.89:6807/1856844856","heartbeat_front_addr":"172.21.15.89:6805/1856844856","state":["exists","up"]},{"osd":5,"uuid":"4cb2c00c-fae0-4dd8-9d27-ec4f0093ccf7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6814","nonce":50395527}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6817","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6820","nonce":50395527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6831","nonce":50395527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6823","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6826","nonce":50395527}]},"public_addr":"172.21.15.89:6814/50395527","cluster_addr":"172.21.15.89:6820/50395527","heartbeat_back_addr":"172.21.15.89:6831/50395527","heartbeat_front_addr":"172.21.15.89:6826/50395527","state":["exists","up"]},{"osd":6,"uuid":"81ce0c19-2554-4cae-acbe-66a05e9e9b9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6809","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6812","nonce":3470471752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6815","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6818","nonce":3470471752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6827","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6829","nonce":3470471752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6821","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6824","nonce":3470471752}]},"public_addr":"172.21.15.89:6812/3470471752","cluster_addr":"172.21.15.89:6818/3470471752","heartbeat_back_addr":"172.21.15.89:6829/3470471752","heartbeat_front_addr":"172.21.15.89:6824/3470471752","state":["exists","up"]},{"osd":7,"uuid":"85e4313b-d37d-442e-8042-153bf20a847d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6811","nonce":1796778630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6813","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6816","nonce":1796778630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6825","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6828","nonce":1796778630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6819","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6822","nonce":1796778630}]},"public_addr":"172.21.15.89:6811/1796778630","cluster_addr":"172.21.15.89:6816/1796778630","heartbeat_back_addr":"172.21.15.89:6828/1796778630","heartbeat_front_addr":"172.21.15.89:6822/1796778630","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.439088+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.529970+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.680512+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:21.044161+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.582111+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.536558+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.409522+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.590229+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-31T15:03:42.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.577+0000 7fb852800640 1 -- 172.21.15.16:0/2031732157 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fb828068380 msgr2=0x7fb82806a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:42.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.577+0000 7fb852800640 1 --2- 172.21.15.16:0/2031732157 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fb828068380 0x7fb82806a840 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fb83c0ad000 tx=0x7fb83c008040 comp rx=0 tx=0).stop 2024-03-31T15:03:42.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.577+0000 7fb852800640 1 -- 172.21.15.16:0/2031732157 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb84c0b23d0 msgr2=0x7fb84c164ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:42.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.577+0000 7fb852800640 1 --2- 172.21.15.16:0/2031732157 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb84c0b23d0 0x7fb84c164ba0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fb83400c500 tx=0x7fb83400c9c0 comp rx=0 tx=0).stop 2024-03-31T15:03:42.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.577+0000 7fb852800640 1 -- 172.21.15.16:0/2031732157 shutdown_connections 2024-03-31T15:03:42.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.577+0000 7fb852800640 1 --2- 172.21.15.16:0/2031732157 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fb828068380 0x7fb82806a840 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:42.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.577+0000 7fb852800640 1 --2- 172.21.15.16:0/2031732157 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb84c145d10 0x7fb84c162b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:42.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.577+0000 7fb852800640 1 --2- 172.21.15.16:0/2031732157 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb84c0b23d0 0x7fb84c164ba0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:42.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.577+0000 7fb852800640 1 --2- 172.21.15.16:0/2031732157 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb84c0728b0 0x7fb84c14daa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:42.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.577+0000 7fb852800640 1 -- 172.21.15.16:0/2031732157 >> 172.21.15.16:0/2031732157 conn(0x7fb84c06b620 msgr2=0x7fb84c0b3690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:03:42.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.577+0000 7fb852800640 1 -- 172.21.15.16:0/2031732157 shutdown_connections 2024-03-31T15:03:42.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.577+0000 7fb852800640 1 -- 172.21.15.16:0/2031732157 wait complete. 2024-03-31T15:03:42.594 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-03-31T15:03:42.594 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-03-31T15:03:42.595 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-03-31T15:03:42.595 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-03-31T15:03:42.596 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-03-31T15:03:42.596 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-03-31T15:03:42.596 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2024-03-31T15:03:42.597 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2024-03-31T15:03:42.836 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.841+0000 7f02ef200640 1 Processor -- start 2024-03-31T15:03:42.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.841+0000 7f02ef200640 1 -- start start 2024-03-31T15:03:42.839 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.841+0000 7f02ef200640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f02e4033e40 0x7f02e40eb180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:42.839 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.841+0000 7f02ef200640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f02e4005ef0 0x7f02e4006320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:42.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.841+0000 7f02ef200640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f02e4034220 con 0x7f02e4047c20 2024-03-31T15:03:42.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.841+0000 7f02ef200640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f02e4050210 con 0x7f02e4033e40 2024-03-31T15:03:42.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.841+0000 7f02ef200640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f02e4037bc0 con 0x7f02e4005ef0 2024-03-31T15:03:42.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.841+0000 7f02ede00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f02e4033e40 0x7f02e40eb180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:42.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.841+0000 7f02ed400640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f02e4047c20 0x7f02e40eb6c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.16:49178/0 (socket says 172.21.15.16:49178) 2024-03-31T15:03:42.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.841+0000 7f02ed400640 1 -- 172.21.15.16:0/2703888154 learned_addr learned my addr 172.21.15.16:0/2703888154 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:03:42.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.841+0000 7f02ee800640 1 --2- 172.21.15.16:0/2703888154 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f02e4005ef0 0x7f02e4006320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:42.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.845+0000 7f02ee800640 1 -- 172.21.15.16:0/2703888154 >> v2:172.21.15.89:3300/0 conn(0x7f02e4033e40 msgr2=0x7f02e40eb180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:03:42.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.845+0000 7f02ee800640 1 --2- 172.21.15.16:0/2703888154 >> v2:172.21.15.89:3300/0 conn(0x7f02e4033e40 0x7f02e40eb180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:42.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.845+0000 7f02ee800640 1 -- 172.21.15.16:0/2703888154 >> v1:172.21.15.89:6789/0 conn(0x7f02e4047c20 legacy=0x7f02e40eb6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:42.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.845+0000 7f02ee800640 1 -- 172.21.15.16:0/2703888154 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02e404fa30 con 0x7f02e4005ef0 2024-03-31T15:03:42.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.845+0000 7f02eca00640 1 -- 172.21.15.16:0/2703888154 <== mon.1 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3797325571 0 0) 0x7f02e4034220 con 0x7f02e4047c20 2024-03-31T15:03:42.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.845+0000 7f1c97600640 1 Processor -- start 2024-03-31T15:03:42.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.849+0000 7f1c97600640 1 -- start start 2024-03-31T15:03:42.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.849+0000 7f02ee800640 1 --2- 172.21.15.16:0/2703888154 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f02e4005ef0 0x7f02e4006320 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f02e806af70 tx=0x7f02e809a4c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=48d95fe8caeb0407 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:03:42.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.849+0000 7f02eca00640 1 -- 172.21.15.16:0/2703888154 <== mon.2 v2:172.21.15.89:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02e812a640 con 0x7f02e4005ef0 2024-03-31T15:03:42.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.849+0000 7f02eca00640 1 -- 172.21.15.16:0/2703888154 <== mon.2 v2:172.21.15.89:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f02e812a820 con 0x7f02e4005ef0 2024-03-31T15:03:42.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.849+0000 7f02eca00640 1 -- 172.21.15.16:0/2703888154 <== mon.2 v2:172.21.15.89:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02e812ac50 con 0x7f02e4005ef0 2024-03-31T15:03:42.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.849+0000 7f1c97600640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f1c900b3030 0x7f1c90072f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:42.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.849+0000 7f1c97600640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f1c900b5130 0x7f1c900cc790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:42.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.849+0000 7f1c97600640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1c900cf750 con 0x7f1c90165ea0 2024-03-31T15:03:42.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.849+0000 7f1c97600640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f1c900734d0 con 0x7f1c900b3030 2024-03-31T15:03:42.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.849+0000 7f1c97600640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f1c901671d0 con 0x7f1c900b5130 2024-03-31T15:03:42.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.849+0000 7f1c94c00640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f1c90165ea0 0x7f1c900b4a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.16:49192/0 (socket says 172.21.15.16:49192) 2024-03-31T15:03:42.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.849+0000 7f1c94c00640 1 -- 172.21.15.16:0/2949760155 learned_addr learned my addr 172.21.15.16:0/2949760155 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:03:42.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.849+0000 7f1c8fe00640 1 --2- 172.21.15.16:0/2949760155 >> v2:172.21.15.16:3300/0 conn(0x7f1c900b3030 0x7f1c90072f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:42.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.849+0000 7f1c8f400640 1 -- 172.21.15.16:0/2949760155 <== mon.1 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2364060739 0 0) 0x7f1c900cf750 con 0x7f1c90165ea0 2024-03-31T15:03:42.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.849+0000 7f1c8f400640 1 -- 172.21.15.16:0/2949760155 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1c78003530 con 0x7f1c90165ea0 2024-03-31T15:03:42.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.849+0000 7f1c95600640 1 --2- 172.21.15.16:0/2949760155 >> v2:172.21.15.89:3300/0 conn(0x7f1c900b5130 0x7f1c900cc790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:42.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.849+0000 7f02ef200640 1 -- 172.21.15.16:0/2703888154 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f02e4005ef0 msgr2=0x7f02e4006320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:42.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.849+0000 7f02ef200640 1 --2- 172.21.15.16:0/2703888154 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f02e4005ef0 0x7f02e4006320 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f02e806af70 tx=0x7f02e809a4c0 comp rx=0 tx=0).stop 2024-03-31T15:03:42.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.849+0000 7f02ef200640 1 -- 172.21.15.16:0/2703888154 shutdown_connections 2024-03-31T15:03:42.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.849+0000 7f1c8fe00640 1 -- 172.21.15.16:0/2949760155 >> v2:172.21.15.89:3300/0 conn(0x7f1c900b5130 msgr2=0x7f1c900cc790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:03:42.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.849+0000 7f1c8fe00640 1 --2- 172.21.15.16:0/2949760155 >> v2:172.21.15.89:3300/0 conn(0x7f1c900b5130 0x7f1c900cc790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:42.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.849+0000 7f1c8fe00640 1 -- 172.21.15.16:0/2949760155 >> v1:172.21.15.89:6789/0 conn(0x7f1c90165ea0 legacy=0x7f1c900b4a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:42.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.853+0000 7f1c8f400640 1 -- 172.21.15.16:0/2949760155 <== mon.1 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4289682114 0 0) 0x7f1c78003530 con 0x7f1c90165ea0 2024-03-31T15:03:42.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.853+0000 7f1c8fe00640 1 -- 172.21.15.16:0/2949760155 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c900cf9f0 con 0x7f1c900b3030 2024-03-31T15:03:42.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.853+0000 7f02ef200640 1 --2- 172.21.15.16:0/2703888154 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f02e4005ef0 0x7f02e4006320 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:42.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.853+0000 7f02ef200640 1 --2- 172.21.15.16:0/2703888154 >> v2:172.21.15.89:3300/0 conn(0x7f02e4033e40 0x7f02e40eb180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:42.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.853+0000 7f02ef200640 1 -- 172.21.15.16:0/2703888154 >> 172.21.15.16:0/2703888154 conn(0x7f02e4009fc0 msgr2=0x7f02e400a3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:03:42.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.853+0000 7f4badc00640 1 Processor -- start 2024-03-31T15:03:42.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.853+0000 7f4badc00640 1 -- start start 2024-03-31T15:03:42.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.853+0000 7f02ef200640 1 -- 172.21.15.16:0/2703888154 shutdown_connections 2024-03-31T15:03:42.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.853+0000 7f02ef200640 1 -- 172.21.15.16:0/2703888154 wait complete. 2024-03-31T15:03:42.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.853+0000 7efc10800640 1 Processor -- start 2024-03-31T15:03:42.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.853+0000 7efc10800640 1 -- start start 2024-03-31T15:03:42.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.853+0000 7f4badc00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f4ba80d2720 0x7f4ba806c550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:42.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.853+0000 7fcfc1400640 1 Processor -- start 2024-03-31T15:03:42.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.853+0000 7f4ba6000640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f4ba80b39b0 0x7f4ba80d2010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:33968/0 (socket says 172.21.15.16:33968) 2024-03-31T15:03:42.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.853+0000 7f4badc00640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4ba80d3d00 con 0x7f4ba80b39b0 2024-03-31T15:03:42.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.853+0000 7f4badc00640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4ba80d3a60 con 0x7f4ba80b35d0 2024-03-31T15:03:42.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.853+0000 7f4badc00640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f4ba80b7980 con 0x7f4ba80d2720 2024-03-31T15:03:42.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.853+0000 7f4ba7400640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f4ba80d2720 0x7f4ba806c550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:42.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.853+0000 7f4ba7400640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f4ba80d2720 0x7f4ba806c550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.16:35940/0 (socket says 172.21.15.16:35940) 2024-03-31T15:03:42.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.853+0000 7f4ba7400640 1 -- 172.21.15.16:0/2738272882 learned_addr learned my addr 172.21.15.16:0/2738272882 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:03:42.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.857+0000 7f02ef200640 1 Processor -- start 2024-03-31T15:03:42.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.857+0000 7f02ef200640 1 -- start start 2024-03-31T15:03:42.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.857+0000 7f1c8fe00640 1 --2- 172.21.15.16:0/2949760155 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c900b3030 0x7f1c90072f90 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f1c9006f990 tx=0x7f1c8401c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=845f1322ea49897 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:03:42.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.857+0000 7efc10800640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7efc080b35d0 0x7efc080b7e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:42.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.857+0000 7efc10800640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7efc080d2720 0x7efc0806c550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:42.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.857+0000 7efc10800640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efc080d3d00 con 0x7efc080b39b0 2024-03-31T15:03:42.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.857+0000 7efc10800640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7efc080b7980 con 0x7efc080b35d0 2024-03-31T15:03:42.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.857+0000 7efc10800640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7efc080d3a60 con 0x7efc080d2720 2024-03-31T15:03:42.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.853+0000 7fcfc1400640 1 -- start start 2024-03-31T15:03:42.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.857+0000 7efc0e800640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7efc080d2720 0x7efc0806c550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:42.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.857+0000 7efc0d400640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7efc080b39b0 0x7efc080d2010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.16:49212/0 (socket says 172.21.15.16:49212) 2024-03-31T15:03:42.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.857+0000 7efc0d400640 1 -- 172.21.15.16:0/169409895 learned_addr learned my addr 172.21.15.16:0/169409895 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:03:42.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.857+0000 7efc0de00640 1 --2- 172.21.15.16:0/169409895 >> v2:172.21.15.89:3300/0 conn(0x7efc080b35d0 0x7efc080b7e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:42.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.857+0000 7fcfc1400640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fcfbc165dc0 0x7fcfbc0ccfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:42.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.857+0000 7fcfc1400640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fcfbc0b48d0 0x7fcfbc0cc130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:42.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.857+0000 7fcfc1400640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fcfbc0b4d30 0x7fcfbc0cc670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:42.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.857+0000 7fcfc1400640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7fcfbc071500 con 0x7fcfbc0b48d0 2024-03-31T15:03:42.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.857+0000 7fcfc1400640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7fcfbc073570 con 0x7fcfbc165dc0 2024-03-31T15:03:42.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.857+0000 7fcfc1400640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fcfbc072f40 con 0x7fcfbc0b4d30 2024-03-31T15:03:42.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.857+0000 7f4ba7400640 1 -- 172.21.15.16:0/2738272882 >> v1:172.21.15.89:6789/0 conn(0x7f4ba80b35d0 legacy=0x7f4ba80b7e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:42.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.857+0000 7f4ba7400640 1 -- 172.21.15.16:0/2738272882 >> v1:172.21.15.16:6789/0 conn(0x7f4ba80b39b0 legacy=0x7f4ba80d2010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:42.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.857+0000 7f4ba7400640 1 -- 172.21.15.16:0/2738272882 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ba80b7330 con 0x7f4ba80d2720 2024-03-31T15:03:42.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.857+0000 7fcfba800640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fcfbc165dc0 0x7fcfbc0ccfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:42.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.857+0000 7fcfba800640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fcfbc165dc0 0x7fcfbc0ccfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.16:35968/0 (socket says 172.21.15.16:35968) 2024-03-31T15:03:42.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.857+0000 7f02ef200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f02e4005ef0 0x7f02e40e49c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:42.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.857+0000 7f1c8f400640 1 -- 172.21.15.16:0/2949760155 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c840b0070 con 0x7f1c900b3030 2024-03-31T15:03:42.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.857+0000 7f1c8f400640 1 -- 172.21.15.16:0/2949760155 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1c840ab400 con 0x7f1c900b3030 2024-03-31T15:03:42.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.857+0000 7f1c8f400640 1 -- 172.21.15.16:0/2949760155 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c7c0016a0 con 0x7f1c900b3030 2024-03-31T15:03:42.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.857+0000 7efc0e800640 1 -- 172.21.15.16:0/169409895 >> v2:172.21.15.89:3300/0 conn(0x7efc080b35d0 msgr2=0x7efc080b7e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:03:42.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.857+0000 7efc0e800640 1 --2- 172.21.15.16:0/169409895 >> v2:172.21.15.89:3300/0 conn(0x7efc080b35d0 0x7efc080b7e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:42.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.857+0000 7efc0e800640 1 -- 172.21.15.16:0/169409895 >> v1:172.21.15.89:6789/0 conn(0x7efc080b39b0 legacy=0x7efc080d2010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:42.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.857+0000 7efc0ca00640 1 -- 172.21.15.16:0/169409895 <== mon.1 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2009048713 0 0) 0x7efc080d3d00 con 0x7efc080b39b0 2024-03-31T15:03:42.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.857+0000 7efc0e800640 1 -- 172.21.15.16:0/169409895 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc080b7330 con 0x7efc080d2720 2024-03-31T15:03:42.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.857+0000 7fcfb9e00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fcfbc0b48d0 0x7fcfbc0cc130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:42.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.857+0000 7fcfbb200640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fcfbc0b4d30 0x7fcfbc0cc670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:42.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.857+0000 7fcfba800640 1 -- 172.21.15.16:0/3461248746 learned_addr learned my addr 172.21.15.16:0/3461248746 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:03:42.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.857+0000 7fcfba800640 1 -- 172.21.15.16:0/3461248746 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fcfbc0b4d30 msgr2=0x7fcfbc0cc670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:03:42.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.857+0000 7fcfba800640 1 --2- 172.21.15.16:0/3461248746 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fcfbc0b4d30 0x7fcfbc0cc670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:42.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.857+0000 7fcfba800640 1 -- 172.21.15.16:0/3461248746 >> v2:172.21.15.16:3300/0 conn(0x7fcfbc0b48d0 msgr2=0x7fcfbc0cc130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:03:42.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.857+0000 7efc0de00640 1 --2- 172.21.15.16:0/169409895 >> v2:172.21.15.89:3300/0 conn(0x7efc080b35d0 0x7efc080b7e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:03:42.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.857+0000 7f02ef200640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f02e4033e40 0x7f02e404be40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:42.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.857+0000 7f02ef200640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f02e4047c20 0x7f02e404c380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:42.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.857+0000 7f02ed400640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f02e4033e40 0x7f02e404be40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:42.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.861+0000 7f02ed400640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f02e4033e40 0x7f02e404be40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.16:35970/0 (socket says 172.21.15.16:35970) 2024-03-31T15:03:42.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.861+0000 7f02ed400640 1 -- 172.21.15.16:0/2730080888 learned_addr learned my addr 172.21.15.16:0/2730080888 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:03:42.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.861+0000 7f02ee800640 1 --2- 172.21.15.16:0/2730080888 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f02e4047c20 0x7f02e404c380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:42.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.861+0000 7f02ef200640 1 -- 172.21.15.16:0/2730080888 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f02e404fd80 con 0x7f02e4005ef0 2024-03-31T15:03:42.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.861+0000 7f02ede00640 1 --2- 172.21.15.16:0/2730080888 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f02e4005ef0 0x7f02e40e49c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:42.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.861+0000 7f02ef200640 1 -- 172.21.15.16:0/2730080888 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f02e404f090 con 0x7f02e4033e40 2024-03-31T15:03:42.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.861+0000 7f02ef200640 1 -- 172.21.15.16:0/2730080888 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f02e404f5c0 con 0x7f02e4047c20 2024-03-31T15:03:42.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.861+0000 7f02ee800640 1 -- 172.21.15.16:0/2730080888 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f02e4033e40 msgr2=0x7f02e404be40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:42.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.861+0000 7f02ee800640 1 --2- 172.21.15.16:0/2730080888 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f02e4033e40 0x7f02e404be40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:42.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.861+0000 7f02ee800640 1 -- 172.21.15.16:0/2730080888 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f02e4005ef0 msgr2=0x7f02e40e49c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:42.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.861+0000 7f02ee800640 1 --2- 172.21.15.16:0/2730080888 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f02e4005ef0 0x7f02e40e49c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:42.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.861+0000 7f02ee800640 1 -- 172.21.15.16:0/2730080888 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02e40e4f00 con 0x7f02e4047c20 2024-03-31T15:03:42.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.861+0000 7f02ede00640 1 --2- 172.21.15.16:0/2730080888 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f02e4005ef0 0x7f02e40e49c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:03:42.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.861+0000 7f02ee800640 1 --2- 172.21.15.16:0/2730080888 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f02e4047c20 0x7f02e404c380 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f02e8050bd0 tx=0x7f02e8065390 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:03:42.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.861+0000 7f1c97600640 1 -- 172.21.15.16:0/2949760155 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c900b3030 msgr2=0x7f1c90072f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:42.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.861+0000 7f1c97600640 1 --2- 172.21.15.16:0/2949760155 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c900b3030 0x7f1c90072f90 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f1c9006f990 tx=0x7f1c8401c3a0 comp rx=0 tx=0).stop 2024-03-31T15:03:42.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.861+0000 7f1c97600640 1 -- 172.21.15.16:0/2949760155 shutdown_connections 2024-03-31T15:03:42.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.861+0000 7f1c97600640 1 --2- 172.21.15.16:0/2949760155 >> v2:172.21.15.89:3300/0 conn(0x7f1c900b5130 0x7f1c900cc790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:42.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.861+0000 7f1c97600640 1 --2- 172.21.15.16:0/2949760155 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c900b3030 0x7f1c90072f90 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:42.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.861+0000 7f1c97600640 1 -- 172.21.15.16:0/2949760155 >> 172.21.15.16:0/2949760155 conn(0x7f1c900b1d30 msgr2=0x7f1c900b2140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:03:42.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.861+0000 7f1c97600640 1 -- 172.21.15.16:0/2949760155 shutdown_connections 2024-03-31T15:03:42.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.861+0000 7f1c97600640 1 -- 172.21.15.16:0/2949760155 wait complete. 2024-03-31T15:03:42.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.857+0000 7f5ae8000640 1 Processor -- start 2024-03-31T15:03:42.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.861+0000 7f5ae8000640 1 -- start start 2024-03-31T15:03:42.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.861+0000 7f02e2000640 1 -- 172.21.15.16:0/2730080888 <== mon.2 v2:172.21.15.89:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02e8133670 con 0x7f02e4047c20 2024-03-31T15:03:42.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.861+0000 7f02e2000640 1 -- 172.21.15.16:0/2730080888 <== mon.2 v2:172.21.15.89:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f02e8133810 con 0x7f02e4047c20 2024-03-31T15:03:42.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.861+0000 7f02e2000640 1 -- 172.21.15.16:0/2730080888 <== mon.2 v2:172.21.15.89:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02e8133bc0 con 0x7f02e4047c20 2024-03-31T15:03:42.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.861+0000 7f1c97600640 1 Processor -- start 2024-03-31T15:03:42.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.861+0000 7f1c97600640 1 -- start start 2024-03-31T15:03:42.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.857+0000 7fcfba800640 1 --2- 172.21.15.16:0/3461248746 >> v2:172.21.15.16:3300/0 conn(0x7fcfbc0b48d0 0x7fcfbc0cc130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:42.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.857+0000 7fcfba800640 1 -- 172.21.15.16:0/3461248746 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcfbc0b51e0 con 0x7fcfbc165dc0 2024-03-31T15:03:42.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.861+0000 7fcfb9e00640 1 --2- 172.21.15.16:0/3461248746 >> v2:172.21.15.16:3300/0 conn(0x7fcfbc0b48d0 0x7fcfbc0cc130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:03:42.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.861+0000 7f4ba7400640 1 --2- 172.21.15.16:0/2738272882 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4ba80d2720 0x7f4ba806c550 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f4ba80b7b00 tx=0x7f4b9801ed30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4b32e629c87f183a server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:03:42.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.861+0000 7f4ba5600640 1 -- 172.21.15.16:0/2738272882 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b980b5070 con 0x7f4ba80d2720 2024-03-31T15:03:42.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.861+0000 7f4ba5600640 1 -- 172.21.15.16:0/2738272882 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4b980a8a30 con 0x7f4ba80d2720 2024-03-31T15:03:42.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.861+0000 7f4ba5600640 1 -- 172.21.15.16:0/2738272882 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b980a8e00 con 0x7f4ba80d2720 2024-03-31T15:03:42.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.861+0000 7f5ae4c00640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f5ae00cf6a0 0x7f5ae0071500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:33972/0 (socket says 172.21.15.16:33972) 2024-03-31T15:03:42.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.861+0000 7f5ae8000640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f5ae00b3030 0x7f5ae00b34c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:42.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.861+0000 7f5ae5600640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f5ae015a660 0x7f5ae00ce860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.16:49220/0 (socket says 172.21.15.16:49220) 2024-03-31T15:03:42.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.865+0000 7f02ef200640 1 -- 172.21.15.16:0/2730080888 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f02e4035180 con 0x7f02e4047c20 2024-03-31T15:03:42.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.865+0000 7f02ef200640 1 -- 172.21.15.16:0/2730080888 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f02e4035740 con 0x7f02e4047c20 2024-03-31T15:03:42.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.865+0000 7f1c97600640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f1c900b3030 0x7f1c90157240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:42.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.865+0000 7f1c97600640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c900b5130 0x7f1c9015c380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:42.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.865+0000 7f1c97600640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1c90165ea0 0x7f1c9015b7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:42.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.865+0000 7f1c97600640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f1c900cd8b0 con 0x7f1c900b5130 2024-03-31T15:03:42.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.865+0000 7f1c97600640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f1c900cd730 con 0x7f1c90165ea0 2024-03-31T15:03:42.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.865+0000 7f1c97600640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f1c9015aeb0 con 0x7f1c900b3030 2024-03-31T15:03:42.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.865+0000 7f1c94c00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f1c900b3030 0x7f1c90157240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:42.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.865+0000 7f1c94c00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f1c900b3030 0x7f1c90157240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3301/0 says I am v2:172.21.15.16:41486/0 (socket says 172.21.15.16:41486) 2024-03-31T15:03:42.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.865+0000 7f1c94c00640 1 -- 172.21.15.16:0/1585641036 learned_addr learned my addr 172.21.15.16:0/1585641036 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:03:42.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.865+0000 7f1c8fe00640 1 --2- 172.21.15.16:0/1585641036 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c900b5130 0x7f1c9015c380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:42.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.865+0000 7efc0e800640 1 --2- 172.21.15.16:0/169409895 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7efc080d2720 0x7efc0806c550 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7efc04009d90 tx=0x7efc0401f150 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c069bc5eb9f824ac server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:03:42.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.861+0000 7fb4fb600640 1 Processor -- start 2024-03-31T15:03:42.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.861+0000 7fb4fb600640 1 -- start start 2024-03-31T15:03:42.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.865+0000 7fb4fb600640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fb4f415a680 0x7fb4f40b5040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:42.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.865+0000 7fb4fb600640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb4f40af730 0x7fb4f40cdec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:42.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.861+0000 7ff570c00640 1 Processor -- start 2024-03-31T15:03:42.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.861+0000 7f5ae4c00640 1 -- 172.21.15.16:0/3226055332 learned_addr learned my addr 172.21.15.16:0/3226055332 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:03:42.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.865+0000 7f5ae6000640 1 --2- 172.21.15.16:0/3226055332 >> v2:172.21.15.16:3300/0 conn(0x7f5ae00b3030 0x7f5ae00b34c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:42.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.865+0000 7f5ae8000640 1 -- 172.21.15.16:0/3226055332 --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5ae0073a10 con 0x7f5ae00cf6a0 2024-03-31T15:03:42.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.865+0000 7f5ae8000640 1 -- 172.21.15.16:0/3226055332 --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5ae00b5110 con 0x7f5ae015a660 2024-03-31T15:03:42.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.865+0000 7f5ae8000640 1 -- 172.21.15.16:0/3226055332 --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f5ae015aa40 con 0x7f5ae00b3030 2024-03-31T15:03:42.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.865+0000 7f5ae6000640 1 -- 172.21.15.16:0/3226055332 >> v1:172.21.15.89:6789/0 conn(0x7f5ae015a660 legacy=0x7f5ae00ce860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:42.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.865+0000 7f5ae6000640 1 -- 172.21.15.16:0/3226055332 >> v1:172.21.15.16:6789/0 conn(0x7f5ae00cf6a0 legacy=0x7f5ae0071500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:42.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.865+0000 7f5ae6000640 1 -- 172.21.15.16:0/3226055332 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ae00b53f0 con 0x7f5ae00b3030 2024-03-31T15:03:42.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.865+0000 7fcfba800640 1 --2- 172.21.15.16:0/3461248746 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcfbc165dc0 0x7fcfbc0ccfc0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fcfbc0d1060 tx=0x7fcfa801c480 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e04d1c70cbdb823b server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:03:42.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.865+0000 7f4badc00640 1 -- 172.21.15.16:0/2738272882 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4ba80d2720 msgr2=0x7f4ba806c550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:42.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.865+0000 7f4badc00640 1 --2- 172.21.15.16:0/2738272882 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4ba80d2720 0x7f4ba806c550 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f4ba80b7b00 tx=0x7f4b9801ed30 comp rx=0 tx=0).stop 2024-03-31T15:03:42.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.865+0000 7fb4f3e00640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fb4f4068d60 0x7fb4f415e6d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:33976/0 (socket says 172.21.15.16:33976) 2024-03-31T15:03:42.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.865+0000 7fb4f8c00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fb4f415a680 0x7fb4f40b5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:42.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.865+0000 7f1c94c00640 1 -- 172.21.15.16:0/1585641036 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1c90165ea0 msgr2=0x7f1c9015b7e0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-31T15:03:42.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.865+0000 7fb4f8c00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fb4f415a680 0x7fb4f40b5040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.16:35984/0 (socket says 172.21.15.16:35984) 2024-03-31T15:03:42.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.865+0000 7f1c94c00640 1 --2- 172.21.15.16:0/1585641036 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1c90165ea0 0x7f1c9015b7e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:42.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.865+0000 7f1c94c00640 1 -- 172.21.15.16:0/1585641036 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c900b5130 msgr2=0x7f1c9015c380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:42.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.865+0000 7f1c94c00640 1 --2- 172.21.15.16:0/1585641036 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c900b5130 0x7f1c9015c380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:42.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.865+0000 7f1c94c00640 1 -- 172.21.15.16:0/1585641036 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c9015a610 con 0x7f1c900b3030 2024-03-31T15:03:42.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.865+0000 7efc0ca00640 1 -- 172.21.15.16:0/169409895 <== mon.2 v2:172.21.15.89:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc040b4070 con 0x7efc080d2720 2024-03-31T15:03:42.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.865+0000 7efc0ca00640 1 -- 172.21.15.16:0/169409895 <== mon.2 v2:172.21.15.89:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efc040af600 con 0x7efc080d2720 2024-03-31T15:03:42.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.865+0000 7efc0ca00640 1 -- 172.21.15.16:0/169409895 <== mon.2 v2:172.21.15.89:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc040af9d0 con 0x7efc080d2720 2024-03-31T15:03:42.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.865+0000 7fcfb9400640 1 -- 172.21.15.16:0/3461248746 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfa80b1070 con 0x7fcfbc165dc0 2024-03-31T15:03:42.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.865+0000 7fcfb9400640 1 -- 172.21.15.16:0/3461248746 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcfa80a8050 con 0x7fcfbc165dc0 2024-03-31T15:03:42.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.865+0000 7fcfb9400640 1 -- 172.21.15.16:0/3461248746 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfa80b1070 con 0x7fcfbc165dc0 2024-03-31T15:03:42.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.865+0000 7fb4f9600640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb4f40af730 0x7fb4f40cdec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:42.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.865+0000 7fb4f9600640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb4f40af730 0x7fb4f40cdec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3301/0 says I am v2:172.21.15.16:41496/0 (socket says 172.21.15.16:41496) 2024-03-31T15:03:42.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.865+0000 7fb4f3e00640 1 -- 172.21.15.16:0/3714067237 learned_addr learned my addr 172.21.15.16:0/3714067237 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:03:42.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.865+0000 7ff570c00640 1 -- start start 2024-03-31T15:03:42.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.865+0000 7ff570c00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff56c15a660 0x7ff56c0ce860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:42.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.865+0000 7ff570c00640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff56c073a10 con 0x7ff56c0cf6a0 2024-03-31T15:03:42.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.865+0000 7ff570c00640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff56c0b5110 con 0x7ff56c0b3030 2024-03-31T15:03:42.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.865+0000 7ff570c00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7ff56c15aa40 con 0x7ff56c15a660 2024-03-31T15:03:42.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.865+0000 7f1c94c00640 1 --2- 172.21.15.16:0/1585641036 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f1c900b3030 0x7f1c90157240 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f1c7c00c520 tx=0x7f1c7c00c9e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:03:42.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.865+0000 7ff569e00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff56c15a660 0x7ff56c0ce860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:42.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.865+0000 7ff569e00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff56c15a660 0x7ff56c0ce860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3301/0 says I am v2:172.21.15.16:41510/0 (socket says 172.21.15.16:41510) 2024-03-31T15:03:42.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.865+0000 7ff569e00640 1 -- 172.21.15.16:0/2692909161 learned_addr learned my addr 172.21.15.16:0/2692909161 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:03:42.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.865+0000 7fb4fb600640 1 -- 172.21.15.16:0/3714067237 --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb4f40d0410 con 0x7fb4f4068d60 2024-03-31T15:03:42.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.865+0000 7f1c8cc00640 1 -- 172.21.15.16:0/1585641036 <== mon.2 v2:172.21.15.89:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c7c018070 con 0x7f1c900b3030 2024-03-31T15:03:42.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.865+0000 7f1c8cc00640 1 -- 172.21.15.16:0/1585641036 <== mon.2 v2:172.21.15.89:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1c7c007500 con 0x7f1c900b3030 2024-03-31T15:03:42.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.865+0000 7f1c8cc00640 1 -- 172.21.15.16:0/1585641036 <== mon.2 v2:172.21.15.89:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c7c004a30 con 0x7f1c900b3030 2024-03-31T15:03:42.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.865+0000 7f4badc00640 1 -- 172.21.15.16:0/2738272882 shutdown_connections 2024-03-31T15:03:42.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.865+0000 7f4badc00640 1 --2- 172.21.15.16:0/2738272882 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4ba80d2720 0x7f4ba806c550 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:42.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.865+0000 7f4badc00640 1 -- 172.21.15.16:0/2738272882 >> 172.21.15.16:0/2738272882 conn(0x7f4ba80d1020 msgr2=0x7f4ba80d1430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:03:42.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.865+0000 7fb4fb600640 1 -- 172.21.15.16:0/3714067237 --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7fb4f40b3550 con 0x7fb4f415a680 2024-03-31T15:03:42.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.865+0000 7fb4fb600640 1 -- 172.21.15.16:0/3714067237 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fb4f40b3830 con 0x7fb4f40af730 2024-03-31T15:03:42.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.865+0000 7efc10800640 1 -- 172.21.15.16:0/169409895 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7efc080d2720 msgr2=0x7efc0806c550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:42.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.865+0000 7efc10800640 1 --2- 172.21.15.16:0/169409895 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7efc080d2720 0x7efc0806c550 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7efc04009d90 tx=0x7efc0401f150 comp rx=0 tx=0).stop 2024-03-31T15:03:42.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.865+0000 7efc10800640 1 -- 172.21.15.16:0/169409895 shutdown_connections 2024-03-31T15:03:42.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.865+0000 7efc10800640 1 --2- 172.21.15.16:0/169409895 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7efc080d2720 0x7efc0806c550 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:42.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.865+0000 7efc10800640 1 --2- 172.21.15.16:0/169409895 >> v2:172.21.15.89:3300/0 conn(0x7efc080b35d0 0x7efc080b7e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:42.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.865+0000 7efc10800640 1 -- 172.21.15.16:0/169409895 >> 172.21.15.16:0/169409895 conn(0x7efc080d1020 msgr2=0x7efc080d1430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:03:42.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.865+0000 7f1c97600640 1 -- 172.21.15.16:0/1585641036 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1c9015a8f0 con 0x7f1c900b3030 2024-03-31T15:03:42.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.865+0000 7fb4f3400640 1 -- 172.21.15.16:0/3714067237 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 27459833 0 0) 0x7fb4f40d0410 con 0x7fb4f4068d60 2024-03-31T15:03:42.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.865+0000 7f02ef200640 1 -- 172.21.15.16:0/2730080888 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f02e4034f30 con 0x7f02e4047c20 2024-03-31T15:03:42.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.869+0000 7f1c97600640 1 -- 172.21.15.16:0/1585641036 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f1c9015a200 con 0x7f1c900b3030 2024-03-31T15:03:42.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.869+0000 7fcfc1400640 1 -- 172.21.15.16:0/3461248746 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcfbc165dc0 msgr2=0x7fcfbc0ccfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:42.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.869+0000 7fcfc1400640 1 --2- 172.21.15.16:0/3461248746 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcfbc165dc0 0x7fcfbc0ccfc0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fcfbc0d1060 tx=0x7fcfa801c480 comp rx=0 tx=0).stop 2024-03-31T15:03:42.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.869+0000 7f4badc00640 1 -- 172.21.15.16:0/2738272882 shutdown_connections 2024-03-31T15:03:42.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.869+0000 7efc10800640 1 -- 172.21.15.16:0/169409895 shutdown_connections 2024-03-31T15:03:42.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.869+0000 7efc10800640 1 -- 172.21.15.16:0/169409895 wait complete. 2024-03-31T15:03:42.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.869+0000 7fb4f3400640 1 -- 172.21.15.16:0/3714067237 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb4d8003530 con 0x7fb4f4068d60 2024-03-31T15:03:42.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.869+0000 7fb4f9600640 1 -- 172.21.15.16:0/3714067237 >> v2:172.21.15.89:3300/0 conn(0x7fb4f415a680 msgr2=0x7fb4f40b5040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:03:42.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.869+0000 7fb4f9600640 1 --2- 172.21.15.16:0/3714067237 >> v2:172.21.15.89:3300/0 conn(0x7fb4f415a680 0x7fb4f40b5040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:42.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.869+0000 7fb4f9600640 1 -- 172.21.15.16:0/3714067237 >> v1:172.21.15.16:6789/0 conn(0x7fb4f4068d60 legacy=0x7fb4f415e6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:42.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.869+0000 7f1c97600640 1 -- 172.21.15.16:0/1585641036 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f1c6c001040 con 0x7f1c900b3030 2024-03-31T15:03:42.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.869+0000 7fb4f9600640 1 -- 172.21.15.16:0/3714067237 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb4f415e1b0 con 0x7fb4f40af730 2024-03-31T15:03:42.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.869+0000 7ff568a00640 1 -- 172.21.15.16:0/2692909161 <== mon.1 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3911822102 0 0) 0x7ff56c0b5110 con 0x7ff56c0b3030 2024-03-31T15:03:42.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.869+0000 7ff568a00640 1 -- 172.21.15.16:0/2692909161 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff550003530 con 0x7ff56c0b3030 2024-03-31T15:03:42.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.869+0000 7f4badc00640 1 -- 172.21.15.16:0/2738272882 wait complete. 2024-03-31T15:03:42.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.869+0000 7fb4f8c00640 1 --2- 172.21.15.16:0/3714067237 >> v2:172.21.15.89:3300/0 conn(0x7fb4f415a680 0x7fb4f40b5040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:03:42.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.869+0000 7ff569e00640 1 -- 172.21.15.16:0/2692909161 >> v1:172.21.15.89:6789/0 conn(0x7ff56c0b3030 legacy=0x7ff56c06e910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:42.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.869+0000 7ff569e00640 1 -- 172.21.15.16:0/2692909161 >> v1:172.21.15.16:6789/0 conn(0x7ff56c0cf6a0 legacy=0x7ff56c071500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:42.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.869+0000 7ff569e00640 1 -- 172.21.15.16:0/2692909161 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff56c0b53f0 con 0x7ff56c15a660 2024-03-31T15:03:42.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.869+0000 7f02e2000640 1 -- 172.21.15.16:0/2730080888 <== mon.2 v2:172.21.15.89:3301/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f02e813c030 con 0x7f02e4047c20 2024-03-31T15:03:42.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.869+0000 7f02e2000640 1 --2- 172.21.15.16:0/2730080888 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f02cc068480 0x7f02cc06a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:42.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.869+0000 7f02e2000640 1 -- 172.21.15.16:0/2730080888 <== mon.2 v2:172.21.15.89:3301/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f02e819f000 con 0x7f02e4047c20 2024-03-31T15:03:42.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.869+0000 7f02e2000640 1 --2- 172.21.15.16:0/2730080888 >> [v2:172.21.15.16:6800/932712244,v1:172.21.15.16:6801/932712244] conn(0x7f02cc0706a0 0x7f02cc072ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:42.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.869+0000 7f02e2000640 1 -- 172.21.15.16:0/2730080888 --> [v2:172.21.15.16:6800/932712244,v1:172.21.15.16:6801/932712244] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f02e812dec0 con 0x7f02cc0706a0 2024-03-31T15:03:42.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.869+0000 7f02e2000640 1 -- 172.21.15.16:0/2730080888 <== mon.2 v2:172.21.15.89:3301/0 6 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f02e81a25f0 con 0x7f02e4047c20 2024-03-31T15:03:42.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.869+0000 7f02ede00640 1 --2- 172.21.15.16:0/2730080888 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f02cc068480 0x7f02cc06a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:42.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.869+0000 7f02ed400640 1 --2- 172.21.15.16:0/2730080888 >> [v2:172.21.15.16:6800/932712244,v1:172.21.15.16:6801/932712244] conn(0x7f02cc0706a0 0x7f02cc072ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:42.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.869+0000 7fcfc1400640 1 -- 172.21.15.16:0/3461248746 shutdown_connections 2024-03-31T15:03:42.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.869+0000 7f5ae6000640 1 --2- 172.21.15.16:0/3226055332 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5ae00b3030 0x7f5ae00b34c0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f5ad8009c80 tx=0x7f5ad801d7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f49e19d71cf04a33 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:03:42.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.869+0000 7fcfc1400640 1 --2- 172.21.15.16:0/3461248746 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fcfbc0b4d30 0x7fcfbc0cc670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:42.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.869+0000 7fcfc1400640 1 --2- 172.21.15.16:0/3461248746 >> v2:172.21.15.16:3300/0 conn(0x7fcfbc0b48d0 0x7fcfbc0cc130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:42.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.869+0000 7fcfc1400640 1 --2- 172.21.15.16:0/3461248746 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcfbc165dc0 0x7fcfbc0ccfc0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:42.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.869+0000 7fcfc1400640 1 -- 172.21.15.16:0/3461248746 >> 172.21.15.16:0/3461248746 conn(0x7fcfbc06b620 msgr2=0x7fcfbc06ba30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:03:42.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.869+0000 7f4badc00640 1 Processor -- start 2024-03-31T15:03:42.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.869+0000 7f5adfe00640 1 -- 172.21.15.16:0/3226055332 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ad80b3070 con 0x7f5ae00b3030 2024-03-31T15:03:42.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.869+0000 7fcfc1400640 1 -- 172.21.15.16:0/3461248746 shutdown_connections 2024-03-31T15:03:42.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.869+0000 7f1c8cc00640 1 -- 172.21.15.16:0/1585641036 <== mon.2 v2:172.21.15.89:3301/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f1c7c01b070 con 0x7f1c900b3030 2024-03-31T15:03:42.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.869+0000 7fcfc1400640 1 -- 172.21.15.16:0/3461248746 wait complete. 2024-03-31T15:03:42.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.869+0000 7f5adfe00640 1 -- 172.21.15.16:0/3226055332 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5ad80a6850 con 0x7f5ae00b3030 2024-03-31T15:03:42.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.869+0000 7efc10800640 1 Processor -- start 2024-03-31T15:03:42.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.873+0000 7f5adfe00640 1 -- 172.21.15.16:0/3226055332 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ad80b3070 con 0x7f5ae00b3030 2024-03-31T15:03:42.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.873+0000 7efc10800640 1 -- start start 2024-03-31T15:03:42.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.873+0000 7f4badc00640 1 -- start start 2024-03-31T15:03:42.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.873+0000 7f5ae8000640 1 -- 172.21.15.16:0/3226055332 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5ae00b3030 msgr2=0x7f5ae00b34c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:42.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.873+0000 7f5ae8000640 1 --2- 172.21.15.16:0/3226055332 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5ae00b3030 0x7f5ae00b34c0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f5ad8009c80 tx=0x7f5ad801d7a0 comp rx=0 tx=0).stop 2024-03-31T15:03:42.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.873+0000 7f5ae8000640 1 -- 172.21.15.16:0/3226055332 shutdown_connections 2024-03-31T15:03:42.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.873+0000 7f5ae8000640 1 --2- 172.21.15.16:0/3226055332 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5ae00b3030 0x7f5ae00b34c0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:42.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.873+0000 7f5ae8000640 1 -- 172.21.15.16:0/3226055332 >> 172.21.15.16:0/3226055332 conn(0x7f5ae00b1d30 msgr2=0x7f5ae00b2140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:03:42.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.873+0000 7f1c8cc00640 1 --2- 172.21.15.16:0/1585641036 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f1c700683d0 0x7f1c7006a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:42.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.873+0000 7f02ed400640 1 --2- 172.21.15.16:0/2730080888 >> [v2:172.21.15.16:6800/932712244,v1:172.21.15.16:6801/932712244] conn(0x7f02cc0706a0 0x7f02cc072ac0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:03:42.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.873+0000 7f1c8cc00640 1 -- 172.21.15.16:0/1585641036 <== mon.2 v2:172.21.15.89:3301/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f1c7c085ce0 con 0x7f1c900b3030 2024-03-31T15:03:42.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.873+0000 7f1c8fe00640 1 --2- 172.21.15.16:0/1585641036 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f1c700683d0 0x7f1c7006a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:42.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.873+0000 7f1c8cc00640 1 --2- 172.21.15.16:0/1585641036 >> [v2:172.21.15.16:6808/1398737533,v1:172.21.15.16:6810/1398737533] conn(0x7f1c70070650 0x7f1c70072a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:42.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.873+0000 7f1c8cc00640 1 -- 172.21.15.16:0/1585641036 --> [v2:172.21.15.16:6808/1398737533,v1:172.21.15.16:6810/1398737533] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f1c70073120 con 0x7f1c70070650 2024-03-31T15:03:42.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.873+0000 7f1c8cc00640 1 -- 172.21.15.16:0/1585641036 <== mon.2 v2:172.21.15.89:3301/0 6 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f1c7c088300 con 0x7f1c900b3030 2024-03-31T15:03:42.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.873+0000 7efc10800640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efc080b35d0 0x7efc081d7d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:42.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.873+0000 7efc10800640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efc080b39b0 0x7efc081e9970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:42.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.873+0000 7f5ae8000640 1 -- 172.21.15.16:0/3226055332 shutdown_connections 2024-03-31T15:03:42.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.873+0000 7efc10800640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7efc080d2720 0x7efc081ea840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:42.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.873+0000 7efc10800640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7efc080b6a90 con 0x7efc080b39b0 2024-03-31T15:03:42.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.873+0000 7efc10800640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7efc080b6910 con 0x7efc080b35d0 2024-03-31T15:03:42.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.873+0000 7efc10800640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7efc080d3ef0 con 0x7efc080d2720 2024-03-31T15:03:42.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.873+0000 7efc0de00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efc080b35d0 0x7efc081d7d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:42.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.873+0000 7efc0de00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efc080b35d0 0x7efc081d7d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.16:35998/0 (socket says 172.21.15.16:35998) 2024-03-31T15:03:42.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.873+0000 7efc0de00640 1 -- 172.21.15.16:0/3781439136 learned_addr learned my addr 172.21.15.16:0/3781439136 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:03:42.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.873+0000 7efc0d400640 1 --2- 172.21.15.16:0/3781439136 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efc080b39b0 0x7efc081e9970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:42.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.873+0000 7f5ae8000640 1 -- 172.21.15.16:0/3226055332 wait complete. 2024-03-31T15:03:42.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.873+0000 7f02ede00640 1 --2- 172.21.15.16:0/2730080888 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f02cc068480 0x7f02cc06a940 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f02dc009ae0 tx=0x7f02dc008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:03:42.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.873+0000 7f1c95600640 1 --2- 172.21.15.16:0/1585641036 >> [v2:172.21.15.16:6808/1398737533,v1:172.21.15.16:6810/1398737533] conn(0x7f1c70070650 0x7f1c70072a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:42.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.873+0000 7efc0e800640 1 --2- 172.21.15.16:0/3781439136 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7efc080d2720 0x7efc081ea840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:42.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.873+0000 7efc0de00640 1 -- 172.21.15.16:0/3781439136 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7efc080d2720 msgr2=0x7efc081ea840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:42.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.873+0000 7fcfc1400640 1 Processor -- start 2024-03-31T15:03:42.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.873+0000 7fb4f9600640 1 --2- 172.21.15.16:0/3714067237 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb4f40af730 0x7fb4f40cdec0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fb4e00097c0 tx=0x7fb4e001d640 comp rx=0 tx=0).ready entity=mon.2 client_cookie=668f33deecf1cce8 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:03:42.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.873+0000 7ff569e00640 1 --2- 172.21.15.16:0/2692909161 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff56c15a660 0x7ff56c0ce860 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7ff55c009210 tx=0x7ff55c01d7b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f0b64cf7de18f0bc server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:03:42.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.873+0000 7ff568a00640 1 -- 172.21.15.16:0/2692909161 <== mon.2 v2:172.21.15.89:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff55c0a7690 con 0x7ff56c15a660 2024-03-31T15:03:42.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.873+0000 7efc0de00640 1 --2- 172.21.15.16:0/3781439136 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7efc080d2720 0x7efc081ea840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:42.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.873+0000 7efc0de00640 1 -- 172.21.15.16:0/3781439136 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efc080b39b0 msgr2=0x7efc081e9970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:42.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.873+0000 7efc0de00640 1 --2- 172.21.15.16:0/3781439136 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efc080b39b0 0x7efc081e9970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:42.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.873+0000 7f4badc00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f4ba80b35d0 0x7f4ba81e74c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:42.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.873+0000 7efc0de00640 1 -- 172.21.15.16:0/3781439136 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc080cab80 con 0x7efc080b35d0 2024-03-31T15:03:42.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.873+0000 7efc0d400640 1 --2- 172.21.15.16:0/3781439136 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efc080b39b0 0x7efc081e9970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:03:42.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.873+0000 7efc0e800640 1 --2- 172.21.15.16:0/3781439136 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7efc080d2720 0x7efc081ea840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:03:42.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.873+0000 7f4badc00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4ba80b39b0 0x7f4ba80cd930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:42.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.873+0000 7f4badc00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4ba80d2720 0x7f4ba80cde70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:42.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.873+0000 7f4badc00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f4ba80b86a0 con 0x7f4ba80b39b0 2024-03-31T15:03:42.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.873+0000 7f4badc00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f4ba80b8520 con 0x7f4ba80d2720 2024-03-31T15:03:42.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.873+0000 7f4badc00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f4ba80b6910 con 0x7f4ba80b35d0 2024-03-31T15:03:42.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.873+0000 7fb4f3400640 1 -- 172.21.15.16:0/3714067237 <== mon.2 v2:172.21.15.89:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4e00a9610 con 0x7fb4f40af730 2024-03-31T15:03:42.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.873+0000 7f1c8fe00640 1 --2- 172.21.15.16:0/1585641036 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f1c700683d0 0x7f1c7006a890 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f1c840ad000 tx=0x7f1c840a6000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:03:42.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.873+0000 7fcfc1400640 1 -- start start 2024-03-31T15:03:42.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.873+0000 7fb4f3400640 1 -- 172.21.15.16:0/3714067237 <== mon.2 v2:172.21.15.89:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb4e00a97b0 con 0x7fb4f40af730 2024-03-31T15:03:42.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.873+0000 7fb4f3400640 1 -- 172.21.15.16:0/3714067237 <== mon.2 v2:172.21.15.89:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4e00a9b80 con 0x7fb4f40af730 2024-03-31T15:03:42.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.873+0000 7f02e2000640 1 -- 172.21.15.16:0/2730080888 <== osd.0 v2:172.21.15.16:6800/932712244 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f02e812dec0 con 0x7f02cc0706a0 2024-03-31T15:03:42.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.877+0000 7f1c95600640 1 --2- 172.21.15.16:0/1585641036 >> [v2:172.21.15.16:6808/1398737533,v1:172.21.15.16:6810/1398737533] conn(0x7f1c70070650 0x7f1c70072a70 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:03:42.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.873+0000 7f4ba6a00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f4ba80b35d0 0x7f4ba81e74c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:42.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.877+0000 7f4ba6a00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f4ba80b35d0 0x7f4ba81e74c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3301/0 says I am v2:172.21.15.16:41532/0 (socket says 172.21.15.16:41532) 2024-03-31T15:03:42.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.877+0000 7f4ba6a00640 1 -- 172.21.15.16:0/1515004585 learned_addr learned my addr 172.21.15.16:0/1515004585 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:03:42.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.877+0000 7f4ba6000640 1 --2- 172.21.15.16:0/1515004585 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4ba80b39b0 0x7f4ba80cd930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:42.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.877+0000 7f4ba7400640 1 --2- 172.21.15.16:0/1515004585 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4ba80d2720 0x7f4ba80cde70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:42.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.877+0000 7ff568a00640 1 -- 172.21.15.16:0/2692909161 <== mon.2 v2:172.21.15.89:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff55c0a7830 con 0x7ff56c15a660 2024-03-31T15:03:42.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.877+0000 7ff568a00640 1 -- 172.21.15.16:0/2692909161 <== mon.2 v2:172.21.15.89:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff55c0a7c00 con 0x7ff56c15a660 2024-03-31T15:03:42.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.877+0000 7f4ba6a00640 1 -- 172.21.15.16:0/1515004585 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4ba80d2720 msgr2=0x7f4ba80cde70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:42.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.877+0000 7f4ba6a00640 1 --2- 172.21.15.16:0/1515004585 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4ba80d2720 0x7f4ba80cde70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:42.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.877+0000 7f4ba6a00640 1 -- 172.21.15.16:0/1515004585 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4ba80b39b0 msgr2=0x7f4ba80cd930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:42.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.877+0000 7f5ae8000640 1 Processor -- start 2024-03-31T15:03:42.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.877+0000 7f5ae8000640 1 -- start start 2024-03-31T15:03:42.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.877+0000 7fb4fb600640 1 -- 172.21.15.16:0/3714067237 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb4f40af730 msgr2=0x7fb4f40cdec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:42.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.877+0000 7fb4fb600640 1 --2- 172.21.15.16:0/3714067237 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb4f40af730 0x7fb4f40cdec0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fb4e00097c0 tx=0x7fb4e001d640 comp rx=0 tx=0).stop 2024-03-31T15:03:42.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.877+0000 7efc0de00640 1 --2- 172.21.15.16:0/3781439136 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efc080b35d0 0x7efc081d7d10 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7efbfc00e810 tx=0x7efbfc00ecd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:03:42.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.877+0000 7f4ba6a00640 1 --2- 172.21.15.16:0/1515004585 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4ba80b39b0 0x7f4ba80cd930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:42.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.877+0000 7f4ba6a00640 1 -- 172.21.15.16:0/1515004585 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ba80cc290 con 0x7f4ba80b35d0 2024-03-31T15:03:42.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.877+0000 7f4ba6000640 1 --2- 172.21.15.16:0/1515004585 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4ba80b39b0 0x7f4ba80cd930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:03:42.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.877+0000 7ff570c00640 1 -- 172.21.15.16:0/2692909161 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff56c15a660 msgr2=0x7ff56c0ce860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:42.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.877+0000 7ff570c00640 1 --2- 172.21.15.16:0/2692909161 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff56c15a660 0x7ff56c0ce860 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7ff55c009210 tx=0x7ff55c01d7b0 comp rx=0 tx=0).stop 2024-03-31T15:03:42.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.877+0000 7efc02000640 1 -- 172.21.15.16:0/3781439136 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efbfc017070 con 0x7efc080b35d0 2024-03-31T15:03:42.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.877+0000 7efc02000640 1 -- 172.21.15.16:0/3781439136 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efbfc00c8d0 con 0x7efc080b35d0 2024-03-31T15:03:42.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.877+0000 7efc02000640 1 -- 172.21.15.16:0/3781439136 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efbfc00f040 con 0x7efc080b35d0 2024-03-31T15:03:42.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.877+0000 7fb4fb600640 1 -- 172.21.15.16:0/3714067237 shutdown_connections 2024-03-31T15:03:42.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.877+0000 7fb4fb600640 1 --2- 172.21.15.16:0/3714067237 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb4f40af730 0x7fb4f40cdec0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:42.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.877+0000 7fb4fb600640 1 --2- 172.21.15.16:0/3714067237 >> v2:172.21.15.89:3300/0 conn(0x7fb4f415a680 0x7fb4f40b5040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:42.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.877+0000 7fb4fb600640 1 -- 172.21.15.16:0/3714067237 >> 172.21.15.16:0/3714067237 conn(0x7fb4f40b2040 msgr2=0x7fb4f40b2450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:03:42.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.877+0000 7efc10800640 1 -- 172.21.15.16:0/3781439136 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efc080cae60 con 0x7efc080b35d0 2024-03-31T15:03:42.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.877+0000 7fb4fb600640 1 -- 172.21.15.16:0/3714067237 shutdown_connections 2024-03-31T15:03:42.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.877+0000 7ff570c00640 1 -- 172.21.15.16:0/2692909161 shutdown_connections 2024-03-31T15:03:42.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.877+0000 7ff570c00640 1 --2- 172.21.15.16:0/2692909161 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff56c15a660 0x7ff56c0ce860 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:42.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.877+0000 7ff570c00640 1 -- 172.21.15.16:0/2692909161 >> 172.21.15.16:0/2692909161 conn(0x7ff56c0b1d30 msgr2=0x7ff56c0b2140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:03:42.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.877+0000 7f4ba6a00640 1 --2- 172.21.15.16:0/1515004585 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f4ba80b35d0 0x7f4ba81e74c0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f4ba000c710 tx=0x7f4ba000cbd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:03:42.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.877+0000 7efc10800640 1 -- 172.21.15.16:0/3781439136 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efc080cba10 con 0x7efc080b35d0 2024-03-31T15:03:42.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.877+0000 7fcfc1400640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fcfbc0b48d0 0x7fcfbc15ed50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:42.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.877+0000 7fcfc1400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcfbc0b4d30 0x7fcfbc154e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:42.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.877+0000 7f4b9ea00640 1 -- 172.21.15.16:0/1515004585 <== mon.2 v2:172.21.15.89:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ba0003b80 con 0x7f4ba80b35d0 2024-03-31T15:03:42.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.877+0000 7ff570c00640 1 -- 172.21.15.16:0/2692909161 shutdown_connections 2024-03-31T15:03:42.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.877+0000 7f4b9ea00640 1 -- 172.21.15.16:0/1515004585 <== mon.2 v2:172.21.15.89:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4ba0003d40 con 0x7f4ba80b35d0 2024-03-31T15:03:42.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.877+0000 7f4b9ea00640 1 -- 172.21.15.16:0/1515004585 <== mon.2 v2:172.21.15.89:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ba000aa90 con 0x7f4ba80b35d0 2024-03-31T15:03:42.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.877+0000 7f4badc00640 1 -- 172.21.15.16:0/1515004585 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4ba80cfc30 con 0x7f4ba80b35d0 2024-03-31T15:03:42.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.877+0000 7f1c8cc00640 1 -- 172.21.15.16:0/1585641036 <== osd.1 v2:172.21.15.16:6808/1398737533 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f1c70073120 con 0x7f1c70070650 2024-03-31T15:03:42.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.877+0000 7fcfba800640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fcfbc0b48d0 0x7fcfbc15ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:42.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.877+0000 7fcfba800640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fcfbc0b48d0 0x7fcfbc15ed50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3301/0 says I am v2:172.21.15.16:41542/0 (socket says 172.21.15.16:41542) 2024-03-31T15:03:42.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.877+0000 7f5ae8000640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f5ae00b3030 0x7f5ae0141a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:42.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.877+0000 7f5ae8000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5ae00cf6a0 0x7f5ae01f8500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:42.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.877+0000 7f5ae8000640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5ae015a660 0x7f5ae0158cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:42.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.877+0000 7f5ae8000640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f5ae0071c10 con 0x7f5ae00cf6a0 2024-03-31T15:03:42.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.877+0000 7f5ae8000640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f5ae00732a0 con 0x7f5ae015a660 2024-03-31T15:03:42.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.877+0000 7f5ae8000640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f5ae00b4d30 con 0x7f5ae00b3030 2024-03-31T15:03:42.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.877+0000 7f4badc00640 1 -- 172.21.15.16:0/1515004585 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f4ba80cbeb0 con 0x7f4ba80b35d0 2024-03-31T15:03:42.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.877+0000 7efc10800640 1 -- 172.21.15.16:0/3781439136 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7efc08069950 con 0x7efc080b35d0 2024-03-31T15:03:42.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.877+0000 7fcfb9e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcfbc0b4d30 0x7fcfbc154e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:42.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.877+0000 7fcfb9e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcfbc0b4d30 0x7fcfbc154e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39012/0 (socket says 172.21.15.16:39012) 2024-03-31T15:03:42.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.877+0000 7fcfba800640 1 -- 172.21.15.16:0/1303149306 learned_addr learned my addr 172.21.15.16:0/1303149306 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:03:42.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.877+0000 7fcfc1400640 1 --2- 172.21.15.16:0/1303149306 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcfbc165dc0 0x7fcfbc1574c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:42.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.877+0000 7fcfc1400640 1 -- 172.21.15.16:0/1303149306 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fcfbc166320 con 0x7fcfbc0b4d30 2024-03-31T15:03:42.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.877+0000 7fcfc1400640 1 -- 172.21.15.16:0/1303149306 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fcfbc1661a0 con 0x7fcfbc165dc0 2024-03-31T15:03:42.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.877+0000 7fcfc1400640 1 -- 172.21.15.16:0/1303149306 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fcfbc0ce6b0 con 0x7fcfbc0b48d0 2024-03-31T15:03:42.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.877+0000 7ff570c00640 1 -- 172.21.15.16:0/2692909161 wait complete. 2024-03-31T15:03:42.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.877+0000 7f4badc00640 1 -- 172.21.15.16:0/1515004585 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f4ba80ce3b0 con 0x7f4ba80b35d0 2024-03-31T15:03:42.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.877+0000 7fcfbb200640 1 --2- 172.21.15.16:0/1303149306 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcfbc165dc0 0x7fcfbc1574c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:42.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.881+0000 7efc02000640 1 -- 172.21.15.16:0/3781439136 <== mon.1 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7efbfc01a070 con 0x7efc080b35d0 2024-03-31T15:03:42.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.881+0000 7fcfba800640 1 -- 172.21.15.16:0/1303149306 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcfbc165dc0 msgr2=0x7fcfbc1574c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:42.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.881+0000 7fcfba800640 1 --2- 172.21.15.16:0/1303149306 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcfbc165dc0 0x7fcfbc1574c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:42.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.881+0000 7fcfba800640 1 -- 172.21.15.16:0/1303149306 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcfbc0b4d30 msgr2=0x7fcfbc154e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:42.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.881+0000 7fcfba800640 1 --2- 172.21.15.16:0/1303149306 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcfbc0b4d30 0x7fcfbc154e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:42.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.881+0000 7fcfba800640 1 -- 172.21.15.16:0/1303149306 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcfbc15eb00 con 0x7fcfbc0b48d0 2024-03-31T15:03:42.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.881+0000 7fcfbb200640 1 --2- 172.21.15.16:0/1303149306 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcfbc165dc0 0x7fcfbc1574c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:03:42.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.877+0000 7fb4fb600640 1 -- 172.21.15.16:0/3714067237 wait complete. 2024-03-31T15:03:42.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.877+0000 7f5ae5600640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f5ae00b3030 0x7f5ae0141a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:42.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.877+0000 7f5ae5600640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f5ae00b3030 0x7f5ae0141a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3301/0 says I am v2:172.21.15.16:41544/0 (socket says 172.21.15.16:41544) 2024-03-31T15:03:42.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.877+0000 7f5ae5600640 1 -- 172.21.15.16:0/2669765298 learned_addr learned my addr 172.21.15.16:0/2669765298 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:03:42.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.877+0000 7f5ae6000640 1 --2- 172.21.15.16:0/2669765298 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5ae015a660 0x7f5ae0158cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:42.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.881+0000 7f5ae4c00640 1 --2- 172.21.15.16:0/2669765298 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5ae00cf6a0 0x7f5ae01f8500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:42.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.881+0000 7fcfba800640 1 --2- 172.21.15.16:0/1303149306 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fcfbc0b48d0 0x7fcfbc15ed50 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fcfa801ab30 tx=0x7fcfa801acc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:03:42.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.881+0000 7fb4fb600640 1 Processor -- start 2024-03-31T15:03:42.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.881+0000 7f5ae6000640 1 -- 172.21.15.16:0/2669765298 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f5ae00b3030 msgr2=0x7f5ae0141a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:42.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.881+0000 7f5ae6000640 1 --2- 172.21.15.16:0/2669765298 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f5ae00b3030 0x7f5ae0141a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:42.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.881+0000 7f5ae6000640 1 -- 172.21.15.16:0/2669765298 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5ae00cf6a0 msgr2=0x7f5ae01f8500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:42.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.881+0000 7f5ae6000640 1 --2- 172.21.15.16:0/2669765298 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5ae00cf6a0 0x7f5ae01f8500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:42.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.881+0000 7f5ae6000640 1 -- 172.21.15.16:0/2669765298 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ae015bff0 con 0x7f5ae015a660 2024-03-31T15:03:42.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.881+0000 7fcfb2a00640 1 -- 172.21.15.16:0/1303149306 <== mon.2 v2:172.21.15.89:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfa80bb020 con 0x7fcfbc0b48d0 2024-03-31T15:03:42.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.881+0000 7fcfb2a00640 1 -- 172.21.15.16:0/1303149306 <== mon.2 v2:172.21.15.89:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcfa80ace30 con 0x7fcfbc0b48d0 2024-03-31T15:03:42.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.881+0000 7fcfb2a00640 1 -- 172.21.15.16:0/1303149306 <== mon.2 v2:172.21.15.89:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfa80b1070 con 0x7fcfbc0b48d0 2024-03-31T15:03:42.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.881+0000 7fcfc1400640 1 -- 172.21.15.16:0/1303149306 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcfbc15d1e0 con 0x7fcfbc0b48d0 2024-03-31T15:03:42.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.881+0000 7efc02000640 1 --2- 172.21.15.16:0/3781439136 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7efbec0683d0 0x7efbec06a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:42.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.881+0000 7fb4fb600640 1 -- start start 2024-03-31T15:03:42.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.881+0000 7fcfc1400640 1 -- 172.21.15.16:0/1303149306 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fcfbc15d7a0 con 0x7fcfbc0b48d0 2024-03-31T15:03:42.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.881+0000 7efc02000640 1 -- 172.21.15.16:0/3781439136 <== mon.1 v2:172.21.15.89:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7efbfc086500 con 0x7efc080b35d0 2024-03-31T15:03:42.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.881+0000 7efc02000640 1 --2- 172.21.15.16:0/3781439136 >> [v2:172.21.15.16:6819/3018526646,v1:172.21.15.16:6822/3018526646] conn(0x7efbec070650 0x7efbec072a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:42.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.881+0000 7efc02000640 1 -- 172.21.15.16:0/3781439136 --> [v2:172.21.15.16:6819/3018526646,v1:172.21.15.16:6822/3018526646] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7efbfc020e50 con 0x7efbec070650 2024-03-31T15:03:42.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.881+0000 7efc02000640 1 -- 172.21.15.16:0/3781439136 <== mon.1 v2:172.21.15.89:3300/0 6 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7efbfc08d050 con 0x7efc080b35d0 2024-03-31T15:03:42.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.881+0000 7f5ae6000640 1 --2- 172.21.15.16:0/2669765298 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5ae015a660 0x7f5ae0158cb0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f5ad80ad000 tx=0x7f5ad801a540 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:03:42.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.881+0000 7f4b9ea00640 1 -- 172.21.15.16:0/1515004585 <== mon.2 v2:172.21.15.89:3301/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f4ba000ac30 con 0x7f4ba80b35d0 2024-03-31T15:03:42.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.881+0000 7efc0e800640 1 --2- 172.21.15.16:0/3781439136 >> [v2:172.21.15.16:6819/3018526646,v1:172.21.15.16:6822/3018526646] conn(0x7efbec070650 0x7efbec072a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:42.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.881+0000 7efc0d400640 1 --2- 172.21.15.16:0/3781439136 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7efbec0683d0 0x7efbec06a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:42.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.881+0000 7fb4fb600640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb4f4068d60 0x7fb4f4151350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:42.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.881+0000 7fb4fb600640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb4f40af730 0x7fb4f415c580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:42.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.881+0000 7fb4fb600640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb4f415a680 0x7fb4f415cac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:42.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.881+0000 7fb4fb600640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fb4f41673e0 con 0x7fb4f415a680 2024-03-31T15:03:42.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.881+0000 7fb4fb600640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fb4f40ce810 con 0x7fb4f4068d60 2024-03-31T15:03:42.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.881+0000 7fb4fb600640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fb4f4167560 con 0x7fb4f40af730 2024-03-31T15:03:42.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.881+0000 7f4b9ea00640 1 --2- 172.21.15.16:0/1515004585 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f4b88068400 0x7f4b8806a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:42.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.885+0000 7f4b9ea00640 1 -- 172.21.15.16:0/1515004585 <== mon.2 v2:172.21.15.89:3301/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f4ba0086690 con 0x7f4ba80b35d0 2024-03-31T15:03:42.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.881+0000 7f5add600640 1 -- 172.21.15.16:0/2669765298 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ad80bc070 con 0x7f5ae015a660 2024-03-31T15:03:42.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.885+0000 7efc0e800640 1 --2- 172.21.15.16:0/3781439136 >> [v2:172.21.15.16:6819/3018526646,v1:172.21.15.16:6822/3018526646] conn(0x7efbec070650 0x7efbec072a70 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:03:42.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.885+0000 7f4b9ea00640 1 --2- 172.21.15.16:0/1515004585 >> [v2:172.21.15.89:6800/1856844856,v1:172.21.15.89:6801/1856844856] conn(0x7f4b88070680 0x7f4b88072aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:42.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.885+0000 7f4b9ea00640 1 -- 172.21.15.16:0/1515004585 --> [v2:172.21.15.89:6800/1856844856,v1:172.21.15.89:6801/1856844856] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f4ba0010e50 con 0x7f4b88070680 2024-03-31T15:03:42.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.885+0000 7f4b9ea00640 1 -- 172.21.15.16:0/1515004585 <== mon.2 v2:172.21.15.89:3301/0 6 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f4ba008d050 con 0x7f4ba80b35d0 2024-03-31T15:03:42.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.885+0000 7f4ba6000640 1 --2- 172.21.15.16:0/1515004585 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f4b88068400 0x7f4b8806a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:42.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.885+0000 7fb4f9600640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb4f415a680 0x7fb4f415cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:42.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.885+0000 7fb4f9600640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb4f415a680 0x7fb4f415cac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39044/0 (socket says 172.21.15.16:39044) 2024-03-31T15:03:42.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.885+0000 7fb4f9600640 1 -- 172.21.15.16:0/1482123868 learned_addr learned my addr 172.21.15.16:0/1482123868 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:03:42.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.885+0000 7f5add600640 1 -- 172.21.15.16:0/2669765298 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5ad8004040 con 0x7f5ae015a660 2024-03-31T15:03:42.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.885+0000 7f5add600640 1 -- 172.21.15.16:0/2669765298 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ad80b3040 con 0x7f5ae015a660 2024-03-31T15:03:42.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.885+0000 7ff570c00640 1 Processor -- start 2024-03-31T15:03:42.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.885+0000 7fb4f3e00640 1 --2- 172.21.15.16:0/1482123868 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb4f40af730 0x7fb4f415c580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:42.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.885+0000 7fb4f8c00640 1 --2- 172.21.15.16:0/1482123868 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb4f4068d60 0x7fb4f4151350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:42.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.885+0000 7f5ae8000640 1 -- 172.21.15.16:0/2669765298 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5ae015c2d0 con 0x7f5ae015a660 2024-03-31T15:03:42.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.885+0000 7f5ae8000640 1 -- 172.21.15.16:0/2669765298 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5ae015c890 con 0x7f5ae015a660 2024-03-31T15:03:42.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.885+0000 7efc0d400640 1 --2- 172.21.15.16:0/3781439136 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7efbec0683d0 0x7efbec06a890 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7efbf8007a90 tx=0x7efbf80079e0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:03:42.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.885+0000 7ff570c00640 1 -- start start 2024-03-31T15:03:42.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.885+0000 7fcfc1400640 1 -- 172.21.15.16:0/1303149306 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fcfbc069950 con 0x7fcfbc0b48d0 2024-03-31T15:03:42.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.885+0000 7f4ba6000640 1 --2- 172.21.15.16:0/1515004585 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f4b88068400 0x7f4b8806a8c0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f4ba81e8080 tx=0x7f4b94007e90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:03:42.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.885+0000 7efc02000640 1 -- 172.21.15.16:0/3781439136 <== osd.2 v2:172.21.15.16:6819/3018526646 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7efbfc020e50 con 0x7efbec070650 2024-03-31T15:03:42.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.885+0000 7fb4f3e00640 1 -- 172.21.15.16:0/1482123868 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb4f4068d60 msgr2=0x7fb4f4151350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:42.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.885+0000 7fb4f3e00640 1 --2- 172.21.15.16:0/1482123868 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb4f4068d60 0x7fb4f4151350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:42.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.885+0000 7fb4f3e00640 1 -- 172.21.15.16:0/1482123868 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb4f415a680 msgr2=0x7fb4f415cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:42.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.885+0000 7fb4f3e00640 1 --2- 172.21.15.16:0/1482123868 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb4f415a680 0x7fb4f415cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:42.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.885+0000 7fb4f3e00640 1 -- 172.21.15.16:0/1482123868 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb4f4155cf0 con 0x7fb4f40af730 2024-03-31T15:03:42.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.885+0000 7fcfb2a00640 1 -- 172.21.15.16:0/1303149306 <== mon.2 v2:172.21.15.89:3301/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fcfa80be020 con 0x7fcfbc0b48d0 2024-03-31T15:03:42.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.885+0000 7f5ae8000640 1 -- 172.21.15.16:0/2669765298 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f5ae0069950 con 0x7f5ae015a660 2024-03-31T15:03:42.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.885+0000 7fb4f3e00640 1 --2- 172.21.15.16:0/1482123868 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb4f40af730 0x7fb4f415c580 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fb4f4069180 tx=0x7fb4e400ce80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:03:42.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.885+0000 7f4ba7400640 1 --2- 172.21.15.16:0/1515004585 >> [v2:172.21.15.89:6800/1856844856,v1:172.21.15.89:6801/1856844856] conn(0x7f4b88070680 0x7f4b88072aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:42.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.885+0000 7fcfb2a00640 1 --2- 172.21.15.16:0/1303149306 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fcf9c068380 0x7fcf9c06a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:42.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.885+0000 7fb4f0c00640 1 -- 172.21.15.16:0/1482123868 <== mon.2 v2:172.21.15.89:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4e4017070 con 0x7fb4f40af730 2024-03-31T15:03:42.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.885+0000 7fb4f0c00640 1 -- 172.21.15.16:0/1482123868 <== mon.2 v2:172.21.15.89:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb4e400ab70 con 0x7fb4f40af730 2024-03-31T15:03:42.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.885+0000 7fb4f0c00640 1 -- 172.21.15.16:0/1482123868 <== mon.2 v2:172.21.15.89:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4e4012400 con 0x7fb4f40af730 2024-03-31T15:03:42.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.885+0000 7fb4fb600640 1 -- 172.21.15.16:0/1482123868 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb4f4155fd0 con 0x7fb4f40af730 2024-03-31T15:03:42.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.889+0000 7fcfb2a00640 1 -- 172.21.15.16:0/1303149306 <== mon.2 v2:172.21.15.89:3301/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fcfa8120de0 con 0x7fcfbc0b48d0 2024-03-31T15:03:42.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.889+0000 7fcfb9e00640 1 --2- 172.21.15.16:0/1303149306 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fcf9c068380 0x7fcf9c06a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:42.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.889+0000 7fcfb2a00640 1 --2- 172.21.15.16:0/1303149306 >> [v2:172.21.15.16:6809/3360352974,v1:172.21.15.16:6812/3360352974] conn(0x7fcf9c0705a0 0x7fcf9c0729c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:42.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.889+0000 7fcfb2a00640 1 -- 172.21.15.16:0/1303149306 --> [v2:172.21.15.16:6809/3360352974,v1:172.21.15.16:6812/3360352974] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fcfa8006800 con 0x7fcf9c0705a0 2024-03-31T15:03:42.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.889+0000 7fcfb2a00640 1 -- 172.21.15.16:0/1303149306 <== mon.2 v2:172.21.15.89:3301/0 6 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fcfa8123440 con 0x7fcfbc0b48d0 2024-03-31T15:03:42.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.889+0000 7ff570c00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff56c0b3030 0x7ff56c14cd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:42.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.889+0000 7fcfbb200640 1 --2- 172.21.15.16:0/1303149306 >> [v2:172.21.15.16:6809/3360352974,v1:172.21.15.16:6812/3360352974] conn(0x7fcf9c0705a0 0x7fcf9c0729c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:42.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.889+0000 7fcfb9e00640 1 --2- 172.21.15.16:0/1303149306 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fcf9c068380 0x7fcf9c06a840 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fcfbc154520 tx=0x7fcfac008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:03:42.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.889+0000 7ff570c00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff56c0cf6a0 0x7ff56c149020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:42.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.889+0000 7fb4fb600640 1 -- 172.21.15.16:0/1482123868 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fb4f41fbca0 con 0x7fb4f40af730 2024-03-31T15:03:42.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.889+0000 7ff569e00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff56c0b3030 0x7ff56c14cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:42.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.889+0000 7ff569e00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff56c0b3030 0x7ff56c14cd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.16:36054/0 (socket says 172.21.15.16:36054) 2024-03-31T15:03:42.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.889+0000 7ff569e00640 1 -- 172.21.15.16:0/2245740762 learned_addr learned my addr 172.21.15.16:0/2245740762 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:03:42.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.889+0000 7ff570c00640 1 --2- 172.21.15.16:0/2245740762 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff56c15a660 0x7ff56c0cffc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:42.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.889+0000 7ff569400640 1 --2- 172.21.15.16:0/2245740762 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff56c0cf6a0 0x7ff56c149020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:42.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.889+0000 7fcfbb200640 1 --2- 172.21.15.16:0/1303149306 >> [v2:172.21.15.16:6809/3360352974,v1:172.21.15.16:6812/3360352974] conn(0x7fcf9c0705a0 0x7fcf9c0729c0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:03:42.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.889+0000 7ff570c00640 1 -- 172.21.15.16:0/2245740762 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7ff56c06f1a0 con 0x7ff56c15a660 2024-03-31T15:03:42.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.889+0000 7ff570c00640 1 -- 172.21.15.16:0/2245740762 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7ff56c06f020 con 0x7ff56c0b3030 2024-03-31T15:03:42.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.889+0000 7ff56a800640 1 --2- 172.21.15.16:0/2245740762 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff56c15a660 0x7ff56c0cffc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:42.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.889+0000 7ff570c00640 1 -- 172.21.15.16:0/2245740762 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7ff56c0b2850 con 0x7ff56c0cf6a0 2024-03-31T15:03:42.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.889+0000 7f5add600640 1 -- 172.21.15.16:0/2669765298 <== mon.1 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f5ad80bf070 con 0x7f5ae015a660 2024-03-31T15:03:42.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.889+0000 7f5add600640 1 --2- 172.21.15.16:0/2669765298 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f5ac0068450 0x7f5ac006a910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:42.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.889+0000 7f5add600640 1 -- 172.21.15.16:0/2669765298 <== mon.1 v2:172.21.15.89:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f5ad81209f0 con 0x7f5ae015a660 2024-03-31T15:03:42.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.889+0000 7f5add600640 1 --2- 172.21.15.16:0/2669765298 >> [v2:172.21.15.89:6808/1796778630,v1:172.21.15.89:6811/1796778630] conn(0x7f5ac0070670 0x7f5ac0072a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:42.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.889+0000 7f5add600640 1 -- 172.21.15.16:0/2669765298 --> [v2:172.21.15.89:6808/1796778630,v1:172.21.15.89:6811/1796778630] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f5ad80b4e50 con 0x7f5ac0070670 2024-03-31T15:03:42.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.889+0000 7f5add600640 1 -- 172.21.15.16:0/2669765298 <== mon.1 v2:172.21.15.89:3300/0 6 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f5ad81252a0 con 0x7f5ae015a660 2024-03-31T15:03:42.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.889+0000 7fcfb2a00640 1 -- 172.21.15.16:0/1303149306 <== osd.3 v2:172.21.15.16:6809/3360352974 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fcfa8006800 con 0x7fcf9c0705a0 2024-03-31T15:03:42.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.889+0000 7ff569e00640 1 -- 172.21.15.16:0/2245740762 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff56c0cf6a0 msgr2=0x7ff56c149020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:42.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.889+0000 7ff569e00640 1 --2- 172.21.15.16:0/2245740762 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff56c0cf6a0 0x7ff56c149020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:42.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.889+0000 7ff569e00640 1 -- 172.21.15.16:0/2245740762 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff56c15a660 msgr2=0x7ff56c0cffc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:42.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.893+0000 7ff569e00640 1 --2- 172.21.15.16:0/2245740762 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff56c15a660 0x7ff56c0cffc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:42.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.893+0000 7ff569e00640 1 -- 172.21.15.16:0/2245740762 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff56c14f940 con 0x7ff56c0b3030 2024-03-31T15:03:42.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.893+0000 7ff569400640 1 --2- 172.21.15.16:0/2245740762 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff56c0cf6a0 0x7ff56c149020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:03:42.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.893+0000 7ff56a800640 1 --2- 172.21.15.16:0/2245740762 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff56c15a660 0x7ff56c0cffc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:03:42.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.893+0000 7fb4f0c00640 1 -- 172.21.15.16:0/1482123868 <== mon.2 v2:172.21.15.89:3301/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fb4e401a070 con 0x7fb4f40af730 2024-03-31T15:03:42.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.893+0000 7fb4fb600640 1 -- 172.21.15.16:0/1482123868 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fb4f4069950 con 0x7fb4f40af730 2024-03-31T15:03:42.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.893+0000 7f5ae5600640 1 --2- 172.21.15.16:0/2669765298 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f5ac0068450 0x7f5ac006a910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:42.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.893+0000 7f5ae4c00640 1 --2- 172.21.15.16:0/2669765298 >> [v2:172.21.15.89:6808/1796778630,v1:172.21.15.89:6811/1796778630] conn(0x7f5ac0070670 0x7f5ac0072a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:42.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.893+0000 7f5ae4c00640 1 --2- 172.21.15.16:0/2669765298 >> [v2:172.21.15.89:6808/1796778630,v1:172.21.15.89:6811/1796778630] conn(0x7f5ac0070670 0x7f5ac0072a90 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:03:42.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.893+0000 7f5ae5600640 1 --2- 172.21.15.16:0/2669765298 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f5ac0068450 0x7f5ac006a910 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f5acc000f70 tx=0x7f5acc007ef0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:03:42.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.893+0000 7fb4f0c00640 1 --2- 172.21.15.16:0/1482123868 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fb4d40683d0 0x7fb4d406a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:42.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.893+0000 7fb4f0c00640 1 -- 172.21.15.16:0/1482123868 <== mon.2 v2:172.21.15.89:3301/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fb4e408b060 con 0x7fb4f40af730 2024-03-31T15:03:42.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.893+0000 7f4ba7400640 1 --2- 172.21.15.16:0/1515004585 >> [v2:172.21.15.89:6800/1856844856,v1:172.21.15.89:6801/1856844856] conn(0x7f4b88070680 0x7f4b88072aa0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:03:42.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.893+0000 7fb4f0c00640 1 --2- 172.21.15.16:0/1482123868 >> [v2:172.21.15.89:6810/50395527,v1:172.21.15.89:6814/50395527] conn(0x7fb4d4070570 0x7fb4d4072990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:42.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.893+0000 7fb4f0c00640 1 -- 172.21.15.16:0/1482123868 --> [v2:172.21.15.89:6810/50395527,v1:172.21.15.89:6814/50395527] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb4d4073040 con 0x7fb4d4070570 2024-03-31T15:03:42.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.893+0000 7fb4f0c00640 1 -- 172.21.15.16:0/1482123868 <== mon.2 v2:172.21.15.89:3301/0 6 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fb4e40523e0 con 0x7fb4f40af730 2024-03-31T15:03:42.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.893+0000 7fb4f8c00640 1 --2- 172.21.15.16:0/1482123868 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fb4d40683d0 0x7fb4d406a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:42.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.893+0000 7fb4f9600640 1 --2- 172.21.15.16:0/1482123868 >> [v2:172.21.15.89:6810/50395527,v1:172.21.15.89:6814/50395527] conn(0x7fb4d4070570 0x7fb4d4072990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:42.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.893+0000 7ff569e00640 1 --2- 172.21.15.16:0/2245740762 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff56c0b3030 0x7ff56c14cd50 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7ff55c01b3e0 tx=0x7ff55c01a550 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:03:42.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.893+0000 7f5add600640 1 -- 172.21.15.16:0/2669765298 <== osd.7 v2:172.21.15.89:6808/1796778630 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f5ad80b4e50 con 0x7f5ac0070670 2024-03-31T15:03:42.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.893+0000 7f4b9ea00640 1 -- 172.21.15.16:0/1515004585 <== osd.4 v2:172.21.15.89:6800/1856844856 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f4ba0010e50 con 0x7f4b88070680 2024-03-31T15:03:42.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.893+0000 7ff562000640 1 -- 172.21.15.16:0/2245740762 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff55c0a8880 con 0x7ff56c0b3030 2024-03-31T15:03:42.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.893+0000 7ff562000640 1 -- 172.21.15.16:0/2245740762 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff55c0a8a20 con 0x7ff56c0b3030 2024-03-31T15:03:42.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.893+0000 7fb4f8c00640 1 --2- 172.21.15.16:0/1482123868 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fb4d40683d0 0x7fb4d406a890 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fb4f40affc0 tx=0x7fb4ec008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:03:42.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.893+0000 7fb4f9600640 1 --2- 172.21.15.16:0/1482123868 >> [v2:172.21.15.89:6810/50395527,v1:172.21.15.89:6814/50395527] conn(0x7fb4d4070570 0x7fb4d4072990 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:03:42.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.893+0000 7ff570c00640 1 -- 172.21.15.16:0/2245740762 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff56c14fc20 con 0x7ff56c0b3030 2024-03-31T15:03:42.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.893+0000 7ff570c00640 1 -- 172.21.15.16:0/2245740762 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff56c1501e0 con 0x7ff56c0b3030 2024-03-31T15:03:42.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.893+0000 7ff562000640 1 -- 172.21.15.16:0/2245740762 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff55c0a8df0 con 0x7ff56c0b3030 2024-03-31T15:03:42.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.897+0000 7fb4f0c00640 1 -- 172.21.15.16:0/1482123868 <== osd.5 v2:172.21.15.89:6810/50395527 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fb4d4073040 con 0x7fb4d4070570 2024-03-31T15:03:42.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.901+0000 7ff570c00640 1 -- 172.21.15.16:0/2245740762 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7ff544001040 con 0x7ff56c0b3030 2024-03-31T15:03:42.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.901+0000 7ff562000640 1 -- 172.21.15.16:0/2245740762 <== mon.1 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7ff55c0a76f0 con 0x7ff56c0b3030 2024-03-31T15:03:42.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.901+0000 7ff562000640 1 --2- 172.21.15.16:0/2245740762 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7ff54c06c7e0 0x7ff54c06eca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:42.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.901+0000 7ff562000640 1 -- 172.21.15.16:0/2245740762 <== mon.1 v2:172.21.15.89:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7ff55c1203a0 con 0x7ff56c0b3030 2024-03-31T15:03:42.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.905+0000 7ff562000640 1 --2- 172.21.15.16:0/2245740762 >> [v2:172.21.15.89:6809/3470471752,v1:172.21.15.89:6812/3470471752] conn(0x7ff54c074a00 0x7ff54c076e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:42.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.905+0000 7ff562000640 1 -- 172.21.15.16:0/2245740762 --> [v2:172.21.15.89:6809/3470471752,v1:172.21.15.89:6812/3470471752] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff55c0aaec0 con 0x7ff54c074a00 2024-03-31T15:03:42.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.905+0000 7ff562000640 1 -- 172.21.15.16:0/2245740762 <== mon.1 v2:172.21.15.89:3300/0 6 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7ff55c127050 con 0x7ff56c0b3030 2024-03-31T15:03:42.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.905+0000 7ff569400640 1 --2- 172.21.15.16:0/2245740762 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7ff54c06c7e0 0x7ff54c06eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:42.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.905+0000 7ff56a800640 1 --2- 172.21.15.16:0/2245740762 >> [v2:172.21.15.89:6809/3470471752,v1:172.21.15.89:6812/3470471752] conn(0x7ff54c074a00 0x7ff54c076e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:42.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.905+0000 7ff569400640 1 --2- 172.21.15.16:0/2245740762 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7ff54c06c7e0 0x7ff54c06eca0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7ff56c151090 tx=0x7ff558009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:03:42.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.909+0000 7ff56a800640 1 --2- 172.21.15.16:0/2245740762 >> [v2:172.21.15.89:6809/3470471752,v1:172.21.15.89:6812/3470471752] conn(0x7ff54c074a00 0x7ff54c076e20 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:03:42.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.909+0000 7f02ef200640 1 -- 172.21.15.16:0/2730080888 --> [v2:172.21.15.16:6800/932712244,v1:172.21.15.16:6801/932712244] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f02e40052d0 con 0x7f02cc0706a0 2024-03-31T15:03:42.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.909+0000 7efc10800640 1 -- 172.21.15.16:0/3781439136 --> [v2:172.21.15.16:6819/3018526646,v1:172.21.15.16:6822/3018526646] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7efc0806d5d0 con 0x7efbec070650 2024-03-31T15:03:42.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.909+0000 7ff562000640 1 -- 172.21.15.16:0/2245740762 <== osd.6 v2:172.21.15.89:6809/3470471752 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ff55c0aaec0 con 0x7ff54c074a00 2024-03-31T15:03:42.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.909+0000 7f02e2000640 1 -- 172.21.15.16:0/2730080888 <== osd.0 v2:172.21.15.16:6800/932712244 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f02e40052d0 con 0x7f02cc0706a0 2024-03-31T15:03:42.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.909+0000 7efc02000640 1 -- 172.21.15.16:0/3781439136 <== osd.2 v2:172.21.15.16:6819/3018526646 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7efc0806d5d0 con 0x7efbec070650 2024-03-31T15:03:42.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.909+0000 7f02ef200640 1 -- 172.21.15.16:0/2730080888 >> [v2:172.21.15.16:6800/932712244,v1:172.21.15.16:6801/932712244] conn(0x7f02cc0706a0 msgr2=0x7f02cc072ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:42.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.909+0000 7f02ef200640 1 --2- 172.21.15.16:0/2730080888 >> [v2:172.21.15.16:6800/932712244,v1:172.21.15.16:6801/932712244] conn(0x7f02cc0706a0 0x7f02cc072ac0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:42.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.909+0000 7f02ef200640 1 -- 172.21.15.16:0/2730080888 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f02cc068480 msgr2=0x7f02cc06a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:42.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.909+0000 7f02ef200640 1 --2- 172.21.15.16:0/2730080888 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f02cc068480 0x7f02cc06a940 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f02dc009ae0 tx=0x7f02dc008040 comp rx=0 tx=0).stop 2024-03-31T15:03:42.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.909+0000 7f02ef200640 1 -- 172.21.15.16:0/2730080888 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f02e4047c20 msgr2=0x7f02e404c380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:42.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.909+0000 7f02ef200640 1 --2- 172.21.15.16:0/2730080888 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f02e4047c20 0x7f02e404c380 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f02e8050bd0 tx=0x7f02e8065390 comp rx=0 tx=0).stop 2024-03-31T15:03:42.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.909+0000 7f02ee800640 1 -- 172.21.15.16:0/2730080888 reap_dead start 2024-03-31T15:03:42.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.913+0000 7f02ef200640 1 -- 172.21.15.16:0/2730080888 shutdown_connections 2024-03-31T15:03:42.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.913+0000 7f02ef200640 1 -- 172.21.15.16:0/2730080888 >> 172.21.15.16:0/2730080888 conn(0x7f02e4009fc0 msgr2=0x7f02e40e6ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:03:42.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.913+0000 7f1c97600640 1 -- 172.21.15.16:0/1585641036 --> [v2:172.21.15.16:6808/1398737533,v1:172.21.15.16:6810/1398737533] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f1c6c002de0 con 0x7f1c70070650 2024-03-31T15:03:42.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.913+0000 7f02ef200640 1 -- 172.21.15.16:0/2730080888 shutdown_connections 2024-03-31T15:03:42.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.913+0000 7f02ef200640 1 -- 172.21.15.16:0/2730080888 wait complete. 2024-03-31T15:03:42.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.913+0000 7f1c8cc00640 1 -- 172.21.15.16:0/1585641036 <== osd.1 v2:172.21.15.16:6808/1398737533 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f1c6c002de0 con 0x7f1c70070650 2024-03-31T15:03:42.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.913+0000 7f1c97600640 1 -- 172.21.15.16:0/1585641036 >> [v2:172.21.15.16:6808/1398737533,v1:172.21.15.16:6810/1398737533] conn(0x7f1c70070650 msgr2=0x7f1c70072a70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:42.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.913+0000 7f1c97600640 1 --2- 172.21.15.16:0/1585641036 >> [v2:172.21.15.16:6808/1398737533,v1:172.21.15.16:6810/1398737533] conn(0x7f1c70070650 0x7f1c70072a70 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:42.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.913+0000 7efc10800640 1 -- 172.21.15.16:0/3781439136 >> [v2:172.21.15.16:6819/3018526646,v1:172.21.15.16:6822/3018526646] conn(0x7efbec070650 msgr2=0x7efbec072a70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:42.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.913+0000 7f1c97600640 1 -- 172.21.15.16:0/1585641036 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f1c700683d0 msgr2=0x7f1c7006a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:42.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.913+0000 7f1c97600640 1 --2- 172.21.15.16:0/1585641036 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f1c700683d0 0x7f1c7006a890 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f1c840ad000 tx=0x7f1c840a6000 comp rx=0 tx=0).stop 2024-03-31T15:03:42.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.913+0000 7f1c97600640 1 -- 172.21.15.16:0/1585641036 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f1c900b3030 msgr2=0x7f1c90157240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:42.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.913+0000 7f1c97600640 1 --2- 172.21.15.16:0/1585641036 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f1c900b3030 0x7f1c90157240 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f1c7c00c520 tx=0x7f1c7c00c9e0 comp rx=0 tx=0).stop 2024-03-31T15:03:42.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.913+0000 7efc10800640 1 --2- 172.21.15.16:0/3781439136 >> [v2:172.21.15.16:6819/3018526646,v1:172.21.15.16:6822/3018526646] conn(0x7efbec070650 0x7efbec072a70 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:42.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.913+0000 7f1c95600640 1 -- 172.21.15.16:0/1585641036 reap_dead start 2024-03-31T15:03:42.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.913+0000 7efc10800640 1 -- 172.21.15.16:0/3781439136 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7efbec0683d0 msgr2=0x7efbec06a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:42.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.913+0000 7efc10800640 1 --2- 172.21.15.16:0/3781439136 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7efbec0683d0 0x7efbec06a890 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7efbf8007a90 tx=0x7efbf80079e0 comp rx=0 tx=0).stop 2024-03-31T15:03:42.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.913+0000 7f1c97600640 1 -- 172.21.15.16:0/1585641036 shutdown_connections 2024-03-31T15:03:42.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.913+0000 7f1c97600640 1 -- 172.21.15.16:0/1585641036 >> 172.21.15.16:0/1585641036 conn(0x7f1c900b1d30 msgr2=0x7f1c90162230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:03:42.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.913+0000 7efc10800640 1 -- 172.21.15.16:0/3781439136 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efc080b35d0 msgr2=0x7efc081d7d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:42.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.913+0000 7efc10800640 1 --2- 172.21.15.16:0/3781439136 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efc080b35d0 0x7efc081d7d10 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7efbfc00e810 tx=0x7efbfc00ecd0 comp rx=0 tx=0).stop 2024-03-31T15:03:42.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.913+0000 7efc0e800640 1 -- 172.21.15.16:0/3781439136 reap_dead start 2024-03-31T15:03:42.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.917+0000 7f1c97600640 1 -- 172.21.15.16:0/1585641036 shutdown_connections 2024-03-31T15:03:42.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.917+0000 7f1c97600640 1 -- 172.21.15.16:0/1585641036 wait complete. 2024-03-31T15:03:42.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.917+0000 7efc10800640 1 -- 172.21.15.16:0/3781439136 shutdown_connections 2024-03-31T15:03:42.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.917+0000 7efc10800640 1 -- 172.21.15.16:0/3781439136 >> 172.21.15.16:0/3781439136 conn(0x7efc080d1020 msgr2=0x7efc0806c4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:03:42.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.917+0000 7efc10800640 1 -- 172.21.15.16:0/3781439136 shutdown_connections 2024-03-31T15:03:42.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.917+0000 7efc10800640 1 -- 172.21.15.16:0/3781439136 wait complete. 2024-03-31T15:03:42.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.917+0000 7fcfc1400640 1 -- 172.21.15.16:0/1303149306 --> [v2:172.21.15.16:6809/3360352974,v1:172.21.15.16:6812/3360352974] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fcfbc0cfd20 con 0x7fcf9c0705a0 2024-03-31T15:03:42.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.921+0000 7fcfb2a00640 1 -- 172.21.15.16:0/1303149306 <== osd.3 v2:172.21.15.16:6809/3360352974 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fcfbc0cfd20 con 0x7fcf9c0705a0 2024-03-31T15:03:42.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.921+0000 7fcfa7e00640 1 -- 172.21.15.16:0/1303149306 >> [v2:172.21.15.16:6809/3360352974,v1:172.21.15.16:6812/3360352974] conn(0x7fcf9c0705a0 msgr2=0x7fcf9c0729c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:42.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.921+0000 7fcfa7e00640 1 --2- 172.21.15.16:0/1303149306 >> [v2:172.21.15.16:6809/3360352974,v1:172.21.15.16:6812/3360352974] conn(0x7fcf9c0705a0 0x7fcf9c0729c0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:42.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.921+0000 7fcfa7e00640 1 -- 172.21.15.16:0/1303149306 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fcf9c068380 msgr2=0x7fcf9c06a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:42.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.921+0000 7fcfa7e00640 1 --2- 172.21.15.16:0/1303149306 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fcf9c068380 0x7fcf9c06a840 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fcfbc154520 tx=0x7fcfac008040 comp rx=0 tx=0).stop 2024-03-31T15:03:42.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.921+0000 7fcfa7e00640 1 -- 172.21.15.16:0/1303149306 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fcfbc0b48d0 msgr2=0x7fcfbc15ed50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:42.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.921+0000 7fcfa7e00640 1 --2- 172.21.15.16:0/1303149306 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fcfbc0b48d0 0x7fcfbc15ed50 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fcfa801ab30 tx=0x7fcfa801acc0 comp rx=0 tx=0).stop 2024-03-31T15:03:42.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.921+0000 7fcfbb200640 1 -- 172.21.15.16:0/1303149306 reap_dead start 2024-03-31T15:03:42.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.921+0000 7fcfa7e00640 1 -- 172.21.15.16:0/1303149306 shutdown_connections 2024-03-31T15:03:42.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.921+0000 7fcfa7e00640 1 -- 172.21.15.16:0/1303149306 >> 172.21.15.16:0/1303149306 conn(0x7fcfbc06b620 msgr2=0x7fcfbc0b1980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:03:42.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.921+0000 7fcfa7e00640 1 -- 172.21.15.16:0/1303149306 shutdown_connections 2024-03-31T15:03:42.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.925+0000 7fcfa7e00640 1 -- 172.21.15.16:0/1303149306 wait complete. 2024-03-31T15:03:42.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.925+0000 7f4badc00640 1 -- 172.21.15.16:0/1515004585 --> [v2:172.21.15.89:6800/1856844856,v1:172.21.15.89:6801/1856844856] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f4ba8069950 con 0x7f4b88070680 2024-03-31T15:03:42.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.925+0000 7f5ae8000640 1 -- 172.21.15.16:0/2669765298 --> [v2:172.21.15.89:6808/1796778630,v1:172.21.15.89:6811/1796778630] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f5ae0157670 con 0x7f5ac0070670 2024-03-31T15:03:42.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.925+0000 7f4b9ea00640 1 -- 172.21.15.16:0/1515004585 <== osd.4 v2:172.21.15.89:6800/1856844856 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f4ba8069950 con 0x7f4b88070680 2024-03-31T15:03:42.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.925+0000 7f5add600640 1 -- 172.21.15.16:0/2669765298 <== osd.7 v2:172.21.15.89:6808/1796778630 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f5ae0157670 con 0x7f5ac0070670 2024-03-31T15:03:42.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.925+0000 7f5ad2a00640 1 -- 172.21.15.16:0/2669765298 >> [v2:172.21.15.89:6808/1796778630,v1:172.21.15.89:6811/1796778630] conn(0x7f5ac0070670 msgr2=0x7f5ac0072a90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:42.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.925+0000 7f5ad2a00640 1 --2- 172.21.15.16:0/2669765298 >> [v2:172.21.15.89:6808/1796778630,v1:172.21.15.89:6811/1796778630] conn(0x7f5ac0070670 0x7f5ac0072a90 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:42.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.925+0000 7f5ad2a00640 1 -- 172.21.15.16:0/2669765298 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f5ac0068450 msgr2=0x7f5ac006a910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:42.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.925+0000 7f5ad2a00640 1 --2- 172.21.15.16:0/2669765298 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f5ac0068450 0x7f5ac006a910 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f5acc000f70 tx=0x7f5acc007ef0 comp rx=0 tx=0).stop 2024-03-31T15:03:42.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.925+0000 7f5ad2a00640 1 -- 172.21.15.16:0/2669765298 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5ae015a660 msgr2=0x7f5ae0158cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:42.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.925+0000 7f5ad2a00640 1 --2- 172.21.15.16:0/2669765298 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5ae015a660 0x7f5ae0158cb0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f5ad80ad000 tx=0x7f5ad801a540 comp rx=0 tx=0).stop 2024-03-31T15:03:42.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.925+0000 7f5ae6000640 1 -- 172.21.15.16:0/2669765298 reap_dead start 2024-03-31T15:03:42.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.925+0000 7f5ad2a00640 1 -- 172.21.15.16:0/2669765298 shutdown_connections 2024-03-31T15:03:42.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.925+0000 7f5ad2a00640 1 -- 172.21.15.16:0/2669765298 >> 172.21.15.16:0/2669765298 conn(0x7f5ae00b1d30 msgr2=0x7f5ae0156ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:03:42.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.925+0000 7f4badc00640 1 -- 172.21.15.16:0/1515004585 >> [v2:172.21.15.89:6800/1856844856,v1:172.21.15.89:6801/1856844856] conn(0x7f4b88070680 msgr2=0x7f4b88072aa0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:42.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.925+0000 7f4badc00640 1 --2- 172.21.15.16:0/1515004585 >> [v2:172.21.15.89:6800/1856844856,v1:172.21.15.89:6801/1856844856] conn(0x7f4b88070680 0x7f4b88072aa0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:42.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.925+0000 7f5ad2a00640 1 -- 172.21.15.16:0/2669765298 shutdown_connections 2024-03-31T15:03:42.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.925+0000 7f4badc00640 1 -- 172.21.15.16:0/1515004585 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f4b88068400 msgr2=0x7f4b8806a8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:42.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.925+0000 7f4badc00640 1 --2- 172.21.15.16:0/1515004585 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f4b88068400 0x7f4b8806a8c0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f4ba81e8080 tx=0x7f4b94007e90 comp rx=0 tx=0).stop 2024-03-31T15:03:42.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.925+0000 7f4badc00640 1 -- 172.21.15.16:0/1515004585 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f4ba80b35d0 msgr2=0x7f4ba81e74c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:42.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.925+0000 7f4badc00640 1 --2- 172.21.15.16:0/1515004585 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f4ba80b35d0 0x7f4ba81e74c0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f4ba000c710 tx=0x7f4ba000cbd0 comp rx=0 tx=0).stop 2024-03-31T15:03:42.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.925+0000 7f5ad2a00640 1 -- 172.21.15.16:0/2669765298 wait complete. 2024-03-31T15:03:42.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.925+0000 7f4ba7400640 1 -- 172.21.15.16:0/1515004585 reap_dead start 2024-03-31T15:03:42.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.925+0000 7f4badc00640 1 -- 172.21.15.16:0/1515004585 shutdown_connections 2024-03-31T15:03:42.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.925+0000 7f4badc00640 1 -- 172.21.15.16:0/1515004585 >> 172.21.15.16:0/1515004585 conn(0x7f4ba80d1020 msgr2=0x7f4ba80b8820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:03:42.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.925+0000 7f4badc00640 1 -- 172.21.15.16:0/1515004585 shutdown_connections 2024-03-31T15:03:42.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.925+0000 7f4badc00640 1 -- 172.21.15.16:0/1515004585 wait complete. 2024-03-31T15:03:42.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.929+0000 7fb4fb600640 1 -- 172.21.15.16:0/1482123868 --> [v2:172.21.15.89:6810/50395527,v1:172.21.15.89:6814/50395527] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fb4f41518e0 con 0x7fb4d4070570 2024-03-31T15:03:42.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.929+0000 7fb4f0c00640 1 -- 172.21.15.16:0/1482123868 <== osd.5 v2:172.21.15.89:6810/50395527 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fb4f41518e0 con 0x7fb4d4070570 2024-03-31T15:03:42.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.929+0000 7fb4fb600640 1 -- 172.21.15.16:0/1482123868 >> [v2:172.21.15.89:6810/50395527,v1:172.21.15.89:6814/50395527] conn(0x7fb4d4070570 msgr2=0x7fb4d4072990 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:42.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.929+0000 7fb4fb600640 1 --2- 172.21.15.16:0/1482123868 >> [v2:172.21.15.89:6810/50395527,v1:172.21.15.89:6814/50395527] conn(0x7fb4d4070570 0x7fb4d4072990 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:42.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.929+0000 7fb4fb600640 1 -- 172.21.15.16:0/1482123868 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fb4d40683d0 msgr2=0x7fb4d406a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:42.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.929+0000 7fb4fb600640 1 --2- 172.21.15.16:0/1482123868 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fb4d40683d0 0x7fb4d406a890 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fb4f40affc0 tx=0x7fb4ec008040 comp rx=0 tx=0).stop 2024-03-31T15:03:42.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.929+0000 7fb4fb600640 1 -- 172.21.15.16:0/1482123868 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb4f40af730 msgr2=0x7fb4f415c580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:42.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.929+0000 7fb4fb600640 1 --2- 172.21.15.16:0/1482123868 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb4f40af730 0x7fb4f415c580 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fb4f4069180 tx=0x7fb4e400ce80 comp rx=0 tx=0).stop 2024-03-31T15:03:42.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.929+0000 7fb4f9600640 1 -- 172.21.15.16:0/1482123868 reap_dead start 2024-03-31T15:03:42.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.933+0000 7fb4fb600640 1 -- 172.21.15.16:0/1482123868 shutdown_connections 2024-03-31T15:03:42.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.933+0000 7fb4fb600640 1 -- 172.21.15.16:0/1482123868 >> 172.21.15.16:0/1482123868 conn(0x7fb4f40b2040 msgr2=0x7fb4f406dbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:03:42.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.933+0000 7fb4fb600640 1 -- 172.21.15.16:0/1482123868 shutdown_connections 2024-03-31T15:03:42.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.937+0000 7fb4fb600640 1 -- 172.21.15.16:0/1482123868 wait complete. 2024-03-31T15:03:42.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.937+0000 7ff570c00640 1 -- 172.21.15.16:0/2245740762 --> [v2:172.21.15.89:6809/3470471752,v1:172.21.15.89:6812/3470471752] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7ff544002de0 con 0x7ff54c074a00 2024-03-31T15:03:42.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.937+0000 7ff562000640 1 -- 172.21.15.16:0/2245740762 <== osd.6 v2:172.21.15.89:6809/3470471752 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7ff544002de0 con 0x7ff54c074a00 2024-03-31T15:03:42.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.941+0000 7ff570c00640 1 -- 172.21.15.16:0/2245740762 >> [v2:172.21.15.89:6809/3470471752,v1:172.21.15.89:6812/3470471752] conn(0x7ff54c074a00 msgr2=0x7ff54c076e20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:42.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.941+0000 7ff570c00640 1 --2- 172.21.15.16:0/2245740762 >> [v2:172.21.15.89:6809/3470471752,v1:172.21.15.89:6812/3470471752] conn(0x7ff54c074a00 0x7ff54c076e20 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:42.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.941+0000 7ff570c00640 1 -- 172.21.15.16:0/2245740762 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7ff54c06c7e0 msgr2=0x7ff54c06eca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:42.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.941+0000 7ff570c00640 1 --2- 172.21.15.16:0/2245740762 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7ff54c06c7e0 0x7ff54c06eca0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7ff56c151090 tx=0x7ff558009040 comp rx=0 tx=0).stop 2024-03-31T15:03:42.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.941+0000 7ff570c00640 1 -- 172.21.15.16:0/2245740762 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff56c0b3030 msgr2=0x7ff56c14cd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:42.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.941+0000 7ff570c00640 1 --2- 172.21.15.16:0/2245740762 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff56c0b3030 0x7ff56c14cd50 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7ff55c01b3e0 tx=0x7ff55c01a550 comp rx=0 tx=0).stop 2024-03-31T15:03:42.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.941+0000 7ff56a800640 1 -- 172.21.15.16:0/2245740762 reap_dead start 2024-03-31T15:03:42.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.941+0000 7ff570c00640 1 -- 172.21.15.16:0/2245740762 shutdown_connections 2024-03-31T15:03:42.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.941+0000 7ff570c00640 1 -- 172.21.15.16:0/2245740762 >> 172.21.15.16:0/2245740762 conn(0x7ff56c0b1d30 msgr2=0x7ff56c0b12c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:03:42.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.945+0000 7ff570c00640 1 -- 172.21.15.16:0/2245740762 shutdown_connections 2024-03-31T15:03:42.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:42.945+0000 7ff570c00640 1 -- 172.21.15.16:0/2245740762 wait complete. 2024-03-31T15:03:42.951 INFO:teuthology.orchestra.run.smithi016.stdout:60129542152 2024-03-31T15:03:42.951 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-03-31T15:03:42.956 INFO:teuthology.orchestra.run.smithi016.stdout:60129542152 2024-03-31T15:03:42.956 INFO:teuthology.orchestra.run.smithi016.stdout:64424509445 2024-03-31T15:03:42.957 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-03-31T15:03:42.957 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-03-31T15:03:42.962 INFO:teuthology.orchestra.run.smithi016.stdout:60129542152 2024-03-31T15:03:42.962 INFO:teuthology.orchestra.run.smithi016.stdout:64424509446 2024-03-31T15:03:42.963 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-03-31T15:03:42.963 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-03-31T15:03:42.965 INFO:teuthology.orchestra.run.smithi016.stdout:64424509447 2024-03-31T15:03:42.965 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-03-31T15:03:42.974 INFO:teuthology.orchestra.run.smithi016.stdout:64424509448 2024-03-31T15:03:42.974 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-03-31T15:03:42.980 INFO:teuthology.orchestra.run.smithi016.stdout:60129542152 2024-03-31T15:03:42.980 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-03-31T15:03:43.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.201+0000 7f183f000640 1 Processor -- start 2024-03-31T15:03:43.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.201+0000 7f183f000640 1 -- start start 2024-03-31T15:03:43.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.201+0000 7f183f000640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f1838158b50 0x7f18380ae7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:43.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.201+0000 7f183f000640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1838158f30 con 0x7f18380aed30 2024-03-31T15:03:43.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.201+0000 7f183f000640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f183815c240 con 0x7f1838072b70 2024-03-31T15:03:43.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.201+0000 7f183f000640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f18380b3fb0 con 0x7f1838158b50 2024-03-31T15:03:43.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.201+0000 7f183d200640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f18380aed30 0x7f1838072460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:33998/0 (socket says 172.21.15.16:33998) 2024-03-31T15:03:43.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.201+0000 7f183d200640 1 -- 172.21.15.16:0/1650915637 learned_addr learned my addr 172.21.15.16:0/1650915637 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:03:43.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.201+0000 7f1837e00640 1 -- 172.21.15.16:0/1650915637 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2881936991 0 0) 0x7f1838158f30 con 0x7f18380aed30 2024-03-31T15:03:43.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.205+0000 7f1837e00640 1 -- 172.21.15.16:0/1650915637 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1820003530 con 0x7f18380aed30 2024-03-31T15:03:43.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.205+0000 7f183dc00640 1 --2- 172.21.15.16:0/1650915637 >> v2:172.21.15.89:3300/0 conn(0x7f1838158b50 0x7f18380ae7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:43.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.205+0000 7f1837e00640 1 -- 172.21.15.16:0/1650915637 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2900972852 0 0) 0x7f1820003530 con 0x7f18380aed30 2024-03-31T15:03:43.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.205+0000 7f1837e00640 1 -- 172.21.15.16:0/1650915637 >> v2:172.21.15.89:3300/0 conn(0x7f1838158b50 msgr2=0x7f18380ae7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:03:43.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.205+0000 7f1837e00640 1 --2- 172.21.15.16:0/1650915637 >> v2:172.21.15.89:3300/0 conn(0x7f1838158b50 0x7f18380ae7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:43.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.205+0000 7f1837e00640 1 -- 172.21.15.16:0/1650915637 >> v1:172.21.15.89:6789/0 conn(0x7f1838072b70 legacy=0x7f18380b4a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:43.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.205+0000 7f1837e00640 1 -- 172.21.15.16:0/1650915637 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18380b4290 con 0x7f18380aed30 2024-03-31T15:03:43.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.205+0000 7f1837e00640 1 -- 172.21.15.16:0/1650915637 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 214807623 0 0) 0x7f1824001680 con 0x7f18380aed30 2024-03-31T15:03:43.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.205+0000 7f1837e00640 1 -- 172.21.15.16:0/1650915637 >> v1:172.21.15.16:6789/0 conn(0x7f18380aed30 legacy=0x7f1838072460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:43.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.205+0000 7f1837e00640 1 --2- 172.21.15.16:0/1650915637 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f1820003a10 0x7f1820003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:43.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.205+0000 7f1837e00640 1 --2- 172.21.15.16:0/1650915637 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1820004370 0x7f1820004830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:43.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.205+0000 7f1837e00640 1 --2- 172.21.15.16:0/1650915637 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1820005030 0x7f182000ffe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:43.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.205+0000 7f1837e00640 1 -- 172.21.15.16:0/1650915637 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f18380b3fb0 con 0x7f1820004370 2024-03-31T15:03:43.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.205+0000 7f1837e00640 1 -- 172.21.15.16:0/1650915637 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f1820010580 con 0x7f1820005030 2024-03-31T15:03:43.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.205+0000 7f1837e00640 1 -- 172.21.15.16:0/1650915637 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f1820010780 con 0x7f1820003a10 2024-03-31T15:03:43.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.205+0000 7f183dc00640 1 --2- 172.21.15.16:0/1650915637 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f1820003a10 0x7f1820003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:43.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.205+0000 7f183e600640 1 --2- 172.21.15.16:0/1650915637 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1820005030 0x7f182000ffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:43.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.205+0000 7f183d200640 1 --2- 172.21.15.16:0/1650915637 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1820004370 0x7f1820004830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:43.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.205+0000 7f183dc00640 1 -- 172.21.15.16:0/1650915637 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1820005030 msgr2=0x7f182000ffe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:03:43.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.205+0000 7f183dc00640 1 --2- 172.21.15.16:0/1650915637 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1820005030 0x7f182000ffe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:43.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.205+0000 7f183dc00640 1 -- 172.21.15.16:0/1650915637 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1820004370 msgr2=0x7f1820004830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:03:43.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.205+0000 7f183dc00640 1 --2- 172.21.15.16:0/1650915637 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1820004370 0x7f1820004830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:43.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.205+0000 7f183dc00640 1 -- 172.21.15.16:0/1650915637 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1820003530 con 0x7f1820003a10 2024-03-31T15:03:43.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.205+0000 7f183e600640 1 --2- 172.21.15.16:0/1650915637 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1820005030 0x7f182000ffe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:03:43.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.205+0000 7f183e600640 1 -- 172.21.15.16:0/1650915637 reap_dead start 2024-03-31T15:03:43.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.205+0000 7f183d200640 1 --2- 172.21.15.16:0/1650915637 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1820004370 0x7f1820004830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:03:43.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.209+0000 7f183dc00640 1 --2- 172.21.15.16:0/1650915637 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f1820003a10 0x7f1820003e30 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f182c00ba30 tx=0x7f182c01c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=37f0feb7fe0285de server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:03:43.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.209+0000 7f1837e00640 1 -- 172.21.15.16:0/1650915637 <== mon.2 v2:172.21.15.89:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f182c0ae070 con 0x7f1820003a10 2024-03-31T15:03:43.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.209+0000 7f1837e00640 1 -- 172.21.15.16:0/1650915637 <== mon.2 v2:172.21.15.89:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f182c0a9680 con 0x7f1820003a10 2024-03-31T15:03:43.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.209+0000 7f1837e00640 1 -- 172.21.15.16:0/1650915637 <== mon.2 v2:172.21.15.89:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f182c00b600 con 0x7f1820003a10 2024-03-31T15:03:43.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.205+0000 7f7b74800640 1 Processor -- start 2024-03-31T15:03:43.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.205+0000 7f7b74800640 1 -- start start 2024-03-31T15:03:43.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.209+0000 7f7b74800640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f7b6c15a4e0 0x7f7b6c0b1cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:43.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.209+0000 7f7b74800640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f7b6c0ce8b0 0x7f7b6c0b2200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:43.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.209+0000 7f7b74800640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7b6c0cee00 con 0x7f7b6c0b34c0 2024-03-31T15:03:43.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.209+0000 7f7b74800640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f7b6c0b3030 con 0x7f7b6c15a4e0 2024-03-31T15:03:43.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.209+0000 7f7b74800640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f7b6c0b50d0 con 0x7f7b6c0ce8b0 2024-03-31T15:03:43.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.209+0000 7f7b72800640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f7b6c0ce8b0 0x7f7b6c0b2200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:43.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.209+0000 7f7b72800640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f7b6c0ce8b0 0x7f7b6c0b2200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3301/0 says I am v2:172.21.15.16:41580/0 (socket says 172.21.15.16:41580) 2024-03-31T15:03:43.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.209+0000 7f7b72800640 1 -- 172.21.15.16:0/3426571480 learned_addr learned my addr 172.21.15.16:0/3426571480 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:03:43.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.209+0000 7f7b72800640 1 -- 172.21.15.16:0/3426571480 >> v2:172.21.15.16:3300/0 conn(0x7f7b6c15a4e0 msgr2=0x7f7b6c0b1cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:03:43.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.209+0000 7f7b70a00640 1 -- 172.21.15.16:0/3426571480 <== mon.1 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2876040392 0 0) 0x7f7b6c0cee00 con 0x7f7b6c0b34c0 2024-03-31T15:03:43.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.209+0000 7f7b72800640 1 --2- 172.21.15.16:0/3426571480 >> v2:172.21.15.16:3300/0 conn(0x7f7b6c15a4e0 0x7f7b6c0b1cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:43.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.209+0000 7f7b72800640 1 -- 172.21.15.16:0/3426571480 >> v1:172.21.15.89:6789/0 conn(0x7f7b6c0b34c0 legacy=0x7f7b6c0cf9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:43.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.209+0000 7f7b72800640 1 -- 172.21.15.16:0/3426571480 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b6c0b53b0 con 0x7f7b6c0ce8b0 2024-03-31T15:03:43.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.213+0000 7ff998400640 1 Processor -- start 2024-03-31T15:03:43.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.213+0000 7f183f000640 1 -- 172.21.15.16:0/1650915637 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f1820003a10 msgr2=0x7f1820003e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:43.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.213+0000 7f183f000640 1 --2- 172.21.15.16:0/1650915637 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f1820003a10 0x7f1820003e30 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f182c00ba30 tx=0x7f182c01c6a0 comp rx=0 tx=0).stop 2024-03-31T15:03:43.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.213+0000 7f0294a00640 1 Processor -- start 2024-03-31T15:03:43.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.213+0000 7f183f000640 1 -- 172.21.15.16:0/1650915637 shutdown_connections 2024-03-31T15:03:43.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.213+0000 7f183f000640 1 --2- 172.21.15.16:0/1650915637 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f1820003a10 0x7f1820003e30 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:43.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.213+0000 7f183f000640 1 -- 172.21.15.16:0/1650915637 >> 172.21.15.16:0/1650915637 conn(0x7f18380b2b30 msgr2=0x7f18380b2f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:03:43.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.213+0000 7f0294a00640 1 -- start start 2024-03-31T15:03:43.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.213+0000 7ff998400640 1 -- start start 2024-03-31T15:03:43.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.213+0000 7f183f000640 1 -- 172.21.15.16:0/1650915637 shutdown_connections 2024-03-31T15:03:43.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.213+0000 7f183f000640 1 -- 172.21.15.16:0/1650915637 wait complete. 2024-03-31T15:03:43.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.213+0000 7f0294a00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f02880070f0 0x7f02880e2060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:43.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.213+0000 7ff998400640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7ff9900cc840 0x7ff99016d970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:43.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.213+0000 7ff998400640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff990072f40 con 0x7ff9900ccfc0 2024-03-31T15:03:43.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.213+0000 7ff998400640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff9900b51e0 con 0x7ff990165dc0 2024-03-31T15:03:43.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.213+0000 7ff998400640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7ff990071500 con 0x7ff9900cc840 2024-03-31T15:03:43.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.213+0000 7ff995a00640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7ff990165dc0 0x7ff9900b48d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.16:49278/0 (socket says 172.21.15.16:49278) 2024-03-31T15:03:43.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.213+0000 7ff995a00640 1 -- 172.21.15.16:0/2423570522 learned_addr learned my addr 172.21.15.16:0/2423570522 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:03:43.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.213+0000 7f0294a00640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f02880e25a0 con 0x7f0288033fc0 2024-03-31T15:03:43.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.213+0000 7f0294a00640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f02880382e0 con 0x7f0288007df0 2024-03-31T15:03:43.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.213+0000 7f0294a00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f0288006950 con 0x7f02880070f0 2024-03-31T15:03:43.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.213+0000 7f028f400640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f02880070f0 0x7f02880e2060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:43.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.213+0000 7f028f400640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f02880070f0 0x7f02880e2060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3301/0 says I am v2:172.21.15.16:41588/0 (socket says 172.21.15.16:41588) 2024-03-31T15:03:43.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.213+0000 7f028f400640 1 -- 172.21.15.16:0/4175455110 learned_addr learned my addr 172.21.15.16:0/4175455110 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:03:43.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.213+0000 7f434ec00640 1 Processor -- start 2024-03-31T15:03:43.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.213+0000 7f434ec00640 1 -- start start 2024-03-31T15:03:43.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.217+0000 7f183f000640 1 Processor -- start 2024-03-31T15:03:43.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.217+0000 7ff996400640 1 --2- 172.21.15.16:0/2423570522 >> v2:172.21.15.89:3300/0 conn(0x7ff9900cc840 0x7ff99016d970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:43.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.217+0000 7fbf37600640 1 Processor -- start 2024-03-31T15:03:43.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.217+0000 7ff98fe00640 1 -- 172.21.15.16:0/2423570522 <== mon.1 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1791174590 0 0) 0x7ff9900b51e0 con 0x7ff990165dc0 2024-03-31T15:03:43.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.217+0000 7ff98fe00640 1 -- 172.21.15.16:0/2423570522 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff978003530 con 0x7ff990165dc0 2024-03-31T15:03:43.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.217+0000 7f183f000640 1 -- start start 2024-03-31T15:03:43.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.217+0000 7fbf37600640 1 -- start start 2024-03-31T15:03:43.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.217+0000 7ff98fe00640 1 -- 172.21.15.16:0/2423570522 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2588684376 0 0) 0x7ff990072f40 con 0x7ff9900ccfc0 2024-03-31T15:03:43.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.217+0000 7ff98fe00640 1 -- 172.21.15.16:0/2423570522 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff9900b51e0 con 0x7ff9900ccfc0 2024-03-31T15:03:43.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.217+0000 7ff98fe00640 1 -- 172.21.15.16:0/2423570522 <== mon.1 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3115366280 0 0) 0x7ff978003530 con 0x7ff990165dc0 2024-03-31T15:03:43.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.217+0000 7ff98fe00640 1 -- 172.21.15.16:0/2423570522 >> v2:172.21.15.89:3300/0 conn(0x7ff9900cc840 msgr2=0x7ff99016d970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:03:43.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.217+0000 7ff98fe00640 1 --2- 172.21.15.16:0/2423570522 >> v2:172.21.15.89:3300/0 conn(0x7ff9900cc840 0x7ff99016d970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:43.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.217+0000 7ff98fe00640 1 -- 172.21.15.16:0/2423570522 >> v1:172.21.15.16:6789/0 conn(0x7ff9900ccfc0 legacy=0x7ff9900cc130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:43.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.217+0000 7ff98fe00640 1 -- 172.21.15.16:0/2423570522 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff990073200 con 0x7ff990165dc0 2024-03-31T15:03:43.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.217+0000 7ff98fe00640 1 -- 172.21.15.16:0/2423570522 <== mon.1 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 214807623 0 0) 0x7ff97c001680 con 0x7ff990165dc0 2024-03-31T15:03:43.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.217+0000 7ff98fe00640 1 -- 172.21.15.16:0/2423570522 >> v1:172.21.15.89:6789/0 conn(0x7ff990165dc0 legacy=0x7ff9900b48d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:43.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.217+0000 7ff98fe00640 1 --2- 172.21.15.16:0/2423570522 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff978003b50 0x7ff978003f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:43.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.217+0000 7ff98fe00640 1 --2- 172.21.15.16:0/2423570522 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff978004810 0x7ff978004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:43.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.217+0000 7ff98fe00640 1 --2- 172.21.15.16:0/2423570522 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff978005220 0x7ff97800b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:43.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.217+0000 7ff98fe00640 1 -- 172.21.15.16:0/2423570522 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7ff97800c120 con 0x7ff978003b50 2024-03-31T15:03:43.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.217+0000 7ff98fe00640 1 -- 172.21.15.16:0/2423570522 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7ff97800c2d0 con 0x7ff978005220 2024-03-31T15:03:43.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.217+0000 7ff98fe00640 1 -- 172.21.15.16:0/2423570522 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7ff97800c4d0 con 0x7ff978004810 2024-03-31T15:03:43.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.217+0000 7ff995000640 1 --2- 172.21.15.16:0/2423570522 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff978004810 0x7ff978004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:43.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.217+0000 7ff995a00640 1 --2- 172.21.15.16:0/2423570522 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff978003b50 0x7ff978003f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:43.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.217+0000 7f434ec00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f4348162ae0 0x7f43480cc5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:43.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.217+0000 7f434ec00640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f43481666e0 con 0x7f43480d0970 2024-03-31T15:03:43.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.217+0000 7f434ec00640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f43481669c0 con 0x7f434806f580 2024-03-31T15:03:43.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.217+0000 7f434ec00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f4348162ec0 con 0x7f4348162ae0 2024-03-31T15:03:43.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.217+0000 7f183f000640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1820003a10 0x7f18380cfeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:43.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.217+0000 7f183f000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f18381f8780 0x7f18380b1c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:43.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.217+0000 7f183f000640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f18381f8c00 0x7f18380b21c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:43.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.217+0000 7f183f000640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f18380aeb80 con 0x7f18381f8780 2024-03-31T15:03:43.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.217+0000 7f183f000640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f18380aea00 con 0x7f1820003a10 2024-03-31T15:03:43.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.217+0000 7f183f000640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f1838072460 con 0x7f18381f8c00 2024-03-31T15:03:43.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.217+0000 7fbf37600640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fbf3015e750 0x7fbf300af5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:43.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.217+0000 7fbf37600640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fbf300cca10 0x7fbf300cce80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:43.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.217+0000 7f7b72800640 1 --2- 172.21.15.16:0/3426571480 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f7b6c0ce8b0 0x7f7b6c0b2200 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f7b68009e50 tx=0x7f7b6801ca30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=66f56ec5daa2c771 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:03:43.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.217+0000 7f7b70a00640 1 -- 172.21.15.16:0/3426571480 <== mon.2 v2:172.21.15.89:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b680a5920 con 0x7f7b6c0ce8b0 2024-03-31T15:03:43.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.217+0000 7f7b70a00640 1 -- 172.21.15.16:0/3426571480 <== mon.2 v2:172.21.15.89:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7b680a5ac0 con 0x7f7b6c0ce8b0 2024-03-31T15:03:43.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.217+0000 7f7b70a00640 1 -- 172.21.15.16:0/3426571480 <== mon.2 v2:172.21.15.89:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b680ad3f0 con 0x7f7b6c0ce8b0 2024-03-31T15:03:43.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.217+0000 7f7b74800640 1 -- 172.21.15.16:0/3426571480 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f7b6c0ce8b0 msgr2=0x7f7b6c0b2200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:43.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.217+0000 7f7b74800640 1 --2- 172.21.15.16:0/3426571480 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f7b6c0ce8b0 0x7f7b6c0b2200 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f7b68009e50 tx=0x7f7b6801ca30 comp rx=0 tx=0).stop 2024-03-31T15:03:43.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.217+0000 7fbf2fe00640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fbf300ada20 0x7fbf30162ac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.16:49284/0 (socket says 172.21.15.16:49284) 2024-03-31T15:03:43.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.217+0000 7fbf35600640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fbf300cca10 0x7fbf300cce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:43.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.217+0000 7fbf35600640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fbf300cca10 0x7fbf300cce80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.16:36102/0 (socket says 172.21.15.16:36102) 2024-03-31T15:03:43.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.217+0000 7fbf37600640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbf300b4fe0 con 0x7fbf300ada20 2024-03-31T15:03:43.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.217+0000 7fbf37600640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7fbf300cd3c0 con 0x7fbf3015e750 2024-03-31T15:03:43.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.217+0000 7fbf37600640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7fbf300cf750 con 0x7fbf300cca10 2024-03-31T15:03:43.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.217+0000 7fbf2fe00640 1 -- 172.21.15.16:0/418449113 learned_addr learned my addr 172.21.15.16:0/418449113 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:03:43.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.217+0000 7fbf34c00640 1 --2- 172.21.15.16:0/418449113 >> v2:172.21.15.16:3300/0 conn(0x7fbf3015e750 0x7fbf300af5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:43.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.217+0000 7f183e600640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f18381f8c00 0x7f18380b21c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:43.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.217+0000 7f183e600640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f18381f8c00 0x7f18380b21c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3301/0 says I am v2:172.21.15.16:41616/0 (socket says 172.21.15.16:41616) 2024-03-31T15:03:43.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.217+0000 7f183e600640 1 -- 172.21.15.16:0/3318694499 learned_addr learned my addr 172.21.15.16:0/3318694499 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:03:43.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.221+0000 7f183dc00640 1 --2- 172.21.15.16:0/3318694499 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1820003a10 0x7f18380cfeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:43.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.221+0000 7f183e600640 1 -- 172.21.15.16:0/3318694499 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1820003a10 msgr2=0x7f18380cfeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:43.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.221+0000 7f183e600640 1 --2- 172.21.15.16:0/3318694499 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1820003a10 0x7f18380cfeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:43.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.221+0000 7f183e600640 1 -- 172.21.15.16:0/3318694499 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f18381f8780 msgr2=0x7f18380b1c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:43.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.221+0000 7f183e600640 1 --2- 172.21.15.16:0/3318694499 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f18381f8780 0x7f18380b1c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:43.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.221+0000 7f183e600640 1 -- 172.21.15.16:0/3318694499 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f183815c730 con 0x7f18381f8c00 2024-03-31T15:03:43.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.221+0000 7fbf35600640 1 -- 172.21.15.16:0/418449113 >> v2:172.21.15.16:3300/0 conn(0x7fbf3015e750 msgr2=0x7fbf300af5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:03:43.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.221+0000 7fbf35600640 1 --2- 172.21.15.16:0/418449113 >> v2:172.21.15.16:3300/0 conn(0x7fbf3015e750 0x7fbf300af5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:43.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.221+0000 7fbf35600640 1 -- 172.21.15.16:0/418449113 >> v1:172.21.15.89:6789/0 conn(0x7fbf300ada20 legacy=0x7fbf30162ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:43.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.221+0000 7fbf35600640 1 -- 172.21.15.16:0/418449113 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf3015eb30 con 0x7fbf300cca10 2024-03-31T15:03:43.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.217+0000 7ff995000640 1 -- 172.21.15.16:0/2423570522 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff978005220 msgr2=0x7ff97800b9d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-31T15:03:43.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.217+0000 7ff995000640 1 --2- 172.21.15.16:0/2423570522 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff978005220 0x7ff97800b9d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:43.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.217+0000 7ff995000640 1 -- 172.21.15.16:0/2423570522 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff978003b50 msgr2=0x7ff978003f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:03:43.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.217+0000 7ff995000640 1 --2- 172.21.15.16:0/2423570522 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff978003b50 0x7ff978003f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:43.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.217+0000 7ff995000640 1 -- 172.21.15.16:0/2423570522 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff990073200 con 0x7ff978004810 2024-03-31T15:03:43.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.217+0000 7ff996400640 1 -- 172.21.15.16:0/2423570522 reap_dead start 2024-03-31T15:03:43.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.213+0000 7f028f400640 1 -- 172.21.15.16:0/4175455110 >> v1:172.21.15.89:6789/0 conn(0x7f0288007df0 legacy=0x7f0288006240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:03:43.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.213+0000 7f028f400640 1 -- 172.21.15.16:0/4175455110 >> v1:172.21.15.16:6789/0 conn(0x7f0288033fc0 legacy=0x7f028803a7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:03:43.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.213+0000 7f028f400640 1 -- 172.21.15.16:0/4175455110 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02880385c0 con 0x7f02880070f0 2024-03-31T15:03:43.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.217+0000 7f4347e00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f4348162ae0 0x7f43480cc5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:43.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.217+0000 7f4347e00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f4348162ae0 0x7f43480cc5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3301/0 says I am v2:172.21.15.16:41614/0 (socket says 172.21.15.16:41614) 2024-03-31T15:03:43.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.217+0000 7f4347e00640 1 -- 172.21.15.16:0/3053532721 learned_addr learned my addr 172.21.15.16:0/3053532721 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:03:43.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.217+0000 7f4347e00640 1 -- 172.21.15.16:0/3053532721 >> v1:172.21.15.89:6789/0 conn(0x7f434806f580 legacy=0x7f43480d0260 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-31T15:03:43.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.217+0000 7f4347e00640 1 -- 172.21.15.16:0/3053532721 >> v1:172.21.15.16:6789/0 conn(0x7f43480d0970 legacy=0x7f43480ce3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:03:43.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.221+0000 7f4347e00640 1 -- 172.21.15.16:0/3053532721 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4348166ca0 con 0x7f4348162ae0 2024-03-31T15:03:43.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.221+0000 7f183e600640 1 --2- 172.21.15.16:0/3318694499 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f18381f8c00 0x7f18380b21c0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f183806dbf0 tx=0x7f183000ead0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:03:43.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.221+0000 7f7b74800640 1 -- 172.21.15.16:0/3426571480 shutdown_connections 2024-03-31T15:03:43.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.221+0000 7f7b74800640 1 --2- 172.21.15.16:0/3426571480 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f7b6c0ce8b0 0x7f7b6c0b2200 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:43.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.221+0000 7f7b74800640 1 --2- 172.21.15.16:0/3426571480 >> v2:172.21.15.16:3300/0 conn(0x7f7b6c15a4e0 0x7f7b6c0b1cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:43.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.221+0000 7f7b74800640 1 -- 172.21.15.16:0/3426571480 >> 172.21.15.16:0/3426571480 conn(0x7f7b6c06b620 msgr2=0x7f7b6c06ba30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:03:43.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.221+0000 7f1835600640 1 -- 172.21.15.16:0/3318694499 <== mon.2 v2:172.21.15.89:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1830003b90 con 0x7f18381f8c00 2024-03-31T15:03:43.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.221+0000 7f028f400640 1 --2- 172.21.15.16:0/4175455110 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f02880070f0 0x7f02880e2060 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f027c00dcc0 tx=0x7f027c020600 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cb8858b61ad227df server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:03:43.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.221+0000 7f1835600640 1 -- 172.21.15.16:0/3318694499 <== mon.2 v2:172.21.15.89:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1830018070 con 0x7f18381f8c00 2024-03-31T15:03:43.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.221+0000 7f1835600640 1 -- 172.21.15.16:0/3318694499 <== mon.2 v2:172.21.15.89:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18300050d0 con 0x7f18381f8c00 2024-03-31T15:03:43.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.221+0000 7f183f000640 1 -- 172.21.15.16:0/3318694499 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f183815ca10 con 0x7f18381f8c00 2024-03-31T15:03:43.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.221+0000 7f183f000640 1 -- 172.21.15.16:0/3318694499 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f18380ce620 con 0x7f18381f8c00 2024-03-31T15:03:43.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.221+0000 7f7b74800640 1 -- 172.21.15.16:0/3426571480 shutdown_connections 2024-03-31T15:03:43.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.221+0000 7f7b74800640 1 -- 172.21.15.16:0/3426571480 wait complete. 2024-03-31T15:03:43.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.221+0000 7f028e000640 1 -- 172.21.15.16:0/4175455110 <== mon.2 v2:172.21.15.89:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f027c00d0d0 con 0x7f02880070f0 2024-03-31T15:03:43.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.221+0000 7f028e000640 1 -- 172.21.15.16:0/4175455110 <== mon.2 v2:172.21.15.89:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f027c00d270 con 0x7f02880070f0 2024-03-31T15:03:43.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.221+0000 7f183f000640 1 -- 172.21.15.16:0/3318694499 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f183804f8c0 con 0x7f18381f8c00 2024-03-31T15:03:43.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.221+0000 7ff995000640 1 --2- 172.21.15.16:0/2423570522 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff978004810 0x7ff978004c90 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7ff984001970 tx=0x7ff98401c580 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8ae98366a5bfc929 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:03:43.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.221+0000 7f028e000640 1 -- 172.21.15.16:0/4175455110 <== mon.2 v2:172.21.15.89:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f027c0aa6e0 con 0x7f02880070f0 2024-03-31T15:03:43.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.221+0000 7ff98fe00640 1 -- 172.21.15.16:0/2423570522 <== mon.2 v2:172.21.15.89:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9840b0070 con 0x7ff978004810 2024-03-31T15:03:43.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.221+0000 7ff98fe00640 1 -- 172.21.15.16:0/2423570522 <== mon.2 v2:172.21.15.89:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff9840ab620 con 0x7ff978004810 2024-03-31T15:03:43.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.221+0000 7ff98fe00640 1 -- 172.21.15.16:0/2423570522 <== mon.2 v2:172.21.15.89:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff984001570 con 0x7ff978004810 2024-03-31T15:03:43.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.225+0000 7f0294a00640 1 -- 172.21.15.16:0/4175455110 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f02880070f0 msgr2=0x7f02880e2060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:43.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.225+0000 7f0294a00640 1 --2- 172.21.15.16:0/4175455110 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f02880070f0 0x7f02880e2060 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f027c00dcc0 tx=0x7f027c020600 comp rx=0 tx=0).stop 2024-03-31T15:03:43.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.225+0000 7ff998400640 1 -- 172.21.15.16:0/2423570522 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff978004810 msgr2=0x7ff978004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:43.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.225+0000 7ff998400640 1 --2- 172.21.15.16:0/2423570522 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff978004810 0x7ff978004c90 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7ff984001970 tx=0x7ff98401c580 comp rx=0 tx=0).stop 2024-03-31T15:03:43.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.225+0000 7ff998400640 1 -- 172.21.15.16:0/2423570522 shutdown_connections 2024-03-31T15:03:43.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.225+0000 7ff998400640 1 --2- 172.21.15.16:0/2423570522 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff978004810 0x7ff978004c90 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:43.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.225+0000 7ff998400640 1 -- 172.21.15.16:0/2423570522 >> 172.21.15.16:0/2423570522 conn(0x7ff99006b620 msgr2=0x7ff99006ba30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:03:43.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.225+0000 7f1835600640 1 -- 172.21.15.16:0/3318694499 <== mon.2 v2:172.21.15.89:3301/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f1830005270 con 0x7f18381f8c00 2024-03-31T15:03:43.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.225+0000 7fbf35600640 1 --2- 172.21.15.16:0/418449113 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbf300cca10 0x7fbf300cce80 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fbf300b2bc0 tx=0x7fbf1c01ec90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=65469ef2e9c1b9b9 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:03:43.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.225+0000 7ff998400640 1 -- 172.21.15.16:0/2423570522 shutdown_connections 2024-03-31T15:03:43.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.225+0000 7f0294a00640 1 -- 172.21.15.16:0/4175455110 shutdown_connections 2024-03-31T15:03:43.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.225+0000 7f0294a00640 1 --2- 172.21.15.16:0/4175455110 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f02880070f0 0x7f02880e2060 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:43.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.225+0000 7f0294a00640 1 -- 172.21.15.16:0/4175455110 >> 172.21.15.16:0/4175455110 conn(0x7f0288055360 msgr2=0x7f0288055770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:03:43.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.225+0000 7f4347e00640 1 --2- 172.21.15.16:0/3053532721 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f4348162ae0 0x7f43480cc5a0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f43380098a0 tx=0x7f433801c470 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4e72324a9cc00019 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:03:43.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.225+0000 7fbf2f400640 1 -- 172.21.15.16:0/418449113 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf1c0b5070 con 0x7fbf300cca10 2024-03-31T15:03:43.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.225+0000 7f1835600640 1 --2- 172.21.15.16:0/3318694499 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f1818068380 0x7f181806a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:43.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.225+0000 7f0294a00640 1 -- 172.21.15.16:0/4175455110 shutdown_connections 2024-03-31T15:03:43.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.225+0000 7f0294a00640 1 -- 172.21.15.16:0/4175455110 wait complete. 2024-03-31T15:03:43.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.225+0000 7f1835600640 1 -- 172.21.15.16:0/3318694499 <== mon.2 v2:172.21.15.89:3301/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f18300867d0 con 0x7f18381f8c00 2024-03-31T15:03:43.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.225+0000 7f183dc00640 1 --2- 172.21.15.16:0/3318694499 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f1818068380 0x7f181806a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:43.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.225+0000 7fbf2f400640 1 -- 172.21.15.16:0/418449113 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbf1c0a8970 con 0x7fbf300cca10 2024-03-31T15:03:43.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.225+0000 7f7b74800640 1 Processor -- start 2024-03-31T15:03:43.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.225+0000 7f7b74800640 1 -- start start 2024-03-31T15:03:43.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.225+0000 7ff998400640 1 -- 172.21.15.16:0/2423570522 wait complete. 2024-03-31T15:03:43.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.225+0000 7fbf2f400640 1 -- 172.21.15.16:0/418449113 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf1c0b5070 con 0x7fbf300cca10 2024-03-31T15:03:43.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.225+0000 7f183dc00640 1 --2- 172.21.15.16:0/3318694499 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f1818068380 0x7f181806a840 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f182c0b0000 tx=0x7f182c0ac040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:03:43.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.225+0000 7fbf37600640 1 -- 172.21.15.16:0/418449113 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbf300cca10 msgr2=0x7fbf300cce80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:43.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.225+0000 7fbf37600640 1 --2- 172.21.15.16:0/418449113 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbf300cca10 0x7fbf300cce80 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fbf300b2bc0 tx=0x7fbf1c01ec90 comp rx=0 tx=0).stop 2024-03-31T15:03:43.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.229+0000 7f0294a00640 1 Processor -- start 2024-03-31T15:03:43.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.229+0000 7fbf37600640 1 -- 172.21.15.16:0/418449113 shutdown_connections 2024-03-31T15:03:43.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.229+0000 7fbf37600640 1 --2- 172.21.15.16:0/418449113 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbf300cca10 0x7fbf300cce80 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:43.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.229+0000 7fbf37600640 1 --2- 172.21.15.16:0/418449113 >> v2:172.21.15.16:3300/0 conn(0x7fbf3015e750 0x7fbf300af5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:43.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.229+0000 7fbf37600640 1 -- 172.21.15.16:0/418449113 >> 172.21.15.16:0/418449113 conn(0x7fbf300b1d30 msgr2=0x7fbf300b2140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:03:43.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.229+0000 7f0294a00640 1 -- start start 2024-03-31T15:03:43.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.229+0000 7ff998400640 1 Processor -- start 2024-03-31T15:03:43.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.229+0000 7f7b74800640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7b6c0b34c0 0x7f7b6c160a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:43.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.229+0000 7f7b74800640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f7b6c0ce8b0 0x7f7b6c152780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:43.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.229+0000 7f7b74800640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7b6c15a4e0 0x7f7b6c151ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:43.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.229+0000 7f7b74800640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f7b6c06f770 con 0x7f7b6c0b34c0 2024-03-31T15:03:43.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.229+0000 7f7b74800640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f7b6c06f5f0 con 0x7f7b6c15a4e0 2024-03-31T15:03:43.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.229+0000 7f7b74800640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f7b6c1672b0 con 0x7f7b6c0ce8b0 2024-03-31T15:03:43.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.229+0000 7fbf37600640 1 -- 172.21.15.16:0/418449113 shutdown_connections 2024-03-31T15:03:43.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.229+0000 7f7b72800640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7b6c15a4e0 0x7f7b6c151ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:43.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.229+0000 7f7b72800640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7b6c15a4e0 0x7f7b6c151ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.16:36108/0 (socket says 172.21.15.16:36108) 2024-03-31T15:03:43.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.229+0000 7f7b72800640 1 -- 172.21.15.16:0/4009901049 learned_addr learned my addr 172.21.15.16:0/4009901049 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:03:43.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.229+0000 7f7b71e00640 1 --2- 172.21.15.16:0/4009901049 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7b6c0b34c0 0x7f7b6c160a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:43.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.229+0000 7f7b71400640 1 --2- 172.21.15.16:0/4009901049 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f7b6c0ce8b0 0x7f7b6c152780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:43.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.229+0000 7f1835600640 1 -- 172.21.15.16:0/3318694499 <== mon.2 v2:172.21.15.89:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f1830056cf0 con 0x7f18381f8c00 2024-03-31T15:03:43.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.229+0000 7fbf37600640 1 -- 172.21.15.16:0/418449113 wait complete. 2024-03-31T15:03:43.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.229+0000 7ff998400640 1 -- start start 2024-03-31T15:03:43.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.229+0000 7ff998400640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff978004810 0x7ff9901631a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:43.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.229+0000 7ff998400640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff99015cd60 0x7ff9901618d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:43.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.229+0000 7ff998400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff990163e30 0x7ff99014ce10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:43.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.229+0000 7ff998400640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7ff990166320 con 0x7ff990163e30 2024-03-31T15:03:43.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.229+0000 7ff998400640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7ff9901661a0 con 0x7ff99015cd60 2024-03-31T15:03:43.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.229+0000 7ff998400640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7ff9900cdea0 con 0x7ff978004810 2024-03-31T15:03:43.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.229+0000 7ff996400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff990163e30 0x7ff99014ce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:43.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.233+0000 7ff996400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff990163e30 0x7ff99014ce10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39108/0 (socket says 172.21.15.16:39108) 2024-03-31T15:03:43.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.233+0000 7ff996400640 1 -- 172.21.15.16:0/2040406316 learned_addr learned my addr 172.21.15.16:0/2040406316 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:03:43.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.229+0000 7f0294a00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f02880070f0 0x7f0288034cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:43.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.229+0000 7f0294a00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f0288007df0 0x7f02880de7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:43.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.229+0000 7f0294a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0288033fc0 0x7f0288036600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:43.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.229+0000 7f0294a00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f0288034800 con 0x7f0288033fc0 2024-03-31T15:03:43.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.229+0000 7f0294a00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f02880074d0 con 0x7f02880070f0 2024-03-31T15:03:43.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.229+0000 7f0294a00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f02880ed570 con 0x7f0288007df0 2024-03-31T15:03:43.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.229+0000 7f028f400640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f02880070f0 0x7f0288034cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:43.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.229+0000 7f028f400640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f02880070f0 0x7f0288034cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.16:36114/0 (socket says 172.21.15.16:36114) 2024-03-31T15:03:43.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.229+0000 7f028f400640 1 -- 172.21.15.16:0/3511530985 learned_addr learned my addr 172.21.15.16:0/3511530985 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:03:43.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.229+0000 7f028ea00640 1 --2- 172.21.15.16:0/3511530985 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f0288007df0 0x7f02880de7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:43.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.229+0000 7f028fe00640 1 --2- 172.21.15.16:0/3511530985 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0288033fc0 0x7f0288036600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:43.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.229+0000 7f028f400640 1 -- 172.21.15.16:0/3511530985 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f0288007df0 msgr2=0x7f02880de7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:43.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.229+0000 7f028f400640 1 --2- 172.21.15.16:0/3511530985 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f0288007df0 0x7f02880de7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:43.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.229+0000 7f028f400640 1 -- 172.21.15.16:0/3511530985 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0288033fc0 msgr2=0x7f0288036600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:43.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.229+0000 7f028f400640 1 --2- 172.21.15.16:0/3511530985 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0288033fc0 0x7f0288036600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:43.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.229+0000 7f028f400640 1 -- 172.21.15.16:0/3511530985 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02880352a0 con 0x7f02880070f0 2024-03-31T15:03:43.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.233+0000 7f028f400640 1 --2- 172.21.15.16:0/3511530985 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f02880070f0 0x7f0288034cc0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f027c020ea0 tx=0x7f027c01d3a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:03:43.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.229+0000 7f7b72800640 1 -- 172.21.15.16:0/4009901049 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f7b6c0ce8b0 msgr2=0x7f7b6c152780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:43.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.229+0000 7f7b72800640 1 --2- 172.21.15.16:0/4009901049 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f7b6c0ce8b0 0x7f7b6c152780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:43.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.229+0000 7f7b72800640 1 -- 172.21.15.16:0/4009901049 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7b6c0b34c0 msgr2=0x7f7b6c160a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:43.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.229+0000 7f7b72800640 1 --2- 172.21.15.16:0/4009901049 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7b6c0b34c0 0x7f7b6c160a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:43.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.229+0000 7f7b72800640 1 -- 172.21.15.16:0/4009901049 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b6c152cc0 con 0x7f7b6c15a4e0 2024-03-31T15:03:43.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.229+0000 7f7b72800640 1 --2- 172.21.15.16:0/4009901049 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7b6c15a4e0 0x7f7b6c151ad0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f7b68009e50 tx=0x7f7b680197d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:03:43.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.229+0000 7f7b66000640 1 -- 172.21.15.16:0/4009901049 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b680bc020 con 0x7f7b6c15a4e0 2024-03-31T15:03:43.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.229+0000 7f7b66000640 1 -- 172.21.15.16:0/4009901049 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7b68007da0 con 0x7f7b6c15a4e0 2024-03-31T15:03:43.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.229+0000 7f7b74800640 1 -- 172.21.15.16:0/4009901049 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7b6c1fb680 con 0x7f7b6c15a4e0 2024-03-31T15:03:43.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.229+0000 7f7b66000640 1 -- 172.21.15.16:0/4009901049 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b680a5920 con 0x7f7b6c15a4e0 2024-03-31T15:03:43.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.229+0000 7f4346a00640 1 -- 172.21.15.16:0/3053532721 <== mon.2 v2:172.21.15.89:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43380b0070 con 0x7f4348162ae0 2024-03-31T15:03:43.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.229+0000 7f4346a00640 1 -- 172.21.15.16:0/3053532721 <== mon.2 v2:172.21.15.89:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f43380ab600 con 0x7f4348162ae0 2024-03-31T15:03:43.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.229+0000 7f4346a00640 1 -- 172.21.15.16:0/3053532721 <== mon.2 v2:172.21.15.89:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43380ab9d0 con 0x7f4348162ae0 2024-03-31T15:03:43.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.233+0000 7fbf37600640 1 Processor -- start 2024-03-31T15:03:43.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.233+0000 7f7b74800640 1 -- 172.21.15.16:0/4009901049 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7b6c161360 con 0x7f7b6c15a4e0 2024-03-31T15:03:43.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.233+0000 7ff995a00640 1 --2- 172.21.15.16:0/2040406316 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff978004810 0x7ff9901631a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:43.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.233+0000 7f434ec00640 1 -- 172.21.15.16:0/3053532721 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f4348162ae0 msgr2=0x7f43480cc5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:43.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.233+0000 7f0283400640 1 -- 172.21.15.16:0/3511530985 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f027c0b7070 con 0x7f02880070f0 2024-03-31T15:03:43.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.233+0000 7f0283400640 1 -- 172.21.15.16:0/3511530985 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f027c0aabd0 con 0x7f02880070f0 2024-03-31T15:03:43.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.233+0000 7f0283400640 1 -- 172.21.15.16:0/3511530985 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f027c0b3480 con 0x7f02880070f0 2024-03-31T15:03:43.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.233+0000 7ff996400640 1 -- 172.21.15.16:0/2040406316 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff978004810 msgr2=0x7ff9901631a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:43.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.233+0000 7f0294a00640 1 -- 172.21.15.16:0/3511530985 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0288035580 con 0x7f02880070f0 2024-03-31T15:03:43.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.233+0000 7fbf37600640 1 -- start start 2024-03-31T15:03:43.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.233+0000 7f0294a00640 1 -- 172.21.15.16:0/3511530985 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0288035e20 con 0x7f02880070f0 2024-03-31T15:03:43.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.233+0000 7ff996400640 1 --2- 172.21.15.16:0/2040406316 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff978004810 0x7ff9901631a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:43.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.233+0000 7ff996400640 1 -- 172.21.15.16:0/2040406316 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff99015cd60 msgr2=0x7ff9901618d0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-31T15:03:43.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.233+0000 7ff996400640 1 --2- 172.21.15.16:0/2040406316 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff99015cd60 0x7ff9901618d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:43.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.233+0000 7f434ec00640 1 --2- 172.21.15.16:0/3053532721 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f4348162ae0 0x7f43480cc5a0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f43380098a0 tx=0x7f433801c470 comp rx=0 tx=0).stop 2024-03-31T15:03:43.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.233+0000 7ff996400640 1 -- 172.21.15.16:0/2040406316 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff990162100 con 0x7ff990163e30 2024-03-31T15:03:43.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.233+0000 7f434ec00640 1 -- 172.21.15.16:0/3053532721 shutdown_connections 2024-03-31T15:03:43.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.233+0000 7ff996400640 1 --2- 172.21.15.16:0/2040406316 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff990163e30 0x7ff99014ce10 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7ff990073520 tx=0x7ff98800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:03:43.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.233+0000 7f434ec00640 1 --2- 172.21.15.16:0/3053532721 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f4348162ae0 0x7f43480cc5a0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:43.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.233+0000 7f434ec00640 1 -- 172.21.15.16:0/3053532721 >> 172.21.15.16:0/3053532721 conn(0x7f434806b620 msgr2=0x7f434806ba30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:03:43.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.233+0000 7ff98d600640 1 -- 172.21.15.16:0/2040406316 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff98800ed70 con 0x7ff990163e30 2024-03-31T15:03:43.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.233+0000 7ff998400640 1 -- 172.21.15.16:0/2040406316 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff9901623e0 con 0x7ff990163e30 2024-03-31T15:03:43.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.233+0000 7ff98d600640 1 -- 172.21.15.16:0/2040406316 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff988019070 con 0x7ff990163e30 2024-03-31T15:03:43.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.233+0000 7ff98d600640 1 -- 172.21.15.16:0/2040406316 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff988014660 con 0x7ff990163e30 2024-03-31T15:03:43.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.233+0000 7ff998400640 1 -- 172.21.15.16:0/2040406316 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff9901629a0 con 0x7ff990163e30 2024-03-31T15:03:43.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.233+0000 7fbf37600640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbf300ada20 0x7fbf30202080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:43.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.237+0000 7fbf37600640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fbf300cca10 0x7fbf30161360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:43.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.237+0000 7fbf37600640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbf3015e750 0x7fbf301618a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:43.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.237+0000 7fbf2fe00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fbf300cca10 0x7fbf30161360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:43.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.237+0000 7fbf2fe00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fbf300cca10 0x7fbf30161360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3301/0 says I am v2:172.21.15.16:41638/0 (socket says 172.21.15.16:41638) 2024-03-31T15:03:43.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.237+0000 7fbf2fe00640 1 -- 172.21.15.16:0/728813491 learned_addr learned my addr 172.21.15.16:0/728813491 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:03:43.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.237+0000 7fbf34c00640 1 --2- 172.21.15.16:0/728813491 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbf300ada20 0x7fbf30202080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:43.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.237+0000 7fbf35600640 1 --2- 172.21.15.16:0/728813491 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbf3015e750 0x7fbf301618a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:43.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.237+0000 7fbf37600640 1 -- 172.21.15.16:0/728813491 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fbf300cf9a0 con 0x7fbf300ada20 2024-03-31T15:03:43.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.237+0000 7fbf37600640 1 -- 172.21.15.16:0/728813491 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fbf300afae0 con 0x7fbf3015e750 2024-03-31T15:03:43.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.237+0000 7fbf37600640 1 -- 172.21.15.16:0/728813491 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fbf300b5250 con 0x7fbf300cca10 2024-03-31T15:03:43.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.237+0000 7fbf35600640 1 -- 172.21.15.16:0/728813491 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fbf300cca10 msgr2=0x7fbf30161360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:43.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.237+0000 7fbf35600640 1 --2- 172.21.15.16:0/728813491 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fbf300cca10 0x7fbf30161360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:43.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.237+0000 7fbf35600640 1 -- 172.21.15.16:0/728813491 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbf300ada20 msgr2=0x7fbf30202080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:43.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.237+0000 7fbf35600640 1 --2- 172.21.15.16:0/728813491 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbf300ada20 0x7fbf30202080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:43.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.237+0000 7fbf35600640 1 -- 172.21.15.16:0/728813491 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf300721c0 con 0x7fbf3015e750 2024-03-31T15:03:43.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.237+0000 7fbf2fe00640 1 --2- 172.21.15.16:0/728813491 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fbf300cca10 0x7fbf30161360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:03:43.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.237+0000 7fbf34c00640 1 --2- 172.21.15.16:0/728813491 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbf300ada20 0x7fbf30202080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:03:43.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.237+0000 7ff998400640 1 -- 172.21.15.16:0/2040406316 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff990069950 con 0x7ff990163e30 2024-03-31T15:03:43.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.237+0000 7ff98d600640 1 -- 172.21.15.16:0/2040406316 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7ff988014800 con 0x7ff990163e30 2024-03-31T15:03:43.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.237+0000 7f0294a00640 1 -- 172.21.15.16:0/3511530985 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0288035750 con 0x7f02880070f0 2024-03-31T15:03:43.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.233+0000 7f7b74800640 1 -- 172.21.15.16:0/4009901049 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7b6c069950 con 0x7f7b6c15a4e0 2024-03-31T15:03:43.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.237+0000 7f7b66000640 1 -- 172.21.15.16:0/4009901049 <== mon.1 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f7b680bf020 con 0x7f7b6c15a4e0 2024-03-31T15:03:43.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.237+0000 7f434ec00640 1 -- 172.21.15.16:0/3053532721 shutdown_connections 2024-03-31T15:03:43.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.237+0000 7f0283400640 1 -- 172.21.15.16:0/3511530985 <== mon.1 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f027c0b3620 con 0x7f02880070f0 2024-03-31T15:03:43.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.237+0000 7f434ec00640 1 -- 172.21.15.16:0/3053532721 wait complete. 2024-03-31T15:03:43.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.237+0000 7fbf35600640 1 --2- 172.21.15.16:0/728813491 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbf3015e750 0x7fbf301618a0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fbf1c0af000 tx=0x7fbf1c01ba30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:03:43.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.237+0000 7ff98d600640 1 --2- 172.21.15.16:0/2040406316 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7ff9700683b0 0x7ff97006a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:43.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.241+0000 7ff98d600640 1 -- 172.21.15.16:0/2040406316 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7ff988086ee0 con 0x7ff990163e30 2024-03-31T15:03:43.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.241+0000 7ff995a00640 1 --2- 172.21.15.16:0/2040406316 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7ff9700683b0 0x7ff97006a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:43.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.241+0000 7f7b66000640 1 --2- 172.21.15.16:0/4009901049 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f7b50068380 0x7f7b5006a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:43.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.241+0000 7f7b66000640 1 -- 172.21.15.16:0/4009901049 <== mon.1 v2:172.21.15.89:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f7b68121f70 con 0x7f7b6c15a4e0 2024-03-31T15:03:43.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.241+0000 7ff995a00640 1 --2- 172.21.15.16:0/2040406316 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7ff9700683b0 0x7ff97006a870 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7ff97c000fe0 tx=0x7ff97c009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:03:43.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.241+0000 7f7b71e00640 1 --2- 172.21.15.16:0/4009901049 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f7b50068380 0x7f7b5006a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:43.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.241+0000 7f0283400640 1 --2- 172.21.15.16:0/3511530985 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f0270068450 0x7f027006a910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:43.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.241+0000 7f0283400640 1 -- 172.21.15.16:0/3511530985 <== mon.1 v2:172.21.15.89:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f027c120b70 con 0x7f02880070f0 2024-03-31T15:03:43.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.241+0000 7f028ea00640 1 --2- 172.21.15.16:0/3511530985 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f0270068450 0x7f027006a910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:43.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.241+0000 7f7b71e00640 1 --2- 172.21.15.16:0/4009901049 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f7b50068380 0x7f7b5006a840 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f7b60006fd0 tx=0x7f7b60008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:03:43.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.241+0000 7f434ec00640 1 Processor -- start 2024-03-31T15:03:43.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.241+0000 7f434ec00640 1 -- start start 2024-03-31T15:03:43.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.245+0000 7f7b66000640 1 -- 172.21.15.16:0/4009901049 <== mon.1 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f7b680c6030 con 0x7f7b6c15a4e0 2024-03-31T15:03:43.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.245+0000 7f028ea00640 1 --2- 172.21.15.16:0/3511530985 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f0270068450 0x7f027006a910 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f0288184270 tx=0x7f0284009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:03:43.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.245+0000 7f434ec00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f434806f580 0x7f434815ed00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:43.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.245+0000 7f434ec00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f43480d0970 0x7f4348154e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:43.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.245+0000 7f434ec00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4348162ae0 0x7f43480b2760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:43.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.245+0000 7ff98d600640 1 -- 172.21.15.16:0/2040406316 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7ff988023020 con 0x7ff990163e30 2024-03-31T15:03:43.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.245+0000 7f434ec00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f43480cf6a0 con 0x7f434806f580 2024-03-31T15:03:43.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.245+0000 7f4347e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f434806f580 0x7f434815ed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:43.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.245+0000 7f4347e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f434806f580 0x7f434815ed00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39122/0 (socket says 172.21.15.16:39122) 2024-03-31T15:03:43.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.245+0000 7f4347e00640 1 -- 172.21.15.16:0/3043022218 learned_addr learned my addr 172.21.15.16:0/3043022218 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:03:43.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.245+0000 7f434cc00640 1 --2- 172.21.15.16:0/3043022218 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4348162ae0 0x7f43480b2760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:43.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.245+0000 7f4347400640 1 --2- 172.21.15.16:0/3043022218 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f43480d0970 0x7f4348154e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:43.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.245+0000 7f434ec00640 1 -- 172.21.15.16:0/3043022218 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f43480cf520 con 0x7f4348162ae0 2024-03-31T15:03:43.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.245+0000 7f434ec00640 1 -- 172.21.15.16:0/3043022218 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f43481708f0 con 0x7f43480d0970 2024-03-31T15:03:43.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.245+0000 7fbf2cc00640 1 -- 172.21.15.16:0/728813491 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf1c0b5040 con 0x7fbf3015e750 2024-03-31T15:03:43.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.245+0000 7fbf2cc00640 1 -- 172.21.15.16:0/728813491 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbf1c0bf070 con 0x7fbf3015e750 2024-03-31T15:03:43.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.245+0000 7fbf37600640 1 -- 172.21.15.16:0/728813491 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbf300724a0 con 0x7fbf3015e750 2024-03-31T15:03:43.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.245+0000 7fbf37600640 1 -- 172.21.15.16:0/728813491 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbf30072a60 con 0x7fbf3015e750 2024-03-31T15:03:43.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.249+0000 7f4347400640 1 -- 172.21.15.16:0/3043022218 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4348162ae0 msgr2=0x7f43480b2760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:43.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.249+0000 7f4347400640 1 --2- 172.21.15.16:0/3043022218 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4348162ae0 0x7f43480b2760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:43.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.249+0000 7f4347400640 1 -- 172.21.15.16:0/3043022218 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f434806f580 msgr2=0x7f434815ed00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:43.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.249+0000 7f4347400640 1 --2- 172.21.15.16:0/3043022218 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f434806f580 0x7f434815ed00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:43.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.249+0000 7fbf37600640 1 -- 172.21.15.16:0/728813491 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbf30069950 con 0x7fbf3015e750 2024-03-31T15:03:43.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.249+0000 7f0283400640 1 -- 172.21.15.16:0/3511530985 <== mon.1 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f02880e25a0 con 0x7f02880070f0 2024-03-31T15:03:43.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.249+0000 7f4347400640 1 -- 172.21.15.16:0/3043022218 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f434815b9b0 con 0x7f43480d0970 2024-03-31T15:03:43.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.249+0000 7f434cc00640 1 --2- 172.21.15.16:0/3043022218 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4348162ae0 0x7f43480b2760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:03:43.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.249+0000 7f4347e00640 1 --2- 172.21.15.16:0/3043022218 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f434806f580 0x7f434815ed00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:03:43.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.249+0000 7f4347400640 1 --2- 172.21.15.16:0/3043022218 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f43480d0970 0x7f4348154e30 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f433400b410 tx=0x7f433400b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:03:43.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.249+0000 7f433fe00640 1 -- 172.21.15.16:0/3043022218 <== mon.2 v2:172.21.15.89:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4334015020 con 0x7f43480d0970 2024-03-31T15:03:43.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.249+0000 7fbf2cc00640 1 -- 172.21.15.16:0/728813491 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf1c0b5040 con 0x7fbf3015e750 2024-03-31T15:03:43.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.249+0000 7f434ec00640 1 -- 172.21.15.16:0/3043022218 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f434815bc50 con 0x7f43480d0970 2024-03-31T15:03:43.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.249+0000 7f434ec00640 1 -- 172.21.15.16:0/3043022218 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f434815c210 con 0x7f43480d0970 2024-03-31T15:03:43.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.249+0000 7f433fe00640 1 -- 172.21.15.16:0/3043022218 <== mon.2 v2:172.21.15.89:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4334011830 con 0x7f43480d0970 2024-03-31T15:03:43.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.249+0000 7f433fe00640 1 -- 172.21.15.16:0/3043022218 <== mon.2 v2:172.21.15.89:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4334011c20 con 0x7f43480d0970 2024-03-31T15:03:43.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.253+0000 7f434ec00640 1 -- 172.21.15.16:0/3043022218 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f43200052a0 con 0x7f43480d0970 2024-03-31T15:03:43.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.253+0000 7f433fe00640 1 -- 172.21.15.16:0/3043022218 <== mon.2 v2:172.21.15.89:3301/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f4334018020 con 0x7f43480d0970 2024-03-31T15:03:43.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.253+0000 7fbf2cc00640 1 -- 172.21.15.16:0/728813491 <== mon.1 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fbf1c0a8920 con 0x7fbf3015e750 2024-03-31T15:03:43.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.253+0000 7f433fe00640 1 --2- 172.21.15.16:0/3043022218 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f4328068380 0x7f432806a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:43.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.253+0000 7f433fe00640 1 -- 172.21.15.16:0/3043022218 <== mon.2 v2:172.21.15.89:3301/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f4334086790 con 0x7f43480d0970 2024-03-31T15:03:43.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.253+0000 7fbf2cc00640 1 --2- 172.21.15.16:0/728813491 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fbf10068580 0x7fbf1006aa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:43.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.253+0000 7fbf2cc00640 1 -- 172.21.15.16:0/728813491 <== mon.1 v2:172.21.15.89:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fbf1c124140 con 0x7fbf3015e750 2024-03-31T15:03:43.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.257+0000 7fbf34c00640 1 --2- 172.21.15.16:0/728813491 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fbf10068580 0x7fbf1006aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:43.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.257+0000 7f4347e00640 1 --2- 172.21.15.16:0/3043022218 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f4328068380 0x7f432806a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:43.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.257+0000 7fbf2cc00640 1 -- 172.21.15.16:0/728813491 <== mon.1 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fbf1c0187f0 con 0x7fbf3015e750 2024-03-31T15:03:43.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.257+0000 7f4347e00640 1 --2- 172.21.15.16:0/3043022218 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f4328068380 0x7f432806a840 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f433801cd10 tx=0x7f43380a6000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:03:43.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.257+0000 7fbf34c00640 1 --2- 172.21.15.16:0/728813491 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fbf10068580 0x7fbf1006aa40 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fbf28003e70 tx=0x7fbf28009830 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:03:43.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.257+0000 7f433fe00640 1 -- 172.21.15.16:0/3043022218 <== mon.2 v2:172.21.15.89:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f433401f020 con 0x7f43480d0970 2024-03-31T15:03:43.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.281+0000 7fde73200640 1 Processor -- start 2024-03-31T15:03:43.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.281+0000 7fde73200640 1 -- start start 2024-03-31T15:03:43.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.285+0000 7fde73200640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fde740cf160 0x7fde740b4020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:43.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.285+0000 7fde73200640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fde740ceab0 0x7fde74071430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:43.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.285+0000 7fde71e00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fde740cf160 0x7fde740b4020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:43.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.285+0000 7fde71e00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fde740cf160 0x7fde740b4020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.16:36128/0 (socket says 172.21.15.16:36128) 2024-03-31T15:03:43.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.285+0000 7fde71400640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fde740ceab0 0x7fde74071430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:43.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.285+0000 7fde71400640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fde740ceab0 0x7fde74071430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3301/0 says I am v2:172.21.15.16:41656/0 (socket says 172.21.15.16:41656) 2024-03-31T15:03:43.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.285+0000 7fde72800640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fde740cf540 0x7fde7406e330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:34028/0 (socket says 172.21.15.16:34028) 2024-03-31T15:03:43.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.285+0000 7fde71e00640 1 -- 172.21.15.16:0/2092339003 learned_addr learned my addr 172.21.15.16:0/2092339003 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:03:43.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.285+0000 7fde73200640 1 -- 172.21.15.16:0/2092339003 --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fde740b5170 con 0x7fde740cf540 2024-03-31T15:03:43.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.285+0000 7fde73200640 1 -- 172.21.15.16:0/2092339003 --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7fde740b5450 con 0x7fde740cf160 2024-03-31T15:03:43.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.285+0000 7fde73200640 1 -- 172.21.15.16:0/2092339003 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fde7406d850 con 0x7fde740ceab0 2024-03-31T15:03:43.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.285+0000 7f7805c00640 1 Processor -- start 2024-03-31T15:03:43.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.285+0000 7f7805c00640 1 -- start start 2024-03-31T15:03:43.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.285+0000 7fde71e00640 1 -- 172.21.15.16:0/2092339003 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fde740ceab0 msgr2=0x7fde74071430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:03:43.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.285+0000 7fde71e00640 1 --2- 172.21.15.16:0/2092339003 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fde740ceab0 0x7fde74071430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:43.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.285+0000 7fde71e00640 1 -- 172.21.15.16:0/2092339003 >> v1:172.21.15.16:6789/0 conn(0x7fde740cf540 legacy=0x7fde7406e330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:43.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.285+0000 7fde71e00640 1 -- 172.21.15.16:0/2092339003 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde7406db30 con 0x7fde740cf160 2024-03-31T15:03:43.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.285+0000 7f7805c00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f780015be60 0x7f78000d00e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:43.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.285+0000 7f7805c00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f78000b35f0 0x7f78000715f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:43.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.285+0000 7f7805c00640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f780015f710 con 0x7f78000d0620 2024-03-31T15:03:43.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.285+0000 7f7805c00640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f780015c240 con 0x7f78000b35f0 2024-03-31T15:03:43.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.285+0000 7f7805c00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f780015eb00 con 0x7f780015be60 2024-03-31T15:03:43.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.289+0000 7f77ffe00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f780015be60 0x7f78000d00e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:43.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.289+0000 7f77ffe00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f780015be60 0x7f78000d00e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3301/0 says I am v2:172.21.15.16:41666/0 (socket says 172.21.15.16:41666) 2024-03-31T15:03:43.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.289+0000 7f77ffe00640 1 -- 172.21.15.16:0/2621937323 learned_addr learned my addr 172.21.15.16:0/2621937323 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:03:43.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.289+0000 7f7805200640 1 --2- 172.21.15.16:0/2621937323 >> v2:172.21.15.16:3300/0 conn(0x7f78000b35f0 0x7f78000715f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:43.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.289+0000 7f77ffe00640 1 -- 172.21.15.16:0/2621937323 >> v2:172.21.15.16:3300/0 conn(0x7f78000b35f0 msgr2=0x7f78000715f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:03:43.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.289+0000 7f77ffe00640 1 --2- 172.21.15.16:0/2621937323 >> v2:172.21.15.16:3300/0 conn(0x7f78000b35f0 0x7f78000715f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:43.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.289+0000 7f77ffe00640 1 -- 172.21.15.16:0/2621937323 >> v1:172.21.15.16:6789/0 conn(0x7f78000d0620 legacy=0x7f78000b2ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:43.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.289+0000 7f77ffe00640 1 -- 172.21.15.16:0/2621937323 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f780015ede0 con 0x7f780015be60 2024-03-31T15:03:43.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.289+0000 7fde71e00640 1 --2- 172.21.15.16:0/2092339003 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fde740cf160 0x7fde740b4020 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fde6c009fd0 tx=0x7fde6c01c8e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=299ccb7c4373f3d3 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:03:43.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.293+0000 7fde70a00640 1 -- 172.21.15.16:0/2092339003 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde6c007d60 con 0x7fde740cf160 2024-03-31T15:03:43.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.293+0000 7fde70a00640 1 -- 172.21.15.16:0/2092339003 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fde6c0a56e0 con 0x7fde740cf160 2024-03-31T15:03:43.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.293+0000 7fde70a00640 1 -- 172.21.15.16:0/2092339003 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde6c0a5ab0 con 0x7fde740cf160 2024-03-31T15:03:43.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.293+0000 7fde73200640 1 -- 172.21.15.16:0/2092339003 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fde740cf160 msgr2=0x7fde740b4020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:43.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.293+0000 7fde73200640 1 --2- 172.21.15.16:0/2092339003 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fde740cf160 0x7fde740b4020 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fde6c009fd0 tx=0x7fde6c01c8e0 comp rx=0 tx=0).stop 2024-03-31T15:03:43.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.297+0000 7fde73200640 1 -- 172.21.15.16:0/2092339003 shutdown_connections 2024-03-31T15:03:43.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.297+0000 7fde73200640 1 --2- 172.21.15.16:0/2092339003 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fde740ceab0 0x7fde74071430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:43.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.297+0000 7fde73200640 1 --2- 172.21.15.16:0/2092339003 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fde740cf160 0x7fde740b4020 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:43.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.297+0000 7fde73200640 1 -- 172.21.15.16:0/2092339003 >> 172.21.15.16:0/2092339003 conn(0x7fde740b1c50 msgr2=0x7fde740b2060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:03:43.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.297+0000 7f77ffe00640 1 --2- 172.21.15.16:0/2621937323 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f780015be60 0x7f78000d00e0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f77f0009870 tx=0x7f77f001c5a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e79726e54a9c2ce2 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:03:43.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.297+0000 7fde73200640 1 -- 172.21.15.16:0/2092339003 shutdown_connections 2024-03-31T15:03:43.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.297+0000 7fde73200640 1 -- 172.21.15.16:0/2092339003 wait complete. 2024-03-31T15:03:43.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.297+0000 7f77fea00640 1 -- 172.21.15.16:0/2621937323 <== mon.2 v2:172.21.15.89:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77f00ab650 con 0x7f780015be60 2024-03-31T15:03:43.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.297+0000 7f77fea00640 1 -- 172.21.15.16:0/2621937323 <== mon.2 v2:172.21.15.89:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f77f00b0070 con 0x7f780015be60 2024-03-31T15:03:43.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.297+0000 7f77fea00640 1 -- 172.21.15.16:0/2621937323 <== mon.2 v2:172.21.15.89:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77f00aba10 con 0x7f780015be60 2024-03-31T15:03:43.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.297+0000 7f7805c00640 1 -- 172.21.15.16:0/2621937323 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f780015be60 msgr2=0x7f78000d00e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:43.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.297+0000 7f7805c00640 1 --2- 172.21.15.16:0/2621937323 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f780015be60 0x7f78000d00e0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f77f0009870 tx=0x7f77f001c5a0 comp rx=0 tx=0).stop 2024-03-31T15:03:43.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.297+0000 7f7805c00640 1 -- 172.21.15.16:0/2621937323 shutdown_connections 2024-03-31T15:03:43.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.297+0000 7f7805c00640 1 --2- 172.21.15.16:0/2621937323 >> v2:172.21.15.16:3300/0 conn(0x7f78000b35f0 0x7f78000715f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:43.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.297+0000 7f7805c00640 1 --2- 172.21.15.16:0/2621937323 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f780015be60 0x7f78000d00e0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:43.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.297+0000 7f7805c00640 1 -- 172.21.15.16:0/2621937323 >> 172.21.15.16:0/2621937323 conn(0x7f780006b620 msgr2=0x7f780006ba00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:03:43.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.297+0000 7f7805c00640 1 -- 172.21.15.16:0/2621937323 shutdown_connections 2024-03-31T15:03:43.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.297+0000 7fde73200640 1 Processor -- start 2024-03-31T15:03:43.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.301+0000 7fde73200640 1 -- start start 2024-03-31T15:03:43.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.297+0000 7f7805c00640 1 -- 172.21.15.16:0/2621937323 wait complete. 2024-03-31T15:03:43.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.301+0000 7f7805c00640 1 Processor -- start 2024-03-31T15:03:43.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.301+0000 7fde73200640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fde740ceab0 0x7fde74159970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:43.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.301+0000 7fde73200640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fde740cf160 0x7fde741f7df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:43.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.301+0000 7f7805c00640 1 -- start start 2024-03-31T15:03:43.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.301+0000 7fde71e00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fde740ceab0 0x7fde74159970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:43.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.301+0000 7fde71e00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fde740ceab0 0x7fde74159970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.16:36130/0 (socket says 172.21.15.16:36130) 2024-03-31T15:03:43.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.301+0000 7fde73200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fde740cf540 0x7fde7415cdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:43.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.301+0000 7fde71400640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fde740cf160 0x7fde741f7df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:43.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.301+0000 7fde71400640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fde740cf160 0x7fde741f7df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3301/0 says I am v2:172.21.15.16:41674/0 (socket says 172.21.15.16:41674) 2024-03-31T15:03:43.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.305+0000 7fde71400640 1 -- 172.21.15.16:0/4115200107 learned_addr learned my addr 172.21.15.16:0/4115200107 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:03:43.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.305+0000 7f7805c00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f78000b35f0 0x7f7800204ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:43.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.305+0000 7fde73200640 1 -- 172.21.15.16:0/4115200107 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fde740b5950 con 0x7fde740cf540 2024-03-31T15:03:43.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.305+0000 7fde73200640 1 -- 172.21.15.16:0/4115200107 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fde740cf9f0 con 0x7fde740ceab0 2024-03-31T15:03:43.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.305+0000 7f7805c00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f78000d0620 0x7f7800204630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:43.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.305+0000 7f7805c00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f780015be60 0x7f7800151110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:43.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.305+0000 7fde73200640 1 -- 172.21.15.16:0/4115200107 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fde740b5ad0 con 0x7fde740cf160 2024-03-31T15:03:43.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.305+0000 7fde72800640 1 --2- 172.21.15.16:0/4115200107 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fde740cf540 0x7fde7415cdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:43.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.305+0000 7f77ffe00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f78000b35f0 0x7f7800204ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:43.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.305+0000 7f77ffe00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f78000b35f0 0x7f7800204ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3301/0 says I am v2:172.21.15.16:41680/0 (socket says 172.21.15.16:41680) 2024-03-31T15:03:43.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.305+0000 7f7805c00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f78000724c0 con 0x7f780015be60 2024-03-31T15:03:43.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.305+0000 7f77ffe00640 1 -- 172.21.15.16:0/449655726 learned_addr learned my addr 172.21.15.16:0/449655726 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:03:43.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.305+0000 7f7805c00640 1 -- 172.21.15.16:0/449655726 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f7800072340 con 0x7f78000d0620 2024-03-31T15:03:43.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.305+0000 7f7805c00640 1 -- 172.21.15.16:0/449655726 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f7800072640 con 0x7f78000b35f0 2024-03-31T15:03:43.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.305+0000 7f77ff400640 1 --2- 172.21.15.16:0/449655726 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f78000d0620 0x7f7800204630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:43.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.305+0000 7f7805200640 1 --2- 172.21.15.16:0/449655726 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f780015be60 0x7f7800151110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:43.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.305+0000 7f77ffe00640 1 -- 172.21.15.16:0/449655726 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f78000d0620 msgr2=0x7f7800204630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:43.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.305+0000 7f77ffe00640 1 --2- 172.21.15.16:0/449655726 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f78000d0620 0x7f7800204630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:43.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.305+0000 7f77ffe00640 1 -- 172.21.15.16:0/449655726 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f780015be60 msgr2=0x7f7800151110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:43.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.305+0000 7f77ffe00640 1 --2- 172.21.15.16:0/449655726 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f780015be60 0x7f7800151110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:43.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.305+0000 7f77ffe00640 1 -- 172.21.15.16:0/449655726 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f780015dad0 con 0x7f78000b35f0 2024-03-31T15:03:43.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.305+0000 7fde71400640 1 -- 172.21.15.16:0/4115200107 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fde740ceab0 msgr2=0x7fde74159970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:43.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.305+0000 7fde71400640 1 --2- 172.21.15.16:0/4115200107 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fde740ceab0 0x7fde74159970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:43.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.305+0000 7fde71400640 1 -- 172.21.15.16:0/4115200107 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fde740cf540 msgr2=0x7fde7415cdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:43.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.305+0000 7fde71400640 1 --2- 172.21.15.16:0/4115200107 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fde740cf540 0x7fde7415cdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:43.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.305+0000 7fde71400640 1 -- 172.21.15.16:0/4115200107 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde741f8330 con 0x7fde740cf160 2024-03-31T15:03:43.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.305+0000 7fde71400640 1 --2- 172.21.15.16:0/4115200107 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fde740cf160 0x7fde741f7df0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fde6400d800 tx=0x7fde6400dcc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:03:43.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.305+0000 7fde72800640 1 --2- 172.21.15.16:0/4115200107 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fde740cf540 0x7fde7415cdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:03:43.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.305+0000 7fde6a000640 1 -- 172.21.15.16:0/4115200107 <== mon.2 v2:172.21.15.89:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde6400b6c0 con 0x7fde740cf160 2024-03-31T15:03:43.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.305+0000 7fde6a000640 1 -- 172.21.15.16:0/4115200107 <== mon.2 v2:172.21.15.89:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fde64019070 con 0x7fde740cf160 2024-03-31T15:03:43.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.305+0000 7fde6a000640 1 -- 172.21.15.16:0/4115200107 <== mon.2 v2:172.21.15.89:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde6400ba80 con 0x7fde740cf160 2024-03-31T15:03:43.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.305+0000 7fde71e00640 1 --2- 172.21.15.16:0/4115200107 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fde740ceab0 0x7fde74159970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:03:43.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.305+0000 7fde73200640 1 -- 172.21.15.16:0/4115200107 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fde741588d0 con 0x7fde740cf160 2024-03-31T15:03:43.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.305+0000 7f77ffe00640 1 --2- 172.21.15.16:0/449655726 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f78000b35f0 0x7f7800204ff0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f77f00ad040 tx=0x7f77f001ade0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:03:43.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.305+0000 7fde73200640 1 -- 172.21.15.16:0/4115200107 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fde74158e90 con 0x7fde740cf160 2024-03-31T15:03:43.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.305+0000 7f77f7e00640 1 -- 172.21.15.16:0/449655726 <== mon.2 v2:172.21.15.89:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77f00be5f0 con 0x7f78000b35f0 2024-03-31T15:03:43.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.305+0000 7f77f7e00640 1 -- 172.21.15.16:0/449655726 <== mon.2 v2:172.21.15.89:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f77f00be7b0 con 0x7f78000b35f0 2024-03-31T15:03:43.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.309+0000 7fde73200640 1 -- 172.21.15.16:0/4115200107 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fde7404f8c0 con 0x7fde740cf160 2024-03-31T15:03:43.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.309+0000 7f77f7e00640 1 -- 172.21.15.16:0/449655726 <== mon.2 v2:172.21.15.89:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77f00beba0 con 0x7f78000b35f0 2024-03-31T15:03:43.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.309+0000 7f7805c00640 1 -- 172.21.15.16:0/449655726 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f780015ddb0 con 0x7f78000b35f0 2024-03-31T15:03:43.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.309+0000 7f7805c00640 1 -- 172.21.15.16:0/449655726 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f7800160310 con 0x7f78000b35f0 2024-03-31T15:03:43.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.309+0000 7fde6a000640 1 -- 172.21.15.16:0/4115200107 <== mon.2 v2:172.21.15.89:3301/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fde6400bc20 con 0x7fde740cf160 2024-03-31T15:03:43.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.313+0000 7fde6a000640 1 --2- 172.21.15.16:0/4115200107 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fde54068380 0x7fde5406a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:43.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.313+0000 7f77f7e00640 1 -- 172.21.15.16:0/449655726 <== mon.2 v2:172.21.15.89:3301/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f77f00b43f0 con 0x7f78000b35f0 2024-03-31T15:03:43.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.313+0000 7f7805c00640 1 -- 172.21.15.16:0/449655726 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f780004f8c0 con 0x7f78000b35f0 2024-03-31T15:03:43.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.313+0000 7fde6a000640 1 -- 172.21.15.16:0/4115200107 <== mon.2 v2:172.21.15.89:3301/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fde64054da0 con 0x7fde740cf160 2024-03-31T15:03:43.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.313+0000 7fde71e00640 1 --2- 172.21.15.16:0/4115200107 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fde54068380 0x7fde5406a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:43.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.313+0000 7f77f7e00640 1 --2- 172.21.15.16:0/449655726 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f77e0068480 0x7f77e006a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:43.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.313+0000 7fde71e00640 1 --2- 172.21.15.16:0/4115200107 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fde54068380 0x7fde5406a840 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fde6c0ae000 tx=0x7fde6c0a7070 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:03:43.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.313+0000 7f77f7e00640 1 -- 172.21.15.16:0/449655726 <== mon.2 v2:172.21.15.89:3301/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f77f0121440 con 0x7f78000b35f0 2024-03-31T15:03:43.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.313+0000 7f77ff400640 1 --2- 172.21.15.16:0/449655726 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f77e0068480 0x7f77e006a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:43.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.313+0000 7f77ff400640 1 --2- 172.21.15.16:0/449655726 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f77e0068480 0x7f77e006a940 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f78000cf780 tx=0x7f77ec009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:03:43.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.317+0000 7f77f7e00640 1 -- 172.21.15.16:0/449655726 <== mon.2 v2:172.21.15.89:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f77f0016190 con 0x7f78000b35f0 2024-03-31T15:03:43.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.321+0000 7fde6a000640 1 -- 172.21.15.16:0/4115200107 <== mon.2 v2:172.21.15.89:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fde741f8330 con 0x7fde740cf160 2024-03-31T15:03:43.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.569+0000 7f183f000640 1 -- 172.21.15.16:0/3318694499 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7f183806c230 con 0x7f18381f8c00 2024-03-31T15:03:43.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.573+0000 7f1835600640 1 -- 172.21.15.16:0/3318694499 <== mon.2 v2:172.21.15.89:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f18300565f0 con 0x7f18381f8c00 2024-03-31T15:03:43.569 INFO:teuthology.orchestra.run.smithi016.stdout:60129542150 2024-03-31T15:03:43.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.573+0000 7f183f000640 1 -- 172.21.15.16:0/3318694499 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f1818068380 msgr2=0x7f181806a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:43.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.573+0000 7f183f000640 1 --2- 172.21.15.16:0/3318694499 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f1818068380 0x7f181806a840 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f182c0b0000 tx=0x7f182c0ac040 comp rx=0 tx=0).stop 2024-03-31T15:03:43.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.573+0000 7f183f000640 1 -- 172.21.15.16:0/3318694499 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f18381f8c00 msgr2=0x7f18380b21c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:43.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.573+0000 7f183f000640 1 --2- 172.21.15.16:0/3318694499 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f18381f8c00 0x7f18380b21c0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f183806dbf0 tx=0x7f183000ead0 comp rx=0 tx=0).stop 2024-03-31T15:03:43.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.573+0000 7f183f000640 1 -- 172.21.15.16:0/3318694499 shutdown_connections 2024-03-31T15:03:43.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.577+0000 7f183f000640 1 --2- 172.21.15.16:0/3318694499 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f1818068380 0x7f181806a840 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:43.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.577+0000 7f183f000640 1 --2- 172.21.15.16:0/3318694499 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f18381f8c00 0x7f18380b21c0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:43.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.577+0000 7f183f000640 1 --2- 172.21.15.16:0/3318694499 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f18381f8780 0x7f18380b1c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:43.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.577+0000 7f183f000640 1 --2- 172.21.15.16:0/3318694499 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1820003a10 0x7f18380cfeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:43.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.577+0000 7f183f000640 1 -- 172.21.15.16:0/3318694499 >> 172.21.15.16:0/3318694499 conn(0x7f18380b2b30 msgr2=0x7f18380afa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:03:43.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.577+0000 7f183f000640 1 -- 172.21.15.16:0/3318694499 shutdown_connections 2024-03-31T15:03:43.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.577+0000 7f183f000640 1 -- 172.21.15.16:0/3318694499 wait complete. 2024-03-31T15:03:43.585 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.589+0000 7ff998400640 1 -- 172.21.15.16:0/2040406316 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7ff990072f40 con 0x7ff990163e30 2024-03-31T15:03:43.585 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.589+0000 7ff98d600640 1 -- 172.21.15.16:0/2040406316 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7ff9880574a0 con 0x7ff990163e30 2024-03-31T15:03:43.588 INFO:teuthology.orchestra.run.smithi016.stdout:60129542150 2024-03-31T15:03:43.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.597+0000 7ff982a00640 1 -- 172.21.15.16:0/2040406316 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7ff9700683b0 msgr2=0x7ff97006a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:43.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.597+0000 7ff982a00640 1 --2- 172.21.15.16:0/2040406316 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7ff9700683b0 0x7ff97006a870 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7ff97c000fe0 tx=0x7ff97c009040 comp rx=0 tx=0).stop 2024-03-31T15:03:43.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.597+0000 7ff982a00640 1 -- 172.21.15.16:0/2040406316 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff990163e30 msgr2=0x7ff99014ce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:43.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.597+0000 7ff982a00640 1 --2- 172.21.15.16:0/2040406316 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff990163e30 0x7ff99014ce10 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7ff990073520 tx=0x7ff98800e4b0 comp rx=0 tx=0).stop 2024-03-31T15:03:43.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.597+0000 7ff982a00640 1 -- 172.21.15.16:0/2040406316 shutdown_connections 2024-03-31T15:03:43.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.597+0000 7ff982a00640 1 --2- 172.21.15.16:0/2040406316 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7ff9700683b0 0x7ff97006a870 secure :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7ff97c000fe0 tx=0x7ff97c009040 comp rx=0 tx=0).stop 2024-03-31T15:03:43.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.597+0000 7ff982a00640 1 --2- 172.21.15.16:0/2040406316 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff990163e30 0x7ff99014ce10 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:43.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.597+0000 7ff982a00640 1 --2- 172.21.15.16:0/2040406316 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff99015cd60 0x7ff9901618d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:43.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.597+0000 7ff982a00640 1 --2- 172.21.15.16:0/2040406316 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff978004810 0x7ff9901631a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:43.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.597+0000 7ff982a00640 1 -- 172.21.15.16:0/2040406316 >> 172.21.15.16:0/2040406316 conn(0x7ff99006b620 msgr2=0x7ff9900ce770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:03:43.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.601+0000 7ff982a00640 1 -- 172.21.15.16:0/2040406316 shutdown_connections 2024-03-31T15:03:43.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.601+0000 7ff982a00640 1 -- 172.21.15.16:0/2040406316 wait complete. 2024-03-31T15:03:43.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.601+0000 7f7b74800640 1 -- 172.21.15.16:0/4009901049 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7f7b6c0cfdd0 con 0x7f7b6c15a4e0 2024-03-31T15:03:43.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.601+0000 7f7b66000640 1 -- 172.21.15.16:0/4009901049 <== mon.1 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f7b68016620 con 0x7f7b6c15a4e0 2024-03-31T15:03:43.598 INFO:teuthology.orchestra.run.smithi016.stdout:60129542150 2024-03-31T15:03:43.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.605+0000 7f0294a00640 1 -- 172.21.15.16:0/3511530985 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7f0288005720 con 0x7f02880070f0 2024-03-31T15:03:43.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.605+0000 7f0283400640 1 -- 172.21.15.16:0/3511530985 <== mon.1 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f027c0f12c0 con 0x7f02880070f0 2024-03-31T15:03:43.601 INFO:teuthology.orchestra.run.smithi016.stdout:64424509446 2024-03-31T15:03:43.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.605+0000 7f7b74800640 1 -- 172.21.15.16:0/4009901049 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f7b50068380 msgr2=0x7f7b5006a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:43.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.605+0000 7f7b74800640 1 --2- 172.21.15.16:0/4009901049 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f7b50068380 0x7f7b5006a840 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f7b60006fd0 tx=0x7f7b60008040 comp rx=0 tx=0).stop 2024-03-31T15:03:43.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.605+0000 7f7b74800640 1 -- 172.21.15.16:0/4009901049 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7b6c15a4e0 msgr2=0x7f7b6c151ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:43.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.605+0000 7f7b74800640 1 --2- 172.21.15.16:0/4009901049 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7b6c15a4e0 0x7f7b6c151ad0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f7b68009e50 tx=0x7f7b680197d0 comp rx=0 tx=0).stop 2024-03-31T15:03:43.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.605+0000 7fbf37600640 1 -- 172.21.15.16:0/728813491 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7fbf30073a50 con 0x7fbf3015e750 2024-03-31T15:03:43.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.605+0000 7f7b74800640 1 -- 172.21.15.16:0/4009901049 shutdown_connections 2024-03-31T15:03:43.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.605+0000 7f7b74800640 1 --2- 172.21.15.16:0/4009901049 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f7b50068380 0x7f7b5006a840 secure :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f7b60006fd0 tx=0x7f7b60008040 comp rx=0 tx=0).stop 2024-03-31T15:03:43.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.605+0000 7f7b74800640 1 --2- 172.21.15.16:0/4009901049 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7b6c15a4e0 0x7f7b6c151ad0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:43.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.605+0000 7f7b74800640 1 --2- 172.21.15.16:0/4009901049 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f7b6c0ce8b0 0x7f7b6c152780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:43.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.605+0000 7f7b74800640 1 --2- 172.21.15.16:0/4009901049 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7b6c0b34c0 0x7f7b6c160a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:43.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.605+0000 7f7b74800640 1 -- 172.21.15.16:0/4009901049 >> 172.21.15.16:0/4009901049 conn(0x7f7b6c06b620 msgr2=0x7f7b6c15b640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:03:43.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.605+0000 7f7b74800640 1 -- 172.21.15.16:0/4009901049 shutdown_connections 2024-03-31T15:03:43.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.605+0000 7f7b74800640 1 -- 172.21.15.16:0/4009901049 wait complete. 2024-03-31T15:03:43.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.605+0000 7fbf2cc00640 1 -- 172.21.15.16:0/728813491 <== mon.1 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fbf1c0f46e0 con 0x7fbf3015e750 2024-03-31T15:03:43.605 INFO:teuthology.orchestra.run.smithi016.stdout:64424509445 2024-03-31T15:03:43.605 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542152 got 60129542150 for osd.0 2024-03-31T15:03:43.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.609+0000 7f0294a00640 1 -- 172.21.15.16:0/3511530985 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f0270068450 msgr2=0x7f027006a910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:43.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.609+0000 7f0294a00640 1 --2- 172.21.15.16:0/3511530985 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f0270068450 0x7f027006a910 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f0288184270 tx=0x7f0284009040 comp rx=0 tx=0).stop 2024-03-31T15:03:43.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.609+0000 7f0294a00640 1 -- 172.21.15.16:0/3511530985 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f02880070f0 msgr2=0x7f0288034cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:43.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.609+0000 7f0294a00640 1 --2- 172.21.15.16:0/3511530985 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f02880070f0 0x7f0288034cc0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f027c020ea0 tx=0x7f027c01d3a0 comp rx=0 tx=0).stop 2024-03-31T15:03:43.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.609+0000 7f0294a00640 1 -- 172.21.15.16:0/3511530985 shutdown_connections 2024-03-31T15:03:43.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.609+0000 7f0294a00640 1 --2- 172.21.15.16:0/3511530985 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f0270068450 0x7f027006a910 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:43.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.609+0000 7f0294a00640 1 --2- 172.21.15.16:0/3511530985 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0288033fc0 0x7f0288036600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:43.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.609+0000 7f0294a00640 1 --2- 172.21.15.16:0/3511530985 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f0288007df0 0x7f02880de7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:43.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.609+0000 7f0294a00640 1 --2- 172.21.15.16:0/3511530985 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f02880070f0 0x7f0288034cc0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:43.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.609+0000 7f0294a00640 1 -- 172.21.15.16:0/3511530985 >> 172.21.15.16:0/3511530985 conn(0x7f0288055360 msgr2=0x7f0288005b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:03:43.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.609+0000 7fbf37600640 1 -- 172.21.15.16:0/728813491 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fbf10068580 msgr2=0x7fbf1006aa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:43.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.609+0000 7fbf37600640 1 --2- 172.21.15.16:0/728813491 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fbf10068580 0x7fbf1006aa40 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fbf28003e70 tx=0x7fbf28009830 comp rx=0 tx=0).stop 2024-03-31T15:03:43.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.609+0000 7fbf37600640 1 -- 172.21.15.16:0/728813491 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbf3015e750 msgr2=0x7fbf301618a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:43.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.609+0000 7fbf37600640 1 --2- 172.21.15.16:0/728813491 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbf3015e750 0x7fbf301618a0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fbf1c0af000 tx=0x7fbf1c01ba30 comp rx=0 tx=0).stop 2024-03-31T15:03:43.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.609+0000 7f0294a00640 1 -- 172.21.15.16:0/3511530985 shutdown_connections 2024-03-31T15:03:43.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.609+0000 7f0294a00640 1 -- 172.21.15.16:0/3511530985 wait complete. 2024-03-31T15:03:43.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.609+0000 7fbf37600640 1 -- 172.21.15.16:0/728813491 shutdown_connections 2024-03-31T15:03:43.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.609+0000 7fbf37600640 1 --2- 172.21.15.16:0/728813491 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fbf10068580 0x7fbf1006aa40 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:43.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.609+0000 7fbf37600640 1 --2- 172.21.15.16:0/728813491 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbf3015e750 0x7fbf301618a0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:43.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.609+0000 7fbf37600640 1 --2- 172.21.15.16:0/728813491 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fbf300cca10 0x7fbf30161360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:43.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.609+0000 7fbf37600640 1 --2- 172.21.15.16:0/728813491 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbf300ada20 0x7fbf30202080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:43.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.609+0000 7fbf37600640 1 -- 172.21.15.16:0/728813491 >> 172.21.15.16:0/728813491 conn(0x7fbf300b1d30 msgr2=0x7fbf300cfc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:03:43.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.609+0000 7fbf37600640 1 -- 172.21.15.16:0/728813491 shutdown_connections 2024-03-31T15:03:43.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.609+0000 7fbf37600640 1 -- 172.21.15.16:0/728813491 wait complete. 2024-03-31T15:03:43.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.613+0000 7f434ec00640 1 -- 172.21.15.16:0/3043022218 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7f4320005590 con 0x7f43480d0970 2024-03-31T15:03:43.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.613+0000 7f433fe00640 1 -- 172.21.15.16:0/3043022218 <== mon.2 v2:172.21.15.89:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f4334056660 con 0x7f43480d0970 2024-03-31T15:03:43.612 INFO:teuthology.orchestra.run.smithi016.stdout:64424509444 2024-03-31T15:03:43.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.617+0000 7f434ec00640 1 -- 172.21.15.16:0/3043022218 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f4328068380 msgr2=0x7f432806a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:43.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.617+0000 7f434ec00640 1 --2- 172.21.15.16:0/3043022218 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f4328068380 0x7f432806a840 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f433801cd10 tx=0x7f43380a6000 comp rx=0 tx=0).stop 2024-03-31T15:03:43.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.617+0000 7f434ec00640 1 -- 172.21.15.16:0/3043022218 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f43480d0970 msgr2=0x7f4348154e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:43.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.617+0000 7f434ec00640 1 --2- 172.21.15.16:0/3043022218 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f43480d0970 0x7f4348154e30 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f433400b410 tx=0x7f433400b8d0 comp rx=0 tx=0).stop 2024-03-31T15:03:43.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.617+0000 7f434ec00640 1 -- 172.21.15.16:0/3043022218 shutdown_connections 2024-03-31T15:03:43.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.617+0000 7f434ec00640 1 --2- 172.21.15.16:0/3043022218 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f4328068380 0x7f432806a840 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:43.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.617+0000 7f434ec00640 1 --2- 172.21.15.16:0/3043022218 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4348162ae0 0x7f43480b2760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:43.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.617+0000 7f434ec00640 1 --2- 172.21.15.16:0/3043022218 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f43480d0970 0x7f4348154e30 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:43.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.617+0000 7f434ec00640 1 --2- 172.21.15.16:0/3043022218 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f434806f580 0x7f434815ed00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:43.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.617+0000 7f434ec00640 1 -- 172.21.15.16:0/3043022218 >> 172.21.15.16:0/3043022218 conn(0x7f434806b620 msgr2=0x7f4348164680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:03:43.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.617+0000 7f434ec00640 1 -- 172.21.15.16:0/3043022218 shutdown_connections 2024-03-31T15:03:43.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.617+0000 7f434ec00640 1 -- 172.21.15.16:0/3043022218 wait complete. 2024-03-31T15:03:43.639 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542152 got 60129542150 for osd.3 2024-03-31T15:03:43.647 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509445 got 64424509444 for osd.2 2024-03-31T15:03:43.647 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542152 got 60129542150 for osd.1 2024-03-31T15:03:43.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.653+0000 7fde73200640 1 -- 172.21.15.16:0/4115200107 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7fde74155f90 con 0x7fde740cf160 2024-03-31T15:03:43.649 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509447 got 64424509446 for osd.5 2024-03-31T15:03:43.650 INFO:teuthology.orchestra.run.smithi016.stdout:64424509447 2024-03-31T15:03:43.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.653+0000 7fde6a000640 1 -- 172.21.15.16:0/4115200107 <== mon.2 v2:172.21.15.89:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fde741f8330 con 0x7fde740cf160 2024-03-31T15:03:43.651 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509446 got 64424509445 for osd.4 2024-03-31T15:03:43.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.653+0000 7fde73200640 1 -- 172.21.15.16:0/4115200107 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fde54068380 msgr2=0x7fde5406a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:43.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.653+0000 7fde73200640 1 --2- 172.21.15.16:0/4115200107 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fde54068380 0x7fde5406a840 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fde6c0ae000 tx=0x7fde6c0a7070 comp rx=0 tx=0).stop 2024-03-31T15:03:43.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.653+0000 7fde73200640 1 -- 172.21.15.16:0/4115200107 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fde740cf160 msgr2=0x7fde741f7df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:43.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.653+0000 7fde73200640 1 --2- 172.21.15.16:0/4115200107 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fde740cf160 0x7fde741f7df0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fde6400d800 tx=0x7fde6400dcc0 comp rx=0 tx=0).stop 2024-03-31T15:03:43.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.653+0000 7fde73200640 1 -- 172.21.15.16:0/4115200107 shutdown_connections 2024-03-31T15:03:43.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.653+0000 7fde73200640 1 --2- 172.21.15.16:0/4115200107 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fde54068380 0x7fde5406a840 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:43.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.653+0000 7fde73200640 1 --2- 172.21.15.16:0/4115200107 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fde740cf540 0x7fde7415cdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:43.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.653+0000 7fde73200640 1 --2- 172.21.15.16:0/4115200107 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fde740cf160 0x7fde741f7df0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:43.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.653+0000 7fde73200640 1 --2- 172.21.15.16:0/4115200107 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fde740ceab0 0x7fde74159970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:43.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.653+0000 7fde73200640 1 -- 172.21.15.16:0/4115200107 >> 172.21.15.16:0/4115200107 conn(0x7fde740b1c50 msgr2=0x7fde740b11b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:03:43.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.653+0000 7fde73200640 1 -- 172.21.15.16:0/4115200107 shutdown_connections 2024-03-31T15:03:43.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.653+0000 7fde73200640 1 -- 172.21.15.16:0/4115200107 wait complete. 2024-03-31T15:03:43.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.657+0000 7f7805c00640 1 -- 172.21.15.16:0/449655726 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7f7800156180 con 0x7f78000b35f0 2024-03-31T15:03:43.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.661+0000 7f77f7e00640 1 -- 172.21.15.16:0/449655726 <== mon.2 v2:172.21.15.89:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f77f00f19e0 con 0x7f78000b35f0 2024-03-31T15:03:43.656 INFO:teuthology.orchestra.run.smithi016.stdout:60129542150 2024-03-31T15:03:43.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.661+0000 7f7805c00640 1 -- 172.21.15.16:0/449655726 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f77e0068480 msgr2=0x7f77e006a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:43.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.661+0000 7f7805c00640 1 --2- 172.21.15.16:0/449655726 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f77e0068480 0x7f77e006a940 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f78000cf780 tx=0x7f77ec009040 comp rx=0 tx=0).stop 2024-03-31T15:03:43.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.661+0000 7f7805c00640 1 -- 172.21.15.16:0/449655726 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f78000b35f0 msgr2=0x7f7800204ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:43.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.661+0000 7f7805c00640 1 --2- 172.21.15.16:0/449655726 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f78000b35f0 0x7f7800204ff0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f77f00ad040 tx=0x7f77f001ade0 comp rx=0 tx=0).stop 2024-03-31T15:03:43.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.661+0000 7f7805c00640 1 -- 172.21.15.16:0/449655726 shutdown_connections 2024-03-31T15:03:43.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.661+0000 7f7805c00640 1 --2- 172.21.15.16:0/449655726 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f77e0068480 0x7f77e006a940 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:43.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.661+0000 7f7805c00640 1 --2- 172.21.15.16:0/449655726 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f780015be60 0x7f7800151110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:43.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.661+0000 7f7805c00640 1 --2- 172.21.15.16:0/449655726 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f78000d0620 0x7f7800204630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:43.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.661+0000 7f7805c00640 1 --2- 172.21.15.16:0/449655726 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f78000b35f0 0x7f7800204ff0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:43.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.661+0000 7f7805c00640 1 -- 172.21.15.16:0/449655726 >> 172.21.15.16:0/449655726 conn(0x7f780006b620 msgr2=0x7f78000ace90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:03:43.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.661+0000 7f7805c00640 1 -- 172.21.15.16:0/449655726 shutdown_connections 2024-03-31T15:03:43.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:43.661+0000 7f7805c00640 1 -- 172.21.15.16:0/449655726 wait complete. 2024-03-31T15:03:43.674 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509448 got 64424509447 for osd.7 2024-03-31T15:03:43.679 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542152 got 60129542150 for osd.6 2024-03-31T15:03:44.606 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-03-31T15:03:44.640 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-03-31T15:03:44.647 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-03-31T15:03:44.648 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-03-31T15:03:44.650 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-03-31T15:03:44.652 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-03-31T15:03:44.675 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-03-31T15:03:44.680 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-03-31T15:03:44.836 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.845+0000 7f970f200640 1 Processor -- start 2024-03-31T15:03:44.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.845+0000 7f970f200640 1 -- start start 2024-03-31T15:03:44.839 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.845+0000 7f970f200640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f970806c140 0x7f97081511d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:44.839 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.845+0000 7f970f200640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f9708072bf0 0x7f9708073080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:44.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.845+0000 7f9707e00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f970806c140 0x7f97081511d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:44.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.849+0000 7f970d200640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f9708072bf0 0x7f9708073080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:44.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.849+0000 7f970d200640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f9708072bf0 0x7f9708073080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39154/0 (socket says 172.21.15.16:39154) 2024-03-31T15:03:44.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.849+0000 7f970d200640 1 -- 172.21.15.16:0/1469185316 learned_addr learned my addr 172.21.15.16:0/1469185316 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:03:44.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.849+0000 7f970f200640 1 -- 172.21.15.16:0/1469185316 --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f97080d0b70 con 0x7f970815a720 2024-03-31T15:03:44.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.849+0000 7f970f200640 1 -- 172.21.15.16:0/1469185316 --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f9708072750 con 0x7f9708072bf0 2024-03-31T15:03:44.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.849+0000 7f970f200640 1 -- 172.21.15.16:0/1469185316 --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f970815ac70 con 0x7f970806c140 2024-03-31T15:03:44.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.849+0000 7f9707e00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f970806c140 0x7f97081511d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.16:36146/0 (socket says 172.21.15.16:36146) 2024-03-31T15:03:44.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.849+0000 7f970d200640 1 -- 172.21.15.16:0/1469185316 >> v2:172.21.15.89:3300/0 conn(0x7f970806c140 msgr2=0x7f97081511d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:03:44.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.849+0000 7f970d200640 1 --2- 172.21.15.16:0/1469185316 >> v2:172.21.15.89:3300/0 conn(0x7f970806c140 0x7f97081511d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:44.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.849+0000 7f970d200640 1 -- 172.21.15.16:0/1469185316 >> v1:172.21.15.16:6789/0 conn(0x7f970815a720 legacy=0x7f97080d0460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:03:44.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.849+0000 7f970d200640 1 -- 172.21.15.16:0/1469185316 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f970815af50 con 0x7f9708072bf0 2024-03-31T15:03:44.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.853+0000 7f970d200640 1 --2- 172.21.15.16:0/1469185316 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9708072bf0 0x7f9708073080 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f9700009ed0 tx=0x7f97000279c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ce618ca99336afa server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:03:44.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.853+0000 7f9706a00640 1 -- 172.21.15.16:0/1469185316 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97000b1590 con 0x7f9708072bf0 2024-03-31T15:03:44.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.853+0000 7f9706a00640 1 -- 172.21.15.16:0/1469185316 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f97000b1730 con 0x7f9708072bf0 2024-03-31T15:03:44.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.853+0000 7f970f200640 1 -- 172.21.15.16:0/1469185316 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9708072bf0 msgr2=0x7f9708073080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:44.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.853+0000 7f970f200640 1 --2- 172.21.15.16:0/1469185316 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9708072bf0 0x7f9708073080 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f9700009ed0 tx=0x7f97000279c0 comp rx=0 tx=0).stop 2024-03-31T15:03:44.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.857+0000 7f970f200640 1 -- 172.21.15.16:0/1469185316 shutdown_connections 2024-03-31T15:03:44.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.857+0000 7f970f200640 1 --2- 172.21.15.16:0/1469185316 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9708072bf0 0x7f9708073080 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:44.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.857+0000 7f970f200640 1 --2- 172.21.15.16:0/1469185316 >> v2:172.21.15.89:3300/0 conn(0x7f970806c140 0x7f97081511d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:44.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.857+0000 7f970f200640 1 -- 172.21.15.16:0/1469185316 >> 172.21.15.16:0/1469185316 conn(0x7f970806b620 msgr2=0x7f970806ba30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:03:44.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.857+0000 7f970f200640 1 -- 172.21.15.16:0/1469185316 shutdown_connections 2024-03-31T15:03:44.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.857+0000 7f970f200640 1 -- 172.21.15.16:0/1469185316 wait complete. 2024-03-31T15:03:44.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.857+0000 7f970f200640 1 Processor -- start 2024-03-31T15:03:44.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.861+0000 7f970f200640 1 -- start start 2024-03-31T15:03:44.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.861+0000 7f970f200640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f970806c140 0x7f97080cbf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:44.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.861+0000 7f970f200640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9708072bf0 0x7f97081534a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:44.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.861+0000 7f970f200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f970815a720 0x7f9708146680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:44.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.861+0000 7f970f200640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f970815b8e0 con 0x7f970815a720 2024-03-31T15:03:44.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.861+0000 7f970f200640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f970815b510 con 0x7f9708072bf0 2024-03-31T15:03:44.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.861+0000 7f970f200640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f9708071740 con 0x7f970806c140 2024-03-31T15:03:44.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.861+0000 7f9707e00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f970806c140 0x7f97080cbf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:44.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.861+0000 7f9707e00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f970806c140 0x7f97080cbf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3301/0 says I am v2:172.21.15.16:41696/0 (socket says 172.21.15.16:41696) 2024-03-31T15:03:44.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.861+0000 7f9707e00640 1 -- 172.21.15.16:0/3278691659 learned_addr learned my addr 172.21.15.16:0/3278691659 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:03:44.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.861+0000 7f9707400640 1 --2- 172.21.15.16:0/3278691659 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9708072bf0 0x7f97081534a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:44.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.861+0000 7f970d200640 1 --2- 172.21.15.16:0/3278691659 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f970815a720 0x7f9708146680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:44.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.861+0000 7f9707400640 1 -- 172.21.15.16:0/3278691659 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f970806c140 msgr2=0x7f97080cbf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:44.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.861+0000 7f9707400640 1 --2- 172.21.15.16:0/3278691659 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f970806c140 0x7f97080cbf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:44.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.861+0000 7f9707400640 1 -- 172.21.15.16:0/3278691659 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f970815a720 msgr2=0x7f9708146680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:44.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.861+0000 7f9707400640 1 --2- 172.21.15.16:0/3278691659 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f970815a720 0x7f9708146680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:44.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.861+0000 7f9707400640 1 -- 172.21.15.16:0/3278691659 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9708152400 con 0x7f9708072bf0 2024-03-31T15:03:44.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.861+0000 7f9707e00640 1 --2- 172.21.15.16:0/3278691659 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f970806c140 0x7f97080cbf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:03:44.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.861+0000 7f9707400640 1 --2- 172.21.15.16:0/3278691659 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9708072bf0 0x7f97081534a0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f96f400b440 tx=0x7f96f4009f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:03:44.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.861+0000 7f96ffe00640 1 -- 172.21.15.16:0/3278691659 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96f4026020 con 0x7f9708072bf0 2024-03-31T15:03:44.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.861+0000 7f96ffe00640 1 -- 172.21.15.16:0/3278691659 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f96f4022640 con 0x7f9708072bf0 2024-03-31T15:03:44.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.865+0000 7f96ffe00640 1 -- 172.21.15.16:0/3278691659 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96f4022a30 con 0x7f9708072bf0 2024-03-31T15:03:44.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.865+0000 7f970f200640 1 -- 172.21.15.16:0/3278691659 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f97081526e0 con 0x7f9708072bf0 2024-03-31T15:03:44.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.865+0000 7f970f200640 1 -- 172.21.15.16:0/3278691659 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9708152ca0 con 0x7f9708072bf0 2024-03-31T15:03:44.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.865+0000 7f970f200640 1 -- 172.21.15.16:0/3278691659 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f970804f8c0 con 0x7f9708072bf0 2024-03-31T15:03:44.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.869+0000 7f96ffe00640 1 -- 172.21.15.16:0/3278691659 <== mon.1 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f96f4029020 con 0x7f9708072bf0 2024-03-31T15:03:44.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.869+0000 7f96ffe00640 1 --2- 172.21.15.16:0/3278691659 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f96e8068380 0x7f96e806a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:44.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.869+0000 7f9707e00640 1 --2- 172.21.15.16:0/3278691659 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f96e8068380 0x7f96e806a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:44.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.869+0000 7f96ffe00640 1 -- 172.21.15.16:0/3278691659 <== mon.1 v2:172.21.15.89:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f96f40975b0 con 0x7f9708072bf0 2024-03-31T15:03:44.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.869+0000 7f9707e00640 1 --2- 172.21.15.16:0/3278691659 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f96e8068380 0x7f96e806a840 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f96f8007f00 tx=0x7f96f8008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:03:44.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.873+0000 7f96ffe00640 1 -- 172.21.15.16:0/3278691659 <== mon.1 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f96f4030020 con 0x7f9708072bf0 2024-03-31T15:03:44.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.901+0000 7fb675a00640 1 Processor -- start 2024-03-31T15:03:44.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.901+0000 7fb675a00640 1 -- start start 2024-03-31T15:03:44.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.901+0000 7fb675a00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb6700b4660 0x7fb6700b7710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:44.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.901+0000 7fb675a00640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb6700b5dd0 con 0x7fb67015c270 2024-03-31T15:03:44.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.901+0000 7fb675a00640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb6700ccc40 con 0x7fb67015fde0 2024-03-31T15:03:44.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.901+0000 7fb675a00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fb6700cca10 con 0x7fb6700b4660 2024-03-31T15:03:44.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.901+0000 7fb66e000640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb6700b4660 0x7fb6700b7710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:44.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.901+0000 7fb66e000640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb6700b4660 0x7fb6700b7710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3301/0 says I am v2:172.21.15.16:41700/0 (socket says 172.21.15.16:41700) 2024-03-31T15:03:44.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.901+0000 7fb66e000640 1 -- 172.21.15.16:0/687425562 learned_addr learned my addr 172.21.15.16:0/687425562 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:03:44.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.901+0000 7f5352800640 1 Processor -- start 2024-03-31T15:03:44.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.905+0000 7f5352800640 1 -- start start 2024-03-31T15:03:44.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.905+0000 7f573a400640 1 Processor -- start 2024-03-31T15:03:44.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.905+0000 7f573a400640 1 -- start start 2024-03-31T15:03:44.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.901+0000 7fb66e000640 1 -- 172.21.15.16:0/687425562 >> v1:172.21.15.89:6789/0 conn(0x7fb67015fde0 legacy=0x7fb670160250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:03:44.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.901+0000 7fb66e000640 1 -- 172.21.15.16:0/687425562 >> v1:172.21.15.16:6789/0 conn(0x7fb67015c270 legacy=0x7fb6700b7000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:03:44.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.901+0000 7fb66e000640 1 -- 172.21.15.16:0/687425562 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6700ccf20 con 0x7fb6700b4660 2024-03-31T15:03:44.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.905+0000 7f5352800640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f534c147b50 0x7f534c0cfcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:44.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.905+0000 7f5352800640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f534c167130 0x7f534c06f920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:44.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.905+0000 7f5352800640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f534c06fe60 0x7f534c15fb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:44.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.905+0000 7f5352800640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f534c1460d0 con 0x7f534c147b50 2024-03-31T15:03:44.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.905+0000 7f5352800640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f534c163020 con 0x7f534c06fe60 2024-03-31T15:03:44.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.905+0000 7f5352800640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f534c163300 con 0x7f534c167130 2024-03-31T15:03:44.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.905+0000 7f573a400640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f5734147220 0x7f5734072bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:44.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.905+0000 7f573a400640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f5734070b50 0x7f57340b3130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:44.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.905+0000 7f573a400640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f57341620d0 con 0x7f57340b2cd0 2024-03-31T15:03:44.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.905+0000 7f573a400640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f57340b3670 con 0x7f5734147220 2024-03-31T15:03:44.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.905+0000 7f573a400640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f5734162ce0 con 0x7f5734070b50 2024-03-31T15:03:44.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.905+0000 7f534aa00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f534c167130 0x7f534c06f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:44.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.905+0000 7f5733e00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f5734070b50 0x7f57340b3130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:44.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.905+0000 7f5733e00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f5734070b50 0x7f57340b3130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3301/0 says I am v2:172.21.15.16:41718/0 (socket says 172.21.15.16:41718) 2024-03-31T15:03:44.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.905+0000 7f5733e00640 1 -- 172.21.15.16:0/2348681755 learned_addr learned my addr 172.21.15.16:0/2348681755 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:03:44.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.905+0000 7f534be00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f534c06fe60 0x7f534c15fb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:44.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.905+0000 7f534be00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f534c06fe60 0x7f534c15fb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.16:36166/0 (socket says 172.21.15.16:36166) 2024-03-31T15:03:44.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.905+0000 7f534be00640 1 -- 172.21.15.16:0/4290298017 learned_addr learned my addr 172.21.15.16:0/4290298017 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:03:44.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.905+0000 7f5733e00640 1 -- 172.21.15.16:0/2348681755 >> v2:172.21.15.89:3300/0 conn(0x7f5734147220 msgr2=0x7f5734072bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:03:44.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.905+0000 7f5733400640 1 --2- 172.21.15.16:0/2348681755 >> v2:172.21.15.89:3300/0 conn(0x7f5734147220 0x7f5734072bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:44.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.909+0000 7f5733e00640 1 --2- 172.21.15.16:0/2348681755 >> v2:172.21.15.89:3300/0 conn(0x7f5734147220 0x7f5734072bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:44.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.909+0000 7f5733e00640 1 -- 172.21.15.16:0/2348681755 >> v1:172.21.15.16:6789/0 conn(0x7f57340b2cd0 legacy=0x7f573406b620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:03:44.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.909+0000 7f5733e00640 1 -- 172.21.15.16:0/2348681755 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57341623b0 con 0x7f5734070b50 2024-03-31T15:03:44.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.909+0000 7f5733400640 1 --2- 172.21.15.16:0/2348681755 >> v2:172.21.15.89:3300/0 conn(0x7f5734147220 0x7f5734072bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:03:44.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.909+0000 7fa718400640 1 Processor -- start 2024-03-31T15:03:44.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.909+0000 7f534b400640 1 --2- 172.21.15.16:0/4290298017 >> v2:172.21.15.16:3300/0 conn(0x7f534c147b50 0x7f534c0cfcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:44.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.909+0000 7f534be00640 1 -- 172.21.15.16:0/4290298017 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f534c167130 msgr2=0x7f534c06f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:03:44.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.909+0000 7f534be00640 1 --2- 172.21.15.16:0/4290298017 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f534c167130 0x7f534c06f920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:44.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.909+0000 7f534be00640 1 -- 172.21.15.16:0/4290298017 >> v2:172.21.15.16:3300/0 conn(0x7f534c147b50 msgr2=0x7f534c0cfcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:03:44.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.909+0000 7f534be00640 1 --2- 172.21.15.16:0/4290298017 >> v2:172.21.15.16:3300/0 conn(0x7f534c147b50 0x7f534c0cfcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:44.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.909+0000 7f534be00640 1 -- 172.21.15.16:0/4290298017 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f534c1635e0 con 0x7f534c06fe60 2024-03-31T15:03:44.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.909+0000 7f534aa00640 1 --2- 172.21.15.16:0/4290298017 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f534c167130 0x7f534c06f920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:03:44.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.909+0000 7fb66e000640 1 --2- 172.21.15.16:0/687425562 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb6700b4660 0x7fb6700b7710 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fb65c0098a0 tx=0x7fb65c01c500 comp rx=0 tx=0).ready entity=mon.2 client_cookie=29a374c32e003760 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:03:44.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.909+0000 7f534b400640 1 --2- 172.21.15.16:0/4290298017 >> v2:172.21.15.16:3300/0 conn(0x7f534c147b50 0x7f534c0cfcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:03:44.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.909+0000 7fb66d600640 1 -- 172.21.15.16:0/687425562 <== mon.2 v2:172.21.15.89:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb65c0b1070 con 0x7fb6700b4660 2024-03-31T15:03:44.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.909+0000 7fb66d600640 1 -- 172.21.15.16:0/687425562 <== mon.2 v2:172.21.15.89:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb65c0ac410 con 0x7fb6700b4660 2024-03-31T15:03:44.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.909+0000 7fb66d600640 1 -- 172.21.15.16:0/687425562 <== mon.2 v2:172.21.15.89:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb65c0ac7e0 con 0x7fb6700b4660 2024-03-31T15:03:44.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.909+0000 7fa718400640 1 -- start start 2024-03-31T15:03:44.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.909+0000 7fa718400640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fa7100cf6a0 0x7fa710071500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:44.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.909+0000 7fa718400640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fa7100b3030 0x7fa7100b34c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:44.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.909+0000 7fa718400640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa710073a10 con 0x7fa71015a660 2024-03-31T15:03:44.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.909+0000 7fa718400640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7fa71015aa40 con 0x7fa7100cf6a0 2024-03-31T15:03:44.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.909+0000 7fa718400640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fa7100732a0 con 0x7fa7100b3030 2024-03-31T15:03:44.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.909+0000 7fb675a00640 1 -- 172.21.15.16:0/687425562 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb6700b4660 msgr2=0x7fb6700b7710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:44.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.909+0000 7fb675a00640 1 --2- 172.21.15.16:0/687425562 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb6700b4660 0x7fb6700b7710 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fb65c0098a0 tx=0x7fb65c01c500 comp rx=0 tx=0).stop 2024-03-31T15:03:44.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.913+0000 7fa715000640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fa7100cf6a0 0x7fa710071500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:44.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.913+0000 7fa716400640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fa7100b3030 0x7fa7100b34c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:44.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.913+0000 7fa716400640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fa7100b3030 0x7fa7100b34c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3301/0 says I am v2:172.21.15.16:41724/0 (socket says 172.21.15.16:41724) 2024-03-31T15:03:44.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.913+0000 7fa716400640 1 -- 172.21.15.16:0/704624979 learned_addr learned my addr 172.21.15.16:0/704624979 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:03:44.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.913+0000 7f5733e00640 1 --2- 172.21.15.16:0/2348681755 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f5734070b50 0x7f57340b3130 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f5724009870 tx=0x7f572401c4f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cc976eae216c6aa5 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:03:44.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.913+0000 7fa70fe00640 1 -- 172.21.15.16:0/704624979 <== mon.1 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2010917915 0 0) 0x7fa710073a10 con 0x7fa71015a660 2024-03-31T15:03:44.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.913+0000 7fa70fe00640 1 -- 172.21.15.16:0/704624979 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa6f8003530 con 0x7fa71015a660 2024-03-31T15:03:44.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.913+0000 7fa716400640 1 -- 172.21.15.16:0/704624979 >> v2:172.21.15.16:3300/0 conn(0x7fa7100cf6a0 msgr2=0x7fa710071500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:03:44.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.913+0000 7fa716400640 1 --2- 172.21.15.16:0/704624979 >> v2:172.21.15.16:3300/0 conn(0x7fa7100cf6a0 0x7fa710071500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:44.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.913+0000 7fa716400640 1 -- 172.21.15.16:0/704624979 >> v1:172.21.15.89:6789/0 conn(0x7fa71015a660 legacy=0x7fa7100ce860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:44.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.913+0000 7fa716400640 1 -- 172.21.15.16:0/704624979 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7100b5110 con 0x7fa7100b3030 2024-03-31T15:03:44.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.913+0000 7f5732000640 1 -- 172.21.15.16:0/2348681755 <== mon.2 v2:172.21.15.89:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57240b1070 con 0x7f5734070b50 2024-03-31T15:03:44.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.913+0000 7f5732000640 1 -- 172.21.15.16:0/2348681755 <== mon.2 v2:172.21.15.89:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f57240ac630 con 0x7f5734070b50 2024-03-31T15:03:44.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.913+0000 7fb675a00640 1 -- 172.21.15.16:0/687425562 shutdown_connections 2024-03-31T15:03:44.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.913+0000 7fb675a00640 1 --2- 172.21.15.16:0/687425562 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb6700b4660 0x7fb6700b7710 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:44.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.913+0000 7fb675a00640 1 -- 172.21.15.16:0/687425562 >> 172.21.15.16:0/687425562 conn(0x7fb6700b3a60 msgr2=0x7fb6700b3e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:03:44.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.913+0000 7fb675a00640 1 -- 172.21.15.16:0/687425562 shutdown_connections 2024-03-31T15:03:44.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.913+0000 7fb675a00640 1 -- 172.21.15.16:0/687425562 wait complete. 2024-03-31T15:03:44.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.913+0000 7f573a400640 1 -- 172.21.15.16:0/2348681755 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f5734070b50 msgr2=0x7f57340b3130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:44.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.913+0000 7f5732000640 1 -- 172.21.15.16:0/2348681755 <== mon.2 v2:172.21.15.89:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57240b1070 con 0x7f5734070b50 2024-03-31T15:03:44.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.913+0000 7f573a400640 1 --2- 172.21.15.16:0/2348681755 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f5734070b50 0x7f57340b3130 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f5724009870 tx=0x7f572401c4f0 comp rx=0 tx=0).stop 2024-03-31T15:03:44.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.913+0000 7f534be00640 1 --2- 172.21.15.16:0/4290298017 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f534c06fe60 0x7f534c15fb00 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f534c0b3d50 tx=0x7f534401c9f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=17f4b3dea4c775b0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:03:44.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.913+0000 7f534a000640 1 -- 172.21.15.16:0/4290298017 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53440a5780 con 0x7f534c06fe60 2024-03-31T15:03:44.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.913+0000 7f534a000640 1 -- 172.21.15.16:0/4290298017 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f53440a5920 con 0x7f534c06fe60 2024-03-31T15:03:44.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.913+0000 7f534a000640 1 -- 172.21.15.16:0/4290298017 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53440a5cf0 con 0x7f534c06fe60 2024-03-31T15:03:44.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.913+0000 7f5352800640 1 -- 172.21.15.16:0/4290298017 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f534c06fe60 msgr2=0x7f534c15fb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:44.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.913+0000 7f5352800640 1 --2- 172.21.15.16:0/4290298017 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f534c06fe60 0x7f534c15fb00 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f534c0b3d50 tx=0x7f534401c9f0 comp rx=0 tx=0).stop 2024-03-31T15:03:44.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.913+0000 7f573a400640 1 -- 172.21.15.16:0/2348681755 shutdown_connections 2024-03-31T15:03:44.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.913+0000 7f573a400640 1 --2- 172.21.15.16:0/2348681755 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f5734070b50 0x7f57340b3130 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:44.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.913+0000 7f573a400640 1 --2- 172.21.15.16:0/2348681755 >> v2:172.21.15.89:3300/0 conn(0x7f5734147220 0x7f5734072bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:44.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.913+0000 7f573a400640 1 -- 172.21.15.16:0/2348681755 >> 172.21.15.16:0/2348681755 conn(0x7f573406d090 msgr2=0x7f573406e7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:03:44.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.913+0000 7f5352800640 1 -- 172.21.15.16:0/4290298017 shutdown_connections 2024-03-31T15:03:44.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.913+0000 7f5352800640 1 --2- 172.21.15.16:0/4290298017 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f534c06fe60 0x7f534c15fb00 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:44.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.913+0000 7f5352800640 1 --2- 172.21.15.16:0/4290298017 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f534c167130 0x7f534c06f920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:44.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.913+0000 7f5352800640 1 --2- 172.21.15.16:0/4290298017 >> v2:172.21.15.16:3300/0 conn(0x7f534c147b50 0x7f534c0cfcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:44.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.913+0000 7f5352800640 1 -- 172.21.15.16:0/4290298017 >> 172.21.15.16:0/4290298017 conn(0x7f534c067330 msgr2=0x7f534c067740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:03:44.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.913+0000 7f573a400640 1 -- 172.21.15.16:0/2348681755 shutdown_connections 2024-03-31T15:03:44.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.917+0000 7f8720e00640 1 Processor -- start 2024-03-31T15:03:44.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.917+0000 7f8720e00640 1 -- start start 2024-03-31T15:03:44.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.917+0000 7fb675a00640 1 Processor -- start 2024-03-31T15:03:44.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.913+0000 7f573a400640 1 -- 172.21.15.16:0/2348681755 wait complete. 2024-03-31T15:03:44.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.917+0000 7f573a400640 1 Processor -- start 2024-03-31T15:03:44.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.913+0000 7f5352800640 1 -- 172.21.15.16:0/4290298017 shutdown_connections 2024-03-31T15:03:44.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.913+0000 7f5352800640 1 -- 172.21.15.16:0/4290298017 wait complete. 2024-03-31T15:03:44.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.917+0000 7f5352800640 1 Processor -- start 2024-03-31T15:03:44.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.917+0000 7fb675a00640 1 -- start start 2024-03-31T15:03:44.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.917+0000 7f573a400640 1 -- start start 2024-03-31T15:03:44.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.917+0000 7fa716400640 1 --2- 172.21.15.16:0/704624979 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fa7100b3030 0x7fa7100b34c0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fa708007ac0 tx=0x7fa70801f110 comp rx=0 tx=0).ready entity=mon.2 client_cookie=63d67c5a1da3aeab server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:03:44.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.917+0000 7f5352800640 1 -- start start 2024-03-31T15:03:44.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.917+0000 7fa70fe00640 1 -- 172.21.15.16:0/704624979 <== mon.2 v2:172.21.15.89:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7080b3070 con 0x7fa7100b3030 2024-03-31T15:03:44.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.917+0000 7fa70fe00640 1 -- 172.21.15.16:0/704624979 <== mon.2 v2:172.21.15.89:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa7080092d0 con 0x7fa7100b3030 2024-03-31T15:03:44.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.917+0000 7fa70fe00640 1 -- 172.21.15.16:0/704624979 <== mon.2 v2:172.21.15.89:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7080b3070 con 0x7fa7100b3030 2024-03-31T15:03:44.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.917+0000 7f8720e00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f871c0ada60 0x7f871c0af610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:44.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.917+0000 7f8720e00640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f871c0cfdb0 con 0x7f871c16f650 2024-03-31T15:03:44.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.917+0000 7f8720e00640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f871c0b4f40 con 0x7f871c0cce10 2024-03-31T15:03:44.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.917+0000 7f8720e00640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f871c0b5220 con 0x7f871c0ada60 2024-03-31T15:03:44.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.917+0000 7fb675a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb6700b4660 0x7fb67015d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:44.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.921+0000 7fb675a00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb67015c270 0x7fb67015ee50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:44.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.921+0000 7fb675a00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb67015fde0 0x7fb670153790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:44.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.921+0000 7fb66ea00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb6700b4660 0x7fb67015d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:44.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.921+0000 7fa718400640 1 -- 172.21.15.16:0/704624979 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fa7100b3030 msgr2=0x7fa7100b34c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:44.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.921+0000 7fa718400640 1 --2- 172.21.15.16:0/704624979 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fa7100b3030 0x7fa7100b34c0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fa708007ac0 tx=0x7fa70801f110 comp rx=0 tx=0).stop 2024-03-31T15:03:44.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.921+0000 7f8719400640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f871c0ada60 0x7f871c0af610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:44.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.921+0000 7f8719400640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f871c0ada60 0x7f871c0af610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.16:36184/0 (socket says 172.21.15.16:36184) 2024-03-31T15:03:44.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.921+0000 7f8719400640 1 -- 172.21.15.16:0/2555861768 learned_addr learned my addr 172.21.15.16:0/2555861768 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:03:44.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.921+0000 7fb66ea00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb6700b4660 0x7fb67015d1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39196/0 (socket says 172.21.15.16:39196) 2024-03-31T15:03:44.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.921+0000 7fb675a00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fb6700b7c50 con 0x7fb6700b4660 2024-03-31T15:03:44.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.921+0000 7fb675a00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fb67015c650 con 0x7fb67015fde0 2024-03-31T15:03:44.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.921+0000 7fb675a00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fb6700cdce0 con 0x7fb67015c270 2024-03-31T15:03:44.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.921+0000 7fb66e000640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb67015c270 0x7fb67015ee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:44.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.921+0000 7fa718400640 1 -- 172.21.15.16:0/704624979 shutdown_connections 2024-03-31T15:03:44.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.921+0000 7fb66ea00640 1 -- 172.21.15.16:0/570319002 learned_addr learned my addr 172.21.15.16:0/570319002 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:03:44.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.921+0000 7fa718400640 1 --2- 172.21.15.16:0/704624979 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fa7100b3030 0x7fa7100b34c0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:44.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.921+0000 7fa718400640 1 --2- 172.21.15.16:0/704624979 >> v2:172.21.15.16:3300/0 conn(0x7fa7100cf6a0 0x7fa710071500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:44.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.921+0000 7fa718400640 1 -- 172.21.15.16:0/704624979 >> 172.21.15.16:0/704624979 conn(0x7fa7100b1d30 msgr2=0x7fa7100b2140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:03:44.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.921+0000 7f8719400640 1 -- 172.21.15.16:0/2555861768 >> v1:172.21.15.89:6789/0 conn(0x7f871c0cce10 legacy=0x7f871c0cdea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:03:44.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.921+0000 7fa718400640 1 -- 172.21.15.16:0/704624979 shutdown_connections 2024-03-31T15:03:44.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.921+0000 7f8719400640 1 -- 172.21.15.16:0/2555861768 >> v1:172.21.15.16:6789/0 conn(0x7f871c16f650 legacy=0x7f871c0cc700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:03:44.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.921+0000 7f8719400640 1 -- 172.21.15.16:0/2555861768 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f871c0cf7f0 con 0x7f871c0ada60 2024-03-31T15:03:44.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.921+0000 7fb66f400640 1 --2- 172.21.15.16:0/570319002 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb67015fde0 0x7fb670153790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:44.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.921+0000 7f573a400640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f5734070b50 0x7f5734145630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:44.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.921+0000 7f573a400640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f57340b2cd0 0x7f57340ac010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:44.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.921+0000 7f573a400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5734147220 0x7f57340ac550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:44.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.921+0000 7fa718400640 1 -- 172.21.15.16:0/704624979 wait complete. 2024-03-31T15:03:44.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.921+0000 7f5733400640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f5734070b50 0x7f5734145630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:44.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.921+0000 7f573a400640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f5734073290 con 0x7f5734147220 2024-03-31T15:03:44.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.921+0000 7f573a400640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f5734073110 con 0x7f57340b2cd0 2024-03-31T15:03:44.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.921+0000 7f573a400640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f573416b2d0 con 0x7f5734070b50 2024-03-31T15:03:44.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.921+0000 7f5733400640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f5734070b50 0x7f5734145630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3301/0 says I am v2:172.21.15.16:41740/0 (socket says 172.21.15.16:41740) 2024-03-31T15:03:44.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.921+0000 7f5733400640 1 -- 172.21.15.16:0/1231396231 learned_addr learned my addr 172.21.15.16:0/1231396231 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:03:44.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.921+0000 7f5732a00640 1 --2- 172.21.15.16:0/1231396231 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f57340b2cd0 0x7f57340ac010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:44.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.921+0000 7fb66e000640 1 -- 172.21.15.16:0/570319002 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb67015fde0 msgr2=0x7fb670153790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:44.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.921+0000 7fb66e000640 1 --2- 172.21.15.16:0/570319002 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb67015fde0 0x7fb670153790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:44.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.921+0000 7fb66e000640 1 -- 172.21.15.16:0/570319002 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb6700b4660 msgr2=0x7fb67015d1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:44.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.921+0000 7fb66e000640 1 --2- 172.21.15.16:0/570319002 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb6700b4660 0x7fb67015d1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:44.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.921+0000 7fb66e000640 1 -- 172.21.15.16:0/570319002 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb670158e20 con 0x7fb67015c270 2024-03-31T15:03:44.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.921+0000 7f5352800640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f534c06fe60 0x7f534c1430e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:44.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.921+0000 7f5352800640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f534c147b50 0x7f534c144200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:44.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.921+0000 7f5352800640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f534c167130 0x7f534c144740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:44.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.921+0000 7f5352800640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f534c0d0380 con 0x7f534c147b50 2024-03-31T15:03:44.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.921+0000 7f5352800640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f534c0d0200 con 0x7f534c06fe60 2024-03-31T15:03:44.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.921+0000 7f5352800640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f534c0d2050 con 0x7f534c167130 2024-03-31T15:03:44.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.921+0000 7f5733e00640 1 --2- 172.21.15.16:0/1231396231 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5734147220 0x7f57340ac550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:44.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.921+0000 7f534be00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f534c167130 0x7f534c144740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:44.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.921+0000 7f534be00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f534c167130 0x7f534c144740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3301/0 says I am v2:172.21.15.16:41754/0 (socket says 172.21.15.16:41754) 2024-03-31T15:03:44.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.921+0000 7f534be00640 1 -- 172.21.15.16:0/244635572 learned_addr learned my addr 172.21.15.16:0/244635572 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:03:44.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.921+0000 7f534be00640 1 -- 172.21.15.16:0/244635572 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f534c06fe60 msgr2=0x7f534c1430e0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-31T15:03:44.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.921+0000 7f534be00640 1 --2- 172.21.15.16:0/244635572 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f534c06fe60 0x7f534c1430e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:44.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.921+0000 7f534aa00640 1 --2- 172.21.15.16:0/244635572 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f534c147b50 0x7f534c144200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:44.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.921+0000 7f534be00640 1 -- 172.21.15.16:0/244635572 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f534c147b50 msgr2=0x7f534c144200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:44.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.921+0000 7f534be00640 1 --2- 172.21.15.16:0/244635572 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f534c147b50 0x7f534c144200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:44.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.921+0000 7f534be00640 1 -- 172.21.15.16:0/244635572 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f534c144f70 con 0x7f534c167130 2024-03-31T15:03:44.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.921+0000 7f534aa00640 1 --2- 172.21.15.16:0/244635572 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f534c147b50 0x7f534c144200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:03:44.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.925+0000 7f534be00640 1 --2- 172.21.15.16:0/244635572 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f534c167130 0x7f534c144740 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f53440af040 tx=0x7f5344019790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:03:44.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.925+0000 7fa718400640 1 Processor -- start 2024-03-31T15:03:44.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.925+0000 7f5733e00640 1 -- 172.21.15.16:0/1231396231 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f5734070b50 msgr2=0x7f5734145630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:44.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.925+0000 7f5733e00640 1 --2- 172.21.15.16:0/1231396231 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f5734070b50 0x7f5734145630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:44.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.925+0000 7f5733e00640 1 -- 172.21.15.16:0/1231396231 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f57340b2cd0 msgr2=0x7f57340ac010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:44.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.925+0000 7f5733e00640 1 --2- 172.21.15.16:0/1231396231 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f57340b2cd0 0x7f57340ac010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:44.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.925+0000 7f5733e00640 1 -- 172.21.15.16:0/1231396231 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57340aa6e0 con 0x7f5734147220 2024-03-31T15:03:44.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.925+0000 7f5732a00640 1 --2- 172.21.15.16:0/1231396231 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f57340b2cd0 0x7f57340ac010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:03:44.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.925+0000 7f5343400640 1 -- 172.21.15.16:0/244635572 <== mon.2 v2:172.21.15.89:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53440bc020 con 0x7f534c167130 2024-03-31T15:03:44.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.925+0000 7fa718400640 1 -- start start 2024-03-31T15:03:44.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.925+0000 7fb66e000640 1 --2- 172.21.15.16:0/570319002 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb67015c270 0x7fb67015ee50 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fb65c0ac000 tx=0x7fb65c0192a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:03:44.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.925+0000 7fb666a00640 1 -- 172.21.15.16:0/570319002 <== mon.2 v2:172.21.15.89:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb65c0b1050 con 0x7fb67015c270 2024-03-31T15:03:44.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.925+0000 7fb666a00640 1 -- 172.21.15.16:0/570319002 <== mon.2 v2:172.21.15.89:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb65c0b2040 con 0x7fb67015c270 2024-03-31T15:03:44.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.925+0000 7fb666a00640 1 -- 172.21.15.16:0/570319002 <== mon.2 v2:172.21.15.89:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb65c0ace00 con 0x7fb67015c270 2024-03-31T15:03:44.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.925+0000 7fb675a00640 1 -- 172.21.15.16:0/570319002 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb67015d7c0 con 0x7fb67015c270 2024-03-31T15:03:44.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.925+0000 7fb675a00640 1 -- 172.21.15.16:0/570319002 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fb67015dd80 con 0x7fb67015c270 2024-03-31T15:03:44.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.925+0000 7f5733e00640 1 --2- 172.21.15.16:0/1231396231 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5734147220 0x7f57340ac550 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f57240ac000 tx=0x7f5724019290 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:03:44.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.925+0000 7f8719400640 1 --2- 172.21.15.16:0/2555861768 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f871c0ada60 0x7f871c0af610 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f87080098a0 tx=0x7f870801c4c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=43a35724a2d02a07 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:03:44.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.925+0000 7f5343400640 1 -- 172.21.15.16:0/244635572 <== mon.2 v2:172.21.15.89:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f53440b8670 con 0x7f534c167130 2024-03-31T15:03:44.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.925+0000 7fa718400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa7100b3030 0x7fa71014f450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:44.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.925+0000 7fa718400640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa7100cf6a0 0x7fa71015abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:44.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.925+0000 7fa718400640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fa71015a660 0x7fa7101529b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:44.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.925+0000 7f572b400640 1 -- 172.21.15.16:0/1231396231 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57240b1040 con 0x7f5734147220 2024-03-31T15:03:44.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.925+0000 7fb675a00640 1 -- 172.21.15.16:0/570319002 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb67004f8c0 con 0x7fb67015c270 2024-03-31T15:03:44.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.925+0000 7f5343400640 1 -- 172.21.15.16:0/244635572 <== mon.2 v2:172.21.15.89:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53440b8a20 con 0x7f534c167130 2024-03-31T15:03:44.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.925+0000 7f5352800640 1 -- 172.21.15.16:0/244635572 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f534c145250 con 0x7f534c167130 2024-03-31T15:03:44.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.925+0000 7f5352800640 1 -- 172.21.15.16:0/244635572 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f534c145810 con 0x7f534c167130 2024-03-31T15:03:44.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.925+0000 7f572b400640 1 -- 172.21.15.16:0/1231396231 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f57240bc070 con 0x7f5734147220 2024-03-31T15:03:44.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.925+0000 7f8718a00640 1 -- 172.21.15.16:0/2555861768 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87080b0070 con 0x7f871c0ada60 2024-03-31T15:03:44.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.925+0000 7f8718a00640 1 -- 172.21.15.16:0/2555861768 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f87080ab600 con 0x7f871c0ada60 2024-03-31T15:03:44.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.925+0000 7fa715000640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa7100cf6a0 0x7fa71015abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:44.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.925+0000 7f8718a00640 1 -- 172.21.15.16:0/2555861768 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87080ab9d0 con 0x7f871c0ada60 2024-03-31T15:03:44.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.925+0000 7fa715000640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa7100cf6a0 0x7fa71015abc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.16:36212/0 (socket says 172.21.15.16:36212) 2024-03-31T15:03:44.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.925+0000 7fa715a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa7100b3030 0x7fa71014f450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:44.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.925+0000 7fa715a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa7100b3030 0x7fa71014f450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39216/0 (socket says 172.21.15.16:39216) 2024-03-31T15:03:44.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.925+0000 7fa716400640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fa71015a660 0x7fa7101529b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:44.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.925+0000 7fa716400640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fa71015a660 0x7fa7101529b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3301/0 says I am v2:172.21.15.16:41760/0 (socket says 172.21.15.16:41760) 2024-03-31T15:03:44.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.925+0000 7fa716400640 1 -- 172.21.15.16:0/3806957822 learned_addr learned my addr 172.21.15.16:0/3806957822 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:03:44.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.925+0000 7f573a400640 1 -- 172.21.15.16:0/1231396231 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f57340aa9c0 con 0x7f5734147220 2024-03-31T15:03:44.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.925+0000 7f572b400640 1 -- 172.21.15.16:0/1231396231 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57240a8050 con 0x7f5734147220 2024-03-31T15:03:44.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.925+0000 7fa718400640 1 -- 172.21.15.16:0/3806957822 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fa710071bc0 con 0x7fa7100b3030 2024-03-31T15:03:44.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.925+0000 7fa718400640 1 -- 172.21.15.16:0/3806957822 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fa710071a40 con 0x7fa7100cf6a0 2024-03-31T15:03:44.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.925+0000 7fa718400640 1 -- 172.21.15.16:0/3806957822 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fa7101674e0 con 0x7fa71015a660 2024-03-31T15:03:44.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.925+0000 7f8720e00640 1 -- 172.21.15.16:0/2555861768 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f871c0ada60 msgr2=0x7f871c0af610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:44.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.925+0000 7f8720e00640 1 --2- 172.21.15.16:0/2555861768 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f871c0ada60 0x7f871c0af610 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f87080098a0 tx=0x7f870801c4c0 comp rx=0 tx=0).stop 2024-03-31T15:03:44.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.925+0000 7f8720e00640 1 -- 172.21.15.16:0/2555861768 shutdown_connections 2024-03-31T15:03:44.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.925+0000 7f8720e00640 1 --2- 172.21.15.16:0/2555861768 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f871c0ada60 0x7f871c0af610 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:44.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.925+0000 7f8720e00640 1 -- 172.21.15.16:0/2555861768 >> 172.21.15.16:0/2555861768 conn(0x7f871c0b1d30 msgr2=0x7f871c0b2140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:03:44.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.929+0000 7f8720e00640 1 -- 172.21.15.16:0/2555861768 shutdown_connections 2024-03-31T15:03:44.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.929+0000 7fa716400640 1 -- 172.21.15.16:0/3806957822 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa7100cf6a0 msgr2=0x7fa71015abc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:44.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.929+0000 7fa716400640 1 --2- 172.21.15.16:0/3806957822 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa7100cf6a0 0x7fa71015abc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:44.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.929+0000 7fa716400640 1 -- 172.21.15.16:0/3806957822 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa7100b3030 msgr2=0x7fa71014f450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:44.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.929+0000 7fa716400640 1 --2- 172.21.15.16:0/3806957822 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa7100b3030 0x7fa71014f450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:44.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.929+0000 7f573a400640 1 -- 172.21.15.16:0/1231396231 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f57340aaf80 con 0x7f5734147220 2024-03-31T15:03:44.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.929+0000 7fa716400640 1 -- 172.21.15.16:0/3806957822 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa71015b610 con 0x7fa71015a660 2024-03-31T15:03:44.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.929+0000 7fa715a00640 1 --2- 172.21.15.16:0/3806957822 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa7100b3030 0x7fa71014f450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:03:44.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.929+0000 7fa716400640 1 --2- 172.21.15.16:0/3806957822 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fa71015a660 0x7fa7101529b0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fa7080b1040 tx=0x7fa70801beb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:03:44.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.929+0000 7fb666a00640 1 -- 172.21.15.16:0/570319002 <== mon.2 v2:172.21.15.89:3301/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fb65c0bd030 con 0x7fb67015c270 2024-03-31T15:03:44.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.929+0000 7f8720e00640 1 -- 172.21.15.16:0/2555861768 wait complete. 2024-03-31T15:03:44.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.929+0000 7f5352800640 1 -- 172.21.15.16:0/244635572 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f53240052a0 con 0x7f534c167130 2024-03-31T15:03:44.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.929+0000 7f573a400640 1 -- 172.21.15.16:0/1231396231 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5734069950 con 0x7f5734147220 2024-03-31T15:03:44.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.933+0000 7f8720e00640 1 Processor -- start 2024-03-31T15:03:44.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.929+0000 7fb666a00640 1 --2- 172.21.15.16:0/570319002 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fb65006c780 0x7fb65006ec40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:44.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.929+0000 7fb666a00640 1 -- 172.21.15.16:0/570319002 <== mon.2 v2:172.21.15.89:3301/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fb65c120060 con 0x7fb67015c270 2024-03-31T15:03:44.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.933+0000 7f572b400640 1 -- 172.21.15.16:0/1231396231 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f57240dca60 con 0x7f5734147220 2024-03-31T15:03:44.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.933+0000 7f8720e00640 1 -- start start 2024-03-31T15:03:44.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.933+0000 7fb66ea00640 1 --2- 172.21.15.16:0/570319002 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fb65006c780 0x7fb65006ec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:44.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.933+0000 7f5343400640 1 -- 172.21.15.16:0/244635572 <== mon.2 v2:172.21.15.89:3301/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f53440bf020 con 0x7f534c167130 2024-03-31T15:03:44.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.933+0000 7f8720e00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f871c0cce10 0x7f871c165620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:44.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.933+0000 7f8720e00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f871c16f650 0x7f871c08a520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:44.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.933+0000 7f8720e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f871c161510 0x7f871c08aa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:44.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.933+0000 7f8720e00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f871c08ba70 con 0x7f871c161510 2024-03-31T15:03:44.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.933+0000 7f8720e00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f871c08b8f0 con 0x7f871c0cce10 2024-03-31T15:03:44.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.933+0000 7f8720e00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f871c08bbf0 con 0x7f871c16f650 2024-03-31T15:03:44.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.933+0000 7f871a800640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f871c161510 0x7f871c08aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:44.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.933+0000 7fb66ea00640 1 --2- 172.21.15.16:0/570319002 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fb65006c780 0x7fb65006ec40 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fb660004ef0 tx=0x7fb660009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:03:44.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.933+0000 7f8719400640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f871c16f650 0x7f871c08a520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:44.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.933+0000 7f572b400640 1 --2- 172.21.15.16:0/1231396231 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f57140683b0 0x7f571406a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:44.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.933+0000 7f572b400640 1 -- 172.21.15.16:0/1231396231 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f57241204e0 con 0x7f5734147220 2024-03-31T15:03:44.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.933+0000 7f8719400640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f871c16f650 0x7f871c08a520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3301/0 says I am v2:172.21.15.16:41766/0 (socket says 172.21.15.16:41766) 2024-03-31T15:03:44.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.933+0000 7f8719400640 1 -- 172.21.15.16:0/3468984769 learned_addr learned my addr 172.21.15.16:0/3468984769 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:03:44.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.933+0000 7f5343400640 1 --2- 172.21.15.16:0/244635572 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f5328068380 0x7f532806a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:44.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.933+0000 7f534b400640 1 --2- 172.21.15.16:0/244635572 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f5328068380 0x7f532806a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:44.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.933+0000 7f5343400640 1 -- 172.21.15.16:0/244635572 <== mon.2 v2:172.21.15.89:3301/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f5344121a10 con 0x7f534c167130 2024-03-31T15:03:44.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.933+0000 7f5733400640 1 --2- 172.21.15.16:0/1231396231 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f57140683b0 0x7f571406a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:44.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.933+0000 7f871a800640 1 -- 172.21.15.16:0/3468984769 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f871c16f650 msgr2=0x7f871c08a520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:44.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.933+0000 7f871a800640 1 --2- 172.21.15.16:0/3468984769 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f871c16f650 0x7f871c08a520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:44.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.933+0000 7f871a800640 1 -- 172.21.15.16:0/3468984769 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f871c0cce10 msgr2=0x7f871c165620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:44.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.933+0000 7f871a800640 1 --2- 172.21.15.16:0/3468984769 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f871c0cce10 0x7f871c165620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:44.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.933+0000 7f871a800640 1 -- 172.21.15.16:0/3468984769 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f871c169660 con 0x7f871c161510 2024-03-31T15:03:44.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.933+0000 7f534b400640 1 --2- 172.21.15.16:0/244635572 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f5328068380 0x7f532806a840 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f533c006b20 tx=0x7f533c008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:03:44.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.937+0000 7f871a800640 1 --2- 172.21.15.16:0/3468984769 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f871c161510 0x7f871c08aa60 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f871c06e610 tx=0x7f8714009ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:03:44.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.937+0000 7f5733400640 1 --2- 172.21.15.16:0/1231396231 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f57140683b0 0x7f571406a870 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f572c006fd0 tx=0x7f572c008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:03:44.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.937+0000 7fa70d600640 1 -- 172.21.15.16:0/3806957822 <== mon.2 v2:172.21.15.89:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7080b3070 con 0x7fa71015a660 2024-03-31T15:03:44.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.937+0000 7fa718400640 1 -- 172.21.15.16:0/3806957822 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa71015b8f0 con 0x7fa71015a660 2024-03-31T15:03:44.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.937+0000 7fa718400640 1 -- 172.21.15.16:0/3806957822 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fa71015beb0 con 0x7fa71015a660 2024-03-31T15:03:44.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.937+0000 7f572b400640 1 -- 172.21.15.16:0/1231396231 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f57240c6020 con 0x7f5734147220 2024-03-31T15:03:44.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.937+0000 7f8712000640 1 -- 172.21.15.16:0/3468984769 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8714019070 con 0x7f871c161510 2024-03-31T15:03:44.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.937+0000 7f8712000640 1 -- 172.21.15.16:0/3468984769 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f871400cd90 con 0x7f871c161510 2024-03-31T15:03:44.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.937+0000 7f8712000640 1 -- 172.21.15.16:0/3468984769 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8714014680 con 0x7f871c161510 2024-03-31T15:03:44.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.937+0000 7fb666a00640 1 -- 172.21.15.16:0/570319002 <== mon.2 v2:172.21.15.89:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fb65c0c4020 con 0x7fb67015c270 2024-03-31T15:03:44.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.937+0000 7f8720e00640 1 -- 172.21.15.16:0/3468984769 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f871c16ff80 con 0x7f871c161510 2024-03-31T15:03:44.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.937+0000 7f8720e00640 1 -- 172.21.15.16:0/3468984769 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f871c170540 con 0x7f871c161510 2024-03-31T15:03:44.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.941+0000 7fa718400640 1 -- 172.21.15.16:0/3806957822 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa710069950 con 0x7fa71015a660 2024-03-31T15:03:44.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.941+0000 7f8720e00640 1 -- 172.21.15.16:0/3468984769 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f86f40052a0 con 0x7f871c161510 2024-03-31T15:03:44.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.941+0000 7f8712000640 1 -- 172.21.15.16:0/3468984769 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f871401c070 con 0x7f871c161510 2024-03-31T15:03:44.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.945+0000 7f8712000640 1 --2- 172.21.15.16:0/3468984769 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f86fc0683d0 0x7f86fc06a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:44.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.945+0000 7f8712000640 1 -- 172.21.15.16:0/3468984769 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f8714086ab0 con 0x7f871c161510 2024-03-31T15:03:44.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.945+0000 7f8719e00640 1 --2- 172.21.15.16:0/3468984769 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f86fc0683d0 0x7f86fc06a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:44.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.945+0000 7f8719e00640 1 --2- 172.21.15.16:0/3468984769 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f86fc0683d0 0x7f86fc06a890 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f870c003cf0 tx=0x7f870c0079c0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:03:44.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.949+0000 7fa70d600640 1 -- 172.21.15.16:0/3806957822 <== mon.2 v2:172.21.15.89:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa7080092d0 con 0x7fa71015a660 2024-03-31T15:03:44.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.949+0000 7fa70d600640 1 -- 172.21.15.16:0/3806957822 <== mon.2 v2:172.21.15.89:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa708003dd0 con 0x7fa71015a660 2024-03-31T15:03:44.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.949+0000 7f5343400640 1 -- 172.21.15.16:0/244635572 <== mon.2 v2:172.21.15.89:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f53440c6030 con 0x7f534c167130 2024-03-31T15:03:44.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.949+0000 7fa70d600640 1 -- 172.21.15.16:0/3806957822 <== mon.2 v2:172.21.15.89:3301/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fa7080b7070 con 0x7fa71015a660 2024-03-31T15:03:44.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.953+0000 7f8712000640 1 -- 172.21.15.16:0/3468984769 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f8714023080 con 0x7f871c161510 2024-03-31T15:03:44.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.953+0000 7fa70d600640 1 --2- 172.21.15.16:0/3806957822 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fa6f0068380 0x7fa6f006a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:44.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.953+0000 7fa70d600640 1 -- 172.21.15.16:0/3806957822 <== mon.2 v2:172.21.15.89:3301/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fa708121780 con 0x7fa71015a660 2024-03-31T15:03:44.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.953+0000 7fa715a00640 1 --2- 172.21.15.16:0/3806957822 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fa6f0068380 0x7fa6f006a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:44.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.953+0000 7fa715a00640 1 --2- 172.21.15.16:0/3806957822 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fa6f0068380 0x7fa6f006a840 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fa6fc0011f0 tx=0x7fa6fc009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:03:44.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.957+0000 7fa70d600640 1 -- 172.21.15.16:0/3806957822 <== mon.2 v2:172.21.15.89:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fa7080bf060 con 0x7fa71015a660 2024-03-31T15:03:44.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.969+0000 7fb29a400640 1 Processor -- start 2024-03-31T15:03:44.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.969+0000 7fb29a400640 1 -- start start 2024-03-31T15:03:44.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.973+0000 7fb29a400640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb294151180 0x7fb2940d0860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:44.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.973+0000 7fb29a400640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fb29415a830 0x7fb29415e590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:44.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.973+0000 7fb29a400640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb29415b010 con 0x7fb29415ead0 2024-03-31T15:03:44.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.973+0000 7f66ca600640 1 Processor -- start 2024-03-31T15:03:44.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.973+0000 7fb29a400640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7fb29415de20 con 0x7fb29415a830 2024-03-31T15:03:44.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.973+0000 7fb29a400640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fb29415b2f0 con 0x7fb294151180 2024-03-31T15:03:44.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.973+0000 7fb293e00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fb29415a830 0x7fb29415e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:44.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.973+0000 7fb293e00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fb29415a830 0x7fb29415e590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.16:36222/0 (socket says 172.21.15.16:36222) 2024-03-31T15:03:44.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.973+0000 7fb293e00640 1 -- 172.21.15.16:0/3646475342 learned_addr learned my addr 172.21.15.16:0/3646475342 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:03:44.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.973+0000 7fb299000640 1 --2- 172.21.15.16:0/3646475342 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb294151180 0x7fb2940d0860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:44.968 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.977+0000 7fb293400640 1 -- 172.21.15.16:0/3646475342 <== mon.1 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2129816654 0 0) 0x7fb29415b010 con 0x7fb29415ead0 2024-03-31T15:03:44.969 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.977+0000 7fb299000640 1 -- 172.21.15.16:0/3646475342 >> v2:172.21.15.89:3300/0 conn(0x7fb29415a830 msgr2=0x7fb29415e590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:03:44.969 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.977+0000 7fb299000640 1 --2- 172.21.15.16:0/3646475342 >> v2:172.21.15.89:3300/0 conn(0x7fb29415a830 0x7fb29415e590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:44.969 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.977+0000 7fb299000640 1 -- 172.21.15.16:0/3646475342 >> v1:172.21.15.89:6789/0 conn(0x7fb29415ead0 legacy=0x7fb294072c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:44.969 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.977+0000 7fb299000640 1 -- 172.21.15.16:0/3646475342 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb29415b5d0 con 0x7fb294151180 2024-03-31T15:03:44.969 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.973+0000 7f66ca600640 1 -- start start 2024-03-31T15:03:44.969 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.977+0000 7f66ca600640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f66c415a660 0x7f66c40ce860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:44.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.977+0000 7f66ca600640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f66c40cf6a0 0x7f66c4071500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:44.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.977+0000 7f66ca600640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f66c4073a10 con 0x7f66c40b3030 2024-03-31T15:03:44.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.977+0000 7f66ca600640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f66c415aa40 con 0x7f66c40cf6a0 2024-03-31T15:03:44.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.977+0000 7f66ca600640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f66c40732a0 con 0x7f66c415a660 2024-03-31T15:03:44.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.977+0000 7f66c3e00640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f66c40b3030 0x7f66c406e910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.16:49352/0 (socket says 172.21.15.16:49352) 2024-03-31T15:03:44.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.977+0000 7f66c3400640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f66c415a660 0x7f66c40ce860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:44.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.977+0000 7f66c3400640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f66c415a660 0x7f66c40ce860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3301/0 says I am v2:172.21.15.16:41786/0 (socket says 172.21.15.16:41786) 2024-03-31T15:03:44.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.977+0000 7f66c3400640 1 -- 172.21.15.16:0/595570106 learned_addr learned my addr 172.21.15.16:0/595570106 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:03:44.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.977+0000 7f66c2a00640 1 --2- 172.21.15.16:0/595570106 >> v2:172.21.15.89:3300/0 conn(0x7f66c40cf6a0 0x7f66c4071500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:44.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.981+0000 7fb293e00640 1 --2- 172.21.15.16:0/3646475342 >> v2:172.21.15.89:3300/0 conn(0x7fb29415a830 0x7fb29415e590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:03:44.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.977+0000 7f66c3400640 1 -- 172.21.15.16:0/595570106 >> v2:172.21.15.89:3300/0 conn(0x7f66c40cf6a0 msgr2=0x7f66c4071500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:03:44.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.981+0000 7fb299000640 1 --2- 172.21.15.16:0/3646475342 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb294151180 0x7fb2940d0860 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fb2840097c0 tx=0x7fb28401c4b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1e046e4371b4ca74 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:03:44.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.977+0000 7f66c3400640 1 --2- 172.21.15.16:0/595570106 >> v2:172.21.15.89:3300/0 conn(0x7f66c40cf6a0 0x7f66c4071500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:44.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.977+0000 7f66c3400640 1 -- 172.21.15.16:0/595570106 >> v1:172.21.15.89:6789/0 conn(0x7f66c40b3030 legacy=0x7f66c406e910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:44.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.981+0000 7fb293400640 1 -- 172.21.15.16:0/3646475342 <== mon.2 v2:172.21.15.89:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2840aa5f0 con 0x7fb294151180 2024-03-31T15:03:44.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.981+0000 7fb293400640 1 -- 172.21.15.16:0/3646475342 <== mon.2 v2:172.21.15.89:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb2840af070 con 0x7fb294151180 2024-03-31T15:03:44.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.981+0000 7fb293400640 1 -- 172.21.15.16:0/3646475342 <== mon.2 v2:172.21.15.89:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2840aa9b0 con 0x7fb294151180 2024-03-31T15:03:44.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.977+0000 7f66c3400640 1 -- 172.21.15.16:0/595570106 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66c40b5110 con 0x7f66c415a660 2024-03-31T15:03:44.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.981+0000 7fb29a400640 1 -- 172.21.15.16:0/3646475342 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb294151180 msgr2=0x7fb2940d0860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:44.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.981+0000 7fb29a400640 1 --2- 172.21.15.16:0/3646475342 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb294151180 0x7fb2940d0860 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fb2840097c0 tx=0x7fb28401c4b0 comp rx=0 tx=0).stop 2024-03-31T15:03:44.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.985+0000 7fb29a400640 1 -- 172.21.15.16:0/3646475342 shutdown_connections 2024-03-31T15:03:44.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.985+0000 7fb29a400640 1 --2- 172.21.15.16:0/3646475342 >> v2:172.21.15.89:3300/0 conn(0x7fb29415a830 0x7fb29415e590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:44.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.985+0000 7fb29a400640 1 --2- 172.21.15.16:0/3646475342 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb294151180 0x7fb2940d0860 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:44.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.985+0000 7fb29a400640 1 -- 172.21.15.16:0/3646475342 >> 172.21.15.16:0/3646475342 conn(0x7fb29406b620 msgr2=0x7fb29406ba00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:03:44.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.985+0000 7f66c3400640 1 --2- 172.21.15.16:0/595570106 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f66c415a660 0x7f66c40ce860 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f66b0009870 tx=0x7f66b001c570 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c395ea7c405694f4 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:03:44.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.985+0000 7fb29a400640 1 -- 172.21.15.16:0/3646475342 shutdown_connections 2024-03-31T15:03:44.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.985+0000 7f66c2000640 1 -- 172.21.15.16:0/595570106 <== mon.2 v2:172.21.15.89:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66b00a95f0 con 0x7f66c415a660 2024-03-31T15:03:44.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.985+0000 7f66c2000640 1 -- 172.21.15.16:0/595570106 <== mon.2 v2:172.21.15.89:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f66b00a9790 con 0x7f66c415a660 2024-03-31T15:03:44.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.985+0000 7f66c2000640 1 -- 172.21.15.16:0/595570106 <== mon.2 v2:172.21.15.89:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66b00a95f0 con 0x7f66c415a660 2024-03-31T15:03:44.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.985+0000 7fb29a400640 1 -- 172.21.15.16:0/3646475342 wait complete. 2024-03-31T15:03:44.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.985+0000 7fb29a400640 1 Processor -- start 2024-03-31T15:03:44.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.985+0000 7f66ca600640 1 -- 172.21.15.16:0/595570106 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f66c415a660 msgr2=0x7f66c40ce860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:44.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.989+0000 7f66ca600640 1 --2- 172.21.15.16:0/595570106 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f66c415a660 0x7f66c40ce860 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f66b0009870 tx=0x7f66b001c570 comp rx=0 tx=0).stop 2024-03-31T15:03:44.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.989+0000 7fb29a400640 1 -- start start 2024-03-31T15:03:44.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.989+0000 7f66ca600640 1 -- 172.21.15.16:0/595570106 shutdown_connections 2024-03-31T15:03:44.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.989+0000 7f66ca600640 1 --2- 172.21.15.16:0/595570106 >> v2:172.21.15.89:3300/0 conn(0x7f66c40cf6a0 0x7f66c4071500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:44.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.989+0000 7f66ca600640 1 --2- 172.21.15.16:0/595570106 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f66c415a660 0x7f66c40ce860 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:44.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.989+0000 7f66ca600640 1 -- 172.21.15.16:0/595570106 >> 172.21.15.16:0/595570106 conn(0x7f66c40b1d30 msgr2=0x7f66c40b2140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:03:44.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.989+0000 7fb29a400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb294151180 0x7fb2940cc3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:44.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.989+0000 7f66ca600640 1 -- 172.21.15.16:0/595570106 shutdown_connections 2024-03-31T15:03:44.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.989+0000 7fb29a400640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb29415a830 0x7fb2940ce620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:44.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.989+0000 7fb29a400640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb29415ead0 0x7fb2941465c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:44.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.989+0000 7fb293e00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb29415a830 0x7fb2940ce620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:44.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.989+0000 7fb293e00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb29415a830 0x7fb2940ce620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3301/0 says I am v2:172.21.15.16:41788/0 (socket says 172.21.15.16:41788) 2024-03-31T15:03:44.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.989+0000 7fb293e00640 1 -- 172.21.15.16:0/3437885351 learned_addr learned my addr 172.21.15.16:0/3437885351 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:03:44.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.989+0000 7fb299a00640 1 --2- 172.21.15.16:0/3437885351 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb29415ead0 0x7fb2941465c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:44.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.993+0000 7fb29a400640 1 -- 172.21.15.16:0/3437885351 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fb294071ca0 con 0x7fb294151180 2024-03-31T15:03:44.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.993+0000 7fb29a400640 1 -- 172.21.15.16:0/3437885351 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fb294071b20 con 0x7fb29415ead0 2024-03-31T15:03:44.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.993+0000 7fb29a400640 1 -- 172.21.15.16:0/3437885351 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fb294073320 con 0x7fb29415a830 2024-03-31T15:03:44.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.993+0000 7fb299000640 1 --2- 172.21.15.16:0/3437885351 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb294151180 0x7fb2940cc3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:44.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.993+0000 7fb293e00640 1 -- 172.21.15.16:0/3437885351 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb29415ead0 msgr2=0x7fb2941465c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:44.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.993+0000 7fb293e00640 1 --2- 172.21.15.16:0/3437885351 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb29415ead0 0x7fb2941465c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:44.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.993+0000 7fb293e00640 1 -- 172.21.15.16:0/3437885351 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb294151180 msgr2=0x7fb2940cc3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:44.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.993+0000 7fb293e00640 1 --2- 172.21.15.16:0/3437885351 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb294151180 0x7fb2940cc3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:44.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.993+0000 7fb293e00640 1 -- 172.21.15.16:0/3437885351 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2940cc9e0 con 0x7fb29415a830 2024-03-31T15:03:44.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.989+0000 7f66ca600640 1 -- 172.21.15.16:0/595570106 wait complete. 2024-03-31T15:03:44.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.993+0000 7f66ca600640 1 Processor -- start 2024-03-31T15:03:44.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.993+0000 7f66ca600640 1 -- start start 2024-03-31T15:03:44.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.993+0000 7fb293e00640 1 --2- 172.21.15.16:0/3437885351 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb29415a830 0x7fb2940ce620 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fb28000bea0 tx=0x7fb28000bf90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:03:44.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.993+0000 7fb290c00640 1 -- 172.21.15.16:0/3437885351 <== mon.2 v2:172.21.15.89:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb280021650 con 0x7fb29415a830 2024-03-31T15:03:44.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.993+0000 7fb290c00640 1 -- 172.21.15.16:0/3437885351 <== mon.2 v2:172.21.15.89:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb2800217f0 con 0x7fb29415a830 2024-03-31T15:03:44.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.993+0000 7fb290c00640 1 -- 172.21.15.16:0/3437885351 <== mon.2 v2:172.21.15.89:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb280021ba0 con 0x7fb29415a830 2024-03-31T15:03:44.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.993+0000 7fb29a400640 1 -- 172.21.15.16:0/3437885351 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb2940ccc80 con 0x7fb29415a830 2024-03-31T15:03:44.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.993+0000 7f66ca600640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f66c40b3030 0x7f66c414f1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:44.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.997+0000 7f66ca600640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f66c40cf6a0 0x7f66c4164840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:44.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.997+0000 7fb29a400640 1 -- 172.21.15.16:0/3437885351 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fb2940ced00 con 0x7fb29415a830 2024-03-31T15:03:44.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.997+0000 7f66c2a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f66c40cf6a0 0x7f66c4164840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:44.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.997+0000 7f66c2a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f66c40cf6a0 0x7f66c4164840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39234/0 (socket says 172.21.15.16:39234) 2024-03-31T15:03:44.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.997+0000 7f66c2a00640 1 -- 172.21.15.16:0/71752145 learned_addr learned my addr 172.21.15.16:0/71752145 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:03:44.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.997+0000 7f66ca600640 1 --2- 172.21.15.16:0/71752145 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f66c415a660 0x7f66c4071f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:44.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.997+0000 7f66c3400640 1 --2- 172.21.15.16:0/71752145 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f66c40b3030 0x7f66c414f1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:44.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.997+0000 7f66ca600640 1 -- 172.21.15.16:0/71752145 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f66c40b4eb0 con 0x7f66c40cf6a0 2024-03-31T15:03:44.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.997+0000 7f66c3e00640 1 --2- 172.21.15.16:0/71752145 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f66c415a660 0x7f66c4071f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:44.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.997+0000 7fb29a400640 1 -- 172.21.15.16:0/3437885351 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb29404f8c0 con 0x7fb29415a830 2024-03-31T15:03:44.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.997+0000 7f66ca600640 1 -- 172.21.15.16:0/71752145 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f66c40b4d30 con 0x7f66c415a660 2024-03-31T15:03:44.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.997+0000 7f66ca600640 1 -- 172.21.15.16:0/71752145 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f66c4071a40 con 0x7f66c40b3030 2024-03-31T15:03:44.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.997+0000 7f66c3e00640 1 -- 172.21.15.16:0/71752145 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f66c40b3030 msgr2=0x7f66c414f1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:44.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.997+0000 7f66c3e00640 1 --2- 172.21.15.16:0/71752145 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f66c40b3030 0x7f66c414f1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:44.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.997+0000 7f66c3e00640 1 -- 172.21.15.16:0/71752145 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f66c40cf6a0 msgr2=0x7f66c4164840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:44.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.997+0000 7f66c3e00640 1 --2- 172.21.15.16:0/71752145 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f66c40cf6a0 0x7f66c4164840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:44.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.997+0000 7f66c3e00640 1 -- 172.21.15.16:0/71752145 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66c414e140 con 0x7f66c415a660 2024-03-31T15:03:44.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:44.997+0000 7f66c3e00640 1 --2- 172.21.15.16:0/71752145 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f66c415a660 0x7f66c4071f50 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f66b4001e30 tx=0x7f66b400c820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:03:44.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.001+0000 7fb290c00640 1 -- 172.21.15.16:0/3437885351 <== mon.2 v2:172.21.15.89:3301/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fb2800203e0 con 0x7fb29415a830 2024-03-31T15:03:44.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.001+0000 7fb290c00640 1 --2- 172.21.15.16:0/3437885351 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fb274068380 0x7fb27406a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:44.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.001+0000 7fb290c00640 1 -- 172.21.15.16:0/3437885351 <== mon.2 v2:172.21.15.89:3301/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fb280097e70 con 0x7fb29415a830 2024-03-31T15:03:44.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.001+0000 7f66bb400640 1 -- 172.21.15.16:0/71752145 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66b40075a0 con 0x7f66c415a660 2024-03-31T15:03:44.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.001+0000 7f66bb400640 1 -- 172.21.15.16:0/71752145 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f66b4015070 con 0x7f66c415a660 2024-03-31T15:03:44.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.001+0000 7f66ca600640 1 -- 172.21.15.16:0/71752145 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f66c414e420 con 0x7f66c415a660 2024-03-31T15:03:44.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.001+0000 7fb299000640 1 --2- 172.21.15.16:0/3437885351 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fb274068380 0x7fb27406a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:44.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.001+0000 7f66ca600640 1 -- 172.21.15.16:0/71752145 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f66c414e9e0 con 0x7f66c415a660 2024-03-31T15:03:44.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.005+0000 7fb299000640 1 --2- 172.21.15.16:0/3437885351 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fb274068380 0x7fb27406a840 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fb2840ae000 tx=0x7fb2840a7000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:03:44.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.005+0000 7f66bb400640 1 -- 172.21.15.16:0/71752145 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66b4004a30 con 0x7f66c415a660 2024-03-31T15:03:44.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.005+0000 7fb290c00640 1 -- 172.21.15.16:0/3437885351 <== mon.2 v2:172.21.15.89:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fb280025030 con 0x7fb29415a830 2024-03-31T15:03:45.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.005+0000 7f66bb400640 1 -- 172.21.15.16:0/71752145 <== mon.1 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f66b4004bd0 con 0x7f66c415a660 2024-03-31T15:03:45.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.009+0000 7f66ca600640 1 -- 172.21.15.16:0/71752145 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f669c0052a0 con 0x7f66c415a660 2024-03-31T15:03:45.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.009+0000 7f66bb400640 1 --2- 172.21.15.16:0/71752145 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f66a4068450 0x7f66a406a910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:45.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.009+0000 7f66bb400640 1 -- 172.21.15.16:0/71752145 <== mon.1 v2:172.21.15.89:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f66b408a360 con 0x7f66c415a660 2024-03-31T15:03:45.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.009+0000 7f66c3400640 1 --2- 172.21.15.16:0/71752145 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f66a4068450 0x7f66a406a910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:45.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.009+0000 7f66c3400640 1 --2- 172.21.15.16:0/71752145 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f66a4068450 0x7f66a406a910 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f66b00ae000 tx=0x7f66b00b3040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:03:45.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.013+0000 7f66bb400640 1 -- 172.21.15.16:0/71752145 <== mon.1 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f66b405a2b0 con 0x7f66c415a660 2024-03-31T15:03:45.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.241+0000 7f970f200640 1 -- 172.21.15.16:0/3278691659 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7f97080b2df0 con 0x7f9708072bf0 2024-03-31T15:03:45.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.241+0000 7f96ffe00640 1 -- 172.21.15.16:0/3278691659 <== mon.1 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f96f4067b60 con 0x7f9708072bf0 2024-03-31T15:03:45.237 INFO:teuthology.orchestra.run.smithi016.stdout:60129542152 2024-03-31T15:03:45.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.249+0000 7f970f200640 1 -- 172.21.15.16:0/3278691659 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f96e8068380 msgr2=0x7f96e806a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:45.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.249+0000 7f970f200640 1 --2- 172.21.15.16:0/3278691659 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f96e8068380 0x7f96e806a840 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f96f8007f00 tx=0x7f96f8008040 comp rx=0 tx=0).stop 2024-03-31T15:03:45.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.249+0000 7f970f200640 1 -- 172.21.15.16:0/3278691659 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9708072bf0 msgr2=0x7f97081534a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:45.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.249+0000 7f970f200640 1 --2- 172.21.15.16:0/3278691659 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9708072bf0 0x7f97081534a0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f96f400b440 tx=0x7f96f4009f90 comp rx=0 tx=0).stop 2024-03-31T15:03:45.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.249+0000 7f970f200640 1 -- 172.21.15.16:0/3278691659 shutdown_connections 2024-03-31T15:03:45.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.249+0000 7f970f200640 1 --2- 172.21.15.16:0/3278691659 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f96e8068380 0x7f96e806a840 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:45.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.249+0000 7f970f200640 1 --2- 172.21.15.16:0/3278691659 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f970815a720 0x7f9708146680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:45.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.249+0000 7f970f200640 1 --2- 172.21.15.16:0/3278691659 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9708072bf0 0x7f97081534a0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:45.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.249+0000 7f970f200640 1 --2- 172.21.15.16:0/3278691659 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f970806c140 0x7f97080cbf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:45.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.249+0000 7f970f200640 1 -- 172.21.15.16:0/3278691659 >> 172.21.15.16:0/3278691659 conn(0x7f970806b620 msgr2=0x7f97080ad730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:03:45.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.249+0000 7f970f200640 1 -- 172.21.15.16:0/3278691659 shutdown_connections 2024-03-31T15:03:45.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.249+0000 7f970f200640 1 -- 172.21.15.16:0/3278691659 wait complete. 2024-03-31T15:03:45.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.277+0000 7fb675a00640 1 -- 172.21.15.16:0/570319002 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7fb670068ae0 con 0x7fb67015c270 2024-03-31T15:03:45.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.277+0000 7fb666a00640 1 -- 172.21.15.16:0/570319002 <== mon.2 v2:172.21.15.89:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fb65c0160f0 con 0x7fb67015c270 2024-03-31T15:03:45.272 INFO:teuthology.orchestra.run.smithi016.stdout:64424509448 2024-03-31T15:03:45.273 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542152 got 60129542152 for osd.0 2024-03-31T15:03:45.273 DEBUG:teuthology.parallel:result is None 2024-03-31T15:03:45.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.285+0000 7fb675a00640 1 -- 172.21.15.16:0/570319002 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fb65006c780 msgr2=0x7fb65006ec40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:45.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.285+0000 7fb675a00640 1 --2- 172.21.15.16:0/570319002 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fb65006c780 0x7fb65006ec40 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fb660004ef0 tx=0x7fb660009040 comp rx=0 tx=0).stop 2024-03-31T15:03:45.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.285+0000 7fb675a00640 1 -- 172.21.15.16:0/570319002 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb67015c270 msgr2=0x7fb67015ee50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:45.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.285+0000 7fb675a00640 1 --2- 172.21.15.16:0/570319002 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb67015c270 0x7fb67015ee50 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fb65c0ac000 tx=0x7fb65c0192a0 comp rx=0 tx=0).stop 2024-03-31T15:03:45.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.285+0000 7fb675a00640 1 -- 172.21.15.16:0/570319002 shutdown_connections 2024-03-31T15:03:45.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.285+0000 7fb675a00640 1 --2- 172.21.15.16:0/570319002 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fb65006c780 0x7fb65006ec40 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:45.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.285+0000 7fb675a00640 1 --2- 172.21.15.16:0/570319002 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb67015fde0 0x7fb670153790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:45.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.285+0000 7fb675a00640 1 --2- 172.21.15.16:0/570319002 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb67015c270 0x7fb67015ee50 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:45.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.285+0000 7fb675a00640 1 --2- 172.21.15.16:0/570319002 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb6700b4660 0x7fb67015d1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:45.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.285+0000 7fb675a00640 1 -- 172.21.15.16:0/570319002 >> 172.21.15.16:0/570319002 conn(0x7fb6700b3a60 msgr2=0x7fb67009aa60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:03:45.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.285+0000 7fb675a00640 1 -- 172.21.15.16:0/570319002 shutdown_connections 2024-03-31T15:03:45.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.285+0000 7fb675a00640 1 -- 172.21.15.16:0/570319002 wait complete. 2024-03-31T15:03:45.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.293+0000 7f573a400640 1 -- 172.21.15.16:0/1231396231 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7f57340729b0 con 0x7f5734147220 2024-03-31T15:03:45.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.297+0000 7f572b400640 1 -- 172.21.15.16:0/1231396231 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f5724016110 con 0x7f5734147220 2024-03-31T15:03:45.294 INFO:teuthology.orchestra.run.smithi016.stdout:64424509446 2024-03-31T15:03:45.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.301+0000 7f5352800640 1 -- 172.21.15.16:0/244635572 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7f5324005590 con 0x7f534c167130 2024-03-31T15:03:45.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.301+0000 7f8720e00640 1 -- 172.21.15.16:0/3468984769 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7f86f4005860 con 0x7f871c161510 2024-03-31T15:03:45.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.301+0000 7f573a400640 1 -- 172.21.15.16:0/1231396231 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f57140683b0 msgr2=0x7f571406a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:45.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.301+0000 7f573a400640 1 --2- 172.21.15.16:0/1231396231 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f57140683b0 0x7f571406a870 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f572c006fd0 tx=0x7f572c008040 comp rx=0 tx=0).stop 2024-03-31T15:03:45.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.301+0000 7f573a400640 1 -- 172.21.15.16:0/1231396231 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5734147220 msgr2=0x7f57340ac550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:45.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.301+0000 7f573a400640 1 --2- 172.21.15.16:0/1231396231 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5734147220 0x7f57340ac550 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f57240ac000 tx=0x7f5724019290 comp rx=0 tx=0).stop 2024-03-31T15:03:45.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.301+0000 7f5343400640 1 -- 172.21.15.16:0/244635572 <== mon.2 v2:172.21.15.89:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f53440165e0 con 0x7f534c167130 2024-03-31T15:03:45.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.301+0000 7f573a400640 1 -- 172.21.15.16:0/1231396231 shutdown_connections 2024-03-31T15:03:45.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.301+0000 7f573a400640 1 --2- 172.21.15.16:0/1231396231 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f57140683b0 0x7f571406a870 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:45.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.301+0000 7f573a400640 1 --2- 172.21.15.16:0/1231396231 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5734147220 0x7f57340ac550 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:45.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.301+0000 7f573a400640 1 --2- 172.21.15.16:0/1231396231 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f57340b2cd0 0x7f57340ac010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:45.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.301+0000 7f573a400640 1 --2- 172.21.15.16:0/1231396231 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f5734070b50 0x7f5734145630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:45.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.301+0000 7f573a400640 1 -- 172.21.15.16:0/1231396231 >> 172.21.15.16:0/1231396231 conn(0x7f573406d090 msgr2=0x7f573406e7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:03:45.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.301+0000 7f573a400640 1 -- 172.21.15.16:0/1231396231 shutdown_connections 2024-03-31T15:03:45.297 INFO:teuthology.orchestra.run.smithi016.stdout:64424509447 2024-03-31T15:03:45.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.301+0000 7f573a400640 1 -- 172.21.15.16:0/1231396231 wait complete. 2024-03-31T15:03:45.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.305+0000 7f8712000640 1 -- 172.21.15.16:0/3468984769 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f8714056a00 con 0x7f871c161510 2024-03-31T15:03:45.298 INFO:teuthology.orchestra.run.smithi016.stdout:60129542152 2024-03-31T15:03:45.299 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509447 got 64424509448 for osd.5 2024-03-31T15:03:45.299 DEBUG:teuthology.parallel:result is None 2024-03-31T15:03:45.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.305+0000 7f5352800640 1 -- 172.21.15.16:0/244635572 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f5328068380 msgr2=0x7f532806a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:45.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.305+0000 7f5352800640 1 --2- 172.21.15.16:0/244635572 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f5328068380 0x7f532806a840 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f533c006b20 tx=0x7f533c008040 comp rx=0 tx=0).stop 2024-03-31T15:03:45.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.305+0000 7f5352800640 1 -- 172.21.15.16:0/244635572 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f534c167130 msgr2=0x7f534c144740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:45.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.305+0000 7f5352800640 1 --2- 172.21.15.16:0/244635572 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f534c167130 0x7f534c144740 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f53440af040 tx=0x7f5344019790 comp rx=0 tx=0).stop 2024-03-31T15:03:45.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.305+0000 7f5352800640 1 -- 172.21.15.16:0/244635572 shutdown_connections 2024-03-31T15:03:45.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.305+0000 7f5352800640 1 --2- 172.21.15.16:0/244635572 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f5328068380 0x7f532806a840 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:45.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.305+0000 7f5352800640 1 --2- 172.21.15.16:0/244635572 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f534c167130 0x7f534c144740 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:45.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.305+0000 7f5352800640 1 --2- 172.21.15.16:0/244635572 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f534c147b50 0x7f534c144200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:45.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.305+0000 7f5352800640 1 --2- 172.21.15.16:0/244635572 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f534c06fe60 0x7f534c1430e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:45.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.305+0000 7f5352800640 1 -- 172.21.15.16:0/244635572 >> 172.21.15.16:0/244635572 conn(0x7f534c067330 msgr2=0x7f534c06d7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:03:45.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.305+0000 7f5352800640 1 -- 172.21.15.16:0/244635572 shutdown_connections 2024-03-31T15:03:45.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.305+0000 7f5352800640 1 -- 172.21.15.16:0/244635572 wait complete. 2024-03-31T15:03:45.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.305+0000 7f8720e00640 1 -- 172.21.15.16:0/3468984769 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f86fc0683d0 msgr2=0x7f86fc06a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:45.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.305+0000 7f8720e00640 1 --2- 172.21.15.16:0/3468984769 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f86fc0683d0 0x7f86fc06a890 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f870c003cf0 tx=0x7f870c0079c0 comp rx=0 tx=0).stop 2024-03-31T15:03:45.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.305+0000 7f8720e00640 1 -- 172.21.15.16:0/3468984769 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f871c161510 msgr2=0x7f871c08aa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:45.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.305+0000 7f8720e00640 1 --2- 172.21.15.16:0/3468984769 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f871c161510 0x7f871c08aa60 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f871c06e610 tx=0x7f8714009ed0 comp rx=0 tx=0).stop 2024-03-31T15:03:45.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.305+0000 7f8720e00640 1 -- 172.21.15.16:0/3468984769 shutdown_connections 2024-03-31T15:03:45.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.305+0000 7f8720e00640 1 --2- 172.21.15.16:0/3468984769 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f86fc0683d0 0x7f86fc06a890 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:45.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.305+0000 7f8720e00640 1 --2- 172.21.15.16:0/3468984769 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f871c161510 0x7f871c08aa60 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:45.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.305+0000 7f8720e00640 1 --2- 172.21.15.16:0/3468984769 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f871c16f650 0x7f871c08a520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:45.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.305+0000 7f8720e00640 1 --2- 172.21.15.16:0/3468984769 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f871c0cce10 0x7f871c165620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:45.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.305+0000 7f8720e00640 1 -- 172.21.15.16:0/3468984769 >> 172.21.15.16:0/3468984769 conn(0x7f871c0b1d30 msgr2=0x7f871c0ce710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:03:45.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.305+0000 7f8720e00640 1 -- 172.21.15.16:0/3468984769 shutdown_connections 2024-03-31T15:03:45.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.305+0000 7f8720e00640 1 -- 172.21.15.16:0/3468984769 wait complete. 2024-03-31T15:03:45.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.309+0000 7fa718400640 1 -- 172.21.15.16:0/3806957822 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7fa710068980 con 0x7fa71015a660 2024-03-31T15:03:45.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.309+0000 7fa70d600640 1 -- 172.21.15.16:0/3806957822 <== mon.2 v2:172.21.15.89:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fa708018d30 con 0x7fa71015a660 2024-03-31T15:03:45.304 INFO:teuthology.orchestra.run.smithi016.stdout:60129542152 2024-03-31T15:03:45.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.313+0000 7fa718400640 1 -- 172.21.15.16:0/3806957822 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fa6f0068380 msgr2=0x7fa6f006a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:45.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.313+0000 7fa718400640 1 --2- 172.21.15.16:0/3806957822 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fa6f0068380 0x7fa6f006a840 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fa6fc0011f0 tx=0x7fa6fc009040 comp rx=0 tx=0).stop 2024-03-31T15:03:45.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.313+0000 7fa718400640 1 -- 172.21.15.16:0/3806957822 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fa71015a660 msgr2=0x7fa7101529b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:45.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.313+0000 7fa718400640 1 --2- 172.21.15.16:0/3806957822 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fa71015a660 0x7fa7101529b0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fa7080b1040 tx=0x7fa70801beb0 comp rx=0 tx=0).stop 2024-03-31T15:03:45.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.313+0000 7fa718400640 1 -- 172.21.15.16:0/3806957822 shutdown_connections 2024-03-31T15:03:45.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.313+0000 7fa718400640 1 --2- 172.21.15.16:0/3806957822 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fa6f0068380 0x7fa6f006a840 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:45.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.313+0000 7fa718400640 1 --2- 172.21.15.16:0/3806957822 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fa71015a660 0x7fa7101529b0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:45.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.313+0000 7fa718400640 1 --2- 172.21.15.16:0/3806957822 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa7100cf6a0 0x7fa71015abc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:45.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.313+0000 7fa718400640 1 --2- 172.21.15.16:0/3806957822 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa7100b3030 0x7fa71014f450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:45.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.313+0000 7fa718400640 1 -- 172.21.15.16:0/3806957822 >> 172.21.15.16:0/3806957822 conn(0x7fa7100b1d30 msgr2=0x7fa71015df90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:03:45.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.313+0000 7fa718400640 1 -- 172.21.15.16:0/3806957822 shutdown_connections 2024-03-31T15:03:45.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.313+0000 7fa718400640 1 -- 172.21.15.16:0/3806957822 wait complete. 2024-03-31T15:03:45.326 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509446 got 64424509447 for osd.4 2024-03-31T15:03:45.327 DEBUG:teuthology.parallel:result is None 2024-03-31T15:03:45.327 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542152 got 60129542152 for osd.1 2024-03-31T15:03:45.327 DEBUG:teuthology.parallel:result is None 2024-03-31T15:03:45.336 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509445 got 64424509446 for osd.2 2024-03-31T15:03:45.336 DEBUG:teuthology.parallel:result is None 2024-03-31T15:03:45.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.349+0000 7fb29a400640 1 -- 172.21.15.16:0/3437885351 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7fb29406bce0 con 0x7fb29415a830 2024-03-31T15:03:45.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.349+0000 7fb290c00640 1 -- 172.21.15.16:0/3437885351 <== mon.2 v2:172.21.15.89:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fb280067dd0 con 0x7fb29415a830 2024-03-31T15:03:45.341 INFO:teuthology.orchestra.run.smithi016.stdout:60129542152 2024-03-31T15:03:45.342 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542152 got 60129542152 for osd.3 2024-03-31T15:03:45.342 DEBUG:teuthology.parallel:result is None 2024-03-31T15:03:45.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.349+0000 7fb29a400640 1 -- 172.21.15.16:0/3437885351 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fb274068380 msgr2=0x7fb27406a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:45.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.349+0000 7fb29a400640 1 --2- 172.21.15.16:0/3437885351 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fb274068380 0x7fb27406a840 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fb2840ae000 tx=0x7fb2840a7000 comp rx=0 tx=0).stop 2024-03-31T15:03:45.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.349+0000 7fb29a400640 1 -- 172.21.15.16:0/3437885351 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb29415a830 msgr2=0x7fb2940ce620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:45.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.349+0000 7fb29a400640 1 --2- 172.21.15.16:0/3437885351 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb29415a830 0x7fb2940ce620 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fb28000bea0 tx=0x7fb28000bf90 comp rx=0 tx=0).stop 2024-03-31T15:03:45.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.349+0000 7fb29a400640 1 -- 172.21.15.16:0/3437885351 shutdown_connections 2024-03-31T15:03:45.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.349+0000 7fb29a400640 1 --2- 172.21.15.16:0/3437885351 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fb274068380 0x7fb27406a840 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:45.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.349+0000 7fb29a400640 1 --2- 172.21.15.16:0/3437885351 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb29415ead0 0x7fb2941465c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:45.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.349+0000 7fb29a400640 1 --2- 172.21.15.16:0/3437885351 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb29415a830 0x7fb2940ce620 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:45.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.349+0000 7fb29a400640 1 --2- 172.21.15.16:0/3437885351 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb294151180 0x7fb2940cc3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:45.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.349+0000 7fb29a400640 1 -- 172.21.15.16:0/3437885351 >> 172.21.15.16:0/3437885351 conn(0x7fb29406b620 msgr2=0x7fb2941479b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:03:45.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.349+0000 7fb29a400640 1 -- 172.21.15.16:0/3437885351 shutdown_connections 2024-03-31T15:03:45.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.349+0000 7f66ca600640 1 -- 172.21.15.16:0/71752145 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7f669c005590 con 0x7f66c415a660 2024-03-31T15:03:45.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.349+0000 7fb29a400640 1 -- 172.21.15.16:0/3437885351 wait complete. 2024-03-31T15:03:45.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.349+0000 7f66bb400640 1 -- 172.21.15.16:0/71752145 <== mon.1 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f66b405e5b0 con 0x7f66c415a660 2024-03-31T15:03:45.345 INFO:teuthology.orchestra.run.smithi016.stdout:64424509449 2024-03-31T15:03:45.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.353+0000 7f66ca600640 1 -- 172.21.15.16:0/71752145 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f66a4068450 msgr2=0x7f66a406a910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:45.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.353+0000 7f66ca600640 1 --2- 172.21.15.16:0/71752145 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f66a4068450 0x7f66a406a910 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f66b00ae000 tx=0x7f66b00b3040 comp rx=0 tx=0).stop 2024-03-31T15:03:45.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.353+0000 7f66ca600640 1 -- 172.21.15.16:0/71752145 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f66c415a660 msgr2=0x7f66c4071f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:45.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.353+0000 7f66ca600640 1 --2- 172.21.15.16:0/71752145 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f66c415a660 0x7f66c4071f50 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f66b4001e30 tx=0x7f66b400c820 comp rx=0 tx=0).stop 2024-03-31T15:03:45.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.353+0000 7f66ca600640 1 -- 172.21.15.16:0/71752145 shutdown_connections 2024-03-31T15:03:45.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.353+0000 7f66ca600640 1 --2- 172.21.15.16:0/71752145 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f66a4068450 0x7f66a406a910 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:45.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.353+0000 7f66ca600640 1 --2- 172.21.15.16:0/71752145 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f66c415a660 0x7f66c4071f50 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:45.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.353+0000 7f66ca600640 1 --2- 172.21.15.16:0/71752145 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f66c40cf6a0 0x7f66c4164840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:45.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.353+0000 7f66ca600640 1 --2- 172.21.15.16:0/71752145 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f66c40b3030 0x7f66c414f1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:45.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.353+0000 7f66ca600640 1 -- 172.21.15.16:0/71752145 >> 172.21.15.16:0/71752145 conn(0x7f66c40b1d30 msgr2=0x7f66c415abc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:03:45.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.353+0000 7f66ca600640 1 -- 172.21.15.16:0/71752145 shutdown_connections 2024-03-31T15:03:45.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.353+0000 7f66ca600640 1 -- 172.21.15.16:0/71752145 wait complete. 2024-03-31T15:03:45.365 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542152 got 60129542152 for osd.6 2024-03-31T15:03:45.365 DEBUG:teuthology.parallel:result is None 2024-03-31T15:03:45.366 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509448 got 64424509449 for osd.7 2024-03-31T15:03:45.367 DEBUG:teuthology.parallel:result is None 2024-03-31T15:03:45.367 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-03-31T15:03:45.367 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-31T15:03:45.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.557+0000 7f705ec00640 1 Processor -- start 2024-03-31T15:03:45.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.557+0000 7f705ec00640 1 -- start start 2024-03-31T15:03:45.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.561+0000 7f705ec00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f70580728b0 0x7f70580b1cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:45.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.561+0000 7f705ec00640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7058160e70 con 0x7f70580b2200 2024-03-31T15:03:45.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.561+0000 7f705ec00640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f70581602f0 con 0x7f7058145d10 2024-03-31T15:03:45.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.561+0000 7f705ec00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f70581605d0 con 0x7f70580728b0 2024-03-31T15:03:45.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.561+0000 7f7057e00640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f7058145d10 0x7f70580cf790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.16:49366/0 (socket says 172.21.15.16:49366) 2024-03-31T15:03:45.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.561+0000 7f7057e00640 1 -- 172.21.15.16:0/2905357904 learned_addr learned my addr 172.21.15.16:0/2905357904 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:03:45.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.561+0000 7f705cc00640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f70580b2200 0x7f705815cd50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:34088/0 (socket says 172.21.15.16:34088) 2024-03-31T15:03:45.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.561+0000 7f704fe00640 1 --2- 172.21.15.16:0/2905357904 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f70580728b0 0x7f70580b1cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:45.554 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.561+0000 7f7057400640 1 -- 172.21.15.16:0/2905357904 <== mon.1 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 686285409 0 0) 0x7f70581602f0 con 0x7f7058145d10 2024-03-31T15:03:45.554 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.561+0000 7f7057400640 1 -- 172.21.15.16:0/2905357904 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f703c003530 con 0x7f7058145d10 2024-03-31T15:03:45.554 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.561+0000 7f7057400640 1 -- 172.21.15.16:0/2905357904 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2696778979 0 0) 0x7f7058160e70 con 0x7f70580b2200 2024-03-31T15:03:45.554 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.561+0000 7f7057400640 1 -- 172.21.15.16:0/2905357904 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f70581602f0 con 0x7f70580b2200 2024-03-31T15:03:45.554 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.561+0000 7f704fe00640 1 -- 172.21.15.16:0/2905357904 >> v1:172.21.15.89:6789/0 conn(0x7f7058145d10 legacy=0x7f70580cf790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:45.554 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.561+0000 7f704fe00640 1 -- 172.21.15.16:0/2905357904 >> v1:172.21.15.16:6789/0 conn(0x7f70580b2200 legacy=0x7f705815cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:45.554 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.561+0000 7f704fe00640 1 -- 172.21.15.16:0/2905357904 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70581608b0 con 0x7f70580728b0 2024-03-31T15:03:45.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.561+0000 7f704fe00640 1 --2- 172.21.15.16:0/2905357904 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f70580728b0 0x7f70580b1cc0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f70400097f0 tx=0x7f704001c4e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c684c17951643418 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:03:45.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.565+0000 7f7057400640 1 -- 172.21.15.16:0/2905357904 <== mon.2 v2:172.21.15.89:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70400b0070 con 0x7f70580728b0 2024-03-31T15:03:45.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.565+0000 7f7057400640 1 -- 172.21.15.16:0/2905357904 <== mon.2 v2:172.21.15.89:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f70400ab600 con 0x7f70580728b0 2024-03-31T15:03:45.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.565+0000 7f7057400640 1 -- 172.21.15.16:0/2905357904 <== mon.2 v2:172.21.15.89:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70400ab9b0 con 0x7f70580728b0 2024-03-31T15:03:45.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.565+0000 7f705ec00640 1 -- 172.21.15.16:0/2905357904 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f70580728b0 msgr2=0x7f70580b1cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:45.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.565+0000 7f705ec00640 1 --2- 172.21.15.16:0/2905357904 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f70580728b0 0x7f70580b1cc0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f70400097f0 tx=0x7f704001c4e0 comp rx=0 tx=0).stop 2024-03-31T15:03:45.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.565+0000 7f705ec00640 1 -- 172.21.15.16:0/2905357904 shutdown_connections 2024-03-31T15:03:45.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.565+0000 7f705ec00640 1 --2- 172.21.15.16:0/2905357904 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f70580728b0 0x7f70580b1cc0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:45.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.565+0000 7f705ec00640 1 -- 172.21.15.16:0/2905357904 >> 172.21.15.16:0/2905357904 conn(0x7f705806b620 msgr2=0x7f705806ba30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:03:45.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.565+0000 7f705ec00640 1 -- 172.21.15.16:0/2905357904 shutdown_connections 2024-03-31T15:03:45.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.565+0000 7f705ec00640 1 -- 172.21.15.16:0/2905357904 wait complete. 2024-03-31T15:03:45.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.565+0000 7f705ec00640 1 Processor -- start 2024-03-31T15:03:45.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.565+0000 7f705ec00640 1 -- start start 2024-03-31T15:03:45.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.565+0000 7f705ec00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f70580728b0 0x7f705813d700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:45.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.565+0000 7f705ec00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f70580b2200 0x7f705813efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:45.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.565+0000 7f705ec00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f7058145d10 0x7f70580b0360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:45.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.565+0000 7f705ec00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f70580cf350 con 0x7f70580728b0 2024-03-31T15:03:45.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.565+0000 7f705ec00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f70580b2610 con 0x7f70580b2200 2024-03-31T15:03:45.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.565+0000 7f705ec00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f70580cf4d0 con 0x7f7058145d10 2024-03-31T15:03:45.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.565+0000 7f7057e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f70580728b0 0x7f705813d700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:45.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.565+0000 7f7057e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f70580728b0 0x7f705813d700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39242/0 (socket says 172.21.15.16:39242) 2024-03-31T15:03:45.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.565+0000 7f7057e00640 1 -- 172.21.15.16:0/116224532 learned_addr learned my addr 172.21.15.16:0/116224532 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:03:45.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.565+0000 7f704fe00640 1 --2- 172.21.15.16:0/116224532 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f70580b2200 0x7f705813efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:45.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.565+0000 7f705cc00640 1 --2- 172.21.15.16:0/116224532 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f7058145d10 0x7f70580b0360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:45.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.565+0000 7f7057e00640 1 -- 172.21.15.16:0/116224532 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f7058145d10 msgr2=0x7f70580b0360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:45.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.565+0000 7f7057e00640 1 --2- 172.21.15.16:0/116224532 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f7058145d10 0x7f70580b0360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:45.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.565+0000 7f7057e00640 1 -- 172.21.15.16:0/116224532 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f70580b2200 msgr2=0x7f705813efb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:45.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.565+0000 7f7057e00640 1 --2- 172.21.15.16:0/116224532 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f70580b2200 0x7f705813efb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:45.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.565+0000 7f7057e00640 1 -- 172.21.15.16:0/116224532 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f705813dc40 con 0x7f70580728b0 2024-03-31T15:03:45.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.569+0000 7f7057e00640 1 --2- 172.21.15.16:0/116224532 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f70580728b0 0x7f705813d700 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f704800c450 tx=0x7f704800c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:03:45.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.569+0000 7f7054c00640 1 -- 172.21.15.16:0/116224532 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7048017070 con 0x7f70580728b0 2024-03-31T15:03:45.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.569+0000 7f7054c00640 1 -- 172.21.15.16:0/116224532 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7048005cf0 con 0x7f70580728b0 2024-03-31T15:03:45.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.569+0000 7f7054c00640 1 -- 172.21.15.16:0/116224532 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7048007270 con 0x7f70580728b0 2024-03-31T15:03:45.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.569+0000 7f705ec00640 1 -- 172.21.15.16:0/116224532 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f705813c940 con 0x7f70580728b0 2024-03-31T15:03:45.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.569+0000 7f705ec00640 1 -- 172.21.15.16:0/116224532 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f705813cf00 con 0x7f70580728b0 2024-03-31T15:03:45.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.569+0000 7f705ec00640 1 -- 172.21.15.16:0/116224532 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7058069950 con 0x7f70580728b0 2024-03-31T15:03:45.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.569+0000 7f7054c00640 1 -- 172.21.15.16:0/116224532 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f704801a070 con 0x7f70580728b0 2024-03-31T15:03:45.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.569+0000 7f7054c00640 1 --2- 172.21.15.16:0/116224532 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f7034068380 0x7f703406a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:45.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.569+0000 7f7054c00640 1 -- 172.21.15.16:0/116224532 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f7048085b10 con 0x7f70580728b0 2024-03-31T15:03:45.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.569+0000 7f704fe00640 1 --2- 172.21.15.16:0/116224532 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f7034068380 0x7f703406a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:45.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.573+0000 7f704fe00640 1 --2- 172.21.15.16:0/116224532 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f7034068380 0x7f703406a840 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f70400ad000 tx=0x7f7040008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:03:45.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.573+0000 7f7054c00640 1 -- 172.21.15.16:0/116224532 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f7048055b10 con 0x7f70580728b0 2024-03-31T15:03:45.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.733+0000 7f705ec00640 1 -- 172.21.15.16:0/116224532 --> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f705815bfa0 con 0x7f7034068380 2024-03-31T15:03:45.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.741+0000 7f7054c00640 1 -- 172.21.15.16:0/116224532 <== mgr.4100 v2:172.21.15.89:6832/1549407273 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251836 (secure 0 0 0) 0x7f705815bfa0 con 0x7f7034068380 2024-03-31T15:03:45.737 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:03:45.738 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-03-31T15:03:45.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.745+0000 7f705ec00640 1 -- 172.21.15.16:0/116224532 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f7034068380 msgr2=0x7f703406a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:45.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.745+0000 7f705ec00640 1 --2- 172.21.15.16:0/116224532 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f7034068380 0x7f703406a840 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f70400ad000 tx=0x7f7040008040 comp rx=0 tx=0).stop 2024-03-31T15:03:45.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.745+0000 7f705ec00640 1 -- 172.21.15.16:0/116224532 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f70580728b0 msgr2=0x7f705813d700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:45.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.745+0000 7f705ec00640 1 --2- 172.21.15.16:0/116224532 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f70580728b0 0x7f705813d700 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f704800c450 tx=0x7f704800c910 comp rx=0 tx=0).stop 2024-03-31T15:03:45.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.745+0000 7f705ec00640 1 -- 172.21.15.16:0/116224532 shutdown_connections 2024-03-31T15:03:45.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.745+0000 7f705ec00640 1 --2- 172.21.15.16:0/116224532 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f7034068380 0x7f703406a840 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:45.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.745+0000 7f705ec00640 1 --2- 172.21.15.16:0/116224532 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f7058145d10 0x7f70580b0360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:45.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.745+0000 7f705ec00640 1 --2- 172.21.15.16:0/116224532 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f70580b2200 0x7f705813efb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:45.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.745+0000 7f705ec00640 1 --2- 172.21.15.16:0/116224532 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f70580728b0 0x7f705813d700 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:45.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.745+0000 7f705ec00640 1 -- 172.21.15.16:0/116224532 >> 172.21.15.16:0/116224532 conn(0x7f705806b620 msgr2=0x7f7058072d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:03:45.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.745+0000 7f705ec00640 1 -- 172.21.15.16:0/116224532 shutdown_connections 2024-03-31T15:03:45.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.745+0000 7f705ec00640 1 -- 172.21.15.16:0/116224532 wait complete. 2024-03-31T15:03:45.757 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":25,"stamp":"2024-03-31T15:03:44.531310+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":235692,"kb_used_data":2856,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754739028,"statfs":{"total":773094113280,"available":772852764672,"internally_reserved":0,"allocated":2924544,"data_stored":1768716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":238276144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001612"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-31T15:03:38.898721+0000","last_change":"2024-03-31T15:03:36.984154+0000","last_active":"2024-03-31T15:03:38.898721+0000","last_peered":"2024-03-31T15:03:38.898721+0000","last_clean":"2024-03-31T15:03:38.898721+0000","last_became_active":"2024-03-31T15:03:36.983712+0000","last_became_peered":"2024-03-31T15:03:36.983712+0000","last_unstale":"2024-03-31T15:03:38.898721+0000","last_undegraded":"2024-03-31T15:03:38.898721+0000","last_fullsized":"2024-03-31T15:03:38.898721+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:31:52.119827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-31T15:03:37.971509+0000","last_change":"2024-03-31T15:03:36.962443+0000","last_active":"2024-03-31T15:03:37.971509+0000","last_peered":"2024-03-31T15:03:37.971509+0000","last_clean":"2024-03-31T15:03:37.971509+0000","last_became_active":"2024-03-31T15:03:36.962140+0000","last_became_peered":"2024-03-31T15:03:36.962140+0000","last_unstale":"2024-03-31T15:03:37.971509+0000","last_undegraded":"2024-03-31T15:03:37.971509+0000","last_fullsized":"2024-03-31T15:03:37.971509+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:22:33.812426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-31T15:03:39.380530+0000","last_change":"2024-03-31T15:03:36.985502+0000","last_active":"2024-03-31T15:03:39.380530+0000","last_peered":"2024-03-31T15:03:39.380530+0000","last_clean":"2024-03-31T15:03:39.380530+0000","last_became_active":"2024-03-31T15:03:36.985263+0000","last_became_peered":"2024-03-31T15:03:36.985263+0000","last_unstale":"2024-03-31T15:03:39.380530+0000","last_undegraded":"2024-03-31T15:03:39.380530+0000","last_fullsized":"2024-03-31T15:03:39.380530+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:29:47.722244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-31T15:03:39.382049+0000","last_change":"2024-03-31T15:03:36.981333+0000","last_active":"2024-03-31T15:03:39.382049+0000","last_peered":"2024-03-31T15:03:39.382049+0000","last_clean":"2024-03-31T15:03:39.382049+0000","last_became_active":"2024-03-31T15:03:36.980937+0000","last_became_peered":"2024-03-31T15:03:36.980937+0000","last_unstale":"2024-03-31T15:03:39.382049+0000","last_undegraded":"2024-03-31T15:03:39.382049+0000","last_fullsized":"2024-03-31T15:03:39.382049+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:50:01.491908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-31T15:03:38.898625+0000","last_change":"2024-03-31T15:03:36.989117+0000","last_active":"2024-03-31T15:03:38.898625+0000","last_peered":"2024-03-31T15:03:38.898625+0000","last_clean":"2024-03-31T15:03:38.898625+0000","last_became_active":"2024-03-31T15:03:36.986002+0000","last_became_peered":"2024-03-31T15:03:36.986002+0000","last_unstale":"2024-03-31T15:03:38.898625+0000","last_undegraded":"2024-03-31T15:03:38.898625+0000","last_fullsized":"2024-03-31T15:03:38.898625+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:58:07.229805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-31T15:03:39.381819+0000","last_change":"2024-03-31T15:03:36.981121+0000","last_active":"2024-03-31T15:03:39.381819+0000","last_peered":"2024-03-31T15:03:39.381819+0000","last_clean":"2024-03-31T15:03:39.381819+0000","last_became_active":"2024-03-31T15:03:36.980669+0000","last_became_peered":"2024-03-31T15:03:36.980669+0000","last_unstale":"2024-03-31T15:03:39.381819+0000","last_undegraded":"2024-03-31T15:03:39.381819+0000","last_fullsized":"2024-03-31T15:03:39.381819+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T17:34:55.373127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-31T15:03:38.898283+0000","last_change":"2024-03-31T15:03:36.974140+0000","last_active":"2024-03-31T15:03:38.898283+0000","last_peered":"2024-03-31T15:03:38.898283+0000","last_clean":"2024-03-31T15:03:38.898283+0000","last_became_active":"2024-03-31T15:03:36.973867+0000","last_became_peered":"2024-03-31T15:03:36.973867+0000","last_unstale":"2024-03-31T15:03:38.898283+0000","last_undegraded":"2024-03-31T15:03:38.898283+0000","last_fullsized":"2024-03-31T15:03:38.898283+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:12:48.216307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-31T15:03:37.962657+0000","last_change":"2024-03-31T15:03:36.969620+0000","last_active":"2024-03-31T15:03:37.962657+0000","last_peered":"2024-03-31T15:03:37.962657+0000","last_clean":"2024-03-31T15:03:37.962657+0000","last_became_active":"2024-03-31T15:03:36.969317+0000","last_became_peered":"2024-03-31T15:03:36.969317+0000","last_unstale":"2024-03-31T15:03:37.962657+0000","last_undegraded":"2024-03-31T15:03:37.962657+0000","last_fullsized":"2024-03-31T15:03:37.962657+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T17:27:21.111881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-31T15:03:38.898858+0000","last_change":"2024-03-31T15:03:36.990208+0000","last_active":"2024-03-31T15:03:38.898858+0000","last_peered":"2024-03-31T15:03:38.898858+0000","last_clean":"2024-03-31T15:03:38.898858+0000","last_became_active":"2024-03-31T15:03:36.989706+0000","last_became_peered":"2024-03-31T15:03:36.989706+0000","last_unstale":"2024-03-31T15:03:38.898858+0000","last_undegraded":"2024-03-31T15:03:38.898858+0000","last_fullsized":"2024-03-31T15:03:38.898858+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:07:26.921241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-31T15:03:37.967891+0000","last_change":"2024-03-31T15:03:36.982132+0000","last_active":"2024-03-31T15:03:37.967891+0000","last_peered":"2024-03-31T15:03:37.967891+0000","last_clean":"2024-03-31T15:03:37.967891+0000","last_became_active":"2024-03-31T15:03:36.981697+0000","last_became_peered":"2024-03-31T15:03:36.981697+0000","last_unstale":"2024-03-31T15:03:37.967891+0000","last_undegraded":"2024-03-31T15:03:37.967891+0000","last_fullsized":"2024-03-31T15:03:37.967891+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:22:33.812426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-31T15:03:37.967567+0000","last_change":"2024-03-31T15:03:36.980849+0000","last_active":"2024-03-31T15:03:37.967567+0000","last_peered":"2024-03-31T15:03:37.967567+0000","last_clean":"2024-03-31T15:03:37.967567+0000","last_became_active":"2024-03-31T15:03:36.980535+0000","last_became_peered":"2024-03-31T15:03:36.980535+0000","last_unstale":"2024-03-31T15:03:37.967567+0000","last_undegraded":"2024-03-31T15:03:37.967567+0000","last_fullsized":"2024-03-31T15:03:37.967567+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:27:54.317789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-31T15:03:37.966263+0000","last_change":"2024-03-31T15:03:36.962572+0000","last_active":"2024-03-31T15:03:37.966263+0000","last_peered":"2024-03-31T15:03:37.966263+0000","last_clean":"2024-03-31T15:03:37.966263+0000","last_became_active":"2024-03-31T15:03:36.960772+0000","last_became_peered":"2024-03-31T15:03:36.960772+0000","last_unstale":"2024-03-31T15:03:37.966263+0000","last_undegraded":"2024-03-31T15:03:37.966263+0000","last_fullsized":"2024-03-31T15:03:37.966263+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T15:58:19.498104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-31T15:03:37.963021+0000","last_change":"2024-03-31T15:03:36.960134+0000","last_active":"2024-03-31T15:03:37.963021+0000","last_peered":"2024-03-31T15:03:37.963021+0000","last_clean":"2024-03-31T15:03:37.963021+0000","last_became_active":"2024-03-31T15:03:36.959747+0000","last_became_peered":"2024-03-31T15:03:36.959747+0000","last_unstale":"2024-03-31T15:03:37.963021+0000","last_undegraded":"2024-03-31T15:03:37.963021+0000","last_fullsized":"2024-03-31T15:03:37.963021+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T17:50:16.612568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-31T15:03:39.380710+0000","last_change":"2024-03-31T15:03:36.964930+0000","last_active":"2024-03-31T15:03:39.380710+0000","last_peered":"2024-03-31T15:03:39.380710+0000","last_clean":"2024-03-31T15:03:39.380710+0000","last_became_active":"2024-03-31T15:03:36.964006+0000","last_became_peered":"2024-03-31T15:03:36.964006+0000","last_unstale":"2024-03-31T15:03:39.380710+0000","last_undegraded":"2024-03-31T15:03:39.380710+0000","last_fullsized":"2024-03-31T15:03:39.380710+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:37:26.685300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-31T15:03:37.971000+0000","last_change":"2024-03-31T15:03:36.981622+0000","last_active":"2024-03-31T15:03:37.971000+0000","last_peered":"2024-03-31T15:03:37.971000+0000","last_clean":"2024-03-31T15:03:37.971000+0000","last_became_active":"2024-03-31T15:03:36.981267+0000","last_became_peered":"2024-03-31T15:03:36.981267+0000","last_unstale":"2024-03-31T15:03:37.971000+0000","last_undegraded":"2024-03-31T15:03:37.971000+0000","last_fullsized":"2024-03-31T15:03:37.971000+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T17:27:21.111881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-31T15:03:39.381753+0000","last_change":"2024-03-31T15:03:36.977834+0000","last_active":"2024-03-31T15:03:39.381753+0000","last_peered":"2024-03-31T15:03:39.381753+0000","last_clean":"2024-03-31T15:03:39.381753+0000","last_became_active":"2024-03-31T15:03:36.977469+0000","last_became_peered":"2024-03-31T15:03:36.977469+0000","last_unstale":"2024-03-31T15:03:39.381753+0000","last_undegraded":"2024-03-31T15:03:39.381753+0000","last_fullsized":"2024-03-31T15:03:39.381753+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:05:30.818052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-31T15:03:39.382482+0000","last_change":"2024-03-31T15:03:36.981952+0000","last_active":"2024-03-31T15:03:39.382482+0000","last_peered":"2024-03-31T15:03:39.382482+0000","last_clean":"2024-03-31T15:03:39.382482+0000","last_became_active":"2024-03-31T15:03:36.981615+0000","last_became_peered":"2024-03-31T15:03:36.981615+0000","last_unstale":"2024-03-31T15:03:39.382482+0000","last_undegraded":"2024-03-31T15:03:39.382482+0000","last_fullsized":"2024-03-31T15:03:39.382482+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:31:52.119827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-31T15:03:39.381168+0000","last_change":"2024-03-31T15:03:36.981172+0000","last_active":"2024-03-31T15:03:39.381168+0000","last_peered":"2024-03-31T15:03:39.381168+0000","last_clean":"2024-03-31T15:03:39.381168+0000","last_became_active":"2024-03-31T15:03:36.980303+0000","last_became_peered":"2024-03-31T15:03:36.980303+0000","last_unstale":"2024-03-31T15:03:39.381168+0000","last_undegraded":"2024-03-31T15:03:39.381168+0000","last_fullsized":"2024-03-31T15:03:39.381168+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:19:24.339229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-31T15:03:38.898702+0000","last_change":"2024-03-31T15:03:36.991712+0000","last_active":"2024-03-31T15:03:38.898702+0000","last_peered":"2024-03-31T15:03:38.898702+0000","last_clean":"2024-03-31T15:03:38.898702+0000","last_became_active":"2024-03-31T15:03:36.988177+0000","last_became_peered":"2024-03-31T15:03:36.988177+0000","last_unstale":"2024-03-31T15:03:38.898702+0000","last_undegraded":"2024-03-31T15:03:38.898702+0000","last_fullsized":"2024-03-31T15:03:38.898702+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:25:31.055690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-31T15:03:37.967399+0000","last_change":"2024-03-31T15:03:36.968423+0000","last_active":"2024-03-31T15:03:37.967399+0000","last_peered":"2024-03-31T15:03:37.967399+0000","last_clean":"2024-03-31T15:03:37.967399+0000","last_became_active":"2024-03-31T15:03:36.968014+0000","last_became_peered":"2024-03-31T15:03:36.968014+0000","last_unstale":"2024-03-31T15:03:37.967399+0000","last_undegraded":"2024-03-31T15:03:37.967399+0000","last_fullsized":"2024-03-31T15:03:37.967399+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T17:15:21.722119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-31T15:03:39.380826+0000","last_change":"2024-03-31T15:03:36.975734+0000","last_active":"2024-03-31T15:03:39.380826+0000","last_peered":"2024-03-31T15:03:39.380826+0000","last_clean":"2024-03-31T15:03:39.380826+0000","last_became_active":"2024-03-31T15:03:36.970929+0000","last_became_peered":"2024-03-31T15:03:36.970929+0000","last_unstale":"2024-03-31T15:03:39.380826+0000","last_undegraded":"2024-03-31T15:03:39.380826+0000","last_fullsized":"2024-03-31T15:03:39.380826+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T19:12:45.358470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-31T15:03:39.381878+0000","last_change":"2024-03-31T15:03:36.982234+0000","last_active":"2024-03-31T15:03:39.381878+0000","last_peered":"2024-03-31T15:03:39.381878+0000","last_clean":"2024-03-31T15:03:39.381878+0000","last_became_active":"2024-03-31T15:03:36.981359+0000","last_became_peered":"2024-03-31T15:03:36.981359+0000","last_unstale":"2024-03-31T15:03:39.381878+0000","last_undegraded":"2024-03-31T15:03:39.381878+0000","last_fullsized":"2024-03-31T15:03:39.381878+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:12:48.216307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-31T15:03:37.962738+0000","last_change":"2024-03-31T15:03:36.965603+0000","last_active":"2024-03-31T15:03:37.962738+0000","last_peered":"2024-03-31T15:03:37.962738+0000","last_clean":"2024-03-31T15:03:37.962738+0000","last_became_active":"2024-03-31T15:03:36.965339+0000","last_became_peered":"2024-03-31T15:03:36.965339+0000","last_unstale":"2024-03-31T15:03:37.962738+0000","last_undegraded":"2024-03-31T15:03:37.962738+0000","last_fullsized":"2024-03-31T15:03:37.962738+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:37:46.279789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-31T15:03:39.381792+0000","last_change":"2024-03-31T15:03:36.977130+0000","last_active":"2024-03-31T15:03:39.381792+0000","last_peered":"2024-03-31T15:03:39.381792+0000","last_clean":"2024-03-31T15:03:39.381792+0000","last_became_active":"2024-03-31T15:03:36.973022+0000","last_became_peered":"2024-03-31T15:03:36.973022+0000","last_unstale":"2024-03-31T15:03:39.381792+0000","last_undegraded":"2024-03-31T15:03:39.381792+0000","last_fullsized":"2024-03-31T15:03:39.381792+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:34:49.982234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-31T15:03:37.963323+0000","last_change":"2024-03-31T15:03:36.967074+0000","last_active":"2024-03-31T15:03:37.963323+0000","last_peered":"2024-03-31T15:03:37.963323+0000","last_clean":"2024-03-31T15:03:37.963323+0000","last_became_active":"2024-03-31T15:03:36.966802+0000","last_became_peered":"2024-03-31T15:03:36.966802+0000","last_unstale":"2024-03-31T15:03:37.963323+0000","last_undegraded":"2024-03-31T15:03:37.963323+0000","last_fullsized":"2024-03-31T15:03:37.963323+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:49:08.771388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-31T15:03:37.967818+0000","last_change":"2024-03-31T15:03:36.977099+0000","last_active":"2024-03-31T15:03:37.967818+0000","last_peered":"2024-03-31T15:03:37.967818+0000","last_clean":"2024-03-31T15:03:37.967818+0000","last_became_active":"2024-03-31T15:03:36.976814+0000","last_became_peered":"2024-03-31T15:03:36.976814+0000","last_unstale":"2024-03-31T15:03:37.967818+0000","last_undegraded":"2024-03-31T15:03:37.967818+0000","last_fullsized":"2024-03-31T15:03:37.967818+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:37:46.279789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-31T15:03:37.971261+0000","last_change":"2024-03-31T15:03:36.968708+0000","last_active":"2024-03-31T15:03:37.971261+0000","last_peered":"2024-03-31T15:03:37.971261+0000","last_clean":"2024-03-31T15:03:37.971261+0000","last_became_active":"2024-03-31T15:03:36.968292+0000","last_became_peered":"2024-03-31T15:03:36.968292+0000","last_unstale":"2024-03-31T15:03:37.971261+0000","last_undegraded":"2024-03-31T15:03:37.971261+0000","last_fullsized":"2024-03-31T15:03:37.971261+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:37:46.279789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-31T15:03:37.963169+0000","last_change":"2024-03-31T15:03:36.970468+0000","last_active":"2024-03-31T15:03:37.963169+0000","last_peered":"2024-03-31T15:03:37.963169+0000","last_clean":"2024-03-31T15:03:37.963169+0000","last_became_active":"2024-03-31T15:03:36.970233+0000","last_became_peered":"2024-03-31T15:03:36.970233+0000","last_unstale":"2024-03-31T15:03:37.963169+0000","last_undegraded":"2024-03-31T15:03:37.963169+0000","last_fullsized":"2024-03-31T15:03:37.963169+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:16:25.725098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-31T15:03:39.380875+0000","last_change":"2024-03-31T15:03:36.962951+0000","last_active":"2024-03-31T15:03:39.380875+0000","last_peered":"2024-03-31T15:03:39.380875+0000","last_clean":"2024-03-31T15:03:39.380875+0000","last_became_active":"2024-03-31T15:03:36.961876+0000","last_became_peered":"2024-03-31T15:03:36.961876+0000","last_unstale":"2024-03-31T15:03:39.380875+0000","last_undegraded":"2024-03-31T15:03:39.380875+0000","last_fullsized":"2024-03-31T15:03:39.380875+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T17:15:21.722119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-31T15:03:39.381964+0000","last_change":"2024-03-31T15:03:36.979570+0000","last_active":"2024-03-31T15:03:39.381964+0000","last_peered":"2024-03-31T15:03:39.381964+0000","last_clean":"2024-03-31T15:03:39.381964+0000","last_became_active":"2024-03-31T15:03:36.979287+0000","last_became_peered":"2024-03-31T15:03:36.979287+0000","last_unstale":"2024-03-31T15:03:39.381964+0000","last_undegraded":"2024-03-31T15:03:39.381964+0000","last_fullsized":"2024-03-31T15:03:39.381964+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:07:48.318096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-31T15:03:39.380389+0000","last_change":"2024-03-31T15:03:36.988981+0000","last_active":"2024-03-31T15:03:39.380389+0000","last_peered":"2024-03-31T15:03:39.380389+0000","last_clean":"2024-03-31T15:03:39.380389+0000","last_became_active":"2024-03-31T15:03:36.988582+0000","last_became_peered":"2024-03-31T15:03:36.988582+0000","last_unstale":"2024-03-31T15:03:39.380389+0000","last_undegraded":"2024-03-31T15:03:39.380389+0000","last_fullsized":"2024-03-31T15:03:39.380389+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:48:54.968379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-31T15:03:37.967288+0000","last_change":"2024-03-31T15:03:36.975267+0000","last_active":"2024-03-31T15:03:37.967288+0000","last_peered":"2024-03-31T15:03:37.967288+0000","last_clean":"2024-03-31T15:03:37.967288+0000","last_became_active":"2024-03-31T15:03:36.972549+0000","last_became_peered":"2024-03-31T15:03:36.972549+0000","last_unstale":"2024-03-31T15:03:37.967288+0000","last_undegraded":"2024-03-31T15:03:37.967288+0000","last_fullsized":"2024-03-31T15:03:37.967288+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T19:12:45.358470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-31T15:03:37.966007+0000","last_change":"2024-03-31T15:03:36.964214+0000","last_active":"2024-03-31T15:03:37.966007+0000","last_peered":"2024-03-31T15:03:37.966007+0000","last_clean":"2024-03-31T15:03:37.966007+0000","last_became_active":"2024-03-31T15:03:36.963926+0000","last_became_peered":"2024-03-31T15:03:36.963926+0000","last_unstale":"2024-03-31T15:03:37.966007+0000","last_undegraded":"2024-03-31T15:03:37.966007+0000","last_fullsized":"2024-03-31T15:03:37.966007+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:26:54.177359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-31T15:03:37.963122+0000","last_change":"2024-03-31T15:03:35.966649+0000","last_active":"2024-03-31T15:03:37.963122+0000","last_peered":"2024-03-31T15:03:37.963122+0000","last_clean":"2024-03-31T15:03:37.963122+0000","last_became_active":"2024-03-31T15:03:35.964612+0000","last_became_peered":"2024-03-31T15:03:35.964612+0000","last_unstale":"2024-03-31T15:03:37.963122+0000","last_undegraded":"2024-03-31T15:03:37.963122+0000","last_fullsized":"2024-03-31T15:03:37.963122+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_clean_scrub_stamp":"2024-03-31T15:03:34.926145+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T15:38:42.097818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-31T15:03:39.380987+0000","last_change":"2024-03-31T15:03:36.961519+0000","last_active":"2024-03-31T15:03:39.380987+0000","last_peered":"2024-03-31T15:03:39.380987+0000","last_clean":"2024-03-31T15:03:39.380987+0000","last_became_active":"2024-03-31T15:03:36.961032+0000","last_became_peered":"2024-03-31T15:03:36.961032+0000","last_unstale":"2024-03-31T15:03:39.380987+0000","last_undegraded":"2024-03-31T15:03:39.380987+0000","last_fullsized":"2024-03-31T15:03:39.380987+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:27:54.317789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-31T15:03:37.967261+0000","last_change":"2024-03-31T15:03:35.961058+0000","last_active":"2024-03-31T15:03:37.967261+0000","last_peered":"2024-03-31T15:03:37.967261+0000","last_clean":"2024-03-31T15:03:37.967261+0000","last_became_active":"2024-03-31T15:03:35.960684+0000","last_became_peered":"2024-03-31T15:03:35.960684+0000","last_unstale":"2024-03-31T15:03:37.967261+0000","last_undegraded":"2024-03-31T15:03:37.967261+0000","last_fullsized":"2024-03-31T15:03:37.967261+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_clean_scrub_stamp":"2024-03-31T15:03:34.926145+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:07:06.210288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-31T15:03:37.962722+0000","last_change":"2024-03-31T15:03:35.963374+0000","last_active":"2024-03-31T15:03:37.962722+0000","last_peered":"2024-03-31T15:03:37.962722+0000","last_clean":"2024-03-31T15:03:37.962722+0000","last_became_active":"2024-03-31T15:03:35.963009+0000","last_became_peered":"2024-03-31T15:03:35.963009+0000","last_unstale":"2024-03-31T15:03:37.962722+0000","last_undegraded":"2024-03-31T15:03:37.962722+0000","last_fullsized":"2024-03-31T15:03:37.962722+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_clean_scrub_stamp":"2024-03-31T15:03:34.926145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:36:14.576891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-31T15:03:37.962735+0000","last_change":"2024-03-31T15:03:36.972478+0000","last_active":"2024-03-31T15:03:37.962735+0000","last_peered":"2024-03-31T15:03:37.962735+0000","last_clean":"2024-03-31T15:03:37.962735+0000","last_became_active":"2024-03-31T15:03:36.972297+0000","last_became_peered":"2024-03-31T15:03:36.972297+0000","last_unstale":"2024-03-31T15:03:37.962735+0000","last_undegraded":"2024-03-31T15:03:37.962735+0000","last_fullsized":"2024-03-31T15:03:37.962735+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:27:54.317789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-31T15:03:39.381858+0000","last_change":"2024-03-31T15:03:35.963562+0000","last_active":"2024-03-31T15:03:39.381858+0000","last_peered":"2024-03-31T15:03:39.381858+0000","last_clean":"2024-03-31T15:03:39.381858+0000","last_became_active":"2024-03-31T15:03:35.961838+0000","last_became_peered":"2024-03-31T15:03:35.961838+0000","last_unstale":"2024-03-31T15:03:39.381858+0000","last_undegraded":"2024-03-31T15:03:39.381858+0000","last_fullsized":"2024-03-31T15:03:39.381858+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_clean_scrub_stamp":"2024-03-31T15:03:34.926145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:27:19.823151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-31T15:03:37.970977+0000","last_change":"2024-03-31T15:03:36.970365+0000","last_active":"2024-03-31T15:03:37.970977+0000","last_peered":"2024-03-31T15:03:37.970977+0000","last_clean":"2024-03-31T15:03:37.970977+0000","last_became_active":"2024-03-31T15:03:36.970060+0000","last_became_peered":"2024-03-31T15:03:36.970060+0000","last_unstale":"2024-03-31T15:03:37.970977+0000","last_undegraded":"2024-03-31T15:03:37.970977+0000","last_fullsized":"2024-03-31T15:03:37.970977+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T17:15:21.722119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-31T15:03:37.967603+0000","last_change":"2024-03-31T15:03:35.956977+0000","last_active":"2024-03-31T15:03:37.967603+0000","last_peered":"2024-03-31T15:03:37.967603+0000","last_clean":"2024-03-31T15:03:37.967603+0000","last_became_active":"2024-03-31T15:03:35.954617+0000","last_became_peered":"2024-03-31T15:03:35.954617+0000","last_unstale":"2024-03-31T15:03:37.967603+0000","last_undegraded":"2024-03-31T15:03:37.967603+0000","last_fullsized":"2024-03-31T15:03:37.967603+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_clean_scrub_stamp":"2024-03-31T15:03:34.926145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T17:31:20.984335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-31T15:03:39.382328+0000","last_change":"2024-03-31T15:03:36.981437+0000","last_active":"2024-03-31T15:03:39.382328+0000","last_peered":"2024-03-31T15:03:39.382328+0000","last_clean":"2024-03-31T15:03:39.382328+0000","last_became_active":"2024-03-31T15:03:36.981050+0000","last_became_peered":"2024-03-31T15:03:36.981050+0000","last_unstale":"2024-03-31T15:03:39.382328+0000","last_undegraded":"2024-03-31T15:03:39.382328+0000","last_fullsized":"2024-03-31T15:03:39.382328+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T17:07:57.546757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-31T15:03:39.382317+0000","last_change":"2024-03-31T15:03:35.954630+0000","last_active":"2024-03-31T15:03:39.382317+0000","last_peered":"2024-03-31T15:03:39.382317+0000","last_clean":"2024-03-31T15:03:39.382317+0000","last_became_active":"2024-03-31T15:03:35.954149+0000","last_became_peered":"2024-03-31T15:03:35.954149+0000","last_unstale":"2024-03-31T15:03:39.382317+0000","last_undegraded":"2024-03-31T15:03:39.382317+0000","last_fullsized":"2024-03-31T15:03:39.382317+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_clean_scrub_stamp":"2024-03-31T15:03:34.926145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:23:12.843244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-31T15:03:39.381016+0000","last_change":"2024-03-31T15:03:36.988008+0000","last_active":"2024-03-31T15:03:39.381016+0000","last_peered":"2024-03-31T15:03:39.381016+0000","last_clean":"2024-03-31T15:03:39.381016+0000","last_became_active":"2024-03-31T15:03:36.987732+0000","last_became_peered":"2024-03-31T15:03:36.987732+0000","last_unstale":"2024-03-31T15:03:39.381016+0000","last_undegraded":"2024-03-31T15:03:39.381016+0000","last_fullsized":"2024-03-31T15:03:39.381016+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:37:46.279789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"20'3","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-31T15:03:37.971313+0000","last_change":"2024-03-31T15:03:35.959827+0000","last_active":"2024-03-31T15:03:37.971313+0000","last_peered":"2024-03-31T15:03:37.971313+0000","last_clean":"2024-03-31T15:03:37.971313+0000","last_became_active":"2024-03-31T15:03:35.959564+0000","last_became_peered":"2024-03-31T15:03:35.959564+0000","last_unstale":"2024-03-31T15:03:37.971313+0000","last_undegraded":"2024-03-31T15:03:37.971313+0000","last_fullsized":"2024-03-31T15:03:37.971313+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_clean_scrub_stamp":"2024-03-31T15:03:34.926145+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:36:14.576891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-31T15:03:37.966441+0000","last_change":"2024-03-31T15:03:36.982726+0000","last_active":"2024-03-31T15:03:37.966441+0000","last_peered":"2024-03-31T15:03:37.966441+0000","last_clean":"2024-03-31T15:03:37.966441+0000","last_became_active":"2024-03-31T15:03:36.981893+0000","last_became_peered":"2024-03-31T15:03:36.981893+0000","last_unstale":"2024-03-31T15:03:37.966441+0000","last_undegraded":"2024-03-31T15:03:37.966441+0000","last_fullsized":"2024-03-31T15:03:37.966441+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:19:24.339229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"20'6","reported_seq":20,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-31T15:03:37.967382+0000","last_change":"2024-03-31T15:03:35.970154+0000","last_active":"2024-03-31T15:03:37.967382+0000","last_peered":"2024-03-31T15:03:37.967382+0000","last_clean":"2024-03-31T15:03:37.967382+0000","last_became_active":"2024-03-31T15:03:35.969877+0000","last_became_peered":"2024-03-31T15:03:35.969877+0000","last_unstale":"2024-03-31T15:03:37.967382+0000","last_undegraded":"2024-03-31T15:03:37.967382+0000","last_fullsized":"2024-03-31T15:03:37.967382+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_clean_scrub_stamp":"2024-03-31T15:03:34.926145+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:36:14.576891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-31T15:03:38.898534+0000","last_change":"2024-03-31T15:03:36.980541+0000","last_active":"2024-03-31T15:03:38.898534+0000","last_peered":"2024-03-31T15:03:38.898534+0000","last_clean":"2024-03-31T15:03:38.898534+0000","last_became_active":"2024-03-31T15:03:36.980276+0000","last_became_peered":"2024-03-31T15:03:36.980276+0000","last_unstale":"2024-03-31T15:03:38.898534+0000","last_undegraded":"2024-03-31T15:03:38.898534+0000","last_fullsized":"2024-03-31T15:03:38.898534+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:41:13.119207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-31T15:03:39.380977+0000","last_change":"2024-03-31T15:03:35.960798+0000","last_active":"2024-03-31T15:03:39.380977+0000","last_peered":"2024-03-31T15:03:39.380977+0000","last_clean":"2024-03-31T15:03:39.380977+0000","last_became_active":"2024-03-31T15:03:35.960522+0000","last_became_peered":"2024-03-31T15:03:35.960522+0000","last_unstale":"2024-03-31T15:03:39.380977+0000","last_undegraded":"2024-03-31T15:03:39.380977+0000","last_fullsized":"2024-03-31T15:03:39.380977+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_clean_scrub_stamp":"2024-03-31T15:03:34.926145+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T16:07:33.984402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-31T15:03:39.380750+0000","last_change":"2024-03-31T15:03:36.972506+0000","last_active":"2024-03-31T15:03:39.380750+0000","last_peered":"2024-03-31T15:03:39.380750+0000","last_clean":"2024-03-31T15:03:39.380750+0000","last_became_active":"2024-03-31T15:03:36.972194+0000","last_became_peered":"2024-03-31T15:03:36.972194+0000","last_unstale":"2024-03-31T15:03:39.380750+0000","last_undegraded":"2024-03-31T15:03:39.380750+0000","last_fullsized":"2024-03-31T15:03:39.380750+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:48:13.691269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-31T15:03:39.381975+0000","last_change":"2024-03-31T15:03:35.966260+0000","last_active":"2024-03-31T15:03:39.381975+0000","last_peered":"2024-03-31T15:03:39.381975+0000","last_clean":"2024-03-31T15:03:39.381975+0000","last_became_active":"2024-03-31T15:03:35.965676+0000","last_became_peered":"2024-03-31T15:03:35.965676+0000","last_unstale":"2024-03-31T15:03:39.381975+0000","last_undegraded":"2024-03-31T15:03:39.381975+0000","last_fullsized":"2024-03-31T15:03:39.381975+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_clean_scrub_stamp":"2024-03-31T15:03:34.926145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:26:53.171901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-31T15:03:38.898638+0000","last_change":"2024-03-31T15:03:36.988567+0000","last_active":"2024-03-31T15:03:38.898638+0000","last_peered":"2024-03-31T15:03:38.898638+0000","last_clean":"2024-03-31T15:03:38.898638+0000","last_became_active":"2024-03-31T15:03:36.985668+0000","last_became_peered":"2024-03-31T15:03:36.985668+0000","last_unstale":"2024-03-31T15:03:38.898638+0000","last_undegraded":"2024-03-31T15:03:38.898638+0000","last_fullsized":"2024-03-31T15:03:38.898638+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:14:42.246629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-31T15:03:37.967494+0000","last_change":"2024-03-31T15:03:35.961562+0000","last_active":"2024-03-31T15:03:37.967494+0000","last_peered":"2024-03-31T15:03:37.967494+0000","last_clean":"2024-03-31T15:03:37.967494+0000","last_became_active":"2024-03-31T15:03:35.961280+0000","last_became_peered":"2024-03-31T15:03:35.961280+0000","last_unstale":"2024-03-31T15:03:37.967494+0000","last_undegraded":"2024-03-31T15:03:37.967494+0000","last_fullsized":"2024-03-31T15:03:37.967494+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_clean_scrub_stamp":"2024-03-31T15:03:34.926145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T15:38:42.097818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-31T15:03:37.967298+0000","last_change":"2024-03-31T15:03:36.979417+0000","last_active":"2024-03-31T15:03:37.967298+0000","last_peered":"2024-03-31T15:03:37.967298+0000","last_clean":"2024-03-31T15:03:37.967298+0000","last_became_active":"2024-03-31T15:03:36.973021+0000","last_became_peered":"2024-03-31T15:03:36.973021+0000","last_unstale":"2024-03-31T15:03:37.967298+0000","last_undegraded":"2024-03-31T15:03:37.967298+0000","last_fullsized":"2024-03-31T15:03:37.967298+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:48:13.691269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"20'8","reported_seq":22,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-31T15:03:39.380233+0000","last_change":"2024-03-31T15:03:35.955608+0000","last_active":"2024-03-31T15:03:39.380233+0000","last_peered":"2024-03-31T15:03:39.380233+0000","last_clean":"2024-03-31T15:03:39.380233+0000","last_became_active":"2024-03-31T15:03:35.955295+0000","last_became_peered":"2024-03-31T15:03:35.955295+0000","last_unstale":"2024-03-31T15:03:39.380233+0000","last_undegraded":"2024-03-31T15:03:39.380233+0000","last_fullsized":"2024-03-31T15:03:39.380233+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_clean_scrub_stamp":"2024-03-31T15:03:34.926145+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:23:12.843244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-31T15:03:38.898341+0000","last_change":"2024-03-31T15:03:36.990806+0000","last_active":"2024-03-31T15:03:38.898341+0000","last_peered":"2024-03-31T15:03:38.898341+0000","last_clean":"2024-03-31T15:03:38.898341+0000","last_became_active":"2024-03-31T15:03:36.990542+0000","last_became_peered":"2024-03-31T15:03:36.990542+0000","last_unstale":"2024-03-31T15:03:38.898341+0000","last_undegraded":"2024-03-31T15:03:38.898341+0000","last_fullsized":"2024-03-31T15:03:38.898341+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:07:48.318096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-31T15:03:37.966654+0000","last_change":"2024-03-31T15:03:35.968824+0000","last_active":"2024-03-31T15:03:37.966654+0000","last_peered":"2024-03-31T15:03:37.966654+0000","last_clean":"2024-03-31T15:03:37.966654+0000","last_became_active":"2024-03-31T15:03:35.968431+0000","last_became_peered":"2024-03-31T15:03:35.968431+0000","last_unstale":"2024-03-31T15:03:37.966654+0000","last_undegraded":"2024-03-31T15:03:37.966654+0000","last_fullsized":"2024-03-31T15:03:37.966654+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_clean_scrub_stamp":"2024-03-31T15:03:34.926145+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:30:58.752050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-31T15:03:38.898638+0000","last_change":"2024-03-31T15:03:36.993283+0000","last_active":"2024-03-31T15:03:38.898638+0000","last_peered":"2024-03-31T15:03:38.898638+0000","last_clean":"2024-03-31T15:03:38.898638+0000","last_became_active":"2024-03-31T15:03:36.992977+0000","last_became_peered":"2024-03-31T15:03:36.992977+0000","last_unstale":"2024-03-31T15:03:38.898638+0000","last_undegraded":"2024-03-31T15:03:38.898638+0000","last_fullsized":"2024-03-31T15:03:38.898638+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T17:07:57.546757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-31T15:03:39.381206+0000","last_change":"2024-03-31T15:03:35.969251+0000","last_active":"2024-03-31T15:03:39.381206+0000","last_peered":"2024-03-31T15:03:39.381206+0000","last_clean":"2024-03-31T15:03:39.381206+0000","last_became_active":"2024-03-31T15:03:35.968876+0000","last_became_peered":"2024-03-31T15:03:35.968876+0000","last_unstale":"2024-03-31T15:03:39.381206+0000","last_undegraded":"2024-03-31T15:03:39.381206+0000","last_fullsized":"2024-03-31T15:03:39.381206+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_clean_scrub_stamp":"2024-03-31T15:03:34.926145+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:27:19.823151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-31T15:03:38.898891+0000","last_change":"2024-03-31T15:03:36.986107+0000","last_active":"2024-03-31T15:03:38.898891+0000","last_peered":"2024-03-31T15:03:38.898891+0000","last_clean":"2024-03-31T15:03:38.898891+0000","last_became_active":"2024-03-31T15:03:36.985749+0000","last_became_peered":"2024-03-31T15:03:36.985749+0000","last_unstale":"2024-03-31T15:03:38.898891+0000","last_undegraded":"2024-03-31T15:03:38.898891+0000","last_fullsized":"2024-03-31T15:03:38.898891+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:22:28.970344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-31T15:03:37.971354+0000","last_change":"2024-03-31T15:03:35.965507+0000","last_active":"2024-03-31T15:03:37.971354+0000","last_peered":"2024-03-31T15:03:37.971354+0000","last_clean":"2024-03-31T15:03:37.971354+0000","last_became_active":"2024-03-31T15:03:35.964843+0000","last_became_peered":"2024-03-31T15:03:35.964843+0000","last_unstale":"2024-03-31T15:03:37.971354+0000","last_undegraded":"2024-03-31T15:03:37.971354+0000","last_fullsized":"2024-03-31T15:03:37.971354+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_clean_scrub_stamp":"2024-03-31T15:03:34.926145+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T17:31:20.984335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-31T15:03:39.382209+0000","last_change":"2024-03-31T15:03:36.979781+0000","last_active":"2024-03-31T15:03:39.382209+0000","last_peered":"2024-03-31T15:03:39.382209+0000","last_clean":"2024-03-31T15:03:39.382209+0000","last_became_active":"2024-03-31T15:03:36.979517+0000","last_became_peered":"2024-03-31T15:03:36.979517+0000","last_unstale":"2024-03-31T15:03:39.382209+0000","last_undegraded":"2024-03-31T15:03:39.382209+0000","last_fullsized":"2024-03-31T15:03:39.382209+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:58:07.229805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-31T15:03:39.380445+0000","last_change":"2024-03-31T15:03:35.960305+0000","last_active":"2024-03-31T15:03:39.380445+0000","last_peered":"2024-03-31T15:03:39.380445+0000","last_clean":"2024-03-31T15:03:39.380445+0000","last_became_active":"2024-03-31T15:03:35.960006+0000","last_became_peered":"2024-03-31T15:03:35.960006+0000","last_unstale":"2024-03-31T15:03:39.380445+0000","last_undegraded":"2024-03-31T15:03:39.380445+0000","last_fullsized":"2024-03-31T15:03:39.380445+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_clean_scrub_stamp":"2024-03-31T15:03:34.926145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:11:41.679652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-31T15:03:38.898523+0000","last_change":"2024-03-31T15:03:36.985221+0000","last_active":"2024-03-31T15:03:38.898523+0000","last_peered":"2024-03-31T15:03:38.898523+0000","last_clean":"2024-03-31T15:03:38.898523+0000","last_became_active":"2024-03-31T15:03:36.984963+0000","last_became_peered":"2024-03-31T15:03:36.984963+0000","last_unstale":"2024-03-31T15:03:38.898523+0000","last_undegraded":"2024-03-31T15:03:38.898523+0000","last_fullsized":"2024-03-31T15:03:38.898523+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:50:01.491908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":170,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-31T15:03:38.898259+0000","last_change":"2024-03-31T15:03:24.329890+0000","last_active":"2024-03-31T15:03:38.898259+0000","last_peered":"2024-03-31T15:03:38.898259+0000","last_clean":"2024-03-31T15:03:38.898259+0000","last_became_active":"2024-03-31T15:03:24.328882+0000","last_became_peered":"2024-03-31T15:03:24.328882+0000","last_unstale":"2024-03-31T15:03:38.898259+0000","last_undegraded":"2024-03-31T15:03:38.898259+0000","last_fullsized":"2024-03-31T15:03:38.898259+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:22.885535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:22.885535+0000","last_clean_scrub_stamp":"2024-03-31T15:03:22.885535+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:18:29.004296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-31T15:03:39.381810+0000","last_change":"2024-03-31T15:03:35.952977+0000","last_active":"2024-03-31T15:03:39.381810+0000","last_peered":"2024-03-31T15:03:39.381810+0000","last_clean":"2024-03-31T15:03:39.381810+0000","last_became_active":"2024-03-31T15:03:35.951288+0000","last_became_peered":"2024-03-31T15:03:35.951288+0000","last_unstale":"2024-03-31T15:03:39.381810+0000","last_undegraded":"2024-03-31T15:03:39.381810+0000","last_fullsized":"2024-03-31T15:03:39.381810+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_clean_scrub_stamp":"2024-03-31T15:03:34.926145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:11:41.679652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-31T15:03:37.962984+0000","last_change":"2024-03-31T15:03:36.980592+0000","last_active":"2024-03-31T15:03:37.962984+0000","last_peered":"2024-03-31T15:03:37.962984+0000","last_clean":"2024-03-31T15:03:37.962984+0000","last_became_active":"2024-03-31T15:03:36.980285+0000","last_became_peered":"2024-03-31T15:03:36.980285+0000","last_unstale":"2024-03-31T15:03:37.962984+0000","last_undegraded":"2024-03-31T15:03:37.962984+0000","last_fullsized":"2024-03-31T15:03:37.962984+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T17:01:42.202122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-31T15:03:38.898198+0000","last_change":"2024-03-31T15:03:35.968364+0000","last_active":"2024-03-31T15:03:38.898198+0000","last_peered":"2024-03-31T15:03:38.898198+0000","last_clean":"2024-03-31T15:03:38.898198+0000","last_became_active":"2024-03-31T15:03:35.968047+0000","last_became_peered":"2024-03-31T15:03:35.968047+0000","last_unstale":"2024-03-31T15:03:38.898198+0000","last_undegraded":"2024-03-31T15:03:38.898198+0000","last_fullsized":"2024-03-31T15:03:38.898198+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_clean_scrub_stamp":"2024-03-31T15:03:34.926145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:26:53.171901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-31T15:03:39.380580+0000","last_change":"2024-03-31T15:03:36.963584+0000","last_active":"2024-03-31T15:03:39.380580+0000","last_peered":"2024-03-31T15:03:39.380580+0000","last_clean":"2024-03-31T15:03:39.380580+0000","last_became_active":"2024-03-31T15:03:36.963216+0000","last_became_peered":"2024-03-31T15:03:36.963216+0000","last_unstale":"2024-03-31T15:03:39.380580+0000","last_undegraded":"2024-03-31T15:03:39.380580+0000","last_fullsized":"2024-03-31T15:03:39.380580+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:29:57.089580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-31T15:03:39.380342+0000","last_change":"2024-03-31T15:03:35.963196+0000","last_active":"2024-03-31T15:03:39.380342+0000","last_peered":"2024-03-31T15:03:39.380342+0000","last_clean":"2024-03-31T15:03:39.380342+0000","last_became_active":"2024-03-31T15:03:35.962746+0000","last_became_peered":"2024-03-31T15:03:35.962746+0000","last_unstale":"2024-03-31T15:03:39.380342+0000","last_undegraded":"2024-03-31T15:03:39.380342+0000","last_fullsized":"2024-03-31T15:03:39.380342+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_clean_scrub_stamp":"2024-03-31T15:03:34.926145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:00:32.840818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-31T15:03:37.970979+0000","last_change":"2024-03-31T15:03:36.972190+0000","last_active":"2024-03-31T15:03:37.970979+0000","last_peered":"2024-03-31T15:03:37.970979+0000","last_clean":"2024-03-31T15:03:37.970979+0000","last_became_active":"2024-03-31T15:03:36.971866+0000","last_became_peered":"2024-03-31T15:03:36.971866+0000","last_unstale":"2024-03-31T15:03:37.970979+0000","last_undegraded":"2024-03-31T15:03:37.970979+0000","last_fullsized":"2024-03-31T15:03:37.970979+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T19:12:45.358470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-31T15:03:39.382038+0000","last_change":"2024-03-31T15:03:35.953803+0000","last_active":"2024-03-31T15:03:39.382038+0000","last_peered":"2024-03-31T15:03:39.382038+0000","last_clean":"2024-03-31T15:03:39.382038+0000","last_became_active":"2024-03-31T15:03:35.953459+0000","last_became_peered":"2024-03-31T15:03:35.953459+0000","last_unstale":"2024-03-31T15:03:39.382038+0000","last_undegraded":"2024-03-31T15:03:39.382038+0000","last_fullsized":"2024-03-31T15:03:39.382038+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_clean_scrub_stamp":"2024-03-31T15:03:34.926145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T15:58:18.492645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-31T15:03:37.966372+0000","last_change":"2024-03-31T15:03:36.966599+0000","last_active":"2024-03-31T15:03:37.966372+0000","last_peered":"2024-03-31T15:03:37.966372+0000","last_clean":"2024-03-31T15:03:37.966372+0000","last_became_active":"2024-03-31T15:03:36.964867+0000","last_became_peered":"2024-03-31T15:03:36.964867+0000","last_unstale":"2024-03-31T15:03:37.966372+0000","last_undegraded":"2024-03-31T15:03:37.966372+0000","last_fullsized":"2024-03-31T15:03:37.966372+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:48:54.968379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-31T15:03:37.971175+0000","last_change":"2024-03-31T15:03:35.958174+0000","last_active":"2024-03-31T15:03:37.971175+0000","last_peered":"2024-03-31T15:03:37.971175+0000","last_clean":"2024-03-31T15:03:37.971175+0000","last_became_active":"2024-03-31T15:03:35.957870+0000","last_became_peered":"2024-03-31T15:03:35.957870+0000","last_unstale":"2024-03-31T15:03:37.971175+0000","last_undegraded":"2024-03-31T15:03:37.971175+0000","last_fullsized":"2024-03-31T15:03:37.971175+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_clean_scrub_stamp":"2024-03-31T15:03:34.926145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T15:38:42.097818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-31T15:03:39.382417+0000","last_change":"2024-03-31T15:03:36.980337+0000","last_active":"2024-03-31T15:03:39.382417+0000","last_peered":"2024-03-31T15:03:39.382417+0000","last_clean":"2024-03-31T15:03:39.382417+0000","last_became_active":"2024-03-31T15:03:36.980082+0000","last_became_peered":"2024-03-31T15:03:36.980082+0000","last_unstale":"2024-03-31T15:03:39.382417+0000","last_undegraded":"2024-03-31T15:03:39.382417+0000","last_fullsized":"2024-03-31T15:03:39.382417+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:25:31.055690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-31T15:03:38.898776+0000","last_change":"2024-03-31T15:03:35.969598+0000","last_active":"2024-03-31T15:03:38.898776+0000","last_peered":"2024-03-31T15:03:38.898776+0000","last_clean":"2024-03-31T15:03:38.898776+0000","last_became_active":"2024-03-31T15:03:35.969253+0000","last_became_peered":"2024-03-31T15:03:35.969253+0000","last_unstale":"2024-03-31T15:03:38.898776+0000","last_undegraded":"2024-03-31T15:03:38.898776+0000","last_fullsized":"2024-03-31T15:03:38.898776+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_clean_scrub_stamp":"2024-03-31T15:03:34.926145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:55:57.509587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-31T15:03:37.971241+0000","last_change":"2024-03-31T15:03:36.983933+0000","last_active":"2024-03-31T15:03:37.971241+0000","last_peered":"2024-03-31T15:03:37.971241+0000","last_clean":"2024-03-31T15:03:37.971241+0000","last_became_active":"2024-03-31T15:03:36.983650+0000","last_became_peered":"2024-03-31T15:03:36.983650+0000","last_unstale":"2024-03-31T15:03:37.971241+0000","last_undegraded":"2024-03-31T15:03:37.971241+0000","last_fullsized":"2024-03-31T15:03:37.971241+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:27:54.317789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"21'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-31T15:03:37.971523+0000","last_change":"2024-03-31T15:03:35.959254+0000","last_active":"2024-03-31T15:03:37.971523+0000","last_peered":"2024-03-31T15:03:37.971523+0000","last_clean":"2024-03-31T15:03:37.971523+0000","last_became_active":"2024-03-31T15:03:35.958903+0000","last_became_peered":"2024-03-31T15:03:35.958903+0000","last_unstale":"2024-03-31T15:03:37.971523+0000","last_undegraded":"2024-03-31T15:03:37.971523+0000","last_fullsized":"2024-03-31T15:03:37.971523+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_clean_scrub_stamp":"2024-03-31T15:03:34.926145+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T15:07:57.066606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-31T15:03:37.963339+0000","last_change":"2024-03-31T15:03:36.973216+0000","last_active":"2024-03-31T15:03:37.963339+0000","last_peered":"2024-03-31T15:03:37.963339+0000","last_clean":"2024-03-31T15:03:37.963339+0000","last_became_active":"2024-03-31T15:03:36.972719+0000","last_became_peered":"2024-03-31T15:03:36.972719+0000","last_unstale":"2024-03-31T15:03:37.963339+0000","last_undegraded":"2024-03-31T15:03:37.963339+0000","last_fullsized":"2024-03-31T15:03:37.963339+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:12:46.855478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-31T15:03:37.967788+0000","last_change":"2024-03-31T15:03:35.964289+0000","last_active":"2024-03-31T15:03:37.967788+0000","last_peered":"2024-03-31T15:03:37.967788+0000","last_clean":"2024-03-31T15:03:37.967788+0000","last_became_active":"2024-03-31T15:03:35.963897+0000","last_became_peered":"2024-03-31T15:03:35.963897+0000","last_unstale":"2024-03-31T15:03:37.967788+0000","last_undegraded":"2024-03-31T15:03:37.967788+0000","last_fullsized":"2024-03-31T15:03:37.967788+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_clean_scrub_stamp":"2024-03-31T15:03:34.926145+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T15:07:57.066606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-31T15:03:39.381063+0000","last_change":"2024-03-31T15:03:36.967603+0000","last_active":"2024-03-31T15:03:39.381063+0000","last_peered":"2024-03-31T15:03:39.381063+0000","last_clean":"2024-03-31T15:03:39.381063+0000","last_became_active":"2024-03-31T15:03:36.967206+0000","last_became_peered":"2024-03-31T15:03:36.967206+0000","last_unstale":"2024-03-31T15:03:39.381063+0000","last_undegraded":"2024-03-31T15:03:39.381063+0000","last_fullsized":"2024-03-31T15:03:39.381063+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:22:33.812426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-31T15:03:38.898165+0000","last_change":"2024-03-31T15:03:35.967340+0000","last_active":"2024-03-31T15:03:38.898165+0000","last_peered":"2024-03-31T15:03:38.898165+0000","last_clean":"2024-03-31T15:03:38.898165+0000","last_became_active":"2024-03-31T15:03:35.967046+0000","last_became_peered":"2024-03-31T15:03:35.967046+0000","last_unstale":"2024-03-31T15:03:38.898165+0000","last_undegraded":"2024-03-31T15:03:38.898165+0000","last_fullsized":"2024-03-31T15:03:38.898165+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_clean_scrub_stamp":"2024-03-31T15:03:34.926145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:11:41.679652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-31T15:03:37.965991+0000","last_change":"2024-03-31T15:03:36.991196+0000","last_active":"2024-03-31T15:03:37.965991+0000","last_peered":"2024-03-31T15:03:37.965991+0000","last_clean":"2024-03-31T15:03:37.965991+0000","last_became_active":"2024-03-31T15:03:36.990914+0000","last_became_peered":"2024-03-31T15:03:36.990914+0000","last_unstale":"2024-03-31T15:03:37.965991+0000","last_undegraded":"2024-03-31T15:03:37.965991+0000","last_fullsized":"2024-03-31T15:03:37.965991+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:11:42.685111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-31T15:03:37.971080+0000","last_change":"2024-03-31T15:03:35.969671+0000","last_active":"2024-03-31T15:03:37.971080+0000","last_peered":"2024-03-31T15:03:37.971080+0000","last_clean":"2024-03-31T15:03:37.971080+0000","last_became_active":"2024-03-31T15:03:35.969412+0000","last_became_peered":"2024-03-31T15:03:35.969412+0000","last_unstale":"2024-03-31T15:03:37.971080+0000","last_undegraded":"2024-03-31T15:03:37.971080+0000","last_fullsized":"2024-03-31T15:03:37.971080+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_clean_scrub_stamp":"2024-03-31T15:03:34.926145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:07:06.210288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-31T15:03:37.962862+0000","last_change":"2024-03-31T15:03:36.959849+0000","last_active":"2024-03-31T15:03:37.962862+0000","last_peered":"2024-03-31T15:03:37.962862+0000","last_clean":"2024-03-31T15:03:37.962862+0000","last_became_active":"2024-03-31T15:03:36.959611+0000","last_became_peered":"2024-03-31T15:03:36.959611+0000","last_unstale":"2024-03-31T15:03:37.962862+0000","last_undegraded":"2024-03-31T15:03:37.962862+0000","last_fullsized":"2024-03-31T15:03:37.962862+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:22:33.812426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-31T15:03:37.971127+0000","last_change":"2024-03-31T15:03:35.963705+0000","last_active":"2024-03-31T15:03:37.971127+0000","last_peered":"2024-03-31T15:03:37.971127+0000","last_clean":"2024-03-31T15:03:37.971127+0000","last_became_active":"2024-03-31T15:03:35.963241+0000","last_became_peered":"2024-03-31T15:03:35.963241+0000","last_unstale":"2024-03-31T15:03:37.971127+0000","last_undegraded":"2024-03-31T15:03:37.971127+0000","last_fullsized":"2024-03-31T15:03:37.971127+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_clean_scrub_stamp":"2024-03-31T15:03:34.926145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T16:00:45.563865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-31T15:03:37.963082+0000","last_change":"2024-03-31T15:03:36.970959+0000","last_active":"2024-03-31T15:03:37.963082+0000","last_peered":"2024-03-31T15:03:37.963082+0000","last_clean":"2024-03-31T15:03:37.963082+0000","last_became_active":"2024-03-31T15:03:36.970431+0000","last_became_peered":"2024-03-31T15:03:36.970431+0000","last_unstale":"2024-03-31T15:03:37.963082+0000","last_undegraded":"2024-03-31T15:03:37.963082+0000","last_fullsized":"2024-03-31T15:03:37.963082+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:15:27.562105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-31T15:03:38.898739+0000","last_change":"2024-03-31T15:03:35.964624+0000","last_active":"2024-03-31T15:03:38.898739+0000","last_peered":"2024-03-31T15:03:38.898739+0000","last_clean":"2024-03-31T15:03:38.898739+0000","last_became_active":"2024-03-31T15:03:35.964307+0000","last_became_peered":"2024-03-31T15:03:35.964307+0000","last_unstale":"2024-03-31T15:03:38.898739+0000","last_undegraded":"2024-03-31T15:03:38.898739+0000","last_fullsized":"2024-03-31T15:03:38.898739+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_clean_scrub_stamp":"2024-03-31T15:03:34.926145+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T15:58:18.492645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-31T15:03:37.967441+0000","last_change":"2024-03-31T15:03:36.969447+0000","last_active":"2024-03-31T15:03:37.967441+0000","last_peered":"2024-03-31T15:03:37.967441+0000","last_clean":"2024-03-31T15:03:37.967441+0000","last_became_active":"2024-03-31T15:03:36.967372+0000","last_became_peered":"2024-03-31T15:03:36.967372+0000","last_unstale":"2024-03-31T15:03:37.967441+0000","last_undegraded":"2024-03-31T15:03:37.967441+0000","last_fullsized":"2024-03-31T15:03:37.967441+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T17:27:21.111881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-31T15:03:39.382283+0000","last_change":"2024-03-31T15:03:36.980538+0000","last_active":"2024-03-31T15:03:39.382283+0000","last_peered":"2024-03-31T15:03:39.382283+0000","last_clean":"2024-03-31T15:03:39.382283+0000","last_became_active":"2024-03-31T15:03:36.980255+0000","last_became_peered":"2024-03-31T15:03:36.980255+0000","last_unstale":"2024-03-31T15:03:39.382283+0000","last_undegraded":"2024-03-31T15:03:39.382283+0000","last_fullsized":"2024-03-31T15:03:39.382283+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:14:42.246629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-31T15:03:38.898402+0000","last_change":"2024-03-31T15:03:35.962539+0000","last_active":"2024-03-31T15:03:38.898402+0000","last_peered":"2024-03-31T15:03:38.898402+0000","last_clean":"2024-03-31T15:03:38.898402+0000","last_became_active":"2024-03-31T15:03:35.961927+0000","last_became_peered":"2024-03-31T15:03:35.961927+0000","last_unstale":"2024-03-31T15:03:38.898402+0000","last_undegraded":"2024-03-31T15:03:38.898402+0000","last_fullsized":"2024-03-31T15:03:38.898402+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_clean_scrub_stamp":"2024-03-31T15:03:34.926145+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:48:53.962920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-31T15:03:39.382100+0000","last_change":"2024-03-31T15:03:36.981810+0000","last_active":"2024-03-31T15:03:39.382100+0000","last_peered":"2024-03-31T15:03:39.382100+0000","last_clean":"2024-03-31T15:03:39.382100+0000","last_became_active":"2024-03-31T15:03:36.981486+0000","last_became_peered":"2024-03-31T15:03:36.981486+0000","last_unstale":"2024-03-31T15:03:39.382100+0000","last_undegraded":"2024-03-31T15:03:39.382100+0000","last_fullsized":"2024-03-31T15:03:39.382100+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:41:13.119207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-31T15:03:38.898669+0000","last_change":"2024-03-31T15:03:35.972149+0000","last_active":"2024-03-31T15:03:38.898669+0000","last_peered":"2024-03-31T15:03:38.898669+0000","last_clean":"2024-03-31T15:03:38.898669+0000","last_became_active":"2024-03-31T15:03:35.971844+0000","last_became_peered":"2024-03-31T15:03:35.971844+0000","last_unstale":"2024-03-31T15:03:38.898669+0000","last_undegraded":"2024-03-31T15:03:38.898669+0000","last_fullsized":"2024-03-31T15:03:38.898669+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_clean_scrub_stamp":"2024-03-31T15:03:34.926145+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:19:23.333770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-31T15:03:39.380930+0000","last_change":"2024-03-31T15:03:36.988093+0000","last_active":"2024-03-31T15:03:39.380930+0000","last_peered":"2024-03-31T15:03:39.380930+0000","last_clean":"2024-03-31T15:03:39.380930+0000","last_became_active":"2024-03-31T15:03:36.987794+0000","last_became_peered":"2024-03-31T15:03:36.987794+0000","last_unstale":"2024-03-31T15:03:39.380930+0000","last_undegraded":"2024-03-31T15:03:39.380930+0000","last_fullsized":"2024-03-31T15:03:39.380930+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T17:27:21.111881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-31T15:03:37.963349+0000","last_change":"2024-03-31T15:03:35.969099+0000","last_active":"2024-03-31T15:03:37.963349+0000","last_peered":"2024-03-31T15:03:37.963349+0000","last_clean":"2024-03-31T15:03:37.963349+0000","last_became_active":"2024-03-31T15:03:35.968350+0000","last_became_peered":"2024-03-31T15:03:35.968350+0000","last_unstale":"2024-03-31T15:03:37.963349+0000","last_undegraded":"2024-03-31T15:03:37.963349+0000","last_fullsized":"2024-03-31T15:03:37.963349+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_clean_scrub_stamp":"2024-03-31T15:03:34.926145+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T17:31:20.984335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-31T15:03:37.963327+0000","last_change":"2024-03-31T15:03:36.960526+0000","last_active":"2024-03-31T15:03:37.963327+0000","last_peered":"2024-03-31T15:03:37.963327+0000","last_clean":"2024-03-31T15:03:37.963327+0000","last_became_active":"2024-03-31T15:03:36.960245+0000","last_became_peered":"2024-03-31T15:03:36.960245+0000","last_unstale":"2024-03-31T15:03:37.963327+0000","last_undegraded":"2024-03-31T15:03:37.963327+0000","last_fullsized":"2024-03-31T15:03:37.963327+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T15:57:32.539376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"20'4","reported_seq":18,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-31T15:03:39.381081+0000","last_change":"2024-03-31T15:03:35.965202+0000","last_active":"2024-03-31T15:03:39.381081+0000","last_peered":"2024-03-31T15:03:39.381081+0000","last_clean":"2024-03-31T15:03:39.381081+0000","last_became_active":"2024-03-31T15:03:35.964867+0000","last_became_peered":"2024-03-31T15:03:35.964867+0000","last_unstale":"2024-03-31T15:03:39.381081+0000","last_undegraded":"2024-03-31T15:03:39.381081+0000","last_fullsized":"2024-03-31T15:03:39.381081+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_clean_scrub_stamp":"2024-03-31T15:03:34.926145+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T16:35:11.030977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509449,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":788,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":806912,"data_stored":663845,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509448,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":212992,"data_stored":72985,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":15,"seq":64424509447,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":221184,"data_stored":73021,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509446,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31388,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340452,"statfs":{"total":96636764160,"available":96604622848,"internally_reserved":0,"allocated":208896,"data_stored":72895,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542152,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27244,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344596,"statfs":{"total":96636764160,"available":96608866304,"internally_reserved":0,"allocated":221184,"data_stored":74507,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542152,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31404,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340436,"statfs":{"total":96636764160,"available":96604606464,"internally_reserved":0,"allocated":221184,"data_stored":74507,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542152,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31400,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340440,"statfs":{"total":96636764160,"available":96604610560,"internally_reserved":0,"allocated":217088,"data_stored":73055,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542152,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31976,"kb_used_data":796,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339864,"statfs":{"total":96636764160,"available":96604020736,"internally_reserved":0,"allocated":815104,"data_stored":663901,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-31T15:03:45.761 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-31T15:03:45.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.949+0000 7f7e7c200640 1 Processor -- start 2024-03-31T15:03:45.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.949+0000 7f7e7c200640 1 -- start start 2024-03-31T15:03:45.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.953+0000 7f7e7c200640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f7e74145d10 0x7f7e740cf790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:45.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.953+0000 7f7e7c200640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f7e740b23d0 0x7f7e7415cd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:45.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.953+0000 7f7e7c200640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7e74160e70 con 0x7f7e740728b0 2024-03-31T15:03:45.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.953+0000 7f7e7c200640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f7e741602f0 con 0x7f7e74145d10 2024-03-31T15:03:45.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.953+0000 7f7e7c200640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f7e741605d0 con 0x7f7e740b23d0 2024-03-31T15:03:45.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.953+0000 7f7e78e00640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f7e740728b0 0x7f7e740b1cc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:34102/0 (socket says 172.21.15.16:34102) 2024-03-31T15:03:45.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.953+0000 7f7e78e00640 1 -- 172.21.15.16:0/3074032888 learned_addr learned my addr 172.21.15.16:0/3074032888 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:03:45.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.953+0000 7f7e79800640 1 --2- 172.21.15.16:0/3074032888 >> v2:172.21.15.89:3300/0 conn(0x7f7e74145d10 0x7f7e740cf790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:45.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.953+0000 7f7e7a200640 1 --2- 172.21.15.16:0/3074032888 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f7e740b23d0 0x7f7e7415cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:45.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.953+0000 7f7e73e00640 1 -- 172.21.15.16:0/3074032888 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3955081270 0 0) 0x7f7e74160e70 con 0x7f7e740728b0 2024-03-31T15:03:45.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.953+0000 7f7e73e00640 1 -- 172.21.15.16:0/3074032888 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7e5c003530 con 0x7f7e740728b0 2024-03-31T15:03:45.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.953+0000 7f7e79800640 1 -- 172.21.15.16:0/3074032888 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f7e740b23d0 msgr2=0x7f7e7415cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:03:45.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.953+0000 7f7e79800640 1 --2- 172.21.15.16:0/3074032888 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f7e740b23d0 0x7f7e7415cd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:45.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.953+0000 7f7e79800640 1 -- 172.21.15.16:0/3074032888 >> v1:172.21.15.16:6789/0 conn(0x7f7e740728b0 legacy=0x7f7e740b1cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:45.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.953+0000 7f7e79800640 1 -- 172.21.15.16:0/3074032888 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e740cfcd0 con 0x7f7e74145d10 2024-03-31T15:03:45.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.953+0000 7f7e7a200640 1 --2- 172.21.15.16:0/3074032888 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f7e740b23d0 0x7f7e7415cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:03:45.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.957+0000 7f7e79800640 1 --2- 172.21.15.16:0/3074032888 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7e74145d10 0x7f7e740cf790 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f7e680097c0 tx=0x7f7e6801c4a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=307768b49d1956b0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:03:45.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.957+0000 7f7e73e00640 1 -- 172.21.15.16:0/3074032888 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e680b0070 con 0x7f7e74145d10 2024-03-31T15:03:45.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.957+0000 7f7e73e00640 1 -- 172.21.15.16:0/3074032888 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7e680ab600 con 0x7f7e74145d10 2024-03-31T15:03:45.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.957+0000 7f7e73e00640 1 -- 172.21.15.16:0/3074032888 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e680ab9d0 con 0x7f7e74145d10 2024-03-31T15:03:45.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.957+0000 7f7e7c200640 1 -- 172.21.15.16:0/3074032888 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7e74145d10 msgr2=0x7f7e740cf790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:45.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.957+0000 7f7e7c200640 1 --2- 172.21.15.16:0/3074032888 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7e74145d10 0x7f7e740cf790 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f7e680097c0 tx=0x7f7e6801c4a0 comp rx=0 tx=0).stop 2024-03-31T15:03:45.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.957+0000 7f7e7c200640 1 -- 172.21.15.16:0/3074032888 shutdown_connections 2024-03-31T15:03:45.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.957+0000 7f7e7c200640 1 --2- 172.21.15.16:0/3074032888 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f7e740b23d0 0x7f7e7415cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:45.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.957+0000 7f7e7c200640 1 --2- 172.21.15.16:0/3074032888 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7e74145d10 0x7f7e740cf790 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:45.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.957+0000 7f7e7c200640 1 -- 172.21.15.16:0/3074032888 >> 172.21.15.16:0/3074032888 conn(0x7f7e7406b620 msgr2=0x7f7e7406ba30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:03:45.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.957+0000 7f7e7c200640 1 -- 172.21.15.16:0/3074032888 shutdown_connections 2024-03-31T15:03:45.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.957+0000 7f7e7c200640 1 -- 172.21.15.16:0/3074032888 wait complete. 2024-03-31T15:03:45.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.957+0000 7f7e7c200640 1 Processor -- start 2024-03-31T15:03:45.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.957+0000 7f7e7c200640 1 -- start start 2024-03-31T15:03:45.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.957+0000 7f7e7c200640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f7e740728b0 0x7f7e741451c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:45.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.957+0000 7f7e7c200640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7e740b23d0 0x7f7e74165ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:45.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.961+0000 7f7e7c200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7e74145d10 0x7f7e741634f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:45.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.961+0000 7f7e7c200640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f7e740cf4d0 con 0x7f7e74145d10 2024-03-31T15:03:45.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.961+0000 7f7e7c200640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f7e740cf350 con 0x7f7e740b23d0 2024-03-31T15:03:45.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.961+0000 7f7e7c200640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f7e74072cc0 con 0x7f7e740728b0 2024-03-31T15:03:45.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.961+0000 7f7e7a200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7e74145d10 0x7f7e741634f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:45.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.961+0000 7f7e7a200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7e74145d10 0x7f7e741634f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39252/0 (socket says 172.21.15.16:39252) 2024-03-31T15:03:45.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.961+0000 7f7e7a200640 1 -- 172.21.15.16:0/1555941097 learned_addr learned my addr 172.21.15.16:0/1555941097 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:03:45.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.961+0000 7f7e79800640 1 --2- 172.21.15.16:0/1555941097 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f7e740728b0 0x7f7e741451c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:45.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.961+0000 7f7e78e00640 1 --2- 172.21.15.16:0/1555941097 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7e740b23d0 0x7f7e74165ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:45.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.961+0000 7f7e7a200640 1 -- 172.21.15.16:0/1555941097 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f7e740728b0 msgr2=0x7f7e741451c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:45.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.961+0000 7f7e7a200640 1 --2- 172.21.15.16:0/1555941097 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f7e740728b0 0x7f7e741451c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:45.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.961+0000 7f7e7a200640 1 -- 172.21.15.16:0/1555941097 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7e740b23d0 msgr2=0x7f7e74165ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:45.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.961+0000 7f7e7a200640 1 --2- 172.21.15.16:0/1555941097 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7e740b23d0 0x7f7e74165ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:45.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.961+0000 7f7e7a200640 1 -- 172.21.15.16:0/1555941097 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e74161500 con 0x7f7e74145d10 2024-03-31T15:03:45.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.961+0000 7f7e7a200640 1 --2- 172.21.15.16:0/1555941097 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7e74145d10 0x7f7e741634f0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f7e6c009f00 tx=0x7f7e6c00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:03:45.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.961+0000 7f7e71600640 1 -- 172.21.15.16:0/1555941097 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e6c018030 con 0x7f7e74145d10 2024-03-31T15:03:45.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.961+0000 7f7e71600640 1 -- 172.21.15.16:0/1555941097 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7e6c00ce50 con 0x7f7e74145d10 2024-03-31T15:03:45.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.961+0000 7f7e7c200640 1 -- 172.21.15.16:0/1555941097 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7e74162be0 con 0x7f7e74145d10 2024-03-31T15:03:45.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.961+0000 7f7e71600640 1 -- 172.21.15.16:0/1555941097 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e6c003bc0 con 0x7f7e74145d10 2024-03-31T15:03:45.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.961+0000 7f7e7c200640 1 -- 172.21.15.16:0/1555941097 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7e74161a50 con 0x7f7e74145d10 2024-03-31T15:03:45.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.961+0000 7f7e7c200640 1 -- 172.21.15.16:0/1555941097 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7e74069950 con 0x7f7e74145d10 2024-03-31T15:03:45.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.961+0000 7f7e71600640 1 -- 172.21.15.16:0/1555941097 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f7e6c01b070 con 0x7f7e74145d10 2024-03-31T15:03:45.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.965+0000 7f7e71600640 1 --2- 172.21.15.16:0/1555941097 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f7e54068380 0x7f7e5406a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:45.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.965+0000 7f7e71600640 1 -- 172.21.15.16:0/1555941097 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f7e6c0872e0 con 0x7f7e74145d10 2024-03-31T15:03:45.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.965+0000 7f7e79800640 1 --2- 172.21.15.16:0/1555941097 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f7e54068380 0x7f7e5406a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:45.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.965+0000 7f7e79800640 1 --2- 172.21.15.16:0/1555941097 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f7e54068380 0x7f7e5406a840 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f7e680ad000 tx=0x7f7e68008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:03:45.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:45.969+0000 7f7e71600640 1 -- 172.21.15.16:0/1555941097 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f7e6c010040 con 0x7f7e74145d10 2024-03-31T15:03:46.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:46.129+0000 7f7e7c200640 1 -- 172.21.15.16:0/1555941097 --> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f7e740b19a0 con 0x7f7e54068380 2024-03-31T15:03:46.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:46.137+0000 7f7e71600640 1 -- 172.21.15.16:0/1555941097 <== mgr.4100 v2:172.21.15.89:6832/1549407273 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251836 (secure 0 0 0) 0x7f7e740b19a0 con 0x7f7e54068380 2024-03-31T15:03:46.132 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:03:46.134 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-03-31T15:03:46.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:46.141+0000 7f7e7c200640 1 -- 172.21.15.16:0/1555941097 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f7e54068380 msgr2=0x7f7e5406a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:46.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:46.141+0000 7f7e7c200640 1 --2- 172.21.15.16:0/1555941097 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f7e54068380 0x7f7e5406a840 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f7e680ad000 tx=0x7f7e68008040 comp rx=0 tx=0).stop 2024-03-31T15:03:46.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:46.141+0000 7f7e7c200640 1 -- 172.21.15.16:0/1555941097 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7e74145d10 msgr2=0x7f7e741634f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:46.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:46.141+0000 7f7e7c200640 1 --2- 172.21.15.16:0/1555941097 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7e74145d10 0x7f7e741634f0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f7e6c009f00 tx=0x7f7e6c00c490 comp rx=0 tx=0).stop 2024-03-31T15:03:46.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:46.141+0000 7f7e7c200640 1 -- 172.21.15.16:0/1555941097 shutdown_connections 2024-03-31T15:03:46.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:46.141+0000 7f7e7c200640 1 --2- 172.21.15.16:0/1555941097 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f7e54068380 0x7f7e5406a840 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:46.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:46.141+0000 7f7e7c200640 1 --2- 172.21.15.16:0/1555941097 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7e74145d10 0x7f7e741634f0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:46.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:46.141+0000 7f7e7c200640 1 --2- 172.21.15.16:0/1555941097 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7e740b23d0 0x7f7e74165ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:46.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:46.141+0000 7f7e7c200640 1 --2- 172.21.15.16:0/1555941097 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f7e740728b0 0x7f7e741451c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:46.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:46.141+0000 7f7e7c200640 1 -- 172.21.15.16:0/1555941097 >> 172.21.15.16:0/1555941097 conn(0x7f7e7406b620 msgr2=0x7f7e740b41e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:03:46.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:46.141+0000 7f7e7c200640 1 -- 172.21.15.16:0/1555941097 shutdown_connections 2024-03-31T15:03:46.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:46.141+0000 7f7e7c200640 1 -- 172.21.15.16:0/1555941097 wait complete. 2024-03-31T15:03:46.152 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":25,"stamp":"2024-03-31T15:03:44.531310+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":235692,"kb_used_data":2856,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754739028,"statfs":{"total":773094113280,"available":772852764672,"internally_reserved":0,"allocated":2924544,"data_stored":1768716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":238276144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001612"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-31T15:03:38.898721+0000","last_change":"2024-03-31T15:03:36.984154+0000","last_active":"2024-03-31T15:03:38.898721+0000","last_peered":"2024-03-31T15:03:38.898721+0000","last_clean":"2024-03-31T15:03:38.898721+0000","last_became_active":"2024-03-31T15:03:36.983712+0000","last_became_peered":"2024-03-31T15:03:36.983712+0000","last_unstale":"2024-03-31T15:03:38.898721+0000","last_undegraded":"2024-03-31T15:03:38.898721+0000","last_fullsized":"2024-03-31T15:03:38.898721+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:31:52.119827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-31T15:03:37.971509+0000","last_change":"2024-03-31T15:03:36.962443+0000","last_active":"2024-03-31T15:03:37.971509+0000","last_peered":"2024-03-31T15:03:37.971509+0000","last_clean":"2024-03-31T15:03:37.971509+0000","last_became_active":"2024-03-31T15:03:36.962140+0000","last_became_peered":"2024-03-31T15:03:36.962140+0000","last_unstale":"2024-03-31T15:03:37.971509+0000","last_undegraded":"2024-03-31T15:03:37.971509+0000","last_fullsized":"2024-03-31T15:03:37.971509+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:22:33.812426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-31T15:03:39.380530+0000","last_change":"2024-03-31T15:03:36.985502+0000","last_active":"2024-03-31T15:03:39.380530+0000","last_peered":"2024-03-31T15:03:39.380530+0000","last_clean":"2024-03-31T15:03:39.380530+0000","last_became_active":"2024-03-31T15:03:36.985263+0000","last_became_peered":"2024-03-31T15:03:36.985263+0000","last_unstale":"2024-03-31T15:03:39.380530+0000","last_undegraded":"2024-03-31T15:03:39.380530+0000","last_fullsized":"2024-03-31T15:03:39.380530+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:29:47.722244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-31T15:03:39.382049+0000","last_change":"2024-03-31T15:03:36.981333+0000","last_active":"2024-03-31T15:03:39.382049+0000","last_peered":"2024-03-31T15:03:39.382049+0000","last_clean":"2024-03-31T15:03:39.382049+0000","last_became_active":"2024-03-31T15:03:36.980937+0000","last_became_peered":"2024-03-31T15:03:36.980937+0000","last_unstale":"2024-03-31T15:03:39.382049+0000","last_undegraded":"2024-03-31T15:03:39.382049+0000","last_fullsized":"2024-03-31T15:03:39.382049+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:50:01.491908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-31T15:03:38.898625+0000","last_change":"2024-03-31T15:03:36.989117+0000","last_active":"2024-03-31T15:03:38.898625+0000","last_peered":"2024-03-31T15:03:38.898625+0000","last_clean":"2024-03-31T15:03:38.898625+0000","last_became_active":"2024-03-31T15:03:36.986002+0000","last_became_peered":"2024-03-31T15:03:36.986002+0000","last_unstale":"2024-03-31T15:03:38.898625+0000","last_undegraded":"2024-03-31T15:03:38.898625+0000","last_fullsized":"2024-03-31T15:03:38.898625+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:58:07.229805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-31T15:03:39.381819+0000","last_change":"2024-03-31T15:03:36.981121+0000","last_active":"2024-03-31T15:03:39.381819+0000","last_peered":"2024-03-31T15:03:39.381819+0000","last_clean":"2024-03-31T15:03:39.381819+0000","last_became_active":"2024-03-31T15:03:36.980669+0000","last_became_peered":"2024-03-31T15:03:36.980669+0000","last_unstale":"2024-03-31T15:03:39.381819+0000","last_undegraded":"2024-03-31T15:03:39.381819+0000","last_fullsized":"2024-03-31T15:03:39.381819+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T17:34:55.373127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-31T15:03:38.898283+0000","last_change":"2024-03-31T15:03:36.974140+0000","last_active":"2024-03-31T15:03:38.898283+0000","last_peered":"2024-03-31T15:03:38.898283+0000","last_clean":"2024-03-31T15:03:38.898283+0000","last_became_active":"2024-03-31T15:03:36.973867+0000","last_became_peered":"2024-03-31T15:03:36.973867+0000","last_unstale":"2024-03-31T15:03:38.898283+0000","last_undegraded":"2024-03-31T15:03:38.898283+0000","last_fullsized":"2024-03-31T15:03:38.898283+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:12:48.216307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-31T15:03:37.962657+0000","last_change":"2024-03-31T15:03:36.969620+0000","last_active":"2024-03-31T15:03:37.962657+0000","last_peered":"2024-03-31T15:03:37.962657+0000","last_clean":"2024-03-31T15:03:37.962657+0000","last_became_active":"2024-03-31T15:03:36.969317+0000","last_became_peered":"2024-03-31T15:03:36.969317+0000","last_unstale":"2024-03-31T15:03:37.962657+0000","last_undegraded":"2024-03-31T15:03:37.962657+0000","last_fullsized":"2024-03-31T15:03:37.962657+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T17:27:21.111881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-31T15:03:38.898858+0000","last_change":"2024-03-31T15:03:36.990208+0000","last_active":"2024-03-31T15:03:38.898858+0000","last_peered":"2024-03-31T15:03:38.898858+0000","last_clean":"2024-03-31T15:03:38.898858+0000","last_became_active":"2024-03-31T15:03:36.989706+0000","last_became_peered":"2024-03-31T15:03:36.989706+0000","last_unstale":"2024-03-31T15:03:38.898858+0000","last_undegraded":"2024-03-31T15:03:38.898858+0000","last_fullsized":"2024-03-31T15:03:38.898858+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:07:26.921241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-31T15:03:37.967891+0000","last_change":"2024-03-31T15:03:36.982132+0000","last_active":"2024-03-31T15:03:37.967891+0000","last_peered":"2024-03-31T15:03:37.967891+0000","last_clean":"2024-03-31T15:03:37.967891+0000","last_became_active":"2024-03-31T15:03:36.981697+0000","last_became_peered":"2024-03-31T15:03:36.981697+0000","last_unstale":"2024-03-31T15:03:37.967891+0000","last_undegraded":"2024-03-31T15:03:37.967891+0000","last_fullsized":"2024-03-31T15:03:37.967891+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:22:33.812426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-31T15:03:37.967567+0000","last_change":"2024-03-31T15:03:36.980849+0000","last_active":"2024-03-31T15:03:37.967567+0000","last_peered":"2024-03-31T15:03:37.967567+0000","last_clean":"2024-03-31T15:03:37.967567+0000","last_became_active":"2024-03-31T15:03:36.980535+0000","last_became_peered":"2024-03-31T15:03:36.980535+0000","last_unstale":"2024-03-31T15:03:37.967567+0000","last_undegraded":"2024-03-31T15:03:37.967567+0000","last_fullsized":"2024-03-31T15:03:37.967567+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:27:54.317789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-31T15:03:37.966263+0000","last_change":"2024-03-31T15:03:36.962572+0000","last_active":"2024-03-31T15:03:37.966263+0000","last_peered":"2024-03-31T15:03:37.966263+0000","last_clean":"2024-03-31T15:03:37.966263+0000","last_became_active":"2024-03-31T15:03:36.960772+0000","last_became_peered":"2024-03-31T15:03:36.960772+0000","last_unstale":"2024-03-31T15:03:37.966263+0000","last_undegraded":"2024-03-31T15:03:37.966263+0000","last_fullsized":"2024-03-31T15:03:37.966263+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T15:58:19.498104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-31T15:03:37.963021+0000","last_change":"2024-03-31T15:03:36.960134+0000","last_active":"2024-03-31T15:03:37.963021+0000","last_peered":"2024-03-31T15:03:37.963021+0000","last_clean":"2024-03-31T15:03:37.963021+0000","last_became_active":"2024-03-31T15:03:36.959747+0000","last_became_peered":"2024-03-31T15:03:36.959747+0000","last_unstale":"2024-03-31T15:03:37.963021+0000","last_undegraded":"2024-03-31T15:03:37.963021+0000","last_fullsized":"2024-03-31T15:03:37.963021+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T17:50:16.612568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-31T15:03:39.380710+0000","last_change":"2024-03-31T15:03:36.964930+0000","last_active":"2024-03-31T15:03:39.380710+0000","last_peered":"2024-03-31T15:03:39.380710+0000","last_clean":"2024-03-31T15:03:39.380710+0000","last_became_active":"2024-03-31T15:03:36.964006+0000","last_became_peered":"2024-03-31T15:03:36.964006+0000","last_unstale":"2024-03-31T15:03:39.380710+0000","last_undegraded":"2024-03-31T15:03:39.380710+0000","last_fullsized":"2024-03-31T15:03:39.380710+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:37:26.685300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-31T15:03:37.971000+0000","last_change":"2024-03-31T15:03:36.981622+0000","last_active":"2024-03-31T15:03:37.971000+0000","last_peered":"2024-03-31T15:03:37.971000+0000","last_clean":"2024-03-31T15:03:37.971000+0000","last_became_active":"2024-03-31T15:03:36.981267+0000","last_became_peered":"2024-03-31T15:03:36.981267+0000","last_unstale":"2024-03-31T15:03:37.971000+0000","last_undegraded":"2024-03-31T15:03:37.971000+0000","last_fullsized":"2024-03-31T15:03:37.971000+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T17:27:21.111881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-31T15:03:39.381753+0000","last_change":"2024-03-31T15:03:36.977834+0000","last_active":"2024-03-31T15:03:39.381753+0000","last_peered":"2024-03-31T15:03:39.381753+0000","last_clean":"2024-03-31T15:03:39.381753+0000","last_became_active":"2024-03-31T15:03:36.977469+0000","last_became_peered":"2024-03-31T15:03:36.977469+0000","last_unstale":"2024-03-31T15:03:39.381753+0000","last_undegraded":"2024-03-31T15:03:39.381753+0000","last_fullsized":"2024-03-31T15:03:39.381753+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:05:30.818052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-31T15:03:39.382482+0000","last_change":"2024-03-31T15:03:36.981952+0000","last_active":"2024-03-31T15:03:39.382482+0000","last_peered":"2024-03-31T15:03:39.382482+0000","last_clean":"2024-03-31T15:03:39.382482+0000","last_became_active":"2024-03-31T15:03:36.981615+0000","last_became_peered":"2024-03-31T15:03:36.981615+0000","last_unstale":"2024-03-31T15:03:39.382482+0000","last_undegraded":"2024-03-31T15:03:39.382482+0000","last_fullsized":"2024-03-31T15:03:39.382482+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:31:52.119827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-31T15:03:39.381168+0000","last_change":"2024-03-31T15:03:36.981172+0000","last_active":"2024-03-31T15:03:39.381168+0000","last_peered":"2024-03-31T15:03:39.381168+0000","last_clean":"2024-03-31T15:03:39.381168+0000","last_became_active":"2024-03-31T15:03:36.980303+0000","last_became_peered":"2024-03-31T15:03:36.980303+0000","last_unstale":"2024-03-31T15:03:39.381168+0000","last_undegraded":"2024-03-31T15:03:39.381168+0000","last_fullsized":"2024-03-31T15:03:39.381168+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:19:24.339229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-31T15:03:38.898702+0000","last_change":"2024-03-31T15:03:36.991712+0000","last_active":"2024-03-31T15:03:38.898702+0000","last_peered":"2024-03-31T15:03:38.898702+0000","last_clean":"2024-03-31T15:03:38.898702+0000","last_became_active":"2024-03-31T15:03:36.988177+0000","last_became_peered":"2024-03-31T15:03:36.988177+0000","last_unstale":"2024-03-31T15:03:38.898702+0000","last_undegraded":"2024-03-31T15:03:38.898702+0000","last_fullsized":"2024-03-31T15:03:38.898702+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:25:31.055690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-31T15:03:37.967399+0000","last_change":"2024-03-31T15:03:36.968423+0000","last_active":"2024-03-31T15:03:37.967399+0000","last_peered":"2024-03-31T15:03:37.967399+0000","last_clean":"2024-03-31T15:03:37.967399+0000","last_became_active":"2024-03-31T15:03:36.968014+0000","last_became_peered":"2024-03-31T15:03:36.968014+0000","last_unstale":"2024-03-31T15:03:37.967399+0000","last_undegraded":"2024-03-31T15:03:37.967399+0000","last_fullsized":"2024-03-31T15:03:37.967399+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T17:15:21.722119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-31T15:03:39.380826+0000","last_change":"2024-03-31T15:03:36.975734+0000","last_active":"2024-03-31T15:03:39.380826+0000","last_peered":"2024-03-31T15:03:39.380826+0000","last_clean":"2024-03-31T15:03:39.380826+0000","last_became_active":"2024-03-31T15:03:36.970929+0000","last_became_peered":"2024-03-31T15:03:36.970929+0000","last_unstale":"2024-03-31T15:03:39.380826+0000","last_undegraded":"2024-03-31T15:03:39.380826+0000","last_fullsized":"2024-03-31T15:03:39.380826+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T19:12:45.358470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-31T15:03:39.381878+0000","last_change":"2024-03-31T15:03:36.982234+0000","last_active":"2024-03-31T15:03:39.381878+0000","last_peered":"2024-03-31T15:03:39.381878+0000","last_clean":"2024-03-31T15:03:39.381878+0000","last_became_active":"2024-03-31T15:03:36.981359+0000","last_became_peered":"2024-03-31T15:03:36.981359+0000","last_unstale":"2024-03-31T15:03:39.381878+0000","last_undegraded":"2024-03-31T15:03:39.381878+0000","last_fullsized":"2024-03-31T15:03:39.381878+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:12:48.216307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-31T15:03:37.962738+0000","last_change":"2024-03-31T15:03:36.965603+0000","last_active":"2024-03-31T15:03:37.962738+0000","last_peered":"2024-03-31T15:03:37.962738+0000","last_clean":"2024-03-31T15:03:37.962738+0000","last_became_active":"2024-03-31T15:03:36.965339+0000","last_became_peered":"2024-03-31T15:03:36.965339+0000","last_unstale":"2024-03-31T15:03:37.962738+0000","last_undegraded":"2024-03-31T15:03:37.962738+0000","last_fullsized":"2024-03-31T15:03:37.962738+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:37:46.279789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-31T15:03:39.381792+0000","last_change":"2024-03-31T15:03:36.977130+0000","last_active":"2024-03-31T15:03:39.381792+0000","last_peered":"2024-03-31T15:03:39.381792+0000","last_clean":"2024-03-31T15:03:39.381792+0000","last_became_active":"2024-03-31T15:03:36.973022+0000","last_became_peered":"2024-03-31T15:03:36.973022+0000","last_unstale":"2024-03-31T15:03:39.381792+0000","last_undegraded":"2024-03-31T15:03:39.381792+0000","last_fullsized":"2024-03-31T15:03:39.381792+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:34:49.982234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-31T15:03:37.963323+0000","last_change":"2024-03-31T15:03:36.967074+0000","last_active":"2024-03-31T15:03:37.963323+0000","last_peered":"2024-03-31T15:03:37.963323+0000","last_clean":"2024-03-31T15:03:37.963323+0000","last_became_active":"2024-03-31T15:03:36.966802+0000","last_became_peered":"2024-03-31T15:03:36.966802+0000","last_unstale":"2024-03-31T15:03:37.963323+0000","last_undegraded":"2024-03-31T15:03:37.963323+0000","last_fullsized":"2024-03-31T15:03:37.963323+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:49:08.771388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-31T15:03:37.967818+0000","last_change":"2024-03-31T15:03:36.977099+0000","last_active":"2024-03-31T15:03:37.967818+0000","last_peered":"2024-03-31T15:03:37.967818+0000","last_clean":"2024-03-31T15:03:37.967818+0000","last_became_active":"2024-03-31T15:03:36.976814+0000","last_became_peered":"2024-03-31T15:03:36.976814+0000","last_unstale":"2024-03-31T15:03:37.967818+0000","last_undegraded":"2024-03-31T15:03:37.967818+0000","last_fullsized":"2024-03-31T15:03:37.967818+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:37:46.279789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-31T15:03:37.971261+0000","last_change":"2024-03-31T15:03:36.968708+0000","last_active":"2024-03-31T15:03:37.971261+0000","last_peered":"2024-03-31T15:03:37.971261+0000","last_clean":"2024-03-31T15:03:37.971261+0000","last_became_active":"2024-03-31T15:03:36.968292+0000","last_became_peered":"2024-03-31T15:03:36.968292+0000","last_unstale":"2024-03-31T15:03:37.971261+0000","last_undegraded":"2024-03-31T15:03:37.971261+0000","last_fullsized":"2024-03-31T15:03:37.971261+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:37:46.279789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-31T15:03:37.963169+0000","last_change":"2024-03-31T15:03:36.970468+0000","last_active":"2024-03-31T15:03:37.963169+0000","last_peered":"2024-03-31T15:03:37.963169+0000","last_clean":"2024-03-31T15:03:37.963169+0000","last_became_active":"2024-03-31T15:03:36.970233+0000","last_became_peered":"2024-03-31T15:03:36.970233+0000","last_unstale":"2024-03-31T15:03:37.963169+0000","last_undegraded":"2024-03-31T15:03:37.963169+0000","last_fullsized":"2024-03-31T15:03:37.963169+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:16:25.725098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-31T15:03:39.380875+0000","last_change":"2024-03-31T15:03:36.962951+0000","last_active":"2024-03-31T15:03:39.380875+0000","last_peered":"2024-03-31T15:03:39.380875+0000","last_clean":"2024-03-31T15:03:39.380875+0000","last_became_active":"2024-03-31T15:03:36.961876+0000","last_became_peered":"2024-03-31T15:03:36.961876+0000","last_unstale":"2024-03-31T15:03:39.380875+0000","last_undegraded":"2024-03-31T15:03:39.380875+0000","last_fullsized":"2024-03-31T15:03:39.380875+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T17:15:21.722119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-31T15:03:39.381964+0000","last_change":"2024-03-31T15:03:36.979570+0000","last_active":"2024-03-31T15:03:39.381964+0000","last_peered":"2024-03-31T15:03:39.381964+0000","last_clean":"2024-03-31T15:03:39.381964+0000","last_became_active":"2024-03-31T15:03:36.979287+0000","last_became_peered":"2024-03-31T15:03:36.979287+0000","last_unstale":"2024-03-31T15:03:39.381964+0000","last_undegraded":"2024-03-31T15:03:39.381964+0000","last_fullsized":"2024-03-31T15:03:39.381964+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:07:48.318096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-31T15:03:39.380389+0000","last_change":"2024-03-31T15:03:36.988981+0000","last_active":"2024-03-31T15:03:39.380389+0000","last_peered":"2024-03-31T15:03:39.380389+0000","last_clean":"2024-03-31T15:03:39.380389+0000","last_became_active":"2024-03-31T15:03:36.988582+0000","last_became_peered":"2024-03-31T15:03:36.988582+0000","last_unstale":"2024-03-31T15:03:39.380389+0000","last_undegraded":"2024-03-31T15:03:39.380389+0000","last_fullsized":"2024-03-31T15:03:39.380389+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:48:54.968379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-31T15:03:37.967288+0000","last_change":"2024-03-31T15:03:36.975267+0000","last_active":"2024-03-31T15:03:37.967288+0000","last_peered":"2024-03-31T15:03:37.967288+0000","last_clean":"2024-03-31T15:03:37.967288+0000","last_became_active":"2024-03-31T15:03:36.972549+0000","last_became_peered":"2024-03-31T15:03:36.972549+0000","last_unstale":"2024-03-31T15:03:37.967288+0000","last_undegraded":"2024-03-31T15:03:37.967288+0000","last_fullsized":"2024-03-31T15:03:37.967288+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T19:12:45.358470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-31T15:03:37.966007+0000","last_change":"2024-03-31T15:03:36.964214+0000","last_active":"2024-03-31T15:03:37.966007+0000","last_peered":"2024-03-31T15:03:37.966007+0000","last_clean":"2024-03-31T15:03:37.966007+0000","last_became_active":"2024-03-31T15:03:36.963926+0000","last_became_peered":"2024-03-31T15:03:36.963926+0000","last_unstale":"2024-03-31T15:03:37.966007+0000","last_undegraded":"2024-03-31T15:03:37.966007+0000","last_fullsized":"2024-03-31T15:03:37.966007+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:26:54.177359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-31T15:03:37.963122+0000","last_change":"2024-03-31T15:03:35.966649+0000","last_active":"2024-03-31T15:03:37.963122+0000","last_peered":"2024-03-31T15:03:37.963122+0000","last_clean":"2024-03-31T15:03:37.963122+0000","last_became_active":"2024-03-31T15:03:35.964612+0000","last_became_peered":"2024-03-31T15:03:35.964612+0000","last_unstale":"2024-03-31T15:03:37.963122+0000","last_undegraded":"2024-03-31T15:03:37.963122+0000","last_fullsized":"2024-03-31T15:03:37.963122+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_clean_scrub_stamp":"2024-03-31T15:03:34.926145+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T15:38:42.097818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-31T15:03:39.380987+0000","last_change":"2024-03-31T15:03:36.961519+0000","last_active":"2024-03-31T15:03:39.380987+0000","last_peered":"2024-03-31T15:03:39.380987+0000","last_clean":"2024-03-31T15:03:39.380987+0000","last_became_active":"2024-03-31T15:03:36.961032+0000","last_became_peered":"2024-03-31T15:03:36.961032+0000","last_unstale":"2024-03-31T15:03:39.380987+0000","last_undegraded":"2024-03-31T15:03:39.380987+0000","last_fullsized":"2024-03-31T15:03:39.380987+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:27:54.317789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-31T15:03:37.967261+0000","last_change":"2024-03-31T15:03:35.961058+0000","last_active":"2024-03-31T15:03:37.967261+0000","last_peered":"2024-03-31T15:03:37.967261+0000","last_clean":"2024-03-31T15:03:37.967261+0000","last_became_active":"2024-03-31T15:03:35.960684+0000","last_became_peered":"2024-03-31T15:03:35.960684+0000","last_unstale":"2024-03-31T15:03:37.967261+0000","last_undegraded":"2024-03-31T15:03:37.967261+0000","last_fullsized":"2024-03-31T15:03:37.967261+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_clean_scrub_stamp":"2024-03-31T15:03:34.926145+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:07:06.210288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-31T15:03:37.962722+0000","last_change":"2024-03-31T15:03:35.963374+0000","last_active":"2024-03-31T15:03:37.962722+0000","last_peered":"2024-03-31T15:03:37.962722+0000","last_clean":"2024-03-31T15:03:37.962722+0000","last_became_active":"2024-03-31T15:03:35.963009+0000","last_became_peered":"2024-03-31T15:03:35.963009+0000","last_unstale":"2024-03-31T15:03:37.962722+0000","last_undegraded":"2024-03-31T15:03:37.962722+0000","last_fullsized":"2024-03-31T15:03:37.962722+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_clean_scrub_stamp":"2024-03-31T15:03:34.926145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:36:14.576891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-31T15:03:37.962735+0000","last_change":"2024-03-31T15:03:36.972478+0000","last_active":"2024-03-31T15:03:37.962735+0000","last_peered":"2024-03-31T15:03:37.962735+0000","last_clean":"2024-03-31T15:03:37.962735+0000","last_became_active":"2024-03-31T15:03:36.972297+0000","last_became_peered":"2024-03-31T15:03:36.972297+0000","last_unstale":"2024-03-31T15:03:37.962735+0000","last_undegraded":"2024-03-31T15:03:37.962735+0000","last_fullsized":"2024-03-31T15:03:37.962735+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:27:54.317789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-31T15:03:39.381858+0000","last_change":"2024-03-31T15:03:35.963562+0000","last_active":"2024-03-31T15:03:39.381858+0000","last_peered":"2024-03-31T15:03:39.381858+0000","last_clean":"2024-03-31T15:03:39.381858+0000","last_became_active":"2024-03-31T15:03:35.961838+0000","last_became_peered":"2024-03-31T15:03:35.961838+0000","last_unstale":"2024-03-31T15:03:39.381858+0000","last_undegraded":"2024-03-31T15:03:39.381858+0000","last_fullsized":"2024-03-31T15:03:39.381858+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_clean_scrub_stamp":"2024-03-31T15:03:34.926145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:27:19.823151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-31T15:03:37.970977+0000","last_change":"2024-03-31T15:03:36.970365+0000","last_active":"2024-03-31T15:03:37.970977+0000","last_peered":"2024-03-31T15:03:37.970977+0000","last_clean":"2024-03-31T15:03:37.970977+0000","last_became_active":"2024-03-31T15:03:36.970060+0000","last_became_peered":"2024-03-31T15:03:36.970060+0000","last_unstale":"2024-03-31T15:03:37.970977+0000","last_undegraded":"2024-03-31T15:03:37.970977+0000","last_fullsized":"2024-03-31T15:03:37.970977+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T17:15:21.722119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-31T15:03:37.967603+0000","last_change":"2024-03-31T15:03:35.956977+0000","last_active":"2024-03-31T15:03:37.967603+0000","last_peered":"2024-03-31T15:03:37.967603+0000","last_clean":"2024-03-31T15:03:37.967603+0000","last_became_active":"2024-03-31T15:03:35.954617+0000","last_became_peered":"2024-03-31T15:03:35.954617+0000","last_unstale":"2024-03-31T15:03:37.967603+0000","last_undegraded":"2024-03-31T15:03:37.967603+0000","last_fullsized":"2024-03-31T15:03:37.967603+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_clean_scrub_stamp":"2024-03-31T15:03:34.926145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T17:31:20.984335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-31T15:03:39.382328+0000","last_change":"2024-03-31T15:03:36.981437+0000","last_active":"2024-03-31T15:03:39.382328+0000","last_peered":"2024-03-31T15:03:39.382328+0000","last_clean":"2024-03-31T15:03:39.382328+0000","last_became_active":"2024-03-31T15:03:36.981050+0000","last_became_peered":"2024-03-31T15:03:36.981050+0000","last_unstale":"2024-03-31T15:03:39.382328+0000","last_undegraded":"2024-03-31T15:03:39.382328+0000","last_fullsized":"2024-03-31T15:03:39.382328+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T17:07:57.546757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-31T15:03:39.382317+0000","last_change":"2024-03-31T15:03:35.954630+0000","last_active":"2024-03-31T15:03:39.382317+0000","last_peered":"2024-03-31T15:03:39.382317+0000","last_clean":"2024-03-31T15:03:39.382317+0000","last_became_active":"2024-03-31T15:03:35.954149+0000","last_became_peered":"2024-03-31T15:03:35.954149+0000","last_unstale":"2024-03-31T15:03:39.382317+0000","last_undegraded":"2024-03-31T15:03:39.382317+0000","last_fullsized":"2024-03-31T15:03:39.382317+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_clean_scrub_stamp":"2024-03-31T15:03:34.926145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:23:12.843244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-31T15:03:39.381016+0000","last_change":"2024-03-31T15:03:36.988008+0000","last_active":"2024-03-31T15:03:39.381016+0000","last_peered":"2024-03-31T15:03:39.381016+0000","last_clean":"2024-03-31T15:03:39.381016+0000","last_became_active":"2024-03-31T15:03:36.987732+0000","last_became_peered":"2024-03-31T15:03:36.987732+0000","last_unstale":"2024-03-31T15:03:39.381016+0000","last_undegraded":"2024-03-31T15:03:39.381016+0000","last_fullsized":"2024-03-31T15:03:39.381016+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:37:46.279789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"20'3","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-31T15:03:37.971313+0000","last_change":"2024-03-31T15:03:35.959827+0000","last_active":"2024-03-31T15:03:37.971313+0000","last_peered":"2024-03-31T15:03:37.971313+0000","last_clean":"2024-03-31T15:03:37.971313+0000","last_became_active":"2024-03-31T15:03:35.959564+0000","last_became_peered":"2024-03-31T15:03:35.959564+0000","last_unstale":"2024-03-31T15:03:37.971313+0000","last_undegraded":"2024-03-31T15:03:37.971313+0000","last_fullsized":"2024-03-31T15:03:37.971313+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_clean_scrub_stamp":"2024-03-31T15:03:34.926145+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:36:14.576891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-31T15:03:37.966441+0000","last_change":"2024-03-31T15:03:36.982726+0000","last_active":"2024-03-31T15:03:37.966441+0000","last_peered":"2024-03-31T15:03:37.966441+0000","last_clean":"2024-03-31T15:03:37.966441+0000","last_became_active":"2024-03-31T15:03:36.981893+0000","last_became_peered":"2024-03-31T15:03:36.981893+0000","last_unstale":"2024-03-31T15:03:37.966441+0000","last_undegraded":"2024-03-31T15:03:37.966441+0000","last_fullsized":"2024-03-31T15:03:37.966441+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:19:24.339229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"20'6","reported_seq":20,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-31T15:03:37.967382+0000","last_change":"2024-03-31T15:03:35.970154+0000","last_active":"2024-03-31T15:03:37.967382+0000","last_peered":"2024-03-31T15:03:37.967382+0000","last_clean":"2024-03-31T15:03:37.967382+0000","last_became_active":"2024-03-31T15:03:35.969877+0000","last_became_peered":"2024-03-31T15:03:35.969877+0000","last_unstale":"2024-03-31T15:03:37.967382+0000","last_undegraded":"2024-03-31T15:03:37.967382+0000","last_fullsized":"2024-03-31T15:03:37.967382+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_clean_scrub_stamp":"2024-03-31T15:03:34.926145+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:36:14.576891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-31T15:03:38.898534+0000","last_change":"2024-03-31T15:03:36.980541+0000","last_active":"2024-03-31T15:03:38.898534+0000","last_peered":"2024-03-31T15:03:38.898534+0000","last_clean":"2024-03-31T15:03:38.898534+0000","last_became_active":"2024-03-31T15:03:36.980276+0000","last_became_peered":"2024-03-31T15:03:36.980276+0000","last_unstale":"2024-03-31T15:03:38.898534+0000","last_undegraded":"2024-03-31T15:03:38.898534+0000","last_fullsized":"2024-03-31T15:03:38.898534+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:41:13.119207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-31T15:03:39.380977+0000","last_change":"2024-03-31T15:03:35.960798+0000","last_active":"2024-03-31T15:03:39.380977+0000","last_peered":"2024-03-31T15:03:39.380977+0000","last_clean":"2024-03-31T15:03:39.380977+0000","last_became_active":"2024-03-31T15:03:35.960522+0000","last_became_peered":"2024-03-31T15:03:35.960522+0000","last_unstale":"2024-03-31T15:03:39.380977+0000","last_undegraded":"2024-03-31T15:03:39.380977+0000","last_fullsized":"2024-03-31T15:03:39.380977+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_clean_scrub_stamp":"2024-03-31T15:03:34.926145+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T16:07:33.984402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-31T15:03:39.380750+0000","last_change":"2024-03-31T15:03:36.972506+0000","last_active":"2024-03-31T15:03:39.380750+0000","last_peered":"2024-03-31T15:03:39.380750+0000","last_clean":"2024-03-31T15:03:39.380750+0000","last_became_active":"2024-03-31T15:03:36.972194+0000","last_became_peered":"2024-03-31T15:03:36.972194+0000","last_unstale":"2024-03-31T15:03:39.380750+0000","last_undegraded":"2024-03-31T15:03:39.380750+0000","last_fullsized":"2024-03-31T15:03:39.380750+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:48:13.691269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-31T15:03:39.381975+0000","last_change":"2024-03-31T15:03:35.966260+0000","last_active":"2024-03-31T15:03:39.381975+0000","last_peered":"2024-03-31T15:03:39.381975+0000","last_clean":"2024-03-31T15:03:39.381975+0000","last_became_active":"2024-03-31T15:03:35.965676+0000","last_became_peered":"2024-03-31T15:03:35.965676+0000","last_unstale":"2024-03-31T15:03:39.381975+0000","last_undegraded":"2024-03-31T15:03:39.381975+0000","last_fullsized":"2024-03-31T15:03:39.381975+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_clean_scrub_stamp":"2024-03-31T15:03:34.926145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:26:53.171901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-31T15:03:38.898638+0000","last_change":"2024-03-31T15:03:36.988567+0000","last_active":"2024-03-31T15:03:38.898638+0000","last_peered":"2024-03-31T15:03:38.898638+0000","last_clean":"2024-03-31T15:03:38.898638+0000","last_became_active":"2024-03-31T15:03:36.985668+0000","last_became_peered":"2024-03-31T15:03:36.985668+0000","last_unstale":"2024-03-31T15:03:38.898638+0000","last_undegraded":"2024-03-31T15:03:38.898638+0000","last_fullsized":"2024-03-31T15:03:38.898638+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:14:42.246629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-31T15:03:37.967494+0000","last_change":"2024-03-31T15:03:35.961562+0000","last_active":"2024-03-31T15:03:37.967494+0000","last_peered":"2024-03-31T15:03:37.967494+0000","last_clean":"2024-03-31T15:03:37.967494+0000","last_became_active":"2024-03-31T15:03:35.961280+0000","last_became_peered":"2024-03-31T15:03:35.961280+0000","last_unstale":"2024-03-31T15:03:37.967494+0000","last_undegraded":"2024-03-31T15:03:37.967494+0000","last_fullsized":"2024-03-31T15:03:37.967494+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_clean_scrub_stamp":"2024-03-31T15:03:34.926145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T15:38:42.097818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-31T15:03:37.967298+0000","last_change":"2024-03-31T15:03:36.979417+0000","last_active":"2024-03-31T15:03:37.967298+0000","last_peered":"2024-03-31T15:03:37.967298+0000","last_clean":"2024-03-31T15:03:37.967298+0000","last_became_active":"2024-03-31T15:03:36.973021+0000","last_became_peered":"2024-03-31T15:03:36.973021+0000","last_unstale":"2024-03-31T15:03:37.967298+0000","last_undegraded":"2024-03-31T15:03:37.967298+0000","last_fullsized":"2024-03-31T15:03:37.967298+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:48:13.691269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"20'8","reported_seq":22,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-31T15:03:39.380233+0000","last_change":"2024-03-31T15:03:35.955608+0000","last_active":"2024-03-31T15:03:39.380233+0000","last_peered":"2024-03-31T15:03:39.380233+0000","last_clean":"2024-03-31T15:03:39.380233+0000","last_became_active":"2024-03-31T15:03:35.955295+0000","last_became_peered":"2024-03-31T15:03:35.955295+0000","last_unstale":"2024-03-31T15:03:39.380233+0000","last_undegraded":"2024-03-31T15:03:39.380233+0000","last_fullsized":"2024-03-31T15:03:39.380233+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_clean_scrub_stamp":"2024-03-31T15:03:34.926145+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:23:12.843244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-31T15:03:38.898341+0000","last_change":"2024-03-31T15:03:36.990806+0000","last_active":"2024-03-31T15:03:38.898341+0000","last_peered":"2024-03-31T15:03:38.898341+0000","last_clean":"2024-03-31T15:03:38.898341+0000","last_became_active":"2024-03-31T15:03:36.990542+0000","last_became_peered":"2024-03-31T15:03:36.990542+0000","last_unstale":"2024-03-31T15:03:38.898341+0000","last_undegraded":"2024-03-31T15:03:38.898341+0000","last_fullsized":"2024-03-31T15:03:38.898341+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:07:48.318096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-31T15:03:37.966654+0000","last_change":"2024-03-31T15:03:35.968824+0000","last_active":"2024-03-31T15:03:37.966654+0000","last_peered":"2024-03-31T15:03:37.966654+0000","last_clean":"2024-03-31T15:03:37.966654+0000","last_became_active":"2024-03-31T15:03:35.968431+0000","last_became_peered":"2024-03-31T15:03:35.968431+0000","last_unstale":"2024-03-31T15:03:37.966654+0000","last_undegraded":"2024-03-31T15:03:37.966654+0000","last_fullsized":"2024-03-31T15:03:37.966654+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_clean_scrub_stamp":"2024-03-31T15:03:34.926145+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:30:58.752050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-31T15:03:38.898638+0000","last_change":"2024-03-31T15:03:36.993283+0000","last_active":"2024-03-31T15:03:38.898638+0000","last_peered":"2024-03-31T15:03:38.898638+0000","last_clean":"2024-03-31T15:03:38.898638+0000","last_became_active":"2024-03-31T15:03:36.992977+0000","last_became_peered":"2024-03-31T15:03:36.992977+0000","last_unstale":"2024-03-31T15:03:38.898638+0000","last_undegraded":"2024-03-31T15:03:38.898638+0000","last_fullsized":"2024-03-31T15:03:38.898638+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T17:07:57.546757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-31T15:03:39.381206+0000","last_change":"2024-03-31T15:03:35.969251+0000","last_active":"2024-03-31T15:03:39.381206+0000","last_peered":"2024-03-31T15:03:39.381206+0000","last_clean":"2024-03-31T15:03:39.381206+0000","last_became_active":"2024-03-31T15:03:35.968876+0000","last_became_peered":"2024-03-31T15:03:35.968876+0000","last_unstale":"2024-03-31T15:03:39.381206+0000","last_undegraded":"2024-03-31T15:03:39.381206+0000","last_fullsized":"2024-03-31T15:03:39.381206+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_clean_scrub_stamp":"2024-03-31T15:03:34.926145+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:27:19.823151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-31T15:03:38.898891+0000","last_change":"2024-03-31T15:03:36.986107+0000","last_active":"2024-03-31T15:03:38.898891+0000","last_peered":"2024-03-31T15:03:38.898891+0000","last_clean":"2024-03-31T15:03:38.898891+0000","last_became_active":"2024-03-31T15:03:36.985749+0000","last_became_peered":"2024-03-31T15:03:36.985749+0000","last_unstale":"2024-03-31T15:03:38.898891+0000","last_undegraded":"2024-03-31T15:03:38.898891+0000","last_fullsized":"2024-03-31T15:03:38.898891+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:22:28.970344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-31T15:03:37.971354+0000","last_change":"2024-03-31T15:03:35.965507+0000","last_active":"2024-03-31T15:03:37.971354+0000","last_peered":"2024-03-31T15:03:37.971354+0000","last_clean":"2024-03-31T15:03:37.971354+0000","last_became_active":"2024-03-31T15:03:35.964843+0000","last_became_peered":"2024-03-31T15:03:35.964843+0000","last_unstale":"2024-03-31T15:03:37.971354+0000","last_undegraded":"2024-03-31T15:03:37.971354+0000","last_fullsized":"2024-03-31T15:03:37.971354+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_clean_scrub_stamp":"2024-03-31T15:03:34.926145+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T17:31:20.984335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-31T15:03:39.382209+0000","last_change":"2024-03-31T15:03:36.979781+0000","last_active":"2024-03-31T15:03:39.382209+0000","last_peered":"2024-03-31T15:03:39.382209+0000","last_clean":"2024-03-31T15:03:39.382209+0000","last_became_active":"2024-03-31T15:03:36.979517+0000","last_became_peered":"2024-03-31T15:03:36.979517+0000","last_unstale":"2024-03-31T15:03:39.382209+0000","last_undegraded":"2024-03-31T15:03:39.382209+0000","last_fullsized":"2024-03-31T15:03:39.382209+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:58:07.229805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-31T15:03:39.380445+0000","last_change":"2024-03-31T15:03:35.960305+0000","last_active":"2024-03-31T15:03:39.380445+0000","last_peered":"2024-03-31T15:03:39.380445+0000","last_clean":"2024-03-31T15:03:39.380445+0000","last_became_active":"2024-03-31T15:03:35.960006+0000","last_became_peered":"2024-03-31T15:03:35.960006+0000","last_unstale":"2024-03-31T15:03:39.380445+0000","last_undegraded":"2024-03-31T15:03:39.380445+0000","last_fullsized":"2024-03-31T15:03:39.380445+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_clean_scrub_stamp":"2024-03-31T15:03:34.926145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:11:41.679652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-31T15:03:38.898523+0000","last_change":"2024-03-31T15:03:36.985221+0000","last_active":"2024-03-31T15:03:38.898523+0000","last_peered":"2024-03-31T15:03:38.898523+0000","last_clean":"2024-03-31T15:03:38.898523+0000","last_became_active":"2024-03-31T15:03:36.984963+0000","last_became_peered":"2024-03-31T15:03:36.984963+0000","last_unstale":"2024-03-31T15:03:38.898523+0000","last_undegraded":"2024-03-31T15:03:38.898523+0000","last_fullsized":"2024-03-31T15:03:38.898523+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:50:01.491908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":170,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-31T15:03:38.898259+0000","last_change":"2024-03-31T15:03:24.329890+0000","last_active":"2024-03-31T15:03:38.898259+0000","last_peered":"2024-03-31T15:03:38.898259+0000","last_clean":"2024-03-31T15:03:38.898259+0000","last_became_active":"2024-03-31T15:03:24.328882+0000","last_became_peered":"2024-03-31T15:03:24.328882+0000","last_unstale":"2024-03-31T15:03:38.898259+0000","last_undegraded":"2024-03-31T15:03:38.898259+0000","last_fullsized":"2024-03-31T15:03:38.898259+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:22.885535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:22.885535+0000","last_clean_scrub_stamp":"2024-03-31T15:03:22.885535+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:18:29.004296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-31T15:03:39.381810+0000","last_change":"2024-03-31T15:03:35.952977+0000","last_active":"2024-03-31T15:03:39.381810+0000","last_peered":"2024-03-31T15:03:39.381810+0000","last_clean":"2024-03-31T15:03:39.381810+0000","last_became_active":"2024-03-31T15:03:35.951288+0000","last_became_peered":"2024-03-31T15:03:35.951288+0000","last_unstale":"2024-03-31T15:03:39.381810+0000","last_undegraded":"2024-03-31T15:03:39.381810+0000","last_fullsized":"2024-03-31T15:03:39.381810+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_clean_scrub_stamp":"2024-03-31T15:03:34.926145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:11:41.679652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-31T15:03:37.962984+0000","last_change":"2024-03-31T15:03:36.980592+0000","last_active":"2024-03-31T15:03:37.962984+0000","last_peered":"2024-03-31T15:03:37.962984+0000","last_clean":"2024-03-31T15:03:37.962984+0000","last_became_active":"2024-03-31T15:03:36.980285+0000","last_became_peered":"2024-03-31T15:03:36.980285+0000","last_unstale":"2024-03-31T15:03:37.962984+0000","last_undegraded":"2024-03-31T15:03:37.962984+0000","last_fullsized":"2024-03-31T15:03:37.962984+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T17:01:42.202122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-31T15:03:38.898198+0000","last_change":"2024-03-31T15:03:35.968364+0000","last_active":"2024-03-31T15:03:38.898198+0000","last_peered":"2024-03-31T15:03:38.898198+0000","last_clean":"2024-03-31T15:03:38.898198+0000","last_became_active":"2024-03-31T15:03:35.968047+0000","last_became_peered":"2024-03-31T15:03:35.968047+0000","last_unstale":"2024-03-31T15:03:38.898198+0000","last_undegraded":"2024-03-31T15:03:38.898198+0000","last_fullsized":"2024-03-31T15:03:38.898198+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_clean_scrub_stamp":"2024-03-31T15:03:34.926145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:26:53.171901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-31T15:03:39.380580+0000","last_change":"2024-03-31T15:03:36.963584+0000","last_active":"2024-03-31T15:03:39.380580+0000","last_peered":"2024-03-31T15:03:39.380580+0000","last_clean":"2024-03-31T15:03:39.380580+0000","last_became_active":"2024-03-31T15:03:36.963216+0000","last_became_peered":"2024-03-31T15:03:36.963216+0000","last_unstale":"2024-03-31T15:03:39.380580+0000","last_undegraded":"2024-03-31T15:03:39.380580+0000","last_fullsized":"2024-03-31T15:03:39.380580+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:29:57.089580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-31T15:03:39.380342+0000","last_change":"2024-03-31T15:03:35.963196+0000","last_active":"2024-03-31T15:03:39.380342+0000","last_peered":"2024-03-31T15:03:39.380342+0000","last_clean":"2024-03-31T15:03:39.380342+0000","last_became_active":"2024-03-31T15:03:35.962746+0000","last_became_peered":"2024-03-31T15:03:35.962746+0000","last_unstale":"2024-03-31T15:03:39.380342+0000","last_undegraded":"2024-03-31T15:03:39.380342+0000","last_fullsized":"2024-03-31T15:03:39.380342+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_clean_scrub_stamp":"2024-03-31T15:03:34.926145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:00:32.840818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-31T15:03:37.970979+0000","last_change":"2024-03-31T15:03:36.972190+0000","last_active":"2024-03-31T15:03:37.970979+0000","last_peered":"2024-03-31T15:03:37.970979+0000","last_clean":"2024-03-31T15:03:37.970979+0000","last_became_active":"2024-03-31T15:03:36.971866+0000","last_became_peered":"2024-03-31T15:03:36.971866+0000","last_unstale":"2024-03-31T15:03:37.970979+0000","last_undegraded":"2024-03-31T15:03:37.970979+0000","last_fullsized":"2024-03-31T15:03:37.970979+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T19:12:45.358470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-31T15:03:39.382038+0000","last_change":"2024-03-31T15:03:35.953803+0000","last_active":"2024-03-31T15:03:39.382038+0000","last_peered":"2024-03-31T15:03:39.382038+0000","last_clean":"2024-03-31T15:03:39.382038+0000","last_became_active":"2024-03-31T15:03:35.953459+0000","last_became_peered":"2024-03-31T15:03:35.953459+0000","last_unstale":"2024-03-31T15:03:39.382038+0000","last_undegraded":"2024-03-31T15:03:39.382038+0000","last_fullsized":"2024-03-31T15:03:39.382038+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_clean_scrub_stamp":"2024-03-31T15:03:34.926145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T15:58:18.492645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-31T15:03:37.966372+0000","last_change":"2024-03-31T15:03:36.966599+0000","last_active":"2024-03-31T15:03:37.966372+0000","last_peered":"2024-03-31T15:03:37.966372+0000","last_clean":"2024-03-31T15:03:37.966372+0000","last_became_active":"2024-03-31T15:03:36.964867+0000","last_became_peered":"2024-03-31T15:03:36.964867+0000","last_unstale":"2024-03-31T15:03:37.966372+0000","last_undegraded":"2024-03-31T15:03:37.966372+0000","last_fullsized":"2024-03-31T15:03:37.966372+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:48:54.968379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-31T15:03:37.971175+0000","last_change":"2024-03-31T15:03:35.958174+0000","last_active":"2024-03-31T15:03:37.971175+0000","last_peered":"2024-03-31T15:03:37.971175+0000","last_clean":"2024-03-31T15:03:37.971175+0000","last_became_active":"2024-03-31T15:03:35.957870+0000","last_became_peered":"2024-03-31T15:03:35.957870+0000","last_unstale":"2024-03-31T15:03:37.971175+0000","last_undegraded":"2024-03-31T15:03:37.971175+0000","last_fullsized":"2024-03-31T15:03:37.971175+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_clean_scrub_stamp":"2024-03-31T15:03:34.926145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T15:38:42.097818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-31T15:03:39.382417+0000","last_change":"2024-03-31T15:03:36.980337+0000","last_active":"2024-03-31T15:03:39.382417+0000","last_peered":"2024-03-31T15:03:39.382417+0000","last_clean":"2024-03-31T15:03:39.382417+0000","last_became_active":"2024-03-31T15:03:36.980082+0000","last_became_peered":"2024-03-31T15:03:36.980082+0000","last_unstale":"2024-03-31T15:03:39.382417+0000","last_undegraded":"2024-03-31T15:03:39.382417+0000","last_fullsized":"2024-03-31T15:03:39.382417+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:25:31.055690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-31T15:03:38.898776+0000","last_change":"2024-03-31T15:03:35.969598+0000","last_active":"2024-03-31T15:03:38.898776+0000","last_peered":"2024-03-31T15:03:38.898776+0000","last_clean":"2024-03-31T15:03:38.898776+0000","last_became_active":"2024-03-31T15:03:35.969253+0000","last_became_peered":"2024-03-31T15:03:35.969253+0000","last_unstale":"2024-03-31T15:03:38.898776+0000","last_undegraded":"2024-03-31T15:03:38.898776+0000","last_fullsized":"2024-03-31T15:03:38.898776+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_clean_scrub_stamp":"2024-03-31T15:03:34.926145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:55:57.509587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-31T15:03:37.971241+0000","last_change":"2024-03-31T15:03:36.983933+0000","last_active":"2024-03-31T15:03:37.971241+0000","last_peered":"2024-03-31T15:03:37.971241+0000","last_clean":"2024-03-31T15:03:37.971241+0000","last_became_active":"2024-03-31T15:03:36.983650+0000","last_became_peered":"2024-03-31T15:03:36.983650+0000","last_unstale":"2024-03-31T15:03:37.971241+0000","last_undegraded":"2024-03-31T15:03:37.971241+0000","last_fullsized":"2024-03-31T15:03:37.971241+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:27:54.317789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"21'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-31T15:03:37.971523+0000","last_change":"2024-03-31T15:03:35.959254+0000","last_active":"2024-03-31T15:03:37.971523+0000","last_peered":"2024-03-31T15:03:37.971523+0000","last_clean":"2024-03-31T15:03:37.971523+0000","last_became_active":"2024-03-31T15:03:35.958903+0000","last_became_peered":"2024-03-31T15:03:35.958903+0000","last_unstale":"2024-03-31T15:03:37.971523+0000","last_undegraded":"2024-03-31T15:03:37.971523+0000","last_fullsized":"2024-03-31T15:03:37.971523+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_clean_scrub_stamp":"2024-03-31T15:03:34.926145+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T15:07:57.066606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-31T15:03:37.963339+0000","last_change":"2024-03-31T15:03:36.973216+0000","last_active":"2024-03-31T15:03:37.963339+0000","last_peered":"2024-03-31T15:03:37.963339+0000","last_clean":"2024-03-31T15:03:37.963339+0000","last_became_active":"2024-03-31T15:03:36.972719+0000","last_became_peered":"2024-03-31T15:03:36.972719+0000","last_unstale":"2024-03-31T15:03:37.963339+0000","last_undegraded":"2024-03-31T15:03:37.963339+0000","last_fullsized":"2024-03-31T15:03:37.963339+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:12:46.855478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-31T15:03:37.967788+0000","last_change":"2024-03-31T15:03:35.964289+0000","last_active":"2024-03-31T15:03:37.967788+0000","last_peered":"2024-03-31T15:03:37.967788+0000","last_clean":"2024-03-31T15:03:37.967788+0000","last_became_active":"2024-03-31T15:03:35.963897+0000","last_became_peered":"2024-03-31T15:03:35.963897+0000","last_unstale":"2024-03-31T15:03:37.967788+0000","last_undegraded":"2024-03-31T15:03:37.967788+0000","last_fullsized":"2024-03-31T15:03:37.967788+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_clean_scrub_stamp":"2024-03-31T15:03:34.926145+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T15:07:57.066606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-31T15:03:39.381063+0000","last_change":"2024-03-31T15:03:36.967603+0000","last_active":"2024-03-31T15:03:39.381063+0000","last_peered":"2024-03-31T15:03:39.381063+0000","last_clean":"2024-03-31T15:03:39.381063+0000","last_became_active":"2024-03-31T15:03:36.967206+0000","last_became_peered":"2024-03-31T15:03:36.967206+0000","last_unstale":"2024-03-31T15:03:39.381063+0000","last_undegraded":"2024-03-31T15:03:39.381063+0000","last_fullsized":"2024-03-31T15:03:39.381063+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:22:33.812426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-31T15:03:38.898165+0000","last_change":"2024-03-31T15:03:35.967340+0000","last_active":"2024-03-31T15:03:38.898165+0000","last_peered":"2024-03-31T15:03:38.898165+0000","last_clean":"2024-03-31T15:03:38.898165+0000","last_became_active":"2024-03-31T15:03:35.967046+0000","last_became_peered":"2024-03-31T15:03:35.967046+0000","last_unstale":"2024-03-31T15:03:38.898165+0000","last_undegraded":"2024-03-31T15:03:38.898165+0000","last_fullsized":"2024-03-31T15:03:38.898165+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_clean_scrub_stamp":"2024-03-31T15:03:34.926145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:11:41.679652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-31T15:03:37.965991+0000","last_change":"2024-03-31T15:03:36.991196+0000","last_active":"2024-03-31T15:03:37.965991+0000","last_peered":"2024-03-31T15:03:37.965991+0000","last_clean":"2024-03-31T15:03:37.965991+0000","last_became_active":"2024-03-31T15:03:36.990914+0000","last_became_peered":"2024-03-31T15:03:36.990914+0000","last_unstale":"2024-03-31T15:03:37.965991+0000","last_undegraded":"2024-03-31T15:03:37.965991+0000","last_fullsized":"2024-03-31T15:03:37.965991+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:11:42.685111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-31T15:03:37.971080+0000","last_change":"2024-03-31T15:03:35.969671+0000","last_active":"2024-03-31T15:03:37.971080+0000","last_peered":"2024-03-31T15:03:37.971080+0000","last_clean":"2024-03-31T15:03:37.971080+0000","last_became_active":"2024-03-31T15:03:35.969412+0000","last_became_peered":"2024-03-31T15:03:35.969412+0000","last_unstale":"2024-03-31T15:03:37.971080+0000","last_undegraded":"2024-03-31T15:03:37.971080+0000","last_fullsized":"2024-03-31T15:03:37.971080+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_clean_scrub_stamp":"2024-03-31T15:03:34.926145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:07:06.210288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-31T15:03:37.962862+0000","last_change":"2024-03-31T15:03:36.959849+0000","last_active":"2024-03-31T15:03:37.962862+0000","last_peered":"2024-03-31T15:03:37.962862+0000","last_clean":"2024-03-31T15:03:37.962862+0000","last_became_active":"2024-03-31T15:03:36.959611+0000","last_became_peered":"2024-03-31T15:03:36.959611+0000","last_unstale":"2024-03-31T15:03:37.962862+0000","last_undegraded":"2024-03-31T15:03:37.962862+0000","last_fullsized":"2024-03-31T15:03:37.962862+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:22:33.812426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-31T15:03:37.971127+0000","last_change":"2024-03-31T15:03:35.963705+0000","last_active":"2024-03-31T15:03:37.971127+0000","last_peered":"2024-03-31T15:03:37.971127+0000","last_clean":"2024-03-31T15:03:37.971127+0000","last_became_active":"2024-03-31T15:03:35.963241+0000","last_became_peered":"2024-03-31T15:03:35.963241+0000","last_unstale":"2024-03-31T15:03:37.971127+0000","last_undegraded":"2024-03-31T15:03:37.971127+0000","last_fullsized":"2024-03-31T15:03:37.971127+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_clean_scrub_stamp":"2024-03-31T15:03:34.926145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T16:00:45.563865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-31T15:03:37.963082+0000","last_change":"2024-03-31T15:03:36.970959+0000","last_active":"2024-03-31T15:03:37.963082+0000","last_peered":"2024-03-31T15:03:37.963082+0000","last_clean":"2024-03-31T15:03:37.963082+0000","last_became_active":"2024-03-31T15:03:36.970431+0000","last_became_peered":"2024-03-31T15:03:36.970431+0000","last_unstale":"2024-03-31T15:03:37.963082+0000","last_undegraded":"2024-03-31T15:03:37.963082+0000","last_fullsized":"2024-03-31T15:03:37.963082+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:15:27.562105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-31T15:03:38.898739+0000","last_change":"2024-03-31T15:03:35.964624+0000","last_active":"2024-03-31T15:03:38.898739+0000","last_peered":"2024-03-31T15:03:38.898739+0000","last_clean":"2024-03-31T15:03:38.898739+0000","last_became_active":"2024-03-31T15:03:35.964307+0000","last_became_peered":"2024-03-31T15:03:35.964307+0000","last_unstale":"2024-03-31T15:03:38.898739+0000","last_undegraded":"2024-03-31T15:03:38.898739+0000","last_fullsized":"2024-03-31T15:03:38.898739+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_clean_scrub_stamp":"2024-03-31T15:03:34.926145+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T15:58:18.492645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-31T15:03:37.967441+0000","last_change":"2024-03-31T15:03:36.969447+0000","last_active":"2024-03-31T15:03:37.967441+0000","last_peered":"2024-03-31T15:03:37.967441+0000","last_clean":"2024-03-31T15:03:37.967441+0000","last_became_active":"2024-03-31T15:03:36.967372+0000","last_became_peered":"2024-03-31T15:03:36.967372+0000","last_unstale":"2024-03-31T15:03:37.967441+0000","last_undegraded":"2024-03-31T15:03:37.967441+0000","last_fullsized":"2024-03-31T15:03:37.967441+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T17:27:21.111881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-31T15:03:39.382283+0000","last_change":"2024-03-31T15:03:36.980538+0000","last_active":"2024-03-31T15:03:39.382283+0000","last_peered":"2024-03-31T15:03:39.382283+0000","last_clean":"2024-03-31T15:03:39.382283+0000","last_became_active":"2024-03-31T15:03:36.980255+0000","last_became_peered":"2024-03-31T15:03:36.980255+0000","last_unstale":"2024-03-31T15:03:39.382283+0000","last_undegraded":"2024-03-31T15:03:39.382283+0000","last_fullsized":"2024-03-31T15:03:39.382283+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:14:42.246629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-31T15:03:38.898402+0000","last_change":"2024-03-31T15:03:35.962539+0000","last_active":"2024-03-31T15:03:38.898402+0000","last_peered":"2024-03-31T15:03:38.898402+0000","last_clean":"2024-03-31T15:03:38.898402+0000","last_became_active":"2024-03-31T15:03:35.961927+0000","last_became_peered":"2024-03-31T15:03:35.961927+0000","last_unstale":"2024-03-31T15:03:38.898402+0000","last_undegraded":"2024-03-31T15:03:38.898402+0000","last_fullsized":"2024-03-31T15:03:38.898402+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_clean_scrub_stamp":"2024-03-31T15:03:34.926145+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:48:53.962920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-31T15:03:39.382100+0000","last_change":"2024-03-31T15:03:36.981810+0000","last_active":"2024-03-31T15:03:39.382100+0000","last_peered":"2024-03-31T15:03:39.382100+0000","last_clean":"2024-03-31T15:03:39.382100+0000","last_became_active":"2024-03-31T15:03:36.981486+0000","last_became_peered":"2024-03-31T15:03:36.981486+0000","last_unstale":"2024-03-31T15:03:39.382100+0000","last_undegraded":"2024-03-31T15:03:39.382100+0000","last_fullsized":"2024-03-31T15:03:39.382100+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:41:13.119207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-31T15:03:38.898669+0000","last_change":"2024-03-31T15:03:35.972149+0000","last_active":"2024-03-31T15:03:38.898669+0000","last_peered":"2024-03-31T15:03:38.898669+0000","last_clean":"2024-03-31T15:03:38.898669+0000","last_became_active":"2024-03-31T15:03:35.971844+0000","last_became_peered":"2024-03-31T15:03:35.971844+0000","last_unstale":"2024-03-31T15:03:38.898669+0000","last_undegraded":"2024-03-31T15:03:38.898669+0000","last_fullsized":"2024-03-31T15:03:38.898669+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_clean_scrub_stamp":"2024-03-31T15:03:34.926145+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:19:23.333770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-31T15:03:39.380930+0000","last_change":"2024-03-31T15:03:36.988093+0000","last_active":"2024-03-31T15:03:39.380930+0000","last_peered":"2024-03-31T15:03:39.380930+0000","last_clean":"2024-03-31T15:03:39.380930+0000","last_became_active":"2024-03-31T15:03:36.987794+0000","last_became_peered":"2024-03-31T15:03:36.987794+0000","last_unstale":"2024-03-31T15:03:39.380930+0000","last_undegraded":"2024-03-31T15:03:39.380930+0000","last_fullsized":"2024-03-31T15:03:39.380930+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T17:27:21.111881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-31T15:03:37.963349+0000","last_change":"2024-03-31T15:03:35.969099+0000","last_active":"2024-03-31T15:03:37.963349+0000","last_peered":"2024-03-31T15:03:37.963349+0000","last_clean":"2024-03-31T15:03:37.963349+0000","last_became_active":"2024-03-31T15:03:35.968350+0000","last_became_peered":"2024-03-31T15:03:35.968350+0000","last_unstale":"2024-03-31T15:03:37.963349+0000","last_undegraded":"2024-03-31T15:03:37.963349+0000","last_fullsized":"2024-03-31T15:03:37.963349+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_clean_scrub_stamp":"2024-03-31T15:03:34.926145+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T17:31:20.984335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-31T15:03:37.963327+0000","last_change":"2024-03-31T15:03:36.960526+0000","last_active":"2024-03-31T15:03:37.963327+0000","last_peered":"2024-03-31T15:03:37.963327+0000","last_clean":"2024-03-31T15:03:37.963327+0000","last_became_active":"2024-03-31T15:03:36.960245+0000","last_became_peered":"2024-03-31T15:03:36.960245+0000","last_unstale":"2024-03-31T15:03:37.963327+0000","last_undegraded":"2024-03-31T15:03:37.963327+0000","last_fullsized":"2024-03-31T15:03:37.963327+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:35.931603+0000","last_clean_scrub_stamp":"2024-03-31T15:03:35.931603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T15:57:32.539376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"20'4","reported_seq":18,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-31T15:03:39.381081+0000","last_change":"2024-03-31T15:03:35.965202+0000","last_active":"2024-03-31T15:03:39.381081+0000","last_peered":"2024-03-31T15:03:39.381081+0000","last_clean":"2024-03-31T15:03:39.381081+0000","last_became_active":"2024-03-31T15:03:35.964867+0000","last_became_peered":"2024-03-31T15:03:35.964867+0000","last_unstale":"2024-03-31T15:03:39.381081+0000","last_undegraded":"2024-03-31T15:03:39.381081+0000","last_fullsized":"2024-03-31T15:03:39.381081+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:34.926145+0000","last_clean_scrub_stamp":"2024-03-31T15:03:34.926145+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T16:35:11.030977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509449,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":788,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":806912,"data_stored":663845,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509448,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":212992,"data_stored":72985,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":15,"seq":64424509447,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":221184,"data_stored":73021,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509446,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31388,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340452,"statfs":{"total":96636764160,"available":96604622848,"internally_reserved":0,"allocated":208896,"data_stored":72895,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542152,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27244,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344596,"statfs":{"total":96636764160,"available":96608866304,"internally_reserved":0,"allocated":221184,"data_stored":74507,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542152,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31404,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340436,"statfs":{"total":96636764160,"available":96604606464,"internally_reserved":0,"allocated":221184,"data_stored":74507,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542152,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31400,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340440,"statfs":{"total":96636764160,"available":96604610560,"internally_reserved":0,"allocated":217088,"data_stored":73055,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542152,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31976,"kb_used_data":796,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339864,"statfs":{"total":96636764160,"available":96604020736,"internally_reserved":0,"allocated":815104,"data_stored":663901,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-31T15:03:46.157 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-03-31T15:03:46.157 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-03-31T15:03:46.157 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-03-31T15:03:46.157 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-03-31T15:03:46.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:46.341+0000 7fcd27c00640 1 Processor -- start 2024-03-31T15:03:46.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:46.341+0000 7fcd27c00640 1 -- start start 2024-03-31T15:03:46.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:46.345+0000 7fcd27c00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fcd20141590 0x7fcd20144340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:46.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:46.345+0000 7fcd27c00640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcd20144980 con 0x7fcd201411b0 2024-03-31T15:03:46.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:46.345+0000 7fcd27c00640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcd20161fe0 con 0x7fcd20070030 2024-03-31T15:03:46.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:46.345+0000 7fcd27c00640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7fcd2015f910 con 0x7fcd20141590 2024-03-31T15:03:46.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:46.345+0000 7fcd25200640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fcd201411b0 0x7fcd2006f920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:34104/0 (socket says 172.21.15.16:34104) 2024-03-31T15:03:46.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:46.345+0000 7fcd25200640 1 -- 172.21.15.16:0/3150711670 learned_addr learned my addr 172.21.15.16:0/3150711670 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:03:46.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:46.345+0000 7fcd25c00640 1 --2- 172.21.15.16:0/3150711670 >> v2:172.21.15.89:3300/0 conn(0x7fcd20141590 0x7fcd20144340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:46.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:46.345+0000 7fcd1fe00640 1 -- 172.21.15.16:0/3150711670 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1552946349 0 0) 0x7fcd20144980 con 0x7fcd201411b0 2024-03-31T15:03:46.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:46.345+0000 7fcd1fe00640 1 -- 172.21.15.16:0/3150711670 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcd04003530 con 0x7fcd201411b0 2024-03-31T15:03:46.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:46.345+0000 7fcd1fe00640 1 -- 172.21.15.16:0/3150711670 <== mon.1 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1908183339 0 0) 0x7fcd20161fe0 con 0x7fcd20070030 2024-03-31T15:03:46.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:46.345+0000 7fcd1fe00640 1 -- 172.21.15.16:0/3150711670 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcd20144980 con 0x7fcd20070030 2024-03-31T15:03:46.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:46.345+0000 7fcd25c00640 1 -- 172.21.15.16:0/3150711670 >> v1:172.21.15.89:6789/0 conn(0x7fcd20070030 legacy=0x7fcd20143c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:46.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:46.345+0000 7fcd25c00640 1 -- 172.21.15.16:0/3150711670 >> v1:172.21.15.16:6789/0 conn(0x7fcd201411b0 legacy=0x7fcd2006f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:46.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:46.345+0000 7fcd25c00640 1 -- 172.21.15.16:0/3150711670 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcd201622c0 con 0x7fcd20141590 2024-03-31T15:03:46.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:46.345+0000 7fcd25c00640 1 --2- 172.21.15.16:0/3150711670 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcd20141590 0x7fcd20144340 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fcd18009d90 tx=0x7fcd1801c970 comp rx=0 tx=0).ready entity=mon.1 client_cookie=44cdd967b9c21b97 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:03:46.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:46.349+0000 7fcd1fe00640 1 -- 172.21.15.16:0/3150711670 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd180b2070 con 0x7fcd20141590 2024-03-31T15:03:46.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:46.349+0000 7fcd1fe00640 1 -- 172.21.15.16:0/3150711670 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcd180a5940 con 0x7fcd20141590 2024-03-31T15:03:46.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:46.349+0000 7fcd1fe00640 1 -- 172.21.15.16:0/3150711670 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd180a5d10 con 0x7fcd20141590 2024-03-31T15:03:46.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:46.349+0000 7fcd27c00640 1 -- 172.21.15.16:0/3150711670 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcd20141590 msgr2=0x7fcd20144340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:46.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:46.349+0000 7fcd27c00640 1 --2- 172.21.15.16:0/3150711670 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcd20141590 0x7fcd20144340 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fcd18009d90 tx=0x7fcd1801c970 comp rx=0 tx=0).stop 2024-03-31T15:03:46.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:46.349+0000 7fcd27c00640 1 -- 172.21.15.16:0/3150711670 shutdown_connections 2024-03-31T15:03:46.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:46.349+0000 7fcd27c00640 1 --2- 172.21.15.16:0/3150711670 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcd20141590 0x7fcd20144340 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:46.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:46.349+0000 7fcd27c00640 1 -- 172.21.15.16:0/3150711670 >> 172.21.15.16:0/3150711670 conn(0x7fcd20067930 msgr2=0x7fcd200b3ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:03:46.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:46.349+0000 7fcd27c00640 1 -- 172.21.15.16:0/3150711670 shutdown_connections 2024-03-31T15:03:46.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:46.349+0000 7fcd27c00640 1 -- 172.21.15.16:0/3150711670 wait complete. 2024-03-31T15:03:46.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:46.349+0000 7fcd27c00640 1 Processor -- start 2024-03-31T15:03:46.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:46.349+0000 7fcd27c00640 1 -- start start 2024-03-31T15:03:46.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:46.349+0000 7fcd27c00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fcd20070030 0x7fcd2013a3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:46.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:46.349+0000 7fcd27c00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcd201411b0 0x7fcd20150360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:46.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:46.349+0000 7fcd27c00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcd20141590 0x7fcd20155560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:46.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:46.349+0000 7fcd27c00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fcd2016be80 con 0x7fcd201411b0 2024-03-31T15:03:46.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:46.349+0000 7fcd27c00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fcd2016bd00 con 0x7fcd20141590 2024-03-31T15:03:46.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:46.349+0000 7fcd27c00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fcd2016c000 con 0x7fcd20070030 2024-03-31T15:03:46.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:46.349+0000 7fcd17e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcd201411b0 0x7fcd20150360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:46.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:46.349+0000 7fcd17e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcd201411b0 0x7fcd20150360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39258/0 (socket says 172.21.15.16:39258) 2024-03-31T15:03:46.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:46.349+0000 7fcd17e00640 1 -- 172.21.15.16:0/961519810 learned_addr learned my addr 172.21.15.16:0/961519810 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:03:46.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:46.349+0000 7fcd25200640 1 --2- 172.21.15.16:0/961519810 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fcd20070030 0x7fcd2013a3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:46.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:46.349+0000 7fcd25c00640 1 --2- 172.21.15.16:0/961519810 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcd20141590 0x7fcd20155560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:46.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:46.353+0000 7fcd17e00640 1 -- 172.21.15.16:0/961519810 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fcd20070030 msgr2=0x7fcd2013a3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:46.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:46.353+0000 7fcd17e00640 1 --2- 172.21.15.16:0/961519810 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fcd20070030 0x7fcd2013a3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:46.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:46.353+0000 7fcd17e00640 1 -- 172.21.15.16:0/961519810 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcd20141590 msgr2=0x7fcd20155560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:46.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:46.353+0000 7fcd17e00640 1 --2- 172.21.15.16:0/961519810 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcd20141590 0x7fcd20155560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:46.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:46.353+0000 7fcd17e00640 1 -- 172.21.15.16:0/961519810 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcd20151e40 con 0x7fcd201411b0 2024-03-31T15:03:46.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:46.353+0000 7fcd17e00640 1 --2- 172.21.15.16:0/961519810 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcd201411b0 0x7fcd20150360 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fcd0800c500 tx=0x7fcd0800c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:03:46.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:46.353+0000 7fcd1d600640 1 -- 172.21.15.16:0/961519810 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd080052d0 con 0x7fcd201411b0 2024-03-31T15:03:46.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:46.353+0000 7fcd1d600640 1 -- 172.21.15.16:0/961519810 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcd08005490 con 0x7fcd201411b0 2024-03-31T15:03:46.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:46.353+0000 7fcd1d600640 1 -- 172.21.15.16:0/961519810 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd08012850 con 0x7fcd201411b0 2024-03-31T15:03:46.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:46.353+0000 7fcd27c00640 1 -- 172.21.15.16:0/961519810 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcd20155aa0 con 0x7fcd201411b0 2024-03-31T15:03:46.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:46.353+0000 7fcd27c00640 1 -- 172.21.15.16:0/961519810 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcd20152390 con 0x7fcd201411b0 2024-03-31T15:03:46.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:46.353+0000 7fcd27c00640 1 -- 172.21.15.16:0/961519810 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcd20069950 con 0x7fcd201411b0 2024-03-31T15:03:46.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:46.353+0000 7fcd1d600640 1 -- 172.21.15.16:0/961519810 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fcd080129f0 con 0x7fcd201411b0 2024-03-31T15:03:46.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:46.353+0000 7fcd1d600640 1 --2- 172.21.15.16:0/961519810 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fccfc0683b0 0x7fccfc06a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:46.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:46.353+0000 7fcd1d600640 1 -- 172.21.15.16:0/961519810 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fcd08086530 con 0x7fcd201411b0 2024-03-31T15:03:46.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:46.353+0000 7fcd25200640 1 --2- 172.21.15.16:0/961519810 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fccfc0683b0 0x7fccfc06a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:46.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:46.357+0000 7fcd25200640 1 --2- 172.21.15.16:0/961519810 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fccfc0683b0 0x7fccfc06a870 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fcd100011f0 tx=0x7fcd10007d40 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:03:46.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:46.357+0000 7fcd1d600640 1 -- 172.21.15.16:0/961519810 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fcd080564b0 con 0x7fcd201411b0 2024-03-31T15:03:46.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:46.561+0000 7fcd27c00640 1 -- 172.21.15.16:0/961519810 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7fcd200b4a20 con 0x7fcd201411b0 2024-03-31T15:03:46.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:46.561+0000 7fcd1d600640 1 -- 172.21.15.16:0/961519810 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+46 (secure 0 0 0) 0x7fcd0805a7b0 con 0x7fcd201411b0 2024-03-31T15:03:46.555 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:03:46.555 INFO:teuthology.orchestra.run.smithi016.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-03-31T15:03:46.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:46.565+0000 7fcd27c00640 1 -- 172.21.15.16:0/961519810 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fccfc0683b0 msgr2=0x7fccfc06a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:46.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:46.565+0000 7fcd27c00640 1 --2- 172.21.15.16:0/961519810 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fccfc0683b0 0x7fccfc06a870 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fcd100011f0 tx=0x7fcd10007d40 comp rx=0 tx=0).stop 2024-03-31T15:03:46.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:46.565+0000 7fcd27c00640 1 -- 172.21.15.16:0/961519810 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcd201411b0 msgr2=0x7fcd20150360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:46.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:46.565+0000 7fcd27c00640 1 --2- 172.21.15.16:0/961519810 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcd201411b0 0x7fcd20150360 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fcd0800c500 tx=0x7fcd0800c9c0 comp rx=0 tx=0).stop 2024-03-31T15:03:46.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:46.565+0000 7fcd27c00640 1 -- 172.21.15.16:0/961519810 shutdown_connections 2024-03-31T15:03:46.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:46.565+0000 7fcd27c00640 1 --2- 172.21.15.16:0/961519810 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fccfc0683b0 0x7fccfc06a870 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:46.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:46.565+0000 7fcd27c00640 1 --2- 172.21.15.16:0/961519810 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcd20141590 0x7fcd20155560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:46.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:46.565+0000 7fcd27c00640 1 --2- 172.21.15.16:0/961519810 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcd201411b0 0x7fcd20150360 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:46.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:46.565+0000 7fcd27c00640 1 --2- 172.21.15.16:0/961519810 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fcd20070030 0x7fcd2013a3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:46.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:46.565+0000 7fcd27c00640 1 -- 172.21.15.16:0/961519810 >> 172.21.15.16:0/961519810 conn(0x7fcd20067930 msgr2=0x7fcd2006d940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:03:46.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:46.565+0000 7fcd27c00640 1 -- 172.21.15.16:0/961519810 shutdown_connections 2024-03-31T15:03:46.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:46.565+0000 7fcd27c00640 1 -- 172.21.15.16:0/961519810 wait complete. 2024-03-31T15:03:46.576 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-03-31T15:03:46.577 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-31T15:03:46.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:46.725+0000 7fc515600640 1 Processor -- start 2024-03-31T15:03:46.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:46.725+0000 7fc515600640 1 -- start start 2024-03-31T15:03:46.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:46.725+0000 7fc515600640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fc5100d1280 0x7fc5100d1660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:46.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:46.725+0000 7fc515600640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc5100d1ba0 con 0x7fc5100b6460 2024-03-31T15:03:46.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:46.725+0000 7fc515600640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc5100d3990 con 0x7fc5100b9c60 2024-03-31T15:03:46.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:46.725+0000 7fc515600640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7fc5100d26a0 con 0x7fc5100d1280 2024-03-31T15:03:46.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:46.725+0000 7fc50ea00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fc5100d1280 0x7fc5100d1660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:46.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:46.725+0000 7fc50ea00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fc5100d1280 0x7fc5100d1660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39266/0 (socket says 172.21.15.16:39266) 2024-03-31T15:03:46.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:46.725+0000 7fc50ea00640 1 -- 172.21.15.16:0/4085341499 learned_addr learned my addr 172.21.15.16:0/4085341499 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:03:46.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:46.725+0000 7fc50f400640 1 --1- 172.21.15.16:0/4085341499 >> v1:172.21.15.16:6789/0 conn(0x7fc5100b6460 0x7fc510169080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:34108/0 (socket says 172.21.15.16:34108) 2024-03-31T15:03:46.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:46.725+0000 7fc50ea00640 1 -- 172.21.15.16:0/4085341499 >> v1:172.21.15.89:6789/0 conn(0x7fc5100b9c60 legacy=0x7fc5100b5d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:46.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:46.725+0000 7fc50ea00640 1 -- 172.21.15.16:0/4085341499 >> v1:172.21.15.16:6789/0 conn(0x7fc5100b6460 legacy=0x7fc510169080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:46.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:46.725+0000 7fc50ea00640 1 -- 172.21.15.16:0/4085341499 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5100d28f0 con 0x7fc5100d1280 2024-03-31T15:03:46.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:46.729+0000 7fc50ea00640 1 --2- 172.21.15.16:0/4085341499 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc5100d1280 0x7fc5100d1660 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fc500009870 tx=0x7fc50001c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce099e077aa8e880 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:03:46.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:46.729+0000 7fc50e000640 1 -- 172.21.15.16:0/4085341499 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5000b1070 con 0x7fc5100d1280 2024-03-31T15:03:46.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:46.729+0000 7fc50e000640 1 -- 172.21.15.16:0/4085341499 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc5000ac620 con 0x7fc5100d1280 2024-03-31T15:03:46.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:46.729+0000 7fc50e000640 1 -- 172.21.15.16:0/4085341499 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5000ac9f0 con 0x7fc5100d1280 2024-03-31T15:03:46.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:46.729+0000 7fc515600640 1 -- 172.21.15.16:0/4085341499 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc5100d1280 msgr2=0x7fc5100d1660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:46.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:46.729+0000 7fc515600640 1 --2- 172.21.15.16:0/4085341499 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc5100d1280 0x7fc5100d1660 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fc500009870 tx=0x7fc50001c4d0 comp rx=0 tx=0).stop 2024-03-31T15:03:46.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:46.729+0000 7fc515600640 1 -- 172.21.15.16:0/4085341499 shutdown_connections 2024-03-31T15:03:46.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:46.729+0000 7fc515600640 1 --2- 172.21.15.16:0/4085341499 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc5100d1280 0x7fc5100d1660 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:46.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:46.729+0000 7fc515600640 1 -- 172.21.15.16:0/4085341499 >> 172.21.15.16:0/4085341499 conn(0x7fc510067730 msgr2=0x7fc510067b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:03:46.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:46.729+0000 7fc515600640 1 -- 172.21.15.16:0/4085341499 shutdown_connections 2024-03-31T15:03:46.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:46.729+0000 7fc515600640 1 -- 172.21.15.16:0/4085341499 wait complete. 2024-03-31T15:03:46.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:46.729+0000 7fc515600640 1 Processor -- start 2024-03-31T15:03:46.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:46.733+0000 7fc515600640 1 -- start start 2024-03-31T15:03:46.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:46.733+0000 7fc515600640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc5100b6460 0x7fc510154980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:46.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:46.733+0000 7fc515600640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fc5100b9c60 0x7fc5100cd470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:46.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:46.733+0000 7fc515600640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc5100d1280 0x7fc5100cd9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:46.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:46.733+0000 7fc515600640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fc5100d3e90 con 0x7fc5100d1280 2024-03-31T15:03:46.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:46.733+0000 7fc515600640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fc510161d80 con 0x7fc5100b6460 2024-03-31T15:03:46.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:46.733+0000 7fc515600640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fc5100d4010 con 0x7fc5100b9c60 2024-03-31T15:03:46.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:46.733+0000 7fc50f400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc5100d1280 0x7fc5100cd9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:46.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:46.733+0000 7fc50ea00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc5100b6460 0x7fc510154980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:46.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:46.733+0000 7fc50ea00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc5100b6460 0x7fc510154980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.16:36314/0 (socket says 172.21.15.16:36314) 2024-03-31T15:03:46.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:46.733+0000 7fc50f400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc5100d1280 0x7fc5100cd9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39272/0 (socket says 172.21.15.16:39272) 2024-03-31T15:03:46.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:46.733+0000 7fc50f400640 1 -- 172.21.15.16:0/1157386020 learned_addr learned my addr 172.21.15.16:0/1157386020 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:03:46.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:46.733+0000 7fc507e00640 1 --2- 172.21.15.16:0/1157386020 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fc5100b9c60 0x7fc5100cd470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:46.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:46.733+0000 7fc50f400640 1 -- 172.21.15.16:0/1157386020 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fc5100b9c60 msgr2=0x7fc5100cd470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:46.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:46.733+0000 7fc50f400640 1 --2- 172.21.15.16:0/1157386020 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fc5100b9c60 0x7fc5100cd470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:46.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:46.733+0000 7fc50f400640 1 -- 172.21.15.16:0/1157386020 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc5100b6460 msgr2=0x7fc510154980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:46.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:46.733+0000 7fc50f400640 1 --2- 172.21.15.16:0/1157386020 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc5100b6460 0x7fc510154980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:46.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:46.733+0000 7fc50f400640 1 -- 172.21.15.16:0/1157386020 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5100ce2d0 con 0x7fc5100d1280 2024-03-31T15:03:46.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:46.733+0000 7fc50f400640 1 --2- 172.21.15.16:0/1157386020 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc5100d1280 0x7fc5100cd9b0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fc50800c730 tx=0x7fc50800cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:03:46.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:46.733+0000 7fc506a00640 1 -- 172.21.15.16:0/1157386020 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc508017070 con 0x7fc5100d1280 2024-03-31T15:03:46.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:46.733+0000 7fc506a00640 1 -- 172.21.15.16:0/1157386020 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc50800ce40 con 0x7fc5100d1280 2024-03-31T15:03:46.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:46.733+0000 7fc506a00640 1 -- 172.21.15.16:0/1157386020 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc50800a840 con 0x7fc5100d1280 2024-03-31T15:03:46.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:46.733+0000 7fc515600640 1 -- 172.21.15.16:0/1157386020 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc510157080 con 0x7fc5100d1280 2024-03-31T15:03:46.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:46.733+0000 7fc515600640 1 -- 172.21.15.16:0/1157386020 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc510157640 con 0x7fc5100d1280 2024-03-31T15:03:46.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:46.733+0000 7fc515600640 1 -- 172.21.15.16:0/1157386020 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc510069950 con 0x7fc5100d1280 2024-03-31T15:03:46.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:46.737+0000 7fc506a00640 1 -- 172.21.15.16:0/1157386020 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fc50801a070 con 0x7fc5100d1280 2024-03-31T15:03:46.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:46.737+0000 7fc506a00640 1 --2- 172.21.15.16:0/1157386020 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fc4ec0683b0 0x7fc4ec06a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:46.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:46.737+0000 7fc506a00640 1 -- 172.21.15.16:0/1157386020 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fc508085f40 con 0x7fc5100d1280 2024-03-31T15:03:46.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:46.737+0000 7fc50ea00640 1 --2- 172.21.15.16:0/1157386020 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fc4ec0683b0 0x7fc4ec06a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:46.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:46.737+0000 7fc50ea00640 1 --2- 172.21.15.16:0/1157386020 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fc4ec0683b0 0x7fc4ec06a870 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fc5000ae000 tx=0x7fc5000a7000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:03:46.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:46.741+0000 7fc506a00640 1 -- 172.21.15.16:0/1157386020 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fc5100ce2d0 con 0x7fc5100d1280 2024-03-31T15:03:46.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:46.905+0000 7fc515600640 1 -- 172.21.15.16:0/1157386020 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc5100cfd40 con 0x7fc5100d1280 2024-03-31T15:03:46.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:46.905+0000 7fc506a00640 1 -- 172.21.15.16:0/1157386020 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) ==== 74+0+16533 (secure 0 0 0) 0x7fc508055e10 con 0x7fc5100d1280 2024-03-31T15:03:46.899 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:03:46.900 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":21,"fsid":"286eb5da-fb47-4550-b290-d9519c13b9a5","created":"2024-03-31T15:03:11.438952+0000","modified":"2024-03-31T15:03:37.957829+0000","last_up_change":"2024-03-31T15:03:22.885535+0000","last_in_change":"2024-03-31T15:03:13.941667+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-31T15:03:22.550787+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-31T15:03:34.488425+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-31T15:03:35.299116+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"92a66450-ef19-40d8-ac3d-d760b23b0bda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6801","nonce":932712244}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6803","nonce":932712244}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6807","nonce":932712244}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6805","nonce":932712244}]},"public_addr":"172.21.15.16:6801/932712244","cluster_addr":"172.21.15.16:6803/932712244","heartbeat_back_addr":"172.21.15.16:6807/932712244","heartbeat_front_addr":"172.21.15.16:6805/932712244","state":["exists","up"]},{"osd":1,"uuid":"da9d1f2b-1968-4959-b843-faa231988e92","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6810","nonce":1398737533}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6811","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6813","nonce":1398737533}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6821","nonce":1398737533}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6815","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6816","nonce":1398737533}]},"public_addr":"172.21.15.16:6810/1398737533","cluster_addr":"172.21.15.16:6813/1398737533","heartbeat_back_addr":"172.21.15.16:6821/1398737533","heartbeat_front_addr":"172.21.15.16:6816/1398737533","state":["exists","up"]},{"osd":2,"uuid":"01a73662-9a21-4b1f-bcbb-42b9636a4638","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6819","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6822","nonce":3018526646}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6825","nonce":3018526646}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6831","nonce":3018526646}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6827","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6829","nonce":3018526646}]},"public_addr":"172.21.15.16:6822/3018526646","cluster_addr":"172.21.15.16:6825/3018526646","heartbeat_back_addr":"172.21.15.16:6831/3018526646","heartbeat_front_addr":"172.21.15.16:6829/3018526646","state":["exists","up"]},{"osd":3,"uuid":"c7771fd3-e18e-4500-ac38-23cf10ebcd75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6809","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6812","nonce":3360352974}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6817","nonce":3360352974}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6828","nonce":3360352974}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6823","nonce":3360352974}]},"public_addr":"172.21.15.16:6812/3360352974","cluster_addr":"172.21.15.16:6817/3360352974","heartbeat_back_addr":"172.21.15.16:6828/3360352974","heartbeat_front_addr":"172.21.15.16:6823/3360352974","state":["exists","up"]},{"osd":4,"uuid":"88260730-1482-4fbd-a40f-cb1124e9df4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6801","nonce":1856844856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6803","nonce":1856844856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6807","nonce":1856844856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6805","nonce":1856844856}]},"public_addr":"172.21.15.89:6801/1856844856","cluster_addr":"172.21.15.89:6803/1856844856","heartbeat_back_addr":"172.21.15.89:6807/1856844856","heartbeat_front_addr":"172.21.15.89:6805/1856844856","state":["exists","up"]},{"osd":5,"uuid":"4cb2c00c-fae0-4dd8-9d27-ec4f0093ccf7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6814","nonce":50395527}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6817","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6820","nonce":50395527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6831","nonce":50395527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6823","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6826","nonce":50395527}]},"public_addr":"172.21.15.89:6814/50395527","cluster_addr":"172.21.15.89:6820/50395527","heartbeat_back_addr":"172.21.15.89:6831/50395527","heartbeat_front_addr":"172.21.15.89:6826/50395527","state":["exists","up"]},{"osd":6,"uuid":"81ce0c19-2554-4cae-acbe-66a05e9e9b9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6809","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6812","nonce":3470471752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6815","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6818","nonce":3470471752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6827","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6829","nonce":3470471752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6821","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6824","nonce":3470471752}]},"public_addr":"172.21.15.89:6812/3470471752","cluster_addr":"172.21.15.89:6818/3470471752","heartbeat_back_addr":"172.21.15.89:6829/3470471752","heartbeat_front_addr":"172.21.15.89:6824/3470471752","state":["exists","up"]},{"osd":7,"uuid":"85e4313b-d37d-442e-8042-153bf20a847d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6811","nonce":1796778630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6813","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6816","nonce":1796778630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6825","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6828","nonce":1796778630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6819","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6822","nonce":1796778630}]},"public_addr":"172.21.15.89:6811/1796778630","cluster_addr":"172.21.15.89:6816/1796778630","heartbeat_back_addr":"172.21.15.89:6828/1796778630","heartbeat_front_addr":"172.21.15.89:6822/1796778630","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.439088+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.529970+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.680512+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:21.044161+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.582111+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.536558+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.409522+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.590229+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-31T15:03:46.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:46.909+0000 7fc515600640 1 -- 172.21.15.16:0/1157386020 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fc4ec0683b0 msgr2=0x7fc4ec06a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:46.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:46.909+0000 7fc515600640 1 --2- 172.21.15.16:0/1157386020 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fc4ec0683b0 0x7fc4ec06a870 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fc5000ae000 tx=0x7fc5000a7000 comp rx=0 tx=0).stop 2024-03-31T15:03:46.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:46.909+0000 7fc515600640 1 -- 172.21.15.16:0/1157386020 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc5100d1280 msgr2=0x7fc5100cd9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:46.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:46.909+0000 7fc515600640 1 --2- 172.21.15.16:0/1157386020 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc5100d1280 0x7fc5100cd9b0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fc50800c730 tx=0x7fc50800cbf0 comp rx=0 tx=0).stop 2024-03-31T15:03:46.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:46.909+0000 7fc515600640 1 -- 172.21.15.16:0/1157386020 shutdown_connections 2024-03-31T15:03:46.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:46.909+0000 7fc515600640 1 --2- 172.21.15.16:0/1157386020 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fc4ec0683b0 0x7fc4ec06a870 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:46.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:46.909+0000 7fc515600640 1 --2- 172.21.15.16:0/1157386020 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc5100d1280 0x7fc5100cd9b0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:46.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:46.909+0000 7fc515600640 1 --2- 172.21.15.16:0/1157386020 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fc5100b9c60 0x7fc5100cd470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:46.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:46.909+0000 7fc515600640 1 --2- 172.21.15.16:0/1157386020 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc5100b6460 0x7fc510154980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:46.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:46.909+0000 7fc515600640 1 -- 172.21.15.16:0/1157386020 >> 172.21.15.16:0/1157386020 conn(0x7fc510067730 msgr2=0x7fc5100b8c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:03:46.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:46.909+0000 7fc515600640 1 -- 172.21.15.16:0/1157386020 shutdown_connections 2024-03-31T15:03:46.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:46.909+0000 7fc515600640 1 -- 172.21.15.16:0/1157386020 wait complete. 2024-03-31T15:03:46.920 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-31T15:03:22.550787+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-31T15:03:34.488425+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-31T15:03:35.299116+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-31T15:03:46.921 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-31T15:03:47.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:47.073+0000 7f842b000640 1 Processor -- start 2024-03-31T15:03:47.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:47.073+0000 7f842b000640 1 -- start start 2024-03-31T15:03:47.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:47.073+0000 7f842b000640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f842415f6c0 0x7f84240ad8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:47.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:47.073+0000 7f842b000640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f84240cde30 0x7f84240cf660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:47.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:47.073+0000 7f842b000640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f842406e5f0 con 0x7f84240cda50 2024-03-31T15:03:47.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:47.073+0000 7f842b000640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f84240cec10 con 0x7f842415f6c0 2024-03-31T15:03:47.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:47.073+0000 7f842b000640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f84240cf1d0 con 0x7f84240cde30 2024-03-31T15:03:47.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:47.073+0000 7f841b400640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f842415f6c0 0x7f84240ad8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:47.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:47.073+0000 7f841b400640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f842415f6c0 0x7f84240ad8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53848/0 (socket says 172.21.15.16:53848) 2024-03-31T15:03:47.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:47.073+0000 7f841b400640 1 -- 172.21.15.16:0/2407589425 learned_addr learned my addr 172.21.15.16:0/2407589425 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:03:47.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:47.073+0000 7f8429000640 1 --2- 172.21.15.16:0/2407589425 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f84240cde30 0x7f84240cf660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:47.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:47.073+0000 7f841b400640 1 -- 172.21.15.16:0/2407589425 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f84240cde30 msgr2=0x7f84240cf660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:03:47.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:47.073+0000 7f841b400640 1 --2- 172.21.15.16:0/2407589425 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f84240cde30 0x7f84240cf660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:47.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:47.073+0000 7f841b400640 1 -- 172.21.15.16:0/2407589425 >> v1:172.21.15.89:6789/0 conn(0x7f84240cda50 legacy=0x7f842406de10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:47.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:47.073+0000 7f841b400640 1 -- 172.21.15.16:0/2407589425 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f842406e8d0 con 0x7f842415f6c0 2024-03-31T15:03:47.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:47.077+0000 7f841b400640 1 --2- 172.21.15.16:0/2407589425 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f842415f6c0 0x7f84240ad8d0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f840c009870 tx=0x7f840c01ed50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a940502cd5b0d51 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:03:47.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:47.077+0000 7f8423400640 1 -- 172.21.15.16:0/2407589425 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f840c0b5070 con 0x7f842415f6c0 2024-03-31T15:03:47.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:47.077+0000 7f8423400640 1 -- 172.21.15.16:0/2407589425 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f840c0a8a50 con 0x7f842415f6c0 2024-03-31T15:03:47.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:47.077+0000 7f8423400640 1 -- 172.21.15.16:0/2407589425 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f840c0a8e20 con 0x7f842415f6c0 2024-03-31T15:03:47.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:47.077+0000 7f842b000640 1 -- 172.21.15.16:0/2407589425 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f842415f6c0 msgr2=0x7f84240ad8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:47.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:47.077+0000 7f842b000640 1 --2- 172.21.15.16:0/2407589425 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f842415f6c0 0x7f84240ad8d0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f840c009870 tx=0x7f840c01ed50 comp rx=0 tx=0).stop 2024-03-31T15:03:47.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:47.077+0000 7f842b000640 1 -- 172.21.15.16:0/2407589425 shutdown_connections 2024-03-31T15:03:47.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:47.077+0000 7f842b000640 1 --2- 172.21.15.16:0/2407589425 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f84240cde30 0x7f84240cf660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:47.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:47.077+0000 7f842b000640 1 --2- 172.21.15.16:0/2407589425 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f842415f6c0 0x7f84240ad8d0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:47.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:47.077+0000 7f842b000640 1 -- 172.21.15.16:0/2407589425 >> 172.21.15.16:0/2407589425 conn(0x7f84240b2db0 msgr2=0x7f84240cd340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:03:47.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:47.077+0000 7f842b000640 1 -- 172.21.15.16:0/2407589425 shutdown_connections 2024-03-31T15:03:47.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:47.077+0000 7f842b000640 1 -- 172.21.15.16:0/2407589425 wait complete. 2024-03-31T15:03:47.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:47.077+0000 7f842b000640 1 Processor -- start 2024-03-31T15:03:47.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:47.077+0000 7f842b000640 1 -- start start 2024-03-31T15:03:47.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:47.081+0000 7f842b000640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f84240cda50 0x7f842414a910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:47.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:47.081+0000 7f842b000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f84240cde30 0x7f8424152100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:47.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:47.081+0000 7f842b000640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f842415f6c0 0x7f8424152640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:47.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:47.081+0000 7f842b000640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f84240b39a0 con 0x7f84240cde30 2024-03-31T15:03:47.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:47.081+0000 7f842b000640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f842415e960 con 0x7f842415f6c0 2024-03-31T15:03:47.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:47.081+0000 7f842b000640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f84240ce2c0 con 0x7f84240cda50 2024-03-31T15:03:47.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:47.081+0000 7f841b400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f84240cde30 0x7f8424152100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:47.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:47.081+0000 7f841b400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f84240cde30 0x7f8424152100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53856/0 (socket says 172.21.15.16:53856) 2024-03-31T15:03:47.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:47.081+0000 7f841b400640 1 -- 172.21.15.16:0/1983552956 learned_addr learned my addr 172.21.15.16:0/1983552956 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:03:47.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:47.081+0000 7f8423e00640 1 --2- 172.21.15.16:0/1983552956 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f84240cda50 0x7f842414a910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:47.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:47.081+0000 7f8429000640 1 --2- 172.21.15.16:0/1983552956 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f842415f6c0 0x7f8424152640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:47.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:47.081+0000 7f841b400640 1 -- 172.21.15.16:0/1983552956 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f84240cda50 msgr2=0x7f842414a910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:47.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:47.081+0000 7f841b400640 1 --2- 172.21.15.16:0/1983552956 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f84240cda50 0x7f842414a910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:47.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:47.081+0000 7f841b400640 1 -- 172.21.15.16:0/1983552956 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f842415f6c0 msgr2=0x7f8424152640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:47.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:47.081+0000 7f841b400640 1 --2- 172.21.15.16:0/1983552956 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f842415f6c0 0x7f8424152640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:47.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:47.081+0000 7f841b400640 1 -- 172.21.15.16:0/1983552956 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8424152cc0 con 0x7f84240cde30 2024-03-31T15:03:47.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:47.081+0000 7f841b400640 1 --2- 172.21.15.16:0/1983552956 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f84240cde30 0x7f8424152100 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f840c01d400 tx=0x7f840c01baf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:03:47.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:47.081+0000 7f8420c00640 1 -- 172.21.15.16:0/1983552956 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f840c0c0070 con 0x7f84240cde30 2024-03-31T15:03:47.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:47.081+0000 7f8420c00640 1 -- 172.21.15.16:0/1983552956 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f840c0b08c0 con 0x7f84240cde30 2024-03-31T15:03:47.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:47.081+0000 7f842b000640 1 -- 172.21.15.16:0/1983552956 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8424152fa0 con 0x7f84240cde30 2024-03-31T15:03:47.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:47.081+0000 7f8420c00640 1 -- 172.21.15.16:0/1983552956 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f840c0b5040 con 0x7f84240cde30 2024-03-31T15:03:47.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:47.081+0000 7f842b000640 1 -- 172.21.15.16:0/1983552956 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8424153560 con 0x7f84240cde30 2024-03-31T15:03:47.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:47.081+0000 7f842b000640 1 -- 172.21.15.16:0/1983552956 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8424069950 con 0x7f84240cde30 2024-03-31T15:03:47.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:47.081+0000 7f8420c00640 1 -- 172.21.15.16:0/1983552956 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f840c0c3070 con 0x7f84240cde30 2024-03-31T15:03:47.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:47.085+0000 7f8420c00640 1 --2- 172.21.15.16:0/1983552956 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f8400068380 0x7f840006a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:47.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:47.085+0000 7f8420c00640 1 -- 172.21.15.16:0/1983552956 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f840c123940 con 0x7f84240cde30 2024-03-31T15:03:47.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:47.085+0000 7f8423e00640 1 --2- 172.21.15.16:0/1983552956 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f8400068380 0x7f840006a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:47.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:47.085+0000 7f8423e00640 1 --2- 172.21.15.16:0/1983552956 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f8400068380 0x7f840006a840 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f84241f7a00 tx=0x7f8414009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:03:47.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:47.089+0000 7f8420c00640 1 -- 172.21.15.16:0/1983552956 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f840c0188b0 con 0x7f84240cde30 2024-03-31T15:03:47.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:47.253+0000 7f842b000640 1 -- 172.21.15.16:0/1983552956 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f84240b55c0 con 0x7f84240cde30 2024-03-31T15:03:47.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:47.253+0000 7f8420c00640 1 -- 172.21.15.16:0/1983552956 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) ==== 93+0+10 (secure 0 0 0) 0x7f840c0f38a0 con 0x7f84240cde30 2024-03-31T15:03:47.245 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2024-03-31T15:03:47.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:47.253+0000 7f842b000640 1 -- 172.21.15.16:0/1983552956 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f8400068380 msgr2=0x7f840006a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:47.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:47.253+0000 7f842b000640 1 --2- 172.21.15.16:0/1983552956 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f8400068380 0x7f840006a840 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f84241f7a00 tx=0x7f8414009040 comp rx=0 tx=0).stop 2024-03-31T15:03:47.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:47.253+0000 7f842b000640 1 -- 172.21.15.16:0/1983552956 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f84240cde30 msgr2=0x7f8424152100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:47.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:47.253+0000 7f842b000640 1 --2- 172.21.15.16:0/1983552956 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f84240cde30 0x7f8424152100 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f840c01d400 tx=0x7f840c01baf0 comp rx=0 tx=0).stop 2024-03-31T15:03:47.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:47.253+0000 7f842b000640 1 -- 172.21.15.16:0/1983552956 shutdown_connections 2024-03-31T15:03:47.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:47.253+0000 7f842b000640 1 --2- 172.21.15.16:0/1983552956 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f8400068380 0x7f840006a840 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:47.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:47.253+0000 7f842b000640 1 --2- 172.21.15.16:0/1983552956 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f842415f6c0 0x7f8424152640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:47.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:47.253+0000 7f842b000640 1 --2- 172.21.15.16:0/1983552956 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f84240cde30 0x7f8424152100 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:47.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:47.253+0000 7f842b000640 1 --2- 172.21.15.16:0/1983552956 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f84240cda50 0x7f842414a910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:47.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:47.253+0000 7f842b000640 1 -- 172.21.15.16:0/1983552956 >> 172.21.15.16:0/1983552956 conn(0x7f84240b2db0 msgr2=0x7f84240cf560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:03:47.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:47.253+0000 7f842b000640 1 -- 172.21.15.16:0/1983552956 shutdown_connections 2024-03-31T15:03:47.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:47.253+0000 7f842b000640 1 -- 172.21.15.16:0/1983552956 wait complete. 2024-03-31T15:03:47.267 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-31T15:03:47.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:47.417+0000 7f6636000640 1 Processor -- start 2024-03-31T15:03:47.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:47.417+0000 7f6636000640 1 -- start start 2024-03-31T15:03:47.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:47.417+0000 7f6636000640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f66301567e0 0x7f6630072c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:47.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:47.417+0000 7f6636000640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f663015c7e0 con 0x7f663006dbe0 2024-03-31T15:03:47.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:47.417+0000 7f6636000640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f66300736b0 con 0x7f66300731c0 2024-03-31T15:03:47.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:47.417+0000 7f6636000640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f663015caa0 con 0x7f66301567e0 2024-03-31T15:03:47.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:47.421+0000 7f662f400640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f66301567e0 0x7f6630072c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:47.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:47.421+0000 7f662f400640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f66301567e0 0x7f6630072c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53864/0 (socket says 172.21.15.16:53864) 2024-03-31T15:03:47.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:47.421+0000 7f662fe00640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f663006dbe0 0x7f6630071550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:34882/0 (socket says 172.21.15.16:34882) 2024-03-31T15:03:47.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:47.421+0000 7f6626a00640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f66300731c0 0x7f663006d4d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.16:58724/0 (socket says 172.21.15.16:58724) 2024-03-31T15:03:47.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:47.421+0000 7f662f400640 1 -- 172.21.15.16:0/68663839 learned_addr learned my addr 172.21.15.16:0/68663839 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:03:47.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:47.421+0000 7f662f400640 1 -- 172.21.15.16:0/68663839 >> v1:172.21.15.89:6789/0 conn(0x7f66300731c0 legacy=0x7f663006d4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:47.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:47.421+0000 7f662ea00640 1 -- 172.21.15.16:0/68663839 <== mon.1 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 171856893 0 0) 0x7f66300736b0 con 0x7f66300731c0 2024-03-31T15:03:47.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:47.421+0000 7f662f400640 1 -- 172.21.15.16:0/68663839 >> v1:172.21.15.16:6789/0 conn(0x7f663006dbe0 legacy=0x7f6630071550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:47.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:47.421+0000 7f662f400640 1 -- 172.21.15.16:0/68663839 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f663015cd80 con 0x7f66301567e0 2024-03-31T15:03:47.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:47.421+0000 7f662f400640 1 --2- 172.21.15.16:0/68663839 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f66301567e0 0x7f6630072c80 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f6620009870 tx=0x7f662001c3f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5bb527a935aa982a server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:03:47.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:47.421+0000 7f662ea00640 1 -- 172.21.15.16:0/68663839 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66200b1070 con 0x7f66301567e0 2024-03-31T15:03:47.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:47.421+0000 7f662ea00640 1 -- 172.21.15.16:0/68663839 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f66200ac5f0 con 0x7f66301567e0 2024-03-31T15:03:47.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:47.421+0000 7f662ea00640 1 -- 172.21.15.16:0/68663839 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66200ac9c0 con 0x7f66301567e0 2024-03-31T15:03:47.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:47.425+0000 7f6636000640 1 -- 172.21.15.16:0/68663839 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f66301567e0 msgr2=0x7f6630072c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:47.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:47.425+0000 7f6636000640 1 --2- 172.21.15.16:0/68663839 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f66301567e0 0x7f6630072c80 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f6620009870 tx=0x7f662001c3f0 comp rx=0 tx=0).stop 2024-03-31T15:03:47.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:47.425+0000 7f6636000640 1 -- 172.21.15.16:0/68663839 shutdown_connections 2024-03-31T15:03:47.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:47.425+0000 7f6636000640 1 --2- 172.21.15.16:0/68663839 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f66301567e0 0x7f6630072c80 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:47.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:47.425+0000 7f6636000640 1 -- 172.21.15.16:0/68663839 >> 172.21.15.16:0/68663839 conn(0x7f663006ee40 msgr2=0x7f663006b620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:03:47.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:47.425+0000 7f6636000640 1 -- 172.21.15.16:0/68663839 shutdown_connections 2024-03-31T15:03:47.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:47.425+0000 7f6636000640 1 -- 172.21.15.16:0/68663839 wait complete. 2024-03-31T15:03:47.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:47.425+0000 7f6636000640 1 Processor -- start 2024-03-31T15:03:47.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:47.425+0000 7f6636000640 1 -- start start 2024-03-31T15:03:47.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:47.425+0000 7f6636000640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f663006dbe0 0x7f66301579d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:47.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:47.425+0000 7f6636000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f66300731c0 0x7f663015b560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:47.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:47.425+0000 7f6636000640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f66301567e0 0x7f663015baa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:47.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:47.425+0000 7f6636000640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f66300afbb0 con 0x7f66300731c0 2024-03-31T15:03:47.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:47.425+0000 7f6636000640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f66300afa30 con 0x7f663006dbe0 2024-03-31T15:03:47.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:47.425+0000 7f6636000640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f663015d580 con 0x7f66301567e0 2024-03-31T15:03:47.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:47.425+0000 7f6626a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f66300731c0 0x7f663015b560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:47.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:47.425+0000 7f6626a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f66300731c0 0x7f663015b560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53876/0 (socket says 172.21.15.16:53876) 2024-03-31T15:03:47.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:47.425+0000 7f662f400640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f663006dbe0 0x7f66301579d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:47.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:47.425+0000 7f6626a00640 1 -- 172.21.15.16:0/3081934457 learned_addr learned my addr 172.21.15.16:0/3081934457 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:03:47.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:47.425+0000 7f662fe00640 1 --2- 172.21.15.16:0/3081934457 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f66301567e0 0x7f663015baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:47.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:47.425+0000 7f6626a00640 1 -- 172.21.15.16:0/3081934457 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f66301567e0 msgr2=0x7f663015baa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:47.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:47.425+0000 7f6626a00640 1 --2- 172.21.15.16:0/3081934457 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f66301567e0 0x7f663015baa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:47.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:47.425+0000 7f6626a00640 1 -- 172.21.15.16:0/3081934457 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f663006dbe0 msgr2=0x7f66301579d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:47.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:47.425+0000 7f6626a00640 1 --2- 172.21.15.16:0/3081934457 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f663006dbe0 0x7f66301579d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:47.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:47.425+0000 7f6626a00640 1 -- 172.21.15.16:0/3081934457 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66301583e0 con 0x7f66300731c0 2024-03-31T15:03:47.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:47.425+0000 7f662f400640 1 --2- 172.21.15.16:0/3081934457 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f663006dbe0 0x7f66301579d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:03:47.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:47.429+0000 7f6626a00640 1 --2- 172.21.15.16:0/3081934457 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f66300731c0 0x7f663015b560 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f661800c480 tx=0x7f661800c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:03:47.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:47.429+0000 7f6627e00640 1 -- 172.21.15.16:0/3081934457 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6618018070 con 0x7f66300731c0 2024-03-31T15:03:47.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:47.429+0000 7f6627e00640 1 -- 172.21.15.16:0/3081934457 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f66180074e0 con 0x7f66300731c0 2024-03-31T15:03:47.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:47.429+0000 7f6627e00640 1 -- 172.21.15.16:0/3081934457 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6618004a30 con 0x7f66300731c0 2024-03-31T15:03:47.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:47.429+0000 7f6636000640 1 -- 172.21.15.16:0/3081934457 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f66301586c0 con 0x7f66300731c0 2024-03-31T15:03:47.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:47.429+0000 7f6636000640 1 -- 172.21.15.16:0/3081934457 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6630158c80 con 0x7f66300731c0 2024-03-31T15:03:47.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:47.429+0000 7f6636000640 1 -- 172.21.15.16:0/3081934457 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6630069950 con 0x7f66300731c0 2024-03-31T15:03:47.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:47.429+0000 7f6627e00640 1 -- 172.21.15.16:0/3081934457 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f661801b070 con 0x7f66300731c0 2024-03-31T15:03:47.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:47.429+0000 7f6627e00640 1 --2- 172.21.15.16:0/3081934457 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f660c0683b0 0x7f660c06a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:47.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:47.429+0000 7f6627e00640 1 -- 172.21.15.16:0/3081934457 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f6618085bf0 con 0x7f66300731c0 2024-03-31T15:03:47.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:47.429+0000 7f662f400640 1 --2- 172.21.15.16:0/3081934457 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f660c0683b0 0x7f660c06a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:47.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:47.433+0000 7f662f400640 1 --2- 172.21.15.16:0/3081934457 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f660c0683b0 0x7f660c06a870 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f66200ae000 tx=0x7f6620008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:03:47.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:47.433+0000 7f6627e00640 1 -- 172.21.15.16:0/3081934457 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f6618055bf0 con 0x7f66300731c0 2024-03-31T15:03:47.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:47.597+0000 7f6636000640 1 -- 172.21.15.16:0/3081934457 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f663006b900 con 0x7f66300731c0 2024-03-31T15:03:47.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:47.597+0000 7f6627e00640 1 -- 172.21.15.16:0/3081934457 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) ==== 104+0+11 (secure 0 0 0) 0x7f6618059ef0 con 0x7f66300731c0 2024-03-31T15:03:47.589 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2024-03-31T15:03:47.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:47.597+0000 7f6636000640 1 -- 172.21.15.16:0/3081934457 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f660c0683b0 msgr2=0x7f660c06a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:47.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:47.597+0000 7f6636000640 1 --2- 172.21.15.16:0/3081934457 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f660c0683b0 0x7f660c06a870 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f66200ae000 tx=0x7f6620008040 comp rx=0 tx=0).stop 2024-03-31T15:03:47.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:47.597+0000 7f6636000640 1 -- 172.21.15.16:0/3081934457 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f66300731c0 msgr2=0x7f663015b560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:47.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:47.597+0000 7f6636000640 1 --2- 172.21.15.16:0/3081934457 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f66300731c0 0x7f663015b560 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f661800c480 tx=0x7f661800c940 comp rx=0 tx=0).stop 2024-03-31T15:03:47.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:47.597+0000 7f6636000640 1 -- 172.21.15.16:0/3081934457 shutdown_connections 2024-03-31T15:03:47.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:47.597+0000 7f6636000640 1 --2- 172.21.15.16:0/3081934457 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f660c0683b0 0x7f660c06a870 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:47.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:47.597+0000 7f6636000640 1 --2- 172.21.15.16:0/3081934457 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f66301567e0 0x7f663015baa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:47.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:47.597+0000 7f6636000640 1 --2- 172.21.15.16:0/3081934457 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f66300731c0 0x7f663015b560 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:47.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:47.597+0000 7f6636000640 1 --2- 172.21.15.16:0/3081934457 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f663006dbe0 0x7f66301579d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:47.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:47.597+0000 7f6636000640 1 -- 172.21.15.16:0/3081934457 >> 172.21.15.16:0/3081934457 conn(0x7f663006ee40 msgr2=0x7f663015e180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:03:47.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:47.597+0000 7f6636000640 1 -- 172.21.15.16:0/3081934457 shutdown_connections 2024-03-31T15:03:47.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:47.597+0000 7f6636000640 1 -- 172.21.15.16:0/3081934457 wait complete. 2024-03-31T15:03:47.611 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-31T15:03:47.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:47.761+0000 7fe067e00640 1 Processor -- start 2024-03-31T15:03:47.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:47.761+0000 7fe067e00640 1 -- start start 2024-03-31T15:03:47.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:47.761+0000 7fe067e00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fe0601411b0 0x7fe06006f920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:47.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:47.761+0000 7fe067e00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fe060141590 0x7fe06006fe60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:47.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:47.761+0000 7fe067e00640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe060144980 con 0x7fe060143c30 2024-03-31T15:03:47.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:47.761+0000 7fe067e00640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7fe06015f910 con 0x7fe060141590 2024-03-31T15:03:47.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:47.761+0000 7fe067e00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fe060162910 con 0x7fe0601411b0 2024-03-31T15:03:47.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:47.761+0000 7fe065e00640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fe060143c30 0x7fe0601440a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:34890/0 (socket says 172.21.15.16:34890) 2024-03-31T15:03:47.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:47.761+0000 7fe065e00640 1 -- 172.21.15.16:0/2843638624 learned_addr learned my addr 172.21.15.16:0/2843638624 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:03:47.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:47.761+0000 7fe065400640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fe0601411b0 0x7fe06006f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:47.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:47.761+0000 7fe064a00640 1 --2- 172.21.15.16:0/2843638624 >> v2:172.21.15.89:3300/0 conn(0x7fe060141590 0x7fe06006fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:47.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:47.761+0000 7fe064a00640 1 -- 172.21.15.16:0/2843638624 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fe0601411b0 msgr2=0x7fe06006f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:03:47.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:47.761+0000 7fe064a00640 1 --2- 172.21.15.16:0/2843638624 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fe0601411b0 0x7fe06006f920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:47.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:47.761+0000 7fe064a00640 1 -- 172.21.15.16:0/2843638624 >> v1:172.21.15.16:6789/0 conn(0x7fe060143c30 legacy=0x7fe0601440a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:47.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:47.761+0000 7fe064a00640 1 -- 172.21.15.16:0/2843638624 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe060161fe0 con 0x7fe060141590 2024-03-31T15:03:47.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:47.761+0000 7fe05fe00640 1 -- 172.21.15.16:0/2843638624 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2124221773 0 0) 0x7fe060144980 con 0x7fe060143c30 2024-03-31T15:03:47.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:47.765+0000 7fe064a00640 1 --2- 172.21.15.16:0/2843638624 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe060141590 0x7fe06006fe60 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fe04c0097c0 tx=0x7fe04c01c4a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e3ee6651e14903fe server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:03:47.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:47.765+0000 7fe05fe00640 1 -- 172.21.15.16:0/2843638624 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe04c0b2070 con 0x7fe060141590 2024-03-31T15:03:47.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:47.765+0000 7fe05fe00640 1 -- 172.21.15.16:0/2843638624 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe04c0ad640 con 0x7fe060141590 2024-03-31T15:03:47.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:47.765+0000 7fe05fe00640 1 -- 172.21.15.16:0/2843638624 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe04c0ada10 con 0x7fe060141590 2024-03-31T15:03:47.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:47.765+0000 7fe067e00640 1 -- 172.21.15.16:0/2843638624 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe060141590 msgr2=0x7fe06006fe60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:47.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:47.765+0000 7fe067e00640 1 --2- 172.21.15.16:0/2843638624 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe060141590 0x7fe06006fe60 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fe04c0097c0 tx=0x7fe04c01c4a0 comp rx=0 tx=0).stop 2024-03-31T15:03:47.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:47.765+0000 7fe067e00640 1 -- 172.21.15.16:0/2843638624 shutdown_connections 2024-03-31T15:03:47.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:47.765+0000 7fe067e00640 1 --2- 172.21.15.16:0/2843638624 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe060141590 0x7fe06006fe60 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:47.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:47.765+0000 7fe067e00640 1 --2- 172.21.15.16:0/2843638624 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fe0601411b0 0x7fe06006f920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:47.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:47.765+0000 7fe067e00640 1 -- 172.21.15.16:0/2843638624 >> 172.21.15.16:0/2843638624 conn(0x7fe060067930 msgr2=0x7fe0600b3ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:03:47.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:47.765+0000 7fe067e00640 1 -- 172.21.15.16:0/2843638624 shutdown_connections 2024-03-31T15:03:47.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:47.765+0000 7fe067e00640 1 -- 172.21.15.16:0/2843638624 wait complete. 2024-03-31T15:03:47.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:47.769+0000 7fe067e00640 1 Processor -- start 2024-03-31T15:03:47.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:47.769+0000 7fe067e00640 1 -- start start 2024-03-31T15:03:47.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:47.769+0000 7fe067e00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fe0601411b0 0x7fe06013c860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:47.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:47.769+0000 7fe067e00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe060141590 0x7fe0601513c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:47.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:47.769+0000 7fe067e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe060143c30 0x7fe06013fe50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:47.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:47.769+0000 7fe067e00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fe06015f290 con 0x7fe060143c30 2024-03-31T15:03:47.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:47.769+0000 7fe067e00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fe06015f110 con 0x7fe060141590 2024-03-31T15:03:47.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:47.769+0000 7fe067e00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fe06015f410 con 0x7fe0601411b0 2024-03-31T15:03:47.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:47.769+0000 7fe065e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe060143c30 0x7fe06013fe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:47.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:47.769+0000 7fe065e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe060143c30 0x7fe06013fe50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53884/0 (socket says 172.21.15.16:53884) 2024-03-31T15:03:47.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:47.769+0000 7fe065e00640 1 -- 172.21.15.16:0/820558661 learned_addr learned my addr 172.21.15.16:0/820558661 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:03:47.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:47.769+0000 7fe064a00640 1 --2- 172.21.15.16:0/820558661 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe060141590 0x7fe0601513c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:47.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:47.769+0000 7fe065400640 1 --2- 172.21.15.16:0/820558661 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fe0601411b0 0x7fe06013c860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:47.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:47.769+0000 7fe065e00640 1 -- 172.21.15.16:0/820558661 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fe0601411b0 msgr2=0x7fe06013c860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:47.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:47.769+0000 7fe065e00640 1 --2- 172.21.15.16:0/820558661 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fe0601411b0 0x7fe06013c860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:47.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:47.769+0000 7fe065e00640 1 -- 172.21.15.16:0/820558661 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe060141590 msgr2=0x7fe0601513c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:47.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:47.769+0000 7fe065e00640 1 --2- 172.21.15.16:0/820558661 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe060141590 0x7fe0601513c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:47.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:47.769+0000 7fe065e00640 1 -- 172.21.15.16:0/820558661 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe0600b2c80 con 0x7fe060143c30 2024-03-31T15:03:47.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:47.769+0000 7fe065e00640 1 --2- 172.21.15.16:0/820558661 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe060143c30 0x7fe06013fe50 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fe05800c700 tx=0x7fe05800cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:03:47.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:47.769+0000 7fe05d600640 1 -- 172.21.15.16:0/820558661 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe058018070 con 0x7fe060143c30 2024-03-31T15:03:47.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:47.769+0000 7fe05d600640 1 -- 172.21.15.16:0/820558661 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe05800ce10 con 0x7fe060143c30 2024-03-31T15:03:47.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:47.769+0000 7fe05d600640 1 -- 172.21.15.16:0/820558661 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe05800a880 con 0x7fe060143c30 2024-03-31T15:03:47.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:47.769+0000 7fe067e00640 1 -- 172.21.15.16:0/820558661 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe0600b2f60 con 0x7fe060143c30 2024-03-31T15:03:47.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:47.769+0000 7fe067e00640 1 -- 172.21.15.16:0/820558661 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe0600b3520 con 0x7fe060143c30 2024-03-31T15:03:47.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:47.769+0000 7fe067e00640 1 -- 172.21.15.16:0/820558661 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe060069950 con 0x7fe060143c30 2024-03-31T15:03:47.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:47.773+0000 7fe05d600640 1 -- 172.21.15.16:0/820558661 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fe05801b070 con 0x7fe060143c30 2024-03-31T15:03:47.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:47.773+0000 7fe05d600640 1 --2- 172.21.15.16:0/820558661 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fe040068380 0x7fe04006a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:47.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:47.773+0000 7fe05d600640 1 -- 172.21.15.16:0/820558661 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fe058085f30 con 0x7fe060143c30 2024-03-31T15:03:47.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:47.773+0000 7fe065400640 1 --2- 172.21.15.16:0/820558661 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fe040068380 0x7fe04006a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:47.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:47.773+0000 7fe065400640 1 --2- 172.21.15.16:0/820558661 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fe040068380 0x7fe04006a840 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fe054009b60 tx=0x7fe054009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:03:47.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:47.777+0000 7fe05d600640 1 -- 172.21.15.16:0/820558661 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fe058055e00 con 0x7fe060143c30 2024-03-31T15:03:47.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:47.941+0000 7fe067e00640 1 -- 172.21.15.16:0/820558661 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fe0600b4a20 con 0x7fe060143c30 2024-03-31T15:03:47.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:47.941+0000 7fe05d600640 1 -- 172.21.15.16:0/820558661 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) ==== 100+0+11 (secure 0 0 0) 0x7fe05805a100 con 0x7fe060143c30 2024-03-31T15:03:47.935 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 64 2024-03-31T15:03:47.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:47.945+0000 7fe067e00640 1 -- 172.21.15.16:0/820558661 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fe040068380 msgr2=0x7fe04006a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:47.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:47.945+0000 7fe067e00640 1 --2- 172.21.15.16:0/820558661 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fe040068380 0x7fe04006a840 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fe054009b60 tx=0x7fe054009190 comp rx=0 tx=0).stop 2024-03-31T15:03:47.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:47.945+0000 7fe067e00640 1 -- 172.21.15.16:0/820558661 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe060143c30 msgr2=0x7fe06013fe50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:47.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:47.945+0000 7fe067e00640 1 --2- 172.21.15.16:0/820558661 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe060143c30 0x7fe06013fe50 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fe05800c700 tx=0x7fe05800cbc0 comp rx=0 tx=0).stop 2024-03-31T15:03:47.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:47.945+0000 7fe067e00640 1 -- 172.21.15.16:0/820558661 shutdown_connections 2024-03-31T15:03:47.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:47.945+0000 7fe067e00640 1 --2- 172.21.15.16:0/820558661 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fe040068380 0x7fe04006a840 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:47.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:47.945+0000 7fe067e00640 1 --2- 172.21.15.16:0/820558661 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe060143c30 0x7fe06013fe50 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:47.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:47.945+0000 7fe067e00640 1 --2- 172.21.15.16:0/820558661 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe060141590 0x7fe0601513c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:47.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:47.945+0000 7fe067e00640 1 --2- 172.21.15.16:0/820558661 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fe0601411b0 0x7fe06013c860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:47.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:47.945+0000 7fe067e00640 1 -- 172.21.15.16:0/820558661 >> 172.21.15.16:0/820558661 conn(0x7fe060067930 msgr2=0x7fe0601604f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:03:47.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:47.945+0000 7fe067e00640 1 -- 172.21.15.16:0/820558661 shutdown_connections 2024-03-31T15:03:47.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:47.945+0000 7fe067e00640 1 -- 172.21.15.16:0/820558661 wait complete. 2024-03-31T15:03:47.955 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-31T15:03:48.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:48.101+0000 7f2dbaa00640 1 Processor -- start 2024-03-31T15:03:48.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:48.101+0000 7f2dbaa00640 1 -- start start 2024-03-31T15:03:48.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:48.105+0000 7f2dbaa00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f2db415b3d0 0x7f2db40b1cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:48.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:48.105+0000 7f2dbaa00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f2db415eee0 0x7f2db40b2200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:48.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:48.105+0000 7f2dbaa00640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2db415f3c0 con 0x7f2db40ce2f0 2024-03-31T15:03:48.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:48.105+0000 7f2dbaa00640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f2db415b7b0 con 0x7f2db415b3d0 2024-03-31T15:03:48.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:48.105+0000 7f2dbaa00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f2db415f6a0 con 0x7f2db415eee0 2024-03-31T15:03:48.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:48.105+0000 7f2db3e00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f2db415b3d0 0x7f2db40b1cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:48.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:48.105+0000 7f2db8a00640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f2db40ce2f0 0x7f2db40cc9c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.16:58726/0 (socket says 172.21.15.16:58726) 2024-03-31T15:03:48.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:48.105+0000 7f2db8a00640 1 -- 172.21.15.16:0/686061898 learned_addr learned my addr 172.21.15.16:0/686061898 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:03:48.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:48.105+0000 7f2db3400640 1 --2- 172.21.15.16:0/686061898 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f2db415eee0 0x7f2db40b2200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:48.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:48.105+0000 7f2db3e00640 1 -- 172.21.15.16:0/686061898 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f2db415eee0 msgr2=0x7f2db40b2200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:03:48.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:48.105+0000 7f2db3e00640 1 --2- 172.21.15.16:0/686061898 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f2db415eee0 0x7f2db40b2200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:48.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:48.105+0000 7f2db3e00640 1 -- 172.21.15.16:0/686061898 >> v1:172.21.15.89:6789/0 conn(0x7f2db40ce2f0 legacy=0x7f2db40cc9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:48.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:48.105+0000 7f2db3e00640 1 -- 172.21.15.16:0/686061898 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2db415f980 con 0x7f2db415b3d0 2024-03-31T15:03:48.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:48.109+0000 7f2db3e00640 1 --2- 172.21.15.16:0/686061898 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2db415b3d0 0x7f2db40b1cc0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f2da40097c0 tx=0x7f2da401c470 comp rx=0 tx=0).ready entity=mon.1 client_cookie=921f9ebbeb6077c6 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:03:48.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:48.109+0000 7f2db2a00640 1 -- 172.21.15.16:0/686061898 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2da40ab430 con 0x7f2db415b3d0 2024-03-31T15:03:48.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:48.109+0000 7f2db2a00640 1 -- 172.21.15.16:0/686061898 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2da40b0070 con 0x7f2db415b3d0 2024-03-31T15:03:48.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:48.109+0000 7f2db2a00640 1 -- 172.21.15.16:0/686061898 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2da40ab7f0 con 0x7f2db415b3d0 2024-03-31T15:03:48.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:48.109+0000 7f2dbaa00640 1 -- 172.21.15.16:0/686061898 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2db415b3d0 msgr2=0x7f2db40b1cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:48.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:48.109+0000 7f2dbaa00640 1 --2- 172.21.15.16:0/686061898 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2db415b3d0 0x7f2db40b1cc0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f2da40097c0 tx=0x7f2da401c470 comp rx=0 tx=0).stop 2024-03-31T15:03:48.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:48.109+0000 7f2dbaa00640 1 -- 172.21.15.16:0/686061898 shutdown_connections 2024-03-31T15:03:48.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:48.109+0000 7f2dbaa00640 1 --2- 172.21.15.16:0/686061898 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f2db415eee0 0x7f2db40b2200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:48.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:48.109+0000 7f2dbaa00640 1 --2- 172.21.15.16:0/686061898 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2db415b3d0 0x7f2db40b1cc0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:48.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:48.109+0000 7f2dbaa00640 1 -- 172.21.15.16:0/686061898 >> 172.21.15.16:0/686061898 conn(0x7f2db406b620 msgr2=0x7f2db406ba30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:03:48.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:48.109+0000 7f2dbaa00640 1 -- 172.21.15.16:0/686061898 shutdown_connections 2024-03-31T15:03:48.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:48.109+0000 7f2dbaa00640 1 -- 172.21.15.16:0/686061898 wait complete. 2024-03-31T15:03:48.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:48.109+0000 7f2dbaa00640 1 Processor -- start 2024-03-31T15:03:48.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:48.109+0000 7f2dbaa00640 1 -- start start 2024-03-31T15:03:48.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:48.109+0000 7f2dbaa00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f2db40ce2f0 0x7f2db40b0840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:48.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:48.109+0000 7f2dbaa00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2db415b3d0 0x7f2db41465d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:48.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:48.109+0000 7f2dbaa00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2db415eee0 0x7f2db4158f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:48.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:48.109+0000 7f2dbaa00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f2db4168610 con 0x7f2db415b3d0 2024-03-31T15:03:48.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:48.109+0000 7f2dbaa00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f2db4168490 con 0x7f2db415eee0 2024-03-31T15:03:48.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:48.109+0000 7f2dbaa00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f2db4168790 con 0x7f2db40ce2f0 2024-03-31T15:03:48.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:48.109+0000 7f2db3400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2db415b3d0 0x7f2db41465d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:48.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:48.113+0000 7f2db3400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2db415b3d0 0x7f2db41465d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53892/0 (socket says 172.21.15.16:53892) 2024-03-31T15:03:48.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:48.113+0000 7f2db3400640 1 -- 172.21.15.16:0/4236588773 learned_addr learned my addr 172.21.15.16:0/4236588773 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:03:48.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:48.113+0000 7f2db3e00640 1 --2- 172.21.15.16:0/4236588773 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f2db40ce2f0 0x7f2db40b0840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:48.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:48.113+0000 7f2db8a00640 1 --2- 172.21.15.16:0/4236588773 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2db415eee0 0x7f2db4158f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:48.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:48.113+0000 7f2db3400640 1 -- 172.21.15.16:0/4236588773 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f2db40ce2f0 msgr2=0x7f2db40b0840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:48.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:48.113+0000 7f2db3400640 1 --2- 172.21.15.16:0/4236588773 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f2db40ce2f0 0x7f2db40b0840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:48.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:48.113+0000 7f2db3400640 1 -- 172.21.15.16:0/4236588773 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2db415eee0 msgr2=0x7f2db4158f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:48.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:48.113+0000 7f2db3400640 1 --2- 172.21.15.16:0/4236588773 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2db415eee0 0x7f2db4158f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:48.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:48.113+0000 7f2db3400640 1 -- 172.21.15.16:0/4236588773 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2db40b0e20 con 0x7f2db415b3d0 2024-03-31T15:03:48.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:48.113+0000 7f2db3e00640 1 --2- 172.21.15.16:0/4236588773 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f2db40ce2f0 0x7f2db40b0840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:03:48.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:48.113+0000 7f2db3400640 1 --2- 172.21.15.16:0/4236588773 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2db415b3d0 0x7f2db41465d0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f2da000e930 tx=0x7f2da000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:03:48.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:48.113+0000 7f2dabe00640 1 -- 172.21.15.16:0/4236588773 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2da0017070 con 0x7f2db415b3d0 2024-03-31T15:03:48.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:48.113+0000 7f2dabe00640 1 -- 172.21.15.16:0/4236588773 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2da000ca00 con 0x7f2db415b3d0 2024-03-31T15:03:48.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:48.113+0000 7f2dbaa00640 1 -- 172.21.15.16:0/4236588773 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2db41599c0 con 0x7f2db415b3d0 2024-03-31T15:03:48.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:48.113+0000 7f2dabe00640 1 -- 172.21.15.16:0/4236588773 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2da000cdb0 con 0x7f2db415b3d0 2024-03-31T15:03:48.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:48.113+0000 7f2dbaa00640 1 -- 172.21.15.16:0/4236588773 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2db4159f80 con 0x7f2db415b3d0 2024-03-31T15:03:48.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:48.113+0000 7f2dbaa00640 1 -- 172.21.15.16:0/4236588773 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2db4069950 con 0x7f2db415b3d0 2024-03-31T15:03:48.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:48.113+0000 7f2dabe00640 1 -- 172.21.15.16:0/4236588773 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f2da001a070 con 0x7f2db415b3d0 2024-03-31T15:03:48.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:48.113+0000 7f2dabe00640 1 --2- 172.21.15.16:0/4236588773 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f2d94068380 0x7f2d9406a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:48.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:48.113+0000 7f2dabe00640 1 -- 172.21.15.16:0/4236588773 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f2da0086d80 con 0x7f2db415b3d0 2024-03-31T15:03:48.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:48.117+0000 7f2db3e00640 1 --2- 172.21.15.16:0/4236588773 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f2d94068380 0x7f2d9406a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:48.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:48.117+0000 7f2db3e00640 1 --2- 172.21.15.16:0/4236588773 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f2d94068380 0x7f2d9406a840 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f2da40af000 tx=0x7f2da4008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:03:48.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:48.117+0000 7f2dabe00640 1 -- 172.21.15.16:0/4236588773 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f2da0056d10 con 0x7f2db415b3d0 2024-03-31T15:03:48.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:48.317+0000 7f2dbaa00640 1 -- 172.21.15.16:0/4236588773 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2db40cdea0 con 0x7f2db415b3d0 2024-03-31T15:03:48.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:48.317+0000 7f2dabe00640 1 -- 172.21.15.16:0/4236588773 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+5172 (secure 0 0 0) 0x7f2da005b010 con 0x7f2db415b3d0 2024-03-31T15:03:48.310 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:03:48.310 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4276,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/2832752118","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6833","nonce":2832752118},{"type":"v1","addr":"172.21.15.16:6835","nonce":2832752118}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4278,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6834/153294668","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":153294668},{"type":"v1","addr":"172.21.15.16:6834","nonce":153294668}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4288,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/1653319357","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":1653319357},{"type":"v1","addr":"172.21.15.89:6835","nonce":1653319357}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-31T15:03:36.349865+0000","modified":"2024-03-31T15:03:39.378181+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4293},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.89:6837/2583453131","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":2583453131},{"type":"v1","addr":"172.21.15.89:6837","nonce":2583453131}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4293,"qdb_cluster":[4293]},"id":1}]} 2024-03-31T15:03:48.310 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 8 2024-03-31T15:03:48.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:48.317+0000 7f2dbaa00640 1 -- 172.21.15.16:0/4236588773 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f2d94068380 msgr2=0x7f2d9406a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:48.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:48.317+0000 7f2dbaa00640 1 --2- 172.21.15.16:0/4236588773 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f2d94068380 0x7f2d9406a840 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f2da40af000 tx=0x7f2da4008040 comp rx=0 tx=0).stop 2024-03-31T15:03:48.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:48.317+0000 7f2dbaa00640 1 -- 172.21.15.16:0/4236588773 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2db415b3d0 msgr2=0x7f2db41465d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:48.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:48.317+0000 7f2dbaa00640 1 --2- 172.21.15.16:0/4236588773 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2db415b3d0 0x7f2db41465d0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f2da000e930 tx=0x7f2da000edf0 comp rx=0 tx=0).stop 2024-03-31T15:03:48.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:48.317+0000 7f2dbaa00640 1 -- 172.21.15.16:0/4236588773 shutdown_connections 2024-03-31T15:03:48.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:48.317+0000 7f2dbaa00640 1 --2- 172.21.15.16:0/4236588773 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f2d94068380 0x7f2d9406a840 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:48.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:48.317+0000 7f2dbaa00640 1 --2- 172.21.15.16:0/4236588773 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2db415eee0 0x7f2db4158f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:48.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:48.317+0000 7f2dbaa00640 1 --2- 172.21.15.16:0/4236588773 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2db415b3d0 0x7f2db41465d0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:48.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:48.317+0000 7f2dbaa00640 1 --2- 172.21.15.16:0/4236588773 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f2db40ce2f0 0x7f2db40b0840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:48.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:48.317+0000 7f2dbaa00640 1 -- 172.21.15.16:0/4236588773 >> 172.21.15.16:0/4236588773 conn(0x7f2db406b620 msgr2=0x7f2db40ce9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:03:48.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:48.317+0000 7f2dbaa00640 1 -- 172.21.15.16:0/4236588773 shutdown_connections 2024-03-31T15:03:48.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:48.317+0000 7f2dbaa00640 1 -- 172.21.15.16:0/4236588773 wait complete. 2024-03-31T15:03:48.330 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-31T15:03:48.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:48.477+0000 7f4533a00640 1 Processor -- start 2024-03-31T15:03:48.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:48.477+0000 7f4533a00640 1 -- start start 2024-03-31T15:03:48.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:48.481+0000 7f4533a00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f452c145d10 0x7f452c0cf790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:48.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:48.481+0000 7f4533a00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f452c0b23d0 0x7f452c15cd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:48.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:48.481+0000 7f4533a00640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f452c160e70 con 0x7f452c0728b0 2024-03-31T15:03:48.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:48.481+0000 7f4533a00640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f452c1602f0 con 0x7f452c145d10 2024-03-31T15:03:48.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:48.481+0000 7f4533a00640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f452c1605d0 con 0x7f452c0b23d0 2024-03-31T15:03:48.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:48.481+0000 7f4531000640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f452c145d10 0x7f452c0cf790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:48.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:48.481+0000 7f4531000640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f452c145d10 0x7f452c0cf790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53902/0 (socket says 172.21.15.16:53902) 2024-03-31T15:03:48.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:48.481+0000 7f4531000640 1 -- 172.21.15.16:0/2379824222 learned_addr learned my addr 172.21.15.16:0/2379824222 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:03:48.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:48.481+0000 7f4531a00640 1 --2- 172.21.15.16:0/2379824222 >> v2:172.21.15.89:3300/0 conn(0x7f452c0b23d0 0x7f452c15cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:48.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:48.481+0000 7f452b400640 1 -- 172.21.15.16:0/2379824222 <== mon.1 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 806125705 0 0) 0x7f452c160e70 con 0x7f452c0728b0 2024-03-31T15:03:48.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:48.481+0000 7f4531000640 1 -- 172.21.15.16:0/2379824222 >> v2:172.21.15.89:3300/0 conn(0x7f452c0b23d0 msgr2=0x7f452c15cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:03:48.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:48.481+0000 7f4531000640 1 --2- 172.21.15.16:0/2379824222 >> v2:172.21.15.89:3300/0 conn(0x7f452c0b23d0 0x7f452c15cd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:48.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:48.481+0000 7f4531000640 1 -- 172.21.15.16:0/2379824222 >> v1:172.21.15.89:6789/0 conn(0x7f452c0728b0 legacy=0x7f452c0b1cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:48.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:48.481+0000 7f4531000640 1 -- 172.21.15.16:0/2379824222 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f452c0cfcd0 con 0x7f452c145d10 2024-03-31T15:03:48.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:48.481+0000 7f4531000640 1 --2- 172.21.15.16:0/2379824222 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f452c145d10 0x7f452c0cf790 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f452c071590 tx=0x7f452001c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=377d1dff1084e4ed server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:03:48.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:48.481+0000 7f452b400640 1 -- 172.21.15.16:0/2379824222 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45200b0070 con 0x7f452c145d10 2024-03-31T15:03:48.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:48.485+0000 7f452b400640 1 -- 172.21.15.16:0/2379824222 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f45200ab600 con 0x7f452c145d10 2024-03-31T15:03:48.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:48.485+0000 7f452b400640 1 -- 172.21.15.16:0/2379824222 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45200ab9d0 con 0x7f452c145d10 2024-03-31T15:03:48.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:48.485+0000 7f4533a00640 1 -- 172.21.15.16:0/2379824222 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f452c145d10 msgr2=0x7f452c0cf790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:48.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:48.485+0000 7f4533a00640 1 --2- 172.21.15.16:0/2379824222 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f452c145d10 0x7f452c0cf790 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f452c071590 tx=0x7f452001c520 comp rx=0 tx=0).stop 2024-03-31T15:03:48.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:48.485+0000 7f4533a00640 1 -- 172.21.15.16:0/2379824222 shutdown_connections 2024-03-31T15:03:48.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:48.485+0000 7f4533a00640 1 --2- 172.21.15.16:0/2379824222 >> v2:172.21.15.89:3300/0 conn(0x7f452c0b23d0 0x7f452c15cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:48.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:48.485+0000 7f4533a00640 1 --2- 172.21.15.16:0/2379824222 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f452c145d10 0x7f452c0cf790 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:48.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:48.485+0000 7f4533a00640 1 -- 172.21.15.16:0/2379824222 >> 172.21.15.16:0/2379824222 conn(0x7f452c06b620 msgr2=0x7f452c06ba30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:03:48.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:48.485+0000 7f4533a00640 1 -- 172.21.15.16:0/2379824222 shutdown_connections 2024-03-31T15:03:48.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:48.485+0000 7f4533a00640 1 -- 172.21.15.16:0/2379824222 wait complete. 2024-03-31T15:03:48.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:48.485+0000 7f4533a00640 1 Processor -- start 2024-03-31T15:03:48.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:48.485+0000 7f4533a00640 1 -- start start 2024-03-31T15:03:48.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:48.485+0000 7f4533a00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f452c0728b0 0x7f452c13b8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:48.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:48.485+0000 7f4533a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f452c0b23d0 0x7f452c1f5280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:48.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:48.485+0000 7f4533a00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f452c145d10 0x7f452c1f57c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:48.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:48.485+0000 7f4533a00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f452c0cf4d0 con 0x7f452c0b23d0 2024-03-31T15:03:48.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:48.485+0000 7f4533a00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f452c0cf350 con 0x7f452c145d10 2024-03-31T15:03:48.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:48.485+0000 7f4533a00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f452c072d00 con 0x7f452c0728b0 2024-03-31T15:03:48.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:48.485+0000 7f452be00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f452c0b23d0 0x7f452c1f5280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:48.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:48.485+0000 7f452be00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f452c0b23d0 0x7f452c1f5280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53914/0 (socket says 172.21.15.16:53914) 2024-03-31T15:03:48.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:48.485+0000 7f452be00640 1 -- 172.21.15.16:0/3694092219 learned_addr learned my addr 172.21.15.16:0/3694092219 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:03:48.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:48.485+0000 7f4531a00640 1 --2- 172.21.15.16:0/3694092219 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f452c145d10 0x7f452c1f57c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:48.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:48.485+0000 7f4531000640 1 --2- 172.21.15.16:0/3694092219 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f452c0728b0 0x7f452c13b8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:48.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:48.485+0000 7f452be00640 1 -- 172.21.15.16:0/3694092219 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f452c0728b0 msgr2=0x7f452c13b8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:48.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:48.485+0000 7f452be00640 1 --2- 172.21.15.16:0/3694092219 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f452c0728b0 0x7f452c13b8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:48.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:48.485+0000 7f452be00640 1 -- 172.21.15.16:0/3694092219 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f452c145d10 msgr2=0x7f452c1f57c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:48.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:48.485+0000 7f452be00640 1 --2- 172.21.15.16:0/3694092219 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f452c145d10 0x7f452c1f57c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:48.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:48.485+0000 7f452be00640 1 -- 172.21.15.16:0/3694092219 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f452c1f5e00 con 0x7f452c0b23d0 2024-03-31T15:03:48.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:48.489+0000 7f452be00640 1 --2- 172.21.15.16:0/3694092219 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f452c0b23d0 0x7f452c1f5280 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f451800c4d0 tx=0x7f451800c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:03:48.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:48.489+0000 7f4528c00640 1 -- 172.21.15.16:0/3694092219 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4518017070 con 0x7f452c0b23d0 2024-03-31T15:03:48.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:48.489+0000 7f4528c00640 1 -- 172.21.15.16:0/3694092219 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4518005480 con 0x7f452c0b23d0 2024-03-31T15:03:48.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:48.489+0000 7f4533a00640 1 -- 172.21.15.16:0/3694092219 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f452c1f60e0 con 0x7f452c0b23d0 2024-03-31T15:03:48.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:48.489+0000 7f4528c00640 1 -- 172.21.15.16:0/3694092219 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4518003840 con 0x7f452c0b23d0 2024-03-31T15:03:48.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:48.489+0000 7f4533a00640 1 -- 172.21.15.16:0/3694092219 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f452c1f6980 con 0x7f452c0b23d0 2024-03-31T15:03:48.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:48.489+0000 7f4533a00640 1 -- 172.21.15.16:0/3694092219 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f452c1f64d0 con 0x7f452c0b23d0 2024-03-31T15:03:48.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:48.489+0000 7f4528c00640 1 -- 172.21.15.16:0/3694092219 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f451801a070 con 0x7f452c0b23d0 2024-03-31T15:03:48.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:48.489+0000 7f4528c00640 1 --2- 172.21.15.16:0/3694092219 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f450c068380 0x7f450c06a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:48.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:48.489+0000 7f4528c00640 1 -- 172.21.15.16:0/3694092219 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f4518085e10 con 0x7f452c0b23d0 2024-03-31T15:03:48.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:48.493+0000 7f4531000640 1 --2- 172.21.15.16:0/3694092219 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f450c068380 0x7f450c06a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:48.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:48.493+0000 7f4531000640 1 --2- 172.21.15.16:0/3694092219 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f450c068380 0x7f450c06a840 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f45200ad000 tx=0x7f4520008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:03:48.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:48.493+0000 7f4528c00640 1 -- 172.21.15.16:0/3694092219 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f4518089030 con 0x7f452c0b23d0 2024-03-31T15:03:48.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:48.661+0000 7f4533a00640 1 -- 172.21.15.16:0/3694092219 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f452c069950 con 0x7f452c0b23d0 2024-03-31T15:03:48.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:48.665+0000 7f4528c00640 1 -- 172.21.15.16:0/3694092219 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) ==== 74+0+16533 (secure 0 0 0) 0x7f4518055da0 con 0x7f452c0b23d0 2024-03-31T15:03:48.657 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:03:48.657 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":21,"fsid":"286eb5da-fb47-4550-b290-d9519c13b9a5","created":"2024-03-31T15:03:11.438952+0000","modified":"2024-03-31T15:03:37.957829+0000","last_up_change":"2024-03-31T15:03:22.885535+0000","last_in_change":"2024-03-31T15:03:13.941667+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-31T15:03:22.550787+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-31T15:03:34.488425+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-31T15:03:35.299116+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"92a66450-ef19-40d8-ac3d-d760b23b0bda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6801","nonce":932712244}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6803","nonce":932712244}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6807","nonce":932712244}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6805","nonce":932712244}]},"public_addr":"172.21.15.16:6801/932712244","cluster_addr":"172.21.15.16:6803/932712244","heartbeat_back_addr":"172.21.15.16:6807/932712244","heartbeat_front_addr":"172.21.15.16:6805/932712244","state":["exists","up"]},{"osd":1,"uuid":"da9d1f2b-1968-4959-b843-faa231988e92","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6810","nonce":1398737533}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6811","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6813","nonce":1398737533}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6821","nonce":1398737533}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6815","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6816","nonce":1398737533}]},"public_addr":"172.21.15.16:6810/1398737533","cluster_addr":"172.21.15.16:6813/1398737533","heartbeat_back_addr":"172.21.15.16:6821/1398737533","heartbeat_front_addr":"172.21.15.16:6816/1398737533","state":["exists","up"]},{"osd":2,"uuid":"01a73662-9a21-4b1f-bcbb-42b9636a4638","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6819","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6822","nonce":3018526646}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6825","nonce":3018526646}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6831","nonce":3018526646}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6827","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6829","nonce":3018526646}]},"public_addr":"172.21.15.16:6822/3018526646","cluster_addr":"172.21.15.16:6825/3018526646","heartbeat_back_addr":"172.21.15.16:6831/3018526646","heartbeat_front_addr":"172.21.15.16:6829/3018526646","state":["exists","up"]},{"osd":3,"uuid":"c7771fd3-e18e-4500-ac38-23cf10ebcd75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6809","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6812","nonce":3360352974}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6817","nonce":3360352974}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6828","nonce":3360352974}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6823","nonce":3360352974}]},"public_addr":"172.21.15.16:6812/3360352974","cluster_addr":"172.21.15.16:6817/3360352974","heartbeat_back_addr":"172.21.15.16:6828/3360352974","heartbeat_front_addr":"172.21.15.16:6823/3360352974","state":["exists","up"]},{"osd":4,"uuid":"88260730-1482-4fbd-a40f-cb1124e9df4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6801","nonce":1856844856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6803","nonce":1856844856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6807","nonce":1856844856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6805","nonce":1856844856}]},"public_addr":"172.21.15.89:6801/1856844856","cluster_addr":"172.21.15.89:6803/1856844856","heartbeat_back_addr":"172.21.15.89:6807/1856844856","heartbeat_front_addr":"172.21.15.89:6805/1856844856","state":["exists","up"]},{"osd":5,"uuid":"4cb2c00c-fae0-4dd8-9d27-ec4f0093ccf7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6814","nonce":50395527}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6817","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6820","nonce":50395527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6831","nonce":50395527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6823","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6826","nonce":50395527}]},"public_addr":"172.21.15.89:6814/50395527","cluster_addr":"172.21.15.89:6820/50395527","heartbeat_back_addr":"172.21.15.89:6831/50395527","heartbeat_front_addr":"172.21.15.89:6826/50395527","state":["exists","up"]},{"osd":6,"uuid":"81ce0c19-2554-4cae-acbe-66a05e9e9b9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6809","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6812","nonce":3470471752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6815","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6818","nonce":3470471752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6827","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6829","nonce":3470471752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6821","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6824","nonce":3470471752}]},"public_addr":"172.21.15.89:6812/3470471752","cluster_addr":"172.21.15.89:6818/3470471752","heartbeat_back_addr":"172.21.15.89:6829/3470471752","heartbeat_front_addr":"172.21.15.89:6824/3470471752","state":["exists","up"]},{"osd":7,"uuid":"85e4313b-d37d-442e-8042-153bf20a847d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6811","nonce":1796778630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6813","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6816","nonce":1796778630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6825","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6828","nonce":1796778630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6819","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6822","nonce":1796778630}]},"public_addr":"172.21.15.89:6811/1796778630","cluster_addr":"172.21.15.89:6816/1796778630","heartbeat_back_addr":"172.21.15.89:6828/1796778630","heartbeat_front_addr":"172.21.15.89:6822/1796778630","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.439088+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.529970+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.680512+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:21.044161+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.582111+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.536558+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.409522+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.590229+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-31T15:03:48.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:48.665+0000 7f4533a00640 1 -- 172.21.15.16:0/3694092219 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f450c068380 msgr2=0x7f450c06a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:48.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:48.665+0000 7f4533a00640 1 --2- 172.21.15.16:0/3694092219 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f450c068380 0x7f450c06a840 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f45200ad000 tx=0x7f4520008040 comp rx=0 tx=0).stop 2024-03-31T15:03:48.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:48.665+0000 7f4533a00640 1 -- 172.21.15.16:0/3694092219 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f452c0b23d0 msgr2=0x7f452c1f5280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:48.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:48.665+0000 7f4533a00640 1 --2- 172.21.15.16:0/3694092219 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f452c0b23d0 0x7f452c1f5280 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f451800c4d0 tx=0x7f451800c990 comp rx=0 tx=0).stop 2024-03-31T15:03:48.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:48.665+0000 7f4533a00640 1 -- 172.21.15.16:0/3694092219 shutdown_connections 2024-03-31T15:03:48.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:48.665+0000 7f4533a00640 1 --2- 172.21.15.16:0/3694092219 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f450c068380 0x7f450c06a840 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:48.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:48.665+0000 7f4533a00640 1 --2- 172.21.15.16:0/3694092219 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f452c145d10 0x7f452c1f57c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:48.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:48.665+0000 7f4533a00640 1 --2- 172.21.15.16:0/3694092219 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f452c0b23d0 0x7f452c1f5280 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:48.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:48.665+0000 7f4533a00640 1 --2- 172.21.15.16:0/3694092219 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f452c0728b0 0x7f452c13b8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:48.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:48.665+0000 7f4533a00640 1 -- 172.21.15.16:0/3694092219 >> 172.21.15.16:0/3694092219 conn(0x7f452c06b620 msgr2=0x7f452c0afba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:03:48.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:48.665+0000 7f4533a00640 1 -- 172.21.15.16:0/3694092219 shutdown_connections 2024-03-31T15:03:48.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:48.665+0000 7f4533a00640 1 -- 172.21.15.16:0/3694092219 wait complete. 2024-03-31T15:03:48.678 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-31T15:03:22.550787+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-31T15:03:34.488425+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-31T15:03:35.299116+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-31T15:03:48.679 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-31T15:03:48.822 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:48.829+0000 7f315c200640 1 Processor -- start 2024-03-31T15:03:48.822 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:48.829+0000 7f315c200640 1 -- start start 2024-03-31T15:03:48.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:48.829+0000 7f315c200640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f31541411b0 0x7f315406f920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:48.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:48.829+0000 7f315c200640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f3154144340 0x7f315406fe60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:48.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:48.829+0000 7f315c200640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3154144980 con 0x7f3154141590 2024-03-31T15:03:48.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:48.829+0000 7f315c200640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f315415f910 con 0x7f31541411b0 2024-03-31T15:03:48.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:48.829+0000 7f315c200640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f3154162910 con 0x7f3154144340 2024-03-31T15:03:48.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:48.829+0000 7f3159800640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f31541411b0 0x7f315406f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:48.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:48.829+0000 7f3159800640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f31541411b0 0x7f315406f920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53922/0 (socket says 172.21.15.16:53922) 2024-03-31T15:03:48.825 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:48.829+0000 7f3159800640 1 -- 172.21.15.16:0/1828115544 learned_addr learned my addr 172.21.15.16:0/1828115544 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:03:48.825 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:48.829+0000 7f315a200640 1 --2- 172.21.15.16:0/1828115544 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f3154144340 0x7f315406fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:48.825 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:48.829+0000 7f3159800640 1 -- 172.21.15.16:0/1828115544 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f3154144340 msgr2=0x7f315406fe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:03:48.825 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:48.829+0000 7f3159800640 1 --2- 172.21.15.16:0/1828115544 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f3154144340 0x7f315406fe60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:48.825 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:48.829+0000 7f3159800640 1 -- 172.21.15.16:0/1828115544 >> v1:172.21.15.89:6789/0 conn(0x7f3154141590 legacy=0x7f3154143c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:48.825 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:48.829+0000 7f3159800640 1 -- 172.21.15.16:0/1828115544 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3154161fe0 con 0x7f31541411b0 2024-03-31T15:03:48.825 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:48.829+0000 7f3153e00640 1 -- 172.21.15.16:0/1828115544 <== mon.1 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2164798700 0 0) 0x7f3154144980 con 0x7f3154141590 2024-03-31T15:03:48.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:48.833+0000 7f3159800640 1 --2- 172.21.15.16:0/1828115544 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f31541411b0 0x7f315406f920 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f3148009870 tx=0x7f314801c560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12faa5413f049e8e server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:03:48.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:48.833+0000 7f3153e00640 1 -- 172.21.15.16:0/1828115544 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31480ab650 con 0x7f31541411b0 2024-03-31T15:03:48.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:48.833+0000 7f3153e00640 1 -- 172.21.15.16:0/1828115544 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f31480b0070 con 0x7f31541411b0 2024-03-31T15:03:48.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:48.833+0000 7f3153e00640 1 -- 172.21.15.16:0/1828115544 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31480aba10 con 0x7f31541411b0 2024-03-31T15:03:48.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:48.833+0000 7f315c200640 1 -- 172.21.15.16:0/1828115544 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f31541411b0 msgr2=0x7f315406f920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:48.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:48.833+0000 7f315c200640 1 --2- 172.21.15.16:0/1828115544 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f31541411b0 0x7f315406f920 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f3148009870 tx=0x7f314801c560 comp rx=0 tx=0).stop 2024-03-31T15:03:48.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:48.833+0000 7f315c200640 1 -- 172.21.15.16:0/1828115544 shutdown_connections 2024-03-31T15:03:48.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:48.833+0000 7f315c200640 1 --2- 172.21.15.16:0/1828115544 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f3154144340 0x7f315406fe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:48.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:48.833+0000 7f315c200640 1 --2- 172.21.15.16:0/1828115544 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f31541411b0 0x7f315406f920 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:48.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:48.833+0000 7f315c200640 1 -- 172.21.15.16:0/1828115544 >> 172.21.15.16:0/1828115544 conn(0x7f3154067930 msgr2=0x7f31540b3ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:03:48.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:48.833+0000 7f315c200640 1 -- 172.21.15.16:0/1828115544 shutdown_connections 2024-03-31T15:03:48.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:48.833+0000 7f315c200640 1 -- 172.21.15.16:0/1828115544 wait complete. 2024-03-31T15:03:48.829 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:48.837+0000 7f315c200640 1 Processor -- start 2024-03-31T15:03:48.829 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:48.837+0000 7f315c200640 1 -- start start 2024-03-31T15:03:48.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:48.837+0000 7f315c200640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f31541411b0 0x7f315413de50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:48.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:48.837+0000 7f315c200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3154141590 0x7f3154160790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:48.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:48.837+0000 7f315c200640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3154144340 0x7f3154160cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:48.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:48.837+0000 7f315c200640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f315415f290 con 0x7f3154141590 2024-03-31T15:03:48.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:48.837+0000 7f315c200640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f315415f110 con 0x7f3154144340 2024-03-31T15:03:48.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:48.837+0000 7f315c200640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f315415f410 con 0x7f31541411b0 2024-03-31T15:03:48.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:48.837+0000 7f3158e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3154141590 0x7f3154160790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:48.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:48.837+0000 7f3158e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3154141590 0x7f3154160790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53930/0 (socket says 172.21.15.16:53930) 2024-03-31T15:03:48.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:48.837+0000 7f3158e00640 1 -- 172.21.15.16:0/2829855657 learned_addr learned my addr 172.21.15.16:0/2829855657 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:03:48.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:48.837+0000 7f3159800640 1 --2- 172.21.15.16:0/2829855657 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f31541411b0 0x7f315413de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:48.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:48.837+0000 7f315a200640 1 --2- 172.21.15.16:0/2829855657 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3154144340 0x7f3154160cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:48.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:48.837+0000 7f3158e00640 1 -- 172.21.15.16:0/2829855657 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f31541411b0 msgr2=0x7f315413de50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:48.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:48.837+0000 7f3158e00640 1 --2- 172.21.15.16:0/2829855657 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f31541411b0 0x7f315413de50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:48.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:48.837+0000 7f3158e00640 1 -- 172.21.15.16:0/2829855657 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3154144340 msgr2=0x7f3154160cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:48.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:48.837+0000 7f3158e00640 1 --2- 172.21.15.16:0/2829855657 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3154144340 0x7f3154160cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:48.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:48.837+0000 7f3158e00640 1 -- 172.21.15.16:0/2829855657 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f315406dec0 con 0x7f3154141590 2024-03-31T15:03:48.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:48.837+0000 7f3158e00640 1 --2- 172.21.15.16:0/2829855657 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3154141590 0x7f3154160790 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f314000c520 tx=0x7f314000c9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:03:48.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:48.837+0000 7f3151600640 1 -- 172.21.15.16:0/2829855657 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3140017070 con 0x7f3154141590 2024-03-31T15:03:48.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:48.837+0000 7f3151600640 1 -- 172.21.15.16:0/2829855657 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3140003600 con 0x7f3154141590 2024-03-31T15:03:48.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:48.837+0000 7f3151600640 1 -- 172.21.15.16:0/2829855657 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31400039f0 con 0x7f3154141590 2024-03-31T15:03:48.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:48.837+0000 7f315c200640 1 -- 172.21.15.16:0/2829855657 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f315406e1a0 con 0x7f3154141590 2024-03-31T15:03:48.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:48.837+0000 7f315c200640 1 -- 172.21.15.16:0/2829855657 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f31541f7230 con 0x7f3154141590 2024-03-31T15:03:48.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:48.841+0000 7f315c200640 1 -- 172.21.15.16:0/2829855657 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3154069950 con 0x7f3154141590 2024-03-31T15:03:48.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:48.841+0000 7f3151600640 1 -- 172.21.15.16:0/2829855657 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f314001a070 con 0x7f3154141590 2024-03-31T15:03:48.835 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:48.841+0000 7f3151600640 1 --2- 172.21.15.16:0/2829855657 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f3134068380 0x7f313406a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:48.835 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:48.841+0000 7f3151600640 1 -- 172.21.15.16:0/2829855657 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f3140085d10 con 0x7f3154141590 2024-03-31T15:03:48.835 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:48.841+0000 7f3159800640 1 --2- 172.21.15.16:0/2829855657 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f3134068380 0x7f313406a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:48.835 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:48.841+0000 7f3159800640 1 --2- 172.21.15.16:0/2829855657 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f3134068380 0x7f313406a840 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f31480af000 tx=0x7f31480a7000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:03:48.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:48.845+0000 7f3151600640 1 -- 172.21.15.16:0/2829855657 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f3140055c70 con 0x7f3154141590 2024-03-31T15:03:49.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:49.009+0000 7f315c200640 1 -- 172.21.15.16:0/2829855657 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f315406cd40 con 0x7f3154141590 2024-03-31T15:03:49.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:49.009+0000 7f3151600640 1 -- 172.21.15.16:0/2829855657 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) ==== 93+0+10 (secure 0 0 0) 0x7f3140059f70 con 0x7f3154141590 2024-03-31T15:03:49.003 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2024-03-31T15:03:49.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:49.013+0000 7f315c200640 1 -- 172.21.15.16:0/2829855657 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f3134068380 msgr2=0x7f313406a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:49.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:49.013+0000 7f315c200640 1 --2- 172.21.15.16:0/2829855657 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f3134068380 0x7f313406a840 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f31480af000 tx=0x7f31480a7000 comp rx=0 tx=0).stop 2024-03-31T15:03:49.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:49.013+0000 7f315c200640 1 -- 172.21.15.16:0/2829855657 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3154141590 msgr2=0x7f3154160790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:49.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:49.013+0000 7f315c200640 1 --2- 172.21.15.16:0/2829855657 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3154141590 0x7f3154160790 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f314000c520 tx=0x7f314000c9e0 comp rx=0 tx=0).stop 2024-03-31T15:03:49.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:49.013+0000 7f315c200640 1 -- 172.21.15.16:0/2829855657 shutdown_connections 2024-03-31T15:03:49.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:49.013+0000 7f315c200640 1 --2- 172.21.15.16:0/2829855657 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f3134068380 0x7f313406a840 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:49.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:49.013+0000 7f315c200640 1 --2- 172.21.15.16:0/2829855657 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3154144340 0x7f3154160cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:49.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:49.013+0000 7f315c200640 1 --2- 172.21.15.16:0/2829855657 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3154141590 0x7f3154160790 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:49.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:49.013+0000 7f315c200640 1 --2- 172.21.15.16:0/2829855657 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f31541411b0 0x7f315413de50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:49.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:49.013+0000 7f315c200640 1 -- 172.21.15.16:0/2829855657 >> 172.21.15.16:0/2829855657 conn(0x7f3154067930 msgr2=0x7f3154141bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:03:49.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:49.013+0000 7f315c200640 1 -- 172.21.15.16:0/2829855657 shutdown_connections 2024-03-31T15:03:49.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:49.013+0000 7f315c200640 1 -- 172.21.15.16:0/2829855657 wait complete. 2024-03-31T15:03:49.025 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-31T15:03:49.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:49.173+0000 7f1e20a00640 1 Processor -- start 2024-03-31T15:03:49.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:49.173+0000 7f1e20a00640 1 -- start start 2024-03-31T15:03:49.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:49.177+0000 7f1e20a00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f1e1c13dff0 0x7f1e1c06d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:49.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:49.177+0000 7f1e20a00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f1e1c0cd0e0 0x7f1e1c06da10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:49.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:49.177+0000 7f1e20a00640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1e1c15fbc0 con 0x7f1e1c13e3d0 2024-03-31T15:03:49.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:49.177+0000 7f1e20a00640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f1e1c15f730 con 0x7f1e1c0cd0e0 2024-03-31T15:03:49.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:49.177+0000 7f1e20a00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f1e1c15ee00 con 0x7f1e1c13dff0 2024-03-31T15:03:49.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:49.177+0000 7f1e1a800640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f1e1c0cd0e0 0x7f1e1c06da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:49.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:49.177+0000 7f1e19e00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f1e1c13dff0 0x7f1e1c06d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:49.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:49.177+0000 7f1e1a800640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f1e1c0cd0e0 0x7f1e1c06da10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53934/0 (socket says 172.21.15.16:53934) 2024-03-31T15:03:49.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:49.177+0000 7f1e1a800640 1 -- 172.21.15.16:0/1327246574 learned_addr learned my addr 172.21.15.16:0/1327246574 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:03:49.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:49.177+0000 7f1e19e00640 1 -- 172.21.15.16:0/1327246574 >> v2:172.21.15.16:3300/0 conn(0x7f1e1c0cd0e0 msgr2=0x7f1e1c06da10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:03:49.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:49.177+0000 7f1e19e00640 1 --2- 172.21.15.16:0/1327246574 >> v2:172.21.15.16:3300/0 conn(0x7f1e1c0cd0e0 0x7f1e1c06da10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:49.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:49.177+0000 7f1e19e00640 1 -- 172.21.15.16:0/1327246574 >> v1:172.21.15.16:6789/0 conn(0x7f1e1c13e3d0 legacy=0x7f1e1c0cc9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:49.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:49.177+0000 7f1e19e00640 1 -- 172.21.15.16:0/1327246574 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e1c15f0e0 con 0x7f1e1c13dff0 2024-03-31T15:03:49.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:49.181+0000 7f1e19e00640 1 --2- 172.21.15.16:0/1327246574 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f1e1c13dff0 0x7f1e1c06d4d0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f1e0c009d90 tx=0x7f1e0c01c8e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3716f8d92dfbf72e server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:03:49.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:49.181+0000 7f1e19400640 1 -- 172.21.15.16:0/1327246574 <== mon.2 v2:172.21.15.89:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e0c0b2070 con 0x7f1e1c13dff0 2024-03-31T15:03:49.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:49.181+0000 7f1e19400640 1 -- 172.21.15.16:0/1327246574 <== mon.2 v2:172.21.15.89:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1e0c0a57a0 con 0x7f1e1c13dff0 2024-03-31T15:03:49.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:49.181+0000 7f1e19400640 1 -- 172.21.15.16:0/1327246574 <== mon.2 v2:172.21.15.89:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e0c0a5b70 con 0x7f1e1c13dff0 2024-03-31T15:03:49.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:49.181+0000 7f1e20a00640 1 -- 172.21.15.16:0/1327246574 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f1e1c13dff0 msgr2=0x7f1e1c06d4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:49.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:49.181+0000 7f1e20a00640 1 --2- 172.21.15.16:0/1327246574 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f1e1c13dff0 0x7f1e1c06d4d0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f1e0c009d90 tx=0x7f1e0c01c8e0 comp rx=0 tx=0).stop 2024-03-31T15:03:49.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:49.181+0000 7f1e20a00640 1 -- 172.21.15.16:0/1327246574 shutdown_connections 2024-03-31T15:03:49.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:49.181+0000 7f1e20a00640 1 --2- 172.21.15.16:0/1327246574 >> v2:172.21.15.16:3300/0 conn(0x7f1e1c0cd0e0 0x7f1e1c06da10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:49.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:49.181+0000 7f1e20a00640 1 --2- 172.21.15.16:0/1327246574 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f1e1c13dff0 0x7f1e1c06d4d0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:49.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:49.181+0000 7f1e20a00640 1 -- 172.21.15.16:0/1327246574 >> 172.21.15.16:0/1327246574 conn(0x7f1e1c06ee40 msgr2=0x7f1e1c06b620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:03:49.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:49.181+0000 7f1e20a00640 1 -- 172.21.15.16:0/1327246574 shutdown_connections 2024-03-31T15:03:49.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:49.181+0000 7f1e20a00640 1 -- 172.21.15.16:0/1327246574 wait complete. 2024-03-31T15:03:49.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:49.181+0000 7f1e20a00640 1 Processor -- start 2024-03-31T15:03:49.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:49.181+0000 7f1e20a00640 1 -- start start 2024-03-31T15:03:49.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:49.185+0000 7f1e20a00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1e1c0cd0e0 0x7f1e1c154930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:49.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:49.185+0000 7f1e20a00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f1e1c13dff0 0x7f1e1c13edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:49.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:49.185+0000 7f1e20a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1e1c13e3d0 0x7f1e1c13bff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:49.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:49.185+0000 7f1e20a00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f1e1c0cf710 con 0x7f1e1c13e3d0 2024-03-31T15:03:49.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:49.185+0000 7f1e20a00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f1e1c0cf590 con 0x7f1e1c0cd0e0 2024-03-31T15:03:49.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:49.185+0000 7f1e20a00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f1e1c0cf890 con 0x7f1e1c13dff0 2024-03-31T15:03:49.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:49.185+0000 7f1e1a800640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1e1c13e3d0 0x7f1e1c13bff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:49.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:49.185+0000 7f1e19e00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1e1c0cd0e0 0x7f1e1c154930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:49.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:49.185+0000 7f1e1a800640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1e1c13e3d0 0x7f1e1c13bff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53938/0 (socket says 172.21.15.16:53938) 2024-03-31T15:03:49.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:49.185+0000 7f1e1a800640 1 -- 172.21.15.16:0/3582675683 learned_addr learned my addr 172.21.15.16:0/3582675683 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:03:49.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:49.185+0000 7f1e11400640 1 --2- 172.21.15.16:0/3582675683 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f1e1c13dff0 0x7f1e1c13edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:49.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:49.185+0000 7f1e1a800640 1 -- 172.21.15.16:0/3582675683 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f1e1c13dff0 msgr2=0x7f1e1c13edb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:49.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:49.185+0000 7f1e1a800640 1 --2- 172.21.15.16:0/3582675683 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f1e1c13dff0 0x7f1e1c13edb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:49.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:49.185+0000 7f1e1a800640 1 -- 172.21.15.16:0/3582675683 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1e1c0cd0e0 msgr2=0x7f1e1c154930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:49.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:49.185+0000 7f1e1a800640 1 --2- 172.21.15.16:0/3582675683 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1e1c0cd0e0 0x7f1e1c154930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:49.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:49.185+0000 7f1e1a800640 1 -- 172.21.15.16:0/3582675683 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e1c154fc0 con 0x7f1e1c13e3d0 2024-03-31T15:03:49.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:49.185+0000 7f1e1a800640 1 --2- 172.21.15.16:0/3582675683 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1e1c13e3d0 0x7f1e1c13bff0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f1e1400e880 tx=0x7f1e1400ed40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:03:49.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:49.185+0000 7f1e12a00640 1 -- 172.21.15.16:0/3582675683 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e14017070 con 0x7f1e1c13e3d0 2024-03-31T15:03:49.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:49.185+0000 7f1e12a00640 1 -- 172.21.15.16:0/3582675683 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1e1400c940 con 0x7f1e1c13e3d0 2024-03-31T15:03:49.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:49.185+0000 7f1e20a00640 1 -- 172.21.15.16:0/3582675683 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1e1c1552a0 con 0x7f1e1c13e3d0 2024-03-31T15:03:49.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:49.185+0000 7f1e12a00640 1 -- 172.21.15.16:0/3582675683 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e1400cd30 con 0x7f1e1c13e3d0 2024-03-31T15:03:49.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:49.185+0000 7f1e20a00640 1 -- 172.21.15.16:0/3582675683 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1e1c155860 con 0x7f1e1c13e3d0 2024-03-31T15:03:49.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:49.185+0000 7f1e20a00640 1 -- 172.21.15.16:0/3582675683 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1e1c069950 con 0x7f1e1c13e3d0 2024-03-31T15:03:49.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:49.185+0000 7f1e12a00640 1 -- 172.21.15.16:0/3582675683 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f1e1401a070 con 0x7f1e1c13e3d0 2024-03-31T15:03:49.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:49.189+0000 7f1e12a00640 1 --2- 172.21.15.16:0/3582675683 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f1df8068380 0x7f1df806a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:49.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:49.189+0000 7f1e12a00640 1 -- 172.21.15.16:0/3582675683 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f1e14086d30 con 0x7f1e1c13e3d0 2024-03-31T15:03:49.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:49.189+0000 7f1e19e00640 1 --2- 172.21.15.16:0/3582675683 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f1df8068380 0x7f1df806a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:49.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:49.189+0000 7f1e19e00640 1 --2- 172.21.15.16:0/3582675683 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f1df8068380 0x7f1df806a840 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f1e0c0af000 tx=0x7f1e0c0a7070 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:03:49.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:49.193+0000 7f1e12a00640 1 -- 172.21.15.16:0/3582675683 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f1e14056c00 con 0x7f1e1c13e3d0 2024-03-31T15:03:49.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:49.353+0000 7f1e20a00640 1 -- 172.21.15.16:0/3582675683 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f1e1c06e4e0 con 0x7f1e1c13e3d0 2024-03-31T15:03:49.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:49.353+0000 7f1e12a00640 1 -- 172.21.15.16:0/3582675683 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) ==== 104+0+11 (secure 0 0 0) 0x7f1e1405af00 con 0x7f1e1c13e3d0 2024-03-31T15:03:49.348 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2024-03-31T15:03:49.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:49.357+0000 7f1e20a00640 1 -- 172.21.15.16:0/3582675683 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f1df8068380 msgr2=0x7f1df806a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:49.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:49.357+0000 7f1e20a00640 1 --2- 172.21.15.16:0/3582675683 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f1df8068380 0x7f1df806a840 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f1e0c0af000 tx=0x7f1e0c0a7070 comp rx=0 tx=0).stop 2024-03-31T15:03:49.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:49.357+0000 7f1e20a00640 1 -- 172.21.15.16:0/3582675683 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1e1c13e3d0 msgr2=0x7f1e1c13bff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:49.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:49.357+0000 7f1e20a00640 1 --2- 172.21.15.16:0/3582675683 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1e1c13e3d0 0x7f1e1c13bff0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f1e1400e880 tx=0x7f1e1400ed40 comp rx=0 tx=0).stop 2024-03-31T15:03:49.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:49.357+0000 7f1e20a00640 1 -- 172.21.15.16:0/3582675683 shutdown_connections 2024-03-31T15:03:49.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:49.357+0000 7f1e20a00640 1 --2- 172.21.15.16:0/3582675683 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f1df8068380 0x7f1df806a840 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:49.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:49.357+0000 7f1e20a00640 1 --2- 172.21.15.16:0/3582675683 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1e1c13e3d0 0x7f1e1c13bff0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:49.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:49.357+0000 7f1e20a00640 1 --2- 172.21.15.16:0/3582675683 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f1e1c13dff0 0x7f1e1c13edb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:49.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:49.357+0000 7f1e20a00640 1 --2- 172.21.15.16:0/3582675683 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1e1c0cd0e0 0x7f1e1c154930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:49.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:49.357+0000 7f1e20a00640 1 -- 172.21.15.16:0/3582675683 >> 172.21.15.16:0/3582675683 conn(0x7f1e1c06ee40 msgr2=0x7f1e1c0b11d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:03:49.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:49.357+0000 7f1e20a00640 1 -- 172.21.15.16:0/3582675683 shutdown_connections 2024-03-31T15:03:49.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:49.357+0000 7f1e20a00640 1 -- 172.21.15.16:0/3582675683 wait complete. 2024-03-31T15:03:49.370 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-31T15:03:49.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:49.521+0000 7fae8ec00640 1 Processor -- start 2024-03-31T15:03:49.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:49.521+0000 7fae8ec00640 1 -- start start 2024-03-31T15:03:49.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:49.521+0000 7fae8ec00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fae8813dff0 0x7fae8806d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:49.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:49.521+0000 7fae8ec00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fae880cd0e0 0x7fae8806da10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:49.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:49.521+0000 7fae8ec00640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fae8815fbc0 con 0x7fae8813e3d0 2024-03-31T15:03:49.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:49.521+0000 7fae8ec00640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7fae8815f730 con 0x7fae8813dff0 2024-03-31T15:03:49.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:49.521+0000 7fae8ec00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fae8815ee00 con 0x7fae880cd0e0 2024-03-31T15:03:49.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:49.521+0000 7fae87e00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fae8813dff0 0x7fae8806d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:49.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:49.521+0000 7fae87e00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fae8813dff0 0x7fae8806d4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53952/0 (socket says 172.21.15.16:53952) 2024-03-31T15:03:49.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:49.521+0000 7fae7fe00640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fae8813e3d0 0x7fae880cc9d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:34910/0 (socket says 172.21.15.16:34910) 2024-03-31T15:03:49.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:49.521+0000 7fae87e00640 1 -- 172.21.15.16:0/1096083340 learned_addr learned my addr 172.21.15.16:0/1096083340 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:03:49.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:49.521+0000 7fae8cc00640 1 --2- 172.21.15.16:0/1096083340 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fae880cd0e0 0x7fae8806da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:49.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:49.521+0000 7fae8cc00640 1 -- 172.21.15.16:0/1096083340 >> v2:172.21.15.16:3300/0 conn(0x7fae8813dff0 msgr2=0x7fae8806d4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:03:49.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:49.521+0000 7fae8cc00640 1 --2- 172.21.15.16:0/1096083340 >> v2:172.21.15.16:3300/0 conn(0x7fae8813dff0 0x7fae8806d4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:49.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:49.521+0000 7fae8cc00640 1 -- 172.21.15.16:0/1096083340 >> v1:172.21.15.16:6789/0 conn(0x7fae8813e3d0 legacy=0x7fae880cc9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:49.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:49.521+0000 7fae8cc00640 1 -- 172.21.15.16:0/1096083340 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae8815f0e0 con 0x7fae880cd0e0 2024-03-31T15:03:49.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:49.521+0000 7fae87400640 1 -- 172.21.15.16:0/1096083340 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2875747961 0 0) 0x7fae8815fbc0 con 0x7fae8813e3d0 2024-03-31T15:03:49.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:49.521+0000 7fae87e00640 1 --2- 172.21.15.16:0/1096083340 >> v2:172.21.15.16:3300/0 conn(0x7fae8813dff0 0x7fae8806d4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:03:49.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:49.525+0000 7fae8cc00640 1 --2- 172.21.15.16:0/1096083340 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fae880cd0e0 0x7fae8806da10 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fae80009d90 tx=0x7fae8001c8e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7c61d18c82833ff1 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:03:49.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:49.525+0000 7fae87400640 1 -- 172.21.15.16:0/1096083340 <== mon.2 v2:172.21.15.89:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae800b2070 con 0x7fae880cd0e0 2024-03-31T15:03:49.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:49.525+0000 7fae87400640 1 -- 172.21.15.16:0/1096083340 <== mon.2 v2:172.21.15.89:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fae800a57b0 con 0x7fae880cd0e0 2024-03-31T15:03:49.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:49.525+0000 7fae87400640 1 -- 172.21.15.16:0/1096083340 <== mon.2 v2:172.21.15.89:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae800a5b80 con 0x7fae880cd0e0 2024-03-31T15:03:49.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:49.525+0000 7fae8ec00640 1 -- 172.21.15.16:0/1096083340 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fae880cd0e0 msgr2=0x7fae8806da10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:49.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:49.525+0000 7fae8ec00640 1 --2- 172.21.15.16:0/1096083340 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fae880cd0e0 0x7fae8806da10 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fae80009d90 tx=0x7fae8001c8e0 comp rx=0 tx=0).stop 2024-03-31T15:03:49.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:49.525+0000 7fae8ec00640 1 -- 172.21.15.16:0/1096083340 shutdown_connections 2024-03-31T15:03:49.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:49.525+0000 7fae8ec00640 1 --2- 172.21.15.16:0/1096083340 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fae880cd0e0 0x7fae8806da10 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:49.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:49.525+0000 7fae8ec00640 1 --2- 172.21.15.16:0/1096083340 >> v2:172.21.15.16:3300/0 conn(0x7fae8813dff0 0x7fae8806d4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:49.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:49.525+0000 7fae8ec00640 1 -- 172.21.15.16:0/1096083340 >> 172.21.15.16:0/1096083340 conn(0x7fae8806ee40 msgr2=0x7fae8806b620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:03:49.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:49.525+0000 7fae8ec00640 1 -- 172.21.15.16:0/1096083340 shutdown_connections 2024-03-31T15:03:49.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:49.525+0000 7fae8ec00640 1 -- 172.21.15.16:0/1096083340 wait complete. 2024-03-31T15:03:49.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:49.529+0000 7fae8ec00640 1 Processor -- start 2024-03-31T15:03:49.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:49.529+0000 7fae8ec00640 1 -- start start 2024-03-31T15:03:49.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:49.529+0000 7fae8ec00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fae880cd0e0 0x7fae8814fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:49.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:49.529+0000 7fae8ec00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fae8813dff0 0x7fae8814f270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:49.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:49.529+0000 7fae8ec00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fae8813e3d0 0x7fae88156d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:49.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:49.529+0000 7fae8ec00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fae880720f0 con 0x7fae880cd0e0 2024-03-31T15:03:49.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:49.529+0000 7fae8ec00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fae88071f70 con 0x7fae8813dff0 2024-03-31T15:03:49.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:49.529+0000 7fae8ec00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fae88072270 con 0x7fae8813e3d0 2024-03-31T15:03:49.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:49.529+0000 7fae87e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fae880cd0e0 0x7fae8814fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:49.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:49.529+0000 7fae87e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fae880cd0e0 0x7fae8814fca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53954/0 (socket says 172.21.15.16:53954) 2024-03-31T15:03:49.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:49.529+0000 7fae87e00640 1 -- 172.21.15.16:0/3784075595 learned_addr learned my addr 172.21.15.16:0/3784075595 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:03:49.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:49.529+0000 7fae8cc00640 1 --2- 172.21.15.16:0/3784075595 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fae8813e3d0 0x7fae88156d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:49.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:49.529+0000 7fae7fe00640 1 --2- 172.21.15.16:0/3784075595 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fae8813dff0 0x7fae8814f270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:49.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:49.529+0000 7fae87e00640 1 -- 172.21.15.16:0/3784075595 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fae8813e3d0 msgr2=0x7fae88156d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:49.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:49.529+0000 7fae87e00640 1 --2- 172.21.15.16:0/3784075595 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fae8813e3d0 0x7fae88156d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:49.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:49.529+0000 7fae87e00640 1 -- 172.21.15.16:0/3784075595 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fae8813dff0 msgr2=0x7fae8814f270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:49.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:49.529+0000 7fae87e00640 1 --2- 172.21.15.16:0/3784075595 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fae8813dff0 0x7fae8814f270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:49.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:49.529+0000 7fae87e00640 1 -- 172.21.15.16:0/3784075595 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae88150330 con 0x7fae880cd0e0 2024-03-31T15:03:49.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:49.529+0000 7fae87e00640 1 --2- 172.21.15.16:0/3784075595 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fae880cd0e0 0x7fae8814fca0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fae78010f10 tx=0x7fae7800e730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:03:49.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:49.529+0000 7fae84c00640 1 -- 172.21.15.16:0/3784075595 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae7801e070 con 0x7fae880cd0e0 2024-03-31T15:03:49.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:49.529+0000 7fae84c00640 1 -- 172.21.15.16:0/3784075595 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fae78011860 con 0x7fae880cd0e0 2024-03-31T15:03:49.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:49.529+0000 7fae84c00640 1 -- 172.21.15.16:0/3784075595 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae78011c10 con 0x7fae880cd0e0 2024-03-31T15:03:49.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:49.529+0000 7fae8ec00640 1 -- 172.21.15.16:0/3784075595 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fae88150610 con 0x7fae880cd0e0 2024-03-31T15:03:49.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:49.529+0000 7fae8ec00640 1 -- 172.21.15.16:0/3784075595 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fae88150bd0 con 0x7fae880cd0e0 2024-03-31T15:03:49.525 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:49.529+0000 7fae8ec00640 1 -- 172.21.15.16:0/3784075595 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fae88069950 con 0x7fae880cd0e0 2024-03-31T15:03:49.525 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:49.533+0000 7fae84c00640 1 -- 172.21.15.16:0/3784075595 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fae78021070 con 0x7fae880cd0e0 2024-03-31T15:03:49.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:49.533+0000 7fae84c00640 1 --2- 172.21.15.16:0/3784075595 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fae64068380 0x7fae6406a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:49.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:49.533+0000 7fae84c00640 1 -- 172.21.15.16:0/3784075595 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fae780894d0 con 0x7fae880cd0e0 2024-03-31T15:03:49.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:49.533+0000 7fae7fe00640 1 --2- 172.21.15.16:0/3784075595 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fae64068380 0x7fae6406a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:49.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:49.533+0000 7fae7fe00640 1 --2- 172.21.15.16:0/3784075595 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fae64068380 0x7fae6406a840 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fae8815e2b0 tx=0x7fae70007df0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:03:49.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:49.537+0000 7fae84c00640 1 -- 172.21.15.16:0/3784075595 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fae780593a0 con 0x7fae880cd0e0 2024-03-31T15:03:49.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:49.701+0000 7fae8ec00640 1 -- 172.21.15.16:0/3784075595 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fae8813bd40 con 0x7fae880cd0e0 2024-03-31T15:03:49.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:49.701+0000 7fae84c00640 1 -- 172.21.15.16:0/3784075595 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) ==== 100+0+11 (secure 0 0 0) 0x7fae7805d6a0 con 0x7fae880cd0e0 2024-03-31T15:03:49.695 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 64 2024-03-31T15:03:49.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:49.705+0000 7fae8ec00640 1 -- 172.21.15.16:0/3784075595 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fae64068380 msgr2=0x7fae6406a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:49.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:49.705+0000 7fae8ec00640 1 --2- 172.21.15.16:0/3784075595 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fae64068380 0x7fae6406a840 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fae8815e2b0 tx=0x7fae70007df0 comp rx=0 tx=0).stop 2024-03-31T15:03:49.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:49.705+0000 7fae8ec00640 1 -- 172.21.15.16:0/3784075595 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fae880cd0e0 msgr2=0x7fae8814fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:49.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:49.705+0000 7fae8ec00640 1 --2- 172.21.15.16:0/3784075595 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fae880cd0e0 0x7fae8814fca0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fae78010f10 tx=0x7fae7800e730 comp rx=0 tx=0).stop 2024-03-31T15:03:49.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:49.705+0000 7fae8ec00640 1 -- 172.21.15.16:0/3784075595 shutdown_connections 2024-03-31T15:03:49.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:49.705+0000 7fae8ec00640 1 --2- 172.21.15.16:0/3784075595 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fae64068380 0x7fae6406a840 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:49.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:49.705+0000 7fae8ec00640 1 --2- 172.21.15.16:0/3784075595 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fae8813e3d0 0x7fae88156d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:49.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:49.705+0000 7fae8ec00640 1 --2- 172.21.15.16:0/3784075595 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fae8813dff0 0x7fae8814f270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:49.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:49.705+0000 7fae8ec00640 1 --2- 172.21.15.16:0/3784075595 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fae880cd0e0 0x7fae8814fca0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:49.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:49.705+0000 7fae8ec00640 1 -- 172.21.15.16:0/3784075595 >> 172.21.15.16:0/3784075595 conn(0x7fae8806ee40 msgr2=0x7fae880ceb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:03:49.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:49.705+0000 7fae8ec00640 1 -- 172.21.15.16:0/3784075595 shutdown_connections 2024-03-31T15:03:49.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:49.705+0000 7fae8ec00640 1 -- 172.21.15.16:0/3784075595 wait complete. 2024-03-31T15:03:49.715 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-31T15:03:49.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:49.861+0000 7fdd1b400640 1 Processor -- start 2024-03-31T15:03:49.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:49.861+0000 7fdd1b400640 1 -- start start 2024-03-31T15:03:49.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:49.861+0000 7fdd1b400640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fdd1406dbc0 0x7fdd1415f650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:49.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:49.861+0000 7fdd1b400640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fdd140cb440 0x7fdd1406f920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:49.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:49.861+0000 7fdd1b400640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdd140cbe90 con 0x7fdd140cb8c0 2024-03-31T15:03:49.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:49.861+0000 7fdd1b400640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7fdd1415fb90 con 0x7fdd140cb440 2024-03-31T15:03:49.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:49.861+0000 7fdd1b400640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fdd1415f1c0 con 0x7fdd1406dbc0 2024-03-31T15:03:49.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:49.865+0000 7fdd18a00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fdd1406dbc0 0x7fdd1415f650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:49.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:49.865+0000 7fdd0be00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fdd140cb440 0x7fdd1406f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:49.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:49.865+0000 7fdd18a00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fdd1406dbc0 0x7fdd1415f650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3301/0 says I am v2:172.21.15.16:51170/0 (socket says 172.21.15.16:51170) 2024-03-31T15:03:49.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:49.865+0000 7fdd18a00640 1 -- 172.21.15.16:0/1910805199 learned_addr learned my addr 172.21.15.16:0/1910805199 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:03:49.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:49.865+0000 7fdd0be00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fdd140cb440 0x7fdd1406f920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.16:51962/0 (socket says 172.21.15.16:51962) 2024-03-31T15:03:49.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:49.865+0000 7fdd18a00640 1 -- 172.21.15.16:0/1910805199 >> v2:172.21.15.89:3300/0 conn(0x7fdd140cb440 msgr2=0x7fdd1406f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:03:49.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:49.865+0000 7fdd18a00640 1 --2- 172.21.15.16:0/1910805199 >> v2:172.21.15.89:3300/0 conn(0x7fdd140cb440 0x7fdd1406f920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:49.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:49.865+0000 7fdd18a00640 1 -- 172.21.15.16:0/1910805199 >> v1:172.21.15.89:6789/0 conn(0x7fdd140cb8c0 legacy=0x7fdd14162180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:49.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:49.865+0000 7fdd18a00640 1 -- 172.21.15.16:0/1910805199 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd140cc170 con 0x7fdd1406dbc0 2024-03-31T15:03:49.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:49.865+0000 7fdd18a00640 1 --2- 172.21.15.16:0/1910805199 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fdd1406dbc0 0x7fdd1415f650 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fdd04009870 tx=0x7fdd0401c4a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=46a4b96b3680a8fd server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:03:49.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:49.865+0000 7fdd13e00640 1 -- 172.21.15.16:0/1910805199 <== mon.2 v2:172.21.15.89:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd040aa620 con 0x7fdd1406dbc0 2024-03-31T15:03:49.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:49.865+0000 7fdd13e00640 1 -- 172.21.15.16:0/1910805199 <== mon.2 v2:172.21.15.89:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdd040af070 con 0x7fdd1406dbc0 2024-03-31T15:03:49.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:49.865+0000 7fdd13e00640 1 -- 172.21.15.16:0/1910805199 <== mon.2 v2:172.21.15.89:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd040aa9e0 con 0x7fdd1406dbc0 2024-03-31T15:03:49.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:49.869+0000 7fdd1b400640 1 -- 172.21.15.16:0/1910805199 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fdd1406dbc0 msgr2=0x7fdd1415f650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:49.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:49.869+0000 7fdd1b400640 1 --2- 172.21.15.16:0/1910805199 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fdd1406dbc0 0x7fdd1415f650 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fdd04009870 tx=0x7fdd0401c4a0 comp rx=0 tx=0).stop 2024-03-31T15:03:49.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:49.869+0000 7fdd1b400640 1 -- 172.21.15.16:0/1910805199 shutdown_connections 2024-03-31T15:03:49.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:49.869+0000 7fdd1b400640 1 --2- 172.21.15.16:0/1910805199 >> v2:172.21.15.89:3300/0 conn(0x7fdd140cb440 0x7fdd1406f920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:49.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:49.869+0000 7fdd1b400640 1 --2- 172.21.15.16:0/1910805199 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fdd1406dbc0 0x7fdd1415f650 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:49.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:49.869+0000 7fdd1b400640 1 -- 172.21.15.16:0/1910805199 >> 172.21.15.16:0/1910805199 conn(0x7fdd14067330 msgr2=0x7fdd14067740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:03:49.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:49.869+0000 7fdd1b400640 1 -- 172.21.15.16:0/1910805199 shutdown_connections 2024-03-31T15:03:49.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:49.869+0000 7fdd1b400640 1 -- 172.21.15.16:0/1910805199 wait complete. 2024-03-31T15:03:49.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:49.869+0000 7fdd1b400640 1 Processor -- start 2024-03-31T15:03:49.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:49.869+0000 7fdd1b400640 1 -- start start 2024-03-31T15:03:49.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:49.869+0000 7fdd1b400640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdd1406dbc0 0x7fdd14157e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:49.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:49.869+0000 7fdd1b400640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fdd140cb440 0x7fdd14140450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:49.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:49.869+0000 7fdd1b400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdd140cb8c0 0x7fdd1413f650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:49.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:49.869+0000 7fdd1b400640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fdd14162ca0 con 0x7fdd140cb8c0 2024-03-31T15:03:49.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:49.869+0000 7fdd1b400640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fdd14070270 con 0x7fdd1406dbc0 2024-03-31T15:03:49.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:49.869+0000 7fdd1b400640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fdd1406c950 con 0x7fdd140cb440 2024-03-31T15:03:49.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:49.869+0000 7fdd0be00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fdd140cb440 0x7fdd14140450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:49.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:49.869+0000 7fdd18a00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdd1406dbc0 0x7fdd14157e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:49.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:49.869+0000 7fdd19400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdd140cb8c0 0x7fdd1413f650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:49.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:49.869+0000 7fdd0be00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fdd140cb440 0x7fdd14140450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3301/0 says I am v2:172.21.15.16:51184/0 (socket says 172.21.15.16:51184) 2024-03-31T15:03:49.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:49.869+0000 7fdd0be00640 1 -- 172.21.15.16:0/3812907937 learned_addr learned my addr 172.21.15.16:0/3812907937 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:03:49.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:49.869+0000 7fdd0be00640 1 -- 172.21.15.16:0/3812907937 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdd1406dbc0 msgr2=0x7fdd14157e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:49.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:49.869+0000 7fdd0be00640 1 --2- 172.21.15.16:0/3812907937 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdd1406dbc0 0x7fdd14157e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:49.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:49.869+0000 7fdd0be00640 1 -- 172.21.15.16:0/3812907937 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdd140cb8c0 msgr2=0x7fdd1413f650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:49.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:49.869+0000 7fdd0be00640 1 --2- 172.21.15.16:0/3812907937 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdd140cb8c0 0x7fdd1413f650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:49.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:49.869+0000 7fdd0be00640 1 -- 172.21.15.16:0/3812907937 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd14158360 con 0x7fdd140cb440 2024-03-31T15:03:49.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:49.869+0000 7fdd18a00640 1 --2- 172.21.15.16:0/3812907937 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdd1406dbc0 0x7fdd14157e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:03:49.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:49.869+0000 7fdd19400640 1 --2- 172.21.15.16:0/3812907937 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdd140cb8c0 0x7fdd1413f650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:03:49.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:49.873+0000 7fdd0be00640 1 --2- 172.21.15.16:0/3812907937 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fdd140cb440 0x7fdd14140450 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fdcfc00e870 tx=0x7fdcfc00ec70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:03:49.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:49.873+0000 7fdd11600640 1 -- 172.21.15.16:0/3812907937 <== mon.2 v2:172.21.15.89:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcfc017070 con 0x7fdd140cb440 2024-03-31T15:03:49.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:49.873+0000 7fdd11600640 1 -- 172.21.15.16:0/3812907937 <== mon.2 v2:172.21.15.89:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdcfc00c940 con 0x7fdd140cb440 2024-03-31T15:03:49.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:49.873+0000 7fdd11600640 1 -- 172.21.15.16:0/3812907937 <== mon.2 v2:172.21.15.89:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcfc00ccf0 con 0x7fdd140cb440 2024-03-31T15:03:49.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:49.873+0000 7fdd1b400640 1 -- 172.21.15.16:0/3812907937 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdd140b6260 con 0x7fdd140cb440 2024-03-31T15:03:49.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:49.873+0000 7fdd1b400640 1 -- 172.21.15.16:0/3812907937 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fdd140b6820 con 0x7fdd140cb440 2024-03-31T15:03:49.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:49.873+0000 7fdd1b400640 1 -- 172.21.15.16:0/3812907937 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdd14069950 con 0x7fdd140cb440 2024-03-31T15:03:49.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:49.873+0000 7fdd11600640 1 -- 172.21.15.16:0/3812907937 <== mon.2 v2:172.21.15.89:3301/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fdcfc01a070 con 0x7fdd140cb440 2024-03-31T15:03:49.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:49.873+0000 7fdd11600640 1 --2- 172.21.15.16:0/3812907937 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fdcf00683b0 0x7fdcf006a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:49.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:49.873+0000 7fdd11600640 1 -- 172.21.15.16:0/3812907937 <== mon.2 v2:172.21.15.89:3301/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fdcfc0867e0 con 0x7fdd140cb440 2024-03-31T15:03:49.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:49.873+0000 7fdd18a00640 1 --2- 172.21.15.16:0/3812907937 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fdcf00683b0 0x7fdcf006a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:49.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:49.877+0000 7fdd18a00640 1 --2- 172.21.15.16:0/3812907937 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fdcf00683b0 0x7fdcf006a870 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fdd040ae000 tx=0x7fdd040a7000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:03:49.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:49.877+0000 7fdd11600640 1 -- 172.21.15.16:0/3812907937 <== mon.2 v2:172.21.15.89:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fdcfc0566b0 con 0x7fdd140cb440 2024-03-31T15:03:50.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:50.077+0000 7fdd1b400640 1 -- 172.21.15.16:0/3812907937 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fdd140cf380 con 0x7fdd140cb440 2024-03-31T15:03:50.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:50.077+0000 7fdd11600640 1 -- 172.21.15.16:0/3812907937 <== mon.2 v2:172.21.15.89:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+5172 (secure 0 0 0) 0x7fdcfc05a9b0 con 0x7fdd140cb440 2024-03-31T15:03:50.070 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:03:50.071 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4276,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/2832752118","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6833","nonce":2832752118},{"type":"v1","addr":"172.21.15.16:6835","nonce":2832752118}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4278,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6834/153294668","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":153294668},{"type":"v1","addr":"172.21.15.16:6834","nonce":153294668}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4288,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/1653319357","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":1653319357},{"type":"v1","addr":"172.21.15.89:6835","nonce":1653319357}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-31T15:03:36.349865+0000","modified":"2024-03-31T15:03:39.378181+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4293},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.89:6837/2583453131","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":2583453131},{"type":"v1","addr":"172.21.15.89:6837","nonce":2583453131}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4293,"qdb_cluster":[4293]},"id":1}]} 2024-03-31T15:03:50.071 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 8 2024-03-31T15:03:50.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:50.081+0000 7fdd1b400640 1 -- 172.21.15.16:0/3812907937 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fdcf00683b0 msgr2=0x7fdcf006a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:50.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:50.081+0000 7fdd1b400640 1 --2- 172.21.15.16:0/3812907937 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fdcf00683b0 0x7fdcf006a870 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fdd040ae000 tx=0x7fdd040a7000 comp rx=0 tx=0).stop 2024-03-31T15:03:50.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:50.081+0000 7fdd1b400640 1 -- 172.21.15.16:0/3812907937 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fdd140cb440 msgr2=0x7fdd14140450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:50.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:50.081+0000 7fdd1b400640 1 --2- 172.21.15.16:0/3812907937 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fdd140cb440 0x7fdd14140450 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fdcfc00e870 tx=0x7fdcfc00ec70 comp rx=0 tx=0).stop 2024-03-31T15:03:50.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:50.081+0000 7fdd1b400640 1 -- 172.21.15.16:0/3812907937 shutdown_connections 2024-03-31T15:03:50.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:50.081+0000 7fdd1b400640 1 --2- 172.21.15.16:0/3812907937 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fdcf00683b0 0x7fdcf006a870 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:50.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:50.081+0000 7fdd1b400640 1 --2- 172.21.15.16:0/3812907937 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdd140cb8c0 0x7fdd1413f650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:50.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:50.081+0000 7fdd1b400640 1 --2- 172.21.15.16:0/3812907937 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fdd140cb440 0x7fdd14140450 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:50.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:50.081+0000 7fdd1b400640 1 --2- 172.21.15.16:0/3812907937 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdd1406dbc0 0x7fdd14157e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:50.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:50.081+0000 7fdd1b400640 1 -- 172.21.15.16:0/3812907937 >> 172.21.15.16:0/3812907937 conn(0x7fdd14067330 msgr2=0x7fdd1415dbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:03:50.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:50.081+0000 7fdd1b400640 1 -- 172.21.15.16:0/3812907937 shutdown_connections 2024-03-31T15:03:50.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:50.081+0000 7fdd1b400640 1 -- 172.21.15.16:0/3812907937 wait complete. 2024-03-31T15:03:50.092 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-31T15:03:50.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:50.237+0000 7f39a0a00640 1 Processor -- start 2024-03-31T15:03:50.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:50.237+0000 7f39a0a00640 1 -- start start 2024-03-31T15:03:50.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:50.241+0000 7f39a0a00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f399c145e40 0x7f399c0cc740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:50.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:50.241+0000 7f39a0a00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f399c070030 0x7f399c0d2010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:50.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:50.241+0000 7f39a0a00640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f399c0d0920 con 0x7f399c0ccc80 2024-03-31T15:03:50.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:50.241+0000 7f39a0a00640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f399c0d06d0 con 0x7f399c145e40 2024-03-31T15:03:50.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:50.241+0000 7f39a0a00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f399c0d0c00 con 0x7f399c070030 2024-03-31T15:03:50.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:50.241+0000 7f3999e00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f399c145e40 0x7f399c0cc740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:50.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:50.241+0000 7f3999e00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f399c145e40 0x7f399c0cc740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.16:51974/0 (socket says 172.21.15.16:51974) 2024-03-31T15:03:50.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:50.241+0000 7f3999e00640 1 -- 172.21.15.16:0/239622426 learned_addr learned my addr 172.21.15.16:0/239622426 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:03:50.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:50.241+0000 7f399a800640 1 --2- 172.21.15.16:0/239622426 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f399c070030 0x7f399c0d2010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:50.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:50.241+0000 7f399a800640 1 -- 172.21.15.16:0/239622426 >> v2:172.21.15.89:3300/0 conn(0x7f399c145e40 msgr2=0x7f399c0cc740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:03:50.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:50.241+0000 7f399a800640 1 --2- 172.21.15.16:0/239622426 >> v2:172.21.15.89:3300/0 conn(0x7f399c145e40 0x7f399c0cc740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:50.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:50.241+0000 7f399a800640 1 -- 172.21.15.16:0/239622426 >> v1:172.21.15.16:6789/0 conn(0x7f399c0ccc80 legacy=0x7f399c06f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:50.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:50.241+0000 7f399a800640 1 -- 172.21.15.16:0/239622426 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f399c0d1810 con 0x7f399c070030 2024-03-31T15:03:50.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:50.241+0000 7f3999e00640 1 --2- 172.21.15.16:0/239622426 >> v2:172.21.15.89:3300/0 conn(0x7f399c145e40 0x7f399c0cc740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:03:50.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:50.245+0000 7f399a800640 1 --2- 172.21.15.16:0/239622426 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f399c070030 0x7f399c0d2010 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f3994009d90 tx=0x7f399401c970 comp rx=0 tx=0).ready entity=mon.2 client_cookie=92e6bc7c09285212 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:03:50.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:50.245+0000 7f3999400640 1 -- 172.21.15.16:0/239622426 <== mon.2 v2:172.21.15.89:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39940b2070 con 0x7f399c070030 2024-03-31T15:03:50.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:50.245+0000 7f3999400640 1 -- 172.21.15.16:0/239622426 <== mon.2 v2:172.21.15.89:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f39940a5830 con 0x7f399c070030 2024-03-31T15:03:50.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:50.245+0000 7f3999400640 1 -- 172.21.15.16:0/239622426 <== mon.2 v2:172.21.15.89:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39940a5c00 con 0x7f399c070030 2024-03-31T15:03:50.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:50.245+0000 7f39a0a00640 1 -- 172.21.15.16:0/239622426 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f399c070030 msgr2=0x7f399c0d2010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:50.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:50.245+0000 7f39a0a00640 1 --2- 172.21.15.16:0/239622426 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f399c070030 0x7f399c0d2010 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f3994009d90 tx=0x7f399401c970 comp rx=0 tx=0).stop 2024-03-31T15:03:50.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:50.245+0000 7f39a0a00640 1 -- 172.21.15.16:0/239622426 shutdown_connections 2024-03-31T15:03:50.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:50.245+0000 7f39a0a00640 1 --2- 172.21.15.16:0/239622426 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f399c070030 0x7f399c0d2010 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:50.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:50.245+0000 7f39a0a00640 1 --2- 172.21.15.16:0/239622426 >> v2:172.21.15.89:3300/0 conn(0x7f399c145e40 0x7f399c0cc740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:50.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:50.245+0000 7f39a0a00640 1 -- 172.21.15.16:0/239622426 >> 172.21.15.16:0/239622426 conn(0x7f399c067330 msgr2=0x7f399c067740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:03:50.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:50.245+0000 7f39a0a00640 1 -- 172.21.15.16:0/239622426 shutdown_connections 2024-03-31T15:03:50.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:50.245+0000 7f39a0a00640 1 -- 172.21.15.16:0/239622426 wait complete. 2024-03-31T15:03:50.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:50.245+0000 7f39a0a00640 1 Processor -- start 2024-03-31T15:03:50.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:50.245+0000 7f39a0a00640 1 -- start start 2024-03-31T15:03:50.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:50.249+0000 7f39a0a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f399c070030 0x7f399c142eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:50.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:50.249+0000 7f39a0a00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f399c0ccc80 0x7f399c146270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:50.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:50.249+0000 7f39a0a00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f399c145e40 0x7f399c147ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:50.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:50.249+0000 7f39a0a00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f399c0b5240 con 0x7f399c070030 2024-03-31T15:03:50.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:50.249+0000 7f39a0a00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f399c0b50c0 con 0x7f399c145e40 2024-03-31T15:03:50.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:50.249+0000 7f39a0a00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f399c167180 con 0x7f399c0ccc80 2024-03-31T15:03:50.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:50.249+0000 7f3999e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f399c070030 0x7f399c142eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:50.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:50.249+0000 7f3999e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f399c070030 0x7f399c142eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53966/0 (socket says 172.21.15.16:53966) 2024-03-31T15:03:50.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:50.249+0000 7f3993e00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f399c0ccc80 0x7f399c146270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:50.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:50.249+0000 7f3999e00640 1 -- 172.21.15.16:0/1866707842 learned_addr learned my addr 172.21.15.16:0/1866707842 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:03:50.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:50.249+0000 7f399a800640 1 --2- 172.21.15.16:0/1866707842 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f399c145e40 0x7f399c147ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:50.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:50.249+0000 7f3993e00640 1 -- 172.21.15.16:0/1866707842 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f399c145e40 msgr2=0x7f399c147ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:50.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:50.249+0000 7f3993e00640 1 --2- 172.21.15.16:0/1866707842 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f399c145e40 0x7f399c147ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:50.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:50.249+0000 7f3993e00640 1 -- 172.21.15.16:0/1866707842 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f399c070030 msgr2=0x7f399c142eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:50.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:50.249+0000 7f3993e00640 1 --2- 172.21.15.16:0/1866707842 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f399c070030 0x7f399c142eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:50.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:50.249+0000 7f3993e00640 1 -- 172.21.15.16:0/1866707842 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f399c146a10 con 0x7f399c0ccc80 2024-03-31T15:03:50.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:50.249+0000 7f3999e00640 1 --2- 172.21.15.16:0/1866707842 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f399c070030 0x7f399c142eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:03:50.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:50.249+0000 7f3993e00640 1 --2- 172.21.15.16:0/1866707842 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f399c0ccc80 0x7f399c146270 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f398400c3c0 tx=0x7f398400c880 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:03:50.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:50.249+0000 7f3992000640 1 -- 172.21.15.16:0/1866707842 <== mon.2 v2:172.21.15.89:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39840052d0 con 0x7f399c0ccc80 2024-03-31T15:03:50.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:50.249+0000 7f3992000640 1 -- 172.21.15.16:0/1866707842 <== mon.2 v2:172.21.15.89:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3984005470 con 0x7f399c0ccc80 2024-03-31T15:03:50.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:50.249+0000 7f3992000640 1 -- 172.21.15.16:0/1866707842 <== mon.2 v2:172.21.15.89:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3984007260 con 0x7f399c0ccc80 2024-03-31T15:03:50.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:50.249+0000 7f39a0a00640 1 -- 172.21.15.16:0/1866707842 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f399c146cf0 con 0x7f399c0ccc80 2024-03-31T15:03:50.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:50.249+0000 7f39a0a00640 1 -- 172.21.15.16:0/1866707842 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f399c1472b0 con 0x7f399c0ccc80 2024-03-31T15:03:50.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:50.249+0000 7f39a0a00640 1 -- 172.21.15.16:0/1866707842 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f399c069950 con 0x7f399c0ccc80 2024-03-31T15:03:50.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:50.253+0000 7f3992000640 1 -- 172.21.15.16:0/1866707842 <== mon.2 v2:172.21.15.89:3301/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f3984007400 con 0x7f399c0ccc80 2024-03-31T15:03:50.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:50.253+0000 7f3992000640 1 --2- 172.21.15.16:0/1866707842 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f3978068380 0x7f397806a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:50.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:50.253+0000 7f3992000640 1 -- 172.21.15.16:0/1866707842 <== mon.2 v2:172.21.15.89:3301/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f3984085e70 con 0x7f399c0ccc80 2024-03-31T15:03:50.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:50.253+0000 7f3999e00640 1 --2- 172.21.15.16:0/1866707842 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f3978068380 0x7f397806a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:50.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:50.253+0000 7f3999e00640 1 --2- 172.21.15.16:0/1866707842 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f3978068380 0x7f397806a840 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f398c009630 tx=0x7f398c009240 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:03:50.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:50.257+0000 7f3992000640 1 -- 172.21.15.16:0/1866707842 <== mon.2 v2:172.21.15.89:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f3984055cb0 con 0x7f399c0ccc80 2024-03-31T15:03:50.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:50.417+0000 7f39a0a00640 1 -- 172.21.15.16:0/1866707842 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f399c0cf3c0 con 0x7f399c0ccc80 2024-03-31T15:03:50.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:50.421+0000 7f3992000640 1 -- 172.21.15.16:0/1866707842 <== mon.2 v2:172.21.15.89:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) ==== 74+0+16533 (secure 0 0 0) 0x7f3984059fb0 con 0x7f399c0ccc80 2024-03-31T15:03:50.413 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:03:50.414 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":21,"fsid":"286eb5da-fb47-4550-b290-d9519c13b9a5","created":"2024-03-31T15:03:11.438952+0000","modified":"2024-03-31T15:03:37.957829+0000","last_up_change":"2024-03-31T15:03:22.885535+0000","last_in_change":"2024-03-31T15:03:13.941667+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-31T15:03:22.550787+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-31T15:03:34.488425+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-31T15:03:35.299116+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"92a66450-ef19-40d8-ac3d-d760b23b0bda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6801","nonce":932712244}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6803","nonce":932712244}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6807","nonce":932712244}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6805","nonce":932712244}]},"public_addr":"172.21.15.16:6801/932712244","cluster_addr":"172.21.15.16:6803/932712244","heartbeat_back_addr":"172.21.15.16:6807/932712244","heartbeat_front_addr":"172.21.15.16:6805/932712244","state":["exists","up"]},{"osd":1,"uuid":"da9d1f2b-1968-4959-b843-faa231988e92","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6810","nonce":1398737533}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6811","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6813","nonce":1398737533}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6821","nonce":1398737533}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6815","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6816","nonce":1398737533}]},"public_addr":"172.21.15.16:6810/1398737533","cluster_addr":"172.21.15.16:6813/1398737533","heartbeat_back_addr":"172.21.15.16:6821/1398737533","heartbeat_front_addr":"172.21.15.16:6816/1398737533","state":["exists","up"]},{"osd":2,"uuid":"01a73662-9a21-4b1f-bcbb-42b9636a4638","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6819","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6822","nonce":3018526646}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6825","nonce":3018526646}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6831","nonce":3018526646}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6827","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6829","nonce":3018526646}]},"public_addr":"172.21.15.16:6822/3018526646","cluster_addr":"172.21.15.16:6825/3018526646","heartbeat_back_addr":"172.21.15.16:6831/3018526646","heartbeat_front_addr":"172.21.15.16:6829/3018526646","state":["exists","up"]},{"osd":3,"uuid":"c7771fd3-e18e-4500-ac38-23cf10ebcd75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6809","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6812","nonce":3360352974}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6817","nonce":3360352974}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6828","nonce":3360352974}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6823","nonce":3360352974}]},"public_addr":"172.21.15.16:6812/3360352974","cluster_addr":"172.21.15.16:6817/3360352974","heartbeat_back_addr":"172.21.15.16:6828/3360352974","heartbeat_front_addr":"172.21.15.16:6823/3360352974","state":["exists","up"]},{"osd":4,"uuid":"88260730-1482-4fbd-a40f-cb1124e9df4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6801","nonce":1856844856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6803","nonce":1856844856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6807","nonce":1856844856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6805","nonce":1856844856}]},"public_addr":"172.21.15.89:6801/1856844856","cluster_addr":"172.21.15.89:6803/1856844856","heartbeat_back_addr":"172.21.15.89:6807/1856844856","heartbeat_front_addr":"172.21.15.89:6805/1856844856","state":["exists","up"]},{"osd":5,"uuid":"4cb2c00c-fae0-4dd8-9d27-ec4f0093ccf7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6814","nonce":50395527}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6817","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6820","nonce":50395527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6831","nonce":50395527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6823","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6826","nonce":50395527}]},"public_addr":"172.21.15.89:6814/50395527","cluster_addr":"172.21.15.89:6820/50395527","heartbeat_back_addr":"172.21.15.89:6831/50395527","heartbeat_front_addr":"172.21.15.89:6826/50395527","state":["exists","up"]},{"osd":6,"uuid":"81ce0c19-2554-4cae-acbe-66a05e9e9b9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6809","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6812","nonce":3470471752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6815","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6818","nonce":3470471752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6827","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6829","nonce":3470471752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6821","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6824","nonce":3470471752}]},"public_addr":"172.21.15.89:6812/3470471752","cluster_addr":"172.21.15.89:6818/3470471752","heartbeat_back_addr":"172.21.15.89:6829/3470471752","heartbeat_front_addr":"172.21.15.89:6824/3470471752","state":["exists","up"]},{"osd":7,"uuid":"85e4313b-d37d-442e-8042-153bf20a847d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6811","nonce":1796778630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6813","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6816","nonce":1796778630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6825","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6828","nonce":1796778630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6819","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6822","nonce":1796778630}]},"public_addr":"172.21.15.89:6811/1796778630","cluster_addr":"172.21.15.89:6816/1796778630","heartbeat_back_addr":"172.21.15.89:6828/1796778630","heartbeat_front_addr":"172.21.15.89:6822/1796778630","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.439088+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.529970+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.680512+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:21.044161+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.582111+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.536558+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.409522+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.590229+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-31T15:03:50.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:50.421+0000 7f39a0a00640 1 -- 172.21.15.16:0/1866707842 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f3978068380 msgr2=0x7f397806a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:50.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:50.421+0000 7f39a0a00640 1 --2- 172.21.15.16:0/1866707842 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f3978068380 0x7f397806a840 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f398c009630 tx=0x7f398c009240 comp rx=0 tx=0).stop 2024-03-31T15:03:50.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:50.421+0000 7f39a0a00640 1 -- 172.21.15.16:0/1866707842 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f399c0ccc80 msgr2=0x7f399c146270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:50.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:50.421+0000 7f39a0a00640 1 --2- 172.21.15.16:0/1866707842 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f399c0ccc80 0x7f399c146270 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f398400c3c0 tx=0x7f398400c880 comp rx=0 tx=0).stop 2024-03-31T15:03:50.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:50.421+0000 7f39a0a00640 1 -- 172.21.15.16:0/1866707842 shutdown_connections 2024-03-31T15:03:50.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:50.421+0000 7f39a0a00640 1 --2- 172.21.15.16:0/1866707842 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f3978068380 0x7f397806a840 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:50.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:50.421+0000 7f39a0a00640 1 --2- 172.21.15.16:0/1866707842 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f399c145e40 0x7f399c147ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:50.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:50.421+0000 7f39a0a00640 1 --2- 172.21.15.16:0/1866707842 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f399c0ccc80 0x7f399c146270 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:50.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:50.421+0000 7f39a0a00640 1 --2- 172.21.15.16:0/1866707842 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f399c070030 0x7f399c142eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:50.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:50.421+0000 7f39a0a00640 1 -- 172.21.15.16:0/1866707842 >> 172.21.15.16:0/1866707842 conn(0x7f399c067330 msgr2=0x7f399c15bb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:03:50.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:50.421+0000 7f39a0a00640 1 -- 172.21.15.16:0/1866707842 shutdown_connections 2024-03-31T15:03:50.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:50.421+0000 7f39a0a00640 1 -- 172.21.15.16:0/1866707842 wait complete. 2024-03-31T15:03:50.435 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-31T15:03:50.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:50.581+0000 7f086ec00640 1 Processor -- start 2024-03-31T15:03:50.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:50.581+0000 7f086ec00640 1 -- start start 2024-03-31T15:03:50.577 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:50.585+0000 7f086ec00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f086806dbc0 0x7f086815f650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:50.577 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:50.585+0000 7f086ec00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f08680cb440 0x7f086806f920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:50.577 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:50.585+0000 7f086ec00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f08680cb8c0 0x7f086806fe60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:50.577 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:50.585+0000 7f086ec00640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f086815fb90 con 0x7f08680cb8c0 2024-03-31T15:03:50.577 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:50.585+0000 7f086ec00640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f086815f1c0 con 0x7f08680cb440 2024-03-31T15:03:50.577 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:50.585+0000 7f086ec00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f08680cbe90 con 0x7f086806dbc0 2024-03-31T15:03:50.577 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:50.585+0000 7f086cc00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f08680cb8c0 0x7f086806fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:50.577 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:50.585+0000 7f086cc00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f08680cb8c0 0x7f086806fe60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53972/0 (socket says 172.21.15.16:53972) 2024-03-31T15:03:50.577 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:50.585+0000 7f086cc00640 1 -- 172.21.15.16:0/397462886 learned_addr learned my addr 172.21.15.16:0/397462886 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:03:50.577 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:50.585+0000 7f0867e00640 1 --2- 172.21.15.16:0/397462886 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f086806dbc0 0x7f086815f650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:50.577 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:50.585+0000 7f085f400640 1 --2- 172.21.15.16:0/397462886 >> v2:172.21.15.89:3300/0 conn(0x7f08680cb440 0x7f086806f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:50.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:50.585+0000 7f086cc00640 1 -- 172.21.15.16:0/397462886 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f086806dbc0 msgr2=0x7f086815f650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:03:50.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:50.585+0000 7f086cc00640 1 --2- 172.21.15.16:0/397462886 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f086806dbc0 0x7f086815f650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:50.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:50.585+0000 7f086cc00640 1 -- 172.21.15.16:0/397462886 >> v2:172.21.15.89:3300/0 conn(0x7f08680cb440 msgr2=0x7f086806f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:03:50.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:50.585+0000 7f086cc00640 1 --2- 172.21.15.16:0/397462886 >> v2:172.21.15.89:3300/0 conn(0x7f08680cb440 0x7f086806f920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:50.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:50.585+0000 7f086cc00640 1 -- 172.21.15.16:0/397462886 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08680cc170 con 0x7f08680cb8c0 2024-03-31T15:03:50.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:50.585+0000 7f086cc00640 1 --2- 172.21.15.16:0/397462886 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f08680cb8c0 0x7f086806fe60 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f08680d31b0 tx=0x7f086001c930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=399b06ee7b5ccad4 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:03:50.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:50.589+0000 7f0867400640 1 -- 172.21.15.16:0/397462886 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08600a5770 con 0x7f08680cb8c0 2024-03-31T15:03:50.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:50.589+0000 7f0867400640 1 -- 172.21.15.16:0/397462886 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f08600a5910 con 0x7f08680cb8c0 2024-03-31T15:03:50.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:50.589+0000 7f0867400640 1 -- 172.21.15.16:0/397462886 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08600a5ce0 con 0x7f08680cb8c0 2024-03-31T15:03:50.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:50.589+0000 7f086ec00640 1 -- 172.21.15.16:0/397462886 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f08680cb8c0 msgr2=0x7f086806fe60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:50.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:50.589+0000 7f086ec00640 1 --2- 172.21.15.16:0/397462886 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f08680cb8c0 0x7f086806fe60 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f08680d31b0 tx=0x7f086001c930 comp rx=0 tx=0).stop 2024-03-31T15:03:50.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:50.589+0000 7f086ec00640 1 -- 172.21.15.16:0/397462886 shutdown_connections 2024-03-31T15:03:50.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:50.589+0000 7f086ec00640 1 --2- 172.21.15.16:0/397462886 >> v2:172.21.15.89:3300/0 conn(0x7f08680cb440 0x7f086806f920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:50.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:50.589+0000 7f086ec00640 1 --2- 172.21.15.16:0/397462886 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f08680cb8c0 0x7f086806fe60 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:50.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:50.589+0000 7f086ec00640 1 --2- 172.21.15.16:0/397462886 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f086806dbc0 0x7f086815f650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:50.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:50.589+0000 7f086ec00640 1 -- 172.21.15.16:0/397462886 >> 172.21.15.16:0/397462886 conn(0x7f0868067330 msgr2=0x7f0868067740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:03:50.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:50.589+0000 7f086ec00640 1 -- 172.21.15.16:0/397462886 shutdown_connections 2024-03-31T15:03:50.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:50.589+0000 7f086ec00640 1 -- 172.21.15.16:0/397462886 wait complete. 2024-03-31T15:03:50.585 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:50.589+0000 7f086ec00640 1 Processor -- start 2024-03-31T15:03:50.585 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:50.589+0000 7f086ec00640 1 -- start start 2024-03-31T15:03:50.585 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:50.589+0000 7f086ec00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f086806dbc0 0x7f08681424c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:50.585 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:50.589+0000 7f086ec00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f08680cb440 0x7f0868141c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:50.585 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:50.589+0000 7f086ec00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f08680cb8c0 0x7f086815b240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:50.585 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:50.589+0000 7f086ec00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f086806cad0 con 0x7f08680cb8c0 2024-03-31T15:03:50.585 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:50.589+0000 7f086ec00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f086806c950 con 0x7f08680cb440 2024-03-31T15:03:50.586 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:50.589+0000 7f086ec00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f0868162990 con 0x7f086806dbc0 2024-03-31T15:03:50.586 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:50.589+0000 7f086cc00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f08680cb8c0 0x7f086815b240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:50.586 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:50.589+0000 7f0867e00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f086806dbc0 0x7f08681424c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:50.586 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:50.589+0000 7f086cc00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f08680cb8c0 0x7f086815b240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53986/0 (socket says 172.21.15.16:53986) 2024-03-31T15:03:50.586 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:50.589+0000 7f086cc00640 1 -- 172.21.15.16:0/2702760974 learned_addr learned my addr 172.21.15.16:0/2702760974 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:03:50.586 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:50.589+0000 7f0867e00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f086806dbc0 0x7f08681424c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3301/0 says I am v2:172.21.15.16:51210/0 (socket says 172.21.15.16:51210) 2024-03-31T15:03:50.586 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:50.589+0000 7f085f400640 1 --2- 172.21.15.16:0/2702760974 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f08680cb440 0x7f0868141c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:50.586 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:50.589+0000 7f086cc00640 1 -- 172.21.15.16:0/2702760974 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f086806dbc0 msgr2=0x7f08681424c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:50.586 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:50.589+0000 7f086cc00640 1 --2- 172.21.15.16:0/2702760974 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f086806dbc0 0x7f08681424c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:50.586 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:50.589+0000 7f086cc00640 1 -- 172.21.15.16:0/2702760974 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f08680cb440 msgr2=0x7f0868141c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:50.586 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:50.589+0000 7f086cc00640 1 --2- 172.21.15.16:0/2702760974 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f08680cb440 0x7f0868141c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:50.586 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:50.589+0000 7f086cc00640 1 -- 172.21.15.16:0/2702760974 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0868142dc0 con 0x7f08680cb8c0 2024-03-31T15:03:50.586 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:50.593+0000 7f086cc00640 1 --2- 172.21.15.16:0/2702760974 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f08680cb8c0 0x7f086815b240 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f086001afe0 tx=0x7f08600196d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:03:50.586 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:50.593+0000 7f0864c00640 1 -- 172.21.15.16:0/2702760974 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08600bc020 con 0x7f08680cb8c0 2024-03-31T15:03:50.587 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:50.593+0000 7f0864c00640 1 -- 172.21.15.16:0/2702760974 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f08600b8690 con 0x7f08680cb8c0 2024-03-31T15:03:50.587 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:50.593+0000 7f0864c00640 1 -- 172.21.15.16:0/2702760974 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0860009510 con 0x7f08680cb8c0 2024-03-31T15:03:50.587 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:50.593+0000 7f086ec00640 1 -- 172.21.15.16:0/2702760974 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f08681430a0 con 0x7f08680cb8c0 2024-03-31T15:03:50.587 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:50.593+0000 7f086ec00640 1 -- 172.21.15.16:0/2702760974 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0868143660 con 0x7f08680cb8c0 2024-03-31T15:03:50.587 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:50.593+0000 7f086ec00640 1 -- 172.21.15.16:0/2702760974 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0868069950 con 0x7f08680cb8c0 2024-03-31T15:03:50.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:50.593+0000 7f0864c00640 1 -- 172.21.15.16:0/2702760974 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f08600bf020 con 0x7f08680cb8c0 2024-03-31T15:03:50.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:50.593+0000 7f0864c00640 1 --2- 172.21.15.16:0/2702760974 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f08440683d0 0x7f084406a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:50.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:50.593+0000 7f0864c00640 1 -- 172.21.15.16:0/2702760974 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f0860121880 con 0x7f08680cb8c0 2024-03-31T15:03:50.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:50.593+0000 7f0867e00640 1 --2- 172.21.15.16:0/2702760974 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f08440683d0 0x7f084406a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:50.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:50.597+0000 7f0867e00640 1 --2- 172.21.15.16:0/2702760974 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f08440683d0 0x7f084406a890 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f0858009ae0 tx=0x7f0858009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:03:50.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:50.597+0000 7f0864c00640 1 -- 172.21.15.16:0/2702760974 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f08600c6030 con 0x7f08680cb8c0 2024-03-31T15:03:50.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:50.793+0000 7f086ec00640 1 -- 172.21.15.16:0/2702760974 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f086806e460 con 0x7f08680cb8c0 2024-03-31T15:03:50.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:50.797+0000 7f0864c00640 1 -- 172.21.15.16:0/2702760974 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+5172 (secure 0 0 0) 0x7f08600165e0 con 0x7f08680cb8c0 2024-03-31T15:03:50.789 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:03:50.789 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4276,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/2832752118","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6833","nonce":2832752118},{"type":"v1","addr":"172.21.15.16:6835","nonce":2832752118}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4278,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6834/153294668","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":153294668},{"type":"v1","addr":"172.21.15.16:6834","nonce":153294668}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4288,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/1653319357","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":1653319357},{"type":"v1","addr":"172.21.15.89:6835","nonce":1653319357}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-31T15:03:36.349865+0000","modified":"2024-03-31T15:03:39.378181+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4293},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.89:6837/2583453131","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":2583453131},{"type":"v1","addr":"172.21.15.89:6837","nonce":2583453131}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4293,"qdb_cluster":[4293]},"id":1}]} 2024-03-31T15:03:50.789 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 8 2024-03-31T15:03:50.790 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:50.797+0000 7f086ec00640 1 -- 172.21.15.16:0/2702760974 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f08440683d0 msgr2=0x7f084406a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:50.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:50.797+0000 7f086ec00640 1 --2- 172.21.15.16:0/2702760974 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f08440683d0 0x7f084406a890 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f0858009ae0 tx=0x7f0858009190 comp rx=0 tx=0).stop 2024-03-31T15:03:50.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:50.797+0000 7f086ec00640 1 -- 172.21.15.16:0/2702760974 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f08680cb8c0 msgr2=0x7f086815b240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:50.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:50.797+0000 7f086ec00640 1 --2- 172.21.15.16:0/2702760974 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f08680cb8c0 0x7f086815b240 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f086001afe0 tx=0x7f08600196d0 comp rx=0 tx=0).stop 2024-03-31T15:03:50.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:50.797+0000 7f086ec00640 1 -- 172.21.15.16:0/2702760974 shutdown_connections 2024-03-31T15:03:50.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:50.797+0000 7f086ec00640 1 --2- 172.21.15.16:0/2702760974 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f08440683d0 0x7f084406a890 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:50.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:50.797+0000 7f086ec00640 1 --2- 172.21.15.16:0/2702760974 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f08680cb8c0 0x7f086815b240 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:50.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:50.797+0000 7f086ec00640 1 --2- 172.21.15.16:0/2702760974 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f08680cb440 0x7f0868141c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:50.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:50.797+0000 7f086ec00640 1 --2- 172.21.15.16:0/2702760974 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f086806dbc0 0x7f08681424c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:50.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:50.797+0000 7f086ec00640 1 -- 172.21.15.16:0/2702760974 >> 172.21.15.16:0/2702760974 conn(0x7f0868067330 msgr2=0x7f086815d9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:03:50.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:50.797+0000 7f086ec00640 1 -- 172.21.15.16:0/2702760974 shutdown_connections 2024-03-31T15:03:50.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:50.797+0000 7f086ec00640 1 -- 172.21.15.16:0/2702760974 wait complete. 2024-03-31T15:03:51.212 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 8, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-31T15:03:36.349865+0000', 'modified': '2024-03-31T15:03:39.378181+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4293}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4293': {'gid': 4293, 'name': 'b', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.89:6837/2583453131', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.89:6836', 'nonce': 2583453131}, {'type': 'v1', 'addr': '172.21.15.89:6837', 'nonce': 2583453131}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4293, 'qdb_cluster': [4293]} 2024-03-31T15:03:51.212 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-31T15:03:51.212 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-03-31T15:03:51.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.365+0000 7f62f2c00640 1 Processor -- start 2024-03-31T15:03:51.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.365+0000 7f62f2c00640 1 -- start start 2024-03-31T15:03:51.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.365+0000 7f62f2c00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f62ec15eee0 0x7f62ec072420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:51.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.365+0000 7f62f2c00640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f62ec15f3c0 con 0x7f62ec0b23d0 2024-03-31T15:03:51.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.365+0000 7f62f2c00640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f62ec15f6a0 con 0x7f62ec15b3d0 2024-03-31T15:03:51.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.365+0000 7f62f2c00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f62ec15b7b0 con 0x7f62ec15eee0 2024-03-31T15:03:51.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.365+0000 7f62e3400640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f62ec0b23d0 0x7f62ec071d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:34916/0 (socket says 172.21.15.16:34916) 2024-03-31T15:03:51.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.365+0000 7f62e3400640 1 -- 172.21.15.16:0/978859322 learned_addr learned my addr 172.21.15.16:0/978859322 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:03:51.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.365+0000 7f62f0c00640 1 --2- 172.21.15.16:0/978859322 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f62ec15eee0 0x7f62ec072420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:51.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.365+0000 7f62eb400640 1 -- 172.21.15.16:0/978859322 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2515803860 0 0) 0x7f62ec15f3c0 con 0x7f62ec0b23d0 2024-03-31T15:03:51.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.365+0000 7f62eb400640 1 -- 172.21.15.16:0/978859322 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f62d0003530 con 0x7f62ec0b23d0 2024-03-31T15:03:51.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.365+0000 7f62eb400640 1 -- 172.21.15.16:0/978859322 <== mon.1 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 521218617 0 0) 0x7f62ec15f6a0 con 0x7f62ec15b3d0 2024-03-31T15:03:51.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.365+0000 7f62eb400640 1 -- 172.21.15.16:0/978859322 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f62ec15f3c0 con 0x7f62ec15b3d0 2024-03-31T15:03:51.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.365+0000 7f62f0c00640 1 -- 172.21.15.16:0/978859322 >> v1:172.21.15.89:6789/0 conn(0x7f62ec15b3d0 legacy=0x7f62ec0b1cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:51.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.365+0000 7f62f0c00640 1 -- 172.21.15.16:0/978859322 >> v1:172.21.15.16:6789/0 conn(0x7f62ec0b23d0 legacy=0x7f62ec071d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:51.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.365+0000 7f62f0c00640 1 -- 172.21.15.16:0/978859322 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62ec15f980 con 0x7f62ec15eee0 2024-03-31T15:03:51.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.369+0000 7f62f0c00640 1 --2- 172.21.15.16:0/978859322 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f62ec15eee0 0x7f62ec072420 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f62e4009dc0 tx=0x7f62e401c960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f6eb77c95e608c29 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:03:51.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.369+0000 7f62eb400640 1 -- 172.21.15.16:0/978859322 <== mon.2 v2:172.21.15.89:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62e40b2070 con 0x7f62ec15eee0 2024-03-31T15:03:51.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.369+0000 7f62eb400640 1 -- 172.21.15.16:0/978859322 <== mon.2 v2:172.21.15.89:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f62e40a59e0 con 0x7f62ec15eee0 2024-03-31T15:03:51.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.369+0000 7f62eb400640 1 -- 172.21.15.16:0/978859322 <== mon.2 v2:172.21.15.89:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62e40a5db0 con 0x7f62ec15eee0 2024-03-31T15:03:51.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.369+0000 7f62f2c00640 1 -- 172.21.15.16:0/978859322 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f62ec15eee0 msgr2=0x7f62ec072420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:51.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.369+0000 7f62f2c00640 1 --2- 172.21.15.16:0/978859322 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f62ec15eee0 0x7f62ec072420 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f62e4009dc0 tx=0x7f62e401c960 comp rx=0 tx=0).stop 2024-03-31T15:03:51.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.369+0000 7f62f2c00640 1 -- 172.21.15.16:0/978859322 shutdown_connections 2024-03-31T15:03:51.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.369+0000 7f62f2c00640 1 --2- 172.21.15.16:0/978859322 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f62ec15eee0 0x7f62ec072420 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:51.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.369+0000 7f62f2c00640 1 -- 172.21.15.16:0/978859322 >> 172.21.15.16:0/978859322 conn(0x7f62ec06b620 msgr2=0x7f62ec06ba30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:03:51.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.369+0000 7f62f2c00640 1 -- 172.21.15.16:0/978859322 shutdown_connections 2024-03-31T15:03:51.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.369+0000 7f62f2c00640 1 -- 172.21.15.16:0/978859322 wait complete. 2024-03-31T15:03:51.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.369+0000 7f62f2c00640 1 Processor -- start 2024-03-31T15:03:51.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.369+0000 7f62f2c00640 1 -- start start 2024-03-31T15:03:51.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.373+0000 7f62f2c00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f62ec0b23d0 0x7f62ec148a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:51.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.373+0000 7f62f2c00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f62ec15b3d0 0x7f62ec154630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:51.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.373+0000 7f62f2c00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f62ec15eee0 0x7f62ec15b960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:51.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.373+0000 7f62f2c00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f62ec073580 con 0x7f62ec15b3d0 2024-03-31T15:03:51.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.373+0000 7f62f2c00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f62ec0b4560 con 0x7f62ec15eee0 2024-03-31T15:03:51.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.373+0000 7f62f2c00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f62ec073700 con 0x7f62ec0b23d0 2024-03-31T15:03:51.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.373+0000 7f62e3400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f62ec15b3d0 0x7f62ec154630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:51.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.373+0000 7f62e3400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f62ec15b3d0 0x7f62ec154630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53990/0 (socket says 172.21.15.16:53990) 2024-03-31T15:03:51.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.373+0000 7f62e3400640 1 -- 172.21.15.16:0/4020542802 learned_addr learned my addr 172.21.15.16:0/4020542802 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:03:51.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.373+0000 7f62ebe00640 1 --2- 172.21.15.16:0/4020542802 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f62ec0b23d0 0x7f62ec148a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:51.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.373+0000 7f62f0c00640 1 --2- 172.21.15.16:0/4020542802 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f62ec15eee0 0x7f62ec15b960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:51.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.373+0000 7f62e3400640 1 -- 172.21.15.16:0/4020542802 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f62ec0b23d0 msgr2=0x7f62ec148a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:51.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.373+0000 7f62e3400640 1 --2- 172.21.15.16:0/4020542802 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f62ec0b23d0 0x7f62ec148a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:51.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.373+0000 7f62e3400640 1 -- 172.21.15.16:0/4020542802 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f62ec15eee0 msgr2=0x7f62ec15b960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:51.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.373+0000 7f62e3400640 1 --2- 172.21.15.16:0/4020542802 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f62ec15eee0 0x7f62ec15b960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:51.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.373+0000 7f62e3400640 1 -- 172.21.15.16:0/4020542802 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62ec0adae0 con 0x7f62ec15b3d0 2024-03-31T15:03:51.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.373+0000 7f62e3400640 1 --2- 172.21.15.16:0/4020542802 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f62ec15b3d0 0x7f62ec154630 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f62d400c420 tx=0x7f62d400c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:03:51.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.373+0000 7f62e8c00640 1 -- 172.21.15.16:0/4020542802 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62d4017070 con 0x7f62ec15b3d0 2024-03-31T15:03:51.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.373+0000 7f62e8c00640 1 -- 172.21.15.16:0/4020542802 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f62d4005d10 con 0x7f62ec15b3d0 2024-03-31T15:03:51.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.373+0000 7f62e8c00640 1 -- 172.21.15.16:0/4020542802 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62d4007310 con 0x7f62ec15b3d0 2024-03-31T15:03:51.368 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.373+0000 7f62f2c00640 1 -- 172.21.15.16:0/4020542802 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f62ec0addc0 con 0x7f62ec15b3d0 2024-03-31T15:03:51.368 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.373+0000 7f62f2c00640 1 -- 172.21.15.16:0/4020542802 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f62ec0ae380 con 0x7f62ec15b3d0 2024-03-31T15:03:51.368 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.377+0000 7f62e8c00640 1 -- 172.21.15.16:0/4020542802 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f62d401a070 con 0x7f62ec15b3d0 2024-03-31T15:03:51.369 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.377+0000 7f62e8c00640 1 --2- 172.21.15.16:0/4020542802 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f62c80683b0 0x7f62c806a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:51.369 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.377+0000 7f62e8c00640 1 -- 172.21.15.16:0/4020542802 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f62d4085ce0 con 0x7f62ec15b3d0 2024-03-31T15:03:51.369 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.377+0000 7f62ebe00640 1 --2- 172.21.15.16:0/4020542802 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f62c80683b0 0x7f62c806a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:51.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.377+0000 7f62f60de000 1 Processor -- start 2024-03-31T15:03:51.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.377+0000 7f62f60de000 1 -- start start 2024-03-31T15:03:51.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.377+0000 7f62ebe00640 1 --2- 172.21.15.16:0/4020542802 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f62c80683b0 0x7f62c806a870 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f62dc007fd0 tx=0x7f62dc009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:03:51.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.377+0000 7f62f60de000 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55a863f2d4a0 0x55a863f2d880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:51.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.377+0000 7f62f60de000 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a863f2ddc0 0x55a863ee0b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:51.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.377+0000 7f62f60de000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a863ee10b0 0x55a863ee1490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:51.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.377+0000 7f62f60de000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55a863d33950 con 0x55a863ee10b0 2024-03-31T15:03:51.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.377+0000 7f62f60de000 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55a863cd0990 con 0x55a863f2ddc0 2024-03-31T15:03:51.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.377+0000 7f62f60de000 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x55a863d2f900 con 0x55a863f2d4a0 2024-03-31T15:03:51.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.377+0000 7f62f0c00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a863ee10b0 0x55a863ee1490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:51.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.377+0000 7f62e3400640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a863f2ddc0 0x55a863ee0b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:51.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.377+0000 7f62f0c00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a863ee10b0 0x55a863ee1490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54002/0 (socket says 172.21.15.16:54002) 2024-03-31T15:03:51.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.377+0000 7f62f0c00640 1 -- 172.21.15.16:0/3953865172 learned_addr learned my addr 172.21.15.16:0/3953865172 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:03:51.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.377+0000 7f62ebe00640 1 --2- 172.21.15.16:0/3953865172 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55a863f2d4a0 0x55a863f2d880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:51.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.377+0000 7f62f0c00640 1 -- 172.21.15.16:0/3953865172 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55a863f2d4a0 msgr2=0x55a863f2d880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:03:51.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.377+0000 7f62f0c00640 1 --2- 172.21.15.16:0/3953865172 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55a863f2d4a0 0x55a863f2d880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:51.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.377+0000 7f62f0c00640 1 -- 172.21.15.16:0/3953865172 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a863f2ddc0 msgr2=0x55a863ee0b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:03:51.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.377+0000 7f62f0c00640 1 --2- 172.21.15.16:0/3953865172 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a863f2ddc0 0x55a863ee0b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:51.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.377+0000 7f62f0c00640 1 -- 172.21.15.16:0/3953865172 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a863d78100 con 0x55a863ee10b0 2024-03-31T15:03:51.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.377+0000 7f62e3400640 1 --2- 172.21.15.16:0/3953865172 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a863f2ddc0 0x55a863ee0b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:03:51.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.377+0000 7f62f0c00640 1 --2- 172.21.15.16:0/3953865172 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a863ee10b0 0x55a863ee1490 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f62e4098560 tx=0x7f62e4004ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=85d7aeda0ed46dd9 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:03:51.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.377+0000 7f62e0c00640 1 -- 172.21.15.16:0/3953865172 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62e40b2040 con 0x55a863ee10b0 2024-03-31T15:03:51.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.377+0000 7f62e0c00640 1 -- 172.21.15.16:0/3953865172 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f62e40b3040 con 0x55a863ee10b0 2024-03-31T15:03:51.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.377+0000 7f62e0c00640 1 -- 172.21.15.16:0/3953865172 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62e40a5cc0 con 0x55a863ee10b0 2024-03-31T15:03:51.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.381+0000 7f62f60de000 1 -- 172.21.15.16:0/3953865172 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a863ee10b0 msgr2=0x55a863ee1490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:51.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.381+0000 7f62f60de000 1 --2- 172.21.15.16:0/3953865172 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a863ee10b0 0x55a863ee1490 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f62e4098560 tx=0x7f62e4004ac0 comp rx=0 tx=0).stop 2024-03-31T15:03:51.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.381+0000 7f62f60de000 1 -- 172.21.15.16:0/3953865172 shutdown_connections 2024-03-31T15:03:51.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.381+0000 7f62f60de000 1 --2- 172.21.15.16:0/3953865172 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a863ee10b0 0x55a863ee1490 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:51.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.381+0000 7f62f60de000 1 --2- 172.21.15.16:0/3953865172 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a863f2ddc0 0x55a863ee0b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:51.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.381+0000 7f62f60de000 1 --2- 172.21.15.16:0/3953865172 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55a863f2d4a0 0x55a863f2d880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:51.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.381+0000 7f62f60de000 1 -- 172.21.15.16:0/3953865172 >> 172.21.15.16:0/3953865172 conn(0x55a863dcbb80 msgr2=0x55a863f10460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:03:51.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.381+0000 7f62f60de000 1 -- 172.21.15.16:0/3953865172 shutdown_connections 2024-03-31T15:03:51.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.381+0000 7f62f60de000 1 -- 172.21.15.16:0/3953865172 wait complete. 2024-03-31T15:03:51.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.381+0000 7f62f60de000 1 -- start start 2024-03-31T15:03:51.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.381+0000 7f62f60de000 1 Processor -- start 2024-03-31T15:03:51.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.381+0000 7f62f60de000 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a863ee10b0 0x55a863efc0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:51.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.381+0000 7f62f60de000 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55a863f2d4a0 0x55a863efc630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:51.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.381+0000 7f62f60de000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a863f2ddc0 0x55a863efcb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:51.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.381+0000 7f62f60de000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55a863d32c60 con 0x55a863f2ddc0 2024-03-31T15:03:51.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.381+0000 7f62f60de000 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55a863cf13e0 con 0x55a863ee10b0 2024-03-31T15:03:51.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.381+0000 7f62f60de000 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x55a863d40060 con 0x55a863f2d4a0 2024-03-31T15:03:51.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.381+0000 7f62f60de000 10 client.0 _ll_register_callbacks cb 0x55a863d940f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T15:03:51.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.381+0000 7f62f0c00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a863f2ddc0 0x55a863efcb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:51.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.381+0000 7f62f0c00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a863f2ddc0 0x55a863efcb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54018/0 (socket says 172.21.15.16:54018) 2024-03-31T15:03:51.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.381+0000 7f62f0c00640 1 -- 172.21.15.16:0/247165317 learned_addr learned my addr 172.21.15.16:0/247165317 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:03:51.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.381+0000 7f62ebe00640 1 --2- 172.21.15.16:0/247165317 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a863ee10b0 0x55a863efc0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:51.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.381+0000 7f62e3400640 1 --2- 172.21.15.16:0/247165317 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55a863f2d4a0 0x55a863efc630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:51.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.381+0000 7f62f0c00640 1 -- 172.21.15.16:0/247165317 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55a863f2d4a0 msgr2=0x55a863efc630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:03:51.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.381+0000 7f62f0c00640 1 --2- 172.21.15.16:0/247165317 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55a863f2d4a0 0x55a863efc630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:51.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.381+0000 7f62f0c00640 1 -- 172.21.15.16:0/247165317 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a863ee10b0 msgr2=0x55a863efc0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:03:51.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.381+0000 7f62f0c00640 1 --2- 172.21.15.16:0/247165317 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a863ee10b0 0x55a863efc0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:51.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.381+0000 7f62f0c00640 1 -- 172.21.15.16:0/247165317 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a863cfd8c0 con 0x55a863f2ddc0 2024-03-31T15:03:51.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.381+0000 7f62f0c00640 1 --2- 172.21.15.16:0/247165317 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a863f2ddc0 0x55a863efcb70 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f62e4098560 tx=0x7f62e401b1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d125310d7cdd7d6c server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:03:51.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.381+0000 7f62d9600640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-31T15:03:51.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.381+0000 7f62d9600640 1 -- 172.21.15.16:0/247165317 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62e40b2040 con 0x55a863f2ddc0 2024-03-31T15:03:51.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.381+0000 7f62d9600640 1 -- 172.21.15.16:0/247165317 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f62e4003d60 con 0x55a863f2ddc0 2024-03-31T15:03:51.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.381+0000 7f62f60de000 1 -- 172.21.15.16:0/247165317 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f62e40b3040 con 0x55a863f2ddc0 2024-03-31T15:03:51.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.381+0000 7f62d9600640 1 -- 172.21.15.16:0/247165317 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=8) ==== 24+0+0 (secure 0 0 0) 0x7f62e40d0050 con 0x55a863f2ddc0 2024-03-31T15:03:51.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.381+0000 7f62f60de000 10 client.4650 fetch_fsmap learned FSMap version 8 2024-03-31T15:03:51.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.381+0000 7f62f60de000 1 -- 172.21.15.16:0/247165317 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=8,monmap=2+}) -- 0x55a863cbbe80 con 0x55a863f2ddc0 2024-03-31T15:03:51.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.381+0000 7f62d9600640 1 -- 172.21.15.16:0/247165317 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f62e40b3040 con 0x55a863f2ddc0 2024-03-31T15:03:51.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.381+0000 7f62d9600640 1 -- 172.21.15.16:0/247165317 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 8) ==== 3494+0+0 (secure 0 0 0) 0x7f62e40d3020 con 0x55a863f2ddc0 2024-03-31T15:03:51.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.385+0000 7f62f60de000 10 client.4650 fetch_fsmap finished waiting for FSMap version 8 2024-03-31T15:03:51.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.385+0000 7f62f60de000 10 client.4650 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-31T15:03:51.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.385+0000 7f62f60de000 20 client.4650 populate_metadata read hostname 'smithi016' 2024-03-31T15:03:51.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.385+0000 7f62f60de000 1 --2- 172.21.15.16:0/247165317 >> [v2:172.21.15.89:6836/2583453131,v1:172.21.15.89:6837/2583453131] conn(0x55a863efd380 0x55a863f3fa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:51.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.385+0000 7f62f60de000 4 client.4650 mds_command: new command op to 4293 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-31T15:03:51.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.385+0000 7f62f60de000 1 -- 172.21.15.16:0/247165317 --> [v2:172.21.15.89:6836/2583453131,v1:172.21.15.89:6837/2583453131] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a863ccebc0 con 0x55a863efd380 2024-03-31T15:03:51.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.385+0000 7f62ebe00640 1 --2- 172.21.15.16:0/247165317 >> [v2:172.21.15.89:6836/2583453131,v1:172.21.15.89:6837/2583453131] conn(0x55a863efd380 0x55a863f3fa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:51.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.385+0000 7f62ebe00640 1 --2- 172.21.15.16:0/247165317 >> [v2:172.21.15.89:6836/2583453131,v1:172.21.15.89:6837/2583453131] conn(0x55a863efd380 0x55a863f3fa20 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8cb07514f6edf2ed server_cookie=de0ff42fda949a25 in_seq=0 out_seq=0 2024-03-31T15:03:51.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.385+0000 7f62d9600640 10 client.4650 ms_handle_connect on v2:172.21.15.89:6836/2583453131 2024-03-31T15:03:51.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.385+0000 7f62d9600640 1 -- 172.21.15.16:0/247165317 <== mds.0 v2:172.21.15.89:6836/2583453131 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55a863ccebc0 con 0x55a863efd380 2024-03-31T15:03:51.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.385+0000 7f62d9600640 10 client.4650 handle_command_reply: tid=0 2024-03-31T15:03:51.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.385+0000 7f62f60de000 1 client.4650 shutdown 2024-03-31T15:03:51.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.385+0000 7f62f60de000 1 -- 172.21.15.16:0/247165317 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a863f2ddc0 msgr2=0x55a863efcb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:51.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.385+0000 7f62f60de000 1 --2- 172.21.15.16:0/247165317 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a863f2ddc0 0x55a863efcb70 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f62e4098560 tx=0x7f62e401b1a0 comp rx=0 tx=0).stop 2024-03-31T15:03:51.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.385+0000 7f62f60de000 1 -- 172.21.15.16:0/247165317 shutdown_connections 2024-03-31T15:03:51.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.385+0000 7f62f60de000 1 --2- 172.21.15.16:0/247165317 >> [v2:172.21.15.89:6836/2583453131,v1:172.21.15.89:6837/2583453131] conn(0x55a863efd380 0x55a863f3fa20 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:51.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.385+0000 7f62f60de000 1 --2- 172.21.15.16:0/247165317 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a863f2ddc0 0x55a863efcb70 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:51.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.385+0000 7f62f60de000 1 --2- 172.21.15.16:0/247165317 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55a863f2d4a0 0x55a863efc630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:51.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.385+0000 7f62d9600640 0 client.4650 ms_handle_reset on v2:172.21.15.89:6836/2583453131 2024-03-31T15:03:51.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.385+0000 7f62f60de000 1 --2- 172.21.15.16:0/247165317 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a863ee10b0 0x55a863efc0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:51.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.385+0000 7f62f60de000 1 -- 172.21.15.16:0/247165317 >> 172.21.15.16:0/247165317 conn(0x55a863dcbb80 msgr2=0x55a863ee0270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:03:51.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.385+0000 7f62f60de000 1 -- 172.21.15.16:0/247165317 shutdown_connections 2024-03-31T15:03:51.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.385+0000 7f62f60de000 1 -- 172.21.15.16:0/247165317 wait complete. 2024-03-31T15:03:51.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.385+0000 7f62f60de000 20 client.4650 trim_cache size 0 max 16384 2024-03-31T15:03:51.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.405+0000 7f62f60de000 1 Processor -- start 2024-03-31T15:03:51.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.405+0000 7f62f60de000 1 -- start start 2024-03-31T15:03:51.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.405+0000 7f62f60de000 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a863ee10b0 0x55a863f2cda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:51.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.405+0000 7f62f60de000 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55a863f2d4a0 0x55a863f0f7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:51.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.405+0000 7f62f60de000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a863f2ddc0 0x55a863f0fd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:51.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.405+0000 7f62f60de000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f62ec203c80 con 0x55a863f2ddc0 2024-03-31T15:03:51.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.405+0000 7f62f60de000 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f62ec0b16c0 con 0x55a863ee10b0 2024-03-31T15:03:51.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.405+0000 7f62f60de000 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f62ec1558e0 con 0x55a863f2d4a0 2024-03-31T15:03:51.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.405+0000 7f62f0c00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a863f2ddc0 0x55a863f0fd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:51.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.405+0000 7f62f0c00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a863f2ddc0 0x55a863f0fd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54020/0 (socket says 172.21.15.16:54020) 2024-03-31T15:03:51.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.405+0000 7f62f0c00640 1 -- 172.21.15.16:0/3025263331 learned_addr learned my addr 172.21.15.16:0/3025263331 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:03:51.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.405+0000 7f62ebe00640 1 --2- 172.21.15.16:0/3025263331 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a863ee10b0 0x55a863f2cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:51.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.405+0000 7f62e3400640 1 --2- 172.21.15.16:0/3025263331 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55a863f2d4a0 0x55a863f0f7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:51.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.405+0000 7f62f0c00640 1 -- 172.21.15.16:0/3025263331 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55a863f2d4a0 msgr2=0x55a863f0f7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:03:51.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.405+0000 7f62f0c00640 1 --2- 172.21.15.16:0/3025263331 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55a863f2d4a0 0x55a863f0f7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:51.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.405+0000 7f62f0c00640 1 -- 172.21.15.16:0/3025263331 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a863ee10b0 msgr2=0x55a863f2cda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:03:51.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.405+0000 7f62f0c00640 1 --2- 172.21.15.16:0/3025263331 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a863ee10b0 0x55a863f2cda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:51.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.405+0000 7f62f0c00640 1 -- 172.21.15.16:0/3025263331 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a863cee490 con 0x55a863f2ddc0 2024-03-31T15:03:51.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.405+0000 7f62f0c00640 1 --2- 172.21.15.16:0/3025263331 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a863f2ddc0 0x55a863f0fd30 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f62e4002240 tx=0x7f62e40bc000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b126d67615045e6 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:03:51.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.405+0000 7f62dbe00640 1 -- 172.21.15.16:0/3025263331 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62e40b2070 con 0x55a863f2ddc0 2024-03-31T15:03:51.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.405+0000 7f62dbe00640 1 -- 172.21.15.16:0/3025263331 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f62e40c8070 con 0x55a863f2ddc0 2024-03-31T15:03:51.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.405+0000 7f62dbe00640 1 -- 172.21.15.16:0/3025263331 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62e40ad640 con 0x55a863f2ddc0 2024-03-31T15:03:51.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.405+0000 7f62f60de000 1 -- 172.21.15.16:0/3025263331 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a863f2ddc0 msgr2=0x55a863f0fd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:51.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.405+0000 7f62f60de000 1 --2- 172.21.15.16:0/3025263331 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a863f2ddc0 0x55a863f0fd30 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f62e4002240 tx=0x7f62e40bc000 comp rx=0 tx=0).stop 2024-03-31T15:03:51.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.405+0000 7f62f60de000 1 -- 172.21.15.16:0/3025263331 shutdown_connections 2024-03-31T15:03:51.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.405+0000 7f62f60de000 1 --2- 172.21.15.16:0/3025263331 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a863f2ddc0 0x55a863f0fd30 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:51.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.405+0000 7f62f60de000 1 --2- 172.21.15.16:0/3025263331 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55a863f2d4a0 0x55a863f0f7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:51.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.405+0000 7f62f60de000 1 --2- 172.21.15.16:0/3025263331 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a863ee10b0 0x55a863f2cda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:51.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.405+0000 7f62f60de000 1 -- 172.21.15.16:0/3025263331 >> 172.21.15.16:0/3025263331 conn(0x55a863dcbb80 msgr2=0x55a863f3aa90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:03:51.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.405+0000 7f62f60de000 1 -- 172.21.15.16:0/3025263331 shutdown_connections 2024-03-31T15:03:51.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.405+0000 7f62f60de000 1 -- 172.21.15.16:0/3025263331 wait complete. 2024-03-31T15:03:51.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.405+0000 7f62f60de000 1 -- start start 2024-03-31T15:03:51.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.405+0000 7f62f60de000 1 Processor -- start 2024-03-31T15:03:51.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.409+0000 7f62f60de000 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55a863ee10b0 0x55a863f997c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:51.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.409+0000 7f62f60de000 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a863f2d4a0 0x55a863ee0320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:51.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.409+0000 7f62f60de000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a863f2ddc0 0x55a863ee0860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:51.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.409+0000 7f62f60de000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55a863d37f80 con 0x55a863f2ddc0 2024-03-31T15:03:51.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.409+0000 7f62f60de000 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f62ec0b1070 con 0x55a863f2d4a0 2024-03-31T15:03:51.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.409+0000 7f62f60de000 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x55a863dfc9c0 con 0x55a863ee10b0 2024-03-31T15:03:51.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.409+0000 7f62f60de000 10 client.0 _ll_register_callbacks cb 0x7f62dc007c80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T15:03:51.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.409+0000 7f62f0c00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a863f2ddc0 0x55a863ee0860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:51.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.409+0000 7f62ebe00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55a863ee10b0 0x55a863f997c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:51.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.409+0000 7f62f0c00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a863f2ddc0 0x55a863ee0860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54028/0 (socket says 172.21.15.16:54028) 2024-03-31T15:03:51.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.409+0000 7f62f0c00640 1 -- 172.21.15.16:0/1360250315 learned_addr learned my addr 172.21.15.16:0/1360250315 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:03:51.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.409+0000 7f62e3400640 1 --2- 172.21.15.16:0/1360250315 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a863f2d4a0 0x55a863ee0320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:51.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.409+0000 7f62f0c00640 1 -- 172.21.15.16:0/1360250315 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55a863ee10b0 msgr2=0x55a863f997c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:03:51.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.409+0000 7f62f0c00640 1 --2- 172.21.15.16:0/1360250315 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55a863ee10b0 0x55a863f997c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:51.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.409+0000 7f62f0c00640 1 -- 172.21.15.16:0/1360250315 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a863f2d4a0 msgr2=0x55a863ee0320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:03:51.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.409+0000 7f62f0c00640 1 --2- 172.21.15.16:0/1360250315 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a863f2d4a0 0x55a863ee0320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:51.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.409+0000 7f62f0c00640 1 -- 172.21.15.16:0/1360250315 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a863cece80 con 0x55a863f2ddc0 2024-03-31T15:03:51.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.409+0000 7f62f0c00640 1 --2- 172.21.15.16:0/1360250315 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a863f2ddc0 0x55a863ee0860 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f62e4098560 tx=0x7f62e40a7070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8d92dfe91545875 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:03:51.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.409+0000 7f62d9600640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-31T15:03:51.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.409+0000 7f62d9600640 1 -- 172.21.15.16:0/1360250315 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62e40b2040 con 0x55a863f2ddc0 2024-03-31T15:03:51.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.409+0000 7f62d9600640 1 -- 172.21.15.16:0/1360250315 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f62e40d43f0 con 0x55a863f2ddc0 2024-03-31T15:03:51.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.409+0000 7f62f60de000 1 -- 172.21.15.16:0/1360250315 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f62e40c8070 con 0x55a863f2ddc0 2024-03-31T15:03:51.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.409+0000 7f62d9600640 1 -- 172.21.15.16:0/1360250315 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=8) ==== 24+0+0 (secure 0 0 0) 0x7f62e40e4050 con 0x55a863f2ddc0 2024-03-31T15:03:51.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.409+0000 7f62f60de000 10 client.4656 fetch_fsmap learned FSMap version 8 2024-03-31T15:03:51.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.409+0000 7f62f60de000 1 -- 172.21.15.16:0/1360250315 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=8,monmap=2+}) -- 0x55a863cda270 con 0x55a863f2ddc0 2024-03-31T15:03:51.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.409+0000 7f62d9600640 1 -- 172.21.15.16:0/1360250315 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f62e40c8070 con 0x55a863f2ddc0 2024-03-31T15:03:51.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.409+0000 7f62d9600640 1 -- 172.21.15.16:0/1360250315 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 8) ==== 3494+0+0 (secure 0 0 0) 0x7f62e40e7020 con 0x55a863f2ddc0 2024-03-31T15:03:51.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.409+0000 7f62f60de000 10 client.4656 fetch_fsmap finished waiting for FSMap version 8 2024-03-31T15:03:51.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.409+0000 7f62f60de000 10 client.4656 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-31T15:03:51.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.409+0000 7f62f60de000 20 client.4656 populate_metadata read hostname 'smithi016' 2024-03-31T15:03:51.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.409+0000 7f62f60de000 1 --2- 172.21.15.16:0/1360250315 >> [v2:172.21.15.89:6836/2583453131,v1:172.21.15.89:6837/2583453131] conn(0x55a863f3fa20 0x55a863f3fe00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:51.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.409+0000 7f62f60de000 4 client.4656 mds_command: new command op to 4293 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-31T15:03:51.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.409+0000 7f62f60de000 1 -- 172.21.15.16:0/1360250315 --> [v2:172.21.15.89:6836/2583453131,v1:172.21.15.89:6837/2583453131] -- command(tid 0: {"prefix": "status"}) -- 0x55a863d40e90 con 0x55a863f3fa20 2024-03-31T15:03:51.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.409+0000 7f62ebe00640 1 --2- 172.21.15.16:0/1360250315 >> [v2:172.21.15.89:6836/2583453131,v1:172.21.15.89:6837/2583453131] conn(0x55a863f3fa20 0x55a863f3fe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:51.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.409+0000 7f62d9600640 10 client.4656 ms_handle_connect on v2:172.21.15.89:6836/2583453131 2024-03-31T15:03:51.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.409+0000 7f62ebe00640 1 --2- 172.21.15.16:0/1360250315 >> [v2:172.21.15.89:6836/2583453131,v1:172.21.15.89:6837/2583453131] conn(0x55a863f3fa20 0x55a863f3fe00 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b8817e82192de206 server_cookie=466d69a658b1cae in_seq=0 out_seq=0 2024-03-31T15:03:51.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.409+0000 7f62d9600640 1 -- 172.21.15.16:0/1360250315 <== mds.0 v2:172.21.15.89:6836/2583453131 1 ==== command_reply(tid 0: 0 ) ==== 8+0+324 (crc 0 0 0) 0x55a863d40e90 con 0x55a863f3fa20 2024-03-31T15:03:51.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.409+0000 7f62d9600640 10 client.4656 handle_command_reply: tid=0 2024-03-31T15:03:51.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.409+0000 7f62f60de000 1 client.4656 shutdown 2024-03-31T15:03:51.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.413+0000 7f62f60de000 1 -- 172.21.15.16:0/1360250315 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a863f2ddc0 msgr2=0x55a863ee0860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:51.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.413+0000 7f62f60de000 1 --2- 172.21.15.16:0/1360250315 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a863f2ddc0 0x55a863ee0860 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f62e4098560 tx=0x7f62e40a7070 comp rx=0 tx=0).stop 2024-03-31T15:03:51.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.413+0000 7f62f60de000 1 -- 172.21.15.16:0/1360250315 shutdown_connections 2024-03-31T15:03:51.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.413+0000 7f62f60de000 1 --2- 172.21.15.16:0/1360250315 >> [v2:172.21.15.89:6836/2583453131,v1:172.21.15.89:6837/2583453131] conn(0x55a863f3fa20 0x55a863f3fe00 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:51.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.413+0000 7f62f60de000 1 --2- 172.21.15.16:0/1360250315 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a863f2ddc0 0x55a863ee0860 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:51.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.413+0000 7f62f60de000 1 --2- 172.21.15.16:0/1360250315 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a863f2d4a0 0x55a863ee0320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:51.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.413+0000 7f62f60de000 1 --2- 172.21.15.16:0/1360250315 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55a863ee10b0 0x55a863f997c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:51.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.413+0000 7f62d9600640 0 client.4656 ms_handle_reset on v2:172.21.15.89:6836/2583453131 2024-03-31T15:03:51.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.413+0000 7f62f60de000 1 -- 172.21.15.16:0/1360250315 >> 172.21.15.16:0/1360250315 conn(0x55a863dcbb80 msgr2=0x55a863f0f7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:03:51.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.413+0000 7f62f60de000 1 -- 172.21.15.16:0/1360250315 shutdown_connections 2024-03-31T15:03:51.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.413+0000 7f62f60de000 1 -- 172.21.15.16:0/1360250315 wait complete. 2024-03-31T15:03:51.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.413+0000 7f62f60de000 20 client.4656 trim_cache size 0 max 16384 2024-03-31T15:03:51.409 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-03-31T15:03:51.409 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_fsid": "286eb5da-fb47-4550-b290-d9519c13b9a5", 2024-03-31T15:03:51.409 INFO:teuthology.orchestra.run.smithi016.stdout: "whoami": 0, 2024-03-31T15:03:51.409 INFO:teuthology.orchestra.run.smithi016.stdout: "id": 4293, 2024-03-31T15:03:51.409 INFO:teuthology.orchestra.run.smithi016.stdout: "want_state": "up:active", 2024-03-31T15:03:51.409 INFO:teuthology.orchestra.run.smithi016.stdout: "state": "up:active", 2024-03-31T15:03:51.409 INFO:teuthology.orchestra.run.smithi016.stdout: "fs_name": "cephfs", 2024-03-31T15:03:51.409 INFO:teuthology.orchestra.run.smithi016.stdout: "rank_uptime": 14.453794689, 2024-03-31T15:03:51.409 INFO:teuthology.orchestra.run.smithi016.stdout: "mdsmap_epoch": 8, 2024-03-31T15:03:51.409 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch": 21, 2024-03-31T15:03:51.410 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch_barrier": 21, 2024-03-31T15:03:51.410 INFO:teuthology.orchestra.run.smithi016.stdout: "uptime": 21.651957792000001 2024-03-31T15:03:51.410 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-03-31T15:03:51.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.413+0000 7f62f2c00640 1 -- 172.21.15.16:0/4020542802 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f62c80683b0 msgr2=0x7f62c806a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:51.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.413+0000 7f62f2c00640 1 --2- 172.21.15.16:0/4020542802 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f62c80683b0 0x7f62c806a870 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f62dc007fd0 tx=0x7f62dc009040 comp rx=0 tx=0).stop 2024-03-31T15:03:51.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.413+0000 7f62f2c00640 1 -- 172.21.15.16:0/4020542802 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f62ec15b3d0 msgr2=0x7f62ec154630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:51.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.413+0000 7f62f2c00640 1 --2- 172.21.15.16:0/4020542802 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f62ec15b3d0 0x7f62ec154630 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f62d400c420 tx=0x7f62d400c8e0 comp rx=0 tx=0).stop 2024-03-31T15:03:51.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.413+0000 7f62f2c00640 1 -- 172.21.15.16:0/4020542802 shutdown_connections 2024-03-31T15:03:51.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.413+0000 7f62f2c00640 1 --2- 172.21.15.16:0/4020542802 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f62c80683b0 0x7f62c806a870 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:51.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.413+0000 7f62f2c00640 1 --2- 172.21.15.16:0/4020542802 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f62ec15eee0 0x7f62ec15b960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:51.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.413+0000 7f62f2c00640 1 --2- 172.21.15.16:0/4020542802 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f62ec15b3d0 0x7f62ec154630 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:51.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.413+0000 7f62f2c00640 1 --2- 172.21.15.16:0/4020542802 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f62ec0b23d0 0x7f62ec148a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:51.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.413+0000 7f62f2c00640 1 -- 172.21.15.16:0/4020542802 >> 172.21.15.16:0/4020542802 conn(0x7f62ec06b620 msgr2=0x7f62ec149320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:03:51.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.413+0000 7f62f2c00640 1 -- 172.21.15.16:0/4020542802 shutdown_connections 2024-03-31T15:03:51.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.413+0000 7f62f2c00640 1 -- 172.21.15.16:0/4020542802 wait complete. 2024-03-31T15:03:51.424 INFO:teuthology.run_tasks:Running task sequential... 2024-03-31T15:03:51.438 INFO:teuthology.task.sequential:In sequential, running task sequential... 2024-03-31T15:03:51.438 INFO:teuthology.task.sequential:In sequential, running task print... 2024-03-31T15:03:51.439 INFO:teuthology.task.print:Enabling mgr modules 2024-03-31T15:03:51.439 INFO:teuthology.run_tasks:Running task ceph-fuse... 2024-03-31T15:03:51.461 INFO:tasks.ceph_fuse:Running ceph_fuse task... 2024-03-31T15:03:51.461 INFO:tasks.ceph_fuse:config is {'client.0': {}, 'client.2': {}, 'client.1': {}, 'client.3': {}} 2024-03-31T15:03:51.461 INFO:tasks.ceph_fuse:client.0 config is {} 2024-03-31T15:03:51.461 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-03-31T15:03:51.461 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-31T15:03:51.461 INFO:tasks.ceph_fuse:client.2 config is {} 2024-03-31T15:03:51.462 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-03-31T15:03:51.462 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.2 2024-03-31T15:03:51.462 INFO:tasks.ceph_fuse:client.1 config is {} 2024-03-31T15:03:51.462 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-03-31T15:03:51.462 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-03-31T15:03:51.462 INFO:tasks.ceph_fuse:client.3 config is {} 2024-03-31T15:03:51.462 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-03-31T15:03:51.462 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.3 2024-03-31T15:03:51.462 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:03:51.462 DEBUG:teuthology.orchestra.run.smithi016:> ip netns list 2024-03-31T15:03:51.472 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:03:51.472 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip link delete ceph-brx 2024-03-31T15:03:51.538 INFO:teuthology.orchestra.run.smithi016.stderr:Cannot find device "ceph-brx" 2024-03-31T15:03:51.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-31T15:03:51.540 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:03:51.540 DEBUG:teuthology.orchestra.run.smithi089:> ip netns list 2024-03-31T15:03:51.549 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:03:51.549 DEBUG:teuthology.orchestra.run.smithi089:> sudo ip link delete ceph-brx 2024-03-31T15:03:51.611 INFO:teuthology.orchestra.run.smithi089.stderr:Cannot find device "ceph-brx" 2024-03-31T15:03:51.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-31T15:03:51.614 INFO:tasks.ceph_fuse:Mounting ceph-fuse clients... 2024-03-31T15:03:51.614 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-31T15:03:51.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.761+0000 7f720da00640 1 Processor -- start 2024-03-31T15:03:51.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.761+0000 7f720da00640 1 -- start start 2024-03-31T15:03:51.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.761+0000 7f720da00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f7208147e70 0x7f720806b620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:51.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.761+0000 7f720da00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f72080736e0 0x7f720815eb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:51.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.761+0000 7f720da00640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f72081621b0 con 0x7f72080d0c50 2024-03-31T15:03:51.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.761+0000 7f720da00640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f7208162490 con 0x7f72080736e0 2024-03-31T15:03:51.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.761+0000 7f720da00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f7208162770 con 0x7f7208147e70 2024-03-31T15:03:51.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.761+0000 7f7207400640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f72080736e0 0x7f720815eb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:51.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.761+0000 7f7206a00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f7208147e70 0x7f720806b620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:51.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.761+0000 7f7206a00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f7208147e70 0x7f720806b620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3301/0 says I am v2:172.21.15.16:51282/0 (socket says 172.21.15.16:51282) 2024-03-31T15:03:51.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.761+0000 7f71ffe00640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f72080d0c50 0x7f7208072fd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.16:58784/0 (socket says 172.21.15.16:58784) 2024-03-31T15:03:51.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.761+0000 7f7206a00640 1 -- 172.21.15.16:0/398476417 learned_addr learned my addr 172.21.15.16:0/398476417 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:03:51.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.761+0000 7f7207400640 1 -- 172.21.15.16:0/398476417 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f7208147e70 msgr2=0x7f720806b620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:03:51.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.761+0000 7f7207400640 1 --2- 172.21.15.16:0/398476417 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f7208147e70 0x7f720806b620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:51.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.761+0000 7f7207400640 1 -- 172.21.15.16:0/398476417 >> v1:172.21.15.89:6789/0 conn(0x7f72080d0c50 legacy=0x7f7208072fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:51.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.761+0000 7f7207400640 1 -- 172.21.15.16:0/398476417 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7208162a50 con 0x7f72080736e0 2024-03-31T15:03:51.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.761+0000 7f7206000640 1 -- 172.21.15.16:0/398476417 <== mon.1 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4026326166 0 0) 0x7f72081621b0 con 0x7f72080d0c50 2024-03-31T15:03:51.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.761+0000 7f7206a00640 1 --2- 172.21.15.16:0/398476417 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f7208147e70 0x7f720806b620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:03:51.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.765+0000 7f7207400640 1 --2- 172.21.15.16:0/398476417 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f72080736e0 0x7f720815eb20 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f7200009d90 tx=0x7f720001c970 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ad327d096696c018 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:03:51.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.765+0000 7f7206000640 1 -- 172.21.15.16:0/398476417 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72000b2070 con 0x7f72080736e0 2024-03-31T15:03:51.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.765+0000 7f7206000640 1 -- 172.21.15.16:0/398476417 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f72000a58b0 con 0x7f72080736e0 2024-03-31T15:03:51.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.765+0000 7f7206000640 1 -- 172.21.15.16:0/398476417 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72000a5c80 con 0x7f72080736e0 2024-03-31T15:03:51.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.765+0000 7f720da00640 1 -- 172.21.15.16:0/398476417 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f72080736e0 msgr2=0x7f720815eb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:51.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.765+0000 7f720da00640 1 --2- 172.21.15.16:0/398476417 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f72080736e0 0x7f720815eb20 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f7200009d90 tx=0x7f720001c970 comp rx=0 tx=0).stop 2024-03-31T15:03:51.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.765+0000 7f720da00640 1 -- 172.21.15.16:0/398476417 shutdown_connections 2024-03-31T15:03:51.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.765+0000 7f720da00640 1 --2- 172.21.15.16:0/398476417 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f72080736e0 0x7f720815eb20 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:51.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.765+0000 7f720da00640 1 --2- 172.21.15.16:0/398476417 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f7208147e70 0x7f720806b620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:51.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.765+0000 7f720da00640 1 -- 172.21.15.16:0/398476417 >> 172.21.15.16:0/398476417 conn(0x7f72080d1060 msgr2=0x7f72080d1470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:03:51.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.765+0000 7f720da00640 1 -- 172.21.15.16:0/398476417 shutdown_connections 2024-03-31T15:03:51.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.765+0000 7f720da00640 1 -- 172.21.15.16:0/398476417 wait complete. 2024-03-31T15:03:51.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.769+0000 7f720da00640 1 Processor -- start 2024-03-31T15:03:51.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.769+0000 7f720da00640 1 -- start start 2024-03-31T15:03:51.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.769+0000 7f720da00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f72080736e0 0x7f72080ad8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:51.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.769+0000 7f720da00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f72080d0c50 0x7f720815db20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:51.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.769+0000 7f720da00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7208147e70 0x7f72080b5630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:51.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.769+0000 7f720da00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f720815f9f0 con 0x7f72080736e0 2024-03-31T15:03:51.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.769+0000 7f720da00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f720815f870 con 0x7f7208147e70 2024-03-31T15:03:51.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.769+0000 7f720da00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f720816b4b0 con 0x7f72080d0c50 2024-03-31T15:03:51.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.769+0000 7f7206a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f72080736e0 0x7f72080ad8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:51.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.769+0000 7f7206a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f72080736e0 0x7f72080ad8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54036/0 (socket says 172.21.15.16:54036) 2024-03-31T15:03:51.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.769+0000 7f7206a00640 1 -- 172.21.15.16:0/3503701666 learned_addr learned my addr 172.21.15.16:0/3503701666 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:03:51.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.769+0000 7f7207400640 1 --2- 172.21.15.16:0/3503701666 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7208147e70 0x7f72080b5630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:51.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.769+0000 7f71ffe00640 1 --2- 172.21.15.16:0/3503701666 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f72080d0c50 0x7f720815db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:51.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.769+0000 7f7206a00640 1 -- 172.21.15.16:0/3503701666 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f72080d0c50 msgr2=0x7f720815db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:51.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.769+0000 7f7206a00640 1 --2- 172.21.15.16:0/3503701666 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f72080d0c50 0x7f720815db20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:51.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.769+0000 7f7206a00640 1 -- 172.21.15.16:0/3503701666 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7208147e70 msgr2=0x7f72080b5630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:51.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.769+0000 7f7206a00640 1 --2- 172.21.15.16:0/3503701666 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7208147e70 0x7f72080b5630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:51.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.769+0000 7f7206a00640 1 -- 172.21.15.16:0/3503701666 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7208146020 con 0x7f72080736e0 2024-03-31T15:03:51.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.769+0000 7f7207400640 1 --2- 172.21.15.16:0/3503701666 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7208147e70 0x7f72080b5630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:03:51.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.769+0000 7f7206a00640 1 --2- 172.21.15.16:0/3503701666 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f72080736e0 0x7f72080ad8d0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f71f800e930 tx=0x7f71f800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:03:51.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.769+0000 7f71fea00640 1 -- 172.21.15.16:0/3503701666 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71f8017070 con 0x7f72080736e0 2024-03-31T15:03:51.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.769+0000 7f71fea00640 1 -- 172.21.15.16:0/3503701666 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f71f800c9a0 con 0x7f72080736e0 2024-03-31T15:03:51.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.769+0000 7f71fea00640 1 -- 172.21.15.16:0/3503701666 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71f800cd70 con 0x7f72080736e0 2024-03-31T15:03:51.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.769+0000 7f720da00640 1 -- 172.21.15.16:0/3503701666 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7208161b80 con 0x7f72080736e0 2024-03-31T15:03:51.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.769+0000 7f720da00640 1 -- 172.21.15.16:0/3503701666 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f72081465a0 con 0x7f72080736e0 2024-03-31T15:03:51.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.773+0000 7f720da00640 1 -- 172.21.15.16:0/3503701666 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7208069950 con 0x7f72080736e0 2024-03-31T15:03:51.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.773+0000 7f71fea00640 1 -- 172.21.15.16:0/3503701666 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f71f801a070 con 0x7f72080736e0 2024-03-31T15:03:51.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.773+0000 7f71fea00640 1 --2- 172.21.15.16:0/3503701666 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f71e4068380 0x7f71e406a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:51.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.773+0000 7f71fea00640 1 -- 172.21.15.16:0/3503701666 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f71f8086ba0 con 0x7f72080736e0 2024-03-31T15:03:51.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.773+0000 7f71ffe00640 1 --2- 172.21.15.16:0/3503701666 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f71e4068380 0x7f71e406a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:51.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.773+0000 7f71ffe00640 1 --2- 172.21.15.16:0/3503701666 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f71e4068380 0x7f71e406a840 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f72080cd830 tx=0x7f71f0007d60 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:03:51.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.777+0000 7f71fea00640 1 -- 172.21.15.16:0/3503701666 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f71f8056a70 con 0x7f72080736e0 2024-03-31T15:03:51.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.981+0000 7f720da00640 1 -- 172.21.15.16:0/3503701666 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f72080d1bf0 con 0x7f72080736e0 2024-03-31T15:03:51.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.981+0000 7f71fea00640 1 -- 172.21.15.16:0/3503701666 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v8) ==== 53+0+73 (secure 0 0 0) 0x7f71f805ad70 con 0x7f72080736e0 2024-03-31T15:03:51.974 INFO:teuthology.orchestra.run.smithi016.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-31T15:03:51.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.981+0000 7f720da00640 1 -- 172.21.15.16:0/3503701666 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f71e4068380 msgr2=0x7f71e406a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:51.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.981+0000 7f720da00640 1 --2- 172.21.15.16:0/3503701666 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f71e4068380 0x7f71e406a840 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f72080cd830 tx=0x7f71f0007d60 comp rx=0 tx=0).stop 2024-03-31T15:03:51.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.981+0000 7f720da00640 1 -- 172.21.15.16:0/3503701666 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f72080736e0 msgr2=0x7f72080ad8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:51.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.981+0000 7f720da00640 1 --2- 172.21.15.16:0/3503701666 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f72080736e0 0x7f72080ad8d0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f71f800e930 tx=0x7f71f800edf0 comp rx=0 tx=0).stop 2024-03-31T15:03:51.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.981+0000 7f720da00640 1 -- 172.21.15.16:0/3503701666 shutdown_connections 2024-03-31T15:03:51.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.981+0000 7f720da00640 1 --2- 172.21.15.16:0/3503701666 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f71e4068380 0x7f71e406a840 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:51.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.981+0000 7f720da00640 1 --2- 172.21.15.16:0/3503701666 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7208147e70 0x7f72080b5630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:51.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.981+0000 7f720da00640 1 --2- 172.21.15.16:0/3503701666 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f72080d0c50 0x7f720815db20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:51.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.981+0000 7f720da00640 1 --2- 172.21.15.16:0/3503701666 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f72080736e0 0x7f72080ad8d0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:51.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.981+0000 7f720da00640 1 -- 172.21.15.16:0/3503701666 >> 172.21.15.16:0/3503701666 conn(0x7f72080d1060 msgr2=0x7f720806e600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:03:51.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.981+0000 7f720da00640 1 -- 172.21.15.16:0/3503701666 shutdown_connections 2024-03-31T15:03:51.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:51.981+0000 7f720da00640 1 -- 172.21.15.16:0/3503701666 wait complete. 2024-03-31T15:03:51.995 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-31T15:03:51.995 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-31T15:03:51.995 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi016.front.sepia.ceph.com 2024-03-31T15:03:51.995 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-31T15:03:51.995 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-31T15:03:51.995 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-31T15:03:51.995 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-31T15:03:51.995 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-31T15:03:51.995 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-31T15:03:51.995 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:03:51.995 DEBUG:teuthology.orchestra.run.smithi016:> ip addr 2024-03-31T15:03:52.005 INFO:teuthology.orchestra.run.smithi016.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-31T15:03:52.005 INFO:teuthology.orchestra.run.smithi016.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-31T15:03:52.005 INFO:teuthology.orchestra.run.smithi016.stdout: inet 127.0.0.1/8 scope host lo 2024-03-31T15:03:52.005 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2024-03-31T15:03:52.005 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 ::1/128 scope host 2024-03-31T15:03:52.005 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2024-03-31T15:03:52.005 INFO:teuthology.orchestra.run.smithi016.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-03-31T15:03:52.005 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:6b:fd:e0 brd ff:ff:ff:ff:ff:ff 2024-03-31T15:03:52.005 INFO:teuthology.orchestra.run.smithi016.stdout: altname enp7s0f0 2024-03-31T15:03:52.005 INFO:teuthology.orchestra.run.smithi016.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-03-31T15:03:52.005 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:6b:fd:e1 brd ff:ff:ff:ff:ff:ff 2024-03-31T15:03:52.005 INFO:teuthology.orchestra.run.smithi016.stdout: altname enp7s0f1 2024-03-31T15:03:52.006 INFO:teuthology.orchestra.run.smithi016.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-31T15:03:52.006 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:bd:15:64 brd ff:ff:ff:ff:ff:ff 2024-03-31T15:03:52.006 INFO:teuthology.orchestra.run.smithi016.stdout: altname enp1s0f0 2024-03-31T15:03:52.006 INFO:teuthology.orchestra.run.smithi016.stdout: inet 172.21.15.16/20 brd 172.21.15.255 scope global dynamic ens1f0 2024-03-31T15:03:52.006 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft 42449sec preferred_lft 42449sec 2024-03-31T15:03:52.006 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 fe80::ec4:7aff:febd:1564/64 scope link 2024-03-31T15:03:52.006 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2024-03-31T15:03:52.006 INFO:teuthology.orchestra.run.smithi016.stdout:5: ens1f1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-03-31T15:03:52.006 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:bd:15:65 brd ff:ff:ff:ff:ff:ff 2024-03-31T15:03:52.006 INFO:teuthology.orchestra.run.smithi016.stdout: altname enp1s0f1 2024-03-31T15:03:52.006 INFO:teuthology.orchestra.run.smithi016.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2024-03-31T15:03:52.006 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 02:42:d4:1f:7d:73 brd ff:ff:ff:ff:ff:ff 2024-03-31T15:03:52.006 INFO:teuthology.orchestra.run.smithi016.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2024-03-31T15:03:52.007 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2024-03-31T15:03:52.007 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-03-31T15:03:52.007 DEBUG:teuthology.orchestra.run.smithi016:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-31T15:03:52.007 DEBUG:teuthology.orchestra.run.smithi016:> set -e 2024-03-31T15:03:52.007 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip link add name ceph-brx type bridge 2024-03-31T15:03:52.007 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip addr flush dev ceph-brx 2024-03-31T15:03:52.007 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip link set ceph-brx up 2024-03-31T15:03:52.008 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-03-31T15:03:52.008 DEBUG:teuthology.orchestra.run.smithi016:> ') 2024-03-31T15:03:52.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:52 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-31T15:03:52.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:52 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-31T15:03:52.161 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:03:52.161 DEBUG:teuthology.orchestra.run.smithi016:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-31T15:03:52.180 INFO:teuthology.orchestra.run.smithi016.stdout:1 2024-03-31T15:03:52.182 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:03:52.182 DEBUG:teuthology.orchestra.run.smithi016:> route 2024-03-31T15:03:52.262 INFO:teuthology.orchestra.run.smithi016.stdout:Kernel IP routing table 2024-03-31T15:03:52.262 INFO:teuthology.orchestra.run.smithi016.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-31T15:03:52.262 INFO:teuthology.orchestra.run.smithi016.stdout:default _gateway 0.0.0.0 UG 0 0 0 ens1f0 2024-03-31T15:03:52.262 INFO:teuthology.orchestra.run.smithi016.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2024-03-31T15:03:52.262 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 0 0 0 ens1f0 2024-03-31T15:03:52.262 INFO:teuthology.orchestra.run.smithi016.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-31T15:03:52.263 DEBUG:teuthology.orchestra.run.smithi016:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-31T15:03:52.263 DEBUG:teuthology.orchestra.run.smithi016:> set -e 2024-03-31T15:03:52.263 DEBUG:teuthology.orchestra.run.smithi016:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-03-31T15:03:52.263 DEBUG:teuthology.orchestra.run.smithi016:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-03-31T15:03:52.263 DEBUG:teuthology.orchestra.run.smithi016:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-03-31T15:03:52.263 DEBUG:teuthology.orchestra.run.smithi016:> ') 2024-03-31T15:03:52.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:52 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-31T15:03:52.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:52 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-31T15:03:52.399 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:03:52.399 DEBUG:teuthology.orchestra.run.smithi016:> ip netns list 2024-03-31T15:03:52.408 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:03:52.409 DEBUG:teuthology.orchestra.run.smithi016:> ip netns list-id 2024-03-31T15:03:52.460 DEBUG:teuthology.orchestra.run.smithi016:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-31T15:03:52.460 DEBUG:teuthology.orchestra.run.smithi016:> set -e 2024-03-31T15:03:52.460 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-31T15:03:52.461 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-03-31T15:03:52.461 DEBUG:teuthology.orchestra.run.smithi016:> ') 2024-03-31T15:03:52.554 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:52 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-31T15:03:52.587 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:52 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-31T15:03:52.594 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-03-31T15:03:52.594 DEBUG:teuthology.orchestra.run.smithi016:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-31T15:03:52.594 DEBUG:teuthology.orchestra.run.smithi016:> set -e 2024-03-31T15:03:52.595 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-03-31T15:03:52.595 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-03-31T15:03:52.595 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-03-31T15:03:52.595 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-03-31T15:03:52.595 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-03-31T15:03:52.595 DEBUG:teuthology.orchestra.run.smithi016:> ') 2024-03-31T15:03:52.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:52 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-31T15:03:52.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:52 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-31T15:03:52.767 DEBUG:teuthology.orchestra.run.smithi016:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-31T15:03:52.767 DEBUG:teuthology.orchestra.run.smithi016:> set -e 2024-03-31T15:03:52.767 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip link set brx.0 up 2024-03-31T15:03:52.767 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip link set dev brx.0 master ceph-brx 2024-03-31T15:03:52.767 DEBUG:teuthology.orchestra.run.smithi016:> ') 2024-03-31T15:03:52.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:52 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-31T15:03:52.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:52 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-31T15:03:52.852 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-03-31T15:03:52.853 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-31T15:03:52.853 DEBUG:teuthology.orchestra.run.smithi016:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-31T15:03:52.862 INFO:teuthology.orchestra.run.smithi016.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-31T15:03:52.862 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-31T15:03:52.862 DEBUG:teuthology.orchestra.run.smithi016:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-31T15:03:52.911 DEBUG:teuthology.orchestra.run.smithi016:> sudo modprobe fuse 2024-03-31T15:03:52.974 DEBUG:teuthology.orchestra.run.smithi016:> cat /proc/self/mounts | awk '{print $2}' 2024-03-31T15:03:53.025 INFO:teuthology.orchestra.run.smithi016.stdout:/sys 2024-03-31T15:03:53.025 INFO:teuthology.orchestra.run.smithi016.stdout:/proc 2024-03-31T15:03:53.025 INFO:teuthology.orchestra.run.smithi016.stdout:/dev 2024-03-31T15:03:53.025 INFO:teuthology.orchestra.run.smithi016.stdout:/dev/pts 2024-03-31T15:03:53.026 INFO:teuthology.orchestra.run.smithi016.stdout:/run 2024-03-31T15:03:53.026 INFO:teuthology.orchestra.run.smithi016.stdout:/ 2024-03-31T15:03:53.026 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/kernel/security 2024-03-31T15:03:53.026 INFO:teuthology.orchestra.run.smithi016.stdout:/dev/shm 2024-03-31T15:03:53.026 INFO:teuthology.orchestra.run.smithi016.stdout:/run/lock 2024-03-31T15:03:53.026 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/cgroup 2024-03-31T15:03:53.026 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/pstore 2024-03-31T15:03:53.026 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/bpf 2024-03-31T15:03:53.026 INFO:teuthology.orchestra.run.smithi016.stdout:/proc/sys/fs/binfmt_misc 2024-03-31T15:03:53.026 INFO:teuthology.orchestra.run.smithi016.stdout:/dev/hugepages 2024-03-31T15:03:53.026 INFO:teuthology.orchestra.run.smithi016.stdout:/dev/mqueue 2024-03-31T15:03:53.026 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/kernel/debug 2024-03-31T15:03:53.026 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/kernel/tracing 2024-03-31T15:03:53.027 INFO:teuthology.orchestra.run.smithi016.stdout:/proc/fs/nfsd 2024-03-31T15:03:53.027 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/fuse/connections 2024-03-31T15:03:53.027 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/kernel/config 2024-03-31T15:03:53.027 INFO:teuthology.orchestra.run.smithi016.stdout:/run/credentials/systemd-sysusers.service 2024-03-31T15:03:53.027 INFO:teuthology.orchestra.run.smithi016.stdout:/run/rpc_pipefs 2024-03-31T15:03:53.027 INFO:teuthology.orchestra.run.smithi016.stdout:/run/user/1000 2024-03-31T15:03:53.027 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/ceph 2024-03-31T15:03:53.027 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/ceph/osd/ceph-0 2024-03-31T15:03:53.027 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/ceph/osd/ceph-1 2024-03-31T15:03:53.027 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/ceph/osd/ceph-2 2024-03-31T15:03:53.027 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/ceph/osd/ceph-3 2024-03-31T15:03:53.028 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns 2024-03-31T15:03:53.028 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-31T15:03:53.028 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-31T15:03:53.029 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:03:53.029 DEBUG:teuthology.orchestra.run.smithi016:> ls /sys/fs/fuse/connections 2024-03-31T15:03:53.083 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-31T15:03:53.084 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-03-31T15:03:53.130 DEBUG:teuthology.orchestra.run.smithi016:> sudo modprobe fuse 2024-03-31T15:03:53.151 DEBUG:teuthology.orchestra.run.smithi016:> cat /proc/self/mounts | awk '{print $2}' 2024-03-31T15:03:53.201 INFO:teuthology.orchestra.run.smithi016.stdout:/sys 2024-03-31T15:03:53.201 INFO:teuthology.orchestra.run.smithi016.stdout:/proc 2024-03-31T15:03:53.201 INFO:teuthology.orchestra.run.smithi016.stdout:/dev 2024-03-31T15:03:53.201 INFO:teuthology.orchestra.run.smithi016.stdout:/dev/pts 2024-03-31T15:03:53.201 INFO:teuthology.orchestra.run.smithi016.stdout:/run 2024-03-31T15:03:53.201 INFO:teuthology.orchestra.run.smithi016.stdout:/ 2024-03-31T15:03:53.202 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/kernel/security 2024-03-31T15:03:53.202 INFO:teuthology.orchestra.run.smithi016.stdout:/dev/shm 2024-03-31T15:03:53.202 INFO:teuthology.orchestra.run.smithi016.stdout:/run/lock 2024-03-31T15:03:53.202 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/cgroup 2024-03-31T15:03:53.202 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/pstore 2024-03-31T15:03:53.202 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/bpf 2024-03-31T15:03:53.202 INFO:teuthology.orchestra.run.smithi016.stdout:/proc/sys/fs/binfmt_misc 2024-03-31T15:03:53.202 INFO:teuthology.orchestra.run.smithi016.stdout:/dev/hugepages 2024-03-31T15:03:53.202 INFO:teuthology.orchestra.run.smithi016.stdout:/dev/mqueue 2024-03-31T15:03:53.202 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/kernel/debug 2024-03-31T15:03:53.202 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/kernel/tracing 2024-03-31T15:03:53.202 INFO:teuthology.orchestra.run.smithi016.stdout:/proc/fs/nfsd 2024-03-31T15:03:53.202 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/fuse/connections 2024-03-31T15:03:53.202 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/kernel/config 2024-03-31T15:03:53.202 INFO:teuthology.orchestra.run.smithi016.stdout:/run/credentials/systemd-sysusers.service 2024-03-31T15:03:53.203 INFO:teuthology.orchestra.run.smithi016.stdout:/run/rpc_pipefs 2024-03-31T15:03:53.203 INFO:teuthology.orchestra.run.smithi016.stdout:/run/user/1000 2024-03-31T15:03:53.203 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/ceph 2024-03-31T15:03:53.203 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/ceph/osd/ceph-0 2024-03-31T15:03:53.203 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/ceph/osd/ceph-1 2024-03-31T15:03:53.203 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/ceph/osd/ceph-2 2024-03-31T15:03:53.203 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/ceph/osd/ceph-3 2024-03-31T15:03:53.203 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns 2024-03-31T15:03:53.203 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-31T15:03:53.203 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-31T15:03:53.204 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:03:53.204 DEBUG:teuthology.orchestra.run.smithi016:> ls /sys/fs/fuse/connections 2024-03-31T15:03:53.225 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi016.stderr:2024-03-31T15:03:53.233+0000 7f8fdc4cc580 -1 init, newargv = 0x55ba5b0b2900 newargc=15 2024-03-31T15:03:53.226 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi016.stderr:ceph-fuse[33298]: starting ceph client 2024-03-31T15:03:53.234 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi016.stderr:ceph-fuse[33298]: starting fuse 2024-03-31T15:03:54.336 DEBUG:teuthology.orchestra.run.smithi016:> sudo modprobe fuse 2024-03-31T15:03:54.361 DEBUG:teuthology.orchestra.run.smithi016:> cat /proc/self/mounts | awk '{print $2}' 2024-03-31T15:03:54.415 INFO:teuthology.orchestra.run.smithi016.stdout:/sys 2024-03-31T15:03:54.415 INFO:teuthology.orchestra.run.smithi016.stdout:/proc 2024-03-31T15:03:54.415 INFO:teuthology.orchestra.run.smithi016.stdout:/dev 2024-03-31T15:03:54.415 INFO:teuthology.orchestra.run.smithi016.stdout:/dev/pts 2024-03-31T15:03:54.415 INFO:teuthology.orchestra.run.smithi016.stdout:/run 2024-03-31T15:03:54.415 INFO:teuthology.orchestra.run.smithi016.stdout:/ 2024-03-31T15:03:54.415 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/kernel/security 2024-03-31T15:03:54.415 INFO:teuthology.orchestra.run.smithi016.stdout:/dev/shm 2024-03-31T15:03:54.415 INFO:teuthology.orchestra.run.smithi016.stdout:/run/lock 2024-03-31T15:03:54.415 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/cgroup 2024-03-31T15:03:54.415 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/pstore 2024-03-31T15:03:54.415 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/bpf 2024-03-31T15:03:54.415 INFO:teuthology.orchestra.run.smithi016.stdout:/proc/sys/fs/binfmt_misc 2024-03-31T15:03:54.416 INFO:teuthology.orchestra.run.smithi016.stdout:/dev/hugepages 2024-03-31T15:03:54.416 INFO:teuthology.orchestra.run.smithi016.stdout:/dev/mqueue 2024-03-31T15:03:54.416 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/kernel/debug 2024-03-31T15:03:54.416 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/kernel/tracing 2024-03-31T15:03:54.416 INFO:teuthology.orchestra.run.smithi016.stdout:/proc/fs/nfsd 2024-03-31T15:03:54.416 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/fuse/connections 2024-03-31T15:03:54.416 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/kernel/config 2024-03-31T15:03:54.416 INFO:teuthology.orchestra.run.smithi016.stdout:/run/credentials/systemd-sysusers.service 2024-03-31T15:03:54.416 INFO:teuthology.orchestra.run.smithi016.stdout:/run/rpc_pipefs 2024-03-31T15:03:54.416 INFO:teuthology.orchestra.run.smithi016.stdout:/run/user/1000 2024-03-31T15:03:54.416 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/ceph 2024-03-31T15:03:54.416 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/ceph/osd/ceph-0 2024-03-31T15:03:54.416 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/ceph/osd/ceph-1 2024-03-31T15:03:54.416 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/ceph/osd/ceph-2 2024-03-31T15:03:54.416 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/ceph/osd/ceph-3 2024-03-31T15:03:54.417 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns 2024-03-31T15:03:54.417 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-31T15:03:54.417 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-31T15:03:54.417 INFO:teuthology.orchestra.run.smithi016.stdout:/home/ubuntu/cephtest/mnt.0 2024-03-31T15:03:54.418 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:03:54.418 DEBUG:teuthology.orchestra.run.smithi016:> ls /sys/fs/fuse/connections 2024-03-31T15:03:54.469 INFO:teuthology.orchestra.run.smithi016.stdout:43 2024-03-31T15:03:54.469 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [43] 2024-03-31T15:03:54.469 DEBUG:teuthology.orchestra.run.smithi016:> sudo stdin-killer -- python3 -c ' 2024-03-31T15:03:54.469 DEBUG:teuthology.orchestra.run.smithi016:> import glob 2024-03-31T15:03:54.469 DEBUG:teuthology.orchestra.run.smithi016:> import re 2024-03-31T15:03:54.469 DEBUG:teuthology.orchestra.run.smithi016:> import os 2024-03-31T15:03:54.469 DEBUG:teuthology.orchestra.run.smithi016:> import subprocess 2024-03-31T15:03:54.469 DEBUG:teuthology.orchestra.run.smithi016:> 2024-03-31T15:03:54.470 DEBUG:teuthology.orchestra.run.smithi016:> def _find_admin_socket(client_name): 2024-03-31T15:03:54.470 DEBUG:teuthology.orchestra.run.smithi016:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-03-31T15:03:54.470 DEBUG:teuthology.orchestra.run.smithi016:> files = glob.glob(asok_path) 2024-03-31T15:03:54.470 DEBUG:teuthology.orchestra.run.smithi016:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-03-31T15:03:54.470 DEBUG:teuthology.orchestra.run.smithi016:> 2024-03-31T15:03:54.470 DEBUG:teuthology.orchestra.run.smithi016:> # Given a non-glob path, it better be there 2024-03-31T15:03:54.470 DEBUG:teuthology.orchestra.run.smithi016:> if "*" not in asok_path: 2024-03-31T15:03:54.470 DEBUG:teuthology.orchestra.run.smithi016:> assert(len(files) == 1) 2024-03-31T15:03:54.470 DEBUG:teuthology.orchestra.run.smithi016:> return files[0] 2024-03-31T15:03:54.470 DEBUG:teuthology.orchestra.run.smithi016:> 2024-03-31T15:03:54.470 DEBUG:teuthology.orchestra.run.smithi016:> for f in files: 2024-03-31T15:03:54.470 DEBUG:teuthology.orchestra.run.smithi016:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-31T15:03:54.470 DEBUG:teuthology.orchestra.run.smithi016:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-31T15:03:54.470 DEBUG:teuthology.orchestra.run.smithi016:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-31T15:03:54.470 DEBUG:teuthology.orchestra.run.smithi016:> contents = proc_f.read() 2024-03-31T15:03:54.471 DEBUG:teuthology.orchestra.run.smithi016:> if mountpoint in contents: 2024-03-31T15:03:54.471 DEBUG:teuthology.orchestra.run.smithi016:> return f 2024-03-31T15:03:54.471 DEBUG:teuthology.orchestra.run.smithi016:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-31T15:03:54.471 DEBUG:teuthology.orchestra.run.smithi016:> 2024-03-31T15:03:54.471 DEBUG:teuthology.orchestra.run.smithi016:> print(_find_admin_socket("client.0")) 2024-03-31T15:03:54.471 DEBUG:teuthology.orchestra.run.smithi016:> ' 2024-03-31T15:03:54.604 INFO:teuthology.orchestra.run.smithi016.stdout:/var/run/ceph/ceph-client.0.33298.asok 2024-03-31T15:03:54.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:54 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-31T15:03:54.616 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.33298.asok 2024-03-31T15:03:54.616 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:03:54.616 DEBUG:teuthology.orchestra.run.smithi016:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.33298.asok status 2024-03-31T15:03:54.794 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-03-31T15:03:54.794 INFO:teuthology.orchestra.run.smithi016.stdout: "metadata": { 2024-03-31T15:03:54.794 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph_sha1": "d2271579d5e735cfd6d436b8349c004f06527cb0", 2024-03-31T15:03:54.794 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph_version": "ceph version 19.0.0-2574-gd2271579 (d2271579d5e735cfd6d436b8349c004f06527cb0) squid (dev)", 2024-03-31T15:03:54.794 INFO:teuthology.orchestra.run.smithi016.stdout: "entity_id": "0", 2024-03-31T15:03:54.795 INFO:teuthology.orchestra.run.smithi016.stdout: "hostname": "smithi016", 2024-03-31T15:03:54.795 INFO:teuthology.orchestra.run.smithi016.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-03-31T15:03:54.795 INFO:teuthology.orchestra.run.smithi016.stdout: "pid": "33298", 2024-03-31T15:03:54.795 INFO:teuthology.orchestra.run.smithi016.stdout: "root": "/" 2024-03-31T15:03:54.795 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-31T15:03:54.795 INFO:teuthology.orchestra.run.smithi016.stdout: "dentry_count": 0, 2024-03-31T15:03:54.795 INFO:teuthology.orchestra.run.smithi016.stdout: "dentry_pinned_count": 0, 2024-03-31T15:03:54.795 INFO:teuthology.orchestra.run.smithi016.stdout: "id": 4681, 2024-03-31T15:03:54.795 INFO:teuthology.orchestra.run.smithi016.stdout: "inst": { 2024-03-31T15:03:54.795 INFO:teuthology.orchestra.run.smithi016.stdout: "name": { 2024-03-31T15:03:54.795 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "client", 2024-03-31T15:03:54.795 INFO:teuthology.orchestra.run.smithi016.stdout: "num": 4681 2024-03-31T15:03:54.795 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-31T15:03:54.796 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": { 2024-03-31T15:03:54.796 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-03-31T15:03:54.796 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:0", 2024-03-31T15:03:54.796 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 282842177 2024-03-31T15:03:54.796 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-31T15:03:54.796 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-31T15:03:54.796 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": { 2024-03-31T15:03:54.796 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-03-31T15:03:54.796 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:0", 2024-03-31T15:03:54.796 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 282842177 2024-03-31T15:03:54.796 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-31T15:03:54.796 INFO:teuthology.orchestra.run.smithi016.stdout: "inst_str": "client.4681 172.21.15.16:0/282842177", 2024-03-31T15:03:54.796 INFO:teuthology.orchestra.run.smithi016.stdout: "addr_str": "172.21.15.16:0/282842177", 2024-03-31T15:03:54.796 INFO:teuthology.orchestra.run.smithi016.stdout: "inode_count": 1, 2024-03-31T15:03:54.797 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_epoch": 8, 2024-03-31T15:03:54.797 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_epoch": 21, 2024-03-31T15:03:54.797 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_epoch_barrier": 0, 2024-03-31T15:03:54.797 INFO:teuthology.orchestra.run.smithi016.stdout: "blocklisted": false, 2024-03-31T15:03:54.797 INFO:teuthology.orchestra.run.smithi016.stdout: "fs_name": "cephfs" 2024-03-31T15:03:54.797 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-03-31T15:03:54.809 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-31T15:03:54.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:54.957+0000 7eff83e00640 1 Processor -- start 2024-03-31T15:03:54.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:54.957+0000 7eff83e00640 1 -- start start 2024-03-31T15:03:54.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:54.957+0000 7eff83e00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7eff8415c1d0 0x7eff840b44e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:54.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:54.957+0000 7eff83e00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7eff840cd4f0 0x7eff8406f5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:54.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:54.957+0000 7eff83e00640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7eff840cd980 con 0x7eff8406fb10 2024-03-31T15:03:54.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:54.957+0000 7eff83e00640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7eff840cfaa0 con 0x7eff8415c1d0 2024-03-31T15:03:54.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:54.957+0000 7eff83e00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7eff840cdc60 con 0x7eff840cd4f0 2024-03-31T15:03:54.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:54.957+0000 7eff7be00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7eff840cd4f0 0x7eff8406f5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:54.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:54.957+0000 7eff7be00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7eff840cd4f0 0x7eff8406f5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3301/0 says I am v2:172.21.15.16:51302/0 (socket says 172.21.15.16:51302) 2024-03-31T15:03:54.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:54.957+0000 7eff7be00640 1 -- 172.21.15.16:0/4129087942 learned_addr learned my addr 172.21.15.16:0/4129087942 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:03:54.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:54.957+0000 7eff82a00640 1 --2- 172.21.15.16:0/4129087942 >> v2:172.21.15.89:3300/0 conn(0x7eff8415c1d0 0x7eff840b44e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:54.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:54.957+0000 7eff7be00640 1 -- 172.21.15.16:0/4129087942 >> v2:172.21.15.89:3300/0 conn(0x7eff8415c1d0 msgr2=0x7eff840b44e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:03:54.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:54.957+0000 7eff7be00640 1 --2- 172.21.15.16:0/4129087942 >> v2:172.21.15.89:3300/0 conn(0x7eff8415c1d0 0x7eff840b44e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:54.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:54.957+0000 7eff7be00640 1 -- 172.21.15.16:0/4129087942 >> v1:172.21.15.89:6789/0 conn(0x7eff8406fb10 legacy=0x7eff8415fd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:54.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:54.957+0000 7eff82000640 1 -- 172.21.15.16:0/4129087942 <== mon.1 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 359240568 0 0) 0x7eff840cd980 con 0x7eff8406fb10 2024-03-31T15:03:54.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:54.957+0000 7eff7be00640 1 -- 172.21.15.16:0/4129087942 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff840cfcf0 con 0x7eff840cd4f0 2024-03-31T15:03:54.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:54.961+0000 7eff7be00640 1 --2- 172.21.15.16:0/4129087942 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7eff840cd4f0 0x7eff8406f5d0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7eff6c009870 tx=0x7eff6c01c530 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7a89dd477817531b server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:03:54.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:54.961+0000 7eff82000640 1 -- 172.21.15.16:0/4129087942 <== mon.2 v2:172.21.15.89:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff6c0aa430 con 0x7eff840cd4f0 2024-03-31T15:03:54.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:54.961+0000 7eff82000640 1 -- 172.21.15.16:0/4129087942 <== mon.2 v2:172.21.15.89:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7eff6c0af070 con 0x7eff840cd4f0 2024-03-31T15:03:54.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:54.961+0000 7eff82000640 1 -- 172.21.15.16:0/4129087942 <== mon.2 v2:172.21.15.89:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff6c0aa7f0 con 0x7eff840cd4f0 2024-03-31T15:03:54.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:54.961+0000 7eff83e00640 1 -- 172.21.15.16:0/4129087942 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7eff840cd4f0 msgr2=0x7eff8406f5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:54.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:54.961+0000 7eff83e00640 1 --2- 172.21.15.16:0/4129087942 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7eff840cd4f0 0x7eff8406f5d0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7eff6c009870 tx=0x7eff6c01c530 comp rx=0 tx=0).stop 2024-03-31T15:03:54.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:54.961+0000 7eff83e00640 1 -- 172.21.15.16:0/4129087942 shutdown_connections 2024-03-31T15:03:54.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:54.961+0000 7eff83e00640 1 --2- 172.21.15.16:0/4129087942 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7eff840cd4f0 0x7eff8406f5d0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:54.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:54.961+0000 7eff83e00640 1 --2- 172.21.15.16:0/4129087942 >> v2:172.21.15.89:3300/0 conn(0x7eff8415c1d0 0x7eff840b44e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:54.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:54.961+0000 7eff83e00640 1 -- 172.21.15.16:0/4129087942 >> 172.21.15.16:0/4129087942 conn(0x7eff840b39c0 msgr2=0x7eff840b3dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:03:54.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:54.961+0000 7eff83e00640 1 -- 172.21.15.16:0/4129087942 shutdown_connections 2024-03-31T15:03:54.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:54.961+0000 7eff83e00640 1 -- 172.21.15.16:0/4129087942 wait complete. 2024-03-31T15:03:54.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:54.965+0000 7eff83e00640 1 Processor -- start 2024-03-31T15:03:54.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:54.965+0000 7eff83e00640 1 -- start start 2024-03-31T15:03:54.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:54.965+0000 7eff83e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7eff8406fb10 0x7eff841556b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:54.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:54.965+0000 7eff83e00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7eff8415c1d0 0x7eff840c99b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:54.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:54.965+0000 7eff83e00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7eff8415a7a0 0x7eff840b5000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:54.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:54.965+0000 7eff83e00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7eff840d0e80 con 0x7eff8406fb10 2024-03-31T15:03:54.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:54.965+0000 7eff83e00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7eff840d0d00 con 0x7eff8415a7a0 2024-03-31T15:03:54.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:54.965+0000 7eff83e00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7eff84160450 con 0x7eff8415c1d0 2024-03-31T15:03:54.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:54.965+0000 7eff82a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7eff8406fb10 0x7eff841556b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:54.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:54.965+0000 7eff82a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7eff8406fb10 0x7eff841556b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54040/0 (socket says 172.21.15.16:54040) 2024-03-31T15:03:54.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:54.965+0000 7eff82a00640 1 -- 172.21.15.16:0/2690592450 learned_addr learned my addr 172.21.15.16:0/2690592450 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:03:54.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:54.965+0000 7eff7be00640 1 --2- 172.21.15.16:0/2690592450 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7eff8415c1d0 0x7eff840c99b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:54.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:54.965+0000 7eff83400640 1 --2- 172.21.15.16:0/2690592450 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7eff8415a7a0 0x7eff840b5000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:54.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:54.965+0000 7eff7be00640 1 -- 172.21.15.16:0/2690592450 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7eff8415a7a0 msgr2=0x7eff840b5000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:54.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:54.965+0000 7eff7be00640 1 --2- 172.21.15.16:0/2690592450 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7eff8415a7a0 0x7eff840b5000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:54.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:54.965+0000 7eff7be00640 1 -- 172.21.15.16:0/2690592450 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7eff8406fb10 msgr2=0x7eff841556b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:54.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:54.965+0000 7eff7be00640 1 --2- 172.21.15.16:0/2690592450 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7eff8406fb10 0x7eff841556b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:54.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:54.965+0000 7eff7be00640 1 -- 172.21.15.16:0/2690592450 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff840cc100 con 0x7eff8415c1d0 2024-03-31T15:03:54.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:54.965+0000 7eff82a00640 1 --2- 172.21.15.16:0/2690592450 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7eff8406fb10 0x7eff841556b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:03:54.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:54.965+0000 7eff7be00640 1 --2- 172.21.15.16:0/2690592450 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7eff8415c1d0 0x7eff840c99b0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7eff6c0ab000 tx=0x7eff6c01ad70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:03:54.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:54.965+0000 7eff7aa00640 1 -- 172.21.15.16:0/2690592450 <== mon.2 v2:172.21.15.89:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff6c0bd630 con 0x7eff8415c1d0 2024-03-31T15:03:54.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:54.965+0000 7eff7aa00640 1 -- 172.21.15.16:0/2690592450 <== mon.2 v2:172.21.15.89:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7eff6c0bd7d0 con 0x7eff8415c1d0 2024-03-31T15:03:54.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:54.965+0000 7eff7aa00640 1 -- 172.21.15.16:0/2690592450 <== mon.2 v2:172.21.15.89:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff6c0bdbc0 con 0x7eff8415c1d0 2024-03-31T15:03:54.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:54.965+0000 7eff83e00640 1 -- 172.21.15.16:0/2690592450 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7eff840cb580 con 0x7eff8415c1d0 2024-03-31T15:03:54.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:54.965+0000 7eff83e00640 1 -- 172.21.15.16:0/2690592450 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7eff840cbb40 con 0x7eff8415c1d0 2024-03-31T15:03:54.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:54.965+0000 7eff83e00640 1 -- 172.21.15.16:0/2690592450 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7eff8404f8c0 con 0x7eff8415c1d0 2024-03-31T15:03:54.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:54.969+0000 7eff7aa00640 1 -- 172.21.15.16:0/2690592450 <== mon.2 v2:172.21.15.89:3301/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7eff6c0dbbb0 con 0x7eff8415c1d0 2024-03-31T15:03:54.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:54.969+0000 7eff7aa00640 1 --2- 172.21.15.16:0/2690592450 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7eff60068380 0x7eff6006a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:54.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:54.969+0000 7eff7aa00640 1 -- 172.21.15.16:0/2690592450 <== mon.2 v2:172.21.15.89:3301/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7eff6c120e70 con 0x7eff8415c1d0 2024-03-31T15:03:54.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:54.969+0000 7eff82a00640 1 --2- 172.21.15.16:0/2690592450 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7eff60068380 0x7eff6006a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:54.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:54.969+0000 7eff82a00640 1 --2- 172.21.15.16:0/2690592450 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7eff60068380 0x7eff6006a840 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7eff841f8440 tx=0x7eff74009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:03:54.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:54.973+0000 7eff7aa00640 1 -- 172.21.15.16:0/2690592450 <== mon.2 v2:172.21.15.89:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7eff6c016150 con 0x7eff8415c1d0 2024-03-31T15:03:55.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:55.177+0000 7eff83e00640 1 -- 172.21.15.16:0/2690592450 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7eff84068ae0 con 0x7eff8415c1d0 2024-03-31T15:03:55.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:55.177+0000 7eff7aa00640 1 -- 172.21.15.16:0/2690592450 <== mon.2 v2:172.21.15.89:3301/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v8) ==== 53+0+73 (secure 0 0 0) 0x7eff6c0f0dc0 con 0x7eff8415c1d0 2024-03-31T15:03:55.171 INFO:teuthology.orchestra.run.smithi016.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-31T15:03:55.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:55.181+0000 7eff83e00640 1 -- 172.21.15.16:0/2690592450 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7eff60068380 msgr2=0x7eff6006a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:55.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:55.181+0000 7eff83e00640 1 --2- 172.21.15.16:0/2690592450 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7eff60068380 0x7eff6006a840 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7eff841f8440 tx=0x7eff74009190 comp rx=0 tx=0).stop 2024-03-31T15:03:55.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:55.181+0000 7eff83e00640 1 -- 172.21.15.16:0/2690592450 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7eff8415c1d0 msgr2=0x7eff840c99b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:55.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:55.181+0000 7eff83e00640 1 --2- 172.21.15.16:0/2690592450 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7eff8415c1d0 0x7eff840c99b0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7eff6c0ab000 tx=0x7eff6c01ad70 comp rx=0 tx=0).stop 2024-03-31T15:03:55.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:55.181+0000 7eff83e00640 1 -- 172.21.15.16:0/2690592450 shutdown_connections 2024-03-31T15:03:55.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:55.181+0000 7eff83e00640 1 --2- 172.21.15.16:0/2690592450 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7eff60068380 0x7eff6006a840 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:55.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:55.181+0000 7eff83e00640 1 --2- 172.21.15.16:0/2690592450 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7eff8415a7a0 0x7eff840b5000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:55.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:55.181+0000 7eff83e00640 1 --2- 172.21.15.16:0/2690592450 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7eff8415c1d0 0x7eff840c99b0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:55.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:55.181+0000 7eff83e00640 1 --2- 172.21.15.16:0/2690592450 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7eff8406fb10 0x7eff841556b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:55.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:55.181+0000 7eff83e00640 1 -- 172.21.15.16:0/2690592450 >> 172.21.15.16:0/2690592450 conn(0x7eff840b39c0 msgr2=0x7eff84070a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:03:55.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:55.181+0000 7eff83e00640 1 -- 172.21.15.16:0/2690592450 shutdown_connections 2024-03-31T15:03:55.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:55.181+0000 7eff83e00640 1 -- 172.21.15.16:0/2690592450 wait complete. 2024-03-31T15:03:55.192 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-31T15:03:55.192 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-31T15:03:55.192 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi016.front.sepia.ceph.com 2024-03-31T15:03:55.192 INFO:tasks.cephfs.mount:self.client.name = client.2 2024-03-31T15:03:55.192 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.2 2024-03-31T15:03:55.192 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-31T15:03:55.192 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-31T15:03:55.192 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-31T15:03:55.192 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.2' 2024-03-31T15:03:55.193 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:03:55.193 DEBUG:teuthology.orchestra.run.smithi016:> ip addr 2024-03-31T15:03:55.202 INFO:teuthology.orchestra.run.smithi016.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-31T15:03:55.202 INFO:teuthology.orchestra.run.smithi016.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-31T15:03:55.202 INFO:teuthology.orchestra.run.smithi016.stdout: inet 127.0.0.1/8 scope host lo 2024-03-31T15:03:55.203 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2024-03-31T15:03:55.203 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 ::1/128 scope host 2024-03-31T15:03:55.203 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2024-03-31T15:03:55.203 INFO:teuthology.orchestra.run.smithi016.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-03-31T15:03:55.203 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:6b:fd:e0 brd ff:ff:ff:ff:ff:ff 2024-03-31T15:03:55.203 INFO:teuthology.orchestra.run.smithi016.stdout: altname enp7s0f0 2024-03-31T15:03:55.203 INFO:teuthology.orchestra.run.smithi016.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-03-31T15:03:55.203 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:6b:fd:e1 brd ff:ff:ff:ff:ff:ff 2024-03-31T15:03:55.203 INFO:teuthology.orchestra.run.smithi016.stdout: altname enp7s0f1 2024-03-31T15:03:55.203 INFO:teuthology.orchestra.run.smithi016.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-31T15:03:55.203 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:bd:15:64 brd ff:ff:ff:ff:ff:ff 2024-03-31T15:03:55.203 INFO:teuthology.orchestra.run.smithi016.stdout: altname enp1s0f0 2024-03-31T15:03:55.203 INFO:teuthology.orchestra.run.smithi016.stdout: inet 172.21.15.16/20 brd 172.21.15.255 scope global dynamic ens1f0 2024-03-31T15:03:55.203 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft 42446sec preferred_lft 42446sec 2024-03-31T15:03:55.203 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 fe80::ec4:7aff:febd:1564/64 scope link 2024-03-31T15:03:55.204 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2024-03-31T15:03:55.204 INFO:teuthology.orchestra.run.smithi016.stdout:5: ens1f1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-03-31T15:03:55.204 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:bd:15:65 brd ff:ff:ff:ff:ff:ff 2024-03-31T15:03:55.204 INFO:teuthology.orchestra.run.smithi016.stdout: altname enp1s0f1 2024-03-31T15:03:55.204 INFO:teuthology.orchestra.run.smithi016.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2024-03-31T15:03:55.204 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 02:42:d4:1f:7d:73 brd ff:ff:ff:ff:ff:ff 2024-03-31T15:03:55.204 INFO:teuthology.orchestra.run.smithi016.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2024-03-31T15:03:55.204 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2024-03-31T15:03:55.204 INFO:teuthology.orchestra.run.smithi016.stdout:7: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-31T15:03:55.204 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 5e:1f:84:6d:79:ae brd ff:ff:ff:ff:ff:ff 2024-03-31T15:03:55.204 INFO:teuthology.orchestra.run.smithi016.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-31T15:03:55.204 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2024-03-31T15:03:55.204 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 fe80::5c1f:84ff:fe6d:79ae/64 scope link 2024-03-31T15:03:55.204 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2024-03-31T15:03:55.205 INFO:teuthology.orchestra.run.smithi016.stdout:8: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-31T15:03:55.205 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0e:5c:f6:1f:82:2f brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-31T15:03:55.205 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 fe80::c5c:f6ff:fe1f:822f/64 scope link 2024-03-31T15:03:55.205 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2024-03-31T15:03:55.205 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:03:55.205 DEBUG:teuthology.orchestra.run.smithi016:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-31T15:03:55.261 INFO:teuthology.orchestra.run.smithi016.stdout:1 2024-03-31T15:03:55.263 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:03:55.263 DEBUG:teuthology.orchestra.run.smithi016:> route 2024-03-31T15:03:55.313 INFO:teuthology.orchestra.run.smithi016.stdout:Kernel IP routing table 2024-03-31T15:03:55.313 INFO:teuthology.orchestra.run.smithi016.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-31T15:03:55.313 INFO:teuthology.orchestra.run.smithi016.stdout:default _gateway 0.0.0.0 UG 0 0 0 ens1f0 2024-03-31T15:03:55.313 INFO:teuthology.orchestra.run.smithi016.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2024-03-31T15:03:55.313 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 0 0 0 ens1f0 2024-03-31T15:03:55.313 INFO:teuthology.orchestra.run.smithi016.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-31T15:03:55.314 DEBUG:teuthology.orchestra.run.smithi016:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-31T15:03:55.314 DEBUG:teuthology.orchestra.run.smithi016:> set -e 2024-03-31T15:03:55.314 DEBUG:teuthology.orchestra.run.smithi016:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-03-31T15:03:55.314 DEBUG:teuthology.orchestra.run.smithi016:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-03-31T15:03:55.314 DEBUG:teuthology.orchestra.run.smithi016:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-03-31T15:03:55.314 DEBUG:teuthology.orchestra.run.smithi016:> ') 2024-03-31T15:03:55.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:55 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-31T15:03:55.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:55 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-31T15:03:55.451 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:03:55.451 DEBUG:teuthology.orchestra.run.smithi016:> ip netns list 2024-03-31T15:03:55.500 INFO:teuthology.orchestra.run.smithi016.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-31T15:03:55.501 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:03:55.501 DEBUG:teuthology.orchestra.run.smithi016:> ip netns list-id 2024-03-31T15:03:55.556 INFO:teuthology.orchestra.run.smithi016.stdout:nsid 0 (iproute2 netns name: ceph-ns--home-ubuntu-cephtest-mnt.0) 2024-03-31T15:03:55.557 DEBUG:teuthology.orchestra.run.smithi016:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-31T15:03:55.557 DEBUG:teuthology.orchestra.run.smithi016:> set -e 2024-03-31T15:03:55.557 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-31T15:03:55.557 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.2 1 2024-03-31T15:03:55.557 DEBUG:teuthology.orchestra.run.smithi016:> ') 2024-03-31T15:03:55.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:55 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-31T15:03:55.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:55 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-31T15:03:55.683 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:03:55.683 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr 2024-03-31T15:03:55.704 INFO:teuthology.orchestra.run.smithi016.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-31T15:03:55.705 INFO:teuthology.orchestra.run.smithi016.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-31T15:03:55.705 INFO:teuthology.orchestra.run.smithi016.stdout: inet 127.0.0.1/8 scope host lo 2024-03-31T15:03:55.705 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2024-03-31T15:03:55.705 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 ::1/128 scope host 2024-03-31T15:03:55.705 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2024-03-31T15:03:55.705 INFO:teuthology.orchestra.run.smithi016.stdout:2: veth0@if8: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-31T15:03:55.705 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 42:f5:2b:c3:09:77 brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-03-31T15:03:55.705 INFO:teuthology.orchestra.run.smithi016.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-03-31T15:03:55.705 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2024-03-31T15:03:55.705 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 fe80::40f5:2bff:fec3:977/64 scope link 2024-03-31T15:03:55.705 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2024-03-31T15:03:55.707 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:03:55.707 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr 2024-03-31T15:03:55.767 INFO:teuthology.orchestra.run.smithi016.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-31T15:03:55.767 INFO:teuthology.orchestra.run.smithi016.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-31T15:03:55.767 INFO:teuthology.orchestra.run.smithi016.stdout: inet 127.0.0.1/8 scope host lo 2024-03-31T15:03:55.768 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2024-03-31T15:03:55.768 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 ::1/128 scope host 2024-03-31T15:03:55.768 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2024-03-31T15:03:55.768 INFO:teuthology.orchestra.run.smithi016.stdout:2: veth0@if8: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-31T15:03:55.768 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 42:f5:2b:c3:09:77 brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-03-31T15:03:55.768 INFO:teuthology.orchestra.run.smithi016.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-03-31T15:03:55.768 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2024-03-31T15:03:55.768 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 fe80::40f5:2bff:fec3:977/64 scope link 2024-03-31T15:03:55.768 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2024-03-31T15:03:55.770 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.2' with 192.168.0.2/16 2024-03-31T15:03:55.770 DEBUG:teuthology.orchestra.run.smithi016:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-31T15:03:55.770 DEBUG:teuthology.orchestra.run.smithi016:> set -e 2024-03-31T15:03:55.770 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.2 type veth peer name brx.1 2024-03-31T15:03:55.770 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip addr add 192.168.0.2/16 brd 192.168.255.255 dev veth0 2024-03-31T15:03:55.770 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip link set veth0 up 2024-03-31T15:03:55.770 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip link set lo up 2024-03-31T15:03:55.771 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip route add default via 192.168.255.254 2024-03-31T15:03:55.771 DEBUG:teuthology.orchestra.run.smithi016:> ') 2024-03-31T15:03:55.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:55 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-31T15:03:55.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:55 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-31T15:03:55.954 DEBUG:teuthology.orchestra.run.smithi016:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-31T15:03:55.954 DEBUG:teuthology.orchestra.run.smithi016:> set -e 2024-03-31T15:03:55.954 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip link set brx.1 up 2024-03-31T15:03:55.955 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip link set dev brx.1 master ceph-brx 2024-03-31T15:03:55.955 DEBUG:teuthology.orchestra.run.smithi016:> ') 2024-03-31T15:03:56.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:56 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-31T15:03:56.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:56 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-31T15:03:56.040 INFO:tasks.cephfs.fuse_mount:Client client.2 config is {} 2024-03-31T15:03:56.040 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-31T15:03:56.040 DEBUG:teuthology.orchestra.run.smithi016:> mkdir -p -v /home/ubuntu/cephtest/mnt.2 2024-03-31T15:03:56.049 INFO:teuthology.orchestra.run.smithi016.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.2' 2024-03-31T15:03:56.049 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-31T15:03:56.049 DEBUG:teuthology.orchestra.run.smithi016:> chmod 0000 /home/ubuntu/cephtest/mnt.2 2024-03-31T15:03:56.101 DEBUG:teuthology.orchestra.run.smithi016:> sudo modprobe fuse 2024-03-31T15:03:56.164 DEBUG:teuthology.orchestra.run.smithi016:> cat /proc/self/mounts | awk '{print $2}' 2024-03-31T15:03:56.218 INFO:teuthology.orchestra.run.smithi016.stdout:/sys 2024-03-31T15:03:56.218 INFO:teuthology.orchestra.run.smithi016.stdout:/proc 2024-03-31T15:03:56.218 INFO:teuthology.orchestra.run.smithi016.stdout:/dev 2024-03-31T15:03:56.218 INFO:teuthology.orchestra.run.smithi016.stdout:/dev/pts 2024-03-31T15:03:56.218 INFO:teuthology.orchestra.run.smithi016.stdout:/run 2024-03-31T15:03:56.218 INFO:teuthology.orchestra.run.smithi016.stdout:/ 2024-03-31T15:03:56.218 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/kernel/security 2024-03-31T15:03:56.218 INFO:teuthology.orchestra.run.smithi016.stdout:/dev/shm 2024-03-31T15:03:56.218 INFO:teuthology.orchestra.run.smithi016.stdout:/run/lock 2024-03-31T15:03:56.218 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/cgroup 2024-03-31T15:03:56.218 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/pstore 2024-03-31T15:03:56.219 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/bpf 2024-03-31T15:03:56.219 INFO:teuthology.orchestra.run.smithi016.stdout:/proc/sys/fs/binfmt_misc 2024-03-31T15:03:56.219 INFO:teuthology.orchestra.run.smithi016.stdout:/dev/hugepages 2024-03-31T15:03:56.219 INFO:teuthology.orchestra.run.smithi016.stdout:/dev/mqueue 2024-03-31T15:03:56.219 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/kernel/debug 2024-03-31T15:03:56.219 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/kernel/tracing 2024-03-31T15:03:56.219 INFO:teuthology.orchestra.run.smithi016.stdout:/proc/fs/nfsd 2024-03-31T15:03:56.219 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/fuse/connections 2024-03-31T15:03:56.219 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/kernel/config 2024-03-31T15:03:56.219 INFO:teuthology.orchestra.run.smithi016.stdout:/run/credentials/systemd-sysusers.service 2024-03-31T15:03:56.219 INFO:teuthology.orchestra.run.smithi016.stdout:/run/rpc_pipefs 2024-03-31T15:03:56.219 INFO:teuthology.orchestra.run.smithi016.stdout:/run/user/1000 2024-03-31T15:03:56.219 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/ceph 2024-03-31T15:03:56.219 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/ceph/osd/ceph-0 2024-03-31T15:03:56.220 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/ceph/osd/ceph-1 2024-03-31T15:03:56.220 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/ceph/osd/ceph-2 2024-03-31T15:03:56.220 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/ceph/osd/ceph-3 2024-03-31T15:03:56.220 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns 2024-03-31T15:03:56.220 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-31T15:03:56.220 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-31T15:03:56.220 INFO:teuthology.orchestra.run.smithi016.stdout:/home/ubuntu/cephtest/mnt.0 2024-03-31T15:03:56.220 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-31T15:03:56.220 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-31T15:03:56.221 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:03:56.221 DEBUG:teuthology.orchestra.run.smithi016:> ls /sys/fs/fuse/connections 2024-03-31T15:03:56.276 INFO:teuthology.orchestra.run.smithi016.stdout:43 2024-03-31T15:03:56.277 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [43] 2024-03-31T15:03:56.277 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.2 --id 2) 2024-03-31T15:03:56.322 DEBUG:teuthology.orchestra.run.smithi016:> sudo modprobe fuse 2024-03-31T15:03:56.343 DEBUG:teuthology.orchestra.run.smithi016:> cat /proc/self/mounts | awk '{print $2}' 2024-03-31T15:03:56.394 INFO:teuthology.orchestra.run.smithi016.stdout:/sys 2024-03-31T15:03:56.394 INFO:teuthology.orchestra.run.smithi016.stdout:/proc 2024-03-31T15:03:56.394 INFO:teuthology.orchestra.run.smithi016.stdout:/dev 2024-03-31T15:03:56.394 INFO:teuthology.orchestra.run.smithi016.stdout:/dev/pts 2024-03-31T15:03:56.394 INFO:teuthology.orchestra.run.smithi016.stdout:/run 2024-03-31T15:03:56.394 INFO:teuthology.orchestra.run.smithi016.stdout:/ 2024-03-31T15:03:56.394 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/kernel/security 2024-03-31T15:03:56.394 INFO:teuthology.orchestra.run.smithi016.stdout:/dev/shm 2024-03-31T15:03:56.394 INFO:teuthology.orchestra.run.smithi016.stdout:/run/lock 2024-03-31T15:03:56.394 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/cgroup 2024-03-31T15:03:56.395 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/pstore 2024-03-31T15:03:56.395 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/bpf 2024-03-31T15:03:56.395 INFO:teuthology.orchestra.run.smithi016.stdout:/proc/sys/fs/binfmt_misc 2024-03-31T15:03:56.395 INFO:teuthology.orchestra.run.smithi016.stdout:/dev/hugepages 2024-03-31T15:03:56.395 INFO:teuthology.orchestra.run.smithi016.stdout:/dev/mqueue 2024-03-31T15:03:56.395 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/kernel/debug 2024-03-31T15:03:56.395 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/kernel/tracing 2024-03-31T15:03:56.395 INFO:teuthology.orchestra.run.smithi016.stdout:/proc/fs/nfsd 2024-03-31T15:03:56.395 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/fuse/connections 2024-03-31T15:03:56.395 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/kernel/config 2024-03-31T15:03:56.395 INFO:teuthology.orchestra.run.smithi016.stdout:/run/credentials/systemd-sysusers.service 2024-03-31T15:03:56.395 INFO:teuthology.orchestra.run.smithi016.stdout:/run/rpc_pipefs 2024-03-31T15:03:56.395 INFO:teuthology.orchestra.run.smithi016.stdout:/run/user/1000 2024-03-31T15:03:56.395 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/ceph 2024-03-31T15:03:56.396 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/ceph/osd/ceph-0 2024-03-31T15:03:56.396 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/ceph/osd/ceph-1 2024-03-31T15:03:56.396 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/ceph/osd/ceph-2 2024-03-31T15:03:56.396 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/ceph/osd/ceph-3 2024-03-31T15:03:56.396 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns 2024-03-31T15:03:56.396 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-31T15:03:56.396 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-31T15:03:56.396 INFO:teuthology.orchestra.run.smithi016.stdout:/home/ubuntu/cephtest/mnt.0 2024-03-31T15:03:56.396 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-31T15:03:56.396 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-31T15:03:56.397 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:03:56.397 DEBUG:teuthology.orchestra.run.smithi016:> ls /sys/fs/fuse/connections 2024-03-31T15:03:56.418 INFO:tasks.cephfs.fuse_mount.ceph-fuse.2.smithi016.stderr:2024-03-31T15:03:56.425+0000 7f57b70a8580 -1 init, newargv = 0x55b505b92710 newargc=15 2024-03-31T15:03:56.419 INFO:tasks.cephfs.fuse_mount.ceph-fuse.2.smithi016.stderr:ceph-fuse[33443]: starting ceph client 2024-03-31T15:03:56.426 INFO:tasks.cephfs.fuse_mount.ceph-fuse.2.smithi016.stderr:ceph-fuse[33443]: starting fuse 2024-03-31T15:03:56.427 INFO:teuthology.orchestra.run.smithi016.stdout:43 2024-03-31T15:03:56.427 INFO:teuthology.orchestra.run.smithi016.stdout:44 2024-03-31T15:03:56.427 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [43, 44] 2024-03-31T15:03:56.427 DEBUG:teuthology.orchestra.run.smithi016:> sudo stdin-killer -- python3 -c ' 2024-03-31T15:03:56.427 DEBUG:teuthology.orchestra.run.smithi016:> import glob 2024-03-31T15:03:56.427 DEBUG:teuthology.orchestra.run.smithi016:> import re 2024-03-31T15:03:56.427 DEBUG:teuthology.orchestra.run.smithi016:> import os 2024-03-31T15:03:56.427 DEBUG:teuthology.orchestra.run.smithi016:> import subprocess 2024-03-31T15:03:56.427 DEBUG:teuthology.orchestra.run.smithi016:> 2024-03-31T15:03:56.427 DEBUG:teuthology.orchestra.run.smithi016:> def _find_admin_socket(client_name): 2024-03-31T15:03:56.428 DEBUG:teuthology.orchestra.run.smithi016:> asok_path = "/var/run/ceph/ceph-client.2.*.asok" 2024-03-31T15:03:56.428 DEBUG:teuthology.orchestra.run.smithi016:> files = glob.glob(asok_path) 2024-03-31T15:03:56.428 DEBUG:teuthology.orchestra.run.smithi016:> mountpoint = "/home/ubuntu/cephtest/mnt.2" 2024-03-31T15:03:56.428 DEBUG:teuthology.orchestra.run.smithi016:> 2024-03-31T15:03:56.428 DEBUG:teuthology.orchestra.run.smithi016:> # Given a non-glob path, it better be there 2024-03-31T15:03:56.428 DEBUG:teuthology.orchestra.run.smithi016:> if "*" not in asok_path: 2024-03-31T15:03:56.428 DEBUG:teuthology.orchestra.run.smithi016:> assert(len(files) == 1) 2024-03-31T15:03:56.428 DEBUG:teuthology.orchestra.run.smithi016:> return files[0] 2024-03-31T15:03:56.428 DEBUG:teuthology.orchestra.run.smithi016:> 2024-03-31T15:03:56.428 DEBUG:teuthology.orchestra.run.smithi016:> for f in files: 2024-03-31T15:03:56.428 DEBUG:teuthology.orchestra.run.smithi016:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-31T15:03:56.428 DEBUG:teuthology.orchestra.run.smithi016:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-31T15:03:56.428 DEBUG:teuthology.orchestra.run.smithi016:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-31T15:03:56.428 DEBUG:teuthology.orchestra.run.smithi016:> contents = proc_f.read() 2024-03-31T15:03:56.428 DEBUG:teuthology.orchestra.run.smithi016:> if mountpoint in contents: 2024-03-31T15:03:56.429 DEBUG:teuthology.orchestra.run.smithi016:> return f 2024-03-31T15:03:56.429 DEBUG:teuthology.orchestra.run.smithi016:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-31T15:03:56.429 DEBUG:teuthology.orchestra.run.smithi016:> 2024-03-31T15:03:56.429 DEBUG:teuthology.orchestra.run.smithi016:> print(_find_admin_socket("client.2")) 2024-03-31T15:03:56.429 DEBUG:teuthology.orchestra.run.smithi016:> ' 2024-03-31T15:03:56.555 INFO:teuthology.orchestra.run.smithi016.stdout:/var/run/ceph/ceph-client.2.33443.asok 2024-03-31T15:03:56.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:56 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-31T15:03:56.566 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.2.33443.asok 2024-03-31T15:03:56.566 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:03:56.566 DEBUG:teuthology.orchestra.run.smithi016:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.2.33443.asok status 2024-03-31T15:03:56.743 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-03-31T15:03:56.743 INFO:teuthology.orchestra.run.smithi016.stdout: "metadata": { 2024-03-31T15:03:56.743 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph_sha1": "d2271579d5e735cfd6d436b8349c004f06527cb0", 2024-03-31T15:03:56.743 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph_version": "ceph version 19.0.0-2574-gd2271579 (d2271579d5e735cfd6d436b8349c004f06527cb0) squid (dev)", 2024-03-31T15:03:56.743 INFO:teuthology.orchestra.run.smithi016.stdout: "entity_id": "2", 2024-03-31T15:03:56.744 INFO:teuthology.orchestra.run.smithi016.stdout: "hostname": "smithi016", 2024-03-31T15:03:56.744 INFO:teuthology.orchestra.run.smithi016.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.2", 2024-03-31T15:03:56.744 INFO:teuthology.orchestra.run.smithi016.stdout: "pid": "33443", 2024-03-31T15:03:56.744 INFO:teuthology.orchestra.run.smithi016.stdout: "root": "/" 2024-03-31T15:03:56.744 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-31T15:03:56.744 INFO:teuthology.orchestra.run.smithi016.stdout: "dentry_count": 0, 2024-03-31T15:03:56.744 INFO:teuthology.orchestra.run.smithi016.stdout: "dentry_pinned_count": 0, 2024-03-31T15:03:56.744 INFO:teuthology.orchestra.run.smithi016.stdout: "id": 4671, 2024-03-31T15:03:56.744 INFO:teuthology.orchestra.run.smithi016.stdout: "inst": { 2024-03-31T15:03:56.744 INFO:teuthology.orchestra.run.smithi016.stdout: "name": { 2024-03-31T15:03:56.744 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "client", 2024-03-31T15:03:56.744 INFO:teuthology.orchestra.run.smithi016.stdout: "num": 4671 2024-03-31T15:03:56.744 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-31T15:03:56.744 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": { 2024-03-31T15:03:56.745 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-03-31T15:03:56.745 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "192.168.0.2:0", 2024-03-31T15:03:56.745 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 1185264044 2024-03-31T15:03:56.745 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-31T15:03:56.745 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-31T15:03:56.745 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": { 2024-03-31T15:03:56.745 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-03-31T15:03:56.745 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "192.168.0.2:0", 2024-03-31T15:03:56.745 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 1185264044 2024-03-31T15:03:56.745 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-31T15:03:56.745 INFO:teuthology.orchestra.run.smithi016.stdout: "inst_str": "client.4671 192.168.0.2:0/1185264044", 2024-03-31T15:03:56.745 INFO:teuthology.orchestra.run.smithi016.stdout: "addr_str": "192.168.0.2:0/1185264044", 2024-03-31T15:03:56.745 INFO:teuthology.orchestra.run.smithi016.stdout: "inode_count": 1, 2024-03-31T15:03:56.745 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_epoch": 8, 2024-03-31T15:03:56.746 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_epoch": 21, 2024-03-31T15:03:56.746 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_epoch_barrier": 0, 2024-03-31T15:03:56.746 INFO:teuthology.orchestra.run.smithi016.stdout: "blocklisted": false, 2024-03-31T15:03:56.746 INFO:teuthology.orchestra.run.smithi016.stdout: "fs_name": "cephfs" 2024-03-31T15:03:56.746 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-03-31T15:03:56.758 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-31T15:03:56.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:56.905+0000 7f2843e00640 1 Processor -- start 2024-03-31T15:03:56.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:56.905+0000 7f2843e00640 1 -- start start 2024-03-31T15:03:56.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:56.905+0000 7f2843e00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f283c15b3d0 0x7f283c0b1cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:56.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:56.905+0000 7f2843e00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f283c0ce2f0 0x7f283c0b2200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:56.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:56.905+0000 7f2843e00640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f283c15f3c0 con 0x7f283c15eee0 2024-03-31T15:03:56.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:56.905+0000 7f2843e00640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f283c15b7b0 con 0x7f283c15b3d0 2024-03-31T15:03:56.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:56.905+0000 7f2843e00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f283c15f6a0 con 0x7f283c0ce2f0 2024-03-31T15:03:56.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:56.905+0000 7f2841400640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f283c15b3d0 0x7f283c0b1cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:56.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:56.905+0000 7f2841e00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f283c0ce2f0 0x7f283c0b2200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:56.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:56.905+0000 7f2841400640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f283c15b3d0 0x7f283c0b1cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54042/0 (socket says 172.21.15.16:54042) 2024-03-31T15:03:56.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:56.905+0000 7f2841400640 1 -- 172.21.15.16:0/2128157772 learned_addr learned my addr 172.21.15.16:0/2128157772 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:03:56.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:56.905+0000 7f2840a00640 1 --1- 172.21.15.16:0/2128157772 >> v1:172.21.15.89:6789/0 conn(0x7f283c15eee0 0x7f283c0cc9c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.16:58792/0 (socket says 172.21.15.16:58792) 2024-03-31T15:03:56.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:56.905+0000 7f283be00640 1 -- 172.21.15.16:0/2128157772 <== mon.1 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1095812833 0 0) 0x7f283c15f3c0 con 0x7f283c15eee0 2024-03-31T15:03:56.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:56.905+0000 7f2841400640 1 -- 172.21.15.16:0/2128157772 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f283c0ce2f0 msgr2=0x7f283c0b2200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:03:56.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:56.905+0000 7f2841400640 1 --2- 172.21.15.16:0/2128157772 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f283c0ce2f0 0x7f283c0b2200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:56.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:56.905+0000 7f2841400640 1 -- 172.21.15.16:0/2128157772 >> v1:172.21.15.89:6789/0 conn(0x7f283c15eee0 legacy=0x7f283c0cc9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:56.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:56.905+0000 7f2841400640 1 -- 172.21.15.16:0/2128157772 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f283c15f980 con 0x7f283c15b3d0 2024-03-31T15:03:56.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:56.909+0000 7f2841400640 1 --2- 172.21.15.16:0/2128157772 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f283c15b3d0 0x7f283c0b1cc0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f2830009870 tx=0x7f283001c530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d0a3ec16b1352881 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:03:56.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:56.909+0000 7f283be00640 1 -- 172.21.15.16:0/2128157772 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28300aa650 con 0x7f283c15b3d0 2024-03-31T15:03:56.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:56.909+0000 7f283be00640 1 -- 172.21.15.16:0/2128157772 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f28300af070 con 0x7f283c15b3d0 2024-03-31T15:03:56.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:56.909+0000 7f283be00640 1 -- 172.21.15.16:0/2128157772 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28300aaa10 con 0x7f283c15b3d0 2024-03-31T15:03:56.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:56.909+0000 7f2843e00640 1 -- 172.21.15.16:0/2128157772 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f283c15b3d0 msgr2=0x7f283c0b1cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:56.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:56.909+0000 7f2843e00640 1 --2- 172.21.15.16:0/2128157772 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f283c15b3d0 0x7f283c0b1cc0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f2830009870 tx=0x7f283001c530 comp rx=0 tx=0).stop 2024-03-31T15:03:56.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:56.909+0000 7f2843e00640 1 -- 172.21.15.16:0/2128157772 shutdown_connections 2024-03-31T15:03:56.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:56.909+0000 7f2843e00640 1 --2- 172.21.15.16:0/2128157772 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f283c0ce2f0 0x7f283c0b2200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:56.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:56.909+0000 7f2843e00640 1 --2- 172.21.15.16:0/2128157772 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f283c15b3d0 0x7f283c0b1cc0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:56.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:56.909+0000 7f2843e00640 1 -- 172.21.15.16:0/2128157772 >> 172.21.15.16:0/2128157772 conn(0x7f283c06b620 msgr2=0x7f283c06ba30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:03:56.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:56.909+0000 7f2843e00640 1 -- 172.21.15.16:0/2128157772 shutdown_connections 2024-03-31T15:03:56.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:56.909+0000 7f2843e00640 1 -- 172.21.15.16:0/2128157772 wait complete. 2024-03-31T15:03:56.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:56.913+0000 7f2843e00640 1 Processor -- start 2024-03-31T15:03:56.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:56.913+0000 7f2843e00640 1 -- start start 2024-03-31T15:03:56.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:56.913+0000 7f2843e00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f283c0ce2f0 0x7f283c15bb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:56.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:56.913+0000 7f2843e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f283c15b3d0 0x7f283c14a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:56.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:56.913+0000 7f2843e00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f283c15eee0 0x7f283c148fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:56.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:56.913+0000 7f2843e00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f283c0d0350 con 0x7f283c15b3d0 2024-03-31T15:03:56.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:56.913+0000 7f2843e00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f283c0d01d0 con 0x7f283c15eee0 2024-03-31T15:03:56.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:56.913+0000 7f2843e00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f283c0d04d0 con 0x7f283c0ce2f0 2024-03-31T15:03:56.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:56.913+0000 7f2840a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f283c15b3d0 0x7f283c14a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:56.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:56.913+0000 7f2841400640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f283c0ce2f0 0x7f283c15bb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:56.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:56.913+0000 7f2841400640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f283c0ce2f0 0x7f283c15bb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3301/0 says I am v2:172.21.15.16:51330/0 (socket says 172.21.15.16:51330) 2024-03-31T15:03:56.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:56.913+0000 7f2840a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f283c15b3d0 0x7f283c14a480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54054/0 (socket says 172.21.15.16:54054) 2024-03-31T15:03:56.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:56.913+0000 7f2841400640 1 -- 172.21.15.16:0/1088636282 learned_addr learned my addr 172.21.15.16:0/1088636282 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:03:56.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:56.913+0000 7f2841e00640 1 --2- 172.21.15.16:0/1088636282 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f283c15eee0 0x7f283c148fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:56.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:56.913+0000 7f2841400640 1 -- 172.21.15.16:0/1088636282 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f283c15eee0 msgr2=0x7f283c148fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:56.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:56.913+0000 7f2841400640 1 --2- 172.21.15.16:0/1088636282 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f283c15eee0 0x7f283c148fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:56.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:56.913+0000 7f2841400640 1 -- 172.21.15.16:0/1088636282 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f283c15b3d0 msgr2=0x7f283c14a480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:56.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:56.913+0000 7f2841400640 1 --2- 172.21.15.16:0/1088636282 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f283c15b3d0 0x7f283c14a480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:56.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:56.913+0000 7f2841400640 1 -- 172.21.15.16:0/1088636282 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f283c149800 con 0x7f283c0ce2f0 2024-03-31T15:03:56.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:56.913+0000 7f2840a00640 1 --2- 172.21.15.16:0/1088636282 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f283c15b3d0 0x7f283c14a480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:03:56.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:56.913+0000 7f2841400640 1 --2- 172.21.15.16:0/1088636282 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f283c0ce2f0 0x7f283c15bb60 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f28300ac040 tx=0x7f283001ad70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:03:56.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:56.913+0000 7f2839600640 1 -- 172.21.15.16:0/1088636282 <== mon.2 v2:172.21.15.89:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28300bd5f0 con 0x7f283c0ce2f0 2024-03-31T15:03:56.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:56.913+0000 7f2839600640 1 -- 172.21.15.16:0/1088636282 <== mon.2 v2:172.21.15.89:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f28300a6040 con 0x7f283c0ce2f0 2024-03-31T15:03:56.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:56.913+0000 7f2839600640 1 -- 172.21.15.16:0/1088636282 <== mon.2 v2:172.21.15.89:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28300bd9f0 con 0x7f283c0ce2f0 2024-03-31T15:03:56.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:56.913+0000 7f2843e00640 1 -- 172.21.15.16:0/1088636282 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f283c149ae0 con 0x7f283c0ce2f0 2024-03-31T15:03:56.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:56.913+0000 7f2843e00640 1 -- 172.21.15.16:0/1088636282 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f283c14a0a0 con 0x7f283c0ce2f0 2024-03-31T15:03:56.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:56.917+0000 7f2843e00640 1 -- 172.21.15.16:0/1088636282 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f283c069950 con 0x7f283c0ce2f0 2024-03-31T15:03:56.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:56.917+0000 7f2839600640 1 -- 172.21.15.16:0/1088636282 <== mon.2 v2:172.21.15.89:3301/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f28300b33f0 con 0x7f283c0ce2f0 2024-03-31T15:03:56.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:56.917+0000 7f2839600640 1 --2- 172.21.15.16:0/1088636282 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f281c068380 0x7f281c06a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:03:56.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:56.917+0000 7f2839600640 1 -- 172.21.15.16:0/1088636282 <== mon.2 v2:172.21.15.89:3301/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f2830120ee0 con 0x7f283c0ce2f0 2024-03-31T15:03:56.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:56.917+0000 7f2840a00640 1 --2- 172.21.15.16:0/1088636282 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f281c068380 0x7f281c06a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:03:56.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:56.917+0000 7f2840a00640 1 --2- 172.21.15.16:0/1088636282 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f281c068380 0x7f281c06a840 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f283c1535d0 tx=0x7f2828007df0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:03:56.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:56.921+0000 7f2839600640 1 -- 172.21.15.16:0/1088636282 <== mon.2 v2:172.21.15.89:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f28300c3060 con 0x7f283c0ce2f0 2024-03-31T15:03:57.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:57.121+0000 7f2843e00640 1 -- 172.21.15.16:0/1088636282 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f283c0cdea0 con 0x7f283c0ce2f0 2024-03-31T15:03:57.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:57.125+0000 7f2839600640 1 -- 172.21.15.16:0/1088636282 <== mon.2 v2:172.21.15.89:3301/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v8) ==== 53+0+73 (secure 0 0 0) 0x7f2830016150 con 0x7f283c0ce2f0 2024-03-31T15:03:57.117 INFO:teuthology.orchestra.run.smithi016.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-31T15:03:57.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:57.125+0000 7f2843e00640 1 -- 172.21.15.16:0/1088636282 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f281c068380 msgr2=0x7f281c06a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:57.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:57.125+0000 7f2843e00640 1 --2- 172.21.15.16:0/1088636282 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f281c068380 0x7f281c06a840 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f283c1535d0 tx=0x7f2828007df0 comp rx=0 tx=0).stop 2024-03-31T15:03:57.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:57.125+0000 7f2843e00640 1 -- 172.21.15.16:0/1088636282 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f283c0ce2f0 msgr2=0x7f283c15bb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:03:57.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:57.125+0000 7f2843e00640 1 --2- 172.21.15.16:0/1088636282 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f283c0ce2f0 0x7f283c15bb60 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f28300ac040 tx=0x7f283001ad70 comp rx=0 tx=0).stop 2024-03-31T15:03:57.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:57.125+0000 7f2843e00640 1 -- 172.21.15.16:0/1088636282 shutdown_connections 2024-03-31T15:03:57.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:57.125+0000 7f2843e00640 1 --2- 172.21.15.16:0/1088636282 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f281c068380 0x7f281c06a840 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:57.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:57.125+0000 7f2843e00640 1 --2- 172.21.15.16:0/1088636282 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f283c15eee0 0x7f283c148fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:57.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:57.125+0000 7f2843e00640 1 --2- 172.21.15.16:0/1088636282 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f283c15b3d0 0x7f283c14a480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:57.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:57.125+0000 7f2843e00640 1 --2- 172.21.15.16:0/1088636282 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f283c0ce2f0 0x7f283c15bb60 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:03:57.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:57.125+0000 7f2843e00640 1 -- 172.21.15.16:0/1088636282 >> 172.21.15.16:0/1088636282 conn(0x7f283c06b620 msgr2=0x7f283c0cf7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:03:57.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:57.125+0000 7f2843e00640 1 -- 172.21.15.16:0/1088636282 shutdown_connections 2024-03-31T15:03:57.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:03:57.125+0000 7f2843e00640 1 -- 172.21.15.16:0/1088636282 wait complete. 2024-03-31T15:03:57.137 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-31T15:03:57.137 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-31T15:03:57.137 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi089.front.sepia.ceph.com 2024-03-31T15:03:57.137 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-03-31T15:03:57.138 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-03-31T15:03:57.138 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-31T15:03:57.138 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-31T15:03:57.138 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-31T15:03:57.138 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.1' 2024-03-31T15:03:57.138 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:03:57.138 DEBUG:teuthology.orchestra.run.smithi089:> ip addr 2024-03-31T15:03:57.147 INFO:teuthology.orchestra.run.smithi089.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-31T15:03:57.147 INFO:teuthology.orchestra.run.smithi089.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-31T15:03:57.147 INFO:teuthology.orchestra.run.smithi089.stdout: inet 127.0.0.1/8 scope host lo 2024-03-31T15:03:57.147 INFO:teuthology.orchestra.run.smithi089.stdout: valid_lft forever preferred_lft forever 2024-03-31T15:03:57.147 INFO:teuthology.orchestra.run.smithi089.stdout: inet6 ::1/128 scope host 2024-03-31T15:03:57.147 INFO:teuthology.orchestra.run.smithi089.stdout: valid_lft forever preferred_lft forever 2024-03-31T15:03:57.147 INFO:teuthology.orchestra.run.smithi089.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-03-31T15:03:57.147 INFO:teuthology.orchestra.run.smithi089.stdout: link/ether 0c:c4:7a:d8:a4:c6 brd ff:ff:ff:ff:ff:ff 2024-03-31T15:03:57.147 INFO:teuthology.orchestra.run.smithi089.stdout: altname enp7s0f0 2024-03-31T15:03:57.147 INFO:teuthology.orchestra.run.smithi089.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-03-31T15:03:57.148 INFO:teuthology.orchestra.run.smithi089.stdout: link/ether 0c:c4:7a:d8:a4:c7 brd ff:ff:ff:ff:ff:ff 2024-03-31T15:03:57.148 INFO:teuthology.orchestra.run.smithi089.stdout: altname enp7s0f1 2024-03-31T15:03:57.148 INFO:teuthology.orchestra.run.smithi089.stdout:4: enp3s0f0: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-03-31T15:03:57.148 INFO:teuthology.orchestra.run.smithi089.stdout: link/ether 0c:c4:7a:8f:d5:00 brd ff:ff:ff:ff:ff:ff 2024-03-31T15:03:57.148 INFO:teuthology.orchestra.run.smithi089.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-31T15:03:57.148 INFO:teuthology.orchestra.run.smithi089.stdout: link/ether 0c:c4:7a:8f:d5:01 brd ff:ff:ff:ff:ff:ff 2024-03-31T15:03:57.148 INFO:teuthology.orchestra.run.smithi089.stdout: inet 172.21.15.89/20 metric 100 brd 172.21.15.255 scope global dynamic enp3s0f1 2024-03-31T15:03:57.148 INFO:teuthology.orchestra.run.smithi089.stdout: valid_lft 42446sec preferred_lft 42446sec 2024-03-31T15:03:57.148 INFO:teuthology.orchestra.run.smithi089.stdout: inet6 fe80::ec4:7aff:fe8f:d501/64 scope link 2024-03-31T15:03:57.148 INFO:teuthology.orchestra.run.smithi089.stdout: valid_lft forever preferred_lft forever 2024-03-31T15:03:57.148 INFO:teuthology.orchestra.run.smithi089.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2024-03-31T15:03:57.148 INFO:teuthology.orchestra.run.smithi089.stdout: link/ether 02:42:b1:c8:45:10 brd ff:ff:ff:ff:ff:ff 2024-03-31T15:03:57.148 INFO:teuthology.orchestra.run.smithi089.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2024-03-31T15:03:57.148 INFO:teuthology.orchestra.run.smithi089.stdout: valid_lft forever preferred_lft forever 2024-03-31T15:03:57.149 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-03-31T15:03:57.149 DEBUG:teuthology.orchestra.run.smithi089:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-31T15:03:57.149 DEBUG:teuthology.orchestra.run.smithi089:> set -e 2024-03-31T15:03:57.150 DEBUG:teuthology.orchestra.run.smithi089:> sudo ip link add name ceph-brx type bridge 2024-03-31T15:03:57.150 DEBUG:teuthology.orchestra.run.smithi089:> sudo ip addr flush dev ceph-brx 2024-03-31T15:03:57.150 DEBUG:teuthology.orchestra.run.smithi089:> sudo ip link set ceph-brx up 2024-03-31T15:03:57.150 DEBUG:teuthology.orchestra.run.smithi089:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-03-31T15:03:57.150 DEBUG:teuthology.orchestra.run.smithi089:> ') 2024-03-31T15:03:57.235 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:57 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-31T15:03:57.295 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:57 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-31T15:03:57.301 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:03:57.302 DEBUG:teuthology.orchestra.run.smithi089:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-31T15:03:57.319 INFO:teuthology.orchestra.run.smithi089.stdout:1 2024-03-31T15:03:57.322 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:03:57.322 DEBUG:teuthology.orchestra.run.smithi089:> route 2024-03-31T15:03:57.390 INFO:teuthology.orchestra.run.smithi089.stdout:Kernel IP routing table 2024-03-31T15:03:57.390 INFO:teuthology.orchestra.run.smithi089.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-31T15:03:57.390 INFO:teuthology.orchestra.run.smithi089.stdout:default _gateway 0.0.0.0 UG 0 0 0 enp3s0f1 2024-03-31T15:03:57.390 INFO:teuthology.orchestra.run.smithi089.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-31T15:03:57.390 INFO:teuthology.orchestra.run.smithi089.stdout:158.69.67.47 _gateway 255.255.255.255 UGH 100 0 0 enp3s0f1 2024-03-31T15:03:57.390 INFO:teuthology.orchestra.run.smithi089.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2024-03-31T15:03:57.390 INFO:teuthology.orchestra.run.smithi089.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-31T15:03:57.390 INFO:teuthology.orchestra.run.smithi089.stdout:ns1.front.sepia 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-03-31T15:03:57.390 INFO:teuthology.orchestra.run.smithi089.stdout:172.21.0.2 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-03-31T15:03:57.391 INFO:teuthology.orchestra.run.smithi089.stdout:_gateway 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-03-31T15:03:57.391 INFO:teuthology.orchestra.run.smithi089.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-31T15:03:57.391 DEBUG:teuthology.orchestra.run.smithi089:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-31T15:03:57.391 DEBUG:teuthology.orchestra.run.smithi089:> set -e 2024-03-31T15:03:57.391 DEBUG:teuthology.orchestra.run.smithi089:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-31T15:03:57.391 DEBUG:teuthology.orchestra.run.smithi089:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-31T15:03:57.391 DEBUG:teuthology.orchestra.run.smithi089:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-31T15:03:57.391 DEBUG:teuthology.orchestra.run.smithi089:> ') 2024-03-31T15:03:57.480 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:57 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-31T15:03:57.520 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:57 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-31T15:03:57.527 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:03:57.527 DEBUG:teuthology.orchestra.run.smithi089:> ip netns list 2024-03-31T15:03:57.536 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:03:57.536 DEBUG:teuthology.orchestra.run.smithi089:> ip netns list-id 2024-03-31T15:03:57.591 DEBUG:teuthology.orchestra.run.smithi089:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-31T15:03:57.591 DEBUG:teuthology.orchestra.run.smithi089:> set -e 2024-03-31T15:03:57.591 DEBUG:teuthology.orchestra.run.smithi089:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-31T15:03:57.591 DEBUG:teuthology.orchestra.run.smithi089:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.1 0 2024-03-31T15:03:57.591 DEBUG:teuthology.orchestra.run.smithi089:> ') 2024-03-31T15:03:57.679 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:57 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-31T15:03:57.709 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:57 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-31T15:03:57.716 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.1' with 192.168.0.1/16 2024-03-31T15:03:57.716 DEBUG:teuthology.orchestra.run.smithi089:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-31T15:03:57.716 DEBUG:teuthology.orchestra.run.smithi089:> set -e 2024-03-31T15:03:57.716 DEBUG:teuthology.orchestra.run.smithi089:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.1 type veth peer name brx.0 2024-03-31T15:03:57.716 DEBUG:teuthology.orchestra.run.smithi089:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-03-31T15:03:57.716 DEBUG:teuthology.orchestra.run.smithi089:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set veth0 up 2024-03-31T15:03:57.716 DEBUG:teuthology.orchestra.run.smithi089:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set lo up 2024-03-31T15:03:57.716 DEBUG:teuthology.orchestra.run.smithi089:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip route add default via 192.168.255.254 2024-03-31T15:03:57.716 DEBUG:teuthology.orchestra.run.smithi089:> ') 2024-03-31T15:03:57.763 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:57 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-31T15:03:57.874 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:57 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-31T15:03:57.883 DEBUG:teuthology.orchestra.run.smithi089:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-31T15:03:57.883 DEBUG:teuthology.orchestra.run.smithi089:> set -e 2024-03-31T15:03:57.883 DEBUG:teuthology.orchestra.run.smithi089:> sudo ip link set brx.0 up 2024-03-31T15:03:57.883 DEBUG:teuthology.orchestra.run.smithi089:> sudo ip link set dev brx.0 master ceph-brx 2024-03-31T15:03:57.883 DEBUG:teuthology.orchestra.run.smithi089:> ') 2024-03-31T15:03:57.930 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:57 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-31T15:03:57.962 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:57 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-31T15:03:57.968 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2024-03-31T15:03:57.969 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-31T15:03:57.969 DEBUG:teuthology.orchestra.run.smithi089:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-03-31T15:03:58.018 INFO:teuthology.orchestra.run.smithi089.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-03-31T15:03:58.018 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-31T15:03:58.018 DEBUG:teuthology.orchestra.run.smithi089:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-03-31T15:03:58.069 DEBUG:teuthology.orchestra.run.smithi089:> sudo modprobe fuse 2024-03-31T15:03:58.129 DEBUG:teuthology.orchestra.run.smithi089:> cat /proc/self/mounts | awk '{print $2}' 2024-03-31T15:03:58.183 INFO:teuthology.orchestra.run.smithi089.stdout:/sys 2024-03-31T15:03:58.183 INFO:teuthology.orchestra.run.smithi089.stdout:/proc 2024-03-31T15:03:58.183 INFO:teuthology.orchestra.run.smithi089.stdout:/dev 2024-03-31T15:03:58.183 INFO:teuthology.orchestra.run.smithi089.stdout:/dev/pts 2024-03-31T15:03:58.183 INFO:teuthology.orchestra.run.smithi089.stdout:/run 2024-03-31T15:03:58.184 INFO:teuthology.orchestra.run.smithi089.stdout:/ 2024-03-31T15:03:58.184 INFO:teuthology.orchestra.run.smithi089.stdout:/sys/kernel/security 2024-03-31T15:03:58.184 INFO:teuthology.orchestra.run.smithi089.stdout:/dev/shm 2024-03-31T15:03:58.184 INFO:teuthology.orchestra.run.smithi089.stdout:/run/lock 2024-03-31T15:03:58.184 INFO:teuthology.orchestra.run.smithi089.stdout:/sys/fs/cgroup 2024-03-31T15:03:58.184 INFO:teuthology.orchestra.run.smithi089.stdout:/sys/fs/pstore 2024-03-31T15:03:58.184 INFO:teuthology.orchestra.run.smithi089.stdout:/sys/fs/bpf 2024-03-31T15:03:58.184 INFO:teuthology.orchestra.run.smithi089.stdout:/proc/sys/fs/binfmt_misc 2024-03-31T15:03:58.184 INFO:teuthology.orchestra.run.smithi089.stdout:/dev/hugepages 2024-03-31T15:03:58.184 INFO:teuthology.orchestra.run.smithi089.stdout:/dev/mqueue 2024-03-31T15:03:58.184 INFO:teuthology.orchestra.run.smithi089.stdout:/sys/kernel/debug 2024-03-31T15:03:58.184 INFO:teuthology.orchestra.run.smithi089.stdout:/sys/kernel/tracing 2024-03-31T15:03:58.184 INFO:teuthology.orchestra.run.smithi089.stdout:/proc/fs/nfsd 2024-03-31T15:03:58.184 INFO:teuthology.orchestra.run.smithi089.stdout:/sys/fs/fuse/connections 2024-03-31T15:03:58.184 INFO:teuthology.orchestra.run.smithi089.stdout:/sys/kernel/config 2024-03-31T15:03:58.185 INFO:teuthology.orchestra.run.smithi089.stdout:/run/credentials/systemd-sysusers.service 2024-03-31T15:03:58.185 INFO:teuthology.orchestra.run.smithi089.stdout:/run/rpc_pipefs 2024-03-31T15:03:58.185 INFO:teuthology.orchestra.run.smithi089.stdout:/run/user/1000 2024-03-31T15:03:58.185 INFO:teuthology.orchestra.run.smithi089.stdout:/var/lib/ceph 2024-03-31T15:03:58.185 INFO:teuthology.orchestra.run.smithi089.stdout:/var/lib/ceph/osd/ceph-4 2024-03-31T15:03:58.185 INFO:teuthology.orchestra.run.smithi089.stdout:/var/lib/ceph/osd/ceph-5 2024-03-31T15:03:58.185 INFO:teuthology.orchestra.run.smithi089.stdout:/var/lib/ceph/osd/ceph-6 2024-03-31T15:03:58.185 INFO:teuthology.orchestra.run.smithi089.stdout:/var/lib/ceph/osd/ceph-7 2024-03-31T15:03:58.185 INFO:teuthology.orchestra.run.smithi089.stdout:/run/netns 2024-03-31T15:03:58.185 INFO:teuthology.orchestra.run.smithi089.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-31T15:03:58.185 INFO:teuthology.orchestra.run.smithi089.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-31T15:03:58.186 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:03:58.186 DEBUG:teuthology.orchestra.run.smithi089:> ls /sys/fs/fuse/connections 2024-03-31T15:03:58.237 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-31T15:03:58.237 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.1 --id 1) 2024-03-31T15:03:58.281 DEBUG:teuthology.orchestra.run.smithi089:> sudo modprobe fuse 2024-03-31T15:03:58.301 DEBUG:teuthology.orchestra.run.smithi089:> cat /proc/self/mounts | awk '{print $2}' 2024-03-31T15:03:58.352 INFO:teuthology.orchestra.run.smithi089.stdout:/sys 2024-03-31T15:03:58.353 INFO:teuthology.orchestra.run.smithi089.stdout:/proc 2024-03-31T15:03:58.353 INFO:teuthology.orchestra.run.smithi089.stdout:/dev 2024-03-31T15:03:58.353 INFO:teuthology.orchestra.run.smithi089.stdout:/dev/pts 2024-03-31T15:03:58.353 INFO:teuthology.orchestra.run.smithi089.stdout:/run 2024-03-31T15:03:58.353 INFO:teuthology.orchestra.run.smithi089.stdout:/ 2024-03-31T15:03:58.353 INFO:teuthology.orchestra.run.smithi089.stdout:/sys/kernel/security 2024-03-31T15:03:58.353 INFO:teuthology.orchestra.run.smithi089.stdout:/dev/shm 2024-03-31T15:03:58.353 INFO:teuthology.orchestra.run.smithi089.stdout:/run/lock 2024-03-31T15:03:58.353 INFO:teuthology.orchestra.run.smithi089.stdout:/sys/fs/cgroup 2024-03-31T15:03:58.353 INFO:teuthology.orchestra.run.smithi089.stdout:/sys/fs/pstore 2024-03-31T15:03:58.353 INFO:teuthology.orchestra.run.smithi089.stdout:/sys/fs/bpf 2024-03-31T15:03:58.353 INFO:teuthology.orchestra.run.smithi089.stdout:/proc/sys/fs/binfmt_misc 2024-03-31T15:03:58.353 INFO:teuthology.orchestra.run.smithi089.stdout:/dev/hugepages 2024-03-31T15:03:58.353 INFO:teuthology.orchestra.run.smithi089.stdout:/dev/mqueue 2024-03-31T15:03:58.354 INFO:teuthology.orchestra.run.smithi089.stdout:/sys/kernel/debug 2024-03-31T15:03:58.354 INFO:teuthology.orchestra.run.smithi089.stdout:/sys/kernel/tracing 2024-03-31T15:03:58.354 INFO:teuthology.orchestra.run.smithi089.stdout:/proc/fs/nfsd 2024-03-31T15:03:58.354 INFO:teuthology.orchestra.run.smithi089.stdout:/sys/fs/fuse/connections 2024-03-31T15:03:58.354 INFO:teuthology.orchestra.run.smithi089.stdout:/sys/kernel/config 2024-03-31T15:03:58.354 INFO:teuthology.orchestra.run.smithi089.stdout:/run/credentials/systemd-sysusers.service 2024-03-31T15:03:58.354 INFO:teuthology.orchestra.run.smithi089.stdout:/run/rpc_pipefs 2024-03-31T15:03:58.354 INFO:teuthology.orchestra.run.smithi089.stdout:/run/user/1000 2024-03-31T15:03:58.354 INFO:teuthology.orchestra.run.smithi089.stdout:/var/lib/ceph 2024-03-31T15:03:58.354 INFO:teuthology.orchestra.run.smithi089.stdout:/var/lib/ceph/osd/ceph-4 2024-03-31T15:03:58.354 INFO:teuthology.orchestra.run.smithi089.stdout:/var/lib/ceph/osd/ceph-5 2024-03-31T15:03:58.354 INFO:teuthology.orchestra.run.smithi089.stdout:/var/lib/ceph/osd/ceph-6 2024-03-31T15:03:58.354 INFO:teuthology.orchestra.run.smithi089.stdout:/var/lib/ceph/osd/ceph-7 2024-03-31T15:03:58.354 INFO:teuthology.orchestra.run.smithi089.stdout:/run/netns 2024-03-31T15:03:58.354 INFO:teuthology.orchestra.run.smithi089.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-31T15:03:58.354 INFO:teuthology.orchestra.run.smithi089.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-31T15:03:58.355 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:03:58.355 DEBUG:teuthology.orchestra.run.smithi089:> ls /sys/fs/fuse/connections 2024-03-31T15:03:58.380 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi089.stderr:2024-03-31T15:03:58.388+0000 7fd193c2a580 -1 init, newargv = 0x555a08c4c9f0 newargc=15 2024-03-31T15:03:58.380 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi089.stderr:ceph-fuse[32929]: starting ceph client 2024-03-31T15:03:58.388 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi089.stderr:ceph-fuse[32929]: starting fuse 2024-03-31T15:03:59.490 DEBUG:teuthology.orchestra.run.smithi089:> sudo modprobe fuse 2024-03-31T15:03:59.510 DEBUG:teuthology.orchestra.run.smithi089:> cat /proc/self/mounts | awk '{print $2}' 2024-03-31T15:03:59.563 INFO:teuthology.orchestra.run.smithi089.stdout:/sys 2024-03-31T15:03:59.564 INFO:teuthology.orchestra.run.smithi089.stdout:/proc 2024-03-31T15:03:59.564 INFO:teuthology.orchestra.run.smithi089.stdout:/dev 2024-03-31T15:03:59.564 INFO:teuthology.orchestra.run.smithi089.stdout:/dev/pts 2024-03-31T15:03:59.564 INFO:teuthology.orchestra.run.smithi089.stdout:/run 2024-03-31T15:03:59.564 INFO:teuthology.orchestra.run.smithi089.stdout:/ 2024-03-31T15:03:59.564 INFO:teuthology.orchestra.run.smithi089.stdout:/sys/kernel/security 2024-03-31T15:03:59.564 INFO:teuthology.orchestra.run.smithi089.stdout:/dev/shm 2024-03-31T15:03:59.564 INFO:teuthology.orchestra.run.smithi089.stdout:/run/lock 2024-03-31T15:03:59.564 INFO:teuthology.orchestra.run.smithi089.stdout:/sys/fs/cgroup 2024-03-31T15:03:59.564 INFO:teuthology.orchestra.run.smithi089.stdout:/sys/fs/pstore 2024-03-31T15:03:59.564 INFO:teuthology.orchestra.run.smithi089.stdout:/sys/fs/bpf 2024-03-31T15:03:59.564 INFO:teuthology.orchestra.run.smithi089.stdout:/proc/sys/fs/binfmt_misc 2024-03-31T15:03:59.564 INFO:teuthology.orchestra.run.smithi089.stdout:/dev/hugepages 2024-03-31T15:03:59.564 INFO:teuthology.orchestra.run.smithi089.stdout:/dev/mqueue 2024-03-31T15:03:59.565 INFO:teuthology.orchestra.run.smithi089.stdout:/sys/kernel/debug 2024-03-31T15:03:59.565 INFO:teuthology.orchestra.run.smithi089.stdout:/sys/kernel/tracing 2024-03-31T15:03:59.565 INFO:teuthology.orchestra.run.smithi089.stdout:/proc/fs/nfsd 2024-03-31T15:03:59.565 INFO:teuthology.orchestra.run.smithi089.stdout:/sys/fs/fuse/connections 2024-03-31T15:03:59.565 INFO:teuthology.orchestra.run.smithi089.stdout:/sys/kernel/config 2024-03-31T15:03:59.565 INFO:teuthology.orchestra.run.smithi089.stdout:/run/credentials/systemd-sysusers.service 2024-03-31T15:03:59.565 INFO:teuthology.orchestra.run.smithi089.stdout:/run/rpc_pipefs 2024-03-31T15:03:59.565 INFO:teuthology.orchestra.run.smithi089.stdout:/run/user/1000 2024-03-31T15:03:59.565 INFO:teuthology.orchestra.run.smithi089.stdout:/var/lib/ceph 2024-03-31T15:03:59.565 INFO:teuthology.orchestra.run.smithi089.stdout:/var/lib/ceph/osd/ceph-4 2024-03-31T15:03:59.565 INFO:teuthology.orchestra.run.smithi089.stdout:/var/lib/ceph/osd/ceph-5 2024-03-31T15:03:59.565 INFO:teuthology.orchestra.run.smithi089.stdout:/var/lib/ceph/osd/ceph-6 2024-03-31T15:03:59.565 INFO:teuthology.orchestra.run.smithi089.stdout:/var/lib/ceph/osd/ceph-7 2024-03-31T15:03:59.566 INFO:teuthology.orchestra.run.smithi089.stdout:/run/netns 2024-03-31T15:03:59.566 INFO:teuthology.orchestra.run.smithi089.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-31T15:03:59.566 INFO:teuthology.orchestra.run.smithi089.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-31T15:03:59.566 INFO:teuthology.orchestra.run.smithi089.stdout:/home/ubuntu/cephtest/mnt.1 2024-03-31T15:03:59.567 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:03:59.567 DEBUG:teuthology.orchestra.run.smithi089:> ls /sys/fs/fuse/connections 2024-03-31T15:03:59.618 INFO:teuthology.orchestra.run.smithi089.stdout:43 2024-03-31T15:03:59.618 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [43] 2024-03-31T15:03:59.618 DEBUG:teuthology.orchestra.run.smithi089:> sudo stdin-killer -- python3 -c ' 2024-03-31T15:03:59.618 DEBUG:teuthology.orchestra.run.smithi089:> import glob 2024-03-31T15:03:59.618 DEBUG:teuthology.orchestra.run.smithi089:> import re 2024-03-31T15:03:59.618 DEBUG:teuthology.orchestra.run.smithi089:> import os 2024-03-31T15:03:59.618 DEBUG:teuthology.orchestra.run.smithi089:> import subprocess 2024-03-31T15:03:59.618 DEBUG:teuthology.orchestra.run.smithi089:> 2024-03-31T15:03:59.618 DEBUG:teuthology.orchestra.run.smithi089:> def _find_admin_socket(client_name): 2024-03-31T15:03:59.619 DEBUG:teuthology.orchestra.run.smithi089:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2024-03-31T15:03:59.619 DEBUG:teuthology.orchestra.run.smithi089:> files = glob.glob(asok_path) 2024-03-31T15:03:59.619 DEBUG:teuthology.orchestra.run.smithi089:> mountpoint = "/home/ubuntu/cephtest/mnt.1" 2024-03-31T15:03:59.619 DEBUG:teuthology.orchestra.run.smithi089:> 2024-03-31T15:03:59.619 DEBUG:teuthology.orchestra.run.smithi089:> # Given a non-glob path, it better be there 2024-03-31T15:03:59.619 DEBUG:teuthology.orchestra.run.smithi089:> if "*" not in asok_path: 2024-03-31T15:03:59.619 DEBUG:teuthology.orchestra.run.smithi089:> assert(len(files) == 1) 2024-03-31T15:03:59.619 DEBUG:teuthology.orchestra.run.smithi089:> return files[0] 2024-03-31T15:03:59.619 DEBUG:teuthology.orchestra.run.smithi089:> 2024-03-31T15:03:59.619 DEBUG:teuthology.orchestra.run.smithi089:> for f in files: 2024-03-31T15:03:59.619 DEBUG:teuthology.orchestra.run.smithi089:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-31T15:03:59.619 DEBUG:teuthology.orchestra.run.smithi089:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-31T15:03:59.619 DEBUG:teuthology.orchestra.run.smithi089:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-31T15:03:59.619 DEBUG:teuthology.orchestra.run.smithi089:> contents = proc_f.read() 2024-03-31T15:03:59.619 DEBUG:teuthology.orchestra.run.smithi089:> if mountpoint in contents: 2024-03-31T15:03:59.620 DEBUG:teuthology.orchestra.run.smithi089:> return f 2024-03-31T15:03:59.620 DEBUG:teuthology.orchestra.run.smithi089:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-31T15:03:59.620 DEBUG:teuthology.orchestra.run.smithi089:> 2024-03-31T15:03:59.620 DEBUG:teuthology.orchestra.run.smithi089:> print(_find_admin_socket("client.1")) 2024-03-31T15:03:59.620 DEBUG:teuthology.orchestra.run.smithi089:> ' 2024-03-31T15:03:59.738 INFO:teuthology.orchestra.run.smithi089.stdout:/var/run/ceph/ceph-client.1.32929.asok 2024-03-31T15:03:59.741 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:03:59 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-31T15:03:59.749 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.32929.asok 2024-03-31T15:03:59.749 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:03:59.749 DEBUG:teuthology.orchestra.run.smithi089:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.32929.asok status 2024-03-31T15:03:59.917 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-03-31T15:03:59.917 INFO:teuthology.orchestra.run.smithi089.stdout: "metadata": { 2024-03-31T15:03:59.917 INFO:teuthology.orchestra.run.smithi089.stdout: "ceph_sha1": "d2271579d5e735cfd6d436b8349c004f06527cb0", 2024-03-31T15:03:59.917 INFO:teuthology.orchestra.run.smithi089.stdout: "ceph_version": "ceph version 19.0.0-2574-gd2271579 (d2271579d5e735cfd6d436b8349c004f06527cb0) squid (dev)", 2024-03-31T15:03:59.917 INFO:teuthology.orchestra.run.smithi089.stdout: "entity_id": "1", 2024-03-31T15:03:59.917 INFO:teuthology.orchestra.run.smithi089.stdout: "hostname": "smithi089", 2024-03-31T15:03:59.917 INFO:teuthology.orchestra.run.smithi089.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2024-03-31T15:03:59.917 INFO:teuthology.orchestra.run.smithi089.stdout: "pid": "32929", 2024-03-31T15:03:59.917 INFO:teuthology.orchestra.run.smithi089.stdout: "root": "/" 2024-03-31T15:03:59.917 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-03-31T15:03:59.917 INFO:teuthology.orchestra.run.smithi089.stdout: "dentry_count": 0, 2024-03-31T15:03:59.917 INFO:teuthology.orchestra.run.smithi089.stdout: "dentry_pinned_count": 0, 2024-03-31T15:03:59.918 INFO:teuthology.orchestra.run.smithi089.stdout: "id": 4711, 2024-03-31T15:03:59.918 INFO:teuthology.orchestra.run.smithi089.stdout: "inst": { 2024-03-31T15:03:59.918 INFO:teuthology.orchestra.run.smithi089.stdout: "name": { 2024-03-31T15:03:59.918 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "client", 2024-03-31T15:03:59.918 INFO:teuthology.orchestra.run.smithi089.stdout: "num": 4711 2024-03-31T15:03:59.918 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-03-31T15:03:59.918 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": { 2024-03-31T15:03:59.918 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-03-31T15:03:59.918 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "192.168.0.1:0", 2024-03-31T15:03:59.918 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 2060829056 2024-03-31T15:03:59.918 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-03-31T15:03:59.919 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-03-31T15:03:59.919 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": { 2024-03-31T15:03:59.919 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-03-31T15:03:59.919 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "192.168.0.1:0", 2024-03-31T15:03:59.919 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 2060829056 2024-03-31T15:03:59.919 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-03-31T15:03:59.919 INFO:teuthology.orchestra.run.smithi089.stdout: "inst_str": "client.4711 192.168.0.1:0/2060829056", 2024-03-31T15:03:59.919 INFO:teuthology.orchestra.run.smithi089.stdout: "addr_str": "192.168.0.1:0/2060829056", 2024-03-31T15:03:59.919 INFO:teuthology.orchestra.run.smithi089.stdout: "inode_count": 1, 2024-03-31T15:03:59.919 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_epoch": 8, 2024-03-31T15:03:59.919 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_epoch": 21, 2024-03-31T15:03:59.919 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_epoch_barrier": 0, 2024-03-31T15:03:59.919 INFO:teuthology.orchestra.run.smithi089.stdout: "blocklisted": false, 2024-03-31T15:03:59.919 INFO:teuthology.orchestra.run.smithi089.stdout: "fs_name": "cephfs" 2024-03-31T15:03:59.920 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-03-31T15:03:59.932 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-31T15:04:00.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:00.093+0000 7ffbc0800640 1 Processor -- start 2024-03-31T15:04:00.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:00.093+0000 7ffbc0800640 1 -- start start 2024-03-31T15:04:00.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:00.093+0000 7ffbc0800640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ffbb815b3d0 0x7ffbb80b1cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:00.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:00.093+0000 7ffbc0800640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ffbb815eee0 0x7ffbb80b2200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:00.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:00.093+0000 7ffbc0800640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7ffbb80ce2f0 0x7ffbb80cc9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:00.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:00.093+0000 7ffbc0800640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7ffbb815b7b0 con 0x7ffbb815eee0 2024-03-31T15:04:00.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:00.093+0000 7ffbc0800640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7ffbb815f3c0 con 0x7ffbb80ce2f0 2024-03-31T15:04:00.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:00.093+0000 7ffbc0800640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7ffbb815f6a0 con 0x7ffbb815b3d0 2024-03-31T15:04:00.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:00.093+0000 7ffbbd400640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ffbb815eee0 0x7ffbb80b2200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:00.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:00.093+0000 7ffbbde00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ffbb815b3d0 0x7ffbb80b1cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:00.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:00.093+0000 7ffbbe800640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7ffbb80ce2f0 0x7ffbb80cc9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:00.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:00.093+0000 7ffbbde00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ffbb815b3d0 0x7ffbb80b1cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3301/0 says I am v2:172.21.15.16:42266/0 (socket says 172.21.15.16:42266) 2024-03-31T15:04:00.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:00.093+0000 7ffbbde00640 1 -- 172.21.15.16:0/1091490330 learned_addr learned my addr 172.21.15.16:0/1091490330 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:04:00.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:00.093+0000 7ffbbd400640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ffbb815eee0 0x7ffbb80b2200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60150/0 (socket says 172.21.15.16:60150) 2024-03-31T15:04:00.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:00.097+0000 7ffbbde00640 1 -- 172.21.15.16:0/1091490330 >> v2:172.21.15.89:3300/0 conn(0x7ffbb80ce2f0 msgr2=0x7ffbb80cc9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:04:00.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:00.097+0000 7ffbbde00640 1 --2- 172.21.15.16:0/1091490330 >> v2:172.21.15.89:3300/0 conn(0x7ffbb80ce2f0 0x7ffbb80cc9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:00.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:00.097+0000 7ffbbde00640 1 -- 172.21.15.16:0/1091490330 >> v2:172.21.15.16:3300/0 conn(0x7ffbb815eee0 msgr2=0x7ffbb80b2200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:04:00.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:00.097+0000 7ffbbde00640 1 --2- 172.21.15.16:0/1091490330 >> v2:172.21.15.16:3300/0 conn(0x7ffbb815eee0 0x7ffbb80b2200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:00.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:00.097+0000 7ffbbde00640 1 -- 172.21.15.16:0/1091490330 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffbb815f980 con 0x7ffbb815b3d0 2024-03-31T15:04:00.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:00.097+0000 7ffbbe800640 1 --2- 172.21.15.16:0/1091490330 >> v2:172.21.15.89:3300/0 conn(0x7ffbb80ce2f0 0x7ffbb80cc9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:04:00.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:00.097+0000 7ffbbd400640 1 --2- 172.21.15.16:0/1091490330 >> v2:172.21.15.16:3300/0 conn(0x7ffbb815eee0 0x7ffbb80b2200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:04:00.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:00.097+0000 7ffbbde00640 1 --2- 172.21.15.16:0/1091490330 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ffbb815b3d0 0x7ffbb80b1cc0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7ffbb80b2df0 tx=0x7ffbac01c4f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=874e7c4d464dab97 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:00.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:00.097+0000 7ffbbca00640 1 -- 172.21.15.16:0/1091490330 <== mon.2 v2:172.21.15.89:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbac0a7070 con 0x7ffbb815b3d0 2024-03-31T15:04:00.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:00.097+0000 7ffbbca00640 1 -- 172.21.15.16:0/1091490330 <== mon.2 v2:172.21.15.89:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffbac0b0070 con 0x7ffbb815b3d0 2024-03-31T15:04:00.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:00.097+0000 7ffbbca00640 1 -- 172.21.15.16:0/1091490330 <== mon.2 v2:172.21.15.89:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbac0ab890 con 0x7ffbb815b3d0 2024-03-31T15:04:00.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:00.097+0000 7ffbc0800640 1 -- 172.21.15.16:0/1091490330 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ffbb815b3d0 msgr2=0x7ffbb80b1cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:00.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:00.097+0000 7ffbc0800640 1 --2- 172.21.15.16:0/1091490330 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ffbb815b3d0 0x7ffbb80b1cc0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7ffbb80b2df0 tx=0x7ffbac01c4f0 comp rx=0 tx=0).stop 2024-03-31T15:04:00.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:00.097+0000 7ffbc0800640 1 -- 172.21.15.16:0/1091490330 shutdown_connections 2024-03-31T15:04:00.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:00.097+0000 7ffbc0800640 1 --2- 172.21.15.16:0/1091490330 >> v2:172.21.15.89:3300/0 conn(0x7ffbb80ce2f0 0x7ffbb80cc9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:00.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:00.097+0000 7ffbc0800640 1 --2- 172.21.15.16:0/1091490330 >> v2:172.21.15.16:3300/0 conn(0x7ffbb815eee0 0x7ffbb80b2200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:00.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:00.097+0000 7ffbc0800640 1 --2- 172.21.15.16:0/1091490330 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ffbb815b3d0 0x7ffbb80b1cc0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:00.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:00.097+0000 7ffbc0800640 1 -- 172.21.15.16:0/1091490330 >> 172.21.15.16:0/1091490330 conn(0x7ffbb806b620 msgr2=0x7ffbb806ba30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:04:00.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:00.101+0000 7ffbc0800640 1 -- 172.21.15.16:0/1091490330 shutdown_connections 2024-03-31T15:04:00.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:00.101+0000 7ffbc0800640 1 -- 172.21.15.16:0/1091490330 wait complete. 2024-03-31T15:04:00.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:00.101+0000 7ffbc0800640 1 Processor -- start 2024-03-31T15:04:00.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:00.101+0000 7ffbc0800640 1 -- start start 2024-03-31T15:04:00.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:00.101+0000 7ffbc0800640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffbb80ce2f0 0x7ffbb81445b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:00.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:00.101+0000 7ffbc0800640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ffbb815b3d0 0x7ffbb8148970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:00.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:00.101+0000 7ffbc0800640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffbb815eee0 0x7ffbb814e220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:00.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:00.101+0000 7ffbc0800640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7ffbb8071e90 con 0x7ffbb815eee0 2024-03-31T15:04:00.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:00.101+0000 7ffbc0800640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7ffbb8071d10 con 0x7ffbb80ce2f0 2024-03-31T15:04:00.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:00.101+0000 7ffbc0800640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7ffbb8072010 con 0x7ffbb815b3d0 2024-03-31T15:04:00.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:00.101+0000 7ffbbe800640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffbb815eee0 0x7ffbb814e220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:00.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:00.101+0000 7ffbbe800640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffbb815eee0 0x7ffbb814e220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60154/0 (socket says 172.21.15.16:60154) 2024-03-31T15:04:00.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:00.101+0000 7ffbbe800640 1 -- 172.21.15.16:0/1967867957 learned_addr learned my addr 172.21.15.16:0/1967867957 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:04:00.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:00.101+0000 7ffbbd400640 1 --2- 172.21.15.16:0/1967867957 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ffbb815b3d0 0x7ffbb8148970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:00.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:00.101+0000 7ffbbde00640 1 --2- 172.21.15.16:0/1967867957 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffbb80ce2f0 0x7ffbb81445b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:00.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:00.101+0000 7ffbbe800640 1 -- 172.21.15.16:0/1967867957 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ffbb815b3d0 msgr2=0x7ffbb8148970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:00.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:00.101+0000 7ffbbe800640 1 --2- 172.21.15.16:0/1967867957 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ffbb815b3d0 0x7ffbb8148970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:00.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:00.101+0000 7ffbbe800640 1 -- 172.21.15.16:0/1967867957 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffbb80ce2f0 msgr2=0x7ffbb81445b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:00.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:00.101+0000 7ffbbe800640 1 --2- 172.21.15.16:0/1967867957 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffbb80ce2f0 0x7ffbb81445b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:00.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:00.101+0000 7ffbbe800640 1 -- 172.21.15.16:0/1967867957 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffbb81476c0 con 0x7ffbb815eee0 2024-03-31T15:04:00.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:00.101+0000 7ffbbe800640 1 --2- 172.21.15.16:0/1967867957 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffbb815eee0 0x7ffbb814e220 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7ffbb400eec0 tx=0x7ffbb400b480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:00.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:00.101+0000 7ffbb2000640 1 -- 172.21.15.16:0/1967867957 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbb4018020 con 0x7ffbb815eee0 2024-03-31T15:04:00.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:00.101+0000 7ffbb2000640 1 -- 172.21.15.16:0/1967867957 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffbb4014400 con 0x7ffbb815eee0 2024-03-31T15:04:00.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:00.101+0000 7ffbb2000640 1 -- 172.21.15.16:0/1967867957 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbb40147d0 con 0x7ffbb815eee0 2024-03-31T15:04:00.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:00.101+0000 7ffbc0800640 1 -- 172.21.15.16:0/1967867957 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffbb81479a0 con 0x7ffbb815eee0 2024-03-31T15:04:00.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:00.105+0000 7ffbc0800640 1 -- 172.21.15.16:0/1967867957 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffbb8147f60 con 0x7ffbb815eee0 2024-03-31T15:04:00.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:00.105+0000 7ffbc0800640 1 -- 172.21.15.16:0/1967867957 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffbb8069950 con 0x7ffbb815eee0 2024-03-31T15:04:00.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:00.105+0000 7ffbb2000640 1 -- 172.21.15.16:0/1967867957 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7ffbb401b020 con 0x7ffbb815eee0 2024-03-31T15:04:00.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:00.105+0000 7ffbb2000640 1 --2- 172.21.15.16:0/1967867957 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7ffb9c0683d0 0x7ffb9c06a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:00.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:00.105+0000 7ffbb2000640 1 -- 172.21.15.16:0/1967867957 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7ffbb4087210 con 0x7ffbb815eee0 2024-03-31T15:04:00.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:00.105+0000 7ffbbde00640 1 --2- 172.21.15.16:0/1967867957 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7ffb9c0683d0 0x7ffb9c06a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:00.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:00.105+0000 7ffbbde00640 1 --2- 172.21.15.16:0/1967867957 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7ffb9c0683d0 0x7ffb9c06a890 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7ffbac0af000 tx=0x7ffbac0a7000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:00.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:00.109+0000 7ffbb2000640 1 -- 172.21.15.16:0/1967867957 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7ffbb4022030 con 0x7ffbb815eee0 2024-03-31T15:04:00.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:00.309+0000 7ffbc0800640 1 -- 172.21.15.16:0/1967867957 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7ffbb80cdea0 con 0x7ffbb815eee0 2024-03-31T15:04:00.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:00.309+0000 7ffbb2000640 1 -- 172.21.15.16:0/1967867957 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v8) ==== 53+0+73 (secure 0 0 0) 0x7ffbb4057160 con 0x7ffbb815eee0 2024-03-31T15:04:00.304 INFO:teuthology.orchestra.run.smithi016.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-31T15:04:00.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:00.313+0000 7ffbc0800640 1 -- 172.21.15.16:0/1967867957 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7ffb9c0683d0 msgr2=0x7ffb9c06a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:00.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:00.313+0000 7ffbc0800640 1 --2- 172.21.15.16:0/1967867957 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7ffb9c0683d0 0x7ffb9c06a890 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7ffbac0af000 tx=0x7ffbac0a7000 comp rx=0 tx=0).stop 2024-03-31T15:04:00.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:00.313+0000 7ffbc0800640 1 -- 172.21.15.16:0/1967867957 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffbb815eee0 msgr2=0x7ffbb814e220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:00.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:00.313+0000 7ffbc0800640 1 --2- 172.21.15.16:0/1967867957 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffbb815eee0 0x7ffbb814e220 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7ffbb400eec0 tx=0x7ffbb400b480 comp rx=0 tx=0).stop 2024-03-31T15:04:00.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:00.313+0000 7ffbc0800640 1 -- 172.21.15.16:0/1967867957 shutdown_connections 2024-03-31T15:04:00.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:00.313+0000 7ffbc0800640 1 --2- 172.21.15.16:0/1967867957 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7ffb9c0683d0 0x7ffb9c06a890 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:00.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:00.313+0000 7ffbc0800640 1 --2- 172.21.15.16:0/1967867957 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffbb815eee0 0x7ffbb814e220 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:00.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:00.313+0000 7ffbc0800640 1 --2- 172.21.15.16:0/1967867957 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ffbb815b3d0 0x7ffbb8148970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:00.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:00.313+0000 7ffbc0800640 1 --2- 172.21.15.16:0/1967867957 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffbb80ce2f0 0x7ffbb81445b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:00.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:00.313+0000 7ffbc0800640 1 -- 172.21.15.16:0/1967867957 >> 172.21.15.16:0/1967867957 conn(0x7ffbb806b620 msgr2=0x7ffbb80b34f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:04:00.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:00.313+0000 7ffbc0800640 1 -- 172.21.15.16:0/1967867957 shutdown_connections 2024-03-31T15:04:00.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:00.313+0000 7ffbc0800640 1 -- 172.21.15.16:0/1967867957 wait complete. 2024-03-31T15:04:00.325 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-31T15:04:00.325 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-31T15:04:00.326 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi089.front.sepia.ceph.com 2024-03-31T15:04:00.326 INFO:tasks.cephfs.mount:self.client.name = client.3 2024-03-31T15:04:00.326 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.3 2024-03-31T15:04:00.326 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-31T15:04:00.326 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-31T15:04:00.326 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-31T15:04:00.326 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.3' 2024-03-31T15:04:00.326 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:04:00.326 DEBUG:teuthology.orchestra.run.smithi089:> ip addr 2024-03-31T15:04:00.336 INFO:teuthology.orchestra.run.smithi089.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-31T15:04:00.336 INFO:teuthology.orchestra.run.smithi089.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-31T15:04:00.336 INFO:teuthology.orchestra.run.smithi089.stdout: inet 127.0.0.1/8 scope host lo 2024-03-31T15:04:00.336 INFO:teuthology.orchestra.run.smithi089.stdout: valid_lft forever preferred_lft forever 2024-03-31T15:04:00.336 INFO:teuthology.orchestra.run.smithi089.stdout: inet6 ::1/128 scope host 2024-03-31T15:04:00.336 INFO:teuthology.orchestra.run.smithi089.stdout: valid_lft forever preferred_lft forever 2024-03-31T15:04:00.336 INFO:teuthology.orchestra.run.smithi089.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-03-31T15:04:00.336 INFO:teuthology.orchestra.run.smithi089.stdout: link/ether 0c:c4:7a:d8:a4:c6 brd ff:ff:ff:ff:ff:ff 2024-03-31T15:04:00.336 INFO:teuthology.orchestra.run.smithi089.stdout: altname enp7s0f0 2024-03-31T15:04:00.336 INFO:teuthology.orchestra.run.smithi089.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-03-31T15:04:00.336 INFO:teuthology.orchestra.run.smithi089.stdout: link/ether 0c:c4:7a:d8:a4:c7 brd ff:ff:ff:ff:ff:ff 2024-03-31T15:04:00.336 INFO:teuthology.orchestra.run.smithi089.stdout: altname enp7s0f1 2024-03-31T15:04:00.337 INFO:teuthology.orchestra.run.smithi089.stdout:4: enp3s0f0: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-03-31T15:04:00.337 INFO:teuthology.orchestra.run.smithi089.stdout: link/ether 0c:c4:7a:8f:d5:00 brd ff:ff:ff:ff:ff:ff 2024-03-31T15:04:00.337 INFO:teuthology.orchestra.run.smithi089.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-31T15:04:00.337 INFO:teuthology.orchestra.run.smithi089.stdout: link/ether 0c:c4:7a:8f:d5:01 brd ff:ff:ff:ff:ff:ff 2024-03-31T15:04:00.337 INFO:teuthology.orchestra.run.smithi089.stdout: inet 172.21.15.89/20 metric 100 brd 172.21.15.255 scope global dynamic enp3s0f1 2024-03-31T15:04:00.337 INFO:teuthology.orchestra.run.smithi089.stdout: valid_lft 42443sec preferred_lft 42443sec 2024-03-31T15:04:00.337 INFO:teuthology.orchestra.run.smithi089.stdout: inet6 fe80::ec4:7aff:fe8f:d501/64 scope link 2024-03-31T15:04:00.337 INFO:teuthology.orchestra.run.smithi089.stdout: valid_lft forever preferred_lft forever 2024-03-31T15:04:00.337 INFO:teuthology.orchestra.run.smithi089.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2024-03-31T15:04:00.337 INFO:teuthology.orchestra.run.smithi089.stdout: link/ether 02:42:b1:c8:45:10 brd ff:ff:ff:ff:ff:ff 2024-03-31T15:04:00.337 INFO:teuthology.orchestra.run.smithi089.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2024-03-31T15:04:00.337 INFO:teuthology.orchestra.run.smithi089.stdout: valid_lft forever preferred_lft forever 2024-03-31T15:04:00.337 INFO:teuthology.orchestra.run.smithi089.stdout:7: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-31T15:04:00.337 INFO:teuthology.orchestra.run.smithi089.stdout: link/ether 5e:1f:84:6d:79:ae brd ff:ff:ff:ff:ff:ff 2024-03-31T15:04:00.338 INFO:teuthology.orchestra.run.smithi089.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-31T15:04:00.338 INFO:teuthology.orchestra.run.smithi089.stdout: valid_lft forever preferred_lft forever 2024-03-31T15:04:00.338 INFO:teuthology.orchestra.run.smithi089.stdout: inet6 fe80::5c1f:84ff:fe6d:79ae/64 scope link 2024-03-31T15:04:00.338 INFO:teuthology.orchestra.run.smithi089.stdout: valid_lft forever preferred_lft forever 2024-03-31T15:04:00.338 INFO:teuthology.orchestra.run.smithi089.stdout:8: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-31T15:04:00.338 INFO:teuthology.orchestra.run.smithi089.stdout: link/ether 0e:5c:f6:1f:82:2f brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-31T15:04:00.338 INFO:teuthology.orchestra.run.smithi089.stdout: inet6 fe80::c5c:f6ff:fe1f:822f/64 scope link 2024-03-31T15:04:00.338 INFO:teuthology.orchestra.run.smithi089.stdout: valid_lft forever preferred_lft forever 2024-03-31T15:04:00.338 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:04:00.338 DEBUG:teuthology.orchestra.run.smithi089:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-31T15:04:00.397 INFO:teuthology.orchestra.run.smithi089.stdout:1 2024-03-31T15:04:00.400 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:04:00.400 DEBUG:teuthology.orchestra.run.smithi089:> route 2024-03-31T15:04:00.453 INFO:teuthology.orchestra.run.smithi089.stdout:Kernel IP routing table 2024-03-31T15:04:00.453 INFO:teuthology.orchestra.run.smithi089.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-31T15:04:00.453 INFO:teuthology.orchestra.run.smithi089.stdout:default _gateway 0.0.0.0 UG 0 0 0 enp3s0f1 2024-03-31T15:04:00.453 INFO:teuthology.orchestra.run.smithi089.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-31T15:04:00.453 INFO:teuthology.orchestra.run.smithi089.stdout:158.69.67.47 _gateway 255.255.255.255 UGH 100 0 0 enp3s0f1 2024-03-31T15:04:00.453 INFO:teuthology.orchestra.run.smithi089.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2024-03-31T15:04:00.454 INFO:teuthology.orchestra.run.smithi089.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-31T15:04:00.454 INFO:teuthology.orchestra.run.smithi089.stdout:ns1.front.sepia 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-03-31T15:04:00.454 INFO:teuthology.orchestra.run.smithi089.stdout:172.21.0.2 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-03-31T15:04:00.454 INFO:teuthology.orchestra.run.smithi089.stdout:_gateway 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-03-31T15:04:00.454 INFO:teuthology.orchestra.run.smithi089.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-31T15:04:00.454 DEBUG:teuthology.orchestra.run.smithi089:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-31T15:04:00.454 DEBUG:teuthology.orchestra.run.smithi089:> set -e 2024-03-31T15:04:00.454 DEBUG:teuthology.orchestra.run.smithi089:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-31T15:04:00.454 DEBUG:teuthology.orchestra.run.smithi089:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-31T15:04:00.454 DEBUG:teuthology.orchestra.run.smithi089:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-31T15:04:00.455 DEBUG:teuthology.orchestra.run.smithi089:> ') 2024-03-31T15:04:00.546 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:04:00 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-31T15:04:00.588 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:04:00 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-31T15:04:00.594 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:04:00.594 DEBUG:teuthology.orchestra.run.smithi089:> ip netns list 2024-03-31T15:04:00.647 INFO:teuthology.orchestra.run.smithi089.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-03-31T15:04:00.648 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:04:00.648 DEBUG:teuthology.orchestra.run.smithi089:> ip netns list-id 2024-03-31T15:04:00.702 INFO:teuthology.orchestra.run.smithi089.stdout:nsid 0 (iproute2 netns name: ceph-ns--home-ubuntu-cephtest-mnt.1) 2024-03-31T15:04:00.703 DEBUG:teuthology.orchestra.run.smithi089:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-31T15:04:00.703 DEBUG:teuthology.orchestra.run.smithi089:> set -e 2024-03-31T15:04:00.703 DEBUG:teuthology.orchestra.run.smithi089:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.3 2024-03-31T15:04:00.704 DEBUG:teuthology.orchestra.run.smithi089:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.3 1 2024-03-31T15:04:00.704 DEBUG:teuthology.orchestra.run.smithi089:> ') 2024-03-31T15:04:00.796 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:04:00 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-31T15:04:00.825 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:04:00 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-31T15:04:00.832 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:04:00.833 DEBUG:teuthology.orchestra.run.smithi089:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr 2024-03-31T15:04:00.899 INFO:teuthology.orchestra.run.smithi089.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-31T15:04:00.899 INFO:teuthology.orchestra.run.smithi089.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-31T15:04:00.899 INFO:teuthology.orchestra.run.smithi089.stdout: inet 127.0.0.1/8 scope host lo 2024-03-31T15:04:00.899 INFO:teuthology.orchestra.run.smithi089.stdout: valid_lft forever preferred_lft forever 2024-03-31T15:04:00.899 INFO:teuthology.orchestra.run.smithi089.stdout: inet6 ::1/128 scope host 2024-03-31T15:04:00.899 INFO:teuthology.orchestra.run.smithi089.stdout: valid_lft forever preferred_lft forever 2024-03-31T15:04:00.899 INFO:teuthology.orchestra.run.smithi089.stdout:2: veth0@if8: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-31T15:04:00.899 INFO:teuthology.orchestra.run.smithi089.stdout: link/ether 22:88:5f:cd:1e:4b brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-03-31T15:04:00.900 INFO:teuthology.orchestra.run.smithi089.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-03-31T15:04:00.900 INFO:teuthology.orchestra.run.smithi089.stdout: valid_lft forever preferred_lft forever 2024-03-31T15:04:00.900 INFO:teuthology.orchestra.run.smithi089.stdout: inet6 fe80::2088:5fff:fecd:1e4b/64 scope link 2024-03-31T15:04:00.900 INFO:teuthology.orchestra.run.smithi089.stdout: valid_lft forever preferred_lft forever 2024-03-31T15:04:00.901 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:04:00.901 DEBUG:teuthology.orchestra.run.smithi089:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr 2024-03-31T15:04:00.963 INFO:teuthology.orchestra.run.smithi089.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-31T15:04:00.963 INFO:teuthology.orchestra.run.smithi089.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-31T15:04:00.963 INFO:teuthology.orchestra.run.smithi089.stdout: inet 127.0.0.1/8 scope host lo 2024-03-31T15:04:00.963 INFO:teuthology.orchestra.run.smithi089.stdout: valid_lft forever preferred_lft forever 2024-03-31T15:04:00.964 INFO:teuthology.orchestra.run.smithi089.stdout: inet6 ::1/128 scope host 2024-03-31T15:04:00.964 INFO:teuthology.orchestra.run.smithi089.stdout: valid_lft forever preferred_lft forever 2024-03-31T15:04:00.964 INFO:teuthology.orchestra.run.smithi089.stdout:2: veth0@if8: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-31T15:04:00.964 INFO:teuthology.orchestra.run.smithi089.stdout: link/ether 22:88:5f:cd:1e:4b brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-03-31T15:04:00.964 INFO:teuthology.orchestra.run.smithi089.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-03-31T15:04:00.964 INFO:teuthology.orchestra.run.smithi089.stdout: valid_lft forever preferred_lft forever 2024-03-31T15:04:00.964 INFO:teuthology.orchestra.run.smithi089.stdout: inet6 fe80::2088:5fff:fecd:1e4b/64 scope link 2024-03-31T15:04:00.964 INFO:teuthology.orchestra.run.smithi089.stdout: valid_lft forever preferred_lft forever 2024-03-31T15:04:00.965 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.3' with 192.168.0.2/16 2024-03-31T15:04:00.965 DEBUG:teuthology.orchestra.run.smithi089:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-31T15:04:00.966 DEBUG:teuthology.orchestra.run.smithi089:> set -e 2024-03-31T15:04:00.966 DEBUG:teuthology.orchestra.run.smithi089:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.3 type veth peer name brx.1 2024-03-31T15:04:00.966 DEBUG:teuthology.orchestra.run.smithi089:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip addr add 192.168.0.2/16 brd 192.168.255.255 dev veth0 2024-03-31T15:04:00.966 DEBUG:teuthology.orchestra.run.smithi089:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip link set veth0 up 2024-03-31T15:04:00.966 DEBUG:teuthology.orchestra.run.smithi089:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip link set lo up 2024-03-31T15:04:00.966 DEBUG:teuthology.orchestra.run.smithi089:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip route add default via 192.168.255.254 2024-03-31T15:04:00.966 DEBUG:teuthology.orchestra.run.smithi089:> ') 2024-03-31T15:04:01.058 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:04:01 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-31T15:04:01.143 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:04:01 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-31T15:04:01.150 DEBUG:teuthology.orchestra.run.smithi089:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-31T15:04:01.150 DEBUG:teuthology.orchestra.run.smithi089:> set -e 2024-03-31T15:04:01.151 DEBUG:teuthology.orchestra.run.smithi089:> sudo ip link set brx.1 up 2024-03-31T15:04:01.151 DEBUG:teuthology.orchestra.run.smithi089:> sudo ip link set dev brx.1 master ceph-brx 2024-03-31T15:04:01.151 DEBUG:teuthology.orchestra.run.smithi089:> ') 2024-03-31T15:04:01.198 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:04:01 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-31T15:04:01.228 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:04:01 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-31T15:04:01.238 INFO:tasks.cephfs.fuse_mount:Client client.3 config is {} 2024-03-31T15:04:01.238 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-31T15:04:01.238 DEBUG:teuthology.orchestra.run.smithi089:> mkdir -p -v /home/ubuntu/cephtest/mnt.3 2024-03-31T15:04:01.290 INFO:teuthology.orchestra.run.smithi089.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.3' 2024-03-31T15:04:01.290 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-31T15:04:01.290 DEBUG:teuthology.orchestra.run.smithi089:> chmod 0000 /home/ubuntu/cephtest/mnt.3 2024-03-31T15:04:01.342 DEBUG:teuthology.orchestra.run.smithi089:> sudo modprobe fuse 2024-03-31T15:04:01.407 DEBUG:teuthology.orchestra.run.smithi089:> cat /proc/self/mounts | awk '{print $2}' 2024-03-31T15:04:01.459 INFO:teuthology.orchestra.run.smithi089.stdout:/sys 2024-03-31T15:04:01.459 INFO:teuthology.orchestra.run.smithi089.stdout:/proc 2024-03-31T15:04:01.459 INFO:teuthology.orchestra.run.smithi089.stdout:/dev 2024-03-31T15:04:01.459 INFO:teuthology.orchestra.run.smithi089.stdout:/dev/pts 2024-03-31T15:04:01.459 INFO:teuthology.orchestra.run.smithi089.stdout:/run 2024-03-31T15:04:01.460 INFO:teuthology.orchestra.run.smithi089.stdout:/ 2024-03-31T15:04:01.460 INFO:teuthology.orchestra.run.smithi089.stdout:/sys/kernel/security 2024-03-31T15:04:01.460 INFO:teuthology.orchestra.run.smithi089.stdout:/dev/shm 2024-03-31T15:04:01.460 INFO:teuthology.orchestra.run.smithi089.stdout:/run/lock 2024-03-31T15:04:01.460 INFO:teuthology.orchestra.run.smithi089.stdout:/sys/fs/cgroup 2024-03-31T15:04:01.460 INFO:teuthology.orchestra.run.smithi089.stdout:/sys/fs/pstore 2024-03-31T15:04:01.460 INFO:teuthology.orchestra.run.smithi089.stdout:/sys/fs/bpf 2024-03-31T15:04:01.460 INFO:teuthology.orchestra.run.smithi089.stdout:/proc/sys/fs/binfmt_misc 2024-03-31T15:04:01.460 INFO:teuthology.orchestra.run.smithi089.stdout:/dev/hugepages 2024-03-31T15:04:01.460 INFO:teuthology.orchestra.run.smithi089.stdout:/dev/mqueue 2024-03-31T15:04:01.460 INFO:teuthology.orchestra.run.smithi089.stdout:/sys/kernel/debug 2024-03-31T15:04:01.460 INFO:teuthology.orchestra.run.smithi089.stdout:/sys/kernel/tracing 2024-03-31T15:04:01.460 INFO:teuthology.orchestra.run.smithi089.stdout:/proc/fs/nfsd 2024-03-31T15:04:01.460 INFO:teuthology.orchestra.run.smithi089.stdout:/sys/fs/fuse/connections 2024-03-31T15:04:01.461 INFO:teuthology.orchestra.run.smithi089.stdout:/sys/kernel/config 2024-03-31T15:04:01.461 INFO:teuthology.orchestra.run.smithi089.stdout:/run/credentials/systemd-sysusers.service 2024-03-31T15:04:01.461 INFO:teuthology.orchestra.run.smithi089.stdout:/run/rpc_pipefs 2024-03-31T15:04:01.461 INFO:teuthology.orchestra.run.smithi089.stdout:/run/user/1000 2024-03-31T15:04:01.461 INFO:teuthology.orchestra.run.smithi089.stdout:/var/lib/ceph 2024-03-31T15:04:01.461 INFO:teuthology.orchestra.run.smithi089.stdout:/var/lib/ceph/osd/ceph-4 2024-03-31T15:04:01.461 INFO:teuthology.orchestra.run.smithi089.stdout:/var/lib/ceph/osd/ceph-5 2024-03-31T15:04:01.461 INFO:teuthology.orchestra.run.smithi089.stdout:/var/lib/ceph/osd/ceph-6 2024-03-31T15:04:01.461 INFO:teuthology.orchestra.run.smithi089.stdout:/var/lib/ceph/osd/ceph-7 2024-03-31T15:04:01.461 INFO:teuthology.orchestra.run.smithi089.stdout:/run/netns 2024-03-31T15:04:01.461 INFO:teuthology.orchestra.run.smithi089.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-31T15:04:01.461 INFO:teuthology.orchestra.run.smithi089.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-31T15:04:01.461 INFO:teuthology.orchestra.run.smithi089.stdout:/home/ubuntu/cephtest/mnt.1 2024-03-31T15:04:01.461 INFO:teuthology.orchestra.run.smithi089.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-03-31T15:04:01.462 INFO:teuthology.orchestra.run.smithi089.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-03-31T15:04:01.462 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:04:01.463 DEBUG:teuthology.orchestra.run.smithi089:> ls /sys/fs/fuse/connections 2024-03-31T15:04:01.513 INFO:teuthology.orchestra.run.smithi089.stdout:43 2024-03-31T15:04:01.514 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [43] 2024-03-31T15:04:01.514 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.3 --id 3) 2024-03-31T15:04:01.557 DEBUG:teuthology.orchestra.run.smithi089:> sudo modprobe fuse 2024-03-31T15:04:01.577 DEBUG:teuthology.orchestra.run.smithi089:> cat /proc/self/mounts | awk '{print $2}' 2024-03-31T15:04:01.629 INFO:teuthology.orchestra.run.smithi089.stdout:/sys 2024-03-31T15:04:01.629 INFO:teuthology.orchestra.run.smithi089.stdout:/proc 2024-03-31T15:04:01.629 INFO:teuthology.orchestra.run.smithi089.stdout:/dev 2024-03-31T15:04:01.629 INFO:teuthology.orchestra.run.smithi089.stdout:/dev/pts 2024-03-31T15:04:01.629 INFO:teuthology.orchestra.run.smithi089.stdout:/run 2024-03-31T15:04:01.629 INFO:teuthology.orchestra.run.smithi089.stdout:/ 2024-03-31T15:04:01.629 INFO:teuthology.orchestra.run.smithi089.stdout:/sys/kernel/security 2024-03-31T15:04:01.630 INFO:teuthology.orchestra.run.smithi089.stdout:/dev/shm 2024-03-31T15:04:01.630 INFO:teuthology.orchestra.run.smithi089.stdout:/run/lock 2024-03-31T15:04:01.630 INFO:teuthology.orchestra.run.smithi089.stdout:/sys/fs/cgroup 2024-03-31T15:04:01.630 INFO:teuthology.orchestra.run.smithi089.stdout:/sys/fs/pstore 2024-03-31T15:04:01.630 INFO:teuthology.orchestra.run.smithi089.stdout:/sys/fs/bpf 2024-03-31T15:04:01.630 INFO:teuthology.orchestra.run.smithi089.stdout:/proc/sys/fs/binfmt_misc 2024-03-31T15:04:01.630 INFO:teuthology.orchestra.run.smithi089.stdout:/dev/hugepages 2024-03-31T15:04:01.630 INFO:teuthology.orchestra.run.smithi089.stdout:/dev/mqueue 2024-03-31T15:04:01.630 INFO:teuthology.orchestra.run.smithi089.stdout:/sys/kernel/debug 2024-03-31T15:04:01.630 INFO:teuthology.orchestra.run.smithi089.stdout:/sys/kernel/tracing 2024-03-31T15:04:01.630 INFO:teuthology.orchestra.run.smithi089.stdout:/proc/fs/nfsd 2024-03-31T15:04:01.630 INFO:teuthology.orchestra.run.smithi089.stdout:/sys/fs/fuse/connections 2024-03-31T15:04:01.630 INFO:teuthology.orchestra.run.smithi089.stdout:/sys/kernel/config 2024-03-31T15:04:01.631 INFO:teuthology.orchestra.run.smithi089.stdout:/run/credentials/systemd-sysusers.service 2024-03-31T15:04:01.631 INFO:teuthology.orchestra.run.smithi089.stdout:/run/rpc_pipefs 2024-03-31T15:04:01.631 INFO:teuthology.orchestra.run.smithi089.stdout:/run/user/1000 2024-03-31T15:04:01.631 INFO:teuthology.orchestra.run.smithi089.stdout:/var/lib/ceph 2024-03-31T15:04:01.631 INFO:teuthology.orchestra.run.smithi089.stdout:/var/lib/ceph/osd/ceph-4 2024-03-31T15:04:01.631 INFO:teuthology.orchestra.run.smithi089.stdout:/var/lib/ceph/osd/ceph-5 2024-03-31T15:04:01.631 INFO:teuthology.orchestra.run.smithi089.stdout:/var/lib/ceph/osd/ceph-6 2024-03-31T15:04:01.631 INFO:teuthology.orchestra.run.smithi089.stdout:/var/lib/ceph/osd/ceph-7 2024-03-31T15:04:01.631 INFO:teuthology.orchestra.run.smithi089.stdout:/run/netns 2024-03-31T15:04:01.631 INFO:teuthology.orchestra.run.smithi089.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-31T15:04:01.631 INFO:teuthology.orchestra.run.smithi089.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-31T15:04:01.631 INFO:teuthology.orchestra.run.smithi089.stdout:/home/ubuntu/cephtest/mnt.1 2024-03-31T15:04:01.631 INFO:teuthology.orchestra.run.smithi089.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-03-31T15:04:01.631 INFO:teuthology.orchestra.run.smithi089.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-03-31T15:04:01.632 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:04:01.632 DEBUG:teuthology.orchestra.run.smithi089:> ls /sys/fs/fuse/connections 2024-03-31T15:04:01.650 INFO:tasks.cephfs.fuse_mount.ceph-fuse.3.smithi089.stderr:2024-03-31T15:04:01.656+0000 7f3696848580 -1 init, newargv = 0x5596204bc400 newargc=15 2024-03-31T15:04:01.650 INFO:tasks.cephfs.fuse_mount.ceph-fuse.3.smithi089.stderr:ceph-fuse[33047]: starting ceph client 2024-03-31T15:04:01.657 INFO:tasks.cephfs.fuse_mount.ceph-fuse.3.smithi089.stderr:ceph-fuse[33047]: starting fuse 2024-03-31T15:04:01.660 INFO:teuthology.orchestra.run.smithi089.stdout:43 2024-03-31T15:04:01.660 INFO:teuthology.orchestra.run.smithi089.stdout:44 2024-03-31T15:04:01.660 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [43, 44] 2024-03-31T15:04:01.660 DEBUG:teuthology.orchestra.run.smithi089:> sudo stdin-killer -- python3 -c ' 2024-03-31T15:04:01.661 DEBUG:teuthology.orchestra.run.smithi089:> import glob 2024-03-31T15:04:01.661 DEBUG:teuthology.orchestra.run.smithi089:> import re 2024-03-31T15:04:01.661 DEBUG:teuthology.orchestra.run.smithi089:> import os 2024-03-31T15:04:01.661 DEBUG:teuthology.orchestra.run.smithi089:> import subprocess 2024-03-31T15:04:01.661 DEBUG:teuthology.orchestra.run.smithi089:> 2024-03-31T15:04:01.661 DEBUG:teuthology.orchestra.run.smithi089:> def _find_admin_socket(client_name): 2024-03-31T15:04:01.661 DEBUG:teuthology.orchestra.run.smithi089:> asok_path = "/var/run/ceph/ceph-client.3.*.asok" 2024-03-31T15:04:01.661 DEBUG:teuthology.orchestra.run.smithi089:> files = glob.glob(asok_path) 2024-03-31T15:04:01.661 DEBUG:teuthology.orchestra.run.smithi089:> mountpoint = "/home/ubuntu/cephtest/mnt.3" 2024-03-31T15:04:01.661 DEBUG:teuthology.orchestra.run.smithi089:> 2024-03-31T15:04:01.661 DEBUG:teuthology.orchestra.run.smithi089:> # Given a non-glob path, it better be there 2024-03-31T15:04:01.661 DEBUG:teuthology.orchestra.run.smithi089:> if "*" not in asok_path: 2024-03-31T15:04:01.661 DEBUG:teuthology.orchestra.run.smithi089:> assert(len(files) == 1) 2024-03-31T15:04:01.661 DEBUG:teuthology.orchestra.run.smithi089:> return files[0] 2024-03-31T15:04:01.662 DEBUG:teuthology.orchestra.run.smithi089:> 2024-03-31T15:04:01.662 DEBUG:teuthology.orchestra.run.smithi089:> for f in files: 2024-03-31T15:04:01.662 DEBUG:teuthology.orchestra.run.smithi089:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-31T15:04:01.662 DEBUG:teuthology.orchestra.run.smithi089:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-31T15:04:01.662 DEBUG:teuthology.orchestra.run.smithi089:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-31T15:04:01.662 DEBUG:teuthology.orchestra.run.smithi089:> contents = proc_f.read() 2024-03-31T15:04:01.662 DEBUG:teuthology.orchestra.run.smithi089:> if mountpoint in contents: 2024-03-31T15:04:01.662 DEBUG:teuthology.orchestra.run.smithi089:> return f 2024-03-31T15:04:01.662 DEBUG:teuthology.orchestra.run.smithi089:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-31T15:04:01.662 DEBUG:teuthology.orchestra.run.smithi089:> 2024-03-31T15:04:01.662 DEBUG:teuthology.orchestra.run.smithi089:> print(_find_admin_socket("client.3")) 2024-03-31T15:04:01.662 DEBUG:teuthology.orchestra.run.smithi089:> ' 2024-03-31T15:04:01.782 INFO:teuthology.orchestra.run.smithi089.stdout:/var/run/ceph/ceph-client.3.33047.asok 2024-03-31T15:04:01.785 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-31T15:04:01 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-31T15:04:01.794 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.3.33047.asok 2024-03-31T15:04:01.794 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:04:01.794 DEBUG:teuthology.orchestra.run.smithi089:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.3.33047.asok status 2024-03-31T15:04:01.970 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-03-31T15:04:01.970 INFO:teuthology.orchestra.run.smithi089.stdout: "metadata": { 2024-03-31T15:04:01.971 INFO:teuthology.orchestra.run.smithi089.stdout: "ceph_sha1": "d2271579d5e735cfd6d436b8349c004f06527cb0", 2024-03-31T15:04:01.971 INFO:teuthology.orchestra.run.smithi089.stdout: "ceph_version": "ceph version 19.0.0-2574-gd2271579 (d2271579d5e735cfd6d436b8349c004f06527cb0) squid (dev)", 2024-03-31T15:04:01.971 INFO:teuthology.orchestra.run.smithi089.stdout: "entity_id": "3", 2024-03-31T15:04:01.971 INFO:teuthology.orchestra.run.smithi089.stdout: "hostname": "smithi089", 2024-03-31T15:04:01.971 INFO:teuthology.orchestra.run.smithi089.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.3", 2024-03-31T15:04:01.971 INFO:teuthology.orchestra.run.smithi089.stdout: "pid": "33047", 2024-03-31T15:04:01.971 INFO:teuthology.orchestra.run.smithi089.stdout: "root": "/" 2024-03-31T15:04:01.971 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-03-31T15:04:01.971 INFO:teuthology.orchestra.run.smithi089.stdout: "dentry_count": 0, 2024-03-31T15:04:01.971 INFO:teuthology.orchestra.run.smithi089.stdout: "dentry_pinned_count": 0, 2024-03-31T15:04:01.971 INFO:teuthology.orchestra.run.smithi089.stdout: "id": 4726, 2024-03-31T15:04:01.971 INFO:teuthology.orchestra.run.smithi089.stdout: "inst": { 2024-03-31T15:04:01.971 INFO:teuthology.orchestra.run.smithi089.stdout: "name": { 2024-03-31T15:04:01.971 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "client", 2024-03-31T15:04:01.972 INFO:teuthology.orchestra.run.smithi089.stdout: "num": 4726 2024-03-31T15:04:01.972 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-03-31T15:04:01.972 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": { 2024-03-31T15:04:01.972 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-03-31T15:04:01.972 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "192.168.0.2:0", 2024-03-31T15:04:01.972 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 3385959794 2024-03-31T15:04:01.972 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-03-31T15:04:01.972 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-03-31T15:04:01.972 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": { 2024-03-31T15:04:01.972 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-03-31T15:04:01.972 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "192.168.0.2:0", 2024-03-31T15:04:01.972 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 3385959794 2024-03-31T15:04:01.972 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-03-31T15:04:01.973 INFO:teuthology.orchestra.run.smithi089.stdout: "inst_str": "client.4726 192.168.0.2:0/3385959794", 2024-03-31T15:04:01.973 INFO:teuthology.orchestra.run.smithi089.stdout: "addr_str": "192.168.0.2:0/3385959794", 2024-03-31T15:04:01.973 INFO:teuthology.orchestra.run.smithi089.stdout: "inode_count": 1, 2024-03-31T15:04:01.973 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_epoch": 8, 2024-03-31T15:04:01.973 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_epoch": 21, 2024-03-31T15:04:01.973 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_epoch_barrier": 0, 2024-03-31T15:04:01.973 INFO:teuthology.orchestra.run.smithi089.stdout: "blocklisted": false, 2024-03-31T15:04:01.973 INFO:teuthology.orchestra.run.smithi089.stdout: "fs_name": "cephfs" 2024-03-31T15:04:01.973 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-03-31T15:04:01.985 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:04:01.985 DEBUG:teuthology.orchestra.run.smithi016:> stat --file-system '--printf=%T 2024-03-31T15:04:01.985 DEBUG:teuthology.orchestra.run.smithi016:> ' -- /home/ubuntu/cephtest/mnt.0 2024-03-31T15:04:01.996 INFO:teuthology.orchestra.run.smithi016.stdout:fuseblk 2024-03-31T15:04:01.996 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-03-31T15:04:01.996 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:04:01.996 DEBUG:teuthology.orchestra.run.smithi016:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-31T15:04:02.064 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:04:02.064 DEBUG:teuthology.orchestra.run.smithi016:> stat --file-system '--printf=%T 2024-03-31T15:04:02.064 DEBUG:teuthology.orchestra.run.smithi016:> ' -- /home/ubuntu/cephtest/mnt.2 2024-03-31T15:04:02.117 INFO:teuthology.orchestra.run.smithi016.stdout:fuseblk 2024-03-31T15:04:02.118 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.2 2024-03-31T15:04:02.118 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:04:02.118 DEBUG:teuthology.orchestra.run.smithi016:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.2 2024-03-31T15:04:02.185 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:04:02.185 DEBUG:teuthology.orchestra.run.smithi089:> stat --file-system '--printf=%T 2024-03-31T15:04:02.185 DEBUG:teuthology.orchestra.run.smithi089:> ' -- /home/ubuntu/cephtest/mnt.1 2024-03-31T15:04:02.195 INFO:teuthology.orchestra.run.smithi089.stdout:fuseblk 2024-03-31T15:04:02.195 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2024-03-31T15:04:02.196 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:04:02.196 DEBUG:teuthology.orchestra.run.smithi089:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-03-31T15:04:02.259 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:04:02.259 DEBUG:teuthology.orchestra.run.smithi089:> stat --file-system '--printf=%T 2024-03-31T15:04:02.259 DEBUG:teuthology.orchestra.run.smithi089:> ' -- /home/ubuntu/cephtest/mnt.3 2024-03-31T15:04:02.311 INFO:teuthology.orchestra.run.smithi089.stdout:fuseblk 2024-03-31T15:04:02.311 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.3 2024-03-31T15:04:02.311 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:04:02.311 DEBUG:teuthology.orchestra.run.smithi089:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.3 2024-03-31T15:04:02.374 INFO:teuthology.run_tasks:Running task cephfs_test_runner... 2024-03-31T15:04:02.390 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-31T15:04:02.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:02.549+0000 7f6c54c00640 1 Processor -- start 2024-03-31T15:04:02.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:02.549+0000 7f6c54c00640 1 -- start start 2024-03-31T15:04:02.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:02.549+0000 7f6c54c00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f6c50145e40 0x7f6c500cc740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:02.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:02.549+0000 7f6c54c00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f6c50070030 0x7f6c500d2010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:02.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:02.549+0000 7f6c54c00640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6c500d0920 con 0x7f6c500ccc80 2024-03-31T15:04:02.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:02.549+0000 7f6c54c00640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f6c500d06d0 con 0x7f6c50070030 2024-03-31T15:04:02.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:02.549+0000 7f6c54c00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f6c500d0c00 con 0x7f6c50145e40 2024-03-31T15:04:02.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:02.549+0000 7f6c4de00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f6c50145e40 0x7f6c500cc740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:02.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:02.549+0000 7f6c45400640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f6c500ccc80 0x7f6c5006f920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:51200/0 (socket says 172.21.15.16:51200) 2024-03-31T15:04:02.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:02.549+0000 7f6c45400640 1 -- 172.21.15.16:0/197737862 learned_addr learned my addr 172.21.15.16:0/197737862 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:04:02.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:02.549+0000 7f6c4e800640 1 --2- 172.21.15.16:0/197737862 >> v2:172.21.15.16:3300/0 conn(0x7f6c50070030 0x7f6c500d2010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:02.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:02.549+0000 7f6c4de00640 1 -- 172.21.15.16:0/197737862 >> v2:172.21.15.16:3300/0 conn(0x7f6c50070030 msgr2=0x7f6c500d2010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:04:02.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:02.549+0000 7f6c4de00640 1 --2- 172.21.15.16:0/197737862 >> v2:172.21.15.16:3300/0 conn(0x7f6c50070030 0x7f6c500d2010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:02.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:02.549+0000 7f6c4de00640 1 -- 172.21.15.16:0/197737862 >> v1:172.21.15.16:6789/0 conn(0x7f6c500ccc80 legacy=0x7f6c5006f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:02.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:02.549+0000 7f6c4de00640 1 -- 172.21.15.16:0/197737862 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c500d1810 con 0x7f6c50145e40 2024-03-31T15:04:02.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:02.553+0000 7f6c4de00640 1 --2- 172.21.15.16:0/197737862 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f6c50145e40 0x7f6c500cc740 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f6c400097c0 tx=0x7f6c4001c4a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e8509e5b685a5962 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:02.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:02.553+0000 7f6c4d400640 1 -- 172.21.15.16:0/197737862 <== mon.2 v2:172.21.15.89:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c400b1070 con 0x7f6c50145e40 2024-03-31T15:04:02.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:02.553+0000 7f6c4d400640 1 -- 172.21.15.16:0/197737862 <== mon.2 v2:172.21.15.89:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6c400ac410 con 0x7f6c50145e40 2024-03-31T15:04:02.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:02.553+0000 7f6c4d400640 1 -- 172.21.15.16:0/197737862 <== mon.2 v2:172.21.15.89:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c400ac7e0 con 0x7f6c50145e40 2024-03-31T15:04:02.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:02.553+0000 7f6c54c00640 1 -- 172.21.15.16:0/197737862 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f6c50145e40 msgr2=0x7f6c500cc740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:02.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:02.553+0000 7f6c54c00640 1 --2- 172.21.15.16:0/197737862 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f6c50145e40 0x7f6c500cc740 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f6c400097c0 tx=0x7f6c4001c4a0 comp rx=0 tx=0).stop 2024-03-31T15:04:02.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:02.553+0000 7f6c54c00640 1 -- 172.21.15.16:0/197737862 shutdown_connections 2024-03-31T15:04:02.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:02.553+0000 7f6c54c00640 1 --2- 172.21.15.16:0/197737862 >> v2:172.21.15.16:3300/0 conn(0x7f6c50070030 0x7f6c500d2010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:02.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:02.553+0000 7f6c54c00640 1 --2- 172.21.15.16:0/197737862 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f6c50145e40 0x7f6c500cc740 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:02.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:02.553+0000 7f6c54c00640 1 -- 172.21.15.16:0/197737862 >> 172.21.15.16:0/197737862 conn(0x7f6c50067330 msgr2=0x7f6c50067740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:04:02.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:02.553+0000 7f6c54c00640 1 -- 172.21.15.16:0/197737862 shutdown_connections 2024-03-31T15:04:02.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:02.553+0000 7f6c54c00640 1 -- 172.21.15.16:0/197737862 wait complete. 2024-03-31T15:04:02.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:02.557+0000 7f6c54c00640 1 Processor -- start 2024-03-31T15:04:02.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:02.557+0000 7f6c54c00640 1 -- start start 2024-03-31T15:04:02.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:02.557+0000 7f6c54c00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f6c50070030 0x7f6c501519e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:02.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:02.557+0000 7f6c54c00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6c500ccc80 0x7f6c50152180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:02.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:02.557+0000 7f6c54c00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6c50145e40 0x7f6c50159cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:02.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:02.557+0000 7f6c54c00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f6c500b5240 con 0x7f6c500ccc80 2024-03-31T15:04:02.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:02.557+0000 7f6c54c00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f6c500b50c0 con 0x7f6c50145e40 2024-03-31T15:04:02.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:02.557+0000 7f6c54c00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f6c50169190 con 0x7f6c50070030 2024-03-31T15:04:02.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:02.557+0000 7f6c45400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6c500ccc80 0x7f6c50152180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:02.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:02.557+0000 7f6c4de00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f6c50070030 0x7f6c501519e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:02.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:02.557+0000 7f6c45400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6c500ccc80 0x7f6c50152180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60180/0 (socket says 172.21.15.16:60180) 2024-03-31T15:04:02.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:02.557+0000 7f6c45400640 1 -- 172.21.15.16:0/3039669425 learned_addr learned my addr 172.21.15.16:0/3039669425 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:04:02.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:02.557+0000 7f6c4e800640 1 --2- 172.21.15.16:0/3039669425 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6c50145e40 0x7f6c50159cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:02.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:02.557+0000 7f6c45400640 1 -- 172.21.15.16:0/3039669425 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f6c50070030 msgr2=0x7f6c501519e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:02.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:02.557+0000 7f6c45400640 1 --2- 172.21.15.16:0/3039669425 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f6c50070030 0x7f6c501519e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:02.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:02.557+0000 7f6c45400640 1 -- 172.21.15.16:0/3039669425 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6c50145e40 msgr2=0x7f6c50159cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:02.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:02.557+0000 7f6c45400640 1 --2- 172.21.15.16:0/3039669425 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6c50145e40 0x7f6c50159cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:02.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:02.557+0000 7f6c45400640 1 -- 172.21.15.16:0/3039669425 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c500b5630 con 0x7f6c500ccc80 2024-03-31T15:04:02.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:02.557+0000 7f6c45400640 1 --2- 172.21.15.16:0/3039669425 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6c500ccc80 0x7f6c50152180 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f6c3800cb10 tx=0x7f6c3800a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:02.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:02.557+0000 7f6c46a00640 1 -- 172.21.15.16:0/3039669425 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c38017070 con 0x7f6c500ccc80 2024-03-31T15:04:02.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:02.557+0000 7f6c46a00640 1 -- 172.21.15.16:0/3039669425 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6c3800ac70 con 0x7f6c500ccc80 2024-03-31T15:04:02.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:02.557+0000 7f6c46a00640 1 -- 172.21.15.16:0/3039669425 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c38012620 con 0x7f6c500ccc80 2024-03-31T15:04:02.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:02.557+0000 7f6c54c00640 1 -- 172.21.15.16:0/3039669425 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6c500b5910 con 0x7f6c500ccc80 2024-03-31T15:04:02.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:02.557+0000 7f6c54c00640 1 -- 172.21.15.16:0/3039669425 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6c500b5ed0 con 0x7f6c500ccc80 2024-03-31T15:04:02.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:02.561+0000 7f6c54c00640 1 -- 172.21.15.16:0/3039669425 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6c50069950 con 0x7f6c500ccc80 2024-03-31T15:04:02.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:02.561+0000 7f6c46a00640 1 -- 172.21.15.16:0/3039669425 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f6c3801a070 con 0x7f6c500ccc80 2024-03-31T15:04:02.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:02.561+0000 7f6c46a00640 1 --2- 172.21.15.16:0/3039669425 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f6c2c068380 0x7f6c2c06a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:02.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:02.561+0000 7f6c46a00640 1 -- 172.21.15.16:0/3039669425 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f6c38086770 con 0x7f6c500ccc80 2024-03-31T15:04:02.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:02.561+0000 7f6c4de00640 1 --2- 172.21.15.16:0/3039669425 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f6c2c068380 0x7f6c2c06a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:02.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:02.561+0000 7f6c4de00640 1 --2- 172.21.15.16:0/3039669425 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f6c2c068380 0x7f6c2c06a840 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f6c400ae000 tx=0x7f6c40008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:02.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:02.565+0000 7f6c46a00640 1 -- 172.21.15.16:0/3039669425 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f6c38056640 con 0x7f6c500ccc80 2024-03-31T15:04:02.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:02.733+0000 7f6c54c00640 1 -- 172.21.15.16:0/3039669425 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f6c500cf3c0 con 0x7f6c500ccc80 2024-03-31T15:04:02.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:02.733+0000 7f6c46a00640 1 -- 172.21.15.16:0/3039669425 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) ==== 74+0+16533 (secure 0 0 0) 0x7f6c3805a940 con 0x7f6c500ccc80 2024-03-31T15:04:02.727 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:04:02.728 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":21,"fsid":"286eb5da-fb47-4550-b290-d9519c13b9a5","created":"2024-03-31T15:03:11.438952+0000","modified":"2024-03-31T15:03:37.957829+0000","last_up_change":"2024-03-31T15:03:22.885535+0000","last_in_change":"2024-03-31T15:03:13.941667+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-31T15:03:22.550787+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-31T15:03:34.488425+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-31T15:03:35.299116+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"92a66450-ef19-40d8-ac3d-d760b23b0bda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6801","nonce":932712244}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6803","nonce":932712244}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6807","nonce":932712244}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6805","nonce":932712244}]},"public_addr":"172.21.15.16:6801/932712244","cluster_addr":"172.21.15.16:6803/932712244","heartbeat_back_addr":"172.21.15.16:6807/932712244","heartbeat_front_addr":"172.21.15.16:6805/932712244","state":["exists","up"]},{"osd":1,"uuid":"da9d1f2b-1968-4959-b843-faa231988e92","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6810","nonce":1398737533}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6811","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6813","nonce":1398737533}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6821","nonce":1398737533}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6815","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6816","nonce":1398737533}]},"public_addr":"172.21.15.16:6810/1398737533","cluster_addr":"172.21.15.16:6813/1398737533","heartbeat_back_addr":"172.21.15.16:6821/1398737533","heartbeat_front_addr":"172.21.15.16:6816/1398737533","state":["exists","up"]},{"osd":2,"uuid":"01a73662-9a21-4b1f-bcbb-42b9636a4638","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6819","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6822","nonce":3018526646}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6825","nonce":3018526646}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6831","nonce":3018526646}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6827","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6829","nonce":3018526646}]},"public_addr":"172.21.15.16:6822/3018526646","cluster_addr":"172.21.15.16:6825/3018526646","heartbeat_back_addr":"172.21.15.16:6831/3018526646","heartbeat_front_addr":"172.21.15.16:6829/3018526646","state":["exists","up"]},{"osd":3,"uuid":"c7771fd3-e18e-4500-ac38-23cf10ebcd75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6809","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6812","nonce":3360352974}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6817","nonce":3360352974}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6828","nonce":3360352974}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6823","nonce":3360352974}]},"public_addr":"172.21.15.16:6812/3360352974","cluster_addr":"172.21.15.16:6817/3360352974","heartbeat_back_addr":"172.21.15.16:6828/3360352974","heartbeat_front_addr":"172.21.15.16:6823/3360352974","state":["exists","up"]},{"osd":4,"uuid":"88260730-1482-4fbd-a40f-cb1124e9df4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6801","nonce":1856844856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6803","nonce":1856844856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6807","nonce":1856844856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6805","nonce":1856844856}]},"public_addr":"172.21.15.89:6801/1856844856","cluster_addr":"172.21.15.89:6803/1856844856","heartbeat_back_addr":"172.21.15.89:6807/1856844856","heartbeat_front_addr":"172.21.15.89:6805/1856844856","state":["exists","up"]},{"osd":5,"uuid":"4cb2c00c-fae0-4dd8-9d27-ec4f0093ccf7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6814","nonce":50395527}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6817","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6820","nonce":50395527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6831","nonce":50395527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6823","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6826","nonce":50395527}]},"public_addr":"172.21.15.89:6814/50395527","cluster_addr":"172.21.15.89:6820/50395527","heartbeat_back_addr":"172.21.15.89:6831/50395527","heartbeat_front_addr":"172.21.15.89:6826/50395527","state":["exists","up"]},{"osd":6,"uuid":"81ce0c19-2554-4cae-acbe-66a05e9e9b9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6809","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6812","nonce":3470471752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6815","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6818","nonce":3470471752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6827","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6829","nonce":3470471752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6821","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6824","nonce":3470471752}]},"public_addr":"172.21.15.89:6812/3470471752","cluster_addr":"172.21.15.89:6818/3470471752","heartbeat_back_addr":"172.21.15.89:6829/3470471752","heartbeat_front_addr":"172.21.15.89:6824/3470471752","state":["exists","up"]},{"osd":7,"uuid":"85e4313b-d37d-442e-8042-153bf20a847d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6811","nonce":1796778630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6813","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6816","nonce":1796778630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6825","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6828","nonce":1796778630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6819","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6822","nonce":1796778630}]},"public_addr":"172.21.15.89:6811/1796778630","cluster_addr":"172.21.15.89:6816/1796778630","heartbeat_back_addr":"172.21.15.89:6828/1796778630","heartbeat_front_addr":"172.21.15.89:6822/1796778630","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.439088+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.529970+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.680512+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:21.044161+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.582111+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.536558+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.409522+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.590229+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-31T15:04:02.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:02.737+0000 7f6c54c00640 1 -- 172.21.15.16:0/3039669425 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f6c2c068380 msgr2=0x7f6c2c06a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:02.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:02.737+0000 7f6c54c00640 1 --2- 172.21.15.16:0/3039669425 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f6c2c068380 0x7f6c2c06a840 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f6c400ae000 tx=0x7f6c40008040 comp rx=0 tx=0).stop 2024-03-31T15:04:02.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:02.737+0000 7f6c54c00640 1 -- 172.21.15.16:0/3039669425 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6c500ccc80 msgr2=0x7f6c50152180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:02.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:02.737+0000 7f6c54c00640 1 --2- 172.21.15.16:0/3039669425 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6c500ccc80 0x7f6c50152180 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f6c3800cb10 tx=0x7f6c3800a060 comp rx=0 tx=0).stop 2024-03-31T15:04:02.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:02.737+0000 7f6c54c00640 1 -- 172.21.15.16:0/3039669425 shutdown_connections 2024-03-31T15:04:02.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:02.737+0000 7f6c54c00640 1 --2- 172.21.15.16:0/3039669425 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f6c2c068380 0x7f6c2c06a840 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:02.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:02.737+0000 7f6c54c00640 1 --2- 172.21.15.16:0/3039669425 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6c50145e40 0x7f6c50159cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:02.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:02.737+0000 7f6c54c00640 1 --2- 172.21.15.16:0/3039669425 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6c500ccc80 0x7f6c50152180 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:02.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:02.737+0000 7f6c54c00640 1 --2- 172.21.15.16:0/3039669425 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f6c50070030 0x7f6c501519e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:02.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:02.737+0000 7f6c54c00640 1 -- 172.21.15.16:0/3039669425 >> 172.21.15.16:0/3039669425 conn(0x7f6c50067330 msgr2=0x7f6c5015fd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:04:02.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:02.737+0000 7f6c54c00640 1 -- 172.21.15.16:0/3039669425 shutdown_connections 2024-03-31T15:04:02.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:02.737+0000 7f6c54c00640 1 -- 172.21.15.16:0/3039669425 wait complete. 2024-03-31T15:04:02.749 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-31T15:03:22.550787+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-31T15:03:34.488425+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-31T15:03:35.299116+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-31T15:04:02.750 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-31T15:04:02.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:02.897+0000 7f999c400640 1 Processor -- start 2024-03-31T15:04:02.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:02.897+0000 7f999c400640 1 -- start start 2024-03-31T15:04:02.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:02.901+0000 7f999c400640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f9994141590 0x7f9994144340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:02.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:02.901+0000 7f999c400640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9994144980 con 0x7f9994070030 2024-03-31T15:04:02.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:02.901+0000 7f999c400640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9994161fe0 con 0x7f99941411b0 2024-03-31T15:04:02.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:02.901+0000 7f999c400640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f999415f910 con 0x7f9994141590 2024-03-31T15:04:02.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:02.901+0000 7f9999000640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f9994070030 0x7f9994143c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:51210/0 (socket says 172.21.15.16:51210) 2024-03-31T15:04:02.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:02.901+0000 7f9999000640 1 -- 172.21.15.16:0/600495644 learned_addr learned my addr 172.21.15.16:0/600495644 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:04:02.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:02.901+0000 7f999a400640 1 --2- 172.21.15.16:0/600495644 >> v2:172.21.15.89:3300/0 conn(0x7f9994141590 0x7f9994144340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:02.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:02.901+0000 7f9993e00640 1 -- 172.21.15.16:0/600495644 <== mon.1 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2230614875 0 0) 0x7f9994161fe0 con 0x7f99941411b0 2024-03-31T15:04:02.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:02.901+0000 7f9993e00640 1 -- 172.21.15.16:0/600495644 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f997c003530 con 0x7f99941411b0 2024-03-31T15:04:02.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:02.901+0000 7f999a400640 1 -- 172.21.15.16:0/600495644 >> v1:172.21.15.89:6789/0 conn(0x7f99941411b0 legacy=0x7f999406f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:02.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:02.901+0000 7f9993e00640 1 -- 172.21.15.16:0/600495644 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3490447523 0 0) 0x7f9994144980 con 0x7f9994070030 2024-03-31T15:04:02.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:02.901+0000 7f999a400640 1 -- 172.21.15.16:0/600495644 >> v1:172.21.15.16:6789/0 conn(0x7f9994070030 legacy=0x7f9994143c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:02.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:02.901+0000 7f999a400640 1 -- 172.21.15.16:0/600495644 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99941622c0 con 0x7f9994141590 2024-03-31T15:04:02.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:02.905+0000 7f999a400640 1 --2- 172.21.15.16:0/600495644 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9994141590 0x7f9994144340 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f998c009d90 tx=0x7f998c01c970 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5d76384182f72de6 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:02.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:02.905+0000 7f9993e00640 1 -- 172.21.15.16:0/600495644 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f998c0b2070 con 0x7f9994141590 2024-03-31T15:04:02.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:02.905+0000 7f9993e00640 1 -- 172.21.15.16:0/600495644 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f998c0a5940 con 0x7f9994141590 2024-03-31T15:04:02.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:02.905+0000 7f9993e00640 1 -- 172.21.15.16:0/600495644 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f998c0a5d10 con 0x7f9994141590 2024-03-31T15:04:02.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:02.905+0000 7f999c400640 1 -- 172.21.15.16:0/600495644 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9994141590 msgr2=0x7f9994144340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:02.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:02.905+0000 7f999c400640 1 --2- 172.21.15.16:0/600495644 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9994141590 0x7f9994144340 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f998c009d90 tx=0x7f998c01c970 comp rx=0 tx=0).stop 2024-03-31T15:04:02.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:02.905+0000 7f999c400640 1 -- 172.21.15.16:0/600495644 shutdown_connections 2024-03-31T15:04:02.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:02.905+0000 7f999c400640 1 --2- 172.21.15.16:0/600495644 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9994141590 0x7f9994144340 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:02.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:02.905+0000 7f999c400640 1 -- 172.21.15.16:0/600495644 >> 172.21.15.16:0/600495644 conn(0x7f9994067930 msgr2=0x7f99940b3ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:04:02.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:02.905+0000 7f999c400640 1 -- 172.21.15.16:0/600495644 shutdown_connections 2024-03-31T15:04:02.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:02.905+0000 7f999c400640 1 -- 172.21.15.16:0/600495644 wait complete. 2024-03-31T15:04:02.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:02.905+0000 7f999c400640 1 Processor -- start 2024-03-31T15:04:02.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:02.905+0000 7f999c400640 1 -- start start 2024-03-31T15:04:02.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:02.909+0000 7f999c400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9994070030 0x7f9994143970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:02.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:02.909+0000 7f999c400640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f99941411b0 0x7f9994142560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:02.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:02.909+0000 7f999c400640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9994141590 0x7f999413c850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:02.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:02.909+0000 7f999c400640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f999416be80 con 0x7f9994070030 2024-03-31T15:04:02.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:02.909+0000 7f999c400640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f999416bd00 con 0x7f9994141590 2024-03-31T15:04:02.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:02.909+0000 7f999c400640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f999416c000 con 0x7f99941411b0 2024-03-31T15:04:02.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:02.909+0000 7f9999a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9994070030 0x7f9994143970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:02.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:02.909+0000 7f9999a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9994070030 0x7f9994143970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60182/0 (socket says 172.21.15.16:60182) 2024-03-31T15:04:02.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:02.909+0000 7f9999a00640 1 -- 172.21.15.16:0/2255122687 learned_addr learned my addr 172.21.15.16:0/2255122687 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:04:02.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:02.909+0000 7f999a400640 1 --2- 172.21.15.16:0/2255122687 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9994141590 0x7f999413c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:02.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:02.909+0000 7f9999000640 1 --2- 172.21.15.16:0/2255122687 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f99941411b0 0x7f9994142560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:02.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:02.909+0000 7f9999a00640 1 -- 172.21.15.16:0/2255122687 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f99941411b0 msgr2=0x7f9994142560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:02.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:02.909+0000 7f9999a00640 1 --2- 172.21.15.16:0/2255122687 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f99941411b0 0x7f9994142560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:02.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:02.909+0000 7f9999a00640 1 -- 172.21.15.16:0/2255122687 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9994141590 msgr2=0x7f999413c850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:02.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:02.909+0000 7f9999a00640 1 --2- 172.21.15.16:0/2255122687 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9994141590 0x7f999413c850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:02.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:02.909+0000 7f9999a00640 1 -- 172.21.15.16:0/2255122687 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f999413d610 con 0x7f9994070030 2024-03-31T15:04:02.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:02.909+0000 7f9999a00640 1 --2- 172.21.15.16:0/2255122687 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9994070030 0x7f9994143970 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f998800c530 tx=0x7f998800c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:02.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:02.909+0000 7f9991600640 1 -- 172.21.15.16:0/2255122687 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99880052d0 con 0x7f9994070030 2024-03-31T15:04:02.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:02.909+0000 7f9991600640 1 -- 172.21.15.16:0/2255122687 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9988005490 con 0x7f9994070030 2024-03-31T15:04:02.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:02.909+0000 7f9991600640 1 -- 172.21.15.16:0/2255122687 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9988012850 con 0x7f9994070030 2024-03-31T15:04:02.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:02.909+0000 7f999c400640 1 -- 172.21.15.16:0/2255122687 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9994142c60 con 0x7f9994070030 2024-03-31T15:04:02.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:02.909+0000 7f999c400640 1 -- 172.21.15.16:0/2255122687 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f999412b5d0 con 0x7f9994070030 2024-03-31T15:04:02.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:02.909+0000 7f999c400640 1 -- 172.21.15.16:0/2255122687 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9994069950 con 0x7f9994070030 2024-03-31T15:04:02.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:02.909+0000 7f9991600640 1 -- 172.21.15.16:0/2255122687 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f99880129f0 con 0x7f9994070030 2024-03-31T15:04:02.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:02.913+0000 7f9991600640 1 --2- 172.21.15.16:0/2255122687 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f9974068380 0x7f997406a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:02.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:02.913+0000 7f9991600640 1 -- 172.21.15.16:0/2255122687 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f9988086550 con 0x7f9994070030 2024-03-31T15:04:02.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:02.913+0000 7f9999000640 1 --2- 172.21.15.16:0/2255122687 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f9974068380 0x7f997406a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:02.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:02.913+0000 7f9999000640 1 --2- 172.21.15.16:0/2255122687 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f9974068380 0x7f997406a840 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f999406c810 tx=0x7f9980007d60 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:02.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:02.917+0000 7f9991600640 1 -- 172.21.15.16:0/2255122687 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f999413d610 con 0x7f9994070030 2024-03-31T15:04:03.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:03.081+0000 7f999c400640 1 -- 172.21.15.16:0/2255122687 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f999415e490 con 0x7f9994070030 2024-03-31T15:04:03.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:03.081+0000 7f9991600640 1 -- 172.21.15.16:0/2255122687 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) ==== 93+0+10 (secure 0 0 0) 0x7f9988056430 con 0x7f9994070030 2024-03-31T15:04:03.074 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2024-03-31T15:04:03.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:03.081+0000 7f999c400640 1 -- 172.21.15.16:0/2255122687 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f9974068380 msgr2=0x7f997406a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:03.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:03.081+0000 7f999c400640 1 --2- 172.21.15.16:0/2255122687 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f9974068380 0x7f997406a840 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f999406c810 tx=0x7f9980007d60 comp rx=0 tx=0).stop 2024-03-31T15:04:03.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:03.081+0000 7f999c400640 1 -- 172.21.15.16:0/2255122687 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9994070030 msgr2=0x7f9994143970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:03.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:03.081+0000 7f999c400640 1 --2- 172.21.15.16:0/2255122687 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9994070030 0x7f9994143970 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f998800c530 tx=0x7f998800c9f0 comp rx=0 tx=0).stop 2024-03-31T15:04:03.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:03.081+0000 7f999c400640 1 -- 172.21.15.16:0/2255122687 shutdown_connections 2024-03-31T15:04:03.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:03.081+0000 7f999c400640 1 --2- 172.21.15.16:0/2255122687 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f9974068380 0x7f997406a840 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:03.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:03.081+0000 7f999c400640 1 --2- 172.21.15.16:0/2255122687 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9994141590 0x7f999413c850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:03.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:03.081+0000 7f999c400640 1 --2- 172.21.15.16:0/2255122687 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f99941411b0 0x7f9994142560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:03.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:03.081+0000 7f999c400640 1 --2- 172.21.15.16:0/2255122687 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9994070030 0x7f9994143970 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:03.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:03.081+0000 7f999c400640 1 -- 172.21.15.16:0/2255122687 >> 172.21.15.16:0/2255122687 conn(0x7f9994067930 msgr2=0x7f99940b4d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:04:03.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:03.085+0000 7f999c400640 1 -- 172.21.15.16:0/2255122687 shutdown_connections 2024-03-31T15:04:03.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:03.085+0000 7f999c400640 1 -- 172.21.15.16:0/2255122687 wait complete. 2024-03-31T15:04:03.096 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-31T15:04:03.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:03.245+0000 7f4a0fe00640 1 Processor -- start 2024-03-31T15:04:03.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:03.245+0000 7f4a0fe00640 1 -- start start 2024-03-31T15:04:03.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:03.245+0000 7f4a0fe00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f4a100d2b40 0x7f4a100d0270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:03.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:03.245+0000 7f4a0fe00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f4a1006fbc0 0x7f4a10070050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:03.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:03.245+0000 7f4a0fe00640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4a100cd750 con 0x7f4a1015c390 2024-03-31T15:04:03.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:03.245+0000 7f4a0fe00640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f4a1015c770 con 0x7f4a1006fbc0 2024-03-31T15:04:03.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:03.245+0000 7f4a0fe00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f4a100cda30 con 0x7f4a100d2b40 2024-03-31T15:04:03.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:03.245+0000 7f4a06000640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f4a100d2b40 0x7f4a100d0270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:03.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:03.245+0000 7f4a0ea00640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f4a1015c390 0x7f4a100cfb60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.16:38292/0 (socket says 172.21.15.16:38292) 2024-03-31T15:04:03.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:03.245+0000 7f4a0ea00640 1 -- 172.21.15.16:0/80027449 learned_addr learned my addr 172.21.15.16:0/80027449 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:04:03.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:03.245+0000 7f4a0f400640 1 --2- 172.21.15.16:0/80027449 >> v2:172.21.15.89:3300/0 conn(0x7f4a1006fbc0 0x7f4a10070050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:03.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:03.249+0000 7f4a06000640 1 -- 172.21.15.16:0/80027449 >> v2:172.21.15.89:3300/0 conn(0x7f4a1006fbc0 msgr2=0x7f4a10070050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:04:03.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:03.249+0000 7f4a06000640 1 --2- 172.21.15.16:0/80027449 >> v2:172.21.15.89:3300/0 conn(0x7f4a1006fbc0 0x7f4a10070050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:03.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:03.249+0000 7f4a0e000640 1 -- 172.21.15.16:0/80027449 <== mon.1 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3062990605 0 0) 0x7f4a100cd750 con 0x7f4a1015c390 2024-03-31T15:04:03.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:03.249+0000 7f4a06000640 1 -- 172.21.15.16:0/80027449 >> v1:172.21.15.89:6789/0 conn(0x7f4a1015c390 legacy=0x7f4a100cfb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:03.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:03.249+0000 7f4a06000640 1 -- 172.21.15.16:0/80027449 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a100cdd10 con 0x7f4a100d2b40 2024-03-31T15:04:03.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:03.249+0000 7f4a06000640 1 --2- 172.21.15.16:0/80027449 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f4a100d2b40 0x7f4a100d0270 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f49f80097c0 tx=0x7f49f801c4a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d4f0c3c2c1c8ca48 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:03.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:03.249+0000 7f4a0e000640 1 -- 172.21.15.16:0/80027449 <== mon.2 v2:172.21.15.89:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49f80b0070 con 0x7f4a100d2b40 2024-03-31T15:04:03.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:03.249+0000 7f4a0e000640 1 -- 172.21.15.16:0/80027449 <== mon.2 v2:172.21.15.89:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f49f80ab600 con 0x7f4a100d2b40 2024-03-31T15:04:03.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:03.249+0000 7f4a0e000640 1 -- 172.21.15.16:0/80027449 <== mon.2 v2:172.21.15.89:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49f80ab9d0 con 0x7f4a100d2b40 2024-03-31T15:04:03.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:03.253+0000 7f4a0fe00640 1 -- 172.21.15.16:0/80027449 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f4a100d2b40 msgr2=0x7f4a100d0270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:03.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:03.253+0000 7f4a0fe00640 1 --2- 172.21.15.16:0/80027449 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f4a100d2b40 0x7f4a100d0270 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f49f80097c0 tx=0x7f49f801c4a0 comp rx=0 tx=0).stop 2024-03-31T15:04:03.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:03.253+0000 7f4a0fe00640 1 -- 172.21.15.16:0/80027449 shutdown_connections 2024-03-31T15:04:03.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:03.253+0000 7f4a0fe00640 1 --2- 172.21.15.16:0/80027449 >> v2:172.21.15.89:3300/0 conn(0x7f4a1006fbc0 0x7f4a10070050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:03.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:03.253+0000 7f4a0fe00640 1 --2- 172.21.15.16:0/80027449 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f4a100d2b40 0x7f4a100d0270 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:03.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:03.253+0000 7f4a0fe00640 1 -- 172.21.15.16:0/80027449 >> 172.21.15.16:0/80027449 conn(0x7f4a100d1f90 msgr2=0x7f4a100d23a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:04:03.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:03.253+0000 7f4a0fe00640 1 -- 172.21.15.16:0/80027449 shutdown_connections 2024-03-31T15:04:03.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:03.253+0000 7f4a0fe00640 1 -- 172.21.15.16:0/80027449 wait complete. 2024-03-31T15:04:03.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:03.253+0000 7f4a0fe00640 1 Processor -- start 2024-03-31T15:04:03.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:03.253+0000 7f4a0fe00640 1 -- start start 2024-03-31T15:04:03.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:03.253+0000 7f4a0fe00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4a1006fbc0 0x7f4a1006d140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:03.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:03.253+0000 7f4a0fe00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4a100d2b40 0x7f4a1015b710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:03.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:03.253+0000 7f4a0fe00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f4a1015c390 0x7f4a10154520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:03.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:03.253+0000 7f4a0fe00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f4a10168e50 con 0x7f4a100d2b40 2024-03-31T15:04:03.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:03.253+0000 7f4a0fe00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f4a100d09c0 con 0x7f4a1006fbc0 2024-03-31T15:04:03.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:03.253+0000 7f4a0fe00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f4a10168fd0 con 0x7f4a1015c390 2024-03-31T15:04:03.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:03.253+0000 7f4a06000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4a100d2b40 0x7f4a1015b710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:03.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:03.253+0000 7f4a06000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4a100d2b40 0x7f4a1015b710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60190/0 (socket says 172.21.15.16:60190) 2024-03-31T15:04:03.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:03.253+0000 7f4a06000640 1 -- 172.21.15.16:0/1353958955 learned_addr learned my addr 172.21.15.16:0/1353958955 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:04:03.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:03.253+0000 7f4a0ea00640 1 --2- 172.21.15.16:0/1353958955 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4a1006fbc0 0x7f4a1006d140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:03.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:03.253+0000 7f4a0f400640 1 --2- 172.21.15.16:0/1353958955 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f4a1015c390 0x7f4a10154520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:03.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:03.253+0000 7f4a06000640 1 -- 172.21.15.16:0/1353958955 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f4a1015c390 msgr2=0x7f4a10154520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:03.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:03.253+0000 7f4a06000640 1 --2- 172.21.15.16:0/1353958955 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f4a1015c390 0x7f4a10154520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:03.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:03.253+0000 7f4a06000640 1 -- 172.21.15.16:0/1353958955 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4a1006fbc0 msgr2=0x7f4a1006d140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:03.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:03.253+0000 7f4a06000640 1 --2- 172.21.15.16:0/1353958955 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4a1006fbc0 0x7f4a1006d140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:03.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:03.253+0000 7f4a06000640 1 -- 172.21.15.16:0/1353958955 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a1006d840 con 0x7f4a100d2b40 2024-03-31T15:04:03.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:03.253+0000 7f4a06000640 1 --2- 172.21.15.16:0/1353958955 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4a100d2b40 0x7f4a1015b710 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f49f801a0d0 tx=0x7f49f801ace0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:03.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:03.257+0000 7f4a07400640 1 -- 172.21.15.16:0/1353958955 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49f80b0050 con 0x7f4a100d2b40 2024-03-31T15:04:03.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:03.257+0000 7f4a07400640 1 -- 172.21.15.16:0/1353958955 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f49f80bc070 con 0x7f4a100d2b40 2024-03-31T15:04:03.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:03.257+0000 7f4a07400640 1 -- 172.21.15.16:0/1353958955 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49f80b5830 con 0x7f4a100d2b40 2024-03-31T15:04:03.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:03.257+0000 7f4a0fe00640 1 -- 172.21.15.16:0/1353958955 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4a1006db20 con 0x7f4a100d2b40 2024-03-31T15:04:03.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:03.257+0000 7f4a0fe00640 1 -- 172.21.15.16:0/1353958955 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4a1006e0e0 con 0x7f4a100d2b40 2024-03-31T15:04:03.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:03.257+0000 7f4a0fe00640 1 -- 172.21.15.16:0/1353958955 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4a1004f8c0 con 0x7f4a100d2b40 2024-03-31T15:04:03.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:03.257+0000 7f4a07400640 1 -- 172.21.15.16:0/1353958955 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f4a10070da0 con 0x7f4a100d2b40 2024-03-31T15:04:03.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:03.257+0000 7f4a07400640 1 --2- 172.21.15.16:0/1353958955 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f49ec0683b0 0x7f49ec06a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:03.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:03.257+0000 7f4a07400640 1 -- 172.21.15.16:0/1353958955 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f49f8120d80 con 0x7f4a100d2b40 2024-03-31T15:04:03.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:03.257+0000 7f4a0ea00640 1 --2- 172.21.15.16:0/1353958955 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f49ec0683b0 0x7f49ec06a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:03.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:03.257+0000 7f4a0ea00640 1 --2- 172.21.15.16:0/1353958955 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f49ec0683b0 0x7f49ec06a870 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f4a00000f70 tx=0x7f4a00007240 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:03.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:03.261+0000 7f4a07400640 1 -- 172.21.15.16:0/1353958955 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f49f8016090 con 0x7f4a100d2b40 2024-03-31T15:04:03.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:03.429+0000 7f4a0fe00640 1 -- 172.21.15.16:0/1353958955 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f4a100cf410 con 0x7f4a100d2b40 2024-03-31T15:04:03.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:03.429+0000 7f4a07400640 1 -- 172.21.15.16:0/1353958955 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) ==== 104+0+11 (secure 0 0 0) 0x7f49f80f0c50 con 0x7f4a100d2b40 2024-03-31T15:04:03.422 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2024-03-31T15:04:03.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:03.429+0000 7f4a0fe00640 1 -- 172.21.15.16:0/1353958955 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f49ec0683b0 msgr2=0x7f49ec06a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:03.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:03.429+0000 7f4a0fe00640 1 --2- 172.21.15.16:0/1353958955 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f49ec0683b0 0x7f49ec06a870 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f4a00000f70 tx=0x7f4a00007240 comp rx=0 tx=0).stop 2024-03-31T15:04:03.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:03.429+0000 7f4a0fe00640 1 -- 172.21.15.16:0/1353958955 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4a100d2b40 msgr2=0x7f4a1015b710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:03.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:03.429+0000 7f4a0fe00640 1 --2- 172.21.15.16:0/1353958955 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4a100d2b40 0x7f4a1015b710 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f49f801a0d0 tx=0x7f49f801ace0 comp rx=0 tx=0).stop 2024-03-31T15:04:03.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:03.429+0000 7f4a0fe00640 1 -- 172.21.15.16:0/1353958955 shutdown_connections 2024-03-31T15:04:03.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:03.429+0000 7f4a0fe00640 1 --2- 172.21.15.16:0/1353958955 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f49ec0683b0 0x7f49ec06a870 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:03.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:03.429+0000 7f4a0fe00640 1 --2- 172.21.15.16:0/1353958955 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f4a1015c390 0x7f4a10154520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:03.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:03.429+0000 7f4a0fe00640 1 --2- 172.21.15.16:0/1353958955 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4a100d2b40 0x7f4a1015b710 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:03.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:03.429+0000 7f4a0fe00640 1 --2- 172.21.15.16:0/1353958955 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4a1006fbc0 0x7f4a1006d140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:03.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:03.429+0000 7f4a0fe00640 1 -- 172.21.15.16:0/1353958955 >> 172.21.15.16:0/1353958955 conn(0x7f4a100d1f90 msgr2=0x7f4a101556d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:04:03.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:03.429+0000 7f4a0fe00640 1 -- 172.21.15.16:0/1353958955 shutdown_connections 2024-03-31T15:04:03.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:03.429+0000 7f4a0fe00640 1 -- 172.21.15.16:0/1353958955 wait complete. 2024-03-31T15:04:03.443 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-31T15:04:03.586 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:03.593+0000 7f9e18a00640 1 Processor -- start 2024-03-31T15:04:03.586 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:03.593+0000 7f9e18a00640 1 -- start start 2024-03-31T15:04:03.587 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:03.593+0000 7f9e18a00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f9e1413e3d0 0x7f9e140cc9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:03.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:03.593+0000 7f9e18a00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f9e1406dbe0 0x7f9e140ccf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:03.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:03.593+0000 7f9e18a00640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9e1415fbc0 con 0x7f9e1413dff0 2024-03-31T15:04:03.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:03.593+0000 7f9e18a00640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f9e140cd450 con 0x7f9e1406dbe0 2024-03-31T15:04:03.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:03.593+0000 7f9e18a00640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f9e1415f730 con 0x7f9e1413e3d0 2024-03-31T15:04:03.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:03.593+0000 7f9e12800640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f9e1406dbe0 0x7f9e140ccf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:03.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:03.593+0000 7f9e09400640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f9e1413e3d0 0x7f9e140cc9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:03.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:03.593+0000 7f9e12800640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f9e1406dbe0 0x7f9e140ccf10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60200/0 (socket says 172.21.15.16:60200) 2024-03-31T15:04:03.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:03.593+0000 7f9e12800640 1 -- 172.21.15.16:0/2636425794 learned_addr learned my addr 172.21.15.16:0/2636425794 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:04:03.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:03.597+0000 7f9e12800640 1 -- 172.21.15.16:0/2636425794 >> v2:172.21.15.89:3300/0 conn(0x7f9e1413e3d0 msgr2=0x7f9e140cc9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:04:03.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:03.597+0000 7f9e12800640 1 --2- 172.21.15.16:0/2636425794 >> v2:172.21.15.89:3300/0 conn(0x7f9e1413e3d0 0x7f9e140cc9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:03.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:03.597+0000 7f9e12800640 1 -- 172.21.15.16:0/2636425794 >> v1:172.21.15.89:6789/0 conn(0x7f9e1413dff0 legacy=0x7f9e1406d4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:03.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:03.597+0000 7f9e12800640 1 -- 172.21.15.16:0/2636425794 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e1415ee00 con 0x7f9e1406dbe0 2024-03-31T15:04:03.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:03.597+0000 7f9e11400640 1 -- 172.21.15.16:0/2636425794 <== mon.1 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 486967639 0 0) 0x7f9e1415fbc0 con 0x7f9e1413dff0 2024-03-31T15:04:03.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:03.597+0000 7f9e09400640 1 --2- 172.21.15.16:0/2636425794 >> v2:172.21.15.89:3300/0 conn(0x7f9e1413e3d0 0x7f9e140cc9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:04:03.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:03.597+0000 7f9e12800640 1 --2- 172.21.15.16:0/2636425794 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9e1406dbe0 0x7f9e140ccf10 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f9e1406f620 tx=0x7f9e0c01c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cbdd06dd886ab621 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:03.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:03.597+0000 7f9e11400640 1 -- 172.21.15.16:0/2636425794 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e0c0a5710 con 0x7f9e1406dbe0 2024-03-31T15:04:03.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:03.597+0000 7f9e11400640 1 -- 172.21.15.16:0/2636425794 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9e0c0a58b0 con 0x7f9e1406dbe0 2024-03-31T15:04:03.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:03.597+0000 7f9e11400640 1 -- 172.21.15.16:0/2636425794 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e0c0a5c80 con 0x7f9e1406dbe0 2024-03-31T15:04:03.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:03.597+0000 7f9e18a00640 1 -- 172.21.15.16:0/2636425794 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9e1406dbe0 msgr2=0x7f9e140ccf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:03.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:03.597+0000 7f9e18a00640 1 --2- 172.21.15.16:0/2636425794 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9e1406dbe0 0x7f9e140ccf10 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f9e1406f620 tx=0x7f9e0c01c960 comp rx=0 tx=0).stop 2024-03-31T15:04:03.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:03.597+0000 7f9e18a00640 1 -- 172.21.15.16:0/2636425794 shutdown_connections 2024-03-31T15:04:03.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:03.597+0000 7f9e18a00640 1 --2- 172.21.15.16:0/2636425794 >> v2:172.21.15.89:3300/0 conn(0x7f9e1413e3d0 0x7f9e140cc9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:03.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:03.597+0000 7f9e18a00640 1 --2- 172.21.15.16:0/2636425794 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9e1406dbe0 0x7f9e140ccf10 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:03.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:03.597+0000 7f9e18a00640 1 -- 172.21.15.16:0/2636425794 >> 172.21.15.16:0/2636425794 conn(0x7f9e1406ee40 msgr2=0x7f9e1406b620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:04:03.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:03.597+0000 7f9e18a00640 1 -- 172.21.15.16:0/2636425794 shutdown_connections 2024-03-31T15:04:03.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:03.597+0000 7f9e18a00640 1 -- 172.21.15.16:0/2636425794 wait complete. 2024-03-31T15:04:03.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:03.601+0000 7f9e18a00640 1 Processor -- start 2024-03-31T15:04:03.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:03.601+0000 7f9e18a00640 1 -- start start 2024-03-31T15:04:03.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:03.601+0000 7f9e18a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9e1406dbe0 0x7f9e1413e7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:03.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:03.601+0000 7f9e18a00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f9e1413dff0 0x7f9e1413fa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:03.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:03.601+0000 7f9e18a00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9e1413e3d0 0x7f9e14152a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:03.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:03.601+0000 7f9e18a00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f9e140cf710 con 0x7f9e1406dbe0 2024-03-31T15:04:03.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:03.601+0000 7f9e18a00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f9e140cf590 con 0x7f9e1413e3d0 2024-03-31T15:04:03.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:03.601+0000 7f9e18a00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f9e140cf890 con 0x7f9e1413dff0 2024-03-31T15:04:03.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:03.601+0000 7f9e11e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9e1406dbe0 0x7f9e1413e7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:03.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:03.601+0000 7f9e12800640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9e1413e3d0 0x7f9e14152a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:03.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:03.601+0000 7f9e12800640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9e1413e3d0 0x7f9e14152a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.16:60564/0 (socket says 172.21.15.16:60564) 2024-03-31T15:04:03.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:03.601+0000 7f9e11e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9e1406dbe0 0x7f9e1413e7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60208/0 (socket says 172.21.15.16:60208) 2024-03-31T15:04:03.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:03.601+0000 7f9e12800640 1 -- 172.21.15.16:0/2592029422 learned_addr learned my addr 172.21.15.16:0/2592029422 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:04:03.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:03.601+0000 7f9e09400640 1 --2- 172.21.15.16:0/2592029422 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f9e1413dff0 0x7f9e1413fa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:03.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:03.601+0000 7f9e12800640 1 -- 172.21.15.16:0/2592029422 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f9e1413dff0 msgr2=0x7f9e1413fa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:03.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:03.601+0000 7f9e12800640 1 --2- 172.21.15.16:0/2592029422 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f9e1413dff0 0x7f9e1413fa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:03.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:03.601+0000 7f9e12800640 1 -- 172.21.15.16:0/2592029422 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9e1406dbe0 msgr2=0x7f9e1413e7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:03.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:03.601+0000 7f9e12800640 1 --2- 172.21.15.16:0/2592029422 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9e1406dbe0 0x7f9e1413e7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:03.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:03.601+0000 7f9e12800640 1 -- 172.21.15.16:0/2592029422 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e14153c10 con 0x7f9e1413e3d0 2024-03-31T15:04:03.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:03.601+0000 7f9e12800640 1 --2- 172.21.15.16:0/2592029422 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9e1413e3d0 0x7f9e14152a10 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f9e0c0af040 tx=0x7f9e0c019700 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:03.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:03.601+0000 7f9e0aa00640 1 -- 172.21.15.16:0/2592029422 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e0c0b8670 con 0x7f9e1413e3d0 2024-03-31T15:04:03.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:03.601+0000 7f9e0aa00640 1 -- 172.21.15.16:0/2592029422 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9e0c0be070 con 0x7f9e1413e3d0 2024-03-31T15:04:03.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:03.601+0000 7f9e0aa00640 1 -- 172.21.15.16:0/2592029422 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e0c0b8a50 con 0x7f9e1413e3d0 2024-03-31T15:04:03.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:03.601+0000 7f9e18a00640 1 -- 172.21.15.16:0/2592029422 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9e14153ef0 con 0x7f9e1413e3d0 2024-03-31T15:04:03.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:03.601+0000 7f9e18a00640 1 -- 172.21.15.16:0/2592029422 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9e141547d0 con 0x7f9e1413e3d0 2024-03-31T15:04:03.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:03.605+0000 7f9e18a00640 1 -- 172.21.15.16:0/2592029422 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9e14069950 con 0x7f9e1413e3d0 2024-03-31T15:04:03.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:03.605+0000 7f9e0aa00640 1 -- 172.21.15.16:0/2592029422 <== mon.1 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f9e0c0b8bf0 con 0x7f9e1413e3d0 2024-03-31T15:04:03.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:03.605+0000 7f9e0aa00640 1 --2- 172.21.15.16:0/2592029422 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f9df0068380 0x7f9df006a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:03.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:03.605+0000 7f9e0aa00640 1 -- 172.21.15.16:0/2592029422 <== mon.1 v2:172.21.15.89:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f9e0c121c90 con 0x7f9e1413e3d0 2024-03-31T15:04:03.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:03.605+0000 7f9e11e00640 1 --2- 172.21.15.16:0/2592029422 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f9df0068380 0x7f9df006a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:03.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:03.605+0000 7f9e11e00640 1 --2- 172.21.15.16:0/2592029422 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f9df0068380 0x7f9df006a840 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f9e04000f70 tx=0x7f9e04009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:03.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:03.609+0000 7f9e0aa00640 1 -- 172.21.15.16:0/2592029422 <== mon.1 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f9e0c0165e0 con 0x7f9e1413e3d0 2024-03-31T15:04:03.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:03.777+0000 7f9e18a00640 1 -- 172.21.15.16:0/2592029422 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f9e141540c0 con 0x7f9e1413e3d0 2024-03-31T15:04:03.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:03.777+0000 7f9e0aa00640 1 -- 172.21.15.16:0/2592029422 <== mon.1 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) ==== 100+0+11 (secure 0 0 0) 0x7f9e0c0a5880 con 0x7f9e1413e3d0 2024-03-31T15:04:03.771 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 64 2024-03-31T15:04:03.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:03.781+0000 7f9e18a00640 1 -- 172.21.15.16:0/2592029422 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f9df0068380 msgr2=0x7f9df006a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:03.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:03.781+0000 7f9e18a00640 1 --2- 172.21.15.16:0/2592029422 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f9df0068380 0x7f9df006a840 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f9e04000f70 tx=0x7f9e04009040 comp rx=0 tx=0).stop 2024-03-31T15:04:03.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:03.781+0000 7f9e18a00640 1 -- 172.21.15.16:0/2592029422 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9e1413e3d0 msgr2=0x7f9e14152a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:03.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:03.781+0000 7f9e18a00640 1 --2- 172.21.15.16:0/2592029422 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9e1413e3d0 0x7f9e14152a10 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f9e0c0af040 tx=0x7f9e0c019700 comp rx=0 tx=0).stop 2024-03-31T15:04:03.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:03.781+0000 7f9e18a00640 1 -- 172.21.15.16:0/2592029422 shutdown_connections 2024-03-31T15:04:03.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:03.781+0000 7f9e18a00640 1 --2- 172.21.15.16:0/2592029422 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f9df0068380 0x7f9df006a840 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:03.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:03.781+0000 7f9e18a00640 1 --2- 172.21.15.16:0/2592029422 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9e1413e3d0 0x7f9e14152a10 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:03.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:03.781+0000 7f9e18a00640 1 --2- 172.21.15.16:0/2592029422 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f9e1413dff0 0x7f9e1413fa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:03.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:03.781+0000 7f9e18a00640 1 --2- 172.21.15.16:0/2592029422 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9e1406dbe0 0x7f9e1413e7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:03.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:03.781+0000 7f9e18a00640 1 -- 172.21.15.16:0/2592029422 >> 172.21.15.16:0/2592029422 conn(0x7f9e1406ee40 msgr2=0x7f9e1413d1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:04:03.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:03.781+0000 7f9e18a00640 1 -- 172.21.15.16:0/2592029422 shutdown_connections 2024-03-31T15:04:03.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:03.781+0000 7f9e18a00640 1 -- 172.21.15.16:0/2592029422 wait complete. 2024-03-31T15:04:03.791 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-31T15:04:03.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:03.937+0000 7fdd63a00640 1 Processor -- start 2024-03-31T15:04:03.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:03.937+0000 7fdd63a00640 1 -- start start 2024-03-31T15:04:03.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:03.941+0000 7fdd63a00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fdd5c15b3d0 0x7fdd5c0b1cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:03.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:03.941+0000 7fdd63a00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fdd5c15eee0 0x7fdd5c0b2200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:03.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:03.941+0000 7fdd63a00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fdd5c0ce2f0 0x7fdd5c0cc9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:03.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:03.941+0000 7fdd63a00640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7fdd5c15b7b0 con 0x7fdd5c15eee0 2024-03-31T15:04:03.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:03.941+0000 7fdd63a00640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7fdd5c15f3c0 con 0x7fdd5c15b3d0 2024-03-31T15:04:03.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:03.941+0000 7fdd63a00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fdd5c15f6a0 con 0x7fdd5c0ce2f0 2024-03-31T15:04:03.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:03.941+0000 7fdd5be00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fdd5c15eee0 0x7fdd5c0b2200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:03.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:03.941+0000 7fdd61000640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fdd5c15b3d0 0x7fdd5c0b1cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:03.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:03.941+0000 7fdd5be00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fdd5c15eee0 0x7fdd5c0b2200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60218/0 (socket says 172.21.15.16:60218) 2024-03-31T15:04:03.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:03.941+0000 7fdd5be00640 1 -- 172.21.15.16:0/3393886009 learned_addr learned my addr 172.21.15.16:0/3393886009 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:04:03.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:03.941+0000 7fdd61a00640 1 --2- 172.21.15.16:0/3393886009 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fdd5c0ce2f0 0x7fdd5c0cc9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:03.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:03.941+0000 7fdd5be00640 1 -- 172.21.15.16:0/3393886009 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fdd5c0ce2f0 msgr2=0x7fdd5c0cc9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:04:03.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:03.941+0000 7fdd5be00640 1 --2- 172.21.15.16:0/3393886009 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fdd5c0ce2f0 0x7fdd5c0cc9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:03.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:03.941+0000 7fdd5be00640 1 -- 172.21.15.16:0/3393886009 >> v2:172.21.15.89:3300/0 conn(0x7fdd5c15b3d0 msgr2=0x7fdd5c0b1cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:04:03.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:03.941+0000 7fdd5be00640 1 --2- 172.21.15.16:0/3393886009 >> v2:172.21.15.89:3300/0 conn(0x7fdd5c15b3d0 0x7fdd5c0b1cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:03.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:03.941+0000 7fdd5be00640 1 -- 172.21.15.16:0/3393886009 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd5c15f980 con 0x7fdd5c15eee0 2024-03-31T15:04:03.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:03.941+0000 7fdd61000640 1 --2- 172.21.15.16:0/3393886009 >> v2:172.21.15.89:3300/0 conn(0x7fdd5c15b3d0 0x7fdd5c0b1cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:04:03.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:03.945+0000 7fdd5be00640 1 --2- 172.21.15.16:0/3393886009 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdd5c15eee0 0x7fdd5c0b2200 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fdd5c0691a0 tx=0x7fdd4801c4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db25d9ccb7684d8a server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:03.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:03.945+0000 7fdd5b400640 1 -- 172.21.15.16:0/3393886009 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd480a7070 con 0x7fdd5c15eee0 2024-03-31T15:04:03.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:03.945+0000 7fdd5b400640 1 -- 172.21.15.16:0/3393886009 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdd480b0070 con 0x7fdd5c15eee0 2024-03-31T15:04:03.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:03.945+0000 7fdd5b400640 1 -- 172.21.15.16:0/3393886009 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd480ab890 con 0x7fdd5c15eee0 2024-03-31T15:04:03.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:03.945+0000 7fdd63a00640 1 -- 172.21.15.16:0/3393886009 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdd5c15eee0 msgr2=0x7fdd5c0b2200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:03.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:03.945+0000 7fdd63a00640 1 --2- 172.21.15.16:0/3393886009 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdd5c15eee0 0x7fdd5c0b2200 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fdd5c0691a0 tx=0x7fdd4801c4f0 comp rx=0 tx=0).stop 2024-03-31T15:04:03.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:03.945+0000 7fdd63a00640 1 -- 172.21.15.16:0/3393886009 shutdown_connections 2024-03-31T15:04:03.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:03.945+0000 7fdd63a00640 1 --2- 172.21.15.16:0/3393886009 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fdd5c0ce2f0 0x7fdd5c0cc9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:03.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:03.945+0000 7fdd63a00640 1 --2- 172.21.15.16:0/3393886009 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdd5c15eee0 0x7fdd5c0b2200 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:03.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:03.945+0000 7fdd63a00640 1 --2- 172.21.15.16:0/3393886009 >> v2:172.21.15.89:3300/0 conn(0x7fdd5c15b3d0 0x7fdd5c0b1cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:03.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:03.945+0000 7fdd63a00640 1 -- 172.21.15.16:0/3393886009 >> 172.21.15.16:0/3393886009 conn(0x7fdd5c06b620 msgr2=0x7fdd5c06ba30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:04:03.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:03.945+0000 7fdd63a00640 1 -- 172.21.15.16:0/3393886009 shutdown_connections 2024-03-31T15:04:03.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:03.945+0000 7fdd63a00640 1 -- 172.21.15.16:0/3393886009 wait complete. 2024-03-31T15:04:03.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:03.945+0000 7fdd63a00640 1 Processor -- start 2024-03-31T15:04:03.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:03.945+0000 7fdd63a00640 1 -- start start 2024-03-31T15:04:03.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:03.945+0000 7fdd63a00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fdd5c0ce2f0 0x7fdd5c15bce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:03.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:03.945+0000 7fdd63a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdd5c15b3d0 0x7fdd5c15d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:03.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:03.949+0000 7fdd63a00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdd5c15eee0 0x7fdd5c0b1100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:03.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:03.949+0000 7fdd63a00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fdd5c071e70 con 0x7fdd5c15b3d0 2024-03-31T15:04:03.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:03.949+0000 7fdd63a00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fdd5c071cf0 con 0x7fdd5c15eee0 2024-03-31T15:04:03.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:03.949+0000 7fdd63a00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fdd5c071ff0 con 0x7fdd5c0ce2f0 2024-03-31T15:04:03.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:03.949+0000 7fdd5be00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdd5c15b3d0 0x7fdd5c15d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:03.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:03.949+0000 7fdd5be00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdd5c15b3d0 0x7fdd5c15d530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60230/0 (socket says 172.21.15.16:60230) 2024-03-31T15:04:03.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:03.949+0000 7fdd5be00640 1 -- 172.21.15.16:0/2269556237 learned_addr learned my addr 172.21.15.16:0/2269556237 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:04:03.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:03.949+0000 7fdd61000640 1 --2- 172.21.15.16:0/2269556237 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fdd5c0ce2f0 0x7fdd5c15bce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:03.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:03.949+0000 7fdd61a00640 1 --2- 172.21.15.16:0/2269556237 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdd5c15eee0 0x7fdd5c0b1100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:03.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:03.949+0000 7fdd5be00640 1 -- 172.21.15.16:0/2269556237 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fdd5c0ce2f0 msgr2=0x7fdd5c15bce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:03.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:03.949+0000 7fdd5be00640 1 --2- 172.21.15.16:0/2269556237 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fdd5c0ce2f0 0x7fdd5c15bce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:03.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:03.949+0000 7fdd5be00640 1 -- 172.21.15.16:0/2269556237 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdd5c15eee0 msgr2=0x7fdd5c0b1100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:03.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:03.949+0000 7fdd5be00640 1 --2- 172.21.15.16:0/2269556237 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdd5c15eee0 0x7fdd5c0b1100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:03.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:03.949+0000 7fdd5be00640 1 -- 172.21.15.16:0/2269556237 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd5c15e160 con 0x7fdd5c15b3d0 2024-03-31T15:04:03.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:03.949+0000 7fdd5be00640 1 --2- 172.21.15.16:0/2269556237 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdd5c15b3d0 0x7fdd5c15d530 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fdd480ad040 tx=0x7fdd4801ad30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:03.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:03.949+0000 7fdd58c00640 1 -- 172.21.15.16:0/2269556237 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd480be5f0 con 0x7fdd5c15b3d0 2024-03-31T15:04:03.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:03.949+0000 7fdd58c00640 1 -- 172.21.15.16:0/2269556237 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdd480be790 con 0x7fdd5c15b3d0 2024-03-31T15:04:03.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:03.949+0000 7fdd58c00640 1 -- 172.21.15.16:0/2269556237 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd480beb80 con 0x7fdd5c15b3d0 2024-03-31T15:04:03.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:03.949+0000 7fdd63a00640 1 -- 172.21.15.16:0/2269556237 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdd5c15e440 con 0x7fdd5c15b3d0 2024-03-31T15:04:03.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:03.949+0000 7fdd63a00640 1 -- 172.21.15.16:0/2269556237 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdd5c0ad9a0 con 0x7fdd5c15b3d0 2024-03-31T15:04:03.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:03.949+0000 7fdd63a00640 1 -- 172.21.15.16:0/2269556237 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdd5c069950 con 0x7fdd5c15b3d0 2024-03-31T15:04:03.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:03.949+0000 7fdd58c00640 1 -- 172.21.15.16:0/2269556237 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fdd480b43f0 con 0x7fdd5c15b3d0 2024-03-31T15:04:03.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:03.953+0000 7fdd58c00640 1 --2- 172.21.15.16:0/2269556237 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fdd3c068380 0x7fdd3c06a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:03.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:03.953+0000 7fdd58c00640 1 -- 172.21.15.16:0/2269556237 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fdd48121540 con 0x7fdd5c15b3d0 2024-03-31T15:04:03.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:03.953+0000 7fdd61000640 1 --2- 172.21.15.16:0/2269556237 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fdd3c068380 0x7fdd3c06a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:03.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:03.953+0000 7fdd61000640 1 --2- 172.21.15.16:0/2269556237 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fdd3c068380 0x7fdd3c06a840 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fdd50009580 tx=0x7fdd50009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:03.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:03.957+0000 7fdd58c00640 1 -- 172.21.15.16:0/2269556237 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fdd480c4060 con 0x7fdd5c15b3d0 2024-03-31T15:04:04.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:04.121+0000 7fdd63a00640 1 -- 172.21.15.16:0/2269556237 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fdd5c0cdea0 con 0x7fdd5c15b3d0 2024-03-31T15:04:04.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:04.125+0000 7fdd58c00640 1 -- 172.21.15.16:0/2269556237 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) ==== 74+0+16533 (secure 0 0 0) 0x7fdd48016110 con 0x7fdd5c15b3d0 2024-03-31T15:04:04.117 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:04:04.117 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":21,"fsid":"286eb5da-fb47-4550-b290-d9519c13b9a5","created":"2024-03-31T15:03:11.438952+0000","modified":"2024-03-31T15:03:37.957829+0000","last_up_change":"2024-03-31T15:03:22.885535+0000","last_in_change":"2024-03-31T15:03:13.941667+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-31T15:03:22.550787+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-31T15:03:34.488425+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-31T15:03:35.299116+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"92a66450-ef19-40d8-ac3d-d760b23b0bda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6801","nonce":932712244}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6803","nonce":932712244}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6807","nonce":932712244}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6805","nonce":932712244}]},"public_addr":"172.21.15.16:6801/932712244","cluster_addr":"172.21.15.16:6803/932712244","heartbeat_back_addr":"172.21.15.16:6807/932712244","heartbeat_front_addr":"172.21.15.16:6805/932712244","state":["exists","up"]},{"osd":1,"uuid":"da9d1f2b-1968-4959-b843-faa231988e92","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6810","nonce":1398737533}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6811","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6813","nonce":1398737533}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6821","nonce":1398737533}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6815","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6816","nonce":1398737533}]},"public_addr":"172.21.15.16:6810/1398737533","cluster_addr":"172.21.15.16:6813/1398737533","heartbeat_back_addr":"172.21.15.16:6821/1398737533","heartbeat_front_addr":"172.21.15.16:6816/1398737533","state":["exists","up"]},{"osd":2,"uuid":"01a73662-9a21-4b1f-bcbb-42b9636a4638","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6819","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6822","nonce":3018526646}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6825","nonce":3018526646}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6831","nonce":3018526646}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6827","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6829","nonce":3018526646}]},"public_addr":"172.21.15.16:6822/3018526646","cluster_addr":"172.21.15.16:6825/3018526646","heartbeat_back_addr":"172.21.15.16:6831/3018526646","heartbeat_front_addr":"172.21.15.16:6829/3018526646","state":["exists","up"]},{"osd":3,"uuid":"c7771fd3-e18e-4500-ac38-23cf10ebcd75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6809","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6812","nonce":3360352974}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6817","nonce":3360352974}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6828","nonce":3360352974}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6823","nonce":3360352974}]},"public_addr":"172.21.15.16:6812/3360352974","cluster_addr":"172.21.15.16:6817/3360352974","heartbeat_back_addr":"172.21.15.16:6828/3360352974","heartbeat_front_addr":"172.21.15.16:6823/3360352974","state":["exists","up"]},{"osd":4,"uuid":"88260730-1482-4fbd-a40f-cb1124e9df4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6801","nonce":1856844856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6803","nonce":1856844856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6807","nonce":1856844856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6805","nonce":1856844856}]},"public_addr":"172.21.15.89:6801/1856844856","cluster_addr":"172.21.15.89:6803/1856844856","heartbeat_back_addr":"172.21.15.89:6807/1856844856","heartbeat_front_addr":"172.21.15.89:6805/1856844856","state":["exists","up"]},{"osd":5,"uuid":"4cb2c00c-fae0-4dd8-9d27-ec4f0093ccf7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6814","nonce":50395527}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6817","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6820","nonce":50395527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6831","nonce":50395527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6823","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6826","nonce":50395527}]},"public_addr":"172.21.15.89:6814/50395527","cluster_addr":"172.21.15.89:6820/50395527","heartbeat_back_addr":"172.21.15.89:6831/50395527","heartbeat_front_addr":"172.21.15.89:6826/50395527","state":["exists","up"]},{"osd":6,"uuid":"81ce0c19-2554-4cae-acbe-66a05e9e9b9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6809","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6812","nonce":3470471752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6815","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6818","nonce":3470471752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6827","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6829","nonce":3470471752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6821","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6824","nonce":3470471752}]},"public_addr":"172.21.15.89:6812/3470471752","cluster_addr":"172.21.15.89:6818/3470471752","heartbeat_back_addr":"172.21.15.89:6829/3470471752","heartbeat_front_addr":"172.21.15.89:6824/3470471752","state":["exists","up"]},{"osd":7,"uuid":"85e4313b-d37d-442e-8042-153bf20a847d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6811","nonce":1796778630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6813","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6816","nonce":1796778630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6825","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6828","nonce":1796778630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6819","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6822","nonce":1796778630}]},"public_addr":"172.21.15.89:6811/1796778630","cluster_addr":"172.21.15.89:6816/1796778630","heartbeat_back_addr":"172.21.15.89:6828/1796778630","heartbeat_front_addr":"172.21.15.89:6822/1796778630","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.439088+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.529970+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.680512+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:21.044161+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.582111+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.536558+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.409522+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.590229+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-31T15:04:04.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:04.125+0000 7fdd63a00640 1 -- 172.21.15.16:0/2269556237 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fdd3c068380 msgr2=0x7fdd3c06a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:04.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:04.125+0000 7fdd63a00640 1 --2- 172.21.15.16:0/2269556237 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fdd3c068380 0x7fdd3c06a840 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fdd50009580 tx=0x7fdd50009190 comp rx=0 tx=0).stop 2024-03-31T15:04:04.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:04.125+0000 7fdd63a00640 1 -- 172.21.15.16:0/2269556237 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdd5c15b3d0 msgr2=0x7fdd5c15d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:04.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:04.125+0000 7fdd63a00640 1 --2- 172.21.15.16:0/2269556237 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdd5c15b3d0 0x7fdd5c15d530 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fdd480ad040 tx=0x7fdd4801ad30 comp rx=0 tx=0).stop 2024-03-31T15:04:04.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:04.125+0000 7fdd63a00640 1 -- 172.21.15.16:0/2269556237 shutdown_connections 2024-03-31T15:04:04.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:04.125+0000 7fdd63a00640 1 --2- 172.21.15.16:0/2269556237 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fdd3c068380 0x7fdd3c06a840 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:04.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:04.125+0000 7fdd63a00640 1 --2- 172.21.15.16:0/2269556237 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdd5c15eee0 0x7fdd5c0b1100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:04.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:04.125+0000 7fdd63a00640 1 --2- 172.21.15.16:0/2269556237 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdd5c15b3d0 0x7fdd5c15d530 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:04.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:04.125+0000 7fdd63a00640 1 --2- 172.21.15.16:0/2269556237 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fdd5c0ce2f0 0x7fdd5c15bce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:04.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:04.125+0000 7fdd63a00640 1 -- 172.21.15.16:0/2269556237 >> 172.21.15.16:0/2269556237 conn(0x7fdd5c06b620 msgr2=0x7fdd5c0b1c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:04:04.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:04.125+0000 7fdd63a00640 1 -- 172.21.15.16:0/2269556237 shutdown_connections 2024-03-31T15:04:04.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:04.125+0000 7fdd63a00640 1 -- 172.21.15.16:0/2269556237 wait complete. 2024-03-31T15:04:04.139 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-31T15:03:22.550787+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-31T15:03:34.488425+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-31T15:03:35.299116+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-31T15:04:04.139 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-31T15:04:04.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:04.289+0000 7f206e200640 1 Processor -- start 2024-03-31T15:04:04.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:04.289+0000 7f206e200640 1 -- start start 2024-03-31T15:04:04.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:04.289+0000 7f206e200640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f2068165150 0x7f2068167540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:04.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:04.289+0000 7f206e200640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f206815c7e0 con 0x7f20681567e0 2024-03-31T15:04:04.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:04.289+0000 7f206e200640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f20680736b0 con 0x7f206806d4d0 2024-03-31T15:04:04.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:04.289+0000 7f206e200640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f206815caa0 con 0x7f2068165150 2024-03-31T15:04:04.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:04.289+0000 7f2067400640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f20681567e0 0x7f2068072c80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:51220/0 (socket says 172.21.15.16:51220) 2024-03-31T15:04:04.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:04.289+0000 7f2067400640 1 -- 172.21.15.16:0/3839971038 learned_addr learned my addr 172.21.15.16:0/3839971038 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:04:04.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:04.289+0000 7f2067e00640 1 --2- 172.21.15.16:0/3839971038 >> v2:172.21.15.89:3300/0 conn(0x7f2068165150 0x7f2068167540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:04.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:04.289+0000 7f2066a00640 1 -- 172.21.15.16:0/3839971038 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 360888818 0 0) 0x7f206815c7e0 con 0x7f20681567e0 2024-03-31T15:04:04.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:04.289+0000 7f2066a00640 1 -- 172.21.15.16:0/3839971038 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f204c003530 con 0x7f20681567e0 2024-03-31T15:04:04.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:04.289+0000 7f2066a00640 1 -- 172.21.15.16:0/3839971038 <== mon.1 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2603737715 0 0) 0x7f20680736b0 con 0x7f206806d4d0 2024-03-31T15:04:04.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:04.289+0000 7f2066a00640 1 -- 172.21.15.16:0/3839971038 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f206815c7e0 con 0x7f206806d4d0 2024-03-31T15:04:04.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:04.289+0000 7f2067e00640 1 -- 172.21.15.16:0/3839971038 >> v1:172.21.15.89:6789/0 conn(0x7f206806d4d0 legacy=0x7f206806d930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:04.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:04.289+0000 7f2067e00640 1 -- 172.21.15.16:0/3839971038 >> v1:172.21.15.16:6789/0 conn(0x7f20681567e0 legacy=0x7f2068072c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:04.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:04.293+0000 7f2067e00640 1 -- 172.21.15.16:0/3839971038 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f206815cd80 con 0x7f2068165150 2024-03-31T15:04:04.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:04.293+0000 7f2067e00640 1 --2- 172.21.15.16:0/3839971038 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2068165150 0x7f2068167540 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f20600097c0 tx=0x7f206001c4a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8f46adff957a68dd server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:04.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:04.293+0000 7f2066a00640 1 -- 172.21.15.16:0/3839971038 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20600b0070 con 0x7f2068165150 2024-03-31T15:04:04.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:04.293+0000 7f2066a00640 1 -- 172.21.15.16:0/3839971038 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f20600ab600 con 0x7f2068165150 2024-03-31T15:04:04.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:04.293+0000 7f2066a00640 1 -- 172.21.15.16:0/3839971038 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20600ab9b0 con 0x7f2068165150 2024-03-31T15:04:04.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:04.293+0000 7f206e200640 1 -- 172.21.15.16:0/3839971038 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2068165150 msgr2=0x7f2068167540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:04.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:04.293+0000 7f206e200640 1 --2- 172.21.15.16:0/3839971038 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2068165150 0x7f2068167540 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f20600097c0 tx=0x7f206001c4a0 comp rx=0 tx=0).stop 2024-03-31T15:04:04.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:04.293+0000 7f206e200640 1 -- 172.21.15.16:0/3839971038 shutdown_connections 2024-03-31T15:04:04.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:04.293+0000 7f206e200640 1 --2- 172.21.15.16:0/3839971038 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2068165150 0x7f2068167540 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:04.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:04.293+0000 7f206e200640 1 -- 172.21.15.16:0/3839971038 >> 172.21.15.16:0/3839971038 conn(0x7f206806ee40 msgr2=0x7f206806b620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:04:04.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:04.293+0000 7f206e200640 1 -- 172.21.15.16:0/3839971038 shutdown_connections 2024-03-31T15:04:04.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:04.293+0000 7f206e200640 1 -- 172.21.15.16:0/3839971038 wait complete. 2024-03-31T15:04:04.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:04.297+0000 7f206e200640 1 Processor -- start 2024-03-31T15:04:04.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:04.297+0000 7f206e200640 1 -- start start 2024-03-31T15:04:04.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:04.297+0000 7f206e200640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f206806d4d0 0x7f2068150330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:04.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:04.297+0000 7f206e200640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f20681567e0 0x7f2068153410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:04.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:04.297+0000 7f206e200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2068165150 0x7f20681635b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:04.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:04.297+0000 7f206e200640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f206815d6e0 con 0x7f2068165150 2024-03-31T15:04:04.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:04.297+0000 7f206e200640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f206815d560 con 0x7f20681567e0 2024-03-31T15:04:04.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:04.297+0000 7f206e200640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f206816a6b0 con 0x7f206806d4d0 2024-03-31T15:04:04.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:04.297+0000 7f2067e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2068165150 0x7f20681635b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:04.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:04.297+0000 7f205ea00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f20681567e0 0x7f2068153410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:04.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:04.297+0000 7f2067400640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f206806d4d0 0x7f2068150330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:04.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:04.297+0000 7f2067e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2068165150 0x7f20681635b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60234/0 (socket says 172.21.15.16:60234) 2024-03-31T15:04:04.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:04.297+0000 7f205ea00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f20681567e0 0x7f2068153410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.16:60608/0 (socket says 172.21.15.16:60608) 2024-03-31T15:04:04.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:04.297+0000 7f2067e00640 1 -- 172.21.15.16:0/2406835708 learned_addr learned my addr 172.21.15.16:0/2406835708 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:04:04.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:04.297+0000 7f2067e00640 1 -- 172.21.15.16:0/2406835708 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f206806d4d0 msgr2=0x7f2068150330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:04.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:04.297+0000 7f2067e00640 1 --2- 172.21.15.16:0/2406835708 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f206806d4d0 0x7f2068150330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:04.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:04.297+0000 7f2067e00640 1 -- 172.21.15.16:0/2406835708 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f20681567e0 msgr2=0x7f2068153410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:04.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:04.297+0000 7f2067e00640 1 --2- 172.21.15.16:0/2406835708 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f20681567e0 0x7f2068153410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:04.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:04.297+0000 7f2067e00640 1 -- 172.21.15.16:0/2406835708 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2068154f00 con 0x7f2068165150 2024-03-31T15:04:04.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:04.297+0000 7f2067e00640 1 --2- 172.21.15.16:0/2406835708 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2068165150 0x7f20681635b0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f20600ae040 tx=0x7f2060019240 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:04.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:04.297+0000 7f205fe00640 1 -- 172.21.15.16:0/2406835708 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20600b0040 con 0x7f2068165150 2024-03-31T15:04:04.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:04.297+0000 7f205fe00640 1 -- 172.21.15.16:0/2406835708 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f20600ab400 con 0x7f2068165150 2024-03-31T15:04:04.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:04.297+0000 7f205fe00640 1 -- 172.21.15.16:0/2406835708 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2060006e20 con 0x7f2068165150 2024-03-31T15:04:04.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:04.297+0000 7f206e200640 1 -- 172.21.15.16:0/2406835708 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2068153e60 con 0x7f2068165150 2024-03-31T15:04:04.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:04.297+0000 7f206e200640 1 -- 172.21.15.16:0/2406835708 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2068154420 con 0x7f2068165150 2024-03-31T15:04:04.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:04.301+0000 7f206e200640 1 -- 172.21.15.16:0/2406835708 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2068069950 con 0x7f2068165150 2024-03-31T15:04:04.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:04.301+0000 7f205fe00640 1 -- 172.21.15.16:0/2406835708 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f20600bc020 con 0x7f2068165150 2024-03-31T15:04:04.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:04.301+0000 7f205fe00640 1 --2- 172.21.15.16:0/2406835708 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f2044068380 0x7f204406a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:04.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:04.301+0000 7f205fe00640 1 -- 172.21.15.16:0/2406835708 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f206011fda0 con 0x7f2068165150 2024-03-31T15:04:04.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:04.301+0000 7f2067400640 1 --2- 172.21.15.16:0/2406835708 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f2044068380 0x7f204406a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:04.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:04.301+0000 7f2067400640 1 --2- 172.21.15.16:0/2406835708 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f2044068380 0x7f204406a840 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f20580012f0 tx=0x7f2058001ae0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:04.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:04.305+0000 7f205fe00640 1 -- 172.21.15.16:0/2406835708 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f2060016090 con 0x7f2068165150 2024-03-31T15:04:04.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:04.469+0000 7f206e200640 1 -- 172.21.15.16:0/2406835708 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f20681519f0 con 0x7f2068165150 2024-03-31T15:04:04.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:04.469+0000 7f205fe00640 1 -- 172.21.15.16:0/2406835708 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) ==== 93+0+10 (secure 0 0 0) 0x7f20600efc70 con 0x7f2068165150 2024-03-31T15:04:04.462 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2024-03-31T15:04:04.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:04.469+0000 7f206e200640 1 -- 172.21.15.16:0/2406835708 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f2044068380 msgr2=0x7f204406a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:04.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:04.469+0000 7f206e200640 1 --2- 172.21.15.16:0/2406835708 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f2044068380 0x7f204406a840 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f20580012f0 tx=0x7f2058001ae0 comp rx=0 tx=0).stop 2024-03-31T15:04:04.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:04.469+0000 7f206e200640 1 -- 172.21.15.16:0/2406835708 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2068165150 msgr2=0x7f20681635b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:04.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:04.469+0000 7f206e200640 1 --2- 172.21.15.16:0/2406835708 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2068165150 0x7f20681635b0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f20600ae040 tx=0x7f2060019240 comp rx=0 tx=0).stop 2024-03-31T15:04:04.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:04.469+0000 7f206e200640 1 -- 172.21.15.16:0/2406835708 shutdown_connections 2024-03-31T15:04:04.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:04.469+0000 7f206e200640 1 --2- 172.21.15.16:0/2406835708 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f2044068380 0x7f204406a840 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:04.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:04.469+0000 7f206e200640 1 --2- 172.21.15.16:0/2406835708 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2068165150 0x7f20681635b0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:04.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:04.469+0000 7f206e200640 1 --2- 172.21.15.16:0/2406835708 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f20681567e0 0x7f2068153410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:04.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:04.469+0000 7f206e200640 1 --2- 172.21.15.16:0/2406835708 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f206806d4d0 0x7f2068150330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:04.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:04.469+0000 7f206e200640 1 -- 172.21.15.16:0/2406835708 >> 172.21.15.16:0/2406835708 conn(0x7f206806ee40 msgr2=0x7f206806d930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:04:04.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:04.469+0000 7f206e200640 1 -- 172.21.15.16:0/2406835708 shutdown_connections 2024-03-31T15:04:04.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:04.469+0000 7f206e200640 1 -- 172.21.15.16:0/2406835708 wait complete. 2024-03-31T15:04:04.483 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-31T15:04:04.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:04.633+0000 7f56ef200640 1 Processor -- start 2024-03-31T15:04:04.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:04.633+0000 7f56ef200640 1 -- start start 2024-03-31T15:04:04.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:04.633+0000 7f56ef200640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f56e8070030 0x7f56e8143c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:04.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:04.633+0000 7f56ef200640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f56e8141590 0x7f56e8144170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:04.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:04.633+0000 7f56ef200640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f56e81446b0 con 0x7f56e81411b0 2024-03-31T15:04:04.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:04.633+0000 7f56ef200640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f56e815f910 con 0x7f56e8070030 2024-03-31T15:04:04.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:04.633+0000 7f56ef200640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f56e8162910 con 0x7f56e8141590 2024-03-31T15:04:04.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:04.633+0000 7f56df400640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f56e8070030 0x7f56e8143c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:04.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:04.633+0000 7f56ed200640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f56e8141590 0x7f56e8144170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:04.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:04.633+0000 7f56e7e00640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f56e81411b0 0x7f56e806f920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.16:38306/0 (socket says 172.21.15.16:38306) 2024-03-31T15:04:04.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:04.633+0000 7f56e7e00640 1 -- 172.21.15.16:0/691595074 learned_addr learned my addr 172.21.15.16:0/691595074 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:04:04.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:04.633+0000 7f56ed200640 1 -- 172.21.15.16:0/691595074 >> v2:172.21.15.89:3300/0 conn(0x7f56e8070030 msgr2=0x7f56e8143c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:04:04.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:04.633+0000 7f56ed200640 1 --2- 172.21.15.16:0/691595074 >> v2:172.21.15.89:3300/0 conn(0x7f56e8070030 0x7f56e8143c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:04.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:04.633+0000 7f56ed200640 1 -- 172.21.15.16:0/691595074 >> v1:172.21.15.89:6789/0 conn(0x7f56e81411b0 legacy=0x7f56e806f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:04.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:04.633+0000 7f56ed200640 1 -- 172.21.15.16:0/691595074 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56e8144980 con 0x7f56e8141590 2024-03-31T15:04:04.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:04.637+0000 7f56ed200640 1 --2- 172.21.15.16:0/691595074 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f56e8141590 0x7f56e8144170 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f56e0009d90 tx=0x7f56e001c940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5c0b72a676fbbca4 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:04.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:04.637+0000 7f56e7400640 1 -- 172.21.15.16:0/691595074 <== mon.2 v2:172.21.15.89:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56e00b2070 con 0x7f56e8141590 2024-03-31T15:04:04.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:04.637+0000 7f56e7400640 1 -- 172.21.15.16:0/691595074 <== mon.2 v2:172.21.15.89:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f56e00a5a00 con 0x7f56e8141590 2024-03-31T15:04:04.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:04.637+0000 7f56e7400640 1 -- 172.21.15.16:0/691595074 <== mon.2 v2:172.21.15.89:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56e00a5dd0 con 0x7f56e8141590 2024-03-31T15:04:04.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:04.637+0000 7f56ef200640 1 -- 172.21.15.16:0/691595074 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f56e8141590 msgr2=0x7f56e8144170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:04.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:04.637+0000 7f56ef200640 1 --2- 172.21.15.16:0/691595074 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f56e8141590 0x7f56e8144170 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f56e0009d90 tx=0x7f56e001c940 comp rx=0 tx=0).stop 2024-03-31T15:04:04.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:04.637+0000 7f56ef200640 1 -- 172.21.15.16:0/691595074 shutdown_connections 2024-03-31T15:04:04.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:04.637+0000 7f56ef200640 1 --2- 172.21.15.16:0/691595074 >> v2:172.21.15.89:3300/0 conn(0x7f56e8070030 0x7f56e8143c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:04.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:04.637+0000 7f56ef200640 1 --2- 172.21.15.16:0/691595074 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f56e8141590 0x7f56e8144170 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:04.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:04.637+0000 7f56ef200640 1 -- 172.21.15.16:0/691595074 >> 172.21.15.16:0/691595074 conn(0x7f56e8067930 msgr2=0x7f56e80b3ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:04:04.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:04.637+0000 7f56ef200640 1 -- 172.21.15.16:0/691595074 shutdown_connections 2024-03-31T15:04:04.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:04.637+0000 7f56ef200640 1 -- 172.21.15.16:0/691595074 wait complete. 2024-03-31T15:04:04.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:04.641+0000 7f56ef200640 1 Processor -- start 2024-03-31T15:04:04.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:04.641+0000 7f56ef200640 1 -- start start 2024-03-31T15:04:04.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:04.641+0000 7f56ef200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f56e8070030 0x7f56e81402a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:04.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:04.641+0000 7f56ef200640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f56e81411b0 0x7f56e806d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:04.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:04.641+0000 7f56ef200640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f56e8141590 0x7f56e806dcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:04.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:04.641+0000 7f56ef200640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f56e815f110 con 0x7f56e8070030 2024-03-31T15:04:04.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:04.641+0000 7f56ef200640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f56e8162d80 con 0x7f56e81411b0 2024-03-31T15:04:04.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:04.641+0000 7f56ef200640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f56e815f290 con 0x7f56e8141590 2024-03-31T15:04:04.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:04.641+0000 7f56e7e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f56e8070030 0x7f56e81402a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:04.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:04.641+0000 7f56e7e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f56e8070030 0x7f56e81402a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60238/0 (socket says 172.21.15.16:60238) 2024-03-31T15:04:04.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:04.641+0000 7f56e7e00640 1 -- 172.21.15.16:0/2972521141 learned_addr learned my addr 172.21.15.16:0/2972521141 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:04:04.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:04.641+0000 7f56ed200640 1 --2- 172.21.15.16:0/2972521141 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f56e8141590 0x7f56e806dcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:04.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:04.641+0000 7f56df400640 1 --2- 172.21.15.16:0/2972521141 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f56e81411b0 0x7f56e806d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:04.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:04.641+0000 7f56e7e00640 1 -- 172.21.15.16:0/2972521141 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f56e8141590 msgr2=0x7f56e806dcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:04.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:04.641+0000 7f56e7e00640 1 --2- 172.21.15.16:0/2972521141 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f56e8141590 0x7f56e806dcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:04.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:04.641+0000 7f56e7e00640 1 -- 172.21.15.16:0/2972521141 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f56e81411b0 msgr2=0x7f56e806d780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:04.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:04.641+0000 7f56e7e00640 1 --2- 172.21.15.16:0/2972521141 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f56e81411b0 0x7f56e806d780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:04.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:04.641+0000 7f56e7e00640 1 -- 172.21.15.16:0/2972521141 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56e8142640 con 0x7f56e8070030 2024-03-31T15:04:04.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:04.641+0000 7f56e7e00640 1 --2- 172.21.15.16:0/2972521141 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f56e8070030 0x7f56e81402a0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f56d800c420 tx=0x7f56d800c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:04.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:04.641+0000 7f56e4c00640 1 -- 172.21.15.16:0/2972521141 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56d8017070 con 0x7f56e8070030 2024-03-31T15:04:04.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:04.641+0000 7f56e4c00640 1 -- 172.21.15.16:0/2972521141 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f56d8005cf0 con 0x7f56e8070030 2024-03-31T15:04:04.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:04.641+0000 7f56e4c00640 1 -- 172.21.15.16:0/2972521141 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56d80072f0 con 0x7f56e8070030 2024-03-31T15:04:04.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:04.641+0000 7f56ef200640 1 -- 172.21.15.16:0/2972521141 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f56e8142920 con 0x7f56e8070030 2024-03-31T15:04:04.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:04.641+0000 7f56ef200640 1 -- 172.21.15.16:0/2972521141 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f56e8142ee0 con 0x7f56e8070030 2024-03-31T15:04:04.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:04.645+0000 7f56ef200640 1 -- 172.21.15.16:0/2972521141 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f56e8069950 con 0x7f56e8070030 2024-03-31T15:04:04.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:04.645+0000 7f56e4c00640 1 -- 172.21.15.16:0/2972521141 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f56d801a070 con 0x7f56e8070030 2024-03-31T15:04:04.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:04.645+0000 7f56e4c00640 1 --2- 172.21.15.16:0/2972521141 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f56c4068380 0x7f56c406a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:04.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:04.645+0000 7f56e4c00640 1 -- 172.21.15.16:0/2972521141 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f56d8085cd0 con 0x7f56e8070030 2024-03-31T15:04:04.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:04.645+0000 7f56df400640 1 --2- 172.21.15.16:0/2972521141 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f56c4068380 0x7f56c406a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:04.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:04.645+0000 7f56df400640 1 --2- 172.21.15.16:0/2972521141 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f56c4068380 0x7f56c406a840 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f56e8158820 tx=0x7f56d0009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:04.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:04.649+0000 7f56e4c00640 1 -- 172.21.15.16:0/2972521141 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f56d8055c30 con 0x7f56e8070030 2024-03-31T15:04:04.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:04.817+0000 7f56ef200640 1 -- 172.21.15.16:0/2972521141 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f56e80d2f00 con 0x7f56e8070030 2024-03-31T15:04:04.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:04.817+0000 7f56e4c00640 1 -- 172.21.15.16:0/2972521141 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) ==== 104+0+11 (secure 0 0 0) 0x7f56d8059f30 con 0x7f56e8070030 2024-03-31T15:04:04.809 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2024-03-31T15:04:04.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:04.817+0000 7f56ef200640 1 -- 172.21.15.16:0/2972521141 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f56c4068380 msgr2=0x7f56c406a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:04.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:04.817+0000 7f56ef200640 1 --2- 172.21.15.16:0/2972521141 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f56c4068380 0x7f56c406a840 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f56e8158820 tx=0x7f56d0009190 comp rx=0 tx=0).stop 2024-03-31T15:04:04.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:04.817+0000 7f56ef200640 1 -- 172.21.15.16:0/2972521141 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f56e8070030 msgr2=0x7f56e81402a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:04.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:04.817+0000 7f56ef200640 1 --2- 172.21.15.16:0/2972521141 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f56e8070030 0x7f56e81402a0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f56d800c420 tx=0x7f56d800c8e0 comp rx=0 tx=0).stop 2024-03-31T15:04:04.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:04.817+0000 7f56ef200640 1 -- 172.21.15.16:0/2972521141 shutdown_connections 2024-03-31T15:04:04.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:04.817+0000 7f56ef200640 1 --2- 172.21.15.16:0/2972521141 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f56c4068380 0x7f56c406a840 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:04.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:04.817+0000 7f56ef200640 1 --2- 172.21.15.16:0/2972521141 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f56e8141590 0x7f56e806dcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:04.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:04.817+0000 7f56ef200640 1 --2- 172.21.15.16:0/2972521141 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f56e81411b0 0x7f56e806d780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:04.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:04.817+0000 7f56ef200640 1 --2- 172.21.15.16:0/2972521141 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f56e8070030 0x7f56e81402a0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:04.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:04.817+0000 7f56ef200640 1 -- 172.21.15.16:0/2972521141 >> 172.21.15.16:0/2972521141 conn(0x7f56e8067930 msgr2=0x7f56e8143e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:04:04.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:04.817+0000 7f56ef200640 1 -- 172.21.15.16:0/2972521141 shutdown_connections 2024-03-31T15:04:04.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:04.817+0000 7f56ef200640 1 -- 172.21.15.16:0/2972521141 wait complete. 2024-03-31T15:04:04.830 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-31T15:04:04.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:04.981+0000 7f5660200640 1 Processor -- start 2024-03-31T15:04:04.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:04.981+0000 7f5660200640 1 -- start start 2024-03-31T15:04:04.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:04.981+0000 7f5660200640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f56580d0a60 0x7f56580cea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:04.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:04.981+0000 7f5660200640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f565815ca70 con 0x7f56580cef80 2024-03-31T15:04:04.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:04.981+0000 7f5660200640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f565815cd50 con 0x7f565806f920 2024-03-31T15:04:04.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:04.981+0000 7f5660200640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f565813f670 con 0x7f56580d0a60 2024-03-31T15:04:04.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:04.981+0000 7f565ce00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f56580d0a60 0x7f56580cea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:04.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:04.981+0000 7f565e200640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f56580cef80 0x7f56581419c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:51226/0 (socket says 172.21.15.16:51226) 2024-03-31T15:04:04.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:04.981+0000 7f565d800640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f565806f920 0x7f565806fd00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.16:38318/0 (socket says 172.21.15.16:38318) 2024-03-31T15:04:04.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:04.981+0000 7f565e200640 1 -- 172.21.15.16:0/986485695 learned_addr learned my addr 172.21.15.16:0/986485695 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:04:04.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:04.985+0000 7f565ce00640 1 -- 172.21.15.16:0/986485695 >> v1:172.21.15.89:6789/0 conn(0x7f565806f920 legacy=0x7f565806fd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:04.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:04.985+0000 7f5657e00640 1 -- 172.21.15.16:0/986485695 <== mon.1 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 165869408 0 0) 0x7f565815cd50 con 0x7f565806f920 2024-03-31T15:04:04.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:04.985+0000 7f565ce00640 1 -- 172.21.15.16:0/986485695 >> v1:172.21.15.16:6789/0 conn(0x7f56580cef80 legacy=0x7f56581419c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:04.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:04.985+0000 7f565ce00640 1 -- 172.21.15.16:0/986485695 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f565815d030 con 0x7f56580d0a60 2024-03-31T15:04:04.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:04.985+0000 7f565ce00640 1 --2- 172.21.15.16:0/986485695 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f56580d0a60 0x7f56580cea40 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f56440097c0 tx=0x7f564401c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8f141c8aa6a4e1f server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:04.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:04.985+0000 7f5657e00640 1 -- 172.21.15.16:0/986485695 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56440b0070 con 0x7f56580d0a60 2024-03-31T15:04:04.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:04.985+0000 7f5657e00640 1 -- 172.21.15.16:0/986485695 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f56440ab600 con 0x7f56580d0a60 2024-03-31T15:04:04.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:04.985+0000 7f5657e00640 1 -- 172.21.15.16:0/986485695 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56440ab9d0 con 0x7f56580d0a60 2024-03-31T15:04:04.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:04.985+0000 7f5660200640 1 -- 172.21.15.16:0/986485695 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f56580d0a60 msgr2=0x7f56580cea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:04.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:04.985+0000 7f5660200640 1 --2- 172.21.15.16:0/986485695 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f56580d0a60 0x7f56580cea40 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f56440097c0 tx=0x7f564401c4a0 comp rx=0 tx=0).stop 2024-03-31T15:04:04.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:04.985+0000 7f5660200640 1 -- 172.21.15.16:0/986485695 shutdown_connections 2024-03-31T15:04:04.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:04.985+0000 7f5660200640 1 --2- 172.21.15.16:0/986485695 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f56580d0a60 0x7f56580cea40 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:04.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:04.985+0000 7f5660200640 1 -- 172.21.15.16:0/986485695 >> 172.21.15.16:0/986485695 conn(0x7f5658067930 msgr2=0x7f56580b3ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:04:04.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:04.985+0000 7f5660200640 1 -- 172.21.15.16:0/986485695 shutdown_connections 2024-03-31T15:04:04.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:04.989+0000 7f5660200640 1 -- 172.21.15.16:0/986485695 wait complete. 2024-03-31T15:04:04.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:04.989+0000 7f5660200640 1 Processor -- start 2024-03-31T15:04:04.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:04.989+0000 7f5660200640 1 -- start start 2024-03-31T15:04:04.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:04.989+0000 7f5660200640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f565806f920 0x7f565813e660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:04.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:04.989+0000 7f5660200640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f56580cef80 0x7f565815a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:04.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:04.989+0000 7f5660200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f56580d0a60 0x7f5658167780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:04.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:04.989+0000 7f5660200640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f565816a240 con 0x7f56580d0a60 2024-03-31T15:04:04.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:04.989+0000 7f5660200640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f565816a0c0 con 0x7f565806f920 2024-03-31T15:04:04.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:04.989+0000 7f5660200640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f565816a3c0 con 0x7f56580cef80 2024-03-31T15:04:04.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:04.989+0000 7f565e200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f56580d0a60 0x7f5658167780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:04.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:04.989+0000 7f565d800640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f565806f920 0x7f565813e660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:04.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:04.989+0000 7f565e200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f56580d0a60 0x7f5658167780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60244/0 (socket says 172.21.15.16:60244) 2024-03-31T15:04:04.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:04.989+0000 7f565e200640 1 -- 172.21.15.16:0/1788406706 learned_addr learned my addr 172.21.15.16:0/1788406706 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:04:04.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:04.989+0000 7f565ce00640 1 --2- 172.21.15.16:0/1788406706 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f56580cef80 0x7f565815a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:04.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:04.989+0000 7f565e200640 1 -- 172.21.15.16:0/1788406706 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f56580cef80 msgr2=0x7f565815a9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:04.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:04.989+0000 7f565e200640 1 --2- 172.21.15.16:0/1788406706 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f56580cef80 0x7f565815a9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:04.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:04.989+0000 7f565e200640 1 -- 172.21.15.16:0/1788406706 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f565806f920 msgr2=0x7f565813e660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:04.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:04.989+0000 7f565e200640 1 --2- 172.21.15.16:0/1788406706 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f565806f920 0x7f565813e660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:04.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:04.989+0000 7f565e200640 1 -- 172.21.15.16:0/1788406706 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56580b37c0 con 0x7f56580d0a60 2024-03-31T15:04:04.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:04.989+0000 7f565e200640 1 --2- 172.21.15.16:0/1788406706 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f56580d0a60 0x7f5658167780 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f565000cb10 tx=0x7f565000a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:04.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:04.989+0000 7f5655600640 1 -- 172.21.15.16:0/1788406706 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5650017070 con 0x7f56580d0a60 2024-03-31T15:04:04.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:04.989+0000 7f5655600640 1 -- 172.21.15.16:0/1788406706 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f565000ac70 con 0x7f56580d0a60 2024-03-31T15:04:04.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:04.989+0000 7f5655600640 1 -- 172.21.15.16:0/1788406706 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5650012620 con 0x7f56580d0a60 2024-03-31T15:04:04.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:04.989+0000 7f5660200640 1 -- 172.21.15.16:0/1788406706 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f56580b2f20 con 0x7f56580d0a60 2024-03-31T15:04:04.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:04.989+0000 7f5660200640 1 -- 172.21.15.16:0/1788406706 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f56580b34e0 con 0x7f56580d0a60 2024-03-31T15:04:04.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:04.993+0000 7f5660200640 1 -- 172.21.15.16:0/1788406706 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f565806d7f0 con 0x7f56580d0a60 2024-03-31T15:04:04.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:04.993+0000 7f5655600640 1 -- 172.21.15.16:0/1788406706 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f565001a070 con 0x7f56580d0a60 2024-03-31T15:04:04.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:04.993+0000 7f5655600640 1 --2- 172.21.15.16:0/1788406706 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f56380683b0 0x7f563806a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:04.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:04.993+0000 7f5655600640 1 -- 172.21.15.16:0/1788406706 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f5650086620 con 0x7f56580d0a60 2024-03-31T15:04:04.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:04.993+0000 7f565d800640 1 --2- 172.21.15.16:0/1788406706 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f56380683b0 0x7f563806a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:04.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:04.993+0000 7f565d800640 1 --2- 172.21.15.16:0/1788406706 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f56380683b0 0x7f563806a870 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f564c007fd0 tx=0x7f564c007d60 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:04.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:04.997+0000 7f5655600640 1 -- 172.21.15.16:0/1788406706 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f565000ae20 con 0x7f56580d0a60 2024-03-31T15:04:05.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:05.161+0000 7f5660200640 1 -- 172.21.15.16:0/1788406706 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f5658069950 con 0x7f56580d0a60 2024-03-31T15:04:05.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:05.161+0000 7f5655600640 1 -- 172.21.15.16:0/1788406706 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) ==== 100+0+11 (secure 0 0 0) 0x7f5650056590 con 0x7f56580d0a60 2024-03-31T15:04:05.155 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 64 2024-03-31T15:04:05.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:05.165+0000 7f5660200640 1 -- 172.21.15.16:0/1788406706 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f56380683b0 msgr2=0x7f563806a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:05.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:05.165+0000 7f5660200640 1 --2- 172.21.15.16:0/1788406706 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f56380683b0 0x7f563806a870 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f564c007fd0 tx=0x7f564c007d60 comp rx=0 tx=0).stop 2024-03-31T15:04:05.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:05.165+0000 7f5660200640 1 -- 172.21.15.16:0/1788406706 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f56580d0a60 msgr2=0x7f5658167780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:05.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:05.165+0000 7f5660200640 1 --2- 172.21.15.16:0/1788406706 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f56580d0a60 0x7f5658167780 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f565000cb10 tx=0x7f565000a060 comp rx=0 tx=0).stop 2024-03-31T15:04:05.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:05.165+0000 7f5660200640 1 -- 172.21.15.16:0/1788406706 shutdown_connections 2024-03-31T15:04:05.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:05.165+0000 7f5660200640 1 --2- 172.21.15.16:0/1788406706 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f56380683b0 0x7f563806a870 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:05.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:05.165+0000 7f5660200640 1 --2- 172.21.15.16:0/1788406706 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f56580d0a60 0x7f5658167780 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:05.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:05.165+0000 7f5660200640 1 --2- 172.21.15.16:0/1788406706 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f56580cef80 0x7f565815a9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:05.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:05.165+0000 7f5660200640 1 --2- 172.21.15.16:0/1788406706 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f565806f920 0x7f565813e660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:05.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:05.165+0000 7f5660200640 1 -- 172.21.15.16:0/1788406706 >> 172.21.15.16:0/1788406706 conn(0x7f5658067930 msgr2=0x7f56581553b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:04:05.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:05.165+0000 7f5660200640 1 -- 172.21.15.16:0/1788406706 shutdown_connections 2024-03-31T15:04:05.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:05.165+0000 7f5660200640 1 -- 172.21.15.16:0/1788406706 wait complete. 2024-03-31T15:04:05.177 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-31T15:04:05.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:05.321+0000 7f50daa00640 1 Processor -- start 2024-03-31T15:04:05.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:05.321+0000 7f50daa00640 1 -- start start 2024-03-31T15:04:05.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:05.325+0000 7f50daa00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f50d4145d10 0x7f50d40cf790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:05.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:05.325+0000 7f50daa00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f50d40b23d0 0x7f50d415cd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:05.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:05.325+0000 7f50daa00640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f50d4160e70 con 0x7f50d40728b0 2024-03-31T15:04:05.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:05.325+0000 7f50daa00640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f50d41602f0 con 0x7f50d4145d10 2024-03-31T15:04:05.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:05.325+0000 7f50daa00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f50d41605d0 con 0x7f50d40b23d0 2024-03-31T15:04:05.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:05.325+0000 7f50d3e00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f50d4145d10 0x7f50d40cf790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:05.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:05.325+0000 7f50d3e00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f50d4145d10 0x7f50d40cf790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60248/0 (socket says 172.21.15.16:60248) 2024-03-31T15:04:05.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:05.325+0000 7f50d8a00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f50d40b23d0 0x7f50d415cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:05.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:05.325+0000 7f50d3e00640 1 -- 172.21.15.16:0/1587934095 learned_addr learned my addr 172.21.15.16:0/1587934095 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:04:05.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:05.325+0000 7f50d8a00640 1 -- 172.21.15.16:0/1587934095 >> v2:172.21.15.16:3300/0 conn(0x7f50d4145d10 msgr2=0x7f50d40cf790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:04:05.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:05.325+0000 7f50d8a00640 1 --2- 172.21.15.16:0/1587934095 >> v2:172.21.15.16:3300/0 conn(0x7f50d4145d10 0x7f50d40cf790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:05.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:05.325+0000 7f50d8a00640 1 -- 172.21.15.16:0/1587934095 >> v1:172.21.15.89:6789/0 conn(0x7f50d40728b0 legacy=0x7f50d40b1cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:05.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:05.325+0000 7f50d8a00640 1 -- 172.21.15.16:0/1587934095 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50d40cfcd0 con 0x7f50d40b23d0 2024-03-31T15:04:05.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:05.329+0000 7f50d8a00640 1 --2- 172.21.15.16:0/1587934095 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f50d40b23d0 0x7f50d415cd50 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f50cc009d90 tx=0x7f50cc01c970 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7103c35e18114f57 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:05.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:05.329+0000 7f50d3400640 1 -- 172.21.15.16:0/1587934095 <== mon.2 v2:172.21.15.89:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50cc0b2070 con 0x7f50d40b23d0 2024-03-31T15:04:05.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:05.329+0000 7f50d3400640 1 -- 172.21.15.16:0/1587934095 <== mon.2 v2:172.21.15.89:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f50cc0a5830 con 0x7f50d40b23d0 2024-03-31T15:04:05.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:05.329+0000 7f50d3400640 1 -- 172.21.15.16:0/1587934095 <== mon.2 v2:172.21.15.89:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50cc0a5c00 con 0x7f50d40b23d0 2024-03-31T15:04:05.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:05.329+0000 7f50daa00640 1 -- 172.21.15.16:0/1587934095 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f50d40b23d0 msgr2=0x7f50d415cd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:05.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:05.329+0000 7f50daa00640 1 --2- 172.21.15.16:0/1587934095 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f50d40b23d0 0x7f50d415cd50 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f50cc009d90 tx=0x7f50cc01c970 comp rx=0 tx=0).stop 2024-03-31T15:04:05.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:05.329+0000 7f50daa00640 1 -- 172.21.15.16:0/1587934095 shutdown_connections 2024-03-31T15:04:05.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:05.329+0000 7f50daa00640 1 --2- 172.21.15.16:0/1587934095 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f50d40b23d0 0x7f50d415cd50 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:05.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:05.329+0000 7f50daa00640 1 --2- 172.21.15.16:0/1587934095 >> v2:172.21.15.16:3300/0 conn(0x7f50d4145d10 0x7f50d40cf790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:05.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:05.329+0000 7f50daa00640 1 -- 172.21.15.16:0/1587934095 >> 172.21.15.16:0/1587934095 conn(0x7f50d406b620 msgr2=0x7f50d406ba30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:04:05.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:05.329+0000 7f50daa00640 1 -- 172.21.15.16:0/1587934095 shutdown_connections 2024-03-31T15:04:05.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:05.329+0000 7f50daa00640 1 -- 172.21.15.16:0/1587934095 wait complete. 2024-03-31T15:04:05.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:05.329+0000 7f50daa00640 1 Processor -- start 2024-03-31T15:04:05.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:05.329+0000 7f50daa00640 1 -- start start 2024-03-31T15:04:05.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:05.333+0000 7f50daa00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f50d40728b0 0x7f50d413da50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:05.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:05.333+0000 7f50daa00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f50d40b23d0 0x7f50d4138b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:05.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:05.333+0000 7f50daa00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f50d4145d10 0x7f50d415db50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:05.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:05.333+0000 7f50daa00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f50d40cf4d0 con 0x7f50d4145d10 2024-03-31T15:04:05.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:05.333+0000 7f50daa00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f50d40cf350 con 0x7f50d40b23d0 2024-03-31T15:04:05.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:05.333+0000 7f50daa00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f50d4072cc0 con 0x7f50d40728b0 2024-03-31T15:04:05.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:05.333+0000 7f50d8a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f50d4145d10 0x7f50d415db50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:05.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:05.333+0000 7f50d8a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f50d4145d10 0x7f50d415db50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60250/0 (socket says 172.21.15.16:60250) 2024-03-31T15:04:05.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:05.333+0000 7f50d3e00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f50d40728b0 0x7f50d413da50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:05.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:05.333+0000 7f50d8a00640 1 -- 172.21.15.16:0/1413196068 learned_addr learned my addr 172.21.15.16:0/1413196068 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:04:05.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:05.333+0000 7f50cbe00640 1 --2- 172.21.15.16:0/1413196068 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f50d40b23d0 0x7f50d4138b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:05.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:05.333+0000 7f50d8a00640 1 -- 172.21.15.16:0/1413196068 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f50d40728b0 msgr2=0x7f50d413da50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:05.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:05.333+0000 7f50d8a00640 1 --2- 172.21.15.16:0/1413196068 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f50d40728b0 0x7f50d413da50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:05.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:05.333+0000 7f50d8a00640 1 -- 172.21.15.16:0/1413196068 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f50d40b23d0 msgr2=0x7f50d4138b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:05.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:05.333+0000 7f50d8a00640 1 --2- 172.21.15.16:0/1413196068 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f50d40b23d0 0x7f50d4138b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:05.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:05.333+0000 7f50d8a00640 1 -- 172.21.15.16:0/1413196068 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50d41390a0 con 0x7f50d4145d10 2024-03-31T15:04:05.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:05.333+0000 7f50d8a00640 1 --2- 172.21.15.16:0/1413196068 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f50d4145d10 0x7f50d415db50 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f50cc009d90 tx=0x7f50cc019710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:05.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:05.333+0000 7f50d0c00640 1 -- 172.21.15.16:0/1413196068 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50cc0b2050 con 0x7f50d4145d10 2024-03-31T15:04:05.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:05.333+0000 7f50d0c00640 1 -- 172.21.15.16:0/1413196068 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f50cc0be070 con 0x7f50d4145d10 2024-03-31T15:04:05.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:05.333+0000 7f50d0c00640 1 -- 172.21.15.16:0/1413196068 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50cc0ad890 con 0x7f50d4145d10 2024-03-31T15:04:05.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:05.333+0000 7f50daa00640 1 -- 172.21.15.16:0/1413196068 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f50d413c9b0 con 0x7f50d4145d10 2024-03-31T15:04:05.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:05.333+0000 7f50daa00640 1 -- 172.21.15.16:0/1413196068 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f50d413cf70 con 0x7f50d4145d10 2024-03-31T15:04:05.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:05.333+0000 7f50daa00640 1 -- 172.21.15.16:0/1413196068 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f50d4069950 con 0x7f50d4145d10 2024-03-31T15:04:05.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:05.333+0000 7f50d0c00640 1 -- 172.21.15.16:0/1413196068 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f50cc0ada30 con 0x7f50d4145d10 2024-03-31T15:04:05.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:05.337+0000 7f50d0c00640 1 --2- 172.21.15.16:0/1413196068 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f50b0068380 0x7f50b006a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:05.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:05.337+0000 7f50d0c00640 1 -- 172.21.15.16:0/1413196068 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f50cc121920 con 0x7f50d4145d10 2024-03-31T15:04:05.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:05.337+0000 7f50d3e00640 1 --2- 172.21.15.16:0/1413196068 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f50b0068380 0x7f50b006a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:05.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:05.337+0000 7f50d3e00640 1 --2- 172.21.15.16:0/1413196068 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f50b0068380 0x7f50b006a840 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f50d40d0ca0 tx=0x7f50c4009240 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:05.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:05.341+0000 7f50d0c00640 1 -- 172.21.15.16:0/1413196068 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f50cc016560 con 0x7f50d4145d10 2024-03-31T15:04:05.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:05.505+0000 7f50daa00640 1 -- 172.21.15.16:0/1413196068 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f50d40d0cd0 con 0x7f50d4145d10 2024-03-31T15:04:05.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:05.505+0000 7f50d0c00640 1 -- 172.21.15.16:0/1413196068 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) ==== 74+0+16533 (secure 0 0 0) 0x7f50cc0f1b00 con 0x7f50d4145d10 2024-03-31T15:04:05.498 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:04:05.499 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":21,"fsid":"286eb5da-fb47-4550-b290-d9519c13b9a5","created":"2024-03-31T15:03:11.438952+0000","modified":"2024-03-31T15:03:37.957829+0000","last_up_change":"2024-03-31T15:03:22.885535+0000","last_in_change":"2024-03-31T15:03:13.941667+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-31T15:03:22.550787+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-31T15:03:34.488425+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-31T15:03:35.299116+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"92a66450-ef19-40d8-ac3d-d760b23b0bda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6801","nonce":932712244}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6803","nonce":932712244}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6807","nonce":932712244}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6805","nonce":932712244}]},"public_addr":"172.21.15.16:6801/932712244","cluster_addr":"172.21.15.16:6803/932712244","heartbeat_back_addr":"172.21.15.16:6807/932712244","heartbeat_front_addr":"172.21.15.16:6805/932712244","state":["exists","up"]},{"osd":1,"uuid":"da9d1f2b-1968-4959-b843-faa231988e92","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6810","nonce":1398737533}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6811","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6813","nonce":1398737533}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6821","nonce":1398737533}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6815","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6816","nonce":1398737533}]},"public_addr":"172.21.15.16:6810/1398737533","cluster_addr":"172.21.15.16:6813/1398737533","heartbeat_back_addr":"172.21.15.16:6821/1398737533","heartbeat_front_addr":"172.21.15.16:6816/1398737533","state":["exists","up"]},{"osd":2,"uuid":"01a73662-9a21-4b1f-bcbb-42b9636a4638","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6819","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6822","nonce":3018526646}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6825","nonce":3018526646}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6831","nonce":3018526646}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6827","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6829","nonce":3018526646}]},"public_addr":"172.21.15.16:6822/3018526646","cluster_addr":"172.21.15.16:6825/3018526646","heartbeat_back_addr":"172.21.15.16:6831/3018526646","heartbeat_front_addr":"172.21.15.16:6829/3018526646","state":["exists","up"]},{"osd":3,"uuid":"c7771fd3-e18e-4500-ac38-23cf10ebcd75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6809","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6812","nonce":3360352974}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6817","nonce":3360352974}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6828","nonce":3360352974}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6823","nonce":3360352974}]},"public_addr":"172.21.15.16:6812/3360352974","cluster_addr":"172.21.15.16:6817/3360352974","heartbeat_back_addr":"172.21.15.16:6828/3360352974","heartbeat_front_addr":"172.21.15.16:6823/3360352974","state":["exists","up"]},{"osd":4,"uuid":"88260730-1482-4fbd-a40f-cb1124e9df4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6801","nonce":1856844856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6803","nonce":1856844856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6807","nonce":1856844856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6805","nonce":1856844856}]},"public_addr":"172.21.15.89:6801/1856844856","cluster_addr":"172.21.15.89:6803/1856844856","heartbeat_back_addr":"172.21.15.89:6807/1856844856","heartbeat_front_addr":"172.21.15.89:6805/1856844856","state":["exists","up"]},{"osd":5,"uuid":"4cb2c00c-fae0-4dd8-9d27-ec4f0093ccf7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6814","nonce":50395527}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6817","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6820","nonce":50395527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6831","nonce":50395527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6823","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6826","nonce":50395527}]},"public_addr":"172.21.15.89:6814/50395527","cluster_addr":"172.21.15.89:6820/50395527","heartbeat_back_addr":"172.21.15.89:6831/50395527","heartbeat_front_addr":"172.21.15.89:6826/50395527","state":["exists","up"]},{"osd":6,"uuid":"81ce0c19-2554-4cae-acbe-66a05e9e9b9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6809","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6812","nonce":3470471752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6815","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6818","nonce":3470471752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6827","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6829","nonce":3470471752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6821","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6824","nonce":3470471752}]},"public_addr":"172.21.15.89:6812/3470471752","cluster_addr":"172.21.15.89:6818/3470471752","heartbeat_back_addr":"172.21.15.89:6829/3470471752","heartbeat_front_addr":"172.21.15.89:6824/3470471752","state":["exists","up"]},{"osd":7,"uuid":"85e4313b-d37d-442e-8042-153bf20a847d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6811","nonce":1796778630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6813","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6816","nonce":1796778630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6825","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6828","nonce":1796778630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6819","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6822","nonce":1796778630}]},"public_addr":"172.21.15.89:6811/1796778630","cluster_addr":"172.21.15.89:6816/1796778630","heartbeat_back_addr":"172.21.15.89:6828/1796778630","heartbeat_front_addr":"172.21.15.89:6822/1796778630","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.439088+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.529970+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.680512+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:21.044161+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.582111+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.536558+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.409522+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.590229+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-31T15:04:05.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:05.505+0000 7f50daa00640 1 -- 172.21.15.16:0/1413196068 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f50b0068380 msgr2=0x7f50b006a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:05.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:05.505+0000 7f50daa00640 1 --2- 172.21.15.16:0/1413196068 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f50b0068380 0x7f50b006a840 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f50d40d0ca0 tx=0x7f50c4009240 comp rx=0 tx=0).stop 2024-03-31T15:04:05.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:05.505+0000 7f50daa00640 1 -- 172.21.15.16:0/1413196068 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f50d4145d10 msgr2=0x7f50d415db50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:05.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:05.505+0000 7f50daa00640 1 --2- 172.21.15.16:0/1413196068 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f50d4145d10 0x7f50d415db50 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f50cc009d90 tx=0x7f50cc019710 comp rx=0 tx=0).stop 2024-03-31T15:04:05.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:05.509+0000 7f50daa00640 1 -- 172.21.15.16:0/1413196068 shutdown_connections 2024-03-31T15:04:05.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:05.509+0000 7f50daa00640 1 --2- 172.21.15.16:0/1413196068 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f50b0068380 0x7f50b006a840 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:05.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:05.509+0000 7f50daa00640 1 --2- 172.21.15.16:0/1413196068 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f50d4145d10 0x7f50d415db50 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:05.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:05.509+0000 7f50daa00640 1 --2- 172.21.15.16:0/1413196068 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f50d40b23d0 0x7f50d4138b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:05.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:05.509+0000 7f50daa00640 1 --2- 172.21.15.16:0/1413196068 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f50d40728b0 0x7f50d413da50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:05.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:05.509+0000 7f50daa00640 1 -- 172.21.15.16:0/1413196068 >> 172.21.15.16:0/1413196068 conn(0x7f50d406b620 msgr2=0x7f50d40b3630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:04:05.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:05.509+0000 7f50daa00640 1 -- 172.21.15.16:0/1413196068 shutdown_connections 2024-03-31T15:04:05.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:05.509+0000 7f50daa00640 1 -- 172.21.15.16:0/1413196068 wait complete. 2024-03-31T15:04:05.520 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-31T15:03:22.550787+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-31T15:03:34.488425+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-31T15:03:35.299116+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-31T15:04:05.520 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-31T15:04:05.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:05.669+0000 7fbaabe00640 1 Processor -- start 2024-03-31T15:04:05.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:05.669+0000 7fbaabe00640 1 -- start start 2024-03-31T15:04:05.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:05.673+0000 7fbaabe00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fbaa413e3d0 0x7fbaa40cc9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:05.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:05.673+0000 7fbaabe00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fbaa406dbe0 0x7fbaa40ccf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:05.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:05.673+0000 7fbaabe00640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbaa415fbc0 con 0x7fbaa413dff0 2024-03-31T15:04:05.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:05.673+0000 7fbaabe00640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7fbaa40cd450 con 0x7fbaa413e3d0 2024-03-31T15:04:05.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:05.673+0000 7fbaabe00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fbaa415f730 con 0x7fbaa406dbe0 2024-03-31T15:04:05.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:05.673+0000 7fbaa8a00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fbaa413e3d0 0x7fbaa40cc9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:05.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:05.673+0000 7fbaa8a00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fbaa413e3d0 0x7fbaa40cc9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60266/0 (socket says 172.21.15.16:60266) 2024-03-31T15:04:05.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:05.673+0000 7fbaa8a00640 1 -- 172.21.15.16:0/2363131678 learned_addr learned my addr 172.21.15.16:0/2363131678 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:04:05.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:05.673+0000 7fbaa9e00640 1 --2- 172.21.15.16:0/2363131678 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fbaa406dbe0 0x7fbaa40ccf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:05.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:05.673+0000 7fbaa8a00640 1 -- 172.21.15.16:0/2363131678 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fbaa406dbe0 msgr2=0x7fbaa40ccf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:04:05.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:05.673+0000 7fbaa8a00640 1 --2- 172.21.15.16:0/2363131678 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fbaa406dbe0 0x7fbaa40ccf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:05.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:05.673+0000 7fbaa8a00640 1 -- 172.21.15.16:0/2363131678 >> v1:172.21.15.89:6789/0 conn(0x7fbaa413dff0 legacy=0x7fbaa406d4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:05.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:05.673+0000 7fbaa8a00640 1 -- 172.21.15.16:0/2363131678 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbaa415ee00 con 0x7fbaa413e3d0 2024-03-31T15:04:05.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:05.677+0000 7fbaa8a00640 1 --2- 172.21.15.16:0/2363131678 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbaa413e3d0 0x7fbaa40cc9d0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fba9000bb10 tx=0x7fba9001eba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f770162361be60b5 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:05.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:05.677+0000 7fbaa3e00640 1 -- 172.21.15.16:0/2363131678 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba900b4070 con 0x7fbaa413e3d0 2024-03-31T15:04:05.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:05.677+0000 7fbaa3e00640 1 -- 172.21.15.16:0/2363131678 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fba900a7a60 con 0x7fbaa413e3d0 2024-03-31T15:04:05.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:05.677+0000 7fbaa3e00640 1 -- 172.21.15.16:0/2363131678 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba900a7e30 con 0x7fbaa413e3d0 2024-03-31T15:04:05.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:05.677+0000 7fbaabe00640 1 -- 172.21.15.16:0/2363131678 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbaa413e3d0 msgr2=0x7fbaa40cc9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:05.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:05.677+0000 7fbaabe00640 1 --2- 172.21.15.16:0/2363131678 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbaa413e3d0 0x7fbaa40cc9d0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fba9000bb10 tx=0x7fba9001eba0 comp rx=0 tx=0).stop 2024-03-31T15:04:05.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:05.677+0000 7fbaabe00640 1 -- 172.21.15.16:0/2363131678 shutdown_connections 2024-03-31T15:04:05.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:05.677+0000 7fbaabe00640 1 --2- 172.21.15.16:0/2363131678 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fbaa406dbe0 0x7fbaa40ccf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:05.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:05.677+0000 7fbaabe00640 1 --2- 172.21.15.16:0/2363131678 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbaa413e3d0 0x7fbaa40cc9d0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:05.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:05.677+0000 7fbaabe00640 1 -- 172.21.15.16:0/2363131678 >> 172.21.15.16:0/2363131678 conn(0x7fbaa406ee40 msgr2=0x7fbaa406b620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:04:05.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:05.677+0000 7fbaabe00640 1 -- 172.21.15.16:0/2363131678 shutdown_connections 2024-03-31T15:04:05.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:05.677+0000 7fbaabe00640 1 -- 172.21.15.16:0/2363131678 wait complete. 2024-03-31T15:04:05.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:05.677+0000 7fbaabe00640 1 Processor -- start 2024-03-31T15:04:05.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:05.677+0000 7fbaabe00640 1 -- start start 2024-03-31T15:04:05.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:05.677+0000 7fbaabe00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbaa406dbe0 0x7fbaa415e480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:05.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:05.677+0000 7fbaabe00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbaa413dff0 0x7fbaa4156b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:05.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:05.677+0000 7fbaabe00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fbaa413e3d0 0x7fbaa4153c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:05.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:05.677+0000 7fbaabe00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fbaa40720f0 con 0x7fbaa413dff0 2024-03-31T15:04:05.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:05.677+0000 7fbaabe00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fbaa4071f70 con 0x7fbaa406dbe0 2024-03-31T15:04:05.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:05.677+0000 7fbaabe00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fbaa4072270 con 0x7fbaa413e3d0 2024-03-31T15:04:05.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:05.681+0000 7fbaa8a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbaa413dff0 0x7fbaa4156b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:05.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:05.681+0000 7fbaa8a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbaa413dff0 0x7fbaa4156b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60276/0 (socket says 172.21.15.16:60276) 2024-03-31T15:04:05.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:05.681+0000 7fbaa8a00640 1 -- 172.21.15.16:0/1407122043 learned_addr learned my addr 172.21.15.16:0/1407122043 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:04:05.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:05.681+0000 7fbaa9e00640 1 --2- 172.21.15.16:0/1407122043 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fbaa413e3d0 0x7fbaa4153c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:05.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:05.681+0000 7fbaa9400640 1 --2- 172.21.15.16:0/1407122043 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbaa406dbe0 0x7fbaa415e480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:05.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:05.681+0000 7fbaa8a00640 1 -- 172.21.15.16:0/1407122043 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fbaa413e3d0 msgr2=0x7fbaa4153c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:05.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:05.681+0000 7fbaa8a00640 1 --2- 172.21.15.16:0/1407122043 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fbaa413e3d0 0x7fbaa4153c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:05.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:05.681+0000 7fbaa8a00640 1 -- 172.21.15.16:0/1407122043 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbaa406dbe0 msgr2=0x7fbaa415e480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:05.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:05.681+0000 7fbaa8a00640 1 --2- 172.21.15.16:0/1407122043 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbaa406dbe0 0x7fbaa415e480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:05.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:05.681+0000 7fbaa8a00640 1 -- 172.21.15.16:0/1407122043 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbaa415e9c0 con 0x7fbaa413dff0 2024-03-31T15:04:05.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:05.681+0000 7fbaa8a00640 1 --2- 172.21.15.16:0/1407122043 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbaa413dff0 0x7fbaa4156b10 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fba9000b2c0 tx=0x7fba9001b940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:05.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:05.681+0000 7fbaa1600640 1 -- 172.21.15.16:0/1407122043 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba900b4070 con 0x7fbaa413dff0 2024-03-31T15:04:05.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:05.681+0000 7fbaa1600640 1 -- 172.21.15.16:0/1407122043 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fba900b6070 con 0x7fbaa413dff0 2024-03-31T15:04:05.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:05.681+0000 7fbaa1600640 1 -- 172.21.15.16:0/1407122043 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba900bb6b0 con 0x7fbaa413dff0 2024-03-31T15:04:05.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:05.681+0000 7fbaabe00640 1 -- 172.21.15.16:0/1407122043 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbaa4154b50 con 0x7fbaa413dff0 2024-03-31T15:04:05.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:05.681+0000 7fbaabe00640 1 -- 172.21.15.16:0/1407122043 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbaa4155110 con 0x7fbaa413dff0 2024-03-31T15:04:05.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:05.681+0000 7fbaabe00640 1 -- 172.21.15.16:0/1407122043 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbaa4069950 con 0x7fbaa413dff0 2024-03-31T15:04:05.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:05.681+0000 7fbaa1600640 1 -- 172.21.15.16:0/1407122043 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fba900a7920 con 0x7fbaa413dff0 2024-03-31T15:04:05.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:05.685+0000 7fbaa1600640 1 --2- 172.21.15.16:0/1407122043 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fba840683b0 0x7fba8406a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:05.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:05.685+0000 7fbaa1600640 1 -- 172.21.15.16:0/1407122043 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fba90120ed0 con 0x7fbaa413dff0 2024-03-31T15:04:05.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:05.685+0000 7fbaa9400640 1 --2- 172.21.15.16:0/1407122043 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fba840683b0 0x7fba8406a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:05.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:05.685+0000 7fbaa9400640 1 --2- 172.21.15.16:0/1407122043 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fba840683b0 0x7fba8406a870 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fba98001810 tx=0x7fba98009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:05.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:05.689+0000 7fbaa1600640 1 -- 172.21.15.16:0/1407122043 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fba90018820 con 0x7fbaa413dff0 2024-03-31T15:04:05.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:05.853+0000 7fbaabe00640 1 -- 172.21.15.16:0/1407122043 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fbaa406e4e0 con 0x7fbaa413dff0 2024-03-31T15:04:05.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:05.853+0000 7fbaa1600640 1 -- 172.21.15.16:0/1407122043 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) ==== 93+0+10 (secure 0 0 0) 0x7fba900f0e30 con 0x7fbaa413dff0 2024-03-31T15:04:05.848 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2024-03-31T15:04:05.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:05.857+0000 7fbaabe00640 1 -- 172.21.15.16:0/1407122043 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fba840683b0 msgr2=0x7fba8406a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:05.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:05.857+0000 7fbaabe00640 1 --2- 172.21.15.16:0/1407122043 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fba840683b0 0x7fba8406a870 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fba98001810 tx=0x7fba98009040 comp rx=0 tx=0).stop 2024-03-31T15:04:05.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:05.857+0000 7fbaabe00640 1 -- 172.21.15.16:0/1407122043 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbaa413dff0 msgr2=0x7fbaa4156b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:05.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:05.857+0000 7fbaabe00640 1 --2- 172.21.15.16:0/1407122043 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbaa413dff0 0x7fbaa4156b10 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fba9000b2c0 tx=0x7fba9001b940 comp rx=0 tx=0).stop 2024-03-31T15:04:05.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:05.857+0000 7fbaabe00640 1 -- 172.21.15.16:0/1407122043 shutdown_connections 2024-03-31T15:04:05.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:05.857+0000 7fbaabe00640 1 --2- 172.21.15.16:0/1407122043 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fba840683b0 0x7fba8406a870 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:05.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:05.857+0000 7fbaabe00640 1 --2- 172.21.15.16:0/1407122043 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fbaa413e3d0 0x7fbaa4153c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:05.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:05.857+0000 7fbaabe00640 1 --2- 172.21.15.16:0/1407122043 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbaa413dff0 0x7fbaa4156b10 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:05.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:05.857+0000 7fbaabe00640 1 --2- 172.21.15.16:0/1407122043 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbaa406dbe0 0x7fbaa415e480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:05.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:05.857+0000 7fbaabe00640 1 -- 172.21.15.16:0/1407122043 >> 172.21.15.16:0/1407122043 conn(0x7fbaa406ee40 msgr2=0x7fbaa40b1e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:04:05.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:05.857+0000 7fbaabe00640 1 -- 172.21.15.16:0/1407122043 shutdown_connections 2024-03-31T15:04:05.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:05.857+0000 7fbaabe00640 1 -- 172.21.15.16:0/1407122043 wait complete. 2024-03-31T15:04:05.869 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-31T15:04:06.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:06.021+0000 7f0281e00640 1 Processor -- start 2024-03-31T15:04:06.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:06.021+0000 7f0281e00640 1 -- start start 2024-03-31T15:04:06.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:06.021+0000 7f0281e00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f027c06f920 0x7f027c06fd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:06.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:06.021+0000 7f0281e00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f027c0d0a60 0x7f027c0cea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:06.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:06.021+0000 7f0281e00640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f027c070240 con 0x7f027c0cef80 2024-03-31T15:04:06.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:06.021+0000 7f0281e00640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f027c13f670 con 0x7f027c06f920 2024-03-31T15:04:06.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:06.021+0000 7f0281e00640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f027c15ca70 con 0x7f027c0d0a60 2024-03-31T15:04:06.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:06.021+0000 7f027aa00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f027c06f920 0x7f027c06fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:06.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:06.021+0000 7f027aa00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f027c06f920 0x7f027c06fd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60292/0 (socket says 172.21.15.16:60292) 2024-03-31T15:04:06.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:06.021+0000 7f027aa00640 1 -- 172.21.15.16:0/2231974955 learned_addr learned my addr 172.21.15.16:0/2231974955 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:04:06.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:06.021+0000 7f027a000640 1 --2- 172.21.15.16:0/2231974955 >> v2:172.21.15.89:3300/0 conn(0x7f027c0d0a60 0x7f027c0cea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:06.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:06.021+0000 7f027aa00640 1 -- 172.21.15.16:0/2231974955 >> v2:172.21.15.89:3300/0 conn(0x7f027c0d0a60 msgr2=0x7f027c0cea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:04:06.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:06.021+0000 7f027aa00640 1 --2- 172.21.15.16:0/2231974955 >> v2:172.21.15.89:3300/0 conn(0x7f027c0d0a60 0x7f027c0cea40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:06.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:06.021+0000 7f027aa00640 1 -- 172.21.15.16:0/2231974955 >> v1:172.21.15.89:6789/0 conn(0x7f027c0cef80 legacy=0x7f027c070e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:06.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:06.021+0000 7f027aa00640 1 -- 172.21.15.16:0/2231974955 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f027c15cd50 con 0x7f027c06f920 2024-03-31T15:04:06.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:06.025+0000 7f027aa00640 1 --2- 172.21.15.16:0/2231974955 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f027c06f920 0x7f027c06fd00 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f027c0736b0 tx=0x7f026c01c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a120d2b75f53405d server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:06.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:06.025+0000 7f0279600640 1 -- 172.21.15.16:0/2231974955 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f026c0b0070 con 0x7f027c06f920 2024-03-31T15:04:06.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:06.025+0000 7f0279600640 1 -- 172.21.15.16:0/2231974955 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f026c0ab600 con 0x7f027c06f920 2024-03-31T15:04:06.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:06.025+0000 7f0279600640 1 -- 172.21.15.16:0/2231974955 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f026c0ab9d0 con 0x7f027c06f920 2024-03-31T15:04:06.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:06.025+0000 7f0281e00640 1 -- 172.21.15.16:0/2231974955 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f027c06f920 msgr2=0x7f027c06fd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:06.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:06.025+0000 7f0281e00640 1 --2- 172.21.15.16:0/2231974955 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f027c06f920 0x7f027c06fd00 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f027c0736b0 tx=0x7f026c01c490 comp rx=0 tx=0).stop 2024-03-31T15:04:06.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:06.025+0000 7f0281e00640 1 -- 172.21.15.16:0/2231974955 shutdown_connections 2024-03-31T15:04:06.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:06.025+0000 7f0281e00640 1 --2- 172.21.15.16:0/2231974955 >> v2:172.21.15.89:3300/0 conn(0x7f027c0d0a60 0x7f027c0cea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:06.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:06.025+0000 7f0281e00640 1 --2- 172.21.15.16:0/2231974955 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f027c06f920 0x7f027c06fd00 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:06.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:06.025+0000 7f0281e00640 1 -- 172.21.15.16:0/2231974955 >> 172.21.15.16:0/2231974955 conn(0x7f027c067930 msgr2=0x7f027c0b3ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:04:06.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:06.025+0000 7f0281e00640 1 -- 172.21.15.16:0/2231974955 shutdown_connections 2024-03-31T15:04:06.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:06.025+0000 7f0281e00640 1 -- 172.21.15.16:0/2231974955 wait complete. 2024-03-31T15:04:06.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:06.025+0000 7f0281e00640 1 Processor -- start 2024-03-31T15:04:06.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:06.029+0000 7f0281e00640 1 -- start start 2024-03-31T15:04:06.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:06.029+0000 7f0281e00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f027c06f920 0x7f027c13edd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:06.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:06.029+0000 7f0281e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f027c0cef80 0x7f027c15e160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:06.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:06.029+0000 7f0281e00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f027c0d0a60 0x7f027c0b29b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:06.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:06.029+0000 7f0281e00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f027c1690b0 con 0x7f027c0cef80 2024-03-31T15:04:06.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:06.029+0000 7f0281e00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f027c0cf390 con 0x7f027c0d0a60 2024-03-31T15:04:06.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:06.029+0000 7f0281e00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f027c169230 con 0x7f027c06f920 2024-03-31T15:04:06.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:06.029+0000 7f027a000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f027c0cef80 0x7f027c15e160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:06.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:06.029+0000 7f027a000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f027c0cef80 0x7f027c15e160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60298/0 (socket says 172.21.15.16:60298) 2024-03-31T15:04:06.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:06.029+0000 7f027a000640 1 -- 172.21.15.16:0/3586168760 learned_addr learned my addr 172.21.15.16:0/3586168760 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:04:06.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:06.029+0000 7f027aa00640 1 --2- 172.21.15.16:0/3586168760 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f027c06f920 0x7f027c13edd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:06.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:06.029+0000 7f027b400640 1 --2- 172.21.15.16:0/3586168760 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f027c0d0a60 0x7f027c0b29b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:06.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:06.029+0000 7f027a000640 1 -- 172.21.15.16:0/3586168760 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f027c06f920 msgr2=0x7f027c13edd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:06.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:06.029+0000 7f027a000640 1 --2- 172.21.15.16:0/3586168760 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f027c06f920 0x7f027c13edd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:06.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:06.029+0000 7f027a000640 1 -- 172.21.15.16:0/3586168760 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f027c0d0a60 msgr2=0x7f027c0b29b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:06.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:06.029+0000 7f027a000640 1 --2- 172.21.15.16:0/3586168760 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f027c0d0a60 0x7f027c0b29b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:06.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:06.029+0000 7f027a000640 1 -- 172.21.15.16:0/3586168760 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f027c155ea0 con 0x7f027c0cef80 2024-03-31T15:04:06.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:06.029+0000 7f027aa00640 1 --2- 172.21.15.16:0/3586168760 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f027c06f920 0x7f027c13edd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:04:06.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:06.029+0000 7f027a000640 1 --2- 172.21.15.16:0/3586168760 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f027c0cef80 0x7f027c15e160 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f026800e840 tx=0x7f026800ec40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:06.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:06.029+0000 7f0273400640 1 -- 172.21.15.16:0/3586168760 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0268017070 con 0x7f027c0cef80 2024-03-31T15:04:06.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:06.029+0000 7f0273400640 1 -- 172.21.15.16:0/3586168760 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f026800ee90 con 0x7f027c0cef80 2024-03-31T15:04:06.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:06.029+0000 7f0281e00640 1 -- 172.21.15.16:0/3586168760 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f027c156180 con 0x7f027c0cef80 2024-03-31T15:04:06.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:06.029+0000 7f0273400640 1 -- 172.21.15.16:0/3586168760 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f026800cb80 con 0x7f027c0cef80 2024-03-31T15:04:06.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:06.029+0000 7f0281e00640 1 -- 172.21.15.16:0/3586168760 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f027c156740 con 0x7f027c0cef80 2024-03-31T15:04:06.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:06.029+0000 7f0281e00640 1 -- 172.21.15.16:0/3586168760 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f027c069950 con 0x7f027c0cef80 2024-03-31T15:04:06.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:06.033+0000 7f0273400640 1 -- 172.21.15.16:0/3586168760 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f026801a070 con 0x7f027c0cef80 2024-03-31T15:04:06.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:06.033+0000 7f0273400640 1 --2- 172.21.15.16:0/3586168760 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f025c0683b0 0x7f025c06a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:06.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:06.033+0000 7f0273400640 1 -- 172.21.15.16:0/3586168760 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f0268086be0 con 0x7f027c0cef80 2024-03-31T15:04:06.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:06.033+0000 7f027aa00640 1 --2- 172.21.15.16:0/3586168760 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f025c0683b0 0x7f025c06a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:06.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:06.033+0000 7f027aa00640 1 --2- 172.21.15.16:0/3586168760 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f025c0683b0 0x7f025c06a870 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f026c0ad000 tx=0x7f026c008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:06.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:06.037+0000 7f0273400640 1 -- 172.21.15.16:0/3586168760 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f0268056ab0 con 0x7f027c0cef80 2024-03-31T15:04:06.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:06.201+0000 7f0281e00640 1 -- 172.21.15.16:0/3586168760 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f027c0b4a20 con 0x7f027c0cef80 2024-03-31T15:04:06.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:06.201+0000 7f0273400640 1 -- 172.21.15.16:0/3586168760 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) ==== 104+0+11 (secure 0 0 0) 0x7f026805adb0 con 0x7f027c0cef80 2024-03-31T15:04:06.195 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2024-03-31T15:04:06.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:06.201+0000 7f0281e00640 1 -- 172.21.15.16:0/3586168760 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f025c0683b0 msgr2=0x7f025c06a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:06.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:06.201+0000 7f0281e00640 1 --2- 172.21.15.16:0/3586168760 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f025c0683b0 0x7f025c06a870 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f026c0ad000 tx=0x7f026c008040 comp rx=0 tx=0).stop 2024-03-31T15:04:06.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:06.201+0000 7f0281e00640 1 -- 172.21.15.16:0/3586168760 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f027c0cef80 msgr2=0x7f027c15e160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:06.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:06.201+0000 7f0281e00640 1 --2- 172.21.15.16:0/3586168760 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f027c0cef80 0x7f027c15e160 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f026800e840 tx=0x7f026800ec40 comp rx=0 tx=0).stop 2024-03-31T15:04:06.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:06.201+0000 7f0281e00640 1 -- 172.21.15.16:0/3586168760 shutdown_connections 2024-03-31T15:04:06.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:06.201+0000 7f0281e00640 1 --2- 172.21.15.16:0/3586168760 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f025c0683b0 0x7f025c06a870 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:06.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:06.201+0000 7f0281e00640 1 --2- 172.21.15.16:0/3586168760 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f027c0d0a60 0x7f027c0b29b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:06.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:06.201+0000 7f0281e00640 1 --2- 172.21.15.16:0/3586168760 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f027c0cef80 0x7f027c15e160 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:06.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:06.201+0000 7f0281e00640 1 --2- 172.21.15.16:0/3586168760 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f027c06f920 0x7f027c13edd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:06.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:06.201+0000 7f0281e00640 1 -- 172.21.15.16:0/3586168760 >> 172.21.15.16:0/3586168760 conn(0x7f027c067930 msgr2=0x7f027c15a800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:04:06.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:06.201+0000 7f0281e00640 1 -- 172.21.15.16:0/3586168760 shutdown_connections 2024-03-31T15:04:06.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:06.201+0000 7f0281e00640 1 -- 172.21.15.16:0/3586168760 wait complete. 2024-03-31T15:04:06.219 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-31T15:04:06.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:06.369+0000 7fe35a200640 1 Processor -- start 2024-03-31T15:04:06.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:06.369+0000 7fe35a200640 1 -- start start 2024-03-31T15:04:06.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:06.369+0000 7fe35a200640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fe35413dff0 0x7fe35406d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:06.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:06.369+0000 7fe35a200640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fe35413e3d0 0x7fe35406da10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:06.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:06.369+0000 7fe35a200640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe35415fbc0 con 0x7fe3540cc9d0 2024-03-31T15:04:06.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:06.369+0000 7fe35a200640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7fe35415f730 con 0x7fe35413dff0 2024-03-31T15:04:06.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:06.369+0000 7fe35a200640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fe35415ee00 con 0x7fe35413e3d0 2024-03-31T15:04:06.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:06.369+0000 7fe353400640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fe35413dff0 0x7fe35406d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:06.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:06.369+0000 7fe34be00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fe35413e3d0 0x7fe35406da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:06.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:06.369+0000 7fe353e00640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fe3540cc9d0 0x7fe3540cce30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.16:38356/0 (socket says 172.21.15.16:38356) 2024-03-31T15:04:06.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:06.369+0000 7fe353e00640 1 -- 172.21.15.16:0/1624183323 learned_addr learned my addr 172.21.15.16:0/1624183323 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:04:06.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:06.373+0000 7fe352a00640 1 -- 172.21.15.16:0/1624183323 <== mon.1 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1805014467 0 0) 0x7fe35415fbc0 con 0x7fe3540cc9d0 2024-03-31T15:04:06.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:06.373+0000 7fe34be00640 1 -- 172.21.15.16:0/1624183323 >> v2:172.21.15.89:3300/0 conn(0x7fe35413dff0 msgr2=0x7fe35406d4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:04:06.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:06.373+0000 7fe34be00640 1 --2- 172.21.15.16:0/1624183323 >> v2:172.21.15.89:3300/0 conn(0x7fe35413dff0 0x7fe35406d4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:06.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:06.373+0000 7fe34be00640 1 -- 172.21.15.16:0/1624183323 >> v1:172.21.15.89:6789/0 conn(0x7fe3540cc9d0 legacy=0x7fe3540cce30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:06.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:06.373+0000 7fe34be00640 1 -- 172.21.15.16:0/1624183323 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe35415f0e0 con 0x7fe35413e3d0 2024-03-31T15:04:06.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:06.373+0000 7fe353400640 1 --2- 172.21.15.16:0/1624183323 >> v2:172.21.15.89:3300/0 conn(0x7fe35413dff0 0x7fe35406d4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:04:06.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:06.373+0000 7fe34be00640 1 --2- 172.21.15.16:0/1624183323 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fe35413e3d0 0x7fe35406da10 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fe33c0097c0 tx=0x7fe33c01c410 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ac288109fdc028ac server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:06.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:06.373+0000 7fe352a00640 1 -- 172.21.15.16:0/1624183323 <== mon.2 v2:172.21.15.89:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe33c0b0070 con 0x7fe35413e3d0 2024-03-31T15:04:06.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:06.373+0000 7fe352a00640 1 -- 172.21.15.16:0/1624183323 <== mon.2 v2:172.21.15.89:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe33c0ab600 con 0x7fe35413e3d0 2024-03-31T15:04:06.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:06.373+0000 7fe352a00640 1 -- 172.21.15.16:0/1624183323 <== mon.2 v2:172.21.15.89:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe33c0ab9d0 con 0x7fe35413e3d0 2024-03-31T15:04:06.368 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:06.373+0000 7fe35a200640 1 -- 172.21.15.16:0/1624183323 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fe35413e3d0 msgr2=0x7fe35406da10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:06.368 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:06.373+0000 7fe35a200640 1 --2- 172.21.15.16:0/1624183323 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fe35413e3d0 0x7fe35406da10 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fe33c0097c0 tx=0x7fe33c01c410 comp rx=0 tx=0).stop 2024-03-31T15:04:06.368 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:06.373+0000 7fe35a200640 1 -- 172.21.15.16:0/1624183323 shutdown_connections 2024-03-31T15:04:06.368 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:06.373+0000 7fe35a200640 1 --2- 172.21.15.16:0/1624183323 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fe35413e3d0 0x7fe35406da10 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:06.368 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:06.373+0000 7fe35a200640 1 --2- 172.21.15.16:0/1624183323 >> v2:172.21.15.89:3300/0 conn(0x7fe35413dff0 0x7fe35406d4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:06.369 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:06.373+0000 7fe35a200640 1 -- 172.21.15.16:0/1624183323 >> 172.21.15.16:0/1624183323 conn(0x7fe35406ee40 msgr2=0x7fe35406b620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:04:06.369 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:06.373+0000 7fe35a200640 1 -- 172.21.15.16:0/1624183323 shutdown_connections 2024-03-31T15:04:06.369 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:06.373+0000 7fe35a200640 1 -- 172.21.15.16:0/1624183323 wait complete. 2024-03-31T15:04:06.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:06.377+0000 7fe35a200640 1 Processor -- start 2024-03-31T15:04:06.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:06.377+0000 7fe35a200640 1 -- start start 2024-03-31T15:04:06.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:06.377+0000 7fe35a200640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fe3540cc9d0 0x7fe3540aeec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:06.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:06.377+0000 7fe35a200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe35413dff0 0x7fe354156050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:06.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:06.377+0000 7fe35a200640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe35413e3d0 0x7fe3540ad8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:06.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:06.377+0000 7fe35a200640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fe3540cf710 con 0x7fe35413dff0 2024-03-31T15:04:06.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:06.377+0000 7fe35a200640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fe3540cf590 con 0x7fe35413e3d0 2024-03-31T15:04:06.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:06.377+0000 7fe35a200640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fe3540cf890 con 0x7fe3540cc9d0 2024-03-31T15:04:06.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:06.377+0000 7fe34be00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe35413dff0 0x7fe354156050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:06.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:06.377+0000 7fe353e00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe35413e3d0 0x7fe3540ad8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:06.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:06.377+0000 7fe34be00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe35413dff0 0x7fe354156050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60314/0 (socket says 172.21.15.16:60314) 2024-03-31T15:04:06.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:06.377+0000 7fe34be00640 1 -- 172.21.15.16:0/3108107239 learned_addr learned my addr 172.21.15.16:0/3108107239 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:04:06.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:06.377+0000 7fe353400640 1 --2- 172.21.15.16:0/3108107239 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fe3540cc9d0 0x7fe3540aeec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:06.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:06.377+0000 7fe34be00640 1 -- 172.21.15.16:0/3108107239 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fe3540cc9d0 msgr2=0x7fe3540aeec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:06.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:06.377+0000 7fe34be00640 1 --2- 172.21.15.16:0/3108107239 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fe3540cc9d0 0x7fe3540aeec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:06.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:06.377+0000 7fe34be00640 1 -- 172.21.15.16:0/3108107239 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe35413e3d0 msgr2=0x7fe3540ad8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:06.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:06.377+0000 7fe34be00640 1 --2- 172.21.15.16:0/3108107239 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe35413e3d0 0x7fe3540ad8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:06.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:06.377+0000 7fe34be00640 1 -- 172.21.15.16:0/3108107239 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe354153470 con 0x7fe35413dff0 2024-03-31T15:04:06.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:06.377+0000 7fe34be00640 1 --2- 172.21.15.16:0/3108107239 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe35413dff0 0x7fe354156050 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fe33c01a040 tx=0x7fe33c0191b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:06.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:06.377+0000 7fe34b400640 1 -- 172.21.15.16:0/3108107239 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe33c0b8070 con 0x7fe35413dff0 2024-03-31T15:04:06.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:06.377+0000 7fe34b400640 1 -- 172.21.15.16:0/3108107239 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe33c0b4610 con 0x7fe35413dff0 2024-03-31T15:04:06.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:06.377+0000 7fe34b400640 1 -- 172.21.15.16:0/3108107239 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe33c0b0040 con 0x7fe35413dff0 2024-03-31T15:04:06.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:06.377+0000 7fe35a200640 1 -- 172.21.15.16:0/3108107239 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe354153710 con 0x7fe35413dff0 2024-03-31T15:04:06.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:06.377+0000 7fe35a200640 1 -- 172.21.15.16:0/3108107239 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe35415b3e0 con 0x7fe35413dff0 2024-03-31T15:04:06.375 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:06.381+0000 7fe35a200640 1 -- 172.21.15.16:0/3108107239 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe354069950 con 0x7fe35413dff0 2024-03-31T15:04:06.375 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:06.381+0000 7fe34b400640 1 -- 172.21.15.16:0/3108107239 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fe33c0bb070 con 0x7fe35413dff0 2024-03-31T15:04:06.375 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:06.381+0000 7fe34b400640 1 --2- 172.21.15.16:0/3108107239 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fe330068380 0x7fe33006a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:06.375 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:06.381+0000 7fe34b400640 1 -- 172.21.15.16:0/3108107239 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fe33c120690 con 0x7fe35413dff0 2024-03-31T15:04:06.375 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:06.381+0000 7fe353400640 1 --2- 172.21.15.16:0/3108107239 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fe330068380 0x7fe33006a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:06.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:06.381+0000 7fe353400640 1 --2- 172.21.15.16:0/3108107239 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fe330068380 0x7fe33006a840 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fe344009580 tx=0x7fe344009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:06.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:06.385+0000 7fe34b400640 1 -- 172.21.15.16:0/3108107239 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fe33c016090 con 0x7fe35413dff0 2024-03-31T15:04:06.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:06.553+0000 7fe35a200640 1 -- 172.21.15.16:0/3108107239 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fe354158960 con 0x7fe35413dff0 2024-03-31T15:04:06.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:06.553+0000 7fe34b400640 1 -- 172.21.15.16:0/3108107239 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) ==== 100+0+11 (secure 0 0 0) 0x7fe33c0f0690 con 0x7fe35413dff0 2024-03-31T15:04:06.547 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 64 2024-03-31T15:04:06.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:06.557+0000 7fe35a200640 1 -- 172.21.15.16:0/3108107239 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fe330068380 msgr2=0x7fe33006a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:06.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:06.557+0000 7fe35a200640 1 --2- 172.21.15.16:0/3108107239 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fe330068380 0x7fe33006a840 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fe344009580 tx=0x7fe344009190 comp rx=0 tx=0).stop 2024-03-31T15:04:06.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:06.557+0000 7fe35a200640 1 -- 172.21.15.16:0/3108107239 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe35413dff0 msgr2=0x7fe354156050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:06.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:06.557+0000 7fe35a200640 1 --2- 172.21.15.16:0/3108107239 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe35413dff0 0x7fe354156050 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fe33c01a040 tx=0x7fe33c0191b0 comp rx=0 tx=0).stop 2024-03-31T15:04:06.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:06.557+0000 7fe35a200640 1 -- 172.21.15.16:0/3108107239 shutdown_connections 2024-03-31T15:04:06.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:06.557+0000 7fe35a200640 1 --2- 172.21.15.16:0/3108107239 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fe330068380 0x7fe33006a840 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:06.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:06.557+0000 7fe35a200640 1 --2- 172.21.15.16:0/3108107239 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe35413e3d0 0x7fe3540ad8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:06.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:06.557+0000 7fe35a200640 1 --2- 172.21.15.16:0/3108107239 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe35413dff0 0x7fe354156050 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:06.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:06.557+0000 7fe35a200640 1 --2- 172.21.15.16:0/3108107239 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fe3540cc9d0 0x7fe3540aeec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:06.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:06.557+0000 7fe35a200640 1 -- 172.21.15.16:0/3108107239 >> 172.21.15.16:0/3108107239 conn(0x7fe35406ee40 msgr2=0x7fe3540b1e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:04:06.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:06.557+0000 7fe35a200640 1 -- 172.21.15.16:0/3108107239 shutdown_connections 2024-03-31T15:04:06.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:06.557+0000 7fe35a200640 1 -- 172.21.15.16:0/3108107239 wait complete. 2024-03-31T15:04:06.568 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-31T15:04:06.602 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-31T15:04:06.614 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-31T15:04:06.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:06.717+0000 7f3c93600640 1 Processor -- start 2024-03-31T15:04:06.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:06.717+0000 7f3c93600640 1 -- start start 2024-03-31T15:04:06.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:06.717+0000 7f3c93600640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f3c8c0d1280 0x7f3c8c0d1660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:06.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:06.717+0000 7f3c93600640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f3c8c0b9c60 0x7f3c8c0b5d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:06.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:06.717+0000 7f3c93600640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3c8c0d1ba0 con 0x7f3c8c0b6290 2024-03-31T15:04:06.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:06.717+0000 7f3c93600640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f3c8c0d26a0 con 0x7f3c8c0d1280 2024-03-31T15:04:06.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:06.717+0000 7f3c93600640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f3c8c0d3990 con 0x7f3c8c0b9c60 2024-03-31T15:04:06.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:06.717+0000 7f3c90c00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f3c8c0d1280 0x7f3c8c0d1660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:06.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:06.717+0000 7f3c90c00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f3c8c0d1280 0x7f3c8c0d1660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60324/0 (socket says 172.21.15.16:60324) 2024-03-31T15:04:06.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:06.717+0000 7f3c90c00640 1 -- 172.21.15.16:0/703862563 learned_addr learned my addr 172.21.15.16:0/703862563 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:04:06.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:06.717+0000 7f3c8be00640 1 --2- 172.21.15.16:0/703862563 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f3c8c0b9c60 0x7f3c8c0b5d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:06.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:06.717+0000 7f3c90c00640 1 -- 172.21.15.16:0/703862563 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f3c8c0b9c60 msgr2=0x7f3c8c0b5d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:04:06.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:06.717+0000 7f3c90c00640 1 --2- 172.21.15.16:0/703862563 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f3c8c0b9c60 0x7f3c8c0b5d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:06.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:06.717+0000 7f3c90c00640 1 -- 172.21.15.16:0/703862563 >> v1:172.21.15.89:6789/0 conn(0x7f3c8c0b6290 legacy=0x7f3c8c0b7470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:06.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:06.717+0000 7f3c90c00640 1 -- 172.21.15.16:0/703862563 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c8c0d28f0 con 0x7f3c8c0d1280 2024-03-31T15:04:06.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:06.721+0000 7f3c90c00640 1 --2- 172.21.15.16:0/703862563 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3c8c0d1280 0x7f3c8c0d1660 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f3c80009870 tx=0x7f3c8001c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=85fbd8d31ac2faf8 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:06.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:06.721+0000 7f3c8b400640 1 -- 172.21.15.16:0/703862563 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c800b1070 con 0x7f3c8c0d1280 2024-03-31T15:04:06.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:06.721+0000 7f3c8b400640 1 -- 172.21.15.16:0/703862563 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3c800ac620 con 0x7f3c8c0d1280 2024-03-31T15:04:06.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:06.721+0000 7f3c8b400640 1 -- 172.21.15.16:0/703862563 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c800ac9f0 con 0x7f3c8c0d1280 2024-03-31T15:04:06.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:06.721+0000 7f3c93600640 1 -- 172.21.15.16:0/703862563 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3c8c0d1280 msgr2=0x7f3c8c0d1660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:06.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:06.721+0000 7f3c93600640 1 --2- 172.21.15.16:0/703862563 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3c8c0d1280 0x7f3c8c0d1660 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f3c80009870 tx=0x7f3c8001c4d0 comp rx=0 tx=0).stop 2024-03-31T15:04:06.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:06.721+0000 7f3c93600640 1 -- 172.21.15.16:0/703862563 shutdown_connections 2024-03-31T15:04:06.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:06.721+0000 7f3c93600640 1 --2- 172.21.15.16:0/703862563 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f3c8c0b9c60 0x7f3c8c0b5d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:06.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:06.721+0000 7f3c93600640 1 --2- 172.21.15.16:0/703862563 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3c8c0d1280 0x7f3c8c0d1660 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:06.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:06.721+0000 7f3c93600640 1 -- 172.21.15.16:0/703862563 >> 172.21.15.16:0/703862563 conn(0x7f3c8c067730 msgr2=0x7f3c8c067b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:04:06.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:06.721+0000 7f3c93600640 1 -- 172.21.15.16:0/703862563 shutdown_connections 2024-03-31T15:04:06.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:06.721+0000 7f3c93600640 1 -- 172.21.15.16:0/703862563 wait complete. 2024-03-31T15:04:06.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:06.725+0000 7f3c93600640 1 Processor -- start 2024-03-31T15:04:06.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:06.725+0000 7f3c93600640 1 -- start start 2024-03-31T15:04:06.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:06.725+0000 7f3c93600640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3c8c0b6290 0x7f3c8c14e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:06.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:06.725+0000 7f3c93600640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f3c8c0b9c60 0x7f3c8c0cd460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:06.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:06.725+0000 7f3c93600640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3c8c0d1280 0x7f3c8c0c8960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:06.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:06.725+0000 7f3c93600640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f3c8c0b6aa0 con 0x7f3c8c0b6290 2024-03-31T15:04:06.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:06.725+0000 7f3c93600640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f3c8c161d80 con 0x7f3c8c0d1280 2024-03-31T15:04:06.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:06.725+0000 7f3c93600640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f3c8c0d3e90 con 0x7f3c8c0b9c60 2024-03-31T15:04:06.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:06.725+0000 7f3c90c00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3c8c0b6290 0x7f3c8c14e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:06.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:06.725+0000 7f3c90c00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3c8c0b6290 0x7f3c8c14e770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60336/0 (socket says 172.21.15.16:60336) 2024-03-31T15:04:06.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:06.725+0000 7f3c90c00640 1 -- 172.21.15.16:0/1167206184 learned_addr learned my addr 172.21.15.16:0/1167206184 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:04:06.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:06.725+0000 7f3c91600640 1 --2- 172.21.15.16:0/1167206184 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3c8c0d1280 0x7f3c8c0c8960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:06.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:06.725+0000 7f3c8be00640 1 --2- 172.21.15.16:0/1167206184 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f3c8c0b9c60 0x7f3c8c0cd460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:06.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:06.725+0000 7f3c90c00640 1 -- 172.21.15.16:0/1167206184 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f3c8c0b9c60 msgr2=0x7f3c8c0cd460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:06.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:06.725+0000 7f3c90c00640 1 --2- 172.21.15.16:0/1167206184 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f3c8c0b9c60 0x7f3c8c0cd460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:06.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:06.725+0000 7f3c90c00640 1 -- 172.21.15.16:0/1167206184 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3c8c0d1280 msgr2=0x7f3c8c0c8960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:06.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:06.725+0000 7f3c90c00640 1 --2- 172.21.15.16:0/1167206184 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3c8c0d1280 0x7f3c8c0c8960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:06.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:06.725+0000 7f3c90c00640 1 -- 172.21.15.16:0/1167206184 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c8c150600 con 0x7f3c8c0b6290 2024-03-31T15:04:06.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:06.725+0000 7f3c91600640 1 --2- 172.21.15.16:0/1167206184 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3c8c0d1280 0x7f3c8c0c8960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:04:06.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:06.725+0000 7f3c90c00640 1 --2- 172.21.15.16:0/1167206184 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3c8c0b6290 0x7f3c8c14e770 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f3c8001a100 tx=0x7f3c8001ad10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:06.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:06.725+0000 7f3c88c00640 1 -- 172.21.15.16:0/1167206184 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c800bb020 con 0x7f3c8c0b6290 2024-03-31T15:04:06.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:06.725+0000 7f3c88c00640 1 -- 172.21.15.16:0/1167206184 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3c800b7640 con 0x7f3c8c0b6290 2024-03-31T15:04:06.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:06.725+0000 7f3c88c00640 1 -- 172.21.15.16:0/1167206184 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c800b1040 con 0x7f3c8c0b6290 2024-03-31T15:04:06.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:06.725+0000 7f3c93600640 1 -- 172.21.15.16:0/1167206184 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3c8c0c97b0 con 0x7f3c8c0b6290 2024-03-31T15:04:06.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:06.725+0000 7f3c93600640 1 -- 172.21.15.16:0/1167206184 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3c8c0cc980 con 0x7f3c8c0b6290 2024-03-31T15:04:06.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:06.725+0000 7f3c93600640 1 -- 172.21.15.16:0/1167206184 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3c8c069950 con 0x7f3c8c0b6290 2024-03-31T15:04:06.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:06.729+0000 7f3c88c00640 1 -- 172.21.15.16:0/1167206184 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f3c800be020 con 0x7f3c8c0b6290 2024-03-31T15:04:06.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:06.729+0000 7f3c88c00640 1 --2- 172.21.15.16:0/1167206184 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f3c6c0683b0 0x7f3c6c06a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:06.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:06.729+0000 7f3c88c00640 1 -- 172.21.15.16:0/1167206184 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f3c80120b70 con 0x7f3c8c0b6290 2024-03-31T15:04:06.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:06.729+0000 7f3c8be00640 1 --2- 172.21.15.16:0/1167206184 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f3c6c0683b0 0x7f3c6c06a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:06.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:06.729+0000 7f3c8be00640 1 --2- 172.21.15.16:0/1167206184 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f3c6c0683b0 0x7f3c6c06a870 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f3c8c0b98f0 tx=0x7f3c78008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:06.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:06.733+0000 7f3c88c00640 1 -- 172.21.15.16:0/1167206184 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f3c800c5030 con 0x7f3c8c0b6290 2024-03-31T15:04:06.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:06.897+0000 7f3c93600640 1 -- 172.21.15.16:0/1167206184 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f3c8c0b6010 con 0x7f3c8c0b6290 2024-03-31T15:04:06.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:06.897+0000 7f3c88c00640 1 -- 172.21.15.16:0/1167206184 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) ==== 74+0+16533 (secure 0 0 0) 0x7f3c80016150 con 0x7f3c8c0b6290 2024-03-31T15:04:06.890 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:04:06.891 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":21,"fsid":"286eb5da-fb47-4550-b290-d9519c13b9a5","created":"2024-03-31T15:03:11.438952+0000","modified":"2024-03-31T15:03:37.957829+0000","last_up_change":"2024-03-31T15:03:22.885535+0000","last_in_change":"2024-03-31T15:03:13.941667+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-31T15:03:22.550787+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-31T15:03:34.488425+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-31T15:03:35.299116+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"92a66450-ef19-40d8-ac3d-d760b23b0bda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6801","nonce":932712244}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6803","nonce":932712244}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6807","nonce":932712244}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6805","nonce":932712244}]},"public_addr":"172.21.15.16:6801/932712244","cluster_addr":"172.21.15.16:6803/932712244","heartbeat_back_addr":"172.21.15.16:6807/932712244","heartbeat_front_addr":"172.21.15.16:6805/932712244","state":["exists","up"]},{"osd":1,"uuid":"da9d1f2b-1968-4959-b843-faa231988e92","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6810","nonce":1398737533}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6811","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6813","nonce":1398737533}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6821","nonce":1398737533}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6815","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6816","nonce":1398737533}]},"public_addr":"172.21.15.16:6810/1398737533","cluster_addr":"172.21.15.16:6813/1398737533","heartbeat_back_addr":"172.21.15.16:6821/1398737533","heartbeat_front_addr":"172.21.15.16:6816/1398737533","state":["exists","up"]},{"osd":2,"uuid":"01a73662-9a21-4b1f-bcbb-42b9636a4638","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6819","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6822","nonce":3018526646}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6825","nonce":3018526646}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6831","nonce":3018526646}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6827","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6829","nonce":3018526646}]},"public_addr":"172.21.15.16:6822/3018526646","cluster_addr":"172.21.15.16:6825/3018526646","heartbeat_back_addr":"172.21.15.16:6831/3018526646","heartbeat_front_addr":"172.21.15.16:6829/3018526646","state":["exists","up"]},{"osd":3,"uuid":"c7771fd3-e18e-4500-ac38-23cf10ebcd75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6809","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6812","nonce":3360352974}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6817","nonce":3360352974}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6828","nonce":3360352974}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6823","nonce":3360352974}]},"public_addr":"172.21.15.16:6812/3360352974","cluster_addr":"172.21.15.16:6817/3360352974","heartbeat_back_addr":"172.21.15.16:6828/3360352974","heartbeat_front_addr":"172.21.15.16:6823/3360352974","state":["exists","up"]},{"osd":4,"uuid":"88260730-1482-4fbd-a40f-cb1124e9df4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6801","nonce":1856844856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6803","nonce":1856844856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6807","nonce":1856844856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6805","nonce":1856844856}]},"public_addr":"172.21.15.89:6801/1856844856","cluster_addr":"172.21.15.89:6803/1856844856","heartbeat_back_addr":"172.21.15.89:6807/1856844856","heartbeat_front_addr":"172.21.15.89:6805/1856844856","state":["exists","up"]},{"osd":5,"uuid":"4cb2c00c-fae0-4dd8-9d27-ec4f0093ccf7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6814","nonce":50395527}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6817","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6820","nonce":50395527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6831","nonce":50395527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6823","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6826","nonce":50395527}]},"public_addr":"172.21.15.89:6814/50395527","cluster_addr":"172.21.15.89:6820/50395527","heartbeat_back_addr":"172.21.15.89:6831/50395527","heartbeat_front_addr":"172.21.15.89:6826/50395527","state":["exists","up"]},{"osd":6,"uuid":"81ce0c19-2554-4cae-acbe-66a05e9e9b9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6809","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6812","nonce":3470471752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6815","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6818","nonce":3470471752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6827","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6829","nonce":3470471752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6821","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6824","nonce":3470471752}]},"public_addr":"172.21.15.89:6812/3470471752","cluster_addr":"172.21.15.89:6818/3470471752","heartbeat_back_addr":"172.21.15.89:6829/3470471752","heartbeat_front_addr":"172.21.15.89:6824/3470471752","state":["exists","up"]},{"osd":7,"uuid":"85e4313b-d37d-442e-8042-153bf20a847d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6811","nonce":1796778630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6813","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6816","nonce":1796778630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6825","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6828","nonce":1796778630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6819","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6822","nonce":1796778630}]},"public_addr":"172.21.15.89:6811/1796778630","cluster_addr":"172.21.15.89:6816/1796778630","heartbeat_back_addr":"172.21.15.89:6828/1796778630","heartbeat_front_addr":"172.21.15.89:6822/1796778630","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.439088+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.529970+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.680512+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:21.044161+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.582111+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.536558+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.409522+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.590229+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-31T15:04:06.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:06.897+0000 7f3c93600640 1 -- 172.21.15.16:0/1167206184 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f3c6c0683b0 msgr2=0x7f3c6c06a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:06.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:06.897+0000 7f3c93600640 1 --2- 172.21.15.16:0/1167206184 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f3c6c0683b0 0x7f3c6c06a870 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f3c8c0b98f0 tx=0x7f3c78008040 comp rx=0 tx=0).stop 2024-03-31T15:04:06.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:06.897+0000 7f3c93600640 1 -- 172.21.15.16:0/1167206184 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3c8c0b6290 msgr2=0x7f3c8c14e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:06.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:06.897+0000 7f3c93600640 1 --2- 172.21.15.16:0/1167206184 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3c8c0b6290 0x7f3c8c14e770 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f3c8001a100 tx=0x7f3c8001ad10 comp rx=0 tx=0).stop 2024-03-31T15:04:06.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:06.897+0000 7f3c93600640 1 -- 172.21.15.16:0/1167206184 shutdown_connections 2024-03-31T15:04:06.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:06.897+0000 7f3c93600640 1 --2- 172.21.15.16:0/1167206184 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f3c6c0683b0 0x7f3c6c06a870 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:06.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:06.897+0000 7f3c93600640 1 --2- 172.21.15.16:0/1167206184 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3c8c0d1280 0x7f3c8c0c8960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:06.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:06.897+0000 7f3c93600640 1 --2- 172.21.15.16:0/1167206184 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f3c8c0b9c60 0x7f3c8c0cd460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:06.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:06.897+0000 7f3c93600640 1 --2- 172.21.15.16:0/1167206184 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3c8c0b6290 0x7f3c8c14e770 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:06.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:06.897+0000 7f3c93600640 1 -- 172.21.15.16:0/1167206184 >> 172.21.15.16:0/1167206184 conn(0x7f3c8c067730 msgr2=0x7f3c8c0b8390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:04:06.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:06.897+0000 7f3c93600640 1 -- 172.21.15.16:0/1167206184 shutdown_connections 2024-03-31T15:04:06.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:06.901+0000 7f3c93600640 1 -- 172.21.15.16:0/1167206184 wait complete. 2024-03-31T15:04:06.912 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-31T15:03:22.550787+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-31T15:03:34.488425+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-31T15:03:35.299116+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-31T15:04:06.912 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-31T15:04:07.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:07.105+0000 7f1307200640 1 Processor -- start 2024-03-31T15:04:07.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:07.105+0000 7f1307200640 1 -- start start 2024-03-31T15:04:07.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:07.109+0000 7f1307200640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f130013dff0 0x7f130006d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:07.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:07.109+0000 7f1307200640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f13000cd0e0 0x7f130006da10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:07.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:07.109+0000 7f1307200640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f130015fbc0 con 0x7f130013e3d0 2024-03-31T15:04:07.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:07.109+0000 7f1307200640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f130015f730 con 0x7f130013dff0 2024-03-31T15:04:07.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:07.109+0000 7f1307200640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f130015ee00 con 0x7f13000cd0e0 2024-03-31T15:04:07.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:07.109+0000 7f12ffe00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f130013dff0 0x7f130006d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:07.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:07.109+0000 7f12f7e00640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f130013e3d0 0x7f13000cc9d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:46570/0 (socket says 172.21.15.16:46570) 2024-03-31T15:04:07.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:07.109+0000 7f12f7e00640 1 -- 172.21.15.16:0/2551453025 learned_addr learned my addr 172.21.15.16:0/2551453025 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:04:07.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:07.109+0000 7f1305200640 1 --2- 172.21.15.16:0/2551453025 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f13000cd0e0 0x7f130006da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:07.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:07.109+0000 7f12ffe00640 1 -- 172.21.15.16:0/2551453025 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f13000cd0e0 msgr2=0x7f130006da10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:04:07.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:07.109+0000 7f12ffe00640 1 --2- 172.21.15.16:0/2551453025 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f13000cd0e0 0x7f130006da10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:07.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:07.109+0000 7f12ffe00640 1 -- 172.21.15.16:0/2551453025 >> v1:172.21.15.16:6789/0 conn(0x7f130013e3d0 legacy=0x7f13000cc9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:07.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:07.109+0000 7f12ffe00640 1 -- 172.21.15.16:0/2551453025 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f130015f0e0 con 0x7f130013dff0 2024-03-31T15:04:07.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:07.109+0000 7f12ff400640 1 -- 172.21.15.16:0/2551453025 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1632289605 0 0) 0x7f130015fbc0 con 0x7f130013e3d0 2024-03-31T15:04:07.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:07.113+0000 7f12ffe00640 1 --2- 172.21.15.16:0/2551453025 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f130013dff0 0x7f130006d4d0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f12f00097c0 tx=0x7f12f001c410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a8737aebc95b6fc server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:07.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:07.113+0000 7f12ff400640 1 -- 172.21.15.16:0/2551453025 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12f00b0070 con 0x7f130013dff0 2024-03-31T15:04:07.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:07.113+0000 7f12ff400640 1 -- 172.21.15.16:0/2551453025 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f12f00ab600 con 0x7f130013dff0 2024-03-31T15:04:07.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:07.113+0000 7f12ff400640 1 -- 172.21.15.16:0/2551453025 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12f00ab9d0 con 0x7f130013dff0 2024-03-31T15:04:07.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:07.113+0000 7f1307200640 1 -- 172.21.15.16:0/2551453025 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f130013dff0 msgr2=0x7f130006d4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:07.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:07.113+0000 7f1307200640 1 --2- 172.21.15.16:0/2551453025 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f130013dff0 0x7f130006d4d0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f12f00097c0 tx=0x7f12f001c410 comp rx=0 tx=0).stop 2024-03-31T15:04:07.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:07.113+0000 7f1307200640 1 -- 172.21.15.16:0/2551453025 shutdown_connections 2024-03-31T15:04:07.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:07.113+0000 7f1307200640 1 --2- 172.21.15.16:0/2551453025 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f13000cd0e0 0x7f130006da10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:07.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:07.113+0000 7f1307200640 1 --2- 172.21.15.16:0/2551453025 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f130013dff0 0x7f130006d4d0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:07.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:07.113+0000 7f1307200640 1 -- 172.21.15.16:0/2551453025 >> 172.21.15.16:0/2551453025 conn(0x7f130006ee40 msgr2=0x7f130006b620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:04:07.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:07.113+0000 7f1307200640 1 -- 172.21.15.16:0/2551453025 shutdown_connections 2024-03-31T15:04:07.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:07.113+0000 7f1307200640 1 -- 172.21.15.16:0/2551453025 wait complete. 2024-03-31T15:04:07.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:07.113+0000 7f1307200640 1 Processor -- start 2024-03-31T15:04:07.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:07.113+0000 7f1307200640 1 -- start start 2024-03-31T15:04:07.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:07.117+0000 7f1307200640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f13000cd0e0 0x7f130015d410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:07.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:07.117+0000 7f1307200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f130013dff0 0x7f130013c560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:07.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:07.117+0000 7f1307200640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f130013e3d0 0x7f1300154b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:07.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:07.117+0000 7f1307200640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f13000cf710 con 0x7f130013dff0 2024-03-31T15:04:07.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:07.117+0000 7f1307200640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f13000cf590 con 0x7f13000cd0e0 2024-03-31T15:04:07.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:07.117+0000 7f1307200640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f13000cf890 con 0x7f130013e3d0 2024-03-31T15:04:07.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:07.117+0000 7f12f7e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f130013dff0 0x7f130013c560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:07.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:07.117+0000 7f12f7e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f130013dff0 0x7f130013c560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60278/0 (socket says 172.21.15.16:60278) 2024-03-31T15:04:07.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:07.117+0000 7f12ffe00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f13000cd0e0 0x7f130015d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:07.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:07.117+0000 7f12f7e00640 1 -- 172.21.15.16:0/1570626403 learned_addr learned my addr 172.21.15.16:0/1570626403 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:04:07.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:07.117+0000 7f1305200640 1 --2- 172.21.15.16:0/1570626403 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f130013e3d0 0x7f1300154b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:07.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:07.117+0000 7f12f7e00640 1 -- 172.21.15.16:0/1570626403 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f130013e3d0 msgr2=0x7f1300154b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:07.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:07.117+0000 7f12f7e00640 1 --2- 172.21.15.16:0/1570626403 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f130013e3d0 0x7f1300154b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:07.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:07.117+0000 7f12f7e00640 1 -- 172.21.15.16:0/1570626403 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f13000cd0e0 msgr2=0x7f130015d410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:07.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:07.117+0000 7f12f7e00640 1 --2- 172.21.15.16:0/1570626403 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f13000cd0e0 0x7f130015d410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:07.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:07.117+0000 7f12f7e00640 1 -- 172.21.15.16:0/1570626403 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13001550c0 con 0x7f130013dff0 2024-03-31T15:04:07.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:07.117+0000 7f12f7e00640 1 --2- 172.21.15.16:0/1570626403 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f130013dff0 0x7f130013c560 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f12e800c4b0 tx=0x7f12e800c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:07.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:07.117+0000 7f12fcc00640 1 -- 172.21.15.16:0/1570626403 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12e8018070 con 0x7f130013dff0 2024-03-31T15:04:07.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:07.117+0000 7f12fcc00640 1 -- 172.21.15.16:0/1570626403 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f12e8007500 con 0x7f130013dff0 2024-03-31T15:04:07.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:07.117+0000 7f12fcc00640 1 -- 172.21.15.16:0/1570626403 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12e8004a30 con 0x7f130013dff0 2024-03-31T15:04:07.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:07.117+0000 7f1307200640 1 -- 172.21.15.16:0/1570626403 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f130015d9f0 con 0x7f130013dff0 2024-03-31T15:04:07.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:07.117+0000 7f1307200640 1 -- 172.21.15.16:0/1570626403 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f13001f7770 con 0x7f130013dff0 2024-03-31T15:04:07.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:07.117+0000 7f1307200640 1 -- 172.21.15.16:0/1570626403 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1300069950 con 0x7f130013dff0 2024-03-31T15:04:07.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:07.117+0000 7f12fcc00640 1 -- 172.21.15.16:0/1570626403 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f12e8004bd0 con 0x7f130013dff0 2024-03-31T15:04:07.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:07.121+0000 7f12fcc00640 1 --2- 172.21.15.16:0/1570626403 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f12dc068380 0x7f12dc06a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:07.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:07.121+0000 7f12fcc00640 1 -- 172.21.15.16:0/1570626403 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f12e8085be0 con 0x7f130013dff0 2024-03-31T15:04:07.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:07.121+0000 7f12ffe00640 1 --2- 172.21.15.16:0/1570626403 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f12dc068380 0x7f12dc06a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:07.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:07.121+0000 7f12ffe00640 1 --2- 172.21.15.16:0/1570626403 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f12dc068380 0x7f12dc06a840 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f12f00ad000 tx=0x7f12f00a6000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:07.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:07.125+0000 7f12fcc00640 1 -- 172.21.15.16:0/1570626403 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f12e8055a30 con 0x7f130013dff0 2024-03-31T15:04:07.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:07.289+0000 7f1307200640 1 -- 172.21.15.16:0/1570626403 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f13000b2c10 con 0x7f130013dff0 2024-03-31T15:04:07.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:07.289+0000 7f12fcc00640 1 -- 172.21.15.16:0/1570626403 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) ==== 93+0+10 (secure 0 0 0) 0x7f12e8059d30 con 0x7f130013dff0 2024-03-31T15:04:07.284 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2024-03-31T15:04:07.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:07.293+0000 7f1307200640 1 -- 172.21.15.16:0/1570626403 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f12dc068380 msgr2=0x7f12dc06a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:07.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:07.293+0000 7f1307200640 1 --2- 172.21.15.16:0/1570626403 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f12dc068380 0x7f12dc06a840 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f12f00ad000 tx=0x7f12f00a6000 comp rx=0 tx=0).stop 2024-03-31T15:04:07.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:07.293+0000 7f1307200640 1 -- 172.21.15.16:0/1570626403 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f130013dff0 msgr2=0x7f130013c560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:07.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:07.293+0000 7f1307200640 1 --2- 172.21.15.16:0/1570626403 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f130013dff0 0x7f130013c560 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f12e800c4b0 tx=0x7f12e800c970 comp rx=0 tx=0).stop 2024-03-31T15:04:07.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:07.293+0000 7f1307200640 1 -- 172.21.15.16:0/1570626403 shutdown_connections 2024-03-31T15:04:07.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:07.293+0000 7f1307200640 1 --2- 172.21.15.16:0/1570626403 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f12dc068380 0x7f12dc06a840 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:07.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:07.293+0000 7f1307200640 1 --2- 172.21.15.16:0/1570626403 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f130013e3d0 0x7f1300154b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:07.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:07.293+0000 7f1307200640 1 --2- 172.21.15.16:0/1570626403 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f130013dff0 0x7f130013c560 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:07.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:07.293+0000 7f1307200640 1 --2- 172.21.15.16:0/1570626403 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f13000cd0e0 0x7f130015d410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:07.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:07.293+0000 7f1307200640 1 -- 172.21.15.16:0/1570626403 >> 172.21.15.16:0/1570626403 conn(0x7f130006ee40 msgr2=0x7f1300158120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:04:07.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:07.293+0000 7f1307200640 1 -- 172.21.15.16:0/1570626403 shutdown_connections 2024-03-31T15:04:07.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:07.293+0000 7f1307200640 1 -- 172.21.15.16:0/1570626403 wait complete. 2024-03-31T15:04:07.305 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-31T15:04:07.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:07.457+0000 7fe044e00640 1 Processor -- start 2024-03-31T15:04:07.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:07.457+0000 7fe044e00640 1 -- start start 2024-03-31T15:04:07.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:07.457+0000 7fe044e00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fe0401411b0 0x7fe04006f920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:07.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:07.457+0000 7fe044e00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fe040141590 0x7fe04006fe60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:07.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:07.457+0000 7fe044e00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fe040143c30 0x7fe0401440a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:07.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:07.457+0000 7fe044e00640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7fe040162910 con 0x7fe040143c30 2024-03-31T15:04:07.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:07.457+0000 7fe044e00640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7fe040144980 con 0x7fe040141590 2024-03-31T15:04:07.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:07.457+0000 7fe044e00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fe040161fe0 con 0x7fe0401411b0 2024-03-31T15:04:07.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:07.457+0000 7fe03e800640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fe040143c30 0x7fe0401440a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:07.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:07.457+0000 7fe03e800640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fe040143c30 0x7fe0401440a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60294/0 (socket says 172.21.15.16:60294) 2024-03-31T15:04:07.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:07.457+0000 7fe03e800640 1 -- 172.21.15.16:0/1855083000 learned_addr learned my addr 172.21.15.16:0/1855083000 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:04:07.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:07.457+0000 7fe037e00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fe040141590 0x7fe04006fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:07.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:07.457+0000 7fe03de00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fe0401411b0 0x7fe04006f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:07.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:07.457+0000 7fe03e800640 1 -- 172.21.15.16:0/1855083000 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fe0401411b0 msgr2=0x7fe04006f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:04:07.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:07.457+0000 7fe03e800640 1 --2- 172.21.15.16:0/1855083000 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fe0401411b0 0x7fe04006f920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:07.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:07.457+0000 7fe03e800640 1 -- 172.21.15.16:0/1855083000 >> v2:172.21.15.89:3300/0 conn(0x7fe040141590 msgr2=0x7fe04006fe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:04:07.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:07.457+0000 7fe03e800640 1 --2- 172.21.15.16:0/1855083000 >> v2:172.21.15.89:3300/0 conn(0x7fe040141590 0x7fe04006fe60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:07.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:07.457+0000 7fe03e800640 1 -- 172.21.15.16:0/1855083000 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe0401622c0 con 0x7fe040143c30 2024-03-31T15:04:07.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:07.461+0000 7fe03e800640 1 --2- 172.21.15.16:0/1855083000 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe040143c30 0x7fe0401440a0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fe040071f30 tx=0x7fe03801c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dda6e9e37bf27a1c server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:07.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:07.461+0000 7fe03d400640 1 -- 172.21.15.16:0/1855083000 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0380a5860 con 0x7fe040143c30 2024-03-31T15:04:07.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:07.461+0000 7fe03d400640 1 -- 172.21.15.16:0/1855083000 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe0380a5a00 con 0x7fe040143c30 2024-03-31T15:04:07.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:07.461+0000 7fe03d400640 1 -- 172.21.15.16:0/1855083000 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0380a5dd0 con 0x7fe040143c30 2024-03-31T15:04:07.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:07.461+0000 7fe044e00640 1 -- 172.21.15.16:0/1855083000 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe040143c30 msgr2=0x7fe0401440a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:07.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:07.461+0000 7fe044e00640 1 --2- 172.21.15.16:0/1855083000 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe040143c30 0x7fe0401440a0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fe040071f30 tx=0x7fe03801c9c0 comp rx=0 tx=0).stop 2024-03-31T15:04:07.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:07.461+0000 7fe044e00640 1 -- 172.21.15.16:0/1855083000 shutdown_connections 2024-03-31T15:04:07.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:07.461+0000 7fe044e00640 1 --2- 172.21.15.16:0/1855083000 >> v2:172.21.15.89:3300/0 conn(0x7fe040141590 0x7fe04006fe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:07.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:07.461+0000 7fe044e00640 1 --2- 172.21.15.16:0/1855083000 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe040143c30 0x7fe0401440a0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:07.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:07.461+0000 7fe044e00640 1 --2- 172.21.15.16:0/1855083000 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fe0401411b0 0x7fe04006f920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:07.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:07.461+0000 7fe044e00640 1 -- 172.21.15.16:0/1855083000 >> 172.21.15.16:0/1855083000 conn(0x7fe040067930 msgr2=0x7fe0400b3ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:04:07.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:07.461+0000 7fe044e00640 1 -- 172.21.15.16:0/1855083000 shutdown_connections 2024-03-31T15:04:07.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:07.461+0000 7fe044e00640 1 -- 172.21.15.16:0/1855083000 wait complete. 2024-03-31T15:04:07.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:07.461+0000 7fe044e00640 1 Processor -- start 2024-03-31T15:04:07.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:07.461+0000 7fe044e00640 1 -- start start 2024-03-31T15:04:07.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:07.465+0000 7fe044e00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fe0401411b0 0x7fe040140660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:07.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:07.465+0000 7fe044e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe040141590 0x7fe040150dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:07.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:07.465+0000 7fe044e00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe040143c30 0x7fe040149260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:07.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:07.465+0000 7fe044e00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fe0400717f0 con 0x7fe040141590 2024-03-31T15:04:07.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:07.465+0000 7fe044e00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fe040071670 con 0x7fe040143c30 2024-03-31T15:04:07.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:07.465+0000 7fe044e00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fe040071970 con 0x7fe0401411b0 2024-03-31T15:04:07.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:07.465+0000 7fe037e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe040141590 0x7fe040150dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:07.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:07.465+0000 7fe037e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe040141590 0x7fe040150dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60302/0 (socket says 172.21.15.16:60302) 2024-03-31T15:04:07.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:07.465+0000 7fe037e00640 1 -- 172.21.15.16:0/815822364 learned_addr learned my addr 172.21.15.16:0/815822364 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:04:07.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:07.465+0000 7fe03e800640 1 --2- 172.21.15.16:0/815822364 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe040143c30 0x7fe040149260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:07.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:07.465+0000 7fe03de00640 1 --2- 172.21.15.16:0/815822364 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fe0401411b0 0x7fe040140660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:07.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:07.465+0000 7fe037e00640 1 -- 172.21.15.16:0/815822364 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fe0401411b0 msgr2=0x7fe040140660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:07.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:07.465+0000 7fe037e00640 1 --2- 172.21.15.16:0/815822364 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fe0401411b0 0x7fe040140660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:07.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:07.465+0000 7fe037e00640 1 -- 172.21.15.16:0/815822364 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe040143c30 msgr2=0x7fe040149260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:07.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:07.465+0000 7fe037e00640 1 --2- 172.21.15.16:0/815822364 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe040143c30 0x7fe040149260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:07.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:07.465+0000 7fe037e00640 1 -- 172.21.15.16:0/815822364 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe040156560 con 0x7fe040141590 2024-03-31T15:04:07.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:07.465+0000 7fe037e00640 1 --2- 172.21.15.16:0/815822364 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe040141590 0x7fe040150dd0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fe02800e930 tx=0x7fe02800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:07.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:07.465+0000 7fe036000640 1 -- 172.21.15.16:0/815822364 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe028017070 con 0x7fe040141590 2024-03-31T15:04:07.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:07.465+0000 7fe036000640 1 -- 172.21.15.16:0/815822364 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe02800ca00 con 0x7fe040141590 2024-03-31T15:04:07.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:07.465+0000 7fe036000640 1 -- 172.21.15.16:0/815822364 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe02800cdf0 con 0x7fe040141590 2024-03-31T15:04:07.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:07.465+0000 7fe044e00640 1 -- 172.21.15.16:0/815822364 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe040156840 con 0x7fe040141590 2024-03-31T15:04:07.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:07.465+0000 7fe044e00640 1 -- 172.21.15.16:0/815822364 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe040156e00 con 0x7fe040141590 2024-03-31T15:04:07.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:07.465+0000 7fe044e00640 1 -- 172.21.15.16:0/815822364 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe040069950 con 0x7fe040141590 2024-03-31T15:04:07.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:07.465+0000 7fe036000640 1 -- 172.21.15.16:0/815822364 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fe02801a070 con 0x7fe040141590 2024-03-31T15:04:07.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:07.469+0000 7fe036000640 1 --2- 172.21.15.16:0/815822364 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fe01c068380 0x7fe01c06a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:07.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:07.469+0000 7fe036000640 1 -- 172.21.15.16:0/815822364 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fe028086be0 con 0x7fe040141590 2024-03-31T15:04:07.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:07.469+0000 7fe03de00640 1 --2- 172.21.15.16:0/815822364 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fe01c068380 0x7fe01c06a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:07.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:07.469+0000 7fe03de00640 1 --2- 172.21.15.16:0/815822364 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fe01c068380 0x7fe01c06a840 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fe030009b60 tx=0x7fe030009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:07.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:07.473+0000 7fe036000640 1 -- 172.21.15.16:0/815822364 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fe028056ca0 con 0x7fe040141590 2024-03-31T15:04:07.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:07.637+0000 7fe044e00640 1 -- 172.21.15.16:0/815822364 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fe040161000 con 0x7fe040141590 2024-03-31T15:04:07.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:07.637+0000 7fe036000640 1 -- 172.21.15.16:0/815822364 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) ==== 104+0+11 (secure 0 0 0) 0x7fe02805afa0 con 0x7fe040141590 2024-03-31T15:04:07.631 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2024-03-31T15:04:07.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:07.641+0000 7fe044e00640 1 -- 172.21.15.16:0/815822364 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fe01c068380 msgr2=0x7fe01c06a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:07.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:07.641+0000 7fe044e00640 1 --2- 172.21.15.16:0/815822364 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fe01c068380 0x7fe01c06a840 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fe030009b60 tx=0x7fe030009190 comp rx=0 tx=0).stop 2024-03-31T15:04:07.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:07.641+0000 7fe044e00640 1 -- 172.21.15.16:0/815822364 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe040141590 msgr2=0x7fe040150dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:07.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:07.641+0000 7fe044e00640 1 --2- 172.21.15.16:0/815822364 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe040141590 0x7fe040150dd0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fe02800e930 tx=0x7fe02800edf0 comp rx=0 tx=0).stop 2024-03-31T15:04:07.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:07.641+0000 7fe044e00640 1 -- 172.21.15.16:0/815822364 shutdown_connections 2024-03-31T15:04:07.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:07.641+0000 7fe044e00640 1 --2- 172.21.15.16:0/815822364 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fe01c068380 0x7fe01c06a840 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:07.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:07.641+0000 7fe044e00640 1 --2- 172.21.15.16:0/815822364 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe040143c30 0x7fe040149260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:07.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:07.641+0000 7fe044e00640 1 --2- 172.21.15.16:0/815822364 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe040141590 0x7fe040150dd0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:07.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:07.641+0000 7fe044e00640 1 --2- 172.21.15.16:0/815822364 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fe0401411b0 0x7fe040140660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:07.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:07.641+0000 7fe044e00640 1 -- 172.21.15.16:0/815822364 >> 172.21.15.16:0/815822364 conn(0x7fe040067930 msgr2=0x7fe04006d790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:04:07.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:07.641+0000 7fe044e00640 1 -- 172.21.15.16:0/815822364 shutdown_connections 2024-03-31T15:04:07.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:07.641+0000 7fe044e00640 1 -- 172.21.15.16:0/815822364 wait complete. 2024-03-31T15:04:07.651 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-31T15:04:07.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:07.801+0000 7fcc78c00640 1 Processor -- start 2024-03-31T15:04:07.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:07.801+0000 7fcc78c00640 1 -- start start 2024-03-31T15:04:07.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:07.801+0000 7fcc78c00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fcc741411b0 0x7fcc7406f920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:07.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:07.801+0000 7fcc78c00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fcc74141590 0x7fcc7406fe60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:07.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:07.801+0000 7fcc78c00640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcc74144980 con 0x7fcc74143c30 2024-03-31T15:04:07.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:07.801+0000 7fcc78c00640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7fcc7415f910 con 0x7fcc741411b0 2024-03-31T15:04:07.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:07.801+0000 7fcc78c00640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7fcc74162910 con 0x7fcc74141590 2024-03-31T15:04:07.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:07.801+0000 7fcc71e00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fcc741411b0 0x7fcc7406f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:07.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:07.801+0000 7fcc71e00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fcc741411b0 0x7fcc7406f920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60304/0 (socket says 172.21.15.16:60304) 2024-03-31T15:04:07.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:07.801+0000 7fcc71e00640 1 -- 172.21.15.16:0/2555169616 learned_addr learned my addr 172.21.15.16:0/2555169616 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:04:07.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:07.801+0000 7fcc6be00640 1 --2- 172.21.15.16:0/2555169616 >> v2:172.21.15.89:3300/0 conn(0x7fcc74141590 0x7fcc7406fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:07.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:07.801+0000 7fcc71e00640 1 -- 172.21.15.16:0/2555169616 >> v2:172.21.15.89:3300/0 conn(0x7fcc74141590 msgr2=0x7fcc7406fe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:04:07.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:07.801+0000 7fcc71e00640 1 --2- 172.21.15.16:0/2555169616 >> v2:172.21.15.89:3300/0 conn(0x7fcc74141590 0x7fcc7406fe60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:07.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:07.801+0000 7fcc71e00640 1 -- 172.21.15.16:0/2555169616 >> v1:172.21.15.89:6789/0 conn(0x7fcc74143c30 legacy=0x7fcc741440a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:07.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:07.801+0000 7fcc71e00640 1 -- 172.21.15.16:0/2555169616 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc74161fe0 con 0x7fcc741411b0 2024-03-31T15:04:07.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:07.805+0000 7fcc71e00640 1 --2- 172.21.15.16:0/2555169616 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcc741411b0 0x7fcc7406f920 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fcc74158fe0 tx=0x7fcc6401c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c98db281c55829b server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:07.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:07.805+0000 7fcc71400640 1 -- 172.21.15.16:0/2555169616 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc640ab610 con 0x7fcc741411b0 2024-03-31T15:04:07.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:07.805+0000 7fcc71400640 1 -- 172.21.15.16:0/2555169616 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcc640b0070 con 0x7fcc741411b0 2024-03-31T15:04:07.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:07.805+0000 7fcc71400640 1 -- 172.21.15.16:0/2555169616 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc640ab9d0 con 0x7fcc741411b0 2024-03-31T15:04:07.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:07.805+0000 7fcc78c00640 1 -- 172.21.15.16:0/2555169616 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcc741411b0 msgr2=0x7fcc7406f920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:07.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:07.805+0000 7fcc78c00640 1 --2- 172.21.15.16:0/2555169616 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcc741411b0 0x7fcc7406f920 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fcc74158fe0 tx=0x7fcc6401c520 comp rx=0 tx=0).stop 2024-03-31T15:04:07.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:07.805+0000 7fcc78c00640 1 -- 172.21.15.16:0/2555169616 shutdown_connections 2024-03-31T15:04:07.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:07.805+0000 7fcc78c00640 1 --2- 172.21.15.16:0/2555169616 >> v2:172.21.15.89:3300/0 conn(0x7fcc74141590 0x7fcc7406fe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:07.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:07.805+0000 7fcc78c00640 1 --2- 172.21.15.16:0/2555169616 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcc741411b0 0x7fcc7406f920 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:07.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:07.805+0000 7fcc78c00640 1 -- 172.21.15.16:0/2555169616 >> 172.21.15.16:0/2555169616 conn(0x7fcc74067930 msgr2=0x7fcc740b3ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:04:07.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:07.805+0000 7fcc78c00640 1 -- 172.21.15.16:0/2555169616 shutdown_connections 2024-03-31T15:04:07.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:07.805+0000 7fcc78c00640 1 -- 172.21.15.16:0/2555169616 wait complete. 2024-03-31T15:04:07.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:07.809+0000 7fcc78c00640 1 Processor -- start 2024-03-31T15:04:07.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:07.809+0000 7fcc78c00640 1 -- start start 2024-03-31T15:04:07.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:07.809+0000 7fcc78c00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcc741411b0 0x7fcc7415dfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:07.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:07.809+0000 7fcc78c00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcc74141590 0x7fcc741569e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:07.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:07.809+0000 7fcc78c00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fcc74143c30 0x7fcc740b29b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:07.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:07.809+0000 7fcc78c00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fcc7415f290 con 0x7fcc74141590 2024-03-31T15:04:07.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:07.809+0000 7fcc78c00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fcc7415f110 con 0x7fcc741411b0 2024-03-31T15:04:07.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:07.809+0000 7fcc78c00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fcc7415f410 con 0x7fcc74143c30 2024-03-31T15:04:07.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:07.809+0000 7fcc71e00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcc741411b0 0x7fcc7415dfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:07.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:07.809+0000 7fcc71e00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcc741411b0 0x7fcc7415dfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.16:44270/0 (socket says 172.21.15.16:44270) 2024-03-31T15:04:07.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:07.809+0000 7fcc71e00640 1 -- 172.21.15.16:0/1441824975 learned_addr learned my addr 172.21.15.16:0/1441824975 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:04:07.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:07.809+0000 7fcc6be00640 1 --2- 172.21.15.16:0/1441824975 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcc74141590 0x7fcc741569e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:07.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:07.809+0000 7fcc72800640 1 --2- 172.21.15.16:0/1441824975 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fcc74143c30 0x7fcc740b29b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:07.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:07.809+0000 7fcc71e00640 1 -- 172.21.15.16:0/1441824975 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fcc74143c30 msgr2=0x7fcc740b29b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:07.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:07.809+0000 7fcc71e00640 1 --2- 172.21.15.16:0/1441824975 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fcc74143c30 0x7fcc740b29b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:07.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:07.809+0000 7fcc71e00640 1 -- 172.21.15.16:0/1441824975 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcc74141590 msgr2=0x7fcc741569e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:07.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:07.809+0000 7fcc71e00640 1 --2- 172.21.15.16:0/1441824975 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcc74141590 0x7fcc741569e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:07.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:07.809+0000 7fcc71e00640 1 -- 172.21.15.16:0/1441824975 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc740d2de0 con 0x7fcc741411b0 2024-03-31T15:04:07.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:07.809+0000 7fcc71e00640 1 --2- 172.21.15.16:0/1441824975 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcc741411b0 0x7fcc7415dfd0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fcc6401abd0 tx=0x7fcc6401ad60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:07.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:07.809+0000 7fcc6a000640 1 -- 172.21.15.16:0/1441824975 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc640be5f0 con 0x7fcc741411b0 2024-03-31T15:04:07.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:07.809+0000 7fcc6a000640 1 -- 172.21.15.16:0/1441824975 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcc640be790 con 0x7fcc741411b0 2024-03-31T15:04:07.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:07.809+0000 7fcc6a000640 1 -- 172.21.15.16:0/1441824975 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc640beb60 con 0x7fcc741411b0 2024-03-31T15:04:07.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:07.809+0000 7fcc78c00640 1 -- 172.21.15.16:0/1441824975 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcc7415c920 con 0x7fcc741411b0 2024-03-31T15:04:07.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:07.809+0000 7fcc78c00640 1 -- 172.21.15.16:0/1441824975 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcc7415cee0 con 0x7fcc741411b0 2024-03-31T15:04:07.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:07.813+0000 7fcc78c00640 1 -- 172.21.15.16:0/1441824975 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcc74069950 con 0x7fcc741411b0 2024-03-31T15:04:07.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:07.813+0000 7fcc6a000640 1 -- 172.21.15.16:0/1441824975 <== mon.1 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fcc640b43f0 con 0x7fcc741411b0 2024-03-31T15:04:07.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:07.813+0000 7fcc6a000640 1 --2- 172.21.15.16:0/1441824975 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fcc50068380 0x7fcc5006a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:07.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:07.813+0000 7fcc6a000640 1 -- 172.21.15.16:0/1441824975 <== mon.1 v2:172.21.15.89:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fcc64120bf0 con 0x7fcc741411b0 2024-03-31T15:04:07.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:07.813+0000 7fcc6be00640 1 --2- 172.21.15.16:0/1441824975 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fcc50068380 0x7fcc5006a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:07.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:07.813+0000 7fcc6be00640 1 --2- 172.21.15.16:0/1441824975 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fcc50068380 0x7fcc5006a840 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fcc7413bdf0 tx=0x7fcc5c009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:07.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:07.817+0000 7fcc6a000640 1 -- 172.21.15.16:0/1441824975 <== mon.1 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fcc640a8050 con 0x7fcc741411b0 2024-03-31T15:04:07.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:07.985+0000 7fcc78c00640 1 -- 172.21.15.16:0/1441824975 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fcc740b58b0 con 0x7fcc741411b0 2024-03-31T15:04:07.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:07.985+0000 7fcc6a000640 1 -- 172.21.15.16:0/1441824975 <== mon.1 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) ==== 100+0+11 (secure 0 0 0) 0x7fcc64016110 con 0x7fcc741411b0 2024-03-31T15:04:07.977 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 64 2024-03-31T15:04:07.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:07.985+0000 7fcc78c00640 1 -- 172.21.15.16:0/1441824975 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fcc50068380 msgr2=0x7fcc5006a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:07.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:07.985+0000 7fcc78c00640 1 --2- 172.21.15.16:0/1441824975 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fcc50068380 0x7fcc5006a840 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fcc7413bdf0 tx=0x7fcc5c009190 comp rx=0 tx=0).stop 2024-03-31T15:04:07.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:07.985+0000 7fcc78c00640 1 -- 172.21.15.16:0/1441824975 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcc741411b0 msgr2=0x7fcc7415dfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:07.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:07.985+0000 7fcc78c00640 1 --2- 172.21.15.16:0/1441824975 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcc741411b0 0x7fcc7415dfd0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fcc6401abd0 tx=0x7fcc6401ad60 comp rx=0 tx=0).stop 2024-03-31T15:04:07.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:07.985+0000 7fcc78c00640 1 -- 172.21.15.16:0/1441824975 shutdown_connections 2024-03-31T15:04:07.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:07.985+0000 7fcc78c00640 1 --2- 172.21.15.16:0/1441824975 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fcc50068380 0x7fcc5006a840 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:07.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:07.985+0000 7fcc78c00640 1 --2- 172.21.15.16:0/1441824975 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fcc74143c30 0x7fcc740b29b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:07.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:07.985+0000 7fcc78c00640 1 --2- 172.21.15.16:0/1441824975 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcc74141590 0x7fcc741569e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:07.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:07.985+0000 7fcc78c00640 1 --2- 172.21.15.16:0/1441824975 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcc741411b0 0x7fcc7415dfd0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:07.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:07.985+0000 7fcc78c00640 1 -- 172.21.15.16:0/1441824975 >> 172.21.15.16:0/1441824975 conn(0x7fcc74067930 msgr2=0x7fcc74143200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:04:07.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:07.985+0000 7fcc78c00640 1 -- 172.21.15.16:0/1441824975 shutdown_connections 2024-03-31T15:04:07.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:07.985+0000 7fcc78c00640 1 -- 172.21.15.16:0/1441824975 wait complete. 2024-03-31T15:04:08.027 INFO:tasks.cephfs_test_runner:Starting test: test_apply_tag (tasks.cephfs.test_forward_scrub.TestForwardScrub) 2024-03-31T15:04:08.027 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-31T15:04:08.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:08.177+0000 7ff5fe800640 1 Processor -- start 2024-03-31T15:04:08.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:08.177+0000 7ff5fe800640 1 -- start start 2024-03-31T15:04:08.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:08.177+0000 7ff5fe800640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7ff5f80d0210 0x7ff5f815c6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:08.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:08.177+0000 7ff5fe800640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff5f80cc810 0x7ff5f809af80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:08.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:08.177+0000 7ff5fe800640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff5f815ccd0 con 0x7ff5f80b4740 2024-03-31T15:04:08.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:08.177+0000 7ff5fe800640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7ff5f815cfb0 con 0x7ff5f80cc810 2024-03-31T15:04:08.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:08.177+0000 7ff5fe800640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7ff5f815d290 con 0x7ff5f80d0210 2024-03-31T15:04:08.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:08.177+0000 7ff5fde00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff5f80cc810 0x7ff5f809af80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:08.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:08.177+0000 7ff5fde00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff5f80cc810 0x7ff5f809af80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60310/0 (socket says 172.21.15.16:60310) 2024-03-31T15:04:08.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:08.177+0000 7ff5fde00640 1 -- 172.21.15.16:0/4240796725 learned_addr learned my addr 172.21.15.16:0/4240796725 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:04:08.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:08.177+0000 7ff5fd400640 1 --2- 172.21.15.16:0/4240796725 >> v2:172.21.15.89:3300/0 conn(0x7ff5f80d0210 0x7ff5f815c6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:08.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:08.177+0000 7ff5fde00640 1 -- 172.21.15.16:0/4240796725 >> v2:172.21.15.89:3300/0 conn(0x7ff5f80d0210 msgr2=0x7ff5f815c6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:04:08.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:08.177+0000 7ff5fde00640 1 --2- 172.21.15.16:0/4240796725 >> v2:172.21.15.89:3300/0 conn(0x7ff5f80d0210 0x7ff5f815c6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:08.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:08.177+0000 7ff5fde00640 1 -- 172.21.15.16:0/4240796725 >> v1:172.21.15.89:6789/0 conn(0x7ff5f80b4740 legacy=0x7ff5f80d05f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:08.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:08.177+0000 7ff5fde00640 1 -- 172.21.15.16:0/4240796725 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5f815d570 con 0x7ff5f80cc810 2024-03-31T15:04:08.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:08.181+0000 7ff5fde00640 1 --2- 172.21.15.16:0/4240796725 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff5f80cc810 0x7ff5f809af80 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7ff5f0009e50 tx=0x7ff5f001c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7b07e1ac98ebc5b server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:08.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:08.181+0000 7ff5f7e00640 1 -- 172.21.15.16:0/4240796725 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5f00a56c0 con 0x7ff5f80cc810 2024-03-31T15:04:08.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:08.181+0000 7ff5f7e00640 1 -- 172.21.15.16:0/4240796725 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff5f00a5860 con 0x7ff5f80cc810 2024-03-31T15:04:08.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:08.181+0000 7ff5f7e00640 1 -- 172.21.15.16:0/4240796725 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5f00a5c30 con 0x7ff5f80cc810 2024-03-31T15:04:08.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:08.181+0000 7ff5fe800640 1 -- 172.21.15.16:0/4240796725 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff5f80cc810 msgr2=0x7ff5f809af80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:08.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:08.181+0000 7ff5fe800640 1 --2- 172.21.15.16:0/4240796725 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff5f80cc810 0x7ff5f809af80 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7ff5f0009e50 tx=0x7ff5f001c9f0 comp rx=0 tx=0).stop 2024-03-31T15:04:08.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:08.181+0000 7ff5fe800640 1 -- 172.21.15.16:0/4240796725 shutdown_connections 2024-03-31T15:04:08.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:08.181+0000 7ff5fe800640 1 --2- 172.21.15.16:0/4240796725 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff5f80cc810 0x7ff5f809af80 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:08.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:08.181+0000 7ff5fe800640 1 --2- 172.21.15.16:0/4240796725 >> v2:172.21.15.89:3300/0 conn(0x7ff5f80d0210 0x7ff5f815c6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:08.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:08.181+0000 7ff5fe800640 1 -- 172.21.15.16:0/4240796725 >> 172.21.15.16:0/4240796725 conn(0x7ff5f80b3c20 msgr2=0x7ff5f80b4030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:04:08.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:08.181+0000 7ff5fe800640 1 -- 172.21.15.16:0/4240796725 shutdown_connections 2024-03-31T15:04:08.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:08.181+0000 7ff5fe800640 1 -- 172.21.15.16:0/4240796725 wait complete. 2024-03-31T15:04:08.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:08.181+0000 7ff5fe800640 1 Processor -- start 2024-03-31T15:04:08.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:08.181+0000 7ff5fe800640 1 -- start start 2024-03-31T15:04:08.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:08.185+0000 7ff5fe800640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff5f80b4740 0x7ff5f815b3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:08.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:08.185+0000 7ff5fe800640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff5f80cc810 0x7ff5f8143590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:08.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:08.185+0000 7ff5fe800640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff5f80d0210 0x7ff5f80b2010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:08.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:08.185+0000 7ff5fe800640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7ff5f8169930 con 0x7ff5f80b4740 2024-03-31T15:04:08.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:08.185+0000 7ff5fe800640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7ff5f81697b0 con 0x7ff5f80cc810 2024-03-31T15:04:08.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:08.185+0000 7ff5fe800640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7ff5f8169ab0 con 0x7ff5f80d0210 2024-03-31T15:04:08.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:08.185+0000 7ff5fd400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff5f80b4740 0x7ff5f815b3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:08.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:08.185+0000 7ff5fd400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff5f80b4740 0x7ff5f815b3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60326/0 (socket says 172.21.15.16:60326) 2024-03-31T15:04:08.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:08.185+0000 7ff5fd400640 1 -- 172.21.15.16:0/3152569279 learned_addr learned my addr 172.21.15.16:0/3152569279 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:04:08.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:08.185+0000 7ff5fde00640 1 --2- 172.21.15.16:0/3152569279 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff5f80d0210 0x7ff5f80b2010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:08.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:08.185+0000 7ff5fca00640 1 --2- 172.21.15.16:0/3152569279 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff5f80cc810 0x7ff5f8143590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:08.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:08.185+0000 7ff5fd400640 1 -- 172.21.15.16:0/3152569279 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff5f80d0210 msgr2=0x7ff5f80b2010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:08.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:08.185+0000 7ff5fd400640 1 --2- 172.21.15.16:0/3152569279 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff5f80d0210 0x7ff5f80b2010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:08.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:08.185+0000 7ff5fd400640 1 -- 172.21.15.16:0/3152569279 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff5f80cc810 msgr2=0x7ff5f8143590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:08.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:08.185+0000 7ff5fd400640 1 --2- 172.21.15.16:0/3152569279 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff5f80cc810 0x7ff5f8143590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:08.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:08.185+0000 7ff5fd400640 1 -- 172.21.15.16:0/3152569279 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5f80b0c90 con 0x7ff5f80b4740 2024-03-31T15:04:08.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:08.185+0000 7ff5fca00640 1 --2- 172.21.15.16:0/3152569279 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff5f80cc810 0x7ff5f8143590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:04:08.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:08.185+0000 7ff5fd400640 1 --2- 172.21.15.16:0/3152569279 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff5f80b4740 0x7ff5f815b3b0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7ff5ec00e930 tx=0x7ff5ec00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:08.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:08.185+0000 7ff5f5600640 1 -- 172.21.15.16:0/3152569279 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5ec017070 con 0x7ff5f80b4740 2024-03-31T15:04:08.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:08.185+0000 7ff5f5600640 1 -- 172.21.15.16:0/3152569279 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff5ec00ca00 con 0x7ff5f80b4740 2024-03-31T15:04:08.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:08.185+0000 7ff5f5600640 1 -- 172.21.15.16:0/3152569279 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5ec00cdb0 con 0x7ff5f80b4740 2024-03-31T15:04:08.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:08.185+0000 7ff5fe800640 1 -- 172.21.15.16:0/3152569279 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff5f80b0f70 con 0x7ff5f80b4740 2024-03-31T15:04:08.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:08.185+0000 7ff5fe800640 1 -- 172.21.15.16:0/3152569279 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff5f80b1530 con 0x7ff5f80b4740 2024-03-31T15:04:08.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:08.185+0000 7ff5fe800640 1 -- 172.21.15.16:0/3152569279 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff5f804f8c0 con 0x7ff5f80b4740 2024-03-31T15:04:08.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:08.185+0000 7ff5f5600640 1 -- 172.21.15.16:0/3152569279 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7ff5ec01a070 con 0x7ff5f80b4740 2024-03-31T15:04:08.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:08.189+0000 7ff5f5600640 1 --2- 172.21.15.16:0/3152569279 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7ff5d8068380 0x7ff5d806a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:08.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:08.189+0000 7ff5f5600640 1 -- 172.21.15.16:0/3152569279 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7ff5ec086d80 con 0x7ff5f80b4740 2024-03-31T15:04:08.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:08.189+0000 7ff5fca00640 1 --2- 172.21.15.16:0/3152569279 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7ff5d8068380 0x7ff5d806a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:08.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:08.189+0000 7ff5fca00640 1 --2- 172.21.15.16:0/3152569279 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7ff5d8068380 0x7ff5d806a840 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7ff5f80b7e80 tx=0x7ff5e4009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:08.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:08.193+0000 7ff5f5600640 1 -- 172.21.15.16:0/3152569279 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7ff5ec056d10 con 0x7ff5f80b4740 2024-03-31T15:04:08.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:08.357+0000 7ff5fe800640 1 -- 172.21.15.16:0/3152569279 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ff5f8068050 con 0x7ff5f80b4740 2024-03-31T15:04:08.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:08.357+0000 7ff5f5600640 1 -- 172.21.15.16:0/3152569279 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) ==== 74+0+16533 (secure 0 0 0) 0x7ff5ec05b010 con 0x7ff5f80b4740 2024-03-31T15:04:08.351 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:04:08.352 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":21,"fsid":"286eb5da-fb47-4550-b290-d9519c13b9a5","created":"2024-03-31T15:03:11.438952+0000","modified":"2024-03-31T15:03:37.957829+0000","last_up_change":"2024-03-31T15:03:22.885535+0000","last_in_change":"2024-03-31T15:03:13.941667+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-31T15:03:22.550787+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-31T15:03:34.488425+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-31T15:03:35.299116+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"92a66450-ef19-40d8-ac3d-d760b23b0bda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6801","nonce":932712244}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6803","nonce":932712244}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6807","nonce":932712244}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6805","nonce":932712244}]},"public_addr":"172.21.15.16:6801/932712244","cluster_addr":"172.21.15.16:6803/932712244","heartbeat_back_addr":"172.21.15.16:6807/932712244","heartbeat_front_addr":"172.21.15.16:6805/932712244","state":["exists","up"]},{"osd":1,"uuid":"da9d1f2b-1968-4959-b843-faa231988e92","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6810","nonce":1398737533}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6811","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6813","nonce":1398737533}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6821","nonce":1398737533}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6815","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6816","nonce":1398737533}]},"public_addr":"172.21.15.16:6810/1398737533","cluster_addr":"172.21.15.16:6813/1398737533","heartbeat_back_addr":"172.21.15.16:6821/1398737533","heartbeat_front_addr":"172.21.15.16:6816/1398737533","state":["exists","up"]},{"osd":2,"uuid":"01a73662-9a21-4b1f-bcbb-42b9636a4638","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6819","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6822","nonce":3018526646}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6825","nonce":3018526646}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6831","nonce":3018526646}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6827","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6829","nonce":3018526646}]},"public_addr":"172.21.15.16:6822/3018526646","cluster_addr":"172.21.15.16:6825/3018526646","heartbeat_back_addr":"172.21.15.16:6831/3018526646","heartbeat_front_addr":"172.21.15.16:6829/3018526646","state":["exists","up"]},{"osd":3,"uuid":"c7771fd3-e18e-4500-ac38-23cf10ebcd75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6809","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6812","nonce":3360352974}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6817","nonce":3360352974}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6828","nonce":3360352974}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6823","nonce":3360352974}]},"public_addr":"172.21.15.16:6812/3360352974","cluster_addr":"172.21.15.16:6817/3360352974","heartbeat_back_addr":"172.21.15.16:6828/3360352974","heartbeat_front_addr":"172.21.15.16:6823/3360352974","state":["exists","up"]},{"osd":4,"uuid":"88260730-1482-4fbd-a40f-cb1124e9df4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6801","nonce":1856844856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6803","nonce":1856844856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6807","nonce":1856844856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6805","nonce":1856844856}]},"public_addr":"172.21.15.89:6801/1856844856","cluster_addr":"172.21.15.89:6803/1856844856","heartbeat_back_addr":"172.21.15.89:6807/1856844856","heartbeat_front_addr":"172.21.15.89:6805/1856844856","state":["exists","up"]},{"osd":5,"uuid":"4cb2c00c-fae0-4dd8-9d27-ec4f0093ccf7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6814","nonce":50395527}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6817","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6820","nonce":50395527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6831","nonce":50395527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6823","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6826","nonce":50395527}]},"public_addr":"172.21.15.89:6814/50395527","cluster_addr":"172.21.15.89:6820/50395527","heartbeat_back_addr":"172.21.15.89:6831/50395527","heartbeat_front_addr":"172.21.15.89:6826/50395527","state":["exists","up"]},{"osd":6,"uuid":"81ce0c19-2554-4cae-acbe-66a05e9e9b9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6809","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6812","nonce":3470471752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6815","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6818","nonce":3470471752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6827","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6829","nonce":3470471752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6821","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6824","nonce":3470471752}]},"public_addr":"172.21.15.89:6812/3470471752","cluster_addr":"172.21.15.89:6818/3470471752","heartbeat_back_addr":"172.21.15.89:6829/3470471752","heartbeat_front_addr":"172.21.15.89:6824/3470471752","state":["exists","up"]},{"osd":7,"uuid":"85e4313b-d37d-442e-8042-153bf20a847d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6811","nonce":1796778630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6813","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6816","nonce":1796778630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6825","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6828","nonce":1796778630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6819","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6822","nonce":1796778630}]},"public_addr":"172.21.15.89:6811/1796778630","cluster_addr":"172.21.15.89:6816/1796778630","heartbeat_back_addr":"172.21.15.89:6828/1796778630","heartbeat_front_addr":"172.21.15.89:6822/1796778630","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.439088+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.529970+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.680512+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:21.044161+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.582111+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.536558+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.409522+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.590229+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-31T15:04:08.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:08.361+0000 7ff5fe800640 1 -- 172.21.15.16:0/3152569279 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7ff5d8068380 msgr2=0x7ff5d806a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:08.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:08.361+0000 7ff5fe800640 1 --2- 172.21.15.16:0/3152569279 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7ff5d8068380 0x7ff5d806a840 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7ff5f80b7e80 tx=0x7ff5e4009040 comp rx=0 tx=0).stop 2024-03-31T15:04:08.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:08.361+0000 7ff5fe800640 1 -- 172.21.15.16:0/3152569279 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff5f80b4740 msgr2=0x7ff5f815b3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:08.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:08.361+0000 7ff5fe800640 1 --2- 172.21.15.16:0/3152569279 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff5f80b4740 0x7ff5f815b3b0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7ff5ec00e930 tx=0x7ff5ec00edf0 comp rx=0 tx=0).stop 2024-03-31T15:04:08.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:08.361+0000 7ff5fe800640 1 -- 172.21.15.16:0/3152569279 shutdown_connections 2024-03-31T15:04:08.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:08.361+0000 7ff5fe800640 1 --2- 172.21.15.16:0/3152569279 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7ff5d8068380 0x7ff5d806a840 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:08.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:08.361+0000 7ff5fe800640 1 --2- 172.21.15.16:0/3152569279 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff5f80d0210 0x7ff5f80b2010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:08.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:08.361+0000 7ff5fe800640 1 --2- 172.21.15.16:0/3152569279 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff5f80cc810 0x7ff5f8143590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:08.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:08.361+0000 7ff5fe800640 1 --2- 172.21.15.16:0/3152569279 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff5f80b4740 0x7ff5f815b3b0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:08.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:08.361+0000 7ff5fe800640 1 -- 172.21.15.16:0/3152569279 >> 172.21.15.16:0/3152569279 conn(0x7ff5f80b3c20 msgr2=0x7ff5f8157500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:04:08.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:08.361+0000 7ff5fe800640 1 -- 172.21.15.16:0/3152569279 shutdown_connections 2024-03-31T15:04:08.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:08.361+0000 7ff5fe800640 1 -- 172.21.15.16:0/3152569279 wait complete. 2024-03-31T15:04:08.373 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-31T15:03:22.550787+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-31T15:03:34.488425+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-31T15:03:35.299116+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-31T15:04:08.373 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-31T15:04:08.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:08.525+0000 7fd08e800640 1 Processor -- start 2024-03-31T15:04:08.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:08.525+0000 7fd08e800640 1 -- start start 2024-03-31T15:04:08.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:08.525+0000 7fd08e800640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fd08815e6e0 0x7fd0880b4f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:08.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:08.525+0000 7fd08e800640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fd088151ad0 0x7fd0880cf2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:08.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:08.525+0000 7fd08e800640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd0880b34d0 con 0x7fd08815eac0 2024-03-31T15:04:08.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:08.525+0000 7fd08e800640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7fd08815f300 con 0x7fd088151ad0 2024-03-31T15:04:08.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:08.525+0000 7fd08e800640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7fd0880b40e0 con 0x7fd08815e6e0 2024-03-31T15:04:08.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:08.525+0000 7fd087e00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fd088151ad0 0x7fd0880cf2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:08.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:08.525+0000 7fd087400640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fd08815e6e0 0x7fd0880b4f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:08.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:08.525+0000 7fd087e00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fd088151ad0 0x7fd0880cf2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60340/0 (socket says 172.21.15.16:60340) 2024-03-31T15:04:08.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:08.525+0000 7fd087400640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fd08815e6e0 0x7fd0880b4f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.16:44304/0 (socket says 172.21.15.16:44304) 2024-03-31T15:04:08.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:08.525+0000 7fd087e00640 1 -- 172.21.15.16:0/1796134558 learned_addr learned my addr 172.21.15.16:0/1796134558 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:04:08.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:08.525+0000 7fd087e00640 1 -- 172.21.15.16:0/1796134558 >> v2:172.21.15.89:3300/0 conn(0x7fd08815e6e0 msgr2=0x7fd0880b4f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:04:08.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:08.525+0000 7fd087e00640 1 --2- 172.21.15.16:0/1796134558 >> v2:172.21.15.89:3300/0 conn(0x7fd08815e6e0 0x7fd0880b4f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:08.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:08.525+0000 7fd087e00640 1 -- 172.21.15.16:0/1796134558 >> v1:172.21.15.89:6789/0 conn(0x7fd08815eac0 legacy=0x7fd0881513c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:08.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:08.525+0000 7fd087e00640 1 -- 172.21.15.16:0/1796134558 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0880b37b0 con 0x7fd088151ad0 2024-03-31T15:04:08.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:08.525+0000 7fd086a00640 1 -- 172.21.15.16:0/1796134558 <== mon.1 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1262051450 0 0) 0x7fd0880b34d0 con 0x7fd08815eac0 2024-03-31T15:04:08.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:08.529+0000 7fd087e00640 1 --2- 172.21.15.16:0/1796134558 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd088151ad0 0x7fd0880cf2c0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fd08806f550 tx=0x7fd08001c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f72c7071b066a6a1 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:08.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:08.529+0000 7fd086a00640 1 -- 172.21.15.16:0/1796134558 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0800a5880 con 0x7fd088151ad0 2024-03-31T15:04:08.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:08.529+0000 7fd086a00640 1 -- 172.21.15.16:0/1796134558 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd0800a5a20 con 0x7fd088151ad0 2024-03-31T15:04:08.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:08.529+0000 7fd086a00640 1 -- 172.21.15.16:0/1796134558 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0800a5df0 con 0x7fd088151ad0 2024-03-31T15:04:08.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:08.529+0000 7fd08e800640 1 -- 172.21.15.16:0/1796134558 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd088151ad0 msgr2=0x7fd0880cf2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:08.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:08.529+0000 7fd08e800640 1 --2- 172.21.15.16:0/1796134558 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd088151ad0 0x7fd0880cf2c0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fd08806f550 tx=0x7fd08001c9c0 comp rx=0 tx=0).stop 2024-03-31T15:04:08.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:08.529+0000 7fd08e800640 1 -- 172.21.15.16:0/1796134558 shutdown_connections 2024-03-31T15:04:08.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:08.529+0000 7fd08e800640 1 --2- 172.21.15.16:0/1796134558 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd088151ad0 0x7fd0880cf2c0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:08.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:08.529+0000 7fd08e800640 1 --2- 172.21.15.16:0/1796134558 >> v2:172.21.15.89:3300/0 conn(0x7fd08815e6e0 0x7fd0880b4f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:08.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:08.529+0000 7fd08e800640 1 -- 172.21.15.16:0/1796134558 >> 172.21.15.16:0/1796134558 conn(0x7fd08806ee40 msgr2=0x7fd08806b620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:04:08.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:08.529+0000 7fd08e800640 1 -- 172.21.15.16:0/1796134558 shutdown_connections 2024-03-31T15:04:08.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:08.529+0000 7fd08e800640 1 -- 172.21.15.16:0/1796134558 wait complete. 2024-03-31T15:04:08.525 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:08.533+0000 7fd08e800640 1 Processor -- start 2024-03-31T15:04:08.525 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:08.533+0000 7fd08e800640 1 -- start start 2024-03-31T15:04:08.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:08.533+0000 7fd08e800640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd088151ad0 0x7fd08815f9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:08.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:08.533+0000 7fd08e800640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fd08815e6e0 0x7fd08813e1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:08.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:08.533+0000 7fd08e800640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd08815eac0 0x7fd08815ca20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:08.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:08.533+0000 7fd08e800640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fd0880b55d0 con 0x7fd088151ad0 2024-03-31T15:04:08.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:08.533+0000 7fd08e800640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fd0880b5450 con 0x7fd08815eac0 2024-03-31T15:04:08.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:08.533+0000 7fd08e800640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fd08806d4d0 con 0x7fd08815e6e0 2024-03-31T15:04:08.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:08.533+0000 7fd087400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd088151ad0 0x7fd08815f9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:08.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:08.533+0000 7fd087400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd088151ad0 0x7fd08815f9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60342/0 (socket says 172.21.15.16:60342) 2024-03-31T15:04:08.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:08.533+0000 7fd087400640 1 -- 172.21.15.16:0/3279817699 learned_addr learned my addr 172.21.15.16:0/3279817699 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:04:08.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:08.533+0000 7fd07fe00640 1 --2- 172.21.15.16:0/3279817699 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fd08815e6e0 0x7fd08813e1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:08.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:08.533+0000 7fd087e00640 1 --2- 172.21.15.16:0/3279817699 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd08815eac0 0x7fd08815ca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:08.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:08.533+0000 7fd087400640 1 -- 172.21.15.16:0/3279817699 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fd08815e6e0 msgr2=0x7fd08813e1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:08.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:08.533+0000 7fd087400640 1 --2- 172.21.15.16:0/3279817699 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fd08815e6e0 0x7fd08813e1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:08.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:08.533+0000 7fd087400640 1 -- 172.21.15.16:0/3279817699 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd08815eac0 msgr2=0x7fd08815ca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:08.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:08.533+0000 7fd087400640 1 --2- 172.21.15.16:0/3279817699 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd08815eac0 0x7fd08815ca20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:08.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:08.533+0000 7fd087400640 1 -- 172.21.15.16:0/3279817699 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd08813e960 con 0x7fd088151ad0 2024-03-31T15:04:08.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:08.533+0000 7fd087e00640 1 --2- 172.21.15.16:0/3279817699 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd08815eac0 0x7fd08815ca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:04:08.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:08.533+0000 7fd087400640 1 --2- 172.21.15.16:0/3279817699 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd088151ad0 0x7fd08815f9c0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fd07800e960 tx=0x7fd07800ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:08.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:08.533+0000 7fd07f400640 1 -- 172.21.15.16:0/3279817699 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd078017070 con 0x7fd088151ad0 2024-03-31T15:04:08.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:08.533+0000 7fd07f400640 1 -- 172.21.15.16:0/3279817699 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd07800ca00 con 0x7fd088151ad0 2024-03-31T15:04:08.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:08.533+0000 7fd08e800640 1 -- 172.21.15.16:0/3279817699 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd08813ec40 con 0x7fd088151ad0 2024-03-31T15:04:08.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:08.533+0000 7fd07f400640 1 -- 172.21.15.16:0/3279817699 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd07800cdb0 con 0x7fd088151ad0 2024-03-31T15:04:08.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:08.533+0000 7fd08e800640 1 -- 172.21.15.16:0/3279817699 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd08813f200 con 0x7fd088151ad0 2024-03-31T15:04:08.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:08.537+0000 7fd08e800640 1 -- 172.21.15.16:0/3279817699 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd088069950 con 0x7fd088151ad0 2024-03-31T15:04:08.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:08.537+0000 7fd07f400640 1 -- 172.21.15.16:0/3279817699 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fd07801a070 con 0x7fd088151ad0 2024-03-31T15:04:08.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:08.537+0000 7fd07f400640 1 --2- 172.21.15.16:0/3279817699 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fd064068380 0x7fd06406a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:08.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:08.537+0000 7fd07f400640 1 -- 172.21.15.16:0/3279817699 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fd078086c80 con 0x7fd088151ad0 2024-03-31T15:04:08.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:08.537+0000 7fd07fe00640 1 --2- 172.21.15.16:0/3279817699 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fd064068380 0x7fd06406a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:08.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:08.537+0000 7fd07fe00640 1 --2- 172.21.15.16:0/3279817699 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fd064068380 0x7fd06406a840 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fd0880ccb80 tx=0x7fd070009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:08.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:08.541+0000 7fd07f400640 1 -- 172.21.15.16:0/3279817699 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fd078056c80 con 0x7fd088151ad0 2024-03-31T15:04:08.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:08.705+0000 7fd08e800640 1 -- 172.21.15.16:0/3279817699 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fd088072790 con 0x7fd088151ad0 2024-03-31T15:04:08.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:08.705+0000 7fd07f400640 1 -- 172.21.15.16:0/3279817699 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) ==== 93+0+10 (secure 0 0 0) 0x7fd07805af80 con 0x7fd088151ad0 2024-03-31T15:04:08.700 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2024-03-31T15:04:08.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:08.709+0000 7fd08e800640 1 -- 172.21.15.16:0/3279817699 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fd064068380 msgr2=0x7fd06406a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:08.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:08.709+0000 7fd08e800640 1 --2- 172.21.15.16:0/3279817699 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fd064068380 0x7fd06406a840 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fd0880ccb80 tx=0x7fd070009040 comp rx=0 tx=0).stop 2024-03-31T15:04:08.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:08.709+0000 7fd08e800640 1 -- 172.21.15.16:0/3279817699 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd088151ad0 msgr2=0x7fd08815f9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:08.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:08.709+0000 7fd08e800640 1 --2- 172.21.15.16:0/3279817699 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd088151ad0 0x7fd08815f9c0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fd07800e960 tx=0x7fd07800ee20 comp rx=0 tx=0).stop 2024-03-31T15:04:08.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:08.709+0000 7fd08e800640 1 -- 172.21.15.16:0/3279817699 shutdown_connections 2024-03-31T15:04:08.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:08.709+0000 7fd08e800640 1 --2- 172.21.15.16:0/3279817699 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fd064068380 0x7fd06406a840 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:08.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:08.709+0000 7fd08e800640 1 --2- 172.21.15.16:0/3279817699 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd08815eac0 0x7fd08815ca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:08.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:08.709+0000 7fd08e800640 1 --2- 172.21.15.16:0/3279817699 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fd08815e6e0 0x7fd08813e1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:08.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:08.709+0000 7fd08e800640 1 --2- 172.21.15.16:0/3279817699 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd088151ad0 0x7fd08815f9c0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:08.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:08.709+0000 7fd08e800640 1 -- 172.21.15.16:0/3279817699 >> 172.21.15.16:0/3279817699 conn(0x7fd08806ee40 msgr2=0x7fd088072bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:04:08.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:08.709+0000 7fd08e800640 1 -- 172.21.15.16:0/3279817699 shutdown_connections 2024-03-31T15:04:08.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:08.709+0000 7fd08e800640 1 -- 172.21.15.16:0/3279817699 wait complete. 2024-03-31T15:04:08.721 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-31T15:04:08.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:08.873+0000 7fc45fa00640 1 Processor -- start 2024-03-31T15:04:08.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:08.873+0000 7fc45fa00640 1 -- start start 2024-03-31T15:04:08.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:08.873+0000 7fc45fa00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fc45806dbe0 0x7fc458166060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:08.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:08.873+0000 7fc45fa00640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc45815fbc0 con 0x7fc45813dff0 2024-03-31T15:04:08.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:08.873+0000 7fc45fa00640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc45815ee00 con 0x7fc45813e3d0 2024-03-31T15:04:08.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:08.873+0000 7fc45fa00640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7fc45815f730 con 0x7fc45806dbe0 2024-03-31T15:04:08.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:08.873+0000 7fc45da00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fc45806dbe0 0x7fc458166060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:08.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:08.873+0000 7fc45d000640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fc45813dff0 0x7fc45806d4d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:46576/0 (socket says 172.21.15.16:46576) 2024-03-31T15:04:08.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:08.873+0000 7fc45d000640 1 -- 172.21.15.16:0/621933819 learned_addr learned my addr 172.21.15.16:0/621933819 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:04:08.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:08.873+0000 7fc45da00640 1 -- 172.21.15.16:0/621933819 >> v1:172.21.15.89:6789/0 conn(0x7fc45813e3d0 legacy=0x7fc4580cc9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:08.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:08.873+0000 7fc45da00640 1 -- 172.21.15.16:0/621933819 >> v1:172.21.15.16:6789/0 conn(0x7fc45813dff0 legacy=0x7fc45806d4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:08.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:08.873+0000 7fc45da00640 1 -- 172.21.15.16:0/621933819 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc45815f0e0 con 0x7fc45806dbe0 2024-03-31T15:04:08.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:08.877+0000 7fc45da00640 1 --2- 172.21.15.16:0/621933819 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc45806dbe0 0x7fc458166060 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fc450009d90 tx=0x7fc45001eaf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e04b9a798631d7 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:08.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:08.877+0000 7fc457400640 1 -- 172.21.15.16:0/621933819 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4500b4070 con 0x7fc45806dbe0 2024-03-31T15:04:08.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:08.877+0000 7fc457400640 1 -- 172.21.15.16:0/621933819 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc4500a79d0 con 0x7fc45806dbe0 2024-03-31T15:04:08.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:08.877+0000 7fc457400640 1 -- 172.21.15.16:0/621933819 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4500a7da0 con 0x7fc45806dbe0 2024-03-31T15:04:08.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:08.877+0000 7fc45fa00640 1 -- 172.21.15.16:0/621933819 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc45806dbe0 msgr2=0x7fc458166060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:08.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:08.877+0000 7fc45fa00640 1 --2- 172.21.15.16:0/621933819 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc45806dbe0 0x7fc458166060 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fc450009d90 tx=0x7fc45001eaf0 comp rx=0 tx=0).stop 2024-03-31T15:04:08.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:08.877+0000 7fc45fa00640 1 -- 172.21.15.16:0/621933819 shutdown_connections 2024-03-31T15:04:08.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:08.877+0000 7fc45fa00640 1 --2- 172.21.15.16:0/621933819 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc45806dbe0 0x7fc458166060 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:08.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:08.877+0000 7fc45fa00640 1 -- 172.21.15.16:0/621933819 >> 172.21.15.16:0/621933819 conn(0x7fc45806ee40 msgr2=0x7fc45806b620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:04:08.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:08.877+0000 7fc45fa00640 1 -- 172.21.15.16:0/621933819 shutdown_connections 2024-03-31T15:04:08.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:08.877+0000 7fc45fa00640 1 -- 172.21.15.16:0/621933819 wait complete. 2024-03-31T15:04:08.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:08.881+0000 7fc45fa00640 1 Processor -- start 2024-03-31T15:04:08.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:08.881+0000 7fc45fa00640 1 -- start start 2024-03-31T15:04:08.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:08.881+0000 7fc45fa00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fc45806dbe0 0x7fc4580aa2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:08.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:08.881+0000 7fc45fa00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc45813dff0 0x7fc4580ae280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:08.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:08.881+0000 7fc45fa00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc45813e3d0 0x7fc4581ffa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:08.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:08.881+0000 7fc45fa00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fc4580720f0 con 0x7fc45813dff0 2024-03-31T15:04:08.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:08.881+0000 7fc45fa00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fc458071f70 con 0x7fc45813e3d0 2024-03-31T15:04:08.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:08.881+0000 7fc45fa00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fc458072270 con 0x7fc45806dbe0 2024-03-31T15:04:08.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:08.881+0000 7fc457e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc45813dff0 0x7fc4580ae280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:08.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:08.881+0000 7fc457e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc45813dff0 0x7fc4580ae280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60346/0 (socket says 172.21.15.16:60346) 2024-03-31T15:04:08.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:08.881+0000 7fc457e00640 1 -- 172.21.15.16:0/286537147 learned_addr learned my addr 172.21.15.16:0/286537147 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:04:08.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:08.881+0000 7fc45d000640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fc45806dbe0 0x7fc4580aa2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:08.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:08.881+0000 7fc45da00640 1 --2- 172.21.15.16:0/286537147 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc45813e3d0 0x7fc4581ffa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:08.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:08.881+0000 7fc457e00640 1 -- 172.21.15.16:0/286537147 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fc45806dbe0 msgr2=0x7fc4580aa2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:08.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:08.881+0000 7fc457e00640 1 --2- 172.21.15.16:0/286537147 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fc45806dbe0 0x7fc4580aa2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:08.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:08.881+0000 7fc457e00640 1 -- 172.21.15.16:0/286537147 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc45813e3d0 msgr2=0x7fc4581ffa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:08.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:08.881+0000 7fc457e00640 1 --2- 172.21.15.16:0/286537147 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc45813e3d0 0x7fc4581ffa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:08.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:08.881+0000 7fc457e00640 1 -- 172.21.15.16:0/286537147 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4580a9560 con 0x7fc45813dff0 2024-03-31T15:04:08.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:08.881+0000 7fc457e00640 1 --2- 172.21.15.16:0/286537147 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc45813dff0 0x7fc4580ae280 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fc44400c370 tx=0x7fc44400c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:08.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:08.881+0000 7fc454c00640 1 -- 172.21.15.16:0/286537147 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4440052d0 con 0x7fc45813dff0 2024-03-31T15:04:08.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:08.881+0000 7fc454c00640 1 -- 172.21.15.16:0/286537147 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc444005470 con 0x7fc45813dff0 2024-03-31T15:04:08.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:08.881+0000 7fc45fa00640 1 -- 172.21.15.16:0/286537147 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc4580aa7f0 con 0x7fc45813dff0 2024-03-31T15:04:08.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:08.881+0000 7fc454c00640 1 -- 172.21.15.16:0/286537147 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc444007260 con 0x7fc45813dff0 2024-03-31T15:04:08.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:08.881+0000 7fc45fa00640 1 -- 172.21.15.16:0/286537147 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc4580a9ab0 con 0x7fc45813dff0 2024-03-31T15:04:08.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:08.881+0000 7fc45fa00640 1 -- 172.21.15.16:0/286537147 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc458069950 con 0x7fc45813dff0 2024-03-31T15:04:08.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:08.885+0000 7fc454c00640 1 -- 172.21.15.16:0/286537147 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fc444007400 con 0x7fc45813dff0 2024-03-31T15:04:08.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:08.885+0000 7fc454c00640 1 --2- 172.21.15.16:0/286537147 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fc438068380 0x7fc43806a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:08.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:08.885+0000 7fc454c00640 1 -- 172.21.15.16:0/286537147 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fc444085dc0 con 0x7fc45813dff0 2024-03-31T15:04:08.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:08.885+0000 7fc45d000640 1 --2- 172.21.15.16:0/286537147 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fc438068380 0x7fc43806a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:08.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:08.885+0000 7fc45d000640 1 --2- 172.21.15.16:0/286537147 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fc438068380 0x7fc43806a840 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fc44c005890 tx=0x7fc44c009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:08.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:08.889+0000 7fc454c00640 1 -- 172.21.15.16:0/286537147 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fc444055d20 con 0x7fc45813dff0 2024-03-31T15:04:09.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:09.057+0000 7fc45fa00640 1 -- 172.21.15.16:0/286537147 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fc4580cc760 con 0x7fc45813dff0 2024-03-31T15:04:09.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:09.057+0000 7fc454c00640 1 -- 172.21.15.16:0/286537147 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) ==== 104+0+11 (secure 0 0 0) 0x7fc44405c030 con 0x7fc45813dff0 2024-03-31T15:04:09.050 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2024-03-31T15:04:09.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:09.061+0000 7fc45fa00640 1 -- 172.21.15.16:0/286537147 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fc438068380 msgr2=0x7fc43806a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:09.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:09.061+0000 7fc45fa00640 1 --2- 172.21.15.16:0/286537147 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fc438068380 0x7fc43806a840 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fc44c005890 tx=0x7fc44c009040 comp rx=0 tx=0).stop 2024-03-31T15:04:09.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:09.061+0000 7fc45fa00640 1 -- 172.21.15.16:0/286537147 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc45813dff0 msgr2=0x7fc4580ae280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:09.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:09.061+0000 7fc45fa00640 1 --2- 172.21.15.16:0/286537147 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc45813dff0 0x7fc4580ae280 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fc44400c370 tx=0x7fc44400c830 comp rx=0 tx=0).stop 2024-03-31T15:04:09.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:09.061+0000 7fc45fa00640 1 -- 172.21.15.16:0/286537147 shutdown_connections 2024-03-31T15:04:09.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:09.061+0000 7fc45fa00640 1 --2- 172.21.15.16:0/286537147 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fc438068380 0x7fc43806a840 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:09.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:09.061+0000 7fc45fa00640 1 --2- 172.21.15.16:0/286537147 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc45813e3d0 0x7fc4581ffa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:09.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:09.061+0000 7fc45fa00640 1 --2- 172.21.15.16:0/286537147 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc45813dff0 0x7fc4580ae280 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:09.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:09.061+0000 7fc45fa00640 1 --2- 172.21.15.16:0/286537147 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fc45806dbe0 0x7fc4580aa2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:09.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:09.061+0000 7fc45fa00640 1 -- 172.21.15.16:0/286537147 >> 172.21.15.16:0/286537147 conn(0x7fc45806ee40 msgr2=0x7fc45815ddc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:04:09.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:09.061+0000 7fc45fa00640 1 -- 172.21.15.16:0/286537147 shutdown_connections 2024-03-31T15:04:09.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:09.061+0000 7fc45fa00640 1 -- 172.21.15.16:0/286537147 wait complete. 2024-03-31T15:04:09.072 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-31T15:04:09.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:09.221+0000 7ff5d7000640 1 Processor -- start 2024-03-31T15:04:09.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:09.221+0000 7ff5d7000640 1 -- start start 2024-03-31T15:04:09.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:09.225+0000 7ff5d7000640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff5d0070030 0x7ff5d0143c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:09.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:09.225+0000 7ff5d7000640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff5d0144980 con 0x7ff5d0141590 2024-03-31T15:04:09.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:09.225+0000 7ff5d7000640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff5d0161fe0 con 0x7ff5d01411b0 2024-03-31T15:04:09.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:09.225+0000 7ff5d7000640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7ff5d015f910 con 0x7ff5d0070030 2024-03-31T15:04:09.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:09.225+0000 7ff5d5000640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7ff5d0141590 0x7ff5d0144170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:46588/0 (socket says 172.21.15.16:46588) 2024-03-31T15:04:09.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:09.225+0000 7ff5d5000640 1 -- 172.21.15.16:0/3484529720 learned_addr learned my addr 172.21.15.16:0/3484529720 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:04:09.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:09.225+0000 7ff5c7400640 1 --2- 172.21.15.16:0/3484529720 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff5d0070030 0x7ff5d0143c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:09.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:09.225+0000 7ff5cf400640 1 -- 172.21.15.16:0/3484529720 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2880031116 0 0) 0x7ff5d0144980 con 0x7ff5d0141590 2024-03-31T15:04:09.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:09.225+0000 7ff5cf400640 1 -- 172.21.15.16:0/3484529720 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff5b4003530 con 0x7ff5d0141590 2024-03-31T15:04:09.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:09.225+0000 7ff5cf400640 1 -- 172.21.15.16:0/3484529720 <== mon.1 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1202181998 0 0) 0x7ff5d0161fe0 con 0x7ff5d01411b0 2024-03-31T15:04:09.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:09.225+0000 7ff5c7400640 1 -- 172.21.15.16:0/3484529720 >> v1:172.21.15.89:6789/0 conn(0x7ff5d01411b0 legacy=0x7ff5d006f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:09.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:09.225+0000 7ff5c7400640 1 -- 172.21.15.16:0/3484529720 >> v1:172.21.15.16:6789/0 conn(0x7ff5d0141590 legacy=0x7ff5d0144170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:09.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:09.225+0000 7ff5c7400640 1 -- 172.21.15.16:0/3484529720 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5d01622c0 con 0x7ff5d0070030 2024-03-31T15:04:09.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:09.229+0000 7ff5c7400640 1 --2- 172.21.15.16:0/3484529720 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff5d0070030 0x7ff5d0143c30 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7ff5b80097f0 tx=0x7ff5b801c4e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bcf5b133d8b5a40c server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:09.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:09.229+0000 7ff5cf400640 1 -- 172.21.15.16:0/3484529720 <== mon.2 v2:172.21.15.89:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5b80b2070 con 0x7ff5d0070030 2024-03-31T15:04:09.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:09.229+0000 7ff5cf400640 1 -- 172.21.15.16:0/3484529720 <== mon.2 v2:172.21.15.89:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff5b80a9070 con 0x7ff5d0070030 2024-03-31T15:04:09.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:09.229+0000 7ff5cf400640 1 -- 172.21.15.16:0/3484529720 <== mon.2 v2:172.21.15.89:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5b80ad880 con 0x7ff5d0070030 2024-03-31T15:04:09.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:09.229+0000 7ff5d7000640 1 -- 172.21.15.16:0/3484529720 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff5d0070030 msgr2=0x7ff5d0143c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:09.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:09.229+0000 7ff5d7000640 1 --2- 172.21.15.16:0/3484529720 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff5d0070030 0x7ff5d0143c30 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7ff5b80097f0 tx=0x7ff5b801c4e0 comp rx=0 tx=0).stop 2024-03-31T15:04:09.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:09.229+0000 7ff5d7000640 1 -- 172.21.15.16:0/3484529720 shutdown_connections 2024-03-31T15:04:09.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:09.229+0000 7ff5d7000640 1 --2- 172.21.15.16:0/3484529720 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff5d0070030 0x7ff5d0143c30 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:09.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:09.229+0000 7ff5d7000640 1 -- 172.21.15.16:0/3484529720 >> 172.21.15.16:0/3484529720 conn(0x7ff5d0067930 msgr2=0x7ff5d00b3ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:04:09.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:09.229+0000 7ff5d7000640 1 -- 172.21.15.16:0/3484529720 shutdown_connections 2024-03-31T15:04:09.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:09.229+0000 7ff5d7000640 1 -- 172.21.15.16:0/3484529720 wait complete. 2024-03-31T15:04:09.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:09.229+0000 7ff5d7000640 1 Processor -- start 2024-03-31T15:04:09.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:09.229+0000 7ff5d7000640 1 -- start start 2024-03-31T15:04:09.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:09.229+0000 7ff5d7000640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff5d0070030 0x7ff5d013fe30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:09.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:09.229+0000 7ff5d7000640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff5d01411b0 0x7ff5d0168810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:09.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:09.229+0000 7ff5d7000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff5d0141590 0x7ff5d015bf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:09.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:09.229+0000 7ff5d7000640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7ff5d016be80 con 0x7ff5d0141590 2024-03-31T15:04:09.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:09.229+0000 7ff5d7000640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7ff5d016bd00 con 0x7ff5d0070030 2024-03-31T15:04:09.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:09.229+0000 7ff5d7000640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7ff5d016c000 con 0x7ff5d01411b0 2024-03-31T15:04:09.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:09.233+0000 7ff5d5000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff5d0141590 0x7ff5d015bf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:09.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:09.233+0000 7ff5cfe00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff5d0070030 0x7ff5d013fe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:09.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:09.233+0000 7ff5c7400640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff5d01411b0 0x7ff5d0168810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:09.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:09.233+0000 7ff5d5000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff5d0141590 0x7ff5d015bf60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60350/0 (socket says 172.21.15.16:60350) 2024-03-31T15:04:09.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:09.233+0000 7ff5d5000640 1 -- 172.21.15.16:0/3150595545 learned_addr learned my addr 172.21.15.16:0/3150595545 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:04:09.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:09.233+0000 7ff5d5000640 1 -- 172.21.15.16:0/3150595545 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff5d01411b0 msgr2=0x7ff5d0168810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:09.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:09.233+0000 7ff5d5000640 1 --2- 172.21.15.16:0/3150595545 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff5d01411b0 0x7ff5d0168810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:09.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:09.233+0000 7ff5d5000640 1 -- 172.21.15.16:0/3150595545 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff5d0070030 msgr2=0x7ff5d013fe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:09.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:09.233+0000 7ff5d5000640 1 --2- 172.21.15.16:0/3150595545 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff5d0070030 0x7ff5d013fe30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:09.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:09.233+0000 7ff5d5000640 1 -- 172.21.15.16:0/3150595545 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5d0167770 con 0x7ff5d0141590 2024-03-31T15:04:09.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:09.233+0000 7ff5d5000640 1 --2- 172.21.15.16:0/3150595545 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff5d0141590 0x7ff5d015bf60 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7ff5c800caa0 tx=0x7ff5c800cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:09.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:09.233+0000 7ff5ccc00640 1 -- 172.21.15.16:0/3150595545 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5c8017070 con 0x7ff5d0141590 2024-03-31T15:04:09.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:09.233+0000 7ff5ccc00640 1 -- 172.21.15.16:0/3150595545 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff5c800ac00 con 0x7ff5d0141590 2024-03-31T15:04:09.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:09.233+0000 7ff5ccc00640 1 -- 172.21.15.16:0/3150595545 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5c8012400 con 0x7ff5d0141590 2024-03-31T15:04:09.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:09.233+0000 7ff5d7000640 1 -- 172.21.15.16:0/3150595545 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff5d0167a50 con 0x7ff5d0141590 2024-03-31T15:04:09.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:09.233+0000 7ff5d7000640 1 -- 172.21.15.16:0/3150595545 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff5d0168010 con 0x7ff5d0141590 2024-03-31T15:04:09.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:09.233+0000 7ff5d7000640 1 -- 172.21.15.16:0/3150595545 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff5d0069950 con 0x7ff5d0141590 2024-03-31T15:04:09.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:09.233+0000 7ff5ccc00640 1 -- 172.21.15.16:0/3150595545 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7ff5c801a070 con 0x7ff5d0141590 2024-03-31T15:04:09.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:09.237+0000 7ff5ccc00640 1 --2- 172.21.15.16:0/3150595545 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7ff5ac068380 0x7ff5ac06a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:09.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:09.237+0000 7ff5ccc00640 1 -- 172.21.15.16:0/3150595545 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7ff5c8086700 con 0x7ff5d0141590 2024-03-31T15:04:09.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:09.237+0000 7ff5cfe00640 1 --2- 172.21.15.16:0/3150595545 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7ff5ac068380 0x7ff5ac06a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:09.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:09.237+0000 7ff5cfe00640 1 --2- 172.21.15.16:0/3150595545 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7ff5ac068380 0x7ff5ac06a840 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7ff5d0070ab0 tx=0x7ff5c0009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:09.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:09.241+0000 7ff5ccc00640 1 -- 172.21.15.16:0/3150595545 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7ff5c80565d0 con 0x7ff5d0141590 2024-03-31T15:04:09.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:09.401+0000 7ff5d7000640 1 -- 172.21.15.16:0/3150595545 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7ff5d006fac0 con 0x7ff5d0141590 2024-03-31T15:04:09.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:09.401+0000 7ff5ccc00640 1 -- 172.21.15.16:0/3150595545 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) ==== 100+0+11 (secure 0 0 0) 0x7ff5c805a8d0 con 0x7ff5d0141590 2024-03-31T15:04:09.396 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 64 2024-03-31T15:04:09.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:09.405+0000 7ff5d7000640 1 -- 172.21.15.16:0/3150595545 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7ff5ac068380 msgr2=0x7ff5ac06a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:09.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:09.405+0000 7ff5d7000640 1 --2- 172.21.15.16:0/3150595545 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7ff5ac068380 0x7ff5ac06a840 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7ff5d0070ab0 tx=0x7ff5c0009040 comp rx=0 tx=0).stop 2024-03-31T15:04:09.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:09.405+0000 7ff5d7000640 1 -- 172.21.15.16:0/3150595545 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff5d0141590 msgr2=0x7ff5d015bf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:09.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:09.405+0000 7ff5d7000640 1 --2- 172.21.15.16:0/3150595545 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff5d0141590 0x7ff5d015bf60 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7ff5c800caa0 tx=0x7ff5c800cf60 comp rx=0 tx=0).stop 2024-03-31T15:04:09.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:09.405+0000 7ff5d7000640 1 -- 172.21.15.16:0/3150595545 shutdown_connections 2024-03-31T15:04:09.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:09.405+0000 7ff5d7000640 1 --2- 172.21.15.16:0/3150595545 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7ff5ac068380 0x7ff5ac06a840 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:09.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:09.405+0000 7ff5d7000640 1 --2- 172.21.15.16:0/3150595545 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff5d0141590 0x7ff5d015bf60 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:09.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:09.405+0000 7ff5d7000640 1 --2- 172.21.15.16:0/3150595545 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff5d01411b0 0x7ff5d0168810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:09.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:09.405+0000 7ff5d7000640 1 --2- 172.21.15.16:0/3150595545 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff5d0070030 0x7ff5d013fe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:09.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:09.405+0000 7ff5d7000640 1 -- 172.21.15.16:0/3150595545 >> 172.21.15.16:0/3150595545 conn(0x7ff5d0067930 msgr2=0x7ff5d0143b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:04:09.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:09.405+0000 7ff5d7000640 1 -- 172.21.15.16:0/3150595545 shutdown_connections 2024-03-31T15:04:09.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:09.405+0000 7ff5d7000640 1 -- 172.21.15.16:0/3150595545 wait complete. 2024-03-31T15:04:09.417 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_apply_tag' 2024-03-31T15:04:09.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:09.565+0000 7f616f200640 1 Processor -- start 2024-03-31T15:04:09.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:09.565+0000 7f616f200640 1 -- start start 2024-03-31T15:04:09.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:09.565+0000 7f616f200640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f616806c630 0x7f616806ca10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:09.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:09.565+0000 7f616f200640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f61680ae2e0 0x7f616806dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:09.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:09.565+0000 7f616f200640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f61680d0390 con 0x7f61680ae740 2024-03-31T15:04:09.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:09.565+0000 7f616f200640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f616806cf50 con 0x7f61680ae2e0 2024-03-31T15:04:09.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:09.565+0000 7f616f200640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f61680b41b0 con 0x7f616806c630 2024-03-31T15:04:09.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:09.565+0000 7f616d400640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f61680ae2e0 0x7f616806dbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:09.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:09.565+0000 7f616e800640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f61680ae740 0x7f61680b4f70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:46602/0 (socket says 172.21.15.16:46602) 2024-03-31T15:04:09.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:09.565+0000 7f616e800640 1 -- 172.21.15.16:0/482092974 learned_addr learned my addr 172.21.15.16:0/482092974 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:04:09.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:09.565+0000 7f616de00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f616806c630 0x7f616806ca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:09.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:09.565+0000 7f616d400640 1 -- 172.21.15.16:0/482092974 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f616806c630 msgr2=0x7f616806ca10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:04:09.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:09.565+0000 7f616d400640 1 --2- 172.21.15.16:0/482092974 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f616806c630 0x7f616806ca10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:09.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:09.565+0000 7f616d400640 1 -- 172.21.15.16:0/482092974 >> v1:172.21.15.16:6789/0 conn(0x7f61680ae740 legacy=0x7f61680b4f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:09.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:09.565+0000 7f616d400640 1 -- 172.21.15.16:0/482092974 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61680b4490 con 0x7f61680ae2e0 2024-03-31T15:04:09.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:09.565+0000 7f616de00640 1 --2- 172.21.15.16:0/482092974 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f616806c630 0x7f616806ca10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:04:09.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:09.569+0000 7f616d400640 1 --2- 172.21.15.16:0/482092974 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f61680ae2e0 0x7f616806dbd0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f61580097c0 tx=0x7f615801c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=66c91b861f393869 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:09.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:09.569+0000 7f616ca00640 1 -- 172.21.15.16:0/482092974 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61580b0070 con 0x7f61680ae2e0 2024-03-31T15:04:09.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:09.569+0000 7f616ca00640 1 -- 172.21.15.16:0/482092974 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f61580ab600 con 0x7f61680ae2e0 2024-03-31T15:04:09.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:09.569+0000 7f616ca00640 1 -- 172.21.15.16:0/482092974 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61580ab9d0 con 0x7f61680ae2e0 2024-03-31T15:04:09.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:09.569+0000 7f616f200640 1 -- 172.21.15.16:0/482092974 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f61680ae2e0 msgr2=0x7f616806dbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:09.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:09.569+0000 7f616f200640 1 --2- 172.21.15.16:0/482092974 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f61680ae2e0 0x7f616806dbd0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f61580097c0 tx=0x7f615801c4a0 comp rx=0 tx=0).stop 2024-03-31T15:04:09.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:09.569+0000 7f616f200640 1 -- 172.21.15.16:0/482092974 shutdown_connections 2024-03-31T15:04:09.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:09.569+0000 7f616f200640 1 --2- 172.21.15.16:0/482092974 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f61680ae2e0 0x7f616806dbd0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:09.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:09.569+0000 7f616f200640 1 --2- 172.21.15.16:0/482092974 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f616806c630 0x7f616806ca10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:09.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:09.569+0000 7f616f200640 1 -- 172.21.15.16:0/482092974 >> 172.21.15.16:0/482092974 conn(0x7f6168071c40 msgr2=0x7f6168072050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:04:09.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:09.569+0000 7f616f200640 1 -- 172.21.15.16:0/482092974 shutdown_connections 2024-03-31T15:04:09.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:09.569+0000 7f616f200640 1 -- 172.21.15.16:0/482092974 wait complete. 2024-03-31T15:04:09.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:09.569+0000 7f616f200640 1 Processor -- start 2024-03-31T15:04:09.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:09.569+0000 7f616f200640 1 -- start start 2024-03-31T15:04:09.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:09.573+0000 7f616f200640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f616806c630 0x7f616814a0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:09.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:09.573+0000 7f616f200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f61680ae2e0 0x7f616815d9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:09.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:09.573+0000 7f616f200640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f61680ae740 0x7f6168144dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:09.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:09.573+0000 7f616f200640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f61680b5800 con 0x7f61680ae2e0 2024-03-31T15:04:09.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:09.573+0000 7f616f200640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f61680b5680 con 0x7f616806c630 2024-03-31T15:04:09.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:09.573+0000 7f616f200640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f616816b9d0 con 0x7f61680ae740 2024-03-31T15:04:09.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:09.573+0000 7f616d400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f61680ae2e0 0x7f616815d9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:09.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:09.573+0000 7f616d400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f61680ae2e0 0x7f616815d9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60364/0 (socket says 172.21.15.16:60364) 2024-03-31T15:04:09.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:09.573+0000 7f616d400640 1 -- 172.21.15.16:0/267066629 learned_addr learned my addr 172.21.15.16:0/267066629 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:04:09.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:09.573+0000 7f616de00640 1 --2- 172.21.15.16:0/267066629 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f616806c630 0x7f616814a0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:09.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:09.573+0000 7f616e800640 1 --2- 172.21.15.16:0/267066629 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f61680ae740 0x7f6168144dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:09.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:09.573+0000 7f616d400640 1 -- 172.21.15.16:0/267066629 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f61680ae740 msgr2=0x7f6168144dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:09.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:09.573+0000 7f616d400640 1 --2- 172.21.15.16:0/267066629 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f61680ae740 0x7f6168144dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:09.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:09.573+0000 7f616d400640 1 -- 172.21.15.16:0/267066629 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f616806c630 msgr2=0x7f616814a0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:09.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:09.573+0000 7f616d400640 1 --2- 172.21.15.16:0/267066629 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f616806c630 0x7f616814a0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:09.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:09.573+0000 7f616d400640 1 -- 172.21.15.16:0/267066629 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61681587c0 con 0x7f61680ae2e0 2024-03-31T15:04:09.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:09.573+0000 7f616d400640 1 --2- 172.21.15.16:0/267066629 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f61680ae2e0 0x7f616815d9d0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f61580ae040 tx=0x7f6158019240 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:09.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:09.573+0000 7f6162000640 1 -- 172.21.15.16:0/267066629 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61580ba070 con 0x7f61680ae2e0 2024-03-31T15:04:09.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:09.573+0000 7f6162000640 1 -- 172.21.15.16:0/267066629 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f61580b43f0 con 0x7f61680ae2e0 2024-03-31T15:04:09.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:09.573+0000 7f6162000640 1 -- 172.21.15.16:0/267066629 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61580b0040 con 0x7f61680ae2e0 2024-03-31T15:04:09.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:09.573+0000 7f616f200640 1 -- 172.21.15.16:0/267066629 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f616814a6a0 con 0x7f61680ae2e0 2024-03-31T15:04:09.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:09.573+0000 7f616f200640 1 -- 172.21.15.16:0/267066629 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f616814ac60 con 0x7f61680ae2e0 2024-03-31T15:04:09.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:09.573+0000 7f616f200640 1 -- 172.21.15.16:0/267066629 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f616804f8c0 con 0x7f61680ae2e0 2024-03-31T15:04:09.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:09.577+0000 7f6162000640 1 -- 172.21.15.16:0/267066629 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f61580bd070 con 0x7f61680ae2e0 2024-03-31T15:04:09.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:09.577+0000 7f6162000640 1 --2- 172.21.15.16:0/267066629 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f614c068380 0x7f614c06a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:09.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:09.577+0000 7f6162000640 1 -- 172.21.15.16:0/267066629 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f615811f8d0 con 0x7f61680ae2e0 2024-03-31T15:04:09.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:09.577+0000 7f616de00640 1 --2- 172.21.15.16:0/267066629 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f614c068380 0x7f614c06a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:09.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:09.577+0000 7f616de00640 1 --2- 172.21.15.16:0/267066629 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f614c068380 0x7f614c06a840 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f615c009580 tx=0x7f615c009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:09.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:09.581+0000 7f6162000640 1 -- 172.21.15.16:0/267066629 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f6158016090 con 0x7f61680ae2e0 2024-03-31T15:04:09.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:09.801+0000 7f616f200640 1 -- 172.21.15.16:0/267066629 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_apply_tag"]} v 0) -- 0x7f61680adc50 con 0x7f61680ae2e0 2024-03-31T15:04:10.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:10.061+0000 7f6162000640 1 -- 172.21.15.16:0/267066629 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_apply_tag"]}]=0 v52) ==== 145+0+0 (secure 0 0 0) 0x7f61580ef7a0 con 0x7f61680ae2e0 2024-03-31T15:04:10.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:10.065+0000 7f616f200640 1 -- 172.21.15.16:0/267066629 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f614c068380 msgr2=0x7f614c06a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:10.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:10.065+0000 7f616f200640 1 --2- 172.21.15.16:0/267066629 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f614c068380 0x7f614c06a840 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f615c009580 tx=0x7f615c009190 comp rx=0 tx=0).stop 2024-03-31T15:04:10.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:10.065+0000 7f616f200640 1 -- 172.21.15.16:0/267066629 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f61680ae2e0 msgr2=0x7f616815d9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:10.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:10.065+0000 7f616f200640 1 --2- 172.21.15.16:0/267066629 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f61680ae2e0 0x7f616815d9d0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f61580ae040 tx=0x7f6158019240 comp rx=0 tx=0).stop 2024-03-31T15:04:10.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:10.065+0000 7f616f200640 1 -- 172.21.15.16:0/267066629 shutdown_connections 2024-03-31T15:04:10.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:10.065+0000 7f616f200640 1 --2- 172.21.15.16:0/267066629 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f614c068380 0x7f614c06a840 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:10.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:10.065+0000 7f616f200640 1 --2- 172.21.15.16:0/267066629 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f61680ae740 0x7f6168144dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:10.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:10.065+0000 7f616f200640 1 --2- 172.21.15.16:0/267066629 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f61680ae2e0 0x7f616815d9d0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:10.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:10.065+0000 7f616f200640 1 --2- 172.21.15.16:0/267066629 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f616806c630 0x7f616814a0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:10.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:10.065+0000 7f616f200640 1 -- 172.21.15.16:0/267066629 >> 172.21.15.16:0/267066629 conn(0x7f6168071c40 msgr2=0x7f616814b060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:04:10.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:10.065+0000 7f616f200640 1 -- 172.21.15.16:0/267066629 shutdown_connections 2024-03-31T15:04:10.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:10.065+0000 7f616f200640 1 -- 172.21.15.16:0/267066629 wait complete. 2024-03-31T15:04:10.077 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-03-31T15:04:10.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:10.229+0000 7fe818e00640 1 Processor -- start 2024-03-31T15:04:10.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:10.229+0000 7fe818e00640 1 -- start start 2024-03-31T15:04:10.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:10.229+0000 7fe818e00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fe81406d4d0 0x7fe81406d930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:10.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:10.229+0000 7fe818e00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fe814071550 0x7fe8140719a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:10.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:10.229+0000 7fe818e00640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe81406de70 con 0x7fe8141567e0 2024-03-31T15:04:10.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:10.229+0000 7fe818e00640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7fe81415c7e0 con 0x7fe814071550 2024-03-31T15:04:10.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:10.229+0000 7fe818e00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fe8140736b0 con 0x7fe81406d4d0 2024-03-31T15:04:10.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:10.229+0000 7fe812800640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fe814071550 0x7fe8140719a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:10.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:10.229+0000 7fe811400640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fe81406d4d0 0x7fe81406d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:10.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:10.229+0000 7fe812800640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fe814071550 0x7fe8140719a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.16:44368/0 (socket says 172.21.15.16:44368) 2024-03-31T15:04:10.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:10.229+0000 7fe812800640 1 -- 172.21.15.16:0/2099233174 learned_addr learned my addr 172.21.15.16:0/2099233174 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:04:10.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:10.229+0000 7fe811400640 1 -- 172.21.15.16:0/2099233174 >> v2:172.21.15.89:3300/0 conn(0x7fe814071550 msgr2=0x7fe8140719a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:04:10.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:10.229+0000 7fe811400640 1 --2- 172.21.15.16:0/2099233174 >> v2:172.21.15.89:3300/0 conn(0x7fe814071550 0x7fe8140719a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:10.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:10.229+0000 7fe811400640 1 -- 172.21.15.16:0/2099233174 >> v1:172.21.15.89:6789/0 conn(0x7fe8141567e0 legacy=0x7fe814072c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:10.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:10.229+0000 7fe810a00640 1 -- 172.21.15.16:0/2099233174 <== mon.1 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3557973346 0 0) 0x7fe81406de70 con 0x7fe8141567e0 2024-03-31T15:04:10.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:10.229+0000 7fe811400640 1 -- 172.21.15.16:0/2099233174 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe81415caa0 con 0x7fe81406d4d0 2024-03-31T15:04:10.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:10.229+0000 7fe812800640 1 --2- 172.21.15.16:0/2099233174 >> v2:172.21.15.89:3300/0 conn(0x7fe814071550 0x7fe8140719a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:04:10.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:10.233+0000 7fe811400640 1 --2- 172.21.15.16:0/2099233174 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fe81406d4d0 0x7fe81406d930 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fe8000097c0 tx=0x7fe80001cf00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d943550bc1ff28d1 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:10.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:10.233+0000 7fe810a00640 1 -- 172.21.15.16:0/2099233174 <== mon.2 v2:172.21.15.89:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8000b1070 con 0x7fe81406d4d0 2024-03-31T15:04:10.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:10.233+0000 7fe810a00640 1 -- 172.21.15.16:0/2099233174 <== mon.2 v2:172.21.15.89:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe8000ac410 con 0x7fe81406d4d0 2024-03-31T15:04:10.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:10.233+0000 7fe810a00640 1 -- 172.21.15.16:0/2099233174 <== mon.2 v2:172.21.15.89:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8000ac7e0 con 0x7fe81406d4d0 2024-03-31T15:04:10.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:10.233+0000 7fe818e00640 1 -- 172.21.15.16:0/2099233174 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fe81406d4d0 msgr2=0x7fe81406d930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:10.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:10.233+0000 7fe818e00640 1 --2- 172.21.15.16:0/2099233174 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fe81406d4d0 0x7fe81406d930 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fe8000097c0 tx=0x7fe80001cf00 comp rx=0 tx=0).stop 2024-03-31T15:04:10.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:10.233+0000 7fe818e00640 1 -- 172.21.15.16:0/2099233174 shutdown_connections 2024-03-31T15:04:10.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:10.233+0000 7fe818e00640 1 --2- 172.21.15.16:0/2099233174 >> v2:172.21.15.89:3300/0 conn(0x7fe814071550 0x7fe8140719a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:10.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:10.233+0000 7fe818e00640 1 --2- 172.21.15.16:0/2099233174 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fe81406d4d0 0x7fe81406d930 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:10.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:10.233+0000 7fe818e00640 1 -- 172.21.15.16:0/2099233174 >> 172.21.15.16:0/2099233174 conn(0x7fe81406ee40 msgr2=0x7fe81406b620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:04:10.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:10.233+0000 7fe818e00640 1 -- 172.21.15.16:0/2099233174 shutdown_connections 2024-03-31T15:04:10.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:10.233+0000 7fe818e00640 1 -- 172.21.15.16:0/2099233174 wait complete. 2024-03-31T15:04:10.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:10.233+0000 7fe818e00640 1 Processor -- start 2024-03-31T15:04:10.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:10.237+0000 7fe818e00640 1 -- start start 2024-03-31T15:04:10.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:10.237+0000 7fe818e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe81406d4d0 0x7fe8141376d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:10.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:10.237+0000 7fe818e00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe814071550 0x7fe81415b770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:10.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:10.237+0000 7fe818e00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fe8141567e0 0x7fe81414a410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:10.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:10.237+0000 7fe818e00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fe8140afbb0 con 0x7fe81406d4d0 2024-03-31T15:04:10.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:10.237+0000 7fe818e00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fe8140afa30 con 0x7fe814071550 2024-03-31T15:04:10.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:10.237+0000 7fe818e00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fe8140722f0 con 0x7fe8141567e0 2024-03-31T15:04:10.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:10.237+0000 7fe811e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe81406d4d0 0x7fe8141376d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:10.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:10.237+0000 7fe811e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe81406d4d0 0x7fe8141376d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60370/0 (socket says 172.21.15.16:60370) 2024-03-31T15:04:10.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:10.237+0000 7fe811e00640 1 -- 172.21.15.16:0/1054658658 learned_addr learned my addr 172.21.15.16:0/1054658658 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:04:10.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:10.237+0000 7fe812800640 1 --2- 172.21.15.16:0/1054658658 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fe8141567e0 0x7fe81414a410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:10.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:10.237+0000 7fe811400640 1 --2- 172.21.15.16:0/1054658658 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe814071550 0x7fe81415b770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:10.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:10.237+0000 7fe811e00640 1 -- 172.21.15.16:0/1054658658 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fe8141567e0 msgr2=0x7fe81414a410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:10.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:10.237+0000 7fe811e00640 1 --2- 172.21.15.16:0/1054658658 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fe8141567e0 0x7fe81414a410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:10.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:10.237+0000 7fe811e00640 1 -- 172.21.15.16:0/1054658658 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe814071550 msgr2=0x7fe81415b770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:10.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:10.237+0000 7fe811e00640 1 --2- 172.21.15.16:0/1054658658 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe814071550 0x7fe81415b770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:10.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:10.237+0000 7fe811e00640 1 -- 172.21.15.16:0/1054658658 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe814137c10 con 0x7fe81406d4d0 2024-03-31T15:04:10.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:10.237+0000 7fe812800640 1 --2- 172.21.15.16:0/1054658658 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fe8141567e0 0x7fe81414a410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:04:10.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:10.237+0000 7fe811e00640 1 --2- 172.21.15.16:0/1054658658 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe81406d4d0 0x7fe8141376d0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fe80400c4d0 tx=0x7fe80400c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:10.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:10.237+0000 7fe80a000640 1 -- 172.21.15.16:0/1054658658 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe804017070 con 0x7fe81406d4d0 2024-03-31T15:04:10.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:10.237+0000 7fe80a000640 1 -- 172.21.15.16:0/1054658658 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe804005480 con 0x7fe81406d4d0 2024-03-31T15:04:10.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:10.237+0000 7fe80a000640 1 -- 172.21.15.16:0/1054658658 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe804003840 con 0x7fe81406d4d0 2024-03-31T15:04:10.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:10.237+0000 7fe818e00640 1 -- 172.21.15.16:0/1054658658 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe81415c110 con 0x7fe81406d4d0 2024-03-31T15:04:10.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:10.237+0000 7fe818e00640 1 -- 172.21.15.16:0/1054658658 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe81414f0c0 con 0x7fe81406d4d0 2024-03-31T15:04:10.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:10.237+0000 7fe818e00640 1 -- 172.21.15.16:0/1054658658 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe814069950 con 0x7fe81406d4d0 2024-03-31T15:04:10.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:10.241+0000 7fe80a000640 1 -- 172.21.15.16:0/1054658658 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fe80401a070 con 0x7fe81406d4d0 2024-03-31T15:04:10.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:10.241+0000 7fe80a000640 1 --2- 172.21.15.16:0/1054658658 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fe7f4068380 0x7fe7f406a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:10.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:10.241+0000 7fe80a000640 1 -- 172.21.15.16:0/1054658658 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fe804085d40 con 0x7fe81406d4d0 2024-03-31T15:04:10.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:10.241+0000 7fe811400640 1 --2- 172.21.15.16:0/1054658658 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fe7f4068380 0x7fe7f406a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:10.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:10.241+0000 7fe811400640 1 --2- 172.21.15.16:0/1054658658 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fe7f4068380 0x7fe7f406a840 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fe8000ae000 tx=0x7fe800008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:10.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:10.245+0000 7fe80a000640 1 -- 172.21.15.16:0/1054658658 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fe804055d60 con 0x7fe81406d4d0 2024-03-31T15:04:10.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:10.409+0000 7fe818e00640 1 -- 172.21.15.16:0/1054658658 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7fe81406e470 con 0x7fe81406d4d0 2024-03-31T15:04:10.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:10.413+0000 7fe80a000640 1 -- 172.21.15.16:0/1054658658 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v10)=0 v10) ==== 122+0+0 (secure 0 0 0) 0x7fe80405a060 con 0x7fe81406d4d0 2024-03-31T15:04:10.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:10.417+0000 7fe818e00640 1 -- 172.21.15.16:0/1054658658 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fe7f4068380 msgr2=0x7fe7f406a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:10.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:10.417+0000 7fe818e00640 1 --2- 172.21.15.16:0/1054658658 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fe7f4068380 0x7fe7f406a840 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fe8000ae000 tx=0x7fe800008040 comp rx=0 tx=0).stop 2024-03-31T15:04:10.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:10.417+0000 7fe818e00640 1 -- 172.21.15.16:0/1054658658 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe81406d4d0 msgr2=0x7fe8141376d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:10.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:10.417+0000 7fe818e00640 1 --2- 172.21.15.16:0/1054658658 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe81406d4d0 0x7fe8141376d0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fe80400c4d0 tx=0x7fe80400c990 comp rx=0 tx=0).stop 2024-03-31T15:04:10.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:10.417+0000 7fe818e00640 1 -- 172.21.15.16:0/1054658658 shutdown_connections 2024-03-31T15:04:10.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:10.417+0000 7fe818e00640 1 --2- 172.21.15.16:0/1054658658 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fe7f4068380 0x7fe7f406a840 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:10.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:10.417+0000 7fe818e00640 1 --2- 172.21.15.16:0/1054658658 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fe8141567e0 0x7fe81414a410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:10.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:10.417+0000 7fe818e00640 1 --2- 172.21.15.16:0/1054658658 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe814071550 0x7fe81415b770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:10.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:10.417+0000 7fe818e00640 1 --2- 172.21.15.16:0/1054658658 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe81406d4d0 0x7fe8141376d0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:10.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:10.417+0000 7fe818e00640 1 -- 172.21.15.16:0/1054658658 >> 172.21.15.16:0/1054658658 conn(0x7fe81406ee40 msgr2=0x7fe8140b1670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:04:10.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:10.417+0000 7fe818e00640 1 -- 172.21.15.16:0/1054658658 shutdown_connections 2024-03-31T15:04:10.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:10.417+0000 7fe818e00640 1 -- 172.21.15.16:0/1054658658 wait complete. 2024-03-31T15:04:10.429 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-31T15:04:10.430 DEBUG:teuthology.orchestra.run.smithi016:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-31T15:04:10.507 DEBUG:teuthology.orchestra.run.smithi089:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-31T15:04:10.589 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-31T15:04:10.589 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-03-31T15:04:10.589 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-31T15:04:10.598 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-03-31T15:04:10.598 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-31T15:04:10.652 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-31T15:04:10.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:10.845+0000 7efe28600640 1 Processor -- start 2024-03-31T15:04:10.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:10.845+0000 7efe28600640 1 -- start start 2024-03-31T15:04:10.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:10.849+0000 7efe28600640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7efe20145e40 0x7efe200cc740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:10.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:10.849+0000 7efe28600640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7efe200ccc80 0x7efe2006f920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:10.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:10.849+0000 7efe28600640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efe200d0920 con 0x7efe2006fe60 2024-03-31T15:04:10.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:10.849+0000 7efe28600640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7efe200d06d0 con 0x7efe200ccc80 2024-03-31T15:04:10.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:10.849+0000 7efe28600640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7efe200d0c00 con 0x7efe20145e40 2024-03-31T15:04:10.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:10.849+0000 7efe25c00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7efe20145e40 0x7efe200cc740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:10.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:10.849+0000 7efe25200640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7efe200ccc80 0x7efe2006f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:10.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:10.849+0000 7efe26600640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7efe2006fe60 0x7efe2015fdf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.16:60884/0 (socket says 172.21.15.16:60884) 2024-03-31T15:04:10.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:10.849+0000 7efe26600640 1 -- 172.21.15.16:0/4012360076 learned_addr learned my addr 172.21.15.16:0/4012360076 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:04:10.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:10.849+0000 7efe25c00640 1 -- 172.21.15.16:0/4012360076 >> v2:172.21.15.89:3300/0 conn(0x7efe200ccc80 msgr2=0x7efe2006f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:04:10.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:10.849+0000 7efe25c00640 1 --2- 172.21.15.16:0/4012360076 >> v2:172.21.15.89:3300/0 conn(0x7efe200ccc80 0x7efe2006f920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:10.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:10.849+0000 7efe25c00640 1 -- 172.21.15.16:0/4012360076 >> v1:172.21.15.89:6789/0 conn(0x7efe2006fe60 legacy=0x7efe2015fdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:10.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:10.849+0000 7efe25c00640 1 -- 172.21.15.16:0/4012360076 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efe200d1810 con 0x7efe20145e40 2024-03-31T15:04:10.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:10.849+0000 7efe25200640 1 --2- 172.21.15.16:0/4012360076 >> v2:172.21.15.89:3300/0 conn(0x7efe200ccc80 0x7efe2006f920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:04:10.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:10.853+0000 7efe25c00640 1 --2- 172.21.15.16:0/4012360076 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7efe20145e40 0x7efe200cc740 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7efe140097c0 tx=0x7efe1401c4a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6ac0c9a702ec7505 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:10.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:10.853+0000 7efe1fe00640 1 -- 172.21.15.16:0/4012360076 <== mon.2 v2:172.21.15.89:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe140b0070 con 0x7efe20145e40 2024-03-31T15:04:10.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:10.853+0000 7efe1fe00640 1 -- 172.21.15.16:0/4012360076 <== mon.2 v2:172.21.15.89:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efe140a4e50 con 0x7efe20145e40 2024-03-31T15:04:10.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:10.853+0000 7efe1fe00640 1 -- 172.21.15.16:0/4012360076 <== mon.2 v2:172.21.15.89:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe140ab820 con 0x7efe20145e40 2024-03-31T15:04:10.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:10.853+0000 7efe28600640 1 -- 172.21.15.16:0/4012360076 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7efe20145e40 msgr2=0x7efe200cc740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:10.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:10.853+0000 7efe28600640 1 --2- 172.21.15.16:0/4012360076 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7efe20145e40 0x7efe200cc740 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7efe140097c0 tx=0x7efe1401c4a0 comp rx=0 tx=0).stop 2024-03-31T15:04:10.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:10.853+0000 7efe28600640 1 -- 172.21.15.16:0/4012360076 shutdown_connections 2024-03-31T15:04:10.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:10.853+0000 7efe28600640 1 --2- 172.21.15.16:0/4012360076 >> v2:172.21.15.89:3300/0 conn(0x7efe200ccc80 0x7efe2006f920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:10.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:10.853+0000 7efe28600640 1 --2- 172.21.15.16:0/4012360076 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7efe20145e40 0x7efe200cc740 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:10.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:10.853+0000 7efe28600640 1 -- 172.21.15.16:0/4012360076 >> 172.21.15.16:0/4012360076 conn(0x7efe20067330 msgr2=0x7efe20067740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:04:10.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:10.853+0000 7efe28600640 1 -- 172.21.15.16:0/4012360076 shutdown_connections 2024-03-31T15:04:10.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:10.853+0000 7efe28600640 1 -- 172.21.15.16:0/4012360076 wait complete. 2024-03-31T15:04:10.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:10.853+0000 7efe28600640 1 Processor -- start 2024-03-31T15:04:10.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:10.853+0000 7efe28600640 1 -- start start 2024-03-31T15:04:10.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:10.857+0000 7efe28600640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7efe2006fe60 0x7efe20158aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:10.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:10.857+0000 7efe28600640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efe200ccc80 0x7efe20151fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:10.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:10.857+0000 7efe28600640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efe20145e40 0x7efe20141d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:10.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:10.857+0000 7efe28600640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7efe200b5240 con 0x7efe200ccc80 2024-03-31T15:04:10.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:10.857+0000 7efe28600640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7efe200b50c0 con 0x7efe20145e40 2024-03-31T15:04:10.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:10.857+0000 7efe28600640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7efe20169190 con 0x7efe2006fe60 2024-03-31T15:04:10.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:10.857+0000 7efe25200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efe200ccc80 0x7efe20151fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:10.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:10.857+0000 7efe25200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efe200ccc80 0x7efe20151fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60386/0 (socket says 172.21.15.16:60386) 2024-03-31T15:04:10.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:10.857+0000 7efe25200640 1 -- 172.21.15.16:0/2074569538 learned_addr learned my addr 172.21.15.16:0/2074569538 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:04:10.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:10.857+0000 7efe25c00640 1 --2- 172.21.15.16:0/2074569538 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7efe2006fe60 0x7efe20158aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:10.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:10.857+0000 7efe26600640 1 --2- 172.21.15.16:0/2074569538 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efe20145e40 0x7efe20141d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:10.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:10.857+0000 7efe25200640 1 -- 172.21.15.16:0/2074569538 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7efe2006fe60 msgr2=0x7efe20158aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:10.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:10.857+0000 7efe25200640 1 --2- 172.21.15.16:0/2074569538 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7efe2006fe60 0x7efe20158aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:10.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:10.857+0000 7efe25200640 1 -- 172.21.15.16:0/2074569538 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efe20145e40 msgr2=0x7efe20141d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:10.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:10.857+0000 7efe25200640 1 --2- 172.21.15.16:0/2074569538 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efe20145e40 0x7efe20141d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:10.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:10.857+0000 7efe25200640 1 -- 172.21.15.16:0/2074569538 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efe201524e0 con 0x7efe200ccc80 2024-03-31T15:04:10.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:10.857+0000 7efe25c00640 1 --2- 172.21.15.16:0/2074569538 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7efe2006fe60 0x7efe20158aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:04:10.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:10.857+0000 7efe25200640 1 --2- 172.21.15.16:0/2074569538 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efe200ccc80 0x7efe20151fa0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7efe0c00ebe0 tx=0x7efe0c00c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:10.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:10.857+0000 7efe1d600640 1 -- 172.21.15.16:0/2074569538 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe0c017070 con 0x7efe200ccc80 2024-03-31T15:04:10.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:10.857+0000 7efe1d600640 1 -- 172.21.15.16:0/2074569538 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efe0c00cc30 con 0x7efe200ccc80 2024-03-31T15:04:10.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:10.857+0000 7efe28600640 1 -- 172.21.15.16:0/2074569538 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efe20159080 con 0x7efe200ccc80 2024-03-31T15:04:10.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:10.857+0000 7efe1d600640 1 -- 172.21.15.16:0/2074569538 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe0c013470 con 0x7efe200ccc80 2024-03-31T15:04:10.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:10.857+0000 7efe28600640 1 -- 172.21.15.16:0/2074569538 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efe20159640 con 0x7efe200ccc80 2024-03-31T15:04:10.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:10.857+0000 7efe28600640 1 -- 172.21.15.16:0/2074569538 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efe20069950 con 0x7efe200ccc80 2024-03-31T15:04:10.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:10.857+0000 7efe1d600640 1 -- 172.21.15.16:0/2074569538 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7efe0c01a070 con 0x7efe200ccc80 2024-03-31T15:04:10.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:10.861+0000 7efe1d600640 1 --2- 172.21.15.16:0/2074569538 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7efe00068380 0x7efe0006a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:10.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:10.861+0000 7efe1d600640 1 -- 172.21.15.16:0/2074569538 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7efe0c086970 con 0x7efe200ccc80 2024-03-31T15:04:10.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:10.861+0000 7efe25c00640 1 --2- 172.21.15.16:0/2074569538 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7efe00068380 0x7efe0006a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:10.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:10.861+0000 7efe25c00640 1 --2- 172.21.15.16:0/2074569538 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7efe00068380 0x7efe0006a840 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7efe140ad000 tx=0x7efe140a6000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:10.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:10.865+0000 7efe1d600640 1 -- 172.21.15.16:0/2074569538 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7efe0c0567b0 con 0x7efe200ccc80 2024-03-31T15:04:11.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:11.029+0000 7efe28600640 1 -- 172.21.15.16:0/2074569538 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7efe200cf3c0 con 0x7efe200ccc80 2024-03-31T15:04:11.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:11.029+0000 7efe1d600640 1 -- 172.21.15.16:0/2074569538 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) ==== 74+0+16533 (secure 0 0 0) 0x7efe0c05aab0 con 0x7efe200ccc80 2024-03-31T15:04:11.024 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:04:11.024 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":21,"fsid":"286eb5da-fb47-4550-b290-d9519c13b9a5","created":"2024-03-31T15:03:11.438952+0000","modified":"2024-03-31T15:03:37.957829+0000","last_up_change":"2024-03-31T15:03:22.885535+0000","last_in_change":"2024-03-31T15:03:13.941667+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-31T15:03:22.550787+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-31T15:03:34.488425+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-31T15:03:35.299116+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"92a66450-ef19-40d8-ac3d-d760b23b0bda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6801","nonce":932712244}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6803","nonce":932712244}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6807","nonce":932712244}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6805","nonce":932712244}]},"public_addr":"172.21.15.16:6801/932712244","cluster_addr":"172.21.15.16:6803/932712244","heartbeat_back_addr":"172.21.15.16:6807/932712244","heartbeat_front_addr":"172.21.15.16:6805/932712244","state":["exists","up"]},{"osd":1,"uuid":"da9d1f2b-1968-4959-b843-faa231988e92","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6810","nonce":1398737533}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6811","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6813","nonce":1398737533}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6821","nonce":1398737533}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6815","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6816","nonce":1398737533}]},"public_addr":"172.21.15.16:6810/1398737533","cluster_addr":"172.21.15.16:6813/1398737533","heartbeat_back_addr":"172.21.15.16:6821/1398737533","heartbeat_front_addr":"172.21.15.16:6816/1398737533","state":["exists","up"]},{"osd":2,"uuid":"01a73662-9a21-4b1f-bcbb-42b9636a4638","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6819","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6822","nonce":3018526646}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6825","nonce":3018526646}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6831","nonce":3018526646}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6827","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6829","nonce":3018526646}]},"public_addr":"172.21.15.16:6822/3018526646","cluster_addr":"172.21.15.16:6825/3018526646","heartbeat_back_addr":"172.21.15.16:6831/3018526646","heartbeat_front_addr":"172.21.15.16:6829/3018526646","state":["exists","up"]},{"osd":3,"uuid":"c7771fd3-e18e-4500-ac38-23cf10ebcd75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6809","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6812","nonce":3360352974}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6817","nonce":3360352974}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6828","nonce":3360352974}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6823","nonce":3360352974}]},"public_addr":"172.21.15.16:6812/3360352974","cluster_addr":"172.21.15.16:6817/3360352974","heartbeat_back_addr":"172.21.15.16:6828/3360352974","heartbeat_front_addr":"172.21.15.16:6823/3360352974","state":["exists","up"]},{"osd":4,"uuid":"88260730-1482-4fbd-a40f-cb1124e9df4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6801","nonce":1856844856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6803","nonce":1856844856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6807","nonce":1856844856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6805","nonce":1856844856}]},"public_addr":"172.21.15.89:6801/1856844856","cluster_addr":"172.21.15.89:6803/1856844856","heartbeat_back_addr":"172.21.15.89:6807/1856844856","heartbeat_front_addr":"172.21.15.89:6805/1856844856","state":["exists","up"]},{"osd":5,"uuid":"4cb2c00c-fae0-4dd8-9d27-ec4f0093ccf7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6814","nonce":50395527}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6817","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6820","nonce":50395527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6831","nonce":50395527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6823","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6826","nonce":50395527}]},"public_addr":"172.21.15.89:6814/50395527","cluster_addr":"172.21.15.89:6820/50395527","heartbeat_back_addr":"172.21.15.89:6831/50395527","heartbeat_front_addr":"172.21.15.89:6826/50395527","state":["exists","up"]},{"osd":6,"uuid":"81ce0c19-2554-4cae-acbe-66a05e9e9b9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6809","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6812","nonce":3470471752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6815","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6818","nonce":3470471752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6827","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6829","nonce":3470471752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6821","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6824","nonce":3470471752}]},"public_addr":"172.21.15.89:6812/3470471752","cluster_addr":"172.21.15.89:6818/3470471752","heartbeat_back_addr":"172.21.15.89:6829/3470471752","heartbeat_front_addr":"172.21.15.89:6824/3470471752","state":["exists","up"]},{"osd":7,"uuid":"85e4313b-d37d-442e-8042-153bf20a847d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6811","nonce":1796778630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6813","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6816","nonce":1796778630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6825","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6828","nonce":1796778630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6819","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6822","nonce":1796778630}]},"public_addr":"172.21.15.89:6811/1796778630","cluster_addr":"172.21.15.89:6816/1796778630","heartbeat_back_addr":"172.21.15.89:6828/1796778630","heartbeat_front_addr":"172.21.15.89:6822/1796778630","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.439088+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.529970+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.680512+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:21.044161+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.582111+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.536558+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.409522+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.590229+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-31T15:04:11.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:11.033+0000 7efe28600640 1 -- 172.21.15.16:0/2074569538 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7efe00068380 msgr2=0x7efe0006a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:11.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:11.033+0000 7efe28600640 1 --2- 172.21.15.16:0/2074569538 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7efe00068380 0x7efe0006a840 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7efe140ad000 tx=0x7efe140a6000 comp rx=0 tx=0).stop 2024-03-31T15:04:11.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:11.033+0000 7efe28600640 1 -- 172.21.15.16:0/2074569538 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efe200ccc80 msgr2=0x7efe20151fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:11.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:11.033+0000 7efe28600640 1 --2- 172.21.15.16:0/2074569538 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efe200ccc80 0x7efe20151fa0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7efe0c00ebe0 tx=0x7efe0c00c520 comp rx=0 tx=0).stop 2024-03-31T15:04:11.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:11.033+0000 7efe28600640 1 -- 172.21.15.16:0/2074569538 shutdown_connections 2024-03-31T15:04:11.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:11.033+0000 7efe28600640 1 --2- 172.21.15.16:0/2074569538 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7efe00068380 0x7efe0006a840 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:11.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:11.033+0000 7efe28600640 1 --2- 172.21.15.16:0/2074569538 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efe20145e40 0x7efe20141d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:11.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:11.033+0000 7efe28600640 1 --2- 172.21.15.16:0/2074569538 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efe200ccc80 0x7efe20151fa0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:11.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:11.033+0000 7efe28600640 1 --2- 172.21.15.16:0/2074569538 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7efe2006fe60 0x7efe20158aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:11.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:11.033+0000 7efe28600640 1 -- 172.21.15.16:0/2074569538 >> 172.21.15.16:0/2074569538 conn(0x7efe20067330 msgr2=0x7efe2015eb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:04:11.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:11.033+0000 7efe28600640 1 -- 172.21.15.16:0/2074569538 shutdown_connections 2024-03-31T15:04:11.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:11.033+0000 7efe28600640 1 -- 172.21.15.16:0/2074569538 wait complete. 2024-03-31T15:04:11.045 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-31T15:03:22.550787+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-31T15:03:34.488425+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-31T15:03:35.299116+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-31T15:04:11.045 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-31T15:04:11.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:11.193+0000 7f19d1a00640 1 Processor -- start 2024-03-31T15:04:11.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:11.193+0000 7f19d1a00640 1 -- start start 2024-03-31T15:04:11.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:11.197+0000 7f19d1a00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f19cc1567e0 0x7f19cc072c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:11.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:11.197+0000 7f19d1a00640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f19cc15c7e0 con 0x7f19cc06dbe0 2024-03-31T15:04:11.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:11.197+0000 7f19d1a00640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f19cc0736b0 con 0x7f19cc0731c0 2024-03-31T15:04:11.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:11.197+0000 7f19d1a00640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f19cc15caa0 con 0x7f19cc1567e0 2024-03-31T15:04:11.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:11.197+0000 7f19cb400640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f19cc06dbe0 0x7f19cc071550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:46614/0 (socket says 172.21.15.16:46614) 2024-03-31T15:04:11.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:11.197+0000 7f19caa00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f19cc1567e0 0x7f19cc072c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:11.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:11.197+0000 7f19cb400640 1 -- 172.21.15.16:0/1223131862 learned_addr learned my addr 172.21.15.16:0/1223131862 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:04:11.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:11.197+0000 7f19caa00640 1 -- 172.21.15.16:0/1223131862 >> v1:172.21.15.89:6789/0 conn(0x7f19cc0731c0 legacy=0x7f19cc06d4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:11.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:11.197+0000 7f19caa00640 1 -- 172.21.15.16:0/1223131862 >> v1:172.21.15.16:6789/0 conn(0x7f19cc06dbe0 legacy=0x7f19cc071550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:11.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:11.197+0000 7f19caa00640 1 -- 172.21.15.16:0/1223131862 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19cc15cd80 con 0x7f19cc1567e0 2024-03-31T15:04:11.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:11.197+0000 7f19caa00640 1 --2- 172.21.15.16:0/1223131862 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f19cc1567e0 0x7f19cc072c80 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f19bc0097c0 tx=0x7f19bc01c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a624f218eeab98d server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:11.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:11.201+0000 7f19c9600640 1 -- 172.21.15.16:0/1223131862 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19bc0b0070 con 0x7f19cc1567e0 2024-03-31T15:04:11.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:11.201+0000 7f19c9600640 1 -- 172.21.15.16:0/1223131862 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f19bc0ab400 con 0x7f19cc1567e0 2024-03-31T15:04:11.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:11.201+0000 7f19c9600640 1 -- 172.21.15.16:0/1223131862 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19bc0ab7d0 con 0x7f19cc1567e0 2024-03-31T15:04:11.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:11.201+0000 7f19d1a00640 1 -- 172.21.15.16:0/1223131862 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f19cc1567e0 msgr2=0x7f19cc072c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:11.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:11.201+0000 7f19d1a00640 1 --2- 172.21.15.16:0/1223131862 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f19cc1567e0 0x7f19cc072c80 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f19bc0097c0 tx=0x7f19bc01c4a0 comp rx=0 tx=0).stop 2024-03-31T15:04:11.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:11.201+0000 7f19d1a00640 1 -- 172.21.15.16:0/1223131862 shutdown_connections 2024-03-31T15:04:11.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:11.201+0000 7f19d1a00640 1 --2- 172.21.15.16:0/1223131862 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f19cc1567e0 0x7f19cc072c80 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:11.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:11.201+0000 7f19d1a00640 1 -- 172.21.15.16:0/1223131862 >> 172.21.15.16:0/1223131862 conn(0x7f19cc06ee40 msgr2=0x7f19cc06b620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:04:11.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:11.201+0000 7f19d1a00640 1 -- 172.21.15.16:0/1223131862 shutdown_connections 2024-03-31T15:04:11.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:11.201+0000 7f19d1a00640 1 -- 172.21.15.16:0/1223131862 wait complete. 2024-03-31T15:04:11.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:11.201+0000 7f19d1a00640 1 Processor -- start 2024-03-31T15:04:11.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:11.201+0000 7f19d1a00640 1 -- start start 2024-03-31T15:04:11.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:11.201+0000 7f19d1a00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f19cc06dbe0 0x7f19cc151a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:11.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:11.201+0000 7f19d1a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f19cc0731c0 0x7f19cc0b0af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:11.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:11.201+0000 7f19d1a00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f19cc1567e0 0x7f19cc0b1030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:11.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:11.201+0000 7f19d1a00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f19cc16a300 con 0x7f19cc0731c0 2024-03-31T15:04:11.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:11.201+0000 7f19d1a00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f19cc16a180 con 0x7f19cc06dbe0 2024-03-31T15:04:11.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:11.201+0000 7f19d1a00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f19cc16a480 con 0x7f19cc1567e0 2024-03-31T15:04:11.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:11.201+0000 7f19ca000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f19cc0731c0 0x7f19cc0b0af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:11.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:11.201+0000 7f19caa00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f19cc06dbe0 0x7f19cc151a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:11.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:11.201+0000 7f19ca000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f19cc0731c0 0x7f19cc0b0af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60418/0 (socket says 172.21.15.16:60418) 2024-03-31T15:04:11.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:11.201+0000 7f19ca000640 1 -- 172.21.15.16:0/1395437625 learned_addr learned my addr 172.21.15.16:0/1395437625 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:04:11.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:11.201+0000 7f19cb400640 1 --2- 172.21.15.16:0/1395437625 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f19cc1567e0 0x7f19cc0b1030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:11.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:11.205+0000 7f19ca000640 1 -- 172.21.15.16:0/1395437625 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f19cc1567e0 msgr2=0x7f19cc0b1030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:11.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:11.205+0000 7f19ca000640 1 --2- 172.21.15.16:0/1395437625 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f19cc1567e0 0x7f19cc0b1030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:11.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:11.205+0000 7f19ca000640 1 -- 172.21.15.16:0/1395437625 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f19cc06dbe0 msgr2=0x7f19cc151a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:11.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:11.205+0000 7f19ca000640 1 --2- 172.21.15.16:0/1395437625 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f19cc06dbe0 0x7f19cc151a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:11.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:11.205+0000 7f19ca000640 1 -- 172.21.15.16:0/1395437625 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19cc151f90 con 0x7f19cc0731c0 2024-03-31T15:04:11.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:11.205+0000 7f19caa00640 1 --2- 172.21.15.16:0/1395437625 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f19cc06dbe0 0x7f19cc151a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:04:11.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:11.205+0000 7f19ca000640 1 --2- 172.21.15.16:0/1395437625 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f19cc0731c0 0x7f19cc0b0af0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f19b800c430 tx=0x7f19b800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:11.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:11.205+0000 7f19c2a00640 1 -- 172.21.15.16:0/1395437625 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19b80052d0 con 0x7f19cc0731c0 2024-03-31T15:04:11.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:11.205+0000 7f19c2a00640 1 -- 172.21.15.16:0/1395437625 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f19b8005470 con 0x7f19cc0731c0 2024-03-31T15:04:11.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:11.205+0000 7f19d1a00640 1 -- 172.21.15.16:0/1395437625 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f19cc15b660 con 0x7f19cc0731c0 2024-03-31T15:04:11.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:11.205+0000 7f19c2a00640 1 -- 172.21.15.16:0/1395437625 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19b8007260 con 0x7f19cc0731c0 2024-03-31T15:04:11.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:11.205+0000 7f19d1a00640 1 -- 172.21.15.16:0/1395437625 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f19cc15bc20 con 0x7f19cc0731c0 2024-03-31T15:04:11.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:11.205+0000 7f19d1a00640 1 -- 172.21.15.16:0/1395437625 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f19cc069950 con 0x7f19cc0731c0 2024-03-31T15:04:11.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:11.205+0000 7f19c2a00640 1 -- 172.21.15.16:0/1395437625 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f19b8007400 con 0x7f19cc0731c0 2024-03-31T15:04:11.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:11.205+0000 7f19c2a00640 1 --2- 172.21.15.16:0/1395437625 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f19ac068380 0x7f19ac06a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:11.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:11.205+0000 7f19c2a00640 1 -- 172.21.15.16:0/1395437625 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f19b8085e70 con 0x7f19cc0731c0 2024-03-31T15:04:11.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:11.205+0000 7f19caa00640 1 --2- 172.21.15.16:0/1395437625 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f19ac068380 0x7f19ac06a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:11.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:11.209+0000 7f19caa00640 1 --2- 172.21.15.16:0/1395437625 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f19ac068380 0x7f19ac06a840 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f19bc0ad000 tx=0x7f19bc008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:11.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:11.209+0000 7f19c2a00640 1 -- 172.21.15.16:0/1395437625 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f19b8055dd0 con 0x7f19cc0731c0 2024-03-31T15:04:11.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:11.373+0000 7f19d1a00640 1 -- 172.21.15.16:0/1395437625 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f19cc06e470 con 0x7f19cc0731c0 2024-03-31T15:04:11.368 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:11.373+0000 7f19c2a00640 1 -- 172.21.15.16:0/1395437625 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) ==== 93+0+10 (secure 0 0 0) 0x7f19b805a0d0 con 0x7f19cc0731c0 2024-03-31T15:04:11.368 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2024-03-31T15:04:11.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:11.377+0000 7f19d1a00640 1 -- 172.21.15.16:0/1395437625 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f19ac068380 msgr2=0x7f19ac06a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:11.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:11.377+0000 7f19d1a00640 1 --2- 172.21.15.16:0/1395437625 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f19ac068380 0x7f19ac06a840 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f19bc0ad000 tx=0x7f19bc008040 comp rx=0 tx=0).stop 2024-03-31T15:04:11.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:11.377+0000 7f19d1a00640 1 -- 172.21.15.16:0/1395437625 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f19cc0731c0 msgr2=0x7f19cc0b0af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:11.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:11.377+0000 7f19d1a00640 1 --2- 172.21.15.16:0/1395437625 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f19cc0731c0 0x7f19cc0b0af0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f19b800c430 tx=0x7f19b800c8f0 comp rx=0 tx=0).stop 2024-03-31T15:04:11.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:11.377+0000 7f19d1a00640 1 -- 172.21.15.16:0/1395437625 shutdown_connections 2024-03-31T15:04:11.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:11.377+0000 7f19d1a00640 1 --2- 172.21.15.16:0/1395437625 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f19ac068380 0x7f19ac06a840 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:11.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:11.377+0000 7f19d1a00640 1 --2- 172.21.15.16:0/1395437625 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f19cc1567e0 0x7f19cc0b1030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:11.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:11.377+0000 7f19d1a00640 1 --2- 172.21.15.16:0/1395437625 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f19cc0731c0 0x7f19cc0b0af0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:11.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:11.377+0000 7f19d1a00640 1 --2- 172.21.15.16:0/1395437625 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f19cc06dbe0 0x7f19cc151a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:11.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:11.377+0000 7f19d1a00640 1 -- 172.21.15.16:0/1395437625 >> 172.21.15.16:0/1395437625 conn(0x7f19cc06ee40 msgr2=0x7f19cc15da10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:04:11.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:11.377+0000 7f19d1a00640 1 -- 172.21.15.16:0/1395437625 shutdown_connections 2024-03-31T15:04:11.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:11.377+0000 7f19d1a00640 1 -- 172.21.15.16:0/1395437625 wait complete. 2024-03-31T15:04:11.390 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-31T15:04:11.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:11.541+0000 7fd569a00640 1 Processor -- start 2024-03-31T15:04:11.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:11.541+0000 7fd569a00640 1 -- start start 2024-03-31T15:04:11.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:11.541+0000 7fd569a00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fd564141590 0x7fd564144340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:11.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:11.541+0000 7fd569a00640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd564144980 con 0x7fd564070030 2024-03-31T15:04:11.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:11.541+0000 7fd569a00640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd564161fe0 con 0x7fd5641411b0 2024-03-31T15:04:11.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:11.541+0000 7fd569a00640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7fd56415f910 con 0x7fd564141590 2024-03-31T15:04:11.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:11.541+0000 7fd55be00640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fd564070030 0x7fd564143c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:46616/0 (socket says 172.21.15.16:46616) 2024-03-31T15:04:11.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:11.541+0000 7fd55be00640 1 -- 172.21.15.16:0/1330620228 learned_addr learned my addr 172.21.15.16:0/1330620228 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:04:11.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:11.541+0000 7fd563400640 1 --2- 172.21.15.16:0/1330620228 >> v2:172.21.15.89:3300/0 conn(0x7fd564141590 0x7fd564144340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:11.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:11.541+0000 7fd562000640 1 -- 172.21.15.16:0/1330620228 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2873772965 0 0) 0x7fd564144980 con 0x7fd564070030 2024-03-31T15:04:11.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:11.541+0000 7fd562000640 1 -- 172.21.15.16:0/1330620228 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd548003530 con 0x7fd564070030 2024-03-31T15:04:11.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:11.541+0000 7fd563400640 1 -- 172.21.15.16:0/1330620228 >> v1:172.21.15.89:6789/0 conn(0x7fd5641411b0 legacy=0x7fd56406f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:11.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:11.541+0000 7fd563400640 1 -- 172.21.15.16:0/1330620228 >> v1:172.21.15.16:6789/0 conn(0x7fd564070030 legacy=0x7fd564143c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:11.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:11.541+0000 7fd563400640 1 -- 172.21.15.16:0/1330620228 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5641622c0 con 0x7fd564141590 2024-03-31T15:04:11.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:11.545+0000 7fd563400640 1 --2- 172.21.15.16:0/1330620228 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd564141590 0x7fd564144340 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fd55c009d90 tx=0x7fd55c01c970 comp rx=0 tx=0).ready entity=mon.1 client_cookie=56f5fa9a8258dff5 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:11.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:11.545+0000 7fd562000640 1 -- 172.21.15.16:0/1330620228 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd55c0b2070 con 0x7fd564141590 2024-03-31T15:04:11.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:11.545+0000 7fd562000640 1 -- 172.21.15.16:0/1330620228 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd55c0a5940 con 0x7fd564141590 2024-03-31T15:04:11.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:11.545+0000 7fd562000640 1 -- 172.21.15.16:0/1330620228 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd55c0a5d10 con 0x7fd564141590 2024-03-31T15:04:11.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:11.545+0000 7fd569a00640 1 -- 172.21.15.16:0/1330620228 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd564141590 msgr2=0x7fd564144340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:11.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:11.545+0000 7fd569a00640 1 --2- 172.21.15.16:0/1330620228 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd564141590 0x7fd564144340 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fd55c009d90 tx=0x7fd55c01c970 comp rx=0 tx=0).stop 2024-03-31T15:04:11.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:11.545+0000 7fd569a00640 1 -- 172.21.15.16:0/1330620228 shutdown_connections 2024-03-31T15:04:11.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:11.545+0000 7fd569a00640 1 --2- 172.21.15.16:0/1330620228 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd564141590 0x7fd564144340 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:11.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:11.545+0000 7fd569a00640 1 -- 172.21.15.16:0/1330620228 >> 172.21.15.16:0/1330620228 conn(0x7fd564067930 msgr2=0x7fd5640b3ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:04:11.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:11.545+0000 7fd569a00640 1 -- 172.21.15.16:0/1330620228 shutdown_connections 2024-03-31T15:04:11.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:11.545+0000 7fd569a00640 1 -- 172.21.15.16:0/1330620228 wait complete. 2024-03-31T15:04:11.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:11.545+0000 7fd569a00640 1 Processor -- start 2024-03-31T15:04:11.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:11.545+0000 7fd569a00640 1 -- start start 2024-03-31T15:04:11.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:11.549+0000 7fd569a00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fd564070030 0x7fd5641563b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:11.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:11.549+0000 7fd569a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd5641411b0 0x7fd56413a4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:11.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:11.549+0000 7fd569a00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd564141590 0x7fd56415d7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:11.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:11.549+0000 7fd569a00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fd56416be80 con 0x7fd5641411b0 2024-03-31T15:04:11.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:11.549+0000 7fd569a00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fd56416bd00 con 0x7fd564141590 2024-03-31T15:04:11.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:11.549+0000 7fd569a00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fd56416c000 con 0x7fd564070030 2024-03-31T15:04:11.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:11.549+0000 7fd55be00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd5641411b0 0x7fd56413a4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:11.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:11.549+0000 7fd55be00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd5641411b0 0x7fd56413a4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60426/0 (socket says 172.21.15.16:60426) 2024-03-31T15:04:11.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:11.549+0000 7fd55be00640 1 -- 172.21.15.16:0/1210185733 learned_addr learned my addr 172.21.15.16:0/1210185733 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:04:11.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:11.549+0000 7fd562a00640 1 --2- 172.21.15.16:0/1210185733 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fd564070030 0x7fd5641563b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:11.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:11.549+0000 7fd563400640 1 --2- 172.21.15.16:0/1210185733 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd564141590 0x7fd56415d7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:11.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:11.549+0000 7fd55be00640 1 -- 172.21.15.16:0/1210185733 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fd564070030 msgr2=0x7fd5641563b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:11.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:11.549+0000 7fd55be00640 1 --2- 172.21.15.16:0/1210185733 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fd564070030 0x7fd5641563b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:11.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:11.549+0000 7fd55be00640 1 -- 172.21.15.16:0/1210185733 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd564141590 msgr2=0x7fd56415d7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:11.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:11.549+0000 7fd55be00640 1 --2- 172.21.15.16:0/1210185733 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd564141590 0x7fd56415d7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:11.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:11.549+0000 7fd55be00640 1 -- 172.21.15.16:0/1210185733 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd56413af80 con 0x7fd5641411b0 2024-03-31T15:04:11.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:11.549+0000 7fd55be00640 1 --2- 172.21.15.16:0/1210185733 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd5641411b0 0x7fd56413a4c0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fd54c00c4d0 tx=0x7fd54c00c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:11.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:11.549+0000 7fd55aa00640 1 -- 172.21.15.16:0/1210185733 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd54c017070 con 0x7fd5641411b0 2024-03-31T15:04:11.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:11.549+0000 7fd55aa00640 1 -- 172.21.15.16:0/1210185733 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd54c005cf0 con 0x7fd5641411b0 2024-03-31T15:04:11.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:11.549+0000 7fd55aa00640 1 -- 172.21.15.16:0/1210185733 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd54c007270 con 0x7fd5641411b0 2024-03-31T15:04:11.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:11.549+0000 7fd569a00640 1 -- 172.21.15.16:0/1210185733 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd56413b260 con 0x7fd5641411b0 2024-03-31T15:04:11.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:11.549+0000 7fd569a00640 1 -- 172.21.15.16:0/1210185733 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd56413b820 con 0x7fd5641411b0 2024-03-31T15:04:11.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:11.549+0000 7fd569a00640 1 -- 172.21.15.16:0/1210185733 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd564069950 con 0x7fd5641411b0 2024-03-31T15:04:11.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:11.549+0000 7fd55aa00640 1 -- 172.21.15.16:0/1210185733 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fd54c01a070 con 0x7fd5641411b0 2024-03-31T15:04:11.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:11.553+0000 7fd55aa00640 1 --2- 172.21.15.16:0/1210185733 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fd540068380 0x7fd54006a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:11.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:11.553+0000 7fd55aa00640 1 -- 172.21.15.16:0/1210185733 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fd54c085cb0 con 0x7fd5641411b0 2024-03-31T15:04:11.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:11.553+0000 7fd562a00640 1 --2- 172.21.15.16:0/1210185733 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fd540068380 0x7fd54006a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:11.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:11.553+0000 7fd562a00640 1 --2- 172.21.15.16:0/1210185733 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fd540068380 0x7fd54006a840 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fd554001810 tx=0x7fd554009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:11.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:11.557+0000 7fd55aa00640 1 -- 172.21.15.16:0/1210185733 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fd54c055c20 con 0x7fd5641411b0 2024-03-31T15:04:11.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:11.721+0000 7fd569a00640 1 -- 172.21.15.16:0/1210185733 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fd5640b5bb0 con 0x7fd5641411b0 2024-03-31T15:04:11.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:11.721+0000 7fd55aa00640 1 -- 172.21.15.16:0/1210185733 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) ==== 104+0+11 (secure 0 0 0) 0x7fd54c059f20 con 0x7fd5641411b0 2024-03-31T15:04:11.715 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2024-03-31T15:04:11.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:11.725+0000 7fd569a00640 1 -- 172.21.15.16:0/1210185733 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fd540068380 msgr2=0x7fd54006a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:11.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:11.725+0000 7fd569a00640 1 --2- 172.21.15.16:0/1210185733 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fd540068380 0x7fd54006a840 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fd554001810 tx=0x7fd554009040 comp rx=0 tx=0).stop 2024-03-31T15:04:11.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:11.725+0000 7fd569a00640 1 -- 172.21.15.16:0/1210185733 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd5641411b0 msgr2=0x7fd56413a4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:11.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:11.725+0000 7fd569a00640 1 --2- 172.21.15.16:0/1210185733 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd5641411b0 0x7fd56413a4c0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fd54c00c4d0 tx=0x7fd54c00c990 comp rx=0 tx=0).stop 2024-03-31T15:04:11.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:11.725+0000 7fd569a00640 1 -- 172.21.15.16:0/1210185733 shutdown_connections 2024-03-31T15:04:11.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:11.725+0000 7fd569a00640 1 --2- 172.21.15.16:0/1210185733 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fd540068380 0x7fd54006a840 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:11.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:11.725+0000 7fd569a00640 1 --2- 172.21.15.16:0/1210185733 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd564141590 0x7fd56415d7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:11.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:11.725+0000 7fd569a00640 1 --2- 172.21.15.16:0/1210185733 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd5641411b0 0x7fd56413a4c0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:11.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:11.725+0000 7fd569a00640 1 --2- 172.21.15.16:0/1210185733 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fd564070030 0x7fd5641563b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:11.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:11.725+0000 7fd569a00640 1 -- 172.21.15.16:0/1210185733 >> 172.21.15.16:0/1210185733 conn(0x7fd564067930 msgr2=0x7fd5641614f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:04:11.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:11.725+0000 7fd569a00640 1 -- 172.21.15.16:0/1210185733 shutdown_connections 2024-03-31T15:04:11.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:11.725+0000 7fd569a00640 1 -- 172.21.15.16:0/1210185733 wait complete. 2024-03-31T15:04:11.736 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-31T15:04:11.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:11.885+0000 7f4030000640 1 Processor -- start 2024-03-31T15:04:11.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:11.885+0000 7f4030000640 1 -- start start 2024-03-31T15:04:11.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:11.885+0000 7f4030000640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f4028141590 0x7f4028144340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:11.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:11.885+0000 7f4030000640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4028144980 con 0x7f4028070030 2024-03-31T15:04:11.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:11.889+0000 7f4030000640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4028161fe0 con 0x7f40281411b0 2024-03-31T15:04:11.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:11.889+0000 7f4030000640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f402815f910 con 0x7f4028141590 2024-03-31T15:04:11.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:11.889+0000 7f402cc00640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f4028070030 0x7f4028143c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:46632/0 (socket says 172.21.15.16:46632) 2024-03-31T15:04:11.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:11.889+0000 7f402cc00640 1 -- 172.21.15.16:0/4104241737 learned_addr learned my addr 172.21.15.16:0/4104241737 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:04:11.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:11.889+0000 7f402e000640 1 --2- 172.21.15.16:0/4104241737 >> v2:172.21.15.89:3300/0 conn(0x7f4028141590 0x7f4028144340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:11.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:11.889+0000 7f4027e00640 1 -- 172.21.15.16:0/4104241737 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2157742449 0 0) 0x7f4028144980 con 0x7f4028070030 2024-03-31T15:04:11.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:11.889+0000 7f4027e00640 1 -- 172.21.15.16:0/4104241737 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4010003530 con 0x7f4028070030 2024-03-31T15:04:11.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:11.889+0000 7f402e000640 1 -- 172.21.15.16:0/4104241737 >> v1:172.21.15.89:6789/0 conn(0x7f40281411b0 legacy=0x7f402806f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:11.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:11.889+0000 7f4027e00640 1 -- 172.21.15.16:0/4104241737 <== mon.1 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1285804959 0 0) 0x7f4028161fe0 con 0x7f40281411b0 2024-03-31T15:04:11.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:11.889+0000 7f402e000640 1 -- 172.21.15.16:0/4104241737 >> v1:172.21.15.16:6789/0 conn(0x7f4028070030 legacy=0x7f4028143c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:11.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:11.889+0000 7f402e000640 1 -- 172.21.15.16:0/4104241737 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40281622c0 con 0x7f4028141590 2024-03-31T15:04:11.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:11.889+0000 7f402e000640 1 --2- 172.21.15.16:0/4104241737 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4028141590 0x7f4028144340 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f4020009d90 tx=0x7f402001c970 comp rx=0 tx=0).ready entity=mon.1 client_cookie=98e70339e9700526 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:11.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:11.889+0000 7f4027e00640 1 -- 172.21.15.16:0/4104241737 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40200b2070 con 0x7f4028141590 2024-03-31T15:04:11.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:11.889+0000 7f4027e00640 1 -- 172.21.15.16:0/4104241737 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f40200a58b0 con 0x7f4028141590 2024-03-31T15:04:11.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:11.889+0000 7f4027e00640 1 -- 172.21.15.16:0/4104241737 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40200a5c80 con 0x7f4028141590 2024-03-31T15:04:11.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:11.893+0000 7f4030000640 1 -- 172.21.15.16:0/4104241737 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4028141590 msgr2=0x7f4028144340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:11.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:11.893+0000 7f4030000640 1 --2- 172.21.15.16:0/4104241737 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4028141590 0x7f4028144340 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f4020009d90 tx=0x7f402001c970 comp rx=0 tx=0).stop 2024-03-31T15:04:11.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:11.893+0000 7f4030000640 1 -- 172.21.15.16:0/4104241737 shutdown_connections 2024-03-31T15:04:11.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:11.893+0000 7f4030000640 1 --2- 172.21.15.16:0/4104241737 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4028141590 0x7f4028144340 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:11.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:11.893+0000 7f4030000640 1 -- 172.21.15.16:0/4104241737 >> 172.21.15.16:0/4104241737 conn(0x7f4028067930 msgr2=0x7f40280b3ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:04:11.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:11.893+0000 7f4030000640 1 -- 172.21.15.16:0/4104241737 shutdown_connections 2024-03-31T15:04:11.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:11.893+0000 7f4030000640 1 -- 172.21.15.16:0/4104241737 wait complete. 2024-03-31T15:04:11.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:11.893+0000 7f4030000640 1 Processor -- start 2024-03-31T15:04:11.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:11.893+0000 7f4030000640 1 -- start start 2024-03-31T15:04:11.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:11.893+0000 7f4030000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4028070030 0x7f4028140150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:11.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:11.893+0000 7f4030000640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f40281411b0 0x7f40280b5030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:11.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:11.893+0000 7f4030000640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f4028141590 0x7f40281f7080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:11.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:11.893+0000 7f4030000640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f402816be80 con 0x7f4028070030 2024-03-31T15:04:11.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:11.893+0000 7f4030000640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f402816bd00 con 0x7f40281411b0 2024-03-31T15:04:11.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:11.893+0000 7f4030000640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f402816c000 con 0x7f4028141590 2024-03-31T15:04:11.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:11.893+0000 7f402d600640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4028070030 0x7f4028140150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:11.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:11.893+0000 7f402cc00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f40281411b0 0x7f40280b5030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:11.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:11.893+0000 7f402cc00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f40281411b0 0x7f40280b5030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.16:44438/0 (socket says 172.21.15.16:44438) 2024-03-31T15:04:11.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:11.893+0000 7f402e000640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f4028141590 0x7f40281f7080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:11.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:11.893+0000 7f402cc00640 1 -- 172.21.15.16:0/2211406812 learned_addr learned my addr 172.21.15.16:0/2211406812 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:04:11.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:11.893+0000 7f402d600640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4028070030 0x7f4028140150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60436/0 (socket says 172.21.15.16:60436) 2024-03-31T15:04:11.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:11.897+0000 7f402cc00640 1 -- 172.21.15.16:0/2211406812 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f4028141590 msgr2=0x7f40281f7080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:11.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:11.897+0000 7f402cc00640 1 --2- 172.21.15.16:0/2211406812 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f4028141590 0x7f40281f7080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:11.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:11.897+0000 7f402cc00640 1 -- 172.21.15.16:0/2211406812 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4028070030 msgr2=0x7f4028140150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:11.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:11.897+0000 7f402cc00640 1 --2- 172.21.15.16:0/2211406812 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4028070030 0x7f4028140150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:11.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:11.897+0000 7f402cc00640 1 -- 172.21.15.16:0/2211406812 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40280b2560 con 0x7f40281411b0 2024-03-31T15:04:11.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:11.897+0000 7f402cc00640 1 --2- 172.21.15.16:0/2211406812 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f40281411b0 0x7f40280b5030 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f401400c450 tx=0x7f401400c910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:11.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:11.897+0000 7f4025600640 1 -- 172.21.15.16:0/2211406812 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4014017070 con 0x7f40281411b0 2024-03-31T15:04:11.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:11.897+0000 7f4025600640 1 -- 172.21.15.16:0/2211406812 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4014005cf0 con 0x7f40281411b0 2024-03-31T15:04:11.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:11.897+0000 7f4025600640 1 -- 172.21.15.16:0/2211406812 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4014007290 con 0x7f40281411b0 2024-03-31T15:04:11.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:11.897+0000 7f4030000640 1 -- 172.21.15.16:0/2211406812 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f40280b2840 con 0x7f40281411b0 2024-03-31T15:04:11.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:11.897+0000 7f4030000640 1 -- 172.21.15.16:0/2211406812 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f40280b2e00 con 0x7f40281411b0 2024-03-31T15:04:11.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:11.897+0000 7f4030000640 1 -- 172.21.15.16:0/2211406812 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4028069950 con 0x7f40281411b0 2024-03-31T15:04:11.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:11.897+0000 7f4025600640 1 -- 172.21.15.16:0/2211406812 <== mon.1 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f401401a070 con 0x7f40281411b0 2024-03-31T15:04:11.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:11.897+0000 7f4025600640 1 --2- 172.21.15.16:0/2211406812 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f4008068380 0x7f400806a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:11.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:11.897+0000 7f4025600640 1 -- 172.21.15.16:0/2211406812 <== mon.1 v2:172.21.15.89:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f4014085ce0 con 0x7f40281411b0 2024-03-31T15:04:11.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:11.897+0000 7f402d600640 1 --2- 172.21.15.16:0/2211406812 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f4008068380 0x7f400806a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:11.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:11.901+0000 7f402d600640 1 --2- 172.21.15.16:0/2211406812 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f4008068380 0x7f400806a840 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f401c000f70 tx=0x7f401c009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:11.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:11.901+0000 7f4025600640 1 -- 172.21.15.16:0/2211406812 <== mon.1 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f4014055c30 con 0x7f40281411b0 2024-03-31T15:04:12.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:12.065+0000 7f4030000640 1 -- 172.21.15.16:0/2211406812 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f4028071800 con 0x7f40281411b0 2024-03-31T15:04:12.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:12.069+0000 7f4025600640 1 -- 172.21.15.16:0/2211406812 <== mon.1 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) ==== 100+0+11 (secure 0 0 0) 0x7f4014059f30 con 0x7f40281411b0 2024-03-31T15:04:12.060 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 64 2024-03-31T15:04:12.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:12.069+0000 7f4030000640 1 -- 172.21.15.16:0/2211406812 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f4008068380 msgr2=0x7f400806a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:12.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:12.069+0000 7f4030000640 1 --2- 172.21.15.16:0/2211406812 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f4008068380 0x7f400806a840 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f401c000f70 tx=0x7f401c009040 comp rx=0 tx=0).stop 2024-03-31T15:04:12.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:12.069+0000 7f4030000640 1 -- 172.21.15.16:0/2211406812 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f40281411b0 msgr2=0x7f40280b5030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:12.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:12.069+0000 7f4030000640 1 --2- 172.21.15.16:0/2211406812 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f40281411b0 0x7f40280b5030 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f401400c450 tx=0x7f401400c910 comp rx=0 tx=0).stop 2024-03-31T15:04:12.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:12.069+0000 7f4030000640 1 -- 172.21.15.16:0/2211406812 shutdown_connections 2024-03-31T15:04:12.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:12.069+0000 7f4030000640 1 --2- 172.21.15.16:0/2211406812 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f4008068380 0x7f400806a840 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:12.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:12.069+0000 7f4030000640 1 --2- 172.21.15.16:0/2211406812 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f4028141590 0x7f40281f7080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:12.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:12.069+0000 7f4030000640 1 --2- 172.21.15.16:0/2211406812 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f40281411b0 0x7f40280b5030 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:12.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:12.069+0000 7f4030000640 1 --2- 172.21.15.16:0/2211406812 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4028070030 0x7f4028140150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:12.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:12.069+0000 7f4030000640 1 -- 172.21.15.16:0/2211406812 >> 172.21.15.16:0/2211406812 conn(0x7f4028067930 msgr2=0x7f4028150e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:04:12.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:12.069+0000 7f4030000640 1 -- 172.21.15.16:0/2211406812 shutdown_connections 2024-03-31T15:04:12.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:12.069+0000 7f4030000640 1 -- 172.21.15.16:0/2211406812 wait complete. 2024-03-31T15:04:12.082 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-31T15:04:12.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:12.229+0000 7fb993400640 1 Processor -- start 2024-03-31T15:04:12.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:12.229+0000 7fb993400640 1 -- start start 2024-03-31T15:04:12.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:12.232+0000 7fb993400640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb98c145e40 0x7fb98c0cc740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:12.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:12.232+0000 7fb993400640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb98c070030 0x7fb98c0d2010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:12.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:12.232+0000 7fb993400640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb98c0d0920 con 0x7fb98c0ccc80 2024-03-31T15:04:12.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:12.232+0000 7fb993400640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7fb98c0d06d0 con 0x7fb98c070030 2024-03-31T15:04:12.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:12.232+0000 7fb993400640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fb98c0d0c00 con 0x7fb98c145e40 2024-03-31T15:04:12.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:12.232+0000 7fb98be00640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fb98c0ccc80 0x7fb98c06f920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:46640/0 (socket says 172.21.15.16:46640) 2024-03-31T15:04:12.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:12.232+0000 7fb98be00640 1 -- 172.21.15.16:0/3656952438 learned_addr learned my addr 172.21.15.16:0/3656952438 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:04:12.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:12.232+0000 7fb990a00640 1 --2- 172.21.15.16:0/3656952438 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb98c145e40 0x7fb98c0cc740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:12.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:12.232+0000 7fb991400640 1 --2- 172.21.15.16:0/3656952438 >> v2:172.21.15.16:3300/0 conn(0x7fb98c070030 0x7fb98c0d2010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:12.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:12.232+0000 7fb991400640 1 -- 172.21.15.16:0/3656952438 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb98c145e40 msgr2=0x7fb98c0cc740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:04:12.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:12.232+0000 7fb991400640 1 --2- 172.21.15.16:0/3656952438 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb98c145e40 0x7fb98c0cc740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:12.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:12.232+0000 7fb991400640 1 -- 172.21.15.16:0/3656952438 >> v1:172.21.15.16:6789/0 conn(0x7fb98c0ccc80 legacy=0x7fb98c06f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:12.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:12.232+0000 7fb991400640 1 -- 172.21.15.16:0/3656952438 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb98c0d1810 con 0x7fb98c070030 2024-03-31T15:04:12.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:12.232+0000 7fb98b400640 1 -- 172.21.15.16:0/3656952438 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2544638074 0 0) 0x7fb98c0d0920 con 0x7fb98c0ccc80 2024-03-31T15:04:12.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:12.236+0000 7fb991400640 1 --2- 172.21.15.16:0/3656952438 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb98c070030 0x7fb98c0d2010 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fb984009d90 tx=0x7fb98401c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b84e10cf394193b server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:12.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:12.236+0000 7fb98b400640 1 -- 172.21.15.16:0/3656952438 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9840b2070 con 0x7fb98c070030 2024-03-31T15:04:12.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:12.236+0000 7fb98b400640 1 -- 172.21.15.16:0/3656952438 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb9840a5940 con 0x7fb98c070030 2024-03-31T15:04:12.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:12.236+0000 7fb98b400640 1 -- 172.21.15.16:0/3656952438 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9840a5d10 con 0x7fb98c070030 2024-03-31T15:04:12.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:12.236+0000 7fb993400640 1 -- 172.21.15.16:0/3656952438 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb98c070030 msgr2=0x7fb98c0d2010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:12.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:12.236+0000 7fb993400640 1 --2- 172.21.15.16:0/3656952438 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb98c070030 0x7fb98c0d2010 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fb984009d90 tx=0x7fb98401c970 comp rx=0 tx=0).stop 2024-03-31T15:04:12.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:12.236+0000 7fb993400640 1 -- 172.21.15.16:0/3656952438 shutdown_connections 2024-03-31T15:04:12.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:12.236+0000 7fb993400640 1 --2- 172.21.15.16:0/3656952438 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb98c070030 0x7fb98c0d2010 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:12.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:12.236+0000 7fb993400640 1 --2- 172.21.15.16:0/3656952438 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb98c145e40 0x7fb98c0cc740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:12.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:12.236+0000 7fb993400640 1 -- 172.21.15.16:0/3656952438 >> 172.21.15.16:0/3656952438 conn(0x7fb98c067330 msgr2=0x7fb98c067740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:04:12.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:12.236+0000 7fb993400640 1 -- 172.21.15.16:0/3656952438 shutdown_connections 2024-03-31T15:04:12.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:12.236+0000 7fb993400640 1 -- 172.21.15.16:0/3656952438 wait complete. 2024-03-31T15:04:12.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:12.236+0000 7fb993400640 1 Processor -- start 2024-03-31T15:04:12.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:12.236+0000 7fb993400640 1 -- start start 2024-03-31T15:04:12.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:12.240+0000 7fb993400640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb98c070030 0x7fb98c141940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:12.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:12.240+0000 7fb993400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb98c0ccc80 0x7fb98c143b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:12.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:12.240+0000 7fb993400640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb98c145e40 0x7fb98c14d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:12.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:12.240+0000 7fb993400640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fb98c0b5240 con 0x7fb98c0ccc80 2024-03-31T15:04:12.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:12.240+0000 7fb993400640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fb98c0b50c0 con 0x7fb98c145e40 2024-03-31T15:04:12.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:12.240+0000 7fb993400640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fb98c169190 con 0x7fb98c070030 2024-03-31T15:04:12.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:12.240+0000 7fb98be00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb98c0ccc80 0x7fb98c143b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:12.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:12.240+0000 7fb98be00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb98c0ccc80 0x7fb98c143b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60444/0 (socket says 172.21.15.16:60444) 2024-03-31T15:04:12.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:12.240+0000 7fb98be00640 1 -- 172.21.15.16:0/310270557 learned_addr learned my addr 172.21.15.16:0/310270557 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:04:12.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:12.240+0000 7fb990a00640 1 --2- 172.21.15.16:0/310270557 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb98c070030 0x7fb98c141940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:12.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:12.240+0000 7fb991400640 1 --2- 172.21.15.16:0/310270557 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb98c145e40 0x7fb98c14d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:12.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:12.240+0000 7fb98be00640 1 -- 172.21.15.16:0/310270557 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb98c070030 msgr2=0x7fb98c141940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:12.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:12.240+0000 7fb98be00640 1 --2- 172.21.15.16:0/310270557 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb98c070030 0x7fb98c141940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:12.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:12.240+0000 7fb98be00640 1 -- 172.21.15.16:0/310270557 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb98c145e40 msgr2=0x7fb98c14d450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:12.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:12.240+0000 7fb98be00640 1 --2- 172.21.15.16:0/310270557 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb98c145e40 0x7fb98c14d450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:12.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:12.240+0000 7fb98be00640 1 -- 172.21.15.16:0/310270557 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb98c1444a0 con 0x7fb98c0ccc80 2024-03-31T15:04:12.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:12.240+0000 7fb990a00640 1 --2- 172.21.15.16:0/310270557 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb98c070030 0x7fb98c141940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:04:12.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:12.240+0000 7fb98be00640 1 --2- 172.21.15.16:0/310270557 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb98c0ccc80 0x7fb98c143b20 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fb97800c520 tx=0x7fb97800c9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:12.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:12.240+0000 7fb988c00640 1 -- 172.21.15.16:0/310270557 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb978018070 con 0x7fb98c0ccc80 2024-03-31T15:04:12.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:12.240+0000 7fb988c00640 1 -- 172.21.15.16:0/310270557 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb9780074e0 con 0x7fb98c0ccc80 2024-03-31T15:04:12.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:12.240+0000 7fb988c00640 1 -- 172.21.15.16:0/310270557 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9780049f0 con 0x7fb98c0ccc80 2024-03-31T15:04:12.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:12.240+0000 7fb993400640 1 -- 172.21.15.16:0/310270557 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb98c144780 con 0x7fb98c0ccc80 2024-03-31T15:04:12.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:12.240+0000 7fb993400640 1 -- 172.21.15.16:0/310270557 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb98c141e80 con 0x7fb98c0ccc80 2024-03-31T15:04:12.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:12.240+0000 7fb993400640 1 -- 172.21.15.16:0/310270557 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb98c069950 con 0x7fb98c0ccc80 2024-03-31T15:04:12.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:12.240+0000 7fb988c00640 1 -- 172.21.15.16:0/310270557 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fb97801b070 con 0x7fb98c0ccc80 2024-03-31T15:04:12.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:12.244+0000 7fb988c00640 1 --2- 172.21.15.16:0/310270557 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fb96c0683d0 0x7fb96c06a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:12.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:12.244+0000 7fb988c00640 1 -- 172.21.15.16:0/310270557 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fb978085d50 con 0x7fb98c0ccc80 2024-03-31T15:04:12.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:12.244+0000 7fb990a00640 1 --2- 172.21.15.16:0/310270557 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fb96c0683d0 0x7fb96c06a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:12.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:12.244+0000 7fb990a00640 1 --2- 172.21.15.16:0/310270557 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fb96c0683d0 0x7fb96c06a890 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fb980009eb0 tx=0x7fb980009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:12.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:12.248+0000 7fb988c00640 1 -- 172.21.15.16:0/310270557 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fb978055ca0 con 0x7fb98c0ccc80 2024-03-31T15:04:12.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:12.424+0000 7fb993400640 1 -- 172.21.15.16:0/310270557 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fb98c0cf3c0 con 0x7fb98c0ccc80 2024-03-31T15:04:12.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:12.424+0000 7fb988c00640 1 -- 172.21.15.16:0/310270557 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v21) ==== 80+0+0 (secure 0 0 0) 0x7fb978059fa0 con 0x7fb98c0ccc80 2024-03-31T15:04:12.417 INFO:teuthology.orchestra.run.smithi016.stderr:listed 0 entries 2024-03-31T15:04:12.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:12.424+0000 7fb993400640 1 -- 172.21.15.16:0/310270557 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fb96c0683d0 msgr2=0x7fb96c06a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:12.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:12.424+0000 7fb993400640 1 --2- 172.21.15.16:0/310270557 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fb96c0683d0 0x7fb96c06a890 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fb980009eb0 tx=0x7fb980009190 comp rx=0 tx=0).stop 2024-03-31T15:04:12.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:12.424+0000 7fb993400640 1 -- 172.21.15.16:0/310270557 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb98c0ccc80 msgr2=0x7fb98c143b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:12.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:12.424+0000 7fb993400640 1 --2- 172.21.15.16:0/310270557 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb98c0ccc80 0x7fb98c143b20 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fb97800c520 tx=0x7fb97800c9e0 comp rx=0 tx=0).stop 2024-03-31T15:04:12.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:12.424+0000 7fb993400640 1 -- 172.21.15.16:0/310270557 shutdown_connections 2024-03-31T15:04:12.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:12.424+0000 7fb993400640 1 --2- 172.21.15.16:0/310270557 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fb96c0683d0 0x7fb96c06a890 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:12.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:12.424+0000 7fb993400640 1 --2- 172.21.15.16:0/310270557 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb98c145e40 0x7fb98c14d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:12.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:12.424+0000 7fb993400640 1 --2- 172.21.15.16:0/310270557 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb98c0ccc80 0x7fb98c143b20 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:12.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:12.424+0000 7fb993400640 1 --2- 172.21.15.16:0/310270557 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb98c070030 0x7fb98c141940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:12.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:12.424+0000 7fb993400640 1 -- 172.21.15.16:0/310270557 >> 172.21.15.16:0/310270557 conn(0x7fb98c067330 msgr2=0x7fb98c0cff60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:04:12.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:12.424+0000 7fb993400640 1 -- 172.21.15.16:0/310270557 shutdown_connections 2024-03-31T15:04:12.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:12.424+0000 7fb993400640 1 -- 172.21.15.16:0/310270557 wait complete. 2024-03-31T15:04:12.440 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-03-31T15:04:12.440 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-31T15:04:12.449 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-31T15:04:12.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:12.640+0000 7fef52e00640 1 Processor -- start 2024-03-31T15:04:12.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:12.640+0000 7fef52e00640 1 -- start start 2024-03-31T15:04:12.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:12.640+0000 7fef52e00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fef4c145d10 0x7fef4c0cf790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:12.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:12.640+0000 7fef52e00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fef4c0728b0 0x7fef4c0b1cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:12.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:12.640+0000 7fef52e00640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fef4c160e70 con 0x7fef4c0b2200 2024-03-31T15:04:12.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:12.640+0000 7fef52e00640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7fef4c1602f0 con 0x7fef4c145d10 2024-03-31T15:04:12.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:12.640+0000 7fef52e00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fef4c1605d0 con 0x7fef4c0728b0 2024-03-31T15:04:12.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:12.640+0000 7fef4be00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fef4c145d10 0x7fef4c0cf790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:12.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:12.640+0000 7fef50e00640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fef4c0b2200 0x7fef4c15cd50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:46650/0 (socket says 172.21.15.16:46650) 2024-03-31T15:04:12.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:12.640+0000 7fef50e00640 1 -- 172.21.15.16:0/252707483 learned_addr learned my addr 172.21.15.16:0/252707483 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:04:12.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:12.640+0000 7fef43400640 1 --2- 172.21.15.16:0/252707483 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fef4c0728b0 0x7fef4c0b1cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:12.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:12.644+0000 7fef43400640 1 -- 172.21.15.16:0/252707483 >> v2:172.21.15.16:3300/0 conn(0x7fef4c145d10 msgr2=0x7fef4c0cf790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:04:12.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:12.644+0000 7fef43400640 1 --2- 172.21.15.16:0/252707483 >> v2:172.21.15.16:3300/0 conn(0x7fef4c145d10 0x7fef4c0cf790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:12.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:12.644+0000 7fef43400640 1 -- 172.21.15.16:0/252707483 >> v1:172.21.15.16:6789/0 conn(0x7fef4c0b2200 legacy=0x7fef4c15cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:12.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:12.644+0000 7fef43400640 1 -- 172.21.15.16:0/252707483 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef4c0cfcd0 con 0x7fef4c0728b0 2024-03-31T15:04:12.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:12.644+0000 7fef4b400640 1 -- 172.21.15.16:0/252707483 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2770961340 0 0) 0x7fef4c160e70 con 0x7fef4c0b2200 2024-03-31T15:04:12.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:12.644+0000 7fef43400640 1 --2- 172.21.15.16:0/252707483 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fef4c0728b0 0x7fef4c0b1cc0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fef340097c0 tx=0x7fef3401c4a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7ee4c0c8bcc85f82 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:12.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:12.644+0000 7fef4b400640 1 -- 172.21.15.16:0/252707483 <== mon.2 v2:172.21.15.89:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef340b0070 con 0x7fef4c0728b0 2024-03-31T15:04:12.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:12.644+0000 7fef4b400640 1 -- 172.21.15.16:0/252707483 <== mon.2 v2:172.21.15.89:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fef340ab600 con 0x7fef4c0728b0 2024-03-31T15:04:12.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:12.644+0000 7fef4b400640 1 -- 172.21.15.16:0/252707483 <== mon.2 v2:172.21.15.89:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef340ab9d0 con 0x7fef4c0728b0 2024-03-31T15:04:12.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:12.644+0000 7fef52e00640 1 -- 172.21.15.16:0/252707483 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fef4c0728b0 msgr2=0x7fef4c0b1cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:12.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:12.644+0000 7fef52e00640 1 --2- 172.21.15.16:0/252707483 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fef4c0728b0 0x7fef4c0b1cc0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fef340097c0 tx=0x7fef3401c4a0 comp rx=0 tx=0).stop 2024-03-31T15:04:12.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:12.644+0000 7fef52e00640 1 -- 172.21.15.16:0/252707483 shutdown_connections 2024-03-31T15:04:12.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:12.644+0000 7fef52e00640 1 --2- 172.21.15.16:0/252707483 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fef4c0728b0 0x7fef4c0b1cc0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:12.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:12.644+0000 7fef52e00640 1 --2- 172.21.15.16:0/252707483 >> v2:172.21.15.16:3300/0 conn(0x7fef4c145d10 0x7fef4c0cf790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:12.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:12.648+0000 7fef52e00640 1 -- 172.21.15.16:0/252707483 >> 172.21.15.16:0/252707483 conn(0x7fef4c06b620 msgr2=0x7fef4c06ba30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:04:12.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:12.648+0000 7fef52e00640 1 -- 172.21.15.16:0/252707483 shutdown_connections 2024-03-31T15:04:12.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:12.648+0000 7fef52e00640 1 -- 172.21.15.16:0/252707483 wait complete. 2024-03-31T15:04:12.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:12.648+0000 7fef52e00640 1 Processor -- start 2024-03-31T15:04:12.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:12.648+0000 7fef52e00640 1 -- start start 2024-03-31T15:04:12.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:12.648+0000 7fef52e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fef4c0728b0 0x7fef4c144800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:12.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:12.648+0000 7fef52e00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fef4c0b2200 0x7fef4c13bc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:12.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:12.648+0000 7fef52e00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fef4c145d10 0x7fef4c139480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:12.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:12.648+0000 7fef52e00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fef4c15d870 con 0x7fef4c0728b0 2024-03-31T15:04:12.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:12.648+0000 7fef52e00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fef4c0b2650 con 0x7fef4c145d10 2024-03-31T15:04:12.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:12.648+0000 7fef52e00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fef4c0cf350 con 0x7fef4c0b2200 2024-03-31T15:04:12.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:12.648+0000 7fef4be00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fef4c0728b0 0x7fef4c144800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:12.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:12.648+0000 7fef4be00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fef4c0728b0 0x7fef4c144800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60472/0 (socket says 172.21.15.16:60472) 2024-03-31T15:04:12.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:12.648+0000 7fef4be00640 1 -- 172.21.15.16:0/1196701719 learned_addr learned my addr 172.21.15.16:0/1196701719 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:04:12.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:12.648+0000 7fef50e00640 1 --2- 172.21.15.16:0/1196701719 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fef4c145d10 0x7fef4c139480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:12.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:12.648+0000 7fef43400640 1 --2- 172.21.15.16:0/1196701719 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fef4c0b2200 0x7fef4c13bc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:12.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:12.648+0000 7fef4be00640 1 -- 172.21.15.16:0/1196701719 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fef4c0b2200 msgr2=0x7fef4c13bc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:12.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:12.648+0000 7fef4be00640 1 --2- 172.21.15.16:0/1196701719 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fef4c0b2200 0x7fef4c13bc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:12.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:12.648+0000 7fef4be00640 1 -- 172.21.15.16:0/1196701719 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fef4c145d10 msgr2=0x7fef4c139480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:12.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:12.648+0000 7fef4be00640 1 --2- 172.21.15.16:0/1196701719 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fef4c145d10 0x7fef4c139480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:12.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:12.648+0000 7fef4be00640 1 -- 172.21.15.16:0/1196701719 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef4c13ca60 con 0x7fef4c0728b0 2024-03-31T15:04:12.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:12.648+0000 7fef50e00640 1 --2- 172.21.15.16:0/1196701719 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fef4c145d10 0x7fef4c139480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:04:12.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:12.648+0000 7fef4be00640 1 --2- 172.21.15.16:0/1196701719 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fef4c0728b0 0x7fef4c144800 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fef3c00e930 tx=0x7fef3c00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:12.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:12.648+0000 7fef48c00640 1 -- 172.21.15.16:0/1196701719 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef3c017070 con 0x7fef4c0728b0 2024-03-31T15:04:12.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:12.652+0000 7fef48c00640 1 -- 172.21.15.16:0/1196701719 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fef3c00c9a0 con 0x7fef4c0728b0 2024-03-31T15:04:12.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:12.652+0000 7fef48c00640 1 -- 172.21.15.16:0/1196701719 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef3c00cd50 con 0x7fef4c0728b0 2024-03-31T15:04:12.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:12.652+0000 7fef52e00640 1 -- 172.21.15.16:0/1196701719 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fef4c13cd40 con 0x7fef4c0728b0 2024-03-31T15:04:12.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:12.652+0000 7fef52e00640 1 -- 172.21.15.16:0/1196701719 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fef4c144000 con 0x7fef4c0728b0 2024-03-31T15:04:12.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:12.652+0000 7fef52e00640 1 -- 172.21.15.16:0/1196701719 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fef4c069950 con 0x7fef4c0728b0 2024-03-31T15:04:12.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:12.652+0000 7fef48c00640 1 -- 172.21.15.16:0/1196701719 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fef3c01a070 con 0x7fef4c0728b0 2024-03-31T15:04:12.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:12.652+0000 7fef48c00640 1 --2- 172.21.15.16:0/1196701719 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fef280683b0 0x7fef2806a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:12.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:12.652+0000 7fef48c00640 1 -- 172.21.15.16:0/1196701719 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fef3c086c20 con 0x7fef4c0728b0 2024-03-31T15:04:12.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:12.652+0000 7fef43400640 1 --2- 172.21.15.16:0/1196701719 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fef280683b0 0x7fef2806a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:12.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:12.656+0000 7fef43400640 1 --2- 172.21.15.16:0/1196701719 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fef280683b0 0x7fef2806a870 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fef340ad000 tx=0x7fef340a6000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:12.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:12.656+0000 7fef48c00640 1 -- 172.21.15.16:0/1196701719 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fef3c056c40 con 0x7fef4c0728b0 2024-03-31T15:04:12.822 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:12.828+0000 7fef52e00640 1 -- 172.21.15.16:0/1196701719 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fef4c0b1dd0 con 0x7fef4c0728b0 2024-03-31T15:04:12.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:12.828+0000 7fef48c00640 1 -- 172.21.15.16:0/1196701719 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) ==== 74+0+16533 (secure 0 0 0) 0x7fef3c05af40 con 0x7fef4c0728b0 2024-03-31T15:04:12.823 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:04:12.823 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":21,"fsid":"286eb5da-fb47-4550-b290-d9519c13b9a5","created":"2024-03-31T15:03:11.438952+0000","modified":"2024-03-31T15:03:37.957829+0000","last_up_change":"2024-03-31T15:03:22.885535+0000","last_in_change":"2024-03-31T15:03:13.941667+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-31T15:03:22.550787+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-31T15:03:34.488425+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-31T15:03:35.299116+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"92a66450-ef19-40d8-ac3d-d760b23b0bda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6801","nonce":932712244}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6803","nonce":932712244}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6807","nonce":932712244}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6805","nonce":932712244}]},"public_addr":"172.21.15.16:6801/932712244","cluster_addr":"172.21.15.16:6803/932712244","heartbeat_back_addr":"172.21.15.16:6807/932712244","heartbeat_front_addr":"172.21.15.16:6805/932712244","state":["exists","up"]},{"osd":1,"uuid":"da9d1f2b-1968-4959-b843-faa231988e92","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6810","nonce":1398737533}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6811","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6813","nonce":1398737533}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6821","nonce":1398737533}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6815","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6816","nonce":1398737533}]},"public_addr":"172.21.15.16:6810/1398737533","cluster_addr":"172.21.15.16:6813/1398737533","heartbeat_back_addr":"172.21.15.16:6821/1398737533","heartbeat_front_addr":"172.21.15.16:6816/1398737533","state":["exists","up"]},{"osd":2,"uuid":"01a73662-9a21-4b1f-bcbb-42b9636a4638","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6819","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6822","nonce":3018526646}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6825","nonce":3018526646}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6831","nonce":3018526646}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6827","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6829","nonce":3018526646}]},"public_addr":"172.21.15.16:6822/3018526646","cluster_addr":"172.21.15.16:6825/3018526646","heartbeat_back_addr":"172.21.15.16:6831/3018526646","heartbeat_front_addr":"172.21.15.16:6829/3018526646","state":["exists","up"]},{"osd":3,"uuid":"c7771fd3-e18e-4500-ac38-23cf10ebcd75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6809","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6812","nonce":3360352974}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6817","nonce":3360352974}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6828","nonce":3360352974}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6823","nonce":3360352974}]},"public_addr":"172.21.15.16:6812/3360352974","cluster_addr":"172.21.15.16:6817/3360352974","heartbeat_back_addr":"172.21.15.16:6828/3360352974","heartbeat_front_addr":"172.21.15.16:6823/3360352974","state":["exists","up"]},{"osd":4,"uuid":"88260730-1482-4fbd-a40f-cb1124e9df4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6801","nonce":1856844856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6803","nonce":1856844856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6807","nonce":1856844856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6805","nonce":1856844856}]},"public_addr":"172.21.15.89:6801/1856844856","cluster_addr":"172.21.15.89:6803/1856844856","heartbeat_back_addr":"172.21.15.89:6807/1856844856","heartbeat_front_addr":"172.21.15.89:6805/1856844856","state":["exists","up"]},{"osd":5,"uuid":"4cb2c00c-fae0-4dd8-9d27-ec4f0093ccf7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6814","nonce":50395527}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6817","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6820","nonce":50395527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6831","nonce":50395527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6823","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6826","nonce":50395527}]},"public_addr":"172.21.15.89:6814/50395527","cluster_addr":"172.21.15.89:6820/50395527","heartbeat_back_addr":"172.21.15.89:6831/50395527","heartbeat_front_addr":"172.21.15.89:6826/50395527","state":["exists","up"]},{"osd":6,"uuid":"81ce0c19-2554-4cae-acbe-66a05e9e9b9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6809","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6812","nonce":3470471752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6815","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6818","nonce":3470471752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6827","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6829","nonce":3470471752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6821","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6824","nonce":3470471752}]},"public_addr":"172.21.15.89:6812/3470471752","cluster_addr":"172.21.15.89:6818/3470471752","heartbeat_back_addr":"172.21.15.89:6829/3470471752","heartbeat_front_addr":"172.21.15.89:6824/3470471752","state":["exists","up"]},{"osd":7,"uuid":"85e4313b-d37d-442e-8042-153bf20a847d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6811","nonce":1796778630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6813","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6816","nonce":1796778630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6825","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6828","nonce":1796778630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6819","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6822","nonce":1796778630}]},"public_addr":"172.21.15.89:6811/1796778630","cluster_addr":"172.21.15.89:6816/1796778630","heartbeat_back_addr":"172.21.15.89:6828/1796778630","heartbeat_front_addr":"172.21.15.89:6822/1796778630","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.439088+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.529970+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.680512+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:21.044161+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.582111+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.536558+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.409522+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.590229+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-31T15:04:12.825 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:12.832+0000 7fef52e00640 1 -- 172.21.15.16:0/1196701719 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fef280683b0 msgr2=0x7fef2806a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:12.825 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:12.832+0000 7fef52e00640 1 --2- 172.21.15.16:0/1196701719 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fef280683b0 0x7fef2806a870 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fef340ad000 tx=0x7fef340a6000 comp rx=0 tx=0).stop 2024-03-31T15:04:12.825 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:12.832+0000 7fef52e00640 1 -- 172.21.15.16:0/1196701719 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fef4c0728b0 msgr2=0x7fef4c144800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:12.825 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:12.832+0000 7fef52e00640 1 --2- 172.21.15.16:0/1196701719 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fef4c0728b0 0x7fef4c144800 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fef3c00e930 tx=0x7fef3c00edf0 comp rx=0 tx=0).stop 2024-03-31T15:04:12.825 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:12.832+0000 7fef52e00640 1 -- 172.21.15.16:0/1196701719 shutdown_connections 2024-03-31T15:04:12.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:12.832+0000 7fef52e00640 1 --2- 172.21.15.16:0/1196701719 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fef280683b0 0x7fef2806a870 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:12.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:12.832+0000 7fef52e00640 1 --2- 172.21.15.16:0/1196701719 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fef4c145d10 0x7fef4c139480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:12.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:12.832+0000 7fef52e00640 1 --2- 172.21.15.16:0/1196701719 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fef4c0b2200 0x7fef4c13bc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:12.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:12.832+0000 7fef52e00640 1 --2- 172.21.15.16:0/1196701719 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fef4c0728b0 0x7fef4c144800 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:12.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:12.832+0000 7fef52e00640 1 -- 172.21.15.16:0/1196701719 >> 172.21.15.16:0/1196701719 conn(0x7fef4c06b620 msgr2=0x7fef4c0745a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:04:12.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:12.832+0000 7fef52e00640 1 -- 172.21.15.16:0/1196701719 shutdown_connections 2024-03-31T15:04:12.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:12.832+0000 7fef52e00640 1 -- 172.21.15.16:0/1196701719 wait complete. 2024-03-31T15:04:12.844 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-31T15:03:22.550787+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-31T15:03:34.488425+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-31T15:03:35.299116+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-31T15:04:12.845 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-31T15:04:12.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:12.992+0000 7fbff5400640 1 Processor -- start 2024-03-31T15:04:12.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:12.992+0000 7fbff5400640 1 -- start start 2024-03-31T15:04:12.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:12.996+0000 7fbff5400640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fbff006b620 0x7fbff006ba00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:12.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:12.996+0000 7fbff5400640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fbff00b60a0 0x7fbff006fc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:12.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:12.996+0000 7fbff5400640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbff006bf40 con 0x7fbff00d0c00 2024-03-31T15:04:12.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:12.996+0000 7fbff5400640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7fbff006f820 con 0x7fbff006b620 2024-03-31T15:04:12.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:12.996+0000 7fbff5400640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fbff00b4a40 con 0x7fbff00b60a0 2024-03-31T15:04:12.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:12.996+0000 7fbfee800640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fbff006b620 0x7fbff006ba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:12.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:12.996+0000 7fbfee800640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fbff006b620 0x7fbff006ba00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60474/0 (socket says 172.21.15.16:60474) 2024-03-31T15:04:12.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:12.996+0000 7fbfee800640 1 -- 172.21.15.16:0/2350128765 learned_addr learned my addr 172.21.15.16:0/2350128765 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:04:12.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:12.996+0000 7fbfe5e00640 1 --2- 172.21.15.16:0/2350128765 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fbff00b60a0 0x7fbff006fc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:12.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:12.996+0000 7fbfee800640 1 -- 172.21.15.16:0/2350128765 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fbff00b60a0 msgr2=0x7fbff006fc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:04:12.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:12.996+0000 7fbfee800640 1 --2- 172.21.15.16:0/2350128765 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fbff00b60a0 0x7fbff006fc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:12.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:12.996+0000 7fbfee800640 1 -- 172.21.15.16:0/2350128765 >> v1:172.21.15.89:6789/0 conn(0x7fbff00d0c00 legacy=0x7fbff0070180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:12.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:12.996+0000 7fbfee800640 1 -- 172.21.15.16:0/2350128765 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbff006f540 con 0x7fbff006b620 2024-03-31T15:04:12.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:12.996+0000 7fbfede00640 1 -- 172.21.15.16:0/2350128765 <== mon.1 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2423348028 0 0) 0x7fbff006bf40 con 0x7fbff00d0c00 2024-03-31T15:04:12.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:12.996+0000 7fbfee800640 1 --2- 172.21.15.16:0/2350128765 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbff006b620 0x7fbff006ba00 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fbfe0009e50 tx=0x7fbfe001c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=358720401e90ccb4 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:12.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:13.000+0000 7fbfede00640 1 -- 172.21.15.16:0/2350128765 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfe00a5630 con 0x7fbff006b620 2024-03-31T15:04:12.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:13.000+0000 7fbfede00640 1 -- 172.21.15.16:0/2350128765 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbfe00a57d0 con 0x7fbff006b620 2024-03-31T15:04:12.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:13.000+0000 7fbfede00640 1 -- 172.21.15.16:0/2350128765 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfe00a5ba0 con 0x7fbff006b620 2024-03-31T15:04:12.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:13.000+0000 7fbff5400640 1 -- 172.21.15.16:0/2350128765 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbff006b620 msgr2=0x7fbff006ba00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:12.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:13.000+0000 7fbff5400640 1 --2- 172.21.15.16:0/2350128765 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbff006b620 0x7fbff006ba00 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fbfe0009e50 tx=0x7fbfe001c970 comp rx=0 tx=0).stop 2024-03-31T15:04:12.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:13.000+0000 7fbff5400640 1 -- 172.21.15.16:0/2350128765 shutdown_connections 2024-03-31T15:04:12.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:13.000+0000 7fbff5400640 1 --2- 172.21.15.16:0/2350128765 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fbff00b60a0 0x7fbff006fc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:12.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:13.000+0000 7fbff5400640 1 --2- 172.21.15.16:0/2350128765 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbff006b620 0x7fbff006ba00 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:12.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:13.000+0000 7fbff5400640 1 -- 172.21.15.16:0/2350128765 >> 172.21.15.16:0/2350128765 conn(0x7fbff006e670 msgr2=0x7fbff006ea50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:04:12.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:13.000+0000 7fbff5400640 1 -- 172.21.15.16:0/2350128765 shutdown_connections 2024-03-31T15:04:12.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:13.000+0000 7fbff5400640 1 -- 172.21.15.16:0/2350128765 wait complete. 2024-03-31T15:04:12.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:13.000+0000 7fbff5400640 1 Processor -- start 2024-03-31T15:04:12.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:13.000+0000 7fbff5400640 1 -- start start 2024-03-31T15:04:12.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:13.000+0000 7fbff5400640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbff006b620 0x7fbff0159170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:12.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:13.000+0000 7fbff5400640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fbff00b60a0 0x7fbff0160e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:12.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:13.000+0000 7fbff5400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbff00d0c00 0x7fbff015ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:12.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:13.000+0000 7fbff5400640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fbff015e8d0 con 0x7fbff00d0c00 2024-03-31T15:04:12.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:13.000+0000 7fbff5400640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fbff015e750 con 0x7fbff006b620 2024-03-31T15:04:12.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:13.000+0000 7fbff5400640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fbff006f160 con 0x7fbff00b60a0 2024-03-31T15:04:12.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:13.000+0000 7fbfef200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbff00d0c00 0x7fbff015ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:12.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:13.000+0000 7fbfee800640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbff006b620 0x7fbff0159170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:12.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:13.000+0000 7fbfef200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbff00d0c00 0x7fbff015ea50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60478/0 (socket says 172.21.15.16:60478) 2024-03-31T15:04:12.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:13.000+0000 7fbfef200640 1 -- 172.21.15.16:0/261106764 learned_addr learned my addr 172.21.15.16:0/261106764 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:04:12.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:13.000+0000 7fbfe5e00640 1 --2- 172.21.15.16:0/261106764 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fbff00b60a0 0x7fbff0160e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:12.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:13.004+0000 7fbfef200640 1 -- 172.21.15.16:0/261106764 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fbff00b60a0 msgr2=0x7fbff0160e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:12.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:13.004+0000 7fbfef200640 1 --2- 172.21.15.16:0/261106764 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fbff00b60a0 0x7fbff0160e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:12.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:13.004+0000 7fbfef200640 1 -- 172.21.15.16:0/261106764 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbff006b620 msgr2=0x7fbff0159170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:12.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:13.004+0000 7fbfef200640 1 --2- 172.21.15.16:0/261106764 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbff006b620 0x7fbff0159170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:12.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:13.004+0000 7fbfef200640 1 -- 172.21.15.16:0/261106764 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbff015a630 con 0x7fbff00d0c00 2024-03-31T15:04:12.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:13.004+0000 7fbfef200640 1 --2- 172.21.15.16:0/261106764 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbff00d0c00 0x7fbff015ea50 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fbfe800c450 tx=0x7fbfe800c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:12.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:13.004+0000 7fbfe7400640 1 -- 172.21.15.16:0/261106764 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfe8017070 con 0x7fbff00d0c00 2024-03-31T15:04:12.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:13.004+0000 7fbfe7400640 1 -- 172.21.15.16:0/261106764 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbfe8003600 con 0x7fbff00d0c00 2024-03-31T15:04:12.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:13.004+0000 7fbfe7400640 1 -- 172.21.15.16:0/261106764 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfe80039f0 con 0x7fbff00d0c00 2024-03-31T15:04:12.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:13.004+0000 7fbff5400640 1 -- 172.21.15.16:0/261106764 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbff015a910 con 0x7fbff00d0c00 2024-03-31T15:04:12.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:13.004+0000 7fbff5400640 1 -- 172.21.15.16:0/261106764 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbff0158c50 con 0x7fbff00d0c00 2024-03-31T15:04:12.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:13.004+0000 7fbff5400640 1 -- 172.21.15.16:0/261106764 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbff0069950 con 0x7fbff00d0c00 2024-03-31T15:04:12.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:13.004+0000 7fbfe7400640 1 -- 172.21.15.16:0/261106764 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fbfe801a070 con 0x7fbff00d0c00 2024-03-31T15:04:13.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:13.004+0000 7fbfe7400640 1 --2- 172.21.15.16:0/261106764 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fbfcc0683b0 0x7fbfcc06a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:13.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:13.004+0000 7fbfe7400640 1 -- 172.21.15.16:0/261106764 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fbfe8085c00 con 0x7fbff00d0c00 2024-03-31T15:04:13.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:13.008+0000 7fbfee800640 1 --2- 172.21.15.16:0/261106764 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fbfcc0683b0 0x7fbfcc06a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:13.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:13.008+0000 7fbfee800640 1 --2- 172.21.15.16:0/261106764 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fbfcc0683b0 0x7fbfcc06a870 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fbfe00ae000 tx=0x7fbfe00a7070 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:13.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:13.008+0000 7fbfe7400640 1 -- 172.21.15.16:0/261106764 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fbfe808c080 con 0x7fbff00d0c00 2024-03-31T15:04:13.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:13.180+0000 7fbff5400640 1 -- 172.21.15.16:0/261106764 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fbff0070f00 con 0x7fbff00d0c00 2024-03-31T15:04:13.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:13.180+0000 7fbfe7400640 1 -- 172.21.15.16:0/261106764 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) ==== 93+0+10 (secure 0 0 0) 0x7fbfe8055c00 con 0x7fbff00d0c00 2024-03-31T15:04:13.175 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2024-03-31T15:04:13.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:13.184+0000 7fbff5400640 1 -- 172.21.15.16:0/261106764 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fbfcc0683b0 msgr2=0x7fbfcc06a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:13.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:13.184+0000 7fbff5400640 1 --2- 172.21.15.16:0/261106764 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fbfcc0683b0 0x7fbfcc06a870 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fbfe00ae000 tx=0x7fbfe00a7070 comp rx=0 tx=0).stop 2024-03-31T15:04:13.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:13.184+0000 7fbff5400640 1 -- 172.21.15.16:0/261106764 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbff00d0c00 msgr2=0x7fbff015ea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:13.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:13.184+0000 7fbff5400640 1 --2- 172.21.15.16:0/261106764 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbff00d0c00 0x7fbff015ea50 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fbfe800c450 tx=0x7fbfe800c910 comp rx=0 tx=0).stop 2024-03-31T15:04:13.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:13.184+0000 7fbff5400640 1 -- 172.21.15.16:0/261106764 shutdown_connections 2024-03-31T15:04:13.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:13.184+0000 7fbff5400640 1 --2- 172.21.15.16:0/261106764 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fbfcc0683b0 0x7fbfcc06a870 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:13.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:13.184+0000 7fbff5400640 1 --2- 172.21.15.16:0/261106764 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbff00d0c00 0x7fbff015ea50 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:13.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:13.184+0000 7fbff5400640 1 --2- 172.21.15.16:0/261106764 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fbff00b60a0 0x7fbff0160e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:13.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:13.184+0000 7fbff5400640 1 --2- 172.21.15.16:0/261106764 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbff006b620 0x7fbff0159170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:13.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:13.184+0000 7fbff5400640 1 -- 172.21.15.16:0/261106764 >> 172.21.15.16:0/261106764 conn(0x7fbff006e670 msgr2=0x7fbff015bad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:04:13.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:13.184+0000 7fbff5400640 1 -- 172.21.15.16:0/261106764 shutdown_connections 2024-03-31T15:04:13.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:13.184+0000 7fbff5400640 1 -- 172.21.15.16:0/261106764 wait complete. 2024-03-31T15:04:13.196 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-31T15:04:13.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:13.344+0000 7f21a7000640 1 Processor -- start 2024-03-31T15:04:13.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:13.344+0000 7f21a7000640 1 -- start start 2024-03-31T15:04:13.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:13.344+0000 7f21a7000640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f21a00cda50 0x7f21a015f6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:13.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:13.344+0000 7f21a7000640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f21a00ad8d0 0x7f21a006de10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:13.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:13.344+0000 7f21a7000640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f21a006e350 con 0x7f21a00cde30 2024-03-31T15:04:13.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:13.344+0000 7f21a7000640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f21a00cec10 con 0x7f21a00ad8d0 2024-03-31T15:04:13.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:13.344+0000 7f21a7000640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f21a00cf1d0 con 0x7f21a00cda50 2024-03-31T15:04:13.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:13.344+0000 7f219fe00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f21a00cda50 0x7f21a015f6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:13.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:13.344+0000 7f2197400640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f21a00ad8d0 0x7f21a006de10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:13.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:13.344+0000 7f219fe00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f21a00cda50 0x7f21a015f6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3301/0 says I am v2:172.21.15.16:54198/0 (socket says 172.21.15.16:54198) 2024-03-31T15:04:13.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:13.344+0000 7f219fe00640 1 -- 172.21.15.16:0/3995812231 learned_addr learned my addr 172.21.15.16:0/3995812231 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:04:13.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:13.344+0000 7f21a5000640 1 --1- 172.21.15.16:0/3995812231 >> v1:172.21.15.16:6789/0 conn(0x7f21a00cde30 0x7f21a00cf660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:46660/0 (socket says 172.21.15.16:46660) 2024-03-31T15:04:13.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:13.344+0000 7f219fe00640 1 -- 172.21.15.16:0/3995812231 >> v2:172.21.15.16:3300/0 conn(0x7f21a00ad8d0 msgr2=0x7f21a006de10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:04:13.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:13.344+0000 7f219fe00640 1 --2- 172.21.15.16:0/3995812231 >> v2:172.21.15.16:3300/0 conn(0x7f21a00ad8d0 0x7f21a006de10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:13.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:13.344+0000 7f219fe00640 1 -- 172.21.15.16:0/3995812231 >> v1:172.21.15.16:6789/0 conn(0x7f21a00cde30 legacy=0x7f21a00cf660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:13.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:13.344+0000 7f219fe00640 1 -- 172.21.15.16:0/3995812231 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f21a00add10 con 0x7f21a00cda50 2024-03-31T15:04:13.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:13.348+0000 7f2197400640 1 --2- 172.21.15.16:0/3995812231 >> v2:172.21.15.16:3300/0 conn(0x7f21a00ad8d0 0x7f21a006de10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:04:13.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:13.348+0000 7f219fe00640 1 --2- 172.21.15.16:0/3995812231 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f21a00cda50 0x7f21a015f6c0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f2190009870 tx=0x7f219001ed50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=217acd9104b1448e server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:13.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:13.348+0000 7f219f400640 1 -- 172.21.15.16:0/3995812231 <== mon.2 v2:172.21.15.89:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21900b5070 con 0x7f21a00cda50 2024-03-31T15:04:13.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:13.348+0000 7f219f400640 1 -- 172.21.15.16:0/3995812231 <== mon.2 v2:172.21.15.89:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f21900a8a50 con 0x7f21a00cda50 2024-03-31T15:04:13.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:13.348+0000 7f219f400640 1 -- 172.21.15.16:0/3995812231 <== mon.2 v2:172.21.15.89:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21900a8e20 con 0x7f21a00cda50 2024-03-31T15:04:13.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:13.348+0000 7f21a7000640 1 -- 172.21.15.16:0/3995812231 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f21a00cda50 msgr2=0x7f21a015f6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:13.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:13.348+0000 7f21a7000640 1 --2- 172.21.15.16:0/3995812231 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f21a00cda50 0x7f21a015f6c0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f2190009870 tx=0x7f219001ed50 comp rx=0 tx=0).stop 2024-03-31T15:04:13.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:13.348+0000 7f21a7000640 1 -- 172.21.15.16:0/3995812231 shutdown_connections 2024-03-31T15:04:13.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:13.348+0000 7f21a7000640 1 --2- 172.21.15.16:0/3995812231 >> v2:172.21.15.16:3300/0 conn(0x7f21a00ad8d0 0x7f21a006de10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:13.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:13.348+0000 7f21a7000640 1 --2- 172.21.15.16:0/3995812231 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f21a00cda50 0x7f21a015f6c0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:13.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:13.348+0000 7f21a7000640 1 -- 172.21.15.16:0/3995812231 >> 172.21.15.16:0/3995812231 conn(0x7f21a00b2db0 msgr2=0x7f21a00cd340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:04:13.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:13.348+0000 7f21a7000640 1 -- 172.21.15.16:0/3995812231 shutdown_connections 2024-03-31T15:04:13.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:13.348+0000 7f21a7000640 1 -- 172.21.15.16:0/3995812231 wait complete. 2024-03-31T15:04:13.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:13.352+0000 7f21a7000640 1 Processor -- start 2024-03-31T15:04:13.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:13.352+0000 7f21a7000640 1 -- start start 2024-03-31T15:04:13.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:13.352+0000 7f21a7000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f21a00ad8d0 0x7f21a01e76c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:13.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:13.352+0000 7f21a7000640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f21a00cda50 0x7f21a015a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:13.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:13.352+0000 7f21a7000640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f21a00cde30 0x7f21a01e59b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:13.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:13.352+0000 7f21a7000640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f21a00b39a0 con 0x7f21a00ad8d0 2024-03-31T15:04:13.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:13.352+0000 7f21a7000640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f21a015e960 con 0x7f21a00cde30 2024-03-31T15:04:13.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:13.352+0000 7f21a7000640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f21a006e8d0 con 0x7f21a00cda50 2024-03-31T15:04:13.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:13.352+0000 7f219fe00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f21a00ad8d0 0x7f21a01e76c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:13.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:13.352+0000 7f219fe00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f21a00ad8d0 0x7f21a01e76c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60498/0 (socket says 172.21.15.16:60498) 2024-03-31T15:04:13.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:13.352+0000 7f2197400640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f21a00cda50 0x7f21a015a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:13.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:13.352+0000 7f219fe00640 1 -- 172.21.15.16:0/1936020297 learned_addr learned my addr 172.21.15.16:0/1936020297 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:04:13.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:13.352+0000 7f21a5000640 1 --2- 172.21.15.16:0/1936020297 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f21a00cde30 0x7f21a01e59b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:13.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:13.352+0000 7f219fe00640 1 -- 172.21.15.16:0/1936020297 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f21a00cda50 msgr2=0x7f21a015a870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:13.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:13.352+0000 7f219fe00640 1 --2- 172.21.15.16:0/1936020297 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f21a00cda50 0x7f21a015a870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:13.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:13.352+0000 7f219fe00640 1 -- 172.21.15.16:0/1936020297 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f21a00cde30 msgr2=0x7f21a01e59b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:13.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:13.352+0000 7f219fe00640 1 --2- 172.21.15.16:0/1936020297 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f21a00cde30 0x7f21a01e59b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:13.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:13.352+0000 7f219fe00640 1 -- 172.21.15.16:0/1936020297 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f21a01e5ef0 con 0x7f21a00ad8d0 2024-03-31T15:04:13.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:13.352+0000 7f2197400640 1 --2- 172.21.15.16:0/1936020297 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f21a00cda50 0x7f21a015a870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:04:13.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:13.352+0000 7f219fe00640 1 --2- 172.21.15.16:0/1936020297 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f21a00ad8d0 0x7f21a01e76c0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f21900b3040 tx=0x7f219001baf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:13.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:13.352+0000 7f219cc00640 1 -- 172.21.15.16:0/1936020297 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21900c0070 con 0x7f21a00ad8d0 2024-03-31T15:04:13.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:13.352+0000 7f219cc00640 1 -- 172.21.15.16:0/1936020297 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f21900b0b90 con 0x7f21a00ad8d0 2024-03-31T15:04:13.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:13.352+0000 7f219cc00640 1 -- 172.21.15.16:0/1936020297 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21900b5040 con 0x7f21a00ad8d0 2024-03-31T15:04:13.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:13.352+0000 7f21a7000640 1 -- 172.21.15.16:0/1936020297 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f21a01e6190 con 0x7f21a00ad8d0 2024-03-31T15:04:13.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:13.356+0000 7f21a7000640 1 -- 172.21.15.16:0/1936020297 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f21a01e6750 con 0x7f21a00ad8d0 2024-03-31T15:04:13.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:13.356+0000 7f21a7000640 1 -- 172.21.15.16:0/1936020297 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f21a0069950 con 0x7f21a00ad8d0 2024-03-31T15:04:13.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:13.356+0000 7f219cc00640 1 -- 172.21.15.16:0/1936020297 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f21900ba3d0 con 0x7f21a00ad8d0 2024-03-31T15:04:13.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:13.356+0000 7f219cc00640 1 --2- 172.21.15.16:0/1936020297 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f217c068380 0x7f217c06a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:13.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:13.356+0000 7f219cc00640 1 -- 172.21.15.16:0/1936020297 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f2190123d90 con 0x7f21a00ad8d0 2024-03-31T15:04:13.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:13.356+0000 7f2197400640 1 --2- 172.21.15.16:0/1936020297 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f217c068380 0x7f217c06a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:13.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:13.356+0000 7f2197400640 1 --2- 172.21.15.16:0/1936020297 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f217c068380 0x7f217c06a840 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f21a006c760 tx=0x7f2188009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:13.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:13.360+0000 7f219cc00640 1 -- 172.21.15.16:0/1936020297 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f21900188b0 con 0x7f21a00ad8d0 2024-03-31T15:04:13.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:13.524+0000 7f21a7000640 1 -- 172.21.15.16:0/1936020297 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f21a006c4b0 con 0x7f21a00ad8d0 2024-03-31T15:04:13.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:13.524+0000 7f219cc00640 1 -- 172.21.15.16:0/1936020297 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) ==== 104+0+11 (secure 0 0 0) 0x7f21900f3c60 con 0x7f21a00ad8d0 2024-03-31T15:04:13.518 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2024-03-31T15:04:13.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:13.524+0000 7f21a7000640 1 -- 172.21.15.16:0/1936020297 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f217c068380 msgr2=0x7f217c06a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:13.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:13.524+0000 7f21a7000640 1 --2- 172.21.15.16:0/1936020297 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f217c068380 0x7f217c06a840 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f21a006c760 tx=0x7f2188009190 comp rx=0 tx=0).stop 2024-03-31T15:04:13.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:13.524+0000 7f21a7000640 1 -- 172.21.15.16:0/1936020297 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f21a00ad8d0 msgr2=0x7f21a01e76c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:13.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:13.524+0000 7f21a7000640 1 --2- 172.21.15.16:0/1936020297 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f21a00ad8d0 0x7f21a01e76c0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f21900b3040 tx=0x7f219001baf0 comp rx=0 tx=0).stop 2024-03-31T15:04:13.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:13.524+0000 7f21a7000640 1 -- 172.21.15.16:0/1936020297 shutdown_connections 2024-03-31T15:04:13.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:13.524+0000 7f21a7000640 1 --2- 172.21.15.16:0/1936020297 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f217c068380 0x7f217c06a840 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:13.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:13.524+0000 7f21a7000640 1 --2- 172.21.15.16:0/1936020297 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f21a00cde30 0x7f21a01e59b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:13.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:13.524+0000 7f21a7000640 1 --2- 172.21.15.16:0/1936020297 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f21a00cda50 0x7f21a015a870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:13.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:13.524+0000 7f21a7000640 1 --2- 172.21.15.16:0/1936020297 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f21a00ad8d0 0x7f21a01e76c0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:13.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:13.524+0000 7f21a7000640 1 -- 172.21.15.16:0/1936020297 >> 172.21.15.16:0/1936020297 conn(0x7f21a00b2db0 msgr2=0x7f21a015c2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:04:13.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:13.524+0000 7f21a7000640 1 -- 172.21.15.16:0/1936020297 shutdown_connections 2024-03-31T15:04:13.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:13.524+0000 7f21a7000640 1 -- 172.21.15.16:0/1936020297 wait complete. 2024-03-31T15:04:13.540 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-31T15:04:13.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:13.688+0000 7f84a4c00640 1 Processor -- start 2024-03-31T15:04:13.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:13.688+0000 7f84a4c00640 1 -- start start 2024-03-31T15:04:13.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:13.692+0000 7f84a4c00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f84a01411b0 0x7f84a006f920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:13.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:13.692+0000 7f84a4c00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f84a0141590 0x7f84a006fe60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:13.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:13.692+0000 7f84a4c00640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f84a0144980 con 0x7f84a0143c30 2024-03-31T15:04:13.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:13.692+0000 7f84a4c00640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f84a015f910 con 0x7f84a01411b0 2024-03-31T15:04:13.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:13.692+0000 7f84a4c00640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f84a0162910 con 0x7f84a0141590 2024-03-31T15:04:13.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:13.692+0000 7f849de00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f84a01411b0 0x7f84a006f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:13.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:13.692+0000 7f849de00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f84a01411b0 0x7f84a006f920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60512/0 (socket says 172.21.15.16:60512) 2024-03-31T15:04:13.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:13.692+0000 7f849de00640 1 -- 172.21.15.16:0/776735885 learned_addr learned my addr 172.21.15.16:0/776735885 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:04:13.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:13.692+0000 7f8497e00640 1 --2- 172.21.15.16:0/776735885 >> v2:172.21.15.89:3300/0 conn(0x7f84a0141590 0x7f84a006fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:13.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:13.692+0000 7f849de00640 1 -- 172.21.15.16:0/776735885 >> v2:172.21.15.89:3300/0 conn(0x7f84a0141590 msgr2=0x7f84a006fe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:04:13.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:13.692+0000 7f849de00640 1 --2- 172.21.15.16:0/776735885 >> v2:172.21.15.89:3300/0 conn(0x7f84a0141590 0x7f84a006fe60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:13.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:13.692+0000 7f849de00640 1 -- 172.21.15.16:0/776735885 >> v1:172.21.15.89:6789/0 conn(0x7f84a0143c30 legacy=0x7f84a01440a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:13.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:13.692+0000 7f849de00640 1 -- 172.21.15.16:0/776735885 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84a0161fe0 con 0x7f84a01411b0 2024-03-31T15:04:13.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:13.692+0000 7f849de00640 1 --2- 172.21.15.16:0/776735885 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f84a01411b0 0x7f84a006f920 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f84a0158fe0 tx=0x7f849001c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ff5366d37be33c6 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:13.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:13.696+0000 7f849d400640 1 -- 172.21.15.16:0/776735885 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84900ab610 con 0x7f84a01411b0 2024-03-31T15:04:13.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:13.696+0000 7f849d400640 1 -- 172.21.15.16:0/776735885 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f84900b0070 con 0x7f84a01411b0 2024-03-31T15:04:13.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:13.696+0000 7f849d400640 1 -- 172.21.15.16:0/776735885 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84900ab9d0 con 0x7f84a01411b0 2024-03-31T15:04:13.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:13.696+0000 7f84a4c00640 1 -- 172.21.15.16:0/776735885 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f84a01411b0 msgr2=0x7f84a006f920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:13.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:13.696+0000 7f84a4c00640 1 --2- 172.21.15.16:0/776735885 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f84a01411b0 0x7f84a006f920 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f84a0158fe0 tx=0x7f849001c520 comp rx=0 tx=0).stop 2024-03-31T15:04:13.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:13.696+0000 7f84a4c00640 1 -- 172.21.15.16:0/776735885 shutdown_connections 2024-03-31T15:04:13.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:13.696+0000 7f84a4c00640 1 --2- 172.21.15.16:0/776735885 >> v2:172.21.15.89:3300/0 conn(0x7f84a0141590 0x7f84a006fe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:13.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:13.696+0000 7f84a4c00640 1 --2- 172.21.15.16:0/776735885 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f84a01411b0 0x7f84a006f920 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:13.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:13.696+0000 7f84a4c00640 1 -- 172.21.15.16:0/776735885 >> 172.21.15.16:0/776735885 conn(0x7f84a0067930 msgr2=0x7f84a00b3ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:04:13.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:13.696+0000 7f84a4c00640 1 -- 172.21.15.16:0/776735885 shutdown_connections 2024-03-31T15:04:13.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:13.696+0000 7f84a4c00640 1 -- 172.21.15.16:0/776735885 wait complete. 2024-03-31T15:04:13.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:13.696+0000 7f84a4c00640 1 Processor -- start 2024-03-31T15:04:13.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:13.696+0000 7f84a4c00640 1 -- start start 2024-03-31T15:04:13.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:13.696+0000 7f84a4c00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f84a01411b0 0x7f84a015dfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:13.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:13.696+0000 7f84a4c00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f84a0141590 0x7f84a01569e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:13.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:13.696+0000 7f84a4c00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f84a0143c30 0x7f84a00b29b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:13.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:13.696+0000 7f84a4c00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f84a015f290 con 0x7f84a0141590 2024-03-31T15:04:13.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:13.696+0000 7f84a4c00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f84a015f110 con 0x7f84a0143c30 2024-03-31T15:04:13.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:13.696+0000 7f84a4c00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f84a015f410 con 0x7f84a01411b0 2024-03-31T15:04:13.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:13.696+0000 7f8497e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f84a0141590 0x7f84a01569e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:13.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:13.696+0000 7f8497e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f84a0141590 0x7f84a01569e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60514/0 (socket says 172.21.15.16:60514) 2024-03-31T15:04:13.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:13.696+0000 7f8497e00640 1 -- 172.21.15.16:0/2118514490 learned_addr learned my addr 172.21.15.16:0/2118514490 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:04:13.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:13.696+0000 7f849de00640 1 --2- 172.21.15.16:0/2118514490 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f84a01411b0 0x7f84a015dfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:13.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:13.696+0000 7f849e800640 1 --2- 172.21.15.16:0/2118514490 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f84a0143c30 0x7f84a00b29b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:13.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:13.700+0000 7f8497e00640 1 -- 172.21.15.16:0/2118514490 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f84a01411b0 msgr2=0x7f84a015dfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:13.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:13.700+0000 7f8497e00640 1 --2- 172.21.15.16:0/2118514490 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f84a01411b0 0x7f84a015dfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:13.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:13.700+0000 7f8497e00640 1 -- 172.21.15.16:0/2118514490 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f84a0143c30 msgr2=0x7f84a00b29b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:13.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:13.700+0000 7f8497e00640 1 --2- 172.21.15.16:0/2118514490 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f84a0143c30 0x7f84a00b29b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:13.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:13.700+0000 7f8497e00640 1 -- 172.21.15.16:0/2118514490 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84a00d2de0 con 0x7f84a0141590 2024-03-31T15:04:13.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:13.700+0000 7f8497e00640 1 --2- 172.21.15.16:0/2118514490 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f84a0141590 0x7f84a01569e0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f848800e900 tx=0x7f848800edc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:13.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:13.700+0000 7f8496000640 1 -- 172.21.15.16:0/2118514490 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8488017070 con 0x7f84a0141590 2024-03-31T15:04:13.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:13.700+0000 7f8496000640 1 -- 172.21.15.16:0/2118514490 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f848800ca00 con 0x7f84a0141590 2024-03-31T15:04:13.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:13.700+0000 7f8496000640 1 -- 172.21.15.16:0/2118514490 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f848800cdd0 con 0x7f84a0141590 2024-03-31T15:04:13.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:13.700+0000 7f84a4c00640 1 -- 172.21.15.16:0/2118514490 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f84a015c920 con 0x7f84a0141590 2024-03-31T15:04:13.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:13.700+0000 7f84a4c00640 1 -- 172.21.15.16:0/2118514490 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f84a015cee0 con 0x7f84a0141590 2024-03-31T15:04:13.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:13.700+0000 7f84a4c00640 1 -- 172.21.15.16:0/2118514490 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f84a0069950 con 0x7f84a0141590 2024-03-31T15:04:13.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:13.700+0000 7f8496000640 1 -- 172.21.15.16:0/2118514490 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f848801a070 con 0x7f84a0141590 2024-03-31T15:04:13.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:13.700+0000 7f8496000640 1 --2- 172.21.15.16:0/2118514490 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f847c068380 0x7f847c06a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:13.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:13.700+0000 7f8496000640 1 -- 172.21.15.16:0/2118514490 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f8488086d50 con 0x7f84a0141590 2024-03-31T15:04:13.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:13.700+0000 7f849de00640 1 --2- 172.21.15.16:0/2118514490 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f847c068380 0x7f847c06a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:13.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:13.704+0000 7f849de00640 1 --2- 172.21.15.16:0/2118514490 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f847c068380 0x7f847c06a840 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f84900af000 tx=0x7f84900a7000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:13.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:13.704+0000 7f8496000640 1 -- 172.21.15.16:0/2118514490 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f8488056c20 con 0x7f84a0141590 2024-03-31T15:04:13.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:13.868+0000 7f84a4c00640 1 -- 172.21.15.16:0/2118514490 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f84a00b58b0 con 0x7f84a0141590 2024-03-31T15:04:13.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:13.872+0000 7f8496000640 1 -- 172.21.15.16:0/2118514490 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) ==== 100+0+11 (secure 0 0 0) 0x7f848805af20 con 0x7f84a0141590 2024-03-31T15:04:13.865 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 64 2024-03-31T15:04:13.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:13.872+0000 7f84a4c00640 1 -- 172.21.15.16:0/2118514490 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f847c068380 msgr2=0x7f847c06a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:13.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:13.872+0000 7f84a4c00640 1 --2- 172.21.15.16:0/2118514490 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f847c068380 0x7f847c06a840 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f84900af000 tx=0x7f84900a7000 comp rx=0 tx=0).stop 2024-03-31T15:04:13.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:13.872+0000 7f84a4c00640 1 -- 172.21.15.16:0/2118514490 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f84a0141590 msgr2=0x7f84a01569e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:13.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:13.872+0000 7f84a4c00640 1 --2- 172.21.15.16:0/2118514490 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f84a0141590 0x7f84a01569e0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f848800e900 tx=0x7f848800edc0 comp rx=0 tx=0).stop 2024-03-31T15:04:13.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:13.872+0000 7f84a4c00640 1 -- 172.21.15.16:0/2118514490 shutdown_connections 2024-03-31T15:04:13.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:13.872+0000 7f84a4c00640 1 --2- 172.21.15.16:0/2118514490 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f847c068380 0x7f847c06a840 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:13.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:13.872+0000 7f84a4c00640 1 --2- 172.21.15.16:0/2118514490 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f84a0143c30 0x7f84a00b29b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:13.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:13.872+0000 7f84a4c00640 1 --2- 172.21.15.16:0/2118514490 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f84a0141590 0x7f84a01569e0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:13.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:13.872+0000 7f84a4c00640 1 --2- 172.21.15.16:0/2118514490 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f84a01411b0 0x7f84a015dfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:13.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:13.872+0000 7f84a4c00640 1 -- 172.21.15.16:0/2118514490 >> 172.21.15.16:0/2118514490 conn(0x7f84a0067930 msgr2=0x7f84a0143200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:04:13.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:13.872+0000 7f84a4c00640 1 -- 172.21.15.16:0/2118514490 shutdown_connections 2024-03-31T15:04:13.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:13.872+0000 7f84a4c00640 1 -- 172.21.15.16:0/2118514490 wait complete. 2024-03-31T15:04:13.886 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-31T15:04:14.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:14.032+0000 7fce1fc00640 1 Processor -- start 2024-03-31T15:04:14.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:14.032+0000 7fce1fc00640 1 -- start start 2024-03-31T15:04:14.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:14.036+0000 7fce1fc00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fce18145d10 0x7fce180cf790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:14.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:14.036+0000 7fce1fc00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fce180728b0 0x7fce180b1cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:14.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:14.036+0000 7fce1fc00640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fce18160e70 con 0x7fce180b2200 2024-03-31T15:04:14.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:14.036+0000 7fce1fc00640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7fce181602f0 con 0x7fce18145d10 2024-03-31T15:04:14.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:14.036+0000 7fce1fc00640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7fce181605d0 con 0x7fce180728b0 2024-03-31T15:04:14.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:14.036+0000 7fce1d200640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fce18145d10 0x7fce180cf790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:14.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:14.036+0000 7fce1d200640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fce18145d10 0x7fce180cf790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60524/0 (socket says 172.21.15.16:60524) 2024-03-31T15:04:14.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:14.036+0000 7fce1d200640 1 -- 172.21.15.16:0/1416951689 learned_addr learned my addr 172.21.15.16:0/1416951689 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:04:14.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:14.036+0000 7fce0fe00640 1 --2- 172.21.15.16:0/1416951689 >> v2:172.21.15.89:3300/0 conn(0x7fce180728b0 0x7fce180b1cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:14.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:14.036+0000 7fce1d200640 1 -- 172.21.15.16:0/1416951689 >> v2:172.21.15.89:3300/0 conn(0x7fce180728b0 msgr2=0x7fce180b1cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:04:14.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:14.036+0000 7fce1d200640 1 --2- 172.21.15.16:0/1416951689 >> v2:172.21.15.89:3300/0 conn(0x7fce180728b0 0x7fce180b1cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:14.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:14.036+0000 7fce1d200640 1 -- 172.21.15.16:0/1416951689 >> v1:172.21.15.89:6789/0 conn(0x7fce180b2200 legacy=0x7fce1815cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:14.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:14.036+0000 7fce1d200640 1 -- 172.21.15.16:0/1416951689 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce180cfcd0 con 0x7fce18145d10 2024-03-31T15:04:14.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:14.040+0000 7fce1d200640 1 --2- 172.21.15.16:0/1416951689 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fce18145d10 0x7fce180cf790 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fce180b2980 tx=0x7fce0801c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd77b73e8763c9b2 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:14.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:14.040+0000 7fce17e00640 1 -- 172.21.15.16:0/1416951689 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce080b0070 con 0x7fce18145d10 2024-03-31T15:04:14.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:14.040+0000 7fce17e00640 1 -- 172.21.15.16:0/1416951689 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fce080ab600 con 0x7fce18145d10 2024-03-31T15:04:14.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:14.040+0000 7fce17e00640 1 -- 172.21.15.16:0/1416951689 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce080ab9d0 con 0x7fce18145d10 2024-03-31T15:04:14.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:14.040+0000 7fce1fc00640 1 -- 172.21.15.16:0/1416951689 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fce18145d10 msgr2=0x7fce180cf790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:14.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:14.040+0000 7fce1fc00640 1 --2- 172.21.15.16:0/1416951689 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fce18145d10 0x7fce180cf790 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fce180b2980 tx=0x7fce0801c520 comp rx=0 tx=0).stop 2024-03-31T15:04:14.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:14.040+0000 7fce1fc00640 1 -- 172.21.15.16:0/1416951689 shutdown_connections 2024-03-31T15:04:14.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:14.040+0000 7fce1fc00640 1 --2- 172.21.15.16:0/1416951689 >> v2:172.21.15.89:3300/0 conn(0x7fce180728b0 0x7fce180b1cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:14.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:14.040+0000 7fce1fc00640 1 --2- 172.21.15.16:0/1416951689 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fce18145d10 0x7fce180cf790 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:14.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:14.040+0000 7fce1fc00640 1 -- 172.21.15.16:0/1416951689 >> 172.21.15.16:0/1416951689 conn(0x7fce1806b620 msgr2=0x7fce1806ba30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:04:14.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:14.040+0000 7fce1fc00640 1 -- 172.21.15.16:0/1416951689 shutdown_connections 2024-03-31T15:04:14.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:14.040+0000 7fce1fc00640 1 -- 172.21.15.16:0/1416951689 wait complete. 2024-03-31T15:04:14.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:14.040+0000 7fce1fc00640 1 Processor -- start 2024-03-31T15:04:14.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:14.040+0000 7fce1fc00640 1 -- start start 2024-03-31T15:04:14.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:14.040+0000 7fce1fc00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fce180728b0 0x7fce18166160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:14.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:14.040+0000 7fce1fc00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fce180b2200 0x7fce18140e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:14.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:14.040+0000 7fce1fc00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fce18145d10 0x7fce1813baa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:14.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:14.040+0000 7fce1fc00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fce1815d870 con 0x7fce180728b0 2024-03-31T15:04:14.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:14.040+0000 7fce1fc00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fce180b2610 con 0x7fce180b2200 2024-03-31T15:04:14.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:14.040+0000 7fce1fc00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fce180cf350 con 0x7fce18145d10 2024-03-31T15:04:14.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:14.040+0000 7fce1d200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fce180728b0 0x7fce18166160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:14.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:14.040+0000 7fce1d200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fce180728b0 0x7fce18166160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60528/0 (socket says 172.21.15.16:60528) 2024-03-31T15:04:14.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:14.040+0000 7fce1d200640 1 -- 172.21.15.16:0/4274417768 learned_addr learned my addr 172.21.15.16:0/4274417768 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:04:14.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:14.044+0000 7fce0fe00640 1 --2- 172.21.15.16:0/4274417768 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fce180b2200 0x7fce18140e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:14.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:14.044+0000 7fce1dc00640 1 --2- 172.21.15.16:0/4274417768 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fce18145d10 0x7fce1813baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:14.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:14.044+0000 7fce1d200640 1 -- 172.21.15.16:0/4274417768 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fce18145d10 msgr2=0x7fce1813baa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:14.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:14.044+0000 7fce1d200640 1 --2- 172.21.15.16:0/4274417768 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fce18145d10 0x7fce1813baa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:14.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:14.044+0000 7fce1d200640 1 -- 172.21.15.16:0/4274417768 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fce180b2200 msgr2=0x7fce18140e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:14.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:14.044+0000 7fce1d200640 1 --2- 172.21.15.16:0/4274417768 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fce180b2200 0x7fce18140e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:14.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:14.044+0000 7fce1d200640 1 -- 172.21.15.16:0/4274417768 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce180b3f10 con 0x7fce180728b0 2024-03-31T15:04:14.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:14.044+0000 7fce1d200640 1 --2- 172.21.15.16:0/4274417768 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fce180728b0 0x7fce18166160 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fce0801abd0 tx=0x7fce0801ad60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:14.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:14.044+0000 7fce15600640 1 -- 172.21.15.16:0/4274417768 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce080b0040 con 0x7fce180728b0 2024-03-31T15:04:14.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:14.044+0000 7fce15600640 1 -- 172.21.15.16:0/4274417768 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fce080ab400 con 0x7fce180728b0 2024-03-31T15:04:14.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:14.044+0000 7fce15600640 1 -- 172.21.15.16:0/4274417768 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce080b6660 con 0x7fce180728b0 2024-03-31T15:04:14.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:14.044+0000 7fce1fc00640 1 -- 172.21.15.16:0/4274417768 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fce181653a0 con 0x7fce180728b0 2024-03-31T15:04:14.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:14.044+0000 7fce1fc00640 1 -- 172.21.15.16:0/4274417768 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fce18165960 con 0x7fce180728b0 2024-03-31T15:04:14.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:14.044+0000 7fce1fc00640 1 -- 172.21.15.16:0/4274417768 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fce18069950 con 0x7fce180728b0 2024-03-31T15:04:14.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:14.044+0000 7fce15600640 1 -- 172.21.15.16:0/4274417768 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fce080be030 con 0x7fce180728b0 2024-03-31T15:04:14.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:14.044+0000 7fce15600640 1 --2- 172.21.15.16:0/4274417768 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fcdf4068380 0x7fcdf406a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:14.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:14.044+0000 7fce15600640 1 -- 172.21.15.16:0/4274417768 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fce081200f0 con 0x7fce180728b0 2024-03-31T15:04:14.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:14.048+0000 7fce0fe00640 1 --2- 172.21.15.16:0/4274417768 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fcdf4068380 0x7fcdf406a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:14.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:14.048+0000 7fce0fe00640 1 --2- 172.21.15.16:0/4274417768 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fcdf4068380 0x7fcdf406a840 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fce180b52d0 tx=0x7fce00009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:14.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:14.048+0000 7fce15600640 1 -- 172.21.15.16:0/4274417768 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fce080c5060 con 0x7fce180728b0 2024-03-31T15:04:14.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:14.228+0000 7fce1fc00640 1 -- 172.21.15.16:0/4274417768 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fce1815ee70 con 0x7fce180728b0 2024-03-31T15:04:14.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:14.228+0000 7fce15600640 1 -- 172.21.15.16:0/4274417768 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v21) ==== 80+0+0 (secure 0 0 0) 0x7fce08016110 con 0x7fce180728b0 2024-03-31T15:04:14.223 INFO:teuthology.orchestra.run.smithi016.stderr:listed 0 entries 2024-03-31T15:04:14.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:14.232+0000 7fce1fc00640 1 -- 172.21.15.16:0/4274417768 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fcdf4068380 msgr2=0x7fcdf406a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:14.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:14.232+0000 7fce1fc00640 1 --2- 172.21.15.16:0/4274417768 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fcdf4068380 0x7fcdf406a840 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fce180b52d0 tx=0x7fce00009190 comp rx=0 tx=0).stop 2024-03-31T15:04:14.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:14.232+0000 7fce1fc00640 1 -- 172.21.15.16:0/4274417768 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fce180728b0 msgr2=0x7fce18166160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:14.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:14.232+0000 7fce1fc00640 1 --2- 172.21.15.16:0/4274417768 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fce180728b0 0x7fce18166160 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fce0801abd0 tx=0x7fce0801ad60 comp rx=0 tx=0).stop 2024-03-31T15:04:14.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:14.232+0000 7fce1fc00640 1 -- 172.21.15.16:0/4274417768 shutdown_connections 2024-03-31T15:04:14.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:14.232+0000 7fce1fc00640 1 --2- 172.21.15.16:0/4274417768 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fcdf4068380 0x7fcdf406a840 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:14.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:14.232+0000 7fce1fc00640 1 --2- 172.21.15.16:0/4274417768 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fce18145d10 0x7fce1813baa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:14.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:14.232+0000 7fce1fc00640 1 --2- 172.21.15.16:0/4274417768 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fce180b2200 0x7fce18140e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:14.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:14.232+0000 7fce1fc00640 1 --2- 172.21.15.16:0/4274417768 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fce180728b0 0x7fce18166160 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:14.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:14.232+0000 7fce1fc00640 1 -- 172.21.15.16:0/4274417768 >> 172.21.15.16:0/4274417768 conn(0x7fce1806b620 msgr2=0x7fce1815e3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:04:14.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:14.232+0000 7fce1fc00640 1 -- 172.21.15.16:0/4274417768 shutdown_connections 2024-03-31T15:04:14.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:14.232+0000 7fce1fc00640 1 -- 172.21.15.16:0/4274417768 wait complete. 2024-03-31T15:04:14.246 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi016.front.sepia.ceph.com... 2024-03-31T15:04:14.246 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:04:14.246 DEBUG:teuthology.orchestra.run.smithi016:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-03-31T15:04:14.274 INFO:teuthology.orchestra.run:waiting for 300 2024-03-31T15:04:14.274 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi016.front.sepia.ceph.com 2024-03-31T15:04:14.274 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:04:14.274 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-31T15:04:14.333 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-03-31T15:04:14.333 DEBUG:teuthology.orchestra.run.smithi089:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-31T15:04:14.341 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-03-31T15:04:14.342 DEBUG:teuthology.orchestra.run.smithi089:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-31T15:04:14.390 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-31T15:04:14.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:14.536+0000 7fb317800640 1 Processor -- start 2024-03-31T15:04:14.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:14.536+0000 7fb317800640 1 -- start start 2024-03-31T15:04:14.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:14.536+0000 7fb317800640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb3100728b0 0x7fb3100b1cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:14.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:14.536+0000 7fb317800640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb310160e70 con 0x7fb3100b2200 2024-03-31T15:04:14.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:14.536+0000 7fb317800640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb3101602f0 con 0x7fb310145d10 2024-03-31T15:04:14.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:14.536+0000 7fb317800640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fb3101605d0 con 0x7fb3100728b0 2024-03-31T15:04:14.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:14.536+0000 7fb315800640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fb3100b2200 0x7fb31015cd50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:46668/0 (socket says 172.21.15.16:46668) 2024-03-31T15:04:14.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:14.536+0000 7fb315800640 1 -- 172.21.15.16:0/1507830206 learned_addr learned my addr 172.21.15.16:0/1507830206 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:04:14.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:14.536+0000 7fb30fe00640 1 --2- 172.21.15.16:0/1507830206 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb3100728b0 0x7fb3100b1cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:14.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:14.536+0000 7fb30f400640 1 -- 172.21.15.16:0/1507830206 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2326824670 0 0) 0x7fb310160e70 con 0x7fb3100b2200 2024-03-31T15:04:14.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:14.536+0000 7fb30f400640 1 -- 172.21.15.16:0/1507830206 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb2f8003530 con 0x7fb3100b2200 2024-03-31T15:04:14.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:14.536+0000 7fb30f400640 1 -- 172.21.15.16:0/1507830206 <== mon.1 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 868355946 0 0) 0x7fb3101602f0 con 0x7fb310145d10 2024-03-31T15:04:14.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:14.536+0000 7fb30f400640 1 -- 172.21.15.16:0/1507830206 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb310160e70 con 0x7fb310145d10 2024-03-31T15:04:14.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:14.536+0000 7fb30fe00640 1 -- 172.21.15.16:0/1507830206 >> v1:172.21.15.89:6789/0 conn(0x7fb310145d10 legacy=0x7fb3100cf790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:14.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:14.536+0000 7fb30fe00640 1 -- 172.21.15.16:0/1507830206 >> v1:172.21.15.16:6789/0 conn(0x7fb3100b2200 legacy=0x7fb31015cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:14.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:14.536+0000 7fb30fe00640 1 -- 172.21.15.16:0/1507830206 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3101608b0 con 0x7fb3100728b0 2024-03-31T15:04:14.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:14.540+0000 7fb30fe00640 1 --2- 172.21.15.16:0/1507830206 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb3100728b0 0x7fb3100b1cc0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fb2fc0097f0 tx=0x7fb2fc01c4e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=639f5bd65810bc4a server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:14.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:14.540+0000 7fb30f400640 1 -- 172.21.15.16:0/1507830206 <== mon.2 v2:172.21.15.89:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2fc0b0070 con 0x7fb3100728b0 2024-03-31T15:04:14.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:14.540+0000 7fb30f400640 1 -- 172.21.15.16:0/1507830206 <== mon.2 v2:172.21.15.89:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb2fc0ab600 con 0x7fb3100728b0 2024-03-31T15:04:14.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:14.540+0000 7fb30f400640 1 -- 172.21.15.16:0/1507830206 <== mon.2 v2:172.21.15.89:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2fc0ab9d0 con 0x7fb3100728b0 2024-03-31T15:04:14.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:14.540+0000 7fb317800640 1 -- 172.21.15.16:0/1507830206 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb3100728b0 msgr2=0x7fb3100b1cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:14.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:14.540+0000 7fb317800640 1 --2- 172.21.15.16:0/1507830206 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb3100728b0 0x7fb3100b1cc0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fb2fc0097f0 tx=0x7fb2fc01c4e0 comp rx=0 tx=0).stop 2024-03-31T15:04:14.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:14.540+0000 7fb317800640 1 -- 172.21.15.16:0/1507830206 shutdown_connections 2024-03-31T15:04:14.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:14.540+0000 7fb317800640 1 --2- 172.21.15.16:0/1507830206 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb3100728b0 0x7fb3100b1cc0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:14.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:14.540+0000 7fb317800640 1 -- 172.21.15.16:0/1507830206 >> 172.21.15.16:0/1507830206 conn(0x7fb31006b620 msgr2=0x7fb31006ba30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:04:14.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:14.540+0000 7fb317800640 1 -- 172.21.15.16:0/1507830206 shutdown_connections 2024-03-31T15:04:14.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:14.540+0000 7fb317800640 1 -- 172.21.15.16:0/1507830206 wait complete. 2024-03-31T15:04:14.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:14.544+0000 7fb317800640 1 Processor -- start 2024-03-31T15:04:14.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:14.544+0000 7fb317800640 1 -- start start 2024-03-31T15:04:14.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:14.544+0000 7fb317800640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb3100728b0 0x7fb31013a0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:14.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:14.544+0000 7fb317800640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb3100b2200 0x7fb31013e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:14.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:14.544+0000 7fb317800640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb310145d10 0x7fb3101599c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:14.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:14.544+0000 7fb317800640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fb3100cf350 con 0x7fb3100728b0 2024-03-31T15:04:14.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:14.544+0000 7fb317800640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fb3100b2610 con 0x7fb310145d10 2024-03-31T15:04:14.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:14.544+0000 7fb317800640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fb3100cf4d0 con 0x7fb3100b2200 2024-03-31T15:04:14.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:14.544+0000 7fb314e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb3100728b0 0x7fb31013a0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:14.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:14.544+0000 7fb314e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb3100728b0 0x7fb31013a0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60542/0 (socket says 172.21.15.16:60542) 2024-03-31T15:04:14.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:14.544+0000 7fb314e00640 1 -- 172.21.15.16:0/710379148 learned_addr learned my addr 172.21.15.16:0/710379148 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:04:14.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:14.544+0000 7fb315800640 1 --2- 172.21.15.16:0/710379148 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb310145d10 0x7fb3101599c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:14.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:14.544+0000 7fb30fe00640 1 --2- 172.21.15.16:0/710379148 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb3100b2200 0x7fb31013e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:14.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:14.544+0000 7fb314e00640 1 -- 172.21.15.16:0/710379148 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb3100b2200 msgr2=0x7fb31013e4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:14.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:14.544+0000 7fb314e00640 1 --2- 172.21.15.16:0/710379148 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb3100b2200 0x7fb31013e4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:14.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:14.544+0000 7fb314e00640 1 -- 172.21.15.16:0/710379148 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb310145d10 msgr2=0x7fb3101599c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:14.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:14.544+0000 7fb314e00640 1 --2- 172.21.15.16:0/710379148 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb310145d10 0x7fb3101599c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:14.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:14.544+0000 7fb314e00640 1 -- 172.21.15.16:0/710379148 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb31015a390 con 0x7fb3100728b0 2024-03-31T15:04:14.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:14.544+0000 7fb30fe00640 1 --2- 172.21.15.16:0/710379148 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb3100b2200 0x7fb31013e4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:04:14.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:14.544+0000 7fb314e00640 1 --2- 172.21.15.16:0/710379148 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb3100728b0 0x7fb31013a0f0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fb30400c430 tx=0x7fb30400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:14.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:14.544+0000 7fb30cc00640 1 -- 172.21.15.16:0/710379148 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb304017070 con 0x7fb3100728b0 2024-03-31T15:04:14.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:14.544+0000 7fb30cc00640 1 -- 172.21.15.16:0/710379148 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb3040052f0 con 0x7fb3100728b0 2024-03-31T15:04:14.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:14.544+0000 7fb30cc00640 1 -- 172.21.15.16:0/710379148 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb304003840 con 0x7fb3100728b0 2024-03-31T15:04:14.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:14.544+0000 7fb317800640 1 -- 172.21.15.16:0/710379148 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb310159f00 con 0x7fb3100728b0 2024-03-31T15:04:14.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:14.544+0000 7fb317800640 1 -- 172.21.15.16:0/710379148 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb31013eed0 con 0x7fb3100728b0 2024-03-31T15:04:14.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:14.548+0000 7fb317800640 1 -- 172.21.15.16:0/710379148 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb310069950 con 0x7fb3100728b0 2024-03-31T15:04:14.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:14.548+0000 7fb30cc00640 1 -- 172.21.15.16:0/710379148 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fb30401a070 con 0x7fb3100728b0 2024-03-31T15:04:14.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:14.548+0000 7fb30cc00640 1 --2- 172.21.15.16:0/710379148 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fb2f0068380 0x7fb2f006a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:14.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:14.548+0000 7fb30cc00640 1 -- 172.21.15.16:0/710379148 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fb304085df0 con 0x7fb3100728b0 2024-03-31T15:04:14.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:14.548+0000 7fb30fe00640 1 --2- 172.21.15.16:0/710379148 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fb2f0068380 0x7fb2f006a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:14.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:14.548+0000 7fb30fe00640 1 --2- 172.21.15.16:0/710379148 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fb2f0068380 0x7fb2f006a840 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fb2fc0ad000 tx=0x7fb2fc008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:14.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:14.552+0000 7fb30cc00640 1 -- 172.21.15.16:0/710379148 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fb304055c40 con 0x7fb3100728b0 2024-03-31T15:04:14.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:14.720+0000 7fb317800640 1 -- 172.21.15.16:0/710379148 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fb310143040 con 0x7fb3100728b0 2024-03-31T15:04:14.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:14.720+0000 7fb30cc00640 1 -- 172.21.15.16:0/710379148 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) ==== 74+0+16533 (secure 0 0 0) 0x7fb304059f40 con 0x7fb3100728b0 2024-03-31T15:04:14.715 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:04:14.715 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":21,"fsid":"286eb5da-fb47-4550-b290-d9519c13b9a5","created":"2024-03-31T15:03:11.438952+0000","modified":"2024-03-31T15:03:37.957829+0000","last_up_change":"2024-03-31T15:03:22.885535+0000","last_in_change":"2024-03-31T15:03:13.941667+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-31T15:03:22.550787+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-31T15:03:34.488425+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-31T15:03:35.299116+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"92a66450-ef19-40d8-ac3d-d760b23b0bda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6801","nonce":932712244}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6803","nonce":932712244}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6807","nonce":932712244}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6805","nonce":932712244}]},"public_addr":"172.21.15.16:6801/932712244","cluster_addr":"172.21.15.16:6803/932712244","heartbeat_back_addr":"172.21.15.16:6807/932712244","heartbeat_front_addr":"172.21.15.16:6805/932712244","state":["exists","up"]},{"osd":1,"uuid":"da9d1f2b-1968-4959-b843-faa231988e92","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6810","nonce":1398737533}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6811","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6813","nonce":1398737533}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6821","nonce":1398737533}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6815","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6816","nonce":1398737533}]},"public_addr":"172.21.15.16:6810/1398737533","cluster_addr":"172.21.15.16:6813/1398737533","heartbeat_back_addr":"172.21.15.16:6821/1398737533","heartbeat_front_addr":"172.21.15.16:6816/1398737533","state":["exists","up"]},{"osd":2,"uuid":"01a73662-9a21-4b1f-bcbb-42b9636a4638","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6819","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6822","nonce":3018526646}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6825","nonce":3018526646}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6831","nonce":3018526646}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6827","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6829","nonce":3018526646}]},"public_addr":"172.21.15.16:6822/3018526646","cluster_addr":"172.21.15.16:6825/3018526646","heartbeat_back_addr":"172.21.15.16:6831/3018526646","heartbeat_front_addr":"172.21.15.16:6829/3018526646","state":["exists","up"]},{"osd":3,"uuid":"c7771fd3-e18e-4500-ac38-23cf10ebcd75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6809","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6812","nonce":3360352974}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6817","nonce":3360352974}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6828","nonce":3360352974}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6823","nonce":3360352974}]},"public_addr":"172.21.15.16:6812/3360352974","cluster_addr":"172.21.15.16:6817/3360352974","heartbeat_back_addr":"172.21.15.16:6828/3360352974","heartbeat_front_addr":"172.21.15.16:6823/3360352974","state":["exists","up"]},{"osd":4,"uuid":"88260730-1482-4fbd-a40f-cb1124e9df4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6801","nonce":1856844856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6803","nonce":1856844856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6807","nonce":1856844856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6805","nonce":1856844856}]},"public_addr":"172.21.15.89:6801/1856844856","cluster_addr":"172.21.15.89:6803/1856844856","heartbeat_back_addr":"172.21.15.89:6807/1856844856","heartbeat_front_addr":"172.21.15.89:6805/1856844856","state":["exists","up"]},{"osd":5,"uuid":"4cb2c00c-fae0-4dd8-9d27-ec4f0093ccf7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6814","nonce":50395527}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6817","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6820","nonce":50395527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6831","nonce":50395527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6823","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6826","nonce":50395527}]},"public_addr":"172.21.15.89:6814/50395527","cluster_addr":"172.21.15.89:6820/50395527","heartbeat_back_addr":"172.21.15.89:6831/50395527","heartbeat_front_addr":"172.21.15.89:6826/50395527","state":["exists","up"]},{"osd":6,"uuid":"81ce0c19-2554-4cae-acbe-66a05e9e9b9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6809","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6812","nonce":3470471752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6815","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6818","nonce":3470471752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6827","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6829","nonce":3470471752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6821","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6824","nonce":3470471752}]},"public_addr":"172.21.15.89:6812/3470471752","cluster_addr":"172.21.15.89:6818/3470471752","heartbeat_back_addr":"172.21.15.89:6829/3470471752","heartbeat_front_addr":"172.21.15.89:6824/3470471752","state":["exists","up"]},{"osd":7,"uuid":"85e4313b-d37d-442e-8042-153bf20a847d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6811","nonce":1796778630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6813","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6816","nonce":1796778630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6825","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6828","nonce":1796778630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6819","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6822","nonce":1796778630}]},"public_addr":"172.21.15.89:6811/1796778630","cluster_addr":"172.21.15.89:6816/1796778630","heartbeat_back_addr":"172.21.15.89:6828/1796778630","heartbeat_front_addr":"172.21.15.89:6822/1796778630","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.439088+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.529970+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.680512+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:21.044161+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.582111+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.536558+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.409522+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.590229+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-31T15:04:14.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:14.720+0000 7fb317800640 1 -- 172.21.15.16:0/710379148 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fb2f0068380 msgr2=0x7fb2f006a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:14.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:14.720+0000 7fb317800640 1 --2- 172.21.15.16:0/710379148 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fb2f0068380 0x7fb2f006a840 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fb2fc0ad000 tx=0x7fb2fc008040 comp rx=0 tx=0).stop 2024-03-31T15:04:14.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:14.724+0000 7fb317800640 1 -- 172.21.15.16:0/710379148 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb3100728b0 msgr2=0x7fb31013a0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:14.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:14.724+0000 7fb317800640 1 --2- 172.21.15.16:0/710379148 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb3100728b0 0x7fb31013a0f0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fb30400c430 tx=0x7fb30400c8f0 comp rx=0 tx=0).stop 2024-03-31T15:04:14.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:14.724+0000 7fb317800640 1 -- 172.21.15.16:0/710379148 shutdown_connections 2024-03-31T15:04:14.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:14.724+0000 7fb317800640 1 --2- 172.21.15.16:0/710379148 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fb2f0068380 0x7fb2f006a840 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:14.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:14.724+0000 7fb317800640 1 --2- 172.21.15.16:0/710379148 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb310145d10 0x7fb3101599c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:14.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:14.724+0000 7fb317800640 1 --2- 172.21.15.16:0/710379148 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb3100b2200 0x7fb31013e4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:14.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:14.724+0000 7fb317800640 1 --2- 172.21.15.16:0/710379148 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb3100728b0 0x7fb31013a0f0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:14.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:14.724+0000 7fb317800640 1 -- 172.21.15.16:0/710379148 >> 172.21.15.16:0/710379148 conn(0x7fb31006b620 msgr2=0x7fb3100b27f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:04:14.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:14.724+0000 7fb317800640 1 -- 172.21.15.16:0/710379148 shutdown_connections 2024-03-31T15:04:14.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:14.724+0000 7fb317800640 1 -- 172.21.15.16:0/710379148 wait complete. 2024-03-31T15:04:14.735 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-31T15:03:22.550787+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-31T15:03:34.488425+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-31T15:03:35.299116+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-31T15:04:14.736 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-31T15:04:14.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:14.884+0000 7f42fb200640 1 Processor -- start 2024-03-31T15:04:14.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:14.884+0000 7f42fb200640 1 -- start start 2024-03-31T15:04:14.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:14.888+0000 7f42fb200640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f42f415c390 0x7f42f40cfb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:14.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:14.888+0000 7f42fb200640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f42f406fbc0 0x7f42f4070050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:14.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:14.888+0000 7f42fb200640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f42f40cd750 con 0x7f42f40d2b40 2024-03-31T15:04:14.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:14.888+0000 7f42fb200640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f42f415c770 con 0x7f42f406fbc0 2024-03-31T15:04:14.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:14.888+0000 7f42fb200640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f42f40cda30 con 0x7f42f415c390 2024-03-31T15:04:14.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:14.888+0000 7f42fa800640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f42f406fbc0 0x7f42f4070050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:14.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:14.888+0000 7f42f9e00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f42f415c390 0x7f42f40cfb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:14.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:14.888+0000 7f42fa800640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f42f406fbc0 0x7f42f4070050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60558/0 (socket says 172.21.15.16:60558) 2024-03-31T15:04:14.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:14.888+0000 7f42fa800640 1 -- 172.21.15.16:0/1817435647 learned_addr learned my addr 172.21.15.16:0/1817435647 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:04:14.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:14.888+0000 7f42f9400640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f42f40d2b40 0x7f42f40d00a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.16:60950/0 (socket says 172.21.15.16:60950) 2024-03-31T15:04:14.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:14.888+0000 7f42f9e00640 1 -- 172.21.15.16:0/1817435647 >> v2:172.21.15.16:3300/0 conn(0x7f42f406fbc0 msgr2=0x7f42f4070050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:04:14.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:14.888+0000 7f42f9e00640 1 --2- 172.21.15.16:0/1817435647 >> v2:172.21.15.16:3300/0 conn(0x7f42f406fbc0 0x7f42f4070050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:14.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:14.888+0000 7f42f9e00640 1 -- 172.21.15.16:0/1817435647 >> v1:172.21.15.89:6789/0 conn(0x7f42f40d2b40 legacy=0x7f42f40d00a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:14.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:14.888+0000 7f42f9e00640 1 -- 172.21.15.16:0/1817435647 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42f40cdd10 con 0x7f42f415c390 2024-03-31T15:04:14.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:14.888+0000 7f42f8a00640 1 -- 172.21.15.16:0/1817435647 <== mon.1 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2093917669 0 0) 0x7f42f40cd750 con 0x7f42f40d2b40 2024-03-31T15:04:14.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:14.892+0000 7f42f9e00640 1 --2- 172.21.15.16:0/1817435647 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f42f415c390 0x7f42f40cfb60 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f42e80097c0 tx=0x7f42e801c4a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6f2d1c34387b425 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:14.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:14.892+0000 7f42f8a00640 1 -- 172.21.15.16:0/1817435647 <== mon.2 v2:172.21.15.89:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42e80b0070 con 0x7f42f415c390 2024-03-31T15:04:14.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:14.892+0000 7f42f8a00640 1 -- 172.21.15.16:0/1817435647 <== mon.2 v2:172.21.15.89:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f42e80ab600 con 0x7f42f415c390 2024-03-31T15:04:14.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:14.892+0000 7f42f8a00640 1 -- 172.21.15.16:0/1817435647 <== mon.2 v2:172.21.15.89:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42e80ab9d0 con 0x7f42f415c390 2024-03-31T15:04:14.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:14.892+0000 7f42fb200640 1 -- 172.21.15.16:0/1817435647 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f42f415c390 msgr2=0x7f42f40cfb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:14.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:14.892+0000 7f42fb200640 1 --2- 172.21.15.16:0/1817435647 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f42f415c390 0x7f42f40cfb60 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f42e80097c0 tx=0x7f42e801c4a0 comp rx=0 tx=0).stop 2024-03-31T15:04:14.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:14.892+0000 7f42fb200640 1 -- 172.21.15.16:0/1817435647 shutdown_connections 2024-03-31T15:04:14.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:14.892+0000 7f42fb200640 1 --2- 172.21.15.16:0/1817435647 >> v2:172.21.15.16:3300/0 conn(0x7f42f406fbc0 0x7f42f4070050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:14.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:14.892+0000 7f42fb200640 1 --2- 172.21.15.16:0/1817435647 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f42f415c390 0x7f42f40cfb60 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:14.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:14.892+0000 7f42fb200640 1 -- 172.21.15.16:0/1817435647 >> 172.21.15.16:0/1817435647 conn(0x7f42f40d1f90 msgr2=0x7f42f40d23a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:04:14.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:14.892+0000 7f42fb200640 1 -- 172.21.15.16:0/1817435647 shutdown_connections 2024-03-31T15:04:14.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:14.892+0000 7f42fb200640 1 -- 172.21.15.16:0/1817435647 wait complete. 2024-03-31T15:04:14.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:14.892+0000 7f42fb200640 1 Processor -- start 2024-03-31T15:04:14.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:14.892+0000 7f42fb200640 1 -- start start 2024-03-31T15:04:14.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:14.892+0000 7f42fb200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f42f406fbc0 0x7f42f4156860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:14.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:14.892+0000 7f42fb200640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f42f40d2b40 0x7f42f40c6960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:14.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:14.892+0000 7f42fb200640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f42f415c390 0x7f42f40c9060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:14.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:14.892+0000 7f42fb200640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f42f4168e50 con 0x7f42f406fbc0 2024-03-31T15:04:14.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:14.892+0000 7f42fb200640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f42f40d09c0 con 0x7f42f40d2b40 2024-03-31T15:04:14.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:14.892+0000 7f42fb200640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f42f4168fd0 con 0x7f42f415c390 2024-03-31T15:04:14.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:14.892+0000 7f42f9e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f42f406fbc0 0x7f42f4156860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:14.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:14.892+0000 7f42f9e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f42f406fbc0 0x7f42f4156860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60568/0 (socket says 172.21.15.16:60568) 2024-03-31T15:04:14.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:14.892+0000 7f42f9e00640 1 -- 172.21.15.16:0/2761865501 learned_addr learned my addr 172.21.15.16:0/2761865501 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:04:14.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:14.892+0000 7f42fa800640 1 --2- 172.21.15.16:0/2761865501 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f42f415c390 0x7f42f40c9060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:14.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:14.892+0000 7f42f9400640 1 --2- 172.21.15.16:0/2761865501 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f42f40d2b40 0x7f42f40c6960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:14.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:14.896+0000 7f42f9e00640 1 -- 172.21.15.16:0/2761865501 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f42f415c390 msgr2=0x7f42f40c9060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:14.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:14.896+0000 7f42f9e00640 1 --2- 172.21.15.16:0/2761865501 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f42f415c390 0x7f42f40c9060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:14.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:14.896+0000 7f42f9e00640 1 -- 172.21.15.16:0/2761865501 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f42f40d2b40 msgr2=0x7f42f40c6960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:14.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:14.896+0000 7f42f9e00640 1 --2- 172.21.15.16:0/2761865501 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f42f40d2b40 0x7f42f40c6960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:14.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:14.896+0000 7f42f9e00640 1 -- 172.21.15.16:0/2761865501 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42f41557e0 con 0x7f42f406fbc0 2024-03-31T15:04:14.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:14.896+0000 7f42f9400640 1 --2- 172.21.15.16:0/2761865501 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f42f40d2b40 0x7f42f40c6960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:04:14.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:14.896+0000 7f42f9e00640 1 --2- 172.21.15.16:0/2761865501 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f42f406fbc0 0x7f42f4156860 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f42e801a0d0 tx=0x7f42e8019240 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:14.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:14.896+0000 7f42ee000640 1 -- 172.21.15.16:0/2761865501 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42e80bb070 con 0x7f42f406fbc0 2024-03-31T15:04:14.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:14.896+0000 7f42ee000640 1 -- 172.21.15.16:0/2761865501 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f42e80b7460 con 0x7f42f406fbc0 2024-03-31T15:04:14.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:14.896+0000 7f42fb200640 1 -- 172.21.15.16:0/2761865501 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f42f4155ac0 con 0x7f42f406fbc0 2024-03-31T15:04:14.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:14.896+0000 7f42ee000640 1 -- 172.21.15.16:0/2761865501 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42e80b0040 con 0x7f42f406fbc0 2024-03-31T15:04:14.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:14.896+0000 7f42fb200640 1 -- 172.21.15.16:0/2761865501 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f42f4156080 con 0x7f42f406fbc0 2024-03-31T15:04:14.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:14.896+0000 7f42fb200640 1 -- 172.21.15.16:0/2761865501 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f42f404f8c0 con 0x7f42f406fbc0 2024-03-31T15:04:14.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:14.896+0000 7f42ee000640 1 -- 172.21.15.16:0/2761865501 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f42e80be070 con 0x7f42f406fbc0 2024-03-31T15:04:14.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:14.896+0000 7f42ee000640 1 --2- 172.21.15.16:0/2761865501 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f42d8068380 0x7f42d806a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:14.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:14.896+0000 7f42ee000640 1 -- 172.21.15.16:0/2761865501 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f42e8120f20 con 0x7f42f406fbc0 2024-03-31T15:04:14.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:14.896+0000 7f42f9400640 1 --2- 172.21.15.16:0/2761865501 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f42d8068380 0x7f42d806a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:14.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:14.900+0000 7f42f9400640 1 --2- 172.21.15.16:0/2761865501 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f42d8068380 0x7f42d806a840 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f42f40b0f20 tx=0x7f42e4007cb0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:14.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:14.900+0000 7f42ee000640 1 -- 172.21.15.16:0/2761865501 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f42e8016090 con 0x7f42f406fbc0 2024-03-31T15:04:15.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:15.064+0000 7f42fb200640 1 -- 172.21.15.16:0/2761865501 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f42f40b5f00 con 0x7f42f406fbc0 2024-03-31T15:04:15.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:15.068+0000 7f42ee000640 1 -- 172.21.15.16:0/2761865501 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) ==== 93+0+10 (secure 0 0 0) 0x7f42e80f0e00 con 0x7f42f406fbc0 2024-03-31T15:04:15.061 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2024-03-31T15:04:15.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:15.068+0000 7f42fb200640 1 -- 172.21.15.16:0/2761865501 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f42d8068380 msgr2=0x7f42d806a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:15.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:15.068+0000 7f42fb200640 1 --2- 172.21.15.16:0/2761865501 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f42d8068380 0x7f42d806a840 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f42f40b0f20 tx=0x7f42e4007cb0 comp rx=0 tx=0).stop 2024-03-31T15:04:15.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:15.068+0000 7f42fb200640 1 -- 172.21.15.16:0/2761865501 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f42f406fbc0 msgr2=0x7f42f4156860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:15.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:15.068+0000 7f42fb200640 1 --2- 172.21.15.16:0/2761865501 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f42f406fbc0 0x7f42f4156860 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f42e801a0d0 tx=0x7f42e8019240 comp rx=0 tx=0).stop 2024-03-31T15:04:15.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:15.068+0000 7f42fb200640 1 -- 172.21.15.16:0/2761865501 shutdown_connections 2024-03-31T15:04:15.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:15.068+0000 7f42fb200640 1 --2- 172.21.15.16:0/2761865501 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f42d8068380 0x7f42d806a840 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:15.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:15.068+0000 7f42fb200640 1 --2- 172.21.15.16:0/2761865501 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f42f415c390 0x7f42f40c9060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:15.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:15.068+0000 7f42fb200640 1 --2- 172.21.15.16:0/2761865501 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f42f40d2b40 0x7f42f40c6960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:15.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:15.068+0000 7f42fb200640 1 --2- 172.21.15.16:0/2761865501 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f42f406fbc0 0x7f42f4156860 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:15.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:15.068+0000 7f42fb200640 1 -- 172.21.15.16:0/2761865501 >> 172.21.15.16:0/2761865501 conn(0x7f42f40d1f90 msgr2=0x7f42f4157680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:04:15.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:15.068+0000 7f42fb200640 1 -- 172.21.15.16:0/2761865501 shutdown_connections 2024-03-31T15:04:15.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:15.068+0000 7f42fb200640 1 -- 172.21.15.16:0/2761865501 wait complete. 2024-03-31T15:04:15.083 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-31T15:04:15.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:15.232+0000 7f682dc00640 1 Processor -- start 2024-03-31T15:04:15.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:15.232+0000 7f682dc00640 1 -- start start 2024-03-31T15:04:15.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:15.232+0000 7f682dc00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f682813dff0 0x7f682806d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:15.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:15.232+0000 7f682dc00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f68280cd0e0 0x7f682806da10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:15.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:15.232+0000 7f682dc00640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f682815fbc0 con 0x7f682813e3d0 2024-03-31T15:04:15.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:15.232+0000 7f682dc00640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f682815f730 con 0x7f682813dff0 2024-03-31T15:04:15.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:15.232+0000 7f682dc00640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f682815ee00 con 0x7f68280cd0e0 2024-03-31T15:04:15.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:15.232+0000 7f6826a00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f682813dff0 0x7f682806d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:15.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:15.232+0000 7f681fe00640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f682813e3d0 0x7f68280cc9d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:46672/0 (socket says 172.21.15.16:46672) 2024-03-31T15:04:15.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:15.232+0000 7f681fe00640 1 -- 172.21.15.16:0/3187884117 learned_addr learned my addr 172.21.15.16:0/3187884117 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:04:15.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:15.232+0000 7f6827400640 1 --2- 172.21.15.16:0/3187884117 >> v2:172.21.15.89:3300/0 conn(0x7f68280cd0e0 0x7f682806da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:15.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:15.236+0000 7f6826a00640 1 -- 172.21.15.16:0/3187884117 >> v2:172.21.15.89:3300/0 conn(0x7f68280cd0e0 msgr2=0x7f682806da10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:04:15.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:15.236+0000 7f6826a00640 1 --2- 172.21.15.16:0/3187884117 >> v2:172.21.15.89:3300/0 conn(0x7f68280cd0e0 0x7f682806da10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:15.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:15.236+0000 7f6826a00640 1 -- 172.21.15.16:0/3187884117 >> v1:172.21.15.16:6789/0 conn(0x7f682813e3d0 legacy=0x7f68280cc9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:15.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:15.236+0000 7f6826a00640 1 -- 172.21.15.16:0/3187884117 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f682815f0e0 con 0x7f682813dff0 2024-03-31T15:04:15.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:15.236+0000 7f6826000640 1 -- 172.21.15.16:0/3187884117 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3537904019 0 0) 0x7f682815fbc0 con 0x7f682813e3d0 2024-03-31T15:04:15.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:15.236+0000 7f6826a00640 1 --2- 172.21.15.16:0/3187884117 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f682813dff0 0x7f682806d4d0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f682815b6e0 tx=0x7f681801c3f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5888ae2c08a43e21 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:15.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:15.236+0000 7f6826000640 1 -- 172.21.15.16:0/3187884117 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68180b0070 con 0x7f682813dff0 2024-03-31T15:04:15.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:15.236+0000 7f6826000640 1 -- 172.21.15.16:0/3187884117 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f68180ab600 con 0x7f682813dff0 2024-03-31T15:04:15.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:15.236+0000 7f6826000640 1 -- 172.21.15.16:0/3187884117 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68180ab9d0 con 0x7f682813dff0 2024-03-31T15:04:15.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:15.236+0000 7f682dc00640 1 -- 172.21.15.16:0/3187884117 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f682813dff0 msgr2=0x7f682806d4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:15.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:15.236+0000 7f682dc00640 1 --2- 172.21.15.16:0/3187884117 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f682813dff0 0x7f682806d4d0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f682815b6e0 tx=0x7f681801c3f0 comp rx=0 tx=0).stop 2024-03-31T15:04:15.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:15.240+0000 7f682dc00640 1 -- 172.21.15.16:0/3187884117 shutdown_connections 2024-03-31T15:04:15.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:15.240+0000 7f682dc00640 1 --2- 172.21.15.16:0/3187884117 >> v2:172.21.15.89:3300/0 conn(0x7f68280cd0e0 0x7f682806da10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:15.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:15.240+0000 7f682dc00640 1 --2- 172.21.15.16:0/3187884117 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f682813dff0 0x7f682806d4d0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:15.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:15.240+0000 7f682dc00640 1 -- 172.21.15.16:0/3187884117 >> 172.21.15.16:0/3187884117 conn(0x7f682806ee40 msgr2=0x7f682806b620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:04:15.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:15.240+0000 7f682dc00640 1 -- 172.21.15.16:0/3187884117 shutdown_connections 2024-03-31T15:04:15.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:15.240+0000 7f682dc00640 1 -- 172.21.15.16:0/3187884117 wait complete. 2024-03-31T15:04:15.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:15.240+0000 7f682dc00640 1 Processor -- start 2024-03-31T15:04:15.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:15.240+0000 7f682dc00640 1 -- start start 2024-03-31T15:04:15.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:15.240+0000 7f682dc00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f68280cd0e0 0x7f682813b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:15.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:15.240+0000 7f682dc00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f682813dff0 0x7f68280b0ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:15.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:15.240+0000 7f682dc00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f682813e3d0 0x7f6828144f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:15.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:15.240+0000 7f682dc00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f68280cf710 con 0x7f682813e3d0 2024-03-31T15:04:15.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:15.240+0000 7f682dc00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f68280cf590 con 0x7f682813dff0 2024-03-31T15:04:15.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:15.240+0000 7f682dc00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f68280cf890 con 0x7f68280cd0e0 2024-03-31T15:04:15.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:15.240+0000 7f6827400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f682813e3d0 0x7f6828144f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:15.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:15.240+0000 7f6827400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f682813e3d0 0x7f6828144f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60586/0 (socket says 172.21.15.16:60586) 2024-03-31T15:04:15.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:15.240+0000 7f6827400640 1 -- 172.21.15.16:0/246642241 learned_addr learned my addr 172.21.15.16:0/246642241 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:04:15.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:15.240+0000 7f681fe00640 1 --2- 172.21.15.16:0/246642241 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f682813dff0 0x7f68280b0ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:15.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:15.240+0000 7f6826a00640 1 --2- 172.21.15.16:0/246642241 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f68280cd0e0 0x7f682813b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:15.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:15.240+0000 7f6827400640 1 -- 172.21.15.16:0/246642241 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f68280cd0e0 msgr2=0x7f682813b6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:15.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:15.240+0000 7f6827400640 1 --2- 172.21.15.16:0/246642241 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f68280cd0e0 0x7f682813b6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:15.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:15.240+0000 7f6827400640 1 -- 172.21.15.16:0/246642241 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f682813dff0 msgr2=0x7f68280b0ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:15.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:15.240+0000 7f6827400640 1 --2- 172.21.15.16:0/246642241 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f682813dff0 0x7f68280b0ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:15.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:15.240+0000 7f6827400640 1 -- 172.21.15.16:0/246642241 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f682815bb30 con 0x7f682813e3d0 2024-03-31T15:04:15.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:15.240+0000 7f6827400640 1 --2- 172.21.15.16:0/246642241 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f682813e3d0 0x7f6828144f30 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f682000efc0 tx=0x7f682000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:15.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:15.240+0000 7f681ea00640 1 -- 172.21.15.16:0/246642241 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6820018020 con 0x7f682813e3d0 2024-03-31T15:04:15.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:15.240+0000 7f681ea00640 1 -- 172.21.15.16:0/246642241 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6820014480 con 0x7f682813e3d0 2024-03-31T15:04:15.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:15.240+0000 7f682dc00640 1 -- 172.21.15.16:0/246642241 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f682815be10 con 0x7f682813e3d0 2024-03-31T15:04:15.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:15.240+0000 7f681ea00640 1 -- 172.21.15.16:0/246642241 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6820014870 con 0x7f682813e3d0 2024-03-31T15:04:15.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:15.244+0000 7f682dc00640 1 -- 172.21.15.16:0/246642241 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f682815c3d0 con 0x7f682813e3d0 2024-03-31T15:04:15.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:15.244+0000 7f682dc00640 1 -- 172.21.15.16:0/246642241 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6828069950 con 0x7f682813e3d0 2024-03-31T15:04:15.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:15.244+0000 7f681ea00640 1 -- 172.21.15.16:0/246642241 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f682001b020 con 0x7f682813e3d0 2024-03-31T15:04:15.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:15.244+0000 7f681ea00640 1 --2- 172.21.15.16:0/246642241 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f6804068380 0x7f680406a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:15.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:15.244+0000 7f681ea00640 1 -- 172.21.15.16:0/246642241 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f6820087410 con 0x7f682813e3d0 2024-03-31T15:04:15.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:15.244+0000 7f6826a00640 1 --2- 172.21.15.16:0/246642241 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f6804068380 0x7f680406a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:15.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:15.244+0000 7f6826a00640 1 --2- 172.21.15.16:0/246642241 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f6804068380 0x7f680406a840 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f68180ad000 tx=0x7f68180a4f90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:15.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:15.248+0000 7f681ea00640 1 -- 172.21.15.16:0/246642241 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f6820022070 con 0x7f682813e3d0 2024-03-31T15:04:15.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:15.412+0000 7f682dc00640 1 -- 172.21.15.16:0/246642241 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f682806e4a0 con 0x7f682813e3d0 2024-03-31T15:04:15.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:15.412+0000 7f681ea00640 1 -- 172.21.15.16:0/246642241 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) ==== 104+0+11 (secure 0 0 0) 0x7f6820010040 con 0x7f682813e3d0 2024-03-31T15:04:15.406 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2024-03-31T15:04:15.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:15.412+0000 7f682dc00640 1 -- 172.21.15.16:0/246642241 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f6804068380 msgr2=0x7f680406a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:15.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:15.412+0000 7f682dc00640 1 --2- 172.21.15.16:0/246642241 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f6804068380 0x7f680406a840 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f68180ad000 tx=0x7f68180a4f90 comp rx=0 tx=0).stop 2024-03-31T15:04:15.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:15.412+0000 7f682dc00640 1 -- 172.21.15.16:0/246642241 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f682813e3d0 msgr2=0x7f6828144f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:15.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:15.412+0000 7f682dc00640 1 --2- 172.21.15.16:0/246642241 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f682813e3d0 0x7f6828144f30 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f682000efc0 tx=0x7f682000c490 comp rx=0 tx=0).stop 2024-03-31T15:04:15.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:15.412+0000 7f682dc00640 1 -- 172.21.15.16:0/246642241 shutdown_connections 2024-03-31T15:04:15.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:15.412+0000 7f682dc00640 1 --2- 172.21.15.16:0/246642241 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f6804068380 0x7f680406a840 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:15.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:15.412+0000 7f682dc00640 1 --2- 172.21.15.16:0/246642241 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f682813e3d0 0x7f6828144f30 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:15.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:15.412+0000 7f682dc00640 1 --2- 172.21.15.16:0/246642241 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f682813dff0 0x7f68280b0ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:15.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:15.412+0000 7f682dc00640 1 --2- 172.21.15.16:0/246642241 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f68280cd0e0 0x7f682813b6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:15.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:15.412+0000 7f682dc00640 1 -- 172.21.15.16:0/246642241 >> 172.21.15.16:0/246642241 conn(0x7f682806ee40 msgr2=0x7f6828071c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:04:15.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:15.412+0000 7f682dc00640 1 -- 172.21.15.16:0/246642241 shutdown_connections 2024-03-31T15:04:15.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:15.412+0000 7f682dc00640 1 -- 172.21.15.16:0/246642241 wait complete. 2024-03-31T15:04:15.427 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-31T15:04:15.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:15.576+0000 7f1ac8e00640 1 Processor -- start 2024-03-31T15:04:15.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:15.576+0000 7f1ac8e00640 1 -- start start 2024-03-31T15:04:15.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:15.576+0000 7f1ac8e00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f1ac4141590 0x7f1ac4144340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:15.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:15.576+0000 7f1ac8e00640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1ac4144980 con 0x7f1ac4070030 2024-03-31T15:04:15.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:15.576+0000 7f1ac8e00640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1ac4161fe0 con 0x7f1ac41411b0 2024-03-31T15:04:15.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:15.576+0000 7f1ac8e00640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f1ac415f910 con 0x7f1ac4141590 2024-03-31T15:04:15.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:15.576+0000 7f1abbe00640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f1ac4070030 0x7f1ac4143c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:46684/0 (socket says 172.21.15.16:46684) 2024-03-31T15:04:15.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:15.576+0000 7f1abbe00640 1 -- 172.21.15.16:0/2613930083 learned_addr learned my addr 172.21.15.16:0/2613930083 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:04:15.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:15.576+0000 7f1ac2800640 1 --2- 172.21.15.16:0/2613930083 >> v2:172.21.15.16:3300/0 conn(0x7f1ac4141590 0x7f1ac4144340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:15.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:15.580+0000 7f1ac2800640 1 -- 172.21.15.16:0/2613930083 >> v1:172.21.15.89:6789/0 conn(0x7f1ac41411b0 legacy=0x7f1ac406f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:15.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:15.580+0000 7f1ac2800640 1 -- 172.21.15.16:0/2613930083 >> v1:172.21.15.16:6789/0 conn(0x7f1ac4070030 legacy=0x7f1ac4143c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:15.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:15.580+0000 7f1ac1400640 1 -- 172.21.15.16:0/2613930083 <== mon.1 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4029313395 0 0) 0x7f1ac4161fe0 con 0x7f1ac41411b0 2024-03-31T15:04:15.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:15.580+0000 7f1ac2800640 1 -- 172.21.15.16:0/2613930083 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ac41622c0 con 0x7f1ac4141590 2024-03-31T15:04:15.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:15.580+0000 7f1ac2800640 1 --2- 172.21.15.16:0/2613930083 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1ac4141590 0x7f1ac4144340 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f1abc009d90 tx=0x7f1abc01c800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72397d55efc2fa45 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:15.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:15.580+0000 7f1ac1400640 1 -- 172.21.15.16:0/2613930083 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1abc0b2070 con 0x7f1ac4141590 2024-03-31T15:04:15.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:15.580+0000 7f1ac1400640 1 -- 172.21.15.16:0/2613930083 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1abc0a5860 con 0x7f1ac4141590 2024-03-31T15:04:15.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:15.580+0000 7f1ac1400640 1 -- 172.21.15.16:0/2613930083 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1abc0a5c30 con 0x7f1ac4141590 2024-03-31T15:04:15.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:15.580+0000 7f1ac8e00640 1 -- 172.21.15.16:0/2613930083 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1ac4141590 msgr2=0x7f1ac4144340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:15.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:15.580+0000 7f1ac8e00640 1 --2- 172.21.15.16:0/2613930083 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1ac4141590 0x7f1ac4144340 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f1abc009d90 tx=0x7f1abc01c800 comp rx=0 tx=0).stop 2024-03-31T15:04:15.577 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:15.584+0000 7f1ac8e00640 1 -- 172.21.15.16:0/2613930083 shutdown_connections 2024-03-31T15:04:15.577 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:15.584+0000 7f1ac8e00640 1 --2- 172.21.15.16:0/2613930083 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1ac4141590 0x7f1ac4144340 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:15.577 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:15.584+0000 7f1ac8e00640 1 -- 172.21.15.16:0/2613930083 >> 172.21.15.16:0/2613930083 conn(0x7f1ac4067930 msgr2=0x7f1ac40b3ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:04:15.577 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:15.584+0000 7f1ac8e00640 1 -- 172.21.15.16:0/2613930083 shutdown_connections 2024-03-31T15:04:15.577 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:15.584+0000 7f1ac8e00640 1 -- 172.21.15.16:0/2613930083 wait complete. 2024-03-31T15:04:15.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:15.584+0000 7f1ac8e00640 1 Processor -- start 2024-03-31T15:04:15.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:15.584+0000 7f1ac8e00640 1 -- start start 2024-03-31T15:04:15.580 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:15.584+0000 7f1ac8e00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f1ac4070030 0x7f1ac413cd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:15.580 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:15.584+0000 7f1ac8e00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1ac41411b0 0x7f1ac415d820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:15.580 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:15.584+0000 7f1ac8e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1ac4141590 0x7f1ac415b830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:15.580 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:15.584+0000 7f1ac8e00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f1ac4162da0 con 0x7f1ac4141590 2024-03-31T15:04:15.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:15.584+0000 7f1ac8e00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f1ac4162910 con 0x7f1ac41411b0 2024-03-31T15:04:15.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:15.584+0000 7f1ac8e00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f1ac41692d0 con 0x7f1ac4070030 2024-03-31T15:04:15.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:15.584+0000 7f1ac2800640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1ac4141590 0x7f1ac415b830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:15.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:15.584+0000 7f1ac2800640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1ac4141590 0x7f1ac415b830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60596/0 (socket says 172.21.15.16:60596) 2024-03-31T15:04:15.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:15.584+0000 7f1ac2800640 1 -- 172.21.15.16:0/534393584 learned_addr learned my addr 172.21.15.16:0/534393584 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:04:15.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:15.584+0000 7f1abbe00640 1 --2- 172.21.15.16:0/534393584 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1ac41411b0 0x7f1ac415d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:15.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:15.584+0000 7f1ac1e00640 1 --2- 172.21.15.16:0/534393584 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f1ac4070030 0x7f1ac413cd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:15.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:15.584+0000 7f1ac2800640 1 -- 172.21.15.16:0/534393584 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f1ac4070030 msgr2=0x7f1ac413cd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:15.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:15.584+0000 7f1ac2800640 1 --2- 172.21.15.16:0/534393584 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f1ac4070030 0x7f1ac413cd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:15.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:15.584+0000 7f1ac2800640 1 -- 172.21.15.16:0/534393584 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1ac41411b0 msgr2=0x7f1ac415d820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:15.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:15.584+0000 7f1ac2800640 1 --2- 172.21.15.16:0/534393584 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1ac41411b0 0x7f1ac415d820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:15.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:15.584+0000 7f1ac2800640 1 -- 172.21.15.16:0/534393584 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ac415e300 con 0x7f1ac4141590 2024-03-31T15:04:15.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:15.584+0000 7f1ac1e00640 1 --2- 172.21.15.16:0/534393584 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f1ac4070030 0x7f1ac413cd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:04:15.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:15.584+0000 7f1ac2800640 1 --2- 172.21.15.16:0/534393584 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1ac4141590 0x7f1ac415b830 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f1abc01a430 tx=0x7f1abc0195a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:15.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:15.584+0000 7f1aba000640 1 -- 172.21.15.16:0/534393584 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1abc0b2040 con 0x7f1ac4141590 2024-03-31T15:04:15.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:15.584+0000 7f1aba000640 1 -- 172.21.15.16:0/534393584 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1abc0bf070 con 0x7f1ac4141590 2024-03-31T15:04:15.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:15.584+0000 7f1aba000640 1 -- 172.21.15.16:0/534393584 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1abc0adc10 con 0x7f1ac4141590 2024-03-31T15:04:15.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:15.584+0000 7f1ac8e00640 1 -- 172.21.15.16:0/534393584 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1ac41592b0 con 0x7f1ac4141590 2024-03-31T15:04:15.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:15.588+0000 7f1ac8e00640 1 -- 172.21.15.16:0/534393584 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1ac41561e0 con 0x7f1ac4141590 2024-03-31T15:04:15.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:15.588+0000 7f1ac8e00640 1 -- 172.21.15.16:0/534393584 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1ac41590a0 con 0x7f1ac4141590 2024-03-31T15:04:15.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:15.588+0000 7f1aba000640 1 -- 172.21.15.16:0/534393584 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f1abc0b8470 con 0x7f1ac4141590 2024-03-31T15:04:15.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:15.588+0000 7f1aba000640 1 --2- 172.21.15.16:0/534393584 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f1aa0068380 0x7f1aa006a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:15.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:15.588+0000 7f1aba000640 1 -- 172.21.15.16:0/534393584 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f1abc122080 con 0x7f1ac4141590 2024-03-31T15:04:15.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:15.588+0000 7f1ac1e00640 1 --2- 172.21.15.16:0/534393584 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f1aa0068380 0x7f1aa006a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:15.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:15.588+0000 7f1ac1e00640 1 --2- 172.21.15.16:0/534393584 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f1aa0068380 0x7f1aa006a840 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f1ab4007b30 tx=0x7f1ab4009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:15.587 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:15.592+0000 7f1aba000640 1 -- 172.21.15.16:0/534393584 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f1abc016480 con 0x7f1ac4141590 2024-03-31T15:04:15.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:15.760+0000 7f1ac8e00640 1 -- 172.21.15.16:0/534393584 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f1ac415b610 con 0x7f1ac4141590 2024-03-31T15:04:15.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:15.760+0000 7f1aba000640 1 -- 172.21.15.16:0/534393584 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) ==== 100+0+11 (secure 0 0 0) 0x7f1abc0f26c0 con 0x7f1ac4141590 2024-03-31T15:04:15.755 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 64 2024-03-31T15:04:15.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:15.764+0000 7f1ac8e00640 1 -- 172.21.15.16:0/534393584 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f1aa0068380 msgr2=0x7f1aa006a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:15.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:15.764+0000 7f1ac8e00640 1 --2- 172.21.15.16:0/534393584 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f1aa0068380 0x7f1aa006a840 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f1ab4007b30 tx=0x7f1ab4009040 comp rx=0 tx=0).stop 2024-03-31T15:04:15.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:15.764+0000 7f1ac8e00640 1 -- 172.21.15.16:0/534393584 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1ac4141590 msgr2=0x7f1ac415b830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:15.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:15.764+0000 7f1ac8e00640 1 --2- 172.21.15.16:0/534393584 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1ac4141590 0x7f1ac415b830 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f1abc01a430 tx=0x7f1abc0195a0 comp rx=0 tx=0).stop 2024-03-31T15:04:15.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:15.764+0000 7f1ac8e00640 1 -- 172.21.15.16:0/534393584 shutdown_connections 2024-03-31T15:04:15.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:15.764+0000 7f1ac8e00640 1 --2- 172.21.15.16:0/534393584 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f1aa0068380 0x7f1aa006a840 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:15.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:15.764+0000 7f1ac8e00640 1 --2- 172.21.15.16:0/534393584 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1ac4141590 0x7f1ac415b830 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:15.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:15.764+0000 7f1ac8e00640 1 --2- 172.21.15.16:0/534393584 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1ac41411b0 0x7f1ac415d820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:15.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:15.764+0000 7f1ac8e00640 1 --2- 172.21.15.16:0/534393584 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f1ac4070030 0x7f1ac413cd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:15.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:15.764+0000 7f1ac8e00640 1 -- 172.21.15.16:0/534393584 >> 172.21.15.16:0/534393584 conn(0x7f1ac4067930 msgr2=0x7f1ac4161990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:04:15.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:15.764+0000 7f1ac8e00640 1 -- 172.21.15.16:0/534393584 shutdown_connections 2024-03-31T15:04:15.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:15.764+0000 7f1ac8e00640 1 -- 172.21.15.16:0/534393584 wait complete. 2024-03-31T15:04:15.777 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-31T15:04:15.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:15.924+0000 7f1572200640 1 Processor -- start 2024-03-31T15:04:15.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:15.924+0000 7f1572200640 1 -- start start 2024-03-31T15:04:15.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:15.924+0000 7f1572200640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f156c06f920 0x7f156c06fd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:15.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:15.924+0000 7f1572200640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f156c0d2010 0x7f156c0d24a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:15.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:15.924+0000 7f1572200640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f156c0d0920 con 0x7f156c145e40 2024-03-31T15:04:15.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:15.924+0000 7f1572200640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f156c0702c0 con 0x7f156c06f920 2024-03-31T15:04:15.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:15.924+0000 7f1572200640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f156c0d06d0 con 0x7f156c0d2010 2024-03-31T15:04:15.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:15.924+0000 7f1562a00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f156c06f920 0x7f156c06fd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:15.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:15.924+0000 7f1562a00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f156c06f920 0x7f156c06fd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60600/0 (socket says 172.21.15.16:60600) 2024-03-31T15:04:15.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:15.924+0000 7f1562a00640 1 -- 172.21.15.16:0/1351406211 learned_addr learned my addr 172.21.15.16:0/1351406211 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:04:15.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:15.924+0000 7f156be00640 1 --2- 172.21.15.16:0/1351406211 >> v2:172.21.15.89:3300/0 conn(0x7f156c0d2010 0x7f156c0d24a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:15.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:15.924+0000 7f1562a00640 1 -- 172.21.15.16:0/1351406211 >> v2:172.21.15.89:3300/0 conn(0x7f156c0d2010 msgr2=0x7f156c0d24a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:04:15.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:15.924+0000 7f1562a00640 1 --2- 172.21.15.16:0/1351406211 >> v2:172.21.15.89:3300/0 conn(0x7f156c0d2010 0x7f156c0d24a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:15.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:15.924+0000 7f1562a00640 1 -- 172.21.15.16:0/1351406211 >> v1:172.21.15.89:6789/0 conn(0x7f156c145e40 legacy=0x7f156c0cc740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:15.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:15.924+0000 7f1562a00640 1 -- 172.21.15.16:0/1351406211 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f156c0d0c00 con 0x7f156c06f920 2024-03-31T15:04:15.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:15.928+0000 7f1562a00640 1 --2- 172.21.15.16:0/1351406211 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f156c06f920 0x7f156c06fd80 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f156c160e90 tx=0x7f155401c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7230572dbe977aac server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:15.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:15.928+0000 7f156aa00640 1 -- 172.21.15.16:0/1351406211 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15540a57a0 con 0x7f156c06f920 2024-03-31T15:04:15.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:15.928+0000 7f156aa00640 1 -- 172.21.15.16:0/1351406211 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f15540a5940 con 0x7f156c06f920 2024-03-31T15:04:15.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:15.928+0000 7f156aa00640 1 -- 172.21.15.16:0/1351406211 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15540a5d10 con 0x7f156c06f920 2024-03-31T15:04:15.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:15.928+0000 7f1572200640 1 -- 172.21.15.16:0/1351406211 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f156c06f920 msgr2=0x7f156c06fd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:15.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:15.928+0000 7f1572200640 1 --2- 172.21.15.16:0/1351406211 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f156c06f920 0x7f156c06fd80 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f156c160e90 tx=0x7f155401c9f0 comp rx=0 tx=0).stop 2024-03-31T15:04:15.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:15.928+0000 7f1572200640 1 -- 172.21.15.16:0/1351406211 shutdown_connections 2024-03-31T15:04:15.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:15.928+0000 7f1572200640 1 --2- 172.21.15.16:0/1351406211 >> v2:172.21.15.89:3300/0 conn(0x7f156c0d2010 0x7f156c0d24a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:15.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:15.928+0000 7f1572200640 1 --2- 172.21.15.16:0/1351406211 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f156c06f920 0x7f156c06fd80 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:15.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:15.928+0000 7f1572200640 1 -- 172.21.15.16:0/1351406211 >> 172.21.15.16:0/1351406211 conn(0x7f156c067330 msgr2=0x7f156c067740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:04:15.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:15.928+0000 7f1572200640 1 -- 172.21.15.16:0/1351406211 shutdown_connections 2024-03-31T15:04:15.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:15.928+0000 7f1572200640 1 -- 172.21.15.16:0/1351406211 wait complete. 2024-03-31T15:04:15.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:15.932+0000 7f1572200640 1 Processor -- start 2024-03-31T15:04:15.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:15.932+0000 7f1572200640 1 -- start start 2024-03-31T15:04:15.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:15.932+0000 7f1572200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f156c06f920 0x7f156c143f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:15.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:15.932+0000 7f1572200640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f156c0d2010 0x7f156c144960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:15.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:15.932+0000 7f1572200640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f156c145e40 0x7f156c1533b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:15.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:15.932+0000 7f1572200640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f156c0d0ee0 con 0x7f156c06f920 2024-03-31T15:04:15.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:15.932+0000 7f1572200640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f156c0ce9e0 con 0x7f156c145e40 2024-03-31T15:04:15.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:15.932+0000 7f1572200640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f156c0d2df0 con 0x7f156c0d2010 2024-03-31T15:04:15.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:15.932+0000 7f156b400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f156c06f920 0x7f156c143f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:15.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:15.932+0000 7f156b400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f156c06f920 0x7f156c143f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60602/0 (socket says 172.21.15.16:60602) 2024-03-31T15:04:15.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:15.932+0000 7f156b400640 1 -- 172.21.15.16:0/1933915424 learned_addr learned my addr 172.21.15.16:0/1933915424 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:04:15.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:15.932+0000 7f156be00640 1 --2- 172.21.15.16:0/1933915424 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f156c145e40 0x7f156c1533b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:15.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:15.932+0000 7f1562a00640 1 --2- 172.21.15.16:0/1933915424 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f156c0d2010 0x7f156c144960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:15.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:15.932+0000 7f156b400640 1 -- 172.21.15.16:0/1933915424 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f156c0d2010 msgr2=0x7f156c144960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:15.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:15.932+0000 7f156b400640 1 --2- 172.21.15.16:0/1933915424 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f156c0d2010 0x7f156c144960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:15.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:15.932+0000 7f156b400640 1 -- 172.21.15.16:0/1933915424 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f156c145e40 msgr2=0x7f156c1533b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:15.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:15.932+0000 7f156b400640 1 --2- 172.21.15.16:0/1933915424 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f156c145e40 0x7f156c1533b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:15.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:15.932+0000 7f156b400640 1 -- 172.21.15.16:0/1933915424 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f156c0cbb10 con 0x7f156c06f920 2024-03-31T15:04:15.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:15.932+0000 7f156b400640 1 --2- 172.21.15.16:0/1933915424 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f156c06f920 0x7f156c143f40 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f155c00c370 tx=0x7f155c00c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:15.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:15.932+0000 7f1563e00640 1 -- 172.21.15.16:0/1933915424 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f155c0052d0 con 0x7f156c06f920 2024-03-31T15:04:15.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:15.932+0000 7f1563e00640 1 -- 172.21.15.16:0/1933915424 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f155c005470 con 0x7f156c06f920 2024-03-31T15:04:15.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:15.932+0000 7f1572200640 1 -- 172.21.15.16:0/1933915424 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f156c0cbdf0 con 0x7f156c06f920 2024-03-31T15:04:15.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:15.932+0000 7f1563e00640 1 -- 172.21.15.16:0/1933915424 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f155c007260 con 0x7f156c06f920 2024-03-31T15:04:15.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:15.932+0000 7f1572200640 1 -- 172.21.15.16:0/1933915424 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f156c144630 con 0x7f156c06f920 2024-03-31T15:04:15.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:15.932+0000 7f1572200640 1 -- 172.21.15.16:0/1933915424 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f156c069950 con 0x7f156c06f920 2024-03-31T15:04:15.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:15.936+0000 7f1563e00640 1 -- 172.21.15.16:0/1933915424 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f155c007400 con 0x7f156c06f920 2024-03-31T15:04:15.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:15.936+0000 7f1563e00640 1 --2- 172.21.15.16:0/1933915424 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f1548068380 0x7f154806a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:15.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:15.936+0000 7f1563e00640 1 -- 172.21.15.16:0/1933915424 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f155c085db0 con 0x7f156c06f920 2024-03-31T15:04:15.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:15.936+0000 7f1562a00640 1 --2- 172.21.15.16:0/1933915424 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f1548068380 0x7f154806a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:15.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:15.936+0000 7f1562a00640 1 --2- 172.21.15.16:0/1933915424 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f1548068380 0x7f154806a840 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f15540ae000 tx=0x7f15540a7070 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:15.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:15.940+0000 7f1563e00640 1 -- 172.21.15.16:0/1933915424 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f155c08a070 con 0x7f156c06f920 2024-03-31T15:04:16.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:16.120+0000 7f1572200640 1 -- 172.21.15.16:0/1933915424 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f156c0cf3c0 con 0x7f156c06f920 2024-03-31T15:04:16.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:16.120+0000 7f1563e00640 1 -- 172.21.15.16:0/1933915424 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v21) ==== 80+0+0 (secure 0 0 0) 0x7f155c055d10 con 0x7f156c06f920 2024-03-31T15:04:16.113 INFO:teuthology.orchestra.run.smithi016.stderr:listed 0 entries 2024-03-31T15:04:16.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:16.120+0000 7f1572200640 1 -- 172.21.15.16:0/1933915424 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f1548068380 msgr2=0x7f154806a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:16.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:16.120+0000 7f1572200640 1 --2- 172.21.15.16:0/1933915424 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f1548068380 0x7f154806a840 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f15540ae000 tx=0x7f15540a7070 comp rx=0 tx=0).stop 2024-03-31T15:04:16.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:16.120+0000 7f1572200640 1 -- 172.21.15.16:0/1933915424 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f156c06f920 msgr2=0x7f156c143f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:16.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:16.120+0000 7f1572200640 1 --2- 172.21.15.16:0/1933915424 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f156c06f920 0x7f156c143f40 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f155c00c370 tx=0x7f155c00c830 comp rx=0 tx=0).stop 2024-03-31T15:04:16.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:16.120+0000 7f1572200640 1 -- 172.21.15.16:0/1933915424 shutdown_connections 2024-03-31T15:04:16.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:16.120+0000 7f1572200640 1 --2- 172.21.15.16:0/1933915424 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f1548068380 0x7f154806a840 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:16.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:16.120+0000 7f1572200640 1 --2- 172.21.15.16:0/1933915424 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f156c145e40 0x7f156c1533b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:16.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:16.120+0000 7f1572200640 1 --2- 172.21.15.16:0/1933915424 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f156c0d2010 0x7f156c144960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:16.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:16.120+0000 7f1572200640 1 --2- 172.21.15.16:0/1933915424 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f156c06f920 0x7f156c143f40 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:16.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:16.120+0000 7f1572200640 1 -- 172.21.15.16:0/1933915424 >> 172.21.15.16:0/1933915424 conn(0x7f156c067330 msgr2=0x7f156c0b2540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:04:16.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:16.120+0000 7f1572200640 1 -- 172.21.15.16:0/1933915424 shutdown_connections 2024-03-31T15:04:16.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:16.120+0000 7f1572200640 1 -- 172.21.15.16:0/1933915424 wait complete. 2024-03-31T15:04:16.135 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-03-31T15:04:16.135 DEBUG:teuthology.orchestra.run.smithi089:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-31T15:04:16.144 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-31T15:04:16.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:16.288+0000 7f489c800640 1 Processor -- start 2024-03-31T15:04:16.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:16.288+0000 7f489c800640 1 -- start start 2024-03-31T15:04:16.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:16.288+0000 7f489c800640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f489406dbc0 0x7f489415f650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:16.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:16.288+0000 7f489c800640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f4894070030 0x7f4894162180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:16.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:16.288+0000 7f489c800640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f48940cbe90 con 0x7f48940cb440 2024-03-31T15:04:16.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:16.288+0000 7f489c800640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f489415fb90 con 0x7f489406dbc0 2024-03-31T15:04:16.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:16.288+0000 7f489c800640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f489415f1c0 con 0x7f4894070030 2024-03-31T15:04:16.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:16.288+0000 7f4899e00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f489406dbc0 0x7f489415f650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:16.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:16.288+0000 7f4899400640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f48940cb440 0x7f489406f920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:46692/0 (socket says 172.21.15.16:46692) 2024-03-31T15:04:16.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:16.288+0000 7f4899400640 1 -- 172.21.15.16:0/2453507282 learned_addr learned my addr 172.21.15.16:0/2453507282 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:04:16.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:16.288+0000 7f489a800640 1 --2- 172.21.15.16:0/2453507282 >> v2:172.21.15.89:3300/0 conn(0x7f4894070030 0x7f4894162180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:16.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:16.292+0000 7f4899e00640 1 -- 172.21.15.16:0/2453507282 >> v2:172.21.15.89:3300/0 conn(0x7f4894070030 msgr2=0x7f4894162180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:04:16.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:16.292+0000 7f4899e00640 1 --2- 172.21.15.16:0/2453507282 >> v2:172.21.15.89:3300/0 conn(0x7f4894070030 0x7f4894162180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:16.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:16.292+0000 7f4899e00640 1 -- 172.21.15.16:0/2453507282 >> v1:172.21.15.16:6789/0 conn(0x7f48940cb440 legacy=0x7f489406f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:16.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:16.292+0000 7f4899e00640 1 -- 172.21.15.16:0/2453507282 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48940cc170 con 0x7f489406dbc0 2024-03-31T15:04:16.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:16.292+0000 7f4898a00640 1 -- 172.21.15.16:0/2453507282 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 297206078 0 0) 0x7f48940cbe90 con 0x7f48940cb440 2024-03-31T15:04:16.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:16.292+0000 7f4899e00640 1 --2- 172.21.15.16:0/2453507282 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f489406dbc0 0x7f489415f650 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f4894073be0 tx=0x7f488801c3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d1302234196de618 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:16.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:16.292+0000 7f4898a00640 1 -- 172.21.15.16:0/2453507282 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48880b0070 con 0x7f489406dbc0 2024-03-31T15:04:16.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:16.292+0000 7f4898a00640 1 -- 172.21.15.16:0/2453507282 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f48880ab600 con 0x7f489406dbc0 2024-03-31T15:04:16.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:16.292+0000 7f4898a00640 1 -- 172.21.15.16:0/2453507282 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48880ab9d0 con 0x7f489406dbc0 2024-03-31T15:04:16.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:16.292+0000 7f489c800640 1 -- 172.21.15.16:0/2453507282 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f489406dbc0 msgr2=0x7f489415f650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:16.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:16.292+0000 7f489c800640 1 --2- 172.21.15.16:0/2453507282 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f489406dbc0 0x7f489415f650 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f4894073be0 tx=0x7f488801c3c0 comp rx=0 tx=0).stop 2024-03-31T15:04:16.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:16.292+0000 7f489c800640 1 -- 172.21.15.16:0/2453507282 shutdown_connections 2024-03-31T15:04:16.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:16.292+0000 7f489c800640 1 --2- 172.21.15.16:0/2453507282 >> v2:172.21.15.89:3300/0 conn(0x7f4894070030 0x7f4894162180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:16.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:16.292+0000 7f489c800640 1 --2- 172.21.15.16:0/2453507282 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f489406dbc0 0x7f489415f650 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:16.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:16.296+0000 7f489c800640 1 -- 172.21.15.16:0/2453507282 >> 172.21.15.16:0/2453507282 conn(0x7f4894067330 msgr2=0x7f4894067740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:04:16.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:16.296+0000 7f489c800640 1 -- 172.21.15.16:0/2453507282 shutdown_connections 2024-03-31T15:04:16.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:16.296+0000 7f489c800640 1 -- 172.21.15.16:0/2453507282 wait complete. 2024-03-31T15:04:16.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:16.296+0000 7f489c800640 1 Processor -- start 2024-03-31T15:04:16.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:16.296+0000 7f489c800640 1 -- start start 2024-03-31T15:04:16.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:16.296+0000 7f489c800640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f489406dbc0 0x7f489415ac50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:16.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:16.296+0000 7f489c800640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4894070030 0x7f4894154b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:16.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:16.296+0000 7f489c800640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f48940cb440 0x7f4894155070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:16.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:16.296+0000 7f489c800640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f489406cad0 con 0x7f489406dbc0 2024-03-31T15:04:16.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:16.296+0000 7f489c800640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f489406c950 con 0x7f4894070030 2024-03-31T15:04:16.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:16.296+0000 7f489c800640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f489416b520 con 0x7f48940cb440 2024-03-31T15:04:16.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:16.296+0000 7f4899400640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4894070030 0x7f4894154b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:16.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:16.296+0000 7f4899400640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4894070030 0x7f4894154b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.16:44604/0 (socket says 172.21.15.16:44604) 2024-03-31T15:04:16.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:16.296+0000 7f4899400640 1 -- 172.21.15.16:0/3747349811 learned_addr learned my addr 172.21.15.16:0/3747349811 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:04:16.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:16.296+0000 7f489a800640 1 --2- 172.21.15.16:0/3747349811 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f48940cb440 0x7f4894155070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:16.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:16.296+0000 7f4899e00640 1 --2- 172.21.15.16:0/3747349811 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f489406dbc0 0x7f489415ac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:16.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:16.296+0000 7f4899400640 1 -- 172.21.15.16:0/3747349811 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f48940cb440 msgr2=0x7f4894155070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:16.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:16.296+0000 7f4899400640 1 --2- 172.21.15.16:0/3747349811 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f48940cb440 0x7f4894155070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:16.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:16.296+0000 7f4899400640 1 -- 172.21.15.16:0/3747349811 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f489406dbc0 msgr2=0x7f489415ac50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:16.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:16.296+0000 7f4899400640 1 --2- 172.21.15.16:0/3747349811 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f489406dbc0 0x7f489415ac50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:16.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:16.296+0000 7f4899400640 1 -- 172.21.15.16:0/3747349811 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4894163220 con 0x7f4894070030 2024-03-31T15:04:16.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:16.296+0000 7f4899400640 1 --2- 172.21.15.16:0/3747349811 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4894070030 0x7f4894154b30 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f48840011f0 tx=0x7f488400c9b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:16.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:16.300+0000 7f488e000640 1 -- 172.21.15.16:0/3747349811 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4884018070 con 0x7f4894070030 2024-03-31T15:04:16.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:16.300+0000 7f488e000640 1 -- 172.21.15.16:0/3747349811 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f48840074e0 con 0x7f4894070030 2024-03-31T15:04:16.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:16.300+0000 7f488e000640 1 -- 172.21.15.16:0/3747349811 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4884004a10 con 0x7f4894070030 2024-03-31T15:04:16.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:16.300+0000 7f489c800640 1 -- 172.21.15.16:0/3747349811 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f489415b330 con 0x7f4894070030 2024-03-31T15:04:16.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:16.300+0000 7f489c800640 1 -- 172.21.15.16:0/3747349811 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f489415b8f0 con 0x7f4894070030 2024-03-31T15:04:16.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:16.300+0000 7f489c800640 1 -- 172.21.15.16:0/3747349811 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4894069950 con 0x7f4894070030 2024-03-31T15:04:16.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:16.300+0000 7f488e000640 1 -- 172.21.15.16:0/3747349811 <== mon.1 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f4884004bb0 con 0x7f4894070030 2024-03-31T15:04:16.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:16.300+0000 7f488e000640 1 --2- 172.21.15.16:0/3747349811 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f4878068380 0x7f487806a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:16.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:16.300+0000 7f488e000640 1 -- 172.21.15.16:0/3747349811 <== mon.1 v2:172.21.15.89:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f4884085c20 con 0x7f4894070030 2024-03-31T15:04:16.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:16.300+0000 7f4899e00640 1 --2- 172.21.15.16:0/3747349811 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f4878068380 0x7f487806a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:16.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:16.304+0000 7f4899e00640 1 --2- 172.21.15.16:0/3747349811 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f4878068380 0x7f487806a840 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f48880ad000 tx=0x7f4888008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:16.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:16.304+0000 7f488e000640 1 -- 172.21.15.16:0/3747349811 <== mon.1 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f4884055af0 con 0x7f4894070030 2024-03-31T15:04:16.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:16.472+0000 7f489c800640 1 -- 172.21.15.16:0/3747349811 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f48940cf380 con 0x7f4894070030 2024-03-31T15:04:16.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:16.472+0000 7f488e000640 1 -- 172.21.15.16:0/3747349811 <== mon.1 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) ==== 74+0+16533 (secure 0 0 0) 0x7f4884059df0 con 0x7f4894070030 2024-03-31T15:04:16.467 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:04:16.467 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":21,"fsid":"286eb5da-fb47-4550-b290-d9519c13b9a5","created":"2024-03-31T15:03:11.438952+0000","modified":"2024-03-31T15:03:37.957829+0000","last_up_change":"2024-03-31T15:03:22.885535+0000","last_in_change":"2024-03-31T15:03:13.941667+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-31T15:03:22.550787+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-31T15:03:34.488425+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-31T15:03:35.299116+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"92a66450-ef19-40d8-ac3d-d760b23b0bda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6801","nonce":932712244}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6803","nonce":932712244}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6807","nonce":932712244}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6805","nonce":932712244}]},"public_addr":"172.21.15.16:6801/932712244","cluster_addr":"172.21.15.16:6803/932712244","heartbeat_back_addr":"172.21.15.16:6807/932712244","heartbeat_front_addr":"172.21.15.16:6805/932712244","state":["exists","up"]},{"osd":1,"uuid":"da9d1f2b-1968-4959-b843-faa231988e92","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6810","nonce":1398737533}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6811","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6813","nonce":1398737533}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6821","nonce":1398737533}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6815","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6816","nonce":1398737533}]},"public_addr":"172.21.15.16:6810/1398737533","cluster_addr":"172.21.15.16:6813/1398737533","heartbeat_back_addr":"172.21.15.16:6821/1398737533","heartbeat_front_addr":"172.21.15.16:6816/1398737533","state":["exists","up"]},{"osd":2,"uuid":"01a73662-9a21-4b1f-bcbb-42b9636a4638","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6819","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6822","nonce":3018526646}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6825","nonce":3018526646}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6831","nonce":3018526646}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6827","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6829","nonce":3018526646}]},"public_addr":"172.21.15.16:6822/3018526646","cluster_addr":"172.21.15.16:6825/3018526646","heartbeat_back_addr":"172.21.15.16:6831/3018526646","heartbeat_front_addr":"172.21.15.16:6829/3018526646","state":["exists","up"]},{"osd":3,"uuid":"c7771fd3-e18e-4500-ac38-23cf10ebcd75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6809","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6812","nonce":3360352974}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6817","nonce":3360352974}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6828","nonce":3360352974}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6823","nonce":3360352974}]},"public_addr":"172.21.15.16:6812/3360352974","cluster_addr":"172.21.15.16:6817/3360352974","heartbeat_back_addr":"172.21.15.16:6828/3360352974","heartbeat_front_addr":"172.21.15.16:6823/3360352974","state":["exists","up"]},{"osd":4,"uuid":"88260730-1482-4fbd-a40f-cb1124e9df4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6801","nonce":1856844856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6803","nonce":1856844856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6807","nonce":1856844856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6805","nonce":1856844856}]},"public_addr":"172.21.15.89:6801/1856844856","cluster_addr":"172.21.15.89:6803/1856844856","heartbeat_back_addr":"172.21.15.89:6807/1856844856","heartbeat_front_addr":"172.21.15.89:6805/1856844856","state":["exists","up"]},{"osd":5,"uuid":"4cb2c00c-fae0-4dd8-9d27-ec4f0093ccf7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6814","nonce":50395527}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6817","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6820","nonce":50395527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6831","nonce":50395527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6823","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6826","nonce":50395527}]},"public_addr":"172.21.15.89:6814/50395527","cluster_addr":"172.21.15.89:6820/50395527","heartbeat_back_addr":"172.21.15.89:6831/50395527","heartbeat_front_addr":"172.21.15.89:6826/50395527","state":["exists","up"]},{"osd":6,"uuid":"81ce0c19-2554-4cae-acbe-66a05e9e9b9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6809","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6812","nonce":3470471752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6815","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6818","nonce":3470471752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6827","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6829","nonce":3470471752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6821","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6824","nonce":3470471752}]},"public_addr":"172.21.15.89:6812/3470471752","cluster_addr":"172.21.15.89:6818/3470471752","heartbeat_back_addr":"172.21.15.89:6829/3470471752","heartbeat_front_addr":"172.21.15.89:6824/3470471752","state":["exists","up"]},{"osd":7,"uuid":"85e4313b-d37d-442e-8042-153bf20a847d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6811","nonce":1796778630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6813","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6816","nonce":1796778630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6825","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6828","nonce":1796778630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6819","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6822","nonce":1796778630}]},"public_addr":"172.21.15.89:6811/1796778630","cluster_addr":"172.21.15.89:6816/1796778630","heartbeat_back_addr":"172.21.15.89:6828/1796778630","heartbeat_front_addr":"172.21.15.89:6822/1796778630","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.439088+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.529970+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.680512+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:21.044161+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.582111+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.536558+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.409522+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.590229+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-31T15:04:16.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:16.472+0000 7f489c800640 1 -- 172.21.15.16:0/3747349811 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f4878068380 msgr2=0x7f487806a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:16.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:16.472+0000 7f489c800640 1 --2- 172.21.15.16:0/3747349811 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f4878068380 0x7f487806a840 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f48880ad000 tx=0x7f4888008040 comp rx=0 tx=0).stop 2024-03-31T15:04:16.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:16.472+0000 7f489c800640 1 -- 172.21.15.16:0/3747349811 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4894070030 msgr2=0x7f4894154b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:16.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:16.472+0000 7f489c800640 1 --2- 172.21.15.16:0/3747349811 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4894070030 0x7f4894154b30 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f48840011f0 tx=0x7f488400c9b0 comp rx=0 tx=0).stop 2024-03-31T15:04:16.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:16.476+0000 7f489c800640 1 -- 172.21.15.16:0/3747349811 shutdown_connections 2024-03-31T15:04:16.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:16.476+0000 7f489c800640 1 --2- 172.21.15.16:0/3747349811 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f4878068380 0x7f487806a840 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:16.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:16.476+0000 7f489c800640 1 --2- 172.21.15.16:0/3747349811 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f48940cb440 0x7f4894155070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:16.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:16.476+0000 7f489c800640 1 --2- 172.21.15.16:0/3747349811 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4894070030 0x7f4894154b30 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:16.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:16.476+0000 7f489c800640 1 --2- 172.21.15.16:0/3747349811 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f489406dbc0 0x7f489415ac50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:16.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:16.476+0000 7f489c800640 1 -- 172.21.15.16:0/3747349811 >> 172.21.15.16:0/3747349811 conn(0x7f4894067330 msgr2=0x7f489415de70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:04:16.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:16.476+0000 7f489c800640 1 -- 172.21.15.16:0/3747349811 shutdown_connections 2024-03-31T15:04:16.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:16.476+0000 7f489c800640 1 -- 172.21.15.16:0/3747349811 wait complete. 2024-03-31T15:04:16.487 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-31T15:03:22.550787+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-31T15:03:34.488425+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-31T15:03:35.299116+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-31T15:04:16.488 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-31T15:04:16.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:16.636+0000 7f0230600640 1 Processor -- start 2024-03-31T15:04:16.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:16.636+0000 7f0230600640 1 -- start start 2024-03-31T15:04:16.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:16.636+0000 7f0230600640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f022813dff0 0x7f022806d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:16.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:16.636+0000 7f0230600640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f02280cd0e0 0x7f022806da10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:16.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:16.636+0000 7f0230600640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f022815fbc0 con 0x7f022813e3d0 2024-03-31T15:04:16.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:16.636+0000 7f0230600640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f022815f730 con 0x7f022813dff0 2024-03-31T15:04:16.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:16.636+0000 7f0230600640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f022815ee00 con 0x7f02280cd0e0 2024-03-31T15:04:16.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:16.640+0000 7f022dc00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f022813dff0 0x7f022806d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:16.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:16.640+0000 7f022e600640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f02280cd0e0 0x7f022806da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:16.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:16.640+0000 7f022d200640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f022813e3d0 0x7f02280cc9d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.16:60980/0 (socket says 172.21.15.16:60980) 2024-03-31T15:04:16.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:16.640+0000 7f022d200640 1 -- 172.21.15.16:0/1463206579 learned_addr learned my addr 172.21.15.16:0/1463206579 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:04:16.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:16.640+0000 7f022e600640 1 -- 172.21.15.16:0/1463206579 >> v2:172.21.15.89:3300/0 conn(0x7f022813dff0 msgr2=0x7f022806d4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:04:16.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:16.640+0000 7f022e600640 1 --2- 172.21.15.16:0/1463206579 >> v2:172.21.15.89:3300/0 conn(0x7f022813dff0 0x7f022806d4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:16.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:16.640+0000 7f022e600640 1 -- 172.21.15.16:0/1463206579 >> v1:172.21.15.89:6789/0 conn(0x7f022813e3d0 legacy=0x7f02280cc9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:16.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:16.640+0000 7f022e600640 1 -- 172.21.15.16:0/1463206579 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f022815f0e0 con 0x7f02280cd0e0 2024-03-31T15:04:16.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:16.640+0000 7f022dc00640 1 --2- 172.21.15.16:0/1463206579 >> v2:172.21.15.89:3300/0 conn(0x7f022813dff0 0x7f022806d4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:04:16.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:16.640+0000 7f022e600640 1 --2- 172.21.15.16:0/1463206579 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f02280cd0e0 0x7f022806da10 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f0220009d90 tx=0x7f022001c8e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c2218ff2ca600b37 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:16.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:16.640+0000 7f0227e00640 1 -- 172.21.15.16:0/1463206579 <== mon.2 v2:172.21.15.89:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02200b2070 con 0x7f02280cd0e0 2024-03-31T15:04:16.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:16.640+0000 7f0227e00640 1 -- 172.21.15.16:0/1463206579 <== mon.2 v2:172.21.15.89:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f02200a58b0 con 0x7f02280cd0e0 2024-03-31T15:04:16.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:16.640+0000 7f0227e00640 1 -- 172.21.15.16:0/1463206579 <== mon.2 v2:172.21.15.89:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02200a5c80 con 0x7f02280cd0e0 2024-03-31T15:04:16.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:16.644+0000 7f0230600640 1 -- 172.21.15.16:0/1463206579 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f02280cd0e0 msgr2=0x7f022806da10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:16.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:16.644+0000 7f0230600640 1 --2- 172.21.15.16:0/1463206579 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f02280cd0e0 0x7f022806da10 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f0220009d90 tx=0x7f022001c8e0 comp rx=0 tx=0).stop 2024-03-31T15:04:16.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:16.644+0000 7f0230600640 1 -- 172.21.15.16:0/1463206579 shutdown_connections 2024-03-31T15:04:16.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:16.644+0000 7f0230600640 1 --2- 172.21.15.16:0/1463206579 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f02280cd0e0 0x7f022806da10 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:16.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:16.644+0000 7f0230600640 1 --2- 172.21.15.16:0/1463206579 >> v2:172.21.15.89:3300/0 conn(0x7f022813dff0 0x7f022806d4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:16.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:16.644+0000 7f0230600640 1 -- 172.21.15.16:0/1463206579 >> 172.21.15.16:0/1463206579 conn(0x7f022806ee40 msgr2=0x7f022806b620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:04:16.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:16.644+0000 7f0230600640 1 -- 172.21.15.16:0/1463206579 shutdown_connections 2024-03-31T15:04:16.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:16.644+0000 7f0230600640 1 -- 172.21.15.16:0/1463206579 wait complete. 2024-03-31T15:04:16.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:16.644+0000 7f0230600640 1 Processor -- start 2024-03-31T15:04:16.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:16.644+0000 7f0230600640 1 -- start start 2024-03-31T15:04:16.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:16.644+0000 7f0230600640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f02280cd0e0 0x7f022815e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:16.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:16.644+0000 7f0230600640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f022813dff0 0x7f0228152200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:16.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:16.644+0000 7f0230600640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f022813e3d0 0x7f022812d2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:16.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:16.644+0000 7f0230600640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f02280cf710 con 0x7f022813dff0 2024-03-31T15:04:16.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:16.644+0000 7f0230600640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f02280cf590 con 0x7f022813e3d0 2024-03-31T15:04:16.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:16.644+0000 7f0230600640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f02280cf890 con 0x7f02280cd0e0 2024-03-31T15:04:16.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:16.644+0000 7f022d200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f022813dff0 0x7f0228152200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:16.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:16.644+0000 7f022d200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f022813dff0 0x7f0228152200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60622/0 (socket says 172.21.15.16:60622) 2024-03-31T15:04:16.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:16.644+0000 7f022d200640 1 -- 172.21.15.16:0/63469752 learned_addr learned my addr 172.21.15.16:0/63469752 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:04:16.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:16.644+0000 7f022dc00640 1 --2- 172.21.15.16:0/63469752 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f02280cd0e0 0x7f022815e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:16.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:16.644+0000 7f022e600640 1 --2- 172.21.15.16:0/63469752 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f022813e3d0 0x7f022812d2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:16.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:16.644+0000 7f022d200640 1 -- 172.21.15.16:0/63469752 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f02280cd0e0 msgr2=0x7f022815e0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:16.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:16.644+0000 7f022d200640 1 --2- 172.21.15.16:0/63469752 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f02280cd0e0 0x7f022815e0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:16.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:16.644+0000 7f022d200640 1 -- 172.21.15.16:0/63469752 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f022813e3d0 msgr2=0x7f022812d2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:16.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:16.644+0000 7f022d200640 1 --2- 172.21.15.16:0/63469752 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f022813e3d0 0x7f022812d2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:16.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:16.644+0000 7f022d200640 1 -- 172.21.15.16:0/63469752 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f022815e620 con 0x7f022813dff0 2024-03-31T15:04:16.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:16.648+0000 7f022d200640 1 --2- 172.21.15.16:0/63469752 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f022813dff0 0x7f0228152200 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f021400c370 tx=0x7f021400c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:16.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:16.648+0000 7f0225600640 1 -- 172.21.15.16:0/63469752 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0214017070 con 0x7f022813dff0 2024-03-31T15:04:16.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:16.648+0000 7f0225600640 1 -- 172.21.15.16:0/63469752 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0214005cf0 con 0x7f022813dff0 2024-03-31T15:04:16.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:16.648+0000 7f0230600640 1 -- 172.21.15.16:0/63469752 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f02281553b0 con 0x7f022813dff0 2024-03-31T15:04:16.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:16.648+0000 7f0225600640 1 -- 172.21.15.16:0/63469752 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02140072f0 con 0x7f022813dff0 2024-03-31T15:04:16.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:16.648+0000 7f0230600640 1 -- 172.21.15.16:0/63469752 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f02281566e0 con 0x7f022813dff0 2024-03-31T15:04:16.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:16.648+0000 7f0230600640 1 -- 172.21.15.16:0/63469752 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0228156230 con 0x7f022813dff0 2024-03-31T15:04:16.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:16.648+0000 7f0225600640 1 -- 172.21.15.16:0/63469752 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f021401a070 con 0x7f022813dff0 2024-03-31T15:04:16.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:16.648+0000 7f0225600640 1 --2- 172.21.15.16:0/63469752 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f02080683b0 0x7f020806a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:16.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:16.648+0000 7f0225600640 1 -- 172.21.15.16:0/63469752 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f0214085cb0 con 0x7f022813dff0 2024-03-31T15:04:16.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:16.648+0000 7f022dc00640 1 --2- 172.21.15.16:0/63469752 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f02080683b0 0x7f020806a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:16.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:16.652+0000 7f022dc00640 1 --2- 172.21.15.16:0/63469752 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f02080683b0 0x7f020806a870 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f021c009580 tx=0x7f021c009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:16.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:16.652+0000 7f0225600640 1 -- 172.21.15.16:0/63469752 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f0214055b80 con 0x7f022813dff0 2024-03-31T15:04:16.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:16.816+0000 7f0230600640 1 -- 172.21.15.16:0/63469752 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f022806e4e0 con 0x7f022813dff0 2024-03-31T15:04:16.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:16.816+0000 7f0225600640 1 -- 172.21.15.16:0/63469752 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) ==== 93+0+10 (secure 0 0 0) 0x7f0214059e80 con 0x7f022813dff0 2024-03-31T15:04:16.812 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2024-03-31T15:04:16.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:16.820+0000 7f0230600640 1 -- 172.21.15.16:0/63469752 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f02080683b0 msgr2=0x7f020806a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:16.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:16.820+0000 7f0230600640 1 --2- 172.21.15.16:0/63469752 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f02080683b0 0x7f020806a870 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f021c009580 tx=0x7f021c009190 comp rx=0 tx=0).stop 2024-03-31T15:04:16.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:16.820+0000 7f0230600640 1 -- 172.21.15.16:0/63469752 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f022813dff0 msgr2=0x7f0228152200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:16.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:16.820+0000 7f0230600640 1 --2- 172.21.15.16:0/63469752 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f022813dff0 0x7f0228152200 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f021400c370 tx=0x7f021400c830 comp rx=0 tx=0).stop 2024-03-31T15:04:16.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:16.820+0000 7f0230600640 1 -- 172.21.15.16:0/63469752 shutdown_connections 2024-03-31T15:04:16.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:16.820+0000 7f0230600640 1 --2- 172.21.15.16:0/63469752 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f02080683b0 0x7f020806a870 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:16.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:16.820+0000 7f0230600640 1 --2- 172.21.15.16:0/63469752 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f022813e3d0 0x7f022812d2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:16.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:16.820+0000 7f0230600640 1 --2- 172.21.15.16:0/63469752 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f022813dff0 0x7f0228152200 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:16.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:16.820+0000 7f0230600640 1 --2- 172.21.15.16:0/63469752 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f02280cd0e0 0x7f022815e0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:16.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:16.820+0000 7f0230600640 1 -- 172.21.15.16:0/63469752 >> 172.21.15.16:0/63469752 conn(0x7f022806ee40 msgr2=0x7f022806b620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:04:16.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:16.820+0000 7f0230600640 1 -- 172.21.15.16:0/63469752 shutdown_connections 2024-03-31T15:04:16.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:16.820+0000 7f0230600640 1 -- 172.21.15.16:0/63469752 wait complete. 2024-03-31T15:04:16.832 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-31T15:04:16.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:16.984+0000 7f8fe5200640 1 Processor -- start 2024-03-31T15:04:16.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:16.984+0000 7f8fe5200640 1 -- start start 2024-03-31T15:04:16.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:16.984+0000 7f8fe5200640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f8fe015e5e0 0x7f8fe006c7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:16.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:16.984+0000 7f8fe5200640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f8fe00b2fe0 0x7f8fe006cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:16.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:16.984+0000 7f8fe5200640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8fe015e9c0 con 0x7f8fe006d510 2024-03-31T15:04:16.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:16.984+0000 7f8fe5200640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f8fe00b5760 con 0x7f8fe00b2fe0 2024-03-31T15:04:16.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:16.984+0000 7f8fe5200640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f8fe00b62e0 con 0x7f8fe015e5e0 2024-03-31T15:04:16.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:16.984+0000 7f8fdea00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f8fe00b2fe0 0x7f8fe006cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:16.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:16.984+0000 7f8fdf400640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f8fe015e5e0 0x7f8fe006c7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:16.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:16.984+0000 7f8fdfe00640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f8fe006d510 0x7f8fe00d0090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.16:60986/0 (socket says 172.21.15.16:60986) 2024-03-31T15:04:16.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:16.984+0000 7f8fdfe00640 1 -- 172.21.15.16:0/3320378779 learned_addr learned my addr 172.21.15.16:0/3320378779 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:04:16.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:16.984+0000 7f8fde000640 1 -- 172.21.15.16:0/3320378779 <== mon.1 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 379078981 0 0) 0x7f8fe015e9c0 con 0x7f8fe006d510 2024-03-31T15:04:16.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:16.984+0000 7f8fde000640 1 -- 172.21.15.16:0/3320378779 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8fc4003530 con 0x7f8fe006d510 2024-03-31T15:04:16.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:16.984+0000 7f8fdea00640 1 -- 172.21.15.16:0/3320378779 >> v2:172.21.15.89:3300/0 conn(0x7f8fe015e5e0 msgr2=0x7f8fe006c7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:04:16.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:16.984+0000 7f8fdea00640 1 --2- 172.21.15.16:0/3320378779 >> v2:172.21.15.89:3300/0 conn(0x7f8fe015e5e0 0x7f8fe006c7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:16.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:16.984+0000 7f8fdea00640 1 -- 172.21.15.16:0/3320378779 >> v1:172.21.15.89:6789/0 conn(0x7f8fe006d510 legacy=0x7f8fe00d0090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:16.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:16.984+0000 7f8fdea00640 1 -- 172.21.15.16:0/3320378779 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8fe00b59b0 con 0x7f8fe00b2fe0 2024-03-31T15:04:16.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:16.984+0000 7f8fdf400640 1 --2- 172.21.15.16:0/3320378779 >> v2:172.21.15.89:3300/0 conn(0x7f8fe015e5e0 0x7f8fe006c7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:04:16.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:16.988+0000 7f8fdea00640 1 --2- 172.21.15.16:0/3320378779 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8fe00b2fe0 0x7f8fe006cfd0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f8fe0072a90 tx=0x7f8fcc01c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=29906db86be5cec4 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:16.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:16.988+0000 7f8fde000640 1 -- 172.21.15.16:0/3320378779 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fcc0b0070 con 0x7f8fe00b2fe0 2024-03-31T15:04:16.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:16.988+0000 7f8fde000640 1 -- 172.21.15.16:0/3320378779 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8fcc0ab600 con 0x7f8fe00b2fe0 2024-03-31T15:04:16.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:16.988+0000 7f8fde000640 1 -- 172.21.15.16:0/3320378779 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fcc0ab9d0 con 0x7f8fe00b2fe0 2024-03-31T15:04:16.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:16.988+0000 7f8fe5200640 1 -- 172.21.15.16:0/3320378779 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8fe00b2fe0 msgr2=0x7f8fe006cfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:16.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:16.988+0000 7f8fe5200640 1 --2- 172.21.15.16:0/3320378779 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8fe00b2fe0 0x7f8fe006cfd0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f8fe0072a90 tx=0x7f8fcc01c480 comp rx=0 tx=0).stop 2024-03-31T15:04:16.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:16.988+0000 7f8fe5200640 1 -- 172.21.15.16:0/3320378779 shutdown_connections 2024-03-31T15:04:16.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:16.988+0000 7f8fe5200640 1 --2- 172.21.15.16:0/3320378779 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8fe00b2fe0 0x7f8fe006cfd0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:16.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:16.988+0000 7f8fe5200640 1 --2- 172.21.15.16:0/3320378779 >> v2:172.21.15.89:3300/0 conn(0x7f8fe015e5e0 0x7f8fe006c7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:16.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:16.988+0000 7f8fe5200640 1 -- 172.21.15.16:0/3320378779 >> 172.21.15.16:0/3320378779 conn(0x7f8fe00d1f00 msgr2=0x7f8fe00d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:04:16.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:16.988+0000 7f8fe5200640 1 -- 172.21.15.16:0/3320378779 shutdown_connections 2024-03-31T15:04:16.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:16.988+0000 7f8fe5200640 1 -- 172.21.15.16:0/3320378779 wait complete. 2024-03-31T15:04:16.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:16.992+0000 7f8fe5200640 1 Processor -- start 2024-03-31T15:04:16.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:16.992+0000 7f8fe5200640 1 -- start start 2024-03-31T15:04:16.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:16.992+0000 7f8fe5200640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8fe006d510 0x7f8fe01540b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:16.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:16.992+0000 7f8fe5200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8fe00b2fe0 0x7f8fe0152320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:16.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:16.992+0000 7f8fe5200640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f8fe015e5e0 0x7f8fe015d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:16.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:16.992+0000 7f8fe5200640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f8fe016b350 con 0x7f8fe00b2fe0 2024-03-31T15:04:16.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:16.992+0000 7f8fe5200640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f8fe006d920 con 0x7f8fe006d510 2024-03-31T15:04:16.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:16.992+0000 7f8fe5200640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f8fe016b4d0 con 0x7f8fe015e5e0 2024-03-31T15:04:16.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:16.992+0000 7f8fdea00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8fe00b2fe0 0x7f8fe0152320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:16.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:16.992+0000 7f8fdea00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8fe00b2fe0 0x7f8fe0152320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60644/0 (socket says 172.21.15.16:60644) 2024-03-31T15:04:16.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:16.992+0000 7f8fdea00640 1 -- 172.21.15.16:0/1726979482 learned_addr learned my addr 172.21.15.16:0/1726979482 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:04:16.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:16.992+0000 7f8fdf400640 1 --2- 172.21.15.16:0/1726979482 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8fe006d510 0x7f8fe01540b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:16.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:16.992+0000 7f8fdfe00640 1 --2- 172.21.15.16:0/1726979482 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f8fe015e5e0 0x7f8fe015d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:16.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:16.992+0000 7f8fdea00640 1 -- 172.21.15.16:0/1726979482 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f8fe015e5e0 msgr2=0x7f8fe015d180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:16.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:16.992+0000 7f8fdea00640 1 --2- 172.21.15.16:0/1726979482 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f8fe015e5e0 0x7f8fe015d180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:16.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:16.992+0000 7f8fdea00640 1 -- 172.21.15.16:0/1726979482 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8fe006d510 msgr2=0x7f8fe01540b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:16.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:16.992+0000 7f8fdea00640 1 --2- 172.21.15.16:0/1726979482 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8fe006d510 0x7f8fe01540b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:16.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:16.992+0000 7f8fdea00640 1 -- 172.21.15.16:0/1726979482 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8fe01535d0 con 0x7f8fe00b2fe0 2024-03-31T15:04:16.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:16.992+0000 7f8fdfe00640 1 --2- 172.21.15.16:0/1726979482 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f8fe015e5e0 0x7f8fe015d180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:04:16.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:16.992+0000 7f8fdea00640 1 --2- 172.21.15.16:0/1726979482 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8fe00b2fe0 0x7f8fe0152320 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f8fcc0ae040 tx=0x7f8fcc019220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:16.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:16.992+0000 7f8fd7400640 1 -- 172.21.15.16:0/1726979482 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fcc0b0040 con 0x7f8fe00b2fe0 2024-03-31T15:04:16.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:16.992+0000 7f8fd7400640 1 -- 172.21.15.16:0/1726979482 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8fcc0b9070 con 0x7f8fe00b2fe0 2024-03-31T15:04:16.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:16.992+0000 7f8fd7400640 1 -- 172.21.15.16:0/1726979482 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fcc0b4670 con 0x7f8fe00b2fe0 2024-03-31T15:04:16.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:16.992+0000 7f8fe5200640 1 -- 172.21.15.16:0/1726979482 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8fe0161230 con 0x7f8fe00b2fe0 2024-03-31T15:04:16.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:16.992+0000 7f8fe5200640 1 -- 172.21.15.16:0/1726979482 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8fe01617f0 con 0x7f8fe00b2fe0 2024-03-31T15:04:16.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:16.996+0000 7f8fe5200640 1 -- 172.21.15.16:0/1726979482 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8fe004f8c0 con 0x7f8fe00b2fe0 2024-03-31T15:04:16.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:16.996+0000 7f8fd7400640 1 -- 172.21.15.16:0/1726979482 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f8fcc0b4810 con 0x7f8fe00b2fe0 2024-03-31T15:04:16.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:16.996+0000 7f8fd7400640 1 --2- 172.21.15.16:0/1726979482 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f8fc0068380 0x7f8fc006a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:16.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:16.996+0000 7f8fd7400640 1 -- 172.21.15.16:0/1726979482 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f8fcc120780 con 0x7f8fe00b2fe0 2024-03-31T15:04:16.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:16.996+0000 7f8fdf400640 1 --2- 172.21.15.16:0/1726979482 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f8fc0068380 0x7f8fc006a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:16.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:16.996+0000 7f8fdf400640 1 --2- 172.21.15.16:0/1726979482 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f8fc0068380 0x7f8fc006a840 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f8fe006e5b0 tx=0x7f8fd0009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:16.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:17.000+0000 7f8fd7400640 1 -- 172.21.15.16:0/1726979482 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f8fcc016070 con 0x7f8fe00b2fe0 2024-03-31T15:04:17.164 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:17.168+0000 7f8fe5200640 1 -- 172.21.15.16:0/1726979482 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f8fe015aff0 con 0x7f8fe00b2fe0 2024-03-31T15:04:17.164 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:17.168+0000 7f8fd7400640 1 -- 172.21.15.16:0/1726979482 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v22) ==== 104+0+11 (secure 0 0 0) 0x7f8fcc0f0ca0 con 0x7f8fe00b2fe0 2024-03-31T15:04:17.164 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2024-03-31T15:04:17.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:17.172+0000 7f8fe5200640 1 -- 172.21.15.16:0/1726979482 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f8fc0068380 msgr2=0x7f8fc006a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:17.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:17.172+0000 7f8fe5200640 1 --2- 172.21.15.16:0/1726979482 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f8fc0068380 0x7f8fc006a840 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f8fe006e5b0 tx=0x7f8fd0009190 comp rx=0 tx=0).stop 2024-03-31T15:04:17.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:17.172+0000 7f8fe5200640 1 -- 172.21.15.16:0/1726979482 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8fe00b2fe0 msgr2=0x7f8fe0152320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:17.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:17.172+0000 7f8fe5200640 1 --2- 172.21.15.16:0/1726979482 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8fe00b2fe0 0x7f8fe0152320 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f8fcc0ae040 tx=0x7f8fcc019220 comp rx=0 tx=0).stop 2024-03-31T15:04:17.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:17.172+0000 7f8fe5200640 1 -- 172.21.15.16:0/1726979482 shutdown_connections 2024-03-31T15:04:17.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:17.172+0000 7f8fe5200640 1 --2- 172.21.15.16:0/1726979482 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f8fc0068380 0x7f8fc006a840 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:17.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:17.172+0000 7f8fe5200640 1 --2- 172.21.15.16:0/1726979482 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f8fe015e5e0 0x7f8fe015d180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:17.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:17.172+0000 7f8fe5200640 1 --2- 172.21.15.16:0/1726979482 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8fe00b2fe0 0x7f8fe0152320 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:17.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:17.172+0000 7f8fe5200640 1 --2- 172.21.15.16:0/1726979482 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8fe006d510 0x7f8fe01540b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:17.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:17.172+0000 7f8fe5200640 1 -- 172.21.15.16:0/1726979482 >> 172.21.15.16:0/1726979482 conn(0x7f8fe00d1f00 msgr2=0x7f8fe006fae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:04:17.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:17.172+0000 7f8fe5200640 1 -- 172.21.15.16:0/1726979482 shutdown_connections 2024-03-31T15:04:17.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:17.172+0000 7f8fe5200640 1 -- 172.21.15.16:0/1726979482 wait complete. 2024-03-31T15:04:17.185 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-31T15:04:17.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:17.336+0000 7f194b000640 1 Processor -- start 2024-03-31T15:04:17.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:17.336+0000 7f194b000640 1 -- start start 2024-03-31T15:04:17.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:17.336+0000 7f194b000640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f19441567e0 0x7f1944072c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:17.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:17.336+0000 7f194b000640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f19440731c0 0x7f194406d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:17.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:17.336+0000 7f194b000640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f194415c7e0 con 0x7f194406da10 2024-03-31T15:04:17.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:17.336+0000 7f194b000640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f194406de80 con 0x7f19441567e0 2024-03-31T15:04:17.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:17.336+0000 7f194b000640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f19440736b0 con 0x7f19440731c0 2024-03-31T15:04:17.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:17.336+0000 7f1943e00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f19441567e0 0x7f1944072c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:17.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:17.336+0000 7f1943400640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f19440731c0 0x7f194406d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:17.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:17.336+0000 7f1943e00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f19441567e0 0x7f1944072c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49088/0 (socket says 172.21.15.16:49088) 2024-03-31T15:04:17.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:17.336+0000 7f1943e00640 1 -- 172.21.15.16:0/4141359774 learned_addr learned my addr 172.21.15.16:0/4141359774 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:04:17.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:17.336+0000 7f1943e00640 1 -- 172.21.15.16:0/4141359774 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f19440731c0 msgr2=0x7f194406d4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:04:17.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:17.336+0000 7f1943e00640 1 --2- 172.21.15.16:0/4141359774 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f19440731c0 0x7f194406d4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:17.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:17.336+0000 7f1943e00640 1 -- 172.21.15.16:0/4141359774 >> v1:172.21.15.89:6789/0 conn(0x7f194406da10 legacy=0x7f194415fdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:17.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:17.336+0000 7f1943e00640 1 -- 172.21.15.16:0/4141359774 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f194415caa0 con 0x7f19441567e0 2024-03-31T15:04:17.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:17.340+0000 7f1943e00640 1 --2- 172.21.15.16:0/4141359774 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f19441567e0 0x7f1944072c80 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f1934009870 tx=0x7f193401c560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f077cc05d00bd488 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:17.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:17.340+0000 7f1942a00640 1 -- 172.21.15.16:0/4141359774 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19340b0070 con 0x7f19441567e0 2024-03-31T15:04:17.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:17.340+0000 7f1942a00640 1 -- 172.21.15.16:0/4141359774 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f19340ab600 con 0x7f19441567e0 2024-03-31T15:04:17.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:17.340+0000 7f1942a00640 1 -- 172.21.15.16:0/4141359774 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19340ab9d0 con 0x7f19441567e0 2024-03-31T15:04:17.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:17.340+0000 7f194b000640 1 -- 172.21.15.16:0/4141359774 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f19441567e0 msgr2=0x7f1944072c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:17.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:17.340+0000 7f194b000640 1 --2- 172.21.15.16:0/4141359774 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f19441567e0 0x7f1944072c80 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f1934009870 tx=0x7f193401c560 comp rx=0 tx=0).stop 2024-03-31T15:04:17.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:17.340+0000 7f194b000640 1 -- 172.21.15.16:0/4141359774 shutdown_connections 2024-03-31T15:04:17.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:17.340+0000 7f194b000640 1 --2- 172.21.15.16:0/4141359774 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f19440731c0 0x7f194406d4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:17.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:17.340+0000 7f194b000640 1 --2- 172.21.15.16:0/4141359774 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f19441567e0 0x7f1944072c80 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:17.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:17.340+0000 7f194b000640 1 -- 172.21.15.16:0/4141359774 >> 172.21.15.16:0/4141359774 conn(0x7f194406ee40 msgr2=0x7f194406b620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:04:17.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:17.340+0000 7f194b000640 1 -- 172.21.15.16:0/4141359774 shutdown_connections 2024-03-31T15:04:17.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:17.340+0000 7f194b000640 1 -- 172.21.15.16:0/4141359774 wait complete. 2024-03-31T15:04:17.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:17.344+0000 7f194b000640 1 Processor -- start 2024-03-31T15:04:17.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:17.344+0000 7f194b000640 1 -- start start 2024-03-31T15:04:17.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:17.344+0000 7f194b000640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f194406da10 0x7f19440cd430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:17.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:17.344+0000 7f194b000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f19440731c0 0x7f1944159670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:17.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:17.344+0000 7f194b000640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f19441567e0 0x7f19441f7430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:17.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:17.344+0000 7f194b000640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f194415d700 con 0x7f19440731c0 2024-03-31T15:04:17.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:17.344+0000 7f194b000640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f194415d580 con 0x7f19441567e0 2024-03-31T15:04:17.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:17.344+0000 7f194b000640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f1944169170 con 0x7f194406da10 2024-03-31T15:04:17.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:17.344+0000 7f1943400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f19440731c0 0x7f1944159670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:17.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:17.344+0000 7f1943400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f19440731c0 0x7f1944159670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49102/0 (socket says 172.21.15.16:49102) 2024-03-31T15:04:17.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:17.344+0000 7f1943400640 1 -- 172.21.15.16:0/2821745008 learned_addr learned my addr 172.21.15.16:0/2821745008 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:04:17.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:17.344+0000 7f1943e00640 1 --2- 172.21.15.16:0/2821745008 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f194406da10 0x7f19440cd430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:17.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:17.344+0000 7f1949000640 1 --2- 172.21.15.16:0/2821745008 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f19441567e0 0x7f19441f7430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:17.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:17.344+0000 7f1943400640 1 -- 172.21.15.16:0/2821745008 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f194406da10 msgr2=0x7f19440cd430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:17.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:17.344+0000 7f1943400640 1 --2- 172.21.15.16:0/2821745008 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f194406da10 0x7f19440cd430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:17.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:17.344+0000 7f1943400640 1 -- 172.21.15.16:0/2821745008 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f19441567e0 msgr2=0x7f19441f7430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:17.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:17.344+0000 7f1943400640 1 --2- 172.21.15.16:0/2821745008 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f19441567e0 0x7f19441f7430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:17.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:17.344+0000 7f1943400640 1 -- 172.21.15.16:0/2821745008 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1944159d50 con 0x7f19440731c0 2024-03-31T15:04:17.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:17.344+0000 7f1943400640 1 --2- 172.21.15.16:0/2821745008 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f19440731c0 0x7f1944159670 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f193000e870 tx=0x7f193000ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:17.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:17.344+0000 7f193be00640 1 -- 172.21.15.16:0/2821745008 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1930017070 con 0x7f19440731c0 2024-03-31T15:04:17.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:17.344+0000 7f193be00640 1 -- 172.21.15.16:0/2821745008 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f193000c940 con 0x7f19440731c0 2024-03-31T15:04:17.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:17.344+0000 7f193be00640 1 -- 172.21.15.16:0/2821745008 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f193000cd30 con 0x7f19440731c0 2024-03-31T15:04:17.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:17.344+0000 7f194b000640 1 -- 172.21.15.16:0/2821745008 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1944157980 con 0x7f19440731c0 2024-03-31T15:04:17.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:17.344+0000 7f194b000640 1 -- 172.21.15.16:0/2821745008 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1944157f40 con 0x7f19440731c0 2024-03-31T15:04:17.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:17.348+0000 7f194b000640 1 -- 172.21.15.16:0/2821745008 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1944158140 con 0x7f19440731c0 2024-03-31T15:04:17.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:17.348+0000 7f193be00640 1 -- 172.21.15.16:0/2821745008 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f193001a070 con 0x7f19440731c0 2024-03-31T15:04:17.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:17.348+0000 7f193be00640 1 --2- 172.21.15.16:0/2821745008 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f19240683b0 0x7f192406a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:17.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:17.348+0000 7f193be00640 1 -- 172.21.15.16:0/2821745008 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5825+0+0 (secure 0 0 0) 0x7f1930086810 con 0x7f19440731c0 2024-03-31T15:04:17.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:17.348+0000 7f1943e00640 1 --2- 172.21.15.16:0/2821745008 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f19240683b0 0x7f192406a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:17.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:17.348+0000 7f1943e00640 1 --2- 172.21.15.16:0/2821745008 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f19240683b0 0x7f192406a870 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f19340ad000 tx=0x7f19340a6000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:17.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:17.352+0000 7f193be00640 1 -- 172.21.15.16:0/2821745008 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f1944158140 con 0x7f19440731c0 2024-03-31T15:04:17.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:17.516+0000 7f194b000640 1 -- 172.21.15.16:0/2821745008 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f1944069950 con 0x7f19440731c0 2024-03-31T15:04:17.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:17.516+0000 7f193be00640 1 -- 172.21.15.16:0/2821745008 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v22) ==== 100+0+11 (secure 0 0 0) 0x7f1930056660 con 0x7f19440731c0 2024-03-31T15:04:17.510 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 64 2024-03-31T15:04:17.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:17.516+0000 7f194b000640 1 -- 172.21.15.16:0/2821745008 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f19240683b0 msgr2=0x7f192406a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:17.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:17.516+0000 7f194b000640 1 --2- 172.21.15.16:0/2821745008 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f19240683b0 0x7f192406a870 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f19340ad000 tx=0x7f19340a6000 comp rx=0 tx=0).stop 2024-03-31T15:04:17.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:17.516+0000 7f194b000640 1 -- 172.21.15.16:0/2821745008 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f19440731c0 msgr2=0x7f1944159670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:17.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:17.516+0000 7f194b000640 1 --2- 172.21.15.16:0/2821745008 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f19440731c0 0x7f1944159670 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f193000e870 tx=0x7f193000ec70 comp rx=0 tx=0).stop 2024-03-31T15:04:17.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:17.516+0000 7f194b000640 1 -- 172.21.15.16:0/2821745008 shutdown_connections 2024-03-31T15:04:17.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:17.516+0000 7f194b000640 1 --2- 172.21.15.16:0/2821745008 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f19240683b0 0x7f192406a870 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:17.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:17.516+0000 7f194b000640 1 --2- 172.21.15.16:0/2821745008 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f19441567e0 0x7f19441f7430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:17.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:17.516+0000 7f194b000640 1 --2- 172.21.15.16:0/2821745008 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f19440731c0 0x7f1944159670 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:17.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:17.516+0000 7f194b000640 1 --2- 172.21.15.16:0/2821745008 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f194406da10 0x7f19440cd430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:17.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:17.516+0000 7f194b000640 1 -- 172.21.15.16:0/2821745008 >> 172.21.15.16:0/2821745008 conn(0x7f194406ee40 msgr2=0x7f19440b1e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:04:17.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:17.516+0000 7f194b000640 1 -- 172.21.15.16:0/2821745008 shutdown_connections 2024-03-31T15:04:17.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:17.516+0000 7f194b000640 1 -- 172.21.15.16:0/2821745008 wait complete. 2024-03-31T15:04:17.532 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-31T15:04:17.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:17.680+0000 7fec6fe00640 1 Processor -- start 2024-03-31T15:04:17.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:17.680+0000 7fec6fe00640 1 -- start start 2024-03-31T15:04:17.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:17.680+0000 7fec6fe00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fec6815f650 0x7fec6806f920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:17.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:17.680+0000 7fec6fe00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fec6806fe60 0x7fec68162180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:17.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:17.680+0000 7fec6fe00640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fec680cbe90 con 0x7fec6806dbc0 2024-03-31T15:04:17.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:17.680+0000 7fec6fe00640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7fec6815f1c0 con 0x7fec6815f650 2024-03-31T15:04:17.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:17.680+0000 7fec6fe00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fec680cc170 con 0x7fec6806fe60 2024-03-31T15:04:17.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:17.680+0000 7fec6d400640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fec6806dbc0 0x7fec680cb440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:44316/0 (socket says 172.21.15.16:44316) 2024-03-31T15:04:17.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:17.680+0000 7fec6d400640 1 -- 172.21.15.16:0/2648253098 learned_addr learned my addr 172.21.15.16:0/2648253098 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:04:17.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:17.680+0000 7fec6de00640 1 --2- 172.21.15.16:0/2648253098 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fec6806fe60 0x7fec68162180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:17.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:17.680+0000 7fec6ca00640 1 --2- 172.21.15.16:0/2648253098 >> v2:172.21.15.16:3300/0 conn(0x7fec6815f650 0x7fec6806f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:17.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:17.680+0000 7fec67e00640 1 -- 172.21.15.16:0/2648253098 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3738414307 0 0) 0x7fec680cbe90 con 0x7fec6806dbc0 2024-03-31T15:04:17.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:17.684+0000 7fec67e00640 1 -- 172.21.15.16:0/2648253098 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fec50003530 con 0x7fec6806dbc0 2024-03-31T15:04:17.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:17.684+0000 7fec6de00640 1 -- 172.21.15.16:0/2648253098 >> v2:172.21.15.16:3300/0 conn(0x7fec6815f650 msgr2=0x7fec6806f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:04:17.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:17.684+0000 7fec6de00640 1 --2- 172.21.15.16:0/2648253098 >> v2:172.21.15.16:3300/0 conn(0x7fec6815f650 0x7fec6806f920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:17.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:17.684+0000 7fec6de00640 1 -- 172.21.15.16:0/2648253098 >> v1:172.21.15.16:6789/0 conn(0x7fec6806dbc0 legacy=0x7fec680cb440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:17.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:17.684+0000 7fec6de00640 1 -- 172.21.15.16:0/2648253098 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec6815ee50 con 0x7fec6806fe60 2024-03-31T15:04:17.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:17.684+0000 7fec6de00640 1 --2- 172.21.15.16:0/2648253098 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fec6806fe60 0x7fec68162180 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fec60009d90 tx=0x7fec6001c8e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8ed073dbf38d6fa1 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:17.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:17.684+0000 7fec67e00640 1 -- 172.21.15.16:0/2648253098 <== mon.2 v2:172.21.15.89:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec600b2070 con 0x7fec6806fe60 2024-03-31T15:04:17.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:17.684+0000 7fec67e00640 1 -- 172.21.15.16:0/2648253098 <== mon.2 v2:172.21.15.89:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fec600a5850 con 0x7fec6806fe60 2024-03-31T15:04:17.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:17.684+0000 7fec67e00640 1 -- 172.21.15.16:0/2648253098 <== mon.2 v2:172.21.15.89:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec600a5c20 con 0x7fec6806fe60 2024-03-31T15:04:17.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:17.684+0000 7fec6fe00640 1 -- 172.21.15.16:0/2648253098 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fec6806fe60 msgr2=0x7fec68162180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:17.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:17.684+0000 7fec6fe00640 1 --2- 172.21.15.16:0/2648253098 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fec6806fe60 0x7fec68162180 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fec60009d90 tx=0x7fec6001c8e0 comp rx=0 tx=0).stop 2024-03-31T15:04:17.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:17.684+0000 7fec6fe00640 1 -- 172.21.15.16:0/2648253098 shutdown_connections 2024-03-31T15:04:17.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:17.684+0000 7fec6fe00640 1 --2- 172.21.15.16:0/2648253098 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fec6806fe60 0x7fec68162180 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:17.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:17.684+0000 7fec6fe00640 1 --2- 172.21.15.16:0/2648253098 >> v2:172.21.15.16:3300/0 conn(0x7fec6815f650 0x7fec6806f920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:17.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:17.688+0000 7fec6fe00640 1 -- 172.21.15.16:0/2648253098 >> 172.21.15.16:0/2648253098 conn(0x7fec68067330 msgr2=0x7fec68067740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:04:17.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:17.688+0000 7fec6fe00640 1 -- 172.21.15.16:0/2648253098 shutdown_connections 2024-03-31T15:04:17.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:17.688+0000 7fec6fe00640 1 -- 172.21.15.16:0/2648253098 wait complete. 2024-03-31T15:04:17.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:17.688+0000 7fec6fe00640 1 Processor -- start 2024-03-31T15:04:17.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:17.688+0000 7fec6fe00640 1 -- start start 2024-03-31T15:04:17.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:17.688+0000 7fec6fe00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fec6806dbc0 0x7fec68143f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:17.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:17.688+0000 7fec6fe00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fec6806fe60 0x7fec6815aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:17.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:17.688+0000 7fec6fe00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fec6815f650 0x7fec68144910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:17.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:17.688+0000 7fec6fe00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fec6806c950 con 0x7fec6806dbc0 2024-03-31T15:04:17.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:17.688+0000 7fec6fe00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fec6806dfa0 con 0x7fec6815f650 2024-03-31T15:04:17.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:17.688+0000 7fec6fe00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fec6806cad0 con 0x7fec6806fe60 2024-03-31T15:04:17.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:17.688+0000 7fec6d400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fec6806dbc0 0x7fec68143f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:17.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:17.688+0000 7fec6d400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fec6806dbc0 0x7fec68143f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49114/0 (socket says 172.21.15.16:49114) 2024-03-31T15:04:17.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:17.688+0000 7fec6ca00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fec6806fe60 0x7fec6815aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:17.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:17.688+0000 7fec6d400640 1 -- 172.21.15.16:0/646437770 learned_addr learned my addr 172.21.15.16:0/646437770 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:04:17.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:17.688+0000 7fec6de00640 1 --2- 172.21.15.16:0/646437770 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fec6815f650 0x7fec68144910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:17.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:17.688+0000 7fec6d400640 1 -- 172.21.15.16:0/646437770 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fec6806fe60 msgr2=0x7fec6815aaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:17.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:17.688+0000 7fec6d400640 1 --2- 172.21.15.16:0/646437770 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fec6806fe60 0x7fec6815aaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:17.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:17.688+0000 7fec6d400640 1 -- 172.21.15.16:0/646437770 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fec6815f650 msgr2=0x7fec68144910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:17.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:17.688+0000 7fec6d400640 1 --2- 172.21.15.16:0/646437770 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fec6815f650 0x7fec68144910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:17.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:17.688+0000 7fec6d400640 1 -- 172.21.15.16:0/646437770 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec68140b20 con 0x7fec6806dbc0 2024-03-31T15:04:17.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:17.688+0000 7fec6d400640 1 --2- 172.21.15.16:0/646437770 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fec6806dbc0 0x7fec68143f80 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fec5c00c3c0 tx=0x7fec5c00c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:17.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:17.688+0000 7fec65600640 1 -- 172.21.15.16:0/646437770 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec5c017070 con 0x7fec6806dbc0 2024-03-31T15:04:17.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:17.688+0000 7fec65600640 1 -- 172.21.15.16:0/646437770 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fec5c005cf0 con 0x7fec6806dbc0 2024-03-31T15:04:17.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:17.688+0000 7fec65600640 1 -- 172.21.15.16:0/646437770 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec5c0072f0 con 0x7fec6806dbc0 2024-03-31T15:04:17.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:17.688+0000 7fec6fe00640 1 -- 172.21.15.16:0/646437770 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fec68140e00 con 0x7fec6806dbc0 2024-03-31T15:04:17.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:17.692+0000 7fec6fe00640 1 -- 172.21.15.16:0/646437770 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fec68144560 con 0x7fec6806dbc0 2024-03-31T15:04:17.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:17.692+0000 7fec6fe00640 1 -- 172.21.15.16:0/646437770 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fec68069950 con 0x7fec6806dbc0 2024-03-31T15:04:17.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:17.692+0000 7fec65600640 1 -- 172.21.15.16:0/646437770 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fec5c01a070 con 0x7fec6806dbc0 2024-03-31T15:04:17.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:17.692+0000 7fec65600640 1 --2- 172.21.15.16:0/646437770 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fec48068380 0x7fec4806a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:17.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:17.692+0000 7fec65600640 1 -- 172.21.15.16:0/646437770 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5825+0+0 (secure 0 0 0) 0x7fec5c085de0 con 0x7fec6806dbc0 2024-03-31T15:04:17.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:17.692+0000 7fec6ca00640 1 --2- 172.21.15.16:0/646437770 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fec48068380 0x7fec4806a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:17.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:17.692+0000 7fec6ca00640 1 --2- 172.21.15.16:0/646437770 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fec48068380 0x7fec4806a840 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fec680d1bf0 tx=0x7fec54009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:17.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:17.696+0000 7fec65600640 1 -- 172.21.15.16:0/646437770 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fec5c055c30 con 0x7fec6806dbc0 2024-03-31T15:04:17.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:17.872+0000 7fec6fe00640 1 -- 172.21.15.16:0/646437770 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fec680cf380 con 0x7fec6806dbc0 2024-03-31T15:04:17.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:17.872+0000 7fec65600640 1 -- 172.21.15.16:0/646437770 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v22) ==== 80+0+0 (secure 0 0 0) 0x7fec5c059f30 con 0x7fec6806dbc0 2024-03-31T15:04:17.868 INFO:teuthology.orchestra.run.smithi016.stderr:listed 0 entries 2024-03-31T15:04:17.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:17.876+0000 7fec6fe00640 1 -- 172.21.15.16:0/646437770 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fec48068380 msgr2=0x7fec4806a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:17.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:17.876+0000 7fec6fe00640 1 --2- 172.21.15.16:0/646437770 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fec48068380 0x7fec4806a840 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fec680d1bf0 tx=0x7fec54009190 comp rx=0 tx=0).stop 2024-03-31T15:04:17.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:17.876+0000 7fec6fe00640 1 -- 172.21.15.16:0/646437770 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fec6806dbc0 msgr2=0x7fec68143f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:17.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:17.876+0000 7fec6fe00640 1 --2- 172.21.15.16:0/646437770 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fec6806dbc0 0x7fec68143f80 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fec5c00c3c0 tx=0x7fec5c00c880 comp rx=0 tx=0).stop 2024-03-31T15:04:17.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:17.876+0000 7fec6fe00640 1 -- 172.21.15.16:0/646437770 shutdown_connections 2024-03-31T15:04:17.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:17.876+0000 7fec6fe00640 1 --2- 172.21.15.16:0/646437770 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fec48068380 0x7fec4806a840 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:17.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:17.876+0000 7fec6fe00640 1 --2- 172.21.15.16:0/646437770 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fec6815f650 0x7fec68144910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:17.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:17.876+0000 7fec6fe00640 1 --2- 172.21.15.16:0/646437770 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fec6806fe60 0x7fec6815aaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:17.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:17.876+0000 7fec6fe00640 1 --2- 172.21.15.16:0/646437770 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fec6806dbc0 0x7fec68143f80 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:17.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:17.876+0000 7fec6fe00640 1 -- 172.21.15.16:0/646437770 >> 172.21.15.16:0/646437770 conn(0x7fec68067330 msgr2=0x7fec680cc6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:04:17.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:17.876+0000 7fec6fe00640 1 -- 172.21.15.16:0/646437770 shutdown_connections 2024-03-31T15:04:17.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:17.876+0000 7fec6fe00640 1 -- 172.21.15.16:0/646437770 wait complete. 2024-03-31T15:04:17.890 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi089.front.sepia.ceph.com... 2024-03-31T15:04:17.890 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:04:17.890 DEBUG:teuthology.orchestra.run.smithi089:> sudo fusermount -u /home/ubuntu/cephtest/mnt.1 2024-03-31T15:04:17.918 INFO:teuthology.orchestra.run:waiting for 300 2024-03-31T15:04:17.918 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi089.front.sepia.ceph.com 2024-03-31T15:04:17.918 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:04:17.918 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-03-31T15:04:17.989 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-03-31T15:04:17.989 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-31T15:04:17.998 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-03-31T15:04:17.998 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-31T15:04:18.048 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-31T15:04:18.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:18.236+0000 7f3891c00640 1 Processor -- start 2024-03-31T15:04:18.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:18.240+0000 7f3891c00640 1 -- start start 2024-03-31T15:04:18.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:18.240+0000 7f3891c00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f388c15b3d0 0x7f388c0b1cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:18.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:18.240+0000 7f3891c00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f388c0ce2f0 0x7f388c0b2200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:18.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:18.240+0000 7f3891c00640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f388c15f3c0 con 0x7f388c15eee0 2024-03-31T15:04:18.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:18.240+0000 7f3891c00640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f388c15b7b0 con 0x7f388c0ce2f0 2024-03-31T15:04:18.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:18.240+0000 7f3891c00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f388c15f6a0 con 0x7f388c15b3d0 2024-03-31T15:04:18.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:18.240+0000 7f388a000640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f388c15eee0 0x7f388c0cc9c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:44324/0 (socket says 172.21.15.16:44324) 2024-03-31T15:04:18.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:18.240+0000 7f388a000640 1 -- 172.21.15.16:0/3942962992 learned_addr learned my addr 172.21.15.16:0/3942962992 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:04:18.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:18.240+0000 7f388b400640 1 --2- 172.21.15.16:0/3942962992 >> v2:172.21.15.89:3300/0 conn(0x7f388c0ce2f0 0x7f388c0b2200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:18.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:18.240+0000 7f388aa00640 1 --2- 172.21.15.16:0/3942962992 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f388c15b3d0 0x7f388c0b1cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:18.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:18.240+0000 7f3889600640 1 -- 172.21.15.16:0/3942962992 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1171129293 0 0) 0x7f388c15f3c0 con 0x7f388c15eee0 2024-03-31T15:04:18.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:18.240+0000 7f3889600640 1 -- 172.21.15.16:0/3942962992 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3870003530 con 0x7f388c15eee0 2024-03-31T15:04:18.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:18.240+0000 7f388aa00640 1 -- 172.21.15.16:0/3942962992 >> v2:172.21.15.89:3300/0 conn(0x7f388c0ce2f0 msgr2=0x7f388c0b2200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:04:18.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:18.240+0000 7f388aa00640 1 --2- 172.21.15.16:0/3942962992 >> v2:172.21.15.89:3300/0 conn(0x7f388c0ce2f0 0x7f388c0b2200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:18.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:18.240+0000 7f388aa00640 1 -- 172.21.15.16:0/3942962992 >> v1:172.21.15.16:6789/0 conn(0x7f388c15eee0 legacy=0x7f388c0cc9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:18.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:18.240+0000 7f388aa00640 1 -- 172.21.15.16:0/3942962992 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f388c15f980 con 0x7f388c15b3d0 2024-03-31T15:04:18.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:18.240+0000 7f388b400640 1 --2- 172.21.15.16:0/3942962992 >> v2:172.21.15.89:3300/0 conn(0x7f388c0ce2f0 0x7f388c0b2200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:04:18.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:18.244+0000 7f388aa00640 1 --2- 172.21.15.16:0/3942962992 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f388c15b3d0 0x7f388c0b1cc0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f387c0097c0 tx=0x7f387c01c470 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dcde38f70f669117 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:18.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:18.244+0000 7f3889600640 1 -- 172.21.15.16:0/3942962992 <== mon.2 v2:172.21.15.89:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f387c0ab430 con 0x7f388c15b3d0 2024-03-31T15:04:18.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:18.244+0000 7f3889600640 1 -- 172.21.15.16:0/3942962992 <== mon.2 v2:172.21.15.89:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f387c0b0070 con 0x7f388c15b3d0 2024-03-31T15:04:18.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:18.244+0000 7f3889600640 1 -- 172.21.15.16:0/3942962992 <== mon.2 v2:172.21.15.89:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f387c0ab7f0 con 0x7f388c15b3d0 2024-03-31T15:04:18.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:18.244+0000 7f3891c00640 1 -- 172.21.15.16:0/3942962992 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f388c15b3d0 msgr2=0x7f388c0b1cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:18.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:18.244+0000 7f3891c00640 1 --2- 172.21.15.16:0/3942962992 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f388c15b3d0 0x7f388c0b1cc0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f387c0097c0 tx=0x7f387c01c470 comp rx=0 tx=0).stop 2024-03-31T15:04:18.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:18.244+0000 7f3891c00640 1 -- 172.21.15.16:0/3942962992 shutdown_connections 2024-03-31T15:04:18.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:18.244+0000 7f3891c00640 1 --2- 172.21.15.16:0/3942962992 >> v2:172.21.15.89:3300/0 conn(0x7f388c0ce2f0 0x7f388c0b2200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:18.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:18.244+0000 7f3891c00640 1 --2- 172.21.15.16:0/3942962992 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f388c15b3d0 0x7f388c0b1cc0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:18.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:18.244+0000 7f3891c00640 1 -- 172.21.15.16:0/3942962992 >> 172.21.15.16:0/3942962992 conn(0x7f388c06b620 msgr2=0x7f388c06ba30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:04:18.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:18.244+0000 7f3891c00640 1 -- 172.21.15.16:0/3942962992 shutdown_connections 2024-03-31T15:04:18.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:18.244+0000 7f3891c00640 1 -- 172.21.15.16:0/3942962992 wait complete. 2024-03-31T15:04:18.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:18.248+0000 7f3891c00640 1 Processor -- start 2024-03-31T15:04:18.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:18.248+0000 7f3891c00640 1 -- start start 2024-03-31T15:04:18.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:18.248+0000 7f3891c00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f388c0ce2f0 0x7f388c15df10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:18.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:18.248+0000 7f3891c00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f388c15b3d0 0x7f388c15e450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:18.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:18.248+0000 7f3891c00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f388c15eee0 0x7f388c0ae9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:18.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:18.248+0000 7f3891c00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f388c166640 con 0x7f388c15b3d0 2024-03-31T15:04:18.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:18.248+0000 7f3891c00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f388c1664c0 con 0x7f388c0ce2f0 2024-03-31T15:04:18.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:18.248+0000 7f3891c00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f388c1667c0 con 0x7f388c15eee0 2024-03-31T15:04:18.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:18.248+0000 7f388a000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f388c15b3d0 0x7f388c15e450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:18.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:18.248+0000 7f388a000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f388c15b3d0 0x7f388c15e450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49124/0 (socket says 172.21.15.16:49124) 2024-03-31T15:04:18.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:18.248+0000 7f388a000640 1 -- 172.21.15.16:0/1517060318 learned_addr learned my addr 172.21.15.16:0/1517060318 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:04:18.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:18.248+0000 7f388aa00640 1 --2- 172.21.15.16:0/1517060318 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f388c0ce2f0 0x7f388c15df10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:18.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:18.248+0000 7f388b400640 1 --2- 172.21.15.16:0/1517060318 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f388c15eee0 0x7f388c0ae9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:18.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:18.248+0000 7f388a000640 1 -- 172.21.15.16:0/1517060318 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f388c15eee0 msgr2=0x7f388c0ae9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:18.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:18.248+0000 7f388a000640 1 --2- 172.21.15.16:0/1517060318 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f388c15eee0 0x7f388c0ae9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:18.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:18.248+0000 7f388a000640 1 -- 172.21.15.16:0/1517060318 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f388c0ce2f0 msgr2=0x7f388c15df10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:18.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:18.248+0000 7f388a000640 1 --2- 172.21.15.16:0/1517060318 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f388c0ce2f0 0x7f388c15df10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:18.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:18.248+0000 7f388a000640 1 -- 172.21.15.16:0/1517060318 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f388c1424c0 con 0x7f388c15b3d0 2024-03-31T15:04:18.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:18.248+0000 7f388aa00640 1 --2- 172.21.15.16:0/1517060318 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f388c0ce2f0 0x7f388c15df10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:04:18.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:18.248+0000 7f388a000640 1 --2- 172.21.15.16:0/1517060318 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f388c15b3d0 0x7f388c15e450 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f387800c3f0 tx=0x7f387800c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:18.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:18.248+0000 7f3882a00640 1 -- 172.21.15.16:0/1517060318 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3878017070 con 0x7f388c15b3d0 2024-03-31T15:04:18.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:18.248+0000 7f3882a00640 1 -- 172.21.15.16:0/1517060318 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3878005cf0 con 0x7f388c15b3d0 2024-03-31T15:04:18.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:18.248+0000 7f3891c00640 1 -- 172.21.15.16:0/1517060318 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f388c1427a0 con 0x7f388c15b3d0 2024-03-31T15:04:18.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:18.248+0000 7f3882a00640 1 -- 172.21.15.16:0/1517060318 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38780072f0 con 0x7f388c15b3d0 2024-03-31T15:04:18.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:18.248+0000 7f3891c00640 1 -- 172.21.15.16:0/1517060318 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f388c142d60 con 0x7f388c15b3d0 2024-03-31T15:04:18.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:18.252+0000 7f3891c00640 1 -- 172.21.15.16:0/1517060318 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f388c069950 con 0x7f388c15b3d0 2024-03-31T15:04:18.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:18.252+0000 7f3882a00640 1 -- 172.21.15.16:0/1517060318 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f387801a070 con 0x7f388c15b3d0 2024-03-31T15:04:18.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:18.252+0000 7f3882a00640 1 --2- 172.21.15.16:0/1517060318 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f386c068380 0x7f386c06a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:18.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:18.252+0000 7f3882a00640 1 -- 172.21.15.16:0/1517060318 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f3878085d20 con 0x7f388c15b3d0 2024-03-31T15:04:18.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:18.252+0000 7f388aa00640 1 --2- 172.21.15.16:0/1517060318 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f386c068380 0x7f386c06a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:18.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:18.252+0000 7f388aa00640 1 --2- 172.21.15.16:0/1517060318 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f386c068380 0x7f386c06a840 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f387c0af000 tx=0x7f387c008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:18.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:18.256+0000 7f3882a00640 1 -- 172.21.15.16:0/1517060318 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f3878055c10 con 0x7f388c15b3d0 2024-03-31T15:04:18.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:18.424+0000 7f3891c00640 1 -- 172.21.15.16:0/1517060318 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f388c0cdea0 con 0x7f388c15b3d0 2024-03-31T15:04:18.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:18.424+0000 7f3882a00640 1 -- 172.21.15.16:0/1517060318 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v23) ==== 74+0+16780 (secure 0 0 0) 0x7f3878059f10 con 0x7f388c15b3d0 2024-03-31T15:04:18.418 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:04:18.419 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":23,"fsid":"286eb5da-fb47-4550-b290-d9519c13b9a5","created":"2024-03-31T15:03:11.438952+0000","modified":"2024-03-31T15:04:18.087947+0000","last_up_change":"2024-03-31T15:03:22.885535+0000","last_in_change":"2024-03-31T15:03:13.941667+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-31T15:03:22.550787+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-31T15:03:34.488425+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-31T15:03:35.299116+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"92a66450-ef19-40d8-ac3d-d760b23b0bda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6801","nonce":932712244}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6803","nonce":932712244}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6807","nonce":932712244}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6805","nonce":932712244}]},"public_addr":"172.21.15.16:6801/932712244","cluster_addr":"172.21.15.16:6803/932712244","heartbeat_back_addr":"172.21.15.16:6807/932712244","heartbeat_front_addr":"172.21.15.16:6805/932712244","state":["exists","up"]},{"osd":1,"uuid":"da9d1f2b-1968-4959-b843-faa231988e92","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6810","nonce":1398737533}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6811","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6813","nonce":1398737533}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6821","nonce":1398737533}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6815","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6816","nonce":1398737533}]},"public_addr":"172.21.15.16:6810/1398737533","cluster_addr":"172.21.15.16:6813/1398737533","heartbeat_back_addr":"172.21.15.16:6821/1398737533","heartbeat_front_addr":"172.21.15.16:6816/1398737533","state":["exists","up"]},{"osd":2,"uuid":"01a73662-9a21-4b1f-bcbb-42b9636a4638","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6819","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6822","nonce":3018526646}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6825","nonce":3018526646}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6831","nonce":3018526646}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6827","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6829","nonce":3018526646}]},"public_addr":"172.21.15.16:6822/3018526646","cluster_addr":"172.21.15.16:6825/3018526646","heartbeat_back_addr":"172.21.15.16:6831/3018526646","heartbeat_front_addr":"172.21.15.16:6829/3018526646","state":["exists","up"]},{"osd":3,"uuid":"c7771fd3-e18e-4500-ac38-23cf10ebcd75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6809","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6812","nonce":3360352974}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6817","nonce":3360352974}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6828","nonce":3360352974}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6823","nonce":3360352974}]},"public_addr":"172.21.15.16:6812/3360352974","cluster_addr":"172.21.15.16:6817/3360352974","heartbeat_back_addr":"172.21.15.16:6828/3360352974","heartbeat_front_addr":"172.21.15.16:6823/3360352974","state":["exists","up"]},{"osd":4,"uuid":"88260730-1482-4fbd-a40f-cb1124e9df4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6801","nonce":1856844856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6803","nonce":1856844856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6807","nonce":1856844856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6805","nonce":1856844856}]},"public_addr":"172.21.15.89:6801/1856844856","cluster_addr":"172.21.15.89:6803/1856844856","heartbeat_back_addr":"172.21.15.89:6807/1856844856","heartbeat_front_addr":"172.21.15.89:6805/1856844856","state":["exists","up"]},{"osd":5,"uuid":"4cb2c00c-fae0-4dd8-9d27-ec4f0093ccf7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6814","nonce":50395527}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6817","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6820","nonce":50395527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6831","nonce":50395527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6823","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6826","nonce":50395527}]},"public_addr":"172.21.15.89:6814/50395527","cluster_addr":"172.21.15.89:6820/50395527","heartbeat_back_addr":"172.21.15.89:6831/50395527","heartbeat_front_addr":"172.21.15.89:6826/50395527","state":["exists","up"]},{"osd":6,"uuid":"81ce0c19-2554-4cae-acbe-66a05e9e9b9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6809","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6812","nonce":3470471752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6815","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6818","nonce":3470471752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6827","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6829","nonce":3470471752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6821","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6824","nonce":3470471752}]},"public_addr":"172.21.15.89:6812/3470471752","cluster_addr":"172.21.15.89:6818/3470471752","heartbeat_back_addr":"172.21.15.89:6829/3470471752","heartbeat_front_addr":"172.21.15.89:6824/3470471752","state":["exists","up"]},{"osd":7,"uuid":"85e4313b-d37d-442e-8042-153bf20a847d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6811","nonce":1796778630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6813","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6816","nonce":1796778630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6825","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6828","nonce":1796778630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6819","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6822","nonce":1796778630}]},"public_addr":"172.21.15.89:6811/1796778630","cluster_addr":"172.21.15.89:6816/1796778630","heartbeat_back_addr":"172.21.15.89:6828/1796778630","heartbeat_front_addr":"172.21.15.89:6822/1796778630","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.439088+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.529970+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.680512+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:21.044161+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.582111+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.536558+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.409522+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.590229+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.2","mappings":[{"from":3,"to":2}]},{"pgid":"3.7","mappings":[{"from":3,"to":2}]},{"pgid":"3.d","mappings":[{"from":5,"to":1}]},{"pgid":"3.22","mappings":[{"from":5,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-31T15:04:18.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:18.424+0000 7f3891c00640 1 -- 172.21.15.16:0/1517060318 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f386c068380 msgr2=0x7f386c06a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:18.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:18.424+0000 7f3891c00640 1 --2- 172.21.15.16:0/1517060318 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f386c068380 0x7f386c06a840 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f387c0af000 tx=0x7f387c008040 comp rx=0 tx=0).stop 2024-03-31T15:04:18.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:18.424+0000 7f3891c00640 1 -- 172.21.15.16:0/1517060318 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f388c15b3d0 msgr2=0x7f388c15e450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:18.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:18.424+0000 7f3891c00640 1 --2- 172.21.15.16:0/1517060318 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f388c15b3d0 0x7f388c15e450 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f387800c3f0 tx=0x7f387800c7f0 comp rx=0 tx=0).stop 2024-03-31T15:04:18.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:18.424+0000 7f3891c00640 1 -- 172.21.15.16:0/1517060318 shutdown_connections 2024-03-31T15:04:18.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:18.424+0000 7f3891c00640 1 --2- 172.21.15.16:0/1517060318 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f386c068380 0x7f386c06a840 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:18.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:18.424+0000 7f3891c00640 1 --2- 172.21.15.16:0/1517060318 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f388c15eee0 0x7f388c0ae9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:18.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:18.424+0000 7f3891c00640 1 --2- 172.21.15.16:0/1517060318 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f388c15b3d0 0x7f388c15e450 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:18.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:18.424+0000 7f3891c00640 1 --2- 172.21.15.16:0/1517060318 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f388c0ce2f0 0x7f388c15df10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:18.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:18.424+0000 7f3891c00640 1 -- 172.21.15.16:0/1517060318 >> 172.21.15.16:0/1517060318 conn(0x7f388c06b620 msgr2=0x7f388c0b4760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:04:18.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:18.428+0000 7f3891c00640 1 -- 172.21.15.16:0/1517060318 shutdown_connections 2024-03-31T15:04:18.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:18.428+0000 7f3891c00640 1 -- 172.21.15.16:0/1517060318 wait complete. 2024-03-31T15:04:18.439 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-31T15:03:22.550787+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-31T15:03:34.488425+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-31T15:03:35.299116+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-31T15:04:18.440 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-31T15:04:18.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:18.588+0000 7f19b3600640 1 Processor -- start 2024-03-31T15:04:18.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:18.588+0000 7f19b3600640 1 -- start start 2024-03-31T15:04:18.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:18.588+0000 7f19b3600640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f19ac13dff0 0x7f19ac06d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:18.585 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:18.588+0000 7f19b3600640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f19ac0cd0e0 0x7f19ac06da10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:18.585 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:18.592+0000 7f19b3600640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f19ac15fbc0 con 0x7f19ac13e3d0 2024-03-31T15:04:18.585 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:18.592+0000 7f19b3600640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f19ac15f730 con 0x7f19ac0cd0e0 2024-03-31T15:04:18.585 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:18.592+0000 7f19b3600640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f19ac15ee00 con 0x7f19ac13dff0 2024-03-31T15:04:18.585 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:18.592+0000 7f19b1600640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f19ac0cd0e0 0x7f19ac06da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:18.585 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:18.592+0000 7f19b1600640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f19ac0cd0e0 0x7f19ac06da10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49138/0 (socket says 172.21.15.16:49138) 2024-03-31T15:04:18.585 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:18.592+0000 7f19b1600640 1 -- 172.21.15.16:0/130314467 learned_addr learned my addr 172.21.15.16:0/130314467 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:04:18.585 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:18.592+0000 7f19b0c00640 1 --2- 172.21.15.16:0/130314467 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f19ac13dff0 0x7f19ac06d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:18.586 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:18.592+0000 7f19b1600640 1 -- 172.21.15.16:0/130314467 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f19ac13dff0 msgr2=0x7f19ac06d4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:04:18.586 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:18.592+0000 7f19b1600640 1 --2- 172.21.15.16:0/130314467 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f19ac13dff0 0x7f19ac06d4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:18.586 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:18.592+0000 7f19b1600640 1 -- 172.21.15.16:0/130314467 >> v1:172.21.15.89:6789/0 conn(0x7f19ac13e3d0 legacy=0x7f19ac0cc9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:18.586 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:18.592+0000 7f19b1600640 1 -- 172.21.15.16:0/130314467 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19ac15f0e0 con 0x7f19ac0cd0e0 2024-03-31T15:04:18.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:18.592+0000 7f19b1600640 1 --2- 172.21.15.16:0/130314467 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f19ac0cd0e0 0x7f19ac06da10 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f19a4009e50 tx=0x7f19a401c9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7589ce137e4b4cf2 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:18.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:18.592+0000 7f19ab400640 1 -- 172.21.15.16:0/130314467 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19a40a5670 con 0x7f19ac0cd0e0 2024-03-31T15:04:18.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:18.592+0000 7f19ab400640 1 -- 172.21.15.16:0/130314467 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f19a40a5810 con 0x7f19ac0cd0e0 2024-03-31T15:04:18.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:18.592+0000 7f19ab400640 1 -- 172.21.15.16:0/130314467 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19a40a5be0 con 0x7f19ac0cd0e0 2024-03-31T15:04:18.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:18.596+0000 7f19b3600640 1 -- 172.21.15.16:0/130314467 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f19ac0cd0e0 msgr2=0x7f19ac06da10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:18.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:18.596+0000 7f19b3600640 1 --2- 172.21.15.16:0/130314467 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f19ac0cd0e0 0x7f19ac06da10 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f19a4009e50 tx=0x7f19a401c9a0 comp rx=0 tx=0).stop 2024-03-31T15:04:18.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:18.596+0000 7f19b3600640 1 -- 172.21.15.16:0/130314467 shutdown_connections 2024-03-31T15:04:18.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:18.596+0000 7f19b3600640 1 --2- 172.21.15.16:0/130314467 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f19ac0cd0e0 0x7f19ac06da10 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:18.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:18.596+0000 7f19b3600640 1 --2- 172.21.15.16:0/130314467 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f19ac13dff0 0x7f19ac06d4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:18.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:18.596+0000 7f19b3600640 1 -- 172.21.15.16:0/130314467 >> 172.21.15.16:0/130314467 conn(0x7f19ac06ee40 msgr2=0x7f19ac06b620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:04:18.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:18.596+0000 7f19b3600640 1 -- 172.21.15.16:0/130314467 shutdown_connections 2024-03-31T15:04:18.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:18.596+0000 7f19b3600640 1 -- 172.21.15.16:0/130314467 wait complete. 2024-03-31T15:04:18.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:18.596+0000 7f19b3600640 1 Processor -- start 2024-03-31T15:04:18.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:18.596+0000 7f19b3600640 1 -- start start 2024-03-31T15:04:18.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:18.596+0000 7f19b3600640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f19ac0cd0e0 0x7f19ac154f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:18.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:18.596+0000 7f19b3600640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f19ac13dff0 0x7f19ac0cc320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:18.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:18.596+0000 7f19b3600640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f19ac13e3d0 0x7f19ac15c130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:18.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:18.596+0000 7f19b3600640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f19ac0720f0 con 0x7f19ac0cd0e0 2024-03-31T15:04:18.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:18.596+0000 7f19b3600640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f19ac071f70 con 0x7f19ac13e3d0 2024-03-31T15:04:18.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:18.596+0000 7f19b3600640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f19ac072270 con 0x7f19ac13dff0 2024-03-31T15:04:18.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:18.596+0000 7f19b0c00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f19ac0cd0e0 0x7f19ac154f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:18.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:18.596+0000 7f19b0c00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f19ac0cd0e0 0x7f19ac154f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49146/0 (socket says 172.21.15.16:49146) 2024-03-31T15:04:18.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:18.596+0000 7f19b0c00640 1 -- 172.21.15.16:0/1682605323 learned_addr learned my addr 172.21.15.16:0/1682605323 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:04:18.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:18.596+0000 7f19abe00640 1 --2- 172.21.15.16:0/1682605323 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f19ac13dff0 0x7f19ac0cc320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:18.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:18.596+0000 7f19b1600640 1 --2- 172.21.15.16:0/1682605323 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f19ac13e3d0 0x7f19ac15c130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:18.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:18.596+0000 7f19b0c00640 1 -- 172.21.15.16:0/1682605323 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f19ac13dff0 msgr2=0x7f19ac0cc320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:18.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:18.596+0000 7f19b0c00640 1 --2- 172.21.15.16:0/1682605323 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f19ac13dff0 0x7f19ac0cc320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:18.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:18.596+0000 7f19b0c00640 1 -- 172.21.15.16:0/1682605323 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f19ac13e3d0 msgr2=0x7f19ac15c130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:18.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:18.596+0000 7f19b0c00640 1 --2- 172.21.15.16:0/1682605323 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f19ac13e3d0 0x7f19ac15c130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:18.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:18.596+0000 7f19b0c00640 1 -- 172.21.15.16:0/1682605323 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19ac0cca90 con 0x7f19ac0cd0e0 2024-03-31T15:04:18.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:18.600+0000 7f19b0c00640 1 --2- 172.21.15.16:0/1682605323 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f19ac0cd0e0 0x7f19ac154f20 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f19a000e930 tx=0x7f19a000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:18.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:18.600+0000 7f19a8c00640 1 -- 172.21.15.16:0/1682605323 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19a0017070 con 0x7f19ac0cd0e0 2024-03-31T15:04:18.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:18.600+0000 7f19a8c00640 1 -- 172.21.15.16:0/1682605323 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f19a000ca00 con 0x7f19ac0cd0e0 2024-03-31T15:04:18.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:18.600+0000 7f19a8c00640 1 -- 172.21.15.16:0/1682605323 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19a000cdb0 con 0x7f19ac0cd0e0 2024-03-31T15:04:18.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:18.600+0000 7f19b3600640 1 -- 172.21.15.16:0/1682605323 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f19ac0ccd70 con 0x7f19ac0cd0e0 2024-03-31T15:04:18.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:18.600+0000 7f19b3600640 1 -- 172.21.15.16:0/1682605323 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f19ac15c670 con 0x7f19ac0cd0e0 2024-03-31T15:04:18.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:18.600+0000 7f19b3600640 1 -- 172.21.15.16:0/1682605323 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f19ac069950 con 0x7f19ac0cd0e0 2024-03-31T15:04:18.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:18.600+0000 7f19a8c00640 1 -- 172.21.15.16:0/1682605323 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f19a001a070 con 0x7f19ac0cd0e0 2024-03-31T15:04:18.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:18.600+0000 7f19a8c00640 1 --2- 172.21.15.16:0/1682605323 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f198c068380 0x7f198c06a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:18.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:18.600+0000 7f19a8c00640 1 -- 172.21.15.16:0/1682605323 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f19a0086d70 con 0x7f19ac0cd0e0 2024-03-31T15:04:18.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:18.600+0000 7f19abe00640 1 --2- 172.21.15.16:0/1682605323 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f198c068380 0x7f198c06a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:18.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:18.604+0000 7f19abe00640 1 --2- 172.21.15.16:0/1682605323 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f198c068380 0x7f198c06a840 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f19ac0b11a0 tx=0x7f1998009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:18.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:18.604+0000 7f19a8c00640 1 -- 172.21.15.16:0/1682605323 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f19a0056d10 con 0x7f19ac0cd0e0 2024-03-31T15:04:18.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:18.768+0000 7f19b3600640 1 -- 172.21.15.16:0/1682605323 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f19ac0ce1b0 con 0x7f19ac0cd0e0 2024-03-31T15:04:18.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:18.768+0000 7f19a8c00640 1 -- 172.21.15.16:0/1682605323 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v23) ==== 93+0+10 (secure 0 0 0) 0x7f19a005b010 con 0x7f19ac0cd0e0 2024-03-31T15:04:18.764 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2024-03-31T15:04:18.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:18.772+0000 7f19b3600640 1 -- 172.21.15.16:0/1682605323 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f198c068380 msgr2=0x7f198c06a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:18.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:18.772+0000 7f19b3600640 1 --2- 172.21.15.16:0/1682605323 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f198c068380 0x7f198c06a840 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f19ac0b11a0 tx=0x7f1998009040 comp rx=0 tx=0).stop 2024-03-31T15:04:18.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:18.772+0000 7f19b3600640 1 -- 172.21.15.16:0/1682605323 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f19ac0cd0e0 msgr2=0x7f19ac154f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:18.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:18.772+0000 7f19b3600640 1 --2- 172.21.15.16:0/1682605323 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f19ac0cd0e0 0x7f19ac154f20 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f19a000e930 tx=0x7f19a000edf0 comp rx=0 tx=0).stop 2024-03-31T15:04:18.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:18.772+0000 7f19b3600640 1 -- 172.21.15.16:0/1682605323 shutdown_connections 2024-03-31T15:04:18.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:18.772+0000 7f19b3600640 1 --2- 172.21.15.16:0/1682605323 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f198c068380 0x7f198c06a840 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:18.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:18.772+0000 7f19b3600640 1 --2- 172.21.15.16:0/1682605323 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f19ac13e3d0 0x7f19ac15c130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:18.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:18.772+0000 7f19b3600640 1 --2- 172.21.15.16:0/1682605323 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f19ac13dff0 0x7f19ac0cc320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:18.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:18.772+0000 7f19b3600640 1 --2- 172.21.15.16:0/1682605323 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f19ac0cd0e0 0x7f19ac154f20 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:18.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:18.772+0000 7f19b3600640 1 -- 172.21.15.16:0/1682605323 >> 172.21.15.16:0/1682605323 conn(0x7f19ac06ee40 msgr2=0x7f19ac1595f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:04:18.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:18.772+0000 7f19b3600640 1 -- 172.21.15.16:0/1682605323 shutdown_connections 2024-03-31T15:04:18.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:18.772+0000 7f19b3600640 1 -- 172.21.15.16:0/1682605323 wait complete. 2024-03-31T15:04:18.786 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-31T15:04:18.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:18.936+0000 7f359ac00640 1 Processor -- start 2024-03-31T15:04:18.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:18.936+0000 7f359ac00640 1 -- start start 2024-03-31T15:04:18.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:18.936+0000 7f359ac00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f359413dff0 0x7f359406d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:18.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:18.936+0000 7f359ac00640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f359415fbc0 con 0x7f359413e3d0 2024-03-31T15:04:18.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:18.936+0000 7f359ac00640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f359415ee00 con 0x7f35940cd0e0 2024-03-31T15:04:18.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:18.936+0000 7f359ac00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f359415f730 con 0x7f359413dff0 2024-03-31T15:04:18.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:18.936+0000 7f358b400640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f359413e3d0 0x7f35940cc9d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:44336/0 (socket says 172.21.15.16:44336) 2024-03-31T15:04:18.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:18.936+0000 7f358b400640 1 -- 172.21.15.16:0/2681873356 learned_addr learned my addr 172.21.15.16:0/2681873356 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:04:18.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:18.936+0000 7f3593e00640 1 --2- 172.21.15.16:0/2681873356 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f359413dff0 0x7f359406d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:18.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:18.936+0000 7f3593400640 1 -- 172.21.15.16:0/2681873356 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1801921362 0 0) 0x7f359415fbc0 con 0x7f359413e3d0 2024-03-31T15:04:18.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:18.936+0000 7f3593400640 1 -- 172.21.15.16:0/2681873356 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3578005740 con 0x7f359413e3d0 2024-03-31T15:04:18.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:18.936+0000 7f3593400640 1 -- 172.21.15.16:0/2681873356 <== mon.1 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2684766815 0 0) 0x7f359415ee00 con 0x7f35940cd0e0 2024-03-31T15:04:18.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:18.936+0000 7f3593400640 1 -- 172.21.15.16:0/2681873356 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f359415fbc0 con 0x7f35940cd0e0 2024-03-31T15:04:18.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:18.936+0000 7f3593e00640 1 -- 172.21.15.16:0/2681873356 >> v1:172.21.15.89:6789/0 conn(0x7f35940cd0e0 legacy=0x7f35940ceb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:18.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:18.936+0000 7f3593e00640 1 -- 172.21.15.16:0/2681873356 >> v1:172.21.15.16:6789/0 conn(0x7f359413e3d0 legacy=0x7f35940cc9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:18.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:18.936+0000 7f3593e00640 1 -- 172.21.15.16:0/2681873356 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f359415f0e0 con 0x7f359413dff0 2024-03-31T15:04:18.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:18.940+0000 7f3593e00640 1 --2- 172.21.15.16:0/2681873356 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f359413dff0 0x7f359406d4d0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f35840097f0 tx=0x7f358401c450 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9eec7717714de1da server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:18.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:18.940+0000 7f3593400640 1 -- 172.21.15.16:0/2681873356 <== mon.2 v2:172.21.15.89:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35840b0070 con 0x7f359413dff0 2024-03-31T15:04:18.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:18.940+0000 7f3593400640 1 -- 172.21.15.16:0/2681873356 <== mon.2 v2:172.21.15.89:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f35840ab600 con 0x7f359413dff0 2024-03-31T15:04:18.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:18.940+0000 7f3593400640 1 -- 172.21.15.16:0/2681873356 <== mon.2 v2:172.21.15.89:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35840ab9d0 con 0x7f359413dff0 2024-03-31T15:04:18.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:18.940+0000 7f359ac00640 1 -- 172.21.15.16:0/2681873356 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f359413dff0 msgr2=0x7f359406d4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:18.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:18.940+0000 7f359ac00640 1 --2- 172.21.15.16:0/2681873356 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f359413dff0 0x7f359406d4d0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f35840097f0 tx=0x7f358401c450 comp rx=0 tx=0).stop 2024-03-31T15:04:18.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:18.940+0000 7f359ac00640 1 -- 172.21.15.16:0/2681873356 shutdown_connections 2024-03-31T15:04:18.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:18.940+0000 7f359ac00640 1 --2- 172.21.15.16:0/2681873356 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f359413dff0 0x7f359406d4d0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:18.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:18.940+0000 7f359ac00640 1 -- 172.21.15.16:0/2681873356 >> 172.21.15.16:0/2681873356 conn(0x7f359406ee40 msgr2=0x7f359406b620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:04:18.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:18.940+0000 7f359ac00640 1 -- 172.21.15.16:0/2681873356 shutdown_connections 2024-03-31T15:04:18.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:18.940+0000 7f359ac00640 1 -- 172.21.15.16:0/2681873356 wait complete. 2024-03-31T15:04:18.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:18.944+0000 7f359ac00640 1 Processor -- start 2024-03-31T15:04:18.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:18.944+0000 7f359ac00640 1 -- start start 2024-03-31T15:04:18.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:18.944+0000 7f359ac00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f35940cd0e0 0x7f3594158bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:18.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:18.944+0000 7f359ac00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f359413dff0 0x7f35940ae2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:18.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:18.944+0000 7f359ac00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f359413e3d0 0x7f3594141370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:18.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:18.944+0000 7f359ac00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f359406de20 con 0x7f359413dff0 2024-03-31T15:04:18.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:18.944+0000 7f359ac00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f35940cfca0 con 0x7f359413e3d0 2024-03-31T15:04:18.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:18.944+0000 7f359ac00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f3594071550 con 0x7f35940cd0e0 2024-03-31T15:04:18.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:18.944+0000 7f358b400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f359413dff0 0x7f35940ae2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:18.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:18.944+0000 7f3593e00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f35940cd0e0 0x7f3594158bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:18.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:18.944+0000 7f358b400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f359413dff0 0x7f35940ae2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49148/0 (socket says 172.21.15.16:49148) 2024-03-31T15:04:18.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:18.944+0000 7f358b400640 1 -- 172.21.15.16:0/3582390439 learned_addr learned my addr 172.21.15.16:0/3582390439 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:04:18.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:18.944+0000 7f3598c00640 1 --2- 172.21.15.16:0/3582390439 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f359413e3d0 0x7f3594141370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:18.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:18.944+0000 7f358b400640 1 -- 172.21.15.16:0/3582390439 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f35940cd0e0 msgr2=0x7f3594158bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:18.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:18.944+0000 7f358b400640 1 --2- 172.21.15.16:0/3582390439 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f35940cd0e0 0x7f3594158bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:18.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:18.944+0000 7f358b400640 1 -- 172.21.15.16:0/3582390439 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f359413e3d0 msgr2=0x7f3594141370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:18.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:18.944+0000 7f358b400640 1 --2- 172.21.15.16:0/3582390439 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f359413e3d0 0x7f3594141370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:18.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:18.944+0000 7f358b400640 1 -- 172.21.15.16:0/3582390439 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f359413e7b0 con 0x7f359413dff0 2024-03-31T15:04:18.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:18.944+0000 7f358b400640 1 --2- 172.21.15.16:0/3582390439 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f359413dff0 0x7f35940ae2e0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f357c00cbc0 tx=0x7f357c00a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:18.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:18.944+0000 7f3590c00640 1 -- 172.21.15.16:0/3582390439 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f357c017070 con 0x7f359413dff0 2024-03-31T15:04:18.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:18.944+0000 7f3590c00640 1 -- 172.21.15.16:0/3582390439 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f357c00ad60 con 0x7f359413dff0 2024-03-31T15:04:18.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:18.944+0000 7f3590c00640 1 -- 172.21.15.16:0/3582390439 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f357c012640 con 0x7f359413dff0 2024-03-31T15:04:18.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:18.944+0000 7f359ac00640 1 -- 172.21.15.16:0/3582390439 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f359413ea90 con 0x7f359413dff0 2024-03-31T15:04:18.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:18.944+0000 7f359ac00640 1 -- 172.21.15.16:0/3582390439 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f359413f050 con 0x7f359413dff0 2024-03-31T15:04:18.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:18.944+0000 7f359ac00640 1 -- 172.21.15.16:0/3582390439 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3594069950 con 0x7f359413dff0 2024-03-31T15:04:18.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:18.948+0000 7f3590c00640 1 -- 172.21.15.16:0/3582390439 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f357c01a070 con 0x7f359413dff0 2024-03-31T15:04:18.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:18.948+0000 7f3590c00640 1 --2- 172.21.15.16:0/3582390439 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f3570068380 0x7f357006a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:18.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:18.948+0000 7f3590c00640 1 -- 172.21.15.16:0/3582390439 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f357c0867d0 con 0x7f359413dff0 2024-03-31T15:04:18.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:18.948+0000 7f3593e00640 1 --2- 172.21.15.16:0/3582390439 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f3570068380 0x7f357006a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:18.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:18.948+0000 7f3593e00640 1 --2- 172.21.15.16:0/3582390439 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f3570068380 0x7f357006a840 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f35840ad000 tx=0x7f35840a6000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:18.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:18.952+0000 7f3590c00640 1 -- 172.21.15.16:0/3582390439 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f357c0566b0 con 0x7f359413dff0 2024-03-31T15:04:19.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:19.120+0000 7f359ac00640 1 -- 172.21.15.16:0/3582390439 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f359406e4e0 con 0x7f359413dff0 2024-03-31T15:04:19.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:19.120+0000 7f3590c00640 1 -- 172.21.15.16:0/3582390439 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v23) ==== 104+0+11 (secure 0 0 0) 0x7f357c05a9b0 con 0x7f359413dff0 2024-03-31T15:04:19.116 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2024-03-31T15:04:19.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:19.124+0000 7f359ac00640 1 -- 172.21.15.16:0/3582390439 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f3570068380 msgr2=0x7f357006a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:19.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:19.124+0000 7f359ac00640 1 --2- 172.21.15.16:0/3582390439 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f3570068380 0x7f357006a840 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f35840ad000 tx=0x7f35840a6000 comp rx=0 tx=0).stop 2024-03-31T15:04:19.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:19.124+0000 7f359ac00640 1 -- 172.21.15.16:0/3582390439 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f359413dff0 msgr2=0x7f35940ae2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:19.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:19.124+0000 7f359ac00640 1 --2- 172.21.15.16:0/3582390439 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f359413dff0 0x7f35940ae2e0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f357c00cbc0 tx=0x7f357c00a430 comp rx=0 tx=0).stop 2024-03-31T15:04:19.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:19.124+0000 7f359ac00640 1 -- 172.21.15.16:0/3582390439 shutdown_connections 2024-03-31T15:04:19.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:19.124+0000 7f359ac00640 1 --2- 172.21.15.16:0/3582390439 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f3570068380 0x7f357006a840 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:19.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:19.124+0000 7f359ac00640 1 --2- 172.21.15.16:0/3582390439 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f359413e3d0 0x7f3594141370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:19.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:19.124+0000 7f359ac00640 1 --2- 172.21.15.16:0/3582390439 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f359413dff0 0x7f35940ae2e0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:19.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:19.124+0000 7f359ac00640 1 --2- 172.21.15.16:0/3582390439 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f35940cd0e0 0x7f3594158bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:19.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:19.124+0000 7f359ac00640 1 -- 172.21.15.16:0/3582390439 >> 172.21.15.16:0/3582390439 conn(0x7f359406ee40 msgr2=0x7f35940b2560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:04:19.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:19.124+0000 7f359ac00640 1 -- 172.21.15.16:0/3582390439 shutdown_connections 2024-03-31T15:04:19.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:19.124+0000 7f359ac00640 1 -- 172.21.15.16:0/3582390439 wait complete. 2024-03-31T15:04:19.135 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-31T15:04:19.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:19.284+0000 7fabb5200640 1 Processor -- start 2024-03-31T15:04:19.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:19.284+0000 7fabb5200640 1 -- start start 2024-03-31T15:04:19.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:19.284+0000 7fabb5200640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fabb006f920 0x7fabb006fd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:19.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:19.284+0000 7fabb5200640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fabb00d0a60 0x7fabb00cea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:19.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:19.284+0000 7fabb5200640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fabb00cef80 0x7fabb0070e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:19.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:19.284+0000 7fabb5200640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7fabb013f670 con 0x7fabb00cef80 2024-03-31T15:04:19.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:19.284+0000 7fabb5200640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7fabb0070240 con 0x7fabb00d0a60 2024-03-31T15:04:19.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:19.284+0000 7fabb5200640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fabb015ca70 con 0x7fabb006f920 2024-03-31T15:04:19.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:19.284+0000 7fabae800640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fabb00cef80 0x7fabb0070e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:19.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:19.284+0000 7fabade00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fabb006f920 0x7fabb006fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:19.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:19.284+0000 7faba5400640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fabb00d0a60 0x7fabb00cea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:19.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:19.284+0000 7fabae800640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fabb00cef80 0x7fabb0070e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49162/0 (socket says 172.21.15.16:49162) 2024-03-31T15:04:19.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:19.284+0000 7fabade00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fabb006f920 0x7fabb006fd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3301/0 says I am v2:172.21.15.16:37582/0 (socket says 172.21.15.16:37582) 2024-03-31T15:04:19.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:19.284+0000 7fabae800640 1 -- 172.21.15.16:0/537509452 learned_addr learned my addr 172.21.15.16:0/537509452 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:04:19.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:19.284+0000 7fabae800640 1 -- 172.21.15.16:0/537509452 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fabb006f920 msgr2=0x7fabb006fd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:04:19.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:19.284+0000 7fabae800640 1 --2- 172.21.15.16:0/537509452 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fabb006f920 0x7fabb006fd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:19.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:19.284+0000 7fabae800640 1 -- 172.21.15.16:0/537509452 >> v2:172.21.15.89:3300/0 conn(0x7fabb00d0a60 msgr2=0x7fabb00cea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:04:19.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:19.284+0000 7fabae800640 1 --2- 172.21.15.16:0/537509452 >> v2:172.21.15.89:3300/0 conn(0x7fabb00d0a60 0x7fabb00cea40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:19.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:19.284+0000 7fabae800640 1 -- 172.21.15.16:0/537509452 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fabb015cd50 con 0x7fabb00cef80 2024-03-31T15:04:19.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:19.284+0000 7fabade00640 1 --2- 172.21.15.16:0/537509452 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fabb006f920 0x7fabb006fd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:04:19.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:19.288+0000 7fabae800640 1 --2- 172.21.15.16:0/537509452 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fabb00cef80 0x7fabb0070e60 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fabb0071f30 tx=0x7faba801c730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1891eb36458b3c1c server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:19.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:19.288+0000 7fabad400640 1 -- 172.21.15.16:0/537509452 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faba80ac070 con 0x7fabb00cef80 2024-03-31T15:04:19.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:19.288+0000 7fabad400640 1 -- 172.21.15.16:0/537509452 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faba80a8420 con 0x7fabb00cef80 2024-03-31T15:04:19.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:19.288+0000 7fabad400640 1 -- 172.21.15.16:0/537509452 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faba80a87f0 con 0x7fabb00cef80 2024-03-31T15:04:19.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:19.288+0000 7fabb5200640 1 -- 172.21.15.16:0/537509452 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fabb00cef80 msgr2=0x7fabb0070e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:19.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:19.288+0000 7fabb5200640 1 --2- 172.21.15.16:0/537509452 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fabb00cef80 0x7fabb0070e60 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fabb0071f30 tx=0x7faba801c730 comp rx=0 tx=0).stop 2024-03-31T15:04:19.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:19.288+0000 7fabb5200640 1 -- 172.21.15.16:0/537509452 shutdown_connections 2024-03-31T15:04:19.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:19.288+0000 7fabb5200640 1 --2- 172.21.15.16:0/537509452 >> v2:172.21.15.89:3300/0 conn(0x7fabb00d0a60 0x7fabb00cea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:19.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:19.288+0000 7fabb5200640 1 --2- 172.21.15.16:0/537509452 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fabb00cef80 0x7fabb0070e60 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:19.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:19.288+0000 7fabb5200640 1 --2- 172.21.15.16:0/537509452 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fabb006f920 0x7fabb006fd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:19.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:19.288+0000 7fabb5200640 1 -- 172.21.15.16:0/537509452 >> 172.21.15.16:0/537509452 conn(0x7fabb0067930 msgr2=0x7fabb00b3ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:04:19.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:19.288+0000 7fabb5200640 1 -- 172.21.15.16:0/537509452 shutdown_connections 2024-03-31T15:04:19.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:19.288+0000 7fabb5200640 1 -- 172.21.15.16:0/537509452 wait complete. 2024-03-31T15:04:19.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:19.292+0000 7fabb5200640 1 Processor -- start 2024-03-31T15:04:19.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:19.292+0000 7fabb5200640 1 -- start start 2024-03-31T15:04:19.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:19.292+0000 7fabb5200640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fabb006f920 0x7fabb013eb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:19.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:19.292+0000 7fabb5200640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fabb00cef80 0x7fabb006c730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:19.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:19.292+0000 7fabb5200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fabb00d0a60 0x7fabb00b2ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:19.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:19.292+0000 7fabb5200640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fabb01419c0 con 0x7fabb00d0a60 2024-03-31T15:04:19.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:19.292+0000 7fabb5200640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fabb015c720 con 0x7fabb006f920 2024-03-31T15:04:19.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:19.292+0000 7fabb5200640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fabb0141b40 con 0x7fabb00cef80 2024-03-31T15:04:19.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:19.292+0000 7fabae800640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fabb00d0a60 0x7fabb00b2ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:19.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:19.292+0000 7fabade00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fabb006f920 0x7fabb013eb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:19.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:19.292+0000 7faba5400640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fabb00cef80 0x7fabb006c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:19.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:19.292+0000 7fabade00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fabb006f920 0x7fabb013eb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.16:49000/0 (socket says 172.21.15.16:49000) 2024-03-31T15:04:19.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:19.292+0000 7faba5400640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fabb00cef80 0x7fabb006c730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3301/0 says I am v2:172.21.15.16:37586/0 (socket says 172.21.15.16:37586) 2024-03-31T15:04:19.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:19.292+0000 7fabade00640 1 -- 172.21.15.16:0/3736331440 learned_addr learned my addr 172.21.15.16:0/3736331440 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:04:19.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:19.292+0000 7fabade00640 1 -- 172.21.15.16:0/3736331440 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fabb00cef80 msgr2=0x7fabb006c730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:19.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:19.292+0000 7fabade00640 1 --2- 172.21.15.16:0/3736331440 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fabb00cef80 0x7fabb006c730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:19.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:19.292+0000 7fabade00640 1 -- 172.21.15.16:0/3736331440 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fabb00d0a60 msgr2=0x7fabb00b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:19.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:19.292+0000 7fabade00640 1 --2- 172.21.15.16:0/3736331440 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fabb00d0a60 0x7fabb00b2ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:19.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:19.292+0000 7fabade00640 1 -- 172.21.15.16:0/3736331440 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fabb013f060 con 0x7fabb006f920 2024-03-31T15:04:19.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:19.292+0000 7fabade00640 1 --2- 172.21.15.16:0/3736331440 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fabb006f920 0x7fabb013eb20 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7faba000ebe0 tx=0x7faba000c520 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:19.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:19.292+0000 7faba6a00640 1 -- 172.21.15.16:0/3736331440 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faba0017070 con 0x7fabb006f920 2024-03-31T15:04:19.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:19.292+0000 7faba6a00640 1 -- 172.21.15.16:0/3736331440 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faba000cc30 con 0x7fabb006f920 2024-03-31T15:04:19.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:19.292+0000 7fabb5200640 1 -- 172.21.15.16:0/3736331440 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fabb006cf80 con 0x7fabb006f920 2024-03-31T15:04:19.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:19.292+0000 7faba6a00640 1 -- 172.21.15.16:0/3736331440 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faba0013470 con 0x7fabb006f920 2024-03-31T15:04:19.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:19.292+0000 7fabb5200640 1 -- 172.21.15.16:0/3736331440 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fabb006d8a0 con 0x7fabb006f920 2024-03-31T15:04:19.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:19.296+0000 7fabb5200640 1 -- 172.21.15.16:0/3736331440 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fabb0069950 con 0x7fabb006f920 2024-03-31T15:04:19.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:19.296+0000 7faba6a00640 1 -- 172.21.15.16:0/3736331440 <== mon.1 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7faba001a070 con 0x7fabb006f920 2024-03-31T15:04:19.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:19.296+0000 7faba6a00640 1 --2- 172.21.15.16:0/3736331440 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fab8c0683b0 0x7fab8c06a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:19.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:19.296+0000 7faba6a00640 1 -- 172.21.15.16:0/3736331440 <== mon.1 v2:172.21.15.89:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7faba0086910 con 0x7fabb006f920 2024-03-31T15:04:19.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:19.296+0000 7faba5400640 1 --2- 172.21.15.16:0/3736331440 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fab8c0683b0 0x7fab8c06a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:19.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:19.296+0000 7faba5400640 1 --2- 172.21.15.16:0/3736331440 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fab8c0683b0 0x7fab8c06a870 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fabb00b7360 tx=0x7fab98009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:19.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:19.300+0000 7faba6a00640 1 -- 172.21.15.16:0/3736331440 <== mon.1 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7faba0056760 con 0x7fabb006f920 2024-03-31T15:04:19.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:19.464+0000 7fabb5200640 1 -- 172.21.15.16:0/3736331440 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fabb00b4a20 con 0x7fabb006f920 2024-03-31T15:04:19.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:19.464+0000 7faba6a00640 1 -- 172.21.15.16:0/3736331440 <== mon.1 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v23) ==== 100+0+11 (secure 0 0 0) 0x7faba005aa60 con 0x7fabb006f920 2024-03-31T15:04:19.461 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 64 2024-03-31T15:04:19.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:19.468+0000 7fabb5200640 1 -- 172.21.15.16:0/3736331440 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fab8c0683b0 msgr2=0x7fab8c06a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:19.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:19.468+0000 7fabb5200640 1 --2- 172.21.15.16:0/3736331440 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fab8c0683b0 0x7fab8c06a870 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fabb00b7360 tx=0x7fab98009190 comp rx=0 tx=0).stop 2024-03-31T15:04:19.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:19.468+0000 7fabb5200640 1 -- 172.21.15.16:0/3736331440 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fabb006f920 msgr2=0x7fabb013eb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:19.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:19.468+0000 7fabb5200640 1 --2- 172.21.15.16:0/3736331440 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fabb006f920 0x7fabb013eb20 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7faba000ebe0 tx=0x7faba000c520 comp rx=0 tx=0).stop 2024-03-31T15:04:19.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:19.468+0000 7fabb5200640 1 -- 172.21.15.16:0/3736331440 shutdown_connections 2024-03-31T15:04:19.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:19.468+0000 7fabb5200640 1 --2- 172.21.15.16:0/3736331440 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fab8c0683b0 0x7fab8c06a870 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:19.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:19.468+0000 7fabb5200640 1 --2- 172.21.15.16:0/3736331440 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fabb00d0a60 0x7fabb00b2ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:19.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:19.468+0000 7fabb5200640 1 --2- 172.21.15.16:0/3736331440 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fabb00cef80 0x7fabb006c730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:19.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:19.468+0000 7fabb5200640 1 --2- 172.21.15.16:0/3736331440 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fabb006f920 0x7fabb013eb20 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:19.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:19.468+0000 7fabb5200640 1 -- 172.21.15.16:0/3736331440 >> 172.21.15.16:0/3736331440 conn(0x7fabb0067930 msgr2=0x7fabb0140e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:04:19.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:19.468+0000 7fabb5200640 1 -- 172.21.15.16:0/3736331440 shutdown_connections 2024-03-31T15:04:19.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:19.468+0000 7fabb5200640 1 -- 172.21.15.16:0/3736331440 wait complete. 2024-03-31T15:04:19.481 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-31T15:04:19.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:19.628+0000 7f97c6000640 1 Processor -- start 2024-03-31T15:04:19.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:19.628+0000 7f97c6000640 1 -- start start 2024-03-31T15:04:19.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:19.628+0000 7f97c6000640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f97c0145d10 0x7f97c00cf790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:19.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:19.628+0000 7f97c6000640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f97c00728b0 0x7f97c00b1cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:19.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:19.628+0000 7f97c6000640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f97c00b2200 0x7f97c015cd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:19.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:19.628+0000 7f97c6000640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f97c0160e70 con 0x7f97c0145d10 2024-03-31T15:04:19.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:19.628+0000 7f97c6000640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f97c00cfcd0 con 0x7f97c00728b0 2024-03-31T15:04:19.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:19.628+0000 7f97c6000640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f97c01602f0 con 0x7f97c00b2200 2024-03-31T15:04:19.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:19.628+0000 7f97bf400640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f97c0145d10 0x7f97c00cf790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:19.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:19.628+0000 7f97bf400640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f97c0145d10 0x7f97c00cf790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49186/0 (socket says 172.21.15.16:49186) 2024-03-31T15:04:19.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:19.628+0000 7f97bfe00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f97c00b2200 0x7f97c015cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:19.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:19.628+0000 7f97b6a00640 1 --2- 172.21.15.16:0/1146856158 >> v2:172.21.15.89:3300/0 conn(0x7f97c00728b0 0x7f97c00b1cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:19.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:19.628+0000 7f97bf400640 1 -- 172.21.15.16:0/1146856158 learned_addr learned my addr 172.21.15.16:0/1146856158 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:04:19.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:19.628+0000 7f97bf400640 1 -- 172.21.15.16:0/1146856158 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f97c00b2200 msgr2=0x7f97c015cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:04:19.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:19.628+0000 7f97bf400640 1 --2- 172.21.15.16:0/1146856158 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f97c00b2200 0x7f97c015cd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:19.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:19.628+0000 7f97bf400640 1 -- 172.21.15.16:0/1146856158 >> v2:172.21.15.89:3300/0 conn(0x7f97c00728b0 msgr2=0x7f97c00b1cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:04:19.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:19.628+0000 7f97bf400640 1 --2- 172.21.15.16:0/1146856158 >> v2:172.21.15.89:3300/0 conn(0x7f97c00728b0 0x7f97c00b1cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:19.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:19.628+0000 7f97bf400640 1 -- 172.21.15.16:0/1146856158 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97c01605d0 con 0x7f97c0145d10 2024-03-31T15:04:19.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:19.628+0000 7f97bfe00640 1 --2- 172.21.15.16:0/1146856158 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f97c00b2200 0x7f97c015cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:04:19.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:19.632+0000 7f97bf400640 1 --2- 172.21.15.16:0/1146856158 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f97c0145d10 0x7f97c00cf790 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f97c00b2980 tx=0x7f97b001c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cbdd6f84be166b67 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:19.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:19.632+0000 7f97bea00640 1 -- 172.21.15.16:0/1146856158 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97b00b0070 con 0x7f97c0145d10 2024-03-31T15:04:19.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:19.632+0000 7f97bea00640 1 -- 172.21.15.16:0/1146856158 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f97b00ab600 con 0x7f97c0145d10 2024-03-31T15:04:19.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:19.632+0000 7f97bea00640 1 -- 172.21.15.16:0/1146856158 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97b00ab9d0 con 0x7f97c0145d10 2024-03-31T15:04:19.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:19.632+0000 7f97c6000640 1 -- 172.21.15.16:0/1146856158 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f97c0145d10 msgr2=0x7f97c00cf790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:19.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:19.632+0000 7f97c6000640 1 --2- 172.21.15.16:0/1146856158 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f97c0145d10 0x7f97c00cf790 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f97c00b2980 tx=0x7f97b001c520 comp rx=0 tx=0).stop 2024-03-31T15:04:19.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:19.632+0000 7f97c6000640 1 -- 172.21.15.16:0/1146856158 shutdown_connections 2024-03-31T15:04:19.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:19.632+0000 7f97c6000640 1 --2- 172.21.15.16:0/1146856158 >> v2:172.21.15.89:3300/0 conn(0x7f97c00728b0 0x7f97c00b1cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:19.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:19.632+0000 7f97c6000640 1 --2- 172.21.15.16:0/1146856158 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f97c00b2200 0x7f97c015cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:19.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:19.632+0000 7f97c6000640 1 --2- 172.21.15.16:0/1146856158 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f97c0145d10 0x7f97c00cf790 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:19.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:19.632+0000 7f97c6000640 1 -- 172.21.15.16:0/1146856158 >> 172.21.15.16:0/1146856158 conn(0x7f97c006b620 msgr2=0x7f97c006ba30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:04:19.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:19.632+0000 7f97c6000640 1 -- 172.21.15.16:0/1146856158 shutdown_connections 2024-03-31T15:04:19.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:19.632+0000 7f97c6000640 1 -- 172.21.15.16:0/1146856158 wait complete. 2024-03-31T15:04:19.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:19.636+0000 7f97c6000640 1 Processor -- start 2024-03-31T15:04:19.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:19.636+0000 7f97c6000640 1 -- start start 2024-03-31T15:04:19.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:19.636+0000 7f97c6000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f97c00728b0 0x7f97c0159c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:19.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:19.636+0000 7f97c6000640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f97c00b2200 0x7f97c01586d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:19.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:19.636+0000 7f97c6000640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f97c0145d10 0x7f97c0157e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:19.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:19.636+0000 7f97c6000640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f97c00cf350 con 0x7f97c00728b0 2024-03-31T15:04:19.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:19.636+0000 7f97c6000640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f97c00b2610 con 0x7f97c00b2200 2024-03-31T15:04:19.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:19.636+0000 7f97c6000640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f97c00cf4d0 con 0x7f97c0145d10 2024-03-31T15:04:19.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:19.636+0000 7f97bf400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f97c00728b0 0x7f97c0159c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:19.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:19.636+0000 7f97bfe00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f97c0145d10 0x7f97c0157e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:19.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:19.636+0000 7f97bf400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f97c00728b0 0x7f97c0159c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49188/0 (socket says 172.21.15.16:49188) 2024-03-31T15:04:19.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:19.636+0000 7f97bf400640 1 -- 172.21.15.16:0/426218432 learned_addr learned my addr 172.21.15.16:0/426218432 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:04:19.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:19.636+0000 7f97b6a00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f97c00b2200 0x7f97c01586d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:19.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:19.636+0000 7f97b6a00640 1 -- 172.21.15.16:0/426218432 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f97c0145d10 msgr2=0x7f97c0157e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:19.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:19.636+0000 7f97b6a00640 1 --2- 172.21.15.16:0/426218432 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f97c0145d10 0x7f97c0157e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:19.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:19.636+0000 7f97b6a00640 1 -- 172.21.15.16:0/426218432 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f97c00728b0 msgr2=0x7f97c0159c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:19.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:19.636+0000 7f97b6a00640 1 --2- 172.21.15.16:0/426218432 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f97c00728b0 0x7f97c0159c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:19.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:19.636+0000 7f97b6a00640 1 -- 172.21.15.16:0/426218432 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97c0158c10 con 0x7f97c00b2200 2024-03-31T15:04:19.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:19.636+0000 7f97bf400640 1 --2- 172.21.15.16:0/426218432 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f97c00728b0 0x7f97c0159c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:04:19.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:19.636+0000 7f97bfe00640 1 --2- 172.21.15.16:0/426218432 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f97c0145d10 0x7f97c0157e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:04:19.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:19.636+0000 7f97b6a00640 1 --2- 172.21.15.16:0/426218432 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f97c00b2200 0x7f97c01586d0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f97a800e870 tx=0x7f97a800ec70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:19.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:19.636+0000 7f97b7e00640 1 -- 172.21.15.16:0/426218432 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97a8017070 con 0x7f97c00b2200 2024-03-31T15:04:19.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:19.636+0000 7f97b7e00640 1 -- 172.21.15.16:0/426218432 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f97a800c730 con 0x7f97c00b2200 2024-03-31T15:04:19.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:19.636+0000 7f97b7e00640 1 -- 172.21.15.16:0/426218432 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97a800cae0 con 0x7f97c00b2200 2024-03-31T15:04:19.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:19.636+0000 7f97c6000640 1 -- 172.21.15.16:0/426218432 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f97c0158eb0 con 0x7f97c00b2200 2024-03-31T15:04:19.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:19.636+0000 7f97c6000640 1 -- 172.21.15.16:0/426218432 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f97c0159470 con 0x7f97c00b2200 2024-03-31T15:04:19.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:19.636+0000 7f97c6000640 1 -- 172.21.15.16:0/426218432 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f97c0069950 con 0x7f97c00b2200 2024-03-31T15:04:19.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:19.640+0000 7f97b7e00640 1 -- 172.21.15.16:0/426218432 <== mon.1 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f97a801a070 con 0x7f97c00b2200 2024-03-31T15:04:19.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:19.640+0000 7f97b7e00640 1 --2- 172.21.15.16:0/426218432 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f979c068380 0x7f979c06a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:19.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:19.640+0000 7f97b7e00640 1 -- 172.21.15.16:0/426218432 <== mon.1 v2:172.21.15.89:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f97a8086ac0 con 0x7f97c00b2200 2024-03-31T15:04:19.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:19.640+0000 7f97bf400640 1 --2- 172.21.15.16:0/426218432 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f979c068380 0x7f979c06a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:19.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:19.640+0000 7f97bf400640 1 --2- 172.21.15.16:0/426218432 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f979c068380 0x7f979c06a840 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f97b001c560 tx=0x7f97b0008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:19.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:19.644+0000 7f97b7e00640 1 -- 172.21.15.16:0/426218432 <== mon.1 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f97a8056920 con 0x7f97c00b2200 2024-03-31T15:04:19.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:19.820+0000 7f97c6000640 1 -- 172.21.15.16:0/426218432 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f97c00b15f0 con 0x7f97c00b2200 2024-03-31T15:04:19.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:19.820+0000 7f97b7e00640 1 -- 172.21.15.16:0/426218432 <== mon.1 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v23) ==== 80+0+0 (secure 0 0 0) 0x7f97a805ac20 con 0x7f97c00b2200 2024-03-31T15:04:19.815 INFO:teuthology.orchestra.run.smithi016.stderr:listed 0 entries 2024-03-31T15:04:19.818 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:19.824+0000 7f97c6000640 1 -- 172.21.15.16:0/426218432 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f979c068380 msgr2=0x7f979c06a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:19.818 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:19.824+0000 7f97c6000640 1 --2- 172.21.15.16:0/426218432 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f979c068380 0x7f979c06a840 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f97b001c560 tx=0x7f97b0008040 comp rx=0 tx=0).stop 2024-03-31T15:04:19.818 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:19.824+0000 7f97c6000640 1 -- 172.21.15.16:0/426218432 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f97c00b2200 msgr2=0x7f97c01586d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:19.818 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:19.824+0000 7f97c6000640 1 --2- 172.21.15.16:0/426218432 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f97c00b2200 0x7f97c01586d0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f97a800e870 tx=0x7f97a800ec70 comp rx=0 tx=0).stop 2024-03-31T15:04:19.818 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:19.824+0000 7f97c6000640 1 -- 172.21.15.16:0/426218432 shutdown_connections 2024-03-31T15:04:19.818 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:19.824+0000 7f97c6000640 1 --2- 172.21.15.16:0/426218432 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f979c068380 0x7f979c06a840 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:19.818 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:19.824+0000 7f97c6000640 1 --2- 172.21.15.16:0/426218432 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f97c0145d10 0x7f97c0157e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:19.818 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:19.824+0000 7f97c6000640 1 --2- 172.21.15.16:0/426218432 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f97c00b2200 0x7f97c01586d0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:19.818 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:19.824+0000 7f97c6000640 1 --2- 172.21.15.16:0/426218432 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f97c00728b0 0x7f97c0159c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:19.818 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:19.824+0000 7f97c6000640 1 -- 172.21.15.16:0/426218432 >> 172.21.15.16:0/426218432 conn(0x7f97c006b620 msgr2=0x7f97c00b0260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:04:19.818 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:19.824+0000 7f97c6000640 1 -- 172.21.15.16:0/426218432 shutdown_connections 2024-03-31T15:04:19.818 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:19.824+0000 7f97c6000640 1 -- 172.21.15.16:0/426218432 wait complete. 2024-03-31T15:04:19.837 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-03-31T15:04:19.837 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-31T15:04:19.846 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-31T15:04:20.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:20.032+0000 7f1ea1200640 1 Processor -- start 2024-03-31T15:04:20.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:20.032+0000 7f1ea1200640 1 -- start start 2024-03-31T15:04:20.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:20.036+0000 7f1ea1200640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f1e9c0b23d0 0x7f1e9c071d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:20.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:20.036+0000 7f1ea1200640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f1e9c15eee0 0x7f1e9c072250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:20.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:20.036+0000 7f1ea1200640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1e9c15f3c0 con 0x7f1e9c15b3d0 2024-03-31T15:04:20.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:20.036+0000 7f1ea1200640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f1e9c15b7b0 con 0x7f1e9c15eee0 2024-03-31T15:04:20.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:20.036+0000 7f1ea1200640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f1e9c15f6a0 con 0x7f1e9c0b23d0 2024-03-31T15:04:20.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:20.036+0000 7f1e99e00640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f1e9c15b3d0 0x7f1e9c0b1cc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:44352/0 (socket says 172.21.15.16:44352) 2024-03-31T15:04:20.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:20.036+0000 7f1e99e00640 1 -- 172.21.15.16:0/2542920717 learned_addr learned my addr 172.21.15.16:0/2542920717 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:04:20.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:20.036+0000 7f1e9a800640 1 --2- 172.21.15.16:0/2542920717 >> v2:172.21.15.89:3300/0 conn(0x7f1e9c15eee0 0x7f1e9c072250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:20.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:20.036+0000 7f1e93e00640 1 --2- 172.21.15.16:0/2542920717 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f1e9c0b23d0 0x7f1e9c071d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:20.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:20.036+0000 7f1e99400640 1 -- 172.21.15.16:0/2542920717 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4047762020 0 0) 0x7f1e9c15f3c0 con 0x7f1e9c15b3d0 2024-03-31T15:04:20.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:20.036+0000 7f1e99400640 1 -- 172.21.15.16:0/2542920717 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1e80003530 con 0x7f1e9c15b3d0 2024-03-31T15:04:20.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:20.036+0000 7f1e9a800640 1 -- 172.21.15.16:0/2542920717 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f1e9c0b23d0 msgr2=0x7f1e9c071d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:04:20.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:20.036+0000 7f1e9a800640 1 --2- 172.21.15.16:0/2542920717 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f1e9c0b23d0 0x7f1e9c071d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:20.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:20.036+0000 7f1e9a800640 1 -- 172.21.15.16:0/2542920717 >> v1:172.21.15.16:6789/0 conn(0x7f1e9c15b3d0 legacy=0x7f1e9c0b1cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:20.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:20.036+0000 7f1e9a800640 1 -- 172.21.15.16:0/2542920717 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e9c15f980 con 0x7f1e9c15eee0 2024-03-31T15:04:20.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:20.036+0000 7f1e93e00640 1 --2- 172.21.15.16:0/2542920717 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f1e9c0b23d0 0x7f1e9c071d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:04:20.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:20.040+0000 7f1e9a800640 1 --2- 172.21.15.16:0/2542920717 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1e9c15eee0 0x7f1e9c072250 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f1e94009d90 tx=0x7f1e9401c940 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2163d72bbdf81a9f server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:20.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:20.040+0000 7f1e99400640 1 -- 172.21.15.16:0/2542920717 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e940b2070 con 0x7f1e9c15eee0 2024-03-31T15:04:20.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:20.040+0000 7f1e99400640 1 -- 172.21.15.16:0/2542920717 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1e940a59c0 con 0x7f1e9c15eee0 2024-03-31T15:04:20.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:20.040+0000 7f1e99400640 1 -- 172.21.15.16:0/2542920717 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e940a5d70 con 0x7f1e9c15eee0 2024-03-31T15:04:20.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:20.040+0000 7f1ea1200640 1 -- 172.21.15.16:0/2542920717 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1e9c15eee0 msgr2=0x7f1e9c072250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:20.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:20.040+0000 7f1ea1200640 1 --2- 172.21.15.16:0/2542920717 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1e9c15eee0 0x7f1e9c072250 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f1e94009d90 tx=0x7f1e9401c940 comp rx=0 tx=0).stop 2024-03-31T15:04:20.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:20.040+0000 7f1ea1200640 1 -- 172.21.15.16:0/2542920717 shutdown_connections 2024-03-31T15:04:20.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:20.040+0000 7f1ea1200640 1 --2- 172.21.15.16:0/2542920717 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1e9c15eee0 0x7f1e9c072250 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:20.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:20.040+0000 7f1ea1200640 1 --2- 172.21.15.16:0/2542920717 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f1e9c0b23d0 0x7f1e9c071d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:20.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:20.040+0000 7f1ea1200640 1 -- 172.21.15.16:0/2542920717 >> 172.21.15.16:0/2542920717 conn(0x7f1e9c06b620 msgr2=0x7f1e9c06ba30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:04:20.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:20.040+0000 7f1ea1200640 1 -- 172.21.15.16:0/2542920717 shutdown_connections 2024-03-31T15:04:20.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:20.040+0000 7f1ea1200640 1 -- 172.21.15.16:0/2542920717 wait complete. 2024-03-31T15:04:20.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:20.040+0000 7f1ea1200640 1 Processor -- start 2024-03-31T15:04:20.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:20.040+0000 7f1ea1200640 1 -- start start 2024-03-31T15:04:20.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:20.044+0000 7f1ea1200640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1e9c0b23d0 0x7f1e9c150ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:20.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:20.044+0000 7f1ea1200640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f1e9c15b3d0 0x7f1e9c152f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:20.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:20.044+0000 7f1ea1200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1e9c15eee0 0x7f1e9c14ff70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:20.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:20.044+0000 7f1ea1200640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f1e9c073910 con 0x7f1e9c15eee0 2024-03-31T15:04:20.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:20.044+0000 7f1ea1200640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f1e9c073790 con 0x7f1e9c0b23d0 2024-03-31T15:04:20.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:20.044+0000 7f1ea1200640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f1e9c072790 con 0x7f1e9c15b3d0 2024-03-31T15:04:20.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:20.044+0000 7f1e99e00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1e9c0b23d0 0x7f1e9c150ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:20.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:20.044+0000 7f1e9a800640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1e9c15eee0 0x7f1e9c14ff70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:20.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:20.044+0000 7f1e99e00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1e9c0b23d0 0x7f1e9c150ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.16:49046/0 (socket says 172.21.15.16:49046) 2024-03-31T15:04:20.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:20.044+0000 7f1e99e00640 1 -- 172.21.15.16:0/2926491984 learned_addr learned my addr 172.21.15.16:0/2926491984 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:04:20.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:20.044+0000 7f1e93e00640 1 --2- 172.21.15.16:0/2926491984 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f1e9c15b3d0 0x7f1e9c152f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:20.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:20.044+0000 7f1e99e00640 1 -- 172.21.15.16:0/2926491984 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f1e9c15b3d0 msgr2=0x7f1e9c152f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:20.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:20.044+0000 7f1e99e00640 1 --2- 172.21.15.16:0/2926491984 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f1e9c15b3d0 0x7f1e9c152f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:20.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:20.044+0000 7f1e99e00640 1 -- 172.21.15.16:0/2926491984 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1e9c15eee0 msgr2=0x7f1e9c14ff70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:20.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:20.044+0000 7f1e99e00640 1 --2- 172.21.15.16:0/2926491984 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1e9c15eee0 0x7f1e9c14ff70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:20.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:20.044+0000 7f1e99e00640 1 -- 172.21.15.16:0/2926491984 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e9c151ef0 con 0x7f1e9c0b23d0 2024-03-31T15:04:20.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:20.044+0000 7f1e99e00640 1 --2- 172.21.15.16:0/2926491984 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1e9c0b23d0 0x7f1e9c150ac0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f1e8c00c450 tx=0x7f1e8c00c910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:20.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:20.044+0000 7f1e92000640 1 -- 172.21.15.16:0/2926491984 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e8c017070 con 0x7f1e9c0b23d0 2024-03-31T15:04:20.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:20.044+0000 7f1e92000640 1 -- 172.21.15.16:0/2926491984 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1e8c005cf0 con 0x7f1e9c0b23d0 2024-03-31T15:04:20.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:20.044+0000 7f1e92000640 1 -- 172.21.15.16:0/2926491984 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e8c007290 con 0x7f1e9c0b23d0 2024-03-31T15:04:20.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:20.044+0000 7f1ea1200640 1 -- 172.21.15.16:0/2926491984 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1e9c1521d0 con 0x7f1e9c0b23d0 2024-03-31T15:04:20.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:20.044+0000 7f1ea1200640 1 -- 172.21.15.16:0/2926491984 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1e9c152790 con 0x7f1e9c0b23d0 2024-03-31T15:04:20.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:20.044+0000 7f1ea1200640 1 -- 172.21.15.16:0/2926491984 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1e9c069950 con 0x7f1e9c0b23d0 2024-03-31T15:04:20.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:20.048+0000 7f1e92000640 1 -- 172.21.15.16:0/2926491984 <== mon.1 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f1e8c01a070 con 0x7f1e9c0b23d0 2024-03-31T15:04:20.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:20.048+0000 7f1e92000640 1 --2- 172.21.15.16:0/2926491984 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f1e78068380 0x7f1e7806a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:20.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:20.048+0000 7f1e92000640 1 -- 172.21.15.16:0/2926491984 <== mon.1 v2:172.21.15.89:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f1e8c085e00 con 0x7f1e9c0b23d0 2024-03-31T15:04:20.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:20.048+0000 7f1e93e00640 1 --2- 172.21.15.16:0/2926491984 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f1e78068380 0x7f1e7806a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:20.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:20.048+0000 7f1e93e00640 1 --2- 172.21.15.16:0/2926491984 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f1e78068380 0x7f1e7806a840 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f1e84009580 tx=0x7f1e84009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:20.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:20.052+0000 7f1e92000640 1 -- 172.21.15.16:0/2926491984 <== mon.1 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f1e8c055d50 con 0x7f1e9c0b23d0 2024-03-31T15:04:20.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:20.220+0000 7f1ea1200640 1 -- 172.21.15.16:0/2926491984 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f1e9c0cdea0 con 0x7f1e9c0b23d0 2024-03-31T15:04:20.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:20.220+0000 7f1e92000640 1 -- 172.21.15.16:0/2926491984 <== mon.1 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v23) ==== 74+0+16780 (secure 0 0 0) 0x7f1e8c05a050 con 0x7f1e9c0b23d0 2024-03-31T15:04:20.215 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:04:20.216 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":23,"fsid":"286eb5da-fb47-4550-b290-d9519c13b9a5","created":"2024-03-31T15:03:11.438952+0000","modified":"2024-03-31T15:04:18.087947+0000","last_up_change":"2024-03-31T15:03:22.885535+0000","last_in_change":"2024-03-31T15:03:13.941667+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-31T15:03:22.550787+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-31T15:03:34.488425+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-31T15:03:35.299116+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"92a66450-ef19-40d8-ac3d-d760b23b0bda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6801","nonce":932712244}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6803","nonce":932712244}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6807","nonce":932712244}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6805","nonce":932712244}]},"public_addr":"172.21.15.16:6801/932712244","cluster_addr":"172.21.15.16:6803/932712244","heartbeat_back_addr":"172.21.15.16:6807/932712244","heartbeat_front_addr":"172.21.15.16:6805/932712244","state":["exists","up"]},{"osd":1,"uuid":"da9d1f2b-1968-4959-b843-faa231988e92","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6810","nonce":1398737533}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6811","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6813","nonce":1398737533}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6821","nonce":1398737533}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6815","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6816","nonce":1398737533}]},"public_addr":"172.21.15.16:6810/1398737533","cluster_addr":"172.21.15.16:6813/1398737533","heartbeat_back_addr":"172.21.15.16:6821/1398737533","heartbeat_front_addr":"172.21.15.16:6816/1398737533","state":["exists","up"]},{"osd":2,"uuid":"01a73662-9a21-4b1f-bcbb-42b9636a4638","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6819","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6822","nonce":3018526646}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6825","nonce":3018526646}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6831","nonce":3018526646}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6827","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6829","nonce":3018526646}]},"public_addr":"172.21.15.16:6822/3018526646","cluster_addr":"172.21.15.16:6825/3018526646","heartbeat_back_addr":"172.21.15.16:6831/3018526646","heartbeat_front_addr":"172.21.15.16:6829/3018526646","state":["exists","up"]},{"osd":3,"uuid":"c7771fd3-e18e-4500-ac38-23cf10ebcd75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6809","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6812","nonce":3360352974}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6817","nonce":3360352974}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6828","nonce":3360352974}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6823","nonce":3360352974}]},"public_addr":"172.21.15.16:6812/3360352974","cluster_addr":"172.21.15.16:6817/3360352974","heartbeat_back_addr":"172.21.15.16:6828/3360352974","heartbeat_front_addr":"172.21.15.16:6823/3360352974","state":["exists","up"]},{"osd":4,"uuid":"88260730-1482-4fbd-a40f-cb1124e9df4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6801","nonce":1856844856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6803","nonce":1856844856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6807","nonce":1856844856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6805","nonce":1856844856}]},"public_addr":"172.21.15.89:6801/1856844856","cluster_addr":"172.21.15.89:6803/1856844856","heartbeat_back_addr":"172.21.15.89:6807/1856844856","heartbeat_front_addr":"172.21.15.89:6805/1856844856","state":["exists","up"]},{"osd":5,"uuid":"4cb2c00c-fae0-4dd8-9d27-ec4f0093ccf7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6814","nonce":50395527}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6817","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6820","nonce":50395527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6831","nonce":50395527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6823","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6826","nonce":50395527}]},"public_addr":"172.21.15.89:6814/50395527","cluster_addr":"172.21.15.89:6820/50395527","heartbeat_back_addr":"172.21.15.89:6831/50395527","heartbeat_front_addr":"172.21.15.89:6826/50395527","state":["exists","up"]},{"osd":6,"uuid":"81ce0c19-2554-4cae-acbe-66a05e9e9b9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6809","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6812","nonce":3470471752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6815","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6818","nonce":3470471752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6827","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6829","nonce":3470471752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6821","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6824","nonce":3470471752}]},"public_addr":"172.21.15.89:6812/3470471752","cluster_addr":"172.21.15.89:6818/3470471752","heartbeat_back_addr":"172.21.15.89:6829/3470471752","heartbeat_front_addr":"172.21.15.89:6824/3470471752","state":["exists","up"]},{"osd":7,"uuid":"85e4313b-d37d-442e-8042-153bf20a847d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6811","nonce":1796778630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6813","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6816","nonce":1796778630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6825","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6828","nonce":1796778630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6819","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6822","nonce":1796778630}]},"public_addr":"172.21.15.89:6811/1796778630","cluster_addr":"172.21.15.89:6816/1796778630","heartbeat_back_addr":"172.21.15.89:6828/1796778630","heartbeat_front_addr":"172.21.15.89:6822/1796778630","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.439088+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.529970+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.680512+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:21.044161+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.582111+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.536558+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.409522+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.590229+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.2","mappings":[{"from":3,"to":2}]},{"pgid":"3.7","mappings":[{"from":3,"to":2}]},{"pgid":"3.d","mappings":[{"from":5,"to":1}]},{"pgid":"3.22","mappings":[{"from":5,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-31T15:04:20.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:20.224+0000 7f1ea1200640 1 -- 172.21.15.16:0/2926491984 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f1e78068380 msgr2=0x7f1e7806a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:20.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:20.224+0000 7f1ea1200640 1 --2- 172.21.15.16:0/2926491984 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f1e78068380 0x7f1e7806a840 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f1e84009580 tx=0x7f1e84009190 comp rx=0 tx=0).stop 2024-03-31T15:04:20.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:20.224+0000 7f1ea1200640 1 -- 172.21.15.16:0/2926491984 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1e9c0b23d0 msgr2=0x7f1e9c150ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:20.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:20.224+0000 7f1ea1200640 1 --2- 172.21.15.16:0/2926491984 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1e9c0b23d0 0x7f1e9c150ac0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f1e8c00c450 tx=0x7f1e8c00c910 comp rx=0 tx=0).stop 2024-03-31T15:04:20.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:20.224+0000 7f1ea1200640 1 -- 172.21.15.16:0/2926491984 shutdown_connections 2024-03-31T15:04:20.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:20.224+0000 7f1ea1200640 1 --2- 172.21.15.16:0/2926491984 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f1e78068380 0x7f1e7806a840 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:20.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:20.224+0000 7f1ea1200640 1 --2- 172.21.15.16:0/2926491984 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1e9c15eee0 0x7f1e9c14ff70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:20.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:20.224+0000 7f1ea1200640 1 --2- 172.21.15.16:0/2926491984 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f1e9c15b3d0 0x7f1e9c152f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:20.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:20.224+0000 7f1ea1200640 1 --2- 172.21.15.16:0/2926491984 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1e9c0b23d0 0x7f1e9c150ac0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:20.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:20.224+0000 7f1ea1200640 1 -- 172.21.15.16:0/2926491984 >> 172.21.15.16:0/2926491984 conn(0x7f1e9c06b620 msgr2=0x7f1e9c0b1cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:04:20.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:20.224+0000 7f1ea1200640 1 -- 172.21.15.16:0/2926491984 shutdown_connections 2024-03-31T15:04:20.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:20.224+0000 7f1ea1200640 1 -- 172.21.15.16:0/2926491984 wait complete. 2024-03-31T15:04:20.236 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-31T15:03:22.550787+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-31T15:03:34.488425+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-31T15:03:35.299116+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-31T15:04:20.236 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-31T15:04:20.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:20.384+0000 7f1331600640 1 Processor -- start 2024-03-31T15:04:20.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:20.384+0000 7f1331600640 1 -- start start 2024-03-31T15:04:20.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:20.388+0000 7f1331600640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f132c15c390 0x7f132c0cfb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:20.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:20.388+0000 7f1331600640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f132c0cd750 con 0x7f132c06fbc0 2024-03-31T15:04:20.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:20.388+0000 7f1331600640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f132c0cda30 con 0x7f132c0d2b40 2024-03-31T15:04:20.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:20.388+0000 7f1331600640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f132c15c770 con 0x7f132c15c390 2024-03-31T15:04:20.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:20.388+0000 7f1330c00640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f132c06fbc0 0x7f132c070070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:44360/0 (socket says 172.21.15.16:44360) 2024-03-31T15:04:20.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:20.388+0000 7f1330c00640 1 -- 172.21.15.16:0/1308504920 learned_addr learned my addr 172.21.15.16:0/1308504920 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:04:20.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:20.388+0000 7f132be00640 1 --2- 172.21.15.16:0/1308504920 >> v2:172.21.15.89:3300/0 conn(0x7f132c15c390 0x7f132c0cfb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:20.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:20.388+0000 7f132b400640 1 -- 172.21.15.16:0/1308504920 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2253137395 0 0) 0x7f132c0cd750 con 0x7f132c06fbc0 2024-03-31T15:04:20.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:20.388+0000 7f132b400640 1 -- 172.21.15.16:0/1308504920 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1310003530 con 0x7f132c06fbc0 2024-03-31T15:04:20.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:20.388+0000 7f132b400640 1 -- 172.21.15.16:0/1308504920 <== mon.1 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3602432043 0 0) 0x7f132c0cda30 con 0x7f132c0d2b40 2024-03-31T15:04:20.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:20.388+0000 7f132b400640 1 -- 172.21.15.16:0/1308504920 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f132c0cd750 con 0x7f132c0d2b40 2024-03-31T15:04:20.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:20.388+0000 7f132be00640 1 -- 172.21.15.16:0/1308504920 >> v1:172.21.15.89:6789/0 conn(0x7f132c0d2b40 legacy=0x7f132c0d00a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:20.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:20.388+0000 7f132be00640 1 -- 172.21.15.16:0/1308504920 >> v1:172.21.15.16:6789/0 conn(0x7f132c06fbc0 legacy=0x7f132c070070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:20.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:20.388+0000 7f132be00640 1 -- 172.21.15.16:0/1308504920 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f132c0cdd10 con 0x7f132c15c390 2024-03-31T15:04:20.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:20.392+0000 7f132be00640 1 --2- 172.21.15.16:0/1308504920 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f132c15c390 0x7f132c0cfb60 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f131c0097c0 tx=0x7f131c01c4a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1c933fd49c3ad189 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:20.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:20.392+0000 7f132b400640 1 -- 172.21.15.16:0/1308504920 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f131c0b0070 con 0x7f132c15c390 2024-03-31T15:04:20.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:20.392+0000 7f132b400640 1 -- 172.21.15.16:0/1308504920 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f131c0ab600 con 0x7f132c15c390 2024-03-31T15:04:20.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:20.392+0000 7f132b400640 1 -- 172.21.15.16:0/1308504920 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f131c0ab9d0 con 0x7f132c15c390 2024-03-31T15:04:20.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:20.392+0000 7f1331600640 1 -- 172.21.15.16:0/1308504920 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f132c15c390 msgr2=0x7f132c0cfb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:20.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:20.392+0000 7f1331600640 1 --2- 172.21.15.16:0/1308504920 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f132c15c390 0x7f132c0cfb60 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f131c0097c0 tx=0x7f131c01c4a0 comp rx=0 tx=0).stop 2024-03-31T15:04:20.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:20.392+0000 7f1331600640 1 -- 172.21.15.16:0/1308504920 shutdown_connections 2024-03-31T15:04:20.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:20.392+0000 7f1331600640 1 --2- 172.21.15.16:0/1308504920 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f132c15c390 0x7f132c0cfb60 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:20.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:20.392+0000 7f1331600640 1 -- 172.21.15.16:0/1308504920 >> 172.21.15.16:0/1308504920 conn(0x7f132c0d1f90 msgr2=0x7f132c0d23a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:04:20.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:20.392+0000 7f1331600640 1 -- 172.21.15.16:0/1308504920 shutdown_connections 2024-03-31T15:04:20.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:20.392+0000 7f1331600640 1 -- 172.21.15.16:0/1308504920 wait complete. 2024-03-31T15:04:20.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:20.392+0000 7f1331600640 1 Processor -- start 2024-03-31T15:04:20.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:20.392+0000 7f1331600640 1 -- start start 2024-03-31T15:04:20.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:20.396+0000 7f1331600640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f132c06fbc0 0x7f132c15f450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:20.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:20.396+0000 7f1331600640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f132c0d2b40 0x7f132c0b6c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:20.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:20.396+0000 7f1331600640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f132c15c390 0x7f132c15ca00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:20.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:20.396+0000 7f1331600640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f132c169fe0 con 0x7f132c06fbc0 2024-03-31T15:04:20.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:20.396+0000 7f1331600640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f132c169e60 con 0x7f132c15c390 2024-03-31T15:04:20.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:20.396+0000 7f1331600640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f132c16a160 con 0x7f132c0d2b40 2024-03-31T15:04:20.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:20.396+0000 7f132be00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f132c06fbc0 0x7f132c15f450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:20.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:20.396+0000 7f132be00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f132c06fbc0 0x7f132c15f450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49196/0 (socket says 172.21.15.16:49196) 2024-03-31T15:04:20.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:20.396+0000 7f132be00640 1 -- 172.21.15.16:0/2344862451 learned_addr learned my addr 172.21.15.16:0/2344862451 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:04:20.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:20.396+0000 7f1330c00640 1 --2- 172.21.15.16:0/2344862451 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f132c15c390 0x7f132c15ca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:20.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:20.396+0000 7f1323400640 1 --2- 172.21.15.16:0/2344862451 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f132c0d2b40 0x7f132c0b6c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:20.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:20.396+0000 7f132be00640 1 -- 172.21.15.16:0/2344862451 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f132c0d2b40 msgr2=0x7f132c0b6c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:20.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:20.396+0000 7f132be00640 1 --2- 172.21.15.16:0/2344862451 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f132c0d2b40 0x7f132c0b6c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:20.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:20.396+0000 7f132be00640 1 -- 172.21.15.16:0/2344862451 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f132c15c390 msgr2=0x7f132c15ca00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:20.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:20.396+0000 7f132be00640 1 --2- 172.21.15.16:0/2344862451 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f132c15c390 0x7f132c15ca00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:20.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:20.396+0000 7f132be00640 1 -- 172.21.15.16:0/2344862451 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f132c15cf40 con 0x7f132c06fbc0 2024-03-31T15:04:20.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:20.396+0000 7f132be00640 1 --2- 172.21.15.16:0/2344862451 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f132c06fbc0 0x7f132c15f450 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f131c0ae040 tx=0x7f131c01ace0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:20.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:20.396+0000 7f1328c00640 1 -- 172.21.15.16:0/2344862451 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f131c0b0050 con 0x7f132c06fbc0 2024-03-31T15:04:20.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:20.396+0000 7f1328c00640 1 -- 172.21.15.16:0/2344862451 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f131c0bc070 con 0x7f132c06fbc0 2024-03-31T15:04:20.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:20.396+0000 7f1328c00640 1 -- 172.21.15.16:0/2344862451 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f131c006e20 con 0x7f132c06fbc0 2024-03-31T15:04:20.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:20.396+0000 7f1331600640 1 -- 172.21.15.16:0/2344862451 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f132c15e3b0 con 0x7f132c06fbc0 2024-03-31T15:04:20.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:20.396+0000 7f1331600640 1 -- 172.21.15.16:0/2344862451 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f132c15e970 con 0x7f132c06fbc0 2024-03-31T15:04:20.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:20.396+0000 7f1331600640 1 -- 172.21.15.16:0/2344862451 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f132c04f8c0 con 0x7f132c06fbc0 2024-03-31T15:04:20.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:20.396+0000 7f1328c00640 1 -- 172.21.15.16:0/2344862451 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f131c0b5870 con 0x7f132c06fbc0 2024-03-31T15:04:20.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:20.400+0000 7f1328c00640 1 --2- 172.21.15.16:0/2344862451 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f13080683b0 0x7f130806a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:20.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:20.400+0000 7f1328c00640 1 -- 172.21.15.16:0/2344862451 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f131c120970 con 0x7f132c06fbc0 2024-03-31T15:04:20.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:20.400+0000 7f1323400640 1 --2- 172.21.15.16:0/2344862451 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f13080683b0 0x7f130806a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:20.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:20.400+0000 7f1323400640 1 --2- 172.21.15.16:0/2344862451 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f13080683b0 0x7f130806a870 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f132c156bd0 tx=0x7f1314009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:20.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:20.404+0000 7f1328c00640 1 -- 172.21.15.16:0/2344862451 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f131c0c5020 con 0x7f132c06fbc0 2024-03-31T15:04:20.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:20.568+0000 7f1331600640 1 -- 172.21.15.16:0/2344862451 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f132c159dd0 con 0x7f132c06fbc0 2024-03-31T15:04:20.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:20.568+0000 7f1328c00640 1 -- 172.21.15.16:0/2344862451 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v23) ==== 93+0+10 (secure 0 0 0) 0x7f131c016090 con 0x7f132c06fbc0 2024-03-31T15:04:20.563 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2024-03-31T15:04:20.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:20.572+0000 7f1331600640 1 -- 172.21.15.16:0/2344862451 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f13080683b0 msgr2=0x7f130806a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:20.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:20.572+0000 7f1331600640 1 --2- 172.21.15.16:0/2344862451 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f13080683b0 0x7f130806a870 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f132c156bd0 tx=0x7f1314009040 comp rx=0 tx=0).stop 2024-03-31T15:04:20.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:20.572+0000 7f1331600640 1 -- 172.21.15.16:0/2344862451 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f132c06fbc0 msgr2=0x7f132c15f450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:20.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:20.572+0000 7f1331600640 1 --2- 172.21.15.16:0/2344862451 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f132c06fbc0 0x7f132c15f450 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f131c0ae040 tx=0x7f131c01ace0 comp rx=0 tx=0).stop 2024-03-31T15:04:20.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:20.572+0000 7f1331600640 1 -- 172.21.15.16:0/2344862451 shutdown_connections 2024-03-31T15:04:20.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:20.572+0000 7f1331600640 1 --2- 172.21.15.16:0/2344862451 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f13080683b0 0x7f130806a870 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:20.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:20.572+0000 7f1331600640 1 --2- 172.21.15.16:0/2344862451 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f132c15c390 0x7f132c15ca00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:20.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:20.572+0000 7f1331600640 1 --2- 172.21.15.16:0/2344862451 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f132c0d2b40 0x7f132c0b6c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:20.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:20.572+0000 7f1331600640 1 --2- 172.21.15.16:0/2344862451 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f132c06fbc0 0x7f132c15f450 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:20.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:20.572+0000 7f1331600640 1 -- 172.21.15.16:0/2344862451 >> 172.21.15.16:0/2344862451 conn(0x7f132c0d1f90 msgr2=0x7f132c06df60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:04:20.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:20.572+0000 7f1331600640 1 -- 172.21.15.16:0/2344862451 shutdown_connections 2024-03-31T15:04:20.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:20.572+0000 7f1331600640 1 -- 172.21.15.16:0/2344862451 wait complete. 2024-03-31T15:04:20.583 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-31T15:04:20.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:20.732+0000 7f7148c00640 1 Processor -- start 2024-03-31T15:04:20.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:20.732+0000 7f7148c00640 1 -- start start 2024-03-31T15:04:20.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:20.732+0000 7f7148c00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f71441411b0 0x7f714406f920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:20.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:20.732+0000 7f7148c00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f7144141590 0x7f714406fe60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:20.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:20.732+0000 7f7148c00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f7144143c30 0x7f71441440a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:20.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:20.732+0000 7f7148c00640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f7144162910 con 0x7f71441411b0 2024-03-31T15:04:20.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:20.732+0000 7f7148c00640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f7144144980 con 0x7f7144143c30 2024-03-31T15:04:20.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:20.732+0000 7f7148c00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f7144161fe0 con 0x7f7144141590 2024-03-31T15:04:20.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:20.732+0000 7f7141e00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f71441411b0 0x7f714406f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:20.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:20.732+0000 7f7141e00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f71441411b0 0x7f714406f920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49210/0 (socket says 172.21.15.16:49210) 2024-03-31T15:04:20.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:20.732+0000 7f7141e00640 1 -- 172.21.15.16:0/3482889657 learned_addr learned my addr 172.21.15.16:0/3482889657 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:04:20.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:20.732+0000 7f713be00640 1 --2- 172.21.15.16:0/3482889657 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f7144141590 0x7f714406fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:20.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:20.732+0000 7f7142800640 1 --2- 172.21.15.16:0/3482889657 >> v2:172.21.15.89:3300/0 conn(0x7f7144143c30 0x7f71441440a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:20.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:20.732+0000 7f7141e00640 1 -- 172.21.15.16:0/3482889657 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f7144141590 msgr2=0x7f714406fe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:04:20.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:20.736+0000 7f7141e00640 1 --2- 172.21.15.16:0/3482889657 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f7144141590 0x7f714406fe60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:20.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:20.736+0000 7f7141e00640 1 -- 172.21.15.16:0/3482889657 >> v2:172.21.15.89:3300/0 conn(0x7f7144143c30 msgr2=0x7f71441440a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:04:20.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:20.736+0000 7f7141e00640 1 --2- 172.21.15.16:0/3482889657 >> v2:172.21.15.89:3300/0 conn(0x7f7144143c30 0x7f71441440a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:20.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:20.736+0000 7f7141e00640 1 -- 172.21.15.16:0/3482889657 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71441622c0 con 0x7f71441411b0 2024-03-31T15:04:20.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:20.736+0000 7f7141e00640 1 --2- 172.21.15.16:0/3482889657 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f71441411b0 0x7f714406f920 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f71440b1c00 tx=0x7f713401c4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b640348b09751950 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:20.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:20.736+0000 7f7141400640 1 -- 172.21.15.16:0/3482889657 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71340ab640 con 0x7f71441411b0 2024-03-31T15:04:20.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:20.736+0000 7f7141400640 1 -- 172.21.15.16:0/3482889657 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f71340b0070 con 0x7f71441411b0 2024-03-31T15:04:20.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:20.736+0000 7f7141400640 1 -- 172.21.15.16:0/3482889657 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71340aba00 con 0x7f71441411b0 2024-03-31T15:04:20.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:20.736+0000 7f7148c00640 1 -- 172.21.15.16:0/3482889657 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f71441411b0 msgr2=0x7f714406f920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:20.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:20.736+0000 7f7148c00640 1 --2- 172.21.15.16:0/3482889657 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f71441411b0 0x7f714406f920 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f71440b1c00 tx=0x7f713401c4f0 comp rx=0 tx=0).stop 2024-03-31T15:04:20.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:20.736+0000 7f7148c00640 1 -- 172.21.15.16:0/3482889657 shutdown_connections 2024-03-31T15:04:20.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:20.736+0000 7f7148c00640 1 --2- 172.21.15.16:0/3482889657 >> v2:172.21.15.89:3300/0 conn(0x7f7144143c30 0x7f71441440a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:20.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:20.736+0000 7f7148c00640 1 --2- 172.21.15.16:0/3482889657 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f7144141590 0x7f714406fe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:20.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:20.736+0000 7f7148c00640 1 --2- 172.21.15.16:0/3482889657 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f71441411b0 0x7f714406f920 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:20.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:20.736+0000 7f7148c00640 1 -- 172.21.15.16:0/3482889657 >> 172.21.15.16:0/3482889657 conn(0x7f7144067930 msgr2=0x7f71440b3ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:04:20.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:20.736+0000 7f7148c00640 1 -- 172.21.15.16:0/3482889657 shutdown_connections 2024-03-31T15:04:20.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:20.736+0000 7f7148c00640 1 -- 172.21.15.16:0/3482889657 wait complete. 2024-03-31T15:04:20.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:20.740+0000 7f7148c00640 1 Processor -- start 2024-03-31T15:04:20.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:20.740+0000 7f7148c00640 1 -- start start 2024-03-31T15:04:20.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:20.740+0000 7f7148c00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f71441411b0 0x7f7144156710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:20.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:20.740+0000 7f7148c00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f7144141590 0x7f7144161400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:20.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:20.740+0000 7f7148c00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7144143c30 0x7f714406c730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:20.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:20.740+0000 7f7148c00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f71440717f0 con 0x7f7144143c30 2024-03-31T15:04:20.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:20.740+0000 7f7148c00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f7144071670 con 0x7f71441411b0 2024-03-31T15:04:20.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:20.740+0000 7f7148c00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f7144071970 con 0x7f7144141590 2024-03-31T15:04:20.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:20.740+0000 7f7142800640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7144143c30 0x7f714406c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:20.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:20.740+0000 7f7142800640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7144143c30 0x7f714406c730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49220/0 (socket says 172.21.15.16:49220) 2024-03-31T15:04:20.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:20.740+0000 7f7142800640 1 -- 172.21.15.16:0/1961945006 learned_addr learned my addr 172.21.15.16:0/1961945006 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:04:20.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:20.740+0000 7f7141e00640 1 --2- 172.21.15.16:0/1961945006 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f71441411b0 0x7f7144156710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:20.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:20.740+0000 7f713be00640 1 --2- 172.21.15.16:0/1961945006 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f7144141590 0x7f7144161400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:20.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:20.740+0000 7f7142800640 1 -- 172.21.15.16:0/1961945006 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f7144141590 msgr2=0x7f7144161400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:20.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:20.740+0000 7f7142800640 1 --2- 172.21.15.16:0/1961945006 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f7144141590 0x7f7144161400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:20.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:20.740+0000 7f7142800640 1 -- 172.21.15.16:0/1961945006 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f71441411b0 msgr2=0x7f7144156710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:20.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:20.740+0000 7f7142800640 1 --2- 172.21.15.16:0/1961945006 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f71441411b0 0x7f7144156710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:20.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:20.740+0000 7f7142800640 1 -- 172.21.15.16:0/1961945006 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f714415eaa0 con 0x7f7144143c30 2024-03-31T15:04:20.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:20.740+0000 7f7142800640 1 --2- 172.21.15.16:0/1961945006 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7144143c30 0x7f714406c730 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f713c00efc0 tx=0x7f713c00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:20.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:20.740+0000 7f713a000640 1 -- 172.21.15.16:0/1961945006 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f713c018020 con 0x7f7144143c30 2024-03-31T15:04:20.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:20.740+0000 7f713a000640 1 -- 172.21.15.16:0/1961945006 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f713c014480 con 0x7f7144143c30 2024-03-31T15:04:20.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:20.740+0000 7f7148c00640 1 -- 172.21.15.16:0/1961945006 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7144161b70 con 0x7f7144143c30 2024-03-31T15:04:20.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:20.740+0000 7f713a000640 1 -- 172.21.15.16:0/1961945006 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f713c014870 con 0x7f7144143c30 2024-03-31T15:04:20.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:20.740+0000 7f7148c00640 1 -- 172.21.15.16:0/1961945006 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f714413cae0 con 0x7f7144143c30 2024-03-31T15:04:20.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:20.744+0000 7f7148c00640 1 -- 172.21.15.16:0/1961945006 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7144069950 con 0x7f7144143c30 2024-03-31T15:04:20.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:20.744+0000 7f713a000640 1 -- 172.21.15.16:0/1961945006 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f713c01b020 con 0x7f7144143c30 2024-03-31T15:04:20.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:20.744+0000 7f713a000640 1 --2- 172.21.15.16:0/1961945006 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f7120068380 0x7f712006a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:20.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:20.744+0000 7f713a000640 1 -- 172.21.15.16:0/1961945006 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f713c0872c0 con 0x7f7144143c30 2024-03-31T15:04:20.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:20.744+0000 7f7141e00640 1 --2- 172.21.15.16:0/1961945006 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f7120068380 0x7f712006a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:20.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:20.744+0000 7f7141e00640 1 --2- 172.21.15.16:0/1961945006 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f7120068380 0x7f712006a840 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f71340af000 tx=0x7f71340a7000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:20.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:20.748+0000 7f713a000640 1 -- 172.21.15.16:0/1961945006 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f713c022070 con 0x7f7144143c30 2024-03-31T15:04:20.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:20.916+0000 7f7148c00640 1 -- 172.21.15.16:0/1961945006 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f7144141ca0 con 0x7f7144143c30 2024-03-31T15:04:20.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:20.916+0000 7f713a000640 1 -- 172.21.15.16:0/1961945006 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v23) ==== 104+0+11 (secure 0 0 0) 0x7f713c010040 con 0x7f7144143c30 2024-03-31T15:04:20.912 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2024-03-31T15:04:20.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:20.920+0000 7f7148c00640 1 -- 172.21.15.16:0/1961945006 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f7120068380 msgr2=0x7f712006a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:20.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:20.920+0000 7f7148c00640 1 --2- 172.21.15.16:0/1961945006 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f7120068380 0x7f712006a840 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f71340af000 tx=0x7f71340a7000 comp rx=0 tx=0).stop 2024-03-31T15:04:20.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:20.920+0000 7f7148c00640 1 -- 172.21.15.16:0/1961945006 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7144143c30 msgr2=0x7f714406c730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:20.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:20.920+0000 7f7148c00640 1 --2- 172.21.15.16:0/1961945006 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7144143c30 0x7f714406c730 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f713c00efc0 tx=0x7f713c00c490 comp rx=0 tx=0).stop 2024-03-31T15:04:20.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:20.920+0000 7f7148c00640 1 -- 172.21.15.16:0/1961945006 shutdown_connections 2024-03-31T15:04:20.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:20.920+0000 7f7148c00640 1 --2- 172.21.15.16:0/1961945006 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f7120068380 0x7f712006a840 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:20.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:20.920+0000 7f7148c00640 1 --2- 172.21.15.16:0/1961945006 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7144143c30 0x7f714406c730 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:20.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:20.920+0000 7f7148c00640 1 --2- 172.21.15.16:0/1961945006 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f7144141590 0x7f7144161400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:20.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:20.920+0000 7f7148c00640 1 --2- 172.21.15.16:0/1961945006 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f71441411b0 0x7f7144156710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:20.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:20.920+0000 7f7148c00640 1 -- 172.21.15.16:0/1961945006 >> 172.21.15.16:0/1961945006 conn(0x7f7144067930 msgr2=0x7f714415da70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:04:20.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:20.920+0000 7f7148c00640 1 -- 172.21.15.16:0/1961945006 shutdown_connections 2024-03-31T15:04:20.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:20.920+0000 7f7148c00640 1 -- 172.21.15.16:0/1961945006 wait complete. 2024-03-31T15:04:20.932 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-31T15:04:21.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:21.080+0000 7f7636800640 1 Processor -- start 2024-03-31T15:04:21.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:21.080+0000 7f7636800640 1 -- start start 2024-03-31T15:04:21.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:21.084+0000 7f7636800640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f76300d0a60 0x7f76300cea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:21.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:21.084+0000 7f7636800640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f76300cef80 0x7f76301419c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:21.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:21.084+0000 7f7636800640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f763015ca70 con 0x7f763006f920 2024-03-31T15:04:21.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:21.084+0000 7f7636800640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f763013f670 con 0x7f76300d0a60 2024-03-31T15:04:21.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:21.084+0000 7f7636800640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f763015cd50 con 0x7f76300cef80 2024-03-31T15:04:21.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:21.084+0000 7f7626a00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f76300d0a60 0x7f76300cea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:21.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:21.084+0000 7f762fe00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f76300cef80 0x7f76301419c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:21.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:21.084+0000 7f7626a00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f76300d0a60 0x7f76300cea40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.16:49094/0 (socket says 172.21.15.16:49094) 2024-03-31T15:04:21.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:21.084+0000 7f762f400640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f763006f920 0x7f763006fd00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.16:52784/0 (socket says 172.21.15.16:52784) 2024-03-31T15:04:21.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:21.084+0000 7f7626a00640 1 -- 172.21.15.16:0/3457083063 learned_addr learned my addr 172.21.15.16:0/3457083063 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:04:21.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:21.084+0000 7f762fe00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f76300cef80 0x7f76301419c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3301/0 says I am v2:172.21.15.16:37652/0 (socket says 172.21.15.16:37652) 2024-03-31T15:04:21.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:21.084+0000 7f762ea00640 1 -- 172.21.15.16:0/3457083063 <== mon.1 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2108021445 0 0) 0x7f763015ca70 con 0x7f763006f920 2024-03-31T15:04:21.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:21.084+0000 7f7626a00640 1 -- 172.21.15.16:0/3457083063 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f76300cef80 msgr2=0x7f76301419c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:04:21.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:21.084+0000 7f7626a00640 1 --2- 172.21.15.16:0/3457083063 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f76300cef80 0x7f76301419c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:21.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:21.084+0000 7f7626a00640 1 -- 172.21.15.16:0/3457083063 >> v1:172.21.15.89:6789/0 conn(0x7f763006f920 legacy=0x7f763006fd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:21.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:21.084+0000 7f7626a00640 1 -- 172.21.15.16:0/3457083063 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f763015d030 con 0x7f76300d0a60 2024-03-31T15:04:21.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:21.084+0000 7f7626a00640 1 --2- 172.21.15.16:0/3457083063 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f76300d0a60 0x7f76300cea40 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f7618009870 tx=0x7f761801c560 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c31acc5d9c564425 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:21.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:21.088+0000 7f762ea00640 1 -- 172.21.15.16:0/3457083063 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76180b2070 con 0x7f76300d0a60 2024-03-31T15:04:21.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:21.088+0000 7f762ea00640 1 -- 172.21.15.16:0/3457083063 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f76180a9070 con 0x7f76300d0a60 2024-03-31T15:04:21.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:21.088+0000 7f762ea00640 1 -- 172.21.15.16:0/3457083063 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76180ad880 con 0x7f76300d0a60 2024-03-31T15:04:21.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:21.088+0000 7f7636800640 1 -- 172.21.15.16:0/3457083063 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f76300d0a60 msgr2=0x7f76300cea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:21.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:21.088+0000 7f7636800640 1 --2- 172.21.15.16:0/3457083063 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f76300d0a60 0x7f76300cea40 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f7618009870 tx=0x7f761801c560 comp rx=0 tx=0).stop 2024-03-31T15:04:21.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:21.088+0000 7f7636800640 1 -- 172.21.15.16:0/3457083063 shutdown_connections 2024-03-31T15:04:21.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:21.088+0000 7f7636800640 1 --2- 172.21.15.16:0/3457083063 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f76300d0a60 0x7f76300cea40 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:21.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:21.088+0000 7f7636800640 1 --2- 172.21.15.16:0/3457083063 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f76300cef80 0x7f76301419c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:21.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:21.088+0000 7f7636800640 1 -- 172.21.15.16:0/3457083063 >> 172.21.15.16:0/3457083063 conn(0x7f7630067930 msgr2=0x7f76300b3ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:04:21.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:21.088+0000 7f7636800640 1 -- 172.21.15.16:0/3457083063 shutdown_connections 2024-03-31T15:04:21.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:21.088+0000 7f7636800640 1 -- 172.21.15.16:0/3457083063 wait complete. 2024-03-31T15:04:21.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:21.088+0000 7f7636800640 1 Processor -- start 2024-03-31T15:04:21.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:21.088+0000 7f7636800640 1 -- start start 2024-03-31T15:04:21.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:21.088+0000 7f7636800640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f763006f920 0x7f7630140e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:21.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:21.088+0000 7f7636800640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f76300cef80 0x7f763013e8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:21.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:21.088+0000 7f7636800640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f76300d0a60 0x7f763013ee20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:21.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:21.088+0000 7f7636800640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f7630169230 con 0x7f76300cef80 2024-03-31T15:04:21.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:21.088+0000 7f7636800640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f76301690b0 con 0x7f76300d0a60 2024-03-31T15:04:21.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:21.088+0000 7f7636800640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f76301693b0 con 0x7f763006f920 2024-03-31T15:04:21.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:21.088+0000 7f762f400640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f763006f920 0x7f7630140e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:21.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:21.088+0000 7f762fe00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f76300d0a60 0x7f763013ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:21.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:21.088+0000 7f7626a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f76300cef80 0x7f763013e8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:21.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:21.088+0000 7f762fe00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f76300d0a60 0x7f763013ee20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.16:49098/0 (socket says 172.21.15.16:49098) 2024-03-31T15:04:21.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:21.088+0000 7f762fe00640 1 -- 172.21.15.16:0/347670535 learned_addr learned my addr 172.21.15.16:0/347670535 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:04:21.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:21.088+0000 7f762f400640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f763006f920 0x7f7630140e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3301/0 says I am v2:172.21.15.16:37660/0 (socket says 172.21.15.16:37660) 2024-03-31T15:04:21.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:21.092+0000 7f762f400640 1 -- 172.21.15.16:0/347670535 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f76300d0a60 msgr2=0x7f763013ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:21.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:21.092+0000 7f762f400640 1 --2- 172.21.15.16:0/347670535 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f76300d0a60 0x7f763013ee20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:21.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:21.092+0000 7f762f400640 1 -- 172.21.15.16:0/347670535 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f76300cef80 msgr2=0x7f763013e8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:21.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:21.092+0000 7f762f400640 1 --2- 172.21.15.16:0/347670535 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f76300cef80 0x7f763013e8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:21.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:21.092+0000 7f762f400640 1 -- 172.21.15.16:0/347670535 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f763006e5d0 con 0x7f763006f920 2024-03-31T15:04:21.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:21.092+0000 7f762f400640 1 --2- 172.21.15.16:0/347670535 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f763006f920 0x7f7630140e20 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f762000c7a0 tx=0x7f762000cc60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:21.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:21.092+0000 7f7627e00640 1 -- 172.21.15.16:0/347670535 <== mon.2 v2:172.21.15.89:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7620018070 con 0x7f763006f920 2024-03-31T15:04:21.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:21.092+0000 7f7627e00640 1 -- 172.21.15.16:0/347670535 <== mon.2 v2:172.21.15.89:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f762000a640 con 0x7f763006f920 2024-03-31T15:04:21.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:21.092+0000 7f7627e00640 1 -- 172.21.15.16:0/347670535 <== mon.2 v2:172.21.15.89:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f762000aa30 con 0x7f763006f920 2024-03-31T15:04:21.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:21.092+0000 7f7636800640 1 -- 172.21.15.16:0/347670535 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f763015d850 con 0x7f763006f920 2024-03-31T15:04:21.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:21.092+0000 7f7636800640 1 -- 172.21.15.16:0/347670535 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f763015de10 con 0x7f763006f920 2024-03-31T15:04:21.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:21.092+0000 7f7636800640 1 -- 172.21.15.16:0/347670535 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7630069950 con 0x7f763006f920 2024-03-31T15:04:21.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:21.092+0000 7f7627e00640 1 -- 172.21.15.16:0/347670535 <== mon.2 v2:172.21.15.89:3301/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f762000abd0 con 0x7f763006f920 2024-03-31T15:04:21.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:21.092+0000 7f7627e00640 1 --2- 172.21.15.16:0/347670535 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f760c068380 0x7f760c06a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:21.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:21.092+0000 7f7627e00640 1 -- 172.21.15.16:0/347670535 <== mon.2 v2:172.21.15.89:3301/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f7620086070 con 0x7f763006f920 2024-03-31T15:04:21.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:21.096+0000 7f7626a00640 1 --2- 172.21.15.16:0/347670535 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f760c068380 0x7f760c06a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:21.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:21.096+0000 7f7626a00640 1 --2- 172.21.15.16:0/347670535 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f760c068380 0x7f760c06a840 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f76180af000 tx=0x7f76180a7000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:21.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:21.100+0000 7f7627e00640 1 -- 172.21.15.16:0/347670535 <== mon.2 v2:172.21.15.89:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f7620055ec0 con 0x7f763006f920 2024-03-31T15:04:21.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:21.264+0000 7f7636800640 1 -- 172.21.15.16:0/347670535 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f76300b4a20 con 0x7f763006f920 2024-03-31T15:04:21.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:21.264+0000 7f7627e00640 1 -- 172.21.15.16:0/347670535 <== mon.2 v2:172.21.15.89:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v23) ==== 100+0+11 (secure 0 0 0) 0x7f762005a1c0 con 0x7f763006f920 2024-03-31T15:04:21.261 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 64 2024-03-31T15:04:21.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:21.268+0000 7f7636800640 1 -- 172.21.15.16:0/347670535 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f760c068380 msgr2=0x7f760c06a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:21.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:21.268+0000 7f7636800640 1 --2- 172.21.15.16:0/347670535 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f760c068380 0x7f760c06a840 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f76180af000 tx=0x7f76180a7000 comp rx=0 tx=0).stop 2024-03-31T15:04:21.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:21.268+0000 7f7636800640 1 -- 172.21.15.16:0/347670535 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f763006f920 msgr2=0x7f7630140e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:21.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:21.268+0000 7f7636800640 1 --2- 172.21.15.16:0/347670535 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f763006f920 0x7f7630140e20 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f762000c7a0 tx=0x7f762000cc60 comp rx=0 tx=0).stop 2024-03-31T15:04:21.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:21.268+0000 7f7636800640 1 -- 172.21.15.16:0/347670535 shutdown_connections 2024-03-31T15:04:21.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:21.268+0000 7f7636800640 1 --2- 172.21.15.16:0/347670535 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f760c068380 0x7f760c06a840 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:21.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:21.268+0000 7f7636800640 1 --2- 172.21.15.16:0/347670535 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f76300d0a60 0x7f763013ee20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:21.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:21.268+0000 7f7636800640 1 --2- 172.21.15.16:0/347670535 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f76300cef80 0x7f763013e8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:21.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:21.268+0000 7f7636800640 1 --2- 172.21.15.16:0/347670535 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f763006f920 0x7f7630140e20 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:21.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:21.268+0000 7f7636800640 1 -- 172.21.15.16:0/347670535 >> 172.21.15.16:0/347670535 conn(0x7f7630067930 msgr2=0x7f76300d20a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:04:21.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:21.268+0000 7f7636800640 1 -- 172.21.15.16:0/347670535 shutdown_connections 2024-03-31T15:04:21.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:21.268+0000 7f7636800640 1 -- 172.21.15.16:0/347670535 wait complete. 2024-03-31T15:04:21.281 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-31T15:04:21.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:21.428+0000 7f737c800640 1 Processor -- start 2024-03-31T15:04:21.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:21.428+0000 7f737c800640 1 -- start start 2024-03-31T15:04:21.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:21.428+0000 7f737c800640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f737406dbc0 0x7f737415f650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:21.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:21.428+0000 7f737c800640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f73740cb440 0x7f737406f920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:21.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:21.428+0000 7f737c800640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f73740cbe90 con 0x7f73740cb8c0 2024-03-31T15:04:21.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:21.428+0000 7f737c800640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f737415fb90 con 0x7f73740cb440 2024-03-31T15:04:21.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:21.428+0000 7f737c800640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f737415f1c0 con 0x7f737406dbc0 2024-03-31T15:04:21.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:21.428+0000 7f7379400640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f73740cb440 0x7f737406f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:21.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:21.428+0000 7f737a800640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f73740cb8c0 0x7f7374162180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:44376/0 (socket says 172.21.15.16:44376) 2024-03-31T15:04:21.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:21.428+0000 7f737a800640 1 -- 172.21.15.16:0/2310030555 learned_addr learned my addr 172.21.15.16:0/2310030555 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:04:21.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:21.428+0000 7f7379e00640 1 --2- 172.21.15.16:0/2310030555 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f737406dbc0 0x7f737415f650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:21.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:21.428+0000 7f7378a00640 1 -- 172.21.15.16:0/2310030555 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 838742487 0 0) 0x7f73740cbe90 con 0x7f73740cb8c0 2024-03-31T15:04:21.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:21.428+0000 7f7378a00640 1 -- 172.21.15.16:0/2310030555 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f735c003530 con 0x7f73740cb8c0 2024-03-31T15:04:21.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:21.428+0000 7f7379e00640 1 -- 172.21.15.16:0/2310030555 >> v2:172.21.15.16:3300/0 conn(0x7f73740cb440 msgr2=0x7f737406f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:04:21.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:21.428+0000 7f7379e00640 1 --2- 172.21.15.16:0/2310030555 >> v2:172.21.15.16:3300/0 conn(0x7f73740cb440 0x7f737406f920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:21.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:21.428+0000 7f7379e00640 1 -- 172.21.15.16:0/2310030555 >> v1:172.21.15.16:6789/0 conn(0x7f73740cb8c0 legacy=0x7f7374162180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:21.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:21.428+0000 7f7379e00640 1 -- 172.21.15.16:0/2310030555 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73740cc170 con 0x7f737406dbc0 2024-03-31T15:04:21.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:21.428+0000 7f7379400640 1 --2- 172.21.15.16:0/2310030555 >> v2:172.21.15.16:3300/0 conn(0x7f73740cb440 0x7f737406f920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:04:21.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:21.432+0000 7f7379e00640 1 --2- 172.21.15.16:0/2310030555 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f737406dbc0 0x7f737415f650 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f73680097c0 tx=0x7f736801c3e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a9e27cf6016cbc8f server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:21.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:21.432+0000 7f7378a00640 1 -- 172.21.15.16:0/2310030555 <== mon.2 v2:172.21.15.89:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73680ab600 con 0x7f737406dbc0 2024-03-31T15:04:21.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:21.432+0000 7f7378a00640 1 -- 172.21.15.16:0/2310030555 <== mon.2 v2:172.21.15.89:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f73680b0070 con 0x7f737406dbc0 2024-03-31T15:04:21.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:21.432+0000 7f7378a00640 1 -- 172.21.15.16:0/2310030555 <== mon.2 v2:172.21.15.89:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73680ab9a0 con 0x7f737406dbc0 2024-03-31T15:04:21.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:21.432+0000 7f737c800640 1 -- 172.21.15.16:0/2310030555 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f737406dbc0 msgr2=0x7f737415f650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:21.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:21.432+0000 7f737c800640 1 --2- 172.21.15.16:0/2310030555 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f737406dbc0 0x7f737415f650 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f73680097c0 tx=0x7f736801c3e0 comp rx=0 tx=0).stop 2024-03-31T15:04:21.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:21.432+0000 7f737c800640 1 -- 172.21.15.16:0/2310030555 shutdown_connections 2024-03-31T15:04:21.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:21.432+0000 7f737c800640 1 --2- 172.21.15.16:0/2310030555 >> v2:172.21.15.16:3300/0 conn(0x7f73740cb440 0x7f737406f920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:21.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:21.432+0000 7f737c800640 1 --2- 172.21.15.16:0/2310030555 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f737406dbc0 0x7f737415f650 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:21.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:21.432+0000 7f737c800640 1 -- 172.21.15.16:0/2310030555 >> 172.21.15.16:0/2310030555 conn(0x7f7374067330 msgr2=0x7f7374067740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:04:21.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:21.432+0000 7f737c800640 1 -- 172.21.15.16:0/2310030555 shutdown_connections 2024-03-31T15:04:21.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:21.432+0000 7f737c800640 1 -- 172.21.15.16:0/2310030555 wait complete. 2024-03-31T15:04:21.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:21.436+0000 7f737c800640 1 Processor -- start 2024-03-31T15:04:21.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:21.436+0000 7f737c800640 1 -- start start 2024-03-31T15:04:21.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:21.436+0000 7f737c800640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f737406dbc0 0x7f73741567d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:21.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:21.436+0000 7f737c800640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f73740cb440 0x7f737413d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:21.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:21.436+0000 7f737c800640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f73740cb8c0 0x7f7374156e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:21.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:21.436+0000 7f737c800640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f7374162ca0 con 0x7f73740cb440 2024-03-31T15:04:21.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:21.436+0000 7f737c800640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f7374070270 con 0x7f73740cb8c0 2024-03-31T15:04:21.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:21.436+0000 7f737c800640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f737406c950 con 0x7f737406dbc0 2024-03-31T15:04:21.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:21.436+0000 7f7379400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f73740cb440 0x7f737413d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:21.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:21.436+0000 7f7379e00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f737406dbc0 0x7f73741567d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:21.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:21.436+0000 7f7379400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f73740cb440 0x7f737413d3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49246/0 (socket says 172.21.15.16:49246) 2024-03-31T15:04:21.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:21.436+0000 7f7379400640 1 -- 172.21.15.16:0/127808691 learned_addr learned my addr 172.21.15.16:0/127808691 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:04:21.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:21.436+0000 7f737a800640 1 --2- 172.21.15.16:0/127808691 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f73740cb8c0 0x7f7374156e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:21.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:21.436+0000 7f7379400640 1 -- 172.21.15.16:0/127808691 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f737406dbc0 msgr2=0x7f73741567d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:21.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:21.436+0000 7f7379400640 1 --2- 172.21.15.16:0/127808691 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f737406dbc0 0x7f73741567d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:21.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:21.436+0000 7f7379400640 1 -- 172.21.15.16:0/127808691 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f73740cb8c0 msgr2=0x7f7374156e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:21.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:21.436+0000 7f7379400640 1 --2- 172.21.15.16:0/127808691 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f73740cb8c0 0x7f7374156e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:21.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:21.436+0000 7f7379400640 1 -- 172.21.15.16:0/127808691 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73740b6070 con 0x7f73740cb440 2024-03-31T15:04:21.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:21.436+0000 7f7379400640 1 --2- 172.21.15.16:0/127808691 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f73740cb440 0x7f737413d3e0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f736400e9b0 tx=0x7f736400ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:21.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:21.436+0000 7f736e000640 1 -- 172.21.15.16:0/127808691 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7364017070 con 0x7f73740cb440 2024-03-31T15:04:21.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:21.436+0000 7f736e000640 1 -- 172.21.15.16:0/127808691 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f736400ca00 con 0x7f73740cb440 2024-03-31T15:04:21.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:21.436+0000 7f736e000640 1 -- 172.21.15.16:0/127808691 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f736400cdd0 con 0x7f73740cb440 2024-03-31T15:04:21.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:21.436+0000 7f737c800640 1 -- 172.21.15.16:0/127808691 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f73741413a0 con 0x7f73740cb440 2024-03-31T15:04:21.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:21.436+0000 7f737c800640 1 -- 172.21.15.16:0/127808691 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7374141960 con 0x7f73740cb440 2024-03-31T15:04:21.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:21.440+0000 7f737c800640 1 -- 172.21.15.16:0/127808691 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7374069950 con 0x7f73740cb440 2024-03-31T15:04:21.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:21.440+0000 7f736e000640 1 -- 172.21.15.16:0/127808691 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f736401a070 con 0x7f73740cb440 2024-03-31T15:04:21.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:21.440+0000 7f736e000640 1 --2- 172.21.15.16:0/127808691 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f7358068380 0x7f735806a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:21.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:21.440+0000 7f736e000640 1 -- 172.21.15.16:0/127808691 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f7364086910 con 0x7f73740cb440 2024-03-31T15:04:21.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:21.440+0000 7f7379e00640 1 --2- 172.21.15.16:0/127808691 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f7358068380 0x7f735806a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:21.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:21.440+0000 7f7379e00640 1 --2- 172.21.15.16:0/127808691 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f7358068380 0x7f735806a840 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f73680af000 tx=0x7f73680a7000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:21.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:21.444+0000 7f736e000640 1 -- 172.21.15.16:0/127808691 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f73640566d0 con 0x7f73740cb440 2024-03-31T15:04:21.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:21.620+0000 7f737c800640 1 -- 172.21.15.16:0/127808691 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f73740cf380 con 0x7f73740cb440 2024-03-31T15:04:21.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:21.620+0000 7f736e000640 1 -- 172.21.15.16:0/127808691 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v23) ==== 80+0+0 (secure 0 0 0) 0x7f736405a9d0 con 0x7f73740cb440 2024-03-31T15:04:21.615 INFO:teuthology.orchestra.run.smithi016.stderr:listed 0 entries 2024-03-31T15:04:21.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:21.624+0000 7f737c800640 1 -- 172.21.15.16:0/127808691 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f7358068380 msgr2=0x7f735806a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:21.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:21.624+0000 7f737c800640 1 --2- 172.21.15.16:0/127808691 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f7358068380 0x7f735806a840 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f73680af000 tx=0x7f73680a7000 comp rx=0 tx=0).stop 2024-03-31T15:04:21.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:21.624+0000 7f737c800640 1 -- 172.21.15.16:0/127808691 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f73740cb440 msgr2=0x7f737413d3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:21.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:21.624+0000 7f737c800640 1 --2- 172.21.15.16:0/127808691 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f73740cb440 0x7f737413d3e0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f736400e9b0 tx=0x7f736400ee70 comp rx=0 tx=0).stop 2024-03-31T15:04:21.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:21.624+0000 7f737c800640 1 -- 172.21.15.16:0/127808691 shutdown_connections 2024-03-31T15:04:21.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:21.624+0000 7f737c800640 1 --2- 172.21.15.16:0/127808691 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f7358068380 0x7f735806a840 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:21.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:21.624+0000 7f737c800640 1 --2- 172.21.15.16:0/127808691 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f73740cb8c0 0x7f7374156e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:21.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:21.624+0000 7f737c800640 1 --2- 172.21.15.16:0/127808691 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f73740cb440 0x7f737413d3e0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:21.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:21.624+0000 7f737c800640 1 --2- 172.21.15.16:0/127808691 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f737406dbc0 0x7f73741567d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:21.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:21.624+0000 7f737c800640 1 -- 172.21.15.16:0/127808691 >> 172.21.15.16:0/127808691 conn(0x7f7374067330 msgr2=0x7f737415d010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:04:21.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:21.624+0000 7f737c800640 1 -- 172.21.15.16:0/127808691 shutdown_connections 2024-03-31T15:04:21.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:21.624+0000 7f737c800640 1 -- 172.21.15.16:0/127808691 wait complete. 2024-03-31T15:04:21.635 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi016.front.sepia.ceph.com... 2024-03-31T15:04:21.635 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:04:21.636 DEBUG:teuthology.orchestra.run.smithi016:> sudo fusermount -u /home/ubuntu/cephtest/mnt.2 2024-03-31T15:04:21.656 INFO:teuthology.orchestra.run:waiting for 300 2024-03-31T15:04:21.656 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi016.front.sepia.ceph.com 2024-03-31T15:04:21.656 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:04:21.656 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-03-31T15:04:21.709 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-03-31T15:04:21.709 DEBUG:teuthology.orchestra.run.smithi089:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-31T15:04:21.718 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-03-31T15:04:21.718 DEBUG:teuthology.orchestra.run.smithi089:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-31T15:04:21.770 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-31T15:04:21.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:21.916+0000 7f1ad8e00640 1 Processor -- start 2024-03-31T15:04:21.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:21.916+0000 7f1ad8e00640 1 -- start start 2024-03-31T15:04:21.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:21.916+0000 7f1ad8e00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f1ad4145d10 0x7f1ad40cf790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:21.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:21.916+0000 7f1ad8e00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f1ad40b23d0 0x7f1ad415cd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:21.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:21.916+0000 7f1ad8e00640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1ad4160e70 con 0x7f1ad40728b0 2024-03-31T15:04:21.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:21.916+0000 7f1ad8e00640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f1ad41602f0 con 0x7f1ad4145d10 2024-03-31T15:04:21.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:21.916+0000 7f1ad8e00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f1ad41605d0 con 0x7f1ad40b23d0 2024-03-31T15:04:21.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:21.916+0000 7f1ad1e00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f1ad4145d10 0x7f1ad40cf790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:21.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:21.916+0000 7f1ac9400640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f1ad40728b0 0x7f1ad40b1cc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:44392/0 (socket says 172.21.15.16:44392) 2024-03-31T15:04:21.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:21.916+0000 7f1ac9400640 1 -- 172.21.15.16:0/1083188706 learned_addr learned my addr 172.21.15.16:0/1083188706 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:04:21.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:21.916+0000 7f1ad2800640 1 --2- 172.21.15.16:0/1083188706 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f1ad40b23d0 0x7f1ad415cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:21.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:21.920+0000 7f1ad1e00640 1 -- 172.21.15.16:0/1083188706 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f1ad40b23d0 msgr2=0x7f1ad415cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:04:21.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:21.920+0000 7f1ad1e00640 1 --2- 172.21.15.16:0/1083188706 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f1ad40b23d0 0x7f1ad415cd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:21.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:21.920+0000 7f1ad1e00640 1 -- 172.21.15.16:0/1083188706 >> v1:172.21.15.16:6789/0 conn(0x7f1ad40728b0 legacy=0x7f1ad40b1cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:21.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:21.920+0000 7f1ad1e00640 1 -- 172.21.15.16:0/1083188706 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ad40cfcd0 con 0x7f1ad4145d10 2024-03-31T15:04:21.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:21.920+0000 7f1ad1400640 1 -- 172.21.15.16:0/1083188706 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1431846619 0 0) 0x7f1ad4160e70 con 0x7f1ad40728b0 2024-03-31T15:04:21.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:21.920+0000 7f1ad1e00640 1 --2- 172.21.15.16:0/1083188706 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1ad4145d10 0x7f1ad40cf790 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f1ac40097c0 tx=0x7f1ac401c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a19f3fd735cf4db server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:21.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:21.920+0000 7f1ad1400640 1 -- 172.21.15.16:0/1083188706 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ac40b0070 con 0x7f1ad4145d10 2024-03-31T15:04:21.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:21.920+0000 7f1ad1400640 1 -- 172.21.15.16:0/1083188706 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1ac40ab600 con 0x7f1ad4145d10 2024-03-31T15:04:21.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:21.920+0000 7f1ad1400640 1 -- 172.21.15.16:0/1083188706 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ac40ab9d0 con 0x7f1ad4145d10 2024-03-31T15:04:21.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:21.920+0000 7f1ad8e00640 1 -- 172.21.15.16:0/1083188706 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1ad4145d10 msgr2=0x7f1ad40cf790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:21.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:21.920+0000 7f1ad8e00640 1 --2- 172.21.15.16:0/1083188706 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1ad4145d10 0x7f1ad40cf790 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f1ac40097c0 tx=0x7f1ac401c4a0 comp rx=0 tx=0).stop 2024-03-31T15:04:21.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:21.920+0000 7f1ad8e00640 1 -- 172.21.15.16:0/1083188706 shutdown_connections 2024-03-31T15:04:21.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:21.920+0000 7f1ad8e00640 1 --2- 172.21.15.16:0/1083188706 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f1ad40b23d0 0x7f1ad415cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:21.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:21.920+0000 7f1ad8e00640 1 --2- 172.21.15.16:0/1083188706 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1ad4145d10 0x7f1ad40cf790 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:21.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:21.920+0000 7f1ad8e00640 1 -- 172.21.15.16:0/1083188706 >> 172.21.15.16:0/1083188706 conn(0x7f1ad406b620 msgr2=0x7f1ad406ba30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:04:21.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:21.924+0000 7f1ad8e00640 1 -- 172.21.15.16:0/1083188706 shutdown_connections 2024-03-31T15:04:21.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:21.924+0000 7f1ad8e00640 1 -- 172.21.15.16:0/1083188706 wait complete. 2024-03-31T15:04:21.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:21.924+0000 7f1ad8e00640 1 Processor -- start 2024-03-31T15:04:21.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:21.924+0000 7f1ad8e00640 1 -- start start 2024-03-31T15:04:21.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:21.924+0000 7f1ad8e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1ad40728b0 0x7f1ad41621d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:21.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:21.924+0000 7f1ad8e00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f1ad40b23d0 0x7f1ad415bcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:21.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:21.924+0000 7f1ad8e00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1ad4145d10 0x7f1ad4144d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:21.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:21.924+0000 7f1ad8e00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f1ad40cf4d0 con 0x7f1ad40728b0 2024-03-31T15:04:21.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:21.924+0000 7f1ad8e00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f1ad40cf350 con 0x7f1ad4145d10 2024-03-31T15:04:21.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:21.924+0000 7f1ad8e00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f1ad4072d00 con 0x7f1ad40b23d0 2024-03-31T15:04:21.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:21.924+0000 7f1ad1e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1ad40728b0 0x7f1ad41621d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:21.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:21.924+0000 7f1ad1e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1ad40728b0 0x7f1ad41621d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49268/0 (socket says 172.21.15.16:49268) 2024-03-31T15:04:21.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:21.924+0000 7f1ad1e00640 1 -- 172.21.15.16:0/895768065 learned_addr learned my addr 172.21.15.16:0/895768065 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:04:21.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:21.924+0000 7f1ac9400640 1 --2- 172.21.15.16:0/895768065 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f1ad40b23d0 0x7f1ad415bcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:21.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:21.924+0000 7f1ad2800640 1 --2- 172.21.15.16:0/895768065 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1ad4145d10 0x7f1ad4144d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:21.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:21.924+0000 7f1ad1e00640 1 -- 172.21.15.16:0/895768065 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f1ad40b23d0 msgr2=0x7f1ad415bcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:21.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:21.924+0000 7f1ad1e00640 1 --2- 172.21.15.16:0/895768065 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f1ad40b23d0 0x7f1ad415bcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:21.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:21.924+0000 7f1ad1e00640 1 -- 172.21.15.16:0/895768065 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1ad4145d10 msgr2=0x7f1ad4144d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:21.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:21.924+0000 7f1ad1e00640 1 --2- 172.21.15.16:0/895768065 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1ad4145d10 0x7f1ad4144d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:21.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:21.924+0000 7f1ad1e00640 1 -- 172.21.15.16:0/895768065 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ad415c200 con 0x7f1ad40728b0 2024-03-31T15:04:21.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:21.924+0000 7f1ad1e00640 1 --2- 172.21.15.16:0/895768065 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1ad40728b0 0x7f1ad41621d0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f1ac401a0d0 tx=0x7f1ac4019240 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:21.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:21.924+0000 7f1acaa00640 1 -- 172.21.15.16:0/895768065 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ac40b0040 con 0x7f1ad40728b0 2024-03-31T15:04:21.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:21.924+0000 7f1acaa00640 1 -- 172.21.15.16:0/895768065 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1ac40ba070 con 0x7f1ad40728b0 2024-03-31T15:04:21.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:21.924+0000 7f1acaa00640 1 -- 172.21.15.16:0/895768065 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ac4006e20 con 0x7f1ad40728b0 2024-03-31T15:04:21.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:21.924+0000 7f1ad8e00640 1 -- 172.21.15.16:0/895768065 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1ad4161130 con 0x7f1ad40728b0 2024-03-31T15:04:21.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:21.928+0000 7f1ad8e00640 1 -- 172.21.15.16:0/895768065 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1ad41616f0 con 0x7f1ad40728b0 2024-03-31T15:04:21.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:21.928+0000 7f1ad8e00640 1 -- 172.21.15.16:0/895768065 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1ad4069950 con 0x7f1ad40728b0 2024-03-31T15:04:21.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:21.928+0000 7f1acaa00640 1 -- 172.21.15.16:0/895768065 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f1ac40b4630 con 0x7f1ad40728b0 2024-03-31T15:04:21.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:21.928+0000 7f1acaa00640 1 --2- 172.21.15.16:0/895768065 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f1ab0068380 0x7f1ab006a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:21.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:21.928+0000 7f1acaa00640 1 -- 172.21.15.16:0/895768065 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f1ac4120630 con 0x7f1ad40728b0 2024-03-31T15:04:21.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:21.928+0000 7f1ac9400640 1 --2- 172.21.15.16:0/895768065 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f1ab0068380 0x7f1ab006a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:21.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:21.928+0000 7f1ac9400640 1 --2- 172.21.15.16:0/895768065 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f1ab0068380 0x7f1ab006a840 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f1ad4143d70 tx=0x7f1abc007df0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:21.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:21.932+0000 7f1acaa00640 1 -- 172.21.15.16:0/895768065 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f1ac4016090 con 0x7f1ad40728b0 2024-03-31T15:04:22.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:22.100+0000 7f1ad8e00640 1 -- 172.21.15.16:0/895768065 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f1ad415fb60 con 0x7f1ad40728b0 2024-03-31T15:04:22.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:22.100+0000 7f1acaa00640 1 -- 172.21.15.16:0/895768065 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v23) ==== 74+0+16780 (secure 0 0 0) 0x7f1ac40f0ad0 con 0x7f1ad40728b0 2024-03-31T15:04:22.095 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:04:22.095 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":23,"fsid":"286eb5da-fb47-4550-b290-d9519c13b9a5","created":"2024-03-31T15:03:11.438952+0000","modified":"2024-03-31T15:04:18.087947+0000","last_up_change":"2024-03-31T15:03:22.885535+0000","last_in_change":"2024-03-31T15:03:13.941667+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-31T15:03:22.550787+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-31T15:03:34.488425+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-31T15:03:35.299116+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"92a66450-ef19-40d8-ac3d-d760b23b0bda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6801","nonce":932712244}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6803","nonce":932712244}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6807","nonce":932712244}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6805","nonce":932712244}]},"public_addr":"172.21.15.16:6801/932712244","cluster_addr":"172.21.15.16:6803/932712244","heartbeat_back_addr":"172.21.15.16:6807/932712244","heartbeat_front_addr":"172.21.15.16:6805/932712244","state":["exists","up"]},{"osd":1,"uuid":"da9d1f2b-1968-4959-b843-faa231988e92","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6810","nonce":1398737533}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6811","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6813","nonce":1398737533}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6821","nonce":1398737533}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6815","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6816","nonce":1398737533}]},"public_addr":"172.21.15.16:6810/1398737533","cluster_addr":"172.21.15.16:6813/1398737533","heartbeat_back_addr":"172.21.15.16:6821/1398737533","heartbeat_front_addr":"172.21.15.16:6816/1398737533","state":["exists","up"]},{"osd":2,"uuid":"01a73662-9a21-4b1f-bcbb-42b9636a4638","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6819","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6822","nonce":3018526646}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6825","nonce":3018526646}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6831","nonce":3018526646}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6827","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6829","nonce":3018526646}]},"public_addr":"172.21.15.16:6822/3018526646","cluster_addr":"172.21.15.16:6825/3018526646","heartbeat_back_addr":"172.21.15.16:6831/3018526646","heartbeat_front_addr":"172.21.15.16:6829/3018526646","state":["exists","up"]},{"osd":3,"uuid":"c7771fd3-e18e-4500-ac38-23cf10ebcd75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6809","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6812","nonce":3360352974}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6817","nonce":3360352974}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6828","nonce":3360352974}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6823","nonce":3360352974}]},"public_addr":"172.21.15.16:6812/3360352974","cluster_addr":"172.21.15.16:6817/3360352974","heartbeat_back_addr":"172.21.15.16:6828/3360352974","heartbeat_front_addr":"172.21.15.16:6823/3360352974","state":["exists","up"]},{"osd":4,"uuid":"88260730-1482-4fbd-a40f-cb1124e9df4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6801","nonce":1856844856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6803","nonce":1856844856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6807","nonce":1856844856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6805","nonce":1856844856}]},"public_addr":"172.21.15.89:6801/1856844856","cluster_addr":"172.21.15.89:6803/1856844856","heartbeat_back_addr":"172.21.15.89:6807/1856844856","heartbeat_front_addr":"172.21.15.89:6805/1856844856","state":["exists","up"]},{"osd":5,"uuid":"4cb2c00c-fae0-4dd8-9d27-ec4f0093ccf7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6814","nonce":50395527}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6817","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6820","nonce":50395527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6831","nonce":50395527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6823","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6826","nonce":50395527}]},"public_addr":"172.21.15.89:6814/50395527","cluster_addr":"172.21.15.89:6820/50395527","heartbeat_back_addr":"172.21.15.89:6831/50395527","heartbeat_front_addr":"172.21.15.89:6826/50395527","state":["exists","up"]},{"osd":6,"uuid":"81ce0c19-2554-4cae-acbe-66a05e9e9b9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6809","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6812","nonce":3470471752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6815","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6818","nonce":3470471752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6827","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6829","nonce":3470471752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6821","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6824","nonce":3470471752}]},"public_addr":"172.21.15.89:6812/3470471752","cluster_addr":"172.21.15.89:6818/3470471752","heartbeat_back_addr":"172.21.15.89:6829/3470471752","heartbeat_front_addr":"172.21.15.89:6824/3470471752","state":["exists","up"]},{"osd":7,"uuid":"85e4313b-d37d-442e-8042-153bf20a847d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6811","nonce":1796778630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6813","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6816","nonce":1796778630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6825","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6828","nonce":1796778630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6819","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6822","nonce":1796778630}]},"public_addr":"172.21.15.89:6811/1796778630","cluster_addr":"172.21.15.89:6816/1796778630","heartbeat_back_addr":"172.21.15.89:6828/1796778630","heartbeat_front_addr":"172.21.15.89:6822/1796778630","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.439088+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.529970+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.680512+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:21.044161+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.582111+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.536558+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.409522+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.590229+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.2","mappings":[{"from":3,"to":2}]},{"pgid":"3.7","mappings":[{"from":3,"to":2}]},{"pgid":"3.d","mappings":[{"from":5,"to":1}]},{"pgid":"3.22","mappings":[{"from":5,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-31T15:04:22.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:22.104+0000 7f1ad8e00640 1 -- 172.21.15.16:0/895768065 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f1ab0068380 msgr2=0x7f1ab006a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:22.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:22.104+0000 7f1ad8e00640 1 --2- 172.21.15.16:0/895768065 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f1ab0068380 0x7f1ab006a840 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f1ad4143d70 tx=0x7f1abc007df0 comp rx=0 tx=0).stop 2024-03-31T15:04:22.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:22.104+0000 7f1ad8e00640 1 -- 172.21.15.16:0/895768065 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1ad40728b0 msgr2=0x7f1ad41621d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:22.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:22.104+0000 7f1ad8e00640 1 --2- 172.21.15.16:0/895768065 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1ad40728b0 0x7f1ad41621d0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f1ac401a0d0 tx=0x7f1ac4019240 comp rx=0 tx=0).stop 2024-03-31T15:04:22.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:22.104+0000 7f1ad8e00640 1 -- 172.21.15.16:0/895768065 shutdown_connections 2024-03-31T15:04:22.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:22.104+0000 7f1ad8e00640 1 --2- 172.21.15.16:0/895768065 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f1ab0068380 0x7f1ab006a840 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:22.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:22.104+0000 7f1ad8e00640 1 --2- 172.21.15.16:0/895768065 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1ad4145d10 0x7f1ad4144d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:22.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:22.104+0000 7f1ad8e00640 1 --2- 172.21.15.16:0/895768065 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f1ad40b23d0 0x7f1ad415bcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:22.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:22.104+0000 7f1ad8e00640 1 --2- 172.21.15.16:0/895768065 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1ad40728b0 0x7f1ad41621d0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:22.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:22.104+0000 7f1ad8e00640 1 -- 172.21.15.16:0/895768065 >> 172.21.15.16:0/895768065 conn(0x7f1ad406b620 msgr2=0x7f1ad40cff50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:04:22.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:22.104+0000 7f1ad8e00640 1 -- 172.21.15.16:0/895768065 shutdown_connections 2024-03-31T15:04:22.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:22.104+0000 7f1ad8e00640 1 -- 172.21.15.16:0/895768065 wait complete. 2024-03-31T15:04:22.118 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-31T15:03:22.550787+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-31T15:03:34.488425+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-31T15:03:35.299116+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-31T15:04:22.119 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-31T15:04:22.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:22.268+0000 7ff201c00640 1 Processor -- start 2024-03-31T15:04:22.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:22.268+0000 7ff201c00640 1 -- start start 2024-03-31T15:04:22.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:22.268+0000 7ff201c00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff1fc06f920 0x7ff1fc06fd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:22.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:22.268+0000 7ff201c00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff1fc0d0a60 0x7ff1fc0cea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:22.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:22.268+0000 7ff201c00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7ff1fc0cef80 0x7ff1fc070e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:22.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:22.268+0000 7ff201c00640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7ff1fc13f670 con 0x7ff1fc06f920 2024-03-31T15:04:22.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:22.268+0000 7ff201c00640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7ff1fc070240 con 0x7ff1fc0cef80 2024-03-31T15:04:22.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:22.268+0000 7ff201c00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7ff1fc15ca70 con 0x7ff1fc0d0a60 2024-03-31T15:04:22.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:22.268+0000 7ff1faa00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff1fc06f920 0x7ff1fc06fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:22.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:22.268+0000 7ff1faa00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff1fc06f920 0x7ff1fc06fd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49284/0 (socket says 172.21.15.16:49284) 2024-03-31T15:04:22.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:22.268+0000 7ff1faa00640 1 -- 172.21.15.16:0/3234078777 learned_addr learned my addr 172.21.15.16:0/3234078777 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:04:22.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:22.268+0000 7ff1f2000640 1 --2- 172.21.15.16:0/3234078777 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff1fc0d0a60 0x7ff1fc0cea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:22.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:22.268+0000 7ff1fb400640 1 --2- 172.21.15.16:0/3234078777 >> v2:172.21.15.89:3300/0 conn(0x7ff1fc0cef80 0x7ff1fc070e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:22.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:22.268+0000 7ff1faa00640 1 -- 172.21.15.16:0/3234078777 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff1fc0d0a60 msgr2=0x7ff1fc0cea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:04:22.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:22.268+0000 7ff1faa00640 1 --2- 172.21.15.16:0/3234078777 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff1fc0d0a60 0x7ff1fc0cea40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:22.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:22.268+0000 7ff1faa00640 1 -- 172.21.15.16:0/3234078777 >> v2:172.21.15.89:3300/0 conn(0x7ff1fc0cef80 msgr2=0x7ff1fc070e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:04:22.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:22.268+0000 7ff1faa00640 1 --2- 172.21.15.16:0/3234078777 >> v2:172.21.15.89:3300/0 conn(0x7ff1fc0cef80 0x7ff1fc070e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:22.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:22.268+0000 7ff1faa00640 1 -- 172.21.15.16:0/3234078777 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1fc15cd50 con 0x7ff1fc06f920 2024-03-31T15:04:22.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:22.272+0000 7ff1faa00640 1 --2- 172.21.15.16:0/3234078777 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff1fc06f920 0x7ff1fc06fd00 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7ff1fc0b4cd0 tx=0x7ff1ec01c260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5dcd78f3424c400 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:22.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:22.272+0000 7ff1fa000640 1 -- 172.21.15.16:0/3234078777 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1ec0b0070 con 0x7ff1fc06f920 2024-03-31T15:04:22.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:22.272+0000 7ff1fa000640 1 -- 172.21.15.16:0/3234078777 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff1ec0ab600 con 0x7ff1fc06f920 2024-03-31T15:04:22.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:22.272+0000 7ff1fa000640 1 -- 172.21.15.16:0/3234078777 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1ec0ab9d0 con 0x7ff1fc06f920 2024-03-31T15:04:22.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:22.272+0000 7ff201c00640 1 -- 172.21.15.16:0/3234078777 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff1fc06f920 msgr2=0x7ff1fc06fd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:22.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:22.272+0000 7ff201c00640 1 --2- 172.21.15.16:0/3234078777 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff1fc06f920 0x7ff1fc06fd00 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7ff1fc0b4cd0 tx=0x7ff1ec01c260 comp rx=0 tx=0).stop 2024-03-31T15:04:22.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:22.272+0000 7ff201c00640 1 -- 172.21.15.16:0/3234078777 shutdown_connections 2024-03-31T15:04:22.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:22.272+0000 7ff201c00640 1 --2- 172.21.15.16:0/3234078777 >> v2:172.21.15.89:3300/0 conn(0x7ff1fc0cef80 0x7ff1fc070e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:22.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:22.272+0000 7ff201c00640 1 --2- 172.21.15.16:0/3234078777 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff1fc0d0a60 0x7ff1fc0cea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:22.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:22.272+0000 7ff201c00640 1 --2- 172.21.15.16:0/3234078777 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff1fc06f920 0x7ff1fc06fd00 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:22.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:22.272+0000 7ff201c00640 1 -- 172.21.15.16:0/3234078777 >> 172.21.15.16:0/3234078777 conn(0x7ff1fc067930 msgr2=0x7ff1fc0b3ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:04:22.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:22.272+0000 7ff201c00640 1 -- 172.21.15.16:0/3234078777 shutdown_connections 2024-03-31T15:04:22.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:22.272+0000 7ff201c00640 1 -- 172.21.15.16:0/3234078777 wait complete. 2024-03-31T15:04:22.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:22.276+0000 7ff201c00640 1 Processor -- start 2024-03-31T15:04:22.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:22.276+0000 7ff201c00640 1 -- start start 2024-03-31T15:04:22.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:22.276+0000 7ff201c00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff1fc06f920 0x7ff1fc1527b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:22.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:22.276+0000 7ff201c00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff1fc0cef80 0x7ff1fc0d0ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:22.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:22.276+0000 7ff201c00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff1fc0d0a60 0x7ff1fc0d13e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:22.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:22.276+0000 7ff201c00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7ff1fc1419c0 con 0x7ff1fc06f920 2024-03-31T15:04:22.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:22.276+0000 7ff201c00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7ff1fc0cf390 con 0x7ff1fc0cef80 2024-03-31T15:04:22.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:22.276+0000 7ff201c00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7ff1fc141b40 con 0x7ff1fc0d0a60 2024-03-31T15:04:22.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:22.276+0000 7ff1faa00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff1fc06f920 0x7ff1fc1527b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:22.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:22.276+0000 7ff1faa00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff1fc06f920 0x7ff1fc1527b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49288/0 (socket says 172.21.15.16:49288) 2024-03-31T15:04:22.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:22.276+0000 7ff1faa00640 1 -- 172.21.15.16:0/3145092728 learned_addr learned my addr 172.21.15.16:0/3145092728 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:04:22.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:22.276+0000 7ff1f2000640 1 --2- 172.21.15.16:0/3145092728 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff1fc0cef80 0x7ff1fc0d0ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:22.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:22.276+0000 7ff1fb400640 1 --2- 172.21.15.16:0/3145092728 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff1fc0d0a60 0x7ff1fc0d13e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:22.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:22.276+0000 7ff1faa00640 1 -- 172.21.15.16:0/3145092728 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff1fc0d0a60 msgr2=0x7ff1fc0d13e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:22.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:22.276+0000 7ff1faa00640 1 --2- 172.21.15.16:0/3145092728 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff1fc0d0a60 0x7ff1fc0d13e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:22.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:22.276+0000 7ff1faa00640 1 -- 172.21.15.16:0/3145092728 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff1fc0cef80 msgr2=0x7ff1fc0d0ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:22.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:22.276+0000 7ff1faa00640 1 --2- 172.21.15.16:0/3145092728 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff1fc0cef80 0x7ff1fc0d0ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:22.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:22.276+0000 7ff1faa00640 1 -- 172.21.15.16:0/3145092728 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1fc152cf0 con 0x7ff1fc06f920 2024-03-31T15:04:22.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:22.276+0000 7ff1faa00640 1 --2- 172.21.15.16:0/3145092728 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff1fc06f920 0x7ff1fc1527b0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7ff1ec0ae040 tx=0x7ff1ec019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:22.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:22.276+0000 7ff1f3400640 1 -- 172.21.15.16:0/3145092728 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1ec0b0040 con 0x7ff1fc06f920 2024-03-31T15:04:22.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:22.276+0000 7ff1f3400640 1 -- 172.21.15.16:0/3145092728 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff1ec0b9070 con 0x7ff1fc06f920 2024-03-31T15:04:22.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:22.276+0000 7ff1f3400640 1 -- 172.21.15.16:0/3145092728 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1ec0b4680 con 0x7ff1fc06f920 2024-03-31T15:04:22.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:22.276+0000 7ff201c00640 1 -- 172.21.15.16:0/3145092728 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff1fc141160 con 0x7ff1fc06f920 2024-03-31T15:04:22.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:22.276+0000 7ff201c00640 1 -- 172.21.15.16:0/3145092728 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff1fc06c7d0 con 0x7ff1fc06f920 2024-03-31T15:04:22.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:22.280+0000 7ff201c00640 1 -- 172.21.15.16:0/3145092728 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff1fc069950 con 0x7ff1fc06f920 2024-03-31T15:04:22.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:22.280+0000 7ff1f3400640 1 -- 172.21.15.16:0/3145092728 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7ff1ec0b4820 con 0x7ff1fc06f920 2024-03-31T15:04:22.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:22.280+0000 7ff1f3400640 1 --2- 172.21.15.16:0/3145092728 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7ff1d80683b0 0x7ff1d806a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:22.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:22.280+0000 7ff1f3400640 1 -- 172.21.15.16:0/3145092728 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7ff1ec1205b0 con 0x7ff1fc06f920 2024-03-31T15:04:22.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:22.280+0000 7ff1f2000640 1 --2- 172.21.15.16:0/3145092728 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7ff1d80683b0 0x7ff1d806a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:22.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:22.280+0000 7ff1f2000640 1 --2- 172.21.15.16:0/3145092728 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7ff1d80683b0 0x7ff1d806a870 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7ff1fc0b4050 tx=0x7ff1e4009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:22.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:22.284+0000 7ff1f3400640 1 -- 172.21.15.16:0/3145092728 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7ff1ec016110 con 0x7ff1fc06f920 2024-03-31T15:04:22.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:22.448+0000 7ff201c00640 1 -- 172.21.15.16:0/3145092728 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7ff1fc0b4a20 con 0x7ff1fc06f920 2024-03-31T15:04:22.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:22.448+0000 7ff1f3400640 1 -- 172.21.15.16:0/3145092728 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v23) ==== 93+0+10 (secure 0 0 0) 0x7ff1ec0f0ae0 con 0x7ff1fc06f920 2024-03-31T15:04:22.442 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2024-03-31T15:04:22.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:22.448+0000 7ff201c00640 1 -- 172.21.15.16:0/3145092728 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7ff1d80683b0 msgr2=0x7ff1d806a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:22.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:22.448+0000 7ff201c00640 1 --2- 172.21.15.16:0/3145092728 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7ff1d80683b0 0x7ff1d806a870 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7ff1fc0b4050 tx=0x7ff1e4009190 comp rx=0 tx=0).stop 2024-03-31T15:04:22.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:22.448+0000 7ff201c00640 1 -- 172.21.15.16:0/3145092728 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff1fc06f920 msgr2=0x7ff1fc1527b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:22.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:22.448+0000 7ff201c00640 1 --2- 172.21.15.16:0/3145092728 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff1fc06f920 0x7ff1fc1527b0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7ff1ec0ae040 tx=0x7ff1ec019000 comp rx=0 tx=0).stop 2024-03-31T15:04:22.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:22.452+0000 7ff201c00640 1 -- 172.21.15.16:0/3145092728 shutdown_connections 2024-03-31T15:04:22.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:22.452+0000 7ff201c00640 1 --2- 172.21.15.16:0/3145092728 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7ff1d80683b0 0x7ff1d806a870 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:22.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:22.452+0000 7ff201c00640 1 --2- 172.21.15.16:0/3145092728 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff1fc0d0a60 0x7ff1fc0d13e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:22.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:22.452+0000 7ff201c00640 1 --2- 172.21.15.16:0/3145092728 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff1fc0cef80 0x7ff1fc0d0ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:22.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:22.452+0000 7ff201c00640 1 --2- 172.21.15.16:0/3145092728 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff1fc06f920 0x7ff1fc1527b0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:22.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:22.452+0000 7ff201c00640 1 -- 172.21.15.16:0/3145092728 >> 172.21.15.16:0/3145092728 conn(0x7ff1fc067930 msgr2=0x7ff1fc155800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:04:22.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:22.452+0000 7ff201c00640 1 -- 172.21.15.16:0/3145092728 shutdown_connections 2024-03-31T15:04:22.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:22.452+0000 7ff201c00640 1 -- 172.21.15.16:0/3145092728 wait complete. 2024-03-31T15:04:22.464 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-31T15:04:22.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:22.612+0000 7fe5e6a00640 1 Processor -- start 2024-03-31T15:04:22.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:22.612+0000 7fe5e6a00640 1 -- start start 2024-03-31T15:04:22.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:22.616+0000 7fe5e6a00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fe5e006ca10 0x7fe5e006ce90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:22.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:22.616+0000 7fe5e6a00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fe5e00ced10 0x7fe5e00cf180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:22.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:22.616+0000 7fe5e6a00640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe5e00cd890 con 0x7fe5e00d07e0 2024-03-31T15:04:22.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:22.616+0000 7fe5e6a00640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7fe5e00cdb30 con 0x7fe5e006ca10 2024-03-31T15:04:22.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:22.616+0000 7fe5e6a00640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7fe5e00cde10 con 0x7fe5e00ced10 2024-03-31T15:04:22.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:22.616+0000 7fe5e4c00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fe5e006ca10 0x7fe5e006ce90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:22.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:22.616+0000 7fe5e5600640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fe5e00d07e0 0x7fe5e006e170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:44398/0 (socket says 172.21.15.16:44398) 2024-03-31T15:04:22.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:22.616+0000 7fe5e5600640 1 -- 172.21.15.16:0/1889904553 learned_addr learned my addr 172.21.15.16:0/1889904553 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:04:22.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:22.616+0000 7fe5e6000640 1 --2- 172.21.15.16:0/1889904553 >> v2:172.21.15.89:3300/0 conn(0x7fe5e00ced10 0x7fe5e00cf180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:22.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:22.616+0000 7fe5e4c00640 1 -- 172.21.15.16:0/1889904553 >> v2:172.21.15.89:3300/0 conn(0x7fe5e00ced10 msgr2=0x7fe5e00cf180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:04:22.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:22.616+0000 7fe5e4c00640 1 --2- 172.21.15.16:0/1889904553 >> v2:172.21.15.89:3300/0 conn(0x7fe5e00ced10 0x7fe5e00cf180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:22.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:22.616+0000 7fe5e4c00640 1 -- 172.21.15.16:0/1889904553 >> v1:172.21.15.16:6789/0 conn(0x7fe5e00d07e0 legacy=0x7fe5e006e170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:22.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:22.616+0000 7fe5e4c00640 1 -- 172.21.15.16:0/1889904553 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe5e00ce0f0 con 0x7fe5e006ca10 2024-03-31T15:04:22.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:22.616+0000 7fe5dfe00640 1 -- 172.21.15.16:0/1889904553 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1476150768 0 0) 0x7fe5e00cd890 con 0x7fe5e00d07e0 2024-03-31T15:04:22.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:22.620+0000 7fe5e4c00640 1 --2- 172.21.15.16:0/1889904553 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe5e006ca10 0x7fe5e006ce90 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fe5e006f400 tx=0x7fe5cc01c450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=909d5d35d431f37e server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:22.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:22.620+0000 7fe5dfe00640 1 -- 172.21.15.16:0/1889904553 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5cc0b0070 con 0x7fe5e006ca10 2024-03-31T15:04:22.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:22.620+0000 7fe5dfe00640 1 -- 172.21.15.16:0/1889904553 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe5cc0ab600 con 0x7fe5e006ca10 2024-03-31T15:04:22.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:22.620+0000 7fe5dfe00640 1 -- 172.21.15.16:0/1889904553 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5cc0ab9d0 con 0x7fe5e006ca10 2024-03-31T15:04:22.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:22.620+0000 7fe5e6a00640 1 -- 172.21.15.16:0/1889904553 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe5e006ca10 msgr2=0x7fe5e006ce90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:22.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:22.620+0000 7fe5e6a00640 1 --2- 172.21.15.16:0/1889904553 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe5e006ca10 0x7fe5e006ce90 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fe5e006f400 tx=0x7fe5cc01c450 comp rx=0 tx=0).stop 2024-03-31T15:04:22.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:22.620+0000 7fe5e6a00640 1 -- 172.21.15.16:0/1889904553 shutdown_connections 2024-03-31T15:04:22.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:22.620+0000 7fe5e6a00640 1 --2- 172.21.15.16:0/1889904553 >> v2:172.21.15.89:3300/0 conn(0x7fe5e00ced10 0x7fe5e00cf180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:22.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:22.620+0000 7fe5e6a00640 1 --2- 172.21.15.16:0/1889904553 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe5e006ca10 0x7fe5e006ce90 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:22.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:22.620+0000 7fe5e6a00640 1 -- 172.21.15.16:0/1889904553 >> 172.21.15.16:0/1889904553 conn(0x7fe5e00b2c90 msgr2=0x7fe5e00b30a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:04:22.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:22.620+0000 7fe5e6a00640 1 -- 172.21.15.16:0/1889904553 shutdown_connections 2024-03-31T15:04:22.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:22.620+0000 7fe5e6a00640 1 -- 172.21.15.16:0/1889904553 wait complete. 2024-03-31T15:04:22.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:22.620+0000 7fe5e6a00640 1 Processor -- start 2024-03-31T15:04:22.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:22.620+0000 7fe5e6a00640 1 -- start start 2024-03-31T15:04:22.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:22.620+0000 7fe5e6a00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe5e006ca10 0x7fe5e00b4650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:22.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:22.620+0000 7fe5e6a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe5e00ced10 0x7fe5e0153050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:22.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:22.620+0000 7fe5e6a00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fe5e00d07e0 0x7fe5e00adda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:22.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:22.620+0000 7fe5e6a00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fe5e006ed40 con 0x7fe5e00ced10 2024-03-31T15:04:22.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:22.620+0000 7fe5e6a00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fe5e006ebc0 con 0x7fe5e006ca10 2024-03-31T15:04:22.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:22.620+0000 7fe5e6a00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fe5e015c380 con 0x7fe5e00d07e0 2024-03-31T15:04:22.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:22.620+0000 7fe5e4c00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe5e00ced10 0x7fe5e0153050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:22.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:22.620+0000 7fe5e4c00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe5e00ced10 0x7fe5e0153050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49294/0 (socket says 172.21.15.16:49294) 2024-03-31T15:04:22.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:22.620+0000 7fe5e4c00640 1 -- 172.21.15.16:0/903251573 learned_addr learned my addr 172.21.15.16:0/903251573 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:04:22.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:22.624+0000 7fe5e5600640 1 --2- 172.21.15.16:0/903251573 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe5e006ca10 0x7fe5e00b4650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:22.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:22.624+0000 7fe5e6000640 1 --2- 172.21.15.16:0/903251573 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fe5e00d07e0 0x7fe5e00adda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:22.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:22.624+0000 7fe5e4c00640 1 -- 172.21.15.16:0/903251573 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fe5e00d07e0 msgr2=0x7fe5e00adda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:22.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:22.624+0000 7fe5e4c00640 1 --2- 172.21.15.16:0/903251573 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fe5e00d07e0 0x7fe5e00adda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:22.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:22.624+0000 7fe5e4c00640 1 -- 172.21.15.16:0/903251573 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe5e006ca10 msgr2=0x7fe5e00b4650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:22.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:22.624+0000 7fe5e4c00640 1 --2- 172.21.15.16:0/903251573 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe5e006ca10 0x7fe5e00b4650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:22.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:22.624+0000 7fe5e4c00640 1 -- 172.21.15.16:0/903251573 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe5e0153aa0 con 0x7fe5e00ced10 2024-03-31T15:04:22.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:22.624+0000 7fe5e4c00640 1 --2- 172.21.15.16:0/903251573 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe5e00ced10 0x7fe5e0153050 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fe5cc0ae040 tx=0x7fe5cc0191f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:22.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:22.624+0000 7fe5dd600640 1 -- 172.21.15.16:0/903251573 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5cc0b8020 con 0x7fe5e00ced10 2024-03-31T15:04:22.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:22.624+0000 7fe5dd600640 1 -- 172.21.15.16:0/903251573 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe5cc0b4410 con 0x7fe5e00ced10 2024-03-31T15:04:22.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:22.624+0000 7fe5dd600640 1 -- 172.21.15.16:0/903251573 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5cc0b0040 con 0x7fe5e00ced10 2024-03-31T15:04:22.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:22.624+0000 7fe5e6a00640 1 -- 172.21.15.16:0/903251573 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe5e0153d80 con 0x7fe5e00ced10 2024-03-31T15:04:22.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:22.624+0000 7fe5e6a00640 1 -- 172.21.15.16:0/903251573 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe5e00b3e50 con 0x7fe5e00ced10 2024-03-31T15:04:22.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:22.624+0000 7fe5e6a00640 1 -- 172.21.15.16:0/903251573 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe5e004f8c0 con 0x7fe5e00ced10 2024-03-31T15:04:22.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:22.624+0000 7fe5dd600640 1 -- 172.21.15.16:0/903251573 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fe5cc0bb020 con 0x7fe5e00ced10 2024-03-31T15:04:22.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:22.628+0000 7fe5dd600640 1 --2- 172.21.15.16:0/903251573 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fe5c0068380 0x7fe5c006a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:22.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:22.628+0000 7fe5dd600640 1 -- 172.21.15.16:0/903251573 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7fe5cc1204e0 con 0x7fe5e00ced10 2024-03-31T15:04:22.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:22.628+0000 7fe5e5600640 1 --2- 172.21.15.16:0/903251573 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fe5c0068380 0x7fe5c006a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:22.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:22.628+0000 7fe5e5600640 1 --2- 172.21.15.16:0/903251573 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fe5c0068380 0x7fe5c006a840 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fe5d4000f70 tx=0x7fe5d4007df0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:22.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:22.632+0000 7fe5dd600640 1 -- 172.21.15.16:0/903251573 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fe5cc0c2030 con 0x7fe5e00ced10 2024-03-31T15:04:22.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:22.796+0000 7fe5e6a00640 1 -- 172.21.15.16:0/903251573 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fe5e0072fb0 con 0x7fe5e00ced10 2024-03-31T15:04:22.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:22.796+0000 7fe5dd600640 1 -- 172.21.15.16:0/903251573 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v23) ==== 104+0+11 (secure 0 0 0) 0x7fe5cc016070 con 0x7fe5e00ced10 2024-03-31T15:04:22.792 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2024-03-31T15:04:22.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:22.800+0000 7fe5e6a00640 1 -- 172.21.15.16:0/903251573 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fe5c0068380 msgr2=0x7fe5c006a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:22.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:22.800+0000 7fe5e6a00640 1 --2- 172.21.15.16:0/903251573 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fe5c0068380 0x7fe5c006a840 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fe5d4000f70 tx=0x7fe5d4007df0 comp rx=0 tx=0).stop 2024-03-31T15:04:22.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:22.800+0000 7fe5e6a00640 1 -- 172.21.15.16:0/903251573 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe5e00ced10 msgr2=0x7fe5e0153050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:22.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:22.800+0000 7fe5e6a00640 1 --2- 172.21.15.16:0/903251573 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe5e00ced10 0x7fe5e0153050 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fe5cc0ae040 tx=0x7fe5cc0191f0 comp rx=0 tx=0).stop 2024-03-31T15:04:22.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:22.800+0000 7fe5e6a00640 1 -- 172.21.15.16:0/903251573 shutdown_connections 2024-03-31T15:04:22.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:22.800+0000 7fe5e6a00640 1 --2- 172.21.15.16:0/903251573 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fe5c0068380 0x7fe5c006a840 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:22.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:22.800+0000 7fe5e6a00640 1 --2- 172.21.15.16:0/903251573 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fe5e00d07e0 0x7fe5e00adda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:22.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:22.800+0000 7fe5e6a00640 1 --2- 172.21.15.16:0/903251573 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe5e00ced10 0x7fe5e0153050 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:22.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:22.800+0000 7fe5e6a00640 1 --2- 172.21.15.16:0/903251573 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe5e006ca10 0x7fe5e00b4650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:22.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:22.800+0000 7fe5e6a00640 1 -- 172.21.15.16:0/903251573 >> 172.21.15.16:0/903251573 conn(0x7fe5e00b2c90 msgr2=0x7fe5e006e170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:04:22.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:22.800+0000 7fe5e6a00640 1 -- 172.21.15.16:0/903251573 shutdown_connections 2024-03-31T15:04:22.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:22.800+0000 7fe5e6a00640 1 -- 172.21.15.16:0/903251573 wait complete. 2024-03-31T15:04:22.813 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-31T15:04:22.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:22.960+0000 7f006ee00640 1 Processor -- start 2024-03-31T15:04:22.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:22.960+0000 7f006ee00640 1 -- start start 2024-03-31T15:04:22.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:22.964+0000 7f006ee00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f006806c730 0x7f006806fea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:22.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:22.964+0000 7f006ee00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f00680703e0 0x7f006815dc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:22.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:22.964+0000 7f006ee00640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0068070850 con 0x7f00680b1400 2024-03-31T15:04:22.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:22.964+0000 7f006ee00640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f00680b17e0 con 0x7f006806c730 2024-03-31T15:04:22.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:22.964+0000 7f006ee00640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f00680cc5e0 con 0x7f00680703e0 2024-03-31T15:04:22.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:22.964+0000 7f006d000640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f006806c730 0x7f006806fea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:22.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:22.964+0000 7f006da00640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f00680b1400 0x7f00680d0ce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:44414/0 (socket says 172.21.15.16:44414) 2024-03-31T15:04:22.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:22.964+0000 7f006d000640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f006806c730 0x7f006806fea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49304/0 (socket says 172.21.15.16:49304) 2024-03-31T15:04:22.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:22.964+0000 7f006da00640 1 -- 172.21.15.16:0/248493837 learned_addr learned my addr 172.21.15.16:0/248493837 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:04:22.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:22.964+0000 7f006e400640 1 --2- 172.21.15.16:0/248493837 >> v2:172.21.15.89:3300/0 conn(0x7f00680703e0 0x7f006815dc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:22.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:22.964+0000 7f006d000640 1 -- 172.21.15.16:0/248493837 >> v2:172.21.15.89:3300/0 conn(0x7f00680703e0 msgr2=0x7f006815dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:04:22.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:22.964+0000 7f006d000640 1 --2- 172.21.15.16:0/248493837 >> v2:172.21.15.89:3300/0 conn(0x7f00680703e0 0x7f006815dc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:22.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:22.964+0000 7f006d000640 1 -- 172.21.15.16:0/248493837 >> v1:172.21.15.16:6789/0 conn(0x7f00680b1400 legacy=0x7f00680d0ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:22.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:22.964+0000 7f006d000640 1 -- 172.21.15.16:0/248493837 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00680cc8a0 con 0x7f006806c730 2024-03-31T15:04:22.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:22.964+0000 7f006e400640 1 --2- 172.21.15.16:0/248493837 >> v2:172.21.15.89:3300/0 conn(0x7f00680703e0 0x7f006815dc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:04:22.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:22.968+0000 7f006d000640 1 --2- 172.21.15.16:0/248493837 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f006806c730 0x7f006806fea0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f0068072a30 tx=0x7f005401c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1dca4cc221c217ef server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:22.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:22.968+0000 7f0067e00640 1 -- 172.21.15.16:0/248493837 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00540b0070 con 0x7f006806c730 2024-03-31T15:04:22.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:22.968+0000 7f0067e00640 1 -- 172.21.15.16:0/248493837 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f00540ab400 con 0x7f006806c730 2024-03-31T15:04:22.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:22.968+0000 7f0067e00640 1 -- 172.21.15.16:0/248493837 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00540ab7d0 con 0x7f006806c730 2024-03-31T15:04:22.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:22.968+0000 7f006ee00640 1 -- 172.21.15.16:0/248493837 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f006806c730 msgr2=0x7f006806fea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:22.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:22.968+0000 7f006ee00640 1 --2- 172.21.15.16:0/248493837 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f006806c730 0x7f006806fea0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f0068072a30 tx=0x7f005401c480 comp rx=0 tx=0).stop 2024-03-31T15:04:22.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:22.968+0000 7f006ee00640 1 -- 172.21.15.16:0/248493837 shutdown_connections 2024-03-31T15:04:22.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:22.968+0000 7f006ee00640 1 --2- 172.21.15.16:0/248493837 >> v2:172.21.15.89:3300/0 conn(0x7f00680703e0 0x7f006815dc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:22.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:22.968+0000 7f006ee00640 1 --2- 172.21.15.16:0/248493837 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f006806c730 0x7f006806fea0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:22.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:22.968+0000 7f006ee00640 1 -- 172.21.15.16:0/248493837 >> 172.21.15.16:0/248493837 conn(0x7f00680d1f90 msgr2=0x7f00680d23a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:04:22.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:22.968+0000 7f006ee00640 1 -- 172.21.15.16:0/248493837 shutdown_connections 2024-03-31T15:04:22.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:22.968+0000 7f006ee00640 1 -- 172.21.15.16:0/248493837 wait complete. 2024-03-31T15:04:22.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:22.968+0000 7f006ee00640 1 Processor -- start 2024-03-31T15:04:22.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:22.968+0000 7f006ee00640 1 -- start start 2024-03-31T15:04:22.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:22.972+0000 7f006ee00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f006806c730 0x7f0068157270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:22.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:22.972+0000 7f006ee00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f00680703e0 0x7f00680afb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:22.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:22.972+0000 7f006ee00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f00680b1400 0x7f0068158a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:22.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:22.972+0000 7f006ee00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f00680d13f0 con 0x7f006806c730 2024-03-31T15:04:22.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:22.972+0000 7f006ee00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f006806f930 con 0x7f00680703e0 2024-03-31T15:04:22.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:22.972+0000 7f006ee00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f00680cd660 con 0x7f00680b1400 2024-03-31T15:04:22.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:22.972+0000 7f006da00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f006806c730 0x7f0068157270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:22.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:22.972+0000 7f006da00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f006806c730 0x7f0068157270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49316/0 (socket says 172.21.15.16:49316) 2024-03-31T15:04:22.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:22.972+0000 7f006da00640 1 -- 172.21.15.16:0/2033658182 learned_addr learned my addr 172.21.15.16:0/2033658182 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:04:22.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:22.972+0000 7f006d000640 1 --2- 172.21.15.16:0/2033658182 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f00680703e0 0x7f00680afb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:22.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:22.972+0000 7f006e400640 1 --2- 172.21.15.16:0/2033658182 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f00680b1400 0x7f0068158a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:22.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:22.972+0000 7f006da00640 1 -- 172.21.15.16:0/2033658182 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f00680b1400 msgr2=0x7f0068158a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:22.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:22.972+0000 7f006da00640 1 --2- 172.21.15.16:0/2033658182 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f00680b1400 0x7f0068158a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:22.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:22.972+0000 7f006da00640 1 -- 172.21.15.16:0/2033658182 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f00680703e0 msgr2=0x7f00680afb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:22.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:22.972+0000 7f006da00640 1 --2- 172.21.15.16:0/2033658182 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f00680703e0 0x7f00680afb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:22.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:22.972+0000 7f006da00640 1 -- 172.21.15.16:0/2033658182 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00681579e0 con 0x7f006806c730 2024-03-31T15:04:22.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:22.972+0000 7f006da00640 1 --2- 172.21.15.16:0/2033658182 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f006806c730 0x7f0068157270 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f005c00c3a0 tx=0x7f005c00c7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:22.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:22.972+0000 7f0065600640 1 -- 172.21.15.16:0/2033658182 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f005c017070 con 0x7f006806c730 2024-03-31T15:04:22.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:22.972+0000 7f0065600640 1 -- 172.21.15.16:0/2033658182 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f005c005cf0 con 0x7f006806c730 2024-03-31T15:04:22.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:22.972+0000 7f006ee00640 1 -- 172.21.15.16:0/2033658182 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0068157cc0 con 0x7f006806c730 2024-03-31T15:04:22.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:22.972+0000 7f0065600640 1 -- 172.21.15.16:0/2033658182 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f005c0072f0 con 0x7f006806c730 2024-03-31T15:04:22.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:22.972+0000 7f006ee00640 1 -- 172.21.15.16:0/2033658182 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0068158280 con 0x7f006806c730 2024-03-31T15:04:22.968 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:22.972+0000 7f006ee00640 1 -- 172.21.15.16:0/2033658182 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f006804f8c0 con 0x7f006806c730 2024-03-31T15:04:22.968 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:22.972+0000 7f0065600640 1 -- 172.21.15.16:0/2033658182 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f005c01a070 con 0x7f006806c730 2024-03-31T15:04:22.968 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:22.976+0000 7f0065600640 1 --2- 172.21.15.16:0/2033658182 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f0048068380 0x7f004806a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:22.969 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:22.976+0000 7f0065600640 1 -- 172.21.15.16:0/2033658182 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f005c085cc0 con 0x7f006806c730 2024-03-31T15:04:22.969 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:22.976+0000 7f006d000640 1 --2- 172.21.15.16:0/2033658182 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f0048068380 0x7f004806a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:22.969 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:22.976+0000 7f006d000640 1 --2- 172.21.15.16:0/2033658182 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f0048068380 0x7f004806a840 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f00540ad000 tx=0x7f0054008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:22.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:22.980+0000 7f0065600640 1 -- 172.21.15.16:0/2033658182 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f005c08c080 con 0x7f006806c730 2024-03-31T15:04:23.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:23.144+0000 7f006ee00640 1 -- 172.21.15.16:0/2033658182 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f0068070040 con 0x7f006806c730 2024-03-31T15:04:23.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:23.144+0000 7f0065600640 1 -- 172.21.15.16:0/2033658182 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v23) ==== 100+0+11 (secure 0 0 0) 0x7f005c055b10 con 0x7f006806c730 2024-03-31T15:04:23.137 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 64 2024-03-31T15:04:23.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:23.144+0000 7f006ee00640 1 -- 172.21.15.16:0/2033658182 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f0048068380 msgr2=0x7f004806a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:23.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:23.144+0000 7f006ee00640 1 --2- 172.21.15.16:0/2033658182 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f0048068380 0x7f004806a840 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f00540ad000 tx=0x7f0054008040 comp rx=0 tx=0).stop 2024-03-31T15:04:23.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:23.144+0000 7f006ee00640 1 -- 172.21.15.16:0/2033658182 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f006806c730 msgr2=0x7f0068157270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:23.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:23.144+0000 7f006ee00640 1 --2- 172.21.15.16:0/2033658182 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f006806c730 0x7f0068157270 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f005c00c3a0 tx=0x7f005c00c7a0 comp rx=0 tx=0).stop 2024-03-31T15:04:23.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:23.144+0000 7f006ee00640 1 -- 172.21.15.16:0/2033658182 shutdown_connections 2024-03-31T15:04:23.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:23.144+0000 7f006ee00640 1 --2- 172.21.15.16:0/2033658182 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f0048068380 0x7f004806a840 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:23.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:23.144+0000 7f006ee00640 1 --2- 172.21.15.16:0/2033658182 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f00680b1400 0x7f0068158a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:23.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:23.144+0000 7f006ee00640 1 --2- 172.21.15.16:0/2033658182 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f00680703e0 0x7f00680afb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:23.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:23.144+0000 7f006ee00640 1 --2- 172.21.15.16:0/2033658182 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f006806c730 0x7f0068157270 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:23.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:23.144+0000 7f006ee00640 1 -- 172.21.15.16:0/2033658182 >> 172.21.15.16:0/2033658182 conn(0x7f00680d1f90 msgr2=0x7f00680d0190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:04:23.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:23.144+0000 7f006ee00640 1 -- 172.21.15.16:0/2033658182 shutdown_connections 2024-03-31T15:04:23.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:23.144+0000 7f006ee00640 1 -- 172.21.15.16:0/2033658182 wait complete. 2024-03-31T15:04:23.159 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-31T15:04:23.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:23.308+0000 7f6ca2800640 1 Processor -- start 2024-03-31T15:04:23.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:23.308+0000 7f6ca2800640 1 -- start start 2024-03-31T15:04:23.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:23.308+0000 7f6ca2800640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f6c9c0d0c50 0x7f6c9c072fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:23.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:23.308+0000 7f6ca2800640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f6c9c073510 0x7f6c9c15eb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:23.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:23.308+0000 7f6ca2800640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6c9c06bd30 con 0x7f6c9c147e70 2024-03-31T15:04:23.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:23.308+0000 7f6ca2800640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f6c9c073980 con 0x7f6c9c0d0c50 2024-03-31T15:04:23.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:23.308+0000 7f6ca2800640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f6c9c1621b0 con 0x7f6c9c073510 2024-03-31T15:04:23.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:23.308+0000 7f6c93e00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f6c9c0d0c50 0x7f6c9c072fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:23.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:23.308+0000 7f6c93e00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f6c9c0d0c50 0x7f6c9c072fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49332/0 (socket says 172.21.15.16:49332) 2024-03-31T15:04:23.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:23.308+0000 7f6c9b400640 1 --1- 172.21.15.16:0/4154424670 >> v1:172.21.15.89:6789/0 conn(0x7f6c9c147e70 0x7f6c9c06b620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.16:52792/0 (socket says 172.21.15.16:52792) 2024-03-31T15:04:23.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:23.308+0000 7f6c93e00640 1 -- 172.21.15.16:0/4154424670 learned_addr learned my addr 172.21.15.16:0/4154424670 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:04:23.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:23.308+0000 7f6c9be00640 1 --2- 172.21.15.16:0/4154424670 >> v2:172.21.15.89:3300/0 conn(0x7f6c9c073510 0x7f6c9c15eb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:23.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:23.308+0000 7f6c93e00640 1 -- 172.21.15.16:0/4154424670 >> v2:172.21.15.89:3300/0 conn(0x7f6c9c073510 msgr2=0x7f6c9c15eb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:04:23.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:23.308+0000 7f6c93e00640 1 --2- 172.21.15.16:0/4154424670 >> v2:172.21.15.89:3300/0 conn(0x7f6c9c073510 0x7f6c9c15eb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:23.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:23.308+0000 7f6c93e00640 1 -- 172.21.15.16:0/4154424670 >> v1:172.21.15.89:6789/0 conn(0x7f6c9c147e70 legacy=0x7f6c9c06b620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:23.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:23.308+0000 7f6c93e00640 1 -- 172.21.15.16:0/4154424670 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c9c162490 con 0x7f6c9c0d0c50 2024-03-31T15:04:23.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:23.308+0000 7f6c9aa00640 1 -- 172.21.15.16:0/4154424670 <== mon.1 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1315201128 0 0) 0x7f6c9c06bd30 con 0x7f6c9c147e70 2024-03-31T15:04:23.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:23.312+0000 7f6c93e00640 1 --2- 172.21.15.16:0/4154424670 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6c9c0d0c50 0x7f6c9c072fd0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f6c9c06e280 tx=0x7f6c8401c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6649827eaf08253b server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:23.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:23.312+0000 7f6c9aa00640 1 -- 172.21.15.16:0/4154424670 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c840ab610 con 0x7f6c9c0d0c50 2024-03-31T15:04:23.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:23.312+0000 7f6c9aa00640 1 -- 172.21.15.16:0/4154424670 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6c840b0070 con 0x7f6c9c0d0c50 2024-03-31T15:04:23.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:23.312+0000 7f6c9aa00640 1 -- 172.21.15.16:0/4154424670 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c840ab9d0 con 0x7f6c9c0d0c50 2024-03-31T15:04:23.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:23.312+0000 7f6ca2800640 1 -- 172.21.15.16:0/4154424670 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6c9c0d0c50 msgr2=0x7f6c9c072fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:23.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:23.312+0000 7f6ca2800640 1 --2- 172.21.15.16:0/4154424670 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6c9c0d0c50 0x7f6c9c072fd0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f6c9c06e280 tx=0x7f6c8401c520 comp rx=0 tx=0).stop 2024-03-31T15:04:23.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:23.312+0000 7f6ca2800640 1 -- 172.21.15.16:0/4154424670 shutdown_connections 2024-03-31T15:04:23.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:23.312+0000 7f6ca2800640 1 --2- 172.21.15.16:0/4154424670 >> v2:172.21.15.89:3300/0 conn(0x7f6c9c073510 0x7f6c9c15eb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:23.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:23.312+0000 7f6ca2800640 1 --2- 172.21.15.16:0/4154424670 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6c9c0d0c50 0x7f6c9c072fd0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:23.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:23.312+0000 7f6ca2800640 1 -- 172.21.15.16:0/4154424670 >> 172.21.15.16:0/4154424670 conn(0x7f6c9c0d1060 msgr2=0x7f6c9c0d1470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:04:23.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:23.312+0000 7f6ca2800640 1 -- 172.21.15.16:0/4154424670 shutdown_connections 2024-03-31T15:04:23.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:23.312+0000 7f6ca2800640 1 -- 172.21.15.16:0/4154424670 wait complete. 2024-03-31T15:04:23.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:23.316+0000 7f6ca2800640 1 Processor -- start 2024-03-31T15:04:23.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:23.316+0000 7f6ca2800640 1 -- start start 2024-03-31T15:04:23.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:23.316+0000 7f6ca2800640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6c9c073510 0x7f6c9c15dd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:23.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:23.316+0000 7f6ca2800640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f6c9c0d0c50 0x7f6c9c15d6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:23.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:23.316+0000 7f6ca2800640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6c9c147e70 0x7f6c9c13ed10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:23.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:23.316+0000 7f6ca2800640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f6c9c15f870 con 0x7f6c9c073510 2024-03-31T15:04:23.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:23.316+0000 7f6ca2800640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f6c9c162f70 con 0x7f6c9c147e70 2024-03-31T15:04:23.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:23.316+0000 7f6ca2800640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f6c9c15f9f0 con 0x7f6c9c0d0c50 2024-03-31T15:04:23.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:23.316+0000 7f6c9b400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6c9c073510 0x7f6c9c15dd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:23.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:23.316+0000 7f6c9b400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6c9c073510 0x7f6c9c15dd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49340/0 (socket says 172.21.15.16:49340) 2024-03-31T15:04:23.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:23.316+0000 7f6c9b400640 1 -- 172.21.15.16:0/1033700297 learned_addr learned my addr 172.21.15.16:0/1033700297 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:04:23.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:23.316+0000 7f6c93e00640 1 --2- 172.21.15.16:0/1033700297 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f6c9c0d0c50 0x7f6c9c15d6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:23.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:23.316+0000 7f6c9be00640 1 --2- 172.21.15.16:0/1033700297 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6c9c147e70 0x7f6c9c13ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:23.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:23.316+0000 7f6c9b400640 1 -- 172.21.15.16:0/1033700297 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f6c9c0d0c50 msgr2=0x7f6c9c15d6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:23.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:23.316+0000 7f6c9b400640 1 --2- 172.21.15.16:0/1033700297 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f6c9c0d0c50 0x7f6c9c15d6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:23.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:23.316+0000 7f6c9b400640 1 -- 172.21.15.16:0/1033700297 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6c9c147e70 msgr2=0x7f6c9c13ed10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:23.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:23.316+0000 7f6c9b400640 1 --2- 172.21.15.16:0/1033700297 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6c9c147e70 0x7f6c9c13ed10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:23.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:23.316+0000 7f6c9b400640 1 -- 172.21.15.16:0/1033700297 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c9c15c920 con 0x7f6c9c073510 2024-03-31T15:04:23.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:23.316+0000 7f6c9b400640 1 --2- 172.21.15.16:0/1033700297 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6c9c073510 0x7f6c9c15dd30 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f6c8c00c480 tx=0x7f6c8c00c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:23.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:23.316+0000 7f6c93400640 1 -- 172.21.15.16:0/1033700297 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c8c018070 con 0x7f6c9c073510 2024-03-31T15:04:23.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:23.316+0000 7f6c93400640 1 -- 172.21.15.16:0/1033700297 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6c8c007500 con 0x7f6c9c073510 2024-03-31T15:04:23.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:23.316+0000 7f6c93400640 1 -- 172.21.15.16:0/1033700297 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c8c004a30 con 0x7f6c9c073510 2024-03-31T15:04:23.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:23.316+0000 7f6ca2800640 1 -- 172.21.15.16:0/1033700297 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6c9c15cc00 con 0x7f6c9c073510 2024-03-31T15:04:23.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:23.316+0000 7f6ca2800640 1 -- 172.21.15.16:0/1033700297 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6c9c15d1c0 con 0x7f6c9c073510 2024-03-31T15:04:23.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:23.316+0000 7f6ca2800640 1 -- 172.21.15.16:0/1033700297 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6c9c069950 con 0x7f6c9c073510 2024-03-31T15:04:23.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:23.320+0000 7f6c93400640 1 -- 172.21.15.16:0/1033700297 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f6c8c004bd0 con 0x7f6c9c073510 2024-03-31T15:04:23.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:23.320+0000 7f6c93400640 1 --2- 172.21.15.16:0/1033700297 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f6c78068380 0x7f6c7806a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:23.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:23.320+0000 7f6c93400640 1 -- 172.21.15.16:0/1033700297 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f6c8c085c70 con 0x7f6c9c073510 2024-03-31T15:04:23.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:23.320+0000 7f6c93e00640 1 --2- 172.21.15.16:0/1033700297 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f6c78068380 0x7f6c7806a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:23.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:23.320+0000 7f6c93e00640 1 --2- 172.21.15.16:0/1033700297 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f6c78068380 0x7f6c7806a840 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f6c840af000 tx=0x7f6c840a7000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:23.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:23.324+0000 7f6c93400640 1 -- 172.21.15.16:0/1033700297 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f6c8c055b50 con 0x7f6c9c073510 2024-03-31T15:04:23.494 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:23.500+0000 7f6ca2800640 1 -- 172.21.15.16:0/1033700297 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f6c9c0d1bf0 con 0x7f6c9c073510 2024-03-31T15:04:23.494 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:23.500+0000 7f6c93400640 1 -- 172.21.15.16:0/1033700297 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v23) ==== 80+0+0 (secure 0 0 0) 0x7f6c8c059e50 con 0x7f6c9c073510 2024-03-31T15:04:23.494 INFO:teuthology.orchestra.run.smithi016.stderr:listed 0 entries 2024-03-31T15:04:23.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:23.500+0000 7f6ca2800640 1 -- 172.21.15.16:0/1033700297 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f6c78068380 msgr2=0x7f6c7806a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:23.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:23.500+0000 7f6ca2800640 1 --2- 172.21.15.16:0/1033700297 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f6c78068380 0x7f6c7806a840 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f6c840af000 tx=0x7f6c840a7000 comp rx=0 tx=0).stop 2024-03-31T15:04:23.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:23.500+0000 7f6ca2800640 1 -- 172.21.15.16:0/1033700297 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6c9c073510 msgr2=0x7f6c9c15dd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:23.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:23.500+0000 7f6ca2800640 1 --2- 172.21.15.16:0/1033700297 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6c9c073510 0x7f6c9c15dd30 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f6c8c00c480 tx=0x7f6c8c00c940 comp rx=0 tx=0).stop 2024-03-31T15:04:23.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:23.500+0000 7f6ca2800640 1 -- 172.21.15.16:0/1033700297 shutdown_connections 2024-03-31T15:04:23.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:23.500+0000 7f6ca2800640 1 --2- 172.21.15.16:0/1033700297 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f6c78068380 0x7f6c7806a840 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:23.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:23.500+0000 7f6ca2800640 1 --2- 172.21.15.16:0/1033700297 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6c9c147e70 0x7f6c9c13ed10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:23.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:23.500+0000 7f6ca2800640 1 --2- 172.21.15.16:0/1033700297 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f6c9c0d0c50 0x7f6c9c15d6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:23.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:23.500+0000 7f6ca2800640 1 --2- 172.21.15.16:0/1033700297 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6c9c073510 0x7f6c9c15dd30 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:23.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:23.500+0000 7f6ca2800640 1 -- 172.21.15.16:0/1033700297 >> 172.21.15.16:0/1033700297 conn(0x7f6c9c0d1060 msgr2=0x7f6c9c06e600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:04:23.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:23.500+0000 7f6ca2800640 1 -- 172.21.15.16:0/1033700297 shutdown_connections 2024-03-31T15:04:23.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:23.500+0000 7f6ca2800640 1 -- 172.21.15.16:0/1033700297 wait complete. 2024-03-31T15:04:23.516 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-03-31T15:04:23.516 DEBUG:teuthology.orchestra.run.smithi089:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-31T15:04:23.525 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-31T15:04:23.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:23.672+0000 7f824be00640 1 Processor -- start 2024-03-31T15:04:23.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:23.672+0000 7f824be00640 1 -- start start 2024-03-31T15:04:23.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:23.672+0000 7f824be00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f824406dbc0 0x7f824415f650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:23.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:23.672+0000 7f824be00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f8244070030 0x7f8244162180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:23.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:23.672+0000 7f824be00640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f82440cbe90 con 0x7f82440cb440 2024-03-31T15:04:23.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:23.672+0000 7f824be00640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f824415fb90 con 0x7f8244070030 2024-03-31T15:04:23.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:23.672+0000 7f824be00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f824415f1c0 con 0x7f824406dbc0 2024-03-31T15:04:23.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:23.672+0000 7f8248a00640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f82440cb440 0x7f824406f920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:44428/0 (socket says 172.21.15.16:44428) 2024-03-31T15:04:23.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:23.672+0000 7f8248a00640 1 -- 172.21.15.16:0/2006798325 learned_addr learned my addr 172.21.15.16:0/2006798325 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:04:23.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:23.672+0000 7f8249e00640 1 --2- 172.21.15.16:0/2006798325 >> v2:172.21.15.16:3300/0 conn(0x7f8244070030 0x7f8244162180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:23.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:23.672+0000 7f8249400640 1 --2- 172.21.15.16:0/2006798325 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f824406dbc0 0x7f824415f650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:23.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:23.672+0000 7f8243e00640 1 -- 172.21.15.16:0/2006798325 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 62169481 0 0) 0x7f82440cbe90 con 0x7f82440cb440 2024-03-31T15:04:23.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:23.672+0000 7f8249400640 1 -- 172.21.15.16:0/2006798325 >> v2:172.21.15.16:3300/0 conn(0x7f8244070030 msgr2=0x7f8244162180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:04:23.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:23.672+0000 7f8249400640 1 --2- 172.21.15.16:0/2006798325 >> v2:172.21.15.16:3300/0 conn(0x7f8244070030 0x7f8244162180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:23.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:23.672+0000 7f8249400640 1 -- 172.21.15.16:0/2006798325 >> v1:172.21.15.16:6789/0 conn(0x7f82440cb440 legacy=0x7f824406f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:23.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:23.672+0000 7f8249400640 1 -- 172.21.15.16:0/2006798325 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82440cc170 con 0x7f824406dbc0 2024-03-31T15:04:23.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:23.672+0000 7f8249e00640 1 --2- 172.21.15.16:0/2006798325 >> v2:172.21.15.16:3300/0 conn(0x7f8244070030 0x7f8244162180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:04:23.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:23.676+0000 7f8249400640 1 --2- 172.21.15.16:0/2006798325 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f824406dbc0 0x7f824415f650 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f82380097c0 tx=0x7f823801c3e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8e22172607b58052 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:23.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:23.676+0000 7f8243e00640 1 -- 172.21.15.16:0/2006798325 <== mon.2 v2:172.21.15.89:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82380b0070 con 0x7f824406dbc0 2024-03-31T15:04:23.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:23.676+0000 7f8243e00640 1 -- 172.21.15.16:0/2006798325 <== mon.2 v2:172.21.15.89:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f82380ab600 con 0x7f824406dbc0 2024-03-31T15:04:23.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:23.676+0000 7f8243e00640 1 -- 172.21.15.16:0/2006798325 <== mon.2 v2:172.21.15.89:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82380ab9d0 con 0x7f824406dbc0 2024-03-31T15:04:23.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:23.676+0000 7f824be00640 1 -- 172.21.15.16:0/2006798325 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f824406dbc0 msgr2=0x7f824415f650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:23.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:23.676+0000 7f824be00640 1 --2- 172.21.15.16:0/2006798325 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f824406dbc0 0x7f824415f650 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f82380097c0 tx=0x7f823801c3e0 comp rx=0 tx=0).stop 2024-03-31T15:04:23.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:23.676+0000 7f824be00640 1 -- 172.21.15.16:0/2006798325 shutdown_connections 2024-03-31T15:04:23.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:23.676+0000 7f824be00640 1 --2- 172.21.15.16:0/2006798325 >> v2:172.21.15.16:3300/0 conn(0x7f8244070030 0x7f8244162180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:23.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:23.676+0000 7f824be00640 1 --2- 172.21.15.16:0/2006798325 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f824406dbc0 0x7f824415f650 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:23.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:23.676+0000 7f824be00640 1 -- 172.21.15.16:0/2006798325 >> 172.21.15.16:0/2006798325 conn(0x7f8244067330 msgr2=0x7f8244067740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:04:23.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:23.676+0000 7f824be00640 1 -- 172.21.15.16:0/2006798325 shutdown_connections 2024-03-31T15:04:23.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:23.676+0000 7f824be00640 1 -- 172.21.15.16:0/2006798325 wait complete. 2024-03-31T15:04:23.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:23.676+0000 7f824be00640 1 Processor -- start 2024-03-31T15:04:23.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:23.676+0000 7f824be00640 1 -- start start 2024-03-31T15:04:23.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:23.680+0000 7f824be00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f824406dbc0 0x7f8244159290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:23.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:23.680+0000 7f824be00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8244070030 0x7f8244143080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:23.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:23.680+0000 7f824be00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f82440cb440 0x7f8244159c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:23.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:23.680+0000 7f824be00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f824406cad0 con 0x7f824406dbc0 2024-03-31T15:04:23.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:23.680+0000 7f824be00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f824406c950 con 0x7f8244070030 2024-03-31T15:04:23.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:23.680+0000 7f824be00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f824416b520 con 0x7f82440cb440 2024-03-31T15:04:23.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:23.680+0000 7f8249400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f824406dbc0 0x7f8244159290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:23.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:23.680+0000 7f8249400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f824406dbc0 0x7f8244159290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49354/0 (socket says 172.21.15.16:49354) 2024-03-31T15:04:23.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:23.680+0000 7f8249400640 1 -- 172.21.15.16:0/925280611 learned_addr learned my addr 172.21.15.16:0/925280611 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:04:23.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:23.680+0000 7f8248a00640 1 --2- 172.21.15.16:0/925280611 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8244070030 0x7f8244143080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:23.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:23.680+0000 7f8249e00640 1 --2- 172.21.15.16:0/925280611 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f82440cb440 0x7f8244159c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:23.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:23.680+0000 7f8249400640 1 -- 172.21.15.16:0/925280611 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f82440cb440 msgr2=0x7f8244159c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:23.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:23.680+0000 7f8249400640 1 --2- 172.21.15.16:0/925280611 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f82440cb440 0x7f8244159c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:23.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:23.680+0000 7f8249400640 1 -- 172.21.15.16:0/925280611 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8244070030 msgr2=0x7f8244143080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:23.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:23.680+0000 7f8249400640 1 --2- 172.21.15.16:0/925280611 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8244070030 0x7f8244143080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:23.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:23.680+0000 7f8249400640 1 -- 172.21.15.16:0/925280611 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82441597d0 con 0x7f824406dbc0 2024-03-31T15:04:23.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:23.680+0000 7f8249400640 1 --2- 172.21.15.16:0/925280611 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f824406dbc0 0x7f8244159290 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f823801a010 tx=0x7f8238019180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:23.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:23.680+0000 7f8241600640 1 -- 172.21.15.16:0/925280611 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82380ba070 con 0x7f824406dbc0 2024-03-31T15:04:23.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:23.680+0000 7f8241600640 1 -- 172.21.15.16:0/925280611 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f82380b43f0 con 0x7f824406dbc0 2024-03-31T15:04:23.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:23.680+0000 7f8241600640 1 -- 172.21.15.16:0/925280611 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82380b0040 con 0x7f824406dbc0 2024-03-31T15:04:23.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:23.680+0000 7f824be00640 1 -- 172.21.15.16:0/925280611 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f82441408b0 con 0x7f824406dbc0 2024-03-31T15:04:23.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:23.680+0000 7f824be00640 1 -- 172.21.15.16:0/925280611 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8244140e70 con 0x7f824406dbc0 2024-03-31T15:04:23.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:23.680+0000 7f824be00640 1 -- 172.21.15.16:0/925280611 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8244069950 con 0x7f824406dbc0 2024-03-31T15:04:23.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:23.680+0000 7f8241600640 1 -- 172.21.15.16:0/925280611 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f82380bd070 con 0x7f824406dbc0 2024-03-31T15:04:23.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:23.684+0000 7f8241600640 1 --2- 172.21.15.16:0/925280611 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f82240683b0 0x7f822406a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:23.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:23.684+0000 7f8241600640 1 -- 172.21.15.16:0/925280611 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f823811f7d0 con 0x7f824406dbc0 2024-03-31T15:04:23.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:23.684+0000 7f8248a00640 1 --2- 172.21.15.16:0/925280611 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f82240683b0 0x7f822406a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:23.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:23.684+0000 7f8248a00640 1 --2- 172.21.15.16:0/925280611 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f82240683b0 0x7f822406a870 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f82441fa9f0 tx=0x7f8230007d60 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:23.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:23.688+0000 7f8241600640 1 -- 172.21.15.16:0/925280611 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f8238016090 con 0x7f824406dbc0 2024-03-31T15:04:23.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:23.852+0000 7f824be00640 1 -- 172.21.15.16:0/925280611 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f82441692f0 con 0x7f824406dbc0 2024-03-31T15:04:23.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:23.852+0000 7f8241600640 1 -- 172.21.15.16:0/925280611 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v23) ==== 74+0+16780 (secure 0 0 0) 0x7f82380ef620 con 0x7f824406dbc0 2024-03-31T15:04:23.848 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:04:23.848 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":23,"fsid":"286eb5da-fb47-4550-b290-d9519c13b9a5","created":"2024-03-31T15:03:11.438952+0000","modified":"2024-03-31T15:04:18.087947+0000","last_up_change":"2024-03-31T15:03:22.885535+0000","last_in_change":"2024-03-31T15:03:13.941667+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-31T15:03:22.550787+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-31T15:03:34.488425+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-31T15:03:35.299116+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"92a66450-ef19-40d8-ac3d-d760b23b0bda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6801","nonce":932712244}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6803","nonce":932712244}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6807","nonce":932712244}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6805","nonce":932712244}]},"public_addr":"172.21.15.16:6801/932712244","cluster_addr":"172.21.15.16:6803/932712244","heartbeat_back_addr":"172.21.15.16:6807/932712244","heartbeat_front_addr":"172.21.15.16:6805/932712244","state":["exists","up"]},{"osd":1,"uuid":"da9d1f2b-1968-4959-b843-faa231988e92","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6810","nonce":1398737533}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6811","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6813","nonce":1398737533}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6821","nonce":1398737533}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6815","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6816","nonce":1398737533}]},"public_addr":"172.21.15.16:6810/1398737533","cluster_addr":"172.21.15.16:6813/1398737533","heartbeat_back_addr":"172.21.15.16:6821/1398737533","heartbeat_front_addr":"172.21.15.16:6816/1398737533","state":["exists","up"]},{"osd":2,"uuid":"01a73662-9a21-4b1f-bcbb-42b9636a4638","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6819","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6822","nonce":3018526646}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6825","nonce":3018526646}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6831","nonce":3018526646}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6827","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6829","nonce":3018526646}]},"public_addr":"172.21.15.16:6822/3018526646","cluster_addr":"172.21.15.16:6825/3018526646","heartbeat_back_addr":"172.21.15.16:6831/3018526646","heartbeat_front_addr":"172.21.15.16:6829/3018526646","state":["exists","up"]},{"osd":3,"uuid":"c7771fd3-e18e-4500-ac38-23cf10ebcd75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6809","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6812","nonce":3360352974}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6817","nonce":3360352974}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6828","nonce":3360352974}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6823","nonce":3360352974}]},"public_addr":"172.21.15.16:6812/3360352974","cluster_addr":"172.21.15.16:6817/3360352974","heartbeat_back_addr":"172.21.15.16:6828/3360352974","heartbeat_front_addr":"172.21.15.16:6823/3360352974","state":["exists","up"]},{"osd":4,"uuid":"88260730-1482-4fbd-a40f-cb1124e9df4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6801","nonce":1856844856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6803","nonce":1856844856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6807","nonce":1856844856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6805","nonce":1856844856}]},"public_addr":"172.21.15.89:6801/1856844856","cluster_addr":"172.21.15.89:6803/1856844856","heartbeat_back_addr":"172.21.15.89:6807/1856844856","heartbeat_front_addr":"172.21.15.89:6805/1856844856","state":["exists","up"]},{"osd":5,"uuid":"4cb2c00c-fae0-4dd8-9d27-ec4f0093ccf7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6814","nonce":50395527}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6817","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6820","nonce":50395527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6831","nonce":50395527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6823","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6826","nonce":50395527}]},"public_addr":"172.21.15.89:6814/50395527","cluster_addr":"172.21.15.89:6820/50395527","heartbeat_back_addr":"172.21.15.89:6831/50395527","heartbeat_front_addr":"172.21.15.89:6826/50395527","state":["exists","up"]},{"osd":6,"uuid":"81ce0c19-2554-4cae-acbe-66a05e9e9b9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6809","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6812","nonce":3470471752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6815","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6818","nonce":3470471752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6827","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6829","nonce":3470471752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6821","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6824","nonce":3470471752}]},"public_addr":"172.21.15.89:6812/3470471752","cluster_addr":"172.21.15.89:6818/3470471752","heartbeat_back_addr":"172.21.15.89:6829/3470471752","heartbeat_front_addr":"172.21.15.89:6824/3470471752","state":["exists","up"]},{"osd":7,"uuid":"85e4313b-d37d-442e-8042-153bf20a847d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6811","nonce":1796778630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6813","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6816","nonce":1796778630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6825","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6828","nonce":1796778630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6819","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6822","nonce":1796778630}]},"public_addr":"172.21.15.89:6811/1796778630","cluster_addr":"172.21.15.89:6816/1796778630","heartbeat_back_addr":"172.21.15.89:6828/1796778630","heartbeat_front_addr":"172.21.15.89:6822/1796778630","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.439088+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.529970+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.680512+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:21.044161+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.582111+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.536558+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.409522+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.590229+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.2","mappings":[{"from":3,"to":2}]},{"pgid":"3.7","mappings":[{"from":3,"to":2}]},{"pgid":"3.d","mappings":[{"from":5,"to":1}]},{"pgid":"3.22","mappings":[{"from":5,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-31T15:04:23.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:23.856+0000 7f824be00640 1 -- 172.21.15.16:0/925280611 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f82240683b0 msgr2=0x7f822406a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:23.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:23.856+0000 7f824be00640 1 --2- 172.21.15.16:0/925280611 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f82240683b0 0x7f822406a870 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f82441fa9f0 tx=0x7f8230007d60 comp rx=0 tx=0).stop 2024-03-31T15:04:23.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:23.856+0000 7f824be00640 1 -- 172.21.15.16:0/925280611 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f824406dbc0 msgr2=0x7f8244159290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:23.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:23.856+0000 7f824be00640 1 --2- 172.21.15.16:0/925280611 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f824406dbc0 0x7f8244159290 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f823801a010 tx=0x7f8238019180 comp rx=0 tx=0).stop 2024-03-31T15:04:23.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:23.856+0000 7f824be00640 1 -- 172.21.15.16:0/925280611 shutdown_connections 2024-03-31T15:04:23.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:23.856+0000 7f824be00640 1 --2- 172.21.15.16:0/925280611 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f82240683b0 0x7f822406a870 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:23.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:23.856+0000 7f824be00640 1 --2- 172.21.15.16:0/925280611 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f82440cb440 0x7f8244159c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:23.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:23.856+0000 7f824be00640 1 --2- 172.21.15.16:0/925280611 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8244070030 0x7f8244143080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:23.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:23.856+0000 7f824be00640 1 --2- 172.21.15.16:0/925280611 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f824406dbc0 0x7f8244159290 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:23.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:23.856+0000 7f824be00640 1 -- 172.21.15.16:0/925280611 >> 172.21.15.16:0/925280611 conn(0x7f8244067330 msgr2=0x7f82440d0890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:04:23.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:23.856+0000 7f824be00640 1 -- 172.21.15.16:0/925280611 shutdown_connections 2024-03-31T15:04:23.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:23.856+0000 7f824be00640 1 -- 172.21.15.16:0/925280611 wait complete. 2024-03-31T15:04:23.870 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-31T15:03:22.550787+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-31T15:03:34.488425+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-31T15:03:35.299116+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-31T15:04:23.870 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-31T15:04:24.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:24.020+0000 7fb7f6600640 1 Processor -- start 2024-03-31T15:04:24.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:24.020+0000 7fb7f6600640 1 -- start start 2024-03-31T15:04:24.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:24.020+0000 7fb7f6600640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb7f013dff0 0x7fb7f006d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:24.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:24.020+0000 7fb7f6600640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb7f013e3d0 0x7fb7f006da10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:24.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:24.020+0000 7fb7f6600640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb7f015fbc0 con 0x7fb7f00cc9d0 2024-03-31T15:04:24.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:24.020+0000 7fb7f6600640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7fb7f015f730 con 0x7fb7f013dff0 2024-03-31T15:04:24.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:24.020+0000 7fb7f6600640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fb7f015ee00 con 0x7fb7f013e3d0 2024-03-31T15:04:24.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:24.020+0000 7fb7ef400640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb7f013dff0 0x7fb7f006d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:24.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:24.020+0000 7fb7efe00640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fb7f00cc9d0 0x7fb7f00cce30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:44442/0 (socket says 172.21.15.16:44442) 2024-03-31T15:04:24.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:24.020+0000 7fb7efe00640 1 -- 172.21.15.16:0/1931943469 learned_addr learned my addr 172.21.15.16:0/1931943469 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:04:24.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:24.020+0000 7fb7e7e00640 1 --2- 172.21.15.16:0/1931943469 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb7f013e3d0 0x7fb7f006da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:24.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:24.020+0000 7fb7e7e00640 1 -- 172.21.15.16:0/1931943469 >> v2:172.21.15.16:3300/0 conn(0x7fb7f013dff0 msgr2=0x7fb7f006d4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:04:24.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:24.020+0000 7fb7e7e00640 1 --2- 172.21.15.16:0/1931943469 >> v2:172.21.15.16:3300/0 conn(0x7fb7f013dff0 0x7fb7f006d4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:24.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:24.020+0000 7fb7e7e00640 1 -- 172.21.15.16:0/1931943469 >> v1:172.21.15.16:6789/0 conn(0x7fb7f00cc9d0 legacy=0x7fb7f00cce30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:24.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:24.020+0000 7fb7e7e00640 1 -- 172.21.15.16:0/1931943469 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7f015f0e0 con 0x7fb7f013e3d0 2024-03-31T15:04:24.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:24.020+0000 7fb7ef400640 1 --2- 172.21.15.16:0/1931943469 >> v2:172.21.15.16:3300/0 conn(0x7fb7f013dff0 0x7fb7f006d4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:04:24.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:24.024+0000 7fb7e7e00640 1 --2- 172.21.15.16:0/1931943469 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb7f013e3d0 0x7fb7f006da10 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fb7d80097c0 tx=0x7fb7d801c410 comp rx=0 tx=0).ready entity=mon.2 client_cookie=72a076d63f33db54 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:24.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:24.024+0000 7fb7eea00640 1 -- 172.21.15.16:0/1931943469 <== mon.2 v2:172.21.15.89:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7d80b0070 con 0x7fb7f013e3d0 2024-03-31T15:04:24.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:24.024+0000 7fb7eea00640 1 -- 172.21.15.16:0/1931943469 <== mon.2 v2:172.21.15.89:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb7d80ab600 con 0x7fb7f013e3d0 2024-03-31T15:04:24.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:24.024+0000 7fb7eea00640 1 -- 172.21.15.16:0/1931943469 <== mon.2 v2:172.21.15.89:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7d80ab9d0 con 0x7fb7f013e3d0 2024-03-31T15:04:24.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:24.024+0000 7fb7f6600640 1 -- 172.21.15.16:0/1931943469 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb7f013e3d0 msgr2=0x7fb7f006da10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:24.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:24.024+0000 7fb7f6600640 1 --2- 172.21.15.16:0/1931943469 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb7f013e3d0 0x7fb7f006da10 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fb7d80097c0 tx=0x7fb7d801c410 comp rx=0 tx=0).stop 2024-03-31T15:04:24.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:24.024+0000 7fb7f6600640 1 -- 172.21.15.16:0/1931943469 shutdown_connections 2024-03-31T15:04:24.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:24.024+0000 7fb7f6600640 1 --2- 172.21.15.16:0/1931943469 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb7f013e3d0 0x7fb7f006da10 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:24.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:24.024+0000 7fb7f6600640 1 --2- 172.21.15.16:0/1931943469 >> v2:172.21.15.16:3300/0 conn(0x7fb7f013dff0 0x7fb7f006d4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:24.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:24.024+0000 7fb7f6600640 1 -- 172.21.15.16:0/1931943469 >> 172.21.15.16:0/1931943469 conn(0x7fb7f006ee40 msgr2=0x7fb7f006b620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:04:24.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:24.024+0000 7fb7f6600640 1 -- 172.21.15.16:0/1931943469 shutdown_connections 2024-03-31T15:04:24.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:24.024+0000 7fb7f6600640 1 -- 172.21.15.16:0/1931943469 wait complete. 2024-03-31T15:04:24.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:24.028+0000 7fb7f6600640 1 Processor -- start 2024-03-31T15:04:24.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:24.028+0000 7fb7f6600640 1 -- start start 2024-03-31T15:04:24.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:24.028+0000 7fb7f6600640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb7f00cc9d0 0x7fb7f015a6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:24.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:24.028+0000 7fb7f6600640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb7f013dff0 0x7fb7f0154670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:24.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:24.028+0000 7fb7f6600640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb7f013e3d0 0x7fb7f013c7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:24.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:24.028+0000 7fb7f6600640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fb7f0072170 con 0x7fb7f013e3d0 2024-03-31T15:04:24.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:24.028+0000 7fb7f6600640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fb7f0071ff0 con 0x7fb7f00cc9d0 2024-03-31T15:04:24.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:24.028+0000 7fb7f6600640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fb7f00722f0 con 0x7fb7f013dff0 2024-03-31T15:04:24.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:24.028+0000 7fb7efe00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb7f013e3d0 0x7fb7f013c7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:24.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:24.028+0000 7fb7ef400640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb7f00cc9d0 0x7fb7f015a6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:24.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:24.028+0000 7fb7ef400640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb7f00cc9d0 0x7fb7f015a6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.16:49202/0 (socket says 172.21.15.16:49202) 2024-03-31T15:04:24.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:24.028+0000 7fb7e7e00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb7f013dff0 0x7fb7f0154670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:24.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:24.028+0000 7fb7efe00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb7f013e3d0 0x7fb7f013c7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49366/0 (socket says 172.21.15.16:49366) 2024-03-31T15:04:24.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:24.028+0000 7fb7ef400640 1 -- 172.21.15.16:0/95942371 learned_addr learned my addr 172.21.15.16:0/95942371 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:04:24.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:24.028+0000 7fb7efe00640 1 -- 172.21.15.16:0/95942371 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb7f013dff0 msgr2=0x7fb7f0154670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:24.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:24.028+0000 7fb7efe00640 1 --2- 172.21.15.16:0/95942371 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb7f013dff0 0x7fb7f0154670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:24.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:24.028+0000 7fb7efe00640 1 -- 172.21.15.16:0/95942371 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb7f00cc9d0 msgr2=0x7fb7f015a6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:24.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:24.028+0000 7fb7efe00640 1 --2- 172.21.15.16:0/95942371 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb7f00cc9d0 0x7fb7f015a6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:24.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:24.028+0000 7fb7efe00640 1 -- 172.21.15.16:0/95942371 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7f0154bb0 con 0x7fb7f013e3d0 2024-03-31T15:04:24.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:24.028+0000 7fb7ef400640 1 --2- 172.21.15.16:0/95942371 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb7f00cc9d0 0x7fb7f015a6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:04:24.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:24.028+0000 7fb7efe00640 1 --2- 172.21.15.16:0/95942371 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb7f013e3d0 0x7fb7f013c7f0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fb7e800caa0 tx=0x7fb7e800cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:24.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:24.028+0000 7fb7e7400640 1 -- 172.21.15.16:0/95942371 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7e8017070 con 0x7fb7f013e3d0 2024-03-31T15:04:24.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:24.028+0000 7fb7e7400640 1 -- 172.21.15.16:0/95942371 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb7e800ac00 con 0x7fb7f013e3d0 2024-03-31T15:04:24.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:24.028+0000 7fb7e7400640 1 -- 172.21.15.16:0/95942371 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7e8012400 con 0x7fb7f013e3d0 2024-03-31T15:04:24.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:24.028+0000 7fb7f6600640 1 -- 172.21.15.16:0/95942371 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb7f0154e50 con 0x7fb7f013e3d0 2024-03-31T15:04:24.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:24.028+0000 7fb7f6600640 1 -- 172.21.15.16:0/95942371 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb7f0155410 con 0x7fb7f013e3d0 2024-03-31T15:04:24.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:24.032+0000 7fb7f6600640 1 -- 172.21.15.16:0/95942371 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb7f0069950 con 0x7fb7f013e3d0 2024-03-31T15:04:24.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:24.032+0000 7fb7e7400640 1 -- 172.21.15.16:0/95942371 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fb7e801a070 con 0x7fb7f013e3d0 2024-03-31T15:04:24.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:24.032+0000 7fb7e7400640 1 --2- 172.21.15.16:0/95942371 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fb7cc068380 0x7fb7cc06a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:24.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:24.032+0000 7fb7e7400640 1 -- 172.21.15.16:0/95942371 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7fb7e8086600 con 0x7fb7f013e3d0 2024-03-31T15:04:24.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:24.032+0000 7fb7ef400640 1 --2- 172.21.15.16:0/95942371 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fb7cc068380 0x7fb7cc06a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:24.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:24.032+0000 7fb7ef400640 1 --2- 172.21.15.16:0/95942371 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fb7cc068380 0x7fb7cc06a840 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fb7e000b820 tx=0x7fb7e000b3a0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:24.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:24.036+0000 7fb7e7400640 1 -- 172.21.15.16:0/95942371 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fb7e8056450 con 0x7fb7f013e3d0 2024-03-31T15:04:24.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:24.200+0000 7fb7f6600640 1 -- 172.21.15.16:0/95942371 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fb7f006e240 con 0x7fb7f013e3d0 2024-03-31T15:04:24.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:24.200+0000 7fb7e7400640 1 -- 172.21.15.16:0/95942371 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v23) ==== 93+0+10 (secure 0 0 0) 0x7fb7e805a750 con 0x7fb7f013e3d0 2024-03-31T15:04:24.196 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2024-03-31T15:04:24.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:24.204+0000 7fb7f6600640 1 -- 172.21.15.16:0/95942371 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fb7cc068380 msgr2=0x7fb7cc06a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:24.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:24.204+0000 7fb7f6600640 1 --2- 172.21.15.16:0/95942371 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fb7cc068380 0x7fb7cc06a840 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fb7e000b820 tx=0x7fb7e000b3a0 comp rx=0 tx=0).stop 2024-03-31T15:04:24.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:24.204+0000 7fb7f6600640 1 -- 172.21.15.16:0/95942371 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb7f013e3d0 msgr2=0x7fb7f013c7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:24.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:24.204+0000 7fb7f6600640 1 --2- 172.21.15.16:0/95942371 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb7f013e3d0 0x7fb7f013c7f0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fb7e800caa0 tx=0x7fb7e800cf60 comp rx=0 tx=0).stop 2024-03-31T15:04:24.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:24.204+0000 7fb7f6600640 1 -- 172.21.15.16:0/95942371 shutdown_connections 2024-03-31T15:04:24.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:24.204+0000 7fb7f6600640 1 --2- 172.21.15.16:0/95942371 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fb7cc068380 0x7fb7cc06a840 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:24.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:24.204+0000 7fb7f6600640 1 --2- 172.21.15.16:0/95942371 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb7f013e3d0 0x7fb7f013c7f0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:24.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:24.204+0000 7fb7f6600640 1 --2- 172.21.15.16:0/95942371 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb7f013dff0 0x7fb7f0154670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:24.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:24.204+0000 7fb7f6600640 1 --2- 172.21.15.16:0/95942371 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb7f00cc9d0 0x7fb7f015a6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:24.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:24.204+0000 7fb7f6600640 1 -- 172.21.15.16:0/95942371 >> 172.21.15.16:0/95942371 conn(0x7fb7f006ee40 msgr2=0x7fb7f006b620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:04:24.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:24.204+0000 7fb7f6600640 1 -- 172.21.15.16:0/95942371 shutdown_connections 2024-03-31T15:04:24.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:24.204+0000 7fb7f6600640 1 -- 172.21.15.16:0/95942371 wait complete. 2024-03-31T15:04:24.218 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-31T15:04:24.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:24.368+0000 7fa168200640 1 Processor -- start 2024-03-31T15:04:24.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:24.368+0000 7fa168200640 1 -- start start 2024-03-31T15:04:24.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:24.368+0000 7fa168200640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fa16013dff0 0x7fa16006d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:24.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:24.368+0000 7fa168200640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa16015fbc0 con 0x7fa16013e3d0 2024-03-31T15:04:24.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:24.368+0000 7fa168200640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa16015ee00 con 0x7fa1600cd0e0 2024-03-31T15:04:24.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:24.368+0000 7fa168200640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fa16015f730 con 0x7fa16013dff0 2024-03-31T15:04:24.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:24.368+0000 7fa164e00640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fa16013e3d0 0x7fa1600cc9d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:44452/0 (socket says 172.21.15.16:44452) 2024-03-31T15:04:24.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:24.368+0000 7fa164e00640 1 -- 172.21.15.16:0/141986254 learned_addr learned my addr 172.21.15.16:0/141986254 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:04:24.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:24.368+0000 7fa165800640 1 --2- 172.21.15.16:0/141986254 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fa16013dff0 0x7fa16006d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:24.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:24.368+0000 7fa15fe00640 1 -- 172.21.15.16:0/141986254 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3108558730 0 0) 0x7fa16015fbc0 con 0x7fa16013e3d0 2024-03-31T15:04:24.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:24.368+0000 7fa15fe00640 1 -- 172.21.15.16:0/141986254 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa148003530 con 0x7fa16013e3d0 2024-03-31T15:04:24.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:24.368+0000 7fa15fe00640 1 -- 172.21.15.16:0/141986254 <== mon.1 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2355480570 0 0) 0x7fa16015ee00 con 0x7fa1600cd0e0 2024-03-31T15:04:24.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:24.368+0000 7fa15fe00640 1 -- 172.21.15.16:0/141986254 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa16015fbc0 con 0x7fa1600cd0e0 2024-03-31T15:04:24.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:24.368+0000 7fa165800640 1 -- 172.21.15.16:0/141986254 >> v1:172.21.15.89:6789/0 conn(0x7fa1600cd0e0 legacy=0x7fa1600ceb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:24.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:24.368+0000 7fa165800640 1 -- 172.21.15.16:0/141986254 >> v1:172.21.15.16:6789/0 conn(0x7fa16013e3d0 legacy=0x7fa1600cc9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:24.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:24.368+0000 7fa165800640 1 -- 172.21.15.16:0/141986254 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa16015f0e0 con 0x7fa16013dff0 2024-03-31T15:04:24.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:24.372+0000 7fa165800640 1 --2- 172.21.15.16:0/141986254 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fa16013dff0 0x7fa16006d4d0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fa1540097f0 tx=0x7fa15401c450 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f1367f981d0f737f server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:24.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:24.372+0000 7fa15fe00640 1 -- 172.21.15.16:0/141986254 <== mon.2 v2:172.21.15.89:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1540b0070 con 0x7fa16013dff0 2024-03-31T15:04:24.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:24.372+0000 7fa15fe00640 1 -- 172.21.15.16:0/141986254 <== mon.2 v2:172.21.15.89:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa1540ab600 con 0x7fa16013dff0 2024-03-31T15:04:24.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:24.372+0000 7fa15fe00640 1 -- 172.21.15.16:0/141986254 <== mon.2 v2:172.21.15.89:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1540ab9d0 con 0x7fa16013dff0 2024-03-31T15:04:24.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:24.372+0000 7fa168200640 1 -- 172.21.15.16:0/141986254 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fa16013dff0 msgr2=0x7fa16006d4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:24.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:24.372+0000 7fa168200640 1 --2- 172.21.15.16:0/141986254 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fa16013dff0 0x7fa16006d4d0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fa1540097f0 tx=0x7fa15401c450 comp rx=0 tx=0).stop 2024-03-31T15:04:24.368 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:24.372+0000 7fa168200640 1 -- 172.21.15.16:0/141986254 shutdown_connections 2024-03-31T15:04:24.368 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:24.372+0000 7fa168200640 1 --2- 172.21.15.16:0/141986254 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fa16013dff0 0x7fa16006d4d0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:24.368 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:24.372+0000 7fa168200640 1 -- 172.21.15.16:0/141986254 >> 172.21.15.16:0/141986254 conn(0x7fa16006ee40 msgr2=0x7fa16006b620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:04:24.368 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:24.372+0000 7fa168200640 1 -- 172.21.15.16:0/141986254 shutdown_connections 2024-03-31T15:04:24.368 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:24.372+0000 7fa168200640 1 -- 172.21.15.16:0/141986254 wait complete. 2024-03-31T15:04:24.369 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:24.376+0000 7fa168200640 1 Processor -- start 2024-03-31T15:04:24.369 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:24.376+0000 7fa168200640 1 -- start start 2024-03-31T15:04:24.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:24.376+0000 7fa168200640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa1600cd0e0 0x7fa1600ac8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:24.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:24.376+0000 7fa168200640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fa16013dff0 0x7fa160156590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:24.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:24.376+0000 7fa168200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa16013e3d0 0x7fa160156ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:24.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:24.376+0000 7fa168200640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fa16006de20 con 0x7fa16013e3d0 2024-03-31T15:04:24.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:24.376+0000 7fa168200640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fa1600cfca0 con 0x7fa1600cd0e0 2024-03-31T15:04:24.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:24.376+0000 7fa168200640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fa160071550 con 0x7fa16013dff0 2024-03-31T15:04:24.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:24.376+0000 7fa166200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa16013e3d0 0x7fa160156ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:24.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:24.376+0000 7fa165800640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa1600cd0e0 0x7fa1600ac8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:24.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:24.376+0000 7fa166200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa16013e3d0 0x7fa160156ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49378/0 (socket says 172.21.15.16:49378) 2024-03-31T15:04:24.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:24.376+0000 7fa166200640 1 -- 172.21.15.16:0/1336725500 learned_addr learned my addr 172.21.15.16:0/1336725500 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:04:24.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:24.376+0000 7fa164e00640 1 --2- 172.21.15.16:0/1336725500 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fa16013dff0 0x7fa160156590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:24.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:24.376+0000 7fa165800640 1 -- 172.21.15.16:0/1336725500 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fa16013dff0 msgr2=0x7fa160156590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:24.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:24.376+0000 7fa165800640 1 --2- 172.21.15.16:0/1336725500 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fa16013dff0 0x7fa160156590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:24.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:24.376+0000 7fa165800640 1 -- 172.21.15.16:0/1336725500 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa16013e3d0 msgr2=0x7fa160156ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:24.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:24.376+0000 7fa165800640 1 --2- 172.21.15.16:0/1336725500 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa16013e3d0 0x7fa160156ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:24.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:24.376+0000 7fa165800640 1 -- 172.21.15.16:0/1336725500 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa16015a100 con 0x7fa1600cd0e0 2024-03-31T15:04:24.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:24.376+0000 7fa166200640 1 --2- 172.21.15.16:0/1336725500 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa16013e3d0 0x7fa160156ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:04:24.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:24.376+0000 7fa165800640 1 --2- 172.21.15.16:0/1336725500 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa1600cd0e0 0x7fa1600ac8e0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fa1540ab000 tx=0x7fa15401ac90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:24.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:24.376+0000 7fa15d600640 1 -- 172.21.15.16:0/1336725500 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1540b0050 con 0x7fa1600cd0e0 2024-03-31T15:04:24.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:24.376+0000 7fa15d600640 1 -- 172.21.15.16:0/1336725500 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa1540bc070 con 0x7fa1600cd0e0 2024-03-31T15:04:24.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:24.376+0000 7fa168200640 1 -- 172.21.15.16:0/1336725500 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa16015a3e0 con 0x7fa1600cd0e0 2024-03-31T15:04:24.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:24.376+0000 7fa15d600640 1 -- 172.21.15.16:0/1336725500 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1540b5a30 con 0x7fa1600cd0e0 2024-03-31T15:04:24.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:24.376+0000 7fa168200640 1 -- 172.21.15.16:0/1336725500 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa16015a9a0 con 0x7fa1600cd0e0 2024-03-31T15:04:24.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:24.376+0000 7fa168200640 1 -- 172.21.15.16:0/1336725500 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa160069950 con 0x7fa1600cd0e0 2024-03-31T15:04:24.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:24.380+0000 7fa15d600640 1 -- 172.21.15.16:0/1336725500 <== mon.1 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fa1540b5bd0 con 0x7fa1600cd0e0 2024-03-31T15:04:24.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:24.380+0000 7fa15d600640 1 --2- 172.21.15.16:0/1336725500 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fa140068380 0x7fa14006a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:24.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:24.380+0000 7fa15d600640 1 -- 172.21.15.16:0/1336725500 <== mon.1 v2:172.21.15.89:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7fa154121280 con 0x7fa1600cd0e0 2024-03-31T15:04:24.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:24.380+0000 7fa164e00640 1 --2- 172.21.15.16:0/1336725500 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fa140068380 0x7fa14006a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:24.375 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:24.380+0000 7fa164e00640 1 --2- 172.21.15.16:0/1336725500 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fa140068380 0x7fa14006a840 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fa1601573a0 tx=0x7fa14c007d40 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:24.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:24.388+0000 7fa15d600640 1 -- 172.21.15.16:0/1336725500 <== mon.1 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fa1540160d0 con 0x7fa1600cd0e0 2024-03-31T15:04:24.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:24.552+0000 7fa168200640 1 -- 172.21.15.16:0/1336725500 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fa16006e4e0 con 0x7fa1600cd0e0 2024-03-31T15:04:24.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:24.552+0000 7fa15d600640 1 -- 172.21.15.16:0/1336725500 <== mon.1 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v23) ==== 104+0+11 (secure 0 0 0) 0x7fa1540f11e0 con 0x7fa1600cd0e0 2024-03-31T15:04:24.548 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2024-03-31T15:04:24.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:24.556+0000 7fa168200640 1 -- 172.21.15.16:0/1336725500 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fa140068380 msgr2=0x7fa14006a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:24.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:24.556+0000 7fa168200640 1 --2- 172.21.15.16:0/1336725500 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fa140068380 0x7fa14006a840 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fa1601573a0 tx=0x7fa14c007d40 comp rx=0 tx=0).stop 2024-03-31T15:04:24.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:24.556+0000 7fa168200640 1 -- 172.21.15.16:0/1336725500 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa1600cd0e0 msgr2=0x7fa1600ac8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:24.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:24.556+0000 7fa168200640 1 --2- 172.21.15.16:0/1336725500 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa1600cd0e0 0x7fa1600ac8e0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fa1540ab000 tx=0x7fa15401ac90 comp rx=0 tx=0).stop 2024-03-31T15:04:24.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:24.556+0000 7fa168200640 1 -- 172.21.15.16:0/1336725500 shutdown_connections 2024-03-31T15:04:24.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:24.556+0000 7fa168200640 1 --2- 172.21.15.16:0/1336725500 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fa140068380 0x7fa14006a840 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:24.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:24.556+0000 7fa168200640 1 --2- 172.21.15.16:0/1336725500 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa16013e3d0 0x7fa160156ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:24.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:24.556+0000 7fa168200640 1 --2- 172.21.15.16:0/1336725500 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fa16013dff0 0x7fa160156590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:24.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:24.556+0000 7fa168200640 1 --2- 172.21.15.16:0/1336725500 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa1600cd0e0 0x7fa1600ac8e0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:24.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:24.556+0000 7fa168200640 1 -- 172.21.15.16:0/1336725500 >> 172.21.15.16:0/1336725500 conn(0x7fa16006ee40 msgr2=0x7fa1600b2560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:04:24.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:24.556+0000 7fa168200640 1 -- 172.21.15.16:0/1336725500 shutdown_connections 2024-03-31T15:04:24.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:24.556+0000 7fa168200640 1 -- 172.21.15.16:0/1336725500 wait complete. 2024-03-31T15:04:24.569 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-31T15:04:24.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:24.720+0000 7f136f200640 1 Processor -- start 2024-03-31T15:04:24.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:24.720+0000 7f136f200640 1 -- start start 2024-03-31T15:04:24.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:24.720+0000 7f136f200640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f136813e3d0 0x7f13680cc9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:24.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:24.720+0000 7f136f200640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f136815fbc0 con 0x7f136813dff0 2024-03-31T15:04:24.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:24.720+0000 7f136f200640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f136815ee00 con 0x7f136806dbe0 2024-03-31T15:04:24.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:24.720+0000 7f136f200640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f136815f730 con 0x7f136813e3d0 2024-03-31T15:04:24.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:24.720+0000 7f1367e00640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f136813dff0 0x7f136806d4d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:44466/0 (socket says 172.21.15.16:44466) 2024-03-31T15:04:24.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:24.720+0000 7f1367e00640 1 -- 172.21.15.16:0/3442452321 learned_addr learned my addr 172.21.15.16:0/3442452321 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:04:24.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:24.720+0000 7f135f400640 1 --2- 172.21.15.16:0/3442452321 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f136813e3d0 0x7f13680cc9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:24.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:24.720+0000 7f1367400640 1 -- 172.21.15.16:0/3442452321 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2820027348 0 0) 0x7f136815fbc0 con 0x7f136813dff0 2024-03-31T15:04:24.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:24.720+0000 7f1367400640 1 -- 172.21.15.16:0/3442452321 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f134c003530 con 0x7f136813dff0 2024-03-31T15:04:24.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:24.720+0000 7f135f400640 1 -- 172.21.15.16:0/3442452321 >> v1:172.21.15.89:6789/0 conn(0x7f136806dbe0 legacy=0x7f13680ceb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:24.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:24.720+0000 7f1367400640 1 -- 172.21.15.16:0/3442452321 <== mon.1 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1795905386 0 0) 0x7f136815ee00 con 0x7f136806dbe0 2024-03-31T15:04:24.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:24.720+0000 7f135f400640 1 -- 172.21.15.16:0/3442452321 >> v1:172.21.15.16:6789/0 conn(0x7f136813dff0 legacy=0x7f136806d4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:24.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:24.720+0000 7f135f400640 1 -- 172.21.15.16:0/3442452321 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f136815f0e0 con 0x7f136813e3d0 2024-03-31T15:04:24.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:24.724+0000 7f135f400640 1 --2- 172.21.15.16:0/3442452321 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f136813e3d0 0x7f13680cc9d0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f13500097f0 tx=0x7f135001c450 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2d7fe581996a4de7 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:24.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:24.724+0000 7f1367400640 1 -- 172.21.15.16:0/3442452321 <== mon.2 v2:172.21.15.89:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13500b0070 con 0x7f136813e3d0 2024-03-31T15:04:24.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:24.724+0000 7f1367400640 1 -- 172.21.15.16:0/3442452321 <== mon.2 v2:172.21.15.89:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f13500ab600 con 0x7f136813e3d0 2024-03-31T15:04:24.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:24.724+0000 7f1367400640 1 -- 172.21.15.16:0/3442452321 <== mon.2 v2:172.21.15.89:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13500ab9d0 con 0x7f136813e3d0 2024-03-31T15:04:24.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:24.724+0000 7f136f200640 1 -- 172.21.15.16:0/3442452321 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f136813e3d0 msgr2=0x7f13680cc9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:24.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:24.724+0000 7f136f200640 1 --2- 172.21.15.16:0/3442452321 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f136813e3d0 0x7f13680cc9d0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f13500097f0 tx=0x7f135001c450 comp rx=0 tx=0).stop 2024-03-31T15:04:24.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:24.724+0000 7f136f200640 1 -- 172.21.15.16:0/3442452321 shutdown_connections 2024-03-31T15:04:24.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:24.724+0000 7f136f200640 1 --2- 172.21.15.16:0/3442452321 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f136813e3d0 0x7f13680cc9d0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:24.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:24.724+0000 7f136f200640 1 -- 172.21.15.16:0/3442452321 >> 172.21.15.16:0/3442452321 conn(0x7f136806ee40 msgr2=0x7f136806b620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:04:24.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:24.724+0000 7f136f200640 1 -- 172.21.15.16:0/3442452321 shutdown_connections 2024-03-31T15:04:24.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:24.724+0000 7f136f200640 1 -- 172.21.15.16:0/3442452321 wait complete. 2024-03-31T15:04:24.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:24.724+0000 7f136f200640 1 Processor -- start 2024-03-31T15:04:24.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:24.724+0000 7f136f200640 1 -- start start 2024-03-31T15:04:24.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:24.728+0000 7f136f200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f136806dbe0 0x7f1368139db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:24.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:24.728+0000 7f136f200640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f136813dff0 0x7f1368152550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:24.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:24.728+0000 7f136f200640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f136813e3d0 0x7f136813f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:24.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:24.728+0000 7f136f200640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f1368169080 con 0x7f136806dbe0 2024-03-31T15:04:24.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:24.728+0000 7f136f200640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f13680cfca0 con 0x7f136813dff0 2024-03-31T15:04:24.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:24.728+0000 7f136f200640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f1368169200 con 0x7f136813e3d0 2024-03-31T15:04:24.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:24.728+0000 7f1367e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f136806dbe0 0x7f1368139db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:24.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:24.728+0000 7f1367e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f136806dbe0 0x7f1368139db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49380/0 (socket says 172.21.15.16:49380) 2024-03-31T15:04:24.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:24.728+0000 7f1367e00640 1 -- 172.21.15.16:0/130051165 learned_addr learned my addr 172.21.15.16:0/130051165 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:04:24.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:24.728+0000 7f135f400640 1 --2- 172.21.15.16:0/130051165 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f136813dff0 0x7f1368152550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:24.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:24.728+0000 7f136d200640 1 --2- 172.21.15.16:0/130051165 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f136813e3d0 0x7f136813f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:24.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:24.728+0000 7f1367e00640 1 -- 172.21.15.16:0/130051165 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f136813e3d0 msgr2=0x7f136813f0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:24.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:24.728+0000 7f1367e00640 1 --2- 172.21.15.16:0/130051165 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f136813e3d0 0x7f136813f0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:24.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:24.728+0000 7f1367e00640 1 -- 172.21.15.16:0/130051165 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f136813dff0 msgr2=0x7f1368152550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:24.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:24.728+0000 7f1367e00640 1 --2- 172.21.15.16:0/130051165 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f136813dff0 0x7f1368152550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:24.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:24.728+0000 7f1367e00640 1 -- 172.21.15.16:0/130051165 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1368152f80 con 0x7f136806dbe0 2024-03-31T15:04:24.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:24.728+0000 7f1367e00640 1 --2- 172.21.15.16:0/130051165 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f136806dbe0 0x7f1368139db0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f135800c480 tx=0x7f135800c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:24.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:24.728+0000 7f1364c00640 1 -- 172.21.15.16:0/130051165 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1358017070 con 0x7f136806dbe0 2024-03-31T15:04:24.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:24.728+0000 7f1364c00640 1 -- 172.21.15.16:0/130051165 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1358005cf0 con 0x7f136806dbe0 2024-03-31T15:04:24.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:24.728+0000 7f136f200640 1 -- 172.21.15.16:0/130051165 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1368153260 con 0x7f136806dbe0 2024-03-31T15:04:24.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:24.728+0000 7f1364c00640 1 -- 172.21.15.16:0/130051165 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1358007290 con 0x7f136806dbe0 2024-03-31T15:04:24.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:24.728+0000 7f136f200640 1 -- 172.21.15.16:0/130051165 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1368153820 con 0x7f136806dbe0 2024-03-31T15:04:24.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:24.728+0000 7f136f200640 1 -- 172.21.15.16:0/130051165 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1368069950 con 0x7f136806dbe0 2024-03-31T15:04:24.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:24.728+0000 7f1364c00640 1 -- 172.21.15.16:0/130051165 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f135801a070 con 0x7f136806dbe0 2024-03-31T15:04:24.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:24.732+0000 7f1364c00640 1 --2- 172.21.15.16:0/130051165 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f1344068380 0x7f134406a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:24.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:24.732+0000 7f1364c00640 1 -- 172.21.15.16:0/130051165 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f1358085dc0 con 0x7f136806dbe0 2024-03-31T15:04:24.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:24.732+0000 7f135f400640 1 --2- 172.21.15.16:0/130051165 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f1344068380 0x7f134406a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:24.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:24.732+0000 7f135f400640 1 --2- 172.21.15.16:0/130051165 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f1344068380 0x7f134406a840 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f13500ad000 tx=0x7f13500a6000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:24.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:24.736+0000 7f1364c00640 1 -- 172.21.15.16:0/130051165 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f1358055ca0 con 0x7f136806dbe0 2024-03-31T15:04:24.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:24.900+0000 7f136f200640 1 -- 172.21.15.16:0/130051165 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f136815ca30 con 0x7f136806dbe0 2024-03-31T15:04:24.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:24.900+0000 7f1364c00640 1 -- 172.21.15.16:0/130051165 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v23) ==== 100+0+11 (secure 0 0 0) 0x7f1358059fa0 con 0x7f136806dbe0 2024-03-31T15:04:24.896 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 64 2024-03-31T15:04:24.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:24.904+0000 7f136f200640 1 -- 172.21.15.16:0/130051165 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f1344068380 msgr2=0x7f134406a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:24.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:24.904+0000 7f136f200640 1 --2- 172.21.15.16:0/130051165 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f1344068380 0x7f134406a840 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f13500ad000 tx=0x7f13500a6000 comp rx=0 tx=0).stop 2024-03-31T15:04:24.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:24.904+0000 7f136f200640 1 -- 172.21.15.16:0/130051165 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f136806dbe0 msgr2=0x7f1368139db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:24.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:24.904+0000 7f136f200640 1 --2- 172.21.15.16:0/130051165 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f136806dbe0 0x7f1368139db0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f135800c480 tx=0x7f135800c940 comp rx=0 tx=0).stop 2024-03-31T15:04:24.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:24.904+0000 7f136f200640 1 -- 172.21.15.16:0/130051165 shutdown_connections 2024-03-31T15:04:24.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:24.904+0000 7f136f200640 1 --2- 172.21.15.16:0/130051165 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f1344068380 0x7f134406a840 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:24.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:24.904+0000 7f136f200640 1 --2- 172.21.15.16:0/130051165 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f136813e3d0 0x7f136813f0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:24.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:24.904+0000 7f136f200640 1 --2- 172.21.15.16:0/130051165 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f136813dff0 0x7f1368152550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:24.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:24.904+0000 7f136f200640 1 --2- 172.21.15.16:0/130051165 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f136806dbe0 0x7f1368139db0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:24.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:24.904+0000 7f136f200640 1 -- 172.21.15.16:0/130051165 >> 172.21.15.16:0/130051165 conn(0x7f136806ee40 msgr2=0x7f136815b750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:04:24.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:24.904+0000 7f136f200640 1 -- 172.21.15.16:0/130051165 shutdown_connections 2024-03-31T15:04:24.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:24.904+0000 7f136f200640 1 -- 172.21.15.16:0/130051165 wait complete. 2024-03-31T15:04:24.917 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-31T15:04:25.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:25.064+0000 7f2a29e00640 1 Processor -- start 2024-03-31T15:04:25.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:25.064+0000 7f2a29e00640 1 -- start start 2024-03-31T15:04:25.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:25.068+0000 7f2a29e00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f2a2406dbc0 0x7f2a2415f650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:25.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:25.068+0000 7f2a29e00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f2a240cb440 0x7f2a2406f920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:25.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:25.068+0000 7f2a29e00640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2a240cbe90 con 0x7f2a240cb8c0 2024-03-31T15:04:25.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:25.068+0000 7f2a29e00640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f2a2415fb90 con 0x7f2a2406dbc0 2024-03-31T15:04:25.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:25.068+0000 7f2a29e00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f2a2415f1c0 con 0x7f2a240cb440 2024-03-31T15:04:25.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:25.068+0000 7f2a22a00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f2a2406dbc0 0x7f2a2415f650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:25.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:25.068+0000 7f2a1a000640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f2a240cb440 0x7f2a2406f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:25.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:25.068+0000 7f2a22a00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f2a2406dbc0 0x7f2a2415f650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.16:49216/0 (socket says 172.21.15.16:49216) 2024-03-31T15:04:25.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:25.068+0000 7f2a23400640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f2a240cb8c0 0x7f2a24162180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.16:52808/0 (socket says 172.21.15.16:52808) 2024-03-31T15:04:25.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:25.068+0000 7f2a22a00640 1 -- 172.21.15.16:0/204800198 learned_addr learned my addr 172.21.15.16:0/204800198 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:04:25.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:25.068+0000 7f2a22a00640 1 -- 172.21.15.16:0/204800198 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f2a240cb440 msgr2=0x7f2a2406f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:04:25.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:25.068+0000 7f2a22a00640 1 --2- 172.21.15.16:0/204800198 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f2a240cb440 0x7f2a2406f920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:25.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:25.068+0000 7f2a22a00640 1 -- 172.21.15.16:0/204800198 >> v1:172.21.15.89:6789/0 conn(0x7f2a240cb8c0 legacy=0x7f2a24162180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:25.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:25.068+0000 7f2a22a00640 1 -- 172.21.15.16:0/204800198 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a240cc170 con 0x7f2a2406dbc0 2024-03-31T15:04:25.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:25.072+0000 7f2a22a00640 1 --2- 172.21.15.16:0/204800198 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2a2406dbc0 0x7f2a2415f650 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f2a14009e50 tx=0x7f2a1401c970 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c16237bca8ece8d7 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:25.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:25.072+0000 7f2a22000640 1 -- 172.21.15.16:0/204800198 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a140a5850 con 0x7f2a2406dbc0 2024-03-31T15:04:25.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:25.072+0000 7f2a22000640 1 -- 172.21.15.16:0/204800198 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2a140a59f0 con 0x7f2a2406dbc0 2024-03-31T15:04:25.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:25.072+0000 7f2a22000640 1 -- 172.21.15.16:0/204800198 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a140a5dc0 con 0x7f2a2406dbc0 2024-03-31T15:04:25.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:25.072+0000 7f2a29e00640 1 -- 172.21.15.16:0/204800198 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2a2406dbc0 msgr2=0x7f2a2415f650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:25.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:25.072+0000 7f2a29e00640 1 --2- 172.21.15.16:0/204800198 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2a2406dbc0 0x7f2a2415f650 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f2a14009e50 tx=0x7f2a1401c970 comp rx=0 tx=0).stop 2024-03-31T15:04:25.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:25.072+0000 7f2a29e00640 1 -- 172.21.15.16:0/204800198 shutdown_connections 2024-03-31T15:04:25.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:25.072+0000 7f2a29e00640 1 --2- 172.21.15.16:0/204800198 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f2a240cb440 0x7f2a2406f920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:25.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:25.072+0000 7f2a29e00640 1 --2- 172.21.15.16:0/204800198 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2a2406dbc0 0x7f2a2415f650 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:25.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:25.072+0000 7f2a29e00640 1 -- 172.21.15.16:0/204800198 >> 172.21.15.16:0/204800198 conn(0x7f2a24067330 msgr2=0x7f2a24067740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:04:25.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:25.072+0000 7f2a29e00640 1 -- 172.21.15.16:0/204800198 shutdown_connections 2024-03-31T15:04:25.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:25.072+0000 7f2a29e00640 1 -- 172.21.15.16:0/204800198 wait complete. 2024-03-31T15:04:25.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:25.072+0000 7f2a29e00640 1 Processor -- start 2024-03-31T15:04:25.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:25.072+0000 7f2a29e00640 1 -- start start 2024-03-31T15:04:25.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:25.076+0000 7f2a29e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2a2406dbc0 0x7f2a24163290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:25.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:25.076+0000 7f2a29e00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2a240cb440 0x7f2a2415aaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:25.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:25.076+0000 7f2a29e00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f2a240cb8c0 0x7f2a240b57a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:25.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:25.076+0000 7f2a29e00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f2a24162ca0 con 0x7f2a2406dbc0 2024-03-31T15:04:25.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:25.076+0000 7f2a29e00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f2a24070270 con 0x7f2a240cb440 2024-03-31T15:04:25.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:25.076+0000 7f2a29e00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f2a2406c950 con 0x7f2a240cb8c0 2024-03-31T15:04:25.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:25.076+0000 7f2a22a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2a2406dbc0 0x7f2a24163290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:25.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:25.076+0000 7f2a1a000640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2a240cb440 0x7f2a2415aaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:25.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:25.076+0000 7f2a22a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2a2406dbc0 0x7f2a24163290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49392/0 (socket says 172.21.15.16:49392) 2024-03-31T15:04:25.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:25.076+0000 7f2a1a000640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2a240cb440 0x7f2a2415aaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.16:49230/0 (socket says 172.21.15.16:49230) 2024-03-31T15:04:25.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:25.076+0000 7f2a23400640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f2a240cb8c0 0x7f2a240b57a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:25.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:25.076+0000 7f2a22a00640 1 -- 172.21.15.16:0/2246459192 learned_addr learned my addr 172.21.15.16:0/2246459192 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:04:25.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:25.076+0000 7f2a22a00640 1 -- 172.21.15.16:0/2246459192 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f2a240cb8c0 msgr2=0x7f2a240b57a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:25.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:25.076+0000 7f2a22a00640 1 --2- 172.21.15.16:0/2246459192 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f2a240cb8c0 0x7f2a240b57a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:25.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:25.076+0000 7f2a22a00640 1 -- 172.21.15.16:0/2246459192 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2a240cb440 msgr2=0x7f2a2415aaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:25.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:25.076+0000 7f2a22a00640 1 --2- 172.21.15.16:0/2246459192 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2a240cb440 0x7f2a2415aaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:25.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:25.076+0000 7f2a22a00640 1 -- 172.21.15.16:0/2246459192 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a24163990 con 0x7f2a2406dbc0 2024-03-31T15:04:25.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:25.076+0000 7f2a1a000640 1 --2- 172.21.15.16:0/2246459192 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2a240cb440 0x7f2a2415aaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:04:25.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:25.076+0000 7f2a22a00640 1 --2- 172.21.15.16:0/2246459192 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2a2406dbc0 0x7f2a24163290 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f2a1401a5a0 tx=0x7f2a14019710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:25.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:25.076+0000 7f2a1be00640 1 -- 172.21.15.16:0/2246459192 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a140bc020 con 0x7f2a2406dbc0 2024-03-31T15:04:25.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:25.076+0000 7f2a1be00640 1 -- 172.21.15.16:0/2246459192 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2a140b8670 con 0x7f2a2406dbc0 2024-03-31T15:04:25.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:25.076+0000 7f2a29e00640 1 -- 172.21.15.16:0/2246459192 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2a24163c70 con 0x7f2a2406dbc0 2024-03-31T15:04:25.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:25.076+0000 7f2a1be00640 1 -- 172.21.15.16:0/2246459192 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a140b8a40 con 0x7f2a2406dbc0 2024-03-31T15:04:25.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:25.076+0000 7f2a29e00640 1 -- 172.21.15.16:0/2246459192 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2a240b5280 con 0x7f2a2406dbc0 2024-03-31T15:04:25.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:25.076+0000 7f2a29e00640 1 -- 172.21.15.16:0/2246459192 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2a24069950 con 0x7f2a2406dbc0 2024-03-31T15:04:25.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:25.076+0000 7f2a1be00640 1 -- 172.21.15.16:0/2246459192 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f2a140bf020 con 0x7f2a2406dbc0 2024-03-31T15:04:25.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:25.080+0000 7f2a1be00640 1 --2- 172.21.15.16:0/2246459192 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f2a00068380 0x7f2a0006a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:25.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:25.080+0000 7f2a1be00640 1 -- 172.21.15.16:0/2246459192 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f2a141219d0 con 0x7f2a2406dbc0 2024-03-31T15:04:25.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:25.080+0000 7f2a1a000640 1 --2- 172.21.15.16:0/2246459192 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f2a00068380 0x7f2a0006a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:25.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:25.080+0000 7f2a1a000640 1 --2- 172.21.15.16:0/2246459192 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f2a00068380 0x7f2a0006a840 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f2a24068930 tx=0x7f2a0c009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:25.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:25.084+0000 7f2a1be00640 1 -- 172.21.15.16:0/2246459192 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f2a140c6030 con 0x7f2a2406dbc0 2024-03-31T15:04:25.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:25.260+0000 7f2a29e00640 1 -- 172.21.15.16:0/2246459192 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f2a240b5450 con 0x7f2a2406dbc0 2024-03-31T15:04:25.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:25.260+0000 7f2a1be00640 1 -- 172.21.15.16:0/2246459192 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v23) ==== 80+0+0 (secure 0 0 0) 0x7f2a14016620 con 0x7f2a2406dbc0 2024-03-31T15:04:25.255 INFO:teuthology.orchestra.run.smithi016.stderr:listed 0 entries 2024-03-31T15:04:25.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:25.264+0000 7f2a29e00640 1 -- 172.21.15.16:0/2246459192 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f2a00068380 msgr2=0x7f2a0006a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:25.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:25.264+0000 7f2a29e00640 1 --2- 172.21.15.16:0/2246459192 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f2a00068380 0x7f2a0006a840 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f2a24068930 tx=0x7f2a0c009190 comp rx=0 tx=0).stop 2024-03-31T15:04:25.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:25.264+0000 7f2a29e00640 1 -- 172.21.15.16:0/2246459192 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2a2406dbc0 msgr2=0x7f2a24163290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:25.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:25.264+0000 7f2a29e00640 1 --2- 172.21.15.16:0/2246459192 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2a2406dbc0 0x7f2a24163290 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f2a1401a5a0 tx=0x7f2a14019710 comp rx=0 tx=0).stop 2024-03-31T15:04:25.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:25.264+0000 7f2a29e00640 1 -- 172.21.15.16:0/2246459192 shutdown_connections 2024-03-31T15:04:25.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:25.264+0000 7f2a29e00640 1 --2- 172.21.15.16:0/2246459192 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f2a00068380 0x7f2a0006a840 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:25.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:25.264+0000 7f2a29e00640 1 --2- 172.21.15.16:0/2246459192 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f2a240cb8c0 0x7f2a240b57a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:25.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:25.264+0000 7f2a29e00640 1 --2- 172.21.15.16:0/2246459192 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2a240cb440 0x7f2a2415aaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:25.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:25.264+0000 7f2a29e00640 1 --2- 172.21.15.16:0/2246459192 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2a2406dbc0 0x7f2a24163290 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:25.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:25.264+0000 7f2a29e00640 1 -- 172.21.15.16:0/2246459192 >> 172.21.15.16:0/2246459192 conn(0x7f2a24067330 msgr2=0x7f2a2415c7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:04:25.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:25.264+0000 7f2a29e00640 1 -- 172.21.15.16:0/2246459192 shutdown_connections 2024-03-31T15:04:25.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:25.264+0000 7f2a29e00640 1 -- 172.21.15.16:0/2246459192 wait complete. 2024-03-31T15:04:25.276 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi089.front.sepia.ceph.com... 2024-03-31T15:04:25.276 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:04:25.276 DEBUG:teuthology.orchestra.run.smithi089:> sudo fusermount -u /home/ubuntu/cephtest/mnt.3 2024-03-31T15:04:25.297 INFO:teuthology.orchestra.run:waiting for 300 2024-03-31T15:04:25.297 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi089.front.sepia.ceph.com 2024-03-31T15:04:25.297 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:04:25.297 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-03-31T15:04:25.347 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-03-31T15:04:25.347 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-31T15:04:25.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:25.492+0000 7fa581400640 1 Processor -- start 2024-03-31T15:04:25.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:25.492+0000 7fa581400640 1 -- start start 2024-03-31T15:04:25.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:25.496+0000 7fa581400640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fa57c0d0c50 0x7fa57c072fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:25.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:25.496+0000 7fa581400640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fa57c073510 0x7fa57c15eb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:25.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:25.496+0000 7fa581400640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa57c06bd30 con 0x7fa57c147e70 2024-03-31T15:04:25.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:25.496+0000 7fa581400640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7fa57c0739a0 con 0x7fa57c073510 2024-03-31T15:04:25.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:25.496+0000 7fa581400640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fa57c1621b0 con 0x7fa57c0d0c50 2024-03-31T15:04:25.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:25.496+0000 7fa57a800640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fa57c147e70 0x7fa57c06b620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:44476/0 (socket says 172.21.15.16:44476) 2024-03-31T15:04:25.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:25.496+0000 7fa57a800640 1 -- 172.21.15.16:0/983277777 learned_addr learned my addr 172.21.15.16:0/983277777 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:04:25.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:25.496+0000 7fa57b200640 1 --2- 172.21.15.16:0/983277777 >> v2:172.21.15.16:3300/0 conn(0x7fa57c073510 0x7fa57c15eb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:25.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:25.496+0000 7fa571e00640 1 --2- 172.21.15.16:0/983277777 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fa57c0d0c50 0x7fa57c072fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:25.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:25.496+0000 7fa579e00640 1 -- 172.21.15.16:0/983277777 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3505543714 0 0) 0x7fa57c06bd30 con 0x7fa57c147e70 2024-03-31T15:04:25.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:25.496+0000 7fa579e00640 1 -- 172.21.15.16:0/983277777 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa560003530 con 0x7fa57c147e70 2024-03-31T15:04:25.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:25.496+0000 7fa57b200640 1 -- 172.21.15.16:0/983277777 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fa57c0d0c50 msgr2=0x7fa57c072fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:04:25.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:25.496+0000 7fa57b200640 1 --2- 172.21.15.16:0/983277777 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fa57c0d0c50 0x7fa57c072fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:25.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:25.496+0000 7fa57b200640 1 -- 172.21.15.16:0/983277777 >> v1:172.21.15.16:6789/0 conn(0x7fa57c147e70 legacy=0x7fa57c06b620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:25.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:25.496+0000 7fa57b200640 1 -- 172.21.15.16:0/983277777 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa57c162490 con 0x7fa57c073510 2024-03-31T15:04:25.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:25.496+0000 7fa571e00640 1 --2- 172.21.15.16:0/983277777 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fa57c0d0c50 0x7fa57c072fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:04:25.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:25.496+0000 7fa57b200640 1 --2- 172.21.15.16:0/983277777 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa57c073510 0x7fa57c15eb20 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fa574009d90 tx=0x7fa57401c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b59543d1def3694 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:25.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:25.500+0000 7fa579e00640 1 -- 172.21.15.16:0/983277777 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5740b2070 con 0x7fa57c073510 2024-03-31T15:04:25.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:25.500+0000 7fa579e00640 1 -- 172.21.15.16:0/983277777 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa5740a5920 con 0x7fa57c073510 2024-03-31T15:04:25.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:25.500+0000 7fa579e00640 1 -- 172.21.15.16:0/983277777 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5740a5cf0 con 0x7fa57c073510 2024-03-31T15:04:25.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:25.500+0000 7fa581400640 1 -- 172.21.15.16:0/983277777 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa57c073510 msgr2=0x7fa57c15eb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:25.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:25.500+0000 7fa581400640 1 --2- 172.21.15.16:0/983277777 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa57c073510 0x7fa57c15eb20 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fa574009d90 tx=0x7fa57401c970 comp rx=0 tx=0).stop 2024-03-31T15:04:25.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:25.500+0000 7fa581400640 1 -- 172.21.15.16:0/983277777 shutdown_connections 2024-03-31T15:04:25.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:25.500+0000 7fa581400640 1 --2- 172.21.15.16:0/983277777 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa57c073510 0x7fa57c15eb20 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:25.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:25.500+0000 7fa581400640 1 --2- 172.21.15.16:0/983277777 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fa57c0d0c50 0x7fa57c072fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:25.494 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:25.500+0000 7fa581400640 1 -- 172.21.15.16:0/983277777 >> 172.21.15.16:0/983277777 conn(0x7fa57c0d1060 msgr2=0x7fa57c0d1470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:04:25.494 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:25.500+0000 7fa581400640 1 -- 172.21.15.16:0/983277777 shutdown_connections 2024-03-31T15:04:25.494 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:25.500+0000 7fa581400640 1 -- 172.21.15.16:0/983277777 wait complete. 2024-03-31T15:04:25.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:25.500+0000 7fa581400640 1 Processor -- start 2024-03-31T15:04:25.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:25.500+0000 7fa581400640 1 -- start start 2024-03-31T15:04:25.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:25.500+0000 7fa581400640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fa57c073510 0x7fa57c147340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:25.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:25.500+0000 7fa581400640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa57c0d0c50 0x7fa57c15bec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:25.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:25.500+0000 7fa581400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa57c147e70 0x7fa57c06e600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:25.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:25.500+0000 7fa581400640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fa57c15f870 con 0x7fa57c147e70 2024-03-31T15:04:25.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:25.500+0000 7fa581400640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fa57c162f70 con 0x7fa57c0d0c50 2024-03-31T15:04:25.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:25.500+0000 7fa581400640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fa57c15f9f0 con 0x7fa57c073510 2024-03-31T15:04:25.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:25.500+0000 7fa57a800640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fa57c073510 0x7fa57c147340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:25.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:25.500+0000 7fa57a800640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fa57c073510 0x7fa57c147340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3301/0 says I am v2:172.21.15.16:37862/0 (socket says 172.21.15.16:37862) 2024-03-31T15:04:25.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:25.500+0000 7fa571e00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa57c0d0c50 0x7fa57c15bec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:25.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:25.500+0000 7fa57a800640 1 -- 172.21.15.16:0/2594297219 learned_addr learned my addr 172.21.15.16:0/2594297219 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:04:25.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:25.500+0000 7fa57b200640 1 --2- 172.21.15.16:0/2594297219 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa57c147e70 0x7fa57c06e600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:25.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:25.504+0000 7fa57a800640 1 -- 172.21.15.16:0/2594297219 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa57c0d0c50 msgr2=0x7fa57c15bec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:25.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:25.504+0000 7fa57a800640 1 --2- 172.21.15.16:0/2594297219 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa57c0d0c50 0x7fa57c15bec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:25.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:25.504+0000 7fa57a800640 1 -- 172.21.15.16:0/2594297219 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa57c147e70 msgr2=0x7fa57c06e600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:25.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:25.504+0000 7fa57a800640 1 --2- 172.21.15.16:0/2594297219 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa57c147e70 0x7fa57c06e600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:25.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:25.504+0000 7fa57a800640 1 -- 172.21.15.16:0/2594297219 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa57c15cbd0 con 0x7fa57c073510 2024-03-31T15:04:25.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:25.504+0000 7fa57b200640 1 --2- 172.21.15.16:0/2594297219 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa57c147e70 0x7fa57c06e600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:04:25.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:25.504+0000 7fa57a800640 1 --2- 172.21.15.16:0/2594297219 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fa57c073510 0x7fa57c147340 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fa56c00c450 tx=0x7fa56c00c910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:25.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:25.504+0000 7fa573400640 1 -- 172.21.15.16:0/2594297219 <== mon.2 v2:172.21.15.89:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa56c017070 con 0x7fa57c073510 2024-03-31T15:04:25.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:25.504+0000 7fa573400640 1 -- 172.21.15.16:0/2594297219 <== mon.2 v2:172.21.15.89:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa56c005cf0 con 0x7fa57c073510 2024-03-31T15:04:25.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:25.504+0000 7fa581400640 1 -- 172.21.15.16:0/2594297219 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa57c15ceb0 con 0x7fa57c073510 2024-03-31T15:04:25.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:25.504+0000 7fa573400640 1 -- 172.21.15.16:0/2594297219 <== mon.2 v2:172.21.15.89:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa56c007290 con 0x7fa57c073510 2024-03-31T15:04:25.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:25.504+0000 7fa581400640 1 -- 172.21.15.16:0/2594297219 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fa57c15d470 con 0x7fa57c073510 2024-03-31T15:04:25.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:25.504+0000 7fa581400640 1 -- 172.21.15.16:0/2594297219 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa57c069950 con 0x7fa57c073510 2024-03-31T15:04:25.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:25.504+0000 7fa573400640 1 -- 172.21.15.16:0/2594297219 <== mon.2 v2:172.21.15.89:3301/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fa56c01a070 con 0x7fa57c073510 2024-03-31T15:04:25.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:25.504+0000 7fa573400640 1 --2- 172.21.15.16:0/2594297219 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fa558068380 0x7fa55806a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:25.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:25.504+0000 7fa573400640 1 -- 172.21.15.16:0/2594297219 <== mon.2 v2:172.21.15.89:3301/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7fa56c085db0 con 0x7fa57c073510 2024-03-31T15:04:25.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:25.504+0000 7fa571e00640 1 --2- 172.21.15.16:0/2594297219 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fa558068380 0x7fa55806a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:25.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:25.508+0000 7fa571e00640 1 --2- 172.21.15.16:0/2594297219 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fa558068380 0x7fa55806a840 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fa564009580 tx=0x7fa564009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:25.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:25.508+0000 7fa573400640 1 -- 172.21.15.16:0/2594297219 <== mon.2 v2:172.21.15.89:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fa56c08a070 con 0x7fa57c073510 2024-03-31T15:04:25.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:25.708+0000 7fa581400640 1 -- 172.21.15.16:0/2594297219 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa57c0d1bf0 con 0x7fa57c073510 2024-03-31T15:04:25.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:25.712+0000 7fa573400640 1 -- 172.21.15.16:0/2594297219 <== mon.2 v2:172.21.15.89:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+5172 (secure 0 0 0) 0x7fa56c055c00 con 0x7fa57c073510 2024-03-31T15:04:25.705 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:04:25.705 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4276,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/2832752118","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6833","nonce":2832752118},{"type":"v1","addr":"172.21.15.16:6835","nonce":2832752118}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4278,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6834/153294668","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":153294668},{"type":"v1","addr":"172.21.15.16:6834","nonce":153294668}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4288,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/1653319357","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":1653319357},{"type":"v1","addr":"172.21.15.89:6835","nonce":1653319357}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-31T15:03:36.349865+0000","modified":"2024-03-31T15:03:39.378181+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4293},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.89:6837/2583453131","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":2583453131},{"type":"v1","addr":"172.21.15.89:6837","nonce":2583453131}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4293,"qdb_cluster":[4293]},"id":1}]} 2024-03-31T15:04:25.705 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 8 2024-03-31T15:04:25.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:25.712+0000 7fa581400640 1 -- 172.21.15.16:0/2594297219 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fa558068380 msgr2=0x7fa55806a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:25.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:25.712+0000 7fa581400640 1 --2- 172.21.15.16:0/2594297219 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fa558068380 0x7fa55806a840 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fa564009580 tx=0x7fa564009190 comp rx=0 tx=0).stop 2024-03-31T15:04:25.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:25.712+0000 7fa581400640 1 -- 172.21.15.16:0/2594297219 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fa57c073510 msgr2=0x7fa57c147340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:25.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:25.712+0000 7fa581400640 1 --2- 172.21.15.16:0/2594297219 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fa57c073510 0x7fa57c147340 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fa56c00c450 tx=0x7fa56c00c910 comp rx=0 tx=0).stop 2024-03-31T15:04:25.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:25.712+0000 7fa581400640 1 -- 172.21.15.16:0/2594297219 shutdown_connections 2024-03-31T15:04:25.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:25.712+0000 7fa581400640 1 --2- 172.21.15.16:0/2594297219 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fa558068380 0x7fa55806a840 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:25.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:25.712+0000 7fa581400640 1 --2- 172.21.15.16:0/2594297219 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa57c147e70 0x7fa57c06e600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:25.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:25.712+0000 7fa581400640 1 --2- 172.21.15.16:0/2594297219 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa57c0d0c50 0x7fa57c15bec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:25.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:25.712+0000 7fa581400640 1 --2- 172.21.15.16:0/2594297219 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fa57c073510 0x7fa57c147340 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:25.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:25.712+0000 7fa581400640 1 -- 172.21.15.16:0/2594297219 >> 172.21.15.16:0/2594297219 conn(0x7fa57c0d1060 msgr2=0x7fa57c0b2670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:04:25.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:25.712+0000 7fa581400640 1 -- 172.21.15.16:0/2594297219 shutdown_connections 2024-03-31T15:04:25.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:25.712+0000 7fa581400640 1 -- 172.21.15.16:0/2594297219 wait complete. 2024-03-31T15:04:25.727 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-31T15:04:25.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:25.872+0000 7f29d1000640 1 Processor -- start 2024-03-31T15:04:25.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:25.872+0000 7f29d1000640 1 -- start start 2024-03-31T15:04:25.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:25.876+0000 7f29d1000640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f29cc070030 0x7f29cc162180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:25.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:25.876+0000 7f29d1000640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f29cc0cbe90 con 0x7f29cc15f650 2024-03-31T15:04:25.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:25.876+0000 7f29d1000640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f29cc0cc170 con 0x7f29cc06dbc0 2024-03-31T15:04:25.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:25.876+0000 7f29d1000640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f29cc15f1c0 con 0x7f29cc070030 2024-03-31T15:04:25.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:25.876+0000 7f29c1400640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f29cc15f650 0x7f29cc06f920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:44492/0 (socket says 172.21.15.16:44492) 2024-03-31T15:04:25.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:25.876+0000 7f29c1400640 1 -- 172.21.15.16:0/2206373250 learned_addr learned my addr 172.21.15.16:0/2206373250 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:04:25.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:25.876+0000 7f29ca800640 1 --2- 172.21.15.16:0/2206373250 >> v2:172.21.15.89:3300/0 conn(0x7f29cc070030 0x7f29cc162180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:25.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:25.876+0000 7f29c9400640 1 -- 172.21.15.16:0/2206373250 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 509816002 0 0) 0x7f29cc0cbe90 con 0x7f29cc15f650 2024-03-31T15:04:25.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:25.876+0000 7f29c9400640 1 -- 172.21.15.16:0/2206373250 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f29b0003530 con 0x7f29cc15f650 2024-03-31T15:04:25.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:25.876+0000 7f29c9400640 1 -- 172.21.15.16:0/2206373250 <== mon.1 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2614451106 0 0) 0x7f29cc0cc170 con 0x7f29cc06dbc0 2024-03-31T15:04:25.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:25.876+0000 7f29c9400640 1 -- 172.21.15.16:0/2206373250 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f29cc0cbe90 con 0x7f29cc06dbc0 2024-03-31T15:04:25.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:25.876+0000 7f29ca800640 1 -- 172.21.15.16:0/2206373250 >> v1:172.21.15.89:6789/0 conn(0x7f29cc06dbc0 legacy=0x7f29cc0cb440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:25.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:25.876+0000 7f29ca800640 1 -- 172.21.15.16:0/2206373250 >> v1:172.21.15.16:6789/0 conn(0x7f29cc15f650 legacy=0x7f29cc06f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:25.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:25.876+0000 7f29ca800640 1 -- 172.21.15.16:0/2206373250 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29cc15ee50 con 0x7f29cc070030 2024-03-31T15:04:25.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:25.880+0000 7f29ca800640 1 --2- 172.21.15.16:0/2206373250 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f29cc070030 0x7f29cc162180 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f29c4009d90 tx=0x7f29c401c8e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7aa86489a02173f4 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:25.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:25.880+0000 7f29c9400640 1 -- 172.21.15.16:0/2206373250 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29c40b2070 con 0x7f29cc070030 2024-03-31T15:04:25.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:25.880+0000 7f29c9400640 1 -- 172.21.15.16:0/2206373250 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f29c40a5960 con 0x7f29cc070030 2024-03-31T15:04:25.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:25.880+0000 7f29c9400640 1 -- 172.21.15.16:0/2206373250 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29c40a5d30 con 0x7f29cc070030 2024-03-31T15:04:25.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:25.880+0000 7f29d1000640 1 -- 172.21.15.16:0/2206373250 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f29cc070030 msgr2=0x7f29cc162180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:25.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:25.880+0000 7f29d1000640 1 --2- 172.21.15.16:0/2206373250 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f29cc070030 0x7f29cc162180 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f29c4009d90 tx=0x7f29c401c8e0 comp rx=0 tx=0).stop 2024-03-31T15:04:25.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:25.880+0000 7f29d1000640 1 -- 172.21.15.16:0/2206373250 shutdown_connections 2024-03-31T15:04:25.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:25.880+0000 7f29d1000640 1 --2- 172.21.15.16:0/2206373250 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f29cc070030 0x7f29cc162180 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:25.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:25.880+0000 7f29d1000640 1 -- 172.21.15.16:0/2206373250 >> 172.21.15.16:0/2206373250 conn(0x7f29cc067330 msgr2=0x7f29cc067740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:04:25.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:25.880+0000 7f29d1000640 1 -- 172.21.15.16:0/2206373250 shutdown_connections 2024-03-31T15:04:25.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:25.880+0000 7f29d1000640 1 -- 172.21.15.16:0/2206373250 wait complete. 2024-03-31T15:04:25.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:25.880+0000 7f29d1000640 1 Processor -- start 2024-03-31T15:04:25.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:25.880+0000 7f29d1000640 1 -- start start 2024-03-31T15:04:25.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:25.880+0000 7f29d1000640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f29cc06dbc0 0x7f29cc143ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:25.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:25.880+0000 7f29d1000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f29cc070030 0x7f29cc155020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:25.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:25.880+0000 7f29d1000640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f29cc15f650 0x7f29cc155560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:25.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:25.880+0000 7f29d1000640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f29cc06c950 con 0x7f29cc070030 2024-03-31T15:04:25.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:25.880+0000 7f29d1000640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f29cc06dfa0 con 0x7f29cc06dbc0 2024-03-31T15:04:25.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:25.880+0000 7f29d1000640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f29cc06cad0 con 0x7f29cc15f650 2024-03-31T15:04:25.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:25.884+0000 7f29c1400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f29cc070030 0x7f29cc155020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:25.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:25.884+0000 7f29c1400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f29cc070030 0x7f29cc155020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49420/0 (socket says 172.21.15.16:49420) 2024-03-31T15:04:25.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:25.884+0000 7f29c1400640 1 -- 172.21.15.16:0/4160839662 learned_addr learned my addr 172.21.15.16:0/4160839662 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:04:25.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:25.884+0000 7f29c9e00640 1 --2- 172.21.15.16:0/4160839662 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f29cc06dbc0 0x7f29cc143ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:25.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:25.884+0000 7f29ca800640 1 --2- 172.21.15.16:0/4160839662 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f29cc15f650 0x7f29cc155560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:25.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:25.884+0000 7f29c1400640 1 -- 172.21.15.16:0/4160839662 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f29cc15f650 msgr2=0x7f29cc155560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:25.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:25.884+0000 7f29c1400640 1 --2- 172.21.15.16:0/4160839662 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f29cc15f650 0x7f29cc155560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:25.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:25.884+0000 7f29c1400640 1 -- 172.21.15.16:0/4160839662 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f29cc06dbc0 msgr2=0x7f29cc143ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:25.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:25.884+0000 7f29c1400640 1 --2- 172.21.15.16:0/4160839662 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f29cc06dbc0 0x7f29cc143ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:25.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:25.884+0000 7f29c1400640 1 -- 172.21.15.16:0/4160839662 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29cc13c730 con 0x7f29cc070030 2024-03-31T15:04:25.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:25.884+0000 7f29c1400640 1 --2- 172.21.15.16:0/4160839662 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f29cc070030 0x7f29cc155020 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f29bc00c4d0 tx=0x7f29bc00c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:25.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:25.884+0000 7f29c2a00640 1 -- 172.21.15.16:0/4160839662 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29bc017070 con 0x7f29cc070030 2024-03-31T15:04:25.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:25.884+0000 7f29c2a00640 1 -- 172.21.15.16:0/4160839662 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f29bc005cf0 con 0x7f29cc070030 2024-03-31T15:04:25.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:25.884+0000 7f29c2a00640 1 -- 172.21.15.16:0/4160839662 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29bc007270 con 0x7f29cc070030 2024-03-31T15:04:25.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:25.884+0000 7f29d1000640 1 -- 172.21.15.16:0/4160839662 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f29cc13c450 con 0x7f29cc070030 2024-03-31T15:04:25.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:25.884+0000 7f29d1000640 1 -- 172.21.15.16:0/4160839662 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f29cc0b2120 con 0x7f29cc070030 2024-03-31T15:04:25.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:25.884+0000 7f29d1000640 1 -- 172.21.15.16:0/4160839662 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f29cc069950 con 0x7f29cc070030 2024-03-31T15:04:25.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:25.884+0000 7f29c2a00640 1 -- 172.21.15.16:0/4160839662 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f29bc01a070 con 0x7f29cc070030 2024-03-31T15:04:25.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:25.884+0000 7f29c2a00640 1 --2- 172.21.15.16:0/4160839662 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f29a8068380 0x7f29a806a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:25.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:25.888+0000 7f29c2a00640 1 -- 172.21.15.16:0/4160839662 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f29bc085ce0 con 0x7f29cc070030 2024-03-31T15:04:25.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:25.888+0000 7f29c9e00640 1 --2- 172.21.15.16:0/4160839662 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f29a8068380 0x7f29a806a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:25.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:25.888+0000 7f29c9e00640 1 --2- 172.21.15.16:0/4160839662 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f29a8068380 0x7f29a806a840 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f29cc13b0f0 tx=0x7f29b4009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:25.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:25.888+0000 7f29c2a00640 1 -- 172.21.15.16:0/4160839662 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f29bc055c60 con 0x7f29cc070030 2024-03-31T15:04:26.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:26.060+0000 7f29d1000640 1 -- 172.21.15.16:0/4160839662 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f29cc0cf380 con 0x7f29cc070030 2024-03-31T15:04:26.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:26.060+0000 7f29c2a00640 1 -- 172.21.15.16:0/4160839662 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v23) ==== 74+0+16780 (secure 0 0 0) 0x7f29bc059f60 con 0x7f29cc070030 2024-03-31T15:04:26.054 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:04:26.055 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":23,"fsid":"286eb5da-fb47-4550-b290-d9519c13b9a5","created":"2024-03-31T15:03:11.438952+0000","modified":"2024-03-31T15:04:18.087947+0000","last_up_change":"2024-03-31T15:03:22.885535+0000","last_in_change":"2024-03-31T15:03:13.941667+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-31T15:03:22.550787+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-31T15:03:34.488425+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-31T15:03:35.299116+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"92a66450-ef19-40d8-ac3d-d760b23b0bda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6801","nonce":932712244}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6803","nonce":932712244}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6807","nonce":932712244}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6805","nonce":932712244}]},"public_addr":"172.21.15.16:6801/932712244","cluster_addr":"172.21.15.16:6803/932712244","heartbeat_back_addr":"172.21.15.16:6807/932712244","heartbeat_front_addr":"172.21.15.16:6805/932712244","state":["exists","up"]},{"osd":1,"uuid":"da9d1f2b-1968-4959-b843-faa231988e92","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6810","nonce":1398737533}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6811","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6813","nonce":1398737533}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6821","nonce":1398737533}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6815","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6816","nonce":1398737533}]},"public_addr":"172.21.15.16:6810/1398737533","cluster_addr":"172.21.15.16:6813/1398737533","heartbeat_back_addr":"172.21.15.16:6821/1398737533","heartbeat_front_addr":"172.21.15.16:6816/1398737533","state":["exists","up"]},{"osd":2,"uuid":"01a73662-9a21-4b1f-bcbb-42b9636a4638","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6819","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6822","nonce":3018526646}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6825","nonce":3018526646}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6831","nonce":3018526646}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6827","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6829","nonce":3018526646}]},"public_addr":"172.21.15.16:6822/3018526646","cluster_addr":"172.21.15.16:6825/3018526646","heartbeat_back_addr":"172.21.15.16:6831/3018526646","heartbeat_front_addr":"172.21.15.16:6829/3018526646","state":["exists","up"]},{"osd":3,"uuid":"c7771fd3-e18e-4500-ac38-23cf10ebcd75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6809","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6812","nonce":3360352974}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6817","nonce":3360352974}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6828","nonce":3360352974}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6823","nonce":3360352974}]},"public_addr":"172.21.15.16:6812/3360352974","cluster_addr":"172.21.15.16:6817/3360352974","heartbeat_back_addr":"172.21.15.16:6828/3360352974","heartbeat_front_addr":"172.21.15.16:6823/3360352974","state":["exists","up"]},{"osd":4,"uuid":"88260730-1482-4fbd-a40f-cb1124e9df4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6801","nonce":1856844856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6803","nonce":1856844856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6807","nonce":1856844856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6805","nonce":1856844856}]},"public_addr":"172.21.15.89:6801/1856844856","cluster_addr":"172.21.15.89:6803/1856844856","heartbeat_back_addr":"172.21.15.89:6807/1856844856","heartbeat_front_addr":"172.21.15.89:6805/1856844856","state":["exists","up"]},{"osd":5,"uuid":"4cb2c00c-fae0-4dd8-9d27-ec4f0093ccf7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6814","nonce":50395527}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6817","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6820","nonce":50395527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6831","nonce":50395527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6823","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6826","nonce":50395527}]},"public_addr":"172.21.15.89:6814/50395527","cluster_addr":"172.21.15.89:6820/50395527","heartbeat_back_addr":"172.21.15.89:6831/50395527","heartbeat_front_addr":"172.21.15.89:6826/50395527","state":["exists","up"]},{"osd":6,"uuid":"81ce0c19-2554-4cae-acbe-66a05e9e9b9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6809","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6812","nonce":3470471752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6815","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6818","nonce":3470471752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6827","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6829","nonce":3470471752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6821","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6824","nonce":3470471752}]},"public_addr":"172.21.15.89:6812/3470471752","cluster_addr":"172.21.15.89:6818/3470471752","heartbeat_back_addr":"172.21.15.89:6829/3470471752","heartbeat_front_addr":"172.21.15.89:6824/3470471752","state":["exists","up"]},{"osd":7,"uuid":"85e4313b-d37d-442e-8042-153bf20a847d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6811","nonce":1796778630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6813","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6816","nonce":1796778630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6825","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6828","nonce":1796778630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6819","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6822","nonce":1796778630}]},"public_addr":"172.21.15.89:6811/1796778630","cluster_addr":"172.21.15.89:6816/1796778630","heartbeat_back_addr":"172.21.15.89:6828/1796778630","heartbeat_front_addr":"172.21.15.89:6822/1796778630","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.439088+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.529970+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.680512+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:21.044161+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.582111+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.536558+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.409522+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.590229+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.2","mappings":[{"from":3,"to":2}]},{"pgid":"3.7","mappings":[{"from":3,"to":2}]},{"pgid":"3.d","mappings":[{"from":5,"to":1}]},{"pgid":"3.22","mappings":[{"from":5,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-31T15:04:26.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:26.060+0000 7f29d1000640 1 -- 172.21.15.16:0/4160839662 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f29a8068380 msgr2=0x7f29a806a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:26.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:26.060+0000 7f29d1000640 1 --2- 172.21.15.16:0/4160839662 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f29a8068380 0x7f29a806a840 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f29cc13b0f0 tx=0x7f29b4009040 comp rx=0 tx=0).stop 2024-03-31T15:04:26.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:26.060+0000 7f29d1000640 1 -- 172.21.15.16:0/4160839662 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f29cc070030 msgr2=0x7f29cc155020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:26.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:26.060+0000 7f29d1000640 1 --2- 172.21.15.16:0/4160839662 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f29cc070030 0x7f29cc155020 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f29bc00c4d0 tx=0x7f29bc00c990 comp rx=0 tx=0).stop 2024-03-31T15:04:26.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:26.060+0000 7f29d1000640 1 -- 172.21.15.16:0/4160839662 shutdown_connections 2024-03-31T15:04:26.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:26.060+0000 7f29d1000640 1 --2- 172.21.15.16:0/4160839662 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f29a8068380 0x7f29a806a840 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:26.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:26.060+0000 7f29d1000640 1 --2- 172.21.15.16:0/4160839662 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f29cc15f650 0x7f29cc155560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:26.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:26.060+0000 7f29d1000640 1 --2- 172.21.15.16:0/4160839662 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f29cc070030 0x7f29cc155020 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:26.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:26.060+0000 7f29d1000640 1 --2- 172.21.15.16:0/4160839662 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f29cc06dbc0 0x7f29cc143ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:26.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:26.060+0000 7f29d1000640 1 -- 172.21.15.16:0/4160839662 >> 172.21.15.16:0/4160839662 conn(0x7f29cc067330 msgr2=0x7f29cc15c810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:04:26.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:26.064+0000 7f29d1000640 1 -- 172.21.15.16:0/4160839662 shutdown_connections 2024-03-31T15:04:26.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:26.064+0000 7f29d1000640 1 -- 172.21.15.16:0/4160839662 wait complete. 2024-03-31T15:04:26.075 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-31T15:03:22.550787+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-31T15:03:34.488425+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-31T15:03:35.299116+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-31T15:04:26.076 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-31T15:04:26.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:26.224+0000 7f3222c00640 1 Processor -- start 2024-03-31T15:04:26.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:26.224+0000 7f3222c00640 1 -- start start 2024-03-31T15:04:26.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:26.228+0000 7f3222c00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f321c0b1400 0x7f321c06c730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:26.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:26.228+0000 7f3222c00640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f321c0b17e0 con 0x7f321c0d0ce0 2024-03-31T15:04:26.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:26.228+0000 7f3222c00640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f321c0cc5e0 con 0x7f321c0705b0 2024-03-31T15:04:26.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:26.228+0000 7f3222c00640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f321c0cc8a0 con 0x7f321c0b1400 2024-03-31T15:04:26.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:26.228+0000 7f3221800640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f321c0b1400 0x7f321c06c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:26.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:26.228+0000 7f3221800640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f321c0b1400 0x7f321c06c730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49426/0 (socket says 172.21.15.16:49426) 2024-03-31T15:04:26.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:26.228+0000 7f3220e00640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f321c0d0ce0 0x7f321c06fea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:44500/0 (socket says 172.21.15.16:44500) 2024-03-31T15:04:26.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:26.228+0000 7f3221800640 1 -- 172.21.15.16:0/2151568304 learned_addr learned my addr 172.21.15.16:0/2151568304 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:04:26.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:26.228+0000 7f3221800640 1 -- 172.21.15.16:0/2151568304 >> v1:172.21.15.89:6789/0 conn(0x7f321c0705b0 legacy=0x7f321c15dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:26.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:26.228+0000 7f3221800640 1 -- 172.21.15.16:0/2151568304 >> v1:172.21.15.16:6789/0 conn(0x7f321c0d0ce0 legacy=0x7f321c06fea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:26.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:26.228+0000 7f3221800640 1 -- 172.21.15.16:0/2151568304 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f321c0ccb80 con 0x7f321c0b1400 2024-03-31T15:04:26.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:26.232+0000 7f3221800640 1 --2- 172.21.15.16:0/2151568304 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f321c0b1400 0x7f321c06c730 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f3210009870 tx=0x7f321001c530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8fb74b01d660138a server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:26.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:26.232+0000 7f321be00640 1 -- 172.21.15.16:0/2151568304 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32100aa680 con 0x7f321c0b1400 2024-03-31T15:04:26.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:26.232+0000 7f321be00640 1 -- 172.21.15.16:0/2151568304 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f32100af070 con 0x7f321c0b1400 2024-03-31T15:04:26.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:26.232+0000 7f321be00640 1 -- 172.21.15.16:0/2151568304 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32100aaa40 con 0x7f321c0b1400 2024-03-31T15:04:26.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:26.232+0000 7f3222c00640 1 -- 172.21.15.16:0/2151568304 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f321c0b1400 msgr2=0x7f321c06c730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:26.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:26.232+0000 7f3222c00640 1 --2- 172.21.15.16:0/2151568304 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f321c0b1400 0x7f321c06c730 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f3210009870 tx=0x7f321001c530 comp rx=0 tx=0).stop 2024-03-31T15:04:26.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:26.232+0000 7f3222c00640 1 -- 172.21.15.16:0/2151568304 shutdown_connections 2024-03-31T15:04:26.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:26.232+0000 7f3222c00640 1 --2- 172.21.15.16:0/2151568304 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f321c0b1400 0x7f321c06c730 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:26.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:26.232+0000 7f3222c00640 1 -- 172.21.15.16:0/2151568304 >> 172.21.15.16:0/2151568304 conn(0x7f321c0d1f90 msgr2=0x7f321c0d23a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:04:26.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:26.232+0000 7f3222c00640 1 -- 172.21.15.16:0/2151568304 shutdown_connections 2024-03-31T15:04:26.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:26.232+0000 7f3222c00640 1 -- 172.21.15.16:0/2151568304 wait complete. 2024-03-31T15:04:26.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:26.232+0000 7f3222c00640 1 Processor -- start 2024-03-31T15:04:26.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:26.232+0000 7f3222c00640 1 -- start start 2024-03-31T15:04:26.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:26.232+0000 7f3222c00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f321c0705b0 0x7f321c1f8f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:26.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:26.232+0000 7f3222c00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f321c0b1400 0x7f321c0cbd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:26.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:26.232+0000 7f3222c00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f321c0d0ce0 0x7f321c14dea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:26.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:26.232+0000 7f3222c00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f321c0cd660 con 0x7f321c0b1400 2024-03-31T15:04:26.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:26.232+0000 7f3222c00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f321c06f930 con 0x7f321c0705b0 2024-03-31T15:04:26.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:26.232+0000 7f3222c00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f321c0cd7e0 con 0x7f321c0d0ce0 2024-03-31T15:04:26.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:26.232+0000 7f3220e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f321c0b1400 0x7f321c0cbd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:26.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:26.236+0000 7f3221800640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f321c0705b0 0x7f321c1f8f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:26.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:26.236+0000 7f3220e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f321c0b1400 0x7f321c0cbd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49428/0 (socket says 172.21.15.16:49428) 2024-03-31T15:04:26.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:26.236+0000 7f3220e00640 1 -- 172.21.15.16:0/599800655 learned_addr learned my addr 172.21.15.16:0/599800655 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:04:26.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:26.236+0000 7f3222200640 1 --2- 172.21.15.16:0/599800655 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f321c0d0ce0 0x7f321c14dea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:26.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:26.236+0000 7f3220e00640 1 -- 172.21.15.16:0/599800655 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f321c0d0ce0 msgr2=0x7f321c14dea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:26.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:26.236+0000 7f3220e00640 1 --2- 172.21.15.16:0/599800655 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f321c0d0ce0 0x7f321c14dea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:26.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:26.236+0000 7f3220e00640 1 -- 172.21.15.16:0/599800655 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f321c0705b0 msgr2=0x7f321c1f8f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:26.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:26.236+0000 7f3220e00640 1 --2- 172.21.15.16:0/599800655 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f321c0705b0 0x7f321c1f8f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:26.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:26.236+0000 7f3220e00640 1 -- 172.21.15.16:0/599800655 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f321c1535e0 con 0x7f321c0b1400 2024-03-31T15:04:26.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:26.236+0000 7f3221800640 1 --2- 172.21.15.16:0/599800655 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f321c0705b0 0x7f321c1f8f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:04:26.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:26.236+0000 7f3220e00640 1 --2- 172.21.15.16:0/599800655 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f321c0b1400 0x7f321c0cbd60 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f320800c430 tx=0x7f320800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:26.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:26.236+0000 7f3219600640 1 -- 172.21.15.16:0/599800655 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32080074e0 con 0x7f321c0b1400 2024-03-31T15:04:26.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:26.236+0000 7f3219600640 1 -- 172.21.15.16:0/599800655 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3208007680 con 0x7f321c0b1400 2024-03-31T15:04:26.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:26.236+0000 7f3222c00640 1 -- 172.21.15.16:0/599800655 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f321c14e5f0 con 0x7f321c0b1400 2024-03-31T15:04:26.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:26.236+0000 7f3219600640 1 -- 172.21.15.16:0/599800655 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3208004a10 con 0x7f321c0b1400 2024-03-31T15:04:26.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:26.236+0000 7f3222c00640 1 -- 172.21.15.16:0/599800655 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f321c153b80 con 0x7f321c0b1400 2024-03-31T15:04:26.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:26.236+0000 7f3222c00640 1 -- 172.21.15.16:0/599800655 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f321c04f8c0 con 0x7f321c0b1400 2024-03-31T15:04:26.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:26.236+0000 7f3219600640 1 -- 172.21.15.16:0/599800655 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f320801a020 con 0x7f321c0b1400 2024-03-31T15:04:26.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:26.236+0000 7f3219600640 1 --2- 172.21.15.16:0/599800655 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f31fc068380 0x7f31fc06a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:26.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:26.236+0000 7f3219600640 1 -- 172.21.15.16:0/599800655 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f32080860c0 con 0x7f321c0b1400 2024-03-31T15:04:26.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:26.240+0000 7f3221800640 1 --2- 172.21.15.16:0/599800655 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f31fc068380 0x7f31fc06a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:26.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:26.240+0000 7f3221800640 1 --2- 172.21.15.16:0/599800655 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f31fc068380 0x7f31fc06a840 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f32100ae000 tx=0x7f32100a7000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:26.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:26.244+0000 7f3219600640 1 -- 172.21.15.16:0/599800655 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f3208055fb0 con 0x7f321c0b1400 2024-03-31T15:04:26.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:26.408+0000 7f3222c00640 1 -- 172.21.15.16:0/599800655 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f321c153d50 con 0x7f321c0b1400 2024-03-31T15:04:26.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:26.408+0000 7f3219600640 1 -- 172.21.15.16:0/599800655 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v23) ==== 93+0+10 (secure 0 0 0) 0x7f320805a2b0 con 0x7f321c0b1400 2024-03-31T15:04:26.402 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2024-03-31T15:04:26.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:26.408+0000 7f3222c00640 1 -- 172.21.15.16:0/599800655 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f31fc068380 msgr2=0x7f31fc06a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:26.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:26.408+0000 7f3222c00640 1 --2- 172.21.15.16:0/599800655 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f31fc068380 0x7f31fc06a840 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f32100ae000 tx=0x7f32100a7000 comp rx=0 tx=0).stop 2024-03-31T15:04:26.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:26.408+0000 7f3222c00640 1 -- 172.21.15.16:0/599800655 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f321c0b1400 msgr2=0x7f321c0cbd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:26.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:26.408+0000 7f3222c00640 1 --2- 172.21.15.16:0/599800655 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f321c0b1400 0x7f321c0cbd60 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f320800c430 tx=0x7f320800c8f0 comp rx=0 tx=0).stop 2024-03-31T15:04:26.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:26.408+0000 7f3222c00640 1 -- 172.21.15.16:0/599800655 shutdown_connections 2024-03-31T15:04:26.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:26.408+0000 7f3222c00640 1 --2- 172.21.15.16:0/599800655 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f31fc068380 0x7f31fc06a840 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:26.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:26.408+0000 7f3222c00640 1 --2- 172.21.15.16:0/599800655 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f321c0d0ce0 0x7f321c14dea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:26.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:26.408+0000 7f3222c00640 1 --2- 172.21.15.16:0/599800655 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f321c0b1400 0x7f321c0cbd60 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:26.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:26.408+0000 7f3222c00640 1 --2- 172.21.15.16:0/599800655 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f321c0705b0 0x7f321c1f8f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:26.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:26.408+0000 7f3222c00640 1 -- 172.21.15.16:0/599800655 >> 172.21.15.16:0/599800655 conn(0x7f321c0d1f90 msgr2=0x7f321c0d2370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:04:26.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:26.408+0000 7f3222c00640 1 -- 172.21.15.16:0/599800655 shutdown_connections 2024-03-31T15:04:26.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:26.408+0000 7f3222c00640 1 -- 172.21.15.16:0/599800655 wait complete. 2024-03-31T15:04:26.423 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-31T15:04:26.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:26.572+0000 7f75cae00640 1 Processor -- start 2024-03-31T15:04:26.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:26.572+0000 7f75cae00640 1 -- start start 2024-03-31T15:04:26.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:26.572+0000 7f75cae00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f75c41411b0 0x7f75c406f920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:26.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:26.572+0000 7f75cae00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f75c4144340 0x7f75c406fe60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:26.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:26.572+0000 7f75cae00640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f75c4144980 con 0x7f75c4141590 2024-03-31T15:04:26.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:26.572+0000 7f75cae00640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f75c415f910 con 0x7f75c41411b0 2024-03-31T15:04:26.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:26.572+0000 7f75cae00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f75c4162910 con 0x7f75c4144340 2024-03-31T15:04:26.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:26.576+0000 7f75c3e00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f75c41411b0 0x7f75c406f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:26.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:26.576+0000 7f75bb400640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f75c4141590 0x7f75c4143c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:44504/0 (socket says 172.21.15.16:44504) 2024-03-31T15:04:26.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:26.576+0000 7f75c3e00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f75c41411b0 0x7f75c406f920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.16:49278/0 (socket says 172.21.15.16:49278) 2024-03-31T15:04:26.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:26.576+0000 7f75bb400640 1 -- 172.21.15.16:0/1928570323 learned_addr learned my addr 172.21.15.16:0/1928570323 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:04:26.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:26.576+0000 7f75c8e00640 1 --2- 172.21.15.16:0/1928570323 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f75c4144340 0x7f75c406fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:26.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:26.576+0000 7f75c3e00640 1 -- 172.21.15.16:0/1928570323 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f75c4144340 msgr2=0x7f75c406fe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:04:26.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:26.576+0000 7f75c3e00640 1 --2- 172.21.15.16:0/1928570323 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f75c4144340 0x7f75c406fe60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:26.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:26.576+0000 7f75c3e00640 1 -- 172.21.15.16:0/1928570323 >> v1:172.21.15.16:6789/0 conn(0x7f75c4141590 legacy=0x7f75c4143c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:26.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:26.576+0000 7f75c3e00640 1 -- 172.21.15.16:0/1928570323 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75c4161fe0 con 0x7f75c41411b0 2024-03-31T15:04:26.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:26.576+0000 7f75c3400640 1 -- 172.21.15.16:0/1928570323 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2631646747 0 0) 0x7f75c4144980 con 0x7f75c4141590 2024-03-31T15:04:26.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:26.576+0000 7f75c3e00640 1 --2- 172.21.15.16:0/1928570323 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f75c41411b0 0x7f75c406f920 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f75b4009d90 tx=0x7f75b401c970 comp rx=0 tx=0).ready entity=mon.1 client_cookie=502696306a039455 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:26.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:26.576+0000 7f75c3400640 1 -- 172.21.15.16:0/1928570323 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75b40b2070 con 0x7f75c41411b0 2024-03-31T15:04:26.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:26.576+0000 7f75c3400640 1 -- 172.21.15.16:0/1928570323 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f75b40a5920 con 0x7f75c41411b0 2024-03-31T15:04:26.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:26.576+0000 7f75c3400640 1 -- 172.21.15.16:0/1928570323 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75b40a5cf0 con 0x7f75c41411b0 2024-03-31T15:04:26.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:26.580+0000 7f75cae00640 1 -- 172.21.15.16:0/1928570323 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f75c41411b0 msgr2=0x7f75c406f920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:26.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:26.580+0000 7f75cae00640 1 --2- 172.21.15.16:0/1928570323 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f75c41411b0 0x7f75c406f920 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f75b4009d90 tx=0x7f75b401c970 comp rx=0 tx=0).stop 2024-03-31T15:04:26.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:26.580+0000 7f75cae00640 1 -- 172.21.15.16:0/1928570323 shutdown_connections 2024-03-31T15:04:26.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:26.580+0000 7f75cae00640 1 --2- 172.21.15.16:0/1928570323 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f75c4144340 0x7f75c406fe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:26.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:26.580+0000 7f75cae00640 1 --2- 172.21.15.16:0/1928570323 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f75c41411b0 0x7f75c406f920 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:26.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:26.580+0000 7f75cae00640 1 -- 172.21.15.16:0/1928570323 >> 172.21.15.16:0/1928570323 conn(0x7f75c4067930 msgr2=0x7f75c40b3ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:04:26.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:26.580+0000 7f75cae00640 1 -- 172.21.15.16:0/1928570323 shutdown_connections 2024-03-31T15:04:26.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:26.580+0000 7f75cae00640 1 -- 172.21.15.16:0/1928570323 wait complete. 2024-03-31T15:04:26.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:26.580+0000 7f75cae00640 1 Processor -- start 2024-03-31T15:04:26.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:26.580+0000 7f75cae00640 1 -- start start 2024-03-31T15:04:26.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:26.580+0000 7f75cae00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f75c41411b0 0x7f75c4155940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:26.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:26.580+0000 7f75cae00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f75c4141590 0x7f75c41574f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:26.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:26.580+0000 7f75cae00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f75c4144340 0x7f75c4157a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:26.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:26.580+0000 7f75cae00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f75c415f290 con 0x7f75c41411b0 2024-03-31T15:04:26.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:26.580+0000 7f75cae00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f75c415f110 con 0x7f75c4144340 2024-03-31T15:04:26.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:26.580+0000 7f75cae00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f75c415f410 con 0x7f75c4141590 2024-03-31T15:04:26.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:26.580+0000 7f75c3e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f75c41411b0 0x7f75c4155940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:26.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:26.580+0000 7f75c3e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f75c41411b0 0x7f75c4155940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49438/0 (socket says 172.21.15.16:49438) 2024-03-31T15:04:26.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:26.580+0000 7f75c3e00640 1 -- 172.21.15.16:0/3667300333 learned_addr learned my addr 172.21.15.16:0/3667300333 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:04:26.577 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:26.580+0000 7f75bb400640 1 --2- 172.21.15.16:0/3667300333 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f75c4141590 0x7f75c41574f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:26.577 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:26.580+0000 7f75c8e00640 1 --2- 172.21.15.16:0/3667300333 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f75c4144340 0x7f75c4157a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:26.577 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:26.580+0000 7f75c3e00640 1 -- 172.21.15.16:0/3667300333 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f75c4141590 msgr2=0x7f75c41574f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:26.577 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:26.580+0000 7f75c3e00640 1 --2- 172.21.15.16:0/3667300333 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f75c4141590 0x7f75c41574f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:26.577 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:26.580+0000 7f75c3e00640 1 -- 172.21.15.16:0/3667300333 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f75c4144340 msgr2=0x7f75c4157a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:26.577 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:26.580+0000 7f75c3e00640 1 --2- 172.21.15.16:0/3667300333 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f75c4144340 0x7f75c4157a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:26.577 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:26.580+0000 7f75c3e00640 1 -- 172.21.15.16:0/3667300333 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75c412cbe0 con 0x7f75c41411b0 2024-03-31T15:04:26.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:26.584+0000 7f75c3e00640 1 --2- 172.21.15.16:0/3667300333 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f75c41411b0 0x7f75c4155940 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f75b40b0040 tx=0x7f75b4019710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:26.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:26.584+0000 7f75c0c00640 1 -- 172.21.15.16:0/3667300333 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75b40bd020 con 0x7f75c41411b0 2024-03-31T15:04:26.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:26.584+0000 7f75c0c00640 1 -- 172.21.15.16:0/3667300333 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f75b40b8670 con 0x7f75c41411b0 2024-03-31T15:04:26.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:26.584+0000 7f75c0c00640 1 -- 172.21.15.16:0/3667300333 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75b40b2040 con 0x7f75c41411b0 2024-03-31T15:04:26.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:26.584+0000 7f75cae00640 1 -- 172.21.15.16:0/3667300333 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f75c412cec0 con 0x7f75c41411b0 2024-03-31T15:04:26.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:26.584+0000 7f75cae00640 1 -- 172.21.15.16:0/3667300333 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f75c412d480 con 0x7f75c41411b0 2024-03-31T15:04:26.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:26.584+0000 7f75cae00640 1 -- 172.21.15.16:0/3667300333 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f75c4069950 con 0x7f75c41411b0 2024-03-31T15:04:26.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:26.584+0000 7f75c0c00640 1 -- 172.21.15.16:0/3667300333 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f75b40c0020 con 0x7f75c41411b0 2024-03-31T15:04:26.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:26.584+0000 7f75c0c00640 1 --2- 172.21.15.16:0/3667300333 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f75a0068380 0x7f75a006a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:26.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:26.584+0000 7f75c0c00640 1 -- 172.21.15.16:0/3667300333 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f75b4121890 con 0x7f75c41411b0 2024-03-31T15:04:26.580 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:26.584+0000 7f75bb400640 1 --2- 172.21.15.16:0/3667300333 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f75a0068380 0x7f75a006a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:26.580 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:26.588+0000 7f75bb400640 1 --2- 172.21.15.16:0/3667300333 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f75a0068380 0x7f75a006a840 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f75c415e3d0 tx=0x7f75ac007d60 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:26.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:26.588+0000 7f75c0c00640 1 -- 172.21.15.16:0/3667300333 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f75b40c7030 con 0x7f75c41411b0 2024-03-31T15:04:26.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:26.756+0000 7f75cae00640 1 -- 172.21.15.16:0/3667300333 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f75c4161070 con 0x7f75c41411b0 2024-03-31T15:04:26.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:26.756+0000 7f75c0c00640 1 -- 172.21.15.16:0/3667300333 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v23) ==== 104+0+11 (secure 0 0 0) 0x7f75b4016560 con 0x7f75c41411b0 2024-03-31T15:04:26.749 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2024-03-31T15:04:26.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:26.756+0000 7f75cae00640 1 -- 172.21.15.16:0/3667300333 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f75a0068380 msgr2=0x7f75a006a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:26.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:26.756+0000 7f75cae00640 1 --2- 172.21.15.16:0/3667300333 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f75a0068380 0x7f75a006a840 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f75c415e3d0 tx=0x7f75ac007d60 comp rx=0 tx=0).stop 2024-03-31T15:04:26.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:26.756+0000 7f75cae00640 1 -- 172.21.15.16:0/3667300333 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f75c41411b0 msgr2=0x7f75c4155940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:26.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:26.756+0000 7f75cae00640 1 --2- 172.21.15.16:0/3667300333 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f75c41411b0 0x7f75c4155940 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f75b40b0040 tx=0x7f75b4019710 comp rx=0 tx=0).stop 2024-03-31T15:04:26.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:26.756+0000 7f75cae00640 1 -- 172.21.15.16:0/3667300333 shutdown_connections 2024-03-31T15:04:26.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:26.756+0000 7f75cae00640 1 --2- 172.21.15.16:0/3667300333 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f75a0068380 0x7f75a006a840 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:26.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:26.756+0000 7f75cae00640 1 --2- 172.21.15.16:0/3667300333 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f75c4144340 0x7f75c4157a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:26.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:26.756+0000 7f75cae00640 1 --2- 172.21.15.16:0/3667300333 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f75c4141590 0x7f75c41574f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:26.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:26.756+0000 7f75cae00640 1 --2- 172.21.15.16:0/3667300333 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f75c41411b0 0x7f75c4155940 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:26.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:26.756+0000 7f75cae00640 1 -- 172.21.15.16:0/3667300333 >> 172.21.15.16:0/3667300333 conn(0x7f75c4067930 msgr2=0x7f75c40b3c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:04:26.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:26.756+0000 7f75cae00640 1 -- 172.21.15.16:0/3667300333 shutdown_connections 2024-03-31T15:04:26.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:26.756+0000 7f75cae00640 1 -- 172.21.15.16:0/3667300333 wait complete. 2024-03-31T15:04:26.770 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-31T15:04:26.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:26.916+0000 7fb5d1000640 1 Processor -- start 2024-03-31T15:04:26.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:26.916+0000 7fb5d1000640 1 -- start start 2024-03-31T15:04:26.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:26.920+0000 7fb5d1000640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fb5cc13e3d0 0x7fb5cc0cc9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:26.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:26.920+0000 7fb5d1000640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb5cc06dbe0 0x7fb5cc0ccf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:26.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:26.920+0000 7fb5d1000640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb5cc15fbc0 con 0x7fb5cc13dff0 2024-03-31T15:04:26.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:26.920+0000 7fb5d1000640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7fb5cc0cd450 con 0x7fb5cc13e3d0 2024-03-31T15:04:26.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:26.920+0000 7fb5d1000640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fb5cc15f730 con 0x7fb5cc06dbe0 2024-03-31T15:04:26.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:26.920+0000 7fb5c9e00640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fb5cc13dff0 0x7fb5cc06d4d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:44514/0 (socket says 172.21.15.16:44514) 2024-03-31T15:04:26.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:26.920+0000 7fb5c9e00640 1 -- 172.21.15.16:0/423759049 learned_addr learned my addr 172.21.15.16:0/423759049 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:04:26.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:26.920+0000 7fb5c1400640 1 --2- 172.21.15.16:0/423759049 >> v2:172.21.15.89:3300/0 conn(0x7fb5cc13e3d0 0x7fb5cc0cc9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:26.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:26.920+0000 7fb5ca800640 1 --2- 172.21.15.16:0/423759049 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb5cc06dbe0 0x7fb5cc0ccf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:26.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:26.920+0000 7fb5c9400640 1 -- 172.21.15.16:0/423759049 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2898908139 0 0) 0x7fb5cc15fbc0 con 0x7fb5cc13dff0 2024-03-31T15:04:26.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:26.920+0000 7fb5c1400640 1 -- 172.21.15.16:0/423759049 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb5cc06dbe0 msgr2=0x7fb5cc0ccf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:04:26.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:26.920+0000 7fb5c1400640 1 --2- 172.21.15.16:0/423759049 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb5cc06dbe0 0x7fb5cc0ccf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:26.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:26.920+0000 7fb5c1400640 1 -- 172.21.15.16:0/423759049 >> v1:172.21.15.16:6789/0 conn(0x7fb5cc13dff0 legacy=0x7fb5cc06d4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:26.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:26.920+0000 7fb5c1400640 1 -- 172.21.15.16:0/423759049 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5cc15ee00 con 0x7fb5cc13e3d0 2024-03-31T15:04:26.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:26.924+0000 7fb5c1400640 1 --2- 172.21.15.16:0/423759049 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb5cc13e3d0 0x7fb5cc0cc9d0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fb5b40097c0 tx=0x7fb5b401c410 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b1d9932f3f389dda server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:26.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:26.924+0000 7fb5c9400640 1 -- 172.21.15.16:0/423759049 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5b40b0070 con 0x7fb5cc13e3d0 2024-03-31T15:04:26.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:26.924+0000 7fb5c9400640 1 -- 172.21.15.16:0/423759049 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb5b40ab600 con 0x7fb5cc13e3d0 2024-03-31T15:04:26.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:26.924+0000 7fb5c9400640 1 -- 172.21.15.16:0/423759049 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5b40ab9d0 con 0x7fb5cc13e3d0 2024-03-31T15:04:26.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:26.924+0000 7fb5d1000640 1 -- 172.21.15.16:0/423759049 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb5cc13e3d0 msgr2=0x7fb5cc0cc9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:26.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:26.924+0000 7fb5d1000640 1 --2- 172.21.15.16:0/423759049 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb5cc13e3d0 0x7fb5cc0cc9d0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fb5b40097c0 tx=0x7fb5b401c410 comp rx=0 tx=0).stop 2024-03-31T15:04:26.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:26.924+0000 7fb5d1000640 1 -- 172.21.15.16:0/423759049 shutdown_connections 2024-03-31T15:04:26.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:26.924+0000 7fb5d1000640 1 --2- 172.21.15.16:0/423759049 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb5cc06dbe0 0x7fb5cc0ccf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:26.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:26.924+0000 7fb5d1000640 1 --2- 172.21.15.16:0/423759049 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb5cc13e3d0 0x7fb5cc0cc9d0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:26.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:26.924+0000 7fb5d1000640 1 -- 172.21.15.16:0/423759049 >> 172.21.15.16:0/423759049 conn(0x7fb5cc06ee40 msgr2=0x7fb5cc06b620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:04:26.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:26.924+0000 7fb5d1000640 1 -- 172.21.15.16:0/423759049 shutdown_connections 2024-03-31T15:04:26.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:26.924+0000 7fb5d1000640 1 -- 172.21.15.16:0/423759049 wait complete. 2024-03-31T15:04:26.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:26.924+0000 7fb5d1000640 1 Processor -- start 2024-03-31T15:04:26.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:26.924+0000 7fb5d1000640 1 -- start start 2024-03-31T15:04:26.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:26.928+0000 7fb5d1000640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb5cc06dbe0 0x7fb5cc1f7150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:26.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:26.928+0000 7fb5d1000640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb5cc13dff0 0x7fb5cc0b3560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:26.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:26.928+0000 7fb5d1000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb5cc13e3d0 0x7fb5cc0b3fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:26.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:26.928+0000 7fb5d1000640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fb5cc0cf710 con 0x7fb5cc13e3d0 2024-03-31T15:04:26.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:26.928+0000 7fb5d1000640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fb5cc0cf590 con 0x7fb5cc13dff0 2024-03-31T15:04:26.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:26.928+0000 7fb5d1000640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fb5cc0cf890 con 0x7fb5cc06dbe0 2024-03-31T15:04:26.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:26.928+0000 7fb5ca800640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb5cc13e3d0 0x7fb5cc0b3fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:26.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:26.928+0000 7fb5ca800640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb5cc13e3d0 0x7fb5cc0b3fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49440/0 (socket says 172.21.15.16:49440) 2024-03-31T15:04:26.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:26.928+0000 7fb5ca800640 1 -- 172.21.15.16:0/1996809079 learned_addr learned my addr 172.21.15.16:0/1996809079 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:04:26.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:26.928+0000 7fb5c1400640 1 --2- 172.21.15.16:0/1996809079 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb5cc13dff0 0x7fb5cc0b3560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:26.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:26.928+0000 7fb5c9e00640 1 --2- 172.21.15.16:0/1996809079 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb5cc06dbe0 0x7fb5cc1f7150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:26.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:26.928+0000 7fb5c1400640 1 -- 172.21.15.16:0/1996809079 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb5cc06dbe0 msgr2=0x7fb5cc1f7150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:26.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:26.928+0000 7fb5c1400640 1 --2- 172.21.15.16:0/1996809079 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb5cc06dbe0 0x7fb5cc1f7150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:26.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:26.928+0000 7fb5c1400640 1 -- 172.21.15.16:0/1996809079 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb5cc13e3d0 msgr2=0x7fb5cc0b3fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:26.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:26.928+0000 7fb5c1400640 1 --2- 172.21.15.16:0/1996809079 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb5cc13e3d0 0x7fb5cc0b3fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:26.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:26.928+0000 7fb5c1400640 1 -- 172.21.15.16:0/1996809079 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5cc139550 con 0x7fb5cc13dff0 2024-03-31T15:04:26.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:26.928+0000 7fb5ca800640 1 --2- 172.21.15.16:0/1996809079 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb5cc13e3d0 0x7fb5cc0b3fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:04:26.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:26.928+0000 7fb5c1400640 1 --2- 172.21.15.16:0/1996809079 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb5cc13dff0 0x7fb5cc0b3560 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fb5b40ab000 tx=0x7fb5b40191b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:26.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:26.928+0000 7fb5c2a00640 1 -- 172.21.15.16:0/1996809079 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5b40b8020 con 0x7fb5cc13dff0 2024-03-31T15:04:26.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:26.928+0000 7fb5c2a00640 1 -- 172.21.15.16:0/1996809079 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb5b40b4410 con 0x7fb5cc13dff0 2024-03-31T15:04:26.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:26.928+0000 7fb5d1000640 1 -- 172.21.15.16:0/1996809079 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb5cc157540 con 0x7fb5cc13dff0 2024-03-31T15:04:26.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:26.928+0000 7fb5c2a00640 1 -- 172.21.15.16:0/1996809079 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5b40b0040 con 0x7fb5cc13dff0 2024-03-31T15:04:26.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:26.928+0000 7fb5d1000640 1 -- 172.21.15.16:0/1996809079 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb5cc157b00 con 0x7fb5cc13dff0 2024-03-31T15:04:26.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:26.928+0000 7fb5d1000640 1 -- 172.21.15.16:0/1996809079 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb5cc069950 con 0x7fb5cc13dff0 2024-03-31T15:04:26.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:26.928+0000 7fb5c2a00640 1 -- 172.21.15.16:0/1996809079 <== mon.1 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fb5b40bb020 con 0x7fb5cc13dff0 2024-03-31T15:04:26.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:26.932+0000 7fb5c2a00640 1 --2- 172.21.15.16:0/1996809079 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fb5a8068380 0x7fb5a806a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:26.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:26.932+0000 7fb5c2a00640 1 -- 172.21.15.16:0/1996809079 <== mon.1 v2:172.21.15.89:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7fb5b4120380 con 0x7fb5cc13dff0 2024-03-31T15:04:26.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:26.932+0000 7fb5c9e00640 1 --2- 172.21.15.16:0/1996809079 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fb5a8068380 0x7fb5a806a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:26.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:26.932+0000 7fb5c9e00640 1 --2- 172.21.15.16:0/1996809079 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fb5a8068380 0x7fb5a806a840 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fb5bc004fd0 tx=0x7fb5bc001770 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:26.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:26.936+0000 7fb5c2a00640 1 -- 172.21.15.16:0/1996809079 <== mon.1 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fb5b40c2030 con 0x7fb5cc13dff0 2024-03-31T15:04:27.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:27.100+0000 7fb5d1000640 1 -- 172.21.15.16:0/1996809079 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fb5cc06e4e0 con 0x7fb5cc13dff0 2024-03-31T15:04:27.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:27.100+0000 7fb5c2a00640 1 -- 172.21.15.16:0/1996809079 <== mon.1 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v23) ==== 100+0+11 (secure 0 0 0) 0x7fb5b4016090 con 0x7fb5cc13dff0 2024-03-31T15:04:27.095 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 64 2024-03-31T15:04:27.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:27.104+0000 7fb5d1000640 1 -- 172.21.15.16:0/1996809079 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fb5a8068380 msgr2=0x7fb5a806a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:27.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:27.104+0000 7fb5d1000640 1 --2- 172.21.15.16:0/1996809079 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fb5a8068380 0x7fb5a806a840 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fb5bc004fd0 tx=0x7fb5bc001770 comp rx=0 tx=0).stop 2024-03-31T15:04:27.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:27.104+0000 7fb5d1000640 1 -- 172.21.15.16:0/1996809079 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb5cc13dff0 msgr2=0x7fb5cc0b3560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:27.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:27.104+0000 7fb5d1000640 1 --2- 172.21.15.16:0/1996809079 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb5cc13dff0 0x7fb5cc0b3560 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fb5b40ab000 tx=0x7fb5b40191b0 comp rx=0 tx=0).stop 2024-03-31T15:04:27.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:27.104+0000 7fb5d1000640 1 -- 172.21.15.16:0/1996809079 shutdown_connections 2024-03-31T15:04:27.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:27.104+0000 7fb5d1000640 1 --2- 172.21.15.16:0/1996809079 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fb5a8068380 0x7fb5a806a840 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:27.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:27.104+0000 7fb5d1000640 1 --2- 172.21.15.16:0/1996809079 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb5cc13e3d0 0x7fb5cc0b3fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:27.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:27.104+0000 7fb5d1000640 1 --2- 172.21.15.16:0/1996809079 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb5cc13dff0 0x7fb5cc0b3560 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:27.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:27.104+0000 7fb5d1000640 1 --2- 172.21.15.16:0/1996809079 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb5cc06dbe0 0x7fb5cc1f7150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:27.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:27.104+0000 7fb5d1000640 1 -- 172.21.15.16:0/1996809079 >> 172.21.15.16:0/1996809079 conn(0x7fb5cc06ee40 msgr2=0x7fb5cc06d4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:04:27.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:27.104+0000 7fb5d1000640 1 -- 172.21.15.16:0/1996809079 shutdown_connections 2024-03-31T15:04:27.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:27.104+0000 7fb5d1000640 1 -- 172.21.15.16:0/1996809079 wait complete. 2024-03-31T15:04:27.117 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-31T15:04:27.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:27.264+0000 7fe7bba00640 1 Processor -- start 2024-03-31T15:04:27.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:27.264+0000 7fe7bba00640 1 -- start start 2024-03-31T15:04:27.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:27.264+0000 7fe7bba00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fe7b4145e40 0x7fe7b40cc740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:27.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:27.264+0000 7fe7bba00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fe7b4070030 0x7fe7b40d2010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:27.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:27.264+0000 7fe7bba00640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe7b40d0920 con 0x7fe7b40ccc80 2024-03-31T15:04:27.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:27.264+0000 7fe7bba00640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7fe7b40d06d0 con 0x7fe7b4145e40 2024-03-31T15:04:27.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:27.264+0000 7fe7bba00640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7fe7b40d0c00 con 0x7fe7b4070030 2024-03-31T15:04:27.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:27.264+0000 7fe7b9000640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fe7b4145e40 0x7fe7b40cc740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:27.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:27.264+0000 7fe7b9000640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fe7b4145e40 0x7fe7b40cc740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43682/0 (socket says 172.21.15.16:43682) 2024-03-31T15:04:27.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:27.264+0000 7fe7b3e00640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fe7b40ccc80 0x7fe7b406f920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:33666/0 (socket says 172.21.15.16:33666) 2024-03-31T15:04:27.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:27.264+0000 7fe7b9000640 1 -- 172.21.15.16:0/1874737887 learned_addr learned my addr 172.21.15.16:0/1874737887 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:04:27.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:27.264+0000 7fe7b9a00640 1 --2- 172.21.15.16:0/1874737887 >> v2:172.21.15.89:3300/0 conn(0x7fe7b4070030 0x7fe7b40d2010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:27.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:27.264+0000 7fe7b9000640 1 -- 172.21.15.16:0/1874737887 >> v2:172.21.15.89:3300/0 conn(0x7fe7b4070030 msgr2=0x7fe7b40d2010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:04:27.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:27.264+0000 7fe7b9000640 1 --2- 172.21.15.16:0/1874737887 >> v2:172.21.15.89:3300/0 conn(0x7fe7b4070030 0x7fe7b40d2010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:27.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:27.264+0000 7fe7b9000640 1 -- 172.21.15.16:0/1874737887 >> v1:172.21.15.16:6789/0 conn(0x7fe7b40ccc80 legacy=0x7fe7b406f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:27.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:27.264+0000 7fe7b9000640 1 -- 172.21.15.16:0/1874737887 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7b40d1810 con 0x7fe7b4145e40 2024-03-31T15:04:27.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:27.264+0000 7fe7b3400640 1 -- 172.21.15.16:0/1874737887 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 512306947 0 0) 0x7fe7b40d0920 con 0x7fe7b40ccc80 2024-03-31T15:04:27.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:27.268+0000 7fe7b9000640 1 --2- 172.21.15.16:0/1874737887 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe7b4145e40 0x7fe7b40cc740 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fe7b415fd20 tx=0x7fe7a801c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f80149905929df7 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:27.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:27.268+0000 7fe7b3400640 1 -- 172.21.15.16:0/1874737887 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7a80ab430 con 0x7fe7b4145e40 2024-03-31T15:04:27.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:27.268+0000 7fe7b3400640 1 -- 172.21.15.16:0/1874737887 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe7a80b0070 con 0x7fe7b4145e40 2024-03-31T15:04:27.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:27.268+0000 7fe7b3400640 1 -- 172.21.15.16:0/1874737887 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7a80ab7f0 con 0x7fe7b4145e40 2024-03-31T15:04:27.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:27.268+0000 7fe7bba00640 1 -- 172.21.15.16:0/1874737887 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe7b4145e40 msgr2=0x7fe7b40cc740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:27.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:27.268+0000 7fe7bba00640 1 --2- 172.21.15.16:0/1874737887 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe7b4145e40 0x7fe7b40cc740 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fe7b415fd20 tx=0x7fe7a801c520 comp rx=0 tx=0).stop 2024-03-31T15:04:27.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:27.268+0000 7fe7bba00640 1 -- 172.21.15.16:0/1874737887 shutdown_connections 2024-03-31T15:04:27.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:27.268+0000 7fe7bba00640 1 --2- 172.21.15.16:0/1874737887 >> v2:172.21.15.89:3300/0 conn(0x7fe7b4070030 0x7fe7b40d2010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:27.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:27.268+0000 7fe7bba00640 1 --2- 172.21.15.16:0/1874737887 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe7b4145e40 0x7fe7b40cc740 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:27.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:27.268+0000 7fe7bba00640 1 -- 172.21.15.16:0/1874737887 >> 172.21.15.16:0/1874737887 conn(0x7fe7b4067330 msgr2=0x7fe7b4067740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:04:27.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:27.268+0000 7fe7bba00640 1 -- 172.21.15.16:0/1874737887 shutdown_connections 2024-03-31T15:04:27.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:27.268+0000 7fe7bba00640 1 -- 172.21.15.16:0/1874737887 wait complete. 2024-03-31T15:04:27.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:27.272+0000 7fe7bba00640 1 Processor -- start 2024-03-31T15:04:27.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:27.272+0000 7fe7bba00640 1 -- start start 2024-03-31T15:04:27.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:27.272+0000 7fe7bba00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe7b4070030 0x7fe7b4155cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:27.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:27.272+0000 7fe7bba00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fe7b40ccc80 0x7fe7b415abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:27.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:27.272+0000 7fe7bba00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe7b4145e40 0x7fe7b415b110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:27.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:27.272+0000 7fe7bba00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fe7b40b5240 con 0x7fe7b4070030 2024-03-31T15:04:27.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:27.272+0000 7fe7bba00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fe7b40b50c0 con 0x7fe7b4145e40 2024-03-31T15:04:27.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:27.272+0000 7fe7bba00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fe7b4169190 con 0x7fe7b40ccc80 2024-03-31T15:04:27.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:27.272+0000 7fe7b9000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe7b4070030 0x7fe7b4155cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:27.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:27.272+0000 7fe7b9000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe7b4070030 0x7fe7b4155cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43690/0 (socket says 172.21.15.16:43690) 2024-03-31T15:04:27.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:27.272+0000 7fe7b9000640 1 -- 172.21.15.16:0/3969910309 learned_addr learned my addr 172.21.15.16:0/3969910309 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:04:27.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:27.272+0000 7fe7b3e00640 1 --2- 172.21.15.16:0/3969910309 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fe7b40ccc80 0x7fe7b415abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:27.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:27.272+0000 7fe7b9a00640 1 --2- 172.21.15.16:0/3969910309 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe7b4145e40 0x7fe7b415b110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:27.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:27.272+0000 7fe7b9000640 1 -- 172.21.15.16:0/3969910309 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fe7b40ccc80 msgr2=0x7fe7b415abd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:27.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:27.272+0000 7fe7b9000640 1 --2- 172.21.15.16:0/3969910309 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fe7b40ccc80 0x7fe7b415abd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:27.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:27.272+0000 7fe7b9000640 1 -- 172.21.15.16:0/3969910309 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe7b4145e40 msgr2=0x7fe7b415b110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:27.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:27.272+0000 7fe7b9000640 1 --2- 172.21.15.16:0/3969910309 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe7b4145e40 0x7fe7b415b110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:27.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:27.272+0000 7fe7b9000640 1 -- 172.21.15.16:0/3969910309 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7b415d430 con 0x7fe7b4070030 2024-03-31T15:04:27.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:27.272+0000 7fe7b3e00640 1 --2- 172.21.15.16:0/3969910309 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fe7b40ccc80 0x7fe7b415abd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:04:27.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:27.272+0000 7fe7b9000640 1 --2- 172.21.15.16:0/3969910309 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe7b4070030 0x7fe7b4155cb0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fe7a8019130 tx=0x7fe7a801ad60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:27.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:27.272+0000 7fe7b0c00640 1 -- 172.21.15.16:0/3969910309 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7a8004020 con 0x7fe7b4070030 2024-03-31T15:04:27.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:27.272+0000 7fe7b0c00640 1 -- 172.21.15.16:0/3969910309 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe7a80041c0 con 0x7fe7b4070030 2024-03-31T15:04:27.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:27.272+0000 7fe7b0c00640 1 -- 172.21.15.16:0/3969910309 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7a8003620 con 0x7fe7b4070030 2024-03-31T15:04:27.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:27.272+0000 7fe7bba00640 1 -- 172.21.15.16:0/3969910309 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe7b415d710 con 0x7fe7b4070030 2024-03-31T15:04:27.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:27.272+0000 7fe7bba00640 1 -- 172.21.15.16:0/3969910309 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe7b415dcd0 con 0x7fe7b4070030 2024-03-31T15:04:27.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:27.272+0000 7fe7bba00640 1 -- 172.21.15.16:0/3969910309 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe7b4069950 con 0x7fe7b4070030 2024-03-31T15:04:27.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:27.276+0000 7fe7b0c00640 1 -- 172.21.15.16:0/3969910309 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fe7a80037c0 con 0x7fe7b4070030 2024-03-31T15:04:27.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:27.276+0000 7fe7b0c00640 1 --2- 172.21.15.16:0/3969910309 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fe794068380 0x7fe79406a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:27.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:27.276+0000 7fe7b0c00640 1 -- 172.21.15.16:0/3969910309 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7fe7a81204a0 con 0x7fe7b4070030 2024-03-31T15:04:27.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:27.276+0000 7fe7b3e00640 1 --2- 172.21.15.16:0/3969910309 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fe794068380 0x7fe79406a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:27.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:27.276+0000 7fe7b3e00640 1 --2- 172.21.15.16:0/3969910309 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fe794068380 0x7fe79406a840 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fe7a0007fd0 tx=0x7fe7a0007c20 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:27.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:27.280+0000 7fe7b0c00640 1 -- 172.21.15.16:0/3969910309 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fe7a80c4020 con 0x7fe7b4070030 2024-03-31T15:04:27.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:27.476+0000 7fe7bba00640 1 -- 172.21.15.16:0/3969910309 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe7b40cf3c0 con 0x7fe7b4070030 2024-03-31T15:04:27.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:27.476+0000 7fe7b0c00640 1 -- 172.21.15.16:0/3969910309 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+5172 (secure 0 0 0) 0x7fe7a8016110 con 0x7fe7b4070030 2024-03-31T15:04:27.472 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:04:27.472 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4276,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/2832752118","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6833","nonce":2832752118},{"type":"v1","addr":"172.21.15.16:6835","nonce":2832752118}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4278,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6834/153294668","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":153294668},{"type":"v1","addr":"172.21.15.16:6834","nonce":153294668}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4288,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/1653319357","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":1653319357},{"type":"v1","addr":"172.21.15.89:6835","nonce":1653319357}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-31T15:03:36.349865+0000","modified":"2024-03-31T15:03:39.378181+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4293},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.89:6837/2583453131","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":2583453131},{"type":"v1","addr":"172.21.15.89:6837","nonce":2583453131}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4293,"qdb_cluster":[4293]},"id":1}]} 2024-03-31T15:04:27.472 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 8 2024-03-31T15:04:27.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:27.480+0000 7fe7bba00640 1 -- 172.21.15.16:0/3969910309 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fe794068380 msgr2=0x7fe79406a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:27.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:27.480+0000 7fe7bba00640 1 --2- 172.21.15.16:0/3969910309 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fe794068380 0x7fe79406a840 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fe7a0007fd0 tx=0x7fe7a0007c20 comp rx=0 tx=0).stop 2024-03-31T15:04:27.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:27.480+0000 7fe7bba00640 1 -- 172.21.15.16:0/3969910309 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe7b4070030 msgr2=0x7fe7b4155cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:27.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:27.480+0000 7fe7bba00640 1 --2- 172.21.15.16:0/3969910309 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe7b4070030 0x7fe7b4155cb0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fe7a8019130 tx=0x7fe7a801ad60 comp rx=0 tx=0).stop 2024-03-31T15:04:27.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:27.480+0000 7fe7bba00640 1 -- 172.21.15.16:0/3969910309 shutdown_connections 2024-03-31T15:04:27.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:27.480+0000 7fe7bba00640 1 --2- 172.21.15.16:0/3969910309 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fe794068380 0x7fe79406a840 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:27.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:27.480+0000 7fe7bba00640 1 --2- 172.21.15.16:0/3969910309 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe7b4145e40 0x7fe7b415b110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:27.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:27.480+0000 7fe7bba00640 1 --2- 172.21.15.16:0/3969910309 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fe7b40ccc80 0x7fe7b415abd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:27.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:27.480+0000 7fe7bba00640 1 --2- 172.21.15.16:0/3969910309 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe7b4070030 0x7fe7b4155cb0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:27.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:27.480+0000 7fe7bba00640 1 -- 172.21.15.16:0/3969910309 >> 172.21.15.16:0/3969910309 conn(0x7fe7b4067330 msgr2=0x7fe7b40b3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:04:27.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:27.480+0000 7fe7bba00640 1 -- 172.21.15.16:0/3969910309 shutdown_connections 2024-03-31T15:04:27.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:27.480+0000 7fe7bba00640 1 -- 172.21.15.16:0/3969910309 wait complete. 2024-03-31T15:04:27.492 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-31T15:04:27.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:27.640+0000 7f58cbc00640 1 Processor -- start 2024-03-31T15:04:27.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:27.640+0000 7f58cbc00640 1 -- start start 2024-03-31T15:04:27.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:27.640+0000 7f58cbc00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f58c4145aa0 0x7f58c415c8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:27.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:27.640+0000 7f58cbc00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f58c406c1c0 0x7f58c4072e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:27.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:27.640+0000 7f58cbc00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f58c406b620 0x7f58c406bab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:27.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:27.640+0000 7f58cbc00640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f58c406fd50 con 0x7f58c406b620 2024-03-31T15:04:27.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:27.640+0000 7f58cbc00640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f58c40d09f0 con 0x7f58c406c1c0 2024-03-31T15:04:27.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:27.640+0000 7f58cbc00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f58c40d0cd0 con 0x7f58c4145aa0 2024-03-31T15:04:27.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:27.640+0000 7f58c9c00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f58c406b620 0x7f58c406bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:27.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:27.640+0000 7f58c9c00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f58c406b620 0x7f58c406bab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43698/0 (socket says 172.21.15.16:43698) 2024-03-31T15:04:27.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:27.640+0000 7f58c9c00640 1 -- 172.21.15.16:0/1215213898 learned_addr learned my addr 172.21.15.16:0/1215213898 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:04:27.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:27.640+0000 7f58c3e00640 1 --2- 172.21.15.16:0/1215213898 >> v2:172.21.15.89:3300/0 conn(0x7f58c406c1c0 0x7f58c4072e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:27.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:27.640+0000 7f58c9200640 1 --2- 172.21.15.16:0/1215213898 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f58c4145aa0 0x7f58c415c8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:27.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:27.640+0000 7f58c9c00640 1 -- 172.21.15.16:0/1215213898 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f58c4145aa0 msgr2=0x7f58c415c8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:04:27.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:27.640+0000 7f58c9c00640 1 --2- 172.21.15.16:0/1215213898 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f58c4145aa0 0x7f58c415c8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:27.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:27.640+0000 7f58c9c00640 1 -- 172.21.15.16:0/1215213898 >> v2:172.21.15.89:3300/0 conn(0x7f58c406c1c0 msgr2=0x7f58c4072e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:04:27.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:27.640+0000 7f58c9c00640 1 --2- 172.21.15.16:0/1215213898 >> v2:172.21.15.89:3300/0 conn(0x7f58c406c1c0 0x7f58c4072e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:27.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:27.640+0000 7f58c9c00640 1 -- 172.21.15.16:0/1215213898 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58c40d06e0 con 0x7f58c406b620 2024-03-31T15:04:27.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:27.640+0000 7f58c3e00640 1 --2- 172.21.15.16:0/1215213898 >> v2:172.21.15.89:3300/0 conn(0x7f58c406c1c0 0x7f58c4072e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:04:27.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:27.644+0000 7f58c9c00640 1 --2- 172.21.15.16:0/1215213898 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f58c406b620 0x7f58c406bab0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f58c4069210 tx=0x7f58bc01c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a322caedf5e92753 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:27.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:27.644+0000 7f58c3400640 1 -- 172.21.15.16:0/1215213898 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58bc0a5670 con 0x7f58c406b620 2024-03-31T15:04:27.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:27.644+0000 7f58c3400640 1 -- 172.21.15.16:0/1215213898 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f58bc0a5810 con 0x7f58c406b620 2024-03-31T15:04:27.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:27.644+0000 7f58c3400640 1 -- 172.21.15.16:0/1215213898 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58bc0a5be0 con 0x7f58c406b620 2024-03-31T15:04:27.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:27.644+0000 7f58cbc00640 1 -- 172.21.15.16:0/1215213898 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f58c406b620 msgr2=0x7f58c406bab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:27.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:27.644+0000 7f58cbc00640 1 --2- 172.21.15.16:0/1215213898 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f58c406b620 0x7f58c406bab0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f58c4069210 tx=0x7f58bc01c9f0 comp rx=0 tx=0).stop 2024-03-31T15:04:27.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:27.644+0000 7f58cbc00640 1 -- 172.21.15.16:0/1215213898 shutdown_connections 2024-03-31T15:04:27.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:27.644+0000 7f58cbc00640 1 --2- 172.21.15.16:0/1215213898 >> v2:172.21.15.89:3300/0 conn(0x7f58c406c1c0 0x7f58c4072e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:27.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:27.644+0000 7f58cbc00640 1 --2- 172.21.15.16:0/1215213898 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f58c406b620 0x7f58c406bab0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:27.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:27.644+0000 7f58cbc00640 1 --2- 172.21.15.16:0/1215213898 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f58c4145aa0 0x7f58c415c8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:27.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:27.644+0000 7f58cbc00640 1 -- 172.21.15.16:0/1215213898 >> 172.21.15.16:0/1215213898 conn(0x7f58c40d1060 msgr2=0x7f58c40d1470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:04:27.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:27.644+0000 7f58cbc00640 1 -- 172.21.15.16:0/1215213898 shutdown_connections 2024-03-31T15:04:27.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:27.644+0000 7f58cbc00640 1 -- 172.21.15.16:0/1215213898 wait complete. 2024-03-31T15:04:27.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:27.648+0000 7f58cbc00640 1 Processor -- start 2024-03-31T15:04:27.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:27.648+0000 7f58cbc00640 1 -- start start 2024-03-31T15:04:27.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:27.648+0000 7f58cbc00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f58c406b620 0x7f58c4154870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:27.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:27.648+0000 7f58cbc00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f58c406c1c0 0x7f58c413d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:27.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:27.648+0000 7f58cbc00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f58c4145aa0 0x7f58c4156aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:27.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:27.648+0000 7f58cbc00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f58c40ce650 con 0x7f58c4145aa0 2024-03-31T15:04:27.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:27.648+0000 7f58cbc00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f58c40ce4d0 con 0x7f58c406c1c0 2024-03-31T15:04:27.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:27.648+0000 7f58cbc00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f58c40ce7d0 con 0x7f58c406b620 2024-03-31T15:04:27.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:27.648+0000 7f58c9c00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f58c4145aa0 0x7f58c4156aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:27.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:27.648+0000 7f58c9c00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f58c4145aa0 0x7f58c4156aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43712/0 (socket says 172.21.15.16:43712) 2024-03-31T15:04:27.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:27.648+0000 7f58c9200640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f58c406b620 0x7f58c4154870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:27.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:27.648+0000 7f58c3e00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f58c406c1c0 0x7f58c413d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:27.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:27.648+0000 7f58c9c00640 1 -- 172.21.15.16:0/3580620354 learned_addr learned my addr 172.21.15.16:0/3580620354 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:04:27.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:27.648+0000 7f58c9c00640 1 -- 172.21.15.16:0/3580620354 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f58c406b620 msgr2=0x7f58c4154870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:27.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:27.648+0000 7f58c9c00640 1 --2- 172.21.15.16:0/3580620354 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f58c406b620 0x7f58c4154870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:27.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:27.648+0000 7f58c9c00640 1 -- 172.21.15.16:0/3580620354 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f58c406c1c0 msgr2=0x7f58c413d3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:27.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:27.648+0000 7f58c9c00640 1 --2- 172.21.15.16:0/3580620354 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f58c406c1c0 0x7f58c413d3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:27.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:27.648+0000 7f58c9c00640 1 -- 172.21.15.16:0/3580620354 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58c4159f70 con 0x7f58c4145aa0 2024-03-31T15:04:27.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:27.648+0000 7f58c3e00640 1 --2- 172.21.15.16:0/3580620354 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f58c406c1c0 0x7f58c413d3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:04:27.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:27.648+0000 7f58c9c00640 1 --2- 172.21.15.16:0/3580620354 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f58c4145aa0 0x7f58c4156aa0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f58bc01b0a0 tx=0x7f58bc019790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:27.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:27.648+0000 7f58c0c00640 1 -- 172.21.15.16:0/3580620354 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58bc0bc020 con 0x7f58c4145aa0 2024-03-31T15:04:27.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:27.648+0000 7f58c0c00640 1 -- 172.21.15.16:0/3580620354 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f58bc0b8690 con 0x7f58c4145aa0 2024-03-31T15:04:27.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:27.648+0000 7f58c0c00640 1 -- 172.21.15.16:0/3580620354 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58bc0a5670 con 0x7f58c4145aa0 2024-03-31T15:04:27.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:27.648+0000 7f58cbc00640 1 -- 172.21.15.16:0/3580620354 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f58c4159180 con 0x7f58c4145aa0 2024-03-31T15:04:27.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:27.648+0000 7f58cbc00640 1 -- 172.21.15.16:0/3580620354 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f58c4159740 con 0x7f58c4145aa0 2024-03-31T15:04:27.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:27.652+0000 7f58cbc00640 1 -- 172.21.15.16:0/3580620354 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f58c4069950 con 0x7f58c4145aa0 2024-03-31T15:04:27.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:27.652+0000 7f58c0c00640 1 -- 172.21.15.16:0/3580620354 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f58bc0bf020 con 0x7f58c4145aa0 2024-03-31T15:04:27.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:27.652+0000 7f58c0c00640 1 --2- 172.21.15.16:0/3580620354 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f58a40683b0 0x7f58a406a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:27.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:27.652+0000 7f58c0c00640 1 -- 172.21.15.16:0/3580620354 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f58bc1210a0 con 0x7f58c4145aa0 2024-03-31T15:04:27.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:27.652+0000 7f58c9200640 1 --2- 172.21.15.16:0/3580620354 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f58a40683b0 0x7f58a406a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:27.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:27.652+0000 7f58c9200640 1 --2- 172.21.15.16:0/3580620354 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f58a40683b0 0x7f58a406a870 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f58b8009ae0 tx=0x7f58b8009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:27.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:27.656+0000 7f58c0c00640 1 -- 172.21.15.16:0/3580620354 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f58bc0c7020 con 0x7f58c4145aa0 2024-03-31T15:04:27.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:27.820+0000 7f58cbc00640 1 -- 172.21.15.16:0/3580620354 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f58c4141520 con 0x7f58c4145aa0 2024-03-31T15:04:27.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:27.820+0000 7f58c0c00640 1 -- 172.21.15.16:0/3580620354 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v23) ==== 74+0+16780 (secure 0 0 0) 0x7f58bc0165e0 con 0x7f58c4145aa0 2024-03-31T15:04:27.816 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:04:27.816 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":23,"fsid":"286eb5da-fb47-4550-b290-d9519c13b9a5","created":"2024-03-31T15:03:11.438952+0000","modified":"2024-03-31T15:04:18.087947+0000","last_up_change":"2024-03-31T15:03:22.885535+0000","last_in_change":"2024-03-31T15:03:13.941667+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-31T15:03:22.550787+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-31T15:03:34.488425+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-31T15:03:35.299116+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"92a66450-ef19-40d8-ac3d-d760b23b0bda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6801","nonce":932712244}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6803","nonce":932712244}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6807","nonce":932712244}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6805","nonce":932712244}]},"public_addr":"172.21.15.16:6801/932712244","cluster_addr":"172.21.15.16:6803/932712244","heartbeat_back_addr":"172.21.15.16:6807/932712244","heartbeat_front_addr":"172.21.15.16:6805/932712244","state":["exists","up"]},{"osd":1,"uuid":"da9d1f2b-1968-4959-b843-faa231988e92","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6810","nonce":1398737533}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6811","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6813","nonce":1398737533}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6821","nonce":1398737533}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6815","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6816","nonce":1398737533}]},"public_addr":"172.21.15.16:6810/1398737533","cluster_addr":"172.21.15.16:6813/1398737533","heartbeat_back_addr":"172.21.15.16:6821/1398737533","heartbeat_front_addr":"172.21.15.16:6816/1398737533","state":["exists","up"]},{"osd":2,"uuid":"01a73662-9a21-4b1f-bcbb-42b9636a4638","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6819","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6822","nonce":3018526646}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6825","nonce":3018526646}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6831","nonce":3018526646}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6827","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6829","nonce":3018526646}]},"public_addr":"172.21.15.16:6822/3018526646","cluster_addr":"172.21.15.16:6825/3018526646","heartbeat_back_addr":"172.21.15.16:6831/3018526646","heartbeat_front_addr":"172.21.15.16:6829/3018526646","state":["exists","up"]},{"osd":3,"uuid":"c7771fd3-e18e-4500-ac38-23cf10ebcd75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6809","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6812","nonce":3360352974}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6817","nonce":3360352974}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6828","nonce":3360352974}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6823","nonce":3360352974}]},"public_addr":"172.21.15.16:6812/3360352974","cluster_addr":"172.21.15.16:6817/3360352974","heartbeat_back_addr":"172.21.15.16:6828/3360352974","heartbeat_front_addr":"172.21.15.16:6823/3360352974","state":["exists","up"]},{"osd":4,"uuid":"88260730-1482-4fbd-a40f-cb1124e9df4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6801","nonce":1856844856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6803","nonce":1856844856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6807","nonce":1856844856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6805","nonce":1856844856}]},"public_addr":"172.21.15.89:6801/1856844856","cluster_addr":"172.21.15.89:6803/1856844856","heartbeat_back_addr":"172.21.15.89:6807/1856844856","heartbeat_front_addr":"172.21.15.89:6805/1856844856","state":["exists","up"]},{"osd":5,"uuid":"4cb2c00c-fae0-4dd8-9d27-ec4f0093ccf7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6814","nonce":50395527}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6817","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6820","nonce":50395527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6831","nonce":50395527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6823","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6826","nonce":50395527}]},"public_addr":"172.21.15.89:6814/50395527","cluster_addr":"172.21.15.89:6820/50395527","heartbeat_back_addr":"172.21.15.89:6831/50395527","heartbeat_front_addr":"172.21.15.89:6826/50395527","state":["exists","up"]},{"osd":6,"uuid":"81ce0c19-2554-4cae-acbe-66a05e9e9b9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6809","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6812","nonce":3470471752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6815","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6818","nonce":3470471752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6827","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6829","nonce":3470471752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6821","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6824","nonce":3470471752}]},"public_addr":"172.21.15.89:6812/3470471752","cluster_addr":"172.21.15.89:6818/3470471752","heartbeat_back_addr":"172.21.15.89:6829/3470471752","heartbeat_front_addr":"172.21.15.89:6824/3470471752","state":["exists","up"]},{"osd":7,"uuid":"85e4313b-d37d-442e-8042-153bf20a847d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6811","nonce":1796778630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6813","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6816","nonce":1796778630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6825","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6828","nonce":1796778630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6819","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6822","nonce":1796778630}]},"public_addr":"172.21.15.89:6811/1796778630","cluster_addr":"172.21.15.89:6816/1796778630","heartbeat_back_addr":"172.21.15.89:6828/1796778630","heartbeat_front_addr":"172.21.15.89:6822/1796778630","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.439088+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.529970+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.680512+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:21.044161+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.582111+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.536558+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.409522+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.590229+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.2","mappings":[{"from":3,"to":2}]},{"pgid":"3.7","mappings":[{"from":3,"to":2}]},{"pgid":"3.d","mappings":[{"from":5,"to":1}]},{"pgid":"3.22","mappings":[{"from":5,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-31T15:04:27.818 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:27.824+0000 7f58cbc00640 1 -- 172.21.15.16:0/3580620354 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f58a40683b0 msgr2=0x7f58a406a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:27.818 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:27.824+0000 7f58cbc00640 1 --2- 172.21.15.16:0/3580620354 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f58a40683b0 0x7f58a406a870 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f58b8009ae0 tx=0x7f58b8009190 comp rx=0 tx=0).stop 2024-03-31T15:04:27.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:27.824+0000 7f58cbc00640 1 -- 172.21.15.16:0/3580620354 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f58c4145aa0 msgr2=0x7f58c4156aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:27.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:27.824+0000 7f58cbc00640 1 --2- 172.21.15.16:0/3580620354 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f58c4145aa0 0x7f58c4156aa0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f58bc01b0a0 tx=0x7f58bc019790 comp rx=0 tx=0).stop 2024-03-31T15:04:27.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:27.824+0000 7f58cbc00640 1 -- 172.21.15.16:0/3580620354 shutdown_connections 2024-03-31T15:04:27.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:27.824+0000 7f58cbc00640 1 --2- 172.21.15.16:0/3580620354 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f58a40683b0 0x7f58a406a870 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:27.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:27.824+0000 7f58cbc00640 1 --2- 172.21.15.16:0/3580620354 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f58c4145aa0 0x7f58c4156aa0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:27.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:27.824+0000 7f58cbc00640 1 --2- 172.21.15.16:0/3580620354 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f58c406c1c0 0x7f58c413d3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:27.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:27.824+0000 7f58cbc00640 1 --2- 172.21.15.16:0/3580620354 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f58c406b620 0x7f58c4154870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:27.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:27.824+0000 7f58cbc00640 1 -- 172.21.15.16:0/3580620354 >> 172.21.15.16:0/3580620354 conn(0x7f58c40d1060 msgr2=0x7f58c40b25b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:04:27.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:27.824+0000 7f58cbc00640 1 -- 172.21.15.16:0/3580620354 shutdown_connections 2024-03-31T15:04:27.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:27.824+0000 7f58cbc00640 1 -- 172.21.15.16:0/3580620354 wait complete. 2024-03-31T15:04:27.837 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-31T15:04:27.837 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-31T15:04:27.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:27.980+0000 7f5c5be00640 1 Processor -- start 2024-03-31T15:04:27.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:27.980+0000 7f5c5be00640 1 -- start start 2024-03-31T15:04:27.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:27.984+0000 7f5c5be00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f5c540b23d0 0x7f5c54071d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:27.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:27.984+0000 7f5c5be00640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5c5415f3c0 con 0x7f5c5415eee0 2024-03-31T15:04:27.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:27.984+0000 7f5c5be00640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5c5415f6a0 con 0x7f5c5415b3d0 2024-03-31T15:04:27.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:27.984+0000 7f5c5be00640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f5c5415b7b0 con 0x7f5c540b23d0 2024-03-31T15:04:27.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:27.984+0000 7f5c58a00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f5c540b23d0 0x7f5c54071d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:27.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:27.984+0000 7f5c58a00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f5c540b23d0 0x7f5c54071d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43728/0 (socket says 172.21.15.16:43728) 2024-03-31T15:04:27.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:27.984+0000 7f5c58a00640 1 -- 172.21.15.16:0/235405841 learned_addr learned my addr 172.21.15.16:0/235405841 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:04:27.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:27.984+0000 7f5c59e00640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f5c5415eee0 0x7f5c54072250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:33676/0 (socket says 172.21.15.16:33676) 2024-03-31T15:04:27.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:27.984+0000 7f5c58a00640 1 -- 172.21.15.16:0/235405841 >> v1:172.21.15.89:6789/0 conn(0x7f5c5415b3d0 legacy=0x7f5c540b1cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:27.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:27.984+0000 7f5c58a00640 1 -- 172.21.15.16:0/235405841 >> v1:172.21.15.16:6789/0 conn(0x7f5c5415eee0 legacy=0x7f5c54072250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:27.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:27.984+0000 7f5c53e00640 1 -- 172.21.15.16:0/235405841 <== mon.1 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3039199155 0 0) 0x7f5c5415f6a0 con 0x7f5c5415b3d0 2024-03-31T15:04:27.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:27.984+0000 7f5c58a00640 1 -- 172.21.15.16:0/235405841 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c5415f980 con 0x7f5c540b23d0 2024-03-31T15:04:27.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:27.988+0000 7f5c58a00640 1 --2- 172.21.15.16:0/235405841 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5c540b23d0 0x7f5c54071d10 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f5c4000bb10 tx=0x7f5c4001eac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55291ed628abc9cf server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:27.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:27.988+0000 7f5c53e00640 1 -- 172.21.15.16:0/235405841 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c400b4070 con 0x7f5c540b23d0 2024-03-31T15:04:27.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:27.988+0000 7f5c53e00640 1 -- 172.21.15.16:0/235405841 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5c400a7b30 con 0x7f5c540b23d0 2024-03-31T15:04:27.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:27.988+0000 7f5c53e00640 1 -- 172.21.15.16:0/235405841 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c400af440 con 0x7f5c540b23d0 2024-03-31T15:04:27.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:27.988+0000 7f5c5be00640 1 -- 172.21.15.16:0/235405841 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5c540b23d0 msgr2=0x7f5c54071d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:27.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:27.988+0000 7f5c5be00640 1 --2- 172.21.15.16:0/235405841 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5c540b23d0 0x7f5c54071d10 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f5c4000bb10 tx=0x7f5c4001eac0 comp rx=0 tx=0).stop 2024-03-31T15:04:27.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:27.988+0000 7f5c5be00640 1 -- 172.21.15.16:0/235405841 shutdown_connections 2024-03-31T15:04:27.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:27.988+0000 7f5c5be00640 1 --2- 172.21.15.16:0/235405841 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5c540b23d0 0x7f5c54071d10 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:27.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:27.988+0000 7f5c5be00640 1 -- 172.21.15.16:0/235405841 >> 172.21.15.16:0/235405841 conn(0x7f5c5406b620 msgr2=0x7f5c5406ba30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:04:27.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:27.988+0000 7f5c5be00640 1 -- 172.21.15.16:0/235405841 shutdown_connections 2024-03-31T15:04:27.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:27.988+0000 7f5c5be00640 1 -- 172.21.15.16:0/235405841 wait complete. 2024-03-31T15:04:27.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:27.988+0000 7f5c5be00640 1 Processor -- start 2024-03-31T15:04:27.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:27.988+0000 7f5c5be00640 1 -- start start 2024-03-31T15:04:27.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:27.992+0000 7f5c5be00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f5c540b23d0 0x7f5c540b10a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:27.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:27.992+0000 7f5c5be00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5c5415b3d0 0x7f5c541526d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:27.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:27.992+0000 7f5c5be00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5c5415eee0 0x7f5c54152c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:27.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:27.992+0000 7f5c5be00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f5c54072960 con 0x7f5c5415b3d0 2024-03-31T15:04:27.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:27.992+0000 7f5c5be00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f5c540717c0 con 0x7f5c5415eee0 2024-03-31T15:04:27.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:27.992+0000 7f5c5be00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f5c54072ae0 con 0x7f5c540b23d0 2024-03-31T15:04:27.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:27.992+0000 7f5c58a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5c5415b3d0 0x7f5c541526d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:27.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:27.992+0000 7f5c58a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5c5415b3d0 0x7f5c541526d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43740/0 (socket says 172.21.15.16:43740) 2024-03-31T15:04:27.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:27.992+0000 7f5c58a00640 1 -- 172.21.15.16:0/101986417 learned_addr learned my addr 172.21.15.16:0/101986417 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:04:27.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:27.992+0000 7f5c59400640 1 --2- 172.21.15.16:0/101986417 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f5c540b23d0 0x7f5c540b10a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:27.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:27.992+0000 7f5c59e00640 1 --2- 172.21.15.16:0/101986417 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5c5415eee0 0x7f5c54152c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:27.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:27.992+0000 7f5c58a00640 1 -- 172.21.15.16:0/101986417 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f5c540b23d0 msgr2=0x7f5c540b10a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:27.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:27.992+0000 7f5c58a00640 1 --2- 172.21.15.16:0/101986417 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f5c540b23d0 0x7f5c540b10a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:27.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:27.992+0000 7f5c58a00640 1 -- 172.21.15.16:0/101986417 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5c5415eee0 msgr2=0x7f5c54152c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:27.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:27.992+0000 7f5c58a00640 1 --2- 172.21.15.16:0/101986417 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5c5415eee0 0x7f5c54152c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:27.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:27.992+0000 7f5c58a00640 1 -- 172.21.15.16:0/101986417 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c541f8660 con 0x7f5c5415b3d0 2024-03-31T15:04:27.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:27.992+0000 7f5c58a00640 1 --2- 172.21.15.16:0/101986417 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5c5415b3d0 0x7f5c541526d0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f5c4001b6d0 tx=0x7f5c4001d300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:27.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:27.992+0000 7f5c51600640 1 -- 172.21.15.16:0/101986417 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c400bf070 con 0x7f5c5415b3d0 2024-03-31T15:04:27.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:27.992+0000 7f5c51600640 1 -- 172.21.15.16:0/101986417 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5c400b8770 con 0x7f5c5415b3d0 2024-03-31T15:04:27.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:27.992+0000 7f5c51600640 1 -- 172.21.15.16:0/101986417 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c400b4070 con 0x7f5c5415b3d0 2024-03-31T15:04:27.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:27.992+0000 7f5c5be00640 1 -- 172.21.15.16:0/101986417 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5c54159d90 con 0x7f5c5415b3d0 2024-03-31T15:04:27.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:27.992+0000 7f5c5be00640 1 -- 172.21.15.16:0/101986417 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5c5415a350 con 0x7f5c5415b3d0 2024-03-31T15:04:27.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:27.992+0000 7f5c5be00640 1 -- 172.21.15.16:0/101986417 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5c54069950 con 0x7f5c5415b3d0 2024-03-31T15:04:27.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:27.992+0000 7f5c51600640 1 -- 172.21.15.16:0/101986417 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f5c400b8b60 con 0x7f5c5415b3d0 2024-03-31T15:04:27.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:27.996+0000 7f5c51600640 1 --2- 172.21.15.16:0/101986417 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f5c34068380 0x7f5c3406a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:27.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:27.996+0000 7f5c51600640 1 -- 172.21.15.16:0/101986417 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f5c40121670 con 0x7f5c5415b3d0 2024-03-31T15:04:27.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:27.996+0000 7f5c59400640 1 --2- 172.21.15.16:0/101986417 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f5c34068380 0x7f5c3406a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:27.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:27.996+0000 7f5c59400640 1 --2- 172.21.15.16:0/101986417 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f5c34068380 0x7f5c3406a840 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f5c48000f70 tx=0x7f5c48009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:27.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:28.000+0000 7f5c51600640 1 -- 172.21.15.16:0/101986417 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f5c40018740 con 0x7f5c5415b3d0 2024-03-31T15:04:28.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:28.196+0000 7f5c5be00640 1 -- 172.21.15.16:0/101986417 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5c540cdea0 con 0x7f5c5415b3d0 2024-03-31T15:04:28.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:28.196+0000 7f5c51600640 1 -- 172.21.15.16:0/101986417 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+5172 (secure 0 0 0) 0x7f5c400f14c0 con 0x7f5c5415b3d0 2024-03-31T15:04:28.192 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:04:28.192 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4276,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/2832752118","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6833","nonce":2832752118},{"type":"v1","addr":"172.21.15.16:6835","nonce":2832752118}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4278,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6834/153294668","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":153294668},{"type":"v1","addr":"172.21.15.16:6834","nonce":153294668}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4288,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/1653319357","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":1653319357},{"type":"v1","addr":"172.21.15.89:6835","nonce":1653319357}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-31T15:03:36.349865+0000","modified":"2024-03-31T15:03:39.378181+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4293},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.89:6837/2583453131","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":2583453131},{"type":"v1","addr":"172.21.15.89:6837","nonce":2583453131}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4293,"qdb_cluster":[4293]},"id":1}]} 2024-03-31T15:04:28.192 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 8 2024-03-31T15:04:28.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:28.200+0000 7f5c5be00640 1 -- 172.21.15.16:0/101986417 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f5c34068380 msgr2=0x7f5c3406a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:28.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:28.200+0000 7f5c5be00640 1 --2- 172.21.15.16:0/101986417 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f5c34068380 0x7f5c3406a840 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f5c48000f70 tx=0x7f5c48009040 comp rx=0 tx=0).stop 2024-03-31T15:04:28.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:28.200+0000 7f5c5be00640 1 -- 172.21.15.16:0/101986417 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5c5415b3d0 msgr2=0x7f5c541526d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:28.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:28.200+0000 7f5c5be00640 1 --2- 172.21.15.16:0/101986417 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5c5415b3d0 0x7f5c541526d0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f5c4001b6d0 tx=0x7f5c4001d300 comp rx=0 tx=0).stop 2024-03-31T15:04:28.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:28.200+0000 7f5c5be00640 1 -- 172.21.15.16:0/101986417 shutdown_connections 2024-03-31T15:04:28.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:28.200+0000 7f5c5be00640 1 --2- 172.21.15.16:0/101986417 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f5c34068380 0x7f5c3406a840 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:28.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:28.200+0000 7f5c5be00640 1 --2- 172.21.15.16:0/101986417 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5c5415eee0 0x7f5c54152c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:28.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:28.200+0000 7f5c5be00640 1 --2- 172.21.15.16:0/101986417 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5c5415b3d0 0x7f5c541526d0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:28.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:28.200+0000 7f5c5be00640 1 --2- 172.21.15.16:0/101986417 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f5c540b23d0 0x7f5c540b10a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:28.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:28.200+0000 7f5c5be00640 1 -- 172.21.15.16:0/101986417 >> 172.21.15.16:0/101986417 conn(0x7f5c5406b620 msgr2=0x7f5c540b3c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:04:28.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:28.200+0000 7f5c5be00640 1 -- 172.21.15.16:0/101986417 shutdown_connections 2024-03-31T15:04:28.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:28.200+0000 7f5c5be00640 1 -- 172.21.15.16:0/101986417 wait complete. 2024-03-31T15:04:28.213 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-31T15:04:28.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:28.356+0000 7fc952a00640 1 Processor -- start 2024-03-31T15:04:28.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:28.356+0000 7fc952a00640 1 -- start start 2024-03-31T15:04:28.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:28.360+0000 7fc952a00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fc94c15b3d0 0x7fc94c0b1cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:28.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:28.360+0000 7fc952a00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fc94c15eee0 0x7fc94c0b2200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:28.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:28.360+0000 7fc952a00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fc94c0ce2f0 0x7fc94c0cc9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:28.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:28.360+0000 7fc952a00640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7fc94c15b7b0 con 0x7fc94c15b3d0 2024-03-31T15:04:28.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:28.360+0000 7fc952a00640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7fc94c15f3c0 con 0x7fc94c15eee0 2024-03-31T15:04:28.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:28.360+0000 7fc952a00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fc94c15f6a0 con 0x7fc94c0ce2f0 2024-03-31T15:04:28.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:28.360+0000 7fc94be00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fc94c15b3d0 0x7fc94c0b1cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:28.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:28.360+0000 7fc94be00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fc94c15b3d0 0x7fc94c0b1cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43754/0 (socket says 172.21.15.16:43754) 2024-03-31T15:04:28.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:28.360+0000 7fc94be00640 1 -- 172.21.15.16:0/2996458812 learned_addr learned my addr 172.21.15.16:0/2996458812 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:04:28.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:28.360+0000 7fc943400640 1 --2- 172.21.15.16:0/2996458812 >> v2:172.21.15.89:3300/0 conn(0x7fc94c15eee0 0x7fc94c0b2200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:28.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:28.360+0000 7fc950a00640 1 --2- 172.21.15.16:0/2996458812 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fc94c0ce2f0 0x7fc94c0cc9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:28.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:28.360+0000 7fc94be00640 1 -- 172.21.15.16:0/2996458812 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fc94c0ce2f0 msgr2=0x7fc94c0cc9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:04:28.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:28.360+0000 7fc94be00640 1 --2- 172.21.15.16:0/2996458812 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fc94c0ce2f0 0x7fc94c0cc9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:28.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:28.360+0000 7fc94be00640 1 -- 172.21.15.16:0/2996458812 >> v2:172.21.15.89:3300/0 conn(0x7fc94c15eee0 msgr2=0x7fc94c0b2200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:04:28.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:28.360+0000 7fc94be00640 1 --2- 172.21.15.16:0/2996458812 >> v2:172.21.15.89:3300/0 conn(0x7fc94c15eee0 0x7fc94c0b2200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:28.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:28.360+0000 7fc94be00640 1 -- 172.21.15.16:0/2996458812 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc94c15f980 con 0x7fc94c15b3d0 2024-03-31T15:04:28.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:28.364+0000 7fc94be00640 1 --2- 172.21.15.16:0/2996458812 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc94c15b3d0 0x7fc94c0b1cc0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fc94c0cdae0 tx=0x7fc93c01c4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=89368bc8134e181a server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:28.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:28.364+0000 7fc94b400640 1 -- 172.21.15.16:0/2996458812 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc93c0a7070 con 0x7fc94c15b3d0 2024-03-31T15:04:28.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:28.364+0000 7fc94b400640 1 -- 172.21.15.16:0/2996458812 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc93c0b0070 con 0x7fc94c15b3d0 2024-03-31T15:04:28.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:28.364+0000 7fc94b400640 1 -- 172.21.15.16:0/2996458812 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc93c0ab890 con 0x7fc94c15b3d0 2024-03-31T15:04:28.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:28.364+0000 7fc952a00640 1 -- 172.21.15.16:0/2996458812 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc94c15b3d0 msgr2=0x7fc94c0b1cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:28.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:28.364+0000 7fc952a00640 1 --2- 172.21.15.16:0/2996458812 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc94c15b3d0 0x7fc94c0b1cc0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fc94c0cdae0 tx=0x7fc93c01c4f0 comp rx=0 tx=0).stop 2024-03-31T15:04:28.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:28.364+0000 7fc952a00640 1 -- 172.21.15.16:0/2996458812 shutdown_connections 2024-03-31T15:04:28.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:28.364+0000 7fc952a00640 1 --2- 172.21.15.16:0/2996458812 >> v2:172.21.15.89:3300/0 conn(0x7fc94c15eee0 0x7fc94c0b2200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:28.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:28.364+0000 7fc952a00640 1 --2- 172.21.15.16:0/2996458812 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fc94c0ce2f0 0x7fc94c0cc9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:28.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:28.364+0000 7fc952a00640 1 --2- 172.21.15.16:0/2996458812 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc94c15b3d0 0x7fc94c0b1cc0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:28.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:28.364+0000 7fc952a00640 1 -- 172.21.15.16:0/2996458812 >> 172.21.15.16:0/2996458812 conn(0x7fc94c06b620 msgr2=0x7fc94c06ba30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:04:28.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:28.364+0000 7fc952a00640 1 -- 172.21.15.16:0/2996458812 shutdown_connections 2024-03-31T15:04:28.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:28.364+0000 7fc952a00640 1 -- 172.21.15.16:0/2996458812 wait complete. 2024-03-31T15:04:28.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:28.364+0000 7fc952a00640 1 Processor -- start 2024-03-31T15:04:28.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:28.364+0000 7fc952a00640 1 -- start start 2024-03-31T15:04:28.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:28.364+0000 7fc952a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc94c0ce2f0 0x7fc94c156fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:28.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:28.364+0000 7fc952a00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fc94c15b3d0 0x7fc94c152d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:28.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:28.368+0000 7fc952a00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc94c15eee0 0x7fc94c149170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:28.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:28.368+0000 7fc952a00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fc94c071e90 con 0x7fc94c0ce2f0 2024-03-31T15:04:28.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:28.368+0000 7fc952a00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fc94c071d10 con 0x7fc94c15eee0 2024-03-31T15:04:28.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:28.368+0000 7fc952a00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fc94c072010 con 0x7fc94c15b3d0 2024-03-31T15:04:28.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:28.368+0000 7fc94be00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc94c0ce2f0 0x7fc94c156fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:28.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:28.368+0000 7fc94be00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc94c0ce2f0 0x7fc94c156fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43768/0 (socket says 172.21.15.16:43768) 2024-03-31T15:04:28.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:28.368+0000 7fc94be00640 1 -- 172.21.15.16:0/1184926861 learned_addr learned my addr 172.21.15.16:0/1184926861 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:04:28.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:28.368+0000 7fc943400640 1 --2- 172.21.15.16:0/1184926861 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fc94c15b3d0 0x7fc94c152d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:28.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:28.368+0000 7fc950a00640 1 --2- 172.21.15.16:0/1184926861 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc94c15eee0 0x7fc94c149170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:28.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:28.368+0000 7fc94be00640 1 -- 172.21.15.16:0/1184926861 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fc94c15b3d0 msgr2=0x7fc94c152d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:28.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:28.368+0000 7fc94be00640 1 --2- 172.21.15.16:0/1184926861 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fc94c15b3d0 0x7fc94c152d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:28.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:28.368+0000 7fc94be00640 1 -- 172.21.15.16:0/1184926861 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc94c15eee0 msgr2=0x7fc94c149170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:28.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:28.368+0000 7fc94be00640 1 --2- 172.21.15.16:0/1184926861 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc94c15eee0 0x7fc94c149170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:28.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:28.368+0000 7fc94be00640 1 -- 172.21.15.16:0/1184926861 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc94c0aff30 con 0x7fc94c0ce2f0 2024-03-31T15:04:28.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:28.368+0000 7fc950a00640 1 --2- 172.21.15.16:0/1184926861 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc94c15eee0 0x7fc94c149170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:04:28.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:28.368+0000 7fc94be00640 1 --2- 172.21.15.16:0/1184926861 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc94c0ce2f0 0x7fc94c156fd0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fc93c01aba0 tx=0x7fc93c01ad30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:28.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:28.368+0000 7fc948c00640 1 -- 172.21.15.16:0/1184926861 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc93c0be5f0 con 0x7fc94c0ce2f0 2024-03-31T15:04:28.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:28.368+0000 7fc948c00640 1 -- 172.21.15.16:0/1184926861 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc93c0be790 con 0x7fc94c0ce2f0 2024-03-31T15:04:28.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:28.368+0000 7fc948c00640 1 -- 172.21.15.16:0/1184926861 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc93c0beb60 con 0x7fc94c0ce2f0 2024-03-31T15:04:28.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:28.368+0000 7fc952a00640 1 -- 172.21.15.16:0/1184926861 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc94c0b0210 con 0x7fc94c0ce2f0 2024-03-31T15:04:28.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:28.368+0000 7fc952a00640 1 -- 172.21.15.16:0/1184926861 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc94c157510 con 0x7fc94c0ce2f0 2024-03-31T15:04:28.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:28.368+0000 7fc952a00640 1 -- 172.21.15.16:0/1184926861 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc94c0b0600 con 0x7fc94c0ce2f0 2024-03-31T15:04:28.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:28.368+0000 7fc948c00640 1 -- 172.21.15.16:0/1184926861 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fc93c0b43f0 con 0x7fc94c0ce2f0 2024-03-31T15:04:28.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:28.372+0000 7fc948c00640 1 --2- 172.21.15.16:0/1184926861 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fc928068380 0x7fc92806a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:28.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:28.372+0000 7fc948c00640 1 -- 172.21.15.16:0/1184926861 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7fc93c121710 con 0x7fc94c0ce2f0 2024-03-31T15:04:28.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:28.372+0000 7fc943400640 1 --2- 172.21.15.16:0/1184926861 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fc928068380 0x7fc92806a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:28.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:28.372+0000 7fc943400640 1 --2- 172.21.15.16:0/1184926861 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fc928068380 0x7fc92806a840 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fc94c14e750 tx=0x7fc934009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:28.369 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:28.376+0000 7fc948c00640 1 -- 172.21.15.16:0/1184926861 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fc93c0c4060 con 0x7fc94c0ce2f0 2024-03-31T15:04:28.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:28.576+0000 7fc952a00640 1 -- 172.21.15.16:0/1184926861 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc94c0b03e0 con 0x7fc94c0ce2f0 2024-03-31T15:04:28.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:28.576+0000 7fc948c00640 1 -- 172.21.15.16:0/1184926861 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+5172 (secure 0 0 0) 0x7fc93c016110 con 0x7fc94c0ce2f0 2024-03-31T15:04:28.570 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:04:28.571 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4276,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/2832752118","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6833","nonce":2832752118},{"type":"v1","addr":"172.21.15.16:6835","nonce":2832752118}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4278,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6834/153294668","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":153294668},{"type":"v1","addr":"172.21.15.16:6834","nonce":153294668}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4288,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/1653319357","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":1653319357},{"type":"v1","addr":"172.21.15.89:6835","nonce":1653319357}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-31T15:03:36.349865+0000","modified":"2024-03-31T15:03:39.378181+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4293},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.89:6837/2583453131","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":2583453131},{"type":"v1","addr":"172.21.15.89:6837","nonce":2583453131}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4293,"qdb_cluster":[4293]},"id":1}]} 2024-03-31T15:04:28.571 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 8 2024-03-31T15:04:28.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:28.576+0000 7fc952a00640 1 -- 172.21.15.16:0/1184926861 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fc928068380 msgr2=0x7fc92806a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:28.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:28.576+0000 7fc952a00640 1 --2- 172.21.15.16:0/1184926861 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fc928068380 0x7fc92806a840 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fc94c14e750 tx=0x7fc934009190 comp rx=0 tx=0).stop 2024-03-31T15:04:28.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:28.576+0000 7fc952a00640 1 -- 172.21.15.16:0/1184926861 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc94c0ce2f0 msgr2=0x7fc94c156fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:28.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:28.576+0000 7fc952a00640 1 --2- 172.21.15.16:0/1184926861 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc94c0ce2f0 0x7fc94c156fd0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fc93c01aba0 tx=0x7fc93c01ad30 comp rx=0 tx=0).stop 2024-03-31T15:04:28.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:28.576+0000 7fc952a00640 1 -- 172.21.15.16:0/1184926861 shutdown_connections 2024-03-31T15:04:28.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:28.576+0000 7fc952a00640 1 --2- 172.21.15.16:0/1184926861 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fc928068380 0x7fc92806a840 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:28.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:28.576+0000 7fc952a00640 1 --2- 172.21.15.16:0/1184926861 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc94c15eee0 0x7fc94c149170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:28.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:28.576+0000 7fc952a00640 1 --2- 172.21.15.16:0/1184926861 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fc94c15b3d0 0x7fc94c152d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:28.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:28.576+0000 7fc952a00640 1 --2- 172.21.15.16:0/1184926861 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc94c0ce2f0 0x7fc94c156fd0 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:28.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:28.576+0000 7fc952a00640 1 -- 172.21.15.16:0/1184926861 >> 172.21.15.16:0/1184926861 conn(0x7fc94c06b620 msgr2=0x7fc94c0728a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:04:28.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:28.576+0000 7fc952a00640 1 -- 172.21.15.16:0/1184926861 shutdown_connections 2024-03-31T15:04:28.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:28.576+0000 7fc952a00640 1 -- 172.21.15.16:0/1184926861 wait complete. 2024-03-31T15:04:28.591 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-31T15:04:28.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:28.736+0000 7f7c8ac00640 1 Processor -- start 2024-03-31T15:04:28.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:28.736+0000 7f7c8ac00640 1 -- start start 2024-03-31T15:04:28.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:28.736+0000 7f7c8ac00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f7c8415b3d0 0x7f7c840b1cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:28.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:28.736+0000 7f7c8ac00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f7c8415eee0 0x7f7c840b2200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:28.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:28.736+0000 7f7c8ac00640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7c8415f3c0 con 0x7f7c840ce2f0 2024-03-31T15:04:28.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:28.736+0000 7f7c8ac00640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f7c8415b7b0 con 0x7f7c8415eee0 2024-03-31T15:04:28.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:28.736+0000 7f7c8ac00640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f7c8415f6a0 con 0x7f7c8415b3d0 2024-03-31T15:04:28.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:28.740+0000 7f7c7b400640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f7c8415eee0 0x7f7c840b2200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:28.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:28.740+0000 7f7c7b400640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f7c8415eee0 0x7f7c840b2200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43778/0 (socket says 172.21.15.16:43778) 2024-03-31T15:04:28.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:28.740+0000 7f7c7b400640 1 -- 172.21.15.16:0/2079742832 learned_addr learned my addr 172.21.15.16:0/2079742832 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:04:28.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:28.740+0000 7f7c83e00640 1 --2- 172.21.15.16:0/2079742832 >> v2:172.21.15.89:3300/0 conn(0x7f7c8415b3d0 0x7f7c840b1cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:28.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:28.740+0000 7f7c7b400640 1 -- 172.21.15.16:0/2079742832 >> v2:172.21.15.89:3300/0 conn(0x7f7c8415b3d0 msgr2=0x7f7c840b1cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:04:28.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:28.740+0000 7f7c7b400640 1 --2- 172.21.15.16:0/2079742832 >> v2:172.21.15.89:3300/0 conn(0x7f7c8415b3d0 0x7f7c840b1cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:28.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:28.740+0000 7f7c7b400640 1 -- 172.21.15.16:0/2079742832 >> v1:172.21.15.89:6789/0 conn(0x7f7c840ce2f0 legacy=0x7f7c840cc9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:28.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:28.740+0000 7f7c7b400640 1 -- 172.21.15.16:0/2079742832 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c8415f980 con 0x7f7c8415eee0 2024-03-31T15:04:28.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:28.740+0000 7f7c7b400640 1 --2- 172.21.15.16:0/2079742832 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7c8415eee0 0x7f7c840b2200 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f7c840691a0 tx=0x7f7c6c01c4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8c872987c97f450 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:28.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:28.740+0000 7f7c83400640 1 -- 172.21.15.16:0/2079742832 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c6c0aa610 con 0x7f7c8415eee0 2024-03-31T15:04:28.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:28.740+0000 7f7c83400640 1 -- 172.21.15.16:0/2079742832 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7c6c0af070 con 0x7f7c8415eee0 2024-03-31T15:04:28.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:28.740+0000 7f7c83400640 1 -- 172.21.15.16:0/2079742832 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c6c0aa9d0 con 0x7f7c8415eee0 2024-03-31T15:04:28.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:28.744+0000 7f7c8ac00640 1 -- 172.21.15.16:0/2079742832 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7c8415eee0 msgr2=0x7f7c840b2200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:28.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:28.744+0000 7f7c8ac00640 1 --2- 172.21.15.16:0/2079742832 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7c8415eee0 0x7f7c840b2200 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f7c840691a0 tx=0x7f7c6c01c4f0 comp rx=0 tx=0).stop 2024-03-31T15:04:28.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:28.744+0000 7f7c8ac00640 1 -- 172.21.15.16:0/2079742832 shutdown_connections 2024-03-31T15:04:28.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:28.744+0000 7f7c8ac00640 1 --2- 172.21.15.16:0/2079742832 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7c8415eee0 0x7f7c840b2200 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:28.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:28.744+0000 7f7c8ac00640 1 --2- 172.21.15.16:0/2079742832 >> v2:172.21.15.89:3300/0 conn(0x7f7c8415b3d0 0x7f7c840b1cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:28.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:28.744+0000 7f7c8ac00640 1 -- 172.21.15.16:0/2079742832 >> 172.21.15.16:0/2079742832 conn(0x7f7c8406b620 msgr2=0x7f7c8406ba30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:04:28.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:28.744+0000 7f7c8ac00640 1 -- 172.21.15.16:0/2079742832 shutdown_connections 2024-03-31T15:04:28.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:28.744+0000 7f7c8ac00640 1 -- 172.21.15.16:0/2079742832 wait complete. 2024-03-31T15:04:28.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:28.744+0000 7f7c8ac00640 1 Processor -- start 2024-03-31T15:04:28.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:28.744+0000 7f7c8ac00640 1 -- start start 2024-03-31T15:04:28.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:28.744+0000 7f7c8ac00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7c840ce2f0 0x7f7c8414ba00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:28.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:28.744+0000 7f7c8ac00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f7c8415b3d0 0x7f7c84153260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:28.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:28.744+0000 7f7c8ac00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7c8415eee0 0x7f7c8415e060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:28.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:28.744+0000 7f7c8ac00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f7c84168610 con 0x7f7c840ce2f0 2024-03-31T15:04:28.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:28.744+0000 7f7c8ac00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f7c84168490 con 0x7f7c8415eee0 2024-03-31T15:04:28.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:28.744+0000 7f7c8ac00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f7c84168790 con 0x7f7c8415b3d0 2024-03-31T15:04:28.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:28.744+0000 7f7c83e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7c840ce2f0 0x7f7c8414ba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:28.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:28.744+0000 7f7c83e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7c840ce2f0 0x7f7c8414ba00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43794/0 (socket says 172.21.15.16:43794) 2024-03-31T15:04:28.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:28.744+0000 7f7c83e00640 1 -- 172.21.15.16:0/311299703 learned_addr learned my addr 172.21.15.16:0/311299703 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:04:28.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:28.744+0000 7f7c7b400640 1 --2- 172.21.15.16:0/311299703 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f7c8415b3d0 0x7f7c84153260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:28.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:28.744+0000 7f7c88c00640 1 --2- 172.21.15.16:0/311299703 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7c8415eee0 0x7f7c8415e060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:28.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:28.744+0000 7f7c83e00640 1 -- 172.21.15.16:0/311299703 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f7c8415b3d0 msgr2=0x7f7c84153260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:28.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:28.744+0000 7f7c83e00640 1 --2- 172.21.15.16:0/311299703 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f7c8415b3d0 0x7f7c84153260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:28.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:28.744+0000 7f7c83e00640 1 -- 172.21.15.16:0/311299703 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7c8415eee0 msgr2=0x7f7c8415e060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:28.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:28.744+0000 7f7c83e00640 1 --2- 172.21.15.16:0/311299703 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7c8415eee0 0x7f7c8415e060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:28.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:28.744+0000 7f7c83e00640 1 -- 172.21.15.16:0/311299703 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c8415c930 con 0x7f7c840ce2f0 2024-03-31T15:04:28.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:28.744+0000 7f7c83e00640 1 --2- 172.21.15.16:0/311299703 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7c840ce2f0 0x7f7c8414ba00 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f7c7400efc0 tx=0x7f7c7400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:28.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:28.748+0000 7f7c80c00640 1 -- 172.21.15.16:0/311299703 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c74018020 con 0x7f7c840ce2f0 2024-03-31T15:04:28.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:28.748+0000 7f7c80c00640 1 -- 172.21.15.16:0/311299703 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7c74014480 con 0x7f7c840ce2f0 2024-03-31T15:04:28.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:28.748+0000 7f7c80c00640 1 -- 172.21.15.16:0/311299703 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c74014850 con 0x7f7c840ce2f0 2024-03-31T15:04:28.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:28.748+0000 7f7c8ac00640 1 -- 172.21.15.16:0/311299703 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7c840afb10 con 0x7f7c840ce2f0 2024-03-31T15:04:28.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:28.748+0000 7f7c8ac00640 1 -- 172.21.15.16:0/311299703 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7c840b00d0 con 0x7f7c840ce2f0 2024-03-31T15:04:28.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:28.748+0000 7f7c8ac00640 1 -- 172.21.15.16:0/311299703 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7c84069950 con 0x7f7c840ce2f0 2024-03-31T15:04:28.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:28.748+0000 7f7c80c00640 1 -- 172.21.15.16:0/311299703 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f7c7401b020 con 0x7f7c840ce2f0 2024-03-31T15:04:28.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:28.748+0000 7f7c80c00640 1 --2- 172.21.15.16:0/311299703 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f7c60068380 0x7f7c6006a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:28.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:28.748+0000 7f7c80c00640 1 -- 172.21.15.16:0/311299703 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f7c740877a0 con 0x7f7c840ce2f0 2024-03-31T15:04:28.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:28.748+0000 7f7c7b400640 1 --2- 172.21.15.16:0/311299703 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f7c60068380 0x7f7c6006a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:28.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:28.752+0000 7f7c7b400640 1 --2- 172.21.15.16:0/311299703 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f7c60068380 0x7f7c6006a840 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f7c6c0ae000 tx=0x7f7c6c0a7000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:28.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:28.752+0000 7f7c80c00640 1 -- 172.21.15.16:0/311299703 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f7c74022030 con 0x7f7c840ce2f0 2024-03-31T15:04:28.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:28.920+0000 7f7c8ac00640 1 -- 172.21.15.16:0/311299703 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f7c840ce900 con 0x7f7c840ce2f0 2024-03-31T15:04:28.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:28.920+0000 7f7c80c00640 1 -- 172.21.15.16:0/311299703 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v23) ==== 74+0+16780 (secure 0 0 0) 0x7f7c74010040 con 0x7f7c840ce2f0 2024-03-31T15:04:28.914 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:04:28.915 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":23,"fsid":"286eb5da-fb47-4550-b290-d9519c13b9a5","created":"2024-03-31T15:03:11.438952+0000","modified":"2024-03-31T15:04:18.087947+0000","last_up_change":"2024-03-31T15:03:22.885535+0000","last_in_change":"2024-03-31T15:03:13.941667+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-31T15:03:22.550787+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-31T15:03:34.488425+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-31T15:03:35.299116+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"92a66450-ef19-40d8-ac3d-d760b23b0bda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6801","nonce":932712244}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6803","nonce":932712244}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6807","nonce":932712244}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6805","nonce":932712244}]},"public_addr":"172.21.15.16:6801/932712244","cluster_addr":"172.21.15.16:6803/932712244","heartbeat_back_addr":"172.21.15.16:6807/932712244","heartbeat_front_addr":"172.21.15.16:6805/932712244","state":["exists","up"]},{"osd":1,"uuid":"da9d1f2b-1968-4959-b843-faa231988e92","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6810","nonce":1398737533}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6811","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6813","nonce":1398737533}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6821","nonce":1398737533}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6815","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6816","nonce":1398737533}]},"public_addr":"172.21.15.16:6810/1398737533","cluster_addr":"172.21.15.16:6813/1398737533","heartbeat_back_addr":"172.21.15.16:6821/1398737533","heartbeat_front_addr":"172.21.15.16:6816/1398737533","state":["exists","up"]},{"osd":2,"uuid":"01a73662-9a21-4b1f-bcbb-42b9636a4638","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6819","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6822","nonce":3018526646}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6825","nonce":3018526646}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6831","nonce":3018526646}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6827","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6829","nonce":3018526646}]},"public_addr":"172.21.15.16:6822/3018526646","cluster_addr":"172.21.15.16:6825/3018526646","heartbeat_back_addr":"172.21.15.16:6831/3018526646","heartbeat_front_addr":"172.21.15.16:6829/3018526646","state":["exists","up"]},{"osd":3,"uuid":"c7771fd3-e18e-4500-ac38-23cf10ebcd75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6809","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6812","nonce":3360352974}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6817","nonce":3360352974}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6828","nonce":3360352974}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6823","nonce":3360352974}]},"public_addr":"172.21.15.16:6812/3360352974","cluster_addr":"172.21.15.16:6817/3360352974","heartbeat_back_addr":"172.21.15.16:6828/3360352974","heartbeat_front_addr":"172.21.15.16:6823/3360352974","state":["exists","up"]},{"osd":4,"uuid":"88260730-1482-4fbd-a40f-cb1124e9df4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6801","nonce":1856844856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6803","nonce":1856844856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6807","nonce":1856844856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6805","nonce":1856844856}]},"public_addr":"172.21.15.89:6801/1856844856","cluster_addr":"172.21.15.89:6803/1856844856","heartbeat_back_addr":"172.21.15.89:6807/1856844856","heartbeat_front_addr":"172.21.15.89:6805/1856844856","state":["exists","up"]},{"osd":5,"uuid":"4cb2c00c-fae0-4dd8-9d27-ec4f0093ccf7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6814","nonce":50395527}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6817","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6820","nonce":50395527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6831","nonce":50395527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6823","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6826","nonce":50395527}]},"public_addr":"172.21.15.89:6814/50395527","cluster_addr":"172.21.15.89:6820/50395527","heartbeat_back_addr":"172.21.15.89:6831/50395527","heartbeat_front_addr":"172.21.15.89:6826/50395527","state":["exists","up"]},{"osd":6,"uuid":"81ce0c19-2554-4cae-acbe-66a05e9e9b9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6809","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6812","nonce":3470471752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6815","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6818","nonce":3470471752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6827","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6829","nonce":3470471752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6821","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6824","nonce":3470471752}]},"public_addr":"172.21.15.89:6812/3470471752","cluster_addr":"172.21.15.89:6818/3470471752","heartbeat_back_addr":"172.21.15.89:6829/3470471752","heartbeat_front_addr":"172.21.15.89:6824/3470471752","state":["exists","up"]},{"osd":7,"uuid":"85e4313b-d37d-442e-8042-153bf20a847d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6811","nonce":1796778630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6813","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6816","nonce":1796778630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6825","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6828","nonce":1796778630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6819","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6822","nonce":1796778630}]},"public_addr":"172.21.15.89:6811/1796778630","cluster_addr":"172.21.15.89:6816/1796778630","heartbeat_back_addr":"172.21.15.89:6828/1796778630","heartbeat_front_addr":"172.21.15.89:6822/1796778630","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.439088+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.529970+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.680512+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:21.044161+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.582111+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.536558+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.409522+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.590229+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.2","mappings":[{"from":3,"to":2}]},{"pgid":"3.7","mappings":[{"from":3,"to":2}]},{"pgid":"3.d","mappings":[{"from":5,"to":1}]},{"pgid":"3.22","mappings":[{"from":5,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-31T15:04:28.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:28.920+0000 7f7c8ac00640 1 -- 172.21.15.16:0/311299703 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f7c60068380 msgr2=0x7f7c6006a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:28.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:28.920+0000 7f7c8ac00640 1 --2- 172.21.15.16:0/311299703 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f7c60068380 0x7f7c6006a840 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f7c6c0ae000 tx=0x7f7c6c0a7000 comp rx=0 tx=0).stop 2024-03-31T15:04:28.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:28.920+0000 7f7c8ac00640 1 -- 172.21.15.16:0/311299703 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7c840ce2f0 msgr2=0x7f7c8414ba00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:28.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:28.920+0000 7f7c8ac00640 1 --2- 172.21.15.16:0/311299703 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7c840ce2f0 0x7f7c8414ba00 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f7c7400efc0 tx=0x7f7c7400c490 comp rx=0 tx=0).stop 2024-03-31T15:04:28.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:28.920+0000 7f7c8ac00640 1 -- 172.21.15.16:0/311299703 shutdown_connections 2024-03-31T15:04:28.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:28.920+0000 7f7c8ac00640 1 --2- 172.21.15.16:0/311299703 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f7c60068380 0x7f7c6006a840 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:28.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:28.920+0000 7f7c8ac00640 1 --2- 172.21.15.16:0/311299703 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7c8415eee0 0x7f7c8415e060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:28.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:28.920+0000 7f7c8ac00640 1 --2- 172.21.15.16:0/311299703 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f7c8415b3d0 0x7f7c84153260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:28.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:28.920+0000 7f7c8ac00640 1 --2- 172.21.15.16:0/311299703 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7c840ce2f0 0x7f7c8414ba00 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:28.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:28.920+0000 7f7c8ac00640 1 -- 172.21.15.16:0/311299703 >> 172.21.15.16:0/311299703 conn(0x7f7c8406b620 msgr2=0x7f7c840cffc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:04:28.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:28.920+0000 7f7c8ac00640 1 -- 172.21.15.16:0/311299703 shutdown_connections 2024-03-31T15:04:28.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:28.920+0000 7f7c8ac00640 1 -- 172.21.15.16:0/311299703 wait complete. 2024-03-31T15:04:28.937 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-31T15:04:29.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:29.084+0000 7fbc07e00640 1 Processor -- start 2024-03-31T15:04:29.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:29.084+0000 7fbc07e00640 1 -- start start 2024-03-31T15:04:29.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:29.084+0000 7fbc07e00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fbc0815bd50 0x7fbc080b30e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:29.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:29.084+0000 7fbc07e00640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbc0815c720 con 0x7fbc0815b970 2024-03-31T15:04:29.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:29.084+0000 7fbc07e00640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbc0815ca00 con 0x7fbc0815c170 2024-03-31T15:04:29.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:29.084+0000 7fbc07e00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fbc0815cce0 con 0x7fbc0815bd50 2024-03-31T15:04:29.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:29.084+0000 7fbc06a00640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fbc0815b970 0x7fbc080b26b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:33680/0 (socket says 172.21.15.16:33680) 2024-03-31T15:04:29.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:29.084+0000 7fbc06a00640 1 -- 172.21.15.16:0/4081590234 learned_addr learned my addr 172.21.15.16:0/4081590234 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:04:29.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:29.084+0000 7fbbfe000640 1 --2- 172.21.15.16:0/4081590234 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fbc0815bd50 0x7fbc080b30e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:29.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:29.084+0000 7fbbfe000640 1 -- 172.21.15.16:0/4081590234 >> v1:172.21.15.89:6789/0 conn(0x7fbc0815c170 legacy=0x7fbc0815d7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:29.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:29.084+0000 7fbbfe000640 1 -- 172.21.15.16:0/4081590234 >> v1:172.21.15.16:6789/0 conn(0x7fbc0815b970 legacy=0x7fbc080b26b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:29.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:29.084+0000 7fbbfe000640 1 -- 172.21.15.16:0/4081590234 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc0815cfc0 con 0x7fbc0815bd50 2024-03-31T15:04:29.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:29.084+0000 7fbc06000640 1 -- 172.21.15.16:0/4081590234 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1221780512 0 0) 0x7fbc0815c720 con 0x7fbc0815b970 2024-03-31T15:04:29.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:29.088+0000 7fbbfe000640 1 --2- 172.21.15.16:0/4081590234 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fbc0815bd50 0x7fbc080b30e0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fbbf00097f0 tx=0x7fbbf001c420 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2604c2d6ed150172 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:29.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:29.088+0000 7fbc06000640 1 -- 172.21.15.16:0/4081590234 <== mon.2 v2:172.21.15.89:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbf00b0070 con 0x7fbc0815bd50 2024-03-31T15:04:29.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:29.088+0000 7fbc06000640 1 -- 172.21.15.16:0/4081590234 <== mon.2 v2:172.21.15.89:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbbf00ab600 con 0x7fbc0815bd50 2024-03-31T15:04:29.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:29.088+0000 7fbc06000640 1 -- 172.21.15.16:0/4081590234 <== mon.2 v2:172.21.15.89:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbf00ab9d0 con 0x7fbc0815bd50 2024-03-31T15:04:29.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:29.088+0000 7fbc07e00640 1 -- 172.21.15.16:0/4081590234 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fbc0815bd50 msgr2=0x7fbc080b30e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:29.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:29.088+0000 7fbc07e00640 1 --2- 172.21.15.16:0/4081590234 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fbc0815bd50 0x7fbc080b30e0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fbbf00097f0 tx=0x7fbbf001c420 comp rx=0 tx=0).stop 2024-03-31T15:04:29.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:29.088+0000 7fbc07e00640 1 -- 172.21.15.16:0/4081590234 shutdown_connections 2024-03-31T15:04:29.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:29.088+0000 7fbc07e00640 1 --2- 172.21.15.16:0/4081590234 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fbc0815bd50 0x7fbc080b30e0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:29.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:29.088+0000 7fbc07e00640 1 -- 172.21.15.16:0/4081590234 >> 172.21.15.16:0/4081590234 conn(0x7fbc080b5820 msgr2=0x7fbc080b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:04:29.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:29.088+0000 7fbc07e00640 1 -- 172.21.15.16:0/4081590234 shutdown_connections 2024-03-31T15:04:29.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:29.088+0000 7fbc07e00640 1 -- 172.21.15.16:0/4081590234 wait complete. 2024-03-31T15:04:29.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:29.092+0000 7fbc07e00640 1 Processor -- start 2024-03-31T15:04:29.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:29.092+0000 7fbc07e00640 1 -- start start 2024-03-31T15:04:29.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:29.092+0000 7fbc07e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbc0815b970 0x7fbc0814ac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:29.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:29.092+0000 7fbc07e00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fbc0815bd50 0x7fbc08155fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:29.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:29.092+0000 7fbc07e00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbc0815c170 0x7fbc0814d730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:29.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:29.092+0000 7fbc07e00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fbc08168cb0 con 0x7fbc0815b970 2024-03-31T15:04:29.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:29.092+0000 7fbc07e00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fbc0815e970 con 0x7fbc0815c170 2024-03-31T15:04:29.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:29.092+0000 7fbc07e00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fbc08168e30 con 0x7fbc0815bd50 2024-03-31T15:04:29.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:29.092+0000 7fbc06a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbc0815b970 0x7fbc0814ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:29.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:29.092+0000 7fbbfe000640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fbc0815bd50 0x7fbc08155fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:29.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:29.092+0000 7fbbfe000640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fbc0815bd50 0x7fbc08155fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3301/0 says I am v2:172.21.15.16:42702/0 (socket says 172.21.15.16:42702) 2024-03-31T15:04:29.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:29.092+0000 7fbc06a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbc0815b970 0x7fbc0814ac60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43804/0 (socket says 172.21.15.16:43804) 2024-03-31T15:04:29.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:29.092+0000 7fbbfe000640 1 -- 172.21.15.16:0/3832889997 learned_addr learned my addr 172.21.15.16:0/3832889997 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:04:29.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:29.092+0000 7fbc07400640 1 --2- 172.21.15.16:0/3832889997 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbc0815c170 0x7fbc0814d730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:29.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:29.092+0000 7fbbfe000640 1 -- 172.21.15.16:0/3832889997 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbc0815c170 msgr2=0x7fbc0814d730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:29.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:29.092+0000 7fbbfe000640 1 --2- 172.21.15.16:0/3832889997 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbc0815c170 0x7fbc0814d730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:29.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:29.092+0000 7fbbfe000640 1 -- 172.21.15.16:0/3832889997 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbc0815b970 msgr2=0x7fbc0814ac60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:29.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:29.092+0000 7fbbfe000640 1 --2- 172.21.15.16:0/3832889997 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbc0815b970 0x7fbc0814ac60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:29.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:29.092+0000 7fbbfe000640 1 -- 172.21.15.16:0/3832889997 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc0814c3f0 con 0x7fbc0815bd50 2024-03-31T15:04:29.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:29.092+0000 7fbc06a00640 1 --2- 172.21.15.16:0/3832889997 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbc0815b970 0x7fbc0814ac60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:04:29.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:29.092+0000 7fbbfe000640 1 --2- 172.21.15.16:0/3832889997 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fbc0815bd50 0x7fbc08155fa0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fbbf0019030 tx=0x7fbbf001ac60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:29.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:29.092+0000 7fbbffe00640 1 -- 172.21.15.16:0/3832889997 <== mon.2 v2:172.21.15.89:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbf00b0050 con 0x7fbc0815bd50 2024-03-31T15:04:29.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:29.092+0000 7fbbffe00640 1 -- 172.21.15.16:0/3832889997 <== mon.2 v2:172.21.15.89:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbbf00b1040 con 0x7fbc0815bd50 2024-03-31T15:04:29.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:29.092+0000 7fbbffe00640 1 -- 172.21.15.16:0/3832889997 <== mon.2 v2:172.21.15.89:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbf00abe10 con 0x7fbc0815bd50 2024-03-31T15:04:29.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:29.092+0000 7fbc07e00640 1 -- 172.21.15.16:0/3832889997 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbc081564e0 con 0x7fbc0815bd50 2024-03-31T15:04:29.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:29.092+0000 7fbc07e00640 1 -- 172.21.15.16:0/3832889997 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fbc0814c970 con 0x7fbc0815bd50 2024-03-31T15:04:29.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:29.096+0000 7fbc07e00640 1 -- 172.21.15.16:0/3832889997 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbc0804f8c0 con 0x7fbc0815bd50 2024-03-31T15:04:29.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:29.096+0000 7fbbffe00640 1 -- 172.21.15.16:0/3832889997 <== mon.2 v2:172.21.15.89:3301/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fbbf00be030 con 0x7fbc0815bd50 2024-03-31T15:04:29.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:29.096+0000 7fbbffe00640 1 --2- 172.21.15.16:0/3832889997 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fbbe4068380 0x7fbbe406a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:29.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:29.096+0000 7fbbffe00640 1 -- 172.21.15.16:0/3832889997 <== mon.2 v2:172.21.15.89:3301/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7fbbf0120950 con 0x7fbc0815bd50 2024-03-31T15:04:29.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:29.096+0000 7fbc06a00640 1 --2- 172.21.15.16:0/3832889997 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fbbe4068380 0x7fbbe406a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:29.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:29.096+0000 7fbc06a00640 1 --2- 172.21.15.16:0/3832889997 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fbbe4068380 0x7fbbe406a840 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fbbf8007a90 tx=0x7fbbf80079e0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:29.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:29.104+0000 7fbbffe00640 1 -- 172.21.15.16:0/3832889997 <== mon.2 v2:172.21.15.89:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fbbf00c5060 con 0x7fbc0815bd50 2024-03-31T15:04:29.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:29.304+0000 7fbc07e00640 1 -- 172.21.15.16:0/3832889997 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7fbc0815daa0 con 0x7fbc0815bd50 2024-03-31T15:04:29.301 INFO:tasks.ceph.mon.a.smithi016.stderr:2024-03-31T15:04:29.308+0000 7f7dcbc00640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-31T15:04:29.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:29.316+0000 7fbbffe00640 1 -- 172.21.15.16:0/3832889997 <== mon.2 v2:172.21.15.89:3301/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v9) ==== 161+0+0 (secure 0 0 0) 0x7fbbf00160d0 con 0x7fbc0815bd50 2024-03-31T15:04:29.309 INFO:teuthology.orchestra.run.smithi016.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-31T15:04:29.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:29.316+0000 7fbc07e00640 1 -- 172.21.15.16:0/3832889997 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fbbe4068380 msgr2=0x7fbbe406a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:29.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:29.316+0000 7fbc07e00640 1 --2- 172.21.15.16:0/3832889997 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fbbe4068380 0x7fbbe406a840 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fbbf8007a90 tx=0x7fbbf80079e0 comp rx=0 tx=0).stop 2024-03-31T15:04:29.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:29.316+0000 7fbc07e00640 1 -- 172.21.15.16:0/3832889997 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fbc0815bd50 msgr2=0x7fbc08155fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:29.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:29.316+0000 7fbc07e00640 1 --2- 172.21.15.16:0/3832889997 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fbc0815bd50 0x7fbc08155fa0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fbbf0019030 tx=0x7fbbf001ac60 comp rx=0 tx=0).stop 2024-03-31T15:04:29.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:29.316+0000 7fbc07e00640 1 -- 172.21.15.16:0/3832889997 shutdown_connections 2024-03-31T15:04:29.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:29.316+0000 7fbc07e00640 1 --2- 172.21.15.16:0/3832889997 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fbbe4068380 0x7fbbe406a840 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:29.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:29.316+0000 7fbc07e00640 1 --2- 172.21.15.16:0/3832889997 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbc0815c170 0x7fbc0814d730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:29.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:29.316+0000 7fbc07e00640 1 --2- 172.21.15.16:0/3832889997 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fbc0815bd50 0x7fbc08155fa0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:29.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:29.316+0000 7fbc07e00640 1 --2- 172.21.15.16:0/3832889997 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbc0815b970 0x7fbc0814ac60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:29.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:29.316+0000 7fbc07e00640 1 -- 172.21.15.16:0/3832889997 >> 172.21.15.16:0/3832889997 conn(0x7fbc080b5820 msgr2=0x7fbc08159f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:04:29.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:29.316+0000 7fbc07e00640 1 -- 172.21.15.16:0/3832889997 shutdown_connections 2024-03-31T15:04:29.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:29.316+0000 7fbc07e00640 1 -- 172.21.15.16:0/3832889997 wait complete. 2024-03-31T15:04:29.313 INFO:tasks.ceph.mds.b.smithi089.stderr: -3337> 2024-03-31T15:03:40.749+0000 7f0735c00640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-31T15:04:29.333 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-31T15:04:29.362 INFO:tasks.ceph.mds.b.smithi089.stdout:starting mds.b at 2024-03-31T15:04:29.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:29.480+0000 7fc364600640 1 Processor -- start 2024-03-31T15:04:29.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:29.480+0000 7fc364600640 1 -- start start 2024-03-31T15:04:29.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:29.484+0000 7fc364600640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fc35c06d4d0 0x7fc35c06d930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:29.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:29.484+0000 7fc364600640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fc35c071550 0x7fc35c0719a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:29.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:29.484+0000 7fc364600640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc35c06de70 con 0x7fc35c1567e0 2024-03-31T15:04:29.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:29.484+0000 7fc364600640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7fc35c15c7e0 con 0x7fc35c071550 2024-03-31T15:04:29.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:29.484+0000 7fc364600640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fc35c0736b0 con 0x7fc35c06d4d0 2024-03-31T15:04:29.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:29.484+0000 7fc361c00640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fc35c1567e0 0x7fc35c072c80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:33690/0 (socket says 172.21.15.16:33690) 2024-03-31T15:04:29.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:29.484+0000 7fc361c00640 1 -- 172.21.15.16:0/1837958853 learned_addr learned my addr 172.21.15.16:0/1837958853 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:04:29.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:29.484+0000 7fc362600640 1 --2- 172.21.15.16:0/1837958853 >> v2:172.21.15.16:3300/0 conn(0x7fc35c071550 0x7fc35c0719a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:29.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:29.484+0000 7fc361200640 1 --2- 172.21.15.16:0/1837958853 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fc35c06d4d0 0x7fc35c06d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:29.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:29.484+0000 7fc35be00640 1 -- 172.21.15.16:0/1837958853 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1818452123 0 0) 0x7fc35c06de70 con 0x7fc35c1567e0 2024-03-31T15:04:29.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:29.484+0000 7fc35be00640 1 -- 172.21.15.16:0/1837958853 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc344003530 con 0x7fc35c1567e0 2024-03-31T15:04:29.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:29.484+0000 7fc361200640 1 -- 172.21.15.16:0/1837958853 >> v2:172.21.15.16:3300/0 conn(0x7fc35c071550 msgr2=0x7fc35c0719a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:04:29.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:29.484+0000 7fc361200640 1 --2- 172.21.15.16:0/1837958853 >> v2:172.21.15.16:3300/0 conn(0x7fc35c071550 0x7fc35c0719a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:29.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:29.484+0000 7fc361200640 1 -- 172.21.15.16:0/1837958853 >> v1:172.21.15.16:6789/0 conn(0x7fc35c1567e0 legacy=0x7fc35c072c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:29.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:29.484+0000 7fc361200640 1 -- 172.21.15.16:0/1837958853 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc35c15caa0 con 0x7fc35c06d4d0 2024-03-31T15:04:29.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:29.484+0000 7fc361200640 1 --2- 172.21.15.16:0/1837958853 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fc35c06d4d0 0x7fc35c06d930 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fc3480097c0 tx=0x7fc34801cf00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b737ab1e72aadb71 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:29.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:29.488+0000 7fc35be00640 1 -- 172.21.15.16:0/1837958853 <== mon.2 v2:172.21.15.89:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3480b1070 con 0x7fc35c06d4d0 2024-03-31T15:04:29.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:29.488+0000 7fc35be00640 1 -- 172.21.15.16:0/1837958853 <== mon.2 v2:172.21.15.89:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc3480ac410 con 0x7fc35c06d4d0 2024-03-31T15:04:29.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:29.488+0000 7fc35be00640 1 -- 172.21.15.16:0/1837958853 <== mon.2 v2:172.21.15.89:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3480ac7c0 con 0x7fc35c06d4d0 2024-03-31T15:04:29.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:29.488+0000 7fc364600640 1 -- 172.21.15.16:0/1837958853 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fc35c06d4d0 msgr2=0x7fc35c06d930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:29.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:29.488+0000 7fc364600640 1 --2- 172.21.15.16:0/1837958853 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fc35c06d4d0 0x7fc35c06d930 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fc3480097c0 tx=0x7fc34801cf00 comp rx=0 tx=0).stop 2024-03-31T15:04:29.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:29.488+0000 7fc364600640 1 -- 172.21.15.16:0/1837958853 shutdown_connections 2024-03-31T15:04:29.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:29.488+0000 7fc364600640 1 --2- 172.21.15.16:0/1837958853 >> v2:172.21.15.16:3300/0 conn(0x7fc35c071550 0x7fc35c0719a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:29.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:29.488+0000 7fc364600640 1 --2- 172.21.15.16:0/1837958853 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fc35c06d4d0 0x7fc35c06d930 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:29.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:29.488+0000 7fc364600640 1 -- 172.21.15.16:0/1837958853 >> 172.21.15.16:0/1837958853 conn(0x7fc35c06ee40 msgr2=0x7fc35c06b620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:04:29.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:29.488+0000 7fc364600640 1 -- 172.21.15.16:0/1837958853 shutdown_connections 2024-03-31T15:04:29.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:29.488+0000 7fc364600640 1 -- 172.21.15.16:0/1837958853 wait complete. 2024-03-31T15:04:29.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:29.488+0000 7fc364600640 1 Processor -- start 2024-03-31T15:04:29.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:29.488+0000 7fc364600640 1 -- start start 2024-03-31T15:04:29.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:29.488+0000 7fc364600640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fc35c06d4d0 0x7fc35c14e680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:29.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:29.488+0000 7fc364600640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc35c071550 0x7fc35c15de70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:29.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:29.488+0000 7fc364600640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc35c1567e0 0x7fc35c148b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:29.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:29.488+0000 7fc364600640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fc35c0afbb0 con 0x7fc35c071550 2024-03-31T15:04:29.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:29.488+0000 7fc364600640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fc35c0afa30 con 0x7fc35c1567e0 2024-03-31T15:04:29.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:29.488+0000 7fc364600640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fc35c0722f0 con 0x7fc35c06d4d0 2024-03-31T15:04:29.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:29.488+0000 7fc361200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc35c071550 0x7fc35c15de70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:29.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:29.488+0000 7fc361200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc35c071550 0x7fc35c15de70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43822/0 (socket says 172.21.15.16:43822) 2024-03-31T15:04:29.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:29.488+0000 7fc361200640 1 -- 172.21.15.16:0/3714308372 learned_addr learned my addr 172.21.15.16:0/3714308372 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:04:29.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:29.488+0000 7fc362600640 1 --2- 172.21.15.16:0/3714308372 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc35c1567e0 0x7fc35c148b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:29.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:29.488+0000 7fc361c00640 1 --2- 172.21.15.16:0/3714308372 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fc35c06d4d0 0x7fc35c14e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:29.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:29.488+0000 7fc361200640 1 -- 172.21.15.16:0/3714308372 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fc35c06d4d0 msgr2=0x7fc35c14e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:29.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:29.488+0000 7fc361200640 1 --2- 172.21.15.16:0/3714308372 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fc35c06d4d0 0x7fc35c14e680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:29.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:29.488+0000 7fc361200640 1 -- 172.21.15.16:0/3714308372 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc35c1567e0 msgr2=0x7fc35c148b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:29.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:29.488+0000 7fc361200640 1 --2- 172.21.15.16:0/3714308372 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc35c1567e0 0x7fc35c148b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:29.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:29.492+0000 7fc361200640 1 -- 172.21.15.16:0/3714308372 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc35c06e280 con 0x7fc35c071550 2024-03-31T15:04:29.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:29.492+0000 7fc361200640 1 --2- 172.21.15.16:0/3714308372 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc35c071550 0x7fc35c15de70 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fc34801a0d0 tx=0x7fc348019240 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:29.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:29.492+0000 7fc359600640 1 -- 172.21.15.16:0/3714308372 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3480b9020 con 0x7fc35c071550 2024-03-31T15:04:29.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:29.492+0000 7fc359600640 1 -- 172.21.15.16:0/3714308372 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc3480acca0 con 0x7fc35c071550 2024-03-31T15:04:29.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:29.492+0000 7fc364600640 1 -- 172.21.15.16:0/3714308372 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc35c15e3b0 con 0x7fc35c071550 2024-03-31T15:04:29.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:29.492+0000 7fc359600640 1 -- 172.21.15.16:0/3714308372 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3480b1040 con 0x7fc35c071550 2024-03-31T15:04:29.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:29.492+0000 7fc364600640 1 -- 172.21.15.16:0/3714308372 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc35c1490b0 con 0x7fc35c071550 2024-03-31T15:04:29.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:29.492+0000 7fc364600640 1 -- 172.21.15.16:0/3714308372 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc35c069950 con 0x7fc35c071550 2024-03-31T15:04:29.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:29.492+0000 7fc359600640 1 -- 172.21.15.16:0/3714308372 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fc3480bc020 con 0x7fc35c071550 2024-03-31T15:04:29.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:29.492+0000 7fc359600640 1 --2- 172.21.15.16:0/3714308372 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fc33c0683d0 0x7fc33c06a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:29.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:29.492+0000 7fc359600640 1 -- 172.21.15.16:0/3714308372 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7fc348120120 con 0x7fc35c071550 2024-03-31T15:04:29.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:29.492+0000 7fc361c00640 1 --2- 172.21.15.16:0/3714308372 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fc33c0683d0 0x7fc33c06a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:29.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:29.496+0000 7fc361c00640 1 --2- 172.21.15.16:0/3714308372 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fc33c0683d0 0x7fc33c06a890 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fc350007fd0 tx=0x7fc350009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:29.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:29.496+0000 7fc359600640 1 -- 172.21.15.16:0/3714308372 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fc3480c3030 con 0x7fc35c071550 2024-03-31T15:04:29.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:29.696+0000 7fc364600640 1 -- 172.21.15.16:0/3714308372 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7fc35c14f580 con 0x7fc35c071550 2024-03-31T15:04:30.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:30.312+0000 7fc359600640 1 -- 172.21.15.16:0/3714308372 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v10) ==== 104+0+0 (secure 0 0 0) 0x7fc35c15e3b0 con 0x7fc35c071550 2024-03-31T15:04:30.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:30.312+0000 7fc364600640 1 -- 172.21.15.16:0/3714308372 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fc33c0683d0 msgr2=0x7fc33c06a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:30.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:30.312+0000 7fc364600640 1 --2- 172.21.15.16:0/3714308372 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fc33c0683d0 0x7fc33c06a890 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fc350007fd0 tx=0x7fc350009040 comp rx=0 tx=0).stop 2024-03-31T15:04:30.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:30.312+0000 7fc364600640 1 -- 172.21.15.16:0/3714308372 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc35c071550 msgr2=0x7fc35c15de70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:30.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:30.312+0000 7fc364600640 1 --2- 172.21.15.16:0/3714308372 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc35c071550 0x7fc35c15de70 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fc34801a0d0 tx=0x7fc348019240 comp rx=0 tx=0).stop 2024-03-31T15:04:30.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:30.312+0000 7fc364600640 1 -- 172.21.15.16:0/3714308372 shutdown_connections 2024-03-31T15:04:30.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:30.312+0000 7fc364600640 1 --2- 172.21.15.16:0/3714308372 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fc33c0683d0 0x7fc33c06a890 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:30.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:30.312+0000 7fc364600640 1 --2- 172.21.15.16:0/3714308372 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc35c1567e0 0x7fc35c148b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:30.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:30.312+0000 7fc364600640 1 --2- 172.21.15.16:0/3714308372 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc35c071550 0x7fc35c15de70 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:30.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:30.312+0000 7fc364600640 1 --2- 172.21.15.16:0/3714308372 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fc35c06d4d0 0x7fc35c14e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:30.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:30.312+0000 7fc364600640 1 -- 172.21.15.16:0/3714308372 >> 172.21.15.16:0/3714308372 conn(0x7fc35c06ee40 msgr2=0x7fc35c06b620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:04:30.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:30.312+0000 7fc364600640 1 -- 172.21.15.16:0/3714308372 shutdown_connections 2024-03-31T15:04:30.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:30.312+0000 7fc364600640 1 -- 172.21.15.16:0/3714308372 wait complete. 2024-03-31T15:04:30.332 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-31T15:04:30.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:30.488+0000 7f7e0d800640 1 Processor -- start 2024-03-31T15:04:30.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:30.488+0000 7f7e0d800640 1 -- start start 2024-03-31T15:04:30.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:30.488+0000 7f7e0d800640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f7e080d2b40 0x7f7e080d0270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:30.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:30.488+0000 7f7e0d800640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7e080cd750 con 0x7f7e0815c390 2024-03-31T15:04:30.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:30.488+0000 7f7e0d800640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7e080cda30 con 0x7f7e0806fbc0 2024-03-31T15:04:30.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:30.488+0000 7f7e0d800640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f7e0815c770 con 0x7f7e080d2b40 2024-03-31T15:04:30.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:30.488+0000 7f7e07e00640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f7e0815c390 0x7f7e080cfb60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:33702/0 (socket says 172.21.15.16:33702) 2024-03-31T15:04:30.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:30.488+0000 7f7e07e00640 1 -- 172.21.15.16:0/720751416 learned_addr learned my addr 172.21.15.16:0/720751416 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:04:30.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:30.488+0000 7f7e07400640 1 --2- 172.21.15.16:0/720751416 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f7e080d2b40 0x7f7e080d0270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:30.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:30.488+0000 7f7e06a00640 1 -- 172.21.15.16:0/720751416 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2786008090 0 0) 0x7f7e080cd750 con 0x7f7e0815c390 2024-03-31T15:04:30.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:30.488+0000 7f7e06a00640 1 -- 172.21.15.16:0/720751416 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7dec003530 con 0x7f7e0815c390 2024-03-31T15:04:30.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:30.488+0000 7f7e06a00640 1 -- 172.21.15.16:0/720751416 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 129146314 0 0) 0x7f7dec003530 con 0x7f7e0815c390 2024-03-31T15:04:30.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:30.488+0000 7f7e06a00640 1 -- 172.21.15.16:0/720751416 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f7e080d2b40 msgr2=0x7f7e080d0270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:04:30.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:30.488+0000 7f7e06a00640 1 --2- 172.21.15.16:0/720751416 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f7e080d2b40 0x7f7e080d0270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:30.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:30.488+0000 7f7e06a00640 1 -- 172.21.15.16:0/720751416 >> v1:172.21.15.89:6789/0 conn(0x7f7e0806fbc0 legacy=0x7f7e08070050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:30.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:30.488+0000 7f7e06a00640 1 -- 172.21.15.16:0/720751416 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e080cdd10 con 0x7f7e0815c390 2024-03-31T15:04:30.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:30.488+0000 7f7e06a00640 1 -- 172.21.15.16:0/720751416 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 214807623 0 0) 0x7f7df8001680 con 0x7f7e0815c390 2024-03-31T15:04:30.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:30.488+0000 7f7e06a00640 1 -- 172.21.15.16:0/720751416 >> v1:172.21.15.16:6789/0 conn(0x7f7e0815c390 legacy=0x7f7e080cfb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:30.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:30.488+0000 7f7e06a00640 1 --2- 172.21.15.16:0/720751416 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f7e080d2b40 0x7f7dec0039f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:30.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:30.488+0000 7f7e06a00640 1 --2- 172.21.15.16:0/720751416 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7dec003fc0 0x7f7dec004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:30.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:30.488+0000 7f7e06a00640 1 --2- 172.21.15.16:0/720751416 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7dec004d50 0x7f7dec0051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:30.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:30.488+0000 7f7e06a00640 1 -- 172.21.15.16:0/720751416 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f7dec00bae0 con 0x7f7dec004d50 2024-03-31T15:04:30.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:30.488+0000 7f7e06a00640 1 -- 172.21.15.16:0/720751416 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f7dec00bc90 con 0x7f7dec003fc0 2024-03-31T15:04:30.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:30.488+0000 7f7e06a00640 1 -- 172.21.15.16:0/720751416 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f7dec00be90 con 0x7f7e080d2b40 2024-03-31T15:04:30.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:30.488+0000 7f7e0ce00640 1 --2- 172.21.15.16:0/720751416 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7dec004d50 0x7f7dec0051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:30.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:30.488+0000 7f7e07400640 1 --2- 172.21.15.16:0/720751416 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7dec003fc0 0x7f7dec004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:30.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:30.488+0000 7f7e07e00640 1 --2- 172.21.15.16:0/720751416 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f7e080d2b40 0x7f7dec0039f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:30.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:30.488+0000 7f7e0ce00640 1 -- 172.21.15.16:0/720751416 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f7e080d2b40 msgr2=0x7f7dec0039f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:04:30.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:30.488+0000 7f7e0ce00640 1 --2- 172.21.15.16:0/720751416 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f7e080d2b40 0x7f7dec0039f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:30.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:30.488+0000 7f7e0ce00640 1 -- 172.21.15.16:0/720751416 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7dec003fc0 msgr2=0x7f7dec004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:04:30.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:30.488+0000 7f7e0ce00640 1 --2- 172.21.15.16:0/720751416 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7dec003fc0 0x7f7dec004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:30.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:30.488+0000 7f7e0ce00640 1 -- 172.21.15.16:0/720751416 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7dec003530 con 0x7f7dec004d50 2024-03-31T15:04:30.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:30.492+0000 7f7e0ce00640 1 --2- 172.21.15.16:0/720751416 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7dec004d50 0x7f7dec0051d0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f7e00001e70 tx=0x7f7e0001c780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=787977a07b5579b7 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:30.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:30.492+0000 7f7e06a00640 1 -- 172.21.15.16:0/720751416 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e00001a90 con 0x7f7dec004d50 2024-03-31T15:04:30.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:30.492+0000 7f7e06a00640 1 -- 172.21.15.16:0/720751416 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7e00003b80 con 0x7f7dec004d50 2024-03-31T15:04:30.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:30.492+0000 7f7e06a00640 1 -- 172.21.15.16:0/720751416 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e00005080 con 0x7f7dec004d50 2024-03-31T15:04:30.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:30.492+0000 7f7e0d800640 1 -- 172.21.15.16:0/720751416 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7dec004d50 msgr2=0x7f7dec0051d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:30.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:30.492+0000 7f7e0d800640 1 --2- 172.21.15.16:0/720751416 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7dec004d50 0x7f7dec0051d0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f7e00001e70 tx=0x7f7e0001c780 comp rx=0 tx=0).stop 2024-03-31T15:04:30.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:30.492+0000 7f7e0ce00640 1 -- 172.21.15.16:0/720751416 reap_dead start 2024-03-31T15:04:30.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:30.492+0000 7f7e0d800640 1 -- 172.21.15.16:0/720751416 shutdown_connections 2024-03-31T15:04:30.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:30.492+0000 7f7e0d800640 1 -- 172.21.15.16:0/720751416 >> 172.21.15.16:0/720751416 conn(0x7f7e080d1f90 msgr2=0x7f7e080d23a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:04:30.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:30.492+0000 7f7e0d800640 1 -- 172.21.15.16:0/720751416 shutdown_connections 2024-03-31T15:04:30.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:30.492+0000 7f7e0d800640 1 -- 172.21.15.16:0/720751416 wait complete. 2024-03-31T15:04:30.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:30.496+0000 7f7e0d800640 1 Processor -- start 2024-03-31T15:04:30.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:30.496+0000 7f7e0d800640 1 -- start start 2024-03-31T15:04:30.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:30.496+0000 7f7e0d800640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f7e0814af60 0x7f7e0806c730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:30.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:30.496+0000 7f7e0d800640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7e080b3640 0x7f7e080b3ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:30.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:30.496+0000 7f7e0d800640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7e080b4000 0x7f7e0814b720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:30.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:30.496+0000 7f7e0d800640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f7e080d0270 con 0x7f7e080b3640 2024-03-31T15:04:30.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:30.496+0000 7f7e0d800640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f7e080cf870 con 0x7f7e080b4000 2024-03-31T15:04:30.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:30.496+0000 7f7e0d800640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f7e080d03f0 con 0x7f7e0814af60 2024-03-31T15:04:30.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:30.496+0000 7f7e07400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7e080b3640 0x7f7e080b3ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:30.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:30.496+0000 7f7e07400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7e080b3640 0x7f7e080b3ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43836/0 (socket says 172.21.15.16:43836) 2024-03-31T15:04:30.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:30.496+0000 7f7e07400640 1 -- 172.21.15.16:0/653882622 learned_addr learned my addr 172.21.15.16:0/653882622 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:04:30.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:30.496+0000 7f7e0ce00640 1 --2- 172.21.15.16:0/653882622 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7e080b4000 0x7f7e0814b720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:30.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:30.496+0000 7f7e07e00640 1 --2- 172.21.15.16:0/653882622 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f7e0814af60 0x7f7e0806c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:30.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:30.496+0000 7f7e07400640 1 -- 172.21.15.16:0/653882622 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f7e0814af60 msgr2=0x7f7e0806c730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:30.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:30.496+0000 7f7e07400640 1 --2- 172.21.15.16:0/653882622 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f7e0814af60 0x7f7e0806c730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:30.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:30.496+0000 7f7e07400640 1 -- 172.21.15.16:0/653882622 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7e080b4000 msgr2=0x7f7e0814b720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:30.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:30.496+0000 7f7e07400640 1 --2- 172.21.15.16:0/653882622 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7e080b4000 0x7f7e0814b720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:30.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:30.496+0000 7f7e07400640 1 -- 172.21.15.16:0/653882622 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e0806cdd0 con 0x7f7e080b3640 2024-03-31T15:04:30.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:30.496+0000 7f7e07400640 1 --2- 172.21.15.16:0/653882622 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7e080b3640 0x7f7e080b3ac0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f7df4011a00 tx=0x7f7df4011ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:30.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:30.496+0000 7f7dffe00640 1 -- 172.21.15.16:0/653882622 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7df400fb50 con 0x7f7e080b3640 2024-03-31T15:04:30.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:30.496+0000 7f7dffe00640 1 -- 172.21.15.16:0/653882622 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7df400fcf0 con 0x7f7e080b3640 2024-03-31T15:04:30.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:30.496+0000 7f7dffe00640 1 -- 172.21.15.16:0/653882622 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7df400fea0 con 0x7f7e080b3640 2024-03-31T15:04:30.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:30.496+0000 7f7e0d800640 1 -- 172.21.15.16:0/653882622 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7e0806d0b0 con 0x7f7e080b3640 2024-03-31T15:04:30.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:30.496+0000 7f7e0d800640 1 -- 172.21.15.16:0/653882622 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7e0806d670 con 0x7f7e080b3640 2024-03-31T15:04:30.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:30.500+0000 7f7e0d800640 1 -- 172.21.15.16:0/653882622 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7e0804f8c0 con 0x7f7e080b3640 2024-03-31T15:04:30.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:30.500+0000 7f7dffe00640 1 -- 172.21.15.16:0/653882622 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f7df4012040 con 0x7f7e080b3640 2024-03-31T15:04:30.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:30.500+0000 7f7dffe00640 1 --2- 172.21.15.16:0/653882622 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f7de8068380 0x7f7de806a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:30.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:30.500+0000 7f7dffe00640 1 -- 172.21.15.16:0/653882622 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7f7df40871c0 con 0x7f7e080b3640 2024-03-31T15:04:30.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:30.500+0000 7f7e07e00640 1 --2- 172.21.15.16:0/653882622 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f7de8068380 0x7f7de806a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:30.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:30.500+0000 7f7e07e00640 1 --2- 172.21.15.16:0/653882622 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f7de8068380 0x7f7de806a840 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f7df8001ec0 tx=0x7f7df8009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:30.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:30.504+0000 7f7dffe00640 1 -- 172.21.15.16:0/653882622 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f7df4019030 con 0x7f7e080b3640 2024-03-31T15:04:30.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:30.668+0000 7f7e0d800640 1 -- 172.21.15.16:0/653882622 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7f7e080cfbf0 con 0x7f7e080b3640 2024-03-31T15:04:31.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:31.316+0000 7f7dffe00640 1 -- 172.21.15.16:0/653882622 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v25) ==== 181+0+0 (secure 0 0 0) 0x7f7df4057760 con 0x7f7e080b3640 2024-03-31T15:04:31.311 INFO:teuthology.orchestra.run.smithi016.stderr:pool 'cephfs_metadata' removed 2024-03-31T15:04:31.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:31.320+0000 7f7e0d800640 1 -- 172.21.15.16:0/653882622 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f7de8068380 msgr2=0x7f7de806a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:31.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:31.320+0000 7f7e0d800640 1 --2- 172.21.15.16:0/653882622 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f7de8068380 0x7f7de806a840 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f7df8001ec0 tx=0x7f7df8009040 comp rx=0 tx=0).stop 2024-03-31T15:04:31.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:31.320+0000 7f7e0d800640 1 -- 172.21.15.16:0/653882622 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7e080b3640 msgr2=0x7f7e080b3ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:31.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:31.320+0000 7f7e0d800640 1 --2- 172.21.15.16:0/653882622 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7e080b3640 0x7f7e080b3ac0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f7df4011a00 tx=0x7f7df4011ec0 comp rx=0 tx=0).stop 2024-03-31T15:04:31.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:31.320+0000 7f7e0d800640 1 -- 172.21.15.16:0/653882622 shutdown_connections 2024-03-31T15:04:31.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:31.320+0000 7f7e0d800640 1 --2- 172.21.15.16:0/653882622 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f7de8068380 0x7f7de806a840 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:31.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:31.320+0000 7f7e0d800640 1 --2- 172.21.15.16:0/653882622 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7e080b4000 0x7f7e0814b720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:31.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:31.320+0000 7f7e0d800640 1 --2- 172.21.15.16:0/653882622 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7e080b3640 0x7f7e080b3ac0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:31.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:31.320+0000 7f7e0d800640 1 --2- 172.21.15.16:0/653882622 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f7e0814af60 0x7f7e0806c730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:31.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:31.320+0000 7f7e0d800640 1 -- 172.21.15.16:0/653882622 >> 172.21.15.16:0/653882622 conn(0x7f7e080d1f90 msgr2=0x7f7e080ce7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:04:31.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:31.320+0000 7f7e0d800640 1 -- 172.21.15.16:0/653882622 shutdown_connections 2024-03-31T15:04:31.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:31.320+0000 7f7e0d800640 1 -- 172.21.15.16:0/653882622 wait complete. 2024-03-31T15:04:31.338 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-31T15:04:31.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:31.492+0000 7f0d61600640 1 Processor -- start 2024-03-31T15:04:31.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:31.492+0000 7f0d61600640 1 -- start start 2024-03-31T15:04:31.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:31.492+0000 7f0d61600640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f0d5c0af470 0x7f0d5c0cf1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:31.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:31.492+0000 7f0d61600640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0d5c0cfb00 con 0x7f0d5c15c1b0 2024-03-31T15:04:31.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:31.492+0000 7f0d61600640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0d5c15c8a0 con 0x7f0d5c0cf6f0 2024-03-31T15:04:31.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:31.492+0000 7f0d61600640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f0d5c15d660 con 0x7f0d5c0af470 2024-03-31T15:04:31.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:31.492+0000 7f0d5be00640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f0d5c15c1b0 0x7f0d5c0acd90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:33704/0 (socket says 172.21.15.16:33704) 2024-03-31T15:04:31.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:31.492+0000 7f0d5be00640 1 -- 172.21.15.16:0/2175580805 learned_addr learned my addr 172.21.15.16:0/2175580805 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:04:31.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:31.492+0000 7f0d53400640 1 --2- 172.21.15.16:0/2175580805 >> v2:172.21.15.89:3300/0 conn(0x7f0d5c0af470 0x7f0d5c0cf1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:31.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:31.496+0000 7f0d5b400640 1 -- 172.21.15.16:0/2175580805 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1353597353 0 0) 0x7f0d4c0014a0 con 0x7f0d5c15c1b0 2024-03-31T15:04:31.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:31.496+0000 7f0d5b400640 1 -- 172.21.15.16:0/2175580805 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0d40003530 con 0x7f0d5c15c1b0 2024-03-31T15:04:31.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:31.496+0000 7f0d5b400640 1 -- 172.21.15.16:0/2175580805 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3751211912 0 0) 0x7f0d40003530 con 0x7f0d5c15c1b0 2024-03-31T15:04:31.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:31.496+0000 7f0d5b400640 1 -- 172.21.15.16:0/2175580805 >> v2:172.21.15.89:3300/0 conn(0x7f0d5c0af470 msgr2=0x7f0d5c0cf1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:04:31.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:31.496+0000 7f0d5b400640 1 --2- 172.21.15.16:0/2175580805 >> v2:172.21.15.89:3300/0 conn(0x7f0d5c0af470 0x7f0d5c0cf1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:31.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:31.496+0000 7f0d5b400640 1 -- 172.21.15.16:0/2175580805 >> v1:172.21.15.89:6789/0 conn(0x7f0d5c0cf6f0 legacy=0x7f0d5c0b32f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:31.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:31.496+0000 7f0d5b400640 1 -- 172.21.15.16:0/2175580805 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d5c15cb80 con 0x7f0d5c15c1b0 2024-03-31T15:04:31.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:31.496+0000 7f0d5b400640 1 -- 172.21.15.16:0/2175580805 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 214807623 0 0) 0x7f0d4c001bd0 con 0x7f0d5c15c1b0 2024-03-31T15:04:31.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:31.496+0000 7f0d5b400640 1 -- 172.21.15.16:0/2175580805 >> v1:172.21.15.16:6789/0 conn(0x7f0d5c15c1b0 legacy=0x7f0d5c0acd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:31.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:31.496+0000 7f0d5b400640 1 --2- 172.21.15.16:0/2175580805 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0d400039c0 0x7f0d40003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:31.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:31.496+0000 7f0d5b400640 1 --2- 172.21.15.16:0/2175580805 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0d40004810 0x7f0d40004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:31.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:31.496+0000 7f0d5b400640 1 --2- 172.21.15.16:0/2175580805 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f0d400051d0 0x7f0d4000b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:31.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:31.496+0000 7f0d5b400640 1 -- 172.21.15.16:0/2175580805 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f0d5c15d660 con 0x7f0d40004810 2024-03-31T15:04:31.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:31.496+0000 7f0d5b400640 1 -- 172.21.15.16:0/2175580805 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f0d4000c020 con 0x7f0d400039c0 2024-03-31T15:04:31.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:31.496+0000 7f0d5b400640 1 -- 172.21.15.16:0/2175580805 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f0d4000c220 con 0x7f0d400051d0 2024-03-31T15:04:31.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:31.496+0000 7f0d53400640 1 --2- 172.21.15.16:0/2175580805 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0d40004810 0x7f0d40004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:31.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:31.496+0000 7f0d5be00640 1 --2- 172.21.15.16:0/2175580805 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0d400039c0 0x7f0d40003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:31.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:31.496+0000 7f0d60c00640 1 --2- 172.21.15.16:0/2175580805 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f0d400051d0 0x7f0d4000b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:31.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:31.496+0000 7f0d53400640 1 -- 172.21.15.16:0/2175580805 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f0d400051d0 msgr2=0x7f0d4000b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:04:31.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:31.496+0000 7f0d53400640 1 --2- 172.21.15.16:0/2175580805 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f0d400051d0 0x7f0d4000b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:31.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:31.496+0000 7f0d53400640 1 -- 172.21.15.16:0/2175580805 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0d400039c0 msgr2=0x7f0d40003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:04:31.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:31.496+0000 7f0d53400640 1 --2- 172.21.15.16:0/2175580805 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0d400039c0 0x7f0d40003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:31.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:31.496+0000 7f0d53400640 1 -- 172.21.15.16:0/2175580805 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d40003530 con 0x7f0d40004810 2024-03-31T15:04:31.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:31.496+0000 7f0d5be00640 1 --2- 172.21.15.16:0/2175580805 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0d400039c0 0x7f0d40003da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:04:31.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:31.496+0000 7f0d60c00640 1 -- 172.21.15.16:0/2175580805 reap_dead start 2024-03-31T15:04:31.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:31.500+0000 7f0d53400640 1 --2- 172.21.15.16:0/2175580805 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0d40004810 0x7f0d40004c90 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f0d44009ae0 tx=0x7f0d4401ca70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fdc99027810d425 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:31.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:31.500+0000 7f0d5b400640 1 -- 172.21.15.16:0/2175580805 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d440b2070 con 0x7f0d40004810 2024-03-31T15:04:31.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:31.500+0000 7f0d5b400640 1 -- 172.21.15.16:0/2175580805 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0d440a5660 con 0x7f0d40004810 2024-03-31T15:04:31.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:31.500+0000 7f0d5b400640 1 -- 172.21.15.16:0/2175580805 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d440096b0 con 0x7f0d40004810 2024-03-31T15:04:31.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:31.500+0000 7f0d61600640 1 -- 172.21.15.16:0/2175580805 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0d40004810 msgr2=0x7f0d40004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:31.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:31.500+0000 7f0d61600640 1 --2- 172.21.15.16:0/2175580805 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0d40004810 0x7f0d40004c90 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f0d44009ae0 tx=0x7f0d4401ca70 comp rx=0 tx=0).stop 2024-03-31T15:04:31.494 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:31.500+0000 7f0d61600640 1 -- 172.21.15.16:0/2175580805 shutdown_connections 2024-03-31T15:04:31.494 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:31.500+0000 7f0d61600640 1 --2- 172.21.15.16:0/2175580805 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0d40004810 0x7f0d40004c90 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:31.494 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:31.500+0000 7f0d61600640 1 -- 172.21.15.16:0/2175580805 >> 172.21.15.16:0/2175580805 conn(0x7f0d5c0b2040 msgr2=0x7f0d5c0cfdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:04:31.494 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:31.500+0000 7f0d61600640 1 -- 172.21.15.16:0/2175580805 shutdown_connections 2024-03-31T15:04:31.494 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:31.500+0000 7f0d61600640 1 -- 172.21.15.16:0/2175580805 wait complete. 2024-03-31T15:04:31.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:31.500+0000 7f0d61600640 1 Processor -- start 2024-03-31T15:04:31.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:31.500+0000 7f0d61600640 1 -- start start 2024-03-31T15:04:31.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:31.500+0000 7f0d61600640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0d40004810 0x7f0d5c1e5ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:31.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:31.500+0000 7f0d61600640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0d5c09d280 0x7f0d5c156bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:31.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:31.500+0000 7f0d61600640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f0d5c071940 0x7f0d5c071dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:31.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:31.500+0000 7f0d61600640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f0d5c15d140 con 0x7f0d40004810 2024-03-31T15:04:31.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:31.500+0000 7f0d61600640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f0d5c0ad570 con 0x7f0d5c09d280 2024-03-31T15:04:31.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:31.500+0000 7f0d61600640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f0d5c15c590 con 0x7f0d5c071940 2024-03-31T15:04:31.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:31.500+0000 7f0d5be00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0d40004810 0x7f0d5c1e5ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:31.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:31.504+0000 7f0d5be00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0d40004810 0x7f0d5c1e5ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43846/0 (socket says 172.21.15.16:43846) 2024-03-31T15:04:31.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:31.504+0000 7f0d5be00640 1 -- 172.21.15.16:0/3890588003 learned_addr learned my addr 172.21.15.16:0/3890588003 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:04:31.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:31.504+0000 7f0d60c00640 1 --2- 172.21.15.16:0/3890588003 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f0d5c071940 0x7f0d5c071dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:31.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:31.504+0000 7f0d53400640 1 --2- 172.21.15.16:0/3890588003 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0d5c09d280 0x7f0d5c156bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:31.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:31.504+0000 7f0d5be00640 1 -- 172.21.15.16:0/3890588003 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f0d5c071940 msgr2=0x7f0d5c071dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:31.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:31.504+0000 7f0d5be00640 1 --2- 172.21.15.16:0/3890588003 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f0d5c071940 0x7f0d5c071dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:31.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:31.504+0000 7f0d5be00640 1 -- 172.21.15.16:0/3890588003 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0d5c09d280 msgr2=0x7f0d5c156bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:31.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:31.504+0000 7f0d5be00640 1 --2- 172.21.15.16:0/3890588003 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0d5c09d280 0x7f0d5c156bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:31.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:31.504+0000 7f0d5be00640 1 -- 172.21.15.16:0/3890588003 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d5c1e78e0 con 0x7f0d40004810 2024-03-31T15:04:31.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:31.504+0000 7f0d5be00640 1 --2- 172.21.15.16:0/3890588003 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0d40004810 0x7f0d5c1e5ac0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f0d4c007a50 tx=0x7f0d4c00e530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:31.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:31.504+0000 7f0d58c00640 1 -- 172.21.15.16:0/3890588003 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d4c00ee50 con 0x7f0d40004810 2024-03-31T15:04:31.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:31.504+0000 7f0d58c00640 1 -- 172.21.15.16:0/3890588003 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0d4c013420 con 0x7f0d40004810 2024-03-31T15:04:31.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:31.504+0000 7f0d61600640 1 -- 172.21.15.16:0/3890588003 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0d5c1e7bc0 con 0x7f0d40004810 2024-03-31T15:04:31.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:31.504+0000 7f0d58c00640 1 -- 172.21.15.16:0/3890588003 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d4c0137d0 con 0x7f0d40004810 2024-03-31T15:04:31.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:31.504+0000 7f0d61600640 1 -- 172.21.15.16:0/3890588003 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0d5c1e8180 con 0x7f0d40004810 2024-03-31T15:04:31.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:31.504+0000 7f0d61600640 1 -- 172.21.15.16:0/3890588003 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0d5c04f8c0 con 0x7f0d40004810 2024-03-31T15:04:31.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:31.504+0000 7f0d58c00640 1 -- 172.21.15.16:0/3890588003 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f0d4c013970 con 0x7f0d40004810 2024-03-31T15:04:31.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:31.504+0000 7f0d58c00640 1 --2- 172.21.15.16:0/3890588003 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f0d38068380 0x7f0d3806a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:31.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:31.508+0000 7f0d58c00640 1 -- 172.21.15.16:0/3890588003 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 5473+0+0 (secure 0 0 0) 0x7f0d4c087340 con 0x7f0d40004810 2024-03-31T15:04:31.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:31.508+0000 7f0d53400640 1 --2- 172.21.15.16:0/3890588003 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f0d38068380 0x7f0d3806a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:31.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:31.508+0000 7f0d53400640 1 --2- 172.21.15.16:0/3890588003 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f0d38068380 0x7f0d3806a840 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f0d440af000 tx=0x7f0d440046d0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:31.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:31.512+0000 7f0d58c00640 1 -- 172.21.15.16:0/3890588003 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f0d5c0cfb00 con 0x7f0d40004810 2024-03-31T15:04:31.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:31.684+0000 7f0d61600640 1 -- 172.21.15.16:0/3890588003 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7f0d5c09b5e0 con 0x7f0d40004810 2024-03-31T15:04:32.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:32.320+0000 7f0d58c00640 1 -- 172.21.15.16:0/3890588003 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v26) ==== 169+0+0 (secure 0 0 0) 0x7f0d4c0572f0 con 0x7f0d40004810 2024-03-31T15:04:32.317 INFO:teuthology.orchestra.run.smithi016.stderr:pool 'cephfs_data' removed 2024-03-31T15:04:32.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:32.328+0000 7f0d61600640 1 -- 172.21.15.16:0/3890588003 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f0d38068380 msgr2=0x7f0d3806a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:32.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:32.328+0000 7f0d61600640 1 --2- 172.21.15.16:0/3890588003 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f0d38068380 0x7f0d3806a840 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f0d440af000 tx=0x7f0d440046d0 comp rx=0 tx=0).stop 2024-03-31T15:04:32.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:32.328+0000 7f0d61600640 1 -- 172.21.15.16:0/3890588003 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0d40004810 msgr2=0x7f0d5c1e5ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:32.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:32.328+0000 7f0d61600640 1 --2- 172.21.15.16:0/3890588003 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0d40004810 0x7f0d5c1e5ac0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f0d4c007a50 tx=0x7f0d4c00e530 comp rx=0 tx=0).stop 2024-03-31T15:04:32.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:32.328+0000 7f0d61600640 1 -- 172.21.15.16:0/3890588003 shutdown_connections 2024-03-31T15:04:32.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:32.328+0000 7f0d61600640 1 --2- 172.21.15.16:0/3890588003 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f0d38068380 0x7f0d3806a840 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:32.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:32.328+0000 7f0d61600640 1 --2- 172.21.15.16:0/3890588003 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f0d5c071940 0x7f0d5c071dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:32.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:32.328+0000 7f0d61600640 1 --2- 172.21.15.16:0/3890588003 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0d5c09d280 0x7f0d5c156bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:32.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:32.328+0000 7f0d61600640 1 --2- 172.21.15.16:0/3890588003 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0d40004810 0x7f0d5c1e5ac0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:32.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:32.328+0000 7f0d61600640 1 -- 172.21.15.16:0/3890588003 >> 172.21.15.16:0/3890588003 conn(0x7f0d5c0b2040 msgr2=0x7f0d5c0acd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:04:32.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:32.328+0000 7f0d61600640 1 -- 172.21.15.16:0/3890588003 shutdown_connections 2024-03-31T15:04:32.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:32.328+0000 7f0d61600640 1 -- 172.21.15.16:0/3890588003 wait complete. 2024-03-31T15:04:32.347 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-31T15:04:32.348 INFO:tasks.ceph.mds.a:Stopping old one... 2024-03-31T15:04:32.348 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-31T15:04:32.348 INFO:teuthology.orchestra.run:waiting for 300 2024-03-31T15:04:32.348 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-31T15:04:32.348 INFO:tasks.ceph.mds.c:Stopping old one... 2024-03-31T15:04:32.348 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-31T15:04:32.348 INFO:teuthology.orchestra.run:waiting for 300 2024-03-31T15:04:32.348 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-31T15:04:32.349 INFO:tasks.ceph.mds.b:Stopping old one... 2024-03-31T15:04:32.349 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-31T15:04:32.349 INFO:teuthology.orchestra.run:waiting for 300 2024-03-31T15:04:32.349 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-31T15:04:32.349 INFO:tasks.ceph.mds.d:Stopping old one... 2024-03-31T15:04:32.349 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-31T15:04:32.349 INFO:teuthology.orchestra.run:waiting for 300 2024-03-31T15:04:32.363 INFO:tasks.ceph.mds.b:Stopped 2024-03-31T15:04:32.363 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-31T15:04:32.364 INFO:tasks.ceph.mds.d:Stopped 2024-03-31T15:04:32.364 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-31T15:04:32.365 INFO:tasks.ceph.mds.c:Stopped 2024-03-31T15:04:32.365 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-31T15:04:32.365 INFO:tasks.ceph.mds.a:Stopped 2024-03-31T15:04:32.366 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-31T15:04:32.411 INFO:tasks.ceph.mds.c:Started 2024-03-31T15:04:32.411 DEBUG:teuthology.parallel:result is None 2024-03-31T15:04:32.413 INFO:tasks.ceph.mds.a:Started 2024-03-31T15:04:32.413 DEBUG:teuthology.parallel:result is None 2024-03-31T15:04:32.414 INFO:tasks.ceph.mds.b:Started 2024-03-31T15:04:32.414 DEBUG:teuthology.parallel:result is None 2024-03-31T15:04:32.416 INFO:tasks.ceph.mds.d:Started 2024-03-31T15:04:32.416 DEBUG:teuthology.parallel:result is None 2024-03-31T15:04:32.416 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-03-31T15:04:32.501 INFO:tasks.ceph.mds.a.smithi016.stdout:starting mds.a at 2024-03-31T15:04:32.501 INFO:tasks.ceph.mds.b.smithi089.stdout:starting mds.b at 2024-03-31T15:04:32.501 INFO:tasks.ceph.mds.d.smithi089.stdout:starting mds.d at 2024-03-31T15:04:32.502 INFO:tasks.ceph.mds.c.smithi016.stdout:starting mds.c at 2024-03-31T15:04:32.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:32.572+0000 7fd89be00640 1 Processor -- start 2024-03-31T15:04:32.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:32.572+0000 7fd89be00640 1 -- start start 2024-03-31T15:04:32.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:32.572+0000 7fd89be00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fd894145d10 0x7fd8940cf790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:32.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:32.572+0000 7fd89be00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fd8940b23d0 0x7fd89415cd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:32.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:32.572+0000 7fd89be00640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd894160e70 con 0x7fd8940728b0 2024-03-31T15:04:32.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:32.572+0000 7fd89be00640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7fd8941602f0 con 0x7fd8940b23d0 2024-03-31T15:04:32.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:32.572+0000 7fd89be00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fd8941605d0 con 0x7fd894145d10 2024-03-31T15:04:32.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:32.572+0000 7fd898a00640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fd8940728b0 0x7fd8940b1cc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:33724/0 (socket says 172.21.15.16:33724) 2024-03-31T15:04:32.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:32.572+0000 7fd898a00640 1 -- 172.21.15.16:0/3931737784 learned_addr learned my addr 172.21.15.16:0/3931737784 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:04:32.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:32.572+0000 7fd899e00640 1 --2- 172.21.15.16:0/3931737784 >> v2:172.21.15.89:3300/0 conn(0x7fd8940b23d0 0x7fd89415cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:32.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:32.572+0000 7fd899400640 1 --2- 172.21.15.16:0/3931737784 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fd894145d10 0x7fd8940cf790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:32.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:32.572+0000 7fd893e00640 1 -- 172.21.15.16:0/3931737784 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2894086908 0 0) 0x7fd894160e70 con 0x7fd8940728b0 2024-03-31T15:04:32.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:32.572+0000 7fd893e00640 1 -- 172.21.15.16:0/3931737784 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd87c003530 con 0x7fd8940728b0 2024-03-31T15:04:32.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:32.576+0000 7fd899e00640 1 -- 172.21.15.16:0/3931737784 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fd894145d10 msgr2=0x7fd8940cf790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:04:32.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:32.576+0000 7fd899e00640 1 --2- 172.21.15.16:0/3931737784 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fd894145d10 0x7fd8940cf790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:32.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:32.576+0000 7fd893e00640 1 -- 172.21.15.16:0/3931737784 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 817797172 0 0) 0x7fd87c003530 con 0x7fd8940728b0 2024-03-31T15:04:32.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:32.576+0000 7fd899e00640 1 -- 172.21.15.16:0/3931737784 >> v1:172.21.15.16:6789/0 conn(0x7fd8940728b0 legacy=0x7fd8940b1cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:32.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:32.576+0000 7fd899e00640 1 -- 172.21.15.16:0/3931737784 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8940cfcd0 con 0x7fd8940b23d0 2024-03-31T15:04:32.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:32.576+0000 7fd899e00640 1 --2- 172.21.15.16:0/3931737784 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd8940b23d0 0x7fd89415cd50 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fd88c009d90 tx=0x7fd88c01c890 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5aa48980e5ee25dd server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:32.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:32.576+0000 7fd893e00640 1 -- 172.21.15.16:0/3931737784 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd88c0ac070 con 0x7fd8940b23d0 2024-03-31T15:04:32.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:32.576+0000 7fd893e00640 1 -- 172.21.15.16:0/3931737784 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd88c0a64a0 con 0x7fd8940b23d0 2024-03-31T15:04:32.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:32.576+0000 7fd893e00640 1 -- 172.21.15.16:0/3931737784 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd880001680 con 0x7fd8940b23d0 2024-03-31T15:04:32.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:32.576+0000 7fd89be00640 1 -- 172.21.15.16:0/3931737784 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd8940b23d0 msgr2=0x7fd89415cd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:32.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:32.576+0000 7fd89be00640 1 --2- 172.21.15.16:0/3931737784 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd8940b23d0 0x7fd89415cd50 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fd88c009d90 tx=0x7fd88c01c890 comp rx=0 tx=0).stop 2024-03-31T15:04:32.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:32.576+0000 7fd89be00640 1 -- 172.21.15.16:0/3931737784 shutdown_connections 2024-03-31T15:04:32.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:32.576+0000 7fd89be00640 1 --2- 172.21.15.16:0/3931737784 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd8940b23d0 0x7fd89415cd50 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:32.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:32.576+0000 7fd89be00640 1 --2- 172.21.15.16:0/3931737784 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fd894145d10 0x7fd8940cf790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:32.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:32.576+0000 7fd89be00640 1 -- 172.21.15.16:0/3931737784 >> 172.21.15.16:0/3931737784 conn(0x7fd89406b620 msgr2=0x7fd89406ba30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:04:32.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:32.576+0000 7fd89be00640 1 -- 172.21.15.16:0/3931737784 shutdown_connections 2024-03-31T15:04:32.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:32.580+0000 7fd89be00640 1 -- 172.21.15.16:0/3931737784 wait complete. 2024-03-31T15:04:32.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:32.580+0000 7fd89be00640 1 Processor -- start 2024-03-31T15:04:32.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:32.580+0000 7fd89be00640 1 -- start start 2024-03-31T15:04:32.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:32.580+0000 7fd89be00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd8940728b0 0x7fd894155f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:32.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:32.580+0000 7fd89be00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fd8940b23d0 0x7fd89415e560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:32.577 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:32.580+0000 7fd89be00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd894145d10 0x7fd89413eae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:32.577 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:32.580+0000 7fd89be00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fd8940cf4d0 con 0x7fd8940728b0 2024-03-31T15:04:32.577 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:32.580+0000 7fd89be00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fd8940cf350 con 0x7fd894145d10 2024-03-31T15:04:32.577 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:32.580+0000 7fd89be00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fd894072d10 con 0x7fd8940b23d0 2024-03-31T15:04:32.577 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:32.580+0000 7fd899400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd8940728b0 0x7fd894155f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:32.577 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:32.580+0000 7fd899400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd8940728b0 0x7fd894155f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43878/0 (socket says 172.21.15.16:43878) 2024-03-31T15:04:32.577 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:32.580+0000 7fd899400640 1 -- 172.21.15.16:0/1619304497 learned_addr learned my addr 172.21.15.16:0/1619304497 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:04:32.577 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:32.580+0000 7fd899e00640 1 --2- 172.21.15.16:0/1619304497 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd894145d10 0x7fd89413eae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:32.577 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:32.580+0000 7fd898a00640 1 --2- 172.21.15.16:0/1619304497 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fd8940b23d0 0x7fd89415e560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:32.577 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:32.580+0000 7fd899400640 1 -- 172.21.15.16:0/1619304497 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fd8940b23d0 msgr2=0x7fd89415e560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:32.577 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:32.580+0000 7fd899400640 1 --2- 172.21.15.16:0/1619304497 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fd8940b23d0 0x7fd89415e560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:32.577 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:32.580+0000 7fd899400640 1 -- 172.21.15.16:0/1619304497 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd894145d10 msgr2=0x7fd89413eae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:32.577 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:32.580+0000 7fd899400640 1 --2- 172.21.15.16:0/1619304497 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd894145d10 0x7fd89413eae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:32.577 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:32.580+0000 7fd899400640 1 -- 172.21.15.16:0/1619304497 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8941564b0 con 0x7fd8940728b0 2024-03-31T15:04:32.577 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:32.580+0000 7fd898a00640 1 --2- 172.21.15.16:0/1619304497 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fd8940b23d0 0x7fd89415e560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:04:32.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:32.580+0000 7fd899400640 1 --2- 172.21.15.16:0/1619304497 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd8940728b0 0x7fd894155f70 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fd88800e870 tx=0x7fd88800ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:32.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:32.584+0000 7fd891600640 1 -- 172.21.15.16:0/1619304497 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd888017070 con 0x7fd8940728b0 2024-03-31T15:04:32.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:32.584+0000 7fd89be00640 1 -- 172.21.15.16:0/1619304497 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd89415eed0 con 0x7fd8940728b0 2024-03-31T15:04:32.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:32.584+0000 7fd891600640 1 -- 172.21.15.16:0/1619304497 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd88800c940 con 0x7fd8940728b0 2024-03-31T15:04:32.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:32.584+0000 7fd891600640 1 -- 172.21.15.16:0/1619304497 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd88800cd10 con 0x7fd8940728b0 2024-03-31T15:04:32.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:32.584+0000 7fd89be00640 1 -- 172.21.15.16:0/1619304497 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd8940afcd0 con 0x7fd8940728b0 2024-03-31T15:04:32.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:32.584+0000 7fd89be00640 1 -- 172.21.15.16:0/1619304497 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd894069950 con 0x7fd8940728b0 2024-03-31T15:04:32.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:32.584+0000 7fd891600640 1 -- 172.21.15.16:0/1619304497 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fd88801a070 con 0x7fd8940728b0 2024-03-31T15:04:32.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:32.584+0000 7fd891600640 1 --2- 172.21.15.16:0/1619304497 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fd874068380 0x7fd87406a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:32.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:32.584+0000 7fd891600640 1 -- 172.21.15.16:0/1619304497 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 4943+0+0 (secure 0 0 0) 0x7fd888086590 con 0x7fd8940728b0 2024-03-31T15:04:32.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:32.584+0000 7fd898a00640 1 --2- 172.21.15.16:0/1619304497 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fd874068380 0x7fd87406a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:32.580 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:32.584+0000 7fd898a00640 1 --2- 172.21.15.16:0/1619304497 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fd874068380 0x7fd87406a840 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fd89415a1d0 tx=0x7fd880009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:32.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:32.588+0000 7fd891600640 1 -- 172.21.15.16:0/1619304497 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fd8941564b0 con 0x7fd8940728b0 2024-03-31T15:04:32.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:32.764+0000 7fd89be00640 1 -- 172.21.15.16:0/1619304497 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) -- 0x7fd8940cdb60 con 0x7fd8940728b0 2024-03-31T15:04:33.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:33.336+0000 7fd891600640 1 -- 172.21.15.16:0/1619304497 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v28) ==== 97+0+0 (secure 0 0 0) 0x7fd888056770 con 0x7fd8940728b0 2024-03-31T15:04:33.331 INFO:teuthology.orchestra.run.smithi016.stderr: removed all blocklist entries 2024-03-31T15:04:33.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:33.336+0000 7fd89be00640 1 -- 172.21.15.16:0/1619304497 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fd874068380 msgr2=0x7fd87406a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:33.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:33.336+0000 7fd89be00640 1 --2- 172.21.15.16:0/1619304497 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fd874068380 0x7fd87406a840 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fd89415a1d0 tx=0x7fd880009040 comp rx=0 tx=0).stop 2024-03-31T15:04:33.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:33.340+0000 7fd89be00640 1 -- 172.21.15.16:0/1619304497 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd8940728b0 msgr2=0x7fd894155f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:33.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:33.340+0000 7fd89be00640 1 --2- 172.21.15.16:0/1619304497 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd8940728b0 0x7fd894155f70 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fd88800e870 tx=0x7fd88800ed30 comp rx=0 tx=0).stop 2024-03-31T15:04:33.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:33.340+0000 7fd89be00640 1 -- 172.21.15.16:0/1619304497 shutdown_connections 2024-03-31T15:04:33.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:33.340+0000 7fd89be00640 1 --2- 172.21.15.16:0/1619304497 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fd874068380 0x7fd87406a840 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:33.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:33.340+0000 7fd89be00640 1 --2- 172.21.15.16:0/1619304497 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd894145d10 0x7fd89413eae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:33.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:33.340+0000 7fd89be00640 1 --2- 172.21.15.16:0/1619304497 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fd8940b23d0 0x7fd89415e560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:33.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:33.340+0000 7fd89be00640 1 --2- 172.21.15.16:0/1619304497 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd8940728b0 0x7fd894155f70 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:33.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:33.340+0000 7fd89be00640 1 -- 172.21.15.16:0/1619304497 >> 172.21.15.16:0/1619304497 conn(0x7fd89406b620 msgr2=0x7fd8940cd450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:04:33.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:33.340+0000 7fd89be00640 1 -- 172.21.15.16:0/1619304497 shutdown_connections 2024-03-31T15:04:33.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:33.340+0000 7fd89be00640 1 -- 172.21.15.16:0/1619304497 wait complete. 2024-03-31T15:04:33.357 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-03-31T15:04:33.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:33.504+0000 7f8669e00640 1 Processor -- start 2024-03-31T15:04:33.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:33.504+0000 7f8669e00640 1 -- start start 2024-03-31T15:04:33.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:33.504+0000 7f8669e00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f866406dbc0 0x7f866415f650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:33.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:33.504+0000 7f8669e00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f8664070030 0x7f8664162180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:33.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:33.504+0000 7f8669e00640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f86640cbe90 con 0x7f86640cb440 2024-03-31T15:04:33.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:33.504+0000 7f8669e00640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f866415fb90 con 0x7f8664070030 2024-03-31T15:04:33.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:33.504+0000 7f8669e00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f866415f1c0 con 0x7f866406dbc0 2024-03-31T15:04:33.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:33.504+0000 7f8663400640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f8664070030 0x7f8664162180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:33.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:33.504+0000 7f865be00640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f86640cb440 0x7f866406f920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:33730/0 (socket says 172.21.15.16:33730) 2024-03-31T15:04:33.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:33.504+0000 7f865be00640 1 -- 172.21.15.16:0/994053553 learned_addr learned my addr 172.21.15.16:0/994053553 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:04:33.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:33.504+0000 7f8662a00640 1 --2- 172.21.15.16:0/994053553 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f866406dbc0 0x7f866415f650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:33.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:33.504+0000 7f8662000640 1 -- 172.21.15.16:0/994053553 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 142430487 0 0) 0x7f86640cbe90 con 0x7f86640cb440 2024-03-31T15:04:33.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:33.508+0000 7f8662000640 1 -- 172.21.15.16:0/994053553 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8648003530 con 0x7f86640cb440 2024-03-31T15:04:33.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:33.508+0000 7f8662a00640 1 -- 172.21.15.16:0/994053553 >> v2:172.21.15.16:3300/0 conn(0x7f8664070030 msgr2=0x7f8664162180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:04:33.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:33.508+0000 7f8662a00640 1 --2- 172.21.15.16:0/994053553 >> v2:172.21.15.16:3300/0 conn(0x7f8664070030 0x7f8664162180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:33.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:33.508+0000 7f8662a00640 1 -- 172.21.15.16:0/994053553 >> v1:172.21.15.16:6789/0 conn(0x7f86640cb440 legacy=0x7f866406f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:33.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:33.508+0000 7f8662a00640 1 -- 172.21.15.16:0/994053553 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86640cc170 con 0x7f866406dbc0 2024-03-31T15:04:33.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:33.508+0000 7f8662a00640 1 --2- 172.21.15.16:0/994053553 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f866406dbc0 0x7f866415f650 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f86540097c0 tx=0x7f865401c3e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3f442d9f2d0e92cd server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:33.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:33.508+0000 7f8662000640 1 -- 172.21.15.16:0/994053553 <== mon.2 v2:172.21.15.89:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86540b0070 con 0x7f866406dbc0 2024-03-31T15:04:33.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:33.508+0000 7f8662000640 1 -- 172.21.15.16:0/994053553 <== mon.2 v2:172.21.15.89:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f86540ab600 con 0x7f866406dbc0 2024-03-31T15:04:33.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:33.508+0000 7f8662000640 1 -- 172.21.15.16:0/994053553 <== mon.2 v2:172.21.15.89:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86540ab9d0 con 0x7f866406dbc0 2024-03-31T15:04:33.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:33.508+0000 7f8669e00640 1 -- 172.21.15.16:0/994053553 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f866406dbc0 msgr2=0x7f866415f650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:33.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:33.508+0000 7f8669e00640 1 --2- 172.21.15.16:0/994053553 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f866406dbc0 0x7f866415f650 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f86540097c0 tx=0x7f865401c3e0 comp rx=0 tx=0).stop 2024-03-31T15:04:33.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:33.508+0000 7f8669e00640 1 -- 172.21.15.16:0/994053553 shutdown_connections 2024-03-31T15:04:33.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:33.508+0000 7f8669e00640 1 --2- 172.21.15.16:0/994053553 >> v2:172.21.15.16:3300/0 conn(0x7f8664070030 0x7f8664162180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:33.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:33.508+0000 7f8669e00640 1 --2- 172.21.15.16:0/994053553 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f866406dbc0 0x7f866415f650 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:33.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:33.508+0000 7f8669e00640 1 -- 172.21.15.16:0/994053553 >> 172.21.15.16:0/994053553 conn(0x7f8664067330 msgr2=0x7f8664067740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:04:33.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:33.508+0000 7f8669e00640 1 -- 172.21.15.16:0/994053553 shutdown_connections 2024-03-31T15:04:33.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:33.508+0000 7f8669e00640 1 -- 172.21.15.16:0/994053553 wait complete. 2024-03-31T15:04:33.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:33.512+0000 7f8669e00640 1 Processor -- start 2024-03-31T15:04:33.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:33.512+0000 7f8669e00640 1 -- start start 2024-03-31T15:04:33.508 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:33.512+0000 7f8669e00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f866406dbc0 0x7f86641f6d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:33.508 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:33.512+0000 7f8669e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8664070030 0x7f86641fad80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:33.508 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:33.512+0000 7f8669e00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f86640cb440 0x7f866414da00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:33.508 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:33.512+0000 7f8669e00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f866406cad0 con 0x7f8664070030 2024-03-31T15:04:33.508 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:33.512+0000 7f8669e00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f866406c950 con 0x7f866406dbc0 2024-03-31T15:04:33.508 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:33.512+0000 7f8669e00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f866416b520 con 0x7f86640cb440 2024-03-31T15:04:33.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:33.512+0000 7f865be00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8664070030 0x7f86641fad80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:33.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:33.512+0000 7f865be00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8664070030 0x7f86641fad80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43898/0 (socket says 172.21.15.16:43898) 2024-03-31T15:04:33.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:33.512+0000 7f865be00640 1 -- 172.21.15.16:0/3564931543 learned_addr learned my addr 172.21.15.16:0/3564931543 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:04:33.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:33.512+0000 7f8662a00640 1 --2- 172.21.15.16:0/3564931543 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f866406dbc0 0x7f86641f6d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:33.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:33.512+0000 7f8663400640 1 --2- 172.21.15.16:0/3564931543 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f86640cb440 0x7f866414da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:33.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:33.512+0000 7f865be00640 1 -- 172.21.15.16:0/3564931543 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f86640cb440 msgr2=0x7f866414da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:33.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:33.512+0000 7f865be00640 1 --2- 172.21.15.16:0/3564931543 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f86640cb440 0x7f866414da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:33.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:33.512+0000 7f865be00640 1 -- 172.21.15.16:0/3564931543 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f866406dbc0 msgr2=0x7f86641f6d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:33.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:33.512+0000 7f865be00640 1 --2- 172.21.15.16:0/3564931543 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f866406dbc0 0x7f86641f6d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:33.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:33.512+0000 7f865be00640 1 -- 172.21.15.16:0/3564931543 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86641f7400 con 0x7f8664070030 2024-03-31T15:04:33.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:33.512+0000 7f8662a00640 1 --2- 172.21.15.16:0/3564931543 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f866406dbc0 0x7f86641f6d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:04:33.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:33.512+0000 7f8663400640 1 --2- 172.21.15.16:0/3564931543 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f86640cb440 0x7f866414da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:04:33.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:33.512+0000 7f865be00640 1 --2- 172.21.15.16:0/3564931543 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8664070030 0x7f86641fad80 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f864c00c450 tx=0x7f864c00c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:33.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:33.512+0000 7f865b400640 1 -- 172.21.15.16:0/3564931543 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f864c017070 con 0x7f8664070030 2024-03-31T15:04:33.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:33.512+0000 7f865b400640 1 -- 172.21.15.16:0/3564931543 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f864c005cf0 con 0x7f8664070030 2024-03-31T15:04:33.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:33.512+0000 7f8669e00640 1 -- 172.21.15.16:0/3564931543 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f86641f76e0 con 0x7f8664070030 2024-03-31T15:04:33.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:33.512+0000 7f865b400640 1 -- 172.21.15.16:0/3564931543 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f864c007290 con 0x7f8664070030 2024-03-31T15:04:33.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:33.512+0000 7f8669e00640 1 -- 172.21.15.16:0/3564931543 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f86641f7ca0 con 0x7f8664070030 2024-03-31T15:04:33.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:33.516+0000 7f8669e00640 1 -- 172.21.15.16:0/3564931543 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8664069950 con 0x7f8664070030 2024-03-31T15:04:33.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:33.516+0000 7f865b400640 1 -- 172.21.15.16:0/3564931543 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f864c01a070 con 0x7f8664070030 2024-03-31T15:04:33.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:33.516+0000 7f865b400640 1 --2- 172.21.15.16:0/3564931543 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f8640068380 0x7f864006a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:33.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:33.516+0000 7f865b400640 1 -- 172.21.15.16:0/3564931543 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 4857+0+0 (secure 0 0 0) 0x7f864c085ad0 con 0x7f8664070030 2024-03-31T15:04:33.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:33.516+0000 7f8662a00640 1 --2- 172.21.15.16:0/3564931543 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f8640068380 0x7f864006a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:33.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:33.516+0000 7f8662a00640 1 --2- 172.21.15.16:0/3564931543 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f8640068380 0x7f864006a840 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f86540ad000 tx=0x7f8654008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:33.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:33.520+0000 7f865b400640 1 -- 172.21.15.16:0/3564931543 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f864c056350 con 0x7f8664070030 2024-03-31T15:04:33.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:33.740+0000 7f8669e00640 1 -- 172.21.15.16:0/3564931543 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) -- 0x7f86640b4c10 con 0x7f8664070030 2024-03-31T15:04:33.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:33.740+0000 7f865b400640 1 -- 172.21.15.16:0/3564931543 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v6) ==== 80+0+6645 (secure 0 0 0) 0x7f864c055c50 con 0x7f8664070030 2024-03-31T15:04:33.734 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:04:33.735 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-03-31T15:04:33.735 INFO:teuthology.orchestra.run.smithi016.stdout: "auth_dump": [ 2024-03-31T15:04:33.735 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-31T15:04:33.735 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "mds.a", 2024-03-31T15:04:33.735 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAOewlmhp1ANRAAYS9zo+p4tLU1YK0ySOZE/Q==", 2024-03-31T15:04:33.735 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2024-03-31T15:04:33.735 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2024-03-31T15:04:33.735 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow *", 2024-03-31T15:04:33.735 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow *", 2024-03-31T15:04:33.735 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2024-03-31T15:04:33.735 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-31T15:04:33.735 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-31T15:04:33.735 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-31T15:04:33.735 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "mds.b", 2024-03-31T15:04:33.736 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAPewlmXxupBhAAtpkt93aJOFqhc7GXLIlkBg==", 2024-03-31T15:04:33.736 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2024-03-31T15:04:33.736 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2024-03-31T15:04:33.736 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow *", 2024-03-31T15:04:33.736 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow *", 2024-03-31T15:04:33.736 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2024-03-31T15:04:33.736 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-31T15:04:33.736 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-31T15:04:33.736 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-31T15:04:33.736 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "mds.c", 2024-03-31T15:04:33.736 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAPewlmjaieARAA32teg02vE/Q8RtqLtnArMw==", 2024-03-31T15:04:33.736 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2024-03-31T15:04:33.736 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2024-03-31T15:04:33.736 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow *", 2024-03-31T15:04:33.736 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow *", 2024-03-31T15:04:33.736 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2024-03-31T15:04:33.737 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-31T15:04:33.737 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-31T15:04:33.737 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-31T15:04:33.737 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "mds.d", 2024-03-31T15:04:33.737 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAPewlmo+4PDhAAmJD0siZB3LjWkvdMq1d8aw==", 2024-03-31T15:04:33.737 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2024-03-31T15:04:33.737 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2024-03-31T15:04:33.737 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow *", 2024-03-31T15:04:33.737 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow *", 2024-03-31T15:04:33.737 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2024-03-31T15:04:33.737 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-31T15:04:33.737 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-31T15:04:33.737 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-31T15:04:33.737 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.0", 2024-03-31T15:04:33.737 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAQewlmVP6kLhAAwFr4jwQgz1N3oIUZuJvmzw==", 2024-03-31T15:04:33.738 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2024-03-31T15:04:33.738 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2024-03-31T15:04:33.738 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2024-03-31T15:04:33.738 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2024-03-31T15:04:33.738 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-31T15:04:33.738 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-31T15:04:33.738 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-31T15:04:33.738 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.1", 2024-03-31T15:04:33.738 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQATewlmBh0gNhAAR6LiuFypwA7d3SwKlWuI/Q==", 2024-03-31T15:04:33.738 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2024-03-31T15:04:33.738 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2024-03-31T15:04:33.738 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2024-03-31T15:04:33.738 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2024-03-31T15:04:33.738 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-31T15:04:33.738 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-31T15:04:33.739 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-31T15:04:33.739 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.2", 2024-03-31T15:04:33.739 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAXewlmtQwhARAACbzc7Woo9BoVFPfK3Pns+g==", 2024-03-31T15:04:33.739 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2024-03-31T15:04:33.739 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2024-03-31T15:04:33.739 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2024-03-31T15:04:33.739 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2024-03-31T15:04:33.739 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-31T15:04:33.739 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-31T15:04:33.739 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-31T15:04:33.739 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.3", 2024-03-31T15:04:33.739 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAaewlmmjWABhAA/hwvARNb51nd5vsnYVD/sg==", 2024-03-31T15:04:33.739 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2024-03-31T15:04:33.739 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2024-03-31T15:04:33.740 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2024-03-31T15:04:33.740 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2024-03-31T15:04:33.740 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-31T15:04:33.740 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-31T15:04:33.740 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-31T15:04:33.740 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.4", 2024-03-31T15:04:33.740 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAgewlmby07CxAAChxE/qPuPboIlg3fDxSOeA==", 2024-03-31T15:04:33.740 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2024-03-31T15:04:33.740 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2024-03-31T15:04:33.740 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2024-03-31T15:04:33.740 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2024-03-31T15:04:33.740 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-31T15:04:33.740 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-31T15:04:33.740 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-31T15:04:33.740 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.5", 2024-03-31T15:04:33.741 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAjewlmlD8yExAA9pgP0heBywrgqc39iU8rpQ==", 2024-03-31T15:04:33.741 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2024-03-31T15:04:33.741 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2024-03-31T15:04:33.741 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2024-03-31T15:04:33.741 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2024-03-31T15:04:33.741 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-31T15:04:33.741 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-31T15:04:33.741 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-31T15:04:33.741 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.6", 2024-03-31T15:04:33.741 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAmewlmzAsTGBAALMjVcQ08sIiTDFaBcq8L6A==", 2024-03-31T15:04:33.741 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2024-03-31T15:04:33.741 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2024-03-31T15:04:33.741 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2024-03-31T15:04:33.741 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2024-03-31T15:04:33.741 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-31T15:04:33.741 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-31T15:04:33.742 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-31T15:04:33.742 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.7", 2024-03-31T15:04:33.742 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQApewlmb+CEHRAAXE+iBumDsb2c4zH3C8f6YA==", 2024-03-31T15:04:33.742 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2024-03-31T15:04:33.742 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2024-03-31T15:04:33.742 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2024-03-31T15:04:33.742 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2024-03-31T15:04:33.742 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-31T15:04:33.742 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-31T15:04:33.742 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-31T15:04:33.742 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "client.0", 2024-03-31T15:04:33.742 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAPewlmlMIDEhAAF0Ll1HcrWHmH6xDKq2Qq9A==", 2024-03-31T15:04:33.742 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2024-03-31T15:04:33.743 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2024-03-31T15:04:33.743 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow r", 2024-03-31T15:04:33.743 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow rw", 2024-03-31T15:04:33.743 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow rwx" 2024-03-31T15:04:33.743 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-31T15:04:33.743 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-31T15:04:33.743 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-31T15:04:33.743 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "client.1", 2024-03-31T15:04:33.743 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAPewlmdsvqGxAAaVdwWwri0IE/2sg7psilWw==", 2024-03-31T15:04:33.743 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2024-03-31T15:04:33.743 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2024-03-31T15:04:33.743 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow r", 2024-03-31T15:04:33.743 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow rw", 2024-03-31T15:04:33.743 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow rwx" 2024-03-31T15:04:33.743 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-31T15:04:33.744 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-31T15:04:33.744 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-31T15:04:33.744 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "client.2", 2024-03-31T15:04:33.744 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAPewlmQapPGBAAbwXSlCi+zyQ9118WyAJmrQ==", 2024-03-31T15:04:33.744 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2024-03-31T15:04:33.744 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2024-03-31T15:04:33.744 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow r", 2024-03-31T15:04:33.744 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow rw", 2024-03-31T15:04:33.744 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow rwx" 2024-03-31T15:04:33.744 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-31T15:04:33.744 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-31T15:04:33.744 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-31T15:04:33.744 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "client.3", 2024-03-31T15:04:33.744 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAPewlmCw7qIRAAEeIjF85JbyfJ6d3FtMOVhQ==", 2024-03-31T15:04:33.744 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2024-03-31T15:04:33.744 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2024-03-31T15:04:33.745 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow r", 2024-03-31T15:04:33.745 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow rw", 2024-03-31T15:04:33.745 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow rwx" 2024-03-31T15:04:33.745 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-31T15:04:33.745 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-31T15:04:33.745 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-31T15:04:33.745 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "client.admin", 2024-03-31T15:04:33.745 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAOewlmXyPZGRAAtZOheiYhzGxdeJGqqyGxTw==", 2024-03-31T15:04:33.745 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2024-03-31T15:04:33.745 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow *", 2024-03-31T15:04:33.745 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow *", 2024-03-31T15:04:33.745 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow *", 2024-03-31T15:04:33.745 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2024-03-31T15:04:33.745 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-31T15:04:33.745 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-31T15:04:33.745 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-31T15:04:33.746 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "client.bootstrap-mds", 2024-03-31T15:04:33.746 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAvewlmEsowGhAA4MBPXx7TFBrBaOkxQq+yzQ==", 2024-03-31T15:04:33.746 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2024-03-31T15:04:33.746 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile bootstrap-mds" 2024-03-31T15:04:33.746 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-31T15:04:33.746 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-31T15:04:33.746 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-31T15:04:33.746 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "client.bootstrap-mgr", 2024-03-31T15:04:33.746 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAvewlmu9QwGhAAOtmdEWOehFTK5bQ72kYNUA==", 2024-03-31T15:04:33.746 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2024-03-31T15:04:33.746 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile bootstrap-mgr" 2024-03-31T15:04:33.746 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-31T15:04:33.746 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-31T15:04:33.746 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-31T15:04:33.746 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "client.bootstrap-osd", 2024-03-31T15:04:33.746 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAvewlmV90wGhAAhzVh+SeDgRstqCImeNQFnw==", 2024-03-31T15:04:33.747 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2024-03-31T15:04:33.747 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile bootstrap-osd" 2024-03-31T15:04:33.747 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-31T15:04:33.747 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-31T15:04:33.747 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-31T15:04:33.747 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "client.bootstrap-rbd", 2024-03-31T15:04:33.747 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAvewlmu+UwGhAAK+GHYzx8iwZ/EwWb4zftew==", 2024-03-31T15:04:33.747 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2024-03-31T15:04:33.747 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile bootstrap-rbd" 2024-03-31T15:04:33.747 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-31T15:04:33.747 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-31T15:04:33.747 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-31T15:04:33.747 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "client.bootstrap-rbd-mirror", 2024-03-31T15:04:33.747 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAvewlmNe4wGhAADT0VdJ+sQFG9FTvTCF3NnA==", 2024-03-31T15:04:33.747 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2024-03-31T15:04:33.747 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile bootstrap-rbd-mirror" 2024-03-31T15:04:33.748 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-31T15:04:33.748 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-31T15:04:33.748 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-31T15:04:33.748 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "client.bootstrap-rgw", 2024-03-31T15:04:33.748 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAvewlmE/cwGhAAl1qSxGlrcKBBunCQ/E6cwQ==", 2024-03-31T15:04:33.748 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2024-03-31T15:04:33.748 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile bootstrap-rgw" 2024-03-31T15:04:33.748 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-31T15:04:33.748 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-31T15:04:33.748 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-31T15:04:33.748 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "mgr.x", 2024-03-31T15:04:33.748 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAOewlm7UkKMRAA8hqgeV86mTCkRcxdXNdWmw==", 2024-03-31T15:04:33.748 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2024-03-31T15:04:33.748 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow *", 2024-03-31T15:04:33.748 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile mgr", 2024-03-31T15:04:33.748 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2024-03-31T15:04:33.749 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-31T15:04:33.749 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-31T15:04:33.749 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-31T15:04:33.749 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "mgr.y", 2024-03-31T15:04:33.749 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAOewlmq0cpLRAAKTgh04tRaHrpMhxdwgvXYA==", 2024-03-31T15:04:33.749 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2024-03-31T15:04:33.749 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow *", 2024-03-31T15:04:33.749 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile mgr", 2024-03-31T15:04:33.749 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2024-03-31T15:04:33.749 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-31T15:04:33.749 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-31T15:04:33.749 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-03-31T15:04:33.749 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-03-31T15:04:33.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:33.744+0000 7f8669e00640 1 -- 172.21.15.16:0/3564931543 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f8640068380 msgr2=0x7f864006a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:33.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:33.744+0000 7f8669e00640 1 --2- 172.21.15.16:0/3564931543 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f8640068380 0x7f864006a840 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f86540ad000 tx=0x7f8654008040 comp rx=0 tx=0).stop 2024-03-31T15:04:33.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:33.744+0000 7f8669e00640 1 -- 172.21.15.16:0/3564931543 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8664070030 msgr2=0x7f86641fad80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:33.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:33.744+0000 7f8669e00640 1 --2- 172.21.15.16:0/3564931543 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8664070030 0x7f86641fad80 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f864c00c450 tx=0x7f864c00c910 comp rx=0 tx=0).stop 2024-03-31T15:04:33.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:33.744+0000 7f8669e00640 1 -- 172.21.15.16:0/3564931543 shutdown_connections 2024-03-31T15:04:33.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:33.744+0000 7f8669e00640 1 --2- 172.21.15.16:0/3564931543 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f8640068380 0x7f864006a840 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:33.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:33.744+0000 7f8669e00640 1 --2- 172.21.15.16:0/3564931543 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f86640cb440 0x7f866414da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:33.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:33.744+0000 7f8669e00640 1 --2- 172.21.15.16:0/3564931543 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8664070030 0x7f86641fad80 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:33.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:33.744+0000 7f8669e00640 1 --2- 172.21.15.16:0/3564931543 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f866406dbc0 0x7f86641f6d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:33.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:33.744+0000 7f8669e00640 1 -- 172.21.15.16:0/3564931543 >> 172.21.15.16:0/3564931543 conn(0x7f8664067330 msgr2=0x7f8664067710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:04:33.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:33.744+0000 7f8669e00640 1 -- 172.21.15.16:0/3564931543 shutdown_connections 2024-03-31T15:04:33.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:33.744+0000 7f8669e00640 1 -- 172.21.15.16:0/3564931543 wait complete. 2024-03-31T15:04:33.756 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-mds 2024-03-31T15:04:33.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:33.904+0000 7f9489200640 1 Processor -- start 2024-03-31T15:04:33.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:33.904+0000 7f9489200640 1 -- start start 2024-03-31T15:04:33.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:33.904+0000 7f9489200640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f94840b23d0 0x7f9484071d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:33.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:33.904+0000 7f9489200640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f948415eee0 0x7f9484072250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:33.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:33.904+0000 7f9489200640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f948415f3c0 con 0x7f948415b3d0 2024-03-31T15:04:33.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:33.904+0000 7f9489200640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f948415b7b0 con 0x7f94840b23d0 2024-03-31T15:04:33.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:33.904+0000 7f9489200640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f948415f6a0 con 0x7f948415eee0 2024-03-31T15:04:33.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:33.904+0000 7f947be00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f94840b23d0 0x7f9484071d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:33.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:33.904+0000 7f947be00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f94840b23d0 0x7f9484071d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43910/0 (socket says 172.21.15.16:43910) 2024-03-31T15:04:33.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:33.904+0000 7f947be00640 1 -- 172.21.15.16:0/2594966103 learned_addr learned my addr 172.21.15.16:0/2594966103 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:04:33.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:33.904+0000 7f9482800640 1 --2- 172.21.15.16:0/2594966103 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f948415eee0 0x7f9484072250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:33.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:33.904+0000 7f947be00640 1 -- 172.21.15.16:0/2594966103 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f948415eee0 msgr2=0x7f9484072250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:04:33.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:33.904+0000 7f947be00640 1 --2- 172.21.15.16:0/2594966103 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f948415eee0 0x7f9484072250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:33.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:33.904+0000 7f947be00640 1 -- 172.21.15.16:0/2594966103 >> v1:172.21.15.89:6789/0 conn(0x7f948415b3d0 legacy=0x7f94840b1cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:33.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:33.904+0000 7f947be00640 1 -- 172.21.15.16:0/2594966103 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f948415f980 con 0x7f94840b23d0 2024-03-31T15:04:33.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:33.908+0000 7f947be00640 1 --2- 172.21.15.16:0/2594966103 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f94840b23d0 0x7f9484071d10 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f946c009870 tx=0x7f946c01c530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=765a16e54fc3d07c server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:33.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:33.908+0000 7f9481400640 1 -- 172.21.15.16:0/2594966103 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f946c0aa650 con 0x7f94840b23d0 2024-03-31T15:04:33.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:33.908+0000 7f9481400640 1 -- 172.21.15.16:0/2594966103 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f946c0af070 con 0x7f94840b23d0 2024-03-31T15:04:33.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:33.908+0000 7f9481400640 1 -- 172.21.15.16:0/2594966103 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f946c0aaa10 con 0x7f94840b23d0 2024-03-31T15:04:33.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:33.908+0000 7f9489200640 1 -- 172.21.15.16:0/2594966103 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f94840b23d0 msgr2=0x7f9484071d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:33.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:33.908+0000 7f9489200640 1 --2- 172.21.15.16:0/2594966103 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f94840b23d0 0x7f9484071d10 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f946c009870 tx=0x7f946c01c530 comp rx=0 tx=0).stop 2024-03-31T15:04:33.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:33.908+0000 7f9489200640 1 -- 172.21.15.16:0/2594966103 shutdown_connections 2024-03-31T15:04:33.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:33.908+0000 7f9489200640 1 --2- 172.21.15.16:0/2594966103 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f948415eee0 0x7f9484072250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:33.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:33.908+0000 7f9489200640 1 --2- 172.21.15.16:0/2594966103 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f94840b23d0 0x7f9484071d10 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:33.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:33.908+0000 7f9489200640 1 -- 172.21.15.16:0/2594966103 >> 172.21.15.16:0/2594966103 conn(0x7f948406b620 msgr2=0x7f948406ba30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:04:33.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:33.908+0000 7f9489200640 1 -- 172.21.15.16:0/2594966103 shutdown_connections 2024-03-31T15:04:33.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:33.908+0000 7f9489200640 1 -- 172.21.15.16:0/2594966103 wait complete. 2024-03-31T15:04:33.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:33.912+0000 7f9489200640 1 Processor -- start 2024-03-31T15:04:33.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:33.912+0000 7f9489200640 1 -- start start 2024-03-31T15:04:33.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:33.912+0000 7f9489200640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f94840b23d0 0x7f9484147870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:33.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:33.912+0000 7f9489200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f948415b3d0 0x7f94840cf2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:33.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:33.912+0000 7f9489200640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f948415eee0 0x7f94840cf800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:33.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:33.912+0000 7f9489200640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f9484072790 con 0x7f948415b3d0 2024-03-31T15:04:33.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:33.912+0000 7f9489200640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f94840737d0 con 0x7f948415eee0 2024-03-31T15:04:33.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:33.912+0000 7f9489200640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f9484072910 con 0x7f94840b23d0 2024-03-31T15:04:33.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:33.912+0000 7f947be00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f948415b3d0 0x7f94840cf2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:33.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:33.912+0000 7f947be00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f948415b3d0 0x7f94840cf2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43916/0 (socket says 172.21.15.16:43916) 2024-03-31T15:04:33.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:33.912+0000 7f947be00640 1 -- 172.21.15.16:0/542926014 learned_addr learned my addr 172.21.15.16:0/542926014 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:04:33.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:33.912+0000 7f9482800640 1 --2- 172.21.15.16:0/542926014 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f948415eee0 0x7f94840cf800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:33.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:33.912+0000 7f9481e00640 1 --2- 172.21.15.16:0/542926014 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f94840b23d0 0x7f9484147870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:33.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:33.912+0000 7f947be00640 1 -- 172.21.15.16:0/542926014 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f94840b23d0 msgr2=0x7f9484147870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:33.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:33.912+0000 7f947be00640 1 --2- 172.21.15.16:0/542926014 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f94840b23d0 0x7f9484147870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:33.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:33.912+0000 7f947be00640 1 -- 172.21.15.16:0/542926014 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f948415eee0 msgr2=0x7f94840cf800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:33.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:33.912+0000 7f947be00640 1 --2- 172.21.15.16:0/542926014 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f948415eee0 0x7f94840cf800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:33.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:33.912+0000 7f947be00640 1 -- 172.21.15.16:0/542926014 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f948415d8e0 con 0x7f948415b3d0 2024-03-31T15:04:33.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:33.912+0000 7f947be00640 1 --2- 172.21.15.16:0/542926014 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f948415b3d0 0x7f94840cf2c0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f946c0ac040 tx=0x7f946c01ad70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:33.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:33.912+0000 7f947a000640 1 -- 172.21.15.16:0/542926014 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f946c0bd5f0 con 0x7f948415b3d0 2024-03-31T15:04:33.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:33.912+0000 7f947a000640 1 -- 172.21.15.16:0/542926014 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f946c0a6040 con 0x7f948415b3d0 2024-03-31T15:04:33.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:33.912+0000 7f947a000640 1 -- 172.21.15.16:0/542926014 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f946c0bd9d0 con 0x7f948415b3d0 2024-03-31T15:04:33.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:33.912+0000 7f9489200640 1 -- 172.21.15.16:0/542926014 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f948415dbc0 con 0x7f948415b3d0 2024-03-31T15:04:33.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:33.912+0000 7f9489200640 1 -- 172.21.15.16:0/542926014 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f948415e180 con 0x7f948415b3d0 2024-03-31T15:04:33.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:33.912+0000 7f9489200640 1 -- 172.21.15.16:0/542926014 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9484069950 con 0x7f948415b3d0 2024-03-31T15:04:33.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:33.916+0000 7f947a000640 1 -- 172.21.15.16:0/542926014 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f946c0b33f0 con 0x7f948415b3d0 2024-03-31T15:04:33.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:33.916+0000 7f947a000640 1 --2- 172.21.15.16:0/542926014 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f9460068380 0x7f946006a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:33.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:33.916+0000 7f947a000640 1 -- 172.21.15.16:0/542926014 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 4857+0+0 (secure 0 0 0) 0x7f946c1208a0 con 0x7f948415b3d0 2024-03-31T15:04:33.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:33.916+0000 7f9481e00640 1 --2- 172.21.15.16:0/542926014 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f9460068380 0x7f946006a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:33.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:33.916+0000 7f9481e00640 1 --2- 172.21.15.16:0/542926014 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f9460068380 0x7f946006a840 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f94841550b0 tx=0x7f9474009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:33.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:33.920+0000 7f947a000640 1 -- 172.21.15.16:0/542926014 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f946c016150 con 0x7f948415b3d0 2024-03-31T15:04:34.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:34.144+0000 7f9489200640 1 -- 172.21.15.16:0/542926014 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-mds"} v 0) -- 0x7f9484166050 con 0x7f948415b3d0 2024-03-31T15:04:34.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:34.148+0000 7f947a000640 1 -- 172.21.15.16:0/542926014 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-mds"}]=0 v7) ==== 90+0+0 (secure 0 0 0) 0x7f946c0f0ac0 con 0x7f948415b3d0 2024-03-31T15:04:34.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:34.148+0000 7f9489200640 1 -- 172.21.15.16:0/542926014 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f9460068380 msgr2=0x7f946006a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:34.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:34.148+0000 7f9489200640 1 --2- 172.21.15.16:0/542926014 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f9460068380 0x7f946006a840 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f94841550b0 tx=0x7f9474009040 comp rx=0 tx=0).stop 2024-03-31T15:04:34.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:34.148+0000 7f9489200640 1 -- 172.21.15.16:0/542926014 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f948415b3d0 msgr2=0x7f94840cf2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:34.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:34.148+0000 7f9489200640 1 --2- 172.21.15.16:0/542926014 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f948415b3d0 0x7f94840cf2c0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f946c0ac040 tx=0x7f946c01ad70 comp rx=0 tx=0).stop 2024-03-31T15:04:34.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:34.148+0000 7f9489200640 1 -- 172.21.15.16:0/542926014 shutdown_connections 2024-03-31T15:04:34.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:34.148+0000 7f9489200640 1 --2- 172.21.15.16:0/542926014 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f9460068380 0x7f946006a840 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:34.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:34.148+0000 7f9489200640 1 --2- 172.21.15.16:0/542926014 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f948415eee0 0x7f94840cf800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:34.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:34.148+0000 7f9489200640 1 --2- 172.21.15.16:0/542926014 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f948415b3d0 0x7f94840cf2c0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:34.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:34.148+0000 7f9489200640 1 --2- 172.21.15.16:0/542926014 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f94840b23d0 0x7f9484147870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:34.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:34.148+0000 7f9489200640 1 -- 172.21.15.16:0/542926014 >> 172.21.15.16:0/542926014 conn(0x7f948406b620 msgr2=0x7f9484148280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:04:34.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:34.152+0000 7f9489200640 1 -- 172.21.15.16:0/542926014 shutdown_connections 2024-03-31T15:04:34.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:34.152+0000 7f9489200640 1 -- 172.21.15.16:0/542926014 wait complete. 2024-03-31T15:04:34.163 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-mgr 2024-03-31T15:04:34.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:34.308+0000 7f39ff000640 1 Processor -- start 2024-03-31T15:04:34.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:34.308+0000 7f39ff000640 1 -- start start 2024-03-31T15:04:34.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:34.312+0000 7f39ff000640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f39f815b3d0 0x7f39f80b1cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:34.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:34.312+0000 7f39ff000640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f39f815eee0 0x7f39f80b2200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:34.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:34.312+0000 7f39ff000640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f39f815f3c0 con 0x7f39f80ce2f0 2024-03-31T15:04:34.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:34.312+0000 7f39ff000640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f39f815b7b0 con 0x7f39f815b3d0 2024-03-31T15:04:34.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:34.312+0000 7f39ff000640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f39f815f6a0 con 0x7f39f815eee0 2024-03-31T15:04:34.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:34.312+0000 7f39fd000640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f39f80ce2f0 0x7f39f80cc9c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:33744/0 (socket says 172.21.15.16:33744) 2024-03-31T15:04:34.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:34.312+0000 7f39fd000640 1 -- 172.21.15.16:0/3077887645 learned_addr learned my addr 172.21.15.16:0/3077887645 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:04:34.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:34.312+0000 7f39f7e00640 1 --2- 172.21.15.16:0/3077887645 >> v2:172.21.15.89:3300/0 conn(0x7f39f815b3d0 0x7f39f80b1cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:34.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:34.312+0000 7f39efe00640 1 --2- 172.21.15.16:0/3077887645 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f39f815eee0 0x7f39f80b2200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:34.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:34.312+0000 7f39f7400640 1 -- 172.21.15.16:0/3077887645 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4020818808 0 0) 0x7f39f815f3c0 con 0x7f39f80ce2f0 2024-03-31T15:04:34.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:34.312+0000 7f39f7400640 1 -- 172.21.15.16:0/3077887645 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f39dc003530 con 0x7f39f80ce2f0 2024-03-31T15:04:34.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:34.312+0000 7f39f7e00640 1 -- 172.21.15.16:0/3077887645 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f39f815eee0 msgr2=0x7f39f80b2200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:04:34.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:34.312+0000 7f39f7e00640 1 --2- 172.21.15.16:0/3077887645 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f39f815eee0 0x7f39f80b2200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:34.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:34.312+0000 7f39f7e00640 1 -- 172.21.15.16:0/3077887645 >> v1:172.21.15.16:6789/0 conn(0x7f39f80ce2f0 legacy=0x7f39f80cc9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:34.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:34.312+0000 7f39f7e00640 1 -- 172.21.15.16:0/3077887645 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39f815f980 con 0x7f39f815b3d0 2024-03-31T15:04:34.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:34.312+0000 7f39efe00640 1 --2- 172.21.15.16:0/3077887645 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f39f815eee0 0x7f39f80b2200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:04:34.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:34.316+0000 7f39f7e00640 1 --2- 172.21.15.16:0/3077887645 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f39f815b3d0 0x7f39f80b1cc0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f39e800ba60 tx=0x7f39e801eb40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e3f0cab01b5662d2 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:34.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:34.316+0000 7f39f7400640 1 -- 172.21.15.16:0/3077887645 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39e80b4070 con 0x7f39f815b3d0 2024-03-31T15:04:34.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:34.316+0000 7f39f7400640 1 -- 172.21.15.16:0/3077887645 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f39e80a7bd0 con 0x7f39f815b3d0 2024-03-31T15:04:34.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:34.316+0000 7f39f7400640 1 -- 172.21.15.16:0/3077887645 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39e80af440 con 0x7f39f815b3d0 2024-03-31T15:04:34.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:34.316+0000 7f39ff000640 1 -- 172.21.15.16:0/3077887645 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f39f815b3d0 msgr2=0x7f39f80b1cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:34.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:34.316+0000 7f39ff000640 1 --2- 172.21.15.16:0/3077887645 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f39f815b3d0 0x7f39f80b1cc0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f39e800ba60 tx=0x7f39e801eb40 comp rx=0 tx=0).stop 2024-03-31T15:04:34.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:34.316+0000 7f39ff000640 1 -- 172.21.15.16:0/3077887645 shutdown_connections 2024-03-31T15:04:34.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:34.316+0000 7f39ff000640 1 --2- 172.21.15.16:0/3077887645 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f39f815eee0 0x7f39f80b2200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:34.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:34.316+0000 7f39ff000640 1 --2- 172.21.15.16:0/3077887645 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f39f815b3d0 0x7f39f80b1cc0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:34.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:34.316+0000 7f39ff000640 1 -- 172.21.15.16:0/3077887645 >> 172.21.15.16:0/3077887645 conn(0x7f39f806b620 msgr2=0x7f39f806ba30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:04:34.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:34.316+0000 7f39ff000640 1 -- 172.21.15.16:0/3077887645 shutdown_connections 2024-03-31T15:04:34.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:34.316+0000 7f39ff000640 1 -- 172.21.15.16:0/3077887645 wait complete. 2024-03-31T15:04:34.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:34.316+0000 7f39ff000640 1 Processor -- start 2024-03-31T15:04:34.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:34.316+0000 7f39ff000640 1 -- start start 2024-03-31T15:04:34.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:34.320+0000 7f39ff000640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f39f80ce2f0 0x7f39f81fafa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:34.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:34.320+0000 7f39ff000640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f39f815b3d0 0x7f39f814a930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:34.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:34.320+0000 7f39ff000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f39f815eee0 0x7f39f80b1b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:34.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:34.320+0000 7f39ff000640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f39f80d0310 con 0x7f39f815eee0 2024-03-31T15:04:34.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:34.320+0000 7f39ff000640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f39f80d0190 con 0x7f39f815b3d0 2024-03-31T15:04:34.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:34.320+0000 7f39ff000640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f39f80d0490 con 0x7f39f80ce2f0 2024-03-31T15:04:34.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:34.320+0000 7f39fd000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f39f815eee0 0x7f39f80b1b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:34.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:34.320+0000 7f39fd000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f39f815eee0 0x7f39f80b1b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43918/0 (socket says 172.21.15.16:43918) 2024-03-31T15:04:34.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:34.320+0000 7f39fd000640 1 -- 172.21.15.16:0/3035513426 learned_addr learned my addr 172.21.15.16:0/3035513426 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:04:34.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:34.320+0000 7f39fd000640 1 -- 172.21.15.16:0/3035513426 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f39f80ce2f0 msgr2=0x7f39f81fafa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:34.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:34.320+0000 7f39fd000640 1 --2- 172.21.15.16:0/3035513426 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f39f80ce2f0 0x7f39f81fafa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:34.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:34.320+0000 7f39fd000640 1 -- 172.21.15.16:0/3035513426 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f39f815b3d0 msgr2=0x7f39f814a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:34.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:34.320+0000 7f39fd000640 1 --2- 172.21.15.16:0/3035513426 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f39f815b3d0 0x7f39f814a930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:34.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:34.320+0000 7f39fd000640 1 -- 172.21.15.16:0/3035513426 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39f80afa10 con 0x7f39f815eee0 2024-03-31T15:04:34.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:34.320+0000 7f39fd000640 1 --2- 172.21.15.16:0/3035513426 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f39f815eee0 0x7f39f80b1b50 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f39f000cbe0 tx=0x7f39f000a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:34.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:34.320+0000 7f39f4c00640 1 -- 172.21.15.16:0/3035513426 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39f0017070 con 0x7f39f815eee0 2024-03-31T15:04:34.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:34.320+0000 7f39f4c00640 1 -- 172.21.15.16:0/3035513426 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f39f000ad40 con 0x7f39f815eee0 2024-03-31T15:04:34.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:34.320+0000 7f39ff000640 1 -- 172.21.15.16:0/3035513426 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f39f80afcb0 con 0x7f39f815eee0 2024-03-31T15:04:34.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:34.320+0000 7f39f4c00640 1 -- 172.21.15.16:0/3035513426 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39f0012600 con 0x7f39f815eee0 2024-03-31T15:04:34.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:34.320+0000 7f39ff000640 1 -- 172.21.15.16:0/3035513426 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f39f80b0270 con 0x7f39f815eee0 2024-03-31T15:04:34.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:34.320+0000 7f39ff000640 1 -- 172.21.15.16:0/3035513426 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f39f8069950 con 0x7f39f815eee0 2024-03-31T15:04:34.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:34.320+0000 7f39f4c00640 1 -- 172.21.15.16:0/3035513426 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f39f001a070 con 0x7f39f815eee0 2024-03-31T15:04:34.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:34.324+0000 7f39f4c00640 1 --2- 172.21.15.16:0/3035513426 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f39d4068380 0x7f39d406a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:34.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:34.324+0000 7f39f4c00640 1 -- 172.21.15.16:0/3035513426 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 4857+0+0 (secure 0 0 0) 0x7f39f0086320 con 0x7f39f815eee0 2024-03-31T15:04:34.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:34.324+0000 7f39f7e00640 1 --2- 172.21.15.16:0/3035513426 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f39d4068380 0x7f39d406a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:34.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:34.324+0000 7f39f7e00640 1 --2- 172.21.15.16:0/3035513426 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f39d4068380 0x7f39d406a840 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f39e80b1000 tx=0x7f39e8004740 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:34.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:34.328+0000 7f39f4c00640 1 -- 172.21.15.16:0/3035513426 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f39f00565d0 con 0x7f39f815eee0 2024-03-31T15:04:34.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:34.552+0000 7f39ff000640 1 -- 172.21.15.16:0/3035513426 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-mgr"} v 0) -- 0x7f39f815d740 con 0x7f39f815eee0 2024-03-31T15:04:34.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:34.552+0000 7f39f4c00640 1 -- 172.21.15.16:0/3035513426 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-mgr"}]=0 v8) ==== 90+0+0 (secure 0 0 0) 0x7f39f005a8d0 con 0x7f39f815eee0 2024-03-31T15:04:34.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:34.556+0000 7f39ff000640 1 -- 172.21.15.16:0/3035513426 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f39d4068380 msgr2=0x7f39d406a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:34.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:34.556+0000 7f39ff000640 1 --2- 172.21.15.16:0/3035513426 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f39d4068380 0x7f39d406a840 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f39e80b1000 tx=0x7f39e8004740 comp rx=0 tx=0).stop 2024-03-31T15:04:34.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:34.556+0000 7f39ff000640 1 -- 172.21.15.16:0/3035513426 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f39f815eee0 msgr2=0x7f39f80b1b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:34.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:34.556+0000 7f39ff000640 1 --2- 172.21.15.16:0/3035513426 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f39f815eee0 0x7f39f80b1b50 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f39f000cbe0 tx=0x7f39f000a430 comp rx=0 tx=0).stop 2024-03-31T15:04:34.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:34.556+0000 7f39ff000640 1 -- 172.21.15.16:0/3035513426 shutdown_connections 2024-03-31T15:04:34.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:34.556+0000 7f39ff000640 1 --2- 172.21.15.16:0/3035513426 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f39d4068380 0x7f39d406a840 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:34.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:34.556+0000 7f39ff000640 1 --2- 172.21.15.16:0/3035513426 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f39f815eee0 0x7f39f80b1b50 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:34.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:34.556+0000 7f39ff000640 1 --2- 172.21.15.16:0/3035513426 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f39f815b3d0 0x7f39f814a930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:34.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:34.556+0000 7f39ff000640 1 --2- 172.21.15.16:0/3035513426 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f39f80ce2f0 0x7f39f81fafa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:34.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:34.556+0000 7f39ff000640 1 -- 172.21.15.16:0/3035513426 >> 172.21.15.16:0/3035513426 conn(0x7f39f806b620 msgr2=0x7f39f80b3890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:04:34.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:34.556+0000 7f39ff000640 1 -- 172.21.15.16:0/3035513426 shutdown_connections 2024-03-31T15:04:34.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:34.556+0000 7f39ff000640 1 -- 172.21.15.16:0/3035513426 wait complete. 2024-03-31T15:04:34.570 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-osd 2024-03-31T15:04:34.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:34.716+0000 7f4db3200640 1 Processor -- start 2024-03-31T15:04:34.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:34.716+0000 7f4db3200640 1 -- start start 2024-03-31T15:04:34.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:34.720+0000 7f4db3200640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f4dac0d0c50 0x7f4dac072fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:34.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:34.720+0000 7f4db3200640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f4dac073510 0x7f4dac15eb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:34.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:34.720+0000 7f4db3200640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4dac06bd30 con 0x7f4dac147e70 2024-03-31T15:04:34.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:34.720+0000 7f4db3200640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f4dac073980 con 0x7f4dac0d0c50 2024-03-31T15:04:34.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:34.720+0000 7f4db3200640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f4dac1621b0 con 0x7f4dac073510 2024-03-31T15:04:34.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:34.720+0000 7f4da3e00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f4dac0d0c50 0x7f4dac072fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:34.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:34.720+0000 7f4dabe00640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f4dac147e70 0x7f4dac06b620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:33746/0 (socket says 172.21.15.16:33746) 2024-03-31T15:04:34.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:34.720+0000 7f4dabe00640 1 -- 172.21.15.16:0/4074702178 learned_addr learned my addr 172.21.15.16:0/4074702178 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:04:34.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:34.720+0000 7f4da3e00640 1 -- 172.21.15.16:0/4074702178 >> v2:172.21.15.89:3300/0 conn(0x7f4dac073510 msgr2=0x7f4dac15eb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:04:34.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:34.720+0000 7f4db1200640 1 --2- 172.21.15.16:0/4074702178 >> v2:172.21.15.89:3300/0 conn(0x7f4dac073510 0x7f4dac15eb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:34.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:34.720+0000 7f4dab400640 1 -- 172.21.15.16:0/4074702178 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3274042600 0 0) 0x7f4dac06bd30 con 0x7f4dac147e70 2024-03-31T15:04:34.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:34.720+0000 7f4da3e00640 1 --2- 172.21.15.16:0/4074702178 >> v2:172.21.15.89:3300/0 conn(0x7f4dac073510 0x7f4dac15eb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:34.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:34.720+0000 7f4da3e00640 1 -- 172.21.15.16:0/4074702178 >> v1:172.21.15.16:6789/0 conn(0x7f4dac147e70 legacy=0x7f4dac06b620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:34.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:34.720+0000 7f4da3e00640 1 -- 172.21.15.16:0/4074702178 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4dac162490 con 0x7f4dac0d0c50 2024-03-31T15:04:34.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:34.720+0000 7f4db1200640 1 --2- 172.21.15.16:0/4074702178 >> v2:172.21.15.89:3300/0 conn(0x7f4dac073510 0x7f4dac15eb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:04:34.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:34.724+0000 7f4da3e00640 1 --2- 172.21.15.16:0/4074702178 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4dac0d0c50 0x7f4dac072fd0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f4dac06e280 tx=0x7f4d9401c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=225146ba82b087e9 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:34.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:34.724+0000 7f4dab400640 1 -- 172.21.15.16:0/4074702178 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d940b0070 con 0x7f4dac0d0c50 2024-03-31T15:04:34.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:34.724+0000 7f4dab400640 1 -- 172.21.15.16:0/4074702178 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4d940ab600 con 0x7f4dac0d0c50 2024-03-31T15:04:34.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:34.724+0000 7f4dab400640 1 -- 172.21.15.16:0/4074702178 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d940ab9d0 con 0x7f4dac0d0c50 2024-03-31T15:04:34.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:34.724+0000 7f4db3200640 1 -- 172.21.15.16:0/4074702178 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4dac0d0c50 msgr2=0x7f4dac072fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:34.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:34.724+0000 7f4db3200640 1 --2- 172.21.15.16:0/4074702178 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4dac0d0c50 0x7f4dac072fd0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f4dac06e280 tx=0x7f4d9401c480 comp rx=0 tx=0).stop 2024-03-31T15:04:34.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:34.724+0000 7f4db3200640 1 -- 172.21.15.16:0/4074702178 shutdown_connections 2024-03-31T15:04:34.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:34.724+0000 7f4db3200640 1 --2- 172.21.15.16:0/4074702178 >> v2:172.21.15.89:3300/0 conn(0x7f4dac073510 0x7f4dac15eb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:34.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:34.724+0000 7f4db3200640 1 --2- 172.21.15.16:0/4074702178 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4dac0d0c50 0x7f4dac072fd0 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:34.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:34.724+0000 7f4db3200640 1 -- 172.21.15.16:0/4074702178 >> 172.21.15.16:0/4074702178 conn(0x7f4dac0d1060 msgr2=0x7f4dac0d1470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:04:34.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:34.724+0000 7f4db3200640 1 -- 172.21.15.16:0/4074702178 shutdown_connections 2024-03-31T15:04:34.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:34.724+0000 7f4db3200640 1 -- 172.21.15.16:0/4074702178 wait complete. 2024-03-31T15:04:34.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:34.724+0000 7f4db3200640 1 Processor -- start 2024-03-31T15:04:34.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:34.724+0000 7f4db3200640 1 -- start start 2024-03-31T15:04:34.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:34.724+0000 7f4db3200640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4dac073510 0x7f4dac15bf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:34.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:34.724+0000 7f4db3200640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f4dac0d0c50 0x7f4dac0b5940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:34.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:34.724+0000 7f4db3200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4dac147e70 0x7f4dac0aed40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:34.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:34.724+0000 7f4db3200640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f4dac15f870 con 0x7f4dac147e70 2024-03-31T15:04:34.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:34.724+0000 7f4db3200640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f4dac162f70 con 0x7f4dac073510 2024-03-31T15:04:34.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:34.724+0000 7f4db3200640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f4dac15f9f0 con 0x7f4dac0d0c50 2024-03-31T15:04:34.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:34.724+0000 7f4db1200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4dac147e70 0x7f4dac0aed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:34.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:34.724+0000 7f4db1200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4dac147e70 0x7f4dac0aed40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43934/0 (socket says 172.21.15.16:43934) 2024-03-31T15:04:34.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:34.724+0000 7f4db1200640 1 -- 172.21.15.16:0/270869004 learned_addr learned my addr 172.21.15.16:0/270869004 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:04:34.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:34.728+0000 7f4dabe00640 1 --2- 172.21.15.16:0/270869004 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4dac073510 0x7f4dac15bf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:34.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:34.728+0000 7f4da3e00640 1 --2- 172.21.15.16:0/270869004 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f4dac0d0c50 0x7f4dac0b5940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:34.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:34.728+0000 7f4db1200640 1 -- 172.21.15.16:0/270869004 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f4dac0d0c50 msgr2=0x7f4dac0b5940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:34.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:34.728+0000 7f4db1200640 1 --2- 172.21.15.16:0/270869004 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f4dac0d0c50 0x7f4dac0b5940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:34.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:34.728+0000 7f4db1200640 1 -- 172.21.15.16:0/270869004 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4dac073510 msgr2=0x7f4dac15bf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:34.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:34.728+0000 7f4db1200640 1 --2- 172.21.15.16:0/270869004 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4dac073510 0x7f4dac15bf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:34.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:34.728+0000 7f4db1200640 1 -- 172.21.15.16:0/270869004 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4dac0b4980 con 0x7f4dac147e70 2024-03-31T15:04:34.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:34.728+0000 7f4da3e00640 1 --2- 172.21.15.16:0/270869004 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f4dac0d0c50 0x7f4dac0b5940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:04:34.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:34.728+0000 7f4db1200640 1 --2- 172.21.15.16:0/270869004 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4dac147e70 0x7f4dac0aed40 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f4da400cd80 tx=0x7f4da400a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:34.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:34.728+0000 7f4da8c00640 1 -- 172.21.15.16:0/270869004 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4da400f680 con 0x7f4dac147e70 2024-03-31T15:04:34.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:34.728+0000 7f4da8c00640 1 -- 172.21.15.16:0/270869004 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4da400f840 con 0x7f4dac147e70 2024-03-31T15:04:34.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:34.728+0000 7f4db3200640 1 -- 172.21.15.16:0/270869004 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4dac0ae260 con 0x7f4dac147e70 2024-03-31T15:04:34.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:34.728+0000 7f4da8c00640 1 -- 172.21.15.16:0/270869004 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4da400fc30 con 0x7f4dac147e70 2024-03-31T15:04:34.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:34.728+0000 7f4db3200640 1 -- 172.21.15.16:0/270869004 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4dac15dc50 con 0x7f4dac147e70 2024-03-31T15:04:34.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:34.728+0000 7f4db3200640 1 -- 172.21.15.16:0/270869004 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4dac069950 con 0x7f4dac147e70 2024-03-31T15:04:34.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:34.728+0000 7f4da8c00640 1 -- 172.21.15.16:0/270869004 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f4da40213f0 con 0x7f4dac147e70 2024-03-31T15:04:34.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:34.728+0000 7f4da8c00640 1 --2- 172.21.15.16:0/270869004 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f4d88068380 0x7f4d8806a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:34.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:34.732+0000 7f4da8c00640 1 -- 172.21.15.16:0/270869004 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 4857+0+0 (secure 0 0 0) 0x7f4da4087420 con 0x7f4dac147e70 2024-03-31T15:04:34.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:34.732+0000 7f4dabe00640 1 --2- 172.21.15.16:0/270869004 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f4d88068380 0x7f4d8806a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:34.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:34.732+0000 7f4dabe00640 1 --2- 172.21.15.16:0/270869004 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f4d88068380 0x7f4d8806a840 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f4d9c000f70 tx=0x7f4d9c007c10 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:34.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:34.736+0000 7f4da8c00640 1 -- 172.21.15.16:0/270869004 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f4da4014030 con 0x7f4dac147e70 2024-03-31T15:04:34.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:34.956+0000 7f4db3200640 1 -- 172.21.15.16:0/270869004 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-osd"} v 0) -- 0x7f4dac04f980 con 0x7f4dac147e70 2024-03-31T15:04:34.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:34.960+0000 7f4da8c00640 1 -- 172.21.15.16:0/270869004 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-osd"}]=0 v9) ==== 90+0+0 (secure 0 0 0) 0x7f4da4057d20 con 0x7f4dac147e70 2024-03-31T15:04:34.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:34.960+0000 7f4db3200640 1 -- 172.21.15.16:0/270869004 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f4d88068380 msgr2=0x7f4d8806a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:34.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:34.960+0000 7f4db3200640 1 --2- 172.21.15.16:0/270869004 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f4d88068380 0x7f4d8806a840 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f4d9c000f70 tx=0x7f4d9c007c10 comp rx=0 tx=0).stop 2024-03-31T15:04:34.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:34.960+0000 7f4db3200640 1 -- 172.21.15.16:0/270869004 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4dac147e70 msgr2=0x7f4dac0aed40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:34.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:34.960+0000 7f4db3200640 1 --2- 172.21.15.16:0/270869004 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4dac147e70 0x7f4dac0aed40 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f4da400cd80 tx=0x7f4da400a650 comp rx=0 tx=0).stop 2024-03-31T15:04:34.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:34.960+0000 7f4db3200640 1 -- 172.21.15.16:0/270869004 shutdown_connections 2024-03-31T15:04:34.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:34.960+0000 7f4db3200640 1 --2- 172.21.15.16:0/270869004 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f4d88068380 0x7f4d8806a840 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:34.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:34.960+0000 7f4db3200640 1 --2- 172.21.15.16:0/270869004 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4dac147e70 0x7f4dac0aed40 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:34.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:34.960+0000 7f4db3200640 1 --2- 172.21.15.16:0/270869004 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f4dac0d0c50 0x7f4dac0b5940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:34.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:34.960+0000 7f4db3200640 1 --2- 172.21.15.16:0/270869004 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4dac073510 0x7f4dac15bf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:34.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:34.960+0000 7f4db3200640 1 -- 172.21.15.16:0/270869004 >> 172.21.15.16:0/270869004 conn(0x7f4dac0d1060 msgr2=0x7f4dac160030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:04:34.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:34.960+0000 7f4db3200640 1 -- 172.21.15.16:0/270869004 shutdown_connections 2024-03-31T15:04:34.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:34.960+0000 7f4db3200640 1 -- 172.21.15.16:0/270869004 wait complete. 2024-03-31T15:04:34.974 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rbd 2024-03-31T15:04:35.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:35.120+0000 7f19a2400640 1 Processor -- start 2024-03-31T15:04:35.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:35.120+0000 7f19a2400640 1 -- start start 2024-03-31T15:04:35.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:35.124+0000 7f19a2400640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f199c15eee0 0x7f199c072420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:35.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:35.124+0000 7f19a2400640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f199c15f3c0 con 0x7f199c15b3d0 2024-03-31T15:04:35.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:35.124+0000 7f19a2400640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f199c15f6a0 con 0x7f199c0b23d0 2024-03-31T15:04:35.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:35.124+0000 7f19a2400640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f199c15b7b0 con 0x7f199c15eee0 2024-03-31T15:04:35.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:35.124+0000 7f199b400640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f199c15b3d0 0x7f199c0b1cc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:33762/0 (socket says 172.21.15.16:33762) 2024-03-31T15:04:35.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:35.124+0000 7f199b400640 1 -- 172.21.15.16:0/873121938 learned_addr learned my addr 172.21.15.16:0/873121938 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:04:35.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:35.124+0000 7f199be00640 1 --2- 172.21.15.16:0/873121938 >> v2:172.21.15.89:3300/0 conn(0x7f199c15eee0 0x7f199c072420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:35.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:35.124+0000 7f199aa00640 1 -- 172.21.15.16:0/873121938 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2008944738 0 0) 0x7f199c15f3c0 con 0x7f199c15b3d0 2024-03-31T15:04:35.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:35.124+0000 7f199aa00640 1 -- 172.21.15.16:0/873121938 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1980003530 con 0x7f199c15b3d0 2024-03-31T15:04:35.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:35.124+0000 7f199be00640 1 -- 172.21.15.16:0/873121938 >> v1:172.21.15.89:6789/0 conn(0x7f199c0b23d0 legacy=0x7f199c071d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:35.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:35.124+0000 7f199be00640 1 -- 172.21.15.16:0/873121938 >> v1:172.21.15.16:6789/0 conn(0x7f199c15b3d0 legacy=0x7f199c0b1cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:35.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:35.124+0000 7f199aa00640 1 -- 172.21.15.16:0/873121938 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3059539447 0 0) 0x7f1980003530 con 0x7f199c15b3d0 2024-03-31T15:04:35.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:35.124+0000 7f199be00640 1 -- 172.21.15.16:0/873121938 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f199c15f980 con 0x7f199c15eee0 2024-03-31T15:04:35.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:35.128+0000 7f199be00640 1 --2- 172.21.15.16:0/873121938 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f199c15eee0 0x7f199c072420 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f1994009d90 tx=0x7f199401c940 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cd54d13c41999273 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:35.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:35.128+0000 7f199aa00640 1 -- 172.21.15.16:0/873121938 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19940b2070 con 0x7f199c15eee0 2024-03-31T15:04:35.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:35.128+0000 7f199aa00640 1 -- 172.21.15.16:0/873121938 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f19940a59c0 con 0x7f199c15eee0 2024-03-31T15:04:35.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:35.128+0000 7f199aa00640 1 -- 172.21.15.16:0/873121938 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19940a5d90 con 0x7f199c15eee0 2024-03-31T15:04:35.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:35.128+0000 7f19a2400640 1 -- 172.21.15.16:0/873121938 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f199c15eee0 msgr2=0x7f199c072420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:35.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:35.128+0000 7f19a2400640 1 --2- 172.21.15.16:0/873121938 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f199c15eee0 0x7f199c072420 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f1994009d90 tx=0x7f199401c940 comp rx=0 tx=0).stop 2024-03-31T15:04:35.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:35.128+0000 7f19a2400640 1 -- 172.21.15.16:0/873121938 shutdown_connections 2024-03-31T15:04:35.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:35.128+0000 7f19a2400640 1 --2- 172.21.15.16:0/873121938 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f199c15eee0 0x7f199c072420 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:35.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:35.128+0000 7f19a2400640 1 -- 172.21.15.16:0/873121938 >> 172.21.15.16:0/873121938 conn(0x7f199c06b620 msgr2=0x7f199c06ba30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:04:35.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:35.128+0000 7f19a2400640 1 -- 172.21.15.16:0/873121938 shutdown_connections 2024-03-31T15:04:35.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:35.128+0000 7f19a2400640 1 -- 172.21.15.16:0/873121938 wait complete. 2024-03-31T15:04:35.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:35.128+0000 7f19a2400640 1 Processor -- start 2024-03-31T15:04:35.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:35.128+0000 7f19a2400640 1 -- start start 2024-03-31T15:04:35.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:35.128+0000 7f19a2400640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f199c0b23d0 0x7f199c14eca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:35.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:35.128+0000 7f19a2400640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f199c15b3d0 0x7f199c149540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:35.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:35.128+0000 7f19a2400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f199c15eee0 0x7f199c1572f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:35.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:35.128+0000 7f19a2400640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f199c073580 con 0x7f199c15eee0 2024-03-31T15:04:35.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:35.128+0000 7f19a2400640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f199c0b4560 con 0x7f199c15b3d0 2024-03-31T15:04:35.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:35.128+0000 7f19a2400640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f199c073700 con 0x7f199c0b23d0 2024-03-31T15:04:35.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:35.128+0000 7f199be00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f199c15eee0 0x7f199c1572f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:35.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:35.128+0000 7f199be00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f199c15eee0 0x7f199c1572f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43948/0 (socket says 172.21.15.16:43948) 2024-03-31T15:04:35.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:35.128+0000 7f199be00640 1 -- 172.21.15.16:0/1646443127 learned_addr learned my addr 172.21.15.16:0/1646443127 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:04:35.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:35.132+0000 7f199b400640 1 --2- 172.21.15.16:0/1646443127 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f199c0b23d0 0x7f199c14eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:35.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:35.132+0000 7f1992a00640 1 --2- 172.21.15.16:0/1646443127 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f199c15b3d0 0x7f199c149540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:35.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:35.132+0000 7f199be00640 1 -- 172.21.15.16:0/1646443127 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f199c0b23d0 msgr2=0x7f199c14eca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:35.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:35.132+0000 7f199be00640 1 --2- 172.21.15.16:0/1646443127 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f199c0b23d0 0x7f199c14eca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:35.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:35.132+0000 7f199be00640 1 -- 172.21.15.16:0/1646443127 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f199c15b3d0 msgr2=0x7f199c149540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:35.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:35.132+0000 7f199be00640 1 --2- 172.21.15.16:0/1646443127 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f199c15b3d0 0x7f199c149540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:35.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:35.132+0000 7f199be00640 1 -- 172.21.15.16:0/1646443127 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f199c149a80 con 0x7f199c15eee0 2024-03-31T15:04:35.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:35.132+0000 7f1992a00640 1 --2- 172.21.15.16:0/1646443127 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f199c15b3d0 0x7f199c149540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:04:35.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:35.132+0000 7f199be00640 1 --2- 172.21.15.16:0/1646443127 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f199c15eee0 0x7f199c1572f0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f1994009d90 tx=0x7f19940196e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:35.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:35.132+0000 7f1993e00640 1 -- 172.21.15.16:0/1646443127 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19940bd030 con 0x7f199c15eee0 2024-03-31T15:04:35.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:35.132+0000 7f1993e00640 1 -- 172.21.15.16:0/1646443127 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f19940039a0 con 0x7f199c15eee0 2024-03-31T15:04:35.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:35.132+0000 7f1993e00640 1 -- 172.21.15.16:0/1646443127 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19940b2040 con 0x7f199c15eee0 2024-03-31T15:04:35.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:35.132+0000 7f19a2400640 1 -- 172.21.15.16:0/1646443127 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f199c14f2b0 con 0x7f199c15eee0 2024-03-31T15:04:35.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:35.132+0000 7f19a2400640 1 -- 172.21.15.16:0/1646443127 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f199c1467c0 con 0x7f199c15eee0 2024-03-31T15:04:35.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:35.132+0000 7f19a2400640 1 -- 172.21.15.16:0/1646443127 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f199c069950 con 0x7f199c15eee0 2024-03-31T15:04:35.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:35.132+0000 7f1993e00640 1 -- 172.21.15.16:0/1646443127 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f19940c0070 con 0x7f199c15eee0 2024-03-31T15:04:35.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:35.136+0000 7f1993e00640 1 --2- 172.21.15.16:0/1646443127 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f1978068380 0x7f197806a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:35.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:35.136+0000 7f1993e00640 1 -- 172.21.15.16:0/1646443127 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 4857+0+0 (secure 0 0 0) 0x7f1994121160 con 0x7f199c15eee0 2024-03-31T15:04:35.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:35.136+0000 7f199b400640 1 --2- 172.21.15.16:0/1646443127 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f1978068380 0x7f197806a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:35.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:35.136+0000 7f199b400640 1 --2- 172.21.15.16:0/1646443127 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f1978068380 0x7f197806a840 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f198c005890 tx=0x7f198c009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:35.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:35.140+0000 7f1993e00640 1 -- 172.21.15.16:0/1646443127 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f1994016560 con 0x7f199c15eee0 2024-03-31T15:04:35.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:35.360+0000 7f19a2400640 1 -- 172.21.15.16:0/1646443127 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rbd"} v 0) -- 0x7f199c0b5030 con 0x7f199c15eee0 2024-03-31T15:04:35.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:35.360+0000 7f1993e00640 1 -- 172.21.15.16:0/1646443127 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rbd"}]=0 v10) ==== 90+0+0 (secure 0 0 0) 0x7f19940f19e0 con 0x7f199c15eee0 2024-03-31T15:04:35.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:35.364+0000 7f19a2400640 1 -- 172.21.15.16:0/1646443127 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f1978068380 msgr2=0x7f197806a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:35.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:35.364+0000 7f19a2400640 1 --2- 172.21.15.16:0/1646443127 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f1978068380 0x7f197806a840 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f198c005890 tx=0x7f198c009040 comp rx=0 tx=0).stop 2024-03-31T15:04:35.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:35.364+0000 7f19a2400640 1 -- 172.21.15.16:0/1646443127 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f199c15eee0 msgr2=0x7f199c1572f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:35.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:35.364+0000 7f19a2400640 1 --2- 172.21.15.16:0/1646443127 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f199c15eee0 0x7f199c1572f0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f1994009d90 tx=0x7f19940196e0 comp rx=0 tx=0).stop 2024-03-31T15:04:35.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:35.364+0000 7f19a2400640 1 -- 172.21.15.16:0/1646443127 shutdown_connections 2024-03-31T15:04:35.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:35.364+0000 7f19a2400640 1 --2- 172.21.15.16:0/1646443127 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f1978068380 0x7f197806a840 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:35.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:35.364+0000 7f19a2400640 1 --2- 172.21.15.16:0/1646443127 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f199c15eee0 0x7f199c1572f0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:35.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:35.364+0000 7f19a2400640 1 --2- 172.21.15.16:0/1646443127 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f199c15b3d0 0x7f199c149540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:35.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:35.364+0000 7f19a2400640 1 --2- 172.21.15.16:0/1646443127 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f199c0b23d0 0x7f199c14eca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:35.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:35.364+0000 7f19a2400640 1 -- 172.21.15.16:0/1646443127 >> 172.21.15.16:0/1646443127 conn(0x7f199c06b620 msgr2=0x7f199c0b1980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:04:35.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:35.364+0000 7f19a2400640 1 -- 172.21.15.16:0/1646443127 shutdown_connections 2024-03-31T15:04:35.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:35.364+0000 7f19a2400640 1 -- 172.21.15.16:0/1646443127 wait complete. 2024-03-31T15:04:35.377 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rbd-mirror 2024-03-31T15:04:35.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:35.524+0000 7f967ec00640 1 Processor -- start 2024-03-31T15:04:35.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:35.524+0000 7f967ec00640 1 -- start start 2024-03-31T15:04:35.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:35.524+0000 7f967ec00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f967806f920 0x7f967806fd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:35.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:35.524+0000 7f967ec00640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f96780d0920 con 0x7f96780d2010 2024-03-31T15:04:35.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:35.524+0000 7f967ec00640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f96780d0c00 con 0x7f9678145e40 2024-03-31T15:04:35.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:35.524+0000 7f967ec00640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f96780702c0 con 0x7f967806f920 2024-03-31T15:04:35.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:35.524+0000 7f967cc00640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f96780d2010 0x7f96780d24a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:33772/0 (socket says 172.21.15.16:33772) 2024-03-31T15:04:35.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:35.524+0000 7f967cc00640 1 -- 172.21.15.16:0/1529618998 learned_addr learned my addr 172.21.15.16:0/1529618998 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:04:35.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:35.524+0000 7f966f400640 1 --2- 172.21.15.16:0/1529618998 >> v2:172.21.15.89:3300/0 conn(0x7f967806f920 0x7f967806fd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:35.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:35.524+0000 7f9677400640 1 -- 172.21.15.16:0/1529618998 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1360702979 0 0) 0x7f96780d0920 con 0x7f96780d2010 2024-03-31T15:04:35.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:35.524+0000 7f9677400640 1 -- 172.21.15.16:0/1529618998 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f965c003530 con 0x7f96780d2010 2024-03-31T15:04:35.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:35.524+0000 7f9677400640 1 -- 172.21.15.16:0/1529618998 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3535030676 0 0) 0x7f965c003530 con 0x7f96780d2010 2024-03-31T15:04:35.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:35.524+0000 7f9677400640 1 -- 172.21.15.16:0/1529618998 >> v2:172.21.15.89:3300/0 conn(0x7f967806f920 msgr2=0x7f967806fd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:04:35.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:35.524+0000 7f9677400640 1 --2- 172.21.15.16:0/1529618998 >> v2:172.21.15.89:3300/0 conn(0x7f967806f920 0x7f967806fd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:35.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:35.524+0000 7f9677400640 1 -- 172.21.15.16:0/1529618998 >> v1:172.21.15.89:6789/0 conn(0x7f9678145e40 legacy=0x7f96780cc740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:35.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:35.528+0000 7f9677400640 1 -- 172.21.15.16:0/1529618998 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96780d1810 con 0x7f96780d2010 2024-03-31T15:04:35.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:35.528+0000 7f9677400640 1 -- 172.21.15.16:0/1529618998 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 214807623 0 0) 0x7f9670001a00 con 0x7f96780d2010 2024-03-31T15:04:35.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:35.528+0000 7f9677400640 1 -- 172.21.15.16:0/1529618998 >> v1:172.21.15.16:6789/0 conn(0x7f96780d2010 legacy=0x7f96780d24a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:35.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:35.528+0000 7f9677400640 1 --2- 172.21.15.16:0/1529618998 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f965c0039c0 0x7f965c003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:35.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:35.528+0000 7f9677400640 1 --2- 172.21.15.16:0/1529618998 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f965c004810 0x7f965c004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:35.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:35.528+0000 7f9677400640 1 --2- 172.21.15.16:0/1529618998 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f965c0051d0 0x7f965c00b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:35.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:35.528+0000 7f9677400640 1 -- 172.21.15.16:0/1529618998 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f965c00c040 con 0x7f965c004810 2024-03-31T15:04:35.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:35.528+0000 7f9677400640 1 -- 172.21.15.16:0/1529618998 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f965c00c1f0 con 0x7f965c0039c0 2024-03-31T15:04:35.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:35.528+0000 7f9677400640 1 -- 172.21.15.16:0/1529618998 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f965c00c3f0 con 0x7f965c0051d0 2024-03-31T15:04:35.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:35.528+0000 7f966f400640 1 --2- 172.21.15.16:0/1529618998 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f965c004810 0x7f965c004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:35.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:35.528+0000 7f9677e00640 1 --2- 172.21.15.16:0/1529618998 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f965c0039c0 0x7f965c003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:35.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:35.528+0000 7f967cc00640 1 --2- 172.21.15.16:0/1529618998 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f965c0051d0 0x7f965c00b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:35.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:35.528+0000 7f966f400640 1 -- 172.21.15.16:0/1529618998 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f965c0051d0 msgr2=0x7f965c00b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:04:35.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:35.528+0000 7f966f400640 1 --2- 172.21.15.16:0/1529618998 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f965c0051d0 0x7f965c00b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:35.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:35.528+0000 7f966f400640 1 -- 172.21.15.16:0/1529618998 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f965c0039c0 msgr2=0x7f965c003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:04:35.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:35.528+0000 7f966f400640 1 --2- 172.21.15.16:0/1529618998 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f965c0039c0 0x7f965c003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:35.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:35.528+0000 7f966f400640 1 -- 172.21.15.16:0/1529618998 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f965c003530 con 0x7f965c004810 2024-03-31T15:04:35.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:35.528+0000 7f967cc00640 1 -- 172.21.15.16:0/1529618998 reap_dead start 2024-03-31T15:04:35.525 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:35.528+0000 7f966f400640 1 --2- 172.21.15.16:0/1529618998 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f965c004810 0x7f965c004c90 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f9660009ae0 tx=0x7f966001ca70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad60b71f1bac8afe server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:35.525 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:35.528+0000 7f9677400640 1 -- 172.21.15.16:0/1529618998 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96600b2070 con 0x7f965c004810 2024-03-31T15:04:35.525 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:35.528+0000 7f9677400640 1 -- 172.21.15.16:0/1529618998 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f96600a5660 con 0x7f965c004810 2024-03-31T15:04:35.525 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:35.528+0000 7f9677400640 1 -- 172.21.15.16:0/1529618998 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96600096b0 con 0x7f965c004810 2024-03-31T15:04:35.525 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:35.532+0000 7f967ec00640 1 -- 172.21.15.16:0/1529618998 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f965c004810 msgr2=0x7f965c004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:35.525 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:35.532+0000 7f967ec00640 1 --2- 172.21.15.16:0/1529618998 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f965c004810 0x7f965c004c90 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f9660009ae0 tx=0x7f966001ca70 comp rx=0 tx=0).stop 2024-03-31T15:04:35.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:35.532+0000 7f967ec00640 1 -- 172.21.15.16:0/1529618998 shutdown_connections 2024-03-31T15:04:35.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:35.532+0000 7f967ec00640 1 --2- 172.21.15.16:0/1529618998 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f965c004810 0x7f965c004c90 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:35.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:35.532+0000 7f967ec00640 1 -- 172.21.15.16:0/1529618998 >> 172.21.15.16:0/1529618998 conn(0x7f9678067330 msgr2=0x7f9678067740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:04:35.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:35.532+0000 7f967ec00640 1 -- 172.21.15.16:0/1529618998 shutdown_connections 2024-03-31T15:04:35.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:35.532+0000 7f967ec00640 1 -- 172.21.15.16:0/1529618998 wait complete. 2024-03-31T15:04:35.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:35.532+0000 7f967ec00640 1 Processor -- start 2024-03-31T15:04:35.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:35.532+0000 7f967ec00640 1 -- start start 2024-03-31T15:04:35.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:35.532+0000 7f967ec00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f965c004810 0x7f9678145110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:35.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:35.532+0000 7f967ec00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f96781f28e0 0x7f96781578a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:35.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:35.532+0000 7f967ec00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9678154ef0 0x7f9678155370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:35.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:35.532+0000 7f967ec00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f96780ce9e0 con 0x7f965c004810 2024-03-31T15:04:35.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:35.532+0000 7f967ec00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f96780d06d0 con 0x7f9678154ef0 2024-03-31T15:04:35.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:35.532+0000 7f967ec00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f96780d0ee0 con 0x7f96781f28e0 2024-03-31T15:04:35.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:35.532+0000 7f9677e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f965c004810 0x7f9678145110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:35.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:35.532+0000 7f9677e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f965c004810 0x7f9678145110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43960/0 (socket says 172.21.15.16:43960) 2024-03-31T15:04:35.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:35.532+0000 7f9677e00640 1 -- 172.21.15.16:0/565605677 learned_addr learned my addr 172.21.15.16:0/565605677 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:04:35.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:35.532+0000 7f966f400640 1 --2- 172.21.15.16:0/565605677 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f96781f28e0 0x7f96781578a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:35.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:35.532+0000 7f967cc00640 1 --2- 172.21.15.16:0/565605677 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9678154ef0 0x7f9678155370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:35.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:35.532+0000 7f9677e00640 1 -- 172.21.15.16:0/565605677 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f96781f28e0 msgr2=0x7f96781578a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:35.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:35.532+0000 7f9677e00640 1 --2- 172.21.15.16:0/565605677 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f96781f28e0 0x7f96781578a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:35.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:35.532+0000 7f9677e00640 1 -- 172.21.15.16:0/565605677 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9678154ef0 msgr2=0x7f9678155370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:35.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:35.532+0000 7f9677e00640 1 --2- 172.21.15.16:0/565605677 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9678154ef0 0x7f9678155370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:35.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:35.532+0000 7f9677e00640 1 -- 172.21.15.16:0/565605677 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96781f2e30 con 0x7f965c004810 2024-03-31T15:04:35.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:35.532+0000 7f967cc00640 1 --2- 172.21.15.16:0/565605677 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9678154ef0 0x7f9678155370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:04:35.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:35.536+0000 7f9677e00640 1 --2- 172.21.15.16:0/565605677 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f965c004810 0x7f9678145110 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f966800f850 tx=0x7f966800fd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:35.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:35.536+0000 7f9674c00640 1 -- 172.21.15.16:0/565605677 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9668001cf0 con 0x7f965c004810 2024-03-31T15:04:35.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:35.536+0000 7f9674c00640 1 -- 172.21.15.16:0/565605677 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9668018070 con 0x7f965c004810 2024-03-31T15:04:35.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:35.536+0000 7f9674c00640 1 -- 172.21.15.16:0/565605677 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f966800d6d0 con 0x7f965c004810 2024-03-31T15:04:35.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:35.536+0000 7f967ec00640 1 -- 172.21.15.16:0/565605677 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9678159cb0 con 0x7f965c004810 2024-03-31T15:04:35.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:35.536+0000 7f967ec00640 1 -- 172.21.15.16:0/565605677 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f967815a270 con 0x7f965c004810 2024-03-31T15:04:35.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:35.536+0000 7f967ec00640 1 -- 172.21.15.16:0/565605677 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9678069950 con 0x7f965c004810 2024-03-31T15:04:35.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:35.536+0000 7f9674c00640 1 -- 172.21.15.16:0/565605677 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f966800d870 con 0x7f965c004810 2024-03-31T15:04:35.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:35.536+0000 7f9674c00640 1 --2- 172.21.15.16:0/565605677 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f9654068380 0x7f965406a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:35.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:35.536+0000 7f9674c00640 1 -- 172.21.15.16:0/565605677 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 4857+0+0 (secure 0 0 0) 0x7f96680863b0 con 0x7f965c004810 2024-03-31T15:04:35.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:35.536+0000 7f966f400640 1 --2- 172.21.15.16:0/565605677 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f9654068380 0x7f965406a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:35.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:35.540+0000 7f966f400640 1 --2- 172.21.15.16:0/565605677 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f9654068380 0x7f965406a840 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f96600af000 tx=0x7f96600046d0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:35.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:35.540+0000 7f9674c00640 1 -- 172.21.15.16:0/565605677 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f9668056c20 con 0x7f965c004810 2024-03-31T15:04:35.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:35.764+0000 7f967ec00640 1 -- 172.21.15.16:0/565605677 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rbd-mirror"} v 0) -- 0x7f96780cf3c0 con 0x7f965c004810 2024-03-31T15:04:35.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:35.764+0000 7f9674c00640 1 -- 172.21.15.16:0/565605677 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rbd-mirror"}]=0 v11) ==== 97+0+0 (secure 0 0 0) 0x7f9668056520 con 0x7f965c004810 2024-03-31T15:04:35.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:35.768+0000 7f967ec00640 1 -- 172.21.15.16:0/565605677 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f9654068380 msgr2=0x7f965406a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:35.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:35.768+0000 7f967ec00640 1 --2- 172.21.15.16:0/565605677 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f9654068380 0x7f965406a840 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f96600af000 tx=0x7f96600046d0 comp rx=0 tx=0).stop 2024-03-31T15:04:35.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:35.768+0000 7f967ec00640 1 -- 172.21.15.16:0/565605677 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f965c004810 msgr2=0x7f9678145110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:35.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:35.768+0000 7f967ec00640 1 --2- 172.21.15.16:0/565605677 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f965c004810 0x7f9678145110 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f966800f850 tx=0x7f966800fd10 comp rx=0 tx=0).stop 2024-03-31T15:04:35.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:35.768+0000 7f967ec00640 1 -- 172.21.15.16:0/565605677 shutdown_connections 2024-03-31T15:04:35.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:35.768+0000 7f967ec00640 1 --2- 172.21.15.16:0/565605677 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f9654068380 0x7f965406a840 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:35.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:35.768+0000 7f967ec00640 1 --2- 172.21.15.16:0/565605677 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9678154ef0 0x7f9678155370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:35.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:35.768+0000 7f967ec00640 1 --2- 172.21.15.16:0/565605677 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f96781f28e0 0x7f96781578a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:35.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:35.768+0000 7f967ec00640 1 --2- 172.21.15.16:0/565605677 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f965c004810 0x7f9678145110 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:35.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:35.768+0000 7f967ec00640 1 -- 172.21.15.16:0/565605677 >> 172.21.15.16:0/565605677 conn(0x7f9678067330 msgr2=0x7f96780b71d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:04:35.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:35.768+0000 7f967ec00640 1 -- 172.21.15.16:0/565605677 shutdown_connections 2024-03-31T15:04:35.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:35.768+0000 7f967ec00640 1 -- 172.21.15.16:0/565605677 wait complete. 2024-03-31T15:04:35.781 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rgw 2024-03-31T15:04:35.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:35.928+0000 7efe45c00640 1 Processor -- start 2024-03-31T15:04:35.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:35.928+0000 7efe45c00640 1 -- start start 2024-03-31T15:04:35.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:35.928+0000 7efe45c00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7efe400728b0 0x7efe400b1cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:35.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:35.928+0000 7efe45c00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7efe400b2200 0x7efe4015cd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:35.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:35.928+0000 7efe45c00640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efe40160e70 con 0x7efe40145d10 2024-03-31T15:04:35.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:35.928+0000 7efe45c00640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7efe401602f0 con 0x7efe400b2200 2024-03-31T15:04:35.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:35.928+0000 7efe45c00640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7efe401605d0 con 0x7efe400728b0 2024-03-31T15:04:35.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:35.928+0000 7efe3ea00640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7efe40145d10 0x7efe400cf790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:33782/0 (socket says 172.21.15.16:33782) 2024-03-31T15:04:35.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:35.928+0000 7efe3f400640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7efe400b2200 0x7efe4015cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:35.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:35.928+0000 7efe3ea00640 1 -- 172.21.15.16:0/1875542125 learned_addr learned my addr 172.21.15.16:0/1875542125 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:04:35.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:35.928+0000 7efe36000640 1 --2- 172.21.15.16:0/1875542125 >> v2:172.21.15.89:3300/0 conn(0x7efe400728b0 0x7efe400b1cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:35.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:35.928+0000 7efe3f400640 1 -- 172.21.15.16:0/1875542125 >> v2:172.21.15.89:3300/0 conn(0x7efe400728b0 msgr2=0x7efe400b1cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:04:35.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:35.928+0000 7efe3f400640 1 --2- 172.21.15.16:0/1875542125 >> v2:172.21.15.89:3300/0 conn(0x7efe400728b0 0x7efe400b1cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:35.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:35.928+0000 7efe3f400640 1 -- 172.21.15.16:0/1875542125 >> v1:172.21.15.16:6789/0 conn(0x7efe40145d10 legacy=0x7efe400cf790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:35.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:35.928+0000 7efe3f400640 1 -- 172.21.15.16:0/1875542125 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efe401608b0 con 0x7efe400b2200 2024-03-31T15:04:35.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:35.928+0000 7efe3e000640 1 -- 172.21.15.16:0/1875542125 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2923228942 0 0) 0x7efe40160e70 con 0x7efe40145d10 2024-03-31T15:04:35.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:35.932+0000 7efe3f400640 1 --2- 172.21.15.16:0/1875542125 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efe400b2200 0x7efe4015cd50 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7efe400b2980 tx=0x7efe3801c930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8562d1641fb90745 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:35.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:35.932+0000 7efe3e000640 1 -- 172.21.15.16:0/1875542125 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe380b2070 con 0x7efe400b2200 2024-03-31T15:04:35.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:35.932+0000 7efe3e000640 1 -- 172.21.15.16:0/1875542125 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efe380a57f0 con 0x7efe400b2200 2024-03-31T15:04:35.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:35.932+0000 7efe3e000640 1 -- 172.21.15.16:0/1875542125 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe380a5bc0 con 0x7efe400b2200 2024-03-31T15:04:35.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:35.932+0000 7efe45c00640 1 -- 172.21.15.16:0/1875542125 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efe400b2200 msgr2=0x7efe4015cd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:35.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:35.932+0000 7efe45c00640 1 --2- 172.21.15.16:0/1875542125 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efe400b2200 0x7efe4015cd50 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7efe400b2980 tx=0x7efe3801c930 comp rx=0 tx=0).stop 2024-03-31T15:04:35.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:35.932+0000 7efe45c00640 1 -- 172.21.15.16:0/1875542125 shutdown_connections 2024-03-31T15:04:35.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:35.932+0000 7efe45c00640 1 --2- 172.21.15.16:0/1875542125 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efe400b2200 0x7efe4015cd50 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:35.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:35.932+0000 7efe45c00640 1 --2- 172.21.15.16:0/1875542125 >> v2:172.21.15.89:3300/0 conn(0x7efe400728b0 0x7efe400b1cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:35.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:35.932+0000 7efe45c00640 1 -- 172.21.15.16:0/1875542125 >> 172.21.15.16:0/1875542125 conn(0x7efe4006b620 msgr2=0x7efe4006ba30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:04:35.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:35.932+0000 7efe45c00640 1 -- 172.21.15.16:0/1875542125 shutdown_connections 2024-03-31T15:04:35.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:35.932+0000 7efe45c00640 1 -- 172.21.15.16:0/1875542125 wait complete. 2024-03-31T15:04:35.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:35.936+0000 7efe45c00640 1 Processor -- start 2024-03-31T15:04:35.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:35.936+0000 7efe45c00640 1 -- start start 2024-03-31T15:04:35.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:35.936+0000 7efe45c00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efe400728b0 0x7efe4015e690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:35.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:35.936+0000 7efe45c00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7efe400b2200 0x7efe4015ee50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:35.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:35.936+0000 7efe45c00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efe40145d10 0x7efe40141040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:35.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:35.936+0000 7efe45c00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7efe40072e40 con 0x7efe400728b0 2024-03-31T15:04:35.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:35.936+0000 7efe45c00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7efe40072cc0 con 0x7efe40145d10 2024-03-31T15:04:35.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:35.936+0000 7efe45c00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7efe401690f0 con 0x7efe400b2200 2024-03-31T15:04:35.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:35.936+0000 7efe3ea00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efe400728b0 0x7efe4015e690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:35.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:35.936+0000 7efe3ea00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efe400728b0 0x7efe4015e690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43964/0 (socket says 172.21.15.16:43964) 2024-03-31T15:04:35.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:35.936+0000 7efe3ea00640 1 -- 172.21.15.16:0/680542835 learned_addr learned my addr 172.21.15.16:0/680542835 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:04:35.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:35.936+0000 7efe3f400640 1 --2- 172.21.15.16:0/680542835 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efe40145d10 0x7efe40141040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:35.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:35.936+0000 7efe36000640 1 --2- 172.21.15.16:0/680542835 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7efe400b2200 0x7efe4015ee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:35.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:35.936+0000 7efe3ea00640 1 -- 172.21.15.16:0/680542835 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7efe400b2200 msgr2=0x7efe4015ee50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:35.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:35.936+0000 7efe3ea00640 1 --2- 172.21.15.16:0/680542835 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7efe400b2200 0x7efe4015ee50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:35.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:35.936+0000 7efe3ea00640 1 -- 172.21.15.16:0/680542835 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efe40145d10 msgr2=0x7efe40141040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:35.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:35.936+0000 7efe3ea00640 1 --2- 172.21.15.16:0/680542835 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efe40145d10 0x7efe40141040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:35.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:35.936+0000 7efe3ea00640 1 -- 172.21.15.16:0/680542835 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efe4015dbf0 con 0x7efe400728b0 2024-03-31T15:04:35.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:35.936+0000 7efe3ea00640 1 --2- 172.21.15.16:0/680542835 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efe400728b0 0x7efe4015e690 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7efe3000c480 tx=0x7efe3000c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:35.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:35.936+0000 7efe37400640 1 -- 172.21.15.16:0/680542835 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe30018070 con 0x7efe400728b0 2024-03-31T15:04:35.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:35.936+0000 7efe37400640 1 -- 172.21.15.16:0/680542835 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efe30007500 con 0x7efe400728b0 2024-03-31T15:04:35.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:35.936+0000 7efe45c00640 1 -- 172.21.15.16:0/680542835 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efe4015f750 con 0x7efe400728b0 2024-03-31T15:04:35.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:35.936+0000 7efe37400640 1 -- 172.21.15.16:0/680542835 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe30004a30 con 0x7efe400728b0 2024-03-31T15:04:35.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:35.940+0000 7efe45c00640 1 -- 172.21.15.16:0/680542835 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efe4015fa30 con 0x7efe400728b0 2024-03-31T15:04:35.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:35.940+0000 7efe45c00640 1 -- 172.21.15.16:0/680542835 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efe40069950 con 0x7efe400728b0 2024-03-31T15:04:35.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:35.940+0000 7efe37400640 1 -- 172.21.15.16:0/680542835 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7efe30004bd0 con 0x7efe400728b0 2024-03-31T15:04:35.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:35.940+0000 7efe37400640 1 --2- 172.21.15.16:0/680542835 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7efe1c068380 0x7efe1c06a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:35.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:35.940+0000 7efe37400640 1 -- 172.21.15.16:0/680542835 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 4857+0+0 (secure 0 0 0) 0x7efe30085920 con 0x7efe400728b0 2024-03-31T15:04:35.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:35.940+0000 7efe36000640 1 --2- 172.21.15.16:0/680542835 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7efe1c068380 0x7efe1c06a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:35.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:35.940+0000 7efe36000640 1 --2- 172.21.15.16:0/680542835 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7efe1c068380 0x7efe1c06a840 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7efe40141b10 tx=0x7efe28008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:35.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:35.944+0000 7efe37400640 1 -- 172.21.15.16:0/680542835 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7efe30055b50 con 0x7efe400728b0 2024-03-31T15:04:36.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:36.164+0000 7efe45c00640 1 -- 172.21.15.16:0/680542835 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rgw"} v 0) -- 0x7efe400b1400 con 0x7efe400728b0 2024-03-31T15:04:36.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:36.168+0000 7efe37400640 1 -- 172.21.15.16:0/680542835 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rgw"}]=0 v12) ==== 90+0+0 (secure 0 0 0) 0x7efe30059e50 con 0x7efe400728b0 2024-03-31T15:04:36.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:36.172+0000 7efe45c00640 1 -- 172.21.15.16:0/680542835 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7efe1c068380 msgr2=0x7efe1c06a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:36.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:36.172+0000 7efe45c00640 1 --2- 172.21.15.16:0/680542835 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7efe1c068380 0x7efe1c06a840 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7efe40141b10 tx=0x7efe28008040 comp rx=0 tx=0).stop 2024-03-31T15:04:36.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:36.172+0000 7efe45c00640 1 -- 172.21.15.16:0/680542835 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efe400728b0 msgr2=0x7efe4015e690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:36.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:36.172+0000 7efe45c00640 1 --2- 172.21.15.16:0/680542835 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efe400728b0 0x7efe4015e690 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7efe3000c480 tx=0x7efe3000c940 comp rx=0 tx=0).stop 2024-03-31T15:04:36.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:36.172+0000 7efe45c00640 1 -- 172.21.15.16:0/680542835 shutdown_connections 2024-03-31T15:04:36.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:36.172+0000 7efe45c00640 1 --2- 172.21.15.16:0/680542835 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7efe1c068380 0x7efe1c06a840 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:36.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:36.172+0000 7efe45c00640 1 --2- 172.21.15.16:0/680542835 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efe40145d10 0x7efe40141040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:36.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:36.172+0000 7efe45c00640 1 --2- 172.21.15.16:0/680542835 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7efe400b2200 0x7efe4015ee50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:36.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:36.172+0000 7efe45c00640 1 --2- 172.21.15.16:0/680542835 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efe400728b0 0x7efe4015e690 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:36.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:36.172+0000 7efe45c00640 1 -- 172.21.15.16:0/680542835 >> 172.21.15.16:0/680542835 conn(0x7efe4006b620 msgr2=0x7efe400b1920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:04:36.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:36.172+0000 7efe45c00640 1 -- 172.21.15.16:0/680542835 shutdown_connections 2024-03-31T15:04:36.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:36.172+0000 7efe45c00640 1 -- 172.21.15.16:0/680542835 wait complete. 2024-03-31T15:04:36.191 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-31T15:04:36.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:36.336+0000 7f1caba00640 1 Processor -- start 2024-03-31T15:04:36.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:36.336+0000 7f1caba00640 1 -- start start 2024-03-31T15:04:36.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:36.336+0000 7f1caba00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f1ca40728b0 0x7f1ca40b1cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:36.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:36.336+0000 7f1caba00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f1ca40b2200 0x7f1ca415cd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:36.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:36.336+0000 7f1caba00640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1ca4160e70 con 0x7f1ca4145d10 2024-03-31T15:04:36.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:36.336+0000 7f1caba00640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f1ca41602f0 con 0x7f1ca40728b0 2024-03-31T15:04:36.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:36.336+0000 7f1caba00640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f1ca41605d0 con 0x7f1ca40b2200 2024-03-31T15:04:36.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:36.336+0000 7f1c9be00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f1ca40728b0 0x7f1ca40b1cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:36.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:36.336+0000 7f1c9be00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f1ca40728b0 0x7f1ca40b1cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43970/0 (socket says 172.21.15.16:43970) 2024-03-31T15:04:36.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:36.336+0000 7f1c9be00640 1 -- 172.21.15.16:0/1169925576 learned_addr learned my addr 172.21.15.16:0/1169925576 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:04:36.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:36.336+0000 7f1ca9a00640 1 --2- 172.21.15.16:0/1169925576 >> v2:172.21.15.89:3300/0 conn(0x7f1ca40b2200 0x7f1ca415cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:36.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:36.340+0000 7f1c9be00640 1 -- 172.21.15.16:0/1169925576 >> v2:172.21.15.89:3300/0 conn(0x7f1ca40b2200 msgr2=0x7f1ca415cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:04:36.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:36.340+0000 7f1c9be00640 1 --2- 172.21.15.16:0/1169925576 >> v2:172.21.15.89:3300/0 conn(0x7f1ca40b2200 0x7f1ca415cd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:36.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:36.340+0000 7f1c9be00640 1 -- 172.21.15.16:0/1169925576 >> v1:172.21.15.89:6789/0 conn(0x7f1ca4145d10 legacy=0x7f1ca40cf790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:36.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:36.340+0000 7f1c9be00640 1 -- 172.21.15.16:0/1169925576 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ca41608b0 con 0x7f1ca40728b0 2024-03-31T15:04:36.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:36.340+0000 7f1ca9a00640 1 --2- 172.21.15.16:0/1169925576 >> v2:172.21.15.89:3300/0 conn(0x7f1ca40b2200 0x7f1ca415cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:04:36.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:36.340+0000 7f1c9be00640 1 --2- 172.21.15.16:0/1169925576 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1ca40728b0 0x7f1ca40b1cc0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f1ca40b5400 tx=0x7f1c8c01c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b1995160caf8ec67 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:36.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:36.340+0000 7f1ca3e00640 1 -- 172.21.15.16:0/1169925576 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c8c0b0070 con 0x7f1ca40728b0 2024-03-31T15:04:36.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:36.340+0000 7f1ca3e00640 1 -- 172.21.15.16:0/1169925576 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1c8c0ab600 con 0x7f1ca40728b0 2024-03-31T15:04:36.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:36.340+0000 7f1ca3e00640 1 -- 172.21.15.16:0/1169925576 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c8c0ab9d0 con 0x7f1ca40728b0 2024-03-31T15:04:36.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:36.340+0000 7f1caba00640 1 -- 172.21.15.16:0/1169925576 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1ca40728b0 msgr2=0x7f1ca40b1cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:36.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:36.340+0000 7f1caba00640 1 --2- 172.21.15.16:0/1169925576 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1ca40728b0 0x7f1ca40b1cc0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f1ca40b5400 tx=0x7f1c8c01c520 comp rx=0 tx=0).stop 2024-03-31T15:04:36.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:36.340+0000 7f1caba00640 1 -- 172.21.15.16:0/1169925576 shutdown_connections 2024-03-31T15:04:36.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:36.340+0000 7f1caba00640 1 --2- 172.21.15.16:0/1169925576 >> v2:172.21.15.89:3300/0 conn(0x7f1ca40b2200 0x7f1ca415cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:36.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:36.340+0000 7f1caba00640 1 --2- 172.21.15.16:0/1169925576 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1ca40728b0 0x7f1ca40b1cc0 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:36.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:36.340+0000 7f1caba00640 1 -- 172.21.15.16:0/1169925576 >> 172.21.15.16:0/1169925576 conn(0x7f1ca406b620 msgr2=0x7f1ca406ba30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:04:36.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:36.344+0000 7f1caba00640 1 -- 172.21.15.16:0/1169925576 shutdown_connections 2024-03-31T15:04:36.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:36.344+0000 7f1caba00640 1 -- 172.21.15.16:0/1169925576 wait complete. 2024-03-31T15:04:36.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:36.344+0000 7f1caba00640 1 Processor -- start 2024-03-31T15:04:36.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:36.344+0000 7f1caba00640 1 -- start start 2024-03-31T15:04:36.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:36.344+0000 7f1caba00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1ca40728b0 0x7f1ca413d960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:36.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:36.344+0000 7f1caba00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1ca40b2200 0x7f1ca415aac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:36.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:36.344+0000 7f1caba00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f1ca4145d10 0x7f1ca415b000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:36.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:36.344+0000 7f1caba00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f1ca4072d40 con 0x7f1ca40728b0 2024-03-31T15:04:36.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:36.344+0000 7f1caba00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f1ca40b2610 con 0x7f1ca40b2200 2024-03-31T15:04:36.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:36.344+0000 7f1caba00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f1ca4072ec0 con 0x7f1ca4145d10 2024-03-31T15:04:36.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:36.344+0000 7f1ca9000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1ca40728b0 0x7f1ca413d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:36.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:36.344+0000 7f1ca9000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1ca40728b0 0x7f1ca413d960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43986/0 (socket says 172.21.15.16:43986) 2024-03-31T15:04:36.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:36.344+0000 7f1ca9000640 1 -- 172.21.15.16:0/2762693030 learned_addr learned my addr 172.21.15.16:0/2762693030 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:04:36.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:36.344+0000 7f1c9be00640 1 --2- 172.21.15.16:0/2762693030 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1ca40b2200 0x7f1ca415aac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:36.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:36.344+0000 7f1ca9a00640 1 --2- 172.21.15.16:0/2762693030 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f1ca4145d10 0x7f1ca415b000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:36.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:36.344+0000 7f1ca9000640 1 -- 172.21.15.16:0/2762693030 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f1ca4145d10 msgr2=0x7f1ca415b000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:36.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:36.344+0000 7f1ca9000640 1 --2- 172.21.15.16:0/2762693030 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f1ca4145d10 0x7f1ca415b000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:36.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:36.344+0000 7f1ca9000640 1 -- 172.21.15.16:0/2762693030 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1ca40b2200 msgr2=0x7f1ca415aac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:36.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:36.344+0000 7f1ca9000640 1 --2- 172.21.15.16:0/2762693030 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1ca40b2200 0x7f1ca415aac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:36.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:36.344+0000 7f1ca9000640 1 -- 172.21.15.16:0/2762693030 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ca413e360 con 0x7f1ca40728b0 2024-03-31T15:04:36.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:36.344+0000 7f1ca9000640 1 --2- 172.21.15.16:0/2762693030 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1ca40728b0 0x7f1ca413d960 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f1c9400c3f0 tx=0x7f1c9400c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:36.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:36.348+0000 7f1ca1600640 1 -- 172.21.15.16:0/2762693030 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c94017070 con 0x7f1ca40728b0 2024-03-31T15:04:36.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:36.348+0000 7f1ca1600640 1 -- 172.21.15.16:0/2762693030 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1c940052d0 con 0x7f1ca40728b0 2024-03-31T15:04:36.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:36.348+0000 7f1ca1600640 1 -- 172.21.15.16:0/2762693030 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c94003840 con 0x7f1ca40728b0 2024-03-31T15:04:36.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:36.348+0000 7f1caba00640 1 -- 172.21.15.16:0/2762693030 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1ca413e640 con 0x7f1ca40728b0 2024-03-31T15:04:36.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:36.348+0000 7f1caba00640 1 -- 172.21.15.16:0/2762693030 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1ca413ec00 con 0x7f1ca40728b0 2024-03-31T15:04:36.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:36.348+0000 7f1caba00640 1 -- 172.21.15.16:0/2762693030 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1ca4069950 con 0x7f1ca40728b0 2024-03-31T15:04:36.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:36.348+0000 7f1ca1600640 1 -- 172.21.15.16:0/2762693030 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f1c9401a070 con 0x7f1ca40728b0 2024-03-31T15:04:36.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:36.348+0000 7f1ca1600640 1 --2- 172.21.15.16:0/2762693030 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f1c80068380 0x7f1c8006a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:36.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:36.352+0000 7f1ca1600640 1 -- 172.21.15.16:0/2762693030 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 4857+0+0 (secure 0 0 0) 0x7f1c94085970 con 0x7f1ca40728b0 2024-03-31T15:04:36.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:36.352+0000 7f1c9be00640 1 --2- 172.21.15.16:0/2762693030 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f1c80068380 0x7f1c8006a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:36.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:36.352+0000 7f1c9be00640 1 --2- 172.21.15.16:0/2762693030 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f1c80068380 0x7f1c8006a840 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f1c8c0ad000 tx=0x7f1c8c008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:36.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:36.356+0000 7f1ca1600640 1 -- 172.21.15.16:0/2762693030 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f1c94005480 con 0x7f1ca40728b0 2024-03-31T15:04:36.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:36.520+0000 7f1caba00640 1 -- 172.21.15.16:0/2762693030 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f1ca41658b0 con 0x7f1ca40728b0 2024-03-31T15:04:36.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:36.520+0000 7f1ca1600640 1 -- 172.21.15.16:0/2762693030 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v30) ==== 74+0+12819 (secure 0 0 0) 0x7f1c94055c30 con 0x7f1ca40728b0 2024-03-31T15:04:36.516 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:04:36.516 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":30,"fsid":"286eb5da-fb47-4550-b290-d9519c13b9a5","created":"2024-03-31T15:03:11.438952+0000","modified":"2024-03-31T15:04:33.345647+0000","last_up_change":"2024-03-31T15:03:22.885535+0000","last_in_change":"2024-03-31T15:03:13.941667+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-31T15:03:22.550787+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"92a66450-ef19-40d8-ac3d-d760b23b0bda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6801","nonce":932712244}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6803","nonce":932712244}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6807","nonce":932712244}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6805","nonce":932712244}]},"public_addr":"172.21.15.16:6801/932712244","cluster_addr":"172.21.15.16:6803/932712244","heartbeat_back_addr":"172.21.15.16:6807/932712244","heartbeat_front_addr":"172.21.15.16:6805/932712244","state":["exists","up"]},{"osd":1,"uuid":"da9d1f2b-1968-4959-b843-faa231988e92","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6810","nonce":1398737533}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6811","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6813","nonce":1398737533}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6821","nonce":1398737533}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6815","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6816","nonce":1398737533}]},"public_addr":"172.21.15.16:6810/1398737533","cluster_addr":"172.21.15.16:6813/1398737533","heartbeat_back_addr":"172.21.15.16:6821/1398737533","heartbeat_front_addr":"172.21.15.16:6816/1398737533","state":["exists","up"]},{"osd":2,"uuid":"01a73662-9a21-4b1f-bcbb-42b9636a4638","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6819","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6822","nonce":3018526646}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6825","nonce":3018526646}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6831","nonce":3018526646}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6827","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6829","nonce":3018526646}]},"public_addr":"172.21.15.16:6822/3018526646","cluster_addr":"172.21.15.16:6825/3018526646","heartbeat_back_addr":"172.21.15.16:6831/3018526646","heartbeat_front_addr":"172.21.15.16:6829/3018526646","state":["exists","up"]},{"osd":3,"uuid":"c7771fd3-e18e-4500-ac38-23cf10ebcd75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6809","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6812","nonce":3360352974}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6817","nonce":3360352974}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6828","nonce":3360352974}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6823","nonce":3360352974}]},"public_addr":"172.21.15.16:6812/3360352974","cluster_addr":"172.21.15.16:6817/3360352974","heartbeat_back_addr":"172.21.15.16:6828/3360352974","heartbeat_front_addr":"172.21.15.16:6823/3360352974","state":["exists","up"]},{"osd":4,"uuid":"88260730-1482-4fbd-a40f-cb1124e9df4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6801","nonce":1856844856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6803","nonce":1856844856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6807","nonce":1856844856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6805","nonce":1856844856}]},"public_addr":"172.21.15.89:6801/1856844856","cluster_addr":"172.21.15.89:6803/1856844856","heartbeat_back_addr":"172.21.15.89:6807/1856844856","heartbeat_front_addr":"172.21.15.89:6805/1856844856","state":["exists","up"]},{"osd":5,"uuid":"4cb2c00c-fae0-4dd8-9d27-ec4f0093ccf7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6814","nonce":50395527}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6817","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6820","nonce":50395527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6831","nonce":50395527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6823","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6826","nonce":50395527}]},"public_addr":"172.21.15.89:6814/50395527","cluster_addr":"172.21.15.89:6820/50395527","heartbeat_back_addr":"172.21.15.89:6831/50395527","heartbeat_front_addr":"172.21.15.89:6826/50395527","state":["exists","up"]},{"osd":6,"uuid":"81ce0c19-2554-4cae-acbe-66a05e9e9b9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6809","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6812","nonce":3470471752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6815","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6818","nonce":3470471752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6827","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6829","nonce":3470471752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6821","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6824","nonce":3470471752}]},"public_addr":"172.21.15.89:6812/3470471752","cluster_addr":"172.21.15.89:6818/3470471752","heartbeat_back_addr":"172.21.15.89:6829/3470471752","heartbeat_front_addr":"172.21.15.89:6824/3470471752","state":["exists","up"]},{"osd":7,"uuid":"85e4313b-d37d-442e-8042-153bf20a847d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6811","nonce":1796778630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6813","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6816","nonce":1796778630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6825","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6828","nonce":1796778630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6819","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6822","nonce":1796778630}]},"public_addr":"172.21.15.89:6811/1796778630","cluster_addr":"172.21.15.89:6816/1796778630","heartbeat_back_addr":"172.21.15.89:6828/1796778630","heartbeat_front_addr":"172.21.15.89:6822/1796778630","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.439088+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.529970+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.680512+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:21.044161+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.582111+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.536558+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.409522+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.590229+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-31T15:04:36.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:36.524+0000 7f1caba00640 1 -- 172.21.15.16:0/2762693030 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f1c80068380 msgr2=0x7f1c8006a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:36.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:36.524+0000 7f1caba00640 1 --2- 172.21.15.16:0/2762693030 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f1c80068380 0x7f1c8006a840 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f1c8c0ad000 tx=0x7f1c8c008040 comp rx=0 tx=0).stop 2024-03-31T15:04:36.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:36.524+0000 7f1caba00640 1 -- 172.21.15.16:0/2762693030 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1ca40728b0 msgr2=0x7f1ca413d960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:36.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:36.524+0000 7f1caba00640 1 --2- 172.21.15.16:0/2762693030 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1ca40728b0 0x7f1ca413d960 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f1c9400c3f0 tx=0x7f1c9400c8b0 comp rx=0 tx=0).stop 2024-03-31T15:04:36.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:36.524+0000 7f1caba00640 1 -- 172.21.15.16:0/2762693030 shutdown_connections 2024-03-31T15:04:36.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:36.524+0000 7f1caba00640 1 --2- 172.21.15.16:0/2762693030 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f1c80068380 0x7f1c8006a840 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:36.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:36.524+0000 7f1caba00640 1 --2- 172.21.15.16:0/2762693030 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f1ca4145d10 0x7f1ca415b000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:36.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:36.524+0000 7f1caba00640 1 --2- 172.21.15.16:0/2762693030 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1ca40b2200 0x7f1ca415aac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:36.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:36.524+0000 7f1caba00640 1 --2- 172.21.15.16:0/2762693030 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1ca40728b0 0x7f1ca413d960 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:36.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:36.524+0000 7f1caba00640 1 -- 172.21.15.16:0/2762693030 >> 172.21.15.16:0/2762693030 conn(0x7f1ca406b620 msgr2=0x7f1ca4165e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:04:36.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:36.524+0000 7f1caba00640 1 -- 172.21.15.16:0/2762693030 shutdown_connections 2024-03-31T15:04:36.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:36.524+0000 7f1caba00640 1 -- 172.21.15.16:0/2762693030 wait complete. 2024-03-31T15:04:36.537 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-31T15:03:22.550787+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-31T15:04:36.538 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-31T15:04:36.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:36.688+0000 7f0abc800640 1 Processor -- start 2024-03-31T15:04:36.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:36.688+0000 7f0abc800640 1 -- start start 2024-03-31T15:04:36.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:36.688+0000 7f0abc800640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f0ab413dff0 0x7f0ab406d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:36.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:36.688+0000 7f0abc800640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0ab415fbc0 con 0x7f0ab40cd0e0 2024-03-31T15:04:36.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:36.688+0000 7f0abc800640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0ab415ee00 con 0x7f0ab413e3d0 2024-03-31T15:04:36.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:36.688+0000 7f0abc800640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f0ab415f730 con 0x7f0ab413dff0 2024-03-31T15:04:36.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:36.688+0000 7f0ab9e00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f0ab413dff0 0x7f0ab406d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:36.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:36.688+0000 7f0aba800640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f0ab40cd0e0 0x7f0ab40ceb70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:33786/0 (socket says 172.21.15.16:33786) 2024-03-31T15:04:36.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:36.688+0000 7f0aba800640 1 -- 172.21.15.16:0/138223802 learned_addr learned my addr 172.21.15.16:0/138223802 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:04:36.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:36.688+0000 7f0ab9e00640 1 -- 172.21.15.16:0/138223802 >> v1:172.21.15.89:6789/0 conn(0x7f0ab413e3d0 legacy=0x7f0ab40cc9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:36.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:36.688+0000 7f0ab9e00640 1 -- 172.21.15.16:0/138223802 >> v1:172.21.15.16:6789/0 conn(0x7f0ab40cd0e0 legacy=0x7f0ab40ceb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:36.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:36.688+0000 7f0ab9e00640 1 -- 172.21.15.16:0/138223802 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ab415f0e0 con 0x7f0ab413dff0 2024-03-31T15:04:36.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:36.688+0000 7f0ab8a00640 1 -- 172.21.15.16:0/138223802 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 119704971 0 0) 0x7f0ab415fbc0 con 0x7f0ab40cd0e0 2024-03-31T15:04:36.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:36.692+0000 7f0ab9e00640 1 --2- 172.21.15.16:0/138223802 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0ab413dff0 0x7f0ab406d4d0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f0aa80097c0 tx=0x7f0aa801c410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c91cd107e156949 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:36.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:36.692+0000 7f0ab8a00640 1 -- 172.21.15.16:0/138223802 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0aa80b0070 con 0x7f0ab413dff0 2024-03-31T15:04:36.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:36.692+0000 7f0ab8a00640 1 -- 172.21.15.16:0/138223802 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0aa80ab600 con 0x7f0ab413dff0 2024-03-31T15:04:36.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:36.692+0000 7f0ab8a00640 1 -- 172.21.15.16:0/138223802 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0aa80ab9d0 con 0x7f0ab413dff0 2024-03-31T15:04:36.686 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-31T15:04:36.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:36.692+0000 7f0abc800640 1 -- 172.21.15.16:0/138223802 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0ab413dff0 msgr2=0x7f0ab406d4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:36.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:36.692+0000 7f0abc800640 1 --2- 172.21.15.16:0/138223802 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0ab413dff0 0x7f0ab406d4d0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f0aa80097c0 tx=0x7f0aa801c410 comp rx=0 tx=0).stop 2024-03-31T15:04:36.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:36.692+0000 7f0abc800640 1 -- 172.21.15.16:0/138223802 shutdown_connections 2024-03-31T15:04:36.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:36.692+0000 7f0abc800640 1 --2- 172.21.15.16:0/138223802 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0ab413dff0 0x7f0ab406d4d0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:36.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:36.692+0000 7f0abc800640 1 -- 172.21.15.16:0/138223802 >> 172.21.15.16:0/138223802 conn(0x7f0ab406ee40 msgr2=0x7f0ab406b620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:04:36.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:36.692+0000 7f0abc800640 1 -- 172.21.15.16:0/138223802 shutdown_connections 2024-03-31T15:04:36.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:36.692+0000 7f0abc800640 1 -- 172.21.15.16:0/138223802 wait complete. 2024-03-31T15:04:36.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:36.692+0000 7f0abc800640 1 Processor -- start 2024-03-31T15:04:36.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:36.692+0000 7f0abc800640 1 -- start start 2024-03-31T15:04:36.691 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-31T15:04:36.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:36.696+0000 7f0abc800640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0ab40cd0e0 0x7f0ab41533d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:36.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:36.696+0000 7f0abc800640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f0ab413dff0 0x7f0ab4159600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:36.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:36.696+0000 7f0abc800640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0ab413e3d0 0x7f0ab415ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:36.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:36.696+0000 7f0abc800640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f0ab4169080 con 0x7f0ab413e3d0 2024-03-31T15:04:36.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:36.696+0000 7f0abc800640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f0ab40cfca0 con 0x7f0ab40cd0e0 2024-03-31T15:04:36.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:36.696+0000 7f0abc800640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f0ab4169200 con 0x7f0ab413dff0 2024-03-31T15:04:36.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:36.696+0000 7f0aba800640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0ab413e3d0 0x7f0ab415ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:36.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:36.696+0000 7f0aba800640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0ab413e3d0 0x7f0ab415ddb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:44002/0 (socket says 172.21.15.16:44002) 2024-03-31T15:04:36.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:36.696+0000 7f0aba800640 1 -- 172.21.15.16:0/1861348568 learned_addr learned my addr 172.21.15.16:0/1861348568 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:04:36.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:36.696+0000 7f0ab9e00640 1 --2- 172.21.15.16:0/1861348568 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0ab40cd0e0 0x7f0ab41533d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:36.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:36.696+0000 7f0ab9400640 1 --2- 172.21.15.16:0/1861348568 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f0ab413dff0 0x7f0ab4159600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:36.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:36.696+0000 7f0aba800640 1 -- 172.21.15.16:0/1861348568 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f0ab413dff0 msgr2=0x7f0ab4159600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:36.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:36.696+0000 7f0aba800640 1 --2- 172.21.15.16:0/1861348568 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f0ab413dff0 0x7f0ab4159600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:36.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:36.696+0000 7f0aba800640 1 -- 172.21.15.16:0/1861348568 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0ab40cd0e0 msgr2=0x7f0ab41533d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:36.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:36.696+0000 7f0aba800640 1 --2- 172.21.15.16:0/1861348568 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0ab40cd0e0 0x7f0ab41533d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:36.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:36.696+0000 7f0aba800640 1 -- 172.21.15.16:0/1861348568 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ab4159b40 con 0x7f0ab413e3d0 2024-03-31T15:04:36.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:36.696+0000 7f0aba800640 1 --2- 172.21.15.16:0/1861348568 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0ab413e3d0 0x7f0ab415ddb0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f0ab000c7a0 tx=0x7f0ab000cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:36.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:36.696+0000 7f0aae000640 1 -- 172.21.15.16:0/1861348568 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ab0018070 con 0x7f0ab413e3d0 2024-03-31T15:04:36.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:36.696+0000 7f0aae000640 1 -- 172.21.15.16:0/1861348568 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0ab000a640 con 0x7f0ab413e3d0 2024-03-31T15:04:36.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:36.696+0000 7f0abc800640 1 -- 172.21.15.16:0/1861348568 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0ab4152330 con 0x7f0ab413e3d0 2024-03-31T15:04:36.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:36.696+0000 7f0aae000640 1 -- 172.21.15.16:0/1861348568 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ab000aab0 con 0x7f0ab413e3d0 2024-03-31T15:04:36.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:36.696+0000 7f0abc800640 1 -- 172.21.15.16:0/1861348568 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0ab41528f0 con 0x7f0ab413e3d0 2024-03-31T15:04:36.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:36.696+0000 7f0abc800640 1 -- 172.21.15.16:0/1861348568 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0ab4069950 con 0x7f0ab413e3d0 2024-03-31T15:04:36.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:36.700+0000 7f0aae000640 1 -- 172.21.15.16:0/1861348568 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f0ab001b070 con 0x7f0ab413e3d0 2024-03-31T15:04:36.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:36.700+0000 7f0aae000640 1 --2- 172.21.15.16:0/1861348568 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f0a98068380 0x7f0a9806a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:36.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:36.700+0000 7f0aae000640 1 -- 172.21.15.16:0/1861348568 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 4857+0+0 (secure 0 0 0) 0x7f0ab0085c00 con 0x7f0ab413e3d0 2024-03-31T15:04:36.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:36.700+0000 7f0ab9e00640 1 --2- 172.21.15.16:0/1861348568 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f0a98068380 0x7f0a9806a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:36.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:36.700+0000 7f0ab9e00640 1 --2- 172.21.15.16:0/1861348568 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f0a98068380 0x7f0a9806a840 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f0aa80ad000 tx=0x7f0aa80a6000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:36.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:36.704+0000 7f0aae000640 1 -- 172.21.15.16:0/1861348568 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f0ab0055e20 con 0x7f0ab413e3d0 2024-03-31T15:04:36.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:36.868+0000 7f0abc800640 1 -- 172.21.15.16:0/1861348568 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f0ab40b28f0 con 0x7f0ab413e3d0 2024-03-31T15:04:36.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:36.868+0000 7f0aae000640 1 -- 172.21.15.16:0/1861348568 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v30) ==== 93+0+10 (secure 0 0 0) 0x7f0ab005a120 con 0x7f0ab413e3d0 2024-03-31T15:04:36.863 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2024-03-31T15:04:36.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:36.872+0000 7f0abc800640 1 -- 172.21.15.16:0/1861348568 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f0a98068380 msgr2=0x7f0a9806a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:36.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:36.872+0000 7f0abc800640 1 --2- 172.21.15.16:0/1861348568 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f0a98068380 0x7f0a9806a840 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f0aa80ad000 tx=0x7f0aa80a6000 comp rx=0 tx=0).stop 2024-03-31T15:04:36.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:36.872+0000 7f0abc800640 1 -- 172.21.15.16:0/1861348568 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0ab413e3d0 msgr2=0x7f0ab415ddb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:36.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:36.872+0000 7f0abc800640 1 --2- 172.21.15.16:0/1861348568 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0ab413e3d0 0x7f0ab415ddb0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f0ab000c7a0 tx=0x7f0ab000cc60 comp rx=0 tx=0).stop 2024-03-31T15:04:36.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:36.872+0000 7f0abc800640 1 -- 172.21.15.16:0/1861348568 shutdown_connections 2024-03-31T15:04:36.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:36.872+0000 7f0abc800640 1 --2- 172.21.15.16:0/1861348568 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f0a98068380 0x7f0a9806a840 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:36.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:36.872+0000 7f0abc800640 1 --2- 172.21.15.16:0/1861348568 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0ab413e3d0 0x7f0ab415ddb0 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:36.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:36.872+0000 7f0abc800640 1 --2- 172.21.15.16:0/1861348568 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f0ab413dff0 0x7f0ab4159600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:36.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:36.872+0000 7f0abc800640 1 --2- 172.21.15.16:0/1861348568 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0ab40cd0e0 0x7f0ab41533d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:36.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:36.872+0000 7f0abc800640 1 -- 172.21.15.16:0/1861348568 >> 172.21.15.16:0/1861348568 conn(0x7f0ab406ee40 msgr2=0x7f0ab415bfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:04:36.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:36.872+0000 7f0abc800640 1 -- 172.21.15.16:0/1861348568 shutdown_connections 2024-03-31T15:04:36.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:36.872+0000 7f0abc800640 1 -- 172.21.15.16:0/1861348568 wait complete. 2024-03-31T15:04:36.885 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-31T15:04:37.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:37.028+0000 7ff5a2000640 1 Processor -- start 2024-03-31T15:04:37.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:37.032+0000 7ff5a2000640 1 -- start start 2024-03-31T15:04:37.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:37.032+0000 7ff5a2000640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff59c070030 0x7ff59c06c730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:37.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:37.032+0000 7ff5a2000640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff59c147660 con 0x7ff59c147dd0 2024-03-31T15:04:37.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:37.032+0000 7ff5a2000640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff59c162340 con 0x7ff59c147280 2024-03-31T15:04:37.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:37.032+0000 7ff5a2000640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7ff59c1620f0 con 0x7ff59c070030 2024-03-31T15:04:37.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:37.032+0000 7ff592a00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff59c070030 0x7ff59c06c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:37.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:37.032+0000 7ff59be00640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7ff59c147dd0 0x7ff59c0cc8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:33792/0 (socket says 172.21.15.16:33792) 2024-03-31T15:04:37.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:37.032+0000 7ff59be00640 1 -- 172.21.15.16:0/1985180158 learned_addr learned my addr 172.21.15.16:0/1985180158 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:04:37.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:37.032+0000 7ff592a00640 1 -- 172.21.15.16:0/1985180158 >> v1:172.21.15.89:6789/0 conn(0x7ff59c147280 legacy=0x7ff59c06f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:37.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:37.032+0000 7ff592a00640 1 -- 172.21.15.16:0/1985180158 >> v1:172.21.15.16:6789/0 conn(0x7ff59c147dd0 legacy=0x7ff59c0cc8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:37.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:37.032+0000 7ff592a00640 1 -- 172.21.15.16:0/1985180158 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff59c162620 con 0x7ff59c070030 2024-03-31T15:04:37.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:37.036+0000 7ff592a00640 1 --2- 172.21.15.16:0/1985180158 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff59c070030 0x7ff59c06c730 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7ff5840097c0 tx=0x7ff58401c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2fbe1a15944d3efc server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:37.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:37.036+0000 7ff59aa00640 1 -- 172.21.15.16:0/1985180158 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5840b1070 con 0x7ff59c070030 2024-03-31T15:04:37.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:37.036+0000 7ff59aa00640 1 -- 172.21.15.16:0/1985180158 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff5840ac410 con 0x7ff59c070030 2024-03-31T15:04:37.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:37.036+0000 7ff59aa00640 1 -- 172.21.15.16:0/1985180158 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5840ac7e0 con 0x7ff59c070030 2024-03-31T15:04:37.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:37.036+0000 7ff5a2000640 1 -- 172.21.15.16:0/1985180158 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff59c070030 msgr2=0x7ff59c06c730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:37.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:37.036+0000 7ff5a2000640 1 --2- 172.21.15.16:0/1985180158 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff59c070030 0x7ff59c06c730 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7ff5840097c0 tx=0x7ff58401c4a0 comp rx=0 tx=0).stop 2024-03-31T15:04:37.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:37.036+0000 7ff5a2000640 1 -- 172.21.15.16:0/1985180158 shutdown_connections 2024-03-31T15:04:37.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:37.036+0000 7ff5a2000640 1 --2- 172.21.15.16:0/1985180158 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff59c070030 0x7ff59c06c730 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:37.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:37.036+0000 7ff5a2000640 1 -- 172.21.15.16:0/1985180158 >> 172.21.15.16:0/1985180158 conn(0x7ff59c067330 msgr2=0x7ff59c067740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:04:37.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:37.036+0000 7ff5a2000640 1 -- 172.21.15.16:0/1985180158 shutdown_connections 2024-03-31T15:04:37.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:37.036+0000 7ff5a2000640 1 -- 172.21.15.16:0/1985180158 wait complete. 2024-03-31T15:04:37.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:37.036+0000 7ff5a2000640 1 Processor -- start 2024-03-31T15:04:37.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:37.036+0000 7ff5a2000640 1 -- start start 2024-03-31T15:04:37.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:37.040+0000 7ff5a2000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff59c070030 0x7ff59c13f430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:37.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:37.040+0000 7ff5a2000640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff59c147280 0x7ff59c15a350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:37.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:37.040+0000 7ff5a2000640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff59c147dd0 0x7ff59c15a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:37.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:37.040+0000 7ff5a2000640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7ff59c0cdc20 con 0x7ff59c070030 2024-03-31T15:04:37.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:37.040+0000 7ff5a2000640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7ff59c0cdaa0 con 0x7ff59c147dd0 2024-03-31T15:04:37.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:37.040+0000 7ff5a2000640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7ff59c16c4a0 con 0x7ff59c147280 2024-03-31T15:04:37.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:37.040+0000 7ff59b400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff59c070030 0x7ff59c13f430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:37.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:37.040+0000 7ff59b400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff59c070030 0x7ff59c13f430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54112/0 (socket says 172.21.15.16:54112) 2024-03-31T15:04:37.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:37.040+0000 7ff59b400640 1 -- 172.21.15.16:0/3412605834 learned_addr learned my addr 172.21.15.16:0/3412605834 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:04:37.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:37.040+0000 7ff592a00640 1 --2- 172.21.15.16:0/3412605834 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff59c147280 0x7ff59c15a350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:37.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:37.040+0000 7ff59be00640 1 --2- 172.21.15.16:0/3412605834 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff59c147dd0 0x7ff59c15a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:37.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:37.040+0000 7ff59b400640 1 -- 172.21.15.16:0/3412605834 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff59c147280 msgr2=0x7ff59c15a350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:37.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:37.040+0000 7ff59b400640 1 --2- 172.21.15.16:0/3412605834 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff59c147280 0x7ff59c15a350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:37.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:37.040+0000 7ff59b400640 1 -- 172.21.15.16:0/3412605834 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff59c147dd0 msgr2=0x7ff59c15a890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:37.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:37.040+0000 7ff59b400640 1 --2- 172.21.15.16:0/3412605834 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff59c147dd0 0x7ff59c15a890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:37.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:37.040+0000 7ff59b400640 1 -- 172.21.15.16:0/3412605834 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff59c157e90 con 0x7ff59c070030 2024-03-31T15:04:37.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:37.040+0000 7ff59be00640 1 --2- 172.21.15.16:0/3412605834 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff59c147dd0 0x7ff59c15a890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:04:37.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:37.040+0000 7ff59b400640 1 --2- 172.21.15.16:0/3412605834 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff59c070030 0x7ff59c13f430 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7ff58c00c530 tx=0x7ff58c00c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:37.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:37.040+0000 7ff593e00640 1 -- 172.21.15.16:0/3412605834 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff58c016020 con 0x7ff59c070030 2024-03-31T15:04:37.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:37.040+0000 7ff593e00640 1 -- 172.21.15.16:0/3412605834 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff58c005220 con 0x7ff59c070030 2024-03-31T15:04:37.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:37.040+0000 7ff593e00640 1 -- 172.21.15.16:0/3412605834 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff58c007030 con 0x7ff59c070030 2024-03-31T15:04:37.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:37.040+0000 7ff5a2000640 1 -- 172.21.15.16:0/3412605834 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff59c141a30 con 0x7ff59c070030 2024-03-31T15:04:37.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:37.040+0000 7ff5a2000640 1 -- 172.21.15.16:0/3412605834 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff59c13f970 con 0x7ff59c070030 2024-03-31T15:04:37.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:37.040+0000 7ff5a2000640 1 -- 172.21.15.16:0/3412605834 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff59c069950 con 0x7ff59c070030 2024-03-31T15:04:37.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:37.040+0000 7ff593e00640 1 -- 172.21.15.16:0/3412605834 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7ff58c0071d0 con 0x7ff59c070030 2024-03-31T15:04:37.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:37.044+0000 7ff593e00640 1 --2- 172.21.15.16:0/3412605834 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7ff578068380 0x7ff57806a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:37.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:37.044+0000 7ff593e00640 1 -- 172.21.15.16:0/3412605834 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 4857+0+0 (secure 0 0 0) 0x7ff58c085f00 con 0x7ff59c070030 2024-03-31T15:04:37.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:37.044+0000 7ff592a00640 1 --2- 172.21.15.16:0/3412605834 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7ff578068380 0x7ff57806a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:37.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:37.044+0000 7ff592a00640 1 --2- 172.21.15.16:0/3412605834 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7ff578068380 0x7ff57806a840 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7ff5840ae000 tx=0x7ff584008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:37.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:37.048+0000 7ff593e00640 1 -- 172.21.15.16:0/3412605834 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7ff58c020020 con 0x7ff59c070030 2024-03-31T15:04:37.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:37.216+0000 7ff5a2000640 1 -- 172.21.15.16:0/3412605834 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7ff59c0d2880 con 0x7ff59c070030 2024-03-31T15:04:37.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:37.216+0000 7ff593e00640 1 -- 172.21.15.16:0/3412605834 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v30) ==== 84+0+65 (secure 0 0 0) 0x7ff58c0074c0 con 0x7ff59c070030 2024-03-31T15:04:37.211 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:04:37.211 INFO:teuthology.orchestra.run.smithi016.stdout:[ 2024-03-31T15:04:37.212 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-31T15:04:37.212 INFO:teuthology.orchestra.run.smithi016.stdout: "poolnum": 1, 2024-03-31T15:04:37.212 INFO:teuthology.orchestra.run.smithi016.stdout: "poolname": ".mgr" 2024-03-31T15:04:37.212 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-31T15:04:37.212 INFO:teuthology.orchestra.run.smithi016.stdout:] 2024-03-31T15:04:37.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:37.220+0000 7ff5a2000640 1 -- 172.21.15.16:0/3412605834 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7ff578068380 msgr2=0x7ff57806a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:37.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:37.220+0000 7ff5a2000640 1 --2- 172.21.15.16:0/3412605834 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7ff578068380 0x7ff57806a840 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7ff5840ae000 tx=0x7ff584008040 comp rx=0 tx=0).stop 2024-03-31T15:04:37.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:37.220+0000 7ff5a2000640 1 -- 172.21.15.16:0/3412605834 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff59c070030 msgr2=0x7ff59c13f430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:37.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:37.220+0000 7ff5a2000640 1 --2- 172.21.15.16:0/3412605834 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff59c070030 0x7ff59c13f430 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7ff58c00c530 tx=0x7ff58c00c9f0 comp rx=0 tx=0).stop 2024-03-31T15:04:37.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:37.220+0000 7ff5a2000640 1 -- 172.21.15.16:0/3412605834 shutdown_connections 2024-03-31T15:04:37.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:37.220+0000 7ff5a2000640 1 --2- 172.21.15.16:0/3412605834 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7ff578068380 0x7ff57806a840 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:37.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:37.220+0000 7ff5a2000640 1 --2- 172.21.15.16:0/3412605834 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff59c147dd0 0x7ff59c15a890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:37.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:37.220+0000 7ff5a2000640 1 --2- 172.21.15.16:0/3412605834 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff59c147280 0x7ff59c15a350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:37.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:37.220+0000 7ff5a2000640 1 --2- 172.21.15.16:0/3412605834 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff59c070030 0x7ff59c13f430 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:37.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:37.220+0000 7ff5a2000640 1 -- 172.21.15.16:0/3412605834 >> 172.21.15.16:0/3412605834 conn(0x7ff59c067330 msgr2=0x7ff59c143010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:04:37.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:37.220+0000 7ff5a2000640 1 -- 172.21.15.16:0/3412605834 shutdown_connections 2024-03-31T15:04:37.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:37.220+0000 7ff5a2000640 1 -- 172.21.15.16:0/3412605834 wait complete. 2024-03-31T15:04:37.232 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-31T15:04:37.232 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-31T15:04:37.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:37.384+0000 7f902b600640 1 Processor -- start 2024-03-31T15:04:37.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:37.384+0000 7f902b600640 1 -- start start 2024-03-31T15:04:37.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:37.384+0000 7f902b600640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f9024070490 0x7f90240708f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:37.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:37.384+0000 7f902b600640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f902406b620 0x7f902406bad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:37.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:37.384+0000 7f902b600640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f90240d0a30 con 0x7f9024145ca0 2024-03-31T15:04:37.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:37.384+0000 7f902b600640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f902406fa40 con 0x7f9024070490 2024-03-31T15:04:37.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:37.384+0000 7f902b600640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f90240d0d10 con 0x7f902406b620 2024-03-31T15:04:37.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:37.384+0000 7f9028c00640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f9024145ca0 0x7f90240725a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:38762/0 (socket says 172.21.15.16:38762) 2024-03-31T15:04:37.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:37.384+0000 7f9028c00640 1 -- 172.21.15.16:0/4254394348 learned_addr learned my addr 172.21.15.16:0/4254394348 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:04:37.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:37.384+0000 7f901be00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f9024070490 0x7f90240708f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:37.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:37.384+0000 7f9029600640 1 --2- 172.21.15.16:0/4254394348 >> v2:172.21.15.89:3300/0 conn(0x7f902406b620 0x7f902406bad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:37.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:37.384+0000 7f901be00640 1 -- 172.21.15.16:0/4254394348 >> v2:172.21.15.89:3300/0 conn(0x7f902406b620 msgr2=0x7f902406bad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:04:37.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:37.384+0000 7f901be00640 1 --2- 172.21.15.16:0/4254394348 >> v2:172.21.15.89:3300/0 conn(0x7f902406b620 0x7f902406bad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:37.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:37.384+0000 7f901be00640 1 -- 172.21.15.16:0/4254394348 >> v1:172.21.15.16:6789/0 conn(0x7f9024145ca0 legacy=0x7f90240725a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:37.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:37.384+0000 7f901be00640 1 -- 172.21.15.16:0/4254394348 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f902406fc90 con 0x7f9024070490 2024-03-31T15:04:37.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:37.384+0000 7f9023e00640 1 -- 172.21.15.16:0/4254394348 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4119071989 0 0) 0x7f90240d0a30 con 0x7f9024145ca0 2024-03-31T15:04:37.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:37.388+0000 7f901be00640 1 --2- 172.21.15.16:0/4254394348 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9024070490 0x7f90240708f0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f90240b0cf0 tx=0x7f900c01c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a98d3fa24a73551 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:37.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:37.388+0000 7f9023e00640 1 -- 172.21.15.16:0/4254394348 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f900c0b0070 con 0x7f9024070490 2024-03-31T15:04:37.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:37.388+0000 7f9023e00640 1 -- 172.21.15.16:0/4254394348 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f900c0ab600 con 0x7f9024070490 2024-03-31T15:04:37.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:37.388+0000 7f9023e00640 1 -- 172.21.15.16:0/4254394348 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f900c0ab9d0 con 0x7f9024070490 2024-03-31T15:04:37.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:37.388+0000 7f902b600640 1 -- 172.21.15.16:0/4254394348 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9024070490 msgr2=0x7f90240708f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:37.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:37.388+0000 7f902b600640 1 --2- 172.21.15.16:0/4254394348 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9024070490 0x7f90240708f0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f90240b0cf0 tx=0x7f900c01c480 comp rx=0 tx=0).stop 2024-03-31T15:04:37.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:37.388+0000 7f902b600640 1 -- 172.21.15.16:0/4254394348 shutdown_connections 2024-03-31T15:04:37.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:37.388+0000 7f902b600640 1 --2- 172.21.15.16:0/4254394348 >> v2:172.21.15.89:3300/0 conn(0x7f902406b620 0x7f902406bad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:37.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:37.388+0000 7f902b600640 1 --2- 172.21.15.16:0/4254394348 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9024070490 0x7f90240708f0 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:37.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:37.388+0000 7f902b600640 1 -- 172.21.15.16:0/4254394348 >> 172.21.15.16:0/4254394348 conn(0x7f90240b2dd0 msgr2=0x7f90240b31e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:04:37.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:37.388+0000 7f902b600640 1 -- 172.21.15.16:0/4254394348 shutdown_connections 2024-03-31T15:04:37.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:37.388+0000 7f902b600640 1 -- 172.21.15.16:0/4254394348 wait complete. 2024-03-31T15:04:37.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:37.392+0000 7f902b600640 1 Processor -- start 2024-03-31T15:04:37.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:37.392+0000 7f902b600640 1 -- start start 2024-03-31T15:04:37.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:37.392+0000 7f902b600640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f902406b620 0x7f9024156b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:37.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:37.392+0000 7f902b600640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f9024070490 0x7f90240b1d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:37.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:37.392+0000 7f902b600640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9024145ca0 0x7f90240b22d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:37.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:37.392+0000 7f902b600640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f90240d07c0 con 0x7f9024145ca0 2024-03-31T15:04:37.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:37.392+0000 7f902b600640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f90240d0640 con 0x7f902406b620 2024-03-31T15:04:37.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:37.392+0000 7f902b600640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f90241690c0 con 0x7f9024070490 2024-03-31T15:04:37.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:37.392+0000 7f9029600640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9024145ca0 0x7f90240b22d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:37.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:37.392+0000 7f9029600640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9024145ca0 0x7f90240b22d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54132/0 (socket says 172.21.15.16:54132) 2024-03-31T15:04:37.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:37.392+0000 7f9029600640 1 -- 172.21.15.16:0/4242532537 learned_addr learned my addr 172.21.15.16:0/4242532537 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:04:37.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:37.392+0000 7f9028c00640 1 --2- 172.21.15.16:0/4242532537 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f902406b620 0x7f9024156b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:37.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:37.392+0000 7f901be00640 1 --2- 172.21.15.16:0/4242532537 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f9024070490 0x7f90240b1d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:37.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:37.392+0000 7f9029600640 1 -- 172.21.15.16:0/4242532537 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f9024070490 msgr2=0x7f90240b1d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:37.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:37.392+0000 7f9029600640 1 --2- 172.21.15.16:0/4242532537 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f9024070490 0x7f90240b1d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:37.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:37.392+0000 7f9029600640 1 -- 172.21.15.16:0/4242532537 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f902406b620 msgr2=0x7f9024156b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:37.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:37.392+0000 7f9029600640 1 --2- 172.21.15.16:0/4242532537 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f902406b620 0x7f9024156b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:37.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:37.392+0000 7f9029600640 1 -- 172.21.15.16:0/4242532537 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f902415a070 con 0x7f9024145ca0 2024-03-31T15:04:37.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:37.392+0000 7f9029600640 1 --2- 172.21.15.16:0/4242532537 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9024145ca0 0x7f90240b22d0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f901c009d30 tx=0x7f901c00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:37.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:37.392+0000 7f9021600640 1 -- 172.21.15.16:0/4242532537 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f901c018020 con 0x7f9024145ca0 2024-03-31T15:04:37.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:37.392+0000 7f9021600640 1 -- 172.21.15.16:0/4242532537 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f901c014480 con 0x7f9024145ca0 2024-03-31T15:04:37.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:37.392+0000 7f9021600640 1 -- 172.21.15.16:0/4242532537 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f901c014870 con 0x7f9024145ca0 2024-03-31T15:04:37.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:37.392+0000 7f902b600640 1 -- 172.21.15.16:0/4242532537 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f902415a350 con 0x7f9024145ca0 2024-03-31T15:04:37.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:37.392+0000 7f902b600640 1 -- 172.21.15.16:0/4242532537 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f902415a910 con 0x7f9024145ca0 2024-03-31T15:04:37.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:37.392+0000 7f902b600640 1 -- 172.21.15.16:0/4242532537 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9024069950 con 0x7f9024145ca0 2024-03-31T15:04:37.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:37.396+0000 7f9021600640 1 -- 172.21.15.16:0/4242532537 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f901c01b020 con 0x7f9024145ca0 2024-03-31T15:04:37.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:37.396+0000 7f9021600640 1 --2- 172.21.15.16:0/4242532537 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f9000068380 0x7f900006a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:37.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:37.396+0000 7f9021600640 1 -- 172.21.15.16:0/4242532537 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 4857+0+0 (secure 0 0 0) 0x7f901c087240 con 0x7f9024145ca0 2024-03-31T15:04:37.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:37.396+0000 7f9028c00640 1 --2- 172.21.15.16:0/4242532537 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f9000068380 0x7f900006a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:37.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:37.396+0000 7f9028c00640 1 --2- 172.21.15.16:0/4242532537 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f9000068380 0x7f900006a840 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f9014000f70 tx=0x7f9014007b60 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:37.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:37.400+0000 7f9021600640 1 -- 172.21.15.16:0/4242532537 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f901c022070 con 0x7f9024145ca0 2024-03-31T15:04:37.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:37.564+0000 7f902b600640 1 -- 172.21.15.16:0/4242532537 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7f90240b3b10 con 0x7f9024145ca0 2024-03-31T15:04:38.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:38.360+0000 7f9021600640 1 -- 172.21.15.16:0/4242532537 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v31) ==== 138+0+0 (secure 0 0 0) 0x7f901c010040 con 0x7f9024145ca0 2024-03-31T15:04:38.356 INFO:teuthology.orchestra.run.smithi016.stderr:pool 'cephfs_metadata' created 2024-03-31T15:04:38.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:38.364+0000 7f902b600640 1 -- 172.21.15.16:0/4242532537 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f9000068380 msgr2=0x7f900006a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:38.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:38.364+0000 7f902b600640 1 --2- 172.21.15.16:0/4242532537 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f9000068380 0x7f900006a840 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f9014000f70 tx=0x7f9014007b60 comp rx=0 tx=0).stop 2024-03-31T15:04:38.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:38.364+0000 7f902b600640 1 -- 172.21.15.16:0/4242532537 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9024145ca0 msgr2=0x7f90240b22d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:38.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:38.364+0000 7f902b600640 1 --2- 172.21.15.16:0/4242532537 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9024145ca0 0x7f90240b22d0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f901c009d30 tx=0x7f901c00c490 comp rx=0 tx=0).stop 2024-03-31T15:04:38.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:38.364+0000 7f902b600640 1 -- 172.21.15.16:0/4242532537 shutdown_connections 2024-03-31T15:04:38.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:38.364+0000 7f902b600640 1 --2- 172.21.15.16:0/4242532537 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f9000068380 0x7f900006a840 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:38.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:38.364+0000 7f902b600640 1 --2- 172.21.15.16:0/4242532537 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9024145ca0 0x7f90240b22d0 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:38.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:38.364+0000 7f902b600640 1 --2- 172.21.15.16:0/4242532537 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f9024070490 0x7f90240b1d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:38.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:38.364+0000 7f902b600640 1 --2- 172.21.15.16:0/4242532537 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f902406b620 0x7f9024156b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:38.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:38.364+0000 7f902b600640 1 -- 172.21.15.16:0/4242532537 >> 172.21.15.16:0/4242532537 conn(0x7f90240b2dd0 msgr2=0x7f9024072ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:04:38.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:38.364+0000 7f902b600640 1 -- 172.21.15.16:0/4242532537 shutdown_connections 2024-03-31T15:04:38.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:38.364+0000 7f902b600640 1 -- 172.21.15.16:0/4242532537 wait complete. 2024-03-31T15:04:38.387 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-31T15:04:38.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:38.544+0000 7feeeae00640 1 Processor -- start 2024-03-31T15:04:38.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:38.544+0000 7feeeae00640 1 -- start start 2024-03-31T15:04:38.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:38.548+0000 7feeeae00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7feee413dff0 0x7feee406d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:38.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:38.548+0000 7feeeae00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7feee413e3d0 0x7feee406da10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:38.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:38.548+0000 7feeeae00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7feee40cc9d0 0x7feee40cce30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:38.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:38.548+0000 7feeeae00640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7feee415f730 con 0x7feee413dff0 2024-03-31T15:04:38.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:38.548+0000 7feeeae00640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7feee415fbc0 con 0x7feee40cc9d0 2024-03-31T15:04:38.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:38.548+0000 7feeeae00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7feee415ee00 con 0x7feee413e3d0 2024-03-31T15:04:38.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:38.548+0000 7feee3e00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7feee413dff0 0x7feee406d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:38.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:38.548+0000 7feee3e00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7feee413dff0 0x7feee406d4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54138/0 (socket says 172.21.15.16:54138) 2024-03-31T15:04:38.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:38.548+0000 7feee3e00640 1 -- 172.21.15.16:0/2525684024 learned_addr learned my addr 172.21.15.16:0/2525684024 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:04:38.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:38.548+0000 7feedbe00640 1 --2- 172.21.15.16:0/2525684024 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7feee413e3d0 0x7feee406da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:38.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:38.548+0000 7feee3e00640 1 -- 172.21.15.16:0/2525684024 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7feee413e3d0 msgr2=0x7feee406da10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:04:38.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:38.548+0000 7feee8e00640 1 --2- 172.21.15.16:0/2525684024 >> v2:172.21.15.89:3300/0 conn(0x7feee40cc9d0 0x7feee40cce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:38.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:38.548+0000 7feee3e00640 1 --2- 172.21.15.16:0/2525684024 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7feee413e3d0 0x7feee406da10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:38.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:38.548+0000 7feee3e00640 1 -- 172.21.15.16:0/2525684024 >> v2:172.21.15.89:3300/0 conn(0x7feee40cc9d0 msgr2=0x7feee40cce30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:04:38.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:38.548+0000 7feee3e00640 1 --2- 172.21.15.16:0/2525684024 >> v2:172.21.15.89:3300/0 conn(0x7feee40cc9d0 0x7feee40cce30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:38.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:38.548+0000 7feee3e00640 1 -- 172.21.15.16:0/2525684024 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feee415f0e0 con 0x7feee413dff0 2024-03-31T15:04:38.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:38.548+0000 7feedbe00640 1 --2- 172.21.15.16:0/2525684024 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7feee413e3d0 0x7feee406da10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:04:38.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:38.548+0000 7feee8e00640 1 --2- 172.21.15.16:0/2525684024 >> v2:172.21.15.89:3300/0 conn(0x7feee40cc9d0 0x7feee40cce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:04:38.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:38.552+0000 7feee3e00640 1 --2- 172.21.15.16:0/2525684024 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feee413dff0 0x7feee406d4d0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7feee40cc980 tx=0x7feed401cef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14e7fcc24166eed2 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:38.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:38.552+0000 7feee3400640 1 -- 172.21.15.16:0/2525684024 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feed40b2070 con 0x7feee413dff0 2024-03-31T15:04:38.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:38.552+0000 7feee3400640 1 -- 172.21.15.16:0/2525684024 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feed40a9070 con 0x7feee413dff0 2024-03-31T15:04:38.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:38.552+0000 7feee3400640 1 -- 172.21.15.16:0/2525684024 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feed40ad880 con 0x7feee413dff0 2024-03-31T15:04:38.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:38.552+0000 7feeeae00640 1 -- 172.21.15.16:0/2525684024 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feee413dff0 msgr2=0x7feee406d4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:38.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:38.552+0000 7feeeae00640 1 --2- 172.21.15.16:0/2525684024 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feee413dff0 0x7feee406d4d0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7feee40cc980 tx=0x7feed401cef0 comp rx=0 tx=0).stop 2024-03-31T15:04:38.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:38.552+0000 7feeeae00640 1 -- 172.21.15.16:0/2525684024 shutdown_connections 2024-03-31T15:04:38.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:38.552+0000 7feeeae00640 1 --2- 172.21.15.16:0/2525684024 >> v2:172.21.15.89:3300/0 conn(0x7feee40cc9d0 0x7feee40cce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:38.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:38.552+0000 7feeeae00640 1 --2- 172.21.15.16:0/2525684024 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7feee413e3d0 0x7feee406da10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:38.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:38.552+0000 7feeeae00640 1 --2- 172.21.15.16:0/2525684024 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feee413dff0 0x7feee406d4d0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:38.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:38.552+0000 7feeeae00640 1 -- 172.21.15.16:0/2525684024 >> 172.21.15.16:0/2525684024 conn(0x7feee406ee40 msgr2=0x7feee406b620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:04:38.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:38.552+0000 7feeeae00640 1 -- 172.21.15.16:0/2525684024 shutdown_connections 2024-03-31T15:04:38.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:38.552+0000 7feeeae00640 1 -- 172.21.15.16:0/2525684024 wait complete. 2024-03-31T15:04:38.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:38.552+0000 7feeeae00640 1 Processor -- start 2024-03-31T15:04:38.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:38.552+0000 7feeeae00640 1 -- start start 2024-03-31T15:04:38.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:38.556+0000 7feeeae00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feee40cc9d0 0x7feee40b5190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:38.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:38.556+0000 7feeeae00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7feee413dff0 0x7feee41f7ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:38.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:38.556+0000 7feeeae00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7feee413e3d0 0x7feee413bd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:38.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:38.556+0000 7feeeae00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7feee4073200 con 0x7feee40cc9d0 2024-03-31T15:04:38.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:38.556+0000 7feeeae00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7feee4073080 con 0x7feee413e3d0 2024-03-31T15:04:38.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:38.556+0000 7feeeae00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7feee4073380 con 0x7feee413dff0 2024-03-31T15:04:38.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:38.556+0000 7feee3e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feee40cc9d0 0x7feee40b5190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:38.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:38.556+0000 7feee3e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feee40cc9d0 0x7feee40b5190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54152/0 (socket says 172.21.15.16:54152) 2024-03-31T15:04:38.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:38.556+0000 7feee3e00640 1 -- 172.21.15.16:0/672506504 learned_addr learned my addr 172.21.15.16:0/672506504 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:04:38.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:38.556+0000 7feedbe00640 1 --2- 172.21.15.16:0/672506504 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7feee413dff0 0x7feee41f7ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:38.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:38.556+0000 7feee8e00640 1 --2- 172.21.15.16:0/672506504 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7feee413e3d0 0x7feee413bd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:38.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:38.556+0000 7feee3e00640 1 -- 172.21.15.16:0/672506504 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7feee413dff0 msgr2=0x7feee41f7ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:38.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:38.556+0000 7feee3e00640 1 --2- 172.21.15.16:0/672506504 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7feee413dff0 0x7feee41f7ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:38.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:38.556+0000 7feee3e00640 1 -- 172.21.15.16:0/672506504 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7feee413e3d0 msgr2=0x7feee413bd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:38.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:38.556+0000 7feee3e00640 1 --2- 172.21.15.16:0/672506504 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7feee413e3d0 0x7feee413bd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:38.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:38.556+0000 7feee3e00640 1 -- 172.21.15.16:0/672506504 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feee41f7240 con 0x7feee40cc9d0 2024-03-31T15:04:38.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:38.556+0000 7feee3e00640 1 --2- 172.21.15.16:0/672506504 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feee40cc9d0 0x7feee40b5190 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7feed40b0040 tx=0x7feed401acd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:38.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:38.556+0000 7feee0c00640 1 -- 172.21.15.16:0/672506504 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feed40bc020 con 0x7feee40cc9d0 2024-03-31T15:04:38.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:38.556+0000 7feee0c00640 1 -- 172.21.15.16:0/672506504 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feed40b8420 con 0x7feee40cc9d0 2024-03-31T15:04:38.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:38.556+0000 7feeeae00640 1 -- 172.21.15.16:0/672506504 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feee4156c30 con 0x7feee40cc9d0 2024-03-31T15:04:38.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:38.556+0000 7feee0c00640 1 -- 172.21.15.16:0/672506504 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feed40b2040 con 0x7feee40cc9d0 2024-03-31T15:04:38.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:38.556+0000 7feeeae00640 1 -- 172.21.15.16:0/672506504 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feee41f77c0 con 0x7feee40cc9d0 2024-03-31T15:04:38.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:38.556+0000 7feeeae00640 1 -- 172.21.15.16:0/672506504 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feee4069950 con 0x7feee40cc9d0 2024-03-31T15:04:38.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:38.556+0000 7feee0c00640 1 -- 172.21.15.16:0/672506504 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7feed40bf020 con 0x7feee40cc9d0 2024-03-31T15:04:38.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:38.560+0000 7feee0c00640 1 --2- 172.21.15.16:0/672506504 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7feec0068380 0x7feec006a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:38.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:38.560+0000 7feee0c00640 1 -- 172.21.15.16:0/672506504 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 5227+0+0 (secure 0 0 0) 0x7feed4121280 con 0x7feee40cc9d0 2024-03-31T15:04:38.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:38.560+0000 7feedbe00640 1 --2- 172.21.15.16:0/672506504 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7feec0068380 0x7feec006a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:38.554 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:38.560+0000 7feedbe00640 1 --2- 172.21.15.16:0/672506504 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7feec0068380 0x7feec006a840 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7feecc006fd0 tx=0x7feecc008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:38.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:38.564+0000 7feee0c00640 1 -- 172.21.15.16:0/672506504 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7feed40c6070 con 0x7feee40cc9d0 2024-03-31T15:04:38.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:38.728+0000 7feeeae00640 1 -- 172.21.15.16:0/672506504 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7feee41f8280 con 0x7feee40cc9d0 2024-03-31T15:04:39.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:39.368+0000 7feee0c00640 1 -- 172.21.15.16:0/672506504 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v32) ==== 170+0+0 (secure 0 0 0) 0x7feed4016110 con 0x7feee40cc9d0 2024-03-31T15:04:39.363 INFO:teuthology.orchestra.run.smithi016.stderr:pool 'cephfs_data' created 2024-03-31T15:04:39.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:39.372+0000 7feeeae00640 1 -- 172.21.15.16:0/672506504 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7feec0068380 msgr2=0x7feec006a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:39.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:39.372+0000 7feeeae00640 1 --2- 172.21.15.16:0/672506504 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7feec0068380 0x7feec006a840 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7feecc006fd0 tx=0x7feecc008040 comp rx=0 tx=0).stop 2024-03-31T15:04:39.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:39.372+0000 7feeeae00640 1 -- 172.21.15.16:0/672506504 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feee40cc9d0 msgr2=0x7feee40b5190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:39.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:39.372+0000 7feeeae00640 1 --2- 172.21.15.16:0/672506504 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feee40cc9d0 0x7feee40b5190 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7feed40b0040 tx=0x7feed401acd0 comp rx=0 tx=0).stop 2024-03-31T15:04:39.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:39.372+0000 7feeeae00640 1 -- 172.21.15.16:0/672506504 shutdown_connections 2024-03-31T15:04:39.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:39.372+0000 7feeeae00640 1 --2- 172.21.15.16:0/672506504 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7feec0068380 0x7feec006a840 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:39.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:39.372+0000 7feeeae00640 1 --2- 172.21.15.16:0/672506504 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7feee413e3d0 0x7feee413bd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:39.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:39.372+0000 7feeeae00640 1 --2- 172.21.15.16:0/672506504 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7feee413dff0 0x7feee41f7ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:39.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:39.372+0000 7feeeae00640 1 --2- 172.21.15.16:0/672506504 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feee40cc9d0 0x7feee40b5190 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:39.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:39.372+0000 7feeeae00640 1 -- 172.21.15.16:0/672506504 >> 172.21.15.16:0/672506504 conn(0x7feee406ee40 msgr2=0x7feee413d1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:04:39.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:39.372+0000 7feeeae00640 1 -- 172.21.15.16:0/672506504 shutdown_connections 2024-03-31T15:04:39.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:39.372+0000 7feeeae00640 1 -- 172.21.15.16:0/672506504 wait complete. 2024-03-31T15:04:39.408 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-31T15:04:39.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:39.560+0000 7f49c8e00640 1 Processor -- start 2024-03-31T15:04:39.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:39.560+0000 7f49c8e00640 1 -- start start 2024-03-31T15:04:39.554 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:39.560+0000 7f49c8e00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f49c413dff0 0x7f49c406d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:39.554 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:39.560+0000 7f49c8e00640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f49c415fbc0 con 0x7f49c40cd0e0 2024-03-31T15:04:39.554 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:39.560+0000 7f49c8e00640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f49c415ee00 con 0x7f49c413e3d0 2024-03-31T15:04:39.554 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:39.560+0000 7f49c8e00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f49c415f730 con 0x7f49c413dff0 2024-03-31T15:04:39.554 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:39.560+0000 7f49c2800640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f49c40cd0e0 0x7f49c40ceb70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:38776/0 (socket says 172.21.15.16:38776) 2024-03-31T15:04:39.554 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:39.560+0000 7f49c2800640 1 -- 172.21.15.16:0/2325239149 learned_addr learned my addr 172.21.15.16:0/2325239149 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:04:39.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:39.560+0000 7f49c1e00640 1 --2- 172.21.15.16:0/2325239149 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f49c413dff0 0x7f49c406d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:39.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:39.560+0000 7f49c1400640 1 -- 172.21.15.16:0/2325239149 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3867916984 0 0) 0x7f49c415fbc0 con 0x7f49c40cd0e0 2024-03-31T15:04:39.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:39.560+0000 7f49c1400640 1 -- 172.21.15.16:0/2325239149 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f49a8003430 con 0x7f49c40cd0e0 2024-03-31T15:04:39.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:39.560+0000 7f49c1400640 1 -- 172.21.15.16:0/2325239149 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2759147918 0 0) 0x7f49a8003430 con 0x7f49c40cd0e0 2024-03-31T15:04:39.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:39.560+0000 7f49c1400640 1 -- 172.21.15.16:0/2325239149 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f49c413dff0 msgr2=0x7f49c406d4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:04:39.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:39.560+0000 7f49c1400640 1 --2- 172.21.15.16:0/2325239149 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f49c413dff0 0x7f49c406d4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:39.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:39.560+0000 7f49c1400640 1 -- 172.21.15.16:0/2325239149 >> v1:172.21.15.89:6789/0 conn(0x7f49c413e3d0 legacy=0x7f49c40cc9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:39.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:39.560+0000 7f49c1400640 1 -- 172.21.15.16:0/2325239149 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49c415f0e0 con 0x7f49c40cd0e0 2024-03-31T15:04:39.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:39.560+0000 7f49c1400640 1 -- 172.21.15.16:0/2325239149 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 214807623 0 0) 0x7f49bc001a00 con 0x7f49c40cd0e0 2024-03-31T15:04:39.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:39.560+0000 7f49c1400640 1 -- 172.21.15.16:0/2325239149 >> v1:172.21.15.16:6789/0 conn(0x7f49c40cd0e0 legacy=0x7f49c40ceb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:39.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:39.560+0000 7f49c1400640 1 --2- 172.21.15.16:0/2325239149 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f49c413dff0 0x7f49a8004ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:39.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:39.560+0000 7f49c1400640 1 --2- 172.21.15.16:0/2325239149 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f49a8005050 0x7f49a80036b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:39.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:39.560+0000 7f49c1400640 1 --2- 172.21.15.16:0/2325239149 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f49a8003d40 0x7f49a800fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:39.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:39.560+0000 7f49c1400640 1 -- 172.21.15.16:0/2325239149 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f49c415f730 con 0x7f49a8003d40 2024-03-31T15:04:39.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:39.560+0000 7f49c1400640 1 -- 172.21.15.16:0/2325239149 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f49a80100c0 con 0x7f49a8005050 2024-03-31T15:04:39.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:39.560+0000 7f49c1400640 1 -- 172.21.15.16:0/2325239149 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f49a8010240 con 0x7f49c413dff0 2024-03-31T15:04:39.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:39.560+0000 7f49c2800640 1 --2- 172.21.15.16:0/2325239149 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f49a8003d40 0x7f49a800fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:39.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:39.560+0000 7f49c1e00640 1 --2- 172.21.15.16:0/2325239149 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f49c413dff0 0x7f49a8004ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:39.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:39.560+0000 7f49bbe00640 1 --2- 172.21.15.16:0/2325239149 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f49a8005050 0x7f49a80036b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:39.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:39.560+0000 7f49c2800640 1 -- 172.21.15.16:0/2325239149 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f49c413dff0 msgr2=0x7f49a8004ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:04:39.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:39.560+0000 7f49c2800640 1 --2- 172.21.15.16:0/2325239149 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f49c413dff0 0x7f49a8004ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:39.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:39.560+0000 7f49c2800640 1 -- 172.21.15.16:0/2325239149 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f49a8005050 msgr2=0x7f49a80036b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:04:39.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:39.560+0000 7f49c2800640 1 --2- 172.21.15.16:0/2325239149 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f49a8005050 0x7f49a80036b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:39.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:39.560+0000 7f49c2800640 1 -- 172.21.15.16:0/2325239149 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49a8003430 con 0x7f49a8003d40 2024-03-31T15:04:39.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:39.564+0000 7f49c2800640 1 --2- 172.21.15.16:0/2325239149 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f49a8003d40 0x7f49a800fb80 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f49bc00a860 tx=0x7f49bc01fae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c956a7c78af3eee server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:39.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:39.564+0000 7f49c1400640 1 -- 172.21.15.16:0/2325239149 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49bc00a480 con 0x7f49a8003d40 2024-03-31T15:04:39.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:39.564+0000 7f49c1400640 1 -- 172.21.15.16:0/2325239149 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f49bc0aa710 con 0x7f49a8003d40 2024-03-31T15:04:39.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:39.564+0000 7f49c1400640 1 -- 172.21.15.16:0/2325239149 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49bc0aab80 con 0x7f49a8003d40 2024-03-31T15:04:39.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:39.564+0000 7f49c8e00640 1 -- 172.21.15.16:0/2325239149 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f49a8003d40 msgr2=0x7f49a800fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:39.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:39.564+0000 7f49c8e00640 1 --2- 172.21.15.16:0/2325239149 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f49a8003d40 0x7f49a800fb80 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f49bc00a860 tx=0x7f49bc01fae0 comp rx=0 tx=0).stop 2024-03-31T15:04:39.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:39.564+0000 7f49c2800640 1 -- 172.21.15.16:0/2325239149 reap_dead start 2024-03-31T15:04:39.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:39.564+0000 7f49c8e00640 1 -- 172.21.15.16:0/2325239149 shutdown_connections 2024-03-31T15:04:39.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:39.564+0000 7f49c8e00640 1 -- 172.21.15.16:0/2325239149 >> 172.21.15.16:0/2325239149 conn(0x7f49c406ee40 msgr2=0x7f49c406b620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:04:39.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:39.564+0000 7f49c8e00640 1 -- 172.21.15.16:0/2325239149 shutdown_connections 2024-03-31T15:04:39.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:39.564+0000 7f49c8e00640 1 -- 172.21.15.16:0/2325239149 wait complete. 2024-03-31T15:04:39.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:39.568+0000 7f49c8e00640 1 Processor -- start 2024-03-31T15:04:39.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:39.568+0000 7f49c8e00640 1 -- start start 2024-03-31T15:04:39.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:39.568+0000 7f49c8e00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f49a8003d40 0x7f49c40a9f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:39.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:39.568+0000 7f49c8e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f49c41568e0 0x7f49c4157010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:39.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:39.568+0000 7f49c8e00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f49c40b0190 0x7f49c40b0610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:39.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:39.568+0000 7f49c8e00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f49c406d650 con 0x7f49c41568e0 2024-03-31T15:04:39.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:39.568+0000 7f49c8e00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f49c406d4d0 con 0x7f49a8003d40 2024-03-31T15:04:39.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:39.568+0000 7f49c8e00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f49c406d7d0 con 0x7f49c40b0190 2024-03-31T15:04:39.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:39.568+0000 7f49bbe00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f49c41568e0 0x7f49c4157010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:39.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:39.568+0000 7f49bbe00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f49c41568e0 0x7f49c4157010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54164/0 (socket says 172.21.15.16:54164) 2024-03-31T15:04:39.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:39.568+0000 7f49bbe00640 1 -- 172.21.15.16:0/3971640516 learned_addr learned my addr 172.21.15.16:0/3971640516 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:04:39.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:39.568+0000 7f49c2800640 1 --2- 172.21.15.16:0/3971640516 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f49c40b0190 0x7f49c40b0610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:39.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:39.568+0000 7f49c1e00640 1 --2- 172.21.15.16:0/3971640516 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f49a8003d40 0x7f49c40a9f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:39.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:39.568+0000 7f49bbe00640 1 -- 172.21.15.16:0/3971640516 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f49c40b0190 msgr2=0x7f49c40b0610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:39.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:39.568+0000 7f49bbe00640 1 --2- 172.21.15.16:0/3971640516 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f49c40b0190 0x7f49c40b0610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:39.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:39.568+0000 7f49bbe00640 1 -- 172.21.15.16:0/3971640516 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f49a8003d40 msgr2=0x7f49c40a9f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:39.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:39.568+0000 7f49bbe00640 1 --2- 172.21.15.16:0/3971640516 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f49a8003d40 0x7f49c40a9f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:39.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:39.568+0000 7f49bbe00640 1 -- 172.21.15.16:0/3971640516 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49c4157910 con 0x7f49c41568e0 2024-03-31T15:04:39.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:39.568+0000 7f49bbe00640 1 --2- 172.21.15.16:0/3971640516 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f49c41568e0 0x7f49c4157010 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f49ac00f820 tx=0x7f49ac00fc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:39.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:39.568+0000 7f49ba000640 1 -- 172.21.15.16:0/3971640516 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49ac0050d0 con 0x7f49c41568e0 2024-03-31T15:04:39.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:39.568+0000 7f49ba000640 1 -- 172.21.15.16:0/3971640516 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f49ac018070 con 0x7f49c41568e0 2024-03-31T15:04:39.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:39.568+0000 7f49ba000640 1 -- 172.21.15.16:0/3971640516 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49ac00d6b0 con 0x7f49c41568e0 2024-03-31T15:04:39.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:39.568+0000 7f49c8e00640 1 -- 172.21.15.16:0/3971640516 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f49c4157bf0 con 0x7f49c41568e0 2024-03-31T15:04:39.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:39.568+0000 7f49c8e00640 1 -- 172.21.15.16:0/3971640516 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f49c4156d60 con 0x7f49c41568e0 2024-03-31T15:04:39.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:39.572+0000 7f49c8e00640 1 -- 172.21.15.16:0/3971640516 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f49c4069950 con 0x7f49c41568e0 2024-03-31T15:04:39.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:39.572+0000 7f49ba000640 1 -- 172.21.15.16:0/3971640516 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f49ac00d850 con 0x7f49c41568e0 2024-03-31T15:04:39.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:39.572+0000 7f49ba000640 1 --2- 172.21.15.16:0/3971640516 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f49a00683b0 0x7f49a006a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:39.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:39.572+0000 7f49ba000640 1 -- 172.21.15.16:0/3971640516 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(32..32 src has 1..32) ==== 5609+0+0 (secure 0 0 0) 0x7f49ac0864b0 con 0x7f49c41568e0 2024-03-31T15:04:39.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:39.572+0000 7f49c1e00640 1 --2- 172.21.15.16:0/3971640516 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f49a00683b0 0x7f49a006a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:39.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:39.572+0000 7f49c1e00640 1 --2- 172.21.15.16:0/3971640516 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f49a00683b0 0x7f49a006a870 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f49c40b0dd0 tx=0x7f49b4009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:39.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:39.576+0000 7f49ba000640 1 -- 172.21.15.16:0/3971640516 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f49ac056b60 con 0x7f49c41568e0 2024-03-31T15:04:39.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:39.776+0000 7f49c8e00640 1 -- 172.21.15.16:0/3971640516 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7f49c413b380 con 0x7f49c41568e0 2024-03-31T15:04:40.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:40.372+0000 7f49ba000640 1 -- 172.21.15.16:0/3971640516 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 4 and data pool 5 v13) ==== 172+0+0 (secure 0 0 0) 0x7f49ac056460 con 0x7f49c41568e0 2024-03-31T15:04:40.366 INFO:teuthology.orchestra.run.smithi016.stderr:new fs with metadata pool 4 and data pool 5 2024-03-31T15:04:40.368 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:40.372+0000 7f49c8e00640 1 -- 172.21.15.16:0/3971640516 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f49a00683b0 msgr2=0x7f49a006a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:40.368 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:40.372+0000 7f49c8e00640 1 --2- 172.21.15.16:0/3971640516 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f49a00683b0 0x7f49a006a870 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f49c40b0dd0 tx=0x7f49b4009190 comp rx=0 tx=0).stop 2024-03-31T15:04:40.368 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:40.372+0000 7f49c8e00640 1 -- 172.21.15.16:0/3971640516 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f49c41568e0 msgr2=0x7f49c4157010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:40.368 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:40.372+0000 7f49c8e00640 1 --2- 172.21.15.16:0/3971640516 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f49c41568e0 0x7f49c4157010 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f49ac00f820 tx=0x7f49ac00fc20 comp rx=0 tx=0).stop 2024-03-31T15:04:40.368 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:40.372+0000 7f49c8e00640 1 -- 172.21.15.16:0/3971640516 shutdown_connections 2024-03-31T15:04:40.369 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:40.372+0000 7f49c8e00640 1 --2- 172.21.15.16:0/3971640516 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f49a00683b0 0x7f49a006a870 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:40.369 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:40.372+0000 7f49c8e00640 1 --2- 172.21.15.16:0/3971640516 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f49c40b0190 0x7f49c40b0610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:40.369 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:40.372+0000 7f49c8e00640 1 --2- 172.21.15.16:0/3971640516 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f49c41568e0 0x7f49c4157010 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:40.369 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:40.372+0000 7f49c8e00640 1 --2- 172.21.15.16:0/3971640516 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f49a8003d40 0x7f49c40a9f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:40.369 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:40.372+0000 7f49c8e00640 1 -- 172.21.15.16:0/3971640516 >> 172.21.15.16:0/3971640516 conn(0x7f49c406ee40 msgr2=0x7f49c415af50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:04:40.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:40.376+0000 7f49c8e00640 1 -- 172.21.15.16:0/3971640516 shutdown_connections 2024-03-31T15:04:40.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:40.380+0000 7f49c8e00640 1 -- 172.21.15.16:0/3971640516 wait complete. 2024-03-31T15:04:40.422 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-31T15:04:40.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:40.572+0000 7f88b9600640 1 Processor -- start 2024-03-31T15:04:40.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:40.572+0000 7f88b9600640 1 -- start start 2024-03-31T15:04:40.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:40.572+0000 7f88b9600640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f88b4072460 0x7f88b4072840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:40.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:40.572+0000 7f88b9600640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f88b40afa30 0x7f88b406ebd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:40.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:40.572+0000 7f88b9600640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f88b40ae890 con 0x7f88b40af5f0 2024-03-31T15:04:40.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:40.572+0000 7f88b9600640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f88b40af160 con 0x7f88b40afa30 2024-03-31T15:04:40.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:40.572+0000 7f88b9600640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f88b4072d80 con 0x7f88b4072460 2024-03-31T15:04:40.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:40.572+0000 7f88b8c00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f88b40afa30 0x7f88b406ebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:40.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:40.572+0000 7f88b3400640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f88b40af5f0 0x7f88b406e4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.16:53524/0 (socket says 172.21.15.16:53524) 2024-03-31T15:04:40.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:40.572+0000 7f88b3e00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f88b4072460 0x7f88b4072840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:40.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:40.572+0000 7f88b8c00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f88b40afa30 0x7f88b406ebd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54166/0 (socket says 172.21.15.16:54166) 2024-03-31T15:04:40.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:40.572+0000 7f88b3400640 1 -- 172.21.15.16:0/58342325 learned_addr learned my addr 172.21.15.16:0/58342325 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:04:40.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:40.572+0000 7f88b8c00640 1 -- 172.21.15.16:0/58342325 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f88b4072460 msgr2=0x7f88b4072840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:04:40.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:40.572+0000 7f88b8c00640 1 --2- 172.21.15.16:0/58342325 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f88b4072460 0x7f88b4072840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:40.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:40.572+0000 7f88b8c00640 1 -- 172.21.15.16:0/58342325 >> v1:172.21.15.89:6789/0 conn(0x7f88b40af5f0 legacy=0x7f88b406e4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:40.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:40.572+0000 7f88b8c00640 1 -- 172.21.15.16:0/58342325 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88b415c1c0 con 0x7f88b40afa30 2024-03-31T15:04:40.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:40.572+0000 7f88b2a00640 1 -- 172.21.15.16:0/58342325 <== mon.1 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3553004077 0 0) 0x7f88a0001120 con 0x7f88b40af5f0 2024-03-31T15:04:40.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:40.576+0000 7f88b8c00640 1 --2- 172.21.15.16:0/58342325 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f88b40afa30 0x7f88b406ebd0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f88ac009d90 tx=0x7f88ac01c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd2a12bfd5c6986 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:40.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:40.576+0000 7f88b2a00640 1 -- 172.21.15.16:0/58342325 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88ac0b2070 con 0x7f88b40afa30 2024-03-31T15:04:40.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:40.576+0000 7f88b2a00640 1 -- 172.21.15.16:0/58342325 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f88ac0a5920 con 0x7f88b40afa30 2024-03-31T15:04:40.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:40.576+0000 7f88b2a00640 1 -- 172.21.15.16:0/58342325 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88ac0a5cf0 con 0x7f88b40afa30 2024-03-31T15:04:40.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:40.576+0000 7f88b9600640 1 -- 172.21.15.16:0/58342325 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f88b40afa30 msgr2=0x7f88b406ebd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:40.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:40.576+0000 7f88b9600640 1 --2- 172.21.15.16:0/58342325 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f88b40afa30 0x7f88b406ebd0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f88ac009d90 tx=0x7f88ac01c970 comp rx=0 tx=0).stop 2024-03-31T15:04:40.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:40.576+0000 7f88b9600640 1 -- 172.21.15.16:0/58342325 shutdown_connections 2024-03-31T15:04:40.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:40.576+0000 7f88b9600640 1 --2- 172.21.15.16:0/58342325 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f88b40afa30 0x7f88b406ebd0 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:40.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:40.576+0000 7f88b9600640 1 --2- 172.21.15.16:0/58342325 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f88b4072460 0x7f88b4072840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:40.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:40.576+0000 7f88b9600640 1 -- 172.21.15.16:0/58342325 >> 172.21.15.16:0/58342325 conn(0x7f88b40b2b30 msgr2=0x7f88b40b2f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:04:40.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:40.576+0000 7f88b9600640 1 -- 172.21.15.16:0/58342325 shutdown_connections 2024-03-31T15:04:40.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:40.576+0000 7f88b9600640 1 -- 172.21.15.16:0/58342325 wait complete. 2024-03-31T15:04:40.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:40.576+0000 7f88b9600640 1 Processor -- start 2024-03-31T15:04:40.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:40.580+0000 7f88b9600640 1 -- start start 2024-03-31T15:04:40.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:40.580+0000 7f88b9600640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f88b4072460 0x7f88b4142960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:40.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:40.580+0000 7f88b9600640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f88b40af5f0 0x7f88b41529a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:40.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:40.580+0000 7f88b9600640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f88b40afa30 0x7f88b41596d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:40.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:40.580+0000 7f88b9600640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f88b415bdf0 con 0x7f88b40af5f0 2024-03-31T15:04:40.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:40.580+0000 7f88b9600640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f88b415bc70 con 0x7f88b4072460 2024-03-31T15:04:40.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:40.580+0000 7f88b9600640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f88b415bf70 con 0x7f88b40afa30 2024-03-31T15:04:40.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:40.580+0000 7f88b3400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f88b40af5f0 0x7f88b41529a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:40.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:40.580+0000 7f88b3400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f88b40af5f0 0x7f88b41529a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54182/0 (socket says 172.21.15.16:54182) 2024-03-31T15:04:40.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:40.580+0000 7f88b3400640 1 -- 172.21.15.16:0/3223814760 learned_addr learned my addr 172.21.15.16:0/3223814760 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:04:40.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:40.580+0000 7f88b8c00640 1 --2- 172.21.15.16:0/3223814760 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f88b40afa30 0x7f88b41596d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:40.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:40.580+0000 7f88b3e00640 1 --2- 172.21.15.16:0/3223814760 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f88b4072460 0x7f88b4142960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:40.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:40.580+0000 7f88b3400640 1 -- 172.21.15.16:0/3223814760 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f88b40afa30 msgr2=0x7f88b41596d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:40.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:40.580+0000 7f88b3400640 1 --2- 172.21.15.16:0/3223814760 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f88b40afa30 0x7f88b41596d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:40.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:40.580+0000 7f88b3400640 1 -- 172.21.15.16:0/3223814760 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f88b4072460 msgr2=0x7f88b4142960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:40.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:40.580+0000 7f88b3400640 1 --2- 172.21.15.16:0/3223814760 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f88b4072460 0x7f88b4142960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:40.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:40.580+0000 7f88b3400640 1 -- 172.21.15.16:0/3223814760 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88b41547b0 con 0x7f88b40af5f0 2024-03-31T15:04:40.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:40.580+0000 7f88b3400640 1 --2- 172.21.15.16:0/3223814760 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f88b40af5f0 0x7f88b41529a0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f88a000c780 tx=0x7f88a000cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:40.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:40.580+0000 7f88abe00640 1 -- 172.21.15.16:0/3223814760 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88a0018070 con 0x7f88b40af5f0 2024-03-31T15:04:40.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:40.580+0000 7f88abe00640 1 -- 172.21.15.16:0/3223814760 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f88a00072f0 con 0x7f88b40af5f0 2024-03-31T15:04:40.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:40.580+0000 7f88b9600640 1 -- 172.21.15.16:0/3223814760 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f88b4151be0 con 0x7f88b40af5f0 2024-03-31T15:04:40.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:40.580+0000 7f88abe00640 1 -- 172.21.15.16:0/3223814760 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88a00047f0 con 0x7f88b40af5f0 2024-03-31T15:04:40.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:40.580+0000 7f88b9600640 1 -- 172.21.15.16:0/3223814760 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f88b41521a0 con 0x7f88b40af5f0 2024-03-31T15:04:40.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:40.580+0000 7f88b9600640 1 -- 172.21.15.16:0/3223814760 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f88b404f8c0 con 0x7f88b40af5f0 2024-03-31T15:04:40.577 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:40.584+0000 7f88abe00640 1 -- 172.21.15.16:0/3223814760 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f88a001b070 con 0x7f88b40af5f0 2024-03-31T15:04:40.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:40.584+0000 7f88abe00640 1 --2- 172.21.15.16:0/3223814760 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f8894068380 0x7f889406a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:40.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:40.584+0000 7f88abe00640 1 -- 172.21.15.16:0/3223814760 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5709+0+0 (secure 0 0 0) 0x7f88a0085cc0 con 0x7f88b40af5f0 2024-03-31T15:04:40.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:40.584+0000 7f88b3e00640 1 --2- 172.21.15.16:0/3223814760 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f8894068380 0x7f889406a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:40.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:40.584+0000 7f88b3e00640 1 --2- 172.21.15.16:0/3223814760 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f8894068380 0x7f889406a840 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f88a4009b60 tx=0x7f88a4009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:40.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:40.588+0000 7f88abe00640 1 -- 172.21.15.16:0/3223814760 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f88b40ae890 con 0x7f88b40af5f0 2024-03-31T15:04:40.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:40.752+0000 7f88b9600640 1 -- 172.21.15.16:0/3223814760 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f88b40cd630 con 0x7f88b40af5f0 2024-03-31T15:04:40.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:40.756+0000 7f88abe00640 1 -- 172.21.15.16:0/3223814760 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v33) ==== 74+0+16610 (secure 0 0 0) 0x7f88a0055cc0 con 0x7f88b40af5f0 2024-03-31T15:04:40.749 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:04:40.749 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":33,"fsid":"286eb5da-fb47-4550-b290-d9519c13b9a5","created":"2024-03-31T15:03:11.438952+0000","modified":"2024-03-31T15:04:39.779996+0000","last_up_change":"2024-03-31T15:03:22.885535+0000","last_in_change":"2024-03-31T15:03:13.941667+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-31T15:03:22.550787+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-03-31T15:04:37.577341+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-03-31T15:04:38.738566+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"92a66450-ef19-40d8-ac3d-d760b23b0bda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6801","nonce":932712244}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6803","nonce":932712244}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6807","nonce":932712244}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6805","nonce":932712244}]},"public_addr":"172.21.15.16:6801/932712244","cluster_addr":"172.21.15.16:6803/932712244","heartbeat_back_addr":"172.21.15.16:6807/932712244","heartbeat_front_addr":"172.21.15.16:6805/932712244","state":["exists","up"]},{"osd":1,"uuid":"da9d1f2b-1968-4959-b843-faa231988e92","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6810","nonce":1398737533}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6811","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6813","nonce":1398737533}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6821","nonce":1398737533}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6815","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6816","nonce":1398737533}]},"public_addr":"172.21.15.16:6810/1398737533","cluster_addr":"172.21.15.16:6813/1398737533","heartbeat_back_addr":"172.21.15.16:6821/1398737533","heartbeat_front_addr":"172.21.15.16:6816/1398737533","state":["exists","up"]},{"osd":2,"uuid":"01a73662-9a21-4b1f-bcbb-42b9636a4638","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6819","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6822","nonce":3018526646}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6825","nonce":3018526646}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6831","nonce":3018526646}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6827","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6829","nonce":3018526646}]},"public_addr":"172.21.15.16:6822/3018526646","cluster_addr":"172.21.15.16:6825/3018526646","heartbeat_back_addr":"172.21.15.16:6831/3018526646","heartbeat_front_addr":"172.21.15.16:6829/3018526646","state":["exists","up"]},{"osd":3,"uuid":"c7771fd3-e18e-4500-ac38-23cf10ebcd75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6809","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6812","nonce":3360352974}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6817","nonce":3360352974}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6828","nonce":3360352974}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6823","nonce":3360352974}]},"public_addr":"172.21.15.16:6812/3360352974","cluster_addr":"172.21.15.16:6817/3360352974","heartbeat_back_addr":"172.21.15.16:6828/3360352974","heartbeat_front_addr":"172.21.15.16:6823/3360352974","state":["exists","up"]},{"osd":4,"uuid":"88260730-1482-4fbd-a40f-cb1124e9df4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6801","nonce":1856844856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6803","nonce":1856844856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6807","nonce":1856844856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6805","nonce":1856844856}]},"public_addr":"172.21.15.89:6801/1856844856","cluster_addr":"172.21.15.89:6803/1856844856","heartbeat_back_addr":"172.21.15.89:6807/1856844856","heartbeat_front_addr":"172.21.15.89:6805/1856844856","state":["exists","up"]},{"osd":5,"uuid":"4cb2c00c-fae0-4dd8-9d27-ec4f0093ccf7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6814","nonce":50395527}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6817","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6820","nonce":50395527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6831","nonce":50395527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6823","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6826","nonce":50395527}]},"public_addr":"172.21.15.89:6814/50395527","cluster_addr":"172.21.15.89:6820/50395527","heartbeat_back_addr":"172.21.15.89:6831/50395527","heartbeat_front_addr":"172.21.15.89:6826/50395527","state":["exists","up"]},{"osd":6,"uuid":"81ce0c19-2554-4cae-acbe-66a05e9e9b9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6809","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6812","nonce":3470471752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6815","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6818","nonce":3470471752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6827","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6829","nonce":3470471752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6821","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6824","nonce":3470471752}]},"public_addr":"172.21.15.89:6812/3470471752","cluster_addr":"172.21.15.89:6818/3470471752","heartbeat_back_addr":"172.21.15.89:6829/3470471752","heartbeat_front_addr":"172.21.15.89:6824/3470471752","state":["exists","up"]},{"osd":7,"uuid":"85e4313b-d37d-442e-8042-153bf20a847d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6811","nonce":1796778630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6813","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6816","nonce":1796778630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6825","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6828","nonce":1796778630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6819","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6822","nonce":1796778630}]},"public_addr":"172.21.15.89:6811/1796778630","cluster_addr":"172.21.15.89:6816/1796778630","heartbeat_back_addr":"172.21.15.89:6828/1796778630","heartbeat_front_addr":"172.21.15.89:6822/1796778630","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.439088+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.529970+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.680512+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:21.044161+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.582111+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.536558+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.409522+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.590229+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-31T15:04:40.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:40.756+0000 7f88b9600640 1 -- 172.21.15.16:0/3223814760 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f8894068380 msgr2=0x7f889406a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:40.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:40.756+0000 7f88b9600640 1 --2- 172.21.15.16:0/3223814760 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f8894068380 0x7f889406a840 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f88a4009b60 tx=0x7f88a4009190 comp rx=0 tx=0).stop 2024-03-31T15:04:40.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:40.756+0000 7f88b9600640 1 -- 172.21.15.16:0/3223814760 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f88b40af5f0 msgr2=0x7f88b41529a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:40.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:40.756+0000 7f88b9600640 1 --2- 172.21.15.16:0/3223814760 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f88b40af5f0 0x7f88b41529a0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f88a000c780 tx=0x7f88a000cc40 comp rx=0 tx=0).stop 2024-03-31T15:04:40.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:40.756+0000 7f88b9600640 1 -- 172.21.15.16:0/3223814760 shutdown_connections 2024-03-31T15:04:40.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:40.756+0000 7f88b9600640 1 --2- 172.21.15.16:0/3223814760 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f8894068380 0x7f889406a840 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:40.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:40.756+0000 7f88b9600640 1 --2- 172.21.15.16:0/3223814760 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f88b40afa30 0x7f88b41596d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:40.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:40.756+0000 7f88b9600640 1 --2- 172.21.15.16:0/3223814760 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f88b40af5f0 0x7f88b41529a0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:40.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:40.756+0000 7f88b9600640 1 --2- 172.21.15.16:0/3223814760 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f88b4072460 0x7f88b4142960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:40.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:40.756+0000 7f88b9600640 1 -- 172.21.15.16:0/3223814760 >> 172.21.15.16:0/3223814760 conn(0x7f88b40b2b30 msgr2=0x7f88b40b2300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:04:40.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:40.756+0000 7f88b9600640 1 -- 172.21.15.16:0/3223814760 shutdown_connections 2024-03-31T15:04:40.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:40.756+0000 7f88b9600640 1 -- 172.21.15.16:0/3223814760 wait complete. 2024-03-31T15:04:40.771 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-31T15:04:40.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:40.916+0000 7f863c000640 1 Processor -- start 2024-03-31T15:04:40.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:40.916+0000 7f863c000640 1 -- start start 2024-03-31T15:04:40.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:40.916+0000 7f863c000640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f86340b23d0 0x7f8634071d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:40.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:40.916+0000 7f863c000640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f863415eee0 0x7f8634072250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:40.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:40.916+0000 7f863c000640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f863415f3c0 con 0x7f863415b3d0 2024-03-31T15:04:40.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:40.916+0000 7f863c000640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f863415b7b0 con 0x7f86340b23d0 2024-03-31T15:04:40.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:40.916+0000 7f863c000640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f863415f6a0 con 0x7f863415eee0 2024-03-31T15:04:40.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:40.916+0000 7f8639600640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f863415b3d0 0x7f86340b1cc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:38792/0 (socket says 172.21.15.16:38792) 2024-03-31T15:04:40.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:40.916+0000 7f8639600640 1 -- 172.21.15.16:0/2943133925 learned_addr learned my addr 172.21.15.16:0/2943133925 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:04:40.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:40.916+0000 7f8638c00640 1 --2- 172.21.15.16:0/2943133925 >> v2:172.21.15.16:3300/0 conn(0x7f86340b23d0 0x7f8634071d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:40.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:40.916+0000 7f863a000640 1 --2- 172.21.15.16:0/2943133925 >> v2:172.21.15.89:3300/0 conn(0x7f863415eee0 0x7f8634072250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:40.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:40.916+0000 7f8633e00640 1 -- 172.21.15.16:0/2943133925 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 500111364 0 0) 0x7f863415f3c0 con 0x7f863415b3d0 2024-03-31T15:04:40.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:40.916+0000 7f8633e00640 1 -- 172.21.15.16:0/2943133925 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f861c003530 con 0x7f863415b3d0 2024-03-31T15:04:40.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:40.916+0000 7f8638c00640 1 -- 172.21.15.16:0/2943133925 >> v2:172.21.15.89:3300/0 conn(0x7f863415eee0 msgr2=0x7f8634072250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:04:40.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:40.916+0000 7f8638c00640 1 --2- 172.21.15.16:0/2943133925 >> v2:172.21.15.89:3300/0 conn(0x7f863415eee0 0x7f8634072250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:40.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:40.916+0000 7f8638c00640 1 -- 172.21.15.16:0/2943133925 >> v1:172.21.15.16:6789/0 conn(0x7f863415b3d0 legacy=0x7f86340b1cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:40.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:40.916+0000 7f8638c00640 1 -- 172.21.15.16:0/2943133925 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f863415f980 con 0x7f86340b23d0 2024-03-31T15:04:40.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:40.920+0000 7f8638c00640 1 --2- 172.21.15.16:0/2943133925 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f86340b23d0 0x7f8634071d10 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f8634073510 tx=0x7f862001c450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6bb2feb9ba42533 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:40.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:40.920+0000 7f8633e00640 1 -- 172.21.15.16:0/2943133925 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86200ab610 con 0x7f86340b23d0 2024-03-31T15:04:40.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:40.920+0000 7f8633e00640 1 -- 172.21.15.16:0/2943133925 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f86200b0070 con 0x7f86340b23d0 2024-03-31T15:04:40.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:40.920+0000 7f8633e00640 1 -- 172.21.15.16:0/2943133925 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86200ab9d0 con 0x7f86340b23d0 2024-03-31T15:04:40.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:40.920+0000 7f863c000640 1 -- 172.21.15.16:0/2943133925 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f86340b23d0 msgr2=0x7f8634071d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:40.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:40.920+0000 7f863c000640 1 --2- 172.21.15.16:0/2943133925 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f86340b23d0 0x7f8634071d10 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f8634073510 tx=0x7f862001c450 comp rx=0 tx=0).stop 2024-03-31T15:04:40.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:40.920+0000 7f863c000640 1 -- 172.21.15.16:0/2943133925 shutdown_connections 2024-03-31T15:04:40.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:40.920+0000 7f863c000640 1 --2- 172.21.15.16:0/2943133925 >> v2:172.21.15.89:3300/0 conn(0x7f863415eee0 0x7f8634072250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:40.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:40.920+0000 7f863c000640 1 --2- 172.21.15.16:0/2943133925 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f86340b23d0 0x7f8634071d10 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:40.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:40.920+0000 7f863c000640 1 -- 172.21.15.16:0/2943133925 >> 172.21.15.16:0/2943133925 conn(0x7f863406b620 msgr2=0x7f863406ba30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:04:40.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:40.920+0000 7f863c000640 1 -- 172.21.15.16:0/2943133925 shutdown_connections 2024-03-31T15:04:40.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:40.920+0000 7f863c000640 1 -- 172.21.15.16:0/2943133925 wait complete. 2024-03-31T15:04:40.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:40.924+0000 7f863c000640 1 Processor -- start 2024-03-31T15:04:40.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:40.924+0000 7f863c000640 1 -- start start 2024-03-31T15:04:40.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:40.924+0000 7f863c000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f86340b23d0 0x7f8634164df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:40.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:40.924+0000 7f863c000640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f863415b3d0 0x7f863414fa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:40.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:40.924+0000 7f863c000640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f863415eee0 0x7f863414ffa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:40.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:40.924+0000 7f863c000640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f8634073910 con 0x7f86340b23d0 2024-03-31T15:04:40.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:40.924+0000 7f863c000640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f8634073790 con 0x7f863415b3d0 2024-03-31T15:04:40.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:40.924+0000 7f863c000640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f8634072790 con 0x7f863415eee0 2024-03-31T15:04:40.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:40.924+0000 7f8639600640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f86340b23d0 0x7f8634164df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:40.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:40.924+0000 7f8639600640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f86340b23d0 0x7f8634164df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54208/0 (socket says 172.21.15.16:54208) 2024-03-31T15:04:40.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:40.924+0000 7f8639600640 1 -- 172.21.15.16:0/802129180 learned_addr learned my addr 172.21.15.16:0/802129180 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:04:40.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:40.924+0000 7f8638c00640 1 --2- 172.21.15.16:0/802129180 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f863415b3d0 0x7f863414fa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:40.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:40.924+0000 7f863a000640 1 --2- 172.21.15.16:0/802129180 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f863415eee0 0x7f863414ffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:40.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:40.924+0000 7f8639600640 1 -- 172.21.15.16:0/802129180 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f863415eee0 msgr2=0x7f863414ffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:40.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:40.924+0000 7f8639600640 1 --2- 172.21.15.16:0/802129180 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f863415eee0 0x7f863414ffa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:40.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:40.924+0000 7f8639600640 1 -- 172.21.15.16:0/802129180 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f863415b3d0 msgr2=0x7f863414fa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:40.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:40.924+0000 7f8639600640 1 --2- 172.21.15.16:0/802129180 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f863415b3d0 0x7f863414fa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:40.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:40.924+0000 7f8639600640 1 -- 172.21.15.16:0/802129180 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86341564c0 con 0x7f86340b23d0 2024-03-31T15:04:40.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:40.924+0000 7f8639600640 1 --2- 172.21.15.16:0/802129180 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f86340b23d0 0x7f8634164df0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f862800c450 tx=0x7f862800c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:40.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:40.924+0000 7f8631600640 1 -- 172.21.15.16:0/802129180 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8628017070 con 0x7f86340b23d0 2024-03-31T15:04:40.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:40.924+0000 7f8631600640 1 -- 172.21.15.16:0/802129180 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8628005cf0 con 0x7f86340b23d0 2024-03-31T15:04:40.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:40.924+0000 7f863c000640 1 -- 172.21.15.16:0/802129180 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f86341657d0 con 0x7f86340b23d0 2024-03-31T15:04:40.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:40.924+0000 7f8631600640 1 -- 172.21.15.16:0/802129180 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8628007290 con 0x7f86340b23d0 2024-03-31T15:04:40.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:40.924+0000 7f863c000640 1 -- 172.21.15.16:0/802129180 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f86340afee0 con 0x7f86340b23d0 2024-03-31T15:04:40.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:40.928+0000 7f863c000640 1 -- 172.21.15.16:0/802129180 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8634069950 con 0x7f86340b23d0 2024-03-31T15:04:40.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:40.928+0000 7f8631600640 1 -- 172.21.15.16:0/802129180 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f862801a070 con 0x7f86340b23d0 2024-03-31T15:04:40.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:40.928+0000 7f8631600640 1 --2- 172.21.15.16:0/802129180 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f8614068380 0x7f861406a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:40.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:40.928+0000 7f8631600640 1 -- 172.21.15.16:0/802129180 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5709+0+0 (secure 0 0 0) 0x7f8628085ce0 con 0x7f86340b23d0 2024-03-31T15:04:40.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:40.928+0000 7f8638c00640 1 --2- 172.21.15.16:0/802129180 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f8614068380 0x7f861406a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:40.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:40.928+0000 7f8638c00640 1 --2- 172.21.15.16:0/802129180 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f8614068380 0x7f861406a840 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f86200af000 tx=0x7f86200a7000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:40.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:40.932+0000 7f8631600640 1 -- 172.21.15.16:0/802129180 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f86341564c0 con 0x7f86340b23d0 2024-03-31T15:04:41.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:41.096+0000 7f863c000640 1 -- 172.21.15.16:0/802129180 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f86340cdea0 con 0x7f86340b23d0 2024-03-31T15:04:41.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:41.096+0000 7f8631600640 1 -- 172.21.15.16:0/802129180 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v33) ==== 74+0+16610 (secure 0 0 0) 0x7f8628055c50 con 0x7f86340b23d0 2024-03-31T15:04:41.092 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:04:41.092 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":33,"fsid":"286eb5da-fb47-4550-b290-d9519c13b9a5","created":"2024-03-31T15:03:11.438952+0000","modified":"2024-03-31T15:04:39.779996+0000","last_up_change":"2024-03-31T15:03:22.885535+0000","last_in_change":"2024-03-31T15:03:13.941667+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-31T15:03:22.550787+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-03-31T15:04:37.577341+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-03-31T15:04:38.738566+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"92a66450-ef19-40d8-ac3d-d760b23b0bda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6801","nonce":932712244}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6803","nonce":932712244}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6807","nonce":932712244}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6805","nonce":932712244}]},"public_addr":"172.21.15.16:6801/932712244","cluster_addr":"172.21.15.16:6803/932712244","heartbeat_back_addr":"172.21.15.16:6807/932712244","heartbeat_front_addr":"172.21.15.16:6805/932712244","state":["exists","up"]},{"osd":1,"uuid":"da9d1f2b-1968-4959-b843-faa231988e92","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6810","nonce":1398737533}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6811","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6813","nonce":1398737533}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6821","nonce":1398737533}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6815","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6816","nonce":1398737533}]},"public_addr":"172.21.15.16:6810/1398737533","cluster_addr":"172.21.15.16:6813/1398737533","heartbeat_back_addr":"172.21.15.16:6821/1398737533","heartbeat_front_addr":"172.21.15.16:6816/1398737533","state":["exists","up"]},{"osd":2,"uuid":"01a73662-9a21-4b1f-bcbb-42b9636a4638","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6819","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6822","nonce":3018526646}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6825","nonce":3018526646}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6831","nonce":3018526646}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6827","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6829","nonce":3018526646}]},"public_addr":"172.21.15.16:6822/3018526646","cluster_addr":"172.21.15.16:6825/3018526646","heartbeat_back_addr":"172.21.15.16:6831/3018526646","heartbeat_front_addr":"172.21.15.16:6829/3018526646","state":["exists","up"]},{"osd":3,"uuid":"c7771fd3-e18e-4500-ac38-23cf10ebcd75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6809","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6812","nonce":3360352974}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6817","nonce":3360352974}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6828","nonce":3360352974}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6823","nonce":3360352974}]},"public_addr":"172.21.15.16:6812/3360352974","cluster_addr":"172.21.15.16:6817/3360352974","heartbeat_back_addr":"172.21.15.16:6828/3360352974","heartbeat_front_addr":"172.21.15.16:6823/3360352974","state":["exists","up"]},{"osd":4,"uuid":"88260730-1482-4fbd-a40f-cb1124e9df4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6801","nonce":1856844856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6803","nonce":1856844856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6807","nonce":1856844856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6805","nonce":1856844856}]},"public_addr":"172.21.15.89:6801/1856844856","cluster_addr":"172.21.15.89:6803/1856844856","heartbeat_back_addr":"172.21.15.89:6807/1856844856","heartbeat_front_addr":"172.21.15.89:6805/1856844856","state":["exists","up"]},{"osd":5,"uuid":"4cb2c00c-fae0-4dd8-9d27-ec4f0093ccf7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6814","nonce":50395527}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6817","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6820","nonce":50395527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6831","nonce":50395527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6823","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6826","nonce":50395527}]},"public_addr":"172.21.15.89:6814/50395527","cluster_addr":"172.21.15.89:6820/50395527","heartbeat_back_addr":"172.21.15.89:6831/50395527","heartbeat_front_addr":"172.21.15.89:6826/50395527","state":["exists","up"]},{"osd":6,"uuid":"81ce0c19-2554-4cae-acbe-66a05e9e9b9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6809","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6812","nonce":3470471752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6815","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6818","nonce":3470471752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6827","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6829","nonce":3470471752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6821","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6824","nonce":3470471752}]},"public_addr":"172.21.15.89:6812/3470471752","cluster_addr":"172.21.15.89:6818/3470471752","heartbeat_back_addr":"172.21.15.89:6829/3470471752","heartbeat_front_addr":"172.21.15.89:6824/3470471752","state":["exists","up"]},{"osd":7,"uuid":"85e4313b-d37d-442e-8042-153bf20a847d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6811","nonce":1796778630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6813","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6816","nonce":1796778630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6825","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6828","nonce":1796778630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6819","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6822","nonce":1796778630}]},"public_addr":"172.21.15.89:6811/1796778630","cluster_addr":"172.21.15.89:6816/1796778630","heartbeat_back_addr":"172.21.15.89:6828/1796778630","heartbeat_front_addr":"172.21.15.89:6822/1796778630","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.439088+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.529970+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.680512+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:21.044161+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.582111+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.536558+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.409522+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.590229+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-31T15:04:41.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:41.100+0000 7f863c000640 1 -- 172.21.15.16:0/802129180 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f8614068380 msgr2=0x7f861406a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:41.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:41.100+0000 7f863c000640 1 --2- 172.21.15.16:0/802129180 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f8614068380 0x7f861406a840 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f86200af000 tx=0x7f86200a7000 comp rx=0 tx=0).stop 2024-03-31T15:04:41.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:41.100+0000 7f863c000640 1 -- 172.21.15.16:0/802129180 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f86340b23d0 msgr2=0x7f8634164df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:41.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:41.100+0000 7f863c000640 1 --2- 172.21.15.16:0/802129180 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f86340b23d0 0x7f8634164df0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f862800c450 tx=0x7f862800c910 comp rx=0 tx=0).stop 2024-03-31T15:04:41.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:41.100+0000 7f863c000640 1 -- 172.21.15.16:0/802129180 shutdown_connections 2024-03-31T15:04:41.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:41.100+0000 7f863c000640 1 --2- 172.21.15.16:0/802129180 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f8614068380 0x7f861406a840 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:41.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:41.100+0000 7f863c000640 1 --2- 172.21.15.16:0/802129180 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f863415eee0 0x7f863414ffa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:41.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:41.100+0000 7f863c000640 1 --2- 172.21.15.16:0/802129180 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f863415b3d0 0x7f863414fa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:41.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:41.100+0000 7f863c000640 1 --2- 172.21.15.16:0/802129180 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f86340b23d0 0x7f8634164df0 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:41.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:41.100+0000 7f863c000640 1 -- 172.21.15.16:0/802129180 >> 172.21.15.16:0/802129180 conn(0x7f863406b620 msgr2=0x7f86340cf190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:04:41.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:41.100+0000 7f863c000640 1 -- 172.21.15.16:0/802129180 shutdown_connections 2024-03-31T15:04:41.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:41.100+0000 7f863c000640 1 -- 172.21.15.16:0/802129180 wait complete. 2024-03-31T15:04:41.113 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-31T15:04:41.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:41.264+0000 7fe97c800640 1 Processor -- start 2024-03-31T15:04:41.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:41.264+0000 7fe97c800640 1 -- start start 2024-03-31T15:04:41.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:41.264+0000 7fe97c800640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fe9741411b0 0x7fe97406f920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:41.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:41.264+0000 7fe97c800640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe974144980 con 0x7fe974141590 2024-03-31T15:04:41.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:41.264+0000 7fe97c800640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe974161fe0 con 0x7fe974144340 2024-03-31T15:04:41.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:41.264+0000 7fe97c800640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7fe974162910 con 0x7fe9741411b0 2024-03-31T15:04:41.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:41.264+0000 7fe979400640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fe974141590 0x7fe974143c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:38804/0 (socket says 172.21.15.16:38804) 2024-03-31T15:04:41.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:41.264+0000 7fe979400640 1 -- 172.21.15.16:0/955138966 learned_addr learned my addr 172.21.15.16:0/955138966 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:04:41.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:41.264+0000 7fe979e00640 1 --2- 172.21.15.16:0/955138966 >> v2:172.21.15.89:3300/0 conn(0x7fe9741411b0 0x7fe97406f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:41.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:41.264+0000 7fe978a00640 1 -- 172.21.15.16:0/955138966 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 775513472 0 0) 0x7fe974144980 con 0x7fe974141590 2024-03-31T15:04:41.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:41.264+0000 7fe978a00640 1 -- 172.21.15.16:0/955138966 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe95c003530 con 0x7fe974141590 2024-03-31T15:04:41.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:41.264+0000 7fe979e00640 1 -- 172.21.15.16:0/955138966 >> v1:172.21.15.89:6789/0 conn(0x7fe974144340 legacy=0x7fe97415e900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:41.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:41.264+0000 7fe978a00640 1 -- 172.21.15.16:0/955138966 <== mon.1 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3926548340 0 0) 0x7fe974161fe0 con 0x7fe974144340 2024-03-31T15:04:41.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:41.264+0000 7fe979e00640 1 -- 172.21.15.16:0/955138966 >> v1:172.21.15.16:6789/0 conn(0x7fe974141590 legacy=0x7fe974143c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:41.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:41.264+0000 7fe979e00640 1 -- 172.21.15.16:0/955138966 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9741622c0 con 0x7fe9741411b0 2024-03-31T15:04:41.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:41.272+0000 7fe979e00640 1 --2- 172.21.15.16:0/955138966 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe9741411b0 0x7fe97406f920 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fe9680097c0 tx=0x7fe96801c4a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=16869922e34585e0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:41.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:41.272+0000 7fe978a00640 1 -- 172.21.15.16:0/955138966 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9680b0070 con 0x7fe9741411b0 2024-03-31T15:04:41.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:41.272+0000 7fe978a00640 1 -- 172.21.15.16:0/955138966 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe9680ab600 con 0x7fe9741411b0 2024-03-31T15:04:41.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:41.272+0000 7fe978a00640 1 -- 172.21.15.16:0/955138966 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9680ab9d0 con 0x7fe9741411b0 2024-03-31T15:04:41.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:41.272+0000 7fe97c800640 1 -- 172.21.15.16:0/955138966 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe9741411b0 msgr2=0x7fe97406f920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:41.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:41.272+0000 7fe97c800640 1 --2- 172.21.15.16:0/955138966 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe9741411b0 0x7fe97406f920 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fe9680097c0 tx=0x7fe96801c4a0 comp rx=0 tx=0).stop 2024-03-31T15:04:41.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:41.272+0000 7fe97c800640 1 -- 172.21.15.16:0/955138966 shutdown_connections 2024-03-31T15:04:41.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:41.272+0000 7fe97c800640 1 --2- 172.21.15.16:0/955138966 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe9741411b0 0x7fe97406f920 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:41.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:41.272+0000 7fe97c800640 1 -- 172.21.15.16:0/955138966 >> 172.21.15.16:0/955138966 conn(0x7fe974067930 msgr2=0x7fe9740b3ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:04:41.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:41.272+0000 7fe97c800640 1 -- 172.21.15.16:0/955138966 shutdown_connections 2024-03-31T15:04:41.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:41.272+0000 7fe97c800640 1 -- 172.21.15.16:0/955138966 wait complete. 2024-03-31T15:04:41.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:41.276+0000 7fe97c800640 1 Processor -- start 2024-03-31T15:04:41.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:41.276+0000 7fe97c800640 1 -- start start 2024-03-31T15:04:41.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:41.276+0000 7fe97c800640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fe9741411b0 0x7fe9741436c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:41.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:41.276+0000 7fe97c800640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe974141590 0x7fe97413cb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:41.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:41.276+0000 7fe97c800640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe974144340 0x7fe97413d060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:41.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:41.276+0000 7fe97c800640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fe97416c680 con 0x7fe974141590 2024-03-31T15:04:41.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:41.276+0000 7fe97c800640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fe97416c500 con 0x7fe974144340 2024-03-31T15:04:41.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:41.276+0000 7fe97c800640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fe97416c800 con 0x7fe9741411b0 2024-03-31T15:04:41.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:41.276+0000 7fe979400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe974141590 0x7fe97413cb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:41.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:41.276+0000 7fe979400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe974141590 0x7fe97413cb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54218/0 (socket says 172.21.15.16:54218) 2024-03-31T15:04:41.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:41.276+0000 7fe979400640 1 -- 172.21.15.16:0/418762217 learned_addr learned my addr 172.21.15.16:0/418762217 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:04:41.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:41.276+0000 7fe979e00640 1 --2- 172.21.15.16:0/418762217 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fe9741411b0 0x7fe9741436c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:41.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:41.276+0000 7fe97a800640 1 --2- 172.21.15.16:0/418762217 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe974144340 0x7fe97413d060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:41.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:41.276+0000 7fe979400640 1 -- 172.21.15.16:0/418762217 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fe9741411b0 msgr2=0x7fe9741436c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:41.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:41.276+0000 7fe979400640 1 --2- 172.21.15.16:0/418762217 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fe9741411b0 0x7fe9741436c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:41.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:41.276+0000 7fe979400640 1 -- 172.21.15.16:0/418762217 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe974144340 msgr2=0x7fe97413d060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:41.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:41.276+0000 7fe979400640 1 --2- 172.21.15.16:0/418762217 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe974144340 0x7fe97413d060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:41.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:41.276+0000 7fe979400640 1 -- 172.21.15.16:0/418762217 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe974153ae0 con 0x7fe974141590 2024-03-31T15:04:41.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:41.276+0000 7fe979400640 1 --2- 172.21.15.16:0/418762217 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe974141590 0x7fe97413cb20 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fe964001ec0 tx=0x7fe96400c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:41.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:41.276+0000 7fe96e000640 1 -- 172.21.15.16:0/418762217 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe964017070 con 0x7fe974141590 2024-03-31T15:04:41.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:41.276+0000 7fe96e000640 1 -- 172.21.15.16:0/418762217 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe964005d50 con 0x7fe974141590 2024-03-31T15:04:41.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:41.276+0000 7fe96e000640 1 -- 172.21.15.16:0/418762217 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe964007270 con 0x7fe974141590 2024-03-31T15:04:41.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:41.276+0000 7fe97c800640 1 -- 172.21.15.16:0/418762217 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe974153dc0 con 0x7fe974141590 2024-03-31T15:04:41.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:41.276+0000 7fe97c800640 1 -- 172.21.15.16:0/418762217 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe974143c00 con 0x7fe974141590 2024-03-31T15:04:41.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:41.280+0000 7fe97c800640 1 -- 172.21.15.16:0/418762217 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe974069950 con 0x7fe974141590 2024-03-31T15:04:41.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:41.280+0000 7fe96e000640 1 -- 172.21.15.16:0/418762217 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fe96401a070 con 0x7fe974141590 2024-03-31T15:04:41.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:41.280+0000 7fe96e000640 1 --2- 172.21.15.16:0/418762217 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fe958068380 0x7fe95806a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:41.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:41.280+0000 7fe96e000640 1 -- 172.21.15.16:0/418762217 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5709+0+0 (secure 0 0 0) 0x7fe964085c70 con 0x7fe974141590 2024-03-31T15:04:41.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:41.280+0000 7fe979e00640 1 --2- 172.21.15.16:0/418762217 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fe958068380 0x7fe95806a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:41.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:41.284+0000 7fe979e00640 1 --2- 172.21.15.16:0/418762217 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fe958068380 0x7fe95806a840 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fe9680ad000 tx=0x7fe9680a6000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:41.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:41.284+0000 7fe96e000640 1 -- 172.21.15.16:0/418762217 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fe964055c90 con 0x7fe974141590 2024-03-31T15:04:41.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:41.556+0000 7fe97c800640 1 -- 172.21.15.16:0/418762217 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7fe9740b4a20 con 0x7fe974141590 2024-03-31T15:04:42.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:42.392+0000 7fe96e000640 1 -- 172.21.15.16:0/418762217 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v15) ==== 118+0+0 (secure 0 0 0) 0x7fe964059f90 con 0x7fe974141590 2024-03-31T15:04:42.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:42.396+0000 7fe97c800640 1 -- 172.21.15.16:0/418762217 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fe958068380 msgr2=0x7fe95806a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:42.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:42.396+0000 7fe97c800640 1 --2- 172.21.15.16:0/418762217 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fe958068380 0x7fe95806a840 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fe9680ad000 tx=0x7fe9680a6000 comp rx=0 tx=0).stop 2024-03-31T15:04:42.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:42.396+0000 7fe97c800640 1 -- 172.21.15.16:0/418762217 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe974141590 msgr2=0x7fe97413cb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:42.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:42.396+0000 7fe97c800640 1 --2- 172.21.15.16:0/418762217 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe974141590 0x7fe97413cb20 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fe964001ec0 tx=0x7fe96400c960 comp rx=0 tx=0).stop 2024-03-31T15:04:42.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:42.396+0000 7fe97c800640 1 -- 172.21.15.16:0/418762217 shutdown_connections 2024-03-31T15:04:42.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:42.396+0000 7fe97c800640 1 --2- 172.21.15.16:0/418762217 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fe958068380 0x7fe95806a840 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:42.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:42.396+0000 7fe97c800640 1 --2- 172.21.15.16:0/418762217 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe974144340 0x7fe97413d060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:42.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:42.396+0000 7fe97c800640 1 --2- 172.21.15.16:0/418762217 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe974141590 0x7fe97413cb20 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:42.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:42.396+0000 7fe97c800640 1 --2- 172.21.15.16:0/418762217 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fe9741411b0 0x7fe9741436c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:42.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:42.396+0000 7fe97c800640 1 -- 172.21.15.16:0/418762217 >> 172.21.15.16:0/418762217 conn(0x7fe974067930 msgr2=0x7fe974141bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:04:42.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:42.396+0000 7fe97c800640 1 -- 172.21.15.16:0/418762217 shutdown_connections 2024-03-31T15:04:42.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:42.396+0000 7fe97c800640 1 -- 172.21.15.16:0/418762217 wait complete. 2024-03-31T15:04:42.419 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-03-31T15:04:42.419 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-31T15:04:42.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:42.568+0000 7fd81ce00640 1 Processor -- start 2024-03-31T15:04:42.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:42.568+0000 7fd81ce00640 1 -- start start 2024-03-31T15:04:42.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:42.572+0000 7fd81ce00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fd818161e70 0x7fd8180d00a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:42.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:42.572+0000 7fd81ce00640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd818162410 con 0x7fd81806c960 2024-03-31T15:04:42.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:42.572+0000 7fd81ce00640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd8181626f0 con 0x7fd81806ecf0 2024-03-31T15:04:42.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:42.572+0000 7fd81ce00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fd8181629d0 con 0x7fd818161e70 2024-03-31T15:04:42.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:42.572+0000 7fd817400640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fd81806c960 0x7fd81806cd40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:38818/0 (socket says 172.21.15.16:38818) 2024-03-31T15:04:42.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:42.572+0000 7fd817400640 1 -- 172.21.15.16:0/4039793869 learned_addr learned my addr 172.21.15.16:0/4039793869 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:04:42.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:42.572+0000 7fd80fe00640 1 --2- 172.21.15.16:0/4039793869 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fd818161e70 0x7fd8180d00a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:42.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:42.572+0000 7fd816a00640 1 -- 172.21.15.16:0/4039793869 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3387402733 0 0) 0x7fd818162410 con 0x7fd81806c960 2024-03-31T15:04:42.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:42.572+0000 7fd816a00640 1 -- 172.21.15.16:0/4039793869 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd7fc003530 con 0x7fd81806c960 2024-03-31T15:04:42.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:42.572+0000 7fd816a00640 1 -- 172.21.15.16:0/4039793869 <== mon.1 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 36282990 0 0) 0x7fd8181626f0 con 0x7fd81806ecf0 2024-03-31T15:04:42.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:42.572+0000 7fd80fe00640 1 -- 172.21.15.16:0/4039793869 >> v1:172.21.15.89:6789/0 conn(0x7fd81806ecf0 legacy=0x7fd818072910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:42.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:42.572+0000 7fd80fe00640 1 -- 172.21.15.16:0/4039793869 >> v1:172.21.15.16:6789/0 conn(0x7fd81806c960 legacy=0x7fd81806cd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:42.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:42.572+0000 7fd80fe00640 1 -- 172.21.15.16:0/4039793869 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd818162cb0 con 0x7fd818161e70 2024-03-31T15:04:42.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:42.576+0000 7fd80fe00640 1 --2- 172.21.15.16:0/4039793869 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fd818161e70 0x7fd8180d00a0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fd8000097f0 tx=0x7fd80001c420 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c4a22d03d3ed27b2 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:42.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:42.576+0000 7fd816a00640 1 -- 172.21.15.16:0/4039793869 <== mon.2 v2:172.21.15.89:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8000b0070 con 0x7fd818161e70 2024-03-31T15:04:42.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:42.576+0000 7fd816a00640 1 -- 172.21.15.16:0/4039793869 <== mon.2 v2:172.21.15.89:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd8000ab600 con 0x7fd818161e70 2024-03-31T15:04:42.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:42.576+0000 7fd816a00640 1 -- 172.21.15.16:0/4039793869 <== mon.2 v2:172.21.15.89:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8000ab9b0 con 0x7fd818161e70 2024-03-31T15:04:42.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:42.576+0000 7fd81ce00640 1 -- 172.21.15.16:0/4039793869 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fd818161e70 msgr2=0x7fd8180d00a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:42.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:42.576+0000 7fd81ce00640 1 --2- 172.21.15.16:0/4039793869 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fd818161e70 0x7fd8180d00a0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fd8000097f0 tx=0x7fd80001c420 comp rx=0 tx=0).stop 2024-03-31T15:04:42.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:42.576+0000 7fd81ce00640 1 -- 172.21.15.16:0/4039793869 shutdown_connections 2024-03-31T15:04:42.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:42.576+0000 7fd81ce00640 1 --2- 172.21.15.16:0/4039793869 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fd818161e70 0x7fd8180d00a0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:42.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:42.576+0000 7fd81ce00640 1 -- 172.21.15.16:0/4039793869 >> 172.21.15.16:0/4039793869 conn(0x7fd8180b2c90 msgr2=0x7fd8180b30a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:04:42.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:42.576+0000 7fd81ce00640 1 -- 172.21.15.16:0/4039793869 shutdown_connections 2024-03-31T15:04:42.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:42.576+0000 7fd81ce00640 1 -- 172.21.15.16:0/4039793869 wait complete. 2024-03-31T15:04:42.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:42.580+0000 7fd81ce00640 1 Processor -- start 2024-03-31T15:04:42.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:42.580+0000 7fd81ce00640 1 -- start start 2024-03-31T15:04:42.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:42.580+0000 7fd81ce00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd81806c960 0x7fd81815f850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:42.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:42.580+0000 7fd81ce00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd81806ecf0 0x7fd818158100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:42.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:42.580+0000 7fd81ce00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fd818161e70 0x7fd818151890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:42.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:42.580+0000 7fd81ce00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fd81816ca30 con 0x7fd81806c960 2024-03-31T15:04:42.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:42.580+0000 7fd81ce00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fd818073430 con 0x7fd81806ecf0 2024-03-31T15:04:42.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:42.580+0000 7fd81ce00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fd81816cbb0 con 0x7fd818161e70 2024-03-31T15:04:42.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:42.580+0000 7fd817400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd81806c960 0x7fd81815f850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:42.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:42.580+0000 7fd817400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd81806c960 0x7fd81815f850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54220/0 (socket says 172.21.15.16:54220) 2024-03-31T15:04:42.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:42.580+0000 7fd817400640 1 -- 172.21.15.16:0/1547152213 learned_addr learned my addr 172.21.15.16:0/1547152213 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:04:42.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:42.580+0000 7fd817e00640 1 --2- 172.21.15.16:0/1547152213 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fd818161e70 0x7fd818151890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:42.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:42.580+0000 7fd80fe00640 1 --2- 172.21.15.16:0/1547152213 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd81806ecf0 0x7fd818158100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:42.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:42.580+0000 7fd817400640 1 -- 172.21.15.16:0/1547152213 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fd818161e70 msgr2=0x7fd818151890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:42.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:42.580+0000 7fd817400640 1 --2- 172.21.15.16:0/1547152213 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fd818161e70 0x7fd818151890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:42.577 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:42.580+0000 7fd817400640 1 -- 172.21.15.16:0/1547152213 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd81806ecf0 msgr2=0x7fd818158100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:42.577 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:42.580+0000 7fd817400640 1 --2- 172.21.15.16:0/1547152213 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd81806ecf0 0x7fd818158100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:42.577 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:42.580+0000 7fd817400640 1 -- 172.21.15.16:0/1547152213 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd818158640 con 0x7fd81806c960 2024-03-31T15:04:42.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:42.580+0000 7fd817400640 1 --2- 172.21.15.16:0/1547152213 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd81806c960 0x7fd81815f850 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fd80800c450 tx=0x7fd80800c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:42.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:42.580+0000 7fd80f400640 1 -- 172.21.15.16:0/1547152213 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd808017070 con 0x7fd81806c960 2024-03-31T15:04:42.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:42.584+0000 7fd80f400640 1 -- 172.21.15.16:0/1547152213 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd808005cf0 con 0x7fd81806c960 2024-03-31T15:04:42.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:42.584+0000 7fd80f400640 1 -- 172.21.15.16:0/1547152213 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd808007290 con 0x7fd81806c960 2024-03-31T15:04:42.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:42.584+0000 7fd81ce00640 1 -- 172.21.15.16:0/1547152213 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd81815e7b0 con 0x7fd81806c960 2024-03-31T15:04:42.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:42.584+0000 7fd81ce00640 1 -- 172.21.15.16:0/1547152213 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd81815ed70 con 0x7fd81806c960 2024-03-31T15:04:42.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:42.584+0000 7fd81ce00640 1 -- 172.21.15.16:0/1547152213 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd81804f8c0 con 0x7fd81806c960 2024-03-31T15:04:42.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:42.584+0000 7fd80f400640 1 -- 172.21.15.16:0/1547152213 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fd80801a070 con 0x7fd81806c960 2024-03-31T15:04:42.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:42.584+0000 7fd80f400640 1 --2- 172.21.15.16:0/1547152213 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fd7f4068380 0x7fd7f406a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:42.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:42.584+0000 7fd80f400640 1 -- 172.21.15.16:0/1547152213 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5709+0+0 (secure 0 0 0) 0x7fd808085cf0 con 0x7fd81806c960 2024-03-31T15:04:42.580 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:42.584+0000 7fd80fe00640 1 --2- 172.21.15.16:0/1547152213 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fd7f4068380 0x7fd7f406a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:42.580 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:42.588+0000 7fd80fe00640 1 --2- 172.21.15.16:0/1547152213 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fd7f4068380 0x7fd7f406a840 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fd8000ad000 tx=0x7fd8000a6000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:42.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:42.588+0000 7fd80f400640 1 -- 172.21.15.16:0/1547152213 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fd808055c50 con 0x7fd81806c960 2024-03-31T15:04:42.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:42.844+0000 7fd81ce00640 1 -- 172.21.15.16:0/1547152213 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7fd8180ce410 con 0x7fd81806c960 2024-03-31T15:04:43.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:43.396+0000 7fd80f400640 1 -- 172.21.15.16:0/1547152213 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v16) ==== 122+0+0 (secure 0 0 0) 0x7fd808059f50 con 0x7fd81806c960 2024-03-31T15:04:43.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:43.400+0000 7fd81ce00640 1 -- 172.21.15.16:0/1547152213 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fd7f4068380 msgr2=0x7fd7f406a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:43.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:43.400+0000 7fd81ce00640 1 --2- 172.21.15.16:0/1547152213 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fd7f4068380 0x7fd7f406a840 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fd8000ad000 tx=0x7fd8000a6000 comp rx=0 tx=0).stop 2024-03-31T15:04:43.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:43.400+0000 7fd81ce00640 1 -- 172.21.15.16:0/1547152213 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd81806c960 msgr2=0x7fd81815f850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:43.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:43.400+0000 7fd81ce00640 1 --2- 172.21.15.16:0/1547152213 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd81806c960 0x7fd81815f850 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fd80800c450 tx=0x7fd80800c910 comp rx=0 tx=0).stop 2024-03-31T15:04:43.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:43.400+0000 7fd81ce00640 1 -- 172.21.15.16:0/1547152213 shutdown_connections 2024-03-31T15:04:43.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:43.400+0000 7fd81ce00640 1 --2- 172.21.15.16:0/1547152213 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fd7f4068380 0x7fd7f406a840 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:43.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:43.400+0000 7fd81ce00640 1 --2- 172.21.15.16:0/1547152213 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fd818161e70 0x7fd818151890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:43.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:43.400+0000 7fd81ce00640 1 --2- 172.21.15.16:0/1547152213 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd81806ecf0 0x7fd818158100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:43.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:43.400+0000 7fd81ce00640 1 --2- 172.21.15.16:0/1547152213 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd81806c960 0x7fd81815f850 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:43.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:43.400+0000 7fd81ce00640 1 -- 172.21.15.16:0/1547152213 >> 172.21.15.16:0/1547152213 conn(0x7fd8180b2c90 msgr2=0x7fd8180b5bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:04:43.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:43.400+0000 7fd81ce00640 1 -- 172.21.15.16:0/1547152213 shutdown_connections 2024-03-31T15:04:43.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:43.400+0000 7fd81ce00640 1 -- 172.21.15.16:0/1547152213 wait complete. 2024-03-31T15:04:43.415 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-31T15:04:43.415 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-31T15:04:43.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:43.560+0000 7fc128200640 1 Processor -- start 2024-03-31T15:04:43.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:43.560+0000 7fc128200640 1 -- start start 2024-03-31T15:04:43.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:43.564+0000 7fc128200640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fc12006dbc0 0x7fc12015f650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:43.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:43.564+0000 7fc128200640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fc120070030 0x7fc120162180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:43.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:43.564+0000 7fc128200640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc1200cbe90 con 0x7fc1200cb440 2024-03-31T15:04:43.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:43.564+0000 7fc128200640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7fc12015fb90 con 0x7fc120070030 2024-03-31T15:04:43.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:43.564+0000 7fc128200640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fc12015f1c0 con 0x7fc12006dbc0 2024-03-31T15:04:43.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:43.564+0000 7fc124e00640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fc1200cb440 0x7fc12006f920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:38824/0 (socket says 172.21.15.16:38824) 2024-03-31T15:04:43.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:43.564+0000 7fc124e00640 1 -- 172.21.15.16:0/3628165052 learned_addr learned my addr 172.21.15.16:0/3628165052 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:04:43.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:43.564+0000 7fc125800640 1 --2- 172.21.15.16:0/3628165052 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fc12006dbc0 0x7fc12015f650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:43.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:43.564+0000 7fc126200640 1 --2- 172.21.15.16:0/3628165052 >> v2:172.21.15.89:3300/0 conn(0x7fc120070030 0x7fc120162180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:43.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:43.564+0000 7fc11fe00640 1 -- 172.21.15.16:0/3628165052 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3192147707 0 0) 0x7fc1200cbe90 con 0x7fc1200cb440 2024-03-31T15:04:43.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:43.564+0000 7fc11fe00640 1 -- 172.21.15.16:0/3628165052 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc108003530 con 0x7fc1200cb440 2024-03-31T15:04:43.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:43.564+0000 7fc11fe00640 1 -- 172.21.15.16:0/3628165052 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3005201870 0 0) 0x7fc108003530 con 0x7fc1200cb440 2024-03-31T15:04:43.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:43.564+0000 7fc126200640 1 -- 172.21.15.16:0/3628165052 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fc12006dbc0 msgr2=0x7fc12015f650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:04:43.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:43.564+0000 7fc126200640 1 --2- 172.21.15.16:0/3628165052 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fc12006dbc0 0x7fc12015f650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:43.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:43.564+0000 7fc126200640 1 -- 172.21.15.16:0/3628165052 >> v1:172.21.15.16:6789/0 conn(0x7fc1200cb440 legacy=0x7fc12006f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:43.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:43.564+0000 7fc126200640 1 -- 172.21.15.16:0/3628165052 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc1200cc170 con 0x7fc120070030 2024-03-31T15:04:43.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:43.564+0000 7fc125800640 1 --2- 172.21.15.16:0/3628165052 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fc12006dbc0 0x7fc12015f650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:04:43.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:43.568+0000 7fc126200640 1 --2- 172.21.15.16:0/3628165052 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc120070030 0x7fc120162180 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fc118009d90 tx=0x7fc11801c7d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=639b30f0b4e8e56f server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:43.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:43.568+0000 7fc11fe00640 1 -- 172.21.15.16:0/3628165052 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1180b2070 con 0x7fc120070030 2024-03-31T15:04:43.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:43.568+0000 7fc11fe00640 1 -- 172.21.15.16:0/3628165052 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc1180a55f0 con 0x7fc120070030 2024-03-31T15:04:43.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:43.568+0000 7fc11fe00640 1 -- 172.21.15.16:0/3628165052 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc10c001680 con 0x7fc120070030 2024-03-31T15:04:43.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:43.568+0000 7fc128200640 1 -- 172.21.15.16:0/3628165052 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc120070030 msgr2=0x7fc120162180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:43.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:43.568+0000 7fc128200640 1 --2- 172.21.15.16:0/3628165052 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc120070030 0x7fc120162180 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fc118009d90 tx=0x7fc11801c7d0 comp rx=0 tx=0).stop 2024-03-31T15:04:43.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:43.568+0000 7fc128200640 1 -- 172.21.15.16:0/3628165052 shutdown_connections 2024-03-31T15:04:43.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:43.568+0000 7fc128200640 1 --2- 172.21.15.16:0/3628165052 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc120070030 0x7fc120162180 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:43.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:43.568+0000 7fc128200640 1 --2- 172.21.15.16:0/3628165052 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fc12006dbc0 0x7fc12015f650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:43.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:43.568+0000 7fc128200640 1 -- 172.21.15.16:0/3628165052 >> 172.21.15.16:0/3628165052 conn(0x7fc120067330 msgr2=0x7fc120067740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:04:43.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:43.568+0000 7fc128200640 1 -- 172.21.15.16:0/3628165052 shutdown_connections 2024-03-31T15:04:43.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:43.568+0000 7fc128200640 1 -- 172.21.15.16:0/3628165052 wait complete. 2024-03-31T15:04:43.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:43.568+0000 7fc128200640 1 Processor -- start 2024-03-31T15:04:43.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:43.568+0000 7fc128200640 1 -- start start 2024-03-31T15:04:43.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:43.572+0000 7fc128200640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc12006dbc0 0x7fc1200b58a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:43.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:43.572+0000 7fc128200640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fc120070030 0x7fc120154d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:43.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:43.572+0000 7fc128200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc1200cb440 0x7fc1201f8860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:43.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:43.572+0000 7fc128200640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fc12006cad0 con 0x7fc1200cb440 2024-03-31T15:04:43.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:43.572+0000 7fc128200640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fc12006c950 con 0x7fc12006dbc0 2024-03-31T15:04:43.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:43.572+0000 7fc128200640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fc12016b520 con 0x7fc120070030 2024-03-31T15:04:43.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:43.572+0000 7fc126200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc1200cb440 0x7fc1201f8860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:43.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:43.572+0000 7fc126200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc1200cb440 0x7fc1201f8860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54224/0 (socket says 172.21.15.16:54224) 2024-03-31T15:04:43.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:43.572+0000 7fc126200640 1 -- 172.21.15.16:0/1630239998 learned_addr learned my addr 172.21.15.16:0/1630239998 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:04:43.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:43.572+0000 7fc124e00640 1 --2- 172.21.15.16:0/1630239998 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fc120070030 0x7fc120154d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:43.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:43.572+0000 7fc125800640 1 --2- 172.21.15.16:0/1630239998 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc12006dbc0 0x7fc1200b58a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:43.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:43.572+0000 7fc126200640 1 -- 172.21.15.16:0/1630239998 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fc120070030 msgr2=0x7fc120154d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:43.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:43.572+0000 7fc126200640 1 --2- 172.21.15.16:0/1630239998 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fc120070030 0x7fc120154d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:43.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:43.572+0000 7fc126200640 1 -- 172.21.15.16:0/1630239998 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc12006dbc0 msgr2=0x7fc1200b58a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:43.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:43.572+0000 7fc126200640 1 --2- 172.21.15.16:0/1630239998 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc12006dbc0 0x7fc1200b58a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:43.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:43.572+0000 7fc126200640 1 -- 172.21.15.16:0/1630239998 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc1200b63b0 con 0x7fc1200cb440 2024-03-31T15:04:43.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:43.572+0000 7fc125800640 1 --2- 172.21.15.16:0/1630239998 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc12006dbc0 0x7fc1200b58a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:04:43.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:43.572+0000 7fc124e00640 1 --2- 172.21.15.16:0/1630239998 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fc120070030 0x7fc120154d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:04:43.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:43.572+0000 7fc126200640 1 --2- 172.21.15.16:0/1630239998 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc1200cb440 0x7fc1201f8860 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fc1180b0040 tx=0x7fc118019570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:43.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:43.572+0000 7fc11d600640 1 -- 172.21.15.16:0/1630239998 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1180b2050 con 0x7fc1200cb440 2024-03-31T15:04:43.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:43.572+0000 7fc11d600640 1 -- 172.21.15.16:0/1630239998 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc1180be070 con 0x7fc1200cb440 2024-03-31T15:04:43.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:43.572+0000 7fc11d600640 1 -- 172.21.15.16:0/1630239998 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc118009450 con 0x7fc1200cb440 2024-03-31T15:04:43.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:43.572+0000 7fc128200640 1 -- 172.21.15.16:0/1630239998 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc1200b60d0 con 0x7fc1200cb440 2024-03-31T15:04:43.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:43.572+0000 7fc128200640 1 -- 172.21.15.16:0/1630239998 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc1200b6b60 con 0x7fc1200cb440 2024-03-31T15:04:43.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:43.572+0000 7fc128200640 1 -- 172.21.15.16:0/1630239998 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc120069950 con 0x7fc1200cb440 2024-03-31T15:04:43.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:43.572+0000 7fc11d600640 1 -- 172.21.15.16:0/1630239998 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fc1180ada90 con 0x7fc1200cb440 2024-03-31T15:04:43.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:43.576+0000 7fc11d600640 1 --2- 172.21.15.16:0/1630239998 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fc100068380 0x7fc10006a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:43.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:43.576+0000 7fc11d600640 1 -- 172.21.15.16:0/1630239998 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5709+0+0 (secure 0 0 0) 0x7fc1181219a0 con 0x7fc1200cb440 2024-03-31T15:04:43.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:43.576+0000 7fc125800640 1 --2- 172.21.15.16:0/1630239998 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fc100068380 0x7fc10006a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:43.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:43.576+0000 7fc125800640 1 --2- 172.21.15.16:0/1630239998 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fc100068380 0x7fc10006a840 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fc120157d20 tx=0x7fc114008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:43.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:43.584+0000 7fc11d600640 1 -- 172.21.15.16:0/1630239998 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fc118016480 con 0x7fc1200cb440 2024-03-31T15:04:43.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:43.780+0000 7fc128200640 1 -- 172.21.15.16:0/1630239998 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc1200cf380 con 0x7fc1200cb440 2024-03-31T15:04:43.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:43.780+0000 7fc11d600640 1 -- 172.21.15.16:0/1630239998 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) ==== 94+0+5178 (secure 0 0 0) 0x7fc1180f1f50 con 0x7fc1200cb440 2024-03-31T15:04:43.776 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:04:43.777 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":16,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4949,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6836/2073954753","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":2073954753},{"type":"v1","addr":"172.21.15.89:6836","nonce":2073954753}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":4952,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/2143918278","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6835","nonce":2143918278},{"type":"v1","addr":"172.21.15.89:6837","nonce":2143918278}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":5251,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6833/881391342","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":881391342},{"type":"v1","addr":"172.21.15.16:6833","nonce":881391342}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-31T15:04:39.780323+0000","modified":"2024-03-31T15:04:42.846439+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5254},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5254":{"gid":5254,"name":"c","rank":0,"incarnation":13,"state":"up:active","state_seq":3,"addr":"172.21.15.16:6835/1845707550","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":1845707550},{"type":"v1","addr":"172.21.15.16:6835","nonce":1845707550}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5254,"qdb_cluster":[5254]},"id":2}]} 2024-03-31T15:04:43.777 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 16 2024-03-31T15:04:43.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:43.784+0000 7fc128200640 1 -- 172.21.15.16:0/1630239998 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fc100068380 msgr2=0x7fc10006a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:43.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:43.784+0000 7fc128200640 1 --2- 172.21.15.16:0/1630239998 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fc100068380 0x7fc10006a840 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fc120157d20 tx=0x7fc114008040 comp rx=0 tx=0).stop 2024-03-31T15:04:43.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:43.784+0000 7fc128200640 1 -- 172.21.15.16:0/1630239998 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc1200cb440 msgr2=0x7fc1201f8860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:43.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:43.784+0000 7fc128200640 1 --2- 172.21.15.16:0/1630239998 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc1200cb440 0x7fc1201f8860 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fc1180b0040 tx=0x7fc118019570 comp rx=0 tx=0).stop 2024-03-31T15:04:43.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:43.784+0000 7fc128200640 1 -- 172.21.15.16:0/1630239998 shutdown_connections 2024-03-31T15:04:43.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:43.784+0000 7fc128200640 1 --2- 172.21.15.16:0/1630239998 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fc100068380 0x7fc10006a840 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:43.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:43.784+0000 7fc128200640 1 --2- 172.21.15.16:0/1630239998 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc1200cb440 0x7fc1201f8860 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:43.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:43.784+0000 7fc128200640 1 --2- 172.21.15.16:0/1630239998 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fc120070030 0x7fc120154d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:43.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:43.784+0000 7fc128200640 1 --2- 172.21.15.16:0/1630239998 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc12006dbc0 0x7fc1200b58a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:43.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:43.784+0000 7fc128200640 1 -- 172.21.15.16:0/1630239998 >> 172.21.15.16:0/1630239998 conn(0x7fc120067330 msgr2=0x7fc12015d010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:04:43.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:43.784+0000 7fc128200640 1 -- 172.21.15.16:0/1630239998 shutdown_connections 2024-03-31T15:04:43.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:43.784+0000 7fc128200640 1 -- 172.21.15.16:0/1630239998 wait complete. 2024-03-31T15:04:43.798 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-31T15:04:43.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:43.944+0000 7f8064400640 1 Processor -- start 2024-03-31T15:04:43.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:43.944+0000 7f8064400640 1 -- start start 2024-03-31T15:04:43.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:43.944+0000 7f8064400640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f805c145e40 0x7f805c0cc740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:43.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:43.944+0000 7f8064400640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f805c0d0920 con 0x7f805c070030 2024-03-31T15:04:43.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:43.944+0000 7f8064400640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f805c0d0c00 con 0x7f805c0ccc80 2024-03-31T15:04:43.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:43.944+0000 7f8064400640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f805c0d06d0 con 0x7f805c145e40 2024-03-31T15:04:43.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:43.944+0000 7f8062400640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f805c070030 0x7f805c0d2010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:38834/0 (socket says 172.21.15.16:38834) 2024-03-31T15:04:43.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:43.944+0000 7f8062400640 1 -- 172.21.15.16:0/2283964590 learned_addr learned my addr 172.21.15.16:0/2283964590 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:04:43.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:43.944+0000 7f8061a00640 1 --2- 172.21.15.16:0/2283964590 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f805c145e40 0x7f805c0cc740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:43.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:43.944+0000 7f805be00640 1 -- 172.21.15.16:0/2283964590 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2355970377 0 0) 0x7f805c0d0920 con 0x7f805c070030 2024-03-31T15:04:43.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:43.944+0000 7f805be00640 1 -- 172.21.15.16:0/2283964590 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8044003530 con 0x7f805c070030 2024-03-31T15:04:43.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:43.944+0000 7f805be00640 1 -- 172.21.15.16:0/2283964590 <== mon.1 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 189685935 0 0) 0x7f805c0d0c00 con 0x7f805c0ccc80 2024-03-31T15:04:43.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:43.944+0000 7f805be00640 1 -- 172.21.15.16:0/2283964590 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f805c0d0920 con 0x7f805c0ccc80 2024-03-31T15:04:43.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:43.948+0000 7f8061a00640 1 -- 172.21.15.16:0/2283964590 >> v1:172.21.15.89:6789/0 conn(0x7f805c0ccc80 legacy=0x7f805c06f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:43.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:43.948+0000 7f8061a00640 1 -- 172.21.15.16:0/2283964590 >> v1:172.21.15.16:6789/0 conn(0x7f805c070030 legacy=0x7f805c0d2010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:43.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:43.948+0000 7f805be00640 1 -- 172.21.15.16:0/2283964590 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3835991431 0 0) 0x7f8044003530 con 0x7f805c070030 2024-03-31T15:04:43.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:43.948+0000 7f8061a00640 1 -- 172.21.15.16:0/2283964590 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f805c0d1810 con 0x7f805c145e40 2024-03-31T15:04:43.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:43.948+0000 7f8061a00640 1 --2- 172.21.15.16:0/2283964590 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f805c145e40 0x7f805c0cc740 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f80500097f0 tx=0x7f805001c400 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3f858622303d31ee server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:43.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:43.948+0000 7f805be00640 1 -- 172.21.15.16:0/2283964590 <== mon.2 v2:172.21.15.89:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80500b0070 con 0x7f805c145e40 2024-03-31T15:04:43.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:43.948+0000 7f805be00640 1 -- 172.21.15.16:0/2283964590 <== mon.2 v2:172.21.15.89:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f80500ab400 con 0x7f805c145e40 2024-03-31T15:04:43.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:43.948+0000 7f805be00640 1 -- 172.21.15.16:0/2283964590 <== mon.2 v2:172.21.15.89:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8054001a00 con 0x7f805c145e40 2024-03-31T15:04:43.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:43.948+0000 7f8064400640 1 -- 172.21.15.16:0/2283964590 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f805c145e40 msgr2=0x7f805c0cc740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:43.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:43.948+0000 7f8064400640 1 --2- 172.21.15.16:0/2283964590 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f805c145e40 0x7f805c0cc740 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f80500097f0 tx=0x7f805001c400 comp rx=0 tx=0).stop 2024-03-31T15:04:43.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:43.948+0000 7f8064400640 1 -- 172.21.15.16:0/2283964590 shutdown_connections 2024-03-31T15:04:43.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:43.948+0000 7f8064400640 1 --2- 172.21.15.16:0/2283964590 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f805c145e40 0x7f805c0cc740 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:43.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:43.948+0000 7f8064400640 1 -- 172.21.15.16:0/2283964590 >> 172.21.15.16:0/2283964590 conn(0x7f805c067330 msgr2=0x7f805c067740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:04:43.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:43.948+0000 7f8064400640 1 -- 172.21.15.16:0/2283964590 shutdown_connections 2024-03-31T15:04:43.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:43.948+0000 7f8064400640 1 -- 172.21.15.16:0/2283964590 wait complete. 2024-03-31T15:04:43.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:43.952+0000 7f8064400640 1 Processor -- start 2024-03-31T15:04:43.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:43.952+0000 7f8064400640 1 -- start start 2024-03-31T15:04:43.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:43.952+0000 7f8064400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f805c070030 0x7f805c06d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:43.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:43.952+0000 7f8064400640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f805c0ccc80 0x7f805c15d740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:43.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:43.952+0000 7f8064400640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f805c145e40 0x7f805c0cb440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:43.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:43.952+0000 7f8064400640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f805c0b5240 con 0x7f805c070030 2024-03-31T15:04:43.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:43.952+0000 7f8064400640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f805c0b50c0 con 0x7f805c0ccc80 2024-03-31T15:04:43.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:43.952+0000 7f8064400640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f805c16a1a0 con 0x7f805c145e40 2024-03-31T15:04:43.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:43.952+0000 7f8061a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f805c070030 0x7f805c06d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:43.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:43.952+0000 7f8061a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f805c070030 0x7f805c06d780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54236/0 (socket says 172.21.15.16:54236) 2024-03-31T15:04:43.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:43.952+0000 7f8061a00640 1 -- 172.21.15.16:0/1330923856 learned_addr learned my addr 172.21.15.16:0/1330923856 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:04:43.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:43.952+0000 7f8062400640 1 --2- 172.21.15.16:0/1330923856 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f805c145e40 0x7f805c0cb440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:43.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:43.952+0000 7f8061000640 1 --2- 172.21.15.16:0/1330923856 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f805c0ccc80 0x7f805c15d740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:43.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:43.952+0000 7f8061a00640 1 -- 172.21.15.16:0/1330923856 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f805c145e40 msgr2=0x7f805c0cb440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:43.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:43.952+0000 7f8061a00640 1 --2- 172.21.15.16:0/1330923856 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f805c145e40 0x7f805c0cb440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:43.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:43.952+0000 7f8061a00640 1 -- 172.21.15.16:0/1330923856 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f805c0ccc80 msgr2=0x7f805c15d740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:43.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:43.952+0000 7f8061a00640 1 --2- 172.21.15.16:0/1330923856 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f805c0ccc80 0x7f805c15d740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:43.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:43.952+0000 7f8061a00640 1 -- 172.21.15.16:0/1330923856 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f805c06dcc0 con 0x7f805c070030 2024-03-31T15:04:43.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:43.952+0000 7f8061000640 1 --2- 172.21.15.16:0/1330923856 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f805c0ccc80 0x7f805c15d740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:04:43.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:43.952+0000 7f8061a00640 1 --2- 172.21.15.16:0/1330923856 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f805c070030 0x7f805c06d780 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f805001a030 tx=0x7f805001ac40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:43.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:43.952+0000 7f8059600640 1 -- 172.21.15.16:0/1330923856 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80500b0050 con 0x7f805c070030 2024-03-31T15:04:43.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:43.952+0000 7f8059600640 1 -- 172.21.15.16:0/1330923856 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f80500b1040 con 0x7f805c070030 2024-03-31T15:04:43.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:43.952+0000 7f8059600640 1 -- 172.21.15.16:0/1330923856 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8050006e20 con 0x7f805c070030 2024-03-31T15:04:43.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:43.952+0000 7f8064400640 1 -- 172.21.15.16:0/1330923856 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f805c06df40 con 0x7f805c070030 2024-03-31T15:04:43.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:43.956+0000 7f8064400640 1 -- 172.21.15.16:0/1330923856 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f805c06e500 con 0x7f805c070030 2024-03-31T15:04:43.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:43.956+0000 7f8064400640 1 -- 172.21.15.16:0/1330923856 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f805c069950 con 0x7f805c070030 2024-03-31T15:04:43.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:43.956+0000 7f8059600640 1 -- 172.21.15.16:0/1330923856 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f80500be030 con 0x7f805c070030 2024-03-31T15:04:43.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:43.956+0000 7f8059600640 1 --2- 172.21.15.16:0/1330923856 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f803c068380 0x7f803c06a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:43.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:43.956+0000 7f8059600640 1 -- 172.21.15.16:0/1330923856 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5709+0+0 (secure 0 0 0) 0x7f80501204e0 con 0x7f805c070030 2024-03-31T15:04:43.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:43.960+0000 7f8061000640 1 --2- 172.21.15.16:0/1330923856 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f803c068380 0x7f803c06a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:43.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:43.960+0000 7f8061000640 1 --2- 172.21.15.16:0/1330923856 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f803c068380 0x7f803c06a840 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f805c1fa610 tx=0x7f8048009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:43.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:43.960+0000 7f8059600640 1 -- 172.21.15.16:0/1330923856 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f80500c5060 con 0x7f805c070030 2024-03-31T15:04:44.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:44.128+0000 7f8064400640 1 -- 172.21.15.16:0/1330923856 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f805c15bc50 con 0x7f805c070030 2024-03-31T15:04:44.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:44.128+0000 7f8059600640 1 -- 172.21.15.16:0/1330923856 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v34) ==== 74+0+16597 (secure 0 0 0) 0x7f805c06dcc0 con 0x7f805c070030 2024-03-31T15:04:44.123 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:04:44.123 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":34,"fsid":"286eb5da-fb47-4550-b290-d9519c13b9a5","created":"2024-03-31T15:03:11.438952+0000","modified":"2024-03-31T15:04:41.385382+0000","last_up_change":"2024-03-31T15:03:22.885535+0000","last_in_change":"2024-03-31T15:03:13.941667+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-31T15:03:22.550787+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-03-31T15:04:37.577341+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-03-31T15:04:38.738566+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"34","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"92a66450-ef19-40d8-ac3d-d760b23b0bda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6801","nonce":932712244}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6803","nonce":932712244}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6807","nonce":932712244}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6805","nonce":932712244}]},"public_addr":"172.21.15.16:6801/932712244","cluster_addr":"172.21.15.16:6803/932712244","heartbeat_back_addr":"172.21.15.16:6807/932712244","heartbeat_front_addr":"172.21.15.16:6805/932712244","state":["exists","up"]},{"osd":1,"uuid":"da9d1f2b-1968-4959-b843-faa231988e92","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6810","nonce":1398737533}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6811","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6813","nonce":1398737533}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6821","nonce":1398737533}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6815","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6816","nonce":1398737533}]},"public_addr":"172.21.15.16:6810/1398737533","cluster_addr":"172.21.15.16:6813/1398737533","heartbeat_back_addr":"172.21.15.16:6821/1398737533","heartbeat_front_addr":"172.21.15.16:6816/1398737533","state":["exists","up"]},{"osd":2,"uuid":"01a73662-9a21-4b1f-bcbb-42b9636a4638","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6819","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6822","nonce":3018526646}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6825","nonce":3018526646}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6831","nonce":3018526646}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6827","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6829","nonce":3018526646}]},"public_addr":"172.21.15.16:6822/3018526646","cluster_addr":"172.21.15.16:6825/3018526646","heartbeat_back_addr":"172.21.15.16:6831/3018526646","heartbeat_front_addr":"172.21.15.16:6829/3018526646","state":["exists","up"]},{"osd":3,"uuid":"c7771fd3-e18e-4500-ac38-23cf10ebcd75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6809","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6812","nonce":3360352974}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6817","nonce":3360352974}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6828","nonce":3360352974}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6823","nonce":3360352974}]},"public_addr":"172.21.15.16:6812/3360352974","cluster_addr":"172.21.15.16:6817/3360352974","heartbeat_back_addr":"172.21.15.16:6828/3360352974","heartbeat_front_addr":"172.21.15.16:6823/3360352974","state":["exists","up"]},{"osd":4,"uuid":"88260730-1482-4fbd-a40f-cb1124e9df4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6801","nonce":1856844856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6803","nonce":1856844856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6807","nonce":1856844856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6805","nonce":1856844856}]},"public_addr":"172.21.15.89:6801/1856844856","cluster_addr":"172.21.15.89:6803/1856844856","heartbeat_back_addr":"172.21.15.89:6807/1856844856","heartbeat_front_addr":"172.21.15.89:6805/1856844856","state":["exists","up"]},{"osd":5,"uuid":"4cb2c00c-fae0-4dd8-9d27-ec4f0093ccf7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6814","nonce":50395527}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6817","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6820","nonce":50395527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6831","nonce":50395527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6823","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6826","nonce":50395527}]},"public_addr":"172.21.15.89:6814/50395527","cluster_addr":"172.21.15.89:6820/50395527","heartbeat_back_addr":"172.21.15.89:6831/50395527","heartbeat_front_addr":"172.21.15.89:6826/50395527","state":["exists","up"]},{"osd":6,"uuid":"81ce0c19-2554-4cae-acbe-66a05e9e9b9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6809","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6812","nonce":3470471752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6815","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6818","nonce":3470471752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6827","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6829","nonce":3470471752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6821","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6824","nonce":3470471752}]},"public_addr":"172.21.15.89:6812/3470471752","cluster_addr":"172.21.15.89:6818/3470471752","heartbeat_back_addr":"172.21.15.89:6829/3470471752","heartbeat_front_addr":"172.21.15.89:6824/3470471752","state":["exists","up"]},{"osd":7,"uuid":"85e4313b-d37d-442e-8042-153bf20a847d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6811","nonce":1796778630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6813","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6816","nonce":1796778630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6825","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6828","nonce":1796778630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6819","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6822","nonce":1796778630}]},"public_addr":"172.21.15.89:6811/1796778630","cluster_addr":"172.21.15.89:6816/1796778630","heartbeat_back_addr":"172.21.15.89:6828/1796778630","heartbeat_front_addr":"172.21.15.89:6822/1796778630","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.439088+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.529970+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.680512+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:21.044161+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.582111+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.536558+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.409522+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.590229+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-31T15:04:44.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:44.132+0000 7f8064400640 1 -- 172.21.15.16:0/1330923856 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f803c068380 msgr2=0x7f803c06a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:44.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:44.132+0000 7f8064400640 1 --2- 172.21.15.16:0/1330923856 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f803c068380 0x7f803c06a840 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f805c1fa610 tx=0x7f8048009040 comp rx=0 tx=0).stop 2024-03-31T15:04:44.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:44.132+0000 7f8064400640 1 -- 172.21.15.16:0/1330923856 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f805c070030 msgr2=0x7f805c06d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:44.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:44.132+0000 7f8064400640 1 --2- 172.21.15.16:0/1330923856 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f805c070030 0x7f805c06d780 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f805001a030 tx=0x7f805001ac40 comp rx=0 tx=0).stop 2024-03-31T15:04:44.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:44.132+0000 7f8064400640 1 -- 172.21.15.16:0/1330923856 shutdown_connections 2024-03-31T15:04:44.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:44.132+0000 7f8064400640 1 --2- 172.21.15.16:0/1330923856 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f803c068380 0x7f803c06a840 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:44.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:44.132+0000 7f8064400640 1 --2- 172.21.15.16:0/1330923856 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f805c145e40 0x7f805c0cb440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:44.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:44.132+0000 7f8064400640 1 --2- 172.21.15.16:0/1330923856 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f805c0ccc80 0x7f805c15d740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:44.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:44.132+0000 7f8064400640 1 --2- 172.21.15.16:0/1330923856 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f805c070030 0x7f805c06d780 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:44.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:44.132+0000 7f8064400640 1 -- 172.21.15.16:0/1330923856 >> 172.21.15.16:0/1330923856 conn(0x7f805c067330 msgr2=0x7f805c067710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:04:44.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:44.132+0000 7f8064400640 1 -- 172.21.15.16:0/1330923856 shutdown_connections 2024-03-31T15:04:44.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:44.132+0000 7f8064400640 1 -- 172.21.15.16:0/1330923856 wait complete. 2024-03-31T15:04:44.145 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-31T15:04:44.145 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-31T15:04:44.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:44.292+0000 7f2366a00640 1 Processor -- start 2024-03-31T15:04:44.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:44.292+0000 7f2366a00640 1 -- start start 2024-03-31T15:04:44.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:44.292+0000 7f2366a00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f2360145d10 0x7f23600cf790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:44.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:44.292+0000 7f2366a00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f23600b23d0 0x7f236015cd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:44.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:44.292+0000 7f2366a00640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2360160e70 con 0x7f23600728b0 2024-03-31T15:04:44.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:44.292+0000 7f2366a00640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f23601602f0 con 0x7f23600b23d0 2024-03-31T15:04:44.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:44.292+0000 7f2366a00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f23601605d0 con 0x7f2360145d10 2024-03-31T15:04:44.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:44.292+0000 7f235f400640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f23600728b0 0x7f23600b1cc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:38842/0 (socket says 172.21.15.16:38842) 2024-03-31T15:04:44.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:44.292+0000 7f235f400640 1 -- 172.21.15.16:0/2407587247 learned_addr learned my addr 172.21.15.16:0/2407587247 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:04:44.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:44.292+0000 7f2364a00640 1 --2- 172.21.15.16:0/2407587247 >> v2:172.21.15.89:3300/0 conn(0x7f23600b23d0 0x7f236015cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:44.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:44.292+0000 7f235fe00640 1 --2- 172.21.15.16:0/2407587247 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f2360145d10 0x7f23600cf790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:44.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:44.292+0000 7f235ea00640 1 -- 172.21.15.16:0/2407587247 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1675707909 0 0) 0x7f2360160e70 con 0x7f23600728b0 2024-03-31T15:04:44.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:44.292+0000 7f235ea00640 1 -- 172.21.15.16:0/2407587247 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2344003530 con 0x7f23600728b0 2024-03-31T15:04:44.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:44.292+0000 7f2364a00640 1 -- 172.21.15.16:0/2407587247 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f2360145d10 msgr2=0x7f23600cf790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:04:44.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:44.292+0000 7f2364a00640 1 --2- 172.21.15.16:0/2407587247 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f2360145d10 0x7f23600cf790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:44.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:44.292+0000 7f2364a00640 1 -- 172.21.15.16:0/2407587247 >> v1:172.21.15.16:6789/0 conn(0x7f23600728b0 legacy=0x7f23600b1cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:44.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:44.292+0000 7f235ea00640 1 -- 172.21.15.16:0/2407587247 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2248586720 0 0) 0x7f2344003530 con 0x7f23600728b0 2024-03-31T15:04:44.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:44.292+0000 7f2364a00640 1 -- 172.21.15.16:0/2407587247 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f23600cfcd0 con 0x7f23600b23d0 2024-03-31T15:04:44.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:44.296+0000 7f2364a00640 1 --2- 172.21.15.16:0/2407587247 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f23600b23d0 0x7f236015cd50 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f2358009d90 tx=0x7f235801c890 comp rx=0 tx=0).ready entity=mon.1 client_cookie=30720754dab75290 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:44.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:44.296+0000 7f235ea00640 1 -- 172.21.15.16:0/2407587247 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23580ac070 con 0x7f23600b23d0 2024-03-31T15:04:44.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:44.296+0000 7f235ea00640 1 -- 172.21.15.16:0/2407587247 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f23580a64a0 con 0x7f23600b23d0 2024-03-31T15:04:44.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:44.296+0000 7f235ea00640 1 -- 172.21.15.16:0/2407587247 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f234c001680 con 0x7f23600b23d0 2024-03-31T15:04:44.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:44.296+0000 7f2366a00640 1 -- 172.21.15.16:0/2407587247 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f23600b23d0 msgr2=0x7f236015cd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:44.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:44.296+0000 7f2366a00640 1 --2- 172.21.15.16:0/2407587247 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f23600b23d0 0x7f236015cd50 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f2358009d90 tx=0x7f235801c890 comp rx=0 tx=0).stop 2024-03-31T15:04:44.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:44.296+0000 7f2366a00640 1 -- 172.21.15.16:0/2407587247 shutdown_connections 2024-03-31T15:04:44.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:44.296+0000 7f2366a00640 1 --2- 172.21.15.16:0/2407587247 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f23600b23d0 0x7f236015cd50 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:44.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:44.296+0000 7f2366a00640 1 --2- 172.21.15.16:0/2407587247 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f2360145d10 0x7f23600cf790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:44.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:44.296+0000 7f2366a00640 1 -- 172.21.15.16:0/2407587247 >> 172.21.15.16:0/2407587247 conn(0x7f236006b620 msgr2=0x7f236006ba30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:04:44.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:44.296+0000 7f2366a00640 1 -- 172.21.15.16:0/2407587247 shutdown_connections 2024-03-31T15:04:44.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:44.296+0000 7f2366a00640 1 -- 172.21.15.16:0/2407587247 wait complete. 2024-03-31T15:04:44.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:44.300+0000 7f2366a00640 1 Processor -- start 2024-03-31T15:04:44.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:44.300+0000 7f2366a00640 1 -- start start 2024-03-31T15:04:44.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:44.300+0000 7f2366a00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f23600728b0 0x7f236015a6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:44.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:44.300+0000 7f2366a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f23600b23d0 0x7f2360144490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:44.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:44.300+0000 7f2366a00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f2360145d10 0x7f23601449d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:44.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:44.300+0000 7f2366a00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f23600cf4d0 con 0x7f23600b23d0 2024-03-31T15:04:44.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:44.300+0000 7f2366a00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f23600cf350 con 0x7f23600728b0 2024-03-31T15:04:44.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:44.300+0000 7f2366a00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f2360072d10 con 0x7f2360145d10 2024-03-31T15:04:44.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:44.300+0000 7f235f400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f23600b23d0 0x7f2360144490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:44.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:44.300+0000 7f235f400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f23600b23d0 0x7f2360144490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54250/0 (socket says 172.21.15.16:54250) 2024-03-31T15:04:44.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:44.300+0000 7f235f400640 1 -- 172.21.15.16:0/414143925 learned_addr learned my addr 172.21.15.16:0/414143925 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:04:44.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:44.300+0000 7f235fe00640 1 --2- 172.21.15.16:0/414143925 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f23600728b0 0x7f236015a6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:44.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:44.300+0000 7f2364a00640 1 --2- 172.21.15.16:0/414143925 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f2360145d10 0x7f23601449d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:44.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:44.300+0000 7f235f400640 1 -- 172.21.15.16:0/414143925 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f2360145d10 msgr2=0x7f23601449d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:44.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:44.300+0000 7f235f400640 1 --2- 172.21.15.16:0/414143925 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f2360145d10 0x7f23601449d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:44.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:44.300+0000 7f235f400640 1 -- 172.21.15.16:0/414143925 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f23600728b0 msgr2=0x7f236015a6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:44.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:44.300+0000 7f235f400640 1 --2- 172.21.15.16:0/414143925 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f23600728b0 0x7f236015a6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:44.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:44.300+0000 7f235f400640 1 -- 172.21.15.16:0/414143925 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2360140e80 con 0x7f23600b23d0 2024-03-31T15:04:44.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:44.300+0000 7f235f400640 1 --2- 172.21.15.16:0/414143925 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f23600b23d0 0x7f2360144490 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f234c00c750 tx=0x7f234c00cc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:44.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:44.300+0000 7f2357e00640 1 -- 172.21.15.16:0/414143925 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f234c005ae0 con 0x7f23600b23d0 2024-03-31T15:04:44.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:44.300+0000 7f2357e00640 1 -- 172.21.15.16:0/414143925 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f234c005ca0 con 0x7f23600b23d0 2024-03-31T15:04:44.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:44.300+0000 7f2357e00640 1 -- 172.21.15.16:0/414143925 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f234c0072b0 con 0x7f23600b23d0 2024-03-31T15:04:44.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:44.300+0000 7f2366a00640 1 -- 172.21.15.16:0/414143925 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2360141160 con 0x7f23600b23d0 2024-03-31T15:04:44.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:44.300+0000 7f2366a00640 1 -- 172.21.15.16:0/414143925 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2360141720 con 0x7f23600b23d0 2024-03-31T15:04:44.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:44.304+0000 7f2366a00640 1 -- 172.21.15.16:0/414143925 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2360069950 con 0x7f23600b23d0 2024-03-31T15:04:44.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:44.304+0000 7f2357e00640 1 -- 172.21.15.16:0/414143925 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f234c007450 con 0x7f23600b23d0 2024-03-31T15:04:44.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:44.304+0000 7f2357e00640 1 --2- 172.21.15.16:0/414143925 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f234006c790 0x7f234006ec50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:44.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:44.304+0000 7f2357e00640 1 -- 172.21.15.16:0/414143925 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5709+0+0 (secure 0 0 0) 0x7f234c086890 con 0x7f23600b23d0 2024-03-31T15:04:44.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:44.304+0000 7f235fe00640 1 --2- 172.21.15.16:0/414143925 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f234006c790 0x7f234006ec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:44.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:44.304+0000 7f235fe00640 1 --2- 172.21.15.16:0/414143925 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f234006c790 0x7f234006ec50 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f236015c050 tx=0x7f2350009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:44.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:44.308+0000 7f2357e00640 1 -- 172.21.15.16:0/414143925 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f234c056760 con 0x7f23600b23d0 2024-03-31T15:04:44.343 INFO:tasks.ceph.mds.c.smithi016.stderr:2024-03-31T15:04:44.348+0000 7fe1e7c00640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-31T15:04:44.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:44.472+0000 7f2366a00640 1 -- 172.21.15.16:0/414143925 --> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f2360073ab0 con 0x7f234006c790 2024-03-31T15:04:44.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:44.476+0000 7f2357e00640 1 -- 172.21.15.16:0/414143925 <== mgr.4100 v2:172.21.15.89:6832/1549407273 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+238278 (secure 0 0 0) 0x7f2350011a00 con 0x7f234006c790 2024-03-31T15:04:44.475 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:04:44.476 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-03-31T15:04:44.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:44.480+0000 7f2366a00640 1 -- 172.21.15.16:0/414143925 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f234006c790 msgr2=0x7f234006ec50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:44.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:44.480+0000 7f2366a00640 1 --2- 172.21.15.16:0/414143925 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f234006c790 0x7f234006ec50 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f236015c050 tx=0x7f2350009190 comp rx=0 tx=0).stop 2024-03-31T15:04:44.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:44.480+0000 7f2366a00640 1 -- 172.21.15.16:0/414143925 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f23600b23d0 msgr2=0x7f2360144490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:44.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:44.480+0000 7f2366a00640 1 --2- 172.21.15.16:0/414143925 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f23600b23d0 0x7f2360144490 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f234c00c750 tx=0x7f234c00cc10 comp rx=0 tx=0).stop 2024-03-31T15:04:44.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:44.480+0000 7f2366a00640 1 -- 172.21.15.16:0/414143925 shutdown_connections 2024-03-31T15:04:44.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:44.480+0000 7f2366a00640 1 --2- 172.21.15.16:0/414143925 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f234006c790 0x7f234006ec50 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:44.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:44.480+0000 7f2366a00640 1 --2- 172.21.15.16:0/414143925 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f2360145d10 0x7f23601449d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:44.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:44.480+0000 7f2366a00640 1 --2- 172.21.15.16:0/414143925 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f23600b23d0 0x7f2360144490 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:44.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:44.480+0000 7f2366a00640 1 --2- 172.21.15.16:0/414143925 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f23600728b0 0x7f236015a6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:44.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:44.480+0000 7f2366a00640 1 -- 172.21.15.16:0/414143925 >> 172.21.15.16:0/414143925 conn(0x7f236006b620 msgr2=0x7f23600b1bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:04:44.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:44.480+0000 7f2366a00640 1 -- 172.21.15.16:0/414143925 shutdown_connections 2024-03-31T15:04:44.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:44.480+0000 7f2366a00640 1 -- 172.21.15.16:0/414143925 wait complete. 2024-03-31T15:04:44.499 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":67,"stamp":"2024-03-31T15:04:42.542547+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":5,"kb":754974720,"kb_used":236580,"kb_used_data":3732,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754738140,"statfs":{"total":773094113280,"available":772851855360,"internally_reserved":0,"allocated":3821568,"data_stored":2173833,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":238276144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.174068"},"pg_stats":[{"pgid":"5.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:04:39.365755+0000","last_change":"2024-03-31T15:04:39.365755+0000","last_active":"2024-03-31T15:04:39.365755+0000","last_peered":"2024-03-31T15:04:39.365755+0000","last_clean":"2024-03-31T15:04:39.365755+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:04:39.365755+0000","last_undegraded":"2024-03-31T15:04:39.365755+0000","last_fullsized":"2024-03-31T15:04:39.365755+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_clean_scrub_stamp":"2024-03-31T15:04:39.365755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:04:39.365755+0000","last_change":"2024-03-31T15:04:39.365755+0000","last_active":"2024-03-31T15:04:39.365755+0000","last_peered":"2024-03-31T15:04:39.365755+0000","last_clean":"2024-03-31T15:04:39.365755+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:04:39.365755+0000","last_undegraded":"2024-03-31T15:04:39.365755+0000","last_fullsized":"2024-03-31T15:04:39.365755+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_clean_scrub_stamp":"2024-03-31T15:04:39.365755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:04:39.365755+0000","last_change":"2024-03-31T15:04:39.365755+0000","last_active":"2024-03-31T15:04:39.365755+0000","last_peered":"2024-03-31T15:04:39.365755+0000","last_clean":"2024-03-31T15:04:39.365755+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:04:39.365755+0000","last_undegraded":"2024-03-31T15:04:39.365755+0000","last_fullsized":"2024-03-31T15:04:39.365755+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_clean_scrub_stamp":"2024-03-31T15:04:39.365755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:04:39.365755+0000","last_change":"2024-03-31T15:04:39.365755+0000","last_active":"2024-03-31T15:04:39.365755+0000","last_peered":"2024-03-31T15:04:39.365755+0000","last_clean":"2024-03-31T15:04:39.365755+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:04:39.365755+0000","last_undegraded":"2024-03-31T15:04:39.365755+0000","last_fullsized":"2024-03-31T15:04:39.365755+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_clean_scrub_stamp":"2024-03-31T15:04:39.365755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:04:39.365755+0000","last_change":"2024-03-31T15:04:39.365755+0000","last_active":"2024-03-31T15:04:39.365755+0000","last_peered":"2024-03-31T15:04:39.365755+0000","last_clean":"2024-03-31T15:04:39.365755+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:04:39.365755+0000","last_undegraded":"2024-03-31T15:04:39.365755+0000","last_fullsized":"2024-03-31T15:04:39.365755+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_clean_scrub_stamp":"2024-03-31T15:04:39.365755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:04:39.365755+0000","last_change":"2024-03-31T15:04:39.365755+0000","last_active":"2024-03-31T15:04:39.365755+0000","last_peered":"2024-03-31T15:04:39.365755+0000","last_clean":"2024-03-31T15:04:39.365755+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:04:39.365755+0000","last_undegraded":"2024-03-31T15:04:39.365755+0000","last_fullsized":"2024-03-31T15:04:39.365755+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_clean_scrub_stamp":"2024-03-31T15:04:39.365755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:04:39.365755+0000","last_change":"2024-03-31T15:04:39.365755+0000","last_active":"2024-03-31T15:04:39.365755+0000","last_peered":"2024-03-31T15:04:39.365755+0000","last_clean":"2024-03-31T15:04:39.365755+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:04:39.365755+0000","last_undegraded":"2024-03-31T15:04:39.365755+0000","last_fullsized":"2024-03-31T15:04:39.365755+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_clean_scrub_stamp":"2024-03-31T15:04:39.365755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:04:39.365755+0000","last_change":"2024-03-31T15:04:39.365755+0000","last_active":"2024-03-31T15:04:39.365755+0000","last_peered":"2024-03-31T15:04:39.365755+0000","last_clean":"2024-03-31T15:04:39.365755+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:04:39.365755+0000","last_undegraded":"2024-03-31T15:04:39.365755+0000","last_fullsized":"2024-03-31T15:04:39.365755+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_clean_scrub_stamp":"2024-03-31T15:04:39.365755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:04:39.365755+0000","last_change":"2024-03-31T15:04:39.365755+0000","last_active":"2024-03-31T15:04:39.365755+0000","last_peered":"2024-03-31T15:04:39.365755+0000","last_clean":"2024-03-31T15:04:39.365755+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:04:39.365755+0000","last_undegraded":"2024-03-31T15:04:39.365755+0000","last_fullsized":"2024-03-31T15:04:39.365755+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_clean_scrub_stamp":"2024-03-31T15:04:39.365755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:04:39.365755+0000","last_change":"2024-03-31T15:04:39.365755+0000","last_active":"2024-03-31T15:04:39.365755+0000","last_peered":"2024-03-31T15:04:39.365755+0000","last_clean":"2024-03-31T15:04:39.365755+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:04:39.365755+0000","last_undegraded":"2024-03-31T15:04:39.365755+0000","last_fullsized":"2024-03-31T15:04:39.365755+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_clean_scrub_stamp":"2024-03-31T15:04:39.365755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:04:39.365755+0000","last_change":"2024-03-31T15:04:39.365755+0000","last_active":"2024-03-31T15:04:39.365755+0000","last_peered":"2024-03-31T15:04:39.365755+0000","last_clean":"2024-03-31T15:04:39.365755+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:04:39.365755+0000","last_undegraded":"2024-03-31T15:04:39.365755+0000","last_fullsized":"2024-03-31T15:04:39.365755+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_clean_scrub_stamp":"2024-03-31T15:04:39.365755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:04:39.365755+0000","last_change":"2024-03-31T15:04:39.365755+0000","last_active":"2024-03-31T15:04:39.365755+0000","last_peered":"2024-03-31T15:04:39.365755+0000","last_clean":"2024-03-31T15:04:39.365755+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:04:39.365755+0000","last_undegraded":"2024-03-31T15:04:39.365755+0000","last_fullsized":"2024-03-31T15:04:39.365755+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_clean_scrub_stamp":"2024-03-31T15:04:39.365755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:04:39.365755+0000","last_change":"2024-03-31T15:04:39.365755+0000","last_active":"2024-03-31T15:04:39.365755+0000","last_peered":"2024-03-31T15:04:39.365755+0000","last_clean":"2024-03-31T15:04:39.365755+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:04:39.365755+0000","last_undegraded":"2024-03-31T15:04:39.365755+0000","last_fullsized":"2024-03-31T15:04:39.365755+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_clean_scrub_stamp":"2024-03-31T15:04:39.365755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:04:39.365755+0000","last_change":"2024-03-31T15:04:39.365755+0000","last_active":"2024-03-31T15:04:39.365755+0000","last_peered":"2024-03-31T15:04:39.365755+0000","last_clean":"2024-03-31T15:04:39.365755+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:04:39.365755+0000","last_undegraded":"2024-03-31T15:04:39.365755+0000","last_fullsized":"2024-03-31T15:04:39.365755+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_clean_scrub_stamp":"2024-03-31T15:04:39.365755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:04:39.365755+0000","last_change":"2024-03-31T15:04:39.365755+0000","last_active":"2024-03-31T15:04:39.365755+0000","last_peered":"2024-03-31T15:04:39.365755+0000","last_clean":"2024-03-31T15:04:39.365755+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:04:39.365755+0000","last_undegraded":"2024-03-31T15:04:39.365755+0000","last_fullsized":"2024-03-31T15:04:39.365755+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_clean_scrub_stamp":"2024-03-31T15:04:39.365755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:04:39.365755+0000","last_change":"2024-03-31T15:04:39.365755+0000","last_active":"2024-03-31T15:04:39.365755+0000","last_peered":"2024-03-31T15:04:39.365755+0000","last_clean":"2024-03-31T15:04:39.365755+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:04:39.365755+0000","last_undegraded":"2024-03-31T15:04:39.365755+0000","last_fullsized":"2024-03-31T15:04:39.365755+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_clean_scrub_stamp":"2024-03-31T15:04:39.365755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:04:39.365755+0000","last_change":"2024-03-31T15:04:39.365755+0000","last_active":"2024-03-31T15:04:39.365755+0000","last_peered":"2024-03-31T15:04:39.365755+0000","last_clean":"2024-03-31T15:04:39.365755+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:04:39.365755+0000","last_undegraded":"2024-03-31T15:04:39.365755+0000","last_fullsized":"2024-03-31T15:04:39.365755+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_clean_scrub_stamp":"2024-03-31T15:04:39.365755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:04:39.365755+0000","last_change":"2024-03-31T15:04:39.365755+0000","last_active":"2024-03-31T15:04:39.365755+0000","last_peered":"2024-03-31T15:04:39.365755+0000","last_clean":"2024-03-31T15:04:39.365755+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:04:39.365755+0000","last_undegraded":"2024-03-31T15:04:39.365755+0000","last_fullsized":"2024-03-31T15:04:39.365755+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_clean_scrub_stamp":"2024-03-31T15:04:39.365755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:04:39.365755+0000","last_change":"2024-03-31T15:04:39.365755+0000","last_active":"2024-03-31T15:04:39.365755+0000","last_peered":"2024-03-31T15:04:39.365755+0000","last_clean":"2024-03-31T15:04:39.365755+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:04:39.365755+0000","last_undegraded":"2024-03-31T15:04:39.365755+0000","last_fullsized":"2024-03-31T15:04:39.365755+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_clean_scrub_stamp":"2024-03-31T15:04:39.365755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:04:39.365755+0000","last_change":"2024-03-31T15:04:39.365755+0000","last_active":"2024-03-31T15:04:39.365755+0000","last_peered":"2024-03-31T15:04:39.365755+0000","last_clean":"2024-03-31T15:04:39.365755+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:04:39.365755+0000","last_undegraded":"2024-03-31T15:04:39.365755+0000","last_fullsized":"2024-03-31T15:04:39.365755+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_clean_scrub_stamp":"2024-03-31T15:04:39.365755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:04:39.365755+0000","last_change":"2024-03-31T15:04:39.365755+0000","last_active":"2024-03-31T15:04:39.365755+0000","last_peered":"2024-03-31T15:04:39.365755+0000","last_clean":"2024-03-31T15:04:39.365755+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:04:39.365755+0000","last_undegraded":"2024-03-31T15:04:39.365755+0000","last_fullsized":"2024-03-31T15:04:39.365755+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_clean_scrub_stamp":"2024-03-31T15:04:39.365755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:04:39.365755+0000","last_change":"2024-03-31T15:04:39.365755+0000","last_active":"2024-03-31T15:04:39.365755+0000","last_peered":"2024-03-31T15:04:39.365755+0000","last_clean":"2024-03-31T15:04:39.365755+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:04:39.365755+0000","last_undegraded":"2024-03-31T15:04:39.365755+0000","last_fullsized":"2024-03-31T15:04:39.365755+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_clean_scrub_stamp":"2024-03-31T15:04:39.365755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:04:39.365755+0000","last_change":"2024-03-31T15:04:39.365755+0000","last_active":"2024-03-31T15:04:39.365755+0000","last_peered":"2024-03-31T15:04:39.365755+0000","last_clean":"2024-03-31T15:04:39.365755+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:04:39.365755+0000","last_undegraded":"2024-03-31T15:04:39.365755+0000","last_fullsized":"2024-03-31T15:04:39.365755+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_clean_scrub_stamp":"2024-03-31T15:04:39.365755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:04:39.365755+0000","last_change":"2024-03-31T15:04:39.365755+0000","last_active":"2024-03-31T15:04:39.365755+0000","last_peered":"2024-03-31T15:04:39.365755+0000","last_clean":"2024-03-31T15:04:39.365755+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:04:39.365755+0000","last_undegraded":"2024-03-31T15:04:39.365755+0000","last_fullsized":"2024-03-31T15:04:39.365755+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_clean_scrub_stamp":"2024-03-31T15:04:39.365755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:04:39.365755+0000","last_change":"2024-03-31T15:04:39.365755+0000","last_active":"2024-03-31T15:04:39.365755+0000","last_peered":"2024-03-31T15:04:39.365755+0000","last_clean":"2024-03-31T15:04:39.365755+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:04:39.365755+0000","last_undegraded":"2024-03-31T15:04:39.365755+0000","last_fullsized":"2024-03-31T15:04:39.365755+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_clean_scrub_stamp":"2024-03-31T15:04:39.365755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:04:39.365755+0000","last_change":"2024-03-31T15:04:39.365755+0000","last_active":"2024-03-31T15:04:39.365755+0000","last_peered":"2024-03-31T15:04:39.365755+0000","last_clean":"2024-03-31T15:04:39.365755+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:04:39.365755+0000","last_undegraded":"2024-03-31T15:04:39.365755+0000","last_fullsized":"2024-03-31T15:04:39.365755+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_clean_scrub_stamp":"2024-03-31T15:04:39.365755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:04:39.365755+0000","last_change":"2024-03-31T15:04:39.365755+0000","last_active":"2024-03-31T15:04:39.365755+0000","last_peered":"2024-03-31T15:04:39.365755+0000","last_clean":"2024-03-31T15:04:39.365755+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:04:39.365755+0000","last_undegraded":"2024-03-31T15:04:39.365755+0000","last_fullsized":"2024-03-31T15:04:39.365755+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_clean_scrub_stamp":"2024-03-31T15:04:39.365755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:04:39.365755+0000","last_change":"2024-03-31T15:04:39.365755+0000","last_active":"2024-03-31T15:04:39.365755+0000","last_peered":"2024-03-31T15:04:39.365755+0000","last_clean":"2024-03-31T15:04:39.365755+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:04:39.365755+0000","last_undegraded":"2024-03-31T15:04:39.365755+0000","last_fullsized":"2024-03-31T15:04:39.365755+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_clean_scrub_stamp":"2024-03-31T15:04:39.365755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:04:39.365755+0000","last_change":"2024-03-31T15:04:39.365755+0000","last_active":"2024-03-31T15:04:39.365755+0000","last_peered":"2024-03-31T15:04:39.365755+0000","last_clean":"2024-03-31T15:04:39.365755+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:04:39.365755+0000","last_undegraded":"2024-03-31T15:04:39.365755+0000","last_fullsized":"2024-03-31T15:04:39.365755+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_clean_scrub_stamp":"2024-03-31T15:04:39.365755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:04:39.365755+0000","last_change":"2024-03-31T15:04:39.365755+0000","last_active":"2024-03-31T15:04:39.365755+0000","last_peered":"2024-03-31T15:04:39.365755+0000","last_clean":"2024-03-31T15:04:39.365755+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:04:39.365755+0000","last_undegraded":"2024-03-31T15:04:39.365755+0000","last_fullsized":"2024-03-31T15:04:39.365755+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_clean_scrub_stamp":"2024-03-31T15:04:39.365755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:04:39.365755+0000","last_change":"2024-03-31T15:04:39.365755+0000","last_active":"2024-03-31T15:04:39.365755+0000","last_peered":"2024-03-31T15:04:39.365755+0000","last_clean":"2024-03-31T15:04:39.365755+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:04:39.365755+0000","last_undegraded":"2024-03-31T15:04:39.365755+0000","last_fullsized":"2024-03-31T15:04:39.365755+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_clean_scrub_stamp":"2024-03-31T15:04:39.365755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:04:39.365755+0000","last_change":"2024-03-31T15:04:39.365755+0000","last_active":"2024-03-31T15:04:39.365755+0000","last_peered":"2024-03-31T15:04:39.365755+0000","last_clean":"2024-03-31T15:04:39.365755+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:04:39.365755+0000","last_undegraded":"2024-03-31T15:04:39.365755+0000","last_fullsized":"2024-03-31T15:04:39.365755+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_clean_scrub_stamp":"2024-03-31T15:04:39.365755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:04:39.365755+0000","last_change":"2024-03-31T15:04:39.365755+0000","last_active":"2024-03-31T15:04:39.365755+0000","last_peered":"2024-03-31T15:04:39.365755+0000","last_clean":"2024-03-31T15:04:39.365755+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:04:39.365755+0000","last_undegraded":"2024-03-31T15:04:39.365755+0000","last_fullsized":"2024-03-31T15:04:39.365755+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_clean_scrub_stamp":"2024-03-31T15:04:39.365755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:04:38.358174+0000","last_change":"2024-03-31T15:04:38.358174+0000","last_active":"2024-03-31T15:04:38.358174+0000","last_peered":"2024-03-31T15:04:38.358174+0000","last_clean":"2024-03-31T15:04:38.358174+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:04:38.358174+0000","last_undegraded":"2024-03-31T15:04:38.358174+0000","last_fullsized":"2024-03-31T15:04:38.358174+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:04:38.358174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:04:38.358174+0000","last_clean_scrub_stamp":"2024-03-31T15:04:38.358174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:04:39.365755+0000","last_change":"2024-03-31T15:04:39.365755+0000","last_active":"2024-03-31T15:04:39.365755+0000","last_peered":"2024-03-31T15:04:39.365755+0000","last_clean":"2024-03-31T15:04:39.365755+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:04:39.365755+0000","last_undegraded":"2024-03-31T15:04:39.365755+0000","last_fullsized":"2024-03-31T15:04:39.365755+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_clean_scrub_stamp":"2024-03-31T15:04:39.365755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:04:38.358174+0000","last_change":"2024-03-31T15:04:38.358174+0000","last_active":"2024-03-31T15:04:38.358174+0000","last_peered":"2024-03-31T15:04:38.358174+0000","last_clean":"2024-03-31T15:04:38.358174+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:04:38.358174+0000","last_undegraded":"2024-03-31T15:04:38.358174+0000","last_fullsized":"2024-03-31T15:04:38.358174+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:04:38.358174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:04:38.358174+0000","last_clean_scrub_stamp":"2024-03-31T15:04:38.358174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:04:39.365755+0000","last_change":"2024-03-31T15:04:39.365755+0000","last_active":"2024-03-31T15:04:39.365755+0000","last_peered":"2024-03-31T15:04:39.365755+0000","last_clean":"2024-03-31T15:04:39.365755+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:04:39.365755+0000","last_undegraded":"2024-03-31T15:04:39.365755+0000","last_fullsized":"2024-03-31T15:04:39.365755+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_clean_scrub_stamp":"2024-03-31T15:04:39.365755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:04:38.358174+0000","last_change":"2024-03-31T15:04:38.358174+0000","last_active":"2024-03-31T15:04:38.358174+0000","last_peered":"2024-03-31T15:04:38.358174+0000","last_clean":"2024-03-31T15:04:38.358174+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:04:38.358174+0000","last_undegraded":"2024-03-31T15:04:38.358174+0000","last_fullsized":"2024-03-31T15:04:38.358174+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:04:38.358174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:04:38.358174+0000","last_clean_scrub_stamp":"2024-03-31T15:04:38.358174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:04:39.365755+0000","last_change":"2024-03-31T15:04:39.365755+0000","last_active":"2024-03-31T15:04:39.365755+0000","last_peered":"2024-03-31T15:04:39.365755+0000","last_clean":"2024-03-31T15:04:39.365755+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:04:39.365755+0000","last_undegraded":"2024-03-31T15:04:39.365755+0000","last_fullsized":"2024-03-31T15:04:39.365755+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_clean_scrub_stamp":"2024-03-31T15:04:39.365755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-31T15:04:38.372814+0000","last_change":"2024-03-31T15:04:38.372814+0000","last_active":"2024-03-31T15:04:38.358174+0000","last_peered":"2024-03-31T15:04:38.358174+0000","last_clean":"2024-03-31T15:04:38.358174+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:04:38.372814+0000","last_undegraded":"2024-03-31T15:04:38.372814+0000","last_fullsized":"2024-03-31T15:04:38.372814+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:04:38.358174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:04:38.358174+0000","last_clean_scrub_stamp":"2024-03-31T15:04:38.358174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:04:39.365755+0000","last_change":"2024-03-31T15:04:39.365755+0000","last_active":"2024-03-31T15:04:39.365755+0000","last_peered":"2024-03-31T15:04:39.365755+0000","last_clean":"2024-03-31T15:04:39.365755+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:04:39.365755+0000","last_undegraded":"2024-03-31T15:04:39.365755+0000","last_fullsized":"2024-03-31T15:04:39.365755+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_clean_scrub_stamp":"2024-03-31T15:04:39.365755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-31T15:04:38.372099+0000","last_change":"2024-03-31T15:04:38.372099+0000","last_active":"2024-03-31T15:04:38.358174+0000","last_peered":"2024-03-31T15:04:38.358174+0000","last_clean":"2024-03-31T15:04:38.358174+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:04:38.372099+0000","last_undegraded":"2024-03-31T15:04:38.372099+0000","last_fullsized":"2024-03-31T15:04:38.372099+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:04:38.358174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:04:38.358174+0000","last_clean_scrub_stamp":"2024-03-31T15:04:38.358174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:04:39.365755+0000","last_change":"2024-03-31T15:04:39.365755+0000","last_active":"2024-03-31T15:04:39.365755+0000","last_peered":"2024-03-31T15:04:39.365755+0000","last_clean":"2024-03-31T15:04:39.365755+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:04:39.365755+0000","last_undegraded":"2024-03-31T15:04:39.365755+0000","last_fullsized":"2024-03-31T15:04:39.365755+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_clean_scrub_stamp":"2024-03-31T15:04:39.365755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-31T15:04:38.371341+0000","last_change":"2024-03-31T15:04:38.371341+0000","last_active":"2024-03-31T15:04:38.358174+0000","last_peered":"2024-03-31T15:04:38.358174+0000","last_clean":"2024-03-31T15:04:38.358174+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:04:38.371341+0000","last_undegraded":"2024-03-31T15:04:38.371341+0000","last_fullsized":"2024-03-31T15:04:38.371341+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:04:38.358174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:04:38.358174+0000","last_clean_scrub_stamp":"2024-03-31T15:04:38.358174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:04:39.365755+0000","last_change":"2024-03-31T15:04:39.365755+0000","last_active":"2024-03-31T15:04:39.365755+0000","last_peered":"2024-03-31T15:04:39.365755+0000","last_clean":"2024-03-31T15:04:39.365755+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:04:39.365755+0000","last_undegraded":"2024-03-31T15:04:39.365755+0000","last_fullsized":"2024-03-31T15:04:39.365755+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_clean_scrub_stamp":"2024-03-31T15:04:39.365755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:04:38.358174+0000","last_change":"2024-03-31T15:04:38.358174+0000","last_active":"2024-03-31T15:04:38.358174+0000","last_peered":"2024-03-31T15:04:38.358174+0000","last_clean":"2024-03-31T15:04:38.358174+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:04:38.358174+0000","last_undegraded":"2024-03-31T15:04:38.358174+0000","last_fullsized":"2024-03-31T15:04:38.358174+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:04:38.358174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:04:38.358174+0000","last_clean_scrub_stamp":"2024-03-31T15:04:38.358174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:04:39.365755+0000","last_change":"2024-03-31T15:04:39.365755+0000","last_active":"2024-03-31T15:04:39.365755+0000","last_peered":"2024-03-31T15:04:39.365755+0000","last_clean":"2024-03-31T15:04:39.365755+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:04:39.365755+0000","last_undegraded":"2024-03-31T15:04:39.365755+0000","last_fullsized":"2024-03-31T15:04:39.365755+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_clean_scrub_stamp":"2024-03-31T15:04:39.365755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-31T15:04:38.371736+0000","last_change":"2024-03-31T15:04:38.371736+0000","last_active":"2024-03-31T15:04:38.358174+0000","last_peered":"2024-03-31T15:04:38.358174+0000","last_clean":"2024-03-31T15:04:38.358174+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:04:38.371736+0000","last_undegraded":"2024-03-31T15:04:38.371736+0000","last_fullsized":"2024-03-31T15:04:38.371736+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:04:38.358174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:04:38.358174+0000","last_clean_scrub_stamp":"2024-03-31T15:04:38.358174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:04:39.365755+0000","last_change":"2024-03-31T15:04:39.365755+0000","last_active":"2024-03-31T15:04:39.365755+0000","last_peered":"2024-03-31T15:04:39.365755+0000","last_clean":"2024-03-31T15:04:39.365755+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:04:39.365755+0000","last_undegraded":"2024-03-31T15:04:39.365755+0000","last_fullsized":"2024-03-31T15:04:39.365755+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_clean_scrub_stamp":"2024-03-31T15:04:39.365755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:04:38.358174+0000","last_change":"2024-03-31T15:04:38.358174+0000","last_active":"2024-03-31T15:04:38.358174+0000","last_peered":"2024-03-31T15:04:38.358174+0000","last_clean":"2024-03-31T15:04:38.358174+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:04:38.358174+0000","last_undegraded":"2024-03-31T15:04:38.358174+0000","last_fullsized":"2024-03-31T15:04:38.358174+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:04:38.358174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:04:38.358174+0000","last_clean_scrub_stamp":"2024-03-31T15:04:38.358174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:04:39.365755+0000","last_change":"2024-03-31T15:04:39.365755+0000","last_active":"2024-03-31T15:04:39.365755+0000","last_peered":"2024-03-31T15:04:39.365755+0000","last_clean":"2024-03-31T15:04:39.365755+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:04:39.365755+0000","last_undegraded":"2024-03-31T15:04:39.365755+0000","last_fullsized":"2024-03-31T15:04:39.365755+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_clean_scrub_stamp":"2024-03-31T15:04:39.365755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:04:38.358174+0000","last_change":"2024-03-31T15:04:38.358174+0000","last_active":"2024-03-31T15:04:38.358174+0000","last_peered":"2024-03-31T15:04:38.358174+0000","last_clean":"2024-03-31T15:04:38.358174+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:04:38.358174+0000","last_undegraded":"2024-03-31T15:04:38.358174+0000","last_fullsized":"2024-03-31T15:04:38.358174+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:04:38.358174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:04:38.358174+0000","last_clean_scrub_stamp":"2024-03-31T15:04:38.358174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:04:39.365755+0000","last_change":"2024-03-31T15:04:39.365755+0000","last_active":"2024-03-31T15:04:39.365755+0000","last_peered":"2024-03-31T15:04:39.365755+0000","last_clean":"2024-03-31T15:04:39.365755+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:04:39.365755+0000","last_undegraded":"2024-03-31T15:04:39.365755+0000","last_fullsized":"2024-03-31T15:04:39.365755+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_clean_scrub_stamp":"2024-03-31T15:04:39.365755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-31T15:04:38.369953+0000","last_change":"2024-03-31T15:04:38.369953+0000","last_active":"2024-03-31T15:04:38.358174+0000","last_peered":"2024-03-31T15:04:38.358174+0000","last_clean":"2024-03-31T15:04:38.358174+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:04:38.369953+0000","last_undegraded":"2024-03-31T15:04:38.369953+0000","last_fullsized":"2024-03-31T15:04:38.369953+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:04:38.358174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:04:38.358174+0000","last_clean_scrub_stamp":"2024-03-31T15:04:38.358174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:04:39.365755+0000","last_change":"2024-03-31T15:04:39.365755+0000","last_active":"2024-03-31T15:04:39.365755+0000","last_peered":"2024-03-31T15:04:39.365755+0000","last_clean":"2024-03-31T15:04:39.365755+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:04:39.365755+0000","last_undegraded":"2024-03-31T15:04:39.365755+0000","last_fullsized":"2024-03-31T15:04:39.365755+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_clean_scrub_stamp":"2024-03-31T15:04:39.365755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:04:38.358174+0000","last_change":"2024-03-31T15:04:38.358174+0000","last_active":"2024-03-31T15:04:38.358174+0000","last_peered":"2024-03-31T15:04:38.358174+0000","last_clean":"2024-03-31T15:04:38.358174+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:04:38.358174+0000","last_undegraded":"2024-03-31T15:04:38.358174+0000","last_fullsized":"2024-03-31T15:04:38.358174+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:04:38.358174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:04:38.358174+0000","last_clean_scrub_stamp":"2024-03-31T15:04:38.358174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:04:39.365755+0000","last_change":"2024-03-31T15:04:39.365755+0000","last_active":"2024-03-31T15:04:39.365755+0000","last_peered":"2024-03-31T15:04:39.365755+0000","last_clean":"2024-03-31T15:04:39.365755+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:04:39.365755+0000","last_undegraded":"2024-03-31T15:04:39.365755+0000","last_fullsized":"2024-03-31T15:04:39.365755+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_clean_scrub_stamp":"2024-03-31T15:04:39.365755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-31T15:04:38.371677+0000","last_change":"2024-03-31T15:04:38.371677+0000","last_active":"2024-03-31T15:04:38.358174+0000","last_peered":"2024-03-31T15:04:38.358174+0000","last_clean":"2024-03-31T15:04:38.358174+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:04:38.371677+0000","last_undegraded":"2024-03-31T15:04:38.371677+0000","last_fullsized":"2024-03-31T15:04:38.371677+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:04:38.358174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:04:38.358174+0000","last_clean_scrub_stamp":"2024-03-31T15:04:38.358174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:04:39.365755+0000","last_change":"2024-03-31T15:04:39.365755+0000","last_active":"2024-03-31T15:04:39.365755+0000","last_peered":"2024-03-31T15:04:39.365755+0000","last_clean":"2024-03-31T15:04:39.365755+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:04:39.365755+0000","last_undegraded":"2024-03-31T15:04:39.365755+0000","last_fullsized":"2024-03-31T15:04:39.365755+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_clean_scrub_stamp":"2024-03-31T15:04:39.365755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:04:38.358174+0000","last_change":"2024-03-31T15:04:38.358174+0000","last_active":"2024-03-31T15:04:38.358174+0000","last_peered":"2024-03-31T15:04:38.358174+0000","last_clean":"2024-03-31T15:04:38.358174+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:04:38.358174+0000","last_undegraded":"2024-03-31T15:04:38.358174+0000","last_fullsized":"2024-03-31T15:04:38.358174+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:04:38.358174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:04:38.358174+0000","last_clean_scrub_stamp":"2024-03-31T15:04:38.358174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:04:39.365755+0000","last_change":"2024-03-31T15:04:39.365755+0000","last_active":"2024-03-31T15:04:39.365755+0000","last_peered":"2024-03-31T15:04:39.365755+0000","last_clean":"2024-03-31T15:04:39.365755+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:04:39.365755+0000","last_undegraded":"2024-03-31T15:04:39.365755+0000","last_fullsized":"2024-03-31T15:04:39.365755+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_clean_scrub_stamp":"2024-03-31T15:04:39.365755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:04:38.358174+0000","last_change":"2024-03-31T15:04:38.358174+0000","last_active":"2024-03-31T15:04:38.358174+0000","last_peered":"2024-03-31T15:04:38.358174+0000","last_clean":"2024-03-31T15:04:38.358174+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:04:38.358174+0000","last_undegraded":"2024-03-31T15:04:38.358174+0000","last_fullsized":"2024-03-31T15:04:38.358174+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:04:38.358174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:04:38.358174+0000","last_clean_scrub_stamp":"2024-03-31T15:04:38.358174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:04:39.365755+0000","last_change":"2024-03-31T15:04:39.365755+0000","last_active":"2024-03-31T15:04:39.365755+0000","last_peered":"2024-03-31T15:04:39.365755+0000","last_clean":"2024-03-31T15:04:39.365755+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:04:39.365755+0000","last_undegraded":"2024-03-31T15:04:39.365755+0000","last_fullsized":"2024-03-31T15:04:39.365755+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_clean_scrub_stamp":"2024-03-31T15:04:39.365755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:04:38.358174+0000","last_change":"2024-03-31T15:04:38.358174+0000","last_active":"2024-03-31T15:04:38.358174+0000","last_peered":"2024-03-31T15:04:38.358174+0000","last_clean":"2024-03-31T15:04:38.358174+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:04:38.358174+0000","last_undegraded":"2024-03-31T15:04:38.358174+0000","last_fullsized":"2024-03-31T15:04:38.358174+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:04:38.358174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:04:38.358174+0000","last_clean_scrub_stamp":"2024-03-31T15:04:38.358174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:04:39.365755+0000","last_change":"2024-03-31T15:04:39.365755+0000","last_active":"2024-03-31T15:04:39.365755+0000","last_peered":"2024-03-31T15:04:39.365755+0000","last_clean":"2024-03-31T15:04:39.365755+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:04:39.365755+0000","last_undegraded":"2024-03-31T15:04:39.365755+0000","last_fullsized":"2024-03-31T15:04:39.365755+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_clean_scrub_stamp":"2024-03-31T15:04:39.365755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:04:38.358174+0000","last_change":"2024-03-31T15:04:38.358174+0000","last_active":"2024-03-31T15:04:38.358174+0000","last_peered":"2024-03-31T15:04:38.358174+0000","last_clean":"2024-03-31T15:04:38.358174+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:04:38.358174+0000","last_undegraded":"2024-03-31T15:04:38.358174+0000","last_fullsized":"2024-03-31T15:04:38.358174+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:04:38.358174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:04:38.358174+0000","last_clean_scrub_stamp":"2024-03-31T15:04:38.358174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:04:39.365755+0000","last_change":"2024-03-31T15:04:39.365755+0000","last_active":"2024-03-31T15:04:39.365755+0000","last_peered":"2024-03-31T15:04:39.365755+0000","last_clean":"2024-03-31T15:04:39.365755+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:04:39.365755+0000","last_undegraded":"2024-03-31T15:04:39.365755+0000","last_fullsized":"2024-03-31T15:04:39.365755+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_clean_scrub_stamp":"2024-03-31T15:04:39.365755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-31T15:04:38.371085+0000","last_change":"2024-03-31T15:04:38.371085+0000","last_active":"2024-03-31T15:04:38.358174+0000","last_peered":"2024-03-31T15:04:38.358174+0000","last_clean":"2024-03-31T15:04:38.358174+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:04:38.371085+0000","last_undegraded":"2024-03-31T15:04:38.371085+0000","last_fullsized":"2024-03-31T15:04:38.371085+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:04:38.358174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:04:38.358174+0000","last_clean_scrub_stamp":"2024-03-31T15:04:38.358174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:04:39.365755+0000","last_change":"2024-03-31T15:04:39.365755+0000","last_active":"2024-03-31T15:04:39.365755+0000","last_peered":"2024-03-31T15:04:39.365755+0000","last_clean":"2024-03-31T15:04:39.365755+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:04:39.365755+0000","last_undegraded":"2024-03-31T15:04:39.365755+0000","last_fullsized":"2024-03-31T15:04:39.365755+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_clean_scrub_stamp":"2024-03-31T15:04:39.365755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-31T15:04:38.371092+0000","last_change":"2024-03-31T15:04:38.371092+0000","last_active":"2024-03-31T15:04:38.358174+0000","last_peered":"2024-03-31T15:04:38.358174+0000","last_clean":"2024-03-31T15:04:38.358174+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:04:38.371092+0000","last_undegraded":"2024-03-31T15:04:38.371092+0000","last_fullsized":"2024-03-31T15:04:38.371092+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:04:38.358174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:04:38.358174+0000","last_clean_scrub_stamp":"2024-03-31T15:04:38.358174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:04:39.365755+0000","last_change":"2024-03-31T15:04:39.365755+0000","last_active":"2024-03-31T15:04:39.365755+0000","last_peered":"2024-03-31T15:04:39.365755+0000","last_clean":"2024-03-31T15:04:39.365755+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:04:39.365755+0000","last_undegraded":"2024-03-31T15:04:39.365755+0000","last_fullsized":"2024-03-31T15:04:39.365755+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_clean_scrub_stamp":"2024-03-31T15:04:39.365755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-31T15:04:38.371801+0000","last_change":"2024-03-31T15:04:38.371801+0000","last_active":"2024-03-31T15:04:38.358174+0000","last_peered":"2024-03-31T15:04:38.358174+0000","last_clean":"2024-03-31T15:04:38.358174+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:04:38.371801+0000","last_undegraded":"2024-03-31T15:04:38.371801+0000","last_fullsized":"2024-03-31T15:04:38.371801+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:04:38.358174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:04:38.358174+0000","last_clean_scrub_stamp":"2024-03-31T15:04:38.358174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:04:39.365755+0000","last_change":"2024-03-31T15:04:39.365755+0000","last_active":"2024-03-31T15:04:39.365755+0000","last_peered":"2024-03-31T15:04:39.365755+0000","last_clean":"2024-03-31T15:04:39.365755+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:04:39.365755+0000","last_undegraded":"2024-03-31T15:04:39.365755+0000","last_fullsized":"2024-03-31T15:04:39.365755+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_clean_scrub_stamp":"2024-03-31T15:04:39.365755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:04:38.358174+0000","last_change":"2024-03-31T15:04:38.358174+0000","last_active":"2024-03-31T15:04:38.358174+0000","last_peered":"2024-03-31T15:04:38.358174+0000","last_clean":"2024-03-31T15:04:38.358174+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:04:38.358174+0000","last_undegraded":"2024-03-31T15:04:38.358174+0000","last_fullsized":"2024-03-31T15:04:38.358174+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:04:38.358174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:04:38.358174+0000","last_clean_scrub_stamp":"2024-03-31T15:04:38.358174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:04:39.365755+0000","last_change":"2024-03-31T15:04:39.365755+0000","last_active":"2024-03-31T15:04:39.365755+0000","last_peered":"2024-03-31T15:04:39.365755+0000","last_clean":"2024-03-31T15:04:39.365755+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:04:39.365755+0000","last_undegraded":"2024-03-31T15:04:39.365755+0000","last_fullsized":"2024-03-31T15:04:39.365755+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_clean_scrub_stamp":"2024-03-31T15:04:39.365755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:04:38.358174+0000","last_change":"2024-03-31T15:04:38.358174+0000","last_active":"2024-03-31T15:04:38.358174+0000","last_peered":"2024-03-31T15:04:38.358174+0000","last_clean":"2024-03-31T15:04:38.358174+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:04:38.358174+0000","last_undegraded":"2024-03-31T15:04:38.358174+0000","last_fullsized":"2024-03-31T15:04:38.358174+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:04:38.358174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:04:38.358174+0000","last_clean_scrub_stamp":"2024-03-31T15:04:38.358174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:04:39.365755+0000","last_change":"2024-03-31T15:04:39.365755+0000","last_active":"2024-03-31T15:04:39.365755+0000","last_peered":"2024-03-31T15:04:39.365755+0000","last_clean":"2024-03-31T15:04:39.365755+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:04:39.365755+0000","last_undegraded":"2024-03-31T15:04:39.365755+0000","last_fullsized":"2024-03-31T15:04:39.365755+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_clean_scrub_stamp":"2024-03-31T15:04:39.365755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-31T15:04:38.371926+0000","last_change":"2024-03-31T15:04:38.371926+0000","last_active":"2024-03-31T15:04:38.358174+0000","last_peered":"2024-03-31T15:04:38.358174+0000","last_clean":"2024-03-31T15:04:38.358174+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:04:38.371926+0000","last_undegraded":"2024-03-31T15:04:38.371926+0000","last_fullsized":"2024-03-31T15:04:38.371926+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:04:38.358174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:04:38.358174+0000","last_clean_scrub_stamp":"2024-03-31T15:04:38.358174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:04:39.365755+0000","last_change":"2024-03-31T15:04:39.365755+0000","last_active":"2024-03-31T15:04:39.365755+0000","last_peered":"2024-03-31T15:04:39.365755+0000","last_clean":"2024-03-31T15:04:39.365755+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:04:39.365755+0000","last_undegraded":"2024-03-31T15:04:39.365755+0000","last_fullsized":"2024-03-31T15:04:39.365755+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_clean_scrub_stamp":"2024-03-31T15:04:39.365755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-31T15:04:38.369972+0000","last_change":"2024-03-31T15:04:38.369972+0000","last_active":"2024-03-31T15:04:38.358174+0000","last_peered":"2024-03-31T15:04:38.358174+0000","last_clean":"2024-03-31T15:04:38.358174+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:04:38.369972+0000","last_undegraded":"2024-03-31T15:04:38.369972+0000","last_fullsized":"2024-03-31T15:04:38.369972+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:04:38.358174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:04:38.358174+0000","last_clean_scrub_stamp":"2024-03-31T15:04:38.358174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:04:39.365755+0000","last_change":"2024-03-31T15:04:39.365755+0000","last_active":"2024-03-31T15:04:39.365755+0000","last_peered":"2024-03-31T15:04:39.365755+0000","last_clean":"2024-03-31T15:04:39.365755+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:04:39.365755+0000","last_undegraded":"2024-03-31T15:04:39.365755+0000","last_fullsized":"2024-03-31T15:04:39.365755+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_clean_scrub_stamp":"2024-03-31T15:04:39.365755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:04:38.358174+0000","last_change":"2024-03-31T15:04:38.358174+0000","last_active":"2024-03-31T15:04:38.358174+0000","last_peered":"2024-03-31T15:04:38.358174+0000","last_clean":"2024-03-31T15:04:38.358174+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:04:38.358174+0000","last_undegraded":"2024-03-31T15:04:38.358174+0000","last_fullsized":"2024-03-31T15:04:38.358174+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:04:38.358174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:04:38.358174+0000","last_clean_scrub_stamp":"2024-03-31T15:04:38.358174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:04:39.365755+0000","last_change":"2024-03-31T15:04:39.365755+0000","last_active":"2024-03-31T15:04:39.365755+0000","last_peered":"2024-03-31T15:04:39.365755+0000","last_clean":"2024-03-31T15:04:39.365755+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:04:39.365755+0000","last_undegraded":"2024-03-31T15:04:39.365755+0000","last_fullsized":"2024-03-31T15:04:39.365755+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_clean_scrub_stamp":"2024-03-31T15:04:39.365755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:04:38.358174+0000","last_change":"2024-03-31T15:04:38.358174+0000","last_active":"2024-03-31T15:04:38.358174+0000","last_peered":"2024-03-31T15:04:38.358174+0000","last_clean":"2024-03-31T15:04:38.358174+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:04:38.358174+0000","last_undegraded":"2024-03-31T15:04:38.358174+0000","last_fullsized":"2024-03-31T15:04:38.358174+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:04:38.358174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:04:38.358174+0000","last_clean_scrub_stamp":"2024-03-31T15:04:38.358174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:04:39.365755+0000","last_change":"2024-03-31T15:04:39.365755+0000","last_active":"2024-03-31T15:04:39.365755+0000","last_peered":"2024-03-31T15:04:39.365755+0000","last_clean":"2024-03-31T15:04:39.365755+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:04:39.365755+0000","last_undegraded":"2024-03-31T15:04:39.365755+0000","last_fullsized":"2024-03-31T15:04:39.365755+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_clean_scrub_stamp":"2024-03-31T15:04:39.365755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:04:38.358174+0000","last_change":"2024-03-31T15:04:38.358174+0000","last_active":"2024-03-31T15:04:38.358174+0000","last_peered":"2024-03-31T15:04:38.358174+0000","last_clean":"2024-03-31T15:04:38.358174+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:04:38.358174+0000","last_undegraded":"2024-03-31T15:04:38.358174+0000","last_fullsized":"2024-03-31T15:04:38.358174+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:04:38.358174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:04:38.358174+0000","last_clean_scrub_stamp":"2024-03-31T15:04:38.358174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:04:39.365755+0000","last_change":"2024-03-31T15:04:39.365755+0000","last_active":"2024-03-31T15:04:39.365755+0000","last_peered":"2024-03-31T15:04:39.365755+0000","last_clean":"2024-03-31T15:04:39.365755+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:04:39.365755+0000","last_undegraded":"2024-03-31T15:04:39.365755+0000","last_fullsized":"2024-03-31T15:04:39.365755+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_clean_scrub_stamp":"2024-03-31T15:04:39.365755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:04:38.358174+0000","last_change":"2024-03-31T15:04:38.358174+0000","last_active":"2024-03-31T15:04:38.358174+0000","last_peered":"2024-03-31T15:04:38.358174+0000","last_clean":"2024-03-31T15:04:38.358174+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:04:38.358174+0000","last_undegraded":"2024-03-31T15:04:38.358174+0000","last_fullsized":"2024-03-31T15:04:38.358174+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:04:38.358174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:04:38.358174+0000","last_clean_scrub_stamp":"2024-03-31T15:04:38.358174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:04:39.365755+0000","last_change":"2024-03-31T15:04:39.365755+0000","last_active":"2024-03-31T15:04:39.365755+0000","last_peered":"2024-03-31T15:04:39.365755+0000","last_clean":"2024-03-31T15:04:39.365755+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:04:39.365755+0000","last_undegraded":"2024-03-31T15:04:39.365755+0000","last_fullsized":"2024-03-31T15:04:39.365755+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_clean_scrub_stamp":"2024-03-31T15:04:39.365755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:04:38.358174+0000","last_change":"2024-03-31T15:04:38.358174+0000","last_active":"2024-03-31T15:04:38.358174+0000","last_peered":"2024-03-31T15:04:38.358174+0000","last_clean":"2024-03-31T15:04:38.358174+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:04:38.358174+0000","last_undegraded":"2024-03-31T15:04:38.358174+0000","last_fullsized":"2024-03-31T15:04:38.358174+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:04:38.358174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:04:38.358174+0000","last_clean_scrub_stamp":"2024-03-31T15:04:38.358174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:04:39.365755+0000","last_change":"2024-03-31T15:04:39.365755+0000","last_active":"2024-03-31T15:04:39.365755+0000","last_peered":"2024-03-31T15:04:39.365755+0000","last_clean":"2024-03-31T15:04:39.365755+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:04:39.365755+0000","last_undegraded":"2024-03-31T15:04:39.365755+0000","last_fullsized":"2024-03-31T15:04:39.365755+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_clean_scrub_stamp":"2024-03-31T15:04:39.365755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-31T15:04:38.371216+0000","last_change":"2024-03-31T15:04:38.371216+0000","last_active":"2024-03-31T15:04:38.358174+0000","last_peered":"2024-03-31T15:04:38.358174+0000","last_clean":"2024-03-31T15:04:38.358174+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:04:38.371216+0000","last_undegraded":"2024-03-31T15:04:38.371216+0000","last_fullsized":"2024-03-31T15:04:38.371216+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:04:38.358174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:04:38.358174+0000","last_clean_scrub_stamp":"2024-03-31T15:04:38.358174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:04:39.365755+0000","last_change":"2024-03-31T15:04:39.365755+0000","last_active":"2024-03-31T15:04:39.365755+0000","last_peered":"2024-03-31T15:04:39.365755+0000","last_clean":"2024-03-31T15:04:39.365755+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:04:39.365755+0000","last_undegraded":"2024-03-31T15:04:39.365755+0000","last_fullsized":"2024-03-31T15:04:39.365755+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_clean_scrub_stamp":"2024-03-31T15:04:39.365755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:04:38.358174+0000","last_change":"2024-03-31T15:04:38.358174+0000","last_active":"2024-03-31T15:04:38.358174+0000","last_peered":"2024-03-31T15:04:38.358174+0000","last_clean":"2024-03-31T15:04:38.358174+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:04:38.358174+0000","last_undegraded":"2024-03-31T15:04:38.358174+0000","last_fullsized":"2024-03-31T15:04:38.358174+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:04:38.358174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:04:38.358174+0000","last_clean_scrub_stamp":"2024-03-31T15:04:38.358174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:04:39.365755+0000","last_change":"2024-03-31T15:04:39.365755+0000","last_active":"2024-03-31T15:04:39.365755+0000","last_peered":"2024-03-31T15:04:39.365755+0000","last_clean":"2024-03-31T15:04:39.365755+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:04:39.365755+0000","last_undegraded":"2024-03-31T15:04:39.365755+0000","last_fullsized":"2024-03-31T15:04:39.365755+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_clean_scrub_stamp":"2024-03-31T15:04:39.365755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:04:38.358174+0000","last_change":"2024-03-31T15:04:38.358174+0000","last_active":"2024-03-31T15:04:38.358174+0000","last_peered":"2024-03-31T15:04:38.358174+0000","last_clean":"2024-03-31T15:04:38.358174+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:04:38.358174+0000","last_undegraded":"2024-03-31T15:04:38.358174+0000","last_fullsized":"2024-03-31T15:04:38.358174+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:04:38.358174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:04:38.358174+0000","last_clean_scrub_stamp":"2024-03-31T15:04:38.358174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":188,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-31T15:04:38.374449+0000","last_change":"2024-03-31T15:03:24.329890+0000","last_active":"2024-03-31T15:04:38.374449+0000","last_peered":"2024-03-31T15:04:38.374449+0000","last_clean":"2024-03-31T15:04:38.374449+0000","last_became_active":"2024-03-31T15:03:24.328882+0000","last_became_peered":"2024-03-31T15:03:24.328882+0000","last_unstale":"2024-03-31T15:04:38.374449+0000","last_undegraded":"2024-03-31T15:04:38.374449+0000","last_fullsized":"2024-03-31T15:04:38.374449+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:22.885535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:22.885535+0000","last_clean_scrub_stamp":"2024-03-31T15:03:22.885535+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:18:29.004296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509460,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27932,"kb_used_data":912,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343908,"statfs":{"total":96636764160,"available":96608161792,"internally_reserved":0,"allocated":933888,"data_stored":719595,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509459,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27340,"kb_used_data":320,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344500,"statfs":{"total":96636764160,"available":96608768000,"internally_reserved":0,"allocated":327680,"data_stored":123410,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":15,"seq":64424509458,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27356,"kb_used_data":320,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344484,"statfs":{"total":96636764160,"available":96608751616,"internally_reserved":0,"allocated":327680,"data_stored":123410,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509457,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31504,"kb_used_data":320,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340336,"statfs":{"total":96636764160,"available":96604504064,"internally_reserved":0,"allocated":327680,"data_stored":123410,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542163,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27352,"kb_used_data":320,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344488,"statfs":{"total":96636764160,"available":96608755712,"internally_reserved":0,"allocated":327680,"data_stored":123410,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542163,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31504,"kb_used_data":320,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340336,"statfs":{"total":96636764160,"available":96604504064,"internally_reserved":0,"allocated":327680,"data_stored":123410,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542163,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31500,"kb_used_data":320,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340340,"statfs":{"total":96636764160,"available":96604508160,"internally_reserved":0,"allocated":327680,"data_stored":123410,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542163,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32092,"kb_used_data":900,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339748,"statfs":{"total":96636764160,"available":96603901952,"internally_reserved":0,"allocated":921600,"data_stored":713778,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-31T15:04:44.504 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-31T15:04:44.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:44.684+0000 7f945cc00640 1 Processor -- start 2024-03-31T15:04:44.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:44.684+0000 7f945cc00640 1 -- start start 2024-03-31T15:04:44.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:44.688+0000 7f945cc00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f9458167130 0x7f945806f920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:44.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:44.688+0000 7f945cc00640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9458163020 con 0x7f945806fe60 2024-03-31T15:04:44.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:44.688+0000 7f945cc00640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9458163300 con 0x7f9458147b50 2024-03-31T15:04:44.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:44.688+0000 7f945cc00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f94581460d0 con 0x7f9458167130 2024-03-31T15:04:44.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:44.688+0000 7f9456800640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f945806fe60 0x7f945815fb00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:38856/0 (socket says 172.21.15.16:38856) 2024-03-31T15:04:44.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:44.688+0000 7f9456800640 1 -- 172.21.15.16:0/514847264 learned_addr learned my addr 172.21.15.16:0/514847264 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:04:44.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:44.688+0000 7f944d400640 1 --2- 172.21.15.16:0/514847264 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f9458167130 0x7f945806f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:44.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:44.688+0000 7f9455400640 1 -- 172.21.15.16:0/514847264 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3312482748 0 0) 0x7f9458163020 con 0x7f945806fe60 2024-03-31T15:04:44.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:44.688+0000 7f9455400640 1 -- 172.21.15.16:0/514847264 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f943c003530 con 0x7f945806fe60 2024-03-31T15:04:44.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:44.688+0000 7f9455400640 1 -- 172.21.15.16:0/514847264 <== mon.1 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2490779614 0 0) 0x7f9458163300 con 0x7f9458147b50 2024-03-31T15:04:44.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:44.688+0000 7f9455400640 1 -- 172.21.15.16:0/514847264 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9458163020 con 0x7f9458147b50 2024-03-31T15:04:44.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:44.688+0000 7f9455400640 1 -- 172.21.15.16:0/514847264 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 227176820 0 0) 0x7f943c003530 con 0x7f945806fe60 2024-03-31T15:04:44.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:44.688+0000 7f9455400640 1 -- 172.21.15.16:0/514847264 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f9458167130 msgr2=0x7f945806f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:04:44.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:44.688+0000 7f9455400640 1 --2- 172.21.15.16:0/514847264 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f9458167130 0x7f945806f920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:44.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:44.688+0000 7f9455400640 1 -- 172.21.15.16:0/514847264 >> v1:172.21.15.89:6789/0 conn(0x7f9458147b50 legacy=0x7f94580cfcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:44.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:44.688+0000 7f9455400640 1 -- 172.21.15.16:0/514847264 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94581635e0 con 0x7f945806fe60 2024-03-31T15:04:44.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:44.688+0000 7f9455400640 1 -- 172.21.15.16:0/514847264 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 214807623 0 0) 0x7f9450001a00 con 0x7f945806fe60 2024-03-31T15:04:44.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:44.688+0000 7f9455400640 1 -- 172.21.15.16:0/514847264 >> v1:172.21.15.16:6789/0 conn(0x7f945806fe60 legacy=0x7f945815fb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:44.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:44.688+0000 7f9455400640 1 --2- 172.21.15.16:0/514847264 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f943c003b80 0x7f943c003f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:44.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:44.688+0000 7f9455400640 1 --2- 172.21.15.16:0/514847264 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f943c004810 0x7f943c004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:44.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:44.688+0000 7f9455400640 1 --2- 172.21.15.16:0/514847264 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f943c0051d0 0x7f943c00b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:44.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:44.688+0000 7f9455400640 1 -- 172.21.15.16:0/514847264 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f943c00bfc0 con 0x7f943c004810 2024-03-31T15:04:44.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:44.688+0000 7f9455400640 1 -- 172.21.15.16:0/514847264 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f943c00c140 con 0x7f943c003b80 2024-03-31T15:04:44.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:44.688+0000 7f9455400640 1 -- 172.21.15.16:0/514847264 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f943c00c340 con 0x7f943c0051d0 2024-03-31T15:04:44.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:44.688+0000 7f944d400640 1 --2- 172.21.15.16:0/514847264 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f9458167130 0x7f945806f920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:04:44.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:44.688+0000 7f9455e00640 1 --2- 172.21.15.16:0/514847264 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f943c003b80 0x7f943c003f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:44.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:44.688+0000 7f9456800640 1 --2- 172.21.15.16:0/514847264 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f943c0051d0 0x7f943c00b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:44.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:44.688+0000 7f944d400640 1 --2- 172.21.15.16:0/514847264 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f943c004810 0x7f943c004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:44.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:44.688+0000 7f944d400640 1 -- 172.21.15.16:0/514847264 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f943c0051d0 msgr2=0x7f943c00b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:04:44.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:44.688+0000 7f944d400640 1 --2- 172.21.15.16:0/514847264 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f943c0051d0 0x7f943c00b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:44.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:44.688+0000 7f944d400640 1 -- 172.21.15.16:0/514847264 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f943c003b80 msgr2=0x7f943c003f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:04:44.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:44.688+0000 7f944d400640 1 --2- 172.21.15.16:0/514847264 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f943c003b80 0x7f943c003f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:44.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:44.688+0000 7f944d400640 1 -- 172.21.15.16:0/514847264 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f943c003530 con 0x7f943c004810 2024-03-31T15:04:44.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:44.688+0000 7f9455e00640 1 --2- 172.21.15.16:0/514847264 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f943c003b80 0x7f943c003f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:04:44.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:44.692+0000 7f944d400640 1 --2- 172.21.15.16:0/514847264 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f943c004810 0x7f943c004c90 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f94580d15c0 tx=0x7f944001ca70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=973a4a2e8ad4b77 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:44.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:44.692+0000 7f9455400640 1 -- 172.21.15.16:0/514847264 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9440009810 con 0x7f943c004810 2024-03-31T15:04:44.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:44.692+0000 7f9455400640 1 -- 172.21.15.16:0/514847264 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9440004030 con 0x7f943c004810 2024-03-31T15:04:44.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:44.692+0000 7f9455400640 1 -- 172.21.15.16:0/514847264 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9440004400 con 0x7f943c004810 2024-03-31T15:04:44.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:44.692+0000 7f945cc00640 1 -- 172.21.15.16:0/514847264 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f943c004810 msgr2=0x7f943c004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:44.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:44.692+0000 7f945cc00640 1 --2- 172.21.15.16:0/514847264 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f943c004810 0x7f943c004c90 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f94580d15c0 tx=0x7f944001ca70 comp rx=0 tx=0).stop 2024-03-31T15:04:44.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:44.692+0000 7f9456800640 1 -- 172.21.15.16:0/514847264 reap_dead start 2024-03-31T15:04:44.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:44.692+0000 7f945cc00640 1 -- 172.21.15.16:0/514847264 shutdown_connections 2024-03-31T15:04:44.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:44.692+0000 7f945cc00640 1 -- 172.21.15.16:0/514847264 >> 172.21.15.16:0/514847264 conn(0x7f9458067330 msgr2=0x7f9458067740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:04:44.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:44.692+0000 7f945cc00640 1 -- 172.21.15.16:0/514847264 shutdown_connections 2024-03-31T15:04:44.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:44.692+0000 7f945cc00640 1 -- 172.21.15.16:0/514847264 wait complete. 2024-03-31T15:04:44.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:44.696+0000 7f945cc00640 1 Processor -- start 2024-03-31T15:04:44.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:44.696+0000 7f945cc00640 1 -- start start 2024-03-31T15:04:44.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:44.696+0000 7f945cc00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f943c004810 0x7f94581461a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:44.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:44.696+0000 7f945cc00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9458163f60 0x7f94581643e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:44.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:44.696+0000 7f945cc00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f94580b3fd0 0x7f94580b4450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:44.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:44.696+0000 7f945cc00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f94580d05c0 con 0x7f943c004810 2024-03-31T15:04:44.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:44.696+0000 7f945cc00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f9458163de0 con 0x7f9458163f60 2024-03-31T15:04:44.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:44.696+0000 7f945cc00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f9458148f40 con 0x7f94580b3fd0 2024-03-31T15:04:44.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:44.696+0000 7f9455e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f943c004810 0x7f94581461a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:44.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:44.696+0000 7f9455e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f943c004810 0x7f94581461a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54266/0 (socket says 172.21.15.16:54266) 2024-03-31T15:04:44.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:44.696+0000 7f9455e00640 1 -- 172.21.15.16:0/2710810271 learned_addr learned my addr 172.21.15.16:0/2710810271 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:04:44.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:44.696+0000 7f9456800640 1 --2- 172.21.15.16:0/2710810271 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f94580b3fd0 0x7f94580b4450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:44.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:44.696+0000 7f944d400640 1 --2- 172.21.15.16:0/2710810271 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9458163f60 0x7f94581643e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:44.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:44.696+0000 7f9455e00640 1 -- 172.21.15.16:0/2710810271 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f94580b3fd0 msgr2=0x7f94580b4450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:44.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:44.696+0000 7f9455e00640 1 --2- 172.21.15.16:0/2710810271 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f94580b3fd0 0x7f94580b4450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:44.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:44.696+0000 7f9455e00640 1 -- 172.21.15.16:0/2710810271 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9458163f60 msgr2=0x7f94581643e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:44.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:44.696+0000 7f9455e00640 1 --2- 172.21.15.16:0/2710810271 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9458163f60 0x7f94581643e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:44.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:44.696+0000 7f9455e00640 1 -- 172.21.15.16:0/2710810271 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94580b4d40 con 0x7f943c004810 2024-03-31T15:04:44.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:44.696+0000 7f944d400640 1 --2- 172.21.15.16:0/2710810271 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9458163f60 0x7f94581643e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:04:44.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:44.696+0000 7f9456800640 1 --2- 172.21.15.16:0/2710810271 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f94580b3fd0 0x7f94580b4450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:04:44.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:44.696+0000 7f9455e00640 1 --2- 172.21.15.16:0/2710810271 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f943c004810 0x7f94581461a0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f94480129a0 tx=0x7f9448012e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:44.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:44.696+0000 7f944ea00640 1 -- 172.21.15.16:0/2710810271 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9448010c30 con 0x7f943c004810 2024-03-31T15:04:44.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:44.696+0000 7f944ea00640 1 -- 172.21.15.16:0/2710810271 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f944801b070 con 0x7f943c004810 2024-03-31T15:04:44.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:44.696+0000 7f945cc00640 1 -- 172.21.15.16:0/2710810271 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f94580b5de0 con 0x7f943c004810 2024-03-31T15:04:44.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:44.696+0000 7f944ea00640 1 -- 172.21.15.16:0/2710810271 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9448016480 con 0x7f943c004810 2024-03-31T15:04:44.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:44.696+0000 7f945cc00640 1 -- 172.21.15.16:0/2710810271 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f94580b63a0 con 0x7f943c004810 2024-03-31T15:04:44.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:44.700+0000 7f945cc00640 1 -- 172.21.15.16:0/2710810271 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9458069950 con 0x7f943c004810 2024-03-31T15:04:44.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:44.700+0000 7f944ea00640 1 -- 172.21.15.16:0/2710810271 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f9448016620 con 0x7f943c004810 2024-03-31T15:04:44.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:44.700+0000 7f944ea00640 1 --2- 172.21.15.16:0/2710810271 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f9434068380 0x7f943406a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:44.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:44.700+0000 7f944ea00640 1 -- 172.21.15.16:0/2710810271 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5709+0+0 (secure 0 0 0) 0x7f944808ac00 con 0x7f943c004810 2024-03-31T15:04:44.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:44.700+0000 7f944d400640 1 --2- 172.21.15.16:0/2710810271 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f9434068380 0x7f943406a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:44.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:44.704+0000 7f944d400640 1 --2- 172.21.15.16:0/2710810271 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f9434068380 0x7f943406a840 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f94400ae000 tx=0x7f94400046d0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:44.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:44.704+0000 7f944ea00640 1 -- 172.21.15.16:0/2710810271 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f944805b330 con 0x7f943c004810 2024-03-31T15:04:44.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:44.868+0000 7f945cc00640 1 -- 172.21.15.16:0/2710810271 --> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f94580cf3e0 con 0x7f9434068380 2024-03-31T15:04:44.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:44.876+0000 7f944ea00640 1 -- 172.21.15.16:0/2710810271 <== mgr.4100 v2:172.21.15.89:6832/1549407273 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251950 (secure 0 0 0) 0x7f94400fa840 con 0x7f9434068380 2024-03-31T15:04:44.873 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:04:44.874 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-03-31T15:04:44.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:44.880+0000 7f945cc00640 1 -- 172.21.15.16:0/2710810271 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f9434068380 msgr2=0x7f943406a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:44.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:44.880+0000 7f945cc00640 1 --2- 172.21.15.16:0/2710810271 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f9434068380 0x7f943406a840 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f94400ae000 tx=0x7f94400046d0 comp rx=0 tx=0).stop 2024-03-31T15:04:44.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:44.880+0000 7f945cc00640 1 -- 172.21.15.16:0/2710810271 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f943c004810 msgr2=0x7f94581461a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:44.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:44.880+0000 7f945cc00640 1 --2- 172.21.15.16:0/2710810271 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f943c004810 0x7f94581461a0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f94480129a0 tx=0x7f9448012e60 comp rx=0 tx=0).stop 2024-03-31T15:04:44.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:44.880+0000 7f945cc00640 1 -- 172.21.15.16:0/2710810271 shutdown_connections 2024-03-31T15:04:44.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:44.880+0000 7f945cc00640 1 --2- 172.21.15.16:0/2710810271 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f9434068380 0x7f943406a840 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:44.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:44.880+0000 7f945cc00640 1 --2- 172.21.15.16:0/2710810271 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f94580b3fd0 0x7f94580b4450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:44.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:44.880+0000 7f945cc00640 1 --2- 172.21.15.16:0/2710810271 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9458163f60 0x7f94581643e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:44.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:44.880+0000 7f945cc00640 1 --2- 172.21.15.16:0/2710810271 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f943c004810 0x7f94581461a0 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:44.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:44.880+0000 7f945cc00640 1 -- 172.21.15.16:0/2710810271 >> 172.21.15.16:0/2710810271 conn(0x7f9458067330 msgr2=0x7f9458067740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:04:44.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:44.880+0000 7f945cc00640 1 -- 172.21.15.16:0/2710810271 shutdown_connections 2024-03-31T15:04:44.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:44.880+0000 7f945cc00640 1 -- 172.21.15.16:0/2710810271 wait complete. 2024-03-31T15:04:44.893 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":68,"stamp":"2024-03-31T15:04:44.542977+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":156,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":236992,"kb_used_data":4156,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754737728,"statfs":{"total":773094113280,"available":772851433472,"internally_reserved":0,"allocated":4255744,"data_stored":2367808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":238276144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":31,"apply_latency_ms":31,"commit_latency_ns":31000000,"apply_latency_ns":31000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001212"},"pg_stats":[{"pgid":"5.3f","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-31T15:04:41.394601+0000","last_change":"2024-03-31T15:04:40.404357+0000","last_active":"2024-03-31T15:04:41.394601+0000","last_peered":"2024-03-31T15:04:41.394601+0000","last_clean":"2024-03-31T15:04:41.394601+0000","last_became_active":"2024-03-31T15:04:40.404046+0000","last_became_peered":"2024-03-31T15:04:40.404046+0000","last_unstale":"2024-03-31T15:04:41.394601+0000","last_undegraded":"2024-03-31T15:04:41.394601+0000","last_fullsized":"2024-03-31T15:04:41.394601+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_clean_scrub_stamp":"2024-03-31T15:04:39.365755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T17:54:29.313023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.3e","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-31T15:04:42.250678+0000","last_change":"2024-03-31T15:04:40.412717+0000","last_active":"2024-03-31T15:04:42.250678+0000","last_peered":"2024-03-31T15:04:42.250678+0000","last_clean":"2024-03-31T15:04:42.250678+0000","last_became_active":"2024-03-31T15:04:40.412198+0000","last_became_peered":"2024-03-31T15:04:40.412198+0000","last_unstale":"2024-03-31T15:04:42.250678+0000","last_undegraded":"2024-03-31T15:04:42.250678+0000","last_fullsized":"2024-03-31T15:04:42.250678+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_clean_scrub_stamp":"2024-03-31T15:04:39.365755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:14:46.625860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.3d","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-31T15:04:42.251999+0000","last_change":"2024-03-31T15:04:40.405673+0000","last_active":"2024-03-31T15:04:42.251999+0000","last_peered":"2024-03-31T15:04:42.251999+0000","last_clean":"2024-03-31T15:04:42.251999+0000","last_became_active":"2024-03-31T15:04:40.405390+0000","last_became_peered":"2024-03-31T15:04:40.405390+0000","last_unstale":"2024-03-31T15:04:42.251999+0000","last_undegraded":"2024-03-31T15:04:42.251999+0000","last_fullsized":"2024-03-31T15:04:42.251999+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_clean_scrub_stamp":"2024-03-31T15:04:39.365755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:24:38.674845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3c","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-31T15:04:41.431053+0000","last_change":"2024-03-31T15:04:40.411960+0000","last_active":"2024-03-31T15:04:41.431053+0000","last_peered":"2024-03-31T15:04:41.431053+0000","last_clean":"2024-03-31T15:04:41.431053+0000","last_became_active":"2024-03-31T15:04:40.411417+0000","last_became_peered":"2024-03-31T15:04:40.411417+0000","last_unstale":"2024-03-31T15:04:41.431053+0000","last_undegraded":"2024-03-31T15:04:41.431053+0000","last_fullsized":"2024-03-31T15:04:41.431053+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_clean_scrub_stamp":"2024-03-31T15:04:39.365755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:16:50.219029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3b","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-31T15:04:41.392303+0000","last_change":"2024-03-31T15:04:40.396904+0000","last_active":"2024-03-31T15:04:41.392303+0000","last_peered":"2024-03-31T15:04:41.392303+0000","last_clean":"2024-03-31T15:04:41.392303+0000","last_became_active":"2024-03-31T15:04:40.396520+0000","last_became_peered":"2024-03-31T15:04:40.396520+0000","last_unstale":"2024-03-31T15:04:41.392303+0000","last_undegraded":"2024-03-31T15:04:41.392303+0000","last_fullsized":"2024-03-31T15:04:41.392303+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_clean_scrub_stamp":"2024-03-31T15:04:39.365755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T19:25:11.258946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3a","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-31T15:04:42.250331+0000","last_change":"2024-03-31T15:04:40.394354+0000","last_active":"2024-03-31T15:04:42.250331+0000","last_peered":"2024-03-31T15:04:42.250331+0000","last_clean":"2024-03-31T15:04:42.250331+0000","last_became_active":"2024-03-31T15:04:40.394076+0000","last_became_peered":"2024-03-31T15:04:40.394076+0000","last_unstale":"2024-03-31T15:04:42.250331+0000","last_undegraded":"2024-03-31T15:04:42.250331+0000","last_fullsized":"2024-03-31T15:04:42.250331+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_clean_scrub_stamp":"2024-03-31T15:04:39.365755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:19:02.982248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.39","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-31T15:04:42.251637+0000","last_change":"2024-03-31T15:04:40.408100+0000","last_active":"2024-03-31T15:04:42.251637+0000","last_peered":"2024-03-31T15:04:42.251637+0000","last_clean":"2024-03-31T15:04:42.251637+0000","last_became_active":"2024-03-31T15:04:40.407598+0000","last_became_peered":"2024-03-31T15:04:40.407598+0000","last_unstale":"2024-03-31T15:04:42.251637+0000","last_undegraded":"2024-03-31T15:04:42.251637+0000","last_fullsized":"2024-03-31T15:04:42.251637+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_clean_scrub_stamp":"2024-03-31T15:04:39.365755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:49:21.664326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.38","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-31T15:04:42.251471+0000","last_change":"2024-03-31T15:04:40.401055+0000","last_active":"2024-03-31T15:04:42.251471+0000","last_peered":"2024-03-31T15:04:42.251471+0000","last_clean":"2024-03-31T15:04:42.251471+0000","last_became_active":"2024-03-31T15:04:40.400692+0000","last_became_peered":"2024-03-31T15:04:40.400692+0000","last_unstale":"2024-03-31T15:04:42.251471+0000","last_undegraded":"2024-03-31T15:04:42.251471+0000","last_fullsized":"2024-03-31T15:04:42.251471+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_clean_scrub_stamp":"2024-03-31T15:04:39.365755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:02:36.920594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.37","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-31T15:04:41.394700+0000","last_change":"2024-03-31T15:04:40.394494+0000","last_active":"2024-03-31T15:04:41.394700+0000","last_peered":"2024-03-31T15:04:41.394700+0000","last_clean":"2024-03-31T15:04:41.394700+0000","last_became_active":"2024-03-31T15:04:40.394228+0000","last_became_peered":"2024-03-31T15:04:40.394228+0000","last_unstale":"2024-03-31T15:04:41.394700+0000","last_undegraded":"2024-03-31T15:04:41.394700+0000","last_fullsized":"2024-03-31T15:04:41.394700+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_clean_scrub_stamp":"2024-03-31T15:04:39.365755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:26:27.014855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.36","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-31T15:04:41.431400+0000","last_change":"2024-03-31T15:04:40.398581+0000","last_active":"2024-03-31T15:04:41.431400+0000","last_peered":"2024-03-31T15:04:41.431400+0000","last_clean":"2024-03-31T15:04:41.431400+0000","last_became_active":"2024-03-31T15:04:40.398317+0000","last_became_peered":"2024-03-31T15:04:40.398317+0000","last_unstale":"2024-03-31T15:04:41.431400+0000","last_undegraded":"2024-03-31T15:04:41.431400+0000","last_fullsized":"2024-03-31T15:04:41.431400+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_clean_scrub_stamp":"2024-03-31T15:04:39.365755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T19:01:35.364432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.35","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-31T15:04:42.252014+0000","last_change":"2024-03-31T15:04:40.406611+0000","last_active":"2024-03-31T15:04:42.252014+0000","last_peered":"2024-03-31T15:04:42.252014+0000","last_clean":"2024-03-31T15:04:42.252014+0000","last_became_active":"2024-03-31T15:04:40.406319+0000","last_became_peered":"2024-03-31T15:04:40.406319+0000","last_unstale":"2024-03-31T15:04:42.252014+0000","last_undegraded":"2024-03-31T15:04:42.252014+0000","last_fullsized":"2024-03-31T15:04:42.252014+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_clean_scrub_stamp":"2024-03-31T15:04:39.365755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:20:29.402897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.34","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-31T15:04:42.251885+0000","last_change":"2024-03-31T15:04:40.403099+0000","last_active":"2024-03-31T15:04:42.251885+0000","last_peered":"2024-03-31T15:04:42.251885+0000","last_clean":"2024-03-31T15:04:42.251885+0000","last_became_active":"2024-03-31T15:04:40.402793+0000","last_became_peered":"2024-03-31T15:04:40.402793+0000","last_unstale":"2024-03-31T15:04:42.251885+0000","last_undegraded":"2024-03-31T15:04:42.251885+0000","last_fullsized":"2024-03-31T15:04:42.251885+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_clean_scrub_stamp":"2024-03-31T15:04:39.365755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:20:49.393792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.33","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-31T15:04:41.392545+0000","last_change":"2024-03-31T15:04:40.406940+0000","last_active":"2024-03-31T15:04:41.392545+0000","last_peered":"2024-03-31T15:04:41.392545+0000","last_clean":"2024-03-31T15:04:41.392545+0000","last_became_active":"2024-03-31T15:04:40.406643+0000","last_became_peered":"2024-03-31T15:04:40.406643+0000","last_unstale":"2024-03-31T15:04:41.392545+0000","last_undegraded":"2024-03-31T15:04:41.392545+0000","last_fullsized":"2024-03-31T15:04:41.392545+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_clean_scrub_stamp":"2024-03-31T15:04:39.365755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:37:21.007221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.32","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-31T15:04:42.250349+0000","last_change":"2024-03-31T15:04:40.395851+0000","last_active":"2024-03-31T15:04:42.250349+0000","last_peered":"2024-03-31T15:04:42.250349+0000","last_clean":"2024-03-31T15:04:42.250349+0000","last_became_active":"2024-03-31T15:04:40.395550+0000","last_became_peered":"2024-03-31T15:04:40.395550+0000","last_unstale":"2024-03-31T15:04:42.250349+0000","last_undegraded":"2024-03-31T15:04:42.250349+0000","last_fullsized":"2024-03-31T15:04:42.250349+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_clean_scrub_stamp":"2024-03-31T15:04:39.365755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:06:18.551532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.31","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-31T15:04:42.251377+0000","last_change":"2024-03-31T15:04:40.405445+0000","last_active":"2024-03-31T15:04:42.251377+0000","last_peered":"2024-03-31T15:04:42.251377+0000","last_clean":"2024-03-31T15:04:42.251377+0000","last_became_active":"2024-03-31T15:04:40.404925+0000","last_became_peered":"2024-03-31T15:04:40.404925+0000","last_unstale":"2024-03-31T15:04:42.251377+0000","last_undegraded":"2024-03-31T15:04:42.251377+0000","last_fullsized":"2024-03-31T15:04:42.251377+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_clean_scrub_stamp":"2024-03-31T15:04:39.365755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:02:36.920594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.30","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-31T15:04:42.252078+0000","last_change":"2024-03-31T15:04:40.399699+0000","last_active":"2024-03-31T15:04:42.252078+0000","last_peered":"2024-03-31T15:04:42.252078+0000","last_clean":"2024-03-31T15:04:42.252078+0000","last_became_active":"2024-03-31T15:04:40.399105+0000","last_became_peered":"2024-03-31T15:04:40.399105+0000","last_unstale":"2024-03-31T15:04:42.252078+0000","last_undegraded":"2024-03-31T15:04:42.252078+0000","last_fullsized":"2024-03-31T15:04:42.252078+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_clean_scrub_stamp":"2024-03-31T15:04:39.365755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T15:31:24.450431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2f","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-31T15:04:42.252412+0000","last_change":"2024-03-31T15:04:40.404154+0000","last_active":"2024-03-31T15:04:42.252412+0000","last_peered":"2024-03-31T15:04:42.252412+0000","last_clean":"2024-03-31T15:04:42.252412+0000","last_became_active":"2024-03-31T15:04:40.403852+0000","last_became_peered":"2024-03-31T15:04:40.403852+0000","last_unstale":"2024-03-31T15:04:42.252412+0000","last_undegraded":"2024-03-31T15:04:42.252412+0000","last_fullsized":"2024-03-31T15:04:42.252412+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_clean_scrub_stamp":"2024-03-31T15:04:39.365755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:04:06.091897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2e","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-31T15:04:42.250721+0000","last_change":"2024-03-31T15:04:40.414584+0000","last_active":"2024-03-31T15:04:42.250721+0000","last_peered":"2024-03-31T15:04:42.250721+0000","last_clean":"2024-03-31T15:04:42.250721+0000","last_became_active":"2024-03-31T15:04:40.413522+0000","last_became_peered":"2024-03-31T15:04:40.413522+0000","last_unstale":"2024-03-31T15:04:42.250721+0000","last_undegraded":"2024-03-31T15:04:42.250721+0000","last_fullsized":"2024-03-31T15:04:42.250721+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_clean_scrub_stamp":"2024-03-31T15:04:39.365755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T16:15:37.002470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2d","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-31T15:04:42.250616+0000","last_change":"2024-03-31T15:04:40.414545+0000","last_active":"2024-03-31T15:04:42.250616+0000","last_peered":"2024-03-31T15:04:42.250616+0000","last_clean":"2024-03-31T15:04:42.250616+0000","last_became_active":"2024-03-31T15:04:40.413782+0000","last_became_peered":"2024-03-31T15:04:40.413782+0000","last_unstale":"2024-03-31T15:04:42.250616+0000","last_undegraded":"2024-03-31T15:04:42.250616+0000","last_fullsized":"2024-03-31T15:04:42.250616+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_clean_scrub_stamp":"2024-03-31T15:04:39.365755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:47:37.436509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2c","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-31T15:04:41.392702+0000","last_change":"2024-03-31T15:04:40.401434+0000","last_active":"2024-03-31T15:04:41.392702+0000","last_peered":"2024-03-31T15:04:41.392702+0000","last_clean":"2024-03-31T15:04:41.392702+0000","last_became_active":"2024-03-31T15:04:40.401138+0000","last_became_peered":"2024-03-31T15:04:40.401138+0000","last_unstale":"2024-03-31T15:04:41.392702+0000","last_undegraded":"2024-03-31T15:04:41.392702+0000","last_fullsized":"2024-03-31T15:04:41.392702+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_clean_scrub_stamp":"2024-03-31T15:04:39.365755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:32:07.091323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2b","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-31T15:04:41.393850+0000","last_change":"2024-03-31T15:04:40.414816+0000","last_active":"2024-03-31T15:04:41.393850+0000","last_peered":"2024-03-31T15:04:41.393850+0000","last_clean":"2024-03-31T15:04:41.393850+0000","last_became_active":"2024-03-31T15:04:40.414582+0000","last_became_peered":"2024-03-31T15:04:40.414582+0000","last_unstale":"2024-03-31T15:04:41.393850+0000","last_undegraded":"2024-03-31T15:04:41.393850+0000","last_fullsized":"2024-03-31T15:04:41.393850+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_clean_scrub_stamp":"2024-03-31T15:04:39.365755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T16:10:00.592586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2a","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-31T15:04:41.394964+0000","last_change":"2024-03-31T15:04:40.401054+0000","last_active":"2024-03-31T15:04:41.394964+0000","last_peered":"2024-03-31T15:04:41.394964+0000","last_clean":"2024-03-31T15:04:41.394964+0000","last_became_active":"2024-03-31T15:04:40.400741+0000","last_became_peered":"2024-03-31T15:04:40.400741+0000","last_unstale":"2024-03-31T15:04:41.394964+0000","last_undegraded":"2024-03-31T15:04:41.394964+0000","last_fullsized":"2024-03-31T15:04:41.394964+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_clean_scrub_stamp":"2024-03-31T15:04:39.365755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:42:01.248151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.29","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-31T15:04:41.935386+0000","last_change":"2024-03-31T15:04:40.411250+0000","last_active":"2024-03-31T15:04:41.935386+0000","last_peered":"2024-03-31T15:04:41.935386+0000","last_clean":"2024-03-31T15:04:41.935386+0000","last_became_active":"2024-03-31T15:04:40.402059+0000","last_became_peered":"2024-03-31T15:04:40.402059+0000","last_unstale":"2024-03-31T15:04:41.935386+0000","last_undegraded":"2024-03-31T15:04:41.935386+0000","last_fullsized":"2024-03-31T15:04:41.935386+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_clean_scrub_stamp":"2024-03-31T15:04:39.365755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:32:38.931716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.28","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-31T15:04:42.251397+0000","last_change":"2024-03-31T15:04:40.395022+0000","last_active":"2024-03-31T15:04:42.251397+0000","last_peered":"2024-03-31T15:04:42.251397+0000","last_clean":"2024-03-31T15:04:42.251397+0000","last_became_active":"2024-03-31T15:04:40.394130+0000","last_became_peered":"2024-03-31T15:04:40.394130+0000","last_unstale":"2024-03-31T15:04:42.251397+0000","last_undegraded":"2024-03-31T15:04:42.251397+0000","last_fullsized":"2024-03-31T15:04:42.251397+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_clean_scrub_stamp":"2024-03-31T15:04:39.365755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:12:19.845843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.27","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-31T15:04:41.431481+0000","last_change":"2024-03-31T15:04:40.404883+0000","last_active":"2024-03-31T15:04:41.431481+0000","last_peered":"2024-03-31T15:04:41.431481+0000","last_clean":"2024-03-31T15:04:41.431481+0000","last_became_active":"2024-03-31T15:04:40.404443+0000","last_became_peered":"2024-03-31T15:04:40.404443+0000","last_unstale":"2024-03-31T15:04:41.431481+0000","last_undegraded":"2024-03-31T15:04:41.431481+0000","last_fullsized":"2024-03-31T15:04:41.431481+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_clean_scrub_stamp":"2024-03-31T15:04:39.365755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:30:39.361161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.26","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-31T15:04:42.250820+0000","last_change":"2024-03-31T15:04:40.408887+0000","last_active":"2024-03-31T15:04:42.250820+0000","last_peered":"2024-03-31T15:04:42.250820+0000","last_clean":"2024-03-31T15:04:42.250820+0000","last_became_active":"2024-03-31T15:04:40.408602+0000","last_became_peered":"2024-03-31T15:04:40.408602+0000","last_unstale":"2024-03-31T15:04:42.250820+0000","last_undegraded":"2024-03-31T15:04:42.250820+0000","last_fullsized":"2024-03-31T15:04:42.250820+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_clean_scrub_stamp":"2024-03-31T15:04:39.365755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T17:04:36.573109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.25","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-31T15:04:41.431414+0000","last_change":"2024-03-31T15:04:40.394081+0000","last_active":"2024-03-31T15:04:41.431414+0000","last_peered":"2024-03-31T15:04:41.431414+0000","last_clean":"2024-03-31T15:04:41.431414+0000","last_became_active":"2024-03-31T15:04:40.393784+0000","last_became_peered":"2024-03-31T15:04:40.393784+0000","last_unstale":"2024-03-31T15:04:41.431414+0000","last_undegraded":"2024-03-31T15:04:41.431414+0000","last_fullsized":"2024-03-31T15:04:41.431414+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_clean_scrub_stamp":"2024-03-31T15:04:39.365755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:24:35.438931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.24","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-31T15:04:42.252033+0000","last_change":"2024-03-31T15:04:40.418250+0000","last_active":"2024-03-31T15:04:42.252033+0000","last_peered":"2024-03-31T15:04:42.252033+0000","last_clean":"2024-03-31T15:04:42.252033+0000","last_became_active":"2024-03-31T15:04:40.417880+0000","last_became_peered":"2024-03-31T15:04:40.417880+0000","last_unstale":"2024-03-31T15:04:42.252033+0000","last_undegraded":"2024-03-31T15:04:42.252033+0000","last_fullsized":"2024-03-31T15:04:42.252033+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_clean_scrub_stamp":"2024-03-31T15:04:39.365755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:20:29.402897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.23","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-31T15:04:41.394017+0000","last_change":"2024-03-31T15:04:40.406957+0000","last_active":"2024-03-31T15:04:41.394017+0000","last_peered":"2024-03-31T15:04:41.394017+0000","last_clean":"2024-03-31T15:04:41.394017+0000","last_became_active":"2024-03-31T15:04:40.397366+0000","last_became_peered":"2024-03-31T15:04:40.397366+0000","last_unstale":"2024-03-31T15:04:41.394017+0000","last_undegraded":"2024-03-31T15:04:41.394017+0000","last_fullsized":"2024-03-31T15:04:41.394017+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_clean_scrub_stamp":"2024-03-31T15:04:39.365755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:30:55.409106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.22","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-31T15:04:42.251988+0000","last_change":"2024-03-31T15:04:40.397540+0000","last_active":"2024-03-31T15:04:42.251988+0000","last_peered":"2024-03-31T15:04:42.251988+0000","last_clean":"2024-03-31T15:04:42.251988+0000","last_became_active":"2024-03-31T15:04:40.396254+0000","last_became_peered":"2024-03-31T15:04:40.396254+0000","last_unstale":"2024-03-31T15:04:42.251988+0000","last_undegraded":"2024-03-31T15:04:42.251988+0000","last_fullsized":"2024-03-31T15:04:42.251988+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_clean_scrub_stamp":"2024-03-31T15:04:39.365755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:24:38.674845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.21","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-31T15:04:41.393909+0000","last_change":"2024-03-31T15:04:40.399496+0000","last_active":"2024-03-31T15:04:41.393909+0000","last_peered":"2024-03-31T15:04:41.393909+0000","last_clean":"2024-03-31T15:04:41.393909+0000","last_became_active":"2024-03-31T15:04:40.399001+0000","last_became_peered":"2024-03-31T15:04:40.399001+0000","last_unstale":"2024-03-31T15:04:41.393909+0000","last_undegraded":"2024-03-31T15:04:41.393909+0000","last_fullsized":"2024-03-31T15:04:41.393909+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_clean_scrub_stamp":"2024-03-31T15:04:39.365755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:42:37.682542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.20","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-31T15:04:42.251379+0000","last_change":"2024-03-31T15:04:40.408708+0000","last_active":"2024-03-31T15:04:42.251379+0000","last_peered":"2024-03-31T15:04:42.251379+0000","last_clean":"2024-03-31T15:04:42.251379+0000","last_became_active":"2024-03-31T15:04:40.407656+0000","last_became_peered":"2024-03-31T15:04:40.407656+0000","last_unstale":"2024-03-31T15:04:42.251379+0000","last_undegraded":"2024-03-31T15:04:42.251379+0000","last_fullsized":"2024-03-31T15:04:42.251379+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_clean_scrub_stamp":"2024-03-31T15:04:39.365755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T15:34:02.333360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-31T15:04:41.935633+0000","last_change":"2024-03-31T15:04:40.417293+0000","last_active":"2024-03-31T15:04:41.935633+0000","last_peered":"2024-03-31T15:04:41.935633+0000","last_clean":"2024-03-31T15:04:41.935633+0000","last_became_active":"2024-03-31T15:04:40.416638+0000","last_became_peered":"2024-03-31T15:04:40.416638+0000","last_unstale":"2024-03-31T15:04:41.935633+0000","last_undegraded":"2024-03-31T15:04:41.935633+0000","last_fullsized":"2024-03-31T15:04:41.935633+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_clean_scrub_stamp":"2024-03-31T15:04:39.365755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T19:18:11.771501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"33'4","reported_seq":18,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-31T15:04:41.394713+0000","last_change":"2024-03-31T15:04:39.390983+0000","last_active":"2024-03-31T15:04:41.394713+0000","last_peered":"2024-03-31T15:04:41.394713+0000","last_clean":"2024-03-31T15:04:41.394713+0000","last_became_active":"2024-03-31T15:04:39.390687+0000","last_became_peered":"2024-03-31T15:04:39.390687+0000","last_unstale":"2024-03-31T15:04:41.394713+0000","last_undegraded":"2024-03-31T15:04:41.394713+0000","last_fullsized":"2024-03-31T15:04:41.394713+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:04:38.358174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:04:38.358174+0000","last_clean_scrub_stamp":"2024-03-31T15:04:38.358174+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:00:01.104469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-31T15:04:41.394698+0000","last_change":"2024-03-31T15:04:40.408864+0000","last_active":"2024-03-31T15:04:41.394698+0000","last_peered":"2024-03-31T15:04:41.394698+0000","last_clean":"2024-03-31T15:04:41.394698+0000","last_became_active":"2024-03-31T15:04:40.408585+0000","last_became_peered":"2024-03-31T15:04:40.408585+0000","last_unstale":"2024-03-31T15:04:41.394698+0000","last_undegraded":"2024-03-31T15:04:41.394698+0000","last_fullsized":"2024-03-31T15:04:41.394698+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_clean_scrub_stamp":"2024-03-31T15:04:39.365755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T15:42:06.044034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"33'1","reported_seq":15,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-31T15:04:41.935618+0000","last_change":"2024-03-31T15:04:39.391764+0000","last_active":"2024-03-31T15:04:41.935618+0000","last_peered":"2024-03-31T15:04:41.935618+0000","last_clean":"2024-03-31T15:04:41.935618+0000","last_became_active":"2024-03-31T15:04:39.391506+0000","last_became_peered":"2024-03-31T15:04:39.391506+0000","last_unstale":"2024-03-31T15:04:41.935618+0000","last_undegraded":"2024-03-31T15:04:41.935618+0000","last_fullsized":"2024-03-31T15:04:41.935618+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:04:38.358174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:04:38.358174+0000","last_clean_scrub_stamp":"2024-03-31T15:04:38.358174+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T15:59:53.613841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-31T15:04:41.394360+0000","last_change":"2024-03-31T15:04:40.398203+0000","last_active":"2024-03-31T15:04:41.394360+0000","last_peered":"2024-03-31T15:04:41.394360+0000","last_clean":"2024-03-31T15:04:41.394360+0000","last_became_active":"2024-03-31T15:04:40.397874+0000","last_became_peered":"2024-03-31T15:04:40.397874+0000","last_unstale":"2024-03-31T15:04:41.394360+0000","last_undegraded":"2024-03-31T15:04:41.394360+0000","last_fullsized":"2024-03-31T15:04:41.394360+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_clean_scrub_stamp":"2024-03-31T15:04:39.365755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T16:13:45.059927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"33'2","reported_seq":16,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-31T15:04:42.250566+0000","last_change":"2024-03-31T15:04:39.400286+0000","last_active":"2024-03-31T15:04:42.250566+0000","last_peered":"2024-03-31T15:04:42.250566+0000","last_clean":"2024-03-31T15:04:42.250566+0000","last_became_active":"2024-03-31T15:04:39.400010+0000","last_became_peered":"2024-03-31T15:04:39.400010+0000","last_unstale":"2024-03-31T15:04:42.250566+0000","last_undegraded":"2024-03-31T15:04:42.250566+0000","last_fullsized":"2024-03-31T15:04:42.250566+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:04:38.358174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:04:38.358174+0000","last_clean_scrub_stamp":"2024-03-31T15:04:38.358174+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:30:38.353581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-31T15:04:41.431420+0000","last_change":"2024-03-31T15:04:40.393492+0000","last_active":"2024-03-31T15:04:41.431420+0000","last_peered":"2024-03-31T15:04:41.431420+0000","last_clean":"2024-03-31T15:04:41.431420+0000","last_became_active":"2024-03-31T15:04:40.393167+0000","last_became_peered":"2024-03-31T15:04:40.393167+0000","last_unstale":"2024-03-31T15:04:41.431420+0000","last_undegraded":"2024-03-31T15:04:41.431420+0000","last_fullsized":"2024-03-31T15:04:41.431420+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_clean_scrub_stamp":"2024-03-31T15:04:39.365755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T17:44:28.591558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1c","version":"33'2","reported_seq":16,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-31T15:04:41.392630+0000","last_change":"2024-03-31T15:04:39.390046+0000","last_active":"2024-03-31T15:04:41.392630+0000","last_peered":"2024-03-31T15:04:41.392630+0000","last_clean":"2024-03-31T15:04:41.392630+0000","last_became_active":"2024-03-31T15:04:39.389733+0000","last_became_peered":"2024-03-31T15:04:39.389733+0000","last_unstale":"2024-03-31T15:04:41.392630+0000","last_undegraded":"2024-03-31T15:04:41.392630+0000","last_fullsized":"2024-03-31T15:04:41.392630+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:04:38.358174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:04:38.358174+0000","last_clean_scrub_stamp":"2024-03-31T15:04:38.358174+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:23:31.396914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-31T15:04:42.252220+0000","last_change":"2024-03-31T15:04:40.407895+0000","last_active":"2024-03-31T15:04:42.252220+0000","last_peered":"2024-03-31T15:04:42.252220+0000","last_clean":"2024-03-31T15:04:42.252220+0000","last_became_active":"2024-03-31T15:04:40.407606+0000","last_became_peered":"2024-03-31T15:04:40.407606+0000","last_unstale":"2024-03-31T15:04:42.252220+0000","last_undegraded":"2024-03-31T15:04:42.252220+0000","last_fullsized":"2024-03-31T15:04:42.252220+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_clean_scrub_stamp":"2024-03-31T15:04:39.365755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:35:41.708008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"33'1","reported_seq":15,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-31T15:04:41.394007+0000","last_change":"2024-03-31T15:04:39.397526+0000","last_active":"2024-03-31T15:04:41.394007+0000","last_peered":"2024-03-31T15:04:41.394007+0000","last_clean":"2024-03-31T15:04:41.394007+0000","last_became_active":"2024-03-31T15:04:39.397183+0000","last_became_peered":"2024-03-31T15:04:39.397183+0000","last_unstale":"2024-03-31T15:04:41.394007+0000","last_undegraded":"2024-03-31T15:04:41.394007+0000","last_fullsized":"2024-03-31T15:04:41.394007+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:04:38.358174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:04:38.358174+0000","last_clean_scrub_stamp":"2024-03-31T15:04:38.358174+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:15:13.414813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-31T15:04:42.252304+0000","last_change":"2024-03-31T15:04:40.418172+0000","last_active":"2024-03-31T15:04:42.252304+0000","last_peered":"2024-03-31T15:04:42.252304+0000","last_clean":"2024-03-31T15:04:42.252304+0000","last_became_active":"2024-03-31T15:04:40.417767+0000","last_became_peered":"2024-03-31T15:04:40.417767+0000","last_unstale":"2024-03-31T15:04:42.252304+0000","last_undegraded":"2024-03-31T15:04:42.252304+0000","last_fullsized":"2024-03-31T15:04:42.252304+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_clean_scrub_stamp":"2024-03-31T15:04:39.365755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T15:31:24.450431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"33'1","reported_seq":15,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-31T15:04:41.394996+0000","last_change":"2024-03-31T15:04:39.396071+0000","last_active":"2024-03-31T15:04:41.394996+0000","last_peered":"2024-03-31T15:04:41.394996+0000","last_clean":"2024-03-31T15:04:41.394996+0000","last_became_active":"2024-03-31T15:04:39.395806+0000","last_became_peered":"2024-03-31T15:04:39.395806+0000","last_unstale":"2024-03-31T15:04:41.394996+0000","last_undegraded":"2024-03-31T15:04:41.394996+0000","last_fullsized":"2024-03-31T15:04:41.394996+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:04:38.358174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:04:38.358174+0000","last_clean_scrub_stamp":"2024-03-31T15:04:38.358174+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:19:30.634710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-31T15:04:41.394619+0000","last_change":"2024-03-31T15:04:40.401790+0000","last_active":"2024-03-31T15:04:41.394619+0000","last_peered":"2024-03-31T15:04:41.394619+0000","last_clean":"2024-03-31T15:04:41.394619+0000","last_became_active":"2024-03-31T15:04:40.397471+0000","last_became_peered":"2024-03-31T15:04:40.397471+0000","last_unstale":"2024-03-31T15:04:41.394619+0000","last_undegraded":"2024-03-31T15:04:41.394619+0000","last_fullsized":"2024-03-31T15:04:41.394619+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_clean_scrub_stamp":"2024-03-31T15:04:39.365755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:35:17.024260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-31T15:04:42.250107+0000","last_change":"2024-03-31T15:04:39.390920+0000","last_active":"2024-03-31T15:04:42.250107+0000","last_peered":"2024-03-31T15:04:42.250107+0000","last_clean":"2024-03-31T15:04:42.250107+0000","last_became_active":"2024-03-31T15:04:39.390649+0000","last_became_peered":"2024-03-31T15:04:39.390649+0000","last_unstale":"2024-03-31T15:04:42.250107+0000","last_undegraded":"2024-03-31T15:04:42.250107+0000","last_fullsized":"2024-03-31T15:04:42.250107+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:04:38.358174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:04:38.358174+0000","last_clean_scrub_stamp":"2024-03-31T15:04:38.358174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:16:49.211449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-31T15:04:41.430728+0000","last_change":"2024-03-31T15:04:40.405230+0000","last_active":"2024-03-31T15:04:41.430728+0000","last_peered":"2024-03-31T15:04:41.430728+0000","last_clean":"2024-03-31T15:04:41.430728+0000","last_became_active":"2024-03-31T15:04:40.404444+0000","last_became_peered":"2024-03-31T15:04:40.404444+0000","last_unstale":"2024-03-31T15:04:41.430728+0000","last_undegraded":"2024-03-31T15:04:41.430728+0000","last_fullsized":"2024-03-31T15:04:41.430728+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_clean_scrub_stamp":"2024-03-31T15:04:39.365755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:03:40.236898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-31T15:04:41.393594+0000","last_change":"2024-03-31T15:04:39.389091+0000","last_active":"2024-03-31T15:04:41.393594+0000","last_peered":"2024-03-31T15:04:41.393594+0000","last_clean":"2024-03-31T15:04:41.393594+0000","last_became_active":"2024-03-31T15:04:39.388705+0000","last_became_peered":"2024-03-31T15:04:39.388705+0000","last_unstale":"2024-03-31T15:04:41.393594+0000","last_undegraded":"2024-03-31T15:04:41.393594+0000","last_fullsized":"2024-03-31T15:04:41.393594+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:04:38.358174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:04:38.358174+0000","last_clean_scrub_stamp":"2024-03-31T15:04:38.358174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:37:19.999641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-31T15:04:42.252453+0000","last_change":"2024-03-31T15:04:40.402972+0000","last_active":"2024-03-31T15:04:42.252453+0000","last_peered":"2024-03-31T15:04:42.252453+0000","last_clean":"2024-03-31T15:04:42.252453+0000","last_became_active":"2024-03-31T15:04:40.402465+0000","last_became_peered":"2024-03-31T15:04:40.402465+0000","last_unstale":"2024-03-31T15:04:42.252453+0000","last_undegraded":"2024-03-31T15:04:42.252453+0000","last_fullsized":"2024-03-31T15:04:42.252453+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_clean_scrub_stamp":"2024-03-31T15:04:39.365755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:35:41.708008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-31T15:04:42.250739+0000","last_change":"2024-03-31T15:04:39.400525+0000","last_active":"2024-03-31T15:04:42.250739+0000","last_peered":"2024-03-31T15:04:42.250739+0000","last_clean":"2024-03-31T15:04:42.250739+0000","last_became_active":"2024-03-31T15:04:39.398149+0000","last_became_peered":"2024-03-31T15:04:39.398149+0000","last_unstale":"2024-03-31T15:04:42.250739+0000","last_undegraded":"2024-03-31T15:04:42.250739+0000","last_fullsized":"2024-03-31T15:04:42.250739+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:04:38.358174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:04:38.358174+0000","last_clean_scrub_stamp":"2024-03-31T15:04:38.358174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T15:38:38.796454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-31T15:04:42.250687+0000","last_change":"2024-03-31T15:04:40.419569+0000","last_active":"2024-03-31T15:04:42.250687+0000","last_peered":"2024-03-31T15:04:42.250687+0000","last_clean":"2024-03-31T15:04:42.250687+0000","last_became_active":"2024-03-31T15:04:40.419301+0000","last_became_peered":"2024-03-31T15:04:40.419301+0000","last_unstale":"2024-03-31T15:04:42.250687+0000","last_undegraded":"2024-03-31T15:04:42.250687+0000","last_fullsized":"2024-03-31T15:04:42.250687+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_clean_scrub_stamp":"2024-03-31T15:04:39.365755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T15:10:10.481319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-31T15:04:41.935671+0000","last_change":"2024-03-31T15:04:39.401240+0000","last_active":"2024-03-31T15:04:41.935671+0000","last_peered":"2024-03-31T15:04:41.935671+0000","last_clean":"2024-03-31T15:04:41.935671+0000","last_became_active":"2024-03-31T15:04:39.400568+0000","last_became_peered":"2024-03-31T15:04:39.400568+0000","last_unstale":"2024-03-31T15:04:41.935671+0000","last_undegraded":"2024-03-31T15:04:41.935671+0000","last_fullsized":"2024-03-31T15:04:41.935671+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:04:38.358174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:04:38.358174+0000","last_clean_scrub_stamp":"2024-03-31T15:04:38.358174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:06:50.586294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-31T15:04:42.250525+0000","last_change":"2024-03-31T15:04:40.409079+0000","last_active":"2024-03-31T15:04:42.250525+0000","last_peered":"2024-03-31T15:04:42.250525+0000","last_clean":"2024-03-31T15:04:42.250525+0000","last_became_active":"2024-03-31T15:04:40.408851+0000","last_became_peered":"2024-03-31T15:04:40.408851+0000","last_unstale":"2024-03-31T15:04:42.250525+0000","last_undegraded":"2024-03-31T15:04:42.250525+0000","last_fullsized":"2024-03-31T15:04:42.250525+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_clean_scrub_stamp":"2024-03-31T15:04:39.365755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:45:33.596535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.15","version":"33'3","reported_seq":17,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-31T15:04:42.252053+0000","last_change":"2024-03-31T15:04:39.397172+0000","last_active":"2024-03-31T15:04:42.252053+0000","last_peered":"2024-03-31T15:04:42.252053+0000","last_clean":"2024-03-31T15:04:42.252053+0000","last_became_active":"2024-03-31T15:04:39.396861+0000","last_became_peered":"2024-03-31T15:04:39.396861+0000","last_unstale":"2024-03-31T15:04:42.252053+0000","last_undegraded":"2024-03-31T15:04:42.252053+0000","last_fullsized":"2024-03-31T15:04:42.252053+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:04:38.358174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:04:38.358174+0000","last_clean_scrub_stamp":"2024-03-31T15:04:38.358174+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T16:09:59.585006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-31T15:04:42.251833+0000","last_change":"2024-03-31T15:04:40.397670+0000","last_active":"2024-03-31T15:04:42.251833+0000","last_peered":"2024-03-31T15:04:42.251833+0000","last_clean":"2024-03-31T15:04:42.251833+0000","last_became_active":"2024-03-31T15:04:40.397316+0000","last_became_peered":"2024-03-31T15:04:40.397316+0000","last_unstale":"2024-03-31T15:04:42.251833+0000","last_undegraded":"2024-03-31T15:04:42.251833+0000","last_fullsized":"2024-03-31T15:04:42.251833+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_clean_scrub_stamp":"2024-03-31T15:04:39.365755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:49:21.664326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"33'6","reported_seq":20,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-31T15:04:42.250594+0000","last_change":"2024-03-31T15:04:39.400256+0000","last_active":"2024-03-31T15:04:42.250594+0000","last_peered":"2024-03-31T15:04:42.250594+0000","last_clean":"2024-03-31T15:04:42.250594+0000","last_became_active":"2024-03-31T15:04:39.399988+0000","last_became_peered":"2024-03-31T15:04:39.399988+0000","last_unstale":"2024-03-31T15:04:42.250594+0000","last_undegraded":"2024-03-31T15:04:42.250594+0000","last_fullsized":"2024-03-31T15:04:42.250594+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:04:38.358174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:04:38.358174+0000","last_clean_scrub_stamp":"2024-03-31T15:04:38.358174+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T17:44:27.583977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-31T15:04:41.430883+0000","last_change":"2024-03-31T15:04:40.405675+0000","last_active":"2024-03-31T15:04:41.430883+0000","last_peered":"2024-03-31T15:04:41.430883+0000","last_clean":"2024-03-31T15:04:41.430883+0000","last_became_active":"2024-03-31T15:04:40.405392+0000","last_became_peered":"2024-03-31T15:04:40.405392+0000","last_unstale":"2024-03-31T15:04:41.430883+0000","last_undegraded":"2024-03-31T15:04:41.430883+0000","last_fullsized":"2024-03-31T15:04:41.430883+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_clean_scrub_stamp":"2024-03-31T15:04:39.365755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T16:36:02.651807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"33'2","reported_seq":16,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-31T15:04:41.394075+0000","last_change":"2024-03-31T15:04:39.387853+0000","last_active":"2024-03-31T15:04:41.394075+0000","last_peered":"2024-03-31T15:04:41.394075+0000","last_clean":"2024-03-31T15:04:41.394075+0000","last_became_active":"2024-03-31T15:04:39.386509+0000","last_became_peered":"2024-03-31T15:04:39.386509+0000","last_unstale":"2024-03-31T15:04:41.394075+0000","last_undegraded":"2024-03-31T15:04:41.394075+0000","last_fullsized":"2024-03-31T15:04:41.394075+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:04:38.358174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:04:38.358174+0000","last_clean_scrub_stamp":"2024-03-31T15:04:38.358174+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:35:46.096765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-31T15:04:42.250395+0000","last_change":"2024-03-31T15:04:40.416367+0000","last_active":"2024-03-31T15:04:42.250395+0000","last_peered":"2024-03-31T15:04:42.250395+0000","last_clean":"2024-03-31T15:04:42.250395+0000","last_became_active":"2024-03-31T15:04:40.416085+0000","last_became_peered":"2024-03-31T15:04:40.416085+0000","last_unstale":"2024-03-31T15:04:42.250395+0000","last_undegraded":"2024-03-31T15:04:42.250395+0000","last_fullsized":"2024-03-31T15:04:42.250395+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_clean_scrub_stamp":"2024-03-31T15:04:39.365755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T16:36:48.309073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-31T15:04:41.430817+0000","last_change":"2024-03-31T15:04:39.393271+0000","last_active":"2024-03-31T15:04:41.430817+0000","last_peered":"2024-03-31T15:04:41.430817+0000","last_clean":"2024-03-31T15:04:41.430817+0000","last_became_active":"2024-03-31T15:04:39.392620+0000","last_became_peered":"2024-03-31T15:04:39.392620+0000","last_unstale":"2024-03-31T15:04:41.430817+0000","last_undegraded":"2024-03-31T15:04:41.430817+0000","last_fullsized":"2024-03-31T15:04:41.430817+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:04:38.358174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:04:38.358174+0000","last_clean_scrub_stamp":"2024-03-31T15:04:38.358174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T16:57:41.159408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-31T15:04:42.251784+0000","last_change":"2024-03-31T15:04:40.409056+0000","last_active":"2024-03-31T15:04:42.251784+0000","last_peered":"2024-03-31T15:04:42.251784+0000","last_clean":"2024-03-31T15:04:42.251784+0000","last_became_active":"2024-03-31T15:04:40.408835+0000","last_became_peered":"2024-03-31T15:04:40.408835+0000","last_unstale":"2024-03-31T15:04:42.251784+0000","last_undegraded":"2024-03-31T15:04:42.251784+0000","last_fullsized":"2024-03-31T15:04:42.251784+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_clean_scrub_stamp":"2024-03-31T15:04:39.365755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:20:49.393792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-31T15:04:42.250178+0000","last_change":"2024-03-31T15:04:39.398126+0000","last_active":"2024-03-31T15:04:42.250178+0000","last_peered":"2024-03-31T15:04:42.250178+0000","last_clean":"2024-03-31T15:04:42.250178+0000","last_became_active":"2024-03-31T15:04:39.397858+0000","last_became_peered":"2024-03-31T15:04:39.397858+0000","last_unstale":"2024-03-31T15:04:42.250178+0000","last_undegraded":"2024-03-31T15:04:42.250178+0000","last_fullsized":"2024-03-31T15:04:42.250178+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:04:38.358174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:04:38.358174+0000","last_clean_scrub_stamp":"2024-03-31T15:04:38.358174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:24:34.431351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-31T15:04:41.393951+0000","last_change":"2024-03-31T15:04:40.395692+0000","last_active":"2024-03-31T15:04:41.393951+0000","last_peered":"2024-03-31T15:04:41.393951+0000","last_clean":"2024-03-31T15:04:41.393951+0000","last_became_active":"2024-03-31T15:04:40.395406+0000","last_became_peered":"2024-03-31T15:04:40.395406+0000","last_unstale":"2024-03-31T15:04:41.393951+0000","last_undegraded":"2024-03-31T15:04:41.393951+0000","last_fullsized":"2024-03-31T15:04:41.393951+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_clean_scrub_stamp":"2024-03-31T15:04:39.365755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T17:29:16.250559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"33'8","reported_seq":22,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-31T15:04:42.250154+0000","last_change":"2024-03-31T15:04:39.400309+0000","last_active":"2024-03-31T15:04:42.250154+0000","last_peered":"2024-03-31T15:04:42.250154+0000","last_clean":"2024-03-31T15:04:42.250154+0000","last_became_active":"2024-03-31T15:04:39.400045+0000","last_became_peered":"2024-03-31T15:04:39.400045+0000","last_unstale":"2024-03-31T15:04:42.250154+0000","last_undegraded":"2024-03-31T15:04:42.250154+0000","last_fullsized":"2024-03-31T15:04:42.250154+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:04:38.358174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:04:38.358174+0000","last_clean_scrub_stamp":"2024-03-31T15:04:38.358174+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:45:37.237921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-31T15:04:41.394236+0000","last_change":"2024-03-31T15:04:40.391111+0000","last_active":"2024-03-31T15:04:41.394236+0000","last_peered":"2024-03-31T15:04:41.394236+0000","last_clean":"2024-03-31T15:04:41.394236+0000","last_became_active":"2024-03-31T15:04:40.390064+0000","last_became_peered":"2024-03-31T15:04:40.390064+0000","last_unstale":"2024-03-31T15:04:41.394236+0000","last_undegraded":"2024-03-31T15:04:41.394236+0000","last_fullsized":"2024-03-31T15:04:41.394236+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_clean_scrub_stamp":"2024-03-31T15:04:39.365755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T16:04:31.890768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"33'1","reported_seq":15,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-31T15:04:41.431581+0000","last_change":"2024-03-31T15:04:39.400037+0000","last_active":"2024-03-31T15:04:41.431581+0000","last_peered":"2024-03-31T15:04:41.431581+0000","last_clean":"2024-03-31T15:04:41.431581+0000","last_became_active":"2024-03-31T15:04:39.399782+0000","last_became_peered":"2024-03-31T15:04:39.399782+0000","last_unstale":"2024-03-31T15:04:41.431581+0000","last_undegraded":"2024-03-31T15:04:41.431581+0000","last_fullsized":"2024-03-31T15:04:41.431581+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:04:38.358174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:04:38.358174+0000","last_clean_scrub_stamp":"2024-03-31T15:04:38.358174+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:06:50.586294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-31T15:04:42.252583+0000","last_change":"2024-03-31T15:04:40.392938+0000","last_active":"2024-03-31T15:04:42.252583+0000","last_peered":"2024-03-31T15:04:42.252583+0000","last_clean":"2024-03-31T15:04:42.252583+0000","last_became_active":"2024-03-31T15:04:40.392352+0000","last_became_peered":"2024-03-31T15:04:40.392352+0000","last_unstale":"2024-03-31T15:04:42.252583+0000","last_undegraded":"2024-03-31T15:04:42.252583+0000","last_fullsized":"2024-03-31T15:04:42.252583+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_clean_scrub_stamp":"2024-03-31T15:04:39.365755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:04:06.091897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"33'2","reported_seq":16,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-31T15:04:41.394273+0000","last_change":"2024-03-31T15:04:39.395287+0000","last_active":"2024-03-31T15:04:41.394273+0000","last_peered":"2024-03-31T15:04:41.394273+0000","last_clean":"2024-03-31T15:04:41.394273+0000","last_became_active":"2024-03-31T15:04:39.394961+0000","last_became_peered":"2024-03-31T15:04:39.394961+0000","last_unstale":"2024-03-31T15:04:41.394273+0000","last_undegraded":"2024-03-31T15:04:41.394273+0000","last_fullsized":"2024-03-31T15:04:41.394273+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:04:38.358174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:04:38.358174+0000","last_clean_scrub_stamp":"2024-03-31T15:04:38.358174+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:16:57.187197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-31T15:04:41.431013+0000","last_change":"2024-03-31T15:04:40.411675+0000","last_active":"2024-03-31T15:04:41.431013+0000","last_peered":"2024-03-31T15:04:41.431013+0000","last_clean":"2024-03-31T15:04:41.431013+0000","last_became_active":"2024-03-31T15:04:40.410985+0000","last_became_peered":"2024-03-31T15:04:40.410985+0000","last_unstale":"2024-03-31T15:04:41.431013+0000","last_undegraded":"2024-03-31T15:04:41.431013+0000","last_fullsized":"2024-03-31T15:04:41.431013+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_clean_scrub_stamp":"2024-03-31T15:04:39.365755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:45:38.245502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"33'1","reported_seq":15,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-31T15:04:41.394258+0000","last_change":"2024-03-31T15:04:39.392904+0000","last_active":"2024-03-31T15:04:41.394258+0000","last_peered":"2024-03-31T15:04:41.394258+0000","last_clean":"2024-03-31T15:04:41.394258+0000","last_became_active":"2024-03-31T15:04:39.392598+0000","last_became_peered":"2024-03-31T15:04:39.392598+0000","last_unstale":"2024-03-31T15:04:41.394258+0000","last_undegraded":"2024-03-31T15:04:41.394258+0000","last_fullsized":"2024-03-31T15:04:41.394258+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:04:38.358174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:04:38.358174+0000","last_clean_scrub_stamp":"2024-03-31T15:04:38.358174+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:22:50.831391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-31T15:04:41.392792+0000","last_change":"2024-03-31T15:04:40.401126+0000","last_active":"2024-03-31T15:04:41.392792+0000","last_peered":"2024-03-31T15:04:41.392792+0000","last_clean":"2024-03-31T15:04:41.392792+0000","last_became_active":"2024-03-31T15:04:40.400772+0000","last_became_peered":"2024-03-31T15:04:40.400772+0000","last_unstale":"2024-03-31T15:04:41.392792+0000","last_undegraded":"2024-03-31T15:04:41.392792+0000","last_fullsized":"2024-03-31T15:04:41.392792+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_clean_scrub_stamp":"2024-03-31T15:04:39.365755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:42:11.050371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-31T15:04:41.394651+0000","last_change":"2024-03-31T15:04:39.398381+0000","last_active":"2024-03-31T15:04:41.394651+0000","last_peered":"2024-03-31T15:04:41.394651+0000","last_clean":"2024-03-31T15:04:41.394651+0000","last_became_active":"2024-03-31T15:04:39.398040+0000","last_became_peered":"2024-03-31T15:04:39.398040+0000","last_unstale":"2024-03-31T15:04:41.394651+0000","last_undegraded":"2024-03-31T15:04:41.394651+0000","last_fullsized":"2024-03-31T15:04:41.394651+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:04:38.358174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:04:38.358174+0000","last_clean_scrub_stamp":"2024-03-31T15:04:38.358174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T19:25:10.251365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-31T15:04:41.392612+0000","last_change":"2024-03-31T15:04:40.394119+0000","last_active":"2024-03-31T15:04:41.392612+0000","last_peered":"2024-03-31T15:04:41.392612+0000","last_clean":"2024-03-31T15:04:41.392612+0000","last_became_active":"2024-03-31T15:04:40.393767+0000","last_became_peered":"2024-03-31T15:04:40.393767+0000","last_unstale":"2024-03-31T15:04:41.392612+0000","last_undegraded":"2024-03-31T15:04:41.392612+0000","last_fullsized":"2024-03-31T15:04:41.392612+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_clean_scrub_stamp":"2024-03-31T15:04:39.365755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:27:03.826509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"33'2","reported_seq":16,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-31T15:04:41.935536+0000","last_change":"2024-03-31T15:04:39.391434+0000","last_active":"2024-03-31T15:04:41.935536+0000","last_peered":"2024-03-31T15:04:41.935536+0000","last_clean":"2024-03-31T15:04:41.935536+0000","last_became_active":"2024-03-31T15:04:39.390938+0000","last_became_peered":"2024-03-31T15:04:39.390938+0000","last_unstale":"2024-03-31T15:04:41.935536+0000","last_undegraded":"2024-03-31T15:04:41.935536+0000","last_fullsized":"2024-03-31T15:04:41.935536+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:04:38.358174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:04:38.358174+0000","last_clean_scrub_stamp":"2024-03-31T15:04:38.358174+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T16:57:41.159408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-31T15:04:41.392596+0000","last_change":"2024-03-31T15:04:40.419780+0000","last_active":"2024-03-31T15:04:41.392596+0000","last_peered":"2024-03-31T15:04:41.392596+0000","last_clean":"2024-03-31T15:04:41.392596+0000","last_became_active":"2024-03-31T15:04:40.419452+0000","last_became_peered":"2024-03-31T15:04:40.419452+0000","last_unstale":"2024-03-31T15:04:41.392596+0000","last_undegraded":"2024-03-31T15:04:41.392596+0000","last_fullsized":"2024-03-31T15:04:41.392596+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_clean_scrub_stamp":"2024-03-31T15:04:39.365755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:19:31.642290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-31T15:04:41.394063+0000","last_change":"2024-03-31T15:04:39.393910+0000","last_active":"2024-03-31T15:04:41.394063+0000","last_peered":"2024-03-31T15:04:41.394063+0000","last_clean":"2024-03-31T15:04:41.394063+0000","last_became_active":"2024-03-31T15:04:39.393633+0000","last_became_peered":"2024-03-31T15:04:39.393633+0000","last_unstale":"2024-03-31T15:04:41.394063+0000","last_undegraded":"2024-03-31T15:04:41.394063+0000","last_fullsized":"2024-03-31T15:04:41.394063+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:04:38.358174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:04:38.358174+0000","last_clean_scrub_stamp":"2024-03-31T15:04:38.358174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T16:57:41.159408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-31T15:04:41.392247+0000","last_change":"2024-03-31T15:04:40.400735+0000","last_active":"2024-03-31T15:04:41.392247+0000","last_peered":"2024-03-31T15:04:41.392247+0000","last_clean":"2024-03-31T15:04:41.392247+0000","last_became_active":"2024-03-31T15:04:40.400363+0000","last_became_peered":"2024-03-31T15:04:40.400363+0000","last_unstale":"2024-03-31T15:04:41.392247+0000","last_undegraded":"2024-03-31T15:04:41.392247+0000","last_fullsized":"2024-03-31T15:04:41.392247+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_clean_scrub_stamp":"2024-03-31T15:04:39.365755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:15:42.398917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-31T15:04:41.393924+0000","last_change":"2024-03-31T15:04:39.389197+0000","last_active":"2024-03-31T15:04:41.393924+0000","last_peered":"2024-03-31T15:04:41.393924+0000","last_clean":"2024-03-31T15:04:41.393924+0000","last_became_active":"2024-03-31T15:04:39.388443+0000","last_became_peered":"2024-03-31T15:04:39.388443+0000","last_unstale":"2024-03-31T15:04:41.393924+0000","last_undegraded":"2024-03-31T15:04:41.393924+0000","last_fullsized":"2024-03-31T15:04:41.393924+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:04:38.358174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:04:38.358174+0000","last_clean_scrub_stamp":"2024-03-31T15:04:38.358174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:19:57.689140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-31T15:04:42.251616+0000","last_change":"2024-03-31T15:04:40.404171+0000","last_active":"2024-03-31T15:04:42.251616+0000","last_peered":"2024-03-31T15:04:42.251616+0000","last_clean":"2024-03-31T15:04:42.251616+0000","last_became_active":"2024-03-31T15:04:40.403873+0000","last_became_peered":"2024-03-31T15:04:40.403873+0000","last_unstale":"2024-03-31T15:04:42.251616+0000","last_undegraded":"2024-03-31T15:04:42.251616+0000","last_fullsized":"2024-03-31T15:04:42.251616+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_clean_scrub_stamp":"2024-03-31T15:04:39.365755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:12:19.845843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-31T15:04:42.251264+0000","last_change":"2024-03-31T15:04:39.395958+0000","last_active":"2024-03-31T15:04:42.251264+0000","last_peered":"2024-03-31T15:04:42.251264+0000","last_clean":"2024-03-31T15:04:42.251264+0000","last_became_active":"2024-03-31T15:04:39.395076+0000","last_became_peered":"2024-03-31T15:04:39.395076+0000","last_unstale":"2024-03-31T15:04:42.251264+0000","last_undegraded":"2024-03-31T15:04:42.251264+0000","last_fullsized":"2024-03-31T15:04:42.251264+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:04:38.358174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:04:38.358174+0000","last_clean_scrub_stamp":"2024-03-31T15:04:38.358174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:53:17.213942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-31T15:04:41.392795+0000","last_change":"2024-03-31T15:04:40.391143+0000","last_active":"2024-03-31T15:04:41.392795+0000","last_peered":"2024-03-31T15:04:41.392795+0000","last_clean":"2024-03-31T15:04:41.392795+0000","last_became_active":"2024-03-31T15:04:40.390868+0000","last_became_peered":"2024-03-31T15:04:40.390868+0000","last_unstale":"2024-03-31T15:04:41.392795+0000","last_undegraded":"2024-03-31T15:04:41.392795+0000","last_fullsized":"2024-03-31T15:04:41.392795+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_clean_scrub_stamp":"2024-03-31T15:04:39.365755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:34:34.155504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"33'2","reported_seq":16,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-31T15:04:41.431551+0000","last_change":"2024-03-31T15:04:39.397022+0000","last_active":"2024-03-31T15:04:41.431551+0000","last_peered":"2024-03-31T15:04:41.431551+0000","last_clean":"2024-03-31T15:04:41.431551+0000","last_became_active":"2024-03-31T15:04:39.394282+0000","last_became_peered":"2024-03-31T15:04:39.394282+0000","last_unstale":"2024-03-31T15:04:41.431551+0000","last_undegraded":"2024-03-31T15:04:41.431551+0000","last_fullsized":"2024-03-31T15:04:41.431551+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:04:38.358174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:04:38.358174+0000","last_clean_scrub_stamp":"2024-03-31T15:04:38.358174+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T16:11:46.894965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-31T15:04:42.252564+0000","last_change":"2024-03-31T15:04:40.395877+0000","last_active":"2024-03-31T15:04:42.252564+0000","last_peered":"2024-03-31T15:04:42.252564+0000","last_clean":"2024-03-31T15:04:42.252564+0000","last_became_active":"2024-03-31T15:04:40.395387+0000","last_became_peered":"2024-03-31T15:04:40.395387+0000","last_unstale":"2024-03-31T15:04:42.252564+0000","last_undegraded":"2024-03-31T15:04:42.252564+0000","last_fullsized":"2024-03-31T15:04:42.252564+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_clean_scrub_stamp":"2024-03-31T15:04:39.365755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:41:01.296109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"33'1","reported_seq":15,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-31T15:04:41.935733+0000","last_change":"2024-03-31T15:04:39.394794+0000","last_active":"2024-03-31T15:04:41.935733+0000","last_peered":"2024-03-31T15:04:41.935733+0000","last_clean":"2024-03-31T15:04:41.935733+0000","last_became_active":"2024-03-31T15:04:39.394150+0000","last_became_peered":"2024-03-31T15:04:39.394150+0000","last_unstale":"2024-03-31T15:04:41.935733+0000","last_undegraded":"2024-03-31T15:04:41.935733+0000","last_fullsized":"2024-03-31T15:04:41.935733+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:04:38.358174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:04:38.358174+0000","last_clean_scrub_stamp":"2024-03-31T15:04:38.358174+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T16:11:46.894965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-31T15:04:41.935569+0000","last_change":"2024-03-31T15:04:40.419946+0000","last_active":"2024-03-31T15:04:41.935569+0000","last_peered":"2024-03-31T15:04:41.935569+0000","last_clean":"2024-03-31T15:04:41.935569+0000","last_became_active":"2024-03-31T15:04:40.418512+0000","last_became_peered":"2024-03-31T15:04:40.418512+0000","last_unstale":"2024-03-31T15:04:41.935569+0000","last_undegraded":"2024-03-31T15:04:41.935569+0000","last_fullsized":"2024-03-31T15:04:41.935569+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_clean_scrub_stamp":"2024-03-31T15:04:39.365755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T15:42:06.044034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-31T15:04:41.431060+0000","last_change":"2024-03-31T15:04:39.393852+0000","last_active":"2024-03-31T15:04:41.431060+0000","last_peered":"2024-03-31T15:04:41.431060+0000","last_clean":"2024-03-31T15:04:41.431060+0000","last_became_active":"2024-03-31T15:04:39.393010+0000","last_became_peered":"2024-03-31T15:04:39.393010+0000","last_unstale":"2024-03-31T15:04:41.431060+0000","last_undegraded":"2024-03-31T15:04:41.431060+0000","last_fullsized":"2024-03-31T15:04:41.431060+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:04:38.358174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:04:38.358174+0000","last_clean_scrub_stamp":"2024-03-31T15:04:38.358174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T15:59:53.613841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-31T15:04:41.393955+0000","last_change":"2024-03-31T15:04:40.406445+0000","last_active":"2024-03-31T15:04:41.393955+0000","last_peered":"2024-03-31T15:04:41.393955+0000","last_clean":"2024-03-31T15:04:41.393955+0000","last_became_active":"2024-03-31T15:04:40.406179+0000","last_became_peered":"2024-03-31T15:04:40.406179+0000","last_unstale":"2024-03-31T15:04:41.393955+0000","last_undegraded":"2024-03-31T15:04:41.393955+0000","last_fullsized":"2024-03-31T15:04:41.393955+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_clean_scrub_stamp":"2024-03-31T15:04:39.365755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:32:38.931716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-31T15:04:41.935657+0000","last_change":"2024-03-31T15:04:39.391185+0000","last_active":"2024-03-31T15:04:41.935657+0000","last_peered":"2024-03-31T15:04:41.935657+0000","last_clean":"2024-03-31T15:04:41.935657+0000","last_became_active":"2024-03-31T15:04:39.390600+0000","last_became_peered":"2024-03-31T15:04:39.390600+0000","last_unstale":"2024-03-31T15:04:41.935657+0000","last_undegraded":"2024-03-31T15:04:41.935657+0000","last_fullsized":"2024-03-31T15:04:41.935657+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:04:38.358174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:04:38.358174+0000","last_clean_scrub_stamp":"2024-03-31T15:04:38.358174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:00:01.104469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-31T15:04:41.935564+0000","last_change":"2024-03-31T15:04:40.415234+0000","last_active":"2024-03-31T15:04:41.935564+0000","last_peered":"2024-03-31T15:04:41.935564+0000","last_clean":"2024-03-31T15:04:41.935564+0000","last_became_active":"2024-03-31T15:04:40.414902+0000","last_became_peered":"2024-03-31T15:04:40.414902+0000","last_unstale":"2024-03-31T15:04:41.935564+0000","last_undegraded":"2024-03-31T15:04:41.935564+0000","last_fullsized":"2024-03-31T15:04:41.935564+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_clean_scrub_stamp":"2024-03-31T15:04:39.365755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T17:54:29.313023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-31T15:04:41.430901+0000","last_change":"2024-03-31T15:04:39.398000+0000","last_active":"2024-03-31T15:04:41.430901+0000","last_peered":"2024-03-31T15:04:41.430901+0000","last_clean":"2024-03-31T15:04:41.430901+0000","last_became_active":"2024-03-31T15:04:39.397453+0000","last_became_peered":"2024-03-31T15:04:39.397453+0000","last_unstale":"2024-03-31T15:04:41.430901+0000","last_undegraded":"2024-03-31T15:04:41.430901+0000","last_fullsized":"2024-03-31T15:04:41.430901+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:04:38.358174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:04:38.358174+0000","last_clean_scrub_stamp":"2024-03-31T15:04:38.358174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:00:01.104469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-31T15:04:42.250121+0000","last_change":"2024-03-31T15:04:40.414068+0000","last_active":"2024-03-31T15:04:42.250121+0000","last_peered":"2024-03-31T15:04:42.250121+0000","last_clean":"2024-03-31T15:04:42.250121+0000","last_became_active":"2024-03-31T15:04:40.413502+0000","last_became_peered":"2024-03-31T15:04:40.413502+0000","last_unstale":"2024-03-31T15:04:42.250121+0000","last_undegraded":"2024-03-31T15:04:42.250121+0000","last_fullsized":"2024-03-31T15:04:42.250121+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_clean_scrub_stamp":"2024-03-31T15:04:39.365755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:17:47.873149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-31T15:04:41.394164+0000","last_change":"2024-03-31T15:04:39.390901+0000","last_active":"2024-03-31T15:04:41.394164+0000","last_peered":"2024-03-31T15:04:41.394164+0000","last_clean":"2024-03-31T15:04:41.394164+0000","last_became_active":"2024-03-31T15:04:39.390267+0000","last_became_peered":"2024-03-31T15:04:39.390267+0000","last_unstale":"2024-03-31T15:04:41.394164+0000","last_undegraded":"2024-03-31T15:04:41.394164+0000","last_fullsized":"2024-03-31T15:04:41.394164+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:04:38.358174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:04:38.358174+0000","last_clean_scrub_stamp":"2024-03-31T15:04:38.358174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:15:41.391336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-31T15:04:41.393862+0000","last_change":"2024-03-31T15:04:40.402233+0000","last_active":"2024-03-31T15:04:41.393862+0000","last_peered":"2024-03-31T15:04:41.393862+0000","last_clean":"2024-03-31T15:04:41.393862+0000","last_became_active":"2024-03-31T15:04:40.401912+0000","last_became_peered":"2024-03-31T15:04:40.401912+0000","last_unstale":"2024-03-31T15:04:41.393862+0000","last_undegraded":"2024-03-31T15:04:41.393862+0000","last_fullsized":"2024-03-31T15:04:41.393862+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_clean_scrub_stamp":"2024-03-31T15:04:39.365755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T17:02:38.744921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-31T15:04:42.250229+0000","last_change":"2024-03-31T15:04:39.396502+0000","last_active":"2024-03-31T15:04:42.250229+0000","last_peered":"2024-03-31T15:04:42.250229+0000","last_clean":"2024-03-31T15:04:42.250229+0000","last_became_active":"2024-03-31T15:04:39.396219+0000","last_became_peered":"2024-03-31T15:04:39.396219+0000","last_unstale":"2024-03-31T15:04:42.250229+0000","last_undegraded":"2024-03-31T15:04:42.250229+0000","last_fullsized":"2024-03-31T15:04:42.250229+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:04:38.358174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:04:38.358174+0000","last_clean_scrub_stamp":"2024-03-31T15:04:38.358174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T19:01:34.356852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-31T15:04:42.252259+0000","last_change":"2024-03-31T15:04:40.400787+0000","last_active":"2024-03-31T15:04:42.252259+0000","last_peered":"2024-03-31T15:04:42.252259+0000","last_clean":"2024-03-31T15:04:42.252259+0000","last_became_active":"2024-03-31T15:04:40.400486+0000","last_became_peered":"2024-03-31T15:04:40.400486+0000","last_unstale":"2024-03-31T15:04:42.252259+0000","last_undegraded":"2024-03-31T15:04:42.252259+0000","last_fullsized":"2024-03-31T15:04:42.252259+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:04:39.365755+0000","last_clean_scrub_stamp":"2024-03-31T15:04:39.365755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:41:01.296109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":14,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-31T15:04:41.394237+0000","last_change":"2024-03-31T15:04:39.394010+0000","last_active":"2024-03-31T15:04:41.394237+0000","last_peered":"2024-03-31T15:04:41.394237+0000","last_clean":"2024-03-31T15:04:41.394237+0000","last_became_active":"2024-03-31T15:04:39.393747+0000","last_became_peered":"2024-03-31T15:04:39.393747+0000","last_unstale":"2024-03-31T15:04:41.394237+0000","last_undegraded":"2024-03-31T15:04:41.394237+0000","last_fullsized":"2024-03-31T15:04:41.394237+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:04:38.358174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:04:38.358174+0000","last_clean_scrub_stamp":"2024-03-31T15:04:38.358174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:15:15.854175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":194,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-31T15:04:42.251840+0000","last_change":"2024-03-31T15:03:24.329890+0000","last_active":"2024-03-31T15:04:42.251840+0000","last_peered":"2024-03-31T15:04:42.251840+0000","last_clean":"2024-03-31T15:04:42.251840+0000","last_became_active":"2024-03-31T15:03:24.328882+0000","last_became_peered":"2024-03-31T15:03:24.328882+0000","last_unstale":"2024-03-31T15:04:42.251840+0000","last_undegraded":"2024-03-31T15:04:42.251840+0000","last_fullsized":"2024-03-31T15:04:42.251840+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:22.885535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:22.885535+0000","last_clean_scrub_stamp":"2024-03-31T15:03:22.885535+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:18:29.004296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509461,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27972,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343868,"statfs":{"total":96636764160,"available":96608120832,"internally_reserved":0,"allocated":974848,"data_stored":739027,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509460,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27400,"kb_used_data":376,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344440,"statfs":{"total":96636764160,"available":96608706560,"internally_reserved":0,"allocated":385024,"data_stored":149241,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":4,"up_from":15,"seq":64424509459,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27400,"kb_used_data":376,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344440,"statfs":{"total":96636764160,"available":96608706560,"internally_reserved":0,"allocated":385024,"data_stored":148673,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509458,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31556,"kb_used_data":372,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340284,"statfs":{"total":96636764160,"available":96604450816,"internally_reserved":0,"allocated":380928,"data_stored":148639,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542164,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27404,"kb_used_data":376,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344436,"statfs":{"total":96636764160,"available":96608702464,"internally_reserved":0,"allocated":385024,"data_stored":149185,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542164,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31564,"kb_used_data":384,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340276,"statfs":{"total":96636764160,"available":96604442624,"internally_reserved":0,"allocated":393216,"data_stored":150203,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542164,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31572,"kb_used_data":384,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340268,"statfs":{"total":96636764160,"available":96604434432,"internally_reserved":0,"allocated":393216,"data_stored":150203,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542164,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32124,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339716,"statfs":{"total":96636764160,"available":96603869184,"internally_reserved":0,"allocated":958464,"data_stored":732637,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-31T15:04:44.897 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-31T15:04:44.897 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-03-31T15:04:45.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:45.088+0000 7f9449400640 1 Processor -- start 2024-03-31T15:04:45.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:45.088+0000 7f9449400640 1 -- start start 2024-03-31T15:04:45.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:45.088+0000 7f9449400640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f94441411b0 0x7f944406f920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:45.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:45.088+0000 7f9449400640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f9444141590 0x7f944406fe60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:45.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:45.088+0000 7f9449400640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9444144980 con 0x7f9444143c30 2024-03-31T15:04:45.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:45.088+0000 7f9449400640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f944415f910 con 0x7f94441411b0 2024-03-31T15:04:45.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:45.088+0000 7f9449400640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f9444162910 con 0x7f9444141590 2024-03-31T15:04:45.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:45.092+0000 7f9442800640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f94441411b0 0x7f944406f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:45.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:45.092+0000 7f9442800640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f94441411b0 0x7f944406f920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.16:53724/0 (socket says 172.21.15.16:53724) 2024-03-31T15:04:45.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:45.092+0000 7f9442800640 1 -- 172.21.15.16:0/765277114 learned_addr learned my addr 172.21.15.16:0/765277114 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:04:45.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:45.092+0000 7f9439e00640 1 --2- 172.21.15.16:0/765277114 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f9444141590 0x7f944406fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:45.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:45.092+0000 7f9442800640 1 -- 172.21.15.16:0/765277114 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f9444141590 msgr2=0x7f944406fe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:04:45.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:45.092+0000 7f9442800640 1 --2- 172.21.15.16:0/765277114 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f9444141590 0x7f944406fe60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:45.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:45.092+0000 7f9442800640 1 -- 172.21.15.16:0/765277114 >> v1:172.21.15.89:6789/0 conn(0x7f9444143c30 legacy=0x7f94441440a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:45.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:45.092+0000 7f9442800640 1 -- 172.21.15.16:0/765277114 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9444161fe0 con 0x7f94441411b0 2024-03-31T15:04:45.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:45.092+0000 7f9441e00640 1 -- 172.21.15.16:0/765277114 <== mon.1 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1091843465 0 0) 0x7f9444144980 con 0x7f9444143c30 2024-03-31T15:04:45.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:45.092+0000 7f9442800640 1 --2- 172.21.15.16:0/765277114 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f94441411b0 0x7f944406f920 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f9434009870 tx=0x7f943401c560 comp rx=0 tx=0).ready entity=mon.1 client_cookie=641b80ff02f775a4 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:45.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:45.092+0000 7f9441e00640 1 -- 172.21.15.16:0/765277114 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94340ab650 con 0x7f94441411b0 2024-03-31T15:04:45.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:45.092+0000 7f9441e00640 1 -- 172.21.15.16:0/765277114 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f94340b0070 con 0x7f94441411b0 2024-03-31T15:04:45.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:45.092+0000 7f9441e00640 1 -- 172.21.15.16:0/765277114 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94340aba10 con 0x7f94441411b0 2024-03-31T15:04:45.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:45.096+0000 7f9449400640 1 -- 172.21.15.16:0/765277114 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f94441411b0 msgr2=0x7f944406f920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:45.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:45.096+0000 7f9449400640 1 --2- 172.21.15.16:0/765277114 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f94441411b0 0x7f944406f920 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f9434009870 tx=0x7f943401c560 comp rx=0 tx=0).stop 2024-03-31T15:04:45.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:45.096+0000 7f9449400640 1 -- 172.21.15.16:0/765277114 shutdown_connections 2024-03-31T15:04:45.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:45.096+0000 7f9449400640 1 --2- 172.21.15.16:0/765277114 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f9444141590 0x7f944406fe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:45.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:45.096+0000 7f9449400640 1 --2- 172.21.15.16:0/765277114 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f94441411b0 0x7f944406f920 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:45.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:45.096+0000 7f9449400640 1 -- 172.21.15.16:0/765277114 >> 172.21.15.16:0/765277114 conn(0x7f9444067930 msgr2=0x7f94440b3ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:04:45.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:45.096+0000 7f9449400640 1 -- 172.21.15.16:0/765277114 shutdown_connections 2024-03-31T15:04:45.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:45.096+0000 7f9449400640 1 -- 172.21.15.16:0/765277114 wait complete. 2024-03-31T15:04:45.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:45.096+0000 7f9449400640 1 Processor -- start 2024-03-31T15:04:45.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:45.096+0000 7f9449400640 1 -- start start 2024-03-31T15:04:45.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:45.096+0000 7f9449400640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f94441411b0 0x7f9444144060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:45.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:45.096+0000 7f9449400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9444141590 0x7f944413c8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:45.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:45.096+0000 7f9449400640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f9444143c30 0x7f9444152240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:45.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:45.096+0000 7f9449400640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f944415f290 con 0x7f9444141590 2024-03-31T15:04:45.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:45.096+0000 7f9449400640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f944415f110 con 0x7f94441411b0 2024-03-31T15:04:45.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:45.096+0000 7f9449400640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f944415f410 con 0x7f9444143c30 2024-03-31T15:04:45.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:45.096+0000 7f9439e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9444141590 0x7f944413c8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:45.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:45.096+0000 7f9439e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9444141590 0x7f944413c8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54282/0 (socket says 172.21.15.16:54282) 2024-03-31T15:04:45.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:45.096+0000 7f9439e00640 1 -- 172.21.15.16:0/3334752882 learned_addr learned my addr 172.21.15.16:0/3334752882 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:04:45.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:45.096+0000 7f9443200640 1 --2- 172.21.15.16:0/3334752882 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f9444143c30 0x7f9444152240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:45.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:45.096+0000 7f9442800640 1 --2- 172.21.15.16:0/3334752882 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f94441411b0 0x7f9444144060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:45.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:45.096+0000 7f9439e00640 1 -- 172.21.15.16:0/3334752882 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f9444143c30 msgr2=0x7f9444152240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:45.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:45.096+0000 7f9439e00640 1 --2- 172.21.15.16:0/3334752882 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f9444143c30 0x7f9444152240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:45.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:45.096+0000 7f9439e00640 1 -- 172.21.15.16:0/3334752882 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f94441411b0 msgr2=0x7f9444144060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:45.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:45.096+0000 7f9439e00640 1 --2- 172.21.15.16:0/3334752882 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f94441411b0 0x7f9444144060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:45.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:45.096+0000 7f9439e00640 1 -- 172.21.15.16:0/3334752882 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9444156450 con 0x7f9444141590 2024-03-31T15:04:45.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:45.100+0000 7f9439e00640 1 --2- 172.21.15.16:0/3334752882 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9444141590 0x7f944413c8b0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f942c00efc0 tx=0x7f942c00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:45.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:45.100+0000 7f943b400640 1 -- 172.21.15.16:0/3334752882 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f942c018020 con 0x7f9444141590 2024-03-31T15:04:45.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:45.100+0000 7f943b400640 1 -- 172.21.15.16:0/3334752882 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f942c014480 con 0x7f9444141590 2024-03-31T15:04:45.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:45.100+0000 7f9449400640 1 -- 172.21.15.16:0/3334752882 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9444156730 con 0x7f9444141590 2024-03-31T15:04:45.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:45.100+0000 7f943b400640 1 -- 172.21.15.16:0/3334752882 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f942c014830 con 0x7f9444141590 2024-03-31T15:04:45.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:45.100+0000 7f9449400640 1 -- 172.21.15.16:0/3334752882 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9444156cf0 con 0x7f9444141590 2024-03-31T15:04:45.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:45.100+0000 7f9449400640 1 -- 172.21.15.16:0/3334752882 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9444069950 con 0x7f9444141590 2024-03-31T15:04:45.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:45.100+0000 7f943b400640 1 -- 172.21.15.16:0/3334752882 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f942c01b020 con 0x7f9444141590 2024-03-31T15:04:45.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:45.100+0000 7f943b400640 1 --2- 172.21.15.16:0/3334752882 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f9420068380 0x7f942006a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:45.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:45.100+0000 7f943b400640 1 -- 172.21.15.16:0/3334752882 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5709+0+0 (secure 0 0 0) 0x7f942c0876d0 con 0x7f9444141590 2024-03-31T15:04:45.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:45.104+0000 7f9442800640 1 --2- 172.21.15.16:0/3334752882 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f9420068380 0x7f942006a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:45.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:45.104+0000 7f9442800640 1 --2- 172.21.15.16:0/3334752882 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f9420068380 0x7f942006a840 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f94340af000 tx=0x7f94340a7000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:45.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:45.108+0000 7f943b400640 1 -- 172.21.15.16:0/3334752882 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f942c022030 con 0x7f9444141590 2024-03-31T15:04:45.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:45.328+0000 7f9449400640 1 -- 172.21.15.16:0/3334752882 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) -- 0x7f9444155f30 con 0x7f9444141590 2024-03-31T15:04:45.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:45.332+0000 7f943b400640 1 -- 172.21.15.16:0/3334752882 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v13) ==== 190+0+147 (secure 0 0 0) 0x7f942c010040 con 0x7f9444141590 2024-03-31T15:04:45.326 INFO:teuthology.orchestra.run.smithi016.stdout:[client.0] 2024-03-31T15:04:45.326 INFO:teuthology.orchestra.run.smithi016.stdout: key = AQAPewlmlMIDEhAAF0Ll1HcrWHmH6xDKq2Qq9A== 2024-03-31T15:04:45.326 INFO:teuthology.orchestra.run.smithi016.stdout: caps mds = "allow" 2024-03-31T15:04:45.326 INFO:teuthology.orchestra.run.smithi016.stdout: caps mon = "allow r" 2024-03-31T15:04:45.327 INFO:teuthology.orchestra.run.smithi016.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-03-31T15:04:45.327 INFO:teuthology.orchestra.run.smithi016.stderr:updated caps for client.0 2024-03-31T15:04:45.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:45.332+0000 7f9449400640 1 -- 172.21.15.16:0/3334752882 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f9420068380 msgr2=0x7f942006a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:45.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:45.332+0000 7f9449400640 1 --2- 172.21.15.16:0/3334752882 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f9420068380 0x7f942006a840 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f94340af000 tx=0x7f94340a7000 comp rx=0 tx=0).stop 2024-03-31T15:04:45.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:45.332+0000 7f9449400640 1 -- 172.21.15.16:0/3334752882 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9444141590 msgr2=0x7f944413c8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:45.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:45.332+0000 7f9449400640 1 --2- 172.21.15.16:0/3334752882 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9444141590 0x7f944413c8b0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f942c00efc0 tx=0x7f942c00c490 comp rx=0 tx=0).stop 2024-03-31T15:04:45.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:45.332+0000 7f9449400640 1 -- 172.21.15.16:0/3334752882 shutdown_connections 2024-03-31T15:04:45.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:45.332+0000 7f9449400640 1 --2- 172.21.15.16:0/3334752882 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f9420068380 0x7f942006a840 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:45.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:45.332+0000 7f9449400640 1 --2- 172.21.15.16:0/3334752882 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f9444143c30 0x7f9444152240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:45.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:45.332+0000 7f9449400640 1 --2- 172.21.15.16:0/3334752882 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9444141590 0x7f944413c8b0 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:45.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:45.332+0000 7f9449400640 1 --2- 172.21.15.16:0/3334752882 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f94441411b0 0x7f9444144060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:45.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:45.332+0000 7f9449400640 1 -- 172.21.15.16:0/3334752882 >> 172.21.15.16:0/3334752882 conn(0x7f9444067930 msgr2=0x7f9444142990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:04:45.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:45.332+0000 7f9449400640 1 -- 172.21.15.16:0/3334752882 shutdown_connections 2024-03-31T15:04:45.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:45.332+0000 7f9449400640 1 -- 172.21.15.16:0/3334752882 wait complete. 2024-03-31T15:04:45.348 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-31T15:04:45.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:45.496+0000 7f13cfe00640 1 Processor -- start 2024-03-31T15:04:45.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:45.496+0000 7f13cfe00640 1 -- start start 2024-03-31T15:04:45.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:45.496+0000 7f13cfe00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f13c806dbc0 0x7f13c815f650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:45.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:45.496+0000 7f13cfe00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f13c8070030 0x7f13c8162180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:45.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:45.496+0000 7f13cfe00640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f13c80cbe90 con 0x7f13c80cb440 2024-03-31T15:04:45.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:45.496+0000 7f13cfe00640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f13c815fb90 con 0x7f13c8070030 2024-03-31T15:04:45.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:45.496+0000 7f13cfe00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f13c815f1c0 con 0x7f13c806dbc0 2024-03-31T15:04:45.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:45.496+0000 7f13cde00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f13c8070030 0x7f13c8162180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:45.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:45.496+0000 7f13cde00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f13c8070030 0x7f13c8162180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54284/0 (socket says 172.21.15.16:54284) 2024-03-31T15:04:45.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:45.496+0000 7f13cde00640 1 -- 172.21.15.16:0/2913863362 learned_addr learned my addr 172.21.15.16:0/2913863362 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:04:45.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:45.496+0000 7f13cd400640 1 --2- 172.21.15.16:0/2913863362 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f13c806dbc0 0x7f13c815f650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:45.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:45.496+0000 7f13cde00640 1 -- 172.21.15.16:0/2913863362 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f13c806dbc0 msgr2=0x7f13c815f650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:04:45.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:45.496+0000 7f13cde00640 1 --2- 172.21.15.16:0/2913863362 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f13c806dbc0 0x7f13c815f650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:45.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:45.496+0000 7f13cde00640 1 -- 172.21.15.16:0/2913863362 >> v1:172.21.15.89:6789/0 conn(0x7f13c80cb440 legacy=0x7f13c806f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:45.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:45.496+0000 7f13cde00640 1 -- 172.21.15.16:0/2913863362 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13c80cc170 con 0x7f13c8070030 2024-03-31T15:04:45.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:45.500+0000 7f13cde00640 1 --2- 172.21.15.16:0/2913863362 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f13c8070030 0x7f13c8162180 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f13c0009e50 tx=0x7f13c001c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d165d9c025e198c7 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:45.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:45.500+0000 7f13c7e00640 1 -- 172.21.15.16:0/2913863362 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13c00a5740 con 0x7f13c8070030 2024-03-31T15:04:45.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:45.500+0000 7f13c7e00640 1 -- 172.21.15.16:0/2913863362 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f13c00a58e0 con 0x7f13c8070030 2024-03-31T15:04:45.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:45.500+0000 7f13c7e00640 1 -- 172.21.15.16:0/2913863362 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13c00a5cb0 con 0x7f13c8070030 2024-03-31T15:04:45.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:45.500+0000 7f13cfe00640 1 -- 172.21.15.16:0/2913863362 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f13c8070030 msgr2=0x7f13c8162180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:45.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:45.500+0000 7f13cfe00640 1 --2- 172.21.15.16:0/2913863362 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f13c8070030 0x7f13c8162180 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f13c0009e50 tx=0x7f13c001c970 comp rx=0 tx=0).stop 2024-03-31T15:04:45.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:45.500+0000 7f13cfe00640 1 -- 172.21.15.16:0/2913863362 shutdown_connections 2024-03-31T15:04:45.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:45.500+0000 7f13cfe00640 1 --2- 172.21.15.16:0/2913863362 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f13c8070030 0x7f13c8162180 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:45.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:45.500+0000 7f13cfe00640 1 --2- 172.21.15.16:0/2913863362 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f13c806dbc0 0x7f13c815f650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:45.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:45.500+0000 7f13cfe00640 1 -- 172.21.15.16:0/2913863362 >> 172.21.15.16:0/2913863362 conn(0x7f13c8067330 msgr2=0x7f13c8067740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:04:45.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:45.500+0000 7f13cfe00640 1 -- 172.21.15.16:0/2913863362 shutdown_connections 2024-03-31T15:04:45.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:45.500+0000 7f13cfe00640 1 -- 172.21.15.16:0/2913863362 wait complete. 2024-03-31T15:04:45.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:45.504+0000 7f13cfe00640 1 Processor -- start 2024-03-31T15:04:45.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:45.504+0000 7f13cfe00640 1 -- start start 2024-03-31T15:04:45.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:45.504+0000 7f13cfe00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f13c806dbc0 0x7f13c80b6d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:45.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:45.504+0000 7f13cfe00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f13c8070030 0x7f13c80d0cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:45.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:45.504+0000 7f13cfe00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f13c80cb440 0x7f13c80b50f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:45.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:45.504+0000 7f13cfe00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f13c806cad0 con 0x7f13c806dbc0 2024-03-31T15:04:45.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:45.504+0000 7f13cfe00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f13c806c950 con 0x7f13c80cb440 2024-03-31T15:04:45.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:45.504+0000 7f13cfe00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f13c816b520 con 0x7f13c8070030 2024-03-31T15:04:45.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:45.504+0000 7f13cd400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f13c806dbc0 0x7f13c80b6d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:45.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:45.504+0000 7f13cd400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f13c806dbc0 0x7f13c80b6d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54290/0 (socket says 172.21.15.16:54290) 2024-03-31T15:04:45.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:45.504+0000 7f13cd400640 1 -- 172.21.15.16:0/4171875269 learned_addr learned my addr 172.21.15.16:0/4171875269 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:04:45.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:45.504+0000 7f13cde00640 1 --2- 172.21.15.16:0/4171875269 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f13c80cb440 0x7f13c80b50f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:45.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:45.504+0000 7f13cca00640 1 --2- 172.21.15.16:0/4171875269 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f13c8070030 0x7f13c80d0cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:45.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:45.504+0000 7f13cd400640 1 -- 172.21.15.16:0/4171875269 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f13c8070030 msgr2=0x7f13c80d0cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:45.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:45.504+0000 7f13cd400640 1 --2- 172.21.15.16:0/4171875269 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f13c8070030 0x7f13c80d0cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:45.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:45.504+0000 7f13cd400640 1 -- 172.21.15.16:0/4171875269 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f13c80cb440 msgr2=0x7f13c80b50f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:45.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:45.504+0000 7f13cd400640 1 --2- 172.21.15.16:0/4171875269 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f13c80cb440 0x7f13c80b50f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:45.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:45.504+0000 7f13cd400640 1 -- 172.21.15.16:0/4171875269 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13c8159230 con 0x7f13c806dbc0 2024-03-31T15:04:45.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:45.504+0000 7f13cde00640 1 --2- 172.21.15.16:0/4171875269 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f13c80cb440 0x7f13c80b50f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:04:45.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:45.504+0000 7f13cd400640 1 --2- 172.21.15.16:0/4171875269 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f13c806dbc0 0x7f13c80b6d60 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f13bc00d5f0 tx=0x7f13bc00dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:45.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:45.504+0000 7f13c5600640 1 -- 172.21.15.16:0/4171875269 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13bc0125f0 con 0x7f13c806dbc0 2024-03-31T15:04:45.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:45.504+0000 7f13c5600640 1 -- 172.21.15.16:0/4171875269 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f13bc019070 con 0x7f13c806dbc0 2024-03-31T15:04:45.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:45.504+0000 7f13c5600640 1 -- 172.21.15.16:0/4171875269 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13bc0129b0 con 0x7f13c806dbc0 2024-03-31T15:04:45.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:45.504+0000 7f13cfe00640 1 -- 172.21.15.16:0/4171875269 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f13c80b73d0 con 0x7f13c806dbc0 2024-03-31T15:04:45.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:45.504+0000 7f13cfe00640 1 -- 172.21.15.16:0/4171875269 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f13c80b7990 con 0x7f13c806dbc0 2024-03-31T15:04:45.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:45.508+0000 7f13cfe00640 1 -- 172.21.15.16:0/4171875269 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f13c8150ad0 con 0x7f13c806dbc0 2024-03-31T15:04:45.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:45.508+0000 7f13c5600640 1 -- 172.21.15.16:0/4171875269 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f13bc012b50 con 0x7f13c806dbc0 2024-03-31T15:04:45.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:45.508+0000 7f13c5600640 1 --2- 172.21.15.16:0/4171875269 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f13a8068380 0x7f13a806a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:45.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:45.508+0000 7f13c5600640 1 -- 172.21.15.16:0/4171875269 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5709+0+0 (secure 0 0 0) 0x7f13bc086e00 con 0x7f13c806dbc0 2024-03-31T15:04:45.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:45.508+0000 7f13cca00640 1 --2- 172.21.15.16:0/4171875269 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f13a8068380 0x7f13a806a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:45.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:45.512+0000 7f13cca00640 1 --2- 172.21.15.16:0/4171875269 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f13a8068380 0x7f13a806a840 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f13c81fab10 tx=0x7f13b4009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:45.508 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:45.512+0000 7f13c5600640 1 -- 172.21.15.16:0/4171875269 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f13bc0573b0 con 0x7f13c806dbc0 2024-03-31T15:04:45.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:45.712+0000 7f13cfe00640 1 -- 172.21.15.16:0/4171875269 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f13c8069950 con 0x7f13c806dbc0 2024-03-31T15:04:45.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:45.712+0000 7f13c5600640 1 -- 172.21.15.16:0/4171875269 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) ==== 94+0+5178 (secure 0 0 0) 0x7f13bc056cb0 con 0x7f13c806dbc0 2024-03-31T15:04:45.707 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:04:45.707 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":16,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4949,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6836/2073954753","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":2073954753},{"type":"v1","addr":"172.21.15.89:6836","nonce":2073954753}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":4952,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/2143918278","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6835","nonce":2143918278},{"type":"v1","addr":"172.21.15.89:6837","nonce":2143918278}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":5251,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6833/881391342","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":881391342},{"type":"v1","addr":"172.21.15.16:6833","nonce":881391342}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-31T15:04:39.780323+0000","modified":"2024-03-31T15:04:42.846439+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5254},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5254":{"gid":5254,"name":"c","rank":0,"incarnation":13,"state":"up:active","state_seq":3,"addr":"172.21.15.16:6835/1845707550","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":1845707550},{"type":"v1","addr":"172.21.15.16:6835","nonce":1845707550}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5254,"qdb_cluster":[5254]},"id":2}]} 2024-03-31T15:04:45.708 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 16 2024-03-31T15:04:45.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:45.712+0000 7f13cfe00640 1 -- 172.21.15.16:0/4171875269 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f13a8068380 msgr2=0x7f13a806a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:45.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:45.712+0000 7f13cfe00640 1 --2- 172.21.15.16:0/4171875269 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f13a8068380 0x7f13a806a840 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f13c81fab10 tx=0x7f13b4009040 comp rx=0 tx=0).stop 2024-03-31T15:04:45.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:45.712+0000 7f13cfe00640 1 -- 172.21.15.16:0/4171875269 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f13c806dbc0 msgr2=0x7f13c80b6d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:45.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:45.712+0000 7f13cfe00640 1 --2- 172.21.15.16:0/4171875269 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f13c806dbc0 0x7f13c80b6d60 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f13bc00d5f0 tx=0x7f13bc00dab0 comp rx=0 tx=0).stop 2024-03-31T15:04:45.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:45.712+0000 7f13cfe00640 1 -- 172.21.15.16:0/4171875269 shutdown_connections 2024-03-31T15:04:45.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:45.712+0000 7f13cfe00640 1 --2- 172.21.15.16:0/4171875269 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f13a8068380 0x7f13a806a840 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:45.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:45.712+0000 7f13cfe00640 1 --2- 172.21.15.16:0/4171875269 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f13c80cb440 0x7f13c80b50f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:45.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:45.712+0000 7f13cfe00640 1 --2- 172.21.15.16:0/4171875269 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f13c8070030 0x7f13c80d0cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:45.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:45.712+0000 7f13cfe00640 1 --2- 172.21.15.16:0/4171875269 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f13c806dbc0 0x7f13c80b6d60 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:45.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:45.712+0000 7f13cfe00640 1 -- 172.21.15.16:0/4171875269 >> 172.21.15.16:0/4171875269 conn(0x7f13c8067330 msgr2=0x7f13c81615c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:04:45.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:45.712+0000 7f13cfe00640 1 -- 172.21.15.16:0/4171875269 shutdown_connections 2024-03-31T15:04:45.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:45.712+0000 7f13cfe00640 1 -- 172.21.15.16:0/4171875269 wait complete. 2024-03-31T15:04:46.130 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 16, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-31T15:04:39.780323+0000', 'modified': '2024-03-31T15:04:42.846439+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5254}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5254': {'gid': 5254, 'name': 'c', 'rank': 0, 'incarnation': 13, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.16:6835/1845707550', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6834', 'nonce': 1845707550}, {'type': 'v1', 'addr': '172.21.15.16:6835', 'nonce': 1845707550}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [5], 'metadata_pool': 4, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 5254, 'qdb_cluster': [5254]} 2024-03-31T15:04:46.130 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-31T15:04:46.130 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-03-31T15:04:46.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.276+0000 7f694e200640 1 Processor -- start 2024-03-31T15:04:46.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.276+0000 7f694e200640 1 -- start start 2024-03-31T15:04:46.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.280+0000 7f694e200640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f694806f920 0x7f694806fd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:46.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.280+0000 7f694e200640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f694806d490 0x7f694806d8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:46.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.280+0000 7f694e200640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f69480cb440 0x7f69480cb8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:46.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.280+0000 7f694e200640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f6948162be0 con 0x7f694806d490 2024-03-31T15:04:46.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.280+0000 7f694e200640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f6948162010 con 0x7f694806f920 2024-03-31T15:04:46.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.280+0000 7f694e200640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f6948070240 con 0x7f69480cb440 2024-03-31T15:04:46.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.280+0000 7f693ea00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f694806d490 0x7f694806d8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:46.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.280+0000 7f6947e00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f69480cb440 0x7f69480cb8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:46.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.280+0000 7f6947400640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f694806f920 0x7f694806fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:46.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.280+0000 7f693ea00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f694806d490 0x7f694806d8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54296/0 (socket says 172.21.15.16:54296) 2024-03-31T15:04:46.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.280+0000 7f693ea00640 1 -- 172.21.15.16:0/1950647279 learned_addr learned my addr 172.21.15.16:0/1950647279 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:04:46.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.280+0000 7f6947e00640 1 -- 172.21.15.16:0/1950647279 >> v2:172.21.15.89:3300/0 conn(0x7f694806f920 msgr2=0x7f694806fd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:04:46.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.280+0000 7f6947e00640 1 --2- 172.21.15.16:0/1950647279 >> v2:172.21.15.89:3300/0 conn(0x7f694806f920 0x7f694806fd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:46.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.280+0000 7f6947e00640 1 -- 172.21.15.16:0/1950647279 >> v2:172.21.15.16:3300/0 conn(0x7f694806d490 msgr2=0x7f694806d8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:04:46.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.280+0000 7f6947e00640 1 --2- 172.21.15.16:0/1950647279 >> v2:172.21.15.16:3300/0 conn(0x7f694806d490 0x7f694806d8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:46.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.280+0000 7f6947e00640 1 -- 172.21.15.16:0/1950647279 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69481622b0 con 0x7f69480cb440 2024-03-31T15:04:46.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.280+0000 7f6947400640 1 --2- 172.21.15.16:0/1950647279 >> v2:172.21.15.89:3300/0 conn(0x7f694806f920 0x7f694806fd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:04:46.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.280+0000 7f693ea00640 1 --2- 172.21.15.16:0/1950647279 >> v2:172.21.15.16:3300/0 conn(0x7f694806d490 0x7f694806d8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:04:46.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.284+0000 7f6947e00640 1 --2- 172.21.15.16:0/1950647279 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f69480cb440 0x7f69480cb8d0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f69480ced00 tx=0x7f694001c900 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2653e5210280c927 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:46.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.284+0000 7f6946a00640 1 -- 172.21.15.16:0/1950647279 <== mon.2 v2:172.21.15.89:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69400b2070 con 0x7f69480cb440 2024-03-31T15:04:46.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.284+0000 7f6946a00640 1 -- 172.21.15.16:0/1950647279 <== mon.2 v2:172.21.15.89:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f69400a5890 con 0x7f69480cb440 2024-03-31T15:04:46.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.284+0000 7f6946a00640 1 -- 172.21.15.16:0/1950647279 <== mon.2 v2:172.21.15.89:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69400a5c60 con 0x7f69480cb440 2024-03-31T15:04:46.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.284+0000 7f694e200640 1 -- 172.21.15.16:0/1950647279 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f69480cb440 msgr2=0x7f69480cb8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:46.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.284+0000 7f694e200640 1 --2- 172.21.15.16:0/1950647279 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f69480cb440 0x7f69480cb8d0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f69480ced00 tx=0x7f694001c900 comp rx=0 tx=0).stop 2024-03-31T15:04:46.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.284+0000 7f694e200640 1 -- 172.21.15.16:0/1950647279 shutdown_connections 2024-03-31T15:04:46.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.284+0000 7f694e200640 1 --2- 172.21.15.16:0/1950647279 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f69480cb440 0x7f69480cb8d0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:46.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.284+0000 7f694e200640 1 --2- 172.21.15.16:0/1950647279 >> v2:172.21.15.16:3300/0 conn(0x7f694806d490 0x7f694806d8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:46.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.284+0000 7f694e200640 1 --2- 172.21.15.16:0/1950647279 >> v2:172.21.15.89:3300/0 conn(0x7f694806f920 0x7f694806fd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:46.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.284+0000 7f694e200640 1 -- 172.21.15.16:0/1950647279 >> 172.21.15.16:0/1950647279 conn(0x7f6948067330 msgr2=0x7f6948067740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:04:46.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.284+0000 7f694e200640 1 -- 172.21.15.16:0/1950647279 shutdown_connections 2024-03-31T15:04:46.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.284+0000 7f694e200640 1 -- 172.21.15.16:0/1950647279 wait complete. 2024-03-31T15:04:46.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.284+0000 7f694e200640 1 Processor -- start 2024-03-31T15:04:46.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.288+0000 7f694e200640 1 -- start start 2024-03-31T15:04:46.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.288+0000 7f694e200640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f694806d490 0x7f6948142ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:46.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.288+0000 7f694e200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f694806f920 0x7f69481fa810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:46.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.288+0000 7f694e200640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f69480cb440 0x7f694815ffd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:46.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.288+0000 7f694e200640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f69480ce770 con 0x7f694806f920 2024-03-31T15:04:46.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.288+0000 7f694e200640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f694806de30 con 0x7f694806d490 2024-03-31T15:04:46.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.288+0000 7f694e200640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f69481435d0 con 0x7f69480cb440 2024-03-31T15:04:46.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.288+0000 7f693ea00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f694806f920 0x7f69481fa810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:46.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.288+0000 7f693ea00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f694806f920 0x7f69481fa810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54304/0 (socket says 172.21.15.16:54304) 2024-03-31T15:04:46.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.288+0000 7f693ea00640 1 -- 172.21.15.16:0/962980239 learned_addr learned my addr 172.21.15.16:0/962980239 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:04:46.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.288+0000 7f6947400640 1 --2- 172.21.15.16:0/962980239 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f694806d490 0x7f6948142ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:46.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.288+0000 7f6947e00640 1 --2- 172.21.15.16:0/962980239 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f69480cb440 0x7f694815ffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:46.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.288+0000 7f693ea00640 1 -- 172.21.15.16:0/962980239 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f69480cb440 msgr2=0x7f694815ffd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:46.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.288+0000 7f693ea00640 1 --2- 172.21.15.16:0/962980239 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f69480cb440 0x7f694815ffd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:46.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.288+0000 7f693ea00640 1 -- 172.21.15.16:0/962980239 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f694806d490 msgr2=0x7f6948142ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:46.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.288+0000 7f693ea00640 1 --2- 172.21.15.16:0/962980239 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f694806d490 0x7f6948142ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:46.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.288+0000 7f693ea00640 1 -- 172.21.15.16:0/962980239 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69481433e0 con 0x7f694806f920 2024-03-31T15:04:46.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.288+0000 7f6947e00640 1 --2- 172.21.15.16:0/962980239 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f69480cb440 0x7f694815ffd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:04:46.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.288+0000 7f693ea00640 1 --2- 172.21.15.16:0/962980239 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f694806f920 0x7f69481fa810 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f693000e930 tx=0x7f693000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:46.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.288+0000 7f693fe00640 1 -- 172.21.15.16:0/962980239 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6930017070 con 0x7f694806f920 2024-03-31T15:04:46.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.288+0000 7f693fe00640 1 -- 172.21.15.16:0/962980239 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f693000ca00 con 0x7f694806f920 2024-03-31T15:04:46.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.288+0000 7f693fe00640 1 -- 172.21.15.16:0/962980239 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f693000cdd0 con 0x7f694806f920 2024-03-31T15:04:46.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.288+0000 7f694e200640 1 -- 172.21.15.16:0/962980239 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f69481fb260 con 0x7f694806f920 2024-03-31T15:04:46.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.288+0000 7f694e200640 1 -- 172.21.15.16:0/962980239 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6948143820 con 0x7f694806f920 2024-03-31T15:04:46.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.292+0000 7f693fe00640 1 -- 172.21.15.16:0/962980239 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f693001a070 con 0x7f694806f920 2024-03-31T15:04:46.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.292+0000 7f693fe00640 1 --2- 172.21.15.16:0/962980239 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f6924068380 0x7f692406a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:46.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.292+0000 7f693fe00640 1 -- 172.21.15.16:0/962980239 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5709+0+0 (secure 0 0 0) 0x7f6930086880 con 0x7f694806f920 2024-03-31T15:04:46.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.292+0000 7f695178c000 1 Processor -- start 2024-03-31T15:04:46.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.292+0000 7f695178c000 1 -- start start 2024-03-31T15:04:46.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.292+0000 7f6947400640 1 --2- 172.21.15.16:0/962980239 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f6924068380 0x7f692406a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:46.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.292+0000 7f695178c000 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x555933fe34a0 0x555933fe3880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:46.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.292+0000 7f695178c000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555933fe3dc0 0x555933f96b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:46.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.292+0000 7f695178c000 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555933f970b0 0x555933f97490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:46.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.292+0000 7f695178c000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x555933de9950 con 0x555933fe3dc0 2024-03-31T15:04:46.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.292+0000 7f695178c000 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x555933d86990 con 0x555933f970b0 2024-03-31T15:04:46.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.292+0000 7f695178c000 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x555933de5900 con 0x555933fe34a0 2024-03-31T15:04:46.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.292+0000 7f693ea00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555933fe3dc0 0x555933f96b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:46.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.292+0000 7f693ea00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555933fe3dc0 0x555933f96b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54316/0 (socket says 172.21.15.16:54316) 2024-03-31T15:04:46.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.292+0000 7f693ea00640 1 -- 172.21.15.16:0/1275595092 learned_addr learned my addr 172.21.15.16:0/1275595092 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:04:46.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.292+0000 7f6947400640 1 --2- 172.21.15.16:0/962980239 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f6924068380 0x7f692406a840 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f6938009580 tx=0x7f6938009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:46.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.292+0000 7f6947e00640 1 --2- 172.21.15.16:0/1275595092 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555933f970b0 0x555933f97490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:46.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.292+0000 7f6947400640 1 --2- 172.21.15.16:0/1275595092 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x555933fe34a0 0x555933fe3880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:46.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.292+0000 7f693ea00640 1 -- 172.21.15.16:0/1275595092 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x555933fe34a0 msgr2=0x555933fe3880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:04:46.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.292+0000 7f693ea00640 1 --2- 172.21.15.16:0/1275595092 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x555933fe34a0 0x555933fe3880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:46.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.292+0000 7f693ea00640 1 -- 172.21.15.16:0/1275595092 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555933f970b0 msgr2=0x555933f97490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:04:46.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.292+0000 7f693ea00640 1 --2- 172.21.15.16:0/1275595092 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555933f970b0 0x555933f97490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:46.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.292+0000 7f693ea00640 1 -- 172.21.15.16:0/1275595092 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555933e2e100 con 0x555933fe3dc0 2024-03-31T15:04:46.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.292+0000 7f6947400640 1 --2- 172.21.15.16:0/1275595092 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x555933fe34a0 0x555933fe3880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:04:46.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.296+0000 7f693ea00640 1 --2- 172.21.15.16:0/1275595092 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555933fe3dc0 0x555933f96b70 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f69300561b0 tx=0x7f69300561e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=68f9036dc4bdba2 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:46.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.296+0000 7f6937e00640 1 -- 172.21.15.16:0/1275595092 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6930021070 con 0x555933fe3dc0 2024-03-31T15:04:46.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.296+0000 7f6937e00640 1 -- 172.21.15.16:0/1275595092 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6930054a50 con 0x555933fe3dc0 2024-03-31T15:04:46.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.296+0000 7f6937e00640 1 -- 172.21.15.16:0/1275595092 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6930086040 con 0x555933fe3dc0 2024-03-31T15:04:46.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.296+0000 7f695178c000 1 -- 172.21.15.16:0/1275595092 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555933fe3dc0 msgr2=0x555933f96b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:46.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.296+0000 7f695178c000 1 --2- 172.21.15.16:0/1275595092 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555933fe3dc0 0x555933f96b70 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f69300561b0 tx=0x7f69300561e0 comp rx=0 tx=0).stop 2024-03-31T15:04:46.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.296+0000 7f695178c000 1 -- 172.21.15.16:0/1275595092 shutdown_connections 2024-03-31T15:04:46.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.296+0000 7f695178c000 1 --2- 172.21.15.16:0/1275595092 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555933f970b0 0x555933f97490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:46.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.296+0000 7f695178c000 1 --2- 172.21.15.16:0/1275595092 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555933fe3dc0 0x555933f96b70 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:46.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.296+0000 7f695178c000 1 --2- 172.21.15.16:0/1275595092 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x555933fe34a0 0x555933fe3880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:46.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.296+0000 7f695178c000 1 -- 172.21.15.16:0/1275595092 >> 172.21.15.16:0/1275595092 conn(0x555933e81b80 msgr2=0x555933fc6460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:04:46.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.296+0000 7f695178c000 1 -- 172.21.15.16:0/1275595092 shutdown_connections 2024-03-31T15:04:46.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.296+0000 7f695178c000 1 -- 172.21.15.16:0/1275595092 wait complete. 2024-03-31T15:04:46.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.296+0000 7f695178c000 1 -- start start 2024-03-31T15:04:46.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.296+0000 7f695178c000 1 Processor -- start 2024-03-31T15:04:46.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.296+0000 7f695178c000 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x555933f970b0 0x555933fa3e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:46.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.296+0000 7f695178c000 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555933fe34a0 0x555933fa4340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:46.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.296+0000 7f695178c000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555933fe3dc0 0x555933fa4880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:46.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.296+0000 7f695178c000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x555933de8c60 con 0x555933fe3dc0 2024-03-31T15:04:46.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.296+0000 7f695178c000 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x555933da73e0 con 0x555933fe34a0 2024-03-31T15:04:46.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.296+0000 7f695178c000 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x555933df6060 con 0x555933f970b0 2024-03-31T15:04:46.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.296+0000 7f695178c000 10 client.0 _ll_register_callbacks cb 0x555933e4a0f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T15:04:46.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.296+0000 7f6947e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555933fe3dc0 0x555933fa4880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:46.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.296+0000 7f6947400640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x555933f970b0 0x555933fa3e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:46.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.296+0000 7f693ea00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555933fe34a0 0x555933fa4340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:46.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.296+0000 7f6947400640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x555933f970b0 0x555933fa3e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3301/0 says I am v2:172.21.15.16:44360/0 (socket says 172.21.15.16:44360) 2024-03-31T15:04:46.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.296+0000 7f6947400640 1 -- 172.21.15.16:0/460092121 learned_addr learned my addr 172.21.15.16:0/460092121 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:04:46.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.296+0000 7f6947e00640 1 --2- 172.21.15.16:0/460092121 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555933fe3dc0 0x555933fa4880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54322/0 (socket says 172.21.15.16:54322) 2024-03-31T15:04:46.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.296+0000 7f693ea00640 1 -- 172.21.15.16:0/460092121 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x555933f970b0 msgr2=0x555933fa3e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:04:46.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.296+0000 7f693ea00640 1 --2- 172.21.15.16:0/460092121 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x555933f970b0 0x555933fa3e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:46.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.296+0000 7f693ea00640 1 -- 172.21.15.16:0/460092121 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555933fe3dc0 msgr2=0x555933fa4880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:04:46.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.296+0000 7f693ea00640 1 --2- 172.21.15.16:0/460092121 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555933fe3dc0 0x555933fa4880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:46.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.296+0000 7f693ea00640 1 -- 172.21.15.16:0/460092121 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555933db38c0 con 0x555933fe34a0 2024-03-31T15:04:46.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.296+0000 7f6947e00640 1 --2- 172.21.15.16:0/460092121 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555933fe3dc0 0x555933fa4880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:04:46.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.296+0000 7f693ea00640 1 --2- 172.21.15.16:0/460092121 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555933fe34a0 0x555933fa4340 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f6930056450 tx=0x7f69300568d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c7b1426eee3f5e69 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:46.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.296+0000 7f6934c00640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-03-31T15:04:46.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.300+0000 7f6934c00640 1 -- 172.21.15.16:0/460092121 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6930086050 con 0x555933fe34a0 2024-03-31T15:04:46.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.300+0000 7f695178c000 1 -- 172.21.15.16:0/460092121 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6930054a50 con 0x555933fe34a0 2024-03-31T15:04:46.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.300+0000 7f6934c00640 1 -- 172.21.15.16:0/460092121 <== mon.1 v2:172.21.15.89:3300/0 2 ==== osd_map(34..34 src has 1..34) ==== 5709+0+0 (secure 0 0 0) 0x7f6930055da0 con 0x555933fe34a0 2024-03-31T15:04:46.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.300+0000 7f6934c00640 1 -- 172.21.15.16:0/460092121 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=16) ==== 24+0+0 (secure 0 0 0) 0x7f693007b050 con 0x555933fe34a0 2024-03-31T15:04:46.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.300+0000 7f695178c000 10 client.5398 fetch_fsmap learned FSMap version 16 2024-03-31T15:04:46.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.300+0000 7f695178c000 1 -- 172.21.15.16:0/460092121 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=16,monmap=2+}) -- 0x555933d71e80 con 0x555933fe34a0 2024-03-31T15:04:46.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.300+0000 7f6934c00640 1 -- 172.21.15.16:0/460092121 <== mon.1 v2:172.21.15.89:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6930054a50 con 0x555933fe34a0 2024-03-31T15:04:46.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.300+0000 7f6934c00640 1 -- 172.21.15.16:0/460092121 <== mon.1 v2:172.21.15.89:3300/0 5 ==== fsmap(e 16) ==== 3494+0+0 (secure 0 0 0) 0x7f693007e020 con 0x555933fe34a0 2024-03-31T15:04:46.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.300+0000 7f695178c000 10 client.5398 fetch_fsmap finished waiting for FSMap version 16 2024-03-31T15:04:46.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.300+0000 7f695178c000 10 client.5398 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-31T15:04:46.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.300+0000 7f695178c000 20 client.5398 populate_metadata read hostname 'smithi016' 2024-03-31T15:04:46.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.300+0000 7f695178c000 1 --2- 172.21.15.16:0/460092121 >> [v2:172.21.15.16:6834/1845707550,v1:172.21.15.16:6835/1845707550] conn(0x555933fa5150 0x555933ff5a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:46.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.300+0000 7f695178c000 4 client.5398 mds_command: new command op to 5254 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-31T15:04:46.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.300+0000 7f695178c000 1 -- 172.21.15.16:0/460092121 --> [v2:172.21.15.16:6834/1845707550,v1:172.21.15.16:6835/1845707550] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555933d84bc0 con 0x555933fa5150 2024-03-31T15:04:46.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.300+0000 7f6947400640 1 --2- 172.21.15.16:0/460092121 >> [v2:172.21.15.16:6834/1845707550,v1:172.21.15.16:6835/1845707550] conn(0x555933fa5150 0x555933ff5a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:46.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.300+0000 7f6947400640 1 --2- 172.21.15.16:0/460092121 >> [v2:172.21.15.16:6834/1845707550,v1:172.21.15.16:6835/1845707550] conn(0x555933fa5150 0x555933ff5a20 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=61a33d3fc188df40 server_cookie=7c3cb9157c454865 in_seq=0 out_seq=0 2024-03-31T15:04:46.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.300+0000 7f6934c00640 10 client.5398 ms_handle_connect on v2:172.21.15.16:6834/1845707550 2024-03-31T15:04:46.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.300+0000 7f6934c00640 1 -- 172.21.15.16:0/460092121 <== mds.0 v2:172.21.15.16:6834/1845707550 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x555933d84bc0 con 0x555933fa5150 2024-03-31T15:04:46.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.300+0000 7f6934c00640 10 client.5398 handle_command_reply: tid=0 2024-03-31T15:04:46.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.300+0000 7f695178c000 1 client.5398 shutdown 2024-03-31T15:04:46.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.300+0000 7f695178c000 1 -- 172.21.15.16:0/460092121 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555933fe34a0 msgr2=0x555933fa4340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:46.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.300+0000 7f695178c000 1 --2- 172.21.15.16:0/460092121 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555933fe34a0 0x555933fa4340 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f6930056450 tx=0x7f69300568d0 comp rx=0 tx=0).stop 2024-03-31T15:04:46.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.300+0000 7f695178c000 1 -- 172.21.15.16:0/460092121 shutdown_connections 2024-03-31T15:04:46.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.300+0000 7f695178c000 1 --2- 172.21.15.16:0/460092121 >> [v2:172.21.15.16:6834/1845707550,v1:172.21.15.16:6835/1845707550] conn(0x555933fa5150 0x555933ff5a20 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:46.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.300+0000 7f695178c000 1 --2- 172.21.15.16:0/460092121 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555933fe3dc0 0x555933fa4880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:46.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.300+0000 7f695178c000 1 --2- 172.21.15.16:0/460092121 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555933fe34a0 0x555933fa4340 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:46.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.300+0000 7f6934c00640 0 client.5398 ms_handle_reset on v2:172.21.15.16:6834/1845707550 2024-03-31T15:04:46.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.300+0000 7f695178c000 1 --2- 172.21.15.16:0/460092121 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x555933f970b0 0x555933fa3e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:46.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.300+0000 7f695178c000 1 -- 172.21.15.16:0/460092121 >> 172.21.15.16:0/460092121 conn(0x555933e81b80 msgr2=0x555933f96270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:04:46.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.300+0000 7f695178c000 1 -- 172.21.15.16:0/460092121 shutdown_connections 2024-03-31T15:04:46.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.300+0000 7f695178c000 1 -- 172.21.15.16:0/460092121 wait complete. 2024-03-31T15:04:46.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.300+0000 7f695178c000 20 client.5398 trim_cache size 0 max 16384 2024-03-31T15:04:46.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.320+0000 7f695178c000 1 Processor -- start 2024-03-31T15:04:46.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.320+0000 7f695178c000 1 -- start start 2024-03-31T15:04:46.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.320+0000 7f695178c000 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x555933f970b0 0x555933fe2da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:46.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.320+0000 7f695178c000 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555933fe34a0 0x555933fc5d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:46.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.320+0000 7f695178c000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555933fe3dc0 0x555933fc6280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:46.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.320+0000 7f695178c000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f6948202b60 con 0x555933fe3dc0 2024-03-31T15:04:46.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.320+0000 7f695178c000 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f694815e400 con 0x555933fe34a0 2024-03-31T15:04:46.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.320+0000 7f695178c000 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f694815ea50 con 0x555933f970b0 2024-03-31T15:04:46.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.320+0000 7f6947e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555933fe3dc0 0x555933fc6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:46.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.320+0000 7f6947e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555933fe3dc0 0x555933fc6280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54326/0 (socket says 172.21.15.16:54326) 2024-03-31T15:04:46.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.320+0000 7f6947e00640 1 -- 172.21.15.16:0/4150233599 learned_addr learned my addr 172.21.15.16:0/4150233599 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:04:46.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.320+0000 7f6947400640 1 --2- 172.21.15.16:0/4150233599 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x555933f970b0 0x555933fe2da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:46.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.320+0000 7f693ea00640 1 --2- 172.21.15.16:0/4150233599 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555933fe34a0 0x555933fc5d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:46.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.320+0000 7f6947e00640 1 -- 172.21.15.16:0/4150233599 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x555933f970b0 msgr2=0x555933fe2da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:04:46.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.320+0000 7f6947e00640 1 --2- 172.21.15.16:0/4150233599 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x555933f970b0 0x555933fe2da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:46.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.320+0000 7f6947e00640 1 -- 172.21.15.16:0/4150233599 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555933fe34a0 msgr2=0x555933fc5d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:04:46.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.320+0000 7f6947e00640 1 --2- 172.21.15.16:0/4150233599 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555933fe34a0 0x555933fc5d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:46.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.320+0000 7f6947e00640 1 -- 172.21.15.16:0/4150233599 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555933da4490 con 0x555933fe3dc0 2024-03-31T15:04:46.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.320+0000 7f6947e00640 1 --2- 172.21.15.16:0/4150233599 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555933fe3dc0 0x555933fc6280 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f694001afb0 tx=0x7f6940004ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=537b1e0121f28b84 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:46.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.320+0000 7f6937400640 1 -- 172.21.15.16:0/4150233599 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69400c6070 con 0x555933fe3dc0 2024-03-31T15:04:46.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.320+0000 7f6937400640 1 -- 172.21.15.16:0/4150233599 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f69400a5aa0 con 0x555933fe3dc0 2024-03-31T15:04:46.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.320+0000 7f6937400640 1 -- 172.21.15.16:0/4150233599 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69400b2040 con 0x555933fe3dc0 2024-03-31T15:04:46.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.320+0000 7f695178c000 1 -- 172.21.15.16:0/4150233599 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555933fe3dc0 msgr2=0x555933fc6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:46.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.320+0000 7f695178c000 1 --2- 172.21.15.16:0/4150233599 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555933fe3dc0 0x555933fc6280 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f694001afb0 tx=0x7f6940004ac0 comp rx=0 tx=0).stop 2024-03-31T15:04:46.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.320+0000 7f695178c000 1 -- 172.21.15.16:0/4150233599 shutdown_connections 2024-03-31T15:04:46.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.320+0000 7f695178c000 1 --2- 172.21.15.16:0/4150233599 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555933fe3dc0 0x555933fc6280 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:46.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.320+0000 7f695178c000 1 --2- 172.21.15.16:0/4150233599 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555933fe34a0 0x555933fc5d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:46.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.320+0000 7f695178c000 1 --2- 172.21.15.16:0/4150233599 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x555933f970b0 0x555933fe2da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:46.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.320+0000 7f695178c000 1 -- 172.21.15.16:0/4150233599 >> 172.21.15.16:0/4150233599 conn(0x555933e81b80 msgr2=0x555933fc5630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:04:46.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.320+0000 7f695178c000 1 -- 172.21.15.16:0/4150233599 shutdown_connections 2024-03-31T15:04:46.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.320+0000 7f695178c000 1 -- 172.21.15.16:0/4150233599 wait complete. 2024-03-31T15:04:46.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.324+0000 7f695178c000 1 -- start start 2024-03-31T15:04:46.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.324+0000 7f695178c000 1 Processor -- start 2024-03-31T15:04:46.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.324+0000 7f695178c000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555933f970b0 0x555933f96710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:46.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.324+0000 7f695178c000 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x555933fe34a0 0x555934052530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:46.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.324+0000 7f695178c000 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555933fe3dc0 0x555934052a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:46.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.324+0000 7f695178c000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x555933df6060 con 0x555933f970b0 2024-03-31T15:04:46.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.324+0000 7f695178c000 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x555933dedf80 con 0x555933fe3dc0 2024-03-31T15:04:46.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.324+0000 7f695178c000 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x555933da73e0 con 0x555933fe34a0 2024-03-31T15:04:46.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.324+0000 7f695178c000 10 client.0 _ll_register_callbacks cb 0x7f69300560b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T15:04:46.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.324+0000 7f6947400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555933f970b0 0x555933f96710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:46.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.324+0000 7f6947400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555933f970b0 0x555933f96710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54340/0 (socket says 172.21.15.16:54340) 2024-03-31T15:04:46.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.324+0000 7f6947400640 1 -- 172.21.15.16:0/3500653281 learned_addr learned my addr 172.21.15.16:0/3500653281 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:04:46.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.324+0000 7f693ea00640 1 --2- 172.21.15.16:0/3500653281 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x555933fe34a0 0x555934052530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:46.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.324+0000 7f6947e00640 1 --2- 172.21.15.16:0/3500653281 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555933fe3dc0 0x555934052a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:46.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.324+0000 7f6947400640 1 -- 172.21.15.16:0/3500653281 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x555933fe34a0 msgr2=0x555934052530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:04:46.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.324+0000 7f6947400640 1 --2- 172.21.15.16:0/3500653281 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x555933fe34a0 0x555934052530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:46.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.324+0000 7f6947400640 1 -- 172.21.15.16:0/3500653281 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555933fe3dc0 msgr2=0x555934052a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:04:46.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.324+0000 7f6947400640 1 --2- 172.21.15.16:0/3500653281 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555933fe3dc0 0x555934052a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:46.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.324+0000 7f6947400640 1 -- 172.21.15.16:0/3500653281 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555933da2e80 con 0x555933f970b0 2024-03-31T15:04:46.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.324+0000 7f6947400640 1 --2- 172.21.15.16:0/3500653281 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555933f970b0 0x555933f96710 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f693802af50 tx=0x7f6938004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7707c48d620b594 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:46.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.324+0000 7f6934c00640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-31T15:04:46.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.324+0000 7f6934c00640 1 -- 172.21.15.16:0/3500653281 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f693802a460 con 0x555933f970b0 2024-03-31T15:04:46.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.324+0000 7f695178c000 1 -- 172.21.15.16:0/3500653281 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f69400a5aa0 con 0x555933f970b0 2024-03-31T15:04:46.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.324+0000 7f6934c00640 1 -- 172.21.15.16:0/3500653281 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(34..34 src has 1..34) ==== 5709+0+0 (secure 0 0 0) 0x7f693800f040 con 0x555933f970b0 2024-03-31T15:04:46.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.324+0000 7f6934c00640 1 -- 172.21.15.16:0/3500653281 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=16) ==== 24+0+0 (secure 0 0 0) 0x7f693802a600 con 0x555933f970b0 2024-03-31T15:04:46.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.324+0000 7f695178c000 10 client.5403 fetch_fsmap learned FSMap version 16 2024-03-31T15:04:46.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.324+0000 7f695178c000 1 -- 172.21.15.16:0/3500653281 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=16,monmap=2+}) -- 0x555933d90270 con 0x555933f970b0 2024-03-31T15:04:46.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.324+0000 7f6934c00640 1 -- 172.21.15.16:0/3500653281 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f69400a5aa0 con 0x555933f970b0 2024-03-31T15:04:46.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.324+0000 7f6934c00640 1 -- 172.21.15.16:0/3500653281 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 16) ==== 3494+0+0 (secure 0 0 0) 0x7f6938020040 con 0x555933f970b0 2024-03-31T15:04:46.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.324+0000 7f695178c000 10 client.5403 fetch_fsmap finished waiting for FSMap version 16 2024-03-31T15:04:46.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.324+0000 7f695178c000 10 client.5403 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-31T15:04:46.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.324+0000 7f695178c000 20 client.5403 populate_metadata read hostname 'smithi016' 2024-03-31T15:04:46.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.324+0000 7f695178c000 1 --2- 172.21.15.16:0/3500653281 >> [v2:172.21.15.16:6834/1845707550,v1:172.21.15.16:6835/1845707550] conn(0x555933ff5a20 0x555933ff5e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:46.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.324+0000 7f695178c000 4 client.5403 mds_command: new command op to 5254 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-31T15:04:46.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.324+0000 7f695178c000 1 -- 172.21.15.16:0/3500653281 --> [v2:172.21.15.16:6834/1845707550,v1:172.21.15.16:6835/1845707550] -- command(tid 0: {"prefix": "status"}) -- 0x555933df6e90 con 0x555933ff5a20 2024-03-31T15:04:46.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.324+0000 7f693ea00640 1 --2- 172.21.15.16:0/3500653281 >> [v2:172.21.15.16:6834/1845707550,v1:172.21.15.16:6835/1845707550] conn(0x555933ff5a20 0x555933ff5e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:46.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.328+0000 7f6934c00640 10 client.5403 ms_handle_connect on v2:172.21.15.16:6834/1845707550 2024-03-31T15:04:46.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.328+0000 7f693ea00640 1 --2- 172.21.15.16:0/3500653281 >> [v2:172.21.15.16:6834/1845707550,v1:172.21.15.16:6835/1845707550] conn(0x555933ff5a20 0x555933ff5e00 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5f842ac7be1a1e63 server_cookie=9dcd6bedbba1bfb in_seq=0 out_seq=0 2024-03-31T15:04:46.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.328+0000 7f6934c00640 1 -- 172.21.15.16:0/3500653281 <== mds.0 v2:172.21.15.16:6834/1845707550 1 ==== command_reply(tid 0: 0 ) ==== 8+0+318 (crc 0 0 0) 0x555933df6e90 con 0x555933ff5a20 2024-03-31T15:04:46.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.328+0000 7f6934c00640 10 client.5403 handle_command_reply: tid=0 2024-03-31T15:04:46.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.328+0000 7f695178c000 1 client.5403 shutdown 2024-03-31T15:04:46.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.328+0000 7f695178c000 1 -- 172.21.15.16:0/3500653281 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555933f970b0 msgr2=0x555933f96710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:46.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.328+0000 7f695178c000 1 --2- 172.21.15.16:0/3500653281 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555933f970b0 0x555933f96710 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f693802af50 tx=0x7f6938004740 comp rx=0 tx=0).stop 2024-03-31T15:04:46.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.328+0000 7f695178c000 1 -- 172.21.15.16:0/3500653281 shutdown_connections 2024-03-31T15:04:46.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.328+0000 7f695178c000 1 --2- 172.21.15.16:0/3500653281 >> [v2:172.21.15.16:6834/1845707550,v1:172.21.15.16:6835/1845707550] conn(0x555933ff5a20 0x555933ff5e00 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:46.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.328+0000 7f695178c000 1 --2- 172.21.15.16:0/3500653281 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555933fe3dc0 0x555934052a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:46.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.328+0000 7f695178c000 1 --2- 172.21.15.16:0/3500653281 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x555933fe34a0 0x555934052530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:46.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.328+0000 7f695178c000 1 --2- 172.21.15.16:0/3500653281 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555933f970b0 0x555933f96710 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:46.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.328+0000 7f695178c000 1 -- 172.21.15.16:0/3500653281 >> 172.21.15.16:0/3500653281 conn(0x555933e81b80 msgr2=0x555933fc5ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:04:46.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.328+0000 7f6934c00640 0 client.5403 ms_handle_reset on v2:172.21.15.16:6834/1845707550 2024-03-31T15:04:46.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.328+0000 7f695178c000 1 -- 172.21.15.16:0/3500653281 shutdown_connections 2024-03-31T15:04:46.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.328+0000 7f695178c000 1 -- 172.21.15.16:0/3500653281 wait complete. 2024-03-31T15:04:46.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.328+0000 7f695178c000 20 client.5403 trim_cache size 0 max 16384 2024-03-31T15:04:46.326 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-03-31T15:04:46.326 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_fsid": "286eb5da-fb47-4550-b290-d9519c13b9a5", 2024-03-31T15:04:46.326 INFO:teuthology.orchestra.run.smithi016.stdout: "whoami": 0, 2024-03-31T15:04:46.326 INFO:teuthology.orchestra.run.smithi016.stdout: "id": 5254, 2024-03-31T15:04:46.326 INFO:teuthology.orchestra.run.smithi016.stdout: "want_state": "up:active", 2024-03-31T15:04:46.326 INFO:teuthology.orchestra.run.smithi016.stdout: "state": "up:active", 2024-03-31T15:04:46.326 INFO:teuthology.orchestra.run.smithi016.stdout: "fs_name": "cephfs", 2024-03-31T15:04:46.326 INFO:teuthology.orchestra.run.smithi016.stdout: "rank_uptime": 5.954139316, 2024-03-31T15:04:46.327 INFO:teuthology.orchestra.run.smithi016.stdout: "mdsmap_epoch": 16, 2024-03-31T15:04:46.327 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch": 34, 2024-03-31T15:04:46.327 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch_barrier": 34, 2024-03-31T15:04:46.327 INFO:teuthology.orchestra.run.smithi016.stdout: "uptime": 13.813142335 2024-03-31T15:04:46.327 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-03-31T15:04:46.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.328+0000 7f694e200640 1 -- 172.21.15.16:0/962980239 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f6924068380 msgr2=0x7f692406a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:46.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.328+0000 7f694e200640 1 --2- 172.21.15.16:0/962980239 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f6924068380 0x7f692406a840 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f6938009580 tx=0x7f6938009190 comp rx=0 tx=0).stop 2024-03-31T15:04:46.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.328+0000 7f694e200640 1 -- 172.21.15.16:0/962980239 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f694806f920 msgr2=0x7f69481fa810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:46.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.328+0000 7f694e200640 1 --2- 172.21.15.16:0/962980239 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f694806f920 0x7f69481fa810 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f693000e930 tx=0x7f693000edf0 comp rx=0 tx=0).stop 2024-03-31T15:04:46.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.328+0000 7f694e200640 1 -- 172.21.15.16:0/962980239 shutdown_connections 2024-03-31T15:04:46.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.328+0000 7f694e200640 1 --2- 172.21.15.16:0/962980239 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f6924068380 0x7f692406a840 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:46.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.328+0000 7f694e200640 1 --2- 172.21.15.16:0/962980239 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f69480cb440 0x7f694815ffd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:46.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.328+0000 7f694e200640 1 --2- 172.21.15.16:0/962980239 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f694806f920 0x7f69481fa810 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:46.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.328+0000 7f694e200640 1 --2- 172.21.15.16:0/962980239 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f694806d490 0x7f6948142ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:46.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.328+0000 7f694e200640 1 -- 172.21.15.16:0/962980239 >> 172.21.15.16:0/962980239 conn(0x7f6948067330 msgr2=0x7f694806e1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:04:46.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.328+0000 7f694e200640 1 -- 172.21.15.16:0/962980239 shutdown_connections 2024-03-31T15:04:46.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.328+0000 7f694e200640 1 -- 172.21.15.16:0/962980239 wait complete. 2024-03-31T15:04:46.341 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-31T15:04:46.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.536+0000 7f4a88c00640 1 Processor -- start 2024-03-31T15:04:46.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.536+0000 7f4a88c00640 1 -- start start 2024-03-31T15:04:46.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.540+0000 7f4a88c00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f4a8406dbc0 0x7f4a8415f650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:46.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.540+0000 7f4a88c00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f4a840cb440 0x7f4a8406f920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:46.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.540+0000 7f4a88c00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f4a840cb8c0 0x7f4a8406fe60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:46.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.540+0000 7f4a88c00640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f4a8415fb90 con 0x7f4a840cb8c0 2024-03-31T15:04:46.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.540+0000 7f4a88c00640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f4a8415f1c0 con 0x7f4a840cb440 2024-03-31T15:04:46.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.540+0000 7f4a88c00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f4a840cbe90 con 0x7f4a8406dbc0 2024-03-31T15:04:46.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.540+0000 7f4a82800640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f4a840cb8c0 0x7f4a8406fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:46.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.540+0000 7f4a82800640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f4a840cb8c0 0x7f4a8406fe60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54348/0 (socket says 172.21.15.16:54348) 2024-03-31T15:04:46.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.540+0000 7f4a82800640 1 -- 172.21.15.16:0/3575545753 learned_addr learned my addr 172.21.15.16:0/3575545753 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:04:46.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.540+0000 7f4a81e00640 1 --2- 172.21.15.16:0/3575545753 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f4a8406dbc0 0x7f4a8415f650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:46.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.540+0000 7f4a81400640 1 --2- 172.21.15.16:0/3575545753 >> v2:172.21.15.89:3300/0 conn(0x7f4a840cb440 0x7f4a8406f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:46.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.540+0000 7f4a82800640 1 -- 172.21.15.16:0/3575545753 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f4a8406dbc0 msgr2=0x7f4a8415f650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:04:46.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.540+0000 7f4a82800640 1 --2- 172.21.15.16:0/3575545753 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f4a8406dbc0 0x7f4a8415f650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:46.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.540+0000 7f4a82800640 1 -- 172.21.15.16:0/3575545753 >> v2:172.21.15.89:3300/0 conn(0x7f4a840cb440 msgr2=0x7f4a8406f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:04:46.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.540+0000 7f4a82800640 1 --2- 172.21.15.16:0/3575545753 >> v2:172.21.15.89:3300/0 conn(0x7f4a840cb440 0x7f4a8406f920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:46.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.540+0000 7f4a82800640 1 -- 172.21.15.16:0/3575545753 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a840cc170 con 0x7f4a840cb8c0 2024-03-31T15:04:46.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.544+0000 7f4a82800640 1 --2- 172.21.15.16:0/3575545753 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4a840cb8c0 0x7f4a8406fe60 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f4a840d31b0 tx=0x7f4a7c01c930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=63fc33e375761911 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:46.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.544+0000 7f4a80a00640 1 -- 172.21.15.16:0/3575545753 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a7c0a5770 con 0x7f4a840cb8c0 2024-03-31T15:04:46.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.544+0000 7f4a80a00640 1 -- 172.21.15.16:0/3575545753 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4a7c0a5910 con 0x7f4a840cb8c0 2024-03-31T15:04:46.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.544+0000 7f4a80a00640 1 -- 172.21.15.16:0/3575545753 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a7c0a5ce0 con 0x7f4a840cb8c0 2024-03-31T15:04:46.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.544+0000 7f4a88c00640 1 -- 172.21.15.16:0/3575545753 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4a840cb8c0 msgr2=0x7f4a8406fe60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:46.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.544+0000 7f4a88c00640 1 --2- 172.21.15.16:0/3575545753 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4a840cb8c0 0x7f4a8406fe60 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f4a840d31b0 tx=0x7f4a7c01c930 comp rx=0 tx=0).stop 2024-03-31T15:04:46.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.544+0000 7f4a88c00640 1 -- 172.21.15.16:0/3575545753 shutdown_connections 2024-03-31T15:04:46.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.544+0000 7f4a88c00640 1 --2- 172.21.15.16:0/3575545753 >> v2:172.21.15.89:3300/0 conn(0x7f4a840cb440 0x7f4a8406f920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:46.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.544+0000 7f4a88c00640 1 --2- 172.21.15.16:0/3575545753 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4a840cb8c0 0x7f4a8406fe60 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:46.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.544+0000 7f4a88c00640 1 --2- 172.21.15.16:0/3575545753 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f4a8406dbc0 0x7f4a8415f650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:46.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.544+0000 7f4a88c00640 1 -- 172.21.15.16:0/3575545753 >> 172.21.15.16:0/3575545753 conn(0x7f4a84067330 msgr2=0x7f4a84067740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:04:46.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.544+0000 7f4a88c00640 1 -- 172.21.15.16:0/3575545753 shutdown_connections 2024-03-31T15:04:46.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.544+0000 7f4a88c00640 1 -- 172.21.15.16:0/3575545753 wait complete. 2024-03-31T15:04:46.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.548+0000 7f4a88c00640 1 Processor -- start 2024-03-31T15:04:46.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.548+0000 7f4a88c00640 1 -- start start 2024-03-31T15:04:46.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.548+0000 7f4a88c00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4a8406dbc0 0x7f4a841424c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:46.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.548+0000 7f4a88c00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4a840cb440 0x7f4a84141c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:46.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.548+0000 7f4a88c00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f4a840cb8c0 0x7f4a8415b240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:46.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.548+0000 7f4a88c00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f4a8406cad0 con 0x7f4a840cb440 2024-03-31T15:04:46.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.548+0000 7f4a88c00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f4a8406c950 con 0x7f4a8406dbc0 2024-03-31T15:04:46.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.548+0000 7f4a88c00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f4a84162990 con 0x7f4a840cb8c0 2024-03-31T15:04:46.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.548+0000 7f4a81400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4a840cb440 0x7f4a84141c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:46.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.548+0000 7f4a81400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4a840cb440 0x7f4a84141c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54354/0 (socket says 172.21.15.16:54354) 2024-03-31T15:04:46.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.548+0000 7f4a81400640 1 -- 172.21.15.16:0/3021486197 learned_addr learned my addr 172.21.15.16:0/3021486197 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:04:46.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.548+0000 7f4a81e00640 1 --2- 172.21.15.16:0/3021486197 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4a8406dbc0 0x7f4a841424c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:46.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.548+0000 7f4a82800640 1 --2- 172.21.15.16:0/3021486197 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f4a840cb8c0 0x7f4a8415b240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:46.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.548+0000 7f4a81400640 1 -- 172.21.15.16:0/3021486197 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f4a840cb8c0 msgr2=0x7f4a8415b240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:46.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.548+0000 7f4a81400640 1 --2- 172.21.15.16:0/3021486197 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f4a840cb8c0 0x7f4a8415b240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:46.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.548+0000 7f4a81400640 1 -- 172.21.15.16:0/3021486197 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4a8406dbc0 msgr2=0x7f4a841424c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:46.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.548+0000 7f4a81400640 1 --2- 172.21.15.16:0/3021486197 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4a8406dbc0 0x7f4a841424c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:46.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.548+0000 7f4a81400640 1 -- 172.21.15.16:0/3021486197 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a84142dc0 con 0x7f4a840cb440 2024-03-31T15:04:46.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.548+0000 7f4a81400640 1 --2- 172.21.15.16:0/3021486197 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4a840cb440 0x7f4a84141c30 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f4a7000e930 tx=0x7f4a7000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:46.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.548+0000 7f4a7a000640 1 -- 172.21.15.16:0/3021486197 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a70017070 con 0x7f4a840cb440 2024-03-31T15:04:46.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.548+0000 7f4a7a000640 1 -- 172.21.15.16:0/3021486197 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4a7000ca00 con 0x7f4a840cb440 2024-03-31T15:04:46.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.548+0000 7f4a7a000640 1 -- 172.21.15.16:0/3021486197 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a7000cdb0 con 0x7f4a840cb440 2024-03-31T15:04:46.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.548+0000 7f4a88c00640 1 -- 172.21.15.16:0/3021486197 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4a841430a0 con 0x7f4a840cb440 2024-03-31T15:04:46.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.548+0000 7f4a88c00640 1 -- 172.21.15.16:0/3021486197 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4a84143660 con 0x7f4a840cb440 2024-03-31T15:04:46.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.552+0000 7f4a88c00640 1 -- 172.21.15.16:0/3021486197 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4a84069950 con 0x7f4a840cb440 2024-03-31T15:04:46.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.552+0000 7f4a7a000640 1 -- 172.21.15.16:0/3021486197 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f4a7001a070 con 0x7f4a840cb440 2024-03-31T15:04:46.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.552+0000 7f4a7a000640 1 --2- 172.21.15.16:0/3021486197 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f4a64068380 0x7f4a6406a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:46.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.552+0000 7f4a7a000640 1 -- 172.21.15.16:0/3021486197 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5709+0+0 (secure 0 0 0) 0x7f4a70086e00 con 0x7f4a840cb440 2024-03-31T15:04:46.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.552+0000 7f4a81e00640 1 --2- 172.21.15.16:0/3021486197 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f4a64068380 0x7f4a6406a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:46.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.552+0000 7f4a81e00640 1 --2- 172.21.15.16:0/3021486197 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f4a64068380 0x7f4a6406a840 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f4a74009ae0 tx=0x7f4a74009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:46.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.556+0000 7f4a7a000640 1 -- 172.21.15.16:0/3021486197 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f4a70056d10 con 0x7f4a840cb440 2024-03-31T15:04:46.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.764+0000 7f4a88c00640 1 -- 172.21.15.16:0/3021486197 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f4a8406e460 con 0x7f4a840cb440 2024-03-31T15:04:46.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.764+0000 7f4a7a000640 1 -- 172.21.15.16:0/3021486197 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v16) ==== 53+0+73 (secure 0 0 0) 0x7f4a7005b010 con 0x7f4a840cb440 2024-03-31T15:04:46.759 INFO:teuthology.orchestra.run.smithi016.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-31T15:04:46.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.768+0000 7f4a88c00640 1 -- 172.21.15.16:0/3021486197 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f4a64068380 msgr2=0x7f4a6406a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:46.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.768+0000 7f4a88c00640 1 --2- 172.21.15.16:0/3021486197 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f4a64068380 0x7f4a6406a840 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f4a74009ae0 tx=0x7f4a74009190 comp rx=0 tx=0).stop 2024-03-31T15:04:46.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.768+0000 7f4a88c00640 1 -- 172.21.15.16:0/3021486197 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4a840cb440 msgr2=0x7f4a84141c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:46.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.768+0000 7f4a88c00640 1 --2- 172.21.15.16:0/3021486197 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4a840cb440 0x7f4a84141c30 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f4a7000e930 tx=0x7f4a7000edf0 comp rx=0 tx=0).stop 2024-03-31T15:04:46.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.768+0000 7f4a88c00640 1 -- 172.21.15.16:0/3021486197 shutdown_connections 2024-03-31T15:04:46.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.768+0000 7f4a88c00640 1 --2- 172.21.15.16:0/3021486197 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f4a64068380 0x7f4a6406a840 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:46.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.768+0000 7f4a88c00640 1 --2- 172.21.15.16:0/3021486197 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f4a840cb8c0 0x7f4a8415b240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:46.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.768+0000 7f4a88c00640 1 --2- 172.21.15.16:0/3021486197 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4a840cb440 0x7f4a84141c30 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:46.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.768+0000 7f4a88c00640 1 --2- 172.21.15.16:0/3021486197 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4a8406dbc0 0x7f4a841424c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:46.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.768+0000 7f4a88c00640 1 -- 172.21.15.16:0/3021486197 >> 172.21.15.16:0/3021486197 conn(0x7f4a84067330 msgr2=0x7f4a8415d9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:04:46.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.768+0000 7f4a88c00640 1 -- 172.21.15.16:0/3021486197 shutdown_connections 2024-03-31T15:04:46.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46.768+0000 7f4a88c00640 1 -- 172.21.15.16:0/3021486197 wait complete. 2024-03-31T15:04:46.780 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-31T15:04:46.780 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-31T15:04:46.780 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi016.front.sepia.ceph.com 2024-03-31T15:04:46.781 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-31T15:04:46.781 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-31T15:04:46.781 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-31T15:04:46.781 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-31T15:04:46.781 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-31T15:04:46.781 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-31T15:04:46.781 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:04:46.781 DEBUG:teuthology.orchestra.run.smithi016:> ip addr 2024-03-31T15:04:46.790 INFO:teuthology.orchestra.run.smithi016.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-31T15:04:46.791 INFO:teuthology.orchestra.run.smithi016.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-31T15:04:46.791 INFO:teuthology.orchestra.run.smithi016.stdout: inet 127.0.0.1/8 scope host lo 2024-03-31T15:04:46.791 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2024-03-31T15:04:46.791 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 ::1/128 scope host 2024-03-31T15:04:46.791 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2024-03-31T15:04:46.791 INFO:teuthology.orchestra.run.smithi016.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-03-31T15:04:46.791 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:6b:fd:e0 brd ff:ff:ff:ff:ff:ff 2024-03-31T15:04:46.791 INFO:teuthology.orchestra.run.smithi016.stdout: altname enp7s0f0 2024-03-31T15:04:46.791 INFO:teuthology.orchestra.run.smithi016.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-03-31T15:04:46.791 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:6b:fd:e1 brd ff:ff:ff:ff:ff:ff 2024-03-31T15:04:46.791 INFO:teuthology.orchestra.run.smithi016.stdout: altname enp7s0f1 2024-03-31T15:04:46.791 INFO:teuthology.orchestra.run.smithi016.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-31T15:04:46.792 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:bd:15:64 brd ff:ff:ff:ff:ff:ff 2024-03-31T15:04:46.792 INFO:teuthology.orchestra.run.smithi016.stdout: altname enp1s0f0 2024-03-31T15:04:46.792 INFO:teuthology.orchestra.run.smithi016.stdout: inet 172.21.15.16/20 brd 172.21.15.255 scope global dynamic ens1f0 2024-03-31T15:04:46.792 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft 42394sec preferred_lft 42394sec 2024-03-31T15:04:46.792 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 fe80::ec4:7aff:febd:1564/64 scope link 2024-03-31T15:04:46.792 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2024-03-31T15:04:46.792 INFO:teuthology.orchestra.run.smithi016.stdout:5: ens1f1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-03-31T15:04:46.792 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:bd:15:65 brd ff:ff:ff:ff:ff:ff 2024-03-31T15:04:46.792 INFO:teuthology.orchestra.run.smithi016.stdout: altname enp1s0f1 2024-03-31T15:04:46.792 INFO:teuthology.orchestra.run.smithi016.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2024-03-31T15:04:46.792 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 02:42:d4:1f:7d:73 brd ff:ff:ff:ff:ff:ff 2024-03-31T15:04:46.792 INFO:teuthology.orchestra.run.smithi016.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2024-03-31T15:04:46.792 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2024-03-31T15:04:46.792 INFO:teuthology.orchestra.run.smithi016.stdout:7: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-31T15:04:46.792 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 5e:1f:84:6d:79:ae brd ff:ff:ff:ff:ff:ff 2024-03-31T15:04:46.792 INFO:teuthology.orchestra.run.smithi016.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-31T15:04:46.793 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2024-03-31T15:04:46.793 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 fe80::5c1f:84ff:fe6d:79ae/64 scope link 2024-03-31T15:04:46.793 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2024-03-31T15:04:46.793 INFO:teuthology.orchestra.run.smithi016.stdout:8: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-31T15:04:46.793 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0e:5c:f6:1f:82:2f brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-31T15:04:46.793 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 fe80::c5c:f6ff:fe1f:822f/64 scope link 2024-03-31T15:04:46.793 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2024-03-31T15:04:46.793 INFO:teuthology.orchestra.run.smithi016.stdout:9: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-31T15:04:46.793 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether c6:a6:3d:69:d0:db brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-31T15:04:46.793 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 fe80::c4a6:3dff:fe69:d0db/64 scope link 2024-03-31T15:04:46.793 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2024-03-31T15:04:46.794 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:04:46.794 DEBUG:teuthology.orchestra.run.smithi016:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-31T15:04:46.849 INFO:teuthology.orchestra.run.smithi016.stdout:1 2024-03-31T15:04:46.852 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:04:46.852 DEBUG:teuthology.orchestra.run.smithi016:> route 2024-03-31T15:04:46.901 INFO:teuthology.orchestra.run.smithi016.stdout:Kernel IP routing table 2024-03-31T15:04:46.901 INFO:teuthology.orchestra.run.smithi016.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-31T15:04:46.901 INFO:teuthology.orchestra.run.smithi016.stdout:default _gateway 0.0.0.0 UG 0 0 0 ens1f0 2024-03-31T15:04:46.901 INFO:teuthology.orchestra.run.smithi016.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2024-03-31T15:04:46.901 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 0 0 0 ens1f0 2024-03-31T15:04:46.901 INFO:teuthology.orchestra.run.smithi016.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-31T15:04:46.901 DEBUG:teuthology.orchestra.run.smithi016:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-31T15:04:46.901 DEBUG:teuthology.orchestra.run.smithi016:> set -e 2024-03-31T15:04:46.901 DEBUG:teuthology.orchestra.run.smithi016:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-03-31T15:04:46.902 DEBUG:teuthology.orchestra.run.smithi016:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-03-31T15:04:46.902 DEBUG:teuthology.orchestra.run.smithi016:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-03-31T15:04:46.902 DEBUG:teuthology.orchestra.run.smithi016:> ') 2024-03-31T15:04:46.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:46 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-31T15:04:47.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:47 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-31T15:04:47.039 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:04:47.039 DEBUG:teuthology.orchestra.run.smithi016:> ip netns list 2024-03-31T15:04:47.047 INFO:teuthology.orchestra.run.smithi016.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-31T15:04:47.047 INFO:teuthology.orchestra.run.smithi016.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-31T15:04:47.048 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-31T15:04:47.049 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:04:47.049 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip link set brx.0 up 2024-03-31T15:04:47.112 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-03-31T15:04:47.112 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-31T15:04:47.112 DEBUG:teuthology.orchestra.run.smithi016:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-31T15:04:47.162 INFO:teuthology.orchestra.run.smithi016.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-31T15:04:47.163 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-31T15:04:47.163 DEBUG:teuthology.orchestra.run.smithi016:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-31T15:04:47.211 DEBUG:teuthology.orchestra.run.smithi016:> sudo modprobe fuse 2024-03-31T15:04:47.272 DEBUG:teuthology.orchestra.run.smithi016:> cat /proc/self/mounts | awk '{print $2}' 2024-03-31T15:04:47.325 INFO:teuthology.orchestra.run.smithi016.stdout:/sys 2024-03-31T15:04:47.325 INFO:teuthology.orchestra.run.smithi016.stdout:/proc 2024-03-31T15:04:47.326 INFO:teuthology.orchestra.run.smithi016.stdout:/dev 2024-03-31T15:04:47.326 INFO:teuthology.orchestra.run.smithi016.stdout:/dev/pts 2024-03-31T15:04:47.326 INFO:teuthology.orchestra.run.smithi016.stdout:/run 2024-03-31T15:04:47.326 INFO:teuthology.orchestra.run.smithi016.stdout:/ 2024-03-31T15:04:47.326 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/kernel/security 2024-03-31T15:04:47.326 INFO:teuthology.orchestra.run.smithi016.stdout:/dev/shm 2024-03-31T15:04:47.326 INFO:teuthology.orchestra.run.smithi016.stdout:/run/lock 2024-03-31T15:04:47.326 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/cgroup 2024-03-31T15:04:47.326 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/pstore 2024-03-31T15:04:47.326 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/bpf 2024-03-31T15:04:47.326 INFO:teuthology.orchestra.run.smithi016.stdout:/proc/sys/fs/binfmt_misc 2024-03-31T15:04:47.326 INFO:teuthology.orchestra.run.smithi016.stdout:/dev/hugepages 2024-03-31T15:04:47.326 INFO:teuthology.orchestra.run.smithi016.stdout:/dev/mqueue 2024-03-31T15:04:47.326 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/kernel/debug 2024-03-31T15:04:47.327 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/kernel/tracing 2024-03-31T15:04:47.327 INFO:teuthology.orchestra.run.smithi016.stdout:/proc/fs/nfsd 2024-03-31T15:04:47.327 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/fuse/connections 2024-03-31T15:04:47.327 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/kernel/config 2024-03-31T15:04:47.327 INFO:teuthology.orchestra.run.smithi016.stdout:/run/credentials/systemd-sysusers.service 2024-03-31T15:04:47.327 INFO:teuthology.orchestra.run.smithi016.stdout:/run/rpc_pipefs 2024-03-31T15:04:47.327 INFO:teuthology.orchestra.run.smithi016.stdout:/run/user/1000 2024-03-31T15:04:47.327 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/ceph 2024-03-31T15:04:47.327 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/ceph/osd/ceph-0 2024-03-31T15:04:47.327 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/ceph/osd/ceph-1 2024-03-31T15:04:47.327 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/ceph/osd/ceph-2 2024-03-31T15:04:47.327 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/ceph/osd/ceph-3 2024-03-31T15:04:47.327 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns 2024-03-31T15:04:47.327 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-31T15:04:47.327 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-31T15:04:47.327 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-31T15:04:47.328 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-31T15:04:47.328 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:04:47.328 DEBUG:teuthology.orchestra.run.smithi016:> ls /sys/fs/fuse/connections 2024-03-31T15:04:47.380 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-31T15:04:47.380 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-03-31T15:04:47.423 DEBUG:teuthology.orchestra.run.smithi016:> sudo modprobe fuse 2024-03-31T15:04:47.444 DEBUG:teuthology.orchestra.run.smithi016:> cat /proc/self/mounts | awk '{print $2}' 2024-03-31T15:04:47.494 INFO:teuthology.orchestra.run.smithi016.stdout:/sys 2024-03-31T15:04:47.494 INFO:teuthology.orchestra.run.smithi016.stdout:/proc 2024-03-31T15:04:47.494 INFO:teuthology.orchestra.run.smithi016.stdout:/dev 2024-03-31T15:04:47.494 INFO:teuthology.orchestra.run.smithi016.stdout:/dev/pts 2024-03-31T15:04:47.494 INFO:teuthology.orchestra.run.smithi016.stdout:/run 2024-03-31T15:04:47.494 INFO:teuthology.orchestra.run.smithi016.stdout:/ 2024-03-31T15:04:47.494 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/kernel/security 2024-03-31T15:04:47.494 INFO:teuthology.orchestra.run.smithi016.stdout:/dev/shm 2024-03-31T15:04:47.494 INFO:teuthology.orchestra.run.smithi016.stdout:/run/lock 2024-03-31T15:04:47.494 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/cgroup 2024-03-31T15:04:47.494 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/pstore 2024-03-31T15:04:47.494 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/bpf 2024-03-31T15:04:47.495 INFO:teuthology.orchestra.run.smithi016.stdout:/proc/sys/fs/binfmt_misc 2024-03-31T15:04:47.495 INFO:teuthology.orchestra.run.smithi016.stdout:/dev/hugepages 2024-03-31T15:04:47.495 INFO:teuthology.orchestra.run.smithi016.stdout:/dev/mqueue 2024-03-31T15:04:47.495 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/kernel/debug 2024-03-31T15:04:47.495 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/kernel/tracing 2024-03-31T15:04:47.495 INFO:teuthology.orchestra.run.smithi016.stdout:/proc/fs/nfsd 2024-03-31T15:04:47.495 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/fuse/connections 2024-03-31T15:04:47.495 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/kernel/config 2024-03-31T15:04:47.495 INFO:teuthology.orchestra.run.smithi016.stdout:/run/credentials/systemd-sysusers.service 2024-03-31T15:04:47.495 INFO:teuthology.orchestra.run.smithi016.stdout:/run/rpc_pipefs 2024-03-31T15:04:47.495 INFO:teuthology.orchestra.run.smithi016.stdout:/run/user/1000 2024-03-31T15:04:47.495 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/ceph 2024-03-31T15:04:47.495 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/ceph/osd/ceph-0 2024-03-31T15:04:47.495 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/ceph/osd/ceph-1 2024-03-31T15:04:47.495 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/ceph/osd/ceph-2 2024-03-31T15:04:47.495 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/ceph/osd/ceph-3 2024-03-31T15:04:47.496 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns 2024-03-31T15:04:47.496 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-31T15:04:47.496 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-31T15:04:47.496 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-31T15:04:47.496 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-31T15:04:47.497 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:04:47.497 DEBUG:teuthology.orchestra.run.smithi016:> ls /sys/fs/fuse/connections 2024-03-31T15:04:47.521 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi016.stderr:2024-03-31T15:04:47.528+0000 7f37d72d8580 -1 init, newargv = 0x564abc45a2d0 newargc=15 2024-03-31T15:04:47.521 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi016.stderr:ceph-fuse[36341]: starting ceph client 2024-03-31T15:04:47.529 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi016.stderr:ceph-fuse[36341]: starting fuse 2024-03-31T15:04:47.530 INFO:teuthology.orchestra.run.smithi016.stdout:43 2024-03-31T15:04:47.531 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [43] 2024-03-31T15:04:47.531 DEBUG:teuthology.orchestra.run.smithi016:> sudo stdin-killer -- python3 -c ' 2024-03-31T15:04:47.531 DEBUG:teuthology.orchestra.run.smithi016:> import glob 2024-03-31T15:04:47.531 DEBUG:teuthology.orchestra.run.smithi016:> import re 2024-03-31T15:04:47.531 DEBUG:teuthology.orchestra.run.smithi016:> import os 2024-03-31T15:04:47.531 DEBUG:teuthology.orchestra.run.smithi016:> import subprocess 2024-03-31T15:04:47.531 DEBUG:teuthology.orchestra.run.smithi016:> 2024-03-31T15:04:47.531 DEBUG:teuthology.orchestra.run.smithi016:> def _find_admin_socket(client_name): 2024-03-31T15:04:47.531 DEBUG:teuthology.orchestra.run.smithi016:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-03-31T15:04:47.531 DEBUG:teuthology.orchestra.run.smithi016:> files = glob.glob(asok_path) 2024-03-31T15:04:47.531 DEBUG:teuthology.orchestra.run.smithi016:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-03-31T15:04:47.531 DEBUG:teuthology.orchestra.run.smithi016:> 2024-03-31T15:04:47.532 DEBUG:teuthology.orchestra.run.smithi016:> # Given a non-glob path, it better be there 2024-03-31T15:04:47.532 DEBUG:teuthology.orchestra.run.smithi016:> if "*" not in asok_path: 2024-03-31T15:04:47.532 DEBUG:teuthology.orchestra.run.smithi016:> assert(len(files) == 1) 2024-03-31T15:04:47.532 DEBUG:teuthology.orchestra.run.smithi016:> return files[0] 2024-03-31T15:04:47.532 DEBUG:teuthology.orchestra.run.smithi016:> 2024-03-31T15:04:47.532 DEBUG:teuthology.orchestra.run.smithi016:> for f in files: 2024-03-31T15:04:47.532 DEBUG:teuthology.orchestra.run.smithi016:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-31T15:04:47.532 DEBUG:teuthology.orchestra.run.smithi016:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-31T15:04:47.532 DEBUG:teuthology.orchestra.run.smithi016:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-31T15:04:47.532 DEBUG:teuthology.orchestra.run.smithi016:> contents = proc_f.read() 2024-03-31T15:04:47.532 DEBUG:teuthology.orchestra.run.smithi016:> if mountpoint in contents: 2024-03-31T15:04:47.532 DEBUG:teuthology.orchestra.run.smithi016:> return f 2024-03-31T15:04:47.532 DEBUG:teuthology.orchestra.run.smithi016:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-31T15:04:47.532 DEBUG:teuthology.orchestra.run.smithi016:> 2024-03-31T15:04:47.532 DEBUG:teuthology.orchestra.run.smithi016:> print(_find_admin_socket("client.0")) 2024-03-31T15:04:47.532 DEBUG:teuthology.orchestra.run.smithi016:> ' 2024-03-31T15:04:47.659 INFO:teuthology.orchestra.run.smithi016.stdout:/var/run/ceph/ceph-client.0.36341.asok 2024-03-31T15:04:47.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:47 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-31T15:04:47.670 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.36341.asok 2024-03-31T15:04:47.670 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:04:47.670 DEBUG:teuthology.orchestra.run.smithi016:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.36341.asok status 2024-03-31T15:04:47.845 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-03-31T15:04:47.846 INFO:teuthology.orchestra.run.smithi016.stdout: "metadata": { 2024-03-31T15:04:47.846 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph_sha1": "d2271579d5e735cfd6d436b8349c004f06527cb0", 2024-03-31T15:04:47.846 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph_version": "ceph version 19.0.0-2574-gd2271579 (d2271579d5e735cfd6d436b8349c004f06527cb0) squid (dev)", 2024-03-31T15:04:47.846 INFO:teuthology.orchestra.run.smithi016.stdout: "entity_id": "0", 2024-03-31T15:04:47.846 INFO:teuthology.orchestra.run.smithi016.stdout: "hostname": "smithi016", 2024-03-31T15:04:47.846 INFO:teuthology.orchestra.run.smithi016.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-03-31T15:04:47.846 INFO:teuthology.orchestra.run.smithi016.stdout: "pid": "36341", 2024-03-31T15:04:47.846 INFO:teuthology.orchestra.run.smithi016.stdout: "root": "/" 2024-03-31T15:04:47.846 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-31T15:04:47.846 INFO:teuthology.orchestra.run.smithi016.stdout: "dentry_count": 0, 2024-03-31T15:04:47.846 INFO:teuthology.orchestra.run.smithi016.stdout: "dentry_pinned_count": 0, 2024-03-31T15:04:47.846 INFO:teuthology.orchestra.run.smithi016.stdout: "id": 5412, 2024-03-31T15:04:47.846 INFO:teuthology.orchestra.run.smithi016.stdout: "inst": { 2024-03-31T15:04:47.846 INFO:teuthology.orchestra.run.smithi016.stdout: "name": { 2024-03-31T15:04:47.847 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "client", 2024-03-31T15:04:47.847 INFO:teuthology.orchestra.run.smithi016.stdout: "num": 5412 2024-03-31T15:04:47.847 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-31T15:04:47.847 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": { 2024-03-31T15:04:47.847 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-03-31T15:04:47.847 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "192.168.0.1:0", 2024-03-31T15:04:47.847 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2956929756 2024-03-31T15:04:47.847 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-31T15:04:47.847 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-31T15:04:47.847 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": { 2024-03-31T15:04:47.847 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-03-31T15:04:47.847 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "192.168.0.1:0", 2024-03-31T15:04:47.847 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2956929756 2024-03-31T15:04:47.847 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-31T15:04:47.847 INFO:teuthology.orchestra.run.smithi016.stdout: "inst_str": "client.5412 192.168.0.1:0/2956929756", 2024-03-31T15:04:47.848 INFO:teuthology.orchestra.run.smithi016.stdout: "addr_str": "192.168.0.1:0/2956929756", 2024-03-31T15:04:47.848 INFO:teuthology.orchestra.run.smithi016.stdout: "inode_count": 1, 2024-03-31T15:04:47.848 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_epoch": 16, 2024-03-31T15:04:47.848 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_epoch": 34, 2024-03-31T15:04:47.848 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_epoch_barrier": 0, 2024-03-31T15:04:47.848 INFO:teuthology.orchestra.run.smithi016.stdout: "blocklisted": false, 2024-03-31T15:04:47.848 INFO:teuthology.orchestra.run.smithi016.stdout: "fs_name": "cephfs" 2024-03-31T15:04:47.848 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-03-31T15:04:47.862 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:04:47.862 DEBUG:teuthology.orchestra.run.smithi016:> stat --file-system '--printf=%T 2024-03-31T15:04:47.862 DEBUG:teuthology.orchestra.run.smithi016:> ' -- /home/ubuntu/cephtest/mnt.0 2024-03-31T15:04:47.872 INFO:teuthology.orchestra.run.smithi016.stdout:fuseblk 2024-03-31T15:04:47.872 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-03-31T15:04:47.872 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:04:47.873 DEBUG:teuthology.orchestra.run.smithi016:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-31T15:04:47.937 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:04:47.938 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir parentdir) 2024-03-31T15:04:47.999 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:04:47.999 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir parentdir/childdir) 2024-03-31T15:04:48.052 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:04:48.052 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec touch rfile) 2024-03-31T15:04:48.108 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:04:48.108 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec touch parentdir/pfile) 2024-03-31T15:04:48.163 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:04:48.163 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec touch parentdir/childdir/cfile) 2024-03-31T15:04:48.215 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:04:48.216 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec find ./) 2024-03-31T15:04:48.277 INFO:teuthology.orchestra.run.smithi016.stdout:./ 2024-03-31T15:04:48.277 INFO:teuthology.orchestra.run.smithi016.stdout:./rfile 2024-03-31T15:04:48.277 INFO:teuthology.orchestra.run.smithi016.stdout:./parentdir 2024-03-31T15:04:48.277 INFO:teuthology.orchestra.run.smithi016.stdout:./parentdir/pfile 2024-03-31T15:04:48.277 INFO:teuthology.orchestra.run.smithi016.stdout:./parentdir/childdir 2024-03-31T15:04:48.277 INFO:teuthology.orchestra.run.smithi016.stdout:./parentdir/childdir/cfile 2024-03-31T15:04:48.278 DEBUG:teuthology.orchestra.run.smithi016:> stdin-killer -- python3 -c ' 2024-03-31T15:04:48.278 DEBUG:teuthology.orchestra.run.smithi016:> import os 2024-03-31T15:04:48.278 DEBUG:teuthology.orchestra.run.smithi016:> import stat 2024-03-31T15:04:48.278 DEBUG:teuthology.orchestra.run.smithi016:> 2024-03-31T15:04:48.278 DEBUG:teuthology.orchestra.run.smithi016:> print(os.stat("/home/ubuntu/cephtest/mnt.0/./").st_ino) 2024-03-31T15:04:48.278 DEBUG:teuthology.orchestra.run.smithi016:> ' 2024-03-31T15:04:48.388 INFO:teuthology.orchestra.run.smithi016.stdout:1 2024-03-31T15:04:48.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:48 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-31T15:04:48.399 DEBUG:teuthology.orchestra.run.smithi016:> stdin-killer -- python3 -c ' 2024-03-31T15:04:48.399 DEBUG:teuthology.orchestra.run.smithi016:> import os 2024-03-31T15:04:48.399 DEBUG:teuthology.orchestra.run.smithi016:> import stat 2024-03-31T15:04:48.399 DEBUG:teuthology.orchestra.run.smithi016:> 2024-03-31T15:04:48.399 DEBUG:teuthology.orchestra.run.smithi016:> print(os.stat("/home/ubuntu/cephtest/mnt.0/./rfile").st_ino) 2024-03-31T15:04:48.399 DEBUG:teuthology.orchestra.run.smithi016:> ' 2024-03-31T15:04:48.510 INFO:teuthology.orchestra.run.smithi016.stdout:1099511627778 2024-03-31T15:04:48.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:48 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-31T15:04:48.519 DEBUG:teuthology.orchestra.run.smithi016:> stdin-killer -- python3 -c ' 2024-03-31T15:04:48.519 DEBUG:teuthology.orchestra.run.smithi016:> import os 2024-03-31T15:04:48.519 DEBUG:teuthology.orchestra.run.smithi016:> import stat 2024-03-31T15:04:48.519 DEBUG:teuthology.orchestra.run.smithi016:> 2024-03-31T15:04:48.519 DEBUG:teuthology.orchestra.run.smithi016:> print(os.stat("/home/ubuntu/cephtest/mnt.0/./parentdir").st_ino) 2024-03-31T15:04:48.519 DEBUG:teuthology.orchestra.run.smithi016:> ' 2024-03-31T15:04:48.625 INFO:teuthology.orchestra.run.smithi016.stdout:1099511627776 2024-03-31T15:04:48.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:48 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-31T15:04:48.634 DEBUG:teuthology.orchestra.run.smithi016:> stdin-killer -- python3 -c ' 2024-03-31T15:04:48.634 DEBUG:teuthology.orchestra.run.smithi016:> import os 2024-03-31T15:04:48.634 DEBUG:teuthology.orchestra.run.smithi016:> import stat 2024-03-31T15:04:48.635 DEBUG:teuthology.orchestra.run.smithi016:> 2024-03-31T15:04:48.635 DEBUG:teuthology.orchestra.run.smithi016:> print(os.stat("/home/ubuntu/cephtest/mnt.0/./parentdir/pfile").st_ino) 2024-03-31T15:04:48.635 DEBUG:teuthology.orchestra.run.smithi016:> ' 2024-03-31T15:04:48.742 INFO:teuthology.orchestra.run.smithi016.stdout:1099511628279 2024-03-31T15:04:48.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:48 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-31T15:04:48.751 DEBUG:teuthology.orchestra.run.smithi016:> stdin-killer -- python3 -c ' 2024-03-31T15:04:48.751 DEBUG:teuthology.orchestra.run.smithi016:> import os 2024-03-31T15:04:48.751 DEBUG:teuthology.orchestra.run.smithi016:> import stat 2024-03-31T15:04:48.751 DEBUG:teuthology.orchestra.run.smithi016:> 2024-03-31T15:04:48.751 DEBUG:teuthology.orchestra.run.smithi016:> print(os.stat("/home/ubuntu/cephtest/mnt.0/./parentdir/childdir").st_ino) 2024-03-31T15:04:48.751 DEBUG:teuthology.orchestra.run.smithi016:> ' 2024-03-31T15:04:48.862 INFO:teuthology.orchestra.run.smithi016.stdout:1099511627777 2024-03-31T15:04:48.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:48 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-31T15:04:48.871 DEBUG:teuthology.orchestra.run.smithi016:> stdin-killer -- python3 -c ' 2024-03-31T15:04:48.871 DEBUG:teuthology.orchestra.run.smithi016:> import os 2024-03-31T15:04:48.872 DEBUG:teuthology.orchestra.run.smithi016:> import stat 2024-03-31T15:04:48.872 DEBUG:teuthology.orchestra.run.smithi016:> 2024-03-31T15:04:48.872 DEBUG:teuthology.orchestra.run.smithi016:> print(os.stat("/home/ubuntu/cephtest/mnt.0/./parentdir/childdir/cfile").st_ino) 2024-03-31T15:04:48.872 DEBUG:teuthology.orchestra.run.smithi016:> ' 2024-03-31T15:04:48.978 INFO:teuthology.orchestra.run.smithi016.stdout:1099511628280 2024-03-31T15:04:48.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:48 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-31T15:04:48.990 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-03-31T15:04:48.990 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-31T15:04:49.040 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-31T15:04:49.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:49.228+0000 7f08f0400640 1 Processor -- start 2024-03-31T15:04:49.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:49.228+0000 7f08f0400640 1 -- start start 2024-03-31T15:04:49.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:49.232+0000 7f08f0400640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f08e8145e40 0x7f08e80cc740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:49.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:49.232+0000 7f08f0400640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f08e80d0920 con 0x7f08e80ccc80 2024-03-31T15:04:49.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:49.232+0000 7f08f0400640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f08e80d0c00 con 0x7f08e8070030 2024-03-31T15:04:49.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:49.232+0000 7f08f0400640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f08e80d06d0 con 0x7f08e8145e40 2024-03-31T15:04:49.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:49.232+0000 7f08ed000640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f08e80ccc80 0x7f08e806f920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:38172/0 (socket says 172.21.15.16:38172) 2024-03-31T15:04:49.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:49.232+0000 7f08ed000640 1 -- 172.21.15.16:0/1418503529 learned_addr learned my addr 172.21.15.16:0/1418503529 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:04:49.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:49.232+0000 7f08eda00640 1 --2- 172.21.15.16:0/1418503529 >> v2:172.21.15.89:3300/0 conn(0x7f08e8145e40 0x7f08e80cc740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:49.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:49.232+0000 7f08e7e00640 1 -- 172.21.15.16:0/1418503529 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2802363744 0 0) 0x7f08e80d0920 con 0x7f08e80ccc80 2024-03-31T15:04:49.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:49.232+0000 7f08e7e00640 1 -- 172.21.15.16:0/1418503529 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f08d0003530 con 0x7f08e80ccc80 2024-03-31T15:04:49.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:49.232+0000 7f08eda00640 1 -- 172.21.15.16:0/1418503529 >> v1:172.21.15.89:6789/0 conn(0x7f08e8070030 legacy=0x7f08e80d2010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:49.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:49.232+0000 7f08eda00640 1 -- 172.21.15.16:0/1418503529 >> v1:172.21.15.16:6789/0 conn(0x7f08e80ccc80 legacy=0x7f08e806f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:49.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:49.232+0000 7f08e7e00640 1 -- 172.21.15.16:0/1418503529 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2271344926 0 0) 0x7f08d0003530 con 0x7f08e80ccc80 2024-03-31T15:04:49.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:49.232+0000 7f08eda00640 1 -- 172.21.15.16:0/1418503529 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08e80d1810 con 0x7f08e8145e40 2024-03-31T15:04:49.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:49.236+0000 7f08eda00640 1 --2- 172.21.15.16:0/1418503529 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f08e8145e40 0x7f08e80cc740 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f08dc0097c0 tx=0x7f08dc01c3c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a35abbd3a65a3a63 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:49.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:49.236+0000 7f08e7e00640 1 -- 172.21.15.16:0/1418503529 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08dc0b0070 con 0x7f08e8145e40 2024-03-31T15:04:49.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:49.236+0000 7f08e7e00640 1 -- 172.21.15.16:0/1418503529 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f08dc0ab400 con 0x7f08e8145e40 2024-03-31T15:04:49.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:49.236+0000 7f08e7e00640 1 -- 172.21.15.16:0/1418503529 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08d4001680 con 0x7f08e8145e40 2024-03-31T15:04:49.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:49.236+0000 7f08f0400640 1 -- 172.21.15.16:0/1418503529 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f08e8145e40 msgr2=0x7f08e80cc740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:49.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:49.236+0000 7f08f0400640 1 --2- 172.21.15.16:0/1418503529 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f08e8145e40 0x7f08e80cc740 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f08dc0097c0 tx=0x7f08dc01c3c0 comp rx=0 tx=0).stop 2024-03-31T15:04:49.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:49.236+0000 7f08f0400640 1 -- 172.21.15.16:0/1418503529 shutdown_connections 2024-03-31T15:04:49.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:49.236+0000 7f08f0400640 1 --2- 172.21.15.16:0/1418503529 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f08e8145e40 0x7f08e80cc740 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:49.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:49.236+0000 7f08f0400640 1 -- 172.21.15.16:0/1418503529 >> 172.21.15.16:0/1418503529 conn(0x7f08e8067330 msgr2=0x7f08e8067740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:04:49.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:49.236+0000 7f08f0400640 1 -- 172.21.15.16:0/1418503529 shutdown_connections 2024-03-31T15:04:49.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:49.236+0000 7f08f0400640 1 -- 172.21.15.16:0/1418503529 wait complete. 2024-03-31T15:04:49.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:49.236+0000 7f08f0400640 1 Processor -- start 2024-03-31T15:04:49.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:49.236+0000 7f08f0400640 1 -- start start 2024-03-31T15:04:49.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:49.240+0000 7f08f0400640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f08e8070030 0x7f08e815d640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:49.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:49.240+0000 7f08f0400640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f08e80ccc80 0x7f08e8156800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:49.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:49.240+0000 7f08f0400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f08e8145e40 0x7f08e815dc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:49.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:49.240+0000 7f08f0400640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f08e80b5240 con 0x7f08e8145e40 2024-03-31T15:04:49.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:49.240+0000 7f08f0400640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f08e80b50c0 con 0x7f08e8070030 2024-03-31T15:04:49.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:49.240+0000 7f08f0400640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f08e816a1a0 con 0x7f08e80ccc80 2024-03-31T15:04:49.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:49.240+0000 7f08ee400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f08e8145e40 0x7f08e815dc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:49.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:49.240+0000 7f08ee400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f08e8145e40 0x7f08e815dc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51148/0 (socket says 172.21.15.16:51148) 2024-03-31T15:04:49.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:49.240+0000 7f08ee400640 1 -- 172.21.15.16:0/3288237120 learned_addr learned my addr 172.21.15.16:0/3288237120 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:04:49.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:49.240+0000 7f08ed000640 1 --2- 172.21.15.16:0/3288237120 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f08e80ccc80 0x7f08e8156800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:49.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:49.240+0000 7f08eda00640 1 --2- 172.21.15.16:0/3288237120 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f08e8070030 0x7f08e815d640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:49.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:49.240+0000 7f08ee400640 1 -- 172.21.15.16:0/3288237120 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f08e80ccc80 msgr2=0x7f08e8156800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:49.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:49.240+0000 7f08ee400640 1 --2- 172.21.15.16:0/3288237120 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f08e80ccc80 0x7f08e8156800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:49.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:49.240+0000 7f08ee400640 1 -- 172.21.15.16:0/3288237120 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f08e8070030 msgr2=0x7f08e815d640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:49.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:49.240+0000 7f08ee400640 1 --2- 172.21.15.16:0/3288237120 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f08e8070030 0x7f08e815d640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:49.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:49.240+0000 7f08ee400640 1 -- 172.21.15.16:0/3288237120 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08e815e160 con 0x7f08e8145e40 2024-03-31T15:04:49.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:49.240+0000 7f08eda00640 1 --2- 172.21.15.16:0/3288237120 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f08e8070030 0x7f08e815d640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:04:49.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:49.240+0000 7f08ed000640 1 --2- 172.21.15.16:0/3288237120 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f08e80ccc80 0x7f08e8156800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:04:49.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:49.240+0000 7f08ee400640 1 --2- 172.21.15.16:0/3288237120 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f08e8145e40 0x7f08e815dc20 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f08e000c700 tx=0x7f08e000cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:49.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:49.240+0000 7f08e5600640 1 -- 172.21.15.16:0/3288237120 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08e0003b80 con 0x7f08e8145e40 2024-03-31T15:04:49.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:49.240+0000 7f08e5600640 1 -- 172.21.15.16:0/3288237120 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f08e0003d20 con 0x7f08e8145e40 2024-03-31T15:04:49.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:49.240+0000 7f08f0400640 1 -- 172.21.15.16:0/3288237120 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f08e815e400 con 0x7f08e8145e40 2024-03-31T15:04:49.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:49.240+0000 7f08e5600640 1 -- 172.21.15.16:0/3288237120 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08e000aa90 con 0x7f08e8145e40 2024-03-31T15:04:49.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:49.240+0000 7f08f0400640 1 -- 172.21.15.16:0/3288237120 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f08e815e9c0 con 0x7f08e8145e40 2024-03-31T15:04:49.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:49.240+0000 7f08f0400640 1 -- 172.21.15.16:0/3288237120 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f08e8069950 con 0x7f08e8145e40 2024-03-31T15:04:49.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:49.240+0000 7f08e5600640 1 -- 172.21.15.16:0/3288237120 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f08e000ac30 con 0x7f08e8145e40 2024-03-31T15:04:49.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:49.244+0000 7f08e5600640 1 --2- 172.21.15.16:0/3288237120 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f08c80683b0 0x7f08c806a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:49.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:49.244+0000 7f08e5600640 1 -- 172.21.15.16:0/3288237120 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5709+0+0 (secure 0 0 0) 0x7f08e00868a0 con 0x7f08e8145e40 2024-03-31T15:04:49.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:49.244+0000 7f08eda00640 1 --2- 172.21.15.16:0/3288237120 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f08c80683b0 0x7f08c806a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:49.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:49.244+0000 7f08eda00640 1 --2- 172.21.15.16:0/3288237120 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f08c80683b0 0x7f08c806a870 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f08dc0ad000 tx=0x7f08dc008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:49.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:49.248+0000 7f08e5600640 1 -- 172.21.15.16:0/3288237120 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f08e0056dc0 con 0x7f08e8145e40 2024-03-31T15:04:49.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:49.464+0000 7f08f0400640 1 -- 172.21.15.16:0/3288237120 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f08e80cf3c0 con 0x7f08e8145e40 2024-03-31T15:04:49.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:49.468+0000 7f08e5600640 1 -- 172.21.15.16:0/3288237120 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v34) ==== 74+0+16597 (secure 0 0 0) 0x7f08e00566c0 con 0x7f08e8145e40 2024-03-31T15:04:49.462 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:04:49.462 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":34,"fsid":"286eb5da-fb47-4550-b290-d9519c13b9a5","created":"2024-03-31T15:03:11.438952+0000","modified":"2024-03-31T15:04:41.385382+0000","last_up_change":"2024-03-31T15:03:22.885535+0000","last_in_change":"2024-03-31T15:03:13.941667+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-31T15:03:22.550787+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-03-31T15:04:37.577341+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-03-31T15:04:38.738566+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"34","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"92a66450-ef19-40d8-ac3d-d760b23b0bda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6801","nonce":932712244}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6803","nonce":932712244}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6807","nonce":932712244}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6805","nonce":932712244}]},"public_addr":"172.21.15.16:6801/932712244","cluster_addr":"172.21.15.16:6803/932712244","heartbeat_back_addr":"172.21.15.16:6807/932712244","heartbeat_front_addr":"172.21.15.16:6805/932712244","state":["exists","up"]},{"osd":1,"uuid":"da9d1f2b-1968-4959-b843-faa231988e92","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6810","nonce":1398737533}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6811","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6813","nonce":1398737533}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6821","nonce":1398737533}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6815","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6816","nonce":1398737533}]},"public_addr":"172.21.15.16:6810/1398737533","cluster_addr":"172.21.15.16:6813/1398737533","heartbeat_back_addr":"172.21.15.16:6821/1398737533","heartbeat_front_addr":"172.21.15.16:6816/1398737533","state":["exists","up"]},{"osd":2,"uuid":"01a73662-9a21-4b1f-bcbb-42b9636a4638","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6819","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6822","nonce":3018526646}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6825","nonce":3018526646}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6831","nonce":3018526646}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6827","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6829","nonce":3018526646}]},"public_addr":"172.21.15.16:6822/3018526646","cluster_addr":"172.21.15.16:6825/3018526646","heartbeat_back_addr":"172.21.15.16:6831/3018526646","heartbeat_front_addr":"172.21.15.16:6829/3018526646","state":["exists","up"]},{"osd":3,"uuid":"c7771fd3-e18e-4500-ac38-23cf10ebcd75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6809","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6812","nonce":3360352974}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6817","nonce":3360352974}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6828","nonce":3360352974}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6823","nonce":3360352974}]},"public_addr":"172.21.15.16:6812/3360352974","cluster_addr":"172.21.15.16:6817/3360352974","heartbeat_back_addr":"172.21.15.16:6828/3360352974","heartbeat_front_addr":"172.21.15.16:6823/3360352974","state":["exists","up"]},{"osd":4,"uuid":"88260730-1482-4fbd-a40f-cb1124e9df4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6801","nonce":1856844856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6803","nonce":1856844856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6807","nonce":1856844856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6805","nonce":1856844856}]},"public_addr":"172.21.15.89:6801/1856844856","cluster_addr":"172.21.15.89:6803/1856844856","heartbeat_back_addr":"172.21.15.89:6807/1856844856","heartbeat_front_addr":"172.21.15.89:6805/1856844856","state":["exists","up"]},{"osd":5,"uuid":"4cb2c00c-fae0-4dd8-9d27-ec4f0093ccf7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6814","nonce":50395527}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6817","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6820","nonce":50395527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6831","nonce":50395527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6823","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6826","nonce":50395527}]},"public_addr":"172.21.15.89:6814/50395527","cluster_addr":"172.21.15.89:6820/50395527","heartbeat_back_addr":"172.21.15.89:6831/50395527","heartbeat_front_addr":"172.21.15.89:6826/50395527","state":["exists","up"]},{"osd":6,"uuid":"81ce0c19-2554-4cae-acbe-66a05e9e9b9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6809","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6812","nonce":3470471752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6815","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6818","nonce":3470471752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6827","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6829","nonce":3470471752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6821","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6824","nonce":3470471752}]},"public_addr":"172.21.15.89:6812/3470471752","cluster_addr":"172.21.15.89:6818/3470471752","heartbeat_back_addr":"172.21.15.89:6829/3470471752","heartbeat_front_addr":"172.21.15.89:6824/3470471752","state":["exists","up"]},{"osd":7,"uuid":"85e4313b-d37d-442e-8042-153bf20a847d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6811","nonce":1796778630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6813","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6816","nonce":1796778630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6825","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6828","nonce":1796778630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6819","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6822","nonce":1796778630}]},"public_addr":"172.21.15.89:6811/1796778630","cluster_addr":"172.21.15.89:6816/1796778630","heartbeat_back_addr":"172.21.15.89:6828/1796778630","heartbeat_front_addr":"172.21.15.89:6822/1796778630","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.439088+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.529970+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.680512+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:21.044161+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.582111+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.536558+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.409522+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.590229+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-31T15:04:49.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:49.468+0000 7f08f0400640 1 -- 172.21.15.16:0/3288237120 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f08c80683b0 msgr2=0x7f08c806a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:49.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:49.468+0000 7f08f0400640 1 --2- 172.21.15.16:0/3288237120 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f08c80683b0 0x7f08c806a870 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f08dc0ad000 tx=0x7f08dc008040 comp rx=0 tx=0).stop 2024-03-31T15:04:49.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:49.468+0000 7f08f0400640 1 -- 172.21.15.16:0/3288237120 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f08e8145e40 msgr2=0x7f08e815dc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:49.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:49.468+0000 7f08f0400640 1 --2- 172.21.15.16:0/3288237120 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f08e8145e40 0x7f08e815dc20 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f08e000c700 tx=0x7f08e000cbc0 comp rx=0 tx=0).stop 2024-03-31T15:04:49.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:49.468+0000 7f08f0400640 1 -- 172.21.15.16:0/3288237120 shutdown_connections 2024-03-31T15:04:49.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:49.468+0000 7f08f0400640 1 --2- 172.21.15.16:0/3288237120 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f08c80683b0 0x7f08c806a870 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:49.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:49.468+0000 7f08f0400640 1 --2- 172.21.15.16:0/3288237120 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f08e8145e40 0x7f08e815dc20 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:49.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:49.468+0000 7f08f0400640 1 --2- 172.21.15.16:0/3288237120 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f08e80ccc80 0x7f08e8156800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:49.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:49.468+0000 7f08f0400640 1 --2- 172.21.15.16:0/3288237120 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f08e8070030 0x7f08e815d640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:49.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:49.468+0000 7f08f0400640 1 -- 172.21.15.16:0/3288237120 >> 172.21.15.16:0/3288237120 conn(0x7f08e8067330 msgr2=0x7f08e80b3da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:04:49.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:49.468+0000 7f08f0400640 1 -- 172.21.15.16:0/3288237120 shutdown_connections 2024-03-31T15:04:49.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:49.468+0000 7f08f0400640 1 -- 172.21.15.16:0/3288237120 wait complete. 2024-03-31T15:04:49.490 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-31T15:03:22.550787+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-31T15:04:37.577341+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '33', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_data', 'create_time': '2024-03-31T15:04:38.738566+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '34', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-31T15:04:49.490 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-31T15:04:49.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:49.640+0000 7fcc7ca00640 1 Processor -- start 2024-03-31T15:04:49.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:49.640+0000 7fcc7ca00640 1 -- start start 2024-03-31T15:04:49.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:49.640+0000 7fcc7ca00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fcc78161e70 0x7fcc780d00a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:49.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:49.640+0000 7fcc7ca00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fcc7806ecf0 0x7fcc7806f140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:49.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:49.640+0000 7fcc7ca00640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcc78162410 con 0x7fcc7806c960 2024-03-31T15:04:49.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:49.640+0000 7fcc7ca00640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7fcc781626f0 con 0x7fcc78161e70 2024-03-31T15:04:49.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:49.640+0000 7fcc7ca00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fcc781629d0 con 0x7fcc7806ecf0 2024-03-31T15:04:49.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:49.640+0000 7fcc77400640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fcc7806c960 0x7fcc7806cd40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:38188/0 (socket says 172.21.15.16:38188) 2024-03-31T15:04:49.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:49.640+0000 7fcc77400640 1 -- 172.21.15.16:0/1000196689 learned_addr learned my addr 172.21.15.16:0/1000196689 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:04:49.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:49.640+0000 7fcc76a00640 1 --2- 172.21.15.16:0/1000196689 >> v2:172.21.15.89:3300/0 conn(0x7fcc78161e70 0x7fcc780d00a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:49.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:49.640+0000 7fcc77e00640 1 --2- 172.21.15.16:0/1000196689 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fcc7806ecf0 0x7fcc7806f140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:49.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:49.640+0000 7fcc76000640 1 -- 172.21.15.16:0/1000196689 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3226579766 0 0) 0x7fcc78162410 con 0x7fcc7806c960 2024-03-31T15:04:49.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:49.640+0000 7fcc76000640 1 -- 172.21.15.16:0/1000196689 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcc5c003530 con 0x7fcc7806c960 2024-03-31T15:04:49.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:49.640+0000 7fcc76a00640 1 -- 172.21.15.16:0/1000196689 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fcc7806ecf0 msgr2=0x7fcc7806f140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:04:49.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:49.640+0000 7fcc76a00640 1 --2- 172.21.15.16:0/1000196689 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fcc7806ecf0 0x7fcc7806f140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:49.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:49.640+0000 7fcc76a00640 1 -- 172.21.15.16:0/1000196689 >> v1:172.21.15.16:6789/0 conn(0x7fcc7806c960 legacy=0x7fcc7806cd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:49.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:49.640+0000 7fcc76a00640 1 -- 172.21.15.16:0/1000196689 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc78162cb0 con 0x7fcc78161e70 2024-03-31T15:04:49.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:49.640+0000 7fcc77e00640 1 --2- 172.21.15.16:0/1000196689 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fcc7806ecf0 0x7fcc7806f140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:04:49.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:49.644+0000 7fcc76a00640 1 --2- 172.21.15.16:0/1000196689 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcc78161e70 0x7fcc780d00a0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fcc640097c0 tx=0x7fcc6401c3e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fa00037fc5cf5593 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:49.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:49.644+0000 7fcc76000640 1 -- 172.21.15.16:0/1000196689 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc640b0070 con 0x7fcc78161e70 2024-03-31T15:04:49.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:49.644+0000 7fcc76000640 1 -- 172.21.15.16:0/1000196689 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcc640ab600 con 0x7fcc78161e70 2024-03-31T15:04:49.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:49.644+0000 7fcc76000640 1 -- 172.21.15.16:0/1000196689 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc640ab9d0 con 0x7fcc78161e70 2024-03-31T15:04:49.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:49.644+0000 7fcc7ca00640 1 -- 172.21.15.16:0/1000196689 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcc78161e70 msgr2=0x7fcc780d00a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:49.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:49.644+0000 7fcc7ca00640 1 --2- 172.21.15.16:0/1000196689 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcc78161e70 0x7fcc780d00a0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fcc640097c0 tx=0x7fcc6401c3e0 comp rx=0 tx=0).stop 2024-03-31T15:04:49.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:49.644+0000 7fcc7ca00640 1 -- 172.21.15.16:0/1000196689 shutdown_connections 2024-03-31T15:04:49.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:49.644+0000 7fcc7ca00640 1 --2- 172.21.15.16:0/1000196689 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fcc7806ecf0 0x7fcc7806f140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:49.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:49.644+0000 7fcc7ca00640 1 --2- 172.21.15.16:0/1000196689 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcc78161e70 0x7fcc780d00a0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:49.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:49.644+0000 7fcc7ca00640 1 -- 172.21.15.16:0/1000196689 >> 172.21.15.16:0/1000196689 conn(0x7fcc780b2c90 msgr2=0x7fcc780b30a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:04:49.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:49.644+0000 7fcc7ca00640 1 -- 172.21.15.16:0/1000196689 shutdown_connections 2024-03-31T15:04:49.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:49.644+0000 7fcc7ca00640 1 -- 172.21.15.16:0/1000196689 wait complete. 2024-03-31T15:04:49.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:49.648+0000 7fcc7ca00640 1 Processor -- start 2024-03-31T15:04:49.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:49.648+0000 7fcc7ca00640 1 -- start start 2024-03-31T15:04:49.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:49.648+0000 7fcc7ca00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcc7806c960 0x7fcc78161200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:49.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:49.648+0000 7fcc7ca00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcc7806ecf0 0x7fcc781689c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:49.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:49.648+0000 7fcc7ca00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fcc78161e70 0x7fcc780aebf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:49.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:49.648+0000 7fcc7ca00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fcc780732a0 con 0x7fcc7806ecf0 2024-03-31T15:04:49.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:49.648+0000 7fcc7ca00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fcc78073120 con 0x7fcc7806c960 2024-03-31T15:04:49.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:49.648+0000 7fcc7ca00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fcc78160d70 con 0x7fcc78161e70 2024-03-31T15:04:49.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:49.648+0000 7fcc76a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcc7806ecf0 0x7fcc781689c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:49.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:49.648+0000 7fcc76a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcc7806ecf0 0x7fcc781689c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51160/0 (socket says 172.21.15.16:51160) 2024-03-31T15:04:49.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:49.648+0000 7fcc76a00640 1 -- 172.21.15.16:0/1872124575 learned_addr learned my addr 172.21.15.16:0/1872124575 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:04:49.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:49.648+0000 7fcc77400640 1 --2- 172.21.15.16:0/1872124575 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcc7806c960 0x7fcc78161200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:49.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:49.648+0000 7fcc77e00640 1 --2- 172.21.15.16:0/1872124575 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fcc78161e70 0x7fcc780aebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:49.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:49.648+0000 7fcc76a00640 1 -- 172.21.15.16:0/1872124575 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fcc78161e70 msgr2=0x7fcc780aebf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:49.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:49.648+0000 7fcc76a00640 1 --2- 172.21.15.16:0/1872124575 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fcc78161e70 0x7fcc780aebf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:49.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:49.648+0000 7fcc76a00640 1 -- 172.21.15.16:0/1872124575 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcc7806c960 msgr2=0x7fcc78161200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:49.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:49.648+0000 7fcc76a00640 1 --2- 172.21.15.16:0/1872124575 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcc7806c960 0x7fcc78161200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:49.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:49.648+0000 7fcc76a00640 1 -- 172.21.15.16:0/1872124575 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc7815a000 con 0x7fcc7806ecf0 2024-03-31T15:04:49.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:49.648+0000 7fcc76a00640 1 --2- 172.21.15.16:0/1872124575 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcc7806ecf0 0x7fcc781689c0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fcc6401aa90 tx=0x7fcc64019180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:49.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:49.648+0000 7fcc6f400640 1 -- 172.21.15.16:0/1872124575 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc640ba020 con 0x7fcc7806ecf0 2024-03-31T15:04:49.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:49.648+0000 7fcc6f400640 1 -- 172.21.15.16:0/1872124575 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcc640b5460 con 0x7fcc7806ecf0 2024-03-31T15:04:49.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:49.648+0000 7fcc7ca00640 1 -- 172.21.15.16:0/1872124575 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcc7815a2e0 con 0x7fcc7806ecf0 2024-03-31T15:04:49.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:49.648+0000 7fcc6f400640 1 -- 172.21.15.16:0/1872124575 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc640b0040 con 0x7fcc7806ecf0 2024-03-31T15:04:49.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:49.648+0000 7fcc7ca00640 1 -- 172.21.15.16:0/1872124575 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcc78161740 con 0x7fcc7806ecf0 2024-03-31T15:04:49.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:49.652+0000 7fcc7ca00640 1 -- 172.21.15.16:0/1872124575 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcc7804f8c0 con 0x7fcc7806ecf0 2024-03-31T15:04:49.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:49.652+0000 7fcc6f400640 1 -- 172.21.15.16:0/1872124575 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fcc640bd020 con 0x7fcc7806ecf0 2024-03-31T15:04:49.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:49.652+0000 7fcc6f400640 1 --2- 172.21.15.16:0/1872124575 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fcc58068380 0x7fcc5806a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:49.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:49.652+0000 7fcc6f400640 1 -- 172.21.15.16:0/1872124575 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5709+0+0 (secure 0 0 0) 0x7fcc64120cb0 con 0x7fcc7806ecf0 2024-03-31T15:04:49.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:49.652+0000 7fcc77400640 1 --2- 172.21.15.16:0/1872124575 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fcc58068380 0x7fcc5806a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:49.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:49.652+0000 7fcc77400640 1 --2- 172.21.15.16:0/1872124575 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fcc58068380 0x7fcc5806a840 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fcc68005890 tx=0x7fcc68009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:49.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:49.656+0000 7fcc6f400640 1 -- 172.21.15.16:0/1872124575 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fcc640c4030 con 0x7fcc7806ecf0 2024-03-31T15:04:49.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:49.864+0000 7fcc7ca00640 1 -- 172.21.15.16:0/1872124575 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fcc780ce410 con 0x7fcc7806ecf0 2024-03-31T15:04:49.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:49.864+0000 7fcc6f400640 1 -- 172.21.15.16:0/1872124575 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v34) ==== 93+0+10 (secure 0 0 0) 0x7fcc64016090 con 0x7fcc7806ecf0 2024-03-31T15:04:49.858 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2024-03-31T15:04:49.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:49.864+0000 7fcc7ca00640 1 -- 172.21.15.16:0/1872124575 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fcc58068380 msgr2=0x7fcc5806a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:49.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:49.864+0000 7fcc7ca00640 1 --2- 172.21.15.16:0/1872124575 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fcc58068380 0x7fcc5806a840 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fcc68005890 tx=0x7fcc68009040 comp rx=0 tx=0).stop 2024-03-31T15:04:49.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:49.864+0000 7fcc7ca00640 1 -- 172.21.15.16:0/1872124575 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcc7806ecf0 msgr2=0x7fcc781689c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:49.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:49.864+0000 7fcc7ca00640 1 --2- 172.21.15.16:0/1872124575 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcc7806ecf0 0x7fcc781689c0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fcc6401aa90 tx=0x7fcc64019180 comp rx=0 tx=0).stop 2024-03-31T15:04:49.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:49.864+0000 7fcc7ca00640 1 -- 172.21.15.16:0/1872124575 shutdown_connections 2024-03-31T15:04:49.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:49.864+0000 7fcc7ca00640 1 --2- 172.21.15.16:0/1872124575 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fcc58068380 0x7fcc5806a840 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:49.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:49.864+0000 7fcc7ca00640 1 --2- 172.21.15.16:0/1872124575 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fcc78161e70 0x7fcc780aebf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:49.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:49.864+0000 7fcc7ca00640 1 --2- 172.21.15.16:0/1872124575 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcc7806ecf0 0x7fcc781689c0 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:49.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:49.864+0000 7fcc7ca00640 1 --2- 172.21.15.16:0/1872124575 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcc7806c960 0x7fcc78161200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:49.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:49.864+0000 7fcc7ca00640 1 -- 172.21.15.16:0/1872124575 >> 172.21.15.16:0/1872124575 conn(0x7fcc780b2c90 msgr2=0x7fcc780b2190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:04:49.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:49.864+0000 7fcc7ca00640 1 -- 172.21.15.16:0/1872124575 shutdown_connections 2024-03-31T15:04:49.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:49.864+0000 7fcc7ca00640 1 -- 172.21.15.16:0/1872124575 wait complete. 2024-03-31T15:04:49.879 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-31T15:04:50.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:50.028+0000 7f5de6400640 1 Processor -- start 2024-03-31T15:04:50.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:50.028+0000 7f5de6400640 1 -- start start 2024-03-31T15:04:50.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:50.028+0000 7f5de6400640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f5de0070030 0x7f5de0143c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:50.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:50.028+0000 7f5de6400640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f5de0141590 0x7f5de0144170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:50.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:50.028+0000 7f5de6400640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5de01446b0 con 0x7f5de01411b0 2024-03-31T15:04:50.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:50.028+0000 7f5de6400640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f5de015f910 con 0x7f5de0070030 2024-03-31T15:04:50.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:50.028+0000 7f5de6400640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f5de0162910 con 0x7f5de0141590 2024-03-31T15:04:50.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:50.032+0000 7f5dd7e00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f5de0070030 0x7f5de0143c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:50.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:50.032+0000 7f5dd7e00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f5de0070030 0x7f5de0143c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51174/0 (socket says 172.21.15.16:51174) 2024-03-31T15:04:50.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:50.032+0000 7f5dd7e00640 1 -- 172.21.15.16:0/443749130 learned_addr learned my addr 172.21.15.16:0/443749130 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:04:50.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:50.032+0000 7f5ddf400640 1 --1- 172.21.15.16:0/443749130 >> v1:172.21.15.16:6789/0 conn(0x7f5de01411b0 0x7f5de006f920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:38190/0 (socket says 172.21.15.16:38190) 2024-03-31T15:04:50.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:50.032+0000 7f5ddfe00640 1 --2- 172.21.15.16:0/443749130 >> v2:172.21.15.89:3300/0 conn(0x7f5de0141590 0x7f5de0144170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:50.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:50.032+0000 7f5dd7e00640 1 -- 172.21.15.16:0/443749130 >> v2:172.21.15.89:3300/0 conn(0x7f5de0141590 msgr2=0x7f5de0144170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:04:50.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:50.032+0000 7f5dd7e00640 1 --2- 172.21.15.16:0/443749130 >> v2:172.21.15.89:3300/0 conn(0x7f5de0141590 0x7f5de0144170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:50.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:50.032+0000 7f5dd7e00640 1 -- 172.21.15.16:0/443749130 >> v1:172.21.15.16:6789/0 conn(0x7f5de01411b0 legacy=0x7f5de006f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:50.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:50.032+0000 7f5dd7e00640 1 -- 172.21.15.16:0/443749130 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5de0144980 con 0x7f5de0070030 2024-03-31T15:04:50.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:50.032+0000 7f5ddea00640 1 -- 172.21.15.16:0/443749130 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4140900047 0 0) 0x7f5de01446b0 con 0x7f5de01411b0 2024-03-31T15:04:50.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:50.032+0000 7f5dd7e00640 1 --2- 172.21.15.16:0/443749130 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5de0070030 0x7f5de0143c30 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f5dc8009870 tx=0x7f5dc801c4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8066d16fbb21e32 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:50.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:50.032+0000 7f5ddea00640 1 -- 172.21.15.16:0/443749130 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5dc80aa650 con 0x7f5de0070030 2024-03-31T15:04:50.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:50.032+0000 7f5ddea00640 1 -- 172.21.15.16:0/443749130 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5dc80af070 con 0x7f5de0070030 2024-03-31T15:04:50.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:50.032+0000 7f5ddea00640 1 -- 172.21.15.16:0/443749130 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5dc80aaa10 con 0x7f5de0070030 2024-03-31T15:04:50.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:50.036+0000 7f5de6400640 1 -- 172.21.15.16:0/443749130 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5de0070030 msgr2=0x7f5de0143c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:50.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:50.036+0000 7f5de6400640 1 --2- 172.21.15.16:0/443749130 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5de0070030 0x7f5de0143c30 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f5dc8009870 tx=0x7f5dc801c4f0 comp rx=0 tx=0).stop 2024-03-31T15:04:50.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:50.036+0000 7f5de6400640 1 -- 172.21.15.16:0/443749130 shutdown_connections 2024-03-31T15:04:50.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:50.036+0000 7f5de6400640 1 --2- 172.21.15.16:0/443749130 >> v2:172.21.15.89:3300/0 conn(0x7f5de0141590 0x7f5de0144170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:50.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:50.036+0000 7f5de6400640 1 --2- 172.21.15.16:0/443749130 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5de0070030 0x7f5de0143c30 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:50.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:50.036+0000 7f5de6400640 1 -- 172.21.15.16:0/443749130 >> 172.21.15.16:0/443749130 conn(0x7f5de0067930 msgr2=0x7f5de00b3ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:04:50.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:50.036+0000 7f5de6400640 1 -- 172.21.15.16:0/443749130 shutdown_connections 2024-03-31T15:04:50.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:50.036+0000 7f5de6400640 1 -- 172.21.15.16:0/443749130 wait complete. 2024-03-31T15:04:50.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:50.036+0000 7f5de6400640 1 Processor -- start 2024-03-31T15:04:50.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:50.036+0000 7f5de6400640 1 -- start start 2024-03-31T15:04:50.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:50.036+0000 7f5de6400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5de0070030 0x7f5de01551b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:50.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:50.036+0000 7f5de6400640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5de01411b0 0x7f5de013e040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:50.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:50.036+0000 7f5de6400640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f5de0141590 0x7f5de013e580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:50.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:50.036+0000 7f5de6400640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f5de015f110 con 0x7f5de0070030 2024-03-31T15:04:50.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:50.036+0000 7f5de6400640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f5de0162da0 con 0x7f5de01411b0 2024-03-31T15:04:50.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:50.036+0000 7f5de6400640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f5de015f290 con 0x7f5de0141590 2024-03-31T15:04:50.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:50.036+0000 7f5ddf400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5de0070030 0x7f5de01551b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:50.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:50.036+0000 7f5ddf400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5de0070030 0x7f5de01551b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51190/0 (socket says 172.21.15.16:51190) 2024-03-31T15:04:50.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:50.036+0000 7f5ddf400640 1 -- 172.21.15.16:0/1699437961 learned_addr learned my addr 172.21.15.16:0/1699437961 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:04:50.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:50.036+0000 7f5dd7e00640 1 --2- 172.21.15.16:0/1699437961 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5de01411b0 0x7f5de013e040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:50.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:50.036+0000 7f5ddfe00640 1 --2- 172.21.15.16:0/1699437961 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f5de0141590 0x7f5de013e580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:50.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:50.040+0000 7f5ddf400640 1 -- 172.21.15.16:0/1699437961 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f5de0141590 msgr2=0x7f5de013e580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:50.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:50.040+0000 7f5ddf400640 1 --2- 172.21.15.16:0/1699437961 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f5de0141590 0x7f5de013e580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:50.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:50.040+0000 7f5ddf400640 1 -- 172.21.15.16:0/1699437961 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5de01411b0 msgr2=0x7f5de013e040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:50.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:50.040+0000 7f5ddf400640 1 --2- 172.21.15.16:0/1699437961 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5de01411b0 0x7f5de013e040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:50.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:50.040+0000 7f5ddf400640 1 -- 172.21.15.16:0/1699437961 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5de013ec00 con 0x7f5de0070030 2024-03-31T15:04:50.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:50.040+0000 7f5ddfe00640 1 --2- 172.21.15.16:0/1699437961 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f5de0141590 0x7f5de013e580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:04:50.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:50.040+0000 7f5ddf400640 1 --2- 172.21.15.16:0/1699437961 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5de0070030 0x7f5de01551b0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f5dd00011f0 tx=0x7f5dd000c9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:50.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:50.040+0000 7f5dd7400640 1 -- 172.21.15.16:0/1699437961 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5dd0018070 con 0x7f5de0070030 2024-03-31T15:04:50.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:50.040+0000 7f5dd7400640 1 -- 172.21.15.16:0/1699437961 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5dd00074e0 con 0x7f5de0070030 2024-03-31T15:04:50.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:50.040+0000 7f5dd7400640 1 -- 172.21.15.16:0/1699437961 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5dd0004a10 con 0x7f5de0070030 2024-03-31T15:04:50.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:50.040+0000 7f5de6400640 1 -- 172.21.15.16:0/1699437961 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5de015aac0 con 0x7f5de0070030 2024-03-31T15:04:50.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:50.040+0000 7f5de6400640 1 -- 172.21.15.16:0/1699437961 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5de015b080 con 0x7f5de0070030 2024-03-31T15:04:50.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:50.040+0000 7f5de6400640 1 -- 172.21.15.16:0/1699437961 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5de0069950 con 0x7f5de0070030 2024-03-31T15:04:50.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:50.040+0000 7f5dd7400640 1 -- 172.21.15.16:0/1699437961 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f5dd0004bb0 con 0x7f5de0070030 2024-03-31T15:04:50.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:50.040+0000 7f5dd7400640 1 --2- 172.21.15.16:0/1699437961 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f5dbc068380 0x7f5dbc06a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:50.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:50.040+0000 7f5dd7400640 1 -- 172.21.15.16:0/1699437961 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5709+0+0 (secure 0 0 0) 0x7f5dd0085cf0 con 0x7f5de0070030 2024-03-31T15:04:50.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:50.044+0000 7f5dd7e00640 1 --2- 172.21.15.16:0/1699437961 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f5dbc068380 0x7f5dbc06a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:50.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:50.044+0000 7f5dd7e00640 1 --2- 172.21.15.16:0/1699437961 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f5dbc068380 0x7f5dbc06a840 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f5dc80ae000 tx=0x7f5dc80a7000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:50.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:50.044+0000 7f5dd7400640 1 -- 172.21.15.16:0/1699437961 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f5dd0055c50 con 0x7f5de0070030 2024-03-31T15:04:50.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:50.240+0000 7f5de6400640 1 -- 172.21.15.16:0/1699437961 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f5de00b4a20 con 0x7f5de0070030 2024-03-31T15:04:50.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:50.240+0000 7f5dd7400640 1 -- 172.21.15.16:0/1699437961 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v34) ==== 104+0+11 (secure 0 0 0) 0x7f5dd0059f50 con 0x7f5de0070030 2024-03-31T15:04:50.235 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2024-03-31T15:04:50.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:50.244+0000 7f5de6400640 1 -- 172.21.15.16:0/1699437961 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f5dbc068380 msgr2=0x7f5dbc06a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:50.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:50.244+0000 7f5de6400640 1 --2- 172.21.15.16:0/1699437961 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f5dbc068380 0x7f5dbc06a840 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f5dc80ae000 tx=0x7f5dc80a7000 comp rx=0 tx=0).stop 2024-03-31T15:04:50.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:50.244+0000 7f5de6400640 1 -- 172.21.15.16:0/1699437961 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5de0070030 msgr2=0x7f5de01551b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:50.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:50.244+0000 7f5de6400640 1 --2- 172.21.15.16:0/1699437961 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5de0070030 0x7f5de01551b0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f5dd00011f0 tx=0x7f5dd000c9b0 comp rx=0 tx=0).stop 2024-03-31T15:04:50.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:50.244+0000 7f5de6400640 1 -- 172.21.15.16:0/1699437961 shutdown_connections 2024-03-31T15:04:50.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:50.244+0000 7f5de6400640 1 --2- 172.21.15.16:0/1699437961 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f5dbc068380 0x7f5dbc06a840 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:50.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:50.244+0000 7f5de6400640 1 --2- 172.21.15.16:0/1699437961 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f5de0141590 0x7f5de013e580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:50.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:50.244+0000 7f5de6400640 1 --2- 172.21.15.16:0/1699437961 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5de01411b0 0x7f5de013e040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:50.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:50.244+0000 7f5de6400640 1 --2- 172.21.15.16:0/1699437961 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5de0070030 0x7f5de01551b0 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:50.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:50.244+0000 7f5de6400640 1 -- 172.21.15.16:0/1699437961 >> 172.21.15.16:0/1699437961 conn(0x7f5de0067930 msgr2=0x7f5de015fb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:04:50.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:50.244+0000 7f5de6400640 1 -- 172.21.15.16:0/1699437961 shutdown_connections 2024-03-31T15:04:50.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:50.244+0000 7f5de6400640 1 -- 172.21.15.16:0/1699437961 wait complete. 2024-03-31T15:04:50.257 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-31T15:04:50.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:50.404+0000 7f30d2e00640 1 Processor -- start 2024-03-31T15:04:50.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:50.404+0000 7f30d2e00640 1 -- start start 2024-03-31T15:04:50.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:50.408+0000 7f30d2e00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f30cc070030 0x7f30cc143c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:50.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:50.408+0000 7f30d2e00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f30cc141590 0x7f30cc144170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:50.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:50.408+0000 7f30d2e00640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f30cc1446b0 con 0x7f30cc1411b0 2024-03-31T15:04:50.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:50.408+0000 7f30d2e00640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f30cc15f910 con 0x7f30cc070030 2024-03-31T15:04:50.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:50.408+0000 7f30d2e00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f30cc162910 con 0x7f30cc141590 2024-03-31T15:04:50.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:50.408+0000 7f30c3e00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f30cc070030 0x7f30cc143c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:50.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:50.408+0000 7f30c3e00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f30cc070030 0x7f30cc143c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51196/0 (socket says 172.21.15.16:51196) 2024-03-31T15:04:50.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:50.408+0000 7f30cbe00640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f30cc1411b0 0x7f30cc06f920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:38200/0 (socket says 172.21.15.16:38200) 2024-03-31T15:04:50.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:50.408+0000 7f30c3e00640 1 -- 172.21.15.16:0/2261155562 learned_addr learned my addr 172.21.15.16:0/2261155562 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:04:50.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:50.408+0000 7f30d0e00640 1 --2- 172.21.15.16:0/2261155562 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f30cc141590 0x7f30cc144170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:50.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:50.408+0000 7f30c3e00640 1 -- 172.21.15.16:0/2261155562 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f30cc141590 msgr2=0x7f30cc144170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:04:50.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:50.408+0000 7f30c3e00640 1 --2- 172.21.15.16:0/2261155562 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f30cc141590 0x7f30cc144170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:50.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:50.408+0000 7f30c3e00640 1 -- 172.21.15.16:0/2261155562 >> v1:172.21.15.16:6789/0 conn(0x7f30cc1411b0 legacy=0x7f30cc06f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:50.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:50.408+0000 7f30c3e00640 1 -- 172.21.15.16:0/2261155562 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f30cc144980 con 0x7f30cc070030 2024-03-31T15:04:50.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:50.408+0000 7f30cb400640 1 -- 172.21.15.16:0/2261155562 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3941085013 0 0) 0x7f30cc1446b0 con 0x7f30cc1411b0 2024-03-31T15:04:50.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:50.408+0000 7f30c3e00640 1 --2- 172.21.15.16:0/2261155562 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f30cc070030 0x7f30cc143c30 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f30b4009870 tx=0x7f30b401c530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f93fe945c0dc120a server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:50.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:50.412+0000 7f30cb400640 1 -- 172.21.15.16:0/2261155562 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30b40a95f0 con 0x7f30cc070030 2024-03-31T15:04:50.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:50.412+0000 7f30cb400640 1 -- 172.21.15.16:0/2261155562 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f30b40a9790 con 0x7f30cc070030 2024-03-31T15:04:50.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:50.412+0000 7f30cb400640 1 -- 172.21.15.16:0/2261155562 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30b40a9b60 con 0x7f30cc070030 2024-03-31T15:04:50.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:50.412+0000 7f30d2e00640 1 -- 172.21.15.16:0/2261155562 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f30cc070030 msgr2=0x7f30cc143c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:50.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:50.412+0000 7f30d2e00640 1 --2- 172.21.15.16:0/2261155562 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f30cc070030 0x7f30cc143c30 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f30b4009870 tx=0x7f30b401c530 comp rx=0 tx=0).stop 2024-03-31T15:04:50.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:50.412+0000 7f30d2e00640 1 -- 172.21.15.16:0/2261155562 shutdown_connections 2024-03-31T15:04:50.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:50.412+0000 7f30d2e00640 1 --2- 172.21.15.16:0/2261155562 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f30cc141590 0x7f30cc144170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:50.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:50.412+0000 7f30d2e00640 1 --2- 172.21.15.16:0/2261155562 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f30cc070030 0x7f30cc143c30 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:50.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:50.412+0000 7f30d2e00640 1 -- 172.21.15.16:0/2261155562 >> 172.21.15.16:0/2261155562 conn(0x7f30cc067930 msgr2=0x7f30cc0b3ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:04:50.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:50.412+0000 7f30d2e00640 1 -- 172.21.15.16:0/2261155562 shutdown_connections 2024-03-31T15:04:50.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:50.412+0000 7f30d2e00640 1 -- 172.21.15.16:0/2261155562 wait complete. 2024-03-31T15:04:50.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:50.412+0000 7f30d2e00640 1 Processor -- start 2024-03-31T15:04:50.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:50.412+0000 7f30d2e00640 1 -- start start 2024-03-31T15:04:50.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:50.412+0000 7f30d2e00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f30cc070030 0x7f30cc13fe00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:50.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:50.412+0000 7f30d2e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f30cc1411b0 0x7f30cc152580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:50.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:50.412+0000 7f30d2e00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f30cc141590 0x7f30cc13dad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:50.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:50.412+0000 7f30d2e00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f30cc15f110 con 0x7f30cc1411b0 2024-03-31T15:04:50.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:50.412+0000 7f30d2e00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f30cc162da0 con 0x7f30cc070030 2024-03-31T15:04:50.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:50.412+0000 7f30d2e00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f30cc15f290 con 0x7f30cc141590 2024-03-31T15:04:50.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:50.412+0000 7f30c3e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f30cc1411b0 0x7f30cc152580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:50.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:50.412+0000 7f30c3e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f30cc1411b0 0x7f30cc152580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51206/0 (socket says 172.21.15.16:51206) 2024-03-31T15:04:50.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:50.412+0000 7f30c3e00640 1 -- 172.21.15.16:0/2962726255 learned_addr learned my addr 172.21.15.16:0/2962726255 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:04:50.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:50.416+0000 7f30d0e00640 1 --2- 172.21.15.16:0/2962726255 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f30cc141590 0x7f30cc13dad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:50.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:50.416+0000 7f30cbe00640 1 --2- 172.21.15.16:0/2962726255 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f30cc070030 0x7f30cc13fe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:50.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:50.416+0000 7f30c3e00640 1 -- 172.21.15.16:0/2962726255 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f30cc141590 msgr2=0x7f30cc13dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:50.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:50.416+0000 7f30c3e00640 1 --2- 172.21.15.16:0/2962726255 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f30cc141590 0x7f30cc13dad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:50.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:50.416+0000 7f30c3e00640 1 -- 172.21.15.16:0/2962726255 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f30cc070030 msgr2=0x7f30cc13fe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:50.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:50.416+0000 7f30c3e00640 1 --2- 172.21.15.16:0/2962726255 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f30cc070030 0x7f30cc13fe00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:50.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:50.416+0000 7f30c3e00640 1 -- 172.21.15.16:0/2962726255 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f30cc156f90 con 0x7f30cc1411b0 2024-03-31T15:04:50.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:50.416+0000 7f30cbe00640 1 --2- 172.21.15.16:0/2962726255 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f30cc070030 0x7f30cc13fe00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:04:50.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:50.416+0000 7f30c3e00640 1 --2- 172.21.15.16:0/2962726255 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f30cc1411b0 0x7f30cc152580 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f30b40ab040 tx=0x7f30b401ad70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:50.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:50.416+0000 7f30c8c00640 1 -- 172.21.15.16:0/2962726255 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30b40a95f0 con 0x7f30cc1411b0 2024-03-31T15:04:50.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:50.416+0000 7f30c8c00640 1 -- 172.21.15.16:0/2962726255 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f30b40b1820 con 0x7f30cc1411b0 2024-03-31T15:04:50.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:50.416+0000 7f30d2e00640 1 -- 172.21.15.16:0/2962726255 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f30cc157270 con 0x7f30cc1411b0 2024-03-31T15:04:50.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:50.416+0000 7f30c8c00640 1 -- 172.21.15.16:0/2962726255 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30b40b1c10 con 0x7f30cc1411b0 2024-03-31T15:04:50.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:50.416+0000 7f30d2e00640 1 -- 172.21.15.16:0/2962726255 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f30cc157830 con 0x7f30cc1411b0 2024-03-31T15:04:50.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:50.416+0000 7f30d2e00640 1 -- 172.21.15.16:0/2962726255 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f30cc069950 con 0x7f30cc1411b0 2024-03-31T15:04:50.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:50.416+0000 7f30c8c00640 1 -- 172.21.15.16:0/2962726255 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f30b40bc450 con 0x7f30cc1411b0 2024-03-31T15:04:50.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:50.420+0000 7f30c8c00640 1 --2- 172.21.15.16:0/2962726255 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f30a8068380 0x7f30a806a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:50.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:50.420+0000 7f30c8c00640 1 -- 172.21.15.16:0/2962726255 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5709+0+0 (secure 0 0 0) 0x7f30b41213a0 con 0x7f30cc1411b0 2024-03-31T15:04:50.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:50.420+0000 7f30cbe00640 1 --2- 172.21.15.16:0/2962726255 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f30a8068380 0x7f30a806a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:50.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:50.420+0000 7f30cbe00640 1 --2- 172.21.15.16:0/2962726255 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f30a8068380 0x7f30a806a840 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f30bc007fd0 tx=0x7f30bc007c20 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:50.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:50.424+0000 7f30c8c00640 1 -- 172.21.15.16:0/2962726255 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f30b40ad020 con 0x7f30cc1411b0 2024-03-31T15:04:50.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:50.588+0000 7f30d2e00640 1 -- 172.21.15.16:0/2962726255 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f30cc0d2f00 con 0x7f30cc1411b0 2024-03-31T15:04:50.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:50.588+0000 7f30c8c00640 1 -- 172.21.15.16:0/2962726255 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v34) ==== 100+0+11 (secure 0 0 0) 0x7f30b4016150 con 0x7f30cc1411b0 2024-03-31T15:04:50.582 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 64 2024-03-31T15:04:50.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:50.588+0000 7f30d2e00640 1 -- 172.21.15.16:0/2962726255 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f30a8068380 msgr2=0x7f30a806a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:50.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:50.588+0000 7f30d2e00640 1 --2- 172.21.15.16:0/2962726255 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f30a8068380 0x7f30a806a840 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f30bc007fd0 tx=0x7f30bc007c20 comp rx=0 tx=0).stop 2024-03-31T15:04:50.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:50.588+0000 7f30d2e00640 1 -- 172.21.15.16:0/2962726255 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f30cc1411b0 msgr2=0x7f30cc152580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:50.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:50.588+0000 7f30d2e00640 1 --2- 172.21.15.16:0/2962726255 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f30cc1411b0 0x7f30cc152580 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f30b40ab040 tx=0x7f30b401ad70 comp rx=0 tx=0).stop 2024-03-31T15:04:50.585 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:50.588+0000 7f30d2e00640 1 -- 172.21.15.16:0/2962726255 shutdown_connections 2024-03-31T15:04:50.585 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:50.588+0000 7f30d2e00640 1 --2- 172.21.15.16:0/2962726255 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f30a8068380 0x7f30a806a840 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:50.585 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:50.588+0000 7f30d2e00640 1 --2- 172.21.15.16:0/2962726255 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f30cc141590 0x7f30cc13dad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:50.585 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:50.588+0000 7f30d2e00640 1 --2- 172.21.15.16:0/2962726255 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f30cc1411b0 0x7f30cc152580 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:50.585 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:50.588+0000 7f30d2e00640 1 --2- 172.21.15.16:0/2962726255 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f30cc070030 0x7f30cc13fe00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:50.585 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:50.588+0000 7f30d2e00640 1 -- 172.21.15.16:0/2962726255 >> 172.21.15.16:0/2962726255 conn(0x7f30cc067930 msgr2=0x7f30cc15faa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:04:50.585 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:50.588+0000 7f30d2e00640 1 -- 172.21.15.16:0/2962726255 shutdown_connections 2024-03-31T15:04:50.585 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:50.588+0000 7f30d2e00640 1 -- 172.21.15.16:0/2962726255 wait complete. 2024-03-31T15:04:50.604 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-31T15:04:50.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:50.752+0000 7fc7a8a00640 1 Processor -- start 2024-03-31T15:04:50.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:50.752+0000 7fc7a8a00640 1 -- start start 2024-03-31T15:04:50.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:50.752+0000 7fc7a8a00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fc7a406f920 0x7fc7a406fd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:50.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:50.752+0000 7fc7a8a00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fc7a40d2010 0x7fc7a40d24a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:50.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:50.752+0000 7fc7a8a00640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc7a40d0920 con 0x7fc7a4145e40 2024-03-31T15:04:50.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:50.752+0000 7fc7a8a00640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7fc7a40702c0 con 0x7fc7a406f920 2024-03-31T15:04:50.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:50.752+0000 7fc7a8a00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fc7a40d06d0 con 0x7fc7a40d2010 2024-03-31T15:04:50.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:50.752+0000 7fc7a1400640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fc7a406f920 0x7fc7a406fd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:50.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:50.752+0000 7fc7a1e00640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fc7a4145e40 0x7fc7a40cc740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:38212/0 (socket says 172.21.15.16:38212) 2024-03-31T15:04:50.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:50.752+0000 7fc7a1400640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fc7a406f920 0x7fc7a406fd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51214/0 (socket says 172.21.15.16:51214) 2024-03-31T15:04:50.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:50.752+0000 7fc7a1e00640 1 -- 172.21.15.16:0/1185555336 learned_addr learned my addr 172.21.15.16:0/1185555336 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:04:50.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:50.752+0000 7fc7a2800640 1 --2- 172.21.15.16:0/1185555336 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fc7a40d2010 0x7fc7a40d24a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:50.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:50.752+0000 7fc7a1400640 1 -- 172.21.15.16:0/1185555336 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fc7a40d2010 msgr2=0x7fc7a40d24a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:04:50.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:50.752+0000 7fc7a1400640 1 --2- 172.21.15.16:0/1185555336 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fc7a40d2010 0x7fc7a40d24a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:50.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:50.752+0000 7fc7a1400640 1 -- 172.21.15.16:0/1185555336 >> v1:172.21.15.16:6789/0 conn(0x7fc7a4145e40 legacy=0x7fc7a40cc740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:50.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:50.752+0000 7fc7a1400640 1 -- 172.21.15.16:0/1185555336 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc7a40d0c00 con 0x7fc7a406f920 2024-03-31T15:04:50.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:50.752+0000 7fc7a0a00640 1 -- 172.21.15.16:0/1185555336 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2509513424 0 0) 0x7fc7a40d0920 con 0x7fc7a4145e40 2024-03-31T15:04:50.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:50.756+0000 7fc7a1400640 1 --2- 172.21.15.16:0/1185555336 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc7a406f920 0x7fc7a406fd80 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fc7900097c0 tx=0x7fc79001c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d695fddac845dee8 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:50.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:50.756+0000 7fc7a0a00640 1 -- 172.21.15.16:0/1185555336 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7900b2070 con 0x7fc7a406f920 2024-03-31T15:04:50.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:50.756+0000 7fc7a0a00640 1 -- 172.21.15.16:0/1185555336 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc7900ad640 con 0x7fc7a406f920 2024-03-31T15:04:50.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:50.756+0000 7fc7a0a00640 1 -- 172.21.15.16:0/1185555336 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7900ada10 con 0x7fc7a406f920 2024-03-31T15:04:50.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:50.756+0000 7fc7a8a00640 1 -- 172.21.15.16:0/1185555336 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc7a406f920 msgr2=0x7fc7a406fd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:50.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:50.756+0000 7fc7a8a00640 1 --2- 172.21.15.16:0/1185555336 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc7a406f920 0x7fc7a406fd80 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fc7900097c0 tx=0x7fc79001c4a0 comp rx=0 tx=0).stop 2024-03-31T15:04:50.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:50.756+0000 7fc7a8a00640 1 -- 172.21.15.16:0/1185555336 shutdown_connections 2024-03-31T15:04:50.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:50.756+0000 7fc7a8a00640 1 --2- 172.21.15.16:0/1185555336 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fc7a40d2010 0x7fc7a40d24a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:50.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:50.756+0000 7fc7a8a00640 1 --2- 172.21.15.16:0/1185555336 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc7a406f920 0x7fc7a406fd80 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:50.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:50.756+0000 7fc7a8a00640 1 -- 172.21.15.16:0/1185555336 >> 172.21.15.16:0/1185555336 conn(0x7fc7a4067330 msgr2=0x7fc7a4067740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:04:50.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:50.756+0000 7fc7a8a00640 1 -- 172.21.15.16:0/1185555336 shutdown_connections 2024-03-31T15:04:50.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:50.756+0000 7fc7a8a00640 1 -- 172.21.15.16:0/1185555336 wait complete. 2024-03-31T15:04:50.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:50.760+0000 7fc7a8a00640 1 Processor -- start 2024-03-31T15:04:50.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:50.760+0000 7fc7a8a00640 1 -- start start 2024-03-31T15:04:50.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:50.760+0000 7fc7a8a00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fc7a406f920 0x7fc7a413f060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:50.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:50.760+0000 7fc7a8a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc7a40d2010 0x7fc7a4131b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:50.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:50.760+0000 7fc7a8a00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc7a4145e40 0x7fc7a41598a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:50.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:50.760+0000 7fc7a8a00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fc7a40d0ee0 con 0x7fc7a40d2010 2024-03-31T15:04:50.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:50.760+0000 7fc7a8a00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fc7a40ce9e0 con 0x7fc7a4145e40 2024-03-31T15:04:50.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:50.760+0000 7fc7a8a00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fc7a40d2df0 con 0x7fc7a406f920 2024-03-31T15:04:50.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:50.760+0000 7fc7a1400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc7a40d2010 0x7fc7a4131b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:50.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:50.760+0000 7fc7a1400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc7a40d2010 0x7fc7a4131b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51222/0 (socket says 172.21.15.16:51222) 2024-03-31T15:04:50.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:50.760+0000 7fc7a1400640 1 -- 172.21.15.16:0/2848985669 learned_addr learned my addr 172.21.15.16:0/2848985669 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:04:50.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:50.760+0000 7fc7a1e00640 1 --2- 172.21.15.16:0/2848985669 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fc7a406f920 0x7fc7a413f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:50.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:50.760+0000 7fc7a2800640 1 --2- 172.21.15.16:0/2848985669 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc7a4145e40 0x7fc7a41598a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:50.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:50.760+0000 7fc7a1400640 1 -- 172.21.15.16:0/2848985669 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fc7a406f920 msgr2=0x7fc7a413f060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:50.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:50.760+0000 7fc7a1400640 1 --2- 172.21.15.16:0/2848985669 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fc7a406f920 0x7fc7a413f060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:50.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:50.760+0000 7fc7a1400640 1 -- 172.21.15.16:0/2848985669 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc7a4145e40 msgr2=0x7fc7a41598a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:50.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:50.760+0000 7fc7a1400640 1 --2- 172.21.15.16:0/2848985669 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc7a4145e40 0x7fc7a41598a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:50.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:50.760+0000 7fc7a1400640 1 -- 172.21.15.16:0/2848985669 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc7a41431f0 con 0x7fc7a40d2010 2024-03-31T15:04:50.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:50.760+0000 7fc7a1400640 1 --2- 172.21.15.16:0/2848985669 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc7a40d2010 0x7fc7a4131b00 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fc79001a0d0 tx=0x7fc790019240 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:50.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:50.760+0000 7fc79a000640 1 -- 172.21.15.16:0/2848985669 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7900b2040 con 0x7fc7a40d2010 2024-03-31T15:04:50.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:50.760+0000 7fc79a000640 1 -- 172.21.15.16:0/2848985669 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc7900bd070 con 0x7fc7a40d2010 2024-03-31T15:04:50.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:50.760+0000 7fc79a000640 1 -- 172.21.15.16:0/2848985669 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7900b6650 con 0x7fc7a40d2010 2024-03-31T15:04:50.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:50.760+0000 7fc7a8a00640 1 -- 172.21.15.16:0/2848985669 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc7a41438d0 con 0x7fc7a40d2010 2024-03-31T15:04:50.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:50.760+0000 7fc7a8a00640 1 -- 172.21.15.16:0/2848985669 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc7a4143e90 con 0x7fc7a40d2010 2024-03-31T15:04:50.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:50.764+0000 7fc7a8a00640 1 -- 172.21.15.16:0/2848985669 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc7a4069950 con 0x7fc7a40d2010 2024-03-31T15:04:50.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:50.764+0000 7fc79a000640 1 -- 172.21.15.16:0/2848985669 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fc7900b67f0 con 0x7fc7a40d2010 2024-03-31T15:04:50.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:50.764+0000 7fc79a000640 1 --2- 172.21.15.16:0/2848985669 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fc784068380 0x7fc78406a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:50.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:50.764+0000 7fc79a000640 1 -- 172.21.15.16:0/2848985669 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5709+0+0 (secure 0 0 0) 0x7fc790120810 con 0x7fc7a40d2010 2024-03-31T15:04:50.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:50.764+0000 7fc7a1e00640 1 --2- 172.21.15.16:0/2848985669 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fc784068380 0x7fc78406a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:50.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:50.764+0000 7fc7a1e00640 1 --2- 172.21.15.16:0/2848985669 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fc784068380 0x7fc78406a840 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fc794007fd0 tx=0x7fc794007d60 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:50.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:50.768+0000 7fc79a000640 1 -- 172.21.15.16:0/2848985669 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fc790016090 con 0x7fc7a40d2010 2024-03-31T15:04:50.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:50.944+0000 7fc7a8a00640 1 -- 172.21.15.16:0/2848985669 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fc7a40cf3c0 con 0x7fc7a40d2010 2024-03-31T15:04:50.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:50.944+0000 7fc79a000640 1 -- 172.21.15.16:0/2848985669 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v34) ==== 80+0+0 (secure 0 0 0) 0x7fc7900f0790 con 0x7fc7a40d2010 2024-03-31T15:04:50.938 INFO:teuthology.orchestra.run.smithi016.stderr:listed 0 entries 2024-03-31T15:04:50.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:50.944+0000 7fc7a8a00640 1 -- 172.21.15.16:0/2848985669 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fc784068380 msgr2=0x7fc78406a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:50.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:50.944+0000 7fc7a8a00640 1 --2- 172.21.15.16:0/2848985669 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fc784068380 0x7fc78406a840 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fc794007fd0 tx=0x7fc794007d60 comp rx=0 tx=0).stop 2024-03-31T15:04:50.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:50.944+0000 7fc7a8a00640 1 -- 172.21.15.16:0/2848985669 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc7a40d2010 msgr2=0x7fc7a4131b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:50.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:50.944+0000 7fc7a8a00640 1 --2- 172.21.15.16:0/2848985669 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc7a40d2010 0x7fc7a4131b00 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fc79001a0d0 tx=0x7fc790019240 comp rx=0 tx=0).stop 2024-03-31T15:04:50.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:50.944+0000 7fc7a8a00640 1 -- 172.21.15.16:0/2848985669 shutdown_connections 2024-03-31T15:04:50.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:50.944+0000 7fc7a8a00640 1 --2- 172.21.15.16:0/2848985669 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fc784068380 0x7fc78406a840 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:50.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:50.944+0000 7fc7a8a00640 1 --2- 172.21.15.16:0/2848985669 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc7a4145e40 0x7fc7a41598a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:50.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:50.944+0000 7fc7a8a00640 1 --2- 172.21.15.16:0/2848985669 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc7a40d2010 0x7fc7a4131b00 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:50.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:50.944+0000 7fc7a8a00640 1 --2- 172.21.15.16:0/2848985669 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fc7a406f920 0x7fc7a413f060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:50.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:50.944+0000 7fc7a8a00640 1 -- 172.21.15.16:0/2848985669 >> 172.21.15.16:0/2848985669 conn(0x7fc7a4067330 msgr2=0x7fc7a4067710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:04:50.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:50.944+0000 7fc7a8a00640 1 -- 172.21.15.16:0/2848985669 shutdown_connections 2024-03-31T15:04:50.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:50.944+0000 7fc7a8a00640 1 -- 172.21.15.16:0/2848985669 wait complete. 2024-03-31T15:04:50.959 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-03-31T15:04:50.959 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-31T15:04:50.968 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-31T15:04:51.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:51.156+0000 7f924c600640 1 Processor -- start 2024-03-31T15:04:51.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:51.156+0000 7f924c600640 1 -- start start 2024-03-31T15:04:51.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:51.156+0000 7f924c600640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f924406dbc0 0x7f924415f650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:51.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:51.156+0000 7f924c600640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f92440cb440 0x7f924406f920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:51.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:51.156+0000 7f924c600640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f92440cbe90 con 0x7f92440cb8c0 2024-03-31T15:04:51.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:51.156+0000 7f924c600640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f924415fb90 con 0x7f92440cb440 2024-03-31T15:04:51.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:51.156+0000 7f924c600640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f924415f1c0 con 0x7f924406dbc0 2024-03-31T15:04:51.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:51.160+0000 7f9249200640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f92440cb440 0x7f924406f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:51.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:51.160+0000 7f9249200640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f92440cb440 0x7f924406f920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51236/0 (socket says 172.21.15.16:51236) 2024-03-31T15:04:51.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:51.160+0000 7f9249200640 1 -- 172.21.15.16:0/3430520995 learned_addr learned my addr 172.21.15.16:0/3430520995 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:04:51.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:51.160+0000 7f924a600640 1 --1- 172.21.15.16:0/3430520995 >> v1:172.21.15.16:6789/0 conn(0x7f92440cb8c0 0x7f9244162180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:38218/0 (socket says 172.21.15.16:38218) 2024-03-31T15:04:51.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:51.160+0000 7f9249c00640 1 --2- 172.21.15.16:0/3430520995 >> v2:172.21.15.89:3300/0 conn(0x7f924406dbc0 0x7f924415f650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:51.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:51.160+0000 7f9249200640 1 -- 172.21.15.16:0/3430520995 >> v2:172.21.15.89:3300/0 conn(0x7f924406dbc0 msgr2=0x7f924415f650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:04:51.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:51.160+0000 7f9249200640 1 --2- 172.21.15.16:0/3430520995 >> v2:172.21.15.89:3300/0 conn(0x7f924406dbc0 0x7f924415f650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:51.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:51.160+0000 7f9249200640 1 -- 172.21.15.16:0/3430520995 >> v1:172.21.15.16:6789/0 conn(0x7f92440cb8c0 legacy=0x7f9244162180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:51.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:51.160+0000 7f9249200640 1 -- 172.21.15.16:0/3430520995 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92440cc170 con 0x7f92440cb440 2024-03-31T15:04:51.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:51.160+0000 7f9243e00640 1 -- 172.21.15.16:0/3430520995 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3997662950 0 0) 0x7f92440cbe90 con 0x7f92440cb8c0 2024-03-31T15:04:51.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:51.160+0000 7f9249200640 1 --2- 172.21.15.16:0/3430520995 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f92440cb440 0x7f924406f920 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f92440b8380 tx=0x7f923001c460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1cd06828798b346a server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:51.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:51.160+0000 7f9243e00640 1 -- 172.21.15.16:0/3430520995 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92300b0070 con 0x7f92440cb440 2024-03-31T15:04:51.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:51.160+0000 7f9243e00640 1 -- 172.21.15.16:0/3430520995 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f92300ab600 con 0x7f92440cb440 2024-03-31T15:04:51.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:51.160+0000 7f9243e00640 1 -- 172.21.15.16:0/3430520995 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92300ab9d0 con 0x7f92440cb440 2024-03-31T15:04:51.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:51.164+0000 7f924c600640 1 -- 172.21.15.16:0/3430520995 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f92440cb440 msgr2=0x7f924406f920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:51.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:51.164+0000 7f924c600640 1 --2- 172.21.15.16:0/3430520995 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f92440cb440 0x7f924406f920 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f92440b8380 tx=0x7f923001c460 comp rx=0 tx=0).stop 2024-03-31T15:04:51.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:51.164+0000 7f924c600640 1 -- 172.21.15.16:0/3430520995 shutdown_connections 2024-03-31T15:04:51.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:51.164+0000 7f924c600640 1 --2- 172.21.15.16:0/3430520995 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f92440cb440 0x7f924406f920 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:51.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:51.164+0000 7f924c600640 1 --2- 172.21.15.16:0/3430520995 >> v2:172.21.15.89:3300/0 conn(0x7f924406dbc0 0x7f924415f650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:51.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:51.164+0000 7f924c600640 1 -- 172.21.15.16:0/3430520995 >> 172.21.15.16:0/3430520995 conn(0x7f9244067330 msgr2=0x7f9244067740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:04:51.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:51.164+0000 7f924c600640 1 -- 172.21.15.16:0/3430520995 shutdown_connections 2024-03-31T15:04:51.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:51.164+0000 7f924c600640 1 -- 172.21.15.16:0/3430520995 wait complete. 2024-03-31T15:04:51.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:51.164+0000 7f924c600640 1 Processor -- start 2024-03-31T15:04:51.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:51.164+0000 7f924c600640 1 -- start start 2024-03-31T15:04:51.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:51.164+0000 7f924c600640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f924406dbc0 0x7f9244144320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:51.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:51.164+0000 7f924c600640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f92440cb440 0x7f92440b6480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:51.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:51.164+0000 7f924c600640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f92440cb8c0 0x7f9244163460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:51.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:51.164+0000 7f924c600640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f924406c950 con 0x7f92440cb8c0 2024-03-31T15:04:51.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:51.164+0000 7f924c600640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f9244162ca0 con 0x7f924406dbc0 2024-03-31T15:04:51.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:51.164+0000 7f924c600640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f924406cad0 con 0x7f92440cb440 2024-03-31T15:04:51.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:51.164+0000 7f924a600640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f92440cb8c0 0x7f9244163460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:51.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:51.164+0000 7f924a600640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f92440cb8c0 0x7f9244163460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51250/0 (socket says 172.21.15.16:51250) 2024-03-31T15:04:51.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:51.164+0000 7f924a600640 1 -- 172.21.15.16:0/981367833 learned_addr learned my addr 172.21.15.16:0/981367833 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:04:51.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:51.164+0000 7f9249c00640 1 --2- 172.21.15.16:0/981367833 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f924406dbc0 0x7f9244144320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:51.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:51.164+0000 7f9249200640 1 --2- 172.21.15.16:0/981367833 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f92440cb440 0x7f92440b6480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:51.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:51.168+0000 7f924a600640 1 -- 172.21.15.16:0/981367833 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f92440cb440 msgr2=0x7f92440b6480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:51.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:51.168+0000 7f924a600640 1 --2- 172.21.15.16:0/981367833 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f92440cb440 0x7f92440b6480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:51.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:51.168+0000 7f924a600640 1 -- 172.21.15.16:0/981367833 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f924406dbc0 msgr2=0x7f9244144320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:51.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:51.168+0000 7f924a600640 1 --2- 172.21.15.16:0/981367833 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f924406dbc0 0x7f9244144320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:51.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:51.168+0000 7f924a600640 1 -- 172.21.15.16:0/981367833 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92440b6eb0 con 0x7f92440cb8c0 2024-03-31T15:04:51.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:51.168+0000 7f924a600640 1 --2- 172.21.15.16:0/981367833 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f92440cb8c0 0x7f9244163460 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f923c00c7d0 tx=0x7f923c00cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:51.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:51.168+0000 7f9241600640 1 -- 172.21.15.16:0/981367833 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f923c018070 con 0x7f92440cb8c0 2024-03-31T15:04:51.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:51.168+0000 7f9241600640 1 -- 172.21.15.16:0/981367833 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f923c00a640 con 0x7f92440cb8c0 2024-03-31T15:04:51.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:51.168+0000 7f9241600640 1 -- 172.21.15.16:0/981367833 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f923c00aa70 con 0x7f92440cb8c0 2024-03-31T15:04:51.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:51.168+0000 7f924c600640 1 -- 172.21.15.16:0/981367833 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f92440b6bd0 con 0x7f92440cb8c0 2024-03-31T15:04:51.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:51.168+0000 7f924c600640 1 -- 172.21.15.16:0/981367833 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f92441595d0 con 0x7f92440cb8c0 2024-03-31T15:04:51.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:51.168+0000 7f924c600640 1 -- 172.21.15.16:0/981367833 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9244159ad0 con 0x7f92440cb8c0 2024-03-31T15:04:51.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:51.168+0000 7f9241600640 1 -- 172.21.15.16:0/981367833 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f923c00ac10 con 0x7f92440cb8c0 2024-03-31T15:04:51.164 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:51.172+0000 7f9241600640 1 --2- 172.21.15.16:0/981367833 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f9224068380 0x7f922406a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:51.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:51.172+0000 7f9241600640 1 -- 172.21.15.16:0/981367833 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5709+0+0 (secure 0 0 0) 0x7f923c085ec0 con 0x7f92440cb8c0 2024-03-31T15:04:51.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:51.172+0000 7f9249c00640 1 --2- 172.21.15.16:0/981367833 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f9224068380 0x7f922406a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:51.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:51.172+0000 7f9249c00640 1 --2- 172.21.15.16:0/981367833 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f9224068380 0x7f922406a840 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f92440b8230 tx=0x7f9238009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:51.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:51.176+0000 7f9241600640 1 -- 172.21.15.16:0/981367833 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f9244159ad0 con 0x7f92440cb8c0 2024-03-31T15:04:51.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:51.340+0000 7f924c600640 1 -- 172.21.15.16:0/981367833 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f9244069950 con 0x7f92440cb8c0 2024-03-31T15:04:51.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:51.340+0000 7f9241600640 1 -- 172.21.15.16:0/981367833 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v34) ==== 74+0+16597 (secure 0 0 0) 0x7f923c055ec0 con 0x7f92440cb8c0 2024-03-31T15:04:51.335 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:04:51.335 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":34,"fsid":"286eb5da-fb47-4550-b290-d9519c13b9a5","created":"2024-03-31T15:03:11.438952+0000","modified":"2024-03-31T15:04:41.385382+0000","last_up_change":"2024-03-31T15:03:22.885535+0000","last_in_change":"2024-03-31T15:03:13.941667+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-31T15:03:22.550787+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-03-31T15:04:37.577341+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-03-31T15:04:38.738566+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"34","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"92a66450-ef19-40d8-ac3d-d760b23b0bda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6801","nonce":932712244}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6803","nonce":932712244}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6807","nonce":932712244}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6805","nonce":932712244}]},"public_addr":"172.21.15.16:6801/932712244","cluster_addr":"172.21.15.16:6803/932712244","heartbeat_back_addr":"172.21.15.16:6807/932712244","heartbeat_front_addr":"172.21.15.16:6805/932712244","state":["exists","up"]},{"osd":1,"uuid":"da9d1f2b-1968-4959-b843-faa231988e92","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6810","nonce":1398737533}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6811","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6813","nonce":1398737533}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6821","nonce":1398737533}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6815","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6816","nonce":1398737533}]},"public_addr":"172.21.15.16:6810/1398737533","cluster_addr":"172.21.15.16:6813/1398737533","heartbeat_back_addr":"172.21.15.16:6821/1398737533","heartbeat_front_addr":"172.21.15.16:6816/1398737533","state":["exists","up"]},{"osd":2,"uuid":"01a73662-9a21-4b1f-bcbb-42b9636a4638","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6819","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6822","nonce":3018526646}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6825","nonce":3018526646}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6831","nonce":3018526646}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6827","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6829","nonce":3018526646}]},"public_addr":"172.21.15.16:6822/3018526646","cluster_addr":"172.21.15.16:6825/3018526646","heartbeat_back_addr":"172.21.15.16:6831/3018526646","heartbeat_front_addr":"172.21.15.16:6829/3018526646","state":["exists","up"]},{"osd":3,"uuid":"c7771fd3-e18e-4500-ac38-23cf10ebcd75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6809","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6812","nonce":3360352974}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6817","nonce":3360352974}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6828","nonce":3360352974}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6823","nonce":3360352974}]},"public_addr":"172.21.15.16:6812/3360352974","cluster_addr":"172.21.15.16:6817/3360352974","heartbeat_back_addr":"172.21.15.16:6828/3360352974","heartbeat_front_addr":"172.21.15.16:6823/3360352974","state":["exists","up"]},{"osd":4,"uuid":"88260730-1482-4fbd-a40f-cb1124e9df4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6801","nonce":1856844856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6803","nonce":1856844856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6807","nonce":1856844856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6805","nonce":1856844856}]},"public_addr":"172.21.15.89:6801/1856844856","cluster_addr":"172.21.15.89:6803/1856844856","heartbeat_back_addr":"172.21.15.89:6807/1856844856","heartbeat_front_addr":"172.21.15.89:6805/1856844856","state":["exists","up"]},{"osd":5,"uuid":"4cb2c00c-fae0-4dd8-9d27-ec4f0093ccf7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6814","nonce":50395527}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6817","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6820","nonce":50395527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6831","nonce":50395527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6823","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6826","nonce":50395527}]},"public_addr":"172.21.15.89:6814/50395527","cluster_addr":"172.21.15.89:6820/50395527","heartbeat_back_addr":"172.21.15.89:6831/50395527","heartbeat_front_addr":"172.21.15.89:6826/50395527","state":["exists","up"]},{"osd":6,"uuid":"81ce0c19-2554-4cae-acbe-66a05e9e9b9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6809","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6812","nonce":3470471752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6815","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6818","nonce":3470471752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6827","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6829","nonce":3470471752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6821","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6824","nonce":3470471752}]},"public_addr":"172.21.15.89:6812/3470471752","cluster_addr":"172.21.15.89:6818/3470471752","heartbeat_back_addr":"172.21.15.89:6829/3470471752","heartbeat_front_addr":"172.21.15.89:6824/3470471752","state":["exists","up"]},{"osd":7,"uuid":"85e4313b-d37d-442e-8042-153bf20a847d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6811","nonce":1796778630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6813","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6816","nonce":1796778630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6825","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6828","nonce":1796778630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6819","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6822","nonce":1796778630}]},"public_addr":"172.21.15.89:6811/1796778630","cluster_addr":"172.21.15.89:6816/1796778630","heartbeat_back_addr":"172.21.15.89:6828/1796778630","heartbeat_front_addr":"172.21.15.89:6822/1796778630","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.439088+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.529970+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.680512+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:21.044161+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.582111+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.536558+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.409522+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.590229+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-31T15:04:51.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:51.344+0000 7f924c600640 1 -- 172.21.15.16:0/981367833 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f9224068380 msgr2=0x7f922406a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:51.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:51.344+0000 7f924c600640 1 --2- 172.21.15.16:0/981367833 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f9224068380 0x7f922406a840 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f92440b8230 tx=0x7f9238009190 comp rx=0 tx=0).stop 2024-03-31T15:04:51.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:51.344+0000 7f924c600640 1 -- 172.21.15.16:0/981367833 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f92440cb8c0 msgr2=0x7f9244163460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:51.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:51.344+0000 7f924c600640 1 --2- 172.21.15.16:0/981367833 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f92440cb8c0 0x7f9244163460 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f923c00c7d0 tx=0x7f923c00cc90 comp rx=0 tx=0).stop 2024-03-31T15:04:51.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:51.344+0000 7f924c600640 1 -- 172.21.15.16:0/981367833 shutdown_connections 2024-03-31T15:04:51.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:51.344+0000 7f924c600640 1 --2- 172.21.15.16:0/981367833 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f9224068380 0x7f922406a840 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:51.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:51.344+0000 7f924c600640 1 --2- 172.21.15.16:0/981367833 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f92440cb8c0 0x7f9244163460 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:51.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:51.344+0000 7f924c600640 1 --2- 172.21.15.16:0/981367833 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f92440cb440 0x7f92440b6480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:51.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:51.344+0000 7f924c600640 1 --2- 172.21.15.16:0/981367833 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f924406dbc0 0x7f9244144320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:51.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:51.344+0000 7f924c600640 1 -- 172.21.15.16:0/981367833 >> 172.21.15.16:0/981367833 conn(0x7f9244067330 msgr2=0x7f9244067710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:04:51.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:51.344+0000 7f924c600640 1 -- 172.21.15.16:0/981367833 shutdown_connections 2024-03-31T15:04:51.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:51.344+0000 7f924c600640 1 -- 172.21.15.16:0/981367833 wait complete. 2024-03-31T15:04:51.357 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-31T15:03:22.550787+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-31T15:04:37.577341+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '33', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_data', 'create_time': '2024-03-31T15:04:38.738566+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '34', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-31T15:04:51.357 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-31T15:04:51.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:51.508+0000 7f83b6e00640 1 Processor -- start 2024-03-31T15:04:51.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:51.508+0000 7f83b6e00640 1 -- start start 2024-03-31T15:04:51.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:51.508+0000 7f83b6e00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f83b01411b0 0x7f83b006f920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:51.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:51.508+0000 7f83b6e00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f83b0144340 0x7f83b006fe60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:51.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:51.508+0000 7f83b6e00640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f83b0144980 con 0x7f83b0141590 2024-03-31T15:04:51.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:51.508+0000 7f83b6e00640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f83b015f910 con 0x7f83b0144340 2024-03-31T15:04:51.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:51.508+0000 7f83b6e00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f83b0162910 con 0x7f83b01411b0 2024-03-31T15:04:51.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:51.508+0000 7f83afe00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f83b01411b0 0x7f83b006f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:51.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:51.508+0000 7f83afe00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f83b01411b0 0x7f83b006f920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3301/0 says I am v2:172.21.15.16:37060/0 (socket says 172.21.15.16:37060) 2024-03-31T15:04:51.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:51.508+0000 7f83afe00640 1 -- 172.21.15.16:0/2788459347 learned_addr learned my addr 172.21.15.16:0/2788459347 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:04:51.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:51.508+0000 7f83b4e00640 1 --2- 172.21.15.16:0/2788459347 >> v2:172.21.15.89:3300/0 conn(0x7f83b0144340 0x7f83b006fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:51.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:51.508+0000 7f83afe00640 1 -- 172.21.15.16:0/2788459347 >> v2:172.21.15.89:3300/0 conn(0x7f83b0144340 msgr2=0x7f83b006fe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:04:51.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:51.508+0000 7f83afe00640 1 --2- 172.21.15.16:0/2788459347 >> v2:172.21.15.89:3300/0 conn(0x7f83b0144340 0x7f83b006fe60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:51.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:51.508+0000 7f83afe00640 1 -- 172.21.15.16:0/2788459347 >> v1:172.21.15.89:6789/0 conn(0x7f83b0141590 legacy=0x7f83b0143c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:51.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:51.508+0000 7f83afe00640 1 -- 172.21.15.16:0/2788459347 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83b0161fe0 con 0x7f83b01411b0 2024-03-31T15:04:51.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:51.512+0000 7f83afe00640 1 --2- 172.21.15.16:0/2788459347 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f83b01411b0 0x7f83b006f920 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f83a0009870 tx=0x7f83a001c560 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a6466f081b2fe80f server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:51.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:51.512+0000 7f83aea00640 1 -- 172.21.15.16:0/2788459347 <== mon.2 v2:172.21.15.89:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83a00ab650 con 0x7f83b01411b0 2024-03-31T15:04:51.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:51.512+0000 7f83aea00640 1 -- 172.21.15.16:0/2788459347 <== mon.2 v2:172.21.15.89:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f83a00b0070 con 0x7f83b01411b0 2024-03-31T15:04:51.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:51.512+0000 7f83aea00640 1 -- 172.21.15.16:0/2788459347 <== mon.2 v2:172.21.15.89:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83a00aba10 con 0x7f83b01411b0 2024-03-31T15:04:51.507 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:51.512+0000 7f83b6e00640 1 -- 172.21.15.16:0/2788459347 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f83b01411b0 msgr2=0x7f83b006f920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:51.507 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:51.512+0000 7f83b6e00640 1 --2- 172.21.15.16:0/2788459347 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f83b01411b0 0x7f83b006f920 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f83a0009870 tx=0x7f83a001c560 comp rx=0 tx=0).stop 2024-03-31T15:04:51.507 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:51.512+0000 7f83b6e00640 1 -- 172.21.15.16:0/2788459347 shutdown_connections 2024-03-31T15:04:51.507 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:51.512+0000 7f83b6e00640 1 --2- 172.21.15.16:0/2788459347 >> v2:172.21.15.89:3300/0 conn(0x7f83b0144340 0x7f83b006fe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:51.507 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:51.512+0000 7f83b6e00640 1 --2- 172.21.15.16:0/2788459347 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f83b01411b0 0x7f83b006f920 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:51.507 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:51.512+0000 7f83b6e00640 1 -- 172.21.15.16:0/2788459347 >> 172.21.15.16:0/2788459347 conn(0x7f83b0067930 msgr2=0x7f83b00b3ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:04:51.507 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:51.512+0000 7f83b6e00640 1 -- 172.21.15.16:0/2788459347 shutdown_connections 2024-03-31T15:04:51.507 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:51.512+0000 7f83b6e00640 1 -- 172.21.15.16:0/2788459347 wait complete. 2024-03-31T15:04:51.508 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:51.512+0000 7f83b6e00640 1 Processor -- start 2024-03-31T15:04:51.508 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:51.516+0000 7f83b6e00640 1 -- start start 2024-03-31T15:04:51.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:51.516+0000 7f83b6e00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f83b01411b0 0x7f83b0156e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:51.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:51.516+0000 7f83b6e00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f83b0141590 0x7f83b013f870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:51.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:51.516+0000 7f83b6e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f83b0144340 0x7f83b015c660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:51.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:51.516+0000 7f83b6e00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f83b015f290 con 0x7f83b0144340 2024-03-31T15:04:51.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:51.516+0000 7f83b6e00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f83b015f110 con 0x7f83b0141590 2024-03-31T15:04:51.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:51.516+0000 7f83b6e00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f83b015f410 con 0x7f83b01411b0 2024-03-31T15:04:51.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:51.516+0000 7f83b4e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f83b0144340 0x7f83b015c660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:51.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:51.516+0000 7f83b4e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f83b0144340 0x7f83b015c660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51254/0 (socket says 172.21.15.16:51254) 2024-03-31T15:04:51.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:51.516+0000 7f83b4e00640 1 -- 172.21.15.16:0/1309232702 learned_addr learned my addr 172.21.15.16:0/1309232702 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:04:51.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:51.516+0000 7f83afe00640 1 --2- 172.21.15.16:0/1309232702 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f83b01411b0 0x7f83b0156e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:51.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:51.516+0000 7f83af400640 1 --2- 172.21.15.16:0/1309232702 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f83b0141590 0x7f83b013f870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:51.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:51.516+0000 7f83b4e00640 1 -- 172.21.15.16:0/1309232702 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f83b01411b0 msgr2=0x7f83b0156e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:51.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:51.516+0000 7f83b4e00640 1 --2- 172.21.15.16:0/1309232702 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f83b01411b0 0x7f83b0156e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:51.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:51.516+0000 7f83b4e00640 1 -- 172.21.15.16:0/1309232702 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f83b0141590 msgr2=0x7f83b013f870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:51.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:51.516+0000 7f83b4e00640 1 --2- 172.21.15.16:0/1309232702 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f83b0141590 0x7f83b013f870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:51.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:51.516+0000 7f83b4e00640 1 -- 172.21.15.16:0/1309232702 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83b0155dd0 con 0x7f83b0144340 2024-03-31T15:04:51.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:51.516+0000 7f83afe00640 1 --2- 172.21.15.16:0/1309232702 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f83b01411b0 0x7f83b0156e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:04:51.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:51.516+0000 7f83b4e00640 1 --2- 172.21.15.16:0/1309232702 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f83b0144340 0x7f83b015c660 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f83a800efc0 tx=0x7f83a800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:51.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:51.516+0000 7f83a7e00640 1 -- 172.21.15.16:0/1309232702 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83a8018020 con 0x7f83b0144340 2024-03-31T15:04:51.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:51.516+0000 7f83a7e00640 1 -- 172.21.15.16:0/1309232702 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f83a8014480 con 0x7f83b0144340 2024-03-31T15:04:51.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:51.516+0000 7f83a7e00640 1 -- 172.21.15.16:0/1309232702 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83a8014870 con 0x7f83b0144340 2024-03-31T15:04:51.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:51.516+0000 7f83b6e00640 1 -- 172.21.15.16:0/1309232702 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f83b01560b0 con 0x7f83b0144340 2024-03-31T15:04:51.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:51.516+0000 7f83b6e00640 1 -- 172.21.15.16:0/1309232702 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f83b0156670 con 0x7f83b0144340 2024-03-31T15:04:51.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:51.516+0000 7f83b6e00640 1 -- 172.21.15.16:0/1309232702 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f83b0069950 con 0x7f83b0144340 2024-03-31T15:04:51.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:51.520+0000 7f83a7e00640 1 -- 172.21.15.16:0/1309232702 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f83a801b020 con 0x7f83b0144340 2024-03-31T15:04:51.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:51.520+0000 7f83a7e00640 1 --2- 172.21.15.16:0/1309232702 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f8390068380 0x7f839006a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:51.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:51.520+0000 7f83a7e00640 1 -- 172.21.15.16:0/1309232702 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5709+0+0 (secure 0 0 0) 0x7f83a80873e0 con 0x7f83b0144340 2024-03-31T15:04:51.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:51.520+0000 7f83afe00640 1 --2- 172.21.15.16:0/1309232702 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f8390068380 0x7f839006a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:51.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:51.520+0000 7f83afe00640 1 --2- 172.21.15.16:0/1309232702 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f8390068380 0x7f839006a840 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f83a00af000 tx=0x7f83a00a7000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:51.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:51.524+0000 7f83a7e00640 1 -- 172.21.15.16:0/1309232702 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f83a8022070 con 0x7f83b0144340 2024-03-31T15:04:51.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:51.688+0000 7f83b6e00640 1 -- 172.21.15.16:0/1309232702 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f83b00b4a20 con 0x7f83b0144340 2024-03-31T15:04:51.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:51.688+0000 7f83a7e00640 1 -- 172.21.15.16:0/1309232702 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v34) ==== 93+0+10 (secure 0 0 0) 0x7f83a8010040 con 0x7f83b0144340 2024-03-31T15:04:51.682 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2024-03-31T15:04:51.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:51.688+0000 7f83b6e00640 1 -- 172.21.15.16:0/1309232702 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f8390068380 msgr2=0x7f839006a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:51.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:51.688+0000 7f83b6e00640 1 --2- 172.21.15.16:0/1309232702 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f8390068380 0x7f839006a840 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f83a00af000 tx=0x7f83a00a7000 comp rx=0 tx=0).stop 2024-03-31T15:04:51.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:51.688+0000 7f83b6e00640 1 -- 172.21.15.16:0/1309232702 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f83b0144340 msgr2=0x7f83b015c660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:51.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:51.688+0000 7f83b6e00640 1 --2- 172.21.15.16:0/1309232702 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f83b0144340 0x7f83b015c660 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f83a800efc0 tx=0x7f83a800c490 comp rx=0 tx=0).stop 2024-03-31T15:04:51.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:51.688+0000 7f83b6e00640 1 -- 172.21.15.16:0/1309232702 shutdown_connections 2024-03-31T15:04:51.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:51.688+0000 7f83b6e00640 1 --2- 172.21.15.16:0/1309232702 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f8390068380 0x7f839006a840 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:51.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:51.688+0000 7f83b6e00640 1 --2- 172.21.15.16:0/1309232702 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f83b0144340 0x7f83b015c660 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:51.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:51.688+0000 7f83b6e00640 1 --2- 172.21.15.16:0/1309232702 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f83b0141590 0x7f83b013f870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:51.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:51.688+0000 7f83b6e00640 1 --2- 172.21.15.16:0/1309232702 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f83b01411b0 0x7f83b0156e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:51.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:51.688+0000 7f83b6e00640 1 -- 172.21.15.16:0/1309232702 >> 172.21.15.16:0/1309232702 conn(0x7f83b0067930 msgr2=0x7f83b006df70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:04:51.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:51.688+0000 7f83b6e00640 1 -- 172.21.15.16:0/1309232702 shutdown_connections 2024-03-31T15:04:51.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:51.688+0000 7f83b6e00640 1 -- 172.21.15.16:0/1309232702 wait complete. 2024-03-31T15:04:51.704 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-31T15:04:51.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:51.852+0000 7f1383800640 1 Processor -- start 2024-03-31T15:04:51.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:51.852+0000 7f1383800640 1 -- start start 2024-03-31T15:04:51.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:51.852+0000 7f1383800640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f137c06f920 0x7f137c06fd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:51.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:51.852+0000 7f1383800640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f137c0d0a60 0x7f137c0cea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:51.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:51.852+0000 7f1383800640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f137c070240 con 0x7f137c0cef80 2024-03-31T15:04:51.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:51.852+0000 7f1383800640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f137c13f670 con 0x7f137c06f920 2024-03-31T15:04:51.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:51.852+0000 7f1383800640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f137c15ca70 con 0x7f137c0d0a60 2024-03-31T15:04:51.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:51.852+0000 7f1380e00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f137c06f920 0x7f137c06fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:51.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:51.852+0000 7f1380e00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f137c06f920 0x7f137c06fd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51264/0 (socket says 172.21.15.16:51264) 2024-03-31T15:04:51.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:51.852+0000 7f1380e00640 1 -- 172.21.15.16:0/650560655 learned_addr learned my addr 172.21.15.16:0/650560655 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:04:51.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:51.856+0000 7f137be00640 1 --2- 172.21.15.16:0/650560655 >> v2:172.21.15.89:3300/0 conn(0x7f137c0d0a60 0x7f137c0cea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:51.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:51.856+0000 7f1380e00640 1 -- 172.21.15.16:0/650560655 >> v2:172.21.15.89:3300/0 conn(0x7f137c0d0a60 msgr2=0x7f137c0cea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:04:51.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:51.856+0000 7f1380e00640 1 --2- 172.21.15.16:0/650560655 >> v2:172.21.15.89:3300/0 conn(0x7f137c0d0a60 0x7f137c0cea40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:51.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:51.856+0000 7f1380e00640 1 -- 172.21.15.16:0/650560655 >> v1:172.21.15.89:6789/0 conn(0x7f137c0cef80 legacy=0x7f137c070e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:04:51.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:51.856+0000 7f1380e00640 1 -- 172.21.15.16:0/650560655 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f137c15cd50 con 0x7f137c06f920 2024-03-31T15:04:51.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:51.856+0000 7f1380e00640 1 --2- 172.21.15.16:0/650560655 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f137c06f920 0x7f137c06fd00 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f137c0736b0 tx=0x7f137001c450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6e32d961e01b825 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:51.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:51.856+0000 7f137b400640 1 -- 172.21.15.16:0/650560655 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13700b0070 con 0x7f137c06f920 2024-03-31T15:04:51.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:51.856+0000 7f137b400640 1 -- 172.21.15.16:0/650560655 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f13700ab400 con 0x7f137c06f920 2024-03-31T15:04:51.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:51.856+0000 7f137b400640 1 -- 172.21.15.16:0/650560655 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13700ab7d0 con 0x7f137c06f920 2024-03-31T15:04:51.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:51.860+0000 7f1383800640 1 -- 172.21.15.16:0/650560655 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f137c06f920 msgr2=0x7f137c06fd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:51.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:51.860+0000 7f1383800640 1 --2- 172.21.15.16:0/650560655 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f137c06f920 0x7f137c06fd00 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f137c0736b0 tx=0x7f137001c450 comp rx=0 tx=0).stop 2024-03-31T15:04:51.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:51.860+0000 7f1383800640 1 -- 172.21.15.16:0/650560655 shutdown_connections 2024-03-31T15:04:51.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:51.860+0000 7f1383800640 1 --2- 172.21.15.16:0/650560655 >> v2:172.21.15.89:3300/0 conn(0x7f137c0d0a60 0x7f137c0cea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:51.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:51.860+0000 7f1383800640 1 --2- 172.21.15.16:0/650560655 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f137c06f920 0x7f137c06fd00 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:51.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:51.860+0000 7f1383800640 1 -- 172.21.15.16:0/650560655 >> 172.21.15.16:0/650560655 conn(0x7f137c067930 msgr2=0x7f137c0b3ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:04:51.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:51.860+0000 7f1383800640 1 -- 172.21.15.16:0/650560655 shutdown_connections 2024-03-31T15:04:51.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:51.860+0000 7f1383800640 1 -- 172.21.15.16:0/650560655 wait complete. 2024-03-31T15:04:51.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:51.860+0000 7f1383800640 1 Processor -- start 2024-03-31T15:04:51.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:51.860+0000 7f1383800640 1 -- start start 2024-03-31T15:04:51.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:51.860+0000 7f1383800640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f137c06f920 0x7f137c15ed90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:51.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:51.860+0000 7f1383800640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f137c0cef80 0x7f137c13e6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:51.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:51.860+0000 7f1383800640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f137c0d0a60 0x7f137c0ce6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:51.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:51.860+0000 7f1383800640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f137c1690b0 con 0x7f137c06f920 2024-03-31T15:04:51.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:51.860+0000 7f1383800640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f137c0cf390 con 0x7f137c0cef80 2024-03-31T15:04:51.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:51.860+0000 7f1383800640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f137c169230 con 0x7f137c0d0a60 2024-03-31T15:04:51.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:51.860+0000 7f1380e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f137c06f920 0x7f137c15ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:51.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:51.860+0000 7f1380e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f137c06f920 0x7f137c15ed90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51278/0 (socket says 172.21.15.16:51278) 2024-03-31T15:04:51.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:51.860+0000 7f1380e00640 1 -- 172.21.15.16:0/15949850 learned_addr learned my addr 172.21.15.16:0/15949850 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:04:51.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:51.860+0000 7f1381800640 1 --2- 172.21.15.16:0/15949850 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f137c0d0a60 0x7f137c0ce6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:51.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:51.860+0000 7f137be00640 1 --2- 172.21.15.16:0/15949850 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f137c0cef80 0x7f137c13e6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:51.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:51.860+0000 7f1380e00640 1 -- 172.21.15.16:0/15949850 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f137c0d0a60 msgr2=0x7f137c0ce6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:51.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:51.860+0000 7f1380e00640 1 --2- 172.21.15.16:0/15949850 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f137c0d0a60 0x7f137c0ce6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:51.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:51.860+0000 7f1380e00640 1 -- 172.21.15.16:0/15949850 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f137c0cef80 msgr2=0x7f137c13e6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:51.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:51.860+0000 7f1380e00640 1 --2- 172.21.15.16:0/15949850 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f137c0cef80 0x7f137c13e6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:51.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:51.860+0000 7f1380e00640 1 -- 172.21.15.16:0/15949850 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f137c13ec10 con 0x7f137c06f920 2024-03-31T15:04:51.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:51.860+0000 7f1380e00640 1 --2- 172.21.15.16:0/15949850 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f137c06f920 0x7f137c15ed90 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f13700ae040 tx=0x7f137001ac90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:51.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:51.864+0000 7f1378c00640 1 -- 172.21.15.16:0/15949850 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13700ba020 con 0x7f137c06f920 2024-03-31T15:04:51.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:51.864+0000 7f1378c00640 1 -- 172.21.15.16:0/15949850 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f13700a4e00 con 0x7f137c06f920 2024-03-31T15:04:51.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:51.864+0000 7f1378c00640 1 -- 172.21.15.16:0/15949850 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13700b0040 con 0x7f137c06f920 2024-03-31T15:04:51.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:51.864+0000 7f1383800640 1 -- 172.21.15.16:0/15949850 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f137c155340 con 0x7f137c06f920 2024-03-31T15:04:51.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:51.864+0000 7f1383800640 1 -- 172.21.15.16:0/15949850 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f137c1571d0 con 0x7f137c06f920 2024-03-31T15:04:51.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:51.864+0000 7f1383800640 1 -- 172.21.15.16:0/15949850 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f137c069950 con 0x7f137c06f920 2024-03-31T15:04:51.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:51.864+0000 7f1378c00640 1 -- 172.21.15.16:0/15949850 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f13700bd020 con 0x7f137c06f920 2024-03-31T15:04:51.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:51.864+0000 7f1378c00640 1 --2- 172.21.15.16:0/15949850 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f135c068380 0x7f135c06a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:51.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:51.864+0000 7f1378c00640 1 -- 172.21.15.16:0/15949850 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5709+0+0 (secure 0 0 0) 0x7f1370120b30 con 0x7f137c06f920 2024-03-31T15:04:51.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:51.864+0000 7f137be00640 1 --2- 172.21.15.16:0/15949850 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f135c068380 0x7f135c06a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:51.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:51.868+0000 7f137be00640 1 --2- 172.21.15.16:0/15949850 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f135c068380 0x7f135c06a840 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f137c153940 tx=0x7f1368008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:51.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:51.872+0000 7f1378c00640 1 -- 172.21.15.16:0/15949850 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f13700c4030 con 0x7f137c06f920 2024-03-31T15:04:52.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:52.036+0000 7f1383800640 1 -- 172.21.15.16:0/15949850 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f137c0b6260 con 0x7f137c06f920 2024-03-31T15:04:52.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:52.036+0000 7f1378c00640 1 -- 172.21.15.16:0/15949850 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v34) ==== 104+0+11 (secure 0 0 0) 0x7f13700160d0 con 0x7f137c06f920 2024-03-31T15:04:52.032 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2024-03-31T15:04:52.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:52.040+0000 7f1383800640 1 -- 172.21.15.16:0/15949850 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f135c068380 msgr2=0x7f135c06a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:52.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:52.040+0000 7f1383800640 1 --2- 172.21.15.16:0/15949850 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f135c068380 0x7f135c06a840 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f137c153940 tx=0x7f1368008040 comp rx=0 tx=0).stop 2024-03-31T15:04:52.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:52.040+0000 7f1383800640 1 -- 172.21.15.16:0/15949850 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f137c06f920 msgr2=0x7f137c15ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:52.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:52.040+0000 7f1383800640 1 --2- 172.21.15.16:0/15949850 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f137c06f920 0x7f137c15ed90 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f13700ae040 tx=0x7f137001ac90 comp rx=0 tx=0).stop 2024-03-31T15:04:52.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:52.040+0000 7f1383800640 1 -- 172.21.15.16:0/15949850 shutdown_connections 2024-03-31T15:04:52.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:52.040+0000 7f1383800640 1 --2- 172.21.15.16:0/15949850 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f135c068380 0x7f135c06a840 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:52.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:52.040+0000 7f1383800640 1 --2- 172.21.15.16:0/15949850 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f137c0d0a60 0x7f137c0ce6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:52.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:52.040+0000 7f1383800640 1 --2- 172.21.15.16:0/15949850 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f137c0cef80 0x7f137c13e6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:52.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:52.040+0000 7f1383800640 1 --2- 172.21.15.16:0/15949850 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f137c06f920 0x7f137c15ed90 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:52.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:52.040+0000 7f1383800640 1 -- 172.21.15.16:0/15949850 >> 172.21.15.16:0/15949850 conn(0x7f137c067930 msgr2=0x7f137c0b20d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:04:52.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:52.040+0000 7f1383800640 1 -- 172.21.15.16:0/15949850 shutdown_connections 2024-03-31T15:04:52.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:52.040+0000 7f1383800640 1 -- 172.21.15.16:0/15949850 wait complete. 2024-03-31T15:04:52.054 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-31T15:04:52.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:52.204+0000 7f422c800640 1 Processor -- start 2024-03-31T15:04:52.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:52.204+0000 7f422c800640 1 -- start start 2024-03-31T15:04:52.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:52.204+0000 7f422c800640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f422415c3d0 0x7f422406fc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:52.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:52.204+0000 7f422c800640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f42240d0880 0x7f42240b4250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:52.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:52.204+0000 7f422c800640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4224070520 con 0x7f42240cdc30 2024-03-31T15:04:52.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:52.204+0000 7f422c800640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f42240cfda0 con 0x7f42240d0880 2024-03-31T15:04:52.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:52.204+0000 7f422c800640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f42240d0080 con 0x7f422415c3d0 2024-03-31T15:04:52.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:52.204+0000 7f422a800640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f42240d0880 0x7f42240b4250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:52.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:52.204+0000 7f422a800640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f42240d0880 0x7f42240b4250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51284/0 (socket says 172.21.15.16:51284) 2024-03-31T15:04:52.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:52.204+0000 7f422a800640 1 -- 172.21.15.16:0/3196213267 learned_addr learned my addr 172.21.15.16:0/3196213267 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:04:52.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:52.204+0000 7f4229400640 1 --2- 172.21.15.16:0/3196213267 >> v2:172.21.15.89:3300/0 conn(0x7f422415c3d0 0x7f422406fc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:52.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:52.204+0000 7f422a800640 1 -- 172.21.15.16:0/3196213267 >> v2:172.21.15.89:3300/0 conn(0x7f422415c3d0 msgr2=0x7f422406fc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:04:52.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:52.204+0000 7f422a800640 1 --2- 172.21.15.16:0/3196213267 >> v2:172.21.15.89:3300/0 conn(0x7f422415c3d0 0x7f422406fc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:52.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:52.204+0000 7f422a800640 1 -- 172.21.15.16:0/3196213267 >> v1:172.21.15.89:6789/0 conn(0x7f42240cdc30 legacy=0x7f42240ce010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:52.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:52.204+0000 7f422a800640 1 -- 172.21.15.16:0/3196213267 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42240d0360 con 0x7f42240d0880 2024-03-31T15:04:52.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:52.208+0000 7f422a800640 1 --2- 172.21.15.16:0/3196213267 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f42240d0880 0x7f42240b4250 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f42240722f0 tx=0x7f422001ebf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e2114a74323267b server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:52.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:52.208+0000 7f4228a00640 1 -- 172.21.15.16:0/3196213267 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f422000b2d0 con 0x7f42240d0880 2024-03-31T15:04:52.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:52.208+0000 7f4228a00640 1 -- 172.21.15.16:0/3196213267 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f422000b470 con 0x7f42240d0880 2024-03-31T15:04:52.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:52.208+0000 7f4228a00640 1 -- 172.21.15.16:0/3196213267 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42200a7b10 con 0x7f42240d0880 2024-03-31T15:04:52.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:52.208+0000 7f422c800640 1 -- 172.21.15.16:0/3196213267 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f42240d0880 msgr2=0x7f42240b4250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:52.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:52.208+0000 7f422c800640 1 --2- 172.21.15.16:0/3196213267 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f42240d0880 0x7f42240b4250 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f42240722f0 tx=0x7f422001ebf0 comp rx=0 tx=0).stop 2024-03-31T15:04:52.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:52.208+0000 7f422c800640 1 -- 172.21.15.16:0/3196213267 shutdown_connections 2024-03-31T15:04:52.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:52.208+0000 7f422c800640 1 --2- 172.21.15.16:0/3196213267 >> v2:172.21.15.89:3300/0 conn(0x7f422415c3d0 0x7f422406fc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:52.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:52.208+0000 7f422c800640 1 --2- 172.21.15.16:0/3196213267 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f42240d0880 0x7f42240b4250 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:52.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:52.208+0000 7f422c800640 1 -- 172.21.15.16:0/3196213267 >> 172.21.15.16:0/3196213267 conn(0x7f422406e670 msgr2=0x7f422406ea50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:04:52.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:52.208+0000 7f422c800640 1 -- 172.21.15.16:0/3196213267 shutdown_connections 2024-03-31T15:04:52.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:52.208+0000 7f422c800640 1 -- 172.21.15.16:0/3196213267 wait complete. 2024-03-31T15:04:52.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:52.212+0000 7f422c800640 1 Processor -- start 2024-03-31T15:04:52.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:52.212+0000 7f422c800640 1 -- start start 2024-03-31T15:04:52.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:52.212+0000 7f422c800640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f42240cdc30 0x7f422413db40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:52.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:52.212+0000 7f422c800640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f42240d0880 0x7f42240b2750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:52.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:52.212+0000 7f422c800640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f422415c3d0 0x7f42240b5650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:52.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:52.212+0000 7f422c800640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f4224159100 con 0x7f422415c3d0 2024-03-31T15:04:52.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:52.212+0000 7f422c800640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f4224158f80 con 0x7f42240d0880 2024-03-31T15:04:52.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:52.212+0000 7f422c800640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f422406be30 con 0x7f42240cdc30 2024-03-31T15:04:52.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:52.212+0000 7f422a800640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f422415c3d0 0x7f42240b5650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:52.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:52.212+0000 7f422a800640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f422415c3d0 0x7f42240b5650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51286/0 (socket says 172.21.15.16:51286) 2024-03-31T15:04:52.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:52.212+0000 7f422a800640 1 -- 172.21.15.16:0/2466940735 learned_addr learned my addr 172.21.15.16:0/2466940735 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:04:52.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:52.212+0000 7f4229400640 1 --2- 172.21.15.16:0/2466940735 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f42240d0880 0x7f42240b2750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:52.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:52.212+0000 7f4229e00640 1 --2- 172.21.15.16:0/2466940735 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f42240cdc30 0x7f422413db40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:52.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:52.212+0000 7f422a800640 1 -- 172.21.15.16:0/2466940735 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f42240cdc30 msgr2=0x7f422413db40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:52.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:52.212+0000 7f422a800640 1 --2- 172.21.15.16:0/2466940735 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f42240cdc30 0x7f422413db40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:52.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:52.212+0000 7f422a800640 1 -- 172.21.15.16:0/2466940735 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f42240d0880 msgr2=0x7f42240b2750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:52.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:52.212+0000 7f422a800640 1 --2- 172.21.15.16:0/2466940735 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f42240d0880 0x7f42240b2750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:52.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:52.212+0000 7f422a800640 1 -- 172.21.15.16:0/2466940735 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f422414edd0 con 0x7f422415c3d0 2024-03-31T15:04:52.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:52.212+0000 7f422a800640 1 --2- 172.21.15.16:0/2466940735 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f422415c3d0 0x7f42240b5650 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f422001c820 tx=0x7f422001b990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:52.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:52.212+0000 7f421e000640 1 -- 172.21.15.16:0/2466940735 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4220009800 con 0x7f422415c3d0 2024-03-31T15:04:52.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:52.212+0000 7f421e000640 1 -- 172.21.15.16:0/2466940735 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f42200be070 con 0x7f422415c3d0 2024-03-31T15:04:52.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:52.212+0000 7f421e000640 1 -- 172.21.15.16:0/2466940735 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4220009c20 con 0x7f422415c3d0 2024-03-31T15:04:52.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:52.212+0000 7f422c800640 1 -- 172.21.15.16:0/2466940735 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f422414f0b0 con 0x7f422415c3d0 2024-03-31T15:04:52.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:52.212+0000 7f422c800640 1 -- 172.21.15.16:0/2466940735 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f422414f670 con 0x7f422415c3d0 2024-03-31T15:04:52.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:52.216+0000 7f422c800640 1 -- 172.21.15.16:0/2466940735 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4224069950 con 0x7f422415c3d0 2024-03-31T15:04:52.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:52.216+0000 7f421e000640 1 -- 172.21.15.16:0/2466940735 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f42200b6070 con 0x7f422415c3d0 2024-03-31T15:04:52.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:52.216+0000 7f421e000640 1 --2- 172.21.15.16:0/2466940735 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f4208068380 0x7f420806a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:52.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:52.216+0000 7f421e000640 1 -- 172.21.15.16:0/2466940735 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5709+0+0 (secure 0 0 0) 0x7f4220122740 con 0x7f422415c3d0 2024-03-31T15:04:52.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:52.216+0000 7f4229e00640 1 --2- 172.21.15.16:0/2466940735 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f4208068380 0x7f420806a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:52.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:52.216+0000 7f4229e00640 1 --2- 172.21.15.16:0/2466940735 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f4208068380 0x7f420806a840 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f4218001810 tx=0x7f4218009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:52.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:52.220+0000 7f421e000640 1 -- 172.21.15.16:0/2466940735 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f42200187e0 con 0x7f422415c3d0 2024-03-31T15:04:52.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:52.384+0000 7f422c800640 1 -- 172.21.15.16:0/2466940735 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f42240b7530 con 0x7f422415c3d0 2024-03-31T15:04:52.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:52.384+0000 7f421e000640 1 -- 172.21.15.16:0/2466940735 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v34) ==== 100+0+11 (secure 0 0 0) 0x7f42200f2970 con 0x7f422415c3d0 2024-03-31T15:04:52.378 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 64 2024-03-31T15:04:52.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:52.384+0000 7f422c800640 1 -- 172.21.15.16:0/2466940735 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f4208068380 msgr2=0x7f420806a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:52.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:52.384+0000 7f422c800640 1 --2- 172.21.15.16:0/2466940735 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f4208068380 0x7f420806a840 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f4218001810 tx=0x7f4218009040 comp rx=0 tx=0).stop 2024-03-31T15:04:52.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:52.384+0000 7f422c800640 1 -- 172.21.15.16:0/2466940735 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f422415c3d0 msgr2=0x7f42240b5650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:52.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:52.384+0000 7f422c800640 1 --2- 172.21.15.16:0/2466940735 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f422415c3d0 0x7f42240b5650 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f422001c820 tx=0x7f422001b990 comp rx=0 tx=0).stop 2024-03-31T15:04:52.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:52.384+0000 7f422c800640 1 -- 172.21.15.16:0/2466940735 shutdown_connections 2024-03-31T15:04:52.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:52.384+0000 7f422c800640 1 --2- 172.21.15.16:0/2466940735 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f4208068380 0x7f420806a840 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:52.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:52.384+0000 7f422c800640 1 --2- 172.21.15.16:0/2466940735 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f422415c3d0 0x7f42240b5650 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:52.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:52.384+0000 7f422c800640 1 --2- 172.21.15.16:0/2466940735 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f42240d0880 0x7f42240b2750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:52.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:52.384+0000 7f422c800640 1 --2- 172.21.15.16:0/2466940735 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f42240cdc30 0x7f422413db40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:52.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:52.384+0000 7f422c800640 1 -- 172.21.15.16:0/2466940735 >> 172.21.15.16:0/2466940735 conn(0x7f422406e670 msgr2=0x7f4224164350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:04:52.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:52.384+0000 7f422c800640 1 -- 172.21.15.16:0/2466940735 shutdown_connections 2024-03-31T15:04:52.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:52.384+0000 7f422c800640 1 -- 172.21.15.16:0/2466940735 wait complete. 2024-03-31T15:04:52.400 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-31T15:04:52.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:52.552+0000 7fbfb3e00640 1 Processor -- start 2024-03-31T15:04:52.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:52.552+0000 7fbfb3e00640 1 -- start start 2024-03-31T15:04:52.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:52.552+0000 7fbfb3e00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fbfb415bf80 0x7fbfb4070070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:52.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:52.552+0000 7fbfb3e00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fbfb40726d0 0x7fbfb40cfd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:52.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:52.552+0000 7fbfb3e00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fbfb406c630 0x7fbfb406ca80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:52.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:52.552+0000 7fbfb3e00640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7fbfb406cfc0 con 0x7fbfb40726d0 2024-03-31T15:04:52.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:52.552+0000 7fbfb3e00640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7fbfb40cf890 con 0x7fbfb415bf80 2024-03-31T15:04:52.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:52.552+0000 7fbfb3e00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fbfb40cefc0 con 0x7fbfb406c630 2024-03-31T15:04:52.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:52.556+0000 7fbfaa000640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fbfb40726d0 0x7fbfb40cfd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:52.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:52.556+0000 7fbfaa000640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fbfb40726d0 0x7fbfb40cfd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51294/0 (socket says 172.21.15.16:51294) 2024-03-31T15:04:52.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:52.556+0000 7fbfaa000640 1 -- 172.21.15.16:0/4156203896 learned_addr learned my addr 172.21.15.16:0/4156203896 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:04:52.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:52.556+0000 7fbfb2a00640 1 --2- 172.21.15.16:0/4156203896 >> v2:172.21.15.89:3300/0 conn(0x7fbfb415bf80 0x7fbfb4070070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:52.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:52.556+0000 7fbfb3400640 1 --2- 172.21.15.16:0/4156203896 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fbfb406c630 0x7fbfb406ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:52.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:52.556+0000 7fbfaa000640 1 -- 172.21.15.16:0/4156203896 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fbfb406c630 msgr2=0x7fbfb406ca80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:04:52.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:52.556+0000 7fbfaa000640 1 --2- 172.21.15.16:0/4156203896 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fbfb406c630 0x7fbfb406ca80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:52.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:52.556+0000 7fbfaa000640 1 -- 172.21.15.16:0/4156203896 >> v2:172.21.15.89:3300/0 conn(0x7fbfb415bf80 msgr2=0x7fbfb4070070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:04:52.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:52.556+0000 7fbfaa000640 1 --2- 172.21.15.16:0/4156203896 >> v2:172.21.15.89:3300/0 conn(0x7fbfb415bf80 0x7fbfb4070070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:52.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:52.556+0000 7fbfaa000640 1 -- 172.21.15.16:0/4156203896 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbfb40705b0 con 0x7fbfb40726d0 2024-03-31T15:04:52.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:52.556+0000 7fbfb2a00640 1 --2- 172.21.15.16:0/4156203896 >> v2:172.21.15.89:3300/0 conn(0x7fbfb415bf80 0x7fbfb4070070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:04:52.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:52.556+0000 7fbfaa000640 1 --2- 172.21.15.16:0/4156203896 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbfb40726d0 0x7fbfb40cfd20 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fbfb4073430 tx=0x7fbf9c01c460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5473d81a31136463 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:52.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:52.556+0000 7fbfb2000640 1 -- 172.21.15.16:0/4156203896 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf9c0ab620 con 0x7fbfb40726d0 2024-03-31T15:04:52.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:52.556+0000 7fbfb2000640 1 -- 172.21.15.16:0/4156203896 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbf9c0b0070 con 0x7fbfb40726d0 2024-03-31T15:04:52.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:52.556+0000 7fbfb2000640 1 -- 172.21.15.16:0/4156203896 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf9c0ab9e0 con 0x7fbfb40726d0 2024-03-31T15:04:52.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:52.560+0000 7fbfb3e00640 1 -- 172.21.15.16:0/4156203896 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbfb40726d0 msgr2=0x7fbfb40cfd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:52.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:52.560+0000 7fbfb3e00640 1 --2- 172.21.15.16:0/4156203896 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbfb40726d0 0x7fbfb40cfd20 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fbfb4073430 tx=0x7fbf9c01c460 comp rx=0 tx=0).stop 2024-03-31T15:04:52.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:52.560+0000 7fbfb3e00640 1 -- 172.21.15.16:0/4156203896 shutdown_connections 2024-03-31T15:04:52.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:52.560+0000 7fbfb3e00640 1 --2- 172.21.15.16:0/4156203896 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fbfb406c630 0x7fbfb406ca80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:52.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:52.560+0000 7fbfb3e00640 1 --2- 172.21.15.16:0/4156203896 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbfb40726d0 0x7fbfb40cfd20 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:52.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:52.560+0000 7fbfb3e00640 1 --2- 172.21.15.16:0/4156203896 >> v2:172.21.15.89:3300/0 conn(0x7fbfb415bf80 0x7fbfb4070070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:52.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:52.560+0000 7fbfb3e00640 1 -- 172.21.15.16:0/4156203896 >> 172.21.15.16:0/4156203896 conn(0x7fbfb4071bb0 msgr2=0x7fbfb4071fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:04:52.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:52.560+0000 7fbfb3e00640 1 -- 172.21.15.16:0/4156203896 shutdown_connections 2024-03-31T15:04:52.554 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:52.560+0000 7fbfb3e00640 1 -- 172.21.15.16:0/4156203896 wait complete. 2024-03-31T15:04:52.554 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:52.560+0000 7fbfb3e00640 1 Processor -- start 2024-03-31T15:04:52.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:52.560+0000 7fbfb3e00640 1 -- start start 2024-03-31T15:04:52.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:52.560+0000 7fbfb3e00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbfb406c630 0x7fbfb414aa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:52.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:52.560+0000 7fbfb3e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbfb40726d0 0x7fbfb4150300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:52.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:52.560+0000 7fbfb3e00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fbfb415bf80 0x7fbfb406b620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:52.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:52.560+0000 7fbfb3e00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fbfb415c4e0 con 0x7fbfb40726d0 2024-03-31T15:04:52.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:52.560+0000 7fbfb3e00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fbfb415c360 con 0x7fbfb406c630 2024-03-31T15:04:52.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:52.560+0000 7fbfb3e00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fbfb40d0260 con 0x7fbfb415bf80 2024-03-31T15:04:52.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:52.560+0000 7fbfaa000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbfb40726d0 0x7fbfb4150300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:52.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:52.560+0000 7fbfaa000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbfb40726d0 0x7fbfb4150300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51298/0 (socket says 172.21.15.16:51298) 2024-03-31T15:04:52.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:52.560+0000 7fbfaa000640 1 -- 172.21.15.16:0/4081270759 learned_addr learned my addr 172.21.15.16:0/4081270759 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:04:52.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:52.560+0000 7fbfb2a00640 1 --2- 172.21.15.16:0/4081270759 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbfb406c630 0x7fbfb414aa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:52.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:52.560+0000 7fbfb3400640 1 --2- 172.21.15.16:0/4081270759 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fbfb415bf80 0x7fbfb406b620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:52.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:52.564+0000 7fbfaa000640 1 -- 172.21.15.16:0/4081270759 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fbfb415bf80 msgr2=0x7fbfb406b620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:52.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:52.564+0000 7fbfaa000640 1 --2- 172.21.15.16:0/4081270759 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fbfb415bf80 0x7fbfb406b620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:52.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:52.564+0000 7fbfaa000640 1 -- 172.21.15.16:0/4081270759 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbfb406c630 msgr2=0x7fbfb414aa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:52.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:52.564+0000 7fbfaa000640 1 --2- 172.21.15.16:0/4081270759 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbfb406c630 0x7fbfb414aa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:52.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:52.564+0000 7fbfaa000640 1 -- 172.21.15.16:0/4081270759 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbfb4150d80 con 0x7fbfb40726d0 2024-03-31T15:04:52.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:52.564+0000 7fbfaa000640 1 --2- 172.21.15.16:0/4081270759 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbfb40726d0 0x7fbfb4150300 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fbf9c01ab10 tx=0x7fbf9c01aca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:52.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:52.564+0000 7fbfabe00640 1 -- 172.21.15.16:0/4081270759 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf9c0abdb0 con 0x7fbfb40726d0 2024-03-31T15:04:52.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:52.564+0000 7fbfabe00640 1 -- 172.21.15.16:0/4081270759 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbf9c0be3f0 con 0x7fbfb40726d0 2024-03-31T15:04:52.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:52.564+0000 7fbfabe00640 1 -- 172.21.15.16:0/4081270759 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf9c0be7a0 con 0x7fbfb40726d0 2024-03-31T15:04:52.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:52.564+0000 7fbfb3e00640 1 -- 172.21.15.16:0/4081270759 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbfb415dcc0 con 0x7fbfb40726d0 2024-03-31T15:04:52.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:52.564+0000 7fbfb3e00640 1 -- 172.21.15.16:0/4081270759 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbfb414b4d0 con 0x7fbfb40726d0 2024-03-31T15:04:52.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:52.564+0000 7fbfb3e00640 1 -- 172.21.15.16:0/4081270759 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbfb404f8c0 con 0x7fbfb40726d0 2024-03-31T15:04:52.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:52.564+0000 7fbfabe00640 1 -- 172.21.15.16:0/4081270759 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fbf9c0b43f0 con 0x7fbfb40726d0 2024-03-31T15:04:52.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:52.568+0000 7fbfabe00640 1 --2- 172.21.15.16:0/4081270759 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fbf90068380 0x7fbf9006a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:52.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:52.568+0000 7fbfabe00640 1 -- 172.21.15.16:0/4081270759 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5709+0+0 (secure 0 0 0) 0x7fbf9c120910 con 0x7fbfb40726d0 2024-03-31T15:04:52.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:52.568+0000 7fbfb2a00640 1 --2- 172.21.15.16:0/4081270759 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fbf90068380 0x7fbf9006a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:52.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:52.568+0000 7fbfb2a00640 1 --2- 172.21.15.16:0/4081270759 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fbf90068380 0x7fbf9006a840 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fbfb41519c0 tx=0x7fbfa4008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:52.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:52.572+0000 7fbfabe00640 1 -- 172.21.15.16:0/4081270759 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fbfb40705b0 con 0x7fbfb40726d0 2024-03-31T15:04:52.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:52.748+0000 7fbfb3e00640 1 -- 172.21.15.16:0/4081270759 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fbfb40678f0 con 0x7fbfb40726d0 2024-03-31T15:04:52.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:52.748+0000 7fbfabe00640 1 -- 172.21.15.16:0/4081270759 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v34) ==== 80+0+0 (secure 0 0 0) 0x7fbf9c016110 con 0x7fbfb40726d0 2024-03-31T15:04:52.744 INFO:teuthology.orchestra.run.smithi016.stderr:listed 0 entries 2024-03-31T15:04:52.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:52.752+0000 7fbfb3e00640 1 -- 172.21.15.16:0/4081270759 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fbf90068380 msgr2=0x7fbf9006a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:52.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:52.752+0000 7fbfb3e00640 1 --2- 172.21.15.16:0/4081270759 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fbf90068380 0x7fbf9006a840 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fbfb41519c0 tx=0x7fbfa4008040 comp rx=0 tx=0).stop 2024-03-31T15:04:52.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:52.752+0000 7fbfb3e00640 1 -- 172.21.15.16:0/4081270759 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbfb40726d0 msgr2=0x7fbfb4150300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:52.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:52.752+0000 7fbfb3e00640 1 --2- 172.21.15.16:0/4081270759 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbfb40726d0 0x7fbfb4150300 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fbf9c01ab10 tx=0x7fbf9c01aca0 comp rx=0 tx=0).stop 2024-03-31T15:04:52.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:52.752+0000 7fbfb3e00640 1 -- 172.21.15.16:0/4081270759 shutdown_connections 2024-03-31T15:04:52.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:52.752+0000 7fbfb3e00640 1 --2- 172.21.15.16:0/4081270759 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fbf90068380 0x7fbf9006a840 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:52.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:52.752+0000 7fbfb3e00640 1 --2- 172.21.15.16:0/4081270759 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fbfb415bf80 0x7fbfb406b620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:52.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:52.752+0000 7fbfb3e00640 1 --2- 172.21.15.16:0/4081270759 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbfb40726d0 0x7fbfb4150300 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:52.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:52.752+0000 7fbfb3e00640 1 --2- 172.21.15.16:0/4081270759 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbfb406c630 0x7fbfb414aa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:52.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:52.752+0000 7fbfb3e00640 1 -- 172.21.15.16:0/4081270759 >> 172.21.15.16:0/4081270759 conn(0x7fbfb4071bb0 msgr2=0x7fbfb41554f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:04:52.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:52.752+0000 7fbfb3e00640 1 -- 172.21.15.16:0/4081270759 shutdown_connections 2024-03-31T15:04:52.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:52.752+0000 7fbfb3e00640 1 -- 172.21.15.16:0/4081270759 wait complete. 2024-03-31T15:04:52.765 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi016.front.sepia.ceph.com... 2024-03-31T15:04:52.765 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:04:52.766 DEBUG:teuthology.orchestra.run.smithi016:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-03-31T15:04:52.784 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi016.stderr:ceph-fuse[36341]: fuse finished with error 0 and tester_r 0 2024-03-31T15:04:52.787 INFO:teuthology.orchestra.run:waiting for 300 2024-03-31T15:04:58.889 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi016.front.sepia.ceph.com 2024-03-31T15:04:58.889 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:04:58.889 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-31T15:04:58.899 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-31T15:04:59.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:59.084+0000 7f0cdd200640 1 Processor -- start 2024-03-31T15:04:59.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:59.084+0000 7f0cdd200640 1 -- start start 2024-03-31T15:04:59.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:59.088+0000 7f0cdd200640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f0cd8145d10 0x7f0cd80cf790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:59.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:59.088+0000 7f0cdd200640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0cd8160e70 con 0x7f0cd80b23d0 2024-03-31T15:04:59.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:59.088+0000 7f0cdd200640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0cd80cfcd0 con 0x7f0cd80728b0 2024-03-31T15:04:59.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:59.088+0000 7f0cdd200640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f0cd81602f0 con 0x7f0cd8145d10 2024-03-31T15:04:59.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:59.088+0000 7f0cd6800640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f0cd80b23d0 0x7f0cd815cd50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:40902/0 (socket says 172.21.15.16:40902) 2024-03-31T15:04:59.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:59.088+0000 7f0cd6800640 1 -- 172.21.15.16:0/3582200163 learned_addr learned my addr 172.21.15.16:0/3582200163 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:04:59.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:59.088+0000 7f0cd5e00640 1 --2- 172.21.15.16:0/3582200163 >> v2:172.21.15.89:3300/0 conn(0x7f0cd8145d10 0x7f0cd80cf790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:59.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:59.088+0000 7f0cd5400640 1 -- 172.21.15.16:0/3582200163 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4188313584 0 0) 0x7f0cd8160e70 con 0x7f0cd80b23d0 2024-03-31T15:04:59.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:59.088+0000 7f0cd5400640 1 -- 172.21.15.16:0/3582200163 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0cbc003530 con 0x7f0cd80b23d0 2024-03-31T15:04:59.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:59.088+0000 7f0cd5400640 1 -- 172.21.15.16:0/3582200163 <== mon.1 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 639126255 0 0) 0x7f0cd80cfcd0 con 0x7f0cd80728b0 2024-03-31T15:04:59.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:59.088+0000 7f0cd5400640 1 -- 172.21.15.16:0/3582200163 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0cd8160e70 con 0x7f0cd80728b0 2024-03-31T15:04:59.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:59.088+0000 7f0cd5e00640 1 -- 172.21.15.16:0/3582200163 >> v1:172.21.15.89:6789/0 conn(0x7f0cd80728b0 legacy=0x7f0cd80b1cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:59.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:59.088+0000 7f0cd5e00640 1 -- 172.21.15.16:0/3582200163 >> v1:172.21.15.16:6789/0 conn(0x7f0cd80b23d0 legacy=0x7f0cd815cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:59.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:59.088+0000 7f0cd5400640 1 -- 172.21.15.16:0/3582200163 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 644660724 0 0) 0x7f0cbc003530 con 0x7f0cd80b23d0 2024-03-31T15:04:59.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:59.088+0000 7f0cd5e00640 1 -- 172.21.15.16:0/3582200163 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0cd81605d0 con 0x7f0cd8145d10 2024-03-31T15:04:59.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:59.092+0000 7f0cd5e00640 1 --2- 172.21.15.16:0/3582200163 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0cd8145d10 0x7f0cd80cf790 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f0cc80097c0 tx=0x7f0cc801c3c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=473f5c618497e85d server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:59.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:59.092+0000 7f0cd5400640 1 -- 172.21.15.16:0/3582200163 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cc80b0070 con 0x7f0cd8145d10 2024-03-31T15:04:59.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:59.092+0000 7f0cd5400640 1 -- 172.21.15.16:0/3582200163 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0cc80ab400 con 0x7f0cd8145d10 2024-03-31T15:04:59.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:59.092+0000 7f0cd5400640 1 -- 172.21.15.16:0/3582200163 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cd0001a00 con 0x7f0cd8145d10 2024-03-31T15:04:59.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:59.092+0000 7f0cdd200640 1 -- 172.21.15.16:0/3582200163 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0cd8145d10 msgr2=0x7f0cd80cf790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:59.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:59.092+0000 7f0cdd200640 1 --2- 172.21.15.16:0/3582200163 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0cd8145d10 0x7f0cd80cf790 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f0cc80097c0 tx=0x7f0cc801c3c0 comp rx=0 tx=0).stop 2024-03-31T15:04:59.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:59.092+0000 7f0cdd200640 1 -- 172.21.15.16:0/3582200163 shutdown_connections 2024-03-31T15:04:59.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:59.092+0000 7f0cdd200640 1 --2- 172.21.15.16:0/3582200163 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0cd8145d10 0x7f0cd80cf790 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:59.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:59.092+0000 7f0cdd200640 1 -- 172.21.15.16:0/3582200163 >> 172.21.15.16:0/3582200163 conn(0x7f0cd806b620 msgr2=0x7f0cd806ba30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:04:59.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:59.092+0000 7f0cdd200640 1 -- 172.21.15.16:0/3582200163 shutdown_connections 2024-03-31T15:04:59.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:59.092+0000 7f0cdd200640 1 -- 172.21.15.16:0/3582200163 wait complete. 2024-03-31T15:04:59.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:59.092+0000 7f0cdd200640 1 Processor -- start 2024-03-31T15:04:59.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:59.092+0000 7f0cdd200640 1 -- start start 2024-03-31T15:04:59.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:59.096+0000 7f0cdd200640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0cd80728b0 0x7f0cd81545c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:59.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:59.096+0000 7f0cdd200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0cd80b23d0 0x7f0cd80b0520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:59.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:59.096+0000 7f0cdd200640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f0cd8145d10 0x7f0cd80b0a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:59.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:59.096+0000 7f0cdd200640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f0cd80cf350 con 0x7f0cd80b23d0 2024-03-31T15:04:59.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:59.096+0000 7f0cdd200640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f0cd815d870 con 0x7f0cd80728b0 2024-03-31T15:04:59.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:59.096+0000 7f0cdd200640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f0cd80cf4d0 con 0x7f0cd8145d10 2024-03-31T15:04:59.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:59.096+0000 7f0ccfe00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0cd80b23d0 0x7f0cd80b0520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:59.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:59.096+0000 7f0ccfe00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0cd80b23d0 0x7f0cd80b0520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34906/0 (socket says 172.21.15.16:34906) 2024-03-31T15:04:59.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:59.096+0000 7f0ccfe00640 1 -- 172.21.15.16:0/1615425328 learned_addr learned my addr 172.21.15.16:0/1615425328 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:04:59.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:59.096+0000 7f0cd5e00640 1 --2- 172.21.15.16:0/1615425328 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0cd80728b0 0x7f0cd81545c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:59.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:59.096+0000 7f0cd6800640 1 --2- 172.21.15.16:0/1615425328 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f0cd8145d10 0x7f0cd80b0a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:59.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:59.096+0000 7f0ccfe00640 1 -- 172.21.15.16:0/1615425328 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f0cd8145d10 msgr2=0x7f0cd80b0a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:59.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:59.096+0000 7f0ccfe00640 1 --2- 172.21.15.16:0/1615425328 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f0cd8145d10 0x7f0cd80b0a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:59.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:59.096+0000 7f0ccfe00640 1 -- 172.21.15.16:0/1615425328 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0cd80728b0 msgr2=0x7f0cd81545c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:59.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:59.096+0000 7f0ccfe00640 1 --2- 172.21.15.16:0/1615425328 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0cd80728b0 0x7f0cd81545c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:59.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:59.096+0000 7f0ccfe00640 1 -- 172.21.15.16:0/1615425328 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0cd815af30 con 0x7f0cd80b23d0 2024-03-31T15:04:59.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:59.096+0000 7f0ccfe00640 1 --2- 172.21.15.16:0/1615425328 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0cd80b23d0 0x7f0cd80b0520 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f0cc000c530 tx=0x7f0cc000c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:59.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:59.096+0000 7f0cce000640 1 -- 172.21.15.16:0/1615425328 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cc00052d0 con 0x7f0cd80b23d0 2024-03-31T15:04:59.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:59.096+0000 7f0cce000640 1 -- 172.21.15.16:0/1615425328 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0cc0005490 con 0x7f0cd80b23d0 2024-03-31T15:04:59.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:59.096+0000 7f0cce000640 1 -- 172.21.15.16:0/1615425328 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cc0012850 con 0x7f0cd80b23d0 2024-03-31T15:04:59.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:59.096+0000 7f0cdd200640 1 -- 172.21.15.16:0/1615425328 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0cd815b210 con 0x7f0cd80b23d0 2024-03-31T15:04:59.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:59.096+0000 7f0cdd200640 1 -- 172.21.15.16:0/1615425328 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0cd815b7d0 con 0x7f0cd80b23d0 2024-03-31T15:04:59.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:59.096+0000 7f0cdd200640 1 -- 172.21.15.16:0/1615425328 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0cd8069950 con 0x7f0cd80b23d0 2024-03-31T15:04:59.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:59.100+0000 7f0cce000640 1 -- 172.21.15.16:0/1615425328 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f0cc00129f0 con 0x7f0cd80b23d0 2024-03-31T15:04:59.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:59.100+0000 7f0cce000640 1 --2- 172.21.15.16:0/1615425328 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f0cb4068380 0x7f0cb406a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:59.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:59.100+0000 7f0cce000640 1 -- 172.21.15.16:0/1615425328 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5709+0+0 (secure 0 0 0) 0x7f0cc0086550 con 0x7f0cd80b23d0 2024-03-31T15:04:59.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:59.100+0000 7f0cd5e00640 1 --2- 172.21.15.16:0/1615425328 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f0cb4068380 0x7f0cb406a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:59.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:59.100+0000 7f0cd5e00640 1 --2- 172.21.15.16:0/1615425328 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f0cb4068380 0x7f0cb406a840 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f0cc80ad000 tx=0x7f0cc8008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:59.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:59.108+0000 7f0cce000640 1 -- 172.21.15.16:0/1615425328 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f0cc0056430 con 0x7f0cd80b23d0 2024-03-31T15:04:59.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:59.304+0000 7f0cdd200640 1 -- 172.21.15.16:0/1615425328 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0cd8155620 con 0x7f0cd80b23d0 2024-03-31T15:04:59.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:59.304+0000 7f0cce000640 1 -- 172.21.15.16:0/1615425328 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) ==== 94+0+5178 (secure 0 0 0) 0x7f0cc005a730 con 0x7f0cd80b23d0 2024-03-31T15:04:59.300 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:04:59.300 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":16,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4949,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6836/2073954753","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":2073954753},{"type":"v1","addr":"172.21.15.89:6836","nonce":2073954753}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":4952,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/2143918278","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6835","nonce":2143918278},{"type":"v1","addr":"172.21.15.89:6837","nonce":2143918278}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":5251,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6833/881391342","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":881391342},{"type":"v1","addr":"172.21.15.16:6833","nonce":881391342}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-31T15:04:39.780323+0000","modified":"2024-03-31T15:04:42.846439+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5254},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5254":{"gid":5254,"name":"c","rank":0,"incarnation":13,"state":"up:active","state_seq":3,"addr":"172.21.15.16:6835/1845707550","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":1845707550},{"type":"v1","addr":"172.21.15.16:6835","nonce":1845707550}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5254,"qdb_cluster":[5254]},"id":2}]} 2024-03-31T15:04:59.300 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 16 2024-03-31T15:04:59.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:59.308+0000 7f0cdd200640 1 -- 172.21.15.16:0/1615425328 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f0cb4068380 msgr2=0x7f0cb406a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:59.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:59.308+0000 7f0cdd200640 1 --2- 172.21.15.16:0/1615425328 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f0cb4068380 0x7f0cb406a840 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f0cc80ad000 tx=0x7f0cc8008040 comp rx=0 tx=0).stop 2024-03-31T15:04:59.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:59.308+0000 7f0cdd200640 1 -- 172.21.15.16:0/1615425328 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0cd80b23d0 msgr2=0x7f0cd80b0520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:59.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:59.308+0000 7f0cdd200640 1 --2- 172.21.15.16:0/1615425328 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0cd80b23d0 0x7f0cd80b0520 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f0cc000c530 tx=0x7f0cc000c9f0 comp rx=0 tx=0).stop 2024-03-31T15:04:59.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:59.308+0000 7f0cdd200640 1 -- 172.21.15.16:0/1615425328 shutdown_connections 2024-03-31T15:04:59.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:59.308+0000 7f0cdd200640 1 --2- 172.21.15.16:0/1615425328 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f0cb4068380 0x7f0cb406a840 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:59.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:59.308+0000 7f0cdd200640 1 --2- 172.21.15.16:0/1615425328 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f0cd8145d10 0x7f0cd80b0a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:59.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:59.308+0000 7f0cdd200640 1 --2- 172.21.15.16:0/1615425328 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0cd80b23d0 0x7f0cd80b0520 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:59.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:59.308+0000 7f0cdd200640 1 --2- 172.21.15.16:0/1615425328 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0cd80728b0 0x7f0cd81545c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:59.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:59.308+0000 7f0cdd200640 1 -- 172.21.15.16:0/1615425328 >> 172.21.15.16:0/1615425328 conn(0x7f0cd806b620 msgr2=0x7f0cd815e830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:04:59.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:59.308+0000 7f0cdd200640 1 -- 172.21.15.16:0/1615425328 shutdown_connections 2024-03-31T15:04:59.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:59.308+0000 7f0cdd200640 1 -- 172.21.15.16:0/1615425328 wait complete. 2024-03-31T15:04:59.322 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json flush journal 2024-03-31T15:04:59.495 INFO:teuthology.orchestra.run.smithi016.stdout:{"message":"","return_code":0} 2024-03-31T15:04:59.495 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2024-03-31T15:04:59.495 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-31T15:04:59.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:59.640+0000 7f8ab2400640 1 Processor -- start 2024-03-31T15:04:59.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:59.644+0000 7f8ab2400640 1 -- start start 2024-03-31T15:04:59.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:59.644+0000 7f8ab2400640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f8aac06f920 0x7f8aac06fd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:59.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:59.644+0000 7f8ab2400640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f8aac0d2010 0x7f8aac0d24a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:59.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:59.644+0000 7f8ab2400640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8aac0d0920 con 0x7f8aac145e40 2024-03-31T15:04:59.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:59.644+0000 7f8ab2400640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f8aac0702c0 con 0x7f8aac0d2010 2024-03-31T15:04:59.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:59.644+0000 7f8ab2400640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f8aac0d06d0 con 0x7f8aac06f920 2024-03-31T15:04:59.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:59.644+0000 7f8aabe00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f8aac0d2010 0x7f8aac0d24a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:59.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:59.644+0000 7f8aaaa00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f8aac06f920 0x7f8aac06fd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:59.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:59.644+0000 7f8aab400640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f8aac145e40 0x7f8aac0cc740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.16:39966/0 (socket says 172.21.15.16:39966) 2024-03-31T15:04:59.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:59.644+0000 7f8aab400640 1 -- 172.21.15.16:0/1267150814 learned_addr learned my addr 172.21.15.16:0/1267150814 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:04:59.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:59.644+0000 7f8aabe00640 1 -- 172.21.15.16:0/1267150814 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f8aac06f920 msgr2=0x7f8aac06fd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:04:59.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:59.644+0000 7f8aabe00640 1 --2- 172.21.15.16:0/1267150814 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f8aac06f920 0x7f8aac06fd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:59.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:59.644+0000 7f8aabe00640 1 -- 172.21.15.16:0/1267150814 >> v1:172.21.15.89:6789/0 conn(0x7f8aac145e40 legacy=0x7f8aac0cc740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:59.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:59.644+0000 7f8aabe00640 1 -- 172.21.15.16:0/1267150814 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8aac0d0c00 con 0x7f8aac0d2010 2024-03-31T15:04:59.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:59.644+0000 7f8aaaa00640 1 --2- 172.21.15.16:0/1267150814 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f8aac06f920 0x7f8aac06fd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:04:59.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:59.644+0000 7f8aaa000640 1 -- 172.21.15.16:0/1267150814 <== mon.1 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2311072205 0 0) 0x7f8aac0d0920 con 0x7f8aac145e40 2024-03-31T15:04:59.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:59.648+0000 7f8aabe00640 1 --2- 172.21.15.16:0/1267150814 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8aac0d2010 0x7f8aac0d24a0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f8aa4009d90 tx=0x7f8aa401c970 comp rx=0 tx=0).ready entity=mon.1 client_cookie=32643df0dcacf1a7 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:59.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:59.648+0000 7f8aaa000640 1 -- 172.21.15.16:0/1267150814 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8aa40b2070 con 0x7f8aac0d2010 2024-03-31T15:04:59.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:59.648+0000 7f8aaa000640 1 -- 172.21.15.16:0/1267150814 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8aa40a5920 con 0x7f8aac0d2010 2024-03-31T15:04:59.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:59.648+0000 7f8aaa000640 1 -- 172.21.15.16:0/1267150814 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8aa40a5cf0 con 0x7f8aac0d2010 2024-03-31T15:04:59.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:59.648+0000 7f8ab2400640 1 -- 172.21.15.16:0/1267150814 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8aac0d2010 msgr2=0x7f8aac0d24a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:59.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:59.648+0000 7f8ab2400640 1 --2- 172.21.15.16:0/1267150814 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8aac0d2010 0x7f8aac0d24a0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f8aa4009d90 tx=0x7f8aa401c970 comp rx=0 tx=0).stop 2024-03-31T15:04:59.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:59.648+0000 7f8ab2400640 1 -- 172.21.15.16:0/1267150814 shutdown_connections 2024-03-31T15:04:59.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:59.648+0000 7f8ab2400640 1 --2- 172.21.15.16:0/1267150814 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8aac0d2010 0x7f8aac0d24a0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:59.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:59.648+0000 7f8ab2400640 1 --2- 172.21.15.16:0/1267150814 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f8aac06f920 0x7f8aac06fd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:59.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:59.648+0000 7f8ab2400640 1 -- 172.21.15.16:0/1267150814 >> 172.21.15.16:0/1267150814 conn(0x7f8aac067330 msgr2=0x7f8aac067740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:04:59.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:59.648+0000 7f8ab2400640 1 -- 172.21.15.16:0/1267150814 shutdown_connections 2024-03-31T15:04:59.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:59.648+0000 7f8ab2400640 1 -- 172.21.15.16:0/1267150814 wait complete. 2024-03-31T15:04:59.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:59.652+0000 7f8ab2400640 1 Processor -- start 2024-03-31T15:04:59.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:59.652+0000 7f8ab2400640 1 -- start start 2024-03-31T15:04:59.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:59.652+0000 7f8ab2400640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8aac06f920 0x7f8aac13f910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:59.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:59.652+0000 7f8ab2400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8aac0d2010 0x7f8aac13fe50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:59.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:59.652+0000 7f8ab2400640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f8aac145e40 0x7f8aac13c430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:59.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:59.652+0000 7f8ab2400640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f8aac0d0ee0 con 0x7f8aac0d2010 2024-03-31T15:04:59.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:59.652+0000 7f8ab2400640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f8aac0ce9e0 con 0x7f8aac06f920 2024-03-31T15:04:59.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:59.652+0000 7f8ab2400640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f8aac0d2df0 con 0x7f8aac145e40 2024-03-31T15:04:59.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:59.652+0000 7f8aaaa00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8aac0d2010 0x7f8aac13fe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:59.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:59.652+0000 7f8aaaa00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8aac0d2010 0x7f8aac13fe50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34922/0 (socket says 172.21.15.16:34922) 2024-03-31T15:04:59.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:59.652+0000 7f8aaaa00640 1 -- 172.21.15.16:0/1226122931 learned_addr learned my addr 172.21.15.16:0/1226122931 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:04:59.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:59.652+0000 7f8aab400640 1 --2- 172.21.15.16:0/1226122931 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8aac06f920 0x7f8aac13f910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:59.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:59.652+0000 7f8aabe00640 1 --2- 172.21.15.16:0/1226122931 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f8aac145e40 0x7f8aac13c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:59.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:59.652+0000 7f8aaaa00640 1 -- 172.21.15.16:0/1226122931 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f8aac145e40 msgr2=0x7f8aac13c430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:59.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:59.652+0000 7f8aaaa00640 1 --2- 172.21.15.16:0/1226122931 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f8aac145e40 0x7f8aac13c430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:59.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:59.652+0000 7f8aaaa00640 1 -- 172.21.15.16:0/1226122931 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8aac06f920 msgr2=0x7f8aac13f910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:59.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:59.652+0000 7f8aaaa00640 1 --2- 172.21.15.16:0/1226122931 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8aac06f920 0x7f8aac13f910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:59.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:59.652+0000 7f8aaaa00640 1 -- 172.21.15.16:0/1226122931 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8aac15bdd0 con 0x7f8aac0d2010 2024-03-31T15:04:59.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:59.652+0000 7f8aaaa00640 1 --2- 172.21.15.16:0/1226122931 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8aac0d2010 0x7f8aac13fe50 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f8a9800ec10 tx=0x7f8a9800c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:59.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:59.652+0000 7f8aa3400640 1 -- 172.21.15.16:0/1226122931 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a98017070 con 0x7f8aac0d2010 2024-03-31T15:04:59.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:59.652+0000 7f8aa3400640 1 -- 172.21.15.16:0/1226122931 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8a9800cc30 con 0x7f8aac0d2010 2024-03-31T15:04:59.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:59.652+0000 7f8aa3400640 1 -- 172.21.15.16:0/1226122931 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a98013470 con 0x7f8aac0d2010 2024-03-31T15:04:59.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:59.652+0000 7f8ab2400640 1 -- 172.21.15.16:0/1226122931 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8aac15c0b0 con 0x7f8aac0d2010 2024-03-31T15:04:59.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:59.652+0000 7f8ab2400640 1 -- 172.21.15.16:0/1226122931 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8aac15c670 con 0x7f8aac0d2010 2024-03-31T15:04:59.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:59.652+0000 7f8ab2400640 1 -- 172.21.15.16:0/1226122931 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8aac069950 con 0x7f8aac0d2010 2024-03-31T15:04:59.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:59.656+0000 7f8aa3400640 1 -- 172.21.15.16:0/1226122931 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f8a9801a070 con 0x7f8aac0d2010 2024-03-31T15:04:59.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:59.656+0000 7f8aa3400640 1 --2- 172.21.15.16:0/1226122931 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f8a8c068380 0x7f8a8c06a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:04:59.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:59.656+0000 7f8aa3400640 1 -- 172.21.15.16:0/1226122931 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5709+0+0 (secure 0 0 0) 0x7f8a98086d30 con 0x7f8aac0d2010 2024-03-31T15:04:59.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:59.656+0000 7f8aab400640 1 --2- 172.21.15.16:0/1226122931 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f8a8c068380 0x7f8a8c06a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:04:59.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:59.656+0000 7f8aab400640 1 --2- 172.21.15.16:0/1226122931 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f8a8c068380 0x7f8a8c06a840 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f8a9c000f70 tx=0x7f8a9c007df0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:04:59.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:59.660+0000 7f8aa3400640 1 -- 172.21.15.16:0/1226122931 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f8a98056c00 con 0x7f8aac0d2010 2024-03-31T15:04:59.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:59.864+0000 7f8ab2400640 1 -- 172.21.15.16:0/1226122931 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8aac0b7f60 con 0x7f8aac0d2010 2024-03-31T15:04:59.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:59.864+0000 7f8aa3400640 1 -- 172.21.15.16:0/1226122931 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) ==== 94+0+5178 (secure 0 0 0) 0x7f8a9805af00 con 0x7f8aac0d2010 2024-03-31T15:04:59.858 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:04:59.858 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":16,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4949,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6836/2073954753","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":2073954753},{"type":"v1","addr":"172.21.15.89:6836","nonce":2073954753}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":4952,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/2143918278","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6835","nonce":2143918278},{"type":"v1","addr":"172.21.15.89:6837","nonce":2143918278}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":5251,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6833/881391342","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":881391342},{"type":"v1","addr":"172.21.15.16:6833","nonce":881391342}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-31T15:04:39.780323+0000","modified":"2024-03-31T15:04:42.846439+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5254},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5254":{"gid":5254,"name":"c","rank":0,"incarnation":13,"state":"up:active","state_seq":3,"addr":"172.21.15.16:6835/1845707550","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":1845707550},{"type":"v1","addr":"172.21.15.16:6835","nonce":1845707550}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5254,"qdb_cluster":[5254]},"id":2}]} 2024-03-31T15:04:59.858 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 16 2024-03-31T15:04:59.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:59.864+0000 7f8ab2400640 1 -- 172.21.15.16:0/1226122931 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f8a8c068380 msgr2=0x7f8a8c06a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:59.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:59.864+0000 7f8ab2400640 1 --2- 172.21.15.16:0/1226122931 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f8a8c068380 0x7f8a8c06a840 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f8a9c000f70 tx=0x7f8a9c007df0 comp rx=0 tx=0).stop 2024-03-31T15:04:59.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:59.864+0000 7f8ab2400640 1 -- 172.21.15.16:0/1226122931 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8aac0d2010 msgr2=0x7f8aac13fe50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:04:59.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:59.864+0000 7f8ab2400640 1 --2- 172.21.15.16:0/1226122931 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8aac0d2010 0x7f8aac13fe50 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f8a9800ec10 tx=0x7f8a9800c520 comp rx=0 tx=0).stop 2024-03-31T15:04:59.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:59.864+0000 7f8ab2400640 1 -- 172.21.15.16:0/1226122931 shutdown_connections 2024-03-31T15:04:59.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:59.864+0000 7f8ab2400640 1 --2- 172.21.15.16:0/1226122931 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f8a8c068380 0x7f8a8c06a840 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:59.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:59.864+0000 7f8ab2400640 1 --2- 172.21.15.16:0/1226122931 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f8aac145e40 0x7f8aac13c430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:59.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:59.864+0000 7f8ab2400640 1 --2- 172.21.15.16:0/1226122931 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8aac0d2010 0x7f8aac13fe50 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:59.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:59.864+0000 7f8ab2400640 1 --2- 172.21.15.16:0/1226122931 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8aac06f920 0x7f8aac13f910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:04:59.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:59.864+0000 7f8ab2400640 1 -- 172.21.15.16:0/1226122931 >> 172.21.15.16:0/1226122931 conn(0x7f8aac067330 msgr2=0x7f8aac0cc370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:04:59.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:59.864+0000 7f8ab2400640 1 -- 172.21.15.16:0/1226122931 shutdown_connections 2024-03-31T15:04:59.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:04:59.864+0000 7f8ab2400640 1 -- 172.21.15.16:0/1226122931 wait complete. 2024-03-31T15:04:59.879 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json tag path /parentdir mytag 2024-03-31T15:05:00.048 INFO:teuthology.orchestra.run.smithi016.stdout:{"return_code":0,"scrub_tag":"mytag","mode":"asynchronous"} 2024-03-31T15:05:00.048 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mode": "asynchronous", "return_code": 0, "scrub_tag": "mytag" } 2024-03-31T15:05:06.748 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-31T15:05:06.752 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-31T15:05:10.049 DEBUG:teuthology.orchestra.run.smithi016:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata getxattr 10000000000.00000000 scrub_tag 2024-03-31T15:05:10.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.103+0000 7f4c3b648080 1 Processor -- start 2024-03-31T15:05:10.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.103+0000 7f4c3b648080 1 -- start start 2024-03-31T15:05:10.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.103+0000 7f4c3b648080 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55ffbe9c7260 0x55ffbea3be60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:10.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.103+0000 7f4c3b648080 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ffbe9cc9f0 0x55ffbea5bb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:10.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.103+0000 7f4c3b648080 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ffbea58ab0 0x55ffbea5c080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:10.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.103+0000 7f4c3b648080 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55ffbe9274b0 con 0x55ffbea58ab0 2024-03-31T15:05:10.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.103+0000 7f4c3b648080 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55ffbe92b500 con 0x55ffbe9cc9f0 2024-03-31T15:05:10.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.103+0000 7f4c3b648080 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x55ffbe9b1eb0 con 0x55ffbe9c7260 2024-03-31T15:05:10.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.103+0000 7f4c38600640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ffbea58ab0 0x55ffbea5c080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:10.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.103+0000 7f4c38600640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ffbea58ab0 0x55ffbea5c080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34688/0 (socket says 172.21.15.16:34688) 2024-03-31T15:05:10.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.103+0000 7f4c38600640 1 -- 172.21.15.16:0/372506013 learned_addr learned my addr 172.21.15.16:0/372506013 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:05:10.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.103+0000 7f4c37200640 1 --2- 172.21.15.16:0/372506013 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ffbe9cc9f0 0x55ffbea5bb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:10.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.103+0000 7f4c37c00640 1 --2- 172.21.15.16:0/372506013 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55ffbe9c7260 0x55ffbea3be60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:10.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.103+0000 7f4c38600640 1 -- 172.21.15.16:0/372506013 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55ffbe9c7260 msgr2=0x55ffbea3be60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:05:10.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.103+0000 7f4c38600640 1 --2- 172.21.15.16:0/372506013 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55ffbe9c7260 0x55ffbea3be60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:10.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.103+0000 7f4c38600640 1 -- 172.21.15.16:0/372506013 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ffbe9cc9f0 msgr2=0x55ffbea5bb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:05:10.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.103+0000 7f4c38600640 1 --2- 172.21.15.16:0/372506013 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ffbe9cc9f0 0x55ffbea5bb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:10.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.103+0000 7f4c38600640 1 -- 172.21.15.16:0/372506013 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ffbea3c3a0 con 0x55ffbea58ab0 2024-03-31T15:05:10.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.103+0000 7f4c38600640 1 --2- 172.21.15.16:0/372506013 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ffbea58ab0 0x55ffbea5c080 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f4c300016e0 tx=0x7f4c30006f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=abb3af00948094f5 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:05:10.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.103+0000 7f4c35e00640 1 -- 172.21.15.16:0/372506013 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c30016070 con 0x55ffbea58ab0 2024-03-31T15:05:10.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.103+0000 7f4c35e00640 1 -- 172.21.15.16:0/372506013 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4c3000f710 con 0x55ffbea58ab0 2024-03-31T15:05:10.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.103+0000 7f4c35e00640 1 -- 172.21.15.16:0/372506013 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c3000fae0 con 0x55ffbea58ab0 2024-03-31T15:05:10.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.103+0000 7f4c3b648080 1 -- 172.21.15.16:0/372506013 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ffbea58ab0 msgr2=0x55ffbea5c080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:10.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.103+0000 7f4c3b648080 1 --2- 172.21.15.16:0/372506013 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ffbea58ab0 0x55ffbea5c080 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f4c300016e0 tx=0x7f4c30006f90 comp rx=0 tx=0).stop 2024-03-31T15:05:10.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.103+0000 7f4c3b648080 1 -- 172.21.15.16:0/372506013 shutdown_connections 2024-03-31T15:05:10.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.103+0000 7f4c3b648080 1 --2- 172.21.15.16:0/372506013 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ffbea58ab0 0x55ffbea5c080 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:10.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.103+0000 7f4c3b648080 1 --2- 172.21.15.16:0/372506013 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ffbe9cc9f0 0x55ffbea5bb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:10.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.103+0000 7f4c3b648080 1 --2- 172.21.15.16:0/372506013 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55ffbe9c7260 0x55ffbea3be60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:10.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.103+0000 7f4c3b648080 1 -- 172.21.15.16:0/372506013 >> 172.21.15.16:0/372506013 conn(0x55ffbe9ca0b0 msgr2=0x55ffbe9cbb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:05:10.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.103+0000 7f4c3b648080 1 -- 172.21.15.16:0/372506013 shutdown_connections 2024-03-31T15:05:10.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.103+0000 7f4c3b648080 1 -- 172.21.15.16:0/372506013 wait complete. 2024-03-31T15:05:10.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.107+0000 7f4c3b648080 1 Processor -- start 2024-03-31T15:05:10.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.107+0000 7f4c3b648080 1 -- start start 2024-03-31T15:05:10.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.107+0000 7f4c3b648080 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55ffbe9c7260 0x55ffbea596c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:10.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.107+0000 7f4c3b648080 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ffbe9cc9f0 0x55ffbea59c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:10.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.107+0000 7f4c3b648080 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ffbea58ab0 0x55ffbeaf1d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:10.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.107+0000 7f4c3b648080 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55ffbea580b0 con 0x55ffbea58ab0 2024-03-31T15:05:10.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.107+0000 7f4c3b648080 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55ffbea57e10 con 0x55ffbe9cc9f0 2024-03-31T15:05:10.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.107+0000 7f4c3b648080 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x55ffbea549f0 con 0x55ffbe9c7260 2024-03-31T15:05:10.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.107+0000 7f4c38600640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ffbea58ab0 0x55ffbeaf1d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:10.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.107+0000 7f4c38600640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ffbea58ab0 0x55ffbeaf1d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34698/0 (socket says 172.21.15.16:34698) 2024-03-31T15:05:10.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.107+0000 7f4c38600640 1 -- 172.21.15.16:0/2468710798 learned_addr learned my addr 172.21.15.16:0/2468710798 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:05:10.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.107+0000 7f4c37200640 1 --2- 172.21.15.16:0/2468710798 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ffbe9cc9f0 0x55ffbea59c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:10.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.107+0000 7f4c37c00640 1 --2- 172.21.15.16:0/2468710798 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55ffbe9c7260 0x55ffbea596c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:10.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.107+0000 7f4c38600640 1 -- 172.21.15.16:0/2468710798 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55ffbe9c7260 msgr2=0x55ffbea596c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:10.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.107+0000 7f4c38600640 1 --2- 172.21.15.16:0/2468710798 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55ffbe9c7260 0x55ffbea596c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:10.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.107+0000 7f4c38600640 1 -- 172.21.15.16:0/2468710798 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ffbe9cc9f0 msgr2=0x55ffbea59c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:10.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.107+0000 7f4c38600640 1 --2- 172.21.15.16:0/2468710798 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ffbe9cc9f0 0x55ffbea59c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:10.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.107+0000 7f4c38600640 1 -- 172.21.15.16:0/2468710798 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ffbeaf48d0 con 0x55ffbea58ab0 2024-03-31T15:05:10.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.107+0000 7f4c38600640 1 --2- 172.21.15.16:0/2468710798 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ffbea58ab0 0x55ffbeaf1d40 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f4c30014040 tx=0x7f4c30001260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:05:10.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.107+0000 7f4c34a00640 1 -- 172.21.15.16:0/2468710798 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c30021020 con 0x55ffbea58ab0 2024-03-31T15:05:10.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.107+0000 7f4c34a00640 1 -- 172.21.15.16:0/2468710798 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4c3000fec0 con 0x55ffbea58ab0 2024-03-31T15:05:10.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.107+0000 7f4c34a00640 1 -- 172.21.15.16:0/2468710798 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c30016030 con 0x55ffbea58ab0 2024-03-31T15:05:10.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.107+0000 7f4c3b648080 1 -- 172.21.15.16:0/2468710798 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55ffbeaf4b70 con 0x55ffbea58ab0 2024-03-31T15:05:10.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.107+0000 7f4c3b648080 1 -- 172.21.15.16:0/2468710798 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ffbe9812c0 con 0x55ffbea58ab0 2024-03-31T15:05:10.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.111+0000 7f4c34a00640 1 -- 172.21.15.16:0/2468710798 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f4c30024020 con 0x55ffbea58ab0 2024-03-31T15:05:10.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.111+0000 7f4c34a00640 1 --2- 172.21.15.16:0/2468710798 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f4c18066d90 0x7f4c18069250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:10.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.111+0000 7f4c34a00640 1 -- 172.21.15.16:0/2468710798 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5709+0+0 (secure 0 0 0) 0x7f4c30086b20 con 0x55ffbea58ab0 2024-03-31T15:05:10.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.111+0000 7f4c3b648080 1 --2- 172.21.15.16:0/2468710798 >> [v2:172.21.15.16:6809/3360352974,v1:172.21.15.16:6812/3360352974] conn(0x55ffbeaf4400 0x55ffbe981760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:10.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.111+0000 7f4c3b648080 1 -- 172.21.15.16:0/2468710798 --> [v2:172.21.15.16:6809/3360352974,v1:172.21.15.16:6812/3360352974] -- osd_op(unknown.0.0:1 4.10 4:0cf6ad0f:::10000000000.00000000:head [getxattr scrub_tag in=9b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e34) -- 0x55ffbe981ca0 con 0x55ffbeaf4400 2024-03-31T15:05:10.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.111+0000 7f4c37c00640 1 --2- 172.21.15.16:0/2468710798 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f4c18066d90 0x7f4c18069250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:10.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.111+0000 7f4c37200640 1 --2- 172.21.15.16:0/2468710798 >> [v2:172.21.15.16:6809/3360352974,v1:172.21.15.16:6812/3360352974] conn(0x55ffbeaf4400 0x55ffbe981760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:10.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.111+0000 7f4c37c00640 1 --2- 172.21.15.16:0/2468710798 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f4c18066d90 0x7f4c18069250 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f4c280ad000 tx=0x7f4c280a6000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:05:10.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.111+0000 7f4c37200640 1 --2- 172.21.15.16:0/2468710798 >> [v2:172.21.15.16:6809/3360352974,v1:172.21.15.16:6812/3360352974] conn(0x55ffbeaf4400 0x55ffbe981760 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:05:10.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.111+0000 7f4c37200640 1 -- 172.21.15.16:0/2468710798 <== osd.3 v2:172.21.15.16:6809/3360352974 1 ==== osd_op_reply(1 10000000000.00000000 [getxattr (9) out=9b] v0'0 uv11 ondisk = 0) ==== 164+0+9 (crc 0 0 0) 0x7f4c2400f800 con 0x55ffbeaf4400 2024-03-31T15:05:10.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.111+0000 7f4c3b648080 1 -- 172.21.15.16:0/2468710798 >> [v2:172.21.15.16:6809/3360352974,v1:172.21.15.16:6812/3360352974] conn(0x55ffbeaf4400 msgr2=0x55ffbe981760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:10.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.111+0000 7f4c3b648080 1 --2- 172.21.15.16:0/2468710798 >> [v2:172.21.15.16:6809/3360352974,v1:172.21.15.16:6812/3360352974] conn(0x55ffbeaf4400 0x55ffbe981760 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:10.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.111+0000 7f4c3b648080 1 -- 172.21.15.16:0/2468710798 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f4c18066d90 msgr2=0x7f4c18069250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:10.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.111+0000 7f4c3b648080 1 --2- 172.21.15.16:0/2468710798 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f4c18066d90 0x7f4c18069250 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f4c280ad000 tx=0x7f4c280a6000 comp rx=0 tx=0).stop 2024-03-31T15:05:10.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.111+0000 7f4c3b648080 1 -- 172.21.15.16:0/2468710798 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ffbea58ab0 msgr2=0x55ffbeaf1d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:10.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.111+0000 7f4c3b648080 1 --2- 172.21.15.16:0/2468710798 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ffbea58ab0 0x55ffbeaf1d40 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f4c30014040 tx=0x7f4c30001260 comp rx=0 tx=0).stop 2024-03-31T15:05:10.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.111+0000 7f4c38600640 1 -- 172.21.15.16:0/2468710798 reap_dead start 2024-03-31T15:05:10.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.111+0000 7f4c3b648080 1 -- 172.21.15.16:0/2468710798 shutdown_connections 2024-03-31T15:05:10.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.111+0000 7f4c3b648080 1 -- 172.21.15.16:0/2468710798 >> 172.21.15.16:0/2468710798 conn(0x55ffbe9ca0b0 msgr2=0x55ffbea5bb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:05:10.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.111+0000 7f4c3b648080 1 -- 172.21.15.16:0/2468710798 shutdown_connections 2024-03-31T15:05:10.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.111+0000 7f4c3b648080 1 -- 172.21.15.16:0/2468710798 wait complete. 2024-03-31T15:05:10.112 INFO:teuthology.orchestra.run.smithi016.stdout:mytag 2024-03-31T15:05:10.112 DEBUG:teuthology.orchestra.run.smithi016:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata getxattr 10000000001.00000000 scrub_tag 2024-03-31T15:05:10.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.211+0000 7f36665fc080 1 Processor -- start 2024-03-31T15:05:10.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.211+0000 7f36665fc080 1 -- start start 2024-03-31T15:05:10.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.211+0000 7f36665fc080 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x564aa1109260 0x564aa10f3190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:10.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.211+0000 7f36665fc080 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564aa110e2e0 0x564aa10c31b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:10.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.211+0000 7f36665fc080 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564aa119aab0 0x564aa10f0530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:10.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.211+0000 7f36665fc080 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x564aa10f52b0 con 0x564aa119aab0 2024-03-31T15:05:10.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.211+0000 7f36665fc080 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x564aa10694b0 con 0x564aa110e2e0 2024-03-31T15:05:10.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.211+0000 7f36665fc080 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x564aa10f5430 con 0x564aa1109260 2024-03-31T15:05:10.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.211+0000 7f3664200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564aa119aab0 0x564aa10f0530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:10.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.211+0000 7f3663800640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x564aa1109260 0x564aa10f3190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:10.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.211+0000 7f3664200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564aa119aab0 0x564aa10f0530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34720/0 (socket says 172.21.15.16:34720) 2024-03-31T15:05:10.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.211+0000 7f3664200640 1 -- 172.21.15.16:0/4168124241 learned_addr learned my addr 172.21.15.16:0/4168124241 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:05:10.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.211+0000 7f3662e00640 1 --2- 172.21.15.16:0/4168124241 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564aa110e2e0 0x564aa10c31b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:10.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.211+0000 7f3664200640 1 -- 172.21.15.16:0/4168124241 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x564aa1109260 msgr2=0x564aa10f3190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:05:10.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.211+0000 7f3664200640 1 --2- 172.21.15.16:0/4168124241 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x564aa1109260 0x564aa10f3190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:10.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.211+0000 7f3664200640 1 -- 172.21.15.16:0/4168124241 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564aa110e2e0 msgr2=0x564aa10c31b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:05:10.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.211+0000 7f3664200640 1 --2- 172.21.15.16:0/4168124241 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564aa110e2e0 0x564aa10c31b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:10.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.211+0000 7f3664200640 1 -- 172.21.15.16:0/4168124241 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564aa119d9a0 con 0x564aa119aab0 2024-03-31T15:05:10.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.211+0000 7f3664200640 1 --2- 172.21.15.16:0/4168124241 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564aa119aab0 0x564aa10f0530 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f365c0ad040 tx=0x7f365c003340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=49187541a513bb02 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:05:10.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.211+0000 7f3661a00640 1 -- 172.21.15.16:0/4168124241 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f365c0af040 con 0x564aa119aab0 2024-03-31T15:05:10.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.211+0000 7f3661a00640 1 -- 172.21.15.16:0/4168124241 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f365c0b8070 con 0x564aa119aab0 2024-03-31T15:05:10.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.211+0000 7f3661a00640 1 -- 172.21.15.16:0/4168124241 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f365c0abd40 con 0x564aa119aab0 2024-03-31T15:05:10.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.215+0000 7f36665fc080 1 -- 172.21.15.16:0/4168124241 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564aa119aab0 msgr2=0x564aa10f0530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:10.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.215+0000 7f36665fc080 1 --2- 172.21.15.16:0/4168124241 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564aa119aab0 0x564aa10f0530 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f365c0ad040 tx=0x7f365c003340 comp rx=0 tx=0).stop 2024-03-31T15:05:10.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.215+0000 7f36665fc080 1 -- 172.21.15.16:0/4168124241 shutdown_connections 2024-03-31T15:05:10.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.215+0000 7f36665fc080 1 --2- 172.21.15.16:0/4168124241 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564aa119aab0 0x564aa10f0530 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:10.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.215+0000 7f36665fc080 1 --2- 172.21.15.16:0/4168124241 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564aa110e2e0 0x564aa10c31b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:10.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.215+0000 7f36665fc080 1 --2- 172.21.15.16:0/4168124241 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x564aa1109260 0x564aa10f3190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:10.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.215+0000 7f36665fc080 1 -- 172.21.15.16:0/4168124241 >> 172.21.15.16:0/4168124241 conn(0x564aa110c0b0 msgr2=0x564aa110c490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:05:10.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.215+0000 7f36665fc080 1 -- 172.21.15.16:0/4168124241 shutdown_connections 2024-03-31T15:05:10.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.215+0000 7f36665fc080 1 -- 172.21.15.16:0/4168124241 wait complete. 2024-03-31T15:05:10.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.215+0000 7f36665fc080 1 Processor -- start 2024-03-31T15:05:10.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.215+0000 7f36665fc080 1 -- start start 2024-03-31T15:05:10.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.215+0000 7f36665fc080 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564aa1109260 0x564aa110b2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:10.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.215+0000 7f36665fc080 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564aa110e2e0 0x564aa118cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:10.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.215+0000 7f36665fc080 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x564aa119aab0 0x564aa118d520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:10.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.215+0000 7f36665fc080 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x564aa119dc40 con 0x564aa110e2e0 2024-03-31T15:05:10.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.215+0000 7f36665fc080 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x564aa11990d0 con 0x564aa1109260 2024-03-31T15:05:10.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.215+0000 7f36665fc080 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x564aa1196bc0 con 0x564aa119aab0 2024-03-31T15:05:10.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.215+0000 7f3662e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564aa110e2e0 0x564aa118cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:10.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.215+0000 7f3662e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564aa110e2e0 0x564aa118cfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34724/0 (socket says 172.21.15.16:34724) 2024-03-31T15:05:10.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.215+0000 7f3663800640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564aa1109260 0x564aa110b2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:10.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.215+0000 7f3662e00640 1 -- 172.21.15.16:0/4252121071 learned_addr learned my addr 172.21.15.16:0/4252121071 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:05:10.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.215+0000 7f3664200640 1 --2- 172.21.15.16:0/4252121071 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x564aa119aab0 0x564aa118d520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:10.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.215+0000 7f3662e00640 1 -- 172.21.15.16:0/4252121071 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x564aa119aab0 msgr2=0x564aa118d520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:10.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.215+0000 7f3662e00640 1 --2- 172.21.15.16:0/4252121071 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x564aa119aab0 0x564aa118d520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:10.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.215+0000 7f3662e00640 1 -- 172.21.15.16:0/4252121071 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564aa1109260 msgr2=0x564aa110b2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:10.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.215+0000 7f3662e00640 1 --2- 172.21.15.16:0/4252121071 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564aa1109260 0x564aa110b2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:10.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.215+0000 7f3662e00640 1 -- 172.21.15.16:0/4252121071 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564aa1192340 con 0x564aa110e2e0 2024-03-31T15:05:10.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.215+0000 7f3662e00640 1 --2- 172.21.15.16:0/4252121071 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564aa110e2e0 0x564aa118cfe0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f36500087d0 tx=0x7f3650008c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:05:10.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.215+0000 7f365be00640 1 -- 172.21.15.16:0/4252121071 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3650019070 con 0x564aa110e2e0 2024-03-31T15:05:10.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.215+0000 7f365be00640 1 -- 172.21.15.16:0/4252121071 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3650010b40 con 0x564aa110e2e0 2024-03-31T15:05:10.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.215+0000 7f365be00640 1 -- 172.21.15.16:0/4252121071 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3650015450 con 0x564aa110e2e0 2024-03-31T15:05:10.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.215+0000 7f36665fc080 1 -- 172.21.15.16:0/4252121071 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x564aa11925e0 con 0x564aa110e2e0 2024-03-31T15:05:10.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.215+0000 7f36665fc080 1 -- 172.21.15.16:0/4252121071 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564aa1192b20 con 0x564aa110e2e0 2024-03-31T15:05:10.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.219+0000 7f365be00640 1 -- 172.21.15.16:0/4252121071 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f365001c070 con 0x564aa110e2e0 2024-03-31T15:05:10.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.219+0000 7f365be00640 1 --2- 172.21.15.16:0/4252121071 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f3644066d90 0x7f3644069250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:10.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.219+0000 7f365be00640 1 -- 172.21.15.16:0/4252121071 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5709+0+0 (secure 0 0 0) 0x7f3650086140 con 0x564aa110e2e0 2024-03-31T15:05:10.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.219+0000 7f3663800640 1 --2- 172.21.15.16:0/4252121071 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f3644066d90 0x7f3644069250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:10.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.219+0000 7f36665fc080 1 --2- 172.21.15.16:0/4252121071 >> [v2:172.21.15.16:6808/1398737533,v1:172.21.15.16:6810/1398737533] conn(0x564aa110b7e0 0x564aa1192fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:10.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.219+0000 7f36665fc080 1 -- 172.21.15.16:0/4252121071 --> [v2:172.21.15.16:6808/1398737533,v1:172.21.15.16:6810/1398737533] -- osd_op(unknown.0.0:1 4.7 4:e2e07930:::10000000001.00000000:head [getxattr scrub_tag in=9b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e34) -- 0x564aa1193500 con 0x564aa110b7e0 2024-03-31T15:05:10.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.219+0000 7f3664200640 1 --2- 172.21.15.16:0/4252121071 >> [v2:172.21.15.16:6808/1398737533,v1:172.21.15.16:6810/1398737533] conn(0x564aa110b7e0 0x564aa1192fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:10.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.219+0000 7f3663800640 1 --2- 172.21.15.16:0/4252121071 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f3644066d90 0x7f3644069250 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f3654000c10 tx=0x7f3654001730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:05:10.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.219+0000 7f3664200640 1 --2- 172.21.15.16:0/4252121071 >> [v2:172.21.15.16:6808/1398737533,v1:172.21.15.16:6810/1398737533] conn(0x564aa110b7e0 0x564aa1192fc0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:05:10.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.219+0000 7f3664200640 1 -- 172.21.15.16:0/4252121071 <== osd.1 v2:172.21.15.16:6808/1398737533 1 ==== osd_op_reply(1 10000000001.00000000 [getxattr (9) out=9b] v0'0 uv5 ondisk = 0) ==== 164+0+9 (crc 0 0 0) 0x7f365c0a8070 con 0x564aa110b7e0 2024-03-31T15:05:10.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.219+0000 7f36665fc080 1 -- 172.21.15.16:0/4252121071 >> [v2:172.21.15.16:6808/1398737533,v1:172.21.15.16:6810/1398737533] conn(0x564aa110b7e0 msgr2=0x564aa1192fc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:10.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.219+0000 7f36665fc080 1 --2- 172.21.15.16:0/4252121071 >> [v2:172.21.15.16:6808/1398737533,v1:172.21.15.16:6810/1398737533] conn(0x564aa110b7e0 0x564aa1192fc0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:10.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.219+0000 7f36665fc080 1 -- 172.21.15.16:0/4252121071 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f3644066d90 msgr2=0x7f3644069250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:10.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.219+0000 7f36665fc080 1 --2- 172.21.15.16:0/4252121071 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f3644066d90 0x7f3644069250 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f3654000c10 tx=0x7f3654001730 comp rx=0 tx=0).stop 2024-03-31T15:05:10.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.219+0000 7f36665fc080 1 -- 172.21.15.16:0/4252121071 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564aa110e2e0 msgr2=0x564aa118cfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:10.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.219+0000 7f36665fc080 1 --2- 172.21.15.16:0/4252121071 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564aa110e2e0 0x564aa118cfe0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f36500087d0 tx=0x7f3650008c90 comp rx=0 tx=0).stop 2024-03-31T15:05:10.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.219+0000 7f3664200640 1 -- 172.21.15.16:0/4252121071 reap_dead start 2024-03-31T15:05:10.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.219+0000 7f36665fc080 1 -- 172.21.15.16:0/4252121071 shutdown_connections 2024-03-31T15:05:10.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.219+0000 7f36665fc080 1 -- 172.21.15.16:0/4252121071 >> 172.21.15.16:0/4252121071 conn(0x564aa110c0b0 msgr2=0x564aa110c490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:05:10.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.223+0000 7f36665fc080 1 -- 172.21.15.16:0/4252121071 shutdown_connections 2024-03-31T15:05:10.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.223+0000 7f36665fc080 1 -- 172.21.15.16:0/4252121071 wait complete. 2024-03-31T15:05:10.221 INFO:teuthology.orchestra.run.smithi016.stdout:mytag 2024-03-31T15:05:10.221 DEBUG:teuthology.orchestra.run.smithi016:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_data getxattr 100000001f7.00000000 scrub_tag 2024-03-31T15:05:10.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.315+0000 7f4889455080 1 Processor -- start 2024-03-31T15:05:10.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.315+0000 7f4889455080 1 -- start start 2024-03-31T15:05:10.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.315+0000 7f4889455080 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x562194399350 0x56219437d720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:10.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.315+0000 7f4889455080 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562194399730 0x56219437ea80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:10.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.315+0000 7f4889455080 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5621944099a0 0x562194405d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:10.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.315+0000 7f4889455080 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x56219440ad10 con 0x5621944099a0 2024-03-31T15:05:10.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.315+0000 7f4889455080 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x5621942f54b0 con 0x562194399730 2024-03-31T15:05:10.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.315+0000 7f4889455080 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x56219440ae90 con 0x562194399350 2024-03-31T15:05:10.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.315+0000 7f4887000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5621944099a0 0x562194405d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:10.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.315+0000 7f4887000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5621944099a0 0x562194405d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34736/0 (socket says 172.21.15.16:34736) 2024-03-31T15:05:10.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.315+0000 7f4887000640 1 -- 172.21.15.16:0/3760942816 learned_addr learned my addr 172.21.15.16:0/3760942816 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:05:10.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.315+0000 7f4886600640 1 --2- 172.21.15.16:0/3760942816 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x562194399350 0x56219437d720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:10.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.315+0000 7f4885c00640 1 --2- 172.21.15.16:0/3760942816 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562194399730 0x56219437ea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:10.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.315+0000 7f4887000640 1 -- 172.21.15.16:0/3760942816 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x562194399350 msgr2=0x56219437d720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:05:10.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.315+0000 7f4887000640 1 --2- 172.21.15.16:0/3760942816 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x562194399350 0x56219437d720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:10.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.315+0000 7f4887000640 1 -- 172.21.15.16:0/3760942816 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562194399730 msgr2=0x56219437ea80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:05:10.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.315+0000 7f4887000640 1 --2- 172.21.15.16:0/3760942816 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562194399730 0x56219437ea80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:10.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.315+0000 7f4887000640 1 -- 172.21.15.16:0/3760942816 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56219437f4c0 con 0x5621944099a0 2024-03-31T15:05:10.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.315+0000 7f4887000640 1 --2- 172.21.15.16:0/3760942816 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5621944099a0 0x562194405d20 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f488000a950 tx=0x7f4880005d90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=63fceac0d33e0bf3 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:05:10.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.315+0000 7f487fe00640 1 -- 172.21.15.16:0/3760942816 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f488000d960 con 0x5621944099a0 2024-03-31T15:05:10.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.315+0000 7f487fe00640 1 -- 172.21.15.16:0/3760942816 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f488000db00 con 0x5621944099a0 2024-03-31T15:05:10.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.315+0000 7f487fe00640 1 -- 172.21.15.16:0/3760942816 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4880012450 con 0x5621944099a0 2024-03-31T15:05:10.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.315+0000 7f4889455080 1 -- 172.21.15.16:0/3760942816 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5621944099a0 msgr2=0x562194405d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:10.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.315+0000 7f4889455080 1 --2- 172.21.15.16:0/3760942816 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5621944099a0 0x562194405d20 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f488000a950 tx=0x7f4880005d90 comp rx=0 tx=0).stop 2024-03-31T15:05:10.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.319+0000 7f4889455080 1 -- 172.21.15.16:0/3760942816 shutdown_connections 2024-03-31T15:05:10.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.319+0000 7f4889455080 1 --2- 172.21.15.16:0/3760942816 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5621944099a0 0x562194405d20 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:10.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.319+0000 7f4889455080 1 --2- 172.21.15.16:0/3760942816 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562194399730 0x56219437ea80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:10.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.319+0000 7f4889455080 1 --2- 172.21.15.16:0/3760942816 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x562194399350 0x56219437d720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:10.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.319+0000 7f4889455080 1 -- 172.21.15.16:0/3760942816 >> 172.21.15.16:0/3760942816 conn(0x56219437b050 msgr2=0x562194408f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:05:10.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.319+0000 7f4889455080 1 -- 172.21.15.16:0/3760942816 shutdown_connections 2024-03-31T15:05:10.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.319+0000 7f4889455080 1 -- 172.21.15.16:0/3760942816 wait complete. 2024-03-31T15:05:10.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.319+0000 7f4889455080 1 Processor -- start 2024-03-31T15:05:10.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.319+0000 7f4889455080 1 -- start start 2024-03-31T15:05:10.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.319+0000 7f4889455080 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x562194399350 0x562194405b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:10.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.319+0000 7f4889455080 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562194399730 0x562194421fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:10.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.319+0000 7f4889455080 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5621944099a0 0x562194422500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:10.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.319+0000 7f4889455080 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x56219437fa00 con 0x562194399730 2024-03-31T15:05:10.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.319+0000 7f4889455080 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x56219437f760 con 0x5621944099a0 2024-03-31T15:05:10.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.319+0000 7f4889455080 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x56219437fca0 con 0x562194399350 2024-03-31T15:05:10.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.319+0000 7f4885c00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562194399730 0x562194421fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:10.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.319+0000 7f4885c00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562194399730 0x562194421fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34738/0 (socket says 172.21.15.16:34738) 2024-03-31T15:05:10.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.319+0000 7f4885c00640 1 -- 172.21.15.16:0/1995263626 learned_addr learned my addr 172.21.15.16:0/1995263626 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:05:10.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.319+0000 7f4886600640 1 --2- 172.21.15.16:0/1995263626 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x562194399350 0x562194405b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:10.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.319+0000 7f4887000640 1 --2- 172.21.15.16:0/1995263626 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5621944099a0 0x562194422500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:10.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.319+0000 7f4885c00640 1 -- 172.21.15.16:0/1995263626 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x562194399350 msgr2=0x562194405b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:10.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.319+0000 7f4885c00640 1 --2- 172.21.15.16:0/1995263626 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x562194399350 0x562194405b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:10.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.319+0000 7f4885c00640 1 -- 172.21.15.16:0/1995263626 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5621944099a0 msgr2=0x562194422500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:10.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.319+0000 7f4885c00640 1 --2- 172.21.15.16:0/1995263626 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5621944099a0 0x562194422500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:10.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.319+0000 7f4885c00640 1 -- 172.21.15.16:0/1995263626 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562194397f20 con 0x562194399730 2024-03-31T15:05:10.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.319+0000 7f4887000640 1 --2- 172.21.15.16:0/1995263626 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5621944099a0 0x562194422500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:05:10.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.319+0000 7f4885c00640 1 --2- 172.21.15.16:0/1995263626 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562194399730 0x562194421fc0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f487400a5c0 tx=0x7f4874019c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:05:10.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.319+0000 7f487ea00640 1 -- 172.21.15.16:0/1995263626 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48740c1070 con 0x562194399730 2024-03-31T15:05:10.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.319+0000 7f487ea00640 1 -- 172.21.15.16:0/1995263626 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f48740bc480 con 0x562194399730 2024-03-31T15:05:10.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.319+0000 7f487ea00640 1 -- 172.21.15.16:0/1995263626 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48740bc870 con 0x562194399730 2024-03-31T15:05:10.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.319+0000 7f4889455080 1 -- 172.21.15.16:0/1995263626 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x5621943981c0 con 0x562194399730 2024-03-31T15:05:10.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.319+0000 7f4889455080 1 -- 172.21.15.16:0/1995263626 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562194427910 con 0x562194399730 2024-03-31T15:05:10.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.323+0000 7f487ea00640 1 -- 172.21.15.16:0/1995263626 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f48740c4070 con 0x562194399730 2024-03-31T15:05:10.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.323+0000 7f487ea00640 1 --2- 172.21.15.16:0/1995263626 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f4868066d90 0x7f4868069250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:10.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.323+0000 7f487ea00640 1 -- 172.21.15.16:0/1995263626 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5709+0+0 (secure 0 0 0) 0x7f4874120680 con 0x562194399730 2024-03-31T15:05:10.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.323+0000 7f4889455080 1 --2- 172.21.15.16:0/1995263626 >> [v2:172.21.15.16:6808/1398737533,v1:172.21.15.16:6810/1398737533] conn(0x5621944060c0 0x5621943960b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:10.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.323+0000 7f4889455080 1 -- 172.21.15.16:0/1995263626 --> [v2:172.21.15.16:6808/1398737533,v1:172.21.15.16:6810/1398737533] -- osd_op(unknown.0.0:1 5.36 5:6fe38dc2:::100000001f7.00000000:head [getxattr scrub_tag in=9b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e34) -- 0x562194424330 con 0x5621944060c0 2024-03-31T15:05:10.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.323+0000 7f4886600640 1 --2- 172.21.15.16:0/1995263626 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f4868066d90 0x7f4868069250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:10.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.323+0000 7f4887000640 1 --2- 172.21.15.16:0/1995263626 >> [v2:172.21.15.16:6808/1398737533,v1:172.21.15.16:6810/1398737533] conn(0x5621944060c0 0x5621943960b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:10.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.323+0000 7f4886600640 1 --2- 172.21.15.16:0/1995263626 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f4868066d90 0x7f4868069250 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f4878011570 tx=0x7f48780017a0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:05:10.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.323+0000 7f4887000640 1 --2- 172.21.15.16:0/1995263626 >> [v2:172.21.15.16:6808/1398737533,v1:172.21.15.16:6810/1398737533] conn(0x5621944060c0 0x5621943960b0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:05:10.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.323+0000 7f4887000640 1 -- 172.21.15.16:0/1995263626 <== osd.1 v2:172.21.15.16:6808/1398737533 1 ==== osd_op_reply(1 100000001f7.00000000 [getxattr (9) out=9b] v0'0 uv17 ondisk = 0) ==== 164+0+9 (crc 0 0 0) 0x7f4880019020 con 0x5621944060c0 2024-03-31T15:05:10.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.323+0000 7f4889455080 1 -- 172.21.15.16:0/1995263626 >> [v2:172.21.15.16:6808/1398737533,v1:172.21.15.16:6810/1398737533] conn(0x5621944060c0 msgr2=0x5621943960b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:10.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.323+0000 7f4889455080 1 --2- 172.21.15.16:0/1995263626 >> [v2:172.21.15.16:6808/1398737533,v1:172.21.15.16:6810/1398737533] conn(0x5621944060c0 0x5621943960b0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:10.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.323+0000 7f4889455080 1 -- 172.21.15.16:0/1995263626 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f4868066d90 msgr2=0x7f4868069250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:10.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.323+0000 7f4889455080 1 --2- 172.21.15.16:0/1995263626 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f4868066d90 0x7f4868069250 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f4878011570 tx=0x7f48780017a0 comp rx=0 tx=0).stop 2024-03-31T15:05:10.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.323+0000 7f4889455080 1 -- 172.21.15.16:0/1995263626 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562194399730 msgr2=0x562194421fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:10.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.323+0000 7f4889455080 1 --2- 172.21.15.16:0/1995263626 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562194399730 0x562194421fc0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f487400a5c0 tx=0x7f4874019c20 comp rx=0 tx=0).stop 2024-03-31T15:05:10.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.323+0000 7f4887000640 1 -- 172.21.15.16:0/1995263626 reap_dead start 2024-03-31T15:05:10.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.323+0000 7f4889455080 1 -- 172.21.15.16:0/1995263626 shutdown_connections 2024-03-31T15:05:10.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.323+0000 7f4889455080 1 -- 172.21.15.16:0/1995263626 >> 172.21.15.16:0/1995263626 conn(0x56219437b050 msgr2=0x56219437ea80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:05:10.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.323+0000 7f4889455080 1 -- 172.21.15.16:0/1995263626 shutdown_connections 2024-03-31T15:05:10.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.323+0000 7f4889455080 1 -- 172.21.15.16:0/1995263626 wait complete. 2024-03-31T15:05:10.327 INFO:teuthology.orchestra.run.smithi016.stdout:mytag 2024-03-31T15:05:10.327 DEBUG:teuthology.orchestra.run.smithi016:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_data getxattr 100000001f8.00000000 scrub_tag 2024-03-31T15:05:10.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.419+0000 7f6aca8bf080 1 Processor -- start 2024-03-31T15:05:10.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.419+0000 7f6aca8bf080 1 -- start start 2024-03-31T15:05:10.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.419+0000 7f6aca8bf080 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e11b486e70 0x55e11b52b5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:10.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.419+0000 7f6aca8bf080 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e11b4a4c60 0x55e11b488810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:10.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.419+0000 7f6aca8bf080 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55e11b4a5040 0x55e11b5263d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:10.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.419+0000 7f6aca8bf080 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55e11b4a24b0 con 0x55e11b486e70 2024-03-31T15:05:10.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.419+0000 7f6aca8bf080 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55e11b3fd4b0 con 0x55e11b4a4c60 2024-03-31T15:05:10.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.419+0000 7f6aca8bf080 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x55e11b489c80 con 0x55e11b4a5040 2024-03-31T15:05:10.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.419+0000 7f6ac7a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e11b486e70 0x55e11b52b5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:10.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.419+0000 7f6ac7a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e11b486e70 0x55e11b52b5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34752/0 (socket says 172.21.15.16:34752) 2024-03-31T15:05:10.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.419+0000 7f6ac7a00640 1 -- 172.21.15.16:0/288661081 learned_addr learned my addr 172.21.15.16:0/288661081 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:05:10.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.419+0000 7f6ac8400640 1 --2- 172.21.15.16:0/288661081 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55e11b4a5040 0x55e11b5263d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:10.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.419+0000 7f6ac7000640 1 --2- 172.21.15.16:0/288661081 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e11b4a4c60 0x55e11b488810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:10.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.419+0000 7f6ac7a00640 1 -- 172.21.15.16:0/288661081 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55e11b4a5040 msgr2=0x55e11b5263d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:05:10.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.419+0000 7f6ac7a00640 1 --2- 172.21.15.16:0/288661081 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55e11b4a5040 0x55e11b5263d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:10.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.419+0000 7f6ac7a00640 1 -- 172.21.15.16:0/288661081 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e11b4a4c60 msgr2=0x55e11b488810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:05:10.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.419+0000 7f6ac7a00640 1 --2- 172.21.15.16:0/288661081 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e11b4a4c60 0x55e11b488810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:10.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.419+0000 7f6ac7a00640 1 -- 172.21.15.16:0/288661081 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e11b456b40 con 0x55e11b486e70 2024-03-31T15:05:10.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.419+0000 7f6ac7a00640 1 --2- 172.21.15.16:0/288661081 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e11b486e70 0x55e11b52b5f0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f6ab800a750 tx=0x7f6ab800ac10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af685c7c9b81329a server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:05:10.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.419+0000 7f6ac5c00640 1 -- 172.21.15.16:0/288661081 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ab800ae60 con 0x55e11b486e70 2024-03-31T15:05:10.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.419+0000 7f6ac5c00640 1 -- 172.21.15.16:0/288661081 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6ab8008c00 con 0x55e11b486e70 2024-03-31T15:05:10.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.419+0000 7f6ac5c00640 1 -- 172.21.15.16:0/288661081 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ab80133f0 con 0x55e11b486e70 2024-03-31T15:05:10.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.423+0000 7f6aca8bf080 1 -- 172.21.15.16:0/288661081 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e11b486e70 msgr2=0x55e11b52b5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:10.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.423+0000 7f6aca8bf080 1 --2- 172.21.15.16:0/288661081 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e11b486e70 0x55e11b52b5f0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f6ab800a750 tx=0x7f6ab800ac10 comp rx=0 tx=0).stop 2024-03-31T15:05:10.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.423+0000 7f6aca8bf080 1 -- 172.21.15.16:0/288661081 shutdown_connections 2024-03-31T15:05:10.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.423+0000 7f6aca8bf080 1 --2- 172.21.15.16:0/288661081 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55e11b4a5040 0x55e11b5263d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:10.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.423+0000 7f6aca8bf080 1 --2- 172.21.15.16:0/288661081 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e11b4a4c60 0x55e11b488810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:10.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.423+0000 7f6aca8bf080 1 --2- 172.21.15.16:0/288661081 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e11b486e70 0x55e11b52b5f0 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:10.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.423+0000 7f6aca8bf080 1 -- 172.21.15.16:0/288661081 >> 172.21.15.16:0/288661081 conn(0x55e11b485420 msgr2=0x55e11b52da80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:05:10.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.423+0000 7f6aca8bf080 1 -- 172.21.15.16:0/288661081 shutdown_connections 2024-03-31T15:05:10.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.423+0000 7f6aca8bf080 1 -- 172.21.15.16:0/288661081 wait complete. 2024-03-31T15:05:10.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.423+0000 7f6aca8bf080 1 Processor -- start 2024-03-31T15:05:10.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.423+0000 7f6aca8bf080 1 -- start start 2024-03-31T15:05:10.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.423+0000 7f6aca8bf080 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e11b486e70 0x55e11b513c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:10.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.423+0000 7f6aca8bf080 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e11b4a4c60 0x55e11b52be10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:10.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.423+0000 7f6aca8bf080 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55e11b4a5040 0x55e11b52c350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:10.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.423+0000 7f6aca8bf080 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55e11b456de0 con 0x55e11b4a4c60 2024-03-31T15:05:10.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.423+0000 7f6aca8bf080 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55e11b52a360 con 0x55e11b486e70 2024-03-31T15:05:10.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.423+0000 7f6aca8bf080 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x55e11b457080 con 0x55e11b4a5040 2024-03-31T15:05:10.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.423+0000 7f6ac7000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e11b4a4c60 0x55e11b52be10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:10.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.423+0000 7f6ac7000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e11b4a4c60 0x55e11b52be10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34756/0 (socket says 172.21.15.16:34756) 2024-03-31T15:05:10.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.423+0000 7f6ac7000640 1 -- 172.21.15.16:0/1246699409 learned_addr learned my addr 172.21.15.16:0/1246699409 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:05:10.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.423+0000 7f6ac7a00640 1 --2- 172.21.15.16:0/1246699409 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e11b486e70 0x55e11b513c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:10.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.423+0000 7f6ac8400640 1 --2- 172.21.15.16:0/1246699409 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55e11b4a5040 0x55e11b52c350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:10.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.427+0000 7f6ac7000640 1 -- 172.21.15.16:0/1246699409 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55e11b4a5040 msgr2=0x55e11b52c350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:10.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.427+0000 7f6ac7000640 1 --2- 172.21.15.16:0/1246699409 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55e11b4a5040 0x55e11b52c350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:10.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.427+0000 7f6ac7000640 1 -- 172.21.15.16:0/1246699409 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e11b486e70 msgr2=0x55e11b513c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:10.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.427+0000 7f6ac7000640 1 --2- 172.21.15.16:0/1246699409 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e11b486e70 0x55e11b513c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:10.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.427+0000 7f6ac7000640 1 -- 172.21.15.16:0/1246699409 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e11b49d100 con 0x55e11b4a4c60 2024-03-31T15:05:10.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.427+0000 7f6ac7000640 1 --2- 172.21.15.16:0/1246699409 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e11b4a4c60 0x55e11b52be10 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f6ab4003ef0 tx=0x7f6ab400b5c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:05:10.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.427+0000 7f6abfe00640 1 -- 172.21.15.16:0/1246699409 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ab4014020 con 0x55e11b4a4c60 2024-03-31T15:05:10.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.427+0000 7f6abfe00640 1 -- 172.21.15.16:0/1246699409 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6ab40103f0 con 0x55e11b4a4c60 2024-03-31T15:05:10.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.427+0000 7f6abfe00640 1 -- 172.21.15.16:0/1246699409 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ab400a040 con 0x55e11b4a4c60 2024-03-31T15:05:10.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.427+0000 7f6aca8bf080 1 -- 172.21.15.16:0/1246699409 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55e11b49d3a0 con 0x55e11b4a4c60 2024-03-31T15:05:10.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.427+0000 7f6aca8bf080 1 -- 172.21.15.16:0/1246699409 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e11b49d8e0 con 0x55e11b4a4c60 2024-03-31T15:05:10.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.427+0000 7f6abfe00640 1 -- 172.21.15.16:0/1246699409 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f6ab4017020 con 0x55e11b4a4c60 2024-03-31T15:05:10.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.431+0000 7f6abfe00640 1 --2- 172.21.15.16:0/1246699409 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f6aa8066d90 0x7f6aa8069250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:10.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.431+0000 7f6abfe00640 1 -- 172.21.15.16:0/1246699409 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5709+0+0 (secure 0 0 0) 0x7f6ab4085290 con 0x55e11b4a4c60 2024-03-31T15:05:10.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.431+0000 7f6ac7a00640 1 --2- 172.21.15.16:0/1246699409 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f6aa8066d90 0x7f6aa8069250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:10.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.431+0000 7f6aca8bf080 1 --2- 172.21.15.16:0/1246699409 >> [v2:172.21.15.16:6808/1398737533,v1:172.21.15.16:6810/1398737533] conn(0x55e11b5141d0 0x55e11b49dd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:10.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.431+0000 7f6aca8bf080 1 -- 172.21.15.16:0/1246699409 --> [v2:172.21.15.16:6808/1398737533,v1:172.21.15.16:6810/1398737533] -- osd_op(unknown.0.0:1 5.19 5:997b7129:::100000001f8.00000000:head [getxattr scrub_tag in=9b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e34) -- 0x55e11b49e2c0 con 0x55e11b5141d0 2024-03-31T15:05:10.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.431+0000 7f6ac8400640 1 --2- 172.21.15.16:0/1246699409 >> [v2:172.21.15.16:6808/1398737533,v1:172.21.15.16:6810/1398737533] conn(0x55e11b5141d0 0x55e11b49dd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:10.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.431+0000 7f6ac7a00640 1 --2- 172.21.15.16:0/1246699409 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f6aa8066d90 0x7f6aa8069250 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f6ab800ac50 tx=0x7f6ab800c000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:05:10.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.431+0000 7f6ac8400640 1 --2- 172.21.15.16:0/1246699409 >> [v2:172.21.15.16:6808/1398737533,v1:172.21.15.16:6810/1398737533] conn(0x55e11b5141d0 0x55e11b49dd80 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:05:10.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.431+0000 7f6ac8400640 1 -- 172.21.15.16:0/1246699409 <== osd.1 v2:172.21.15.16:6808/1398737533 1 ==== osd_op_reply(1 100000001f8.00000000 [getxattr (9) out=9b] v0'0 uv12 ondisk = 0) ==== 164+0+9 (crc 0 0 0) 0x7f6ac00b6420 con 0x55e11b5141d0 2024-03-31T15:05:10.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.431+0000 7f6aca8bf080 1 -- 172.21.15.16:0/1246699409 >> [v2:172.21.15.16:6808/1398737533,v1:172.21.15.16:6810/1398737533] conn(0x55e11b5141d0 msgr2=0x55e11b49dd80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:10.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.431+0000 7f6aca8bf080 1 --2- 172.21.15.16:0/1246699409 >> [v2:172.21.15.16:6808/1398737533,v1:172.21.15.16:6810/1398737533] conn(0x55e11b5141d0 0x55e11b49dd80 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:10.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.431+0000 7f6aca8bf080 1 -- 172.21.15.16:0/1246699409 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f6aa8066d90 msgr2=0x7f6aa8069250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:10.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.431+0000 7f6aca8bf080 1 --2- 172.21.15.16:0/1246699409 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f6aa8066d90 0x7f6aa8069250 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f6ab800ac50 tx=0x7f6ab800c000 comp rx=0 tx=0).stop 2024-03-31T15:05:10.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.431+0000 7f6aca8bf080 1 -- 172.21.15.16:0/1246699409 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e11b4a4c60 msgr2=0x55e11b52be10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:10.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.431+0000 7f6aca8bf080 1 --2- 172.21.15.16:0/1246699409 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e11b4a4c60 0x55e11b52be10 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f6ab4003ef0 tx=0x7f6ab400b5c0 comp rx=0 tx=0).stop 2024-03-31T15:05:10.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.431+0000 7f6ac8400640 1 -- 172.21.15.16:0/1246699409 reap_dead start 2024-03-31T15:05:10.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.431+0000 7f6aca8bf080 1 -- 172.21.15.16:0/1246699409 shutdown_connections 2024-03-31T15:05:10.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.431+0000 7f6aca8bf080 1 -- 172.21.15.16:0/1246699409 >> 172.21.15.16:0/1246699409 conn(0x55e11b485420 msgr2=0x55e11b5c2da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:05:10.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.431+0000 7f6aca8bf080 1 -- 172.21.15.16:0/1246699409 shutdown_connections 2024-03-31T15:05:10.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.431+0000 7f6aca8bf080 1 -- 172.21.15.16:0/1246699409 wait complete. 2024-03-31T15:05:10.431 INFO:teuthology.orchestra.run.smithi016.stdout:mytag 2024-03-31T15:05:10.431 DEBUG:teuthology.orchestra.run.smithi016:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_data getxattr 10000000002.00000000 scrub_tag 2024-03-31T15:05:10.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.523+0000 7f6053311080 1 Processor -- start 2024-03-31T15:05:10.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.523+0000 7f6053311080 1 -- start start 2024-03-31T15:05:10.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.523+0000 7f6053311080 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fc8c216350 0x55fc8c289d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:10.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.523+0000 7f6053311080 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55fc8c216730 0x55fc8c281ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:10.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.523+0000 7f6053311080 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fc8c2869a0 0x55fc8c2823e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:10.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.523+0000 7f6053311080 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55fc8c1724b0 con 0x55fc8c2869a0 2024-03-31T15:05:10.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.523+0000 7f6053311080 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55fc8c176500 con 0x55fc8c216350 2024-03-31T15:05:10.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.523+0000 7f6053311080 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x55fc8c287f20 con 0x55fc8c216730 2024-03-31T15:05:10.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.523+0000 7f6051000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fc8c2869a0 0x55fc8c2823e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:10.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.523+0000 7f6051000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fc8c2869a0 0x55fc8c2823e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34766/0 (socket says 172.21.15.16:34766) 2024-03-31T15:05:10.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.523+0000 7f6051000640 1 -- 172.21.15.16:0/3376123761 learned_addr learned my addr 172.21.15.16:0/3376123761 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:05:10.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.523+0000 7f6050600640 1 --2- 172.21.15.16:0/3376123761 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fc8c216350 0x55fc8c289d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:10.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.523+0000 7f604fc00640 1 --2- 172.21.15.16:0/3376123761 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55fc8c216730 0x55fc8c281ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:10.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.523+0000 7f6051000640 1 -- 172.21.15.16:0/3376123761 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55fc8c216730 msgr2=0x55fc8c281ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:05:10.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.523+0000 7f6051000640 1 --2- 172.21.15.16:0/3376123761 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55fc8c216730 0x55fc8c281ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:10.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.523+0000 7f6051000640 1 -- 172.21.15.16:0/3376123761 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fc8c216350 msgr2=0x55fc8c289d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:05:10.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.523+0000 7f6051000640 1 --2- 172.21.15.16:0/3376123761 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fc8c216350 0x55fc8c289d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:10.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.523+0000 7f6051000640 1 -- 172.21.15.16:0/3376123761 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fc8c282a70 con 0x55fc8c2869a0 2024-03-31T15:05:10.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.523+0000 7f604fc00640 1 --2- 172.21.15.16:0/3376123761 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55fc8c216730 0x55fc8c281ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:05:10.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.523+0000 7f6050600640 1 --2- 172.21.15.16:0/3376123761 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fc8c216350 0x55fc8c289d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:05:10.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.523+0000 7f6051000640 1 --2- 172.21.15.16:0/3376123761 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fc8c2869a0 0x55fc8c2823e0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f60480016a0 tx=0x7f6048006f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a41b81a2034c145a server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:05:10.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.523+0000 7f604e800640 1 -- 172.21.15.16:0/3376123761 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6048004530 con 0x55fc8c2869a0 2024-03-31T15:05:10.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.523+0000 7f604e800640 1 -- 172.21.15.16:0/3376123761 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f60480046d0 con 0x55fc8c2869a0 2024-03-31T15:05:10.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.523+0000 7f604e800640 1 -- 172.21.15.16:0/3376123761 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6048004ac0 con 0x55fc8c2869a0 2024-03-31T15:05:10.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.527+0000 7f6053311080 1 -- 172.21.15.16:0/3376123761 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fc8c2869a0 msgr2=0x55fc8c2823e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:10.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.527+0000 7f6053311080 1 --2- 172.21.15.16:0/3376123761 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fc8c2869a0 0x55fc8c2823e0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f60480016a0 tx=0x7f6048006f90 comp rx=0 tx=0).stop 2024-03-31T15:05:10.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.527+0000 7f6053311080 1 -- 172.21.15.16:0/3376123761 shutdown_connections 2024-03-31T15:05:10.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.527+0000 7f6053311080 1 --2- 172.21.15.16:0/3376123761 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fc8c2869a0 0x55fc8c2823e0 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:10.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.527+0000 7f6053311080 1 --2- 172.21.15.16:0/3376123761 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55fc8c216730 0x55fc8c281ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:10.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.527+0000 7f6053311080 1 --2- 172.21.15.16:0/3376123761 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fc8c216350 0x55fc8c289d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:10.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.527+0000 7f6053311080 1 -- 172.21.15.16:0/3376123761 >> 172.21.15.16:0/3376123761 conn(0x55fc8c1f8050 msgr2=0x55fc8c2a43f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:05:10.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.527+0000 7f6053311080 1 -- 172.21.15.16:0/3376123761 shutdown_connections 2024-03-31T15:05:10.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.527+0000 7f6053311080 1 -- 172.21.15.16:0/3376123761 wait complete. 2024-03-31T15:05:10.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.527+0000 7f6053311080 1 Processor -- start 2024-03-31T15:05:10.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.527+0000 7f6053311080 1 -- start start 2024-03-31T15:05:10.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.527+0000 7f6053311080 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55fc8c216350 0x55fc8c2a61a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:10.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.527+0000 7f6053311080 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fc8c216730 0x55fc8c28acf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:10.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.527+0000 7f6053311080 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fc8c2869a0 0x55fc8c28b230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:10.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.527+0000 7f6053311080 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55fc8c2895d0 con 0x55fc8c2869a0 2024-03-31T15:05:10.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.527+0000 7f6053311080 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55fc8c282d10 con 0x55fc8c216730 2024-03-31T15:05:10.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.527+0000 7f6053311080 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x55fc8c289870 con 0x55fc8c216350 2024-03-31T15:05:10.525 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.527+0000 7f604fc00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fc8c216730 0x55fc8c28acf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:10.525 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.527+0000 7f604fc00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fc8c216730 0x55fc8c28acf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.16:56878/0 (socket says 172.21.15.16:56878) 2024-03-31T15:05:10.525 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.527+0000 7f604fc00640 1 -- 172.21.15.16:0/286578459 learned_addr learned my addr 172.21.15.16:0/286578459 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:05:10.525 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.527+0000 7f6050600640 1 --2- 172.21.15.16:0/286578459 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55fc8c216350 0x55fc8c2a61a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:10.525 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.527+0000 7f6051000640 1 --2- 172.21.15.16:0/286578459 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fc8c2869a0 0x55fc8c28b230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:10.525 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.527+0000 7f604fc00640 1 -- 172.21.15.16:0/286578459 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55fc8c216350 msgr2=0x55fc8c2a61a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:10.525 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.527+0000 7f604fc00640 1 --2- 172.21.15.16:0/286578459 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55fc8c216350 0x55fc8c2a61a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:10.525 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.527+0000 7f604fc00640 1 -- 172.21.15.16:0/286578459 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fc8c2869a0 msgr2=0x55fc8c28b230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:10.525 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.527+0000 7f604fc00640 1 --2- 172.21.15.16:0/286578459 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fc8c2869a0 0x55fc8c28b230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:10.525 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.527+0000 7f604fc00640 1 -- 172.21.15.16:0/286578459 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fc8c1fbb70 con 0x55fc8c216730 2024-03-31T15:05:10.525 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.527+0000 7f6051000640 1 --2- 172.21.15.16:0/286578459 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fc8c2869a0 0x55fc8c28b230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:05:10.525 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.527+0000 7f604fc00640 1 --2- 172.21.15.16:0/286578459 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fc8c216730 0x55fc8c28acf0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f603c00df90 tx=0x7f603c00b680 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:05:10.525 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.527+0000 7f604d400640 1 -- 172.21.15.16:0/286578459 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f603c014020 con 0x55fc8c216730 2024-03-31T15:05:10.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.527+0000 7f6053311080 1 -- 172.21.15.16:0/286578459 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55fc8c215580 con 0x55fc8c216730 2024-03-31T15:05:10.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.531+0000 7f604d400640 1 -- 172.21.15.16:0/286578459 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f603c001640 con 0x55fc8c216730 2024-03-31T15:05:10.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.531+0000 7f604d400640 1 -- 172.21.15.16:0/286578459 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f603c00a040 con 0x55fc8c216730 2024-03-31T15:05:10.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.531+0000 7f6053311080 1 -- 172.21.15.16:0/286578459 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fc8c215ac0 con 0x55fc8c216730 2024-03-31T15:05:10.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.531+0000 7f604d400640 1 -- 172.21.15.16:0/286578459 <== mon.1 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f603c017020 con 0x55fc8c216730 2024-03-31T15:05:10.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.531+0000 7f604d400640 1 --2- 172.21.15.16:0/286578459 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f6030066d60 0x7f6030069220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:10.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.531+0000 7f604d400640 1 -- 172.21.15.16:0/286578459 <== mon.1 v2:172.21.15.89:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5709+0+0 (secure 0 0 0) 0x7f603c085530 con 0x55fc8c216730 2024-03-31T15:05:10.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.531+0000 7f6053311080 1 --2- 172.21.15.16:0/286578459 >> [v2:172.21.15.16:6808/1398737533,v1:172.21.15.16:6810/1398737533] conn(0x55fc8c33d850 0x55fc8c2143e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:10.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.531+0000 7f6053311080 1 -- 172.21.15.16:0/286578459 --> [v2:172.21.15.16:6808/1398737533,v1:172.21.15.16:6810/1398737533] -- osd_op(unknown.0.0:1 5.12 5:4bec3886:::10000000002.00000000:head [getxattr scrub_tag in=9b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e34) -- 0x55fc8c214920 con 0x55fc8c33d850 2024-03-31T15:05:10.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.531+0000 7f6050600640 1 --2- 172.21.15.16:0/286578459 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f6030066d60 0x7f6030069220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:10.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.531+0000 7f6051000640 1 --2- 172.21.15.16:0/286578459 >> [v2:172.21.15.16:6808/1398737533,v1:172.21.15.16:6810/1398737533] conn(0x55fc8c33d850 0x55fc8c2143e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:10.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.531+0000 7f6051000640 1 --2- 172.21.15.16:0/286578459 >> [v2:172.21.15.16:6808/1398737533,v1:172.21.15.16:6810/1398737533] conn(0x55fc8c33d850 0x55fc8c2143e0 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:05:10.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.535+0000 7f6050600640 1 --2- 172.21.15.16:0/286578459 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f6030066d60 0x7f6030069220 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f60400ae000 tx=0x7f60400a6000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:05:10.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.535+0000 7f6051000640 1 -- 172.21.15.16:0/286578459 <== osd.1 v2:172.21.15.16:6808/1398737533 1 ==== osd_op_reply(1 10000000002.00000000 [getxattr] v0'0 uv0 ondisk = -61 ((61) No data available)) ==== 164+0+0 (crc 0 0 0) 0x7f604800f030 con 0x55fc8c33d850 2024-03-31T15:05:10.529 INFO:teuthology.orchestra.run.smithi016.stderr:error getting xattr cephfs_data/10000000002.00000000/scrub_tag: (61) No data available 2024-03-31T15:05:10.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.535+0000 7f6053311080 1 -- 172.21.15.16:0/286578459 >> [v2:172.21.15.16:6808/1398737533,v1:172.21.15.16:6810/1398737533] conn(0x55fc8c33d850 msgr2=0x55fc8c2143e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:10.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.535+0000 7f6053311080 1 --2- 172.21.15.16:0/286578459 >> [v2:172.21.15.16:6808/1398737533,v1:172.21.15.16:6810/1398737533] conn(0x55fc8c33d850 0x55fc8c2143e0 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:10.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.535+0000 7f6053311080 1 -- 172.21.15.16:0/286578459 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f6030066d60 msgr2=0x7f6030069220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:10.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.535+0000 7f6053311080 1 --2- 172.21.15.16:0/286578459 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f6030066d60 0x7f6030069220 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f60400ae000 tx=0x7f60400a6000 comp rx=0 tx=0).stop 2024-03-31T15:05:10.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.535+0000 7f6053311080 1 -- 172.21.15.16:0/286578459 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fc8c216730 msgr2=0x55fc8c28acf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:10.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.535+0000 7f6053311080 1 --2- 172.21.15.16:0/286578459 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fc8c216730 0x55fc8c28acf0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f603c00df90 tx=0x7f603c00b680 comp rx=0 tx=0).stop 2024-03-31T15:05:10.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.535+0000 7f6051000640 1 -- 172.21.15.16:0/286578459 reap_dead start 2024-03-31T15:05:10.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.535+0000 7f6053311080 1 -- 172.21.15.16:0/286578459 shutdown_connections 2024-03-31T15:05:10.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.535+0000 7f6053311080 1 -- 172.21.15.16:0/286578459 >> 172.21.15.16:0/286578459 conn(0x55fc8c1f8050 msgr2=0x55fc8c1cae20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:05:10.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.535+0000 7f6053311080 1 -- 172.21.15.16:0/286578459 shutdown_connections 2024-03-31T15:05:10.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:10.535+0000 7f6053311080 1 -- 172.21.15.16:0/286578459 wait complete. 2024-03-31T15:05:10.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-31T15:05:10.534 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-31T15:05:10.534 DEBUG:teuthology.orchestra.run.smithi016:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-31T15:05:10.646 DEBUG:teuthology.orchestra.run.smithi089:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-31T15:05:10.741 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-31T15:05:10.741 INFO:tasks.cephfs.mount:Terminating background process 2024-03-31T15:05:10.741 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-03-31T15:05:10.742 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-31T15:05:10.750 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16357708k,nr_inodes=4089427,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275996k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=6539 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275992k,nr_inodes=818998,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275996k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-31T15:05:10.750 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi016.front.sepia.ceph.com 2024-03-31T15:05:10.750 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:05:10.750 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-31T15:05:10.799 INFO:teuthology.orchestra.run.smithi016.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-03-31T15:05:10.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-31T15:05:10.800 INFO:tasks.cephfs.mount:Terminating background process 2024-03-31T15:05:10.801 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-03-31T15:05:10.801 DEBUG:teuthology.orchestra.run.smithi089:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-31T15:05:10.810 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16354452k,nr_inodes=4088613,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275344k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=901 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275340k,nr_inodes=818835,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275344k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-31T15:05:10.810 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi089.front.sepia.ceph.com 2024-03-31T15:05:10.810 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:05:10.810 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-03-31T15:05:10.858 INFO:teuthology.orchestra.run.smithi089.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.1': No such file or directory 2024-03-31T15:05:10.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-31T15:05:10.859 INFO:tasks.cephfs.mount:Terminating background process 2024-03-31T15:05:10.859 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-03-31T15:05:10.859 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-31T15:05:10.867 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16357708k,nr_inodes=4089427,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275996k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=6539 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275992k,nr_inodes=818998,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275996k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-31T15:05:10.867 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi016.front.sepia.ceph.com 2024-03-31T15:05:10.867 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:05:10.868 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-03-31T15:05:10.915 INFO:teuthology.orchestra.run.smithi016.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2024-03-31T15:05:10.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-31T15:05:10.916 INFO:tasks.cephfs.mount:Terminating background process 2024-03-31T15:05:10.916 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-03-31T15:05:10.917 DEBUG:teuthology.orchestra.run.smithi089:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-31T15:05:10.925 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16354452k,nr_inodes=4088613,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275344k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=901 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275340k,nr_inodes=818835,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275344k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-31T15:05:10.925 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi089.front.sepia.ceph.com 2024-03-31T15:05:10.925 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:05:10.925 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-03-31T15:05:10.974 INFO:teuthology.orchestra.run.smithi089.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2024-03-31T15:05:10.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-31T15:05:10.975 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-31T15:05:11.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:11.119+0000 7f147f400640 1 Processor -- start 2024-03-31T15:05:11.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:11.119+0000 7f147f400640 1 -- start start 2024-03-31T15:05:11.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:11.119+0000 7f147f400640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f147806f920 0x7f147806fd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:11.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:11.119+0000 7f147f400640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f14780d2010 0x7f14780d24a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:11.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:11.119+0000 7f147f400640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f14780d0920 con 0x7f1478145e40 2024-03-31T15:05:11.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:11.119+0000 7f147f400640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f14780702c0 con 0x7f147806f920 2024-03-31T15:05:11.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:11.119+0000 7f147f400640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f14780d06d0 con 0x7f14780d2010 2024-03-31T15:05:11.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:11.119+0000 7f147ca00640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f1478145e40 0x7f14780cc740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:36440/0 (socket says 172.21.15.16:36440) 2024-03-31T15:05:11.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:11.119+0000 7f147ca00640 1 -- 172.21.15.16:0/2943833482 learned_addr learned my addr 172.21.15.16:0/2943833482 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:05:11.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:11.119+0000 7f146fe00640 1 --2- 172.21.15.16:0/2943833482 >> v2:172.21.15.89:3300/0 conn(0x7f147806f920 0x7f147806fd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:11.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:11.119+0000 7f147d400640 1 --2- 172.21.15.16:0/2943833482 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f14780d2010 0x7f14780d24a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:11.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:11.123+0000 7f1477e00640 1 -- 172.21.15.16:0/2943833482 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 803056184 0 0) 0x7f14780d0920 con 0x7f1478145e40 2024-03-31T15:05:11.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:11.123+0000 7f1477e00640 1 -- 172.21.15.16:0/2943833482 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f145c003530 con 0x7f1478145e40 2024-03-31T15:05:11.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:11.123+0000 7f1477e00640 1 -- 172.21.15.16:0/2943833482 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 429871106 0 0) 0x7f145c003530 con 0x7f1478145e40 2024-03-31T15:05:11.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:11.123+0000 7f1477e00640 1 -- 172.21.15.16:0/2943833482 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f14780d2010 msgr2=0x7f14780d24a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:05:11.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:11.123+0000 7f1477e00640 1 --2- 172.21.15.16:0/2943833482 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f14780d2010 0x7f14780d24a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:11.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:11.123+0000 7f1477e00640 1 -- 172.21.15.16:0/2943833482 >> v2:172.21.15.89:3300/0 conn(0x7f147806f920 msgr2=0x7f147806fd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:05:11.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:11.123+0000 7f1477e00640 1 --2- 172.21.15.16:0/2943833482 >> v2:172.21.15.89:3300/0 conn(0x7f147806f920 0x7f147806fd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:11.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:11.123+0000 7f1477e00640 1 -- 172.21.15.16:0/2943833482 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14780d0c00 con 0x7f1478145e40 2024-03-31T15:05:11.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:11.123+0000 7f1477e00640 1 -- 172.21.15.16:0/2943833482 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 214807623 0 0) 0x7f1468001680 con 0x7f1478145e40 2024-03-31T15:05:11.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:11.123+0000 7f1477e00640 1 -- 172.21.15.16:0/2943833482 >> v1:172.21.15.16:6789/0 conn(0x7f1478145e40 legacy=0x7f14780cc740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:11.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:11.123+0000 7f1477e00640 1 --2- 172.21.15.16:0/2943833482 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f14780d2010 0x7f145c0039c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:11.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:11.123+0000 7f1477e00640 1 --2- 172.21.15.16:0/2943833482 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f145c003f90 0x7f145c004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:11.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:11.123+0000 7f1477e00640 1 --2- 172.21.15.16:0/2943833482 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f145c004d50 0x7f145c0051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:11.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:11.123+0000 7f1477e00640 1 -- 172.21.15.16:0/2943833482 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f14780d06d0 con 0x7f145c003f90 2024-03-31T15:05:11.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:11.123+0000 7f1477e00640 1 -- 172.21.15.16:0/2943833482 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f145c00b9d0 con 0x7f145c004d50 2024-03-31T15:05:11.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:11.123+0000 7f1477e00640 1 -- 172.21.15.16:0/2943833482 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f145c00bb50 con 0x7f14780d2010 2024-03-31T15:05:11.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:11.123+0000 7f146fe00640 1 --2- 172.21.15.16:0/2943833482 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f145c003f90 0x7f145c004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:11.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:11.123+0000 7f147ca00640 1 --2- 172.21.15.16:0/2943833482 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f14780d2010 0x7f145c0039c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:11.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:11.123+0000 7f147d400640 1 --2- 172.21.15.16:0/2943833482 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f145c004d50 0x7f145c0051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:11.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:11.123+0000 7f146fe00640 1 -- 172.21.15.16:0/2943833482 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f14780d2010 msgr2=0x7f145c0039c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:05:11.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:11.123+0000 7f146fe00640 1 --2- 172.21.15.16:0/2943833482 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f14780d2010 0x7f145c0039c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:11.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:11.123+0000 7f146fe00640 1 -- 172.21.15.16:0/2943833482 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f145c004d50 msgr2=0x7f145c0051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:05:11.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:11.123+0000 7f146fe00640 1 --2- 172.21.15.16:0/2943833482 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f145c004d50 0x7f145c0051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:11.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:11.123+0000 7f146fe00640 1 -- 172.21.15.16:0/2943833482 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f145c003530 con 0x7f145c003f90 2024-03-31T15:05:11.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:11.123+0000 7f147d400640 1 --2- 172.21.15.16:0/2943833482 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f145c004d50 0x7f145c0051d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:05:11.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:11.127+0000 7f146fe00640 1 --2- 172.21.15.16:0/2943833482 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f145c003f90 0x7f145c004810 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f1460009b10 tx=0x7f146001cab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d8577bcd4dab391 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:05:11.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:11.127+0000 7f1477e00640 1 -- 172.21.15.16:0/2943833482 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14600b2070 con 0x7f145c003f90 2024-03-31T15:05:11.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:11.127+0000 7f1477e00640 1 -- 172.21.15.16:0/2943833482 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f14600a56a0 con 0x7f145c003f90 2024-03-31T15:05:11.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:11.127+0000 7f1477e00640 1 -- 172.21.15.16:0/2943833482 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14600096b0 con 0x7f145c003f90 2024-03-31T15:05:11.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:11.127+0000 7f147f400640 1 -- 172.21.15.16:0/2943833482 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f145c003f90 msgr2=0x7f145c004810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:11.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:11.127+0000 7f147f400640 1 --2- 172.21.15.16:0/2943833482 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f145c003f90 0x7f145c004810 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f1460009b10 tx=0x7f146001cab0 comp rx=0 tx=0).stop 2024-03-31T15:05:11.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:11.127+0000 7f147d400640 1 -- 172.21.15.16:0/2943833482 reap_dead start 2024-03-31T15:05:11.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:11.127+0000 7f147f400640 1 -- 172.21.15.16:0/2943833482 shutdown_connections 2024-03-31T15:05:11.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:11.127+0000 7f147f400640 1 -- 172.21.15.16:0/2943833482 >> 172.21.15.16:0/2943833482 conn(0x7f1478067330 msgr2=0x7f1478067740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:05:11.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:11.127+0000 7f147f400640 1 -- 172.21.15.16:0/2943833482 shutdown_connections 2024-03-31T15:05:11.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:11.127+0000 7f147f400640 1 -- 172.21.15.16:0/2943833482 wait complete. 2024-03-31T15:05:11.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:11.131+0000 7f147f400640 1 Processor -- start 2024-03-31T15:05:11.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:11.131+0000 7f147f400640 1 -- start start 2024-03-31T15:05:11.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:11.131+0000 7f147f400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f145c003f90 0x7f1478141a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:11.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:11.131+0000 7f147f400640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f14780b1c30 0x7f14780b20b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:11.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:11.131+0000 7f147f400640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f14780abb80 0x7f14780ac030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:11.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:11.131+0000 7f147f400640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f14780cce50 con 0x7f145c003f90 2024-03-31T15:05:11.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:11.131+0000 7f147f400640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f14780d1af0 con 0x7f14780b1c30 2024-03-31T15:05:11.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:11.131+0000 7f147f400640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f14780ce9e0 con 0x7f14780abb80 2024-03-31T15:05:11.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:11.131+0000 7f147ca00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f145c003f90 0x7f1478141a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:11.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:11.131+0000 7f147ca00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f145c003f90 0x7f1478141a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34792/0 (socket says 172.21.15.16:34792) 2024-03-31T15:05:11.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:11.131+0000 7f147ca00640 1 -- 172.21.15.16:0/737613399 learned_addr learned my addr 172.21.15.16:0/737613399 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:05:11.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:11.131+0000 7f147d400640 1 --2- 172.21.15.16:0/737613399 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f14780abb80 0x7f14780ac030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:11.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:11.131+0000 7f146fe00640 1 --2- 172.21.15.16:0/737613399 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f14780b1c30 0x7f14780b20b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:11.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:11.131+0000 7f147ca00640 1 -- 172.21.15.16:0/737613399 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f14780abb80 msgr2=0x7f14780ac030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:11.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:11.131+0000 7f147ca00640 1 --2- 172.21.15.16:0/737613399 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f14780abb80 0x7f14780ac030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:11.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:11.131+0000 7f147ca00640 1 -- 172.21.15.16:0/737613399 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f14780b1c30 msgr2=0x7f14780b20b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:11.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:11.131+0000 7f147ca00640 1 --2- 172.21.15.16:0/737613399 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f14780b1c30 0x7f14780b20b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:11.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:11.131+0000 7f147ca00640 1 -- 172.21.15.16:0/737613399 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14781f4740 con 0x7f145c003f90 2024-03-31T15:05:11.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:11.131+0000 7f146fe00640 1 --2- 172.21.15.16:0/737613399 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f14780b1c30 0x7f14780b20b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:05:11.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:11.131+0000 7f147ca00640 1 --2- 172.21.15.16:0/737613399 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f145c003f90 0x7f1478141a70 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f1468012310 tx=0x7f14680127d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:05:11.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:11.131+0000 7f1475600640 1 -- 172.21.15.16:0/737613399 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f146801c070 con 0x7f145c003f90 2024-03-31T15:05:11.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:11.131+0000 7f1475600640 1 -- 172.21.15.16:0/737613399 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1468014750 con 0x7f145c003f90 2024-03-31T15:05:11.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:11.131+0000 7f1475600640 1 -- 172.21.15.16:0/737613399 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1468014b20 con 0x7f145c003f90 2024-03-31T15:05:11.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:11.131+0000 7f147f400640 1 -- 172.21.15.16:0/737613399 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f14781f49e0 con 0x7f145c003f90 2024-03-31T15:05:11.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:11.131+0000 7f147f400640 1 -- 172.21.15.16:0/737613399 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f14781f4fa0 con 0x7f145c003f90 2024-03-31T15:05:11.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:11.135+0000 7f147f400640 1 -- 172.21.15.16:0/737613399 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1478069950 con 0x7f145c003f90 2024-03-31T15:05:11.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:11.135+0000 7f1475600640 1 -- 172.21.15.16:0/737613399 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f1468014cc0 con 0x7f145c003f90 2024-03-31T15:05:11.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:11.135+0000 7f1475600640 1 --2- 172.21.15.16:0/737613399 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f14540683b0 0x7f145406a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:11.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:11.135+0000 7f1475600640 1 -- 172.21.15.16:0/737613399 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5709+0+0 (secure 0 0 0) 0x7f1468089420 con 0x7f145c003f90 2024-03-31T15:05:11.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:11.135+0000 7f146fe00640 1 --2- 172.21.15.16:0/737613399 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f14540683b0 0x7f145406a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:11.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:11.135+0000 7f146fe00640 1 --2- 172.21.15.16:0/737613399 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f14540683b0 0x7f145406a870 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f14600af000 tx=0x7f14600046d0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:05:11.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:11.139+0000 7f1475600640 1 -- 172.21.15.16:0/737613399 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f1468059940 con 0x7f145c003f90 2024-03-31T15:05:11.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:11.339+0000 7f147f400640 1 -- 172.21.15.16:0/737613399 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f14780cf3c0 con 0x7f145c003f90 2024-03-31T15:05:11.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:11.339+0000 7f1475600640 1 -- 172.21.15.16:0/737613399 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) ==== 94+0+5178 (secure 0 0 0) 0x7f1468059240 con 0x7f145c003f90 2024-03-31T15:05:11.334 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:05:11.335 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":16,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4949,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6836/2073954753","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":2073954753},{"type":"v1","addr":"172.21.15.89:6836","nonce":2073954753}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":4952,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/2143918278","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6835","nonce":2143918278},{"type":"v1","addr":"172.21.15.89:6837","nonce":2143918278}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":5251,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6833/881391342","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":881391342},{"type":"v1","addr":"172.21.15.16:6833","nonce":881391342}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-31T15:04:39.780323+0000","modified":"2024-03-31T15:04:42.846439+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5254},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5254":{"gid":5254,"name":"c","rank":0,"incarnation":13,"state":"up:active","state_seq":3,"addr":"172.21.15.16:6835/1845707550","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":1845707550},{"type":"v1","addr":"172.21.15.16:6835","nonce":1845707550}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5254,"qdb_cluster":[5254]},"id":2}]} 2024-03-31T15:05:11.335 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 16 2024-03-31T15:05:11.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:11.339+0000 7f147f400640 1 -- 172.21.15.16:0/737613399 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f14540683b0 msgr2=0x7f145406a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:11.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:11.339+0000 7f147f400640 1 --2- 172.21.15.16:0/737613399 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f14540683b0 0x7f145406a870 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f14600af000 tx=0x7f14600046d0 comp rx=0 tx=0).stop 2024-03-31T15:05:11.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:11.339+0000 7f147f400640 1 -- 172.21.15.16:0/737613399 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f145c003f90 msgr2=0x7f1478141a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:11.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:11.339+0000 7f147f400640 1 --2- 172.21.15.16:0/737613399 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f145c003f90 0x7f1478141a70 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f1468012310 tx=0x7f14680127d0 comp rx=0 tx=0).stop 2024-03-31T15:05:11.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:11.339+0000 7f147f400640 1 -- 172.21.15.16:0/737613399 shutdown_connections 2024-03-31T15:05:11.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:11.339+0000 7f147f400640 1 --2- 172.21.15.16:0/737613399 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f14540683b0 0x7f145406a870 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:11.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:11.339+0000 7f147f400640 1 --2- 172.21.15.16:0/737613399 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f14780abb80 0x7f14780ac030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:11.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:11.339+0000 7f147f400640 1 --2- 172.21.15.16:0/737613399 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f14780b1c30 0x7f14780b20b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:11.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:11.339+0000 7f147f400640 1 --2- 172.21.15.16:0/737613399 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f145c003f90 0x7f1478141a70 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:11.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:11.339+0000 7f147f400640 1 -- 172.21.15.16:0/737613399 >> 172.21.15.16:0/737613399 conn(0x7f1478067330 msgr2=0x7f14780cfca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:05:11.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:11.339+0000 7f147f400640 1 -- 172.21.15.16:0/737613399 shutdown_connections 2024-03-31T15:05:11.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:11.339+0000 7f147f400640 1 -- 172.21.15.16:0/737613399 wait complete. 2024-03-31T15:05:11.362 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-31T15:05:11.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:11.511+0000 7f7578200640 1 Processor -- start 2024-03-31T15:05:11.507 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:11.511+0000 7f7578200640 1 -- start start 2024-03-31T15:05:11.508 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:11.511+0000 7f7578200640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f757006dbc0 0x7f757015f650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:11.508 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:11.511+0000 7f7578200640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f75700cbe90 con 0x7f75700cb440 2024-03-31T15:05:11.508 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:11.511+0000 7f7578200640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f75700cc170 con 0x7f7570070030 2024-03-31T15:05:11.508 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:11.511+0000 7f7578200640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f757015fb90 con 0x7f757006dbc0 2024-03-31T15:05:11.508 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:11.511+0000 7f7575800640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f757006dbc0 0x7f757015f650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:11.508 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:11.511+0000 7f7574e00640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f75700cb440 0x7f757006f920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:36456/0 (socket says 172.21.15.16:36456) 2024-03-31T15:05:11.508 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:11.511+0000 7f7574e00640 1 -- 172.21.15.16:0/3408640301 learned_addr learned my addr 172.21.15.16:0/3408640301 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:05:11.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:11.515+0000 7f7575800640 1 -- 172.21.15.16:0/3408640301 >> v1:172.21.15.89:6789/0 conn(0x7f7570070030 legacy=0x7f7570162180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:11.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:11.515+0000 7f7575800640 1 -- 172.21.15.16:0/3408640301 >> v1:172.21.15.16:6789/0 conn(0x7f75700cb440 legacy=0x7f757006f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:11.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:11.515+0000 7f7575800640 1 -- 172.21.15.16:0/3408640301 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f757015ee50 con 0x7f757006dbc0 2024-03-31T15:05:11.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:11.515+0000 7f756fe00640 1 -- 172.21.15.16:0/3408640301 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4280630218 0 0) 0x7f75700cbe90 con 0x7f75700cb440 2024-03-31T15:05:11.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:11.515+0000 7f7575800640 1 --2- 172.21.15.16:0/3408640301 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f757006dbc0 0x7f757015f650 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f75640097c0 tx=0x7f756401c410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d1ff0904b55950b1 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:05:11.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:11.515+0000 7f756fe00640 1 -- 172.21.15.16:0/3408640301 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75640b0070 con 0x7f757006dbc0 2024-03-31T15:05:11.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:11.515+0000 7f756fe00640 1 -- 172.21.15.16:0/3408640301 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f75640ab600 con 0x7f757006dbc0 2024-03-31T15:05:11.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:11.515+0000 7f756fe00640 1 -- 172.21.15.16:0/3408640301 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75640ab9d0 con 0x7f757006dbc0 2024-03-31T15:05:11.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:11.515+0000 7f7578200640 1 -- 172.21.15.16:0/3408640301 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f757006dbc0 msgr2=0x7f757015f650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:11.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:11.515+0000 7f7578200640 1 --2- 172.21.15.16:0/3408640301 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f757006dbc0 0x7f757015f650 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f75640097c0 tx=0x7f756401c410 comp rx=0 tx=0).stop 2024-03-31T15:05:11.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:11.515+0000 7f7578200640 1 -- 172.21.15.16:0/3408640301 shutdown_connections 2024-03-31T15:05:11.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:11.515+0000 7f7578200640 1 --2- 172.21.15.16:0/3408640301 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f757006dbc0 0x7f757015f650 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:11.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:11.515+0000 7f7578200640 1 -- 172.21.15.16:0/3408640301 >> 172.21.15.16:0/3408640301 conn(0x7f7570067330 msgr2=0x7f7570067740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:05:11.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:11.515+0000 7f7578200640 1 -- 172.21.15.16:0/3408640301 shutdown_connections 2024-03-31T15:05:11.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:11.515+0000 7f7578200640 1 -- 172.21.15.16:0/3408640301 wait complete. 2024-03-31T15:05:11.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:11.519+0000 7f7578200640 1 Processor -- start 2024-03-31T15:05:11.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:11.519+0000 7f7578200640 1 -- start start 2024-03-31T15:05:11.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:11.523+0000 7f7578200640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f757006dbc0 0x7f75701406a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:11.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:11.523+0000 7f7578200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7570070030 0x7f75700cd740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:11.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:11.523+0000 7f7578200640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f75700cb440 0x7f75700cdc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:11.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:11.523+0000 7f7578200640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f757006cad0 con 0x7f7570070030 2024-03-31T15:05:11.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:11.523+0000 7f7578200640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f757006c950 con 0x7f757006dbc0 2024-03-31T15:05:11.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:11.523+0000 7f7578200640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f757016cd40 con 0x7f75700cb440 2024-03-31T15:05:11.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:11.523+0000 7f7574e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7570070030 0x7f75700cd740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:11.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:11.523+0000 7f7575800640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f757006dbc0 0x7f75701406a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:11.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:11.523+0000 7f7574e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7570070030 0x7f75700cd740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34820/0 (socket says 172.21.15.16:34820) 2024-03-31T15:05:11.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:11.523+0000 7f7574e00640 1 -- 172.21.15.16:0/1229265817 learned_addr learned my addr 172.21.15.16:0/1229265817 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:05:11.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:11.523+0000 7f7576200640 1 --2- 172.21.15.16:0/1229265817 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f75700cb440 0x7f75700cdc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:11.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:11.523+0000 7f7574e00640 1 -- 172.21.15.16:0/1229265817 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f75700cb440 msgr2=0x7f75700cdc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:11.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:11.523+0000 7f7574e00640 1 --2- 172.21.15.16:0/1229265817 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f75700cb440 0x7f75700cdc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:11.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:11.523+0000 7f7574e00640 1 -- 172.21.15.16:0/1229265817 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f757006dbc0 msgr2=0x7f75701406a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:11.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:11.523+0000 7f7574e00640 1 --2- 172.21.15.16:0/1229265817 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f757006dbc0 0x7f75701406a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:11.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:11.523+0000 7f7574e00640 1 -- 172.21.15.16:0/1229265817 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75701f8c70 con 0x7f7570070030 2024-03-31T15:05:11.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:11.523+0000 7f7575800640 1 --2- 172.21.15.16:0/1229265817 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f757006dbc0 0x7f75701406a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:05:11.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:11.523+0000 7f7574e00640 1 --2- 172.21.15.16:0/1229265817 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7570070030 0x7f75700cd740 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f755c00c410 tx=0x7f755c00c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:05:11.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:11.523+0000 7f756d600640 1 -- 172.21.15.16:0/1229265817 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f755c0074e0 con 0x7f7570070030 2024-03-31T15:05:11.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:11.523+0000 7f756d600640 1 -- 172.21.15.16:0/1229265817 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f755c0047f0 con 0x7f7570070030 2024-03-31T15:05:11.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:11.523+0000 7f756d600640 1 -- 172.21.15.16:0/1229265817 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f755c007680 con 0x7f7570070030 2024-03-31T15:05:11.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:11.523+0000 7f7578200640 1 -- 172.21.15.16:0/1229265817 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f75701f8f50 con 0x7f7570070030 2024-03-31T15:05:11.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:11.523+0000 7f7578200640 1 -- 172.21.15.16:0/1229265817 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f75700b2260 con 0x7f7570070030 2024-03-31T15:05:11.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:11.523+0000 7f7578200640 1 -- 172.21.15.16:0/1229265817 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7570069950 con 0x7f7570070030 2024-03-31T15:05:11.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:11.523+0000 7f756d600640 1 -- 172.21.15.16:0/1229265817 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f755c011070 con 0x7f7570070030 2024-03-31T15:05:11.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:11.527+0000 7f756d600640 1 --2- 172.21.15.16:0/1229265817 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f7550068380 0x7f755006a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:11.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:11.527+0000 7f756d600640 1 -- 172.21.15.16:0/1229265817 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5709+0+0 (secure 0 0 0) 0x7f755c086530 con 0x7f7570070030 2024-03-31T15:05:11.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:11.527+0000 7f7575800640 1 --2- 172.21.15.16:0/1229265817 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f7550068380 0x7f755006a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:11.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:11.527+0000 7f7575800640 1 --2- 172.21.15.16:0/1229265817 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f7550068380 0x7f755006a840 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f75640ad000 tx=0x7f7564008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:05:11.525 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:11.531+0000 7f756d600640 1 -- 172.21.15.16:0/1229265817 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f755c056400 con 0x7f7570070030 2024-03-31T15:05:11.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:11.695+0000 7f7578200640 1 -- 172.21.15.16:0/1229265817 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f75700b2430 con 0x7f7570070030 2024-03-31T15:05:11.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:11.699+0000 7f756d600640 1 -- 172.21.15.16:0/1229265817 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v34) ==== 74+0+16597 (secure 0 0 0) 0x7f755c05a700 con 0x7f7570070030 2024-03-31T15:05:11.694 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:05:11.694 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":34,"fsid":"286eb5da-fb47-4550-b290-d9519c13b9a5","created":"2024-03-31T15:03:11.438952+0000","modified":"2024-03-31T15:04:41.385382+0000","last_up_change":"2024-03-31T15:03:22.885535+0000","last_in_change":"2024-03-31T15:03:13.941667+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-31T15:03:22.550787+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-03-31T15:04:37.577341+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-03-31T15:04:38.738566+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"34","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"92a66450-ef19-40d8-ac3d-d760b23b0bda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6801","nonce":932712244}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6803","nonce":932712244}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6807","nonce":932712244}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6805","nonce":932712244}]},"public_addr":"172.21.15.16:6801/932712244","cluster_addr":"172.21.15.16:6803/932712244","heartbeat_back_addr":"172.21.15.16:6807/932712244","heartbeat_front_addr":"172.21.15.16:6805/932712244","state":["exists","up"]},{"osd":1,"uuid":"da9d1f2b-1968-4959-b843-faa231988e92","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6810","nonce":1398737533}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6811","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6813","nonce":1398737533}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6821","nonce":1398737533}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6815","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6816","nonce":1398737533}]},"public_addr":"172.21.15.16:6810/1398737533","cluster_addr":"172.21.15.16:6813/1398737533","heartbeat_back_addr":"172.21.15.16:6821/1398737533","heartbeat_front_addr":"172.21.15.16:6816/1398737533","state":["exists","up"]},{"osd":2,"uuid":"01a73662-9a21-4b1f-bcbb-42b9636a4638","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6819","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6822","nonce":3018526646}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6825","nonce":3018526646}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6831","nonce":3018526646}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6827","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6829","nonce":3018526646}]},"public_addr":"172.21.15.16:6822/3018526646","cluster_addr":"172.21.15.16:6825/3018526646","heartbeat_back_addr":"172.21.15.16:6831/3018526646","heartbeat_front_addr":"172.21.15.16:6829/3018526646","state":["exists","up"]},{"osd":3,"uuid":"c7771fd3-e18e-4500-ac38-23cf10ebcd75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6809","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6812","nonce":3360352974}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6817","nonce":3360352974}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6828","nonce":3360352974}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6823","nonce":3360352974}]},"public_addr":"172.21.15.16:6812/3360352974","cluster_addr":"172.21.15.16:6817/3360352974","heartbeat_back_addr":"172.21.15.16:6828/3360352974","heartbeat_front_addr":"172.21.15.16:6823/3360352974","state":["exists","up"]},{"osd":4,"uuid":"88260730-1482-4fbd-a40f-cb1124e9df4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6801","nonce":1856844856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6803","nonce":1856844856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6807","nonce":1856844856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6805","nonce":1856844856}]},"public_addr":"172.21.15.89:6801/1856844856","cluster_addr":"172.21.15.89:6803/1856844856","heartbeat_back_addr":"172.21.15.89:6807/1856844856","heartbeat_front_addr":"172.21.15.89:6805/1856844856","state":["exists","up"]},{"osd":5,"uuid":"4cb2c00c-fae0-4dd8-9d27-ec4f0093ccf7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6814","nonce":50395527}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6817","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6820","nonce":50395527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6831","nonce":50395527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6823","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6826","nonce":50395527}]},"public_addr":"172.21.15.89:6814/50395527","cluster_addr":"172.21.15.89:6820/50395527","heartbeat_back_addr":"172.21.15.89:6831/50395527","heartbeat_front_addr":"172.21.15.89:6826/50395527","state":["exists","up"]},{"osd":6,"uuid":"81ce0c19-2554-4cae-acbe-66a05e9e9b9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6809","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6812","nonce":3470471752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6815","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6818","nonce":3470471752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6827","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6829","nonce":3470471752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6821","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6824","nonce":3470471752}]},"public_addr":"172.21.15.89:6812/3470471752","cluster_addr":"172.21.15.89:6818/3470471752","heartbeat_back_addr":"172.21.15.89:6829/3470471752","heartbeat_front_addr":"172.21.15.89:6824/3470471752","state":["exists","up"]},{"osd":7,"uuid":"85e4313b-d37d-442e-8042-153bf20a847d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6811","nonce":1796778630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6813","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6816","nonce":1796778630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6825","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6828","nonce":1796778630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6819","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6822","nonce":1796778630}]},"public_addr":"172.21.15.89:6811/1796778630","cluster_addr":"172.21.15.89:6816/1796778630","heartbeat_back_addr":"172.21.15.89:6828/1796778630","heartbeat_front_addr":"172.21.15.89:6822/1796778630","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.439088+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.529970+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.680512+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:21.044161+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.582111+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.536558+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.409522+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.590229+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-31T15:05:11.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:11.699+0000 7f7578200640 1 -- 172.21.15.16:0/1229265817 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f7550068380 msgr2=0x7f755006a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:11.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:11.699+0000 7f7578200640 1 --2- 172.21.15.16:0/1229265817 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f7550068380 0x7f755006a840 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f75640ad000 tx=0x7f7564008040 comp rx=0 tx=0).stop 2024-03-31T15:05:11.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:11.699+0000 7f7578200640 1 -- 172.21.15.16:0/1229265817 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7570070030 msgr2=0x7f75700cd740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:11.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:11.699+0000 7f7578200640 1 --2- 172.21.15.16:0/1229265817 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7570070030 0x7f75700cd740 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f755c00c410 tx=0x7f755c00c8d0 comp rx=0 tx=0).stop 2024-03-31T15:05:11.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:11.699+0000 7f7578200640 1 -- 172.21.15.16:0/1229265817 shutdown_connections 2024-03-31T15:05:11.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:11.699+0000 7f7578200640 1 --2- 172.21.15.16:0/1229265817 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f7550068380 0x7f755006a840 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:11.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:11.699+0000 7f7578200640 1 --2- 172.21.15.16:0/1229265817 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f75700cb440 0x7f75700cdc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:11.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:11.699+0000 7f7578200640 1 --2- 172.21.15.16:0/1229265817 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7570070030 0x7f75700cd740 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:11.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:11.699+0000 7f7578200640 1 --2- 172.21.15.16:0/1229265817 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f757006dbc0 0x7f75701406a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:11.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:11.699+0000 7f7578200640 1 -- 172.21.15.16:0/1229265817 >> 172.21.15.16:0/1229265817 conn(0x7f7570067330 msgr2=0x7f75700d0890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:05:11.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:11.699+0000 7f7578200640 1 -- 172.21.15.16:0/1229265817 shutdown_connections 2024-03-31T15:05:11.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:11.699+0000 7f7578200640 1 -- 172.21.15.16:0/1229265817 wait complete. 2024-03-31T15:05:11.716 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-31T15:03:22.550787+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-31T15:04:37.577341+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '33', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_data', 'create_time': '2024-03-31T15:04:38.738566+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '34', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-31T15:05:11.716 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-31T15:05:11.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:11.867+0000 7fc91e000640 1 Processor -- start 2024-03-31T15:05:11.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:11.867+0000 7fc91e000640 1 -- start start 2024-03-31T15:05:11.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:11.867+0000 7fc91e000640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fc91806f920 0x7fc91806fd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:11.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:11.867+0000 7fc91e000640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fc9180d0a60 0x7fc9180cea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:11.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:11.867+0000 7fc91e000640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fc9180cef80 0x7fc918070e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:11.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:11.867+0000 7fc91e000640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7fc91813f670 con 0x7fc91806f920 2024-03-31T15:05:11.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:11.867+0000 7fc91e000640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7fc918070240 con 0x7fc9180d0a60 2024-03-31T15:05:11.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:11.867+0000 7fc91e000640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fc91815ca70 con 0x7fc9180cef80 2024-03-31T15:05:11.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:11.867+0000 7fc917400640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fc91806f920 0x7fc91806fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:11.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:11.867+0000 7fc917400640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fc91806f920 0x7fc91806fd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34830/0 (socket says 172.21.15.16:34830) 2024-03-31T15:05:11.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:11.867+0000 7fc917400640 1 -- 172.21.15.16:0/625487637 learned_addr learned my addr 172.21.15.16:0/625487637 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:05:11.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:11.867+0000 7fc917e00640 1 --2- 172.21.15.16:0/625487637 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fc9180cef80 0x7fc918070e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:11.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:11.867+0000 7fc90fe00640 1 --2- 172.21.15.16:0/625487637 >> v2:172.21.15.89:3300/0 conn(0x7fc9180d0a60 0x7fc9180cea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:11.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:11.867+0000 7fc917400640 1 -- 172.21.15.16:0/625487637 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fc9180cef80 msgr2=0x7fc918070e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:05:11.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:11.867+0000 7fc917400640 1 --2- 172.21.15.16:0/625487637 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fc9180cef80 0x7fc918070e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:11.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:11.867+0000 7fc917400640 1 -- 172.21.15.16:0/625487637 >> v2:172.21.15.89:3300/0 conn(0x7fc9180d0a60 msgr2=0x7fc9180cea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:05:11.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:11.867+0000 7fc917400640 1 --2- 172.21.15.16:0/625487637 >> v2:172.21.15.89:3300/0 conn(0x7fc9180d0a60 0x7fc9180cea40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:11.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:11.867+0000 7fc917400640 1 -- 172.21.15.16:0/625487637 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc91815cd50 con 0x7fc91806f920 2024-03-31T15:05:11.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:11.871+0000 7fc917400640 1 --2- 172.21.15.16:0/625487637 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc91806f920 0x7fc91806fd00 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7fc9180736b0 tx=0x7fc90801c260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7936095a3069c83d server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:05:11.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:11.871+0000 7fc916a00640 1 -- 172.21.15.16:0/625487637 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9080b0070 con 0x7fc91806f920 2024-03-31T15:05:11.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:11.871+0000 7fc916a00640 1 -- 172.21.15.16:0/625487637 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc9080ab600 con 0x7fc91806f920 2024-03-31T15:05:11.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:11.871+0000 7fc916a00640 1 -- 172.21.15.16:0/625487637 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9080ab9d0 con 0x7fc91806f920 2024-03-31T15:05:11.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:11.871+0000 7fc91e000640 1 -- 172.21.15.16:0/625487637 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc91806f920 msgr2=0x7fc91806fd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:11.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:11.871+0000 7fc91e000640 1 --2- 172.21.15.16:0/625487637 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc91806f920 0x7fc91806fd00 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7fc9180736b0 tx=0x7fc90801c260 comp rx=0 tx=0).stop 2024-03-31T15:05:11.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:11.871+0000 7fc91e000640 1 -- 172.21.15.16:0/625487637 shutdown_connections 2024-03-31T15:05:11.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:11.871+0000 7fc91e000640 1 --2- 172.21.15.16:0/625487637 >> v2:172.21.15.89:3300/0 conn(0x7fc9180d0a60 0x7fc9180cea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:11.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:11.871+0000 7fc91e000640 1 --2- 172.21.15.16:0/625487637 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fc9180cef80 0x7fc918070e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:11.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:11.871+0000 7fc91e000640 1 --2- 172.21.15.16:0/625487637 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc91806f920 0x7fc91806fd00 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:11.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:11.871+0000 7fc91e000640 1 -- 172.21.15.16:0/625487637 >> 172.21.15.16:0/625487637 conn(0x7fc918067930 msgr2=0x7fc9180b3ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:05:11.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:11.871+0000 7fc91e000640 1 -- 172.21.15.16:0/625487637 shutdown_connections 2024-03-31T15:05:11.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:11.871+0000 7fc91e000640 1 -- 172.21.15.16:0/625487637 wait complete. 2024-03-31T15:05:11.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:11.875+0000 7fc91e000640 1 Processor -- start 2024-03-31T15:05:11.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:11.875+0000 7fc91e000640 1 -- start start 2024-03-31T15:05:11.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:11.875+0000 7fc91e000640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fc91806f920 0x7fc91815dab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:11.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:11.875+0000 7fc91e000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc9180cef80 0x7fc918157da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:11.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:11.875+0000 7fc91e000640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc9180d0a60 0x7fc9181582e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:11.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:11.875+0000 7fc91e000640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fc9181419c0 con 0x7fc9180cef80 2024-03-31T15:05:11.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:11.875+0000 7fc91e000640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fc9180cf390 con 0x7fc9180d0a60 2024-03-31T15:05:11.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:11.875+0000 7fc91e000640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fc918141b40 con 0x7fc91806f920 2024-03-31T15:05:11.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:11.875+0000 7fc90fe00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc9180cef80 0x7fc918157da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:11.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:11.875+0000 7fc90fe00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc9180cef80 0x7fc918157da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34832/0 (socket says 172.21.15.16:34832) 2024-03-31T15:05:11.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:11.875+0000 7fc90fe00640 1 -- 172.21.15.16:0/3446819892 learned_addr learned my addr 172.21.15.16:0/3446819892 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:05:11.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:11.875+0000 7fc917400640 1 --2- 172.21.15.16:0/3446819892 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fc91806f920 0x7fc91815dab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:11.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:11.875+0000 7fc917e00640 1 --2- 172.21.15.16:0/3446819892 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc9180d0a60 0x7fc9181582e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:11.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:11.875+0000 7fc90fe00640 1 -- 172.21.15.16:0/3446819892 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fc91806f920 msgr2=0x7fc91815dab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:11.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:11.875+0000 7fc90fe00640 1 --2- 172.21.15.16:0/3446819892 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fc91806f920 0x7fc91815dab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:11.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:11.875+0000 7fc90fe00640 1 -- 172.21.15.16:0/3446819892 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc9180d0a60 msgr2=0x7fc9181582e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:11.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:11.875+0000 7fc90fe00640 1 --2- 172.21.15.16:0/3446819892 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc9180d0a60 0x7fc9181582e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:11.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:11.875+0000 7fc90fe00640 1 -- 172.21.15.16:0/3446819892 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9180b3bc0 con 0x7fc9180cef80 2024-03-31T15:05:11.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:11.875+0000 7fc90fe00640 1 --2- 172.21.15.16:0/3446819892 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc9180cef80 0x7fc918157da0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7fc90000e840 tx=0x7fc90000ec40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:05:11.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:11.875+0000 7fc90f400640 1 -- 172.21.15.16:0/3446819892 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc900017070 con 0x7fc9180cef80 2024-03-31T15:05:11.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:11.875+0000 7fc90f400640 1 -- 172.21.15.16:0/3446819892 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc90000ee90 con 0x7fc9180cef80 2024-03-31T15:05:11.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:11.875+0000 7fc90f400640 1 -- 172.21.15.16:0/3446819892 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc90000cb80 con 0x7fc9180cef80 2024-03-31T15:05:11.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:11.875+0000 7fc91e000640 1 -- 172.21.15.16:0/3446819892 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc918158910 con 0x7fc9180cef80 2024-03-31T15:05:11.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:11.875+0000 7fc91e000640 1 -- 172.21.15.16:0/3446819892 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc918158ed0 con 0x7fc9180cef80 2024-03-31T15:05:11.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:11.875+0000 7fc91e000640 1 -- 172.21.15.16:0/3446819892 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc918069950 con 0x7fc9180cef80 2024-03-31T15:05:11.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:11.879+0000 7fc90f400640 1 -- 172.21.15.16:0/3446819892 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fc90001a070 con 0x7fc9180cef80 2024-03-31T15:05:11.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:11.879+0000 7fc90f400640 1 --2- 172.21.15.16:0/3446819892 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fc8f4068380 0x7fc8f406a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:11.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:11.879+0000 7fc90f400640 1 -- 172.21.15.16:0/3446819892 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5709+0+0 (secure 0 0 0) 0x7fc900086780 con 0x7fc9180cef80 2024-03-31T15:05:11.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:11.879+0000 7fc917400640 1 --2- 172.21.15.16:0/3446819892 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fc8f4068380 0x7fc8f406a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:11.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:11.879+0000 7fc917400640 1 --2- 172.21.15.16:0/3446819892 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fc8f4068380 0x7fc8f406a840 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fc9080ad000 tx=0x7fc908008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:05:11.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:11.883+0000 7fc90f400640 1 -- 172.21.15.16:0/3446819892 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fc9000566e0 con 0x7fc9180cef80 2024-03-31T15:05:12.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:12.047+0000 7fc91e000640 1 -- 172.21.15.16:0/3446819892 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fc9180b4a20 con 0x7fc9180cef80 2024-03-31T15:05:12.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:12.051+0000 7fc90f400640 1 -- 172.21.15.16:0/3446819892 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v34) ==== 93+0+10 (secure 0 0 0) 0x7fc90005a9e0 con 0x7fc9180cef80 2024-03-31T15:05:12.045 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2024-03-31T15:05:12.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:12.051+0000 7fc91e000640 1 -- 172.21.15.16:0/3446819892 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fc8f4068380 msgr2=0x7fc8f406a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:12.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:12.051+0000 7fc91e000640 1 --2- 172.21.15.16:0/3446819892 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fc8f4068380 0x7fc8f406a840 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fc9080ad000 tx=0x7fc908008040 comp rx=0 tx=0).stop 2024-03-31T15:05:12.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:12.051+0000 7fc91e000640 1 -- 172.21.15.16:0/3446819892 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc9180cef80 msgr2=0x7fc918157da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:12.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:12.051+0000 7fc91e000640 1 --2- 172.21.15.16:0/3446819892 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc9180cef80 0x7fc918157da0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7fc90000e840 tx=0x7fc90000ec40 comp rx=0 tx=0).stop 2024-03-31T15:05:12.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:12.051+0000 7fc91e000640 1 -- 172.21.15.16:0/3446819892 shutdown_connections 2024-03-31T15:05:12.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:12.051+0000 7fc91e000640 1 --2- 172.21.15.16:0/3446819892 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fc8f4068380 0x7fc8f406a840 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:12.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:12.051+0000 7fc91e000640 1 --2- 172.21.15.16:0/3446819892 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc9180d0a60 0x7fc9181582e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:12.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:12.051+0000 7fc91e000640 1 --2- 172.21.15.16:0/3446819892 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc9180cef80 0x7fc918157da0 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:12.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:12.051+0000 7fc91e000640 1 --2- 172.21.15.16:0/3446819892 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fc91806f920 0x7fc91815dab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:12.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:12.051+0000 7fc91e000640 1 -- 172.21.15.16:0/3446819892 >> 172.21.15.16:0/3446819892 conn(0x7fc918067930 msgr2=0x7fc91813fa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:05:12.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:12.051+0000 7fc91e000640 1 -- 172.21.15.16:0/3446819892 shutdown_connections 2024-03-31T15:05:12.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:12.051+0000 7fc91e000640 1 -- 172.21.15.16:0/3446819892 wait complete. 2024-03-31T15:05:12.066 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-31T15:05:12.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:12.215+0000 7f382ba00640 1 Processor -- start 2024-03-31T15:05:12.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:12.215+0000 7f382ba00640 1 -- start start 2024-03-31T15:05:12.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:12.215+0000 7f382ba00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f38241567e0 0x7f3824072c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:12.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:12.215+0000 7f382ba00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f38240731c0 0x7f382406d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:12.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:12.215+0000 7f382ba00640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f382415c7e0 con 0x7f382406da10 2024-03-31T15:05:12.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:12.215+0000 7f382ba00640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f382406de20 con 0x7f38240731c0 2024-03-31T15:05:12.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:12.215+0000 7f382ba00640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f38240736b0 con 0x7f38241567e0 2024-03-31T15:05:12.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:12.215+0000 7f381be00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f38240731c0 0x7f382406d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:12.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:12.215+0000 7f381be00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f38240731c0 0x7f382406d4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34838/0 (socket says 172.21.15.16:34838) 2024-03-31T15:05:12.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:12.215+0000 7f381be00640 1 -- 172.21.15.16:0/2752337688 learned_addr learned my addr 172.21.15.16:0/2752337688 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:05:12.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:12.215+0000 7f381be00640 1 -- 172.21.15.16:0/2752337688 >> v2:172.21.15.89:3300/0 conn(0x7f38241567e0 msgr2=0x7f3824072c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:05:12.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:12.215+0000 7f3829000640 1 --2- 172.21.15.16:0/2752337688 >> v2:172.21.15.89:3300/0 conn(0x7f38241567e0 0x7f3824072c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:12.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:12.215+0000 7f381be00640 1 --2- 172.21.15.16:0/2752337688 >> v2:172.21.15.89:3300/0 conn(0x7f38241567e0 0x7f3824072c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:12.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:12.215+0000 7f381be00640 1 -- 172.21.15.16:0/2752337688 >> v1:172.21.15.89:6789/0 conn(0x7f382406da10 legacy=0x7f382415fdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:05:12.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:12.215+0000 7f381be00640 1 -- 172.21.15.16:0/2752337688 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f382415caa0 con 0x7f38240731c0 2024-03-31T15:05:12.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:12.215+0000 7f3829000640 1 --2- 172.21.15.16:0/2752337688 >> v2:172.21.15.89:3300/0 conn(0x7f38241567e0 0x7f3824072c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:05:12.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:12.219+0000 7f381be00640 1 --2- 172.21.15.16:0/2752337688 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f38240731c0 0x7f382406d4d0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f38240cffe0 tx=0x7f380c01c4e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f573ad09a4ffaf27 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:05:12.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:12.219+0000 7f3823e00640 1 -- 172.21.15.16:0/2752337688 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f380c0b0070 con 0x7f38240731c0 2024-03-31T15:05:12.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:12.219+0000 7f3823e00640 1 -- 172.21.15.16:0/2752337688 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f380c0ab600 con 0x7f38240731c0 2024-03-31T15:05:12.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:12.219+0000 7f3823e00640 1 -- 172.21.15.16:0/2752337688 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f380c0ab9d0 con 0x7f38240731c0 2024-03-31T15:05:12.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:12.219+0000 7f382ba00640 1 -- 172.21.15.16:0/2752337688 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f38240731c0 msgr2=0x7f382406d4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:12.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:12.219+0000 7f382ba00640 1 --2- 172.21.15.16:0/2752337688 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f38240731c0 0x7f382406d4d0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f38240cffe0 tx=0x7f380c01c4e0 comp rx=0 tx=0).stop 2024-03-31T15:05:12.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:12.219+0000 7f382ba00640 1 -- 172.21.15.16:0/2752337688 shutdown_connections 2024-03-31T15:05:12.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:12.219+0000 7f382ba00640 1 --2- 172.21.15.16:0/2752337688 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f38240731c0 0x7f382406d4d0 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:12.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:12.219+0000 7f382ba00640 1 --2- 172.21.15.16:0/2752337688 >> v2:172.21.15.89:3300/0 conn(0x7f38241567e0 0x7f3824072c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:12.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:12.219+0000 7f382ba00640 1 -- 172.21.15.16:0/2752337688 >> 172.21.15.16:0/2752337688 conn(0x7f382406ee40 msgr2=0x7f382406b620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:05:12.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:12.219+0000 7f382ba00640 1 -- 172.21.15.16:0/2752337688 shutdown_connections 2024-03-31T15:05:12.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:12.219+0000 7f382ba00640 1 -- 172.21.15.16:0/2752337688 wait complete. 2024-03-31T15:05:12.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:12.223+0000 7f382ba00640 1 Processor -- start 2024-03-31T15:05:12.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:12.223+0000 7f382ba00640 1 -- start start 2024-03-31T15:05:12.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:12.223+0000 7f382ba00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f382406da10 0x7f382414ddf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:12.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:12.223+0000 7f382ba00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f38240731c0 0x7f382414e330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:12.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:12.223+0000 7f382ba00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f38241567e0 0x7f3824145310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:12.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:12.223+0000 7f382ba00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f382415d6e0 con 0x7f382406da10 2024-03-31T15:05:12.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:12.223+0000 7f382ba00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f382415d560 con 0x7f38241567e0 2024-03-31T15:05:12.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:12.223+0000 7f382ba00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f3824169170 con 0x7f38240731c0 2024-03-31T15:05:12.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:12.223+0000 7f3829000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f382406da10 0x7f382414ddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:12.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:12.223+0000 7f3829000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f382406da10 0x7f382414ddf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34846/0 (socket says 172.21.15.16:34846) 2024-03-31T15:05:12.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:12.223+0000 7f3829000640 1 -- 172.21.15.16:0/566843079 learned_addr learned my addr 172.21.15.16:0/566843079 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:05:12.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:12.223+0000 7f381be00640 1 --2- 172.21.15.16:0/566843079 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f38240731c0 0x7f382414e330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:12.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:12.223+0000 7f3829a00640 1 --2- 172.21.15.16:0/566843079 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f38241567e0 0x7f3824145310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:12.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:12.223+0000 7f3829000640 1 -- 172.21.15.16:0/566843079 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f38240731c0 msgr2=0x7f382414e330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:12.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:12.223+0000 7f3829000640 1 --2- 172.21.15.16:0/566843079 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f38240731c0 0x7f382414e330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:12.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:12.223+0000 7f3829000640 1 -- 172.21.15.16:0/566843079 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f38241567e0 msgr2=0x7f3824145310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:12.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:12.223+0000 7f3829000640 1 --2- 172.21.15.16:0/566843079 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f38241567e0 0x7f3824145310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:12.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:12.223+0000 7f3829000640 1 -- 172.21.15.16:0/566843079 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38241544d0 con 0x7f382406da10 2024-03-31T15:05:12.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:12.223+0000 7f3829a00640 1 --2- 172.21.15.16:0/566843079 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f38241567e0 0x7f3824145310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:05:12.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:12.223+0000 7f3829000640 1 --2- 172.21.15.16:0/566843079 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f382406da10 0x7f382414ddf0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f381400c7f0 tx=0x7f381400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:05:12.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:12.223+0000 7f3821600640 1 -- 172.21.15.16:0/566843079 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3814015070 con 0x7f382406da10 2024-03-31T15:05:12.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:12.223+0000 7f3821600640 1 -- 172.21.15.16:0/566843079 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3814007820 con 0x7f382406da10 2024-03-31T15:05:12.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:12.223+0000 7f382ba00640 1 -- 172.21.15.16:0/566843079 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f38241547b0 con 0x7f382406da10 2024-03-31T15:05:12.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:12.223+0000 7f3821600640 1 -- 172.21.15.16:0/566843079 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3814007c10 con 0x7f382406da10 2024-03-31T15:05:12.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:12.223+0000 7f382ba00640 1 -- 172.21.15.16:0/566843079 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3824154d70 con 0x7f382406da10 2024-03-31T15:05:12.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:12.223+0000 7f382ba00640 1 -- 172.21.15.16:0/566843079 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3824069950 con 0x7f382406da10 2024-03-31T15:05:12.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:12.227+0000 7f3821600640 1 -- 172.21.15.16:0/566843079 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f3814018070 con 0x7f382406da10 2024-03-31T15:05:12.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:12.227+0000 7f3821600640 1 --2- 172.21.15.16:0/566843079 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f3800068380 0x7f380006a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:12.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:12.227+0000 7f3821600640 1 -- 172.21.15.16:0/566843079 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5709+0+0 (secure 0 0 0) 0x7f3814086a60 con 0x7f382406da10 2024-03-31T15:05:12.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:12.227+0000 7f381be00640 1 --2- 172.21.15.16:0/566843079 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f3800068380 0x7f380006a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:12.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:12.227+0000 7f381be00640 1 --2- 172.21.15.16:0/566843079 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f3800068380 0x7f380006a840 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f380c0ad000 tx=0x7f380c008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:05:12.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:12.231+0000 7f3821600640 1 -- 172.21.15.16:0/566843079 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f38140569d0 con 0x7f382406da10 2024-03-31T15:05:12.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:12.395+0000 7f382ba00640 1 -- 172.21.15.16:0/566843079 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f38240b17b0 con 0x7f382406da10 2024-03-31T15:05:12.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:12.395+0000 7f3821600640 1 -- 172.21.15.16:0/566843079 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v34) ==== 104+0+11 (secure 0 0 0) 0x7f381405acd0 con 0x7f382406da10 2024-03-31T15:05:12.392 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2024-03-31T15:05:12.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:12.399+0000 7f382ba00640 1 -- 172.21.15.16:0/566843079 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f3800068380 msgr2=0x7f380006a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:12.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:12.399+0000 7f382ba00640 1 --2- 172.21.15.16:0/566843079 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f3800068380 0x7f380006a840 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f380c0ad000 tx=0x7f380c008040 comp rx=0 tx=0).stop 2024-03-31T15:05:12.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:12.399+0000 7f382ba00640 1 -- 172.21.15.16:0/566843079 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f382406da10 msgr2=0x7f382414ddf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:12.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:12.399+0000 7f382ba00640 1 --2- 172.21.15.16:0/566843079 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f382406da10 0x7f382414ddf0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f381400c7f0 tx=0x7f381400ccb0 comp rx=0 tx=0).stop 2024-03-31T15:05:12.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:12.399+0000 7f382ba00640 1 -- 172.21.15.16:0/566843079 shutdown_connections 2024-03-31T15:05:12.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:12.399+0000 7f382ba00640 1 --2- 172.21.15.16:0/566843079 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f3800068380 0x7f380006a840 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:12.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:12.399+0000 7f382ba00640 1 --2- 172.21.15.16:0/566843079 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f38241567e0 0x7f3824145310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:12.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:12.399+0000 7f382ba00640 1 --2- 172.21.15.16:0/566843079 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f38240731c0 0x7f382414e330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:12.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:12.399+0000 7f382ba00640 1 --2- 172.21.15.16:0/566843079 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f382406da10 0x7f382414ddf0 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:12.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:12.399+0000 7f382ba00640 1 -- 172.21.15.16:0/566843079 >> 172.21.15.16:0/566843079 conn(0x7f382406ee40 msgr2=0x7f38240afea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:05:12.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:12.399+0000 7f382ba00640 1 -- 172.21.15.16:0/566843079 shutdown_connections 2024-03-31T15:05:12.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:12.399+0000 7f382ba00640 1 -- 172.21.15.16:0/566843079 wait complete. 2024-03-31T15:05:12.418 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-31T15:05:12.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:12.567+0000 7f82cf800640 1 Processor -- start 2024-03-31T15:05:12.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:12.567+0000 7f82cf800640 1 -- start start 2024-03-31T15:05:12.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:12.567+0000 7f82cf800640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f82c81567e0 0x7f82c8072c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:12.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:12.567+0000 7f82cf800640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f82c815c7e0 con 0x7f82c80731c0 2024-03-31T15:05:12.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:12.567+0000 7f82cf800640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f82c80736b0 con 0x7f82c806dbe0 2024-03-31T15:05:12.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:12.567+0000 7f82cf800640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f82c815caa0 con 0x7f82c81567e0 2024-03-31T15:05:12.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:12.571+0000 7f82bfe00640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f82c80731c0 0x7f82c806d4d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:36458/0 (socket says 172.21.15.16:36458) 2024-03-31T15:05:12.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:12.571+0000 7f82bfe00640 1 -- 172.21.15.16:0/2979099687 learned_addr learned my addr 172.21.15.16:0/2979099687 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:05:12.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:12.571+0000 7f82cce00640 1 --2- 172.21.15.16:0/2979099687 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f82c81567e0 0x7f82c8072c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:12.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:12.571+0000 7f82c7e00640 1 -- 172.21.15.16:0/2979099687 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3024014756 0 0) 0x7f82c815c7e0 con 0x7f82c80731c0 2024-03-31T15:05:12.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:12.571+0000 7f82c7e00640 1 -- 172.21.15.16:0/2979099687 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f82ac003530 con 0x7f82c80731c0 2024-03-31T15:05:12.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:12.571+0000 7f82c7e00640 1 -- 172.21.15.16:0/2979099687 <== mon.1 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 318626164 0 0) 0x7f82c80736b0 con 0x7f82c806dbe0 2024-03-31T15:05:12.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:12.571+0000 7f82c7e00640 1 -- 172.21.15.16:0/2979099687 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f82c815c7e0 con 0x7f82c806dbe0 2024-03-31T15:05:12.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:12.571+0000 7f82c7e00640 1 -- 172.21.15.16:0/2979099687 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2626094974 0 0) 0x7f82ac003530 con 0x7f82c80731c0 2024-03-31T15:05:12.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:12.571+0000 7f82c7e00640 1 -- 172.21.15.16:0/2979099687 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f82c81567e0 msgr2=0x7f82c8072c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:05:12.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:12.571+0000 7f82c7e00640 1 --2- 172.21.15.16:0/2979099687 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f82c81567e0 0x7f82c8072c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:12.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:12.571+0000 7f82c7e00640 1 -- 172.21.15.16:0/2979099687 >> v1:172.21.15.89:6789/0 conn(0x7f82c806dbe0 legacy=0x7f82c8071550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:12.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:12.571+0000 7f82c7e00640 1 -- 172.21.15.16:0/2979099687 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82c815cd80 con 0x7f82c80731c0 2024-03-31T15:05:12.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:12.571+0000 7f82c7e00640 1 -- 172.21.15.16:0/2979099687 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 214807623 0 0) 0x7f82b0001680 con 0x7f82c80731c0 2024-03-31T15:05:12.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:12.571+0000 7f82c7e00640 1 -- 172.21.15.16:0/2979099687 >> v1:172.21.15.16:6789/0 conn(0x7f82c80731c0 legacy=0x7f82c806d4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:12.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:12.571+0000 7f82c7e00640 1 --2- 172.21.15.16:0/2979099687 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f82c81567e0 0x7f82ac003b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:12.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:12.571+0000 7f82c7e00640 1 --2- 172.21.15.16:0/2979099687 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f82ac0040e0 0x7f82ac004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:12.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:12.571+0000 7f82c7e00640 1 --2- 172.21.15.16:0/2979099687 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f82ac004d50 0x7f82ac0051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:12.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:12.571+0000 7f82c7e00640 1 -- 172.21.15.16:0/2979099687 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f82c815caa0 con 0x7f82ac004d50 2024-03-31T15:05:12.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:12.571+0000 7f82c7e00640 1 -- 172.21.15.16:0/2979099687 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f82ac00bab0 con 0x7f82ac0040e0 2024-03-31T15:05:12.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:12.571+0000 7f82c7e00640 1 -- 172.21.15.16:0/2979099687 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f82ac00bcb0 con 0x7f82c81567e0 2024-03-31T15:05:12.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:12.571+0000 7f82cd800640 1 --2- 172.21.15.16:0/2979099687 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f82ac004d50 0x7f82ac0051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:12.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:12.571+0000 7f82cce00640 1 --2- 172.21.15.16:0/2979099687 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f82c81567e0 0x7f82ac003b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:12.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:12.571+0000 7f82bfe00640 1 --2- 172.21.15.16:0/2979099687 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f82ac0040e0 0x7f82ac004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:12.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:12.571+0000 7f82cd800640 1 -- 172.21.15.16:0/2979099687 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f82c81567e0 msgr2=0x7f82ac003b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:05:12.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:12.571+0000 7f82cd800640 1 --2- 172.21.15.16:0/2979099687 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f82c81567e0 0x7f82ac003b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:12.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:12.571+0000 7f82cd800640 1 -- 172.21.15.16:0/2979099687 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f82ac0040e0 msgr2=0x7f82ac004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:05:12.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:12.571+0000 7f82cd800640 1 --2- 172.21.15.16:0/2979099687 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f82ac0040e0 0x7f82ac004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:12.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:12.571+0000 7f82cd800640 1 -- 172.21.15.16:0/2979099687 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82ac003530 con 0x7f82ac004d50 2024-03-31T15:05:12.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:12.575+0000 7f82cd800640 1 --2- 172.21.15.16:0/2979099687 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f82ac004d50 0x7f82ac0051d0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f82c806e010 tx=0x7f82c001c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9774c5d408ff122e server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:05:12.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:12.575+0000 7f82c7e00640 1 -- 172.21.15.16:0/2979099687 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82c0001c20 con 0x7f82ac004d50 2024-03-31T15:05:12.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:12.575+0000 7f82c7e00640 1 -- 172.21.15.16:0/2979099687 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f82c0003ba0 con 0x7f82ac004d50 2024-03-31T15:05:12.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:12.575+0000 7f82c7e00640 1 -- 172.21.15.16:0/2979099687 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82c00050d0 con 0x7f82ac004d50 2024-03-31T15:05:12.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:12.575+0000 7f82cf800640 1 -- 172.21.15.16:0/2979099687 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f82ac004d50 msgr2=0x7f82ac0051d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:12.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:12.575+0000 7f82cf800640 1 --2- 172.21.15.16:0/2979099687 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f82ac004d50 0x7f82ac0051d0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f82c806e010 tx=0x7f82c001c8a0 comp rx=0 tx=0).stop 2024-03-31T15:05:12.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:12.575+0000 7f82cd800640 1 -- 172.21.15.16:0/2979099687 reap_dead start 2024-03-31T15:05:12.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:12.575+0000 7f82cf800640 1 -- 172.21.15.16:0/2979099687 shutdown_connections 2024-03-31T15:05:12.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:12.575+0000 7f82cf800640 1 -- 172.21.15.16:0/2979099687 >> 172.21.15.16:0/2979099687 conn(0x7f82c806ee40 msgr2=0x7f82c806b620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:05:12.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:12.575+0000 7f82cf800640 1 -- 172.21.15.16:0/2979099687 shutdown_connections 2024-03-31T15:05:12.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:12.575+0000 7f82cf800640 1 -- 172.21.15.16:0/2979099687 wait complete. 2024-03-31T15:05:12.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:12.575+0000 7f82cf800640 1 Processor -- start 2024-03-31T15:05:12.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:12.575+0000 7f82cf800640 1 -- start start 2024-03-31T15:05:12.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:12.579+0000 7f82cf800640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f82ac004d50 0x7f82c81f3130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:12.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:12.579+0000 7f82cf800640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f82c8150d00 0x7f82c814f630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:12.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:12.579+0000 7f82cf800640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f82c814fb70 0x7f82c8153c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:12.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:12.579+0000 7f82cf800640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f82c80afbb0 con 0x7f82ac004d50 2024-03-31T15:05:12.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:12.579+0000 7f82cf800640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f82c80afa30 con 0x7f82c8150d00 2024-03-31T15:05:12.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:12.579+0000 7f82cf800640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f82c815d580 con 0x7f82c814fb70 2024-03-31T15:05:12.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:12.579+0000 7f82cce00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f82ac004d50 0x7f82c81f3130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:12.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:12.579+0000 7f82cce00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f82ac004d50 0x7f82c81f3130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34862/0 (socket says 172.21.15.16:34862) 2024-03-31T15:05:12.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:12.579+0000 7f82cce00640 1 -- 172.21.15.16:0/1245415274 learned_addr learned my addr 172.21.15.16:0/1245415274 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:05:12.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:12.579+0000 7f82cd800640 1 --2- 172.21.15.16:0/1245415274 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f82c814fb70 0x7f82c8153c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:12.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:12.579+0000 7f82bfe00640 1 --2- 172.21.15.16:0/1245415274 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f82c8150d00 0x7f82c814f630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:12.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:12.579+0000 7f82cce00640 1 -- 172.21.15.16:0/1245415274 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f82c814fb70 msgr2=0x7f82c8153c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:12.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:12.579+0000 7f82cce00640 1 --2- 172.21.15.16:0/1245415274 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f82c814fb70 0x7f82c8153c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:12.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:12.579+0000 7f82cce00640 1 -- 172.21.15.16:0/1245415274 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f82c8150d00 msgr2=0x7f82c814f630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:12.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:12.579+0000 7f82cce00640 1 --2- 172.21.15.16:0/1245415274 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f82c8150d00 0x7f82c814f630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:12.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:12.579+0000 7f82cce00640 1 -- 172.21.15.16:0/1245415274 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82c8072490 con 0x7f82ac004d50 2024-03-31T15:05:12.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:12.579+0000 7f82cd800640 1 --2- 172.21.15.16:0/1245415274 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f82c814fb70 0x7f82c8153c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:05:12.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:12.579+0000 7f82bfe00640 1 --2- 172.21.15.16:0/1245415274 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f82c8150d00 0x7f82c814f630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:05:12.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:12.579+0000 7f82cce00640 1 --2- 172.21.15.16:0/1245415274 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f82ac004d50 0x7f82c81f3130 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f82b8011a00 tx=0x7f82b8011ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:05:12.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:12.579+0000 7f82c5600640 1 -- 172.21.15.16:0/1245415274 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82b800fa90 con 0x7f82ac004d50 2024-03-31T15:05:12.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:12.579+0000 7f82c5600640 1 -- 172.21.15.16:0/1245415274 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f82b800fc30 con 0x7f82ac004d50 2024-03-31T15:05:12.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:12.579+0000 7f82c5600640 1 -- 172.21.15.16:0/1245415274 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82b8014440 con 0x7f82ac004d50 2024-03-31T15:05:12.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:12.579+0000 7f82cf800640 1 -- 172.21.15.16:0/1245415274 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f82c81f2670 con 0x7f82ac004d50 2024-03-31T15:05:12.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:12.579+0000 7f82cf800640 1 -- 172.21.15.16:0/1245415274 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f82c81503d0 con 0x7f82ac004d50 2024-03-31T15:05:12.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:12.579+0000 7f82cf800640 1 -- 172.21.15.16:0/1245415274 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f82c8069950 con 0x7f82ac004d50 2024-03-31T15:05:12.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:12.579+0000 7f82c5600640 1 -- 172.21.15.16:0/1245415274 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f82b8012040 con 0x7f82ac004d50 2024-03-31T15:05:12.577 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:12.583+0000 7f82c5600640 1 --2- 172.21.15.16:0/1245415274 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f82a4068380 0x7f82a406a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:12.577 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:12.583+0000 7f82c5600640 1 -- 172.21.15.16:0/1245415274 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5709+0+0 (secure 0 0 0) 0x7f82b80870a0 con 0x7f82ac004d50 2024-03-31T15:05:12.577 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:12.583+0000 7f82bfe00640 1 --2- 172.21.15.16:0/1245415274 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f82a4068380 0x7f82a406a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:12.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:12.583+0000 7f82bfe00640 1 --2- 172.21.15.16:0/1245415274 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f82a4068380 0x7f82a406a840 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f82b0001060 tx=0x7f82b0009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:05:12.580 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:12.587+0000 7f82c5600640 1 -- 172.21.15.16:0/1245415274 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f82b8019030 con 0x7f82ac004d50 2024-03-31T15:05:12.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:12.751+0000 7f82cf800640 1 -- 172.21.15.16:0/1245415274 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f82c80716f0 con 0x7f82ac004d50 2024-03-31T15:05:12.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:12.751+0000 7f82c5600640 1 -- 172.21.15.16:0/1245415274 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v34) ==== 100+0+11 (secure 0 0 0) 0x7f82b8057680 con 0x7f82ac004d50 2024-03-31T15:05:12.749 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 64 2024-03-31T15:05:12.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:12.755+0000 7f82cf800640 1 -- 172.21.15.16:0/1245415274 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f82a4068380 msgr2=0x7f82a406a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:12.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:12.755+0000 7f82cf800640 1 --2- 172.21.15.16:0/1245415274 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f82a4068380 0x7f82a406a840 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f82b0001060 tx=0x7f82b0009040 comp rx=0 tx=0).stop 2024-03-31T15:05:12.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:12.755+0000 7f82cf800640 1 -- 172.21.15.16:0/1245415274 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f82ac004d50 msgr2=0x7f82c81f3130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:12.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:12.755+0000 7f82cf800640 1 --2- 172.21.15.16:0/1245415274 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f82ac004d50 0x7f82c81f3130 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f82b8011a00 tx=0x7f82b8011ec0 comp rx=0 tx=0).stop 2024-03-31T15:05:12.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:12.755+0000 7f82cf800640 1 -- 172.21.15.16:0/1245415274 shutdown_connections 2024-03-31T15:05:12.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:12.755+0000 7f82cf800640 1 --2- 172.21.15.16:0/1245415274 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f82a4068380 0x7f82a406a840 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:12.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:12.755+0000 7f82cf800640 1 --2- 172.21.15.16:0/1245415274 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f82c814fb70 0x7f82c8153c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:12.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:12.755+0000 7f82cf800640 1 --2- 172.21.15.16:0/1245415274 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f82c8150d00 0x7f82c814f630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:12.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:12.755+0000 7f82cf800640 1 --2- 172.21.15.16:0/1245415274 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f82ac004d50 0x7f82c81f3130 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:12.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:12.755+0000 7f82cf800640 1 -- 172.21.15.16:0/1245415274 >> 172.21.15.16:0/1245415274 conn(0x7f82c806ee40 msgr2=0x7f82c815f230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:05:12.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:12.755+0000 7f82cf800640 1 -- 172.21.15.16:0/1245415274 shutdown_connections 2024-03-31T15:05:12.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:12.755+0000 7f82cf800640 1 -- 172.21.15.16:0/1245415274 wait complete. 2024-03-31T15:05:12.769 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-31T15:05:12.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:12.915+0000 7fdeb9c00640 1 Processor -- start 2024-03-31T15:05:12.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:12.915+0000 7fdeb9c00640 1 -- start start 2024-03-31T15:05:12.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:12.915+0000 7fdeb9c00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fdeb40b9c60 0x7fdeb40b7470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:12.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:12.915+0000 7fdeb9c00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fdeb40b79b0 0x7fdeb40b7e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:12.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:12.915+0000 7fdeb9c00640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdeb40d3990 con 0x7fdeb40d1280 2024-03-31T15:05:12.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:12.915+0000 7fdeb9c00640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7fdeb40d26a0 con 0x7fdeb40b79b0 2024-03-31T15:05:12.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:12.915+0000 7fdeb9c00640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7fdeb40d28f0 con 0x7fdeb40b9c60 2024-03-31T15:05:12.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:12.915+0000 7fdeb3400640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fdeb40b79b0 0x7fdeb40b7e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:12.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:12.915+0000 7fdeb3400640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fdeb40b79b0 0x7fdeb40b7e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34872/0 (socket says 172.21.15.16:34872) 2024-03-31T15:05:12.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:12.915+0000 7fdeb3400640 1 -- 172.21.15.16:0/867238591 learned_addr learned my addr 172.21.15.16:0/867238591 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:05:12.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:12.915+0000 7fdeabe00640 1 --2- 172.21.15.16:0/867238591 >> v2:172.21.15.89:3300/0 conn(0x7fdeb40b9c60 0x7fdeb40b7470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:12.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:12.915+0000 7fdeb3400640 1 -- 172.21.15.16:0/867238591 >> v2:172.21.15.89:3300/0 conn(0x7fdeb40b9c60 msgr2=0x7fdeb40b7470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:05:12.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:12.915+0000 7fdeb3400640 1 --2- 172.21.15.16:0/867238591 >> v2:172.21.15.89:3300/0 conn(0x7fdeb40b9c60 0x7fdeb40b7470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:12.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:12.915+0000 7fdeb3400640 1 -- 172.21.15.16:0/867238591 >> v1:172.21.15.89:6789/0 conn(0x7fdeb40d1280 legacy=0x7fdeb40d1660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:05:12.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:12.915+0000 7fdeb3400640 1 -- 172.21.15.16:0/867238591 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdeb40d2bd0 con 0x7fdeb40b79b0 2024-03-31T15:05:12.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:12.919+0000 7fdeb3400640 1 --2- 172.21.15.16:0/867238591 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdeb40b79b0 0x7fdeb40b7e60 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fdeb40b6df0 tx=0x7fdeac01c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=185e08cd29b00e8d server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:05:12.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:12.919+0000 7fdeb2000640 1 -- 172.21.15.16:0/867238591 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdeac0a5710 con 0x7fdeb40b79b0 2024-03-31T15:05:12.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:12.919+0000 7fdeb2000640 1 -- 172.21.15.16:0/867238591 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdeac0a58b0 con 0x7fdeb40b79b0 2024-03-31T15:05:12.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:12.919+0000 7fdeb2000640 1 -- 172.21.15.16:0/867238591 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdeac0a5c80 con 0x7fdeb40b79b0 2024-03-31T15:05:12.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:12.919+0000 7fdeb9c00640 1 -- 172.21.15.16:0/867238591 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdeb40b79b0 msgr2=0x7fdeb40b7e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:12.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:12.919+0000 7fdeb9c00640 1 --2- 172.21.15.16:0/867238591 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdeb40b79b0 0x7fdeb40b7e60 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fdeb40b6df0 tx=0x7fdeac01c960 comp rx=0 tx=0).stop 2024-03-31T15:05:12.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:12.919+0000 7fdeb9c00640 1 -- 172.21.15.16:0/867238591 shutdown_connections 2024-03-31T15:05:12.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:12.919+0000 7fdeb9c00640 1 --2- 172.21.15.16:0/867238591 >> v2:172.21.15.89:3300/0 conn(0x7fdeb40b9c60 0x7fdeb40b7470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:12.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:12.919+0000 7fdeb9c00640 1 --2- 172.21.15.16:0/867238591 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdeb40b79b0 0x7fdeb40b7e60 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:12.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:12.919+0000 7fdeb9c00640 1 -- 172.21.15.16:0/867238591 >> 172.21.15.16:0/867238591 conn(0x7fdeb4067730 msgr2=0x7fdeb4067b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:05:12.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:12.919+0000 7fdeb9c00640 1 -- 172.21.15.16:0/867238591 shutdown_connections 2024-03-31T15:05:12.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:12.919+0000 7fdeb9c00640 1 -- 172.21.15.16:0/867238591 wait complete. 2024-03-31T15:05:12.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:12.923+0000 7fdeb9c00640 1 Processor -- start 2024-03-31T15:05:12.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:12.923+0000 7fdeb9c00640 1 -- start start 2024-03-31T15:05:12.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:12.923+0000 7fdeb9c00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fdeb40b79b0 0x7fdeb4155da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:12.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:12.923+0000 7fdeb9c00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdeb40b9c60 0x7fdeb4160560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:12.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:12.923+0000 7fdeb9c00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdeb40d1280 0x7fdeb415c8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:12.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:12.923+0000 7fdeb9c00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fdeb40d4010 con 0x7fdeb40b9c60 2024-03-31T15:05:12.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:12.923+0000 7fdeb9c00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fdeb40d3e90 con 0x7fdeb40d1280 2024-03-31T15:05:12.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:12.923+0000 7fdeb9c00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fdeb416b8a0 con 0x7fdeb40b79b0 2024-03-31T15:05:12.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:12.923+0000 7fdeabe00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdeb40b9c60 0x7fdeb4160560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:12.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:12.923+0000 7fdeabe00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdeb40b9c60 0x7fdeb4160560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34884/0 (socket says 172.21.15.16:34884) 2024-03-31T15:05:12.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:12.923+0000 7fdeb2a00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fdeb40b79b0 0x7fdeb4155da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:12.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:12.923+0000 7fdeabe00640 1 -- 172.21.15.16:0/894359134 learned_addr learned my addr 172.21.15.16:0/894359134 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:05:12.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:12.923+0000 7fdeb3400640 1 --2- 172.21.15.16:0/894359134 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdeb40d1280 0x7fdeb415c8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:12.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:12.923+0000 7fdeabe00640 1 -- 172.21.15.16:0/894359134 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fdeb40b79b0 msgr2=0x7fdeb4155da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:12.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:12.923+0000 7fdeabe00640 1 --2- 172.21.15.16:0/894359134 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fdeb40b79b0 0x7fdeb4155da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:12.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:12.923+0000 7fdeabe00640 1 -- 172.21.15.16:0/894359134 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdeb40d1280 msgr2=0x7fdeb415c8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:12.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:12.923+0000 7fdeabe00640 1 --2- 172.21.15.16:0/894359134 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdeb40d1280 0x7fdeb415c8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:12.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:12.923+0000 7fdeabe00640 1 -- 172.21.15.16:0/894359134 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdeb41f79b0 con 0x7fdeb40b9c60 2024-03-31T15:05:12.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:12.923+0000 7fdeb2a00640 1 --2- 172.21.15.16:0/894359134 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fdeb40b79b0 0x7fdeb4155da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:05:12.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:12.923+0000 7fdeabe00640 1 --2- 172.21.15.16:0/894359134 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdeb40b9c60 0x7fdeb4160560 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7fde9c00d7d0 tx=0x7fde9c00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:05:12.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:12.923+0000 7fdeaaa00640 1 -- 172.21.15.16:0/894359134 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde9c004020 con 0x7fdeb40b9c60 2024-03-31T15:05:12.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:12.923+0000 7fdeaaa00640 1 -- 172.21.15.16:0/894359134 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fde9c0041c0 con 0x7fdeb40b9c60 2024-03-31T15:05:12.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:12.923+0000 7fdeb9c00640 1 -- 172.21.15.16:0/894359134 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdeb41f7c90 con 0x7fdeb40b9c60 2024-03-31T15:05:12.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:12.923+0000 7fdeaaa00640 1 -- 172.21.15.16:0/894359134 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde9c0043b0 con 0x7fdeb40b9c60 2024-03-31T15:05:12.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:12.923+0000 7fdeb9c00640 1 -- 172.21.15.16:0/894359134 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdeb41f8250 con 0x7fdeb40b9c60 2024-03-31T15:05:12.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:12.923+0000 7fdeb9c00640 1 -- 172.21.15.16:0/894359134 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdeb4069950 con 0x7fdeb40b9c60 2024-03-31T15:05:12.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:12.927+0000 7fdeaaa00640 1 -- 172.21.15.16:0/894359134 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fde9c01b020 con 0x7fdeb40b9c60 2024-03-31T15:05:12.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:12.927+0000 7fdeaaa00640 1 --2- 172.21.15.16:0/894359134 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fde90068380 0x7fde9006a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:12.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:12.927+0000 7fdeaaa00640 1 -- 172.21.15.16:0/894359134 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5709+0+0 (secure 0 0 0) 0x7fde9c086600 con 0x7fdeb40b9c60 2024-03-31T15:05:12.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:12.927+0000 7fdeb2a00640 1 --2- 172.21.15.16:0/894359134 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fde90068380 0x7fde9006a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:12.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:12.927+0000 7fdeb2a00640 1 --2- 172.21.15.16:0/894359134 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fde90068380 0x7fde9006a840 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fdea4001b60 tx=0x7fdea4009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:05:12.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:12.931+0000 7fdeaaa00640 1 -- 172.21.15.16:0/894359134 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fde9c056600 con 0x7fdeb40b9c60 2024-03-31T15:05:13.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:13.135+0000 7fdeb9c00640 1 -- 172.21.15.16:0/894359134 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fdeb40b9250 con 0x7fdeb40b9c60 2024-03-31T15:05:13.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:13.135+0000 7fdeaaa00640 1 -- 172.21.15.16:0/894359134 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) ==== 94+0+5178 (secure 0 0 0) 0x7fde9c05a900 con 0x7fdeb40b9c60 2024-03-31T15:05:13.131 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:05:13.131 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":16,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4949,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6836/2073954753","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":2073954753},{"type":"v1","addr":"172.21.15.89:6836","nonce":2073954753}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":4952,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/2143918278","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6835","nonce":2143918278},{"type":"v1","addr":"172.21.15.89:6837","nonce":2143918278}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":5251,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6833/881391342","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":881391342},{"type":"v1","addr":"172.21.15.16:6833","nonce":881391342}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-31T15:04:39.780323+0000","modified":"2024-03-31T15:04:42.846439+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5254},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5254":{"gid":5254,"name":"c","rank":0,"incarnation":13,"state":"up:active","state_seq":3,"addr":"172.21.15.16:6835/1845707550","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":1845707550},{"type":"v1","addr":"172.21.15.16:6835","nonce":1845707550}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5254,"qdb_cluster":[5254]},"id":2}]} 2024-03-31T15:05:13.131 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 16 2024-03-31T15:05:13.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:13.139+0000 7fdeb9c00640 1 -- 172.21.15.16:0/894359134 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fde90068380 msgr2=0x7fde9006a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:13.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:13.139+0000 7fdeb9c00640 1 --2- 172.21.15.16:0/894359134 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fde90068380 0x7fde9006a840 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fdea4001b60 tx=0x7fdea4009040 comp rx=0 tx=0).stop 2024-03-31T15:05:13.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:13.139+0000 7fdeb9c00640 1 -- 172.21.15.16:0/894359134 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdeb40b9c60 msgr2=0x7fdeb4160560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:13.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:13.139+0000 7fdeb9c00640 1 --2- 172.21.15.16:0/894359134 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdeb40b9c60 0x7fdeb4160560 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7fde9c00d7d0 tx=0x7fde9c00dc90 comp rx=0 tx=0).stop 2024-03-31T15:05:13.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:13.139+0000 7fdeb9c00640 1 -- 172.21.15.16:0/894359134 shutdown_connections 2024-03-31T15:05:13.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:13.139+0000 7fdeb9c00640 1 --2- 172.21.15.16:0/894359134 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fde90068380 0x7fde9006a840 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:13.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:13.139+0000 7fdeb9c00640 1 --2- 172.21.15.16:0/894359134 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdeb40d1280 0x7fdeb415c8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:13.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:13.139+0000 7fdeb9c00640 1 --2- 172.21.15.16:0/894359134 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdeb40b9c60 0x7fdeb4160560 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:13.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:13.139+0000 7fdeb9c00640 1 --2- 172.21.15.16:0/894359134 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fdeb40b79b0 0x7fdeb4155da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:13.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:13.139+0000 7fdeb9c00640 1 -- 172.21.15.16:0/894359134 >> 172.21.15.16:0/894359134 conn(0x7fdeb4067730 msgr2=0x7fdeb40d4b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:05:13.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:13.139+0000 7fdeb9c00640 1 -- 172.21.15.16:0/894359134 shutdown_connections 2024-03-31T15:05:13.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:13.139+0000 7fdeb9c00640 1 -- 172.21.15.16:0/894359134 wait complete. 2024-03-31T15:05:13.152 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-31T15:05:13.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:13.299+0000 7fdee9c00640 1 Processor -- start 2024-03-31T15:05:13.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:13.299+0000 7fdee9c00640 1 -- start start 2024-03-31T15:05:13.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:13.303+0000 7fdee9c00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fdee40ade10 0x7fdee40b0480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:13.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:13.303+0000 7fdee9c00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fdee406bd30 0x7fdee406e6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:13.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:13.303+0000 7fdee9c00640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdee415c4f0 con 0x7fdee4145ab0 2024-03-31T15:05:13.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:13.303+0000 7fdee9c00640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7fdee40ae2a0 con 0x7fdee406bd30 2024-03-31T15:05:13.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:13.303+0000 7fdee9c00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fdee415c7c0 con 0x7fdee40ade10 2024-03-31T15:05:13.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:13.303+0000 7fdedbe00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fdee40ade10 0x7fdee40b0480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:13.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:13.303+0000 7fdee9200640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fdee406bd30 0x7fdee406e6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:13.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:13.303+0000 7fdedbe00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fdee40ade10 0x7fdee40b0480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3301/0 says I am v2:172.21.15.16:37730/0 (socket says 172.21.15.16:37730) 2024-03-31T15:05:13.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:13.303+0000 7fdee9200640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fdee406bd30 0x7fdee406e6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.16:56992/0 (socket says 172.21.15.16:56992) 2024-03-31T15:05:13.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:13.303+0000 7fdedbe00640 1 -- 172.21.15.16:0/2992519707 learned_addr learned my addr 172.21.15.16:0/2992519707 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:05:13.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:13.303+0000 7fdee3e00640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fdee4145ab0 0x7fdee406b620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.16:46962/0 (socket says 172.21.15.16:46962) 2024-03-31T15:05:13.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:13.303+0000 7fdee9200640 1 -- 172.21.15.16:0/2992519707 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fdee40ade10 msgr2=0x7fdee40b0480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:05:13.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:13.303+0000 7fdee9200640 1 --2- 172.21.15.16:0/2992519707 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fdee40ade10 0x7fdee40b0480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:13.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:13.303+0000 7fdee9200640 1 -- 172.21.15.16:0/2992519707 >> v1:172.21.15.89:6789/0 conn(0x7fdee4145ab0 legacy=0x7fdee406b620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:13.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:13.303+0000 7fdee9200640 1 -- 172.21.15.16:0/2992519707 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdee415caa0 con 0x7fdee406bd30 2024-03-31T15:05:13.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:13.303+0000 7fdedbe00640 1 --2- 172.21.15.16:0/2992519707 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fdee40ade10 0x7fdee40b0480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:05:13.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:13.307+0000 7fdee9200640 1 --2- 172.21.15.16:0/2992519707 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdee406bd30 0x7fdee406e6d0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fdedc009d90 tx=0x7fdedc01c800 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4087e34bb0989f1d server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:05:13.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:13.307+0000 7fdee3400640 1 -- 172.21.15.16:0/2992519707 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdedc0ac070 con 0x7fdee406bd30 2024-03-31T15:05:13.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:13.307+0000 7fdee3400640 1 -- 172.21.15.16:0/2992519707 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdedc0a6610 con 0x7fdee406bd30 2024-03-31T15:05:13.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:13.307+0000 7fdee3400640 1 -- 172.21.15.16:0/2992519707 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdedc0a69e0 con 0x7fdee406bd30 2024-03-31T15:05:13.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:13.307+0000 7fdee9c00640 1 -- 172.21.15.16:0/2992519707 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdee406bd30 msgr2=0x7fdee406e6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:13.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:13.307+0000 7fdee9c00640 1 --2- 172.21.15.16:0/2992519707 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdee406bd30 0x7fdee406e6d0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fdedc009d90 tx=0x7fdedc01c800 comp rx=0 tx=0).stop 2024-03-31T15:05:13.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:13.307+0000 7fdee9c00640 1 -- 172.21.15.16:0/2992519707 shutdown_connections 2024-03-31T15:05:13.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:13.307+0000 7fdee9c00640 1 --2- 172.21.15.16:0/2992519707 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdee406bd30 0x7fdee406e6d0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:13.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:13.307+0000 7fdee9c00640 1 --2- 172.21.15.16:0/2992519707 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fdee40ade10 0x7fdee40b0480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:13.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:13.307+0000 7fdee9c00640 1 -- 172.21.15.16:0/2992519707 >> 172.21.15.16:0/2992519707 conn(0x7fdee40b2b30 msgr2=0x7fdee40b2f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:05:13.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:13.307+0000 7fdee9c00640 1 -- 172.21.15.16:0/2992519707 shutdown_connections 2024-03-31T15:05:13.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:13.307+0000 7fdee9c00640 1 -- 172.21.15.16:0/2992519707 wait complete. 2024-03-31T15:05:13.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:13.307+0000 7fdee9c00640 1 Processor -- start 2024-03-31T15:05:13.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:13.307+0000 7fdee9c00640 1 -- start start 2024-03-31T15:05:13.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:13.311+0000 7fdee9c00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fdee406bd30 0x7fdee41418a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:13.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:13.311+0000 7fdee9c00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdee4145ab0 0x7fdee413a130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:13.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:13.311+0000 7fdee9c00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdee413a670 0x7fdee413d1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:13.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:13.311+0000 7fdee9c00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fdee4168f10 con 0x7fdee4145ab0 2024-03-31T15:05:13.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:13.311+0000 7fdee9c00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fdee4168d90 con 0x7fdee413a670 2024-03-31T15:05:13.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:13.311+0000 7fdee9c00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fdee4169090 con 0x7fdee406bd30 2024-03-31T15:05:13.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:13.311+0000 7fdedbe00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdee4145ab0 0x7fdee413a130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:13.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:13.311+0000 7fdedbe00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdee4145ab0 0x7fdee413a130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34892/0 (socket says 172.21.15.16:34892) 2024-03-31T15:05:13.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:13.311+0000 7fdedbe00640 1 -- 172.21.15.16:0/2348239385 learned_addr learned my addr 172.21.15.16:0/2348239385 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:05:13.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:13.311+0000 7fdee9200640 1 --2- 172.21.15.16:0/2348239385 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdee413a670 0x7fdee413d1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:13.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:13.311+0000 7fdee3e00640 1 --2- 172.21.15.16:0/2348239385 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fdee406bd30 0x7fdee41418a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:13.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:13.311+0000 7fdedbe00640 1 -- 172.21.15.16:0/2348239385 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fdee406bd30 msgr2=0x7fdee41418a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:13.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:13.311+0000 7fdedbe00640 1 --2- 172.21.15.16:0/2348239385 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fdee406bd30 0x7fdee41418a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:13.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:13.311+0000 7fdedbe00640 1 -- 172.21.15.16:0/2348239385 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdee413a670 msgr2=0x7fdee413d1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:13.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:13.311+0000 7fdedbe00640 1 --2- 172.21.15.16:0/2348239385 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdee413a670 0x7fdee413d1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:13.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:13.311+0000 7fdedbe00640 1 -- 172.21.15.16:0/2348239385 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdee415c0d0 con 0x7fdee4145ab0 2024-03-31T15:05:13.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:13.311+0000 7fdedbe00640 1 --2- 172.21.15.16:0/2348239385 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdee4145ab0 0x7fdee413a130 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7fdecc00ea00 tx=0x7fdecc00eec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:05:13.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:13.311+0000 7fdee0c00640 1 -- 172.21.15.16:0/2348239385 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdecc017070 con 0x7fdee4145ab0 2024-03-31T15:05:13.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:13.311+0000 7fdee0c00640 1 -- 172.21.15.16:0/2348239385 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdecc00ca60 con 0x7fdee4145ab0 2024-03-31T15:05:13.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:13.311+0000 7fdee0c00640 1 -- 172.21.15.16:0/2348239385 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdecc00ce10 con 0x7fdee4145ab0 2024-03-31T15:05:13.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:13.311+0000 7fdee9c00640 1 -- 172.21.15.16:0/2348239385 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdee413f670 con 0x7fdee4145ab0 2024-03-31T15:05:13.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:13.311+0000 7fdee9c00640 1 -- 172.21.15.16:0/2348239385 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdee413fbf0 con 0x7fdee4145ab0 2024-03-31T15:05:13.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:13.311+0000 7fdee9c00640 1 -- 172.21.15.16:0/2348239385 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdee404f8c0 con 0x7fdee4145ab0 2024-03-31T15:05:13.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:13.315+0000 7fdee0c00640 1 -- 172.21.15.16:0/2348239385 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fdecc01a070 con 0x7fdee4145ab0 2024-03-31T15:05:13.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:13.315+0000 7fdee0c00640 1 --2- 172.21.15.16:0/2348239385 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fdec0068380 0x7fdec006a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:13.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:13.315+0000 7fdee0c00640 1 -- 172.21.15.16:0/2348239385 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5709+0+0 (secure 0 0 0) 0x7fdecc0868d0 con 0x7fdee4145ab0 2024-03-31T15:05:13.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:13.315+0000 7fdee3e00640 1 --2- 172.21.15.16:0/2348239385 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fdec0068380 0x7fdec006a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:13.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:13.315+0000 7fdee3e00640 1 --2- 172.21.15.16:0/2348239385 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fdec0068380 0x7fdec006a840 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fded4000f70 tx=0x7fded4007e30 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:05:13.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:13.323+0000 7fdee0c00640 1 -- 172.21.15.16:0/2348239385 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fdecc056830 con 0x7fdee4145ab0 2024-03-31T15:05:13.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:13.491+0000 7fdee9c00640 1 -- 172.21.15.16:0/2348239385 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fdee40d04b0 con 0x7fdee4145ab0 2024-03-31T15:05:13.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:13.491+0000 7fdee0c00640 1 -- 172.21.15.16:0/2348239385 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v34) ==== 74+0+16597 (secure 0 0 0) 0x7fdecc05ab30 con 0x7fdee4145ab0 2024-03-31T15:05:13.486 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:05:13.486 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":34,"fsid":"286eb5da-fb47-4550-b290-d9519c13b9a5","created":"2024-03-31T15:03:11.438952+0000","modified":"2024-03-31T15:04:41.385382+0000","last_up_change":"2024-03-31T15:03:22.885535+0000","last_in_change":"2024-03-31T15:03:13.941667+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-31T15:03:22.550787+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-03-31T15:04:37.577341+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-03-31T15:04:38.738566+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"34","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"92a66450-ef19-40d8-ac3d-d760b23b0bda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6801","nonce":932712244}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6803","nonce":932712244}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6807","nonce":932712244}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6805","nonce":932712244}]},"public_addr":"172.21.15.16:6801/932712244","cluster_addr":"172.21.15.16:6803/932712244","heartbeat_back_addr":"172.21.15.16:6807/932712244","heartbeat_front_addr":"172.21.15.16:6805/932712244","state":["exists","up"]},{"osd":1,"uuid":"da9d1f2b-1968-4959-b843-faa231988e92","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6810","nonce":1398737533}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6811","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6813","nonce":1398737533}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6821","nonce":1398737533}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6815","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6816","nonce":1398737533}]},"public_addr":"172.21.15.16:6810/1398737533","cluster_addr":"172.21.15.16:6813/1398737533","heartbeat_back_addr":"172.21.15.16:6821/1398737533","heartbeat_front_addr":"172.21.15.16:6816/1398737533","state":["exists","up"]},{"osd":2,"uuid":"01a73662-9a21-4b1f-bcbb-42b9636a4638","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6819","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6822","nonce":3018526646}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6825","nonce":3018526646}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6831","nonce":3018526646}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6827","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6829","nonce":3018526646}]},"public_addr":"172.21.15.16:6822/3018526646","cluster_addr":"172.21.15.16:6825/3018526646","heartbeat_back_addr":"172.21.15.16:6831/3018526646","heartbeat_front_addr":"172.21.15.16:6829/3018526646","state":["exists","up"]},{"osd":3,"uuid":"c7771fd3-e18e-4500-ac38-23cf10ebcd75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6809","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6812","nonce":3360352974}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6817","nonce":3360352974}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6828","nonce":3360352974}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6823","nonce":3360352974}]},"public_addr":"172.21.15.16:6812/3360352974","cluster_addr":"172.21.15.16:6817/3360352974","heartbeat_back_addr":"172.21.15.16:6828/3360352974","heartbeat_front_addr":"172.21.15.16:6823/3360352974","state":["exists","up"]},{"osd":4,"uuid":"88260730-1482-4fbd-a40f-cb1124e9df4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6801","nonce":1856844856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6803","nonce":1856844856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6807","nonce":1856844856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6805","nonce":1856844856}]},"public_addr":"172.21.15.89:6801/1856844856","cluster_addr":"172.21.15.89:6803/1856844856","heartbeat_back_addr":"172.21.15.89:6807/1856844856","heartbeat_front_addr":"172.21.15.89:6805/1856844856","state":["exists","up"]},{"osd":5,"uuid":"4cb2c00c-fae0-4dd8-9d27-ec4f0093ccf7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6814","nonce":50395527}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6817","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6820","nonce":50395527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6831","nonce":50395527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6823","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6826","nonce":50395527}]},"public_addr":"172.21.15.89:6814/50395527","cluster_addr":"172.21.15.89:6820/50395527","heartbeat_back_addr":"172.21.15.89:6831/50395527","heartbeat_front_addr":"172.21.15.89:6826/50395527","state":["exists","up"]},{"osd":6,"uuid":"81ce0c19-2554-4cae-acbe-66a05e9e9b9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6809","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6812","nonce":3470471752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6815","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6818","nonce":3470471752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6827","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6829","nonce":3470471752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6821","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6824","nonce":3470471752}]},"public_addr":"172.21.15.89:6812/3470471752","cluster_addr":"172.21.15.89:6818/3470471752","heartbeat_back_addr":"172.21.15.89:6829/3470471752","heartbeat_front_addr":"172.21.15.89:6824/3470471752","state":["exists","up"]},{"osd":7,"uuid":"85e4313b-d37d-442e-8042-153bf20a847d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6811","nonce":1796778630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6813","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6816","nonce":1796778630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6825","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6828","nonce":1796778630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6819","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6822","nonce":1796778630}]},"public_addr":"172.21.15.89:6811/1796778630","cluster_addr":"172.21.15.89:6816/1796778630","heartbeat_back_addr":"172.21.15.89:6828/1796778630","heartbeat_front_addr":"172.21.15.89:6822/1796778630","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.439088+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.529970+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.680512+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:21.044161+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.582111+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.536558+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.409522+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.590229+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-31T15:05:13.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:13.491+0000 7fdee9c00640 1 -- 172.21.15.16:0/2348239385 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fdec0068380 msgr2=0x7fdec006a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:13.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:13.491+0000 7fdee9c00640 1 --2- 172.21.15.16:0/2348239385 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fdec0068380 0x7fdec006a840 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fded4000f70 tx=0x7fded4007e30 comp rx=0 tx=0).stop 2024-03-31T15:05:13.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:13.491+0000 7fdee9c00640 1 -- 172.21.15.16:0/2348239385 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdee4145ab0 msgr2=0x7fdee413a130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:13.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:13.491+0000 7fdee9c00640 1 --2- 172.21.15.16:0/2348239385 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdee4145ab0 0x7fdee413a130 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7fdecc00ea00 tx=0x7fdecc00eec0 comp rx=0 tx=0).stop 2024-03-31T15:05:13.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:13.491+0000 7fdee9c00640 1 -- 172.21.15.16:0/2348239385 shutdown_connections 2024-03-31T15:05:13.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:13.491+0000 7fdee9c00640 1 --2- 172.21.15.16:0/2348239385 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fdec0068380 0x7fdec006a840 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:13.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:13.491+0000 7fdee9c00640 1 --2- 172.21.15.16:0/2348239385 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdee413a670 0x7fdee413d1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:13.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:13.491+0000 7fdee9c00640 1 --2- 172.21.15.16:0/2348239385 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdee4145ab0 0x7fdee413a130 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:13.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:13.491+0000 7fdee9c00640 1 --2- 172.21.15.16:0/2348239385 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fdee406bd30 0x7fdee41418a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:13.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:13.491+0000 7fdee9c00640 1 -- 172.21.15.16:0/2348239385 >> 172.21.15.16:0/2348239385 conn(0x7fdee40b2b30 msgr2=0x7fdee4140700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:05:13.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:13.491+0000 7fdee9c00640 1 -- 172.21.15.16:0/2348239385 shutdown_connections 2024-03-31T15:05:13.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:13.491+0000 7fdee9c00640 1 -- 172.21.15.16:0/2348239385 wait complete. 2024-03-31T15:05:13.507 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-31T15:05:13.509 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-31T15:05:13.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:13.655+0000 7f6fef400640 1 Processor -- start 2024-03-31T15:05:13.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:13.655+0000 7f6fef400640 1 -- start start 2024-03-31T15:05:13.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:13.655+0000 7f6fef400640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f6fe8147e70 0x7f6fe806b620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:13.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:13.655+0000 7f6fef400640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f6fe80d0c50 0x7f6fe8072fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:13.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:13.655+0000 7f6fef400640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6fe81621b0 con 0x7f6fe8073510 2024-03-31T15:05:13.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:13.655+0000 7f6fef400640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f6fe80739a0 con 0x7f6fe80d0c50 2024-03-31T15:05:13.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:13.655+0000 7f6fef400640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f6fe8162490 con 0x7f6fe8147e70 2024-03-31T15:05:13.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:13.655+0000 7f6fed400640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f6fe8073510 0x7f6fe80cdab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:36462/0 (socket says 172.21.15.16:36462) 2024-03-31T15:05:13.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:13.655+0000 7f6fed400640 1 -- 172.21.15.16:0/3608427379 learned_addr learned my addr 172.21.15.16:0/3608427379 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:05:13.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:13.655+0000 7f6fdfe00640 1 --2- 172.21.15.16:0/3608427379 >> v2:172.21.15.16:3300/0 conn(0x7f6fe80d0c50 0x7f6fe8072fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:13.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:13.655+0000 7f6feca00640 1 --2- 172.21.15.16:0/3608427379 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f6fe8147e70 0x7f6fe806b620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:13.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:13.655+0000 7f6fe7e00640 1 -- 172.21.15.16:0/3608427379 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2562849200 0 0) 0x7f6fe81621b0 con 0x7f6fe8073510 2024-03-31T15:05:13.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:13.655+0000 7f6fe7e00640 1 -- 172.21.15.16:0/3608427379 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6fcc003530 con 0x7f6fe8073510 2024-03-31T15:05:13.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:13.655+0000 7f6fdfe00640 1 -- 172.21.15.16:0/3608427379 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f6fe8147e70 msgr2=0x7f6fe806b620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:05:13.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:13.655+0000 7f6fdfe00640 1 --2- 172.21.15.16:0/3608427379 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f6fe8147e70 0x7f6fe806b620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:13.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:13.655+0000 7f6fdfe00640 1 -- 172.21.15.16:0/3608427379 >> v1:172.21.15.16:6789/0 conn(0x7f6fe8073510 legacy=0x7f6fe80cdab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:13.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:13.655+0000 7f6fdfe00640 1 -- 172.21.15.16:0/3608427379 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6fe8162770 con 0x7f6fe80d0c50 2024-03-31T15:05:13.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:13.659+0000 7f6fdfe00640 1 --2- 172.21.15.16:0/3608427379 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6fe80d0c50 0x7f6fe8072fd0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f6fd00097c0 tx=0x7f6fd001c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b29f496fa4a20d24 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:05:13.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:13.659+0000 7f6fe7e00640 1 -- 172.21.15.16:0/3608427379 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fd00b0070 con 0x7f6fe80d0c50 2024-03-31T15:05:13.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:13.659+0000 7f6fe7e00640 1 -- 172.21.15.16:0/3608427379 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6fd00a4e50 con 0x7f6fe80d0c50 2024-03-31T15:05:13.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:13.659+0000 7f6fe7e00640 1 -- 172.21.15.16:0/3608427379 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fd00ab820 con 0x7f6fe80d0c50 2024-03-31T15:05:13.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:13.659+0000 7f6fef400640 1 -- 172.21.15.16:0/3608427379 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6fe80d0c50 msgr2=0x7f6fe8072fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:13.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:13.659+0000 7f6fef400640 1 --2- 172.21.15.16:0/3608427379 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6fe80d0c50 0x7f6fe8072fd0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f6fd00097c0 tx=0x7f6fd001c4a0 comp rx=0 tx=0).stop 2024-03-31T15:05:13.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:13.659+0000 7f6fef400640 1 -- 172.21.15.16:0/3608427379 shutdown_connections 2024-03-31T15:05:13.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:13.659+0000 7f6fef400640 1 --2- 172.21.15.16:0/3608427379 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6fe80d0c50 0x7f6fe8072fd0 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:13.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:13.659+0000 7f6fef400640 1 --2- 172.21.15.16:0/3608427379 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f6fe8147e70 0x7f6fe806b620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:13.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:13.659+0000 7f6fef400640 1 -- 172.21.15.16:0/3608427379 >> 172.21.15.16:0/3608427379 conn(0x7f6fe80d1060 msgr2=0x7f6fe80d1470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:05:13.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:13.659+0000 7f6fef400640 1 -- 172.21.15.16:0/3608427379 shutdown_connections 2024-03-31T15:05:13.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:13.659+0000 7f6fef400640 1 -- 172.21.15.16:0/3608427379 wait complete. 2024-03-31T15:05:13.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:13.663+0000 7f6fef400640 1 Processor -- start 2024-03-31T15:05:13.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:13.663+0000 7f6fef400640 1 -- start start 2024-03-31T15:05:13.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:13.663+0000 7f6fef400640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f6fe8073510 0x7f6fe80b0f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:13.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:13.663+0000 7f6fef400640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6fe80d0c50 0x7f6fe81466b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:13.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:13.663+0000 7f6fef400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6fe8147e70 0x7f6fe815ba10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:13.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:13.663+0000 7f6fef400640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f6fe806bce0 con 0x7f6fe8147e70 2024-03-31T15:05:13.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:13.663+0000 7f6fef400640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f6fe806bb60 con 0x7f6fe80d0c50 2024-03-31T15:05:13.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:13.663+0000 7f6fef400640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f6fe816b4b0 con 0x7f6fe8073510 2024-03-31T15:05:13.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:13.663+0000 7f6fed400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6fe8147e70 0x7f6fe815ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:13.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:13.663+0000 7f6feca00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f6fe8073510 0x7f6fe80b0f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:13.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:13.663+0000 7f6fdfe00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6fe80d0c50 0x7f6fe81466b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:13.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:13.663+0000 7f6fed400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6fe8147e70 0x7f6fe815ba10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34908/0 (socket says 172.21.15.16:34908) 2024-03-31T15:05:13.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:13.663+0000 7f6fdfe00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6fe80d0c50 0x7f6fe81466b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.16:57012/0 (socket says 172.21.15.16:57012) 2024-03-31T15:05:13.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:13.663+0000 7f6fed400640 1 -- 172.21.15.16:0/4279063987 learned_addr learned my addr 172.21.15.16:0/4279063987 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:05:13.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:13.663+0000 7f6feca00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f6fe8073510 0x7f6fe80b0f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3301/0 says I am v2:172.21.15.16:37766/0 (socket says 172.21.15.16:37766) 2024-03-31T15:05:13.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:13.663+0000 7f6fdfe00640 1 -- 172.21.15.16:0/4279063987 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f6fe8073510 msgr2=0x7f6fe80b0f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:13.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:13.663+0000 7f6fdfe00640 1 --2- 172.21.15.16:0/4279063987 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f6fe8073510 0x7f6fe80b0f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:13.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:13.663+0000 7f6fdfe00640 1 -- 172.21.15.16:0/4279063987 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6fe8147e70 msgr2=0x7f6fe815ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:13.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:13.663+0000 7f6fdfe00640 1 --2- 172.21.15.16:0/4279063987 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6fe8147e70 0x7f6fe815ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:13.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:13.663+0000 7f6fdfe00640 1 -- 172.21.15.16:0/4279063987 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6fe815ded0 con 0x7f6fe80d0c50 2024-03-31T15:05:13.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:13.663+0000 7f6fdfe00640 1 --2- 172.21.15.16:0/4279063987 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6fe80d0c50 0x7f6fe81466b0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f6fd00ab000 tx=0x7f6fd0019240 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:05:13.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:13.663+0000 7f6fe5600640 1 -- 172.21.15.16:0/4279063987 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fd00ba070 con 0x7f6fe80d0c50 2024-03-31T15:05:13.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:13.663+0000 7f6fe5600640 1 -- 172.21.15.16:0/4279063987 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6fd00abda0 con 0x7f6fe80d0c50 2024-03-31T15:05:13.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:13.663+0000 7f6fef400640 1 -- 172.21.15.16:0/4279063987 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6fe815ca20 con 0x7f6fe80d0c50 2024-03-31T15:05:13.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:13.663+0000 7f6fe5600640 1 -- 172.21.15.16:0/4279063987 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fd00b0040 con 0x7f6fe80d0c50 2024-03-31T15:05:13.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:13.663+0000 7f6fef400640 1 -- 172.21.15.16:0/4279063987 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6fe815cfe0 con 0x7f6fe80d0c50 2024-03-31T15:05:13.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:13.663+0000 7f6fef400640 1 -- 172.21.15.16:0/4279063987 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6fe8069950 con 0x7f6fe80d0c50 2024-03-31T15:05:13.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:13.667+0000 7f6fe5600640 1 -- 172.21.15.16:0/4279063987 <== mon.1 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f6fd00bd070 con 0x7f6fe80d0c50 2024-03-31T15:05:13.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:13.667+0000 7f6fe5600640 1 --2- 172.21.15.16:0/4279063987 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f6fc4068380 0x7f6fc406a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:13.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:13.667+0000 7f6fe5600640 1 -- 172.21.15.16:0/4279063987 <== mon.1 v2:172.21.15.89:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5709+0+0 (secure 0 0 0) 0x7f6fd011f860 con 0x7f6fe80d0c50 2024-03-31T15:05:13.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:13.667+0000 7f6feca00640 1 --2- 172.21.15.16:0/4279063987 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f6fc4068380 0x7f6fc406a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:13.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:13.667+0000 7f6feca00640 1 --2- 172.21.15.16:0/4279063987 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f6fc4068380 0x7f6fc406a840 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f6fe8169470 tx=0x7f6fd8009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:05:13.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:13.671+0000 7f6fe5600640 1 -- 172.21.15.16:0/4279063987 <== mon.1 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f6fe815ded0 con 0x7f6fe80d0c50 2024-03-31T15:05:13.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:13.867+0000 7f6fef400640 1 -- 172.21.15.16:0/4279063987 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6fe80d1bf0 con 0x7f6fe80d0c50 2024-03-31T15:05:13.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:13.871+0000 7f6fe5600640 1 -- 172.21.15.16:0/4279063987 <== mon.1 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) ==== 94+0+5178 (secure 0 0 0) 0x7f6fd0016090 con 0x7f6fe80d0c50 2024-03-31T15:05:13.865 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:05:13.865 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":16,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4949,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6836/2073954753","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":2073954753},{"type":"v1","addr":"172.21.15.89:6836","nonce":2073954753}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":4952,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/2143918278","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6835","nonce":2143918278},{"type":"v1","addr":"172.21.15.89:6837","nonce":2143918278}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":5251,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6833/881391342","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":881391342},{"type":"v1","addr":"172.21.15.16:6833","nonce":881391342}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-31T15:04:39.780323+0000","modified":"2024-03-31T15:04:42.846439+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5254},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5254":{"gid":5254,"name":"c","rank":0,"incarnation":13,"state":"up:active","state_seq":3,"addr":"172.21.15.16:6835/1845707550","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":1845707550},{"type":"v1","addr":"172.21.15.16:6835","nonce":1845707550}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5254,"qdb_cluster":[5254]},"id":2}]} 2024-03-31T15:05:13.865 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 16 2024-03-31T15:05:13.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:13.871+0000 7f6fef400640 1 -- 172.21.15.16:0/4279063987 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f6fc4068380 msgr2=0x7f6fc406a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:13.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:13.871+0000 7f6fef400640 1 --2- 172.21.15.16:0/4279063987 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f6fc4068380 0x7f6fc406a840 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f6fe8169470 tx=0x7f6fd8009190 comp rx=0 tx=0).stop 2024-03-31T15:05:13.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:13.871+0000 7f6fef400640 1 -- 172.21.15.16:0/4279063987 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6fe80d0c50 msgr2=0x7f6fe81466b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:13.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:13.871+0000 7f6fef400640 1 --2- 172.21.15.16:0/4279063987 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6fe80d0c50 0x7f6fe81466b0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f6fd00ab000 tx=0x7f6fd0019240 comp rx=0 tx=0).stop 2024-03-31T15:05:13.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:13.871+0000 7f6fef400640 1 -- 172.21.15.16:0/4279063987 shutdown_connections 2024-03-31T15:05:13.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:13.871+0000 7f6fef400640 1 --2- 172.21.15.16:0/4279063987 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f6fc4068380 0x7f6fc406a840 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:13.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:13.871+0000 7f6fef400640 1 --2- 172.21.15.16:0/4279063987 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6fe8147e70 0x7f6fe815ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:13.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:13.871+0000 7f6fef400640 1 --2- 172.21.15.16:0/4279063987 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6fe80d0c50 0x7f6fe81466b0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:13.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:13.871+0000 7f6fef400640 1 --2- 172.21.15.16:0/4279063987 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f6fe8073510 0x7f6fe80b0f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:13.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:13.871+0000 7f6fef400640 1 -- 172.21.15.16:0/4279063987 >> 172.21.15.16:0/4279063987 conn(0x7f6fe80d1060 msgr2=0x7f6fe806ee90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:05:13.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:13.871+0000 7f6fef400640 1 -- 172.21.15.16:0/4279063987 shutdown_connections 2024-03-31T15:05:13.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:13.871+0000 7f6fef400640 1 -- 172.21.15.16:0/4279063987 wait complete. 2024-03-31T15:05:13.887 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-31T15:05:14.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:14.031+0000 7f9592400640 1 Processor -- start 2024-03-31T15:05:14.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:14.031+0000 7f9592400640 1 -- start start 2024-03-31T15:05:14.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:14.035+0000 7f9592400640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f958c06fd80 0x7f958c0d2720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:14.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:14.035+0000 7f9592400640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f958c0d0920 con 0x7f958c06f920 2024-03-31T15:05:14.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:14.035+0000 7f9592400640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f958c0d0c00 con 0x7f958c145e40 2024-03-31T15:05:14.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:14.035+0000 7f9592400640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f958c0d06d0 con 0x7f958c06fd80 2024-03-31T15:05:14.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:14.035+0000 7f958be00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f958c06fd80 0x7f958c0d2720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:14.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:14.035+0000 7f9583e00640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f958c06f920 0x7f958c0d2010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:36472/0 (socket says 172.21.15.16:36472) 2024-03-31T15:05:14.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:14.035+0000 7f9583e00640 1 -- 172.21.15.16:0/3582447308 learned_addr learned my addr 172.21.15.16:0/3582447308 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:05:14.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:14.035+0000 7f958be00640 1 -- 172.21.15.16:0/3582447308 >> v1:172.21.15.89:6789/0 conn(0x7f958c145e40 legacy=0x7f958c0cc740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:14.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:14.035+0000 7f958be00640 1 -- 172.21.15.16:0/3582447308 >> v1:172.21.15.16:6789/0 conn(0x7f958c06f920 legacy=0x7f958c0d2010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:14.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:14.035+0000 7f958be00640 1 -- 172.21.15.16:0/3582447308 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f958c0d1810 con 0x7f958c06fd80 2024-03-31T15:05:14.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:14.035+0000 7f958be00640 1 --2- 172.21.15.16:0/3582447308 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f958c06fd80 0x7f958c0d2720 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f9584009d90 tx=0x7f958401c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=93cfa2f47a6a9b56 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:05:14.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:14.039+0000 7f958aa00640 1 -- 172.21.15.16:0/3582447308 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95840b2070 con 0x7f958c06fd80 2024-03-31T15:05:14.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:14.039+0000 7f958aa00640 1 -- 172.21.15.16:0/3582447308 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f95840a5920 con 0x7f958c06fd80 2024-03-31T15:05:14.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:14.039+0000 7f958aa00640 1 -- 172.21.15.16:0/3582447308 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95840a5cf0 con 0x7f958c06fd80 2024-03-31T15:05:14.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:14.039+0000 7f9592400640 1 -- 172.21.15.16:0/3582447308 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f958c06fd80 msgr2=0x7f958c0d2720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:14.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:14.039+0000 7f9592400640 1 --2- 172.21.15.16:0/3582447308 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f958c06fd80 0x7f958c0d2720 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f9584009d90 tx=0x7f958401c970 comp rx=0 tx=0).stop 2024-03-31T15:05:14.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:14.039+0000 7f9592400640 1 -- 172.21.15.16:0/3582447308 shutdown_connections 2024-03-31T15:05:14.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:14.039+0000 7f9592400640 1 --2- 172.21.15.16:0/3582447308 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f958c06fd80 0x7f958c0d2720 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:14.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:14.039+0000 7f9592400640 1 -- 172.21.15.16:0/3582447308 >> 172.21.15.16:0/3582447308 conn(0x7f958c067330 msgr2=0x7f958c067740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:05:14.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:14.039+0000 7f9592400640 1 -- 172.21.15.16:0/3582447308 shutdown_connections 2024-03-31T15:05:14.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:14.039+0000 7f9592400640 1 -- 172.21.15.16:0/3582447308 wait complete. 2024-03-31T15:05:14.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:14.039+0000 7f9592400640 1 Processor -- start 2024-03-31T15:05:14.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:14.039+0000 7f9592400640 1 -- start start 2024-03-31T15:05:14.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:14.043+0000 7f9592400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f958c06f920 0x7f958c142b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:14.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:14.043+0000 7f9592400640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f958c06fd80 0x7f958c1f5ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:14.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:14.043+0000 7f9592400640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f958c145e40 0x7f958c1f6020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:14.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:14.043+0000 7f9592400640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f958c0d0ee0 con 0x7f958c06f920 2024-03-31T15:05:14.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:14.043+0000 7f9592400640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f958c0ce9e0 con 0x7f958c145e40 2024-03-31T15:05:14.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:14.043+0000 7f9592400640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f958c0b50c0 con 0x7f958c06fd80 2024-03-31T15:05:14.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:14.043+0000 7f958b400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f958c06f920 0x7f958c142b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:14.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:14.043+0000 7f958b400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f958c06f920 0x7f958c142b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34940/0 (socket says 172.21.15.16:34940) 2024-03-31T15:05:14.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:14.043+0000 7f958b400640 1 -- 172.21.15.16:0/780023751 learned_addr learned my addr 172.21.15.16:0/780023751 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:05:14.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:14.043+0000 7f958be00640 1 --2- 172.21.15.16:0/780023751 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f958c145e40 0x7f958c1f6020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:14.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:14.043+0000 7f9583e00640 1 --2- 172.21.15.16:0/780023751 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f958c06fd80 0x7f958c1f5ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:14.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:14.043+0000 7f958b400640 1 -- 172.21.15.16:0/780023751 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f958c06fd80 msgr2=0x7f958c1f5ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:14.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:14.043+0000 7f958b400640 1 --2- 172.21.15.16:0/780023751 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f958c06fd80 0x7f958c1f5ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:14.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:14.043+0000 7f958b400640 1 -- 172.21.15.16:0/780023751 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f958c145e40 msgr2=0x7f958c1f6020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:14.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:14.043+0000 7f958b400640 1 --2- 172.21.15.16:0/780023751 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f958c145e40 0x7f958c1f6020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:14.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:14.043+0000 7f958b400640 1 -- 172.21.15.16:0/780023751 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f958c15d160 con 0x7f958c06f920 2024-03-31T15:05:14.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:14.043+0000 7f9583e00640 1 --2- 172.21.15.16:0/780023751 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f958c06fd80 0x7f958c1f5ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:05:14.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:14.043+0000 7f958b400640 1 --2- 172.21.15.16:0/780023751 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f958c06f920 0x7f958c142b10 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f957c00c450 tx=0x7f957c00c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:05:14.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:14.043+0000 7f9583400640 1 -- 172.21.15.16:0/780023751 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f957c016020 con 0x7f958c06f920 2024-03-31T15:05:14.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:14.043+0000 7f9583400640 1 -- 172.21.15.16:0/780023751 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f957c0052d0 con 0x7f958c06f920 2024-03-31T15:05:14.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:14.043+0000 7f9583400640 1 -- 172.21.15.16:0/780023751 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f957c007260 con 0x7f958c06f920 2024-03-31T15:05:14.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:14.043+0000 7f9592400640 1 -- 172.21.15.16:0/780023751 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f958c15d440 con 0x7f958c06f920 2024-03-31T15:05:14.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:14.043+0000 7f9592400640 1 -- 172.21.15.16:0/780023751 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f958c15da00 con 0x7f958c06f920 2024-03-31T15:05:14.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:14.043+0000 7f9592400640 1 -- 172.21.15.16:0/780023751 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f958c069950 con 0x7f958c06f920 2024-03-31T15:05:14.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:14.047+0000 7f9583400640 1 -- 172.21.15.16:0/780023751 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f957c007400 con 0x7f958c06f920 2024-03-31T15:05:14.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:14.047+0000 7f9583400640 1 --2- 172.21.15.16:0/780023751 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f9568068380 0x7f956806a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:14.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:14.047+0000 7f9583400640 1 -- 172.21.15.16:0/780023751 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5709+0+0 (secure 0 0 0) 0x7f957c0862f0 con 0x7f958c06f920 2024-03-31T15:05:14.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:14.047+0000 7f9583e00640 1 --2- 172.21.15.16:0/780023751 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f9568068380 0x7f956806a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:14.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:14.047+0000 7f9583e00640 1 --2- 172.21.15.16:0/780023751 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f9568068380 0x7f956806a840 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f958c1599e0 tx=0x7f9574009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:05:14.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:14.051+0000 7f9583400640 1 -- 172.21.15.16:0/780023751 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f957c020020 con 0x7f958c06f920 2024-03-31T15:05:14.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:14.247+0000 7f9592400640 1 -- 172.21.15.16:0/780023751 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f958c0cf3c0 con 0x7f958c06f920 2024-03-31T15:05:14.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:14.251+0000 7f9583400640 1 -- 172.21.15.16:0/780023751 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) ==== 94+0+5178 (secure 0 0 0) 0x7f957c005480 con 0x7f958c06f920 2024-03-31T15:05:14.245 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:05:14.245 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":16,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4949,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6836/2073954753","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":2073954753},{"type":"v1","addr":"172.21.15.89:6836","nonce":2073954753}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":4952,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/2143918278","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6835","nonce":2143918278},{"type":"v1","addr":"172.21.15.89:6837","nonce":2143918278}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":5251,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6833/881391342","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":881391342},{"type":"v1","addr":"172.21.15.16:6833","nonce":881391342}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-31T15:04:39.780323+0000","modified":"2024-03-31T15:04:42.846439+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5254},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5254":{"gid":5254,"name":"c","rank":0,"incarnation":13,"state":"up:active","state_seq":3,"addr":"172.21.15.16:6835/1845707550","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":1845707550},{"type":"v1","addr":"172.21.15.16:6835","nonce":1845707550}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5254,"qdb_cluster":[5254]},"id":2}]} 2024-03-31T15:05:14.245 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 16 2024-03-31T15:05:14.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:14.251+0000 7f9592400640 1 -- 172.21.15.16:0/780023751 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f9568068380 msgr2=0x7f956806a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:14.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:14.251+0000 7f9592400640 1 --2- 172.21.15.16:0/780023751 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f9568068380 0x7f956806a840 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f958c1599e0 tx=0x7f9574009040 comp rx=0 tx=0).stop 2024-03-31T15:05:14.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:14.251+0000 7f9592400640 1 -- 172.21.15.16:0/780023751 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f958c06f920 msgr2=0x7f958c142b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:14.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:14.251+0000 7f9592400640 1 --2- 172.21.15.16:0/780023751 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f958c06f920 0x7f958c142b10 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f957c00c450 tx=0x7f957c00c910 comp rx=0 tx=0).stop 2024-03-31T15:05:14.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:14.251+0000 7f9592400640 1 -- 172.21.15.16:0/780023751 shutdown_connections 2024-03-31T15:05:14.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:14.251+0000 7f9592400640 1 --2- 172.21.15.16:0/780023751 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f9568068380 0x7f956806a840 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:14.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:14.251+0000 7f9592400640 1 --2- 172.21.15.16:0/780023751 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f958c145e40 0x7f958c1f6020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:14.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:14.251+0000 7f9592400640 1 --2- 172.21.15.16:0/780023751 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f958c06fd80 0x7f958c1f5ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:14.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:14.251+0000 7f9592400640 1 --2- 172.21.15.16:0/780023751 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f958c06f920 0x7f958c142b10 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:14.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:14.251+0000 7f9592400640 1 -- 172.21.15.16:0/780023751 >> 172.21.15.16:0/780023751 conn(0x7f958c067330 msgr2=0x7f958c0cc4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:05:14.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:14.251+0000 7f9592400640 1 -- 172.21.15.16:0/780023751 shutdown_connections 2024-03-31T15:05:14.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:14.251+0000 7f9592400640 1 -- 172.21.15.16:0/780023751 wait complete. 2024-03-31T15:05:14.266 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-31T15:05:14.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:14.411+0000 7fefe1600640 1 Processor -- start 2024-03-31T15:05:14.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:14.411+0000 7fefe1600640 1 -- start start 2024-03-31T15:05:14.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:14.411+0000 7fefe1600640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fefdc06f920 0x7fefdc06fd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:14.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:14.411+0000 7fefe1600640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fefdc0d0920 con 0x7fefdc0d2010 2024-03-31T15:05:14.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:14.411+0000 7fefe1600640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fefdc0d0c00 con 0x7fefdc145e40 2024-03-31T15:05:14.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:14.411+0000 7fefe1600640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fefdc0702c0 con 0x7fefdc06f920 2024-03-31T15:05:14.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:14.411+0000 7fefdb400640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fefdc0d2010 0x7fefdc0d24a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:36478/0 (socket says 172.21.15.16:36478) 2024-03-31T15:05:14.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:14.411+0000 7fefdb400640 1 -- 172.21.15.16:0/528029576 learned_addr learned my addr 172.21.15.16:0/528029576 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:05:14.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:14.411+0000 7fefd2000640 1 --2- 172.21.15.16:0/528029576 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fefdc06f920 0x7fefdc06fd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:14.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:14.411+0000 7fefda000640 1 -- 172.21.15.16:0/528029576 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2598645138 0 0) 0x7fefdc0d0920 con 0x7fefdc0d2010 2024-03-31T15:05:14.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:14.411+0000 7fefda000640 1 -- 172.21.15.16:0/528029576 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fefc0003530 con 0x7fefdc0d2010 2024-03-31T15:05:14.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:14.415+0000 7fefd2000640 1 -- 172.21.15.16:0/528029576 >> v1:172.21.15.89:6789/0 conn(0x7fefdc145e40 legacy=0x7fefdc0cc740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:14.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:14.415+0000 7fefda000640 1 -- 172.21.15.16:0/528029576 <== mon.1 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2708066719 0 0) 0x7fefdc0d0c00 con 0x7fefdc145e40 2024-03-31T15:05:14.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:14.415+0000 7fefd2000640 1 -- 172.21.15.16:0/528029576 >> v1:172.21.15.16:6789/0 conn(0x7fefdc0d2010 legacy=0x7fefdc0d24a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:14.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:14.415+0000 7fefd2000640 1 -- 172.21.15.16:0/528029576 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fefdc0d1810 con 0x7fefdc06f920 2024-03-31T15:05:14.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:14.415+0000 7fefd2000640 1 --2- 172.21.15.16:0/528029576 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fefdc06f920 0x7fefdc06fd80 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fefc40097f0 tx=0x7fefc401c320 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5aad7723fa1d4004 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:05:14.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:14.415+0000 7fefda000640 1 -- 172.21.15.16:0/528029576 <== mon.2 v2:172.21.15.89:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefc40b0070 con 0x7fefdc06f920 2024-03-31T15:05:14.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:14.415+0000 7fefda000640 1 -- 172.21.15.16:0/528029576 <== mon.2 v2:172.21.15.89:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fefc40ab400 con 0x7fefdc06f920 2024-03-31T15:05:14.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:14.415+0000 7fefda000640 1 -- 172.21.15.16:0/528029576 <== mon.2 v2:172.21.15.89:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefd4001a00 con 0x7fefdc06f920 2024-03-31T15:05:14.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:14.415+0000 7fefe1600640 1 -- 172.21.15.16:0/528029576 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fefdc06f920 msgr2=0x7fefdc06fd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:14.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:14.415+0000 7fefe1600640 1 --2- 172.21.15.16:0/528029576 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fefdc06f920 0x7fefdc06fd80 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fefc40097f0 tx=0x7fefc401c320 comp rx=0 tx=0).stop 2024-03-31T15:05:14.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:14.415+0000 7fefe1600640 1 -- 172.21.15.16:0/528029576 shutdown_connections 2024-03-31T15:05:14.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:14.415+0000 7fefe1600640 1 --2- 172.21.15.16:0/528029576 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fefdc06f920 0x7fefdc06fd80 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:14.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:14.415+0000 7fefe1600640 1 -- 172.21.15.16:0/528029576 >> 172.21.15.16:0/528029576 conn(0x7fefdc067330 msgr2=0x7fefdc067740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:05:14.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:14.415+0000 7fefe1600640 1 -- 172.21.15.16:0/528029576 shutdown_connections 2024-03-31T15:05:14.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:14.419+0000 7fefe1600640 1 -- 172.21.15.16:0/528029576 wait complete. 2024-03-31T15:05:14.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:14.419+0000 7fefe1600640 1 Processor -- start 2024-03-31T15:05:14.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:14.419+0000 7fefe1600640 1 -- start start 2024-03-31T15:05:14.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:14.419+0000 7fefe1600640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fefdc06f920 0x7fefdc142a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:14.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:14.419+0000 7fefe1600640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fefdc0d2010 0x7fefdc15ffe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:14.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:14.419+0000 7fefe1600640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fefdc145e40 0x7fefdc144490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:14.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:14.419+0000 7fefe1600640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fefdc0d0ee0 con 0x7fefdc06f920 2024-03-31T15:05:14.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:14.419+0000 7fefe1600640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fefdc0d06d0 con 0x7fefdc0d2010 2024-03-31T15:05:14.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:14.419+0000 7fefe1600640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fefdc0b50c0 con 0x7fefdc145e40 2024-03-31T15:05:14.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:14.419+0000 7fefdaa00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fefdc06f920 0x7fefdc142a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:14.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:14.419+0000 7fefdaa00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fefdc06f920 0x7fefdc142a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34942/0 (socket says 172.21.15.16:34942) 2024-03-31T15:05:14.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:14.419+0000 7fefdaa00640 1 -- 172.21.15.16:0/742886675 learned_addr learned my addr 172.21.15.16:0/742886675 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:05:14.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:14.419+0000 7fefd2000640 1 --2- 172.21.15.16:0/742886675 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fefdc0d2010 0x7fefdc15ffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:14.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:14.419+0000 7fefdb400640 1 --2- 172.21.15.16:0/742886675 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fefdc145e40 0x7fefdc144490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:14.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:14.419+0000 7fefdaa00640 1 -- 172.21.15.16:0/742886675 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fefdc145e40 msgr2=0x7fefdc144490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:14.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:14.419+0000 7fefdaa00640 1 --2- 172.21.15.16:0/742886675 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fefdc145e40 0x7fefdc144490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:14.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:14.419+0000 7fefdaa00640 1 -- 172.21.15.16:0/742886675 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fefdc0d2010 msgr2=0x7fefdc15ffe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:14.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:14.419+0000 7fefdaa00640 1 --2- 172.21.15.16:0/742886675 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fefdc0d2010 0x7fefdc15ffe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:14.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:14.419+0000 7fefdaa00640 1 -- 172.21.15.16:0/742886675 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fefdc15cec0 con 0x7fefdc06f920 2024-03-31T15:05:14.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:14.419+0000 7fefd2000640 1 --2- 172.21.15.16:0/742886675 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fefdc0d2010 0x7fefdc15ffe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:05:14.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:14.419+0000 7fefdb400640 1 --2- 172.21.15.16:0/742886675 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fefdc145e40 0x7fefdc144490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:05:14.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:14.419+0000 7fefdaa00640 1 --2- 172.21.15.16:0/742886675 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fefdc06f920 0x7fefdc142a40 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7fefcc00c410 tx=0x7fefcc00c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:05:14.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:14.419+0000 7fefd3400640 1 -- 172.21.15.16:0/742886675 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefcc017070 con 0x7fefdc06f920 2024-03-31T15:05:14.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:14.419+0000 7fefd3400640 1 -- 172.21.15.16:0/742886675 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fefcc003600 con 0x7fefdc06f920 2024-03-31T15:05:14.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:14.419+0000 7fefe1600640 1 -- 172.21.15.16:0/742886675 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fefdc15d1a0 con 0x7fefdc06f920 2024-03-31T15:05:14.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:14.419+0000 7fefd3400640 1 -- 172.21.15.16:0/742886675 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefcc0039f0 con 0x7fefdc06f920 2024-03-31T15:05:14.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:14.419+0000 7fefe1600640 1 -- 172.21.15.16:0/742886675 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fefdc142f80 con 0x7fefdc06f920 2024-03-31T15:05:14.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:14.423+0000 7fefe1600640 1 -- 172.21.15.16:0/742886675 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fefdc069950 con 0x7fefdc06f920 2024-03-31T15:05:14.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:14.423+0000 7fefd3400640 1 -- 172.21.15.16:0/742886675 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fefcc01a070 con 0x7fefdc06f920 2024-03-31T15:05:14.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:14.423+0000 7fefd3400640 1 --2- 172.21.15.16:0/742886675 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fefb8068380 0x7fefb806a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:14.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:14.423+0000 7fefd3400640 1 -- 172.21.15.16:0/742886675 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5709+0+0 (secure 0 0 0) 0x7fefcc085e60 con 0x7fefdc06f920 2024-03-31T15:05:14.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:14.423+0000 7fefd2000640 1 --2- 172.21.15.16:0/742886675 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fefb8068380 0x7fefb806a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:14.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:14.423+0000 7fefd2000640 1 --2- 172.21.15.16:0/742886675 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fefb8068380 0x7fefb806a840 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fefc40ad000 tx=0x7fefc4008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:05:14.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:14.427+0000 7fefd3400640 1 -- 172.21.15.16:0/742886675 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fefcc056380 con 0x7fefdc06f920 2024-03-31T15:05:14.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:14.595+0000 7fefe1600640 1 -- 172.21.15.16:0/742886675 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fefdc0cf3c0 con 0x7fefdc06f920 2024-03-31T15:05:14.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:14.595+0000 7fefd3400640 1 -- 172.21.15.16:0/742886675 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v34) ==== 74+0+16597 (secure 0 0 0) 0x7fefcc001090 con 0x7fefdc06f920 2024-03-31T15:05:14.592 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:05:14.593 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":34,"fsid":"286eb5da-fb47-4550-b290-d9519c13b9a5","created":"2024-03-31T15:03:11.438952+0000","modified":"2024-03-31T15:04:41.385382+0000","last_up_change":"2024-03-31T15:03:22.885535+0000","last_in_change":"2024-03-31T15:03:13.941667+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-31T15:03:22.550787+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-03-31T15:04:37.577341+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-03-31T15:04:38.738566+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"34","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"92a66450-ef19-40d8-ac3d-d760b23b0bda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6801","nonce":932712244}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6803","nonce":932712244}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6807","nonce":932712244}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6805","nonce":932712244}]},"public_addr":"172.21.15.16:6801/932712244","cluster_addr":"172.21.15.16:6803/932712244","heartbeat_back_addr":"172.21.15.16:6807/932712244","heartbeat_front_addr":"172.21.15.16:6805/932712244","state":["exists","up"]},{"osd":1,"uuid":"da9d1f2b-1968-4959-b843-faa231988e92","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6810","nonce":1398737533}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6811","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6813","nonce":1398737533}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6821","nonce":1398737533}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6815","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6816","nonce":1398737533}]},"public_addr":"172.21.15.16:6810/1398737533","cluster_addr":"172.21.15.16:6813/1398737533","heartbeat_back_addr":"172.21.15.16:6821/1398737533","heartbeat_front_addr":"172.21.15.16:6816/1398737533","state":["exists","up"]},{"osd":2,"uuid":"01a73662-9a21-4b1f-bcbb-42b9636a4638","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6819","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6822","nonce":3018526646}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6825","nonce":3018526646}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6831","nonce":3018526646}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6827","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6829","nonce":3018526646}]},"public_addr":"172.21.15.16:6822/3018526646","cluster_addr":"172.21.15.16:6825/3018526646","heartbeat_back_addr":"172.21.15.16:6831/3018526646","heartbeat_front_addr":"172.21.15.16:6829/3018526646","state":["exists","up"]},{"osd":3,"uuid":"c7771fd3-e18e-4500-ac38-23cf10ebcd75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6809","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6812","nonce":3360352974}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6817","nonce":3360352974}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6828","nonce":3360352974}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6823","nonce":3360352974}]},"public_addr":"172.21.15.16:6812/3360352974","cluster_addr":"172.21.15.16:6817/3360352974","heartbeat_back_addr":"172.21.15.16:6828/3360352974","heartbeat_front_addr":"172.21.15.16:6823/3360352974","state":["exists","up"]},{"osd":4,"uuid":"88260730-1482-4fbd-a40f-cb1124e9df4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6801","nonce":1856844856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6803","nonce":1856844856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6807","nonce":1856844856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6805","nonce":1856844856}]},"public_addr":"172.21.15.89:6801/1856844856","cluster_addr":"172.21.15.89:6803/1856844856","heartbeat_back_addr":"172.21.15.89:6807/1856844856","heartbeat_front_addr":"172.21.15.89:6805/1856844856","state":["exists","up"]},{"osd":5,"uuid":"4cb2c00c-fae0-4dd8-9d27-ec4f0093ccf7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6814","nonce":50395527}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6817","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6820","nonce":50395527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6831","nonce":50395527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6823","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6826","nonce":50395527}]},"public_addr":"172.21.15.89:6814/50395527","cluster_addr":"172.21.15.89:6820/50395527","heartbeat_back_addr":"172.21.15.89:6831/50395527","heartbeat_front_addr":"172.21.15.89:6826/50395527","state":["exists","up"]},{"osd":6,"uuid":"81ce0c19-2554-4cae-acbe-66a05e9e9b9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6809","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6812","nonce":3470471752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6815","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6818","nonce":3470471752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6827","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6829","nonce":3470471752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6821","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6824","nonce":3470471752}]},"public_addr":"172.21.15.89:6812/3470471752","cluster_addr":"172.21.15.89:6818/3470471752","heartbeat_back_addr":"172.21.15.89:6829/3470471752","heartbeat_front_addr":"172.21.15.89:6824/3470471752","state":["exists","up"]},{"osd":7,"uuid":"85e4313b-d37d-442e-8042-153bf20a847d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6811","nonce":1796778630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6813","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6816","nonce":1796778630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6825","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6828","nonce":1796778630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6819","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6822","nonce":1796778630}]},"public_addr":"172.21.15.89:6811/1796778630","cluster_addr":"172.21.15.89:6816/1796778630","heartbeat_back_addr":"172.21.15.89:6828/1796778630","heartbeat_front_addr":"172.21.15.89:6822/1796778630","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.439088+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.529970+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.680512+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:21.044161+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.582111+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.536558+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.409522+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.590229+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-31T15:05:14.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:14.599+0000 7fefe1600640 1 -- 172.21.15.16:0/742886675 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fefb8068380 msgr2=0x7fefb806a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:14.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:14.599+0000 7fefe1600640 1 --2- 172.21.15.16:0/742886675 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fefb8068380 0x7fefb806a840 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fefc40ad000 tx=0x7fefc4008040 comp rx=0 tx=0).stop 2024-03-31T15:05:14.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:14.599+0000 7fefe1600640 1 -- 172.21.15.16:0/742886675 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fefdc06f920 msgr2=0x7fefdc142a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:14.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:14.599+0000 7fefe1600640 1 --2- 172.21.15.16:0/742886675 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fefdc06f920 0x7fefdc142a40 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7fefcc00c410 tx=0x7fefcc00c8d0 comp rx=0 tx=0).stop 2024-03-31T15:05:14.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:14.599+0000 7fefe1600640 1 -- 172.21.15.16:0/742886675 shutdown_connections 2024-03-31T15:05:14.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:14.599+0000 7fefe1600640 1 --2- 172.21.15.16:0/742886675 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fefb8068380 0x7fefb806a840 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:14.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:14.599+0000 7fefe1600640 1 --2- 172.21.15.16:0/742886675 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fefdc145e40 0x7fefdc144490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:14.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:14.599+0000 7fefe1600640 1 --2- 172.21.15.16:0/742886675 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fefdc0d2010 0x7fefdc15ffe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:14.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:14.599+0000 7fefe1600640 1 --2- 172.21.15.16:0/742886675 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fefdc06f920 0x7fefdc142a40 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:14.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:14.599+0000 7fefe1600640 1 -- 172.21.15.16:0/742886675 >> 172.21.15.16:0/742886675 conn(0x7fefdc067330 msgr2=0x7fefdc0b2aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:05:14.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:14.599+0000 7fefe1600640 1 -- 172.21.15.16:0/742886675 shutdown_connections 2024-03-31T15:05:14.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:14.599+0000 7fefe1600640 1 -- 172.21.15.16:0/742886675 wait complete. 2024-03-31T15:05:14.614 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-31T15:05:14.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:14.759+0000 7f8669400640 1 Processor -- start 2024-03-31T15:05:14.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:14.759+0000 7f8669400640 1 -- start start 2024-03-31T15:05:14.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:14.759+0000 7f8669400640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f866406f920 0x7f866406fd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:14.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:14.759+0000 7f8669400640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f86640d2010 0x7f86640d24a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:14.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:14.759+0000 7f8669400640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f86640d0920 con 0x7f8664145e40 2024-03-31T15:05:14.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:14.759+0000 7f8669400640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f86640702c0 con 0x7f86640d2010 2024-03-31T15:05:14.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:14.759+0000 7f8669400640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f86640d06d0 con 0x7f866406f920 2024-03-31T15:05:14.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:14.763+0000 7f8663200640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f86640d2010 0x7f86640d24a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:14.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:14.763+0000 7f8662800640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f8664145e40 0x7f86640cc740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:36488/0 (socket says 172.21.15.16:36488) 2024-03-31T15:05:14.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:14.763+0000 7f8662800640 1 -- 172.21.15.16:0/2091002928 learned_addr learned my addr 172.21.15.16:0/2091002928 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:05:14.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:14.763+0000 7f8661e00640 1 --2- 172.21.15.16:0/2091002928 >> v2:172.21.15.89:3300/0 conn(0x7f866406f920 0x7f866406fd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:14.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:14.763+0000 7f8663200640 1 -- 172.21.15.16:0/2091002928 >> v2:172.21.15.89:3300/0 conn(0x7f866406f920 msgr2=0x7f866406fd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:05:14.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:14.763+0000 7f8663200640 1 --2- 172.21.15.16:0/2091002928 >> v2:172.21.15.89:3300/0 conn(0x7f866406f920 0x7f866406fd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:14.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:14.763+0000 7f8663200640 1 -- 172.21.15.16:0/2091002928 >> v1:172.21.15.16:6789/0 conn(0x7f8664145e40 legacy=0x7f86640cc740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:14.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:14.763+0000 7f8663200640 1 -- 172.21.15.16:0/2091002928 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86640d0c00 con 0x7f86640d2010 2024-03-31T15:05:14.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:14.763+0000 7f8661e00640 1 --2- 172.21.15.16:0/2091002928 >> v2:172.21.15.89:3300/0 conn(0x7f866406f920 0x7f866406fd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:05:14.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:14.763+0000 7f8661400640 1 -- 172.21.15.16:0/2091002928 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 125744767 0 0) 0x7f86640d0920 con 0x7f8664145e40 2024-03-31T15:05:14.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:14.767+0000 7f8663200640 1 --2- 172.21.15.16:0/2091002928 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f86640d2010 0x7f86640d24a0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f8664073cf0 tx=0x7f865c01c930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ba619207ce04980 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:05:14.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:14.767+0000 7f8661400640 1 -- 172.21.15.16:0/2091002928 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f865c0b2070 con 0x7f86640d2010 2024-03-31T15:05:14.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:14.767+0000 7f8661400640 1 -- 172.21.15.16:0/2091002928 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f865c0a58e0 con 0x7f86640d2010 2024-03-31T15:05:14.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:14.767+0000 7f8661400640 1 -- 172.21.15.16:0/2091002928 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f865c0a5cb0 con 0x7f86640d2010 2024-03-31T15:05:14.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:14.767+0000 7f8669400640 1 -- 172.21.15.16:0/2091002928 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f86640d2010 msgr2=0x7f86640d24a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:14.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:14.767+0000 7f8669400640 1 --2- 172.21.15.16:0/2091002928 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f86640d2010 0x7f86640d24a0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f8664073cf0 tx=0x7f865c01c930 comp rx=0 tx=0).stop 2024-03-31T15:05:14.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:14.767+0000 7f8669400640 1 -- 172.21.15.16:0/2091002928 shutdown_connections 2024-03-31T15:05:14.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:14.767+0000 7f8669400640 1 --2- 172.21.15.16:0/2091002928 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f86640d2010 0x7f86640d24a0 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:14.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:14.767+0000 7f8669400640 1 --2- 172.21.15.16:0/2091002928 >> v2:172.21.15.89:3300/0 conn(0x7f866406f920 0x7f866406fd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:14.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:14.767+0000 7f8669400640 1 -- 172.21.15.16:0/2091002928 >> 172.21.15.16:0/2091002928 conn(0x7f8664067330 msgr2=0x7f8664067740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:05:14.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:14.767+0000 7f8669400640 1 -- 172.21.15.16:0/2091002928 shutdown_connections 2024-03-31T15:05:14.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:14.767+0000 7f8669400640 1 -- 172.21.15.16:0/2091002928 wait complete. 2024-03-31T15:05:14.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:14.771+0000 7f8669400640 1 Processor -- start 2024-03-31T15:05:14.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:14.771+0000 7f8669400640 1 -- start start 2024-03-31T15:05:14.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:14.771+0000 7f8669400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f866406f920 0x7f866413ef90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:14.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:14.771+0000 7f8669400640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f86640d2010 0x7f86641450b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:14.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:14.771+0000 7f8669400640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8664145e40 0x7f86640b6820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:14.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:14.771+0000 7f8669400640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f86640d0ee0 con 0x7f866406f920 2024-03-31T15:05:14.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:14.771+0000 7f8669400640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f86640ce9e0 con 0x7f8664145e40 2024-03-31T15:05:14.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:14.771+0000 7f8669400640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f86640d2df0 con 0x7f86640d2010 2024-03-31T15:05:14.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:14.771+0000 7f8662800640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f866406f920 0x7f866413ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:14.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:14.771+0000 7f8662800640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f866406f920 0x7f866413ef90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34950/0 (socket says 172.21.15.16:34950) 2024-03-31T15:05:14.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:14.771+0000 7f8662800640 1 -- 172.21.15.16:0/1740619943 learned_addr learned my addr 172.21.15.16:0/1740619943 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:05:14.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:14.771+0000 7f8663200640 1 --2- 172.21.15.16:0/1740619943 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8664145e40 0x7f86640b6820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:14.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:14.771+0000 7f8661e00640 1 --2- 172.21.15.16:0/1740619943 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f86640d2010 0x7f86641450b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:14.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:14.771+0000 7f8662800640 1 -- 172.21.15.16:0/1740619943 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f86640d2010 msgr2=0x7f86641450b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:14.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:14.771+0000 7f8662800640 1 --2- 172.21.15.16:0/1740619943 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f86640d2010 0x7f86641450b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:14.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:14.771+0000 7f8662800640 1 -- 172.21.15.16:0/1740619943 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8664145e40 msgr2=0x7f86640b6820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:14.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:14.771+0000 7f8662800640 1 --2- 172.21.15.16:0/1740619943 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8664145e40 0x7f86640b6820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:14.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:14.771+0000 7f8662800640 1 -- 172.21.15.16:0/1740619943 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f866415cc90 con 0x7f866406f920 2024-03-31T15:05:14.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:14.775+0000 7f8662800640 1 --2- 172.21.15.16:0/1740619943 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f866406f920 0x7f866413ef90 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f865400c480 tx=0x7f865400c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:05:14.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:14.775+0000 7f865aa00640 1 -- 172.21.15.16:0/1740619943 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8654018070 con 0x7f866406f920 2024-03-31T15:05:14.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:14.775+0000 7f865aa00640 1 -- 172.21.15.16:0/1740619943 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8654007500 con 0x7f866406f920 2024-03-31T15:05:14.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:14.775+0000 7f865aa00640 1 -- 172.21.15.16:0/1740619943 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8654004a30 con 0x7f866406f920 2024-03-31T15:05:14.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:14.775+0000 7f8669400640 1 -- 172.21.15.16:0/1740619943 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f866415cf70 con 0x7f866406f920 2024-03-31T15:05:14.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:14.775+0000 7f8669400640 1 -- 172.21.15.16:0/1740619943 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f866415d530 con 0x7f866406f920 2024-03-31T15:05:14.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:14.775+0000 7f8669400640 1 -- 172.21.15.16:0/1740619943 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8664069950 con 0x7f866406f920 2024-03-31T15:05:14.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:14.775+0000 7f865aa00640 1 -- 172.21.15.16:0/1740619943 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f8654004bd0 con 0x7f866406f920 2024-03-31T15:05:14.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:14.775+0000 7f865aa00640 1 --2- 172.21.15.16:0/1740619943 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f86440683b0 0x7f864406a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:14.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:14.775+0000 7f865aa00640 1 -- 172.21.15.16:0/1740619943 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5709+0+0 (secure 0 0 0) 0x7f8654085ba0 con 0x7f866406f920 2024-03-31T15:05:14.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:14.775+0000 7f8661e00640 1 --2- 172.21.15.16:0/1740619943 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f86440683b0 0x7f864406a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:14.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:14.779+0000 7f8661e00640 1 --2- 172.21.15.16:0/1740619943 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f86440683b0 0x7f864406a870 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f866406fea0 tx=0x7f8650008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:05:14.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:14.779+0000 7f865aa00640 1 -- 172.21.15.16:0/1740619943 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f86540561f0 con 0x7f866406f920 2024-03-31T15:05:14.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:14.995+0000 7f8669400640 1 -- 172.21.15.16:0/1740619943 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f86640cf3c0 con 0x7f866406f920 2024-03-31T15:05:14.991 INFO:tasks.ceph.mon.a.smithi016.stderr:2024-03-31T15:05:14.995+0000 7f7dcbc00640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-31T15:05:14.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:14.999+0000 7f865aa00640 1 -- 172.21.15.16:0/1740619943 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v17) ==== 161+0+0 (secure 0 0 0) 0x7f8654055af0 con 0x7f866406f920 2024-03-31T15:05:14.993 INFO:teuthology.orchestra.run.smithi016.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-31T15:05:14.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:14.999+0000 7f8669400640 1 -- 172.21.15.16:0/1740619943 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f86440683b0 msgr2=0x7f864406a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:14.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:14.999+0000 7f8669400640 1 --2- 172.21.15.16:0/1740619943 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f86440683b0 0x7f864406a870 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f866406fea0 tx=0x7f8650008040 comp rx=0 tx=0).stop 2024-03-31T15:05:14.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:14.999+0000 7f8669400640 1 -- 172.21.15.16:0/1740619943 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f866406f920 msgr2=0x7f866413ef90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:14.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:14.999+0000 7f8669400640 1 --2- 172.21.15.16:0/1740619943 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f866406f920 0x7f866413ef90 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f865400c480 tx=0x7f865400c940 comp rx=0 tx=0).stop 2024-03-31T15:05:14.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:14.999+0000 7f8669400640 1 -- 172.21.15.16:0/1740619943 shutdown_connections 2024-03-31T15:05:14.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:14.999+0000 7f8669400640 1 --2- 172.21.15.16:0/1740619943 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f86440683b0 0x7f864406a870 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:14.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:14.999+0000 7f8669400640 1 --2- 172.21.15.16:0/1740619943 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8664145e40 0x7f86640b6820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:14.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:14.999+0000 7f8669400640 1 --2- 172.21.15.16:0/1740619943 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f86640d2010 0x7f86641450b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:14.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:14.999+0000 7f8669400640 1 --2- 172.21.15.16:0/1740619943 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f866406f920 0x7f866413ef90 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:14.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:14.999+0000 7f8669400640 1 -- 172.21.15.16:0/1740619943 >> 172.21.15.16:0/1740619943 conn(0x7f8664067330 msgr2=0x7f86640b1d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:05:14.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:14.999+0000 7f8669400640 1 -- 172.21.15.16:0/1740619943 shutdown_connections 2024-03-31T15:05:14.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:14.999+0000 7f8669400640 1 -- 172.21.15.16:0/1740619943 wait complete. 2024-03-31T15:05:15.007 INFO:tasks.ceph.mds.c.smithi016.stderr: -6003> 2024-03-31T15:04:44.348+0000 7fe1e7c00640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-31T15:05:15.023 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-31T15:05:15.058 INFO:tasks.ceph.mds.c.smithi016.stdout:starting mds.c at 2024-03-31T15:05:15.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:15.171+0000 7f2c34e00640 1 Processor -- start 2024-03-31T15:05:15.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:15.171+0000 7f2c34e00640 1 -- start start 2024-03-31T15:05:15.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:15.175+0000 7f2c34e00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f2c301411b0 0x7f2c3006f920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:15.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:15.175+0000 7f2c34e00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f2c30144340 0x7f2c3006fe60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:15.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:15.175+0000 7f2c34e00640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2c30144980 con 0x7f2c30141590 2024-03-31T15:05:15.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:15.175+0000 7f2c34e00640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f2c3015f910 con 0x7f2c301411b0 2024-03-31T15:05:15.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:15.175+0000 7f2c34e00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f2c30162910 con 0x7f2c30144340 2024-03-31T15:05:15.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:15.175+0000 7f2c2de00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f2c301411b0 0x7f2c3006f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:15.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:15.175+0000 7f2c2e800640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f2c30144340 0x7f2c3006fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:15.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:15.175+0000 7f2c25400640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f2c30141590 0x7f2c30143c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.16:47010/0 (socket says 172.21.15.16:47010) 2024-03-31T15:05:15.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:15.175+0000 7f2c25400640 1 -- 172.21.15.16:0/420574762 learned_addr learned my addr 172.21.15.16:0/420574762 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:05:15.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:15.175+0000 7f2c2de00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f2c301411b0 0x7f2c3006f920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.16:57074/0 (socket says 172.21.15.16:57074) 2024-03-31T15:05:15.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:15.175+0000 7f2c2de00640 1 -- 172.21.15.16:0/420574762 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f2c30144340 msgr2=0x7f2c3006fe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:05:15.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:15.175+0000 7f2c2de00640 1 --2- 172.21.15.16:0/420574762 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f2c30144340 0x7f2c3006fe60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:15.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:15.175+0000 7f2c2de00640 1 -- 172.21.15.16:0/420574762 >> v1:172.21.15.89:6789/0 conn(0x7f2c30141590 legacy=0x7f2c30143c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:15.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:15.175+0000 7f2c2de00640 1 -- 172.21.15.16:0/420574762 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c30161fe0 con 0x7f2c301411b0 2024-03-31T15:05:15.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:15.179+0000 7f2c2de00640 1 --2- 172.21.15.16:0/420574762 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2c301411b0 0x7f2c3006f920 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f2c200097c0 tx=0x7f2c2001c4a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=222735236f86eb72 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:05:15.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:15.179+0000 7f2c2d400640 1 -- 172.21.15.16:0/420574762 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c200b2070 con 0x7f2c301411b0 2024-03-31T15:05:15.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:15.179+0000 7f2c2d400640 1 -- 172.21.15.16:0/420574762 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2c200ad640 con 0x7f2c301411b0 2024-03-31T15:05:15.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:15.179+0000 7f2c2d400640 1 -- 172.21.15.16:0/420574762 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c200ada10 con 0x7f2c301411b0 2024-03-31T15:05:15.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:15.179+0000 7f2c34e00640 1 -- 172.21.15.16:0/420574762 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2c301411b0 msgr2=0x7f2c3006f920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:15.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:15.179+0000 7f2c34e00640 1 --2- 172.21.15.16:0/420574762 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2c301411b0 0x7f2c3006f920 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f2c200097c0 tx=0x7f2c2001c4a0 comp rx=0 tx=0).stop 2024-03-31T15:05:15.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:15.179+0000 7f2c34e00640 1 -- 172.21.15.16:0/420574762 shutdown_connections 2024-03-31T15:05:15.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:15.179+0000 7f2c34e00640 1 --2- 172.21.15.16:0/420574762 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f2c30144340 0x7f2c3006fe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:15.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:15.179+0000 7f2c34e00640 1 --2- 172.21.15.16:0/420574762 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2c301411b0 0x7f2c3006f920 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:15.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:15.179+0000 7f2c34e00640 1 -- 172.21.15.16:0/420574762 >> 172.21.15.16:0/420574762 conn(0x7f2c30067930 msgr2=0x7f2c300b3ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:05:15.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:15.179+0000 7f2c34e00640 1 -- 172.21.15.16:0/420574762 shutdown_connections 2024-03-31T15:05:15.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:15.179+0000 7f2c34e00640 1 -- 172.21.15.16:0/420574762 wait complete. 2024-03-31T15:05:15.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:15.179+0000 7f2c34e00640 1 Processor -- start 2024-03-31T15:05:15.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:15.179+0000 7f2c34e00640 1 -- start start 2024-03-31T15:05:15.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:15.183+0000 7f2c34e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2c301411b0 0x7f2c30156aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:15.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:15.183+0000 7f2c34e00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f2c30141590 0x7f2c30160870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:15.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:15.183+0000 7f2c34e00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2c30144340 0x7f2c30160db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:15.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:15.183+0000 7f2c34e00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f2c3015f290 con 0x7f2c301411b0 2024-03-31T15:05:15.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:15.183+0000 7f2c34e00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f2c3015f110 con 0x7f2c30144340 2024-03-31T15:05:15.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:15.183+0000 7f2c34e00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f2c3015f410 con 0x7f2c30141590 2024-03-31T15:05:15.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:15.183+0000 7f2c2de00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2c301411b0 0x7f2c30156aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:15.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:15.183+0000 7f2c2de00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2c301411b0 0x7f2c30156aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34966/0 (socket says 172.21.15.16:34966) 2024-03-31T15:05:15.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:15.183+0000 7f2c2de00640 1 -- 172.21.15.16:0/4277908117 learned_addr learned my addr 172.21.15.16:0/4277908117 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:05:15.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:15.183+0000 7f2c2de00640 1 -- 172.21.15.16:0/4277908117 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f2c30141590 msgr2=0x7f2c30160870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:15.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:15.183+0000 7f2c2de00640 1 --2- 172.21.15.16:0/4277908117 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f2c30141590 0x7f2c30160870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:15.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:15.183+0000 7f2c2de00640 1 -- 172.21.15.16:0/4277908117 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2c30144340 msgr2=0x7f2c30160db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:15.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:15.183+0000 7f2c2de00640 1 --2- 172.21.15.16:0/4277908117 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2c30144340 0x7f2c30160db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:15.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:15.183+0000 7f2c2de00640 1 -- 172.21.15.16:0/4277908117 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c30142660 con 0x7f2c301411b0 2024-03-31T15:05:15.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:15.183+0000 7f2c2de00640 1 --2- 172.21.15.16:0/4277908117 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2c301411b0 0x7f2c30156aa0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f2c200097c0 tx=0x7f2c20019240 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:05:15.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:15.183+0000 7f2c26a00640 1 -- 172.21.15.16:0/4277908117 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c200ba020 con 0x7f2c301411b0 2024-03-31T15:05:15.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:15.183+0000 7f2c26a00640 1 -- 172.21.15.16:0/4277908117 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2c200b6410 con 0x7f2c301411b0 2024-03-31T15:05:15.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:15.183+0000 7f2c34e00640 1 -- 172.21.15.16:0/4277908117 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2c301f7890 con 0x7f2c301411b0 2024-03-31T15:05:15.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:15.183+0000 7f2c26a00640 1 -- 172.21.15.16:0/4277908117 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c200b2040 con 0x7f2c301411b0 2024-03-31T15:05:15.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:15.183+0000 7f2c34e00640 1 -- 172.21.15.16:0/4277908117 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2c301f7e50 con 0x7f2c301411b0 2024-03-31T15:05:15.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:15.183+0000 7f2c34e00640 1 -- 172.21.15.16:0/4277908117 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2c30069950 con 0x7f2c301411b0 2024-03-31T15:05:15.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:15.183+0000 7f2c26a00640 1 -- 172.21.15.16:0/4277908117 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f2c200bd020 con 0x7f2c301411b0 2024-03-31T15:05:15.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:15.187+0000 7f2c26a00640 1 --2- 172.21.15.16:0/4277908117 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f2c0c068380 0x7f2c0c06a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:15.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:15.187+0000 7f2c26a00640 1 -- 172.21.15.16:0/4277908117 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 5795+0+0 (secure 0 0 0) 0x7f2c20120830 con 0x7f2c301411b0 2024-03-31T15:05:15.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:15.187+0000 7f2c25400640 1 --2- 172.21.15.16:0/4277908117 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f2c0c068380 0x7f2c0c06a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:15.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:15.187+0000 7f2c25400640 1 --2- 172.21.15.16:0/4277908117 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f2c0c068380 0x7f2c0c06a840 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f2c180012f0 tx=0x7f2c18001640 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:05:15.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:15.191+0000 7f2c26a00640 1 -- 172.21.15.16:0/4277908117 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f2c200c4030 con 0x7f2c301411b0 2024-03-31T15:05:15.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:15.391+0000 7f2c34e00640 1 -- 172.21.15.16:0/4277908117 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7f2c300b4a20 con 0x7f2c301411b0 2024-03-31T15:05:15.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:15.999+0000 7f2c26a00640 1 -- 172.21.15.16:0/4277908117 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v18) ==== 104+0+0 (secure 0 0 0) 0x7f2c20016090 con 0x7f2c301411b0 2024-03-31T15:05:15.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:16.003+0000 7f2c34e00640 1 -- 172.21.15.16:0/4277908117 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f2c0c068380 msgr2=0x7f2c0c06a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:15.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:16.003+0000 7f2c34e00640 1 --2- 172.21.15.16:0/4277908117 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f2c0c068380 0x7f2c0c06a840 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f2c180012f0 tx=0x7f2c18001640 comp rx=0 tx=0).stop 2024-03-31T15:05:15.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:16.003+0000 7f2c34e00640 1 -- 172.21.15.16:0/4277908117 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2c301411b0 msgr2=0x7f2c30156aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:15.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:16.003+0000 7f2c34e00640 1 --2- 172.21.15.16:0/4277908117 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2c301411b0 0x7f2c30156aa0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f2c200097c0 tx=0x7f2c20019240 comp rx=0 tx=0).stop 2024-03-31T15:05:15.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:16.003+0000 7f2c34e00640 1 -- 172.21.15.16:0/4277908117 shutdown_connections 2024-03-31T15:05:15.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:16.003+0000 7f2c34e00640 1 --2- 172.21.15.16:0/4277908117 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f2c0c068380 0x7f2c0c06a840 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:15.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:16.003+0000 7f2c34e00640 1 --2- 172.21.15.16:0/4277908117 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2c30144340 0x7f2c30160db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:15.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:16.003+0000 7f2c34e00640 1 --2- 172.21.15.16:0/4277908117 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f2c30141590 0x7f2c30160870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:15.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:16.003+0000 7f2c34e00640 1 --2- 172.21.15.16:0/4277908117 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2c301411b0 0x7f2c30156aa0 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:15.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:16.003+0000 7f2c34e00640 1 -- 172.21.15.16:0/4277908117 >> 172.21.15.16:0/4277908117 conn(0x7f2c30067930 msgr2=0x7f2c30155d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:05:15.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:16.003+0000 7f2c34e00640 1 -- 172.21.15.16:0/4277908117 shutdown_connections 2024-03-31T15:05:15.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:16.003+0000 7f2c34e00640 1 -- 172.21.15.16:0/4277908117 wait complete. 2024-03-31T15:05:16.022 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-31T15:05:16.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:16.219+0000 7fd9fbe00640 1 Processor -- start 2024-03-31T15:05:16.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:16.223+0000 7fd9fbe00640 1 -- start start 2024-03-31T15:05:16.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:16.223+0000 7fd9fbe00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fd9f4143f40 0x7fd9f40730c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:16.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:16.223+0000 7fd9fbe00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fd9f40b45d0 0x7fd9f40b1be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:16.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:16.223+0000 7fd9fbe00640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd9f4144320 con 0x7fd9f40b2120 2024-03-31T15:05:16.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:16.223+0000 7fd9fbe00640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7fd9f40b2570 con 0x7fd9f4143f40 2024-03-31T15:05:16.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:16.223+0000 7fd9fbe00640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7fd9f40b4a30 con 0x7fd9f40b45d0 2024-03-31T15:05:16.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:16.223+0000 7fd9f9400640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fd9f4143f40 0x7fd9f40730c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:16.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:16.223+0000 7fd9f9e00640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fd9f40b2120 0x7fd9f4071490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:36504/0 (socket says 172.21.15.16:36504) 2024-03-31T15:05:16.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:16.223+0000 7fd9f9e00640 1 -- 172.21.15.16:0/1924312484 learned_addr learned my addr 172.21.15.16:0/1924312484 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:05:16.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:16.223+0000 7fd9f8a00640 1 --2- 172.21.15.16:0/1924312484 >> v2:172.21.15.89:3300/0 conn(0x7fd9f40b45d0 0x7fd9f40b1be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:16.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:16.223+0000 7fd9f9400640 1 -- 172.21.15.16:0/1924312484 >> v2:172.21.15.89:3300/0 conn(0x7fd9f40b45d0 msgr2=0x7fd9f40b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:05:16.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:16.223+0000 7fd9f9400640 1 --2- 172.21.15.16:0/1924312484 >> v2:172.21.15.89:3300/0 conn(0x7fd9f40b45d0 0x7fd9f40b1be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:16.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:16.223+0000 7fd9f9400640 1 -- 172.21.15.16:0/1924312484 >> v1:172.21.15.16:6789/0 conn(0x7fd9f40b2120 legacy=0x7fd9f4071490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:16.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:16.223+0000 7fd9f9400640 1 -- 172.21.15.16:0/1924312484 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9f415ef80 con 0x7fd9f4143f40 2024-03-31T15:05:16.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:16.223+0000 7fd9f8a00640 1 --2- 172.21.15.16:0/1924312484 >> v2:172.21.15.89:3300/0 conn(0x7fd9f40b45d0 0x7fd9f40b1be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:05:16.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:16.227+0000 7fd9f9400640 1 --2- 172.21.15.16:0/1924312484 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd9f4143f40 0x7fd9f40730c0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7fd9f41450e0 tx=0x7fd9e801eb50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e227a44da890782 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:05:16.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:16.227+0000 7fd9f3e00640 1 -- 172.21.15.16:0/1924312484 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9e80b4070 con 0x7fd9f4143f40 2024-03-31T15:05:16.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:16.227+0000 7fd9f3e00640 1 -- 172.21.15.16:0/1924312484 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd9e80a7940 con 0x7fd9f4143f40 2024-03-31T15:05:16.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:16.227+0000 7fd9f3e00640 1 -- 172.21.15.16:0/1924312484 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9e80a7d10 con 0x7fd9f4143f40 2024-03-31T15:05:16.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:16.227+0000 7fd9fbe00640 1 -- 172.21.15.16:0/1924312484 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd9f4143f40 msgr2=0x7fd9f40730c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:16.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:16.227+0000 7fd9fbe00640 1 --2- 172.21.15.16:0/1924312484 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd9f4143f40 0x7fd9f40730c0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7fd9f41450e0 tx=0x7fd9e801eb50 comp rx=0 tx=0).stop 2024-03-31T15:05:16.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:16.227+0000 7fd9fbe00640 1 -- 172.21.15.16:0/1924312484 shutdown_connections 2024-03-31T15:05:16.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:16.227+0000 7fd9fbe00640 1 --2- 172.21.15.16:0/1924312484 >> v2:172.21.15.89:3300/0 conn(0x7fd9f40b45d0 0x7fd9f40b1be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:16.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:16.227+0000 7fd9fbe00640 1 --2- 172.21.15.16:0/1924312484 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd9f4143f40 0x7fd9f40730c0 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:16.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:16.227+0000 7fd9fbe00640 1 -- 172.21.15.16:0/1924312484 >> 172.21.15.16:0/1924312484 conn(0x7fd9f406b620 msgr2=0x7fd9f406ba30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:05:16.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:16.227+0000 7fd9fbe00640 1 -- 172.21.15.16:0/1924312484 shutdown_connections 2024-03-31T15:05:16.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:16.227+0000 7fd9fbe00640 1 -- 172.21.15.16:0/1924312484 wait complete. 2024-03-31T15:05:16.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:16.227+0000 7fd9fbe00640 1 Processor -- start 2024-03-31T15:05:16.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:16.227+0000 7fd9fbe00640 1 -- start start 2024-03-31T15:05:16.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:16.231+0000 7fd9fbe00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd9f40b2120 0x7fd9f415fe60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:16.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:16.231+0000 7fd9fbe00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fd9f40b45d0 0x7fd9f4140700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:16.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:16.231+0000 7fd9fbe00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd9f4143f40 0x7fd9f415d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:16.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:16.231+0000 7fd9fbe00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fd9f40cd910 con 0x7fd9f4143f40 2024-03-31T15:05:16.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:16.231+0000 7fd9fbe00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fd9f40cd790 con 0x7fd9f40b2120 2024-03-31T15:05:16.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:16.231+0000 7fd9fbe00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fd9f40cfea0 con 0x7fd9f40b45d0 2024-03-31T15:05:16.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:16.231+0000 7fd9f9e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd9f4143f40 0x7fd9f415d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:16.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:16.231+0000 7fd9f9e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd9f4143f40 0x7fd9f415d3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34982/0 (socket says 172.21.15.16:34982) 2024-03-31T15:05:16.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:16.231+0000 7fd9f9e00640 1 -- 172.21.15.16:0/2453854174 learned_addr learned my addr 172.21.15.16:0/2453854174 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:05:16.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:16.231+0000 7fd9f8a00640 1 --2- 172.21.15.16:0/2453854174 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fd9f40b45d0 0x7fd9f4140700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:16.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:16.231+0000 7fd9f9400640 1 --2- 172.21.15.16:0/2453854174 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd9f40b2120 0x7fd9f415fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:16.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:16.231+0000 7fd9f9e00640 1 -- 172.21.15.16:0/2453854174 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fd9f40b45d0 msgr2=0x7fd9f4140700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:16.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:16.231+0000 7fd9f9e00640 1 --2- 172.21.15.16:0/2453854174 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fd9f40b45d0 0x7fd9f4140700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:16.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:16.231+0000 7fd9f9e00640 1 -- 172.21.15.16:0/2453854174 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd9f40b2120 msgr2=0x7fd9f415fe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:16.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:16.231+0000 7fd9f9e00640 1 --2- 172.21.15.16:0/2453854174 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd9f40b2120 0x7fd9f415fe60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:16.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:16.231+0000 7fd9f9e00640 1 -- 172.21.15.16:0/2453854174 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9f4140c40 con 0x7fd9f4143f40 2024-03-31T15:05:16.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:16.231+0000 7fd9f8a00640 1 --2- 172.21.15.16:0/2453854174 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fd9f40b45d0 0x7fd9f4140700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:05:16.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:16.231+0000 7fd9f9400640 1 --2- 172.21.15.16:0/2453854174 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd9f40b2120 0x7fd9f415fe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:05:16.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:16.231+0000 7fd9f9e00640 1 --2- 172.21.15.16:0/2453854174 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd9f4143f40 0x7fd9f415d3c0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7fd9ec00cbf0 tx=0x7fd9ec00a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:05:16.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:16.231+0000 7fd9f1600640 1 -- 172.21.15.16:0/2453854174 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9ec017070 con 0x7fd9f4143f40 2024-03-31T15:05:16.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:16.231+0000 7fd9f1600640 1 -- 172.21.15.16:0/2453854174 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd9ec00ad40 con 0x7fd9f4143f40 2024-03-31T15:05:16.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:16.231+0000 7fd9fbe00640 1 -- 172.21.15.16:0/2453854174 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd9f412d7f0 con 0x7fd9f4143f40 2024-03-31T15:05:16.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:16.231+0000 7fd9f1600640 1 -- 172.21.15.16:0/2453854174 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9ec012600 con 0x7fd9f4143f40 2024-03-31T15:05:16.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:16.231+0000 7fd9fbe00640 1 -- 172.21.15.16:0/2453854174 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd9f412ddb0 con 0x7fd9f4143f40 2024-03-31T15:05:16.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:16.231+0000 7fd9fbe00640 1 -- 172.21.15.16:0/2453854174 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd9f4069950 con 0x7fd9f4143f40 2024-03-31T15:05:16.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:16.231+0000 7fd9f1600640 1 -- 172.21.15.16:0/2453854174 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fd9ec01a070 con 0x7fd9f4143f40 2024-03-31T15:05:16.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:16.235+0000 7fd9f1600640 1 --2- 172.21.15.16:0/2453854174 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fd9d4068380 0x7fd9d406a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:16.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:16.235+0000 7fd9f1600640 1 -- 172.21.15.16:0/2453854174 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 5795+0+0 (secure 0 0 0) 0x7fd9ec086590 con 0x7fd9f4143f40 2024-03-31T15:05:16.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:16.235+0000 7fd9f9400640 1 --2- 172.21.15.16:0/2453854174 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fd9d4068380 0x7fd9d406a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:16.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:16.235+0000 7fd9f9400640 1 --2- 172.21.15.16:0/2453854174 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fd9d4068380 0x7fd9d406a840 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fd9e80b1000 tx=0x7fd9e8008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:05:16.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:16.239+0000 7fd9f1600640 1 -- 172.21.15.16:0/2453854174 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fd9ec056530 con 0x7fd9f4143f40 2024-03-31T15:05:16.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:16.403+0000 7fd9fbe00640 1 -- 172.21.15.16:0/2453854174 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7fd9f4143770 con 0x7fd9f4143f40 2024-03-31T15:05:16.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:17.003+0000 7fd9f1600640 1 -- 172.21.15.16:0/2453854174 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v36) ==== 181+0+0 (secure 0 0 0) 0x7fd9ec05a830 con 0x7fd9f4143f40 2024-03-31T15:05:16.997 INFO:teuthology.orchestra.run.smithi016.stderr:pool 'cephfs_metadata' removed 2024-03-31T15:05:17.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:17.003+0000 7fd9fbe00640 1 -- 172.21.15.16:0/2453854174 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fd9d4068380 msgr2=0x7fd9d406a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:17.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:17.003+0000 7fd9fbe00640 1 --2- 172.21.15.16:0/2453854174 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fd9d4068380 0x7fd9d406a840 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fd9e80b1000 tx=0x7fd9e8008040 comp rx=0 tx=0).stop 2024-03-31T15:05:17.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:17.003+0000 7fd9fbe00640 1 -- 172.21.15.16:0/2453854174 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd9f4143f40 msgr2=0x7fd9f415d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:17.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:17.003+0000 7fd9fbe00640 1 --2- 172.21.15.16:0/2453854174 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd9f4143f40 0x7fd9f415d3c0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7fd9ec00cbf0 tx=0x7fd9ec00a430 comp rx=0 tx=0).stop 2024-03-31T15:05:17.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:17.003+0000 7fd9fbe00640 1 -- 172.21.15.16:0/2453854174 shutdown_connections 2024-03-31T15:05:17.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:17.003+0000 7fd9fbe00640 1 --2- 172.21.15.16:0/2453854174 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fd9d4068380 0x7fd9d406a840 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:17.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:17.003+0000 7fd9fbe00640 1 --2- 172.21.15.16:0/2453854174 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd9f4143f40 0x7fd9f415d3c0 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:17.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:17.003+0000 7fd9fbe00640 1 --2- 172.21.15.16:0/2453854174 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fd9f40b45d0 0x7fd9f4140700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:17.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:17.003+0000 7fd9fbe00640 1 --2- 172.21.15.16:0/2453854174 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd9f40b2120 0x7fd9f415fe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:17.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:17.003+0000 7fd9fbe00640 1 -- 172.21.15.16:0/2453854174 >> 172.21.15.16:0/2453854174 conn(0x7fd9f406b620 msgr2=0x7fd9f40ae720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:05:17.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:17.003+0000 7fd9fbe00640 1 -- 172.21.15.16:0/2453854174 shutdown_connections 2024-03-31T15:05:17.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:17.003+0000 7fd9fbe00640 1 -- 172.21.15.16:0/2453854174 wait complete. 2024-03-31T15:05:17.027 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-31T15:05:17.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:17.179+0000 7fc73de00640 1 Processor -- start 2024-03-31T15:05:17.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:17.179+0000 7fc73de00640 1 -- start start 2024-03-31T15:05:17.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:17.179+0000 7fc73de00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fc738147230 0x7fc7380b3dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:17.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:17.179+0000 7fc73de00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fc7380d1040 0x7fc7380cb4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:17.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:17.179+0000 7fc73de00640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc7380d14b0 con 0x7fc7380b4300 2024-03-31T15:05:17.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:17.179+0000 7fc73de00640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7fc7380b4760 con 0x7fc7380d1040 2024-03-31T15:05:17.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:17.179+0000 7fc73de00640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7fc7380d2c30 con 0x7fc738147230 2024-03-31T15:05:17.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:17.183+0000 7fc736000640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fc7380b4300 0x7fc7380d0930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:55498/0 (socket says 172.21.15.16:55498) 2024-03-31T15:05:17.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:17.183+0000 7fc736000640 1 -- 172.21.15.16:0/4121490409 learned_addr learned my addr 172.21.15.16:0/4121490409 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:05:17.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:17.183+0000 7fc737400640 1 --2- 172.21.15.16:0/4121490409 >> v2:172.21.15.16:3300/0 conn(0x7fc7380d1040 0x7fc7380cb4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:17.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:17.183+0000 7fc736a00640 1 --2- 172.21.15.16:0/4121490409 >> v2:172.21.15.89:3300/0 conn(0x7fc738147230 0x7fc7380b3dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:17.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:17.183+0000 7fc737400640 1 -- 172.21.15.16:0/4121490409 >> v2:172.21.15.89:3300/0 conn(0x7fc738147230 msgr2=0x7fc7380b3dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:05:17.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:17.183+0000 7fc737400640 1 --2- 172.21.15.16:0/4121490409 >> v2:172.21.15.89:3300/0 conn(0x7fc738147230 0x7fc7380b3dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:17.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:17.183+0000 7fc737400640 1 -- 172.21.15.16:0/4121490409 >> v1:172.21.15.16:6789/0 conn(0x7fc7380b4300 legacy=0x7fc7380d0930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:17.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:17.183+0000 7fc737400640 1 -- 172.21.15.16:0/4121490409 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc7380d2020 con 0x7fc7380d1040 2024-03-31T15:05:17.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:17.183+0000 7fc735600640 1 -- 172.21.15.16:0/4121490409 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3396607351 0 0) 0x7fc7380d14b0 con 0x7fc7380b4300 2024-03-31T15:05:17.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:17.183+0000 7fc736a00640 1 --2- 172.21.15.16:0/4121490409 >> v2:172.21.15.89:3300/0 conn(0x7fc738147230 0x7fc7380b3dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:05:17.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:17.183+0000 7fc737400640 1 --2- 172.21.15.16:0/4121490409 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc7380d1040 0x7fc7380cb4b0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7fc738067e00 tx=0x7fc73001c930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd630c73ee04cb94 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:05:17.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:17.183+0000 7fc735600640 1 -- 172.21.15.16:0/4121490409 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7300b2070 con 0x7fc7380d1040 2024-03-31T15:05:17.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:17.183+0000 7fc735600640 1 -- 172.21.15.16:0/4121490409 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc7300a58e0 con 0x7fc7380d1040 2024-03-31T15:05:17.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:17.183+0000 7fc735600640 1 -- 172.21.15.16:0/4121490409 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7300a5cb0 con 0x7fc7380d1040 2024-03-31T15:05:17.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:17.187+0000 7fc73de00640 1 -- 172.21.15.16:0/4121490409 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc7380d1040 msgr2=0x7fc7380cb4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:17.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:17.187+0000 7fc73de00640 1 --2- 172.21.15.16:0/4121490409 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc7380d1040 0x7fc7380cb4b0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7fc738067e00 tx=0x7fc73001c930 comp rx=0 tx=0).stop 2024-03-31T15:05:17.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:17.187+0000 7fc73de00640 1 -- 172.21.15.16:0/4121490409 shutdown_connections 2024-03-31T15:05:17.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:17.187+0000 7fc73de00640 1 --2- 172.21.15.16:0/4121490409 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc7380d1040 0x7fc7380cb4b0 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:17.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:17.187+0000 7fc73de00640 1 --2- 172.21.15.16:0/4121490409 >> v2:172.21.15.89:3300/0 conn(0x7fc738147230 0x7fc7380b3dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:17.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:17.187+0000 7fc73de00640 1 -- 172.21.15.16:0/4121490409 >> 172.21.15.16:0/4121490409 conn(0x7fc73806b7f0 msgr2=0x7fc738067330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:05:17.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:17.187+0000 7fc73de00640 1 -- 172.21.15.16:0/4121490409 shutdown_connections 2024-03-31T15:05:17.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:17.187+0000 7fc73de00640 1 -- 172.21.15.16:0/4121490409 wait complete. 2024-03-31T15:05:17.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:17.187+0000 7fc73de00640 1 Processor -- start 2024-03-31T15:05:17.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:17.187+0000 7fc73de00640 1 -- start start 2024-03-31T15:05:17.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:17.187+0000 7fc73de00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fc7380b4300 0x7fc7380b4ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:17.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:17.187+0000 7fc73de00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc7380d1040 0x7fc7381423e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:17.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:17.187+0000 7fc73de00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc738147230 0x7fc738142920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:17.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:17.187+0000 7fc73de00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fc7380cc200 con 0x7fc7380d1040 2024-03-31T15:05:17.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:17.187+0000 7fc73de00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fc73806cfd0 con 0x7fc738147230 2024-03-31T15:05:17.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:17.187+0000 7fc73de00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fc7380cc380 con 0x7fc7380b4300 2024-03-31T15:05:17.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:17.187+0000 7fc736000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc7380d1040 0x7fc7381423e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:17.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:17.187+0000 7fc736000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc7380d1040 0x7fc7381423e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49234/0 (socket says 172.21.15.16:49234) 2024-03-31T15:05:17.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:17.187+0000 7fc736000640 1 -- 172.21.15.16:0/2993083231 learned_addr learned my addr 172.21.15.16:0/2993083231 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:05:17.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:17.187+0000 7fc737400640 1 --2- 172.21.15.16:0/2993083231 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc738147230 0x7fc738142920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:17.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:17.187+0000 7fc736a00640 1 --2- 172.21.15.16:0/2993083231 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fc7380b4300 0x7fc7380b4ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:17.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:17.187+0000 7fc736000640 1 -- 172.21.15.16:0/2993083231 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fc7380b4300 msgr2=0x7fc7380b4ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:17.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:17.187+0000 7fc736000640 1 --2- 172.21.15.16:0/2993083231 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fc7380b4300 0x7fc7380b4ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:17.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:17.187+0000 7fc736000640 1 -- 172.21.15.16:0/2993083231 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc738147230 msgr2=0x7fc738142920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:17.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:17.187+0000 7fc736000640 1 --2- 172.21.15.16:0/2993083231 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc738147230 0x7fc738142920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:17.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:17.187+0000 7fc736000640 1 -- 172.21.15.16:0/2993083231 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc7380b7da0 con 0x7fc7380d1040 2024-03-31T15:05:17.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:17.187+0000 7fc736a00640 1 --2- 172.21.15.16:0/2993083231 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fc7380b4300 0x7fc7380b4ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:05:17.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:17.187+0000 7fc736000640 1 --2- 172.21.15.16:0/2993083231 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc7380d1040 0x7fc7381423e0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7fc72400c4b0 tx=0x7fc72400c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:05:17.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:17.191+0000 7fc72f400640 1 -- 172.21.15.16:0/2993083231 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc724018070 con 0x7fc7380d1040 2024-03-31T15:05:17.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:17.191+0000 7fc72f400640 1 -- 172.21.15.16:0/2993083231 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc724007500 con 0x7fc7380d1040 2024-03-31T15:05:17.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:17.191+0000 7fc72f400640 1 -- 172.21.15.16:0/2993083231 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc724004a30 con 0x7fc7380d1040 2024-03-31T15:05:17.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:17.191+0000 7fc73de00640 1 -- 172.21.15.16:0/2993083231 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc7380b63c0 con 0x7fc7380d1040 2024-03-31T15:05:17.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:17.191+0000 7fc73de00640 1 -- 172.21.15.16:0/2993083231 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc7380b6980 con 0x7fc7380d1040 2024-03-31T15:05:17.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:17.191+0000 7fc73de00640 1 -- 172.21.15.16:0/2993083231 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc7380b61f0 con 0x7fc7380d1040 2024-03-31T15:05:17.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:17.191+0000 7fc72f400640 1 -- 172.21.15.16:0/2993083231 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fc724004bd0 con 0x7fc7380d1040 2024-03-31T15:05:17.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:17.191+0000 7fc72f400640 1 --2- 172.21.15.16:0/2993083231 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fc718068380 0x7fc71806a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:17.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:17.191+0000 7fc72f400640 1 -- 172.21.15.16:0/2993083231 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 5357+0+0 (secure 0 0 0) 0x7fc724085a60 con 0x7fc7380d1040 2024-03-31T15:05:17.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:17.191+0000 7fc736a00640 1 --2- 172.21.15.16:0/2993083231 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fc718068380 0x7fc71806a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:17.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:17.195+0000 7fc736a00640 1 --2- 172.21.15.16:0/2993083231 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fc718068380 0x7fc71806a840 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fc728006fd0 tx=0x7fc728008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:05:17.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:17.195+0000 7fc72f400640 1 -- 172.21.15.16:0/2993083231 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fc724055a90 con 0x7fc7380d1040 2024-03-31T15:05:17.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:17.359+0000 7fc73de00640 1 -- 172.21.15.16:0/2993083231 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7fc738145ac0 con 0x7fc7380d1040 2024-03-31T15:05:18.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:18.003+0000 7fc72f400640 1 -- 172.21.15.16:0/2993083231 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v37) ==== 169+0+0 (secure 0 0 0) 0x7fc724059d90 con 0x7fc7380d1040 2024-03-31T15:05:18.001 INFO:teuthology.orchestra.run.smithi016.stderr:pool 'cephfs_data' removed 2024-03-31T15:05:18.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:18.007+0000 7fc73de00640 1 -- 172.21.15.16:0/2993083231 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fc718068380 msgr2=0x7fc71806a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:18.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:18.007+0000 7fc73de00640 1 --2- 172.21.15.16:0/2993083231 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fc718068380 0x7fc71806a840 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fc728006fd0 tx=0x7fc728008040 comp rx=0 tx=0).stop 2024-03-31T15:05:18.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:18.007+0000 7fc73de00640 1 -- 172.21.15.16:0/2993083231 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc7380d1040 msgr2=0x7fc7381423e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:18.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:18.007+0000 7fc73de00640 1 --2- 172.21.15.16:0/2993083231 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc7380d1040 0x7fc7381423e0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7fc72400c4b0 tx=0x7fc72400c970 comp rx=0 tx=0).stop 2024-03-31T15:05:18.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:18.007+0000 7fc73de00640 1 -- 172.21.15.16:0/2993083231 shutdown_connections 2024-03-31T15:05:18.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:18.007+0000 7fc73de00640 1 --2- 172.21.15.16:0/2993083231 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fc718068380 0x7fc71806a840 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:18.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:18.007+0000 7fc73de00640 1 --2- 172.21.15.16:0/2993083231 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc738147230 0x7fc738142920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:18.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:18.007+0000 7fc73de00640 1 --2- 172.21.15.16:0/2993083231 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc7380d1040 0x7fc7381423e0 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:18.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:18.007+0000 7fc73de00640 1 --2- 172.21.15.16:0/2993083231 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fc7380b4300 0x7fc7380b4ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:18.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:18.007+0000 7fc73de00640 1 -- 172.21.15.16:0/2993083231 >> 172.21.15.16:0/2993083231 conn(0x7fc73806b7f0 msgr2=0x7fc738140390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:05:18.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:18.007+0000 7fc73de00640 1 -- 172.21.15.16:0/2993083231 shutdown_connections 2024-03-31T15:05:18.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:18.007+0000 7fc73de00640 1 -- 172.21.15.16:0/2993083231 wait complete. 2024-03-31T15:05:18.029 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-03-31T15:05:18.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:18.175+0000 7fa7e6400640 1 Processor -- start 2024-03-31T15:05:18.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:18.175+0000 7fa7e6400640 1 -- start start 2024-03-31T15:05:18.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:18.179+0000 7fa7e6400640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fa7e013dff0 0x7fa7e006d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:18.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:18.179+0000 7fa7e6400640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fa7e013e3d0 0x7fa7e006da10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:18.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:18.179+0000 7fa7e6400640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa7e015fbc0 con 0x7fa7e00cc9d0 2024-03-31T15:05:18.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:18.179+0000 7fa7e6400640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7fa7e015f730 con 0x7fa7e013dff0 2024-03-31T15:05:18.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:18.179+0000 7fa7e6400640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fa7e015ee00 con 0x7fa7e013e3d0 2024-03-31T15:05:18.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:18.179+0000 7fa7df400640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fa7e013dff0 0x7fa7e006d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:18.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:18.179+0000 7fa7d7e00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fa7e013e3d0 0x7fa7e006da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:18.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:18.179+0000 7fa7dfe00640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fa7e00cc9d0 0x7fa7e00cce30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.16:40938/0 (socket says 172.21.15.16:40938) 2024-03-31T15:05:18.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:18.179+0000 7fa7d7e00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fa7e013e3d0 0x7fa7e006da10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3301/0 says I am v2:172.21.15.16:37500/0 (socket says 172.21.15.16:37500) 2024-03-31T15:05:18.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:18.179+0000 7fa7dfe00640 1 -- 172.21.15.16:0/3778824955 learned_addr learned my addr 172.21.15.16:0/3778824955 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:05:18.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:18.179+0000 7fa7d7e00640 1 -- 172.21.15.16:0/3778824955 >> v2:172.21.15.89:3300/0 conn(0x7fa7e013dff0 msgr2=0x7fa7e006d4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:05:18.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:18.179+0000 7fa7d7e00640 1 --2- 172.21.15.16:0/3778824955 >> v2:172.21.15.89:3300/0 conn(0x7fa7e013dff0 0x7fa7e006d4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:18.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:18.179+0000 7fa7d7e00640 1 -- 172.21.15.16:0/3778824955 >> v1:172.21.15.89:6789/0 conn(0x7fa7e00cc9d0 legacy=0x7fa7e00cce30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:18.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:18.179+0000 7fa7d7e00640 1 -- 172.21.15.16:0/3778824955 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7e015f0e0 con 0x7fa7e013e3d0 2024-03-31T15:05:18.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:18.183+0000 7fa7d7e00640 1 --2- 172.21.15.16:0/3778824955 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fa7e013e3d0 0x7fa7e006da10 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fa7c80097c0 tx=0x7fa7c801c410 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bedee2f6ba94d160 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:05:18.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:18.183+0000 7fa7dea00640 1 -- 172.21.15.16:0/3778824955 <== mon.2 v2:172.21.15.89:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7c80b0070 con 0x7fa7e013e3d0 2024-03-31T15:05:18.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:18.183+0000 7fa7dea00640 1 -- 172.21.15.16:0/3778824955 <== mon.2 v2:172.21.15.89:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa7c80ab600 con 0x7fa7e013e3d0 2024-03-31T15:05:18.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:18.183+0000 7fa7dea00640 1 -- 172.21.15.16:0/3778824955 <== mon.2 v2:172.21.15.89:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7c80ab9d0 con 0x7fa7e013e3d0 2024-03-31T15:05:18.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:18.183+0000 7fa7e6400640 1 -- 172.21.15.16:0/3778824955 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fa7e013e3d0 msgr2=0x7fa7e006da10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:18.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:18.183+0000 7fa7e6400640 1 --2- 172.21.15.16:0/3778824955 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fa7e013e3d0 0x7fa7e006da10 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fa7c80097c0 tx=0x7fa7c801c410 comp rx=0 tx=0).stop 2024-03-31T15:05:18.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:18.183+0000 7fa7e6400640 1 -- 172.21.15.16:0/3778824955 shutdown_connections 2024-03-31T15:05:18.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:18.183+0000 7fa7e6400640 1 --2- 172.21.15.16:0/3778824955 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fa7e013e3d0 0x7fa7e006da10 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:18.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:18.183+0000 7fa7e6400640 1 --2- 172.21.15.16:0/3778824955 >> v2:172.21.15.89:3300/0 conn(0x7fa7e013dff0 0x7fa7e006d4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:18.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:18.183+0000 7fa7e6400640 1 -- 172.21.15.16:0/3778824955 >> 172.21.15.16:0/3778824955 conn(0x7fa7e006ee40 msgr2=0x7fa7e006b620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:05:18.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:18.183+0000 7fa7e6400640 1 -- 172.21.15.16:0/3778824955 shutdown_connections 2024-03-31T15:05:18.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:18.183+0000 7fa7e6400640 1 -- 172.21.15.16:0/3778824955 wait complete. 2024-03-31T15:05:18.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:18.183+0000 7fa7e6400640 1 Processor -- start 2024-03-31T15:05:18.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:18.183+0000 7fa7e6400640 1 -- start start 2024-03-31T15:05:18.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:18.183+0000 7fa7e6400640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa7e00cc9d0 0x7fa7e01f5220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:18.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:18.183+0000 7fa7e6400640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fa7e013dff0 0x7fa7e012b710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:18.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:18.183+0000 7fa7e6400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa7e013e3d0 0x7fa7e013be70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:18.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:18.183+0000 7fa7e6400640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fa7e00cf790 con 0x7fa7e013e3d0 2024-03-31T15:05:18.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:18.183+0000 7fa7e6400640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fa7e00cf610 con 0x7fa7e00cc9d0 2024-03-31T15:05:18.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:18.183+0000 7fa7e6400640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fa7e00cf910 con 0x7fa7e013dff0 2024-03-31T15:05:18.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:18.187+0000 7fa7dfe00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa7e013e3d0 0x7fa7e013be70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:18.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:18.187+0000 7fa7dfe00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa7e013e3d0 0x7fa7e013be70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49248/0 (socket says 172.21.15.16:49248) 2024-03-31T15:05:18.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:18.187+0000 7fa7dfe00640 1 -- 172.21.15.16:0/1232962859 learned_addr learned my addr 172.21.15.16:0/1232962859 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:05:18.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:18.187+0000 7fa7d7e00640 1 --2- 172.21.15.16:0/1232962859 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fa7e013dff0 0x7fa7e012b710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:18.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:18.187+0000 7fa7df400640 1 --2- 172.21.15.16:0/1232962859 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa7e00cc9d0 0x7fa7e01f5220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:18.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:18.187+0000 7fa7dfe00640 1 -- 172.21.15.16:0/1232962859 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fa7e013dff0 msgr2=0x7fa7e012b710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:18.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:18.187+0000 7fa7dfe00640 1 --2- 172.21.15.16:0/1232962859 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fa7e013dff0 0x7fa7e012b710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:18.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:18.187+0000 7fa7dfe00640 1 -- 172.21.15.16:0/1232962859 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa7e00cc9d0 msgr2=0x7fa7e01f5220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:18.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:18.187+0000 7fa7dfe00640 1 --2- 172.21.15.16:0/1232962859 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa7e00cc9d0 0x7fa7e01f5220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:18.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:18.187+0000 7fa7dfe00640 1 -- 172.21.15.16:0/1232962859 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7e012bc50 con 0x7fa7e013e3d0 2024-03-31T15:05:18.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:18.187+0000 7fa7dfe00640 1 --2- 172.21.15.16:0/1232962859 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa7e013e3d0 0x7fa7e013be70 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7fa7d800cad0 tx=0x7fa7d800cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:05:18.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:18.187+0000 7fa7d7400640 1 -- 172.21.15.16:0/1232962859 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7d8017070 con 0x7fa7e013e3d0 2024-03-31T15:05:18.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:18.187+0000 7fa7d7400640 1 -- 172.21.15.16:0/1232962859 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa7d800ac30 con 0x7fa7e013e3d0 2024-03-31T15:05:18.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:18.187+0000 7fa7d7400640 1 -- 172.21.15.16:0/1232962859 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7d8012620 con 0x7fa7e013e3d0 2024-03-31T15:05:18.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:18.187+0000 7fa7e6400640 1 -- 172.21.15.16:0/1232962859 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa7e013a500 con 0x7fa7e013e3d0 2024-03-31T15:05:18.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:18.187+0000 7fa7e6400640 1 -- 172.21.15.16:0/1232962859 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa7e0153700 con 0x7fa7e013e3d0 2024-03-31T15:05:18.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:18.187+0000 7fa7e6400640 1 -- 172.21.15.16:0/1232962859 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa7e013aa00 con 0x7fa7e013e3d0 2024-03-31T15:05:18.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:18.187+0000 7fa7d7400640 1 -- 172.21.15.16:0/1232962859 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fa7d801a070 con 0x7fa7e013e3d0 2024-03-31T15:05:18.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:18.191+0000 7fa7d7400640 1 --2- 172.21.15.16:0/1232962859 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fa7bc068380 0x7fa7bc06a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:18.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:18.191+0000 7fa7d7400640 1 -- 172.21.15.16:0/1232962859 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 4943+0+0 (secure 0 0 0) 0x7fa7d80863a0 con 0x7fa7e013e3d0 2024-03-31T15:05:18.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:18.191+0000 7fa7df400640 1 --2- 172.21.15.16:0/1232962859 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fa7bc068380 0x7fa7bc06a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:18.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:18.191+0000 7fa7df400640 1 --2- 172.21.15.16:0/1232962859 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fa7bc068380 0x7fa7bc06a840 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fa7d0009b60 tx=0x7fa7d0009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:05:18.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:18.195+0000 7fa7d7400640 1 -- 172.21.15.16:0/1232962859 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fa7d800ae20 con 0x7fa7e013e3d0 2024-03-31T15:05:18.369 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:18.375+0000 7fa7e6400640 1 -- 172.21.15.16:0/1232962859 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) -- 0x7fa7e0069950 con 0x7fa7e013e3d0 2024-03-31T15:05:18.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:18.379+0000 7fa7d7400640 1 -- 172.21.15.16:0/1232962859 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v11) ==== 104+0+0 (secure 0 0 0) 0x7fa7d8056570 con 0x7fa7e013e3d0 2024-03-31T15:05:18.375 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:18.379+0000 7fa7e6400640 1 -- 172.21.15.16:0/1232962859 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fa7bc068380 msgr2=0x7fa7bc06a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:18.375 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:18.379+0000 7fa7e6400640 1 --2- 172.21.15.16:0/1232962859 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fa7bc068380 0x7fa7bc06a840 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fa7d0009b60 tx=0x7fa7d0009190 comp rx=0 tx=0).stop 2024-03-31T15:05:18.375 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:18.379+0000 7fa7e6400640 1 -- 172.21.15.16:0/1232962859 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa7e013e3d0 msgr2=0x7fa7e013be70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:18.375 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:18.379+0000 7fa7e6400640 1 --2- 172.21.15.16:0/1232962859 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa7e013e3d0 0x7fa7e013be70 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7fa7d800cad0 tx=0x7fa7d800cf90 comp rx=0 tx=0).stop 2024-03-31T15:05:18.375 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:18.379+0000 7fa7e6400640 1 -- 172.21.15.16:0/1232962859 shutdown_connections 2024-03-31T15:05:18.375 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:18.379+0000 7fa7e6400640 1 --2- 172.21.15.16:0/1232962859 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fa7bc068380 0x7fa7bc06a840 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:18.375 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:18.379+0000 7fa7e6400640 1 --2- 172.21.15.16:0/1232962859 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa7e013e3d0 0x7fa7e013be70 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:18.375 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:18.379+0000 7fa7e6400640 1 --2- 172.21.15.16:0/1232962859 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fa7e013dff0 0x7fa7e012b710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:18.375 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:18.379+0000 7fa7e6400640 1 --2- 172.21.15.16:0/1232962859 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa7e00cc9d0 0x7fa7e01f5220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:18.375 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:18.379+0000 7fa7e6400640 1 -- 172.21.15.16:0/1232962859 >> 172.21.15.16:0/1232962859 conn(0x7fa7e006ee40 msgr2=0x7fa7e01595d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:05:18.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:18.379+0000 7fa7e6400640 1 -- 172.21.15.16:0/1232962859 shutdown_connections 2024-03-31T15:05:18.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:18.379+0000 7fa7e6400640 1 -- 172.21.15.16:0/1232962859 wait complete. 2024-03-31T15:05:18.393 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_apply_tag' 2024-03-31T15:05:18.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:18.539+0000 7f01ce600640 1 Processor -- start 2024-03-31T15:05:18.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:18.539+0000 7f01ce600640 1 -- start start 2024-03-31T15:05:18.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:18.539+0000 7f01ce600640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f01c806f920 0x7f01c806fd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:18.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:18.539+0000 7f01ce600640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f01c80d2010 0x7f01c80d24a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:18.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:18.539+0000 7f01ce600640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f01c80d0920 con 0x7f01c8145e40 2024-03-31T15:05:18.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:18.539+0000 7f01ce600640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f01c80702c0 con 0x7f01c80d2010 2024-03-31T15:05:18.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:18.539+0000 7f01ce600640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f01c80d06d0 con 0x7f01c806f920 2024-03-31T15:05:18.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:18.539+0000 7f01c7e00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f01c80d2010 0x7f01c80d24a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:18.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:18.539+0000 7f01bea00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f01c806f920 0x7f01c806fd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:18.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:18.539+0000 7f01c7e00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f01c80d2010 0x7f01c80d24a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.16:55038/0 (socket says 172.21.15.16:55038) 2024-03-31T15:05:18.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:18.539+0000 7f01c7400640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f01c8145e40 0x7f01c80cc740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.16:40944/0 (socket says 172.21.15.16:40944) 2024-03-31T15:05:18.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:18.539+0000 7f01c7e00640 1 -- 172.21.15.16:0/1268914257 learned_addr learned my addr 172.21.15.16:0/1268914257 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:05:18.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:18.539+0000 7f01bea00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f01c806f920 0x7f01c806fd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3301/0 says I am v2:172.21.15.16:37522/0 (socket says 172.21.15.16:37522) 2024-03-31T15:05:18.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:18.539+0000 7f01c6a00640 1 -- 172.21.15.16:0/1268914257 <== mon.1 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1071216297 0 0) 0x7f01c80d0920 con 0x7f01c8145e40 2024-03-31T15:05:18.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:18.539+0000 7f01c6a00640 1 -- 172.21.15.16:0/1268914257 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f01ac003530 con 0x7f01c8145e40 2024-03-31T15:05:18.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:18.539+0000 7f01bea00640 1 -- 172.21.15.16:0/1268914257 >> v2:172.21.15.89:3300/0 conn(0x7f01c80d2010 msgr2=0x7f01c80d24a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:05:18.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:18.539+0000 7f01bea00640 1 --2- 172.21.15.16:0/1268914257 >> v2:172.21.15.89:3300/0 conn(0x7f01c80d2010 0x7f01c80d24a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:18.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:18.539+0000 7f01bea00640 1 -- 172.21.15.16:0/1268914257 >> v1:172.21.15.89:6789/0 conn(0x7f01c8145e40 legacy=0x7f01c80cc740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:18.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:18.539+0000 7f01bea00640 1 -- 172.21.15.16:0/1268914257 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01c80d0c00 con 0x7f01c806f920 2024-03-31T15:05:18.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:18.539+0000 7f01c7e00640 1 --2- 172.21.15.16:0/1268914257 >> v2:172.21.15.89:3300/0 conn(0x7f01c80d2010 0x7f01c80d24a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:05:18.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:18.543+0000 7f01bea00640 1 --2- 172.21.15.16:0/1268914257 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f01c806f920 0x7f01c806fd80 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f01b80097c0 tx=0x7f01b801c4a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a0fd4a8b265f5f81 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:05:18.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:18.543+0000 7f01c6a00640 1 -- 172.21.15.16:0/1268914257 <== mon.2 v2:172.21.15.89:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01b80b2070 con 0x7f01c806f920 2024-03-31T15:05:18.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:18.543+0000 7f01c6a00640 1 -- 172.21.15.16:0/1268914257 <== mon.2 v2:172.21.15.89:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f01b80ad640 con 0x7f01c806f920 2024-03-31T15:05:18.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:18.543+0000 7f01c6a00640 1 -- 172.21.15.16:0/1268914257 <== mon.2 v2:172.21.15.89:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01b80ada10 con 0x7f01c806f920 2024-03-31T15:05:18.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:18.543+0000 7f01ce600640 1 -- 172.21.15.16:0/1268914257 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f01c806f920 msgr2=0x7f01c806fd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:18.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:18.543+0000 7f01ce600640 1 --2- 172.21.15.16:0/1268914257 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f01c806f920 0x7f01c806fd80 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f01b80097c0 tx=0x7f01b801c4a0 comp rx=0 tx=0).stop 2024-03-31T15:05:18.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:18.543+0000 7f01ce600640 1 -- 172.21.15.16:0/1268914257 shutdown_connections 2024-03-31T15:05:18.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:18.543+0000 7f01ce600640 1 --2- 172.21.15.16:0/1268914257 >> v2:172.21.15.89:3300/0 conn(0x7f01c80d2010 0x7f01c80d24a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:18.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:18.543+0000 7f01ce600640 1 --2- 172.21.15.16:0/1268914257 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f01c806f920 0x7f01c806fd80 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:18.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:18.543+0000 7f01ce600640 1 -- 172.21.15.16:0/1268914257 >> 172.21.15.16:0/1268914257 conn(0x7f01c8067330 msgr2=0x7f01c8067740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:05:18.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:18.543+0000 7f01ce600640 1 -- 172.21.15.16:0/1268914257 shutdown_connections 2024-03-31T15:05:18.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:18.543+0000 7f01ce600640 1 -- 172.21.15.16:0/1268914257 wait complete. 2024-03-31T15:05:18.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:18.547+0000 7f01ce600640 1 Processor -- start 2024-03-31T15:05:18.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:18.547+0000 7f01ce600640 1 -- start start 2024-03-31T15:05:18.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:18.547+0000 7f01ce600640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f01c806f920 0x7f01c81faad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:18.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:18.547+0000 7f01ce600640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f01c80d2010 0x7f01c81fb010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:18.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:18.547+0000 7f01ce600640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f01c8145e40 0x7f01c81f6960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:18.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:18.547+0000 7f01ce600640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f01c80d0ee0 con 0x7f01c806f920 2024-03-31T15:05:18.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:18.547+0000 7f01ce600640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f01c80ce9e0 con 0x7f01c8145e40 2024-03-31T15:05:18.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:18.547+0000 7f01ce600640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f01c80d2df0 con 0x7f01c80d2010 2024-03-31T15:05:18.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:18.547+0000 7f01c7400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f01c806f920 0x7f01c81faad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:18.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:18.547+0000 7f01c7400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f01c806f920 0x7f01c81faad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49252/0 (socket says 172.21.15.16:49252) 2024-03-31T15:05:18.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:18.547+0000 7f01c7400640 1 -- 172.21.15.16:0/3612149256 learned_addr learned my addr 172.21.15.16:0/3612149256 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:05:18.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:18.547+0000 7f01c7e00640 1 --2- 172.21.15.16:0/3612149256 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f01c8145e40 0x7f01c81f6960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:18.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:18.547+0000 7f01bea00640 1 --2- 172.21.15.16:0/3612149256 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f01c80d2010 0x7f01c81fb010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:18.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:18.547+0000 7f01c7400640 1 -- 172.21.15.16:0/3612149256 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f01c80d2010 msgr2=0x7f01c81fb010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:18.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:18.547+0000 7f01c7400640 1 --2- 172.21.15.16:0/3612149256 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f01c80d2010 0x7f01c81fb010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:18.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:18.547+0000 7f01c7400640 1 -- 172.21.15.16:0/3612149256 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f01c8145e40 msgr2=0x7f01c81f6960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:18.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:18.547+0000 7f01c7400640 1 --2- 172.21.15.16:0/3612149256 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f01c8145e40 0x7f01c81f6960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:18.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:18.547+0000 7f01c7400640 1 -- 172.21.15.16:0/3612149256 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01c80b5f70 con 0x7f01c806f920 2024-03-31T15:05:18.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:18.547+0000 7f01c7400640 1 --2- 172.21.15.16:0/3612149256 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f01c806f920 0x7f01c81faad0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f01b000c390 tx=0x7f01b000c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:05:18.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:18.547+0000 7f01bfe00640 1 -- 172.21.15.16:0/3612149256 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01b00074e0 con 0x7f01c806f920 2024-03-31T15:05:18.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:18.547+0000 7f01bfe00640 1 -- 172.21.15.16:0/3612149256 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f01b00047f0 con 0x7f01c806f920 2024-03-31T15:05:18.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:18.547+0000 7f01bfe00640 1 -- 172.21.15.16:0/3612149256 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01b00076a0 con 0x7f01c806f920 2024-03-31T15:05:18.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:18.547+0000 7f01ce600640 1 -- 172.21.15.16:0/3612149256 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f01c80b6250 con 0x7f01c806f920 2024-03-31T15:05:18.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:18.547+0000 7f01ce600640 1 -- 172.21.15.16:0/3612149256 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f01c80b6810 con 0x7f01c806f920 2024-03-31T15:05:18.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:18.551+0000 7f01ce600640 1 -- 172.21.15.16:0/3612149256 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f01c8069950 con 0x7f01c806f920 2024-03-31T15:05:18.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:18.551+0000 7f01bfe00640 1 -- 172.21.15.16:0/3612149256 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f01b001a020 con 0x7f01c806f920 2024-03-31T15:05:18.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:18.551+0000 7f01bfe00640 1 --2- 172.21.15.16:0/3612149256 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f01a4068380 0x7f01a406a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:18.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:18.551+0000 7f01bfe00640 1 -- 172.21.15.16:0/3612149256 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 4943+0+0 (secure 0 0 0) 0x7f01b0085db0 con 0x7f01c806f920 2024-03-31T15:05:18.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:18.551+0000 7f01bea00640 1 --2- 172.21.15.16:0/3612149256 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f01a4068380 0x7f01a406a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:18.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:18.551+0000 7f01bea00640 1 --2- 172.21.15.16:0/3612149256 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f01a4068380 0x7f01a406a840 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f01b80af000 tx=0x7f01b8008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:05:18.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:18.555+0000 7f01bfe00640 1 -- 172.21.15.16:0/3612149256 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f01b008a070 con 0x7f01c806f920 2024-03-31T15:05:18.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:18.779+0000 7f01ce600640 1 -- 172.21.15.16:0/3612149256 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_apply_tag"]} v 0) -- 0x7f01c80d1160 con 0x7f01c806f920 2024-03-31T15:05:19.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:19.015+0000 7f01bfe00640 1 -- 172.21.15.16:0/3612149256 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_apply_tag"]}]=0 v113) ==== 142+0+0 (secure 0 0 0) 0x7f01b0056010 con 0x7f01c806f920 2024-03-31T15:05:19.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:19.015+0000 7f01ce600640 1 -- 172.21.15.16:0/3612149256 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f01a4068380 msgr2=0x7f01a406a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:19.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:19.015+0000 7f01ce600640 1 --2- 172.21.15.16:0/3612149256 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f01a4068380 0x7f01a406a840 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f01b80af000 tx=0x7f01b8008040 comp rx=0 tx=0).stop 2024-03-31T15:05:19.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:19.015+0000 7f01ce600640 1 -- 172.21.15.16:0/3612149256 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f01c806f920 msgr2=0x7f01c81faad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:19.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:19.015+0000 7f01ce600640 1 --2- 172.21.15.16:0/3612149256 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f01c806f920 0x7f01c81faad0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f01b000c390 tx=0x7f01b000c850 comp rx=0 tx=0).stop 2024-03-31T15:05:19.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:19.015+0000 7f01ce600640 1 -- 172.21.15.16:0/3612149256 shutdown_connections 2024-03-31T15:05:19.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:19.015+0000 7f01ce600640 1 --2- 172.21.15.16:0/3612149256 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f01a4068380 0x7f01a406a840 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:19.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:19.015+0000 7f01ce600640 1 --2- 172.21.15.16:0/3612149256 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f01c8145e40 0x7f01c81f6960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:19.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:19.015+0000 7f01ce600640 1 --2- 172.21.15.16:0/3612149256 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f01c80d2010 0x7f01c81fb010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:19.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:19.015+0000 7f01ce600640 1 --2- 172.21.15.16:0/3612149256 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f01c806f920 0x7f01c81faad0 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:19.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:19.015+0000 7f01ce600640 1 -- 172.21.15.16:0/3612149256 >> 172.21.15.16:0/3612149256 conn(0x7f01c8067330 msgr2=0x7f01c80cc440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:05:19.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:19.015+0000 7f01ce600640 1 -- 172.21.15.16:0/3612149256 shutdown_connections 2024-03-31T15:05:19.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:19.015+0000 7f01ce600640 1 -- 172.21.15.16:0/3612149256 wait complete. 2024-03-31T15:05:19.032 INFO:tasks.cephfs_test_runner:test_apply_tag (tasks.cephfs.test_forward_scrub.TestForwardScrub) ... ok 2024-03-31T15:05:19.032 INFO:tasks.cephfs_test_runner:Starting test: test_backtrace_repair (tasks.cephfs.test_forward_scrub.TestForwardScrub) That the MDS can repair an inodes backtrace in the data pool 2024-03-31T15:05:19.032 INFO:tasks.cephfs_test_runner:test_backtrace_repair (tasks.cephfs.test_forward_scrub.TestForwardScrub) 2024-03-31T15:05:19.033 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-31T15:05:19.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:19.179+0000 7fda89400640 1 Processor -- start 2024-03-31T15:05:19.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:19.179+0000 7fda89400640 1 -- start start 2024-03-31T15:05:19.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:19.179+0000 7fda89400640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fda84071c70 0x7fda840ad4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:19.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:19.179+0000 7fda89400640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fda840b24f0 con 0x7fda840af590 2024-03-31T15:05:19.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:19.179+0000 7fda89400640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fda840b27b0 con 0x7fda8415c230 2024-03-31T15:05:19.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:19.179+0000 7fda89400640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7fda840cfed0 con 0x7fda84071c70 2024-03-31T15:05:19.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:19.179+0000 7fda88a00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fda84071c70 0x7fda840ad4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:19.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:19.179+0000 7fda83400640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fda840af590 0x7fda840afc10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:55502/0 (socket says 172.21.15.16:55502) 2024-03-31T15:05:19.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:19.179+0000 7fda83400640 1 -- 172.21.15.16:0/1492261353 learned_addr learned my addr 172.21.15.16:0/1492261353 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:05:19.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:19.179+0000 7fda88a00640 1 -- 172.21.15.16:0/1492261353 >> v1:172.21.15.89:6789/0 conn(0x7fda8415c230 legacy=0x7fda840aee80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:05:19.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:19.179+0000 7fda82a00640 1 -- 172.21.15.16:0/1492261353 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2261749213 0 0) 0x7fda840b24f0 con 0x7fda840af590 2024-03-31T15:05:19.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:19.179+0000 7fda88a00640 1 -- 172.21.15.16:0/1492261353 >> v1:172.21.15.16:6789/0 conn(0x7fda840af590 legacy=0x7fda840afc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:19.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:19.179+0000 7fda88a00640 1 -- 172.21.15.16:0/1492261353 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda84073660 con 0x7fda84071c70 2024-03-31T15:05:19.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:19.183+0000 7fda88a00640 1 --2- 172.21.15.16:0/1492261353 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fda84071c70 0x7fda840ad4a0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fda7c009d90 tx=0x7fda7c01c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5fe9412d15c888a2 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:05:19.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:19.183+0000 7fda82a00640 1 -- 172.21.15.16:0/1492261353 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda7c0b2070 con 0x7fda84071c70 2024-03-31T15:05:19.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:19.183+0000 7fda82a00640 1 -- 172.21.15.16:0/1492261353 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fda7c0a5720 con 0x7fda84071c70 2024-03-31T15:05:19.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:19.183+0000 7fda82a00640 1 -- 172.21.15.16:0/1492261353 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda7c0a5af0 con 0x7fda84071c70 2024-03-31T15:05:19.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:19.183+0000 7fda89400640 1 -- 172.21.15.16:0/1492261353 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fda84071c70 msgr2=0x7fda840ad4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:19.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:19.183+0000 7fda89400640 1 --2- 172.21.15.16:0/1492261353 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fda84071c70 0x7fda840ad4a0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fda7c009d90 tx=0x7fda7c01c970 comp rx=0 tx=0).stop 2024-03-31T15:05:19.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:19.183+0000 7fda89400640 1 -- 172.21.15.16:0/1492261353 shutdown_connections 2024-03-31T15:05:19.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:19.183+0000 7fda89400640 1 --2- 172.21.15.16:0/1492261353 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fda84071c70 0x7fda840ad4a0 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:19.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:19.183+0000 7fda89400640 1 -- 172.21.15.16:0/1492261353 >> 172.21.15.16:0/1492261353 conn(0x7fda840b5820 msgr2=0x7fda840b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:05:19.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:19.183+0000 7fda89400640 1 -- 172.21.15.16:0/1492261353 shutdown_connections 2024-03-31T15:05:19.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:19.183+0000 7fda89400640 1 -- 172.21.15.16:0/1492261353 wait complete. 2024-03-31T15:05:19.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:19.183+0000 7fda89400640 1 Processor -- start 2024-03-31T15:05:19.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:19.183+0000 7fda89400640 1 -- start start 2024-03-31T15:05:19.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:19.187+0000 7fda89400640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fda84071c70 0x7fda8414d580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:19.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:19.187+0000 7fda89400640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fda840af590 0x7fda8413b5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:19.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:19.187+0000 7fda89400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fda8415c230 0x7fda840cbc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:19.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:19.187+0000 7fda89400640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fda8415c610 con 0x7fda8415c230 2024-03-31T15:05:19.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:19.187+0000 7fda89400640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fda840b2a90 con 0x7fda840af590 2024-03-31T15:05:19.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:19.187+0000 7fda89400640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fda8416aa60 con 0x7fda84071c70 2024-03-31T15:05:19.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:19.187+0000 7fda88a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fda8415c230 0x7fda840cbc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:19.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:19.187+0000 7fda88a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fda8415c230 0x7fda840cbc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49274/0 (socket says 172.21.15.16:49274) 2024-03-31T15:05:19.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:19.187+0000 7fda88a00640 1 -- 172.21.15.16:0/1930865432 learned_addr learned my addr 172.21.15.16:0/1930865432 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:05:19.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:19.187+0000 7fda83400640 1 --2- 172.21.15.16:0/1930865432 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fda840af590 0x7fda8413b5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:19.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:19.187+0000 7fda83e00640 1 --2- 172.21.15.16:0/1930865432 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fda84071c70 0x7fda8414d580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:19.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:19.187+0000 7fda88a00640 1 -- 172.21.15.16:0/1930865432 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fda84071c70 msgr2=0x7fda8414d580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:19.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:19.187+0000 7fda88a00640 1 --2- 172.21.15.16:0/1930865432 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fda84071c70 0x7fda8414d580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:19.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:19.187+0000 7fda88a00640 1 -- 172.21.15.16:0/1930865432 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fda840af590 msgr2=0x7fda8413b5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:19.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:19.187+0000 7fda88a00640 1 --2- 172.21.15.16:0/1930865432 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fda840af590 0x7fda8413b5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:19.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:19.187+0000 7fda88a00640 1 -- 172.21.15.16:0/1930865432 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda840cf530 con 0x7fda8415c230 2024-03-31T15:05:19.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:19.187+0000 7fda88a00640 1 --2- 172.21.15.16:0/1930865432 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fda8415c230 0x7fda840cbc30 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7fda7c0b0040 tx=0x7fda7c019710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:05:19.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:19.187+0000 7fda7be00640 1 -- 172.21.15.16:0/1930865432 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda7c0b2050 con 0x7fda8415c230 2024-03-31T15:05:19.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:19.187+0000 7fda7be00640 1 -- 172.21.15.16:0/1930865432 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fda7c0be070 con 0x7fda8415c230 2024-03-31T15:05:19.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:19.187+0000 7fda89400640 1 -- 172.21.15.16:0/1930865432 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fda840cf810 con 0x7fda8415c230 2024-03-31T15:05:19.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:19.187+0000 7fda7be00640 1 -- 172.21.15.16:0/1930865432 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda7c0adc40 con 0x7fda8415c230 2024-03-31T15:05:19.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:19.187+0000 7fda89400640 1 -- 172.21.15.16:0/1930865432 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fda8413bae0 con 0x7fda8415c230 2024-03-31T15:05:19.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:19.187+0000 7fda89400640 1 -- 172.21.15.16:0/1930865432 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fda8404f8c0 con 0x7fda8415c230 2024-03-31T15:05:19.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:19.191+0000 7fda7be00640 1 -- 172.21.15.16:0/1930865432 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fda7c0c3460 con 0x7fda8415c230 2024-03-31T15:05:19.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:19.191+0000 7fda7be00640 1 --2- 172.21.15.16:0/1930865432 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fda64068380 0x7fda6406a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:19.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:19.191+0000 7fda7be00640 1 -- 172.21.15.16:0/1930865432 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 4943+0+0 (secure 0 0 0) 0x7fda7c121890 con 0x7fda8415c230 2024-03-31T15:05:19.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:19.191+0000 7fda83e00640 1 --2- 172.21.15.16:0/1930865432 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fda64068380 0x7fda6406a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:19.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:19.191+0000 7fda83e00640 1 --2- 172.21.15.16:0/1930865432 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fda64068380 0x7fda6406a840 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fda74001770 tx=0x7fda74009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:05:19.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:19.195+0000 7fda7be00640 1 -- 172.21.15.16:0/1930865432 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fda7c016560 con 0x7fda8415c230 2024-03-31T15:05:19.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:19.359+0000 7fda89400640 1 -- 172.21.15.16:0/1930865432 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fda840cf9e0 con 0x7fda8415c230 2024-03-31T15:05:19.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:19.359+0000 7fda7be00640 1 -- 172.21.15.16:0/1930865432 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v37) ==== 74+0+12948 (secure 0 0 0) 0x7fda7c0f1ce0 con 0x7fda8415c230 2024-03-31T15:05:19.355 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:05:19.355 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":37,"fsid":"286eb5da-fb47-4550-b290-d9519c13b9a5","created":"2024-03-31T15:03:11.438952+0000","modified":"2024-03-31T15:05:18.005792+0000","last_up_change":"2024-03-31T15:03:22.885535+0000","last_in_change":"2024-03-31T15:03:13.941667+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-31T15:03:22.550787+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"92a66450-ef19-40d8-ac3d-d760b23b0bda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6801","nonce":932712244}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6803","nonce":932712244}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6807","nonce":932712244}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6805","nonce":932712244}]},"public_addr":"172.21.15.16:6801/932712244","cluster_addr":"172.21.15.16:6803/932712244","heartbeat_back_addr":"172.21.15.16:6807/932712244","heartbeat_front_addr":"172.21.15.16:6805/932712244","state":["exists","up"]},{"osd":1,"uuid":"da9d1f2b-1968-4959-b843-faa231988e92","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6810","nonce":1398737533}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6811","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6813","nonce":1398737533}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6821","nonce":1398737533}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6815","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6816","nonce":1398737533}]},"public_addr":"172.21.15.16:6810/1398737533","cluster_addr":"172.21.15.16:6813/1398737533","heartbeat_back_addr":"172.21.15.16:6821/1398737533","heartbeat_front_addr":"172.21.15.16:6816/1398737533","state":["exists","up"]},{"osd":2,"uuid":"01a73662-9a21-4b1f-bcbb-42b9636a4638","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6819","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6822","nonce":3018526646}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6825","nonce":3018526646}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6831","nonce":3018526646}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6827","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6829","nonce":3018526646}]},"public_addr":"172.21.15.16:6822/3018526646","cluster_addr":"172.21.15.16:6825/3018526646","heartbeat_back_addr":"172.21.15.16:6831/3018526646","heartbeat_front_addr":"172.21.15.16:6829/3018526646","state":["exists","up"]},{"osd":3,"uuid":"c7771fd3-e18e-4500-ac38-23cf10ebcd75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6809","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6812","nonce":3360352974}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6817","nonce":3360352974}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6828","nonce":3360352974}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6823","nonce":3360352974}]},"public_addr":"172.21.15.16:6812/3360352974","cluster_addr":"172.21.15.16:6817/3360352974","heartbeat_back_addr":"172.21.15.16:6828/3360352974","heartbeat_front_addr":"172.21.15.16:6823/3360352974","state":["exists","up"]},{"osd":4,"uuid":"88260730-1482-4fbd-a40f-cb1124e9df4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6801","nonce":1856844856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6803","nonce":1856844856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6807","nonce":1856844856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6805","nonce":1856844856}]},"public_addr":"172.21.15.89:6801/1856844856","cluster_addr":"172.21.15.89:6803/1856844856","heartbeat_back_addr":"172.21.15.89:6807/1856844856","heartbeat_front_addr":"172.21.15.89:6805/1856844856","state":["exists","up"]},{"osd":5,"uuid":"4cb2c00c-fae0-4dd8-9d27-ec4f0093ccf7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6814","nonce":50395527}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6817","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6820","nonce":50395527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6831","nonce":50395527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6823","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6826","nonce":50395527}]},"public_addr":"172.21.15.89:6814/50395527","cluster_addr":"172.21.15.89:6820/50395527","heartbeat_back_addr":"172.21.15.89:6831/50395527","heartbeat_front_addr":"172.21.15.89:6826/50395527","state":["exists","up"]},{"osd":6,"uuid":"81ce0c19-2554-4cae-acbe-66a05e9e9b9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6809","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6812","nonce":3470471752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6815","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6818","nonce":3470471752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6827","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6829","nonce":3470471752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6821","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6824","nonce":3470471752}]},"public_addr":"172.21.15.89:6812/3470471752","cluster_addr":"172.21.15.89:6818/3470471752","heartbeat_back_addr":"172.21.15.89:6829/3470471752","heartbeat_front_addr":"172.21.15.89:6824/3470471752","state":["exists","up"]},{"osd":7,"uuid":"85e4313b-d37d-442e-8042-153bf20a847d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6811","nonce":1796778630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6813","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6816","nonce":1796778630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6825","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6828","nonce":1796778630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6819","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6822","nonce":1796778630}]},"public_addr":"172.21.15.89:6811/1796778630","cluster_addr":"172.21.15.89:6816/1796778630","heartbeat_back_addr":"172.21.15.89:6828/1796778630","heartbeat_front_addr":"172.21.15.89:6822/1796778630","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.439088+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.529970+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.680512+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:21.044161+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.582111+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.536558+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.409522+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.590229+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:6835/1845707550":"2024-04-01T15:05:14.999965+0000","172.21.15.16:6834/1845707550":"2024-04-01T15:05:14.999965+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-31T15:05:19.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:19.363+0000 7fda89400640 1 -- 172.21.15.16:0/1930865432 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fda64068380 msgr2=0x7fda6406a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:19.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:19.363+0000 7fda89400640 1 --2- 172.21.15.16:0/1930865432 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fda64068380 0x7fda6406a840 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fda74001770 tx=0x7fda74009040 comp rx=0 tx=0).stop 2024-03-31T15:05:19.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:19.363+0000 7fda89400640 1 -- 172.21.15.16:0/1930865432 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fda8415c230 msgr2=0x7fda840cbc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:19.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:19.363+0000 7fda89400640 1 --2- 172.21.15.16:0/1930865432 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fda8415c230 0x7fda840cbc30 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7fda7c0b0040 tx=0x7fda7c019710 comp rx=0 tx=0).stop 2024-03-31T15:05:19.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:19.363+0000 7fda89400640 1 -- 172.21.15.16:0/1930865432 shutdown_connections 2024-03-31T15:05:19.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:19.363+0000 7fda89400640 1 --2- 172.21.15.16:0/1930865432 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fda64068380 0x7fda6406a840 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:19.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:19.363+0000 7fda89400640 1 --2- 172.21.15.16:0/1930865432 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fda8415c230 0x7fda840cbc30 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:19.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:19.363+0000 7fda89400640 1 --2- 172.21.15.16:0/1930865432 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fda840af590 0x7fda8413b5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:19.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:19.363+0000 7fda89400640 1 --2- 172.21.15.16:0/1930865432 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fda84071c70 0x7fda8414d580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:19.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:19.363+0000 7fda89400640 1 -- 172.21.15.16:0/1930865432 >> 172.21.15.16:0/1930865432 conn(0x7fda840b5820 msgr2=0x7fda84159830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:05:19.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:19.363+0000 7fda89400640 1 -- 172.21.15.16:0/1930865432 shutdown_connections 2024-03-31T15:05:19.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:19.363+0000 7fda89400640 1 -- 172.21.15.16:0/1930865432 wait complete. 2024-03-31T15:05:19.377 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-31T15:03:22.550787+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-31T15:05:19.377 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-31T15:05:19.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:19.523+0000 7f71a4200640 1 Processor -- start 2024-03-31T15:05:19.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:19.523+0000 7f71a4200640 1 -- start start 2024-03-31T15:05:19.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:19.527+0000 7f71a4200640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f719c141590 0x7f719c144340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:19.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:19.527+0000 7f71a4200640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f719c144980 con 0x7f719c1411b0 2024-03-31T15:05:19.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:19.527+0000 7f71a4200640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f719c161fe0 con 0x7f719c070030 2024-03-31T15:05:19.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:19.527+0000 7f71a4200640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f719c15f910 con 0x7f719c141590 2024-03-31T15:05:19.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:19.527+0000 7f71a2200640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f719c141590 0x7f719c144340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:19.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:19.527+0000 7f71a1800640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f719c1411b0 0x7f719c06f920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:55518/0 (socket says 172.21.15.16:55518) 2024-03-31T15:05:19.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:19.527+0000 7f71a1800640 1 -- 172.21.15.16:0/293796044 learned_addr learned my addr 172.21.15.16:0/293796044 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:05:19.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:19.527+0000 7f71a2200640 1 -- 172.21.15.16:0/293796044 >> v1:172.21.15.89:6789/0 conn(0x7f719c070030 legacy=0x7f719c143c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:19.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:19.527+0000 7f71a2200640 1 -- 172.21.15.16:0/293796044 >> v1:172.21.15.16:6789/0 conn(0x7f719c1411b0 legacy=0x7f719c06f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:19.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:19.527+0000 7f71a2200640 1 -- 172.21.15.16:0/293796044 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f719c1622c0 con 0x7f719c141590 2024-03-31T15:05:19.525 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:19.531+0000 7f71a2200640 1 --2- 172.21.15.16:0/293796044 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f719c141590 0x7f719c144340 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f7194009d90 tx=0x7f719401c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=310e1acae5ca554c server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:05:19.525 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:19.531+0000 7f719be00640 1 -- 172.21.15.16:0/293796044 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71940b2070 con 0x7f719c141590 2024-03-31T15:05:19.525 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:19.531+0000 7f719be00640 1 -- 172.21.15.16:0/293796044 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f71940a5940 con 0x7f719c141590 2024-03-31T15:05:19.525 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:19.531+0000 7f719be00640 1 -- 172.21.15.16:0/293796044 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71940a5d10 con 0x7f719c141590 2024-03-31T15:05:19.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:19.531+0000 7f71a4200640 1 -- 172.21.15.16:0/293796044 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f719c141590 msgr2=0x7f719c144340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:19.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:19.531+0000 7f71a4200640 1 --2- 172.21.15.16:0/293796044 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f719c141590 0x7f719c144340 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f7194009d90 tx=0x7f719401c970 comp rx=0 tx=0).stop 2024-03-31T15:05:19.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:19.531+0000 7f71a4200640 1 -- 172.21.15.16:0/293796044 shutdown_connections 2024-03-31T15:05:19.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:19.531+0000 7f71a4200640 1 --2- 172.21.15.16:0/293796044 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f719c141590 0x7f719c144340 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:19.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:19.531+0000 7f71a4200640 1 -- 172.21.15.16:0/293796044 >> 172.21.15.16:0/293796044 conn(0x7f719c067930 msgr2=0x7f719c0b3ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:05:19.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:19.531+0000 7f71a4200640 1 -- 172.21.15.16:0/293796044 shutdown_connections 2024-03-31T15:05:19.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:19.531+0000 7f71a4200640 1 -- 172.21.15.16:0/293796044 wait complete. 2024-03-31T15:05:19.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:19.531+0000 7f71a4200640 1 Processor -- start 2024-03-31T15:05:19.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:19.531+0000 7f71a4200640 1 -- start start 2024-03-31T15:05:19.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:19.531+0000 7f71a4200640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f719c070030 0x7f719c13d280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:19.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:19.531+0000 7f71a4200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f719c1411b0 0x7f719c157d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:19.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:19.531+0000 7f71a4200640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f719c141590 0x7f719c0b51f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:19.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:19.531+0000 7f71a4200640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f719c16be80 con 0x7f719c1411b0 2024-03-31T15:05:19.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:19.531+0000 7f71a4200640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f719c16bd00 con 0x7f719c141590 2024-03-31T15:05:19.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:19.531+0000 7f71a4200640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f719c16c000 con 0x7f719c070030 2024-03-31T15:05:19.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:19.535+0000 7f71a0e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f719c1411b0 0x7f719c157d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:19.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:19.535+0000 7f71a0e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f719c1411b0 0x7f719c157d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49296/0 (socket says 172.21.15.16:49296) 2024-03-31T15:05:19.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:19.535+0000 7f71a1800640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f719c070030 0x7f719c13d280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:19.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:19.535+0000 7f71a0e00640 1 -- 172.21.15.16:0/2618706411 learned_addr learned my addr 172.21.15.16:0/2618706411 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:05:19.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:19.535+0000 7f71a2200640 1 --2- 172.21.15.16:0/2618706411 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f719c141590 0x7f719c0b51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:19.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:19.535+0000 7f71a0e00640 1 -- 172.21.15.16:0/2618706411 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f719c070030 msgr2=0x7f719c13d280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:19.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:19.535+0000 7f71a0e00640 1 --2- 172.21.15.16:0/2618706411 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f719c070030 0x7f719c13d280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:19.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:19.535+0000 7f71a0e00640 1 -- 172.21.15.16:0/2618706411 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f719c141590 msgr2=0x7f719c0b51f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:19.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:19.535+0000 7f71a0e00640 1 --2- 172.21.15.16:0/2618706411 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f719c141590 0x7f719c0b51f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:19.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:19.535+0000 7f71a0e00640 1 -- 172.21.15.16:0/2618706411 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f719c1614f0 con 0x7f719c1411b0 2024-03-31T15:05:19.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:19.535+0000 7f71a0e00640 1 --2- 172.21.15.16:0/2618706411 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f719c1411b0 0x7f719c157d90 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f718800c420 tx=0x7f718800c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:05:19.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:19.535+0000 7f7199600640 1 -- 172.21.15.16:0/2618706411 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71880052d0 con 0x7f719c1411b0 2024-03-31T15:05:19.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:19.535+0000 7f7199600640 1 -- 172.21.15.16:0/2618706411 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7188005470 con 0x7f719c1411b0 2024-03-31T15:05:19.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:19.535+0000 7f71a4200640 1 -- 172.21.15.16:0/2618706411 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f719c142520 con 0x7f719c1411b0 2024-03-31T15:05:19.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:19.535+0000 7f7199600640 1 -- 172.21.15.16:0/2618706411 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7188007260 con 0x7f719c1411b0 2024-03-31T15:05:19.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:19.535+0000 7f71a4200640 1 -- 172.21.15.16:0/2618706411 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f719c13b8d0 con 0x7f719c1411b0 2024-03-31T15:05:19.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:19.535+0000 7f71a4200640 1 -- 172.21.15.16:0/2618706411 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f719c069950 con 0x7f719c1411b0 2024-03-31T15:05:19.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:19.535+0000 7f7199600640 1 -- 172.21.15.16:0/2618706411 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f7188007400 con 0x7f719c1411b0 2024-03-31T15:05:19.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:19.535+0000 7f7199600640 1 --2- 172.21.15.16:0/2618706411 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f717c068380 0x7f717c06a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:19.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:19.539+0000 7f7199600640 1 -- 172.21.15.16:0/2618706411 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 4943+0+0 (secure 0 0 0) 0x7f7188085b40 con 0x7f719c1411b0 2024-03-31T15:05:19.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:19.539+0000 7f71a1800640 1 --2- 172.21.15.16:0/2618706411 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f717c068380 0x7f717c06a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:19.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:19.539+0000 7f71a1800640 1 --2- 172.21.15.16:0/2618706411 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f717c068380 0x7f717c06a840 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f7190005890 tx=0x7f7190009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:05:19.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:19.543+0000 7f7199600640 1 -- 172.21.15.16:0/2618706411 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f7188055da0 con 0x7f719c1411b0 2024-03-31T15:05:19.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:19.703+0000 7f71a4200640 1 -- 172.21.15.16:0/2618706411 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f719c0b4a20 con 0x7f719c1411b0 2024-03-31T15:05:19.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:19.707+0000 7f7199600640 1 -- 172.21.15.16:0/2618706411 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v37) ==== 93+0+10 (secure 0 0 0) 0x7f718805a0a0 con 0x7f719c1411b0 2024-03-31T15:05:19.701 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2024-03-31T15:05:19.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:19.707+0000 7f71a4200640 1 -- 172.21.15.16:0/2618706411 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f717c068380 msgr2=0x7f717c06a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:19.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:19.707+0000 7f71a4200640 1 --2- 172.21.15.16:0/2618706411 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f717c068380 0x7f717c06a840 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f7190005890 tx=0x7f7190009040 comp rx=0 tx=0).stop 2024-03-31T15:05:19.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:19.707+0000 7f71a4200640 1 -- 172.21.15.16:0/2618706411 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f719c1411b0 msgr2=0x7f719c157d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:19.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:19.707+0000 7f71a4200640 1 --2- 172.21.15.16:0/2618706411 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f719c1411b0 0x7f719c157d90 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f718800c420 tx=0x7f718800c8e0 comp rx=0 tx=0).stop 2024-03-31T15:05:19.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:19.707+0000 7f71a4200640 1 -- 172.21.15.16:0/2618706411 shutdown_connections 2024-03-31T15:05:19.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:19.707+0000 7f71a4200640 1 --2- 172.21.15.16:0/2618706411 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f717c068380 0x7f717c06a840 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:19.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:19.707+0000 7f71a4200640 1 --2- 172.21.15.16:0/2618706411 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f719c141590 0x7f719c0b51f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:19.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:19.707+0000 7f71a4200640 1 --2- 172.21.15.16:0/2618706411 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f719c1411b0 0x7f719c157d90 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:19.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:19.707+0000 7f71a4200640 1 --2- 172.21.15.16:0/2618706411 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f719c070030 0x7f719c13d280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:19.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:19.707+0000 7f71a4200640 1 -- 172.21.15.16:0/2618706411 >> 172.21.15.16:0/2618706411 conn(0x7f719c067930 msgr2=0x7f719c160620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:05:19.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:19.707+0000 7f71a4200640 1 -- 172.21.15.16:0/2618706411 shutdown_connections 2024-03-31T15:05:19.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:19.707+0000 7f71a4200640 1 -- 172.21.15.16:0/2618706411 wait complete. 2024-03-31T15:05:19.722 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_backtrace_repair' 2024-03-31T15:05:19.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:19.867+0000 7efea6c00640 1 Processor -- start 2024-03-31T15:05:19.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:19.867+0000 7efea6c00640 1 -- start start 2024-03-31T15:05:19.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:19.867+0000 7efea6c00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7efea0145d10 0x7efea00cf790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:19.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:19.867+0000 7efea6c00640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efea0160e70 con 0x7efea00728b0 2024-03-31T15:05:19.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:19.867+0000 7efea6c00640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efea00cfcd0 con 0x7efea00b23d0 2024-03-31T15:05:19.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:19.867+0000 7efea6c00640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7efea01602f0 con 0x7efea0145d10 2024-03-31T15:05:19.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:19.867+0000 7efe9fe00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7efea0145d10 0x7efea00cf790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:19.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:19.867+0000 7efe97e00640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7efea00728b0 0x7efea00b1cc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:55524/0 (socket says 172.21.15.16:55524) 2024-03-31T15:05:19.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:19.867+0000 7efe9fe00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7efea0145d10 0x7efea00cf790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49306/0 (socket says 172.21.15.16:49306) 2024-03-31T15:05:19.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:19.867+0000 7efe97e00640 1 -- 172.21.15.16:0/3297997710 learned_addr learned my addr 172.21.15.16:0/3297997710 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:05:19.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:19.871+0000 7efe9fe00640 1 -- 172.21.15.16:0/3297997710 >> v1:172.21.15.89:6789/0 conn(0x7efea00b23d0 legacy=0x7efea015cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:05:19.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:19.871+0000 7efe9fe00640 1 -- 172.21.15.16:0/3297997710 >> v1:172.21.15.16:6789/0 conn(0x7efea00728b0 legacy=0x7efea00b1cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:19.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:19.871+0000 7efe9fe00640 1 -- 172.21.15.16:0/3297997710 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efea01605d0 con 0x7efea0145d10 2024-03-31T15:05:19.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:19.871+0000 7efe9fe00640 1 --2- 172.21.15.16:0/3297997710 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efea0145d10 0x7efea00cf790 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7efe900097c0 tx=0x7efe9001c460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c17df29b61b19bd server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:05:19.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:19.871+0000 7efe9f400640 1 -- 172.21.15.16:0/3297997710 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe900b0070 con 0x7efea0145d10 2024-03-31T15:05:19.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:19.871+0000 7efe9f400640 1 -- 172.21.15.16:0/3297997710 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efe900ab600 con 0x7efea0145d10 2024-03-31T15:05:19.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:19.871+0000 7efe9f400640 1 -- 172.21.15.16:0/3297997710 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe900ab9d0 con 0x7efea0145d10 2024-03-31T15:05:19.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:19.871+0000 7efea6c00640 1 -- 172.21.15.16:0/3297997710 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efea0145d10 msgr2=0x7efea00cf790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:19.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:19.871+0000 7efea6c00640 1 --2- 172.21.15.16:0/3297997710 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efea0145d10 0x7efea00cf790 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7efe900097c0 tx=0x7efe9001c460 comp rx=0 tx=0).stop 2024-03-31T15:05:19.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:19.871+0000 7efea6c00640 1 -- 172.21.15.16:0/3297997710 shutdown_connections 2024-03-31T15:05:19.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:19.871+0000 7efea6c00640 1 --2- 172.21.15.16:0/3297997710 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efea0145d10 0x7efea00cf790 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:19.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:19.871+0000 7efea6c00640 1 -- 172.21.15.16:0/3297997710 >> 172.21.15.16:0/3297997710 conn(0x7efea006b620 msgr2=0x7efea006ba30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:05:19.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:19.871+0000 7efea6c00640 1 -- 172.21.15.16:0/3297997710 shutdown_connections 2024-03-31T15:05:19.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:19.871+0000 7efea6c00640 1 -- 172.21.15.16:0/3297997710 wait complete. 2024-03-31T15:05:19.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:19.875+0000 7efea6c00640 1 Processor -- start 2024-03-31T15:05:19.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:19.875+0000 7efea6c00640 1 -- start start 2024-03-31T15:05:19.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:19.875+0000 7efea6c00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7efea00728b0 0x7efea00b4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:19.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:19.875+0000 7efea6c00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efea00b23d0 0x7efea01fadc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:19.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:19.875+0000 7efea6c00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efea0145d10 0x7efea013bf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:19.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:19.875+0000 7efea6c00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7efea00cf350 con 0x7efea00b23d0 2024-03-31T15:05:19.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:19.875+0000 7efea6c00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7efea015d870 con 0x7efea0145d10 2024-03-31T15:05:19.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:19.875+0000 7efea6c00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7efea00cf4d0 con 0x7efea00728b0 2024-03-31T15:05:19.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:19.875+0000 7efe97e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efea00b23d0 0x7efea01fadc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:19.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:19.875+0000 7efe97e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efea00b23d0 0x7efea01fadc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49312/0 (socket says 172.21.15.16:49312) 2024-03-31T15:05:19.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:19.875+0000 7efe97e00640 1 -- 172.21.15.16:0/4190542079 learned_addr learned my addr 172.21.15.16:0/4190542079 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:05:19.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:19.875+0000 7efea4c00640 1 --2- 172.21.15.16:0/4190542079 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efea0145d10 0x7efea013bf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:19.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:19.875+0000 7efe9fe00640 1 --2- 172.21.15.16:0/4190542079 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7efea00728b0 0x7efea00b4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:19.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:19.875+0000 7efe97e00640 1 -- 172.21.15.16:0/4190542079 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7efea00728b0 msgr2=0x7efea00b4d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:19.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:19.875+0000 7efe97e00640 1 --2- 172.21.15.16:0/4190542079 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7efea00728b0 0x7efea00b4d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:19.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:19.875+0000 7efe97e00640 1 -- 172.21.15.16:0/4190542079 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efea0145d10 msgr2=0x7efea013bf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:19.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:19.875+0000 7efe97e00640 1 --2- 172.21.15.16:0/4190542079 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efea0145d10 0x7efea013bf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:19.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:19.875+0000 7efe97e00640 1 -- 172.21.15.16:0/4190542079 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efea00b3130 con 0x7efea00b23d0 2024-03-31T15:05:19.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:19.875+0000 7efea4c00640 1 --2- 172.21.15.16:0/4190542079 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efea0145d10 0x7efea013bf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:05:19.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:19.875+0000 7efe9fe00640 1 --2- 172.21.15.16:0/4190542079 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7efea00728b0 0x7efea00b4d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:05:19.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:19.875+0000 7efe97e00640 1 --2- 172.21.15.16:0/4190542079 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efea00b23d0 0x7efea01fadc0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7efe8800c420 tx=0x7efe8800c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:05:19.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:19.875+0000 7efe9cc00640 1 -- 172.21.15.16:0/4190542079 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe88017070 con 0x7efea00b23d0 2024-03-31T15:05:19.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:19.875+0000 7efe9cc00640 1 -- 172.21.15.16:0/4190542079 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efe88005cf0 con 0x7efea00b23d0 2024-03-31T15:05:19.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:19.875+0000 7efe9cc00640 1 -- 172.21.15.16:0/4190542079 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe880072f0 con 0x7efea00b23d0 2024-03-31T15:05:19.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:19.875+0000 7efea6c00640 1 -- 172.21.15.16:0/4190542079 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efea01fb300 con 0x7efea00b23d0 2024-03-31T15:05:19.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:19.879+0000 7efea6c00640 1 -- 172.21.15.16:0/4190542079 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efea01fb880 con 0x7efea00b23d0 2024-03-31T15:05:19.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:19.879+0000 7efea6c00640 1 -- 172.21.15.16:0/4190542079 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efea0069950 con 0x7efea00b23d0 2024-03-31T15:05:19.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:19.879+0000 7efe9cc00640 1 -- 172.21.15.16:0/4190542079 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7efe8801a070 con 0x7efea00b23d0 2024-03-31T15:05:19.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:19.879+0000 7efe9cc00640 1 --2- 172.21.15.16:0/4190542079 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7efe7c068380 0x7efe7c06a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:19.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:19.879+0000 7efe9cc00640 1 -- 172.21.15.16:0/4190542079 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 4943+0+0 (secure 0 0 0) 0x7efe88085a90 con 0x7efea00b23d0 2024-03-31T15:05:19.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:19.879+0000 7efe9fe00640 1 --2- 172.21.15.16:0/4190542079 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7efe7c068380 0x7efe7c06a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:19.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:19.879+0000 7efe9fe00640 1 --2- 172.21.15.16:0/4190542079 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7efe7c068380 0x7efe7c06a840 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7efe900ad000 tx=0x7efe900a6000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:05:19.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:19.883+0000 7efe9cc00640 1 -- 172.21.15.16:0/4190542079 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7efe88056340 con 0x7efea00b23d0 2024-03-31T15:05:20.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:20.107+0000 7efea6c00640 1 -- 172.21.15.16:0/4190542079 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_backtrace_repair"]} v 0) -- 0x7efea015cfa0 con 0x7efea00b23d0 2024-03-31T15:05:21.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:21.019+0000 7efe9cc00640 1 -- 172.21.15.16:0/4190542079 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_backtrace_repair"]}]=0 v115) ==== 152+0+0 (secure 0 0 0) 0x7efe88055c40 con 0x7efea00b23d0 2024-03-31T15:05:21.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:21.023+0000 7efea6c00640 1 -- 172.21.15.16:0/4190542079 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7efe7c068380 msgr2=0x7efe7c06a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:21.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:21.023+0000 7efea6c00640 1 --2- 172.21.15.16:0/4190542079 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7efe7c068380 0x7efe7c06a840 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7efe900ad000 tx=0x7efe900a6000 comp rx=0 tx=0).stop 2024-03-31T15:05:21.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:21.023+0000 7efea6c00640 1 -- 172.21.15.16:0/4190542079 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efea00b23d0 msgr2=0x7efea01fadc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:21.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:21.023+0000 7efea6c00640 1 --2- 172.21.15.16:0/4190542079 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efea00b23d0 0x7efea01fadc0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7efe8800c420 tx=0x7efe8800c8e0 comp rx=0 tx=0).stop 2024-03-31T15:05:21.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:21.023+0000 7efea6c00640 1 -- 172.21.15.16:0/4190542079 shutdown_connections 2024-03-31T15:05:21.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:21.023+0000 7efea6c00640 1 --2- 172.21.15.16:0/4190542079 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7efe7c068380 0x7efe7c06a840 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:21.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:21.023+0000 7efea6c00640 1 --2- 172.21.15.16:0/4190542079 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efea0145d10 0x7efea013bf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:21.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:21.023+0000 7efea6c00640 1 --2- 172.21.15.16:0/4190542079 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efea00b23d0 0x7efea01fadc0 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:21.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:21.023+0000 7efea6c00640 1 --2- 172.21.15.16:0/4190542079 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7efea00728b0 0x7efea00b4d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:21.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:21.023+0000 7efea6c00640 1 -- 172.21.15.16:0/4190542079 >> 172.21.15.16:0/4190542079 conn(0x7efea006b620 msgr2=0x7efea0167980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:05:21.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:21.023+0000 7efea6c00640 1 -- 172.21.15.16:0/4190542079 shutdown_connections 2024-03-31T15:05:21.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:21.023+0000 7efea6c00640 1 -- 172.21.15.16:0/4190542079 wait complete. 2024-03-31T15:05:21.042 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-03-31T15:05:21.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:21.195+0000 7f8336a00640 1 Processor -- start 2024-03-31T15:05:21.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:21.195+0000 7f8336a00640 1 -- start start 2024-03-31T15:05:21.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:21.195+0000 7f8336a00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f8330147d60 0x7f83300b3130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:21.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:21.195+0000 7f8336a00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f8330073310 0x7f833006d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:21.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:21.195+0000 7f8336a00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f833006da10 0x7f8330161f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:21.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:21.195+0000 7f8336a00640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f8330148140 con 0x7f8330073310 2024-03-31T15:05:21.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:21.195+0000 7f8336a00640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f833006de60 con 0x7f8330147d60 2024-03-31T15:05:21.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:21.195+0000 7f8336a00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f83300737c0 con 0x7f833006da10 2024-03-31T15:05:21.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:21.195+0000 7f8327400640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f8330073310 0x7f833006d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:21.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:21.195+0000 7f8327400640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f8330073310 0x7f833006d4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49322/0 (socket says 172.21.15.16:49322) 2024-03-31T15:05:21.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:21.195+0000 7f8327400640 1 -- 172.21.15.16:0/4057402946 learned_addr learned my addr 172.21.15.16:0/4057402946 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:05:21.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:21.195+0000 7f8334a00640 1 --2- 172.21.15.16:0/4057402946 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f833006da10 0x7f8330161f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:21.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:21.195+0000 7f8327400640 1 -- 172.21.15.16:0/4057402946 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f833006da10 msgr2=0x7f8330161f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:05:21.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:21.195+0000 7f832fe00640 1 --2- 172.21.15.16:0/4057402946 >> v2:172.21.15.89:3300/0 conn(0x7f8330147d60 0x7f83300b3130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:21.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:21.195+0000 7f8327400640 1 --2- 172.21.15.16:0/4057402946 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f833006da10 0x7f8330161f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:21.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:21.195+0000 7f8327400640 1 -- 172.21.15.16:0/4057402946 >> v2:172.21.15.89:3300/0 conn(0x7f8330147d60 msgr2=0x7f83300b3130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:05:21.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:21.195+0000 7f8327400640 1 --2- 172.21.15.16:0/4057402946 >> v2:172.21.15.89:3300/0 conn(0x7f8330147d60 0x7f83300b3130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:21.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:21.195+0000 7f8327400640 1 -- 172.21.15.16:0/4057402946 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83300b37c0 con 0x7f8330073310 2024-03-31T15:05:21.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:21.195+0000 7f8334a00640 1 --2- 172.21.15.16:0/4057402946 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f833006da10 0x7f8330161f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:05:21.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:21.195+0000 7f832fe00640 1 --2- 172.21.15.16:0/4057402946 >> v2:172.21.15.89:3300/0 conn(0x7f8330147d60 0x7f83300b3130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:05:21.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:21.199+0000 7f8327400640 1 --2- 172.21.15.16:0/4057402946 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8330073310 0x7f833006d4d0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f83300b5590 tx=0x7f831801c230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fffc06b0764a193b server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:05:21.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:21.199+0000 7f832f400640 1 -- 172.21.15.16:0/4057402946 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83180b0070 con 0x7f8330073310 2024-03-31T15:05:21.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:21.199+0000 7f832f400640 1 -- 172.21.15.16:0/4057402946 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f83180ab600 con 0x7f8330073310 2024-03-31T15:05:21.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:21.199+0000 7f832f400640 1 -- 172.21.15.16:0/4057402946 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83180ab9d0 con 0x7f8330073310 2024-03-31T15:05:21.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:21.199+0000 7f8336a00640 1 -- 172.21.15.16:0/4057402946 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8330073310 msgr2=0x7f833006d4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:21.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:21.199+0000 7f8336a00640 1 --2- 172.21.15.16:0/4057402946 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8330073310 0x7f833006d4d0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f83300b5590 tx=0x7f831801c230 comp rx=0 tx=0).stop 2024-03-31T15:05:21.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:21.199+0000 7f8336a00640 1 -- 172.21.15.16:0/4057402946 shutdown_connections 2024-03-31T15:05:21.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:21.199+0000 7f8336a00640 1 --2- 172.21.15.16:0/4057402946 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f833006da10 0x7f8330161f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:21.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:21.199+0000 7f8336a00640 1 --2- 172.21.15.16:0/4057402946 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8330073310 0x7f833006d4d0 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:21.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:21.199+0000 7f8336a00640 1 --2- 172.21.15.16:0/4057402946 >> v2:172.21.15.89:3300/0 conn(0x7f8330147d60 0x7f83300b3130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:21.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:21.199+0000 7f8336a00640 1 -- 172.21.15.16:0/4057402946 >> 172.21.15.16:0/4057402946 conn(0x7f833006ee40 msgr2=0x7f833006b620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:05:21.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:21.199+0000 7f8336a00640 1 -- 172.21.15.16:0/4057402946 shutdown_connections 2024-03-31T15:05:21.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:21.199+0000 7f8336a00640 1 -- 172.21.15.16:0/4057402946 wait complete. 2024-03-31T15:05:21.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:21.203+0000 7f8336a00640 1 Processor -- start 2024-03-31T15:05:21.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:21.203+0000 7f8336a00640 1 -- start start 2024-03-31T15:05:21.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:21.203+0000 7f8336a00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f833006da10 0x7f833015cf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:21.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:21.203+0000 7f8336a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8330073310 0x7f83300ae2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:21.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:21.203+0000 7f8336a00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8330147d60 0x7f8330141c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:21.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:21.203+0000 7f8336a00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f83300ada40 con 0x7f8330073310 2024-03-31T15:05:21.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:21.203+0000 7f8336a00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f83300b4340 con 0x7f8330147d60 2024-03-31T15:05:21.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:21.203+0000 7f8336a00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f83300adbc0 con 0x7f833006da10 2024-03-31T15:05:21.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:21.203+0000 7f8327400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8330073310 0x7f83300ae2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:21.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:21.203+0000 7f8327400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8330073310 0x7f83300ae2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49326/0 (socket says 172.21.15.16:49326) 2024-03-31T15:05:21.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:21.203+0000 7f8327400640 1 -- 172.21.15.16:0/4242887512 learned_addr learned my addr 172.21.15.16:0/4242887512 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:05:21.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:21.203+0000 7f832fe00640 1 --2- 172.21.15.16:0/4242887512 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f833006da10 0x7f833015cf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:21.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:21.203+0000 7f8334a00640 1 --2- 172.21.15.16:0/4242887512 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8330147d60 0x7f8330141c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:21.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:21.203+0000 7f8327400640 1 -- 172.21.15.16:0/4242887512 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f833006da10 msgr2=0x7f833015cf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:21.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:21.203+0000 7f8327400640 1 --2- 172.21.15.16:0/4242887512 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f833006da10 0x7f833015cf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:21.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:21.203+0000 7f8327400640 1 -- 172.21.15.16:0/4242887512 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8330147d60 msgr2=0x7f8330141c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:21.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:21.203+0000 7f8327400640 1 --2- 172.21.15.16:0/4242887512 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8330147d60 0x7f8330141c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:21.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:21.203+0000 7f8327400640 1 -- 172.21.15.16:0/4242887512 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f833015d7c0 con 0x7f8330073310 2024-03-31T15:05:21.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:21.203+0000 7f8334a00640 1 --2- 172.21.15.16:0/4242887512 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8330147d60 0x7f8330141c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:05:21.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:21.203+0000 7f8327400640 1 --2- 172.21.15.16:0/4242887512 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8330073310 0x7f83300ae2b0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f831801a8e0 tx=0x7f8318018fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:05:21.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:21.203+0000 7f832cc00640 1 -- 172.21.15.16:0/4242887512 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83180b9070 con 0x7f8330073310 2024-03-31T15:05:21.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:21.203+0000 7f832cc00640 1 -- 172.21.15.16:0/4242887512 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f83180b4600 con 0x7f8330073310 2024-03-31T15:05:21.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:21.203+0000 7f832cc00640 1 -- 172.21.15.16:0/4242887512 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83180b0040 con 0x7f8330073310 2024-03-31T15:05:21.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:21.203+0000 7f8336a00640 1 -- 172.21.15.16:0/4242887512 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f833015daa0 con 0x7f8330073310 2024-03-31T15:05:21.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:21.203+0000 7f8336a00640 1 -- 172.21.15.16:0/4242887512 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f833015e060 con 0x7f8330073310 2024-03-31T15:05:21.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:21.207+0000 7f8336a00640 1 -- 172.21.15.16:0/4242887512 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8330069950 con 0x7f8330073310 2024-03-31T15:05:21.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:21.207+0000 7f832cc00640 1 -- 172.21.15.16:0/4242887512 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f83180ab4c0 con 0x7f8330073310 2024-03-31T15:05:21.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:21.207+0000 7f832cc00640 1 --2- 172.21.15.16:0/4242887512 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f830c068380 0x7f830c06a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:21.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:21.207+0000 7f832cc00640 1 -- 172.21.15.16:0/4242887512 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 4943+0+0 (secure 0 0 0) 0x7f831811f950 con 0x7f8330073310 2024-03-31T15:05:21.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:21.207+0000 7f832fe00640 1 --2- 172.21.15.16:0/4242887512 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f830c068380 0x7f830c06a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:21.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:21.207+0000 7f832fe00640 1 --2- 172.21.15.16:0/4242887512 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f830c068380 0x7f830c06a840 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f832000f500 tx=0x7f8320007770 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:05:21.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:21.211+0000 7f832cc00640 1 -- 172.21.15.16:0/4242887512 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f8318016110 con 0x7f8330073310 2024-03-31T15:05:21.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:21.375+0000 7f8336a00640 1 -- 172.21.15.16:0/4242887512 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7f833006e470 con 0x7f8330073310 2024-03-31T15:05:21.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:21.379+0000 7f832cc00640 1 -- 172.21.15.16:0/4242887512 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v12)=0 v12) ==== 122+0+0 (secure 0 0 0) 0x7f83180efb30 con 0x7f8330073310 2024-03-31T15:05:21.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:21.379+0000 7f8336a00640 1 -- 172.21.15.16:0/4242887512 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f830c068380 msgr2=0x7f830c06a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:21.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:21.379+0000 7f8336a00640 1 --2- 172.21.15.16:0/4242887512 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f830c068380 0x7f830c06a840 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f832000f500 tx=0x7f8320007770 comp rx=0 tx=0).stop 2024-03-31T15:05:21.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:21.383+0000 7f8336a00640 1 -- 172.21.15.16:0/4242887512 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8330073310 msgr2=0x7f83300ae2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:21.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:21.383+0000 7f8336a00640 1 --2- 172.21.15.16:0/4242887512 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8330073310 0x7f83300ae2b0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f831801a8e0 tx=0x7f8318018fd0 comp rx=0 tx=0).stop 2024-03-31T15:05:21.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:21.383+0000 7f8336a00640 1 -- 172.21.15.16:0/4242887512 shutdown_connections 2024-03-31T15:05:21.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:21.383+0000 7f8336a00640 1 --2- 172.21.15.16:0/4242887512 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f830c068380 0x7f830c06a840 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:21.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:21.383+0000 7f8336a00640 1 --2- 172.21.15.16:0/4242887512 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8330147d60 0x7f8330141c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:21.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:21.383+0000 7f8336a00640 1 --2- 172.21.15.16:0/4242887512 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8330073310 0x7f83300ae2b0 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:21.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:21.383+0000 7f8336a00640 1 --2- 172.21.15.16:0/4242887512 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f833006da10 0x7f833015cf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:21.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:21.383+0000 7f8336a00640 1 -- 172.21.15.16:0/4242887512 >> 172.21.15.16:0/4242887512 conn(0x7f833006ee40 msgr2=0x7f8330147030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:05:21.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:21.383+0000 7f8336a00640 1 -- 172.21.15.16:0/4242887512 shutdown_connections 2024-03-31T15:05:21.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:21.383+0000 7f8336a00640 1 -- 172.21.15.16:0/4242887512 wait complete. 2024-03-31T15:05:21.396 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-31T15:05:21.396 DEBUG:teuthology.orchestra.run.smithi016:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-31T15:05:21.466 DEBUG:teuthology.orchestra.run.smithi089:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-31T15:05:21.557 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-31T15:05:21.557 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-03-31T15:05:21.558 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-31T15:05:21.566 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16357708k,nr_inodes=4089427,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275996k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=6539 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275992k,nr_inodes=818998,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275996k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-31T15:05:21.567 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-03-31T15:05:21.567 DEBUG:teuthology.orchestra.run.smithi089:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-31T15:05:21.576 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16354452k,nr_inodes=4088613,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275344k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=901 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275340k,nr_inodes=818835,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275344k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-31T15:05:21.576 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-03-31T15:05:21.576 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-31T15:05:21.616 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16357708k,nr_inodes=4089427,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275996k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=6539 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275992k,nr_inodes=818998,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275996k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-31T15:05:21.616 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-03-31T15:05:21.616 DEBUG:teuthology.orchestra.run.smithi089:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-31T15:05:21.630 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16354452k,nr_inodes=4088613,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275344k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=901 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275340k,nr_inodes=818835,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275344k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-31T15:05:21.631 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-03-31T15:05:21.631 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-31T15:05:21.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:21.803+0000 7f8c37400640 1 Processor -- start 2024-03-31T15:05:21.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:21.803+0000 7f8c37400640 1 -- start start 2024-03-31T15:05:21.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:21.807+0000 7f8c37400640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f8c30145d10 0x7f8c300cf790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:21.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:21.807+0000 7f8c37400640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f8c300728b0 0x7f8c300b1cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:21.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:21.807+0000 7f8c37400640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f8c300b2200 0x7f8c3015cd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:21.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:21.807+0000 7f8c37400640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f8c30160e70 con 0x7f8c30145d10 2024-03-31T15:05:21.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:21.807+0000 7f8c37400640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f8c300cfcd0 con 0x7f8c300728b0 2024-03-31T15:05:21.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:21.807+0000 7f8c37400640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f8c301602f0 con 0x7f8c300b2200 2024-03-31T15:05:21.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:21.807+0000 7f8c34a00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f8c30145d10 0x7f8c300cf790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:21.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:21.807+0000 7f8c34a00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f8c30145d10 0x7f8c300cf790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49336/0 (socket says 172.21.15.16:49336) 2024-03-31T15:05:21.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:21.807+0000 7f8c34a00640 1 -- 172.21.15.16:0/820198609 learned_addr learned my addr 172.21.15.16:0/820198609 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:05:21.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:21.807+0000 7f8c2fe00640 1 --2- 172.21.15.16:0/820198609 >> v2:172.21.15.89:3300/0 conn(0x7f8c300728b0 0x7f8c300b1cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:21.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:21.807+0000 7f8c35400640 1 --2- 172.21.15.16:0/820198609 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f8c300b2200 0x7f8c3015cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:21.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:21.807+0000 7f8c34a00640 1 -- 172.21.15.16:0/820198609 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f8c300b2200 msgr2=0x7f8c3015cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:05:21.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:21.807+0000 7f8c34a00640 1 --2- 172.21.15.16:0/820198609 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f8c300b2200 0x7f8c3015cd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:21.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:21.807+0000 7f8c34a00640 1 -- 172.21.15.16:0/820198609 >> v2:172.21.15.89:3300/0 conn(0x7f8c300728b0 msgr2=0x7f8c300b1cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:05:21.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:21.807+0000 7f8c34a00640 1 --2- 172.21.15.16:0/820198609 >> v2:172.21.15.89:3300/0 conn(0x7f8c300728b0 0x7f8c300b1cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:21.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:21.807+0000 7f8c34a00640 1 -- 172.21.15.16:0/820198609 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c301605d0 con 0x7f8c30145d10 2024-03-31T15:05:21.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:21.807+0000 7f8c2fe00640 1 --2- 172.21.15.16:0/820198609 >> v2:172.21.15.89:3300/0 conn(0x7f8c300728b0 0x7f8c300b1cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:05:21.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:21.807+0000 7f8c35400640 1 --2- 172.21.15.16:0/820198609 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f8c300b2200 0x7f8c3015cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:05:21.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:21.811+0000 7f8c34a00640 1 --2- 172.21.15.16:0/820198609 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8c30145d10 0x7f8c300cf790 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f8c300b2980 tx=0x7f8c2401c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d9cf4f860bc668c server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:05:21.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:21.811+0000 7f8c2f400640 1 -- 172.21.15.16:0/820198609 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c240b0070 con 0x7f8c30145d10 2024-03-31T15:05:21.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:21.811+0000 7f8c2f400640 1 -- 172.21.15.16:0/820198609 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8c240ab600 con 0x7f8c30145d10 2024-03-31T15:05:21.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:21.811+0000 7f8c2f400640 1 -- 172.21.15.16:0/820198609 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c240ab9d0 con 0x7f8c30145d10 2024-03-31T15:05:21.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:21.811+0000 7f8c37400640 1 -- 172.21.15.16:0/820198609 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8c30145d10 msgr2=0x7f8c300cf790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:21.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:21.811+0000 7f8c37400640 1 --2- 172.21.15.16:0/820198609 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8c30145d10 0x7f8c300cf790 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f8c300b2980 tx=0x7f8c2401c520 comp rx=0 tx=0).stop 2024-03-31T15:05:21.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:21.811+0000 7f8c37400640 1 -- 172.21.15.16:0/820198609 shutdown_connections 2024-03-31T15:05:21.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:21.811+0000 7f8c37400640 1 --2- 172.21.15.16:0/820198609 >> v2:172.21.15.89:3300/0 conn(0x7f8c300728b0 0x7f8c300b1cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:21.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:21.811+0000 7f8c37400640 1 --2- 172.21.15.16:0/820198609 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f8c300b2200 0x7f8c3015cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:21.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:21.811+0000 7f8c37400640 1 --2- 172.21.15.16:0/820198609 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8c30145d10 0x7f8c300cf790 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:21.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:21.811+0000 7f8c37400640 1 -- 172.21.15.16:0/820198609 >> 172.21.15.16:0/820198609 conn(0x7f8c3006b620 msgr2=0x7f8c3006ba30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:05:21.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:21.811+0000 7f8c37400640 1 -- 172.21.15.16:0/820198609 shutdown_connections 2024-03-31T15:05:21.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:21.811+0000 7f8c37400640 1 -- 172.21.15.16:0/820198609 wait complete. 2024-03-31T15:05:21.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:21.811+0000 7f8c37400640 1 Processor -- start 2024-03-31T15:05:21.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:21.815+0000 7f8c37400640 1 -- start start 2024-03-31T15:05:21.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:21.815+0000 7f8c37400640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8c300728b0 0x7f8c3015b670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:21.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:21.815+0000 7f8c37400640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f8c300b2200 0x7f8c300b47a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:21.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:21.815+0000 7f8c37400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8c30145d10 0x7f8c301654a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:21.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:21.815+0000 7f8c37400640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f8c300cf350 con 0x7f8c30145d10 2024-03-31T15:05:21.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:21.815+0000 7f8c37400640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f8c300b2610 con 0x7f8c300728b0 2024-03-31T15:05:21.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:21.815+0000 7f8c37400640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f8c300cf4d0 con 0x7f8c300b2200 2024-03-31T15:05:21.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:21.815+0000 7f8c35400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8c30145d10 0x7f8c301654a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:21.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:21.815+0000 7f8c35400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8c30145d10 0x7f8c301654a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49352/0 (socket says 172.21.15.16:49352) 2024-03-31T15:05:21.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:21.815+0000 7f8c35400640 1 -- 172.21.15.16:0/3663934466 learned_addr learned my addr 172.21.15.16:0/3663934466 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:05:21.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:21.815+0000 7f8c2fe00640 1 --2- 172.21.15.16:0/3663934466 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f8c300b2200 0x7f8c300b47a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:21.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:21.815+0000 7f8c34a00640 1 --2- 172.21.15.16:0/3663934466 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8c300728b0 0x7f8c3015b670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:21.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:21.815+0000 7f8c35400640 1 -- 172.21.15.16:0/3663934466 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f8c300b2200 msgr2=0x7f8c300b47a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:21.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:21.815+0000 7f8c35400640 1 --2- 172.21.15.16:0/3663934466 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f8c300b2200 0x7f8c300b47a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:21.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:21.815+0000 7f8c35400640 1 -- 172.21.15.16:0/3663934466 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8c300728b0 msgr2=0x7f8c3015b670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:21.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:21.815+0000 7f8c35400640 1 --2- 172.21.15.16:0/3663934466 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8c300728b0 0x7f8c3015b670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:21.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:21.815+0000 7f8c35400640 1 -- 172.21.15.16:0/3663934466 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c300b4ce0 con 0x7f8c30145d10 2024-03-31T15:05:21.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:21.815+0000 7f8c35400640 1 --2- 172.21.15.16:0/3663934466 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8c30145d10 0x7f8c301654a0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f8c2800cef0 tx=0x7f8c2800a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:05:21.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:21.815+0000 7f8c2cc00640 1 -- 172.21.15.16:0/3663934466 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c2801a070 con 0x7f8c30145d10 2024-03-31T15:05:21.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:21.815+0000 7f8c2cc00640 1 -- 172.21.15.16:0/3663934466 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8c2800d860 con 0x7f8c30145d10 2024-03-31T15:05:21.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:21.815+0000 7f8c37400640 1 -- 172.21.15.16:0/3663934466 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8c30140610 con 0x7f8c30145d10 2024-03-31T15:05:21.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:21.815+0000 7f8c2cc00640 1 -- 172.21.15.16:0/3663934466 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c2800dc50 con 0x7f8c30145d10 2024-03-31T15:05:21.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:21.815+0000 7f8c37400640 1 -- 172.21.15.16:0/3663934466 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8c300b3910 con 0x7f8c30145d10 2024-03-31T15:05:21.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:21.815+0000 7f8c37400640 1 -- 172.21.15.16:0/3663934466 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8c301407e0 con 0x7f8c30145d10 2024-03-31T15:05:21.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:21.819+0000 7f8c2cc00640 1 -- 172.21.15.16:0/3663934466 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f8c2801d070 con 0x7f8c30145d10 2024-03-31T15:05:21.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:21.819+0000 7f8c2cc00640 1 --2- 172.21.15.16:0/3663934466 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f8c10068380 0x7f8c1006a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:21.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:21.819+0000 7f8c2cc00640 1 -- 172.21.15.16:0/3663934466 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 4943+0+0 (secure 0 0 0) 0x7f8c28086fe0 con 0x7f8c30145d10 2024-03-31T15:05:21.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:21.819+0000 7f8c34a00640 1 --2- 172.21.15.16:0/3663934466 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f8c10068380 0x7f8c1006a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:21.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:21.819+0000 7f8c34a00640 1 --2- 172.21.15.16:0/3663934466 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f8c10068380 0x7f8c1006a840 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f8c240ad000 tx=0x7f8c24008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:05:21.818 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:21.823+0000 7f8c2cc00640 1 -- 172.21.15.16:0/3663934466 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f8c28057200 con 0x7f8c30145d10 2024-03-31T15:05:22.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:22.023+0000 7f8c37400640 1 -- 172.21.15.16:0/3663934466 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8c30069950 con 0x7f8c30145d10 2024-03-31T15:05:22.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:22.023+0000 7f8c2cc00640 1 -- 172.21.15.16:0/3663934466 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 18 v18) ==== 94+0+3866 (secure 0 0 0) 0x7f8c2805b500 con 0x7f8c30145d10 2024-03-31T15:05:22.018 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:05:22.018 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":18,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4949,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6836/2073954753","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":2073954753},{"type":"v1","addr":"172.21.15.89:6836","nonce":2073954753}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":4952,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/2143918278","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6835","nonce":2143918278},{"type":"v1","addr":"172.21.15.89:6837","nonce":2143918278}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":5251,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6833/881391342","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":881391342},{"type":"v1","addr":"172.21.15.16:6833","nonce":881391342}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":5619,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/2266539587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":2266539587},{"type":"v1","addr":"172.21.15.16:6835","nonce":2266539587}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":18}],"filesystems":[]} 2024-03-31T15:05:22.018 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 18 2024-03-31T15:05:22.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:22.023+0000 7f8c37400640 1 -- 172.21.15.16:0/3663934466 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f8c10068380 msgr2=0x7f8c1006a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:22.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:22.023+0000 7f8c37400640 1 --2- 172.21.15.16:0/3663934466 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f8c10068380 0x7f8c1006a840 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f8c240ad000 tx=0x7f8c24008040 comp rx=0 tx=0).stop 2024-03-31T15:05:22.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:22.023+0000 7f8c37400640 1 -- 172.21.15.16:0/3663934466 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8c30145d10 msgr2=0x7f8c301654a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:22.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:22.023+0000 7f8c37400640 1 --2- 172.21.15.16:0/3663934466 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8c30145d10 0x7f8c301654a0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f8c2800cef0 tx=0x7f8c2800a710 comp rx=0 tx=0).stop 2024-03-31T15:05:22.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:22.023+0000 7f8c37400640 1 -- 172.21.15.16:0/3663934466 shutdown_connections 2024-03-31T15:05:22.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:22.023+0000 7f8c37400640 1 --2- 172.21.15.16:0/3663934466 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f8c10068380 0x7f8c1006a840 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:22.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:22.023+0000 7f8c37400640 1 --2- 172.21.15.16:0/3663934466 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8c30145d10 0x7f8c301654a0 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:22.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:22.023+0000 7f8c37400640 1 --2- 172.21.15.16:0/3663934466 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f8c300b2200 0x7f8c300b47a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:22.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:22.023+0000 7f8c37400640 1 --2- 172.21.15.16:0/3663934466 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8c300728b0 0x7f8c3015b670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:22.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:22.023+0000 7f8c37400640 1 -- 172.21.15.16:0/3663934466 >> 172.21.15.16:0/3663934466 conn(0x7f8c3006b620 msgr2=0x7f8c300b0260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:05:22.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:22.023+0000 7f8c37400640 1 -- 172.21.15.16:0/3663934466 shutdown_connections 2024-03-31T15:05:22.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:22.023+0000 7f8c37400640 1 -- 172.21.15.16:0/3663934466 wait complete. 2024-03-31T15:05:22.039 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-31T15:05:22.039 INFO:tasks.ceph.mds.a:Stopping old one... 2024-03-31T15:05:22.040 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-31T15:05:22.040 INFO:teuthology.orchestra.run:waiting for 300 2024-03-31T15:05:22.040 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-31T15:05:22.040 INFO:tasks.ceph.mds.c:Stopping old one... 2024-03-31T15:05:22.040 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-31T15:05:22.040 INFO:teuthology.orchestra.run:waiting for 300 2024-03-31T15:05:22.040 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-31T15:05:22.041 INFO:tasks.ceph.mds.b:Stopping old one... 2024-03-31T15:05:22.041 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-31T15:05:22.041 INFO:teuthology.orchestra.run:waiting for 300 2024-03-31T15:05:22.041 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-31T15:05:22.041 INFO:tasks.ceph.mds.d:Stopping old one... 2024-03-31T15:05:22.041 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-31T15:05:22.041 INFO:teuthology.orchestra.run:waiting for 300 2024-03-31T15:05:22.056 INFO:tasks.ceph.mds.a:Stopped 2024-03-31T15:05:22.056 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-31T15:05:22.057 INFO:tasks.ceph.mds.c:Stopped 2024-03-31T15:05:22.057 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-31T15:05:22.058 INFO:tasks.ceph.mds.b:Stopped 2024-03-31T15:05:22.059 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-31T15:05:22.059 INFO:tasks.ceph.mds.d:Stopped 2024-03-31T15:05:22.059 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-31T15:05:22.103 INFO:tasks.ceph.mds.a:Started 2024-03-31T15:05:22.103 DEBUG:teuthology.parallel:result is None 2024-03-31T15:05:22.104 INFO:tasks.ceph.mds.c:Started 2024-03-31T15:05:22.104 DEBUG:teuthology.parallel:result is None 2024-03-31T15:05:22.106 INFO:tasks.ceph.mds.b:Started 2024-03-31T15:05:22.106 DEBUG:teuthology.parallel:result is None 2024-03-31T15:05:22.108 INFO:tasks.ceph.mds.d:Started 2024-03-31T15:05:22.108 DEBUG:teuthology.parallel:result is None 2024-03-31T15:05:22.108 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-03-31T15:05:22.189 INFO:tasks.ceph.mds.a.smithi016.stdout:starting mds.a at 2024-03-31T15:05:22.189 INFO:tasks.ceph.mds.c.smithi016.stdout:starting mds.c at 2024-03-31T15:05:22.198 INFO:tasks.ceph.mds.b.smithi089.stdout:starting mds.b at 2024-03-31T15:05:22.199 INFO:tasks.ceph.mds.d.smithi089.stdout:starting mds.d at 2024-03-31T15:05:22.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:22.259+0000 7f5b7a000640 1 Processor -- start 2024-03-31T15:05:22.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:22.259+0000 7f5b7a000640 1 -- start start 2024-03-31T15:05:22.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:22.259+0000 7f5b7a000640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f5b7415e370 0x7f5b740cf4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:22.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:22.259+0000 7f5b7a000640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5b7415ebd0 con 0x7f5b7415e750 2024-03-31T15:05:22.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:22.259+0000 7f5b7a000640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5b7415ee80 con 0x7f5b740af070 2024-03-31T15:05:22.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:22.259+0000 7f5b7a000640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f5b7415f160 con 0x7f5b7415e370 2024-03-31T15:05:22.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:22.259+0000 7f5b6be00640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f5b7415e750 0x7f5b740cd3a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:55540/0 (socket says 172.21.15.16:55540) 2024-03-31T15:05:22.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:22.259+0000 7f5b6be00640 1 -- 172.21.15.16:0/94755425 learned_addr learned my addr 172.21.15.16:0/94755425 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:05:22.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:22.259+0000 7f5b73400640 1 --2- 172.21.15.16:0/94755425 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f5b7415e370 0x7f5b740cf4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:22.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:22.259+0000 7f5b72a00640 1 -- 172.21.15.16:0/94755425 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3416051805 0 0) 0x7f5b7415ebd0 con 0x7f5b7415e750 2024-03-31T15:05:22.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:22.259+0000 7f5b72a00640 1 -- 172.21.15.16:0/94755425 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5b58003530 con 0x7f5b7415e750 2024-03-31T15:05:22.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:22.259+0000 7f5b72a00640 1 -- 172.21.15.16:0/94755425 <== mon.1 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 445412279 0 0) 0x7f5b7415ee80 con 0x7f5b740af070 2024-03-31T15:05:22.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:22.259+0000 7f5b73400640 1 -- 172.21.15.16:0/94755425 >> v1:172.21.15.89:6789/0 conn(0x7f5b740af070 legacy=0x7f5b740af500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:22.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:22.259+0000 7f5b73400640 1 -- 172.21.15.16:0/94755425 >> v1:172.21.15.16:6789/0 conn(0x7f5b7415e750 legacy=0x7f5b740cd3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:22.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:22.259+0000 7f5b73400640 1 -- 172.21.15.16:0/94755425 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b7415f440 con 0x7f5b7415e370 2024-03-31T15:05:22.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:22.263+0000 7f5b73400640 1 --2- 172.21.15.16:0/94755425 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f5b7415e370 0x7f5b740cf4f0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f5b640097f0 tx=0x7f5b6401c4e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fe4bf0225dd8e88 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:05:22.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:22.263+0000 7f5b72a00640 1 -- 172.21.15.16:0/94755425 <== mon.2 v2:172.21.15.89:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b640b0070 con 0x7f5b7415e370 2024-03-31T15:05:22.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:22.263+0000 7f5b72a00640 1 -- 172.21.15.16:0/94755425 <== mon.2 v2:172.21.15.89:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5b640ab600 con 0x7f5b7415e370 2024-03-31T15:05:22.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:22.263+0000 7f5b72a00640 1 -- 172.21.15.16:0/94755425 <== mon.2 v2:172.21.15.89:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b640ab9d0 con 0x7f5b7415e370 2024-03-31T15:05:22.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:22.263+0000 7f5b7a000640 1 -- 172.21.15.16:0/94755425 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f5b7415e370 msgr2=0x7f5b740cf4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:22.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:22.263+0000 7f5b7a000640 1 --2- 172.21.15.16:0/94755425 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f5b7415e370 0x7f5b740cf4f0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f5b640097f0 tx=0x7f5b6401c4e0 comp rx=0 tx=0).stop 2024-03-31T15:05:22.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:22.263+0000 7f5b7a000640 1 -- 172.21.15.16:0/94755425 shutdown_connections 2024-03-31T15:05:22.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:22.263+0000 7f5b7a000640 1 --2- 172.21.15.16:0/94755425 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f5b7415e370 0x7f5b740cf4f0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:22.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:22.263+0000 7f5b7a000640 1 -- 172.21.15.16:0/94755425 >> 172.21.15.16:0/94755425 conn(0x7f5b7406b620 msgr2=0x7f5b7406ba30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:05:22.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:22.263+0000 7f5b7a000640 1 -- 172.21.15.16:0/94755425 shutdown_connections 2024-03-31T15:05:22.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:22.263+0000 7f5b7a000640 1 -- 172.21.15.16:0/94755425 wait complete. 2024-03-31T15:05:22.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:22.267+0000 7f5b7a000640 1 Processor -- start 2024-03-31T15:05:22.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:22.267+0000 7f5b7a000640 1 -- start start 2024-03-31T15:05:22.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:22.267+0000 7f5b7a000640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f5b740af070 0x7f5b741578b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:22.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:22.267+0000 7f5b7a000640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5b7415e370 0x7f5b74159c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:22.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:22.267+0000 7f5b7a000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5b7415e750 0x7f5b740b2de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:22.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:22.267+0000 7f5b7a000640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f5b74162f40 con 0x7f5b7415e750 2024-03-31T15:05:22.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:22.267+0000 7f5b7a000640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f5b74162dc0 con 0x7f5b7415e370 2024-03-31T15:05:22.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:22.267+0000 7f5b7a000640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f5b741630c0 con 0x7f5b740af070 2024-03-31T15:05:22.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:22.267+0000 7f5b73e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5b7415e750 0x7f5b740b2de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:22.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:22.267+0000 7f5b73e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5b7415e750 0x7f5b740b2de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49430/0 (socket says 172.21.15.16:49430) 2024-03-31T15:05:22.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:22.267+0000 7f5b73e00640 1 -- 172.21.15.16:0/1891501644 learned_addr learned my addr 172.21.15.16:0/1891501644 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:05:22.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:22.267+0000 7f5b6be00640 1 --2- 172.21.15.16:0/1891501644 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5b7415e370 0x7f5b74159c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:22.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:22.267+0000 7f5b73400640 1 --2- 172.21.15.16:0/1891501644 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f5b740af070 0x7f5b741578b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:22.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:22.267+0000 7f5b73e00640 1 -- 172.21.15.16:0/1891501644 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f5b740af070 msgr2=0x7f5b741578b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:22.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:22.267+0000 7f5b73e00640 1 --2- 172.21.15.16:0/1891501644 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f5b740af070 0x7f5b741578b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:22.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:22.267+0000 7f5b73e00640 1 -- 172.21.15.16:0/1891501644 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5b7415e370 msgr2=0x7f5b74159c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:22.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:22.267+0000 7f5b73e00640 1 --2- 172.21.15.16:0/1891501644 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5b7415e370 0x7f5b74159c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:22.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:22.267+0000 7f5b73e00640 1 -- 172.21.15.16:0/1891501644 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b74158220 con 0x7f5b7415e750 2024-03-31T15:05:22.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:22.267+0000 7f5b73e00640 1 --2- 172.21.15.16:0/1891501644 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5b7415e750 0x7f5b740b2de0 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f5b6c00c780 tx=0x7f5b6c00cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:05:22.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:22.267+0000 7f5b6b400640 1 -- 172.21.15.16:0/1891501644 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b6c003db0 con 0x7f5b7415e750 2024-03-31T15:05:22.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:22.267+0000 7f5b6b400640 1 -- 172.21.15.16:0/1891501644 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5b6c00ce30 con 0x7f5b7415e750 2024-03-31T15:05:22.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:22.267+0000 7f5b6b400640 1 -- 172.21.15.16:0/1891501644 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b6c005290 con 0x7f5b7415e750 2024-03-31T15:05:22.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:22.267+0000 7f5b7a000640 1 -- 172.21.15.16:0/1891501644 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5b7415bf10 con 0x7f5b7415e750 2024-03-31T15:05:22.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:22.267+0000 7f5b7a000640 1 -- 172.21.15.16:0/1891501644 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5b74157df0 con 0x7f5b7415e750 2024-03-31T15:05:22.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:22.271+0000 7f5b7a000640 1 -- 172.21.15.16:0/1891501644 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5b7415bd00 con 0x7f5b7415e750 2024-03-31T15:05:22.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:22.271+0000 7f5b6b400640 1 -- 172.21.15.16:0/1891501644 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f5b6c005430 con 0x7f5b7415e750 2024-03-31T15:05:22.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:22.271+0000 7f5b6b400640 1 --2- 172.21.15.16:0/1891501644 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f5b50068380 0x7f5b5006a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:22.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:22.271+0000 7f5b6b400640 1 -- 172.21.15.16:0/1891501644 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 4943+0+0 (secure 0 0 0) 0x7f5b6c086450 con 0x7f5b7415e750 2024-03-31T15:05:22.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:22.271+0000 7f5b73400640 1 --2- 172.21.15.16:0/1891501644 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f5b50068380 0x7f5b5006a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:22.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:22.271+0000 7f5b73400640 1 --2- 172.21.15.16:0/1891501644 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f5b50068380 0x7f5b5006a840 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f5b640ad000 tx=0x7f5b640a6000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:05:22.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:22.275+0000 7f5b6b400640 1 -- 172.21.15.16:0/1891501644 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f5b6c0566b0 con 0x7f5b7415e750 2024-03-31T15:05:22.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:22.451+0000 7f5b7a000640 1 -- 172.21.15.16:0/1891501644 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) -- 0x7f5b74069950 con 0x7f5b7415e750 2024-03-31T15:05:23.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:23.039+0000 7f5b6b400640 1 -- 172.21.15.16:0/1891501644 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v39) ==== 97+0+0 (secure 0 0 0) 0x7f5b6c05a9b0 con 0x7f5b7415e750 2024-03-31T15:05:23.036 INFO:teuthology.orchestra.run.smithi016.stderr: removed all blocklist entries 2024-03-31T15:05:23.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:23.043+0000 7f5b7a000640 1 -- 172.21.15.16:0/1891501644 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f5b50068380 msgr2=0x7f5b5006a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:23.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:23.043+0000 7f5b7a000640 1 --2- 172.21.15.16:0/1891501644 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f5b50068380 0x7f5b5006a840 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f5b640ad000 tx=0x7f5b640a6000 comp rx=0 tx=0).stop 2024-03-31T15:05:23.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:23.043+0000 7f5b7a000640 1 -- 172.21.15.16:0/1891501644 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5b7415e750 msgr2=0x7f5b740b2de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:23.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:23.043+0000 7f5b7a000640 1 --2- 172.21.15.16:0/1891501644 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5b7415e750 0x7f5b740b2de0 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f5b6c00c780 tx=0x7f5b6c00cc40 comp rx=0 tx=0).stop 2024-03-31T15:05:23.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:23.043+0000 7f5b7a000640 1 -- 172.21.15.16:0/1891501644 shutdown_connections 2024-03-31T15:05:23.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:23.043+0000 7f5b7a000640 1 --2- 172.21.15.16:0/1891501644 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f5b50068380 0x7f5b5006a840 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:23.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:23.043+0000 7f5b7a000640 1 --2- 172.21.15.16:0/1891501644 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5b7415e750 0x7f5b740b2de0 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:23.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:23.043+0000 7f5b7a000640 1 --2- 172.21.15.16:0/1891501644 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5b7415e370 0x7f5b74159c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:23.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:23.043+0000 7f5b7a000640 1 --2- 172.21.15.16:0/1891501644 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f5b740af070 0x7f5b741578b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:23.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:23.043+0000 7f5b7a000640 1 -- 172.21.15.16:0/1891501644 >> 172.21.15.16:0/1891501644 conn(0x7f5b7406b620 msgr2=0x7f5b7415fdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:05:23.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:23.043+0000 7f5b7a000640 1 -- 172.21.15.16:0/1891501644 shutdown_connections 2024-03-31T15:05:23.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:23.043+0000 7f5b7a000640 1 -- 172.21.15.16:0/1891501644 wait complete. 2024-03-31T15:05:23.064 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-03-31T15:05:23.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:23.211+0000 7f9e9b200640 1 Processor -- start 2024-03-31T15:05:23.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:23.211+0000 7f9e9b200640 1 -- start start 2024-03-31T15:05:23.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:23.211+0000 7f9e9b200640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f9e94147e70 0x7f9e9406b620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:23.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:23.211+0000 7f9e9b200640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f9e940d0c50 0x7f9e94072fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:23.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:23.211+0000 7f9e9b200640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9e941621b0 con 0x7f9e94073510 2024-03-31T15:05:23.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:23.215+0000 7f9e9b200640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f9e940739a0 con 0x7f9e940d0c50 2024-03-31T15:05:23.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:23.215+0000 7f9e9b200640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f9e94162490 con 0x7f9e94147e70 2024-03-31T15:05:23.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:23.215+0000 7f9e8be00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f9e940d0c50 0x7f9e94072fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:23.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:23.215+0000 7f9e8be00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f9e940d0c50 0x7f9e94072fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49432/0 (socket says 172.21.15.16:49432) 2024-03-31T15:05:23.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:23.215+0000 7f9e8be00640 1 -- 172.21.15.16:0/2191541953 learned_addr learned my addr 172.21.15.16:0/2191541953 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:05:23.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:23.215+0000 7f9e8be00640 1 -- 172.21.15.16:0/2191541953 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f9e94147e70 msgr2=0x7f9e9406b620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:05:23.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:23.215+0000 7f9e8be00640 1 --2- 172.21.15.16:0/2191541953 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f9e94147e70 0x7f9e9406b620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:23.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:23.215+0000 7f9e8be00640 1 -- 172.21.15.16:0/2191541953 >> v1:172.21.15.89:6789/0 conn(0x7f9e94073510 legacy=0x7f9e940cdab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:05:23.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:23.215+0000 7f9e8be00640 1 -- 172.21.15.16:0/2191541953 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e94162770 con 0x7f9e940d0c50 2024-03-31T15:05:23.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:23.215+0000 7f9e8be00640 1 --2- 172.21.15.16:0/2191541953 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9e940d0c50 0x7f9e94072fd0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f9e7c009870 tx=0x7f9e7c01c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e988fdf69bb6354 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:05:23.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:23.215+0000 7f9e93400640 1 -- 172.21.15.16:0/2191541953 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e7c0b0070 con 0x7f9e940d0c50 2024-03-31T15:05:23.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:23.215+0000 7f9e93400640 1 -- 172.21.15.16:0/2191541953 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9e7c0ab600 con 0x7f9e940d0c50 2024-03-31T15:05:23.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:23.215+0000 7f9e93400640 1 -- 172.21.15.16:0/2191541953 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e7c0ab9d0 con 0x7f9e940d0c50 2024-03-31T15:05:23.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:23.219+0000 7f9e9b200640 1 -- 172.21.15.16:0/2191541953 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9e940d0c50 msgr2=0x7f9e94072fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:23.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:23.219+0000 7f9e9b200640 1 --2- 172.21.15.16:0/2191541953 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9e940d0c50 0x7f9e94072fd0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f9e7c009870 tx=0x7f9e7c01c520 comp rx=0 tx=0).stop 2024-03-31T15:05:23.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:23.219+0000 7f9e9b200640 1 -- 172.21.15.16:0/2191541953 shutdown_connections 2024-03-31T15:05:23.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:23.219+0000 7f9e9b200640 1 --2- 172.21.15.16:0/2191541953 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9e940d0c50 0x7f9e94072fd0 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:23.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:23.219+0000 7f9e9b200640 1 --2- 172.21.15.16:0/2191541953 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f9e94147e70 0x7f9e9406b620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:23.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:23.219+0000 7f9e9b200640 1 -- 172.21.15.16:0/2191541953 >> 172.21.15.16:0/2191541953 conn(0x7f9e940d1060 msgr2=0x7f9e940d1470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:05:23.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:23.219+0000 7f9e9b200640 1 -- 172.21.15.16:0/2191541953 shutdown_connections 2024-03-31T15:05:23.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:23.219+0000 7f9e9b200640 1 -- 172.21.15.16:0/2191541953 wait complete. 2024-03-31T15:05:23.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:23.219+0000 7f9e9b200640 1 Processor -- start 2024-03-31T15:05:23.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:23.219+0000 7f9e9b200640 1 -- start start 2024-03-31T15:05:23.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:23.219+0000 7f9e9b200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9e94073510 0x7f9e9415d970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:23.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:23.219+0000 7f9e9b200640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f9e940d0c50 0x7f9e9413ff20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:23.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:23.219+0000 7f9e9b200640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9e94147e70 0x7f9e94159d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:23.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:23.219+0000 7f9e9b200640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f9e9406bce0 con 0x7f9e94073510 2024-03-31T15:05:23.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:23.219+0000 7f9e9b200640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f9e9406bb60 con 0x7f9e94147e70 2024-03-31T15:05:23.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:23.219+0000 7f9e9b200640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f9e9416b4b0 con 0x7f9e940d0c50 2024-03-31T15:05:23.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:23.219+0000 7f9e93e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9e94073510 0x7f9e9415d970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:23.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:23.219+0000 7f9e93e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9e94073510 0x7f9e9415d970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49442/0 (socket says 172.21.15.16:49442) 2024-03-31T15:05:23.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:23.219+0000 7f9e93e00640 1 -- 172.21.15.16:0/501826984 learned_addr learned my addr 172.21.15.16:0/501826984 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:05:23.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:23.219+0000 7f9e93e00640 1 -- 172.21.15.16:0/501826984 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f9e940d0c50 msgr2=0x7f9e9413ff20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:23.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:23.219+0000 7f9e93e00640 1 --2- 172.21.15.16:0/501826984 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f9e940d0c50 0x7f9e9413ff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:23.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:23.219+0000 7f9e93e00640 1 -- 172.21.15.16:0/501826984 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9e94147e70 msgr2=0x7f9e94159d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:23.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:23.219+0000 7f9e93e00640 1 --2- 172.21.15.16:0/501826984 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9e94147e70 0x7f9e94159d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:23.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:23.219+0000 7f9e93e00640 1 -- 172.21.15.16:0/501826984 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e94140c80 con 0x7f9e94073510 2024-03-31T15:05:23.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:23.219+0000 7f9e93e00640 1 --2- 172.21.15.16:0/501826984 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9e94073510 0x7f9e9415d970 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f9e8400b550 tx=0x7f9e8400ba10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:05:23.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:23.223+0000 7f9e90c00640 1 -- 172.21.15.16:0/501826984 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e84014070 con 0x7f9e94073510 2024-03-31T15:05:23.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:23.223+0000 7f9e90c00640 1 -- 172.21.15.16:0/501826984 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9e84010890 con 0x7f9e94073510 2024-03-31T15:05:23.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:23.223+0000 7f9e90c00640 1 -- 172.21.15.16:0/501826984 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e84010c80 con 0x7f9e94073510 2024-03-31T15:05:23.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:23.223+0000 7f9e9b200640 1 -- 172.21.15.16:0/501826984 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9e941409a0 con 0x7f9e94073510 2024-03-31T15:05:23.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:23.223+0000 7f9e9b200640 1 -- 172.21.15.16:0/501826984 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9e941412b0 con 0x7f9e94073510 2024-03-31T15:05:23.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:23.223+0000 7f9e9b200640 1 -- 172.21.15.16:0/501826984 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9e94069950 con 0x7f9e94073510 2024-03-31T15:05:23.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:23.223+0000 7f9e90c00640 1 -- 172.21.15.16:0/501826984 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f9e84017070 con 0x7f9e94073510 2024-03-31T15:05:23.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:23.223+0000 7f9e90c00640 1 --2- 172.21.15.16:0/501826984 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f9e70068380 0x7f9e7006a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:23.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:23.223+0000 7f9e90c00640 1 -- 172.21.15.16:0/501826984 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 4857+0+0 (secure 0 0 0) 0x7f9e84086420 con 0x7f9e94073510 2024-03-31T15:05:23.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:23.223+0000 7f9e8be00640 1 --2- 172.21.15.16:0/501826984 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f9e70068380 0x7f9e7006a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:23.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:23.227+0000 7f9e8be00640 1 --2- 172.21.15.16:0/501826984 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f9e70068380 0x7f9e7006a840 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f9e7c0ad000 tx=0x7f9e7c008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:05:23.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:23.227+0000 7f9e90c00640 1 -- 172.21.15.16:0/501826984 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f9e84056da0 con 0x7f9e94073510 2024-03-31T15:05:23.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:23.451+0000 7f9e9b200640 1 -- 172.21.15.16:0/501826984 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) -- 0x7f9e940d1bf0 con 0x7f9e94073510 2024-03-31T15:05:23.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:23.451+0000 7f9e90c00640 1 -- 172.21.15.16:0/501826984 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v13) ==== 80+0+5304 (secure 0 0 0) 0x7f9e84010e20 con 0x7f9e94073510 2024-03-31T15:05:23.448 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:05:23.448 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-03-31T15:05:23.449 INFO:teuthology.orchestra.run.smithi016.stdout: "auth_dump": [ 2024-03-31T15:05:23.449 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-31T15:05:23.449 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "mds.a", 2024-03-31T15:05:23.449 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAOewlmhp1ANRAAYS9zo+p4tLU1YK0ySOZE/Q==", 2024-03-31T15:05:23.449 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2024-03-31T15:05:23.449 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2024-03-31T15:05:23.449 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow *", 2024-03-31T15:05:23.449 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow *", 2024-03-31T15:05:23.449 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2024-03-31T15:05:23.449 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-31T15:05:23.449 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-31T15:05:23.449 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-31T15:05:23.449 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "mds.b", 2024-03-31T15:05:23.449 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAPewlmXxupBhAAtpkt93aJOFqhc7GXLIlkBg==", 2024-03-31T15:05:23.449 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2024-03-31T15:05:23.450 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2024-03-31T15:05:23.450 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow *", 2024-03-31T15:05:23.450 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow *", 2024-03-31T15:05:23.450 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2024-03-31T15:05:23.450 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-31T15:05:23.450 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-31T15:05:23.450 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-31T15:05:23.450 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "mds.c", 2024-03-31T15:05:23.450 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAPewlmjaieARAA32teg02vE/Q8RtqLtnArMw==", 2024-03-31T15:05:23.450 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2024-03-31T15:05:23.450 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2024-03-31T15:05:23.450 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow *", 2024-03-31T15:05:23.450 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow *", 2024-03-31T15:05:23.451 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2024-03-31T15:05:23.451 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-31T15:05:23.451 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-31T15:05:23.451 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-31T15:05:23.451 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "mds.d", 2024-03-31T15:05:23.451 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAPewlmo+4PDhAAmJD0siZB3LjWkvdMq1d8aw==", 2024-03-31T15:05:23.451 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2024-03-31T15:05:23.451 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2024-03-31T15:05:23.451 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow *", 2024-03-31T15:05:23.451 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow *", 2024-03-31T15:05:23.451 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2024-03-31T15:05:23.451 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-31T15:05:23.451 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-31T15:05:23.451 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-31T15:05:23.451 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.0", 2024-03-31T15:05:23.451 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAQewlmVP6kLhAAwFr4jwQgz1N3oIUZuJvmzw==", 2024-03-31T15:05:23.452 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2024-03-31T15:05:23.452 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2024-03-31T15:05:23.452 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2024-03-31T15:05:23.452 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2024-03-31T15:05:23.452 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-31T15:05:23.452 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-31T15:05:23.452 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-31T15:05:23.452 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.1", 2024-03-31T15:05:23.452 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQATewlmBh0gNhAAR6LiuFypwA7d3SwKlWuI/Q==", 2024-03-31T15:05:23.452 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2024-03-31T15:05:23.452 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2024-03-31T15:05:23.452 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2024-03-31T15:05:23.452 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2024-03-31T15:05:23.452 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-31T15:05:23.453 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-31T15:05:23.453 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-31T15:05:23.453 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.2", 2024-03-31T15:05:23.453 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAXewlmtQwhARAACbzc7Woo9BoVFPfK3Pns+g==", 2024-03-31T15:05:23.453 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2024-03-31T15:05:23.453 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2024-03-31T15:05:23.453 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2024-03-31T15:05:23.453 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2024-03-31T15:05:23.453 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-31T15:05:23.453 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-31T15:05:23.453 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-31T15:05:23.453 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.3", 2024-03-31T15:05:23.453 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAaewlmmjWABhAA/hwvARNb51nd5vsnYVD/sg==", 2024-03-31T15:05:23.453 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2024-03-31T15:05:23.453 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2024-03-31T15:05:23.453 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2024-03-31T15:05:23.454 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2024-03-31T15:05:23.454 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-31T15:05:23.454 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-31T15:05:23.454 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-31T15:05:23.454 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.4", 2024-03-31T15:05:23.454 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAgewlmby07CxAAChxE/qPuPboIlg3fDxSOeA==", 2024-03-31T15:05:23.454 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2024-03-31T15:05:23.454 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2024-03-31T15:05:23.454 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2024-03-31T15:05:23.454 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2024-03-31T15:05:23.454 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-31T15:05:23.454 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-31T15:05:23.454 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-31T15:05:23.454 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.5", 2024-03-31T15:05:23.455 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAjewlmlD8yExAA9pgP0heBywrgqc39iU8rpQ==", 2024-03-31T15:05:23.455 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2024-03-31T15:05:23.455 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2024-03-31T15:05:23.455 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2024-03-31T15:05:23.455 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2024-03-31T15:05:23.455 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-31T15:05:23.455 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-31T15:05:23.455 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-31T15:05:23.455 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.6", 2024-03-31T15:05:23.455 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAmewlmzAsTGBAALMjVcQ08sIiTDFaBcq8L6A==", 2024-03-31T15:05:23.455 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2024-03-31T15:05:23.455 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2024-03-31T15:05:23.455 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2024-03-31T15:05:23.455 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2024-03-31T15:05:23.455 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-31T15:05:23.456 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-31T15:05:23.456 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-31T15:05:23.456 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.7", 2024-03-31T15:05:23.456 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQApewlmb+CEHRAAXE+iBumDsb2c4zH3C8f6YA==", 2024-03-31T15:05:23.456 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2024-03-31T15:05:23.456 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2024-03-31T15:05:23.456 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2024-03-31T15:05:23.456 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2024-03-31T15:05:23.456 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-31T15:05:23.456 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-31T15:05:23.456 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-31T15:05:23.456 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "client.0", 2024-03-31T15:05:23.456 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAPewlmlMIDEhAAF0Ll1HcrWHmH6xDKq2Qq9A==", 2024-03-31T15:05:23.456 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2024-03-31T15:05:23.456 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2024-03-31T15:05:23.456 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow r", 2024-03-31T15:05:23.457 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-03-31T15:05:23.457 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-31T15:05:23.457 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-31T15:05:23.457 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-31T15:05:23.457 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "client.1", 2024-03-31T15:05:23.457 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAPewlmdsvqGxAAaVdwWwri0IE/2sg7psilWw==", 2024-03-31T15:05:23.457 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2024-03-31T15:05:23.457 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2024-03-31T15:05:23.457 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow r", 2024-03-31T15:05:23.457 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow rw", 2024-03-31T15:05:23.457 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow rwx" 2024-03-31T15:05:23.457 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-31T15:05:23.457 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-31T15:05:23.457 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-31T15:05:23.457 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "client.2", 2024-03-31T15:05:23.457 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAPewlmQapPGBAAbwXSlCi+zyQ9118WyAJmrQ==", 2024-03-31T15:05:23.458 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2024-03-31T15:05:23.458 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2024-03-31T15:05:23.458 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow r", 2024-03-31T15:05:23.458 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow rw", 2024-03-31T15:05:23.458 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow rwx" 2024-03-31T15:05:23.458 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-31T15:05:23.458 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-31T15:05:23.458 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-31T15:05:23.458 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "client.3", 2024-03-31T15:05:23.458 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAPewlmCw7qIRAAEeIjF85JbyfJ6d3FtMOVhQ==", 2024-03-31T15:05:23.458 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2024-03-31T15:05:23.458 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2024-03-31T15:05:23.458 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow r", 2024-03-31T15:05:23.458 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow rw", 2024-03-31T15:05:23.458 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow rwx" 2024-03-31T15:05:23.458 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-31T15:05:23.459 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-31T15:05:23.459 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-31T15:05:23.459 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "client.admin", 2024-03-31T15:05:23.459 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAOewlmXyPZGRAAtZOheiYhzGxdeJGqqyGxTw==", 2024-03-31T15:05:23.459 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2024-03-31T15:05:23.459 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow *", 2024-03-31T15:05:23.459 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow *", 2024-03-31T15:05:23.459 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow *", 2024-03-31T15:05:23.459 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2024-03-31T15:05:23.459 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-31T15:05:23.459 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-31T15:05:23.459 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-31T15:05:23.459 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "mgr.x", 2024-03-31T15:05:23.459 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAOewlm7UkKMRAA8hqgeV86mTCkRcxdXNdWmw==", 2024-03-31T15:05:23.459 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2024-03-31T15:05:23.460 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow *", 2024-03-31T15:05:23.460 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile mgr", 2024-03-31T15:05:23.460 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2024-03-31T15:05:23.460 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-31T15:05:23.460 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-31T15:05:23.460 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-31T15:05:23.460 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "mgr.y", 2024-03-31T15:05:23.460 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAOewlmq0cpLRAAKTgh04tRaHrpMhxdwgvXYA==", 2024-03-31T15:05:23.460 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2024-03-31T15:05:23.460 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow *", 2024-03-31T15:05:23.460 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile mgr", 2024-03-31T15:05:23.460 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2024-03-31T15:05:23.460 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-31T15:05:23.460 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-31T15:05:23.460 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-03-31T15:05:23.461 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-03-31T15:05:23.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:23.455+0000 7f9e9b200640 1 -- 172.21.15.16:0/501826984 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f9e70068380 msgr2=0x7f9e7006a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:23.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:23.455+0000 7f9e9b200640 1 --2- 172.21.15.16:0/501826984 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f9e70068380 0x7f9e7006a840 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f9e7c0ad000 tx=0x7f9e7c008040 comp rx=0 tx=0).stop 2024-03-31T15:05:23.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:23.455+0000 7f9e9b200640 1 -- 172.21.15.16:0/501826984 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9e94073510 msgr2=0x7f9e9415d970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:23.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:23.455+0000 7f9e9b200640 1 --2- 172.21.15.16:0/501826984 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9e94073510 0x7f9e9415d970 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f9e8400b550 tx=0x7f9e8400ba10 comp rx=0 tx=0).stop 2024-03-31T15:05:23.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:23.455+0000 7f9e9b200640 1 -- 172.21.15.16:0/501826984 shutdown_connections 2024-03-31T15:05:23.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:23.455+0000 7f9e9b200640 1 --2- 172.21.15.16:0/501826984 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f9e70068380 0x7f9e7006a840 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:23.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:23.455+0000 7f9e9b200640 1 --2- 172.21.15.16:0/501826984 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9e94147e70 0x7f9e94159d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:23.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:23.455+0000 7f9e9b200640 1 --2- 172.21.15.16:0/501826984 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f9e940d0c50 0x7f9e9413ff20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:23.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:23.455+0000 7f9e9b200640 1 --2- 172.21.15.16:0/501826984 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9e94073510 0x7f9e9415d970 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:23.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:23.455+0000 7f9e9b200640 1 -- 172.21.15.16:0/501826984 >> 172.21.15.16:0/501826984 conn(0x7f9e940d1060 msgr2=0x7f9e9415fca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:05:23.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:23.455+0000 7f9e9b200640 1 -- 172.21.15.16:0/501826984 shutdown_connections 2024-03-31T15:05:23.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:23.455+0000 7f9e9b200640 1 -- 172.21.15.16:0/501826984 wait complete. 2024-03-31T15:05:23.469 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-31T15:05:23.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:23.615+0000 7ff932e00640 1 Processor -- start 2024-03-31T15:05:23.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:23.615+0000 7ff932e00640 1 -- start start 2024-03-31T15:05:23.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:23.615+0000 7ff932e00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff92c0728b0 0x7ff92c0b1cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:23.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:23.615+0000 7ff932e00640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff92c160e70 con 0x7ff92c0b2200 2024-03-31T15:05:23.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:23.615+0000 7ff932e00640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff92c1602f0 con 0x7ff92c145d10 2024-03-31T15:05:23.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:23.615+0000 7ff932e00640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7ff92c1605d0 con 0x7ff92c0728b0 2024-03-31T15:05:23.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:23.615+0000 7ff923e00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff92c0728b0 0x7ff92c0b1cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:23.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:23.615+0000 7ff923e00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff92c0728b0 0x7ff92c0b1cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49448/0 (socket says 172.21.15.16:49448) 2024-03-31T15:05:23.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:23.615+0000 7ff923e00640 1 -- 172.21.15.16:0/1370253648 learned_addr learned my addr 172.21.15.16:0/1370253648 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:05:23.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:23.615+0000 7ff930e00640 1 --1- 172.21.15.16:0/1370253648 >> v1:172.21.15.16:6789/0 conn(0x7ff92c0b2200 0x7ff92c15cd50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:55544/0 (socket says 172.21.15.16:55544) 2024-03-31T15:05:23.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:23.619+0000 7ff923e00640 1 -- 172.21.15.16:0/1370253648 >> v1:172.21.15.89:6789/0 conn(0x7ff92c145d10 legacy=0x7ff92c0cf790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:23.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:23.619+0000 7ff923e00640 1 -- 172.21.15.16:0/1370253648 >> v1:172.21.15.16:6789/0 conn(0x7ff92c0b2200 legacy=0x7ff92c15cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:23.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:23.619+0000 7ff923e00640 1 -- 172.21.15.16:0/1370253648 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff92c1608b0 con 0x7ff92c0728b0 2024-03-31T15:05:23.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:23.619+0000 7ff923e00640 1 --2- 172.21.15.16:0/1370253648 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff92c0728b0 0x7ff92c0b1cc0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7ff914009870 tx=0x7ff91401c560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=50942293188490be server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:05:23.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:23.619+0000 7ff92b400640 1 -- 172.21.15.16:0/1370253648 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9140b2070 con 0x7ff92c0728b0 2024-03-31T15:05:23.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:23.619+0000 7ff92b400640 1 -- 172.21.15.16:0/1370253648 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff9140a9070 con 0x7ff92c0728b0 2024-03-31T15:05:23.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:23.619+0000 7ff92b400640 1 -- 172.21.15.16:0/1370253648 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9140ad880 con 0x7ff92c0728b0 2024-03-31T15:05:23.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:23.619+0000 7ff932e00640 1 -- 172.21.15.16:0/1370253648 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff92c0728b0 msgr2=0x7ff92c0b1cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:23.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:23.619+0000 7ff932e00640 1 --2- 172.21.15.16:0/1370253648 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff92c0728b0 0x7ff92c0b1cc0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7ff914009870 tx=0x7ff91401c560 comp rx=0 tx=0).stop 2024-03-31T15:05:23.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:23.623+0000 7ff932e00640 1 -- 172.21.15.16:0/1370253648 shutdown_connections 2024-03-31T15:05:23.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:23.623+0000 7ff932e00640 1 --2- 172.21.15.16:0/1370253648 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff92c0728b0 0x7ff92c0b1cc0 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:23.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:23.623+0000 7ff932e00640 1 -- 172.21.15.16:0/1370253648 >> 172.21.15.16:0/1370253648 conn(0x7ff92c06b620 msgr2=0x7ff92c06ba30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:05:23.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:23.623+0000 7ff932e00640 1 -- 172.21.15.16:0/1370253648 shutdown_connections 2024-03-31T15:05:23.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:23.623+0000 7ff932e00640 1 -- 172.21.15.16:0/1370253648 wait complete. 2024-03-31T15:05:23.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:23.623+0000 7ff932e00640 1 Processor -- start 2024-03-31T15:05:23.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:23.623+0000 7ff932e00640 1 -- start start 2024-03-31T15:05:23.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:23.623+0000 7ff932e00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff92c0728b0 0x7ff92c137f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:23.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:23.623+0000 7ff932e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff92c0b2200 0x7ff92c158800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:23.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:23.623+0000 7ff932e00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff92c145d10 0x7ff92c15bd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:23.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:23.623+0000 7ff932e00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7ff92c0cf350 con 0x7ff92c0b2200 2024-03-31T15:05:23.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:23.623+0000 7ff932e00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7ff92c0b2610 con 0x7ff92c145d10 2024-03-31T15:05:23.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:23.623+0000 7ff932e00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7ff92c0cf4d0 con 0x7ff92c0728b0 2024-03-31T15:05:23.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:23.623+0000 7ff923e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff92c0b2200 0x7ff92c158800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:23.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:23.623+0000 7ff923e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff92c0b2200 0x7ff92c158800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49456/0 (socket says 172.21.15.16:49456) 2024-03-31T15:05:23.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:23.623+0000 7ff923e00640 1 -- 172.21.15.16:0/863940254 learned_addr learned my addr 172.21.15.16:0/863940254 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:05:23.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:23.623+0000 7ff92be00640 1 --2- 172.21.15.16:0/863940254 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff92c0728b0 0x7ff92c137f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:23.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:23.623+0000 7ff930e00640 1 --2- 172.21.15.16:0/863940254 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff92c145d10 0x7ff92c15bd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:23.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:23.623+0000 7ff923e00640 1 -- 172.21.15.16:0/863940254 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff92c0728b0 msgr2=0x7ff92c137f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:23.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:23.623+0000 7ff923e00640 1 --2- 172.21.15.16:0/863940254 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff92c0728b0 0x7ff92c137f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:23.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:23.623+0000 7ff923e00640 1 -- 172.21.15.16:0/863940254 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff92c145d10 msgr2=0x7ff92c15bd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:23.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:23.623+0000 7ff923e00640 1 --2- 172.21.15.16:0/863940254 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff92c145d10 0x7ff92c15bd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:23.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:23.623+0000 7ff923e00640 1 -- 172.21.15.16:0/863940254 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff92c15b590 con 0x7ff92c0b2200 2024-03-31T15:05:23.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:23.623+0000 7ff923e00640 1 --2- 172.21.15.16:0/863940254 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff92c0b2200 0x7ff92c158800 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7ff9140b0040 tx=0x7ff914019300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:05:23.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:23.623+0000 7ff928c00640 1 -- 172.21.15.16:0/863940254 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9140bc070 con 0x7ff92c0b2200 2024-03-31T15:05:23.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:23.627+0000 7ff928c00640 1 -- 172.21.15.16:0/863940254 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff9140b6640 con 0x7ff92c0b2200 2024-03-31T15:05:23.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:23.627+0000 7ff932e00640 1 -- 172.21.15.16:0/863940254 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff92c157dc0 con 0x7ff92c0b2200 2024-03-31T15:05:23.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:23.627+0000 7ff928c00640 1 -- 172.21.15.16:0/863940254 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9140b2040 con 0x7ff92c0b2200 2024-03-31T15:05:23.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:23.627+0000 7ff932e00640 1 -- 172.21.15.16:0/863940254 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff92c158060 con 0x7ff92c0b2200 2024-03-31T15:05:23.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:23.627+0000 7ff932e00640 1 -- 172.21.15.16:0/863940254 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff92c069950 con 0x7ff92c0b2200 2024-03-31T15:05:23.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:23.627+0000 7ff928c00640 1 -- 172.21.15.16:0/863940254 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7ff9140bf070 con 0x7ff92c0b2200 2024-03-31T15:05:23.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:23.627+0000 7ff928c00640 1 --2- 172.21.15.16:0/863940254 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7ff908068380 0x7ff90806a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:23.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:23.627+0000 7ff928c00640 1 -- 172.21.15.16:0/863940254 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 4857+0+0 (secure 0 0 0) 0x7ff91411fb30 con 0x7ff92c0b2200 2024-03-31T15:05:23.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:23.627+0000 7ff92be00640 1 --2- 172.21.15.16:0/863940254 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7ff908068380 0x7ff90806a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:23.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:23.627+0000 7ff92be00640 1 --2- 172.21.15.16:0/863940254 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7ff908068380 0x7ff90806a840 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7ff92c0cd220 tx=0x7ff91c009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:05:23.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:23.631+0000 7ff928c00640 1 -- 172.21.15.16:0/863940254 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7ff914016150 con 0x7ff92c0b2200 2024-03-31T15:05:23.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:23.799+0000 7ff932e00640 1 -- 172.21.15.16:0/863940254 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ff92c06bcc0 con 0x7ff92c0b2200 2024-03-31T15:05:23.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:23.799+0000 7ff928c00640 1 -- 172.21.15.16:0/863940254 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v41) ==== 74+0+12819 (secure 0 0 0) 0x7ff9140efd50 con 0x7ff92c0b2200 2024-03-31T15:05:23.795 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:05:23.795 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":41,"fsid":"286eb5da-fb47-4550-b290-d9519c13b9a5","created":"2024-03-31T15:03:11.438952+0000","modified":"2024-03-31T15:05:23.056232+0000","last_up_change":"2024-03-31T15:03:22.885535+0000","last_in_change":"2024-03-31T15:03:13.941667+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-31T15:03:22.550787+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"92a66450-ef19-40d8-ac3d-d760b23b0bda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6801","nonce":932712244}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6803","nonce":932712244}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6807","nonce":932712244}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6805","nonce":932712244}]},"public_addr":"172.21.15.16:6801/932712244","cluster_addr":"172.21.15.16:6803/932712244","heartbeat_back_addr":"172.21.15.16:6807/932712244","heartbeat_front_addr":"172.21.15.16:6805/932712244","state":["exists","up"]},{"osd":1,"uuid":"da9d1f2b-1968-4959-b843-faa231988e92","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6810","nonce":1398737533}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6811","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6813","nonce":1398737533}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6821","nonce":1398737533}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6815","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6816","nonce":1398737533}]},"public_addr":"172.21.15.16:6810/1398737533","cluster_addr":"172.21.15.16:6813/1398737533","heartbeat_back_addr":"172.21.15.16:6821/1398737533","heartbeat_front_addr":"172.21.15.16:6816/1398737533","state":["exists","up"]},{"osd":2,"uuid":"01a73662-9a21-4b1f-bcbb-42b9636a4638","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6819","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6822","nonce":3018526646}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6825","nonce":3018526646}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6831","nonce":3018526646}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6827","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6829","nonce":3018526646}]},"public_addr":"172.21.15.16:6822/3018526646","cluster_addr":"172.21.15.16:6825/3018526646","heartbeat_back_addr":"172.21.15.16:6831/3018526646","heartbeat_front_addr":"172.21.15.16:6829/3018526646","state":["exists","up"]},{"osd":3,"uuid":"c7771fd3-e18e-4500-ac38-23cf10ebcd75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6809","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6812","nonce":3360352974}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6817","nonce":3360352974}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6828","nonce":3360352974}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6823","nonce":3360352974}]},"public_addr":"172.21.15.16:6812/3360352974","cluster_addr":"172.21.15.16:6817/3360352974","heartbeat_back_addr":"172.21.15.16:6828/3360352974","heartbeat_front_addr":"172.21.15.16:6823/3360352974","state":["exists","up"]},{"osd":4,"uuid":"88260730-1482-4fbd-a40f-cb1124e9df4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6801","nonce":1856844856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6803","nonce":1856844856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6807","nonce":1856844856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6805","nonce":1856844856}]},"public_addr":"172.21.15.89:6801/1856844856","cluster_addr":"172.21.15.89:6803/1856844856","heartbeat_back_addr":"172.21.15.89:6807/1856844856","heartbeat_front_addr":"172.21.15.89:6805/1856844856","state":["exists","up"]},{"osd":5,"uuid":"4cb2c00c-fae0-4dd8-9d27-ec4f0093ccf7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6814","nonce":50395527}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6817","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6820","nonce":50395527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6831","nonce":50395527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6823","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6826","nonce":50395527}]},"public_addr":"172.21.15.89:6814/50395527","cluster_addr":"172.21.15.89:6820/50395527","heartbeat_back_addr":"172.21.15.89:6831/50395527","heartbeat_front_addr":"172.21.15.89:6826/50395527","state":["exists","up"]},{"osd":6,"uuid":"81ce0c19-2554-4cae-acbe-66a05e9e9b9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6809","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6812","nonce":3470471752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6815","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6818","nonce":3470471752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6827","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6829","nonce":3470471752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6821","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6824","nonce":3470471752}]},"public_addr":"172.21.15.89:6812/3470471752","cluster_addr":"172.21.15.89:6818/3470471752","heartbeat_back_addr":"172.21.15.89:6829/3470471752","heartbeat_front_addr":"172.21.15.89:6824/3470471752","state":["exists","up"]},{"osd":7,"uuid":"85e4313b-d37d-442e-8042-153bf20a847d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6811","nonce":1796778630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6813","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6816","nonce":1796778630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6825","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6828","nonce":1796778630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6819","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6822","nonce":1796778630}]},"public_addr":"172.21.15.89:6811/1796778630","cluster_addr":"172.21.15.89:6816/1796778630","heartbeat_back_addr":"172.21.15.89:6828/1796778630","heartbeat_front_addr":"172.21.15.89:6822/1796778630","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.439088+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.529970+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.680512+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:21.044161+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.582111+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.536558+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.409522+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.590229+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-31T15:05:23.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:23.799+0000 7ff932e00640 1 -- 172.21.15.16:0/863940254 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7ff908068380 msgr2=0x7ff90806a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:23.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:23.799+0000 7ff932e00640 1 --2- 172.21.15.16:0/863940254 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7ff908068380 0x7ff90806a840 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7ff92c0cd220 tx=0x7ff91c009040 comp rx=0 tx=0).stop 2024-03-31T15:05:23.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:23.799+0000 7ff932e00640 1 -- 172.21.15.16:0/863940254 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff92c0b2200 msgr2=0x7ff92c158800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:23.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:23.799+0000 7ff932e00640 1 --2- 172.21.15.16:0/863940254 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff92c0b2200 0x7ff92c158800 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7ff9140b0040 tx=0x7ff914019300 comp rx=0 tx=0).stop 2024-03-31T15:05:23.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:23.803+0000 7ff932e00640 1 -- 172.21.15.16:0/863940254 shutdown_connections 2024-03-31T15:05:23.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:23.803+0000 7ff932e00640 1 --2- 172.21.15.16:0/863940254 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7ff908068380 0x7ff90806a840 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:23.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:23.803+0000 7ff932e00640 1 --2- 172.21.15.16:0/863940254 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff92c145d10 0x7ff92c15bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:23.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:23.803+0000 7ff932e00640 1 --2- 172.21.15.16:0/863940254 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff92c0b2200 0x7ff92c158800 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:23.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:23.803+0000 7ff932e00640 1 --2- 172.21.15.16:0/863940254 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff92c0728b0 0x7ff92c137f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:23.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:23.803+0000 7ff932e00640 1 -- 172.21.15.16:0/863940254 >> 172.21.15.16:0/863940254 conn(0x7ff92c06b620 msgr2=0x7ff92c1678d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:05:23.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:23.803+0000 7ff932e00640 1 -- 172.21.15.16:0/863940254 shutdown_connections 2024-03-31T15:05:23.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:23.803+0000 7ff932e00640 1 -- 172.21.15.16:0/863940254 wait complete. 2024-03-31T15:05:23.816 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-31T15:03:22.550787+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-31T15:05:23.816 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-31T15:05:23.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:23.963+0000 7f102b800640 1 Processor -- start 2024-03-31T15:05:23.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:23.963+0000 7f102b800640 1 -- start start 2024-03-31T15:05:23.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:23.963+0000 7f102b800640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f1024141590 0x7f1024144340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:23.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:23.963+0000 7f102b800640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1024144980 con 0x7f1024070030 2024-03-31T15:05:23.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:23.963+0000 7f102b800640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1024161fe0 con 0x7f10241411b0 2024-03-31T15:05:23.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:23.963+0000 7f102b800640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f102415f910 con 0x7f1024141590 2024-03-31T15:05:23.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:23.967+0000 7f1023e00640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f1024070030 0x7f1024143c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:55556/0 (socket says 172.21.15.16:55556) 2024-03-31T15:05:23.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:23.967+0000 7f1023e00640 1 -- 172.21.15.16:0/1764332370 learned_addr learned my addr 172.21.15.16:0/1764332370 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:05:23.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:23.967+0000 7f1029800640 1 --2- 172.21.15.16:0/1764332370 >> v2:172.21.15.89:3300/0 conn(0x7f1024141590 0x7f1024144340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:23.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:23.967+0000 7f1023400640 1 -- 172.21.15.16:0/1764332370 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 534056222 0 0) 0x7f1024144980 con 0x7f1024070030 2024-03-31T15:05:23.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:23.967+0000 7f1023400640 1 -- 172.21.15.16:0/1764332370 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f100c003530 con 0x7f1024070030 2024-03-31T15:05:23.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:23.967+0000 7f1023400640 1 -- 172.21.15.16:0/1764332370 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3071273817 0 0) 0x7f100c003530 con 0x7f1024070030 2024-03-31T15:05:23.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:23.967+0000 7f1023400640 1 -- 172.21.15.16:0/1764332370 >> v2:172.21.15.89:3300/0 conn(0x7f1024141590 msgr2=0x7f1024144340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:05:23.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:23.967+0000 7f1023400640 1 --2- 172.21.15.16:0/1764332370 >> v2:172.21.15.89:3300/0 conn(0x7f1024141590 0x7f1024144340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:23.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:23.967+0000 7f1023400640 1 -- 172.21.15.16:0/1764332370 >> v1:172.21.15.89:6789/0 conn(0x7f10241411b0 legacy=0x7f102406f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:23.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:23.967+0000 7f1023400640 1 -- 172.21.15.16:0/1764332370 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10241622c0 con 0x7f1024070030 2024-03-31T15:05:23.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:23.967+0000 7f1023400640 1 -- 172.21.15.16:0/1764332370 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 214807623 0 0) 0x7f1010001680 con 0x7f1024070030 2024-03-31T15:05:23.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:23.967+0000 7f1029800640 1 --2- 172.21.15.16:0/1764332370 >> v2:172.21.15.89:3300/0 conn(0x7f1024141590 0x7f1024144340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:05:23.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:23.967+0000 7f1023400640 1 -- 172.21.15.16:0/1764332370 >> v1:172.21.15.16:6789/0 conn(0x7f1024070030 legacy=0x7f1024143c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:23.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:23.967+0000 7f1023400640 1 --2- 172.21.15.16:0/1764332370 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f100c003a10 0x7f100c003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:23.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:23.967+0000 7f1023400640 1 --2- 172.21.15.16:0/1764332370 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f100c004810 0x7f100c004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:23.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:23.967+0000 7f1023400640 1 --2- 172.21.15.16:0/1764332370 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f100c0051d0 0x7f100c00b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:23.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:23.967+0000 7f1023400640 1 -- 172.21.15.16:0/1764332370 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f100c00c0a0 con 0x7f100c003a10 2024-03-31T15:05:23.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:23.967+0000 7f1023400640 1 -- 172.21.15.16:0/1764332370 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f100c00c250 con 0x7f100c0051d0 2024-03-31T15:05:23.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:23.967+0000 7f1023400640 1 -- 172.21.15.16:0/1764332370 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f100c00c450 con 0x7f100c004810 2024-03-31T15:05:23.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:23.967+0000 7f1028e00640 1 --2- 172.21.15.16:0/1764332370 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f100c003a10 0x7f100c003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:23.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:23.967+0000 7f1023e00640 1 --2- 172.21.15.16:0/1764332370 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f100c004810 0x7f100c004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:23.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:23.967+0000 7f1029800640 1 --2- 172.21.15.16:0/1764332370 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f100c0051d0 0x7f100c00b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:23.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:23.967+0000 7f1028e00640 1 -- 172.21.15.16:0/1764332370 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f100c004810 msgr2=0x7f100c004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:05:23.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:23.967+0000 7f1028e00640 1 --2- 172.21.15.16:0/1764332370 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f100c004810 0x7f100c004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:23.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:23.967+0000 7f1028e00640 1 -- 172.21.15.16:0/1764332370 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f100c0051d0 msgr2=0x7f100c00b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:05:23.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:23.967+0000 7f1028e00640 1 --2- 172.21.15.16:0/1764332370 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f100c0051d0 0x7f100c00b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:23.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:23.967+0000 7f1028e00640 1 -- 172.21.15.16:0/1764332370 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f100c003530 con 0x7f100c003a10 2024-03-31T15:05:23.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:23.967+0000 7f1029800640 1 --2- 172.21.15.16:0/1764332370 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f100c0051d0 0x7f100c00b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:05:23.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:23.967+0000 7f1023e00640 1 --2- 172.21.15.16:0/1764332370 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f100c004810 0x7f100c004c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:05:23.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:23.967+0000 7f1029800640 1 -- 172.21.15.16:0/1764332370 reap_dead start 2024-03-31T15:05:23.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:23.971+0000 7f1028e00640 1 --2- 172.21.15.16:0/1764332370 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f100c003a10 0x7f100c003e30 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f1018001780 tx=0x7f101801c400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=49f6239a555f6afa server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:05:23.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:23.971+0000 7f1023400640 1 -- 172.21.15.16:0/1764332370 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10180013a0 con 0x7f100c003a10 2024-03-31T15:05:23.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:23.971+0000 7f1023400640 1 -- 172.21.15.16:0/1764332370 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f10180052f0 con 0x7f100c003a10 2024-03-31T15:05:23.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:23.971+0000 7f1023400640 1 -- 172.21.15.16:0/1764332370 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1018003840 con 0x7f100c003a10 2024-03-31T15:05:23.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:23.971+0000 7f102b800640 1 -- 172.21.15.16:0/1764332370 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f100c003a10 msgr2=0x7f100c003e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:23.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:23.971+0000 7f102b800640 1 --2- 172.21.15.16:0/1764332370 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f100c003a10 0x7f100c003e30 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f1018001780 tx=0x7f101801c400 comp rx=0 tx=0).stop 2024-03-31T15:05:23.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:23.971+0000 7f102b800640 1 -- 172.21.15.16:0/1764332370 shutdown_connections 2024-03-31T15:05:23.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:23.971+0000 7f102b800640 1 --2- 172.21.15.16:0/1764332370 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f100c003a10 0x7f100c003e30 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:23.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:23.971+0000 7f102b800640 1 -- 172.21.15.16:0/1764332370 >> 172.21.15.16:0/1764332370 conn(0x7f1024067930 msgr2=0x7f10240b3ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:05:23.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:23.971+0000 7f102b800640 1 -- 172.21.15.16:0/1764332370 shutdown_connections 2024-03-31T15:05:23.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:23.971+0000 7f102b800640 1 -- 172.21.15.16:0/1764332370 wait complete. 2024-03-31T15:05:23.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:23.971+0000 7f102b800640 1 Processor -- start 2024-03-31T15:05:23.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:23.971+0000 7f102b800640 1 -- start start 2024-03-31T15:05:23.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:23.975+0000 7f102b800640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f100c003a10 0x7f102413d460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:23.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:23.975+0000 7f102b800640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f102413e840 0x7f10240b2e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:23.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:23.975+0000 7f102b800640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f1024140020 0x7f102413dbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:23.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:23.975+0000 7f102b800640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f10241445c0 con 0x7f102413e840 2024-03-31T15:05:23.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:23.975+0000 7f102b800640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f1024144440 con 0x7f100c003a10 2024-03-31T15:05:23.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:23.975+0000 7f102b800640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f102415e900 con 0x7f1024140020 2024-03-31T15:05:23.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:23.975+0000 7f1023e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f102413e840 0x7f10240b2e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:23.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:23.975+0000 7f1023e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f102413e840 0x7f10240b2e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49480/0 (socket says 172.21.15.16:49480) 2024-03-31T15:05:23.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:23.975+0000 7f1023e00640 1 -- 172.21.15.16:0/1344669675 learned_addr learned my addr 172.21.15.16:0/1344669675 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:05:23.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:23.975+0000 7f1028e00640 1 --2- 172.21.15.16:0/1344669675 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f100c003a10 0x7f102413d460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:23.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:23.975+0000 7f1023e00640 1 -- 172.21.15.16:0/1344669675 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f1024140020 msgr2=0x7f102413dbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:23.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:23.975+0000 7f1023e00640 1 --2- 172.21.15.16:0/1344669675 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f1024140020 0x7f102413dbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:23.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:23.975+0000 7f1023e00640 1 -- 172.21.15.16:0/1344669675 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f100c003a10 msgr2=0x7f102413d460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:23.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:23.975+0000 7f1023e00640 1 --2- 172.21.15.16:0/1344669675 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f100c003a10 0x7f102413d460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:23.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:23.975+0000 7f1023e00640 1 -- 172.21.15.16:0/1344669675 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10241fc550 con 0x7f102413e840 2024-03-31T15:05:23.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:23.975+0000 7f1028e00640 1 --2- 172.21.15.16:0/1344669675 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f100c003a10 0x7f102413d460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:05:23.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:23.975+0000 7f1023e00640 1 --2- 172.21.15.16:0/1344669675 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f102413e840 0x7f10240b2e00 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f101000e700 tx=0x7f101000ebc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:05:23.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:23.975+0000 7f1020c00640 1 -- 172.21.15.16:0/1344669675 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f101000c6c0 con 0x7f102413e840 2024-03-31T15:05:23.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:23.975+0000 7f1020c00640 1 -- 172.21.15.16:0/1344669675 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1010018070 con 0x7f102413e840 2024-03-31T15:05:23.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:23.975+0000 7f102b800640 1 -- 172.21.15.16:0/1344669675 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f102413f540 con 0x7f102413e840 2024-03-31T15:05:23.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:23.975+0000 7f1020c00640 1 -- 172.21.15.16:0/1344669675 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f101000ca60 con 0x7f102413e840 2024-03-31T15:05:23.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:23.975+0000 7f102b800640 1 -- 172.21.15.16:0/1344669675 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f102413fb00 con 0x7f102413e840 2024-03-31T15:05:23.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:23.975+0000 7f102b800640 1 -- 172.21.15.16:0/1344669675 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f102413f330 con 0x7f102413e840 2024-03-31T15:05:23.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:23.975+0000 7f1020c00640 1 -- 172.21.15.16:0/1344669675 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f101000cc00 con 0x7f102413e840 2024-03-31T15:05:23.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:23.979+0000 7f1020c00640 1 --2- 172.21.15.16:0/1344669675 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f1004068380 0x7f100406a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:23.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:23.979+0000 7f1020c00640 1 -- 172.21.15.16:0/1344669675 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 4857+0+0 (secure 0 0 0) 0x7f1010086450 con 0x7f102413e840 2024-03-31T15:05:23.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:23.979+0000 7f1028e00640 1 --2- 172.21.15.16:0/1344669675 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f1004068380 0x7f100406a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:23.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:23.979+0000 7f1028e00640 1 --2- 172.21.15.16:0/1344669675 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f1004068380 0x7f100406a840 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f1018001750 tx=0x7f10180011a0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:05:23.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:23.983+0000 7f1020c00640 1 -- 172.21.15.16:0/1344669675 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f1010056cc0 con 0x7f102413e840 2024-03-31T15:05:24.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:24.151+0000 7f102b800640 1 -- 172.21.15.16:0/1344669675 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f1024069950 con 0x7f102413e840 2024-03-31T15:05:24.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:24.151+0000 7f1020c00640 1 -- 172.21.15.16:0/1344669675 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v41) ==== 93+0+10 (secure 0 0 0) 0x7f10100565c0 con 0x7f102413e840 2024-03-31T15:05:24.147 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2024-03-31T15:05:24.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:24.155+0000 7f102b800640 1 -- 172.21.15.16:0/1344669675 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f1004068380 msgr2=0x7f100406a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:24.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:24.155+0000 7f102b800640 1 --2- 172.21.15.16:0/1344669675 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f1004068380 0x7f100406a840 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f1018001750 tx=0x7f10180011a0 comp rx=0 tx=0).stop 2024-03-31T15:05:24.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:24.155+0000 7f102b800640 1 -- 172.21.15.16:0/1344669675 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f102413e840 msgr2=0x7f10240b2e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:24.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:24.155+0000 7f102b800640 1 --2- 172.21.15.16:0/1344669675 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f102413e840 0x7f10240b2e00 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f101000e700 tx=0x7f101000ebc0 comp rx=0 tx=0).stop 2024-03-31T15:05:24.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:24.155+0000 7f102b800640 1 -- 172.21.15.16:0/1344669675 shutdown_connections 2024-03-31T15:05:24.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:24.155+0000 7f102b800640 1 --2- 172.21.15.16:0/1344669675 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f1004068380 0x7f100406a840 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:24.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:24.155+0000 7f102b800640 1 --2- 172.21.15.16:0/1344669675 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f1024140020 0x7f102413dbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:24.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:24.155+0000 7f102b800640 1 --2- 172.21.15.16:0/1344669675 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f102413e840 0x7f10240b2e00 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:24.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:24.155+0000 7f102b800640 1 --2- 172.21.15.16:0/1344669675 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f100c003a10 0x7f102413d460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:24.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:24.155+0000 7f102b800640 1 -- 172.21.15.16:0/1344669675 >> 172.21.15.16:0/1344669675 conn(0x7f1024067930 msgr2=0x7f10240d2ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:05:24.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:24.155+0000 7f102b800640 1 -- 172.21.15.16:0/1344669675 shutdown_connections 2024-03-31T15:05:24.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:24.155+0000 7f102b800640 1 -- 172.21.15.16:0/1344669675 wait complete. 2024-03-31T15:05:24.168 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-31T15:05:24.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:24.311+0000 7ff1d1000640 1 Processor -- start 2024-03-31T15:05:24.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:24.311+0000 7ff1d1000640 1 -- start start 2024-03-31T15:05:24.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:24.315+0000 7ff1d1000640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff1cc15fd70 0x7ff1cc0cf250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:24.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:24.315+0000 7ff1d1000640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff1cc160150 0x7ff1cc0b4630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:24.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:24.315+0000 7ff1d1000640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff1cc0cf850 con 0x7ff1cc1529c0 2024-03-31T15:05:24.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:24.315+0000 7ff1d1000640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7ff1cc0d0460 con 0x7ff1cc15fd70 2024-03-31T15:05:24.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:24.315+0000 7ff1d1000640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7ff1cc0cfb30 con 0x7ff1cc160150 2024-03-31T15:05:24.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:24.315+0000 7ff1c9e00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff1cc15fd70 0x7ff1cc0cf250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:24.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:24.315+0000 7ff1ca800640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7ff1cc1529c0 0x7ff1cc152e50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:55560/0 (socket says 172.21.15.16:55560) 2024-03-31T15:05:24.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:24.315+0000 7ff1ca800640 1 -- 172.21.15.16:0/2968788095 learned_addr learned my addr 172.21.15.16:0/2968788095 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:05:24.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:24.315+0000 7ff1c1400640 1 --2- 172.21.15.16:0/2968788095 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff1cc160150 0x7ff1cc0b4630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:24.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:24.315+0000 7ff1c9e00640 1 -- 172.21.15.16:0/2968788095 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff1cc160150 msgr2=0x7ff1cc0b4630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:05:24.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:24.315+0000 7ff1c9e00640 1 --2- 172.21.15.16:0/2968788095 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff1cc160150 0x7ff1cc0b4630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:24.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:24.315+0000 7ff1c9e00640 1 -- 172.21.15.16:0/2968788095 >> v1:172.21.15.16:6789/0 conn(0x7ff1cc1529c0 legacy=0x7ff1cc152e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:24.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:24.315+0000 7ff1c9e00640 1 -- 172.21.15.16:0/2968788095 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1cc0cfe10 con 0x7ff1cc15fd70 2024-03-31T15:05:24.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:24.315+0000 7ff1c9400640 1 -- 172.21.15.16:0/2968788095 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 146901076 0 0) 0x7ff1cc0cf850 con 0x7ff1cc1529c0 2024-03-31T15:05:24.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:24.319+0000 7ff1c9e00640 1 --2- 172.21.15.16:0/2968788095 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff1cc15fd70 0x7ff1cc0cf250 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7ff1bc009d90 tx=0x7ff1bc01c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=57bd2f362bd9e9da server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:05:24.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:24.319+0000 7ff1c9400640 1 -- 172.21.15.16:0/2968788095 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1bc0b2070 con 0x7ff1cc15fd70 2024-03-31T15:05:24.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:24.319+0000 7ff1c9400640 1 -- 172.21.15.16:0/2968788095 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff1bc0a5940 con 0x7ff1cc15fd70 2024-03-31T15:05:24.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:24.319+0000 7ff1c9400640 1 -- 172.21.15.16:0/2968788095 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1bc0a5d10 con 0x7ff1cc15fd70 2024-03-31T15:05:24.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:24.319+0000 7ff1d1000640 1 -- 172.21.15.16:0/2968788095 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff1cc15fd70 msgr2=0x7ff1cc0cf250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:24.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:24.319+0000 7ff1d1000640 1 --2- 172.21.15.16:0/2968788095 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff1cc15fd70 0x7ff1cc0cf250 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7ff1bc009d90 tx=0x7ff1bc01c970 comp rx=0 tx=0).stop 2024-03-31T15:05:24.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:24.319+0000 7ff1d1000640 1 -- 172.21.15.16:0/2968788095 shutdown_connections 2024-03-31T15:05:24.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:24.319+0000 7ff1d1000640 1 --2- 172.21.15.16:0/2968788095 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff1cc160150 0x7ff1cc0b4630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:24.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:24.319+0000 7ff1d1000640 1 --2- 172.21.15.16:0/2968788095 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff1cc15fd70 0x7ff1cc0cf250 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:24.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:24.319+0000 7ff1d1000640 1 -- 172.21.15.16:0/2968788095 >> 172.21.15.16:0/2968788095 conn(0x7ff1cc0d0760 msgr2=0x7ff1cc0d0b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:05:24.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:24.319+0000 7ff1d1000640 1 -- 172.21.15.16:0/2968788095 shutdown_connections 2024-03-31T15:05:24.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:24.319+0000 7ff1d1000640 1 -- 172.21.15.16:0/2968788095 wait complete. 2024-03-31T15:05:24.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:24.319+0000 7ff1d1000640 1 Processor -- start 2024-03-31T15:05:24.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:24.319+0000 7ff1d1000640 1 -- start start 2024-03-31T15:05:24.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:24.319+0000 7ff1d1000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff1cc1529c0 0x7ff1cc14a720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:24.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:24.319+0000 7ff1d1000640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff1cc15fd70 0x7ff1cc1462b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:24.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:24.319+0000 7ff1d1000640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff1cc160150 0x7ff1cc151db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:24.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:24.319+0000 7ff1d1000640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7ff1cc1606b0 con 0x7ff1cc1529c0 2024-03-31T15:05:24.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:24.319+0000 7ff1d1000640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7ff1cc160530 con 0x7ff1cc160150 2024-03-31T15:05:24.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:24.319+0000 7ff1d1000640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7ff1cc06bfa0 con 0x7ff1cc15fd70 2024-03-31T15:05:24.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:24.323+0000 7ff1c9e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff1cc1529c0 0x7ff1cc14a720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:24.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:24.323+0000 7ff1c9e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff1cc1529c0 0x7ff1cc14a720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49500/0 (socket says 172.21.15.16:49500) 2024-03-31T15:05:24.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:24.323+0000 7ff1c9e00640 1 -- 172.21.15.16:0/3783901969 learned_addr learned my addr 172.21.15.16:0/3783901969 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:05:24.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:24.323+0000 7ff1c1400640 1 --2- 172.21.15.16:0/3783901969 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff1cc15fd70 0x7ff1cc1462b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:24.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:24.323+0000 7ff1ca800640 1 --2- 172.21.15.16:0/3783901969 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff1cc160150 0x7ff1cc151db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:24.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:24.323+0000 7ff1c9e00640 1 -- 172.21.15.16:0/3783901969 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff1cc15fd70 msgr2=0x7ff1cc1462b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:24.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:24.323+0000 7ff1c9e00640 1 --2- 172.21.15.16:0/3783901969 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff1cc15fd70 0x7ff1cc1462b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:24.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:24.323+0000 7ff1c9e00640 1 -- 172.21.15.16:0/3783901969 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff1cc160150 msgr2=0x7ff1cc151db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:24.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:24.323+0000 7ff1c9e00640 1 --2- 172.21.15.16:0/3783901969 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff1cc160150 0x7ff1cc151db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:24.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:24.323+0000 7ff1c9e00640 1 -- 172.21.15.16:0/3783901969 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1cc14bc80 con 0x7ff1cc1529c0 2024-03-31T15:05:24.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:24.323+0000 7ff1c9e00640 1 --2- 172.21.15.16:0/3783901969 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff1cc1529c0 0x7ff1cc14a720 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7ff1bc01a5a0 tx=0x7ff1bc019710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:05:24.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:24.323+0000 7ff1c2a00640 1 -- 172.21.15.16:0/3783901969 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1bc0b2040 con 0x7ff1cc1529c0 2024-03-31T15:05:24.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:24.323+0000 7ff1c2a00640 1 -- 172.21.15.16:0/3783901969 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff1bc0bf070 con 0x7ff1cc1529c0 2024-03-31T15:05:24.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:24.323+0000 7ff1d1000640 1 -- 172.21.15.16:0/3783901969 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff1cc14bf20 con 0x7ff1cc1529c0 2024-03-31T15:05:24.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:24.323+0000 7ff1c2a00640 1 -- 172.21.15.16:0/3783901969 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1bc0ad920 con 0x7ff1cc1529c0 2024-03-31T15:05:24.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:24.323+0000 7ff1d1000640 1 -- 172.21.15.16:0/3783901969 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff1cc14ac60 con 0x7ff1cc1529c0 2024-03-31T15:05:24.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:24.323+0000 7ff1d1000640 1 -- 172.21.15.16:0/3783901969 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff1cc069950 con 0x7ff1cc1529c0 2024-03-31T15:05:24.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:24.323+0000 7ff1c2a00640 1 -- 172.21.15.16:0/3783901969 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7ff1bc0adac0 con 0x7ff1cc1529c0 2024-03-31T15:05:24.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:24.327+0000 7ff1c2a00640 1 --2- 172.21.15.16:0/3783901969 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7ff1a8068380 0x7ff1a806a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:24.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:24.327+0000 7ff1c2a00640 1 -- 172.21.15.16:0/3783901969 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 4857+0+0 (secure 0 0 0) 0x7ff1bc1216d0 con 0x7ff1cc1529c0 2024-03-31T15:05:24.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:24.327+0000 7ff1c1400640 1 --2- 172.21.15.16:0/3783901969 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7ff1a8068380 0x7ff1a806a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:24.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:24.327+0000 7ff1c1400640 1 --2- 172.21.15.16:0/3783901969 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7ff1a8068380 0x7ff1a806a840 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7ff1cc1f8380 tx=0x7ff1b4008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:05:24.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:24.331+0000 7ff1c2a00640 1 -- 172.21.15.16:0/3783901969 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7ff1bc016560 con 0x7ff1cc1529c0 2024-03-31T15:05:24.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:24.495+0000 7ff1d1000640 1 -- 172.21.15.16:0/3783901969 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7ff1cc0b5290 con 0x7ff1cc1529c0 2024-03-31T15:05:24.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:24.495+0000 7ff1c2a00640 1 -- 172.21.15.16:0/3783901969 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v41) ==== 84+0+65 (secure 0 0 0) 0x7ff1bc0f19a0 con 0x7ff1cc1529c0 2024-03-31T15:05:24.492 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:05:24.492 INFO:teuthology.orchestra.run.smithi016.stdout:[ 2024-03-31T15:05:24.492 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-31T15:05:24.492 INFO:teuthology.orchestra.run.smithi016.stdout: "poolnum": 1, 2024-03-31T15:05:24.492 INFO:teuthology.orchestra.run.smithi016.stdout: "poolname": ".mgr" 2024-03-31T15:05:24.492 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-31T15:05:24.492 INFO:teuthology.orchestra.run.smithi016.stdout:] 2024-03-31T15:05:24.494 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:24.499+0000 7ff1d1000640 1 -- 172.21.15.16:0/3783901969 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7ff1a8068380 msgr2=0x7ff1a806a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:24.494 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:24.499+0000 7ff1d1000640 1 --2- 172.21.15.16:0/3783901969 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7ff1a8068380 0x7ff1a806a840 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7ff1cc1f8380 tx=0x7ff1b4008040 comp rx=0 tx=0).stop 2024-03-31T15:05:24.494 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:24.499+0000 7ff1d1000640 1 -- 172.21.15.16:0/3783901969 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff1cc1529c0 msgr2=0x7ff1cc14a720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:24.494 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:24.499+0000 7ff1d1000640 1 --2- 172.21.15.16:0/3783901969 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff1cc1529c0 0x7ff1cc14a720 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7ff1bc01a5a0 tx=0x7ff1bc019710 comp rx=0 tx=0).stop 2024-03-31T15:05:24.494 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:24.499+0000 7ff1d1000640 1 -- 172.21.15.16:0/3783901969 shutdown_connections 2024-03-31T15:05:24.494 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:24.499+0000 7ff1d1000640 1 --2- 172.21.15.16:0/3783901969 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7ff1a8068380 0x7ff1a806a840 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:24.494 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:24.499+0000 7ff1d1000640 1 --2- 172.21.15.16:0/3783901969 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff1cc160150 0x7ff1cc151db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:24.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:24.499+0000 7ff1d1000640 1 --2- 172.21.15.16:0/3783901969 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff1cc15fd70 0x7ff1cc1462b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:24.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:24.499+0000 7ff1d1000640 1 --2- 172.21.15.16:0/3783901969 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff1cc1529c0 0x7ff1cc14a720 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:24.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:24.499+0000 7ff1d1000640 1 -- 172.21.15.16:0/3783901969 >> 172.21.15.16:0/3783901969 conn(0x7ff1cc0d0760 msgr2=0x7ff1cc15ddb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:05:24.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:24.499+0000 7ff1d1000640 1 -- 172.21.15.16:0/3783901969 shutdown_connections 2024-03-31T15:05:24.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:24.499+0000 7ff1d1000640 1 -- 172.21.15.16:0/3783901969 wait complete. 2024-03-31T15:05:24.512 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-31T15:05:24.512 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-31T15:05:24.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:24.663+0000 7f4732600640 1 Processor -- start 2024-03-31T15:05:24.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:24.663+0000 7f4732600640 1 -- start start 2024-03-31T15:05:24.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:24.663+0000 7f4732600640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f472c0b22f0 0x7f472c15fde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:24.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:24.663+0000 7f4732600640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f472c160320 0x7f472c140550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:24.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:24.663+0000 7f4732600640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f472c13f7b0 con 0x7f472c13efd0 2024-03-31T15:05:24.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:24.663+0000 7f4732600640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f472c160b80 con 0x7f472c160320 2024-03-31T15:05:24.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:24.663+0000 7f4732600640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f472c13fa70 con 0x7f472c0b22f0 2024-03-31T15:05:24.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:24.663+0000 7f472be00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f472c160320 0x7f472c140550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:24.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:24.663+0000 7f472be00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f472c160320 0x7f472c140550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49514/0 (socket says 172.21.15.16:49514) 2024-03-31T15:05:24.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:24.663+0000 7f472be00640 1 -- 172.21.15.16:0/4111978120 learned_addr learned my addr 172.21.15.16:0/4111978120 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:05:24.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:24.663+0000 7f4723e00640 1 --2- 172.21.15.16:0/4111978120 >> v2:172.21.15.89:3300/0 conn(0x7f472c0b22f0 0x7f472c15fde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:24.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:24.663+0000 7f472be00640 1 -- 172.21.15.16:0/4111978120 >> v2:172.21.15.89:3300/0 conn(0x7f472c0b22f0 msgr2=0x7f472c15fde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:05:24.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:24.663+0000 7f472be00640 1 --2- 172.21.15.16:0/4111978120 >> v2:172.21.15.89:3300/0 conn(0x7f472c0b22f0 0x7f472c15fde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:24.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:24.663+0000 7f472be00640 1 -- 172.21.15.16:0/4111978120 >> v1:172.21.15.89:6789/0 conn(0x7f472c13efd0 legacy=0x7f472c0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:24.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:24.663+0000 7f472be00640 1 -- 172.21.15.16:0/4111978120 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f472c13fd50 con 0x7f472c160320 2024-03-31T15:05:24.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:24.663+0000 7f4723e00640 1 --2- 172.21.15.16:0/4111978120 >> v2:172.21.15.89:3300/0 conn(0x7f472c0b22f0 0x7f472c15fde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:05:24.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:24.667+0000 7f472be00640 1 --2- 172.21.15.16:0/4111978120 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f472c160320 0x7f472c140550 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f472c0ce070 tx=0x7f472401c930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d962dfc26f242f6 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:05:24.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:24.667+0000 7f472aa00640 1 -- 172.21.15.16:0/4111978120 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47240a57d0 con 0x7f472c160320 2024-03-31T15:05:24.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:24.667+0000 7f472aa00640 1 -- 172.21.15.16:0/4111978120 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f47240a5970 con 0x7f472c160320 2024-03-31T15:05:24.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:24.667+0000 7f472aa00640 1 -- 172.21.15.16:0/4111978120 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47240a5d40 con 0x7f472c160320 2024-03-31T15:05:24.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:24.667+0000 7f4732600640 1 -- 172.21.15.16:0/4111978120 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f472c160320 msgr2=0x7f472c140550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:24.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:24.667+0000 7f4732600640 1 --2- 172.21.15.16:0/4111978120 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f472c160320 0x7f472c140550 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f472c0ce070 tx=0x7f472401c930 comp rx=0 tx=0).stop 2024-03-31T15:05:24.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:24.667+0000 7f4732600640 1 -- 172.21.15.16:0/4111978120 shutdown_connections 2024-03-31T15:05:24.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:24.667+0000 7f4732600640 1 --2- 172.21.15.16:0/4111978120 >> v2:172.21.15.89:3300/0 conn(0x7f472c0b22f0 0x7f472c15fde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:24.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:24.667+0000 7f4732600640 1 --2- 172.21.15.16:0/4111978120 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f472c160320 0x7f472c140550 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:24.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:24.667+0000 7f4732600640 1 -- 172.21.15.16:0/4111978120 >> 172.21.15.16:0/4111978120 conn(0x7f472c06b620 msgr2=0x7f472c06ba30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:05:24.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:24.667+0000 7f4732600640 1 -- 172.21.15.16:0/4111978120 shutdown_connections 2024-03-31T15:05:24.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:24.667+0000 7f4732600640 1 -- 172.21.15.16:0/4111978120 wait complete. 2024-03-31T15:05:24.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:24.671+0000 7f4732600640 1 Processor -- start 2024-03-31T15:05:24.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:24.671+0000 7f4732600640 1 -- start start 2024-03-31T15:05:24.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:24.671+0000 7f4732600640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f472c0b22f0 0x7f472c0d0740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:24.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:24.671+0000 7f4732600640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f472c13efd0 0x7f472c0cfa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:24.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:24.671+0000 7f4732600640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f472c160320 0x7f472c0ae800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:24.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:24.671+0000 7f4732600640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f472c160930 con 0x7f472c0b22f0 2024-03-31T15:05:24.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:24.671+0000 7f4732600640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f472c1607b0 con 0x7f472c160320 2024-03-31T15:05:24.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:24.671+0000 7f4732600640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f472c1412a0 con 0x7f472c13efd0 2024-03-31T15:05:24.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:24.671+0000 7f472b400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f472c0b22f0 0x7f472c0d0740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:24.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:24.671+0000 7f472b400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f472c0b22f0 0x7f472c0d0740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49530/0 (socket says 172.21.15.16:49530) 2024-03-31T15:05:24.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:24.671+0000 7f472b400640 1 -- 172.21.15.16:0/1489029826 learned_addr learned my addr 172.21.15.16:0/1489029826 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:05:24.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:24.671+0000 7f4723e00640 1 --2- 172.21.15.16:0/1489029826 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f472c13efd0 0x7f472c0cfa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:24.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:24.671+0000 7f472be00640 1 --2- 172.21.15.16:0/1489029826 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f472c160320 0x7f472c0ae800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:24.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:24.671+0000 7f472b400640 1 -- 172.21.15.16:0/1489029826 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f472c13efd0 msgr2=0x7f472c0cfa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:24.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:24.671+0000 7f472b400640 1 --2- 172.21.15.16:0/1489029826 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f472c13efd0 0x7f472c0cfa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:24.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:24.671+0000 7f472b400640 1 -- 172.21.15.16:0/1489029826 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f472c160320 msgr2=0x7f472c0ae800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:24.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:24.671+0000 7f472b400640 1 --2- 172.21.15.16:0/1489029826 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f472c160320 0x7f472c0ae800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:24.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:24.671+0000 7f472b400640 1 -- 172.21.15.16:0/1489029826 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f472c12d5a0 con 0x7f472c0b22f0 2024-03-31T15:05:24.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:24.671+0000 7f472b400640 1 --2- 172.21.15.16:0/1489029826 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f472c0b22f0 0x7f472c0d0740 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f471c00c530 tx=0x7f471c00c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:05:24.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:24.671+0000 7f4723400640 1 -- 172.21.15.16:0/1489029826 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f471c016020 con 0x7f472c0b22f0 2024-03-31T15:05:24.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:24.671+0000 7f4723400640 1 -- 172.21.15.16:0/1489029826 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f471c005220 con 0x7f472c0b22f0 2024-03-31T15:05:24.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:24.671+0000 7f4723400640 1 -- 172.21.15.16:0/1489029826 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f471c007030 con 0x7f472c0b22f0 2024-03-31T15:05:24.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:24.671+0000 7f4732600640 1 -- 172.21.15.16:0/1489029826 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f472c12d880 con 0x7f472c0b22f0 2024-03-31T15:05:24.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:24.671+0000 7f4732600640 1 -- 172.21.15.16:0/1489029826 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f472c12de40 con 0x7f472c0b22f0 2024-03-31T15:05:24.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:24.671+0000 7f4732600640 1 -- 172.21.15.16:0/1489029826 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f472c12e180 con 0x7f472c0b22f0 2024-03-31T15:05:24.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:24.675+0000 7f4723400640 1 -- 172.21.15.16:0/1489029826 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f471c0071d0 con 0x7f472c0b22f0 2024-03-31T15:05:24.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:24.675+0000 7f4723400640 1 --2- 172.21.15.16:0/1489029826 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f4708068380 0x7f470806a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:24.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:24.675+0000 7f4723400640 1 -- 172.21.15.16:0/1489029826 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 4857+0+0 (secure 0 0 0) 0x7f471c085f60 con 0x7f472c0b22f0 2024-03-31T15:05:24.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:24.675+0000 7f4723e00640 1 --2- 172.21.15.16:0/1489029826 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f4708068380 0x7f470806a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:24.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:24.675+0000 7f4723e00640 1 --2- 172.21.15.16:0/1489029826 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f4708068380 0x7f470806a840 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f472c13e270 tx=0x7f4714008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:05:24.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:24.679+0000 7f4723400640 1 -- 172.21.15.16:0/1489029826 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f471c020020 con 0x7f472c0b22f0 2024-03-31T15:05:24.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:24.843+0000 7f4732600640 1 -- 172.21.15.16:0/1489029826 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7f472c069950 con 0x7f472c0b22f0 2024-03-31T15:05:25.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:25.047+0000 7f4723400640 1 -- 172.21.15.16:0/1489029826 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v42) ==== 138+0+0 (secure 0 0 0) 0x7f471c0074c0 con 0x7f472c0b22f0 2024-03-31T15:05:25.041 INFO:teuthology.orchestra.run.smithi016.stderr:pool 'cephfs_metadata' created 2024-03-31T15:05:25.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:25.047+0000 7f4732600640 1 -- 172.21.15.16:0/1489029826 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f4708068380 msgr2=0x7f470806a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:25.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:25.047+0000 7f4732600640 1 --2- 172.21.15.16:0/1489029826 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f4708068380 0x7f470806a840 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f472c13e270 tx=0x7f4714008040 comp rx=0 tx=0).stop 2024-03-31T15:05:25.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:25.047+0000 7f4732600640 1 -- 172.21.15.16:0/1489029826 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f472c0b22f0 msgr2=0x7f472c0d0740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:25.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:25.047+0000 7f4732600640 1 --2- 172.21.15.16:0/1489029826 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f472c0b22f0 0x7f472c0d0740 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f471c00c530 tx=0x7f471c00c9f0 comp rx=0 tx=0).stop 2024-03-31T15:05:25.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:25.047+0000 7f4732600640 1 -- 172.21.15.16:0/1489029826 shutdown_connections 2024-03-31T15:05:25.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:25.047+0000 7f4732600640 1 --2- 172.21.15.16:0/1489029826 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f4708068380 0x7f470806a840 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:25.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:25.047+0000 7f4732600640 1 --2- 172.21.15.16:0/1489029826 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f472c160320 0x7f472c0ae800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:25.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:25.047+0000 7f4732600640 1 --2- 172.21.15.16:0/1489029826 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f472c13efd0 0x7f472c0cfa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:25.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:25.047+0000 7f4732600640 1 --2- 172.21.15.16:0/1489029826 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f472c0b22f0 0x7f472c0d0740 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:25.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:25.047+0000 7f4732600640 1 -- 172.21.15.16:0/1489029826 >> 172.21.15.16:0/1489029826 conn(0x7f472c06b620 msgr2=0x7f472c06dfa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:05:25.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:25.047+0000 7f4732600640 1 -- 172.21.15.16:0/1489029826 shutdown_connections 2024-03-31T15:05:25.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:25.047+0000 7f4732600640 1 -- 172.21.15.16:0/1489029826 wait complete. 2024-03-31T15:05:25.069 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-31T15:05:25.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:25.223+0000 7f38b4000640 1 Processor -- start 2024-03-31T15:05:25.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:25.223+0000 7f38b4000640 1 -- start start 2024-03-31T15:05:25.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:25.223+0000 7f38b4000640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f38ac1567e0 0x7f38ac072c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:25.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:25.223+0000 7f38b4000640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f38ac06dbe0 0x7f38ac071550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:25.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:25.223+0000 7f38b4000640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f38ac15c7e0 con 0x7f38ac0731c0 2024-03-31T15:05:25.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:25.223+0000 7f38b4000640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f38ac0736b0 con 0x7f38ac1567e0 2024-03-31T15:05:25.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:25.223+0000 7f38b4000640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f38ac15caa0 con 0x7f38ac06dbe0 2024-03-31T15:05:25.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:25.223+0000 7f38b1600640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f38ac1567e0 0x7f38ac072c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:25.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:25.223+0000 7f38b1600640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f38ac1567e0 0x7f38ac072c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49534/0 (socket says 172.21.15.16:49534) 2024-03-31T15:05:25.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:25.223+0000 7f38b1600640 1 -- 172.21.15.16:0/1527477669 learned_addr learned my addr 172.21.15.16:0/1527477669 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:05:25.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:25.223+0000 7f38b2000640 1 --2- 172.21.15.16:0/1527477669 >> v2:172.21.15.89:3300/0 conn(0x7f38ac06dbe0 0x7f38ac071550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:25.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:25.223+0000 7f38b1600640 1 -- 172.21.15.16:0/1527477669 >> v2:172.21.15.89:3300/0 conn(0x7f38ac06dbe0 msgr2=0x7f38ac071550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:05:25.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:25.223+0000 7f38b1600640 1 --2- 172.21.15.16:0/1527477669 >> v2:172.21.15.89:3300/0 conn(0x7f38ac06dbe0 0x7f38ac071550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:25.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:25.223+0000 7f38b1600640 1 -- 172.21.15.16:0/1527477669 >> v1:172.21.15.89:6789/0 conn(0x7f38ac0731c0 legacy=0x7f38ac06d4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:05:25.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:25.223+0000 7f38b1600640 1 -- 172.21.15.16:0/1527477669 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38ac15cd80 con 0x7f38ac1567e0 2024-03-31T15:05:25.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:25.227+0000 7f38b1600640 1 --2- 172.21.15.16:0/1527477669 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f38ac1567e0 0x7f38ac072c80 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f38ac15a3b0 tx=0x7f38a001c9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b9eeba54e883983 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:05:25.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:25.227+0000 7f38abe00640 1 -- 172.21.15.16:0/1527477669 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38a00aa450 con 0x7f38ac1567e0 2024-03-31T15:05:25.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:25.227+0000 7f38abe00640 1 -- 172.21.15.16:0/1527477669 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f38a00af070 con 0x7f38ac1567e0 2024-03-31T15:05:25.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:25.227+0000 7f38abe00640 1 -- 172.21.15.16:0/1527477669 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38a00aa810 con 0x7f38ac1567e0 2024-03-31T15:05:25.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:25.227+0000 7f38b4000640 1 -- 172.21.15.16:0/1527477669 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f38ac1567e0 msgr2=0x7f38ac072c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:25.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:25.227+0000 7f38b4000640 1 --2- 172.21.15.16:0/1527477669 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f38ac1567e0 0x7f38ac072c80 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f38ac15a3b0 tx=0x7f38a001c9b0 comp rx=0 tx=0).stop 2024-03-31T15:05:25.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:25.227+0000 7f38b4000640 1 -- 172.21.15.16:0/1527477669 shutdown_connections 2024-03-31T15:05:25.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:25.227+0000 7f38b4000640 1 --2- 172.21.15.16:0/1527477669 >> v2:172.21.15.89:3300/0 conn(0x7f38ac06dbe0 0x7f38ac071550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:25.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:25.227+0000 7f38b4000640 1 --2- 172.21.15.16:0/1527477669 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f38ac1567e0 0x7f38ac072c80 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:25.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:25.227+0000 7f38b4000640 1 -- 172.21.15.16:0/1527477669 >> 172.21.15.16:0/1527477669 conn(0x7f38ac06ee40 msgr2=0x7f38ac06b620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:05:25.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:25.227+0000 7f38b4000640 1 -- 172.21.15.16:0/1527477669 shutdown_connections 2024-03-31T15:05:25.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:25.227+0000 7f38b4000640 1 -- 172.21.15.16:0/1527477669 wait complete. 2024-03-31T15:05:25.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:25.231+0000 7f38b4000640 1 Processor -- start 2024-03-31T15:05:25.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:25.231+0000 7f38b4000640 1 -- start start 2024-03-31T15:05:25.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:25.231+0000 7f38b4000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f38ac06dbe0 0x7f38ac15eb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:25.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:25.231+0000 7f38b4000640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f38ac0731c0 0x7f38ac1663e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:25.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:25.231+0000 7f38b4000640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f38ac1567e0 0x7f38ac148450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:25.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:25.231+0000 7f38b4000640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f38ac1692f0 con 0x7f38ac06dbe0 2024-03-31T15:05:25.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:25.231+0000 7f38b4000640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f38ac169170 con 0x7f38ac1567e0 2024-03-31T15:05:25.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:25.231+0000 7f38b4000640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f38ac169470 con 0x7f38ac0731c0 2024-03-31T15:05:25.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:25.231+0000 7f38b1600640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f38ac06dbe0 0x7f38ac15eb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:25.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:25.231+0000 7f38b1600640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f38ac06dbe0 0x7f38ac15eb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49546/0 (socket says 172.21.15.16:49546) 2024-03-31T15:05:25.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:25.231+0000 7f38b1600640 1 -- 172.21.15.16:0/1544583367 learned_addr learned my addr 172.21.15.16:0/1544583367 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:05:25.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:25.231+0000 7f38b0c00640 1 --2- 172.21.15.16:0/1544583367 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f38ac0731c0 0x7f38ac1663e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:25.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:25.231+0000 7f38b2000640 1 --2- 172.21.15.16:0/1544583367 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f38ac1567e0 0x7f38ac148450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:25.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:25.231+0000 7f38b1600640 1 -- 172.21.15.16:0/1544583367 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f38ac0731c0 msgr2=0x7f38ac1663e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:25.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:25.231+0000 7f38b1600640 1 --2- 172.21.15.16:0/1544583367 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f38ac0731c0 0x7f38ac1663e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:25.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:25.231+0000 7f38b1600640 1 -- 172.21.15.16:0/1544583367 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f38ac1567e0 msgr2=0x7f38ac148450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:25.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:25.231+0000 7f38b1600640 1 --2- 172.21.15.16:0/1544583367 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f38ac1567e0 0x7f38ac148450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:25.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:25.231+0000 7f38b1600640 1 -- 172.21.15.16:0/1544583367 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38ac1653b0 con 0x7f38ac06dbe0 2024-03-31T15:05:25.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:25.231+0000 7f38b1600640 1 --2- 172.21.15.16:0/1544583367 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f38ac06dbe0 0x7f38ac15eb70 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f38a00ac040 tx=0x7f38a0019750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:05:25.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:25.231+0000 7f38a9600640 1 -- 172.21.15.16:0/1544583367 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38a00b9070 con 0x7f38ac06dbe0 2024-03-31T15:05:25.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:25.231+0000 7f38a9600640 1 -- 172.21.15.16:0/1544583367 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f38a00a6040 con 0x7f38ac06dbe0 2024-03-31T15:05:25.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:25.231+0000 7f38b4000640 1 -- 172.21.15.16:0/1544583367 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f38ac165900 con 0x7f38ac06dbe0 2024-03-31T15:05:25.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:25.231+0000 7f38a9600640 1 -- 172.21.15.16:0/1544583367 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38a00aaac0 con 0x7f38ac06dbe0 2024-03-31T15:05:25.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:25.231+0000 7f38b4000640 1 -- 172.21.15.16:0/1544583367 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f38ac165ec0 con 0x7f38ac06dbe0 2024-03-31T15:05:25.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:25.231+0000 7f38b4000640 1 -- 172.21.15.16:0/1544583367 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f38ac069950 con 0x7f38ac06dbe0 2024-03-31T15:05:25.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:25.235+0000 7f38a9600640 1 -- 172.21.15.16:0/1544583367 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f38a00aac60 con 0x7f38ac06dbe0 2024-03-31T15:05:25.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:25.235+0000 7f38a9600640 1 --2- 172.21.15.16:0/1544583367 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f388c068380 0x7f388c06a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:25.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:25.235+0000 7f38a9600640 1 -- 172.21.15.16:0/1544583367 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 5227+0+0 (secure 0 0 0) 0x7f38a0120510 con 0x7f38ac06dbe0 2024-03-31T15:05:25.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:25.235+0000 7f38b0c00640 1 --2- 172.21.15.16:0/1544583367 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f388c068380 0x7f388c06a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:25.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:25.235+0000 7f38b0c00640 1 --2- 172.21.15.16:0/1544583367 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f388c068380 0x7f388c06a840 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f38ac1fabe0 tx=0x7f3898007e40 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:05:25.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:25.239+0000 7f38a9600640 1 -- 172.21.15.16:0/1544583367 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f38a00c3060 con 0x7f38ac06dbe0 2024-03-31T15:05:25.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:25.403+0000 7f38b4000640 1 -- 172.21.15.16:0/1544583367 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7f38ac160210 con 0x7f38ac06dbe0 2024-03-31T15:05:26.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:26.055+0000 7f38a9600640 1 -- 172.21.15.16:0/1544583367 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v43) ==== 170+0+0 (secure 0 0 0) 0x7f38a00165a0 con 0x7f38ac06dbe0 2024-03-31T15:05:26.050 INFO:teuthology.orchestra.run.smithi016.stderr:pool 'cephfs_data' created 2024-03-31T15:05:26.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:26.059+0000 7f38b4000640 1 -- 172.21.15.16:0/1544583367 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f388c068380 msgr2=0x7f388c06a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:26.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:26.059+0000 7f38b4000640 1 --2- 172.21.15.16:0/1544583367 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f388c068380 0x7f388c06a840 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f38ac1fabe0 tx=0x7f3898007e40 comp rx=0 tx=0).stop 2024-03-31T15:05:26.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:26.059+0000 7f38b4000640 1 -- 172.21.15.16:0/1544583367 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f38ac06dbe0 msgr2=0x7f38ac15eb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:26.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:26.059+0000 7f38b4000640 1 --2- 172.21.15.16:0/1544583367 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f38ac06dbe0 0x7f38ac15eb70 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f38a00ac040 tx=0x7f38a0019750 comp rx=0 tx=0).stop 2024-03-31T15:05:26.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:26.059+0000 7f38b4000640 1 -- 172.21.15.16:0/1544583367 shutdown_connections 2024-03-31T15:05:26.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:26.059+0000 7f38b4000640 1 --2- 172.21.15.16:0/1544583367 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f388c068380 0x7f388c06a840 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:26.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:26.059+0000 7f38b4000640 1 --2- 172.21.15.16:0/1544583367 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f38ac1567e0 0x7f38ac148450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:26.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:26.059+0000 7f38b4000640 1 --2- 172.21.15.16:0/1544583367 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f38ac0731c0 0x7f38ac1663e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:26.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:26.059+0000 7f38b4000640 1 --2- 172.21.15.16:0/1544583367 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f38ac06dbe0 0x7f38ac15eb70 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:26.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:26.059+0000 7f38b4000640 1 -- 172.21.15.16:0/1544583367 >> 172.21.15.16:0/1544583367 conn(0x7f38ac06ee40 msgr2=0x7f38ac15b780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:05:26.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:26.059+0000 7f38b4000640 1 -- 172.21.15.16:0/1544583367 shutdown_connections 2024-03-31T15:05:26.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:26.059+0000 7f38b4000640 1 -- 172.21.15.16:0/1544583367 wait complete. 2024-03-31T15:05:26.090 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-31T15:05:26.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:26.239+0000 7f7a56a00640 1 Processor -- start 2024-03-31T15:05:26.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:26.239+0000 7f7a56a00640 1 -- start start 2024-03-31T15:05:26.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:26.243+0000 7f7a56a00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f7a5013dff0 0x7f7a5006d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:26.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:26.243+0000 7f7a56a00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f7a5013e3d0 0x7f7a5006da10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:26.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:26.243+0000 7f7a56a00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f7a500cc9d0 0x7f7a500cce30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:26.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:26.243+0000 7f7a56a00640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f7a5015f730 con 0x7f7a5013dff0 2024-03-31T15:05:26.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:26.243+0000 7f7a56a00640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f7a5015fbc0 con 0x7f7a500cc9d0 2024-03-31T15:05:26.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:26.243+0000 7f7a56a00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f7a5015ee00 con 0x7f7a5013e3d0 2024-03-31T15:05:26.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:26.243+0000 7f7a4fe00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f7a5013dff0 0x7f7a5006d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:26.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:26.243+0000 7f7a4fe00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f7a5013dff0 0x7f7a5006d4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49554/0 (socket says 172.21.15.16:49554) 2024-03-31T15:05:26.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:26.243+0000 7f7a4fe00640 1 -- 172.21.15.16:0/3462321178 learned_addr learned my addr 172.21.15.16:0/3462321178 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:05:26.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:26.243+0000 7f7a54a00640 1 --2- 172.21.15.16:0/3462321178 >> v2:172.21.15.89:3300/0 conn(0x7f7a500cc9d0 0x7f7a500cce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:26.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:26.243+0000 7f7a47400640 1 --2- 172.21.15.16:0/3462321178 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f7a5013e3d0 0x7f7a5006da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:26.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:26.243+0000 7f7a4fe00640 1 -- 172.21.15.16:0/3462321178 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f7a5013e3d0 msgr2=0x7f7a5006da10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:05:26.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:26.243+0000 7f7a4fe00640 1 --2- 172.21.15.16:0/3462321178 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f7a5013e3d0 0x7f7a5006da10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:26.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:26.243+0000 7f7a4fe00640 1 -- 172.21.15.16:0/3462321178 >> v2:172.21.15.89:3300/0 conn(0x7f7a500cc9d0 msgr2=0x7f7a500cce30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:05:26.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:26.243+0000 7f7a4fe00640 1 --2- 172.21.15.16:0/3462321178 >> v2:172.21.15.89:3300/0 conn(0x7f7a500cc9d0 0x7f7a500cce30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:26.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:26.243+0000 7f7a4fe00640 1 -- 172.21.15.16:0/3462321178 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a5015f0e0 con 0x7f7a5013dff0 2024-03-31T15:05:26.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:26.247+0000 7f7a4fe00640 1 --2- 172.21.15.16:0/3462321178 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7a5013dff0 0x7f7a5006d4d0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f7a500cc980 tx=0x7f7a4001cef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b74f0496ec3361de server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:05:26.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:26.247+0000 7f7a4f400640 1 -- 172.21.15.16:0/3462321178 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a400b2070 con 0x7f7a5013dff0 2024-03-31T15:05:26.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:26.247+0000 7f7a4f400640 1 -- 172.21.15.16:0/3462321178 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7a400a9070 con 0x7f7a5013dff0 2024-03-31T15:05:26.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:26.247+0000 7f7a4f400640 1 -- 172.21.15.16:0/3462321178 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a400ad880 con 0x7f7a5013dff0 2024-03-31T15:05:26.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:26.247+0000 7f7a56a00640 1 -- 172.21.15.16:0/3462321178 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7a5013dff0 msgr2=0x7f7a5006d4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:26.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:26.247+0000 7f7a56a00640 1 --2- 172.21.15.16:0/3462321178 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7a5013dff0 0x7f7a5006d4d0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f7a500cc980 tx=0x7f7a4001cef0 comp rx=0 tx=0).stop 2024-03-31T15:05:26.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:26.247+0000 7f7a56a00640 1 -- 172.21.15.16:0/3462321178 shutdown_connections 2024-03-31T15:05:26.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:26.247+0000 7f7a56a00640 1 --2- 172.21.15.16:0/3462321178 >> v2:172.21.15.89:3300/0 conn(0x7f7a500cc9d0 0x7f7a500cce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:26.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:26.247+0000 7f7a56a00640 1 --2- 172.21.15.16:0/3462321178 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f7a5013e3d0 0x7f7a5006da10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:26.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:26.247+0000 7f7a56a00640 1 --2- 172.21.15.16:0/3462321178 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7a5013dff0 0x7f7a5006d4d0 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:26.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:26.247+0000 7f7a56a00640 1 -- 172.21.15.16:0/3462321178 >> 172.21.15.16:0/3462321178 conn(0x7f7a5006ee40 msgr2=0x7f7a5006b620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:05:26.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:26.247+0000 7f7a56a00640 1 -- 172.21.15.16:0/3462321178 shutdown_connections 2024-03-31T15:05:26.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:26.247+0000 7f7a56a00640 1 -- 172.21.15.16:0/3462321178 wait complete. 2024-03-31T15:05:26.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:26.247+0000 7f7a56a00640 1 Processor -- start 2024-03-31T15:05:26.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:26.247+0000 7f7a56a00640 1 -- start start 2024-03-31T15:05:26.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:26.251+0000 7f7a56a00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f7a500cc9d0 0x7f7a5013c7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:26.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:26.251+0000 7f7a56a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7a5013dff0 0x7f7a5015add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:26.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:26.251+0000 7f7a56a00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7a5013e3d0 0x7f7a500b3560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:26.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:26.251+0000 7f7a56a00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f7a50073200 con 0x7f7a5013dff0 2024-03-31T15:05:26.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:26.251+0000 7f7a56a00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f7a50073080 con 0x7f7a5013e3d0 2024-03-31T15:05:26.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:26.251+0000 7f7a56a00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f7a50073380 con 0x7f7a500cc9d0 2024-03-31T15:05:26.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:26.251+0000 7f7a47400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7a5013dff0 0x7f7a5015add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:26.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:26.251+0000 7f7a47400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7a5013dff0 0x7f7a5015add0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49568/0 (socket says 172.21.15.16:49568) 2024-03-31T15:05:26.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:26.251+0000 7f7a47400640 1 -- 172.21.15.16:0/2085578006 learned_addr learned my addr 172.21.15.16:0/2085578006 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:05:26.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:26.251+0000 7f7a54a00640 1 --2- 172.21.15.16:0/2085578006 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7a5013e3d0 0x7f7a500b3560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:26.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:26.251+0000 7f7a4fe00640 1 --2- 172.21.15.16:0/2085578006 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f7a500cc9d0 0x7f7a5013c7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:26.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:26.251+0000 7f7a47400640 1 -- 172.21.15.16:0/2085578006 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f7a500cc9d0 msgr2=0x7f7a5013c7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:26.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:26.251+0000 7f7a47400640 1 --2- 172.21.15.16:0/2085578006 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f7a500cc9d0 0x7f7a5013c7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:26.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:26.251+0000 7f7a47400640 1 -- 172.21.15.16:0/2085578006 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7a5013e3d0 msgr2=0x7f7a500b3560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:26.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:26.251+0000 7f7a47400640 1 --2- 172.21.15.16:0/2085578006 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7a5013e3d0 0x7f7a500b3560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:26.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:26.251+0000 7f7a47400640 1 -- 172.21.15.16:0/2085578006 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a501faca0 con 0x7f7a5013dff0 2024-03-31T15:05:26.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:26.251+0000 7f7a4fe00640 1 --2- 172.21.15.16:0/2085578006 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f7a500cc9d0 0x7f7a5013c7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:05:26.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:26.251+0000 7f7a54a00640 1 --2- 172.21.15.16:0/2085578006 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7a5013e3d0 0x7f7a500b3560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:05:26.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:26.251+0000 7f7a47400640 1 --2- 172.21.15.16:0/2085578006 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7a5013dff0 0x7f7a5015add0 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f7a3800e8a0 tx=0x7f7a3800eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:05:26.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:26.251+0000 7f7a4cc00640 1 -- 172.21.15.16:0/2085578006 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a38017070 con 0x7f7a5013dff0 2024-03-31T15:05:26.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:26.251+0000 7f7a4cc00640 1 -- 172.21.15.16:0/2085578006 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7a3800c940 con 0x7f7a5013dff0 2024-03-31T15:05:26.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:26.251+0000 7f7a4cc00640 1 -- 172.21.15.16:0/2085578006 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a3800ccf0 con 0x7f7a5013dff0 2024-03-31T15:05:26.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:26.251+0000 7f7a56a00640 1 -- 172.21.15.16:0/2085578006 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7a501faf80 con 0x7f7a5013dff0 2024-03-31T15:05:26.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:26.251+0000 7f7a56a00640 1 -- 172.21.15.16:0/2085578006 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7a501fb540 con 0x7f7a5013dff0 2024-03-31T15:05:26.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:26.251+0000 7f7a56a00640 1 -- 172.21.15.16:0/2085578006 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7a50069950 con 0x7f7a5013dff0 2024-03-31T15:05:26.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:26.251+0000 7f7a4cc00640 1 -- 172.21.15.16:0/2085578006 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f7a3801a070 con 0x7f7a5013dff0 2024-03-31T15:05:26.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:26.255+0000 7f7a4cc00640 1 --2- 172.21.15.16:0/2085578006 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f7a2c068380 0x7f7a2c06a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:26.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:26.255+0000 7f7a4cc00640 1 -- 172.21.15.16:0/2085578006 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 5609+0+0 (secure 0 0 0) 0x7f7a38086b30 con 0x7f7a5013dff0 2024-03-31T15:05:26.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:26.255+0000 7f7a4fe00640 1 --2- 172.21.15.16:0/2085578006 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f7a2c068380 0x7f7a2c06a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:26.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:26.255+0000 7f7a4fe00640 1 --2- 172.21.15.16:0/2085578006 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f7a2c068380 0x7f7a2c06a840 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f7a400af000 tx=0x7f7a40008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:05:26.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:26.259+0000 7f7a4cc00640 1 -- 172.21.15.16:0/2085578006 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f7a380571e0 con 0x7f7a5013dff0 2024-03-31T15:05:26.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:26.455+0000 7f7a56a00640 1 -- 172.21.15.16:0/2085578006 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7f7a500ccec0 con 0x7f7a5013dff0 2024-03-31T15:05:27.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:27.075+0000 7f7a4cc00640 1 -- 172.21.15.16:0/2085578006 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 6 and data pool 7 v21) ==== 172+0+0 (secure 0 0 0) 0x7f7a38056ae0 con 0x7f7a5013dff0 2024-03-31T15:05:27.078 INFO:teuthology.orchestra.run.smithi016.stderr:new fs with metadata pool 6 and data pool 7 2024-03-31T15:05:27.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:27.087+0000 7f7a56a00640 1 -- 172.21.15.16:0/2085578006 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f7a2c068380 msgr2=0x7f7a2c06a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:27.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:27.087+0000 7f7a56a00640 1 --2- 172.21.15.16:0/2085578006 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f7a2c068380 0x7f7a2c06a840 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f7a400af000 tx=0x7f7a40008040 comp rx=0 tx=0).stop 2024-03-31T15:05:27.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:27.087+0000 7f7a56a00640 1 -- 172.21.15.16:0/2085578006 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7a5013dff0 msgr2=0x7f7a5015add0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:27.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:27.087+0000 7f7a56a00640 1 --2- 172.21.15.16:0/2085578006 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7a5013dff0 0x7f7a5015add0 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f7a3800e8a0 tx=0x7f7a3800eca0 comp rx=0 tx=0).stop 2024-03-31T15:05:27.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:27.091+0000 7f7a56a00640 1 -- 172.21.15.16:0/2085578006 shutdown_connections 2024-03-31T15:05:27.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:27.091+0000 7f7a56a00640 1 --2- 172.21.15.16:0/2085578006 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f7a2c068380 0x7f7a2c06a840 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:27.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:27.091+0000 7f7a56a00640 1 --2- 172.21.15.16:0/2085578006 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7a5013e3d0 0x7f7a500b3560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:27.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:27.091+0000 7f7a56a00640 1 --2- 172.21.15.16:0/2085578006 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7a5013dff0 0x7f7a5015add0 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:27.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:27.091+0000 7f7a56a00640 1 --2- 172.21.15.16:0/2085578006 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f7a500cc9d0 0x7f7a5013c7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:27.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:27.091+0000 7f7a56a00640 1 -- 172.21.15.16:0/2085578006 >> 172.21.15.16:0/2085578006 conn(0x7f7a5006ee40 msgr2=0x7f7a500aff10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:05:27.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:27.091+0000 7f7a56a00640 1 -- 172.21.15.16:0/2085578006 shutdown_connections 2024-03-31T15:05:27.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:27.091+0000 7f7a56a00640 1 -- 172.21.15.16:0/2085578006 wait complete. 2024-03-31T15:05:27.114 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-31T15:05:27.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:27.259+0000 7f941f200640 1 Processor -- start 2024-03-31T15:05:27.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:27.259+0000 7f941f200640 1 -- start start 2024-03-31T15:05:27.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:27.259+0000 7f941f200640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f942015f9d0 0x7f942009aa60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:27.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:27.259+0000 7f941f200640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f94200b1420 0x7f942009afa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:27.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:27.259+0000 7f941f200640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f942015fdb0 con 0x7f94200d0da0 2024-03-31T15:05:27.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:27.259+0000 7f941f200640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f94200cdbd0 con 0x7f94200b1420 2024-03-31T15:05:27.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:27.259+0000 7f941f200640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f94200b4a40 con 0x7f942015f9d0 2024-03-31T15:05:27.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:27.259+0000 7f941e800640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f94200b1420 0x7f942009afa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:27.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:27.259+0000 7f941e800640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f94200b1420 0x7f942009afa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34070/0 (socket says 172.21.15.16:34070) 2024-03-31T15:05:27.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:27.259+0000 7f941e800640 1 -- 172.21.15.16:0/1956402357 learned_addr learned my addr 172.21.15.16:0/1956402357 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:05:27.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:27.259+0000 7f9417e00640 1 --1- 172.21.15.16:0/1956402357 >> v1:172.21.15.16:6789/0 conn(0x7f94200d0da0 0x7f94200b1970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:35498/0 (socket says 172.21.15.16:35498) 2024-03-31T15:05:27.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:27.259+0000 7f941de00640 1 --2- 172.21.15.16:0/1956402357 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f942015f9d0 0x7f942009aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:27.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:27.259+0000 7f941e800640 1 -- 172.21.15.16:0/1956402357 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f942015f9d0 msgr2=0x7f942009aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:05:27.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:27.259+0000 7f941e800640 1 --2- 172.21.15.16:0/1956402357 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f942015f9d0 0x7f942009aa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:27.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:27.259+0000 7f941e800640 1 -- 172.21.15.16:0/1956402357 >> v1:172.21.15.16:6789/0 conn(0x7f94200d0da0 legacy=0x7f94200b1970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:27.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:27.259+0000 7f941e800640 1 -- 172.21.15.16:0/1956402357 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94200cffe0 con 0x7f94200b1420 2024-03-31T15:05:27.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:27.259+0000 7f941d400640 1 -- 172.21.15.16:0/1956402357 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 338494941 0 0) 0x7f942015fdb0 con 0x7f94200d0da0 2024-03-31T15:05:27.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:27.263+0000 7f941e800640 1 --2- 172.21.15.16:0/1956402357 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f94200b1420 0x7f942009afa0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f9418009e80 tx=0x7f941801c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f479e81c8d077aa5 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:05:27.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:27.263+0000 7f941d400640 1 -- 172.21.15.16:0/1956402357 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94180a5830 con 0x7f94200b1420 2024-03-31T15:05:27.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:27.263+0000 7f941d400640 1 -- 172.21.15.16:0/1956402357 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f94180a59d0 con 0x7f94200b1420 2024-03-31T15:05:27.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:27.263+0000 7f941d400640 1 -- 172.21.15.16:0/1956402357 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94180a5da0 con 0x7f94200b1420 2024-03-31T15:05:27.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:27.263+0000 7f941f200640 1 -- 172.21.15.16:0/1956402357 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f94200b1420 msgr2=0x7f942009afa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:27.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:27.263+0000 7f941f200640 1 --2- 172.21.15.16:0/1956402357 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f94200b1420 0x7f942009afa0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f9418009e80 tx=0x7f941801c950 comp rx=0 tx=0).stop 2024-03-31T15:05:27.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:27.263+0000 7f941f200640 1 -- 172.21.15.16:0/1956402357 shutdown_connections 2024-03-31T15:05:27.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:27.263+0000 7f941f200640 1 --2- 172.21.15.16:0/1956402357 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f94200b1420 0x7f942009afa0 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:27.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:27.263+0000 7f941f200640 1 --2- 172.21.15.16:0/1956402357 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f942015f9d0 0x7f942009aa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:27.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:27.263+0000 7f941f200640 1 -- 172.21.15.16:0/1956402357 >> 172.21.15.16:0/1956402357 conn(0x7f94200b3b40 msgr2=0x7f94200b3f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:05:27.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:27.263+0000 7f941f200640 1 -- 172.21.15.16:0/1956402357 shutdown_connections 2024-03-31T15:05:27.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:27.263+0000 7f941f200640 1 -- 172.21.15.16:0/1956402357 wait complete. 2024-03-31T15:05:27.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:27.267+0000 7f941f200640 1 Processor -- start 2024-03-31T15:05:27.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:27.267+0000 7f941f200640 1 -- start start 2024-03-31T15:05:27.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:27.267+0000 7f941f200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f94200b1420 0x7f9420155dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:27.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:27.267+0000 7f941f200640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f94200d0da0 0x7f9420155340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:27.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:27.267+0000 7f941f200640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f942015f9d0 0x7f9420157120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:27.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:27.267+0000 7f941f200640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f94200b47e0 con 0x7f94200b1420 2024-03-31T15:05:27.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:27.267+0000 7f941f200640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f94200b4660 con 0x7f94200d0da0 2024-03-31T15:05:27.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:27.267+0000 7f941f200640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f9420169080 con 0x7f942015f9d0 2024-03-31T15:05:27.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:27.267+0000 7f941de00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f94200b1420 0x7f9420155dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:27.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:27.267+0000 7f941de00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f94200b1420 0x7f9420155dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34086/0 (socket says 172.21.15.16:34086) 2024-03-31T15:05:27.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:27.267+0000 7f941de00640 1 -- 172.21.15.16:0/80082794 learned_addr learned my addr 172.21.15.16:0/80082794 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:05:27.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:27.267+0000 7f9417e00640 1 --2- 172.21.15.16:0/80082794 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f94200d0da0 0x7f9420155340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:27.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:27.267+0000 7f941e800640 1 --2- 172.21.15.16:0/80082794 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f942015f9d0 0x7f9420157120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:27.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:27.267+0000 7f941de00640 1 -- 172.21.15.16:0/80082794 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f942015f9d0 msgr2=0x7f9420157120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:27.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:27.267+0000 7f941de00640 1 --2- 172.21.15.16:0/80082794 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f942015f9d0 0x7f9420157120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:27.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:27.267+0000 7f941de00640 1 -- 172.21.15.16:0/80082794 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f94200d0da0 msgr2=0x7f9420155340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:27.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:27.267+0000 7f941de00640 1 --2- 172.21.15.16:0/80082794 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f94200d0da0 0x7f9420155340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:27.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:27.267+0000 7f941de00640 1 -- 172.21.15.16:0/80082794 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94201566a0 con 0x7f94200b1420 2024-03-31T15:05:27.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:27.267+0000 7f941e800640 1 --2- 172.21.15.16:0/80082794 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f942015f9d0 0x7f9420157120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:05:27.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:27.267+0000 7f9417e00640 1 --2- 172.21.15.16:0/80082794 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f94200d0da0 0x7f9420155340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:05:27.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:27.267+0000 7f941de00640 1 --2- 172.21.15.16:0/80082794 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f94200b1420 0x7f9420155dc0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f941000e870 tx=0x7f941000ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:05:27.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:27.267+0000 7f9416000640 1 -- 172.21.15.16:0/80082794 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9410017070 con 0x7f94200b1420 2024-03-31T15:05:27.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:27.267+0000 7f9416000640 1 -- 172.21.15.16:0/80082794 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f941000c940 con 0x7f94200b1420 2024-03-31T15:05:27.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:27.267+0000 7f9416000640 1 -- 172.21.15.16:0/80082794 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f941000ccf0 con 0x7f94200b1420 2024-03-31T15:05:27.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:27.267+0000 7f941f200640 1 -- 172.21.15.16:0/80082794 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9420156980 con 0x7f94200b1420 2024-03-31T15:05:27.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:27.267+0000 7f941f200640 1 -- 172.21.15.16:0/80082794 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f94200c6cb0 con 0x7f94200b1420 2024-03-31T15:05:27.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:27.271+0000 7f941f200640 1 -- 172.21.15.16:0/80082794 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f942004f8c0 con 0x7f94200b1420 2024-03-31T15:05:27.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:27.271+0000 7f9416000640 1 -- 172.21.15.16:0/80082794 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f941001a070 con 0x7f94200b1420 2024-03-31T15:05:27.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:27.271+0000 7f9416000640 1 --2- 172.21.15.16:0/80082794 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f93fc068400 0x7f93fc06a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:27.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:27.271+0000 7f9416000640 1 -- 172.21.15.16:0/80082794 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 5709+0+0 (secure 0 0 0) 0x7f9410086810 con 0x7f94200b1420 2024-03-31T15:05:27.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:27.271+0000 7f9417e00640 1 --2- 172.21.15.16:0/80082794 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f93fc068400 0x7f93fc06a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:27.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:27.271+0000 7f9417e00640 1 --2- 172.21.15.16:0/80082794 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f93fc068400 0x7f93fc06a8c0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f94200a0170 tx=0x7f9408007cb0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:05:27.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:27.275+0000 7f9416000640 1 -- 172.21.15.16:0/80082794 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f9410056e40 con 0x7f94200b1420 2024-03-31T15:05:27.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:27.439+0000 7f941f200640 1 -- 172.21.15.16:0/80082794 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f9420068850 con 0x7f94200b1420 2024-03-31T15:05:27.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:27.443+0000 7f9416000640 1 -- 172.21.15.16:0/80082794 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v44) ==== 74+0+16554 (secure 0 0 0) 0x7f9410056740 con 0x7f94200b1420 2024-03-31T15:05:27.437 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:05:27.437 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":44,"fsid":"286eb5da-fb47-4550-b290-d9519c13b9a5","created":"2024-03-31T15:03:11.438952+0000","modified":"2024-03-31T15:05:26.461531+0000","last_up_change":"2024-03-31T15:03:22.885535+0000","last_in_change":"2024-03-31T15:03:13.941667+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-31T15:03:22.550787+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2024-03-31T15:05:24.856239+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"44","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2024-03-31T15:05:25.417826+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"44","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"92a66450-ef19-40d8-ac3d-d760b23b0bda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6801","nonce":932712244}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6803","nonce":932712244}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6807","nonce":932712244}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6805","nonce":932712244}]},"public_addr":"172.21.15.16:6801/932712244","cluster_addr":"172.21.15.16:6803/932712244","heartbeat_back_addr":"172.21.15.16:6807/932712244","heartbeat_front_addr":"172.21.15.16:6805/932712244","state":["exists","up"]},{"osd":1,"uuid":"da9d1f2b-1968-4959-b843-faa231988e92","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6810","nonce":1398737533}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6811","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6813","nonce":1398737533}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6821","nonce":1398737533}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6815","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6816","nonce":1398737533}]},"public_addr":"172.21.15.16:6810/1398737533","cluster_addr":"172.21.15.16:6813/1398737533","heartbeat_back_addr":"172.21.15.16:6821/1398737533","heartbeat_front_addr":"172.21.15.16:6816/1398737533","state":["exists","up"]},{"osd":2,"uuid":"01a73662-9a21-4b1f-bcbb-42b9636a4638","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6819","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6822","nonce":3018526646}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6825","nonce":3018526646}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6831","nonce":3018526646}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6827","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6829","nonce":3018526646}]},"public_addr":"172.21.15.16:6822/3018526646","cluster_addr":"172.21.15.16:6825/3018526646","heartbeat_back_addr":"172.21.15.16:6831/3018526646","heartbeat_front_addr":"172.21.15.16:6829/3018526646","state":["exists","up"]},{"osd":3,"uuid":"c7771fd3-e18e-4500-ac38-23cf10ebcd75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6809","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6812","nonce":3360352974}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6817","nonce":3360352974}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6828","nonce":3360352974}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6823","nonce":3360352974}]},"public_addr":"172.21.15.16:6812/3360352974","cluster_addr":"172.21.15.16:6817/3360352974","heartbeat_back_addr":"172.21.15.16:6828/3360352974","heartbeat_front_addr":"172.21.15.16:6823/3360352974","state":["exists","up"]},{"osd":4,"uuid":"88260730-1482-4fbd-a40f-cb1124e9df4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6801","nonce":1856844856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6803","nonce":1856844856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6807","nonce":1856844856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6805","nonce":1856844856}]},"public_addr":"172.21.15.89:6801/1856844856","cluster_addr":"172.21.15.89:6803/1856844856","heartbeat_back_addr":"172.21.15.89:6807/1856844856","heartbeat_front_addr":"172.21.15.89:6805/1856844856","state":["exists","up"]},{"osd":5,"uuid":"4cb2c00c-fae0-4dd8-9d27-ec4f0093ccf7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6814","nonce":50395527}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6817","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6820","nonce":50395527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6831","nonce":50395527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6823","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6826","nonce":50395527}]},"public_addr":"172.21.15.89:6814/50395527","cluster_addr":"172.21.15.89:6820/50395527","heartbeat_back_addr":"172.21.15.89:6831/50395527","heartbeat_front_addr":"172.21.15.89:6826/50395527","state":["exists","up"]},{"osd":6,"uuid":"81ce0c19-2554-4cae-acbe-66a05e9e9b9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6809","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6812","nonce":3470471752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6815","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6818","nonce":3470471752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6827","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6829","nonce":3470471752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6821","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6824","nonce":3470471752}]},"public_addr":"172.21.15.89:6812/3470471752","cluster_addr":"172.21.15.89:6818/3470471752","heartbeat_back_addr":"172.21.15.89:6829/3470471752","heartbeat_front_addr":"172.21.15.89:6824/3470471752","state":["exists","up"]},{"osd":7,"uuid":"85e4313b-d37d-442e-8042-153bf20a847d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6811","nonce":1796778630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6813","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6816","nonce":1796778630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6825","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6828","nonce":1796778630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6819","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6822","nonce":1796778630}]},"public_addr":"172.21.15.89:6811/1796778630","cluster_addr":"172.21.15.89:6816/1796778630","heartbeat_back_addr":"172.21.15.89:6828/1796778630","heartbeat_front_addr":"172.21.15.89:6822/1796778630","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.439088+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.529970+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.680512+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:21.044161+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.582111+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.536558+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.409522+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.590229+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-31T15:05:27.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:27.443+0000 7f941f200640 1 -- 172.21.15.16:0/80082794 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f93fc068400 msgr2=0x7f93fc06a8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:27.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:27.443+0000 7f941f200640 1 --2- 172.21.15.16:0/80082794 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f93fc068400 0x7f93fc06a8c0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f94200a0170 tx=0x7f9408007cb0 comp rx=0 tx=0).stop 2024-03-31T15:05:27.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:27.443+0000 7f941f200640 1 -- 172.21.15.16:0/80082794 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f94200b1420 msgr2=0x7f9420155dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:27.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:27.443+0000 7f941f200640 1 --2- 172.21.15.16:0/80082794 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f94200b1420 0x7f9420155dc0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f941000e870 tx=0x7f941000ec70 comp rx=0 tx=0).stop 2024-03-31T15:05:27.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:27.443+0000 7f941f200640 1 -- 172.21.15.16:0/80082794 shutdown_connections 2024-03-31T15:05:27.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:27.443+0000 7f941f200640 1 --2- 172.21.15.16:0/80082794 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f93fc068400 0x7f93fc06a8c0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:27.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:27.443+0000 7f941f200640 1 --2- 172.21.15.16:0/80082794 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f942015f9d0 0x7f9420157120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:27.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:27.443+0000 7f941f200640 1 --2- 172.21.15.16:0/80082794 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f94200d0da0 0x7f9420155340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:27.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:27.443+0000 7f941f200640 1 --2- 172.21.15.16:0/80082794 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f94200b1420 0x7f9420155dc0 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:27.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:27.443+0000 7f941f200640 1 -- 172.21.15.16:0/80082794 >> 172.21.15.16:0/80082794 conn(0x7f94200b3b40 msgr2=0x7f942006f620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:05:27.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:27.443+0000 7f941f200640 1 -- 172.21.15.16:0/80082794 shutdown_connections 2024-03-31T15:05:27.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:27.443+0000 7f941f200640 1 -- 172.21.15.16:0/80082794 wait complete. 2024-03-31T15:05:27.458 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-31T15:05:27.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:27.603+0000 7f2b78c00640 1 Processor -- start 2024-03-31T15:05:27.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:27.603+0000 7f2b78c00640 1 -- start start 2024-03-31T15:05:27.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:27.607+0000 7f2b78c00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f2b7406dbc0 0x7f2b7415f650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:27.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:27.607+0000 7f2b78c00640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2b740cbe90 con 0x7f2b74070030 2024-03-31T15:05:27.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:27.607+0000 7f2b78c00640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2b740cc170 con 0x7f2b740cb440 2024-03-31T15:05:27.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:27.607+0000 7f2b78c00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f2b7415fb90 con 0x7f2b7406dbc0 2024-03-31T15:05:27.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:27.607+0000 7f2b72800640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f2b74070030 0x7f2b74162180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:35502/0 (socket says 172.21.15.16:35502) 2024-03-31T15:05:27.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:27.607+0000 7f2b72800640 1 -- 172.21.15.16:0/1923146822 learned_addr learned my addr 172.21.15.16:0/1923146822 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:05:27.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:27.607+0000 7f2b71e00640 1 --2- 172.21.15.16:0/1923146822 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f2b7406dbc0 0x7f2b7415f650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:27.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:27.607+0000 7f2b71400640 1 -- 172.21.15.16:0/1923146822 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 205504505 0 0) 0x7f2b740cbe90 con 0x7f2b74070030 2024-03-31T15:05:27.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:27.607+0000 7f2b71400640 1 -- 172.21.15.16:0/1923146822 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2b58003530 con 0x7f2b74070030 2024-03-31T15:05:27.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:27.607+0000 7f2b71400640 1 -- 172.21.15.16:0/1923146822 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 967910988 0 0) 0x7f2b58003530 con 0x7f2b74070030 2024-03-31T15:05:27.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:27.607+0000 7f2b71400640 1 -- 172.21.15.16:0/1923146822 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f2b7406dbc0 msgr2=0x7f2b7415f650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:05:27.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:27.607+0000 7f2b71400640 1 --2- 172.21.15.16:0/1923146822 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f2b7406dbc0 0x7f2b7415f650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:27.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:27.607+0000 7f2b71400640 1 -- 172.21.15.16:0/1923146822 >> v1:172.21.15.89:6789/0 conn(0x7f2b740cb440 legacy=0x7f2b7406f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:27.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:27.607+0000 7f2b71400640 1 -- 172.21.15.16:0/1923146822 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b7415ee50 con 0x7f2b74070030 2024-03-31T15:05:27.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:27.607+0000 7f2b71400640 1 -- 172.21.15.16:0/1923146822 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 214807623 0 0) 0x7f2b6c001a00 con 0x7f2b74070030 2024-03-31T15:05:27.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:27.607+0000 7f2b71400640 1 -- 172.21.15.16:0/1923146822 >> v1:172.21.15.16:6789/0 conn(0x7f2b74070030 legacy=0x7f2b74162180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:27.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:27.607+0000 7f2b71400640 1 --2- 172.21.15.16:0/1923146822 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2b580039f0 0x7f2b58003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:27.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:27.607+0000 7f2b71400640 1 --2- 172.21.15.16:0/1923146822 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f2b7406dbc0 0x7f2b58004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:27.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:27.607+0000 7f2b71400640 1 --2- 172.21.15.16:0/1923146822 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2b58004d50 0x7f2b580051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:27.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:27.607+0000 7f2b71400640 1 -- 172.21.15.16:0/1923146822 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f2b7415fb90 con 0x7f2b58004d50 2024-03-31T15:05:27.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:27.607+0000 7f2b71400640 1 -- 172.21.15.16:0/1923146822 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f2b5800bb00 con 0x7f2b580039f0 2024-03-31T15:05:27.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:27.607+0000 7f2b71400640 1 -- 172.21.15.16:0/1923146822 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f2b5800bd00 con 0x7f2b7406dbc0 2024-03-31T15:05:27.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:27.607+0000 7f2b72800640 1 --2- 172.21.15.16:0/1923146822 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2b58004d50 0x7f2b580051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:27.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:27.607+0000 7f2b71e00640 1 --2- 172.21.15.16:0/1923146822 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2b580039f0 0x7f2b58003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:27.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:27.607+0000 7f2b6be00640 1 --2- 172.21.15.16:0/1923146822 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f2b7406dbc0 0x7f2b58004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:27.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:27.607+0000 7f2b72800640 1 -- 172.21.15.16:0/1923146822 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f2b7406dbc0 msgr2=0x7f2b58004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:05:27.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:27.607+0000 7f2b72800640 1 --2- 172.21.15.16:0/1923146822 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f2b7406dbc0 0x7f2b58004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:27.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:27.607+0000 7f2b72800640 1 -- 172.21.15.16:0/1923146822 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2b580039f0 msgr2=0x7f2b58003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:05:27.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:27.607+0000 7f2b72800640 1 --2- 172.21.15.16:0/1923146822 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2b580039f0 0x7f2b58003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:27.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:27.607+0000 7f2b72800640 1 -- 172.21.15.16:0/1923146822 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b58003530 con 0x7f2b58004d50 2024-03-31T15:05:27.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:27.611+0000 7f2b72800640 1 --2- 172.21.15.16:0/1923146822 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2b58004d50 0x7f2b580051d0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f2b6c00a860 tx=0x7f2b6c01fae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c3df74bf77591743 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:05:27.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:27.611+0000 7f2b71400640 1 -- 172.21.15.16:0/1923146822 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b6c00a480 con 0x7f2b58004d50 2024-03-31T15:05:27.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:27.611+0000 7f2b71400640 1 -- 172.21.15.16:0/1923146822 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2b6c0aa710 con 0x7f2b58004d50 2024-03-31T15:05:27.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:27.611+0000 7f2b71400640 1 -- 172.21.15.16:0/1923146822 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b6c0aab20 con 0x7f2b58004d50 2024-03-31T15:05:27.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:27.611+0000 7f2b78c00640 1 -- 172.21.15.16:0/1923146822 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2b58004d50 msgr2=0x7f2b580051d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:27.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:27.611+0000 7f2b78c00640 1 --2- 172.21.15.16:0/1923146822 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2b58004d50 0x7f2b580051d0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f2b6c00a860 tx=0x7f2b6c01fae0 comp rx=0 tx=0).stop 2024-03-31T15:05:27.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:27.611+0000 7f2b72800640 1 -- 172.21.15.16:0/1923146822 reap_dead start 2024-03-31T15:05:27.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:27.611+0000 7f2b78c00640 1 -- 172.21.15.16:0/1923146822 shutdown_connections 2024-03-31T15:05:27.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:27.611+0000 7f2b78c00640 1 -- 172.21.15.16:0/1923146822 >> 172.21.15.16:0/1923146822 conn(0x7f2b74067330 msgr2=0x7f2b74067740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:05:27.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:27.611+0000 7f2b78c00640 1 -- 172.21.15.16:0/1923146822 shutdown_connections 2024-03-31T15:05:27.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:27.611+0000 7f2b78c00640 1 -- 172.21.15.16:0/1923146822 wait complete. 2024-03-31T15:05:27.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:27.615+0000 7f2b78c00640 1 Processor -- start 2024-03-31T15:05:27.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:27.615+0000 7f2b78c00640 1 -- start start 2024-03-31T15:05:27.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:27.615+0000 7f2b78c00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2b74147a30 0x7f2b7415e050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:27.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:27.615+0000 7f2b78c00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f2b7415afa0 0x7f2b741596b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:27.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:27.615+0000 7f2b78c00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2b7415c060 0x7f2b7415c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:27.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:27.615+0000 7f2b78c00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f2b7406dfa0 con 0x7f2b74147a30 2024-03-31T15:05:27.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:27.615+0000 7f2b78c00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f2b7415f1c0 con 0x7f2b7415c060 2024-03-31T15:05:27.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:27.615+0000 7f2b78c00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f2b7406c950 con 0x7f2b7415afa0 2024-03-31T15:05:27.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:27.615+0000 7f2b71e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2b74147a30 0x7f2b7415e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:27.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:27.615+0000 7f2b71e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2b74147a30 0x7f2b7415e050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34090/0 (socket says 172.21.15.16:34090) 2024-03-31T15:05:27.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:27.615+0000 7f2b71e00640 1 -- 172.21.15.16:0/4043348722 learned_addr learned my addr 172.21.15.16:0/4043348722 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:05:27.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:27.615+0000 7f2b6be00640 1 --2- 172.21.15.16:0/4043348722 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f2b7415afa0 0x7f2b741596b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:27.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:27.615+0000 7f2b72800640 1 --2- 172.21.15.16:0/4043348722 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2b7415c060 0x7f2b7415c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:27.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:27.615+0000 7f2b71e00640 1 -- 172.21.15.16:0/4043348722 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f2b7415afa0 msgr2=0x7f2b741596b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:27.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:27.615+0000 7f2b71e00640 1 --2- 172.21.15.16:0/4043348722 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f2b7415afa0 0x7f2b741596b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:27.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:27.615+0000 7f2b71e00640 1 -- 172.21.15.16:0/4043348722 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2b7415c060 msgr2=0x7f2b7415c510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:27.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:27.615+0000 7f2b71e00640 1 --2- 172.21.15.16:0/4043348722 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2b7415c060 0x7f2b7415c510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:27.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:27.615+0000 7f2b71e00640 1 -- 172.21.15.16:0/4043348722 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b74159f00 con 0x7f2b74147a30 2024-03-31T15:05:27.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:27.615+0000 7f2b72800640 1 --2- 172.21.15.16:0/4043348722 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2b7415c060 0x7f2b7415c510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:05:27.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:27.615+0000 7f2b71e00640 1 --2- 172.21.15.16:0/4043348722 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2b74147a30 0x7f2b7415e050 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f2b640119d0 tx=0x7f2b64011e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:05:27.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:27.615+0000 7f2b6a000640 1 -- 172.21.15.16:0/4043348722 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b6400fb50 con 0x7f2b74147a30 2024-03-31T15:05:27.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:27.615+0000 7f2b6a000640 1 -- 172.21.15.16:0/4043348722 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2b6400fcf0 con 0x7f2b74147a30 2024-03-31T15:05:27.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:27.615+0000 7f2b78c00640 1 -- 172.21.15.16:0/4043348722 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2b7415a1e0 con 0x7f2b74147a30 2024-03-31T15:05:27.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:27.615+0000 7f2b6a000640 1 -- 172.21.15.16:0/4043348722 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b6400fea0 con 0x7f2b74147a30 2024-03-31T15:05:27.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:27.615+0000 7f2b78c00640 1 -- 172.21.15.16:0/4043348722 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2b7415a7a0 con 0x7f2b74147a30 2024-03-31T15:05:27.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:27.615+0000 7f2b78c00640 1 -- 172.21.15.16:0/4043348722 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2b74069950 con 0x7f2b74147a30 2024-03-31T15:05:27.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:27.619+0000 7f2b6a000640 1 -- 172.21.15.16:0/4043348722 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f2b64012040 con 0x7f2b74147a30 2024-03-31T15:05:27.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:27.619+0000 7f2b6a000640 1 --2- 172.21.15.16:0/4043348722 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f2b500683b0 0x7f2b5006a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:27.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:27.619+0000 7f2b6a000640 1 -- 172.21.15.16:0/4043348722 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 5709+0+0 (secure 0 0 0) 0x7f2b64087030 con 0x7f2b74147a30 2024-03-31T15:05:27.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:27.619+0000 7f2b6be00640 1 --2- 172.21.15.16:0/4043348722 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f2b500683b0 0x7f2b5006a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:27.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:27.619+0000 7f2b6be00640 1 --2- 172.21.15.16:0/4043348722 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f2b500683b0 0x7f2b5006a870 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f2b7415f8d0 tx=0x7f2b5c009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:05:27.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:27.623+0000 7f2b6a000640 1 -- 172.21.15.16:0/4043348722 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f2b64019030 con 0x7f2b74147a30 2024-03-31T15:05:27.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:27.787+0000 7f2b78c00640 1 -- 172.21.15.16:0/4043348722 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f2b740b83a0 con 0x7f2b74147a30 2024-03-31T15:05:27.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:27.791+0000 7f2b6a000640 1 -- 172.21.15.16:0/4043348722 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v44) ==== 74+0+16554 (secure 0 0 0) 0x7f2b640576b0 con 0x7f2b74147a30 2024-03-31T15:05:27.785 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:05:27.786 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":44,"fsid":"286eb5da-fb47-4550-b290-d9519c13b9a5","created":"2024-03-31T15:03:11.438952+0000","modified":"2024-03-31T15:05:26.461531+0000","last_up_change":"2024-03-31T15:03:22.885535+0000","last_in_change":"2024-03-31T15:03:13.941667+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-31T15:03:22.550787+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2024-03-31T15:05:24.856239+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"44","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2024-03-31T15:05:25.417826+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"44","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"92a66450-ef19-40d8-ac3d-d760b23b0bda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6801","nonce":932712244}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6803","nonce":932712244}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6807","nonce":932712244}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6805","nonce":932712244}]},"public_addr":"172.21.15.16:6801/932712244","cluster_addr":"172.21.15.16:6803/932712244","heartbeat_back_addr":"172.21.15.16:6807/932712244","heartbeat_front_addr":"172.21.15.16:6805/932712244","state":["exists","up"]},{"osd":1,"uuid":"da9d1f2b-1968-4959-b843-faa231988e92","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6810","nonce":1398737533}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6811","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6813","nonce":1398737533}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6821","nonce":1398737533}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6815","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6816","nonce":1398737533}]},"public_addr":"172.21.15.16:6810/1398737533","cluster_addr":"172.21.15.16:6813/1398737533","heartbeat_back_addr":"172.21.15.16:6821/1398737533","heartbeat_front_addr":"172.21.15.16:6816/1398737533","state":["exists","up"]},{"osd":2,"uuid":"01a73662-9a21-4b1f-bcbb-42b9636a4638","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6819","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6822","nonce":3018526646}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6825","nonce":3018526646}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6831","nonce":3018526646}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6827","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6829","nonce":3018526646}]},"public_addr":"172.21.15.16:6822/3018526646","cluster_addr":"172.21.15.16:6825/3018526646","heartbeat_back_addr":"172.21.15.16:6831/3018526646","heartbeat_front_addr":"172.21.15.16:6829/3018526646","state":["exists","up"]},{"osd":3,"uuid":"c7771fd3-e18e-4500-ac38-23cf10ebcd75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6809","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6812","nonce":3360352974}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6817","nonce":3360352974}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6828","nonce":3360352974}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6823","nonce":3360352974}]},"public_addr":"172.21.15.16:6812/3360352974","cluster_addr":"172.21.15.16:6817/3360352974","heartbeat_back_addr":"172.21.15.16:6828/3360352974","heartbeat_front_addr":"172.21.15.16:6823/3360352974","state":["exists","up"]},{"osd":4,"uuid":"88260730-1482-4fbd-a40f-cb1124e9df4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6801","nonce":1856844856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6803","nonce":1856844856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6807","nonce":1856844856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6805","nonce":1856844856}]},"public_addr":"172.21.15.89:6801/1856844856","cluster_addr":"172.21.15.89:6803/1856844856","heartbeat_back_addr":"172.21.15.89:6807/1856844856","heartbeat_front_addr":"172.21.15.89:6805/1856844856","state":["exists","up"]},{"osd":5,"uuid":"4cb2c00c-fae0-4dd8-9d27-ec4f0093ccf7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6814","nonce":50395527}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6817","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6820","nonce":50395527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6831","nonce":50395527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6823","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6826","nonce":50395527}]},"public_addr":"172.21.15.89:6814/50395527","cluster_addr":"172.21.15.89:6820/50395527","heartbeat_back_addr":"172.21.15.89:6831/50395527","heartbeat_front_addr":"172.21.15.89:6826/50395527","state":["exists","up"]},{"osd":6,"uuid":"81ce0c19-2554-4cae-acbe-66a05e9e9b9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6809","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6812","nonce":3470471752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6815","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6818","nonce":3470471752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6827","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6829","nonce":3470471752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6821","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6824","nonce":3470471752}]},"public_addr":"172.21.15.89:6812/3470471752","cluster_addr":"172.21.15.89:6818/3470471752","heartbeat_back_addr":"172.21.15.89:6829/3470471752","heartbeat_front_addr":"172.21.15.89:6824/3470471752","state":["exists","up"]},{"osd":7,"uuid":"85e4313b-d37d-442e-8042-153bf20a847d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6811","nonce":1796778630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6813","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6816","nonce":1796778630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6825","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6828","nonce":1796778630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6819","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6822","nonce":1796778630}]},"public_addr":"172.21.15.89:6811/1796778630","cluster_addr":"172.21.15.89:6816/1796778630","heartbeat_back_addr":"172.21.15.89:6828/1796778630","heartbeat_front_addr":"172.21.15.89:6822/1796778630","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.439088+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.529970+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.680512+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:21.044161+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.582111+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.536558+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.409522+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.590229+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-31T15:05:27.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:27.791+0000 7f2b78c00640 1 -- 172.21.15.16:0/4043348722 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f2b500683b0 msgr2=0x7f2b5006a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:27.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:27.791+0000 7f2b78c00640 1 --2- 172.21.15.16:0/4043348722 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f2b500683b0 0x7f2b5006a870 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f2b7415f8d0 tx=0x7f2b5c009040 comp rx=0 tx=0).stop 2024-03-31T15:05:27.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:27.791+0000 7f2b78c00640 1 -- 172.21.15.16:0/4043348722 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2b74147a30 msgr2=0x7f2b7415e050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:27.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:27.791+0000 7f2b78c00640 1 --2- 172.21.15.16:0/4043348722 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2b74147a30 0x7f2b7415e050 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f2b640119d0 tx=0x7f2b64011e90 comp rx=0 tx=0).stop 2024-03-31T15:05:27.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:27.791+0000 7f2b78c00640 1 -- 172.21.15.16:0/4043348722 shutdown_connections 2024-03-31T15:05:27.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:27.791+0000 7f2b78c00640 1 --2- 172.21.15.16:0/4043348722 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f2b500683b0 0x7f2b5006a870 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:27.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:27.791+0000 7f2b78c00640 1 --2- 172.21.15.16:0/4043348722 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2b7415c060 0x7f2b7415c510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:27.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:27.791+0000 7f2b78c00640 1 --2- 172.21.15.16:0/4043348722 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f2b7415afa0 0x7f2b741596b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:27.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:27.791+0000 7f2b78c00640 1 --2- 172.21.15.16:0/4043348722 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2b74147a30 0x7f2b7415e050 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:27.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:27.791+0000 7f2b78c00640 1 -- 172.21.15.16:0/4043348722 >> 172.21.15.16:0/4043348722 conn(0x7f2b74067330 msgr2=0x7f2b74160480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:05:27.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:27.791+0000 7f2b78c00640 1 -- 172.21.15.16:0/4043348722 shutdown_connections 2024-03-31T15:05:27.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:27.791+0000 7f2b78c00640 1 -- 172.21.15.16:0/4043348722 wait complete. 2024-03-31T15:05:27.807 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-31T15:05:27.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:27.955+0000 7f9c42200640 1 Processor -- start 2024-03-31T15:05:27.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:27.955+0000 7f9c42200640 1 -- start start 2024-03-31T15:05:27.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:27.955+0000 7f9c42200640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f9c3c1567e0 0x7f9c3c072c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:27.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:27.955+0000 7f9c42200640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f9c3c0731c0 0x7f9c3c06d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:27.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:27.955+0000 7f9c42200640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9c3c15c7e0 con 0x7f9c3c06da10 2024-03-31T15:05:27.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:27.955+0000 7f9c42200640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f9c3c06de20 con 0x7f9c3c1567e0 2024-03-31T15:05:27.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:27.955+0000 7f9c42200640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f9c3c0736b0 con 0x7f9c3c0731c0 2024-03-31T15:05:27.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:27.955+0000 7f9c3b400640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f9c3c1567e0 0x7f9c3c072c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:27.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:27.959+0000 7f9c3b400640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f9c3c1567e0 0x7f9c3c072c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34094/0 (socket says 172.21.15.16:34094) 2024-03-31T15:05:27.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:27.959+0000 7f9c3b400640 1 -- 172.21.15.16:0/2085941273 learned_addr learned my addr 172.21.15.16:0/2085941273 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:05:27.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:27.959+0000 7f9c32a00640 1 --2- 172.21.15.16:0/2085941273 >> v2:172.21.15.89:3300/0 conn(0x7f9c3c0731c0 0x7f9c3c06d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:27.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:27.959+0000 7f9c3b400640 1 -- 172.21.15.16:0/2085941273 >> v2:172.21.15.89:3300/0 conn(0x7f9c3c0731c0 msgr2=0x7f9c3c06d4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:05:27.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:27.959+0000 7f9c3b400640 1 --2- 172.21.15.16:0/2085941273 >> v2:172.21.15.89:3300/0 conn(0x7f9c3c0731c0 0x7f9c3c06d4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:27.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:27.959+0000 7f9c3b400640 1 -- 172.21.15.16:0/2085941273 >> v1:172.21.15.89:6789/0 conn(0x7f9c3c06da10 legacy=0x7f9c3c15fdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:05:27.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:27.959+0000 7f9c3b400640 1 -- 172.21.15.16:0/2085941273 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9c3c15caa0 con 0x7f9c3c1567e0 2024-03-31T15:05:27.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:27.959+0000 7f9c32a00640 1 --2- 172.21.15.16:0/2085941273 >> v2:172.21.15.89:3300/0 conn(0x7f9c3c0731c0 0x7f9c3c06d4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:05:27.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:27.959+0000 7f9c3b400640 1 --2- 172.21.15.16:0/2085941273 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9c3c1567e0 0x7f9c3c072c80 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f9c3c06c2e0 tx=0x7f9c2c01c4e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=76cb4c18deb68c53 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:05:27.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:27.959+0000 7f9c3aa00640 1 -- 172.21.15.16:0/2085941273 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c2c0b0070 con 0x7f9c3c1567e0 2024-03-31T15:05:27.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:27.959+0000 7f9c3aa00640 1 -- 172.21.15.16:0/2085941273 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9c2c0ab600 con 0x7f9c3c1567e0 2024-03-31T15:05:27.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:27.959+0000 7f9c3aa00640 1 -- 172.21.15.16:0/2085941273 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c2c0ab9d0 con 0x7f9c3c1567e0 2024-03-31T15:05:27.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:27.963+0000 7f9c42200640 1 -- 172.21.15.16:0/2085941273 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9c3c1567e0 msgr2=0x7f9c3c072c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:27.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:27.963+0000 7f9c42200640 1 --2- 172.21.15.16:0/2085941273 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9c3c1567e0 0x7f9c3c072c80 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f9c3c06c2e0 tx=0x7f9c2c01c4e0 comp rx=0 tx=0).stop 2024-03-31T15:05:27.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:27.963+0000 7f9c42200640 1 -- 172.21.15.16:0/2085941273 shutdown_connections 2024-03-31T15:05:27.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:27.963+0000 7f9c42200640 1 --2- 172.21.15.16:0/2085941273 >> v2:172.21.15.89:3300/0 conn(0x7f9c3c0731c0 0x7f9c3c06d4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:27.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:27.963+0000 7f9c42200640 1 --2- 172.21.15.16:0/2085941273 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9c3c1567e0 0x7f9c3c072c80 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:27.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:27.963+0000 7f9c42200640 1 -- 172.21.15.16:0/2085941273 >> 172.21.15.16:0/2085941273 conn(0x7f9c3c06ee40 msgr2=0x7f9c3c06b620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:05:27.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:27.963+0000 7f9c42200640 1 -- 172.21.15.16:0/2085941273 shutdown_connections 2024-03-31T15:05:27.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:27.963+0000 7f9c42200640 1 -- 172.21.15.16:0/2085941273 wait complete. 2024-03-31T15:05:27.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:27.963+0000 7f9c42200640 1 Processor -- start 2024-03-31T15:05:27.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:27.963+0000 7f9c42200640 1 -- start start 2024-03-31T15:05:27.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:27.963+0000 7f9c42200640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f9c3c06da10 0x7f9c3c147510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:27.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:27.963+0000 7f9c42200640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9c3c0731c0 0x7f9c3c0b49b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:27.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:27.963+0000 7f9c42200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9c3c1567e0 0x7f9c3c136550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:27.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:27.963+0000 7f9c42200640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f9c3c15d700 con 0x7f9c3c1567e0 2024-03-31T15:05:27.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:27.963+0000 7f9c42200640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f9c3c15d580 con 0x7f9c3c0731c0 2024-03-31T15:05:27.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:27.963+0000 7f9c42200640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f9c3c169170 con 0x7f9c3c06da10 2024-03-31T15:05:27.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:27.963+0000 7f9c3be00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9c3c1567e0 0x7f9c3c136550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:27.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:27.963+0000 7f9c3be00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9c3c1567e0 0x7f9c3c136550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34106/0 (socket says 172.21.15.16:34106) 2024-03-31T15:05:27.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:27.963+0000 7f9c3be00640 1 -- 172.21.15.16:0/351748103 learned_addr learned my addr 172.21.15.16:0/351748103 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:05:27.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:27.963+0000 7f9c32a00640 1 --2- 172.21.15.16:0/351748103 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9c3c0731c0 0x7f9c3c0b49b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:27.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:27.963+0000 7f9c3b400640 1 --2- 172.21.15.16:0/351748103 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f9c3c06da10 0x7f9c3c147510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:27.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:27.963+0000 7f9c3be00640 1 -- 172.21.15.16:0/351748103 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f9c3c06da10 msgr2=0x7f9c3c147510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:27.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:27.963+0000 7f9c3be00640 1 --2- 172.21.15.16:0/351748103 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f9c3c06da10 0x7f9c3c147510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:27.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:27.963+0000 7f9c3be00640 1 -- 172.21.15.16:0/351748103 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9c3c0731c0 msgr2=0x7f9c3c0b49b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:27.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:27.963+0000 7f9c3be00640 1 --2- 172.21.15.16:0/351748103 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9c3c0731c0 0x7f9c3c0b49b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:27.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:27.963+0000 7f9c3be00640 1 -- 172.21.15.16:0/351748103 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9c3c0b0a80 con 0x7f9c3c1567e0 2024-03-31T15:05:27.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:27.963+0000 7f9c3be00640 1 --2- 172.21.15.16:0/351748103 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9c3c1567e0 0x7f9c3c136550 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f9c3400c780 tx=0x7f9c3400cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:05:27.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:27.963+0000 7f9c33e00640 1 -- 172.21.15.16:0/351748103 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c34004b70 con 0x7f9c3c1567e0 2024-03-31T15:05:27.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:27.967+0000 7f9c33e00640 1 -- 172.21.15.16:0/351748103 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9c34004d10 con 0x7f9c3c1567e0 2024-03-31T15:05:27.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:27.967+0000 7f9c42200640 1 -- 172.21.15.16:0/351748103 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9c3c0b0d60 con 0x7f9c3c1567e0 2024-03-31T15:05:27.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:27.967+0000 7f9c33e00640 1 -- 172.21.15.16:0/351748103 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c3400aa90 con 0x7f9c3c1567e0 2024-03-31T15:05:27.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:27.967+0000 7f9c42200640 1 -- 172.21.15.16:0/351748103 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9c3c0b1320 con 0x7f9c3c1567e0 2024-03-31T15:05:27.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:27.967+0000 7f9c42200640 1 -- 172.21.15.16:0/351748103 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9c3c069950 con 0x7f9c3c1567e0 2024-03-31T15:05:27.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:27.967+0000 7f9c33e00640 1 -- 172.21.15.16:0/351748103 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f9c3400ac30 con 0x7f9c3c1567e0 2024-03-31T15:05:27.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:27.967+0000 7f9c33e00640 1 --2- 172.21.15.16:0/351748103 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f9c18068380 0x7f9c1806a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:27.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:27.967+0000 7f9c33e00640 1 -- 172.21.15.16:0/351748103 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 5709+0+0 (secure 0 0 0) 0x7f9c34086b90 con 0x7f9c3c1567e0 2024-03-31T15:05:27.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:27.967+0000 7f9c3b400640 1 --2- 172.21.15.16:0/351748103 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f9c18068380 0x7f9c1806a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:27.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:27.971+0000 7f9c3b400640 1 --2- 172.21.15.16:0/351748103 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f9c18068380 0x7f9c1806a840 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f9c2c0ad000 tx=0x7f9c2c0a6000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:05:27.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:27.971+0000 7f9c33e00640 1 -- 172.21.15.16:0/351748103 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f9c34020070 con 0x7f9c3c1567e0 2024-03-31T15:05:28.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:28.183+0000 7f9c42200640 1 -- 172.21.15.16:0/351748103 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f9c3c06e470 con 0x7f9c3c1567e0 2024-03-31T15:05:29.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:29.079+0000 7f9c33e00640 1 -- 172.21.15.16:0/351748103 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v24) ==== 118+0+0 (secure 0 0 0) 0x7f9c34056af0 con 0x7f9c3c1567e0 2024-03-31T15:05:29.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:29.079+0000 7f9c42200640 1 -- 172.21.15.16:0/351748103 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f9c18068380 msgr2=0x7f9c1806a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:29.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:29.079+0000 7f9c42200640 1 --2- 172.21.15.16:0/351748103 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f9c18068380 0x7f9c1806a840 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f9c2c0ad000 tx=0x7f9c2c0a6000 comp rx=0 tx=0).stop 2024-03-31T15:05:29.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:29.079+0000 7f9c42200640 1 -- 172.21.15.16:0/351748103 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9c3c1567e0 msgr2=0x7f9c3c136550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:29.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:29.079+0000 7f9c42200640 1 --2- 172.21.15.16:0/351748103 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9c3c1567e0 0x7f9c3c136550 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f9c3400c780 tx=0x7f9c3400cc40 comp rx=0 tx=0).stop 2024-03-31T15:05:29.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:29.079+0000 7f9c42200640 1 -- 172.21.15.16:0/351748103 shutdown_connections 2024-03-31T15:05:29.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:29.079+0000 7f9c42200640 1 --2- 172.21.15.16:0/351748103 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f9c18068380 0x7f9c1806a840 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:29.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:29.079+0000 7f9c42200640 1 --2- 172.21.15.16:0/351748103 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9c3c1567e0 0x7f9c3c136550 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:29.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:29.079+0000 7f9c42200640 1 --2- 172.21.15.16:0/351748103 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9c3c0731c0 0x7f9c3c0b49b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:29.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:29.079+0000 7f9c42200640 1 --2- 172.21.15.16:0/351748103 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f9c3c06da10 0x7f9c3c147510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:29.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:29.079+0000 7f9c42200640 1 -- 172.21.15.16:0/351748103 >> 172.21.15.16:0/351748103 conn(0x7f9c3c06ee40 msgr2=0x7f9c3c0cd1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:05:29.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:29.079+0000 7f9c42200640 1 -- 172.21.15.16:0/351748103 shutdown_connections 2024-03-31T15:05:29.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:29.079+0000 7f9c42200640 1 -- 172.21.15.16:0/351748103 wait complete. 2024-03-31T15:05:29.096 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-03-31T15:05:29.096 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-31T15:05:29.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:29.243+0000 7faf14000640 1 Processor -- start 2024-03-31T15:05:29.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:29.247+0000 7faf14000640 1 -- start start 2024-03-31T15:05:29.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:29.247+0000 7faf14000640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7faf0c1411b0 0x7faf0c06f920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:29.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:29.247+0000 7faf14000640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7faf0c141590 0x7faf0c06fe60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:29.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:29.247+0000 7faf14000640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7faf0c143c30 0x7faf0c1440a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:29.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:29.247+0000 7faf14000640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7faf0c162910 con 0x7faf0c1411b0 2024-03-31T15:05:29.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:29.247+0000 7faf14000640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7faf0c144980 con 0x7faf0c143c30 2024-03-31T15:05:29.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:29.247+0000 7faf14000640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7faf0c161fe0 con 0x7faf0c141590 2024-03-31T15:05:29.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:29.247+0000 7faf11600640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7faf0c1411b0 0x7faf0c06f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:29.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:29.247+0000 7faf11600640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7faf0c1411b0 0x7faf0c06f920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34110/0 (socket says 172.21.15.16:34110) 2024-03-31T15:05:29.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:29.247+0000 7faf11600640 1 -- 172.21.15.16:0/1278362653 learned_addr learned my addr 172.21.15.16:0/1278362653 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:05:29.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:29.247+0000 7faf10c00640 1 --2- 172.21.15.16:0/1278362653 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7faf0c141590 0x7faf0c06fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:29.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:29.247+0000 7faf12000640 1 --2- 172.21.15.16:0/1278362653 >> v2:172.21.15.89:3300/0 conn(0x7faf0c143c30 0x7faf0c1440a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:29.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:29.247+0000 7faf11600640 1 -- 172.21.15.16:0/1278362653 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7faf0c141590 msgr2=0x7faf0c06fe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:05:29.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:29.247+0000 7faf11600640 1 --2- 172.21.15.16:0/1278362653 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7faf0c141590 0x7faf0c06fe60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:29.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:29.247+0000 7faf11600640 1 -- 172.21.15.16:0/1278362653 >> v2:172.21.15.89:3300/0 conn(0x7faf0c143c30 msgr2=0x7faf0c1440a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:05:29.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:29.247+0000 7faf11600640 1 --2- 172.21.15.16:0/1278362653 >> v2:172.21.15.89:3300/0 conn(0x7faf0c143c30 0x7faf0c1440a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:29.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:29.247+0000 7faf11600640 1 -- 172.21.15.16:0/1278362653 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf0c1622c0 con 0x7faf0c1411b0 2024-03-31T15:05:29.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:29.251+0000 7faf11600640 1 --2- 172.21.15.16:0/1278362653 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faf0c1411b0 0x7faf0c06f920 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7faf0c0b1c00 tx=0x7faf0001c4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8a6f0cfcd7fdd51 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:05:29.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:29.251+0000 7faf0be00640 1 -- 172.21.15.16:0/1278362653 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf000ab640 con 0x7faf0c1411b0 2024-03-31T15:05:29.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:29.251+0000 7faf0be00640 1 -- 172.21.15.16:0/1278362653 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faf000b0070 con 0x7faf0c1411b0 2024-03-31T15:05:29.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:29.251+0000 7faf0be00640 1 -- 172.21.15.16:0/1278362653 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf000aba00 con 0x7faf0c1411b0 2024-03-31T15:05:29.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:29.251+0000 7faf14000640 1 -- 172.21.15.16:0/1278362653 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faf0c1411b0 msgr2=0x7faf0c06f920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:29.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:29.251+0000 7faf14000640 1 --2- 172.21.15.16:0/1278362653 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faf0c1411b0 0x7faf0c06f920 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7faf0c0b1c00 tx=0x7faf0001c4f0 comp rx=0 tx=0).stop 2024-03-31T15:05:29.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:29.251+0000 7faf14000640 1 -- 172.21.15.16:0/1278362653 shutdown_connections 2024-03-31T15:05:29.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:29.251+0000 7faf14000640 1 --2- 172.21.15.16:0/1278362653 >> v2:172.21.15.89:3300/0 conn(0x7faf0c143c30 0x7faf0c1440a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:29.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:29.251+0000 7faf14000640 1 --2- 172.21.15.16:0/1278362653 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7faf0c141590 0x7faf0c06fe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:29.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:29.251+0000 7faf14000640 1 --2- 172.21.15.16:0/1278362653 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faf0c1411b0 0x7faf0c06f920 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:29.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:29.251+0000 7faf14000640 1 -- 172.21.15.16:0/1278362653 >> 172.21.15.16:0/1278362653 conn(0x7faf0c067930 msgr2=0x7faf0c0b3ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:05:29.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:29.251+0000 7faf14000640 1 -- 172.21.15.16:0/1278362653 shutdown_connections 2024-03-31T15:05:29.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:29.251+0000 7faf14000640 1 -- 172.21.15.16:0/1278362653 wait complete. 2024-03-31T15:05:29.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:29.255+0000 7faf14000640 1 Processor -- start 2024-03-31T15:05:29.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:29.255+0000 7faf14000640 1 -- start start 2024-03-31T15:05:29.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:29.255+0000 7faf14000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faf0c1411b0 0x7faf0c158f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:29.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:29.255+0000 7faf14000640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faf0c141590 0x7faf0c13ca00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:29.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:29.255+0000 7faf14000640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7faf0c143c30 0x7faf0c15d060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:29.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:29.255+0000 7faf14000640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7faf0c0717f0 con 0x7faf0c1411b0 2024-03-31T15:05:29.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:29.255+0000 7faf14000640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7faf0c071670 con 0x7faf0c141590 2024-03-31T15:05:29.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:29.255+0000 7faf14000640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7faf0c071970 con 0x7faf0c143c30 2024-03-31T15:05:29.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:29.255+0000 7faf11600640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faf0c1411b0 0x7faf0c158f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:29.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:29.255+0000 7faf11600640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faf0c1411b0 0x7faf0c158f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34118/0 (socket says 172.21.15.16:34118) 2024-03-31T15:05:29.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:29.255+0000 7faf11600640 1 -- 172.21.15.16:0/3797890400 learned_addr learned my addr 172.21.15.16:0/3797890400 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:05:29.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:29.255+0000 7faf10c00640 1 --2- 172.21.15.16:0/3797890400 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faf0c141590 0x7faf0c13ca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:29.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:29.255+0000 7faf12000640 1 --2- 172.21.15.16:0/3797890400 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7faf0c143c30 0x7faf0c15d060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:29.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:29.255+0000 7faf11600640 1 -- 172.21.15.16:0/3797890400 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7faf0c143c30 msgr2=0x7faf0c15d060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:29.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:29.255+0000 7faf11600640 1 --2- 172.21.15.16:0/3797890400 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7faf0c143c30 0x7faf0c15d060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:29.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:29.255+0000 7faf11600640 1 -- 172.21.15.16:0/3797890400 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faf0c141590 msgr2=0x7faf0c13ca00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:29.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:29.255+0000 7faf11600640 1 --2- 172.21.15.16:0/3797890400 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faf0c141590 0x7faf0c13ca00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:29.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:29.255+0000 7faf11600640 1 -- 172.21.15.16:0/3797890400 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf0c159490 con 0x7faf0c1411b0 2024-03-31T15:05:29.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:29.255+0000 7faf11600640 1 --2- 172.21.15.16:0/3797890400 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faf0c1411b0 0x7faf0c158f50 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7faf0001cd90 tx=0x7faf0001ad30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:05:29.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:29.255+0000 7faf09600640 1 -- 172.21.15.16:0/3797890400 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf000be3f0 con 0x7faf0c1411b0 2024-03-31T15:05:29.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:29.255+0000 7faf09600640 1 -- 172.21.15.16:0/3797890400 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faf000be590 con 0x7faf0c1411b0 2024-03-31T15:05:29.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:29.255+0000 7faf14000640 1 -- 172.21.15.16:0/3797890400 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faf0c15db20 con 0x7faf0c1411b0 2024-03-31T15:05:29.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:29.255+0000 7faf09600640 1 -- 172.21.15.16:0/3797890400 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf000be960 con 0x7faf0c1411b0 2024-03-31T15:05:29.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:29.255+0000 7faf14000640 1 -- 172.21.15.16:0/3797890400 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faf0c158470 con 0x7faf0c1411b0 2024-03-31T15:05:29.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:29.255+0000 7faf14000640 1 -- 172.21.15.16:0/3797890400 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faf0c069950 con 0x7faf0c1411b0 2024-03-31T15:05:29.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:29.259+0000 7faf09600640 1 -- 172.21.15.16:0/3797890400 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7faf000b43f0 con 0x7faf0c1411b0 2024-03-31T15:05:29.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:29.259+0000 7faf09600640 1 --2- 172.21.15.16:0/3797890400 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7faeec068380 0x7faeec06a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:29.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:29.259+0000 7faf09600640 1 -- 172.21.15.16:0/3797890400 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5709+0+0 (secure 0 0 0) 0x7faf00120fe0 con 0x7faf0c1411b0 2024-03-31T15:05:29.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:29.259+0000 7faf10c00640 1 --2- 172.21.15.16:0/3797890400 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7faeec068380 0x7faeec06a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:29.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:29.259+0000 7faf10c00640 1 --2- 172.21.15.16:0/3797890400 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7faeec068380 0x7faeec06a840 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7faf0c15b3a0 tx=0x7faef8008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:05:29.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:29.263+0000 7faf09600640 1 -- 172.21.15.16:0/3797890400 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7faf000c4060 con 0x7faf0c1411b0 2024-03-31T15:05:29.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:29.487+0000 7faf14000640 1 -- 172.21.15.16:0/3797890400 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7faf0c15b700 con 0x7faf0c1411b0 2024-03-31T15:05:30.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:30.087+0000 7faf09600640 1 -- 172.21.15.16:0/3797890400 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v25) ==== 122+0+0 (secure 0 0 0) 0x7faf00016110 con 0x7faf0c1411b0 2024-03-31T15:05:30.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:30.091+0000 7faf14000640 1 -- 172.21.15.16:0/3797890400 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7faeec068380 msgr2=0x7faeec06a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:30.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:30.091+0000 7faf14000640 1 --2- 172.21.15.16:0/3797890400 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7faeec068380 0x7faeec06a840 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7faf0c15b3a0 tx=0x7faef8008040 comp rx=0 tx=0).stop 2024-03-31T15:05:30.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:30.091+0000 7faf14000640 1 -- 172.21.15.16:0/3797890400 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faf0c1411b0 msgr2=0x7faf0c158f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:30.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:30.091+0000 7faf14000640 1 --2- 172.21.15.16:0/3797890400 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faf0c1411b0 0x7faf0c158f50 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7faf0001cd90 tx=0x7faf0001ad30 comp rx=0 tx=0).stop 2024-03-31T15:05:30.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:30.091+0000 7faf14000640 1 -- 172.21.15.16:0/3797890400 shutdown_connections 2024-03-31T15:05:30.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:30.091+0000 7faf14000640 1 --2- 172.21.15.16:0/3797890400 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7faeec068380 0x7faeec06a840 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:30.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:30.091+0000 7faf14000640 1 --2- 172.21.15.16:0/3797890400 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7faf0c143c30 0x7faf0c15d060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:30.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:30.091+0000 7faf14000640 1 --2- 172.21.15.16:0/3797890400 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faf0c141590 0x7faf0c13ca00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:30.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:30.091+0000 7faf14000640 1 --2- 172.21.15.16:0/3797890400 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faf0c1411b0 0x7faf0c158f50 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:30.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:30.091+0000 7faf14000640 1 -- 172.21.15.16:0/3797890400 >> 172.21.15.16:0/3797890400 conn(0x7faf0c067930 msgr2=0x7faf0c142580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:05:30.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:30.091+0000 7faf14000640 1 -- 172.21.15.16:0/3797890400 shutdown_connections 2024-03-31T15:05:30.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:30.091+0000 7faf14000640 1 -- 172.21.15.16:0/3797890400 wait complete. 2024-03-31T15:05:30.106 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-31T15:05:30.107 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-31T15:05:30.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:30.251+0000 7fad2ea00640 1 Processor -- start 2024-03-31T15:05:30.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:30.251+0000 7fad2ea00640 1 -- start start 2024-03-31T15:05:30.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:30.255+0000 7fad2ea00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fad2806f920 0x7fad2806fd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:30.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:30.255+0000 7fad2ea00640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fad280d0920 con 0x7fad28145e40 2024-03-31T15:05:30.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:30.255+0000 7fad2ea00640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fad280d0c00 con 0x7fad280d2010 2024-03-31T15:05:30.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:30.255+0000 7fad2ea00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fad280702c0 con 0x7fad2806f920 2024-03-31T15:05:30.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:30.255+0000 7fad27e00640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fad28145e40 0x7fad280cc740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:35508/0 (socket says 172.21.15.16:35508) 2024-03-31T15:05:30.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:30.255+0000 7fad27e00640 1 -- 172.21.15.16:0/2299305957 learned_addr learned my addr 172.21.15.16:0/2299305957 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:05:30.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:30.255+0000 7fad1f400640 1 --2- 172.21.15.16:0/2299305957 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fad2806f920 0x7fad2806fd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:30.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:30.255+0000 7fad27400640 1 -- 172.21.15.16:0/2299305957 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 97115460 0 0) 0x7fad280d0920 con 0x7fad28145e40 2024-03-31T15:05:30.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:30.255+0000 7fad27400640 1 -- 172.21.15.16:0/2299305957 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fad0c003530 con 0x7fad28145e40 2024-03-31T15:05:30.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:30.255+0000 7fad27400640 1 -- 172.21.15.16:0/2299305957 <== mon.1 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1187072101 0 0) 0x7fad280d0c00 con 0x7fad280d2010 2024-03-31T15:05:30.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:30.255+0000 7fad27400640 1 -- 172.21.15.16:0/2299305957 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fad280d0920 con 0x7fad280d2010 2024-03-31T15:05:30.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:30.255+0000 7fad27400640 1 -- 172.21.15.16:0/2299305957 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1224043161 0 0) 0x7fad0c003530 con 0x7fad28145e40 2024-03-31T15:05:30.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:30.255+0000 7fad27400640 1 -- 172.21.15.16:0/2299305957 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fad2806f920 msgr2=0x7fad2806fd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:05:30.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:30.255+0000 7fad27400640 1 --2- 172.21.15.16:0/2299305957 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fad2806f920 0x7fad2806fd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:30.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:30.255+0000 7fad27400640 1 -- 172.21.15.16:0/2299305957 >> v1:172.21.15.89:6789/0 conn(0x7fad280d2010 legacy=0x7fad280d24a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:30.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:30.255+0000 7fad27400640 1 -- 172.21.15.16:0/2299305957 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad280d1810 con 0x7fad28145e40 2024-03-31T15:05:30.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:30.255+0000 7fad27400640 1 -- 172.21.15.16:0/2299305957 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 214807623 0 0) 0x7fad18001680 con 0x7fad28145e40 2024-03-31T15:05:30.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:30.255+0000 7fad27400640 1 -- 172.21.15.16:0/2299305957 >> v1:172.21.15.16:6789/0 conn(0x7fad28145e40 legacy=0x7fad280cc740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:30.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:30.255+0000 7fad27400640 1 --2- 172.21.15.16:0/2299305957 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fad0c003b80 0x7fad0c003f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:30.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:30.255+0000 7fad27400640 1 --2- 172.21.15.16:0/2299305957 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fad0c004810 0x7fad0c004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:30.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:30.255+0000 7fad27400640 1 --2- 172.21.15.16:0/2299305957 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fad2806f920 0x7fad0c0051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:30.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:30.255+0000 7fad27400640 1 -- 172.21.15.16:0/2299305957 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fad0c00ba80 con 0x7fad0c003b80 2024-03-31T15:05:30.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:30.255+0000 7fad27400640 1 -- 172.21.15.16:0/2299305957 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fad0c00bc00 con 0x7fad0c004810 2024-03-31T15:05:30.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:30.255+0000 7fad27400640 1 -- 172.21.15.16:0/2299305957 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fad0c00bdd0 con 0x7fad2806f920 2024-03-31T15:05:30.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:30.255+0000 7fad27e00640 1 --2- 172.21.15.16:0/2299305957 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fad0c003b80 0x7fad0c003f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:30.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:30.255+0000 7fad1f400640 1 --2- 172.21.15.16:0/2299305957 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fad0c004810 0x7fad0c004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:30.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:30.255+0000 7fad2ca00640 1 --2- 172.21.15.16:0/2299305957 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fad2806f920 0x7fad0c0051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:30.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:30.255+0000 7fad27e00640 1 -- 172.21.15.16:0/2299305957 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fad2806f920 msgr2=0x7fad0c0051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:05:30.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:30.255+0000 7fad27e00640 1 --2- 172.21.15.16:0/2299305957 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fad2806f920 0x7fad0c0051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:30.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:30.255+0000 7fad27e00640 1 -- 172.21.15.16:0/2299305957 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fad0c004810 msgr2=0x7fad0c004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:05:30.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:30.255+0000 7fad27e00640 1 --2- 172.21.15.16:0/2299305957 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fad0c004810 0x7fad0c004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:30.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:30.255+0000 7fad27e00640 1 -- 172.21.15.16:0/2299305957 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad0c003530 con 0x7fad0c003b80 2024-03-31T15:05:30.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:30.255+0000 7fad2ca00640 1 --2- 172.21.15.16:0/2299305957 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fad2806f920 0x7fad0c0051d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:05:30.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:30.259+0000 7fad27e00640 1 --2- 172.21.15.16:0/2299305957 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fad0c003b80 0x7fad0c003f80 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7fad1800c190 tx=0x7fad1801cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7514a61368f6c32 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:05:30.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:30.259+0000 7fad27400640 1 -- 172.21.15.16:0/2299305957 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad18007cd0 con 0x7fad0c003b80 2024-03-31T15:05:30.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:30.259+0000 7fad27400640 1 -- 172.21.15.16:0/2299305957 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fad180a9020 con 0x7fad0c003b80 2024-03-31T15:05:30.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:30.259+0000 7fad27400640 1 -- 172.21.15.16:0/2299305957 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad18005ae0 con 0x7fad0c003b80 2024-03-31T15:05:30.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:30.259+0000 7fad2ea00640 1 -- 172.21.15.16:0/2299305957 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fad0c003b80 msgr2=0x7fad0c003f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:30.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:30.259+0000 7fad2ea00640 1 --2- 172.21.15.16:0/2299305957 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fad0c003b80 0x7fad0c003f80 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7fad1800c190 tx=0x7fad1801cbc0 comp rx=0 tx=0).stop 2024-03-31T15:05:30.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:30.259+0000 7fad2ca00640 1 -- 172.21.15.16:0/2299305957 reap_dead start 2024-03-31T15:05:30.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:30.259+0000 7fad2ea00640 1 -- 172.21.15.16:0/2299305957 shutdown_connections 2024-03-31T15:05:30.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:30.259+0000 7fad2ea00640 1 -- 172.21.15.16:0/2299305957 >> 172.21.15.16:0/2299305957 conn(0x7fad28067330 msgr2=0x7fad28067740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:05:30.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:30.259+0000 7fad2ea00640 1 -- 172.21.15.16:0/2299305957 shutdown_connections 2024-03-31T15:05:30.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:30.259+0000 7fad2ea00640 1 -- 172.21.15.16:0/2299305957 wait complete. 2024-03-31T15:05:30.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:30.259+0000 7fad2ea00640 1 Processor -- start 2024-03-31T15:05:30.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:30.259+0000 7fad2ea00640 1 -- start start 2024-03-31T15:05:30.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:30.263+0000 7fad2ea00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fad0c003b80 0x7fad2815c030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:30.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:30.263+0000 7fad2ea00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fad280cb260 0x7fad280cb6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:30.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:30.263+0000 7fad2ea00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fad2815d150 0x7fad2815d5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:30.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:30.263+0000 7fad2ea00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fad28070110 con 0x7fad2815d150 2024-03-31T15:05:30.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:30.263+0000 7fad2ea00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fad280d06d0 con 0x7fad280cb260 2024-03-31T15:05:30.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:30.263+0000 7fad2ea00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fad280d0ee0 con 0x7fad0c003b80 2024-03-31T15:05:30.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:30.263+0000 7fad2ca00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fad2815d150 0x7fad2815d5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:30.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:30.263+0000 7fad2ca00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fad2815d150 0x7fad2815d5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34138/0 (socket says 172.21.15.16:34138) 2024-03-31T15:05:30.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:30.263+0000 7fad2ca00640 1 -- 172.21.15.16:0/327030990 learned_addr learned my addr 172.21.15.16:0/327030990 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:05:30.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:30.263+0000 7fad27e00640 1 --2- 172.21.15.16:0/327030990 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fad0c003b80 0x7fad2815c030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:30.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:30.263+0000 7fad1f400640 1 --2- 172.21.15.16:0/327030990 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fad280cb260 0x7fad280cb6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:30.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:30.263+0000 7fad2ca00640 1 -- 172.21.15.16:0/327030990 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fad0c003b80 msgr2=0x7fad2815c030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:30.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:30.263+0000 7fad2ca00640 1 --2- 172.21.15.16:0/327030990 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fad0c003b80 0x7fad2815c030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:30.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:30.263+0000 7fad2ca00640 1 -- 172.21.15.16:0/327030990 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fad280cb260 msgr2=0x7fad280cb6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:30.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:30.263+0000 7fad2ca00640 1 --2- 172.21.15.16:0/327030990 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fad280cb260 0x7fad280cb6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:30.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:30.263+0000 7fad2ca00640 1 -- 172.21.15.16:0/327030990 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad280c7030 con 0x7fad2815d150 2024-03-31T15:05:30.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:30.263+0000 7fad2ca00640 1 --2- 172.21.15.16:0/327030990 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fad2815d150 0x7fad2815d5d0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7fad28068280 tx=0x7fad2000eb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:05:30.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:30.263+0000 7fad24c00640 1 -- 172.21.15.16:0/327030990 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad20019070 con 0x7fad2815d150 2024-03-31T15:05:30.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:30.263+0000 7fad24c00640 1 -- 172.21.15.16:0/327030990 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fad20006b60 con 0x7fad2815d150 2024-03-31T15:05:30.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:30.263+0000 7fad24c00640 1 -- 172.21.15.16:0/327030990 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad20003980 con 0x7fad2815d150 2024-03-31T15:05:30.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:30.263+0000 7fad2ea00640 1 -- 172.21.15.16:0/327030990 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fad280cbd60 con 0x7fad2815d150 2024-03-31T15:05:30.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:30.263+0000 7fad2ea00640 1 -- 172.21.15.16:0/327030990 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fad280c75b0 con 0x7fad2815d150 2024-03-31T15:05:30.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:30.263+0000 7fad2ea00640 1 -- 172.21.15.16:0/327030990 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fad28069950 con 0x7fad2815d150 2024-03-31T15:05:30.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:30.263+0000 7fad24c00640 1 -- 172.21.15.16:0/327030990 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fad2001c070 con 0x7fad2815d150 2024-03-31T15:05:30.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:30.267+0000 7fad24c00640 1 --2- 172.21.15.16:0/327030990 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fad04068380 0x7fad0406a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:30.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:30.267+0000 7fad24c00640 1 -- 172.21.15.16:0/327030990 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5709+0+0 (secure 0 0 0) 0x7fad20085bb0 con 0x7fad2815d150 2024-03-31T15:05:30.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:30.267+0000 7fad27e00640 1 --2- 172.21.15.16:0/327030990 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fad04068380 0x7fad0406a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:30.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:30.267+0000 7fad27e00640 1 --2- 172.21.15.16:0/327030990 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fad04068380 0x7fad0406a840 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fad180aa000 tx=0x7fad180b1040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:05:30.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:30.271+0000 7fad24c00640 1 -- 172.21.15.16:0/327030990 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fad20055a90 con 0x7fad2815d150 2024-03-31T15:05:30.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:30.471+0000 7fad2ea00640 1 -- 172.21.15.16:0/327030990 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fad280cf3c0 con 0x7fad2815d150 2024-03-31T15:05:30.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:30.471+0000 7fad24c00640 1 -- 172.21.15.16:0/327030990 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+5175 (secure 0 0 0) 0x7fad20089070 con 0x7fad2815d150 2024-03-31T15:05:30.467 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:05:30.467 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":25,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/285985105","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6835","nonce":285985105},{"type":"v1","addr":"172.21.15.89:6837","nonce":285985105}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20},{"gid":5689,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6836/1615931247","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":1615931247},{"type":"v1","addr":"172.21.15.89:6836","nonce":1615931247}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20},{"gid":5694,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/512187852","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6833","nonce":512187852},{"type":"v1","addr":"172.21.15.16:6835","nonce":512187852}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-31T15:05:26.461908+0000","modified":"2024-03-31T15:05:29.491163+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5697},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5697":{"gid":5697,"name":"a","rank":0,"incarnation":21,"state":"up:active","state_seq":3,"addr":"172.21.15.16:6834/2768973409","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":2768973409},{"type":"v1","addr":"172.21.15.16:6834","nonce":2768973409}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5697,"qdb_cluster":[5697]},"id":3}]} 2024-03-31T15:05:30.467 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 25 2024-03-31T15:05:30.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:30.475+0000 7fad2ea00640 1 -- 172.21.15.16:0/327030990 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fad04068380 msgr2=0x7fad0406a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:30.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:30.475+0000 7fad2ea00640 1 --2- 172.21.15.16:0/327030990 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fad04068380 0x7fad0406a840 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fad180aa000 tx=0x7fad180b1040 comp rx=0 tx=0).stop 2024-03-31T15:05:30.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:30.475+0000 7fad2ea00640 1 -- 172.21.15.16:0/327030990 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fad2815d150 msgr2=0x7fad2815d5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:30.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:30.475+0000 7fad2ea00640 1 --2- 172.21.15.16:0/327030990 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fad2815d150 0x7fad2815d5d0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7fad28068280 tx=0x7fad2000eb20 comp rx=0 tx=0).stop 2024-03-31T15:05:30.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:30.475+0000 7fad2ea00640 1 -- 172.21.15.16:0/327030990 shutdown_connections 2024-03-31T15:05:30.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:30.475+0000 7fad2ea00640 1 --2- 172.21.15.16:0/327030990 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fad04068380 0x7fad0406a840 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:30.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:30.475+0000 7fad2ea00640 1 --2- 172.21.15.16:0/327030990 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fad2815d150 0x7fad2815d5d0 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:30.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:30.475+0000 7fad2ea00640 1 --2- 172.21.15.16:0/327030990 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fad280cb260 0x7fad280cb6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:30.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:30.475+0000 7fad2ea00640 1 --2- 172.21.15.16:0/327030990 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fad0c003b80 0x7fad2815c030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:30.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:30.475+0000 7fad2ea00640 1 -- 172.21.15.16:0/327030990 >> 172.21.15.16:0/327030990 conn(0x7fad28067330 msgr2=0x7fad28161050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:05:30.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:30.475+0000 7fad2ea00640 1 -- 172.21.15.16:0/327030990 shutdown_connections 2024-03-31T15:05:30.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:30.475+0000 7fad2ea00640 1 -- 172.21.15.16:0/327030990 wait complete. 2024-03-31T15:05:30.499 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-31T15:05:30.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:30.643+0000 7f5a09c00640 1 Processor -- start 2024-03-31T15:05:30.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:30.643+0000 7f5a09c00640 1 -- start start 2024-03-31T15:05:30.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:30.643+0000 7f5a09c00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f5a04070030 0x7f5a04162180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:30.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:30.643+0000 7f5a09c00640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5a040cbe90 con 0x7f5a0415f650 2024-03-31T15:05:30.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:30.643+0000 7f5a09c00640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5a040cc170 con 0x7f5a0406dbc0 2024-03-31T15:05:30.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:30.643+0000 7f5a09c00640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f5a0415f1c0 con 0x7f5a04070030 2024-03-31T15:05:30.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:30.647+0000 7f59fbe00640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f5a0415f650 0x7f5a0406f920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:35524/0 (socket says 172.21.15.16:35524) 2024-03-31T15:05:30.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:30.647+0000 7f59fbe00640 1 -- 172.21.15.16:0/422638891 learned_addr learned my addr 172.21.15.16:0/422638891 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:05:30.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:30.647+0000 7f5a03400640 1 --2- 172.21.15.16:0/422638891 >> v2:172.21.15.89:3300/0 conn(0x7f5a04070030 0x7f5a04162180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:30.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:30.647+0000 7f5a02000640 1 -- 172.21.15.16:0/422638891 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3232593684 0 0) 0x7f5a040cbe90 con 0x7f5a0415f650 2024-03-31T15:05:30.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:30.647+0000 7f5a02000640 1 -- 172.21.15.16:0/422638891 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f59e8003530 con 0x7f5a0415f650 2024-03-31T15:05:30.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:30.647+0000 7f5a03400640 1 -- 172.21.15.16:0/422638891 >> v1:172.21.15.89:6789/0 conn(0x7f5a0406dbc0 legacy=0x7f5a040cb440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:30.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:30.647+0000 7f5a02000640 1 -- 172.21.15.16:0/422638891 <== mon.1 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4053543001 0 0) 0x7f5a040cc170 con 0x7f5a0406dbc0 2024-03-31T15:05:30.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:30.647+0000 7f5a03400640 1 -- 172.21.15.16:0/422638891 >> v1:172.21.15.16:6789/0 conn(0x7f5a0415f650 legacy=0x7f5a0406f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:30.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:30.647+0000 7f5a03400640 1 -- 172.21.15.16:0/422638891 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a0415ee50 con 0x7f5a04070030 2024-03-31T15:05:30.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:30.647+0000 7f5a03400640 1 --2- 172.21.15.16:0/422638891 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5a04070030 0x7f5a04162180 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f59fc009d90 tx=0x7f59fc01c800 comp rx=0 tx=0).ready entity=mon.1 client_cookie=27abe79e6f37bb4f server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:05:30.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:30.651+0000 7f5a02000640 1 -- 172.21.15.16:0/422638891 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59fc0b2070 con 0x7f5a04070030 2024-03-31T15:05:30.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:30.651+0000 7f5a02000640 1 -- 172.21.15.16:0/422638891 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f59fc0a5800 con 0x7f5a04070030 2024-03-31T15:05:30.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:30.651+0000 7f5a02000640 1 -- 172.21.15.16:0/422638891 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59fc0a5bb0 con 0x7f5a04070030 2024-03-31T15:05:30.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:30.651+0000 7f5a09c00640 1 -- 172.21.15.16:0/422638891 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5a04070030 msgr2=0x7f5a04162180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:30.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:30.651+0000 7f5a09c00640 1 --2- 172.21.15.16:0/422638891 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5a04070030 0x7f5a04162180 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f59fc009d90 tx=0x7f59fc01c800 comp rx=0 tx=0).stop 2024-03-31T15:05:30.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:30.651+0000 7f5a09c00640 1 -- 172.21.15.16:0/422638891 shutdown_connections 2024-03-31T15:05:30.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:30.651+0000 7f5a09c00640 1 --2- 172.21.15.16:0/422638891 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5a04070030 0x7f5a04162180 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:30.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:30.651+0000 7f5a09c00640 1 -- 172.21.15.16:0/422638891 >> 172.21.15.16:0/422638891 conn(0x7f5a04067330 msgr2=0x7f5a04067740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:05:30.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:30.651+0000 7f5a09c00640 1 -- 172.21.15.16:0/422638891 shutdown_connections 2024-03-31T15:05:30.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:30.651+0000 7f5a09c00640 1 -- 172.21.15.16:0/422638891 wait complete. 2024-03-31T15:05:30.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:30.651+0000 7f5a09c00640 1 Processor -- start 2024-03-31T15:05:30.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:30.651+0000 7f5a09c00640 1 -- start start 2024-03-31T15:05:30.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:30.651+0000 7f5a09c00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5a0406dbc0 0x7f5a04159530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:30.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:30.651+0000 7f5a09c00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f5a04070030 0x7f5a040b6820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:30.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:30.651+0000 7f5a09c00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5a0415f650 0x7f5a040b7660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:30.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:30.651+0000 7f5a09c00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f5a0406c950 con 0x7f5a0406dbc0 2024-03-31T15:05:30.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:30.651+0000 7f5a09c00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f5a0406dfa0 con 0x7f5a0415f650 2024-03-31T15:05:30.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:30.651+0000 7f5a09c00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f5a0406cad0 con 0x7f5a04070030 2024-03-31T15:05:30.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:30.651+0000 7f5a02a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5a0406dbc0 0x7f5a04159530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:30.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:30.651+0000 7f5a02a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5a0406dbc0 0x7f5a04159530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34154/0 (socket says 172.21.15.16:34154) 2024-03-31T15:05:30.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:30.651+0000 7f5a02a00640 1 -- 172.21.15.16:0/2983937873 learned_addr learned my addr 172.21.15.16:0/2983937873 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:05:30.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:30.651+0000 7f59fbe00640 1 --2- 172.21.15.16:0/2983937873 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f5a04070030 0x7f5a040b6820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:30.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:30.651+0000 7f5a03400640 1 --2- 172.21.15.16:0/2983937873 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5a0415f650 0x7f5a040b7660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:30.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:30.655+0000 7f5a02a00640 1 -- 172.21.15.16:0/2983937873 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f5a04070030 msgr2=0x7f5a040b6820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:30.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:30.655+0000 7f5a02a00640 1 --2- 172.21.15.16:0/2983937873 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f5a04070030 0x7f5a040b6820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:30.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:30.655+0000 7f5a02a00640 1 -- 172.21.15.16:0/2983937873 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5a0415f650 msgr2=0x7f5a040b7660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:30.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:30.655+0000 7f5a02a00640 1 --2- 172.21.15.16:0/2983937873 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5a0415f650 0x7f5a040b7660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:30.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:30.655+0000 7f5a02a00640 1 -- 172.21.15.16:0/2983937873 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a040b1f40 con 0x7f5a0406dbc0 2024-03-31T15:05:30.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:30.655+0000 7f5a02a00640 1 --2- 172.21.15.16:0/2983937873 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5a0406dbc0 0x7f5a04159530 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f59f4001ec0 tx=0x7f59f400c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:05:30.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:30.655+0000 7f59faa00640 1 -- 172.21.15.16:0/2983937873 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59f4017070 con 0x7f5a0406dbc0 2024-03-31T15:05:30.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:30.655+0000 7f59faa00640 1 -- 172.21.15.16:0/2983937873 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f59f4003600 con 0x7f5a0406dbc0 2024-03-31T15:05:30.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:30.655+0000 7f59faa00640 1 -- 172.21.15.16:0/2983937873 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59f40039f0 con 0x7f5a0406dbc0 2024-03-31T15:05:30.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:30.655+0000 7f5a09c00640 1 -- 172.21.15.16:0/2983937873 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5a040b2220 con 0x7f5a0406dbc0 2024-03-31T15:05:30.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:30.655+0000 7f5a09c00640 1 -- 172.21.15.16:0/2983937873 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5a040b27e0 con 0x7f5a0406dbc0 2024-03-31T15:05:30.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:30.655+0000 7f5a09c00640 1 -- 172.21.15.16:0/2983937873 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5a04069950 con 0x7f5a0406dbc0 2024-03-31T15:05:30.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:30.655+0000 7f59faa00640 1 -- 172.21.15.16:0/2983937873 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f59f401a070 con 0x7f5a0406dbc0 2024-03-31T15:05:30.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:30.655+0000 7f59faa00640 1 --2- 172.21.15.16:0/2983937873 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f59e0068380 0x7f59e006a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:30.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:30.659+0000 7f59faa00640 1 -- 172.21.15.16:0/2983937873 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5709+0+0 (secure 0 0 0) 0x7f59f4085d70 con 0x7f5a0406dbc0 2024-03-31T15:05:30.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:30.659+0000 7f59fbe00640 1 --2- 172.21.15.16:0/2983937873 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f59e0068380 0x7f59e006a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:30.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:30.659+0000 7f59fbe00640 1 --2- 172.21.15.16:0/2983937873 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f59e0068380 0x7f59e006a840 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f5a0406cf30 tx=0x7f59ec007d40 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:05:30.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:30.663+0000 7f59faa00640 1 -- 172.21.15.16:0/2983937873 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f59f4055c40 con 0x7f5a0406dbc0 2024-03-31T15:05:30.822 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:30.827+0000 7f5a09c00640 1 -- 172.21.15.16:0/2983937873 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f5a040cf380 con 0x7f5a0406dbc0 2024-03-31T15:05:30.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:30.827+0000 7f59faa00640 1 -- 172.21.15.16:0/2983937873 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v45) ==== 74+0+16541 (secure 0 0 0) 0x7f59f4059f40 con 0x7f5a0406dbc0 2024-03-31T15:05:30.823 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:05:30.823 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":45,"fsid":"286eb5da-fb47-4550-b290-d9519c13b9a5","created":"2024-03-31T15:03:11.438952+0000","modified":"2024-03-31T15:05:28.074367+0000","last_up_change":"2024-03-31T15:03:22.885535+0000","last_in_change":"2024-03-31T15:03:13.941667+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-31T15:03:22.550787+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2024-03-31T15:05:24.856239+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"44","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2024-03-31T15:05:25.417826+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"45","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"92a66450-ef19-40d8-ac3d-d760b23b0bda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6801","nonce":932712244}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6803","nonce":932712244}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6807","nonce":932712244}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6805","nonce":932712244}]},"public_addr":"172.21.15.16:6801/932712244","cluster_addr":"172.21.15.16:6803/932712244","heartbeat_back_addr":"172.21.15.16:6807/932712244","heartbeat_front_addr":"172.21.15.16:6805/932712244","state":["exists","up"]},{"osd":1,"uuid":"da9d1f2b-1968-4959-b843-faa231988e92","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6810","nonce":1398737533}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6811","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6813","nonce":1398737533}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6821","nonce":1398737533}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6815","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6816","nonce":1398737533}]},"public_addr":"172.21.15.16:6810/1398737533","cluster_addr":"172.21.15.16:6813/1398737533","heartbeat_back_addr":"172.21.15.16:6821/1398737533","heartbeat_front_addr":"172.21.15.16:6816/1398737533","state":["exists","up"]},{"osd":2,"uuid":"01a73662-9a21-4b1f-bcbb-42b9636a4638","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6819","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6822","nonce":3018526646}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6825","nonce":3018526646}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6831","nonce":3018526646}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6827","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6829","nonce":3018526646}]},"public_addr":"172.21.15.16:6822/3018526646","cluster_addr":"172.21.15.16:6825/3018526646","heartbeat_back_addr":"172.21.15.16:6831/3018526646","heartbeat_front_addr":"172.21.15.16:6829/3018526646","state":["exists","up"]},{"osd":3,"uuid":"c7771fd3-e18e-4500-ac38-23cf10ebcd75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6809","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6812","nonce":3360352974}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6817","nonce":3360352974}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6828","nonce":3360352974}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6823","nonce":3360352974}]},"public_addr":"172.21.15.16:6812/3360352974","cluster_addr":"172.21.15.16:6817/3360352974","heartbeat_back_addr":"172.21.15.16:6828/3360352974","heartbeat_front_addr":"172.21.15.16:6823/3360352974","state":["exists","up"]},{"osd":4,"uuid":"88260730-1482-4fbd-a40f-cb1124e9df4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6801","nonce":1856844856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6803","nonce":1856844856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6807","nonce":1856844856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6805","nonce":1856844856}]},"public_addr":"172.21.15.89:6801/1856844856","cluster_addr":"172.21.15.89:6803/1856844856","heartbeat_back_addr":"172.21.15.89:6807/1856844856","heartbeat_front_addr":"172.21.15.89:6805/1856844856","state":["exists","up"]},{"osd":5,"uuid":"4cb2c00c-fae0-4dd8-9d27-ec4f0093ccf7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6814","nonce":50395527}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6817","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6820","nonce":50395527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6831","nonce":50395527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6823","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6826","nonce":50395527}]},"public_addr":"172.21.15.89:6814/50395527","cluster_addr":"172.21.15.89:6820/50395527","heartbeat_back_addr":"172.21.15.89:6831/50395527","heartbeat_front_addr":"172.21.15.89:6826/50395527","state":["exists","up"]},{"osd":6,"uuid":"81ce0c19-2554-4cae-acbe-66a05e9e9b9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6809","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6812","nonce":3470471752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6815","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6818","nonce":3470471752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6827","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6829","nonce":3470471752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6821","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6824","nonce":3470471752}]},"public_addr":"172.21.15.89:6812/3470471752","cluster_addr":"172.21.15.89:6818/3470471752","heartbeat_back_addr":"172.21.15.89:6829/3470471752","heartbeat_front_addr":"172.21.15.89:6824/3470471752","state":["exists","up"]},{"osd":7,"uuid":"85e4313b-d37d-442e-8042-153bf20a847d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6811","nonce":1796778630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6813","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6816","nonce":1796778630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6825","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6828","nonce":1796778630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6819","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6822","nonce":1796778630}]},"public_addr":"172.21.15.89:6811/1796778630","cluster_addr":"172.21.15.89:6816/1796778630","heartbeat_back_addr":"172.21.15.89:6828/1796778630","heartbeat_front_addr":"172.21.15.89:6822/1796778630","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.439088+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.529970+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.680512+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:21.044161+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.582111+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.536558+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.409522+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.590229+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-31T15:05:30.825 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:30.831+0000 7f5a09c00640 1 -- 172.21.15.16:0/2983937873 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f59e0068380 msgr2=0x7f59e006a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:30.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:30.831+0000 7f5a09c00640 1 --2- 172.21.15.16:0/2983937873 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f59e0068380 0x7f59e006a840 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f5a0406cf30 tx=0x7f59ec007d40 comp rx=0 tx=0).stop 2024-03-31T15:05:30.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:30.831+0000 7f5a09c00640 1 -- 172.21.15.16:0/2983937873 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5a0406dbc0 msgr2=0x7f5a04159530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:30.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:30.831+0000 7f5a09c00640 1 --2- 172.21.15.16:0/2983937873 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5a0406dbc0 0x7f5a04159530 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f59f4001ec0 tx=0x7f59f400c8e0 comp rx=0 tx=0).stop 2024-03-31T15:05:30.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:30.831+0000 7f5a09c00640 1 -- 172.21.15.16:0/2983937873 shutdown_connections 2024-03-31T15:05:30.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:30.831+0000 7f5a09c00640 1 --2- 172.21.15.16:0/2983937873 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f59e0068380 0x7f59e006a840 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:30.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:30.831+0000 7f5a09c00640 1 --2- 172.21.15.16:0/2983937873 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5a0415f650 0x7f5a040b7660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:30.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:30.831+0000 7f5a09c00640 1 --2- 172.21.15.16:0/2983937873 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f5a04070030 0x7f5a040b6820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:30.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:30.831+0000 7f5a09c00640 1 --2- 172.21.15.16:0/2983937873 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5a0406dbc0 0x7f5a04159530 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:30.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:30.831+0000 7f5a09c00640 1 -- 172.21.15.16:0/2983937873 >> 172.21.15.16:0/2983937873 conn(0x7f5a04067330 msgr2=0x7f5a04067710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:05:30.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:30.831+0000 7f5a09c00640 1 -- 172.21.15.16:0/2983937873 shutdown_connections 2024-03-31T15:05:30.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:30.831+0000 7f5a09c00640 1 -- 172.21.15.16:0/2983937873 wait complete. 2024-03-31T15:05:30.844 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-31T15:05:30.845 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-31T15:05:30.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:30.991+0000 7f21c2e00640 1 Processor -- start 2024-03-31T15:05:30.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:30.991+0000 7f21c2e00640 1 -- start start 2024-03-31T15:05:30.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:30.991+0000 7f21c2e00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f21bc145d10 0x7f21bc0cf790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:30.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:30.991+0000 7f21c2e00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f21bc0728b0 0x7f21bc0b1cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:30.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:30.991+0000 7f21c2e00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f21bc0b2200 0x7f21bc15cd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:30.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:30.991+0000 7f21c2e00640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f21bc160e70 con 0x7f21bc0728b0 2024-03-31T15:05:30.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:30.991+0000 7f21c2e00640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f21bc0cfcd0 con 0x7f21bc145d10 2024-03-31T15:05:30.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:30.991+0000 7f21c2e00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f21bc1602f0 con 0x7f21bc0b2200 2024-03-31T15:05:30.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:30.991+0000 7f21b3400640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f21bc0728b0 0x7f21bc0b1cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:30.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:30.991+0000 7f21b3400640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f21bc0728b0 0x7f21bc0b1cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34158/0 (socket says 172.21.15.16:34158) 2024-03-31T15:05:30.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:30.991+0000 7f21b3400640 1 -- 172.21.15.16:0/2829818802 learned_addr learned my addr 172.21.15.16:0/2829818802 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:05:30.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:30.991+0000 7f21c0e00640 1 --2- 172.21.15.16:0/2829818802 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f21bc0b2200 0x7f21bc15cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:30.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:30.991+0000 7f21bbe00640 1 --2- 172.21.15.16:0/2829818802 >> v2:172.21.15.89:3300/0 conn(0x7f21bc145d10 0x7f21bc0cf790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:30.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:30.991+0000 7f21b3400640 1 -- 172.21.15.16:0/2829818802 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f21bc0b2200 msgr2=0x7f21bc15cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:05:30.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:30.991+0000 7f21b3400640 1 --2- 172.21.15.16:0/2829818802 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f21bc0b2200 0x7f21bc15cd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:30.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:30.991+0000 7f21b3400640 1 -- 172.21.15.16:0/2829818802 >> v2:172.21.15.89:3300/0 conn(0x7f21bc145d10 msgr2=0x7f21bc0cf790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:05:30.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:30.991+0000 7f21b3400640 1 --2- 172.21.15.16:0/2829818802 >> v2:172.21.15.89:3300/0 conn(0x7f21bc145d10 0x7f21bc0cf790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:30.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:30.991+0000 7f21b3400640 1 -- 172.21.15.16:0/2829818802 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f21bc1605d0 con 0x7f21bc0728b0 2024-03-31T15:05:30.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:30.991+0000 7f21bbe00640 1 --2- 172.21.15.16:0/2829818802 >> v2:172.21.15.89:3300/0 conn(0x7f21bc145d10 0x7f21bc0cf790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:05:30.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:30.991+0000 7f21c0e00640 1 --2- 172.21.15.16:0/2829818802 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f21bc0b2200 0x7f21bc15cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:05:30.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:30.995+0000 7f21b3400640 1 --2- 172.21.15.16:0/2829818802 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f21bc0728b0 0x7f21bc0b1cc0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f21bc0d1060 tx=0x7f21a401c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf74f31930372373 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:05:30.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:30.995+0000 7f21bb400640 1 -- 172.21.15.16:0/2829818802 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21a40b0070 con 0x7f21bc0728b0 2024-03-31T15:05:30.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:30.995+0000 7f21bb400640 1 -- 172.21.15.16:0/2829818802 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f21a40ab600 con 0x7f21bc0728b0 2024-03-31T15:05:30.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:30.995+0000 7f21bb400640 1 -- 172.21.15.16:0/2829818802 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21a40ab9d0 con 0x7f21bc0728b0 2024-03-31T15:05:30.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:30.995+0000 7f21c2e00640 1 -- 172.21.15.16:0/2829818802 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f21bc0728b0 msgr2=0x7f21bc0b1cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:30.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:30.995+0000 7f21c2e00640 1 --2- 172.21.15.16:0/2829818802 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f21bc0728b0 0x7f21bc0b1cc0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f21bc0d1060 tx=0x7f21a401c520 comp rx=0 tx=0).stop 2024-03-31T15:05:30.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:30.995+0000 7f21c2e00640 1 -- 172.21.15.16:0/2829818802 shutdown_connections 2024-03-31T15:05:30.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:30.995+0000 7f21c2e00640 1 --2- 172.21.15.16:0/2829818802 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f21bc0b2200 0x7f21bc15cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:30.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:30.995+0000 7f21c2e00640 1 --2- 172.21.15.16:0/2829818802 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f21bc0728b0 0x7f21bc0b1cc0 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:30.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:30.995+0000 7f21c2e00640 1 --2- 172.21.15.16:0/2829818802 >> v2:172.21.15.89:3300/0 conn(0x7f21bc145d10 0x7f21bc0cf790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:30.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:30.995+0000 7f21c2e00640 1 -- 172.21.15.16:0/2829818802 >> 172.21.15.16:0/2829818802 conn(0x7f21bc06b620 msgr2=0x7f21bc06ba30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:05:30.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:30.995+0000 7f21c2e00640 1 -- 172.21.15.16:0/2829818802 shutdown_connections 2024-03-31T15:05:30.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:30.995+0000 7f21c2e00640 1 -- 172.21.15.16:0/2829818802 wait complete. 2024-03-31T15:05:30.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:30.999+0000 7f21c2e00640 1 Processor -- start 2024-03-31T15:05:30.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:30.999+0000 7f21c2e00640 1 -- start start 2024-03-31T15:05:30.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:30.999+0000 7f21c2e00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f21bc0728b0 0x7f21bc15aff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:30.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:30.999+0000 7f21c2e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f21bc0b2200 0x7f21bc158f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:30.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:30.999+0000 7f21c2e00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f21bc145d10 0x7f21bc15db50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:30.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:30.999+0000 7f21c2e00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f21bc0cf350 con 0x7f21bc0b2200 2024-03-31T15:05:30.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:30.999+0000 7f21c2e00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f21bc0b2610 con 0x7f21bc145d10 2024-03-31T15:05:30.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:30.999+0000 7f21c2e00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f21bc0cf4d0 con 0x7f21bc0728b0 2024-03-31T15:05:30.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:30.999+0000 7f21b3400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f21bc0b2200 0x7f21bc158f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:30.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:30.999+0000 7f21b3400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f21bc0b2200 0x7f21bc158f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34166/0 (socket says 172.21.15.16:34166) 2024-03-31T15:05:30.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:30.999+0000 7f21b3400640 1 -- 172.21.15.16:0/2946255170 learned_addr learned my addr 172.21.15.16:0/2946255170 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:05:30.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:30.999+0000 7f21bbe00640 1 --2- 172.21.15.16:0/2946255170 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f21bc0728b0 0x7f21bc15aff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:30.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:30.999+0000 7f21c0e00640 1 --2- 172.21.15.16:0/2946255170 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f21bc145d10 0x7f21bc15db50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:30.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:30.999+0000 7f21b3400640 1 -- 172.21.15.16:0/2946255170 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f21bc0728b0 msgr2=0x7f21bc15aff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:30.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:30.999+0000 7f21b3400640 1 --2- 172.21.15.16:0/2946255170 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f21bc0728b0 0x7f21bc15aff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:30.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:30.999+0000 7f21b3400640 1 -- 172.21.15.16:0/2946255170 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f21bc145d10 msgr2=0x7f21bc15db50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:30.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:30.999+0000 7f21b3400640 1 --2- 172.21.15.16:0/2946255170 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f21bc145d10 0x7f21bc15db50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:30.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:30.999+0000 7f21b3400640 1 -- 172.21.15.16:0/2946255170 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f21bc15eaf0 con 0x7f21bc0b2200 2024-03-31T15:05:30.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:30.999+0000 7f21bbe00640 1 --2- 172.21.15.16:0/2946255170 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f21bc0728b0 0x7f21bc15aff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:05:30.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:30.999+0000 7f21b3400640 1 --2- 172.21.15.16:0/2946255170 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f21bc0b2200 0x7f21bc158f30 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f21a40ae040 tx=0x7f21a40192c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:05:30.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:30.999+0000 7f21b8c00640 1 -- 172.21.15.16:0/2946255170 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21a40b0040 con 0x7f21bc0b2200 2024-03-31T15:05:30.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:30.999+0000 7f21b8c00640 1 -- 172.21.15.16:0/2946255170 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f21a40b4620 con 0x7f21bc0b2200 2024-03-31T15:05:30.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:30.999+0000 7f21b8c00640 1 -- 172.21.15.16:0/2946255170 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21a40b4a10 con 0x7f21bc0b2200 2024-03-31T15:05:30.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:30.999+0000 7f21c2e00640 1 -- 172.21.15.16:0/2946255170 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f21bc15e530 con 0x7f21bc0b2200 2024-03-31T15:05:30.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:30.999+0000 7f21c2e00640 1 -- 172.21.15.16:0/2946255170 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f21bc141e40 con 0x7f21bc0b2200 2024-03-31T15:05:30.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:30.999+0000 7f21c2e00640 1 -- 172.21.15.16:0/2946255170 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f21bc069950 con 0x7f21bc0b2200 2024-03-31T15:05:30.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:31.003+0000 7f21b8c00640 1 -- 172.21.15.16:0/2946255170 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f21a40ab4c0 con 0x7f21bc0b2200 2024-03-31T15:05:30.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:31.003+0000 7f21b8c00640 1 --2- 172.21.15.16:0/2946255170 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f2198068380 0x7f219806a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:30.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:31.003+0000 7f21b8c00640 1 -- 172.21.15.16:0/2946255170 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5709+0+0 (secure 0 0 0) 0x7f21a4120560 con 0x7f21bc0b2200 2024-03-31T15:05:30.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:31.003+0000 7f21bbe00640 1 --2- 172.21.15.16:0/2946255170 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f2198068380 0x7f219806a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:30.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:31.003+0000 7f21bbe00640 1 --2- 172.21.15.16:0/2946255170 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f2198068380 0x7f219806a840 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f21bc0b4d40 tx=0x7f21ac008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:05:31.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:31.007+0000 7f21b8c00640 1 -- 172.21.15.16:0/2946255170 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f21a4016110 con 0x7f21bc0b2200 2024-03-31T15:05:31.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:31.171+0000 7f21c2e00640 1 -- 172.21.15.16:0/2946255170 --> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f21bc073a60 con 0x7f2198068380 2024-03-31T15:05:31.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:31.179+0000 7f21b8c00640 1 -- 172.21.15.16:0/2946255170 <== mgr.4100 v2:172.21.15.89:6832/1549407273 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251915 (secure 0 0 0) 0x7f21bc073a60 con 0x7f2198068380 2024-03-31T15:05:31.177 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:05:31.179 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-03-31T15:05:31.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:31.183+0000 7f21c2e00640 1 -- 172.21.15.16:0/2946255170 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f2198068380 msgr2=0x7f219806a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:31.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:31.183+0000 7f21c2e00640 1 --2- 172.21.15.16:0/2946255170 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f2198068380 0x7f219806a840 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f21bc0b4d40 tx=0x7f21ac008040 comp rx=0 tx=0).stop 2024-03-31T15:05:31.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:31.183+0000 7f21c2e00640 1 -- 172.21.15.16:0/2946255170 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f21bc0b2200 msgr2=0x7f21bc158f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:31.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:31.183+0000 7f21c2e00640 1 --2- 172.21.15.16:0/2946255170 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f21bc0b2200 0x7f21bc158f30 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f21a40ae040 tx=0x7f21a40192c0 comp rx=0 tx=0).stop 2024-03-31T15:05:31.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:31.183+0000 7f21c2e00640 1 -- 172.21.15.16:0/2946255170 shutdown_connections 2024-03-31T15:05:31.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:31.183+0000 7f21c2e00640 1 --2- 172.21.15.16:0/2946255170 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f2198068380 0x7f219806a840 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:31.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:31.183+0000 7f21c2e00640 1 --2- 172.21.15.16:0/2946255170 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f21bc145d10 0x7f21bc15db50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:31.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:31.183+0000 7f21c2e00640 1 --2- 172.21.15.16:0/2946255170 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f21bc0b2200 0x7f21bc158f30 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:31.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:31.183+0000 7f21c2e00640 1 --2- 172.21.15.16:0/2946255170 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f21bc0728b0 0x7f21bc15aff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:31.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:31.183+0000 7f21c2e00640 1 -- 172.21.15.16:0/2946255170 >> 172.21.15.16:0/2946255170 conn(0x7f21bc06b620 msgr2=0x7f21bc0cea40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:05:31.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:31.183+0000 7f21c2e00640 1 -- 172.21.15.16:0/2946255170 shutdown_connections 2024-03-31T15:05:31.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:31.183+0000 7f21c2e00640 1 -- 172.21.15.16:0/2946255170 wait complete. 2024-03-31T15:05:31.198 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":102,"stamp":"2024-03-31T15:05:30.553448+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":85,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":242208,"kb_used_data":5228,"kb_used_omap":12,"kb_used_meta":236851,"kb_avail":754732512,"statfs":{"total":773094113280,"available":772846092288,"internally_reserved":0,"allocated":5353472,"data_stored":2848144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":242535984},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.500967"},"pg_stats":[{"pgid":"7.3f","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-31T15:05:28.089715+0000","last_change":"2024-03-31T15:05:27.085969+0000","last_active":"2024-03-31T15:05:28.089715+0000","last_peered":"2024-03-31T15:05:28.089715+0000","last_clean":"2024-03-31T15:05:28.089715+0000","last_became_active":"2024-03-31T15:05:27.084521+0000","last_became_peered":"2024-03-31T15:05:27.084521+0000","last_unstale":"2024-03-31T15:05:28.089715+0000","last_undegraded":"2024-03-31T15:05:28.089715+0000","last_fullsized":"2024-03-31T15:05:28.089715+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_clean_scrub_stamp":"2024-03-31T15:05:26.050987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T19:01:25.453513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3e","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-31T15:05:28.164243+0000","last_change":"2024-03-31T15:05:27.086338+0000","last_active":"2024-03-31T15:05:28.164243+0000","last_peered":"2024-03-31T15:05:28.164243+0000","last_clean":"2024-03-31T15:05:28.164243+0000","last_became_active":"2024-03-31T15:05:27.086068+0000","last_became_peered":"2024-03-31T15:05:27.086068+0000","last_unstale":"2024-03-31T15:05:28.164243+0000","last_undegraded":"2024-03-31T15:05:28.164243+0000","last_fullsized":"2024-03-31T15:05:28.164243+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_clean_scrub_stamp":"2024-03-31T15:05:26.050987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:42:42.805582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3d","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-31T15:05:28.092760+0000","last_change":"2024-03-31T15:05:27.078884+0000","last_active":"2024-03-31T15:05:28.092760+0000","last_peered":"2024-03-31T15:05:28.092760+0000","last_clean":"2024-03-31T15:05:28.092760+0000","last_became_active":"2024-03-31T15:05:27.078573+0000","last_became_peered":"2024-03-31T15:05:27.078573+0000","last_unstale":"2024-03-31T15:05:28.092760+0000","last_undegraded":"2024-03-31T15:05:28.092760+0000","last_fullsized":"2024-03-31T15:05:28.092760+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_clean_scrub_stamp":"2024-03-31T15:05:26.050987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:10:37.950151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.3c","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-31T15:05:27.090081+0000","last_change":"2024-03-31T15:05:27.081269+0000","last_active":"2024-03-31T15:05:27.090081+0000","last_peered":"2024-03-31T15:05:27.090081+0000","last_clean":"2024-03-31T15:05:27.090081+0000","last_became_active":"2024-03-31T15:05:27.081033+0000","last_became_peered":"2024-03-31T15:05:27.081033+0000","last_unstale":"2024-03-31T15:05:27.090081+0000","last_undegraded":"2024-03-31T15:05:27.090081+0000","last_fullsized":"2024-03-31T15:05:27.090081+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_clean_scrub_stamp":"2024-03-31T15:05:26.050987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T16:04:24.129485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3b","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-31T15:05:28.163981+0000","last_change":"2024-03-31T15:05:27.085289+0000","last_active":"2024-03-31T15:05:28.163981+0000","last_peered":"2024-03-31T15:05:28.163981+0000","last_clean":"2024-03-31T15:05:28.163981+0000","last_became_active":"2024-03-31T15:05:27.085017+0000","last_became_peered":"2024-03-31T15:05:27.085017+0000","last_unstale":"2024-03-31T15:05:28.163981+0000","last_undegraded":"2024-03-31T15:05:28.163981+0000","last_fullsized":"2024-03-31T15:05:28.163981+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_clean_scrub_stamp":"2024-03-31T15:05:26.050987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:35:33.976512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3a","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-31T15:05:27.089535+0000","last_change":"2024-03-31T15:05:27.084961+0000","last_active":"2024-03-31T15:05:27.089535+0000","last_peered":"2024-03-31T15:05:27.089535+0000","last_clean":"2024-03-31T15:05:27.089535+0000","last_became_active":"2024-03-31T15:05:27.084733+0000","last_became_peered":"2024-03-31T15:05:27.084733+0000","last_unstale":"2024-03-31T15:05:27.089535+0000","last_undegraded":"2024-03-31T15:05:27.089535+0000","last_fullsized":"2024-03-31T15:05:27.089535+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_clean_scrub_stamp":"2024-03-31T15:05:26.050987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:26:46.240631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.39","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-31T15:05:28.089324+0000","last_change":"2024-03-31T15:05:27.092960+0000","last_active":"2024-03-31T15:05:28.089324+0000","last_peered":"2024-03-31T15:05:28.089324+0000","last_clean":"2024-03-31T15:05:28.089324+0000","last_became_active":"2024-03-31T15:05:27.092668+0000","last_became_peered":"2024-03-31T15:05:27.092668+0000","last_unstale":"2024-03-31T15:05:28.089324+0000","last_undegraded":"2024-03-31T15:05:28.089324+0000","last_fullsized":"2024-03-31T15:05:28.089324+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_clean_scrub_stamp":"2024-03-31T15:05:26.050987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T19:42:20.460194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.38","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-31T15:05:27.086950+0000","last_change":"2024-03-31T15:05:27.076883+0000","last_active":"2024-03-31T15:05:27.086950+0000","last_peered":"2024-03-31T15:05:27.086950+0000","last_clean":"2024-03-31T15:05:27.086950+0000","last_became_active":"2024-03-31T15:05:27.075582+0000","last_became_peered":"2024-03-31T15:05:27.075582+0000","last_unstale":"2024-03-31T15:05:27.086950+0000","last_undegraded":"2024-03-31T15:05:27.086950+0000","last_fullsized":"2024-03-31T15:05:27.086950+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_clean_scrub_stamp":"2024-03-31T15:05:26.050987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T17:05:23.258342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.37","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-31T15:05:27.090266+0000","last_change":"2024-03-31T15:05:27.083170+0000","last_active":"2024-03-31T15:05:27.090266+0000","last_peered":"2024-03-31T15:05:27.090266+0000","last_clean":"2024-03-31T15:05:27.090266+0000","last_became_active":"2024-03-31T15:05:27.082904+0000","last_became_peered":"2024-03-31T15:05:27.082904+0000","last_unstale":"2024-03-31T15:05:27.090266+0000","last_undegraded":"2024-03-31T15:05:27.090266+0000","last_fullsized":"2024-03-31T15:05:27.090266+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_clean_scrub_stamp":"2024-03-31T15:05:26.050987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T19:10:54.973176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.36","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-31T15:05:27.090213+0000","last_change":"2024-03-31T15:05:27.078623+0000","last_active":"2024-03-31T15:05:27.090213+0000","last_peered":"2024-03-31T15:05:27.090213+0000","last_clean":"2024-03-31T15:05:27.090213+0000","last_became_active":"2024-03-31T15:05:27.078340+0000","last_became_peered":"2024-03-31T15:05:27.078340+0000","last_unstale":"2024-03-31T15:05:27.090213+0000","last_undegraded":"2024-03-31T15:05:27.090213+0000","last_fullsized":"2024-03-31T15:05:27.090213+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_clean_scrub_stamp":"2024-03-31T15:05:26.050987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T15:48:23.844852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.35","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-31T15:05:27.082725+0000","last_change":"2024-03-31T15:05:27.075031+0000","last_active":"2024-03-31T15:05:27.082725+0000","last_peered":"2024-03-31T15:05:27.082725+0000","last_clean":"2024-03-31T15:05:27.082725+0000","last_became_active":"2024-03-31T15:05:27.074747+0000","last_became_peered":"2024-03-31T15:05:27.074747+0000","last_unstale":"2024-03-31T15:05:27.082725+0000","last_undegraded":"2024-03-31T15:05:27.082725+0000","last_fullsized":"2024-03-31T15:05:27.082725+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_clean_scrub_stamp":"2024-03-31T15:05:26.050987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:48:24.121742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.34","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-31T15:05:28.164095+0000","last_change":"2024-03-31T15:05:27.085453+0000","last_active":"2024-03-31T15:05:28.164095+0000","last_peered":"2024-03-31T15:05:28.164095+0000","last_clean":"2024-03-31T15:05:28.164095+0000","last_became_active":"2024-03-31T15:05:27.082607+0000","last_became_peered":"2024-03-31T15:05:27.082607+0000","last_unstale":"2024-03-31T15:05:28.164095+0000","last_undegraded":"2024-03-31T15:05:28.164095+0000","last_fullsized":"2024-03-31T15:05:28.164095+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_clean_scrub_stamp":"2024-03-31T15:05:26.050987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:51:01.214558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.33","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-31T15:05:28.162637+0000","last_change":"2024-03-31T15:05:27.078631+0000","last_active":"2024-03-31T15:05:28.162637+0000","last_peered":"2024-03-31T15:05:28.162637+0000","last_clean":"2024-03-31T15:05:28.162637+0000","last_became_active":"2024-03-31T15:05:27.078338+0000","last_became_peered":"2024-03-31T15:05:27.078338+0000","last_unstale":"2024-03-31T15:05:28.162637+0000","last_undegraded":"2024-03-31T15:05:28.162637+0000","last_fullsized":"2024-03-31T15:05:28.162637+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_clean_scrub_stamp":"2024-03-31T15:05:26.050987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:04:12.442076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.32","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-31T15:05:27.090036+0000","last_change":"2024-03-31T15:05:27.078798+0000","last_active":"2024-03-31T15:05:27.090036+0000","last_peered":"2024-03-31T15:05:27.090036+0000","last_clean":"2024-03-31T15:05:27.090036+0000","last_became_active":"2024-03-31T15:05:27.078531+0000","last_became_peered":"2024-03-31T15:05:27.078531+0000","last_unstale":"2024-03-31T15:05:27.090036+0000","last_undegraded":"2024-03-31T15:05:27.090036+0000","last_fullsized":"2024-03-31T15:05:27.090036+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_clean_scrub_stamp":"2024-03-31T15:05:26.050987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:29:56.093449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.31","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-31T15:05:27.087091+0000","last_change":"2024-03-31T15:05:27.079236+0000","last_active":"2024-03-31T15:05:27.087091+0000","last_peered":"2024-03-31T15:05:27.087091+0000","last_clean":"2024-03-31T15:05:27.087091+0000","last_became_active":"2024-03-31T15:05:27.078860+0000","last_became_peered":"2024-03-31T15:05:27.078860+0000","last_unstale":"2024-03-31T15:05:27.087091+0000","last_undegraded":"2024-03-31T15:05:27.087091+0000","last_fullsized":"2024-03-31T15:05:27.087091+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_clean_scrub_stamp":"2024-03-31T15:05:26.050987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:29:27.366849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.30","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-31T15:05:27.096822+0000","last_change":"2024-03-31T15:05:27.088665+0000","last_active":"2024-03-31T15:05:27.096822+0000","last_peered":"2024-03-31T15:05:27.096822+0000","last_clean":"2024-03-31T15:05:27.096822+0000","last_became_active":"2024-03-31T15:05:27.088261+0000","last_became_peered":"2024-03-31T15:05:27.088261+0000","last_unstale":"2024-03-31T15:05:27.096822+0000","last_undegraded":"2024-03-31T15:05:27.096822+0000","last_fullsized":"2024-03-31T15:05:27.096822+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_clean_scrub_stamp":"2024-03-31T15:05:26.050987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:48:24.121742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2f","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-31T15:05:28.092830+0000","last_change":"2024-03-31T15:05:27.085866+0000","last_active":"2024-03-31T15:05:28.092830+0000","last_peered":"2024-03-31T15:05:28.092830+0000","last_clean":"2024-03-31T15:05:28.092830+0000","last_became_active":"2024-03-31T15:05:27.085509+0000","last_became_peered":"2024-03-31T15:05:27.085509+0000","last_unstale":"2024-03-31T15:05:28.092830+0000","last_undegraded":"2024-03-31T15:05:28.092830+0000","last_fullsized":"2024-03-31T15:05:28.092830+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_clean_scrub_stamp":"2024-03-31T15:05:26.050987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T16:38:30.957973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2e","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-31T15:05:28.092939+0000","last_change":"2024-03-31T15:05:27.078203+0000","last_active":"2024-03-31T15:05:28.092939+0000","last_peered":"2024-03-31T15:05:28.092939+0000","last_clean":"2024-03-31T15:05:28.092939+0000","last_became_active":"2024-03-31T15:05:27.077911+0000","last_became_peered":"2024-03-31T15:05:27.077911+0000","last_unstale":"2024-03-31T15:05:28.092939+0000","last_undegraded":"2024-03-31T15:05:28.092939+0000","last_fullsized":"2024-03-31T15:05:28.092939+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_clean_scrub_stamp":"2024-03-31T15:05:26.050987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T15:41:40.292384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2d","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-31T15:05:27.095117+0000","last_change":"2024-03-31T15:05:27.088220+0000","last_active":"2024-03-31T15:05:27.095117+0000","last_peered":"2024-03-31T15:05:27.095117+0000","last_clean":"2024-03-31T15:05:27.095117+0000","last_became_active":"2024-03-31T15:05:27.087897+0000","last_became_peered":"2024-03-31T15:05:27.087897+0000","last_unstale":"2024-03-31T15:05:27.095117+0000","last_undegraded":"2024-03-31T15:05:27.095117+0000","last_fullsized":"2024-03-31T15:05:27.095117+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_clean_scrub_stamp":"2024-03-31T15:05:26.050987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:46:20.281767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2c","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-31T15:05:28.162895+0000","last_change":"2024-03-31T15:05:27.091595+0000","last_active":"2024-03-31T15:05:28.162895+0000","last_peered":"2024-03-31T15:05:28.162895+0000","last_clean":"2024-03-31T15:05:28.162895+0000","last_became_active":"2024-03-31T15:05:27.090968+0000","last_became_peered":"2024-03-31T15:05:27.090968+0000","last_unstale":"2024-03-31T15:05:28.162895+0000","last_undegraded":"2024-03-31T15:05:28.162895+0000","last_fullsized":"2024-03-31T15:05:28.162895+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_clean_scrub_stamp":"2024-03-31T15:05:26.050987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T16:28:52.354372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.2b","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-31T15:05:27.091904+0000","last_change":"2024-03-31T15:05:27.083227+0000","last_active":"2024-03-31T15:05:27.091904+0000","last_peered":"2024-03-31T15:05:27.091904+0000","last_clean":"2024-03-31T15:05:27.091904+0000","last_became_active":"2024-03-31T15:05:27.082759+0000","last_became_peered":"2024-03-31T15:05:27.082759+0000","last_unstale":"2024-03-31T15:05:27.091904+0000","last_undegraded":"2024-03-31T15:05:27.091904+0000","last_fullsized":"2024-03-31T15:05:27.091904+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_clean_scrub_stamp":"2024-03-31T15:05:26.050987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:25:02.179421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.2a","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-31T15:05:27.080744+0000","last_change":"2024-03-31T15:05:27.075150+0000","last_active":"2024-03-31T15:05:27.080744+0000","last_peered":"2024-03-31T15:05:27.080744+0000","last_clean":"2024-03-31T15:05:27.080744+0000","last_became_active":"2024-03-31T15:05:27.074725+0000","last_became_peered":"2024-03-31T15:05:27.074725+0000","last_unstale":"2024-03-31T15:05:27.080744+0000","last_undegraded":"2024-03-31T15:05:27.080744+0000","last_fullsized":"2024-03-31T15:05:27.080744+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_clean_scrub_stamp":"2024-03-31T15:05:26.050987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:15:33.311093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.29","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-31T15:05:27.087097+0000","last_change":"2024-03-31T15:05:27.080488+0000","last_active":"2024-03-31T15:05:27.087097+0000","last_peered":"2024-03-31T15:05:27.087097+0000","last_clean":"2024-03-31T15:05:27.087097+0000","last_became_active":"2024-03-31T15:05:27.080219+0000","last_became_peered":"2024-03-31T15:05:27.080219+0000","last_unstale":"2024-03-31T15:05:27.087097+0000","last_undegraded":"2024-03-31T15:05:27.087097+0000","last_fullsized":"2024-03-31T15:05:27.087097+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_clean_scrub_stamp":"2024-03-31T15:05:26.050987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:17:37.439361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.28","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-31T15:05:27.086807+0000","last_change":"2024-03-31T15:05:27.080424+0000","last_active":"2024-03-31T15:05:27.086807+0000","last_peered":"2024-03-31T15:05:27.086807+0000","last_clean":"2024-03-31T15:05:27.086807+0000","last_became_active":"2024-03-31T15:05:27.080116+0000","last_became_peered":"2024-03-31T15:05:27.080116+0000","last_unstale":"2024-03-31T15:05:27.086807+0000","last_undegraded":"2024-03-31T15:05:27.086807+0000","last_fullsized":"2024-03-31T15:05:27.086807+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_clean_scrub_stamp":"2024-03-31T15:05:26.050987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:23:19.302715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.27","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-31T15:05:27.087184+0000","last_change":"2024-03-31T15:05:27.081969+0000","last_active":"2024-03-31T15:05:27.087184+0000","last_peered":"2024-03-31T15:05:27.087184+0000","last_clean":"2024-03-31T15:05:27.087184+0000","last_became_active":"2024-03-31T15:05:27.081644+0000","last_became_peered":"2024-03-31T15:05:27.081644+0000","last_unstale":"2024-03-31T15:05:27.087184+0000","last_undegraded":"2024-03-31T15:05:27.087184+0000","last_fullsized":"2024-03-31T15:05:27.087184+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_clean_scrub_stamp":"2024-03-31T15:05:26.050987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:59:56.699974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.26","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-31T15:05:28.162863+0000","last_change":"2024-03-31T15:05:27.084818+0000","last_active":"2024-03-31T15:05:28.162863+0000","last_peered":"2024-03-31T15:05:28.162863+0000","last_clean":"2024-03-31T15:05:28.162863+0000","last_became_active":"2024-03-31T15:05:27.084498+0000","last_became_peered":"2024-03-31T15:05:27.084498+0000","last_unstale":"2024-03-31T15:05:28.162863+0000","last_undegraded":"2024-03-31T15:05:28.162863+0000","last_fullsized":"2024-03-31T15:05:28.162863+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_clean_scrub_stamp":"2024-03-31T15:05:26.050987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T17:32:20.502679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.25","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-31T15:05:28.092995+0000","last_change":"2024-03-31T15:05:27.086432+0000","last_active":"2024-03-31T15:05:28.092995+0000","last_peered":"2024-03-31T15:05:28.092995+0000","last_clean":"2024-03-31T15:05:28.092995+0000","last_became_active":"2024-03-31T15:05:27.086196+0000","last_became_peered":"2024-03-31T15:05:27.086196+0000","last_unstale":"2024-03-31T15:05:28.092995+0000","last_undegraded":"2024-03-31T15:05:28.092995+0000","last_fullsized":"2024-03-31T15:05:28.092995+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_clean_scrub_stamp":"2024-03-31T15:05:26.050987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T15:14:52.888307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.24","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-31T15:05:28.164161+0000","last_change":"2024-03-31T15:05:27.084714+0000","last_active":"2024-03-31T15:05:28.164161+0000","last_peered":"2024-03-31T15:05:28.164161+0000","last_clean":"2024-03-31T15:05:28.164161+0000","last_became_active":"2024-03-31T15:05:27.082275+0000","last_became_peered":"2024-03-31T15:05:27.082275+0000","last_unstale":"2024-03-31T15:05:28.164161+0000","last_undegraded":"2024-03-31T15:05:28.164161+0000","last_fullsized":"2024-03-31T15:05:28.164161+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_clean_scrub_stamp":"2024-03-31T15:05:26.050987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T19:42:20.460194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.23","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-31T15:05:27.083499+0000","last_change":"2024-03-31T15:05:27.078374+0000","last_active":"2024-03-31T15:05:27.083499+0000","last_peered":"2024-03-31T15:05:27.083499+0000","last_clean":"2024-03-31T15:05:27.083499+0000","last_became_active":"2024-03-31T15:05:27.078099+0000","last_became_peered":"2024-03-31T15:05:27.078099+0000","last_unstale":"2024-03-31T15:05:27.083499+0000","last_undegraded":"2024-03-31T15:05:27.083499+0000","last_fullsized":"2024-03-31T15:05:27.083499+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_clean_scrub_stamp":"2024-03-31T15:05:26.050987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T19:12:41.806694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.22","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-31T15:05:28.092636+0000","last_change":"2024-03-31T15:05:27.081299+0000","last_active":"2024-03-31T15:05:28.092636+0000","last_peered":"2024-03-31T15:05:28.092636+0000","last_clean":"2024-03-31T15:05:28.092636+0000","last_became_active":"2024-03-31T15:05:27.080969+0000","last_became_peered":"2024-03-31T15:05:27.080969+0000","last_unstale":"2024-03-31T15:05:28.092636+0000","last_undegraded":"2024-03-31T15:05:28.092636+0000","last_fullsized":"2024-03-31T15:05:28.092636+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_clean_scrub_stamp":"2024-03-31T15:05:26.050987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:22:40.920851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.21","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-31T15:05:27.090927+0000","last_change":"2024-03-31T15:05:27.088171+0000","last_active":"2024-03-31T15:05:27.090927+0000","last_peered":"2024-03-31T15:05:27.090927+0000","last_clean":"2024-03-31T15:05:27.090927+0000","last_became_active":"2024-03-31T15:05:27.086948+0000","last_became_peered":"2024-03-31T15:05:27.086948+0000","last_unstale":"2024-03-31T15:05:27.090927+0000","last_undegraded":"2024-03-31T15:05:27.090927+0000","last_fullsized":"2024-03-31T15:05:27.090927+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_clean_scrub_stamp":"2024-03-31T15:05:26.050987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T19:33:19.777309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.20","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-31T15:05:28.162277+0000","last_change":"2024-03-31T15:05:27.083002+0000","last_active":"2024-03-31T15:05:28.162277+0000","last_peered":"2024-03-31T15:05:28.162277+0000","last_clean":"2024-03-31T15:05:28.162277+0000","last_became_active":"2024-03-31T15:05:27.082714+0000","last_became_peered":"2024-03-31T15:05:27.082714+0000","last_unstale":"2024-03-31T15:05:28.162277+0000","last_undegraded":"2024-03-31T15:05:28.162277+0000","last_fullsized":"2024-03-31T15:05:28.162277+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_clean_scrub_stamp":"2024-03-31T15:05:26.050987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T19:51:20.669777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-31T15:05:27.090618+0000","last_change":"2024-03-31T15:05:27.080242+0000","last_active":"2024-03-31T15:05:27.090618+0000","last_peered":"2024-03-31T15:05:27.090618+0000","last_clean":"2024-03-31T15:05:27.090618+0000","last_became_active":"2024-03-31T15:05:27.079987+0000","last_became_peered":"2024-03-31T15:05:27.079987+0000","last_unstale":"2024-03-31T15:05:27.090618+0000","last_undegraded":"2024-03-31T15:05:27.090618+0000","last_fullsized":"2024-03-31T15:05:27.090618+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_clean_scrub_stamp":"2024-03-31T15:05:26.050987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T17:05:23.258342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"44'4","reported_seq":16,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-31T15:05:27.104225+0000","last_change":"2024-03-31T15:05:26.082021+0000","last_active":"2024-03-31T15:05:27.104225+0000","last_peered":"2024-03-31T15:05:27.104225+0000","last_clean":"2024-03-31T15:05:27.104225+0000","last_became_active":"2024-03-31T15:05:26.081723+0000","last_became_peered":"2024-03-31T15:05:26.081723+0000","last_unstale":"2024-03-31T15:05:27.104225+0000","last_undegraded":"2024-03-31T15:05:27.104225+0000","last_fullsized":"2024-03-31T15:05:27.104225+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:05:25.044753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:05:25.044753+0000","last_clean_scrub_stamp":"2024-03-31T15:05:25.044753+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:26:05.887334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1f","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-31T15:05:27.087158+0000","last_change":"2024-03-31T15:05:27.080022+0000","last_active":"2024-03-31T15:05:27.087158+0000","last_peered":"2024-03-31T15:05:27.087158+0000","last_clean":"2024-03-31T15:05:27.087158+0000","last_became_active":"2024-03-31T15:05:27.079359+0000","last_became_peered":"2024-03-31T15:05:27.079359+0000","last_unstale":"2024-03-31T15:05:27.087158+0000","last_undegraded":"2024-03-31T15:05:27.087158+0000","last_fullsized":"2024-03-31T15:05:27.087158+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_clean_scrub_stamp":"2024-03-31T15:05:26.050987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:04:33.953699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1e","version":"44'1","reported_seq":15,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-31T15:05:28.093076+0000","last_change":"2024-03-31T15:05:26.076816+0000","last_active":"2024-03-31T15:05:28.093076+0000","last_peered":"2024-03-31T15:05:28.093076+0000","last_clean":"2024-03-31T15:05:28.093076+0000","last_became_active":"2024-03-31T15:05:26.076308+0000","last_became_peered":"2024-03-31T15:05:26.076308+0000","last_unstale":"2024-03-31T15:05:28.093076+0000","last_undegraded":"2024-03-31T15:05:28.093076+0000","last_fullsized":"2024-03-31T15:05:28.093076+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:05:25.044753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:05:25.044753+0000","last_clean_scrub_stamp":"2024-03-31T15:05:25.044753+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:22:07.071870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-31T15:05:28.164275+0000","last_change":"2024-03-31T15:05:27.088970+0000","last_active":"2024-03-31T15:05:28.164275+0000","last_peered":"2024-03-31T15:05:28.164275+0000","last_clean":"2024-03-31T15:05:28.164275+0000","last_became_active":"2024-03-31T15:05:27.085485+0000","last_became_peered":"2024-03-31T15:05:27.085485+0000","last_unstale":"2024-03-31T15:05:28.164275+0000","last_undegraded":"2024-03-31T15:05:28.164275+0000","last_fullsized":"2024-03-31T15:05:28.164275+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_clean_scrub_stamp":"2024-03-31T15:05:26.050987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T15:53:54.088987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1d","version":"44'2","reported_seq":14,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-31T15:05:27.102151+0000","last_change":"2024-03-31T15:05:26.080274+0000","last_active":"2024-03-31T15:05:27.102151+0000","last_peered":"2024-03-31T15:05:27.102151+0000","last_clean":"2024-03-31T15:05:27.102151+0000","last_became_active":"2024-03-31T15:05:26.079983+0000","last_became_peered":"2024-03-31T15:05:26.079983+0000","last_unstale":"2024-03-31T15:05:27.102151+0000","last_undegraded":"2024-03-31T15:05:27.102151+0000","last_fullsized":"2024-03-31T15:05:27.102151+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:05:25.044753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:05:25.044753+0000","last_clean_scrub_stamp":"2024-03-31T15:05:25.044753+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:15:32.304859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-31T15:05:27.090536+0000","last_change":"2024-03-31T15:05:27.079753+0000","last_active":"2024-03-31T15:05:27.090536+0000","last_peered":"2024-03-31T15:05:27.090536+0000","last_clean":"2024-03-31T15:05:27.090536+0000","last_became_active":"2024-03-31T15:05:27.079480+0000","last_became_peered":"2024-03-31T15:05:27.079480+0000","last_unstale":"2024-03-31T15:05:27.090536+0000","last_undegraded":"2024-03-31T15:05:27.090536+0000","last_fullsized":"2024-03-31T15:05:27.090536+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_clean_scrub_stamp":"2024-03-31T15:05:26.050987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T16:16:23.687703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"44'2","reported_seq":16,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-31T15:05:28.089618+0000","last_change":"2024-03-31T15:05:26.079703+0000","last_active":"2024-03-31T15:05:28.089618+0000","last_peered":"2024-03-31T15:05:28.089618+0000","last_clean":"2024-03-31T15:05:28.089618+0000","last_became_active":"2024-03-31T15:05:26.079314+0000","last_became_peered":"2024-03-31T15:05:26.079314+0000","last_unstale":"2024-03-31T15:05:28.089618+0000","last_undegraded":"2024-03-31T15:05:28.089618+0000","last_fullsized":"2024-03-31T15:05:28.089618+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:05:25.044753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:05:25.044753+0000","last_clean_scrub_stamp":"2024-03-31T15:05:25.044753+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T15:41:39.286149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-31T15:05:28.092714+0000","last_change":"2024-03-31T15:05:27.075920+0000","last_active":"2024-03-31T15:05:28.092714+0000","last_peered":"2024-03-31T15:05:28.092714+0000","last_clean":"2024-03-31T15:05:28.092714+0000","last_became_active":"2024-03-31T15:05:27.075611+0000","last_became_peered":"2024-03-31T15:05:27.075611+0000","last_unstale":"2024-03-31T15:05:28.092714+0000","last_undegraded":"2024-03-31T15:05:28.092714+0000","last_fullsized":"2024-03-31T15:05:28.092714+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_clean_scrub_stamp":"2024-03-31T15:05:26.050987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T16:42:27.714002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"44'1","reported_seq":13,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-31T15:05:27.099175+0000","last_change":"2024-03-31T15:05:26.085749+0000","last_active":"2024-03-31T15:05:27.099175+0000","last_peered":"2024-03-31T15:05:27.099175+0000","last_clean":"2024-03-31T15:05:27.099175+0000","last_became_active":"2024-03-31T15:05:26.085260+0000","last_became_peered":"2024-03-31T15:05:26.085260+0000","last_unstale":"2024-03-31T15:05:27.099175+0000","last_undegraded":"2024-03-31T15:05:27.099175+0000","last_fullsized":"2024-03-31T15:05:27.099175+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:05:25.044753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:05:25.044753+0000","last_clean_scrub_stamp":"2024-03-31T15:05:25.044753+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:43:01.014270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1b","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-31T15:05:27.096183+0000","last_change":"2024-03-31T15:05:27.086020+0000","last_active":"2024-03-31T15:05:27.096183+0000","last_peered":"2024-03-31T15:05:27.096183+0000","last_clean":"2024-03-31T15:05:27.096183+0000","last_became_active":"2024-03-31T15:05:27.085701+0000","last_became_peered":"2024-03-31T15:05:27.085701+0000","last_unstale":"2024-03-31T15:05:27.096183+0000","last_undegraded":"2024-03-31T15:05:27.096183+0000","last_fullsized":"2024-03-31T15:05:27.096183+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_clean_scrub_stamp":"2024-03-31T15:05:26.050987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:16:34.982760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1a","version":"44'1","reported_seq":15,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-31T15:05:28.092787+0000","last_change":"2024-03-31T15:05:26.071315+0000","last_active":"2024-03-31T15:05:28.092787+0000","last_peered":"2024-03-31T15:05:28.092787+0000","last_clean":"2024-03-31T15:05:28.092787+0000","last_became_active":"2024-03-31T15:05:26.071008+0000","last_became_peered":"2024-03-31T15:05:26.071008+0000","last_unstale":"2024-03-31T15:05:28.092787+0000","last_undegraded":"2024-03-31T15:05:28.092787+0000","last_fullsized":"2024-03-31T15:05:28.092787+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:05:25.044753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:05:25.044753+0000","last_clean_scrub_stamp":"2024-03-31T15:05:25.044753+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:24:07.692704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-31T15:05:27.086926+0000","last_change":"2024-03-31T15:05:27.076679+0000","last_active":"2024-03-31T15:05:27.086926+0000","last_peered":"2024-03-31T15:05:27.086926+0000","last_clean":"2024-03-31T15:05:27.086926+0000","last_became_active":"2024-03-31T15:05:27.075493+0000","last_became_peered":"2024-03-31T15:05:27.075493+0000","last_unstale":"2024-03-31T15:05:27.086926+0000","last_undegraded":"2024-03-31T15:05:27.086926+0000","last_fullsized":"2024-03-31T15:05:27.086926+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_clean_scrub_stamp":"2024-03-31T15:05:26.050987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T15:10:57.166552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-31T15:05:28.163811+0000","last_change":"2024-03-31T15:05:26.077943+0000","last_active":"2024-03-31T15:05:28.163811+0000","last_peered":"2024-03-31T15:05:28.163811+0000","last_clean":"2024-03-31T15:05:28.163811+0000","last_became_active":"2024-03-31T15:05:26.077665+0000","last_became_peered":"2024-03-31T15:05:26.077665+0000","last_unstale":"2024-03-31T15:05:28.163811+0000","last_undegraded":"2024-03-31T15:05:28.163811+0000","last_fullsized":"2024-03-31T15:05:28.163811+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:05:25.044753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:05:25.044753+0000","last_clean_scrub_stamp":"2024-03-31T15:05:25.044753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T17:50:37.634805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-31T15:05:27.090007+0000","last_change":"2024-03-31T15:05:27.083246+0000","last_active":"2024-03-31T15:05:27.090007+0000","last_peered":"2024-03-31T15:05:27.090007+0000","last_clean":"2024-03-31T15:05:27.090007+0000","last_became_active":"2024-03-31T15:05:27.082988+0000","last_became_peered":"2024-03-31T15:05:27.082988+0000","last_unstale":"2024-03-31T15:05:27.090007+0000","last_undegraded":"2024-03-31T15:05:27.090007+0000","last_fullsized":"2024-03-31T15:05:27.090007+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_clean_scrub_stamp":"2024-03-31T15:05:26.050987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:18:34.558382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-31T15:05:27.064157+0000","last_change":"2024-03-31T15:05:26.078694+0000","last_active":"2024-03-31T15:05:27.064157+0000","last_peered":"2024-03-31T15:05:27.064157+0000","last_clean":"2024-03-31T15:05:27.064157+0000","last_became_active":"2024-03-31T15:05:26.076178+0000","last_became_peered":"2024-03-31T15:05:26.076178+0000","last_unstale":"2024-03-31T15:05:27.064157+0000","last_undegraded":"2024-03-31T15:05:27.064157+0000","last_fullsized":"2024-03-31T15:05:27.064157+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:05:25.044753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:05:25.044753+0000","last_clean_scrub_stamp":"2024-03-31T15:05:25.044753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:19:48.661247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-31T15:05:28.092937+0000","last_change":"2024-03-31T15:05:27.077997+0000","last_active":"2024-03-31T15:05:28.092937+0000","last_peered":"2024-03-31T15:05:28.092937+0000","last_clean":"2024-03-31T15:05:28.092937+0000","last_became_active":"2024-03-31T15:05:27.077615+0000","last_became_peered":"2024-03-31T15:05:27.077615+0000","last_unstale":"2024-03-31T15:05:28.092937+0000","last_undegraded":"2024-03-31T15:05:28.092937+0000","last_fullsized":"2024-03-31T15:05:28.092937+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_clean_scrub_stamp":"2024-03-31T15:05:26.050987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T17:09:12.798678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-31T15:05:28.093097+0000","last_change":"2024-03-31T15:05:26.076164+0000","last_active":"2024-03-31T15:05:28.093097+0000","last_peered":"2024-03-31T15:05:28.093097+0000","last_clean":"2024-03-31T15:05:28.093097+0000","last_became_active":"2024-03-31T15:05:26.075868+0000","last_became_peered":"2024-03-31T15:05:26.075868+0000","last_unstale":"2024-03-31T15:05:28.093097+0000","last_undegraded":"2024-03-31T15:05:28.093097+0000","last_fullsized":"2024-03-31T15:05:28.093097+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:05:25.044753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:05:25.044753+0000","last_clean_scrub_stamp":"2024-03-31T15:05:25.044753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:40:36.292782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-31T15:05:28.089795+0000","last_change":"2024-03-31T15:05:27.089434+0000","last_active":"2024-03-31T15:05:28.089795+0000","last_peered":"2024-03-31T15:05:28.089795+0000","last_clean":"2024-03-31T15:05:28.089795+0000","last_became_active":"2024-03-31T15:05:27.089097+0000","last_became_peered":"2024-03-31T15:05:27.089097+0000","last_unstale":"2024-03-31T15:05:28.089795+0000","last_undegraded":"2024-03-31T15:05:28.089795+0000","last_fullsized":"2024-03-31T15:05:28.089795+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_clean_scrub_stamp":"2024-03-31T15:05:26.050987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:45:25.819165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-31T15:05:27.065533+0000","last_change":"2024-03-31T15:05:26.085575+0000","last_active":"2024-03-31T15:05:27.065533+0000","last_peered":"2024-03-31T15:05:27.065533+0000","last_clean":"2024-03-31T15:05:27.065533+0000","last_became_active":"2024-03-31T15:05:26.085201+0000","last_became_peered":"2024-03-31T15:05:26.085201+0000","last_unstale":"2024-03-31T15:05:27.065533+0000","last_undegraded":"2024-03-31T15:05:27.065533+0000","last_fullsized":"2024-03-31T15:05:27.065533+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:05:25.044753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:05:25.044753+0000","last_clean_scrub_stamp":"2024-03-31T15:05:25.044753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:46:19.275533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.14","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-31T15:05:27.088949+0000","last_change":"2024-03-31T15:05:27.079895+0000","last_active":"2024-03-31T15:05:27.088949+0000","last_peered":"2024-03-31T15:05:27.088949+0000","last_clean":"2024-03-31T15:05:27.088949+0000","last_became_active":"2024-03-31T15:05:27.079538+0000","last_became_peered":"2024-03-31T15:05:27.079538+0000","last_unstale":"2024-03-31T15:05:27.088949+0000","last_undegraded":"2024-03-31T15:05:27.088949+0000","last_fullsized":"2024-03-31T15:05:27.088949+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_clean_scrub_stamp":"2024-03-31T15:05:26.050987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:03:20.812659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"44'3","reported_seq":17,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-31T15:05:28.089681+0000","last_change":"2024-03-31T15:05:26.080315+0000","last_active":"2024-03-31T15:05:28.089681+0000","last_peered":"2024-03-31T15:05:28.089681+0000","last_clean":"2024-03-31T15:05:28.089681+0000","last_became_active":"2024-03-31T15:05:26.079466+0000","last_became_peered":"2024-03-31T15:05:26.079466+0000","last_unstale":"2024-03-31T15:05:28.089681+0000","last_undegraded":"2024-03-31T15:05:28.089681+0000","last_fullsized":"2024-03-31T15:05:28.089681+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:05:25.044753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:05:25.044753+0000","last_clean_scrub_stamp":"2024-03-31T15:05:25.044753+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T16:38:29.951739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.15","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-31T15:05:28.089467+0000","last_change":"2024-03-31T15:05:27.091966+0000","last_active":"2024-03-31T15:05:28.089467+0000","last_peered":"2024-03-31T15:05:28.089467+0000","last_clean":"2024-03-31T15:05:28.089467+0000","last_became_active":"2024-03-31T15:05:27.091670+0000","last_became_peered":"2024-03-31T15:05:27.091670+0000","last_unstale":"2024-03-31T15:05:28.089467+0000","last_undegraded":"2024-03-31T15:05:28.089467+0000","last_fullsized":"2024-03-31T15:05:28.089467+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_clean_scrub_stamp":"2024-03-31T15:05:26.050987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:42:42.805582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.14","version":"44'6","reported_seq":20,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-31T15:05:28.162997+0000","last_change":"2024-03-31T15:05:26.084337+0000","last_active":"2024-03-31T15:05:28.162997+0000","last_peered":"2024-03-31T15:05:28.162997+0000","last_clean":"2024-03-31T15:05:28.162997+0000","last_became_active":"2024-03-31T15:05:26.084031+0000","last_became_peered":"2024-03-31T15:05:26.084031+0000","last_unstale":"2024-03-31T15:05:28.162997+0000","last_undegraded":"2024-03-31T15:05:28.162997+0000","last_fullsized":"2024-03-31T15:05:28.162997+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:05:25.044753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:05:25.044753+0000","last_clean_scrub_stamp":"2024-03-31T15:05:25.044753+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:13:05.524841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-31T15:05:27.095001+0000","last_change":"2024-03-31T15:05:27.087927+0000","last_active":"2024-03-31T15:05:27.095001+0000","last_peered":"2024-03-31T15:05:27.095001+0000","last_clean":"2024-03-31T15:05:27.095001+0000","last_became_active":"2024-03-31T15:05:27.087659+0000","last_became_peered":"2024-03-31T15:05:27.087659+0000","last_unstale":"2024-03-31T15:05:27.095001+0000","last_undegraded":"2024-03-31T15:05:27.095001+0000","last_fullsized":"2024-03-31T15:05:27.095001+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_clean_scrub_stamp":"2024-03-31T15:05:26.050987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T16:37:34.994306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"44'2","reported_seq":14,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-31T15:05:27.100366+0000","last_change":"2024-03-31T15:05:26.086976+0000","last_active":"2024-03-31T15:05:27.100366+0000","last_peered":"2024-03-31T15:05:27.100366+0000","last_clean":"2024-03-31T15:05:27.100366+0000","last_became_active":"2024-03-31T15:05:26.086764+0000","last_became_peered":"2024-03-31T15:05:26.086764+0000","last_unstale":"2024-03-31T15:05:27.100366+0000","last_undegraded":"2024-03-31T15:05:27.100366+0000","last_fullsized":"2024-03-31T15:05:27.100366+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:05:25.044753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:05:25.044753+0000","last_clean_scrub_stamp":"2024-03-31T15:05:25.044753+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T17:09:24.098955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-31T15:05:27.098760+0000","last_change":"2024-03-31T15:05:27.094181+0000","last_active":"2024-03-31T15:05:27.098760+0000","last_peered":"2024-03-31T15:05:27.098760+0000","last_clean":"2024-03-31T15:05:27.098760+0000","last_became_active":"2024-03-31T15:05:27.093442+0000","last_became_peered":"2024-03-31T15:05:27.093442+0000","last_unstale":"2024-03-31T15:05:27.098760+0000","last_undegraded":"2024-03-31T15:05:27.098760+0000","last_fullsized":"2024-03-31T15:05:27.098760+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_clean_scrub_stamp":"2024-03-31T15:05:26.050987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:56:24.208563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-31T15:05:28.089591+0000","last_change":"2024-03-31T15:05:26.081225+0000","last_active":"2024-03-31T15:05:28.089591+0000","last_peered":"2024-03-31T15:05:28.089591+0000","last_clean":"2024-03-31T15:05:28.089591+0000","last_became_active":"2024-03-31T15:05:26.080966+0000","last_became_peered":"2024-03-31T15:05:26.080966+0000","last_unstale":"2024-03-31T15:05:28.089591+0000","last_undegraded":"2024-03-31T15:05:28.089591+0000","last_fullsized":"2024-03-31T15:05:28.089591+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:05:25.044753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:05:25.044753+0000","last_clean_scrub_stamp":"2024-03-31T15:05:25.044753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:10:36.943916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.10","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-31T15:05:27.088591+0000","last_change":"2024-03-31T15:05:27.080688+0000","last_active":"2024-03-31T15:05:27.088591+0000","last_peered":"2024-03-31T15:05:27.088591+0000","last_clean":"2024-03-31T15:05:27.088591+0000","last_became_active":"2024-03-31T15:05:27.080415+0000","last_became_peered":"2024-03-31T15:05:27.080415+0000","last_unstale":"2024-03-31T15:05:27.088591+0000","last_undegraded":"2024-03-31T15:05:27.088591+0000","last_fullsized":"2024-03-31T15:05:27.088591+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_clean_scrub_stamp":"2024-03-31T15:05:26.050987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:26:04.100923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-31T15:05:27.073295+0000","last_change":"2024-03-31T15:05:26.083701+0000","last_active":"2024-03-31T15:05:27.073295+0000","last_peered":"2024-03-31T15:05:27.073295+0000","last_clean":"2024-03-31T15:05:27.073295+0000","last_became_active":"2024-03-31T15:05:26.080775+0000","last_became_peered":"2024-03-31T15:05:26.080775+0000","last_unstale":"2024-03-31T15:05:27.073295+0000","last_undegraded":"2024-03-31T15:05:27.073295+0000","last_fullsized":"2024-03-31T15:05:27.073295+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:05:25.044753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:05:25.044753+0000","last_clean_scrub_stamp":"2024-03-31T15:05:25.044753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T17:43:49.683910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-31T15:05:28.089475+0000","last_change":"2024-03-31T15:05:27.086375+0000","last_active":"2024-03-31T15:05:28.089475+0000","last_peered":"2024-03-31T15:05:28.089475+0000","last_clean":"2024-03-31T15:05:28.089475+0000","last_became_active":"2024-03-31T15:05:27.086051+0000","last_became_peered":"2024-03-31T15:05:27.086051+0000","last_unstale":"2024-03-31T15:05:28.089475+0000","last_undegraded":"2024-03-31T15:05:28.089475+0000","last_fullsized":"2024-03-31T15:05:28.089475+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_clean_scrub_stamp":"2024-03-31T15:05:26.050987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T15:53:54.088987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.10","version":"44'8","reported_seq":20,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-31T15:05:27.105302+0000","last_change":"2024-03-31T15:05:26.081850+0000","last_active":"2024-03-31T15:05:27.105302+0000","last_peered":"2024-03-31T15:05:27.105302+0000","last_clean":"2024-03-31T15:05:27.105302+0000","last_became_active":"2024-03-31T15:05:26.081484+0000","last_became_peered":"2024-03-31T15:05:26.081484+0000","last_unstale":"2024-03-31T15:05:27.105302+0000","last_undegraded":"2024-03-31T15:05:27.105302+0000","last_fullsized":"2024-03-31T15:05:27.105302+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:05:25.044753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:05:25.044753+0000","last_clean_scrub_stamp":"2024-03-31T15:05:25.044753+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:07:04.230530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.e","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-31T15:05:28.163012+0000","last_change":"2024-03-31T15:05:27.092324+0000","last_active":"2024-03-31T15:05:28.163012+0000","last_peered":"2024-03-31T15:05:28.163012+0000","last_clean":"2024-03-31T15:05:28.163012+0000","last_became_active":"2024-03-31T15:05:27.091992+0000","last_became_peered":"2024-03-31T15:05:27.091992+0000","last_unstale":"2024-03-31T15:05:28.163012+0000","last_undegraded":"2024-03-31T15:05:28.163012+0000","last_fullsized":"2024-03-31T15:05:28.163012+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_clean_scrub_stamp":"2024-03-31T15:05:26.050987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:34:07.231805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"44'1","reported_seq":15,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-31T15:05:28.162950+0000","last_change":"2024-03-31T15:05:26.080745+0000","last_active":"2024-03-31T15:05:28.162950+0000","last_peered":"2024-03-31T15:05:28.162950+0000","last_clean":"2024-03-31T15:05:28.162950+0000","last_became_active":"2024-03-31T15:05:26.080436+0000","last_became_peered":"2024-03-31T15:05:26.080436+0000","last_unstale":"2024-03-31T15:05:28.162950+0000","last_undegraded":"2024-03-31T15:05:28.162950+0000","last_fullsized":"2024-03-31T15:05:28.162950+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:05:25.044753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:05:25.044753+0000","last_clean_scrub_stamp":"2024-03-31T15:05:25.044753+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:50:07.343324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-31T15:05:27.095399+0000","last_change":"2024-03-31T15:05:27.086835+0000","last_active":"2024-03-31T15:05:27.095399+0000","last_peered":"2024-03-31T15:05:27.095399+0000","last_clean":"2024-03-31T15:05:27.095399+0000","last_became_active":"2024-03-31T15:05:27.086599+0000","last_became_peered":"2024-03-31T15:05:27.086599+0000","last_unstale":"2024-03-31T15:05:27.095399+0000","last_undegraded":"2024-03-31T15:05:27.095399+0000","last_fullsized":"2024-03-31T15:05:27.095399+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_clean_scrub_stamp":"2024-03-31T15:05:26.050987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:16:43.129582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.e","version":"44'2","reported_seq":16,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-31T15:05:28.093095+0000","last_change":"2024-03-31T15:05:26.076775+0000","last_active":"2024-03-31T15:05:28.093095+0000","last_peered":"2024-03-31T15:05:28.093095+0000","last_clean":"2024-03-31T15:05:28.093095+0000","last_became_active":"2024-03-31T15:05:26.075913+0000","last_became_peered":"2024-03-31T15:05:26.075913+0000","last_unstale":"2024-03-31T15:05:28.093095+0000","last_undegraded":"2024-03-31T15:05:28.093095+0000","last_fullsized":"2024-03-31T15:05:28.093095+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:05:25.044753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:05:25.044753+0000","last_clean_scrub_stamp":"2024-03-31T15:05:25.044753+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T15:23:12.818282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-31T15:05:27.090223+0000","last_change":"2024-03-31T15:05:27.083967+0000","last_active":"2024-03-31T15:05:27.090223+0000","last_peered":"2024-03-31T15:05:27.090223+0000","last_clean":"2024-03-31T15:05:27.090223+0000","last_became_active":"2024-03-31T15:05:27.083691+0000","last_became_peered":"2024-03-31T15:05:27.083691+0000","last_unstale":"2024-03-31T15:05:27.090223+0000","last_undegraded":"2024-03-31T15:05:27.090223+0000","last_fullsized":"2024-03-31T15:05:27.090223+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_clean_scrub_stamp":"2024-03-31T15:05:26.050987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:19:49.667481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.d","version":"44'1","reported_seq":15,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-31T15:05:28.164130+0000","last_change":"2024-03-31T15:05:26.078710+0000","last_active":"2024-03-31T15:05:28.164130+0000","last_peered":"2024-03-31T15:05:28.164130+0000","last_clean":"2024-03-31T15:05:28.164130+0000","last_became_active":"2024-03-31T15:05:26.078389+0000","last_became_peered":"2024-03-31T15:05:26.078389+0000","last_unstale":"2024-03-31T15:05:28.164130+0000","last_undegraded":"2024-03-31T15:05:28.164130+0000","last_fullsized":"2024-03-31T15:05:28.164130+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:05:25.044753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:05:25.044753+0000","last_clean_scrub_stamp":"2024-03-31T15:05:25.044753+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T16:42:26.707767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-31T15:05:27.083584+0000","last_change":"2024-03-31T15:05:27.078424+0000","last_active":"2024-03-31T15:05:27.083584+0000","last_peered":"2024-03-31T15:05:27.083584+0000","last_clean":"2024-03-31T15:05:27.083584+0000","last_became_active":"2024-03-31T15:05:27.078156+0000","last_became_peered":"2024-03-31T15:05:27.078156+0000","last_unstale":"2024-03-31T15:05:27.083584+0000","last_undegraded":"2024-03-31T15:05:27.083584+0000","last_fullsized":"2024-03-31T15:05:27.083584+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_clean_scrub_stamp":"2024-03-31T15:05:26.050987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T16:24:34.638128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-31T15:05:27.075688+0000","last_change":"2024-03-31T15:05:26.088880+0000","last_active":"2024-03-31T15:05:27.075688+0000","last_peered":"2024-03-31T15:05:27.075688+0000","last_clean":"2024-03-31T15:05:27.075688+0000","last_became_active":"2024-03-31T15:05:26.088608+0000","last_became_peered":"2024-03-31T15:05:26.088608+0000","last_unstale":"2024-03-31T15:05:27.075688+0000","last_undegraded":"2024-03-31T15:05:27.075688+0000","last_fullsized":"2024-03-31T15:05:27.075688+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:05:25.044753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:05:25.044753+0000","last_clean_scrub_stamp":"2024-03-31T15:05:25.044753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:52:22.169709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-31T15:05:27.087289+0000","last_change":"2024-03-31T15:05:27.082535+0000","last_active":"2024-03-31T15:05:27.087289+0000","last_peered":"2024-03-31T15:05:27.087289+0000","last_clean":"2024-03-31T15:05:27.087289+0000","last_became_active":"2024-03-31T15:05:27.082264+0000","last_became_peered":"2024-03-31T15:05:27.082264+0000","last_unstale":"2024-03-31T15:05:27.087289+0000","last_undegraded":"2024-03-31T15:05:27.087289+0000","last_fullsized":"2024-03-31T15:05:27.087289+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_clean_scrub_stamp":"2024-03-31T15:05:26.050987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:12:25.160714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.b","version":"44'2","reported_seq":14,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-31T15:05:27.101266+0000","last_change":"2024-03-31T15:05:26.085506+0000","last_active":"2024-03-31T15:05:27.101266+0000","last_peered":"2024-03-31T15:05:27.101266+0000","last_clean":"2024-03-31T15:05:27.101266+0000","last_became_active":"2024-03-31T15:05:26.084906+0000","last_became_peered":"2024-03-31T15:05:26.084906+0000","last_unstale":"2024-03-31T15:05:27.101266+0000","last_undegraded":"2024-03-31T15:05:27.101266+0000","last_fullsized":"2024-03-31T15:05:27.101266+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:05:25.044753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:05:25.044753+0000","last_clean_scrub_stamp":"2024-03-31T15:05:25.044753+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T19:15:58.627228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-31T15:05:27.091745+0000","last_change":"2024-03-31T15:05:27.085483+0000","last_active":"2024-03-31T15:05:27.091745+0000","last_peered":"2024-03-31T15:05:27.091745+0000","last_clean":"2024-03-31T15:05:27.091745+0000","last_became_active":"2024-03-31T15:05:27.085177+0000","last_became_peered":"2024-03-31T15:05:27.085177+0000","last_unstale":"2024-03-31T15:05:27.091745+0000","last_undegraded":"2024-03-31T15:05:27.091745+0000","last_fullsized":"2024-03-31T15:05:27.091745+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_clean_scrub_stamp":"2024-03-31T15:05:26.050987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T15:48:23.844852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-31T15:05:28.163901+0000","last_change":"2024-03-31T15:05:26.074149+0000","last_active":"2024-03-31T15:05:28.163901+0000","last_peered":"2024-03-31T15:05:28.163901+0000","last_clean":"2024-03-31T15:05:28.163901+0000","last_became_active":"2024-03-31T15:05:26.073847+0000","last_became_peered":"2024-03-31T15:05:26.073847+0000","last_unstale":"2024-03-31T15:05:28.163901+0000","last_undegraded":"2024-03-31T15:05:28.163901+0000","last_fullsized":"2024-03-31T15:05:28.163901+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:05:25.044753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:05:25.044753+0000","last_clean_scrub_stamp":"2024-03-31T15:05:25.044753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:26:16.679750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-31T15:05:28.162277+0000","last_change":"2024-03-31T15:05:27.081050+0000","last_active":"2024-03-31T15:05:28.162277+0000","last_peered":"2024-03-31T15:05:28.162277+0000","last_clean":"2024-03-31T15:05:28.162277+0000","last_became_active":"2024-03-31T15:05:27.080718+0000","last_became_peered":"2024-03-31T15:05:27.080718+0000","last_unstale":"2024-03-31T15:05:28.162277+0000","last_undegraded":"2024-03-31T15:05:28.162277+0000","last_fullsized":"2024-03-31T15:05:28.162277+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_clean_scrub_stamp":"2024-03-31T15:05:26.050987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:04:52.777130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-31T15:05:27.063860+0000","last_change":"2024-03-31T15:05:26.084529+0000","last_active":"2024-03-31T15:05:27.063860+0000","last_peered":"2024-03-31T15:05:27.063860+0000","last_clean":"2024-03-31T15:05:27.063860+0000","last_became_active":"2024-03-31T15:05:26.084245+0000","last_became_peered":"2024-03-31T15:05:26.084245+0000","last_unstale":"2024-03-31T15:05:27.063860+0000","last_undegraded":"2024-03-31T15:05:27.063860+0000","last_fullsized":"2024-03-31T15:05:27.063860+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:05:25.044753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:05:25.044753+0000","last_clean_scrub_stamp":"2024-03-31T15:05:25.044753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:18:33.552147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-31T15:05:27.080654+0000","last_change":"2024-03-31T15:05:27.076777+0000","last_active":"2024-03-31T15:05:27.080654+0000","last_peered":"2024-03-31T15:05:27.080654+0000","last_clean":"2024-03-31T15:05:27.080654+0000","last_became_active":"2024-03-31T15:05:27.076515+0000","last_became_peered":"2024-03-31T15:05:27.076515+0000","last_unstale":"2024-03-31T15:05:27.080654+0000","last_undegraded":"2024-03-31T15:05:27.080654+0000","last_fullsized":"2024-03-31T15:05:27.080654+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_clean_scrub_stamp":"2024-03-31T15:05:26.050987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T16:16:23.687703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-31T15:05:28.089018+0000","last_change":"2024-03-31T15:05:26.084900+0000","last_active":"2024-03-31T15:05:28.089018+0000","last_peered":"2024-03-31T15:05:28.089018+0000","last_clean":"2024-03-31T15:05:28.089018+0000","last_became_active":"2024-03-31T15:05:26.084487+0000","last_became_peered":"2024-03-31T15:05:26.084487+0000","last_unstale":"2024-03-31T15:05:28.089018+0000","last_undegraded":"2024-03-31T15:05:28.089018+0000","last_fullsized":"2024-03-31T15:05:28.089018+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:05:25.044753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:05:25.044753+0000","last_clean_scrub_stamp":"2024-03-31T15:05:25.044753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:22:39.914616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-31T15:05:27.090323+0000","last_change":"2024-03-31T15:05:27.084780+0000","last_active":"2024-03-31T15:05:27.090323+0000","last_peered":"2024-03-31T15:05:27.090323+0000","last_clean":"2024-03-31T15:05:27.090323+0000","last_became_active":"2024-03-31T15:05:27.084479+0000","last_became_peered":"2024-03-31T15:05:27.084479+0000","last_unstale":"2024-03-31T15:05:27.090323+0000","last_undegraded":"2024-03-31T15:05:27.090323+0000","last_fullsized":"2024-03-31T15:05:27.090323+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_clean_scrub_stamp":"2024-03-31T15:05:26.050987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:07:05.236765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-31T15:05:27.075356+0000","last_change":"2024-03-31T15:05:26.086483+0000","last_active":"2024-03-31T15:05:27.075356+0000","last_peered":"2024-03-31T15:05:27.075356+0000","last_clean":"2024-03-31T15:05:27.075356+0000","last_became_active":"2024-03-31T15:05:26.086208+0000","last_became_peered":"2024-03-31T15:05:26.086208+0000","last_unstale":"2024-03-31T15:05:27.075356+0000","last_undegraded":"2024-03-31T15:05:27.075356+0000","last_fullsized":"2024-03-31T15:05:27.075356+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:05:25.044753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:05:25.044753+0000","last_clean_scrub_stamp":"2024-03-31T15:05:25.044753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:23:55.245030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-31T15:05:27.090106+0000","last_change":"2024-03-31T15:05:27.081617+0000","last_active":"2024-03-31T15:05:27.090106+0000","last_peered":"2024-03-31T15:05:27.090106+0000","last_clean":"2024-03-31T15:05:27.090106+0000","last_became_active":"2024-03-31T15:05:27.081440+0000","last_became_peered":"2024-03-31T15:05:27.081440+0000","last_unstale":"2024-03-31T15:05:27.090106+0000","last_undegraded":"2024-03-31T15:05:27.090106+0000","last_fullsized":"2024-03-31T15:05:27.090106+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_clean_scrub_stamp":"2024-03-31T15:05:26.050987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T17:19:47.130905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-31T15:05:28.089771+0000","last_change":"2024-03-31T15:05:26.082448+0000","last_active":"2024-03-31T15:05:28.089771+0000","last_peered":"2024-03-31T15:05:28.089771+0000","last_clean":"2024-03-31T15:05:28.089771+0000","last_became_active":"2024-03-31T15:05:26.082046+0000","last_became_peered":"2024-03-31T15:05:26.082046+0000","last_unstale":"2024-03-31T15:05:28.089771+0000","last_undegraded":"2024-03-31T15:05:28.089771+0000","last_fullsized":"2024-03-31T15:05:28.089771+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:05:25.044753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:05:25.044753+0000","last_clean_scrub_stamp":"2024-03-31T15:05:25.044753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T17:09:11.792443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-31T15:05:28.089546+0000","last_change":"2024-03-31T15:05:27.093776+0000","last_active":"2024-03-31T15:05:28.089546+0000","last_peered":"2024-03-31T15:05:28.089546+0000","last_clean":"2024-03-31T15:05:28.089546+0000","last_became_active":"2024-03-31T15:05:27.093513+0000","last_became_peered":"2024-03-31T15:05:27.093513+0000","last_unstale":"2024-03-31T15:05:28.089546+0000","last_undegraded":"2024-03-31T15:05:28.089546+0000","last_fullsized":"2024-03-31T15:05:28.089546+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_clean_scrub_stamp":"2024-03-31T15:05:26.050987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T16:12:10.481371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-31T15:05:27.068619+0000","last_change":"2024-03-31T15:05:26.074576+0000","last_active":"2024-03-31T15:05:27.068619+0000","last_peered":"2024-03-31T15:05:27.068619+0000","last_clean":"2024-03-31T15:05:27.068619+0000","last_became_active":"2024-03-31T15:05:26.074220+0000","last_became_peered":"2024-03-31T15:05:26.074220+0000","last_unstale":"2024-03-31T15:05:27.068619+0000","last_undegraded":"2024-03-31T15:05:27.068619+0000","last_fullsized":"2024-03-31T15:05:27.068619+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:05:25.044753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:05:25.044753+0000","last_clean_scrub_stamp":"2024-03-31T15:05:25.044753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:48:23.115507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-31T15:05:27.094483+0000","last_change":"2024-03-31T15:05:27.092068+0000","last_active":"2024-03-31T15:05:27.094483+0000","last_peered":"2024-03-31T15:05:27.094483+0000","last_clean":"2024-03-31T15:05:27.094483+0000","last_became_active":"2024-03-31T15:05:27.091670+0000","last_became_peered":"2024-03-31T15:05:27.091670+0000","last_unstale":"2024-03-31T15:05:27.094483+0000","last_undegraded":"2024-03-31T15:05:27.094483+0000","last_fullsized":"2024-03-31T15:05:27.094483+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_clean_scrub_stamp":"2024-03-31T15:05:26.050987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:36:11.111173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-31T15:05:28.089604+0000","last_change":"2024-03-31T15:05:26.080545+0000","last_active":"2024-03-31T15:05:28.089604+0000","last_peered":"2024-03-31T15:05:28.089604+0000","last_clean":"2024-03-31T15:05:28.089604+0000","last_became_active":"2024-03-31T15:05:26.080246+0000","last_became_peered":"2024-03-31T15:05:26.080246+0000","last_unstale":"2024-03-31T15:05:28.089604+0000","last_undegraded":"2024-03-31T15:05:28.089604+0000","last_fullsized":"2024-03-31T15:05:28.089604+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:05:25.044753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:05:25.044753+0000","last_clean_scrub_stamp":"2024-03-31T15:05:25.044753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T16:42:26.707767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-31T15:05:27.091882+0000","last_change":"2024-03-31T15:05:27.082977+0000","last_active":"2024-03-31T15:05:27.091882+0000","last_peered":"2024-03-31T15:05:27.091882+0000","last_clean":"2024-03-31T15:05:27.091882+0000","last_became_active":"2024-03-31T15:05:27.082310+0000","last_became_peered":"2024-03-31T15:05:27.082310+0000","last_unstale":"2024-03-31T15:05:27.091882+0000","last_undegraded":"2024-03-31T15:05:27.091882+0000","last_fullsized":"2024-03-31T15:05:27.091882+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_clean_scrub_stamp":"2024-03-31T15:05:26.050987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T15:27:50.855100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-31T15:05:28.092866+0000","last_change":"2024-03-31T15:05:26.084927+0000","last_active":"2024-03-31T15:05:28.092866+0000","last_peered":"2024-03-31T15:05:28.092866+0000","last_clean":"2024-03-31T15:05:28.092866+0000","last_became_active":"2024-03-31T15:05:26.084374+0000","last_became_peered":"2024-03-31T15:05:26.084374+0000","last_unstale":"2024-03-31T15:05:28.092866+0000","last_undegraded":"2024-03-31T15:05:28.092866+0000","last_fullsized":"2024-03-31T15:05:28.092866+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:05:25.044753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:05:25.044753+0000","last_clean_scrub_stamp":"2024-03-31T15:05:25.044753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T19:56:56.774930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-31T15:05:28.163891+0000","last_change":"2024-03-31T15:05:27.084713+0000","last_active":"2024-03-31T15:05:28.163891+0000","last_peered":"2024-03-31T15:05:28.163891+0000","last_clean":"2024-03-31T15:05:28.163891+0000","last_became_active":"2024-03-31T15:05:27.082242+0000","last_became_peered":"2024-03-31T15:05:27.082242+0000","last_unstale":"2024-03-31T15:05:28.163891+0000","last_undegraded":"2024-03-31T15:05:28.163891+0000","last_fullsized":"2024-03-31T15:05:28.163891+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_clean_scrub_stamp":"2024-03-31T15:05:26.050987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:13:39.279396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-31T15:05:27.071927+0000","last_change":"2024-03-31T15:05:26.076418+0000","last_active":"2024-03-31T15:05:27.071927+0000","last_peered":"2024-03-31T15:05:27.071927+0000","last_clean":"2024-03-31T15:05:27.071927+0000","last_became_active":"2024-03-31T15:05:26.074431+0000","last_became_peered":"2024-03-31T15:05:26.074431+0000","last_unstale":"2024-03-31T15:05:27.071927+0000","last_undegraded":"2024-03-31T15:05:27.071927+0000","last_fullsized":"2024-03-31T15:05:27.071927+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:05:25.044753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:05:25.044753+0000","last_clean_scrub_stamp":"2024-03-31T15:05:25.044753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T15:10:56.160318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-31T15:05:28.092624+0000","last_change":"2024-03-31T15:05:27.085017+0000","last_active":"2024-03-31T15:05:28.092624+0000","last_peered":"2024-03-31T15:05:28.092624+0000","last_clean":"2024-03-31T15:05:28.092624+0000","last_became_active":"2024-03-31T15:05:27.084153+0000","last_became_peered":"2024-03-31T15:05:27.084153+0000","last_unstale":"2024-03-31T15:05:28.092624+0000","last_undegraded":"2024-03-31T15:05:28.092624+0000","last_fullsized":"2024-03-31T15:05:28.092624+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_clean_scrub_stamp":"2024-03-31T15:05:26.050987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T17:50:38.641040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-31T15:05:27.064494+0000","last_change":"2024-03-31T15:05:26.087612+0000","last_active":"2024-03-31T15:05:27.064494+0000","last_peered":"2024-03-31T15:05:27.064494+0000","last_clean":"2024-03-31T15:05:27.064494+0000","last_became_active":"2024-03-31T15:05:26.087282+0000","last_became_peered":"2024-03-31T15:05:26.087282+0000","last_unstale":"2024-03-31T15:05:27.064494+0000","last_undegraded":"2024-03-31T15:05:27.064494+0000","last_fullsized":"2024-03-31T15:05:27.064494+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:05:25.044753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:05:25.044753+0000","last_clean_scrub_stamp":"2024-03-31T15:05:25.044753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T16:37:33.988071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-31T15:05:27.094860+0000","last_change":"2024-03-31T15:05:27.088737+0000","last_active":"2024-03-31T15:05:27.094860+0000","last_peered":"2024-03-31T15:05:27.094860+0000","last_clean":"2024-03-31T15:05:27.094860+0000","last_became_active":"2024-03-31T15:05:27.088320+0000","last_became_peered":"2024-03-31T15:05:27.088320+0000","last_unstale":"2024-03-31T15:05:27.094860+0000","last_undegraded":"2024-03-31T15:05:27.094860+0000","last_fullsized":"2024-03-31T15:05:27.094860+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_clean_scrub_stamp":"2024-03-31T15:05:26.050987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:30:51.046353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.7","version":"44'2","reported_seq":16,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-31T15:05:28.164278+0000","last_change":"2024-03-31T15:05:26.082613+0000","last_active":"2024-03-31T15:05:28.164278+0000","last_peered":"2024-03-31T15:05:28.164278+0000","last_clean":"2024-03-31T15:05:28.164278+0000","last_became_active":"2024-03-31T15:05:26.082334+0000","last_became_peered":"2024-03-31T15:05:26.082334+0000","last_unstale":"2024-03-31T15:05:28.164278+0000","last_undegraded":"2024-03-31T15:05:28.164278+0000","last_fullsized":"2024-03-31T15:05:28.164278+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:05:25.044753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:05:25.044753+0000","last_clean_scrub_stamp":"2024-03-31T15:05:25.044753+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:22:39.914616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":215,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-31T15:05:28.089081+0000","last_change":"2024-03-31T15:03:24.329890+0000","last_active":"2024-03-31T15:05:28.089081+0000","last_peered":"2024-03-31T15:05:28.089081+0000","last_clean":"2024-03-31T15:05:28.089081+0000","last_became_active":"2024-03-31T15:03:24.328882+0000","last_became_peered":"2024-03-31T15:03:24.328882+0000","last_unstale":"2024-03-31T15:05:28.089081+0000","last_undegraded":"2024-03-31T15:05:28.089081+0000","last_fullsized":"2024-03-31T15:05:28.089081+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:22.885535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:22.885535+0000","last_clean_scrub_stamp":"2024-03-31T15:03:22.885535+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:18:29.004296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509470,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28120,"kb_used_data":1092,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343720,"statfs":{"total":96636764160,"available":96607969280,"internally_reserved":0,"allocated":1118208,"data_stored":802219,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509469,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27532,"kb_used_data":512,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344308,"statfs":{"total":96636764160,"available":96608571392,"internally_reserved":0,"allocated":524288,"data_stored":211831,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":15,"seq":64424509468,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27536,"kb_used_data":508,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344304,"statfs":{"total":96636764160,"available":96608567296,"internally_reserved":0,"allocated":520192,"data_stored":207073,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509467,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31704,"kb_used_data":524,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340136,"statfs":{"total":96636764160,"available":96604299264,"internally_reserved":0,"allocated":536576,"data_stored":213395,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542173,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31684,"kb_used_data":504,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340156,"statfs":{"total":96636764160,"available":96604319744,"internally_reserved":0,"allocated":516096,"data_stored":206133,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542173,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31688,"kb_used_data":508,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340152,"statfs":{"total":96636764160,"available":96604315648,"internally_reserved":0,"allocated":520192,"data_stored":206099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542173,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31680,"kb_used_data":500,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340160,"statfs":{"total":96636764160,"available":96604323840,"internally_reserved":0,"allocated":512000,"data_stored":205531,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542173,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32264,"kb_used_data":1080,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339576,"statfs":{"total":96636764160,"available":96603725824,"internally_reserved":0,"allocated":1105920,"data_stored":795863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-31T15:05:31.203 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-31T15:05:31.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:31.383+0000 7f48f8a00640 1 Processor -- start 2024-03-31T15:05:31.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:31.383+0000 7f48f8a00640 1 -- start start 2024-03-31T15:05:31.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:31.387+0000 7f48f8a00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f48f406f920 0x7f48f406fd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:31.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:31.387+0000 7f48f8a00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f48f40d2010 0x7f48f40d24a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:31.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:31.387+0000 7f48f8a00640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f48f40d0920 con 0x7f48f4145e40 2024-03-31T15:05:31.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:31.387+0000 7f48f8a00640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f48f40702c0 con 0x7f48f406f920 2024-03-31T15:05:31.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:31.387+0000 7f48f8a00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f48f40d06d0 con 0x7f48f40d2010 2024-03-31T15:05:31.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:31.387+0000 7f48f1e00640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f48f4145e40 0x7f48f40cc740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:35528/0 (socket says 172.21.15.16:35528) 2024-03-31T15:05:31.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:31.387+0000 7f48f1e00640 1 -- 172.21.15.16:0/4114992634 learned_addr learned my addr 172.21.15.16:0/4114992634 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:05:31.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:31.387+0000 7f48e9400640 1 --2- 172.21.15.16:0/4114992634 >> v2:172.21.15.89:3300/0 conn(0x7f48f406f920 0x7f48f406fd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:31.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:31.387+0000 7f48f2800640 1 --2- 172.21.15.16:0/4114992634 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f48f40d2010 0x7f48f40d24a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:31.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:31.387+0000 7f48f1400640 1 -- 172.21.15.16:0/4114992634 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3197935158 0 0) 0x7f48f40d0920 con 0x7f48f4145e40 2024-03-31T15:05:31.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:31.387+0000 7f48f1400640 1 -- 172.21.15.16:0/4114992634 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f48d8003530 con 0x7f48f4145e40 2024-03-31T15:05:31.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:31.387+0000 7f48f1400640 1 -- 172.21.15.16:0/4114992634 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3558393820 0 0) 0x7f48d8003530 con 0x7f48f4145e40 2024-03-31T15:05:31.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:31.387+0000 7f48f1400640 1 -- 172.21.15.16:0/4114992634 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f48f40d2010 msgr2=0x7f48f40d24a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:05:31.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:31.387+0000 7f48f1400640 1 --2- 172.21.15.16:0/4114992634 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f48f40d2010 0x7f48f40d24a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:31.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:31.387+0000 7f48f1400640 1 -- 172.21.15.16:0/4114992634 >> v2:172.21.15.89:3300/0 conn(0x7f48f406f920 msgr2=0x7f48f406fd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:05:31.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:31.387+0000 7f48f1400640 1 --2- 172.21.15.16:0/4114992634 >> v2:172.21.15.89:3300/0 conn(0x7f48f406f920 0x7f48f406fd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:31.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:31.387+0000 7f48f1400640 1 -- 172.21.15.16:0/4114992634 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48f40d0c00 con 0x7f48f4145e40 2024-03-31T15:05:31.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:31.387+0000 7f48f1400640 1 -- 172.21.15.16:0/4114992634 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 214807623 0 0) 0x7f48e4001680 con 0x7f48f4145e40 2024-03-31T15:05:31.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:31.387+0000 7f48f1400640 1 -- 172.21.15.16:0/4114992634 >> v1:172.21.15.16:6789/0 conn(0x7f48f4145e40 legacy=0x7f48f40cc740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:31.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:31.387+0000 7f48f1400640 1 --2- 172.21.15.16:0/4114992634 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f48d80039c0 0x7f48d8003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:31.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:31.387+0000 7f48f1400640 1 --2- 172.21.15.16:0/4114992634 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f48f40d2010 0x7f48d8004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:31.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:31.387+0000 7f48f1400640 1 --2- 172.21.15.16:0/4114992634 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f48d8004d50 0x7f48d80051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:31.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:31.387+0000 7f48f1400640 1 -- 172.21.15.16:0/4114992634 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f48f40d06d0 con 0x7f48d8004d50 2024-03-31T15:05:31.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:31.387+0000 7f48f1400640 1 -- 172.21.15.16:0/4114992634 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f48d800b9d0 con 0x7f48d80039c0 2024-03-31T15:05:31.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:31.387+0000 7f48f1400640 1 -- 172.21.15.16:0/4114992634 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f48d800bb50 con 0x7f48f40d2010 2024-03-31T15:05:31.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:31.387+0000 7f48f2800640 1 --2- 172.21.15.16:0/4114992634 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f48d8004d50 0x7f48d80051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:31.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:31.387+0000 7f48f1e00640 1 --2- 172.21.15.16:0/4114992634 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f48d80039c0 0x7f48d8003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:31.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:31.387+0000 7f48e9400640 1 --2- 172.21.15.16:0/4114992634 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f48f40d2010 0x7f48d8004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:31.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:31.387+0000 7f48f2800640 1 -- 172.21.15.16:0/4114992634 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f48f40d2010 msgr2=0x7f48d8004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:05:31.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:31.387+0000 7f48f2800640 1 --2- 172.21.15.16:0/4114992634 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f48f40d2010 0x7f48d8004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:31.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:31.387+0000 7f48f2800640 1 -- 172.21.15.16:0/4114992634 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f48d80039c0 msgr2=0x7f48d8003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:05:31.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:31.387+0000 7f48f2800640 1 --2- 172.21.15.16:0/4114992634 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f48d80039c0 0x7f48d8003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:31.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:31.387+0000 7f48f2800640 1 -- 172.21.15.16:0/4114992634 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48d8003530 con 0x7f48d8004d50 2024-03-31T15:05:31.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:31.387+0000 7f48e9400640 1 --2- 172.21.15.16:0/4114992634 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f48f40d2010 0x7f48d8004810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:05:31.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:31.391+0000 7f48f2800640 1 --2- 172.21.15.16:0/4114992634 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f48d8004d50 0x7f48d80051d0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f48ec00e170 tx=0x7f48ec01cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae5307f061131d06 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:05:31.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:31.391+0000 7f48f1400640 1 -- 172.21.15.16:0/4114992634 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48ec0b2070 con 0x7f48d8004d50 2024-03-31T15:05:31.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:31.391+0000 7f48f1400640 1 -- 172.21.15.16:0/4114992634 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f48ec0a5c70 con 0x7f48d8004d50 2024-03-31T15:05:31.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:31.391+0000 7f48f1400640 1 -- 172.21.15.16:0/4114992634 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48ec009c10 con 0x7f48d8004d50 2024-03-31T15:05:31.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:31.395+0000 7f48f8a00640 1 -- 172.21.15.16:0/4114992634 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f48d8004d50 msgr2=0x7f48d80051d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:31.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:31.395+0000 7f48f8a00640 1 --2- 172.21.15.16:0/4114992634 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f48d8004d50 0x7f48d80051d0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f48ec00e170 tx=0x7f48ec01cc60 comp rx=0 tx=0).stop 2024-03-31T15:05:31.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:31.395+0000 7f48f2800640 1 -- 172.21.15.16:0/4114992634 reap_dead start 2024-03-31T15:05:31.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:31.395+0000 7f48f8a00640 1 -- 172.21.15.16:0/4114992634 shutdown_connections 2024-03-31T15:05:31.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:31.395+0000 7f48f8a00640 1 -- 172.21.15.16:0/4114992634 >> 172.21.15.16:0/4114992634 conn(0x7f48f4067330 msgr2=0x7f48f4067740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:05:31.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:31.395+0000 7f48f8a00640 1 -- 172.21.15.16:0/4114992634 shutdown_connections 2024-03-31T15:05:31.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:31.395+0000 7f48f8a00640 1 -- 172.21.15.16:0/4114992634 wait complete. 2024-03-31T15:05:31.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:31.395+0000 7f48f8a00640 1 Processor -- start 2024-03-31T15:05:31.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:31.395+0000 7f48f8a00640 1 -- start start 2024-03-31T15:05:31.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:31.395+0000 7f48f8a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f48d8004d50 0x7f48f40c7030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:31.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:31.395+0000 7f48f8a00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f48f4142460 0x7f48f40b66b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:31.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:31.395+0000 7f48f8a00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f48f40b6bf0 0x7f48f415dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:31.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:31.395+0000 7f48f8a00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f48f40cce50 con 0x7f48d8004d50 2024-03-31T15:05:31.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:31.395+0000 7f48f8a00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f48f40d1af0 con 0x7f48f40b6bf0 2024-03-31T15:05:31.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:31.395+0000 7f48f8a00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f48f40ce9e0 con 0x7f48f4142460 2024-03-31T15:05:31.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:31.395+0000 7f48f1e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f48d8004d50 0x7f48f40c7030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:31.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:31.395+0000 7f48f1e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f48d8004d50 0x7f48f40c7030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34182/0 (socket says 172.21.15.16:34182) 2024-03-31T15:05:31.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:31.395+0000 7f48f1e00640 1 -- 172.21.15.16:0/4292639140 learned_addr learned my addr 172.21.15.16:0/4292639140 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:05:31.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:31.395+0000 7f48e9400640 1 --2- 172.21.15.16:0/4292639140 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f48f4142460 0x7f48f40b66b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:31.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:31.395+0000 7f48f2800640 1 --2- 172.21.15.16:0/4292639140 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f48f40b6bf0 0x7f48f415dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:31.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:31.395+0000 7f48f1e00640 1 -- 172.21.15.16:0/4292639140 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f48f4142460 msgr2=0x7f48f40b66b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:31.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:31.395+0000 7f48f1e00640 1 --2- 172.21.15.16:0/4292639140 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f48f4142460 0x7f48f40b66b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:31.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:31.395+0000 7f48f1e00640 1 -- 172.21.15.16:0/4292639140 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f48f40b6bf0 msgr2=0x7f48f415dd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:31.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:31.395+0000 7f48f1e00640 1 --2- 172.21.15.16:0/4292639140 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f48f40b6bf0 0x7f48f415dd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:31.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:31.395+0000 7f48f1e00640 1 -- 172.21.15.16:0/4292639140 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48f40c8670 con 0x7f48d8004d50 2024-03-31T15:05:31.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:31.399+0000 7f48f1e00640 1 --2- 172.21.15.16:0/4292639140 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f48d8004d50 0x7f48f40c7030 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f48e4007340 tx=0x7f48e4007740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:05:31.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:31.399+0000 7f48eaa00640 1 -- 172.21.15.16:0/4292639140 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48e400f960 con 0x7f48d8004d50 2024-03-31T15:05:31.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:31.399+0000 7f48eaa00640 1 -- 172.21.15.16:0/4292639140 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f48e4018070 con 0x7f48d8004d50 2024-03-31T15:05:31.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:31.399+0000 7f48f8a00640 1 -- 172.21.15.16:0/4292639140 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f48f40c8950 con 0x7f48d8004d50 2024-03-31T15:05:31.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:31.399+0000 7f48eaa00640 1 -- 172.21.15.16:0/4292639140 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48e400fd80 con 0x7f48d8004d50 2024-03-31T15:05:31.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:31.399+0000 7f48f8a00640 1 -- 172.21.15.16:0/4292639140 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f48f40c8f10 con 0x7f48d8004d50 2024-03-31T15:05:31.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:31.399+0000 7f48f8a00640 1 -- 172.21.15.16:0/4292639140 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f48f4069950 con 0x7f48d8004d50 2024-03-31T15:05:31.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:31.399+0000 7f48eaa00640 1 -- 172.21.15.16:0/4292639140 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f48e401c410 con 0x7f48d8004d50 2024-03-31T15:05:31.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:31.399+0000 7f48eaa00640 1 --2- 172.21.15.16:0/4292639140 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f48d00683b0 0x7f48d006a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:31.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:31.399+0000 7f48eaa00640 1 -- 172.21.15.16:0/4292639140 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5709+0+0 (secure 0 0 0) 0x7f48e4086570 con 0x7f48d8004d50 2024-03-31T15:05:31.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:31.399+0000 7f48e9400640 1 --2- 172.21.15.16:0/4292639140 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f48d00683b0 0x7f48d006a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:31.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:31.403+0000 7f48e9400640 1 --2- 172.21.15.16:0/4292639140 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f48d00683b0 0x7f48d006a870 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f48f415cc70 tx=0x7f48dc008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:05:31.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:31.403+0000 7f48eaa00640 1 -- 172.21.15.16:0/4292639140 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f48e4056b20 con 0x7f48d8004d50 2024-03-31T15:05:31.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:31.579+0000 7f48f8a00640 1 -- 172.21.15.16:0/4292639140 --> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f48f40b6fd0 con 0x7f48d00683b0 2024-03-31T15:05:31.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:31.587+0000 7f48eaa00640 1 -- 172.21.15.16:0/4292639140 <== mgr.4100 v2:172.21.15.89:6832/1549407273 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251915 (secure 0 0 0) 0x7f48f40b6fd0 con 0x7f48d00683b0 2024-03-31T15:05:31.583 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:05:31.585 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-03-31T15:05:31.586 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:31.587+0000 7f48f8a00640 1 -- 172.21.15.16:0/4292639140 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f48d00683b0 msgr2=0x7f48d006a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:31.586 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:31.587+0000 7f48f8a00640 1 --2- 172.21.15.16:0/4292639140 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f48d00683b0 0x7f48d006a870 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f48f415cc70 tx=0x7f48dc008040 comp rx=0 tx=0).stop 2024-03-31T15:05:31.586 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:31.587+0000 7f48f8a00640 1 -- 172.21.15.16:0/4292639140 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f48d8004d50 msgr2=0x7f48f40c7030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:31.586 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:31.587+0000 7f48f8a00640 1 --2- 172.21.15.16:0/4292639140 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f48d8004d50 0x7f48f40c7030 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f48e4007340 tx=0x7f48e4007740 comp rx=0 tx=0).stop 2024-03-31T15:05:31.586 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:31.591+0000 7f48f8a00640 1 -- 172.21.15.16:0/4292639140 shutdown_connections 2024-03-31T15:05:31.586 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:31.591+0000 7f48f8a00640 1 --2- 172.21.15.16:0/4292639140 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f48d00683b0 0x7f48d006a870 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:31.586 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:31.591+0000 7f48f8a00640 1 --2- 172.21.15.16:0/4292639140 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f48f40b6bf0 0x7f48f415dd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:31.586 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:31.591+0000 7f48f8a00640 1 --2- 172.21.15.16:0/4292639140 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f48f4142460 0x7f48f40b66b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:31.586 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:31.591+0000 7f48f8a00640 1 --2- 172.21.15.16:0/4292639140 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f48d8004d50 0x7f48f40c7030 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:31.587 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:31.591+0000 7f48f8a00640 1 -- 172.21.15.16:0/4292639140 >> 172.21.15.16:0/4292639140 conn(0x7f48f4067330 msgr2=0x7f48f40b7730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:05:31.587 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:31.591+0000 7f48f8a00640 1 -- 172.21.15.16:0/4292639140 shutdown_connections 2024-03-31T15:05:31.587 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:31.591+0000 7f48f8a00640 1 -- 172.21.15.16:0/4292639140 wait complete. 2024-03-31T15:05:31.604 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":102,"stamp":"2024-03-31T15:05:30.553448+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":85,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":242208,"kb_used_data":5228,"kb_used_omap":12,"kb_used_meta":236851,"kb_avail":754732512,"statfs":{"total":773094113280,"available":772846092288,"internally_reserved":0,"allocated":5353472,"data_stored":2848144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":242535984},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.500967"},"pg_stats":[{"pgid":"7.3f","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-31T15:05:28.089715+0000","last_change":"2024-03-31T15:05:27.085969+0000","last_active":"2024-03-31T15:05:28.089715+0000","last_peered":"2024-03-31T15:05:28.089715+0000","last_clean":"2024-03-31T15:05:28.089715+0000","last_became_active":"2024-03-31T15:05:27.084521+0000","last_became_peered":"2024-03-31T15:05:27.084521+0000","last_unstale":"2024-03-31T15:05:28.089715+0000","last_undegraded":"2024-03-31T15:05:28.089715+0000","last_fullsized":"2024-03-31T15:05:28.089715+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_clean_scrub_stamp":"2024-03-31T15:05:26.050987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T19:01:25.453513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3e","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-31T15:05:28.164243+0000","last_change":"2024-03-31T15:05:27.086338+0000","last_active":"2024-03-31T15:05:28.164243+0000","last_peered":"2024-03-31T15:05:28.164243+0000","last_clean":"2024-03-31T15:05:28.164243+0000","last_became_active":"2024-03-31T15:05:27.086068+0000","last_became_peered":"2024-03-31T15:05:27.086068+0000","last_unstale":"2024-03-31T15:05:28.164243+0000","last_undegraded":"2024-03-31T15:05:28.164243+0000","last_fullsized":"2024-03-31T15:05:28.164243+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_clean_scrub_stamp":"2024-03-31T15:05:26.050987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:42:42.805582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3d","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-31T15:05:28.092760+0000","last_change":"2024-03-31T15:05:27.078884+0000","last_active":"2024-03-31T15:05:28.092760+0000","last_peered":"2024-03-31T15:05:28.092760+0000","last_clean":"2024-03-31T15:05:28.092760+0000","last_became_active":"2024-03-31T15:05:27.078573+0000","last_became_peered":"2024-03-31T15:05:27.078573+0000","last_unstale":"2024-03-31T15:05:28.092760+0000","last_undegraded":"2024-03-31T15:05:28.092760+0000","last_fullsized":"2024-03-31T15:05:28.092760+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_clean_scrub_stamp":"2024-03-31T15:05:26.050987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:10:37.950151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.3c","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-31T15:05:27.090081+0000","last_change":"2024-03-31T15:05:27.081269+0000","last_active":"2024-03-31T15:05:27.090081+0000","last_peered":"2024-03-31T15:05:27.090081+0000","last_clean":"2024-03-31T15:05:27.090081+0000","last_became_active":"2024-03-31T15:05:27.081033+0000","last_became_peered":"2024-03-31T15:05:27.081033+0000","last_unstale":"2024-03-31T15:05:27.090081+0000","last_undegraded":"2024-03-31T15:05:27.090081+0000","last_fullsized":"2024-03-31T15:05:27.090081+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_clean_scrub_stamp":"2024-03-31T15:05:26.050987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T16:04:24.129485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3b","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-31T15:05:28.163981+0000","last_change":"2024-03-31T15:05:27.085289+0000","last_active":"2024-03-31T15:05:28.163981+0000","last_peered":"2024-03-31T15:05:28.163981+0000","last_clean":"2024-03-31T15:05:28.163981+0000","last_became_active":"2024-03-31T15:05:27.085017+0000","last_became_peered":"2024-03-31T15:05:27.085017+0000","last_unstale":"2024-03-31T15:05:28.163981+0000","last_undegraded":"2024-03-31T15:05:28.163981+0000","last_fullsized":"2024-03-31T15:05:28.163981+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_clean_scrub_stamp":"2024-03-31T15:05:26.050987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:35:33.976512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3a","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-31T15:05:27.089535+0000","last_change":"2024-03-31T15:05:27.084961+0000","last_active":"2024-03-31T15:05:27.089535+0000","last_peered":"2024-03-31T15:05:27.089535+0000","last_clean":"2024-03-31T15:05:27.089535+0000","last_became_active":"2024-03-31T15:05:27.084733+0000","last_became_peered":"2024-03-31T15:05:27.084733+0000","last_unstale":"2024-03-31T15:05:27.089535+0000","last_undegraded":"2024-03-31T15:05:27.089535+0000","last_fullsized":"2024-03-31T15:05:27.089535+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_clean_scrub_stamp":"2024-03-31T15:05:26.050987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:26:46.240631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.39","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-31T15:05:28.089324+0000","last_change":"2024-03-31T15:05:27.092960+0000","last_active":"2024-03-31T15:05:28.089324+0000","last_peered":"2024-03-31T15:05:28.089324+0000","last_clean":"2024-03-31T15:05:28.089324+0000","last_became_active":"2024-03-31T15:05:27.092668+0000","last_became_peered":"2024-03-31T15:05:27.092668+0000","last_unstale":"2024-03-31T15:05:28.089324+0000","last_undegraded":"2024-03-31T15:05:28.089324+0000","last_fullsized":"2024-03-31T15:05:28.089324+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_clean_scrub_stamp":"2024-03-31T15:05:26.050987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T19:42:20.460194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.38","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-31T15:05:27.086950+0000","last_change":"2024-03-31T15:05:27.076883+0000","last_active":"2024-03-31T15:05:27.086950+0000","last_peered":"2024-03-31T15:05:27.086950+0000","last_clean":"2024-03-31T15:05:27.086950+0000","last_became_active":"2024-03-31T15:05:27.075582+0000","last_became_peered":"2024-03-31T15:05:27.075582+0000","last_unstale":"2024-03-31T15:05:27.086950+0000","last_undegraded":"2024-03-31T15:05:27.086950+0000","last_fullsized":"2024-03-31T15:05:27.086950+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_clean_scrub_stamp":"2024-03-31T15:05:26.050987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T17:05:23.258342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.37","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-31T15:05:27.090266+0000","last_change":"2024-03-31T15:05:27.083170+0000","last_active":"2024-03-31T15:05:27.090266+0000","last_peered":"2024-03-31T15:05:27.090266+0000","last_clean":"2024-03-31T15:05:27.090266+0000","last_became_active":"2024-03-31T15:05:27.082904+0000","last_became_peered":"2024-03-31T15:05:27.082904+0000","last_unstale":"2024-03-31T15:05:27.090266+0000","last_undegraded":"2024-03-31T15:05:27.090266+0000","last_fullsized":"2024-03-31T15:05:27.090266+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_clean_scrub_stamp":"2024-03-31T15:05:26.050987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T19:10:54.973176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.36","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-31T15:05:27.090213+0000","last_change":"2024-03-31T15:05:27.078623+0000","last_active":"2024-03-31T15:05:27.090213+0000","last_peered":"2024-03-31T15:05:27.090213+0000","last_clean":"2024-03-31T15:05:27.090213+0000","last_became_active":"2024-03-31T15:05:27.078340+0000","last_became_peered":"2024-03-31T15:05:27.078340+0000","last_unstale":"2024-03-31T15:05:27.090213+0000","last_undegraded":"2024-03-31T15:05:27.090213+0000","last_fullsized":"2024-03-31T15:05:27.090213+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_clean_scrub_stamp":"2024-03-31T15:05:26.050987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T15:48:23.844852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.35","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-31T15:05:27.082725+0000","last_change":"2024-03-31T15:05:27.075031+0000","last_active":"2024-03-31T15:05:27.082725+0000","last_peered":"2024-03-31T15:05:27.082725+0000","last_clean":"2024-03-31T15:05:27.082725+0000","last_became_active":"2024-03-31T15:05:27.074747+0000","last_became_peered":"2024-03-31T15:05:27.074747+0000","last_unstale":"2024-03-31T15:05:27.082725+0000","last_undegraded":"2024-03-31T15:05:27.082725+0000","last_fullsized":"2024-03-31T15:05:27.082725+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_clean_scrub_stamp":"2024-03-31T15:05:26.050987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:48:24.121742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.34","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-31T15:05:28.164095+0000","last_change":"2024-03-31T15:05:27.085453+0000","last_active":"2024-03-31T15:05:28.164095+0000","last_peered":"2024-03-31T15:05:28.164095+0000","last_clean":"2024-03-31T15:05:28.164095+0000","last_became_active":"2024-03-31T15:05:27.082607+0000","last_became_peered":"2024-03-31T15:05:27.082607+0000","last_unstale":"2024-03-31T15:05:28.164095+0000","last_undegraded":"2024-03-31T15:05:28.164095+0000","last_fullsized":"2024-03-31T15:05:28.164095+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_clean_scrub_stamp":"2024-03-31T15:05:26.050987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:51:01.214558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.33","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-31T15:05:28.162637+0000","last_change":"2024-03-31T15:05:27.078631+0000","last_active":"2024-03-31T15:05:28.162637+0000","last_peered":"2024-03-31T15:05:28.162637+0000","last_clean":"2024-03-31T15:05:28.162637+0000","last_became_active":"2024-03-31T15:05:27.078338+0000","last_became_peered":"2024-03-31T15:05:27.078338+0000","last_unstale":"2024-03-31T15:05:28.162637+0000","last_undegraded":"2024-03-31T15:05:28.162637+0000","last_fullsized":"2024-03-31T15:05:28.162637+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_clean_scrub_stamp":"2024-03-31T15:05:26.050987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:04:12.442076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.32","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-31T15:05:27.090036+0000","last_change":"2024-03-31T15:05:27.078798+0000","last_active":"2024-03-31T15:05:27.090036+0000","last_peered":"2024-03-31T15:05:27.090036+0000","last_clean":"2024-03-31T15:05:27.090036+0000","last_became_active":"2024-03-31T15:05:27.078531+0000","last_became_peered":"2024-03-31T15:05:27.078531+0000","last_unstale":"2024-03-31T15:05:27.090036+0000","last_undegraded":"2024-03-31T15:05:27.090036+0000","last_fullsized":"2024-03-31T15:05:27.090036+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_clean_scrub_stamp":"2024-03-31T15:05:26.050987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:29:56.093449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.31","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-31T15:05:27.087091+0000","last_change":"2024-03-31T15:05:27.079236+0000","last_active":"2024-03-31T15:05:27.087091+0000","last_peered":"2024-03-31T15:05:27.087091+0000","last_clean":"2024-03-31T15:05:27.087091+0000","last_became_active":"2024-03-31T15:05:27.078860+0000","last_became_peered":"2024-03-31T15:05:27.078860+0000","last_unstale":"2024-03-31T15:05:27.087091+0000","last_undegraded":"2024-03-31T15:05:27.087091+0000","last_fullsized":"2024-03-31T15:05:27.087091+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_clean_scrub_stamp":"2024-03-31T15:05:26.050987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:29:27.366849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.30","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-31T15:05:27.096822+0000","last_change":"2024-03-31T15:05:27.088665+0000","last_active":"2024-03-31T15:05:27.096822+0000","last_peered":"2024-03-31T15:05:27.096822+0000","last_clean":"2024-03-31T15:05:27.096822+0000","last_became_active":"2024-03-31T15:05:27.088261+0000","last_became_peered":"2024-03-31T15:05:27.088261+0000","last_unstale":"2024-03-31T15:05:27.096822+0000","last_undegraded":"2024-03-31T15:05:27.096822+0000","last_fullsized":"2024-03-31T15:05:27.096822+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_clean_scrub_stamp":"2024-03-31T15:05:26.050987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:48:24.121742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2f","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-31T15:05:28.092830+0000","last_change":"2024-03-31T15:05:27.085866+0000","last_active":"2024-03-31T15:05:28.092830+0000","last_peered":"2024-03-31T15:05:28.092830+0000","last_clean":"2024-03-31T15:05:28.092830+0000","last_became_active":"2024-03-31T15:05:27.085509+0000","last_became_peered":"2024-03-31T15:05:27.085509+0000","last_unstale":"2024-03-31T15:05:28.092830+0000","last_undegraded":"2024-03-31T15:05:28.092830+0000","last_fullsized":"2024-03-31T15:05:28.092830+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_clean_scrub_stamp":"2024-03-31T15:05:26.050987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T16:38:30.957973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2e","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-31T15:05:28.092939+0000","last_change":"2024-03-31T15:05:27.078203+0000","last_active":"2024-03-31T15:05:28.092939+0000","last_peered":"2024-03-31T15:05:28.092939+0000","last_clean":"2024-03-31T15:05:28.092939+0000","last_became_active":"2024-03-31T15:05:27.077911+0000","last_became_peered":"2024-03-31T15:05:27.077911+0000","last_unstale":"2024-03-31T15:05:28.092939+0000","last_undegraded":"2024-03-31T15:05:28.092939+0000","last_fullsized":"2024-03-31T15:05:28.092939+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_clean_scrub_stamp":"2024-03-31T15:05:26.050987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T15:41:40.292384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2d","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-31T15:05:27.095117+0000","last_change":"2024-03-31T15:05:27.088220+0000","last_active":"2024-03-31T15:05:27.095117+0000","last_peered":"2024-03-31T15:05:27.095117+0000","last_clean":"2024-03-31T15:05:27.095117+0000","last_became_active":"2024-03-31T15:05:27.087897+0000","last_became_peered":"2024-03-31T15:05:27.087897+0000","last_unstale":"2024-03-31T15:05:27.095117+0000","last_undegraded":"2024-03-31T15:05:27.095117+0000","last_fullsized":"2024-03-31T15:05:27.095117+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_clean_scrub_stamp":"2024-03-31T15:05:26.050987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:46:20.281767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2c","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-31T15:05:28.162895+0000","last_change":"2024-03-31T15:05:27.091595+0000","last_active":"2024-03-31T15:05:28.162895+0000","last_peered":"2024-03-31T15:05:28.162895+0000","last_clean":"2024-03-31T15:05:28.162895+0000","last_became_active":"2024-03-31T15:05:27.090968+0000","last_became_peered":"2024-03-31T15:05:27.090968+0000","last_unstale":"2024-03-31T15:05:28.162895+0000","last_undegraded":"2024-03-31T15:05:28.162895+0000","last_fullsized":"2024-03-31T15:05:28.162895+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_clean_scrub_stamp":"2024-03-31T15:05:26.050987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T16:28:52.354372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.2b","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-31T15:05:27.091904+0000","last_change":"2024-03-31T15:05:27.083227+0000","last_active":"2024-03-31T15:05:27.091904+0000","last_peered":"2024-03-31T15:05:27.091904+0000","last_clean":"2024-03-31T15:05:27.091904+0000","last_became_active":"2024-03-31T15:05:27.082759+0000","last_became_peered":"2024-03-31T15:05:27.082759+0000","last_unstale":"2024-03-31T15:05:27.091904+0000","last_undegraded":"2024-03-31T15:05:27.091904+0000","last_fullsized":"2024-03-31T15:05:27.091904+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_clean_scrub_stamp":"2024-03-31T15:05:26.050987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:25:02.179421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.2a","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-31T15:05:27.080744+0000","last_change":"2024-03-31T15:05:27.075150+0000","last_active":"2024-03-31T15:05:27.080744+0000","last_peered":"2024-03-31T15:05:27.080744+0000","last_clean":"2024-03-31T15:05:27.080744+0000","last_became_active":"2024-03-31T15:05:27.074725+0000","last_became_peered":"2024-03-31T15:05:27.074725+0000","last_unstale":"2024-03-31T15:05:27.080744+0000","last_undegraded":"2024-03-31T15:05:27.080744+0000","last_fullsized":"2024-03-31T15:05:27.080744+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_clean_scrub_stamp":"2024-03-31T15:05:26.050987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:15:33.311093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.29","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-31T15:05:27.087097+0000","last_change":"2024-03-31T15:05:27.080488+0000","last_active":"2024-03-31T15:05:27.087097+0000","last_peered":"2024-03-31T15:05:27.087097+0000","last_clean":"2024-03-31T15:05:27.087097+0000","last_became_active":"2024-03-31T15:05:27.080219+0000","last_became_peered":"2024-03-31T15:05:27.080219+0000","last_unstale":"2024-03-31T15:05:27.087097+0000","last_undegraded":"2024-03-31T15:05:27.087097+0000","last_fullsized":"2024-03-31T15:05:27.087097+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_clean_scrub_stamp":"2024-03-31T15:05:26.050987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:17:37.439361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.28","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-31T15:05:27.086807+0000","last_change":"2024-03-31T15:05:27.080424+0000","last_active":"2024-03-31T15:05:27.086807+0000","last_peered":"2024-03-31T15:05:27.086807+0000","last_clean":"2024-03-31T15:05:27.086807+0000","last_became_active":"2024-03-31T15:05:27.080116+0000","last_became_peered":"2024-03-31T15:05:27.080116+0000","last_unstale":"2024-03-31T15:05:27.086807+0000","last_undegraded":"2024-03-31T15:05:27.086807+0000","last_fullsized":"2024-03-31T15:05:27.086807+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_clean_scrub_stamp":"2024-03-31T15:05:26.050987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:23:19.302715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.27","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-31T15:05:27.087184+0000","last_change":"2024-03-31T15:05:27.081969+0000","last_active":"2024-03-31T15:05:27.087184+0000","last_peered":"2024-03-31T15:05:27.087184+0000","last_clean":"2024-03-31T15:05:27.087184+0000","last_became_active":"2024-03-31T15:05:27.081644+0000","last_became_peered":"2024-03-31T15:05:27.081644+0000","last_unstale":"2024-03-31T15:05:27.087184+0000","last_undegraded":"2024-03-31T15:05:27.087184+0000","last_fullsized":"2024-03-31T15:05:27.087184+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_clean_scrub_stamp":"2024-03-31T15:05:26.050987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:59:56.699974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.26","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-31T15:05:28.162863+0000","last_change":"2024-03-31T15:05:27.084818+0000","last_active":"2024-03-31T15:05:28.162863+0000","last_peered":"2024-03-31T15:05:28.162863+0000","last_clean":"2024-03-31T15:05:28.162863+0000","last_became_active":"2024-03-31T15:05:27.084498+0000","last_became_peered":"2024-03-31T15:05:27.084498+0000","last_unstale":"2024-03-31T15:05:28.162863+0000","last_undegraded":"2024-03-31T15:05:28.162863+0000","last_fullsized":"2024-03-31T15:05:28.162863+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_clean_scrub_stamp":"2024-03-31T15:05:26.050987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T17:32:20.502679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.25","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-31T15:05:28.092995+0000","last_change":"2024-03-31T15:05:27.086432+0000","last_active":"2024-03-31T15:05:28.092995+0000","last_peered":"2024-03-31T15:05:28.092995+0000","last_clean":"2024-03-31T15:05:28.092995+0000","last_became_active":"2024-03-31T15:05:27.086196+0000","last_became_peered":"2024-03-31T15:05:27.086196+0000","last_unstale":"2024-03-31T15:05:28.092995+0000","last_undegraded":"2024-03-31T15:05:28.092995+0000","last_fullsized":"2024-03-31T15:05:28.092995+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_clean_scrub_stamp":"2024-03-31T15:05:26.050987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T15:14:52.888307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.24","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-31T15:05:28.164161+0000","last_change":"2024-03-31T15:05:27.084714+0000","last_active":"2024-03-31T15:05:28.164161+0000","last_peered":"2024-03-31T15:05:28.164161+0000","last_clean":"2024-03-31T15:05:28.164161+0000","last_became_active":"2024-03-31T15:05:27.082275+0000","last_became_peered":"2024-03-31T15:05:27.082275+0000","last_unstale":"2024-03-31T15:05:28.164161+0000","last_undegraded":"2024-03-31T15:05:28.164161+0000","last_fullsized":"2024-03-31T15:05:28.164161+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_clean_scrub_stamp":"2024-03-31T15:05:26.050987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T19:42:20.460194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.23","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-31T15:05:27.083499+0000","last_change":"2024-03-31T15:05:27.078374+0000","last_active":"2024-03-31T15:05:27.083499+0000","last_peered":"2024-03-31T15:05:27.083499+0000","last_clean":"2024-03-31T15:05:27.083499+0000","last_became_active":"2024-03-31T15:05:27.078099+0000","last_became_peered":"2024-03-31T15:05:27.078099+0000","last_unstale":"2024-03-31T15:05:27.083499+0000","last_undegraded":"2024-03-31T15:05:27.083499+0000","last_fullsized":"2024-03-31T15:05:27.083499+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_clean_scrub_stamp":"2024-03-31T15:05:26.050987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T19:12:41.806694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.22","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-31T15:05:28.092636+0000","last_change":"2024-03-31T15:05:27.081299+0000","last_active":"2024-03-31T15:05:28.092636+0000","last_peered":"2024-03-31T15:05:28.092636+0000","last_clean":"2024-03-31T15:05:28.092636+0000","last_became_active":"2024-03-31T15:05:27.080969+0000","last_became_peered":"2024-03-31T15:05:27.080969+0000","last_unstale":"2024-03-31T15:05:28.092636+0000","last_undegraded":"2024-03-31T15:05:28.092636+0000","last_fullsized":"2024-03-31T15:05:28.092636+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_clean_scrub_stamp":"2024-03-31T15:05:26.050987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:22:40.920851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.21","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-31T15:05:27.090927+0000","last_change":"2024-03-31T15:05:27.088171+0000","last_active":"2024-03-31T15:05:27.090927+0000","last_peered":"2024-03-31T15:05:27.090927+0000","last_clean":"2024-03-31T15:05:27.090927+0000","last_became_active":"2024-03-31T15:05:27.086948+0000","last_became_peered":"2024-03-31T15:05:27.086948+0000","last_unstale":"2024-03-31T15:05:27.090927+0000","last_undegraded":"2024-03-31T15:05:27.090927+0000","last_fullsized":"2024-03-31T15:05:27.090927+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_clean_scrub_stamp":"2024-03-31T15:05:26.050987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T19:33:19.777309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.20","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-31T15:05:28.162277+0000","last_change":"2024-03-31T15:05:27.083002+0000","last_active":"2024-03-31T15:05:28.162277+0000","last_peered":"2024-03-31T15:05:28.162277+0000","last_clean":"2024-03-31T15:05:28.162277+0000","last_became_active":"2024-03-31T15:05:27.082714+0000","last_became_peered":"2024-03-31T15:05:27.082714+0000","last_unstale":"2024-03-31T15:05:28.162277+0000","last_undegraded":"2024-03-31T15:05:28.162277+0000","last_fullsized":"2024-03-31T15:05:28.162277+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_clean_scrub_stamp":"2024-03-31T15:05:26.050987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T19:51:20.669777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-31T15:05:27.090618+0000","last_change":"2024-03-31T15:05:27.080242+0000","last_active":"2024-03-31T15:05:27.090618+0000","last_peered":"2024-03-31T15:05:27.090618+0000","last_clean":"2024-03-31T15:05:27.090618+0000","last_became_active":"2024-03-31T15:05:27.079987+0000","last_became_peered":"2024-03-31T15:05:27.079987+0000","last_unstale":"2024-03-31T15:05:27.090618+0000","last_undegraded":"2024-03-31T15:05:27.090618+0000","last_fullsized":"2024-03-31T15:05:27.090618+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_clean_scrub_stamp":"2024-03-31T15:05:26.050987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T17:05:23.258342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"44'4","reported_seq":16,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-31T15:05:27.104225+0000","last_change":"2024-03-31T15:05:26.082021+0000","last_active":"2024-03-31T15:05:27.104225+0000","last_peered":"2024-03-31T15:05:27.104225+0000","last_clean":"2024-03-31T15:05:27.104225+0000","last_became_active":"2024-03-31T15:05:26.081723+0000","last_became_peered":"2024-03-31T15:05:26.081723+0000","last_unstale":"2024-03-31T15:05:27.104225+0000","last_undegraded":"2024-03-31T15:05:27.104225+0000","last_fullsized":"2024-03-31T15:05:27.104225+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:05:25.044753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:05:25.044753+0000","last_clean_scrub_stamp":"2024-03-31T15:05:25.044753+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:26:05.887334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1f","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-31T15:05:27.087158+0000","last_change":"2024-03-31T15:05:27.080022+0000","last_active":"2024-03-31T15:05:27.087158+0000","last_peered":"2024-03-31T15:05:27.087158+0000","last_clean":"2024-03-31T15:05:27.087158+0000","last_became_active":"2024-03-31T15:05:27.079359+0000","last_became_peered":"2024-03-31T15:05:27.079359+0000","last_unstale":"2024-03-31T15:05:27.087158+0000","last_undegraded":"2024-03-31T15:05:27.087158+0000","last_fullsized":"2024-03-31T15:05:27.087158+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_clean_scrub_stamp":"2024-03-31T15:05:26.050987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:04:33.953699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1e","version":"44'1","reported_seq":15,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-31T15:05:28.093076+0000","last_change":"2024-03-31T15:05:26.076816+0000","last_active":"2024-03-31T15:05:28.093076+0000","last_peered":"2024-03-31T15:05:28.093076+0000","last_clean":"2024-03-31T15:05:28.093076+0000","last_became_active":"2024-03-31T15:05:26.076308+0000","last_became_peered":"2024-03-31T15:05:26.076308+0000","last_unstale":"2024-03-31T15:05:28.093076+0000","last_undegraded":"2024-03-31T15:05:28.093076+0000","last_fullsized":"2024-03-31T15:05:28.093076+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:05:25.044753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:05:25.044753+0000","last_clean_scrub_stamp":"2024-03-31T15:05:25.044753+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:22:07.071870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-31T15:05:28.164275+0000","last_change":"2024-03-31T15:05:27.088970+0000","last_active":"2024-03-31T15:05:28.164275+0000","last_peered":"2024-03-31T15:05:28.164275+0000","last_clean":"2024-03-31T15:05:28.164275+0000","last_became_active":"2024-03-31T15:05:27.085485+0000","last_became_peered":"2024-03-31T15:05:27.085485+0000","last_unstale":"2024-03-31T15:05:28.164275+0000","last_undegraded":"2024-03-31T15:05:28.164275+0000","last_fullsized":"2024-03-31T15:05:28.164275+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_clean_scrub_stamp":"2024-03-31T15:05:26.050987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T15:53:54.088987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1d","version":"44'2","reported_seq":14,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-31T15:05:27.102151+0000","last_change":"2024-03-31T15:05:26.080274+0000","last_active":"2024-03-31T15:05:27.102151+0000","last_peered":"2024-03-31T15:05:27.102151+0000","last_clean":"2024-03-31T15:05:27.102151+0000","last_became_active":"2024-03-31T15:05:26.079983+0000","last_became_peered":"2024-03-31T15:05:26.079983+0000","last_unstale":"2024-03-31T15:05:27.102151+0000","last_undegraded":"2024-03-31T15:05:27.102151+0000","last_fullsized":"2024-03-31T15:05:27.102151+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:05:25.044753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:05:25.044753+0000","last_clean_scrub_stamp":"2024-03-31T15:05:25.044753+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:15:32.304859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-31T15:05:27.090536+0000","last_change":"2024-03-31T15:05:27.079753+0000","last_active":"2024-03-31T15:05:27.090536+0000","last_peered":"2024-03-31T15:05:27.090536+0000","last_clean":"2024-03-31T15:05:27.090536+0000","last_became_active":"2024-03-31T15:05:27.079480+0000","last_became_peered":"2024-03-31T15:05:27.079480+0000","last_unstale":"2024-03-31T15:05:27.090536+0000","last_undegraded":"2024-03-31T15:05:27.090536+0000","last_fullsized":"2024-03-31T15:05:27.090536+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_clean_scrub_stamp":"2024-03-31T15:05:26.050987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T16:16:23.687703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"44'2","reported_seq":16,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-31T15:05:28.089618+0000","last_change":"2024-03-31T15:05:26.079703+0000","last_active":"2024-03-31T15:05:28.089618+0000","last_peered":"2024-03-31T15:05:28.089618+0000","last_clean":"2024-03-31T15:05:28.089618+0000","last_became_active":"2024-03-31T15:05:26.079314+0000","last_became_peered":"2024-03-31T15:05:26.079314+0000","last_unstale":"2024-03-31T15:05:28.089618+0000","last_undegraded":"2024-03-31T15:05:28.089618+0000","last_fullsized":"2024-03-31T15:05:28.089618+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:05:25.044753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:05:25.044753+0000","last_clean_scrub_stamp":"2024-03-31T15:05:25.044753+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T15:41:39.286149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-31T15:05:28.092714+0000","last_change":"2024-03-31T15:05:27.075920+0000","last_active":"2024-03-31T15:05:28.092714+0000","last_peered":"2024-03-31T15:05:28.092714+0000","last_clean":"2024-03-31T15:05:28.092714+0000","last_became_active":"2024-03-31T15:05:27.075611+0000","last_became_peered":"2024-03-31T15:05:27.075611+0000","last_unstale":"2024-03-31T15:05:28.092714+0000","last_undegraded":"2024-03-31T15:05:28.092714+0000","last_fullsized":"2024-03-31T15:05:28.092714+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_clean_scrub_stamp":"2024-03-31T15:05:26.050987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T16:42:27.714002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"44'1","reported_seq":13,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-31T15:05:27.099175+0000","last_change":"2024-03-31T15:05:26.085749+0000","last_active":"2024-03-31T15:05:27.099175+0000","last_peered":"2024-03-31T15:05:27.099175+0000","last_clean":"2024-03-31T15:05:27.099175+0000","last_became_active":"2024-03-31T15:05:26.085260+0000","last_became_peered":"2024-03-31T15:05:26.085260+0000","last_unstale":"2024-03-31T15:05:27.099175+0000","last_undegraded":"2024-03-31T15:05:27.099175+0000","last_fullsized":"2024-03-31T15:05:27.099175+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:05:25.044753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:05:25.044753+0000","last_clean_scrub_stamp":"2024-03-31T15:05:25.044753+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:43:01.014270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1b","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-31T15:05:27.096183+0000","last_change":"2024-03-31T15:05:27.086020+0000","last_active":"2024-03-31T15:05:27.096183+0000","last_peered":"2024-03-31T15:05:27.096183+0000","last_clean":"2024-03-31T15:05:27.096183+0000","last_became_active":"2024-03-31T15:05:27.085701+0000","last_became_peered":"2024-03-31T15:05:27.085701+0000","last_unstale":"2024-03-31T15:05:27.096183+0000","last_undegraded":"2024-03-31T15:05:27.096183+0000","last_fullsized":"2024-03-31T15:05:27.096183+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_clean_scrub_stamp":"2024-03-31T15:05:26.050987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:16:34.982760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1a","version":"44'1","reported_seq":15,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-31T15:05:28.092787+0000","last_change":"2024-03-31T15:05:26.071315+0000","last_active":"2024-03-31T15:05:28.092787+0000","last_peered":"2024-03-31T15:05:28.092787+0000","last_clean":"2024-03-31T15:05:28.092787+0000","last_became_active":"2024-03-31T15:05:26.071008+0000","last_became_peered":"2024-03-31T15:05:26.071008+0000","last_unstale":"2024-03-31T15:05:28.092787+0000","last_undegraded":"2024-03-31T15:05:28.092787+0000","last_fullsized":"2024-03-31T15:05:28.092787+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:05:25.044753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:05:25.044753+0000","last_clean_scrub_stamp":"2024-03-31T15:05:25.044753+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:24:07.692704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-31T15:05:27.086926+0000","last_change":"2024-03-31T15:05:27.076679+0000","last_active":"2024-03-31T15:05:27.086926+0000","last_peered":"2024-03-31T15:05:27.086926+0000","last_clean":"2024-03-31T15:05:27.086926+0000","last_became_active":"2024-03-31T15:05:27.075493+0000","last_became_peered":"2024-03-31T15:05:27.075493+0000","last_unstale":"2024-03-31T15:05:27.086926+0000","last_undegraded":"2024-03-31T15:05:27.086926+0000","last_fullsized":"2024-03-31T15:05:27.086926+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_clean_scrub_stamp":"2024-03-31T15:05:26.050987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T15:10:57.166552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-31T15:05:28.163811+0000","last_change":"2024-03-31T15:05:26.077943+0000","last_active":"2024-03-31T15:05:28.163811+0000","last_peered":"2024-03-31T15:05:28.163811+0000","last_clean":"2024-03-31T15:05:28.163811+0000","last_became_active":"2024-03-31T15:05:26.077665+0000","last_became_peered":"2024-03-31T15:05:26.077665+0000","last_unstale":"2024-03-31T15:05:28.163811+0000","last_undegraded":"2024-03-31T15:05:28.163811+0000","last_fullsized":"2024-03-31T15:05:28.163811+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:05:25.044753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:05:25.044753+0000","last_clean_scrub_stamp":"2024-03-31T15:05:25.044753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T17:50:37.634805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-31T15:05:27.090007+0000","last_change":"2024-03-31T15:05:27.083246+0000","last_active":"2024-03-31T15:05:27.090007+0000","last_peered":"2024-03-31T15:05:27.090007+0000","last_clean":"2024-03-31T15:05:27.090007+0000","last_became_active":"2024-03-31T15:05:27.082988+0000","last_became_peered":"2024-03-31T15:05:27.082988+0000","last_unstale":"2024-03-31T15:05:27.090007+0000","last_undegraded":"2024-03-31T15:05:27.090007+0000","last_fullsized":"2024-03-31T15:05:27.090007+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_clean_scrub_stamp":"2024-03-31T15:05:26.050987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:18:34.558382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-31T15:05:27.064157+0000","last_change":"2024-03-31T15:05:26.078694+0000","last_active":"2024-03-31T15:05:27.064157+0000","last_peered":"2024-03-31T15:05:27.064157+0000","last_clean":"2024-03-31T15:05:27.064157+0000","last_became_active":"2024-03-31T15:05:26.076178+0000","last_became_peered":"2024-03-31T15:05:26.076178+0000","last_unstale":"2024-03-31T15:05:27.064157+0000","last_undegraded":"2024-03-31T15:05:27.064157+0000","last_fullsized":"2024-03-31T15:05:27.064157+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:05:25.044753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:05:25.044753+0000","last_clean_scrub_stamp":"2024-03-31T15:05:25.044753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:19:48.661247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-31T15:05:28.092937+0000","last_change":"2024-03-31T15:05:27.077997+0000","last_active":"2024-03-31T15:05:28.092937+0000","last_peered":"2024-03-31T15:05:28.092937+0000","last_clean":"2024-03-31T15:05:28.092937+0000","last_became_active":"2024-03-31T15:05:27.077615+0000","last_became_peered":"2024-03-31T15:05:27.077615+0000","last_unstale":"2024-03-31T15:05:28.092937+0000","last_undegraded":"2024-03-31T15:05:28.092937+0000","last_fullsized":"2024-03-31T15:05:28.092937+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_clean_scrub_stamp":"2024-03-31T15:05:26.050987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T17:09:12.798678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-31T15:05:28.093097+0000","last_change":"2024-03-31T15:05:26.076164+0000","last_active":"2024-03-31T15:05:28.093097+0000","last_peered":"2024-03-31T15:05:28.093097+0000","last_clean":"2024-03-31T15:05:28.093097+0000","last_became_active":"2024-03-31T15:05:26.075868+0000","last_became_peered":"2024-03-31T15:05:26.075868+0000","last_unstale":"2024-03-31T15:05:28.093097+0000","last_undegraded":"2024-03-31T15:05:28.093097+0000","last_fullsized":"2024-03-31T15:05:28.093097+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:05:25.044753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:05:25.044753+0000","last_clean_scrub_stamp":"2024-03-31T15:05:25.044753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:40:36.292782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-31T15:05:28.089795+0000","last_change":"2024-03-31T15:05:27.089434+0000","last_active":"2024-03-31T15:05:28.089795+0000","last_peered":"2024-03-31T15:05:28.089795+0000","last_clean":"2024-03-31T15:05:28.089795+0000","last_became_active":"2024-03-31T15:05:27.089097+0000","last_became_peered":"2024-03-31T15:05:27.089097+0000","last_unstale":"2024-03-31T15:05:28.089795+0000","last_undegraded":"2024-03-31T15:05:28.089795+0000","last_fullsized":"2024-03-31T15:05:28.089795+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_clean_scrub_stamp":"2024-03-31T15:05:26.050987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:45:25.819165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-31T15:05:27.065533+0000","last_change":"2024-03-31T15:05:26.085575+0000","last_active":"2024-03-31T15:05:27.065533+0000","last_peered":"2024-03-31T15:05:27.065533+0000","last_clean":"2024-03-31T15:05:27.065533+0000","last_became_active":"2024-03-31T15:05:26.085201+0000","last_became_peered":"2024-03-31T15:05:26.085201+0000","last_unstale":"2024-03-31T15:05:27.065533+0000","last_undegraded":"2024-03-31T15:05:27.065533+0000","last_fullsized":"2024-03-31T15:05:27.065533+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:05:25.044753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:05:25.044753+0000","last_clean_scrub_stamp":"2024-03-31T15:05:25.044753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:46:19.275533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.14","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-31T15:05:27.088949+0000","last_change":"2024-03-31T15:05:27.079895+0000","last_active":"2024-03-31T15:05:27.088949+0000","last_peered":"2024-03-31T15:05:27.088949+0000","last_clean":"2024-03-31T15:05:27.088949+0000","last_became_active":"2024-03-31T15:05:27.079538+0000","last_became_peered":"2024-03-31T15:05:27.079538+0000","last_unstale":"2024-03-31T15:05:27.088949+0000","last_undegraded":"2024-03-31T15:05:27.088949+0000","last_fullsized":"2024-03-31T15:05:27.088949+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_clean_scrub_stamp":"2024-03-31T15:05:26.050987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:03:20.812659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"44'3","reported_seq":17,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-31T15:05:28.089681+0000","last_change":"2024-03-31T15:05:26.080315+0000","last_active":"2024-03-31T15:05:28.089681+0000","last_peered":"2024-03-31T15:05:28.089681+0000","last_clean":"2024-03-31T15:05:28.089681+0000","last_became_active":"2024-03-31T15:05:26.079466+0000","last_became_peered":"2024-03-31T15:05:26.079466+0000","last_unstale":"2024-03-31T15:05:28.089681+0000","last_undegraded":"2024-03-31T15:05:28.089681+0000","last_fullsized":"2024-03-31T15:05:28.089681+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:05:25.044753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:05:25.044753+0000","last_clean_scrub_stamp":"2024-03-31T15:05:25.044753+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T16:38:29.951739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.15","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-31T15:05:28.089467+0000","last_change":"2024-03-31T15:05:27.091966+0000","last_active":"2024-03-31T15:05:28.089467+0000","last_peered":"2024-03-31T15:05:28.089467+0000","last_clean":"2024-03-31T15:05:28.089467+0000","last_became_active":"2024-03-31T15:05:27.091670+0000","last_became_peered":"2024-03-31T15:05:27.091670+0000","last_unstale":"2024-03-31T15:05:28.089467+0000","last_undegraded":"2024-03-31T15:05:28.089467+0000","last_fullsized":"2024-03-31T15:05:28.089467+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_clean_scrub_stamp":"2024-03-31T15:05:26.050987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:42:42.805582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.14","version":"44'6","reported_seq":20,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-31T15:05:28.162997+0000","last_change":"2024-03-31T15:05:26.084337+0000","last_active":"2024-03-31T15:05:28.162997+0000","last_peered":"2024-03-31T15:05:28.162997+0000","last_clean":"2024-03-31T15:05:28.162997+0000","last_became_active":"2024-03-31T15:05:26.084031+0000","last_became_peered":"2024-03-31T15:05:26.084031+0000","last_unstale":"2024-03-31T15:05:28.162997+0000","last_undegraded":"2024-03-31T15:05:28.162997+0000","last_fullsized":"2024-03-31T15:05:28.162997+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:05:25.044753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:05:25.044753+0000","last_clean_scrub_stamp":"2024-03-31T15:05:25.044753+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:13:05.524841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-31T15:05:27.095001+0000","last_change":"2024-03-31T15:05:27.087927+0000","last_active":"2024-03-31T15:05:27.095001+0000","last_peered":"2024-03-31T15:05:27.095001+0000","last_clean":"2024-03-31T15:05:27.095001+0000","last_became_active":"2024-03-31T15:05:27.087659+0000","last_became_peered":"2024-03-31T15:05:27.087659+0000","last_unstale":"2024-03-31T15:05:27.095001+0000","last_undegraded":"2024-03-31T15:05:27.095001+0000","last_fullsized":"2024-03-31T15:05:27.095001+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_clean_scrub_stamp":"2024-03-31T15:05:26.050987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T16:37:34.994306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"44'2","reported_seq":14,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-31T15:05:27.100366+0000","last_change":"2024-03-31T15:05:26.086976+0000","last_active":"2024-03-31T15:05:27.100366+0000","last_peered":"2024-03-31T15:05:27.100366+0000","last_clean":"2024-03-31T15:05:27.100366+0000","last_became_active":"2024-03-31T15:05:26.086764+0000","last_became_peered":"2024-03-31T15:05:26.086764+0000","last_unstale":"2024-03-31T15:05:27.100366+0000","last_undegraded":"2024-03-31T15:05:27.100366+0000","last_fullsized":"2024-03-31T15:05:27.100366+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:05:25.044753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:05:25.044753+0000","last_clean_scrub_stamp":"2024-03-31T15:05:25.044753+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T17:09:24.098955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-31T15:05:27.098760+0000","last_change":"2024-03-31T15:05:27.094181+0000","last_active":"2024-03-31T15:05:27.098760+0000","last_peered":"2024-03-31T15:05:27.098760+0000","last_clean":"2024-03-31T15:05:27.098760+0000","last_became_active":"2024-03-31T15:05:27.093442+0000","last_became_peered":"2024-03-31T15:05:27.093442+0000","last_unstale":"2024-03-31T15:05:27.098760+0000","last_undegraded":"2024-03-31T15:05:27.098760+0000","last_fullsized":"2024-03-31T15:05:27.098760+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_clean_scrub_stamp":"2024-03-31T15:05:26.050987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:56:24.208563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-31T15:05:28.089591+0000","last_change":"2024-03-31T15:05:26.081225+0000","last_active":"2024-03-31T15:05:28.089591+0000","last_peered":"2024-03-31T15:05:28.089591+0000","last_clean":"2024-03-31T15:05:28.089591+0000","last_became_active":"2024-03-31T15:05:26.080966+0000","last_became_peered":"2024-03-31T15:05:26.080966+0000","last_unstale":"2024-03-31T15:05:28.089591+0000","last_undegraded":"2024-03-31T15:05:28.089591+0000","last_fullsized":"2024-03-31T15:05:28.089591+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:05:25.044753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:05:25.044753+0000","last_clean_scrub_stamp":"2024-03-31T15:05:25.044753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:10:36.943916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.10","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-31T15:05:27.088591+0000","last_change":"2024-03-31T15:05:27.080688+0000","last_active":"2024-03-31T15:05:27.088591+0000","last_peered":"2024-03-31T15:05:27.088591+0000","last_clean":"2024-03-31T15:05:27.088591+0000","last_became_active":"2024-03-31T15:05:27.080415+0000","last_became_peered":"2024-03-31T15:05:27.080415+0000","last_unstale":"2024-03-31T15:05:27.088591+0000","last_undegraded":"2024-03-31T15:05:27.088591+0000","last_fullsized":"2024-03-31T15:05:27.088591+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_clean_scrub_stamp":"2024-03-31T15:05:26.050987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:26:04.100923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-31T15:05:27.073295+0000","last_change":"2024-03-31T15:05:26.083701+0000","last_active":"2024-03-31T15:05:27.073295+0000","last_peered":"2024-03-31T15:05:27.073295+0000","last_clean":"2024-03-31T15:05:27.073295+0000","last_became_active":"2024-03-31T15:05:26.080775+0000","last_became_peered":"2024-03-31T15:05:26.080775+0000","last_unstale":"2024-03-31T15:05:27.073295+0000","last_undegraded":"2024-03-31T15:05:27.073295+0000","last_fullsized":"2024-03-31T15:05:27.073295+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:05:25.044753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:05:25.044753+0000","last_clean_scrub_stamp":"2024-03-31T15:05:25.044753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T17:43:49.683910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-31T15:05:28.089475+0000","last_change":"2024-03-31T15:05:27.086375+0000","last_active":"2024-03-31T15:05:28.089475+0000","last_peered":"2024-03-31T15:05:28.089475+0000","last_clean":"2024-03-31T15:05:28.089475+0000","last_became_active":"2024-03-31T15:05:27.086051+0000","last_became_peered":"2024-03-31T15:05:27.086051+0000","last_unstale":"2024-03-31T15:05:28.089475+0000","last_undegraded":"2024-03-31T15:05:28.089475+0000","last_fullsized":"2024-03-31T15:05:28.089475+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_clean_scrub_stamp":"2024-03-31T15:05:26.050987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T15:53:54.088987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.10","version":"44'8","reported_seq":20,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-31T15:05:27.105302+0000","last_change":"2024-03-31T15:05:26.081850+0000","last_active":"2024-03-31T15:05:27.105302+0000","last_peered":"2024-03-31T15:05:27.105302+0000","last_clean":"2024-03-31T15:05:27.105302+0000","last_became_active":"2024-03-31T15:05:26.081484+0000","last_became_peered":"2024-03-31T15:05:26.081484+0000","last_unstale":"2024-03-31T15:05:27.105302+0000","last_undegraded":"2024-03-31T15:05:27.105302+0000","last_fullsized":"2024-03-31T15:05:27.105302+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:05:25.044753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:05:25.044753+0000","last_clean_scrub_stamp":"2024-03-31T15:05:25.044753+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:07:04.230530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.e","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-31T15:05:28.163012+0000","last_change":"2024-03-31T15:05:27.092324+0000","last_active":"2024-03-31T15:05:28.163012+0000","last_peered":"2024-03-31T15:05:28.163012+0000","last_clean":"2024-03-31T15:05:28.163012+0000","last_became_active":"2024-03-31T15:05:27.091992+0000","last_became_peered":"2024-03-31T15:05:27.091992+0000","last_unstale":"2024-03-31T15:05:28.163012+0000","last_undegraded":"2024-03-31T15:05:28.163012+0000","last_fullsized":"2024-03-31T15:05:28.163012+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_clean_scrub_stamp":"2024-03-31T15:05:26.050987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:34:07.231805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"44'1","reported_seq":15,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-31T15:05:28.162950+0000","last_change":"2024-03-31T15:05:26.080745+0000","last_active":"2024-03-31T15:05:28.162950+0000","last_peered":"2024-03-31T15:05:28.162950+0000","last_clean":"2024-03-31T15:05:28.162950+0000","last_became_active":"2024-03-31T15:05:26.080436+0000","last_became_peered":"2024-03-31T15:05:26.080436+0000","last_unstale":"2024-03-31T15:05:28.162950+0000","last_undegraded":"2024-03-31T15:05:28.162950+0000","last_fullsized":"2024-03-31T15:05:28.162950+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:05:25.044753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:05:25.044753+0000","last_clean_scrub_stamp":"2024-03-31T15:05:25.044753+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:50:07.343324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-31T15:05:27.095399+0000","last_change":"2024-03-31T15:05:27.086835+0000","last_active":"2024-03-31T15:05:27.095399+0000","last_peered":"2024-03-31T15:05:27.095399+0000","last_clean":"2024-03-31T15:05:27.095399+0000","last_became_active":"2024-03-31T15:05:27.086599+0000","last_became_peered":"2024-03-31T15:05:27.086599+0000","last_unstale":"2024-03-31T15:05:27.095399+0000","last_undegraded":"2024-03-31T15:05:27.095399+0000","last_fullsized":"2024-03-31T15:05:27.095399+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_clean_scrub_stamp":"2024-03-31T15:05:26.050987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:16:43.129582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.e","version":"44'2","reported_seq":16,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-31T15:05:28.093095+0000","last_change":"2024-03-31T15:05:26.076775+0000","last_active":"2024-03-31T15:05:28.093095+0000","last_peered":"2024-03-31T15:05:28.093095+0000","last_clean":"2024-03-31T15:05:28.093095+0000","last_became_active":"2024-03-31T15:05:26.075913+0000","last_became_peered":"2024-03-31T15:05:26.075913+0000","last_unstale":"2024-03-31T15:05:28.093095+0000","last_undegraded":"2024-03-31T15:05:28.093095+0000","last_fullsized":"2024-03-31T15:05:28.093095+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:05:25.044753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:05:25.044753+0000","last_clean_scrub_stamp":"2024-03-31T15:05:25.044753+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T15:23:12.818282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-31T15:05:27.090223+0000","last_change":"2024-03-31T15:05:27.083967+0000","last_active":"2024-03-31T15:05:27.090223+0000","last_peered":"2024-03-31T15:05:27.090223+0000","last_clean":"2024-03-31T15:05:27.090223+0000","last_became_active":"2024-03-31T15:05:27.083691+0000","last_became_peered":"2024-03-31T15:05:27.083691+0000","last_unstale":"2024-03-31T15:05:27.090223+0000","last_undegraded":"2024-03-31T15:05:27.090223+0000","last_fullsized":"2024-03-31T15:05:27.090223+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_clean_scrub_stamp":"2024-03-31T15:05:26.050987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:19:49.667481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.d","version":"44'1","reported_seq":15,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-31T15:05:28.164130+0000","last_change":"2024-03-31T15:05:26.078710+0000","last_active":"2024-03-31T15:05:28.164130+0000","last_peered":"2024-03-31T15:05:28.164130+0000","last_clean":"2024-03-31T15:05:28.164130+0000","last_became_active":"2024-03-31T15:05:26.078389+0000","last_became_peered":"2024-03-31T15:05:26.078389+0000","last_unstale":"2024-03-31T15:05:28.164130+0000","last_undegraded":"2024-03-31T15:05:28.164130+0000","last_fullsized":"2024-03-31T15:05:28.164130+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:05:25.044753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:05:25.044753+0000","last_clean_scrub_stamp":"2024-03-31T15:05:25.044753+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T16:42:26.707767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-31T15:05:27.083584+0000","last_change":"2024-03-31T15:05:27.078424+0000","last_active":"2024-03-31T15:05:27.083584+0000","last_peered":"2024-03-31T15:05:27.083584+0000","last_clean":"2024-03-31T15:05:27.083584+0000","last_became_active":"2024-03-31T15:05:27.078156+0000","last_became_peered":"2024-03-31T15:05:27.078156+0000","last_unstale":"2024-03-31T15:05:27.083584+0000","last_undegraded":"2024-03-31T15:05:27.083584+0000","last_fullsized":"2024-03-31T15:05:27.083584+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_clean_scrub_stamp":"2024-03-31T15:05:26.050987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T16:24:34.638128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-31T15:05:27.075688+0000","last_change":"2024-03-31T15:05:26.088880+0000","last_active":"2024-03-31T15:05:27.075688+0000","last_peered":"2024-03-31T15:05:27.075688+0000","last_clean":"2024-03-31T15:05:27.075688+0000","last_became_active":"2024-03-31T15:05:26.088608+0000","last_became_peered":"2024-03-31T15:05:26.088608+0000","last_unstale":"2024-03-31T15:05:27.075688+0000","last_undegraded":"2024-03-31T15:05:27.075688+0000","last_fullsized":"2024-03-31T15:05:27.075688+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:05:25.044753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:05:25.044753+0000","last_clean_scrub_stamp":"2024-03-31T15:05:25.044753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:52:22.169709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-31T15:05:27.087289+0000","last_change":"2024-03-31T15:05:27.082535+0000","last_active":"2024-03-31T15:05:27.087289+0000","last_peered":"2024-03-31T15:05:27.087289+0000","last_clean":"2024-03-31T15:05:27.087289+0000","last_became_active":"2024-03-31T15:05:27.082264+0000","last_became_peered":"2024-03-31T15:05:27.082264+0000","last_unstale":"2024-03-31T15:05:27.087289+0000","last_undegraded":"2024-03-31T15:05:27.087289+0000","last_fullsized":"2024-03-31T15:05:27.087289+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_clean_scrub_stamp":"2024-03-31T15:05:26.050987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:12:25.160714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.b","version":"44'2","reported_seq":14,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-31T15:05:27.101266+0000","last_change":"2024-03-31T15:05:26.085506+0000","last_active":"2024-03-31T15:05:27.101266+0000","last_peered":"2024-03-31T15:05:27.101266+0000","last_clean":"2024-03-31T15:05:27.101266+0000","last_became_active":"2024-03-31T15:05:26.084906+0000","last_became_peered":"2024-03-31T15:05:26.084906+0000","last_unstale":"2024-03-31T15:05:27.101266+0000","last_undegraded":"2024-03-31T15:05:27.101266+0000","last_fullsized":"2024-03-31T15:05:27.101266+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:05:25.044753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:05:25.044753+0000","last_clean_scrub_stamp":"2024-03-31T15:05:25.044753+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T19:15:58.627228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-31T15:05:27.091745+0000","last_change":"2024-03-31T15:05:27.085483+0000","last_active":"2024-03-31T15:05:27.091745+0000","last_peered":"2024-03-31T15:05:27.091745+0000","last_clean":"2024-03-31T15:05:27.091745+0000","last_became_active":"2024-03-31T15:05:27.085177+0000","last_became_peered":"2024-03-31T15:05:27.085177+0000","last_unstale":"2024-03-31T15:05:27.091745+0000","last_undegraded":"2024-03-31T15:05:27.091745+0000","last_fullsized":"2024-03-31T15:05:27.091745+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_clean_scrub_stamp":"2024-03-31T15:05:26.050987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T15:48:23.844852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-31T15:05:28.163901+0000","last_change":"2024-03-31T15:05:26.074149+0000","last_active":"2024-03-31T15:05:28.163901+0000","last_peered":"2024-03-31T15:05:28.163901+0000","last_clean":"2024-03-31T15:05:28.163901+0000","last_became_active":"2024-03-31T15:05:26.073847+0000","last_became_peered":"2024-03-31T15:05:26.073847+0000","last_unstale":"2024-03-31T15:05:28.163901+0000","last_undegraded":"2024-03-31T15:05:28.163901+0000","last_fullsized":"2024-03-31T15:05:28.163901+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:05:25.044753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:05:25.044753+0000","last_clean_scrub_stamp":"2024-03-31T15:05:25.044753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:26:16.679750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-31T15:05:28.162277+0000","last_change":"2024-03-31T15:05:27.081050+0000","last_active":"2024-03-31T15:05:28.162277+0000","last_peered":"2024-03-31T15:05:28.162277+0000","last_clean":"2024-03-31T15:05:28.162277+0000","last_became_active":"2024-03-31T15:05:27.080718+0000","last_became_peered":"2024-03-31T15:05:27.080718+0000","last_unstale":"2024-03-31T15:05:28.162277+0000","last_undegraded":"2024-03-31T15:05:28.162277+0000","last_fullsized":"2024-03-31T15:05:28.162277+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_clean_scrub_stamp":"2024-03-31T15:05:26.050987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:04:52.777130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-31T15:05:27.063860+0000","last_change":"2024-03-31T15:05:26.084529+0000","last_active":"2024-03-31T15:05:27.063860+0000","last_peered":"2024-03-31T15:05:27.063860+0000","last_clean":"2024-03-31T15:05:27.063860+0000","last_became_active":"2024-03-31T15:05:26.084245+0000","last_became_peered":"2024-03-31T15:05:26.084245+0000","last_unstale":"2024-03-31T15:05:27.063860+0000","last_undegraded":"2024-03-31T15:05:27.063860+0000","last_fullsized":"2024-03-31T15:05:27.063860+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:05:25.044753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:05:25.044753+0000","last_clean_scrub_stamp":"2024-03-31T15:05:25.044753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:18:33.552147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-31T15:05:27.080654+0000","last_change":"2024-03-31T15:05:27.076777+0000","last_active":"2024-03-31T15:05:27.080654+0000","last_peered":"2024-03-31T15:05:27.080654+0000","last_clean":"2024-03-31T15:05:27.080654+0000","last_became_active":"2024-03-31T15:05:27.076515+0000","last_became_peered":"2024-03-31T15:05:27.076515+0000","last_unstale":"2024-03-31T15:05:27.080654+0000","last_undegraded":"2024-03-31T15:05:27.080654+0000","last_fullsized":"2024-03-31T15:05:27.080654+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_clean_scrub_stamp":"2024-03-31T15:05:26.050987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T16:16:23.687703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-31T15:05:28.089018+0000","last_change":"2024-03-31T15:05:26.084900+0000","last_active":"2024-03-31T15:05:28.089018+0000","last_peered":"2024-03-31T15:05:28.089018+0000","last_clean":"2024-03-31T15:05:28.089018+0000","last_became_active":"2024-03-31T15:05:26.084487+0000","last_became_peered":"2024-03-31T15:05:26.084487+0000","last_unstale":"2024-03-31T15:05:28.089018+0000","last_undegraded":"2024-03-31T15:05:28.089018+0000","last_fullsized":"2024-03-31T15:05:28.089018+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:05:25.044753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:05:25.044753+0000","last_clean_scrub_stamp":"2024-03-31T15:05:25.044753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:22:39.914616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-31T15:05:27.090323+0000","last_change":"2024-03-31T15:05:27.084780+0000","last_active":"2024-03-31T15:05:27.090323+0000","last_peered":"2024-03-31T15:05:27.090323+0000","last_clean":"2024-03-31T15:05:27.090323+0000","last_became_active":"2024-03-31T15:05:27.084479+0000","last_became_peered":"2024-03-31T15:05:27.084479+0000","last_unstale":"2024-03-31T15:05:27.090323+0000","last_undegraded":"2024-03-31T15:05:27.090323+0000","last_fullsized":"2024-03-31T15:05:27.090323+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_clean_scrub_stamp":"2024-03-31T15:05:26.050987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:07:05.236765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-31T15:05:27.075356+0000","last_change":"2024-03-31T15:05:26.086483+0000","last_active":"2024-03-31T15:05:27.075356+0000","last_peered":"2024-03-31T15:05:27.075356+0000","last_clean":"2024-03-31T15:05:27.075356+0000","last_became_active":"2024-03-31T15:05:26.086208+0000","last_became_peered":"2024-03-31T15:05:26.086208+0000","last_unstale":"2024-03-31T15:05:27.075356+0000","last_undegraded":"2024-03-31T15:05:27.075356+0000","last_fullsized":"2024-03-31T15:05:27.075356+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:05:25.044753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:05:25.044753+0000","last_clean_scrub_stamp":"2024-03-31T15:05:25.044753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:23:55.245030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-31T15:05:27.090106+0000","last_change":"2024-03-31T15:05:27.081617+0000","last_active":"2024-03-31T15:05:27.090106+0000","last_peered":"2024-03-31T15:05:27.090106+0000","last_clean":"2024-03-31T15:05:27.090106+0000","last_became_active":"2024-03-31T15:05:27.081440+0000","last_became_peered":"2024-03-31T15:05:27.081440+0000","last_unstale":"2024-03-31T15:05:27.090106+0000","last_undegraded":"2024-03-31T15:05:27.090106+0000","last_fullsized":"2024-03-31T15:05:27.090106+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_clean_scrub_stamp":"2024-03-31T15:05:26.050987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T17:19:47.130905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-31T15:05:28.089771+0000","last_change":"2024-03-31T15:05:26.082448+0000","last_active":"2024-03-31T15:05:28.089771+0000","last_peered":"2024-03-31T15:05:28.089771+0000","last_clean":"2024-03-31T15:05:28.089771+0000","last_became_active":"2024-03-31T15:05:26.082046+0000","last_became_peered":"2024-03-31T15:05:26.082046+0000","last_unstale":"2024-03-31T15:05:28.089771+0000","last_undegraded":"2024-03-31T15:05:28.089771+0000","last_fullsized":"2024-03-31T15:05:28.089771+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:05:25.044753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:05:25.044753+0000","last_clean_scrub_stamp":"2024-03-31T15:05:25.044753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T17:09:11.792443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-31T15:05:28.089546+0000","last_change":"2024-03-31T15:05:27.093776+0000","last_active":"2024-03-31T15:05:28.089546+0000","last_peered":"2024-03-31T15:05:28.089546+0000","last_clean":"2024-03-31T15:05:28.089546+0000","last_became_active":"2024-03-31T15:05:27.093513+0000","last_became_peered":"2024-03-31T15:05:27.093513+0000","last_unstale":"2024-03-31T15:05:28.089546+0000","last_undegraded":"2024-03-31T15:05:28.089546+0000","last_fullsized":"2024-03-31T15:05:28.089546+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_clean_scrub_stamp":"2024-03-31T15:05:26.050987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T16:12:10.481371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-31T15:05:27.068619+0000","last_change":"2024-03-31T15:05:26.074576+0000","last_active":"2024-03-31T15:05:27.068619+0000","last_peered":"2024-03-31T15:05:27.068619+0000","last_clean":"2024-03-31T15:05:27.068619+0000","last_became_active":"2024-03-31T15:05:26.074220+0000","last_became_peered":"2024-03-31T15:05:26.074220+0000","last_unstale":"2024-03-31T15:05:27.068619+0000","last_undegraded":"2024-03-31T15:05:27.068619+0000","last_fullsized":"2024-03-31T15:05:27.068619+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:05:25.044753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:05:25.044753+0000","last_clean_scrub_stamp":"2024-03-31T15:05:25.044753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:48:23.115507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-31T15:05:27.094483+0000","last_change":"2024-03-31T15:05:27.092068+0000","last_active":"2024-03-31T15:05:27.094483+0000","last_peered":"2024-03-31T15:05:27.094483+0000","last_clean":"2024-03-31T15:05:27.094483+0000","last_became_active":"2024-03-31T15:05:27.091670+0000","last_became_peered":"2024-03-31T15:05:27.091670+0000","last_unstale":"2024-03-31T15:05:27.094483+0000","last_undegraded":"2024-03-31T15:05:27.094483+0000","last_fullsized":"2024-03-31T15:05:27.094483+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_clean_scrub_stamp":"2024-03-31T15:05:26.050987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:36:11.111173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-31T15:05:28.089604+0000","last_change":"2024-03-31T15:05:26.080545+0000","last_active":"2024-03-31T15:05:28.089604+0000","last_peered":"2024-03-31T15:05:28.089604+0000","last_clean":"2024-03-31T15:05:28.089604+0000","last_became_active":"2024-03-31T15:05:26.080246+0000","last_became_peered":"2024-03-31T15:05:26.080246+0000","last_unstale":"2024-03-31T15:05:28.089604+0000","last_undegraded":"2024-03-31T15:05:28.089604+0000","last_fullsized":"2024-03-31T15:05:28.089604+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:05:25.044753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:05:25.044753+0000","last_clean_scrub_stamp":"2024-03-31T15:05:25.044753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T16:42:26.707767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-31T15:05:27.091882+0000","last_change":"2024-03-31T15:05:27.082977+0000","last_active":"2024-03-31T15:05:27.091882+0000","last_peered":"2024-03-31T15:05:27.091882+0000","last_clean":"2024-03-31T15:05:27.091882+0000","last_became_active":"2024-03-31T15:05:27.082310+0000","last_became_peered":"2024-03-31T15:05:27.082310+0000","last_unstale":"2024-03-31T15:05:27.091882+0000","last_undegraded":"2024-03-31T15:05:27.091882+0000","last_fullsized":"2024-03-31T15:05:27.091882+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_clean_scrub_stamp":"2024-03-31T15:05:26.050987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T15:27:50.855100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-31T15:05:28.092866+0000","last_change":"2024-03-31T15:05:26.084927+0000","last_active":"2024-03-31T15:05:28.092866+0000","last_peered":"2024-03-31T15:05:28.092866+0000","last_clean":"2024-03-31T15:05:28.092866+0000","last_became_active":"2024-03-31T15:05:26.084374+0000","last_became_peered":"2024-03-31T15:05:26.084374+0000","last_unstale":"2024-03-31T15:05:28.092866+0000","last_undegraded":"2024-03-31T15:05:28.092866+0000","last_fullsized":"2024-03-31T15:05:28.092866+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:05:25.044753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:05:25.044753+0000","last_clean_scrub_stamp":"2024-03-31T15:05:25.044753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T19:56:56.774930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-31T15:05:28.163891+0000","last_change":"2024-03-31T15:05:27.084713+0000","last_active":"2024-03-31T15:05:28.163891+0000","last_peered":"2024-03-31T15:05:28.163891+0000","last_clean":"2024-03-31T15:05:28.163891+0000","last_became_active":"2024-03-31T15:05:27.082242+0000","last_became_peered":"2024-03-31T15:05:27.082242+0000","last_unstale":"2024-03-31T15:05:28.163891+0000","last_undegraded":"2024-03-31T15:05:28.163891+0000","last_fullsized":"2024-03-31T15:05:28.163891+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_clean_scrub_stamp":"2024-03-31T15:05:26.050987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:13:39.279396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-31T15:05:27.071927+0000","last_change":"2024-03-31T15:05:26.076418+0000","last_active":"2024-03-31T15:05:27.071927+0000","last_peered":"2024-03-31T15:05:27.071927+0000","last_clean":"2024-03-31T15:05:27.071927+0000","last_became_active":"2024-03-31T15:05:26.074431+0000","last_became_peered":"2024-03-31T15:05:26.074431+0000","last_unstale":"2024-03-31T15:05:27.071927+0000","last_undegraded":"2024-03-31T15:05:27.071927+0000","last_fullsized":"2024-03-31T15:05:27.071927+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:05:25.044753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:05:25.044753+0000","last_clean_scrub_stamp":"2024-03-31T15:05:25.044753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T15:10:56.160318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-31T15:05:28.092624+0000","last_change":"2024-03-31T15:05:27.085017+0000","last_active":"2024-03-31T15:05:28.092624+0000","last_peered":"2024-03-31T15:05:28.092624+0000","last_clean":"2024-03-31T15:05:28.092624+0000","last_became_active":"2024-03-31T15:05:27.084153+0000","last_became_peered":"2024-03-31T15:05:27.084153+0000","last_unstale":"2024-03-31T15:05:28.092624+0000","last_undegraded":"2024-03-31T15:05:28.092624+0000","last_fullsized":"2024-03-31T15:05:28.092624+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_clean_scrub_stamp":"2024-03-31T15:05:26.050987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T17:50:38.641040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-31T15:05:27.064494+0000","last_change":"2024-03-31T15:05:26.087612+0000","last_active":"2024-03-31T15:05:27.064494+0000","last_peered":"2024-03-31T15:05:27.064494+0000","last_clean":"2024-03-31T15:05:27.064494+0000","last_became_active":"2024-03-31T15:05:26.087282+0000","last_became_peered":"2024-03-31T15:05:26.087282+0000","last_unstale":"2024-03-31T15:05:27.064494+0000","last_undegraded":"2024-03-31T15:05:27.064494+0000","last_fullsized":"2024-03-31T15:05:27.064494+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:05:25.044753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:05:25.044753+0000","last_clean_scrub_stamp":"2024-03-31T15:05:25.044753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T16:37:33.988071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-31T15:05:27.094860+0000","last_change":"2024-03-31T15:05:27.088737+0000","last_active":"2024-03-31T15:05:27.094860+0000","last_peered":"2024-03-31T15:05:27.094860+0000","last_clean":"2024-03-31T15:05:27.094860+0000","last_became_active":"2024-03-31T15:05:27.088320+0000","last_became_peered":"2024-03-31T15:05:27.088320+0000","last_unstale":"2024-03-31T15:05:27.094860+0000","last_undegraded":"2024-03-31T15:05:27.094860+0000","last_fullsized":"2024-03-31T15:05:27.094860+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:05:26.050987+0000","last_clean_scrub_stamp":"2024-03-31T15:05:26.050987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:30:51.046353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.7","version":"44'2","reported_seq":16,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-31T15:05:28.164278+0000","last_change":"2024-03-31T15:05:26.082613+0000","last_active":"2024-03-31T15:05:28.164278+0000","last_peered":"2024-03-31T15:05:28.164278+0000","last_clean":"2024-03-31T15:05:28.164278+0000","last_became_active":"2024-03-31T15:05:26.082334+0000","last_became_peered":"2024-03-31T15:05:26.082334+0000","last_unstale":"2024-03-31T15:05:28.164278+0000","last_undegraded":"2024-03-31T15:05:28.164278+0000","last_fullsized":"2024-03-31T15:05:28.164278+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:05:25.044753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:05:25.044753+0000","last_clean_scrub_stamp":"2024-03-31T15:05:25.044753+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:22:39.914616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":215,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-31T15:05:28.089081+0000","last_change":"2024-03-31T15:03:24.329890+0000","last_active":"2024-03-31T15:05:28.089081+0000","last_peered":"2024-03-31T15:05:28.089081+0000","last_clean":"2024-03-31T15:05:28.089081+0000","last_became_active":"2024-03-31T15:03:24.328882+0000","last_became_peered":"2024-03-31T15:03:24.328882+0000","last_unstale":"2024-03-31T15:05:28.089081+0000","last_undegraded":"2024-03-31T15:05:28.089081+0000","last_fullsized":"2024-03-31T15:05:28.089081+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:22.885535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:22.885535+0000","last_clean_scrub_stamp":"2024-03-31T15:03:22.885535+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:18:29.004296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509470,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28120,"kb_used_data":1092,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343720,"statfs":{"total":96636764160,"available":96607969280,"internally_reserved":0,"allocated":1118208,"data_stored":802219,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509469,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27532,"kb_used_data":512,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344308,"statfs":{"total":96636764160,"available":96608571392,"internally_reserved":0,"allocated":524288,"data_stored":211831,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":15,"seq":64424509468,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27536,"kb_used_data":508,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344304,"statfs":{"total":96636764160,"available":96608567296,"internally_reserved":0,"allocated":520192,"data_stored":207073,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509467,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31704,"kb_used_data":524,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340136,"statfs":{"total":96636764160,"available":96604299264,"internally_reserved":0,"allocated":536576,"data_stored":213395,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542173,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31684,"kb_used_data":504,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340156,"statfs":{"total":96636764160,"available":96604319744,"internally_reserved":0,"allocated":516096,"data_stored":206133,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542173,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31688,"kb_used_data":508,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340152,"statfs":{"total":96636764160,"available":96604315648,"internally_reserved":0,"allocated":520192,"data_stored":206099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542173,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31680,"kb_used_data":500,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340160,"statfs":{"total":96636764160,"available":96604323840,"internally_reserved":0,"allocated":512000,"data_stored":205531,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542173,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32264,"kb_used_data":1080,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339576,"statfs":{"total":96636764160,"available":96603725824,"internally_reserved":0,"allocated":1105920,"data_stored":795863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-31T15:05:31.607 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-31T15:05:31.607 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-03-31T15:05:31.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:31.799+0000 7fc027c00640 1 Processor -- start 2024-03-31T15:05:31.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:31.799+0000 7fc027c00640 1 -- start start 2024-03-31T15:05:31.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:31.803+0000 7fc027c00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fc020070030 0x7fc020143c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:31.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:31.803+0000 7fc027c00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fc020141590 0x7fc020144170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:31.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:31.803+0000 7fc027c00640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc0201446b0 con 0x7fc0201411b0 2024-03-31T15:05:31.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:31.803+0000 7fc027c00640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7fc02015f910 con 0x7fc020070030 2024-03-31T15:05:31.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:31.803+0000 7fc027c00640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7fc020162910 con 0x7fc020141590 2024-03-31T15:05:31.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:31.803+0000 7fc01fe00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fc020070030 0x7fc020143c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:31.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:31.803+0000 7fc01fe00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fc020070030 0x7fc020143c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34188/0 (socket says 172.21.15.16:34188) 2024-03-31T15:05:31.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:31.803+0000 7fc01fe00640 1 -- 172.21.15.16:0/2329118850 learned_addr learned my addr 172.21.15.16:0/2329118850 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:05:31.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:31.803+0000 7fc01fe00640 1 -- 172.21.15.16:0/2329118850 >> v2:172.21.15.89:3300/0 conn(0x7fc020141590 msgr2=0x7fc020144170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:05:31.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:31.803+0000 7fc025c00640 1 --2- 172.21.15.16:0/2329118850 >> v2:172.21.15.89:3300/0 conn(0x7fc020141590 0x7fc020144170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:31.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:31.803+0000 7fc01fe00640 1 --2- 172.21.15.16:0/2329118850 >> v2:172.21.15.89:3300/0 conn(0x7fc020141590 0x7fc020144170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:31.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:31.803+0000 7fc01fe00640 1 -- 172.21.15.16:0/2329118850 >> v1:172.21.15.89:6789/0 conn(0x7fc0201411b0 legacy=0x7fc02006f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:05:31.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:31.803+0000 7fc01fe00640 1 -- 172.21.15.16:0/2329118850 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc020144980 con 0x7fc020070030 2024-03-31T15:05:31.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:31.803+0000 7fc025c00640 1 --2- 172.21.15.16:0/2329118850 >> v2:172.21.15.89:3300/0 conn(0x7fc020141590 0x7fc020144170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:05:31.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:31.803+0000 7fc01fe00640 1 --2- 172.21.15.16:0/2329118850 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc020070030 0x7fc020143c30 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7fc00c009870 tx=0x7fc00c01c4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=91e69861aca451e2 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:05:31.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:31.807+0000 7fc01f400640 1 -- 172.21.15.16:0/2329118850 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc00c0aa610 con 0x7fc020070030 2024-03-31T15:05:31.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:31.807+0000 7fc01f400640 1 -- 172.21.15.16:0/2329118850 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc00c0af070 con 0x7fc020070030 2024-03-31T15:05:31.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:31.807+0000 7fc01f400640 1 -- 172.21.15.16:0/2329118850 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc00c0aa9d0 con 0x7fc020070030 2024-03-31T15:05:31.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:31.807+0000 7fc027c00640 1 -- 172.21.15.16:0/2329118850 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc020070030 msgr2=0x7fc020143c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:31.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:31.807+0000 7fc027c00640 1 --2- 172.21.15.16:0/2329118850 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc020070030 0x7fc020143c30 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7fc00c009870 tx=0x7fc00c01c4b0 comp rx=0 tx=0).stop 2024-03-31T15:05:31.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:31.807+0000 7fc027c00640 1 -- 172.21.15.16:0/2329118850 shutdown_connections 2024-03-31T15:05:31.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:31.807+0000 7fc027c00640 1 --2- 172.21.15.16:0/2329118850 >> v2:172.21.15.89:3300/0 conn(0x7fc020141590 0x7fc020144170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:31.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:31.807+0000 7fc027c00640 1 --2- 172.21.15.16:0/2329118850 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc020070030 0x7fc020143c30 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:31.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:31.807+0000 7fc027c00640 1 -- 172.21.15.16:0/2329118850 >> 172.21.15.16:0/2329118850 conn(0x7fc020067930 msgr2=0x7fc0200b3ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:05:31.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:31.807+0000 7fc027c00640 1 -- 172.21.15.16:0/2329118850 shutdown_connections 2024-03-31T15:05:31.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:31.807+0000 7fc027c00640 1 -- 172.21.15.16:0/2329118850 wait complete. 2024-03-31T15:05:31.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:31.807+0000 7fc027c00640 1 Processor -- start 2024-03-31T15:05:31.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:31.807+0000 7fc027c00640 1 -- start start 2024-03-31T15:05:31.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:31.811+0000 7fc027c00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc020070030 0x7fc02013e490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:31.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:31.811+0000 7fc027c00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fc0201411b0 0x7fc02013ebf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:31.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:31.811+0000 7fc027c00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc020141590 0x7fc0201575d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:31.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:31.811+0000 7fc027c00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fc02015f110 con 0x7fc020070030 2024-03-31T15:05:31.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:31.811+0000 7fc027c00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fc020162da0 con 0x7fc020141590 2024-03-31T15:05:31.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:31.811+0000 7fc027c00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fc02015f290 con 0x7fc0201411b0 2024-03-31T15:05:31.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:31.811+0000 7fc025200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc020070030 0x7fc02013e490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:31.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:31.811+0000 7fc025200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc020070030 0x7fc02013e490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34202/0 (socket says 172.21.15.16:34202) 2024-03-31T15:05:31.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:31.811+0000 7fc025200640 1 -- 172.21.15.16:0/1183571305 learned_addr learned my addr 172.21.15.16:0/1183571305 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:05:31.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:31.811+0000 7fc01fe00640 1 --2- 172.21.15.16:0/1183571305 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fc0201411b0 0x7fc02013ebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:31.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:31.811+0000 7fc025c00640 1 --2- 172.21.15.16:0/1183571305 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc020141590 0x7fc0201575d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:31.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:31.811+0000 7fc025200640 1 -- 172.21.15.16:0/1183571305 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fc0201411b0 msgr2=0x7fc02013ebf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:31.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:31.811+0000 7fc025200640 1 --2- 172.21.15.16:0/1183571305 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fc0201411b0 0x7fc02013ebf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:31.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:31.811+0000 7fc025200640 1 -- 172.21.15.16:0/1183571305 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc020141590 msgr2=0x7fc0201575d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:31.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:31.811+0000 7fc025200640 1 --2- 172.21.15.16:0/1183571305 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc020141590 0x7fc0201575d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:31.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:31.811+0000 7fc025200640 1 -- 172.21.15.16:0/1183571305 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc020156260 con 0x7fc020070030 2024-03-31T15:05:31.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:31.811+0000 7fc025c00640 1 --2- 172.21.15.16:0/1183571305 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc020141590 0x7fc0201575d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:05:31.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:31.811+0000 7fc01fe00640 1 --2- 172.21.15.16:0/1183571305 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fc0201411b0 0x7fc02013ebf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:05:31.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:31.811+0000 7fc025200640 1 --2- 172.21.15.16:0/1183571305 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc020070030 0x7fc02013e490 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7fc01400c4c0 tx=0x7fc014001e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:05:31.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:31.811+0000 7fc01cc00640 1 -- 172.21.15.16:0/1183571305 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc014017070 con 0x7fc020070030 2024-03-31T15:05:31.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:31.811+0000 7fc01cc00640 1 -- 172.21.15.16:0/1183571305 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc014003600 con 0x7fc020070030 2024-03-31T15:05:31.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:31.811+0000 7fc01cc00640 1 -- 172.21.15.16:0/1183571305 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0140039f0 con 0x7fc020070030 2024-03-31T15:05:31.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:31.811+0000 7fc027c00640 1 -- 172.21.15.16:0/1183571305 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc02013e9d0 con 0x7fc020070030 2024-03-31T15:05:31.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:31.811+0000 7fc027c00640 1 -- 172.21.15.16:0/1183571305 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc02013f640 con 0x7fc020070030 2024-03-31T15:05:31.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:31.811+0000 7fc027c00640 1 -- 172.21.15.16:0/1183571305 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc020069950 con 0x7fc020070030 2024-03-31T15:05:31.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:31.815+0000 7fc01cc00640 1 -- 172.21.15.16:0/1183571305 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fc01401a070 con 0x7fc020070030 2024-03-31T15:05:31.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:31.815+0000 7fc01cc00640 1 --2- 172.21.15.16:0/1183571305 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fc000068380 0x7fc00006a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:31.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:31.815+0000 7fc01cc00640 1 -- 172.21.15.16:0/1183571305 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5709+0+0 (secure 0 0 0) 0x7fc014085d90 con 0x7fc020070030 2024-03-31T15:05:31.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:31.815+0000 7fc01fe00640 1 --2- 172.21.15.16:0/1183571305 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fc000068380 0x7fc00006a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:31.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:31.815+0000 7fc01fe00640 1 --2- 172.21.15.16:0/1183571305 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fc000068380 0x7fc00006a840 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fc00c0ae000 tx=0x7fc00c0a7000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:05:31.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:31.819+0000 7fc01cc00640 1 -- 172.21.15.16:0/1183571305 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fc014056340 con 0x7fc020070030 2024-03-31T15:05:32.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:32.043+0000 7fc027c00640 1 -- 172.21.15.16:0/1183571305 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) -- 0x7fc02015cc40 con 0x7fc020070030 2024-03-31T15:05:32.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:32.043+0000 7fc01cc00640 1 -- 172.21.15.16:0/1183571305 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v14) ==== 190+0+147 (secure 0 0 0) 0x7fc014055c40 con 0x7fc020070030 2024-03-31T15:05:32.040 INFO:teuthology.orchestra.run.smithi016.stdout:[client.0] 2024-03-31T15:05:32.040 INFO:teuthology.orchestra.run.smithi016.stdout: key = AQAPewlmlMIDEhAAF0Ll1HcrWHmH6xDKq2Qq9A== 2024-03-31T15:05:32.041 INFO:teuthology.orchestra.run.smithi016.stdout: caps mds = "allow" 2024-03-31T15:05:32.041 INFO:teuthology.orchestra.run.smithi016.stdout: caps mon = "allow r" 2024-03-31T15:05:32.041 INFO:teuthology.orchestra.run.smithi016.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-03-31T15:05:32.041 INFO:teuthology.orchestra.run.smithi016.stderr:updated caps for client.0 2024-03-31T15:05:32.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:32.047+0000 7fc027c00640 1 -- 172.21.15.16:0/1183571305 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fc000068380 msgr2=0x7fc00006a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:32.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:32.047+0000 7fc027c00640 1 --2- 172.21.15.16:0/1183571305 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fc000068380 0x7fc00006a840 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fc00c0ae000 tx=0x7fc00c0a7000 comp rx=0 tx=0).stop 2024-03-31T15:05:32.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:32.047+0000 7fc027c00640 1 -- 172.21.15.16:0/1183571305 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc020070030 msgr2=0x7fc02013e490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:32.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:32.047+0000 7fc027c00640 1 --2- 172.21.15.16:0/1183571305 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc020070030 0x7fc02013e490 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7fc01400c4c0 tx=0x7fc014001e80 comp rx=0 tx=0).stop 2024-03-31T15:05:32.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:32.047+0000 7fc027c00640 1 -- 172.21.15.16:0/1183571305 shutdown_connections 2024-03-31T15:05:32.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:32.047+0000 7fc027c00640 1 --2- 172.21.15.16:0/1183571305 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fc000068380 0x7fc00006a840 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:32.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:32.047+0000 7fc027c00640 1 --2- 172.21.15.16:0/1183571305 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc020141590 0x7fc0201575d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:32.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:32.047+0000 7fc027c00640 1 --2- 172.21.15.16:0/1183571305 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fc0201411b0 0x7fc02013ebf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:32.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:32.047+0000 7fc027c00640 1 --2- 172.21.15.16:0/1183571305 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc020070030 0x7fc02013e490 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:32.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:32.047+0000 7fc027c00640 1 -- 172.21.15.16:0/1183571305 >> 172.21.15.16:0/1183571305 conn(0x7fc020067930 msgr2=0x7fc020160160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:05:32.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:32.047+0000 7fc027c00640 1 -- 172.21.15.16:0/1183571305 shutdown_connections 2024-03-31T15:05:32.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:32.047+0000 7fc027c00640 1 -- 172.21.15.16:0/1183571305 wait complete. 2024-03-31T15:05:32.062 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-31T15:05:32.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:32.207+0000 7f4cfd400640 1 Processor -- start 2024-03-31T15:05:32.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:32.207+0000 7f4cfd400640 1 -- start start 2024-03-31T15:05:32.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:32.207+0000 7f4cfd400640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f4cf8145e40 0x7f4cf80cc740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:32.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:32.207+0000 7f4cfd400640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f4cf8070030 0x7f4cf80d2010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:32.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:32.207+0000 7f4cfd400640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4cf80d0920 con 0x7f4cf80ccc80 2024-03-31T15:05:32.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:32.207+0000 7f4cfd400640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f4cf80d06d0 con 0x7f4cf8070030 2024-03-31T15:05:32.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:32.207+0000 7f4cfd400640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f4cf80d0c00 con 0x7f4cf8145e40 2024-03-31T15:05:32.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:32.207+0000 7f4cede00640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f4cf80ccc80 0x7f4cf806f920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:35542/0 (socket says 172.21.15.16:35542) 2024-03-31T15:05:32.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:32.207+0000 7f4cede00640 1 -- 172.21.15.16:0/2224915389 learned_addr learned my addr 172.21.15.16:0/2224915389 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:05:32.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:32.211+0000 7f4cf7200640 1 --2- 172.21.15.16:0/2224915389 >> v2:172.21.15.89:3300/0 conn(0x7f4cf8070030 0x7f4cf80d2010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:32.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:32.211+0000 7f4cf6800640 1 --2- 172.21.15.16:0/2224915389 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f4cf8145e40 0x7f4cf80cc740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:32.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:32.211+0000 7f4cf5e00640 1 -- 172.21.15.16:0/2224915389 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 997501482 0 0) 0x7f4cf80d0920 con 0x7f4cf80ccc80 2024-03-31T15:05:32.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:32.211+0000 7f4cf5e00640 1 -- 172.21.15.16:0/2224915389 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4cdc003530 con 0x7f4cf80ccc80 2024-03-31T15:05:32.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:32.211+0000 7f4cf6800640 1 -- 172.21.15.16:0/2224915389 >> v2:172.21.15.89:3300/0 conn(0x7f4cf8070030 msgr2=0x7f4cf80d2010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:05:32.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:32.211+0000 7f4cf6800640 1 --2- 172.21.15.16:0/2224915389 >> v2:172.21.15.89:3300/0 conn(0x7f4cf8070030 0x7f4cf80d2010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:32.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:32.211+0000 7f4cf6800640 1 -- 172.21.15.16:0/2224915389 >> v1:172.21.15.16:6789/0 conn(0x7f4cf80ccc80 legacy=0x7f4cf806f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:32.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:32.211+0000 7f4cf6800640 1 -- 172.21.15.16:0/2224915389 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4cf80d1810 con 0x7f4cf8145e40 2024-03-31T15:05:32.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:32.211+0000 7f4cf7200640 1 --2- 172.21.15.16:0/2224915389 >> v2:172.21.15.89:3300/0 conn(0x7f4cf8070030 0x7f4cf80d2010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:05:32.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:32.211+0000 7f4cf6800640 1 --2- 172.21.15.16:0/2224915389 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f4cf8145e40 0x7f4cf80cc740 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f4ce80097c0 tx=0x7f4ce801c4a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=16d1ec8184b4fafe server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:05:32.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:32.211+0000 7f4cf5e00640 1 -- 172.21.15.16:0/2224915389 <== mon.2 v2:172.21.15.89:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ce80b1070 con 0x7f4cf8145e40 2024-03-31T15:05:32.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:32.211+0000 7f4cf5e00640 1 -- 172.21.15.16:0/2224915389 <== mon.2 v2:172.21.15.89:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4ce80ac410 con 0x7f4cf8145e40 2024-03-31T15:05:32.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:32.211+0000 7f4cf5e00640 1 -- 172.21.15.16:0/2224915389 <== mon.2 v2:172.21.15.89:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ce80ac7e0 con 0x7f4cf8145e40 2024-03-31T15:05:32.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:32.215+0000 7f4cfd400640 1 -- 172.21.15.16:0/2224915389 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f4cf8145e40 msgr2=0x7f4cf80cc740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:32.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:32.215+0000 7f4cfd400640 1 --2- 172.21.15.16:0/2224915389 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f4cf8145e40 0x7f4cf80cc740 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f4ce80097c0 tx=0x7f4ce801c4a0 comp rx=0 tx=0).stop 2024-03-31T15:05:32.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:32.215+0000 7f4cfd400640 1 -- 172.21.15.16:0/2224915389 shutdown_connections 2024-03-31T15:05:32.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:32.215+0000 7f4cfd400640 1 --2- 172.21.15.16:0/2224915389 >> v2:172.21.15.89:3300/0 conn(0x7f4cf8070030 0x7f4cf80d2010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:32.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:32.215+0000 7f4cfd400640 1 --2- 172.21.15.16:0/2224915389 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f4cf8145e40 0x7f4cf80cc740 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:32.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:32.215+0000 7f4cfd400640 1 -- 172.21.15.16:0/2224915389 >> 172.21.15.16:0/2224915389 conn(0x7f4cf8067330 msgr2=0x7f4cf8067740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:05:32.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:32.215+0000 7f4cfd400640 1 -- 172.21.15.16:0/2224915389 shutdown_connections 2024-03-31T15:05:32.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:32.215+0000 7f4cfd400640 1 -- 172.21.15.16:0/2224915389 wait complete. 2024-03-31T15:05:32.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:32.215+0000 7f4cfd400640 1 Processor -- start 2024-03-31T15:05:32.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:32.215+0000 7f4cfd400640 1 -- start start 2024-03-31T15:05:32.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:32.215+0000 7f4cfd400640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4cf8070030 0x7f4cf8152d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:32.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:32.215+0000 7f4cfd400640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f4cf80ccc80 0x7f4cf8135260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:32.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:32.215+0000 7f4cfd400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4cf815d550 0x7f4cf81357a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:32.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:32.215+0000 7f4cfd400640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f4cf80b5240 con 0x7f4cf815d550 2024-03-31T15:05:32.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:32.215+0000 7f4cfd400640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f4cf80b50c0 con 0x7f4cf8070030 2024-03-31T15:05:32.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:32.215+0000 7f4cfd400640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f4cf8169190 con 0x7f4cf80ccc80 2024-03-31T15:05:32.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:32.215+0000 7f4cf7200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4cf815d550 0x7f4cf81357a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:32.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:32.215+0000 7f4cf7200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4cf815d550 0x7f4cf81357a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34212/0 (socket says 172.21.15.16:34212) 2024-03-31T15:05:32.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:32.215+0000 7f4cf7200640 1 -- 172.21.15.16:0/3801373706 learned_addr learned my addr 172.21.15.16:0/3801373706 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:05:32.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:32.215+0000 7f4cf6800640 1 --2- 172.21.15.16:0/3801373706 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4cf8070030 0x7f4cf8152d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:32.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:32.215+0000 7f4cede00640 1 --2- 172.21.15.16:0/3801373706 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f4cf80ccc80 0x7f4cf8135260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:32.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:32.215+0000 7f4cf7200640 1 -- 172.21.15.16:0/3801373706 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f4cf80ccc80 msgr2=0x7f4cf8135260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:32.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:32.215+0000 7f4cf7200640 1 --2- 172.21.15.16:0/3801373706 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f4cf80ccc80 0x7f4cf8135260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:32.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:32.215+0000 7f4cf7200640 1 -- 172.21.15.16:0/3801373706 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4cf8070030 msgr2=0x7f4cf8152d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:32.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:32.215+0000 7f4cf7200640 1 --2- 172.21.15.16:0/3801373706 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4cf8070030 0x7f4cf8152d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:32.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:32.215+0000 7f4cf7200640 1 -- 172.21.15.16:0/3801373706 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4cf8153250 con 0x7f4cf815d550 2024-03-31T15:05:32.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:32.219+0000 7f4cf7200640 1 --2- 172.21.15.16:0/3801373706 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4cf815d550 0x7f4cf81357a0 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f4cf000efc0 tx=0x7f4cf0009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:05:32.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:32.219+0000 7f4cef400640 1 -- 172.21.15.16:0/3801373706 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cf0018030 con 0x7f4cf815d550 2024-03-31T15:05:32.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:32.219+0000 7f4cef400640 1 -- 172.21.15.16:0/3801373706 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4cf000cde0 con 0x7f4cf815d550 2024-03-31T15:05:32.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:32.219+0000 7f4cef400640 1 -- 172.21.15.16:0/3801373706 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cf0003bc0 con 0x7f4cf815d550 2024-03-31T15:05:32.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:32.219+0000 7f4cfd400640 1 -- 172.21.15.16:0/3801373706 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4cf815ad50 con 0x7f4cf815d550 2024-03-31T15:05:32.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:32.219+0000 7f4cfd400640 1 -- 172.21.15.16:0/3801373706 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4cf815b310 con 0x7f4cf815d550 2024-03-31T15:05:32.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:32.219+0000 7f4cfd400640 1 -- 172.21.15.16:0/3801373706 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4cf8069950 con 0x7f4cf815d550 2024-03-31T15:05:32.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:32.219+0000 7f4cef400640 1 -- 172.21.15.16:0/3801373706 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f4cf001b070 con 0x7f4cf815d550 2024-03-31T15:05:32.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:32.219+0000 7f4cef400640 1 --2- 172.21.15.16:0/3801373706 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f4cd4068380 0x7f4cd406a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:32.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:32.223+0000 7f4cef400640 1 -- 172.21.15.16:0/3801373706 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5709+0+0 (secure 0 0 0) 0x7f4cf0087260 con 0x7f4cf815d550 2024-03-31T15:05:32.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:32.223+0000 7f4cf6800640 1 --2- 172.21.15.16:0/3801373706 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f4cd4068380 0x7f4cd406a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:32.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:32.223+0000 7f4cf6800640 1 --2- 172.21.15.16:0/3801373706 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f4cd4068380 0x7f4cd406a840 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f4ce80ae000 tx=0x7f4ce8008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:05:32.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:32.227+0000 7f4cef400640 1 -- 172.21.15.16:0/3801373706 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f4cf0022070 con 0x7f4cf815d550 2024-03-31T15:05:32.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:32.427+0000 7f4cfd400640 1 -- 172.21.15.16:0/3801373706 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4cf80cf3c0 con 0x7f4cf815d550 2024-03-31T15:05:32.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:32.427+0000 7f4cef400640 1 -- 172.21.15.16:0/3801373706 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+5175 (secure 0 0 0) 0x7f4cf0010040 con 0x7f4cf815d550 2024-03-31T15:05:32.421 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:05:32.422 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":25,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/285985105","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6835","nonce":285985105},{"type":"v1","addr":"172.21.15.89:6837","nonce":285985105}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20},{"gid":5689,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6836/1615931247","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":1615931247},{"type":"v1","addr":"172.21.15.89:6836","nonce":1615931247}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20},{"gid":5694,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/512187852","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6833","nonce":512187852},{"type":"v1","addr":"172.21.15.16:6835","nonce":512187852}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-31T15:05:26.461908+0000","modified":"2024-03-31T15:05:29.491163+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5697},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5697":{"gid":5697,"name":"a","rank":0,"incarnation":21,"state":"up:active","state_seq":3,"addr":"172.21.15.16:6834/2768973409","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":2768973409},{"type":"v1","addr":"172.21.15.16:6834","nonce":2768973409}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5697,"qdb_cluster":[5697]},"id":3}]} 2024-03-31T15:05:32.422 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 25 2024-03-31T15:05:32.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:32.427+0000 7f4cfd400640 1 -- 172.21.15.16:0/3801373706 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f4cd4068380 msgr2=0x7f4cd406a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:32.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:32.427+0000 7f4cfd400640 1 --2- 172.21.15.16:0/3801373706 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f4cd4068380 0x7f4cd406a840 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f4ce80ae000 tx=0x7f4ce8008040 comp rx=0 tx=0).stop 2024-03-31T15:05:32.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:32.427+0000 7f4cfd400640 1 -- 172.21.15.16:0/3801373706 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4cf815d550 msgr2=0x7f4cf81357a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:32.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:32.427+0000 7f4cfd400640 1 --2- 172.21.15.16:0/3801373706 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4cf815d550 0x7f4cf81357a0 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f4cf000efc0 tx=0x7f4cf0009f90 comp rx=0 tx=0).stop 2024-03-31T15:05:32.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:32.427+0000 7f4cfd400640 1 -- 172.21.15.16:0/3801373706 shutdown_connections 2024-03-31T15:05:32.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:32.427+0000 7f4cfd400640 1 --2- 172.21.15.16:0/3801373706 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f4cd4068380 0x7f4cd406a840 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:32.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:32.427+0000 7f4cfd400640 1 --2- 172.21.15.16:0/3801373706 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4cf815d550 0x7f4cf81357a0 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:32.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:32.427+0000 7f4cfd400640 1 --2- 172.21.15.16:0/3801373706 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f4cf80ccc80 0x7f4cf8135260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:32.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:32.427+0000 7f4cfd400640 1 --2- 172.21.15.16:0/3801373706 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4cf8070030 0x7f4cf8152d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:32.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:32.427+0000 7f4cfd400640 1 -- 172.21.15.16:0/3801373706 >> 172.21.15.16:0/3801373706 conn(0x7f4cf8067330 msgr2=0x7f4cf8159630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:05:32.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:32.427+0000 7f4cfd400640 1 -- 172.21.15.16:0/3801373706 shutdown_connections 2024-03-31T15:05:32.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:32.427+0000 7f4cfd400640 1 -- 172.21.15.16:0/3801373706 wait complete. 2024-03-31T15:05:32.844 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-31T15:05:26.461908+0000', 'modified': '2024-03-31T15:05:29.491163+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5697}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5697': {'gid': 5697, 'name': 'a', 'rank': 0, 'incarnation': 21, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.16:6834/2768973409', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6832', 'nonce': 2768973409}, {'type': 'v1', 'addr': '172.21.15.16:6834', 'nonce': 2768973409}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [7], 'metadata_pool': 6, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 5697, 'qdb_cluster': [5697]} 2024-03-31T15:05:32.844 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-31T15:05:32.844 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-03-31T15:05:32.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:32.991+0000 7f9242e00640 1 Processor -- start 2024-03-31T15:05:32.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:32.991+0000 7f9242e00640 1 -- start start 2024-03-31T15:05:32.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:32.991+0000 7f9242e00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f923c145aa0 0x7f923c15c8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:32.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:32.991+0000 7f9242e00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f923c06c1c0 0x7f923c072e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:32.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:32.991+0000 7f9242e00640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f923c0d09f0 con 0x7f923c06b620 2024-03-31T15:05:32.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:32.991+0000 7f9242e00640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f923c06fd50 con 0x7f923c145aa0 2024-03-31T15:05:32.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:32.991+0000 7f9242e00640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f923c0d0cd0 con 0x7f923c06c1c0 2024-03-31T15:05:32.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:32.991+0000 7f923be00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f923c145aa0 0x7f923c15c8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:32.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:32.991+0000 7f923be00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f923c145aa0 0x7f923c15c8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34214/0 (socket says 172.21.15.16:34214) 2024-03-31T15:05:32.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:32.991+0000 7f923be00640 1 -- 172.21.15.16:0/2378071720 learned_addr learned my addr 172.21.15.16:0/2378071720 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:05:32.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:32.991+0000 7f9233400640 1 --2- 172.21.15.16:0/2378071720 >> v2:172.21.15.89:3300/0 conn(0x7f923c06c1c0 0x7f923c072e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:32.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:32.991+0000 7f923be00640 1 -- 172.21.15.16:0/2378071720 >> v2:172.21.15.89:3300/0 conn(0x7f923c06c1c0 msgr2=0x7f923c072e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:05:32.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:32.991+0000 7f923be00640 1 --2- 172.21.15.16:0/2378071720 >> v2:172.21.15.89:3300/0 conn(0x7f923c06c1c0 0x7f923c072e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:32.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:32.991+0000 7f923be00640 1 -- 172.21.15.16:0/2378071720 >> v1:172.21.15.89:6789/0 conn(0x7f923c06b620 legacy=0x7f923c0701e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:32.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:32.991+0000 7f923be00640 1 -- 172.21.15.16:0/2378071720 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f923c0d06e0 con 0x7f923c145aa0 2024-03-31T15:05:32.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:32.999+0000 7f923be00640 1 --2- 172.21.15.16:0/2378071720 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f923c145aa0 0x7f923c15c8e0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f923c0b7310 tx=0x7f922c01c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8abd1ab15f54e5aa server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:05:32.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:32.999+0000 7f923b400640 1 -- 172.21.15.16:0/2378071720 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f922c0b0070 con 0x7f923c145aa0 2024-03-31T15:05:32.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:32.999+0000 7f923b400640 1 -- 172.21.15.16:0/2378071720 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f922c0ab600 con 0x7f923c145aa0 2024-03-31T15:05:32.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:32.999+0000 7f923b400640 1 -- 172.21.15.16:0/2378071720 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f922c0ab9d0 con 0x7f923c145aa0 2024-03-31T15:05:32.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:32.999+0000 7f9242e00640 1 -- 172.21.15.16:0/2378071720 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f923c145aa0 msgr2=0x7f923c15c8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:32.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:32.999+0000 7f9242e00640 1 --2- 172.21.15.16:0/2378071720 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f923c145aa0 0x7f923c15c8e0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f923c0b7310 tx=0x7f922c01c520 comp rx=0 tx=0).stop 2024-03-31T15:05:32.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:32.999+0000 7f9242e00640 1 -- 172.21.15.16:0/2378071720 shutdown_connections 2024-03-31T15:05:32.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:32.999+0000 7f9242e00640 1 --2- 172.21.15.16:0/2378071720 >> v2:172.21.15.89:3300/0 conn(0x7f923c06c1c0 0x7f923c072e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:32.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:32.999+0000 7f9242e00640 1 --2- 172.21.15.16:0/2378071720 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f923c145aa0 0x7f923c15c8e0 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:32.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:32.999+0000 7f9242e00640 1 -- 172.21.15.16:0/2378071720 >> 172.21.15.16:0/2378071720 conn(0x7f923c0d1060 msgr2=0x7f923c0d1470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:05:32.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:32.999+0000 7f9242e00640 1 -- 172.21.15.16:0/2378071720 shutdown_connections 2024-03-31T15:05:32.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:32.999+0000 7f9242e00640 1 -- 172.21.15.16:0/2378071720 wait complete. 2024-03-31T15:05:32.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:32.999+0000 7f9242e00640 1 Processor -- start 2024-03-31T15:05:32.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:32.999+0000 7f9242e00640 1 -- start start 2024-03-31T15:05:32.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.003+0000 7f9242e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f923c06b620 0x7f923c159fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:32.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.003+0000 7f9242e00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f923c06c1c0 0x7f923c1f6660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:32.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.003+0000 7f9242e00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f923c145aa0 0x7f923c1f6ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:32.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.003+0000 7f9242e00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f923c169070 con 0x7f923c06b620 2024-03-31T15:05:32.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.003+0000 7f9242e00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f923c168ef0 con 0x7f923c145aa0 2024-03-31T15:05:32.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.003+0000 7f9242e00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f923c1691f0 con 0x7f923c06c1c0 2024-03-31T15:05:32.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.003+0000 7f923be00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f923c06b620 0x7f923c159fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:32.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.003+0000 7f923be00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f923c06b620 0x7f923c159fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34220/0 (socket says 172.21.15.16:34220) 2024-03-31T15:05:32.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.003+0000 7f923be00640 1 -- 172.21.15.16:0/4137515676 learned_addr learned my addr 172.21.15.16:0/4137515676 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:05:32.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.003+0000 7f9233400640 1 --2- 172.21.15.16:0/4137515676 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f923c06c1c0 0x7f923c1f6660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:32.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.003+0000 7f9240e00640 1 --2- 172.21.15.16:0/4137515676 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f923c145aa0 0x7f923c1f6ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:32.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.003+0000 7f923be00640 1 -- 172.21.15.16:0/4137515676 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f923c06c1c0 msgr2=0x7f923c1f6660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:32.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.003+0000 7f923be00640 1 --2- 172.21.15.16:0/4137515676 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f923c06c1c0 0x7f923c1f6660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:32.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.003+0000 7f923be00640 1 -- 172.21.15.16:0/4137515676 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f923c145aa0 msgr2=0x7f923c1f6ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:32.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.003+0000 7f923be00640 1 --2- 172.21.15.16:0/4137515676 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f923c145aa0 0x7f923c1f6ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:32.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.003+0000 7f923be00640 1 -- 172.21.15.16:0/4137515676 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f923c140d90 con 0x7f923c06b620 2024-03-31T15:05:32.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.003+0000 7f923be00640 1 --2- 172.21.15.16:0/4137515676 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f923c06b620 0x7f923c159fe0 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f922c01ca00 tx=0x7f922c0192c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:05:32.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.003+0000 7f9238c00640 1 -- 172.21.15.16:0/4137515676 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f922c0bb070 con 0x7f923c06b620 2024-03-31T15:05:32.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.003+0000 7f9238c00640 1 -- 172.21.15.16:0/4137515676 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f922c0b6440 con 0x7f923c06b620 2024-03-31T15:05:32.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.003+0000 7f9242e00640 1 -- 172.21.15.16:0/4137515676 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f923c1f5ec0 con 0x7f923c06b620 2024-03-31T15:05:32.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.003+0000 7f9238c00640 1 -- 172.21.15.16:0/4137515676 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f922c0b0040 con 0x7f923c06b620 2024-03-31T15:05:32.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.003+0000 7f9242e00640 1 -- 172.21.15.16:0/4137515676 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f923c0ce730 con 0x7f923c06b620 2024-03-31T15:05:33.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.007+0000 7f9238c00640 1 -- 172.21.15.16:0/4137515676 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f922c0be070 con 0x7f923c06b620 2024-03-31T15:05:33.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.007+0000 7f9238c00640 1 --2- 172.21.15.16:0/4137515676 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f9218068380 0x7f921806a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:33.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.007+0000 7f9238c00640 1 -- 172.21.15.16:0/4137515676 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5709+0+0 (secure 0 0 0) 0x7f922c1202f0 con 0x7f923c06b620 2024-03-31T15:05:33.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.007+0000 7f9233400640 1 --2- 172.21.15.16:0/4137515676 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f9218068380 0x7f921806a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:33.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.007+0000 7f924623e000 1 Processor -- start 2024-03-31T15:05:33.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.007+0000 7f924623e000 1 -- start start 2024-03-31T15:05:33.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.007+0000 7f9233400640 1 --2- 172.21.15.16:0/4137515676 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f9218068380 0x7f921806a840 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f923c0b26e0 tx=0x7f9224008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:05:33.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.007+0000 7f924623e000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559f874b74a0 0x559f874b7880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:33.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.007+0000 7f924623e000 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x559f874b7dc0 0x559f8746ab70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:33.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.007+0000 7f924623e000 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559f8746b0b0 0x559f8746b490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:33.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.007+0000 7f924623e000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x559f872bd950 con 0x559f874b74a0 2024-03-31T15:05:33.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.007+0000 7f924623e000 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x559f8725a990 con 0x559f8746b0b0 2024-03-31T15:05:33.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.007+0000 7f924623e000 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x559f872b9900 con 0x559f874b7dc0 2024-03-31T15:05:33.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.007+0000 7f923be00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559f874b74a0 0x559f874b7880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:33.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.007+0000 7f923be00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559f874b74a0 0x559f874b7880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34232/0 (socket says 172.21.15.16:34232) 2024-03-31T15:05:33.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.007+0000 7f923be00640 1 -- 172.21.15.16:0/3967968186 learned_addr learned my addr 172.21.15.16:0/3967968186 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:05:33.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.007+0000 7f9240e00640 1 --2- 172.21.15.16:0/3967968186 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559f8746b0b0 0x559f8746b490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:33.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.007+0000 7f9233400640 1 --2- 172.21.15.16:0/3967968186 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x559f874b7dc0 0x559f8746ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:33.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.007+0000 7f923be00640 1 -- 172.21.15.16:0/3967968186 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x559f874b7dc0 msgr2=0x559f8746ab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:05:33.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.007+0000 7f923be00640 1 --2- 172.21.15.16:0/3967968186 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x559f874b7dc0 0x559f8746ab70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:33.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.007+0000 7f923be00640 1 -- 172.21.15.16:0/3967968186 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559f8746b0b0 msgr2=0x559f8746b490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:05:33.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.007+0000 7f923be00640 1 --2- 172.21.15.16:0/3967968186 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559f8746b0b0 0x559f8746b490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:33.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.007+0000 7f923be00640 1 -- 172.21.15.16:0/3967968186 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559f87302100 con 0x559f874b74a0 2024-03-31T15:05:33.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.007+0000 7f9233400640 1 --2- 172.21.15.16:0/3967968186 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x559f874b7dc0 0x559f8746ab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:05:33.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.007+0000 7f923be00640 1 --2- 172.21.15.16:0/3967968186 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559f874b74a0 0x559f874b7880 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f922c0eff60 tx=0x7f922c0f0aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d72dc87aa0f724d server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:05:33.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.007+0000 7f9230c00640 1 -- 172.21.15.16:0/3967968186 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f922c119030 con 0x559f874b74a0 2024-03-31T15:05:33.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.007+0000 7f9230c00640 1 -- 172.21.15.16:0/3967968186 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f922c0f8e40 con 0x559f874b74a0 2024-03-31T15:05:33.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.011+0000 7f9230c00640 1 -- 172.21.15.16:0/3967968186 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f922c0ee560 con 0x559f874b74a0 2024-03-31T15:05:33.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.011+0000 7f924623e000 1 -- 172.21.15.16:0/3967968186 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559f874b74a0 msgr2=0x559f874b7880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:33.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.011+0000 7f924623e000 1 --2- 172.21.15.16:0/3967968186 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559f874b74a0 0x559f874b7880 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f922c0eff60 tx=0x7f922c0f0aa0 comp rx=0 tx=0).stop 2024-03-31T15:05:33.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.011+0000 7f924623e000 1 -- 172.21.15.16:0/3967968186 shutdown_connections 2024-03-31T15:05:33.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.011+0000 7f924623e000 1 --2- 172.21.15.16:0/3967968186 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559f8746b0b0 0x559f8746b490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:33.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.011+0000 7f924623e000 1 --2- 172.21.15.16:0/3967968186 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x559f874b7dc0 0x559f8746ab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:33.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.011+0000 7f924623e000 1 --2- 172.21.15.16:0/3967968186 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559f874b74a0 0x559f874b7880 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:33.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.011+0000 7f924623e000 1 -- 172.21.15.16:0/3967968186 >> 172.21.15.16:0/3967968186 conn(0x559f87355b80 msgr2=0x559f8749a460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:05:33.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.011+0000 7f924623e000 1 -- 172.21.15.16:0/3967968186 shutdown_connections 2024-03-31T15:05:33.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.011+0000 7f924623e000 1 -- 172.21.15.16:0/3967968186 wait complete. 2024-03-31T15:05:33.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.011+0000 7f924623e000 1 -- start start 2024-03-31T15:05:33.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.011+0000 7f924623e000 1 Processor -- start 2024-03-31T15:05:33.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.011+0000 7f924623e000 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x559f8746b0b0 0x559f874c3640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:33.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.011+0000 7f924623e000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559f874b74a0 0x559f874c3b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:33.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.011+0000 7f924623e000 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559f874b7dc0 0x559f87489d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:33.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.011+0000 7f924623e000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x559f872bcc60 con 0x559f874b74a0 2024-03-31T15:05:33.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.011+0000 7f924623e000 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x559f8727b3e0 con 0x559f874b7dc0 2024-03-31T15:05:33.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.011+0000 7f924623e000 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x559f872bd950 con 0x559f8746b0b0 2024-03-31T15:05:33.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.011+0000 7f924623e000 10 client.0 _ll_register_callbacks cb 0x559f8731e0f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T15:05:33.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.011+0000 7f9233400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559f874b74a0 0x559f874c3b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:33.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.011+0000 7f923be00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x559f8746b0b0 0x559f874c3640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:33.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.011+0000 7f9233400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559f874b74a0 0x559f874c3b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34238/0 (socket says 172.21.15.16:34238) 2024-03-31T15:05:33.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.011+0000 7f9233400640 1 -- 172.21.15.16:0/4287010279 learned_addr learned my addr 172.21.15.16:0/4287010279 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:05:33.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.011+0000 7f9240e00640 1 --2- 172.21.15.16:0/4287010279 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559f874b7dc0 0x559f87489d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:33.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.011+0000 7f9233400640 1 -- 172.21.15.16:0/4287010279 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x559f8746b0b0 msgr2=0x559f874c3640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:05:33.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.011+0000 7f9233400640 1 --2- 172.21.15.16:0/4287010279 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x559f8746b0b0 0x559f874c3640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:33.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.011+0000 7f9233400640 1 -- 172.21.15.16:0/4287010279 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559f874b7dc0 msgr2=0x559f87489d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:05:33.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.011+0000 7f9233400640 1 --2- 172.21.15.16:0/4287010279 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559f874b7dc0 0x559f87489d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:33.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.011+0000 7f9233400640 1 -- 172.21.15.16:0/4287010279 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559f872878c0 con 0x559f874b74a0 2024-03-31T15:05:33.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.011+0000 7f9233400640 1 --2- 172.21.15.16:0/4287010279 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559f874b74a0 0x559f874c3b80 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f9224006a60 tx=0x7f92240046d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b3d261c3a2da12a7 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:05:33.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.011+0000 7f9229600640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-31T15:05:33.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.011+0000 7f9229600640 1 -- 172.21.15.16:0/4287010279 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9224026070 con 0x559f874b74a0 2024-03-31T15:05:33.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.011+0000 7f9229600640 1 -- 172.21.15.16:0/4287010279 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(45..45 src has 1..45) ==== 5709+0+0 (secure 0 0 0) 0x7f9224014730 con 0x559f874b74a0 2024-03-31T15:05:33.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.011+0000 7f924623e000 1 -- 172.21.15.16:0/4287010279 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f922c0f8e40 con 0x559f874b74a0 2024-03-31T15:05:33.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.011+0000 7f9229600640 1 -- 172.21.15.16:0/4287010279 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f9224031050 con 0x559f874b74a0 2024-03-31T15:05:33.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.011+0000 7f924623e000 10 client.5841 fetch_fsmap learned FSMap version 25 2024-03-31T15:05:33.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.015+0000 7f924623e000 1 -- 172.21.15.16:0/4287010279 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=2+}) -- 0x559f87245e80 con 0x559f874b74a0 2024-03-31T15:05:33.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.015+0000 7f9229600640 1 -- 172.21.15.16:0/4287010279 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f922c0f8e40 con 0x559f874b74a0 2024-03-31T15:05:33.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.015+0000 7f9229600640 1 -- 172.21.15.16:0/4287010279 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 25) ==== 3494+0+0 (secure 0 0 0) 0x7f9224034020 con 0x559f874b74a0 2024-03-31T15:05:33.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.015+0000 7f924623e000 10 client.5841 fetch_fsmap finished waiting for FSMap version 25 2024-03-31T15:05:33.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.015+0000 7f924623e000 10 client.5841 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-31T15:05:33.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.015+0000 7f924623e000 20 client.5841 populate_metadata read hostname 'smithi016' 2024-03-31T15:05:33.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.015+0000 7f924623e000 1 --2- 172.21.15.16:0/4287010279 >> [v2:172.21.15.16:6832/2768973409,v1:172.21.15.16:6834/2768973409] conn(0x559f8748a750 0x559f8748ab30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:33.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.015+0000 7f924623e000 4 client.5841 mds_command: new command op to 5697 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-31T15:05:33.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.015+0000 7f924623e000 1 -- 172.21.15.16:0/4287010279 --> [v2:172.21.15.16:6832/2768973409,v1:172.21.15.16:6834/2768973409] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559f87258bc0 con 0x559f8748a750 2024-03-31T15:05:33.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.015+0000 7f923be00640 1 --2- 172.21.15.16:0/4287010279 >> [v2:172.21.15.16:6832/2768973409,v1:172.21.15.16:6834/2768973409] conn(0x559f8748a750 0x559f8748ab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:33.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.015+0000 7f923be00640 1 --2- 172.21.15.16:0/4287010279 >> [v2:172.21.15.16:6832/2768973409,v1:172.21.15.16:6834/2768973409] conn(0x559f8748a750 0x559f8748ab30 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8fc55df90e25f3fe server_cookie=f5130b8a966d030e in_seq=0 out_seq=0 2024-03-31T15:05:33.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.015+0000 7f9229600640 10 client.5841 ms_handle_connect on v2:172.21.15.16:6832/2768973409 2024-03-31T15:05:33.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.015+0000 7f9229600640 1 -- 172.21.15.16:0/4287010279 <== mds.0 v2:172.21.15.16:6832/2768973409 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x559f87258bc0 con 0x559f8748a750 2024-03-31T15:05:33.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.015+0000 7f9229600640 10 client.5841 handle_command_reply: tid=0 2024-03-31T15:05:33.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.015+0000 7f924623e000 1 client.5841 shutdown 2024-03-31T15:05:33.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.015+0000 7f924623e000 1 -- 172.21.15.16:0/4287010279 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559f874b74a0 msgr2=0x559f874c3b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:33.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.015+0000 7f924623e000 1 --2- 172.21.15.16:0/4287010279 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559f874b74a0 0x559f874c3b80 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f9224006a60 tx=0x7f92240046d0 comp rx=0 tx=0).stop 2024-03-31T15:05:33.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.015+0000 7f924623e000 1 -- 172.21.15.16:0/4287010279 shutdown_connections 2024-03-31T15:05:33.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.015+0000 7f924623e000 1 --2- 172.21.15.16:0/4287010279 >> [v2:172.21.15.16:6832/2768973409,v1:172.21.15.16:6834/2768973409] conn(0x559f8748a750 0x559f8748ab30 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:33.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.015+0000 7f924623e000 1 --2- 172.21.15.16:0/4287010279 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559f874b7dc0 0x559f87489d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:33.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.015+0000 7f924623e000 1 --2- 172.21.15.16:0/4287010279 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559f874b74a0 0x559f874c3b80 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:33.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.015+0000 7f9229600640 0 client.5841 ms_handle_reset on v2:172.21.15.16:6832/2768973409 2024-03-31T15:05:33.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.015+0000 7f924623e000 1 --2- 172.21.15.16:0/4287010279 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x559f8746b0b0 0x559f874c3640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:33.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.015+0000 7f924623e000 1 -- 172.21.15.16:0/4287010279 >> 172.21.15.16:0/4287010279 conn(0x559f87355b80 msgr2=0x559f8746a270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:05:33.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.015+0000 7f924623e000 1 -- 172.21.15.16:0/4287010279 shutdown_connections 2024-03-31T15:05:33.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.015+0000 7f924623e000 1 -- 172.21.15.16:0/4287010279 wait complete. 2024-03-31T15:05:33.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.015+0000 7f924623e000 20 client.5841 trim_cache size 0 max 16384 2024-03-31T15:05:33.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.043+0000 7f924623e000 1 Processor -- start 2024-03-31T15:05:33.037 INFO:tasks.ceph.mds.a.smithi016.stderr:2024-03-31T15:05:33.043+0000 7fa13f800640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-31T15:05:33.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.043+0000 7f924623e000 1 -- start start 2024-03-31T15:05:33.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.043+0000 7f924623e000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559f8746b0b0 0x559f8746a910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:33.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.043+0000 7f924623e000 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x559f874b74a0 0x559f87499bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:33.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.043+0000 7f924623e000 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559f874b7dc0 0x559f8749a0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:33.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.043+0000 7f924623e000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f923c0ae7a0 con 0x559f8746b0b0 2024-03-31T15:05:33.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.043+0000 7f924623e000 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f923c0b5840 con 0x559f874b7dc0 2024-03-31T15:05:33.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.043+0000 7f924623e000 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f923c0b5e90 con 0x559f874b74a0 2024-03-31T15:05:33.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.043+0000 7f923be00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559f8746b0b0 0x559f8746a910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:33.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.043+0000 7f923be00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559f8746b0b0 0x559f8746a910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34248/0 (socket says 172.21.15.16:34248) 2024-03-31T15:05:33.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.043+0000 7f923be00640 1 -- 172.21.15.16:0/4229112268 learned_addr learned my addr 172.21.15.16:0/4229112268 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:05:33.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.043+0000 7f9240e00640 1 --2- 172.21.15.16:0/4229112268 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559f874b7dc0 0x559f8749a0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:33.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.043+0000 7f9233400640 1 --2- 172.21.15.16:0/4229112268 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x559f874b74a0 0x559f87499bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:33.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.043+0000 7f923be00640 1 -- 172.21.15.16:0/4229112268 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x559f874b74a0 msgr2=0x559f87499bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:05:33.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.043+0000 7f923be00640 1 --2- 172.21.15.16:0/4229112268 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x559f874b74a0 0x559f87499bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:33.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.043+0000 7f923be00640 1 -- 172.21.15.16:0/4229112268 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559f874b7dc0 msgr2=0x559f8749a0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:05:33.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.043+0000 7f923be00640 1 --2- 172.21.15.16:0/4229112268 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559f874b7dc0 0x559f8749a0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:33.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.043+0000 7f923be00640 1 -- 172.21.15.16:0/4229112268 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559f87278490 con 0x559f8746b0b0 2024-03-31T15:05:33.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.043+0000 7f923be00640 1 --2- 172.21.15.16:0/4229112268 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559f8746b0b0 0x559f8746a910 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f922c01a150 tx=0x7f922c120000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d997e7d9a1a7108 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:05:33.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.043+0000 7f922be00640 1 -- 172.21.15.16:0/4229112268 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f922c119040 con 0x559f8746b0b0 2024-03-31T15:05:33.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.043+0000 7f922be00640 1 -- 172.21.15.16:0/4229112268 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f922c0ee560 con 0x559f8746b0b0 2024-03-31T15:05:33.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.043+0000 7f922be00640 1 -- 172.21.15.16:0/4229112268 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f922c11e390 con 0x559f8746b0b0 2024-03-31T15:05:33.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.043+0000 7f924623e000 1 -- 172.21.15.16:0/4229112268 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559f8746b0b0 msgr2=0x559f8746a910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:33.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.043+0000 7f924623e000 1 --2- 172.21.15.16:0/4229112268 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559f8746b0b0 0x559f8746a910 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f922c01a150 tx=0x7f922c120000 comp rx=0 tx=0).stop 2024-03-31T15:05:33.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.043+0000 7f924623e000 1 -- 172.21.15.16:0/4229112268 shutdown_connections 2024-03-31T15:05:33.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.043+0000 7f924623e000 1 --2- 172.21.15.16:0/4229112268 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559f874b7dc0 0x559f8749a0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:33.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.043+0000 7f924623e000 1 --2- 172.21.15.16:0/4229112268 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x559f874b74a0 0x559f87499bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:33.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.043+0000 7f924623e000 1 --2- 172.21.15.16:0/4229112268 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559f8746b0b0 0x559f8746a910 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:33.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.043+0000 7f924623e000 1 -- 172.21.15.16:0/4229112268 >> 172.21.15.16:0/4229112268 conn(0x559f87355b80 msgr2=0x559f87472350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:05:33.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.043+0000 7f924623e000 1 -- 172.21.15.16:0/4229112268 shutdown_connections 2024-03-31T15:05:33.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.043+0000 7f924623e000 1 -- 172.21.15.16:0/4229112268 wait complete. 2024-03-31T15:05:33.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.047+0000 7f924623e000 1 -- start start 2024-03-31T15:05:33.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.047+0000 7f924623e000 1 Processor -- start 2024-03-31T15:05:33.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.047+0000 7f924623e000 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559f8746b0b0 0x559f8746a710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:33.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.047+0000 7f924623e000 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x559f874b74a0 0x559f8752bc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:33.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.047+0000 7f924623e000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559f874b7dc0 0x559f8752c150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:33.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.047+0000 7f924623e000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x559f872c1f80 con 0x559f874b7dc0 2024-03-31T15:05:33.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.047+0000 7f924623e000 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x559f872ca060 con 0x559f8746b0b0 2024-03-31T15:05:33.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.047+0000 7f924623e000 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x559f87241b10 con 0x559f874b74a0 2024-03-31T15:05:33.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.047+0000 7f924623e000 10 client.0 _ll_register_callbacks cb 0x7f92240098e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T15:05:33.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.047+0000 7f9240e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559f874b7dc0 0x559f8752c150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:33.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.047+0000 7f9240e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559f874b7dc0 0x559f8752c150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34260/0 (socket says 172.21.15.16:34260) 2024-03-31T15:05:33.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.047+0000 7f9240e00640 1 -- 172.21.15.16:0/1863658930 learned_addr learned my addr 172.21.15.16:0/1863658930 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:05:33.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.047+0000 7f9233400640 1 --2- 172.21.15.16:0/1863658930 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x559f874b74a0 0x559f8752bc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:33.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.047+0000 7f923be00640 1 --2- 172.21.15.16:0/1863658930 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559f8746b0b0 0x559f8746a710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:33.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.047+0000 7f9240e00640 1 -- 172.21.15.16:0/1863658930 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x559f874b74a0 msgr2=0x559f8752bc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:05:33.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.047+0000 7f9240e00640 1 --2- 172.21.15.16:0/1863658930 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x559f874b74a0 0x559f8752bc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:33.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.047+0000 7f9240e00640 1 -- 172.21.15.16:0/1863658930 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559f8746b0b0 msgr2=0x559f8746a710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:05:33.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.047+0000 7f9240e00640 1 --2- 172.21.15.16:0/1863658930 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559f8746b0b0 0x559f8746a710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:33.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.047+0000 7f9240e00640 1 -- 172.21.15.16:0/1863658930 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559f87276e80 con 0x559f874b7dc0 2024-03-31T15:05:33.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.047+0000 7f9240e00640 1 --2- 172.21.15.16:0/1863658930 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559f874b7dc0 0x559f8752c150 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f9234017a00 tx=0x7f9234017ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c3b40333afcb110 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:05:33.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.047+0000 7f9229600640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-31T15:05:33.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.047+0000 7f9229600640 1 -- 172.21.15.16:0/1863658930 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9234015ca0 con 0x559f874b7dc0 2024-03-31T15:05:33.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.047+0000 7f9229600640 1 -- 172.21.15.16:0/1863658930 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(45..45 src has 1..45) ==== 5709+0+0 (secure 0 0 0) 0x7f9234015e40 con 0x559f874b7dc0 2024-03-31T15:05:33.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.047+0000 7f924623e000 1 -- 172.21.15.16:0/1863658930 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f922c0ee560 con 0x559f874b7dc0 2024-03-31T15:05:33.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.047+0000 7f9229600640 1 -- 172.21.15.16:0/1863658930 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f9234027050 con 0x559f874b7dc0 2024-03-31T15:05:33.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.047+0000 7f924623e000 10 client.5847 fetch_fsmap learned FSMap version 25 2024-03-31T15:05:33.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.047+0000 7f924623e000 1 -- 172.21.15.16:0/1863658930 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=2+}) -- 0x559f87264270 con 0x559f874b7dc0 2024-03-31T15:05:33.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.047+0000 7f9229600640 1 -- 172.21.15.16:0/1863658930 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f922c0ee560 con 0x559f874b7dc0 2024-03-31T15:05:33.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.047+0000 7f9229600640 1 -- 172.21.15.16:0/1863658930 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 25) ==== 3494+0+0 (secure 0 0 0) 0x7f9234006b60 con 0x559f874b7dc0 2024-03-31T15:05:33.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.047+0000 7f924623e000 10 client.5847 fetch_fsmap finished waiting for FSMap version 25 2024-03-31T15:05:33.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.047+0000 7f924623e000 10 client.5847 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-31T15:05:33.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.047+0000 7f924623e000 20 client.5847 populate_metadata read hostname 'smithi016' 2024-03-31T15:05:33.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.047+0000 7f924623e000 1 --2- 172.21.15.16:0/1863658930 >> [v2:172.21.15.16:6832/2768973409,v1:172.21.15.16:6834/2768973409] conn(0x559f8752c690 0x559f87498ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:33.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.047+0000 7f924623e000 4 client.5847 mds_command: new command op to 5697 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-31T15:05:33.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.047+0000 7f924623e000 1 -- 172.21.15.16:0/1863658930 --> [v2:172.21.15.16:6832/2768973409,v1:172.21.15.16:6834/2768973409] -- command(tid 0: {"prefix": "status"}) -- 0x559f872cae90 con 0x559f8752c690 2024-03-31T15:05:33.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.047+0000 7f923be00640 1 --2- 172.21.15.16:0/1863658930 >> [v2:172.21.15.16:6832/2768973409,v1:172.21.15.16:6834/2768973409] conn(0x559f8752c690 0x559f87498ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:33.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.051+0000 7f9229600640 10 client.5847 ms_handle_connect on v2:172.21.15.16:6832/2768973409 2024-03-31T15:05:33.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.051+0000 7f923be00640 1 --2- 172.21.15.16:0/1863658930 >> [v2:172.21.15.16:6832/2768973409,v1:172.21.15.16:6834/2768973409] conn(0x559f8752c690 0x559f87498ba0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e1226ad2b0a0516a server_cookie=460781e695a5ce8f in_seq=0 out_seq=0 2024-03-31T15:05:33.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.051+0000 7f9229600640 1 -- 172.21.15.16:0/1863658930 <== mds.0 v2:172.21.15.16:6832/2768973409 1 ==== command_reply(tid 0: 0 ) ==== 8+0+318 (crc 0 0 0) 0x559f872cae90 con 0x559f8752c690 2024-03-31T15:05:33.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.051+0000 7f9229600640 10 client.5847 handle_command_reply: tid=0 2024-03-31T15:05:33.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.051+0000 7f924623e000 1 client.5847 shutdown 2024-03-31T15:05:33.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.051+0000 7f924623e000 1 -- 172.21.15.16:0/1863658930 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559f874b7dc0 msgr2=0x559f8752c150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:33.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.051+0000 7f924623e000 1 --2- 172.21.15.16:0/1863658930 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559f874b7dc0 0x559f8752c150 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f9234017a00 tx=0x7f9234017ec0 comp rx=0 tx=0).stop 2024-03-31T15:05:33.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.051+0000 7f924623e000 1 -- 172.21.15.16:0/1863658930 shutdown_connections 2024-03-31T15:05:33.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.051+0000 7f924623e000 1 --2- 172.21.15.16:0/1863658930 >> [v2:172.21.15.16:6832/2768973409,v1:172.21.15.16:6834/2768973409] conn(0x559f8752c690 0x559f87498ba0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:33.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.051+0000 7f924623e000 1 --2- 172.21.15.16:0/1863658930 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559f874b7dc0 0x559f8752c150 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:33.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.051+0000 7f924623e000 1 --2- 172.21.15.16:0/1863658930 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x559f874b74a0 0x559f8752bc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:33.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.051+0000 7f924623e000 1 --2- 172.21.15.16:0/1863658930 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559f8746b0b0 0x559f8746a710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:33.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.051+0000 7f924623e000 1 -- 172.21.15.16:0/1863658930 >> 172.21.15.16:0/1863658930 conn(0x559f87355b80 msgr2=0x559f87472350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:05:33.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.051+0000 7f9229600640 0 client.5847 ms_handle_reset on v2:172.21.15.16:6832/2768973409 2024-03-31T15:05:33.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.051+0000 7f924623e000 1 -- 172.21.15.16:0/1863658930 shutdown_connections 2024-03-31T15:05:33.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.051+0000 7f924623e000 1 -- 172.21.15.16:0/1863658930 wait complete. 2024-03-31T15:05:33.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.051+0000 7f924623e000 20 client.5847 trim_cache size 0 max 16384 2024-03-31T15:05:33.050 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-03-31T15:05:33.050 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_fsid": "286eb5da-fb47-4550-b290-d9519c13b9a5", 2024-03-31T15:05:33.050 INFO:teuthology.orchestra.run.smithi016.stdout: "whoami": 0, 2024-03-31T15:05:33.050 INFO:teuthology.orchestra.run.smithi016.stdout: "id": 5697, 2024-03-31T15:05:33.050 INFO:teuthology.orchestra.run.smithi016.stdout: "want_state": "up:active", 2024-03-31T15:05:33.050 INFO:teuthology.orchestra.run.smithi016.stdout: "state": "up:active", 2024-03-31T15:05:33.050 INFO:teuthology.orchestra.run.smithi016.stdout: "fs_name": "cephfs", 2024-03-31T15:05:33.050 INFO:teuthology.orchestra.run.smithi016.stdout: "rank_uptime": 5.974996301, 2024-03-31T15:05:33.051 INFO:teuthology.orchestra.run.smithi016.stdout: "mdsmap_epoch": 25, 2024-03-31T15:05:33.051 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch": 45, 2024-03-31T15:05:33.051 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch_barrier": 45, 2024-03-31T15:05:33.051 INFO:teuthology.orchestra.run.smithi016.stdout: "uptime": 10.853034401 2024-03-31T15:05:33.051 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-03-31T15:05:33.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.051+0000 7f9242e00640 1 -- 172.21.15.16:0/4137515676 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f9218068380 msgr2=0x7f921806a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:33.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.051+0000 7f9242e00640 1 --2- 172.21.15.16:0/4137515676 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f9218068380 0x7f921806a840 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f923c0b26e0 tx=0x7f9224008040 comp rx=0 tx=0).stop 2024-03-31T15:05:33.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.051+0000 7f9242e00640 1 -- 172.21.15.16:0/4137515676 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f923c06b620 msgr2=0x7f923c159fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:33.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.051+0000 7f9242e00640 1 --2- 172.21.15.16:0/4137515676 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f923c06b620 0x7f923c159fe0 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f922c01ca00 tx=0x7f922c0192c0 comp rx=0 tx=0).stop 2024-03-31T15:05:33.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.051+0000 7f9242e00640 1 -- 172.21.15.16:0/4137515676 shutdown_connections 2024-03-31T15:05:33.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.051+0000 7f9242e00640 1 --2- 172.21.15.16:0/4137515676 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f9218068380 0x7f921806a840 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:33.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.051+0000 7f9242e00640 1 --2- 172.21.15.16:0/4137515676 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f923c145aa0 0x7f923c1f6ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:33.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.051+0000 7f9242e00640 1 --2- 172.21.15.16:0/4137515676 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f923c06c1c0 0x7f923c1f6660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:33.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.051+0000 7f9242e00640 1 --2- 172.21.15.16:0/4137515676 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f923c06b620 0x7f923c159fe0 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:33.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.051+0000 7f9242e00640 1 -- 172.21.15.16:0/4137515676 >> 172.21.15.16:0/4137515676 conn(0x7f923c0d1060 msgr2=0x7f923c144b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:05:33.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.051+0000 7f9242e00640 1 -- 172.21.15.16:0/4137515676 shutdown_connections 2024-03-31T15:05:33.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.051+0000 7f9242e00640 1 -- 172.21.15.16:0/4137515676 wait complete. 2024-03-31T15:05:33.065 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-31T15:05:33.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.219+0000 7f03b3c00640 1 Processor -- start 2024-03-31T15:05:33.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.219+0000 7f03b3c00640 1 -- start start 2024-03-31T15:05:33.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.219+0000 7f03b3c00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f03ac06dbc0 0x7f03ac15f650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:33.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.219+0000 7f03b3c00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f03ac070030 0x7f03ac162180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:33.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.219+0000 7f03b3c00640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f03ac0cbe90 con 0x7f03ac0cb440 2024-03-31T15:05:33.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.219+0000 7f03b3c00640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f03ac15fb90 con 0x7f03ac06dbc0 2024-03-31T15:05:33.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.219+0000 7f03b3c00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f03ac15f1c0 con 0x7f03ac070030 2024-03-31T15:05:33.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.219+0000 7f03a3e00640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f03ac0cb440 0x7f03ac06f920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:35552/0 (socket says 172.21.15.16:35552) 2024-03-31T15:05:33.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.219+0000 7f03a3e00640 1 -- 172.21.15.16:0/385755833 learned_addr learned my addr 172.21.15.16:0/385755833 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:05:33.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.219+0000 7f03b1c00640 1 --2- 172.21.15.16:0/385755833 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f03ac070030 0x7f03ac162180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:33.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.219+0000 7f03b1200640 1 --2- 172.21.15.16:0/385755833 >> v2:172.21.15.89:3300/0 conn(0x7f03ac06dbc0 0x7f03ac15f650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:33.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.219+0000 7f03abe00640 1 -- 172.21.15.16:0/385755833 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3968718667 0 0) 0x7f03ac0cbe90 con 0x7f03ac0cb440 2024-03-31T15:05:33.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.219+0000 7f03abe00640 1 -- 172.21.15.16:0/385755833 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0390003530 con 0x7f03ac0cb440 2024-03-31T15:05:33.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.219+0000 7f03b1c00640 1 -- 172.21.15.16:0/385755833 >> v2:172.21.15.89:3300/0 conn(0x7f03ac06dbc0 msgr2=0x7f03ac15f650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:05:33.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.219+0000 7f03b1c00640 1 --2- 172.21.15.16:0/385755833 >> v2:172.21.15.89:3300/0 conn(0x7f03ac06dbc0 0x7f03ac15f650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:33.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.219+0000 7f03abe00640 1 -- 172.21.15.16:0/385755833 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2437000291 0 0) 0x7f0390003530 con 0x7f03ac0cb440 2024-03-31T15:05:33.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.219+0000 7f03b1c00640 1 -- 172.21.15.16:0/385755833 >> v1:172.21.15.16:6789/0 conn(0x7f03ac0cb440 legacy=0x7f03ac06f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:33.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.219+0000 7f03b1c00640 1 -- 172.21.15.16:0/385755833 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03ac0cc170 con 0x7f03ac070030 2024-03-31T15:05:33.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.223+0000 7f03b1c00640 1 --2- 172.21.15.16:0/385755833 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f03ac070030 0x7f03ac162180 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f03a4009d90 tx=0x7f03a401c7d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=858c2e8ad658a810 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:05:33.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.223+0000 7f03abe00640 1 -- 172.21.15.16:0/385755833 <== mon.2 v2:172.21.15.89:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03a40b2070 con 0x7f03ac070030 2024-03-31T15:05:33.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.223+0000 7f03abe00640 1 -- 172.21.15.16:0/385755833 <== mon.2 v2:172.21.15.89:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f03a40a55f0 con 0x7f03ac070030 2024-03-31T15:05:33.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.223+0000 7f03abe00640 1 -- 172.21.15.16:0/385755833 <== mon.2 v2:172.21.15.89:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0394001680 con 0x7f03ac070030 2024-03-31T15:05:33.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.223+0000 7f03b3c00640 1 -- 172.21.15.16:0/385755833 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f03ac070030 msgr2=0x7f03ac162180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:33.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.223+0000 7f03b3c00640 1 --2- 172.21.15.16:0/385755833 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f03ac070030 0x7f03ac162180 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f03a4009d90 tx=0x7f03a401c7d0 comp rx=0 tx=0).stop 2024-03-31T15:05:33.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.223+0000 7f03b3c00640 1 -- 172.21.15.16:0/385755833 shutdown_connections 2024-03-31T15:05:33.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.223+0000 7f03b3c00640 1 --2- 172.21.15.16:0/385755833 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f03ac070030 0x7f03ac162180 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:33.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.223+0000 7f03b3c00640 1 --2- 172.21.15.16:0/385755833 >> v2:172.21.15.89:3300/0 conn(0x7f03ac06dbc0 0x7f03ac15f650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:33.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.223+0000 7f03b3c00640 1 -- 172.21.15.16:0/385755833 >> 172.21.15.16:0/385755833 conn(0x7f03ac067330 msgr2=0x7f03ac067740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:05:33.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.223+0000 7f03b3c00640 1 -- 172.21.15.16:0/385755833 shutdown_connections 2024-03-31T15:05:33.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.223+0000 7f03b3c00640 1 -- 172.21.15.16:0/385755833 wait complete. 2024-03-31T15:05:33.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.227+0000 7f03b3c00640 1 Processor -- start 2024-03-31T15:05:33.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.227+0000 7f03b3c00640 1 -- start start 2024-03-31T15:05:33.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.227+0000 7f03b3c00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f03ac06dbc0 0x7f03ac12b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:33.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.227+0000 7f03b3c00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f03ac070030 0x7f03ac158140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:33.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.227+0000 7f03b3c00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f03ac0cb440 0x7f03ac13dc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:33.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.227+0000 7f03b3c00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f03ac06cad0 con 0x7f03ac070030 2024-03-31T15:05:33.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.227+0000 7f03b3c00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f03ac06c950 con 0x7f03ac06dbc0 2024-03-31T15:05:33.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.227+0000 7f03b3c00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f03ac16b520 con 0x7f03ac0cb440 2024-03-31T15:05:33.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.227+0000 7f03a3e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f03ac070030 0x7f03ac158140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:33.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.227+0000 7f03a3e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f03ac070030 0x7f03ac158140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34276/0 (socket says 172.21.15.16:34276) 2024-03-31T15:05:33.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.227+0000 7f03a3e00640 1 -- 172.21.15.16:0/4089032042 learned_addr learned my addr 172.21.15.16:0/4089032042 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:05:33.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.227+0000 7f03b1200640 1 --2- 172.21.15.16:0/4089032042 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f03ac06dbc0 0x7f03ac12b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:33.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.227+0000 7f03b1c00640 1 --2- 172.21.15.16:0/4089032042 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f03ac0cb440 0x7f03ac13dc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:33.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.227+0000 7f03a3e00640 1 -- 172.21.15.16:0/4089032042 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f03ac0cb440 msgr2=0x7f03ac13dc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:33.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.227+0000 7f03a3e00640 1 --2- 172.21.15.16:0/4089032042 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f03ac0cb440 0x7f03ac13dc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:33.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.227+0000 7f03a3e00640 1 -- 172.21.15.16:0/4089032042 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f03ac06dbc0 msgr2=0x7f03ac12b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:33.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.227+0000 7f03a3e00640 1 --2- 172.21.15.16:0/4089032042 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f03ac06dbc0 0x7f03ac12b0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:33.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.227+0000 7f03a3e00640 1 -- 172.21.15.16:0/4089032042 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03ac12b600 con 0x7f03ac070030 2024-03-31T15:05:33.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.227+0000 7f03a3e00640 1 --2- 172.21.15.16:0/4089032042 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f03ac070030 0x7f03ac158140 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f039400c760 tx=0x7f039400cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:05:33.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.227+0000 7f03a9600640 1 -- 172.21.15.16:0/4089032042 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0394017070 con 0x7f03ac070030 2024-03-31T15:05:33.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.227+0000 7f03a9600640 1 -- 172.21.15.16:0/4089032042 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0394005ae0 con 0x7f03ac070030 2024-03-31T15:05:33.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.227+0000 7f03b3c00640 1 -- 172.21.15.16:0/4089032042 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f03ac143280 con 0x7f03ac070030 2024-03-31T15:05:33.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.227+0000 7f03a9600640 1 -- 172.21.15.16:0/4089032042 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0394007070 con 0x7f03ac070030 2024-03-31T15:05:33.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.227+0000 7f03b3c00640 1 -- 172.21.15.16:0/4089032042 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f03ac143840 con 0x7f03ac070030 2024-03-31T15:05:33.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.231+0000 7f03b3c00640 1 -- 172.21.15.16:0/4089032042 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f03ac069950 con 0x7f03ac070030 2024-03-31T15:05:33.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.231+0000 7f03a9600640 1 -- 172.21.15.16:0/4089032042 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f039401a070 con 0x7f03ac070030 2024-03-31T15:05:33.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.231+0000 7f03a9600640 1 --2- 172.21.15.16:0/4089032042 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f0388068380 0x7f038806a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:33.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.231+0000 7f03a9600640 1 -- 172.21.15.16:0/4089032042 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5709+0+0 (secure 0 0 0) 0x7f0394085e10 con 0x7f03ac070030 2024-03-31T15:05:33.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.231+0000 7f03b1200640 1 --2- 172.21.15.16:0/4089032042 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f0388068380 0x7f038806a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:33.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.231+0000 7f03b1200640 1 --2- 172.21.15.16:0/4089032042 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f0388068380 0x7f038806a840 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f03ac0b8240 tx=0x7f039c009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:05:33.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.235+0000 7f03a9600640 1 -- 172.21.15.16:0/4089032042 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f0394055e10 con 0x7f03ac070030 2024-03-31T15:05:33.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.439+0000 7f03b3c00640 1 -- 172.21.15.16:0/4089032042 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f03ac0cf380 con 0x7f03ac070030 2024-03-31T15:05:33.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.439+0000 7f03a9600640 1 -- 172.21.15.16:0/4089032042 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v25) ==== 53+0+73 (secure 0 0 0) 0x7f039405a110 con 0x7f03ac070030 2024-03-31T15:05:33.435 INFO:teuthology.orchestra.run.smithi016.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-31T15:05:33.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.443+0000 7f03b3c00640 1 -- 172.21.15.16:0/4089032042 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f0388068380 msgr2=0x7f038806a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:33.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.443+0000 7f03b3c00640 1 --2- 172.21.15.16:0/4089032042 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f0388068380 0x7f038806a840 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f03ac0b8240 tx=0x7f039c009190 comp rx=0 tx=0).stop 2024-03-31T15:05:33.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.443+0000 7f03b3c00640 1 -- 172.21.15.16:0/4089032042 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f03ac070030 msgr2=0x7f03ac158140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:33.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.443+0000 7f03b3c00640 1 --2- 172.21.15.16:0/4089032042 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f03ac070030 0x7f03ac158140 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f039400c760 tx=0x7f039400cc20 comp rx=0 tx=0).stop 2024-03-31T15:05:33.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.443+0000 7f03b3c00640 1 -- 172.21.15.16:0/4089032042 shutdown_connections 2024-03-31T15:05:33.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.443+0000 7f03b3c00640 1 --2- 172.21.15.16:0/4089032042 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f0388068380 0x7f038806a840 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:33.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.443+0000 7f03b3c00640 1 --2- 172.21.15.16:0/4089032042 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f03ac0cb440 0x7f03ac13dc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:33.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.443+0000 7f03b3c00640 1 --2- 172.21.15.16:0/4089032042 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f03ac070030 0x7f03ac158140 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:33.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.443+0000 7f03b3c00640 1 --2- 172.21.15.16:0/4089032042 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f03ac06dbc0 0x7f03ac12b0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:33.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.443+0000 7f03b3c00640 1 -- 172.21.15.16:0/4089032042 >> 172.21.15.16:0/4089032042 conn(0x7f03ac067330 msgr2=0x7f03ac15c7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:05:33.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.443+0000 7f03b3c00640 1 -- 172.21.15.16:0/4089032042 shutdown_connections 2024-03-31T15:05:33.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33.443+0000 7f03b3c00640 1 -- 172.21.15.16:0/4089032042 wait complete. 2024-03-31T15:05:33.456 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-31T15:05:33.456 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-31T15:05:33.456 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi016.front.sepia.ceph.com 2024-03-31T15:05:33.456 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-31T15:05:33.456 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-31T15:05:33.456 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-31T15:05:33.456 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-31T15:05:33.456 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-31T15:05:33.456 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-31T15:05:33.457 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:05:33.457 DEBUG:teuthology.orchestra.run.smithi016:> ip addr 2024-03-31T15:05:33.466 INFO:teuthology.orchestra.run.smithi016.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-31T15:05:33.466 INFO:teuthology.orchestra.run.smithi016.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-31T15:05:33.466 INFO:teuthology.orchestra.run.smithi016.stdout: inet 127.0.0.1/8 scope host lo 2024-03-31T15:05:33.466 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2024-03-31T15:05:33.466 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 ::1/128 scope host 2024-03-31T15:05:33.467 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2024-03-31T15:05:33.467 INFO:teuthology.orchestra.run.smithi016.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-03-31T15:05:33.467 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:6b:fd:e0 brd ff:ff:ff:ff:ff:ff 2024-03-31T15:05:33.467 INFO:teuthology.orchestra.run.smithi016.stdout: altname enp7s0f0 2024-03-31T15:05:33.467 INFO:teuthology.orchestra.run.smithi016.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-03-31T15:05:33.467 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:6b:fd:e1 brd ff:ff:ff:ff:ff:ff 2024-03-31T15:05:33.467 INFO:teuthology.orchestra.run.smithi016.stdout: altname enp7s0f1 2024-03-31T15:05:33.467 INFO:teuthology.orchestra.run.smithi016.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-31T15:05:33.467 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:bd:15:64 brd ff:ff:ff:ff:ff:ff 2024-03-31T15:05:33.467 INFO:teuthology.orchestra.run.smithi016.stdout: altname enp1s0f0 2024-03-31T15:05:33.467 INFO:teuthology.orchestra.run.smithi016.stdout: inet 172.21.15.16/20 brd 172.21.15.255 scope global dynamic ens1f0 2024-03-31T15:05:33.467 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft 42347sec preferred_lft 42347sec 2024-03-31T15:05:33.467 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 fe80::ec4:7aff:febd:1564/64 scope link 2024-03-31T15:05:33.468 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2024-03-31T15:05:33.468 INFO:teuthology.orchestra.run.smithi016.stdout:5: ens1f1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-03-31T15:05:33.468 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:bd:15:65 brd ff:ff:ff:ff:ff:ff 2024-03-31T15:05:33.468 INFO:teuthology.orchestra.run.smithi016.stdout: altname enp1s0f1 2024-03-31T15:05:33.468 INFO:teuthology.orchestra.run.smithi016.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2024-03-31T15:05:33.468 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 02:42:d4:1f:7d:73 brd ff:ff:ff:ff:ff:ff 2024-03-31T15:05:33.468 INFO:teuthology.orchestra.run.smithi016.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2024-03-31T15:05:33.468 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2024-03-31T15:05:33.468 INFO:teuthology.orchestra.run.smithi016.stdout:7: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-31T15:05:33.468 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 5e:1f:84:6d:79:ae brd ff:ff:ff:ff:ff:ff 2024-03-31T15:05:33.468 INFO:teuthology.orchestra.run.smithi016.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-31T15:05:33.468 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2024-03-31T15:05:33.468 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 fe80::5c1f:84ff:fe6d:79ae/64 scope link 2024-03-31T15:05:33.468 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2024-03-31T15:05:33.468 INFO:teuthology.orchestra.run.smithi016.stdout:8: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-31T15:05:33.468 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0e:5c:f6:1f:82:2f brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-31T15:05:33.469 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 fe80::c5c:f6ff:fe1f:822f/64 scope link 2024-03-31T15:05:33.469 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2024-03-31T15:05:33.469 INFO:teuthology.orchestra.run.smithi016.stdout:9: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-31T15:05:33.469 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether c6:a6:3d:69:d0:db brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-31T15:05:33.469 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 fe80::c4a6:3dff:fe69:d0db/64 scope link 2024-03-31T15:05:33.469 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2024-03-31T15:05:33.469 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:05:33.469 DEBUG:teuthology.orchestra.run.smithi016:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-31T15:05:33.525 INFO:teuthology.orchestra.run.smithi016.stdout:1 2024-03-31T15:05:33.527 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:05:33.528 DEBUG:teuthology.orchestra.run.smithi016:> route 2024-03-31T15:05:33.576 INFO:teuthology.orchestra.run.smithi016.stdout:Kernel IP routing table 2024-03-31T15:05:33.577 INFO:teuthology.orchestra.run.smithi016.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-31T15:05:33.577 INFO:teuthology.orchestra.run.smithi016.stdout:default _gateway 0.0.0.0 UG 0 0 0 ens1f0 2024-03-31T15:05:33.577 INFO:teuthology.orchestra.run.smithi016.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2024-03-31T15:05:33.577 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 0 0 0 ens1f0 2024-03-31T15:05:33.577 INFO:teuthology.orchestra.run.smithi016.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-31T15:05:33.577 DEBUG:teuthology.orchestra.run.smithi016:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-31T15:05:33.577 DEBUG:teuthology.orchestra.run.smithi016:> set -e 2024-03-31T15:05:33.577 DEBUG:teuthology.orchestra.run.smithi016:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-03-31T15:05:33.577 DEBUG:teuthology.orchestra.run.smithi016:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-03-31T15:05:33.578 DEBUG:teuthology.orchestra.run.smithi016:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-03-31T15:05:33.578 DEBUG:teuthology.orchestra.run.smithi016:> ') 2024-03-31T15:05:33.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-31T15:05:33.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:33 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-31T15:05:33.716 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:05:33.716 DEBUG:teuthology.orchestra.run.smithi016:> ip netns list 2024-03-31T15:05:33.725 INFO:teuthology.orchestra.run.smithi016.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-31T15:05:33.725 INFO:teuthology.orchestra.run.smithi016.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-31T15:05:33.725 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-31T15:05:33.726 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:05:33.726 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip link set brx.0 up 2024-03-31T15:05:33.787 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-03-31T15:05:33.787 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-31T15:05:33.787 DEBUG:teuthology.orchestra.run.smithi016:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-31T15:05:33.836 INFO:teuthology.orchestra.run.smithi016.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-31T15:05:33.836 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-31T15:05:33.836 DEBUG:teuthology.orchestra.run.smithi016:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-31T15:05:33.887 DEBUG:teuthology.orchestra.run.smithi016:> sudo modprobe fuse 2024-03-31T15:05:33.948 DEBUG:teuthology.orchestra.run.smithi016:> cat /proc/self/mounts | awk '{print $2}' 2024-03-31T15:05:34.001 INFO:teuthology.orchestra.run.smithi016.stdout:/sys 2024-03-31T15:05:34.001 INFO:teuthology.orchestra.run.smithi016.stdout:/proc 2024-03-31T15:05:34.001 INFO:teuthology.orchestra.run.smithi016.stdout:/dev 2024-03-31T15:05:34.001 INFO:teuthology.orchestra.run.smithi016.stdout:/dev/pts 2024-03-31T15:05:34.001 INFO:teuthology.orchestra.run.smithi016.stdout:/run 2024-03-31T15:05:34.001 INFO:teuthology.orchestra.run.smithi016.stdout:/ 2024-03-31T15:05:34.001 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/kernel/security 2024-03-31T15:05:34.001 INFO:teuthology.orchestra.run.smithi016.stdout:/dev/shm 2024-03-31T15:05:34.002 INFO:teuthology.orchestra.run.smithi016.stdout:/run/lock 2024-03-31T15:05:34.002 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/cgroup 2024-03-31T15:05:34.002 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/pstore 2024-03-31T15:05:34.002 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/bpf 2024-03-31T15:05:34.002 INFO:teuthology.orchestra.run.smithi016.stdout:/proc/sys/fs/binfmt_misc 2024-03-31T15:05:34.002 INFO:teuthology.orchestra.run.smithi016.stdout:/dev/hugepages 2024-03-31T15:05:34.002 INFO:teuthology.orchestra.run.smithi016.stdout:/dev/mqueue 2024-03-31T15:05:34.002 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/kernel/debug 2024-03-31T15:05:34.002 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/kernel/tracing 2024-03-31T15:05:34.002 INFO:teuthology.orchestra.run.smithi016.stdout:/proc/fs/nfsd 2024-03-31T15:05:34.002 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/fuse/connections 2024-03-31T15:05:34.002 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/kernel/config 2024-03-31T15:05:34.002 INFO:teuthology.orchestra.run.smithi016.stdout:/run/credentials/systemd-sysusers.service 2024-03-31T15:05:34.003 INFO:teuthology.orchestra.run.smithi016.stdout:/run/rpc_pipefs 2024-03-31T15:05:34.003 INFO:teuthology.orchestra.run.smithi016.stdout:/run/user/1000 2024-03-31T15:05:34.003 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/ceph 2024-03-31T15:05:34.003 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/ceph/osd/ceph-0 2024-03-31T15:05:34.003 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/ceph/osd/ceph-1 2024-03-31T15:05:34.003 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/ceph/osd/ceph-2 2024-03-31T15:05:34.003 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/ceph/osd/ceph-3 2024-03-31T15:05:34.003 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns 2024-03-31T15:05:34.003 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-31T15:05:34.003 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-31T15:05:34.003 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-31T15:05:34.003 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-31T15:05:34.004 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:05:34.005 DEBUG:teuthology.orchestra.run.smithi016:> ls /sys/fs/fuse/connections 2024-03-31T15:05:34.055 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-31T15:05:34.056 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-03-31T15:05:34.099 DEBUG:teuthology.orchestra.run.smithi016:> sudo modprobe fuse 2024-03-31T15:05:34.120 DEBUG:teuthology.orchestra.run.smithi016:> cat /proc/self/mounts | awk '{print $2}' 2024-03-31T15:05:34.173 INFO:teuthology.orchestra.run.smithi016.stdout:/sys 2024-03-31T15:05:34.174 INFO:teuthology.orchestra.run.smithi016.stdout:/proc 2024-03-31T15:05:34.174 INFO:teuthology.orchestra.run.smithi016.stdout:/dev 2024-03-31T15:05:34.174 INFO:teuthology.orchestra.run.smithi016.stdout:/dev/pts 2024-03-31T15:05:34.174 INFO:teuthology.orchestra.run.smithi016.stdout:/run 2024-03-31T15:05:34.174 INFO:teuthology.orchestra.run.smithi016.stdout:/ 2024-03-31T15:05:34.174 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/kernel/security 2024-03-31T15:05:34.174 INFO:teuthology.orchestra.run.smithi016.stdout:/dev/shm 2024-03-31T15:05:34.174 INFO:teuthology.orchestra.run.smithi016.stdout:/run/lock 2024-03-31T15:05:34.174 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/cgroup 2024-03-31T15:05:34.174 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/pstore 2024-03-31T15:05:34.174 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/bpf 2024-03-31T15:05:34.174 INFO:teuthology.orchestra.run.smithi016.stdout:/proc/sys/fs/binfmt_misc 2024-03-31T15:05:34.175 INFO:teuthology.orchestra.run.smithi016.stdout:/dev/hugepages 2024-03-31T15:05:34.175 INFO:teuthology.orchestra.run.smithi016.stdout:/dev/mqueue 2024-03-31T15:05:34.175 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/kernel/debug 2024-03-31T15:05:34.175 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/kernel/tracing 2024-03-31T15:05:34.175 INFO:teuthology.orchestra.run.smithi016.stdout:/proc/fs/nfsd 2024-03-31T15:05:34.175 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/fuse/connections 2024-03-31T15:05:34.175 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/kernel/config 2024-03-31T15:05:34.175 INFO:teuthology.orchestra.run.smithi016.stdout:/run/credentials/systemd-sysusers.service 2024-03-31T15:05:34.175 INFO:teuthology.orchestra.run.smithi016.stdout:/run/rpc_pipefs 2024-03-31T15:05:34.175 INFO:teuthology.orchestra.run.smithi016.stdout:/run/user/1000 2024-03-31T15:05:34.175 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/ceph 2024-03-31T15:05:34.175 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/ceph/osd/ceph-0 2024-03-31T15:05:34.175 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/ceph/osd/ceph-1 2024-03-31T15:05:34.175 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/ceph/osd/ceph-2 2024-03-31T15:05:34.175 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/ceph/osd/ceph-3 2024-03-31T15:05:34.175 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns 2024-03-31T15:05:34.176 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-31T15:05:34.176 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-31T15:05:34.176 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-31T15:05:34.176 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-31T15:05:34.177 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:05:34.177 DEBUG:teuthology.orchestra.run.smithi016:> ls /sys/fs/fuse/connections 2024-03-31T15:05:34.197 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi016.stderr:2024-03-31T15:05:34.203+0000 7f6ab1e1a580 -1 init, newargv = 0x55fc1bf20730 newargc=15 2024-03-31T15:05:34.197 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi016.stderr:ceph-fuse[38076]: starting ceph client 2024-03-31T15:05:34.205 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi016.stderr:ceph-fuse[38076]: starting fuse 2024-03-31T15:05:34.207 INFO:teuthology.orchestra.run.smithi016.stdout:43 2024-03-31T15:05:34.207 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [43] 2024-03-31T15:05:34.207 DEBUG:teuthology.orchestra.run.smithi016:> sudo stdin-killer -- python3 -c ' 2024-03-31T15:05:34.207 DEBUG:teuthology.orchestra.run.smithi016:> import glob 2024-03-31T15:05:34.207 DEBUG:teuthology.orchestra.run.smithi016:> import re 2024-03-31T15:05:34.207 DEBUG:teuthology.orchestra.run.smithi016:> import os 2024-03-31T15:05:34.207 DEBUG:teuthology.orchestra.run.smithi016:> import subprocess 2024-03-31T15:05:34.207 DEBUG:teuthology.orchestra.run.smithi016:> 2024-03-31T15:05:34.208 DEBUG:teuthology.orchestra.run.smithi016:> def _find_admin_socket(client_name): 2024-03-31T15:05:34.208 DEBUG:teuthology.orchestra.run.smithi016:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-03-31T15:05:34.208 DEBUG:teuthology.orchestra.run.smithi016:> files = glob.glob(asok_path) 2024-03-31T15:05:34.208 DEBUG:teuthology.orchestra.run.smithi016:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-03-31T15:05:34.208 DEBUG:teuthology.orchestra.run.smithi016:> 2024-03-31T15:05:34.208 DEBUG:teuthology.orchestra.run.smithi016:> # Given a non-glob path, it better be there 2024-03-31T15:05:34.208 DEBUG:teuthology.orchestra.run.smithi016:> if "*" not in asok_path: 2024-03-31T15:05:34.208 DEBUG:teuthology.orchestra.run.smithi016:> assert(len(files) == 1) 2024-03-31T15:05:34.208 DEBUG:teuthology.orchestra.run.smithi016:> return files[0] 2024-03-31T15:05:34.208 DEBUG:teuthology.orchestra.run.smithi016:> 2024-03-31T15:05:34.208 DEBUG:teuthology.orchestra.run.smithi016:> for f in files: 2024-03-31T15:05:34.208 DEBUG:teuthology.orchestra.run.smithi016:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-31T15:05:34.208 DEBUG:teuthology.orchestra.run.smithi016:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-31T15:05:34.208 DEBUG:teuthology.orchestra.run.smithi016:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-31T15:05:34.208 DEBUG:teuthology.orchestra.run.smithi016:> contents = proc_f.read() 2024-03-31T15:05:34.209 DEBUG:teuthology.orchestra.run.smithi016:> if mountpoint in contents: 2024-03-31T15:05:34.209 DEBUG:teuthology.orchestra.run.smithi016:> return f 2024-03-31T15:05:34.209 DEBUG:teuthology.orchestra.run.smithi016:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-31T15:05:34.209 DEBUG:teuthology.orchestra.run.smithi016:> 2024-03-31T15:05:34.209 DEBUG:teuthology.orchestra.run.smithi016:> print(_find_admin_socket("client.0")) 2024-03-31T15:05:34.209 DEBUG:teuthology.orchestra.run.smithi016:> ' 2024-03-31T15:05:34.331 INFO:teuthology.orchestra.run.smithi016.stdout:/var/run/ceph/ceph-client.0.38076.asok 2024-03-31T15:05:34.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:34 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-31T15:05:34.343 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.38076.asok 2024-03-31T15:05:34.343 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:05:34.343 DEBUG:teuthology.orchestra.run.smithi016:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.38076.asok status 2024-03-31T15:05:34.562 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-03-31T15:05:34.562 INFO:teuthology.orchestra.run.smithi016.stdout: "metadata": { 2024-03-31T15:05:34.562 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph_sha1": "d2271579d5e735cfd6d436b8349c004f06527cb0", 2024-03-31T15:05:34.563 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph_version": "ceph version 19.0.0-2574-gd2271579 (d2271579d5e735cfd6d436b8349c004f06527cb0) squid (dev)", 2024-03-31T15:05:34.563 INFO:teuthology.orchestra.run.smithi016.stdout: "entity_id": "0", 2024-03-31T15:05:34.563 INFO:teuthology.orchestra.run.smithi016.stdout: "hostname": "smithi016", 2024-03-31T15:05:34.563 INFO:teuthology.orchestra.run.smithi016.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-03-31T15:05:34.563 INFO:teuthology.orchestra.run.smithi016.stdout: "pid": "38076", 2024-03-31T15:05:34.563 INFO:teuthology.orchestra.run.smithi016.stdout: "root": "/" 2024-03-31T15:05:34.563 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-31T15:05:34.563 INFO:teuthology.orchestra.run.smithi016.stdout: "dentry_count": 0, 2024-03-31T15:05:34.563 INFO:teuthology.orchestra.run.smithi016.stdout: "dentry_pinned_count": 0, 2024-03-31T15:05:34.563 INFO:teuthology.orchestra.run.smithi016.stdout: "id": 5859, 2024-03-31T15:05:34.563 INFO:teuthology.orchestra.run.smithi016.stdout: "inst": { 2024-03-31T15:05:34.563 INFO:teuthology.orchestra.run.smithi016.stdout: "name": { 2024-03-31T15:05:34.563 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "client", 2024-03-31T15:05:34.563 INFO:teuthology.orchestra.run.smithi016.stdout: "num": 5859 2024-03-31T15:05:34.563 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-31T15:05:34.564 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": { 2024-03-31T15:05:34.564 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-03-31T15:05:34.564 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "192.168.0.1:0", 2024-03-31T15:05:34.564 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 4291969114 2024-03-31T15:05:34.564 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-31T15:05:34.564 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-31T15:05:34.564 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": { 2024-03-31T15:05:34.564 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-03-31T15:05:34.564 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "192.168.0.1:0", 2024-03-31T15:05:34.564 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 4291969114 2024-03-31T15:05:34.564 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-31T15:05:34.564 INFO:teuthology.orchestra.run.smithi016.stdout: "inst_str": "client.5859 192.168.0.1:0/4291969114", 2024-03-31T15:05:34.564 INFO:teuthology.orchestra.run.smithi016.stdout: "addr_str": "192.168.0.1:0/4291969114", 2024-03-31T15:05:34.564 INFO:teuthology.orchestra.run.smithi016.stdout: "inode_count": 1, 2024-03-31T15:05:34.565 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_epoch": 25, 2024-03-31T15:05:34.565 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_epoch": 45, 2024-03-31T15:05:34.565 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_epoch_barrier": 0, 2024-03-31T15:05:34.565 INFO:teuthology.orchestra.run.smithi016.stdout: "blocklisted": false, 2024-03-31T15:05:34.565 INFO:teuthology.orchestra.run.smithi016.stdout: "fs_name": "cephfs" 2024-03-31T15:05:34.565 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-03-31T15:05:34.581 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:05:34.581 DEBUG:teuthology.orchestra.run.smithi016:> stat --file-system '--printf=%T 2024-03-31T15:05:34.581 DEBUG:teuthology.orchestra.run.smithi016:> ' -- /home/ubuntu/cephtest/mnt.0 2024-03-31T15:05:34.590 INFO:teuthology.orchestra.run.smithi016.stdout:fuseblk 2024-03-31T15:05:34.590 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-03-31T15:05:34.590 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:05:34.590 DEBUG:teuthology.orchestra.run.smithi016:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-31T15:05:34.653 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:05:34.653 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir parent_a) 2024-03-31T15:05:34.717 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:05:34.717 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec touch parent_a/alpha) 2024-03-31T15:05:34.776 DEBUG:teuthology.orchestra.run.smithi016:> stdin-killer -- python3 -c ' 2024-03-31T15:05:34.776 DEBUG:teuthology.orchestra.run.smithi016:> import os 2024-03-31T15:05:34.776 DEBUG:teuthology.orchestra.run.smithi016:> import stat 2024-03-31T15:05:34.776 DEBUG:teuthology.orchestra.run.smithi016:> 2024-03-31T15:05:34.776 DEBUG:teuthology.orchestra.run.smithi016:> print(os.stat("/home/ubuntu/cephtest/mnt.0/parent_a/alpha").st_ino) 2024-03-31T15:05:34.776 DEBUG:teuthology.orchestra.run.smithi016:> ' 2024-03-31T15:05:34.885 INFO:teuthology.orchestra.run.smithi016.stdout:1099511627777 2024-03-31T15:05:34.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:34 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-31T15:05:34.895 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-31T15:05:35.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.083+0000 7fecfc800640 1 Processor -- start 2024-03-31T15:05:35.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.083+0000 7fecfc800640 1 -- start start 2024-03-31T15:05:35.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.083+0000 7fecfc800640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fecf406bd30 0x7fecf4166ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:35.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.083+0000 7fecfc800640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fecf4166470 con 0x7fecf4073570 2024-03-31T15:05:35.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.083+0000 7fecfc800640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fecf40d09d0 con 0x7fecf4072ce0 2024-03-31T15:05:35.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.083+0000 7fecfc800640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fecf40d0cb0 con 0x7fecf406bd30 2024-03-31T15:05:35.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.083+0000 7fecf9e00640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fecf4073570 0x7fecf40cff80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:35556/0 (socket says 172.21.15.16:35556) 2024-03-31T15:05:35.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.083+0000 7fecf9e00640 1 -- 172.21.15.16:0/2297319765 learned_addr learned my addr 172.21.15.16:0/2297319765 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:05:35.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.083+0000 7fecfa800640 1 --2- 172.21.15.16:0/2297319765 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fecf406bd30 0x7fecf4166ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:35.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.083+0000 7fecf8a00640 1 -- 172.21.15.16:0/2297319765 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2373390642 0 0) 0x7fecf4166470 con 0x7fecf4073570 2024-03-31T15:05:35.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.083+0000 7fecfa800640 1 -- 172.21.15.16:0/2297319765 >> v1:172.21.15.89:6789/0 conn(0x7fecf4072ce0 legacy=0x7fecf406b620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:35.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.083+0000 7fecfa800640 1 -- 172.21.15.16:0/2297319765 >> v1:172.21.15.16:6789/0 conn(0x7fecf4073570 legacy=0x7fecf40cff80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:35.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.083+0000 7fecfa800640 1 -- 172.21.15.16:0/2297319765 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fecf41666e0 con 0x7fecf406bd30 2024-03-31T15:05:35.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.087+0000 7fecfa800640 1 --2- 172.21.15.16:0/2297319765 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fecf406bd30 0x7fecf4166ee0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fecf0009dc0 tx=0x7fecf001c990 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e2638660de8bf3bf server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:05:35.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.087+0000 7fecf8a00640 1 -- 172.21.15.16:0/2297319765 <== mon.2 v2:172.21.15.89:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecf00b2070 con 0x7fecf406bd30 2024-03-31T15:05:35.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.087+0000 7fecf8a00640 1 -- 172.21.15.16:0/2297319765 <== mon.2 v2:172.21.15.89:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fecf00a5850 con 0x7fecf406bd30 2024-03-31T15:05:35.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.087+0000 7fecf8a00640 1 -- 172.21.15.16:0/2297319765 <== mon.2 v2:172.21.15.89:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecf00a5c20 con 0x7fecf406bd30 2024-03-31T15:05:35.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.091+0000 7fecfc800640 1 -- 172.21.15.16:0/2297319765 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fecf406bd30 msgr2=0x7fecf4166ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:35.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.091+0000 7fecfc800640 1 --2- 172.21.15.16:0/2297319765 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fecf406bd30 0x7fecf4166ee0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fecf0009dc0 tx=0x7fecf001c990 comp rx=0 tx=0).stop 2024-03-31T15:05:35.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.091+0000 7fecfc800640 1 -- 172.21.15.16:0/2297319765 shutdown_connections 2024-03-31T15:05:35.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.091+0000 7fecfc800640 1 --2- 172.21.15.16:0/2297319765 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fecf406bd30 0x7fecf4166ee0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:35.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.091+0000 7fecfc800640 1 -- 172.21.15.16:0/2297319765 >> 172.21.15.16:0/2297319765 conn(0x7fecf40d1060 msgr2=0x7fecf40d1470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:05:35.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.091+0000 7fecfc800640 1 -- 172.21.15.16:0/2297319765 shutdown_connections 2024-03-31T15:05:35.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.091+0000 7fecfc800640 1 -- 172.21.15.16:0/2297319765 wait complete. 2024-03-31T15:05:35.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.091+0000 7fecfc800640 1 Processor -- start 2024-03-31T15:05:35.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.091+0000 7fecfc800640 1 -- start start 2024-03-31T15:05:35.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.091+0000 7fecfc800640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fecf406bd30 0x7fecf414c060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:35.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.091+0000 7fecfc800640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fecf4072ce0 0x7fecf41fc280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:35.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.091+0000 7fecfc800640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fecf4073570 0x7fecf4150d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:35.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.091+0000 7fecfc800640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fecf4163870 con 0x7fecf4072ce0 2024-03-31T15:05:35.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.091+0000 7fecfc800640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fecf41636f0 con 0x7fecf4073570 2024-03-31T15:05:35.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.091+0000 7fecfc800640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fecf41639f0 con 0x7fecf406bd30 2024-03-31T15:05:35.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.091+0000 7fecf9400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fecf4072ce0 0x7fecf41fc280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:35.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.091+0000 7fecf9400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fecf4072ce0 0x7fecf41fc280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34290/0 (socket says 172.21.15.16:34290) 2024-03-31T15:05:35.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.091+0000 7fecf9400640 1 -- 172.21.15.16:0/1373033416 learned_addr learned my addr 172.21.15.16:0/1373033416 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:05:35.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.091+0000 7fecf9e00640 1 --2- 172.21.15.16:0/1373033416 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fecf406bd30 0x7fecf414c060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:35.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.091+0000 7fecfa800640 1 --2- 172.21.15.16:0/1373033416 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fecf4073570 0x7fecf4150d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:35.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.091+0000 7fecf9400640 1 -- 172.21.15.16:0/1373033416 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fecf406bd30 msgr2=0x7fecf414c060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:35.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.091+0000 7fecf9400640 1 --2- 172.21.15.16:0/1373033416 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fecf406bd30 0x7fecf414c060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:35.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.091+0000 7fecf9400640 1 -- 172.21.15.16:0/1373033416 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fecf4073570 msgr2=0x7fecf4150d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:35.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.091+0000 7fecf9400640 1 --2- 172.21.15.16:0/1373033416 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fecf4073570 0x7fecf4150d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:35.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.091+0000 7fecf9400640 1 -- 172.21.15.16:0/1373033416 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fecf414cac0 con 0x7fecf4072ce0 2024-03-31T15:05:35.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.091+0000 7fecf9e00640 1 --2- 172.21.15.16:0/1373033416 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fecf406bd30 0x7fecf414c060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:05:35.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.091+0000 7fecf9400640 1 --2- 172.21.15.16:0/1373033416 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fecf4072ce0 0x7fecf41fc280 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7fece400c4d0 tx=0x7fece400c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:05:35.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.095+0000 7fecee000640 1 -- 172.21.15.16:0/1373033416 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fece40052d0 con 0x7fecf4072ce0 2024-03-31T15:05:35.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.095+0000 7fecee000640 1 -- 172.21.15.16:0/1373033416 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fece4005470 con 0x7fecf4072ce0 2024-03-31T15:05:35.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.095+0000 7fecfc800640 1 -- 172.21.15.16:0/1373033416 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fecf414d2e0 con 0x7fecf4072ce0 2024-03-31T15:05:35.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.095+0000 7fecee000640 1 -- 172.21.15.16:0/1373033416 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fece4012830 con 0x7fecf4072ce0 2024-03-31T15:05:35.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.095+0000 7fecfc800640 1 -- 172.21.15.16:0/1373033416 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fecf414d8a0 con 0x7fecf4072ce0 2024-03-31T15:05:35.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.095+0000 7fecfc800640 1 -- 172.21.15.16:0/1373033416 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fecf4069950 con 0x7fecf4072ce0 2024-03-31T15:05:35.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.095+0000 7fecee000640 1 -- 172.21.15.16:0/1373033416 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fece40129d0 con 0x7fecf4072ce0 2024-03-31T15:05:35.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.099+0000 7fecee000640 1 --2- 172.21.15.16:0/1373033416 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fecd8068380 0x7fecd806a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:35.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.099+0000 7fecee000640 1 -- 172.21.15.16:0/1373033416 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5709+0+0 (secure 0 0 0) 0x7fece40863d0 con 0x7fecf4072ce0 2024-03-31T15:05:35.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.099+0000 7fecf9e00640 1 --2- 172.21.15.16:0/1373033416 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fecd8068380 0x7fecd806a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:35.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.099+0000 7fecf9e00640 1 --2- 172.21.15.16:0/1373033416 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fecd8068380 0x7fecd806a840 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fece8007bc0 tx=0x7fece8007b10 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:05:35.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.107+0000 7fecee000640 1 -- 172.21.15.16:0/1373033416 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fece40563d0 con 0x7fecf4072ce0 2024-03-31T15:05:35.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.307+0000 7fecfc800640 1 -- 172.21.15.16:0/1373033416 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fecf40d1bf0 con 0x7fecf4072ce0 2024-03-31T15:05:35.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.307+0000 7fecee000640 1 -- 172.21.15.16:0/1373033416 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+5175 (secure 0 0 0) 0x7fece405a6d0 con 0x7fecf4072ce0 2024-03-31T15:05:35.304 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:05:35.304 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":25,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/285985105","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6835","nonce":285985105},{"type":"v1","addr":"172.21.15.89:6837","nonce":285985105}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20},{"gid":5689,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6836/1615931247","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":1615931247},{"type":"v1","addr":"172.21.15.89:6836","nonce":1615931247}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20},{"gid":5694,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/512187852","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6833","nonce":512187852},{"type":"v1","addr":"172.21.15.16:6835","nonce":512187852}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-31T15:05:26.461908+0000","modified":"2024-03-31T15:05:29.491163+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5697},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5697":{"gid":5697,"name":"a","rank":0,"incarnation":21,"state":"up:active","state_seq":3,"addr":"172.21.15.16:6834/2768973409","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":2768973409},{"type":"v1","addr":"172.21.15.16:6834","nonce":2768973409}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5697,"qdb_cluster":[5697]},"id":3}]} 2024-03-31T15:05:35.304 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 25 2024-03-31T15:05:35.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.311+0000 7fecfc800640 1 -- 172.21.15.16:0/1373033416 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fecd8068380 msgr2=0x7fecd806a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:35.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.311+0000 7fecfc800640 1 --2- 172.21.15.16:0/1373033416 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fecd8068380 0x7fecd806a840 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fece8007bc0 tx=0x7fece8007b10 comp rx=0 tx=0).stop 2024-03-31T15:05:35.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.311+0000 7fecfc800640 1 -- 172.21.15.16:0/1373033416 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fecf4072ce0 msgr2=0x7fecf41fc280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:35.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.311+0000 7fecfc800640 1 --2- 172.21.15.16:0/1373033416 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fecf4072ce0 0x7fecf41fc280 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7fece400c4d0 tx=0x7fece400c990 comp rx=0 tx=0).stop 2024-03-31T15:05:35.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.311+0000 7fecfc800640 1 -- 172.21.15.16:0/1373033416 shutdown_connections 2024-03-31T15:05:35.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.311+0000 7fecfc800640 1 --2- 172.21.15.16:0/1373033416 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fecd8068380 0x7fecd806a840 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:35.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.311+0000 7fecfc800640 1 --2- 172.21.15.16:0/1373033416 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fecf4073570 0x7fecf4150d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:35.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.311+0000 7fecfc800640 1 --2- 172.21.15.16:0/1373033416 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fecf4072ce0 0x7fecf41fc280 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:35.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.311+0000 7fecfc800640 1 --2- 172.21.15.16:0/1373033416 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fecf406bd30 0x7fecf414c060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:35.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.311+0000 7fecfc800640 1 -- 172.21.15.16:0/1373033416 >> 172.21.15.16:0/1373033416 conn(0x7fecf40d1060 msgr2=0x7fecf406b620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:05:35.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.311+0000 7fecfc800640 1 -- 172.21.15.16:0/1373033416 shutdown_connections 2024-03-31T15:05:35.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.311+0000 7fecfc800640 1 -- 172.21.15.16:0/1373033416 wait complete. 2024-03-31T15:05:35.326 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json flush journal 2024-03-31T15:05:35.503 INFO:teuthology.orchestra.run.smithi016.stdout:{"message":"","return_code":0} 2024-03-31T15:05:35.504 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2024-03-31T15:05:35.504 DEBUG:teuthology.orchestra.run.smithi016:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_data getxattr 10000000001.00000000 parent 2024-03-31T15:05:35.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.555+0000 7f313cbcc080 1 Processor -- start 2024-03-31T15:05:35.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.555+0000 7f313cbcc080 1 -- start start 2024-03-31T15:05:35.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.555+0000 7f313cbcc080 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55b4928b6040 0x55b4929880a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:35.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.555+0000 7f313cbcc080 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b492986770 0x55b492984b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:35.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.555+0000 7f313cbcc080 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b492987090 0x55b4929850a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:35.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.555+0000 7f313cbcc080 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55b4928b3740 con 0x55b492987090 2024-03-31T15:05:35.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.555+0000 7f313cbcc080 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55b4928574b0 con 0x55b492986770 2024-03-31T15:05:35.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.555+0000 7f313cbcc080 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x55b492989630 con 0x55b4928b6040 2024-03-31T15:05:35.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.555+0000 7f313a800640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b492987090 0x55b4929850a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:35.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.555+0000 7f313a800640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b492987090 0x55b4929850a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34302/0 (socket says 172.21.15.16:34302) 2024-03-31T15:05:35.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.555+0000 7f313a800640 1 -- 172.21.15.16:0/3537429418 learned_addr learned my addr 172.21.15.16:0/3537429418 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:05:35.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.555+0000 7f3139e00640 1 --2- 172.21.15.16:0/3537429418 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55b4928b6040 0x55b4929880a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:35.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.555+0000 7f3139400640 1 --2- 172.21.15.16:0/3537429418 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b492986770 0x55b492984b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:35.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.559+0000 7f313a800640 1 -- 172.21.15.16:0/3537429418 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55b4928b6040 msgr2=0x55b4929880a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:05:35.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.559+0000 7f313a800640 1 --2- 172.21.15.16:0/3537429418 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55b4928b6040 0x55b4929880a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:35.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.559+0000 7f313a800640 1 -- 172.21.15.16:0/3537429418 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b492986770 msgr2=0x55b492984b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:05:35.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.559+0000 7f313a800640 1 --2- 172.21.15.16:0/3537429418 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b492986770 0x55b492984b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:35.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.559+0000 7f313a800640 1 -- 172.21.15.16:0/3537429418 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b492989b00 con 0x55b492987090 2024-03-31T15:05:35.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.559+0000 7f3139400640 1 --2- 172.21.15.16:0/3537429418 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b492986770 0x55b492984b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:05:35.554 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.559+0000 7f313a800640 1 --2- 172.21.15.16:0/3537429418 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b492987090 0x55b4929850a0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f3134001740 tx=0x7f3134001c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=efddb008472d0423 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:05:35.554 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.559+0000 7f3133e00640 1 -- 172.21.15.16:0/3537429418 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3134016070 con 0x55b492987090 2024-03-31T15:05:35.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.559+0000 7f3133e00640 1 -- 172.21.15.16:0/3537429418 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3134004530 con 0x55b492987090 2024-03-31T15:05:35.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.559+0000 7f3133e00640 1 -- 172.21.15.16:0/3537429418 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3134004920 con 0x55b492987090 2024-03-31T15:05:35.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.559+0000 7f313cbcc080 1 -- 172.21.15.16:0/3537429418 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b492987090 msgr2=0x55b4929850a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:35.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.559+0000 7f313cbcc080 1 --2- 172.21.15.16:0/3537429418 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b492987090 0x55b4929850a0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f3134001740 tx=0x7f3134001c00 comp rx=0 tx=0).stop 2024-03-31T15:05:35.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.559+0000 7f313cbcc080 1 -- 172.21.15.16:0/3537429418 shutdown_connections 2024-03-31T15:05:35.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.559+0000 7f313cbcc080 1 --2- 172.21.15.16:0/3537429418 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b492987090 0x55b4929850a0 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:35.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.559+0000 7f313cbcc080 1 --2- 172.21.15.16:0/3537429418 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b492986770 0x55b492984b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:35.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.559+0000 7f313cbcc080 1 --2- 172.21.15.16:0/3537429418 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55b4928b6040 0x55b4929880a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:35.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.559+0000 7f313cbcc080 1 -- 172.21.15.16:0/3537429418 >> 172.21.15.16:0/3537429418 conn(0x55b4928fa0b0 msgr2=0x55b492985ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:05:35.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.559+0000 7f313cbcc080 1 -- 172.21.15.16:0/3537429418 shutdown_connections 2024-03-31T15:05:35.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.559+0000 7f313cbcc080 1 -- 172.21.15.16:0/3537429418 wait complete. 2024-03-31T15:05:35.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.559+0000 7f313cbcc080 1 Processor -- start 2024-03-31T15:05:35.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.559+0000 7f313cbcc080 1 -- start start 2024-03-31T15:05:35.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.563+0000 7f313cbcc080 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b4928b6040 0x55b4929847c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:35.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.563+0000 7f313cbcc080 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55b492986770 0x55b492968b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:35.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.563+0000 7f313cbcc080 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b492987090 0x55b492982d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:35.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.563+0000 7f313cbcc080 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55b492989860 con 0x55b492987090 2024-03-31T15:05:35.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.563+0000 7f313cbcc080 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55b4929855e0 con 0x55b4928b6040 2024-03-31T15:05:35.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.563+0000 7f313cbcc080 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x55b4928fcc80 con 0x55b492986770 2024-03-31T15:05:35.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.563+0000 7f313a800640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b492987090 0x55b492982d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:35.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.563+0000 7f313a800640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b492987090 0x55b492982d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34306/0 (socket says 172.21.15.16:34306) 2024-03-31T15:05:35.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.563+0000 7f313a800640 1 -- 172.21.15.16:0/3856878569 learned_addr learned my addr 172.21.15.16:0/3856878569 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:05:35.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.563+0000 7f3139e00640 1 --2- 172.21.15.16:0/3856878569 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b4928b6040 0x55b4929847c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:35.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.563+0000 7f3139400640 1 --2- 172.21.15.16:0/3856878569 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55b492986770 0x55b492968b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:35.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.563+0000 7f313a800640 1 -- 172.21.15.16:0/3856878569 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55b492986770 msgr2=0x55b492968b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:35.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.563+0000 7f313a800640 1 --2- 172.21.15.16:0/3856878569 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55b492986770 0x55b492968b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:35.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.563+0000 7f313a800640 1 -- 172.21.15.16:0/3856878569 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b4928b6040 msgr2=0x55b4929847c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:35.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.563+0000 7f313a800640 1 --2- 172.21.15.16:0/3856878569 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b4928b6040 0x55b4929847c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:35.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.563+0000 7f313a800640 1 -- 172.21.15.16:0/3856878569 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b4928f8bc0 con 0x55b492987090 2024-03-31T15:05:35.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.563+0000 7f313a800640 1 --2- 172.21.15.16:0/3856878569 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b492987090 0x55b492982d90 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f31340012f0 tx=0x7f31340014e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:05:35.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.563+0000 7f3132a00640 1 -- 172.21.15.16:0/3856878569 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3134016030 con 0x55b492987090 2024-03-31T15:05:35.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.563+0000 7f3132a00640 1 -- 172.21.15.16:0/3856878569 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3134004d70 con 0x55b492987090 2024-03-31T15:05:35.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.563+0000 7f313cbcc080 1 -- 172.21.15.16:0/3856878569 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55b4929838a0 con 0x55b492987090 2024-03-31T15:05:35.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.563+0000 7f3132a00640 1 -- 172.21.15.16:0/3856878569 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f313401d670 con 0x55b492987090 2024-03-31T15:05:35.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.563+0000 7f313cbcc080 1 -- 172.21.15.16:0/3856878569 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b492983de0 con 0x55b492987090 2024-03-31T15:05:35.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.567+0000 7f3132a00640 1 -- 172.21.15.16:0/3856878569 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f313401d810 con 0x55b492987090 2024-03-31T15:05:35.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.567+0000 7f3132a00640 1 --2- 172.21.15.16:0/3856878569 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f311c066d90 0x7f311c069250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:35.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.567+0000 7f3132a00640 1 -- 172.21.15.16:0/3856878569 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5709+0+0 (secure 0 0 0) 0x7f3134086bd0 con 0x55b492987090 2024-03-31T15:05:35.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.567+0000 7f3139e00640 1 --2- 172.21.15.16:0/3856878569 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f311c066d90 0x7f311c069250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:35.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.567+0000 7f313cbcc080 1 --2- 172.21.15.16:0/3856878569 >> [v2:172.21.15.89:6809/3470471752,v1:172.21.15.89:6812/3470471752] conn(0x55b4928f7f00 0x55b4929695a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:35.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.567+0000 7f313cbcc080 1 -- 172.21.15.16:0/3856878569 --> [v2:172.21.15.89:6809/3470471752,v1:172.21.15.89:6812/3470471752] -- osd_op(unknown.0.0:1 7.7 7:e2e07930:::10000000001.00000000:head [getxattr parent in=6b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e45) -- 0x55b492969ae0 con 0x55b4928f7f00 2024-03-31T15:05:35.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.567+0000 7f3139400640 1 --2- 172.21.15.16:0/3856878569 >> [v2:172.21.15.89:6809/3470471752,v1:172.21.15.89:6812/3470471752] conn(0x55b4928f7f00 0x55b4929695a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:35.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.567+0000 7f3139e00640 1 --2- 172.21.15.16:0/3856878569 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f311c066d90 0x7f311c069250 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f312c0b1000 tx=0x7f312c0a9070 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:05:35.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.567+0000 7f3139400640 1 --2- 172.21.15.16:0/3856878569 >> [v2:172.21.15.89:6809/3470471752,v1:172.21.15.89:6812/3470471752] conn(0x55b4928f7f00 0x55b4929695a0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:05:35.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.567+0000 7f3139400640 1 -- 172.21.15.16:0/3856878569 <== osd.6 v2:172.21.15.89:6809/3470471752 1 ==== osd_op_reply(1 10000000001.00000000 [getxattr (95) out=95b] v0'0 uv1 ondisk = 0) ==== 164+0+95 (crc 0 0 0) 0x7f3128009a30 con 0x55b4928f7f00 2024-03-31T15:05:35.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.567+0000 7f313cbcc080 1 -- 172.21.15.16:0/3856878569 >> [v2:172.21.15.89:6809/3470471752,v1:172.21.15.89:6812/3470471752] conn(0x55b4928f7f00 msgr2=0x55b4929695a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:35.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.567+0000 7f313cbcc080 1 --2- 172.21.15.16:0/3856878569 >> [v2:172.21.15.89:6809/3470471752,v1:172.21.15.89:6812/3470471752] conn(0x55b4928f7f00 0x55b4929695a0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:35.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.567+0000 7f313cbcc080 1 -- 172.21.15.16:0/3856878569 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f311c066d90 msgr2=0x7f311c069250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:35.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.567+0000 7f313cbcc080 1 --2- 172.21.15.16:0/3856878569 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f311c066d90 0x7f311c069250 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f312c0b1000 tx=0x7f312c0a9070 comp rx=0 tx=0).stop 2024-03-31T15:05:35.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.567+0000 7f313cbcc080 1 -- 172.21.15.16:0/3856878569 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b492987090 msgr2=0x55b492982d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:35.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.567+0000 7f313cbcc080 1 --2- 172.21.15.16:0/3856878569 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b492987090 0x55b492982d90 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f31340012f0 tx=0x7f31340014e0 comp rx=0 tx=0).stop 2024-03-31T15:05:35.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.567+0000 7f313a800640 1 -- 172.21.15.16:0/3856878569 reap_dead start 2024-03-31T15:05:35.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.571+0000 7f313cbcc080 1 -- 172.21.15.16:0/3856878569 shutdown_connections 2024-03-31T15:05:35.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.571+0000 7f313cbcc080 1 -- 172.21.15.16:0/3856878569 >> 172.21.15.16:0/3856878569 conn(0x55b4928fa0b0 msgr2=0x55b492985810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:05:35.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.571+0000 7f313cbcc080 1 -- 172.21.15.16:0/3856878569 shutdown_connections 2024-03-31T15:05:35.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.571+0000 7f313cbcc080 1 -- 172.21.15.16:0/3856878569 wait complete. 2024-03-31T15:05:35.568 INFO:teuthology.orchestra.run.smithi016.stdout:Yalphaparent_a 2024-03-31T15:05:35.569 DEBUG:teuthology.orchestra.run.smithi016:> ceph-dencoder type inode_backtrace_t import - decode dump_json 2024-03-31T15:05:35.627 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-03-31T15:05:35.627 INFO:teuthology.orchestra.run.smithi016.stdout: "ino": 1099511627777, 2024-03-31T15:05:35.627 INFO:teuthology.orchestra.run.smithi016.stdout: "ancestors": [ 2024-03-31T15:05:35.627 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-31T15:05:35.628 INFO:teuthology.orchestra.run.smithi016.stdout: "dirino": 1099511627776, 2024-03-31T15:05:35.628 INFO:teuthology.orchestra.run.smithi016.stdout: "dname": "alpha", 2024-03-31T15:05:35.628 INFO:teuthology.orchestra.run.smithi016.stdout: "version": 2 2024-03-31T15:05:35.628 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-31T15:05:35.628 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-31T15:05:35.628 INFO:teuthology.orchestra.run.smithi016.stdout: "dirino": 1, 2024-03-31T15:05:35.628 INFO:teuthology.orchestra.run.smithi016.stdout: "dname": "parent_a", 2024-03-31T15:05:35.628 INFO:teuthology.orchestra.run.smithi016.stdout: "version": 4 2024-03-31T15:05:35.628 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-31T15:05:35.628 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-03-31T15:05:35.628 INFO:teuthology.orchestra.run.smithi016.stdout: "pool": 7, 2024-03-31T15:05:35.629 INFO:teuthology.orchestra.run.smithi016.stdout: "old_pools": [] 2024-03-31T15:05:35.629 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-03-31T15:05:35.629 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:05:35.631 DEBUG:teuthology.orchestra.run.smithi016:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_data setxattr 10000000001.00000000 parent 'oh i'"'"'m sorry did i overwrite your xattr?' 2024-03-31T15:05:35.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.723+0000 7ff099d7a080 1 Processor -- start 2024-03-31T15:05:35.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.723+0000 7ff099d7a080 1 -- start start 2024-03-31T15:05:35.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.723+0000 7ff099d7a080 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560656a77350 0x560656ae6830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:35.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.723+0000 7ff099d7a080 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x560656a77730 0x560656ae6d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:35.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.723+0000 7ff099d7a080 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560656ae79a0 0x560656aeaa60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:35.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.723+0000 7ff099d7a080 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x560656ae8d50 con 0x560656a77350 2024-03-31T15:05:35.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.723+0000 7ff099d7a080 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x5606569d34b0 con 0x560656ae79a0 2024-03-31T15:05:35.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.723+0000 7ff099d7a080 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x560656ae8ed0 con 0x560656a77730 2024-03-31T15:05:35.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.727+0000 7ff097000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560656a77350 0x560656ae6830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:35.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.727+0000 7ff097000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560656a77350 0x560656ae6830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34318/0 (socket says 172.21.15.16:34318) 2024-03-31T15:05:35.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.727+0000 7ff097000640 1 -- 172.21.15.16:0/1780964471 learned_addr learned my addr 172.21.15.16:0/1780964471 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:05:35.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.727+0000 7ff096600640 1 --2- 172.21.15.16:0/1780964471 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x560656a77730 0x560656ae6d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:35.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.727+0000 7ff097a00640 1 --2- 172.21.15.16:0/1780964471 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560656ae79a0 0x560656aeaa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:35.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.727+0000 7ff097000640 1 -- 172.21.15.16:0/1780964471 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x560656a77730 msgr2=0x560656ae6d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:05:35.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.727+0000 7ff097000640 1 --2- 172.21.15.16:0/1780964471 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x560656a77730 0x560656ae6d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:35.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.727+0000 7ff097000640 1 -- 172.21.15.16:0/1780964471 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560656ae79a0 msgr2=0x560656aeaa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:05:35.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.727+0000 7ff097000640 1 --2- 172.21.15.16:0/1780964471 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560656ae79a0 0x560656aeaa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:35.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.727+0000 7ff097000640 1 -- 172.21.15.16:0/1780964471 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560656a5c880 con 0x560656a77350 2024-03-31T15:05:35.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.727+0000 7ff097000640 1 --2- 172.21.15.16:0/1780964471 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560656a77350 0x560656ae6830 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7ff08800b280 tx=0x7ff08800b740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c53d48a35febea8 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:05:35.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.727+0000 7ff095200640 1 -- 172.21.15.16:0/1780964471 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff08800d510 con 0x560656a77350 2024-03-31T15:05:35.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.727+0000 7ff095200640 1 -- 172.21.15.16:0/1780964471 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff088008070 con 0x560656a77350 2024-03-31T15:05:35.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.727+0000 7ff095200640 1 -- 172.21.15.16:0/1780964471 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff08800d8b0 con 0x560656a77350 2024-03-31T15:05:35.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.727+0000 7ff099d7a080 1 -- 172.21.15.16:0/1780964471 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560656a77350 msgr2=0x560656ae6830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:35.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.727+0000 7ff099d7a080 1 --2- 172.21.15.16:0/1780964471 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560656a77350 0x560656ae6830 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7ff08800b280 tx=0x7ff08800b740 comp rx=0 tx=0).stop 2024-03-31T15:05:35.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.727+0000 7ff099d7a080 1 -- 172.21.15.16:0/1780964471 shutdown_connections 2024-03-31T15:05:35.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.727+0000 7ff099d7a080 1 --2- 172.21.15.16:0/1780964471 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560656ae79a0 0x560656aeaa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:35.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.727+0000 7ff099d7a080 1 --2- 172.21.15.16:0/1780964471 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x560656a77730 0x560656ae6d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:35.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.727+0000 7ff099d7a080 1 --2- 172.21.15.16:0/1780964471 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560656a77350 0x560656ae6830 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:35.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.727+0000 7ff099d7a080 1 -- 172.21.15.16:0/1780964471 >> 172.21.15.16:0/1780964471 conn(0x560656a59050 msgr2=0x560656a5dbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:05:35.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.727+0000 7ff099d7a080 1 -- 172.21.15.16:0/1780964471 shutdown_connections 2024-03-31T15:05:35.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.727+0000 7ff099d7a080 1 -- 172.21.15.16:0/1780964471 wait complete. 2024-03-31T15:05:35.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.727+0000 7ff099d7a080 1 Processor -- start 2024-03-31T15:05:35.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.727+0000 7ff099d7a080 1 -- start start 2024-03-31T15:05:35.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.727+0000 7ff099d7a080 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x560656a77350 0x560656b033e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:35.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.727+0000 7ff099d7a080 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560656a77730 0x560656ae5cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:35.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.727+0000 7ff099d7a080 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560656ae79a0 0x560656aebd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:35.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.727+0000 7ff099d7a080 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x560656a5cb20 con 0x560656a77730 2024-03-31T15:05:35.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.727+0000 7ff099d7a080 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x560656a5cdc0 con 0x560656ae79a0 2024-03-31T15:05:35.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.727+0000 7ff099d7a080 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x560656a76d30 con 0x560656a77350 2024-03-31T15:05:35.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.727+0000 7ff096600640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560656a77730 0x560656ae5cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:35.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.731+0000 7ff096600640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560656a77730 0x560656ae5cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34330/0 (socket says 172.21.15.16:34330) 2024-03-31T15:05:35.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.731+0000 7ff096600640 1 -- 172.21.15.16:0/3319137102 learned_addr learned my addr 172.21.15.16:0/3319137102 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:05:35.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.731+0000 7ff097000640 1 --2- 172.21.15.16:0/3319137102 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x560656a77350 0x560656b033e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:35.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.731+0000 7ff097a00640 1 --2- 172.21.15.16:0/3319137102 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560656ae79a0 0x560656aebd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:35.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.731+0000 7ff096600640 1 -- 172.21.15.16:0/3319137102 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x560656a77350 msgr2=0x560656b033e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:35.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.731+0000 7ff096600640 1 --2- 172.21.15.16:0/3319137102 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x560656a77350 0x560656b033e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:35.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.731+0000 7ff096600640 1 -- 172.21.15.16:0/3319137102 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560656ae79a0 msgr2=0x560656aebd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:35.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.731+0000 7ff096600640 1 --2- 172.21.15.16:0/3319137102 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560656ae79a0 0x560656aebd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:35.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.731+0000 7ff096600640 1 -- 172.21.15.16:0/3319137102 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560656b024c0 con 0x560656a77730 2024-03-31T15:05:35.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.731+0000 7ff096600640 1 --2- 172.21.15.16:0/3319137102 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560656a77730 0x560656ae5cd0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7ff084008820 tx=0x7ff084008ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:05:35.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.731+0000 7ff08f400640 1 -- 172.21.15.16:0/3319137102 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff084019070 con 0x560656a77730 2024-03-31T15:05:35.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.731+0000 7ff08f400640 1 -- 172.21.15.16:0/3319137102 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff084010ba0 con 0x560656a77730 2024-03-31T15:05:35.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.731+0000 7ff08f400640 1 -- 172.21.15.16:0/3319137102 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff084015450 con 0x560656a77730 2024-03-31T15:05:35.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.731+0000 7ff099d7a080 1 -- 172.21.15.16:0/3319137102 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x560656b02760 con 0x560656a77730 2024-03-31T15:05:35.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.731+0000 7ff099d7a080 1 -- 172.21.15.16:0/3319137102 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560656b02ca0 con 0x560656a77730 2024-03-31T15:05:35.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.735+0000 7ff08f400640 1 -- 172.21.15.16:0/3319137102 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7ff08401c070 con 0x560656a77730 2024-03-31T15:05:35.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.735+0000 7ff08f400640 1 --2- 172.21.15.16:0/3319137102 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7ff078066d90 0x7ff078069250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:35.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.735+0000 7ff08f400640 1 -- 172.21.15.16:0/3319137102 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5709+0+0 (secure 0 0 0) 0x7ff084086260 con 0x560656a77730 2024-03-31T15:05:35.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.735+0000 7ff099d7a080 1 --2- 172.21.15.16:0/3319137102 >> [v2:172.21.15.89:6809/3470471752,v1:172.21.15.89:6812/3470471752] conn(0x560656b01c70 0x560656b9d4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:35.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.735+0000 7ff099d7a080 1 -- 172.21.15.16:0/3319137102 --> [v2:172.21.15.89:6809/3470471752,v1:172.21.15.89:6812/3470471752] -- osd_op(unknown.0.0:1 7.7 7:e2e07930:::10000000001.00000000:head [setxattr parent (40) in=46b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45) -- 0x560656b02050 con 0x560656b01c70 2024-03-31T15:05:35.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.735+0000 7ff097000640 1 --2- 172.21.15.16:0/3319137102 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7ff078066d90 0x7ff078069250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:35.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.735+0000 7ff097a00640 1 --2- 172.21.15.16:0/3319137102 >> [v2:172.21.15.89:6809/3470471752,v1:172.21.15.89:6812/3470471752] conn(0x560656b01c70 0x560656b9d4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:35.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.735+0000 7ff097a00640 1 --2- 172.21.15.16:0/3319137102 >> [v2:172.21.15.89:6809/3470471752,v1:172.21.15.89:6812/3470471752] conn(0x560656b01c70 0x560656b9d4f0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:05:35.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.735+0000 7ff097000640 1 --2- 172.21.15.16:0/3319137102 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7ff078066d90 0x7ff078069250 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7ff088000c10 tx=0x7ff088008000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:05:35.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.739+0000 7ff097a00640 1 -- 172.21.15.16:0/3319137102 <== osd.6 v2:172.21.15.89:6809/3470471752 1 ==== osd_op_reply(1 10000000001.00000000 [setxattr (40)] v45'2 uv2 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7ff0900b2040 con 0x560656b01c70 2024-03-31T15:05:35.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.739+0000 7ff099d7a080 1 -- 172.21.15.16:0/3319137102 >> [v2:172.21.15.89:6809/3470471752,v1:172.21.15.89:6812/3470471752] conn(0x560656b01c70 msgr2=0x560656b9d4f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:35.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.739+0000 7ff099d7a080 1 --2- 172.21.15.16:0/3319137102 >> [v2:172.21.15.89:6809/3470471752,v1:172.21.15.89:6812/3470471752] conn(0x560656b01c70 0x560656b9d4f0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:35.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.739+0000 7ff099d7a080 1 -- 172.21.15.16:0/3319137102 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7ff078066d90 msgr2=0x7ff078069250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:35.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.739+0000 7ff099d7a080 1 --2- 172.21.15.16:0/3319137102 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7ff078066d90 0x7ff078069250 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7ff088000c10 tx=0x7ff088008000 comp rx=0 tx=0).stop 2024-03-31T15:05:35.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.739+0000 7ff099d7a080 1 -- 172.21.15.16:0/3319137102 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560656a77730 msgr2=0x560656ae5cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:35.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.739+0000 7ff099d7a080 1 --2- 172.21.15.16:0/3319137102 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560656a77730 0x560656ae5cd0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7ff084008820 tx=0x7ff084008ce0 comp rx=0 tx=0).stop 2024-03-31T15:05:35.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.739+0000 7ff097a00640 1 -- 172.21.15.16:0/3319137102 reap_dead start 2024-03-31T15:05:35.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.739+0000 7ff099d7a080 1 -- 172.21.15.16:0/3319137102 shutdown_connections 2024-03-31T15:05:35.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.739+0000 7ff099d7a080 1 -- 172.21.15.16:0/3319137102 >> 172.21.15.16:0/3319137102 conn(0x560656a59050 msgr2=0x560656a751c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:05:35.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.739+0000 7ff099d7a080 1 -- 172.21.15.16:0/3319137102 shutdown_connections 2024-03-31T15:05:35.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.739+0000 7ff099d7a080 1 -- 172.21.15.16:0/3319137102 wait complete. 2024-03-31T15:05:35.739 DEBUG:teuthology.orchestra.run.smithi016:> sudo daemon-helper kill ceph --cluster ceph -w 2024-03-31T15:05:35.742 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:05:35.742 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.3:0 scrub start / repair,recursive 2024-03-31T15:05:35.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.891+0000 7f482c800640 1 Processor -- start 2024-03-31T15:05:35.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.891+0000 7f482c800640 1 -- start start 2024-03-31T15:05:35.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.895+0000 7f482c800640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f48240b7900 0x7f4824160740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:35.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.895+0000 7f482c800640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f48240ce2d0 con 0x7f48240cc7b0 2024-03-31T15:05:35.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.895+0000 7f482c800640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f48240ceb10 con 0x7f48240b44d0 2024-03-31T15:05:35.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.895+0000 7f482c800640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f48240ce550 con 0x7f48240b7900 2024-03-31T15:05:35.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.895+0000 7f4829400640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f48240cc7b0 0x7f48240b3dc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:35564/0 (socket says 172.21.15.16:35564) 2024-03-31T15:05:35.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.895+0000 7f4829400640 1 -- 172.21.15.16:0/4167400286 learned_addr learned my addr 172.21.15.16:0/4167400286 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:05:35.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.895+0000 7f4829e00640 1 --2- 172.21.15.16:0/4167400286 >> v2:172.21.15.89:3300/0 conn(0x7f48240b7900 0x7f4824160740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:35.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.895+0000 7f4828a00640 1 -- 172.21.15.16:0/4167400286 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1208981017 0 0) 0x7f4814001120 con 0x7f48240cc7b0 2024-03-31T15:05:35.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.895+0000 7f4828a00640 1 -- 172.21.15.16:0/4167400286 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f480c003530 con 0x7f48240cc7b0 2024-03-31T15:05:35.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.895+0000 7f4829e00640 1 -- 172.21.15.16:0/4167400286 >> v1:172.21.15.89:6789/0 conn(0x7f48240b44d0 legacy=0x7f48240d1a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:35.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.895+0000 7f4829e00640 1 -- 172.21.15.16:0/4167400286 >> v1:172.21.15.16:6789/0 conn(0x7f48240cc7b0 legacy=0x7f48240b3dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:35.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.895+0000 7f4829e00640 1 -- 172.21.15.16:0/4167400286 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48240ce830 con 0x7f48240b7900 2024-03-31T15:05:35.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.899+0000 7f4829e00640 1 --2- 172.21.15.16:0/4167400286 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f48240b7900 0x7f4824160740 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f48180097c0 tx=0x7f481801c3e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cab44ca5db5a7a66 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:05:35.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.899+0000 7f4828a00640 1 -- 172.21.15.16:0/4167400286 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48180b1070 con 0x7f48240b7900 2024-03-31T15:05:35.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.899+0000 7f4828a00640 1 -- 172.21.15.16:0/4167400286 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f48180a8050 con 0x7f48240b7900 2024-03-31T15:05:35.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.899+0000 7f4828a00640 1 -- 172.21.15.16:0/4167400286 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48180ac860 con 0x7f48240b7900 2024-03-31T15:05:35.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.899+0000 7f482c800640 1 -- 172.21.15.16:0/4167400286 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f48240b7900 msgr2=0x7f4824160740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:35.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.899+0000 7f482c800640 1 --2- 172.21.15.16:0/4167400286 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f48240b7900 0x7f4824160740 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f48180097c0 tx=0x7f481801c3e0 comp rx=0 tx=0).stop 2024-03-31T15:05:35.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.899+0000 7f482c800640 1 -- 172.21.15.16:0/4167400286 shutdown_connections 2024-03-31T15:05:35.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.899+0000 7f482c800640 1 --2- 172.21.15.16:0/4167400286 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f48240b7900 0x7f4824160740 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:35.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.899+0000 7f482c800640 1 -- 172.21.15.16:0/4167400286 >> 172.21.15.16:0/4167400286 conn(0x7f482406b7f0 msgr2=0x7f4824067330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:05:35.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.899+0000 7f482c800640 1 -- 172.21.15.16:0/4167400286 shutdown_connections 2024-03-31T15:05:35.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.899+0000 7f482c800640 1 -- 172.21.15.16:0/4167400286 wait complete. 2024-03-31T15:05:35.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.903+0000 7f482c800640 1 Processor -- start 2024-03-31T15:05:35.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.903+0000 7f482c800640 1 -- start start 2024-03-31T15:05:35.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.903+0000 7f482c800640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f48240b44d0 0x7f48241574c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:35.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.903+0000 7f482c800640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f48240b7900 0x7f4824144a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:35.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.903+0000 7f482c800640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f48240cc7b0 0x7f48240b56d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:35.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.903+0000 7f482c800640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f48240d25a0 con 0x7f48240cc7b0 2024-03-31T15:05:35.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.903+0000 7f482c800640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f48240b7ce0 con 0x7f48240b7900 2024-03-31T15:05:35.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.903+0000 7f482c800640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f482415d540 con 0x7f48240b44d0 2024-03-31T15:05:35.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.903+0000 7f482a800640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f48240cc7b0 0x7f48240b56d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:35.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.903+0000 7f482a800640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f48240cc7b0 0x7f48240b56d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34332/0 (socket says 172.21.15.16:34332) 2024-03-31T15:05:35.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.903+0000 7f482a800640 1 -- 172.21.15.16:0/3284504006 learned_addr learned my addr 172.21.15.16:0/3284504006 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:05:35.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.903+0000 7f4829e00640 1 --2- 172.21.15.16:0/3284504006 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f48240b44d0 0x7f48241574c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:35.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.903+0000 7f4829400640 1 --2- 172.21.15.16:0/3284504006 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f48240b7900 0x7f4824144a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:35.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.903+0000 7f482a800640 1 -- 172.21.15.16:0/3284504006 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f48240b44d0 msgr2=0x7f48241574c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:35.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.903+0000 7f482a800640 1 --2- 172.21.15.16:0/3284504006 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f48240b44d0 0x7f48241574c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:35.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.903+0000 7f482a800640 1 -- 172.21.15.16:0/3284504006 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f48240b7900 msgr2=0x7f4824144a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:35.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.903+0000 7f482a800640 1 --2- 172.21.15.16:0/3284504006 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f48240b7900 0x7f4824144a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:35.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.903+0000 7f482a800640 1 -- 172.21.15.16:0/3284504006 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48240c7e00 con 0x7f48240cc7b0 2024-03-31T15:05:35.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.903+0000 7f482a800640 1 --2- 172.21.15.16:0/3284504006 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f48240cc7b0 0x7f48240b56d0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f482000c730 tx=0x7f482000cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:05:35.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.903+0000 7f481e000640 1 -- 172.21.15.16:0/3284504006 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4820003b80 con 0x7f48240cc7b0 2024-03-31T15:05:35.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.903+0000 7f481e000640 1 -- 172.21.15.16:0/3284504006 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4820003d20 con 0x7f48240cc7b0 2024-03-31T15:05:35.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.903+0000 7f481e000640 1 -- 172.21.15.16:0/3284504006 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f482000aa30 con 0x7f48240cc7b0 2024-03-31T15:05:35.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.903+0000 7f482c800640 1 -- 172.21.15.16:0/3284504006 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4824157c30 con 0x7f48240cc7b0 2024-03-31T15:05:35.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.907+0000 7f482c800640 1 -- 172.21.15.16:0/3284504006 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f48241581f0 con 0x7f48240cc7b0 2024-03-31T15:05:35.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.907+0000 7f481e000640 1 -- 172.21.15.16:0/3284504006 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f482000abd0 con 0x7f48240cc7b0 2024-03-31T15:05:35.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.911+0000 7f482fd46000 1 Processor -- start 2024-03-31T15:05:35.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.911+0000 7f482fd46000 1 -- start start 2024-03-31T15:05:35.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.911+0000 7f481e000640 1 --2- 172.21.15.16:0/3284504006 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f4808068380 0x7f480806a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:35.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.911+0000 7f481e000640 1 -- 172.21.15.16:0/3284504006 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5709+0+0 (secure 0 0 0) 0x7f48200867a0 con 0x7f48240cc7b0 2024-03-31T15:05:35.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.911+0000 7f4829400640 1 --2- 172.21.15.16:0/3284504006 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f4808068380 0x7f480806a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:35.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.911+0000 7f482fd46000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fa0e2a5840 0x55fa0e2a5c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:35.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.911+0000 7f482fd46000 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55fa0e2a6160 0x55fa0e258c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:35.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.911+0000 7f482fd46000 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fa0e2591d0 0x55fa0e2595b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:35.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.911+0000 7f482fd46000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55fa0e143180 con 0x55fa0e2a5840 2024-03-31T15:05:35.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.911+0000 7f482fd46000 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55fa0e0aac90 con 0x55fa0e2591d0 2024-03-31T15:05:35.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.911+0000 7f482fd46000 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x55fa0e047a00 con 0x55fa0e2a6160 2024-03-31T15:05:35.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.911+0000 7f4829e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fa0e2a5840 0x55fa0e2a5c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:35.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.911+0000 7f4829e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fa0e2a5840 0x55fa0e2a5c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34344/0 (socket says 172.21.15.16:34344) 2024-03-31T15:05:35.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.911+0000 7f4829e00640 1 -- 172.21.15.16:0/2605796269 learned_addr learned my addr 172.21.15.16:0/2605796269 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:05:35.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.911+0000 7f482a800640 1 --2- 172.21.15.16:0/2605796269 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fa0e2591d0 0x55fa0e2595b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:35.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.911+0000 7f4829400640 1 --2- 172.21.15.16:0/2605796269 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55fa0e2a6160 0x55fa0e258c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:35.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.911+0000 7f4829400640 1 --2- 172.21.15.16:0/3284504006 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f4808068380 0x7f480806a840 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f48240cc1e0 tx=0x7f4814007b80 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:05:35.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.911+0000 7f482a800640 1 -- 172.21.15.16:0/2605796269 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55fa0e2a6160 msgr2=0x55fa0e258c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:05:35.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.911+0000 7f482a800640 1 --2- 172.21.15.16:0/2605796269 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55fa0e2a6160 0x55fa0e258c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:35.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.911+0000 7f482a800640 1 -- 172.21.15.16:0/2605796269 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fa0e2a5840 msgr2=0x55fa0e2a5c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:05:35.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.911+0000 7f482a800640 1 --2- 172.21.15.16:0/2605796269 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fa0e2a5840 0x55fa0e2a5c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:35.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.911+0000 7f482a800640 1 -- 172.21.15.16:0/2605796269 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fa0e030510 con 0x55fa0e2591d0 2024-03-31T15:05:35.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.911+0000 7f4829e00640 1 --2- 172.21.15.16:0/2605796269 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fa0e2a5840 0x55fa0e2a5c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:05:35.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.911+0000 7f482a800640 1 --2- 172.21.15.16:0/2605796269 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fa0e2591d0 0x55fa0e2595b0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f4820056600 tx=0x7f48200560a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a22e03dc6d496523 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:05:35.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.911+0000 7f4812a00640 1 -- 172.21.15.16:0/2605796269 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f482006e020 con 0x55fa0e2591d0 2024-03-31T15:05:35.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.911+0000 7f4812a00640 1 -- 172.21.15.16:0/2605796269 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f482005a970 con 0x55fa0e2591d0 2024-03-31T15:05:35.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.911+0000 7f4812a00640 1 -- 172.21.15.16:0/2605796269 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f482005f1a0 con 0x55fa0e2591d0 2024-03-31T15:05:35.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.911+0000 7f482fd46000 1 -- 172.21.15.16:0/2605796269 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fa0e2591d0 msgr2=0x55fa0e2595b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:35.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.911+0000 7f482fd46000 1 --2- 172.21.15.16:0/2605796269 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fa0e2591d0 0x55fa0e2595b0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f4820056600 tx=0x7f48200560a0 comp rx=0 tx=0).stop 2024-03-31T15:05:35.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.911+0000 7f482fd46000 1 -- 172.21.15.16:0/2605796269 shutdown_connections 2024-03-31T15:05:35.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.911+0000 7f482fd46000 1 --2- 172.21.15.16:0/2605796269 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fa0e2591d0 0x55fa0e2595b0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:35.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.911+0000 7f482fd46000 1 --2- 172.21.15.16:0/2605796269 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55fa0e2a6160 0x55fa0e258c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:35.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.911+0000 7f482fd46000 1 --2- 172.21.15.16:0/2605796269 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fa0e2a5840 0x55fa0e2a5c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:35.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.911+0000 7f482fd46000 1 -- 172.21.15.16:0/2605796269 >> 172.21.15.16:0/2605796269 conn(0x55fa0e2865e0 msgr2=0x55fa0e2889d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:05:35.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.911+0000 7f482fd46000 1 -- 172.21.15.16:0/2605796269 shutdown_connections 2024-03-31T15:05:35.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.911+0000 7f482fd46000 1 -- 172.21.15.16:0/2605796269 wait complete. 2024-03-31T15:05:35.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.915+0000 7f482fd46000 1 -- start start 2024-03-31T15:05:35.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.915+0000 7f482fd46000 1 Processor -- start 2024-03-31T15:05:35.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.915+0000 7f482fd46000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fa0e2591d0 0x55fa0e25e190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:35.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.915+0000 7f482fd46000 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55fa0e2a5840 0x55fa0e25e6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:35.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.915+0000 7f482fd46000 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fa0e2a6160 0x55fa0e25ec10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:35.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.915+0000 7f482fd46000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55fa0e068200 con 0x55fa0e2591d0 2024-03-31T15:05:35.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.915+0000 7f482fd46000 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55fa0e0a6c40 con 0x55fa0e2a6160 2024-03-31T15:05:35.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.915+0000 7f482fd46000 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x55fa0e076b20 con 0x55fa0e2a5840 2024-03-31T15:05:35.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.915+0000 7f482fd46000 10 client.0 _ll_register_callbacks cb 0x55fa0e246930 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T15:05:35.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.915+0000 7f4829e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fa0e2591d0 0x55fa0e25e190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:35.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.915+0000 7f4829e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fa0e2591d0 0x55fa0e25e190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34360/0 (socket says 172.21.15.16:34360) 2024-03-31T15:05:35.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.915+0000 7f4829e00640 1 -- 172.21.15.16:0/779534036 learned_addr learned my addr 172.21.15.16:0/779534036 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:05:35.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.915+0000 7f4829400640 1 --2- 172.21.15.16:0/779534036 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55fa0e2a5840 0x55fa0e25e6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:35.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.915+0000 7f482a800640 1 --2- 172.21.15.16:0/779534036 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fa0e2a6160 0x55fa0e25ec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:35.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.915+0000 7f4829e00640 1 -- 172.21.15.16:0/779534036 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55fa0e2a5840 msgr2=0x55fa0e25e6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:05:35.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.915+0000 7f4829e00640 1 --2- 172.21.15.16:0/779534036 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55fa0e2a5840 0x55fa0e25e6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:35.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.915+0000 7f4829e00640 1 -- 172.21.15.16:0/779534036 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fa0e2a6160 msgr2=0x55fa0e25ec10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:05:35.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.915+0000 7f4829e00640 1 --2- 172.21.15.16:0/779534036 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fa0e2a6160 0x55fa0e25ec10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:35.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.915+0000 7f4829e00640 1 -- 172.21.15.16:0/779534036 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fa0e0b5ce0 con 0x55fa0e2591d0 2024-03-31T15:05:35.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.915+0000 7f4829e00640 1 --2- 172.21.15.16:0/779534036 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fa0e2591d0 0x55fa0e25e190 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f4818097eb0 tx=0x7f481801ac20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=31f33e83e94e54b1 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:05:35.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.915+0000 7f4807400640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-31T15:05:35.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.915+0000 7f4807400640 1 -- 172.21.15.16:0/779534036 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48180bd030 con 0x55fa0e2591d0 2024-03-31T15:05:35.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.915+0000 7f4807400640 1 -- 172.21.15.16:0/779534036 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(45..45 src has 1..45) ==== 5709+0+0 (secure 0 0 0) 0x7f48180acb60 con 0x55fa0e2591d0 2024-03-31T15:05:35.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.915+0000 7f482fd46000 1 -- 172.21.15.16:0/779534036 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f482005a970 con 0x55fa0e2591d0 2024-03-31T15:05:35.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.919+0000 7f4807400640 1 -- 172.21.15.16:0/779534036 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f48180b1040 con 0x55fa0e2591d0 2024-03-31T15:05:35.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.919+0000 7f482fd46000 10 client.5892 fetch_fsmap learned FSMap version 25 2024-03-31T15:05:35.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.919+0000 7f482fd46000 1 -- 172.21.15.16:0/779534036 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=2+}) -- 0x55fa0e0667c0 con 0x55fa0e2591d0 2024-03-31T15:05:35.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.919+0000 7f4807400640 1 -- 172.21.15.16:0/779534036 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f482005a970 con 0x55fa0e2591d0 2024-03-31T15:05:35.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.919+0000 7f4807400640 1 -- 172.21.15.16:0/779534036 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 25) ==== 3494+0+0 (secure 0 0 0) 0x7f48180b8430 con 0x55fa0e2591d0 2024-03-31T15:05:35.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.919+0000 7f482fd46000 10 client.5892 fetch_fsmap finished waiting for FSMap version 25 2024-03-31T15:05:35.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.919+0000 7f482fd46000 10 client.5892 resolve_mds: resolved 3:0 to role '3:0' aka daemon mds.a 2024-03-31T15:05:35.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.919+0000 7f482fd46000 20 client.5892 populate_metadata read hostname 'smithi016' 2024-03-31T15:05:35.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.919+0000 7f482fd46000 1 --2- 172.21.15.16:0/779534036 >> [v2:172.21.15.16:6832/2768973409,v1:172.21.15.16:6834/2768973409] conn(0x55fa0e25f4e0 0x55fa0e2b6db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:35.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.919+0000 7f482fd46000 4 client.5892 mds_command: new command op to 5697 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-31T15:05:35.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.919+0000 7f482fd46000 1 -- 172.21.15.16:0/779534036 --> [v2:172.21.15.16:6832/2768973409,v1:172.21.15.16:6834/2768973409] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fa0e045c20 con 0x55fa0e25f4e0 2024-03-31T15:05:35.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.919+0000 7f4829400640 1 --2- 172.21.15.16:0/779534036 >> [v2:172.21.15.16:6832/2768973409,v1:172.21.15.16:6834/2768973409] conn(0x55fa0e25f4e0 0x55fa0e2b6db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:35.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.919+0000 7f4807400640 10 client.5892 ms_handle_connect on v2:172.21.15.16:6832/2768973409 2024-03-31T15:05:35.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.919+0000 7f4829400640 1 --2- 172.21.15.16:0/779534036 >> [v2:172.21.15.16:6832/2768973409,v1:172.21.15.16:6834/2768973409] conn(0x55fa0e25f4e0 0x55fa0e2b6db0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9a8a637bfceb9fe8 server_cookie=1c284170bd201c33 in_seq=0 out_seq=0 2024-03-31T15:05:35.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.919+0000 7f4807400640 1 -- 172.21.15.16:0/779534036 <== mds.0 v2:172.21.15.16:6832/2768973409 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55fa0e045c20 con 0x55fa0e25f4e0 2024-03-31T15:05:35.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.919+0000 7f4807400640 10 client.5892 handle_command_reply: tid=0 2024-03-31T15:05:35.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.919+0000 7f482fd46000 1 client.5892 shutdown 2024-03-31T15:05:35.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.919+0000 7f482fd46000 1 -- 172.21.15.16:0/779534036 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fa0e2591d0 msgr2=0x55fa0e25e190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:35.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.919+0000 7f482fd46000 1 --2- 172.21.15.16:0/779534036 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fa0e2591d0 0x55fa0e25e190 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f4818097eb0 tx=0x7f481801ac20 comp rx=0 tx=0).stop 2024-03-31T15:05:35.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.919+0000 7f482fd46000 1 -- 172.21.15.16:0/779534036 shutdown_connections 2024-03-31T15:05:35.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.919+0000 7f482fd46000 1 --2- 172.21.15.16:0/779534036 >> [v2:172.21.15.16:6832/2768973409,v1:172.21.15.16:6834/2768973409] conn(0x55fa0e25f4e0 0x55fa0e2b6db0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:35.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.919+0000 7f482fd46000 1 --2- 172.21.15.16:0/779534036 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fa0e2a6160 0x55fa0e25ec10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:35.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.919+0000 7f482fd46000 1 --2- 172.21.15.16:0/779534036 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55fa0e2a5840 0x55fa0e25e6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:35.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.919+0000 7f482fd46000 1 --2- 172.21.15.16:0/779534036 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fa0e2591d0 0x55fa0e25e190 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:35.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.919+0000 7f482fd46000 1 -- 172.21.15.16:0/779534036 >> 172.21.15.16:0/779534036 conn(0x55fa0e2865e0 msgr2=0x55fa0e2889d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:05:35.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.919+0000 7f4807400640 0 client.5892 ms_handle_reset on v2:172.21.15.16:6832/2768973409 2024-03-31T15:05:35.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.919+0000 7f482fd46000 1 -- 172.21.15.16:0/779534036 shutdown_connections 2024-03-31T15:05:35.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.919+0000 7f482fd46000 1 -- 172.21.15.16:0/779534036 wait complete. 2024-03-31T15:05:35.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.923+0000 7f482fd46000 20 client.5892 trim_cache size 0 max 16384 2024-03-31T15:05:35.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.939+0000 7f482fd46000 1 Processor -- start 2024-03-31T15:05:35.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.939+0000 7f482fd46000 1 -- start start 2024-03-31T15:05:35.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.939+0000 7f482fd46000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fa0e2591d0 0x55fa0e258390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:35.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.939+0000 7f482fd46000 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fa0e2a5840 0x55fa0e2588d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:35.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.939+0000 7f482fd46000 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55fa0e2a6160 0x55fa0e2b6db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:35.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.939+0000 7f482fd46000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f4824201950 con 0x55fa0e2591d0 2024-03-31T15:05:35.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.939+0000 7f482fd46000 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f4824200a10 con 0x55fa0e2a5840 2024-03-31T15:05:35.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.939+0000 7f482fd46000 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f48240c9ea0 con 0x55fa0e2a6160 2024-03-31T15:05:35.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.939+0000 7f4829e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fa0e2591d0 0x55fa0e258390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:35.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.939+0000 7f4829e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fa0e2591d0 0x55fa0e258390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34372/0 (socket says 172.21.15.16:34372) 2024-03-31T15:05:35.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.939+0000 7f4829e00640 1 -- 172.21.15.16:0/1408208159 learned_addr learned my addr 172.21.15.16:0/1408208159 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:05:35.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.939+0000 7f4829400640 1 --2- 172.21.15.16:0/1408208159 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fa0e2a5840 0x55fa0e2588d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:35.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.939+0000 7f482a800640 1 --2- 172.21.15.16:0/1408208159 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55fa0e2a6160 0x55fa0e2b6db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:35.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.939+0000 7f4829e00640 1 -- 172.21.15.16:0/1408208159 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55fa0e2a6160 msgr2=0x55fa0e2b6db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:05:35.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.939+0000 7f4829e00640 1 --2- 172.21.15.16:0/1408208159 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55fa0e2a6160 0x55fa0e2b6db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:35.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.939+0000 7f4829e00640 1 -- 172.21.15.16:0/1408208159 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fa0e2a5840 msgr2=0x55fa0e2588d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:05:35.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.939+0000 7f4829e00640 1 --2- 172.21.15.16:0/1408208159 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fa0e2a5840 0x55fa0e2588d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:35.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.939+0000 7f4829e00640 1 -- 172.21.15.16:0/1408208159 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fa0e2a5f90 con 0x55fa0e2591d0 2024-03-31T15:05:35.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.939+0000 7f4829400640 1 --2- 172.21.15.16:0/1408208159 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fa0e2a5840 0x55fa0e2588d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:05:35.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.943+0000 7f4829e00640 1 --2- 172.21.15.16:0/1408208159 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fa0e2591d0 0x55fa0e258390 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f48180097c0 tx=0x7f48180ae070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d96c8404e0f027f5 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:05:35.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.943+0000 7f4812000640 1 -- 172.21.15.16:0/1408208159 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48180bd040 con 0x55fa0e2591d0 2024-03-31T15:05:35.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.943+0000 7f4812000640 1 -- 172.21.15.16:0/1408208159 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4818000e50 con 0x55fa0e2591d0 2024-03-31T15:05:35.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.943+0000 7f4812000640 1 -- 172.21.15.16:0/1408208159 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48180b8980 con 0x55fa0e2591d0 2024-03-31T15:05:35.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.943+0000 7f3e9ae00640 1 Processor -- start 2024-03-31T15:05:35.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.943+0000 7f3e9ae00640 1 -- start start 2024-03-31T15:05:35.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.943+0000 7f482fd46000 1 -- 172.21.15.16:0/1408208159 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fa0e2591d0 msgr2=0x55fa0e258390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:35.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.943+0000 7f482fd46000 1 --2- 172.21.15.16:0/1408208159 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fa0e2591d0 0x55fa0e258390 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f48180097c0 tx=0x7f48180ae070 comp rx=0 tx=0).stop 2024-03-31T15:05:35.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.943+0000 7f482fd46000 1 -- 172.21.15.16:0/1408208159 shutdown_connections 2024-03-31T15:05:35.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.943+0000 7f482fd46000 1 --2- 172.21.15.16:0/1408208159 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55fa0e2a6160 0x55fa0e2b6db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:35.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.943+0000 7f482fd46000 1 --2- 172.21.15.16:0/1408208159 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fa0e2a5840 0x55fa0e2588d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:35.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.943+0000 7f482fd46000 1 --2- 172.21.15.16:0/1408208159 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fa0e2591d0 0x55fa0e258390 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:35.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.943+0000 7f482fd46000 1 -- 172.21.15.16:0/1408208159 >> 172.21.15.16:0/1408208159 conn(0x55fa0e2865e0 msgr2=0x55fa0e257c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:05:35.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.943+0000 7f482fd46000 1 -- 172.21.15.16:0/1408208159 shutdown_connections 2024-03-31T15:05:35.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.943+0000 7f482fd46000 1 -- 172.21.15.16:0/1408208159 wait complete. 2024-03-31T15:05:35.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.943+0000 7f482fd46000 1 -- start start 2024-03-31T15:05:35.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.943+0000 7f3e9ae00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f3e94142930 0x7f3e941430e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:35.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.943+0000 7f3e9ae00640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3e94073980 con 0x7f3e940b2f50 2024-03-31T15:05:35.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.943+0000 7f3e9ae00640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3e940cdf40 con 0x7f3e9406edd0 2024-03-31T15:05:35.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.943+0000 7f3e9ae00640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f3e94142d10 con 0x7f3e94142930 2024-03-31T15:05:35.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.943+0000 7f482fd46000 1 Processor -- start 2024-03-31T15:05:35.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.943+0000 7f3e99000640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f3e940b2f50 0x7f3e9406e6c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:35578/0 (socket says 172.21.15.16:35578) 2024-03-31T15:05:35.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.943+0000 7f3e99000640 1 -- 172.21.15.16:0/1018570723 learned_addr learned my addr 172.21.15.16:0/1018570723 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:05:35.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.943+0000 7f3e99a00640 1 --2- 172.21.15.16:0/1018570723 >> v2:172.21.15.89:3300/0 conn(0x7f3e94142930 0x7f3e941430e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:35.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.943+0000 7f3e93e00640 1 -- 172.21.15.16:0/1018570723 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1664986069 0 0) 0x7f3e94073980 con 0x7f3e940b2f50 2024-03-31T15:05:35.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.943+0000 7f3e99a00640 1 -- 172.21.15.16:0/1018570723 >> v1:172.21.15.89:6789/0 conn(0x7f3e9406edd0 legacy=0x7f3e940cefe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:35.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.943+0000 7f3e99a00640 1 -- 172.21.15.16:0/1018570723 >> v1:172.21.15.16:6789/0 conn(0x7f3e940b2f50 legacy=0x7f3e9406e6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:35.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.943+0000 7f3e99a00640 1 -- 172.21.15.16:0/1018570723 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e940ce220 con 0x7f3e94142930 2024-03-31T15:05:35.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.943+0000 7f482fd46000 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55fa0e2591d0 0x55fa0e2b95e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:35.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.943+0000 7f482fd46000 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fa0e2a5840 0x55fa0e2b9b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:35.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.943+0000 7f482fd46000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fa0e2a6160 0x55fa0e316800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:35.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.943+0000 7f482fd46000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55fa0e0a6c40 con 0x55fa0e2a6160 2024-03-31T15:05:35.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.943+0000 7f482fd46000 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f48240ca4f0 con 0x55fa0e2a5840 2024-03-31T15:05:35.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.943+0000 7f482fd46000 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x55fa0e0a9fa0 con 0x55fa0e2591d0 2024-03-31T15:05:35.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.943+0000 7f482fd46000 10 client.0 _ll_register_callbacks cb 0x7f4818009640 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T15:05:35.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.943+0000 7f482a800640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fa0e2a6160 0x55fa0e316800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:35.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.943+0000 7f4829400640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fa0e2a5840 0x55fa0e2b9b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:35.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.947+0000 7f4829e00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55fa0e2591d0 0x55fa0e2b95e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:35.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.947+0000 7f482a800640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fa0e2a6160 0x55fa0e316800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34376/0 (socket says 172.21.15.16:34376) 2024-03-31T15:05:35.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.947+0000 7f482a800640 1 -- 172.21.15.16:0/306695795 learned_addr learned my addr 172.21.15.16:0/306695795 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:05:35.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.947+0000 7f4829400640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fa0e2a5840 0x55fa0e2b9b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.16:46072/0 (socket says 172.21.15.16:46072) 2024-03-31T15:05:35.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.947+0000 7f4829e00640 1 -- 172.21.15.16:0/306695795 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fa0e2a5840 msgr2=0x55fa0e2b9b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:05:35.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.947+0000 7f4829e00640 1 --2- 172.21.15.16:0/306695795 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fa0e2a5840 0x55fa0e2b9b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:35.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.947+0000 7f4829e00640 1 -- 172.21.15.16:0/306695795 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fa0e2a6160 msgr2=0x55fa0e316800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:05:35.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.947+0000 7f4829e00640 1 --2- 172.21.15.16:0/306695795 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fa0e2a6160 0x55fa0e316800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:35.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.947+0000 7f4829e00640 1 -- 172.21.15.16:0/306695795 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({osdmap=0}) -- 0x55fa0e066c70 con 0x55fa0e2591d0 2024-03-31T15:05:35.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.947+0000 7f482a800640 1 --2- 172.21.15.16:0/306695795 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fa0e2a6160 0x55fa0e316800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:05:35.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.947+0000 7f4829e00640 1 --2- 172.21.15.16:0/306695795 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55fa0e2591d0 0x55fa0e2b95e0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f481801cc80 tx=0x7f48180a9000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a50f0d646a36c167 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:05:35.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.947+0000 7f4807400640 10 client.0 ms_handle_connect on v2:172.21.15.89:3301/0 2024-03-31T15:05:35.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.947+0000 7f4807400640 1 -- 172.21.15.16:0/306695795 <== mon.2 v2:172.21.15.89:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48180bd070 con 0x55fa0e2591d0 2024-03-31T15:05:35.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.947+0000 7f4807400640 1 -- 172.21.15.16:0/306695795 <== mon.2 v2:172.21.15.89:3301/0 2 ==== osd_map(45..45 src has 1..45) ==== 5709+0+0 (secure 0 0 0) 0x7f4818004040 con 0x55fa0e2591d0 2024-03-31T15:05:35.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.947+0000 7f482fd46000 1 -- 172.21.15.16:0/306695795 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4818000e50 con 0x55fa0e2591d0 2024-03-31T15:05:35.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.947+0000 7f4807400640 1 -- 172.21.15.16:0/306695795 <== mon.2 v2:172.21.15.89:3301/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f48180d1070 con 0x55fa0e2591d0 2024-03-31T15:05:35.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.947+0000 7f482fd46000 10 client.5414 fetch_fsmap learned FSMap version 25 2024-03-31T15:05:35.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.947+0000 7f482fd46000 1 -- 172.21.15.16:0/306695795 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({config=0+,fsmap=25,monmap=2+}) -- 0x55fa0e047c50 con 0x55fa0e2591d0 2024-03-31T15:05:35.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.947+0000 7f4807400640 1 -- 172.21.15.16:0/306695795 <== mon.2 v2:172.21.15.89:3301/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4818000e50 con 0x55fa0e2591d0 2024-03-31T15:05:35.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.947+0000 7f4807400640 1 -- 172.21.15.16:0/306695795 <== mon.2 v2:172.21.15.89:3301/0 5 ==== fsmap(e 25) ==== 3494+0+0 (secure 0 0 0) 0x7f4818004240 con 0x55fa0e2591d0 2024-03-31T15:05:35.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.947+0000 7f482fd46000 10 client.5414 fetch_fsmap finished waiting for FSMap version 25 2024-03-31T15:05:35.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.947+0000 7f482fd46000 10 client.5414 resolve_mds: resolved 3:0 to role '3:0' aka daemon mds.a 2024-03-31T15:05:35.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.947+0000 7f482fd46000 20 client.5414 populate_metadata read hostname 'smithi016' 2024-03-31T15:05:35.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.947+0000 7f482fd46000 1 --2- 172.21.15.16:0/306695795 >> [v2:172.21.15.16:6832/2768973409,v1:172.21.15.16:6834/2768973409] conn(0x55fa0e316d40 0x55fa0e317150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:35.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.947+0000 7f482fd46000 4 client.5414 mds_command: new command op to 5697 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["repair", "recursive"]}] 2024-03-31T15:05:35.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.947+0000 7f482fd46000 1 -- 172.21.15.16:0/306695795 --> [v2:172.21.15.16:6832/2768973409,v1:172.21.15.16:6834/2768973409] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["repair", "recursive"]}) -- 0x55fa0e0b81d0 con 0x55fa0e316d40 2024-03-31T15:05:35.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.947+0000 7f4829400640 1 --2- 172.21.15.16:0/306695795 >> [v2:172.21.15.16:6832/2768973409,v1:172.21.15.16:6834/2768973409] conn(0x55fa0e316d40 0x55fa0e317150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:35.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.947+0000 7f4829400640 1 --2- 172.21.15.16:0/306695795 >> [v2:172.21.15.16:6832/2768973409,v1:172.21.15.16:6834/2768973409] conn(0x55fa0e316d40 0x55fa0e317150 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bdabdafc44bf523c server_cookie=d780d3d737a7f056 in_seq=0 out_seq=0 2024-03-31T15:05:35.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.947+0000 7f4807400640 10 client.5414 ms_handle_connect on v2:172.21.15.16:6832/2768973409 2024-03-31T15:05:35.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.947+0000 7f3e99a00640 1 --2- 172.21.15.16:0/1018570723 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3e94142930 0x7f3e941430e0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f3e880097c0 tx=0x7f3e8801c4a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=29c1072b5877c18e server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:05:35.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.947+0000 7f3e93e00640 1 -- 172.21.15.16:0/1018570723 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e880b1070 con 0x7f3e94142930 2024-03-31T15:05:35.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.947+0000 7f3e93e00640 1 -- 172.21.15.16:0/1018570723 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3e880ac410 con 0x7f3e94142930 2024-03-31T15:05:35.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.947+0000 7f3e93e00640 1 -- 172.21.15.16:0/1018570723 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e880ac7e0 con 0x7f3e94142930 2024-03-31T15:05:35.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.947+0000 7f4807400640 1 -- 172.21.15.16:0/306695795 <== mds.0 v2:172.21.15.16:6832/2768973409 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55fa0e0b81d0 con 0x55fa0e316d40 2024-03-31T15:05:35.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.947+0000 7f4807400640 10 client.5414 handle_command_reply: tid=0 2024-03-31T15:05:35.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.947+0000 7f482fd46000 1 client.5414 shutdown 2024-03-31T15:05:35.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.951+0000 7f482fd46000 1 -- 172.21.15.16:0/306695795 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55fa0e2591d0 msgr2=0x55fa0e2b95e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:35.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.951+0000 7f482fd46000 1 --2- 172.21.15.16:0/306695795 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55fa0e2591d0 0x55fa0e2b95e0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f481801cc80 tx=0x7f48180a9000 comp rx=0 tx=0).stop 2024-03-31T15:05:35.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.951+0000 7f3e9ae00640 1 -- 172.21.15.16:0/1018570723 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3e94142930 msgr2=0x7f3e941430e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:35.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.951+0000 7f3e9ae00640 1 --2- 172.21.15.16:0/1018570723 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3e94142930 0x7f3e941430e0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f3e880097c0 tx=0x7f3e8801c4a0 comp rx=0 tx=0).stop 2024-03-31T15:05:35.950 INFO:tasks.ceph.mds.a.smithi016.stderr:2024-03-31T15:05:35.951+0000 7fa13a800640 -1 mds.0.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000001 [2,head] /parent_a/alpha auth v2 ap=1 DIRTYPARENT s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={5859=0-4194304@1} caps={5859=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@2},l=5859 | request=0 caps=1 dirtyparent=1 dirty=0 authpin=1 scrubqueue=0 0x5639fddc0c00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(-1)0x0:[]//[]","memoryvalue":"(7)0x10000000001:[<0x10000000000/alpha v2>,<0x1/parent_a v4>]//[]","error_str":"failed to decode on-disk backtrace (40 bytes)!"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-31T15:05:35.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.951+0000 7f3e9ae00640 1 -- 172.21.15.16:0/1018570723 shutdown_connections 2024-03-31T15:05:35.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.951+0000 7f3e9ae00640 1 --2- 172.21.15.16:0/1018570723 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3e94142930 0x7f3e941430e0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:35.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.951+0000 7f3e9ae00640 1 -- 172.21.15.16:0/1018570723 >> 172.21.15.16:0/1018570723 conn(0x7f3e940cc9a0 msgr2=0x7f3e940ccdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:05:35.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.951+0000 7f482fd46000 1 -- 172.21.15.16:0/306695795 shutdown_connections 2024-03-31T15:05:35.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.951+0000 7f482fd46000 1 --2- 172.21.15.16:0/306695795 >> [v2:172.21.15.16:6832/2768973409,v1:172.21.15.16:6834/2768973409] conn(0x55fa0e316d40 0x55fa0e317150 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:35.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.951+0000 7f482fd46000 1 --2- 172.21.15.16:0/306695795 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fa0e2a6160 0x55fa0e316800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:35.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.951+0000 7f482fd46000 1 --2- 172.21.15.16:0/306695795 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fa0e2a5840 0x55fa0e2b9b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:35.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.951+0000 7f482fd46000 1 --2- 172.21.15.16:0/306695795 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55fa0e2591d0 0x55fa0e2b95e0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:35.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.951+0000 7f482fd46000 1 -- 172.21.15.16:0/306695795 >> 172.21.15.16:0/306695795 conn(0x55fa0e2865e0 msgr2=0x55fa0e258390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:05:35.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.951+0000 7f4807400640 0 client.5414 ms_handle_reset on v2:172.21.15.16:6832/2768973409 2024-03-31T15:05:35.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.951+0000 7f3e9ae00640 1 -- 172.21.15.16:0/1018570723 shutdown_connections 2024-03-31T15:05:35.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.951+0000 7f482fd46000 1 -- 172.21.15.16:0/306695795 shutdown_connections 2024-03-31T15:05:35.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.951+0000 7f3e9ae00640 1 -- 172.21.15.16:0/1018570723 wait complete. 2024-03-31T15:05:35.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.951+0000 7f482fd46000 1 -- 172.21.15.16:0/306695795 wait complete. 2024-03-31T15:05:35.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.951+0000 7f482fd46000 20 client.5414 trim_cache size 0 max 16384 2024-03-31T15:05:35.953 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-03-31T15:05:35.954 INFO:teuthology.orchestra.run.smithi016.stdout: "return_code": 0, 2024-03-31T15:05:35.954 INFO:teuthology.orchestra.run.smithi016.stdout: "scrub_tag": "453e7a77-7af9-4065-bfad-edf570d63b51", 2024-03-31T15:05:35.954 INFO:teuthology.orchestra.run.smithi016.stdout: "mode": "asynchronous" 2024-03-31T15:05:35.954 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-03-31T15:05:35.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.951+0000 7f482c800640 1 -- 172.21.15.16:0/3284504006 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f4808068380 msgr2=0x7f480806a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:35.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.951+0000 7f482c800640 1 --2- 172.21.15.16:0/3284504006 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f4808068380 0x7f480806a840 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f48240cc1e0 tx=0x7f4814007b80 comp rx=0 tx=0).stop 2024-03-31T15:05:35.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.951+0000 7f482c800640 1 -- 172.21.15.16:0/3284504006 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f48240cc7b0 msgr2=0x7f48240b56d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:35.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.951+0000 7f482c800640 1 --2- 172.21.15.16:0/3284504006 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f48240cc7b0 0x7f48240b56d0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f482000c730 tx=0x7f482000cbf0 comp rx=0 tx=0).stop 2024-03-31T15:05:35.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.951+0000 7f482c800640 1 -- 172.21.15.16:0/3284504006 shutdown_connections 2024-03-31T15:05:35.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.951+0000 7f482c800640 1 --2- 172.21.15.16:0/3284504006 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f4808068380 0x7f480806a840 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:35.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.951+0000 7f482c800640 1 --2- 172.21.15.16:0/3284504006 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f48240cc7b0 0x7f48240b56d0 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:35.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.951+0000 7f482c800640 1 --2- 172.21.15.16:0/3284504006 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f48240b7900 0x7f4824144a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:35.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.951+0000 7f482c800640 1 --2- 172.21.15.16:0/3284504006 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f48240b44d0 0x7f48241574c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:35.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.951+0000 7f482c800640 1 -- 172.21.15.16:0/3284504006 >> 172.21.15.16:0/3284504006 conn(0x7f482406b7f0 msgr2=0x7f48240d2720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:05:35.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.951+0000 7f482c800640 1 -- 172.21.15.16:0/3284504006 shutdown_connections 2024-03-31T15:05:35.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.951+0000 7f482c800640 1 -- 172.21.15.16:0/3284504006 wait complete. 2024-03-31T15:05:35.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.951+0000 7f3e9ae00640 1 Processor -- start 2024-03-31T15:05:35.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.951+0000 7f3e9ae00640 1 -- start start 2024-03-31T15:05:35.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.951+0000 7f3e9ae00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3e9406edd0 0x7f3e94140a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:35.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.951+0000 7f3e9ae00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3e940b2f50 0x7f3e94141240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:35.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.951+0000 7f3e9ae00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f3e94142930 0x7f3e940ad400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:35.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.951+0000 7f3e9ae00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f3e940cfb00 con 0x7f3e940b2f50 2024-03-31T15:05:35.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.951+0000 7f3e9ae00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f3e940ceac0 con 0x7f3e9406edd0 2024-03-31T15:05:35.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.951+0000 7f3e9ae00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f3e940cfc80 con 0x7f3e94142930 2024-03-31T15:05:35.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.951+0000 7f3e99000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3e940b2f50 0x7f3e94141240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:35.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.951+0000 7f3e99000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3e940b2f50 0x7f3e94141240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34388/0 (socket says 172.21.15.16:34388) 2024-03-31T15:05:35.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.951+0000 7f3e99000640 1 -- 172.21.15.16:0/1760146965 learned_addr learned my addr 172.21.15.16:0/1760146965 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:05:35.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.951+0000 7f3e99a00640 1 --2- 172.21.15.16:0/1760146965 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3e9406edd0 0x7f3e94140a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:35.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.951+0000 7f3e9a400640 1 --2- 172.21.15.16:0/1760146965 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f3e94142930 0x7f3e940ad400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:35.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.955+0000 7f3e99000640 1 -- 172.21.15.16:0/1760146965 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f3e94142930 msgr2=0x7f3e940ad400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:35.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.955+0000 7f3e99000640 1 --2- 172.21.15.16:0/1760146965 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f3e94142930 0x7f3e940ad400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:35.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.955+0000 7f3e99000640 1 -- 172.21.15.16:0/1760146965 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3e9406edd0 msgr2=0x7f3e94140a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:35.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.955+0000 7f3e99000640 1 --2- 172.21.15.16:0/1760146965 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3e9406edd0 0x7f3e94140a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:35.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.955+0000 7f3e99000640 1 -- 172.21.15.16:0/1760146965 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e9415ca20 con 0x7f3e940b2f50 2024-03-31T15:05:35.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.955+0000 7f3e99a00640 1 --2- 172.21.15.16:0/1760146965 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3e9406edd0 0x7f3e94140a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:05:35.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.955+0000 7f3e99000640 1 --2- 172.21.15.16:0/1760146965 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3e940b2f50 0x7f3e94141240 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f3e8000c4a0 tx=0x7f3e8000c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:05:35.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.955+0000 7f3e91600640 1 -- 172.21.15.16:0/1760146965 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e80018070 con 0x7f3e940b2f50 2024-03-31T15:05:35.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.955+0000 7f3e91600640 1 -- 172.21.15.16:0/1760146965 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3e800074e0 con 0x7f3e940b2f50 2024-03-31T15:05:35.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.955+0000 7f3e91600640 1 -- 172.21.15.16:0/1760146965 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e80004a30 con 0x7f3e940b2f50 2024-03-31T15:05:35.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.955+0000 7f3e9ae00640 1 -- 172.21.15.16:0/1760146965 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3e9415cd00 con 0x7f3e940b2f50 2024-03-31T15:05:35.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.955+0000 7f3e9ae00640 1 -- 172.21.15.16:0/1760146965 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3e9415d2c0 con 0x7f3e940b2f50 2024-03-31T15:05:35.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.955+0000 7f3e9ae00640 1 -- 172.21.15.16:0/1760146965 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "status"} v 0) -- 0x7f3e9404f8c0 con 0x7f3e940b2f50 2024-03-31T15:05:35.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.955+0000 7f3e91600640 1 -- 172.21.15.16:0/1760146965 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f3e8001b070 con 0x7f3e940b2f50 2024-03-31T15:05:35.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.959+0000 7f3e91600640 1 --2- 172.21.15.16:0/1760146965 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f3e74068380 0x7f3e7406a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:35.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.959+0000 7f3e91600640 1 -- 172.21.15.16:0/1760146965 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5709+0+0 (secure 0 0 0) 0x7f3e800862d0 con 0x7f3e940b2f50 2024-03-31T15:05:35.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.959+0000 7f3e91600640 1 -- 172.21.15.16:0/1760146965 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "status"}]=0 v0) ==== 54+0+501 (secure 0 0 0) 0x7f3e80086780 con 0x7f3e940b2f50 2024-03-31T15:05:35.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.959+0000 7f3e99a00640 1 --2- 172.21.15.16:0/1760146965 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f3e74068380 0x7f3e7406a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:35.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.959+0000 7f3e9ae00640 1 -- 172.21.15.16:0/1760146965 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({log-info=0+}) -- 0x7f3e94167b00 con 0x7f3e940b2f50 2024-03-31T15:05:35.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:35.959+0000 7f3e99a00640 1 --2- 172.21.15.16:0/1760146965 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f3e74068380 0x7f3e7406a840 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f3e880ae000 tx=0x7f3e88008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:05:35.967 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:05:35.968 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.3:0 scrub status 2024-03-31T15:05:36.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:36.115+0000 7fd971000640 1 Processor -- start 2024-03-31T15:05:36.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:36.115+0000 7fd971000640 1 -- start start 2024-03-31T15:05:36.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:36.115+0000 7fd971000640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fd96c0d0070 0x7fd96c0d0450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:36.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:36.115+0000 7fd971000640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fd96c15f290 0x7fd96c15cad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:36.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:36.115+0000 7fd971000640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd96c0d0990 con 0x7fd96c15c670 2024-03-31T15:05:36.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:36.115+0000 7fd971000640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7fd96c15d830 con 0x7fd96c0d0070 2024-03-31T15:05:36.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:36.115+0000 7fd971000640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7fd96c15ddf0 con 0x7fd96c15f290 2024-03-31T15:05:36.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:36.115+0000 7fd96b400640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fd96c0d0070 0x7fd96c0d0450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:36.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:36.115+0000 7fd96b400640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fd96c0d0070 0x7fd96c0d0450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34404/0 (socket says 172.21.15.16:34404) 2024-03-31T15:05:36.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:36.115+0000 7fd96b400640 1 -- 172.21.15.16:0/3966397195 learned_addr learned my addr 172.21.15.16:0/3966397195 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:05:36.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:36.119+0000 7fd96be00640 1 --2- 172.21.15.16:0/3966397195 >> v2:172.21.15.89:3300/0 conn(0x7fd96c15f290 0x7fd96c15cad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:36.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:36.119+0000 7fd96b400640 1 -- 172.21.15.16:0/3966397195 >> v2:172.21.15.89:3300/0 conn(0x7fd96c15f290 msgr2=0x7fd96c15cad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:05:36.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:36.119+0000 7fd96b400640 1 --2- 172.21.15.16:0/3966397195 >> v2:172.21.15.89:3300/0 conn(0x7fd96c15f290 0x7fd96c15cad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:36.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:36.119+0000 7fd96b400640 1 -- 172.21.15.16:0/3966397195 >> v1:172.21.15.89:6789/0 conn(0x7fd96c15c670 legacy=0x7fd96c072b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:36.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:36.119+0000 7fd96b400640 1 -- 172.21.15.16:0/3966397195 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd96c15d160 con 0x7fd96c0d0070 2024-03-31T15:05:36.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:36.119+0000 7fd96be00640 1 --2- 172.21.15.16:0/3966397195 >> v2:172.21.15.89:3300/0 conn(0x7fd96c15f290 0x7fd96c15cad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:05:36.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:36.119+0000 7fd96b400640 1 --2- 172.21.15.16:0/3966397195 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd96c0d0070 0x7fd96c0d0450 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7fd96c072a80 tx=0x7fd95c01c4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=200184ea077f94 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:05:36.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:36.119+0000 7fd96a000640 1 -- 172.21.15.16:0/3966397195 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd95c0ab610 con 0x7fd96c0d0070 2024-03-31T15:05:36.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:36.119+0000 7fd96a000640 1 -- 172.21.15.16:0/3966397195 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd95c0b0070 con 0x7fd96c0d0070 2024-03-31T15:05:36.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:36.119+0000 7fd96a000640 1 -- 172.21.15.16:0/3966397195 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd95c0ab9d0 con 0x7fd96c0d0070 2024-03-31T15:05:36.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:36.123+0000 7fd971000640 1 -- 172.21.15.16:0/3966397195 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd96c0d0070 msgr2=0x7fd96c0d0450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:36.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:36.123+0000 7fd971000640 1 --2- 172.21.15.16:0/3966397195 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd96c0d0070 0x7fd96c0d0450 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7fd96c072a80 tx=0x7fd95c01c4f0 comp rx=0 tx=0).stop 2024-03-31T15:05:36.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:36.123+0000 7fd971000640 1 -- 172.21.15.16:0/3966397195 shutdown_connections 2024-03-31T15:05:36.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:36.123+0000 7fd971000640 1 --2- 172.21.15.16:0/3966397195 >> v2:172.21.15.89:3300/0 conn(0x7fd96c15f290 0x7fd96c15cad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:36.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:36.123+0000 7fd971000640 1 --2- 172.21.15.16:0/3966397195 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd96c0d0070 0x7fd96c0d0450 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:36.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:36.123+0000 7fd971000640 1 -- 172.21.15.16:0/3966397195 >> 172.21.15.16:0/3966397195 conn(0x7fd96c0b5520 msgr2=0x7fd96c0b5930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:05:36.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:36.123+0000 7fd971000640 1 -- 172.21.15.16:0/3966397195 shutdown_connections 2024-03-31T15:05:36.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:36.123+0000 7fd971000640 1 -- 172.21.15.16:0/3966397195 wait complete. 2024-03-31T15:05:36.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:36.123+0000 7fd971000640 1 Processor -- start 2024-03-31T15:05:36.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:36.123+0000 7fd971000640 1 -- start start 2024-03-31T15:05:36.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:36.123+0000 7fd971000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd96c0d0070 0x7fd96c099a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:36.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:36.123+0000 7fd971000640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fd96c15c670 0x7fd96c14d430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:36.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:36.123+0000 7fd971000640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd96c15f290 0x7fd96c1462a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:36.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:36.123+0000 7fd971000640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fd96c168cb0 con 0x7fd96c0d0070 2024-03-31T15:05:36.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:36.123+0000 7fd971000640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fd96c0ad200 con 0x7fd96c15f290 2024-03-31T15:05:36.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:36.123+0000 7fd971000640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fd96c168e30 con 0x7fd96c15c670 2024-03-31T15:05:36.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:36.123+0000 7fd96b400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd96c0d0070 0x7fd96c099a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:36.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:36.123+0000 7fd96b400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd96c0d0070 0x7fd96c099a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34412/0 (socket says 172.21.15.16:34412) 2024-03-31T15:05:36.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:36.123+0000 7fd96b400640 1 -- 172.21.15.16:0/4182909677 learned_addr learned my addr 172.21.15.16:0/4182909677 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:05:36.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:36.123+0000 7fd96be00640 1 --2- 172.21.15.16:0/4182909677 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd96c15f290 0x7fd96c1462a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:36.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:36.123+0000 7fd96b400640 1 -- 172.21.15.16:0/4182909677 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fd96c15c670 msgr2=0x7fd96c14d430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:36.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:36.123+0000 7fd96aa00640 1 --2- 172.21.15.16:0/4182909677 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fd96c15c670 0x7fd96c14d430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:36.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:36.123+0000 7fd96b400640 1 --2- 172.21.15.16:0/4182909677 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fd96c15c670 0x7fd96c14d430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:36.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:36.123+0000 7fd96b400640 1 -- 172.21.15.16:0/4182909677 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd96c15f290 msgr2=0x7fd96c1462a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:36.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:36.123+0000 7fd96b400640 1 --2- 172.21.15.16:0/4182909677 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd96c15f290 0x7fd96c1462a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:36.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:36.123+0000 7fd96b400640 1 -- 172.21.15.16:0/4182909677 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd96c157100 con 0x7fd96c0d0070 2024-03-31T15:05:36.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:36.123+0000 7fd96be00640 1 --2- 172.21.15.16:0/4182909677 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd96c15f290 0x7fd96c1462a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:05:36.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:36.123+0000 7fd96b400640 1 --2- 172.21.15.16:0/4182909677 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd96c0d0070 0x7fd96c099a10 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7fd95c0ad040 tx=0x7fd95c01ad30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:05:36.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:36.127+0000 7fd963400640 1 -- 172.21.15.16:0/4182909677 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd95c0be5f0 con 0x7fd96c0d0070 2024-03-31T15:05:36.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:36.127+0000 7fd963400640 1 -- 172.21.15.16:0/4182909677 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd95c0be790 con 0x7fd96c0d0070 2024-03-31T15:05:36.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:36.127+0000 7fd963400640 1 -- 172.21.15.16:0/4182909677 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd95c0beb60 con 0x7fd96c0d0070 2024-03-31T15:05:36.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:36.127+0000 7fd971000640 1 -- 172.21.15.16:0/4182909677 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd96c14c950 con 0x7fd96c0d0070 2024-03-31T15:05:36.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:36.127+0000 7fd971000640 1 -- 172.21.15.16:0/4182909677 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd96c14cf10 con 0x7fd96c0d0070 2024-03-31T15:05:36.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:36.127+0000 7fd963400640 1 -- 172.21.15.16:0/4182909677 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fd95c0b43f0 con 0x7fd96c0d0070 2024-03-31T15:05:36.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:36.127+0000 7fd963400640 1 --2- 172.21.15.16:0/4182909677 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fd94c068380 0x7fd94c06a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:36.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:36.127+0000 7fd975a8d000 1 Processor -- start 2024-03-31T15:05:36.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:36.127+0000 7fd963400640 1 -- 172.21.15.16:0/4182909677 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5709+0+0 (secure 0 0 0) 0x7fd95c121230 con 0x7fd96c0d0070 2024-03-31T15:05:36.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:36.127+0000 7fd975a8d000 1 -- start start 2024-03-31T15:05:36.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:36.127+0000 7fd96aa00640 1 --2- 172.21.15.16:0/4182909677 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fd94c068380 0x7fd94c06a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:36.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:36.131+0000 7fd975a8d000 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56478bddb2d0 0x56478bddb6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:36.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:36.131+0000 7fd975a8d000 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x56478bddbbf0 0x56478bd8e470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:36.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:36.131+0000 7fd975a8d000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56478bd8e9b0 0x56478bd8ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:36.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:36.131+0000 7fd975a8d000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x56478bc77af0 con 0x56478bd8e9b0 2024-03-31T15:05:36.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:36.131+0000 7fd975a8d000 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x56478bbe0b00 con 0x56478bddb2d0 2024-03-31T15:05:36.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:36.131+0000 7fd975a8d000 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x56478bb7d940 con 0x56478bddbbf0 2024-03-31T15:05:36.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:36.131+0000 7fd96aa00640 1 --2- 172.21.15.16:0/4182909677 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fd94c068380 0x7fd94c06a840 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fd958007fd0 tx=0x7fd958009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:05:36.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:36.131+0000 7fd96be00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56478bd8e9b0 0x56478bd8ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:36.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:36.131+0000 7fd96be00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56478bd8e9b0 0x56478bd8ed90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34426/0 (socket says 172.21.15.16:34426) 2024-03-31T15:05:36.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:36.131+0000 7fd96aa00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x56478bddbbf0 0x56478bd8e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:36.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:36.131+0000 7fd96be00640 1 -- 172.21.15.16:0/2481917332 learned_addr learned my addr 172.21.15.16:0/2481917332 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:05:36.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:36.131+0000 7fd96b400640 1 --2- 172.21.15.16:0/2481917332 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56478bddb2d0 0x56478bddb6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:36.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:36.131+0000 7fd96be00640 1 -- 172.21.15.16:0/2481917332 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x56478bddbbf0 msgr2=0x56478bd8e470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:05:36.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:36.131+0000 7fd96be00640 1 --2- 172.21.15.16:0/2481917332 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x56478bddbbf0 0x56478bd8e470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:36.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:36.131+0000 7fd96be00640 1 -- 172.21.15.16:0/2481917332 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56478bddb2d0 msgr2=0x56478bddb6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:05:36.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:36.131+0000 7fd96be00640 1 --2- 172.21.15.16:0/2481917332 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56478bddb2d0 0x56478bddb6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:36.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:36.131+0000 7fd96be00640 1 -- 172.21.15.16:0/2481917332 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56478bb66380 con 0x56478bd8e9b0 2024-03-31T15:05:36.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:36.131+0000 7fd96be00640 1 --2- 172.21.15.16:0/2481917332 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56478bd8e9b0 0x56478bd8ed90 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7fd964007330 tx=0x7fd9640077f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7acc2eb6faeeec1a server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:05:36.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:36.131+0000 7fd957e00640 1 -- 172.21.15.16:0/2481917332 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9640137d0 con 0x56478bd8e9b0 2024-03-31T15:05:36.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:36.131+0000 7fd957e00640 1 -- 172.21.15.16:0/2481917332 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd964019070 con 0x56478bd8e9b0 2024-03-31T15:05:36.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:36.131+0000 7fd957e00640 1 -- 172.21.15.16:0/2481917332 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd964013b70 con 0x56478bd8e9b0 2024-03-31T15:05:36.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:36.131+0000 7fd975a8d000 1 -- 172.21.15.16:0/2481917332 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56478bd8e9b0 msgr2=0x56478bd8ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:36.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:36.131+0000 7fd975a8d000 1 --2- 172.21.15.16:0/2481917332 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56478bd8e9b0 0x56478bd8ed90 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7fd964007330 tx=0x7fd9640077f0 comp rx=0 tx=0).stop 2024-03-31T15:05:36.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:36.131+0000 7fd975a8d000 1 -- 172.21.15.16:0/2481917332 shutdown_connections 2024-03-31T15:05:36.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:36.131+0000 7fd975a8d000 1 --2- 172.21.15.16:0/2481917332 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56478bd8e9b0 0x56478bd8ed90 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:36.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:36.131+0000 7fd975a8d000 1 --2- 172.21.15.16:0/2481917332 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x56478bddbbf0 0x56478bd8e470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:36.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:36.131+0000 7fd975a8d000 1 --2- 172.21.15.16:0/2481917332 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56478bddb2d0 0x56478bddb6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:36.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:36.131+0000 7fd975a8d000 1 -- 172.21.15.16:0/2481917332 >> 172.21.15.16:0/2481917332 conn(0x56478bdbc070 msgr2=0x56478bdbe460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:05:36.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:36.131+0000 7fd975a8d000 1 -- 172.21.15.16:0/2481917332 shutdown_connections 2024-03-31T15:05:36.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:36.131+0000 7fd975a8d000 1 -- 172.21.15.16:0/2481917332 wait complete. 2024-03-31T15:05:36.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:36.131+0000 7fd975a8d000 1 -- start start 2024-03-31T15:05:36.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:36.135+0000 7fd975a8d000 1 Processor -- start 2024-03-31T15:05:36.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:36.135+0000 7fd975a8d000 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x56478bd8e9b0 0x56478bde2630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:36.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:36.135+0000 7fd975a8d000 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56478bddb2d0 0x56478bde2b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:36.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:36.135+0000 7fd975a8d000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56478bddbbf0 0x56478bd93570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:36.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:36.135+0000 7fd975a8d000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x56478bb9e1e0 con 0x56478bddbbf0 2024-03-31T15:05:36.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:36.135+0000 7fd975a8d000 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x56478bbdcab0 con 0x56478bddb2d0 2024-03-31T15:05:36.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:36.135+0000 7fd975a8d000 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x56478bb97750 con 0x56478bd8e9b0 2024-03-31T15:05:36.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:36.135+0000 7fd975a8d000 10 client.0 _ll_register_callbacks cb 0x56478bd7c370 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T15:05:36.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:36.135+0000 7fd96be00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56478bddbbf0 0x56478bd93570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:36.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:36.135+0000 7fd96be00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56478bddbbf0 0x56478bd93570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34428/0 (socket says 172.21.15.16:34428) 2024-03-31T15:05:36.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:36.135+0000 7fd96be00640 1 -- 172.21.15.16:0/1709582975 learned_addr learned my addr 172.21.15.16:0/1709582975 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:05:36.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:36.135+0000 7fd96aa00640 1 --2- 172.21.15.16:0/1709582975 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56478bddb2d0 0x56478bde2b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:36.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:36.135+0000 7fd96b400640 1 --2- 172.21.15.16:0/1709582975 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x56478bd8e9b0 0x56478bde2630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:36.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:36.135+0000 7fd96be00640 1 -- 172.21.15.16:0/1709582975 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x56478bd8e9b0 msgr2=0x56478bde2630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:05:36.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:36.135+0000 7fd96be00640 1 --2- 172.21.15.16:0/1709582975 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x56478bd8e9b0 0x56478bde2630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:36.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:36.135+0000 7fd96be00640 1 -- 172.21.15.16:0/1709582975 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56478bddb2d0 msgr2=0x56478bde2b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:05:36.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:36.135+0000 7fd96be00640 1 --2- 172.21.15.16:0/1709582975 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56478bddb2d0 0x56478bde2b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:36.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:36.135+0000 7fd96be00640 1 -- 172.21.15.16:0/1709582975 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56478bb9cb70 con 0x56478bddbbf0 2024-03-31T15:05:36.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:36.135+0000 7fd96be00640 1 --2- 172.21.15.16:0/1709582975 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56478bddbbf0 0x56478bd93570 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x56478bbe6ae0 tx=0x7fd964004ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c8ab59a1b069b08 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:05:36.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:36.135+0000 7fd954c00640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-31T15:05:36.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:36.135+0000 7fd954c00640 1 -- 172.21.15.16:0/1709582975 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd96402b060 con 0x56478bddbbf0 2024-03-31T15:05:36.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:36.135+0000 7fd954c00640 1 -- 172.21.15.16:0/1709582975 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(45..45 src has 1..45) ==== 5709+0+0 (secure 0 0 0) 0x7fd964014460 con 0x56478bddbbf0 2024-03-31T15:05:36.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:36.135+0000 7fd975a8d000 1 -- 172.21.15.16:0/1709582975 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd964019070 con 0x56478bddbbf0 2024-03-31T15:05:36.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:36.135+0000 7fd954c00640 1 -- 172.21.15.16:0/1709582975 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fd964031050 con 0x56478bddbbf0 2024-03-31T15:05:36.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:36.135+0000 7fd975a8d000 10 client.5916 fetch_fsmap learned FSMap version 25 2024-03-31T15:05:36.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:36.135+0000 7fd975a8d000 1 -- 172.21.15.16:0/1709582975 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=2+}) -- 0x56478bb7db90 con 0x56478bddbbf0 2024-03-31T15:05:36.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:36.135+0000 7fd954c00640 1 -- 172.21.15.16:0/1709582975 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd964019070 con 0x56478bddbbf0 2024-03-31T15:05:36.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:36.135+0000 7fd954c00640 1 -- 172.21.15.16:0/1709582975 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 25) ==== 3494+0+0 (secure 0 0 0) 0x7fd9640039a0 con 0x56478bddbbf0 2024-03-31T15:05:36.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:36.139+0000 7fd975a8d000 10 client.5916 fetch_fsmap finished waiting for FSMap version 25 2024-03-31T15:05:36.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:36.139+0000 7fd975a8d000 10 client.5916 resolve_mds: resolved 3:0 to role '3:0' aka daemon mds.a 2024-03-31T15:05:36.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:36.139+0000 7fd975a8d000 20 client.5916 populate_metadata read hostname 'smithi016' 2024-03-31T15:05:36.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:36.139+0000 7fd975a8d000 1 --2- 172.21.15.16:0/1709582975 >> [v2:172.21.15.16:6832/2768973409,v1:172.21.15.16:6834/2768973409] conn(0x56478bd94310 0x56478bd94720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:36.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:36.139+0000 7fd975a8d000 4 client.5916 mds_command: new command op to 5697 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-31T15:05:36.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:36.139+0000 7fd975a8d000 1 -- 172.21.15.16:0/1709582975 --> [v2:172.21.15.16:6832/2768973409,v1:172.21.15.16:6834/2768973409] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56478bb7bdd0 con 0x56478bd94310 2024-03-31T15:05:36.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:36.139+0000 7fd96b400640 1 --2- 172.21.15.16:0/1709582975 >> [v2:172.21.15.16:6832/2768973409,v1:172.21.15.16:6834/2768973409] conn(0x56478bd94310 0x56478bd94720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:36.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:36.139+0000 7fd96b400640 1 --2- 172.21.15.16:0/1709582975 >> [v2:172.21.15.16:6832/2768973409,v1:172.21.15.16:6834/2768973409] conn(0x56478bd94310 0x56478bd94720 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e16f1de1066c7f6a server_cookie=400117fe0f9931ac in_seq=0 out_seq=0 2024-03-31T15:05:36.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:36.139+0000 7fd954c00640 10 client.5916 ms_handle_connect on v2:172.21.15.16:6832/2768973409 2024-03-31T15:05:36.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:36.139+0000 7fd954c00640 1 -- 172.21.15.16:0/1709582975 <== mds.0 v2:172.21.15.16:6832/2768973409 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56478bb7bdd0 con 0x56478bd94310 2024-03-31T15:05:36.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:36.139+0000 7fd954c00640 10 client.5916 handle_command_reply: tid=0 2024-03-31T15:05:36.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:36.139+0000 7fd975a8d000 1 client.5916 shutdown 2024-03-31T15:05:36.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:36.139+0000 7fd975a8d000 1 -- 172.21.15.16:0/1709582975 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56478bddbbf0 msgr2=0x56478bd93570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:36.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:36.139+0000 7fd975a8d000 1 --2- 172.21.15.16:0/1709582975 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56478bddbbf0 0x56478bd93570 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x56478bbe6ae0 tx=0x7fd964004ac0 comp rx=0 tx=0).stop 2024-03-31T15:05:36.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:36.139+0000 7fd975a8d000 1 -- 172.21.15.16:0/1709582975 shutdown_connections 2024-03-31T15:05:36.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:36.139+0000 7fd975a8d000 1 --2- 172.21.15.16:0/1709582975 >> [v2:172.21.15.16:6832/2768973409,v1:172.21.15.16:6834/2768973409] conn(0x56478bd94310 0x56478bd94720 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:36.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:36.139+0000 7fd975a8d000 1 --2- 172.21.15.16:0/1709582975 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56478bddbbf0 0x56478bd93570 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:36.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:36.139+0000 7fd975a8d000 1 --2- 172.21.15.16:0/1709582975 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56478bddb2d0 0x56478bde2b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:36.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:36.139+0000 7fd975a8d000 1 --2- 172.21.15.16:0/1709582975 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x56478bd8e9b0 0x56478bde2630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:36.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:36.139+0000 7fd975a8d000 1 -- 172.21.15.16:0/1709582975 >> 172.21.15.16:0/1709582975 conn(0x56478bdbc070 msgr2=0x56478bdbe460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:05:36.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:36.139+0000 7fd954c00640 0 client.5916 ms_handle_reset on v2:172.21.15.16:6832/2768973409 2024-03-31T15:05:36.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:36.139+0000 7fd975a8d000 1 -- 172.21.15.16:0/1709582975 shutdown_connections 2024-03-31T15:05:36.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:36.139+0000 7fd975a8d000 1 -- 172.21.15.16:0/1709582975 wait complete. 2024-03-31T15:05:36.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:36.139+0000 7fd975a8d000 20 client.5916 trim_cache size 0 max 16384 2024-03-31T15:05:36.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:36.155+0000 7fd975a8d000 1 Processor -- start 2024-03-31T15:05:36.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:36.155+0000 7fd975a8d000 1 -- start start 2024-03-31T15:05:36.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:36.159+0000 7fd975a8d000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56478bd8e9b0 0x56478bdd9380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:36.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:36.159+0000 7fd975a8d000 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56478bddb2d0 0x56478bdd98c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:36.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:36.159+0000 7fd975a8d000 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x56478bddbbf0 0x56478bdd9e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:36.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:36.159+0000 7fd975a8d000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fd96c145f20 con 0x56478bd8e9b0 2024-03-31T15:05:36.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:36.159+0000 7fd975a8d000 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fd96c14af10 con 0x56478bddb2d0 2024-03-31T15:05:36.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:36.159+0000 7fd975a8d000 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x56478bbac150 con 0x56478bddbbf0 2024-03-31T15:05:36.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:36.159+0000 7fd96b400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56478bd8e9b0 0x56478bdd9380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:36.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:36.159+0000 7fd96b400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56478bd8e9b0 0x56478bdd9380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34438/0 (socket says 172.21.15.16:34438) 2024-03-31T15:05:36.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:36.159+0000 7fd96b400640 1 -- 172.21.15.16:0/237053221 learned_addr learned my addr 172.21.15.16:0/237053221 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:05:36.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:36.159+0000 7fd96be00640 1 --2- 172.21.15.16:0/237053221 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x56478bddbbf0 0x56478bdd9e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:36.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:36.159+0000 7fd96aa00640 1 --2- 172.21.15.16:0/237053221 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56478bddb2d0 0x56478bdd98c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:36.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:36.159+0000 7fd96b400640 1 -- 172.21.15.16:0/237053221 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x56478bddbbf0 msgr2=0x56478bdd9e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:05:36.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:36.159+0000 7fd96b400640 1 --2- 172.21.15.16:0/237053221 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x56478bddbbf0 0x56478bdd9e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:36.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:36.159+0000 7fd96b400640 1 -- 172.21.15.16:0/237053221 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56478bddb2d0 msgr2=0x56478bdd98c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:05:36.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:36.159+0000 7fd96b400640 1 --2- 172.21.15.16:0/237053221 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56478bddb2d0 0x56478bdd98c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:36.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:36.159+0000 7fd96b400640 1 -- 172.21.15.16:0/237053221 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56478bc61140 con 0x56478bd8e9b0 2024-03-31T15:05:36.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:36.159+0000 7fd96b400640 1 --2- 172.21.15.16:0/237053221 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56478bd8e9b0 0x56478bdd9380 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7fd95c019100 tx=0x7fd95c121040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59090622dac1110b server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:05:36.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:36.159+0000 7fd957400640 1 -- 172.21.15.16:0/237053221 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd95c109030 con 0x56478bd8e9b0 2024-03-31T15:05:36.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:36.159+0000 7fd957400640 1 -- 172.21.15.16:0/237053221 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd95c0b0040 con 0x56478bd8e9b0 2024-03-31T15:05:36.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:36.159+0000 7fd957400640 1 -- 172.21.15.16:0/237053221 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd95c0c4040 con 0x56478bd8e9b0 2024-03-31T15:05:36.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:36.159+0000 7fd975a8d000 1 -- 172.21.15.16:0/237053221 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56478bd8e9b0 msgr2=0x56478bdd9380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:36.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:36.159+0000 7fd975a8d000 1 --2- 172.21.15.16:0/237053221 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56478bd8e9b0 0x56478bdd9380 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7fd95c019100 tx=0x7fd95c121040 comp rx=0 tx=0).stop 2024-03-31T15:05:36.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:36.159+0000 7fd975a8d000 1 -- 172.21.15.16:0/237053221 shutdown_connections 2024-03-31T15:05:36.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:36.159+0000 7fd975a8d000 1 --2- 172.21.15.16:0/237053221 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x56478bddbbf0 0x56478bdd9e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:36.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:36.159+0000 7fd975a8d000 1 --2- 172.21.15.16:0/237053221 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56478bddb2d0 0x56478bdd98c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:36.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:36.159+0000 7fd975a8d000 1 --2- 172.21.15.16:0/237053221 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56478bd8e9b0 0x56478bdd9380 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:36.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:36.159+0000 7fd975a8d000 1 -- 172.21.15.16:0/237053221 >> 172.21.15.16:0/237053221 conn(0x56478bdbc070 msgr2=0x56478bd8cbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:05:36.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:36.159+0000 7fd975a8d000 1 -- 172.21.15.16:0/237053221 shutdown_connections 2024-03-31T15:05:36.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:36.159+0000 7fd975a8d000 1 -- 172.21.15.16:0/237053221 wait complete. 2024-03-31T15:05:36.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:36.159+0000 7fd975a8d000 1 -- start start 2024-03-31T15:05:36.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:36.159+0000 7fd975a8d000 1 Processor -- start 2024-03-31T15:05:36.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:36.159+0000 7fd975a8d000 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x56478bd8e9b0 0x56478be4b6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:36.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:36.159+0000 7fd975a8d000 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56478bddb2d0 0x56478be4bc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:36.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:36.159+0000 7fd975a8d000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56478bddbbf0 0x56478bddf030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:36.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:36.159+0000 7fd975a8d000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x56478bc9b250 con 0x56478bddbbf0 2024-03-31T15:05:36.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:36.159+0000 7fd975a8d000 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x56478bb7b940 con 0x56478bddb2d0 2024-03-31T15:05:36.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:36.159+0000 7fd975a8d000 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x56478bcaf420 con 0x56478bd8e9b0 2024-03-31T15:05:36.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:36.159+0000 7fd975a8d000 10 client.0 _ll_register_callbacks cb 0x7fd95c0ed440 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T15:05:36.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:36.159+0000 7fd96be00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56478bddbbf0 0x56478bddf030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:36.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:36.159+0000 7fd96be00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56478bddbbf0 0x56478bddf030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34450/0 (socket says 172.21.15.16:34450) 2024-03-31T15:05:36.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:36.159+0000 7fd96b400640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x56478bd8e9b0 0x56478be4b6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:36.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:36.163+0000 7fd96be00640 1 -- 172.21.15.16:0/1830946510 learned_addr learned my addr 172.21.15.16:0/1830946510 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:05:36.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:36.163+0000 7fd96aa00640 1 --2- 172.21.15.16:0/1830946510 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56478bddb2d0 0x56478be4bc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:36.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:36.163+0000 7fd96be00640 1 -- 172.21.15.16:0/1830946510 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x56478bd8e9b0 msgr2=0x56478be4b6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:05:36.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:36.163+0000 7fd96be00640 1 --2- 172.21.15.16:0/1830946510 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x56478bd8e9b0 0x56478be4b6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:36.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:36.163+0000 7fd96be00640 1 -- 172.21.15.16:0/1830946510 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56478bddb2d0 msgr2=0x56478be4bc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:05:36.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:36.163+0000 7fd96be00640 1 --2- 172.21.15.16:0/1830946510 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56478bddb2d0 0x56478be4bc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:36.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:36.163+0000 7fd96be00640 1 -- 172.21.15.16:0/1830946510 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56478bb6f1e0 con 0x56478bddbbf0 2024-03-31T15:05:36.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:36.163+0000 7fd96be00640 1 --2- 172.21.15.16:0/1830946510 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56478bddbbf0 0x56478bddf030 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7fd964007cf0 tx=0x7fd96400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=64c2106f6e837e54 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:05:36.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:36.163+0000 7fd954c00640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-31T15:05:36.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:36.163+0000 7fd954c00640 1 -- 172.21.15.16:0/1830946510 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd964013b10 con 0x56478bddbbf0 2024-03-31T15:05:36.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:36.163+0000 7fd954c00640 1 -- 172.21.15.16:0/1830946510 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(45..45 src has 1..45) ==== 5709+0+0 (secure 0 0 0) 0x7fd964036ae0 con 0x56478bddbbf0 2024-03-31T15:05:36.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:36.163+0000 7fd975a8d000 1 -- 172.21.15.16:0/1830946510 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd95c0b0040 con 0x56478bddbbf0 2024-03-31T15:05:36.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:36.163+0000 7fd954c00640 1 -- 172.21.15.16:0/1830946510 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fd96403c050 con 0x56478bddbbf0 2024-03-31T15:05:36.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:36.163+0000 7fd975a8d000 10 client.5922 fetch_fsmap learned FSMap version 25 2024-03-31T15:05:36.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:36.163+0000 7fd975a8d000 1 -- 172.21.15.16:0/1830946510 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=2+}) -- 0x56478bb747b0 con 0x56478bddbbf0 2024-03-31T15:05:36.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:36.163+0000 7fd954c00640 1 -- 172.21.15.16:0/1830946510 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd95c0b0040 con 0x56478bddbbf0 2024-03-31T15:05:36.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:36.163+0000 7fd954c00640 1 -- 172.21.15.16:0/1830946510 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 25) ==== 3494+0+0 (secure 0 0 0) 0x7fd964036ce0 con 0x56478bddbbf0 2024-03-31T15:05:36.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:36.163+0000 7fd975a8d000 10 client.5922 fetch_fsmap finished waiting for FSMap version 25 2024-03-31T15:05:36.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:36.163+0000 7fd975a8d000 10 client.5922 resolve_mds: resolved 3:0 to role '3:0' aka daemon mds.a 2024-03-31T15:05:36.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:36.163+0000 7fd975a8d000 20 client.5922 populate_metadata read hostname 'smithi016' 2024-03-31T15:05:36.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:36.163+0000 7fd975a8d000 1 --2- 172.21.15.16:0/1830946510 >> [v2:172.21.15.16:6832/2768973409,v1:172.21.15.16:6834/2768973409] conn(0x56478bddf570 0x56478bddf980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:05:36.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:36.163+0000 7fd975a8d000 4 client.5922 mds_command: new command op to 5697 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-31T15:05:36.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:36.163+0000 7fd975a8d000 1 -- 172.21.15.16:0/1830946510 --> [v2:172.21.15.16:6832/2768973409,v1:172.21.15.16:6834/2768973409] -- command(tid 0: {"prefix": "scrub status"}) -- 0x56478bd96a70 con 0x56478bddf570 2024-03-31T15:05:36.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:36.163+0000 7fd96b400640 1 --2- 172.21.15.16:0/1830946510 >> [v2:172.21.15.16:6832/2768973409,v1:172.21.15.16:6834/2768973409] conn(0x56478bddf570 0x56478bddf980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:05:36.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:36.163+0000 7fd96b400640 1 --2- 172.21.15.16:0/1830946510 >> [v2:172.21.15.16:6832/2768973409,v1:172.21.15.16:6834/2768973409] conn(0x56478bddf570 0x56478bddf980 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=82a50cd6b949cb0e server_cookie=556a01fbefa9e50e in_seq=0 out_seq=0 2024-03-31T15:05:36.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:36.163+0000 7fd954c00640 10 client.5922 ms_handle_connect on v2:172.21.15.16:6832/2768973409 2024-03-31T15:05:36.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:36.163+0000 7fd954c00640 1 -- 172.21.15.16:0/1830946510 <== mds.0 v2:172.21.15.16:6832/2768973409 1 ==== command_reply(tid 0: 0 ) ==== 8+0+260 (crc 0 0 0) 0x56478bd96a70 con 0x56478bddf570 2024-03-31T15:05:36.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:36.163+0000 7fd954c00640 10 client.5922 handle_command_reply: tid=0 2024-03-31T15:05:36.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:36.163+0000 7fd975a8d000 1 client.5922 shutdown 2024-03-31T15:05:36.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:36.163+0000 7fd975a8d000 1 -- 172.21.15.16:0/1830946510 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56478bddbbf0 msgr2=0x56478bddf030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:36.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:36.163+0000 7fd975a8d000 1 --2- 172.21.15.16:0/1830946510 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56478bddbbf0 0x56478bddf030 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7fd964007cf0 tx=0x7fd96400b040 comp rx=0 tx=0).stop 2024-03-31T15:05:36.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:36.163+0000 7fd975a8d000 1 -- 172.21.15.16:0/1830946510 shutdown_connections 2024-03-31T15:05:36.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:36.163+0000 7fd975a8d000 1 --2- 172.21.15.16:0/1830946510 >> [v2:172.21.15.16:6832/2768973409,v1:172.21.15.16:6834/2768973409] conn(0x56478bddf570 0x56478bddf980 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:36.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:36.163+0000 7fd975a8d000 1 --2- 172.21.15.16:0/1830946510 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56478bddbbf0 0x56478bddf030 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:36.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:36.163+0000 7fd975a8d000 1 --2- 172.21.15.16:0/1830946510 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56478bddb2d0 0x56478be4bc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:36.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:36.163+0000 7fd975a8d000 1 --2- 172.21.15.16:0/1830946510 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x56478bd8e9b0 0x56478be4b6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:36.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:36.163+0000 7fd954c00640 0 client.5922 ms_handle_reset on v2:172.21.15.16:6832/2768973409 2024-03-31T15:05:36.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:36.163+0000 7fd975a8d000 1 -- 172.21.15.16:0/1830946510 >> 172.21.15.16:0/1830946510 conn(0x56478bdbc070 msgr2=0x56478bd8d460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:05:36.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:36.163+0000 7fd975a8d000 1 -- 172.21.15.16:0/1830946510 shutdown_connections 2024-03-31T15:05:36.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:36.163+0000 7fd975a8d000 1 -- 172.21.15.16:0/1830946510 wait complete. 2024-03-31T15:05:36.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:36.163+0000 7fd975a8d000 20 client.5922 trim_cache size 0 max 16384 2024-03-31T15:05:36.163 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-03-31T15:05:36.164 INFO:teuthology.orchestra.run.smithi016.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-31T15:05:36.164 INFO:teuthology.orchestra.run.smithi016.stdout: "scrubs": { 2024-03-31T15:05:36.164 INFO:teuthology.orchestra.run.smithi016.stdout: "453e7a77-7af9-4065-bfad-edf570d63b51": { 2024-03-31T15:05:36.164 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-03-31T15:05:36.164 INFO:teuthology.orchestra.run.smithi016.stdout: "tag": "453e7a77-7af9-4065-bfad-edf570d63b51", 2024-03-31T15:05:36.164 INFO:teuthology.orchestra.run.smithi016.stdout: "options": "recursive,repair" 2024-03-31T15:05:36.164 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-31T15:05:36.164 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-31T15:05:36.164 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-03-31T15:05:36.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:36.163+0000 7fd971000640 1 -- 172.21.15.16:0/4182909677 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fd94c068380 msgr2=0x7fd94c06a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:36.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:36.163+0000 7fd971000640 1 --2- 172.21.15.16:0/4182909677 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fd94c068380 0x7fd94c06a840 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fd958007fd0 tx=0x7fd958009040 comp rx=0 tx=0).stop 2024-03-31T15:05:36.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:36.163+0000 7fd971000640 1 -- 172.21.15.16:0/4182909677 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd96c0d0070 msgr2=0x7fd96c099a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:05:36.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:36.163+0000 7fd971000640 1 --2- 172.21.15.16:0/4182909677 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd96c0d0070 0x7fd96c099a10 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7fd95c0ad040 tx=0x7fd95c01ad30 comp rx=0 tx=0).stop 2024-03-31T15:05:36.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:36.163+0000 7fd971000640 1 -- 172.21.15.16:0/4182909677 shutdown_connections 2024-03-31T15:05:36.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:36.163+0000 7fd971000640 1 --2- 172.21.15.16:0/4182909677 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fd94c068380 0x7fd94c06a840 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:36.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:36.163+0000 7fd971000640 1 --2- 172.21.15.16:0/4182909677 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd96c15f290 0x7fd96c1462a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:36.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:36.163+0000 7fd971000640 1 --2- 172.21.15.16:0/4182909677 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fd96c15c670 0x7fd96c14d430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:36.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:36.163+0000 7fd971000640 1 --2- 172.21.15.16:0/4182909677 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd96c0d0070 0x7fd96c099a10 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:05:36.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:36.163+0000 7fd971000640 1 -- 172.21.15.16:0/4182909677 >> 172.21.15.16:0/4182909677 conn(0x7fd96c0b5520 msgr2=0x7fd96c15b8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:05:36.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:36.163+0000 7fd971000640 1 -- 172.21.15.16:0/4182909677 shutdown_connections 2024-03-31T15:05:36.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:36.163+0000 7fd971000640 1 -- 172.21.15.16:0/4182909677 wait complete. 2024-03-31T15:05:36.178 INFO:tasks.cephfs.filesystem:scrub status for tag:453e7a77-7af9-4065-bfad-edf570d63b51 - {'path': '/', 'tag': '453e7a77-7af9-4065-bfad-edf570d63b51', 'options': 'recursive,repair'} 2024-03-31T15:05:36.804 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-31T15:05:36.807 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-31T15:05:37.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:37.115+0000 7f3e91600640 1 -- 172.21.15.16:0/1760146965 <== mon.0 v2:172.21.15.16:3300/0 7 ==== log(6 entries from seq 1 at 2024-03-31T15:05:35.952729+0000) ==== 1078+0+0 (secure 0 0 0) 0x7f3e800860d0 con 0x7f3e940b2f50 2024-03-31T15:05:37.110 INFO:teuthology.orchestra.run.smithi016.stdout: cluster: 2024-03-31T15:05:37.110 INFO:teuthology.orchestra.run.smithi016.stdout: id: 286eb5da-fb47-4550-b290-d9519c13b9a5 2024-03-31T15:05:37.110 INFO:teuthology.orchestra.run.smithi016.stdout: health: HEALTH_OK 2024-03-31T15:05:37.110 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:05:37.110 INFO:teuthology.orchestra.run.smithi016.stdout: services: 2024-03-31T15:05:37.110 INFO:teuthology.orchestra.run.smithi016.stdout: mon: 3 daemons, quorum a,b,c (age 2m) 2024-03-31T15:05:37.111 INFO:teuthology.orchestra.run.smithi016.stdout: mgr: x(active, since 2m), standbys: y 2024-03-31T15:05:37.111 INFO:teuthology.orchestra.run.smithi016.stdout: mds: 1/1 daemons up, 3 standby 2024-03-31T15:05:37.111 INFO:teuthology.orchestra.run.smithi016.stdout: osd: 8 osds: 8 up (since 2m), 8 in (since 2m) 2024-03-31T15:05:37.111 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:05:37.111 INFO:teuthology.orchestra.run.smithi016.stdout: data: 2024-03-31T15:05:37.111 INFO:teuthology.orchestra.run.smithi016.stdout: volumes: 1/1 healthy 2024-03-31T15:05:37.111 INFO:teuthology.orchestra.run.smithi016.stdout: pools: 3 pools, 97 pgs 2024-03-31T15:05:37.111 INFO:teuthology.orchestra.run.smithi016.stdout: objects: 24 objects, 579 KiB 2024-03-31T15:05:37.111 INFO:teuthology.orchestra.run.smithi016.stdout: usage: 237 MiB used, 720 GiB / 720 GiB avail 2024-03-31T15:05:37.111 INFO:teuthology.orchestra.run.smithi016.stdout: pgs: 97 active+clean 2024-03-31T15:05:37.111 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:05:37.111 INFO:teuthology.orchestra.run.smithi016.stdout: io: 2024-03-31T15:05:37.111 INFO:teuthology.orchestra.run.smithi016.stdout: client: 1.7 KiB/s wr, 0 op/s rd, 5 op/s wr 2024-03-31T15:05:37.111 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:05:37.112 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:05:37.112 INFO:teuthology.orchestra.run.smithi016.stdout:2024-03-31T15:05:35.952729+0000 mds.a [INF] scrub queued for path: / 2024-03-31T15:05:37.112 INFO:teuthology.orchestra.run.smithi016.stdout:2024-03-31T15:05:35.952733+0000 mds.a [INF] scrub summary: idle+waiting paths [/] 2024-03-31T15:05:37.112 INFO:teuthology.orchestra.run.smithi016.stdout:2024-03-31T15:05:35.952740+0000 mds.a [INF] scrub summary: active paths [/] 2024-03-31T15:05:37.112 INFO:teuthology.orchestra.run.smithi016.stdout:2024-03-31T15:05:35.953876+0000 mds.a [WRN] bad backtrace on inode 0x10000000001(/parent_a/alpha), rewriting it 2024-03-31T15:05:37.112 INFO:teuthology.orchestra.run.smithi016.stdout:2024-03-31T15:05:35.953886+0000 mds.a [INF] Scrub repaired inode 0x10000000001 (/parent_a/alpha) 2024-03-31T15:05:37.112 INFO:teuthology.orchestra.run.smithi016.stdout:2024-03-31T15:05:35.954525+0000 mds.a [INF] scrub summary: idle+waiting paths [/] 2024-03-31T15:05:39.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:05:39.123+0000 7f3e91600640 1 -- 172.21.15.16:0/1760146965 <== mon.0 v2:172.21.15.16:3300/0 8 ==== log(1 entries from seq 7 at 2024-03-31T15:05:37.204430+0000) ==== 189+0+0 (secure 0 0 0) 0x7f3e80084910 con 0x7f3e940b2f50 2024-03-31T15:05:39.118 INFO:teuthology.orchestra.run.smithi016.stdout:2024-03-31T15:05:37.204430+0000 mds.a [INF] scrub summary: idle 2024-03-31T15:06:06.179 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:06:06.180 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.3:0 scrub status 2024-03-31T15:06:06.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.327+0000 7f304a400640 1 Processor -- start 2024-03-31T15:06:06.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.327+0000 7f304a400640 1 -- start start 2024-03-31T15:06:06.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.327+0000 7f304a400640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f3044145d50 0x7f3044160860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:06.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.331+0000 7f304a400640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f304415c890 0x7f304406fc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:06.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.331+0000 7f304a400640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3044070a10 con 0x7f304406f160 2024-03-31T15:06:06.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.331+0000 7f304a400640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f30440ce5c0 con 0x7f3044145d50 2024-03-31T15:06:06.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.331+0000 7f304a400640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f30440cd9b0 con 0x7f304415c890 2024-03-31T15:06:06.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.331+0000 7f3043400640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f3044145d50 0x7f3044160860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:06.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.331+0000 7f303be00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f304415c890 0x7f304406fc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:06.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.331+0000 7f3043400640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f3044145d50 0x7f3044160860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.16:43906/0 (socket says 172.21.15.16:43906) 2024-03-31T15:06:06.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.331+0000 7f3043400640 1 -- 172.21.15.16:0/3872574426 learned_addr learned my addr 172.21.15.16:0/3872574426 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:06:06.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.331+0000 7f3043e00640 1 --1- 172.21.15.16:0/3872574426 >> v1:172.21.15.89:6789/0 conn(0x7f304406f160 0x7f304406b620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.16:42442/0 (socket says 172.21.15.16:42442) 2024-03-31T15:06:06.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.331+0000 7f3043400640 1 -- 172.21.15.16:0/3872574426 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f304415c890 msgr2=0x7f304406fc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:06:06.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.331+0000 7f3043400640 1 --2- 172.21.15.16:0/3872574426 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f304415c890 0x7f304406fc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:06.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.331+0000 7f3043400640 1 -- 172.21.15.16:0/3872574426 >> v1:172.21.15.89:6789/0 conn(0x7f304406f160 legacy=0x7f304406b620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:06.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.331+0000 7f3043400640 1 -- 172.21.15.16:0/3872574426 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f30440cdc90 con 0x7f3044145d50 2024-03-31T15:06:06.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.335+0000 7f3043400640 1 --2- 172.21.15.16:0/3872574426 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3044145d50 0x7f3044160860 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f3034009870 tx=0x7f303401c560 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7e7a4dfe2545910c server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:06:06.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.335+0000 7f3042a00640 1 -- 172.21.15.16:0/3872574426 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30340b2070 con 0x7f3044145d50 2024-03-31T15:06:06.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.335+0000 7f3042a00640 1 -- 172.21.15.16:0/3872574426 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f30340a9070 con 0x7f3044145d50 2024-03-31T15:06:06.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.335+0000 7f3042a00640 1 -- 172.21.15.16:0/3872574426 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30340ad880 con 0x7f3044145d50 2024-03-31T15:06:06.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.335+0000 7f304a400640 1 -- 172.21.15.16:0/3872574426 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3044145d50 msgr2=0x7f3044160860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:06.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.335+0000 7f304a400640 1 --2- 172.21.15.16:0/3872574426 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3044145d50 0x7f3044160860 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f3034009870 tx=0x7f303401c560 comp rx=0 tx=0).stop 2024-03-31T15:06:06.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.335+0000 7f304a400640 1 -- 172.21.15.16:0/3872574426 shutdown_connections 2024-03-31T15:06:06.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.335+0000 7f304a400640 1 --2- 172.21.15.16:0/3872574426 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f304415c890 0x7f304406fc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:06.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.335+0000 7f304a400640 1 --2- 172.21.15.16:0/3872574426 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3044145d50 0x7f3044160860 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:06.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.335+0000 7f304a400640 1 -- 172.21.15.16:0/3872574426 >> 172.21.15.16:0/3872574426 conn(0x7f304406e670 msgr2=0x7f304406ea50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:06:06.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.335+0000 7f304a400640 1 -- 172.21.15.16:0/3872574426 shutdown_connections 2024-03-31T15:06:06.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.335+0000 7f304a400640 1 -- 172.21.15.16:0/3872574426 wait complete. 2024-03-31T15:06:06.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.335+0000 7f304a400640 1 Processor -- start 2024-03-31T15:06:06.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.335+0000 7f304a400640 1 -- start start 2024-03-31T15:06:06.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.335+0000 7f304a400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f304406f160 0x7f3044142550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:06.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.335+0000 7f304a400640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3044145d50 0x7f304413a9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:06.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.335+0000 7f304a400640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f304415c890 0x7f30441663a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:06.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.335+0000 7f304a400640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f3044169520 con 0x7f304406f160 2024-03-31T15:06:06.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.335+0000 7f304a400640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f30441693a0 con 0x7f3044145d50 2024-03-31T15:06:06.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.335+0000 7f304a400640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f30441696a0 con 0x7f304415c890 2024-03-31T15:06:06.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.339+0000 7f3043400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f304406f160 0x7f3044142550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:06.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.339+0000 7f3043400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f304406f160 0x7f3044142550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40282/0 (socket says 172.21.15.16:40282) 2024-03-31T15:06:06.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.339+0000 7f3043400640 1 -- 172.21.15.16:0/1749580697 learned_addr learned my addr 172.21.15.16:0/1749580697 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:06:06.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.339+0000 7f3043e00640 1 --2- 172.21.15.16:0/1749580697 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f304415c890 0x7f30441663a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:06.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.339+0000 7f303be00640 1 --2- 172.21.15.16:0/1749580697 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3044145d50 0x7f304413a9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:06.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.339+0000 7f3043400640 1 -- 172.21.15.16:0/1749580697 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f304415c890 msgr2=0x7f30441663a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:06.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.339+0000 7f3043400640 1 --2- 172.21.15.16:0/1749580697 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f304415c890 0x7f30441663a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:06.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.339+0000 7f3043400640 1 -- 172.21.15.16:0/1749580697 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3044145d50 msgr2=0x7f304413a9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:06.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.339+0000 7f3043400640 1 --2- 172.21.15.16:0/1749580697 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3044145d50 0x7f304413a9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:06.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.339+0000 7f3043400640 1 -- 172.21.15.16:0/1749580697 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3044141920 con 0x7f304406f160 2024-03-31T15:06:06.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.339+0000 7f3043400640 1 --2- 172.21.15.16:0/1749580697 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f304406f160 0x7f3044142550 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f30340b0040 tx=0x7f303401ada0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:06:06.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.339+0000 7f303b400640 1 -- 172.21.15.16:0/1749580697 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30340b2040 con 0x7f304406f160 2024-03-31T15:06:06.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.339+0000 7f303b400640 1 -- 172.21.15.16:0/1749580697 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f30340be070 con 0x7f304406f160 2024-03-31T15:06:06.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.339+0000 7f304a400640 1 -- 172.21.15.16:0/1749580697 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f30440b5bf0 con 0x7f304406f160 2024-03-31T15:06:06.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.339+0000 7f303b400640 1 -- 172.21.15.16:0/1749580697 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30340b8420 con 0x7f304406f160 2024-03-31T15:06:06.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.339+0000 7f304a400640 1 -- 172.21.15.16:0/1749580697 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f304413aee0 con 0x7f304406f160 2024-03-31T15:06:06.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.339+0000 7f303b400640 1 -- 172.21.15.16:0/1749580697 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f30340ab070 con 0x7f304406f160 2024-03-31T15:06:06.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.343+0000 7f303b400640 1 --2- 172.21.15.16:0/1749580697 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f3020068380 0x7f302006a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:06.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.343+0000 7f303b400640 1 -- 172.21.15.16:0/1749580697 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5709+0+0 (secure 0 0 0) 0x7f30341208f0 con 0x7f304406f160 2024-03-31T15:06:06.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.343+0000 7f303be00640 1 --2- 172.21.15.16:0/1749580697 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f3020068380 0x7f302006a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:06.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.343+0000 7f304d808000 1 Processor -- start 2024-03-31T15:06:06.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.343+0000 7f304d808000 1 -- start start 2024-03-31T15:06:06.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.343+0000 7f303be00640 1 --2- 172.21.15.16:0/1749580697 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f3020068380 0x7f302006a840 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f304413f3f0 tx=0x7f302c009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:06:06.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.343+0000 7f304d808000 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5635ec73f2d0 0x5635ec73f6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:06.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.343+0000 7f304d808000 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5635ec73fbf0 0x5635ec6f2470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:06.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.343+0000 7f304d808000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5635ec6f29b0 0x5635ec6f2d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:06.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.343+0000 7f304d808000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5635ec5dbaf0 con 0x5635ec6f29b0 2024-03-31T15:06:06.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.343+0000 7f304d808000 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x5635ec544b00 con 0x5635ec73f2d0 2024-03-31T15:06:06.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.343+0000 7f304d808000 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x5635ec4e1940 con 0x5635ec73fbf0 2024-03-31T15:06:06.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.343+0000 7f3043e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5635ec6f29b0 0x5635ec6f2d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:06.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.343+0000 7f303be00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5635ec73fbf0 0x5635ec6f2470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:06.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.343+0000 7f3043400640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5635ec73f2d0 0x5635ec73f6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:06.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.343+0000 7f3043e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5635ec6f29b0 0x5635ec6f2d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40298/0 (socket says 172.21.15.16:40298) 2024-03-31T15:06:06.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.343+0000 7f3043e00640 1 -- 172.21.15.16:0/4273550171 learned_addr learned my addr 172.21.15.16:0/4273550171 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:06:06.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.343+0000 7f3043400640 1 -- 172.21.15.16:0/4273550171 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5635ec73fbf0 msgr2=0x5635ec6f2470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:06:06.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.343+0000 7f3043400640 1 --2- 172.21.15.16:0/4273550171 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5635ec73fbf0 0x5635ec6f2470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:06.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.343+0000 7f3043400640 1 -- 172.21.15.16:0/4273550171 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5635ec6f29b0 msgr2=0x5635ec6f2d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:06:06.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.343+0000 7f3043400640 1 --2- 172.21.15.16:0/4273550171 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5635ec6f29b0 0x5635ec6f2d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:06.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.343+0000 7f3043400640 1 -- 172.21.15.16:0/4273550171 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5635ec4ca380 con 0x5635ec73f2d0 2024-03-31T15:06:06.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.343+0000 7f3043e00640 1 --2- 172.21.15.16:0/4273550171 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5635ec6f29b0 0x5635ec6f2d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:06:06.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.343+0000 7f3043400640 1 --2- 172.21.15.16:0/4273550171 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5635ec73f2d0 0x5635ec73f6b0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f303411ed80 tx=0x7f3034019300 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8c860e3d380bb051 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:06:06.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.343+0000 7f3033e00640 1 -- 172.21.15.16:0/4273550171 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30340f92f0 con 0x5635ec73f2d0 2024-03-31T15:06:06.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.343+0000 7f3033e00640 1 -- 172.21.15.16:0/4273550171 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f30340a9050 con 0x5635ec73f2d0 2024-03-31T15:06:06.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.343+0000 7f3033e00640 1 -- 172.21.15.16:0/4273550171 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30340eeac0 con 0x5635ec73f2d0 2024-03-31T15:06:06.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.343+0000 7f304d808000 1 -- 172.21.15.16:0/4273550171 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5635ec73f2d0 msgr2=0x5635ec73f6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:06.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.343+0000 7f304d808000 1 --2- 172.21.15.16:0/4273550171 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5635ec73f2d0 0x5635ec73f6b0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f303411ed80 tx=0x7f3034019300 comp rx=0 tx=0).stop 2024-03-31T15:06:06.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.343+0000 7f304d808000 1 -- 172.21.15.16:0/4273550171 shutdown_connections 2024-03-31T15:06:06.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.343+0000 7f304d808000 1 --2- 172.21.15.16:0/4273550171 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5635ec6f29b0 0x5635ec6f2d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:06.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.343+0000 7f304d808000 1 --2- 172.21.15.16:0/4273550171 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5635ec73fbf0 0x5635ec6f2470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:06.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.343+0000 7f304d808000 1 --2- 172.21.15.16:0/4273550171 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5635ec73f2d0 0x5635ec73f6b0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:06.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.343+0000 7f304d808000 1 -- 172.21.15.16:0/4273550171 >> 172.21.15.16:0/4273550171 conn(0x5635ec720070 msgr2=0x5635ec722460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:06:06.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.343+0000 7f304d808000 1 -- 172.21.15.16:0/4273550171 shutdown_connections 2024-03-31T15:06:06.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.343+0000 7f304d808000 1 -- 172.21.15.16:0/4273550171 wait complete. 2024-03-31T15:06:06.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.347+0000 7f304d808000 1 -- start start 2024-03-31T15:06:06.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.347+0000 7f304d808000 1 Processor -- start 2024-03-31T15:06:06.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.347+0000 7f304d808000 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5635ec6f29b0 0x5635ec6f7540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:06.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.347+0000 7f304d808000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5635ec73f2d0 0x5635ec6f7a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:06.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.347+0000 7f304d808000 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5635ec73fbf0 0x5635ec6f7fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:06.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.347+0000 7f304d808000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5635ec5021e0 con 0x5635ec73f2d0 2024-03-31T15:06:06.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.347+0000 7f304d808000 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x5635ec540ab0 con 0x5635ec6f29b0 2024-03-31T15:06:06.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.347+0000 7f304d808000 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x5635ec544b00 con 0x5635ec73fbf0 2024-03-31T15:06:06.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.347+0000 7f304d808000 10 client.0 _ll_register_callbacks cb 0x5635ec6e0370 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T15:06:06.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.347+0000 7f303be00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5635ec73f2d0 0x5635ec6f7a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:06.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.347+0000 7f303be00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5635ec73f2d0 0x5635ec6f7a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40306/0 (socket says 172.21.15.16:40306) 2024-03-31T15:06:06.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.347+0000 7f303be00640 1 -- 172.21.15.16:0/1831101451 learned_addr learned my addr 172.21.15.16:0/1831101451 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:06:06.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.347+0000 7f3043e00640 1 --2- 172.21.15.16:0/1831101451 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5635ec73fbf0 0x5635ec6f7fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:06.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.347+0000 7f3043400640 1 --2- 172.21.15.16:0/1831101451 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5635ec6f29b0 0x5635ec6f7540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:06.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.347+0000 7f303be00640 1 -- 172.21.15.16:0/1831101451 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5635ec73fbf0 msgr2=0x5635ec6f7fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:06:06.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.347+0000 7f303be00640 1 --2- 172.21.15.16:0/1831101451 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5635ec73fbf0 0x5635ec6f7fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:06.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.347+0000 7f303be00640 1 -- 172.21.15.16:0/1831101451 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5635ec6f29b0 msgr2=0x5635ec6f7540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:06:06.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.347+0000 7f303be00640 1 --2- 172.21.15.16:0/1831101451 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5635ec6f29b0 0x5635ec6f7540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:06.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.347+0000 7f303be00640 1 -- 172.21.15.16:0/1831101451 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5635ec500b70 con 0x5635ec73f2d0 2024-03-31T15:06:06.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.347+0000 7f303be00640 1 --2- 172.21.15.16:0/1831101451 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5635ec73f2d0 0x5635ec6f7a80 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f302c0149c0 tx=0x7f302c0046d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ad945f9719f5100 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:06:06.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.347+0000 7f3030c00640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-31T15:06:06.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.347+0000 7f3030c00640 1 -- 172.21.15.16:0/1831101451 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f302c027030 con 0x5635ec73f2d0 2024-03-31T15:06:06.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.347+0000 7f3030c00640 1 -- 172.21.15.16:0/1831101451 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(45..45 src has 1..45) ==== 5709+0+0 (secure 0 0 0) 0x7f302c00f040 con 0x5635ec73f2d0 2024-03-31T15:06:06.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.347+0000 7f304d808000 1 -- 172.21.15.16:0/1831101451 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f30340a9050 con 0x5635ec73f2d0 2024-03-31T15:06:06.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.347+0000 7f3030c00640 1 -- 172.21.15.16:0/1831101451 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f302c032050 con 0x5635ec73f2d0 2024-03-31T15:06:06.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.347+0000 7f304d808000 10 client.5931 fetch_fsmap learned FSMap version 25 2024-03-31T15:06:06.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.347+0000 7f304d808000 1 -- 172.21.15.16:0/1831101451 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=2+}) -- 0x5635ec4e1b90 con 0x5635ec73f2d0 2024-03-31T15:06:06.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.347+0000 7f3030c00640 1 -- 172.21.15.16:0/1831101451 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f30340a9050 con 0x5635ec73f2d0 2024-03-31T15:06:06.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.347+0000 7f3030c00640 1 -- 172.21.15.16:0/1831101451 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 25) ==== 3494+0+0 (secure 0 0 0) 0x7f302c035020 con 0x5635ec73f2d0 2024-03-31T15:06:06.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.347+0000 7f304d808000 10 client.5931 fetch_fsmap finished waiting for FSMap version 25 2024-03-31T15:06:06.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.347+0000 7f304d808000 10 client.5931 resolve_mds: resolved 3:0 to role '3:0' aka daemon mds.a 2024-03-31T15:06:06.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.347+0000 7f304d808000 20 client.5931 populate_metadata read hostname 'smithi016' 2024-03-31T15:06:06.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.347+0000 7f304d808000 1 --2- 172.21.15.16:0/1831101451 >> [v2:172.21.15.16:6832/2768973409,v1:172.21.15.16:6834/2768973409] conn(0x5635ec74e570 0x5635ec750960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:06.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.347+0000 7f304d808000 4 client.5931 mds_command: new command op to 5697 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-31T15:06:06.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.347+0000 7f304d808000 1 -- 172.21.15.16:0/1831101451 --> [v2:172.21.15.16:6832/2768973409,v1:172.21.15.16:6834/2768973409] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5635ec4dfdd0 con 0x5635ec74e570 2024-03-31T15:06:06.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.347+0000 7f3043400640 1 --2- 172.21.15.16:0/1831101451 >> [v2:172.21.15.16:6832/2768973409,v1:172.21.15.16:6834/2768973409] conn(0x5635ec74e570 0x5635ec750960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:06.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.351+0000 7f3043400640 1 --2- 172.21.15.16:0/1831101451 >> [v2:172.21.15.16:6832/2768973409,v1:172.21.15.16:6834/2768973409] conn(0x5635ec74e570 0x5635ec750960 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7325e8f7af2dbc66 server_cookie=d4e6ab279447e66 in_seq=0 out_seq=0 2024-03-31T15:06:06.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.351+0000 7f3030c00640 10 client.5931 ms_handle_connect on v2:172.21.15.16:6832/2768973409 2024-03-31T15:06:06.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.351+0000 7f3030c00640 1 -- 172.21.15.16:0/1831101451 <== mds.0 v2:172.21.15.16:6832/2768973409 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5635ec4dfdd0 con 0x5635ec74e570 2024-03-31T15:06:06.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.351+0000 7f3030c00640 10 client.5931 handle_command_reply: tid=0 2024-03-31T15:06:06.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.351+0000 7f304d808000 1 client.5931 shutdown 2024-03-31T15:06:06.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.351+0000 7f304d808000 1 -- 172.21.15.16:0/1831101451 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5635ec73f2d0 msgr2=0x5635ec6f7a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:06.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.351+0000 7f304d808000 1 --2- 172.21.15.16:0/1831101451 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5635ec73f2d0 0x5635ec6f7a80 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f302c0149c0 tx=0x7f302c0046d0 comp rx=0 tx=0).stop 2024-03-31T15:06:06.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.351+0000 7f304d808000 1 -- 172.21.15.16:0/1831101451 shutdown_connections 2024-03-31T15:06:06.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.351+0000 7f304d808000 1 --2- 172.21.15.16:0/1831101451 >> [v2:172.21.15.16:6832/2768973409,v1:172.21.15.16:6834/2768973409] conn(0x5635ec74e570 0x5635ec750960 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:06.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.351+0000 7f304d808000 1 --2- 172.21.15.16:0/1831101451 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5635ec73fbf0 0x5635ec6f7fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:06.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.351+0000 7f304d808000 1 --2- 172.21.15.16:0/1831101451 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5635ec73f2d0 0x5635ec6f7a80 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:06.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.351+0000 7f304d808000 1 --2- 172.21.15.16:0/1831101451 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5635ec6f29b0 0x5635ec6f7540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:06.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.351+0000 7f304d808000 1 -- 172.21.15.16:0/1831101451 >> 172.21.15.16:0/1831101451 conn(0x5635ec720070 msgr2=0x5635ec722460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:06:06.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.351+0000 7f3030c00640 0 client.5931 ms_handle_reset on v2:172.21.15.16:6832/2768973409 2024-03-31T15:06:06.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.351+0000 7f304d808000 1 -- 172.21.15.16:0/1831101451 shutdown_connections 2024-03-31T15:06:06.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.351+0000 7f304d808000 1 -- 172.21.15.16:0/1831101451 wait complete. 2024-03-31T15:06:06.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.351+0000 7f304d808000 20 client.5931 trim_cache size 0 max 16384 2024-03-31T15:06:06.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.383+0000 7f304d808000 1 Processor -- start 2024-03-31T15:06:06.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.383+0000 7f304d808000 1 -- start start 2024-03-31T15:06:06.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.383+0000 7f304d808000 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5635ec6f29b0 0x5635ec7a9b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:06.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.383+0000 7f304d808000 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5635ec73f2d0 0x5635ec7aa040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:06.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.383+0000 7f304d808000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5635ec73fbf0 0x5635ec7aa580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:06.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.383+0000 7f304d808000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f3044203c20 con 0x5635ec73fbf0 2024-03-31T15:06:06.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.383+0000 7f304d808000 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f3044202c80 con 0x5635ec6f29b0 2024-03-31T15:06:06.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.383+0000 7f304d808000 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f3044202010 con 0x5635ec73f2d0 2024-03-31T15:06:06.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.383+0000 7f3043e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5635ec73fbf0 0x5635ec7aa580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:06.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.383+0000 7f3043e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5635ec73fbf0 0x5635ec7aa580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40312/0 (socket says 172.21.15.16:40312) 2024-03-31T15:06:06.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.383+0000 7f3043e00640 1 -- 172.21.15.16:0/307116308 learned_addr learned my addr 172.21.15.16:0/307116308 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:06:06.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.383+0000 7f3043400640 1 --2- 172.21.15.16:0/307116308 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5635ec6f29b0 0x5635ec7a9b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:06.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.383+0000 7f303be00640 1 --2- 172.21.15.16:0/307116308 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5635ec73f2d0 0x5635ec7aa040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:06.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.383+0000 7f3043e00640 1 -- 172.21.15.16:0/307116308 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5635ec73f2d0 msgr2=0x5635ec7aa040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:06:06.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.383+0000 7f3043e00640 1 --2- 172.21.15.16:0/307116308 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5635ec73f2d0 0x5635ec7aa040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:06.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.383+0000 7f3043e00640 1 -- 172.21.15.16:0/307116308 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5635ec6f29b0 msgr2=0x5635ec7a9b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:06:06.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.383+0000 7f3043e00640 1 --2- 172.21.15.16:0/307116308 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5635ec6f29b0 0x5635ec7a9b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:06.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.383+0000 7f3043e00640 1 -- 172.21.15.16:0/307116308 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5635ec73fa20 con 0x5635ec73fbf0 2024-03-31T15:06:06.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.383+0000 7f303be00640 1 --2- 172.21.15.16:0/307116308 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5635ec73f2d0 0x5635ec7aa040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:06:06.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.383+0000 7f3043e00640 1 --2- 172.21.15.16:0/307116308 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5635ec73fbf0 0x5635ec7aa580 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f303c001e60 tx=0x7f303c001680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40221298795e7051 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:06:06.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.383+0000 7f3033400640 1 -- 172.21.15.16:0/307116308 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f303c016420 con 0x5635ec73fbf0 2024-03-31T15:06:06.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.383+0000 7f3033400640 1 -- 172.21.15.16:0/307116308 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f303c0165c0 con 0x5635ec73fbf0 2024-03-31T15:06:06.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.383+0000 7f3033400640 1 -- 172.21.15.16:0/307116308 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f303c016970 con 0x5635ec73fbf0 2024-03-31T15:06:06.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.383+0000 7f304d808000 1 -- 172.21.15.16:0/307116308 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5635ec73fbf0 msgr2=0x5635ec7aa580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:06.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.383+0000 7f304d808000 1 --2- 172.21.15.16:0/307116308 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5635ec73fbf0 0x5635ec7aa580 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f303c001e60 tx=0x7f303c001680 comp rx=0 tx=0).stop 2024-03-31T15:06:06.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.383+0000 7f304d808000 1 -- 172.21.15.16:0/307116308 shutdown_connections 2024-03-31T15:06:06.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.383+0000 7f304d808000 1 --2- 172.21.15.16:0/307116308 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5635ec73fbf0 0x5635ec7aa580 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:06.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.383+0000 7f304d808000 1 --2- 172.21.15.16:0/307116308 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5635ec73f2d0 0x5635ec7aa040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:06.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.383+0000 7f304d808000 1 --2- 172.21.15.16:0/307116308 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5635ec6f29b0 0x5635ec7a9b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:06.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.383+0000 7f304d808000 1 -- 172.21.15.16:0/307116308 >> 172.21.15.16:0/307116308 conn(0x5635ec720070 msgr2=0x5635ec73d380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:06:06.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.383+0000 7f304d808000 1 -- 172.21.15.16:0/307116308 shutdown_connections 2024-03-31T15:06:06.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.383+0000 7f304d808000 1 -- 172.21.15.16:0/307116308 wait complete. 2024-03-31T15:06:06.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.387+0000 7f304d808000 1 -- start start 2024-03-31T15:06:06.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.387+0000 7f304d808000 1 Processor -- start 2024-03-31T15:06:06.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.387+0000 7f304d808000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5635ec6f29b0 0x5635ec7ae7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:06.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.387+0000 7f304d808000 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5635ec73f2d0 0x5635ec7aed10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:06.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.387+0000 7f304d808000 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5635ec73fbf0 0x5635ec7924b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:06.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.387+0000 7f304d808000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5635ec543e10 con 0x5635ec6f29b0 2024-03-31T15:06:06.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.387+0000 7f304d808000 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x5635ec613420 con 0x5635ec73f2d0 2024-03-31T15:06:06.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.387+0000 7f304d808000 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x5635ec544b00 con 0x5635ec73fbf0 2024-03-31T15:06:06.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.387+0000 7f304d808000 10 client.0 _ll_register_callbacks cb 0x7f302c014670 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T15:06:06.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.387+0000 7f3043400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5635ec6f29b0 0x5635ec7ae7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:06.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.387+0000 7f3043400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5635ec6f29b0 0x5635ec7ae7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40324/0 (socket says 172.21.15.16:40324) 2024-03-31T15:06:06.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.387+0000 7f3043400640 1 -- 172.21.15.16:0/647758036 learned_addr learned my addr 172.21.15.16:0/647758036 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:06:06.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.387+0000 7f3043400640 1 -- 172.21.15.16:0/647758036 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5635ec73fbf0 msgr2=0x5635ec7924b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:06:06.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.387+0000 7f303be00640 1 --2- 172.21.15.16:0/647758036 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5635ec73f2d0 0x5635ec7aed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:06.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.387+0000 7f3043400640 1 --2- 172.21.15.16:0/647758036 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5635ec73fbf0 0x5635ec7924b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:06.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.387+0000 7f3043400640 1 -- 172.21.15.16:0/647758036 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5635ec73f2d0 msgr2=0x5635ec7aed10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:06:06.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.387+0000 7f3043400640 1 --2- 172.21.15.16:0/647758036 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5635ec73f2d0 0x5635ec7aed10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:06.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.387+0000 7f3043400640 1 -- 172.21.15.16:0/647758036 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5635ec5c5140 con 0x5635ec6f29b0 2024-03-31T15:06:06.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.387+0000 7f3043400640 1 --2- 172.21.15.16:0/647758036 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5635ec6f29b0 0x5635ec7ae7d0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f303411ed80 tx=0x7f30340f7040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8381c7296c71aae server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:06:06.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.387+0000 7f3030c00640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-31T15:06:06.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.387+0000 7f3030c00640 1 -- 172.21.15.16:0/647758036 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30340f92f0 con 0x5635ec6f29b0 2024-03-31T15:06:06.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.387+0000 7f3030c00640 1 -- 172.21.15.16:0/647758036 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(45..45 src has 1..45) ==== 5709+0+0 (secure 0 0 0) 0x7f30340eeac0 con 0x5635ec6f29b0 2024-03-31T15:06:06.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.387+0000 7f304d808000 1 -- 172.21.15.16:0/647758036 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f303c0165c0 con 0x5635ec6f29b0 2024-03-31T15:06:06.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.387+0000 7f3030c00640 1 -- 172.21.15.16:0/647758036 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f30340ece30 con 0x5635ec6f29b0 2024-03-31T15:06:06.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.387+0000 7f304d808000 10 client.5937 fetch_fsmap learned FSMap version 25 2024-03-31T15:06:06.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.387+0000 7f304d808000 1 -- 172.21.15.16:0/647758036 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=2+}) -- 0x5635ec4d31e0 con 0x5635ec6f29b0 2024-03-31T15:06:06.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.387+0000 7f3030c00640 1 -- 172.21.15.16:0/647758036 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f303c0165c0 con 0x5635ec6f29b0 2024-03-31T15:06:06.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.387+0000 7f3030c00640 1 -- 172.21.15.16:0/647758036 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 25) ==== 3494+0+0 (secure 0 0 0) 0x7f30340b5030 con 0x5635ec6f29b0 2024-03-31T15:06:06.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.387+0000 7f304d808000 10 client.5937 fetch_fsmap finished waiting for FSMap version 25 2024-03-31T15:06:06.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.387+0000 7f304d808000 10 client.5937 resolve_mds: resolved 3:0 to role '3:0' aka daemon mds.a 2024-03-31T15:06:06.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.387+0000 7f304d808000 20 client.5937 populate_metadata read hostname 'smithi016' 2024-03-31T15:06:06.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.387+0000 7f304d808000 1 --2- 172.21.15.16:0/647758036 >> [v2:172.21.15.16:6832/2768973409,v1:172.21.15.16:6834/2768973409] conn(0x5635ec792e40 0x5635ec793250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:06.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.387+0000 7f304d808000 4 client.5937 mds_command: new command op to 5697 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-31T15:06:06.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.387+0000 7f304d808000 1 -- 172.21.15.16:0/647758036 --> [v2:172.21.15.16:6832/2768973409,v1:172.21.15.16:6834/2768973409] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5635ec552040 con 0x5635ec792e40 2024-03-31T15:06:06.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.387+0000 7f303be00640 1 --2- 172.21.15.16:0/647758036 >> [v2:172.21.15.16:6832/2768973409,v1:172.21.15.16:6834/2768973409] conn(0x5635ec792e40 0x5635ec793250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:06.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.387+0000 7f303be00640 1 --2- 172.21.15.16:0/647758036 >> [v2:172.21.15.16:6832/2768973409,v1:172.21.15.16:6834/2768973409] conn(0x5635ec792e40 0x5635ec793250 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9a0e77ab6f6dc3a4 server_cookie=e72b9dcea4600873 in_seq=0 out_seq=0 2024-03-31T15:06:06.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.387+0000 7f3030c00640 10 client.5937 ms_handle_connect on v2:172.21.15.16:6832/2768973409 2024-03-31T15:06:06.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.391+0000 7f3030c00640 1 -- 172.21.15.16:0/647758036 <== mds.0 v2:172.21.15.16:6832/2768973409 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x5635ec552040 con 0x5635ec792e40 2024-03-31T15:06:06.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.391+0000 7f3030c00640 10 client.5937 handle_command_reply: tid=0 2024-03-31T15:06:06.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.391+0000 7f304d808000 1 client.5937 shutdown 2024-03-31T15:06:06.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.391+0000 7f304d808000 1 -- 172.21.15.16:0/647758036 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5635ec6f29b0 msgr2=0x5635ec7ae7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:06.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.391+0000 7f304d808000 1 --2- 172.21.15.16:0/647758036 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5635ec6f29b0 0x5635ec7ae7d0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f303411ed80 tx=0x7f30340f7040 comp rx=0 tx=0).stop 2024-03-31T15:06:06.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.391+0000 7f304d808000 1 -- 172.21.15.16:0/647758036 shutdown_connections 2024-03-31T15:06:06.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.391+0000 7f304d808000 1 --2- 172.21.15.16:0/647758036 >> [v2:172.21.15.16:6832/2768973409,v1:172.21.15.16:6834/2768973409] conn(0x5635ec792e40 0x5635ec793250 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:06.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.391+0000 7f304d808000 1 --2- 172.21.15.16:0/647758036 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5635ec73fbf0 0x5635ec7924b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:06.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.391+0000 7f304d808000 1 --2- 172.21.15.16:0/647758036 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5635ec73f2d0 0x5635ec7aed10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:06.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.391+0000 7f3030c00640 0 client.5937 ms_handle_reset on v2:172.21.15.16:6832/2768973409 2024-03-31T15:06:06.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.391+0000 7f304d808000 1 --2- 172.21.15.16:0/647758036 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5635ec6f29b0 0x5635ec7ae7d0 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:06.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.391+0000 7f304d808000 1 -- 172.21.15.16:0/647758036 >> 172.21.15.16:0/647758036 conn(0x5635ec720070 msgr2=0x5635ec7537d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:06:06.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.391+0000 7f304d808000 1 -- 172.21.15.16:0/647758036 shutdown_connections 2024-03-31T15:06:06.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.391+0000 7f304d808000 1 -- 172.21.15.16:0/647758036 wait complete. 2024-03-31T15:06:06.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.391+0000 7f304d808000 20 client.5937 trim_cache size 0 max 16384 2024-03-31T15:06:06.390 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-03-31T15:06:06.390 INFO:teuthology.orchestra.run.smithi016.stdout: "status": "no active scrubs running", 2024-03-31T15:06:06.390 INFO:teuthology.orchestra.run.smithi016.stdout: "scrubs": {} 2024-03-31T15:06:06.390 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-03-31T15:06:06.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.391+0000 7f304a400640 1 -- 172.21.15.16:0/1749580697 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f3020068380 msgr2=0x7f302006a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:06.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.391+0000 7f304a400640 1 --2- 172.21.15.16:0/1749580697 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f3020068380 0x7f302006a840 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f304413f3f0 tx=0x7f302c009190 comp rx=0 tx=0).stop 2024-03-31T15:06:06.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.391+0000 7f304a400640 1 -- 172.21.15.16:0/1749580697 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f304406f160 msgr2=0x7f3044142550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:06.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.391+0000 7f304a400640 1 --2- 172.21.15.16:0/1749580697 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f304406f160 0x7f3044142550 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f30340b0040 tx=0x7f303401ada0 comp rx=0 tx=0).stop 2024-03-31T15:06:06.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.391+0000 7f304a400640 1 -- 172.21.15.16:0/1749580697 shutdown_connections 2024-03-31T15:06:06.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.391+0000 7f304a400640 1 --2- 172.21.15.16:0/1749580697 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f3020068380 0x7f302006a840 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:06.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.391+0000 7f304a400640 1 --2- 172.21.15.16:0/1749580697 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f304415c890 0x7f30441663a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:06.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.391+0000 7f304a400640 1 --2- 172.21.15.16:0/1749580697 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3044145d50 0x7f304413a9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:06.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.391+0000 7f304a400640 1 --2- 172.21.15.16:0/1749580697 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f304406f160 0x7f3044142550 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:06.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.391+0000 7f304a400640 1 -- 172.21.15.16:0/1749580697 >> 172.21.15.16:0/1749580697 conn(0x7f304406e670 msgr2=0x7f304413e530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:06:06.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.391+0000 7f304a400640 1 -- 172.21.15.16:0/1749580697 shutdown_connections 2024-03-31T15:06:06.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.391+0000 7f304a400640 1 -- 172.21.15.16:0/1749580697 wait complete. 2024-03-31T15:06:06.405 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-31T15:06:06.506 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-31T15:06:06.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.655+0000 7fb440000640 1 Processor -- start 2024-03-31T15:06:06.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.655+0000 7fb440000640 1 -- start start 2024-03-31T15:06:06.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.659+0000 7fb440000640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fb43806dbc0 0x7fb43815f650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:06.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.659+0000 7fb440000640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb4380cb440 0x7fb43806f920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:06.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.659+0000 7fb440000640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb4380cbe90 con 0x7fb4380cb8c0 2024-03-31T15:06:06.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.659+0000 7fb440000640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7fb43815fb90 con 0x7fb4380cb440 2024-03-31T15:06:06.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.659+0000 7fb440000640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7fb43815f1c0 con 0x7fb43806dbc0 2024-03-31T15:06:06.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.659+0000 7fb43cc00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb4380cb440 0x7fb43806f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:06.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.659+0000 7fb43cc00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb4380cb440 0x7fb43806f920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40334/0 (socket says 172.21.15.16:40334) 2024-03-31T15:06:06.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.659+0000 7fb43cc00640 1 -- 172.21.15.16:0/1319641082 learned_addr learned my addr 172.21.15.16:0/1319641082 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:06:06.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.659+0000 7fb43d600640 1 --2- 172.21.15.16:0/1319641082 >> v2:172.21.15.89:3300/0 conn(0x7fb43806dbc0 0x7fb43815f650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:06.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.659+0000 7fb43cc00640 1 -- 172.21.15.16:0/1319641082 >> v2:172.21.15.89:3300/0 conn(0x7fb43806dbc0 msgr2=0x7fb43815f650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:06:06.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.659+0000 7fb43cc00640 1 --2- 172.21.15.16:0/1319641082 >> v2:172.21.15.89:3300/0 conn(0x7fb43806dbc0 0x7fb43815f650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:06.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.659+0000 7fb43cc00640 1 -- 172.21.15.16:0/1319641082 >> v1:172.21.15.89:6789/0 conn(0x7fb4380cb8c0 legacy=0x7fb438162180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:06:06.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.659+0000 7fb43cc00640 1 -- 172.21.15.16:0/1319641082 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb4380cc170 con 0x7fb4380cb440 2024-03-31T15:06:06.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.663+0000 7fb43cc00640 1 --2- 172.21.15.16:0/1319641082 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb4380cb440 0x7fb43806f920 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7fb4380b8380 tx=0x7fb42401c420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0786fa083f31e48 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:06:06.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.663+0000 7fb437e00640 1 -- 172.21.15.16:0/1319641082 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4240b0070 con 0x7fb4380cb440 2024-03-31T15:06:06.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.663+0000 7fb437e00640 1 -- 172.21.15.16:0/1319641082 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb4240ab600 con 0x7fb4380cb440 2024-03-31T15:06:06.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.663+0000 7fb437e00640 1 -- 172.21.15.16:0/1319641082 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4240ab9d0 con 0x7fb4380cb440 2024-03-31T15:06:06.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.663+0000 7fb440000640 1 -- 172.21.15.16:0/1319641082 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb4380cb440 msgr2=0x7fb43806f920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:06.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.663+0000 7fb440000640 1 --2- 172.21.15.16:0/1319641082 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb4380cb440 0x7fb43806f920 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7fb4380b8380 tx=0x7fb42401c420 comp rx=0 tx=0).stop 2024-03-31T15:06:06.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.663+0000 7fb440000640 1 -- 172.21.15.16:0/1319641082 shutdown_connections 2024-03-31T15:06:06.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.663+0000 7fb440000640 1 --2- 172.21.15.16:0/1319641082 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb4380cb440 0x7fb43806f920 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:06.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.663+0000 7fb440000640 1 --2- 172.21.15.16:0/1319641082 >> v2:172.21.15.89:3300/0 conn(0x7fb43806dbc0 0x7fb43815f650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:06.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.663+0000 7fb440000640 1 -- 172.21.15.16:0/1319641082 >> 172.21.15.16:0/1319641082 conn(0x7fb438067330 msgr2=0x7fb438067740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:06:06.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.663+0000 7fb440000640 1 -- 172.21.15.16:0/1319641082 shutdown_connections 2024-03-31T15:06:06.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.663+0000 7fb440000640 1 -- 172.21.15.16:0/1319641082 wait complete. 2024-03-31T15:06:06.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.663+0000 7fb440000640 1 Processor -- start 2024-03-31T15:06:06.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.663+0000 7fb440000640 1 -- start start 2024-03-31T15:06:06.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.663+0000 7fb440000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb43806dbc0 0x7fb4381525e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:06.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.663+0000 7fb440000640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb4380cb440 0x7fb43814c900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:06.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.663+0000 7fb440000640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb4380cb8c0 0x7fb4381532f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:06.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.663+0000 7fb440000640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fb43806c950 con 0x7fb43806dbc0 2024-03-31T15:06:06.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.663+0000 7fb440000640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fb438162ca0 con 0x7fb4380cb8c0 2024-03-31T15:06:06.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.663+0000 7fb440000640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fb43806cad0 con 0x7fb4380cb440 2024-03-31T15:06:06.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.667+0000 7fb43d600640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb43806dbc0 0x7fb4381525e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:06.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.667+0000 7fb43d600640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb43806dbc0 0x7fb4381525e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40348/0 (socket says 172.21.15.16:40348) 2024-03-31T15:06:06.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.667+0000 7fb43d600640 1 -- 172.21.15.16:0/3524530298 learned_addr learned my addr 172.21.15.16:0/3524530298 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:06:06.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.667+0000 7fb43cc00640 1 --2- 172.21.15.16:0/3524530298 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb4380cb440 0x7fb43814c900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:06.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.667+0000 7fb43e000640 1 --2- 172.21.15.16:0/3524530298 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb4380cb8c0 0x7fb4381532f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:06.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.667+0000 7fb43d600640 1 -- 172.21.15.16:0/3524530298 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb4380cb440 msgr2=0x7fb43814c900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:06.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.667+0000 7fb43d600640 1 --2- 172.21.15.16:0/3524530298 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb4380cb440 0x7fb43814c900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:06.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.667+0000 7fb43d600640 1 -- 172.21.15.16:0/3524530298 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb4380cb8c0 msgr2=0x7fb4381532f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:06.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.667+0000 7fb43d600640 1 --2- 172.21.15.16:0/3524530298 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb4380cb8c0 0x7fb4381532f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:06.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.667+0000 7fb43d600640 1 -- 172.21.15.16:0/3524530298 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb438152b20 con 0x7fb43806dbc0 2024-03-31T15:06:06.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.667+0000 7fb43cc00640 1 --2- 172.21.15.16:0/3524530298 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb4380cb440 0x7fb43814c900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:06:06.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.667+0000 7fb43d600640 1 --2- 172.21.15.16:0/3524530298 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb43806dbc0 0x7fb4381525e0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7fb42c00d5f0 tx=0x7fb42c00dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:06:06.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.667+0000 7fb435600640 1 -- 172.21.15.16:0/3524530298 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb42c0125f0 con 0x7fb43806dbc0 2024-03-31T15:06:06.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.667+0000 7fb435600640 1 -- 172.21.15.16:0/3524530298 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb42c019070 con 0x7fb43806dbc0 2024-03-31T15:06:06.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.667+0000 7fb440000640 1 -- 172.21.15.16:0/3524530298 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb4380b72a0 con 0x7fb43806dbc0 2024-03-31T15:06:06.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.667+0000 7fb435600640 1 -- 172.21.15.16:0/3524530298 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb42c0129b0 con 0x7fb43806dbc0 2024-03-31T15:06:06.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.667+0000 7fb440000640 1 -- 172.21.15.16:0/3524530298 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb4380b7860 con 0x7fb43806dbc0 2024-03-31T15:06:06.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.667+0000 7fb440000640 1 -- 172.21.15.16:0/3524530298 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb438069950 con 0x7fb43806dbc0 2024-03-31T15:06:06.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.667+0000 7fb435600640 1 -- 172.21.15.16:0/3524530298 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fb42c012b50 con 0x7fb43806dbc0 2024-03-31T15:06:06.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.671+0000 7fb435600640 1 --2- 172.21.15.16:0/3524530298 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fb418068380 0x7fb41806a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:06.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.671+0000 7fb435600640 1 -- 172.21.15.16:0/3524530298 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5709+0+0 (secure 0 0 0) 0x7fb42c086df0 con 0x7fb43806dbc0 2024-03-31T15:06:06.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.671+0000 7fb43cc00640 1 --2- 172.21.15.16:0/3524530298 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fb418068380 0x7fb41806a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:06.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.671+0000 7fb43cc00640 1 --2- 172.21.15.16:0/3524530298 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fb418068380 0x7fb41806a840 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fb4240ad000 tx=0x7fb424008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:06:06.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.679+0000 7fb435600640 1 -- 172.21.15.16:0/3524530298 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fb42c011340 con 0x7fb43806dbc0 2024-03-31T15:06:06.851 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-31T15:06:06.854 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-31T15:06:06.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.875+0000 7fb440000640 1 -- 172.21.15.16:0/3524530298 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb4380cf380 con 0x7fb43806dbc0 2024-03-31T15:06:06.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.879+0000 7fb435600640 1 -- 172.21.15.16:0/3524530298 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+5175 (secure 0 0 0) 0x7fb42c0573a0 con 0x7fb43806dbc0 2024-03-31T15:06:06.872 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:06:06.872 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":25,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/285985105","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6835","nonce":285985105},{"type":"v1","addr":"172.21.15.89:6837","nonce":285985105}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20},{"gid":5689,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6836/1615931247","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":1615931247},{"type":"v1","addr":"172.21.15.89:6836","nonce":1615931247}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20},{"gid":5694,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/512187852","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6833","nonce":512187852},{"type":"v1","addr":"172.21.15.16:6835","nonce":512187852}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-31T15:05:26.461908+0000","modified":"2024-03-31T15:05:29.491163+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5697},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5697":{"gid":5697,"name":"a","rank":0,"incarnation":21,"state":"up:active","state_seq":3,"addr":"172.21.15.16:6834/2768973409","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":2768973409},{"type":"v1","addr":"172.21.15.16:6834","nonce":2768973409}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5697,"qdb_cluster":[5697]},"id":3}]} 2024-03-31T15:06:06.873 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 25 2024-03-31T15:06:06.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.879+0000 7fb440000640 1 -- 172.21.15.16:0/3524530298 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fb418068380 msgr2=0x7fb41806a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:06.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.879+0000 7fb440000640 1 --2- 172.21.15.16:0/3524530298 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fb418068380 0x7fb41806a840 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fb4240ad000 tx=0x7fb424008040 comp rx=0 tx=0).stop 2024-03-31T15:06:06.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.879+0000 7fb440000640 1 -- 172.21.15.16:0/3524530298 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb43806dbc0 msgr2=0x7fb4381525e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:06.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.879+0000 7fb440000640 1 --2- 172.21.15.16:0/3524530298 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb43806dbc0 0x7fb4381525e0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7fb42c00d5f0 tx=0x7fb42c00dab0 comp rx=0 tx=0).stop 2024-03-31T15:06:06.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.879+0000 7fb440000640 1 -- 172.21.15.16:0/3524530298 shutdown_connections 2024-03-31T15:06:06.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.879+0000 7fb440000640 1 --2- 172.21.15.16:0/3524530298 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fb418068380 0x7fb41806a840 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:06.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.879+0000 7fb440000640 1 --2- 172.21.15.16:0/3524530298 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb4380cb8c0 0x7fb4381532f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:06.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.879+0000 7fb440000640 1 --2- 172.21.15.16:0/3524530298 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb4380cb440 0x7fb43814c900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:06.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.879+0000 7fb440000640 1 --2- 172.21.15.16:0/3524530298 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb43806dbc0 0x7fb4381525e0 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:06.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.879+0000 7fb440000640 1 -- 172.21.15.16:0/3524530298 >> 172.21.15.16:0/3524530298 conn(0x7fb438067330 msgr2=0x7fb4380d0800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:06:06.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.879+0000 7fb440000640 1 -- 172.21.15.16:0/3524530298 shutdown_connections 2024-03-31T15:06:06.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:06.879+0000 7fb440000640 1 -- 172.21.15.16:0/3524530298 wait complete. 2024-03-31T15:06:06.894 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json flush journal 2024-03-31T15:06:07.138 INFO:teuthology.orchestra.run.smithi016.stdout:{"message":"","return_code":0} 2024-03-31T15:06:07.139 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2024-03-31T15:06:07.139 DEBUG:teuthology.orchestra.run.smithi016:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_data getxattr 10000000001.00000000 parent 2024-03-31T15:06:07.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:07.191+0000 7f83f0472080 1 Processor -- start 2024-03-31T15:06:07.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:07.191+0000 7f83f0472080 1 -- start start 2024-03-31T15:06:07.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:07.191+0000 7f83f0472080 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55eeccd8de70 0x55eecce328a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:07.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:07.191+0000 7f83f0472080 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55eeccdabc60 0x55eeccd8fe30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:07.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:07.191+0000 7f83f0472080 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55eeccdac040 0x55eeccd90370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:07.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:07.191+0000 7f83f0472080 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55eeccd91e20 con 0x55eeccdabc60 2024-03-31T15:06:07.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:07.191+0000 7f83f0472080 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55eeccd044b0 con 0x55eeccd8de70 2024-03-31T15:06:07.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:07.191+0000 7f83f0472080 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x55eeccd91fa0 con 0x55eeccdac040 2024-03-31T15:06:07.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:07.191+0000 7f83ecc00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55eeccdabc60 0x55eeccd8fe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:07.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:07.191+0000 7f83ecc00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55eeccdabc60 0x55eeccd8fe30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35002/0 (socket says 172.21.15.16:35002) 2024-03-31T15:06:07.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:07.191+0000 7f83ecc00640 1 -- 172.21.15.16:0/1845241635 learned_addr learned my addr 172.21.15.16:0/1845241635 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:06:07.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:07.191+0000 7f83ed600640 1 --2- 172.21.15.16:0/1845241635 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55eeccd8de70 0x55eecce328a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:07.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:07.191+0000 7f83ecc00640 1 -- 172.21.15.16:0/1845241635 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55eeccdac040 msgr2=0x55eeccd90370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:06:07.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:07.191+0000 7f83ecc00640 1 --2- 172.21.15.16:0/1845241635 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55eeccdac040 0x55eeccd90370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:07.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:07.191+0000 7f83ecc00640 1 -- 172.21.15.16:0/1845241635 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55eeccd8de70 msgr2=0x55eecce328a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:06:07.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:07.191+0000 7f83ecc00640 1 --2- 172.21.15.16:0/1845241635 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55eeccd8de70 0x55eecce328a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:07.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:07.191+0000 7f83ecc00640 1 -- 172.21.15.16:0/1845241635 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55eecce313a0 con 0x55eeccdabc60 2024-03-31T15:06:07.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:07.191+0000 7f83ed600640 1 --2- 172.21.15.16:0/1845241635 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55eeccd8de70 0x55eecce328a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:06:07.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:07.191+0000 7f83ecc00640 1 --2- 172.21.15.16:0/1845241635 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55eeccdabc60 0x55eeccd8fe30 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f83dc018ea0 tx=0x7f83dc003030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8384077113fb4c1c server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:06:07.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:07.195+0000 7f83e7400640 1 -- 172.21.15.16:0/1845241635 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83dc0af040 con 0x55eeccdabc60 2024-03-31T15:06:07.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:07.195+0000 7f83e7400640 1 -- 172.21.15.16:0/1845241635 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f83dc0b9070 con 0x55eeccdabc60 2024-03-31T15:06:07.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:07.195+0000 7f83e7400640 1 -- 172.21.15.16:0/1845241635 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83dc0aae50 con 0x55eeccdabc60 2024-03-31T15:06:07.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:07.195+0000 7f83f0472080 1 -- 172.21.15.16:0/1845241635 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55eeccdabc60 msgr2=0x55eeccd8fe30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:07.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:07.195+0000 7f83f0472080 1 --2- 172.21.15.16:0/1845241635 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55eeccdabc60 0x55eeccd8fe30 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f83dc018ea0 tx=0x7f83dc003030 comp rx=0 tx=0).stop 2024-03-31T15:06:07.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:07.195+0000 7f83f0472080 1 -- 172.21.15.16:0/1845241635 shutdown_connections 2024-03-31T15:06:07.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:07.195+0000 7f83f0472080 1 --2- 172.21.15.16:0/1845241635 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55eeccdac040 0x55eeccd90370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:07.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:07.195+0000 7f83f0472080 1 --2- 172.21.15.16:0/1845241635 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55eeccdabc60 0x55eeccd8fe30 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:07.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:07.195+0000 7f83f0472080 1 --2- 172.21.15.16:0/1845241635 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55eeccd8de70 0x55eecce328a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:07.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:07.195+0000 7f83f0472080 1 -- 172.21.15.16:0/1845241635 >> 172.21.15.16:0/1845241635 conn(0x55eeccd8c420 msgr2=0x55eecce347b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:06:07.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:07.195+0000 7f83f0472080 1 -- 172.21.15.16:0/1845241635 shutdown_connections 2024-03-31T15:06:07.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:07.195+0000 7f83f0472080 1 -- 172.21.15.16:0/1845241635 wait complete. 2024-03-31T15:06:07.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:07.195+0000 7f83f0472080 1 Processor -- start 2024-03-31T15:06:07.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:07.195+0000 7f83f0472080 1 -- start start 2024-03-31T15:06:07.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:07.195+0000 7f83f0472080 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55eeccd8de70 0x55eecce166e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:07.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:07.195+0000 7f83f0472080 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55eeccdabc60 0x55eecce30380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:07.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:07.195+0000 7f83f0472080 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55eeccdac040 0x55eecce308c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:07.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:07.195+0000 7f83f0472080 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55eeccda20f0 con 0x55eeccdac040 2024-03-31T15:06:07.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:07.195+0000 7f83f0472080 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55eeccda1e50 con 0x55eeccdabc60 2024-03-31T15:06:07.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:07.195+0000 7f83f0472080 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x55eeccda2390 con 0x55eeccd8de70 2024-03-31T15:06:07.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:07.195+0000 7f83ee000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55eeccdac040 0x55eecce308c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:07.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:07.195+0000 7f83ed600640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55eeccd8de70 0x55eecce166e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:07.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:07.195+0000 7f83ee000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55eeccdac040 0x55eecce308c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35012/0 (socket says 172.21.15.16:35012) 2024-03-31T15:06:07.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:07.195+0000 7f83ee000640 1 -- 172.21.15.16:0/3357355305 learned_addr learned my addr 172.21.15.16:0/3357355305 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:06:07.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:07.195+0000 7f83ecc00640 1 --2- 172.21.15.16:0/3357355305 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55eeccdabc60 0x55eecce30380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:07.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:07.195+0000 7f83ee000640 1 -- 172.21.15.16:0/3357355305 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55eeccd8de70 msgr2=0x55eecce166e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:07.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:07.195+0000 7f83ee000640 1 --2- 172.21.15.16:0/3357355305 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55eeccd8de70 0x55eecce166e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:07.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:07.195+0000 7f83ee000640 1 -- 172.21.15.16:0/3357355305 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55eeccdabc60 msgr2=0x55eecce30380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:07.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:07.195+0000 7f83ee000640 1 --2- 172.21.15.16:0/3357355305 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55eeccdabc60 0x55eecce30380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:07.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:07.195+0000 7f83ee000640 1 -- 172.21.15.16:0/3357355305 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55eecce18d80 con 0x55eeccdac040 2024-03-31T15:06:07.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:07.195+0000 7f83ee000640 1 --2- 172.21.15.16:0/3357355305 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55eeccdac040 0x55eecce308c0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f83e800a1c0 tx=0x7f83e800a680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:06:07.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:07.199+0000 7f83e6000640 1 -- 172.21.15.16:0/3357355305 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83e80115f0 con 0x55eeccdac040 2024-03-31T15:06:07.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:07.199+0000 7f83e6000640 1 -- 172.21.15.16:0/3357355305 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f83e8018070 con 0x55eeccdac040 2024-03-31T15:06:07.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:07.199+0000 7f83e6000640 1 -- 172.21.15.16:0/3357355305 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83e80119d0 con 0x55eeccdac040 2024-03-31T15:06:07.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:07.199+0000 7f83f0472080 1 -- 172.21.15.16:0/3357355305 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55eecce1a730 con 0x55eeccdac040 2024-03-31T15:06:07.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:07.199+0000 7f83f0472080 1 -- 172.21.15.16:0/3357355305 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55eecce1ac70 con 0x55eeccdac040 2024-03-31T15:06:07.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:07.199+0000 7f83e6000640 1 -- 172.21.15.16:0/3357355305 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f83e8011b70 con 0x55eeccdac040 2024-03-31T15:06:07.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:07.199+0000 7f83e6000640 1 --2- 172.21.15.16:0/3357355305 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f83d0066d90 0x7f83d0069250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:07.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:07.199+0000 7f83e6000640 1 -- 172.21.15.16:0/3357355305 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5709+0+0 (secure 0 0 0) 0x7f83e8085760 con 0x55eeccdac040 2024-03-31T15:06:07.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:07.199+0000 7f83f0472080 1 --2- 172.21.15.16:0/3357355305 >> [v2:172.21.15.89:6809/3470471752,v1:172.21.15.89:6812/3470471752] conn(0x55eecce1b110 0x55eecce1b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:07.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:07.199+0000 7f83f0472080 1 -- 172.21.15.16:0/3357355305 --> [v2:172.21.15.89:6809/3470471752,v1:172.21.15.89:6812/3470471752] -- osd_op(unknown.0.0:1 7.7 7:e2e07930:::10000000001.00000000:head [getxattr parent in=6b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e45) -- 0x55eecce1ba30 con 0x55eecce1b110 2024-03-31T15:06:07.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:07.199+0000 7f83ed600640 1 --2- 172.21.15.16:0/3357355305 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f83d0066d90 0x7f83d0069250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:07.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:07.199+0000 7f83ecc00640 1 --2- 172.21.15.16:0/3357355305 >> [v2:172.21.15.89:6809/3470471752,v1:172.21.15.89:6812/3470471752] conn(0x55eecce1b110 0x55eecce1b4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:07.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:07.199+0000 7f83ed600640 1 --2- 172.21.15.16:0/3357355305 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f83d0066d90 0x7f83d0069250 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f83e0000f70 tx=0x7f83e0001760 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:06:07.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:07.203+0000 7f83ecc00640 1 --2- 172.21.15.16:0/3357355305 >> [v2:172.21.15.89:6809/3470471752,v1:172.21.15.89:6812/3470471752] conn(0x55eecce1b110 0x55eecce1b4f0 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:06:07.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:07.203+0000 7f83ecc00640 1 -- 172.21.15.16:0/3357355305 <== osd.6 v2:172.21.15.89:6809/3470471752 1 ==== osd_op_reply(1 10000000001.00000000 [getxattr (95) out=95b] v0'0 uv3 ondisk = 0) ==== 164+0+95 (crc 0 0 0) 0x7f83dc0aad70 con 0x55eecce1b110 2024-03-31T15:06:07.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:07.203+0000 7f83f0472080 1 -- 172.21.15.16:0/3357355305 >> [v2:172.21.15.89:6809/3470471752,v1:172.21.15.89:6812/3470471752] conn(0x55eecce1b110 msgr2=0x55eecce1b4f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:07.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:07.203+0000 7f83f0472080 1 --2- 172.21.15.16:0/3357355305 >> [v2:172.21.15.89:6809/3470471752,v1:172.21.15.89:6812/3470471752] conn(0x55eecce1b110 0x55eecce1b4f0 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:07.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:07.203+0000 7f83f0472080 1 -- 172.21.15.16:0/3357355305 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f83d0066d90 msgr2=0x7f83d0069250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:07.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:07.203+0000 7f83f0472080 1 --2- 172.21.15.16:0/3357355305 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f83d0066d90 0x7f83d0069250 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f83e0000f70 tx=0x7f83e0001760 comp rx=0 tx=0).stop 2024-03-31T15:06:07.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:07.203+0000 7f83f0472080 1 -- 172.21.15.16:0/3357355305 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55eeccdac040 msgr2=0x55eecce308c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:07.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:07.203+0000 7f83f0472080 1 --2- 172.21.15.16:0/3357355305 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55eeccdac040 0x55eecce308c0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f83e800a1c0 tx=0x7f83e800a680 comp rx=0 tx=0).stop 2024-03-31T15:06:07.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:07.203+0000 7f83ee000640 1 -- 172.21.15.16:0/3357355305 reap_dead start 2024-03-31T15:06:07.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:07.203+0000 7f83f0472080 1 -- 172.21.15.16:0/3357355305 shutdown_connections 2024-03-31T15:06:07.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:07.203+0000 7f83f0472080 1 -- 172.21.15.16:0/3357355305 >> 172.21.15.16:0/3357355305 conn(0x55eeccd8c420 msgr2=0x55eecce2c9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:06:07.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:07.203+0000 7f83f0472080 1 -- 172.21.15.16:0/3357355305 shutdown_connections 2024-03-31T15:06:07.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:07.203+0000 7f83f0472080 1 -- 172.21.15.16:0/3357355305 wait complete. 2024-03-31T15:06:07.201 INFO:teuthology.orchestra.run.smithi016.stdout:Yalphaparent_a 2024-03-31T15:06:07.201 DEBUG:teuthology.orchestra.run.smithi016:> ceph-dencoder type inode_backtrace_t import - decode dump_json 2024-03-31T15:06:07.292 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-03-31T15:06:07.292 INFO:teuthology.orchestra.run.smithi016.stdout: "ino": 1099511627777, 2024-03-31T15:06:07.292 INFO:teuthology.orchestra.run.smithi016.stdout: "ancestors": [ 2024-03-31T15:06:07.292 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-31T15:06:07.293 INFO:teuthology.orchestra.run.smithi016.stdout: "dirino": 1099511627776, 2024-03-31T15:06:07.293 INFO:teuthology.orchestra.run.smithi016.stdout: "dname": "alpha", 2024-03-31T15:06:07.293 INFO:teuthology.orchestra.run.smithi016.stdout: "version": 4 2024-03-31T15:06:07.293 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-31T15:06:07.293 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-31T15:06:07.293 INFO:teuthology.orchestra.run.smithi016.stdout: "dirino": 1, 2024-03-31T15:06:07.293 INFO:teuthology.orchestra.run.smithi016.stdout: "dname": "parent_a", 2024-03-31T15:06:07.293 INFO:teuthology.orchestra.run.smithi016.stdout: "version": 7 2024-03-31T15:06:07.293 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-31T15:06:07.293 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-03-31T15:06:07.293 INFO:teuthology.orchestra.run.smithi016.stdout: "pool": 7, 2024-03-31T15:06:07.293 INFO:teuthology.orchestra.run.smithi016.stdout: "old_pools": [] 2024-03-31T15:06:07.293 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-03-31T15:06:07.293 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:06:07.298 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-31T15:06:07.298 DEBUG:teuthology.orchestra.run.smithi016:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-31T15:06:07.414 DEBUG:teuthology.orchestra.run.smithi089:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-31T15:06:07.513 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-31T15:06:07.514 INFO:tasks.cephfs.mount:Terminating background process 2024-03-31T15:06:07.514 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-03-31T15:06:07.514 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-31T15:06:07.523 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-31T15:06:07.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:07.711+0000 7f5f4ec00640 1 Processor -- start 2024-03-31T15:06:07.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:07.711+0000 7f5f4ec00640 1 -- start start 2024-03-31T15:06:07.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:07.711+0000 7f5f4ec00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f5f480cd4f0 0x7f5f480b44e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:07.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:07.711+0000 7f5f4ec00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f5f4815fd40 0x7f5f481601b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:07.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:07.711+0000 7f5f4ec00640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5f480cd980 con 0x7f5f4815c1d0 2024-03-31T15:06:07.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:07.711+0000 7f5f4ec00640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f5f481606f0 con 0x7f5f4815fd40 2024-03-31T15:06:07.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:07.711+0000 7f5f4ec00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f5f480cfaa0 con 0x7f5f480cd4f0 2024-03-31T15:06:07.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:07.711+0000 7f5f4d800640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f5f4815c1d0 0x7f5f4806f5d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:37118/0 (socket says 172.21.15.16:37118) 2024-03-31T15:06:07.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:07.711+0000 7f5f4d800640 1 -- 172.21.15.16:0/1559668730 learned_addr learned my addr 172.21.15.16:0/1559668730 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:06:07.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:07.711+0000 7f5f4e200640 1 --2- 172.21.15.16:0/1559668730 >> v2:172.21.15.89:3300/0 conn(0x7f5f4815fd40 0x7f5f481601b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:07.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:07.711+0000 7f5f4ce00640 1 --2- 172.21.15.16:0/1559668730 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f5f480cd4f0 0x7f5f480b44e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:07.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:07.711+0000 7f5f47e00640 1 -- 172.21.15.16:0/1559668730 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1786666440 0 0) 0x7f5f480cd980 con 0x7f5f4815c1d0 2024-03-31T15:06:07.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:07.711+0000 7f5f47e00640 1 -- 172.21.15.16:0/1559668730 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5f30003530 con 0x7f5f4815c1d0 2024-03-31T15:06:07.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:07.711+0000 7f5f47e00640 1 -- 172.21.15.16:0/1559668730 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2606567381 0 0) 0x7f5f30003530 con 0x7f5f4815c1d0 2024-03-31T15:06:07.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:07.711+0000 7f5f47e00640 1 -- 172.21.15.16:0/1559668730 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f5f480cd4f0 msgr2=0x7f5f480b44e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:06:07.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:07.711+0000 7f5f47e00640 1 --2- 172.21.15.16:0/1559668730 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f5f480cd4f0 0x7f5f480b44e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:07.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:07.711+0000 7f5f47e00640 1 -- 172.21.15.16:0/1559668730 >> v2:172.21.15.89:3300/0 conn(0x7f5f4815fd40 msgr2=0x7f5f481601b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:06:07.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:07.711+0000 7f5f47e00640 1 --2- 172.21.15.16:0/1559668730 >> v2:172.21.15.89:3300/0 conn(0x7f5f4815fd40 0x7f5f481601b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:07.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:07.711+0000 7f5f47e00640 1 -- 172.21.15.16:0/1559668730 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5f480cdc60 con 0x7f5f4815c1d0 2024-03-31T15:06:07.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:07.711+0000 7f5f47e00640 1 -- 172.21.15.16:0/1559668730 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 214807623 0 0) 0x7f5f3c001680 con 0x7f5f4815c1d0 2024-03-31T15:06:07.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:07.711+0000 7f5f47e00640 1 -- 172.21.15.16:0/1559668730 >> v1:172.21.15.16:6789/0 conn(0x7f5f4815c1d0 legacy=0x7f5f4806f5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:07.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:07.711+0000 7f5f47e00640 1 --2- 172.21.15.16:0/1559668730 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5f300039c0 0x7f5f30003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:07.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:07.711+0000 7f5f47e00640 1 --2- 172.21.15.16:0/1559668730 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f5f30004810 0x7f5f30004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:07.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:07.711+0000 7f5f47e00640 1 --2- 172.21.15.16:0/1559668730 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5f30005190 0x7f5f3000b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:07.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:07.711+0000 7f5f47e00640 1 -- 172.21.15.16:0/1559668730 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f5f481606f0 con 0x7f5f300039c0 2024-03-31T15:06:07.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:07.711+0000 7f5f47e00640 1 -- 172.21.15.16:0/1559668730 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f5f480cfaa0 con 0x7f5f30005190 2024-03-31T15:06:07.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:07.711+0000 7f5f47e00640 1 -- 172.21.15.16:0/1559668730 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f5f30004610 con 0x7f5f30004810 2024-03-31T15:06:07.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:07.711+0000 7f5f4d800640 1 --2- 172.21.15.16:0/1559668730 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5f300039c0 0x7f5f30003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:07.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:07.711+0000 7f5f4ce00640 1 --2- 172.21.15.16:0/1559668730 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f5f30004810 0x7f5f30004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:07.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:07.711+0000 7f5f4e200640 1 --2- 172.21.15.16:0/1559668730 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5f30005190 0x7f5f3000b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:07.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:07.711+0000 7f5f4d800640 1 -- 172.21.15.16:0/1559668730 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f5f30004810 msgr2=0x7f5f30004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:06:07.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:07.711+0000 7f5f4d800640 1 --2- 172.21.15.16:0/1559668730 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f5f30004810 0x7f5f30004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:07.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:07.711+0000 7f5f4d800640 1 -- 172.21.15.16:0/1559668730 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5f30005190 msgr2=0x7f5f3000b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:06:07.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:07.711+0000 7f5f4d800640 1 --2- 172.21.15.16:0/1559668730 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5f30005190 0x7f5f3000b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:07.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:07.711+0000 7f5f4d800640 1 -- 172.21.15.16:0/1559668730 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5f30003530 con 0x7f5f300039c0 2024-03-31T15:06:07.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:07.715+0000 7f5f4d800640 1 --2- 172.21.15.16:0/1559668730 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5f300039c0 0x7f5f30003de0 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f5f3c00c240 tx=0x7f5f3c01cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15947020b249a454 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:06:07.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:07.715+0000 7f5f47e00640 1 -- 172.21.15.16:0/1559668730 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f3c007d60 con 0x7f5f300039c0 2024-03-31T15:06:07.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:07.715+0000 7f5f47e00640 1 -- 172.21.15.16:0/1559668730 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5f3c0a9020 con 0x7f5f300039c0 2024-03-31T15:06:07.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:07.715+0000 7f5f47e00640 1 -- 172.21.15.16:0/1559668730 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f3c005ae0 con 0x7f5f300039c0 2024-03-31T15:06:07.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:07.715+0000 7f5f4ec00640 1 -- 172.21.15.16:0/1559668730 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5f300039c0 msgr2=0x7f5f30003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:07.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:07.715+0000 7f5f4ec00640 1 --2- 172.21.15.16:0/1559668730 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5f300039c0 0x7f5f30003de0 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f5f3c00c240 tx=0x7f5f3c01cc90 comp rx=0 tx=0).stop 2024-03-31T15:06:07.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:07.715+0000 7f5f4e200640 1 -- 172.21.15.16:0/1559668730 reap_dead start 2024-03-31T15:06:07.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:07.715+0000 7f5f4ec00640 1 -- 172.21.15.16:0/1559668730 shutdown_connections 2024-03-31T15:06:07.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:07.715+0000 7f5f4ec00640 1 -- 172.21.15.16:0/1559668730 >> 172.21.15.16:0/1559668730 conn(0x7f5f480b39c0 msgr2=0x7f5f480b3dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:06:07.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:07.715+0000 7f5f4ec00640 1 -- 172.21.15.16:0/1559668730 shutdown_connections 2024-03-31T15:06:07.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:07.715+0000 7f5f4ec00640 1 -- 172.21.15.16:0/1559668730 wait complete. 2024-03-31T15:06:07.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:07.719+0000 7f5f4ec00640 1 Processor -- start 2024-03-31T15:06:07.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:07.719+0000 7f5f4ec00640 1 -- start start 2024-03-31T15:06:07.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:07.719+0000 7f5f4ec00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f5f300039c0 0x7f5f4806d290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:07.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:07.719+0000 7f5f4ec00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5f480ca330 0x7f5f480ca7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:07.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:07.719+0000 7f5f4ec00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5f480b5630 0x7f5f480b5ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:07.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:07.719+0000 7f5f4ec00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f5f4806fce0 con 0x7f5f480ca330 2024-03-31T15:06:07.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:07.719+0000 7f5f4ec00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f5f480b1420 con 0x7f5f480b5630 2024-03-31T15:06:07.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:07.719+0000 7f5f4ec00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f5f480cbe50 con 0x7f5f300039c0 2024-03-31T15:06:07.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:07.719+0000 7f5f4ce00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5f480ca330 0x7f5f480ca7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:07.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:07.719+0000 7f5f4ce00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5f480ca330 0x7f5f480ca7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35024/0 (socket says 172.21.15.16:35024) 2024-03-31T15:06:07.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:07.719+0000 7f5f4ce00640 1 -- 172.21.15.16:0/2061436204 learned_addr learned my addr 172.21.15.16:0/2061436204 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:06:07.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:07.719+0000 7f5f4d800640 1 --2- 172.21.15.16:0/2061436204 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f5f300039c0 0x7f5f4806d290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:07.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:07.719+0000 7f5f4e200640 1 --2- 172.21.15.16:0/2061436204 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5f480b5630 0x7f5f480b5ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:07.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:07.719+0000 7f5f4ce00640 1 -- 172.21.15.16:0/2061436204 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f5f300039c0 msgr2=0x7f5f4806d290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:07.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:07.719+0000 7f5f4ce00640 1 --2- 172.21.15.16:0/2061436204 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f5f300039c0 0x7f5f4806d290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:07.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:07.719+0000 7f5f4ce00640 1 -- 172.21.15.16:0/2061436204 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5f480b5630 msgr2=0x7f5f480b5ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:07.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:07.719+0000 7f5f4ce00640 1 --2- 172.21.15.16:0/2061436204 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5f480b5630 0x7f5f480b5ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:07.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:07.719+0000 7f5f4ce00640 1 -- 172.21.15.16:0/2061436204 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5f480cadb0 con 0x7f5f480ca330 2024-03-31T15:06:07.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:07.719+0000 7f5f4ce00640 1 --2- 172.21.15.16:0/2061436204 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5f480ca330 0x7f5f480ca7b0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f5f34011990 tx=0x7f5f34011e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:06:07.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:07.719+0000 7f5f45600640 1 -- 172.21.15.16:0/2061436204 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f3400fad0 con 0x7f5f480ca330 2024-03-31T15:06:07.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:07.719+0000 7f5f45600640 1 -- 172.21.15.16:0/2061436204 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5f3400fc70 con 0x7f5f480ca330 2024-03-31T15:06:07.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:07.719+0000 7f5f4ec00640 1 -- 172.21.15.16:0/2061436204 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5f480cb090 con 0x7f5f480ca330 2024-03-31T15:06:07.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:07.719+0000 7f5f45600640 1 -- 172.21.15.16:0/2061436204 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f34014440 con 0x7f5f480ca330 2024-03-31T15:06:07.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:07.719+0000 7f5f4ec00640 1 -- 172.21.15.16:0/2061436204 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5f480cb650 con 0x7f5f480ca330 2024-03-31T15:06:07.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:07.723+0000 7f5f4ec00640 1 -- 172.21.15.16:0/2061436204 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5f4804f8c0 con 0x7f5f480ca330 2024-03-31T15:06:07.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:07.723+0000 7f5f45600640 1 -- 172.21.15.16:0/2061436204 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f5f34012040 con 0x7f5f480ca330 2024-03-31T15:06:07.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:07.723+0000 7f5f45600640 1 --2- 172.21.15.16:0/2061436204 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f5f280683b0 0x7f5f2806a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:07.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:07.723+0000 7f5f45600640 1 -- 172.21.15.16:0/2061436204 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5709+0+0 (secure 0 0 0) 0x7f5f34086fc0 con 0x7f5f480ca330 2024-03-31T15:06:07.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:07.723+0000 7f5f4d800640 1 --2- 172.21.15.16:0/2061436204 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f5f280683b0 0x7f5f2806a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:07.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:07.723+0000 7f5f4d800640 1 --2- 172.21.15.16:0/2061436204 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f5f280683b0 0x7f5f2806a870 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f5f3c0aa000 tx=0x7f5f3c001ac0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:06:07.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:07.727+0000 7f5f45600640 1 -- 172.21.15.16:0/2061436204 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f5f34019030 con 0x7f5f480ca330 2024-03-31T15:06:07.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:07.891+0000 7f5f4ec00640 1 -- 172.21.15.16:0/2061436204 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f5f48068ae0 con 0x7f5f480ca330 2024-03-31T15:06:07.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:07.895+0000 7f5f45600640 1 -- 172.21.15.16:0/2061436204 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v45) ==== 74+0+16541 (secure 0 0 0) 0x7f5f34056e90 con 0x7f5f480ca330 2024-03-31T15:06:07.888 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:06:07.889 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":45,"fsid":"286eb5da-fb47-4550-b290-d9519c13b9a5","created":"2024-03-31T15:03:11.438952+0000","modified":"2024-03-31T15:05:28.074367+0000","last_up_change":"2024-03-31T15:03:22.885535+0000","last_in_change":"2024-03-31T15:03:13.941667+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-31T15:03:22.550787+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2024-03-31T15:05:24.856239+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"44","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2024-03-31T15:05:25.417826+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"45","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"92a66450-ef19-40d8-ac3d-d760b23b0bda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6801","nonce":932712244}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6803","nonce":932712244}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6807","nonce":932712244}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6805","nonce":932712244}]},"public_addr":"172.21.15.16:6801/932712244","cluster_addr":"172.21.15.16:6803/932712244","heartbeat_back_addr":"172.21.15.16:6807/932712244","heartbeat_front_addr":"172.21.15.16:6805/932712244","state":["exists","up"]},{"osd":1,"uuid":"da9d1f2b-1968-4959-b843-faa231988e92","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6810","nonce":1398737533}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6811","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6813","nonce":1398737533}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6821","nonce":1398737533}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6815","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6816","nonce":1398737533}]},"public_addr":"172.21.15.16:6810/1398737533","cluster_addr":"172.21.15.16:6813/1398737533","heartbeat_back_addr":"172.21.15.16:6821/1398737533","heartbeat_front_addr":"172.21.15.16:6816/1398737533","state":["exists","up"]},{"osd":2,"uuid":"01a73662-9a21-4b1f-bcbb-42b9636a4638","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6819","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6822","nonce":3018526646}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6825","nonce":3018526646}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6831","nonce":3018526646}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6827","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6829","nonce":3018526646}]},"public_addr":"172.21.15.16:6822/3018526646","cluster_addr":"172.21.15.16:6825/3018526646","heartbeat_back_addr":"172.21.15.16:6831/3018526646","heartbeat_front_addr":"172.21.15.16:6829/3018526646","state":["exists","up"]},{"osd":3,"uuid":"c7771fd3-e18e-4500-ac38-23cf10ebcd75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6809","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6812","nonce":3360352974}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6817","nonce":3360352974}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6828","nonce":3360352974}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6823","nonce":3360352974}]},"public_addr":"172.21.15.16:6812/3360352974","cluster_addr":"172.21.15.16:6817/3360352974","heartbeat_back_addr":"172.21.15.16:6828/3360352974","heartbeat_front_addr":"172.21.15.16:6823/3360352974","state":["exists","up"]},{"osd":4,"uuid":"88260730-1482-4fbd-a40f-cb1124e9df4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6801","nonce":1856844856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6803","nonce":1856844856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6807","nonce":1856844856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6805","nonce":1856844856}]},"public_addr":"172.21.15.89:6801/1856844856","cluster_addr":"172.21.15.89:6803/1856844856","heartbeat_back_addr":"172.21.15.89:6807/1856844856","heartbeat_front_addr":"172.21.15.89:6805/1856844856","state":["exists","up"]},{"osd":5,"uuid":"4cb2c00c-fae0-4dd8-9d27-ec4f0093ccf7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6814","nonce":50395527}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6817","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6820","nonce":50395527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6831","nonce":50395527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6823","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6826","nonce":50395527}]},"public_addr":"172.21.15.89:6814/50395527","cluster_addr":"172.21.15.89:6820/50395527","heartbeat_back_addr":"172.21.15.89:6831/50395527","heartbeat_front_addr":"172.21.15.89:6826/50395527","state":["exists","up"]},{"osd":6,"uuid":"81ce0c19-2554-4cae-acbe-66a05e9e9b9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6809","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6812","nonce":3470471752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6815","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6818","nonce":3470471752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6827","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6829","nonce":3470471752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6821","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6824","nonce":3470471752}]},"public_addr":"172.21.15.89:6812/3470471752","cluster_addr":"172.21.15.89:6818/3470471752","heartbeat_back_addr":"172.21.15.89:6829/3470471752","heartbeat_front_addr":"172.21.15.89:6824/3470471752","state":["exists","up"]},{"osd":7,"uuid":"85e4313b-d37d-442e-8042-153bf20a847d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6811","nonce":1796778630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6813","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6816","nonce":1796778630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6825","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6828","nonce":1796778630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6819","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6822","nonce":1796778630}]},"public_addr":"172.21.15.89:6811/1796778630","cluster_addr":"172.21.15.89:6816/1796778630","heartbeat_back_addr":"172.21.15.89:6828/1796778630","heartbeat_front_addr":"172.21.15.89:6822/1796778630","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.439088+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.529970+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.680512+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:21.044161+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.582111+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.536558+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.409522+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.590229+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-31T15:06:07.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:07.895+0000 7f5f4ec00640 1 -- 172.21.15.16:0/2061436204 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f5f280683b0 msgr2=0x7f5f2806a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:07.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:07.895+0000 7f5f4ec00640 1 --2- 172.21.15.16:0/2061436204 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f5f280683b0 0x7f5f2806a870 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f5f3c0aa000 tx=0x7f5f3c001ac0 comp rx=0 tx=0).stop 2024-03-31T15:06:07.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:07.895+0000 7f5f4ec00640 1 -- 172.21.15.16:0/2061436204 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5f480ca330 msgr2=0x7f5f480ca7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:07.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:07.895+0000 7f5f4ec00640 1 --2- 172.21.15.16:0/2061436204 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5f480ca330 0x7f5f480ca7b0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f5f34011990 tx=0x7f5f34011e50 comp rx=0 tx=0).stop 2024-03-31T15:06:07.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:07.895+0000 7f5f4ec00640 1 -- 172.21.15.16:0/2061436204 shutdown_connections 2024-03-31T15:06:07.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:07.895+0000 7f5f4ec00640 1 --2- 172.21.15.16:0/2061436204 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f5f280683b0 0x7f5f2806a870 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:07.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:07.895+0000 7f5f4ec00640 1 --2- 172.21.15.16:0/2061436204 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5f480b5630 0x7f5f480b5ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:07.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:07.895+0000 7f5f4ec00640 1 --2- 172.21.15.16:0/2061436204 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5f480ca330 0x7f5f480ca7b0 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:07.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:07.895+0000 7f5f4ec00640 1 --2- 172.21.15.16:0/2061436204 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f5f300039c0 0x7f5f4806d290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:07.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:07.895+0000 7f5f4ec00640 1 -- 172.21.15.16:0/2061436204 >> 172.21.15.16:0/2061436204 conn(0x7f5f480b39c0 msgr2=0x7f5f4806f5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:06:07.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:07.895+0000 7f5f4ec00640 1 -- 172.21.15.16:0/2061436204 shutdown_connections 2024-03-31T15:06:07.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:07.895+0000 7f5f4ec00640 1 -- 172.21.15.16:0/2061436204 wait complete. 2024-03-31T15:06:07.910 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-31T15:03:22.550787+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-31T15:05:24.856239+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '44', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 7, 'pool_name': 'cephfs_data', 'create_time': '2024-03-31T15:05:25.417826+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '45', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-31T15:06:07.910 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-31T15:06:08.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:08.059+0000 7f7072600640 1 Processor -- start 2024-03-31T15:06:08.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:08.059+0000 7f7072600640 1 -- start start 2024-03-31T15:06:08.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:08.063+0000 7f7072600640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f706c0cf7a0 0x7f706c0d1890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:08.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:08.063+0000 7f7072600640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f706c0d03d0 con 0x7f706c0cea30 2024-03-31T15:06:08.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:08.063+0000 7f7072600640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f706c162f60 con 0x7f706c0cee10 2024-03-31T15:06:08.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:08.063+0000 7f7072600640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f706c0b47e0 con 0x7f706c0cf7a0 2024-03-31T15:06:08.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:08.063+0000 7f7071200640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f706c0cea30 0x7f706c0b2680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:37120/0 (socket says 172.21.15.16:37120) 2024-03-31T15:06:08.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:08.063+0000 7f7071200640 1 -- 172.21.15.16:0/1149200496 learned_addr learned my addr 172.21.15.16:0/1149200496 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:06:08.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:08.063+0000 7f7063e00640 1 --2- 172.21.15.16:0/1149200496 >> v2:172.21.15.89:3300/0 conn(0x7f706c0cf7a0 0x7f706c0d1890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:08.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:08.063+0000 7f706be00640 1 -- 172.21.15.16:0/1149200496 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2613235686 0 0) 0x7f706c0d03d0 con 0x7f706c0cea30 2024-03-31T15:06:08.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:08.063+0000 7f706be00640 1 -- 172.21.15.16:0/1149200496 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7050003530 con 0x7f706c0cea30 2024-03-31T15:06:08.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:08.063+0000 7f706be00640 1 -- 172.21.15.16:0/1149200496 <== mon.1 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1284925209 0 0) 0x7f706c162f60 con 0x7f706c0cee10 2024-03-31T15:06:08.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:08.063+0000 7f706be00640 1 -- 172.21.15.16:0/1149200496 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f706c0d03d0 con 0x7f706c0cee10 2024-03-31T15:06:08.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:08.063+0000 7f7063e00640 1 -- 172.21.15.16:0/1149200496 >> v1:172.21.15.89:6789/0 conn(0x7f706c0cee10 legacy=0x7f706c162850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:08.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:08.063+0000 7f7063e00640 1 -- 172.21.15.16:0/1149200496 >> v1:172.21.15.16:6789/0 conn(0x7f706c0cea30 legacy=0x7f706c0b2680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:08.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:08.063+0000 7f7063e00640 1 -- 172.21.15.16:0/1149200496 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f706c0cfff0 con 0x7f706c0cf7a0 2024-03-31T15:06:08.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:08.067+0000 7f7063e00640 1 --2- 172.21.15.16:0/1149200496 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f706c0cf7a0 0x7f706c0d1890 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f70540097c0 tx=0x7f705401c410 comp rx=0 tx=0).ready entity=mon.1 client_cookie=79f20d7a1cdf5a3e server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:06:08.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:08.067+0000 7f706be00640 1 -- 172.21.15.16:0/1149200496 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70540b0070 con 0x7f706c0cf7a0 2024-03-31T15:06:08.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:08.067+0000 7f706be00640 1 -- 172.21.15.16:0/1149200496 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f70540ab600 con 0x7f706c0cf7a0 2024-03-31T15:06:08.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:08.067+0000 7f706be00640 1 -- 172.21.15.16:0/1149200496 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70540ab9d0 con 0x7f706c0cf7a0 2024-03-31T15:06:08.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:08.067+0000 7f7072600640 1 -- 172.21.15.16:0/1149200496 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f706c0cf7a0 msgr2=0x7f706c0d1890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:08.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:08.067+0000 7f7072600640 1 --2- 172.21.15.16:0/1149200496 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f706c0cf7a0 0x7f706c0d1890 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f70540097c0 tx=0x7f705401c410 comp rx=0 tx=0).stop 2024-03-31T15:06:08.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:08.067+0000 7f7072600640 1 -- 172.21.15.16:0/1149200496 shutdown_connections 2024-03-31T15:06:08.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:08.067+0000 7f7072600640 1 --2- 172.21.15.16:0/1149200496 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f706c0cf7a0 0x7f706c0d1890 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:08.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:08.067+0000 7f7072600640 1 -- 172.21.15.16:0/1149200496 >> 172.21.15.16:0/1149200496 conn(0x7f706c0b3b40 msgr2=0x7f706c0b3f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:06:08.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:08.067+0000 7f7072600640 1 -- 172.21.15.16:0/1149200496 shutdown_connections 2024-03-31T15:06:08.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:08.067+0000 7f7072600640 1 -- 172.21.15.16:0/1149200496 wait complete. 2024-03-31T15:06:08.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:08.071+0000 7f7072600640 1 Processor -- start 2024-03-31T15:06:08.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:08.071+0000 7f7072600640 1 -- start start 2024-03-31T15:06:08.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:08.071+0000 7f7072600640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f706c0cea30 0x7f706c0cc190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:08.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:08.071+0000 7f7072600640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f706c0cee10 0x7f706c06f640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:08.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:08.071+0000 7f7072600640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f706c0cdcd0 0x7f706c06fb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:08.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:08.071+0000 7f7072600640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f706c16c4a0 con 0x7f706c0cea30 2024-03-31T15:06:08.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:08.071+0000 7f7072600640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f706c0b3870 con 0x7f706c0cee10 2024-03-31T15:06:08.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:08.071+0000 7f7072600640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f706c16c620 con 0x7f706c0cdcd0 2024-03-31T15:06:08.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:08.071+0000 7f7071200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f706c0cea30 0x7f706c0cc190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:08.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:08.071+0000 7f7071200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f706c0cea30 0x7f706c0cc190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35028/0 (socket says 172.21.15.16:35028) 2024-03-31T15:06:08.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:08.071+0000 7f7071200640 1 -- 172.21.15.16:0/2772759323 learned_addr learned my addr 172.21.15.16:0/2772759323 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:06:08.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:08.071+0000 7f7063e00640 1 --2- 172.21.15.16:0/2772759323 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f706c0cee10 0x7f706c06f640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:08.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:08.071+0000 7f7071c00640 1 --2- 172.21.15.16:0/2772759323 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f706c0cdcd0 0x7f706c06fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:08.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:08.071+0000 7f7071200640 1 -- 172.21.15.16:0/2772759323 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f706c0cdcd0 msgr2=0x7f706c06fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:08.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:08.071+0000 7f7071200640 1 --2- 172.21.15.16:0/2772759323 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f706c0cdcd0 0x7f706c06fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:08.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:08.071+0000 7f7071200640 1 -- 172.21.15.16:0/2772759323 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f706c0cee10 msgr2=0x7f706c06f640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:08.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:08.071+0000 7f7071200640 1 --2- 172.21.15.16:0/2772759323 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f706c0cee10 0x7f706c06f640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:08.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:08.071+0000 7f7071200640 1 -- 172.21.15.16:0/2772759323 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f706c0cb3d0 con 0x7f706c0cea30 2024-03-31T15:06:08.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:08.071+0000 7f7071200640 1 --2- 172.21.15.16:0/2772759323 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f706c0cea30 0x7f706c0cc190 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f705c00c4d0 tx=0x7f705c00c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:06:08.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:08.071+0000 7f7069600640 1 -- 172.21.15.16:0/2772759323 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f705c017070 con 0x7f706c0cea30 2024-03-31T15:06:08.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:08.071+0000 7f7069600640 1 -- 172.21.15.16:0/2772759323 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f705c005cf0 con 0x7f706c0cea30 2024-03-31T15:06:08.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:08.071+0000 7f7069600640 1 -- 172.21.15.16:0/2772759323 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f705c0072b0 con 0x7f706c0cea30 2024-03-31T15:06:08.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:08.071+0000 7f7072600640 1 -- 172.21.15.16:0/2772759323 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f706c0cb6b0 con 0x7f706c0cea30 2024-03-31T15:06:08.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:08.075+0000 7f7072600640 1 -- 172.21.15.16:0/2772759323 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f706c0cbc70 con 0x7f706c0cea30 2024-03-31T15:06:08.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:08.075+0000 7f7072600640 1 -- 172.21.15.16:0/2772759323 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f706c04f8c0 con 0x7f706c0cea30 2024-03-31T15:06:08.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:08.075+0000 7f7069600640 1 -- 172.21.15.16:0/2772759323 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f705c01a070 con 0x7f706c0cea30 2024-03-31T15:06:08.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:08.075+0000 7f7069600640 1 --2- 172.21.15.16:0/2772759323 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f7048068380 0x7f704806a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:08.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:08.075+0000 7f7069600640 1 -- 172.21.15.16:0/2772759323 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5709+0+0 (secure 0 0 0) 0x7f705c085d10 con 0x7f706c0cea30 2024-03-31T15:06:08.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:08.075+0000 7f7063e00640 1 --2- 172.21.15.16:0/2772759323 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f7048068380 0x7f704806a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:08.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:08.079+0000 7f7063e00640 1 --2- 172.21.15.16:0/2772759323 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f7048068380 0x7f704806a840 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f70540ad000 tx=0x7f7054008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:06:08.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:08.079+0000 7f7069600640 1 -- 172.21.15.16:0/2772759323 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f705c08b020 con 0x7f706c0cea30 2024-03-31T15:06:08.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:08.251+0000 7f7072600640 1 -- 172.21.15.16:0/2772759323 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f706c075180 con 0x7f706c0cea30 2024-03-31T15:06:08.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:08.251+0000 7f7069600640 1 -- 172.21.15.16:0/2772759323 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v45) ==== 93+0+10 (secure 0 0 0) 0x7f705c055c70 con 0x7f706c0cea30 2024-03-31T15:06:08.245 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2024-03-31T15:06:08.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:08.251+0000 7f7072600640 1 -- 172.21.15.16:0/2772759323 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f7048068380 msgr2=0x7f704806a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:08.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:08.251+0000 7f7072600640 1 --2- 172.21.15.16:0/2772759323 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f7048068380 0x7f704806a840 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f70540ad000 tx=0x7f7054008040 comp rx=0 tx=0).stop 2024-03-31T15:06:08.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:08.251+0000 7f7072600640 1 -- 172.21.15.16:0/2772759323 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f706c0cea30 msgr2=0x7f706c0cc190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:08.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:08.251+0000 7f7072600640 1 --2- 172.21.15.16:0/2772759323 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f706c0cea30 0x7f706c0cc190 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f705c00c4d0 tx=0x7f705c00c990 comp rx=0 tx=0).stop 2024-03-31T15:06:08.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:08.251+0000 7f7072600640 1 -- 172.21.15.16:0/2772759323 shutdown_connections 2024-03-31T15:06:08.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:08.251+0000 7f7072600640 1 --2- 172.21.15.16:0/2772759323 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f7048068380 0x7f704806a840 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:08.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:08.251+0000 7f7072600640 1 --2- 172.21.15.16:0/2772759323 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f706c0cdcd0 0x7f706c06fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:08.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:08.251+0000 7f7072600640 1 --2- 172.21.15.16:0/2772759323 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f706c0cee10 0x7f706c06f640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:08.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:08.251+0000 7f7072600640 1 --2- 172.21.15.16:0/2772759323 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f706c0cea30 0x7f706c0cc190 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:08.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:08.251+0000 7f7072600640 1 -- 172.21.15.16:0/2772759323 >> 172.21.15.16:0/2772759323 conn(0x7f706c0b3b40 msgr2=0x7f706c070490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:06:08.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:08.251+0000 7f7072600640 1 -- 172.21.15.16:0/2772759323 shutdown_connections 2024-03-31T15:06:08.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:08.251+0000 7f7072600640 1 -- 172.21.15.16:0/2772759323 wait complete. 2024-03-31T15:06:08.266 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-31T15:06:08.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:08.415+0000 7fcefc800640 1 Processor -- start 2024-03-31T15:06:08.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:08.415+0000 7fcefc800640 1 -- start start 2024-03-31T15:06:08.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:08.415+0000 7fcefc800640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fcef406f920 0x7fcef406fd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:08.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:08.415+0000 7fcefc800640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fcef40cf150 0x7fcef41419c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:08.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:08.415+0000 7fcefc800640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcef4070240 con 0x7fcef40d0a60 2024-03-31T15:06:08.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:08.415+0000 7fcefc800640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7fcef413f670 con 0x7fcef40cf150 2024-03-31T15:06:08.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:08.415+0000 7fcefc800640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fcef415ca70 con 0x7fcef406f920 2024-03-31T15:06:08.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:08.415+0000 7fcef9e00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fcef406f920 0x7fcef406fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:08.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:08.415+0000 7fcefa800640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fcef40cf150 0x7fcef41419c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:08.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:08.415+0000 7fcef9400640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fcef40d0a60 0x7fcef40cea40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.16:51850/0 (socket says 172.21.15.16:51850) 2024-03-31T15:06:08.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:08.415+0000 7fcefa800640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fcef40cf150 0x7fcef41419c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.16:50994/0 (socket says 172.21.15.16:50994) 2024-03-31T15:06:08.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:08.415+0000 7fcef9400640 1 -- 172.21.15.16:0/3993471236 learned_addr learned my addr 172.21.15.16:0/3993471236 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:06:08.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:08.415+0000 7fcef9e00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fcef406f920 0x7fcef406fd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3301/0 says I am v2:172.21.15.16:52198/0 (socket says 172.21.15.16:52198) 2024-03-31T15:06:08.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:08.419+0000 7fcef9e00640 1 -- 172.21.15.16:0/3993471236 >> v2:172.21.15.89:3300/0 conn(0x7fcef40cf150 msgr2=0x7fcef41419c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:06:08.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:08.419+0000 7fcef9e00640 1 --2- 172.21.15.16:0/3993471236 >> v2:172.21.15.89:3300/0 conn(0x7fcef40cf150 0x7fcef41419c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:08.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:08.419+0000 7fcef9e00640 1 -- 172.21.15.16:0/3993471236 >> v1:172.21.15.89:6789/0 conn(0x7fcef40d0a60 legacy=0x7fcef40cea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:08.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:08.419+0000 7fcef9e00640 1 -- 172.21.15.16:0/3993471236 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcef415cd50 con 0x7fcef406f920 2024-03-31T15:06:08.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:08.419+0000 7fcef9e00640 1 --2- 172.21.15.16:0/3993471236 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fcef406f920 0x7fcef406fd00 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fcee80097c0 tx=0x7fcee801c4a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=79ac3e054f888324 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:06:08.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:08.419+0000 7fcef8a00640 1 -- 172.21.15.16:0/3993471236 <== mon.2 v2:172.21.15.89:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcee80b0070 con 0x7fcef406f920 2024-03-31T15:06:08.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:08.419+0000 7fcef8a00640 1 -- 172.21.15.16:0/3993471236 <== mon.2 v2:172.21.15.89:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcee80ab600 con 0x7fcef406f920 2024-03-31T15:06:08.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:08.419+0000 7fcef8a00640 1 -- 172.21.15.16:0/3993471236 <== mon.2 v2:172.21.15.89:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcee80ab9d0 con 0x7fcef406f920 2024-03-31T15:06:08.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:08.419+0000 7fcefc800640 1 -- 172.21.15.16:0/3993471236 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fcef406f920 msgr2=0x7fcef406fd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:08.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:08.419+0000 7fcefc800640 1 --2- 172.21.15.16:0/3993471236 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fcef406f920 0x7fcef406fd00 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fcee80097c0 tx=0x7fcee801c4a0 comp rx=0 tx=0).stop 2024-03-31T15:06:08.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:08.419+0000 7fcefc800640 1 -- 172.21.15.16:0/3993471236 shutdown_connections 2024-03-31T15:06:08.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:08.419+0000 7fcefc800640 1 --2- 172.21.15.16:0/3993471236 >> v2:172.21.15.89:3300/0 conn(0x7fcef40cf150 0x7fcef41419c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:08.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:08.419+0000 7fcefc800640 1 --2- 172.21.15.16:0/3993471236 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fcef406f920 0x7fcef406fd00 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:08.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:08.419+0000 7fcefc800640 1 -- 172.21.15.16:0/3993471236 >> 172.21.15.16:0/3993471236 conn(0x7fcef4067930 msgr2=0x7fcef40b3ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:06:08.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:08.423+0000 7fcefc800640 1 -- 172.21.15.16:0/3993471236 shutdown_connections 2024-03-31T15:06:08.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:08.423+0000 7fcefc800640 1 -- 172.21.15.16:0/3993471236 wait complete. 2024-03-31T15:06:08.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:08.423+0000 7fcefc800640 1 Processor -- start 2024-03-31T15:06:08.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:08.423+0000 7fcefc800640 1 -- start start 2024-03-31T15:06:08.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:08.423+0000 7fcefc800640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcef406f920 0x7fcef412b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:08.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:08.423+0000 7fcefc800640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcef40cf150 0x7fcef412eec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:08.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:08.423+0000 7fcefc800640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fcef40d0a60 0x7fcef415eaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:08.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:08.423+0000 7fcefc800640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fcef4169230 con 0x7fcef40cf150 2024-03-31T15:06:08.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:08.423+0000 7fcefc800640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fcef41690b0 con 0x7fcef406f920 2024-03-31T15:06:08.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:08.423+0000 7fcefc800640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fcef41693b0 con 0x7fcef40d0a60 2024-03-31T15:06:08.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:08.423+0000 7fcef9400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcef40cf150 0x7fcef412eec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:08.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:08.423+0000 7fcef9400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcef40cf150 0x7fcef412eec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35038/0 (socket says 172.21.15.16:35038) 2024-03-31T15:06:08.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:08.423+0000 7fcef9400640 1 -- 172.21.15.16:0/3002878689 learned_addr learned my addr 172.21.15.16:0/3002878689 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:06:08.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:08.423+0000 7fcef9e00640 1 --2- 172.21.15.16:0/3002878689 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcef406f920 0x7fcef412b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:08.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:08.423+0000 7fcefa800640 1 --2- 172.21.15.16:0/3002878689 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fcef40d0a60 0x7fcef415eaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:08.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:08.423+0000 7fcef9400640 1 -- 172.21.15.16:0/3002878689 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fcef40d0a60 msgr2=0x7fcef415eaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:08.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:08.423+0000 7fcef9400640 1 --2- 172.21.15.16:0/3002878689 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fcef40d0a60 0x7fcef415eaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:08.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:08.423+0000 7fcef9400640 1 -- 172.21.15.16:0/3002878689 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcef406f920 msgr2=0x7fcef412b0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:08.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:08.423+0000 7fcef9400640 1 --2- 172.21.15.16:0/3002878689 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcef406f920 0x7fcef412b0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:08.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:08.423+0000 7fcef9400640 1 -- 172.21.15.16:0/3002878689 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcef412f400 con 0x7fcef40cf150 2024-03-31T15:06:08.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:08.423+0000 7fcef9400640 1 --2- 172.21.15.16:0/3002878689 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcef40cf150 0x7fcef412eec0 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7fcee400c450 tx=0x7fcee400c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:06:08.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:08.423+0000 7fceee000640 1 -- 172.21.15.16:0/3002878689 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcee4017070 con 0x7fcef40cf150 2024-03-31T15:06:08.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:08.423+0000 7fceee000640 1 -- 172.21.15.16:0/3002878689 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcee4005cf0 con 0x7fcef40cf150 2024-03-31T15:06:08.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:08.423+0000 7fceee000640 1 -- 172.21.15.16:0/3002878689 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcee4007290 con 0x7fcef40cf150 2024-03-31T15:06:08.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:08.423+0000 7fcefc800640 1 -- 172.21.15.16:0/3002878689 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcef412b680 con 0x7fcef40cf150 2024-03-31T15:06:08.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:08.427+0000 7fcefc800640 1 -- 172.21.15.16:0/3002878689 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcef412bc40 con 0x7fcef40cf150 2024-03-31T15:06:08.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:08.427+0000 7fcefc800640 1 -- 172.21.15.16:0/3002878689 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcef4069950 con 0x7fcef40cf150 2024-03-31T15:06:08.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:08.427+0000 7fceee000640 1 -- 172.21.15.16:0/3002878689 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fcee401a070 con 0x7fcef40cf150 2024-03-31T15:06:08.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:08.427+0000 7fceee000640 1 --2- 172.21.15.16:0/3002878689 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fced8068380 0x7fced806a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:08.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:08.427+0000 7fceee000640 1 -- 172.21.15.16:0/3002878689 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5709+0+0 (secure 0 0 0) 0x7fcee4085c10 con 0x7fcef40cf150 2024-03-31T15:06:08.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:08.427+0000 7fcef9e00640 1 --2- 172.21.15.16:0/3002878689 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fced8068380 0x7fced806a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:08.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:08.427+0000 7fcef9e00640 1 --2- 172.21.15.16:0/3002878689 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fced8068380 0x7fced806a840 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fcee80ad000 tx=0x7fcee80a6000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:06:08.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:08.431+0000 7fceee000640 1 -- 172.21.15.16:0/3002878689 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fcee4055b80 con 0x7fcef40cf150 2024-03-31T15:06:08.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:08.595+0000 7fcefc800640 1 -- 172.21.15.16:0/3002878689 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fcef41411c0 con 0x7fcef40cf150 2024-03-31T15:06:08.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:08.595+0000 7fceee000640 1 -- 172.21.15.16:0/3002878689 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v45) ==== 104+0+11 (secure 0 0 0) 0x7fcee4059e80 con 0x7fcef40cf150 2024-03-31T15:06:08.591 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2024-03-31T15:06:08.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:08.599+0000 7fcefc800640 1 -- 172.21.15.16:0/3002878689 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fced8068380 msgr2=0x7fced806a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:08.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:08.599+0000 7fcefc800640 1 --2- 172.21.15.16:0/3002878689 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fced8068380 0x7fced806a840 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fcee80ad000 tx=0x7fcee80a6000 comp rx=0 tx=0).stop 2024-03-31T15:06:08.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:08.599+0000 7fcefc800640 1 -- 172.21.15.16:0/3002878689 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcef40cf150 msgr2=0x7fcef412eec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:08.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:08.599+0000 7fcefc800640 1 --2- 172.21.15.16:0/3002878689 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcef40cf150 0x7fcef412eec0 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7fcee400c450 tx=0x7fcee400c910 comp rx=0 tx=0).stop 2024-03-31T15:06:08.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:08.599+0000 7fcefc800640 1 -- 172.21.15.16:0/3002878689 shutdown_connections 2024-03-31T15:06:08.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:08.599+0000 7fcefc800640 1 --2- 172.21.15.16:0/3002878689 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fced8068380 0x7fced806a840 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:08.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:08.599+0000 7fcefc800640 1 --2- 172.21.15.16:0/3002878689 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fcef40d0a60 0x7fcef415eaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:08.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:08.599+0000 7fcefc800640 1 --2- 172.21.15.16:0/3002878689 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcef40cf150 0x7fcef412eec0 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:08.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:08.599+0000 7fcefc800640 1 --2- 172.21.15.16:0/3002878689 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcef406f920 0x7fcef412b0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:08.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:08.599+0000 7fcefc800640 1 -- 172.21.15.16:0/3002878689 >> 172.21.15.16:0/3002878689 conn(0x7fcef4067930 msgr2=0x7fcef40d0ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:06:08.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:08.599+0000 7fcefc800640 1 -- 172.21.15.16:0/3002878689 shutdown_connections 2024-03-31T15:06:08.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:08.599+0000 7fcefc800640 1 -- 172.21.15.16:0/3002878689 wait complete. 2024-03-31T15:06:08.611 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-31T15:06:08.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:08.759+0000 7fc256000640 1 Processor -- start 2024-03-31T15:06:08.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:08.759+0000 7fc256000640 1 -- start start 2024-03-31T15:06:08.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:08.763+0000 7fc256000640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fc2500d0a60 0x7fc2500cea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:08.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:08.763+0000 7fc256000640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fc2500cef80 0x7fc2501419c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:08.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:08.763+0000 7fc256000640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc25015ca70 con 0x7fc25006f920 2024-03-31T15:06:08.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:08.763+0000 7fc256000640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7fc25013f670 con 0x7fc2500d0a60 2024-03-31T15:06:08.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:08.763+0000 7fc256000640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7fc25015cd50 con 0x7fc2500cef80 2024-03-31T15:06:08.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:08.763+0000 7fc24f400640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fc25006f920 0x7fc25006fd00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:37124/0 (socket says 172.21.15.16:37124) 2024-03-31T15:06:08.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:08.763+0000 7fc247e00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fc2500d0a60 0x7fc2500cea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:08.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:08.763+0000 7fc24f400640 1 -- 172.21.15.16:0/1339799184 learned_addr learned my addr 172.21.15.16:0/1339799184 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:06:08.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:08.763+0000 7fc24fe00640 1 --2- 172.21.15.16:0/1339799184 >> v2:172.21.15.89:3300/0 conn(0x7fc2500cef80 0x7fc2501419c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:08.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:08.763+0000 7fc247e00640 1 -- 172.21.15.16:0/1339799184 >> v2:172.21.15.89:3300/0 conn(0x7fc2500cef80 msgr2=0x7fc2501419c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:06:08.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:08.763+0000 7fc247e00640 1 --2- 172.21.15.16:0/1339799184 >> v2:172.21.15.89:3300/0 conn(0x7fc2500cef80 0x7fc2501419c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:08.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:08.763+0000 7fc247e00640 1 -- 172.21.15.16:0/1339799184 >> v1:172.21.15.16:6789/0 conn(0x7fc25006f920 legacy=0x7fc25006fd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:08.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:08.763+0000 7fc247e00640 1 -- 172.21.15.16:0/1339799184 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc25015d030 con 0x7fc2500d0a60 2024-03-31T15:06:08.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:08.767+0000 7fc247e00640 1 --2- 172.21.15.16:0/1339799184 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc2500d0a60 0x7fc2500cea40 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7fc2500b4530 tx=0x7fc23801c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db70a11e11045e40 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:06:08.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:08.767+0000 7fc24ea00640 1 -- 172.21.15.16:0/1339799184 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2380b0070 con 0x7fc2500d0a60 2024-03-31T15:06:08.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:08.767+0000 7fc24ea00640 1 -- 172.21.15.16:0/1339799184 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc2380ab600 con 0x7fc2500d0a60 2024-03-31T15:06:08.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:08.767+0000 7fc24ea00640 1 -- 172.21.15.16:0/1339799184 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2380ab9d0 con 0x7fc2500d0a60 2024-03-31T15:06:08.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:08.767+0000 7fc256000640 1 -- 172.21.15.16:0/1339799184 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc2500d0a60 msgr2=0x7fc2500cea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:08.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:08.767+0000 7fc256000640 1 --2- 172.21.15.16:0/1339799184 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc2500d0a60 0x7fc2500cea40 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7fc2500b4530 tx=0x7fc23801c480 comp rx=0 tx=0).stop 2024-03-31T15:06:08.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:08.767+0000 7fc256000640 1 -- 172.21.15.16:0/1339799184 shutdown_connections 2024-03-31T15:06:08.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:08.767+0000 7fc256000640 1 --2- 172.21.15.16:0/1339799184 >> v2:172.21.15.89:3300/0 conn(0x7fc2500cef80 0x7fc2501419c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:08.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:08.767+0000 7fc256000640 1 --2- 172.21.15.16:0/1339799184 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc2500d0a60 0x7fc2500cea40 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:08.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:08.767+0000 7fc256000640 1 -- 172.21.15.16:0/1339799184 >> 172.21.15.16:0/1339799184 conn(0x7fc250067930 msgr2=0x7fc2500b3ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:06:08.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:08.767+0000 7fc256000640 1 -- 172.21.15.16:0/1339799184 shutdown_connections 2024-03-31T15:06:08.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:08.767+0000 7fc256000640 1 -- 172.21.15.16:0/1339799184 wait complete. 2024-03-31T15:06:08.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:08.767+0000 7fc256000640 1 Processor -- start 2024-03-31T15:06:08.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:08.767+0000 7fc256000640 1 -- start start 2024-03-31T15:06:08.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:08.767+0000 7fc256000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc25006f920 0x7fc25013e670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:08.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:08.767+0000 7fc256000640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc2500cef80 0x7fc250157950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:08.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:08.771+0000 7fc256000640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fc2500d0a60 0x7fc25013dec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:08.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:08.771+0000 7fc256000640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fc250169230 con 0x7fc25006f920 2024-03-31T15:06:08.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:08.771+0000 7fc256000640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fc2501690b0 con 0x7fc2500cef80 2024-03-31T15:06:08.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:08.771+0000 7fc256000640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fc2501693b0 con 0x7fc2500d0a60 2024-03-31T15:06:08.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:08.771+0000 7fc24f400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc25006f920 0x7fc25013e670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:08.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:08.771+0000 7fc24f400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc25006f920 0x7fc25013e670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35042/0 (socket says 172.21.15.16:35042) 2024-03-31T15:06:08.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:08.771+0000 7fc24f400640 1 -- 172.21.15.16:0/1610945964 learned_addr learned my addr 172.21.15.16:0/1610945964 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:06:08.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:08.771+0000 7fc247e00640 1 --2- 172.21.15.16:0/1610945964 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc2500cef80 0x7fc250157950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:08.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:08.771+0000 7fc24fe00640 1 --2- 172.21.15.16:0/1610945964 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fc2500d0a60 0x7fc25013dec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:08.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:08.771+0000 7fc24f400640 1 -- 172.21.15.16:0/1610945964 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fc2500d0a60 msgr2=0x7fc25013dec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:08.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:08.771+0000 7fc24f400640 1 --2- 172.21.15.16:0/1610945964 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fc2500d0a60 0x7fc25013dec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:08.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:08.771+0000 7fc24f400640 1 -- 172.21.15.16:0/1610945964 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc2500cef80 msgr2=0x7fc250157950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:08.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:08.771+0000 7fc24f400640 1 --2- 172.21.15.16:0/1610945964 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc2500cef80 0x7fc250157950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:08.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:08.771+0000 7fc24f400640 1 -- 172.21.15.16:0/1610945964 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc250157e90 con 0x7fc25006f920 2024-03-31T15:06:08.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:08.771+0000 7fc24fe00640 1 --2- 172.21.15.16:0/1610945964 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fc2500d0a60 0x7fc25013dec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:06:08.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:08.771+0000 7fc24f400640 1 --2- 172.21.15.16:0/1610945964 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc25006f920 0x7fc25013e670 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7fc24000c450 tx=0x7fc24000c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:06:08.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:08.771+0000 7fc247400640 1 -- 172.21.15.16:0/1610945964 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc240017070 con 0x7fc25006f920 2024-03-31T15:06:08.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:08.771+0000 7fc247400640 1 -- 172.21.15.16:0/1610945964 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc240005cf0 con 0x7fc25006f920 2024-03-31T15:06:08.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:08.771+0000 7fc247400640 1 -- 172.21.15.16:0/1610945964 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc240007290 con 0x7fc25006f920 2024-03-31T15:06:08.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:08.771+0000 7fc256000640 1 -- 172.21.15.16:0/1610945964 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc250155da0 con 0x7fc25006f920 2024-03-31T15:06:08.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:08.771+0000 7fc256000640 1 -- 172.21.15.16:0/1610945964 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc250156360 con 0x7fc25006f920 2024-03-31T15:06:08.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:08.771+0000 7fc256000640 1 -- 172.21.15.16:0/1610945964 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc250069950 con 0x7fc25006f920 2024-03-31T15:06:08.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:08.771+0000 7fc247400640 1 -- 172.21.15.16:0/1610945964 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fc24001a070 con 0x7fc25006f920 2024-03-31T15:06:08.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:08.775+0000 7fc247400640 1 --2- 172.21.15.16:0/1610945964 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fc22c068380 0x7fc22c06a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:08.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:08.775+0000 7fc247400640 1 -- 172.21.15.16:0/1610945964 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5709+0+0 (secure 0 0 0) 0x7fc240085cd0 con 0x7fc25006f920 2024-03-31T15:06:08.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:08.775+0000 7fc247e00640 1 --2- 172.21.15.16:0/1610945964 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fc22c068380 0x7fc22c06a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:08.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:08.775+0000 7fc247e00640 1 --2- 172.21.15.16:0/1610945964 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fc22c068380 0x7fc22c06a840 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fc2380ad000 tx=0x7fc238008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:06:08.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:08.775+0000 7fc247400640 1 -- 172.21.15.16:0/1610945964 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fc240056280 con 0x7fc25006f920 2024-03-31T15:06:08.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:08.939+0000 7fc256000640 1 -- 172.21.15.16:0/1610945964 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fc2500b4a20 con 0x7fc25006f920 2024-03-31T15:06:08.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:08.943+0000 7fc247400640 1 -- 172.21.15.16:0/1610945964 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v45) ==== 100+0+11 (secure 0 0 0) 0x7fc240055b80 con 0x7fc25006f920 2024-03-31T15:06:08.936 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 64 2024-03-31T15:06:08.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:08.943+0000 7fc256000640 1 -- 172.21.15.16:0/1610945964 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fc22c068380 msgr2=0x7fc22c06a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:08.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:08.943+0000 7fc256000640 1 --2- 172.21.15.16:0/1610945964 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fc22c068380 0x7fc22c06a840 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fc2380ad000 tx=0x7fc238008040 comp rx=0 tx=0).stop 2024-03-31T15:06:08.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:08.943+0000 7fc256000640 1 -- 172.21.15.16:0/1610945964 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc25006f920 msgr2=0x7fc25013e670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:08.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:08.943+0000 7fc256000640 1 --2- 172.21.15.16:0/1610945964 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc25006f920 0x7fc25013e670 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7fc24000c450 tx=0x7fc24000c910 comp rx=0 tx=0).stop 2024-03-31T15:06:08.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:08.943+0000 7fc256000640 1 -- 172.21.15.16:0/1610945964 shutdown_connections 2024-03-31T15:06:08.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:08.943+0000 7fc256000640 1 --2- 172.21.15.16:0/1610945964 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fc22c068380 0x7fc22c06a840 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:08.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:08.943+0000 7fc256000640 1 --2- 172.21.15.16:0/1610945964 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fc2500d0a60 0x7fc25013dec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:08.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:08.943+0000 7fc256000640 1 --2- 172.21.15.16:0/1610945964 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc2500cef80 0x7fc250157950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:08.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:08.943+0000 7fc256000640 1 --2- 172.21.15.16:0/1610945964 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc25006f920 0x7fc25013e670 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:08.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:08.943+0000 7fc256000640 1 -- 172.21.15.16:0/1610945964 >> 172.21.15.16:0/1610945964 conn(0x7fc250067930 msgr2=0x7fc250158950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:06:08.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:08.943+0000 7fc256000640 1 -- 172.21.15.16:0/1610945964 shutdown_connections 2024-03-31T15:06:08.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:08.943+0000 7fc256000640 1 -- 172.21.15.16:0/1610945964 wait complete. 2024-03-31T15:06:08.958 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-31T15:06:09.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:09.103+0000 7ff0f7400640 1 Processor -- start 2024-03-31T15:06:09.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:09.103+0000 7ff0f7400640 1 -- start start 2024-03-31T15:06:09.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:09.103+0000 7ff0f7400640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7ff0f006f920 0x7ff0f006fd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:09.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:09.103+0000 7ff0f7400640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff0f00d0920 con 0x7ff0f00d2010 2024-03-31T15:06:09.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:09.103+0000 7ff0f7400640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff0f00d0c00 con 0x7ff0f0145e40 2024-03-31T15:06:09.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:09.103+0000 7ff0f7400640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7ff0f00702c0 con 0x7ff0f006f920 2024-03-31T15:06:09.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:09.107+0000 7ff0f5400640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7ff0f00d2010 0x7ff0f00d24a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:37128/0 (socket says 172.21.15.16:37128) 2024-03-31T15:06:09.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:09.107+0000 7ff0f5400640 1 -- 172.21.15.16:0/2726289871 learned_addr learned my addr 172.21.15.16:0/2726289871 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:06:09.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:09.107+0000 7ff0efe00640 1 --2- 172.21.15.16:0/2726289871 >> v2:172.21.15.89:3300/0 conn(0x7ff0f006f920 0x7ff0f006fd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:09.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:09.107+0000 7ff0ef400640 1 -- 172.21.15.16:0/2726289871 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 729460020 0 0) 0x7ff0f00d0920 con 0x7ff0f00d2010 2024-03-31T15:06:09.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:09.107+0000 7ff0ef400640 1 -- 172.21.15.16:0/2726289871 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff0d8003530 con 0x7ff0f00d2010 2024-03-31T15:06:09.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:09.107+0000 7ff0ef400640 1 -- 172.21.15.16:0/2726289871 <== mon.1 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3541392770 0 0) 0x7ff0f00d0c00 con 0x7ff0f0145e40 2024-03-31T15:06:09.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:09.107+0000 7ff0ef400640 1 -- 172.21.15.16:0/2726289871 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff0f00d0920 con 0x7ff0f0145e40 2024-03-31T15:06:09.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:09.107+0000 7ff0efe00640 1 -- 172.21.15.16:0/2726289871 >> v1:172.21.15.89:6789/0 conn(0x7ff0f0145e40 legacy=0x7ff0f00cc740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:09.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:09.107+0000 7ff0efe00640 1 -- 172.21.15.16:0/2726289871 >> v1:172.21.15.16:6789/0 conn(0x7ff0f00d2010 legacy=0x7ff0f00d24a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:09.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:09.107+0000 7ff0efe00640 1 -- 172.21.15.16:0/2726289871 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0f00d1810 con 0x7ff0f006f920 2024-03-31T15:06:09.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:09.107+0000 7ff0efe00640 1 --2- 172.21.15.16:0/2726289871 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff0f006f920 0x7ff0f006fd80 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7ff0dc0097c0 tx=0x7ff0dc01c4a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6af75ba8ff69afd1 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:06:09.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:09.111+0000 7ff0ef400640 1 -- 172.21.15.16:0/2726289871 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0dc0b0070 con 0x7ff0f006f920 2024-03-31T15:06:09.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:09.111+0000 7ff0ef400640 1 -- 172.21.15.16:0/2726289871 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff0dc0ab600 con 0x7ff0f006f920 2024-03-31T15:06:09.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:09.111+0000 7ff0ef400640 1 -- 172.21.15.16:0/2726289871 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0dc0ab9d0 con 0x7ff0f006f920 2024-03-31T15:06:09.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:09.111+0000 7ff0f7400640 1 -- 172.21.15.16:0/2726289871 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff0f006f920 msgr2=0x7ff0f006fd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:09.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:09.111+0000 7ff0f7400640 1 --2- 172.21.15.16:0/2726289871 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff0f006f920 0x7ff0f006fd80 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7ff0dc0097c0 tx=0x7ff0dc01c4a0 comp rx=0 tx=0).stop 2024-03-31T15:06:09.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:09.111+0000 7ff0f7400640 1 -- 172.21.15.16:0/2726289871 shutdown_connections 2024-03-31T15:06:09.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:09.111+0000 7ff0f7400640 1 --2- 172.21.15.16:0/2726289871 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff0f006f920 0x7ff0f006fd80 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:09.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:09.111+0000 7ff0f7400640 1 -- 172.21.15.16:0/2726289871 >> 172.21.15.16:0/2726289871 conn(0x7ff0f0067330 msgr2=0x7ff0f0067740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:06:09.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:09.111+0000 7ff0f7400640 1 -- 172.21.15.16:0/2726289871 shutdown_connections 2024-03-31T15:06:09.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:09.111+0000 7ff0f7400640 1 -- 172.21.15.16:0/2726289871 wait complete. 2024-03-31T15:06:09.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:09.111+0000 7ff0f7400640 1 Processor -- start 2024-03-31T15:06:09.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:09.111+0000 7ff0f7400640 1 -- start start 2024-03-31T15:06:09.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:09.111+0000 7ff0f7400640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff0f006f920 0x7ff0f01421e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:09.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:09.111+0000 7ff0f7400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff0f00d2010 0x7ff0f0142720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:09.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:09.111+0000 7ff0f7400640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff0f0145e40 0x7ff0f015c220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:09.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:09.111+0000 7ff0f7400640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7ff0f00d0ee0 con 0x7ff0f00d2010 2024-03-31T15:06:09.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:09.111+0000 7ff0f7400640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7ff0f00d06d0 con 0x7ff0f0145e40 2024-03-31T15:06:09.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:09.111+0000 7ff0f7400640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7ff0f00b50c0 con 0x7ff0f006f920 2024-03-31T15:06:09.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:09.115+0000 7ff0efe00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff0f00d2010 0x7ff0f0142720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:09.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:09.115+0000 7ff0efe00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff0f00d2010 0x7ff0f0142720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35052/0 (socket says 172.21.15.16:35052) 2024-03-31T15:06:09.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:09.115+0000 7ff0efe00640 1 -- 172.21.15.16:0/3499919089 learned_addr learned my addr 172.21.15.16:0/3499919089 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:06:09.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:09.115+0000 7ff0f4a00640 1 --2- 172.21.15.16:0/3499919089 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff0f006f920 0x7ff0f01421e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:09.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:09.115+0000 7ff0f5400640 1 --2- 172.21.15.16:0/3499919089 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff0f0145e40 0x7ff0f015c220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:09.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:09.115+0000 7ff0efe00640 1 -- 172.21.15.16:0/3499919089 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff0f006f920 msgr2=0x7ff0f01421e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:09.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:09.115+0000 7ff0efe00640 1 --2- 172.21.15.16:0/3499919089 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff0f006f920 0x7ff0f01421e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:09.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:09.115+0000 7ff0efe00640 1 -- 172.21.15.16:0/3499919089 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff0f0145e40 msgr2=0x7ff0f015c220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:09.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:09.115+0000 7ff0efe00640 1 --2- 172.21.15.16:0/3499919089 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff0f0145e40 0x7ff0f015c220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:09.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:09.115+0000 7ff0efe00640 1 -- 172.21.15.16:0/3499919089 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0f015e5d0 con 0x7ff0f00d2010 2024-03-31T15:06:09.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:09.115+0000 7ff0efe00640 1 --2- 172.21.15.16:0/3499919089 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff0f00d2010 0x7ff0f0142720 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7ff0dc0ae040 tx=0x7ff0dc01ace0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:06:09.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:09.115+0000 7ff0ecc00640 1 -- 172.21.15.16:0/3499919089 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0dc0b0050 con 0x7ff0f00d2010 2024-03-31T15:06:09.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:09.115+0000 7ff0ecc00640 1 -- 172.21.15.16:0/3499919089 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff0dc0bc070 con 0x7ff0f00d2010 2024-03-31T15:06:09.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:09.115+0000 7ff0ecc00640 1 -- 172.21.15.16:0/3499919089 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0dc006e20 con 0x7ff0f00d2010 2024-03-31T15:06:09.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:09.115+0000 7ff0f7400640 1 -- 172.21.15.16:0/3499919089 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff0f015e8b0 con 0x7ff0f00d2010 2024-03-31T15:06:09.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:09.115+0000 7ff0f7400640 1 -- 172.21.15.16:0/3499919089 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff0f015e280 con 0x7ff0f00d2010 2024-03-31T15:06:09.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:09.115+0000 7ff0f7400640 1 -- 172.21.15.16:0/3499919089 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff0f0069950 con 0x7ff0f00d2010 2024-03-31T15:06:09.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:09.115+0000 7ff0ecc00640 1 -- 172.21.15.16:0/3499919089 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7ff0dc0b5830 con 0x7ff0f00d2010 2024-03-31T15:06:09.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:09.119+0000 7ff0ecc00640 1 --2- 172.21.15.16:0/3499919089 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7ff0d0068380 0x7ff0d006a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:09.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:09.119+0000 7ff0ecc00640 1 -- 172.21.15.16:0/3499919089 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5709+0+0 (secure 0 0 0) 0x7ff0dc120e50 con 0x7ff0f00d2010 2024-03-31T15:06:09.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:09.119+0000 7ff0f4a00640 1 --2- 172.21.15.16:0/3499919089 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7ff0d0068380 0x7ff0d006a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:09.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:09.119+0000 7ff0f4a00640 1 --2- 172.21.15.16:0/3499919089 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7ff0d0068380 0x7ff0d006a840 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7ff0e40011f0 tx=0x7ff0e4009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:06:09.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:09.123+0000 7ff0ecc00640 1 -- 172.21.15.16:0/3499919089 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7ff0dc016090 con 0x7ff0f00d2010 2024-03-31T15:06:09.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:09.299+0000 7ff0f7400640 1 -- 172.21.15.16:0/3499919089 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7ff0f00cecc0 con 0x7ff0f00d2010 2024-03-31T15:06:09.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:09.299+0000 7ff0ecc00640 1 -- 172.21.15.16:0/3499919089 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v45) ==== 80+0+0 (secure 0 0 0) 0x7ff0dc0f0f70 con 0x7ff0f00d2010 2024-03-31T15:06:09.296 INFO:teuthology.orchestra.run.smithi016.stderr:listed 0 entries 2024-03-31T15:06:09.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:09.303+0000 7ff0f7400640 1 -- 172.21.15.16:0/3499919089 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7ff0d0068380 msgr2=0x7ff0d006a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:09.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:09.303+0000 7ff0f7400640 1 --2- 172.21.15.16:0/3499919089 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7ff0d0068380 0x7ff0d006a840 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7ff0e40011f0 tx=0x7ff0e4009040 comp rx=0 tx=0).stop 2024-03-31T15:06:09.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:09.303+0000 7ff0f7400640 1 -- 172.21.15.16:0/3499919089 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff0f00d2010 msgr2=0x7ff0f0142720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:09.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:09.303+0000 7ff0f7400640 1 --2- 172.21.15.16:0/3499919089 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff0f00d2010 0x7ff0f0142720 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7ff0dc0ae040 tx=0x7ff0dc01ace0 comp rx=0 tx=0).stop 2024-03-31T15:06:09.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:09.303+0000 7ff0f7400640 1 -- 172.21.15.16:0/3499919089 shutdown_connections 2024-03-31T15:06:09.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:09.303+0000 7ff0f7400640 1 --2- 172.21.15.16:0/3499919089 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7ff0d0068380 0x7ff0d006a840 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:09.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:09.303+0000 7ff0f7400640 1 --2- 172.21.15.16:0/3499919089 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff0f0145e40 0x7ff0f015c220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:09.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:09.303+0000 7ff0f7400640 1 --2- 172.21.15.16:0/3499919089 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff0f00d2010 0x7ff0f0142720 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:09.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:09.303+0000 7ff0f7400640 1 --2- 172.21.15.16:0/3499919089 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff0f006f920 0x7ff0f01421e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:09.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:09.303+0000 7ff0f7400640 1 -- 172.21.15.16:0/3499919089 >> 172.21.15.16:0/3499919089 conn(0x7ff0f0067330 msgr2=0x7ff0f015ab20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:06:09.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:09.303+0000 7ff0f7400640 1 -- 172.21.15.16:0/3499919089 shutdown_connections 2024-03-31T15:06:09.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:09.303+0000 7ff0f7400640 1 -- 172.21.15.16:0/3499919089 wait complete. 2024-03-31T15:06:09.316 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi016.front.sepia.ceph.com... 2024-03-31T15:06:09.316 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:06:09.317 DEBUG:teuthology.orchestra.run.smithi016:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-03-31T15:06:09.334 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi016.stderr:ceph-fuse[38076]: fuse finished with error 0 and tester_r 0 2024-03-31T15:06:09.337 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi016.front.sepia.ceph.com 2024-03-31T15:06:09.337 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:06:09.337 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-31T15:06:09.417 INFO:tasks.cephfs.mount:Terminating background process 2024-03-31T15:06:09.417 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-03-31T15:06:09.417 DEBUG:teuthology.orchestra.run.smithi089:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-31T15:06:09.426 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16354452k,nr_inodes=4088613,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275344k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=901 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275340k,nr_inodes=818835,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275344k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-31T15:06:09.426 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi089.front.sepia.ceph.com 2024-03-31T15:06:09.426 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:06:09.426 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-03-31T15:06:09.474 INFO:teuthology.orchestra.run.smithi089.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.1': No such file or directory 2024-03-31T15:06:09.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-31T15:06:09.475 INFO:tasks.cephfs.mount:Terminating background process 2024-03-31T15:06:09.475 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-03-31T15:06:09.475 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-31T15:06:09.484 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16357708k,nr_inodes=4089427,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275996k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=6539 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275992k,nr_inodes=818998,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275996k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-31T15:06:09.484 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi016.front.sepia.ceph.com 2024-03-31T15:06:09.484 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:06:09.484 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-03-31T15:06:09.535 INFO:teuthology.orchestra.run.smithi016.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2024-03-31T15:06:09.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-31T15:06:09.536 INFO:tasks.cephfs.mount:Terminating background process 2024-03-31T15:06:09.536 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-03-31T15:06:09.536 DEBUG:teuthology.orchestra.run.smithi089:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-31T15:06:09.544 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16354452k,nr_inodes=4088613,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275344k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=901 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275340k,nr_inodes=818835,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275344k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-31T15:06:09.545 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi089.front.sepia.ceph.com 2024-03-31T15:06:09.545 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:06:09.545 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-03-31T15:06:09.593 INFO:teuthology.orchestra.run.smithi089.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2024-03-31T15:06:09.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-31T15:06:09.595 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-31T15:06:09.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:09.739+0000 7f7471a00640 1 Processor -- start 2024-03-31T15:06:09.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:09.739+0000 7f7471a00640 1 -- start start 2024-03-31T15:06:09.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:09.739+0000 7f7471a00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f746c15f650 0x7f746c06f920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:09.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:09.739+0000 7f7471a00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f746c06fe60 0x7f746c162180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:09.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:09.739+0000 7f7471a00640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f746c0cbe90 con 0x7f746c06dbc0 2024-03-31T15:06:09.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:09.739+0000 7f7471a00640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f746c0702d0 con 0x7f746c15f650 2024-03-31T15:06:09.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:09.739+0000 7f7471a00640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f746c15f1c0 con 0x7f746c06fe60 2024-03-31T15:06:09.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:09.743+0000 7f7463e00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f746c15f650 0x7f746c06f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:09.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:09.743+0000 7f7463e00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f746c15f650 0x7f746c06f920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35060/0 (socket says 172.21.15.16:35060) 2024-03-31T15:06:09.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:09.743+0000 7f7463e00640 1 -- 172.21.15.16:0/1155666495 learned_addr learned my addr 172.21.15.16:0/1155666495 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:06:09.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:09.743+0000 7f746b400640 1 --2- 172.21.15.16:0/1155666495 >> v2:172.21.15.89:3300/0 conn(0x7f746c06fe60 0x7f746c162180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:09.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:09.743+0000 7f7463e00640 1 -- 172.21.15.16:0/1155666495 >> v2:172.21.15.89:3300/0 conn(0x7f746c06fe60 msgr2=0x7f746c162180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:06:09.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:09.743+0000 7f7463e00640 1 --2- 172.21.15.16:0/1155666495 >> v2:172.21.15.89:3300/0 conn(0x7f746c06fe60 0x7f746c162180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:09.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:09.743+0000 7f7463e00640 1 -- 172.21.15.16:0/1155666495 >> v1:172.21.15.89:6789/0 conn(0x7f746c06dbc0 legacy=0x7f746c0cb440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:09.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:09.743+0000 7f7463e00640 1 -- 172.21.15.16:0/1155666495 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f746c0cc170 con 0x7f746c15f650 2024-03-31T15:06:09.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:09.743+0000 7f7463e00640 1 --2- 172.21.15.16:0/1155666495 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f746c15f650 0x7f746c06f920 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f746c073be0 tx=0x7f745401c460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=337adcf5c0967937 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:06:09.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:09.743+0000 7f746a000640 1 -- 172.21.15.16:0/1155666495 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74540ab680 con 0x7f746c15f650 2024-03-31T15:06:09.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:09.743+0000 7f746a000640 1 -- 172.21.15.16:0/1155666495 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f74540b0070 con 0x7f746c15f650 2024-03-31T15:06:09.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:09.743+0000 7f746a000640 1 -- 172.21.15.16:0/1155666495 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74540aba40 con 0x7f746c15f650 2024-03-31T15:06:09.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:09.747+0000 7f7471a00640 1 -- 172.21.15.16:0/1155666495 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f746c15f650 msgr2=0x7f746c06f920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:09.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:09.747+0000 7f7471a00640 1 --2- 172.21.15.16:0/1155666495 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f746c15f650 0x7f746c06f920 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f746c073be0 tx=0x7f745401c460 comp rx=0 tx=0).stop 2024-03-31T15:06:09.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:09.747+0000 7f7471a00640 1 -- 172.21.15.16:0/1155666495 shutdown_connections 2024-03-31T15:06:09.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:09.747+0000 7f7471a00640 1 --2- 172.21.15.16:0/1155666495 >> v2:172.21.15.89:3300/0 conn(0x7f746c06fe60 0x7f746c162180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:09.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:09.747+0000 7f7471a00640 1 --2- 172.21.15.16:0/1155666495 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f746c15f650 0x7f746c06f920 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:09.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:09.747+0000 7f7471a00640 1 -- 172.21.15.16:0/1155666495 >> 172.21.15.16:0/1155666495 conn(0x7f746c067330 msgr2=0x7f746c067740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:06:09.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:09.747+0000 7f7471a00640 1 -- 172.21.15.16:0/1155666495 shutdown_connections 2024-03-31T15:06:09.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:09.747+0000 7f7471a00640 1 -- 172.21.15.16:0/1155666495 wait complete. 2024-03-31T15:06:09.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:09.747+0000 7f7471a00640 1 Processor -- start 2024-03-31T15:06:09.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:09.747+0000 7f7471a00640 1 -- start start 2024-03-31T15:06:09.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:09.747+0000 7f7471a00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f746c06dbc0 0x7f746c0b7540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:09.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:09.747+0000 7f7471a00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f746c06fe60 0x7f746c0cdbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:09.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:09.747+0000 7f7471a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f746c15f650 0x7f746c1406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:09.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:09.747+0000 7f7471a00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f746c06dfa0 con 0x7f746c15f650 2024-03-31T15:06:09.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:09.747+0000 7f7471a00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f746c0cbb50 con 0x7f746c06dbc0 2024-03-31T15:06:09.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:09.747+0000 7f7471a00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f746c06c950 con 0x7f746c06fe60 2024-03-31T15:06:09.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:09.747+0000 7f746b400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f746c15f650 0x7f746c1406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:09.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:09.747+0000 7f746b400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f746c15f650 0x7f746c1406d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35062/0 (socket says 172.21.15.16:35062) 2024-03-31T15:06:09.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:09.747+0000 7f746b400640 1 -- 172.21.15.16:0/1559084405 learned_addr learned my addr 172.21.15.16:0/1559084405 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:06:09.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:09.747+0000 7f746aa00640 1 --2- 172.21.15.16:0/1559084405 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f746c06dbc0 0x7f746c0b7540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:09.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:09.747+0000 7f7463e00640 1 --2- 172.21.15.16:0/1559084405 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f746c06fe60 0x7f746c0cdbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:09.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:09.747+0000 7f746b400640 1 -- 172.21.15.16:0/1559084405 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f746c06fe60 msgr2=0x7f746c0cdbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:09.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:09.747+0000 7f746b400640 1 --2- 172.21.15.16:0/1559084405 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f746c06fe60 0x7f746c0cdbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:09.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:09.747+0000 7f746b400640 1 -- 172.21.15.16:0/1559084405 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f746c06dbc0 msgr2=0x7f746c0b7540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:09.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:09.747+0000 7f746b400640 1 --2- 172.21.15.16:0/1559084405 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f746c06dbc0 0x7f746c0b7540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:09.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:09.747+0000 7f746b400640 1 -- 172.21.15.16:0/1559084405 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f746c159250 con 0x7f746c15f650 2024-03-31T15:06:09.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:09.751+0000 7f746b400640 1 --2- 172.21.15.16:0/1559084405 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f746c15f650 0x7f746c1406d0 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f746400efc0 tx=0x7f746400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:06:09.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:09.751+0000 7f7462a00640 1 -- 172.21.15.16:0/1559084405 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7464018020 con 0x7f746c15f650 2024-03-31T15:06:09.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:09.751+0000 7f7462a00640 1 -- 172.21.15.16:0/1559084405 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7464014480 con 0x7f746c15f650 2024-03-31T15:06:09.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:09.751+0000 7f7462a00640 1 -- 172.21.15.16:0/1559084405 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7464014870 con 0x7f746c15f650 2024-03-31T15:06:09.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:09.751+0000 7f7471a00640 1 -- 172.21.15.16:0/1559084405 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f746c159530 con 0x7f746c15f650 2024-03-31T15:06:09.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:09.751+0000 7f7471a00640 1 -- 172.21.15.16:0/1559084405 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f746c159af0 con 0x7f746c15f650 2024-03-31T15:06:09.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:09.751+0000 7f7471a00640 1 -- 172.21.15.16:0/1559084405 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f746c069950 con 0x7f746c15f650 2024-03-31T15:06:09.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:09.751+0000 7f7462a00640 1 -- 172.21.15.16:0/1559084405 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f746401b020 con 0x7f746c15f650 2024-03-31T15:06:09.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:09.751+0000 7f7462a00640 1 --2- 172.21.15.16:0/1559084405 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f7448068380 0x7f744806a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:09.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:09.751+0000 7f7462a00640 1 -- 172.21.15.16:0/1559084405 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5709+0+0 (secure 0 0 0) 0x7f74640872e0 con 0x7f746c15f650 2024-03-31T15:06:09.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:09.751+0000 7f746aa00640 1 --2- 172.21.15.16:0/1559084405 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f7448068380 0x7f744806a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:09.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:09.755+0000 7f746aa00640 1 --2- 172.21.15.16:0/1559084405 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f7448068380 0x7f744806a840 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f746c15c370 tx=0x7f745c009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:06:09.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:09.755+0000 7f7462a00640 1 -- 172.21.15.16:0/1559084405 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f7464022070 con 0x7f746c15f650 2024-03-31T15:06:09.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:09.955+0000 7f7471a00640 1 -- 172.21.15.16:0/1559084405 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f746c0cf380 con 0x7f746c15f650 2024-03-31T15:06:09.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:09.955+0000 7f7462a00640 1 -- 172.21.15.16:0/1559084405 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+5175 (secure 0 0 0) 0x7f7464010040 con 0x7f746c15f650 2024-03-31T15:06:09.951 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:06:09.951 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":25,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/285985105","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6835","nonce":285985105},{"type":"v1","addr":"172.21.15.89:6837","nonce":285985105}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20},{"gid":5689,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6836/1615931247","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":1615931247},{"type":"v1","addr":"172.21.15.89:6836","nonce":1615931247}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20},{"gid":5694,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/512187852","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6833","nonce":512187852},{"type":"v1","addr":"172.21.15.16:6835","nonce":512187852}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-31T15:05:26.461908+0000","modified":"2024-03-31T15:05:29.491163+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5697},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5697":{"gid":5697,"name":"a","rank":0,"incarnation":21,"state":"up:active","state_seq":3,"addr":"172.21.15.16:6834/2768973409","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":2768973409},{"type":"v1","addr":"172.21.15.16:6834","nonce":2768973409}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5697,"qdb_cluster":[5697]},"id":3}]} 2024-03-31T15:06:09.951 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 25 2024-03-31T15:06:09.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:09.959+0000 7f7471a00640 1 -- 172.21.15.16:0/1559084405 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f7448068380 msgr2=0x7f744806a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:09.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:09.959+0000 7f7471a00640 1 --2- 172.21.15.16:0/1559084405 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f7448068380 0x7f744806a840 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f746c15c370 tx=0x7f745c009040 comp rx=0 tx=0).stop 2024-03-31T15:06:09.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:09.959+0000 7f7471a00640 1 -- 172.21.15.16:0/1559084405 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f746c15f650 msgr2=0x7f746c1406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:09.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:09.959+0000 7f7471a00640 1 --2- 172.21.15.16:0/1559084405 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f746c15f650 0x7f746c1406d0 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f746400efc0 tx=0x7f746400c490 comp rx=0 tx=0).stop 2024-03-31T15:06:09.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:09.959+0000 7f7471a00640 1 -- 172.21.15.16:0/1559084405 shutdown_connections 2024-03-31T15:06:09.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:09.959+0000 7f7471a00640 1 --2- 172.21.15.16:0/1559084405 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f7448068380 0x7f744806a840 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:09.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:09.959+0000 7f7471a00640 1 --2- 172.21.15.16:0/1559084405 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f746c15f650 0x7f746c1406d0 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:09.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:09.959+0000 7f7471a00640 1 --2- 172.21.15.16:0/1559084405 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f746c06fe60 0x7f746c0cdbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:09.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:09.959+0000 7f7471a00640 1 --2- 172.21.15.16:0/1559084405 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f746c06dbc0 0x7f746c0b7540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:09.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:09.959+0000 7f7471a00640 1 -- 172.21.15.16:0/1559084405 >> 172.21.15.16:0/1559084405 conn(0x7f746c067330 msgr2=0x7f746c15fd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:06:09.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:09.959+0000 7f7471a00640 1 -- 172.21.15.16:0/1559084405 shutdown_connections 2024-03-31T15:06:09.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:09.959+0000 7f7471a00640 1 -- 172.21.15.16:0/1559084405 wait complete. 2024-03-31T15:06:09.973 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-31T15:06:10.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:10.119+0000 7f7b83a00640 1 Processor -- start 2024-03-31T15:06:10.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:10.119+0000 7f7b83a00640 1 -- start start 2024-03-31T15:06:10.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:10.119+0000 7f7b83a00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f7b7c06dbc0 0x7f7b7c15f650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:10.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:10.119+0000 7f7b83a00640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7b7c0cbe90 con 0x7f7b7c0cb440 2024-03-31T15:06:10.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:10.119+0000 7f7b83a00640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7b7c0cc170 con 0x7f7b7c070030 2024-03-31T15:06:10.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:10.119+0000 7f7b83a00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f7b7c15fb90 con 0x7f7b7c06dbc0 2024-03-31T15:06:10.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:10.119+0000 7f7b7be00640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f7b7c0cb440 0x7f7b7c06f920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:37144/0 (socket says 172.21.15.16:37144) 2024-03-31T15:06:10.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:10.119+0000 7f7b7be00640 1 -- 172.21.15.16:0/2874843692 learned_addr learned my addr 172.21.15.16:0/2874843692 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:06:10.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:10.119+0000 7f7b81000640 1 --2- 172.21.15.16:0/2874843692 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f7b7c06dbc0 0x7f7b7c15f650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:10.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:10.119+0000 7f7b7b400640 1 -- 172.21.15.16:0/2874843692 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3081483321 0 0) 0x7f7b7c0cbe90 con 0x7f7b7c0cb440 2024-03-31T15:06:10.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:10.123+0000 7f7b7b400640 1 -- 172.21.15.16:0/2874843692 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7b64003530 con 0x7f7b7c0cb440 2024-03-31T15:06:10.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:10.123+0000 7f7b7b400640 1 -- 172.21.15.16:0/2874843692 <== mon.1 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2273374522 0 0) 0x7f7b7c0cc170 con 0x7f7b7c070030 2024-03-31T15:06:10.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:10.123+0000 7f7b7b400640 1 -- 172.21.15.16:0/2874843692 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7b7c0cbe90 con 0x7f7b7c070030 2024-03-31T15:06:10.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:10.123+0000 7f7b81000640 1 -- 172.21.15.16:0/2874843692 >> v1:172.21.15.89:6789/0 conn(0x7f7b7c070030 legacy=0x7f7b7c162180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:10.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:10.123+0000 7f7b81000640 1 -- 172.21.15.16:0/2874843692 >> v1:172.21.15.16:6789/0 conn(0x7f7b7c0cb440 legacy=0x7f7b7c06f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:10.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:10.123+0000 7f7b81000640 1 -- 172.21.15.16:0/2874843692 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b7c15ee50 con 0x7f7b7c06dbc0 2024-03-31T15:06:10.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:10.123+0000 7f7b81000640 1 --2- 172.21.15.16:0/2874843692 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f7b7c06dbc0 0x7f7b7c15f650 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f7b700097f0 tx=0x7f7b7001c450 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ec1d0a1f1c968e4a server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:06:10.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:10.123+0000 7f7b7b400640 1 -- 172.21.15.16:0/2874843692 <== mon.2 v2:172.21.15.89:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b700b0070 con 0x7f7b7c06dbc0 2024-03-31T15:06:10.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:10.123+0000 7f7b7b400640 1 -- 172.21.15.16:0/2874843692 <== mon.2 v2:172.21.15.89:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7b700ab600 con 0x7f7b7c06dbc0 2024-03-31T15:06:10.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:10.123+0000 7f7b7b400640 1 -- 172.21.15.16:0/2874843692 <== mon.2 v2:172.21.15.89:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b700ab9d0 con 0x7f7b7c06dbc0 2024-03-31T15:06:10.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:10.123+0000 7f7b83a00640 1 -- 172.21.15.16:0/2874843692 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f7b7c06dbc0 msgr2=0x7f7b7c15f650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:10.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:10.123+0000 7f7b83a00640 1 --2- 172.21.15.16:0/2874843692 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f7b7c06dbc0 0x7f7b7c15f650 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f7b700097f0 tx=0x7f7b7001c450 comp rx=0 tx=0).stop 2024-03-31T15:06:10.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:10.123+0000 7f7b83a00640 1 -- 172.21.15.16:0/2874843692 shutdown_connections 2024-03-31T15:06:10.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:10.123+0000 7f7b83a00640 1 --2- 172.21.15.16:0/2874843692 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f7b7c06dbc0 0x7f7b7c15f650 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:10.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:10.127+0000 7f7b83a00640 1 -- 172.21.15.16:0/2874843692 >> 172.21.15.16:0/2874843692 conn(0x7f7b7c067330 msgr2=0x7f7b7c067740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:06:10.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:10.127+0000 7f7b83a00640 1 -- 172.21.15.16:0/2874843692 shutdown_connections 2024-03-31T15:06:10.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:10.127+0000 7f7b83a00640 1 -- 172.21.15.16:0/2874843692 wait complete. 2024-03-31T15:06:10.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:10.127+0000 7f7b83a00640 1 Processor -- start 2024-03-31T15:06:10.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:10.127+0000 7f7b83a00640 1 -- start start 2024-03-31T15:06:10.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:10.127+0000 7f7b83a00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7b7c06dbc0 0x7f7b7c0b3830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:10.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:10.127+0000 7f7b83a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7b7c070030 0x7f7b7c140610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:10.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:10.127+0000 7f7b83a00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f7b7c0cb440 0x7f7b7c160d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:10.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:10.127+0000 7f7b83a00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f7b7c06cad0 con 0x7f7b7c070030 2024-03-31T15:06:10.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:10.127+0000 7f7b83a00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f7b7c06c950 con 0x7f7b7c06dbc0 2024-03-31T15:06:10.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:10.127+0000 7f7b83a00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f7b7c16cd40 con 0x7f7b7c0cb440 2024-03-31T15:06:10.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:10.127+0000 7f7b7be00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7b7c070030 0x7f7b7c140610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:10.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:10.127+0000 7f7b7be00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7b7c070030 0x7f7b7c140610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35070/0 (socket says 172.21.15.16:35070) 2024-03-31T15:06:10.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:10.127+0000 7f7b7be00640 1 -- 172.21.15.16:0/1332270805 learned_addr learned my addr 172.21.15.16:0/1332270805 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:06:10.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:10.127+0000 7f7b81000640 1 --2- 172.21.15.16:0/1332270805 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7b7c06dbc0 0x7f7b7c0b3830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:10.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:10.127+0000 7f7b81a00640 1 --2- 172.21.15.16:0/1332270805 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f7b7c0cb440 0x7f7b7c160d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:10.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:10.127+0000 7f7b7be00640 1 -- 172.21.15.16:0/1332270805 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f7b7c0cb440 msgr2=0x7f7b7c160d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:10.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:10.127+0000 7f7b7be00640 1 --2- 172.21.15.16:0/1332270805 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f7b7c0cb440 0x7f7b7c160d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:10.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:10.127+0000 7f7b7be00640 1 -- 172.21.15.16:0/1332270805 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7b7c06dbc0 msgr2=0x7f7b7c0b3830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:10.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:10.127+0000 7f7b7be00640 1 --2- 172.21.15.16:0/1332270805 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7b7c06dbc0 0x7f7b7c0b3830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:10.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:10.127+0000 7f7b7be00640 1 -- 172.21.15.16:0/1332270805 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b7c0b6b90 con 0x7f7b7c070030 2024-03-31T15:06:10.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:10.127+0000 7f7b7be00640 1 --2- 172.21.15.16:0/1332270805 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7b7c070030 0x7f7b7c140610 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f7b6800c420 tx=0x7f7b6800c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:06:10.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:10.127+0000 7f7b78c00640 1 -- 172.21.15.16:0/1332270805 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b68017070 con 0x7f7b7c070030 2024-03-31T15:06:10.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:10.127+0000 7f7b78c00640 1 -- 172.21.15.16:0/1332270805 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7b68005d10 con 0x7f7b7c070030 2024-03-31T15:06:10.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:10.127+0000 7f7b78c00640 1 -- 172.21.15.16:0/1332270805 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b68007310 con 0x7f7b7c070030 2024-03-31T15:06:10.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:10.127+0000 7f7b83a00640 1 -- 172.21.15.16:0/1332270805 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7b7c0b5af0 con 0x7f7b7c070030 2024-03-31T15:06:10.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:10.131+0000 7f7b83a00640 1 -- 172.21.15.16:0/1332270805 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7b7c0b60b0 con 0x7f7b7c070030 2024-03-31T15:06:10.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:10.131+0000 7f7b83a00640 1 -- 172.21.15.16:0/1332270805 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7b7c069950 con 0x7f7b7c070030 2024-03-31T15:06:10.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:10.131+0000 7f7b78c00640 1 -- 172.21.15.16:0/1332270805 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f7b6801a070 con 0x7f7b7c070030 2024-03-31T15:06:10.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:10.131+0000 7f7b78c00640 1 --2- 172.21.15.16:0/1332270805 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f7b5c068380 0x7f7b5c06a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:10.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:10.131+0000 7f7b78c00640 1 -- 172.21.15.16:0/1332270805 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5709+0+0 (secure 0 0 0) 0x7f7b68085e60 con 0x7f7b7c070030 2024-03-31T15:06:10.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:10.131+0000 7f7b81000640 1 --2- 172.21.15.16:0/1332270805 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f7b5c068380 0x7f7b5c06a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:10.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:10.135+0000 7f7b81000640 1 --2- 172.21.15.16:0/1332270805 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f7b5c068380 0x7f7b5c06a840 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f7b700ad000 tx=0x7f7b700a6000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:06:10.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:10.135+0000 7f7b78c00640 1 -- 172.21.15.16:0/1332270805 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f7b68055cb0 con 0x7f7b7c070030 2024-03-31T15:06:10.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:10.303+0000 7f7b83a00640 1 -- 172.21.15.16:0/1332270805 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f7b7c0cf380 con 0x7f7b7c070030 2024-03-31T15:06:10.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:10.303+0000 7f7b78c00640 1 -- 172.21.15.16:0/1332270805 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v45) ==== 74+0+16541 (secure 0 0 0) 0x7f7b68059fb0 con 0x7f7b7c070030 2024-03-31T15:06:10.300 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:06:10.300 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":45,"fsid":"286eb5da-fb47-4550-b290-d9519c13b9a5","created":"2024-03-31T15:03:11.438952+0000","modified":"2024-03-31T15:05:28.074367+0000","last_up_change":"2024-03-31T15:03:22.885535+0000","last_in_change":"2024-03-31T15:03:13.941667+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-31T15:03:22.550787+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2024-03-31T15:05:24.856239+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"44","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2024-03-31T15:05:25.417826+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"45","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"92a66450-ef19-40d8-ac3d-d760b23b0bda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6801","nonce":932712244}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6803","nonce":932712244}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6807","nonce":932712244}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6805","nonce":932712244}]},"public_addr":"172.21.15.16:6801/932712244","cluster_addr":"172.21.15.16:6803/932712244","heartbeat_back_addr":"172.21.15.16:6807/932712244","heartbeat_front_addr":"172.21.15.16:6805/932712244","state":["exists","up"]},{"osd":1,"uuid":"da9d1f2b-1968-4959-b843-faa231988e92","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6810","nonce":1398737533}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6811","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6813","nonce":1398737533}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6821","nonce":1398737533}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6815","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6816","nonce":1398737533}]},"public_addr":"172.21.15.16:6810/1398737533","cluster_addr":"172.21.15.16:6813/1398737533","heartbeat_back_addr":"172.21.15.16:6821/1398737533","heartbeat_front_addr":"172.21.15.16:6816/1398737533","state":["exists","up"]},{"osd":2,"uuid":"01a73662-9a21-4b1f-bcbb-42b9636a4638","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6819","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6822","nonce":3018526646}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6825","nonce":3018526646}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6831","nonce":3018526646}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6827","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6829","nonce":3018526646}]},"public_addr":"172.21.15.16:6822/3018526646","cluster_addr":"172.21.15.16:6825/3018526646","heartbeat_back_addr":"172.21.15.16:6831/3018526646","heartbeat_front_addr":"172.21.15.16:6829/3018526646","state":["exists","up"]},{"osd":3,"uuid":"c7771fd3-e18e-4500-ac38-23cf10ebcd75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6809","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6812","nonce":3360352974}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6817","nonce":3360352974}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6828","nonce":3360352974}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6823","nonce":3360352974}]},"public_addr":"172.21.15.16:6812/3360352974","cluster_addr":"172.21.15.16:6817/3360352974","heartbeat_back_addr":"172.21.15.16:6828/3360352974","heartbeat_front_addr":"172.21.15.16:6823/3360352974","state":["exists","up"]},{"osd":4,"uuid":"88260730-1482-4fbd-a40f-cb1124e9df4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6801","nonce":1856844856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6803","nonce":1856844856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6807","nonce":1856844856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6805","nonce":1856844856}]},"public_addr":"172.21.15.89:6801/1856844856","cluster_addr":"172.21.15.89:6803/1856844856","heartbeat_back_addr":"172.21.15.89:6807/1856844856","heartbeat_front_addr":"172.21.15.89:6805/1856844856","state":["exists","up"]},{"osd":5,"uuid":"4cb2c00c-fae0-4dd8-9d27-ec4f0093ccf7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6814","nonce":50395527}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6817","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6820","nonce":50395527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6831","nonce":50395527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6823","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6826","nonce":50395527}]},"public_addr":"172.21.15.89:6814/50395527","cluster_addr":"172.21.15.89:6820/50395527","heartbeat_back_addr":"172.21.15.89:6831/50395527","heartbeat_front_addr":"172.21.15.89:6826/50395527","state":["exists","up"]},{"osd":6,"uuid":"81ce0c19-2554-4cae-acbe-66a05e9e9b9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6809","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6812","nonce":3470471752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6815","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6818","nonce":3470471752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6827","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6829","nonce":3470471752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6821","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6824","nonce":3470471752}]},"public_addr":"172.21.15.89:6812/3470471752","cluster_addr":"172.21.15.89:6818/3470471752","heartbeat_back_addr":"172.21.15.89:6829/3470471752","heartbeat_front_addr":"172.21.15.89:6824/3470471752","state":["exists","up"]},{"osd":7,"uuid":"85e4313b-d37d-442e-8042-153bf20a847d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6811","nonce":1796778630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6813","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6816","nonce":1796778630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6825","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6828","nonce":1796778630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6819","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6822","nonce":1796778630}]},"public_addr":"172.21.15.89:6811/1796778630","cluster_addr":"172.21.15.89:6816/1796778630","heartbeat_back_addr":"172.21.15.89:6828/1796778630","heartbeat_front_addr":"172.21.15.89:6822/1796778630","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.439088+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.529970+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.680512+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:21.044161+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.582111+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.536558+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.409522+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.590229+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-31T15:06:10.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:10.307+0000 7f7b83a00640 1 -- 172.21.15.16:0/1332270805 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f7b5c068380 msgr2=0x7f7b5c06a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:10.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:10.307+0000 7f7b83a00640 1 --2- 172.21.15.16:0/1332270805 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f7b5c068380 0x7f7b5c06a840 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f7b700ad000 tx=0x7f7b700a6000 comp rx=0 tx=0).stop 2024-03-31T15:06:10.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:10.307+0000 7f7b83a00640 1 -- 172.21.15.16:0/1332270805 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7b7c070030 msgr2=0x7f7b7c140610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:10.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:10.307+0000 7f7b83a00640 1 --2- 172.21.15.16:0/1332270805 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7b7c070030 0x7f7b7c140610 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f7b6800c420 tx=0x7f7b6800c8e0 comp rx=0 tx=0).stop 2024-03-31T15:06:10.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:10.307+0000 7f7b83a00640 1 -- 172.21.15.16:0/1332270805 shutdown_connections 2024-03-31T15:06:10.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:10.307+0000 7f7b83a00640 1 --2- 172.21.15.16:0/1332270805 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f7b5c068380 0x7f7b5c06a840 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:10.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:10.307+0000 7f7b83a00640 1 --2- 172.21.15.16:0/1332270805 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f7b7c0cb440 0x7f7b7c160d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:10.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:10.307+0000 7f7b83a00640 1 --2- 172.21.15.16:0/1332270805 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7b7c070030 0x7f7b7c140610 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:10.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:10.307+0000 7f7b83a00640 1 --2- 172.21.15.16:0/1332270805 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7b7c06dbc0 0x7f7b7c0b3830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:10.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:10.307+0000 7f7b83a00640 1 -- 172.21.15.16:0/1332270805 >> 172.21.15.16:0/1332270805 conn(0x7f7b7c067330 msgr2=0x7f7b7c15dcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:06:10.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:10.307+0000 7f7b83a00640 1 -- 172.21.15.16:0/1332270805 shutdown_connections 2024-03-31T15:06:10.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:10.307+0000 7f7b83a00640 1 -- 172.21.15.16:0/1332270805 wait complete. 2024-03-31T15:06:10.321 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-31T15:03:22.550787+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-31T15:05:24.856239+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '44', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 7, 'pool_name': 'cephfs_data', 'create_time': '2024-03-31T15:05:25.417826+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '45', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-31T15:06:10.322 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-31T15:06:10.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:10.471+0000 7fc96e400640 1 Processor -- start 2024-03-31T15:06:10.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:10.471+0000 7fc96e400640 1 -- start start 2024-03-31T15:06:10.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:10.471+0000 7fc96e400640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fc96806f920 0x7fc96806fd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:10.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:10.471+0000 7fc96e400640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fc9680d0a60 0x7fc9680cea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:10.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:10.471+0000 7fc96e400640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc968070240 con 0x7fc9680cef80 2024-03-31T15:06:10.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:10.471+0000 7fc96e400640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7fc96813f670 con 0x7fc96806f920 2024-03-31T15:06:10.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:10.471+0000 7fc96e400640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7fc96815ca70 con 0x7fc9680d0a60 2024-03-31T15:06:10.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:10.471+0000 7fc967e00640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fc9680cef80 0x7fc968070e60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:37148/0 (socket says 172.21.15.16:37148) 2024-03-31T15:06:10.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:10.471+0000 7fc967400640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fc96806f920 0x7fc96806fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:10.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:10.471+0000 7fc967e00640 1 -- 172.21.15.16:0/68515050 learned_addr learned my addr 172.21.15.16:0/68515050 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:06:10.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:10.475+0000 7fc966a00640 1 --2- 172.21.15.16:0/68515050 >> v2:172.21.15.89:3300/0 conn(0x7fc9680d0a60 0x7fc9680cea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:10.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:10.475+0000 7fc966000640 1 -- 172.21.15.16:0/68515050 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2748787732 0 0) 0x7fc968070240 con 0x7fc9680cef80 2024-03-31T15:06:10.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:10.475+0000 7fc967400640 1 -- 172.21.15.16:0/68515050 >> v2:172.21.15.89:3300/0 conn(0x7fc9680d0a60 msgr2=0x7fc9680cea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:06:10.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:10.475+0000 7fc967400640 1 --2- 172.21.15.16:0/68515050 >> v2:172.21.15.89:3300/0 conn(0x7fc9680d0a60 0x7fc9680cea40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:10.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:10.475+0000 7fc967400640 1 -- 172.21.15.16:0/68515050 >> v1:172.21.15.16:6789/0 conn(0x7fc9680cef80 legacy=0x7fc968070e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:10.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:10.475+0000 7fc967400640 1 -- 172.21.15.16:0/68515050 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc96815cd50 con 0x7fc96806f920 2024-03-31T15:06:10.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:10.475+0000 7fc966a00640 1 --2- 172.21.15.16:0/68515050 >> v2:172.21.15.89:3300/0 conn(0x7fc9680d0a60 0x7fc9680cea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:06:10.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:10.479+0000 7fc967400640 1 --2- 172.21.15.16:0/68515050 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc96806f920 0x7fc96806fd00 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7fc9680736b0 tx=0x7fc95801c3f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a6a9c8a18ec15b54 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:06:10.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:10.479+0000 7fc966000640 1 -- 172.21.15.16:0/68515050 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9580b0070 con 0x7fc96806f920 2024-03-31T15:06:10.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:10.479+0000 7fc966000640 1 -- 172.21.15.16:0/68515050 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc9580ab600 con 0x7fc96806f920 2024-03-31T15:06:10.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:10.479+0000 7fc966000640 1 -- 172.21.15.16:0/68515050 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9580ab9d0 con 0x7fc96806f920 2024-03-31T15:06:10.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:10.479+0000 7fc96e400640 1 -- 172.21.15.16:0/68515050 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc96806f920 msgr2=0x7fc96806fd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:10.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:10.479+0000 7fc96e400640 1 --2- 172.21.15.16:0/68515050 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc96806f920 0x7fc96806fd00 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7fc9680736b0 tx=0x7fc95801c3f0 comp rx=0 tx=0).stop 2024-03-31T15:06:10.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:10.479+0000 7fc96e400640 1 -- 172.21.15.16:0/68515050 shutdown_connections 2024-03-31T15:06:10.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:10.479+0000 7fc96e400640 1 --2- 172.21.15.16:0/68515050 >> v2:172.21.15.89:3300/0 conn(0x7fc9680d0a60 0x7fc9680cea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:10.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:10.479+0000 7fc96e400640 1 --2- 172.21.15.16:0/68515050 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc96806f920 0x7fc96806fd00 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:10.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:10.479+0000 7fc96e400640 1 -- 172.21.15.16:0/68515050 >> 172.21.15.16:0/68515050 conn(0x7fc968067930 msgr2=0x7fc9680b3ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:06:10.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:10.479+0000 7fc96e400640 1 -- 172.21.15.16:0/68515050 shutdown_connections 2024-03-31T15:06:10.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:10.479+0000 7fc96e400640 1 -- 172.21.15.16:0/68515050 wait complete. 2024-03-31T15:06:10.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:10.479+0000 7fc96e400640 1 Processor -- start 2024-03-31T15:06:10.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:10.479+0000 7fc96e400640 1 -- start start 2024-03-31T15:06:10.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:10.483+0000 7fc96e400640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fc96806f920 0x7fc96806d2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:10.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:10.483+0000 7fc96e400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc9680cef80 0x7fc968152180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:10.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:10.483+0000 7fc96e400640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc9680d0a60 0x7fc96815f730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:10.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:10.483+0000 7fc96e400640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fc9681690b0 con 0x7fc9680cef80 2024-03-31T15:06:10.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:10.483+0000 7fc96e400640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fc9680cf390 con 0x7fc9680d0a60 2024-03-31T15:06:10.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:10.483+0000 7fc96e400640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fc968169230 con 0x7fc96806f920 2024-03-31T15:06:10.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:10.483+0000 7fc966a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc9680cef80 0x7fc968152180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:10.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:10.483+0000 7fc966a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc9680cef80 0x7fc968152180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35088/0 (socket says 172.21.15.16:35088) 2024-03-31T15:06:10.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:10.483+0000 7fc966a00640 1 -- 172.21.15.16:0/3977106011 learned_addr learned my addr 172.21.15.16:0/3977106011 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:06:10.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:10.483+0000 7fc967400640 1 --2- 172.21.15.16:0/3977106011 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fc96806f920 0x7fc96806d2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:10.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:10.483+0000 7fc967e00640 1 --2- 172.21.15.16:0/3977106011 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc9680d0a60 0x7fc96815f730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:10.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:10.483+0000 7fc966a00640 1 -- 172.21.15.16:0/3977106011 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fc96806f920 msgr2=0x7fc96806d2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:10.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:10.483+0000 7fc966a00640 1 --2- 172.21.15.16:0/3977106011 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fc96806f920 0x7fc96806d2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:10.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:10.483+0000 7fc966a00640 1 -- 172.21.15.16:0/3977106011 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc9680d0a60 msgr2=0x7fc96815f730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:10.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:10.483+0000 7fc966a00640 1 --2- 172.21.15.16:0/3977106011 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc9680d0a60 0x7fc96815f730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:10.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:10.483+0000 7fc966a00640 1 -- 172.21.15.16:0/3977106011 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc96806d8b0 con 0x7fc9680cef80 2024-03-31T15:06:10.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:10.483+0000 7fc966a00640 1 --2- 172.21.15.16:0/3977106011 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc9680cef80 0x7fc968152180 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7fc95400d800 tx=0x7fc95400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:06:10.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:10.483+0000 7fc95f400640 1 -- 172.21.15.16:0/3977106011 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc95400b6c0 con 0x7fc9680cef80 2024-03-31T15:06:10.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:10.483+0000 7fc95f400640 1 -- 172.21.15.16:0/3977106011 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc954019070 con 0x7fc9680cef80 2024-03-31T15:06:10.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:10.483+0000 7fc96e400640 1 -- 172.21.15.16:0/3977106011 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc96806db90 con 0x7fc9680cef80 2024-03-31T15:06:10.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:10.483+0000 7fc95f400640 1 -- 172.21.15.16:0/3977106011 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc95400ba80 con 0x7fc9680cef80 2024-03-31T15:06:10.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:10.483+0000 7fc96e400640 1 -- 172.21.15.16:0/3977106011 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc96806e150 con 0x7fc9680cef80 2024-03-31T15:06:10.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:10.483+0000 7fc96e400640 1 -- 172.21.15.16:0/3977106011 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc968069950 con 0x7fc9680cef80 2024-03-31T15:06:10.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:10.487+0000 7fc95f400640 1 -- 172.21.15.16:0/3977106011 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fc95400bc20 con 0x7fc9680cef80 2024-03-31T15:06:10.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:10.487+0000 7fc95f400640 1 --2- 172.21.15.16:0/3977106011 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fc9480683b0 0x7fc94806a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:10.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:10.487+0000 7fc95f400640 1 -- 172.21.15.16:0/3977106011 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5709+0+0 (secure 0 0 0) 0x7fc954086e20 con 0x7fc9680cef80 2024-03-31T15:06:10.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:10.487+0000 7fc967400640 1 --2- 172.21.15.16:0/3977106011 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fc9480683b0 0x7fc94806a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:10.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:10.487+0000 7fc967400640 1 --2- 172.21.15.16:0/3977106011 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fc9480683b0 0x7fc94806a870 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fc9580ad000 tx=0x7fc9580a6000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:06:10.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:10.491+0000 7fc95f400640 1 -- 172.21.15.16:0/3977106011 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fc954057340 con 0x7fc9680cef80 2024-03-31T15:06:10.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:10.655+0000 7fc96e400640 1 -- 172.21.15.16:0/3977106011 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fc9680b50e0 con 0x7fc9680cef80 2024-03-31T15:06:10.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:10.655+0000 7fc95f400640 1 -- 172.21.15.16:0/3977106011 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v45) ==== 93+0+10 (secure 0 0 0) 0x7fc954056c40 con 0x7fc9680cef80 2024-03-31T15:06:10.650 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2024-03-31T15:06:10.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:10.655+0000 7fc96e400640 1 -- 172.21.15.16:0/3977106011 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fc9480683b0 msgr2=0x7fc94806a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:10.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:10.655+0000 7fc96e400640 1 --2- 172.21.15.16:0/3977106011 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fc9480683b0 0x7fc94806a870 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fc9580ad000 tx=0x7fc9580a6000 comp rx=0 tx=0).stop 2024-03-31T15:06:10.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:10.655+0000 7fc96e400640 1 -- 172.21.15.16:0/3977106011 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc9680cef80 msgr2=0x7fc968152180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:10.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:10.655+0000 7fc96e400640 1 --2- 172.21.15.16:0/3977106011 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc9680cef80 0x7fc968152180 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7fc95400d800 tx=0x7fc95400dcc0 comp rx=0 tx=0).stop 2024-03-31T15:06:10.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:10.655+0000 7fc96e400640 1 -- 172.21.15.16:0/3977106011 shutdown_connections 2024-03-31T15:06:10.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:10.655+0000 7fc96e400640 1 --2- 172.21.15.16:0/3977106011 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fc9480683b0 0x7fc94806a870 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:10.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:10.655+0000 7fc96e400640 1 --2- 172.21.15.16:0/3977106011 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc9680d0a60 0x7fc96815f730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:10.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:10.655+0000 7fc96e400640 1 --2- 172.21.15.16:0/3977106011 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc9680cef80 0x7fc968152180 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:10.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:10.655+0000 7fc96e400640 1 --2- 172.21.15.16:0/3977106011 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fc96806f920 0x7fc96806d2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:10.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:10.655+0000 7fc96e400640 1 -- 172.21.15.16:0/3977106011 >> 172.21.15.16:0/3977106011 conn(0x7fc968067930 msgr2=0x7fc9681567a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:06:10.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:10.655+0000 7fc96e400640 1 -- 172.21.15.16:0/3977106011 shutdown_connections 2024-03-31T15:06:10.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:10.655+0000 7fc96e400640 1 -- 172.21.15.16:0/3977106011 wait complete. 2024-03-31T15:06:10.671 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-31T15:06:10.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:10.819+0000 7fb47c800640 1 Processor -- start 2024-03-31T15:06:10.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:10.819+0000 7fb47c800640 1 -- start start 2024-03-31T15:06:10.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:10.819+0000 7fb47c800640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fb474070030 0x7fb474143c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:10.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:10.819+0000 7fb47c800640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb474144980 con 0x7fb474141590 2024-03-31T15:06:10.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:10.819+0000 7fb47c800640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb474161fe0 con 0x7fb4741411b0 2024-03-31T15:06:10.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:10.819+0000 7fb47c800640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7fb47415f910 con 0x7fb474070030 2024-03-31T15:06:10.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:10.819+0000 7fb47a800640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fb474141590 0x7fb474144170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:37154/0 (socket says 172.21.15.16:37154) 2024-03-31T15:06:10.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:10.819+0000 7fb47a800640 1 -- 172.21.15.16:0/2469691518 learned_addr learned my addr 172.21.15.16:0/2469691518 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:06:10.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:10.819+0000 7fb479400640 1 --2- 172.21.15.16:0/2469691518 >> v2:172.21.15.89:3300/0 conn(0x7fb474070030 0x7fb474143c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:10.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:10.819+0000 7fb478a00640 1 -- 172.21.15.16:0/2469691518 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3871461898 0 0) 0x7fb474144980 con 0x7fb474141590 2024-03-31T15:06:10.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:10.819+0000 7fb478a00640 1 -- 172.21.15.16:0/2469691518 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb45c003530 con 0x7fb474141590 2024-03-31T15:06:10.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:10.819+0000 7fb478a00640 1 -- 172.21.15.16:0/2469691518 <== mon.1 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1515748966 0 0) 0x7fb474161fe0 con 0x7fb4741411b0 2024-03-31T15:06:10.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:10.819+0000 7fb478a00640 1 -- 172.21.15.16:0/2469691518 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb474144980 con 0x7fb4741411b0 2024-03-31T15:06:10.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:10.823+0000 7fb479400640 1 -- 172.21.15.16:0/2469691518 >> v1:172.21.15.89:6789/0 conn(0x7fb4741411b0 legacy=0x7fb47406f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:10.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:10.823+0000 7fb479400640 1 -- 172.21.15.16:0/2469691518 >> v1:172.21.15.16:6789/0 conn(0x7fb474141590 legacy=0x7fb474144170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:10.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:10.823+0000 7fb478a00640 1 -- 172.21.15.16:0/2469691518 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3773630961 0 0) 0x7fb45c003530 con 0x7fb474141590 2024-03-31T15:06:10.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:10.823+0000 7fb479400640 1 -- 172.21.15.16:0/2469691518 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb4741622c0 con 0x7fb474070030 2024-03-31T15:06:10.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:10.823+0000 7fb479400640 1 --2- 172.21.15.16:0/2469691518 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb474070030 0x7fb474143c30 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fb4640097c0 tx=0x7fb46401c3c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7036ac1669defd5a server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:06:10.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:10.823+0000 7fb478a00640 1 -- 172.21.15.16:0/2469691518 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4640b0070 con 0x7fb474070030 2024-03-31T15:06:10.820 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:10.823+0000 7fb478a00640 1 -- 172.21.15.16:0/2469691518 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb4640ab400 con 0x7fb474070030 2024-03-31T15:06:10.820 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:10.823+0000 7fb478a00640 1 -- 172.21.15.16:0/2469691518 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb470001a00 con 0x7fb474070030 2024-03-31T15:06:10.820 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:10.827+0000 7fb47c800640 1 -- 172.21.15.16:0/2469691518 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb474070030 msgr2=0x7fb474143c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:10.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:10.827+0000 7fb47c800640 1 --2- 172.21.15.16:0/2469691518 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb474070030 0x7fb474143c30 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fb4640097c0 tx=0x7fb46401c3c0 comp rx=0 tx=0).stop 2024-03-31T15:06:10.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:10.827+0000 7fb47c800640 1 -- 172.21.15.16:0/2469691518 shutdown_connections 2024-03-31T15:06:10.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:10.827+0000 7fb47c800640 1 --2- 172.21.15.16:0/2469691518 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb474070030 0x7fb474143c30 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:10.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:10.827+0000 7fb47c800640 1 -- 172.21.15.16:0/2469691518 >> 172.21.15.16:0/2469691518 conn(0x7fb474067930 msgr2=0x7fb4740b3ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:06:10.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:10.827+0000 7fb47c800640 1 -- 172.21.15.16:0/2469691518 shutdown_connections 2024-03-31T15:06:10.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:10.827+0000 7fb47c800640 1 -- 172.21.15.16:0/2469691518 wait complete. 2024-03-31T15:06:10.822 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:10.827+0000 7fb47c800640 1 Processor -- start 2024-03-31T15:06:10.822 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:10.827+0000 7fb47c800640 1 -- start start 2024-03-31T15:06:10.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:10.827+0000 7fb47c800640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb474070030 0x7fb4741483a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:10.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:10.827+0000 7fb47c800640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb4741411b0 0x7fb4741403f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:10.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:10.827+0000 7fb47c800640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb474141590 0x7fb47413a200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:10.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:10.827+0000 7fb47c800640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fb47416be80 con 0x7fb4741411b0 2024-03-31T15:06:10.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:10.827+0000 7fb47c800640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fb47416bd00 con 0x7fb474070030 2024-03-31T15:06:10.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:10.827+0000 7fb47c800640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fb47416c000 con 0x7fb474141590 2024-03-31T15:06:10.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:10.827+0000 7fb479400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb4741411b0 0x7fb4741403f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:10.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:10.827+0000 7fb479400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb4741411b0 0x7fb4741403f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35102/0 (socket says 172.21.15.16:35102) 2024-03-31T15:06:10.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:10.827+0000 7fb479400640 1 -- 172.21.15.16:0/3488968337 learned_addr learned my addr 172.21.15.16:0/3488968337 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:06:10.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:10.827+0000 7fb479e00640 1 --2- 172.21.15.16:0/3488968337 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb474070030 0x7fb4741483a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:10.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:10.827+0000 7fb47a800640 1 --2- 172.21.15.16:0/3488968337 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb474141590 0x7fb47413a200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:10.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:10.827+0000 7fb479400640 1 -- 172.21.15.16:0/3488968337 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb474141590 msgr2=0x7fb47413a200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:10.825 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:10.827+0000 7fb479400640 1 --2- 172.21.15.16:0/3488968337 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb474141590 0x7fb47413a200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:10.825 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:10.827+0000 7fb479400640 1 -- 172.21.15.16:0/3488968337 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb474070030 msgr2=0x7fb4741483a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:10.825 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:10.827+0000 7fb479400640 1 --2- 172.21.15.16:0/3488968337 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb474070030 0x7fb4741483a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:10.825 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:10.827+0000 7fb479400640 1 -- 172.21.15.16:0/3488968337 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb47413f580 con 0x7fb4741411b0 2024-03-31T15:06:10.825 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:10.827+0000 7fb47a800640 1 --2- 172.21.15.16:0/3488968337 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb474141590 0x7fb47413a200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:06:10.825 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:10.831+0000 7fb479400640 1 --2- 172.21.15.16:0/3488968337 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb4741411b0 0x7fb4741403f0 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7fb4640ae040 tx=0x7fb46401ac00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:06:10.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:10.831+0000 7fb46e000640 1 -- 172.21.15.16:0/3488968337 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4640bb020 con 0x7fb4741411b0 2024-03-31T15:06:10.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:10.831+0000 7fb46e000640 1 -- 172.21.15.16:0/3488968337 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb4640b1040 con 0x7fb4741411b0 2024-03-31T15:06:10.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:10.831+0000 7fb47c800640 1 -- 172.21.15.16:0/3488968337 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb47413f860 con 0x7fb4741411b0 2024-03-31T15:06:10.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:10.831+0000 7fb46e000640 1 -- 172.21.15.16:0/3488968337 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4640b0050 con 0x7fb4741411b0 2024-03-31T15:06:10.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:10.831+0000 7fb47c800640 1 -- 172.21.15.16:0/3488968337 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb474140930 con 0x7fb4741411b0 2024-03-31T15:06:10.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:10.831+0000 7fb47c800640 1 -- 172.21.15.16:0/3488968337 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb474069950 con 0x7fb4741411b0 2024-03-31T15:06:10.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:10.831+0000 7fb46e000640 1 -- 172.21.15.16:0/3488968337 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fb4640be070 con 0x7fb4741411b0 2024-03-31T15:06:10.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:10.831+0000 7fb46e000640 1 --2- 172.21.15.16:0/3488968337 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fb458068380 0x7fb45806a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:10.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:10.831+0000 7fb46e000640 1 -- 172.21.15.16:0/3488968337 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5709+0+0 (secure 0 0 0) 0x7fb46411fe40 con 0x7fb4741411b0 2024-03-31T15:06:10.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:10.831+0000 7fb479e00640 1 --2- 172.21.15.16:0/3488968337 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fb458068380 0x7fb45806a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:10.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:10.835+0000 7fb479e00640 1 --2- 172.21.15.16:0/3488968337 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fb458068380 0x7fb45806a840 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fb4680011f0 tx=0x7fb468009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:06:10.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:10.835+0000 7fb46e000640 1 -- 172.21.15.16:0/3488968337 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fb464015fb0 con 0x7fb4741411b0 2024-03-31T15:06:11.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:11.163+0000 7fb47c800640 1 -- 172.21.15.16:0/3488968337 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fb4740b4a20 con 0x7fb4741411b0 2024-03-31T15:06:11.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:11.163+0000 7fb46e000640 1 -- 172.21.15.16:0/3488968337 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v45) ==== 104+0+11 (secure 0 0 0) 0x7fb4640f6070 con 0x7fb4741411b0 2024-03-31T15:06:11.158 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2024-03-31T15:06:11.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:11.163+0000 7fb47c800640 1 -- 172.21.15.16:0/3488968337 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fb458068380 msgr2=0x7fb45806a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:11.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:11.163+0000 7fb47c800640 1 --2- 172.21.15.16:0/3488968337 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fb458068380 0x7fb45806a840 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fb4680011f0 tx=0x7fb468009040 comp rx=0 tx=0).stop 2024-03-31T15:06:11.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:11.163+0000 7fb47c800640 1 -- 172.21.15.16:0/3488968337 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb4741411b0 msgr2=0x7fb4741403f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:11.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:11.163+0000 7fb47c800640 1 --2- 172.21.15.16:0/3488968337 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb4741411b0 0x7fb4741403f0 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7fb4640ae040 tx=0x7fb46401ac00 comp rx=0 tx=0).stop 2024-03-31T15:06:11.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:11.163+0000 7fb47c800640 1 -- 172.21.15.16:0/3488968337 shutdown_connections 2024-03-31T15:06:11.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:11.163+0000 7fb47c800640 1 --2- 172.21.15.16:0/3488968337 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fb458068380 0x7fb45806a840 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:11.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:11.163+0000 7fb47c800640 1 --2- 172.21.15.16:0/3488968337 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb474141590 0x7fb47413a200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:11.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:11.163+0000 7fb47c800640 1 --2- 172.21.15.16:0/3488968337 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb4741411b0 0x7fb4741403f0 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:11.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:11.163+0000 7fb47c800640 1 --2- 172.21.15.16:0/3488968337 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb474070030 0x7fb4741483a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:11.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:11.167+0000 7fb47c800640 1 -- 172.21.15.16:0/3488968337 >> 172.21.15.16:0/3488968337 conn(0x7fb474067930 msgr2=0x7fb47406f920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:06:11.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:11.167+0000 7fb47c800640 1 -- 172.21.15.16:0/3488968337 shutdown_connections 2024-03-31T15:06:11.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:11.167+0000 7fb47c800640 1 -- 172.21.15.16:0/3488968337 wait complete. 2024-03-31T15:06:11.181 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-31T15:06:11.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:11.327+0000 7f5b84a00640 1 Processor -- start 2024-03-31T15:06:11.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:11.327+0000 7f5b84a00640 1 -- start start 2024-03-31T15:06:11.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:11.331+0000 7f5b84a00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f5b801411b0 0x7f5b8006f920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:11.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:11.331+0000 7f5b84a00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f5b80144340 0x7f5b8006fe60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:11.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:11.331+0000 7f5b84a00640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5b80144980 con 0x7f5b80141590 2024-03-31T15:06:11.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:11.331+0000 7f5b84a00640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f5b8015f910 con 0x7f5b801411b0 2024-03-31T15:06:11.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:11.331+0000 7f5b84a00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f5b80162910 con 0x7f5b80144340 2024-03-31T15:06:11.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:11.331+0000 7f5b7e800640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f5b80144340 0x7f5b8006fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:11.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:11.331+0000 7f5b7de00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f5b801411b0 0x7f5b8006f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:11.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:11.331+0000 7f5b7e800640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f5b80144340 0x7f5b8006fe60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3301/0 says I am v2:172.21.15.16:52298/0 (socket says 172.21.15.16:52298) 2024-03-31T15:06:11.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:11.331+0000 7f5b7e800640 1 -- 172.21.15.16:0/2870208625 learned_addr learned my addr 172.21.15.16:0/2870208625 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:06:11.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:11.331+0000 7f5b7d400640 1 --1- 172.21.15.16:0/2870208625 >> v1:172.21.15.89:6789/0 conn(0x7f5b80141590 0x7f5b80143c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.16:51890/0 (socket says 172.21.15.16:51890) 2024-03-31T15:06:11.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:11.331+0000 7f5b7e800640 1 -- 172.21.15.16:0/2870208625 >> v2:172.21.15.89:3300/0 conn(0x7f5b801411b0 msgr2=0x7f5b8006f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:06:11.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:11.331+0000 7f5b7e800640 1 --2- 172.21.15.16:0/2870208625 >> v2:172.21.15.89:3300/0 conn(0x7f5b801411b0 0x7f5b8006f920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:11.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:11.331+0000 7f5b7e800640 1 -- 172.21.15.16:0/2870208625 >> v1:172.21.15.89:6789/0 conn(0x7f5b80141590 legacy=0x7f5b80143c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:11.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:11.331+0000 7f5b7e800640 1 -- 172.21.15.16:0/2870208625 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b80161fe0 con 0x7f5b80144340 2024-03-31T15:06:11.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:11.331+0000 7f5b7de00640 1 --2- 172.21.15.16:0/2870208625 >> v2:172.21.15.89:3300/0 conn(0x7f5b801411b0 0x7f5b8006f920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:06:11.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:11.331+0000 7f5b7ca00640 1 -- 172.21.15.16:0/2870208625 <== mon.1 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3322022690 0 0) 0x7f5b80144980 con 0x7f5b80141590 2024-03-31T15:06:11.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:11.335+0000 7f5b7e800640 1 --2- 172.21.15.16:0/2870208625 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f5b80144340 0x7f5b8006fe60 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f5b78009e50 tx=0x7f5b7801ca30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3ccc062a7a224972 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:06:11.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:11.335+0000 7f5b7ca00640 1 -- 172.21.15.16:0/2870208625 <== mon.2 v2:172.21.15.89:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b780a57f0 con 0x7f5b80144340 2024-03-31T15:06:11.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:11.335+0000 7f5b7ca00640 1 -- 172.21.15.16:0/2870208625 <== mon.2 v2:172.21.15.89:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5b780a5990 con 0x7f5b80144340 2024-03-31T15:06:11.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:11.335+0000 7f5b7ca00640 1 -- 172.21.15.16:0/2870208625 <== mon.2 v2:172.21.15.89:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b780a5d60 con 0x7f5b80144340 2024-03-31T15:06:11.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:11.335+0000 7f5b84a00640 1 -- 172.21.15.16:0/2870208625 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f5b80144340 msgr2=0x7f5b8006fe60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:11.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:11.335+0000 7f5b84a00640 1 --2- 172.21.15.16:0/2870208625 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f5b80144340 0x7f5b8006fe60 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f5b78009e50 tx=0x7f5b7801ca30 comp rx=0 tx=0).stop 2024-03-31T15:06:11.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:11.335+0000 7f5b84a00640 1 -- 172.21.15.16:0/2870208625 shutdown_connections 2024-03-31T15:06:11.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:11.335+0000 7f5b84a00640 1 --2- 172.21.15.16:0/2870208625 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f5b80144340 0x7f5b8006fe60 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:11.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:11.335+0000 7f5b84a00640 1 --2- 172.21.15.16:0/2870208625 >> v2:172.21.15.89:3300/0 conn(0x7f5b801411b0 0x7f5b8006f920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:11.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:11.335+0000 7f5b84a00640 1 -- 172.21.15.16:0/2870208625 >> 172.21.15.16:0/2870208625 conn(0x7f5b80067930 msgr2=0x7f5b800b3ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:06:11.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:11.335+0000 7f5b84a00640 1 -- 172.21.15.16:0/2870208625 shutdown_connections 2024-03-31T15:06:11.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:11.335+0000 7f5b84a00640 1 -- 172.21.15.16:0/2870208625 wait complete. 2024-03-31T15:06:11.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:11.335+0000 7f5b84a00640 1 Processor -- start 2024-03-31T15:06:11.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:11.335+0000 7f5b84a00640 1 -- start start 2024-03-31T15:06:11.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:11.339+0000 7f5b84a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5b801411b0 0x7f5b80139b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:11.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:11.339+0000 7f5b84a00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f5b80141590 0x7f5b80156ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:11.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:11.339+0000 7f5b84a00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5b80144340 0x7f5b80156730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:11.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:11.339+0000 7f5b84a00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f5b8015f290 con 0x7f5b801411b0 2024-03-31T15:06:11.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:11.339+0000 7f5b84a00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f5b8015f110 con 0x7f5b80144340 2024-03-31T15:06:11.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:11.339+0000 7f5b84a00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f5b8015f410 con 0x7f5b80141590 2024-03-31T15:06:11.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:11.339+0000 7f5b7de00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5b801411b0 0x7f5b80139b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:11.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:11.339+0000 7f5b7de00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5b801411b0 0x7f5b80139b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35110/0 (socket says 172.21.15.16:35110) 2024-03-31T15:06:11.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:11.339+0000 7f5b7de00640 1 -- 172.21.15.16:0/3325227585 learned_addr learned my addr 172.21.15.16:0/3325227585 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:06:11.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:11.339+0000 7f5b7e800640 1 --2- 172.21.15.16:0/3325227585 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5b80144340 0x7f5b80156730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:11.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:11.339+0000 7f5b7d400640 1 --2- 172.21.15.16:0/3325227585 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f5b80141590 0x7f5b80156ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:11.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:11.339+0000 7f5b7de00640 1 -- 172.21.15.16:0/3325227585 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f5b80141590 msgr2=0x7f5b80156ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:11.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:11.339+0000 7f5b7de00640 1 --2- 172.21.15.16:0/3325227585 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f5b80141590 0x7f5b80156ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:11.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:11.339+0000 7f5b7de00640 1 -- 172.21.15.16:0/3325227585 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5b80144340 msgr2=0x7f5b80156730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:11.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:11.339+0000 7f5b7de00640 1 --2- 172.21.15.16:0/3325227585 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5b80144340 0x7f5b80156730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:11.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:11.339+0000 7f5b7de00640 1 -- 172.21.15.16:0/3325227585 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b80157530 con 0x7f5b801411b0 2024-03-31T15:06:11.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:11.339+0000 7f5b7de00640 1 --2- 172.21.15.16:0/3325227585 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5b801411b0 0x7f5b80139b50 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f5b7000ec10 tx=0x7f5b7000c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:06:11.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:11.339+0000 7f5b76000640 1 -- 172.21.15.16:0/3325227585 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b70017070 con 0x7f5b801411b0 2024-03-31T15:06:11.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:11.339+0000 7f5b76000640 1 -- 172.21.15.16:0/3325227585 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5b7000cc30 con 0x7f5b801411b0 2024-03-31T15:06:11.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:11.339+0000 7f5b84a00640 1 -- 172.21.15.16:0/3325227585 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5b801577d0 con 0x7f5b801411b0 2024-03-31T15:06:11.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:11.339+0000 7f5b76000640 1 -- 172.21.15.16:0/3325227585 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b70013470 con 0x7f5b801411b0 2024-03-31T15:06:11.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:11.339+0000 7f5b84a00640 1 -- 172.21.15.16:0/3325227585 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5b80157d90 con 0x7f5b801411b0 2024-03-31T15:06:11.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:11.339+0000 7f5b84a00640 1 -- 172.21.15.16:0/3325227585 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5b80069950 con 0x7f5b801411b0 2024-03-31T15:06:11.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:11.343+0000 7f5b76000640 1 -- 172.21.15.16:0/3325227585 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f5b7001a070 con 0x7f5b801411b0 2024-03-31T15:06:11.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:11.343+0000 7f5b76000640 1 --2- 172.21.15.16:0/3325227585 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f5b600683b0 0x7f5b6006a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:11.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:11.343+0000 7f5b76000640 1 -- 172.21.15.16:0/3325227585 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5709+0+0 (secure 0 0 0) 0x7f5b700870a0 con 0x7f5b801411b0 2024-03-31T15:06:11.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:11.343+0000 7f5b7d400640 1 --2- 172.21.15.16:0/3325227585 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f5b600683b0 0x7f5b6006a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:11.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:11.343+0000 7f5b7d400640 1 --2- 172.21.15.16:0/3325227585 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f5b600683b0 0x7f5b6006a870 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f5b80140900 tx=0x7f5b6c007b60 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:06:11.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:11.347+0000 7f5b76000640 1 -- 172.21.15.16:0/3325227585 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f5b700576d0 con 0x7f5b801411b0 2024-03-31T15:06:11.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:11.511+0000 7f5b84a00640 1 -- 172.21.15.16:0/3325227585 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f5b8015e490 con 0x7f5b801411b0 2024-03-31T15:06:11.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:11.511+0000 7f5b76000640 1 -- 172.21.15.16:0/3325227585 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v45) ==== 100+0+11 (secure 0 0 0) 0x7f5b70056fd0 con 0x7f5b801411b0 2024-03-31T15:06:11.506 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 64 2024-03-31T15:06:11.508 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:11.511+0000 7f5b84a00640 1 -- 172.21.15.16:0/3325227585 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f5b600683b0 msgr2=0x7f5b6006a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:11.508 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:11.511+0000 7f5b84a00640 1 --2- 172.21.15.16:0/3325227585 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f5b600683b0 0x7f5b6006a870 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f5b80140900 tx=0x7f5b6c007b60 comp rx=0 tx=0).stop 2024-03-31T15:06:11.508 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:11.511+0000 7f5b84a00640 1 -- 172.21.15.16:0/3325227585 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5b801411b0 msgr2=0x7f5b80139b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:11.508 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:11.511+0000 7f5b84a00640 1 --2- 172.21.15.16:0/3325227585 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5b801411b0 0x7f5b80139b50 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f5b7000ec10 tx=0x7f5b7000c520 comp rx=0 tx=0).stop 2024-03-31T15:06:11.508 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:11.511+0000 7f5b84a00640 1 -- 172.21.15.16:0/3325227585 shutdown_connections 2024-03-31T15:06:11.508 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:11.511+0000 7f5b84a00640 1 --2- 172.21.15.16:0/3325227585 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f5b600683b0 0x7f5b6006a870 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:11.508 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:11.511+0000 7f5b84a00640 1 --2- 172.21.15.16:0/3325227585 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5b80144340 0x7f5b80156730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:11.508 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:11.511+0000 7f5b84a00640 1 --2- 172.21.15.16:0/3325227585 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f5b80141590 0x7f5b80156ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:11.508 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:11.511+0000 7f5b84a00640 1 --2- 172.21.15.16:0/3325227585 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5b801411b0 0x7f5b80139b50 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:11.508 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:11.511+0000 7f5b84a00640 1 -- 172.21.15.16:0/3325227585 >> 172.21.15.16:0/3325227585 conn(0x7f5b80067930 msgr2=0x7f5b801436d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:06:11.508 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:11.511+0000 7f5b84a00640 1 -- 172.21.15.16:0/3325227585 shutdown_connections 2024-03-31T15:06:11.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:11.511+0000 7f5b84a00640 1 -- 172.21.15.16:0/3325227585 wait complete. 2024-03-31T15:06:11.527 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-31T15:06:11.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:11.671+0000 7f42d1a00640 1 Processor -- start 2024-03-31T15:06:11.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:11.671+0000 7f42d1a00640 1 -- start start 2024-03-31T15:06:11.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:11.675+0000 7f42d1a00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f42cc145e40 0x7f42cc0cc740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:11.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:11.675+0000 7f42d1a00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f42cc0ccc80 0x7f42cc06f920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:11.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:11.675+0000 7f42d1a00640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f42cc0d0920 con 0x7f42cc06fe60 2024-03-31T15:06:11.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:11.675+0000 7f42d1a00640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f42cc0d06d0 con 0x7f42cc0ccc80 2024-03-31T15:06:11.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:11.675+0000 7f42d1a00640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f42cc0d0c00 con 0x7f42cc145e40 2024-03-31T15:06:11.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:11.675+0000 7f42c3e00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f42cc0ccc80 0x7f42cc06f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:11.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:11.675+0000 7f42cb400640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f42cc06fe60 0x7f42cc15fdf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:37170/0 (socket says 172.21.15.16:37170) 2024-03-31T15:06:11.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:11.675+0000 7f42cb400640 1 -- 172.21.15.16:0/3302890647 learned_addr learned my addr 172.21.15.16:0/3302890647 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:06:11.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:11.675+0000 7f42caa00640 1 --2- 172.21.15.16:0/3302890647 >> v2:172.21.15.89:3300/0 conn(0x7f42cc145e40 0x7f42cc0cc740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:11.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:11.675+0000 7f42c3e00640 1 -- 172.21.15.16:0/3302890647 >> v2:172.21.15.89:3300/0 conn(0x7f42cc145e40 msgr2=0x7f42cc0cc740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:06:11.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:11.675+0000 7f42c3e00640 1 --2- 172.21.15.16:0/3302890647 >> v2:172.21.15.89:3300/0 conn(0x7f42cc145e40 0x7f42cc0cc740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:11.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:11.675+0000 7f42c3e00640 1 -- 172.21.15.16:0/3302890647 >> v1:172.21.15.16:6789/0 conn(0x7f42cc06fe60 legacy=0x7f42cc15fdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:11.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:11.675+0000 7f42c3e00640 1 -- 172.21.15.16:0/3302890647 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42cc0d1810 con 0x7f42cc0ccc80 2024-03-31T15:06:11.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:11.675+0000 7f42ca000640 1 -- 172.21.15.16:0/3302890647 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4171112263 0 0) 0x7f42cc0d0920 con 0x7f42cc06fe60 2024-03-31T15:06:11.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:11.675+0000 7f42caa00640 1 --2- 172.21.15.16:0/3302890647 >> v2:172.21.15.89:3300/0 conn(0x7f42cc145e40 0x7f42cc0cc740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:06:11.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:11.679+0000 7f42c3e00640 1 --2- 172.21.15.16:0/3302890647 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f42cc0ccc80 0x7f42cc06f920 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f42cc072e60 tx=0x7f42b401c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52b104d0fca3b2d5 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:06:11.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:11.679+0000 7f42ca000640 1 -- 172.21.15.16:0/3302890647 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42b40b0070 con 0x7f42cc0ccc80 2024-03-31T15:06:11.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:11.679+0000 7f42ca000640 1 -- 172.21.15.16:0/3302890647 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f42b40ab600 con 0x7f42cc0ccc80 2024-03-31T15:06:11.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:11.679+0000 7f42ca000640 1 -- 172.21.15.16:0/3302890647 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42b40ab9d0 con 0x7f42cc0ccc80 2024-03-31T15:06:11.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:11.679+0000 7f42d1a00640 1 -- 172.21.15.16:0/3302890647 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f42cc0ccc80 msgr2=0x7f42cc06f920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:11.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:11.679+0000 7f42d1a00640 1 --2- 172.21.15.16:0/3302890647 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f42cc0ccc80 0x7f42cc06f920 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f42cc072e60 tx=0x7f42b401c480 comp rx=0 tx=0).stop 2024-03-31T15:06:11.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:11.679+0000 7f42d1a00640 1 -- 172.21.15.16:0/3302890647 shutdown_connections 2024-03-31T15:06:11.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:11.679+0000 7f42d1a00640 1 --2- 172.21.15.16:0/3302890647 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f42cc0ccc80 0x7f42cc06f920 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:11.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:11.679+0000 7f42d1a00640 1 --2- 172.21.15.16:0/3302890647 >> v2:172.21.15.89:3300/0 conn(0x7f42cc145e40 0x7f42cc0cc740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:11.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:11.679+0000 7f42d1a00640 1 -- 172.21.15.16:0/3302890647 >> 172.21.15.16:0/3302890647 conn(0x7f42cc067330 msgr2=0x7f42cc067740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:06:11.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:11.679+0000 7f42d1a00640 1 -- 172.21.15.16:0/3302890647 shutdown_connections 2024-03-31T15:06:11.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:11.679+0000 7f42d1a00640 1 -- 172.21.15.16:0/3302890647 wait complete. 2024-03-31T15:06:11.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:11.683+0000 7f42d1a00640 1 Processor -- start 2024-03-31T15:06:11.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:11.683+0000 7f42d1a00640 1 -- start start 2024-03-31T15:06:11.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:11.683+0000 7f42d1a00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f42cc06fe60 0x7f42cc160ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:11.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:11.683+0000 7f42d1a00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f42cc0ccc80 0x7f42cc163c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:11.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:11.683+0000 7f42d1a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f42cc145e40 0x7f42cc142400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:11.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:11.683+0000 7f42d1a00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f42cc0b5240 con 0x7f42cc145e40 2024-03-31T15:06:11.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:11.683+0000 7f42d1a00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f42cc0b50c0 con 0x7f42cc0ccc80 2024-03-31T15:06:11.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:11.683+0000 7f42d1a00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f42cc169190 con 0x7f42cc06fe60 2024-03-31T15:06:11.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:11.683+0000 7f42cb400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f42cc145e40 0x7f42cc142400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:11.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:11.683+0000 7f42cb400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f42cc145e40 0x7f42cc142400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35130/0 (socket says 172.21.15.16:35130) 2024-03-31T15:06:11.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:11.683+0000 7f42cb400640 1 -- 172.21.15.16:0/1886307745 learned_addr learned my addr 172.21.15.16:0/1886307745 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:06:11.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:11.683+0000 7f42c3e00640 1 --2- 172.21.15.16:0/1886307745 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f42cc0ccc80 0x7f42cc163c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:11.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:11.683+0000 7f42caa00640 1 --2- 172.21.15.16:0/1886307745 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f42cc06fe60 0x7f42cc160ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:11.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:11.683+0000 7f42cb400640 1 -- 172.21.15.16:0/1886307745 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f42cc06fe60 msgr2=0x7f42cc160ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:11.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:11.683+0000 7f42cb400640 1 --2- 172.21.15.16:0/1886307745 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f42cc06fe60 0x7f42cc160ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:11.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:11.683+0000 7f42cb400640 1 -- 172.21.15.16:0/1886307745 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f42cc0ccc80 msgr2=0x7f42cc163c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:11.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:11.683+0000 7f42cb400640 1 --2- 172.21.15.16:0/1886307745 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f42cc0ccc80 0x7f42cc163c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:11.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:11.683+0000 7f42cb400640 1 -- 172.21.15.16:0/1886307745 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42cc142ef0 con 0x7f42cc145e40 2024-03-31T15:06:11.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:11.683+0000 7f42c3e00640 1 --2- 172.21.15.16:0/1886307745 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f42cc0ccc80 0x7f42cc163c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:06:11.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:11.683+0000 7f42cb400640 1 --2- 172.21.15.16:0/1886307745 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f42cc145e40 0x7f42cc142400 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f42c400c730 tx=0x7f42c400cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:06:11.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:11.683+0000 7f42c2a00640 1 -- 172.21.15.16:0/1886307745 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42c4018070 con 0x7f42cc145e40 2024-03-31T15:06:11.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:11.683+0000 7f42c2a00640 1 -- 172.21.15.16:0/1886307745 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f42c400ce40 con 0x7f42cc145e40 2024-03-31T15:06:11.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:11.683+0000 7f42d1a00640 1 -- 172.21.15.16:0/1886307745 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f42cc1614c0 con 0x7f42cc145e40 2024-03-31T15:06:11.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:11.683+0000 7f42c2a00640 1 -- 172.21.15.16:0/1886307745 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42c400a860 con 0x7f42cc145e40 2024-03-31T15:06:11.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:11.683+0000 7f42d1a00640 1 -- 172.21.15.16:0/1886307745 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f42cc161d60 con 0x7f42cc145e40 2024-03-31T15:06:11.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:11.687+0000 7f42d1a00640 1 -- 172.21.15.16:0/1886307745 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f42cc069950 con 0x7f42cc145e40 2024-03-31T15:06:11.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:11.687+0000 7f42c2a00640 1 -- 172.21.15.16:0/1886307745 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f42c401b070 con 0x7f42cc145e40 2024-03-31T15:06:11.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:11.687+0000 7f42c2a00640 1 --2- 172.21.15.16:0/1886307745 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f42a8068380 0x7f42a806a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:11.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:11.687+0000 7f42c2a00640 1 -- 172.21.15.16:0/1886307745 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5709+0+0 (secure 0 0 0) 0x7f42c4085ef0 con 0x7f42cc145e40 2024-03-31T15:06:11.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:11.687+0000 7f42caa00640 1 --2- 172.21.15.16:0/1886307745 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f42a8068380 0x7f42a806a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:11.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:11.687+0000 7f42caa00640 1 --2- 172.21.15.16:0/1886307745 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f42a8068380 0x7f42a806a840 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f42bc0098b0 tx=0x7f42bc008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:06:11.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:11.691+0000 7f42c2a00640 1 -- 172.21.15.16:0/1886307745 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f42c40564b0 con 0x7f42cc145e40 2024-03-31T15:06:11.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:11.887+0000 7f42d1a00640 1 -- 172.21.15.16:0/1886307745 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f42cc15e520 con 0x7f42cc145e40 2024-03-31T15:06:11.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:11.887+0000 7f42c2a00640 1 -- 172.21.15.16:0/1886307745 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+5175 (secure 0 0 0) 0x7f42c4055db0 con 0x7f42cc145e40 2024-03-31T15:06:11.884 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:06:11.884 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":25,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/285985105","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6835","nonce":285985105},{"type":"v1","addr":"172.21.15.89:6837","nonce":285985105}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20},{"gid":5689,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6836/1615931247","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":1615931247},{"type":"v1","addr":"172.21.15.89:6836","nonce":1615931247}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20},{"gid":5694,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/512187852","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6833","nonce":512187852},{"type":"v1","addr":"172.21.15.16:6835","nonce":512187852}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-31T15:05:26.461908+0000","modified":"2024-03-31T15:05:29.491163+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5697},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5697":{"gid":5697,"name":"a","rank":0,"incarnation":21,"state":"up:active","state_seq":3,"addr":"172.21.15.16:6834/2768973409","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":2768973409},{"type":"v1","addr":"172.21.15.16:6834","nonce":2768973409}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5697,"qdb_cluster":[5697]},"id":3}]} 2024-03-31T15:06:11.884 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 25 2024-03-31T15:06:11.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:11.891+0000 7f42d1a00640 1 -- 172.21.15.16:0/1886307745 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f42a8068380 msgr2=0x7f42a806a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:11.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:11.891+0000 7f42d1a00640 1 --2- 172.21.15.16:0/1886307745 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f42a8068380 0x7f42a806a840 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f42bc0098b0 tx=0x7f42bc008040 comp rx=0 tx=0).stop 2024-03-31T15:06:11.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:11.891+0000 7f42d1a00640 1 -- 172.21.15.16:0/1886307745 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f42cc145e40 msgr2=0x7f42cc142400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:11.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:11.891+0000 7f42d1a00640 1 --2- 172.21.15.16:0/1886307745 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f42cc145e40 0x7f42cc142400 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f42c400c730 tx=0x7f42c400cbf0 comp rx=0 tx=0).stop 2024-03-31T15:06:11.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:11.891+0000 7f42d1a00640 1 -- 172.21.15.16:0/1886307745 shutdown_connections 2024-03-31T15:06:11.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:11.891+0000 7f42d1a00640 1 --2- 172.21.15.16:0/1886307745 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f42a8068380 0x7f42a806a840 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:11.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:11.891+0000 7f42d1a00640 1 --2- 172.21.15.16:0/1886307745 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f42cc145e40 0x7f42cc142400 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:11.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:11.891+0000 7f42d1a00640 1 --2- 172.21.15.16:0/1886307745 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f42cc0ccc80 0x7f42cc163c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:11.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:11.891+0000 7f42d1a00640 1 --2- 172.21.15.16:0/1886307745 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f42cc06fe60 0x7f42cc160ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:11.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:11.891+0000 7f42d1a00640 1 -- 172.21.15.16:0/1886307745 >> 172.21.15.16:0/1886307745 conn(0x7f42cc067330 msgr2=0x7f42cc067710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:06:11.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:11.891+0000 7f42d1a00640 1 -- 172.21.15.16:0/1886307745 shutdown_connections 2024-03-31T15:06:11.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:11.891+0000 7f42d1a00640 1 -- 172.21.15.16:0/1886307745 wait complete. 2024-03-31T15:06:11.904 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-31T15:06:12.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:12.047+0000 7f48de600640 1 Processor -- start 2024-03-31T15:06:12.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:12.047+0000 7f48de600640 1 -- start start 2024-03-31T15:06:12.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:12.051+0000 7f48de600640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f48d8147e70 0x7f48d806b620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:12.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:12.051+0000 7f48de600640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f48d81621b0 con 0x7f48d80736e0 2024-03-31T15:06:12.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:12.051+0000 7f48de600640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f48d8162490 con 0x7f48d80d0c50 2024-03-31T15:06:12.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:12.051+0000 7f48de600640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f48d8162770 con 0x7f48d8147e70 2024-03-31T15:06:12.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:12.051+0000 7f48d7e00640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f48d80736e0 0x7f48d815eb20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:37180/0 (socket says 172.21.15.16:37180) 2024-03-31T15:06:12.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:12.051+0000 7f48d7e00640 1 -- 172.21.15.16:0/3820416323 learned_addr learned my addr 172.21.15.16:0/3820416323 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:06:12.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:12.051+0000 7f48d7400640 1 --2- 172.21.15.16:0/3820416323 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f48d8147e70 0x7f48d806b620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:12.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:12.051+0000 7f48d6a00640 1 -- 172.21.15.16:0/3820416323 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3673161282 0 0) 0x7f48d81621b0 con 0x7f48d80736e0 2024-03-31T15:06:12.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:12.051+0000 7f48d6a00640 1 -- 172.21.15.16:0/3820416323 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f48bc003530 con 0x7f48d80736e0 2024-03-31T15:06:12.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:12.051+0000 7f48d6a00640 1 -- 172.21.15.16:0/3820416323 <== mon.1 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3648088471 0 0) 0x7f48d8162490 con 0x7f48d80d0c50 2024-03-31T15:06:12.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:12.051+0000 7f48d6a00640 1 -- 172.21.15.16:0/3820416323 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f48d81621b0 con 0x7f48d80d0c50 2024-03-31T15:06:12.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:12.051+0000 7f48d7400640 1 -- 172.21.15.16:0/3820416323 >> v1:172.21.15.89:6789/0 conn(0x7f48d80d0c50 legacy=0x7f48d8072fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:12.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:12.051+0000 7f48d7400640 1 -- 172.21.15.16:0/3820416323 >> v1:172.21.15.16:6789/0 conn(0x7f48d80736e0 legacy=0x7f48d815eb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:12.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:12.051+0000 7f48d7400640 1 -- 172.21.15.16:0/3820416323 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48d8162a50 con 0x7f48d8147e70 2024-03-31T15:06:12.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:12.055+0000 7f48d7400640 1 --2- 172.21.15.16:0/3820416323 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f48d8147e70 0x7f48d806b620 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f48c80097f0 tx=0x7f48c801c4e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=58bf67f699889262 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:06:12.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:12.055+0000 7f48d6a00640 1 -- 172.21.15.16:0/3820416323 <== mon.2 v2:172.21.15.89:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48c80b2070 con 0x7f48d8147e70 2024-03-31T15:06:12.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:12.055+0000 7f48d6a00640 1 -- 172.21.15.16:0/3820416323 <== mon.2 v2:172.21.15.89:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f48c80a9070 con 0x7f48d8147e70 2024-03-31T15:06:12.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:12.055+0000 7f48d6a00640 1 -- 172.21.15.16:0/3820416323 <== mon.2 v2:172.21.15.89:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48c80ad880 con 0x7f48d8147e70 2024-03-31T15:06:12.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:12.059+0000 7f48de600640 1 -- 172.21.15.16:0/3820416323 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f48d8147e70 msgr2=0x7f48d806b620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:12.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:12.059+0000 7f48de600640 1 --2- 172.21.15.16:0/3820416323 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f48d8147e70 0x7f48d806b620 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f48c80097f0 tx=0x7f48c801c4e0 comp rx=0 tx=0).stop 2024-03-31T15:06:12.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:12.059+0000 7f48de600640 1 -- 172.21.15.16:0/3820416323 shutdown_connections 2024-03-31T15:06:12.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:12.059+0000 7f48de600640 1 --2- 172.21.15.16:0/3820416323 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f48d8147e70 0x7f48d806b620 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:12.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:12.059+0000 7f48de600640 1 -- 172.21.15.16:0/3820416323 >> 172.21.15.16:0/3820416323 conn(0x7f48d80d1060 msgr2=0x7f48d80d1470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:06:12.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:12.059+0000 7f48de600640 1 -- 172.21.15.16:0/3820416323 shutdown_connections 2024-03-31T15:06:12.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:12.059+0000 7f48de600640 1 -- 172.21.15.16:0/3820416323 wait complete. 2024-03-31T15:06:12.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:12.059+0000 7f48de600640 1 Processor -- start 2024-03-31T15:06:12.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:12.059+0000 7f48de600640 1 -- start start 2024-03-31T15:06:12.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:12.063+0000 7f48de600640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f48d80736e0 0x7f48d8145be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:12.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:12.063+0000 7f48de600640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f48d80d0c50 0x7f48d815dde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:12.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:12.063+0000 7f48de600640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f48d8147e70 0x7f48d80ad8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:12.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:12.063+0000 7f48de600640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f48d816c4c0 con 0x7f48d80d0c50 2024-03-31T15:06:12.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:12.063+0000 7f48de600640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f48d816c640 con 0x7f48d8147e70 2024-03-31T15:06:12.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:12.063+0000 7f48de600640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f48d816c7c0 con 0x7f48d80736e0 2024-03-31T15:06:12.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:12.063+0000 7f48cfe00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f48d80d0c50 0x7f48d815dde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:12.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:12.063+0000 7f48cfe00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f48d80d0c50 0x7f48d815dde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35138/0 (socket says 172.21.15.16:35138) 2024-03-31T15:06:12.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:12.063+0000 7f48cfe00640 1 -- 172.21.15.16:0/747153780 learned_addr learned my addr 172.21.15.16:0/747153780 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:06:12.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:12.063+0000 7f48d7e00640 1 --2- 172.21.15.16:0/747153780 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f48d8147e70 0x7f48d80ad8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:12.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:12.063+0000 7f48d7400640 1 --2- 172.21.15.16:0/747153780 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f48d80736e0 0x7f48d8145be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:12.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:12.063+0000 7f48cfe00640 1 -- 172.21.15.16:0/747153780 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f48d80736e0 msgr2=0x7f48d8145be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:12.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:12.063+0000 7f48cfe00640 1 --2- 172.21.15.16:0/747153780 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f48d80736e0 0x7f48d8145be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:12.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:12.063+0000 7f48cfe00640 1 -- 172.21.15.16:0/747153780 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f48d8147e70 msgr2=0x7f48d80ad8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:12.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:12.063+0000 7f48cfe00640 1 --2- 172.21.15.16:0/747153780 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f48d8147e70 0x7f48d80ad8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:12.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:12.063+0000 7f48cfe00640 1 -- 172.21.15.16:0/747153780 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48d8146120 con 0x7f48d80d0c50 2024-03-31T15:06:12.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:12.063+0000 7f48cfe00640 1 --2- 172.21.15.16:0/747153780 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f48d80d0c50 0x7f48d815dde0 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f48c000c410 tx=0x7f48c000c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:06:12.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:12.063+0000 7f48cf400640 1 -- 172.21.15.16:0/747153780 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48c0017070 con 0x7f48d80d0c50 2024-03-31T15:06:12.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:12.063+0000 7f48cf400640 1 -- 172.21.15.16:0/747153780 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f48c00052f0 con 0x7f48d80d0c50 2024-03-31T15:06:12.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:12.063+0000 7f48cf400640 1 -- 172.21.15.16:0/747153780 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48c0003840 con 0x7f48d80d0c50 2024-03-31T15:06:12.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:12.063+0000 7f48de600640 1 -- 172.21.15.16:0/747153780 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f48d81463a0 con 0x7f48d80d0c50 2024-03-31T15:06:12.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:12.063+0000 7f48de600640 1 -- 172.21.15.16:0/747153780 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f48d8146960 con 0x7f48d80d0c50 2024-03-31T15:06:12.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:12.063+0000 7f48de600640 1 -- 172.21.15.16:0/747153780 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f48d8069950 con 0x7f48d80d0c50 2024-03-31T15:06:12.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:12.067+0000 7f48cf400640 1 -- 172.21.15.16:0/747153780 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f48c001a070 con 0x7f48d80d0c50 2024-03-31T15:06:12.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:12.067+0000 7f48cf400640 1 --2- 172.21.15.16:0/747153780 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f48b4068380 0x7f48b406a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:12.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:12.067+0000 7f48cf400640 1 -- 172.21.15.16:0/747153780 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5709+0+0 (secure 0 0 0) 0x7f48c0085eb0 con 0x7f48d80d0c50 2024-03-31T15:06:12.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:12.067+0000 7f48d7400640 1 --2- 172.21.15.16:0/747153780 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f48b4068380 0x7f48b406a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:12.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:12.067+0000 7f48d7400640 1 --2- 172.21.15.16:0/747153780 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f48b4068380 0x7f48b406a840 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f48c80af000 tx=0x7f48c80a7000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:06:12.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:12.071+0000 7f48cf400640 1 -- 172.21.15.16:0/747153780 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f48d8146120 con 0x7f48d80d0c50 2024-03-31T15:06:12.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:12.283+0000 7f48de600640 1 -- 172.21.15.16:0/747153780 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f48d80ce060 con 0x7f48d80d0c50 2024-03-31T15:06:12.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:12.283+0000 7f48cf400640 1 -- 172.21.15.16:0/747153780 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v45) ==== 74+0+16541 (secure 0 0 0) 0x7f48c00563d0 con 0x7f48d80d0c50 2024-03-31T15:06:12.279 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:06:12.280 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":45,"fsid":"286eb5da-fb47-4550-b290-d9519c13b9a5","created":"2024-03-31T15:03:11.438952+0000","modified":"2024-03-31T15:05:28.074367+0000","last_up_change":"2024-03-31T15:03:22.885535+0000","last_in_change":"2024-03-31T15:03:13.941667+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-31T15:03:22.550787+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2024-03-31T15:05:24.856239+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"44","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2024-03-31T15:05:25.417826+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"45","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"92a66450-ef19-40d8-ac3d-d760b23b0bda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6801","nonce":932712244}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6803","nonce":932712244}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6807","nonce":932712244}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6805","nonce":932712244}]},"public_addr":"172.21.15.16:6801/932712244","cluster_addr":"172.21.15.16:6803/932712244","heartbeat_back_addr":"172.21.15.16:6807/932712244","heartbeat_front_addr":"172.21.15.16:6805/932712244","state":["exists","up"]},{"osd":1,"uuid":"da9d1f2b-1968-4959-b843-faa231988e92","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6810","nonce":1398737533}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6811","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6813","nonce":1398737533}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6821","nonce":1398737533}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6815","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6816","nonce":1398737533}]},"public_addr":"172.21.15.16:6810/1398737533","cluster_addr":"172.21.15.16:6813/1398737533","heartbeat_back_addr":"172.21.15.16:6821/1398737533","heartbeat_front_addr":"172.21.15.16:6816/1398737533","state":["exists","up"]},{"osd":2,"uuid":"01a73662-9a21-4b1f-bcbb-42b9636a4638","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6819","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6822","nonce":3018526646}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6825","nonce":3018526646}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6831","nonce":3018526646}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6827","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6829","nonce":3018526646}]},"public_addr":"172.21.15.16:6822/3018526646","cluster_addr":"172.21.15.16:6825/3018526646","heartbeat_back_addr":"172.21.15.16:6831/3018526646","heartbeat_front_addr":"172.21.15.16:6829/3018526646","state":["exists","up"]},{"osd":3,"uuid":"c7771fd3-e18e-4500-ac38-23cf10ebcd75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6809","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6812","nonce":3360352974}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6817","nonce":3360352974}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6828","nonce":3360352974}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6823","nonce":3360352974}]},"public_addr":"172.21.15.16:6812/3360352974","cluster_addr":"172.21.15.16:6817/3360352974","heartbeat_back_addr":"172.21.15.16:6828/3360352974","heartbeat_front_addr":"172.21.15.16:6823/3360352974","state":["exists","up"]},{"osd":4,"uuid":"88260730-1482-4fbd-a40f-cb1124e9df4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6801","nonce":1856844856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6803","nonce":1856844856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6807","nonce":1856844856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6805","nonce":1856844856}]},"public_addr":"172.21.15.89:6801/1856844856","cluster_addr":"172.21.15.89:6803/1856844856","heartbeat_back_addr":"172.21.15.89:6807/1856844856","heartbeat_front_addr":"172.21.15.89:6805/1856844856","state":["exists","up"]},{"osd":5,"uuid":"4cb2c00c-fae0-4dd8-9d27-ec4f0093ccf7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6814","nonce":50395527}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6817","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6820","nonce":50395527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6831","nonce":50395527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6823","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6826","nonce":50395527}]},"public_addr":"172.21.15.89:6814/50395527","cluster_addr":"172.21.15.89:6820/50395527","heartbeat_back_addr":"172.21.15.89:6831/50395527","heartbeat_front_addr":"172.21.15.89:6826/50395527","state":["exists","up"]},{"osd":6,"uuid":"81ce0c19-2554-4cae-acbe-66a05e9e9b9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6809","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6812","nonce":3470471752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6815","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6818","nonce":3470471752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6827","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6829","nonce":3470471752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6821","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6824","nonce":3470471752}]},"public_addr":"172.21.15.89:6812/3470471752","cluster_addr":"172.21.15.89:6818/3470471752","heartbeat_back_addr":"172.21.15.89:6829/3470471752","heartbeat_front_addr":"172.21.15.89:6824/3470471752","state":["exists","up"]},{"osd":7,"uuid":"85e4313b-d37d-442e-8042-153bf20a847d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6811","nonce":1796778630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6813","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6816","nonce":1796778630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6825","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6828","nonce":1796778630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6819","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6822","nonce":1796778630}]},"public_addr":"172.21.15.89:6811/1796778630","cluster_addr":"172.21.15.89:6816/1796778630","heartbeat_back_addr":"172.21.15.89:6828/1796778630","heartbeat_front_addr":"172.21.15.89:6822/1796778630","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.439088+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.529970+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.680512+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:21.044161+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.582111+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.536558+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.409522+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.590229+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-31T15:06:12.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:12.287+0000 7f48de600640 1 -- 172.21.15.16:0/747153780 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f48b4068380 msgr2=0x7f48b406a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:12.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:12.287+0000 7f48de600640 1 --2- 172.21.15.16:0/747153780 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f48b4068380 0x7f48b406a840 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f48c80af000 tx=0x7f48c80a7000 comp rx=0 tx=0).stop 2024-03-31T15:06:12.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:12.287+0000 7f48de600640 1 -- 172.21.15.16:0/747153780 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f48d80d0c50 msgr2=0x7f48d815dde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:12.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:12.287+0000 7f48de600640 1 --2- 172.21.15.16:0/747153780 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f48d80d0c50 0x7f48d815dde0 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f48c000c410 tx=0x7f48c000c8d0 comp rx=0 tx=0).stop 2024-03-31T15:06:12.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:12.287+0000 7f48de600640 1 -- 172.21.15.16:0/747153780 shutdown_connections 2024-03-31T15:06:12.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:12.287+0000 7f48de600640 1 --2- 172.21.15.16:0/747153780 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f48b4068380 0x7f48b406a840 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:12.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:12.287+0000 7f48de600640 1 --2- 172.21.15.16:0/747153780 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f48d8147e70 0x7f48d80ad8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:12.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:12.287+0000 7f48de600640 1 --2- 172.21.15.16:0/747153780 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f48d80d0c50 0x7f48d815dde0 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:12.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:12.287+0000 7f48de600640 1 --2- 172.21.15.16:0/747153780 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f48d80736e0 0x7f48d8145be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:12.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:12.287+0000 7f48de600640 1 -- 172.21.15.16:0/747153780 >> 172.21.15.16:0/747153780 conn(0x7f48d80d1060 msgr2=0x7f48d8160a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:06:12.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:12.287+0000 7f48de600640 1 -- 172.21.15.16:0/747153780 shutdown_connections 2024-03-31T15:06:12.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:12.287+0000 7f48de600640 1 -- 172.21.15.16:0/747153780 wait complete. 2024-03-31T15:06:12.300 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-31T15:06:12.300 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-31T15:06:12.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:12.447+0000 7fdfe1600640 1 Processor -- start 2024-03-31T15:06:12.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:12.447+0000 7fdfe1600640 1 -- start start 2024-03-31T15:06:12.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:12.447+0000 7fdfe1600640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fdfdc145e40 0x7fdfdc0cc740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:12.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:12.447+0000 7fdfe1600640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fdfdc0ccc80 0x7fdfdc06f920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:12.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:12.447+0000 7fdfe1600640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fdfdc06fe60 0x7fdfdc15fdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:12.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:12.447+0000 7fdfe1600640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7fdfdc0d06d0 con 0x7fdfdc145e40 2024-03-31T15:06:12.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:12.447+0000 7fdfe1600640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7fdfdc0d0920 con 0x7fdfdc0ccc80 2024-03-31T15:06:12.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:12.447+0000 7fdfe1600640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fdfdc0d0c00 con 0x7fdfdc06fe60 2024-03-31T15:06:12.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:12.447+0000 7fdfdaa00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fdfdc145e40 0x7fdfdc0cc740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:12.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:12.447+0000 7fdfdaa00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fdfdc145e40 0x7fdfdc0cc740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35150/0 (socket says 172.21.15.16:35150) 2024-03-31T15:06:12.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:12.447+0000 7fdfdaa00640 1 -- 172.21.15.16:0/3818896873 learned_addr learned my addr 172.21.15.16:0/3818896873 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:06:12.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:12.447+0000 7fdfdb400640 1 --2- 172.21.15.16:0/3818896873 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fdfdc06fe60 0x7fdfdc15fdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:12.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:12.447+0000 7fdfd3e00640 1 --2- 172.21.15.16:0/3818896873 >> v2:172.21.15.89:3300/0 conn(0x7fdfdc0ccc80 0x7fdfdc06f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:12.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:12.447+0000 7fdfdaa00640 1 -- 172.21.15.16:0/3818896873 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fdfdc06fe60 msgr2=0x7fdfdc15fdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:06:12.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:12.447+0000 7fdfdaa00640 1 --2- 172.21.15.16:0/3818896873 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fdfdc06fe60 0x7fdfdc15fdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:12.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:12.447+0000 7fdfdaa00640 1 -- 172.21.15.16:0/3818896873 >> v2:172.21.15.89:3300/0 conn(0x7fdfdc0ccc80 msgr2=0x7fdfdc06f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:06:12.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:12.447+0000 7fdfdaa00640 1 --2- 172.21.15.16:0/3818896873 >> v2:172.21.15.89:3300/0 conn(0x7fdfdc0ccc80 0x7fdfdc06f920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:12.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:12.447+0000 7fdfdaa00640 1 -- 172.21.15.16:0/3818896873 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdfdc0d1810 con 0x7fdfdc145e40 2024-03-31T15:06:12.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:12.451+0000 7fdfdaa00640 1 --2- 172.21.15.16:0/3818896873 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdfdc145e40 0x7fdfdc0cc740 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7fdfdc073cf0 tx=0x7fdfcc01c260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=237333c232c9e3bd server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:06:12.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:12.451+0000 7fdfda000640 1 -- 172.21.15.16:0/3818896873 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfcc0b0070 con 0x7fdfdc145e40 2024-03-31T15:06:12.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:12.451+0000 7fdfda000640 1 -- 172.21.15.16:0/3818896873 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdfcc0ab600 con 0x7fdfdc145e40 2024-03-31T15:06:12.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:12.451+0000 7fdfda000640 1 -- 172.21.15.16:0/3818896873 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfcc0ab9d0 con 0x7fdfdc145e40 2024-03-31T15:06:12.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:12.451+0000 7fdfe1600640 1 -- 172.21.15.16:0/3818896873 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdfdc145e40 msgr2=0x7fdfdc0cc740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:12.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:12.451+0000 7fdfe1600640 1 --2- 172.21.15.16:0/3818896873 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdfdc145e40 0x7fdfdc0cc740 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7fdfdc073cf0 tx=0x7fdfcc01c260 comp rx=0 tx=0).stop 2024-03-31T15:06:12.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:12.451+0000 7fdfe1600640 1 -- 172.21.15.16:0/3818896873 shutdown_connections 2024-03-31T15:06:12.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:12.451+0000 7fdfe1600640 1 --2- 172.21.15.16:0/3818896873 >> v2:172.21.15.89:3300/0 conn(0x7fdfdc0ccc80 0x7fdfdc06f920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:12.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:12.451+0000 7fdfe1600640 1 --2- 172.21.15.16:0/3818896873 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fdfdc06fe60 0x7fdfdc15fdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:12.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:12.451+0000 7fdfe1600640 1 --2- 172.21.15.16:0/3818896873 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdfdc145e40 0x7fdfdc0cc740 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:12.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:12.451+0000 7fdfe1600640 1 -- 172.21.15.16:0/3818896873 >> 172.21.15.16:0/3818896873 conn(0x7fdfdc067330 msgr2=0x7fdfdc067740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:06:12.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:12.451+0000 7fdfe1600640 1 -- 172.21.15.16:0/3818896873 shutdown_connections 2024-03-31T15:06:12.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:12.451+0000 7fdfe1600640 1 -- 172.21.15.16:0/3818896873 wait complete. 2024-03-31T15:06:12.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:12.455+0000 7fdfe1600640 1 Processor -- start 2024-03-31T15:06:12.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:12.455+0000 7fdfe1600640 1 -- start start 2024-03-31T15:06:12.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:12.455+0000 7fdfe1600640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdfdc06fe60 0x7fdfdc0b47c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:12.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:12.455+0000 7fdfe1600640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdfdc0ccc80 0x7fdfdc153250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:12.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:12.455+0000 7fdfe1600640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fdfdc145e40 0x7fdfdc06def0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:12.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:12.455+0000 7fdfe1600640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fdfdc0b5240 con 0x7fdfdc0ccc80 2024-03-31T15:06:12.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:12.455+0000 7fdfe1600640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fdfdc0b50c0 con 0x7fdfdc06fe60 2024-03-31T15:06:12.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:12.455+0000 7fdfe1600640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fdfdc0d2010 con 0x7fdfdc145e40 2024-03-31T15:06:12.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:12.455+0000 7fdfd3e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdfdc0ccc80 0x7fdfdc153250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:12.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:12.455+0000 7fdfd3e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdfdc0ccc80 0x7fdfdc153250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35156/0 (socket says 172.21.15.16:35156) 2024-03-31T15:06:12.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:12.455+0000 7fdfd3e00640 1 -- 172.21.15.16:0/279921489 learned_addr learned my addr 172.21.15.16:0/279921489 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:06:12.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:12.455+0000 7fdfdaa00640 1 --2- 172.21.15.16:0/279921489 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdfdc06fe60 0x7fdfdc0b47c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:12.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:12.455+0000 7fdfdb400640 1 --2- 172.21.15.16:0/279921489 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fdfdc145e40 0x7fdfdc06def0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:12.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:12.455+0000 7fdfd3e00640 1 -- 172.21.15.16:0/279921489 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fdfdc145e40 msgr2=0x7fdfdc06def0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:12.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:12.455+0000 7fdfd3e00640 1 --2- 172.21.15.16:0/279921489 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fdfdc145e40 0x7fdfdc06def0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:12.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:12.455+0000 7fdfd3e00640 1 -- 172.21.15.16:0/279921489 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdfdc06fe60 msgr2=0x7fdfdc0b47c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:12.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:12.455+0000 7fdfd3e00640 1 --2- 172.21.15.16:0/279921489 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdfdc06fe60 0x7fdfdc0b47c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:12.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:12.455+0000 7fdfd3e00640 1 -- 172.21.15.16:0/279921489 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdfdc153af0 con 0x7fdfdc0ccc80 2024-03-31T15:06:12.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:12.455+0000 7fdfdaa00640 1 --2- 172.21.15.16:0/279921489 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdfdc06fe60 0x7fdfdc0b47c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:06:12.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:12.455+0000 7fdfd3e00640 1 --2- 172.21.15.16:0/279921489 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdfdc0ccc80 0x7fdfdc153250 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7fdfc400e960 tx=0x7fdfc400ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:06:12.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:12.455+0000 7fdfd2a00640 1 -- 172.21.15.16:0/279921489 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfc4017070 con 0x7fdfdc0ccc80 2024-03-31T15:06:12.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:12.455+0000 7fdfd2a00640 1 -- 172.21.15.16:0/279921489 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdfc400ca00 con 0x7fdfdc0ccc80 2024-03-31T15:06:12.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:12.455+0000 7fdfe1600640 1 -- 172.21.15.16:0/279921489 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdfdc153dd0 con 0x7fdfdc0ccc80 2024-03-31T15:06:12.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:12.455+0000 7fdfd2a00640 1 -- 172.21.15.16:0/279921489 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfc400cdb0 con 0x7fdfdc0ccc80 2024-03-31T15:06:12.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:12.455+0000 7fdfe1600640 1 -- 172.21.15.16:0/279921489 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdfdc154390 con 0x7fdfdc0ccc80 2024-03-31T15:06:12.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:12.459+0000 7fdfe1600640 1 -- 172.21.15.16:0/279921489 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdfdc069950 con 0x7fdfdc0ccc80 2024-03-31T15:06:12.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:12.459+0000 7fdfd2a00640 1 -- 172.21.15.16:0/279921489 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fdfc401a070 con 0x7fdfdc0ccc80 2024-03-31T15:06:12.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:12.459+0000 7fdfd2a00640 1 --2- 172.21.15.16:0/279921489 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fdfb8068380 0x7fdfb806a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:12.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:12.459+0000 7fdfd2a00640 1 -- 172.21.15.16:0/279921489 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5709+0+0 (secure 0 0 0) 0x7fdfc4086c40 con 0x7fdfdc0ccc80 2024-03-31T15:06:12.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:12.459+0000 7fdfdaa00640 1 --2- 172.21.15.16:0/279921489 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fdfb8068380 0x7fdfb806a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:12.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:12.459+0000 7fdfdaa00640 1 --2- 172.21.15.16:0/279921489 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fdfb8068380 0x7fdfb806a840 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fdfcc0ad000 tx=0x7fdfcc0a6000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:06:12.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:12.463+0000 7fdfd2a00640 1 -- 172.21.15.16:0/279921489 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fdfc4056bb0 con 0x7fdfdc0ccc80 2024-03-31T15:06:12.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:12.663+0000 7fdfe1600640 1 -- 172.21.15.16:0/279921489 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fdfdc145600 con 0x7fdfdc0ccc80 2024-03-31T15:06:12.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:12.663+0000 7fdfd2a00640 1 -- 172.21.15.16:0/279921489 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+5175 (secure 0 0 0) 0x7fdfc405aeb0 con 0x7fdfdc0ccc80 2024-03-31T15:06:12.659 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:06:12.659 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":25,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/285985105","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6835","nonce":285985105},{"type":"v1","addr":"172.21.15.89:6837","nonce":285985105}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20},{"gid":5689,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6836/1615931247","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":1615931247},{"type":"v1","addr":"172.21.15.89:6836","nonce":1615931247}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20},{"gid":5694,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/512187852","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6833","nonce":512187852},{"type":"v1","addr":"172.21.15.16:6835","nonce":512187852}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-31T15:05:26.461908+0000","modified":"2024-03-31T15:05:29.491163+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5697},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5697":{"gid":5697,"name":"a","rank":0,"incarnation":21,"state":"up:active","state_seq":3,"addr":"172.21.15.16:6834/2768973409","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":2768973409},{"type":"v1","addr":"172.21.15.16:6834","nonce":2768973409}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5697,"qdb_cluster":[5697]},"id":3}]} 2024-03-31T15:06:12.659 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 25 2024-03-31T15:06:12.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:12.667+0000 7fdfe1600640 1 -- 172.21.15.16:0/279921489 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fdfb8068380 msgr2=0x7fdfb806a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:12.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:12.667+0000 7fdfe1600640 1 --2- 172.21.15.16:0/279921489 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fdfb8068380 0x7fdfb806a840 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fdfcc0ad000 tx=0x7fdfcc0a6000 comp rx=0 tx=0).stop 2024-03-31T15:06:12.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:12.667+0000 7fdfe1600640 1 -- 172.21.15.16:0/279921489 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdfdc0ccc80 msgr2=0x7fdfdc153250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:12.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:12.667+0000 7fdfe1600640 1 --2- 172.21.15.16:0/279921489 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdfdc0ccc80 0x7fdfdc153250 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7fdfc400e960 tx=0x7fdfc400ee20 comp rx=0 tx=0).stop 2024-03-31T15:06:12.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:12.667+0000 7fdfe1600640 1 -- 172.21.15.16:0/279921489 shutdown_connections 2024-03-31T15:06:12.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:12.667+0000 7fdfe1600640 1 --2- 172.21.15.16:0/279921489 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fdfb8068380 0x7fdfb806a840 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:12.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:12.667+0000 7fdfe1600640 1 --2- 172.21.15.16:0/279921489 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fdfdc145e40 0x7fdfdc06def0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:12.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:12.667+0000 7fdfe1600640 1 --2- 172.21.15.16:0/279921489 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdfdc0ccc80 0x7fdfdc153250 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:12.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:12.667+0000 7fdfe1600640 1 --2- 172.21.15.16:0/279921489 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdfdc06fe60 0x7fdfdc0b47c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:12.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:12.667+0000 7fdfe1600640 1 -- 172.21.15.16:0/279921489 >> 172.21.15.16:0/279921489 conn(0x7fdfdc067330 msgr2=0x7fdfdc0b2ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:06:12.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:12.667+0000 7fdfe1600640 1 -- 172.21.15.16:0/279921489 shutdown_connections 2024-03-31T15:06:12.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:12.667+0000 7fdfe1600640 1 -- 172.21.15.16:0/279921489 wait complete. 2024-03-31T15:06:12.681 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-31T15:06:12.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:12.827+0000 7fed41400640 1 Processor -- start 2024-03-31T15:06:12.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:12.827+0000 7fed41400640 1 -- start start 2024-03-31T15:06:12.822 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:12.827+0000 7fed41400640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fed3c06c1c0 0x7fed3c15c8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:12.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:12.827+0000 7fed41400640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fed3c06b620 0x7fed3c06bab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:12.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:12.827+0000 7fed41400640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fed3c0d09f0 con 0x7fed3c145aa0 2024-03-31T15:06:12.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:12.827+0000 7fed41400640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7fed3c06fd50 con 0x7fed3c06c1c0 2024-03-31T15:06:12.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:12.827+0000 7fed41400640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fed3c0d0cd0 con 0x7fed3c06b620 2024-03-31T15:06:12.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:12.827+0000 7fed3b200640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fed3c06b620 0x7fed3c06bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:12.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:12.827+0000 7fed33e00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fed3c06c1c0 0x7fed3c15c8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:12.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:12.827+0000 7fed3b200640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fed3c06b620 0x7fed3c06bab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3301/0 says I am v2:172.21.15.16:52360/0 (socket says 172.21.15.16:52360) 2024-03-31T15:06:12.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:12.827+0000 7fed33e00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fed3c06c1c0 0x7fed3c15c8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.16:51182/0 (socket says 172.21.15.16:51182) 2024-03-31T15:06:12.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:12.827+0000 7fed3b200640 1 -- 172.21.15.16:0/340786097 learned_addr learned my addr 172.21.15.16:0/340786097 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:06:12.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:12.827+0000 7fed3a800640 1 --1- 172.21.15.16:0/340786097 >> v1:172.21.15.89:6789/0 conn(0x7fed3c145aa0 0x7fed3c072e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.16:51910/0 (socket says 172.21.15.16:51910) 2024-03-31T15:06:12.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:12.827+0000 7fed3b200640 1 -- 172.21.15.16:0/340786097 >> v2:172.21.15.89:3300/0 conn(0x7fed3c06c1c0 msgr2=0x7fed3c15c8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:06:12.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:12.827+0000 7fed3b200640 1 --2- 172.21.15.16:0/340786097 >> v2:172.21.15.89:3300/0 conn(0x7fed3c06c1c0 0x7fed3c15c8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:12.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:12.827+0000 7fed3b200640 1 -- 172.21.15.16:0/340786097 >> v1:172.21.15.89:6789/0 conn(0x7fed3c145aa0 legacy=0x7fed3c072e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:12.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:12.827+0000 7fed3b200640 1 -- 172.21.15.16:0/340786097 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed3c0d06e0 con 0x7fed3c06b620 2024-03-31T15:06:12.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:12.827+0000 7fed33e00640 1 --2- 172.21.15.16:0/340786097 >> v2:172.21.15.89:3300/0 conn(0x7fed3c06c1c0 0x7fed3c15c8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:06:12.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:12.831+0000 7fed3b200640 1 --2- 172.21.15.16:0/340786097 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fed3c06b620 0x7fed3c06bab0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fed34009e50 tx=0x7fed3401ca30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d7d4a69934bc22c0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:06:12.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:12.831+0000 7fed39e00640 1 -- 172.21.15.16:0/340786097 <== mon.2 v2:172.21.15.89:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed340a5700 con 0x7fed3c06b620 2024-03-31T15:06:12.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:12.831+0000 7fed39e00640 1 -- 172.21.15.16:0/340786097 <== mon.2 v2:172.21.15.89:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fed340a58a0 con 0x7fed3c06b620 2024-03-31T15:06:12.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:12.831+0000 7fed39e00640 1 -- 172.21.15.16:0/340786097 <== mon.2 v2:172.21.15.89:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed340a5c70 con 0x7fed3c06b620 2024-03-31T15:06:12.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:12.831+0000 7fed41400640 1 -- 172.21.15.16:0/340786097 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fed3c06b620 msgr2=0x7fed3c06bab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:12.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:12.831+0000 7fed41400640 1 --2- 172.21.15.16:0/340786097 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fed3c06b620 0x7fed3c06bab0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fed34009e50 tx=0x7fed3401ca30 comp rx=0 tx=0).stop 2024-03-31T15:06:12.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:12.831+0000 7fed41400640 1 -- 172.21.15.16:0/340786097 shutdown_connections 2024-03-31T15:06:12.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:12.831+0000 7fed41400640 1 --2- 172.21.15.16:0/340786097 >> v2:172.21.15.89:3300/0 conn(0x7fed3c06c1c0 0x7fed3c15c8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:12.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:12.831+0000 7fed41400640 1 --2- 172.21.15.16:0/340786097 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fed3c06b620 0x7fed3c06bab0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:12.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:12.831+0000 7fed41400640 1 -- 172.21.15.16:0/340786097 >> 172.21.15.16:0/340786097 conn(0x7fed3c0d1060 msgr2=0x7fed3c0d1470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:06:12.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:12.831+0000 7fed41400640 1 -- 172.21.15.16:0/340786097 shutdown_connections 2024-03-31T15:06:12.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:12.831+0000 7fed41400640 1 -- 172.21.15.16:0/340786097 wait complete. 2024-03-31T15:06:12.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:12.835+0000 7fed41400640 1 Processor -- start 2024-03-31T15:06:12.829 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:12.835+0000 7fed41400640 1 -- start start 2024-03-31T15:06:12.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:12.835+0000 7fed41400640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fed3c06b620 0x7fed3c0c6040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:12.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:12.835+0000 7fed41400640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fed3c06c1c0 0x7fed3c0af330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:12.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:12.835+0000 7fed41400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fed3c145aa0 0x7fed3c1f6c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:12.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:12.835+0000 7fed41400640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fed3c0ce6d0 con 0x7fed3c145aa0 2024-03-31T15:06:12.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:12.835+0000 7fed41400640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fed3c0ce550 con 0x7fed3c06c1c0 2024-03-31T15:06:12.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:12.835+0000 7fed41400640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fed3c0ce850 con 0x7fed3c06b620 2024-03-31T15:06:12.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:12.835+0000 7fed3b200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fed3c145aa0 0x7fed3c1f6c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:12.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:12.835+0000 7fed3b200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fed3c145aa0 0x7fed3c1f6c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35164/0 (socket says 172.21.15.16:35164) 2024-03-31T15:06:12.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:12.835+0000 7fed3b200640 1 -- 172.21.15.16:0/4010785469 learned_addr learned my addr 172.21.15.16:0/4010785469 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:06:12.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:12.835+0000 7fed3a800640 1 --2- 172.21.15.16:0/4010785469 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fed3c06b620 0x7fed3c0c6040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:12.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:12.835+0000 7fed33e00640 1 --2- 172.21.15.16:0/4010785469 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fed3c06c1c0 0x7fed3c0af330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:12.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:12.835+0000 7fed3b200640 1 -- 172.21.15.16:0/4010785469 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fed3c06b620 msgr2=0x7fed3c0c6040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:12.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:12.835+0000 7fed3b200640 1 --2- 172.21.15.16:0/4010785469 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fed3c06b620 0x7fed3c0c6040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:12.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:12.835+0000 7fed3b200640 1 -- 172.21.15.16:0/4010785469 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fed3c06c1c0 msgr2=0x7fed3c0af330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:12.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:12.835+0000 7fed3b200640 1 --2- 172.21.15.16:0/4010785469 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fed3c06c1c0 0x7fed3c0af330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:12.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:12.835+0000 7fed3b200640 1 -- 172.21.15.16:0/4010785469 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed3c1437a0 con 0x7fed3c145aa0 2024-03-31T15:06:12.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:12.835+0000 7fed3b200640 1 --2- 172.21.15.16:0/4010785469 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fed3c145aa0 0x7fed3c1f6c90 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7fed3401a660 tx=0x7fed340197d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:06:12.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:12.835+0000 7fed32a00640 1 -- 172.21.15.16:0/4010785469 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed340a5700 con 0x7fed3c145aa0 2024-03-31T15:06:12.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:12.835+0000 7fed32a00640 1 -- 172.21.15.16:0/4010785469 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fed340be070 con 0x7fed3c145aa0 2024-03-31T15:06:12.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:12.835+0000 7fed41400640 1 -- 172.21.15.16:0/4010785469 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fed3c142f00 con 0x7fed3c145aa0 2024-03-31T15:06:12.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:12.835+0000 7fed32a00640 1 -- 172.21.15.16:0/4010785469 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed34009510 con 0x7fed3c145aa0 2024-03-31T15:06:12.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:12.835+0000 7fed41400640 1 -- 172.21.15.16:0/4010785469 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fed3c1434c0 con 0x7fed3c145aa0 2024-03-31T15:06:12.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:12.839+0000 7fed41400640 1 -- 172.21.15.16:0/4010785469 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fed3c069950 con 0x7fed3c145aa0 2024-03-31T15:06:12.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:12.839+0000 7fed32a00640 1 -- 172.21.15.16:0/4010785469 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fed340b8ac0 con 0x7fed3c145aa0 2024-03-31T15:06:12.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:12.839+0000 7fed32a00640 1 --2- 172.21.15.16:0/4010785469 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fed18068380 0x7fed1806a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:12.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:12.839+0000 7fed32a00640 1 -- 172.21.15.16:0/4010785469 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5709+0+0 (secure 0 0 0) 0x7fed34121bc0 con 0x7fed3c145aa0 2024-03-31T15:06:12.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:12.839+0000 7fed3a800640 1 --2- 172.21.15.16:0/4010785469 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fed18068380 0x7fed1806a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:12.835 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:12.839+0000 7fed3a800640 1 --2- 172.21.15.16:0/4010785469 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fed18068380 0x7fed1806a840 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fed2c000f70 tx=0x7fed2c007e90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:06:12.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:12.847+0000 7fed32a00640 1 -- 172.21.15.16:0/4010785469 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fed34016620 con 0x7fed3c145aa0 2024-03-31T15:06:13.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:13.047+0000 7fed41400640 1 -- 172.21.15.16:0/4010785469 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fed3c06d740 con 0x7fed3c145aa0 2024-03-31T15:06:13.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:13.047+0000 7fed32a00640 1 -- 172.21.15.16:0/4010785469 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+5175 (secure 0 0 0) 0x7fed340f1aa0 con 0x7fed3c145aa0 2024-03-31T15:06:13.042 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:06:13.042 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":25,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/285985105","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6835","nonce":285985105},{"type":"v1","addr":"172.21.15.89:6837","nonce":285985105}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20},{"gid":5689,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6836/1615931247","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":1615931247},{"type":"v1","addr":"172.21.15.89:6836","nonce":1615931247}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20},{"gid":5694,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/512187852","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6833","nonce":512187852},{"type":"v1","addr":"172.21.15.16:6835","nonce":512187852}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-31T15:05:26.461908+0000","modified":"2024-03-31T15:05:29.491163+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5697},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5697":{"gid":5697,"name":"a","rank":0,"incarnation":21,"state":"up:active","state_seq":3,"addr":"172.21.15.16:6834/2768973409","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":2768973409},{"type":"v1","addr":"172.21.15.16:6834","nonce":2768973409}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5697,"qdb_cluster":[5697]},"id":3}]} 2024-03-31T15:06:13.042 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 25 2024-03-31T15:06:13.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:13.047+0000 7fed41400640 1 -- 172.21.15.16:0/4010785469 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fed18068380 msgr2=0x7fed1806a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:13.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:13.047+0000 7fed41400640 1 --2- 172.21.15.16:0/4010785469 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fed18068380 0x7fed1806a840 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fed2c000f70 tx=0x7fed2c007e90 comp rx=0 tx=0).stop 2024-03-31T15:06:13.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:13.047+0000 7fed41400640 1 -- 172.21.15.16:0/4010785469 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fed3c145aa0 msgr2=0x7fed3c1f6c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:13.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:13.047+0000 7fed41400640 1 --2- 172.21.15.16:0/4010785469 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fed3c145aa0 0x7fed3c1f6c90 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7fed3401a660 tx=0x7fed340197d0 comp rx=0 tx=0).stop 2024-03-31T15:06:13.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:13.047+0000 7fed41400640 1 -- 172.21.15.16:0/4010785469 shutdown_connections 2024-03-31T15:06:13.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:13.047+0000 7fed41400640 1 --2- 172.21.15.16:0/4010785469 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fed18068380 0x7fed1806a840 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:13.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:13.047+0000 7fed41400640 1 --2- 172.21.15.16:0/4010785469 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fed3c145aa0 0x7fed3c1f6c90 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:13.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:13.047+0000 7fed41400640 1 --2- 172.21.15.16:0/4010785469 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fed3c06c1c0 0x7fed3c0af330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:13.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:13.047+0000 7fed41400640 1 --2- 172.21.15.16:0/4010785469 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fed3c06b620 0x7fed3c0c6040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:13.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:13.047+0000 7fed41400640 1 -- 172.21.15.16:0/4010785469 >> 172.21.15.16:0/4010785469 conn(0x7fed3c0d1060 msgr2=0x7fed3c144dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:06:13.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:13.047+0000 7fed41400640 1 -- 172.21.15.16:0/4010785469 shutdown_connections 2024-03-31T15:06:13.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:13.051+0000 7fed41400640 1 -- 172.21.15.16:0/4010785469 wait complete. 2024-03-31T15:06:13.063 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-31T15:06:13.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:13.207+0000 7f333c800640 1 Processor -- start 2024-03-31T15:06:13.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:13.207+0000 7f333c800640 1 -- start start 2024-03-31T15:06:13.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:13.211+0000 7f333c800640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f333406d490 0x7f333406d8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:13.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:13.211+0000 7f333c800640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3334162010 con 0x7f333406f920 2024-03-31T15:06:13.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:13.211+0000 7f333c800640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f33341622b0 con 0x7f33340cc740 2024-03-31T15:06:13.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:13.211+0000 7f333c800640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f3334162be0 con 0x7f333406d490 2024-03-31T15:06:13.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:13.211+0000 7f3339e00640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f333406f920 0x7f333406fd00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:37184/0 (socket says 172.21.15.16:37184) 2024-03-31T15:06:13.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:13.211+0000 7f3339e00640 1 -- 172.21.15.16:0/2102656688 learned_addr learned my addr 172.21.15.16:0/2102656688 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:06:13.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:13.211+0000 7f3339400640 1 --2- 172.21.15.16:0/2102656688 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f333406d490 0x7f333406d8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:13.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:13.211+0000 7f3338a00640 1 -- 172.21.15.16:0/2102656688 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3109746152 0 0) 0x7f3334162010 con 0x7f333406f920 2024-03-31T15:06:13.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:13.211+0000 7f3338a00640 1 -- 172.21.15.16:0/2102656688 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f331c003530 con 0x7f333406f920 2024-03-31T15:06:13.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:13.211+0000 7f3338a00640 1 -- 172.21.15.16:0/2102656688 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3450066762 0 0) 0x7f331c003530 con 0x7f333406f920 2024-03-31T15:06:13.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:13.211+0000 7f3338a00640 1 -- 172.21.15.16:0/2102656688 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f333406d490 msgr2=0x7f333406d8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:06:13.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:13.211+0000 7f3338a00640 1 --2- 172.21.15.16:0/2102656688 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f333406d490 0x7f333406d8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:13.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:13.211+0000 7f3338a00640 1 -- 172.21.15.16:0/2102656688 >> v1:172.21.15.89:6789/0 conn(0x7f33340cc740 legacy=0x7f33340ccba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:13.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:13.211+0000 7f3338a00640 1 -- 172.21.15.16:0/2102656688 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3334162590 con 0x7f333406f920 2024-03-31T15:06:13.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:13.211+0000 7f3338a00640 1 -- 172.21.15.16:0/2102656688 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 214807623 0 0) 0x7f3328001680 con 0x7f333406f920 2024-03-31T15:06:13.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:13.211+0000 7f3338a00640 1 -- 172.21.15.16:0/2102656688 >> v1:172.21.15.16:6789/0 conn(0x7f333406f920 legacy=0x7f333406fd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:13.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:13.211+0000 7f3338a00640 1 --2- 172.21.15.16:0/2102656688 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f331c0039f0 0x7f331c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:13.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:13.211+0000 7f3338a00640 1 --2- 172.21.15.16:0/2102656688 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f331c004810 0x7f331c004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:13.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:13.211+0000 7f3338a00640 1 --2- 172.21.15.16:0/2102656688 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f333406d490 0x7f331c0051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:13.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:13.211+0000 7f3338a00640 1 -- 172.21.15.16:0/2102656688 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f3334162be0 con 0x7f331c0039f0 2024-03-31T15:06:13.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:13.211+0000 7f3338a00640 1 -- 172.21.15.16:0/2102656688 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f331c00ba80 con 0x7f331c004810 2024-03-31T15:06:13.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:13.211+0000 7f3338a00640 1 -- 172.21.15.16:0/2102656688 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f331c00bc30 con 0x7f333406d490 2024-03-31T15:06:13.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:13.211+0000 7f3339e00640 1 --2- 172.21.15.16:0/2102656688 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f331c0039f0 0x7f331c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:13.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:13.211+0000 7f3339400640 1 --2- 172.21.15.16:0/2102656688 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f331c004810 0x7f331c004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:13.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:13.211+0000 7f333a800640 1 --2- 172.21.15.16:0/2102656688 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f333406d490 0x7f331c0051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:13.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:13.211+0000 7f3339e00640 1 -- 172.21.15.16:0/2102656688 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f333406d490 msgr2=0x7f331c0051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:06:13.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:13.211+0000 7f3339e00640 1 --2- 172.21.15.16:0/2102656688 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f333406d490 0x7f331c0051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:13.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:13.211+0000 7f3339e00640 1 -- 172.21.15.16:0/2102656688 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f331c004810 msgr2=0x7f331c004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:06:13.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:13.211+0000 7f3339e00640 1 --2- 172.21.15.16:0/2102656688 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f331c004810 0x7f331c004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:13.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:13.211+0000 7f3339e00640 1 -- 172.21.15.16:0/2102656688 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f331c003530 con 0x7f331c0039f0 2024-03-31T15:06:13.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:13.211+0000 7f333a800640 1 --2- 172.21.15.16:0/2102656688 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f333406d490 0x7f331c0051d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:06:13.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:13.215+0000 7f3339e00640 1 --2- 172.21.15.16:0/2102656688 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f331c0039f0 0x7f331c003dd0 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f332800c0e0 tx=0x7f332801cad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c3978a9415bb23e6 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:06:13.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:13.215+0000 7f3338a00640 1 -- 172.21.15.16:0/2102656688 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3328007c50 con 0x7f331c0039f0 2024-03-31T15:06:13.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:13.215+0000 7f3338a00640 1 -- 172.21.15.16:0/2102656688 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f33280a9020 con 0x7f331c0039f0 2024-03-31T15:06:13.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:13.215+0000 7f3338a00640 1 -- 172.21.15.16:0/2102656688 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3328005ae0 con 0x7f331c0039f0 2024-03-31T15:06:13.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:13.215+0000 7f333c800640 1 -- 172.21.15.16:0/2102656688 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f331c0039f0 msgr2=0x7f331c003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:13.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:13.215+0000 7f333c800640 1 --2- 172.21.15.16:0/2102656688 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f331c0039f0 0x7f331c003dd0 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f332800c0e0 tx=0x7f332801cad0 comp rx=0 tx=0).stop 2024-03-31T15:06:13.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:13.215+0000 7f333a800640 1 -- 172.21.15.16:0/2102656688 reap_dead start 2024-03-31T15:06:13.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:13.215+0000 7f333c800640 1 -- 172.21.15.16:0/2102656688 shutdown_connections 2024-03-31T15:06:13.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:13.215+0000 7f333c800640 1 -- 172.21.15.16:0/2102656688 >> 172.21.15.16:0/2102656688 conn(0x7f3334067330 msgr2=0x7f3334067740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:06:13.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:13.215+0000 7f333c800640 1 -- 172.21.15.16:0/2102656688 shutdown_connections 2024-03-31T15:06:13.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:13.215+0000 7f333c800640 1 -- 172.21.15.16:0/2102656688 wait complete. 2024-03-31T15:06:13.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:13.215+0000 7f333c800640 1 Processor -- start 2024-03-31T15:06:13.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:13.215+0000 7f333c800640 1 -- start start 2024-03-31T15:06:13.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:13.219+0000 7f333c800640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f331c0039f0 0x7f33341fa360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:13.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:13.219+0000 7f333c800640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f33340b4a20 0x7f3334142e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:13.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:13.219+0000 7f333c800640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f333415bfd0 0x7f333415c480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:13.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:13.219+0000 7f333c800640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f333406dc80 con 0x7f333415bfd0 2024-03-31T15:06:13.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:13.219+0000 7f333c800640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f333406db00 con 0x7f33340b4a20 2024-03-31T15:06:13.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:13.219+0000 7f333c800640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f333406de00 con 0x7f331c0039f0 2024-03-31T15:06:13.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:13.219+0000 7f333a800640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f333415bfd0 0x7f333415c480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:13.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:13.219+0000 7f333a800640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f333415bfd0 0x7f333415c480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35184/0 (socket says 172.21.15.16:35184) 2024-03-31T15:06:13.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:13.219+0000 7f333a800640 1 -- 172.21.15.16:0/1911524409 learned_addr learned my addr 172.21.15.16:0/1911524409 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:06:13.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:13.219+0000 7f3339400640 1 --2- 172.21.15.16:0/1911524409 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f33340b4a20 0x7f3334142e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:13.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:13.219+0000 7f3339e00640 1 --2- 172.21.15.16:0/1911524409 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f331c0039f0 0x7f33341fa360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:13.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:13.219+0000 7f333a800640 1 -- 172.21.15.16:0/1911524409 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f331c0039f0 msgr2=0x7f33341fa360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:13.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:13.219+0000 7f333a800640 1 --2- 172.21.15.16:0/1911524409 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f331c0039f0 0x7f33341fa360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:13.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:13.219+0000 7f333a800640 1 -- 172.21.15.16:0/1911524409 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f33340b4a20 msgr2=0x7f3334142e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:13.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:13.219+0000 7f333a800640 1 --2- 172.21.15.16:0/1911524409 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f33340b4a20 0x7f3334142e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:13.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:13.219+0000 7f333a800640 1 -- 172.21.15.16:0/1911524409 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f333415c9c0 con 0x7f333415bfd0 2024-03-31T15:06:13.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:13.219+0000 7f333a800640 1 --2- 172.21.15.16:0/1911524409 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f333415bfd0 0x7f333415c480 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f331c0039c0 tx=0x7f333000eb60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:06:13.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:13.219+0000 7f332e000640 1 -- 172.21.15.16:0/1911524409 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3330003980 con 0x7f333415bfd0 2024-03-31T15:06:13.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:13.219+0000 7f332e000640 1 -- 172.21.15.16:0/1911524409 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3330018070 con 0x7f333415bfd0 2024-03-31T15:06:13.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:13.219+0000 7f332e000640 1 -- 172.21.15.16:0/1911524409 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3330003d80 con 0x7f333415bfd0 2024-03-31T15:06:13.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:13.219+0000 7f333c800640 1 -- 172.21.15.16:0/1911524409 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f33341facd0 con 0x7f333415bfd0 2024-03-31T15:06:13.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:13.219+0000 7f333c800640 1 -- 172.21.15.16:0/1911524409 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3334160060 con 0x7f333415bfd0 2024-03-31T15:06:13.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:13.219+0000 7f333c800640 1 -- 172.21.15.16:0/1911524409 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f33341fb1f0 con 0x7f333415bfd0 2024-03-31T15:06:13.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:13.223+0000 7f332e000640 1 -- 172.21.15.16:0/1911524409 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f33300050d0 con 0x7f333415bfd0 2024-03-31T15:06:13.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:13.223+0000 7f332e000640 1 --2- 172.21.15.16:0/1911524409 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f3318068380 0x7f331806a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:13.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:13.223+0000 7f332e000640 1 -- 172.21.15.16:0/1911524409 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5709+0+0 (secure 0 0 0) 0x7f3330086750 con 0x7f333415bfd0 2024-03-31T15:06:13.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:13.223+0000 7f3339e00640 1 --2- 172.21.15.16:0/1911524409 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f3318068380 0x7f331806a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:13.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:13.223+0000 7f3339e00640 1 --2- 172.21.15.16:0/1911524409 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f3318068380 0x7f331806a840 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f33280aa000 tx=0x7f33280b1040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:06:13.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:13.227+0000 7f332e000640 1 -- 172.21.15.16:0/1911524409 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f333415c9c0 con 0x7f333415bfd0 2024-03-31T15:06:13.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:13.391+0000 7f333c800640 1 -- 172.21.15.16:0/1911524409 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f3334069950 con 0x7f333415bfd0 2024-03-31T15:06:13.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:13.391+0000 7f332e000640 1 -- 172.21.15.16:0/1911524409 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v45) ==== 74+0+16541 (secure 0 0 0) 0x7f3330056c70 con 0x7f333415bfd0 2024-03-31T15:06:13.387 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:06:13.387 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":45,"fsid":"286eb5da-fb47-4550-b290-d9519c13b9a5","created":"2024-03-31T15:03:11.438952+0000","modified":"2024-03-31T15:05:28.074367+0000","last_up_change":"2024-03-31T15:03:22.885535+0000","last_in_change":"2024-03-31T15:03:13.941667+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-31T15:03:22.550787+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2024-03-31T15:05:24.856239+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"44","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2024-03-31T15:05:25.417826+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"45","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"92a66450-ef19-40d8-ac3d-d760b23b0bda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6801","nonce":932712244}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6803","nonce":932712244}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6807","nonce":932712244}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6805","nonce":932712244}]},"public_addr":"172.21.15.16:6801/932712244","cluster_addr":"172.21.15.16:6803/932712244","heartbeat_back_addr":"172.21.15.16:6807/932712244","heartbeat_front_addr":"172.21.15.16:6805/932712244","state":["exists","up"]},{"osd":1,"uuid":"da9d1f2b-1968-4959-b843-faa231988e92","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6810","nonce":1398737533}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6811","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6813","nonce":1398737533}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6821","nonce":1398737533}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6815","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6816","nonce":1398737533}]},"public_addr":"172.21.15.16:6810/1398737533","cluster_addr":"172.21.15.16:6813/1398737533","heartbeat_back_addr":"172.21.15.16:6821/1398737533","heartbeat_front_addr":"172.21.15.16:6816/1398737533","state":["exists","up"]},{"osd":2,"uuid":"01a73662-9a21-4b1f-bcbb-42b9636a4638","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6819","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6822","nonce":3018526646}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6825","nonce":3018526646}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6831","nonce":3018526646}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6827","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6829","nonce":3018526646}]},"public_addr":"172.21.15.16:6822/3018526646","cluster_addr":"172.21.15.16:6825/3018526646","heartbeat_back_addr":"172.21.15.16:6831/3018526646","heartbeat_front_addr":"172.21.15.16:6829/3018526646","state":["exists","up"]},{"osd":3,"uuid":"c7771fd3-e18e-4500-ac38-23cf10ebcd75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6809","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6812","nonce":3360352974}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6817","nonce":3360352974}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6828","nonce":3360352974}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6823","nonce":3360352974}]},"public_addr":"172.21.15.16:6812/3360352974","cluster_addr":"172.21.15.16:6817/3360352974","heartbeat_back_addr":"172.21.15.16:6828/3360352974","heartbeat_front_addr":"172.21.15.16:6823/3360352974","state":["exists","up"]},{"osd":4,"uuid":"88260730-1482-4fbd-a40f-cb1124e9df4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6801","nonce":1856844856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6803","nonce":1856844856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6807","nonce":1856844856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6805","nonce":1856844856}]},"public_addr":"172.21.15.89:6801/1856844856","cluster_addr":"172.21.15.89:6803/1856844856","heartbeat_back_addr":"172.21.15.89:6807/1856844856","heartbeat_front_addr":"172.21.15.89:6805/1856844856","state":["exists","up"]},{"osd":5,"uuid":"4cb2c00c-fae0-4dd8-9d27-ec4f0093ccf7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6814","nonce":50395527}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6817","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6820","nonce":50395527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6831","nonce":50395527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6823","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6826","nonce":50395527}]},"public_addr":"172.21.15.89:6814/50395527","cluster_addr":"172.21.15.89:6820/50395527","heartbeat_back_addr":"172.21.15.89:6831/50395527","heartbeat_front_addr":"172.21.15.89:6826/50395527","state":["exists","up"]},{"osd":6,"uuid":"81ce0c19-2554-4cae-acbe-66a05e9e9b9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6809","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6812","nonce":3470471752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6815","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6818","nonce":3470471752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6827","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6829","nonce":3470471752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6821","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6824","nonce":3470471752}]},"public_addr":"172.21.15.89:6812/3470471752","cluster_addr":"172.21.15.89:6818/3470471752","heartbeat_back_addr":"172.21.15.89:6829/3470471752","heartbeat_front_addr":"172.21.15.89:6824/3470471752","state":["exists","up"]},{"osd":7,"uuid":"85e4313b-d37d-442e-8042-153bf20a847d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6811","nonce":1796778630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6813","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6816","nonce":1796778630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6825","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6828","nonce":1796778630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6819","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6822","nonce":1796778630}]},"public_addr":"172.21.15.89:6811/1796778630","cluster_addr":"172.21.15.89:6816/1796778630","heartbeat_back_addr":"172.21.15.89:6828/1796778630","heartbeat_front_addr":"172.21.15.89:6822/1796778630","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.439088+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.529970+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.680512+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:21.044161+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.582111+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.536558+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.409522+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.590229+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-31T15:06:13.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:13.395+0000 7f333c800640 1 -- 172.21.15.16:0/1911524409 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f3318068380 msgr2=0x7f331806a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:13.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:13.395+0000 7f333c800640 1 --2- 172.21.15.16:0/1911524409 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f3318068380 0x7f331806a840 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f33280aa000 tx=0x7f33280b1040 comp rx=0 tx=0).stop 2024-03-31T15:06:13.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:13.395+0000 7f333c800640 1 -- 172.21.15.16:0/1911524409 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f333415bfd0 msgr2=0x7f333415c480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:13.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:13.395+0000 7f333c800640 1 --2- 172.21.15.16:0/1911524409 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f333415bfd0 0x7f333415c480 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f331c0039c0 tx=0x7f333000eb60 comp rx=0 tx=0).stop 2024-03-31T15:06:13.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:13.395+0000 7f333c800640 1 -- 172.21.15.16:0/1911524409 shutdown_connections 2024-03-31T15:06:13.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:13.395+0000 7f333c800640 1 --2- 172.21.15.16:0/1911524409 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f3318068380 0x7f331806a840 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:13.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:13.395+0000 7f333c800640 1 --2- 172.21.15.16:0/1911524409 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f333415bfd0 0x7f333415c480 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:13.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:13.395+0000 7f333c800640 1 --2- 172.21.15.16:0/1911524409 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f33340b4a20 0x7f3334142e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:13.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:13.395+0000 7f333c800640 1 --2- 172.21.15.16:0/1911524409 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f331c0039f0 0x7f33341fa360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:13.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:13.395+0000 7f333c800640 1 -- 172.21.15.16:0/1911524409 >> 172.21.15.16:0/1911524409 conn(0x7f3334067330 msgr2=0x7f3334145cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:06:13.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:13.395+0000 7f333c800640 1 -- 172.21.15.16:0/1911524409 shutdown_connections 2024-03-31T15:06:13.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:13.395+0000 7f333c800640 1 -- 172.21.15.16:0/1911524409 wait complete. 2024-03-31T15:06:13.409 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-31T15:06:13.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:13.555+0000 7f4e6ae00640 1 Processor -- start 2024-03-31T15:06:13.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:13.555+0000 7f4e6ae00640 1 -- start start 2024-03-31T15:06:13.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:13.555+0000 7f4e6ae00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f4e64070030 0x7f4e6406c730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:13.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:13.555+0000 7f4e6ae00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f4e64147dd0 0x7f4e640cc8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:13.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:13.555+0000 7f4e6ae00640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4e64147660 con 0x7f4e64147280 2024-03-31T15:06:13.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:13.555+0000 7f4e6ae00640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f4e641620f0 con 0x7f4e64070030 2024-03-31T15:06:13.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:13.555+0000 7f4e6ae00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f4e64162340 con 0x7f4e64147dd0 2024-03-31T15:06:13.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:13.559+0000 7f4e5b400640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f4e64070030 0x7f4e6406c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:13.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:13.559+0000 7f4e63e00640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f4e64147280 0x7f4e6406f920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:37200/0 (socket says 172.21.15.16:37200) 2024-03-31T15:06:13.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:13.559+0000 7f4e63e00640 1 -- 172.21.15.16:0/3967711925 learned_addr learned my addr 172.21.15.16:0/3967711925 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:06:13.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:13.559+0000 7f4e5b400640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f4e64070030 0x7f4e6406c730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35200/0 (socket says 172.21.15.16:35200) 2024-03-31T15:06:13.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:13.559+0000 7f4e68e00640 1 --2- 172.21.15.16:0/3967711925 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f4e64147dd0 0x7f4e640cc8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:13.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:13.559+0000 7f4e5b400640 1 -- 172.21.15.16:0/3967711925 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f4e64147dd0 msgr2=0x7f4e640cc8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:06:13.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:13.559+0000 7f4e5b400640 1 --2- 172.21.15.16:0/3967711925 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f4e64147dd0 0x7f4e640cc8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:13.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:13.559+0000 7f4e5b400640 1 -- 172.21.15.16:0/3967711925 >> v1:172.21.15.16:6789/0 conn(0x7f4e64147280 legacy=0x7f4e6406f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:13.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:13.559+0000 7f4e5b400640 1 -- 172.21.15.16:0/3967711925 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e64162620 con 0x7f4e64070030 2024-03-31T15:06:13.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:13.559+0000 7f4e5b400640 1 --2- 172.21.15.16:0/3967711925 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4e64070030 0x7f4e6406c730 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f4e4c009d90 tx=0x7f4e4c01c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=99ee736383fbaeb3 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:06:13.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:13.559+0000 7f4e63400640 1 -- 172.21.15.16:0/3967711925 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e4c0b2070 con 0x7f4e64070030 2024-03-31T15:06:13.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:13.559+0000 7f4e63400640 1 -- 172.21.15.16:0/3967711925 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4e4c0a5940 con 0x7f4e64070030 2024-03-31T15:06:13.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:13.559+0000 7f4e63400640 1 -- 172.21.15.16:0/3967711925 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e4c0a5d10 con 0x7f4e64070030 2024-03-31T15:06:13.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:13.563+0000 7f4e6ae00640 1 -- 172.21.15.16:0/3967711925 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4e64070030 msgr2=0x7f4e6406c730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:13.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:13.563+0000 7f4e6ae00640 1 --2- 172.21.15.16:0/3967711925 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4e64070030 0x7f4e6406c730 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f4e4c009d90 tx=0x7f4e4c01c970 comp rx=0 tx=0).stop 2024-03-31T15:06:13.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:13.563+0000 7f4e6ae00640 1 -- 172.21.15.16:0/3967711925 shutdown_connections 2024-03-31T15:06:13.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:13.563+0000 7f4e6ae00640 1 --2- 172.21.15.16:0/3967711925 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f4e64147dd0 0x7f4e640cc8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:13.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:13.563+0000 7f4e6ae00640 1 --2- 172.21.15.16:0/3967711925 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4e64070030 0x7f4e6406c730 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:13.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:13.563+0000 7f4e6ae00640 1 -- 172.21.15.16:0/3967711925 >> 172.21.15.16:0/3967711925 conn(0x7f4e64067330 msgr2=0x7f4e64067740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:06:13.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:13.563+0000 7f4e6ae00640 1 -- 172.21.15.16:0/3967711925 shutdown_connections 2024-03-31T15:06:13.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:13.563+0000 7f4e6ae00640 1 -- 172.21.15.16:0/3967711925 wait complete. 2024-03-31T15:06:13.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:13.563+0000 7f4e6ae00640 1 Processor -- start 2024-03-31T15:06:13.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:13.563+0000 7f4e6ae00640 1 -- start start 2024-03-31T15:06:13.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:13.563+0000 7f4e6ae00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f4e64070030 0x7f4e64145790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:13.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:13.563+0000 7f4e6ae00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4e64147280 0x7f4e64145cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:13.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:13.563+0000 7f4e6ae00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4e64147dd0 0x7f4e64146210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:13.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:13.563+0000 7f4e6ae00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f4e640cda50 con 0x7f4e64147dd0 2024-03-31T15:06:13.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:13.563+0000 7f4e6ae00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f4e640cd8d0 con 0x7f4e64147280 2024-03-31T15:06:13.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:13.563+0000 7f4e6ae00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f4e640cdbd0 con 0x7f4e64070030 2024-03-31T15:06:13.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:13.563+0000 7f4e68e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4e64147dd0 0x7f4e64146210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:13.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:13.563+0000 7f4e68e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4e64147dd0 0x7f4e64146210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35210/0 (socket says 172.21.15.16:35210) 2024-03-31T15:06:13.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:13.563+0000 7f4e68e00640 1 -- 172.21.15.16:0/2569957186 learned_addr learned my addr 172.21.15.16:0/2569957186 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:06:13.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:13.563+0000 7f4e63e00640 1 --2- 172.21.15.16:0/2569957186 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f4e64070030 0x7f4e64145790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:13.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:13.563+0000 7f4e5b400640 1 --2- 172.21.15.16:0/2569957186 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4e64147280 0x7f4e64145cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:13.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:13.563+0000 7f4e68e00640 1 -- 172.21.15.16:0/2569957186 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f4e64070030 msgr2=0x7f4e64145790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:13.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:13.563+0000 7f4e68e00640 1 --2- 172.21.15.16:0/2569957186 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f4e64070030 0x7f4e64145790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:13.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:13.563+0000 7f4e68e00640 1 -- 172.21.15.16:0/2569957186 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4e64147280 msgr2=0x7f4e64145cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:13.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:13.563+0000 7f4e68e00640 1 --2- 172.21.15.16:0/2569957186 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4e64147280 0x7f4e64145cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:13.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:13.563+0000 7f4e68e00640 1 -- 172.21.15.16:0/2569957186 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e64144fd0 con 0x7f4e64147dd0 2024-03-31T15:06:13.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:13.567+0000 7f4e68e00640 1 --2- 172.21.15.16:0/2569957186 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4e64147dd0 0x7f4e64146210 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f4e5c00e930 tx=0x7f4e5c00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:06:13.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:13.567+0000 7f4e60c00640 1 -- 172.21.15.16:0/2569957186 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e5c017070 con 0x7f4e64147dd0 2024-03-31T15:06:13.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:13.567+0000 7f4e60c00640 1 -- 172.21.15.16:0/2569957186 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4e5c00c9a0 con 0x7f4e64147dd0 2024-03-31T15:06:13.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:13.567+0000 7f4e60c00640 1 -- 172.21.15.16:0/2569957186 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e5c00cd70 con 0x7f4e64147dd0 2024-03-31T15:06:13.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:13.567+0000 7f4e6ae00640 1 -- 172.21.15.16:0/2569957186 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4e64140420 con 0x7f4e64147dd0 2024-03-31T15:06:13.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:13.567+0000 7f4e6ae00640 1 -- 172.21.15.16:0/2569957186 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4e64145270 con 0x7f4e64147dd0 2024-03-31T15:06:13.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:13.567+0000 7f4e6ae00640 1 -- 172.21.15.16:0/2569957186 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4e64069950 con 0x7f4e64147dd0 2024-03-31T15:06:13.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:13.567+0000 7f4e60c00640 1 -- 172.21.15.16:0/2569957186 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f4e5c01a070 con 0x7f4e64147dd0 2024-03-31T15:06:13.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:13.571+0000 7f4e60c00640 1 --2- 172.21.15.16:0/2569957186 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f4e400683b0 0x7f4e4006a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:13.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:13.571+0000 7f4e60c00640 1 -- 172.21.15.16:0/2569957186 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5709+0+0 (secure 0 0 0) 0x7f4e5c086a60 con 0x7f4e64147dd0 2024-03-31T15:06:13.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:13.571+0000 7f4e63e00640 1 --2- 172.21.15.16:0/2569957186 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f4e400683b0 0x7f4e4006a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:13.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:13.571+0000 7f4e63e00640 1 --2- 172.21.15.16:0/2569957186 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f4e400683b0 0x7f4e4006a870 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f4e54000f70 tx=0x7f4e54009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:06:13.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:13.571+0000 7f4e60c00640 1 -- 172.21.15.16:0/2569957186 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f4e5c057040 con 0x7f4e64147dd0 2024-03-31T15:06:13.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:13.771+0000 7f4e6ae00640 1 -- 172.21.15.16:0/2569957186 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f4e640cf3e0 con 0x7f4e64147dd0 2024-03-31T15:06:13.768 INFO:tasks.ceph.mon.a.smithi016.stderr:2024-03-31T15:06:13.771+0000 7f7dcbc00640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-31T15:06:13.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:13.775+0000 7f4e60c00640 1 -- 172.21.15.16:0/2569957186 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v26) ==== 161+0+0 (secure 0 0 0) 0x7f4e5c056940 con 0x7f4e64147dd0 2024-03-31T15:06:13.771 INFO:teuthology.orchestra.run.smithi016.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-31T15:06:13.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:13.779+0000 7f4e6ae00640 1 -- 172.21.15.16:0/2569957186 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f4e400683b0 msgr2=0x7f4e4006a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:13.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:13.779+0000 7f4e6ae00640 1 --2- 172.21.15.16:0/2569957186 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f4e400683b0 0x7f4e4006a870 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f4e54000f70 tx=0x7f4e54009040 comp rx=0 tx=0).stop 2024-03-31T15:06:13.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:13.779+0000 7f4e6ae00640 1 -- 172.21.15.16:0/2569957186 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4e64147dd0 msgr2=0x7f4e64146210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:13.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:13.779+0000 7f4e6ae00640 1 --2- 172.21.15.16:0/2569957186 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4e64147dd0 0x7f4e64146210 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f4e5c00e930 tx=0x7f4e5c00edf0 comp rx=0 tx=0).stop 2024-03-31T15:06:13.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:13.779+0000 7f4e6ae00640 1 -- 172.21.15.16:0/2569957186 shutdown_connections 2024-03-31T15:06:13.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:13.779+0000 7f4e6ae00640 1 --2- 172.21.15.16:0/2569957186 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f4e400683b0 0x7f4e4006a870 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:13.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:13.779+0000 7f4e6ae00640 1 --2- 172.21.15.16:0/2569957186 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4e64147dd0 0x7f4e64146210 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:13.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:13.779+0000 7f4e6ae00640 1 --2- 172.21.15.16:0/2569957186 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4e64147280 0x7f4e64145cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:13.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:13.779+0000 7f4e6ae00640 1 --2- 172.21.15.16:0/2569957186 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f4e64070030 0x7f4e64145790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:13.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:13.779+0000 7f4e6ae00640 1 -- 172.21.15.16:0/2569957186 >> 172.21.15.16:0/2569957186 conn(0x7f4e64067330 msgr2=0x7f4e6406d450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:06:13.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:13.779+0000 7f4e6ae00640 1 -- 172.21.15.16:0/2569957186 shutdown_connections 2024-03-31T15:06:13.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:13.779+0000 7f4e6ae00640 1 -- 172.21.15.16:0/2569957186 wait complete. 2024-03-31T15:06:13.788 INFO:tasks.ceph.mds.a.smithi016.stderr: -4823> 2024-03-31T15:05:33.043+0000 7fa13f800640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-31T15:06:13.795 INFO:tasks.ceph.mds.a.smithi016.stderr: -3222> 2024-03-31T15:05:35.951+0000 7fa13a800640 -1 mds.0.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000001 [2,head] /parent_a/alpha auth v2 ap=1 DIRTYPARENT s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={5859=0-4194304@1} caps={5859=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@2},l=5859 | request=0 caps=1 dirtyparent=1 dirty=0 authpin=1 scrubqueue=0 0x5639fddc0c00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(-1)0x0:[]//[]","memoryvalue":"(7)0x10000000001:[<0x10000000000/alpha v2>,<0x1/parent_a v4>]//[]","error_str":"failed to decode on-disk backtrace (40 bytes)!"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-31T15:06:13.802 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-31T15:06:13.846 INFO:tasks.ceph.mds.a.smithi016.stdout:starting mds.a at 2024-03-31T15:06:13.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:13.955+0000 7f7ff1600640 1 Processor -- start 2024-03-31T15:06:13.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:13.955+0000 7f7ff1600640 1 -- start start 2024-03-31T15:06:13.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:13.955+0000 7f7ff1600640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f7fec0d0a60 0x7f7fec0cea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:13.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:13.955+0000 7f7ff1600640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7fec15ca70 con 0x7f7fec0cef80 2024-03-31T15:06:13.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:13.955+0000 7f7ff1600640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7fec15cd50 con 0x7f7fec06f920 2024-03-31T15:06:13.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:13.955+0000 7f7ff1600640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f7fec13f670 con 0x7f7fec0d0a60 2024-03-31T15:06:13.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:13.955+0000 7f7feb400640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f7fec0cef80 0x7f7fec1419c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:37220/0 (socket says 172.21.15.16:37220) 2024-03-31T15:06:13.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:13.955+0000 7f7feb400640 1 -- 172.21.15.16:0/1557264793 learned_addr learned my addr 172.21.15.16:0/1557264793 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:06:13.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:13.955+0000 7f7fea000640 1 --2- 172.21.15.16:0/1557264793 >> v2:172.21.15.89:3300/0 conn(0x7f7fec0d0a60 0x7f7fec0cea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:13.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:13.955+0000 7f7fe9600640 1 -- 172.21.15.16:0/1557264793 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1957205915 0 0) 0x7f7fec15ca70 con 0x7f7fec0cef80 2024-03-31T15:06:13.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:13.955+0000 7f7fe9600640 1 -- 172.21.15.16:0/1557264793 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7fd0003530 con 0x7f7fec0cef80 2024-03-31T15:06:13.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:13.955+0000 7f7fea000640 1 -- 172.21.15.16:0/1557264793 >> v1:172.21.15.89:6789/0 conn(0x7f7fec06f920 legacy=0x7f7fec06fd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:13.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:13.955+0000 7f7fea000640 1 -- 172.21.15.16:0/1557264793 >> v1:172.21.15.16:6789/0 conn(0x7f7fec0cef80 legacy=0x7f7fec1419c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:13.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:13.955+0000 7f7fea000640 1 -- 172.21.15.16:0/1557264793 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7fec15d030 con 0x7f7fec0d0a60 2024-03-31T15:06:13.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:13.959+0000 7f7fea000640 1 --2- 172.21.15.16:0/1557264793 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7fec0d0a60 0x7f7fec0cea40 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f7fd80097c0 tx=0x7f7fd801c4a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c1c1728c7f8c5959 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:06:13.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:13.959+0000 7f7fe9600640 1 -- 172.21.15.16:0/1557264793 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fd80b0070 con 0x7f7fec0d0a60 2024-03-31T15:06:13.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:13.959+0000 7f7fe9600640 1 -- 172.21.15.16:0/1557264793 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7fd80ab600 con 0x7f7fec0d0a60 2024-03-31T15:06:13.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:13.959+0000 7f7fe9600640 1 -- 172.21.15.16:0/1557264793 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fd80ab9d0 con 0x7f7fec0d0a60 2024-03-31T15:06:13.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:13.959+0000 7f7ff1600640 1 -- 172.21.15.16:0/1557264793 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7fec0d0a60 msgr2=0x7f7fec0cea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:13.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:13.959+0000 7f7ff1600640 1 --2- 172.21.15.16:0/1557264793 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7fec0d0a60 0x7f7fec0cea40 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f7fd80097c0 tx=0x7f7fd801c4a0 comp rx=0 tx=0).stop 2024-03-31T15:06:13.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:13.959+0000 7f7ff1600640 1 -- 172.21.15.16:0/1557264793 shutdown_connections 2024-03-31T15:06:13.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:13.959+0000 7f7ff1600640 1 --2- 172.21.15.16:0/1557264793 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7fec0d0a60 0x7f7fec0cea40 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:13.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:13.959+0000 7f7ff1600640 1 -- 172.21.15.16:0/1557264793 >> 172.21.15.16:0/1557264793 conn(0x7f7fec067930 msgr2=0x7f7fec0b3ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:06:13.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:13.959+0000 7f7ff1600640 1 -- 172.21.15.16:0/1557264793 shutdown_connections 2024-03-31T15:06:13.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:13.959+0000 7f7ff1600640 1 -- 172.21.15.16:0/1557264793 wait complete. 2024-03-31T15:06:13.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:13.963+0000 7f7ff1600640 1 Processor -- start 2024-03-31T15:06:13.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:13.963+0000 7f7ff1600640 1 -- start start 2024-03-31T15:06:13.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:13.963+0000 7f7ff1600640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f7fec06f920 0x7f7fec0b2480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:13.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:13.963+0000 7f7ff1600640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7fec0cef80 0x7f7fec13f830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:13.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:13.963+0000 7f7ff1600640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7fec0d0a60 0x7f7fec13ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:13.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:13.963+0000 7f7ff1600640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f7fec16a240 con 0x7f7fec0d0a60 2024-03-31T15:06:13.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:13.963+0000 7f7ff1600640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f7fec16a0c0 con 0x7f7fec0cef80 2024-03-31T15:06:13.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:13.963+0000 7f7ff1600640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f7fec16a3c0 con 0x7f7fec06f920 2024-03-31T15:06:13.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:13.963+0000 7f7feb400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7fec0d0a60 0x7f7fec13ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:13.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:13.963+0000 7f7feaa00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f7fec06f920 0x7f7fec0b2480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:13.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:13.963+0000 7f7fea000640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7fec0cef80 0x7f7fec13f830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:13.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:13.963+0000 7f7feaa00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f7fec06f920 0x7f7fec0b2480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3301/0 says I am v2:172.21.15.16:52418/0 (socket says 172.21.15.16:52418) 2024-03-31T15:06:13.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:13.963+0000 7f7feb400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7fec0d0a60 0x7f7fec13ecd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35230/0 (socket says 172.21.15.16:35230) 2024-03-31T15:06:13.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:13.963+0000 7f7feaa00640 1 -- 172.21.15.16:0/314683849 learned_addr learned my addr 172.21.15.16:0/314683849 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:06:13.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:13.963+0000 7f7feaa00640 1 -- 172.21.15.16:0/314683849 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7fec0cef80 msgr2=0x7f7fec13f830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:13.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:13.963+0000 7f7feaa00640 1 --2- 172.21.15.16:0/314683849 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7fec0cef80 0x7f7fec13f830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:13.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:13.963+0000 7f7feaa00640 1 -- 172.21.15.16:0/314683849 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7fec0d0a60 msgr2=0x7f7fec13ecd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:13.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:13.963+0000 7f7feaa00640 1 --2- 172.21.15.16:0/314683849 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7fec0d0a60 0x7f7fec13ecd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:13.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:13.963+0000 7f7feaa00640 1 -- 172.21.15.16:0/314683849 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7fec13fd70 con 0x7f7fec06f920 2024-03-31T15:06:13.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:13.963+0000 7f7feb400640 1 --2- 172.21.15.16:0/314683849 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7fec0d0a60 0x7f7fec13ecd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:06:13.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:13.963+0000 7f7feaa00640 1 --2- 172.21.15.16:0/314683849 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f7fec06f920 0x7f7fec0b2480 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f7fdc00c530 tx=0x7f7fdc00c9f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:06:13.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:13.963+0000 7f7fe2a00640 1 -- 172.21.15.16:0/314683849 <== mon.2 v2:172.21.15.89:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fdc016020 con 0x7f7fec06f920 2024-03-31T15:06:13.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:13.963+0000 7f7fe2a00640 1 -- 172.21.15.16:0/314683849 <== mon.2 v2:172.21.15.89:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7fdc005220 con 0x7f7fec06f920 2024-03-31T15:06:13.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:13.963+0000 7f7ff1600640 1 -- 172.21.15.16:0/314683849 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7fec13f210 con 0x7f7fec06f920 2024-03-31T15:06:13.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:13.963+0000 7f7fe2a00640 1 -- 172.21.15.16:0/314683849 <== mon.2 v2:172.21.15.89:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fdc007030 con 0x7f7fec06f920 2024-03-31T15:06:13.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:13.963+0000 7f7ff1600640 1 -- 172.21.15.16:0/314683849 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f7fec1405b0 con 0x7f7fec06f920 2024-03-31T15:06:13.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:13.967+0000 7f7ff1600640 1 -- 172.21.15.16:0/314683849 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7fec069950 con 0x7f7fec06f920 2024-03-31T15:06:13.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:13.967+0000 7f7fe2a00640 1 -- 172.21.15.16:0/314683849 <== mon.2 v2:172.21.15.89:3301/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f7fdc0071d0 con 0x7f7fec06f920 2024-03-31T15:06:13.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:13.967+0000 7f7fe2a00640 1 --2- 172.21.15.16:0/314683849 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f7fcc0683b0 0x7f7fcc06a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:13.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:13.967+0000 7f7fe2a00640 1 -- 172.21.15.16:0/314683849 <== mon.2 v2:172.21.15.89:3301/0 5 ==== osd_map(46..46 src has 1..46) ==== 5795+0+0 (secure 0 0 0) 0x7f7fdc086310 con 0x7f7fec06f920 2024-03-31T15:06:13.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:13.967+0000 7f7fea000640 1 --2- 172.21.15.16:0/314683849 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f7fcc0683b0 0x7f7fcc06a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:13.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:13.967+0000 7f7fea000640 1 --2- 172.21.15.16:0/314683849 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f7fcc0683b0 0x7f7fcc06a870 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f7fd80ad000 tx=0x7f7fd8008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:06:13.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:13.971+0000 7f7fe2a00640 1 -- 172.21.15.16:0/314683849 <== mon.2 v2:172.21.15.89:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f7fdc020020 con 0x7f7fec06f920 2024-03-31T15:06:14.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:14.175+0000 7f7ff1600640 1 -- 172.21.15.16:0/314683849 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7f7fec0b50e0 con 0x7f7fec06f920 2024-03-31T15:06:14.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:14.779+0000 7f7fe2a00640 1 -- 172.21.15.16:0/314683849 <== mon.2 v2:172.21.15.89:3301/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v27) ==== 104+0+0 (secure 0 0 0) 0x7f7fec13fd70 con 0x7f7fec06f920 2024-03-31T15:06:14.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:14.783+0000 7f7ff1600640 1 -- 172.21.15.16:0/314683849 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f7fcc0683b0 msgr2=0x7f7fcc06a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:14.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:14.783+0000 7f7ff1600640 1 --2- 172.21.15.16:0/314683849 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f7fcc0683b0 0x7f7fcc06a870 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f7fd80ad000 tx=0x7f7fd8008040 comp rx=0 tx=0).stop 2024-03-31T15:06:14.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:14.783+0000 7f7ff1600640 1 -- 172.21.15.16:0/314683849 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f7fec06f920 msgr2=0x7f7fec0b2480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:14.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:14.783+0000 7f7ff1600640 1 --2- 172.21.15.16:0/314683849 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f7fec06f920 0x7f7fec0b2480 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f7fdc00c530 tx=0x7f7fdc00c9f0 comp rx=0 tx=0).stop 2024-03-31T15:06:14.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:14.783+0000 7f7ff1600640 1 -- 172.21.15.16:0/314683849 shutdown_connections 2024-03-31T15:06:14.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:14.783+0000 7f7ff1600640 1 --2- 172.21.15.16:0/314683849 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f7fcc0683b0 0x7f7fcc06a870 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:14.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:14.783+0000 7f7ff1600640 1 --2- 172.21.15.16:0/314683849 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7fec0d0a60 0x7f7fec13ecd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:14.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:14.783+0000 7f7ff1600640 1 --2- 172.21.15.16:0/314683849 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7fec0cef80 0x7f7fec13f830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:14.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:14.783+0000 7f7ff1600640 1 --2- 172.21.15.16:0/314683849 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f7fec06f920 0x7f7fec0b2480 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:14.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:14.783+0000 7f7ff1600640 1 -- 172.21.15.16:0/314683849 >> 172.21.15.16:0/314683849 conn(0x7f7fec067930 msgr2=0x7f7fec06c810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:06:14.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:14.783+0000 7f7ff1600640 1 -- 172.21.15.16:0/314683849 shutdown_connections 2024-03-31T15:06:14.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:14.783+0000 7f7ff1600640 1 -- 172.21.15.16:0/314683849 wait complete. 2024-03-31T15:06:14.802 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-31T15:06:14.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:14.999+0000 7ff555600640 1 Processor -- start 2024-03-31T15:06:14.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:14.999+0000 7ff555600640 1 -- start start 2024-03-31T15:06:14.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:14.999+0000 7ff555600640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff550143f40 0x7ff5500730c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:14.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:14.999+0000 7ff555600640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff550144320 con 0x7ff5500b45d0 2024-03-31T15:06:14.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:14.999+0000 7ff555600640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff55015ef80 con 0x7ff5500b22f0 2024-03-31T15:06:14.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:14.999+0000 7ff555600640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7ff5500b4a30 con 0x7ff550143f40 2024-03-31T15:06:14.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:15.003+0000 7ff54ea00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff550143f40 0x7ff5500730c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:14.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:15.003+0000 7ff54ea00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff550143f40 0x7ff5500730c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35246/0 (socket says 172.21.15.16:35246) 2024-03-31T15:06:14.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:15.003+0000 7ff54ea00640 1 -- 172.21.15.16:0/1830224329 learned_addr learned my addr 172.21.15.16:0/1830224329 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:06:14.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:15.003+0000 7ff547e00640 1 --1- 172.21.15.16:0/1830224329 >> v1:172.21.15.16:6789/0 conn(0x7ff5500b45d0 0x7ff5500b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:37234/0 (socket says 172.21.15.16:37234) 2024-03-31T15:06:14.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:15.003+0000 7ff54ea00640 1 -- 172.21.15.16:0/1830224329 >> v1:172.21.15.89:6789/0 conn(0x7ff5500b22f0 legacy=0x7ff5500b2f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:14.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:15.003+0000 7ff54ea00640 1 -- 172.21.15.16:0/1830224329 >> v1:172.21.15.16:6789/0 conn(0x7ff5500b45d0 legacy=0x7ff5500b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:14.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:15.003+0000 7ff54ea00640 1 -- 172.21.15.16:0/1830224329 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff55015f260 con 0x7ff550143f40 2024-03-31T15:06:14.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:15.003+0000 7ff54ea00640 1 --2- 172.21.15.16:0/1830224329 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff550143f40 0x7ff5500730c0 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7ff540009870 tx=0x7ff54001c560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c66592c2d3794fb7 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:06:14.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:15.003+0000 7ff54e000640 1 -- 172.21.15.16:0/1830224329 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5400b2070 con 0x7ff550143f40 2024-03-31T15:06:15.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:15.003+0000 7ff54e000640 1 -- 172.21.15.16:0/1830224329 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff5400a9070 con 0x7ff550143f40 2024-03-31T15:06:15.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:15.003+0000 7ff54e000640 1 -- 172.21.15.16:0/1830224329 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5400ad880 con 0x7ff550143f40 2024-03-31T15:06:15.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:15.007+0000 7ff555600640 1 -- 172.21.15.16:0/1830224329 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff550143f40 msgr2=0x7ff5500730c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:15.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:15.007+0000 7ff555600640 1 --2- 172.21.15.16:0/1830224329 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff550143f40 0x7ff5500730c0 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7ff540009870 tx=0x7ff54001c560 comp rx=0 tx=0).stop 2024-03-31T15:06:15.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:15.007+0000 7ff555600640 1 -- 172.21.15.16:0/1830224329 shutdown_connections 2024-03-31T15:06:15.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:15.007+0000 7ff555600640 1 --2- 172.21.15.16:0/1830224329 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff550143f40 0x7ff5500730c0 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:15.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:15.007+0000 7ff555600640 1 -- 172.21.15.16:0/1830224329 >> 172.21.15.16:0/1830224329 conn(0x7ff55006b620 msgr2=0x7ff55006ba30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:06:15.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:15.007+0000 7ff555600640 1 -- 172.21.15.16:0/1830224329 shutdown_connections 2024-03-31T15:06:15.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:15.007+0000 7ff555600640 1 -- 172.21.15.16:0/1830224329 wait complete. 2024-03-31T15:06:15.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:15.007+0000 7ff555600640 1 Processor -- start 2024-03-31T15:06:15.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:15.007+0000 7ff555600640 1 -- start start 2024-03-31T15:06:15.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:15.007+0000 7ff555600640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff5500b22f0 0x7ff550159490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:15.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:15.007+0000 7ff555600640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff5500b45d0 0x7ff550140380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:15.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:15.007+0000 7ff555600640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff550143f40 0x7ff55013bed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:15.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:15.007+0000 7ff555600640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7ff5500d0020 con 0x7ff5500b22f0 2024-03-31T15:06:15.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:15.007+0000 7ff555600640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7ff5500cfea0 con 0x7ff5500b45d0 2024-03-31T15:06:15.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:15.007+0000 7ff555600640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7ff5500d01a0 con 0x7ff550143f40 2024-03-31T15:06:15.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:15.007+0000 7ff54ea00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff5500b22f0 0x7ff550159490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:15.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:15.007+0000 7ff54ea00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff5500b22f0 0x7ff550159490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35260/0 (socket says 172.21.15.16:35260) 2024-03-31T15:06:15.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:15.007+0000 7ff54ea00640 1 -- 172.21.15.16:0/3291320154 learned_addr learned my addr 172.21.15.16:0/3291320154 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:06:15.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:15.007+0000 7ff547e00640 1 --2- 172.21.15.16:0/3291320154 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff5500b45d0 0x7ff550140380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:15.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:15.007+0000 7ff54f400640 1 --2- 172.21.15.16:0/3291320154 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff550143f40 0x7ff55013bed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:15.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:15.007+0000 7ff54ea00640 1 -- 172.21.15.16:0/3291320154 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff550143f40 msgr2=0x7ff55013bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:15.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:15.007+0000 7ff54ea00640 1 --2- 172.21.15.16:0/3291320154 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff550143f40 0x7ff55013bed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:15.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:15.007+0000 7ff54ea00640 1 -- 172.21.15.16:0/3291320154 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff5500b45d0 msgr2=0x7ff550140380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:15.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:15.007+0000 7ff54ea00640 1 --2- 172.21.15.16:0/3291320154 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff5500b45d0 0x7ff550140380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:15.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:15.007+0000 7ff54ea00640 1 -- 172.21.15.16:0/3291320154 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff550143550 con 0x7ff5500b22f0 2024-03-31T15:06:15.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:15.007+0000 7ff54f400640 1 --2- 172.21.15.16:0/3291320154 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff550143f40 0x7ff55013bed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:06:15.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:15.007+0000 7ff54ea00640 1 --2- 172.21.15.16:0/3291320154 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff5500b22f0 0x7ff550159490 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7ff5400b0040 tx=0x7ff540019300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:06:15.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:15.011+0000 7ff546a00640 1 -- 172.21.15.16:0/3291320154 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5400b2040 con 0x7ff5500b22f0 2024-03-31T15:06:15.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:15.011+0000 7ff546a00640 1 -- 172.21.15.16:0/3291320154 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff5400bb070 con 0x7ff5500b22f0 2024-03-31T15:06:15.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:15.011+0000 7ff546a00640 1 -- 172.21.15.16:0/3291320154 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5400b6680 con 0x7ff5500b22f0 2024-03-31T15:06:15.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:15.011+0000 7ff555600640 1 -- 172.21.15.16:0/3291320154 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff5501424b0 con 0x7ff5500b22f0 2024-03-31T15:06:15.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:15.011+0000 7ff555600640 1 -- 172.21.15.16:0/3291320154 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff550142a70 con 0x7ff5500b22f0 2024-03-31T15:06:15.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:15.011+0000 7ff555600640 1 -- 172.21.15.16:0/3291320154 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff550069950 con 0x7ff5500b22f0 2024-03-31T15:06:15.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:15.011+0000 7ff546a00640 1 -- 172.21.15.16:0/3291320154 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7ff5400ab070 con 0x7ff5500b22f0 2024-03-31T15:06:15.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:15.011+0000 7ff546a00640 1 --2- 172.21.15.16:0/3291320154 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7ff52c068380 0x7ff52c06a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:15.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:15.011+0000 7ff546a00640 1 -- 172.21.15.16:0/3291320154 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 5795+0+0 (secure 0 0 0) 0x7ff540120320 con 0x7ff5500b22f0 2024-03-31T15:06:15.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:15.015+0000 7ff547e00640 1 --2- 172.21.15.16:0/3291320154 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7ff52c068380 0x7ff52c06a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:15.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:15.015+0000 7ff547e00640 1 --2- 172.21.15.16:0/3291320154 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7ff52c068380 0x7ff52c06a840 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7ff550073070 tx=0x7ff538007e50 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:06:15.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:15.015+0000 7ff546a00640 1 -- 172.21.15.16:0/3291320154 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7ff540016150 con 0x7ff5500b22f0 2024-03-31T15:06:15.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:15.199+0000 7ff555600640 1 -- 172.21.15.16:0/3291320154 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7ff5500b4060 con 0x7ff5500b22f0 2024-03-31T15:06:16.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:16.199+0000 7ff546a00640 1 -- 172.21.15.16:0/3291320154 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v47) ==== 181+0+0 (secure 0 0 0) 0x7ff5400f0910 con 0x7ff5500b22f0 2024-03-31T15:06:16.196 INFO:teuthology.orchestra.run.smithi016.stderr:pool 'cephfs_metadata' removed 2024-03-31T15:06:16.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:16.203+0000 7ff555600640 1 -- 172.21.15.16:0/3291320154 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7ff52c068380 msgr2=0x7ff52c06a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:16.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:16.203+0000 7ff555600640 1 --2- 172.21.15.16:0/3291320154 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7ff52c068380 0x7ff52c06a840 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7ff550073070 tx=0x7ff538007e50 comp rx=0 tx=0).stop 2024-03-31T15:06:16.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:16.203+0000 7ff555600640 1 -- 172.21.15.16:0/3291320154 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff5500b22f0 msgr2=0x7ff550159490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:16.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:16.203+0000 7ff555600640 1 --2- 172.21.15.16:0/3291320154 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff5500b22f0 0x7ff550159490 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7ff5400b0040 tx=0x7ff540019300 comp rx=0 tx=0).stop 2024-03-31T15:06:16.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:16.203+0000 7ff555600640 1 -- 172.21.15.16:0/3291320154 shutdown_connections 2024-03-31T15:06:16.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:16.203+0000 7ff555600640 1 --2- 172.21.15.16:0/3291320154 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7ff52c068380 0x7ff52c06a840 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:16.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:16.203+0000 7ff555600640 1 --2- 172.21.15.16:0/3291320154 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff550143f40 0x7ff55013bed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:16.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:16.203+0000 7ff555600640 1 --2- 172.21.15.16:0/3291320154 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff5500b45d0 0x7ff550140380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:16.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:16.203+0000 7ff555600640 1 --2- 172.21.15.16:0/3291320154 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff5500b22f0 0x7ff550159490 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:16.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:16.203+0000 7ff555600640 1 -- 172.21.15.16:0/3291320154 >> 172.21.15.16:0/3291320154 conn(0x7ff55006b620 msgr2=0x7ff5500ce240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:06:16.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:16.203+0000 7ff555600640 1 -- 172.21.15.16:0/3291320154 shutdown_connections 2024-03-31T15:06:16.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:16.203+0000 7ff555600640 1 -- 172.21.15.16:0/3291320154 wait complete. 2024-03-31T15:06:16.231 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-31T15:06:16.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:16.383+0000 7f1313600640 1 Processor -- start 2024-03-31T15:06:16.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:16.383+0000 7f1313600640 1 -- start start 2024-03-31T15:06:16.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:16.387+0000 7f1313600640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f130c15e6c0 0x7f130c070b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:16.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:16.387+0000 7f1313600640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f130c0b4140 0x7f130c073220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:16.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:16.387+0000 7f1313600640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f130c0b0160 0x7f130c0b05f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:16.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:16.387+0000 7f1313600640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f130c0b4b50 con 0x7f130c15e6c0 2024-03-31T15:06:16.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:16.387+0000 7f1313600640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f130c0b4e30 con 0x7f130c0b0160 2024-03-31T15:06:16.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:16.387+0000 7f1313600640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f130c072740 con 0x7f130c0b4140 2024-03-31T15:06:16.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:16.387+0000 7f1310c00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f130c15e6c0 0x7f130c070b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:16.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:16.387+0000 7f1310c00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f130c15e6c0 0x7f130c070b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35268/0 (socket says 172.21.15.16:35268) 2024-03-31T15:06:16.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:16.387+0000 7f1310c00640 1 -- 172.21.15.16:0/3044572452 learned_addr learned my addr 172.21.15.16:0/3044572452 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:06:16.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:16.387+0000 7f1303e00640 1 --2- 172.21.15.16:0/3044572452 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f130c0b4140 0x7f130c073220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:16.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:16.387+0000 7f1311600640 1 --2- 172.21.15.16:0/3044572452 >> v2:172.21.15.89:3300/0 conn(0x7f130c0b0160 0x7f130c0b05f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:16.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:16.387+0000 7f1310c00640 1 -- 172.21.15.16:0/3044572452 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f130c0b4140 msgr2=0x7f130c073220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:06:16.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:16.387+0000 7f1310c00640 1 --2- 172.21.15.16:0/3044572452 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f130c0b4140 0x7f130c073220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:16.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:16.387+0000 7f1310c00640 1 -- 172.21.15.16:0/3044572452 >> v2:172.21.15.89:3300/0 conn(0x7f130c0b0160 msgr2=0x7f130c0b05f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:06:16.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:16.387+0000 7f1310c00640 1 --2- 172.21.15.16:0/3044572452 >> v2:172.21.15.89:3300/0 conn(0x7f130c0b0160 0x7f130c0b05f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:16.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:16.387+0000 7f1310c00640 1 -- 172.21.15.16:0/3044572452 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f130c072a20 con 0x7f130c15e6c0 2024-03-31T15:06:16.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:16.391+0000 7f1310c00640 1 --2- 172.21.15.16:0/3044572452 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f130c15e6c0 0x7f130c070b00 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f130c072530 tx=0x7f12fc01c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e3777773e2b201a server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:06:16.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:16.391+0000 7f130be00640 1 -- 172.21.15.16:0/3044572452 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12fc0b2070 con 0x7f130c15e6c0 2024-03-31T15:06:16.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:16.391+0000 7f130be00640 1 -- 172.21.15.16:0/3044572452 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f12fc0a9070 con 0x7f130c15e6c0 2024-03-31T15:06:16.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:16.391+0000 7f130be00640 1 -- 172.21.15.16:0/3044572452 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12fc0ad880 con 0x7f130c15e6c0 2024-03-31T15:06:16.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:16.391+0000 7f1313600640 1 -- 172.21.15.16:0/3044572452 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f130c15e6c0 msgr2=0x7f130c070b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:16.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:16.391+0000 7f1313600640 1 --2- 172.21.15.16:0/3044572452 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f130c15e6c0 0x7f130c070b00 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f130c072530 tx=0x7f12fc01c490 comp rx=0 tx=0).stop 2024-03-31T15:06:16.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:16.391+0000 7f1313600640 1 -- 172.21.15.16:0/3044572452 shutdown_connections 2024-03-31T15:06:16.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:16.391+0000 7f1313600640 1 --2- 172.21.15.16:0/3044572452 >> v2:172.21.15.89:3300/0 conn(0x7f130c0b0160 0x7f130c0b05f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:16.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:16.391+0000 7f1313600640 1 --2- 172.21.15.16:0/3044572452 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f130c0b4140 0x7f130c073220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:16.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:16.391+0000 7f1313600640 1 --2- 172.21.15.16:0/3044572452 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f130c15e6c0 0x7f130c070b00 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:16.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:16.391+0000 7f1313600640 1 -- 172.21.15.16:0/3044572452 >> 172.21.15.16:0/3044572452 conn(0x7f130c0b2dd0 msgr2=0x7f130c0b31e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:06:16.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:16.391+0000 7f1313600640 1 -- 172.21.15.16:0/3044572452 shutdown_connections 2024-03-31T15:06:16.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:16.391+0000 7f1313600640 1 -- 172.21.15.16:0/3044572452 wait complete. 2024-03-31T15:06:16.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:16.391+0000 7f1313600640 1 Processor -- start 2024-03-31T15:06:16.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:16.391+0000 7f1313600640 1 -- start start 2024-03-31T15:06:16.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:16.395+0000 7f1313600640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f130c0b0160 0x7f130c14f4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:16.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:16.395+0000 7f1313600640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f130c0b4140 0x7f130c160b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:16.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:16.395+0000 7f1313600640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f130c15e6c0 0x7f130c152570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:16.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:16.395+0000 7f1313600640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f130c06fa40 con 0x7f130c0b0160 2024-03-31T15:06:16.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:16.395+0000 7f1313600640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f130c0ae2f0 con 0x7f130c0b4140 2024-03-31T15:06:16.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:16.395+0000 7f1313600640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f130c06fbc0 con 0x7f130c15e6c0 2024-03-31T15:06:16.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:16.395+0000 7f1310c00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f130c0b0160 0x7f130c14f4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:16.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:16.395+0000 7f1310c00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f130c0b0160 0x7f130c14f4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35270/0 (socket says 172.21.15.16:35270) 2024-03-31T15:06:16.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:16.395+0000 7f1310c00640 1 -- 172.21.15.16:0/2733244973 learned_addr learned my addr 172.21.15.16:0/2733244973 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:06:16.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:16.395+0000 7f1303e00640 1 --2- 172.21.15.16:0/2733244973 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f130c0b4140 0x7f130c160b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:16.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:16.395+0000 7f1311600640 1 --2- 172.21.15.16:0/2733244973 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f130c15e6c0 0x7f130c152570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:16.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:16.395+0000 7f1310c00640 1 -- 172.21.15.16:0/2733244973 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f130c15e6c0 msgr2=0x7f130c152570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:16.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:16.395+0000 7f1310c00640 1 --2- 172.21.15.16:0/2733244973 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f130c15e6c0 0x7f130c152570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:16.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:16.395+0000 7f1310c00640 1 -- 172.21.15.16:0/2733244973 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f130c0b4140 msgr2=0x7f130c160b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:16.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:16.395+0000 7f1310c00640 1 --2- 172.21.15.16:0/2733244973 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f130c0b4140 0x7f130c160b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:16.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:16.395+0000 7f1310c00640 1 -- 172.21.15.16:0/2733244973 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f130c161050 con 0x7f130c0b0160 2024-03-31T15:06:16.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:16.395+0000 7f1310c00640 1 --2- 172.21.15.16:0/2733244973 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f130c0b0160 0x7f130c14f4c0 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f12fc01cd30 tx=0x7f12fc01acd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:06:16.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:16.395+0000 7f1309600640 1 -- 172.21.15.16:0/2733244973 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12fc0bc020 con 0x7f130c0b0160 2024-03-31T15:06:16.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:16.395+0000 7f1309600640 1 -- 172.21.15.16:0/2733244973 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f12fc0b8640 con 0x7f130c0b0160 2024-03-31T15:06:16.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:16.395+0000 7f1309600640 1 -- 172.21.15.16:0/2733244973 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12fc0b2040 con 0x7f130c0b0160 2024-03-31T15:06:16.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:16.395+0000 7f1313600640 1 -- 172.21.15.16:0/2733244973 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f130c1612d0 con 0x7f130c0b0160 2024-03-31T15:06:16.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:16.395+0000 7f1313600640 1 -- 172.21.15.16:0/2733244973 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f130c14fe30 con 0x7f130c0b0160 2024-03-31T15:06:16.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:16.395+0000 7f1313600640 1 -- 172.21.15.16:0/2733244973 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f130c069950 con 0x7f130c0b0160 2024-03-31T15:06:16.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:16.395+0000 7f1309600640 1 -- 172.21.15.16:0/2733244973 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f12fc0bf020 con 0x7f130c0b0160 2024-03-31T15:06:16.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:16.399+0000 7f1309600640 1 --2- 172.21.15.16:0/2733244973 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f12e80683d0 0x7f12e806a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:16.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:16.399+0000 7f1309600640 1 -- 172.21.15.16:0/2733244973 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 5357+0+0 (secure 0 0 0) 0x7f12fc120a10 con 0x7f130c0b0160 2024-03-31T15:06:16.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:16.399+0000 7f1303e00640 1 --2- 172.21.15.16:0/2733244973 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f12e80683d0 0x7f12e806a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:16.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:16.399+0000 7f1303e00640 1 --2- 172.21.15.16:0/2733244973 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f12e80683d0 0x7f12e806a890 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f130c156f00 tx=0x7f12f4008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:06:16.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:16.403+0000 7f1309600640 1 -- 172.21.15.16:0/2733244973 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f12fc0c6030 con 0x7f130c0b0160 2024-03-31T15:06:16.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:16.567+0000 7f1313600640 1 -- 172.21.15.16:0/2733244973 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7f130c0aff40 con 0x7f130c0b0160 2024-03-31T15:06:17.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:17.207+0000 7f1309600640 1 -- 172.21.15.16:0/2733244973 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v48) ==== 169+0+0 (secure 0 0 0) 0x7f130c161050 con 0x7f130c0b0160 2024-03-31T15:06:17.203 INFO:teuthology.orchestra.run.smithi016.stderr:pool 'cephfs_data' removed 2024-03-31T15:06:17.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:17.211+0000 7f1313600640 1 -- 172.21.15.16:0/2733244973 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f12e80683d0 msgr2=0x7f12e806a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:17.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:17.211+0000 7f1313600640 1 --2- 172.21.15.16:0/2733244973 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f12e80683d0 0x7f12e806a890 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f130c156f00 tx=0x7f12f4008040 comp rx=0 tx=0).stop 2024-03-31T15:06:17.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:17.211+0000 7f1313600640 1 -- 172.21.15.16:0/2733244973 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f130c0b0160 msgr2=0x7f130c14f4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:17.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:17.211+0000 7f1313600640 1 --2- 172.21.15.16:0/2733244973 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f130c0b0160 0x7f130c14f4c0 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f12fc01cd30 tx=0x7f12fc01acd0 comp rx=0 tx=0).stop 2024-03-31T15:06:17.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:17.211+0000 7f1313600640 1 -- 172.21.15.16:0/2733244973 shutdown_connections 2024-03-31T15:06:17.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:17.211+0000 7f1313600640 1 --2- 172.21.15.16:0/2733244973 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f12e80683d0 0x7f12e806a890 secure :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f130c156f00 tx=0x7f12f4008040 comp rx=0 tx=0).stop 2024-03-31T15:06:17.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:17.211+0000 7f1313600640 1 --2- 172.21.15.16:0/2733244973 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f130c15e6c0 0x7f130c152570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:17.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:17.211+0000 7f1313600640 1 --2- 172.21.15.16:0/2733244973 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f130c0b4140 0x7f130c160b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:17.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:17.211+0000 7f1313600640 1 --2- 172.21.15.16:0/2733244973 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f130c0b0160 0x7f130c14f4c0 secure :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f12fc01cd30 tx=0x7f12fc01acd0 comp rx=0 tx=0).stop 2024-03-31T15:06:17.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:17.211+0000 7f1313600640 1 -- 172.21.15.16:0/2733244973 >> 172.21.15.16:0/2733244973 conn(0x7f130c0b2dd0 msgr2=0x7f130c1577a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:06:17.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:17.211+0000 7f1313600640 1 -- 172.21.15.16:0/2733244973 shutdown_connections 2024-03-31T15:06:17.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:17.211+0000 7f1313600640 1 -- 172.21.15.16:0/2733244973 wait complete. 2024-03-31T15:06:17.233 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-03-31T15:06:17.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:17.387+0000 7fa670800640 1 Processor -- start 2024-03-31T15:06:17.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:17.387+0000 7fa670800640 1 -- start start 2024-03-31T15:06:17.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:17.391+0000 7fa670800640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fa66813dff0 0x7fa66806d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:17.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:17.391+0000 7fa670800640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fa66813e3d0 0x7fa66806da10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:17.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:17.391+0000 7fa670800640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa66815fbc0 con 0x7fa6680cc9d0 2024-03-31T15:06:17.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:17.391+0000 7fa670800640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7fa66815f730 con 0x7fa66813dff0 2024-03-31T15:06:17.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:17.391+0000 7fa670800640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fa66815ee00 con 0x7fa66813e3d0 2024-03-31T15:06:17.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:17.391+0000 7fa66de00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fa66813dff0 0x7fa66806d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:17.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:17.391+0000 7fa66d400640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fa66813e3d0 0x7fa66806da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:17.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:17.391+0000 7fa66de00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fa66813dff0 0x7fa66806d4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.16:41950/0 (socket says 172.21.15.16:41950) 2024-03-31T15:06:17.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:17.391+0000 7fa66de00640 1 -- 172.21.15.16:0/2399405489 learned_addr learned my addr 172.21.15.16:0/2399405489 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:06:17.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:17.391+0000 7fa66e800640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fa6680cc9d0 0x7fa6680cce30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.16:51588/0 (socket says 172.21.15.16:51588) 2024-03-31T15:06:17.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:17.391+0000 7fa66d400640 1 -- 172.21.15.16:0/2399405489 >> v2:172.21.15.89:3300/0 conn(0x7fa66813dff0 msgr2=0x7fa66806d4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:06:17.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:17.391+0000 7fa66d400640 1 --2- 172.21.15.16:0/2399405489 >> v2:172.21.15.89:3300/0 conn(0x7fa66813dff0 0x7fa66806d4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:17.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:17.391+0000 7fa66d400640 1 -- 172.21.15.16:0/2399405489 >> v1:172.21.15.89:6789/0 conn(0x7fa6680cc9d0 legacy=0x7fa6680cce30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:17.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:17.391+0000 7fa66d400640 1 -- 172.21.15.16:0/2399405489 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa66815f0e0 con 0x7fa66813e3d0 2024-03-31T15:06:17.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:17.391+0000 7fa66de00640 1 --2- 172.21.15.16:0/2399405489 >> v2:172.21.15.89:3300/0 conn(0x7fa66813dff0 0x7fa66806d4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:06:17.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:17.395+0000 7fa66d400640 1 --2- 172.21.15.16:0/2399405489 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fa66813e3d0 0x7fa66806da10 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fa6580097c0 tx=0x7fa65801c410 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f4d71f805b51a9ed server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:06:17.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:17.395+0000 7fa66ca00640 1 -- 172.21.15.16:0/2399405489 <== mon.2 v2:172.21.15.89:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6580b0070 con 0x7fa66813e3d0 2024-03-31T15:06:17.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:17.395+0000 7fa66ca00640 1 -- 172.21.15.16:0/2399405489 <== mon.2 v2:172.21.15.89:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa6580ab600 con 0x7fa66813e3d0 2024-03-31T15:06:17.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:17.395+0000 7fa66ca00640 1 -- 172.21.15.16:0/2399405489 <== mon.2 v2:172.21.15.89:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6580ab9d0 con 0x7fa66813e3d0 2024-03-31T15:06:17.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:17.395+0000 7fa670800640 1 -- 172.21.15.16:0/2399405489 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fa66813e3d0 msgr2=0x7fa66806da10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:17.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:17.395+0000 7fa670800640 1 --2- 172.21.15.16:0/2399405489 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fa66813e3d0 0x7fa66806da10 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fa6580097c0 tx=0x7fa65801c410 comp rx=0 tx=0).stop 2024-03-31T15:06:17.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:17.395+0000 7fa670800640 1 -- 172.21.15.16:0/2399405489 shutdown_connections 2024-03-31T15:06:17.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:17.395+0000 7fa670800640 1 --2- 172.21.15.16:0/2399405489 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fa66813e3d0 0x7fa66806da10 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:17.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:17.395+0000 7fa670800640 1 --2- 172.21.15.16:0/2399405489 >> v2:172.21.15.89:3300/0 conn(0x7fa66813dff0 0x7fa66806d4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:17.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:17.395+0000 7fa670800640 1 -- 172.21.15.16:0/2399405489 >> 172.21.15.16:0/2399405489 conn(0x7fa66806ee40 msgr2=0x7fa66806b620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:06:17.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:17.395+0000 7fa670800640 1 -- 172.21.15.16:0/2399405489 shutdown_connections 2024-03-31T15:06:17.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:17.395+0000 7fa670800640 1 -- 172.21.15.16:0/2399405489 wait complete. 2024-03-31T15:06:17.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:17.395+0000 7fa670800640 1 Processor -- start 2024-03-31T15:06:17.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:17.395+0000 7fa670800640 1 -- start start 2024-03-31T15:06:17.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:17.399+0000 7fa670800640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa6680cc9d0 0x7fa668158700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:17.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:17.399+0000 7fa670800640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fa66813dff0 0x7fa66815a150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:17.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:17.399+0000 7fa670800640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa66813e3d0 0x7fa66815a690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:17.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:17.399+0000 7fa670800640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fa6680cf790 con 0x7fa6680cc9d0 2024-03-31T15:06:17.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:17.399+0000 7fa670800640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fa6680cf610 con 0x7fa66813e3d0 2024-03-31T15:06:17.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:17.399+0000 7fa670800640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fa6680cf910 con 0x7fa66813dff0 2024-03-31T15:06:17.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:17.399+0000 7fa66de00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa6680cc9d0 0x7fa668158700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:17.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:17.399+0000 7fa66de00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa6680cc9d0 0x7fa668158700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35776/0 (socket says 172.21.15.16:35776) 2024-03-31T15:06:17.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:17.399+0000 7fa66de00640 1 -- 172.21.15.16:0/602836705 learned_addr learned my addr 172.21.15.16:0/602836705 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:06:17.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:17.399+0000 7fa66e800640 1 --2- 172.21.15.16:0/602836705 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa66813e3d0 0x7fa66815a690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:17.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:17.399+0000 7fa66d400640 1 --2- 172.21.15.16:0/602836705 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fa66813dff0 0x7fa66815a150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:17.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:17.399+0000 7fa66de00640 1 -- 172.21.15.16:0/602836705 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fa66813dff0 msgr2=0x7fa66815a150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:17.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:17.399+0000 7fa66de00640 1 --2- 172.21.15.16:0/602836705 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fa66813dff0 0x7fa66815a150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:17.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:17.399+0000 7fa66de00640 1 -- 172.21.15.16:0/602836705 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa66813e3d0 msgr2=0x7fa66815a690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:17.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:17.399+0000 7fa66de00640 1 --2- 172.21.15.16:0/602836705 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa66813e3d0 0x7fa66815a690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:17.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:17.399+0000 7fa66de00640 1 -- 172.21.15.16:0/602836705 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa66815acd0 con 0x7fa6680cc9d0 2024-03-31T15:06:17.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:17.399+0000 7fa66e800640 1 --2- 172.21.15.16:0/602836705 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa66813e3d0 0x7fa66815a690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:06:17.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:17.399+0000 7fa66d400640 1 --2- 172.21.15.16:0/602836705 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fa66813dff0 0x7fa66815a150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:06:17.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:17.399+0000 7fa66de00640 1 --2- 172.21.15.16:0/602836705 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa6680cc9d0 0x7fa668158700 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7fa65c00e9d0 tx=0x7fa65c00ee90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:06:17.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:17.399+0000 7fa662000640 1 -- 172.21.15.16:0/602836705 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa65c017070 con 0x7fa6680cc9d0 2024-03-31T15:06:17.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:17.399+0000 7fa662000640 1 -- 172.21.15.16:0/602836705 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa65c00ca60 con 0x7fa6680cc9d0 2024-03-31T15:06:17.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:17.399+0000 7fa662000640 1 -- 172.21.15.16:0/602836705 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa65c00ce10 con 0x7fa6680cc9d0 2024-03-31T15:06:17.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:17.399+0000 7fa670800640 1 -- 172.21.15.16:0/602836705 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa66815afb0 con 0x7fa6680cc9d0 2024-03-31T15:06:17.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:17.399+0000 7fa670800640 1 -- 172.21.15.16:0/602836705 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa668128310 con 0x7fa6680cc9d0 2024-03-31T15:06:17.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:17.399+0000 7fa670800640 1 -- 172.21.15.16:0/602836705 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa668069950 con 0x7fa6680cc9d0 2024-03-31T15:06:17.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:17.399+0000 7fa662000640 1 -- 172.21.15.16:0/602836705 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fa65c01a070 con 0x7fa6680cc9d0 2024-03-31T15:06:17.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:17.403+0000 7fa662000640 1 --2- 172.21.15.16:0/602836705 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fa64c068380 0x7fa64c06a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:17.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:17.403+0000 7fa662000640 1 -- 172.21.15.16:0/602836705 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 4943+0+0 (secure 0 0 0) 0x7fa65c0869c0 con 0x7fa6680cc9d0 2024-03-31T15:06:17.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:17.403+0000 7fa66d400640 1 --2- 172.21.15.16:0/602836705 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fa64c068380 0x7fa64c06a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:17.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:17.403+0000 7fa66d400640 1 --2- 172.21.15.16:0/602836705 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fa64c068380 0x7fa64c06a840 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fa6580ad000 tx=0x7fa658008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:06:17.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:17.407+0000 7fa662000640 1 -- 172.21.15.16:0/602836705 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fa65c057270 con 0x7fa6680cc9d0 2024-03-31T15:06:17.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:17.579+0000 7fa670800640 1 -- 172.21.15.16:0/602836705 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) -- 0x7fa66806e4e0 con 0x7fa6680cc9d0 2024-03-31T15:06:17.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:17.583+0000 7fa662000640 1 -- 172.21.15.16:0/602836705 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v13) ==== 104+0+0 (secure 0 0 0) 0x7fa65c056b70 con 0x7fa6680cc9d0 2024-03-31T15:06:17.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:17.583+0000 7fa670800640 1 -- 172.21.15.16:0/602836705 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fa64c068380 msgr2=0x7fa64c06a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:17.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:17.583+0000 7fa670800640 1 --2- 172.21.15.16:0/602836705 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fa64c068380 0x7fa64c06a840 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fa6580ad000 tx=0x7fa658008040 comp rx=0 tx=0).stop 2024-03-31T15:06:17.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:17.583+0000 7fa670800640 1 -- 172.21.15.16:0/602836705 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa6680cc9d0 msgr2=0x7fa668158700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:17.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:17.583+0000 7fa670800640 1 --2- 172.21.15.16:0/602836705 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa6680cc9d0 0x7fa668158700 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7fa65c00e9d0 tx=0x7fa65c00ee90 comp rx=0 tx=0).stop 2024-03-31T15:06:17.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:17.583+0000 7fa670800640 1 -- 172.21.15.16:0/602836705 shutdown_connections 2024-03-31T15:06:17.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:17.583+0000 7fa670800640 1 --2- 172.21.15.16:0/602836705 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fa64c068380 0x7fa64c06a840 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:17.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:17.583+0000 7fa670800640 1 --2- 172.21.15.16:0/602836705 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa66813e3d0 0x7fa66815a690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:17.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:17.583+0000 7fa670800640 1 --2- 172.21.15.16:0/602836705 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fa66813dff0 0x7fa66815a150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:17.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:17.583+0000 7fa670800640 1 --2- 172.21.15.16:0/602836705 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa6680cc9d0 0x7fa668158700 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:17.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:17.583+0000 7fa670800640 1 -- 172.21.15.16:0/602836705 >> 172.21.15.16:0/602836705 conn(0x7fa66806ee40 msgr2=0x7fa668157670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:06:17.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:17.583+0000 7fa670800640 1 -- 172.21.15.16:0/602836705 shutdown_connections 2024-03-31T15:06:17.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:17.583+0000 7fa670800640 1 -- 172.21.15.16:0/602836705 wait complete. 2024-03-31T15:06:17.598 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_backtrace_repair' 2024-03-31T15:06:17.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:17.743+0000 7f5dd1800640 1 Processor -- start 2024-03-31T15:06:17.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:17.743+0000 7f5dd1800640 1 -- start start 2024-03-31T15:06:17.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:17.747+0000 7f5dd1800640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f5dcc15b3d0 0x7f5dcc0b1cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:17.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:17.747+0000 7f5dd1800640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f5dcc15eee0 0x7f5dcc0b2200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:17.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:17.747+0000 7f5dd1800640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5dcc15f3c0 con 0x7f5dcc0ce2f0 2024-03-31T15:06:17.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:17.747+0000 7f5dd1800640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f5dcc15b7b0 con 0x7f5dcc15eee0 2024-03-31T15:06:17.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:17.747+0000 7f5dd1800640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f5dcc15f6a0 con 0x7f5dcc15b3d0 2024-03-31T15:06:17.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:17.747+0000 7f5dc2000640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f5dcc15eee0 0x7f5dcc0b2200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:17.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:17.747+0000 7f5dcb400640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f5dcc0ce2f0 0x7f5dcc0cc9c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:50628/0 (socket says 172.21.15.16:50628) 2024-03-31T15:06:17.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:17.747+0000 7f5dc2000640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f5dcc15eee0 0x7f5dcc0b2200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35784/0 (socket says 172.21.15.16:35784) 2024-03-31T15:06:17.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:17.747+0000 7f5dcb400640 1 -- 172.21.15.16:0/2151014954 learned_addr learned my addr 172.21.15.16:0/2151014954 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:06:17.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:17.747+0000 7f5dcaa00640 1 --2- 172.21.15.16:0/2151014954 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f5dcc15b3d0 0x7f5dcc0b1cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:17.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:17.747+0000 7f5dc2000640 1 -- 172.21.15.16:0/2151014954 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f5dcc15b3d0 msgr2=0x7f5dcc0b1cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:06:17.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:17.747+0000 7f5dc2000640 1 --2- 172.21.15.16:0/2151014954 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f5dcc15b3d0 0x7f5dcc0b1cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:17.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:17.747+0000 7f5dc2000640 1 -- 172.21.15.16:0/2151014954 >> v1:172.21.15.16:6789/0 conn(0x7f5dcc0ce2f0 legacy=0x7f5dcc0cc9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:17.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:17.747+0000 7f5dc2000640 1 -- 172.21.15.16:0/2151014954 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5dcc15f980 con 0x7f5dcc15eee0 2024-03-31T15:06:17.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:17.747+0000 7f5dcaa00640 1 --2- 172.21.15.16:0/2151014954 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f5dcc15b3d0 0x7f5dcc0b1cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:06:17.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:17.747+0000 7f5dca000640 1 -- 172.21.15.16:0/2151014954 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2286330381 0 0) 0x7f5dcc15f3c0 con 0x7f5dcc0ce2f0 2024-03-31T15:06:17.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:17.747+0000 7f5dc2000640 1 --2- 172.21.15.16:0/2151014954 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5dcc15eee0 0x7f5dcc0b2200 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f5db4009d90 tx=0x7f5db401c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9952592e7572fbc1 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:06:17.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:17.751+0000 7f5dca000640 1 -- 172.21.15.16:0/2151014954 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5db40b2070 con 0x7f5dcc15eee0 2024-03-31T15:06:17.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:17.751+0000 7f5dca000640 1 -- 172.21.15.16:0/2151014954 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5db40a59c0 con 0x7f5dcc15eee0 2024-03-31T15:06:17.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:17.751+0000 7f5dca000640 1 -- 172.21.15.16:0/2151014954 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5db40a5d90 con 0x7f5dcc15eee0 2024-03-31T15:06:17.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:17.751+0000 7f5dd1800640 1 -- 172.21.15.16:0/2151014954 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5dcc15eee0 msgr2=0x7f5dcc0b2200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:17.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:17.751+0000 7f5dd1800640 1 --2- 172.21.15.16:0/2151014954 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5dcc15eee0 0x7f5dcc0b2200 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f5db4009d90 tx=0x7f5db401c940 comp rx=0 tx=0).stop 2024-03-31T15:06:17.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:17.751+0000 7f5dd1800640 1 -- 172.21.15.16:0/2151014954 shutdown_connections 2024-03-31T15:06:17.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:17.751+0000 7f5dd1800640 1 --2- 172.21.15.16:0/2151014954 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5dcc15eee0 0x7f5dcc0b2200 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:17.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:17.751+0000 7f5dd1800640 1 --2- 172.21.15.16:0/2151014954 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f5dcc15b3d0 0x7f5dcc0b1cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:17.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:17.751+0000 7f5dd1800640 1 -- 172.21.15.16:0/2151014954 >> 172.21.15.16:0/2151014954 conn(0x7f5dcc06b620 msgr2=0x7f5dcc06ba30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:06:17.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:17.751+0000 7f5dd1800640 1 -- 172.21.15.16:0/2151014954 shutdown_connections 2024-03-31T15:06:17.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:17.751+0000 7f5dd1800640 1 -- 172.21.15.16:0/2151014954 wait complete. 2024-03-31T15:06:17.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:17.751+0000 7f5dd1800640 1 Processor -- start 2024-03-31T15:06:17.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:17.751+0000 7f5dd1800640 1 -- start start 2024-03-31T15:06:17.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:17.751+0000 7f5dd1800640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5dcc0ce2f0 0x7f5dcc1420e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:17.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:17.751+0000 7f5dd1800640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5dcc15b3d0 0x7f5dcc15a220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:17.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:17.751+0000 7f5dd1800640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f5dcc15eee0 0x7f5dcc15ca80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:17.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:17.751+0000 7f5dd1800640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f5dcc0d03b0 con 0x7f5dcc15b3d0 2024-03-31T15:06:17.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:17.751+0000 7f5dd1800640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f5dcc0d0230 con 0x7f5dcc0ce2f0 2024-03-31T15:06:17.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:17.751+0000 7f5dd1800640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f5dcc0d0530 con 0x7f5dcc15eee0 2024-03-31T15:06:17.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:17.751+0000 7f5dc2000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5dcc15b3d0 0x7f5dcc15a220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:17.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:17.751+0000 7f5dc2000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5dcc15b3d0 0x7f5dcc15a220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35788/0 (socket says 172.21.15.16:35788) 2024-03-31T15:06:17.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:17.751+0000 7f5dc2000640 1 -- 172.21.15.16:0/3833322891 learned_addr learned my addr 172.21.15.16:0/3833322891 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:06:17.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:17.751+0000 7f5dcaa00640 1 --2- 172.21.15.16:0/3833322891 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5dcc0ce2f0 0x7f5dcc1420e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:17.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:17.751+0000 7f5dcb400640 1 --2- 172.21.15.16:0/3833322891 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f5dcc15eee0 0x7f5dcc15ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:17.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:17.755+0000 7f5dc2000640 1 -- 172.21.15.16:0/3833322891 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f5dcc15eee0 msgr2=0x7f5dcc15ca80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:17.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:17.755+0000 7f5dc2000640 1 --2- 172.21.15.16:0/3833322891 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f5dcc15eee0 0x7f5dcc15ca80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:17.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:17.755+0000 7f5dc2000640 1 -- 172.21.15.16:0/3833322891 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5dcc0ce2f0 msgr2=0x7f5dcc1420e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:17.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:17.755+0000 7f5dc2000640 1 --2- 172.21.15.16:0/3833322891 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5dcc0ce2f0 0x7f5dcc1420e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:17.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:17.755+0000 7f5dc2000640 1 -- 172.21.15.16:0/3833322891 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5dcc15bd30 con 0x7f5dcc15b3d0 2024-03-31T15:06:17.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:17.755+0000 7f5dc2000640 1 --2- 172.21.15.16:0/3833322891 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5dcc15b3d0 0x7f5dcc15a220 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f5db40b0040 tx=0x7f5db40196e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:06:17.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:17.755+0000 7f5dc3400640 1 -- 172.21.15.16:0/3833322891 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5db40b2050 con 0x7f5dcc15b3d0 2024-03-31T15:06:17.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:17.755+0000 7f5dc3400640 1 -- 172.21.15.16:0/3833322891 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5db40be070 con 0x7f5dcc15b3d0 2024-03-31T15:06:17.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:17.755+0000 7f5dc3400640 1 -- 172.21.15.16:0/3833322891 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5db40ada50 con 0x7f5dcc15b3d0 2024-03-31T15:06:17.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:17.755+0000 7f5dd1800640 1 -- 172.21.15.16:0/3833322891 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5dcc143990 con 0x7f5dcc15b3d0 2024-03-31T15:06:17.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:17.755+0000 7f5dd1800640 1 -- 172.21.15.16:0/3833322891 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5dcc15c280 con 0x7f5dcc15b3d0 2024-03-31T15:06:17.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:17.755+0000 7f5dd1800640 1 -- 172.21.15.16:0/3833322891 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5dcc069950 con 0x7f5dcc15b3d0 2024-03-31T15:06:17.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:17.755+0000 7f5dc3400640 1 -- 172.21.15.16:0/3833322891 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f5db40adbf0 con 0x7f5dcc15b3d0 2024-03-31T15:06:17.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:17.755+0000 7f5dc3400640 1 --2- 172.21.15.16:0/3833322891 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f5da8068380 0x7f5da806a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:17.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:17.755+0000 7f5dc3400640 1 -- 172.21.15.16:0/3833322891 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 4943+0+0 (secure 0 0 0) 0x7f5db4121b40 con 0x7f5dcc15b3d0 2024-03-31T15:06:17.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:17.759+0000 7f5dcaa00640 1 --2- 172.21.15.16:0/3833322891 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f5da8068380 0x7f5da806a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:17.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:17.759+0000 7f5dcaa00640 1 --2- 172.21.15.16:0/3833322891 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f5da8068380 0x7f5da806a840 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f5dcc0b0650 tx=0x7f5dbc008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:06:17.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:17.763+0000 7f5dc3400640 1 -- 172.21.15.16:0/3833322891 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f5db4016560 con 0x7f5dcc15b3d0 2024-03-31T15:06:17.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:17.983+0000 7f5dd1800640 1 -- 172.21.15.16:0/3833322891 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_backtrace_repair"]} v 0) -- 0x7f5dcc0cdea0 con 0x7f5dcc15b3d0 2024-03-31T15:06:18.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:18.211+0000 7f5dc3400640 1 -- 172.21.15.16:0/3833322891 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_backtrace_repair"]}]=0 v158) ==== 149+0+0 (secure 0 0 0) 0x7f5db40ab340 con 0x7f5dcc15b3d0 2024-03-31T15:06:18.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:18.211+0000 7f5dd1800640 1 -- 172.21.15.16:0/3833322891 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f5da8068380 msgr2=0x7f5da806a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:18.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:18.211+0000 7f5dd1800640 1 --2- 172.21.15.16:0/3833322891 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f5da8068380 0x7f5da806a840 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f5dcc0b0650 tx=0x7f5dbc008040 comp rx=0 tx=0).stop 2024-03-31T15:06:18.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:18.211+0000 7f5dd1800640 1 -- 172.21.15.16:0/3833322891 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5dcc15b3d0 msgr2=0x7f5dcc15a220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:18.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:18.211+0000 7f5dd1800640 1 --2- 172.21.15.16:0/3833322891 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5dcc15b3d0 0x7f5dcc15a220 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f5db40b0040 tx=0x7f5db40196e0 comp rx=0 tx=0).stop 2024-03-31T15:06:18.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:18.211+0000 7f5dd1800640 1 -- 172.21.15.16:0/3833322891 shutdown_connections 2024-03-31T15:06:18.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:18.211+0000 7f5dd1800640 1 --2- 172.21.15.16:0/3833322891 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f5da8068380 0x7f5da806a840 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:18.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:18.211+0000 7f5dd1800640 1 --2- 172.21.15.16:0/3833322891 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f5dcc15eee0 0x7f5dcc15ca80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:18.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:18.211+0000 7f5dd1800640 1 --2- 172.21.15.16:0/3833322891 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5dcc15b3d0 0x7f5dcc15a220 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:18.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:18.211+0000 7f5dd1800640 1 --2- 172.21.15.16:0/3833322891 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5dcc0ce2f0 0x7f5dcc1420e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:18.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:18.211+0000 7f5dd1800640 1 -- 172.21.15.16:0/3833322891 >> 172.21.15.16:0/3833322891 conn(0x7f5dcc06b620 msgr2=0x7f5dcc0cf500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:06:18.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:18.211+0000 7f5dd1800640 1 -- 172.21.15.16:0/3833322891 shutdown_connections 2024-03-31T15:06:18.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:18.211+0000 7f5dd1800640 1 -- 172.21.15.16:0/3833322891 wait complete. 2024-03-31T15:06:18.226 INFO:tasks.cephfs_test_runner:That the MDS can repair an inodes backtrace in the data pool ... ok 2024-03-31T15:06:18.227 INFO:tasks.cephfs_test_runner:Starting test: test_health_status_after_backtrace_repair (tasks.cephfs.test_forward_scrub.TestForwardScrub) Test that the damage health status is cleared 2024-03-31T15:06:18.227 INFO:tasks.cephfs_test_runner:test_health_status_after_backtrace_repair (tasks.cephfs.test_forward_scrub.TestForwardScrub) 2024-03-31T15:06:18.227 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-31T15:06:18.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:18.371+0000 7f880b200640 1 Processor -- start 2024-03-31T15:06:18.368 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:18.371+0000 7f880b200640 1 -- start start 2024-03-31T15:06:18.369 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:18.375+0000 7f880b200640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f880415b3d0 0x7f88040b1cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:18.369 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:18.375+0000 7f880b200640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f880415eee0 0x7f88040b2200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:18.369 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:18.375+0000 7f880b200640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f880415f3c0 con 0x7f88040ce2f0 2024-03-31T15:06:18.369 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:18.375+0000 7f880b200640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f880415b7b0 con 0x7f880415eee0 2024-03-31T15:06:18.369 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:18.375+0000 7f880b200640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f880415f6a0 con 0x7f880415b3d0 2024-03-31T15:06:18.369 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:18.375+0000 7f87fb400640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f880415eee0 0x7f88040b2200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:18.369 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:18.375+0000 7f87fb400640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f880415eee0 0x7f88040b2200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35804/0 (socket says 172.21.15.16:35804) 2024-03-31T15:06:18.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:18.375+0000 7f87fb400640 1 -- 172.21.15.16:0/2423483002 learned_addr learned my addr 172.21.15.16:0/2423483002 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:06:18.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:18.375+0000 7f8803e00640 1 --2- 172.21.15.16:0/2423483002 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f880415b3d0 0x7f88040b1cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:18.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:18.375+0000 7f87fb400640 1 -- 172.21.15.16:0/2423483002 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f880415b3d0 msgr2=0x7f88040b1cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:06:18.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:18.375+0000 7f87fb400640 1 --2- 172.21.15.16:0/2423483002 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f880415b3d0 0x7f88040b1cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:18.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:18.375+0000 7f8803400640 1 -- 172.21.15.16:0/2423483002 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 34235821 0 0) 0x7f880415f3c0 con 0x7f88040ce2f0 2024-03-31T15:06:18.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:18.375+0000 7f87fb400640 1 -- 172.21.15.16:0/2423483002 >> v1:172.21.15.16:6789/0 conn(0x7f88040ce2f0 legacy=0x7f88040cc9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:18.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:18.375+0000 7f87fb400640 1 -- 172.21.15.16:0/2423483002 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f880415f980 con 0x7f880415eee0 2024-03-31T15:06:18.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:18.375+0000 7f8803e00640 1 --2- 172.21.15.16:0/2423483002 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f880415b3d0 0x7f88040b1cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:06:18.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:18.379+0000 7f87fb400640 1 --2- 172.21.15.16:0/2423483002 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f880415eee0 0x7f88040b2200 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f87ec009870 tx=0x7f87ec01e740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=582852d71e0d582c server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:06:18.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:18.379+0000 7f8803400640 1 -- 172.21.15.16:0/2423483002 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87ec0a86d0 con 0x7f880415eee0 2024-03-31T15:06:18.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:18.379+0000 7f8803400640 1 -- 172.21.15.16:0/2423483002 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f87ec0a8870 con 0x7f880415eee0 2024-03-31T15:06:18.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:18.379+0000 7f8803400640 1 -- 172.21.15.16:0/2423483002 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87ec0a8c40 con 0x7f880415eee0 2024-03-31T15:06:18.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:18.379+0000 7f880b200640 1 -- 172.21.15.16:0/2423483002 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f880415eee0 msgr2=0x7f88040b2200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:18.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:18.379+0000 7f880b200640 1 --2- 172.21.15.16:0/2423483002 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f880415eee0 0x7f88040b2200 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f87ec009870 tx=0x7f87ec01e740 comp rx=0 tx=0).stop 2024-03-31T15:06:18.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:18.379+0000 7f880b200640 1 -- 172.21.15.16:0/2423483002 shutdown_connections 2024-03-31T15:06:18.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:18.379+0000 7f880b200640 1 --2- 172.21.15.16:0/2423483002 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f880415eee0 0x7f88040b2200 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:18.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:18.379+0000 7f880b200640 1 --2- 172.21.15.16:0/2423483002 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f880415b3d0 0x7f88040b1cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:18.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:18.379+0000 7f880b200640 1 -- 172.21.15.16:0/2423483002 >> 172.21.15.16:0/2423483002 conn(0x7f880406b620 msgr2=0x7f880406ba30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:06:18.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:18.379+0000 7f880b200640 1 -- 172.21.15.16:0/2423483002 shutdown_connections 2024-03-31T15:06:18.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:18.379+0000 7f880b200640 1 -- 172.21.15.16:0/2423483002 wait complete. 2024-03-31T15:06:18.375 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:18.379+0000 7f880b200640 1 Processor -- start 2024-03-31T15:06:18.375 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:18.379+0000 7f880b200640 1 -- start start 2024-03-31T15:06:18.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:18.383+0000 7f880b200640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f88040ce2f0 0x7f8804156f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:18.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:18.383+0000 7f880b200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f880415b3d0 0x7f88040aed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:18.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:18.383+0000 7f880b200640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f880415eee0 0x7f8804152670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:18.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:18.383+0000 7f880b200640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f88040d03b0 con 0x7f880415b3d0 2024-03-31T15:06:18.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:18.383+0000 7f880b200640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f88040d0230 con 0x7f880415eee0 2024-03-31T15:06:18.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:18.383+0000 7f880b200640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f88040d0530 con 0x7f88040ce2f0 2024-03-31T15:06:18.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:18.383+0000 7f87fb400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f880415b3d0 0x7f88040aed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:18.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:18.383+0000 7f87fb400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f880415b3d0 0x7f88040aed20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35816/0 (socket says 172.21.15.16:35816) 2024-03-31T15:06:18.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:18.383+0000 7f87fb400640 1 -- 172.21.15.16:0/2729793382 learned_addr learned my addr 172.21.15.16:0/2729793382 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:06:18.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:18.383+0000 7f8803e00640 1 --2- 172.21.15.16:0/2729793382 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f88040ce2f0 0x7f8804156f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:18.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:18.383+0000 7f8809200640 1 --2- 172.21.15.16:0/2729793382 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f880415eee0 0x7f8804152670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:18.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:18.383+0000 7f87fb400640 1 -- 172.21.15.16:0/2729793382 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f88040ce2f0 msgr2=0x7f8804156f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:18.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:18.383+0000 7f87fb400640 1 --2- 172.21.15.16:0/2729793382 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f88040ce2f0 0x7f8804156f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:18.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:18.383+0000 7f87fb400640 1 -- 172.21.15.16:0/2729793382 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f880415eee0 msgr2=0x7f8804152670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:18.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:18.383+0000 7f87fb400640 1 --2- 172.21.15.16:0/2729793382 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f880415eee0 0x7f8804152670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:18.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:18.383+0000 7f87fb400640 1 -- 172.21.15.16:0/2729793382 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f880415e150 con 0x7f880415b3d0 2024-03-31T15:06:18.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:18.383+0000 7f87fb400640 1 --2- 172.21.15.16:0/2729793382 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f880415b3d0 0x7f88040aed20 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f87ec0b2040 tx=0x7f87ec01b4e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:06:18.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:18.383+0000 7f8800c00640 1 -- 172.21.15.16:0/2729793382 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87ec0be070 con 0x7f880415b3d0 2024-03-31T15:06:18.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:18.383+0000 7f8800c00640 1 -- 172.21.15.16:0/2729793382 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f87ec0b0850 con 0x7f880415b3d0 2024-03-31T15:06:18.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:18.383+0000 7f8800c00640 1 -- 172.21.15.16:0/2729793382 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87ec0b0c40 con 0x7f880415b3d0 2024-03-31T15:06:18.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:18.383+0000 7f880b200640 1 -- 172.21.15.16:0/2729793382 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f880415e430 con 0x7f880415b3d0 2024-03-31T15:06:18.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:18.383+0000 7f880b200640 1 -- 172.21.15.16:0/2729793382 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f88040af260 con 0x7f880415b3d0 2024-03-31T15:06:18.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:18.383+0000 7f880b200640 1 -- 172.21.15.16:0/2729793382 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8804069950 con 0x7f880415b3d0 2024-03-31T15:06:18.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:18.383+0000 7f8800c00640 1 -- 172.21.15.16:0/2729793382 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f87ec0b83e0 con 0x7f880415b3d0 2024-03-31T15:06:18.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:18.387+0000 7f8800c00640 1 --2- 172.21.15.16:0/2729793382 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f87e0068380 0x7f87e006a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:18.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:18.387+0000 7f8800c00640 1 -- 172.21.15.16:0/2729793382 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 4943+0+0 (secure 0 0 0) 0x7f87ec120a00 con 0x7f880415b3d0 2024-03-31T15:06:18.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:18.387+0000 7f8803e00640 1 --2- 172.21.15.16:0/2729793382 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f87e0068380 0x7f87e006a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:18.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:18.387+0000 7f8803e00640 1 --2- 172.21.15.16:0/2729793382 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f87e0068380 0x7f87e006a840 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f8804153360 tx=0x7f87f4007810 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:06:18.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:18.391+0000 7f8800c00640 1 -- 172.21.15.16:0/2729793382 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f87ec018360 con 0x7f880415b3d0 2024-03-31T15:06:18.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:18.555+0000 7f880b200640 1 -- 172.21.15.16:0/2729793382 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f88040cdea0 con 0x7f880415b3d0 2024-03-31T15:06:18.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:18.555+0000 7f8800c00640 1 -- 172.21.15.16:0/2729793382 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v48) ==== 74+0+12948 (secure 0 0 0) 0x7f87ec0f0cf0 con 0x7f880415b3d0 2024-03-31T15:06:18.551 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:06:18.551 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":48,"fsid":"286eb5da-fb47-4550-b290-d9519c13b9a5","created":"2024-03-31T15:03:11.438952+0000","modified":"2024-03-31T15:06:17.205804+0000","last_up_change":"2024-03-31T15:03:22.885535+0000","last_in_change":"2024-03-31T15:03:13.941667+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-31T15:03:22.550787+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"92a66450-ef19-40d8-ac3d-d760b23b0bda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6801","nonce":932712244}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6803","nonce":932712244}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6807","nonce":932712244}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6805","nonce":932712244}]},"public_addr":"172.21.15.16:6801/932712244","cluster_addr":"172.21.15.16:6803/932712244","heartbeat_back_addr":"172.21.15.16:6807/932712244","heartbeat_front_addr":"172.21.15.16:6805/932712244","state":["exists","up"]},{"osd":1,"uuid":"da9d1f2b-1968-4959-b843-faa231988e92","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6810","nonce":1398737533}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6811","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6813","nonce":1398737533}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6821","nonce":1398737533}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6815","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6816","nonce":1398737533}]},"public_addr":"172.21.15.16:6810/1398737533","cluster_addr":"172.21.15.16:6813/1398737533","heartbeat_back_addr":"172.21.15.16:6821/1398737533","heartbeat_front_addr":"172.21.15.16:6816/1398737533","state":["exists","up"]},{"osd":2,"uuid":"01a73662-9a21-4b1f-bcbb-42b9636a4638","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6819","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6822","nonce":3018526646}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6825","nonce":3018526646}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6831","nonce":3018526646}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6827","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6829","nonce":3018526646}]},"public_addr":"172.21.15.16:6822/3018526646","cluster_addr":"172.21.15.16:6825/3018526646","heartbeat_back_addr":"172.21.15.16:6831/3018526646","heartbeat_front_addr":"172.21.15.16:6829/3018526646","state":["exists","up"]},{"osd":3,"uuid":"c7771fd3-e18e-4500-ac38-23cf10ebcd75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6809","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6812","nonce":3360352974}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6817","nonce":3360352974}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6828","nonce":3360352974}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6823","nonce":3360352974}]},"public_addr":"172.21.15.16:6812/3360352974","cluster_addr":"172.21.15.16:6817/3360352974","heartbeat_back_addr":"172.21.15.16:6828/3360352974","heartbeat_front_addr":"172.21.15.16:6823/3360352974","state":["exists","up"]},{"osd":4,"uuid":"88260730-1482-4fbd-a40f-cb1124e9df4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6801","nonce":1856844856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6803","nonce":1856844856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6807","nonce":1856844856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6805","nonce":1856844856}]},"public_addr":"172.21.15.89:6801/1856844856","cluster_addr":"172.21.15.89:6803/1856844856","heartbeat_back_addr":"172.21.15.89:6807/1856844856","heartbeat_front_addr":"172.21.15.89:6805/1856844856","state":["exists","up"]},{"osd":5,"uuid":"4cb2c00c-fae0-4dd8-9d27-ec4f0093ccf7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6814","nonce":50395527}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6817","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6820","nonce":50395527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6831","nonce":50395527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6823","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6826","nonce":50395527}]},"public_addr":"172.21.15.89:6814/50395527","cluster_addr":"172.21.15.89:6820/50395527","heartbeat_back_addr":"172.21.15.89:6831/50395527","heartbeat_front_addr":"172.21.15.89:6826/50395527","state":["exists","up"]},{"osd":6,"uuid":"81ce0c19-2554-4cae-acbe-66a05e9e9b9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6809","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6812","nonce":3470471752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6815","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6818","nonce":3470471752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6827","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6829","nonce":3470471752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6821","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6824","nonce":3470471752}]},"public_addr":"172.21.15.89:6812/3470471752","cluster_addr":"172.21.15.89:6818/3470471752","heartbeat_back_addr":"172.21.15.89:6829/3470471752","heartbeat_front_addr":"172.21.15.89:6824/3470471752","state":["exists","up"]},{"osd":7,"uuid":"85e4313b-d37d-442e-8042-153bf20a847d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6811","nonce":1796778630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6813","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6816","nonce":1796778630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6825","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6828","nonce":1796778630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6819","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6822","nonce":1796778630}]},"public_addr":"172.21.15.89:6811/1796778630","cluster_addr":"172.21.15.89:6816/1796778630","heartbeat_back_addr":"172.21.15.89:6828/1796778630","heartbeat_front_addr":"172.21.15.89:6822/1796778630","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.439088+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.529970+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.680512+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:21.044161+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.582111+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.536558+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.409522+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.590229+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:6834/2768973409":"2024-04-01T15:06:13.777234+0000","172.21.15.16:6832/2768973409":"2024-04-01T15:06:13.777234+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-31T15:06:18.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:18.559+0000 7f880b200640 1 -- 172.21.15.16:0/2729793382 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f87e0068380 msgr2=0x7f87e006a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:18.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:18.559+0000 7f880b200640 1 --2- 172.21.15.16:0/2729793382 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f87e0068380 0x7f87e006a840 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f8804153360 tx=0x7f87f4007810 comp rx=0 tx=0).stop 2024-03-31T15:06:18.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:18.559+0000 7f880b200640 1 -- 172.21.15.16:0/2729793382 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f880415b3d0 msgr2=0x7f88040aed20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:18.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:18.559+0000 7f880b200640 1 --2- 172.21.15.16:0/2729793382 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f880415b3d0 0x7f88040aed20 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f87ec0b2040 tx=0x7f87ec01b4e0 comp rx=0 tx=0).stop 2024-03-31T15:06:18.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:18.559+0000 7f880b200640 1 -- 172.21.15.16:0/2729793382 shutdown_connections 2024-03-31T15:06:18.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:18.559+0000 7f880b200640 1 --2- 172.21.15.16:0/2729793382 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f87e0068380 0x7f87e006a840 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:18.554 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:18.559+0000 7f880b200640 1 --2- 172.21.15.16:0/2729793382 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f880415eee0 0x7f8804152670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:18.554 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:18.559+0000 7f880b200640 1 --2- 172.21.15.16:0/2729793382 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f880415b3d0 0x7f88040aed20 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:18.554 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:18.559+0000 7f880b200640 1 --2- 172.21.15.16:0/2729793382 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f88040ce2f0 0x7f8804156f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:18.554 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:18.559+0000 7f880b200640 1 -- 172.21.15.16:0/2729793382 >> 172.21.15.16:0/2729793382 conn(0x7f880406b620 msgr2=0x7f88040b1980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:06:18.554 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:18.559+0000 7f880b200640 1 -- 172.21.15.16:0/2729793382 shutdown_connections 2024-03-31T15:06:18.554 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:18.559+0000 7f880b200640 1 -- 172.21.15.16:0/2729793382 wait complete. 2024-03-31T15:06:18.573 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-31T15:03:22.550787+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-31T15:06:18.573 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-31T15:06:18.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:18.723+0000 7fd69b200640 1 Processor -- start 2024-03-31T15:06:18.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:18.723+0000 7fd69b200640 1 -- start start 2024-03-31T15:06:18.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:18.723+0000 7fd69b200640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fd69406b620 0x7fd69406ba00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:18.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:18.723+0000 7fd69b200640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fd6940d0c00 0x7fd694070350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:18.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:18.723+0000 7fd69b200640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd69406bf40 con 0x7fd6940b60a0 2024-03-31T15:06:18.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:18.723+0000 7fd69b200640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7fd69406f820 con 0x7fd69406b620 2024-03-31T15:06:18.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:18.723+0000 7fd69b200640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7fd6940b4a40 con 0x7fd6940d0c00 2024-03-31T15:06:18.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:18.723+0000 7fd693e00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fd69406b620 0x7fd69406ba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:18.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:18.723+0000 7fd693e00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fd69406b620 0x7fd69406ba00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35818/0 (socket says 172.21.15.16:35818) 2024-03-31T15:06:18.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:18.723+0000 7fd693e00640 1 -- 172.21.15.16:0/2559600147 learned_addr learned my addr 172.21.15.16:0/2559600147 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:06:18.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:18.723+0000 7fd693e00640 1 -- 172.21.15.16:0/2559600147 >> v2:172.21.15.89:3300/0 conn(0x7fd6940d0c00 msgr2=0x7fd694070350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:06:18.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:18.723+0000 7fd699200640 1 --2- 172.21.15.16:0/2559600147 >> v2:172.21.15.89:3300/0 conn(0x7fd6940d0c00 0x7fd694070350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:18.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:18.723+0000 7fd693e00640 1 --2- 172.21.15.16:0/2559600147 >> v2:172.21.15.89:3300/0 conn(0x7fd6940d0c00 0x7fd694070350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:18.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:18.723+0000 7fd693e00640 1 -- 172.21.15.16:0/2559600147 >> v1:172.21.15.89:6789/0 conn(0x7fd6940b60a0 legacy=0x7fd69406fc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:06:18.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:18.723+0000 7fd693e00640 1 -- 172.21.15.16:0/2559600147 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd69406f540 con 0x7fd69406b620 2024-03-31T15:06:18.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:18.723+0000 7fd699200640 1 --2- 172.21.15.16:0/2559600147 >> v2:172.21.15.89:3300/0 conn(0x7fd6940d0c00 0x7fd694070350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:06:18.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:18.727+0000 7fd693e00640 1 --2- 172.21.15.16:0/2559600147 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd69406b620 0x7fd69406ba00 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7fd6940b2ee0 tx=0x7fd68401c420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5422b33b13c05cf4 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:06:18.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:18.727+0000 7fd693400640 1 -- 172.21.15.16:0/2559600147 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6840b0070 con 0x7fd69406b620 2024-03-31T15:06:18.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:18.727+0000 7fd693400640 1 -- 172.21.15.16:0/2559600147 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd6840ab600 con 0x7fd69406b620 2024-03-31T15:06:18.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:18.727+0000 7fd693400640 1 -- 172.21.15.16:0/2559600147 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6840ab9d0 con 0x7fd69406b620 2024-03-31T15:06:18.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:18.727+0000 7fd69b200640 1 -- 172.21.15.16:0/2559600147 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd69406b620 msgr2=0x7fd69406ba00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:18.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:18.727+0000 7fd69b200640 1 --2- 172.21.15.16:0/2559600147 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd69406b620 0x7fd69406ba00 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7fd6940b2ee0 tx=0x7fd68401c420 comp rx=0 tx=0).stop 2024-03-31T15:06:18.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:18.727+0000 7fd69b200640 1 -- 172.21.15.16:0/2559600147 shutdown_connections 2024-03-31T15:06:18.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:18.727+0000 7fd69b200640 1 --2- 172.21.15.16:0/2559600147 >> v2:172.21.15.89:3300/0 conn(0x7fd6940d0c00 0x7fd694070350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:18.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:18.727+0000 7fd69b200640 1 --2- 172.21.15.16:0/2559600147 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd69406b620 0x7fd69406ba00 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:18.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:18.727+0000 7fd69b200640 1 -- 172.21.15.16:0/2559600147 >> 172.21.15.16:0/2559600147 conn(0x7fd69406e670 msgr2=0x7fd69406ea50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:06:18.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:18.727+0000 7fd69b200640 1 -- 172.21.15.16:0/2559600147 shutdown_connections 2024-03-31T15:06:18.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:18.727+0000 7fd69b200640 1 -- 172.21.15.16:0/2559600147 wait complete. 2024-03-31T15:06:18.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:18.731+0000 7fd69b200640 1 Processor -- start 2024-03-31T15:06:18.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:18.731+0000 7fd69b200640 1 -- start start 2024-03-31T15:06:18.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:18.731+0000 7fd69b200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd69406b620 0x7fd69415a5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:18.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:18.731+0000 7fd69b200640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd6940b60a0 0x7fd694161480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:18.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:18.731+0000 7fd69b200640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fd6940d0c00 0x7fd694150960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:18.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:18.731+0000 7fd69b200640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fd69415e8d0 con 0x7fd69406b620 2024-03-31T15:06:18.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:18.731+0000 7fd69b200640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fd69415e750 con 0x7fd6940b60a0 2024-03-31T15:06:18.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:18.731+0000 7fd69b200640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fd69406f160 con 0x7fd6940d0c00 2024-03-31T15:06:18.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:18.731+0000 7fd693e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd69406b620 0x7fd69415a5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:18.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:18.731+0000 7fd693e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd69406b620 0x7fd69415a5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35828/0 (socket says 172.21.15.16:35828) 2024-03-31T15:06:18.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:18.731+0000 7fd693e00640 1 -- 172.21.15.16:0/447398569 learned_addr learned my addr 172.21.15.16:0/447398569 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:06:18.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:18.731+0000 7fd693e00640 1 -- 172.21.15.16:0/447398569 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fd6940d0c00 msgr2=0x7fd694150960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:18.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:18.731+0000 7fd693e00640 1 --2- 172.21.15.16:0/447398569 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fd6940d0c00 0x7fd694150960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:18.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:18.731+0000 7fd693e00640 1 -- 172.21.15.16:0/447398569 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd6940b60a0 msgr2=0x7fd694161480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:18.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:18.731+0000 7fd693e00640 1 --2- 172.21.15.16:0/447398569 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd6940b60a0 0x7fd694161480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:18.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:18.731+0000 7fd693e00640 1 -- 172.21.15.16:0/447398569 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd69415eaf0 con 0x7fd69406b620 2024-03-31T15:06:18.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:18.731+0000 7fd693e00640 1 --2- 172.21.15.16:0/447398569 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd69406b620 0x7fd69415a5f0 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7fd6840ae040 tx=0x7fd68401ac60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:06:18.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:18.731+0000 7fd690c00640 1 -- 172.21.15.16:0/447398569 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6840ba020 con 0x7fd69406b620 2024-03-31T15:06:18.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:18.731+0000 7fd690c00640 1 -- 172.21.15.16:0/447398569 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd6840b6640 con 0x7fd69406b620 2024-03-31T15:06:18.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:18.731+0000 7fd690c00640 1 -- 172.21.15.16:0/447398569 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6840b0040 con 0x7fd69406b620 2024-03-31T15:06:18.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:18.731+0000 7fd69b200640 1 -- 172.21.15.16:0/447398569 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd69415edd0 con 0x7fd69406b620 2024-03-31T15:06:18.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:18.731+0000 7fd69b200640 1 -- 172.21.15.16:0/447398569 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd69415f410 con 0x7fd69406b620 2024-03-31T15:06:18.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:18.731+0000 7fd69b200640 1 -- 172.21.15.16:0/447398569 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd694069950 con 0x7fd69406b620 2024-03-31T15:06:18.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:18.735+0000 7fd690c00640 1 -- 172.21.15.16:0/447398569 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fd6840bd020 con 0x7fd69406b620 2024-03-31T15:06:18.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:18.735+0000 7fd690c00640 1 --2- 172.21.15.16:0/447398569 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fd670068380 0x7fd67006a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:18.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:18.735+0000 7fd690c00640 1 -- 172.21.15.16:0/447398569 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 4943+0+0 (secure 0 0 0) 0x7fd684120a40 con 0x7fd69406b620 2024-03-31T15:06:18.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:18.735+0000 7fd68be00640 1 --2- 172.21.15.16:0/447398569 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fd670068380 0x7fd67006a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:18.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:18.735+0000 7fd68be00640 1 --2- 172.21.15.16:0/447398569 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fd670068380 0x7fd67006a840 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fd694153430 tx=0x7fd67c006bd0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:06:18.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:18.739+0000 7fd690c00640 1 -- 172.21.15.16:0/447398569 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fd6840c4030 con 0x7fd69406b620 2024-03-31T15:06:18.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:18.903+0000 7fd69b200640 1 -- 172.21.15.16:0/447398569 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fd694070f00 con 0x7fd69406b620 2024-03-31T15:06:18.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:18.907+0000 7fd690c00640 1 -- 172.21.15.16:0/447398569 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v48) ==== 93+0+10 (secure 0 0 0) 0x7fd6840160d0 con 0x7fd69406b620 2024-03-31T15:06:18.900 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2024-03-31T15:06:18.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:18.907+0000 7fd69b200640 1 -- 172.21.15.16:0/447398569 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fd670068380 msgr2=0x7fd67006a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:18.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:18.907+0000 7fd69b200640 1 --2- 172.21.15.16:0/447398569 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fd670068380 0x7fd67006a840 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fd694153430 tx=0x7fd67c006bd0 comp rx=0 tx=0).stop 2024-03-31T15:06:18.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:18.907+0000 7fd69b200640 1 -- 172.21.15.16:0/447398569 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd69406b620 msgr2=0x7fd69415a5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:18.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:18.907+0000 7fd69b200640 1 --2- 172.21.15.16:0/447398569 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd69406b620 0x7fd69415a5f0 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7fd6840ae040 tx=0x7fd68401ac60 comp rx=0 tx=0).stop 2024-03-31T15:06:18.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:18.907+0000 7fd69b200640 1 -- 172.21.15.16:0/447398569 shutdown_connections 2024-03-31T15:06:18.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:18.907+0000 7fd69b200640 1 --2- 172.21.15.16:0/447398569 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fd670068380 0x7fd67006a840 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:18.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:18.907+0000 7fd69b200640 1 --2- 172.21.15.16:0/447398569 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fd6940d0c00 0x7fd694150960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:18.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:18.907+0000 7fd69b200640 1 --2- 172.21.15.16:0/447398569 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd6940b60a0 0x7fd694161480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:18.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:18.907+0000 7fd69b200640 1 --2- 172.21.15.16:0/447398569 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd69406b620 0x7fd69415a5f0 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:18.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:18.907+0000 7fd69b200640 1 -- 172.21.15.16:0/447398569 >> 172.21.15.16:0/447398569 conn(0x7fd69406e670 msgr2=0x7fd69415c450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:06:18.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:18.907+0000 7fd69b200640 1 -- 172.21.15.16:0/447398569 shutdown_connections 2024-03-31T15:06:18.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:18.907+0000 7fd69b200640 1 -- 172.21.15.16:0/447398569 wait complete. 2024-03-31T15:06:18.921 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_health_status_after_backtrace_repair' 2024-03-31T15:06:19.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:19.067+0000 7f913c400640 1 Processor -- start 2024-03-31T15:06:19.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:19.067+0000 7f913c400640 1 -- start start 2024-03-31T15:06:19.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:19.067+0000 7f913c400640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f91340d0c50 0x7f9134072fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:19.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:19.067+0000 7f913c400640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f9134073510 0x7f913415eb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:19.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:19.067+0000 7f913c400640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f913406bd30 con 0x7f9134147e70 2024-03-31T15:06:19.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:19.067+0000 7f913c400640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f91340739a0 con 0x7f91340d0c50 2024-03-31T15:06:19.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:19.067+0000 7f913c400640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f91341621b0 con 0x7f9134073510 2024-03-31T15:06:19.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:19.067+0000 7f9139000640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f91340d0c50 0x7f9134072fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:19.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:19.071+0000 7f9139000640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f91340d0c50 0x7f9134072fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.16:42014/0 (socket says 172.21.15.16:42014) 2024-03-31T15:06:19.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:19.071+0000 7f913a400640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f9134073510 0x7f913415eb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:19.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:19.071+0000 7f9139000640 1 -- 172.21.15.16:0/1373520992 learned_addr learned my addr 172.21.15.16:0/1373520992 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:06:19.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:19.071+0000 7f9139a00640 1 --1- 172.21.15.16:0/1373520992 >> v1:172.21.15.89:6789/0 conn(0x7f9134147e70 0x7f913406b620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.16:51614/0 (socket says 172.21.15.16:51614) 2024-03-31T15:06:19.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:19.071+0000 7f913a400640 1 -- 172.21.15.16:0/1373520992 >> v2:172.21.15.89:3300/0 conn(0x7f91340d0c50 msgr2=0x7f9134072fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:06:19.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:19.071+0000 7f913a400640 1 --2- 172.21.15.16:0/1373520992 >> v2:172.21.15.89:3300/0 conn(0x7f91340d0c50 0x7f9134072fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:19.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:19.071+0000 7f913a400640 1 -- 172.21.15.16:0/1373520992 >> v1:172.21.15.89:6789/0 conn(0x7f9134147e70 legacy=0x7f913406b620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:19.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:19.071+0000 7f913a400640 1 -- 172.21.15.16:0/1373520992 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9134162490 con 0x7f9134073510 2024-03-31T15:06:19.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:19.071+0000 7f9133e00640 1 -- 172.21.15.16:0/1373520992 <== mon.1 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2686427681 0 0) 0x7f913406bd30 con 0x7f9134147e70 2024-03-31T15:06:19.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:19.071+0000 7f9139000640 1 --2- 172.21.15.16:0/1373520992 >> v2:172.21.15.89:3300/0 conn(0x7f91340d0c50 0x7f9134072fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:06:19.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:19.071+0000 7f913a400640 1 --2- 172.21.15.16:0/1373520992 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f9134073510 0x7f913415eb20 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f912c009d90 tx=0x7f912c01c970 comp rx=0 tx=0).ready entity=mon.2 client_cookie=31604ede8e335dc3 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:06:19.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:19.071+0000 7f9133e00640 1 -- 172.21.15.16:0/1373520992 <== mon.2 v2:172.21.15.89:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f912c0b2070 con 0x7f9134073510 2024-03-31T15:06:19.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:19.071+0000 7f9133e00640 1 -- 172.21.15.16:0/1373520992 <== mon.2 v2:172.21.15.89:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f912c0a5920 con 0x7f9134073510 2024-03-31T15:06:19.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:19.071+0000 7f9133e00640 1 -- 172.21.15.16:0/1373520992 <== mon.2 v2:172.21.15.89:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f912c0a5cf0 con 0x7f9134073510 2024-03-31T15:06:19.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:19.075+0000 7f913c400640 1 -- 172.21.15.16:0/1373520992 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f9134073510 msgr2=0x7f913415eb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:19.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:19.075+0000 7f913c400640 1 --2- 172.21.15.16:0/1373520992 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f9134073510 0x7f913415eb20 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f912c009d90 tx=0x7f912c01c970 comp rx=0 tx=0).stop 2024-03-31T15:06:19.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:19.075+0000 7f913c400640 1 -- 172.21.15.16:0/1373520992 shutdown_connections 2024-03-31T15:06:19.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:19.075+0000 7f913c400640 1 --2- 172.21.15.16:0/1373520992 >> v2:172.21.15.89:3300/0 conn(0x7f91340d0c50 0x7f9134072fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:19.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:19.075+0000 7f913c400640 1 --2- 172.21.15.16:0/1373520992 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f9134073510 0x7f913415eb20 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:19.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:19.075+0000 7f913c400640 1 -- 172.21.15.16:0/1373520992 >> 172.21.15.16:0/1373520992 conn(0x7f91340d1060 msgr2=0x7f91340d1470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:06:19.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:19.075+0000 7f913c400640 1 -- 172.21.15.16:0/1373520992 shutdown_connections 2024-03-31T15:06:19.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:19.075+0000 7f913c400640 1 -- 172.21.15.16:0/1373520992 wait complete. 2024-03-31T15:06:19.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:19.075+0000 7f913c400640 1 Processor -- start 2024-03-31T15:06:19.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:19.075+0000 7f913c400640 1 -- start start 2024-03-31T15:06:19.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:19.075+0000 7f913c400640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f9134073510 0x7f91340adae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:19.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:19.075+0000 7f913c400640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f91340d0c50 0x7f9134145840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:19.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:19.075+0000 7f913c400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9134147e70 0x7f913415d970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:19.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:19.075+0000 7f913c400640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f913415f870 con 0x7f9134147e70 2024-03-31T15:06:19.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:19.075+0000 7f913c400640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f9134162f70 con 0x7f91340d0c50 2024-03-31T15:06:19.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:19.075+0000 7f913c400640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f913415f9f0 con 0x7f9134073510 2024-03-31T15:06:19.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:19.075+0000 7f913a400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9134147e70 0x7f913415d970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:19.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:19.075+0000 7f913a400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9134147e70 0x7f913415d970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35830/0 (socket says 172.21.15.16:35830) 2024-03-31T15:06:19.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:19.075+0000 7f913a400640 1 -- 172.21.15.16:0/837825966 learned_addr learned my addr 172.21.15.16:0/837825966 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:06:19.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:19.075+0000 7f9139000640 1 --2- 172.21.15.16:0/837825966 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f91340d0c50 0x7f9134145840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:19.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:19.075+0000 7f9139a00640 1 --2- 172.21.15.16:0/837825966 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f9134073510 0x7f91340adae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:19.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:19.075+0000 7f913a400640 1 -- 172.21.15.16:0/837825966 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f9134073510 msgr2=0x7f91340adae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:19.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:19.075+0000 7f913a400640 1 --2- 172.21.15.16:0/837825966 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f9134073510 0x7f91340adae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:19.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:19.075+0000 7f913a400640 1 -- 172.21.15.16:0/837825966 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f91340d0c50 msgr2=0x7f9134145840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:19.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:19.075+0000 7f913a400640 1 --2- 172.21.15.16:0/837825966 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f91340d0c50 0x7f9134145840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:19.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:19.075+0000 7f913a400640 1 -- 172.21.15.16:0/837825966 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f91341460c0 con 0x7f9134147e70 2024-03-31T15:06:19.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:19.079+0000 7f913a400640 1 --2- 172.21.15.16:0/837825966 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9134147e70 0x7f913415d970 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f912c0b0040 tx=0x7f912c019710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:06:19.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:19.079+0000 7f9131600640 1 -- 172.21.15.16:0/837825966 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f912c0b2040 con 0x7f9134147e70 2024-03-31T15:06:19.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:19.079+0000 7f9131600640 1 -- 172.21.15.16:0/837825966 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f912c0b8470 con 0x7f9134147e70 2024-03-31T15:06:19.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:19.079+0000 7f913c400640 1 -- 172.21.15.16:0/837825966 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f91341463a0 con 0x7f9134147e70 2024-03-31T15:06:19.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:19.079+0000 7f9131600640 1 -- 172.21.15.16:0/837825966 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f912c0b8820 con 0x7f9134147e70 2024-03-31T15:06:19.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:19.079+0000 7f913c400640 1 -- 172.21.15.16:0/837825966 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9134146960 con 0x7f9134147e70 2024-03-31T15:06:19.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:19.079+0000 7f913c400640 1 -- 172.21.15.16:0/837825966 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9134069950 con 0x7f9134147e70 2024-03-31T15:06:19.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:19.079+0000 7f9131600640 1 -- 172.21.15.16:0/837825966 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f912c0c1020 con 0x7f9134147e70 2024-03-31T15:06:19.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:19.079+0000 7f9131600640 1 --2- 172.21.15.16:0/837825966 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f9114068380 0x7f911406a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:19.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:19.079+0000 7f9131600640 1 -- 172.21.15.16:0/837825966 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 4943+0+0 (secure 0 0 0) 0x7f912c120e90 con 0x7f9134147e70 2024-03-31T15:06:19.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:19.079+0000 7f9139a00640 1 --2- 172.21.15.16:0/837825966 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f9114068380 0x7f911406a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:19.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:19.083+0000 7f9139a00640 1 --2- 172.21.15.16:0/837825966 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f9114068380 0x7f911406a840 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f9128007fd0 tx=0x7f9128007d60 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:06:19.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:19.083+0000 7f9131600640 1 -- 172.21.15.16:0/837825966 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f912c016560 con 0x7f9134147e70 2024-03-31T15:06:19.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:19.311+0000 7f913c400640 1 -- 172.21.15.16:0/837825966 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_health_status_after_backtrace_repair"]} v 0) -- 0x7f91340d1bf0 con 0x7f9134147e70 2024-03-31T15:06:20.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:20.215+0000 7f9131600640 1 -- 172.21.15.16:0/837825966 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_health_status_after_backtrace_repair"]}]=0 v160) ==== 172+0+0 (secure 0 0 0) 0x7f912c0f1100 con 0x7f9134147e70 2024-03-31T15:06:20.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:20.219+0000 7f913c400640 1 -- 172.21.15.16:0/837825966 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f9114068380 msgr2=0x7f911406a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:20.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:20.219+0000 7f913c400640 1 --2- 172.21.15.16:0/837825966 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f9114068380 0x7f911406a840 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f9128007fd0 tx=0x7f9128007d60 comp rx=0 tx=0).stop 2024-03-31T15:06:20.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:20.219+0000 7f913c400640 1 -- 172.21.15.16:0/837825966 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9134147e70 msgr2=0x7f913415d970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:20.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:20.219+0000 7f913c400640 1 --2- 172.21.15.16:0/837825966 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9134147e70 0x7f913415d970 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f912c0b0040 tx=0x7f912c019710 comp rx=0 tx=0).stop 2024-03-31T15:06:20.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:20.219+0000 7f913c400640 1 -- 172.21.15.16:0/837825966 shutdown_connections 2024-03-31T15:06:20.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:20.219+0000 7f913c400640 1 --2- 172.21.15.16:0/837825966 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f9114068380 0x7f911406a840 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:20.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:20.219+0000 7f913c400640 1 --2- 172.21.15.16:0/837825966 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9134147e70 0x7f913415d970 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:20.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:20.219+0000 7f913c400640 1 --2- 172.21.15.16:0/837825966 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f91340d0c50 0x7f9134145840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:20.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:20.219+0000 7f913c400640 1 --2- 172.21.15.16:0/837825966 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f9134073510 0x7f91340adae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:20.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:20.219+0000 7f913c400640 1 -- 172.21.15.16:0/837825966 >> 172.21.15.16:0/837825966 conn(0x7f91340d1060 msgr2=0x7f91340b2600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:06:20.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:20.219+0000 7f913c400640 1 -- 172.21.15.16:0/837825966 shutdown_connections 2024-03-31T15:06:20.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:20.219+0000 7f913c400640 1 -- 172.21.15.16:0/837825966 wait complete. 2024-03-31T15:06:20.237 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-03-31T15:06:20.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:20.391+0000 7f8841c00640 1 Processor -- start 2024-03-31T15:06:20.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:20.391+0000 7f8841c00640 1 -- start start 2024-03-31T15:06:20.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:20.391+0000 7f8841c00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f883c13dff0 0x7f883c06d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:20.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:20.391+0000 7f8841c00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f883c0cd0e0 0x7f883c06da10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:20.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:20.391+0000 7f8841c00640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f883c15fbc0 con 0x7f883c13e3d0 2024-03-31T15:06:20.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:20.391+0000 7f8841c00640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f883c15f730 con 0x7f883c0cd0e0 2024-03-31T15:06:20.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:20.391+0000 7f8841c00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f883c15ee00 con 0x7f883c13dff0 2024-03-31T15:06:20.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:20.395+0000 7f883aa00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f883c13dff0 0x7f883c06d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:20.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:20.395+0000 7f883b400640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f883c0cd0e0 0x7f883c06da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:20.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:20.395+0000 7f883aa00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f883c13dff0 0x7f883c06d4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3301/0 says I am v2:172.21.15.16:43640/0 (socket says 172.21.15.16:43640) 2024-03-31T15:06:20.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:20.395+0000 7f883b400640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f883c0cd0e0 0x7f883c06da10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.16:42032/0 (socket says 172.21.15.16:42032) 2024-03-31T15:06:20.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:20.395+0000 7f883aa00640 1 -- 172.21.15.16:0/2580342761 learned_addr learned my addr 172.21.15.16:0/2580342761 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:06:20.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:20.395+0000 7f8833e00640 1 --1- 172.21.15.16:0/2580342761 >> v1:172.21.15.89:6789/0 conn(0x7f883c13e3d0 0x7f883c0cc9d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.16:51626/0 (socket says 172.21.15.16:51626) 2024-03-31T15:06:20.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:20.395+0000 7f883b400640 1 -- 172.21.15.16:0/2580342761 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f883c13dff0 msgr2=0x7f883c06d4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:06:20.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:20.395+0000 7f883b400640 1 --2- 172.21.15.16:0/2580342761 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f883c13dff0 0x7f883c06d4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:20.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:20.395+0000 7f883b400640 1 -- 172.21.15.16:0/2580342761 >> v1:172.21.15.89:6789/0 conn(0x7f883c13e3d0 legacy=0x7f883c0cc9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:20.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:20.395+0000 7f883b400640 1 -- 172.21.15.16:0/2580342761 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f883c15f0e0 con 0x7f883c0cd0e0 2024-03-31T15:06:20.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:20.395+0000 7f883aa00640 1 --2- 172.21.15.16:0/2580342761 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f883c13dff0 0x7f883c06d4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:06:20.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:20.395+0000 7f883b400640 1 --2- 172.21.15.16:0/2580342761 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f883c0cd0e0 0x7f883c06da10 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f8834009d90 tx=0x7f883401c8e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9d14706e5d73781c server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:06:20.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:20.399+0000 7f883a000640 1 -- 172.21.15.16:0/2580342761 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88340b2070 con 0x7f883c0cd0e0 2024-03-31T15:06:20.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:20.399+0000 7f883a000640 1 -- 172.21.15.16:0/2580342761 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f88340a58b0 con 0x7f883c0cd0e0 2024-03-31T15:06:20.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:20.399+0000 7f883a000640 1 -- 172.21.15.16:0/2580342761 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88340a5c80 con 0x7f883c0cd0e0 2024-03-31T15:06:20.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:20.399+0000 7f8841c00640 1 -- 172.21.15.16:0/2580342761 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f883c0cd0e0 msgr2=0x7f883c06da10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:20.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:20.399+0000 7f8841c00640 1 --2- 172.21.15.16:0/2580342761 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f883c0cd0e0 0x7f883c06da10 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f8834009d90 tx=0x7f883401c8e0 comp rx=0 tx=0).stop 2024-03-31T15:06:20.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:20.399+0000 7f8841c00640 1 -- 172.21.15.16:0/2580342761 shutdown_connections 2024-03-31T15:06:20.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:20.399+0000 7f8841c00640 1 --2- 172.21.15.16:0/2580342761 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f883c0cd0e0 0x7f883c06da10 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:20.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:20.399+0000 7f8841c00640 1 --2- 172.21.15.16:0/2580342761 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f883c13dff0 0x7f883c06d4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:20.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:20.399+0000 7f8841c00640 1 -- 172.21.15.16:0/2580342761 >> 172.21.15.16:0/2580342761 conn(0x7f883c06ee40 msgr2=0x7f883c06b620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:06:20.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:20.399+0000 7f8841c00640 1 -- 172.21.15.16:0/2580342761 shutdown_connections 2024-03-31T15:06:20.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:20.399+0000 7f8841c00640 1 -- 172.21.15.16:0/2580342761 wait complete. 2024-03-31T15:06:20.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:20.399+0000 7f8841c00640 1 Processor -- start 2024-03-31T15:06:20.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:20.399+0000 7f8841c00640 1 -- start start 2024-03-31T15:06:20.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:20.399+0000 7f8841c00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f883c0cd0e0 0x7f883c1fa5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:20.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:20.399+0000 7f8841c00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f883c13dff0 0x7f883c15ded0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:20.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:20.399+0000 7f8841c00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f883c13e3d0 0x7f883c15e410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:20.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:20.399+0000 7f8841c00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f883c0cf710 con 0x7f883c13dff0 2024-03-31T15:06:20.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:20.399+0000 7f8841c00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f883c0cf590 con 0x7f883c13e3d0 2024-03-31T15:06:20.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:20.399+0000 7f8841c00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f883c0cf890 con 0x7f883c0cd0e0 2024-03-31T15:06:20.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:20.399+0000 7f8833e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f883c13dff0 0x7f883c15ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:20.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:20.399+0000 7f8833e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f883c13dff0 0x7f883c15ded0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35834/0 (socket says 172.21.15.16:35834) 2024-03-31T15:06:20.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:20.399+0000 7f8833e00640 1 -- 172.21.15.16:0/3839305582 learned_addr learned my addr 172.21.15.16:0/3839305582 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:06:20.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:20.399+0000 7f883aa00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f883c0cd0e0 0x7f883c1fa5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:20.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:20.399+0000 7f883b400640 1 --2- 172.21.15.16:0/3839305582 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f883c13e3d0 0x7f883c15e410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:20.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:20.403+0000 7f8833e00640 1 -- 172.21.15.16:0/3839305582 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f883c0cd0e0 msgr2=0x7f883c1fa5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:20.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:20.403+0000 7f8833e00640 1 --2- 172.21.15.16:0/3839305582 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f883c0cd0e0 0x7f883c1fa5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:20.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:20.403+0000 7f8833e00640 1 -- 172.21.15.16:0/3839305582 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f883c13e3d0 msgr2=0x7f883c15e410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:20.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:20.403+0000 7f8833e00640 1 --2- 172.21.15.16:0/3839305582 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f883c13e3d0 0x7f883c15e410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:20.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:20.403+0000 7f8833e00640 1 -- 172.21.15.16:0/3839305582 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f883c15c2f0 con 0x7f883c13dff0 2024-03-31T15:06:20.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:20.403+0000 7f8833e00640 1 --2- 172.21.15.16:0/3839305582 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f883c13dff0 0x7f883c15ded0 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f882400c3a0 tx=0x7f882400c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:06:20.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:20.403+0000 7f8832a00640 1 -- 172.21.15.16:0/3839305582 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88240074e0 con 0x7f883c13dff0 2024-03-31T15:06:20.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:20.403+0000 7f8832a00640 1 -- 172.21.15.16:0/3839305582 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8824007680 con 0x7f883c13dff0 2024-03-31T15:06:20.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:20.403+0000 7f8841c00640 1 -- 172.21.15.16:0/3839305582 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f883c15c5d0 con 0x7f883c13dff0 2024-03-31T15:06:20.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:20.403+0000 7f8832a00640 1 -- 172.21.15.16:0/3839305582 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8824004a30 con 0x7f883c13dff0 2024-03-31T15:06:20.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:20.403+0000 7f8841c00640 1 -- 172.21.15.16:0/3839305582 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f883c15cb90 con 0x7f883c13dff0 2024-03-31T15:06:20.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:20.403+0000 7f8841c00640 1 -- 172.21.15.16:0/3839305582 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f883c069950 con 0x7f883c13dff0 2024-03-31T15:06:20.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:20.403+0000 7f8832a00640 1 -- 172.21.15.16:0/3839305582 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f882401a020 con 0x7f883c13dff0 2024-03-31T15:06:20.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:20.403+0000 7f8832a00640 1 --2- 172.21.15.16:0/3839305582 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f8818068380 0x7f881806a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:20.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:20.403+0000 7f8832a00640 1 -- 172.21.15.16:0/3839305582 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 4943+0+0 (secure 0 0 0) 0x7f8824085de0 con 0x7f883c13dff0 2024-03-31T15:06:20.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:20.403+0000 7f883aa00640 1 --2- 172.21.15.16:0/3839305582 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f8818068380 0x7f881806a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:20.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:20.407+0000 7f883aa00640 1 --2- 172.21.15.16:0/3839305582 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f8818068380 0x7f881806a840 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f882c006cb0 tx=0x7f882c009240 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:06:20.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:20.407+0000 7f8832a00640 1 -- 172.21.15.16:0/3839305582 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f8824055fb0 con 0x7f883c13dff0 2024-03-31T15:06:20.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:20.575+0000 7f8841c00640 1 -- 172.21.15.16:0/3839305582 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7f883c06e4e0 con 0x7f883c13dff0 2024-03-31T15:06:20.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:20.579+0000 7f8832a00640 1 -- 172.21.15.16:0/3839305582 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v14)=0 v14) ==== 122+0+0 (secure 0 0 0) 0x7f882405a2b0 con 0x7f883c13dff0 2024-03-31T15:06:20.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:20.579+0000 7f8841c00640 1 -- 172.21.15.16:0/3839305582 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f8818068380 msgr2=0x7f881806a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:20.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:20.579+0000 7f8841c00640 1 --2- 172.21.15.16:0/3839305582 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f8818068380 0x7f881806a840 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f882c006cb0 tx=0x7f882c009240 comp rx=0 tx=0).stop 2024-03-31T15:06:20.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:20.579+0000 7f8841c00640 1 -- 172.21.15.16:0/3839305582 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f883c13dff0 msgr2=0x7f883c15ded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:20.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:20.579+0000 7f8841c00640 1 --2- 172.21.15.16:0/3839305582 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f883c13dff0 0x7f883c15ded0 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f882400c3a0 tx=0x7f882400c860 comp rx=0 tx=0).stop 2024-03-31T15:06:20.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:20.579+0000 7f8841c00640 1 -- 172.21.15.16:0/3839305582 shutdown_connections 2024-03-31T15:06:20.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:20.579+0000 7f8841c00640 1 --2- 172.21.15.16:0/3839305582 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f8818068380 0x7f881806a840 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:20.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:20.579+0000 7f8841c00640 1 --2- 172.21.15.16:0/3839305582 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f883c13e3d0 0x7f883c15e410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:20.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:20.579+0000 7f8841c00640 1 --2- 172.21.15.16:0/3839305582 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f883c13dff0 0x7f883c15ded0 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:20.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:20.579+0000 7f8841c00640 1 --2- 172.21.15.16:0/3839305582 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f883c0cd0e0 0x7f883c1fa5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:20.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:20.579+0000 7f8841c00640 1 -- 172.21.15.16:0/3839305582 >> 172.21.15.16:0/3839305582 conn(0x7f883c06ee40 msgr2=0x7f883c0b3560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:06:20.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:20.579+0000 7f8841c00640 1 -- 172.21.15.16:0/3839305582 shutdown_connections 2024-03-31T15:06:20.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:20.579+0000 7f8841c00640 1 -- 172.21.15.16:0/3839305582 wait complete. 2024-03-31T15:06:20.593 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-31T15:06:20.594 DEBUG:teuthology.orchestra.run.smithi016:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-31T15:06:20.675 DEBUG:teuthology.orchestra.run.smithi089:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-31T15:06:20.757 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-31T15:06:20.758 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-03-31T15:06:20.758 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-31T15:06:20.767 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16357708k,nr_inodes=4089427,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275996k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=6539 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275992k,nr_inodes=818998,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275996k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-31T15:06:20.767 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-03-31T15:06:20.767 DEBUG:teuthology.orchestra.run.smithi089:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-31T15:06:20.776 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16354452k,nr_inodes=4088613,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275344k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=901 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275340k,nr_inodes=818835,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275344k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-31T15:06:20.776 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-03-31T15:06:20.776 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-31T15:06:20.816 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16357708k,nr_inodes=4089427,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275996k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=6539 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275992k,nr_inodes=818998,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275996k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-31T15:06:20.816 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-03-31T15:06:20.816 DEBUG:teuthology.orchestra.run.smithi089:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-31T15:06:20.826 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16354452k,nr_inodes=4088613,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275344k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=901 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275340k,nr_inodes=818835,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275344k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-31T15:06:20.826 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-03-31T15:06:20.826 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-31T15:06:20.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:21.003+0000 7f0608600640 1 Processor -- start 2024-03-31T15:06:21.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:21.003+0000 7f0608600640 1 -- start start 2024-03-31T15:06:21.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:21.007+0000 7f0608600640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f06000d0c50 0x7f0600072fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:21.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:21.007+0000 7f0608600640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f0600073510 0x7f060015eb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:21.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:21.007+0000 7f0608600640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f060006bd30 con 0x7f0600147e70 2024-03-31T15:06:21.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:21.007+0000 7f0608600640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f0600073980 con 0x7f06000d0c50 2024-03-31T15:06:21.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:21.007+0000 7f0608600640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f06001621b0 con 0x7f0600073510 2024-03-31T15:06:21.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:21.007+0000 7f0605200640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f06000d0c50 0x7f0600072fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:21.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:21.007+0000 7f0605c00640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f0600147e70 0x7f060006b620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:50648/0 (socket says 172.21.15.16:50648) 2024-03-31T15:06:21.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:21.007+0000 7f0605c00640 1 -- 172.21.15.16:0/3004150871 learned_addr learned my addr 172.21.15.16:0/3004150871 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:06:21.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:21.007+0000 7f0606600640 1 --2- 172.21.15.16:0/3004150871 >> v2:172.21.15.89:3300/0 conn(0x7f0600073510 0x7f060015eb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:21.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:21.007+0000 7f0605200640 1 -- 172.21.15.16:0/3004150871 >> v2:172.21.15.89:3300/0 conn(0x7f0600073510 msgr2=0x7f060015eb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:06:21.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:21.007+0000 7f0605200640 1 --2- 172.21.15.16:0/3004150871 >> v2:172.21.15.89:3300/0 conn(0x7f0600073510 0x7f060015eb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:21.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:21.007+0000 7f0605200640 1 -- 172.21.15.16:0/3004150871 >> v1:172.21.15.16:6789/0 conn(0x7f0600147e70 legacy=0x7f060006b620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:21.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:21.007+0000 7f0605200640 1 -- 172.21.15.16:0/3004150871 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0600162490 con 0x7f06000d0c50 2024-03-31T15:06:21.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:21.007+0000 7f0606600640 1 --2- 172.21.15.16:0/3004150871 >> v2:172.21.15.89:3300/0 conn(0x7f0600073510 0x7f060015eb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:06:21.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:21.011+0000 7f0605200640 1 --2- 172.21.15.16:0/3004150871 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f06000d0c50 0x7f0600072fd0 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f060006e280 tx=0x7f05ec01c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5109a599dc89063d server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:06:21.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:21.011+0000 7f05ffe00640 1 -- 172.21.15.16:0/3004150871 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05ec0b0070 con 0x7f06000d0c50 2024-03-31T15:06:21.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:21.011+0000 7f05ffe00640 1 -- 172.21.15.16:0/3004150871 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f05ec0ab600 con 0x7f06000d0c50 2024-03-31T15:06:21.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:21.011+0000 7f05ffe00640 1 -- 172.21.15.16:0/3004150871 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05ec0ab9d0 con 0x7f06000d0c50 2024-03-31T15:06:21.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:21.011+0000 7f0608600640 1 -- 172.21.15.16:0/3004150871 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f06000d0c50 msgr2=0x7f0600072fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:21.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:21.011+0000 7f0608600640 1 --2- 172.21.15.16:0/3004150871 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f06000d0c50 0x7f0600072fd0 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f060006e280 tx=0x7f05ec01c480 comp rx=0 tx=0).stop 2024-03-31T15:06:21.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:21.011+0000 7f0608600640 1 -- 172.21.15.16:0/3004150871 shutdown_connections 2024-03-31T15:06:21.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:21.011+0000 7f0608600640 1 --2- 172.21.15.16:0/3004150871 >> v2:172.21.15.89:3300/0 conn(0x7f0600073510 0x7f060015eb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:21.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:21.011+0000 7f0608600640 1 --2- 172.21.15.16:0/3004150871 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f06000d0c50 0x7f0600072fd0 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:21.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:21.011+0000 7f0608600640 1 -- 172.21.15.16:0/3004150871 >> 172.21.15.16:0/3004150871 conn(0x7f06000d1060 msgr2=0x7f06000d1470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:06:21.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:21.011+0000 7f0608600640 1 -- 172.21.15.16:0/3004150871 shutdown_connections 2024-03-31T15:06:21.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:21.011+0000 7f0608600640 1 -- 172.21.15.16:0/3004150871 wait complete. 2024-03-31T15:06:21.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:21.011+0000 7f0608600640 1 Processor -- start 2024-03-31T15:06:21.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:21.011+0000 7f0608600640 1 -- start start 2024-03-31T15:06:21.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:21.011+0000 7f0608600640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0600073510 0x7f060015d910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:21.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:21.011+0000 7f0608600640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f06000d0c50 0x7f060015bd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:21.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:21.011+0000 7f0608600640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0600147e70 0x7f060015c3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:21.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:21.015+0000 7f0608600640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f060015f870 con 0x7f0600073510 2024-03-31T15:06:21.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:21.015+0000 7f0608600640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f0600162f70 con 0x7f0600147e70 2024-03-31T15:06:21.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:21.015+0000 7f0608600640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f060015f9f0 con 0x7f06000d0c50 2024-03-31T15:06:21.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:21.015+0000 7f0605c00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0600073510 0x7f060015d910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:21.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:21.015+0000 7f0605c00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0600073510 0x7f060015d910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35858/0 (socket says 172.21.15.16:35858) 2024-03-31T15:06:21.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:21.015+0000 7f0605c00640 1 -- 172.21.15.16:0/581566002 learned_addr learned my addr 172.21.15.16:0/581566002 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:06:21.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:21.015+0000 7f0605200640 1 --2- 172.21.15.16:0/581566002 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f06000d0c50 0x7f060015bd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:21.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:21.015+0000 7f0606600640 1 --2- 172.21.15.16:0/581566002 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0600147e70 0x7f060015c3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:21.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:21.015+0000 7f0605c00640 1 -- 172.21.15.16:0/581566002 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f06000d0c50 msgr2=0x7f060015bd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:21.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:21.015+0000 7f0605c00640 1 --2- 172.21.15.16:0/581566002 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f06000d0c50 0x7f060015bd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:21.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:21.015+0000 7f0605c00640 1 -- 172.21.15.16:0/581566002 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0600147e70 msgr2=0x7f060015c3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:21.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:21.015+0000 7f0605c00640 1 --2- 172.21.15.16:0/581566002 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0600147e70 0x7f060015c3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:21.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:21.015+0000 7f0605c00640 1 -- 172.21.15.16:0/581566002 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06000b1850 con 0x7f0600073510 2024-03-31T15:06:21.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:21.015+0000 7f0605c00640 1 --2- 172.21.15.16:0/581566002 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0600073510 0x7f060015d910 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f05f400caa0 tx=0x7f05f400cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:06:21.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:21.015+0000 7f05fd600640 1 -- 172.21.15.16:0/581566002 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05f4017070 con 0x7f0600073510 2024-03-31T15:06:21.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:21.015+0000 7f05fd600640 1 -- 172.21.15.16:0/581566002 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f05f400ac00 con 0x7f0600073510 2024-03-31T15:06:21.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:21.015+0000 7f05fd600640 1 -- 172.21.15.16:0/581566002 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05f4012400 con 0x7f0600073510 2024-03-31T15:06:21.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:21.015+0000 7f0608600640 1 -- 172.21.15.16:0/581566002 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f06000b1b30 con 0x7f0600073510 2024-03-31T15:06:21.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:21.015+0000 7f0608600640 1 -- 172.21.15.16:0/581566002 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0600146b00 con 0x7f0600073510 2024-03-31T15:06:21.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:21.015+0000 7f0608600640 1 -- 172.21.15.16:0/581566002 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0600069950 con 0x7f0600073510 2024-03-31T15:06:21.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:21.015+0000 7f05fd600640 1 -- 172.21.15.16:0/581566002 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f05f401a070 con 0x7f0600073510 2024-03-31T15:06:21.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:21.019+0000 7f05fd600640 1 --2- 172.21.15.16:0/581566002 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f05e0068380 0x7f05e006a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:21.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:21.019+0000 7f05fd600640 1 -- 172.21.15.16:0/581566002 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 4943+0+0 (secure 0 0 0) 0x7f05f40863d0 con 0x7f0600073510 2024-03-31T15:06:21.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:21.019+0000 7f0605200640 1 --2- 172.21.15.16:0/581566002 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f05e0068380 0x7f05e006a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:21.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:21.019+0000 7f0605200640 1 --2- 172.21.15.16:0/581566002 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f05e0068380 0x7f05e006a840 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f05ec0ad000 tx=0x7f05ec008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:06:21.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:21.023+0000 7f05fd600640 1 -- 172.21.15.16:0/581566002 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f05f4056c80 con 0x7f0600073510 2024-03-31T15:06:21.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:21.227+0000 7f0608600640 1 -- 172.21.15.16:0/581566002 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f06000d1bf0 con 0x7f0600073510 2024-03-31T15:06:21.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:21.227+0000 7f05fd600640 1 -- 172.21.15.16:0/581566002 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+3863 (secure 0 0 0) 0x7f05f4056580 con 0x7f0600073510 2024-03-31T15:06:21.222 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:06:21.222 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":27,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/285985105","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6835","nonce":285985105},{"type":"v1","addr":"172.21.15.89:6837","nonce":285985105}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20},{"gid":5689,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6836/1615931247","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":1615931247},{"type":"v1","addr":"172.21.15.89:6836","nonce":1615931247}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20},{"gid":5694,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/512187852","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6833","nonce":512187852},{"type":"v1","addr":"172.21.15.16:6835","nonce":512187852}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20},{"gid":6060,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6834/2108510672","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":2108510672},{"type":"v1","addr":"172.21.15.16:6834","nonce":2108510672}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":27}],"filesystems":[]} 2024-03-31T15:06:21.222 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 27 2024-03-31T15:06:21.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:21.227+0000 7f0608600640 1 -- 172.21.15.16:0/581566002 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f05e0068380 msgr2=0x7f05e006a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:21.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:21.227+0000 7f0608600640 1 --2- 172.21.15.16:0/581566002 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f05e0068380 0x7f05e006a840 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f05ec0ad000 tx=0x7f05ec008040 comp rx=0 tx=0).stop 2024-03-31T15:06:21.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:21.227+0000 7f0608600640 1 -- 172.21.15.16:0/581566002 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0600073510 msgr2=0x7f060015d910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:21.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:21.227+0000 7f0608600640 1 --2- 172.21.15.16:0/581566002 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0600073510 0x7f060015d910 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f05f400caa0 tx=0x7f05f400cf60 comp rx=0 tx=0).stop 2024-03-31T15:06:21.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:21.227+0000 7f0608600640 1 -- 172.21.15.16:0/581566002 shutdown_connections 2024-03-31T15:06:21.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:21.227+0000 7f0608600640 1 --2- 172.21.15.16:0/581566002 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f05e0068380 0x7f05e006a840 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:21.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:21.227+0000 7f0608600640 1 --2- 172.21.15.16:0/581566002 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0600147e70 0x7f060015c3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:21.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:21.227+0000 7f0608600640 1 --2- 172.21.15.16:0/581566002 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f06000d0c50 0x7f060015bd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:21.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:21.227+0000 7f0608600640 1 --2- 172.21.15.16:0/581566002 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0600073510 0x7f060015d910 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:21.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:21.227+0000 7f0608600640 1 -- 172.21.15.16:0/581566002 >> 172.21.15.16:0/581566002 conn(0x7f06000d1060 msgr2=0x7f060006fb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:06:21.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:21.227+0000 7f0608600640 1 -- 172.21.15.16:0/581566002 shutdown_connections 2024-03-31T15:06:21.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:21.227+0000 7f0608600640 1 -- 172.21.15.16:0/581566002 wait complete. 2024-03-31T15:06:21.243 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-31T15:06:21.244 INFO:tasks.ceph.mds.a:Stopping old one... 2024-03-31T15:06:21.244 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-31T15:06:21.244 INFO:teuthology.orchestra.run:waiting for 300 2024-03-31T15:06:21.244 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-31T15:06:21.244 INFO:tasks.ceph.mds.c:Stopping old one... 2024-03-31T15:06:21.244 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-31T15:06:21.245 INFO:teuthology.orchestra.run:waiting for 300 2024-03-31T15:06:21.245 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-31T15:06:21.245 INFO:tasks.ceph.mds.b:Stopping old one... 2024-03-31T15:06:21.245 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-31T15:06:21.245 INFO:teuthology.orchestra.run:waiting for 300 2024-03-31T15:06:21.245 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-31T15:06:21.245 INFO:tasks.ceph.mds.d:Stopping old one... 2024-03-31T15:06:21.246 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-31T15:06:21.246 INFO:teuthology.orchestra.run:waiting for 300 2024-03-31T15:06:21.257 INFO:tasks.ceph.mds.b:Stopped 2024-03-31T15:06:21.257 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-31T15:06:21.258 INFO:tasks.ceph.mds.d:Stopped 2024-03-31T15:06:21.258 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-31T15:06:21.259 INFO:tasks.ceph.mds.a:Stopped 2024-03-31T15:06:21.260 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-31T15:06:21.260 INFO:tasks.ceph.mds.c:Stopped 2024-03-31T15:06:21.260 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-31T15:06:21.301 INFO:tasks.ceph.mds.b:Started 2024-03-31T15:06:21.301 DEBUG:teuthology.parallel:result is None 2024-03-31T15:06:21.303 INFO:tasks.ceph.mds.d:Started 2024-03-31T15:06:21.303 DEBUG:teuthology.parallel:result is None 2024-03-31T15:06:21.310 INFO:tasks.ceph.mds.a:Started 2024-03-31T15:06:21.311 DEBUG:teuthology.parallel:result is None 2024-03-31T15:06:21.312 INFO:tasks.ceph.mds.c:Started 2024-03-31T15:06:21.312 DEBUG:teuthology.parallel:result is None 2024-03-31T15:06:21.312 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-03-31T15:06:21.385 INFO:tasks.ceph.mds.b.smithi089.stdout:starting mds.b at 2024-03-31T15:06:21.385 INFO:tasks.ceph.mds.d.smithi089.stdout:starting mds.d at 2024-03-31T15:06:21.397 INFO:tasks.ceph.mds.a.smithi016.stdout:starting mds.a at 2024-03-31T15:06:21.399 INFO:tasks.ceph.mds.c.smithi016.stdout:starting mds.c at 2024-03-31T15:06:21.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:21.463+0000 7f13ac400640 1 Processor -- start 2024-03-31T15:06:21.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:21.463+0000 7f13ac400640 1 -- start start 2024-03-31T15:06:21.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:21.467+0000 7f13ac400640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f13a415b3d0 0x7f13a40b1cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:21.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:21.467+0000 7f13ac400640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f13a40ce2f0 0x7f13a40b2200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:21.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:21.467+0000 7f13ac400640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f13a415f3c0 con 0x7f13a415eee0 2024-03-31T15:06:21.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:21.467+0000 7f13ac400640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f13a415b7b0 con 0x7f13a415b3d0 2024-03-31T15:06:21.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:21.467+0000 7f13ac400640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f13a415f6a0 con 0x7f13a40ce2f0 2024-03-31T15:06:21.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:21.467+0000 7f13a9a00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f13a415b3d0 0x7f13a40b1cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:21.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:21.467+0000 7f13a9a00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f13a415b3d0 0x7f13a40b1cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35878/0 (socket says 172.21.15.16:35878) 2024-03-31T15:06:21.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:21.467+0000 7f13a9a00640 1 -- 172.21.15.16:0/3882331504 learned_addr learned my addr 172.21.15.16:0/3882331504 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:06:21.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:21.467+0000 7f13aa400640 1 --2- 172.21.15.16:0/3882331504 >> v2:172.21.15.89:3300/0 conn(0x7f13a40ce2f0 0x7f13a40b2200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:21.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:21.467+0000 7f13a9a00640 1 -- 172.21.15.16:0/3882331504 >> v2:172.21.15.89:3300/0 conn(0x7f13a40ce2f0 msgr2=0x7f13a40b2200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:06:21.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:21.467+0000 7f13a9a00640 1 --2- 172.21.15.16:0/3882331504 >> v2:172.21.15.89:3300/0 conn(0x7f13a40ce2f0 0x7f13a40b2200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:21.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:21.467+0000 7f13a9a00640 1 -- 172.21.15.16:0/3882331504 >> v1:172.21.15.89:6789/0 conn(0x7f13a415eee0 legacy=0x7f13a40cc9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:21.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:21.467+0000 7f13a9a00640 1 -- 172.21.15.16:0/3882331504 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13a415f980 con 0x7f13a415b3d0 2024-03-31T15:06:21.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:21.471+0000 7f13a9a00640 1 --2- 172.21.15.16:0/3882331504 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f13a415b3d0 0x7f13a40b1cc0 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f13a406f8a0 tx=0x7f139801c4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee8ab8f5f0c31fee server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:06:21.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:21.471+0000 7f13a3e00640 1 -- 172.21.15.16:0/3882331504 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13980aa610 con 0x7f13a415b3d0 2024-03-31T15:06:21.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:21.471+0000 7f13a3e00640 1 -- 172.21.15.16:0/3882331504 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f13980af070 con 0x7f13a415b3d0 2024-03-31T15:06:21.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:21.471+0000 7f13a3e00640 1 -- 172.21.15.16:0/3882331504 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13980aa9d0 con 0x7f13a415b3d0 2024-03-31T15:06:21.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:21.471+0000 7f13ac400640 1 -- 172.21.15.16:0/3882331504 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f13a415b3d0 msgr2=0x7f13a40b1cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:21.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:21.471+0000 7f13ac400640 1 --2- 172.21.15.16:0/3882331504 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f13a415b3d0 0x7f13a40b1cc0 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f13a406f8a0 tx=0x7f139801c4f0 comp rx=0 tx=0).stop 2024-03-31T15:06:21.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:21.471+0000 7f13ac400640 1 -- 172.21.15.16:0/3882331504 shutdown_connections 2024-03-31T15:06:21.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:21.471+0000 7f13ac400640 1 --2- 172.21.15.16:0/3882331504 >> v2:172.21.15.89:3300/0 conn(0x7f13a40ce2f0 0x7f13a40b2200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:21.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:21.471+0000 7f13ac400640 1 --2- 172.21.15.16:0/3882331504 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f13a415b3d0 0x7f13a40b1cc0 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:21.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:21.471+0000 7f13ac400640 1 -- 172.21.15.16:0/3882331504 >> 172.21.15.16:0/3882331504 conn(0x7f13a406b620 msgr2=0x7f13a406ba30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:06:21.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:21.471+0000 7f13ac400640 1 -- 172.21.15.16:0/3882331504 shutdown_connections 2024-03-31T15:06:21.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:21.471+0000 7f13ac400640 1 -- 172.21.15.16:0/3882331504 wait complete. 2024-03-31T15:06:21.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:21.471+0000 7f13ac400640 1 Processor -- start 2024-03-31T15:06:21.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:21.471+0000 7f13ac400640 1 -- start start 2024-03-31T15:06:21.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:21.471+0000 7f13ac400640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f13a40ce2f0 0x7f13a40b0d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:21.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:21.471+0000 7f13ac400640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f13a415b3d0 0x7f13a4149890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:21.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:21.475+0000 7f13ac400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f13a415eee0 0x7f13a4071d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:21.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:21.475+0000 7f13ac400640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f13a4168610 con 0x7f13a415eee0 2024-03-31T15:06:21.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:21.475+0000 7f13ac400640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f13a4168490 con 0x7f13a415b3d0 2024-03-31T15:06:21.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:21.475+0000 7f13ac400640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f13a4168790 con 0x7f13a40ce2f0 2024-03-31T15:06:21.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:21.475+0000 7f13aa400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f13a415eee0 0x7f13a4071d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:21.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:21.475+0000 7f13aa400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f13a415eee0 0x7f13a4071d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35880/0 (socket says 172.21.15.16:35880) 2024-03-31T15:06:21.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:21.475+0000 7f13aa400640 1 -- 172.21.15.16:0/1462505012 learned_addr learned my addr 172.21.15.16:0/1462505012 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:06:21.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:21.475+0000 7f13a9a00640 1 --2- 172.21.15.16:0/1462505012 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f13a40ce2f0 0x7f13a40b0d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:21.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:21.475+0000 7f13a9000640 1 --2- 172.21.15.16:0/1462505012 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f13a415b3d0 0x7f13a4149890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:21.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:21.475+0000 7f13aa400640 1 -- 172.21.15.16:0/1462505012 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f13a40ce2f0 msgr2=0x7f13a40b0d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:21.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:21.475+0000 7f13aa400640 1 --2- 172.21.15.16:0/1462505012 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f13a40ce2f0 0x7f13a40b0d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:21.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:21.475+0000 7f13aa400640 1 -- 172.21.15.16:0/1462505012 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f13a415b3d0 msgr2=0x7f13a4149890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:21.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:21.475+0000 7f13aa400640 1 --2- 172.21.15.16:0/1462505012 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f13a415b3d0 0x7f13a4149890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:21.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:21.475+0000 7f13aa400640 1 -- 172.21.15.16:0/1462505012 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13a414a5b0 con 0x7f13a415eee0 2024-03-31T15:06:21.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:21.475+0000 7f13aa400640 1 --2- 172.21.15.16:0/1462505012 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f13a415eee0 0x7f13a4071d10 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f139c00efc0 tx=0x7f139c00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:06:21.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:21.475+0000 7f13a1600640 1 -- 172.21.15.16:0/1462505012 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f139c018020 con 0x7f13a415eee0 2024-03-31T15:06:21.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:21.475+0000 7f13a1600640 1 -- 172.21.15.16:0/1462505012 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f139c014480 con 0x7f13a415eee0 2024-03-31T15:06:21.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:21.475+0000 7f13ac400640 1 -- 172.21.15.16:0/1462505012 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f13a414a890 con 0x7f13a415eee0 2024-03-31T15:06:21.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:21.475+0000 7f13a1600640 1 -- 172.21.15.16:0/1462505012 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f139c014870 con 0x7f13a415eee0 2024-03-31T15:06:21.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:21.475+0000 7f13ac400640 1 -- 172.21.15.16:0/1462505012 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f13a414ae50 con 0x7f13a415eee0 2024-03-31T15:06:21.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:21.475+0000 7f13ac400640 1 -- 172.21.15.16:0/1462505012 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f13a4069950 con 0x7f13a415eee0 2024-03-31T15:06:21.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:21.475+0000 7f13a1600640 1 -- 172.21.15.16:0/1462505012 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f139c01b020 con 0x7f13a415eee0 2024-03-31T15:06:21.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:21.479+0000 7f13a1600640 1 --2- 172.21.15.16:0/1462505012 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f13840683b0 0x7f138406a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:21.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:21.479+0000 7f13a1600640 1 -- 172.21.15.16:0/1462505012 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 4943+0+0 (secure 0 0 0) 0x7f139c0870f0 con 0x7f13a415eee0 2024-03-31T15:06:21.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:21.479+0000 7f13a9a00640 1 --2- 172.21.15.16:0/1462505012 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f13840683b0 0x7f138406a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:21.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:21.479+0000 7f13a9a00640 1 --2- 172.21.15.16:0/1462505012 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f13840683b0 0x7f138406a870 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f13980ae000 tx=0x7f13980a7000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:06:21.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:21.483+0000 7f13a1600640 1 -- 172.21.15.16:0/1462505012 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f139c022070 con 0x7f13a415eee0 2024-03-31T15:06:21.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:21.659+0000 7f13ac400640 1 -- 172.21.15.16:0/1462505012 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) -- 0x7f13a40cdea0 con 0x7f13a415eee0 2024-03-31T15:06:22.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:22.231+0000 7f13a1600640 1 -- 172.21.15.16:0/1462505012 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v50) ==== 97+0+0 (secure 0 0 0) 0x7f139c010040 con 0x7f13a415eee0 2024-03-31T15:06:22.226 INFO:teuthology.orchestra.run.smithi016.stderr: removed all blocklist entries 2024-03-31T15:06:22.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:22.235+0000 7f13ac400640 1 -- 172.21.15.16:0/1462505012 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f13840683b0 msgr2=0x7f138406a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:22.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:22.235+0000 7f13ac400640 1 --2- 172.21.15.16:0/1462505012 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f13840683b0 0x7f138406a870 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f13980ae000 tx=0x7f13980a7000 comp rx=0 tx=0).stop 2024-03-31T15:06:22.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:22.235+0000 7f13ac400640 1 -- 172.21.15.16:0/1462505012 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f13a415eee0 msgr2=0x7f13a4071d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:22.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:22.235+0000 7f13ac400640 1 --2- 172.21.15.16:0/1462505012 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f13a415eee0 0x7f13a4071d10 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f139c00efc0 tx=0x7f139c00c490 comp rx=0 tx=0).stop 2024-03-31T15:06:22.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:22.235+0000 7f13ac400640 1 -- 172.21.15.16:0/1462505012 shutdown_connections 2024-03-31T15:06:22.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:22.235+0000 7f13ac400640 1 --2- 172.21.15.16:0/1462505012 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f13840683b0 0x7f138406a870 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:22.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:22.235+0000 7f13ac400640 1 --2- 172.21.15.16:0/1462505012 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f13a415eee0 0x7f13a4071d10 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:22.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:22.235+0000 7f13ac400640 1 --2- 172.21.15.16:0/1462505012 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f13a415b3d0 0x7f13a4149890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:22.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:22.235+0000 7f13ac400640 1 --2- 172.21.15.16:0/1462505012 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f13a40ce2f0 0x7f13a40b0d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:22.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:22.235+0000 7f13ac400640 1 -- 172.21.15.16:0/1462505012 >> 172.21.15.16:0/1462505012 conn(0x7f13a406b620 msgr2=0x7f13a40ce730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:06:22.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:22.235+0000 7f13ac400640 1 -- 172.21.15.16:0/1462505012 shutdown_connections 2024-03-31T15:06:22.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:22.235+0000 7f13ac400640 1 -- 172.21.15.16:0/1462505012 wait complete. 2024-03-31T15:06:22.251 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-03-31T15:06:22.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:22.399+0000 7f901da00640 1 Processor -- start 2024-03-31T15:06:22.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:22.399+0000 7f901da00640 1 -- start start 2024-03-31T15:06:22.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:22.399+0000 7f901da00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f90180cef40 0x7f90180b5b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:22.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:22.399+0000 7f901da00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f901815b4d0 0x7f90180cf320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:22.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:22.399+0000 7f901da00640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f90180ce8a0 con 0x7f90180b61e0 2024-03-31T15:06:22.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:22.399+0000 7f901da00640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f90180cf860 con 0x7f90180cef40 2024-03-31T15:06:22.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:22.399+0000 7f901da00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f90180709a0 con 0x7f901815b4d0 2024-03-31T15:06:22.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:22.399+0000 7f9016a00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f90180cef40 0x7f90180b5b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:22.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:22.399+0000 7f9017400640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f90180b61e0 0x7f901815ee10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:50656/0 (socket says 172.21.15.16:50656) 2024-03-31T15:06:22.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:22.399+0000 7f9017400640 1 -- 172.21.15.16:0/1312201474 learned_addr learned my addr 172.21.15.16:0/1312201474 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:06:22.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:22.399+0000 7f900fe00640 1 --2- 172.21.15.16:0/1312201474 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f901815b4d0 0x7f90180cf320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:22.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:22.399+0000 7f9016a00640 1 -- 172.21.15.16:0/1312201474 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f901815b4d0 msgr2=0x7f90180cf320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:06:22.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:22.399+0000 7f9016a00640 1 --2- 172.21.15.16:0/1312201474 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f901815b4d0 0x7f90180cf320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:22.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:22.399+0000 7f9016a00640 1 -- 172.21.15.16:0/1312201474 >> v1:172.21.15.16:6789/0 conn(0x7f90180b61e0 legacy=0x7f901815ee10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:22.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:22.399+0000 7f9016a00640 1 -- 172.21.15.16:0/1312201474 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f901806fbe0 con 0x7f90180cef40 2024-03-31T15:06:22.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:22.399+0000 7f9016000640 1 -- 172.21.15.16:0/1312201474 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 107115237 0 0) 0x7f90180ce8a0 con 0x7f90180b61e0 2024-03-31T15:06:22.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:22.399+0000 7f900fe00640 1 --2- 172.21.15.16:0/1312201474 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f901815b4d0 0x7f90180cf320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:06:22.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:22.403+0000 7f9016a00640 1 --2- 172.21.15.16:0/1312201474 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f90180cef40 0x7f90180b5b50 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f90080097c0 tx=0x7f900801c470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c9c854a1573a3901 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:06:22.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:22.403+0000 7f9016000640 1 -- 172.21.15.16:0/1312201474 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90080a7070 con 0x7f90180cef40 2024-03-31T15:06:22.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:22.403+0000 7f9016000640 1 -- 172.21.15.16:0/1312201474 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f90080b0070 con 0x7f90180cef40 2024-03-31T15:06:22.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:22.403+0000 7f9016000640 1 -- 172.21.15.16:0/1312201474 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90080ab890 con 0x7f90180cef40 2024-03-31T15:06:22.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:22.403+0000 7f901da00640 1 -- 172.21.15.16:0/1312201474 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f90180cef40 msgr2=0x7f90180b5b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:22.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:22.403+0000 7f901da00640 1 --2- 172.21.15.16:0/1312201474 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f90180cef40 0x7f90180b5b50 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f90080097c0 tx=0x7f900801c470 comp rx=0 tx=0).stop 2024-03-31T15:06:22.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:22.403+0000 7f901da00640 1 -- 172.21.15.16:0/1312201474 shutdown_connections 2024-03-31T15:06:22.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:22.403+0000 7f901da00640 1 --2- 172.21.15.16:0/1312201474 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f901815b4d0 0x7f90180cf320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:22.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:22.403+0000 7f901da00640 1 --2- 172.21.15.16:0/1312201474 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f90180cef40 0x7f90180b5b50 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:22.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:22.403+0000 7f901da00640 1 -- 172.21.15.16:0/1312201474 >> 172.21.15.16:0/1312201474 conn(0x7f90180d1060 msgr2=0x7f90180d1470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:06:22.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:22.403+0000 7f901da00640 1 -- 172.21.15.16:0/1312201474 shutdown_connections 2024-03-31T15:06:22.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:22.403+0000 7f901da00640 1 -- 172.21.15.16:0/1312201474 wait complete. 2024-03-31T15:06:22.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:22.403+0000 7f901da00640 1 Processor -- start 2024-03-31T15:06:22.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:22.407+0000 7f901da00640 1 -- start start 2024-03-31T15:06:22.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:22.407+0000 7f901da00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f90180b61e0 0x7f9018155fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:22.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:22.407+0000 7f901da00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f90180cef40 0x7f90181445e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:22.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:22.407+0000 7f901da00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f901815b4d0 0x7f9018144b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:22.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:22.407+0000 7f901da00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f90180ce640 con 0x7f90180cef40 2024-03-31T15:06:22.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:22.407+0000 7f901da00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f90180ce4c0 con 0x7f901815b4d0 2024-03-31T15:06:22.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:22.407+0000 7f901da00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f90180b52f0 con 0x7f90180b61e0 2024-03-31T15:06:22.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:22.407+0000 7f900fe00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f90180cef40 0x7f90181445e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:22.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:22.407+0000 7f900fe00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f90180cef40 0x7f90181445e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35898/0 (socket says 172.21.15.16:35898) 2024-03-31T15:06:22.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:22.407+0000 7f900fe00640 1 -- 172.21.15.16:0/1658748739 learned_addr learned my addr 172.21.15.16:0/1658748739 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:06:22.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:22.407+0000 7f9016a00640 1 --2- 172.21.15.16:0/1658748739 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f90180b61e0 0x7f9018155fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:22.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:22.407+0000 7f9017400640 1 --2- 172.21.15.16:0/1658748739 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f901815b4d0 0x7f9018144b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:22.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:22.407+0000 7f900fe00640 1 -- 172.21.15.16:0/1658748739 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f90180b61e0 msgr2=0x7f9018155fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:22.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:22.407+0000 7f900fe00640 1 --2- 172.21.15.16:0/1658748739 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f90180b61e0 0x7f9018155fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:22.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:22.407+0000 7f900fe00640 1 -- 172.21.15.16:0/1658748739 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f901815b4d0 msgr2=0x7f9018144b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:22.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:22.407+0000 7f900fe00640 1 --2- 172.21.15.16:0/1658748739 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f901815b4d0 0x7f9018144b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:22.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:22.407+0000 7f900fe00640 1 -- 172.21.15.16:0/1658748739 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9018073be0 con 0x7f90180cef40 2024-03-31T15:06:22.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:22.407+0000 7f9017400640 1 --2- 172.21.15.16:0/1658748739 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f901815b4d0 0x7f9018144b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:06:22.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:22.407+0000 7f900fe00640 1 --2- 172.21.15.16:0/1658748739 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f90180cef40 0x7f90181445e0 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f900000d800 tx=0x7f900000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:06:22.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:22.407+0000 7f900ea00640 1 -- 172.21.15.16:0/1658748739 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9000018020 con 0x7f90180cef40 2024-03-31T15:06:22.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:22.407+0000 7f900ea00640 1 -- 172.21.15.16:0/1658748739 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f90000096c0 con 0x7f90180cef40 2024-03-31T15:06:22.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:22.407+0000 7f900ea00640 1 -- 172.21.15.16:0/1658748739 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9000009ab0 con 0x7f90180cef40 2024-03-31T15:06:22.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:22.407+0000 7f901da00640 1 -- 172.21.15.16:0/1658748739 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f901815b8d0 con 0x7f90180cef40 2024-03-31T15:06:22.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:22.407+0000 7f901da00640 1 -- 172.21.15.16:0/1658748739 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f901815be90 con 0x7f90180cef40 2024-03-31T15:06:22.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:22.407+0000 7f901da00640 1 -- 172.21.15.16:0/1658748739 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9018069950 con 0x7f90180cef40 2024-03-31T15:06:22.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:22.411+0000 7f900ea00640 1 -- 172.21.15.16:0/1658748739 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f900001b020 con 0x7f90180cef40 2024-03-31T15:06:22.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:22.411+0000 7f900ea00640 1 --2- 172.21.15.16:0/1658748739 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f8ff4068380 0x7f8ff406a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:22.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:22.411+0000 7f900ea00640 1 -- 172.21.15.16:0/1658748739 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(52..52 src has 1..52) ==== 4857+0+0 (secure 0 0 0) 0x7f9000086bc0 con 0x7f90180cef40 2024-03-31T15:06:22.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:22.411+0000 7f9016a00640 1 --2- 172.21.15.16:0/1658748739 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f8ff4068380 0x7f8ff406a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:22.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:22.411+0000 7f9016a00640 1 --2- 172.21.15.16:0/1658748739 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f8ff4068380 0x7f8ff406a840 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f90080af000 tx=0x7f90080a7000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:06:22.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:22.415+0000 7f900ea00640 1 -- 172.21.15.16:0/1658748739 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f9000022020 con 0x7f90180cef40 2024-03-31T15:06:22.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:22.639+0000 7f901da00640 1 -- 172.21.15.16:0/1658748739 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) -- 0x7f9018070c00 con 0x7f90180cef40 2024-03-31T15:06:22.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:22.639+0000 7f900ea00640 1 -- 172.21.15.16:0/1658748739 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v14) ==== 80+0+5304 (secure 0 0 0) 0x7f90000574c0 con 0x7f90180cef40 2024-03-31T15:06:22.634 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:06:22.634 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-03-31T15:06:22.634 INFO:teuthology.orchestra.run.smithi016.stdout: "auth_dump": [ 2024-03-31T15:06:22.634 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-31T15:06:22.634 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "mds.a", 2024-03-31T15:06:22.634 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAOewlmhp1ANRAAYS9zo+p4tLU1YK0ySOZE/Q==", 2024-03-31T15:06:22.635 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2024-03-31T15:06:22.635 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2024-03-31T15:06:22.635 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow *", 2024-03-31T15:06:22.635 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow *", 2024-03-31T15:06:22.635 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2024-03-31T15:06:22.635 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-31T15:06:22.635 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-31T15:06:22.635 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-31T15:06:22.635 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "mds.b", 2024-03-31T15:06:22.635 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAPewlmXxupBhAAtpkt93aJOFqhc7GXLIlkBg==", 2024-03-31T15:06:22.635 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2024-03-31T15:06:22.635 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2024-03-31T15:06:22.635 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow *", 2024-03-31T15:06:22.635 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow *", 2024-03-31T15:06:22.636 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2024-03-31T15:06:22.636 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-31T15:06:22.636 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-31T15:06:22.636 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-31T15:06:22.636 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "mds.c", 2024-03-31T15:06:22.636 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAPewlmjaieARAA32teg02vE/Q8RtqLtnArMw==", 2024-03-31T15:06:22.636 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2024-03-31T15:06:22.636 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2024-03-31T15:06:22.636 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow *", 2024-03-31T15:06:22.636 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow *", 2024-03-31T15:06:22.636 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2024-03-31T15:06:22.636 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-31T15:06:22.636 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-31T15:06:22.636 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-31T15:06:22.636 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "mds.d", 2024-03-31T15:06:22.637 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAPewlmo+4PDhAAmJD0siZB3LjWkvdMq1d8aw==", 2024-03-31T15:06:22.637 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2024-03-31T15:06:22.637 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2024-03-31T15:06:22.637 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow *", 2024-03-31T15:06:22.637 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow *", 2024-03-31T15:06:22.637 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2024-03-31T15:06:22.637 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-31T15:06:22.637 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-31T15:06:22.637 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-31T15:06:22.637 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.0", 2024-03-31T15:06:22.637 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAQewlmVP6kLhAAwFr4jwQgz1N3oIUZuJvmzw==", 2024-03-31T15:06:22.637 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2024-03-31T15:06:22.637 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2024-03-31T15:06:22.637 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2024-03-31T15:06:22.637 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2024-03-31T15:06:22.638 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-31T15:06:22.638 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-31T15:06:22.638 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-31T15:06:22.638 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.1", 2024-03-31T15:06:22.638 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQATewlmBh0gNhAAR6LiuFypwA7d3SwKlWuI/Q==", 2024-03-31T15:06:22.638 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2024-03-31T15:06:22.638 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2024-03-31T15:06:22.638 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2024-03-31T15:06:22.638 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2024-03-31T15:06:22.638 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-31T15:06:22.638 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-31T15:06:22.638 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-31T15:06:22.638 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.2", 2024-03-31T15:06:22.639 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAXewlmtQwhARAACbzc7Woo9BoVFPfK3Pns+g==", 2024-03-31T15:06:22.639 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2024-03-31T15:06:22.639 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2024-03-31T15:06:22.639 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2024-03-31T15:06:22.639 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2024-03-31T15:06:22.639 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-31T15:06:22.639 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-31T15:06:22.639 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-31T15:06:22.639 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.3", 2024-03-31T15:06:22.639 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAaewlmmjWABhAA/hwvARNb51nd5vsnYVD/sg==", 2024-03-31T15:06:22.639 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2024-03-31T15:06:22.639 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2024-03-31T15:06:22.639 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2024-03-31T15:06:22.639 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2024-03-31T15:06:22.640 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-31T15:06:22.640 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-31T15:06:22.640 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-31T15:06:22.640 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.4", 2024-03-31T15:06:22.640 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAgewlmby07CxAAChxE/qPuPboIlg3fDxSOeA==", 2024-03-31T15:06:22.640 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2024-03-31T15:06:22.640 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2024-03-31T15:06:22.640 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2024-03-31T15:06:22.640 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2024-03-31T15:06:22.640 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-31T15:06:22.640 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-31T15:06:22.640 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-31T15:06:22.640 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.5", 2024-03-31T15:06:22.640 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAjewlmlD8yExAA9pgP0heBywrgqc39iU8rpQ==", 2024-03-31T15:06:22.641 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2024-03-31T15:06:22.641 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2024-03-31T15:06:22.641 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2024-03-31T15:06:22.641 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2024-03-31T15:06:22.641 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-31T15:06:22.641 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-31T15:06:22.641 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-31T15:06:22.641 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.6", 2024-03-31T15:06:22.641 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAmewlmzAsTGBAALMjVcQ08sIiTDFaBcq8L6A==", 2024-03-31T15:06:22.641 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2024-03-31T15:06:22.641 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2024-03-31T15:06:22.641 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2024-03-31T15:06:22.641 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2024-03-31T15:06:22.641 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-31T15:06:22.641 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-31T15:06:22.642 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-31T15:06:22.642 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.7", 2024-03-31T15:06:22.642 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQApewlmb+CEHRAAXE+iBumDsb2c4zH3C8f6YA==", 2024-03-31T15:06:22.642 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2024-03-31T15:06:22.642 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2024-03-31T15:06:22.642 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2024-03-31T15:06:22.642 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2024-03-31T15:06:22.642 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-31T15:06:22.642 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-31T15:06:22.642 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-31T15:06:22.642 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "client.0", 2024-03-31T15:06:22.642 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAPewlmlMIDEhAAF0Ll1HcrWHmH6xDKq2Qq9A==", 2024-03-31T15:06:22.642 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2024-03-31T15:06:22.643 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2024-03-31T15:06:22.643 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow r", 2024-03-31T15:06:22.643 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-03-31T15:06:22.643 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-31T15:06:22.643 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-31T15:06:22.643 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-31T15:06:22.643 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "client.1", 2024-03-31T15:06:22.643 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAPewlmdsvqGxAAaVdwWwri0IE/2sg7psilWw==", 2024-03-31T15:06:22.643 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2024-03-31T15:06:22.643 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2024-03-31T15:06:22.643 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow r", 2024-03-31T15:06:22.643 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow rw", 2024-03-31T15:06:22.643 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow rwx" 2024-03-31T15:06:22.644 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-31T15:06:22.644 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-31T15:06:22.644 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-31T15:06:22.644 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "client.2", 2024-03-31T15:06:22.644 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAPewlmQapPGBAAbwXSlCi+zyQ9118WyAJmrQ==", 2024-03-31T15:06:22.644 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2024-03-31T15:06:22.644 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2024-03-31T15:06:22.644 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow r", 2024-03-31T15:06:22.644 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow rw", 2024-03-31T15:06:22.644 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow rwx" 2024-03-31T15:06:22.644 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-31T15:06:22.644 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-31T15:06:22.645 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-31T15:06:22.645 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "client.3", 2024-03-31T15:06:22.645 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAPewlmCw7qIRAAEeIjF85JbyfJ6d3FtMOVhQ==", 2024-03-31T15:06:22.645 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2024-03-31T15:06:22.645 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2024-03-31T15:06:22.645 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow r", 2024-03-31T15:06:22.645 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow rw", 2024-03-31T15:06:22.645 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow rwx" 2024-03-31T15:06:22.645 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-31T15:06:22.645 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-31T15:06:22.645 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-31T15:06:22.645 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "client.admin", 2024-03-31T15:06:22.645 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAOewlmXyPZGRAAtZOheiYhzGxdeJGqqyGxTw==", 2024-03-31T15:06:22.646 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2024-03-31T15:06:22.646 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow *", 2024-03-31T15:06:22.646 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow *", 2024-03-31T15:06:22.646 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow *", 2024-03-31T15:06:22.646 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2024-03-31T15:06:22.646 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-31T15:06:22.646 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-31T15:06:22.646 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-31T15:06:22.646 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "mgr.x", 2024-03-31T15:06:22.646 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAOewlm7UkKMRAA8hqgeV86mTCkRcxdXNdWmw==", 2024-03-31T15:06:22.646 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2024-03-31T15:06:22.646 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow *", 2024-03-31T15:06:22.646 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile mgr", 2024-03-31T15:06:22.646 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2024-03-31T15:06:22.647 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-31T15:06:22.647 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-31T15:06:22.647 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-31T15:06:22.647 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "mgr.y", 2024-03-31T15:06:22.647 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAOewlmq0cpLRAAKTgh04tRaHrpMhxdwgvXYA==", 2024-03-31T15:06:22.647 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2024-03-31T15:06:22.647 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow *", 2024-03-31T15:06:22.647 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile mgr", 2024-03-31T15:06:22.647 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2024-03-31T15:06:22.647 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-31T15:06:22.647 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-31T15:06:22.647 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-03-31T15:06:22.647 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-03-31T15:06:22.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:22.639+0000 7f901da00640 1 -- 172.21.15.16:0/1658748739 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f8ff4068380 msgr2=0x7f8ff406a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:22.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:22.639+0000 7f901da00640 1 --2- 172.21.15.16:0/1658748739 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f8ff4068380 0x7f8ff406a840 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f90080af000 tx=0x7f90080a7000 comp rx=0 tx=0).stop 2024-03-31T15:06:22.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:22.639+0000 7f901da00640 1 -- 172.21.15.16:0/1658748739 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f90180cef40 msgr2=0x7f90181445e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:22.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:22.639+0000 7f901da00640 1 --2- 172.21.15.16:0/1658748739 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f90180cef40 0x7f90181445e0 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f900000d800 tx=0x7f900000dcc0 comp rx=0 tx=0).stop 2024-03-31T15:06:22.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:22.639+0000 7f901da00640 1 -- 172.21.15.16:0/1658748739 shutdown_connections 2024-03-31T15:06:22.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:22.639+0000 7f901da00640 1 --2- 172.21.15.16:0/1658748739 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f8ff4068380 0x7f8ff406a840 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:22.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:22.639+0000 7f901da00640 1 --2- 172.21.15.16:0/1658748739 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f901815b4d0 0x7f9018144b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:22.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:22.639+0000 7f901da00640 1 --2- 172.21.15.16:0/1658748739 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f90180cef40 0x7f90181445e0 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:22.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:22.639+0000 7f901da00640 1 --2- 172.21.15.16:0/1658748739 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f90180b61e0 0x7f9018155fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:22.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:22.639+0000 7f901da00640 1 -- 172.21.15.16:0/1658748739 >> 172.21.15.16:0/1658748739 conn(0x7f90180d1060 msgr2=0x7f9018165a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:06:22.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:22.639+0000 7f901da00640 1 -- 172.21.15.16:0/1658748739 shutdown_connections 2024-03-31T15:06:22.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:22.639+0000 7f901da00640 1 -- 172.21.15.16:0/1658748739 wait complete. 2024-03-31T15:06:22.655 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-31T15:06:22.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:22.799+0000 7f38cb800640 1 Processor -- start 2024-03-31T15:06:22.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:22.803+0000 7f38cb800640 1 -- start start 2024-03-31T15:06:22.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:22.803+0000 7f38cb800640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f38c40b23d0 0x7f38c415cd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:22.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:22.803+0000 7f38cb800640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f38c4160e70 con 0x7f38c40728b0 2024-03-31T15:06:22.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:22.803+0000 7f38cb800640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f38c41602f0 con 0x7f38c4145d10 2024-03-31T15:06:22.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:22.803+0000 7f38cb800640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f38c41605d0 con 0x7f38c40b23d0 2024-03-31T15:06:22.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:22.803+0000 7f38bbe00640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f38c40728b0 0x7f38c40b1cc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:50666/0 (socket says 172.21.15.16:50666) 2024-03-31T15:06:22.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:22.803+0000 7f38bbe00640 1 -- 172.21.15.16:0/3557982198 learned_addr learned my addr 172.21.15.16:0/3557982198 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:06:22.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:22.803+0000 7f38c9800640 1 --2- 172.21.15.16:0/3557982198 >> v2:172.21.15.89:3300/0 conn(0x7f38c40b23d0 0x7f38c415cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:22.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:22.803+0000 7f38c3e00640 1 -- 172.21.15.16:0/3557982198 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1357583659 0 0) 0x7f38c4160e70 con 0x7f38c40728b0 2024-03-31T15:06:22.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:22.803+0000 7f38c3e00640 1 -- 172.21.15.16:0/3557982198 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f38a8003530 con 0x7f38c40728b0 2024-03-31T15:06:22.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:22.803+0000 7f38c3e00640 1 -- 172.21.15.16:0/3557982198 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1009519232 0 0) 0x7f38a8003530 con 0x7f38c40728b0 2024-03-31T15:06:22.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:22.803+0000 7f38c3e00640 1 -- 172.21.15.16:0/3557982198 >> v2:172.21.15.89:3300/0 conn(0x7f38c40b23d0 msgr2=0x7f38c415cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:06:22.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:22.803+0000 7f38c3e00640 1 --2- 172.21.15.16:0/3557982198 >> v2:172.21.15.89:3300/0 conn(0x7f38c40b23d0 0x7f38c415cd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:22.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:22.803+0000 7f38c3e00640 1 -- 172.21.15.16:0/3557982198 >> v1:172.21.15.89:6789/0 conn(0x7f38c4145d10 legacy=0x7f38c40cf790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:22.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:22.803+0000 7f38c3e00640 1 -- 172.21.15.16:0/3557982198 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38c41608b0 con 0x7f38c40728b0 2024-03-31T15:06:22.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:22.803+0000 7f38c3e00640 1 -- 172.21.15.16:0/3557982198 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 214807623 0 0) 0x7f38ac001680 con 0x7f38c40728b0 2024-03-31T15:06:22.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:22.803+0000 7f38c3e00640 1 -- 172.21.15.16:0/3557982198 >> v1:172.21.15.16:6789/0 conn(0x7f38c40728b0 legacy=0x7f38c40b1cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:22.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:22.803+0000 7f38c3e00640 1 --2- 172.21.15.16:0/3557982198 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f38a80039c0 0x7f38a8003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:22.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:22.803+0000 7f38c3e00640 1 --2- 172.21.15.16:0/3557982198 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f38a8004810 0x7f38a8004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:22.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:22.803+0000 7f38c3e00640 1 --2- 172.21.15.16:0/3557982198 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f38a80051d0 0x7f38a800b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:22.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:22.803+0000 7f38c3e00640 1 -- 172.21.15.16:0/3557982198 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f38c41605d0 con 0x7f38a8004810 2024-03-31T15:06:22.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:22.803+0000 7f38c3e00640 1 -- 172.21.15.16:0/3557982198 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f38a800c070 con 0x7f38a80051d0 2024-03-31T15:06:22.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:22.803+0000 7f38c3e00640 1 -- 172.21.15.16:0/3557982198 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f38a800c270 con 0x7f38a80039c0 2024-03-31T15:06:22.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:22.803+0000 7f38bbe00640 1 --2- 172.21.15.16:0/3557982198 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f38a8004810 0x7f38a8004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:22.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:22.803+0000 7f38c8e00640 1 --2- 172.21.15.16:0/3557982198 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f38a80039c0 0x7f38a8003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:22.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:22.803+0000 7f38c9800640 1 --2- 172.21.15.16:0/3557982198 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f38a80051d0 0x7f38a800b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:22.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:22.803+0000 7f38bbe00640 1 -- 172.21.15.16:0/3557982198 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f38a80039c0 msgr2=0x7f38a8003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:06:22.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:22.803+0000 7f38bbe00640 1 --2- 172.21.15.16:0/3557982198 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f38a80039c0 0x7f38a8003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:22.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:22.803+0000 7f38bbe00640 1 -- 172.21.15.16:0/3557982198 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f38a80051d0 msgr2=0x7f38a800b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:06:22.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:22.803+0000 7f38bbe00640 1 --2- 172.21.15.16:0/3557982198 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f38a80051d0 0x7f38a800b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:22.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:22.803+0000 7f38bbe00640 1 -- 172.21.15.16:0/3557982198 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38a8003530 con 0x7f38a8004810 2024-03-31T15:06:22.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:22.803+0000 7f38c9800640 1 --2- 172.21.15.16:0/3557982198 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f38a80051d0 0x7f38a800b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:06:22.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:22.803+0000 7f38c9800640 1 -- 172.21.15.16:0/3557982198 reap_dead start 2024-03-31T15:06:22.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:22.807+0000 7f38bbe00640 1 --2- 172.21.15.16:0/3557982198 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f38a8004810 0x7f38a8004c90 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f38ac00c240 tx=0x7f38ac01f3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e0ada0bfcb8f500 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:06:22.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:22.807+0000 7f38c3e00640 1 -- 172.21.15.16:0/3557982198 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38ac007d20 con 0x7f38a8004810 2024-03-31T15:06:22.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:22.807+0000 7f38c3e00640 1 -- 172.21.15.16:0/3557982198 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f38ac005ae0 con 0x7f38a8004810 2024-03-31T15:06:22.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:22.807+0000 7f38c3e00640 1 -- 172.21.15.16:0/3557982198 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38ac007070 con 0x7f38a8004810 2024-03-31T15:06:22.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:22.807+0000 7f38cb800640 1 -- 172.21.15.16:0/3557982198 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f38a8004810 msgr2=0x7f38a8004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:22.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:22.807+0000 7f38cb800640 1 --2- 172.21.15.16:0/3557982198 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f38a8004810 0x7f38a8004c90 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f38ac00c240 tx=0x7f38ac01f3b0 comp rx=0 tx=0).stop 2024-03-31T15:06:22.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:22.807+0000 7f38cb800640 1 -- 172.21.15.16:0/3557982198 shutdown_connections 2024-03-31T15:06:22.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:22.807+0000 7f38cb800640 1 --2- 172.21.15.16:0/3557982198 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f38a8004810 0x7f38a8004c90 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:22.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:22.807+0000 7f38cb800640 1 -- 172.21.15.16:0/3557982198 >> 172.21.15.16:0/3557982198 conn(0x7f38c406b620 msgr2=0x7f38c406ba30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:06:22.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:22.807+0000 7f38cb800640 1 -- 172.21.15.16:0/3557982198 shutdown_connections 2024-03-31T15:06:22.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:22.807+0000 7f38cb800640 1 -- 172.21.15.16:0/3557982198 wait complete. 2024-03-31T15:06:22.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:22.811+0000 7f38cb800640 1 Processor -- start 2024-03-31T15:06:22.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:22.811+0000 7f38cb800640 1 -- start start 2024-03-31T15:06:22.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:22.811+0000 7f38cb800640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f38a8004810 0x7f38c40b3b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:22.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:22.811+0000 7f38cb800640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f38c4140e80 0x7f38c413b350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:22.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:22.811+0000 7f38cb800640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f38c41412e0 0x7f38c413b890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:22.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:22.811+0000 7f38cb800640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f38c40cf350 con 0x7f38a8004810 2024-03-31T15:06:22.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:22.811+0000 7f38cb800640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f38c415ff80 con 0x7f38c41412e0 2024-03-31T15:06:22.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:22.811+0000 7f38cb800640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f38c40cf4d0 con 0x7f38c4140e80 2024-03-31T15:06:22.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:22.811+0000 7f38c8e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f38a8004810 0x7f38c40b3b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:22.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:22.811+0000 7f38c8e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f38a8004810 0x7f38c40b3b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35930/0 (socket says 172.21.15.16:35930) 2024-03-31T15:06:22.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:22.811+0000 7f38c8e00640 1 -- 172.21.15.16:0/2563095302 learned_addr learned my addr 172.21.15.16:0/2563095302 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:06:22.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:22.811+0000 7f38bbe00640 1 --2- 172.21.15.16:0/2563095302 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f38c4140e80 0x7f38c413b350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:22.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:22.811+0000 7f38c9800640 1 --2- 172.21.15.16:0/2563095302 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f38c41412e0 0x7f38c413b890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:22.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:22.811+0000 7f38c8e00640 1 -- 172.21.15.16:0/2563095302 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f38c4140e80 msgr2=0x7f38c413b350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:22.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:22.811+0000 7f38c8e00640 1 --2- 172.21.15.16:0/2563095302 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f38c4140e80 0x7f38c413b350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:22.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:22.811+0000 7f38c8e00640 1 -- 172.21.15.16:0/2563095302 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f38c41412e0 msgr2=0x7f38c413b890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:22.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:22.811+0000 7f38c8e00640 1 --2- 172.21.15.16:0/2563095302 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f38c41412e0 0x7f38c413b890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:22.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:22.811+0000 7f38c8e00640 1 -- 172.21.15.16:0/2563095302 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38c415fa70 con 0x7f38a8004810 2024-03-31T15:06:22.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:22.811+0000 7f38c9800640 1 --2- 172.21.15.16:0/2563095302 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f38c41412e0 0x7f38c413b890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:06:22.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:22.811+0000 7f38c8e00640 1 --2- 172.21.15.16:0/2563095302 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f38a8004810 0x7f38c40b3b40 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f38b400f7a0 tx=0x7f38b400fc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:06:22.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:22.811+0000 7f38c1600640 1 -- 172.21.15.16:0/2563095302 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38b40050d0 con 0x7f38a8004810 2024-03-31T15:06:22.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:22.811+0000 7f38c1600640 1 -- 172.21.15.16:0/2563095302 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f38b4018070 con 0x7f38a8004810 2024-03-31T15:06:22.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:22.811+0000 7f38cb800640 1 -- 172.21.15.16:0/2563095302 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f38c415e9d0 con 0x7f38a8004810 2024-03-31T15:06:22.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:22.811+0000 7f38c1600640 1 -- 172.21.15.16:0/2563095302 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38b4005490 con 0x7f38a8004810 2024-03-31T15:06:22.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:22.811+0000 7f38cb800640 1 -- 172.21.15.16:0/2563095302 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f38c415ef90 con 0x7f38a8004810 2024-03-31T15:06:22.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:22.815+0000 7f38cb800640 1 -- 172.21.15.16:0/2563095302 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f38c4069950 con 0x7f38a8004810 2024-03-31T15:06:22.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:22.815+0000 7f38c1600640 1 -- 172.21.15.16:0/2563095302 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f38b400d730 con 0x7f38a8004810 2024-03-31T15:06:22.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:22.815+0000 7f38c1600640 1 --2- 172.21.15.16:0/2563095302 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f38a0068520 0x7f38a006a9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:22.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:22.815+0000 7f38c1600640 1 -- 172.21.15.16:0/2563095302 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(52..52 src has 1..52) ==== 4857+0+0 (secure 0 0 0) 0x7f38b4086470 con 0x7f38a8004810 2024-03-31T15:06:22.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:22.815+0000 7f38bbe00640 1 --2- 172.21.15.16:0/2563095302 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f38a0068520 0x7f38a006a9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:22.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:22.815+0000 7f38bbe00640 1 --2- 172.21.15.16:0/2563095302 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f38a0068520 0x7f38a006a9e0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f38ac0af000 tx=0x7f38ac0aa000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:06:22.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:22.819+0000 7f38c1600640 1 -- 172.21.15.16:0/2563095302 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f38b4056ce0 con 0x7f38a8004810 2024-03-31T15:06:22.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:22.983+0000 7f38cb800640 1 -- 172.21.15.16:0/2563095302 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f38c40b06f0 con 0x7f38a8004810 2024-03-31T15:06:22.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:22.983+0000 7f38c1600640 1 -- 172.21.15.16:0/2563095302 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v52) ==== 74+0+12819 (secure 0 0 0) 0x7f38b40565e0 con 0x7f38a8004810 2024-03-31T15:06:22.980 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:06:22.980 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":52,"fsid":"286eb5da-fb47-4550-b290-d9519c13b9a5","created":"2024-03-31T15:03:11.438952+0000","modified":"2024-03-31T15:06:22.242109+0000","last_up_change":"2024-03-31T15:03:22.885535+0000","last_in_change":"2024-03-31T15:03:13.941667+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-31T15:03:22.550787+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"92a66450-ef19-40d8-ac3d-d760b23b0bda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6801","nonce":932712244}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6803","nonce":932712244}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6807","nonce":932712244}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6805","nonce":932712244}]},"public_addr":"172.21.15.16:6801/932712244","cluster_addr":"172.21.15.16:6803/932712244","heartbeat_back_addr":"172.21.15.16:6807/932712244","heartbeat_front_addr":"172.21.15.16:6805/932712244","state":["exists","up"]},{"osd":1,"uuid":"da9d1f2b-1968-4959-b843-faa231988e92","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6810","nonce":1398737533}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6811","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6813","nonce":1398737533}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6821","nonce":1398737533}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6815","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6816","nonce":1398737533}]},"public_addr":"172.21.15.16:6810/1398737533","cluster_addr":"172.21.15.16:6813/1398737533","heartbeat_back_addr":"172.21.15.16:6821/1398737533","heartbeat_front_addr":"172.21.15.16:6816/1398737533","state":["exists","up"]},{"osd":2,"uuid":"01a73662-9a21-4b1f-bcbb-42b9636a4638","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6819","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6822","nonce":3018526646}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6825","nonce":3018526646}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6831","nonce":3018526646}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6827","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6829","nonce":3018526646}]},"public_addr":"172.21.15.16:6822/3018526646","cluster_addr":"172.21.15.16:6825/3018526646","heartbeat_back_addr":"172.21.15.16:6831/3018526646","heartbeat_front_addr":"172.21.15.16:6829/3018526646","state":["exists","up"]},{"osd":3,"uuid":"c7771fd3-e18e-4500-ac38-23cf10ebcd75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6809","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6812","nonce":3360352974}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6817","nonce":3360352974}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6828","nonce":3360352974}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6823","nonce":3360352974}]},"public_addr":"172.21.15.16:6812/3360352974","cluster_addr":"172.21.15.16:6817/3360352974","heartbeat_back_addr":"172.21.15.16:6828/3360352974","heartbeat_front_addr":"172.21.15.16:6823/3360352974","state":["exists","up"]},{"osd":4,"uuid":"88260730-1482-4fbd-a40f-cb1124e9df4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6801","nonce":1856844856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6803","nonce":1856844856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6807","nonce":1856844856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6805","nonce":1856844856}]},"public_addr":"172.21.15.89:6801/1856844856","cluster_addr":"172.21.15.89:6803/1856844856","heartbeat_back_addr":"172.21.15.89:6807/1856844856","heartbeat_front_addr":"172.21.15.89:6805/1856844856","state":["exists","up"]},{"osd":5,"uuid":"4cb2c00c-fae0-4dd8-9d27-ec4f0093ccf7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6814","nonce":50395527}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6817","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6820","nonce":50395527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6831","nonce":50395527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6823","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6826","nonce":50395527}]},"public_addr":"172.21.15.89:6814/50395527","cluster_addr":"172.21.15.89:6820/50395527","heartbeat_back_addr":"172.21.15.89:6831/50395527","heartbeat_front_addr":"172.21.15.89:6826/50395527","state":["exists","up"]},{"osd":6,"uuid":"81ce0c19-2554-4cae-acbe-66a05e9e9b9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6809","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6812","nonce":3470471752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6815","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6818","nonce":3470471752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6827","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6829","nonce":3470471752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6821","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6824","nonce":3470471752}]},"public_addr":"172.21.15.89:6812/3470471752","cluster_addr":"172.21.15.89:6818/3470471752","heartbeat_back_addr":"172.21.15.89:6829/3470471752","heartbeat_front_addr":"172.21.15.89:6824/3470471752","state":["exists","up"]},{"osd":7,"uuid":"85e4313b-d37d-442e-8042-153bf20a847d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6811","nonce":1796778630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6813","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6816","nonce":1796778630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6825","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6828","nonce":1796778630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6819","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6822","nonce":1796778630}]},"public_addr":"172.21.15.89:6811/1796778630","cluster_addr":"172.21.15.89:6816/1796778630","heartbeat_back_addr":"172.21.15.89:6828/1796778630","heartbeat_front_addr":"172.21.15.89:6822/1796778630","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.439088+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.529970+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.680512+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:21.044161+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.582111+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.536558+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.409522+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.590229+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-31T15:06:22.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:22.987+0000 7f38cb800640 1 -- 172.21.15.16:0/2563095302 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f38a0068520 msgr2=0x7f38a006a9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:22.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:22.987+0000 7f38cb800640 1 --2- 172.21.15.16:0/2563095302 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f38a0068520 0x7f38a006a9e0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f38ac0af000 tx=0x7f38ac0aa000 comp rx=0 tx=0).stop 2024-03-31T15:06:22.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:22.987+0000 7f38cb800640 1 -- 172.21.15.16:0/2563095302 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f38a8004810 msgr2=0x7f38c40b3b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:22.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:22.987+0000 7f38cb800640 1 --2- 172.21.15.16:0/2563095302 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f38a8004810 0x7f38c40b3b40 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f38b400f7a0 tx=0x7f38b400fc60 comp rx=0 tx=0).stop 2024-03-31T15:06:22.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:22.987+0000 7f38cb800640 1 -- 172.21.15.16:0/2563095302 shutdown_connections 2024-03-31T15:06:22.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:22.987+0000 7f38cb800640 1 --2- 172.21.15.16:0/2563095302 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f38a0068520 0x7f38a006a9e0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:22.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:22.987+0000 7f38cb800640 1 --2- 172.21.15.16:0/2563095302 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f38c41412e0 0x7f38c413b890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:22.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:22.987+0000 7f38cb800640 1 --2- 172.21.15.16:0/2563095302 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f38c4140e80 0x7f38c413b350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:22.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:22.987+0000 7f38cb800640 1 --2- 172.21.15.16:0/2563095302 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f38a8004810 0x7f38c40b3b40 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:22.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:22.987+0000 7f38cb800640 1 -- 172.21.15.16:0/2563095302 >> 172.21.15.16:0/2563095302 conn(0x7f38c406b620 msgr2=0x7f38c40cd2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:06:22.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:22.987+0000 7f38cb800640 1 -- 172.21.15.16:0/2563095302 shutdown_connections 2024-03-31T15:06:22.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:22.987+0000 7f38cb800640 1 -- 172.21.15.16:0/2563095302 wait complete. 2024-03-31T15:06:23.002 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-31T15:03:22.550787+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-31T15:06:23.002 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-31T15:06:23.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:23.151+0000 7ff152e00640 1 Processor -- start 2024-03-31T15:06:23.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:23.151+0000 7ff152e00640 1 -- start start 2024-03-31T15:06:23.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:23.151+0000 7ff152e00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff14c0d0a60 0x7ff14c0cea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:23.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:23.151+0000 7ff152e00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7ff14c0cef80 0x7ff14c1419c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:23.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:23.151+0000 7ff152e00640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff14c15ca70 con 0x7ff14c06f920 2024-03-31T15:06:23.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:23.151+0000 7ff152e00640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7ff14c13f670 con 0x7ff14c0d0a60 2024-03-31T15:06:23.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:23.151+0000 7ff152e00640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7ff14c15cd50 con 0x7ff14c0cef80 2024-03-31T15:06:23.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:23.151+0000 7ff14be00640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7ff14c06f920 0x7ff14c06fd00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:50682/0 (socket says 172.21.15.16:50682) 2024-03-31T15:06:23.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:23.151+0000 7ff14be00640 1 -- 172.21.15.16:0/4135448143 learned_addr learned my addr 172.21.15.16:0/4135448143 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:06:23.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:23.151+0000 7ff143400640 1 --2- 172.21.15.16:0/4135448143 >> v2:172.21.15.16:3300/0 conn(0x7ff14c0d0a60 0x7ff14c0cea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:23.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:23.151+0000 7ff150e00640 1 --2- 172.21.15.16:0/4135448143 >> v2:172.21.15.89:3300/0 conn(0x7ff14c0cef80 0x7ff14c1419c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:23.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:23.151+0000 7ff143400640 1 -- 172.21.15.16:0/4135448143 >> v2:172.21.15.89:3300/0 conn(0x7ff14c0cef80 msgr2=0x7ff14c1419c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:06:23.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:23.151+0000 7ff143400640 1 --2- 172.21.15.16:0/4135448143 >> v2:172.21.15.89:3300/0 conn(0x7ff14c0cef80 0x7ff14c1419c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:23.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:23.151+0000 7ff143400640 1 -- 172.21.15.16:0/4135448143 >> v1:172.21.15.16:6789/0 conn(0x7ff14c06f920 legacy=0x7ff14c06fd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:23.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:23.151+0000 7ff14b400640 1 -- 172.21.15.16:0/4135448143 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1829366647 0 0) 0x7ff14c15ca70 con 0x7ff14c06f920 2024-03-31T15:06:23.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:23.151+0000 7ff143400640 1 -- 172.21.15.16:0/4135448143 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff14c15d030 con 0x7ff14c0d0a60 2024-03-31T15:06:23.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:23.155+0000 7ff143400640 1 --2- 172.21.15.16:0/4135448143 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff14c0d0a60 0x7ff14c0cea40 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7ff14c0b4530 tx=0x7ff13401c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=79e074411660a30c server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:06:23.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:23.155+0000 7ff14b400640 1 -- 172.21.15.16:0/4135448143 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1340b0070 con 0x7ff14c0d0a60 2024-03-31T15:06:23.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:23.155+0000 7ff14b400640 1 -- 172.21.15.16:0/4135448143 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff1340ab600 con 0x7ff14c0d0a60 2024-03-31T15:06:23.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:23.155+0000 7ff14b400640 1 -- 172.21.15.16:0/4135448143 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1340ab9d0 con 0x7ff14c0d0a60 2024-03-31T15:06:23.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:23.155+0000 7ff152e00640 1 -- 172.21.15.16:0/4135448143 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff14c0d0a60 msgr2=0x7ff14c0cea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:23.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:23.155+0000 7ff152e00640 1 --2- 172.21.15.16:0/4135448143 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff14c0d0a60 0x7ff14c0cea40 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7ff14c0b4530 tx=0x7ff13401c480 comp rx=0 tx=0).stop 2024-03-31T15:06:23.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:23.155+0000 7ff152e00640 1 -- 172.21.15.16:0/4135448143 shutdown_connections 2024-03-31T15:06:23.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:23.155+0000 7ff152e00640 1 --2- 172.21.15.16:0/4135448143 >> v2:172.21.15.89:3300/0 conn(0x7ff14c0cef80 0x7ff14c1419c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:23.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:23.155+0000 7ff152e00640 1 --2- 172.21.15.16:0/4135448143 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff14c0d0a60 0x7ff14c0cea40 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:23.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:23.155+0000 7ff152e00640 1 -- 172.21.15.16:0/4135448143 >> 172.21.15.16:0/4135448143 conn(0x7ff14c067930 msgr2=0x7ff14c0b3ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:06:23.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:23.155+0000 7ff152e00640 1 -- 172.21.15.16:0/4135448143 shutdown_connections 2024-03-31T15:06:23.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:23.155+0000 7ff152e00640 1 -- 172.21.15.16:0/4135448143 wait complete. 2024-03-31T15:06:23.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:23.159+0000 7ff152e00640 1 Processor -- start 2024-03-31T15:06:23.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:23.159+0000 7ff152e00640 1 -- start start 2024-03-31T15:06:23.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:23.159+0000 7ff152e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff14c06f920 0x7ff14c13e660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:23.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:23.159+0000 7ff152e00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff14c0cef80 0x7ff14c151890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:23.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:23.159+0000 7ff152e00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff14c0d0a60 0x7ff14c15d850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:23.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:23.159+0000 7ff152e00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7ff14c169230 con 0x7ff14c06f920 2024-03-31T15:06:23.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:23.159+0000 7ff152e00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7ff14c1690b0 con 0x7ff14c0cef80 2024-03-31T15:06:23.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:23.159+0000 7ff152e00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7ff14c1693b0 con 0x7ff14c0d0a60 2024-03-31T15:06:23.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:23.159+0000 7ff14be00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff14c06f920 0x7ff14c13e660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:23.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:23.159+0000 7ff14be00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff14c06f920 0x7ff14c13e660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35948/0 (socket says 172.21.15.16:35948) 2024-03-31T15:06:23.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:23.159+0000 7ff14be00640 1 -- 172.21.15.16:0/527316499 learned_addr learned my addr 172.21.15.16:0/527316499 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:06:23.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:23.159+0000 7ff150e00640 1 --2- 172.21.15.16:0/527316499 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff14c0d0a60 0x7ff14c15d850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:23.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:23.159+0000 7ff143400640 1 --2- 172.21.15.16:0/527316499 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff14c0cef80 0x7ff14c151890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:23.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:23.159+0000 7ff14be00640 1 -- 172.21.15.16:0/527316499 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff14c0d0a60 msgr2=0x7ff14c15d850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:23.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:23.159+0000 7ff14be00640 1 --2- 172.21.15.16:0/527316499 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff14c0d0a60 0x7ff14c15d850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:23.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:23.159+0000 7ff14be00640 1 -- 172.21.15.16:0/527316499 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff14c0cef80 msgr2=0x7ff14c151890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:23.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:23.159+0000 7ff14be00640 1 --2- 172.21.15.16:0/527316499 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff14c0cef80 0x7ff14c151890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:23.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:23.159+0000 7ff14be00640 1 -- 172.21.15.16:0/527316499 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff14c15f900 con 0x7ff14c06f920 2024-03-31T15:06:23.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:23.159+0000 7ff14be00640 1 --2- 172.21.15.16:0/527316499 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff14c06f920 0x7ff14c13e660 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7ff13c00c520 tx=0x7ff13c00c9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:06:23.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:23.159+0000 7ff148c00640 1 -- 172.21.15.16:0/527316499 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff13c018070 con 0x7ff14c06f920 2024-03-31T15:06:23.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:23.159+0000 7ff148c00640 1 -- 172.21.15.16:0/527316499 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff13c0074e0 con 0x7ff14c06f920 2024-03-31T15:06:23.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:23.159+0000 7ff152e00640 1 -- 172.21.15.16:0/527316499 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff14c15fba0 con 0x7ff14c06f920 2024-03-31T15:06:23.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:23.159+0000 7ff148c00640 1 -- 172.21.15.16:0/527316499 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff13c0049f0 con 0x7ff14c06f920 2024-03-31T15:06:23.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:23.159+0000 7ff152e00640 1 -- 172.21.15.16:0/527316499 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff14c152250 con 0x7ff14c06f920 2024-03-31T15:06:23.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:23.159+0000 7ff152e00640 1 -- 172.21.15.16:0/527316499 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff14c069950 con 0x7ff14c06f920 2024-03-31T15:06:23.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:23.163+0000 7ff148c00640 1 -- 172.21.15.16:0/527316499 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7ff13c01b070 con 0x7ff14c06f920 2024-03-31T15:06:23.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:23.163+0000 7ff148c00640 1 --2- 172.21.15.16:0/527316499 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7ff128068380 0x7ff12806a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:23.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:23.163+0000 7ff148c00640 1 -- 172.21.15.16:0/527316499 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(52..52 src has 1..52) ==== 4857+0+0 (secure 0 0 0) 0x7ff13c085ab0 con 0x7ff14c06f920 2024-03-31T15:06:23.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:23.163+0000 7ff143400640 1 --2- 172.21.15.16:0/527316499 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7ff128068380 0x7ff12806a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:23.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:23.163+0000 7ff143400640 1 --2- 172.21.15.16:0/527316499 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7ff128068380 0x7ff12806a840 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7ff1340ad000 tx=0x7ff134008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:06:23.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:23.167+0000 7ff148c00640 1 -- 172.21.15.16:0/527316499 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7ff13c0563b0 con 0x7ff14c06f920 2024-03-31T15:06:23.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:23.335+0000 7ff152e00640 1 -- 172.21.15.16:0/527316499 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7ff14c0b50e0 con 0x7ff14c06f920 2024-03-31T15:06:23.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:23.335+0000 7ff148c00640 1 -- 172.21.15.16:0/527316499 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v52) ==== 93+0+10 (secure 0 0 0) 0x7ff13c055cb0 con 0x7ff14c06f920 2024-03-31T15:06:23.330 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2024-03-31T15:06:23.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:23.335+0000 7ff152e00640 1 -- 172.21.15.16:0/527316499 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7ff128068380 msgr2=0x7ff12806a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:23.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:23.335+0000 7ff152e00640 1 --2- 172.21.15.16:0/527316499 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7ff128068380 0x7ff12806a840 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7ff1340ad000 tx=0x7ff134008040 comp rx=0 tx=0).stop 2024-03-31T15:06:23.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:23.335+0000 7ff152e00640 1 -- 172.21.15.16:0/527316499 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff14c06f920 msgr2=0x7ff14c13e660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:23.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:23.335+0000 7ff152e00640 1 --2- 172.21.15.16:0/527316499 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff14c06f920 0x7ff14c13e660 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7ff13c00c520 tx=0x7ff13c00c9e0 comp rx=0 tx=0).stop 2024-03-31T15:06:23.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:23.335+0000 7ff152e00640 1 -- 172.21.15.16:0/527316499 shutdown_connections 2024-03-31T15:06:23.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:23.339+0000 7ff152e00640 1 --2- 172.21.15.16:0/527316499 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7ff128068380 0x7ff12806a840 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:23.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:23.339+0000 7ff152e00640 1 --2- 172.21.15.16:0/527316499 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff14c0d0a60 0x7ff14c15d850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:23.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:23.339+0000 7ff152e00640 1 --2- 172.21.15.16:0/527316499 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff14c0cef80 0x7ff14c151890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:23.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:23.339+0000 7ff152e00640 1 --2- 172.21.15.16:0/527316499 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff14c06f920 0x7ff14c13e660 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:23.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:23.339+0000 7ff152e00640 1 -- 172.21.15.16:0/527316499 >> 172.21.15.16:0/527316499 conn(0x7ff14c067930 msgr2=0x7ff14c0b6640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:06:23.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:23.339+0000 7ff152e00640 1 -- 172.21.15.16:0/527316499 shutdown_connections 2024-03-31T15:06:23.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:23.339+0000 7ff152e00640 1 -- 172.21.15.16:0/527316499 wait complete. 2024-03-31T15:06:23.352 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-31T15:06:23.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:23.499+0000 7fe547800640 1 Processor -- start 2024-03-31T15:06:23.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:23.499+0000 7fe547800640 1 -- start start 2024-03-31T15:06:23.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:23.499+0000 7fe547800640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fe54006c1c0 0x7fe54015c8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:23.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:23.499+0000 7fe547800640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fe54006b620 0x7fe54006bab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:23.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:23.499+0000 7fe547800640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe5400d09f0 con 0x7fe540145aa0 2024-03-31T15:06:23.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:23.499+0000 7fe547800640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7fe54006fd50 con 0x7fe54006b620 2024-03-31T15:06:23.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:23.499+0000 7fe547800640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fe5400d0cd0 con 0x7fe54006c1c0 2024-03-31T15:06:23.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:23.499+0000 7fe545800640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fe54006b620 0x7fe54006bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:23.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:23.499+0000 7fe544e00640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fe540145aa0 0x7fe540072e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:50684/0 (socket says 172.21.15.16:50684) 2024-03-31T15:06:23.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:23.499+0000 7fe544e00640 1 -- 172.21.15.16:0/839051271 learned_addr learned my addr 172.21.15.16:0/839051271 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:06:23.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:23.499+0000 7fe537e00640 1 --2- 172.21.15.16:0/839051271 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fe54006c1c0 0x7fe54015c8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:23.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:23.499+0000 7fe545800640 1 -- 172.21.15.16:0/839051271 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fe54006c1c0 msgr2=0x7fe54015c8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:06:23.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:23.499+0000 7fe545800640 1 --2- 172.21.15.16:0/839051271 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fe54006c1c0 0x7fe54015c8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:23.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:23.499+0000 7fe545800640 1 -- 172.21.15.16:0/839051271 >> v1:172.21.15.16:6789/0 conn(0x7fe540145aa0 legacy=0x7fe540072e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:23.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:23.499+0000 7fe545800640 1 -- 172.21.15.16:0/839051271 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe5400d06e0 con 0x7fe54006b620 2024-03-31T15:06:23.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:23.499+0000 7fe53fe00640 1 -- 172.21.15.16:0/839051271 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1543187953 0 0) 0x7fe5400d09f0 con 0x7fe540145aa0 2024-03-31T15:06:23.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:23.503+0000 7fe545800640 1 --2- 172.21.15.16:0/839051271 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe54006b620 0x7fe54006bab0 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7fe538009d90 tx=0x7fe53801c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e4e9476b8b4eeb server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:06:23.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:23.503+0000 7fe53fe00640 1 -- 172.21.15.16:0/839051271 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5380b2070 con 0x7fe54006b620 2024-03-31T15:06:23.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:23.503+0000 7fe53fe00640 1 -- 172.21.15.16:0/839051271 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe5380a5830 con 0x7fe54006b620 2024-03-31T15:06:23.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:23.503+0000 7fe53fe00640 1 -- 172.21.15.16:0/839051271 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5380a5c00 con 0x7fe54006b620 2024-03-31T15:06:23.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:23.503+0000 7fe547800640 1 -- 172.21.15.16:0/839051271 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe54006b620 msgr2=0x7fe54006bab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:23.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:23.503+0000 7fe547800640 1 --2- 172.21.15.16:0/839051271 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe54006b620 0x7fe54006bab0 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7fe538009d90 tx=0x7fe53801c970 comp rx=0 tx=0).stop 2024-03-31T15:06:23.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:23.503+0000 7fe547800640 1 -- 172.21.15.16:0/839051271 shutdown_connections 2024-03-31T15:06:23.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:23.503+0000 7fe547800640 1 --2- 172.21.15.16:0/839051271 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe54006b620 0x7fe54006bab0 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:23.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:23.503+0000 7fe547800640 1 --2- 172.21.15.16:0/839051271 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fe54006c1c0 0x7fe54015c8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:23.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:23.503+0000 7fe547800640 1 -- 172.21.15.16:0/839051271 >> 172.21.15.16:0/839051271 conn(0x7fe5400d1060 msgr2=0x7fe5400d1470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:06:23.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:23.503+0000 7fe547800640 1 -- 172.21.15.16:0/839051271 shutdown_connections 2024-03-31T15:06:23.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:23.503+0000 7fe547800640 1 -- 172.21.15.16:0/839051271 wait complete. 2024-03-31T15:06:23.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:23.507+0000 7fe547800640 1 Processor -- start 2024-03-31T15:06:23.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:23.507+0000 7fe547800640 1 -- start start 2024-03-31T15:06:23.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:23.507+0000 7fe547800640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe54006b620 0x7fe5401f7270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:23.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:23.507+0000 7fe547800640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe54006c1c0 0x7fe5400af9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:23.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:23.507+0000 7fe547800640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fe540145aa0 0x7fe5400aff00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:23.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:23.507+0000 7fe547800640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fe5400ce690 con 0x7fe54006c1c0 2024-03-31T15:06:23.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:23.507+0000 7fe547800640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fe5400ce510 con 0x7fe54006b620 2024-03-31T15:06:23.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:23.507+0000 7fe547800640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fe5400ce810 con 0x7fe540145aa0 2024-03-31T15:06:23.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:23.507+0000 7fe537e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe54006c1c0 0x7fe5400af9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:23.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:23.507+0000 7fe537e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe54006c1c0 0x7fe5400af9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35968/0 (socket says 172.21.15.16:35968) 2024-03-31T15:06:23.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:23.507+0000 7fe537e00640 1 -- 172.21.15.16:0/513459083 learned_addr learned my addr 172.21.15.16:0/513459083 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:06:23.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:23.507+0000 7fe544e00640 1 --2- 172.21.15.16:0/513459083 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe54006b620 0x7fe5401f7270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:23.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:23.507+0000 7fe545800640 1 --2- 172.21.15.16:0/513459083 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fe540145aa0 0x7fe5400aff00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:23.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:23.507+0000 7fe537e00640 1 -- 172.21.15.16:0/513459083 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fe540145aa0 msgr2=0x7fe5400aff00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:23.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:23.507+0000 7fe537e00640 1 --2- 172.21.15.16:0/513459083 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fe540145aa0 0x7fe5400aff00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:23.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:23.507+0000 7fe537e00640 1 -- 172.21.15.16:0/513459083 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe54006b620 msgr2=0x7fe5401f7270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:23.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:23.507+0000 7fe537e00640 1 --2- 172.21.15.16:0/513459083 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe54006b620 0x7fe5401f7270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:23.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:23.507+0000 7fe537e00640 1 -- 172.21.15.16:0/513459083 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe540142a30 con 0x7fe54006c1c0 2024-03-31T15:06:23.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:23.507+0000 7fe537e00640 1 --2- 172.21.15.16:0/513459083 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe54006c1c0 0x7fe5400af9c0 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7fe52800d7d0 tx=0x7fe52800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:06:23.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:23.507+0000 7fe53d600640 1 -- 172.21.15.16:0/513459083 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe52800b6c0 con 0x7fe54006c1c0 2024-03-31T15:06:23.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:23.507+0000 7fe53d600640 1 -- 172.21.15.16:0/513459083 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe528019070 con 0x7fe54006c1c0 2024-03-31T15:06:23.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:23.507+0000 7fe53d600640 1 -- 172.21.15.16:0/513459083 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe52800ba80 con 0x7fe54006c1c0 2024-03-31T15:06:23.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:23.507+0000 7fe547800640 1 -- 172.21.15.16:0/513459083 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe540142d10 con 0x7fe54006c1c0 2024-03-31T15:06:23.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:23.507+0000 7fe547800640 1 -- 172.21.15.16:0/513459083 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe5401432d0 con 0x7fe54006c1c0 2024-03-31T15:06:23.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:23.507+0000 7fe547800640 1 -- 172.21.15.16:0/513459083 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe540069950 con 0x7fe54006c1c0 2024-03-31T15:06:23.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:23.511+0000 7fe53d600640 1 -- 172.21.15.16:0/513459083 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fe52800bc20 con 0x7fe54006c1c0 2024-03-31T15:06:23.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:23.511+0000 7fe53d600640 1 --2- 172.21.15.16:0/513459083 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fe51c0683b0 0x7fe51c06a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:23.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:23.511+0000 7fe53d600640 1 -- 172.21.15.16:0/513459083 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(52..52 src has 1..52) ==== 4857+0+0 (secure 0 0 0) 0x7fe528086990 con 0x7fe54006c1c0 2024-03-31T15:06:23.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:23.511+0000 7fe544e00640 1 --2- 172.21.15.16:0/513459083 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fe51c0683b0 0x7fe51c06a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:23.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:23.511+0000 7fe544e00640 1 --2- 172.21.15.16:0/513459083 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fe51c0683b0 0x7fe51c06a870 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fe530007fd0 tx=0x7fe530007d60 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:06:23.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:23.515+0000 7fe53d600640 1 -- 172.21.15.16:0/513459083 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fe528056bb0 con 0x7fe54006c1c0 2024-03-31T15:06:23.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:23.679+0000 7fe547800640 1 -- 172.21.15.16:0/513459083 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7fe54015dd40 con 0x7fe54006c1c0 2024-03-31T15:06:23.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:23.683+0000 7fe53d600640 1 -- 172.21.15.16:0/513459083 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v52) ==== 84+0+65 (secure 0 0 0) 0x7fe52805aeb0 con 0x7fe54006c1c0 2024-03-31T15:06:23.676 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:06:23.676 INFO:teuthology.orchestra.run.smithi016.stdout:[ 2024-03-31T15:06:23.676 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-31T15:06:23.676 INFO:teuthology.orchestra.run.smithi016.stdout: "poolnum": 1, 2024-03-31T15:06:23.676 INFO:teuthology.orchestra.run.smithi016.stdout: "poolname": ".mgr" 2024-03-31T15:06:23.676 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-31T15:06:23.677 INFO:teuthology.orchestra.run.smithi016.stdout:] 2024-03-31T15:06:23.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:23.683+0000 7fe547800640 1 -- 172.21.15.16:0/513459083 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fe51c0683b0 msgr2=0x7fe51c06a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:23.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:23.683+0000 7fe547800640 1 --2- 172.21.15.16:0/513459083 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fe51c0683b0 0x7fe51c06a870 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fe530007fd0 tx=0x7fe530007d60 comp rx=0 tx=0).stop 2024-03-31T15:06:23.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:23.683+0000 7fe547800640 1 -- 172.21.15.16:0/513459083 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe54006c1c0 msgr2=0x7fe5400af9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:23.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:23.683+0000 7fe547800640 1 --2- 172.21.15.16:0/513459083 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe54006c1c0 0x7fe5400af9c0 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7fe52800d7d0 tx=0x7fe52800dc90 comp rx=0 tx=0).stop 2024-03-31T15:06:23.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:23.683+0000 7fe547800640 1 -- 172.21.15.16:0/513459083 shutdown_connections 2024-03-31T15:06:23.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:23.683+0000 7fe547800640 1 --2- 172.21.15.16:0/513459083 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fe51c0683b0 0x7fe51c06a870 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:23.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:23.683+0000 7fe547800640 1 --2- 172.21.15.16:0/513459083 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fe540145aa0 0x7fe5400aff00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:23.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:23.683+0000 7fe547800640 1 --2- 172.21.15.16:0/513459083 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe54006c1c0 0x7fe5400af9c0 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:23.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:23.683+0000 7fe547800640 1 --2- 172.21.15.16:0/513459083 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe54006b620 0x7fe5401f7270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:23.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:23.683+0000 7fe547800640 1 -- 172.21.15.16:0/513459083 >> 172.21.15.16:0/513459083 conn(0x7fe5400d1060 msgr2=0x7fe54015ca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:06:23.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:23.683+0000 7fe547800640 1 -- 172.21.15.16:0/513459083 shutdown_connections 2024-03-31T15:06:23.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:23.683+0000 7fe547800640 1 -- 172.21.15.16:0/513459083 wait complete. 2024-03-31T15:06:23.697 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-31T15:06:23.697 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-31T15:06:23.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:23.847+0000 7f762ee00640 1 Processor -- start 2024-03-31T15:06:23.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:23.847+0000 7f762ee00640 1 -- start start 2024-03-31T15:06:23.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:23.847+0000 7f762ee00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f76280b44d0 0x7f76280d0930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:23.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:23.847+0000 7f762ee00640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f76280d2020 con 0x7f76280d0e70 2024-03-31T15:06:23.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:23.847+0000 7f762ee00640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f76280d2300 con 0x7f7628147230 2024-03-31T15:06:23.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:23.847+0000 7f762ee00640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f76280d2c30 con 0x7f76280b44d0 2024-03-31T15:06:23.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:23.847+0000 7f762ce00640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f76280d0e70 0x7f76280cb4b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:50686/0 (socket says 172.21.15.16:50686) 2024-03-31T15:06:23.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:23.847+0000 7f762ce00640 1 -- 172.21.15.16:0/2533410802 learned_addr learned my addr 172.21.15.16:0/2533410802 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:06:23.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:23.847+0000 7f761fe00640 1 --2- 172.21.15.16:0/2533410802 >> v2:172.21.15.89:3300/0 conn(0x7f76280b44d0 0x7f76280d0930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:23.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:23.851+0000 7f7627400640 1 -- 172.21.15.16:0/2533410802 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3228527955 0 0) 0x7f76280d2020 con 0x7f76280d0e70 2024-03-31T15:06:23.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:23.851+0000 7f7627400640 1 -- 172.21.15.16:0/2533410802 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f760c003530 con 0x7f76280d0e70 2024-03-31T15:06:23.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:23.851+0000 7f7627400640 1 -- 172.21.15.16:0/2533410802 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1029099417 0 0) 0x7f760c003530 con 0x7f76280d0e70 2024-03-31T15:06:23.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:23.851+0000 7f7627400640 1 -- 172.21.15.16:0/2533410802 >> v2:172.21.15.89:3300/0 conn(0x7f76280b44d0 msgr2=0x7f76280d0930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:06:23.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:23.851+0000 7f7627400640 1 --2- 172.21.15.16:0/2533410802 >> v2:172.21.15.89:3300/0 conn(0x7f76280b44d0 0x7f76280d0930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:23.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:23.851+0000 7f7627400640 1 -- 172.21.15.16:0/2533410802 >> v1:172.21.15.89:6789/0 conn(0x7f7628147230 legacy=0x7f76280b3dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:23.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:23.851+0000 7f7627400640 1 -- 172.21.15.16:0/2533410802 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f76280d25e0 con 0x7f76280d0e70 2024-03-31T15:06:23.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:23.851+0000 7f7627400640 1 -- 172.21.15.16:0/2533410802 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 214807623 0 0) 0x7f7620001a00 con 0x7f76280d0e70 2024-03-31T15:06:23.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:23.851+0000 7f761fe00640 1 --2- 172.21.15.16:0/2533410802 >> v2:172.21.15.89:3300/0 conn(0x7f76280b44d0 0x7f76280d0930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:06:23.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:23.851+0000 7f7627400640 1 -- 172.21.15.16:0/2533410802 >> v1:172.21.15.16:6789/0 conn(0x7f76280d0e70 legacy=0x7f76280cb4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:23.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:23.851+0000 7f7627400640 1 --2- 172.21.15.16:0/2533410802 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f760c0039c0 0x7f760c003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:23.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:23.851+0000 7f7627400640 1 --2- 172.21.15.16:0/2533410802 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f760c004810 0x7f760c004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:23.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:23.851+0000 7f7627400640 1 --2- 172.21.15.16:0/2533410802 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f760c0051d0 0x7f760c00b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:23.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:23.851+0000 7f7627400640 1 -- 172.21.15.16:0/2533410802 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f76280d2c30 con 0x7f760c004810 2024-03-31T15:06:23.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:23.851+0000 7f7627400640 1 -- 172.21.15.16:0/2533410802 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f760c00c070 con 0x7f760c0039c0 2024-03-31T15:06:23.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:23.851+0000 7f7627400640 1 -- 172.21.15.16:0/2533410802 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f760c00c270 con 0x7f760c0051d0 2024-03-31T15:06:23.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:23.851+0000 7f761fe00640 1 --2- 172.21.15.16:0/2533410802 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f760c004810 0x7f760c004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:23.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:23.851+0000 7f7627e00640 1 --2- 172.21.15.16:0/2533410802 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f760c0039c0 0x7f760c003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:23.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:23.851+0000 7f762ce00640 1 --2- 172.21.15.16:0/2533410802 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f760c0051d0 0x7f760c00b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:23.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:23.851+0000 7f761fe00640 1 -- 172.21.15.16:0/2533410802 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f760c0051d0 msgr2=0x7f760c00b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:06:23.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:23.851+0000 7f761fe00640 1 --2- 172.21.15.16:0/2533410802 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f760c0051d0 0x7f760c00b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:23.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:23.851+0000 7f761fe00640 1 -- 172.21.15.16:0/2533410802 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f760c0039c0 msgr2=0x7f760c003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:06:23.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:23.851+0000 7f761fe00640 1 --2- 172.21.15.16:0/2533410802 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f760c0039c0 0x7f760c003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:23.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:23.851+0000 7f761fe00640 1 -- 172.21.15.16:0/2533410802 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f760c003530 con 0x7f760c004810 2024-03-31T15:06:23.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:23.851+0000 7f762ce00640 1 --2- 172.21.15.16:0/2533410802 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f760c0051d0 0x7f760c00b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:06:23.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:23.851+0000 7f762ce00640 1 -- 172.21.15.16:0/2533410802 reap_dead start 2024-03-31T15:06:23.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:23.855+0000 7f761fe00640 1 --2- 172.21.15.16:0/2533410802 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f760c004810 0x7f760c004c90 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f7610009cb0 tx=0x7f761001cc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40dce690c9d7e066 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:06:23.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:23.855+0000 7f7627400640 1 -- 172.21.15.16:0/2533410802 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76100b2070 con 0x7f760c004810 2024-03-31T15:06:23.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:23.855+0000 7f7627400640 1 -- 172.21.15.16:0/2533410802 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f76100a58d0 con 0x7f760c004810 2024-03-31T15:06:23.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:23.855+0000 7f7627400640 1 -- 172.21.15.16:0/2533410802 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7610009880 con 0x7f760c004810 2024-03-31T15:06:23.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:23.855+0000 7f762ee00640 1 -- 172.21.15.16:0/2533410802 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f760c004810 msgr2=0x7f760c004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:23.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:23.855+0000 7f762ee00640 1 --2- 172.21.15.16:0/2533410802 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f760c004810 0x7f760c004c90 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f7610009cb0 tx=0x7f761001cc10 comp rx=0 tx=0).stop 2024-03-31T15:06:23.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:23.855+0000 7f762ee00640 1 -- 172.21.15.16:0/2533410802 shutdown_connections 2024-03-31T15:06:23.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:23.855+0000 7f762ee00640 1 --2- 172.21.15.16:0/2533410802 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f760c004810 0x7f760c004c90 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:23.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:23.855+0000 7f762ee00640 1 -- 172.21.15.16:0/2533410802 >> 172.21.15.16:0/2533410802 conn(0x7f762806b7f0 msgr2=0x7f7628067330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:06:23.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:23.855+0000 7f762ee00640 1 -- 172.21.15.16:0/2533410802 shutdown_connections 2024-03-31T15:06:23.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:23.855+0000 7f762ee00640 1 -- 172.21.15.16:0/2533410802 wait complete. 2024-03-31T15:06:23.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:23.855+0000 7f762ee00640 1 Processor -- start 2024-03-31T15:06:23.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:23.855+0000 7f762ee00640 1 -- start start 2024-03-31T15:06:23.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:23.855+0000 7f762ee00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f760c004810 0x7f76280ce200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:23.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:23.855+0000 7f762ee00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7628142f40 0x7f76281433c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:23.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:23.855+0000 7f762ee00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7628144d20 0x7f76281451a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:23.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:23.855+0000 7f762ee00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f76280d12c0 con 0x7f7628144d20 2024-03-31T15:06:23.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:23.855+0000 7f762ee00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f762806cfd0 con 0x7f7628142f40 2024-03-31T15:06:23.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:23.855+0000 7f762ee00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f76280d1440 con 0x7f760c004810 2024-03-31T15:06:23.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:23.855+0000 7f762ce00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7628144d20 0x7f76281451a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:23.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:23.855+0000 7f762ce00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7628144d20 0x7f76281451a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35990/0 (socket says 172.21.15.16:35990) 2024-03-31T15:06:23.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:23.855+0000 7f762ce00640 1 -- 172.21.15.16:0/2069117031 learned_addr learned my addr 172.21.15.16:0/2069117031 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:06:23.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:23.855+0000 7f761fe00640 1 --2- 172.21.15.16:0/2069117031 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7628142f40 0x7f76281433c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:23.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:23.855+0000 7f7627e00640 1 --2- 172.21.15.16:0/2069117031 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f760c004810 0x7f76280ce200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:23.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:23.859+0000 7f762ce00640 1 -- 172.21.15.16:0/2069117031 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f760c004810 msgr2=0x7f76280ce200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:23.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:23.859+0000 7f762ce00640 1 --2- 172.21.15.16:0/2069117031 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f760c004810 0x7f76280ce200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:23.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:23.859+0000 7f762ce00640 1 -- 172.21.15.16:0/2069117031 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7628142f40 msgr2=0x7f76281433c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:23.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:23.859+0000 7f762ce00640 1 --2- 172.21.15.16:0/2069117031 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7628142f40 0x7f76281433c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:23.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:23.859+0000 7f762ce00640 1 -- 172.21.15.16:0/2069117031 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7628143a00 con 0x7f7628144d20 2024-03-31T15:06:23.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:23.859+0000 7f7627e00640 1 --2- 172.21.15.16:0/2069117031 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f760c004810 0x7f76280ce200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:06:23.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:23.859+0000 7f761fe00640 1 --2- 172.21.15.16:0/2069117031 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7628142f40 0x7f76281433c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:06:23.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:23.859+0000 7f762ce00640 1 --2- 172.21.15.16:0/2069117031 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7628144d20 0x7f76281451a0 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f7620001b70 tx=0x7f76200119f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:06:23.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:23.859+0000 7f7624c00640 1 -- 172.21.15.16:0/2069117031 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7620010040 con 0x7f7628144d20 2024-03-31T15:06:23.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:23.859+0000 7f7624c00640 1 -- 172.21.15.16:0/2069117031 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f762001a070 con 0x7f7628144d20 2024-03-31T15:06:23.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:23.859+0000 7f7624c00640 1 -- 172.21.15.16:0/2069117031 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7620013a00 con 0x7f7628144d20 2024-03-31T15:06:23.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:23.859+0000 7f762ee00640 1 -- 172.21.15.16:0/2069117031 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7628143ce0 con 0x7f7628144d20 2024-03-31T15:06:23.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:23.859+0000 7f762ee00640 1 -- 172.21.15.16:0/2069117031 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f76281442a0 con 0x7f7628144d20 2024-03-31T15:06:23.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:23.859+0000 7f762ee00640 1 -- 172.21.15.16:0/2069117031 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7628069950 con 0x7f7628144d20 2024-03-31T15:06:23.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:23.859+0000 7f7624c00640 1 -- 172.21.15.16:0/2069117031 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f7620013ba0 con 0x7f7628144d20 2024-03-31T15:06:23.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:23.859+0000 7f7624c00640 1 --2- 172.21.15.16:0/2069117031 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f76040683b0 0x7f760406a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:23.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:23.859+0000 7f7624c00640 1 -- 172.21.15.16:0/2069117031 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(52..52 src has 1..52) ==== 4857+0+0 (secure 0 0 0) 0x7f76200892b0 con 0x7f7628144d20 2024-03-31T15:06:23.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:23.863+0000 7f7627e00640 1 --2- 172.21.15.16:0/2069117031 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f76040683b0 0x7f760406a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:23.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:23.863+0000 7f7627e00640 1 --2- 172.21.15.16:0/2069117031 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f76040683b0 0x7f760406a870 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f7618000c10 tx=0x7f7618009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:06:23.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:23.863+0000 7f7624c00640 1 -- 172.21.15.16:0/2069117031 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f7620059c50 con 0x7f7628144d20 2024-03-31T15:06:24.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:24.031+0000 7f762ee00640 1 -- 172.21.15.16:0/2069117031 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7f7628163020 con 0x7f7628144d20 2024-03-31T15:06:24.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:24.243+0000 7f7624c00640 1 -- 172.21.15.16:0/2069117031 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v53) ==== 138+0+0 (secure 0 0 0) 0x7f7620013e90 con 0x7f7628144d20 2024-03-31T15:06:24.238 INFO:teuthology.orchestra.run.smithi016.stderr:pool 'cephfs_metadata' created 2024-03-31T15:06:24.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:24.243+0000 7f762ee00640 1 -- 172.21.15.16:0/2069117031 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f76040683b0 msgr2=0x7f760406a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:24.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:24.243+0000 7f762ee00640 1 --2- 172.21.15.16:0/2069117031 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f76040683b0 0x7f760406a870 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f7618000c10 tx=0x7f7618009040 comp rx=0 tx=0).stop 2024-03-31T15:06:24.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:24.243+0000 7f762ee00640 1 -- 172.21.15.16:0/2069117031 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7628144d20 msgr2=0x7f76281451a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:24.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:24.243+0000 7f762ee00640 1 --2- 172.21.15.16:0/2069117031 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7628144d20 0x7f76281451a0 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f7620001b70 tx=0x7f76200119f0 comp rx=0 tx=0).stop 2024-03-31T15:06:24.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:24.243+0000 7f762ee00640 1 -- 172.21.15.16:0/2069117031 shutdown_connections 2024-03-31T15:06:24.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:24.243+0000 7f762ee00640 1 --2- 172.21.15.16:0/2069117031 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f76040683b0 0x7f760406a870 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:24.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:24.243+0000 7f762ee00640 1 --2- 172.21.15.16:0/2069117031 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7628144d20 0x7f76281451a0 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:24.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:24.243+0000 7f762ee00640 1 --2- 172.21.15.16:0/2069117031 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7628142f40 0x7f76281433c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:24.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:24.243+0000 7f762ee00640 1 --2- 172.21.15.16:0/2069117031 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f760c004810 0x7f76280ce200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:24.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:24.243+0000 7f762ee00640 1 -- 172.21.15.16:0/2069117031 >> 172.21.15.16:0/2069117031 conn(0x7f762806b7f0 msgr2=0x7f7628146350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:06:24.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:24.243+0000 7f762ee00640 1 -- 172.21.15.16:0/2069117031 shutdown_connections 2024-03-31T15:06:24.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:24.243+0000 7f762ee00640 1 -- 172.21.15.16:0/2069117031 wait complete. 2024-03-31T15:06:24.264 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-31T15:06:24.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:24.415+0000 7f781bc00640 1 Processor -- start 2024-03-31T15:06:24.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:24.415+0000 7f781bc00640 1 -- start start 2024-03-31T15:06:24.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:24.415+0000 7f781bc00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f781413dff0 0x7f781406d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:24.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:24.419+0000 7f781bc00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f78140cd0e0 0x7f781406da10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:24.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:24.419+0000 7f781bc00640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f781415fbc0 con 0x7f781413e3d0 2024-03-31T15:06:24.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:24.419+0000 7f781bc00640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f781415f730 con 0x7f781413dff0 2024-03-31T15:06:24.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:24.419+0000 7f781bc00640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f781415ee00 con 0x7f78140cd0e0 2024-03-31T15:06:24.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:24.419+0000 7f7819200640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f781413dff0 0x7f781406d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:24.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:24.419+0000 7f7813e00640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f781413e3d0 0x7f78140cc9d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:50688/0 (socket says 172.21.15.16:50688) 2024-03-31T15:06:24.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:24.419+0000 7f7819200640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f781413dff0 0x7f781406d4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35994/0 (socket says 172.21.15.16:35994) 2024-03-31T15:06:24.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:24.419+0000 7f7813e00640 1 -- 172.21.15.16:0/763389382 learned_addr learned my addr 172.21.15.16:0/763389382 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:06:24.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:24.419+0000 7f7819c00640 1 --2- 172.21.15.16:0/763389382 >> v2:172.21.15.89:3300/0 conn(0x7f78140cd0e0 0x7f781406da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:24.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:24.419+0000 7f7819200640 1 -- 172.21.15.16:0/763389382 >> v2:172.21.15.89:3300/0 conn(0x7f78140cd0e0 msgr2=0x7f781406da10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:06:24.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:24.419+0000 7f7819200640 1 --2- 172.21.15.16:0/763389382 >> v2:172.21.15.89:3300/0 conn(0x7f78140cd0e0 0x7f781406da10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:24.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:24.419+0000 7f7819200640 1 -- 172.21.15.16:0/763389382 >> v1:172.21.15.16:6789/0 conn(0x7f781413e3d0 legacy=0x7f78140cc9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:24.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:24.419+0000 7f7819200640 1 -- 172.21.15.16:0/763389382 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f781415f0e0 con 0x7f781413dff0 2024-03-31T15:06:24.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:24.419+0000 7f7813400640 1 -- 172.21.15.16:0/763389382 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3115924612 0 0) 0x7f781415fbc0 con 0x7f781413e3d0 2024-03-31T15:06:24.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:24.419+0000 7f7819200640 1 --2- 172.21.15.16:0/763389382 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f781413dff0 0x7f781406d4d0 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f781415b6e0 tx=0x7f780801eac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea346bbf864de63a server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:06:24.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:24.419+0000 7f7813400640 1 -- 172.21.15.16:0/763389382 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78080b4070 con 0x7f781413dff0 2024-03-31T15:06:24.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:24.423+0000 7f7813400640 1 -- 172.21.15.16:0/763389382 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f78080a79a0 con 0x7f781413dff0 2024-03-31T15:06:24.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:24.423+0000 7f7813400640 1 -- 172.21.15.16:0/763389382 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78080a7d70 con 0x7f781413dff0 2024-03-31T15:06:24.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:24.423+0000 7f781bc00640 1 -- 172.21.15.16:0/763389382 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f781413dff0 msgr2=0x7f781406d4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:24.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:24.423+0000 7f781bc00640 1 --2- 172.21.15.16:0/763389382 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f781413dff0 0x7f781406d4d0 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f781415b6e0 tx=0x7f780801eac0 comp rx=0 tx=0).stop 2024-03-31T15:06:24.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:24.423+0000 7f781bc00640 1 -- 172.21.15.16:0/763389382 shutdown_connections 2024-03-31T15:06:24.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:24.423+0000 7f781bc00640 1 --2- 172.21.15.16:0/763389382 >> v2:172.21.15.89:3300/0 conn(0x7f78140cd0e0 0x7f781406da10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:24.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:24.423+0000 7f781bc00640 1 --2- 172.21.15.16:0/763389382 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f781413dff0 0x7f781406d4d0 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:24.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:24.423+0000 7f781bc00640 1 -- 172.21.15.16:0/763389382 >> 172.21.15.16:0/763389382 conn(0x7f781406ee40 msgr2=0x7f781406b620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:06:24.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:24.423+0000 7f781bc00640 1 -- 172.21.15.16:0/763389382 shutdown_connections 2024-03-31T15:06:24.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:24.423+0000 7f781bc00640 1 -- 172.21.15.16:0/763389382 wait complete. 2024-03-31T15:06:24.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:24.423+0000 7f781bc00640 1 Processor -- start 2024-03-31T15:06:24.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:24.423+0000 7f781bc00640 1 -- start start 2024-03-31T15:06:24.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:24.423+0000 7f781bc00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f78140cd0e0 0x7f781413d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:24.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:24.423+0000 7f781bc00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f781413dff0 0x7f78140ad8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:24.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:24.423+0000 7f781bc00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f781413e3d0 0x7f7814158c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:24.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:24.423+0000 7f781bc00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f78140720f0 con 0x7f78140cd0e0 2024-03-31T15:06:24.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:24.423+0000 7f781bc00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f7814071f70 con 0x7f781413dff0 2024-03-31T15:06:24.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:24.423+0000 7f781bc00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f7814072270 con 0x7f781413e3d0 2024-03-31T15:06:24.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:24.423+0000 7f7819200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f78140cd0e0 0x7f781413d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:24.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:24.423+0000 7f7819200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f78140cd0e0 0x7f781413d4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35996/0 (socket says 172.21.15.16:35996) 2024-03-31T15:06:24.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:24.423+0000 7f7819200640 1 -- 172.21.15.16:0/2937548875 learned_addr learned my addr 172.21.15.16:0/2937548875 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:06:24.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:24.423+0000 7f7819c00640 1 --2- 172.21.15.16:0/2937548875 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f781413e3d0 0x7f7814158c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:24.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:24.423+0000 7f7813e00640 1 --2- 172.21.15.16:0/2937548875 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f781413dff0 0x7f78140ad8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:24.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:24.423+0000 7f7819200640 1 -- 172.21.15.16:0/2937548875 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f781413e3d0 msgr2=0x7f7814158c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:24.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:24.423+0000 7f7819200640 1 --2- 172.21.15.16:0/2937548875 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f781413e3d0 0x7f7814158c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:24.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:24.423+0000 7f7819200640 1 -- 172.21.15.16:0/2937548875 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f781413dff0 msgr2=0x7f78140ad8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:24.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:24.423+0000 7f7819200640 1 --2- 172.21.15.16:0/2937548875 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f781413dff0 0x7f78140ad8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:24.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:24.423+0000 7f7819200640 1 -- 172.21.15.16:0/2937548875 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78140ade10 con 0x7f78140cd0e0 2024-03-31T15:06:24.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:24.423+0000 7f7819c00640 1 --2- 172.21.15.16:0/2937548875 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f781413e3d0 0x7f7814158c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:06:24.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:24.427+0000 7f7819200640 1 --2- 172.21.15.16:0/2937548875 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f78140cd0e0 0x7f781413d4e0 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f780801b6d0 tx=0x7f780801d300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:06:24.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:24.427+0000 7f7810c00640 1 -- 172.21.15.16:0/2937548875 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78080b4040 con 0x7f78140cd0e0 2024-03-31T15:06:24.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:24.427+0000 7f7810c00640 1 -- 172.21.15.16:0/2937548875 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7808004020 con 0x7f78140cd0e0 2024-03-31T15:06:24.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:24.427+0000 7f7810c00640 1 -- 172.21.15.16:0/2937548875 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78080043d0 con 0x7f78140cd0e0 2024-03-31T15:06:24.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:24.427+0000 7f781bc00640 1 -- 172.21.15.16:0/2937548875 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f78141591c0 con 0x7f78140cd0e0 2024-03-31T15:06:24.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:24.427+0000 7f781bc00640 1 -- 172.21.15.16:0/2937548875 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f78140ae650 con 0x7f78140cd0e0 2024-03-31T15:06:24.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:24.427+0000 7f781bc00640 1 -- 172.21.15.16:0/2937548875 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7814069950 con 0x7f78140cd0e0 2024-03-31T15:06:24.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:24.427+0000 7f7810c00640 1 -- 172.21.15.16:0/2937548875 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f78080c1060 con 0x7f78140cd0e0 2024-03-31T15:06:24.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:24.427+0000 7f7810c00640 1 --2- 172.21.15.16:0/2937548875 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f77f4068380 0x7f77f406a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:24.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:24.427+0000 7f7810c00640 1 -- 172.21.15.16:0/2937548875 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(53..53 src has 1..53) ==== 5227+0+0 (secure 0 0 0) 0x7f7808121080 con 0x7f78140cd0e0 2024-03-31T15:06:24.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:24.431+0000 7f7813e00640 1 --2- 172.21.15.16:0/2937548875 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f77f4068380 0x7f77f406a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:24.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:24.431+0000 7f7813e00640 1 --2- 172.21.15.16:0/2937548875 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f77f4068380 0x7f77f406a840 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f781415da70 tx=0x7f7800007b60 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:06:24.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:24.431+0000 7f7810c00640 1 -- 172.21.15.16:0/2937548875 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f78140ade10 con 0x7f78140cd0e0 2024-03-31T15:06:24.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:24.599+0000 7f781bc00640 1 -- 172.21.15.16:0/2937548875 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7f78140b1590 con 0x7f78140cd0e0 2024-03-31T15:06:25.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:25.247+0000 7f7810c00640 1 -- 172.21.15.16:0/2937548875 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v54) ==== 170+0+0 (secure 0 0 0) 0x7f7808018740 con 0x7f78140cd0e0 2024-03-31T15:06:25.241 INFO:teuthology.orchestra.run.smithi016.stderr:pool 'cephfs_data' created 2024-03-31T15:06:25.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:25.251+0000 7f781bc00640 1 -- 172.21.15.16:0/2937548875 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f77f4068380 msgr2=0x7f77f406a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:25.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:25.251+0000 7f781bc00640 1 --2- 172.21.15.16:0/2937548875 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f77f4068380 0x7f77f406a840 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f781415da70 tx=0x7f7800007b60 comp rx=0 tx=0).stop 2024-03-31T15:06:25.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:25.251+0000 7f781bc00640 1 -- 172.21.15.16:0/2937548875 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f78140cd0e0 msgr2=0x7f781413d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:25.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:25.251+0000 7f781bc00640 1 --2- 172.21.15.16:0/2937548875 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f78140cd0e0 0x7f781413d4e0 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f780801b6d0 tx=0x7f780801d300 comp rx=0 tx=0).stop 2024-03-31T15:06:25.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:25.251+0000 7f781bc00640 1 -- 172.21.15.16:0/2937548875 shutdown_connections 2024-03-31T15:06:25.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:25.251+0000 7f781bc00640 1 --2- 172.21.15.16:0/2937548875 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f77f4068380 0x7f77f406a840 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:25.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:25.251+0000 7f781bc00640 1 --2- 172.21.15.16:0/2937548875 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f781413e3d0 0x7f7814158c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:25.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:25.251+0000 7f781bc00640 1 --2- 172.21.15.16:0/2937548875 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f781413dff0 0x7f78140ad8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:25.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:25.251+0000 7f781bc00640 1 --2- 172.21.15.16:0/2937548875 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f78140cd0e0 0x7f781413d4e0 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:25.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:25.251+0000 7f781bc00640 1 -- 172.21.15.16:0/2937548875 >> 172.21.15.16:0/2937548875 conn(0x7f781406ee40 msgr2=0x7f78140b0d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:06:25.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:25.251+0000 7f781bc00640 1 -- 172.21.15.16:0/2937548875 shutdown_connections 2024-03-31T15:06:25.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:25.251+0000 7f781bc00640 1 -- 172.21.15.16:0/2937548875 wait complete. 2024-03-31T15:06:25.281 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-31T15:06:25.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:25.435+0000 7f8874a00640 1 Processor -- start 2024-03-31T15:06:25.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:25.435+0000 7f8874a00640 1 -- start start 2024-03-31T15:06:25.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:25.435+0000 7f8874a00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f887006f920 0x7f887006fd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:25.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:25.435+0000 7f8874a00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f88700d0a60 0x7f88700cea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:25.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:25.435+0000 7f8874a00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f88700cef80 0x7f8870070e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:25.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:25.435+0000 7f8874a00640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f887013f670 con 0x7f887006f920 2024-03-31T15:06:25.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:25.435+0000 7f8874a00640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f8870070240 con 0x7f88700cef80 2024-03-31T15:06:25.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:25.435+0000 7f8874a00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f887015ca70 con 0x7f88700d0a60 2024-03-31T15:06:25.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:25.435+0000 7f886de00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f887006f920 0x7f887006fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:25.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:25.435+0000 7f886de00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f887006f920 0x7f887006fd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36008/0 (socket says 172.21.15.16:36008) 2024-03-31T15:06:25.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:25.435+0000 7f886de00640 1 -- 172.21.15.16:0/4225053950 learned_addr learned my addr 172.21.15.16:0/4225053950 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:06:25.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:25.435+0000 7f886e800640 1 --2- 172.21.15.16:0/4225053950 >> v2:172.21.15.89:3300/0 conn(0x7f88700cef80 0x7f8870070e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:25.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:25.435+0000 7f8865400640 1 --2- 172.21.15.16:0/4225053950 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f88700d0a60 0x7f88700cea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:25.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:25.435+0000 7f886de00640 1 -- 172.21.15.16:0/4225053950 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f88700d0a60 msgr2=0x7f88700cea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:06:25.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:25.435+0000 7f886de00640 1 --2- 172.21.15.16:0/4225053950 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f88700d0a60 0x7f88700cea40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:25.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:25.435+0000 7f886de00640 1 -- 172.21.15.16:0/4225053950 >> v2:172.21.15.89:3300/0 conn(0x7f88700cef80 msgr2=0x7f8870070e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:06:25.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:25.435+0000 7f886de00640 1 --2- 172.21.15.16:0/4225053950 >> v2:172.21.15.89:3300/0 conn(0x7f88700cef80 0x7f8870070e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:25.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:25.435+0000 7f886de00640 1 -- 172.21.15.16:0/4225053950 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f887015cd50 con 0x7f887006f920 2024-03-31T15:06:25.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:25.435+0000 7f8865400640 1 --2- 172.21.15.16:0/4225053950 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f88700d0a60 0x7f88700cea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:06:25.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:25.439+0000 7f886de00640 1 --2- 172.21.15.16:0/4225053950 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f887006f920 0x7f887006fd00 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f88700b4cd0 tx=0x7f886001c260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b7cfbd636ff972ff server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:06:25.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:25.439+0000 7f886d400640 1 -- 172.21.15.16:0/4225053950 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88600b0070 con 0x7f887006f920 2024-03-31T15:06:25.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:25.439+0000 7f886d400640 1 -- 172.21.15.16:0/4225053950 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f88600ab600 con 0x7f887006f920 2024-03-31T15:06:25.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:25.439+0000 7f886d400640 1 -- 172.21.15.16:0/4225053950 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88600ab9d0 con 0x7f887006f920 2024-03-31T15:06:25.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:25.439+0000 7f8874a00640 1 -- 172.21.15.16:0/4225053950 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f887006f920 msgr2=0x7f887006fd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:25.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:25.439+0000 7f8874a00640 1 --2- 172.21.15.16:0/4225053950 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f887006f920 0x7f887006fd00 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f88700b4cd0 tx=0x7f886001c260 comp rx=0 tx=0).stop 2024-03-31T15:06:25.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:25.439+0000 7f8874a00640 1 -- 172.21.15.16:0/4225053950 shutdown_connections 2024-03-31T15:06:25.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:25.439+0000 7f8874a00640 1 --2- 172.21.15.16:0/4225053950 >> v2:172.21.15.89:3300/0 conn(0x7f88700cef80 0x7f8870070e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:25.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:25.439+0000 7f8874a00640 1 --2- 172.21.15.16:0/4225053950 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f88700d0a60 0x7f88700cea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:25.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:25.439+0000 7f8874a00640 1 --2- 172.21.15.16:0/4225053950 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f887006f920 0x7f887006fd00 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:25.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:25.439+0000 7f8874a00640 1 -- 172.21.15.16:0/4225053950 >> 172.21.15.16:0/4225053950 conn(0x7f8870067930 msgr2=0x7f88700b3ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:06:25.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:25.439+0000 7f8874a00640 1 -- 172.21.15.16:0/4225053950 shutdown_connections 2024-03-31T15:06:25.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:25.439+0000 7f8874a00640 1 -- 172.21.15.16:0/4225053950 wait complete. 2024-03-31T15:06:25.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:25.443+0000 7f8874a00640 1 Processor -- start 2024-03-31T15:06:25.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:25.443+0000 7f8874a00640 1 -- start start 2024-03-31T15:06:25.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:25.443+0000 7f8874a00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f887006f920 0x7f88700ce700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:25.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:25.443+0000 7f8874a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f88700cef80 0x7f887015d850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:25.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:25.443+0000 7f8874a00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f88700d0a60 0x7f887013e880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:25.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:25.443+0000 7f8874a00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f88701419c0 con 0x7f88700cef80 2024-03-31T15:06:25.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:25.443+0000 7f8874a00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f88700cf390 con 0x7f88700d0a60 2024-03-31T15:06:25.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:25.443+0000 7f8874a00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f8870141b40 con 0x7f887006f920 2024-03-31T15:06:25.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:25.443+0000 7f8865400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f88700cef80 0x7f887015d850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:25.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:25.443+0000 7f8865400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f88700cef80 0x7f887015d850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36012/0 (socket says 172.21.15.16:36012) 2024-03-31T15:06:25.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:25.443+0000 7f8865400640 1 -- 172.21.15.16:0/2707056979 learned_addr learned my addr 172.21.15.16:0/2707056979 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:06:25.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:25.443+0000 7f886e800640 1 --2- 172.21.15.16:0/2707056979 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f88700d0a60 0x7f887013e880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:25.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:25.443+0000 7f886de00640 1 --2- 172.21.15.16:0/2707056979 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f887006f920 0x7f88700ce700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:25.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:25.443+0000 7f8865400640 1 -- 172.21.15.16:0/2707056979 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f887006f920 msgr2=0x7f88700ce700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:25.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:25.443+0000 7f8865400640 1 --2- 172.21.15.16:0/2707056979 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f887006f920 0x7f88700ce700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:25.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:25.443+0000 7f8865400640 1 -- 172.21.15.16:0/2707056979 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f88700d0a60 msgr2=0x7f887013e880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:25.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:25.443+0000 7f8865400640 1 --2- 172.21.15.16:0/2707056979 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f88700d0a60 0x7f887013e880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:25.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:25.443+0000 7f8865400640 1 -- 172.21.15.16:0/2707056979 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f887015dd90 con 0x7f88700cef80 2024-03-31T15:06:25.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:25.443+0000 7f886de00640 1 --2- 172.21.15.16:0/2707056979 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f887006f920 0x7f88700ce700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:06:25.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:25.443+0000 7f8865400640 1 --2- 172.21.15.16:0/2707056979 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f88700cef80 0x7f887015d850 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f885800d800 tx=0x7f885800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:06:25.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:25.443+0000 7f8866a00640 1 -- 172.21.15.16:0/2707056979 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f885800b6c0 con 0x7f88700cef80 2024-03-31T15:06:25.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:25.443+0000 7f8866a00640 1 -- 172.21.15.16:0/2707056979 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8858019070 con 0x7f88700cef80 2024-03-31T15:06:25.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:25.443+0000 7f8874a00640 1 -- 172.21.15.16:0/2707056979 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f887015e030 con 0x7f88700cef80 2024-03-31T15:06:25.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:25.443+0000 7f8866a00640 1 -- 172.21.15.16:0/2707056979 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f885800baa0 con 0x7f88700cef80 2024-03-31T15:06:25.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:25.443+0000 7f8874a00640 1 -- 172.21.15.16:0/2707056979 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f887015e5f0 con 0x7f88700cef80 2024-03-31T15:06:25.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:25.443+0000 7f8874a00640 1 -- 172.21.15.16:0/2707056979 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8870069950 con 0x7f88700cef80 2024-03-31T15:06:25.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:25.447+0000 7f8866a00640 1 -- 172.21.15.16:0/2707056979 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f885800bc40 con 0x7f88700cef80 2024-03-31T15:06:25.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:25.447+0000 7f8866a00640 1 --2- 172.21.15.16:0/2707056979 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f884c068380 0x7f884c06a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:25.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:25.447+0000 7f8866a00640 1 -- 172.21.15.16:0/2707056979 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(54..54 src has 1..54) ==== 5609+0+0 (secure 0 0 0) 0x7f8858086e10 con 0x7f88700cef80 2024-03-31T15:06:25.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:25.447+0000 7f886de00640 1 --2- 172.21.15.16:0/2707056979 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f884c068380 0x7f884c06a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:25.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:25.447+0000 7f886de00640 1 --2- 172.21.15.16:0/2707056979 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f884c068380 0x7f884c06a840 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f88600ad000 tx=0x7f8860008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:06:25.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:25.451+0000 7f8866a00640 1 -- 172.21.15.16:0/2707056979 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f885808a030 con 0x7f88700cef80 2024-03-31T15:06:25.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:25.659+0000 7f8874a00640 1 -- 172.21.15.16:0/2707056979 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7f88700b4a20 con 0x7f88700cef80 2024-03-31T15:06:26.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:26.259+0000 7f8866a00640 1 -- 172.21.15.16:0/2707056979 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 8 and data pool 9 v30) ==== 172+0+0 (secure 0 0 0) 0x7f8858057390 con 0x7f88700cef80 2024-03-31T15:06:26.255 INFO:teuthology.orchestra.run.smithi016.stderr:new fs with metadata pool 8 and data pool 9 2024-03-31T15:06:26.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:26.267+0000 7f8874a00640 1 -- 172.21.15.16:0/2707056979 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f884c068380 msgr2=0x7f884c06a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:26.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:26.267+0000 7f8874a00640 1 --2- 172.21.15.16:0/2707056979 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f884c068380 0x7f884c06a840 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f88600ad000 tx=0x7f8860008040 comp rx=0 tx=0).stop 2024-03-31T15:06:26.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:26.267+0000 7f8874a00640 1 -- 172.21.15.16:0/2707056979 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f88700cef80 msgr2=0x7f887015d850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:26.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:26.267+0000 7f8874a00640 1 --2- 172.21.15.16:0/2707056979 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f88700cef80 0x7f887015d850 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f885800d800 tx=0x7f885800dcc0 comp rx=0 tx=0).stop 2024-03-31T15:06:26.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:26.267+0000 7f8874a00640 1 -- 172.21.15.16:0/2707056979 shutdown_connections 2024-03-31T15:06:26.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:26.267+0000 7f8874a00640 1 --2- 172.21.15.16:0/2707056979 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f884c068380 0x7f884c06a840 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:26.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:26.267+0000 7f8874a00640 1 --2- 172.21.15.16:0/2707056979 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f88700d0a60 0x7f887013e880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:26.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:26.267+0000 7f8874a00640 1 --2- 172.21.15.16:0/2707056979 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f88700cef80 0x7f887015d850 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:26.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:26.267+0000 7f8874a00640 1 --2- 172.21.15.16:0/2707056979 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f887006f920 0x7f88700ce700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:26.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:26.267+0000 7f8874a00640 1 -- 172.21.15.16:0/2707056979 >> 172.21.15.16:0/2707056979 conn(0x7f8870067930 msgr2=0x7f88700b3c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:06:26.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:26.271+0000 7f8874a00640 1 -- 172.21.15.16:0/2707056979 shutdown_connections 2024-03-31T15:06:26.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:26.271+0000 7f8874a00640 1 -- 172.21.15.16:0/2707056979 wait complete. 2024-03-31T15:06:26.311 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-31T15:06:26.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:26.459+0000 7f91d0600640 1 Processor -- start 2024-03-31T15:06:26.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:26.459+0000 7f91d0600640 1 -- start start 2024-03-31T15:06:26.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:26.459+0000 7f91d0600640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f91c8145d10 0x7f91c80cf790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:26.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:26.459+0000 7f91d0600640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f91c80728b0 0x7f91c80b1cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:26.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:26.459+0000 7f91d0600640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f91c80b2200 0x7f91c815cd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:26.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:26.459+0000 7f91d0600640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f91c8160e70 con 0x7f91c80728b0 2024-03-31T15:06:26.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:26.459+0000 7f91d0600640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f91c80cfcd0 con 0x7f91c8145d10 2024-03-31T15:06:26.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:26.459+0000 7f91d0600640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f91c81602f0 con 0x7f91c80b2200 2024-03-31T15:06:26.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:26.459+0000 7f91cd200640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f91c80728b0 0x7f91c80b1cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:26.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:26.459+0000 7f91cd200640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f91c80728b0 0x7f91c80b1cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36024/0 (socket says 172.21.15.16:36024) 2024-03-31T15:06:26.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:26.459+0000 7f91cd200640 1 -- 172.21.15.16:0/3862443879 learned_addr learned my addr 172.21.15.16:0/3862443879 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:06:26.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:26.459+0000 7f91cdc00640 1 --2- 172.21.15.16:0/3862443879 >> v2:172.21.15.89:3300/0 conn(0x7f91c8145d10 0x7f91c80cf790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:26.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:26.459+0000 7f91ce600640 1 --2- 172.21.15.16:0/3862443879 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f91c80b2200 0x7f91c815cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:26.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:26.459+0000 7f91cd200640 1 -- 172.21.15.16:0/3862443879 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f91c80b2200 msgr2=0x7f91c815cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:06:26.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:26.459+0000 7f91cd200640 1 --2- 172.21.15.16:0/3862443879 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f91c80b2200 0x7f91c815cd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:26.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:26.459+0000 7f91cd200640 1 -- 172.21.15.16:0/3862443879 >> v2:172.21.15.89:3300/0 conn(0x7f91c8145d10 msgr2=0x7f91c80cf790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:06:26.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:26.459+0000 7f91cd200640 1 --2- 172.21.15.16:0/3862443879 >> v2:172.21.15.89:3300/0 conn(0x7f91c8145d10 0x7f91c80cf790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:26.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:26.459+0000 7f91cd200640 1 -- 172.21.15.16:0/3862443879 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f91c81605d0 con 0x7f91c80728b0 2024-03-31T15:06:26.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:26.463+0000 7f91cd200640 1 --2- 172.21.15.16:0/3862443879 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f91c80728b0 0x7f91c80b1cc0 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f91c80d1060 tx=0x7f91b401c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8302dde7d663ed58 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:06:26.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:26.463+0000 7f91c7e00640 1 -- 172.21.15.16:0/3862443879 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91b40b0070 con 0x7f91c80728b0 2024-03-31T15:06:26.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:26.463+0000 7f91c7e00640 1 -- 172.21.15.16:0/3862443879 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f91b40ab600 con 0x7f91c80728b0 2024-03-31T15:06:26.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:26.463+0000 7f91c7e00640 1 -- 172.21.15.16:0/3862443879 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91b40ab9d0 con 0x7f91c80728b0 2024-03-31T15:06:26.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:26.463+0000 7f91d0600640 1 -- 172.21.15.16:0/3862443879 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f91c80728b0 msgr2=0x7f91c80b1cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:26.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:26.463+0000 7f91d0600640 1 --2- 172.21.15.16:0/3862443879 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f91c80728b0 0x7f91c80b1cc0 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f91c80d1060 tx=0x7f91b401c520 comp rx=0 tx=0).stop 2024-03-31T15:06:26.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:26.463+0000 7f91d0600640 1 -- 172.21.15.16:0/3862443879 shutdown_connections 2024-03-31T15:06:26.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:26.463+0000 7f91d0600640 1 --2- 172.21.15.16:0/3862443879 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f91c80b2200 0x7f91c815cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:26.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:26.463+0000 7f91d0600640 1 --2- 172.21.15.16:0/3862443879 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f91c80728b0 0x7f91c80b1cc0 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:26.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:26.463+0000 7f91d0600640 1 --2- 172.21.15.16:0/3862443879 >> v2:172.21.15.89:3300/0 conn(0x7f91c8145d10 0x7f91c80cf790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:26.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:26.463+0000 7f91d0600640 1 -- 172.21.15.16:0/3862443879 >> 172.21.15.16:0/3862443879 conn(0x7f91c806b620 msgr2=0x7f91c806ba30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:06:26.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:26.463+0000 7f91d0600640 1 -- 172.21.15.16:0/3862443879 shutdown_connections 2024-03-31T15:06:26.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:26.463+0000 7f91d0600640 1 -- 172.21.15.16:0/3862443879 wait complete. 2024-03-31T15:06:26.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:26.467+0000 7f91d0600640 1 Processor -- start 2024-03-31T15:06:26.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:26.467+0000 7f91d0600640 1 -- start start 2024-03-31T15:06:26.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:26.467+0000 7f91d0600640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f91c80728b0 0x7f91c813f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:26.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:26.467+0000 7f91d0600640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f91c80b2200 0x7f91c815bdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:26.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:26.467+0000 7f91d0600640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f91c8145d10 0x7f91c815ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:26.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:26.467+0000 7f91d0600640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f91c80cf350 con 0x7f91c80728b0 2024-03-31T15:06:26.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:26.467+0000 7f91d0600640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f91c80b2610 con 0x7f91c8145d10 2024-03-31T15:06:26.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:26.467+0000 7f91d0600640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f91c80cf4d0 con 0x7f91c80b2200 2024-03-31T15:06:26.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:26.467+0000 7f91cdc00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f91c80728b0 0x7f91c813f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:26.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:26.467+0000 7f91cdc00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f91c80728b0 0x7f91c813f0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36028/0 (socket says 172.21.15.16:36028) 2024-03-31T15:06:26.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:26.467+0000 7f91cdc00640 1 -- 172.21.15.16:0/459418050 learned_addr learned my addr 172.21.15.16:0/459418050 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:06:26.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:26.467+0000 7f91cd200640 1 --2- 172.21.15.16:0/459418050 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f91c80b2200 0x7f91c815bdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:26.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:26.467+0000 7f91ce600640 1 --2- 172.21.15.16:0/459418050 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f91c8145d10 0x7f91c815ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:26.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:26.467+0000 7f91cdc00640 1 -- 172.21.15.16:0/459418050 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f91c80b2200 msgr2=0x7f91c815bdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:26.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:26.467+0000 7f91cdc00640 1 --2- 172.21.15.16:0/459418050 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f91c80b2200 0x7f91c815bdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:26.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:26.467+0000 7f91cdc00640 1 -- 172.21.15.16:0/459418050 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f91c8145d10 msgr2=0x7f91c815ecd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:26.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:26.467+0000 7f91cdc00640 1 --2- 172.21.15.16:0/459418050 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f91c8145d10 0x7f91c815ecd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:26.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:26.467+0000 7f91cdc00640 1 -- 172.21.15.16:0/459418050 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f91c813b750 con 0x7f91c80728b0 2024-03-31T15:06:26.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:26.467+0000 7f91cd200640 1 --2- 172.21.15.16:0/459418050 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f91c80b2200 0x7f91c815bdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:06:26.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:26.467+0000 7f91cdc00640 1 --2- 172.21.15.16:0/459418050 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f91c80728b0 0x7f91c813f0e0 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f91bc00e930 tx=0x7f91bc00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:06:26.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:26.467+0000 7f91c5600640 1 -- 172.21.15.16:0/459418050 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91bc017070 con 0x7f91c80728b0 2024-03-31T15:06:26.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:26.467+0000 7f91c5600640 1 -- 172.21.15.16:0/459418050 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f91bc00ca00 con 0x7f91c80728b0 2024-03-31T15:06:26.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:26.467+0000 7f91c5600640 1 -- 172.21.15.16:0/459418050 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91bc00cdb0 con 0x7f91c80728b0 2024-03-31T15:06:26.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:26.467+0000 7f91d0600640 1 -- 172.21.15.16:0/459418050 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f91c813c4a0 con 0x7f91c80728b0 2024-03-31T15:06:26.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:26.467+0000 7f91d0600640 1 -- 172.21.15.16:0/459418050 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f91c8139fd0 con 0x7f91c80728b0 2024-03-31T15:06:26.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:26.467+0000 7f91d0600640 1 -- 172.21.15.16:0/459418050 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f91c8069950 con 0x7f91c80728b0 2024-03-31T15:06:26.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:26.471+0000 7f91c5600640 1 -- 172.21.15.16:0/459418050 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f91bc01a070 con 0x7f91c80728b0 2024-03-31T15:06:26.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:26.471+0000 7f91c5600640 1 --2- 172.21.15.16:0/459418050 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f91a8068380 0x7f91a806a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:26.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:26.471+0000 7f91c5600640 1 -- 172.21.15.16:0/459418050 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(55..55 src has 1..55) ==== 5709+0+0 (secure 0 0 0) 0x7f91bc086940 con 0x7f91c80728b0 2024-03-31T15:06:26.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:26.471+0000 7f91cd200640 1 --2- 172.21.15.16:0/459418050 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f91a8068380 0x7f91a806a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:26.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:26.471+0000 7f91cd200640 1 --2- 172.21.15.16:0/459418050 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f91a8068380 0x7f91a806a840 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f91b40ad000 tx=0x7f91b4008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:06:26.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:26.475+0000 7f91c5600640 1 -- 172.21.15.16:0/459418050 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f91bc057050 con 0x7f91c80728b0 2024-03-31T15:06:26.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:26.639+0000 7f91d0600640 1 -- 172.21.15.16:0/459418050 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f91c80b0650 con 0x7f91c80728b0 2024-03-31T15:06:26.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:26.639+0000 7f91c5600640 1 -- 172.21.15.16:0/459418050 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v55) ==== 74+0+16606 (secure 0 0 0) 0x7f91bc056950 con 0x7f91c80728b0 2024-03-31T15:06:26.634 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:06:26.635 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":55,"fsid":"286eb5da-fb47-4550-b290-d9519c13b9a5","created":"2024-03-31T15:03:11.438952+0000","modified":"2024-03-31T15:06:25.664685+0000","last_up_change":"2024-03-31T15:03:22.885535+0000","last_in_change":"2024-03-31T15:03:13.941667+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-31T15:03:22.550787+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-03-31T15:06:24.043886+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"55","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-03-31T15:06:24.613108+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"55","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"92a66450-ef19-40d8-ac3d-d760b23b0bda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6801","nonce":932712244}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6803","nonce":932712244}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6807","nonce":932712244}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6805","nonce":932712244}]},"public_addr":"172.21.15.16:6801/932712244","cluster_addr":"172.21.15.16:6803/932712244","heartbeat_back_addr":"172.21.15.16:6807/932712244","heartbeat_front_addr":"172.21.15.16:6805/932712244","state":["exists","up"]},{"osd":1,"uuid":"da9d1f2b-1968-4959-b843-faa231988e92","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6810","nonce":1398737533}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6811","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6813","nonce":1398737533}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6821","nonce":1398737533}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6815","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6816","nonce":1398737533}]},"public_addr":"172.21.15.16:6810/1398737533","cluster_addr":"172.21.15.16:6813/1398737533","heartbeat_back_addr":"172.21.15.16:6821/1398737533","heartbeat_front_addr":"172.21.15.16:6816/1398737533","state":["exists","up"]},{"osd":2,"uuid":"01a73662-9a21-4b1f-bcbb-42b9636a4638","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6819","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6822","nonce":3018526646}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6825","nonce":3018526646}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6831","nonce":3018526646}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6827","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6829","nonce":3018526646}]},"public_addr":"172.21.15.16:6822/3018526646","cluster_addr":"172.21.15.16:6825/3018526646","heartbeat_back_addr":"172.21.15.16:6831/3018526646","heartbeat_front_addr":"172.21.15.16:6829/3018526646","state":["exists","up"]},{"osd":3,"uuid":"c7771fd3-e18e-4500-ac38-23cf10ebcd75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6809","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6812","nonce":3360352974}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6817","nonce":3360352974}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6828","nonce":3360352974}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6823","nonce":3360352974}]},"public_addr":"172.21.15.16:6812/3360352974","cluster_addr":"172.21.15.16:6817/3360352974","heartbeat_back_addr":"172.21.15.16:6828/3360352974","heartbeat_front_addr":"172.21.15.16:6823/3360352974","state":["exists","up"]},{"osd":4,"uuid":"88260730-1482-4fbd-a40f-cb1124e9df4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6801","nonce":1856844856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6803","nonce":1856844856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6807","nonce":1856844856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6805","nonce":1856844856}]},"public_addr":"172.21.15.89:6801/1856844856","cluster_addr":"172.21.15.89:6803/1856844856","heartbeat_back_addr":"172.21.15.89:6807/1856844856","heartbeat_front_addr":"172.21.15.89:6805/1856844856","state":["exists","up"]},{"osd":5,"uuid":"4cb2c00c-fae0-4dd8-9d27-ec4f0093ccf7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6814","nonce":50395527}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6817","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6820","nonce":50395527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6831","nonce":50395527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6823","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6826","nonce":50395527}]},"public_addr":"172.21.15.89:6814/50395527","cluster_addr":"172.21.15.89:6820/50395527","heartbeat_back_addr":"172.21.15.89:6831/50395527","heartbeat_front_addr":"172.21.15.89:6826/50395527","state":["exists","up"]},{"osd":6,"uuid":"81ce0c19-2554-4cae-acbe-66a05e9e9b9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6809","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6812","nonce":3470471752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6815","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6818","nonce":3470471752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6827","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6829","nonce":3470471752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6821","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6824","nonce":3470471752}]},"public_addr":"172.21.15.89:6812/3470471752","cluster_addr":"172.21.15.89:6818/3470471752","heartbeat_back_addr":"172.21.15.89:6829/3470471752","heartbeat_front_addr":"172.21.15.89:6824/3470471752","state":["exists","up"]},{"osd":7,"uuid":"85e4313b-d37d-442e-8042-153bf20a847d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6811","nonce":1796778630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6813","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6816","nonce":1796778630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6825","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6828","nonce":1796778630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6819","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6822","nonce":1796778630}]},"public_addr":"172.21.15.89:6811/1796778630","cluster_addr":"172.21.15.89:6816/1796778630","heartbeat_back_addr":"172.21.15.89:6828/1796778630","heartbeat_front_addr":"172.21.15.89:6822/1796778630","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.439088+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.529970+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.680512+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:21.044161+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.582111+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.536558+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.409522+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.590229+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-31T15:06:26.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:26.643+0000 7f91d0600640 1 -- 172.21.15.16:0/459418050 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f91a8068380 msgr2=0x7f91a806a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:26.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:26.643+0000 7f91d0600640 1 --2- 172.21.15.16:0/459418050 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f91a8068380 0x7f91a806a840 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f91b40ad000 tx=0x7f91b4008040 comp rx=0 tx=0).stop 2024-03-31T15:06:26.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:26.643+0000 7f91d0600640 1 -- 172.21.15.16:0/459418050 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f91c80728b0 msgr2=0x7f91c813f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:26.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:26.643+0000 7f91d0600640 1 --2- 172.21.15.16:0/459418050 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f91c80728b0 0x7f91c813f0e0 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f91bc00e930 tx=0x7f91bc00edf0 comp rx=0 tx=0).stop 2024-03-31T15:06:26.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:26.643+0000 7f91d0600640 1 -- 172.21.15.16:0/459418050 shutdown_connections 2024-03-31T15:06:26.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:26.643+0000 7f91d0600640 1 --2- 172.21.15.16:0/459418050 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f91a8068380 0x7f91a806a840 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:26.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:26.643+0000 7f91d0600640 1 --2- 172.21.15.16:0/459418050 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f91c8145d10 0x7f91c815ecd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:26.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:26.643+0000 7f91d0600640 1 --2- 172.21.15.16:0/459418050 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f91c80b2200 0x7f91c815bdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:26.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:26.643+0000 7f91d0600640 1 --2- 172.21.15.16:0/459418050 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f91c80728b0 0x7f91c813f0e0 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:26.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:26.643+0000 7f91d0600640 1 -- 172.21.15.16:0/459418050 >> 172.21.15.16:0/459418050 conn(0x7f91c806b620 msgr2=0x7f91c80afba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:06:26.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:26.643+0000 7f91d0600640 1 -- 172.21.15.16:0/459418050 shutdown_connections 2024-03-31T15:06:26.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:26.643+0000 7f91d0600640 1 -- 172.21.15.16:0/459418050 wait complete. 2024-03-31T15:06:26.656 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-31T15:06:26.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:26.799+0000 7f19fce00640 1 Processor -- start 2024-03-31T15:06:26.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:26.803+0000 7f19fce00640 1 -- start start 2024-03-31T15:06:26.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:26.803+0000 7f19fce00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f19f806f920 0x7f19f806fd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:26.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:26.803+0000 7f19fce00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f19f80cc740 0x7f19f80ccbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:26.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:26.803+0000 7f19fce00640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f19f8162010 con 0x7f19f806d490 2024-03-31T15:06:26.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:26.803+0000 7f19fce00640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f19f8162be0 con 0x7f19f806f920 2024-03-31T15:06:26.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:26.803+0000 7f19fce00640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f19f8070240 con 0x7f19f80cc740 2024-03-31T15:06:26.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:26.803+0000 7f19f5e00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f19f806f920 0x7f19f806fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:26.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:26.803+0000 7f19f5e00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f19f806f920 0x7f19f806fd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36038/0 (socket says 172.21.15.16:36038) 2024-03-31T15:06:26.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:26.803+0000 7f19f5e00640 1 -- 172.21.15.16:0/2129365093 learned_addr learned my addr 172.21.15.16:0/2129365093 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:06:26.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:26.803+0000 7f19f6800640 1 --2- 172.21.15.16:0/2129365093 >> v2:172.21.15.89:3300/0 conn(0x7f19f80cc740 0x7f19f80ccbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:26.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:26.803+0000 7f19f5e00640 1 -- 172.21.15.16:0/2129365093 >> v2:172.21.15.89:3300/0 conn(0x7f19f80cc740 msgr2=0x7f19f80ccbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:06:26.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:26.803+0000 7f19f5e00640 1 --2- 172.21.15.16:0/2129365093 >> v2:172.21.15.89:3300/0 conn(0x7f19f80cc740 0x7f19f80ccbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:26.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:26.803+0000 7f19f5e00640 1 -- 172.21.15.16:0/2129365093 >> v1:172.21.15.89:6789/0 conn(0x7f19f806d490 legacy=0x7f19f806d8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:06:26.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:26.803+0000 7f19f5e00640 1 -- 172.21.15.16:0/2129365093 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19f81622b0 con 0x7f19f806f920 2024-03-31T15:06:26.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:26.803+0000 7f19f6800640 1 --2- 172.21.15.16:0/2129365093 >> v2:172.21.15.89:3300/0 conn(0x7f19f80cc740 0x7f19f80ccbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:06:26.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:26.807+0000 7f19f5e00640 1 --2- 172.21.15.16:0/2129365093 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f19f806f920 0x7f19f806fd00 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f19f8073cd0 tx=0x7f19e801c4e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=341dd0f2528d854 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:06:26.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:26.807+0000 7f19f5400640 1 -- 172.21.15.16:0/2129365093 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19e80b1070 con 0x7f19f806f920 2024-03-31T15:06:26.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:26.807+0000 7f19f5400640 1 -- 172.21.15.16:0/2129365093 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f19e80ac630 con 0x7f19f806f920 2024-03-31T15:06:26.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:26.807+0000 7f19f5400640 1 -- 172.21.15.16:0/2129365093 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19e80a8050 con 0x7f19f806f920 2024-03-31T15:06:26.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:26.807+0000 7f19fce00640 1 -- 172.21.15.16:0/2129365093 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f19f806f920 msgr2=0x7f19f806fd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:26.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:26.807+0000 7f19fce00640 1 --2- 172.21.15.16:0/2129365093 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f19f806f920 0x7f19f806fd00 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f19f8073cd0 tx=0x7f19e801c4e0 comp rx=0 tx=0).stop 2024-03-31T15:06:26.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:26.807+0000 7f19fce00640 1 -- 172.21.15.16:0/2129365093 shutdown_connections 2024-03-31T15:06:26.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:26.807+0000 7f19fce00640 1 --2- 172.21.15.16:0/2129365093 >> v2:172.21.15.89:3300/0 conn(0x7f19f80cc740 0x7f19f80ccbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:26.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:26.807+0000 7f19fce00640 1 --2- 172.21.15.16:0/2129365093 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f19f806f920 0x7f19f806fd00 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:26.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:26.807+0000 7f19fce00640 1 -- 172.21.15.16:0/2129365093 >> 172.21.15.16:0/2129365093 conn(0x7f19f8067330 msgr2=0x7f19f8067740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:06:26.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:26.807+0000 7f19fce00640 1 -- 172.21.15.16:0/2129365093 shutdown_connections 2024-03-31T15:06:26.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:26.807+0000 7f19fce00640 1 -- 172.21.15.16:0/2129365093 wait complete. 2024-03-31T15:06:26.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:26.807+0000 7f19fce00640 1 Processor -- start 2024-03-31T15:06:26.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:26.807+0000 7f19fce00640 1 -- start start 2024-03-31T15:06:26.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:26.811+0000 7f19fce00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f19f806d490 0x7f19f815c280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:26.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:26.811+0000 7f19fce00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f19f806f920 0x7f19f813f590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:26.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:26.811+0000 7f19fce00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f19f80cc740 0x7f19f813fad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:26.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:26.811+0000 7f19fce00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f19f816b530 con 0x7f19f806f920 2024-03-31T15:06:26.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:26.811+0000 7f19fce00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f19f816b3b0 con 0x7f19f806d490 2024-03-31T15:06:26.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:26.811+0000 7f19fce00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f19f816b6b0 con 0x7f19f80cc740 2024-03-31T15:06:26.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:26.811+0000 7f19efe00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f19f806f920 0x7f19f813f590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:26.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:26.811+0000 7f19efe00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f19f806f920 0x7f19f813f590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36048/0 (socket says 172.21.15.16:36048) 2024-03-31T15:06:26.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:26.811+0000 7f19efe00640 1 -- 172.21.15.16:0/3507892988 learned_addr learned my addr 172.21.15.16:0/3507892988 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:06:26.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:26.811+0000 7f19f5e00640 1 --2- 172.21.15.16:0/3507892988 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f19f806d490 0x7f19f815c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:26.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:26.811+0000 7f19f6800640 1 --2- 172.21.15.16:0/3507892988 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f19f80cc740 0x7f19f813fad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:26.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:26.811+0000 7f19efe00640 1 -- 172.21.15.16:0/3507892988 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f19f80cc740 msgr2=0x7f19f813fad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:26.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:26.811+0000 7f19efe00640 1 --2- 172.21.15.16:0/3507892988 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f19f80cc740 0x7f19f813fad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:26.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:26.811+0000 7f19efe00640 1 -- 172.21.15.16:0/3507892988 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f19f806d490 msgr2=0x7f19f815c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:26.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:26.811+0000 7f19efe00640 1 --2- 172.21.15.16:0/3507892988 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f19f806d490 0x7f19f815c280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:26.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:26.811+0000 7f19efe00640 1 -- 172.21.15.16:0/3507892988 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19f80b6600 con 0x7f19f806f920 2024-03-31T15:06:26.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:26.811+0000 7f19f5e00640 1 --2- 172.21.15.16:0/3507892988 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f19f806d490 0x7f19f815c280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:06:26.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:26.811+0000 7f19efe00640 1 --2- 172.21.15.16:0/3507892988 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f19f806f920 0x7f19f813f590 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f19e000b410 tx=0x7f19e000b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:06:26.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:26.811+0000 7f19ee000640 1 -- 172.21.15.16:0/3507892988 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19e0015030 con 0x7f19f806f920 2024-03-31T15:06:26.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:26.811+0000 7f19ee000640 1 -- 172.21.15.16:0/3507892988 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f19e0004020 con 0x7f19f806f920 2024-03-31T15:06:26.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:26.811+0000 7f19ee000640 1 -- 172.21.15.16:0/3507892988 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19e0004450 con 0x7f19f806f920 2024-03-31T15:06:26.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:26.811+0000 7f19fce00640 1 -- 172.21.15.16:0/3507892988 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f19f8160ca0 con 0x7f19f806f920 2024-03-31T15:06:26.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:26.811+0000 7f19fce00640 1 -- 172.21.15.16:0/3507892988 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f19f8161260 con 0x7f19f806f920 2024-03-31T15:06:26.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:26.811+0000 7f19fce00640 1 -- 172.21.15.16:0/3507892988 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f19f8161460 con 0x7f19f806f920 2024-03-31T15:06:26.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:26.811+0000 7f19ee000640 1 -- 172.21.15.16:0/3507892988 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f19e0018070 con 0x7f19f806f920 2024-03-31T15:06:26.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:26.815+0000 7f19ee000640 1 --2- 172.21.15.16:0/3507892988 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f19d4068380 0x7f19d406a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:26.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:26.815+0000 7f19ee000640 1 -- 172.21.15.16:0/3507892988 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(55..55 src has 1..55) ==== 5709+0+0 (secure 0 0 0) 0x7f19e0086490 con 0x7f19f806f920 2024-03-31T15:06:26.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:26.815+0000 7f19f5e00640 1 --2- 172.21.15.16:0/3507892988 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f19d4068380 0x7f19d406a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:26.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:26.815+0000 7f19f5e00640 1 --2- 172.21.15.16:0/3507892988 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f19d4068380 0x7f19d406a840 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f19e80ae000 tx=0x7f19e8008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:06:26.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:26.819+0000 7f19ee000640 1 -- 172.21.15.16:0/3507892988 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f19f8161460 con 0x7f19f806f920 2024-03-31T15:06:26.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:26.987+0000 7f19fce00640 1 -- 172.21.15.16:0/3507892988 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f19f8069950 con 0x7f19f806f920 2024-03-31T15:06:26.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:26.987+0000 7f19ee000640 1 -- 172.21.15.16:0/3507892988 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v55) ==== 74+0+16606 (secure 0 0 0) 0x7f19f80b6600 con 0x7f19f806f920 2024-03-31T15:06:26.982 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:06:26.982 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":55,"fsid":"286eb5da-fb47-4550-b290-d9519c13b9a5","created":"2024-03-31T15:03:11.438952+0000","modified":"2024-03-31T15:06:25.664685+0000","last_up_change":"2024-03-31T15:03:22.885535+0000","last_in_change":"2024-03-31T15:03:13.941667+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-31T15:03:22.550787+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-03-31T15:06:24.043886+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"55","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-03-31T15:06:24.613108+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"55","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"92a66450-ef19-40d8-ac3d-d760b23b0bda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6801","nonce":932712244}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6803","nonce":932712244}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6807","nonce":932712244}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6805","nonce":932712244}]},"public_addr":"172.21.15.16:6801/932712244","cluster_addr":"172.21.15.16:6803/932712244","heartbeat_back_addr":"172.21.15.16:6807/932712244","heartbeat_front_addr":"172.21.15.16:6805/932712244","state":["exists","up"]},{"osd":1,"uuid":"da9d1f2b-1968-4959-b843-faa231988e92","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6810","nonce":1398737533}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6811","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6813","nonce":1398737533}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6821","nonce":1398737533}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6815","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6816","nonce":1398737533}]},"public_addr":"172.21.15.16:6810/1398737533","cluster_addr":"172.21.15.16:6813/1398737533","heartbeat_back_addr":"172.21.15.16:6821/1398737533","heartbeat_front_addr":"172.21.15.16:6816/1398737533","state":["exists","up"]},{"osd":2,"uuid":"01a73662-9a21-4b1f-bcbb-42b9636a4638","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6819","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6822","nonce":3018526646}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6825","nonce":3018526646}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6831","nonce":3018526646}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6827","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6829","nonce":3018526646}]},"public_addr":"172.21.15.16:6822/3018526646","cluster_addr":"172.21.15.16:6825/3018526646","heartbeat_back_addr":"172.21.15.16:6831/3018526646","heartbeat_front_addr":"172.21.15.16:6829/3018526646","state":["exists","up"]},{"osd":3,"uuid":"c7771fd3-e18e-4500-ac38-23cf10ebcd75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6809","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6812","nonce":3360352974}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6817","nonce":3360352974}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6828","nonce":3360352974}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6823","nonce":3360352974}]},"public_addr":"172.21.15.16:6812/3360352974","cluster_addr":"172.21.15.16:6817/3360352974","heartbeat_back_addr":"172.21.15.16:6828/3360352974","heartbeat_front_addr":"172.21.15.16:6823/3360352974","state":["exists","up"]},{"osd":4,"uuid":"88260730-1482-4fbd-a40f-cb1124e9df4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6801","nonce":1856844856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6803","nonce":1856844856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6807","nonce":1856844856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6805","nonce":1856844856}]},"public_addr":"172.21.15.89:6801/1856844856","cluster_addr":"172.21.15.89:6803/1856844856","heartbeat_back_addr":"172.21.15.89:6807/1856844856","heartbeat_front_addr":"172.21.15.89:6805/1856844856","state":["exists","up"]},{"osd":5,"uuid":"4cb2c00c-fae0-4dd8-9d27-ec4f0093ccf7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6814","nonce":50395527}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6817","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6820","nonce":50395527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6831","nonce":50395527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6823","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6826","nonce":50395527}]},"public_addr":"172.21.15.89:6814/50395527","cluster_addr":"172.21.15.89:6820/50395527","heartbeat_back_addr":"172.21.15.89:6831/50395527","heartbeat_front_addr":"172.21.15.89:6826/50395527","state":["exists","up"]},{"osd":6,"uuid":"81ce0c19-2554-4cae-acbe-66a05e9e9b9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6809","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6812","nonce":3470471752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6815","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6818","nonce":3470471752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6827","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6829","nonce":3470471752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6821","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6824","nonce":3470471752}]},"public_addr":"172.21.15.89:6812/3470471752","cluster_addr":"172.21.15.89:6818/3470471752","heartbeat_back_addr":"172.21.15.89:6829/3470471752","heartbeat_front_addr":"172.21.15.89:6824/3470471752","state":["exists","up"]},{"osd":7,"uuid":"85e4313b-d37d-442e-8042-153bf20a847d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6811","nonce":1796778630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6813","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6816","nonce":1796778630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6825","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6828","nonce":1796778630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6819","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6822","nonce":1796778630}]},"public_addr":"172.21.15.89:6811/1796778630","cluster_addr":"172.21.15.89:6816/1796778630","heartbeat_back_addr":"172.21.15.89:6828/1796778630","heartbeat_front_addr":"172.21.15.89:6822/1796778630","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.439088+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.529970+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.680512+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:21.044161+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.582111+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.536558+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.409522+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.590229+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-31T15:06:26.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:26.987+0000 7f19fce00640 1 -- 172.21.15.16:0/3507892988 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f19d4068380 msgr2=0x7f19d406a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:26.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:26.987+0000 7f19fce00640 1 --2- 172.21.15.16:0/3507892988 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f19d4068380 0x7f19d406a840 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f19e80ae000 tx=0x7f19e8008040 comp rx=0 tx=0).stop 2024-03-31T15:06:26.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:26.987+0000 7f19fce00640 1 -- 172.21.15.16:0/3507892988 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f19f806f920 msgr2=0x7f19f813f590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:26.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:26.987+0000 7f19fce00640 1 --2- 172.21.15.16:0/3507892988 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f19f806f920 0x7f19f813f590 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f19e000b410 tx=0x7f19e000b8d0 comp rx=0 tx=0).stop 2024-03-31T15:06:26.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:26.987+0000 7f19fce00640 1 -- 172.21.15.16:0/3507892988 shutdown_connections 2024-03-31T15:06:26.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:26.987+0000 7f19fce00640 1 --2- 172.21.15.16:0/3507892988 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f19d4068380 0x7f19d406a840 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:26.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:26.987+0000 7f19fce00640 1 --2- 172.21.15.16:0/3507892988 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f19f80cc740 0x7f19f813fad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:26.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:26.987+0000 7f19fce00640 1 --2- 172.21.15.16:0/3507892988 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f19f806f920 0x7f19f813f590 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:26.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:26.987+0000 7f19fce00640 1 --2- 172.21.15.16:0/3507892988 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f19f806d490 0x7f19f815c280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:26.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:26.987+0000 7f19fce00640 1 -- 172.21.15.16:0/3507892988 >> 172.21.15.16:0/3507892988 conn(0x7f19f8067330 msgr2=0x7f19f80b3630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:06:26.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:26.987+0000 7f19fce00640 1 -- 172.21.15.16:0/3507892988 shutdown_connections 2024-03-31T15:06:26.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:26.987+0000 7f19fce00640 1 -- 172.21.15.16:0/3507892988 wait complete. 2024-03-31T15:06:27.003 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-31T15:06:27.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:27.151+0000 7f5375a00640 1 Processor -- start 2024-03-31T15:06:27.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:27.151+0000 7f5375a00640 1 -- start start 2024-03-31T15:06:27.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:27.151+0000 7f5375a00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f53700cda50 0x7f537015f6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:27.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:27.151+0000 7f5375a00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f53700cde30 0x7f53700cf660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:27.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:27.151+0000 7f5375a00640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f53700add10 con 0x7f53700ad8d0 2024-03-31T15:06:27.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:27.151+0000 7f5375a00640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f53700cec10 con 0x7f53700cda50 2024-03-31T15:06:27.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:27.151+0000 7f5375a00640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f53700cf1d0 con 0x7f53700cde30 2024-03-31T15:06:27.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:27.151+0000 7f536ea00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f53700cda50 0x7f537015f6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:27.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:27.151+0000 7f536ea00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f53700cda50 0x7f537015f6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48974/0 (socket says 172.21.15.16:48974) 2024-03-31T15:06:27.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:27.151+0000 7f536ea00640 1 -- 172.21.15.16:0/429577556 learned_addr learned my addr 172.21.15.16:0/429577556 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:06:27.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:27.151+0000 7f536f400640 1 --2- 172.21.15.16:0/429577556 >> v2:172.21.15.89:3300/0 conn(0x7f53700cde30 0x7f53700cf660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:27.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:27.151+0000 7f536ea00640 1 -- 172.21.15.16:0/429577556 >> v2:172.21.15.89:3300/0 conn(0x7f53700cde30 msgr2=0x7f53700cf660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:06:27.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:27.151+0000 7f536ea00640 1 --2- 172.21.15.16:0/429577556 >> v2:172.21.15.89:3300/0 conn(0x7f53700cde30 0x7f53700cf660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:27.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:27.151+0000 7f536ea00640 1 -- 172.21.15.16:0/429577556 >> v1:172.21.15.89:6789/0 conn(0x7f53700ad8d0 legacy=0x7f537006de10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:06:27.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:27.151+0000 7f536ea00640 1 -- 172.21.15.16:0/429577556 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f537006e5f0 con 0x7f53700cda50 2024-03-31T15:06:27.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:27.151+0000 7f536f400640 1 --2- 172.21.15.16:0/429577556 >> v2:172.21.15.89:3300/0 conn(0x7f53700cde30 0x7f53700cf660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:06:27.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:27.155+0000 7f536ea00640 1 --2- 172.21.15.16:0/429577556 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f53700cda50 0x7f537015f6c0 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f53700af930 tx=0x7f536001ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97b6c7e8632429df server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:06:27.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:27.155+0000 7f536e000640 1 -- 172.21.15.16:0/429577556 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53600b5070 con 0x7f53700cda50 2024-03-31T15:06:27.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:27.155+0000 7f536e000640 1 -- 172.21.15.16:0/429577556 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f53600a89d0 con 0x7f53700cda50 2024-03-31T15:06:27.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:27.155+0000 7f536e000640 1 -- 172.21.15.16:0/429577556 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53600a8da0 con 0x7f53700cda50 2024-03-31T15:06:27.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:27.155+0000 7f5375a00640 1 -- 172.21.15.16:0/429577556 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f53700cda50 msgr2=0x7f537015f6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:27.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:27.155+0000 7f5375a00640 1 --2- 172.21.15.16:0/429577556 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f53700cda50 0x7f537015f6c0 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f53700af930 tx=0x7f536001ecd0 comp rx=0 tx=0).stop 2024-03-31T15:06:27.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:27.155+0000 7f5375a00640 1 -- 172.21.15.16:0/429577556 shutdown_connections 2024-03-31T15:06:27.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:27.155+0000 7f5375a00640 1 --2- 172.21.15.16:0/429577556 >> v2:172.21.15.89:3300/0 conn(0x7f53700cde30 0x7f53700cf660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:27.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:27.155+0000 7f5375a00640 1 --2- 172.21.15.16:0/429577556 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f53700cda50 0x7f537015f6c0 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:27.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:27.155+0000 7f5375a00640 1 -- 172.21.15.16:0/429577556 >> 172.21.15.16:0/429577556 conn(0x7f53700b2db0 msgr2=0x7f53700cd340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:06:27.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:27.155+0000 7f5375a00640 1 -- 172.21.15.16:0/429577556 shutdown_connections 2024-03-31T15:06:27.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:27.155+0000 7f5375a00640 1 -- 172.21.15.16:0/429577556 wait complete. 2024-03-31T15:06:27.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:27.159+0000 7f5375a00640 1 Processor -- start 2024-03-31T15:06:27.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:27.159+0000 7f5375a00640 1 -- start start 2024-03-31T15:06:27.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:27.159+0000 7f5375a00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f53700ad8d0 0x7f537015ab90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:27.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:27.159+0000 7f5375a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f53700cda50 0x7f53701440c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:27.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:27.159+0000 7f5375a00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f53700cde30 0x7f5370157d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:27.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:27.159+0000 7f5375a00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f53700b39a0 con 0x7f53700cda50 2024-03-31T15:06:27.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:27.159+0000 7f5375a00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f537015e960 con 0x7f53700ad8d0 2024-03-31T15:06:27.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:27.159+0000 7f5375a00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f53700cee60 con 0x7f53700cde30 2024-03-31T15:06:27.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:27.159+0000 7f5367e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f53700cda50 0x7f53701440c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:27.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:27.159+0000 7f5367e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f53700cda50 0x7f53701440c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48976/0 (socket says 172.21.15.16:48976) 2024-03-31T15:06:27.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:27.159+0000 7f5367e00640 1 -- 172.21.15.16:0/735124376 learned_addr learned my addr 172.21.15.16:0/735124376 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:06:27.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:27.159+0000 7f536ea00640 1 --2- 172.21.15.16:0/735124376 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f53700ad8d0 0x7f537015ab90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:27.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:27.159+0000 7f536f400640 1 --2- 172.21.15.16:0/735124376 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f53700cde30 0x7f5370157d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:27.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:27.159+0000 7f5367e00640 1 -- 172.21.15.16:0/735124376 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f53700cde30 msgr2=0x7f5370157d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:27.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:27.159+0000 7f5367e00640 1 --2- 172.21.15.16:0/735124376 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f53700cde30 0x7f5370157d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:27.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:27.159+0000 7f5367e00640 1 -- 172.21.15.16:0/735124376 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f53700ad8d0 msgr2=0x7f537015ab90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:27.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:27.159+0000 7f5367e00640 1 --2- 172.21.15.16:0/735124376 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f53700ad8d0 0x7f537015ab90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:27.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:27.159+0000 7f5367e00640 1 -- 172.21.15.16:0/735124376 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5370159e10 con 0x7f53700cda50 2024-03-31T15:06:27.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:27.159+0000 7f5367e00640 1 --2- 172.21.15.16:0/735124376 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f53700cda50 0x7f53701440c0 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f535800c410 tx=0x7f535800c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:06:27.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:27.159+0000 7f5366a00640 1 -- 172.21.15.16:0/735124376 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5358011790 con 0x7f53700cda50 2024-03-31T15:06:27.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:27.159+0000 7f5366a00640 1 -- 172.21.15.16:0/735124376 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5358017070 con 0x7f53700cda50 2024-03-31T15:06:27.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:27.159+0000 7f5375a00640 1 -- 172.21.15.16:0/735124376 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f537015b0d0 con 0x7f53700cda50 2024-03-31T15:06:27.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:27.159+0000 7f5366a00640 1 -- 172.21.15.16:0/735124376 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5358011bb0 con 0x7f53700cda50 2024-03-31T15:06:27.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:27.159+0000 7f5375a00640 1 -- 172.21.15.16:0/735124376 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f53701586c0 con 0x7f53700cda50 2024-03-31T15:06:27.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:27.163+0000 7f5375a00640 1 -- 172.21.15.16:0/735124376 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5370069950 con 0x7f53700cda50 2024-03-31T15:06:27.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:27.163+0000 7f5366a00640 1 -- 172.21.15.16:0/735124376 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f535801c460 con 0x7f53700cda50 2024-03-31T15:06:27.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:27.163+0000 7f5366a00640 1 --2- 172.21.15.16:0/735124376 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f534c0683b0 0x7f534c06a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:27.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:27.163+0000 7f5366a00640 1 -- 172.21.15.16:0/735124376 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(55..55 src has 1..55) ==== 5709+0+0 (secure 0 0 0) 0x7f53580867f0 con 0x7f53700cda50 2024-03-31T15:06:27.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:27.163+0000 7f536ea00640 1 --2- 172.21.15.16:0/735124376 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f534c0683b0 0x7f534c06a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:27.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:27.163+0000 7f536ea00640 1 --2- 172.21.15.16:0/735124376 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f534c0683b0 0x7f534c06a870 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f53600b2000 tx=0x7f5360009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:06:27.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:27.167+0000 7f5366a00640 1 -- 172.21.15.16:0/735124376 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f5358003810 con 0x7f53700cda50 2024-03-31T15:06:27.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:27.371+0000 7f5375a00640 1 -- 172.21.15.16:0/735124376 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f53700aeb60 con 0x7f53700cda50 2024-03-31T15:06:28.231 INFO:tasks.ceph.mds.c.smithi016.stderr:2024-03-31T15:06:28.235+0000 7f4ef6400640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-31T15:06:28.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:28.283+0000 7f5366a00640 1 -- 172.21.15.16:0/735124376 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v33) ==== 118+0+0 (secure 0 0 0) 0x7f5358056d10 con 0x7f53700cda50 2024-03-31T15:06:28.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:28.287+0000 7f5375a00640 1 -- 172.21.15.16:0/735124376 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f534c0683b0 msgr2=0x7f534c06a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:28.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:28.287+0000 7f5375a00640 1 --2- 172.21.15.16:0/735124376 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f534c0683b0 0x7f534c06a870 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f53600b2000 tx=0x7f5360009040 comp rx=0 tx=0).stop 2024-03-31T15:06:28.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:28.287+0000 7f5375a00640 1 -- 172.21.15.16:0/735124376 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f53700cda50 msgr2=0x7f53701440c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:28.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:28.287+0000 7f5375a00640 1 --2- 172.21.15.16:0/735124376 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f53700cda50 0x7f53701440c0 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f535800c410 tx=0x7f535800c8d0 comp rx=0 tx=0).stop 2024-03-31T15:06:28.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:28.287+0000 7f5375a00640 1 -- 172.21.15.16:0/735124376 shutdown_connections 2024-03-31T15:06:28.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:28.287+0000 7f5375a00640 1 --2- 172.21.15.16:0/735124376 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f534c0683b0 0x7f534c06a870 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:28.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:28.287+0000 7f5375a00640 1 --2- 172.21.15.16:0/735124376 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f53700cde30 0x7f5370157d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:28.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:28.287+0000 7f5375a00640 1 --2- 172.21.15.16:0/735124376 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f53700cda50 0x7f53701440c0 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:28.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:28.287+0000 7f5375a00640 1 --2- 172.21.15.16:0/735124376 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f53700ad8d0 0x7f537015ab90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:28.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:28.287+0000 7f5375a00640 1 -- 172.21.15.16:0/735124376 >> 172.21.15.16:0/735124376 conn(0x7f53700b2db0 msgr2=0x7f537015c930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:06:28.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:28.287+0000 7f5375a00640 1 -- 172.21.15.16:0/735124376 shutdown_connections 2024-03-31T15:06:28.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:28.287+0000 7f5375a00640 1 -- 172.21.15.16:0/735124376 wait complete. 2024-03-31T15:06:28.301 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-03-31T15:06:28.301 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-31T15:06:28.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:28.455+0000 7ffa7be00640 1 Processor -- start 2024-03-31T15:06:28.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:28.455+0000 7ffa7be00640 1 -- start start 2024-03-31T15:06:28.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:28.455+0000 7ffa7be00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ffa7c0d07e0 0x7ffa7c06e170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:28.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:28.455+0000 7ffa7be00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7ffa7c06e6b0 0x7ffa7c06ca10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:28.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:28.455+0000 7ffa7be00640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffa7c0cfd20 con 0x7ffa7c06cf50 2024-03-31T15:06:28.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:28.455+0000 7ffa7be00640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7ffa7c0cd890 con 0x7ffa7c0d07e0 2024-03-31T15:06:28.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:28.455+0000 7ffa7be00640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7ffa7c0cdb30 con 0x7ffa7c06e6b0 2024-03-31T15:06:28.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:28.455+0000 7ffa7aa00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ffa7c0d07e0 0x7ffa7c06e170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:28.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:28.455+0000 7ffa7aa00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ffa7c0d07e0 0x7ffa7c06e170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48978/0 (socket says 172.21.15.16:48978) 2024-03-31T15:06:28.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:28.455+0000 7ffa7aa00640 1 -- 172.21.15.16:0/1863211904 learned_addr learned my addr 172.21.15.16:0/1863211904 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:06:28.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:28.455+0000 7ffa72000640 1 --2- 172.21.15.16:0/1863211904 >> v2:172.21.15.89:3300/0 conn(0x7ffa7c06e6b0 0x7ffa7c06ca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:28.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:28.455+0000 7ffa7aa00640 1 -- 172.21.15.16:0/1863211904 >> v2:172.21.15.89:3300/0 conn(0x7ffa7c06e6b0 msgr2=0x7ffa7c06ca10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:06:28.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:28.455+0000 7ffa7aa00640 1 --2- 172.21.15.16:0/1863211904 >> v2:172.21.15.89:3300/0 conn(0x7ffa7c06e6b0 0x7ffa7c06ca10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:28.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:28.455+0000 7ffa7aa00640 1 -- 172.21.15.16:0/1863211904 >> v1:172.21.15.16:6789/0 conn(0x7ffa7c06cf50 legacy=0x7ffa7c15ebd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:28.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:28.455+0000 7ffa7aa00640 1 -- 172.21.15.16:0/1863211904 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa7c0cde10 con 0x7ffa7c0d07e0 2024-03-31T15:06:28.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:28.455+0000 7ffa7a000640 1 -- 172.21.15.16:0/1863211904 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 651549993 0 0) 0x7ffa74001120 con 0x7ffa7c06cf50 2024-03-31T15:06:28.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:28.463+0000 7ffa7aa00640 1 --2- 172.21.15.16:0/1863211904 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffa7c0d07e0 0x7ffa7c06e170 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7ffa7c0734b0 tx=0x7ffa6c01ebf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb83a31070835ea server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:06:28.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:28.463+0000 7ffa7a000640 1 -- 172.21.15.16:0/1863211904 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa6c0b4070 con 0x7ffa7c0d07e0 2024-03-31T15:06:28.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:28.463+0000 7ffa7a000640 1 -- 172.21.15.16:0/1863211904 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffa6c0a7ab0 con 0x7ffa7c0d07e0 2024-03-31T15:06:28.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:28.463+0000 7ffa7a000640 1 -- 172.21.15.16:0/1863211904 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa6c0af440 con 0x7ffa7c0d07e0 2024-03-31T15:06:28.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:28.463+0000 7ffa7be00640 1 -- 172.21.15.16:0/1863211904 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffa7c0d07e0 msgr2=0x7ffa7c06e170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:28.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:28.463+0000 7ffa7be00640 1 --2- 172.21.15.16:0/1863211904 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffa7c0d07e0 0x7ffa7c06e170 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7ffa7c0734b0 tx=0x7ffa6c01ebf0 comp rx=0 tx=0).stop 2024-03-31T15:06:28.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:28.463+0000 7ffa7be00640 1 -- 172.21.15.16:0/1863211904 shutdown_connections 2024-03-31T15:06:28.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:28.463+0000 7ffa7be00640 1 --2- 172.21.15.16:0/1863211904 >> v2:172.21.15.89:3300/0 conn(0x7ffa7c06e6b0 0x7ffa7c06ca10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:28.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:28.463+0000 7ffa7be00640 1 --2- 172.21.15.16:0/1863211904 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffa7c0d07e0 0x7ffa7c06e170 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:28.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:28.463+0000 7ffa7be00640 1 -- 172.21.15.16:0/1863211904 >> 172.21.15.16:0/1863211904 conn(0x7ffa7c0b2c90 msgr2=0x7ffa7c0b30a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:06:28.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:28.463+0000 7ffa7be00640 1 -- 172.21.15.16:0/1863211904 shutdown_connections 2024-03-31T15:06:28.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:28.463+0000 7ffa7be00640 1 -- 172.21.15.16:0/1863211904 wait complete. 2024-03-31T15:06:28.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:28.463+0000 7ffa7be00640 1 Processor -- start 2024-03-31T15:06:28.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:28.463+0000 7ffa7be00640 1 -- start start 2024-03-31T15:06:28.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:28.463+0000 7ffa7be00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ffa7c06cf50 0x7ffa7c1595a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:28.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:28.463+0000 7ffa7be00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffa7c06e6b0 0x7ffa7c06bbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:28.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:28.463+0000 7ffa7be00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffa7c0d07e0 0x7ffa7c147750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:28.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:28.463+0000 7ffa7be00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7ffa7c06ed40 con 0x7ffa7c06e6b0 2024-03-31T15:06:28.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:28.463+0000 7ffa7be00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7ffa7c06ebc0 con 0x7ffa7c0d07e0 2024-03-31T15:06:28.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:28.463+0000 7ffa7be00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7ffa7c15c380 con 0x7ffa7c06cf50 2024-03-31T15:06:28.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:28.463+0000 7ffa72000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffa7c06e6b0 0x7ffa7c06bbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:28.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:28.467+0000 7ffa72000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffa7c06e6b0 0x7ffa7c06bbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48980/0 (socket says 172.21.15.16:48980) 2024-03-31T15:06:28.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:28.467+0000 7ffa72000640 1 -- 172.21.15.16:0/4012224463 learned_addr learned my addr 172.21.15.16:0/4012224463 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:06:28.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:28.467+0000 7ffa7aa00640 1 --2- 172.21.15.16:0/4012224463 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ffa7c06cf50 0x7ffa7c1595a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:28.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:28.467+0000 7ffa7b400640 1 --2- 172.21.15.16:0/4012224463 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffa7c0d07e0 0x7ffa7c147750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:28.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:28.467+0000 7ffa72000640 1 -- 172.21.15.16:0/4012224463 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ffa7c06cf50 msgr2=0x7ffa7c1595a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:28.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:28.467+0000 7ffa72000640 1 --2- 172.21.15.16:0/4012224463 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ffa7c06cf50 0x7ffa7c1595a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:28.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:28.467+0000 7ffa72000640 1 -- 172.21.15.16:0/4012224463 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffa7c0d07e0 msgr2=0x7ffa7c147750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:28.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:28.467+0000 7ffa72000640 1 --2- 172.21.15.16:0/4012224463 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffa7c0d07e0 0x7ffa7c147750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:28.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:28.467+0000 7ffa72000640 1 -- 172.21.15.16:0/4012224463 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa7c158500 con 0x7ffa7c06e6b0 2024-03-31T15:06:28.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:28.467+0000 7ffa72000640 1 --2- 172.21.15.16:0/4012224463 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffa7c06e6b0 0x7ffa7c06bbc0 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7ffa64009fd0 tx=0x7ffa6400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:06:28.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:28.467+0000 7ffa73400640 1 -- 172.21.15.16:0/4012224463 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa64014620 con 0x7ffa7c06e6b0 2024-03-31T15:06:28.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:28.467+0000 7ffa73400640 1 -- 172.21.15.16:0/4012224463 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffa64019070 con 0x7ffa7c06e6b0 2024-03-31T15:06:28.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:28.467+0000 7ffa73400640 1 -- 172.21.15.16:0/4012224463 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa64014a00 con 0x7ffa7c06e6b0 2024-03-31T15:06:28.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:28.467+0000 7ffa7be00640 1 -- 172.21.15.16:0/4012224463 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffa7c1587e0 con 0x7ffa7c06e6b0 2024-03-31T15:06:28.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:28.467+0000 7ffa7be00640 1 -- 172.21.15.16:0/4012224463 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffa7c158da0 con 0x7ffa7c06e6b0 2024-03-31T15:06:28.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:28.467+0000 7ffa7be00640 1 -- 172.21.15.16:0/4012224463 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffa7c04f8c0 con 0x7ffa7c06e6b0 2024-03-31T15:06:28.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:28.467+0000 7ffa73400640 1 -- 172.21.15.16:0/4012224463 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7ffa64014ba0 con 0x7ffa7c06e6b0 2024-03-31T15:06:28.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:28.471+0000 7ffa73400640 1 --2- 172.21.15.16:0/4012224463 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7ffa580683b0 0x7ffa5806a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:28.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:28.471+0000 7ffa73400640 1 -- 172.21.15.16:0/4012224463 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 5709+0+0 (secure 0 0 0) 0x7ffa64087290 con 0x7ffa7c06e6b0 2024-03-31T15:06:28.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:28.471+0000 7ffa7aa00640 1 --2- 172.21.15.16:0/4012224463 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7ffa580683b0 0x7ffa5806a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:28.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:28.471+0000 7ffa7aa00640 1 --2- 172.21.15.16:0/4012224463 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7ffa580683b0 0x7ffa5806a870 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7ffa6c0b1000 tx=0x7ffa6c0a9070 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:06:28.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:28.471+0000 7ffa73400640 1 -- 172.21.15.16:0/4012224463 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7ffa640577b0 con 0x7ffa7c06e6b0 2024-03-31T15:06:28.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:28.703+0000 7ffa7be00640 1 -- 172.21.15.16:0/4012224463 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7ffa7c06fa00 con 0x7ffa7c06e6b0 2024-03-31T15:06:29.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:29.287+0000 7ffa73400640 1 -- 172.21.15.16:0/4012224463 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v34) ==== 122+0+0 (secure 0 0 0) 0x7ffa640570b0 con 0x7ffa7c06e6b0 2024-03-31T15:06:29.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:29.291+0000 7ffa7be00640 1 -- 172.21.15.16:0/4012224463 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7ffa580683b0 msgr2=0x7ffa5806a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:29.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:29.291+0000 7ffa7be00640 1 --2- 172.21.15.16:0/4012224463 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7ffa580683b0 0x7ffa5806a870 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7ffa6c0b1000 tx=0x7ffa6c0a9070 comp rx=0 tx=0).stop 2024-03-31T15:06:29.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:29.291+0000 7ffa7be00640 1 -- 172.21.15.16:0/4012224463 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffa7c06e6b0 msgr2=0x7ffa7c06bbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:29.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:29.291+0000 7ffa7be00640 1 --2- 172.21.15.16:0/4012224463 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffa7c06e6b0 0x7ffa7c06bbc0 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7ffa64009fd0 tx=0x7ffa6400e4b0 comp rx=0 tx=0).stop 2024-03-31T15:06:29.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:29.291+0000 7ffa7be00640 1 -- 172.21.15.16:0/4012224463 shutdown_connections 2024-03-31T15:06:29.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:29.291+0000 7ffa7be00640 1 --2- 172.21.15.16:0/4012224463 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7ffa580683b0 0x7ffa5806a870 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:29.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:29.291+0000 7ffa7be00640 1 --2- 172.21.15.16:0/4012224463 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffa7c0d07e0 0x7ffa7c147750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:29.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:29.291+0000 7ffa7be00640 1 --2- 172.21.15.16:0/4012224463 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffa7c06e6b0 0x7ffa7c06bbc0 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:29.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:29.291+0000 7ffa7be00640 1 --2- 172.21.15.16:0/4012224463 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ffa7c06cf50 0x7ffa7c1595a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:29.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:29.291+0000 7ffa7be00640 1 -- 172.21.15.16:0/4012224463 >> 172.21.15.16:0/4012224463 conn(0x7ffa7c0b2c90 msgr2=0x7ffa7c0afbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:06:29.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:29.291+0000 7ffa7be00640 1 -- 172.21.15.16:0/4012224463 shutdown_connections 2024-03-31T15:06:29.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:29.291+0000 7ffa7be00640 1 -- 172.21.15.16:0/4012224463 wait complete. 2024-03-31T15:06:29.313 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-31T15:06:29.314 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-31T15:06:29.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:29.459+0000 7f3310e00640 1 Processor -- start 2024-03-31T15:06:29.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:29.459+0000 7f3310e00640 1 -- start start 2024-03-31T15:06:29.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:29.463+0000 7f3310e00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f330c145e40 0x7f330c0cc740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:29.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:29.463+0000 7f3310e00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f330c0ccc80 0x7f330c06f920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:29.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:29.463+0000 7f3310e00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f330c06fe60 0x7f330c15fdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:29.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:29.463+0000 7f3310e00640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f330c0d06d0 con 0x7f330c145e40 2024-03-31T15:06:29.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:29.463+0000 7f3310e00640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f330c0d0920 con 0x7f330c0ccc80 2024-03-31T15:06:29.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:29.463+0000 7f3310e00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f330c0d0c00 con 0x7f330c06fe60 2024-03-31T15:06:29.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:29.463+0000 7f3309e00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f330c145e40 0x7f330c0cc740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:29.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:29.463+0000 7f3309e00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f330c145e40 0x7f330c0cc740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48984/0 (socket says 172.21.15.16:48984) 2024-03-31T15:06:29.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:29.463+0000 7f3309e00640 1 -- 172.21.15.16:0/4222425657 learned_addr learned my addr 172.21.15.16:0/4222425657 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:06:29.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:29.463+0000 7f3303e00640 1 --2- 172.21.15.16:0/4222425657 >> v2:172.21.15.89:3300/0 conn(0x7f330c0ccc80 0x7f330c06f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:29.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:29.463+0000 7f330a800640 1 --2- 172.21.15.16:0/4222425657 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f330c06fe60 0x7f330c15fdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:29.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:29.463+0000 7f3309e00640 1 -- 172.21.15.16:0/4222425657 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f330c06fe60 msgr2=0x7f330c15fdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:06:29.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:29.463+0000 7f3309e00640 1 --2- 172.21.15.16:0/4222425657 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f330c06fe60 0x7f330c15fdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:29.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:29.463+0000 7f3309e00640 1 -- 172.21.15.16:0/4222425657 >> v2:172.21.15.89:3300/0 conn(0x7f330c0ccc80 msgr2=0x7f330c06f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:06:29.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:29.463+0000 7f3309e00640 1 --2- 172.21.15.16:0/4222425657 >> v2:172.21.15.89:3300/0 conn(0x7f330c0ccc80 0x7f330c06f920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:29.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:29.463+0000 7f3309e00640 1 -- 172.21.15.16:0/4222425657 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f330c0d1810 con 0x7f330c145e40 2024-03-31T15:06:29.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:29.467+0000 7f3309e00640 1 --2- 172.21.15.16:0/4222425657 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f330c145e40 0x7f330c0cc740 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f330c073cf0 tx=0x7f32fc01c260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3176c1e6118d7485 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:06:29.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:29.467+0000 7f3309400640 1 -- 172.21.15.16:0/4222425657 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32fc0b0070 con 0x7f330c145e40 2024-03-31T15:06:29.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:29.467+0000 7f3309400640 1 -- 172.21.15.16:0/4222425657 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f32fc0ab600 con 0x7f330c145e40 2024-03-31T15:06:29.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:29.467+0000 7f3309400640 1 -- 172.21.15.16:0/4222425657 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32fc0ab9d0 con 0x7f330c145e40 2024-03-31T15:06:29.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:29.467+0000 7f3310e00640 1 -- 172.21.15.16:0/4222425657 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f330c145e40 msgr2=0x7f330c0cc740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:29.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:29.467+0000 7f3310e00640 1 --2- 172.21.15.16:0/4222425657 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f330c145e40 0x7f330c0cc740 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f330c073cf0 tx=0x7f32fc01c260 comp rx=0 tx=0).stop 2024-03-31T15:06:29.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:29.467+0000 7f3310e00640 1 -- 172.21.15.16:0/4222425657 shutdown_connections 2024-03-31T15:06:29.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:29.467+0000 7f3310e00640 1 --2- 172.21.15.16:0/4222425657 >> v2:172.21.15.89:3300/0 conn(0x7f330c0ccc80 0x7f330c06f920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:29.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:29.467+0000 7f3310e00640 1 --2- 172.21.15.16:0/4222425657 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f330c06fe60 0x7f330c15fdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:29.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:29.467+0000 7f3310e00640 1 --2- 172.21.15.16:0/4222425657 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f330c145e40 0x7f330c0cc740 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:29.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:29.467+0000 7f3310e00640 1 -- 172.21.15.16:0/4222425657 >> 172.21.15.16:0/4222425657 conn(0x7f330c067330 msgr2=0x7f330c067740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:06:29.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:29.467+0000 7f3310e00640 1 -- 172.21.15.16:0/4222425657 shutdown_connections 2024-03-31T15:06:29.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:29.467+0000 7f3310e00640 1 -- 172.21.15.16:0/4222425657 wait complete. 2024-03-31T15:06:29.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:29.467+0000 7f3310e00640 1 Processor -- start 2024-03-31T15:06:29.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:29.467+0000 7f3310e00640 1 -- start start 2024-03-31T15:06:29.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:29.471+0000 7f3310e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f330c06fe60 0x7f330c06e3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:29.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:29.471+0000 7f3310e00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f330c0ccc80 0x7f330c145160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:29.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:29.471+0000 7f3310e00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f330c145e40 0x7f330c14f580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:29.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:29.471+0000 7f3310e00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f330c0b5240 con 0x7f330c06fe60 2024-03-31T15:06:29.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:29.471+0000 7f3310e00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f330c0b50c0 con 0x7f330c145e40 2024-03-31T15:06:29.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:29.471+0000 7f3310e00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f330c0d2010 con 0x7f330c0ccc80 2024-03-31T15:06:29.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:29.471+0000 7f3309e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f330c06fe60 0x7f330c06e3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:29.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:29.471+0000 7f3309e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f330c06fe60 0x7f330c06e3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48990/0 (socket says 172.21.15.16:48990) 2024-03-31T15:06:29.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:29.471+0000 7f3309e00640 1 -- 172.21.15.16:0/3630109187 learned_addr learned my addr 172.21.15.16:0/3630109187 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:06:29.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:29.471+0000 7f3309e00640 1 -- 172.21.15.16:0/3630109187 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f330c0ccc80 msgr2=0x7f330c145160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:29.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:29.471+0000 7f3309e00640 1 --2- 172.21.15.16:0/3630109187 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f330c0ccc80 0x7f330c145160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:29.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:29.471+0000 7f3309e00640 1 -- 172.21.15.16:0/3630109187 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f330c145e40 msgr2=0x7f330c14f580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:29.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:29.471+0000 7f3309e00640 1 --2- 172.21.15.16:0/3630109187 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f330c145e40 0x7f330c14f580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:29.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:29.471+0000 7f3309e00640 1 -- 172.21.15.16:0/3630109187 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f330c06d910 con 0x7f330c06fe60 2024-03-31T15:06:29.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:29.471+0000 7f3309e00640 1 --2- 172.21.15.16:0/3630109187 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f330c06fe60 0x7f330c06e3b0 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f32fc0ae040 tx=0x7f32fc01aaa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:06:29.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:29.471+0000 7f3302000640 1 -- 172.21.15.16:0/3630109187 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32fc0b0040 con 0x7f330c06fe60 2024-03-31T15:06:29.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:29.471+0000 7f3302000640 1 -- 172.21.15.16:0/3630109187 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f32fc0bc070 con 0x7f330c06fe60 2024-03-31T15:06:29.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:29.471+0000 7f3310e00640 1 -- 172.21.15.16:0/3630109187 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f330c157970 con 0x7f330c06fe60 2024-03-31T15:06:29.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:29.471+0000 7f3302000640 1 -- 172.21.15.16:0/3630109187 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32fc0b6680 con 0x7f330c06fe60 2024-03-31T15:06:29.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:29.471+0000 7f3310e00640 1 -- 172.21.15.16:0/3630109187 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f330c06dbb0 con 0x7f330c06fe60 2024-03-31T15:06:29.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:29.471+0000 7f3310e00640 1 -- 172.21.15.16:0/3630109187 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f330c069950 con 0x7f330c06fe60 2024-03-31T15:06:29.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:29.471+0000 7f3302000640 1 -- 172.21.15.16:0/3630109187 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f32fc0b6820 con 0x7f330c06fe60 2024-03-31T15:06:29.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:29.475+0000 7f3302000640 1 --2- 172.21.15.16:0/3630109187 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f32e8068400 0x7f32e806a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:29.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:29.475+0000 7f3302000640 1 -- 172.21.15.16:0/3630109187 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 5709+0+0 (secure 0 0 0) 0x7f32fc120c70 con 0x7f330c06fe60 2024-03-31T15:06:29.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:29.475+0000 7f3303e00640 1 --2- 172.21.15.16:0/3630109187 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f32e8068400 0x7f32e806a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:29.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:29.475+0000 7f3303e00640 1 --2- 172.21.15.16:0/3630109187 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f32e8068400 0x7f32e806a8c0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f330c0b5f50 tx=0x7f32f4009520 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:06:29.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:29.483+0000 7f3302000640 1 -- 172.21.15.16:0/3630109187 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f32fc016110 con 0x7f330c06fe60 2024-03-31T15:06:29.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:29.683+0000 7f3310e00640 1 -- 172.21.15.16:0/3630109187 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f330c0cf3c0 con 0x7f330c06fe60 2024-03-31T15:06:29.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:29.683+0000 7f3302000640 1 -- 172.21.15.16:0/3630109187 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 34 v34) ==== 94+0+5181 (secure 0 0 0) 0x7f32fc0f12d0 con 0x7f330c06fe60 2024-03-31T15:06:29.679 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:06:29.679 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":34,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5579,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6833/3721971977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":3721971977},{"type":"v1","addr":"172.21.15.16:6833","nonce":3721971977}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":29},{"gid":6109,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6836/1166206553","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":1166206553},{"type":"v1","addr":"172.21.15.89:6836","nonce":1166206553}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28},{"gid":6112,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/1826113083","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6835","nonce":1826113083},{"type":"v1","addr":"172.21.15.89:6837","nonce":1826113083}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":29}],"filesystems":[{"mdsmap":{"epoch":34,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-31T15:06:25.665001+0000","modified":"2024-03-31T15:06:28.707528+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6150},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6150":{"gid":6150,"name":"c","rank":0,"incarnation":30,"state":"up:active","state_seq":3,"addr":"172.21.15.16:6835/2613117528","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":2613117528},{"type":"v1","addr":"172.21.15.16:6835","nonce":2613117528}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6150,"qdb_cluster":[6150]},"id":4}]} 2024-03-31T15:06:29.680 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 34 2024-03-31T15:06:29.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:29.687+0000 7f3310e00640 1 -- 172.21.15.16:0/3630109187 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f32e8068400 msgr2=0x7f32e806a8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:29.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:29.687+0000 7f3310e00640 1 --2- 172.21.15.16:0/3630109187 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f32e8068400 0x7f32e806a8c0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f330c0b5f50 tx=0x7f32f4009520 comp rx=0 tx=0).stop 2024-03-31T15:06:29.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:29.687+0000 7f3310e00640 1 -- 172.21.15.16:0/3630109187 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f330c06fe60 msgr2=0x7f330c06e3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:29.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:29.687+0000 7f3310e00640 1 --2- 172.21.15.16:0/3630109187 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f330c06fe60 0x7f330c06e3b0 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f32fc0ae040 tx=0x7f32fc01aaa0 comp rx=0 tx=0).stop 2024-03-31T15:06:29.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:29.687+0000 7f3310e00640 1 -- 172.21.15.16:0/3630109187 shutdown_connections 2024-03-31T15:06:29.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:29.687+0000 7f3310e00640 1 --2- 172.21.15.16:0/3630109187 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f32e8068400 0x7f32e806a8c0 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:29.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:29.687+0000 7f3310e00640 1 --2- 172.21.15.16:0/3630109187 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f330c145e40 0x7f330c14f580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:29.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:29.687+0000 7f3310e00640 1 --2- 172.21.15.16:0/3630109187 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f330c0ccc80 0x7f330c145160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:29.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:29.687+0000 7f3310e00640 1 --2- 172.21.15.16:0/3630109187 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f330c06fe60 0x7f330c06e3b0 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:29.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:29.687+0000 7f3310e00640 1 -- 172.21.15.16:0/3630109187 >> 172.21.15.16:0/3630109187 conn(0x7f330c067330 msgr2=0x7f330c15ccd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:06:29.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:29.687+0000 7f3310e00640 1 -- 172.21.15.16:0/3630109187 shutdown_connections 2024-03-31T15:06:29.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:29.687+0000 7f3310e00640 1 -- 172.21.15.16:0/3630109187 wait complete. 2024-03-31T15:06:29.700 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-31T15:06:29.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:29.847+0000 7f3b92000640 1 Processor -- start 2024-03-31T15:06:29.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:29.847+0000 7f3b92000640 1 -- start start 2024-03-31T15:06:29.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:29.847+0000 7f3b92000640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f3b8c145d10 0x7f3b8c0cf790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:29.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:29.847+0000 7f3b92000640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f3b8c0728b0 0x7f3b8c0b1cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:29.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:29.847+0000 7f3b92000640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f3b8c0b2200 0x7f3b8c15cd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:29.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:29.847+0000 7f3b92000640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f3b8c160e70 con 0x7f3b8c145d10 2024-03-31T15:06:29.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:29.847+0000 7f3b92000640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f3b8c0cfcd0 con 0x7f3b8c0728b0 2024-03-31T15:06:29.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:29.847+0000 7f3b92000640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f3b8c1602f0 con 0x7f3b8c0b2200 2024-03-31T15:06:29.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:29.847+0000 7f3b8b400640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f3b8c145d10 0x7f3b8c0cf790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:29.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:29.847+0000 7f3b8b400640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f3b8c145d10 0x7f3b8c0cf790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49000/0 (socket says 172.21.15.16:49000) 2024-03-31T15:06:29.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:29.847+0000 7f3b8b400640 1 -- 172.21.15.16:0/2350143789 learned_addr learned my addr 172.21.15.16:0/2350143789 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:06:29.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:29.847+0000 7f3b83e00640 1 --2- 172.21.15.16:0/2350143789 >> v2:172.21.15.89:3300/0 conn(0x7f3b8c0728b0 0x7f3b8c0b1cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:29.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:29.847+0000 7f3b8be00640 1 --2- 172.21.15.16:0/2350143789 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f3b8c0b2200 0x7f3b8c15cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:29.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:29.847+0000 7f3b8b400640 1 -- 172.21.15.16:0/2350143789 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f3b8c0b2200 msgr2=0x7f3b8c15cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:06:29.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:29.847+0000 7f3b8b400640 1 --2- 172.21.15.16:0/2350143789 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f3b8c0b2200 0x7f3b8c15cd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:29.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:29.847+0000 7f3b8b400640 1 -- 172.21.15.16:0/2350143789 >> v2:172.21.15.89:3300/0 conn(0x7f3b8c0728b0 msgr2=0x7f3b8c0b1cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:06:29.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:29.847+0000 7f3b8b400640 1 --2- 172.21.15.16:0/2350143789 >> v2:172.21.15.89:3300/0 conn(0x7f3b8c0728b0 0x7f3b8c0b1cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:29.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:29.847+0000 7f3b8b400640 1 -- 172.21.15.16:0/2350143789 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b8c1605d0 con 0x7f3b8c145d10 2024-03-31T15:06:29.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:29.847+0000 7f3b83e00640 1 --2- 172.21.15.16:0/2350143789 >> v2:172.21.15.89:3300/0 conn(0x7f3b8c0728b0 0x7f3b8c0b1cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:06:29.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:29.847+0000 7f3b8be00640 1 --2- 172.21.15.16:0/2350143789 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f3b8c0b2200 0x7f3b8c15cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:06:29.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:29.851+0000 7f3b8b400640 1 --2- 172.21.15.16:0/2350143789 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3b8c145d10 0x7f3b8c0cf790 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f3b8c0b2980 tx=0x7f3b7c01c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a5c7c43d254cf7a server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:06:29.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:29.851+0000 7f3b8aa00640 1 -- 172.21.15.16:0/2350143789 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b7c0b0070 con 0x7f3b8c145d10 2024-03-31T15:06:29.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:29.851+0000 7f3b8aa00640 1 -- 172.21.15.16:0/2350143789 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3b7c0ab600 con 0x7f3b8c145d10 2024-03-31T15:06:29.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:29.851+0000 7f3b8aa00640 1 -- 172.21.15.16:0/2350143789 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b7c0ab9d0 con 0x7f3b8c145d10 2024-03-31T15:06:29.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:29.851+0000 7f3b92000640 1 -- 172.21.15.16:0/2350143789 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3b8c145d10 msgr2=0x7f3b8c0cf790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:29.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:29.851+0000 7f3b92000640 1 --2- 172.21.15.16:0/2350143789 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3b8c145d10 0x7f3b8c0cf790 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f3b8c0b2980 tx=0x7f3b7c01c520 comp rx=0 tx=0).stop 2024-03-31T15:06:29.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:29.851+0000 7f3b92000640 1 -- 172.21.15.16:0/2350143789 shutdown_connections 2024-03-31T15:06:29.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:29.851+0000 7f3b92000640 1 --2- 172.21.15.16:0/2350143789 >> v2:172.21.15.89:3300/0 conn(0x7f3b8c0728b0 0x7f3b8c0b1cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:29.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:29.851+0000 7f3b92000640 1 --2- 172.21.15.16:0/2350143789 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f3b8c0b2200 0x7f3b8c15cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:29.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:29.851+0000 7f3b92000640 1 --2- 172.21.15.16:0/2350143789 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3b8c145d10 0x7f3b8c0cf790 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:29.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:29.851+0000 7f3b92000640 1 -- 172.21.15.16:0/2350143789 >> 172.21.15.16:0/2350143789 conn(0x7f3b8c06b620 msgr2=0x7f3b8c06ba30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:06:29.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:29.851+0000 7f3b92000640 1 -- 172.21.15.16:0/2350143789 shutdown_connections 2024-03-31T15:06:29.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:29.851+0000 7f3b92000640 1 -- 172.21.15.16:0/2350143789 wait complete. 2024-03-31T15:06:29.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:29.855+0000 7f3b92000640 1 Processor -- start 2024-03-31T15:06:29.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:29.855+0000 7f3b92000640 1 -- start start 2024-03-31T15:06:29.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:29.859+0000 7f3b92000640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f3b8c0728b0 0x7f3b8c15b670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:29.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:29.859+0000 7f3b92000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3b8c0b2200 0x7f3b8c0b47a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:29.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:29.859+0000 7f3b92000640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3b8c145d10 0x7f3b8c1654a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:29.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:29.859+0000 7f3b92000640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f3b8c0cf350 con 0x7f3b8c0b2200 2024-03-31T15:06:29.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:29.859+0000 7f3b92000640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f3b8c0b2610 con 0x7f3b8c145d10 2024-03-31T15:06:29.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:29.859+0000 7f3b92000640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f3b8c0cf4d0 con 0x7f3b8c0728b0 2024-03-31T15:06:29.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:29.859+0000 7f3b83e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3b8c0b2200 0x7f3b8c0b47a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:29.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:29.859+0000 7f3b83e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3b8c0b2200 0x7f3b8c0b47a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49016/0 (socket says 172.21.15.16:49016) 2024-03-31T15:06:29.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:29.859+0000 7f3b83e00640 1 -- 172.21.15.16:0/3393759987 learned_addr learned my addr 172.21.15.16:0/3393759987 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:06:29.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:29.859+0000 7f3b8be00640 1 --2- 172.21.15.16:0/3393759987 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3b8c145d10 0x7f3b8c1654a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:29.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:29.859+0000 7f3b8b400640 1 --2- 172.21.15.16:0/3393759987 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f3b8c0728b0 0x7f3b8c15b670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:29.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:29.859+0000 7f3b83e00640 1 -- 172.21.15.16:0/3393759987 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f3b8c0728b0 msgr2=0x7f3b8c15b670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:29.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:29.859+0000 7f3b83e00640 1 --2- 172.21.15.16:0/3393759987 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f3b8c0728b0 0x7f3b8c15b670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:29.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:29.859+0000 7f3b83e00640 1 -- 172.21.15.16:0/3393759987 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3b8c145d10 msgr2=0x7f3b8c1654a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:29.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:29.859+0000 7f3b83e00640 1 --2- 172.21.15.16:0/3393759987 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3b8c145d10 0x7f3b8c1654a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:29.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:29.859+0000 7f3b83e00640 1 -- 172.21.15.16:0/3393759987 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b8c0b4ce0 con 0x7f3b8c0b2200 2024-03-31T15:06:29.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:29.859+0000 7f3b8be00640 1 --2- 172.21.15.16:0/3393759987 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3b8c145d10 0x7f3b8c1654a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:06:29.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:29.859+0000 7f3b83e00640 1 --2- 172.21.15.16:0/3393759987 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3b8c0b2200 0x7f3b8c0b47a0 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f3b7400c7f0 tx=0x7f3b7400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:06:29.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:29.859+0000 7f3b83400640 1 -- 172.21.15.16:0/3393759987 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b74015070 con 0x7f3b8c0b2200 2024-03-31T15:06:29.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:29.859+0000 7f3b83400640 1 -- 172.21.15.16:0/3393759987 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3b74007820 con 0x7f3b8c0b2200 2024-03-31T15:06:29.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:29.859+0000 7f3b92000640 1 -- 172.21.15.16:0/3393759987 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3b8c140610 con 0x7f3b8c0b2200 2024-03-31T15:06:29.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:29.859+0000 7f3b83400640 1 -- 172.21.15.16:0/3393759987 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b74007c10 con 0x7f3b8c0b2200 2024-03-31T15:06:29.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:29.859+0000 7f3b92000640 1 -- 172.21.15.16:0/3393759987 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3b8c0b3910 con 0x7f3b8c0b2200 2024-03-31T15:06:29.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:29.859+0000 7f3b92000640 1 -- 172.21.15.16:0/3393759987 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3b8c069950 con 0x7f3b8c0b2200 2024-03-31T15:06:29.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:29.859+0000 7f3b83400640 1 -- 172.21.15.16:0/3393759987 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f3b74018070 con 0x7f3b8c0b2200 2024-03-31T15:06:29.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:29.863+0000 7f3b83400640 1 --2- 172.21.15.16:0/3393759987 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f3b68068380 0x7f3b6806a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:29.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:29.863+0000 7f3b83400640 1 -- 172.21.15.16:0/3393759987 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 5709+0+0 (secure 0 0 0) 0x7f3b74086be0 con 0x7f3b8c0b2200 2024-03-31T15:06:29.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:29.863+0000 7f3b8b400640 1 --2- 172.21.15.16:0/3393759987 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f3b68068380 0x7f3b6806a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:29.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:29.863+0000 7f3b8b400640 1 --2- 172.21.15.16:0/3393759987 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f3b68068380 0x7f3b6806a840 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f3b7c0ad000 tx=0x7f3b7c008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:06:29.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:29.867+0000 7f3b83400640 1 -- 172.21.15.16:0/3393759987 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f3b74057070 con 0x7f3b8c0b2200 2024-03-31T15:06:30.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:30.031+0000 7f3b92000640 1 -- 172.21.15.16:0/3393759987 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f3b8c0b4f10 con 0x7f3b8c0b2200 2024-03-31T15:06:30.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:30.031+0000 7f3b83400640 1 -- 172.21.15.16:0/3393759987 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v56) ==== 74+0+16593 (secure 0 0 0) 0x7f3b74056970 con 0x7f3b8c0b2200 2024-03-31T15:06:30.028 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:06:30.028 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":56,"fsid":"286eb5da-fb47-4550-b290-d9519c13b9a5","created":"2024-03-31T15:03:11.438952+0000","modified":"2024-03-31T15:06:27.274379+0000","last_up_change":"2024-03-31T15:03:22.885535+0000","last_in_change":"2024-03-31T15:03:13.941667+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-31T15:03:22.550787+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-03-31T15:06:24.043886+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"55","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-03-31T15:06:24.613108+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"56","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"92a66450-ef19-40d8-ac3d-d760b23b0bda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6801","nonce":932712244}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6803","nonce":932712244}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6807","nonce":932712244}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6805","nonce":932712244}]},"public_addr":"172.21.15.16:6801/932712244","cluster_addr":"172.21.15.16:6803/932712244","heartbeat_back_addr":"172.21.15.16:6807/932712244","heartbeat_front_addr":"172.21.15.16:6805/932712244","state":["exists","up"]},{"osd":1,"uuid":"da9d1f2b-1968-4959-b843-faa231988e92","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6810","nonce":1398737533}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6811","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6813","nonce":1398737533}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6821","nonce":1398737533}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6815","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6816","nonce":1398737533}]},"public_addr":"172.21.15.16:6810/1398737533","cluster_addr":"172.21.15.16:6813/1398737533","heartbeat_back_addr":"172.21.15.16:6821/1398737533","heartbeat_front_addr":"172.21.15.16:6816/1398737533","state":["exists","up"]},{"osd":2,"uuid":"01a73662-9a21-4b1f-bcbb-42b9636a4638","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6819","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6822","nonce":3018526646}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6825","nonce":3018526646}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6831","nonce":3018526646}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6827","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6829","nonce":3018526646}]},"public_addr":"172.21.15.16:6822/3018526646","cluster_addr":"172.21.15.16:6825/3018526646","heartbeat_back_addr":"172.21.15.16:6831/3018526646","heartbeat_front_addr":"172.21.15.16:6829/3018526646","state":["exists","up"]},{"osd":3,"uuid":"c7771fd3-e18e-4500-ac38-23cf10ebcd75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6809","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6812","nonce":3360352974}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6817","nonce":3360352974}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6828","nonce":3360352974}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6823","nonce":3360352974}]},"public_addr":"172.21.15.16:6812/3360352974","cluster_addr":"172.21.15.16:6817/3360352974","heartbeat_back_addr":"172.21.15.16:6828/3360352974","heartbeat_front_addr":"172.21.15.16:6823/3360352974","state":["exists","up"]},{"osd":4,"uuid":"88260730-1482-4fbd-a40f-cb1124e9df4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6801","nonce":1856844856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6803","nonce":1856844856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6807","nonce":1856844856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6805","nonce":1856844856}]},"public_addr":"172.21.15.89:6801/1856844856","cluster_addr":"172.21.15.89:6803/1856844856","heartbeat_back_addr":"172.21.15.89:6807/1856844856","heartbeat_front_addr":"172.21.15.89:6805/1856844856","state":["exists","up"]},{"osd":5,"uuid":"4cb2c00c-fae0-4dd8-9d27-ec4f0093ccf7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6814","nonce":50395527}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6817","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6820","nonce":50395527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6831","nonce":50395527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6823","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6826","nonce":50395527}]},"public_addr":"172.21.15.89:6814/50395527","cluster_addr":"172.21.15.89:6820/50395527","heartbeat_back_addr":"172.21.15.89:6831/50395527","heartbeat_front_addr":"172.21.15.89:6826/50395527","state":["exists","up"]},{"osd":6,"uuid":"81ce0c19-2554-4cae-acbe-66a05e9e9b9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6809","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6812","nonce":3470471752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6815","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6818","nonce":3470471752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6827","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6829","nonce":3470471752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6821","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6824","nonce":3470471752}]},"public_addr":"172.21.15.89:6812/3470471752","cluster_addr":"172.21.15.89:6818/3470471752","heartbeat_back_addr":"172.21.15.89:6829/3470471752","heartbeat_front_addr":"172.21.15.89:6824/3470471752","state":["exists","up"]},{"osd":7,"uuid":"85e4313b-d37d-442e-8042-153bf20a847d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6811","nonce":1796778630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6813","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6816","nonce":1796778630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6825","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6828","nonce":1796778630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6819","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6822","nonce":1796778630}]},"public_addr":"172.21.15.89:6811/1796778630","cluster_addr":"172.21.15.89:6816/1796778630","heartbeat_back_addr":"172.21.15.89:6828/1796778630","heartbeat_front_addr":"172.21.15.89:6822/1796778630","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.439088+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.529970+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.680512+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:21.044161+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.582111+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.536558+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.409522+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.590229+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-31T15:06:30.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:30.035+0000 7f3b92000640 1 -- 172.21.15.16:0/3393759987 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f3b68068380 msgr2=0x7f3b6806a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:30.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:30.035+0000 7f3b92000640 1 --2- 172.21.15.16:0/3393759987 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f3b68068380 0x7f3b6806a840 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f3b7c0ad000 tx=0x7f3b7c008040 comp rx=0 tx=0).stop 2024-03-31T15:06:30.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:30.035+0000 7f3b92000640 1 -- 172.21.15.16:0/3393759987 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3b8c0b2200 msgr2=0x7f3b8c0b47a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:30.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:30.035+0000 7f3b92000640 1 --2- 172.21.15.16:0/3393759987 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3b8c0b2200 0x7f3b8c0b47a0 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f3b7400c7f0 tx=0x7f3b7400ccb0 comp rx=0 tx=0).stop 2024-03-31T15:06:30.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:30.035+0000 7f3b92000640 1 -- 172.21.15.16:0/3393759987 shutdown_connections 2024-03-31T15:06:30.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:30.035+0000 7f3b92000640 1 --2- 172.21.15.16:0/3393759987 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f3b68068380 0x7f3b6806a840 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:30.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:30.035+0000 7f3b92000640 1 --2- 172.21.15.16:0/3393759987 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3b8c145d10 0x7f3b8c1654a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:30.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:30.035+0000 7f3b92000640 1 --2- 172.21.15.16:0/3393759987 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3b8c0b2200 0x7f3b8c0b47a0 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:30.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:30.035+0000 7f3b92000640 1 --2- 172.21.15.16:0/3393759987 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f3b8c0728b0 0x7f3b8c15b670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:30.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:30.035+0000 7f3b92000640 1 -- 172.21.15.16:0/3393759987 >> 172.21.15.16:0/3393759987 conn(0x7f3b8c06b620 msgr2=0x7f3b8c0b0260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:06:30.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:30.035+0000 7f3b92000640 1 -- 172.21.15.16:0/3393759987 shutdown_connections 2024-03-31T15:06:30.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:30.035+0000 7f3b92000640 1 -- 172.21.15.16:0/3393759987 wait complete. 2024-03-31T15:06:30.058 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-31T15:06:30.059 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-31T15:06:30.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:30.203+0000 7faf69a00640 1 Processor -- start 2024-03-31T15:06:30.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:30.203+0000 7faf69a00640 1 -- start start 2024-03-31T15:06:30.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:30.203+0000 7faf69a00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7faf640728b0 0x7faf640b1cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:30.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:30.203+0000 7faf69a00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7faf640b2200 0x7faf6415cd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:30.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:30.203+0000 7faf69a00640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faf64160e70 con 0x7faf64145d10 2024-03-31T15:06:30.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:30.203+0000 7faf69a00640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7faf641602f0 con 0x7faf640b2200 2024-03-31T15:06:30.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:30.203+0000 7faf69a00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7faf641605d0 con 0x7faf640728b0 2024-03-31T15:06:30.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:30.203+0000 7faf63400640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7faf640b2200 0x7faf6415cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:30.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:30.203+0000 7faf63400640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7faf640b2200 0x7faf6415cd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.16:38854/0 (socket says 172.21.15.16:38854) 2024-03-31T15:06:30.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:30.203+0000 7faf63400640 1 -- 172.21.15.16:0/2796753291 learned_addr learned my addr 172.21.15.16:0/2796753291 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:06:30.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:30.203+0000 7faf5be00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7faf640728b0 0x7faf640b1cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:30.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:30.203+0000 7faf62a00640 1 --1- 172.21.15.16:0/2796753291 >> v1:172.21.15.89:6789/0 conn(0x7faf64145d10 0x7faf640cf790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.16:50928/0 (socket says 172.21.15.16:50928) 2024-03-31T15:06:30.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:30.207+0000 7faf63400640 1 -- 172.21.15.16:0/2796753291 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7faf640728b0 msgr2=0x7faf640b1cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:06:30.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:30.207+0000 7faf63400640 1 --2- 172.21.15.16:0/2796753291 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7faf640728b0 0x7faf640b1cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:30.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:30.207+0000 7faf63400640 1 -- 172.21.15.16:0/2796753291 >> v1:172.21.15.89:6789/0 conn(0x7faf64145d10 legacy=0x7faf640cf790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:30.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:30.207+0000 7faf63400640 1 -- 172.21.15.16:0/2796753291 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf641608b0 con 0x7faf640b2200 2024-03-31T15:06:30.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:30.207+0000 7faf63400640 1 --2- 172.21.15.16:0/2796753291 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faf640b2200 0x7faf6415cd50 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7faf5c009e50 tx=0x7faf5c01ca30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f4210adbfa1f8425 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:06:30.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:30.207+0000 7faf62000640 1 -- 172.21.15.16:0/2796753291 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf5c0a5700 con 0x7faf640b2200 2024-03-31T15:06:30.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:30.207+0000 7faf62000640 1 -- 172.21.15.16:0/2796753291 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faf5c0a58a0 con 0x7faf640b2200 2024-03-31T15:06:30.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:30.207+0000 7faf62000640 1 -- 172.21.15.16:0/2796753291 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf5c0a5c70 con 0x7faf640b2200 2024-03-31T15:06:30.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:30.211+0000 7faf69a00640 1 -- 172.21.15.16:0/2796753291 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faf640b2200 msgr2=0x7faf6415cd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:30.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:30.211+0000 7faf69a00640 1 --2- 172.21.15.16:0/2796753291 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faf640b2200 0x7faf6415cd50 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7faf5c009e50 tx=0x7faf5c01ca30 comp rx=0 tx=0).stop 2024-03-31T15:06:30.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:30.211+0000 7faf69a00640 1 -- 172.21.15.16:0/2796753291 shutdown_connections 2024-03-31T15:06:30.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:30.211+0000 7faf69a00640 1 --2- 172.21.15.16:0/2796753291 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faf640b2200 0x7faf6415cd50 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:30.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:30.211+0000 7faf69a00640 1 --2- 172.21.15.16:0/2796753291 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7faf640728b0 0x7faf640b1cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:30.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:30.211+0000 7faf69a00640 1 -- 172.21.15.16:0/2796753291 >> 172.21.15.16:0/2796753291 conn(0x7faf6406b620 msgr2=0x7faf6406ba30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:06:30.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:30.211+0000 7faf69a00640 1 -- 172.21.15.16:0/2796753291 shutdown_connections 2024-03-31T15:06:30.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:30.211+0000 7faf69a00640 1 -- 172.21.15.16:0/2796753291 wait complete. 2024-03-31T15:06:30.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:30.211+0000 7faf69a00640 1 Processor -- start 2024-03-31T15:06:30.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:30.211+0000 7faf69a00640 1 -- start start 2024-03-31T15:06:30.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:30.211+0000 7faf69a00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faf640728b0 0x7faf6413c810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:30.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:30.211+0000 7faf69a00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7faf640b2200 0x7faf64138ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:30.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:30.211+0000 7faf69a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faf64145d10 0x7faf641654a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:30.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:30.211+0000 7faf69a00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7faf64072e10 con 0x7faf64145d10 2024-03-31T15:06:30.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:30.211+0000 7faf69a00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7faf64072c90 con 0x7faf640728b0 2024-03-31T15:06:30.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:30.211+0000 7faf69a00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7faf641690f0 con 0x7faf640b2200 2024-03-31T15:06:30.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:30.211+0000 7faf63400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faf64145d10 0x7faf641654a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:30.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:30.211+0000 7faf63400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faf64145d10 0x7faf641654a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49030/0 (socket says 172.21.15.16:49030) 2024-03-31T15:06:30.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:30.211+0000 7faf63400640 1 -- 172.21.15.16:0/455255137 learned_addr learned my addr 172.21.15.16:0/455255137 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:06:30.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:30.211+0000 7faf62a00640 1 --2- 172.21.15.16:0/455255137 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faf640728b0 0x7faf6413c810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:30.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:30.211+0000 7faf5be00640 1 --2- 172.21.15.16:0/455255137 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7faf640b2200 0x7faf64138ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:30.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:30.211+0000 7faf63400640 1 -- 172.21.15.16:0/455255137 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7faf640b2200 msgr2=0x7faf64138ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:30.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:30.211+0000 7faf63400640 1 --2- 172.21.15.16:0/455255137 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7faf640b2200 0x7faf64138ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:30.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:30.211+0000 7faf63400640 1 -- 172.21.15.16:0/455255137 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faf640728b0 msgr2=0x7faf6413c810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:30.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:30.211+0000 7faf63400640 1 --2- 172.21.15.16:0/455255137 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faf640728b0 0x7faf6413c810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:30.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:30.211+0000 7faf63400640 1 -- 172.21.15.16:0/455255137 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf6413f690 con 0x7faf64145d10 2024-03-31T15:06:30.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:30.211+0000 7faf63400640 1 --2- 172.21.15.16:0/455255137 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faf64145d10 0x7faf641654a0 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7faf5c01a660 tx=0x7faf5c0197d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:06:30.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:30.215+0000 7faf5aa00640 1 -- 172.21.15.16:0/455255137 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf5c009510 con 0x7faf64145d10 2024-03-31T15:06:30.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:30.215+0000 7faf5aa00640 1 -- 172.21.15.16:0/455255137 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faf5c0be070 con 0x7faf64145d10 2024-03-31T15:06:30.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:30.215+0000 7faf5aa00640 1 -- 172.21.15.16:0/455255137 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf5c0b88d0 con 0x7faf64145d10 2024-03-31T15:06:30.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:30.215+0000 7faf69a00640 1 -- 172.21.15.16:0/455255137 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faf6413cd50 con 0x7faf64145d10 2024-03-31T15:06:30.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:30.215+0000 7faf69a00640 1 -- 172.21.15.16:0/455255137 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faf6413ebb0 con 0x7faf64145d10 2024-03-31T15:06:30.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:30.215+0000 7faf69a00640 1 -- 172.21.15.16:0/455255137 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faf64069950 con 0x7faf64145d10 2024-03-31T15:06:30.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:30.215+0000 7faf5aa00640 1 -- 172.21.15.16:0/455255137 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7faf5c0b8a70 con 0x7faf64145d10 2024-03-31T15:06:30.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:30.215+0000 7faf5aa00640 1 --2- 172.21.15.16:0/455255137 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7faf40068380 0x7faf4006a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:30.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:30.215+0000 7faf5aa00640 1 -- 172.21.15.16:0/455255137 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 5709+0+0 (secure 0 0 0) 0x7faf5c121900 con 0x7faf64145d10 2024-03-31T15:06:30.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:30.215+0000 7faf62a00640 1 --2- 172.21.15.16:0/455255137 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7faf40068380 0x7faf4006a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:30.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:30.219+0000 7faf62a00640 1 --2- 172.21.15.16:0/455255137 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7faf40068380 0x7faf4006a840 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7faf64166360 tx=0x7faf54007e90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:06:30.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:30.223+0000 7faf5aa00640 1 -- 172.21.15.16:0/455255137 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7faf5c016620 con 0x7faf64145d10 2024-03-31T15:06:30.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:30.395+0000 7faf69a00640 1 -- 172.21.15.16:0/455255137 --> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7faf64073ab0 con 0x7faf40068380 2024-03-31T15:06:30.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:30.403+0000 7faf5aa00640 1 -- 172.21.15.16:0/455255137 <== mgr.4100 v2:172.21.15.89:6832/1549407273 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+244909 (secure 0 0 0) 0x7faf540074f0 con 0x7faf40068380 2024-03-31T15:06:30.400 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:06:30.401 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-03-31T15:06:30.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:30.407+0000 7faf69a00640 1 -- 172.21.15.16:0/455255137 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7faf40068380 msgr2=0x7faf4006a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:30.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:30.407+0000 7faf69a00640 1 --2- 172.21.15.16:0/455255137 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7faf40068380 0x7faf4006a840 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7faf64166360 tx=0x7faf54007e90 comp rx=0 tx=0).stop 2024-03-31T15:06:30.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:30.407+0000 7faf69a00640 1 -- 172.21.15.16:0/455255137 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faf64145d10 msgr2=0x7faf641654a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:30.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:30.407+0000 7faf69a00640 1 --2- 172.21.15.16:0/455255137 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faf64145d10 0x7faf641654a0 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7faf5c01a660 tx=0x7faf5c0197d0 comp rx=0 tx=0).stop 2024-03-31T15:06:30.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:30.407+0000 7faf69a00640 1 -- 172.21.15.16:0/455255137 shutdown_connections 2024-03-31T15:06:30.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:30.407+0000 7faf69a00640 1 --2- 172.21.15.16:0/455255137 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7faf40068380 0x7faf4006a840 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:30.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:30.407+0000 7faf69a00640 1 --2- 172.21.15.16:0/455255137 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faf64145d10 0x7faf641654a0 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:30.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:30.407+0000 7faf69a00640 1 --2- 172.21.15.16:0/455255137 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7faf640b2200 0x7faf64138ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:30.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:30.407+0000 7faf69a00640 1 --2- 172.21.15.16:0/455255137 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faf640728b0 0x7faf6413c810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:30.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:30.407+0000 7faf69a00640 1 -- 172.21.15.16:0/455255137 >> 172.21.15.16:0/455255137 conn(0x7faf6406b620 msgr2=0x7faf640cff50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:06:30.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:30.407+0000 7faf69a00640 1 -- 172.21.15.16:0/455255137 shutdown_connections 2024-03-31T15:06:30.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:30.407+0000 7faf69a00640 1 -- 172.21.15.16:0/455255137 wait complete. 2024-03-31T15:06:30.420 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":142,"stamp":"2024-03-31T15:06:28.567712+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":591006,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":177,"num_write_kb":2425,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":134,"ondisk_log_size":134,"up":104,"acting":104,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":39,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":5,"kb":754974720,"kb_used":243136,"kb_used_data":6144,"kb_used_omap":12,"kb_used_meta":236851,"kb_avail":754731584,"statfs":{"total":773094113280,"available":772845142016,"internally_reserved":0,"allocated":6291456,"data_stored":3263164,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":242535984},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":19,"apply_latency_ms":19,"commit_latency_ns":19000000,"apply_latency_ns":19000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":638,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.318195"},"pg_stats":[{"pgid":"9.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:06:25.246926+0000","last_change":"2024-03-31T15:06:25.246926+0000","last_active":"2024-03-31T15:06:25.246926+0000","last_peered":"2024-03-31T15:06:25.246926+0000","last_clean":"2024-03-31T15:06:25.246926+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:06:25.246926+0000","last_undegraded":"2024-03-31T15:06:25.246926+0000","last_fullsized":"2024-03-31T15:06:25.246926+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_clean_scrub_stamp":"2024-03-31T15:06:25.246926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.3e","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:06:26.300576+0000","last_change":"2024-03-31T15:06:26.291088+0000","last_active":"2024-03-31T15:06:26.300576+0000","last_peered":"2024-03-31T15:06:26.300576+0000","last_clean":"2024-03-31T15:06:26.300576+0000","last_became_active":"2024-03-31T15:06:26.290740+0000","last_became_peered":"2024-03-31T15:06:26.290740+0000","last_unstale":"2024-03-31T15:06:26.300576+0000","last_undegraded":"2024-03-31T15:06:26.300576+0000","last_fullsized":"2024-03-31T15:06:26.300576+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_clean_scrub_stamp":"2024-03-31T15:06:25.246926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T17:23:30.654478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:06:25.246926+0000","last_change":"2024-03-31T15:06:25.246926+0000","last_active":"2024-03-31T15:06:25.246926+0000","last_peered":"2024-03-31T15:06:25.246926+0000","last_clean":"2024-03-31T15:06:25.246926+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:06:25.246926+0000","last_undegraded":"2024-03-31T15:06:25.246926+0000","last_fullsized":"2024-03-31T15:06:25.246926+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_clean_scrub_stamp":"2024-03-31T15:06:25.246926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:06:25.246926+0000","last_change":"2024-03-31T15:06:25.246926+0000","last_active":"2024-03-31T15:06:25.246926+0000","last_peered":"2024-03-31T15:06:25.246926+0000","last_clean":"2024-03-31T15:06:25.246926+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:06:25.246926+0000","last_undegraded":"2024-03-31T15:06:25.246926+0000","last_fullsized":"2024-03-31T15:06:25.246926+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_clean_scrub_stamp":"2024-03-31T15:06:25.246926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:06:25.246926+0000","last_change":"2024-03-31T15:06:25.246926+0000","last_active":"2024-03-31T15:06:25.246926+0000","last_peered":"2024-03-31T15:06:25.246926+0000","last_clean":"2024-03-31T15:06:25.246926+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:06:25.246926+0000","last_undegraded":"2024-03-31T15:06:25.246926+0000","last_fullsized":"2024-03-31T15:06:25.246926+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_clean_scrub_stamp":"2024-03-31T15:06:25.246926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.3a","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-31T15:06:27.347135+0000","last_change":"2024-03-31T15:06:26.297322+0000","last_active":"2024-03-31T15:06:27.347135+0000","last_peered":"2024-03-31T15:06:27.347135+0000","last_clean":"2024-03-31T15:06:27.347135+0000","last_became_active":"2024-03-31T15:06:26.296993+0000","last_became_peered":"2024-03-31T15:06:26.296993+0000","last_unstale":"2024-03-31T15:06:27.347135+0000","last_undegraded":"2024-03-31T15:06:27.347135+0000","last_fullsized":"2024-03-31T15:06:27.347135+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_clean_scrub_stamp":"2024-03-31T15:06:25.246926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:13:41.882384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:06:25.246926+0000","last_change":"2024-03-31T15:06:25.246926+0000","last_active":"2024-03-31T15:06:25.246926+0000","last_peered":"2024-03-31T15:06:25.246926+0000","last_clean":"2024-03-31T15:06:25.246926+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:06:25.246926+0000","last_undegraded":"2024-03-31T15:06:25.246926+0000","last_fullsized":"2024-03-31T15:06:25.246926+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_clean_scrub_stamp":"2024-03-31T15:06:25.246926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.38","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:06:26.295455+0000","last_change":"2024-03-31T15:06:26.284320+0000","last_active":"2024-03-31T15:06:26.295455+0000","last_peered":"2024-03-31T15:06:26.295455+0000","last_clean":"2024-03-31T15:06:26.295455+0000","last_became_active":"2024-03-31T15:06:26.283991+0000","last_became_peered":"2024-03-31T15:06:26.283991+0000","last_unstale":"2024-03-31T15:06:26.295455+0000","last_undegraded":"2024-03-31T15:06:26.295455+0000","last_fullsized":"2024-03-31T15:06:26.295455+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_clean_scrub_stamp":"2024-03-31T15:06:25.246926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:25:45.811474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:06:25.246926+0000","last_change":"2024-03-31T15:06:25.246926+0000","last_active":"2024-03-31T15:06:25.246926+0000","last_peered":"2024-03-31T15:06:25.246926+0000","last_clean":"2024-03-31T15:06:25.246926+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:06:25.246926+0000","last_undegraded":"2024-03-31T15:06:25.246926+0000","last_fullsized":"2024-03-31T15:06:25.246926+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_clean_scrub_stamp":"2024-03-31T15:06:25.246926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.36","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:06:26.297518+0000","last_change":"2024-03-31T15:06:26.291183+0000","last_active":"2024-03-31T15:06:26.297518+0000","last_peered":"2024-03-31T15:06:26.297518+0000","last_clean":"2024-03-31T15:06:26.297518+0000","last_became_active":"2024-03-31T15:06:26.290916+0000","last_became_peered":"2024-03-31T15:06:26.290916+0000","last_unstale":"2024-03-31T15:06:26.297518+0000","last_undegraded":"2024-03-31T15:06:26.297518+0000","last_fullsized":"2024-03-31T15:06:26.297518+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_clean_scrub_stamp":"2024-03-31T15:06:25.246926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T17:39:02.515165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.35","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:06:26.301309+0000","last_change":"2024-03-31T15:06:26.294325+0000","last_active":"2024-03-31T15:06:26.301309+0000","last_peered":"2024-03-31T15:06:26.301309+0000","last_clean":"2024-03-31T15:06:26.301309+0000","last_became_active":"2024-03-31T15:06:26.294054+0000","last_became_peered":"2024-03-31T15:06:26.294054+0000","last_unstale":"2024-03-31T15:06:26.301309+0000","last_undegraded":"2024-03-31T15:06:26.301309+0000","last_fullsized":"2024-03-31T15:06:26.301309+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_clean_scrub_stamp":"2024-03-31T15:06:25.246926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:28:08.464687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:06:25.246926+0000","last_change":"2024-03-31T15:06:25.246926+0000","last_active":"2024-03-31T15:06:25.246926+0000","last_peered":"2024-03-31T15:06:25.246926+0000","last_clean":"2024-03-31T15:06:25.246926+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:06:25.246926+0000","last_undegraded":"2024-03-31T15:06:25.246926+0000","last_fullsized":"2024-03-31T15:06:25.246926+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_clean_scrub_stamp":"2024-03-31T15:06:25.246926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.33","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-31T15:06:27.286942+0000","last_change":"2024-03-31T15:06:26.292241+0000","last_active":"2024-03-31T15:06:27.286942+0000","last_peered":"2024-03-31T15:06:27.286942+0000","last_clean":"2024-03-31T15:06:27.286942+0000","last_became_active":"2024-03-31T15:06:26.291928+0000","last_became_peered":"2024-03-31T15:06:26.291928+0000","last_unstale":"2024-03-31T15:06:27.286942+0000","last_undegraded":"2024-03-31T15:06:27.286942+0000","last_fullsized":"2024-03-31T15:06:27.286942+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_clean_scrub_stamp":"2024-03-31T15:06:25.246926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:59:13.871341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:06:25.246926+0000","last_change":"2024-03-31T15:06:25.246926+0000","last_active":"2024-03-31T15:06:25.246926+0000","last_peered":"2024-03-31T15:06:25.246926+0000","last_clean":"2024-03-31T15:06:25.246926+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:06:25.246926+0000","last_undegraded":"2024-03-31T15:06:25.246926+0000","last_fullsized":"2024-03-31T15:06:25.246926+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_clean_scrub_stamp":"2024-03-31T15:06:25.246926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:06:25.246926+0000","last_change":"2024-03-31T15:06:25.246926+0000","last_active":"2024-03-31T15:06:25.246926+0000","last_peered":"2024-03-31T15:06:25.246926+0000","last_clean":"2024-03-31T15:06:25.246926+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:06:25.246926+0000","last_undegraded":"2024-03-31T15:06:25.246926+0000","last_fullsized":"2024-03-31T15:06:25.246926+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_clean_scrub_stamp":"2024-03-31T15:06:25.246926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.30","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-31T15:06:27.347466+0000","last_change":"2024-03-31T15:06:26.298305+0000","last_active":"2024-03-31T15:06:27.347466+0000","last_peered":"2024-03-31T15:06:27.347466+0000","last_clean":"2024-03-31T15:06:27.347466+0000","last_became_active":"2024-03-31T15:06:26.296491+0000","last_became_peered":"2024-03-31T15:06:26.296491+0000","last_unstale":"2024-03-31T15:06:27.347466+0000","last_undegraded":"2024-03-31T15:06:27.347466+0000","last_fullsized":"2024-03-31T15:06:27.347466+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_clean_scrub_stamp":"2024-03-31T15:06:25.246926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:50:32.468918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.2f","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:06:26.300835+0000","last_change":"2024-03-31T15:06:26.294641+0000","last_active":"2024-03-31T15:06:26.300835+0000","last_peered":"2024-03-31T15:06:26.300835+0000","last_clean":"2024-03-31T15:06:26.300835+0000","last_became_active":"2024-03-31T15:06:26.294358+0000","last_became_peered":"2024-03-31T15:06:26.294358+0000","last_unstale":"2024-03-31T15:06:26.300835+0000","last_undegraded":"2024-03-31T15:06:26.300835+0000","last_fullsized":"2024-03-31T15:06:26.300835+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_clean_scrub_stamp":"2024-03-31T15:06:25.246926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T16:42:20.504930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:06:25.246926+0000","last_change":"2024-03-31T15:06:25.246926+0000","last_active":"2024-03-31T15:06:25.246926+0000","last_peered":"2024-03-31T15:06:25.246926+0000","last_clean":"2024-03-31T15:06:25.246926+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:06:25.246926+0000","last_undegraded":"2024-03-31T15:06:25.246926+0000","last_fullsized":"2024-03-31T15:06:25.246926+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_clean_scrub_stamp":"2024-03-31T15:06:25.246926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:06:25.246926+0000","last_change":"2024-03-31T15:06:25.246926+0000","last_active":"2024-03-31T15:06:25.246926+0000","last_peered":"2024-03-31T15:06:25.246926+0000","last_clean":"2024-03-31T15:06:25.246926+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:06:25.246926+0000","last_undegraded":"2024-03-31T15:06:25.246926+0000","last_fullsized":"2024-03-31T15:06:25.246926+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_clean_scrub_stamp":"2024-03-31T15:06:25.246926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.2c","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:06:26.300517+0000","last_change":"2024-03-31T15:06:26.297250+0000","last_active":"2024-03-31T15:06:26.300517+0000","last_peered":"2024-03-31T15:06:26.300517+0000","last_clean":"2024-03-31T15:06:26.300517+0000","last_became_active":"2024-03-31T15:06:26.296974+0000","last_became_peered":"2024-03-31T15:06:26.296974+0000","last_unstale":"2024-03-31T15:06:26.300517+0000","last_undegraded":"2024-03-31T15:06:26.300517+0000","last_fullsized":"2024-03-31T15:06:26.300517+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_clean_scrub_stamp":"2024-03-31T15:06:25.246926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:38:46.324919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:06:25.246926+0000","last_change":"2024-03-31T15:06:25.246926+0000","last_active":"2024-03-31T15:06:25.246926+0000","last_peered":"2024-03-31T15:06:25.246926+0000","last_clean":"2024-03-31T15:06:25.246926+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:06:25.246926+0000","last_undegraded":"2024-03-31T15:06:25.246926+0000","last_fullsized":"2024-03-31T15:06:25.246926+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_clean_scrub_stamp":"2024-03-31T15:06:25.246926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.2a","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:06:26.300547+0000","last_change":"2024-03-31T15:06:26.296387+0000","last_active":"2024-03-31T15:06:26.300547+0000","last_peered":"2024-03-31T15:06:26.300547+0000","last_clean":"2024-03-31T15:06:26.300547+0000","last_became_active":"2024-03-31T15:06:26.295902+0000","last_became_peered":"2024-03-31T15:06:26.295902+0000","last_unstale":"2024-03-31T15:06:26.300547+0000","last_undegraded":"2024-03-31T15:06:26.300547+0000","last_fullsized":"2024-03-31T15:06:26.300547+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_clean_scrub_stamp":"2024-03-31T15:06:25.246926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:27:24.902121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:06:25.246926+0000","last_change":"2024-03-31T15:06:25.246926+0000","last_active":"2024-03-31T15:06:25.246926+0000","last_peered":"2024-03-31T15:06:25.246926+0000","last_clean":"2024-03-31T15:06:25.246926+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:06:25.246926+0000","last_undegraded":"2024-03-31T15:06:25.246926+0000","last_fullsized":"2024-03-31T15:06:25.246926+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_clean_scrub_stamp":"2024-03-31T15:06:25.246926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.28","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-31T15:06:27.286270+0000","last_change":"2024-03-31T15:06:26.288990+0000","last_active":"2024-03-31T15:06:27.286270+0000","last_peered":"2024-03-31T15:06:27.286270+0000","last_clean":"2024-03-31T15:06:27.286270+0000","last_became_active":"2024-03-31T15:06:26.288124+0000","last_became_peered":"2024-03-31T15:06:26.288124+0000","last_unstale":"2024-03-31T15:06:27.286270+0000","last_undegraded":"2024-03-31T15:06:27.286270+0000","last_fullsized":"2024-03-31T15:06:27.286270+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_clean_scrub_stamp":"2024-03-31T15:06:25.246926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T15:16:19.589166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:06:25.246926+0000","last_change":"2024-03-31T15:06:25.246926+0000","last_active":"2024-03-31T15:06:25.246926+0000","last_peered":"2024-03-31T15:06:25.246926+0000","last_clean":"2024-03-31T15:06:25.246926+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:06:25.246926+0000","last_undegraded":"2024-03-31T15:06:25.246926+0000","last_fullsized":"2024-03-31T15:06:25.246926+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_clean_scrub_stamp":"2024-03-31T15:06:25.246926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:06:25.246926+0000","last_change":"2024-03-31T15:06:25.246926+0000","last_active":"2024-03-31T15:06:25.246926+0000","last_peered":"2024-03-31T15:06:25.246926+0000","last_clean":"2024-03-31T15:06:25.246926+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:06:25.246926+0000","last_undegraded":"2024-03-31T15:06:25.246926+0000","last_fullsized":"2024-03-31T15:06:25.246926+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_clean_scrub_stamp":"2024-03-31T15:06:25.246926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.25","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-31T15:06:27.347347+0000","last_change":"2024-03-31T15:06:26.300765+0000","last_active":"2024-03-31T15:06:27.347347+0000","last_peered":"2024-03-31T15:06:27.347347+0000","last_clean":"2024-03-31T15:06:27.347347+0000","last_became_active":"2024-03-31T15:06:26.300480+0000","last_became_peered":"2024-03-31T15:06:26.300480+0000","last_unstale":"2024-03-31T15:06:27.347347+0000","last_undegraded":"2024-03-31T15:06:27.347347+0000","last_fullsized":"2024-03-31T15:06:27.347347+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_clean_scrub_stamp":"2024-03-31T15:06:25.246926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T19:09:44.028119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:06:25.246926+0000","last_change":"2024-03-31T15:06:25.246926+0000","last_active":"2024-03-31T15:06:25.246926+0000","last_peered":"2024-03-31T15:06:25.246926+0000","last_clean":"2024-03-31T15:06:25.246926+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:06:25.246926+0000","last_undegraded":"2024-03-31T15:06:25.246926+0000","last_fullsized":"2024-03-31T15:06:25.246926+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_clean_scrub_stamp":"2024-03-31T15:06:25.246926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.23","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:06:26.299003+0000","last_change":"2024-03-31T15:06:26.289798+0000","last_active":"2024-03-31T15:06:26.299003+0000","last_peered":"2024-03-31T15:06:26.299003+0000","last_clean":"2024-03-31T15:06:26.299003+0000","last_became_active":"2024-03-31T15:06:26.289464+0000","last_became_peered":"2024-03-31T15:06:26.289464+0000","last_unstale":"2024-03-31T15:06:26.299003+0000","last_undegraded":"2024-03-31T15:06:26.299003+0000","last_fullsized":"2024-03-31T15:06:26.299003+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_clean_scrub_stamp":"2024-03-31T15:06:25.246926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T15:23:39.306094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:06:25.246926+0000","last_change":"2024-03-31T15:06:25.246926+0000","last_active":"2024-03-31T15:06:25.246926+0000","last_peered":"2024-03-31T15:06:25.246926+0000","last_clean":"2024-03-31T15:06:25.246926+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:06:25.246926+0000","last_undegraded":"2024-03-31T15:06:25.246926+0000","last_fullsized":"2024-03-31T15:06:25.246926+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_clean_scrub_stamp":"2024-03-31T15:06:25.246926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:06:25.246926+0000","last_change":"2024-03-31T15:06:25.246926+0000","last_active":"2024-03-31T15:06:25.246926+0000","last_peered":"2024-03-31T15:06:25.246926+0000","last_clean":"2024-03-31T15:06:25.246926+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:06:25.246926+0000","last_undegraded":"2024-03-31T15:06:25.246926+0000","last_fullsized":"2024-03-31T15:06:25.246926+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_clean_scrub_stamp":"2024-03-31T15:06:25.246926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.20","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-31T15:06:27.347113+0000","last_change":"2024-03-31T15:06:26.303201+0000","last_active":"2024-03-31T15:06:27.347113+0000","last_peered":"2024-03-31T15:06:27.347113+0000","last_clean":"2024-03-31T15:06:27.347113+0000","last_became_active":"2024-03-31T15:06:26.302949+0000","last_became_peered":"2024-03-31T15:06:26.302949+0000","last_unstale":"2024-03-31T15:06:27.347113+0000","last_undegraded":"2024-03-31T15:06:27.347113+0000","last_fullsized":"2024-03-31T15:06:27.347113+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_clean_scrub_stamp":"2024-03-31T15:06:25.246926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T15:38:58.967933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1e","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-31T15:06:27.347530+0000","last_change":"2024-03-31T15:06:26.301437+0000","last_active":"2024-03-31T15:06:27.347530+0000","last_peered":"2024-03-31T15:06:27.347530+0000","last_clean":"2024-03-31T15:06:27.347530+0000","last_became_active":"2024-03-31T15:06:26.299941+0000","last_became_peered":"2024-03-31T15:06:26.299941+0000","last_unstale":"2024-03-31T15:06:27.347530+0000","last_undegraded":"2024-03-31T15:06:27.347530+0000","last_fullsized":"2024-03-31T15:06:27.347530+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_clean_scrub_stamp":"2024-03-31T15:06:25.246926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:56:13.492863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1f","version":"55'4","reported_seq":18,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-31T15:06:27.347398+0000","last_change":"2024-03-31T15:06:25.270858+0000","last_active":"2024-03-31T15:06:27.347398+0000","last_peered":"2024-03-31T15:06:27.347398+0000","last_clean":"2024-03-31T15:06:27.347398+0000","last_became_active":"2024-03-31T15:06:25.270569+0000","last_became_peered":"2024-03-31T15:06:25.270569+0000","last_unstale":"2024-03-31T15:06:27.347398+0000","last_undegraded":"2024-03-31T15:06:27.347398+0000","last_fullsized":"2024-03-31T15:06:27.347398+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:06:24.244870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:06:24.244870+0000","last_clean_scrub_stamp":"2024-03-31T15:06:24.244870+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T15:22:02.328114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1f","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:06:26.295037+0000","last_change":"2024-03-31T15:06:26.288476+0000","last_active":"2024-03-31T15:06:26.295037+0000","last_peered":"2024-03-31T15:06:26.295037+0000","last_clean":"2024-03-31T15:06:26.295037+0000","last_became_active":"2024-03-31T15:06:26.288157+0000","last_became_peered":"2024-03-31T15:06:26.288157+0000","last_unstale":"2024-03-31T15:06:26.295037+0000","last_undegraded":"2024-03-31T15:06:26.295037+0000","last_fullsized":"2024-03-31T15:06:26.295037+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_clean_scrub_stamp":"2024-03-31T15:06:25.246926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T19:34:18.973247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1e","version":"55'1","reported_seq":13,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:06:26.296315+0000","last_change":"2024-03-31T15:06:25.259815+0000","last_active":"2024-03-31T15:06:26.296315+0000","last_peered":"2024-03-31T15:06:26.296315+0000","last_clean":"2024-03-31T15:06:26.296315+0000","last_became_active":"2024-03-31T15:06:25.259531+0000","last_became_peered":"2024-03-31T15:06:25.259531+0000","last_unstale":"2024-03-31T15:06:26.296315+0000","last_undegraded":"2024-03-31T15:06:26.296315+0000","last_fullsized":"2024-03-31T15:06:26.296315+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:06:24.244870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:06:24.244870+0000","last_clean_scrub_stamp":"2024-03-31T15:06:24.244870+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T17:06:13.957394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.1c","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-31T15:06:27.347741+0000","last_change":"2024-03-31T15:06:26.308014+0000","last_active":"2024-03-31T15:06:27.347741+0000","last_peered":"2024-03-31T15:06:27.347741+0000","last_clean":"2024-03-31T15:06:27.347741+0000","last_became_active":"2024-03-31T15:06:26.307138+0000","last_became_peered":"2024-03-31T15:06:26.307138+0000","last_unstale":"2024-03-31T15:06:27.347741+0000","last_undegraded":"2024-03-31T15:06:27.347741+0000","last_fullsized":"2024-03-31T15:06:27.347741+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_clean_scrub_stamp":"2024-03-31T15:06:25.246926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:43:16.624229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1d","version":"55'2","reported_seq":16,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-31T15:06:27.347550+0000","last_change":"2024-03-31T15:06:25.276727+0000","last_active":"2024-03-31T15:06:27.347550+0000","last_peered":"2024-03-31T15:06:27.347550+0000","last_clean":"2024-03-31T15:06:27.347550+0000","last_became_active":"2024-03-31T15:06:25.276430+0000","last_became_peered":"2024-03-31T15:06:25.276430+0000","last_unstale":"2024-03-31T15:06:27.347550+0000","last_undegraded":"2024-03-31T15:06:27.347550+0000","last_fullsized":"2024-03-31T15:06:27.347550+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:06:24.244870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:06:24.244870+0000","last_clean_scrub_stamp":"2024-03-31T15:06:24.244870+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:15:43.405488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1d","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-31T15:06:27.347431+0000","last_change":"2024-03-31T15:06:26.300634+0000","last_active":"2024-03-31T15:06:27.347431+0000","last_peered":"2024-03-31T15:06:27.347431+0000","last_clean":"2024-03-31T15:06:27.347431+0000","last_became_active":"2024-03-31T15:06:26.300114+0000","last_became_peered":"2024-03-31T15:06:26.300114+0000","last_unstale":"2024-03-31T15:06:27.347431+0000","last_undegraded":"2024-03-31T15:06:27.347431+0000","last_fullsized":"2024-03-31T15:06:27.347431+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_clean_scrub_stamp":"2024-03-31T15:06:25.246926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:46:19.150623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:06:24.244870+0000","last_change":"2024-03-31T15:06:24.244870+0000","last_active":"2024-03-31T15:06:24.244870+0000","last_peered":"2024-03-31T15:06:24.244870+0000","last_clean":"2024-03-31T15:06:24.244870+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:06:24.244870+0000","last_undegraded":"2024-03-31T15:06:24.244870+0000","last_fullsized":"2024-03-31T15:06:24.244870+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:06:24.244870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:06:24.244870+0000","last_clean_scrub_stamp":"2024-03-31T15:06:24.244870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.1a","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:06:26.300477+0000","last_change":"2024-03-31T15:06:26.296178+0000","last_active":"2024-03-31T15:06:26.300477+0000","last_peered":"2024-03-31T15:06:26.300477+0000","last_clean":"2024-03-31T15:06:26.300477+0000","last_became_active":"2024-03-31T15:06:26.295534+0000","last_became_peered":"2024-03-31T15:06:26.295534+0000","last_unstale":"2024-03-31T15:06:26.300477+0000","last_undegraded":"2024-03-31T15:06:26.300477+0000","last_fullsized":"2024-03-31T15:06:26.300477+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_clean_scrub_stamp":"2024-03-31T15:06:25.246926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T17:46:44.001115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:06:24.244870+0000","last_change":"2024-03-31T15:06:24.244870+0000","last_active":"2024-03-31T15:06:24.244870+0000","last_peered":"2024-03-31T15:06:24.244870+0000","last_clean":"2024-03-31T15:06:24.244870+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:06:24.244870+0000","last_undegraded":"2024-03-31T15:06:24.244870+0000","last_fullsized":"2024-03-31T15:06:24.244870+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:06:24.244870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:06:24.244870+0000","last_clean_scrub_stamp":"2024-03-31T15:06:24.244870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.1b","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:06:26.305001+0000","last_change":"2024-03-31T15:06:26.303096+0000","last_active":"2024-03-31T15:06:26.305001+0000","last_peered":"2024-03-31T15:06:26.305001+0000","last_clean":"2024-03-31T15:06:26.305001+0000","last_became_active":"2024-03-31T15:06:26.302647+0000","last_became_peered":"2024-03-31T15:06:26.302647+0000","last_unstale":"2024-03-31T15:06:26.305001+0000","last_undegraded":"2024-03-31T15:06:26.305001+0000","last_fullsized":"2024-03-31T15:06:26.305001+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_clean_scrub_stamp":"2024-03-31T15:06:25.246926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:53:42.345165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1a","version":"55'1","reported_seq":15,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-31T15:06:27.347230+0000","last_change":"2024-03-31T15:06:25.274617+0000","last_active":"2024-03-31T15:06:27.347230+0000","last_peered":"2024-03-31T15:06:27.347230+0000","last_clean":"2024-03-31T15:06:27.347230+0000","last_became_active":"2024-03-31T15:06:25.274071+0000","last_became_peered":"2024-03-31T15:06:25.274071+0000","last_unstale":"2024-03-31T15:06:27.347230+0000","last_undegraded":"2024-03-31T15:06:27.347230+0000","last_fullsized":"2024-03-31T15:06:27.347230+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:06:24.244870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:06:24.244870+0000","last_clean_scrub_stamp":"2024-03-31T15:06:24.244870+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:23:48.591532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-31T15:06:27.286419+0000","last_change":"2024-03-31T15:06:26.278004+0000","last_active":"2024-03-31T15:06:27.286419+0000","last_peered":"2024-03-31T15:06:27.286419+0000","last_clean":"2024-03-31T15:06:27.286419+0000","last_became_active":"2024-03-31T15:06:26.277630+0000","last_became_peered":"2024-03-31T15:06:26.277630+0000","last_unstale":"2024-03-31T15:06:27.286419+0000","last_undegraded":"2024-03-31T15:06:27.286419+0000","last_fullsized":"2024-03-31T15:06:27.286419+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_clean_scrub_stamp":"2024-03-31T15:06:25.246926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:59:09.402257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:06:24.244870+0000","last_change":"2024-03-31T15:06:24.244870+0000","last_active":"2024-03-31T15:06:24.244870+0000","last_peered":"2024-03-31T15:06:24.244870+0000","last_clean":"2024-03-31T15:06:24.244870+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:06:24.244870+0000","last_undegraded":"2024-03-31T15:06:24.244870+0000","last_fullsized":"2024-03-31T15:06:24.244870+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:06:24.244870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:06:24.244870+0000","last_clean_scrub_stamp":"2024-03-31T15:06:24.244870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:06:26.306949+0000","last_change":"2024-03-31T15:06:26.305278+0000","last_active":"2024-03-31T15:06:26.306949+0000","last_peered":"2024-03-31T15:06:26.306949+0000","last_clean":"2024-03-31T15:06:26.306949+0000","last_became_active":"2024-03-31T15:06:26.301678+0000","last_became_peered":"2024-03-31T15:06:26.301678+0000","last_unstale":"2024-03-31T15:06:26.306949+0000","last_undegraded":"2024-03-31T15:06:26.306949+0000","last_fullsized":"2024-03-31T15:06:26.306949+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_clean_scrub_stamp":"2024-03-31T15:06:25.246926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:04:57.292580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:06:24.244870+0000","last_change":"2024-03-31T15:06:24.244870+0000","last_active":"2024-03-31T15:06:24.244870+0000","last_peered":"2024-03-31T15:06:24.244870+0000","last_clean":"2024-03-31T15:06:24.244870+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:06:24.244870+0000","last_undegraded":"2024-03-31T15:06:24.244870+0000","last_fullsized":"2024-03-31T15:06:24.244870+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:06:24.244870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:06:24.244870+0000","last_clean_scrub_stamp":"2024-03-31T15:06:24.244870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:06:25.246926+0000","last_change":"2024-03-31T15:06:25.246926+0000","last_active":"2024-03-31T15:06:25.246926+0000","last_peered":"2024-03-31T15:06:25.246926+0000","last_clean":"2024-03-31T15:06:25.246926+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:06:25.246926+0000","last_undegraded":"2024-03-31T15:06:25.246926+0000","last_fullsized":"2024-03-31T15:06:25.246926+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_clean_scrub_stamp":"2024-03-31T15:06:25.246926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:06:24.244870+0000","last_change":"2024-03-31T15:06:24.244870+0000","last_active":"2024-03-31T15:06:24.244870+0000","last_peered":"2024-03-31T15:06:24.244870+0000","last_clean":"2024-03-31T15:06:24.244870+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:06:24.244870+0000","last_undegraded":"2024-03-31T15:06:24.244870+0000","last_fullsized":"2024-03-31T15:06:24.244870+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:06:24.244870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:06:24.244870+0000","last_clean_scrub_stamp":"2024-03-31T15:06:24.244870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:06:25.246926+0000","last_change":"2024-03-31T15:06:25.246926+0000","last_active":"2024-03-31T15:06:25.246926+0000","last_peered":"2024-03-31T15:06:25.246926+0000","last_clean":"2024-03-31T15:06:25.246926+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:06:25.246926+0000","last_undegraded":"2024-03-31T15:06:25.246926+0000","last_fullsized":"2024-03-31T15:06:25.246926+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_clean_scrub_stamp":"2024-03-31T15:06:25.246926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:06:26.274393+0000","last_change":"2024-03-31T15:06:25.278704+0000","last_active":"2024-03-31T15:06:26.274393+0000","last_peered":"2024-03-31T15:06:26.274393+0000","last_clean":"2024-03-31T15:06:26.274393+0000","last_became_active":"2024-03-31T15:06:25.278406+0000","last_became_peered":"2024-03-31T15:06:25.278406+0000","last_unstale":"2024-03-31T15:06:26.274393+0000","last_undegraded":"2024-03-31T15:06:26.274393+0000","last_fullsized":"2024-03-31T15:06:26.274393+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:06:24.244870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:06:24.244870+0000","last_clean_scrub_stamp":"2024-03-31T15:06:24.244870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T16:27:52.154972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.14","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:06:26.294558+0000","last_change":"2024-03-31T15:06:26.289438+0000","last_active":"2024-03-31T15:06:26.294558+0000","last_peered":"2024-03-31T15:06:26.294558+0000","last_clean":"2024-03-31T15:06:26.294558+0000","last_became_active":"2024-03-31T15:06:26.289182+0000","last_became_peered":"2024-03-31T15:06:26.289182+0000","last_unstale":"2024-03-31T15:06:26.294558+0000","last_undegraded":"2024-03-31T15:06:26.294558+0000","last_fullsized":"2024-03-31T15:06:26.294558+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_clean_scrub_stamp":"2024-03-31T15:06:25.246926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:37:10.307112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:06:24.244870+0000","last_change":"2024-03-31T15:06:24.244870+0000","last_active":"2024-03-31T15:06:24.244870+0000","last_peered":"2024-03-31T15:06:24.244870+0000","last_clean":"2024-03-31T15:06:24.244870+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:06:24.244870+0000","last_undegraded":"2024-03-31T15:06:24.244870+0000","last_fullsized":"2024-03-31T15:06:24.244870+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:06:24.244870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:06:24.244870+0000","last_clean_scrub_stamp":"2024-03-31T15:06:24.244870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:06:25.246926+0000","last_change":"2024-03-31T15:06:25.246926+0000","last_active":"2024-03-31T15:06:25.246926+0000","last_peered":"2024-03-31T15:06:25.246926+0000","last_clean":"2024-03-31T15:06:25.246926+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:06:25.246926+0000","last_undegraded":"2024-03-31T15:06:25.246926+0000","last_fullsized":"2024-03-31T15:06:25.246926+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_clean_scrub_stamp":"2024-03-31T15:06:25.246926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:06:24.244870+0000","last_change":"2024-03-31T15:06:24.244870+0000","last_active":"2024-03-31T15:06:24.244870+0000","last_peered":"2024-03-31T15:06:24.244870+0000","last_clean":"2024-03-31T15:06:24.244870+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:06:24.244870+0000","last_undegraded":"2024-03-31T15:06:24.244870+0000","last_fullsized":"2024-03-31T15:06:24.244870+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:06:24.244870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:06:24.244870+0000","last_clean_scrub_stamp":"2024-03-31T15:06:24.244870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:06:25.246926+0000","last_change":"2024-03-31T15:06:25.246926+0000","last_active":"2024-03-31T15:06:25.246926+0000","last_peered":"2024-03-31T15:06:25.246926+0000","last_clean":"2024-03-31T15:06:25.246926+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:06:25.246926+0000","last_undegraded":"2024-03-31T15:06:25.246926+0000","last_fullsized":"2024-03-31T15:06:25.246926+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_clean_scrub_stamp":"2024-03-31T15:06:25.246926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.13","version":"55'2","reported_seq":14,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:06:26.303833+0000","last_change":"2024-03-31T15:06:25.278639+0000","last_active":"2024-03-31T15:06:26.303833+0000","last_peered":"2024-03-31T15:06:26.303833+0000","last_clean":"2024-03-31T15:06:26.303833+0000","last_became_active":"2024-03-31T15:06:25.278335+0000","last_became_peered":"2024-03-31T15:06:25.278335+0000","last_unstale":"2024-03-31T15:06:26.303833+0000","last_undegraded":"2024-03-31T15:06:26.303833+0000","last_fullsized":"2024-03-31T15:06:26.303833+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:06:24.244870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:06:24.244870+0000","last_clean_scrub_stamp":"2024-03-31T15:06:24.244870+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:56:12.490806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:06:25.246926+0000","last_change":"2024-03-31T15:06:25.246926+0000","last_active":"2024-03-31T15:06:25.246926+0000","last_peered":"2024-03-31T15:06:25.246926+0000","last_clean":"2024-03-31T15:06:25.246926+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:06:25.246926+0000","last_undegraded":"2024-03-31T15:06:25.246926+0000","last_fullsized":"2024-03-31T15:06:25.246926+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_clean_scrub_stamp":"2024-03-31T15:06:25.246926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":14,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-31T15:06:27.286832+0000","last_change":"2024-03-31T15:06:25.267602+0000","last_active":"2024-03-31T15:06:27.286832+0000","last_peered":"2024-03-31T15:06:27.286832+0000","last_clean":"2024-03-31T15:06:27.286832+0000","last_became_active":"2024-03-31T15:06:25.267283+0000","last_became_peered":"2024-03-31T15:06:25.267283+0000","last_unstale":"2024-03-31T15:06:27.286832+0000","last_undegraded":"2024-03-31T15:06:27.286832+0000","last_fullsized":"2024-03-31T15:06:27.286832+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:06:24.244870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:06:24.244870+0000","last_clean_scrub_stamp":"2024-03-31T15:06:24.244870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T19:34:17.971191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:06:25.246926+0000","last_change":"2024-03-31T15:06:25.246926+0000","last_active":"2024-03-31T15:06:25.246926+0000","last_peered":"2024-03-31T15:06:25.246926+0000","last_clean":"2024-03-31T15:06:25.246926+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:06:25.246926+0000","last_undegraded":"2024-03-31T15:06:25.246926+0000","last_fullsized":"2024-03-31T15:06:25.246926+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_clean_scrub_stamp":"2024-03-31T15:06:25.246926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:06:24.244870+0000","last_change":"2024-03-31T15:06:24.244870+0000","last_active":"2024-03-31T15:06:24.244870+0000","last_peered":"2024-03-31T15:06:24.244870+0000","last_clean":"2024-03-31T15:06:24.244870+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:06:24.244870+0000","last_undegraded":"2024-03-31T15:06:24.244870+0000","last_fullsized":"2024-03-31T15:06:24.244870+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:06:24.244870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:06:24.244870+0000","last_clean_scrub_stamp":"2024-03-31T15:06:24.244870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:06:25.246926+0000","last_change":"2024-03-31T15:06:25.246926+0000","last_active":"2024-03-31T15:06:25.246926+0000","last_peered":"2024-03-31T15:06:25.246926+0000","last_clean":"2024-03-31T15:06:25.246926+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:06:25.246926+0000","last_undegraded":"2024-03-31T15:06:25.246926+0000","last_fullsized":"2024-03-31T15:06:25.246926+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_clean_scrub_stamp":"2024-03-31T15:06:25.246926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:06:24.244870+0000","last_change":"2024-03-31T15:06:24.244870+0000","last_active":"2024-03-31T15:06:24.244870+0000","last_peered":"2024-03-31T15:06:24.244870+0000","last_clean":"2024-03-31T15:06:24.244870+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:06:24.244870+0000","last_undegraded":"2024-03-31T15:06:24.244870+0000","last_fullsized":"2024-03-31T15:06:24.244870+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:06:24.244870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:06:24.244870+0000","last_clean_scrub_stamp":"2024-03-31T15:06:24.244870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:06:25.246926+0000","last_change":"2024-03-31T15:06:25.246926+0000","last_active":"2024-03-31T15:06:25.246926+0000","last_peered":"2024-03-31T15:06:25.246926+0000","last_clean":"2024-03-31T15:06:25.246926+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:06:25.246926+0000","last_undegraded":"2024-03-31T15:06:25.246926+0000","last_fullsized":"2024-03-31T15:06:25.246926+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_clean_scrub_stamp":"2024-03-31T15:06:25.246926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.f","version":"55'1","reported_seq":13,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:06:26.299395+0000","last_change":"2024-03-31T15:06:25.279268+0000","last_active":"2024-03-31T15:06:26.299395+0000","last_peered":"2024-03-31T15:06:26.299395+0000","last_clean":"2024-03-31T15:06:26.299395+0000","last_became_active":"2024-03-31T15:06:25.277056+0000","last_became_peered":"2024-03-31T15:06:25.277056+0000","last_unstale":"2024-03-31T15:06:26.299395+0000","last_undegraded":"2024-03-31T15:06:26.299395+0000","last_fullsized":"2024-03-31T15:06:26.299395+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:06:24.244870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:06:24.244870+0000","last_clean_scrub_stamp":"2024-03-31T15:06:24.244870+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:43:15.622172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:06:25.246926+0000","last_change":"2024-03-31T15:06:25.246926+0000","last_active":"2024-03-31T15:06:25.246926+0000","last_peered":"2024-03-31T15:06:25.246926+0000","last_clean":"2024-03-31T15:06:25.246926+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:06:25.246926+0000","last_undegraded":"2024-03-31T15:06:25.246926+0000","last_fullsized":"2024-03-31T15:06:25.246926+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_clean_scrub_stamp":"2024-03-31T15:06:25.246926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.e","version":"55'2","reported_seq":16,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-31T15:06:27.347624+0000","last_change":"2024-03-31T15:06:25.280643+0000","last_active":"2024-03-31T15:06:27.347624+0000","last_peered":"2024-03-31T15:06:27.347624+0000","last_clean":"2024-03-31T15:06:27.347624+0000","last_became_active":"2024-03-31T15:06:25.280285+0000","last_became_peered":"2024-03-31T15:06:25.280285+0000","last_unstale":"2024-03-31T15:06:27.347624+0000","last_undegraded":"2024-03-31T15:06:27.347624+0000","last_fullsized":"2024-03-31T15:06:27.347624+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:06:24.244870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:06:24.244870+0000","last_clean_scrub_stamp":"2024-03-31T15:06:24.244870+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:25:14.095342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:06:26.300497+0000","last_change":"2024-03-31T15:06:26.291977+0000","last_active":"2024-03-31T15:06:26.300497+0000","last_peered":"2024-03-31T15:06:26.300497+0000","last_clean":"2024-03-31T15:06:26.300497+0000","last_became_active":"2024-03-31T15:06:26.291666+0000","last_became_peered":"2024-03-31T15:06:26.291666+0000","last_unstale":"2024-03-31T15:06:26.300497+0000","last_undegraded":"2024-03-31T15:06:26.300497+0000","last_fullsized":"2024-03-31T15:06:26.300497+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_clean_scrub_stamp":"2024-03-31T15:06:25.246926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T19:35:04.972072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:06:24.244870+0000","last_change":"2024-03-31T15:06:24.244870+0000","last_active":"2024-03-31T15:06:24.244870+0000","last_peered":"2024-03-31T15:06:24.244870+0000","last_clean":"2024-03-31T15:06:24.244870+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:06:24.244870+0000","last_undegraded":"2024-03-31T15:06:24.244870+0000","last_fullsized":"2024-03-31T15:06:24.244870+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:06:24.244870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:06:24.244870+0000","last_clean_scrub_stamp":"2024-03-31T15:06:24.244870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:06:25.246926+0000","last_change":"2024-03-31T15:06:25.246926+0000","last_active":"2024-03-31T15:06:25.246926+0000","last_peered":"2024-03-31T15:06:25.246926+0000","last_clean":"2024-03-31T15:06:25.246926+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:06:25.246926+0000","last_undegraded":"2024-03-31T15:06:25.246926+0000","last_fullsized":"2024-03-31T15:06:25.246926+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_clean_scrub_stamp":"2024-03-31T15:06:25.246926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":14,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-31T15:06:27.347749+0000","last_change":"2024-03-31T15:06:25.273987+0000","last_active":"2024-03-31T15:06:27.347749+0000","last_peered":"2024-03-31T15:06:27.347749+0000","last_clean":"2024-03-31T15:06:27.347749+0000","last_became_active":"2024-03-31T15:06:25.273413+0000","last_became_peered":"2024-03-31T15:06:25.273413+0000","last_unstale":"2024-03-31T15:06:27.347749+0000","last_undegraded":"2024-03-31T15:06:27.347749+0000","last_fullsized":"2024-03-31T15:06:27.347749+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:06:24.244870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:06:24.244870+0000","last_clean_scrub_stamp":"2024-03-31T15:06:24.244870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T16:02:01.742318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:06:25.246926+0000","last_change":"2024-03-31T15:06:25.246926+0000","last_active":"2024-03-31T15:06:25.246926+0000","last_peered":"2024-03-31T15:06:25.246926+0000","last_clean":"2024-03-31T15:06:25.246926+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:06:25.246926+0000","last_undegraded":"2024-03-31T15:06:25.246926+0000","last_fullsized":"2024-03-31T15:06:25.246926+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_clean_scrub_stamp":"2024-03-31T15:06:25.246926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.b","version":"55'2","reported_seq":14,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:06:26.298941+0000","last_change":"2024-03-31T15:06:25.271864+0000","last_active":"2024-03-31T15:06:26.298941+0000","last_peered":"2024-03-31T15:06:26.298941+0000","last_clean":"2024-03-31T15:06:26.298941+0000","last_became_active":"2024-03-31T15:06:25.271554+0000","last_became_peered":"2024-03-31T15:06:25.271554+0000","last_unstale":"2024-03-31T15:06:26.298941+0000","last_undegraded":"2024-03-31T15:06:26.298941+0000","last_fullsized":"2024-03-31T15:06:26.298941+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:06:24.244870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:06:24.244870+0000","last_clean_scrub_stamp":"2024-03-31T15:06:24.244870+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:24:14.868237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.b","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-31T15:06:27.347288+0000","last_change":"2024-03-31T15:06:26.297362+0000","last_active":"2024-03-31T15:06:27.347288+0000","last_peered":"2024-03-31T15:06:27.347288+0000","last_clean":"2024-03-31T15:06:27.347288+0000","last_became_active":"2024-03-31T15:06:26.297013+0000","last_became_peered":"2024-03-31T15:06:26.297013+0000","last_unstale":"2024-03-31T15:06:27.347288+0000","last_undegraded":"2024-03-31T15:06:27.347288+0000","last_fullsized":"2024-03-31T15:06:27.347288+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_clean_scrub_stamp":"2024-03-31T15:06:25.246926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:57:13.747232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:06:26.272769+0000","last_change":"2024-03-31T15:06:25.277123+0000","last_active":"2024-03-31T15:06:26.272769+0000","last_peered":"2024-03-31T15:06:26.272769+0000","last_clean":"2024-03-31T15:06:26.272769+0000","last_became_active":"2024-03-31T15:06:25.276798+0000","last_became_peered":"2024-03-31T15:06:25.276798+0000","last_unstale":"2024-03-31T15:06:26.272769+0000","last_undegraded":"2024-03-31T15:06:26.272769+0000","last_fullsized":"2024-03-31T15:06:26.272769+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:06:24.244870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:06:24.244870+0000","last_clean_scrub_stamp":"2024-03-31T15:06:24.244870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:13:23.354596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-31T15:06:27.347116+0000","last_change":"2024-03-31T15:06:26.292230+0000","last_active":"2024-03-31T15:06:27.347116+0000","last_peered":"2024-03-31T15:06:27.347116+0000","last_clean":"2024-03-31T15:06:27.347116+0000","last_became_active":"2024-03-31T15:06:26.291968+0000","last_became_peered":"2024-03-31T15:06:26.291968+0000","last_unstale":"2024-03-31T15:06:27.347116+0000","last_undegraded":"2024-03-31T15:06:27.347116+0000","last_fullsized":"2024-03-31T15:06:27.347116+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_clean_scrub_stamp":"2024-03-31T15:06:25.246926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T16:29:20.020911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:06:24.244870+0000","last_change":"2024-03-31T15:06:24.244870+0000","last_active":"2024-03-31T15:06:24.244870+0000","last_peered":"2024-03-31T15:06:24.244870+0000","last_clean":"2024-03-31T15:06:24.244870+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:06:24.244870+0000","last_undegraded":"2024-03-31T15:06:24.244870+0000","last_fullsized":"2024-03-31T15:06:24.244870+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:06:24.244870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:06:24.244870+0000","last_clean_scrub_stamp":"2024-03-31T15:06:24.244870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:06:25.246926+0000","last_change":"2024-03-31T15:06:25.246926+0000","last_active":"2024-03-31T15:06:25.246926+0000","last_peered":"2024-03-31T15:06:25.246926+0000","last_clean":"2024-03-31T15:06:25.246926+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:06:25.246926+0000","last_undegraded":"2024-03-31T15:06:25.246926+0000","last_fullsized":"2024-03-31T15:06:25.246926+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_clean_scrub_stamp":"2024-03-31T15:06:25.246926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.7","version":"55'2","reported_seq":14,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:06:26.305707+0000","last_change":"2024-03-31T15:06:25.279243+0000","last_active":"2024-03-31T15:06:26.305707+0000","last_peered":"2024-03-31T15:06:26.305707+0000","last_clean":"2024-03-31T15:06:26.305707+0000","last_became_active":"2024-03-31T15:06:25.278951+0000","last_became_peered":"2024-03-31T15:06:25.278951+0000","last_unstale":"2024-03-31T15:06:26.305707+0000","last_undegraded":"2024-03-31T15:06:26.305707+0000","last_fullsized":"2024-03-31T15:06:26.305707+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:06:24.244870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:06:24.244870+0000","last_clean_scrub_stamp":"2024-03-31T15:06:24.244870+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:26:00.373303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:06:25.246926+0000","last_change":"2024-03-31T15:06:25.246926+0000","last_active":"2024-03-31T15:06:25.246926+0000","last_peered":"2024-03-31T15:06:25.246926+0000","last_clean":"2024-03-31T15:06:25.246926+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:06:25.246926+0000","last_undegraded":"2024-03-31T15:06:25.246926+0000","last_fullsized":"2024-03-31T15:06:25.246926+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_clean_scrub_stamp":"2024-03-31T15:06:25.246926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":14,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-31T15:06:27.347708+0000","last_change":"2024-03-31T15:06:25.274928+0000","last_active":"2024-03-31T15:06:27.347708+0000","last_peered":"2024-03-31T15:06:27.347708+0000","last_clean":"2024-03-31T15:06:27.347708+0000","last_became_active":"2024-03-31T15:06:25.274355+0000","last_became_peered":"2024-03-31T15:06:25.274355+0000","last_unstale":"2024-03-31T15:06:27.347708+0000","last_undegraded":"2024-03-31T15:06:27.347708+0000","last_fullsized":"2024-03-31T15:06:27.347708+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:06:24.244870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:06:24.244870+0000","last_clean_scrub_stamp":"2024-03-31T15:06:24.244870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:18:03.273932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:06:26.297337+0000","last_change":"2024-03-31T15:06:26.292751+0000","last_active":"2024-03-31T15:06:26.297337+0000","last_peered":"2024-03-31T15:06:26.297337+0000","last_clean":"2024-03-31T15:06:26.297337+0000","last_became_active":"2024-03-31T15:06:26.292454+0000","last_became_peered":"2024-03-31T15:06:26.292454+0000","last_unstale":"2024-03-31T15:06:26.297337+0000","last_undegraded":"2024-03-31T15:06:26.297337+0000","last_fullsized":"2024-03-31T15:06:26.297337+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_clean_scrub_stamp":"2024-03-31T15:06:25.246926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T16:59:43.968592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":14,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-31T15:06:27.286952+0000","last_change":"2024-03-31T15:06:25.264755+0000","last_active":"2024-03-31T15:06:27.286952+0000","last_peered":"2024-03-31T15:06:27.286952+0000","last_clean":"2024-03-31T15:06:27.286952+0000","last_became_active":"2024-03-31T15:06:25.264394+0000","last_became_peered":"2024-03-31T15:06:25.264394+0000","last_unstale":"2024-03-31T15:06:27.286952+0000","last_undegraded":"2024-03-31T15:06:27.286952+0000","last_fullsized":"2024-03-31T15:06:27.286952+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:06:24.244870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:06:24.244870+0000","last_clean_scrub_stamp":"2024-03-31T15:06:24.244870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:27:44.434513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-31T15:06:27.347507+0000","last_change":"2024-03-31T15:06:26.300390+0000","last_active":"2024-03-31T15:06:27.347507+0000","last_peered":"2024-03-31T15:06:27.347507+0000","last_clean":"2024-03-31T15:06:27.347507+0000","last_became_active":"2024-03-31T15:06:26.299754+0000","last_became_peered":"2024-03-31T15:06:26.299754+0000","last_unstale":"2024-03-31T15:06:27.347507+0000","last_undegraded":"2024-03-31T15:06:27.347507+0000","last_fullsized":"2024-03-31T15:06:27.347507+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_clean_scrub_stamp":"2024-03-31T15:06:25.246926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:31:27.245880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:06:24.244870+0000","last_change":"2024-03-31T15:06:24.244870+0000","last_active":"2024-03-31T15:06:24.244870+0000","last_peered":"2024-03-31T15:06:24.244870+0000","last_clean":"2024-03-31T15:06:24.244870+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:06:24.244870+0000","last_undegraded":"2024-03-31T15:06:24.244870+0000","last_fullsized":"2024-03-31T15:06:24.244870+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:06:24.244870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:06:24.244870+0000","last_clean_scrub_stamp":"2024-03-31T15:06:24.244870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:06:26.300456+0000","last_change":"2024-03-31T15:06:26.293083+0000","last_active":"2024-03-31T15:06:26.300456+0000","last_peered":"2024-03-31T15:06:26.300456+0000","last_clean":"2024-03-31T15:06:26.300456+0000","last_became_active":"2024-03-31T15:06:26.290726+0000","last_became_peered":"2024-03-31T15:06:26.290726+0000","last_unstale":"2024-03-31T15:06:26.300456+0000","last_undegraded":"2024-03-31T15:06:26.300456+0000","last_fullsized":"2024-03-31T15:06:26.300456+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_clean_scrub_stamp":"2024-03-31T15:06:25.246926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:06:19.784539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:06:26.272845+0000","last_change":"2024-03-31T15:06:25.276449+0000","last_active":"2024-03-31T15:06:26.272845+0000","last_peered":"2024-03-31T15:06:26.272845+0000","last_clean":"2024-03-31T15:06:26.272845+0000","last_became_active":"2024-03-31T15:06:25.276104+0000","last_became_peered":"2024-03-31T15:06:25.276104+0000","last_unstale":"2024-03-31T15:06:26.272845+0000","last_undegraded":"2024-03-31T15:06:26.272845+0000","last_fullsized":"2024-03-31T15:06:26.272845+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:06:24.244870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:06:24.244870+0000","last_clean_scrub_stamp":"2024-03-31T15:06:24.244870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T15:28:49.048982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:06:25.246926+0000","last_change":"2024-03-31T15:06:25.246926+0000","last_active":"2024-03-31T15:06:25.246926+0000","last_peered":"2024-03-31T15:06:25.246926+0000","last_clean":"2024-03-31T15:06:25.246926+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:06:25.246926+0000","last_undegraded":"2024-03-31T15:06:25.246926+0000","last_fullsized":"2024-03-31T15:06:25.246926+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_clean_scrub_stamp":"2024-03-31T15:06:25.246926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:06:24.244870+0000","last_change":"2024-03-31T15:06:24.244870+0000","last_active":"2024-03-31T15:06:24.244870+0000","last_peered":"2024-03-31T15:06:24.244870+0000","last_clean":"2024-03-31T15:06:24.244870+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:06:24.244870+0000","last_undegraded":"2024-03-31T15:06:24.244870+0000","last_fullsized":"2024-03-31T15:06:24.244870+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:06:24.244870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:06:24.244870+0000","last_clean_scrub_stamp":"2024-03-31T15:06:24.244870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:06:26.285376+0000","last_change":"2024-03-31T15:06:26.278792+0000","last_active":"2024-03-31T15:06:26.285376+0000","last_peered":"2024-03-31T15:06:26.285376+0000","last_clean":"2024-03-31T15:06:26.285376+0000","last_became_active":"2024-03-31T15:06:26.278505+0000","last_became_peered":"2024-03-31T15:06:26.278505+0000","last_unstale":"2024-03-31T15:06:26.285376+0000","last_undegraded":"2024-03-31T15:06:26.285376+0000","last_fullsized":"2024-03-31T15:06:26.285376+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_clean_scrub_stamp":"2024-03-31T15:06:25.246926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:27:45.436570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:06:26.272617+0000","last_change":"2024-03-31T15:06:25.277849+0000","last_active":"2024-03-31T15:06:26.272617+0000","last_peered":"2024-03-31T15:06:26.272617+0000","last_clean":"2024-03-31T15:06:26.272617+0000","last_became_active":"2024-03-31T15:06:25.277570+0000","last_became_peered":"2024-03-31T15:06:25.277570+0000","last_unstale":"2024-03-31T15:06:26.272617+0000","last_undegraded":"2024-03-31T15:06:26.272617+0000","last_fullsized":"2024-03-31T15:06:26.272617+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:06:24.244870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:06:24.244870+0000","last_clean_scrub_stamp":"2024-03-31T15:06:24.244870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:27:02.294805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:06:26.305453+0000","last_change":"2024-03-31T15:06:26.302007+0000","last_active":"2024-03-31T15:06:26.305453+0000","last_peered":"2024-03-31T15:06:26.305453+0000","last_clean":"2024-03-31T15:06:26.305453+0000","last_became_active":"2024-03-31T15:06:26.294389+0000","last_became_peered":"2024-03-31T15:06:26.294389+0000","last_unstale":"2024-03-31T15:06:26.305453+0000","last_undegraded":"2024-03-31T15:06:26.305453+0000","last_fullsized":"2024-03-31T15:06:26.305453+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_clean_scrub_stamp":"2024-03-31T15:06:25.246926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:59:03.602178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:06:26.269316+0000","last_change":"2024-03-31T15:06:25.274053+0000","last_active":"2024-03-31T15:06:26.269316+0000","last_peered":"2024-03-31T15:06:26.269316+0000","last_clean":"2024-03-31T15:06:26.269316+0000","last_became_active":"2024-03-31T15:06:25.273747+0000","last_became_peered":"2024-03-31T15:06:25.273747+0000","last_unstale":"2024-03-31T15:06:26.269316+0000","last_undegraded":"2024-03-31T15:06:26.269316+0000","last_fullsized":"2024-03-31T15:06:26.269316+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:06:24.244870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:06:24.244870+0000","last_clean_scrub_stamp":"2024-03-31T15:06:24.244870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:31:26.243823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:06:26.303655+0000","last_change":"2024-03-31T15:06:26.301578+0000","last_active":"2024-03-31T15:06:26.303655+0000","last_peered":"2024-03-31T15:06:26.303655+0000","last_clean":"2024-03-31T15:06:26.303655+0000","last_became_active":"2024-03-31T15:06:26.301293+0000","last_became_peered":"2024-03-31T15:06:26.301293+0000","last_unstale":"2024-03-31T15:06:26.303655+0000","last_undegraded":"2024-03-31T15:06:26.303655+0000","last_fullsized":"2024-03-31T15:06:26.303655+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_clean_scrub_stamp":"2024-03-31T15:06:25.246926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:02:22.084360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":14,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-31T15:06:27.286677+0000","last_change":"2024-03-31T15:06:25.271451+0000","last_active":"2024-03-31T15:06:27.286677+0000","last_peered":"2024-03-31T15:06:27.286677+0000","last_clean":"2024-03-31T15:06:27.286677+0000","last_became_active":"2024-03-31T15:06:25.270955+0000","last_became_peered":"2024-03-31T15:06:25.270955+0000","last_unstale":"2024-03-31T15:06:27.286677+0000","last_undegraded":"2024-03-31T15:06:27.286677+0000","last_fullsized":"2024-03-31T15:06:27.286677+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:06:24.244870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:06:24.244870+0000","last_clean_scrub_stamp":"2024-03-31T15:06:24.244870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:37:09.305055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":226,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-31T15:06:22.969424+0000","last_change":"2024-03-31T15:03:24.329890+0000","last_active":"2024-03-31T15:06:22.969424+0000","last_peered":"2024-03-31T15:06:22.969424+0000","last_clean":"2024-03-31T15:06:22.969424+0000","last_became_active":"2024-03-31T15:03:24.328882+0000","last_became_peered":"2024-03-31T15:03:24.328882+0000","last_unstale":"2024-03-31T15:06:22.969424+0000","last_undegraded":"2024-03-31T15:06:22.969424+0000","last_fullsized":"2024-03-31T15:06:22.969424+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:22.885535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:22.885535+0000","last_clean_scrub_stamp":"2024-03-31T15:03:22.885535+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:18:29.004296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":4},{"poolid":8,"num_pg":32,"stat_sum":{"num_bytes":638,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":2340,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":38,"acting":38,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509481,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28200,"kb_used_data":1180,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343640,"statfs":{"total":96636764160,"available":96607887360,"internally_reserved":0,"allocated":1208320,"data_stored":840162,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509480,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27624,"kb_used_data":600,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344216,"statfs":{"total":96636764160,"available":96608477184,"internally_reserved":0,"allocated":614400,"data_stored":249794,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":15,"seq":64424509479,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27632,"kb_used_data":600,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344208,"statfs":{"total":96636764160,"available":96608468992,"internally_reserved":0,"allocated":614400,"data_stored":249794,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509478,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31780,"kb_used_data":600,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340060,"statfs":{"total":96636764160,"available":96604221440,"internally_reserved":0,"allocated":614400,"data_stored":249794,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542185,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31828,"kb_used_data":648,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340012,"statfs":{"total":96636764160,"available":96604172288,"internally_reserved":0,"allocated":663552,"data_stored":274953,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542185,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31832,"kb_used_data":644,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340008,"statfs":{"total":96636764160,"available":96604168192,"internally_reserved":0,"allocated":659456,"data_stored":268709,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542185,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31828,"kb_used_data":640,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340012,"statfs":{"total":96636764160,"available":96604172288,"internally_reserved":0,"allocated":655360,"data_stored":268743,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542185,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32412,"kb_used_data":1232,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339428,"statfs":{"total":96636764160,"available":96603574272,"internally_reserved":0,"allocated":1261568,"data_stored":861215,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-31T15:06:30.424 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-31T15:06:30.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:30.607+0000 7f8bf0800640 1 Processor -- start 2024-03-31T15:06:30.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:30.607+0000 7f8bf0800640 1 -- start start 2024-03-31T15:06:30.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:30.611+0000 7f8bf0800640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f8be80b23d0 0x7f8be8071d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:30.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:30.611+0000 7f8bf0800640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f8be815eee0 0x7f8be8072250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:30.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:30.611+0000 7f8bf0800640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8be815f3c0 con 0x7f8be815b3d0 2024-03-31T15:06:30.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:30.611+0000 7f8bf0800640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f8be815b7b0 con 0x7f8be815eee0 2024-03-31T15:06:30.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:30.611+0000 7f8bf0800640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f8be815f6a0 con 0x7f8be80b23d0 2024-03-31T15:06:30.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:30.611+0000 7f8bede00640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f8be815b3d0 0x7f8be80b1cc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:56388/0 (socket says 172.21.15.16:56388) 2024-03-31T15:06:30.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:30.611+0000 7f8bede00640 1 -- 172.21.15.16:0/548313215 learned_addr learned my addr 172.21.15.16:0/548313215 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:06:30.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:30.611+0000 7f8bed400640 1 --2- 172.21.15.16:0/548313215 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f8be80b23d0 0x7f8be8071d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:30.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:30.611+0000 7f8bee800640 1 --2- 172.21.15.16:0/548313215 >> v2:172.21.15.89:3300/0 conn(0x7f8be815eee0 0x7f8be8072250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:30.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:30.611+0000 7f8beca00640 1 -- 172.21.15.16:0/548313215 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1192217138 0 0) 0x7f8be815f3c0 con 0x7f8be815b3d0 2024-03-31T15:06:30.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:30.611+0000 7f8beca00640 1 -- 172.21.15.16:0/548313215 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8bd0003530 con 0x7f8be815b3d0 2024-03-31T15:06:30.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:30.611+0000 7f8bed400640 1 -- 172.21.15.16:0/548313215 >> v2:172.21.15.89:3300/0 conn(0x7f8be815eee0 msgr2=0x7f8be8072250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:06:30.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:30.611+0000 7f8bed400640 1 --2- 172.21.15.16:0/548313215 >> v2:172.21.15.89:3300/0 conn(0x7f8be815eee0 0x7f8be8072250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:30.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:30.611+0000 7f8bed400640 1 -- 172.21.15.16:0/548313215 >> v1:172.21.15.16:6789/0 conn(0x7f8be815b3d0 legacy=0x7f8be80b1cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:30.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:30.611+0000 7f8beca00640 1 -- 172.21.15.16:0/548313215 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3587727971 0 0) 0x7f8bd0003530 con 0x7f8be815b3d0 2024-03-31T15:06:30.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:30.611+0000 7f8bed400640 1 -- 172.21.15.16:0/548313215 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8be815f980 con 0x7f8be80b23d0 2024-03-31T15:06:30.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:30.615+0000 7f8bed400640 1 --2- 172.21.15.16:0/548313215 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f8be80b23d0 0x7f8be8071d10 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f8bd80097c0 tx=0x7f8bd801c390 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1d6e8a4c92b9c11a server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:06:30.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:30.615+0000 7f8beca00640 1 -- 172.21.15.16:0/548313215 <== mon.2 v2:172.21.15.89:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bd80b0070 con 0x7f8be80b23d0 2024-03-31T15:06:30.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:30.615+0000 7f8beca00640 1 -- 172.21.15.16:0/548313215 <== mon.2 v2:172.21.15.89:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8bd80ab400 con 0x7f8be80b23d0 2024-03-31T15:06:30.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:30.615+0000 7f8beca00640 1 -- 172.21.15.16:0/548313215 <== mon.2 v2:172.21.15.89:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bdc001680 con 0x7f8be80b23d0 2024-03-31T15:06:30.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:30.615+0000 7f8bf0800640 1 -- 172.21.15.16:0/548313215 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f8be80b23d0 msgr2=0x7f8be8071d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:30.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:30.615+0000 7f8bf0800640 1 --2- 172.21.15.16:0/548313215 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f8be80b23d0 0x7f8be8071d10 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f8bd80097c0 tx=0x7f8bd801c390 comp rx=0 tx=0).stop 2024-03-31T15:06:30.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:30.615+0000 7f8bf0800640 1 -- 172.21.15.16:0/548313215 shutdown_connections 2024-03-31T15:06:30.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:30.615+0000 7f8bf0800640 1 --2- 172.21.15.16:0/548313215 >> v2:172.21.15.89:3300/0 conn(0x7f8be815eee0 0x7f8be8072250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:30.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:30.615+0000 7f8bf0800640 1 --2- 172.21.15.16:0/548313215 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f8be80b23d0 0x7f8be8071d10 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:30.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:30.615+0000 7f8bf0800640 1 -- 172.21.15.16:0/548313215 >> 172.21.15.16:0/548313215 conn(0x7f8be806b620 msgr2=0x7f8be806ba30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:06:30.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:30.615+0000 7f8bf0800640 1 -- 172.21.15.16:0/548313215 shutdown_connections 2024-03-31T15:06:30.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:30.615+0000 7f8bf0800640 1 -- 172.21.15.16:0/548313215 wait complete. 2024-03-31T15:06:30.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:30.615+0000 7f8bf0800640 1 Processor -- start 2024-03-31T15:06:30.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:30.615+0000 7f8bf0800640 1 -- start start 2024-03-31T15:06:30.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:30.619+0000 7f8bf0800640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8be80b23d0 0x7f8be80ced10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:30.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:30.619+0000 7f8bf0800640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f8be815b3d0 0x7f8be8141320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:30.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:30.619+0000 7f8bf0800640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8be815eee0 0x7f8be8141860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:30.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:30.619+0000 7f8bf0800640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f8be8072790 con 0x7f8be815eee0 2024-03-31T15:06:30.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:30.619+0000 7f8bf0800640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f8be80737d0 con 0x7f8be80b23d0 2024-03-31T15:06:30.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:30.619+0000 7f8bf0800640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f8be8072910 con 0x7f8be815b3d0 2024-03-31T15:06:30.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:30.619+0000 7f8bee800640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8be815eee0 0x7f8be8141860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:30.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:30.619+0000 7f8bee800640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8be815eee0 0x7f8be8141860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49042/0 (socket says 172.21.15.16:49042) 2024-03-31T15:06:30.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:30.619+0000 7f8bee800640 1 -- 172.21.15.16:0/3711419121 learned_addr learned my addr 172.21.15.16:0/3711419121 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:06:30.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:30.619+0000 7f8bed400640 1 --2- 172.21.15.16:0/3711419121 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f8be815b3d0 0x7f8be8141320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:30.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:30.619+0000 7f8bede00640 1 --2- 172.21.15.16:0/3711419121 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8be80b23d0 0x7f8be80ced10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:30.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:30.619+0000 7f8bee800640 1 -- 172.21.15.16:0/3711419121 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f8be815b3d0 msgr2=0x7f8be8141320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:30.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:30.619+0000 7f8bee800640 1 --2- 172.21.15.16:0/3711419121 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f8be815b3d0 0x7f8be8141320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:30.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:30.619+0000 7f8bee800640 1 -- 172.21.15.16:0/3711419121 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8be80b23d0 msgr2=0x7f8be80ced10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:30.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:30.619+0000 7f8bee800640 1 --2- 172.21.15.16:0/3711419121 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8be80b23d0 0x7f8be80ced10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:30.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:30.619+0000 7f8bee800640 1 -- 172.21.15.16:0/3711419121 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8be8141da0 con 0x7f8be815eee0 2024-03-31T15:06:30.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:30.619+0000 7f8bed400640 1 --2- 172.21.15.16:0/3711419121 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f8be815b3d0 0x7f8be8141320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:06:30.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:30.619+0000 7f8bede00640 1 --2- 172.21.15.16:0/3711419121 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8be80b23d0 0x7f8be80ced10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:06:30.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:30.619+0000 7f8bee800640 1 --2- 172.21.15.16:0/3711419121 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8be815eee0 0x7f8be8141860 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f8be400efc0 tx=0x7f8be400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:06:30.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:30.619+0000 7f8be2000640 1 -- 172.21.15.16:0/3711419121 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8be4018020 con 0x7f8be815eee0 2024-03-31T15:06:30.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:30.619+0000 7f8be2000640 1 -- 172.21.15.16:0/3711419121 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8be4014480 con 0x7f8be815eee0 2024-03-31T15:06:30.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:30.619+0000 7f8be2000640 1 -- 172.21.15.16:0/3711419121 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8be4014870 con 0x7f8be815eee0 2024-03-31T15:06:30.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:30.619+0000 7f8bf0800640 1 -- 172.21.15.16:0/3711419121 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8be8148f70 con 0x7f8be815eee0 2024-03-31T15:06:30.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:30.619+0000 7f8bf0800640 1 -- 172.21.15.16:0/3711419121 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8be8165eb0 con 0x7f8be815eee0 2024-03-31T15:06:30.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:30.619+0000 7f8bf0800640 1 -- 172.21.15.16:0/3711419121 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8be81659c0 con 0x7f8be815eee0 2024-03-31T15:06:30.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:30.623+0000 7f8be2000640 1 -- 172.21.15.16:0/3711419121 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f8be401b020 con 0x7f8be815eee0 2024-03-31T15:06:30.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:30.623+0000 7f8be2000640 1 --2- 172.21.15.16:0/3711419121 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f8bcc068380 0x7f8bcc06a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:30.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:30.623+0000 7f8be2000640 1 -- 172.21.15.16:0/3711419121 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 5709+0+0 (secure 0 0 0) 0x7f8be4087830 con 0x7f8be815eee0 2024-03-31T15:06:30.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:30.623+0000 7f8bede00640 1 --2- 172.21.15.16:0/3711419121 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f8bcc068380 0x7f8bcc06a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:30.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:30.623+0000 7f8bede00640 1 --2- 172.21.15.16:0/3711419121 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f8bcc068380 0x7f8bcc06a840 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f8bdc007a10 tx=0x7f8bdc009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:06:30.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:30.627+0000 7f8be2000640 1 -- 172.21.15.16:0/3711419121 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f8be4022030 con 0x7f8be815eee0 2024-03-31T15:06:30.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:30.791+0000 7f8bf0800640 1 -- 172.21.15.16:0/3711419121 --> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f8be8069950 con 0x7f8bcc068380 2024-03-31T15:06:30.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:30.795+0000 7f8be2000640 1 -- 172.21.15.16:0/3711419121 <== mgr.4100 v2:172.21.15.89:6832/1549407273 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251931 (secure 0 0 0) 0x7f8be8069950 con 0x7f8bcc068380 2024-03-31T15:06:30.794 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:06:30.796 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-03-31T15:06:30.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:30.799+0000 7f8bf0800640 1 -- 172.21.15.16:0/3711419121 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f8bcc068380 msgr2=0x7f8bcc06a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:30.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:30.799+0000 7f8bf0800640 1 --2- 172.21.15.16:0/3711419121 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f8bcc068380 0x7f8bcc06a840 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f8bdc007a10 tx=0x7f8bdc009040 comp rx=0 tx=0).stop 2024-03-31T15:06:30.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:30.799+0000 7f8bf0800640 1 -- 172.21.15.16:0/3711419121 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8be815eee0 msgr2=0x7f8be8141860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:30.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:30.799+0000 7f8bf0800640 1 --2- 172.21.15.16:0/3711419121 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8be815eee0 0x7f8be8141860 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f8be400efc0 tx=0x7f8be400c490 comp rx=0 tx=0).stop 2024-03-31T15:06:30.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:30.799+0000 7f8bf0800640 1 -- 172.21.15.16:0/3711419121 shutdown_connections 2024-03-31T15:06:30.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:30.799+0000 7f8bf0800640 1 --2- 172.21.15.16:0/3711419121 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f8bcc068380 0x7f8bcc06a840 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:30.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:30.799+0000 7f8bf0800640 1 --2- 172.21.15.16:0/3711419121 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8be815eee0 0x7f8be8141860 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:30.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:30.799+0000 7f8bf0800640 1 --2- 172.21.15.16:0/3711419121 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f8be815b3d0 0x7f8be8141320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:30.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:30.799+0000 7f8bf0800640 1 --2- 172.21.15.16:0/3711419121 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8be80b23d0 0x7f8be80ced10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:30.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:30.799+0000 7f8bf0800640 1 -- 172.21.15.16:0/3711419121 >> 172.21.15.16:0/3711419121 conn(0x7f8be806b620 msgr2=0x7f8be814f670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:06:30.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:30.799+0000 7f8bf0800640 1 -- 172.21.15.16:0/3711419121 shutdown_connections 2024-03-31T15:06:30.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:30.799+0000 7f8bf0800640 1 -- 172.21.15.16:0/3711419121 wait complete. 2024-03-31T15:06:30.815 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":143,"stamp":"2024-03-31T15:06:30.568287+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":102,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":247520,"kb_used_data":6360,"kb_used_omap":12,"kb_used_meta":241011,"kb_avail":754727200,"statfs":{"total":773094113280,"available":772840652800,"internally_reserved":0,"allocated":6512640,"data_stored":3359980,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":246795824},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":33,"apply_latency_ms":33,"commit_latency_ns":33000000,"apply_latency_ns":33000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001667"},"pg_stats":[{"pgid":"9.3f","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-31T15:06:28.607369+0000","last_change":"2024-03-31T15:06:26.305076+0000","last_active":"2024-03-31T15:06:28.607369+0000","last_peered":"2024-03-31T15:06:28.607369+0000","last_clean":"2024-03-31T15:06:28.607369+0000","last_became_active":"2024-03-31T15:06:26.304748+0000","last_became_peered":"2024-03-31T15:06:26.304748+0000","last_unstale":"2024-03-31T15:06:28.607369+0000","last_undegraded":"2024-03-31T15:06:28.607369+0000","last_fullsized":"2024-03-31T15:06:28.607369+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_clean_scrub_stamp":"2024-03-31T15:06:25.246926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T19:43:19.656133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3e","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:06:26.300576+0000","last_change":"2024-03-31T15:06:26.291088+0000","last_active":"2024-03-31T15:06:26.300576+0000","last_peered":"2024-03-31T15:06:26.300576+0000","last_clean":"2024-03-31T15:06:26.300576+0000","last_became_active":"2024-03-31T15:06:26.290740+0000","last_became_peered":"2024-03-31T15:06:26.290740+0000","last_unstale":"2024-03-31T15:06:26.300576+0000","last_undegraded":"2024-03-31T15:06:26.300576+0000","last_fullsized":"2024-03-31T15:06:26.300576+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_clean_scrub_stamp":"2024-03-31T15:06:25.246926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T17:23:30.654478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.3d","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-31T15:06:27.286075+0000","last_change":"2024-03-31T15:06:26.285191+0000","last_active":"2024-03-31T15:06:27.286075+0000","last_peered":"2024-03-31T15:06:27.286075+0000","last_clean":"2024-03-31T15:06:27.286075+0000","last_became_active":"2024-03-31T15:06:26.284871+0000","last_became_peered":"2024-03-31T15:06:26.284871+0000","last_unstale":"2024-03-31T15:06:27.286075+0000","last_undegraded":"2024-03-31T15:06:27.286075+0000","last_fullsized":"2024-03-31T15:06:27.286075+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_clean_scrub_stamp":"2024-03-31T15:06:25.246926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T17:30:45.401084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.3c","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-31T15:06:27.284028+0000","last_change":"2024-03-31T15:06:26.295039+0000","last_active":"2024-03-31T15:06:27.284028+0000","last_peered":"2024-03-31T15:06:27.284028+0000","last_clean":"2024-03-31T15:06:27.284028+0000","last_became_active":"2024-03-31T15:06:26.294762+0000","last_became_peered":"2024-03-31T15:06:26.294762+0000","last_unstale":"2024-03-31T15:06:27.284028+0000","last_undegraded":"2024-03-31T15:06:27.284028+0000","last_fullsized":"2024-03-31T15:06:27.284028+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_clean_scrub_stamp":"2024-03-31T15:06:25.246926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:25:01.258373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3b","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-31T15:06:28.605508+0000","last_change":"2024-03-31T15:06:26.298806+0000","last_active":"2024-03-31T15:06:28.605508+0000","last_peered":"2024-03-31T15:06:28.605508+0000","last_clean":"2024-03-31T15:06:28.605508+0000","last_became_active":"2024-03-31T15:06:26.297567+0000","last_became_peered":"2024-03-31T15:06:26.297567+0000","last_unstale":"2024-03-31T15:06:28.605508+0000","last_undegraded":"2024-03-31T15:06:28.605508+0000","last_fullsized":"2024-03-31T15:06:28.605508+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_clean_scrub_stamp":"2024-03-31T15:06:25.246926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:14:38.475335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3a","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-31T15:06:27.347135+0000","last_change":"2024-03-31T15:06:26.297322+0000","last_active":"2024-03-31T15:06:27.347135+0000","last_peered":"2024-03-31T15:06:27.347135+0000","last_clean":"2024-03-31T15:06:27.347135+0000","last_became_active":"2024-03-31T15:06:26.296993+0000","last_became_peered":"2024-03-31T15:06:26.296993+0000","last_unstale":"2024-03-31T15:06:27.347135+0000","last_undegraded":"2024-03-31T15:06:27.347135+0000","last_fullsized":"2024-03-31T15:06:27.347135+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_clean_scrub_stamp":"2024-03-31T15:06:25.246926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:13:41.882384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.39","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-31T15:06:27.505294+0000","last_change":"2024-03-31T15:06:26.292263+0000","last_active":"2024-03-31T15:06:27.505294+0000","last_peered":"2024-03-31T15:06:27.505294+0000","last_clean":"2024-03-31T15:06:27.505294+0000","last_became_active":"2024-03-31T15:06:26.289178+0000","last_became_peered":"2024-03-31T15:06:26.289178+0000","last_unstale":"2024-03-31T15:06:27.505294+0000","last_undegraded":"2024-03-31T15:06:27.505294+0000","last_fullsized":"2024-03-31T15:06:27.505294+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_clean_scrub_stamp":"2024-03-31T15:06:25.246926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:32:10.984838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.38","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:06:26.295455+0000","last_change":"2024-03-31T15:06:26.284320+0000","last_active":"2024-03-31T15:06:26.295455+0000","last_peered":"2024-03-31T15:06:26.295455+0000","last_clean":"2024-03-31T15:06:26.295455+0000","last_became_active":"2024-03-31T15:06:26.283991+0000","last_became_peered":"2024-03-31T15:06:26.283991+0000","last_unstale":"2024-03-31T15:06:26.295455+0000","last_undegraded":"2024-03-31T15:06:26.295455+0000","last_fullsized":"2024-03-31T15:06:26.295455+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_clean_scrub_stamp":"2024-03-31T15:06:25.246926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:25:45.811474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.37","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-31T15:06:27.505282+0000","last_change":"2024-03-31T15:06:26.304683+0000","last_active":"2024-03-31T15:06:27.505282+0000","last_peered":"2024-03-31T15:06:27.505282+0000","last_clean":"2024-03-31T15:06:27.505282+0000","last_became_active":"2024-03-31T15:06:26.304347+0000","last_became_peered":"2024-03-31T15:06:26.304347+0000","last_unstale":"2024-03-31T15:06:27.505282+0000","last_undegraded":"2024-03-31T15:06:27.505282+0000","last_fullsized":"2024-03-31T15:06:27.505282+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_clean_scrub_stamp":"2024-03-31T15:06:25.246926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:50:33.745864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.36","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:06:26.297518+0000","last_change":"2024-03-31T15:06:26.291183+0000","last_active":"2024-03-31T15:06:26.297518+0000","last_peered":"2024-03-31T15:06:26.297518+0000","last_clean":"2024-03-31T15:06:26.297518+0000","last_became_active":"2024-03-31T15:06:26.290916+0000","last_became_peered":"2024-03-31T15:06:26.290916+0000","last_unstale":"2024-03-31T15:06:26.297518+0000","last_undegraded":"2024-03-31T15:06:26.297518+0000","last_fullsized":"2024-03-31T15:06:26.297518+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_clean_scrub_stamp":"2024-03-31T15:06:25.246926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T17:39:02.515165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.35","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:06:26.301309+0000","last_change":"2024-03-31T15:06:26.294325+0000","last_active":"2024-03-31T15:06:26.301309+0000","last_peered":"2024-03-31T15:06:26.301309+0000","last_clean":"2024-03-31T15:06:26.301309+0000","last_became_active":"2024-03-31T15:06:26.294054+0000","last_became_peered":"2024-03-31T15:06:26.294054+0000","last_unstale":"2024-03-31T15:06:26.301309+0000","last_undegraded":"2024-03-31T15:06:26.301309+0000","last_fullsized":"2024-03-31T15:06:26.301309+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_clean_scrub_stamp":"2024-03-31T15:06:25.246926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:28:08.464687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.34","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-31T15:06:27.284144+0000","last_change":"2024-03-31T15:06:26.286751+0000","last_active":"2024-03-31T15:06:27.284144+0000","last_peered":"2024-03-31T15:06:27.284144+0000","last_clean":"2024-03-31T15:06:27.284144+0000","last_became_active":"2024-03-31T15:06:26.285763+0000","last_became_peered":"2024-03-31T15:06:26.285763+0000","last_unstale":"2024-03-31T15:06:27.284144+0000","last_undegraded":"2024-03-31T15:06:27.284144+0000","last_fullsized":"2024-03-31T15:06:27.284144+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_clean_scrub_stamp":"2024-03-31T15:06:25.246926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T19:56:39.238117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.33","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-31T15:06:27.286942+0000","last_change":"2024-03-31T15:06:26.292241+0000","last_active":"2024-03-31T15:06:27.286942+0000","last_peered":"2024-03-31T15:06:27.286942+0000","last_clean":"2024-03-31T15:06:27.286942+0000","last_became_active":"2024-03-31T15:06:26.291928+0000","last_became_peered":"2024-03-31T15:06:26.291928+0000","last_unstale":"2024-03-31T15:06:27.286942+0000","last_undegraded":"2024-03-31T15:06:27.286942+0000","last_fullsized":"2024-03-31T15:06:27.286942+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_clean_scrub_stamp":"2024-03-31T15:06:25.246926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:59:13.871341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.32","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-31T15:06:27.504978+0000","last_change":"2024-03-31T15:06:26.289642+0000","last_active":"2024-03-31T15:06:27.504978+0000","last_peered":"2024-03-31T15:06:27.504978+0000","last_clean":"2024-03-31T15:06:27.504978+0000","last_became_active":"2024-03-31T15:06:26.289419+0000","last_became_peered":"2024-03-31T15:06:26.289419+0000","last_unstale":"2024-03-31T15:06:27.504978+0000","last_undegraded":"2024-03-31T15:06:27.504978+0000","last_fullsized":"2024-03-31T15:06:27.504978+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_clean_scrub_stamp":"2024-03-31T15:06:25.246926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T16:46:35.821268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.31","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-31T15:06:27.505299+0000","last_change":"2024-03-31T15:06:26.281328+0000","last_active":"2024-03-31T15:06:27.505299+0000","last_peered":"2024-03-31T15:06:27.505299+0000","last_clean":"2024-03-31T15:06:27.505299+0000","last_became_active":"2024-03-31T15:06:26.280565+0000","last_became_peered":"2024-03-31T15:06:26.280565+0000","last_unstale":"2024-03-31T15:06:27.505299+0000","last_undegraded":"2024-03-31T15:06:27.505299+0000","last_fullsized":"2024-03-31T15:06:27.505299+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_clean_scrub_stamp":"2024-03-31T15:06:25.246926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:05:30.824600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.30","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-31T15:06:27.347466+0000","last_change":"2024-03-31T15:06:26.298305+0000","last_active":"2024-03-31T15:06:27.347466+0000","last_peered":"2024-03-31T15:06:27.347466+0000","last_clean":"2024-03-31T15:06:27.347466+0000","last_became_active":"2024-03-31T15:06:26.296491+0000","last_became_peered":"2024-03-31T15:06:26.296491+0000","last_unstale":"2024-03-31T15:06:27.347466+0000","last_undegraded":"2024-03-31T15:06:27.347466+0000","last_fullsized":"2024-03-31T15:06:27.347466+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_clean_scrub_stamp":"2024-03-31T15:06:25.246926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:50:32.468918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.2f","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:06:26.300835+0000","last_change":"2024-03-31T15:06:26.294641+0000","last_active":"2024-03-31T15:06:26.300835+0000","last_peered":"2024-03-31T15:06:26.300835+0000","last_clean":"2024-03-31T15:06:26.300835+0000","last_became_active":"2024-03-31T15:06:26.294358+0000","last_became_peered":"2024-03-31T15:06:26.294358+0000","last_unstale":"2024-03-31T15:06:26.300835+0000","last_undegraded":"2024-03-31T15:06:26.300835+0000","last_fullsized":"2024-03-31T15:06:26.300835+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_clean_scrub_stamp":"2024-03-31T15:06:25.246926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T16:42:20.504930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2e","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-31T15:06:27.286095+0000","last_change":"2024-03-31T15:06:26.293021+0000","last_active":"2024-03-31T15:06:27.286095+0000","last_peered":"2024-03-31T15:06:27.286095+0000","last_clean":"2024-03-31T15:06:27.286095+0000","last_became_active":"2024-03-31T15:06:26.292540+0000","last_became_peered":"2024-03-31T15:06:26.292540+0000","last_unstale":"2024-03-31T15:06:27.286095+0000","last_undegraded":"2024-03-31T15:06:27.286095+0000","last_fullsized":"2024-03-31T15:06:27.286095+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_clean_scrub_stamp":"2024-03-31T15:06:25.246926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:34:47.288253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2d","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-31T15:06:28.606866+0000","last_change":"2024-03-31T15:06:26.303270+0000","last_active":"2024-03-31T15:06:28.606866+0000","last_peered":"2024-03-31T15:06:28.606866+0000","last_clean":"2024-03-31T15:06:28.606866+0000","last_became_active":"2024-03-31T15:06:26.302964+0000","last_became_peered":"2024-03-31T15:06:26.302964+0000","last_unstale":"2024-03-31T15:06:28.606866+0000","last_undegraded":"2024-03-31T15:06:28.606866+0000","last_fullsized":"2024-03-31T15:06:28.606866+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_clean_scrub_stamp":"2024-03-31T15:06:25.246926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:52:00.410497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.2c","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:06:26.300517+0000","last_change":"2024-03-31T15:06:26.297250+0000","last_active":"2024-03-31T15:06:26.300517+0000","last_peered":"2024-03-31T15:06:26.300517+0000","last_clean":"2024-03-31T15:06:26.300517+0000","last_became_active":"2024-03-31T15:06:26.296974+0000","last_became_peered":"2024-03-31T15:06:26.296974+0000","last_unstale":"2024-03-31T15:06:26.300517+0000","last_undegraded":"2024-03-31T15:06:26.300517+0000","last_fullsized":"2024-03-31T15:06:26.300517+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_clean_scrub_stamp":"2024-03-31T15:06:25.246926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:38:46.324919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2b","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-31T15:06:27.285783+0000","last_change":"2024-03-31T15:06:26.282691+0000","last_active":"2024-03-31T15:06:27.285783+0000","last_peered":"2024-03-31T15:06:27.285783+0000","last_clean":"2024-03-31T15:06:27.285783+0000","last_became_active":"2024-03-31T15:06:26.282391+0000","last_became_peered":"2024-03-31T15:06:26.282391+0000","last_unstale":"2024-03-31T15:06:27.285783+0000","last_undegraded":"2024-03-31T15:06:27.285783+0000","last_fullsized":"2024-03-31T15:06:27.285783+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_clean_scrub_stamp":"2024-03-31T15:06:25.246926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:10:50.564026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2a","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:06:26.300547+0000","last_change":"2024-03-31T15:06:26.296387+0000","last_active":"2024-03-31T15:06:26.300547+0000","last_peered":"2024-03-31T15:06:26.300547+0000","last_clean":"2024-03-31T15:06:26.300547+0000","last_became_active":"2024-03-31T15:06:26.295902+0000","last_became_peered":"2024-03-31T15:06:26.295902+0000","last_unstale":"2024-03-31T15:06:26.300547+0000","last_undegraded":"2024-03-31T15:06:26.300547+0000","last_fullsized":"2024-03-31T15:06:26.300547+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_clean_scrub_stamp":"2024-03-31T15:06:25.246926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:27:24.902121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.29","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-31T15:06:27.285277+0000","last_change":"2024-03-31T15:06:26.286772+0000","last_active":"2024-03-31T15:06:27.285277+0000","last_peered":"2024-03-31T15:06:27.285277+0000","last_clean":"2024-03-31T15:06:27.285277+0000","last_became_active":"2024-03-31T15:06:26.286228+0000","last_became_peered":"2024-03-31T15:06:26.286228+0000","last_unstale":"2024-03-31T15:06:27.285277+0000","last_undegraded":"2024-03-31T15:06:27.285277+0000","last_fullsized":"2024-03-31T15:06:27.285277+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_clean_scrub_stamp":"2024-03-31T15:06:25.246926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T19:56:39.238117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.28","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-31T15:06:27.286270+0000","last_change":"2024-03-31T15:06:26.288990+0000","last_active":"2024-03-31T15:06:27.286270+0000","last_peered":"2024-03-31T15:06:27.286270+0000","last_clean":"2024-03-31T15:06:27.286270+0000","last_became_active":"2024-03-31T15:06:26.288124+0000","last_became_peered":"2024-03-31T15:06:26.288124+0000","last_unstale":"2024-03-31T15:06:27.286270+0000","last_undegraded":"2024-03-31T15:06:27.286270+0000","last_fullsized":"2024-03-31T15:06:27.286270+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_clean_scrub_stamp":"2024-03-31T15:06:25.246926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T15:16:19.589166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.27","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-31T15:06:28.607373+0000","last_change":"2024-03-31T15:06:26.298744+0000","last_active":"2024-03-31T15:06:28.607373+0000","last_peered":"2024-03-31T15:06:28.607373+0000","last_clean":"2024-03-31T15:06:28.607373+0000","last_became_active":"2024-03-31T15:06:26.296995+0000","last_became_peered":"2024-03-31T15:06:26.296995+0000","last_unstale":"2024-03-31T15:06:28.607373+0000","last_undegraded":"2024-03-31T15:06:28.607373+0000","last_fullsized":"2024-03-31T15:06:28.607373+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_clean_scrub_stamp":"2024-03-31T15:06:25.246926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:43:42.001520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.26","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-31T15:06:27.284159+0000","last_change":"2024-03-31T15:06:26.290297+0000","last_active":"2024-03-31T15:06:27.284159+0000","last_peered":"2024-03-31T15:06:27.284159+0000","last_clean":"2024-03-31T15:06:27.284159+0000","last_became_active":"2024-03-31T15:06:26.289695+0000","last_became_peered":"2024-03-31T15:06:26.289695+0000","last_unstale":"2024-03-31T15:06:27.284159+0000","last_undegraded":"2024-03-31T15:06:27.284159+0000","last_fullsized":"2024-03-31T15:06:27.284159+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_clean_scrub_stamp":"2024-03-31T15:06:25.246926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:53:46.360339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.25","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-31T15:06:27.347347+0000","last_change":"2024-03-31T15:06:26.300765+0000","last_active":"2024-03-31T15:06:27.347347+0000","last_peered":"2024-03-31T15:06:27.347347+0000","last_clean":"2024-03-31T15:06:27.347347+0000","last_became_active":"2024-03-31T15:06:26.300480+0000","last_became_peered":"2024-03-31T15:06:26.300480+0000","last_unstale":"2024-03-31T15:06:27.347347+0000","last_undegraded":"2024-03-31T15:06:27.347347+0000","last_fullsized":"2024-03-31T15:06:27.347347+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_clean_scrub_stamp":"2024-03-31T15:06:25.246926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T19:09:44.028119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.24","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-31T15:06:28.606136+0000","last_change":"2024-03-31T15:06:26.295262+0000","last_active":"2024-03-31T15:06:28.606136+0000","last_peered":"2024-03-31T15:06:28.606136+0000","last_clean":"2024-03-31T15:06:28.606136+0000","last_became_active":"2024-03-31T15:06:26.294843+0000","last_became_peered":"2024-03-31T15:06:26.294843+0000","last_unstale":"2024-03-31T15:06:28.606136+0000","last_undegraded":"2024-03-31T15:06:28.606136+0000","last_fullsized":"2024-03-31T15:06:28.606136+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_clean_scrub_stamp":"2024-03-31T15:06:25.246926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:36:33.172451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.23","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:06:26.299003+0000","last_change":"2024-03-31T15:06:26.289798+0000","last_active":"2024-03-31T15:06:26.299003+0000","last_peered":"2024-03-31T15:06:26.299003+0000","last_clean":"2024-03-31T15:06:26.299003+0000","last_became_active":"2024-03-31T15:06:26.289464+0000","last_became_peered":"2024-03-31T15:06:26.289464+0000","last_unstale":"2024-03-31T15:06:26.299003+0000","last_undegraded":"2024-03-31T15:06:26.299003+0000","last_fullsized":"2024-03-31T15:06:26.299003+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_clean_scrub_stamp":"2024-03-31T15:06:25.246926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T15:23:39.306094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.22","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-31T15:06:27.285655+0000","last_change":"2024-03-31T15:06:26.284890+0000","last_active":"2024-03-31T15:06:27.285655+0000","last_peered":"2024-03-31T15:06:27.285655+0000","last_clean":"2024-03-31T15:06:27.285655+0000","last_became_active":"2024-03-31T15:06:26.284449+0000","last_became_peered":"2024-03-31T15:06:26.284449+0000","last_unstale":"2024-03-31T15:06:27.285655+0000","last_undegraded":"2024-03-31T15:06:27.285655+0000","last_fullsized":"2024-03-31T15:06:27.285655+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_clean_scrub_stamp":"2024-03-31T15:06:25.246926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:16:24.241917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.21","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-31T15:06:28.605399+0000","last_change":"2024-03-31T15:06:26.287703+0000","last_active":"2024-03-31T15:06:28.605399+0000","last_peered":"2024-03-31T15:06:28.605399+0000","last_clean":"2024-03-31T15:06:28.605399+0000","last_became_active":"2024-03-31T15:06:26.287413+0000","last_became_peered":"2024-03-31T15:06:26.287413+0000","last_unstale":"2024-03-31T15:06:28.605399+0000","last_undegraded":"2024-03-31T15:06:28.605399+0000","last_fullsized":"2024-03-31T15:06:28.605399+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_clean_scrub_stamp":"2024-03-31T15:06:25.246926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:28:34.107112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.20","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-31T15:06:27.347113+0000","last_change":"2024-03-31T15:06:26.303201+0000","last_active":"2024-03-31T15:06:27.347113+0000","last_peered":"2024-03-31T15:06:27.347113+0000","last_clean":"2024-03-31T15:06:27.347113+0000","last_became_active":"2024-03-31T15:06:26.302949+0000","last_became_peered":"2024-03-31T15:06:26.302949+0000","last_unstale":"2024-03-31T15:06:27.347113+0000","last_undegraded":"2024-03-31T15:06:27.347113+0000","last_fullsized":"2024-03-31T15:06:27.347113+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_clean_scrub_stamp":"2024-03-31T15:06:25.246926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T15:38:58.967933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1e","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-31T15:06:27.347530+0000","last_change":"2024-03-31T15:06:26.301437+0000","last_active":"2024-03-31T15:06:27.347530+0000","last_peered":"2024-03-31T15:06:27.347530+0000","last_clean":"2024-03-31T15:06:27.347530+0000","last_became_active":"2024-03-31T15:06:26.299941+0000","last_became_peered":"2024-03-31T15:06:26.299941+0000","last_unstale":"2024-03-31T15:06:27.347530+0000","last_undegraded":"2024-03-31T15:06:27.347530+0000","last_fullsized":"2024-03-31T15:06:27.347530+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_clean_scrub_stamp":"2024-03-31T15:06:25.246926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:56:13.492863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1f","version":"55'4","reported_seq":18,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-31T15:06:27.347398+0000","last_change":"2024-03-31T15:06:25.270858+0000","last_active":"2024-03-31T15:06:27.347398+0000","last_peered":"2024-03-31T15:06:27.347398+0000","last_clean":"2024-03-31T15:06:27.347398+0000","last_became_active":"2024-03-31T15:06:25.270569+0000","last_became_peered":"2024-03-31T15:06:25.270569+0000","last_unstale":"2024-03-31T15:06:27.347398+0000","last_undegraded":"2024-03-31T15:06:27.347398+0000","last_fullsized":"2024-03-31T15:06:27.347398+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:06:24.244870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:06:24.244870+0000","last_clean_scrub_stamp":"2024-03-31T15:06:24.244870+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T15:22:02.328114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1f","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:06:26.295037+0000","last_change":"2024-03-31T15:06:26.288476+0000","last_active":"2024-03-31T15:06:26.295037+0000","last_peered":"2024-03-31T15:06:26.295037+0000","last_clean":"2024-03-31T15:06:26.295037+0000","last_became_active":"2024-03-31T15:06:26.288157+0000","last_became_peered":"2024-03-31T15:06:26.288157+0000","last_unstale":"2024-03-31T15:06:26.295037+0000","last_undegraded":"2024-03-31T15:06:26.295037+0000","last_fullsized":"2024-03-31T15:06:26.295037+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_clean_scrub_stamp":"2024-03-31T15:06:25.246926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T19:34:18.973247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1e","version":"55'1","reported_seq":13,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:06:26.296315+0000","last_change":"2024-03-31T15:06:25.259815+0000","last_active":"2024-03-31T15:06:26.296315+0000","last_peered":"2024-03-31T15:06:26.296315+0000","last_clean":"2024-03-31T15:06:26.296315+0000","last_became_active":"2024-03-31T15:06:25.259531+0000","last_became_peered":"2024-03-31T15:06:25.259531+0000","last_unstale":"2024-03-31T15:06:26.296315+0000","last_undegraded":"2024-03-31T15:06:26.296315+0000","last_fullsized":"2024-03-31T15:06:26.296315+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:06:24.244870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:06:24.244870+0000","last_clean_scrub_stamp":"2024-03-31T15:06:24.244870+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T17:06:13.957394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.1c","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-31T15:06:27.347741+0000","last_change":"2024-03-31T15:06:26.308014+0000","last_active":"2024-03-31T15:06:27.347741+0000","last_peered":"2024-03-31T15:06:27.347741+0000","last_clean":"2024-03-31T15:06:27.347741+0000","last_became_active":"2024-03-31T15:06:26.307138+0000","last_became_peered":"2024-03-31T15:06:26.307138+0000","last_unstale":"2024-03-31T15:06:27.347741+0000","last_undegraded":"2024-03-31T15:06:27.347741+0000","last_fullsized":"2024-03-31T15:06:27.347741+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_clean_scrub_stamp":"2024-03-31T15:06:25.246926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:43:16.624229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1d","version":"55'2","reported_seq":16,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-31T15:06:27.347550+0000","last_change":"2024-03-31T15:06:25.276727+0000","last_active":"2024-03-31T15:06:27.347550+0000","last_peered":"2024-03-31T15:06:27.347550+0000","last_clean":"2024-03-31T15:06:27.347550+0000","last_became_active":"2024-03-31T15:06:25.276430+0000","last_became_peered":"2024-03-31T15:06:25.276430+0000","last_unstale":"2024-03-31T15:06:27.347550+0000","last_undegraded":"2024-03-31T15:06:27.347550+0000","last_fullsized":"2024-03-31T15:06:27.347550+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:06:24.244870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:06:24.244870+0000","last_clean_scrub_stamp":"2024-03-31T15:06:24.244870+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:15:43.405488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1d","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-31T15:06:27.347431+0000","last_change":"2024-03-31T15:06:26.300634+0000","last_active":"2024-03-31T15:06:27.347431+0000","last_peered":"2024-03-31T15:06:27.347431+0000","last_clean":"2024-03-31T15:06:27.347431+0000","last_became_active":"2024-03-31T15:06:26.300114+0000","last_became_peered":"2024-03-31T15:06:26.300114+0000","last_unstale":"2024-03-31T15:06:27.347431+0000","last_undegraded":"2024-03-31T15:06:27.347431+0000","last_fullsized":"2024-03-31T15:06:27.347431+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_clean_scrub_stamp":"2024-03-31T15:06:25.246926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:46:19.150623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1c","version":"55'2","reported_seq":16,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-31T15:06:27.284155+0000","last_change":"2024-03-31T15:06:25.273198+0000","last_active":"2024-03-31T15:06:27.284155+0000","last_peered":"2024-03-31T15:06:27.284155+0000","last_clean":"2024-03-31T15:06:27.284155+0000","last_became_active":"2024-03-31T15:06:25.272869+0000","last_became_peered":"2024-03-31T15:06:25.272869+0000","last_unstale":"2024-03-31T15:06:27.284155+0000","last_undegraded":"2024-03-31T15:06:27.284155+0000","last_fullsized":"2024-03-31T15:06:27.284155+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:06:24.244870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:06:24.244870+0000","last_clean_scrub_stamp":"2024-03-31T15:06:24.244870+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:50:32.743807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.1a","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:06:26.300477+0000","last_change":"2024-03-31T15:06:26.296178+0000","last_active":"2024-03-31T15:06:26.300477+0000","last_peered":"2024-03-31T15:06:26.300477+0000","last_clean":"2024-03-31T15:06:26.300477+0000","last_became_active":"2024-03-31T15:06:26.295534+0000","last_became_peered":"2024-03-31T15:06:26.295534+0000","last_unstale":"2024-03-31T15:06:26.300477+0000","last_undegraded":"2024-03-31T15:06:26.300477+0000","last_fullsized":"2024-03-31T15:06:26.300477+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_clean_scrub_stamp":"2024-03-31T15:06:25.246926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T17:46:44.001115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1b","version":"55'1","reported_seq":15,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-31T15:06:27.505187+0000","last_change":"2024-03-31T15:06:25.279029+0000","last_active":"2024-03-31T15:06:27.505187+0000","last_peered":"2024-03-31T15:06:27.505187+0000","last_clean":"2024-03-31T15:06:27.505187+0000","last_became_active":"2024-03-31T15:06:25.278594+0000","last_became_peered":"2024-03-31T15:06:25.278594+0000","last_unstale":"2024-03-31T15:06:27.505187+0000","last_undegraded":"2024-03-31T15:06:27.505187+0000","last_fullsized":"2024-03-31T15:06:27.505187+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:06:24.244870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:06:24.244870+0000","last_clean_scrub_stamp":"2024-03-31T15:06:24.244870+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T16:37:50.440602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1b","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:06:26.305001+0000","last_change":"2024-03-31T15:06:26.303096+0000","last_active":"2024-03-31T15:06:26.305001+0000","last_peered":"2024-03-31T15:06:26.305001+0000","last_clean":"2024-03-31T15:06:26.305001+0000","last_became_active":"2024-03-31T15:06:26.302647+0000","last_became_peered":"2024-03-31T15:06:26.302647+0000","last_unstale":"2024-03-31T15:06:26.305001+0000","last_undegraded":"2024-03-31T15:06:26.305001+0000","last_fullsized":"2024-03-31T15:06:26.305001+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_clean_scrub_stamp":"2024-03-31T15:06:25.246926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:53:42.345165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1a","version":"55'1","reported_seq":15,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-31T15:06:27.347230+0000","last_change":"2024-03-31T15:06:25.274617+0000","last_active":"2024-03-31T15:06:27.347230+0000","last_peered":"2024-03-31T15:06:27.347230+0000","last_clean":"2024-03-31T15:06:27.347230+0000","last_became_active":"2024-03-31T15:06:25.274071+0000","last_became_peered":"2024-03-31T15:06:25.274071+0000","last_unstale":"2024-03-31T15:06:27.347230+0000","last_undegraded":"2024-03-31T15:06:27.347230+0000","last_fullsized":"2024-03-31T15:06:27.347230+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:06:24.244870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:06:24.244870+0000","last_clean_scrub_stamp":"2024-03-31T15:06:24.244870+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:23:48.591532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-31T15:06:27.286419+0000","last_change":"2024-03-31T15:06:26.278004+0000","last_active":"2024-03-31T15:06:27.286419+0000","last_peered":"2024-03-31T15:06:27.286419+0000","last_clean":"2024-03-31T15:06:27.286419+0000","last_became_active":"2024-03-31T15:06:26.277630+0000","last_became_peered":"2024-03-31T15:06:26.277630+0000","last_unstale":"2024-03-31T15:06:27.286419+0000","last_undegraded":"2024-03-31T15:06:27.286419+0000","last_fullsized":"2024-03-31T15:06:27.286419+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_clean_scrub_stamp":"2024-03-31T15:06:25.246926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:59:09.402257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":14,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-31T15:06:27.283834+0000","last_change":"2024-03-31T15:06:25.274081+0000","last_active":"2024-03-31T15:06:27.283834+0000","last_peered":"2024-03-31T15:06:27.283834+0000","last_clean":"2024-03-31T15:06:27.283834+0000","last_became_active":"2024-03-31T15:06:25.273757+0000","last_became_peered":"2024-03-31T15:06:25.273757+0000","last_unstale":"2024-03-31T15:06:27.283834+0000","last_undegraded":"2024-03-31T15:06:27.283834+0000","last_fullsized":"2024-03-31T15:06:27.283834+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:06:24.244870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:06:24.244870+0000","last_clean_scrub_stamp":"2024-03-31T15:06:24.244870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:27:49.498401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:06:26.306949+0000","last_change":"2024-03-31T15:06:26.305278+0000","last_active":"2024-03-31T15:06:26.306949+0000","last_peered":"2024-03-31T15:06:26.306949+0000","last_clean":"2024-03-31T15:06:26.306949+0000","last_became_active":"2024-03-31T15:06:26.301678+0000","last_became_peered":"2024-03-31T15:06:26.301678+0000","last_unstale":"2024-03-31T15:06:26.306949+0000","last_undegraded":"2024-03-31T15:06:26.306949+0000","last_fullsized":"2024-03-31T15:06:26.306949+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_clean_scrub_stamp":"2024-03-31T15:06:25.246926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:04:57.292580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":14,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-31T15:06:27.505014+0000","last_change":"2024-03-31T15:06:25.277728+0000","last_active":"2024-03-31T15:06:27.505014+0000","last_peered":"2024-03-31T15:06:27.505014+0000","last_clean":"2024-03-31T15:06:27.505014+0000","last_became_active":"2024-03-31T15:06:25.277290+0000","last_became_peered":"2024-03-31T15:06:25.277290+0000","last_unstale":"2024-03-31T15:06:27.505014+0000","last_undegraded":"2024-03-31T15:06:27.505014+0000","last_fullsized":"2024-03-31T15:06:27.505014+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:06:24.244870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:06:24.244870+0000","last_clean_scrub_stamp":"2024-03-31T15:06:24.244870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:20:35.570759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-31T15:06:27.505338+0000","last_change":"2024-03-31T15:06:26.294062+0000","last_active":"2024-03-31T15:06:27.505338+0000","last_peered":"2024-03-31T15:06:27.505338+0000","last_clean":"2024-03-31T15:06:27.505338+0000","last_became_active":"2024-03-31T15:06:26.293783+0000","last_became_peered":"2024-03-31T15:06:26.293783+0000","last_unstale":"2024-03-31T15:06:27.505338+0000","last_undegraded":"2024-03-31T15:06:27.505338+0000","last_fullsized":"2024-03-31T15:06:27.505338+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_clean_scrub_stamp":"2024-03-31T15:06:25.246926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:27:50.500458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":14,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-31T15:06:27.286242+0000","last_change":"2024-03-31T15:06:25.268373+0000","last_active":"2024-03-31T15:06:27.286242+0000","last_peered":"2024-03-31T15:06:27.286242+0000","last_clean":"2024-03-31T15:06:27.286242+0000","last_became_active":"2024-03-31T15:06:25.267981+0000","last_became_peered":"2024-03-31T15:06:25.267981+0000","last_unstale":"2024-03-31T15:06:27.286242+0000","last_undegraded":"2024-03-31T15:06:27.286242+0000","last_fullsized":"2024-03-31T15:06:27.286242+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:06:24.244870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:06:24.244870+0000","last_clean_scrub_stamp":"2024-03-31T15:06:24.244870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:32:09.982781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-31T15:06:27.285891+0000","last_change":"2024-03-31T15:06:26.281908+0000","last_active":"2024-03-31T15:06:27.285891+0000","last_peered":"2024-03-31T15:06:27.285891+0000","last_clean":"2024-03-31T15:06:27.285891+0000","last_became_active":"2024-03-31T15:06:26.281645+0000","last_became_peered":"2024-03-31T15:06:26.281645+0000","last_unstale":"2024-03-31T15:06:27.285891+0000","last_undegraded":"2024-03-31T15:06:27.285891+0000","last_fullsized":"2024-03-31T15:06:27.285891+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_clean_scrub_stamp":"2024-03-31T15:06:25.246926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:52:59.815514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:06:26.274393+0000","last_change":"2024-03-31T15:06:25.278704+0000","last_active":"2024-03-31T15:06:26.274393+0000","last_peered":"2024-03-31T15:06:26.274393+0000","last_clean":"2024-03-31T15:06:26.274393+0000","last_became_active":"2024-03-31T15:06:25.278406+0000","last_became_peered":"2024-03-31T15:06:25.278406+0000","last_unstale":"2024-03-31T15:06:26.274393+0000","last_undegraded":"2024-03-31T15:06:26.274393+0000","last_fullsized":"2024-03-31T15:06:26.274393+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:06:24.244870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:06:24.244870+0000","last_clean_scrub_stamp":"2024-03-31T15:06:24.244870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T16:27:52.154972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.14","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:06:26.294558+0000","last_change":"2024-03-31T15:06:26.289438+0000","last_active":"2024-03-31T15:06:26.294558+0000","last_peered":"2024-03-31T15:06:26.294558+0000","last_clean":"2024-03-31T15:06:26.294558+0000","last_became_active":"2024-03-31T15:06:26.289182+0000","last_became_peered":"2024-03-31T15:06:26.289182+0000","last_unstale":"2024-03-31T15:06:26.294558+0000","last_undegraded":"2024-03-31T15:06:26.294558+0000","last_fullsized":"2024-03-31T15:06:26.294558+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_clean_scrub_stamp":"2024-03-31T15:06:25.246926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:37:10.307112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.15","version":"55'3","reported_seq":17,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-31T15:06:27.286481+0000","last_change":"2024-03-31T15:06:25.277683+0000","last_active":"2024-03-31T15:06:27.286481+0000","last_peered":"2024-03-31T15:06:27.286481+0000","last_clean":"2024-03-31T15:06:27.286481+0000","last_became_active":"2024-03-31T15:06:25.277342+0000","last_became_peered":"2024-03-31T15:06:25.277342+0000","last_unstale":"2024-03-31T15:06:27.286481+0000","last_undegraded":"2024-03-31T15:06:27.286481+0000","last_fullsized":"2024-03-31T15:06:27.286481+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:06:24.244870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:06:24.244870+0000","last_clean_scrub_stamp":"2024-03-31T15:06:24.244870+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:05:29.822543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.15","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-31T15:06:27.286221+0000","last_change":"2024-03-31T15:06:26.285235+0000","last_active":"2024-03-31T15:06:27.286221+0000","last_peered":"2024-03-31T15:06:27.286221+0000","last_clean":"2024-03-31T15:06:27.286221+0000","last_became_active":"2024-03-31T15:06:26.283775+0000","last_became_peered":"2024-03-31T15:06:26.283775+0000","last_unstale":"2024-03-31T15:06:27.286221+0000","last_undegraded":"2024-03-31T15:06:27.286221+0000","last_fullsized":"2024-03-31T15:06:27.286221+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_clean_scrub_stamp":"2024-03-31T15:06:25.246926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:46:47.862478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.14","version":"55'6","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-31T15:06:27.505169+0000","last_change":"2024-03-31T15:06:25.276935+0000","last_active":"2024-03-31T15:06:27.505169+0000","last_peered":"2024-03-31T15:06:27.505169+0000","last_clean":"2024-03-31T15:06:27.505169+0000","last_became_active":"2024-03-31T15:06:25.276499+0000","last_became_peered":"2024-03-31T15:06:25.276499+0000","last_unstale":"2024-03-31T15:06:27.505169+0000","last_undegraded":"2024-03-31T15:06:27.505169+0000","last_fullsized":"2024-03-31T15:06:27.505169+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:06:24.244870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:06:24.244870+0000","last_clean_scrub_stamp":"2024-03-31T15:06:24.244870+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T16:42:59.818466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-31T15:06:27.284530+0000","last_change":"2024-03-31T15:06:26.282459+0000","last_active":"2024-03-31T15:06:27.284530+0000","last_peered":"2024-03-31T15:06:27.284530+0000","last_clean":"2024-03-31T15:06:27.284530+0000","last_became_active":"2024-03-31T15:06:26.281971+0000","last_became_peered":"2024-03-31T15:06:26.281971+0000","last_unstale":"2024-03-31T15:06:27.284530+0000","last_undegraded":"2024-03-31T15:06:27.284530+0000","last_fullsized":"2024-03-31T15:06:27.284530+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_clean_scrub_stamp":"2024-03-31T15:06:25.246926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:52:59.815514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.13","version":"55'2","reported_seq":14,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:06:26.303833+0000","last_change":"2024-03-31T15:06:25.278639+0000","last_active":"2024-03-31T15:06:26.303833+0000","last_peered":"2024-03-31T15:06:26.303833+0000","last_clean":"2024-03-31T15:06:26.303833+0000","last_became_active":"2024-03-31T15:06:25.278335+0000","last_became_peered":"2024-03-31T15:06:25.278335+0000","last_unstale":"2024-03-31T15:06:26.303833+0000","last_undegraded":"2024-03-31T15:06:26.303833+0000","last_fullsized":"2024-03-31T15:06:26.303833+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:06:24.244870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:06:24.244870+0000","last_clean_scrub_stamp":"2024-03-31T15:06:24.244870+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:56:12.490806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.13","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-31T15:06:27.285790+0000","last_change":"2024-03-31T15:06:26.277529+0000","last_active":"2024-03-31T15:06:27.285790+0000","last_peered":"2024-03-31T15:06:27.285790+0000","last_clean":"2024-03-31T15:06:27.285790+0000","last_became_active":"2024-03-31T15:06:26.277199+0000","last_became_peered":"2024-03-31T15:06:26.277199+0000","last_unstale":"2024-03-31T15:06:27.285790+0000","last_undegraded":"2024-03-31T15:06:27.285790+0000","last_fullsized":"2024-03-31T15:06:27.285790+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_clean_scrub_stamp":"2024-03-31T15:06:25.246926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:23:45.049194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":14,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-31T15:06:27.286832+0000","last_change":"2024-03-31T15:06:25.267602+0000","last_active":"2024-03-31T15:06:27.286832+0000","last_peered":"2024-03-31T15:06:27.286832+0000","last_clean":"2024-03-31T15:06:27.286832+0000","last_became_active":"2024-03-31T15:06:25.267283+0000","last_became_peered":"2024-03-31T15:06:25.267283+0000","last_unstale":"2024-03-31T15:06:27.286832+0000","last_undegraded":"2024-03-31T15:06:27.286832+0000","last_fullsized":"2024-03-31T15:06:27.286832+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:06:24.244870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:06:24.244870+0000","last_clean_scrub_stamp":"2024-03-31T15:06:24.244870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T19:34:17.971191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.10","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-31T15:06:27.505164+0000","last_change":"2024-03-31T15:06:26.292381+0000","last_active":"2024-03-31T15:06:27.505164+0000","last_peered":"2024-03-31T15:06:27.505164+0000","last_clean":"2024-03-31T15:06:27.505164+0000","last_became_active":"2024-03-31T15:06:26.290816+0000","last_became_peered":"2024-03-31T15:06:26.290816+0000","last_unstale":"2024-03-31T15:06:27.505164+0000","last_undegraded":"2024-03-31T15:06:27.505164+0000","last_fullsized":"2024-03-31T15:06:27.505164+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_clean_scrub_stamp":"2024-03-31T15:06:25.246926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T16:28:36.415394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":14,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-31T15:06:27.284513+0000","last_change":"2024-03-31T15:06:25.271605+0000","last_active":"2024-03-31T15:06:27.284513+0000","last_peered":"2024-03-31T15:06:27.284513+0000","last_clean":"2024-03-31T15:06:27.284513+0000","last_became_active":"2024-03-31T15:06:25.271317+0000","last_became_peered":"2024-03-31T15:06:25.271317+0000","last_unstale":"2024-03-31T15:06:27.284513+0000","last_undegraded":"2024-03-31T15:06:27.284513+0000","last_fullsized":"2024-03-31T15:06:27.284513+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:06:24.244870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:06:24.244870+0000","last_clean_scrub_stamp":"2024-03-31T15:06:24.244870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:20:38.020781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-31T15:06:27.285391+0000","last_change":"2024-03-31T15:06:26.286817+0000","last_active":"2024-03-31T15:06:27.285391+0000","last_peered":"2024-03-31T15:06:27.285391+0000","last_clean":"2024-03-31T15:06:27.285391+0000","last_became_active":"2024-03-31T15:06:26.286533+0000","last_became_peered":"2024-03-31T15:06:26.286533+0000","last_unstale":"2024-03-31T15:06:27.285391+0000","last_undegraded":"2024-03-31T15:06:27.285391+0000","last_fullsized":"2024-03-31T15:06:27.285391+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_clean_scrub_stamp":"2024-03-31T15:06:25.246926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:53:46.360339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.10","version":"55'8","reported_seq":22,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-31T15:06:27.285182+0000","last_change":"2024-03-31T15:06:25.269450+0000","last_active":"2024-03-31T15:06:27.285182+0000","last_peered":"2024-03-31T15:06:27.285182+0000","last_clean":"2024-03-31T15:06:27.285182+0000","last_became_active":"2024-03-31T15:06:25.269170+0000","last_became_peered":"2024-03-31T15:06:25.269170+0000","last_unstale":"2024-03-31T15:06:27.285182+0000","last_undegraded":"2024-03-31T15:06:27.285182+0000","last_fullsized":"2024-03-31T15:06:27.285182+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:06:24.244870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:06:24.244870+0000","last_clean_scrub_stamp":"2024-03-31T15:06:24.244870+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:50:32.743807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.e","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-31T15:06:27.286259+0000","last_change":"2024-03-31T15:06:26.286921+0000","last_active":"2024-03-31T15:06:27.286259+0000","last_peered":"2024-03-31T15:06:27.286259+0000","last_clean":"2024-03-31T15:06:27.286259+0000","last_became_active":"2024-03-31T15:06:26.284462+0000","last_became_peered":"2024-03-31T15:06:26.284462+0000","last_unstale":"2024-03-31T15:06:27.286259+0000","last_undegraded":"2024-03-31T15:06:27.286259+0000","last_fullsized":"2024-03-31T15:06:27.286259+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_clean_scrub_stamp":"2024-03-31T15:06:25.246926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T16:41:32.500664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.f","version":"55'1","reported_seq":13,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:06:26.299395+0000","last_change":"2024-03-31T15:06:25.279268+0000","last_active":"2024-03-31T15:06:26.299395+0000","last_peered":"2024-03-31T15:06:26.299395+0000","last_clean":"2024-03-31T15:06:26.299395+0000","last_became_active":"2024-03-31T15:06:25.277056+0000","last_became_peered":"2024-03-31T15:06:25.277056+0000","last_unstale":"2024-03-31T15:06:26.299395+0000","last_undegraded":"2024-03-31T15:06:26.299395+0000","last_fullsized":"2024-03-31T15:06:26.299395+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:06:24.244870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:06:24.244870+0000","last_clean_scrub_stamp":"2024-03-31T15:06:24.244870+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:43:15.622172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.f","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-31T15:06:27.284387+0000","last_change":"2024-03-31T15:06:26.279710+0000","last_active":"2024-03-31T15:06:27.284387+0000","last_peered":"2024-03-31T15:06:27.284387+0000","last_clean":"2024-03-31T15:06:27.284387+0000","last_became_active":"2024-03-31T15:06:26.279379+0000","last_became_peered":"2024-03-31T15:06:26.279379+0000","last_unstale":"2024-03-31T15:06:27.284387+0000","last_undegraded":"2024-03-31T15:06:27.284387+0000","last_fullsized":"2024-03-31T15:06:27.284387+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_clean_scrub_stamp":"2024-03-31T15:06:25.246926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:10:50.564026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.e","version":"55'2","reported_seq":16,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-31T15:06:27.347624+0000","last_change":"2024-03-31T15:06:25.280643+0000","last_active":"2024-03-31T15:06:27.347624+0000","last_peered":"2024-03-31T15:06:27.347624+0000","last_clean":"2024-03-31T15:06:27.347624+0000","last_became_active":"2024-03-31T15:06:25.280285+0000","last_became_peered":"2024-03-31T15:06:25.280285+0000","last_unstale":"2024-03-31T15:06:27.347624+0000","last_undegraded":"2024-03-31T15:06:27.347624+0000","last_fullsized":"2024-03-31T15:06:27.347624+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:06:24.244870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:06:24.244870+0000","last_clean_scrub_stamp":"2024-03-31T15:06:24.244870+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:25:14.095342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:06:26.300497+0000","last_change":"2024-03-31T15:06:26.291977+0000","last_active":"2024-03-31T15:06:26.300497+0000","last_peered":"2024-03-31T15:06:26.300497+0000","last_clean":"2024-03-31T15:06:26.300497+0000","last_became_active":"2024-03-31T15:06:26.291666+0000","last_became_peered":"2024-03-31T15:06:26.291666+0000","last_unstale":"2024-03-31T15:06:26.300497+0000","last_undegraded":"2024-03-31T15:06:26.300497+0000","last_fullsized":"2024-03-31T15:06:26.300497+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_clean_scrub_stamp":"2024-03-31T15:06:25.246926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T19:35:04.972072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"55'1","reported_seq":15,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-31T15:06:27.284064+0000","last_change":"2024-03-31T15:06:25.269900+0000","last_active":"2024-03-31T15:06:27.284064+0000","last_peered":"2024-03-31T15:06:27.284064+0000","last_clean":"2024-03-31T15:06:27.284064+0000","last_became_active":"2024-03-31T15:06:25.269638+0000","last_became_peered":"2024-03-31T15:06:25.269638+0000","last_unstale":"2024-03-31T15:06:27.284064+0000","last_undegraded":"2024-03-31T15:06:27.284064+0000","last_fullsized":"2024-03-31T15:06:27.284064+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:06:24.244870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:06:24.244870+0000","last_clean_scrub_stamp":"2024-03-31T15:06:24.244870+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:05:29.822543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.d","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-31T15:06:27.286366+0000","last_change":"2024-03-31T15:06:26.281237+0000","last_active":"2024-03-31T15:06:27.286366+0000","last_peered":"2024-03-31T15:06:27.286366+0000","last_clean":"2024-03-31T15:06:27.286366+0000","last_became_active":"2024-03-31T15:06:26.280933+0000","last_became_peered":"2024-03-31T15:06:26.280933+0000","last_unstale":"2024-03-31T15:06:27.286366+0000","last_undegraded":"2024-03-31T15:06:27.286366+0000","last_fullsized":"2024-03-31T15:06:27.286366+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_clean_scrub_stamp":"2024-03-31T15:06:25.246926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:52:49.002409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":14,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-31T15:06:27.347749+0000","last_change":"2024-03-31T15:06:25.273987+0000","last_active":"2024-03-31T15:06:27.347749+0000","last_peered":"2024-03-31T15:06:27.347749+0000","last_clean":"2024-03-31T15:06:27.347749+0000","last_became_active":"2024-03-31T15:06:25.273413+0000","last_became_peered":"2024-03-31T15:06:25.273413+0000","last_unstale":"2024-03-31T15:06:27.347749+0000","last_undegraded":"2024-03-31T15:06:27.347749+0000","last_fullsized":"2024-03-31T15:06:27.347749+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:06:24.244870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:06:24.244870+0000","last_clean_scrub_stamp":"2024-03-31T15:06:24.244870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T16:02:01.742318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-31T15:06:27.284681+0000","last_change":"2024-03-31T15:06:26.282219+0000","last_active":"2024-03-31T15:06:27.284681+0000","last_peered":"2024-03-31T15:06:27.284681+0000","last_clean":"2024-03-31T15:06:27.284681+0000","last_became_active":"2024-03-31T15:06:26.281600+0000","last_became_peered":"2024-03-31T15:06:26.281600+0000","last_unstale":"2024-03-31T15:06:27.284681+0000","last_undegraded":"2024-03-31T15:06:27.284681+0000","last_fullsized":"2024-03-31T15:06:27.284681+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_clean_scrub_stamp":"2024-03-31T15:06:25.246926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T15:42:16.759739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.b","version":"55'2","reported_seq":14,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:06:26.298941+0000","last_change":"2024-03-31T15:06:25.271864+0000","last_active":"2024-03-31T15:06:26.298941+0000","last_peered":"2024-03-31T15:06:26.298941+0000","last_clean":"2024-03-31T15:06:26.298941+0000","last_became_active":"2024-03-31T15:06:25.271554+0000","last_became_peered":"2024-03-31T15:06:25.271554+0000","last_unstale":"2024-03-31T15:06:26.298941+0000","last_undegraded":"2024-03-31T15:06:26.298941+0000","last_fullsized":"2024-03-31T15:06:26.298941+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:06:24.244870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:06:24.244870+0000","last_clean_scrub_stamp":"2024-03-31T15:06:24.244870+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:24:14.868237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.b","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-31T15:06:27.347288+0000","last_change":"2024-03-31T15:06:26.297362+0000","last_active":"2024-03-31T15:06:27.347288+0000","last_peered":"2024-03-31T15:06:27.347288+0000","last_clean":"2024-03-31T15:06:27.347288+0000","last_became_active":"2024-03-31T15:06:26.297013+0000","last_became_peered":"2024-03-31T15:06:26.297013+0000","last_unstale":"2024-03-31T15:06:27.347288+0000","last_undegraded":"2024-03-31T15:06:27.347288+0000","last_fullsized":"2024-03-31T15:06:27.347288+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_clean_scrub_stamp":"2024-03-31T15:06:25.246926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:57:13.747232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:06:26.272769+0000","last_change":"2024-03-31T15:06:25.277123+0000","last_active":"2024-03-31T15:06:26.272769+0000","last_peered":"2024-03-31T15:06:26.272769+0000","last_clean":"2024-03-31T15:06:26.272769+0000","last_became_active":"2024-03-31T15:06:25.276798+0000","last_became_peered":"2024-03-31T15:06:25.276798+0000","last_unstale":"2024-03-31T15:06:26.272769+0000","last_undegraded":"2024-03-31T15:06:26.272769+0000","last_fullsized":"2024-03-31T15:06:26.272769+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:06:24.244870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:06:24.244870+0000","last_clean_scrub_stamp":"2024-03-31T15:06:24.244870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:13:23.354596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-31T15:06:27.347116+0000","last_change":"2024-03-31T15:06:26.292230+0000","last_active":"2024-03-31T15:06:27.347116+0000","last_peered":"2024-03-31T15:06:27.347116+0000","last_clean":"2024-03-31T15:06:27.347116+0000","last_became_active":"2024-03-31T15:06:26.291968+0000","last_became_peered":"2024-03-31T15:06:26.291968+0000","last_unstale":"2024-03-31T15:06:27.347116+0000","last_undegraded":"2024-03-31T15:06:27.347116+0000","last_fullsized":"2024-03-31T15:06:27.347116+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_clean_scrub_stamp":"2024-03-31T15:06:25.246926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T16:29:20.020911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":14,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-31T15:06:27.283906+0000","last_change":"2024-03-31T15:06:25.277911+0000","last_active":"2024-03-31T15:06:27.283906+0000","last_peered":"2024-03-31T15:06:27.283906+0000","last_clean":"2024-03-31T15:06:27.283906+0000","last_became_active":"2024-03-31T15:06:25.277608+0000","last_became_peered":"2024-03-31T15:06:25.277608+0000","last_unstale":"2024-03-31T15:06:27.283906+0000","last_undegraded":"2024-03-31T15:06:27.283906+0000","last_fullsized":"2024-03-31T15:06:27.283906+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:06:24.244870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:06:24.244870+0000","last_clean_scrub_stamp":"2024-03-31T15:06:24.244870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T16:28:35.413337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-31T15:06:27.284238+0000","last_change":"2024-03-31T15:06:26.296321+0000","last_active":"2024-03-31T15:06:27.284238+0000","last_peered":"2024-03-31T15:06:27.284238+0000","last_clean":"2024-03-31T15:06:27.284238+0000","last_became_active":"2024-03-31T15:06:26.294992+0000","last_became_peered":"2024-03-31T15:06:26.294992+0000","last_unstale":"2024-03-31T15:06:27.284238+0000","last_undegraded":"2024-03-31T15:06:27.284238+0000","last_fullsized":"2024-03-31T15:06:27.284238+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_clean_scrub_stamp":"2024-03-31T15:06:25.246926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:16:24.241917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.7","version":"55'2","reported_seq":14,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:06:26.305707+0000","last_change":"2024-03-31T15:06:25.279243+0000","last_active":"2024-03-31T15:06:26.305707+0000","last_peered":"2024-03-31T15:06:26.305707+0000","last_clean":"2024-03-31T15:06:26.305707+0000","last_became_active":"2024-03-31T15:06:25.278951+0000","last_became_peered":"2024-03-31T15:06:25.278951+0000","last_unstale":"2024-03-31T15:06:26.305707+0000","last_undegraded":"2024-03-31T15:06:26.305707+0000","last_fullsized":"2024-03-31T15:06:26.305707+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:06:24.244870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:06:24.244870+0000","last_clean_scrub_stamp":"2024-03-31T15:06:24.244870+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:26:00.373303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-31T15:06:27.284434+0000","last_change":"2024-03-31T15:06:26.284804+0000","last_active":"2024-03-31T15:06:27.284434+0000","last_peered":"2024-03-31T15:06:27.284434+0000","last_clean":"2024-03-31T15:06:27.284434+0000","last_became_active":"2024-03-31T15:06:26.284537+0000","last_became_peered":"2024-03-31T15:06:26.284537+0000","last_unstale":"2024-03-31T15:06:27.284434+0000","last_undegraded":"2024-03-31T15:06:27.284434+0000","last_fullsized":"2024-03-31T15:06:27.284434+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_clean_scrub_stamp":"2024-03-31T15:06:25.246926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:23:45.049194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":14,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-31T15:06:27.347708+0000","last_change":"2024-03-31T15:06:25.274928+0000","last_active":"2024-03-31T15:06:27.347708+0000","last_peered":"2024-03-31T15:06:27.347708+0000","last_clean":"2024-03-31T15:06:27.347708+0000","last_became_active":"2024-03-31T15:06:25.274355+0000","last_became_peered":"2024-03-31T15:06:25.274355+0000","last_unstale":"2024-03-31T15:06:27.347708+0000","last_undegraded":"2024-03-31T15:06:27.347708+0000","last_fullsized":"2024-03-31T15:06:27.347708+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:06:24.244870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:06:24.244870+0000","last_clean_scrub_stamp":"2024-03-31T15:06:24.244870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:18:03.273932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:06:26.297337+0000","last_change":"2024-03-31T15:06:26.292751+0000","last_active":"2024-03-31T15:06:26.297337+0000","last_peered":"2024-03-31T15:06:26.297337+0000","last_clean":"2024-03-31T15:06:26.297337+0000","last_became_active":"2024-03-31T15:06:26.292454+0000","last_became_peered":"2024-03-31T15:06:26.292454+0000","last_unstale":"2024-03-31T15:06:26.297337+0000","last_undegraded":"2024-03-31T15:06:26.297337+0000","last_fullsized":"2024-03-31T15:06:26.297337+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_clean_scrub_stamp":"2024-03-31T15:06:25.246926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T16:59:43.968592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":14,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-31T15:06:27.286952+0000","last_change":"2024-03-31T15:06:25.264755+0000","last_active":"2024-03-31T15:06:27.286952+0000","last_peered":"2024-03-31T15:06:27.286952+0000","last_clean":"2024-03-31T15:06:27.286952+0000","last_became_active":"2024-03-31T15:06:25.264394+0000","last_became_peered":"2024-03-31T15:06:25.264394+0000","last_unstale":"2024-03-31T15:06:27.286952+0000","last_undegraded":"2024-03-31T15:06:27.286952+0000","last_fullsized":"2024-03-31T15:06:27.286952+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:06:24.244870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:06:24.244870+0000","last_clean_scrub_stamp":"2024-03-31T15:06:24.244870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:27:44.434513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-31T15:06:27.347507+0000","last_change":"2024-03-31T15:06:26.300390+0000","last_active":"2024-03-31T15:06:27.347507+0000","last_peered":"2024-03-31T15:06:27.347507+0000","last_clean":"2024-03-31T15:06:27.347507+0000","last_became_active":"2024-03-31T15:06:26.299754+0000","last_became_peered":"2024-03-31T15:06:26.299754+0000","last_unstale":"2024-03-31T15:06:27.347507+0000","last_undegraded":"2024-03-31T15:06:27.347507+0000","last_fullsized":"2024-03-31T15:06:27.347507+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_clean_scrub_stamp":"2024-03-31T15:06:25.246926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:31:27.245880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":14,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-31T15:06:27.284263+0000","last_change":"2024-03-31T15:06:25.274117+0000","last_active":"2024-03-31T15:06:27.284263+0000","last_peered":"2024-03-31T15:06:27.284263+0000","last_clean":"2024-03-31T15:06:27.284263+0000","last_became_active":"2024-03-31T15:06:25.273837+0000","last_became_peered":"2024-03-31T15:06:25.273837+0000","last_unstale":"2024-03-31T15:06:27.284263+0000","last_undegraded":"2024-03-31T15:06:27.284263+0000","last_fullsized":"2024-03-31T15:06:27.284263+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:06:24.244870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:06:24.244870+0000","last_clean_scrub_stamp":"2024-03-31T15:06:24.244870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:32:09.982781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:06:26.300456+0000","last_change":"2024-03-31T15:06:26.293083+0000","last_active":"2024-03-31T15:06:26.300456+0000","last_peered":"2024-03-31T15:06:26.300456+0000","last_clean":"2024-03-31T15:06:26.300456+0000","last_became_active":"2024-03-31T15:06:26.290726+0000","last_became_peered":"2024-03-31T15:06:26.290726+0000","last_unstale":"2024-03-31T15:06:26.300456+0000","last_undegraded":"2024-03-31T15:06:26.300456+0000","last_fullsized":"2024-03-31T15:06:26.300456+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_clean_scrub_stamp":"2024-03-31T15:06:25.246926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:06:19.784539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:06:26.272845+0000","last_change":"2024-03-31T15:06:25.276449+0000","last_active":"2024-03-31T15:06:26.272845+0000","last_peered":"2024-03-31T15:06:26.272845+0000","last_clean":"2024-03-31T15:06:26.272845+0000","last_became_active":"2024-03-31T15:06:25.276104+0000","last_became_peered":"2024-03-31T15:06:25.276104+0000","last_unstale":"2024-03-31T15:06:26.272845+0000","last_undegraded":"2024-03-31T15:06:26.272845+0000","last_fullsized":"2024-03-31T15:06:26.272845+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:06:24.244870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:06:24.244870+0000","last_clean_scrub_stamp":"2024-03-31T15:06:24.244870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T15:28:49.048982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-31T15:06:27.285997+0000","last_change":"2024-03-31T15:06:26.293236+0000","last_active":"2024-03-31T15:06:27.285997+0000","last_peered":"2024-03-31T15:06:27.285997+0000","last_clean":"2024-03-31T15:06:27.285997+0000","last_became_active":"2024-03-31T15:06:26.292917+0000","last_became_peered":"2024-03-31T15:06:26.292917+0000","last_unstale":"2024-03-31T15:06:27.285997+0000","last_undegraded":"2024-03-31T15:06:27.285997+0000","last_fullsized":"2024-03-31T15:06:27.285997+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_clean_scrub_stamp":"2024-03-31T15:06:25.246926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T15:42:16.759739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":14,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-31T15:06:27.505117+0000","last_change":"2024-03-31T15:06:25.276897+0000","last_active":"2024-03-31T15:06:27.505117+0000","last_peered":"2024-03-31T15:06:27.505117+0000","last_clean":"2024-03-31T15:06:27.505117+0000","last_became_active":"2024-03-31T15:06:25.276494+0000","last_became_peered":"2024-03-31T15:06:25.276494+0000","last_unstale":"2024-03-31T15:06:27.505117+0000","last_undegraded":"2024-03-31T15:06:27.505117+0000","last_fullsized":"2024-03-31T15:06:27.505117+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:06:24.244870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:06:24.244870+0000","last_clean_scrub_stamp":"2024-03-31T15:06:24.244870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:36:22.933745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:06:26.285376+0000","last_change":"2024-03-31T15:06:26.278792+0000","last_active":"2024-03-31T15:06:26.285376+0000","last_peered":"2024-03-31T15:06:26.285376+0000","last_clean":"2024-03-31T15:06:26.285376+0000","last_became_active":"2024-03-31T15:06:26.278505+0000","last_became_peered":"2024-03-31T15:06:26.278505+0000","last_unstale":"2024-03-31T15:06:26.285376+0000","last_undegraded":"2024-03-31T15:06:26.285376+0000","last_fullsized":"2024-03-31T15:06:26.285376+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_clean_scrub_stamp":"2024-03-31T15:06:25.246926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:27:45.436570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:06:26.272617+0000","last_change":"2024-03-31T15:06:25.277849+0000","last_active":"2024-03-31T15:06:26.272617+0000","last_peered":"2024-03-31T15:06:26.272617+0000","last_clean":"2024-03-31T15:06:26.272617+0000","last_became_active":"2024-03-31T15:06:25.277570+0000","last_became_peered":"2024-03-31T15:06:25.277570+0000","last_unstale":"2024-03-31T15:06:26.272617+0000","last_undegraded":"2024-03-31T15:06:26.272617+0000","last_fullsized":"2024-03-31T15:06:26.272617+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:06:24.244870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:06:24.244870+0000","last_clean_scrub_stamp":"2024-03-31T15:06:24.244870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:27:02.294805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:06:26.305453+0000","last_change":"2024-03-31T15:06:26.302007+0000","last_active":"2024-03-31T15:06:26.305453+0000","last_peered":"2024-03-31T15:06:26.305453+0000","last_clean":"2024-03-31T15:06:26.305453+0000","last_became_active":"2024-03-31T15:06:26.294389+0000","last_became_peered":"2024-03-31T15:06:26.294389+0000","last_unstale":"2024-03-31T15:06:26.305453+0000","last_undegraded":"2024-03-31T15:06:26.305453+0000","last_fullsized":"2024-03-31T15:06:26.305453+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_clean_scrub_stamp":"2024-03-31T15:06:25.246926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:59:03.602178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:06:26.269316+0000","last_change":"2024-03-31T15:06:25.274053+0000","last_active":"2024-03-31T15:06:26.269316+0000","last_peered":"2024-03-31T15:06:26.269316+0000","last_clean":"2024-03-31T15:06:26.269316+0000","last_became_active":"2024-03-31T15:06:25.273747+0000","last_became_peered":"2024-03-31T15:06:25.273747+0000","last_unstale":"2024-03-31T15:06:26.269316+0000","last_undegraded":"2024-03-31T15:06:26.269316+0000","last_fullsized":"2024-03-31T15:06:26.269316+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:06:24.244870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:06:24.244870+0000","last_clean_scrub_stamp":"2024-03-31T15:06:24.244870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:31:26.243823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:06:26.303655+0000","last_change":"2024-03-31T15:06:26.301578+0000","last_active":"2024-03-31T15:06:26.303655+0000","last_peered":"2024-03-31T15:06:26.303655+0000","last_clean":"2024-03-31T15:06:26.303655+0000","last_became_active":"2024-03-31T15:06:26.301293+0000","last_became_peered":"2024-03-31T15:06:26.301293+0000","last_unstale":"2024-03-31T15:06:26.303655+0000","last_undegraded":"2024-03-31T15:06:26.303655+0000","last_fullsized":"2024-03-31T15:06:26.303655+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:06:25.246926+0000","last_clean_scrub_stamp":"2024-03-31T15:06:25.246926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:02:22.084360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":14,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-31T15:06:27.286677+0000","last_change":"2024-03-31T15:06:25.271451+0000","last_active":"2024-03-31T15:06:27.286677+0000","last_peered":"2024-03-31T15:06:27.286677+0000","last_clean":"2024-03-31T15:06:27.286677+0000","last_became_active":"2024-03-31T15:06:25.270955+0000","last_became_peered":"2024-03-31T15:06:25.270955+0000","last_unstale":"2024-03-31T15:06:27.286677+0000","last_undegraded":"2024-03-31T15:06:27.286677+0000","last_fullsized":"2024-03-31T15:06:27.286677+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:06:24.244870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:06:24.244870+0000","last_clean_scrub_stamp":"2024-03-31T15:06:24.244870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:37:09.305055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":234,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-31T15:06:27.285247+0000","last_change":"2024-03-31T15:03:24.329890+0000","last_active":"2024-03-31T15:06:27.285247+0000","last_peered":"2024-03-31T15:06:27.285247+0000","last_clean":"2024-03-31T15:06:27.285247+0000","last_became_active":"2024-03-31T15:03:24.328882+0000","last_became_peered":"2024-03-31T15:03:24.328882+0000","last_unstale":"2024-03-31T15:06:27.285247+0000","last_undegraded":"2024-03-31T15:06:27.285247+0000","last_fullsized":"2024-03-31T15:06:27.285247+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:22.885535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:22.885535+0000","last_clean_scrub_stamp":"2024-03-31T15:03:22.885535+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:18:29.004296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":8,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509482,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32420,"kb_used_data":1236,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339420,"statfs":{"total":96636764160,"available":96603566080,"internally_reserved":0,"allocated":1265664,"data_stored":865481,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509481,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27684,"kb_used_data":656,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344156,"statfs":{"total":96636764160,"available":96608415744,"internally_reserved":0,"allocated":671744,"data_stored":275605,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":4,"up_from":15,"seq":64424509480,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27688,"kb_used_data":660,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344152,"statfs":{"total":96636764160,"available":96608411648,"internally_reserved":0,"allocated":675840,"data_stored":276565,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509479,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31828,"kb_used_data":644,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340012,"statfs":{"total":96636764160,"available":96604172288,"internally_reserved":0,"allocated":659456,"data_stored":268709,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542185,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31828,"kb_used_data":648,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340012,"statfs":{"total":96636764160,"available":96604172288,"internally_reserved":0,"allocated":663552,"data_stored":274953,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542185,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31832,"kb_used_data":644,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340008,"statfs":{"total":96636764160,"available":96604168192,"internally_reserved":0,"allocated":659456,"data_stored":268709,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542185,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31828,"kb_used_data":640,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340012,"statfs":{"total":96636764160,"available":96604172288,"internally_reserved":0,"allocated":655360,"data_stored":268743,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542185,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32412,"kb_used_data":1232,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339428,"statfs":{"total":96636764160,"available":96603574272,"internally_reserved":0,"allocated":1261568,"data_stored":861215,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-31T15:06:30.818 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-31T15:06:30.819 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-03-31T15:06:31.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:31.007+0000 7f29a5000640 1 Processor -- start 2024-03-31T15:06:31.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:31.007+0000 7f29a5000640 1 -- start start 2024-03-31T15:06:31.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:31.007+0000 7f29a5000640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f29a00d0a60 0x7f29a00cea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:31.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:31.007+0000 7f29a5000640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f29a00cef80 0x7f29a01419c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:31.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:31.007+0000 7f29a5000640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f29a015ca70 con 0x7f29a006f920 2024-03-31T15:06:31.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:31.007+0000 7f29a5000640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f29a013f670 con 0x7f29a00d0a60 2024-03-31T15:06:31.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:31.007+0000 7f29a5000640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f29a015cd50 con 0x7f29a00cef80 2024-03-31T15:06:31.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:31.007+0000 7f2995400640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f29a00d0a60 0x7f29a00cea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:31.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:31.007+0000 7f2995400640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f29a00d0a60 0x7f29a00cea40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49056/0 (socket says 172.21.15.16:49056) 2024-03-31T15:06:31.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:31.007+0000 7f2995400640 1 -- 172.21.15.16:0/2926688987 learned_addr learned my addr 172.21.15.16:0/2926688987 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:06:31.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:31.011+0000 7f299e800640 1 --2- 172.21.15.16:0/2926688987 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f29a00cef80 0x7f29a01419c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:31.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:31.011+0000 7f2995400640 1 -- 172.21.15.16:0/2926688987 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f29a00cef80 msgr2=0x7f29a01419c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:06:31.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:31.011+0000 7f2995400640 1 --2- 172.21.15.16:0/2926688987 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f29a00cef80 0x7f29a01419c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:31.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:31.011+0000 7f2995400640 1 -- 172.21.15.16:0/2926688987 >> v1:172.21.15.89:6789/0 conn(0x7f29a006f920 legacy=0x7f29a006fd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:06:31.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:31.011+0000 7f2995400640 1 -- 172.21.15.16:0/2926688987 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29a015d030 con 0x7f29a00d0a60 2024-03-31T15:06:31.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:31.011+0000 7f2995400640 1 --2- 172.21.15.16:0/2926688987 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f29a00d0a60 0x7f29a00cea40 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f2988009870 tx=0x7f298801c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ab2158ad880b837 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:06:31.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:31.011+0000 7f299d400640 1 -- 172.21.15.16:0/2926688987 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29880b0070 con 0x7f29a00d0a60 2024-03-31T15:06:31.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:31.011+0000 7f299d400640 1 -- 172.21.15.16:0/2926688987 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f29880ab600 con 0x7f29a00d0a60 2024-03-31T15:06:31.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:31.011+0000 7f299d400640 1 -- 172.21.15.16:0/2926688987 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29880ab9d0 con 0x7f29a00d0a60 2024-03-31T15:06:31.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:31.015+0000 7f29a5000640 1 -- 172.21.15.16:0/2926688987 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f29a00d0a60 msgr2=0x7f29a00cea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:31.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:31.015+0000 7f29a5000640 1 --2- 172.21.15.16:0/2926688987 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f29a00d0a60 0x7f29a00cea40 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f2988009870 tx=0x7f298801c520 comp rx=0 tx=0).stop 2024-03-31T15:06:31.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:31.015+0000 7f29a5000640 1 -- 172.21.15.16:0/2926688987 shutdown_connections 2024-03-31T15:06:31.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:31.015+0000 7f29a5000640 1 --2- 172.21.15.16:0/2926688987 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f29a00cef80 0x7f29a01419c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:31.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:31.015+0000 7f29a5000640 1 --2- 172.21.15.16:0/2926688987 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f29a00d0a60 0x7f29a00cea40 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:31.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:31.015+0000 7f29a5000640 1 -- 172.21.15.16:0/2926688987 >> 172.21.15.16:0/2926688987 conn(0x7f29a0067930 msgr2=0x7f29a00b3ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:06:31.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:31.015+0000 7f29a5000640 1 -- 172.21.15.16:0/2926688987 shutdown_connections 2024-03-31T15:06:31.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:31.015+0000 7f29a5000640 1 -- 172.21.15.16:0/2926688987 wait complete. 2024-03-31T15:06:31.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:31.015+0000 7f29a5000640 1 Processor -- start 2024-03-31T15:06:31.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:31.015+0000 7f29a5000640 1 -- start start 2024-03-31T15:06:31.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:31.015+0000 7f29a5000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f29a006f920 0x7f29a013e5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:31.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:31.015+0000 7f29a5000640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f29a00cef80 0x7f29a00b37d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:31.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:31.015+0000 7f29a5000640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f29a00d0a60 0x7f29a006d8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:31.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:31.015+0000 7f29a5000640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f29a0169230 con 0x7f29a006f920 2024-03-31T15:06:31.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:31.015+0000 7f29a5000640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f29a01690b0 con 0x7f29a00cef80 2024-03-31T15:06:31.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:31.015+0000 7f29a5000640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f29a01693b0 con 0x7f29a00d0a60 2024-03-31T15:06:31.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:31.015+0000 7f299de00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f29a006f920 0x7f29a013e5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:31.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:31.015+0000 7f2995400640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f29a00cef80 0x7f29a00b37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:31.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:31.015+0000 7f299de00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f29a006f920 0x7f29a013e5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49060/0 (socket says 172.21.15.16:49060) 2024-03-31T15:06:31.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:31.015+0000 7f299de00640 1 -- 172.21.15.16:0/1497983998 learned_addr learned my addr 172.21.15.16:0/1497983998 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:06:31.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:31.015+0000 7f299e800640 1 --2- 172.21.15.16:0/1497983998 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f29a00d0a60 0x7f29a006d8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:31.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:31.015+0000 7f299de00640 1 -- 172.21.15.16:0/1497983998 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f29a00d0a60 msgr2=0x7f29a006d8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:31.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:31.015+0000 7f299de00640 1 --2- 172.21.15.16:0/1497983998 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f29a00d0a60 0x7f29a006d8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:31.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:31.015+0000 7f299de00640 1 -- 172.21.15.16:0/1497983998 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f29a00cef80 msgr2=0x7f29a00b37d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:31.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:31.015+0000 7f299de00640 1 --2- 172.21.15.16:0/1497983998 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f29a00cef80 0x7f29a00b37d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:31.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:31.015+0000 7f299de00640 1 -- 172.21.15.16:0/1497983998 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29a013eb10 con 0x7f29a006f920 2024-03-31T15:06:31.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:31.015+0000 7f2995400640 1 --2- 172.21.15.16:0/1497983998 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f29a00cef80 0x7f29a00b37d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:06:31.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:31.015+0000 7f299de00640 1 --2- 172.21.15.16:0/1497983998 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f29a006f920 0x7f29a013e5d0 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f299000c7c0 tx=0x7f299000cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:06:31.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:31.019+0000 7f2996a00640 1 -- 172.21.15.16:0/1497983998 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2990016020 con 0x7f29a006f920 2024-03-31T15:06:31.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:31.019+0000 7f2996a00640 1 -- 172.21.15.16:0/1497983998 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f299000aa50 con 0x7f29a006f920 2024-03-31T15:06:31.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:31.019+0000 7f29a5000640 1 -- 172.21.15.16:0/1497983998 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f29a01fb0d0 con 0x7f29a006f920 2024-03-31T15:06:31.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:31.019+0000 7f2996a00640 1 -- 172.21.15.16:0/1497983998 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2990012420 con 0x7f29a006f920 2024-03-31T15:06:31.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:31.019+0000 7f29a5000640 1 -- 172.21.15.16:0/1497983998 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f29a01fb690 con 0x7f29a006f920 2024-03-31T15:06:31.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:31.019+0000 7f29a5000640 1 -- 172.21.15.16:0/1497983998 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f29a0069950 con 0x7f29a006f920 2024-03-31T15:06:31.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:31.019+0000 7f2996a00640 1 -- 172.21.15.16:0/1497983998 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f2990019020 con 0x7f29a006f920 2024-03-31T15:06:31.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:31.023+0000 7f2996a00640 1 --2- 172.21.15.16:0/1497983998 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f297c068380 0x7f297c06a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:31.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:31.023+0000 7f2996a00640 1 -- 172.21.15.16:0/1497983998 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 5709+0+0 (secure 0 0 0) 0x7f2990086940 con 0x7f29a006f920 2024-03-31T15:06:31.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:31.023+0000 7f2995400640 1 --2- 172.21.15.16:0/1497983998 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f297c068380 0x7f297c06a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:31.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:31.023+0000 7f2995400640 1 --2- 172.21.15.16:0/1497983998 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f297c068380 0x7f297c06a840 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f29880ad000 tx=0x7f29880a6000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:06:31.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:31.027+0000 7f2996a00640 1 -- 172.21.15.16:0/1497983998 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f2990020030 con 0x7f29a006f920 2024-03-31T15:06:31.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:31.419+0000 7f29a5000640 1 -- 172.21.15.16:0/1497983998 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) -- 0x7f29a00b50e0 con 0x7f29a006f920 2024-03-31T15:06:31.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:31.423+0000 7f2996a00640 1 -- 172.21.15.16:0/1497983998 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v15) ==== 190+0+147 (secure 0 0 0) 0x7f299005d0c0 con 0x7f29a006f920 2024-03-31T15:06:31.416 INFO:teuthology.orchestra.run.smithi016.stdout:[client.0] 2024-03-31T15:06:31.417 INFO:teuthology.orchestra.run.smithi016.stdout: key = AQAPewlmlMIDEhAAF0Ll1HcrWHmH6xDKq2Qq9A== 2024-03-31T15:06:31.417 INFO:teuthology.orchestra.run.smithi016.stdout: caps mds = "allow" 2024-03-31T15:06:31.417 INFO:teuthology.orchestra.run.smithi016.stdout: caps mon = "allow r" 2024-03-31T15:06:31.417 INFO:teuthology.orchestra.run.smithi016.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-03-31T15:06:31.417 INFO:teuthology.orchestra.run.smithi016.stderr:updated caps for client.0 2024-03-31T15:06:31.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:31.423+0000 7f29a5000640 1 -- 172.21.15.16:0/1497983998 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f297c068380 msgr2=0x7f297c06a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:31.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:31.423+0000 7f29a5000640 1 --2- 172.21.15.16:0/1497983998 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f297c068380 0x7f297c06a840 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f29880ad000 tx=0x7f29880a6000 comp rx=0 tx=0).stop 2024-03-31T15:06:31.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:31.423+0000 7f29a5000640 1 -- 172.21.15.16:0/1497983998 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f29a006f920 msgr2=0x7f29a013e5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:31.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:31.423+0000 7f29a5000640 1 --2- 172.21.15.16:0/1497983998 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f29a006f920 0x7f29a013e5d0 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f299000c7c0 tx=0x7f299000cc80 comp rx=0 tx=0).stop 2024-03-31T15:06:31.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:31.423+0000 7f29a5000640 1 -- 172.21.15.16:0/1497983998 shutdown_connections 2024-03-31T15:06:31.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:31.423+0000 7f29a5000640 1 --2- 172.21.15.16:0/1497983998 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f297c068380 0x7f297c06a840 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:31.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:31.423+0000 7f29a5000640 1 --2- 172.21.15.16:0/1497983998 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f29a00d0a60 0x7f29a006d8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:31.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:31.423+0000 7f29a5000640 1 --2- 172.21.15.16:0/1497983998 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f29a00cef80 0x7f29a00b37d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:31.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:31.423+0000 7f29a5000640 1 --2- 172.21.15.16:0/1497983998 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f29a006f920 0x7f29a013e5d0 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:31.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:31.423+0000 7f29a5000640 1 -- 172.21.15.16:0/1497983998 >> 172.21.15.16:0/1497983998 conn(0x7f29a0067930 msgr2=0x7f29a0157e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:06:31.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:31.423+0000 7f29a5000640 1 -- 172.21.15.16:0/1497983998 shutdown_connections 2024-03-31T15:06:31.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:31.423+0000 7f29a5000640 1 -- 172.21.15.16:0/1497983998 wait complete. 2024-03-31T15:06:31.438 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-31T15:06:31.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:31.583+0000 7ff69fe00640 1 Processor -- start 2024-03-31T15:06:31.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:31.583+0000 7ff69fe00640 1 -- start start 2024-03-31T15:06:31.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:31.587+0000 7ff69fe00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff6a00b0410 0x7ff6a006f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:31.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:31.587+0000 7ff69fe00640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff6a0145de0 con 0x7ff6a0145a00 2024-03-31T15:06:31.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:31.587+0000 7ff69fe00640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff6a015fc40 con 0x7ff6a006bd30 2024-03-31T15:06:31.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:31.587+0000 7ff69fe00640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7ff6a0160850 con 0x7ff6a00b0410 2024-03-31T15:06:31.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:31.587+0000 7ff69ea00640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7ff6a0145a00 0x7ff6a006b620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:56404/0 (socket says 172.21.15.16:56404) 2024-03-31T15:06:31.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:31.587+0000 7ff69ea00640 1 -- 172.21.15.16:0/3700887890 learned_addr learned my addr 172.21.15.16:0/3700887890 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:06:31.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:31.587+0000 7ff69f400640 1 --2- 172.21.15.16:0/3700887890 >> v2:172.21.15.16:3300/0 conn(0x7ff6a00b0410 0x7ff6a006f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:31.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:31.587+0000 7ff69f400640 1 -- 172.21.15.16:0/3700887890 >> v1:172.21.15.89:6789/0 conn(0x7ff6a006bd30 legacy=0x7ff6a006e990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:31.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:31.587+0000 7ff69f400640 1 -- 172.21.15.16:0/3700887890 >> v1:172.21.15.16:6789/0 conn(0x7ff6a0145a00 legacy=0x7ff6a006b620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:31.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:31.587+0000 7ff69f400640 1 -- 172.21.15.16:0/3700887890 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6a015ff20 con 0x7ff6a00b0410 2024-03-31T15:06:31.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:31.591+0000 7ff69f400640 1 --2- 172.21.15.16:0/3700887890 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff6a00b0410 0x7ff6a006f0a0 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7ff698009d90 tx=0x7ff6a006fdf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c1313db40ad248b0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:06:31.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:31.591+0000 7ff69e000640 1 -- 172.21.15.16:0/3700887890 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6980b2070 con 0x7ff6a00b0410 2024-03-31T15:06:31.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:31.591+0000 7ff69e000640 1 -- 172.21.15.16:0/3700887890 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff6980a5a00 con 0x7ff6a00b0410 2024-03-31T15:06:31.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:31.591+0000 7ff69e000640 1 -- 172.21.15.16:0/3700887890 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6980a5dd0 con 0x7ff6a00b0410 2024-03-31T15:06:31.585 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:31.591+0000 7ff69fe00640 1 -- 172.21.15.16:0/3700887890 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff6a00b0410 msgr2=0x7ff6a006f0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:31.585 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:31.591+0000 7ff69fe00640 1 --2- 172.21.15.16:0/3700887890 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff6a00b0410 0x7ff6a006f0a0 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7ff698009d90 tx=0x7ff6a006fdf0 comp rx=0 tx=0).stop 2024-03-31T15:06:31.585 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:31.591+0000 7ff69fe00640 1 -- 172.21.15.16:0/3700887890 shutdown_connections 2024-03-31T15:06:31.586 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:31.591+0000 7ff69fe00640 1 --2- 172.21.15.16:0/3700887890 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff6a00b0410 0x7ff6a006f0a0 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:31.586 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:31.591+0000 7ff69fe00640 1 -- 172.21.15.16:0/3700887890 >> 172.21.15.16:0/3700887890 conn(0x7ff6a00b2b30 msgr2=0x7ff6a00b2f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:06:31.586 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:31.591+0000 7ff69fe00640 1 -- 172.21.15.16:0/3700887890 shutdown_connections 2024-03-31T15:06:31.586 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:31.591+0000 7ff69fe00640 1 -- 172.21.15.16:0/3700887890 wait complete. 2024-03-31T15:06:31.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:31.595+0000 7ff69fe00640 1 Processor -- start 2024-03-31T15:06:31.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:31.595+0000 7ff69fe00640 1 -- start start 2024-03-31T15:06:31.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:31.595+0000 7ff69fe00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff6a006bd30 0x7ff6a00b4f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:31.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:31.595+0000 7ff69fe00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff6a00b0410 0x7ff6a009b720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:31.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:31.595+0000 7ff69fe00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff6a0145a00 0x7ff6a01fac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:31.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:31.595+0000 7ff69fe00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7ff6a016a0d0 con 0x7ff6a00b0410 2024-03-31T15:06:31.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:31.595+0000 7ff69fe00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7ff6a0169f50 con 0x7ff6a0145a00 2024-03-31T15:06:31.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:31.595+0000 7ff69fe00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7ff6a016a250 con 0x7ff6a006bd30 2024-03-31T15:06:31.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:31.595+0000 7ff696000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff6a00b0410 0x7ff6a009b720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:31.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:31.595+0000 7ff696000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff6a00b0410 0x7ff6a009b720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49076/0 (socket says 172.21.15.16:49076) 2024-03-31T15:06:31.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:31.595+0000 7ff696000640 1 -- 172.21.15.16:0/948031438 learned_addr learned my addr 172.21.15.16:0/948031438 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:06:31.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:31.595+0000 7ff69ea00640 1 --2- 172.21.15.16:0/948031438 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff6a006bd30 0x7ff6a00b4f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:31.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:31.595+0000 7ff69f400640 1 --2- 172.21.15.16:0/948031438 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff6a0145a00 0x7ff6a01fac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:31.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:31.595+0000 7ff696000640 1 -- 172.21.15.16:0/948031438 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff6a006bd30 msgr2=0x7ff6a00b4f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:31.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:31.595+0000 7ff696000640 1 --2- 172.21.15.16:0/948031438 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff6a006bd30 0x7ff6a00b4f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:31.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:31.595+0000 7ff696000640 1 -- 172.21.15.16:0/948031438 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff6a0145a00 msgr2=0x7ff6a01fac10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:31.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:31.595+0000 7ff696000640 1 --2- 172.21.15.16:0/948031438 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff6a0145a00 0x7ff6a01fac10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:31.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:31.595+0000 7ff696000640 1 -- 172.21.15.16:0/948031438 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6a013ed40 con 0x7ff6a00b0410 2024-03-31T15:06:31.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:31.595+0000 7ff696000640 1 --2- 172.21.15.16:0/948031438 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff6a00b0410 0x7ff6a009b720 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7ff68800c530 tx=0x7ff68800c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:06:31.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:31.595+0000 7ff697400640 1 -- 172.21.15.16:0/948031438 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff688016020 con 0x7ff6a00b0410 2024-03-31T15:06:31.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:31.595+0000 7ff697400640 1 -- 172.21.15.16:0/948031438 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff688005220 con 0x7ff6a00b0410 2024-03-31T15:06:31.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:31.595+0000 7ff697400640 1 -- 172.21.15.16:0/948031438 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff688007030 con 0x7ff6a00b0410 2024-03-31T15:06:31.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:31.595+0000 7ff69fe00640 1 -- 172.21.15.16:0/948031438 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff6a015c680 con 0x7ff6a00b0410 2024-03-31T15:06:31.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:31.595+0000 7ff69fe00640 1 -- 172.21.15.16:0/948031438 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff6a015cc40 con 0x7ff6a00b0410 2024-03-31T15:06:31.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:31.599+0000 7ff69fe00640 1 -- 172.21.15.16:0/948031438 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff6a004f8c0 con 0x7ff6a00b0410 2024-03-31T15:06:31.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:31.599+0000 7ff697400640 1 -- 172.21.15.16:0/948031438 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7ff6880071d0 con 0x7ff6a00b0410 2024-03-31T15:06:31.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:31.599+0000 7ff697400640 1 --2- 172.21.15.16:0/948031438 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7ff67c068380 0x7ff67c06a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:31.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:31.599+0000 7ff697400640 1 -- 172.21.15.16:0/948031438 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 5709+0+0 (secure 0 0 0) 0x7ff688086240 con 0x7ff6a00b0410 2024-03-31T15:06:31.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:31.599+0000 7ff69ea00640 1 --2- 172.21.15.16:0/948031438 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7ff67c068380 0x7ff67c06a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:31.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:31.603+0000 7ff69ea00640 1 --2- 172.21.15.16:0/948031438 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7ff67c068380 0x7ff67c06a840 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7ff690005890 tx=0x7ff690009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:06:31.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:31.603+0000 7ff697400640 1 -- 172.21.15.16:0/948031438 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7ff688020020 con 0x7ff6a00b0410 2024-03-31T15:06:31.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:31.803+0000 7ff69fe00640 1 -- 172.21.15.16:0/948031438 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff6a015ce10 con 0x7ff6a00b0410 2024-03-31T15:06:31.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:31.803+0000 7ff697400640 1 -- 172.21.15.16:0/948031438 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 34 v34) ==== 94+0+5181 (secure 0 0 0) 0x7ff6880074c0 con 0x7ff6a00b0410 2024-03-31T15:06:31.798 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:06:31.798 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":34,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5579,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6833/3721971977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":3721971977},{"type":"v1","addr":"172.21.15.16:6833","nonce":3721971977}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":29},{"gid":6109,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6836/1166206553","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":1166206553},{"type":"v1","addr":"172.21.15.89:6836","nonce":1166206553}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28},{"gid":6112,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/1826113083","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6835","nonce":1826113083},{"type":"v1","addr":"172.21.15.89:6837","nonce":1826113083}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":29}],"filesystems":[{"mdsmap":{"epoch":34,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-31T15:06:25.665001+0000","modified":"2024-03-31T15:06:28.707528+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6150},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6150":{"gid":6150,"name":"c","rank":0,"incarnation":30,"state":"up:active","state_seq":3,"addr":"172.21.15.16:6835/2613117528","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":2613117528},{"type":"v1","addr":"172.21.15.16:6835","nonce":2613117528}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6150,"qdb_cluster":[6150]},"id":4}]} 2024-03-31T15:06:31.798 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 34 2024-03-31T15:06:31.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:31.803+0000 7ff69fe00640 1 -- 172.21.15.16:0/948031438 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7ff67c068380 msgr2=0x7ff67c06a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:31.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:31.803+0000 7ff69fe00640 1 --2- 172.21.15.16:0/948031438 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7ff67c068380 0x7ff67c06a840 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7ff690005890 tx=0x7ff690009040 comp rx=0 tx=0).stop 2024-03-31T15:06:31.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:31.803+0000 7ff69fe00640 1 -- 172.21.15.16:0/948031438 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff6a00b0410 msgr2=0x7ff6a009b720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:31.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:31.803+0000 7ff69fe00640 1 --2- 172.21.15.16:0/948031438 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff6a00b0410 0x7ff6a009b720 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7ff68800c530 tx=0x7ff68800c9f0 comp rx=0 tx=0).stop 2024-03-31T15:06:31.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:31.803+0000 7ff69fe00640 1 -- 172.21.15.16:0/948031438 shutdown_connections 2024-03-31T15:06:31.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:31.803+0000 7ff69fe00640 1 --2- 172.21.15.16:0/948031438 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7ff67c068380 0x7ff67c06a840 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:31.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:31.803+0000 7ff69fe00640 1 --2- 172.21.15.16:0/948031438 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff6a0145a00 0x7ff6a01fac10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:31.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:31.803+0000 7ff69fe00640 1 --2- 172.21.15.16:0/948031438 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff6a00b0410 0x7ff6a009b720 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:31.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:31.803+0000 7ff69fe00640 1 --2- 172.21.15.16:0/948031438 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff6a006bd30 0x7ff6a00b4f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:31.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:31.803+0000 7ff69fe00640 1 -- 172.21.15.16:0/948031438 >> 172.21.15.16:0/948031438 conn(0x7ff6a00b2b30 msgr2=0x7ff6a00b2320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:06:31.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:31.803+0000 7ff69fe00640 1 -- 172.21.15.16:0/948031438 shutdown_connections 2024-03-31T15:06:31.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:31.807+0000 7ff69fe00640 1 -- 172.21.15.16:0/948031438 wait complete. 2024-03-31T15:06:32.223 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 34, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-31T15:06:25.665001+0000', 'modified': '2024-03-31T15:06:28.707528+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6150}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6150': {'gid': 6150, 'name': 'c', 'rank': 0, 'incarnation': 30, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.16:6835/2613117528', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6834', 'nonce': 2613117528}, {'type': 'v1', 'addr': '172.21.15.16:6835', 'nonce': 2613117528}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [9], 'metadata_pool': 8, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 6150, 'qdb_cluster': [6150]} 2024-03-31T15:06:32.223 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-31T15:06:32.223 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-03-31T15:06:32.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.371+0000 7fe17bc00640 1 Processor -- start 2024-03-31T15:06:32.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.371+0000 7fe17bc00640 1 -- start start 2024-03-31T15:06:32.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.371+0000 7fe17bc00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fe174145e40 0x7fe1740cc740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:32.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.371+0000 7fe17bc00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fe174070030 0x7fe1740d2010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:32.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.371+0000 7fe17bc00640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe1740d0920 con 0x7fe1740ccc80 2024-03-31T15:06:32.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.371+0000 7fe17bc00640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7fe1740d06d0 con 0x7fe174145e40 2024-03-31T15:06:32.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.371+0000 7fe17bc00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fe1740d0c00 con 0x7fe174070030 2024-03-31T15:06:32.368 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.371+0000 7fe179200640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fe174145e40 0x7fe1740cc740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:32.368 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.371+0000 7fe179c00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fe174070030 0x7fe1740d2010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:32.368 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.371+0000 7fe179c00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fe174070030 0x7fe1740d2010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3301/0 says I am v2:172.21.15.16:42018/0 (socket says 172.21.15.16:42018) 2024-03-31T15:06:32.368 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.371+0000 7fe173e00640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fe1740ccc80 0x7fe17406f920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.16:50964/0 (socket says 172.21.15.16:50964) 2024-03-31T15:06:32.368 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.371+0000 7fe179c00640 1 -- 172.21.15.16:0/429638862 learned_addr learned my addr 172.21.15.16:0/429638862 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:06:32.368 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.371+0000 7fe179200640 1 --2- 172.21.15.16:0/429638862 >> v2:172.21.15.89:3300/0 conn(0x7fe174145e40 0x7fe1740cc740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.16:38902/0 (socket says 172.21.15.16:38902) 2024-03-31T15:06:32.368 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.375+0000 7fe179c00640 1 -- 172.21.15.16:0/429638862 >> v2:172.21.15.89:3300/0 conn(0x7fe174145e40 msgr2=0x7fe1740cc740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:06:32.369 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.375+0000 7fe179c00640 1 --2- 172.21.15.16:0/429638862 >> v2:172.21.15.89:3300/0 conn(0x7fe174145e40 0x7fe1740cc740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:32.369 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.375+0000 7fe179c00640 1 -- 172.21.15.16:0/429638862 >> v1:172.21.15.89:6789/0 conn(0x7fe1740ccc80 legacy=0x7fe17406f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:32.369 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.375+0000 7fe179c00640 1 -- 172.21.15.16:0/429638862 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe1740d1810 con 0x7fe174070030 2024-03-31T15:06:32.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.375+0000 7fe179c00640 1 --2- 172.21.15.16:0/429638862 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fe174070030 0x7fe1740d2010 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fe16c009e50 tx=0x7fe16c01ca30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2ccedafe4ddc3a54 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:06:32.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.375+0000 7fe173400640 1 -- 172.21.15.16:0/429638862 <== mon.2 v2:172.21.15.89:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe16c0a5810 con 0x7fe174070030 2024-03-31T15:06:32.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.375+0000 7fe173400640 1 -- 172.21.15.16:0/429638862 <== mon.2 v2:172.21.15.89:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe16c0a59b0 con 0x7fe174070030 2024-03-31T15:06:32.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.375+0000 7fe173400640 1 -- 172.21.15.16:0/429638862 <== mon.2 v2:172.21.15.89:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe16c0a5d80 con 0x7fe174070030 2024-03-31T15:06:32.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.379+0000 7fe17bc00640 1 -- 172.21.15.16:0/429638862 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fe174070030 msgr2=0x7fe1740d2010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:32.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.379+0000 7fe17bc00640 1 --2- 172.21.15.16:0/429638862 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fe174070030 0x7fe1740d2010 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fe16c009e50 tx=0x7fe16c01ca30 comp rx=0 tx=0).stop 2024-03-31T15:06:32.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.379+0000 7fe17bc00640 1 -- 172.21.15.16:0/429638862 shutdown_connections 2024-03-31T15:06:32.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.379+0000 7fe17bc00640 1 --2- 172.21.15.16:0/429638862 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fe174070030 0x7fe1740d2010 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:32.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.379+0000 7fe17bc00640 1 --2- 172.21.15.16:0/429638862 >> v2:172.21.15.89:3300/0 conn(0x7fe174145e40 0x7fe1740cc740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:32.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.379+0000 7fe17bc00640 1 -- 172.21.15.16:0/429638862 >> 172.21.15.16:0/429638862 conn(0x7fe174067330 msgr2=0x7fe174067740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:06:32.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.379+0000 7fe17bc00640 1 -- 172.21.15.16:0/429638862 shutdown_connections 2024-03-31T15:06:32.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.379+0000 7fe17bc00640 1 -- 172.21.15.16:0/429638862 wait complete. 2024-03-31T15:06:32.375 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.379+0000 7fe17bc00640 1 Processor -- start 2024-03-31T15:06:32.375 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.379+0000 7fe17bc00640 1 -- start start 2024-03-31T15:06:32.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.383+0000 7fe17bc00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe174070030 0x7fe17413e400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:32.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.383+0000 7fe17bc00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fe1740ccc80 0x7fe174140190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:32.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.383+0000 7fe17bc00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe174145e40 0x7fe17413c130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:32.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.383+0000 7fe17bc00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fe1740b5240 con 0x7fe174145e40 2024-03-31T15:06:32.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.383+0000 7fe17bc00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fe1740b50c0 con 0x7fe174070030 2024-03-31T15:06:32.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.383+0000 7fe17bc00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fe174169190 con 0x7fe1740ccc80 2024-03-31T15:06:32.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.383+0000 7fe179c00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe174145e40 0x7fe17413c130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:32.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.383+0000 7fe179c00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe174145e40 0x7fe17413c130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49080/0 (socket says 172.21.15.16:49080) 2024-03-31T15:06:32.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.383+0000 7fe179c00640 1 -- 172.21.15.16:0/2610020467 learned_addr learned my addr 172.21.15.16:0/2610020467 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:06:32.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.383+0000 7fe173e00640 1 --2- 172.21.15.16:0/2610020467 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fe1740ccc80 0x7fe174140190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:32.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.383+0000 7fe179200640 1 --2- 172.21.15.16:0/2610020467 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe174070030 0x7fe17413e400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:32.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.383+0000 7fe179c00640 1 -- 172.21.15.16:0/2610020467 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fe1740ccc80 msgr2=0x7fe174140190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:32.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.383+0000 7fe179c00640 1 --2- 172.21.15.16:0/2610020467 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fe1740ccc80 0x7fe174140190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:32.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.383+0000 7fe179c00640 1 -- 172.21.15.16:0/2610020467 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe174070030 msgr2=0x7fe17413e400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:32.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.383+0000 7fe179c00640 1 --2- 172.21.15.16:0/2610020467 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe174070030 0x7fe17413e400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:32.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.383+0000 7fe179c00640 1 -- 172.21.15.16:0/2610020467 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe17413e940 con 0x7fe174145e40 2024-03-31T15:06:32.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.383+0000 7fe179c00640 1 --2- 172.21.15.16:0/2610020467 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe174145e40 0x7fe17413c130 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7fe16c0af040 tx=0x7fe16c0197d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:06:32.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.383+0000 7fe170c00640 1 -- 172.21.15.16:0/2610020467 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe16c0bc020 con 0x7fe174145e40 2024-03-31T15:06:32.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.383+0000 7fe170c00640 1 -- 172.21.15.16:0/2610020467 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe16c0b8670 con 0x7fe174145e40 2024-03-31T15:06:32.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.383+0000 7fe170c00640 1 -- 172.21.15.16:0/2610020467 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe16c0b8a60 con 0x7fe174145e40 2024-03-31T15:06:32.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.383+0000 7fe17bc00640 1 -- 172.21.15.16:0/2610020467 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe17415e230 con 0x7fe174145e40 2024-03-31T15:06:32.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.383+0000 7fe17bc00640 1 -- 172.21.15.16:0/2610020467 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe17413f990 con 0x7fe174145e40 2024-03-31T15:06:32.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.387+0000 7fe170c00640 1 -- 172.21.15.16:0/2610020467 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fe16c0bf020 con 0x7fe174145e40 2024-03-31T15:06:32.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.387+0000 7fe170c00640 1 --2- 172.21.15.16:0/2610020467 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fe154068380 0x7fe15406a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:32.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.387+0000 7fe170c00640 1 -- 172.21.15.16:0/2610020467 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 5709+0+0 (secure 0 0 0) 0x7fe16c1218a0 con 0x7fe174145e40 2024-03-31T15:06:32.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.387+0000 7fe17effa000 1 Processor -- start 2024-03-31T15:06:32.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.387+0000 7fe17effa000 1 -- start start 2024-03-31T15:06:32.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.387+0000 7fe179200640 1 --2- 172.21.15.16:0/2610020467 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fe154068380 0x7fe15406a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:32.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.387+0000 7fe179200640 1 --2- 172.21.15.16:0/2610020467 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fe154068380 0x7fe15406a840 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fe17415e090 tx=0x7fe168009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:06:32.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.387+0000 7fe17effa000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dd9ec1d4a0 0x55dd9ec1d880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:32.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.387+0000 7fe17effa000 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55dd9ec1ddc0 0x55dd9ebd0b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:32.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.387+0000 7fe17effa000 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55dd9ebd10b0 0x55dd9ebd1490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:32.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.387+0000 7fe17effa000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55dd9ea23950 con 0x55dd9ec1d4a0 2024-03-31T15:06:32.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.387+0000 7fe17effa000 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55dd9e9c0990 con 0x55dd9ebd10b0 2024-03-31T15:06:32.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.387+0000 7fe17effa000 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x55dd9ea1f900 con 0x55dd9ec1ddc0 2024-03-31T15:06:32.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.387+0000 7fe179200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dd9ec1d4a0 0x55dd9ec1d880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:32.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.387+0000 7fe179200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dd9ec1d4a0 0x55dd9ec1d880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49086/0 (socket says 172.21.15.16:49086) 2024-03-31T15:06:32.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.387+0000 7fe179200640 1 -- 172.21.15.16:0/3071102069 learned_addr learned my addr 172.21.15.16:0/3071102069 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:06:32.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.387+0000 7fe173e00640 1 --2- 172.21.15.16:0/3071102069 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55dd9ec1ddc0 0x55dd9ebd0b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:32.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.387+0000 7fe179c00640 1 --2- 172.21.15.16:0/3071102069 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55dd9ebd10b0 0x55dd9ebd1490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:32.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.387+0000 7fe179200640 1 -- 172.21.15.16:0/3071102069 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55dd9ec1ddc0 msgr2=0x55dd9ebd0b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:06:32.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.387+0000 7fe179200640 1 --2- 172.21.15.16:0/3071102069 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55dd9ec1ddc0 0x55dd9ebd0b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:32.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.387+0000 7fe179200640 1 -- 172.21.15.16:0/3071102069 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55dd9ebd10b0 msgr2=0x55dd9ebd1490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:06:32.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.387+0000 7fe179200640 1 --2- 172.21.15.16:0/3071102069 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55dd9ebd10b0 0x55dd9ebd1490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:32.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.387+0000 7fe179200640 1 -- 172.21.15.16:0/3071102069 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dd9ea68100 con 0x55dd9ec1d4a0 2024-03-31T15:06:32.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.387+0000 7fe179c00640 1 --2- 172.21.15.16:0/3071102069 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55dd9ebd10b0 0x55dd9ebd1490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:06:32.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.391+0000 7fe179200640 1 --2- 172.21.15.16:0/3071102069 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dd9ec1d4a0 0x55dd9ec1d880 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7fe168011de0 tx=0x7fe168014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=61bfbfdb609c32d0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:06:32.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.391+0000 7fe165600640 1 -- 172.21.15.16:0/3071102069 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe168014bd0 con 0x55dd9ec1d4a0 2024-03-31T15:06:32.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.391+0000 7fe165600640 1 -- 172.21.15.16:0/3071102069 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe16801b070 con 0x55dd9ec1d4a0 2024-03-31T15:06:32.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.391+0000 7fe165600640 1 -- 172.21.15.16:0/3071102069 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe168004030 con 0x55dd9ec1d4a0 2024-03-31T15:06:32.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.391+0000 7fe17effa000 1 -- 172.21.15.16:0/3071102069 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dd9ec1d4a0 msgr2=0x55dd9ec1d880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:32.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.391+0000 7fe17effa000 1 --2- 172.21.15.16:0/3071102069 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dd9ec1d4a0 0x55dd9ec1d880 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7fe168011de0 tx=0x7fe168014000 comp rx=0 tx=0).stop 2024-03-31T15:06:32.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.391+0000 7fe17effa000 1 -- 172.21.15.16:0/3071102069 shutdown_connections 2024-03-31T15:06:32.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.391+0000 7fe17effa000 1 --2- 172.21.15.16:0/3071102069 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55dd9ebd10b0 0x55dd9ebd1490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:32.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.391+0000 7fe17effa000 1 --2- 172.21.15.16:0/3071102069 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55dd9ec1ddc0 0x55dd9ebd0b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:32.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.391+0000 7fe17effa000 1 --2- 172.21.15.16:0/3071102069 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dd9ec1d4a0 0x55dd9ec1d880 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:32.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.391+0000 7fe17effa000 1 -- 172.21.15.16:0/3071102069 >> 172.21.15.16:0/3071102069 conn(0x55dd9eabbb80 msgr2=0x55dd9ec00460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:06:32.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.391+0000 7fe17effa000 1 -- 172.21.15.16:0/3071102069 shutdown_connections 2024-03-31T15:06:32.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.391+0000 7fe17effa000 1 -- 172.21.15.16:0/3071102069 wait complete. 2024-03-31T15:06:32.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.391+0000 7fe17effa000 1 -- start start 2024-03-31T15:06:32.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.391+0000 7fe17effa000 1 Processor -- start 2024-03-31T15:06:32.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.391+0000 7fe17effa000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dd9ebd10b0 0x55dd9ebdde00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:32.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.391+0000 7fe17effa000 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55dd9ec1d4a0 0x55dd9ebde340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:32.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.391+0000 7fe17effa000 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55dd9ec1ddc0 0x55dd9ebde880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:32.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.391+0000 7fe17effa000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55dd9ea22c60 con 0x55dd9ebd10b0 2024-03-31T15:06:32.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.391+0000 7fe17effa000 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55dd9e9e13e0 con 0x55dd9ec1d4a0 2024-03-31T15:06:32.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.391+0000 7fe17effa000 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x55dd9ea30060 con 0x55dd9ec1ddc0 2024-03-31T15:06:32.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.391+0000 7fe17effa000 10 client.0 _ll_register_callbacks cb 0x55dd9ea840f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T15:06:32.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.391+0000 7fe179200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dd9ebd10b0 0x55dd9ebdde00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:32.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.391+0000 7fe179200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dd9ebd10b0 0x55dd9ebdde00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49096/0 (socket says 172.21.15.16:49096) 2024-03-31T15:06:32.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.391+0000 7fe179200640 1 -- 172.21.15.16:0/110253771 learned_addr learned my addr 172.21.15.16:0/110253771 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:06:32.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.391+0000 7fe179c00640 1 --2- 172.21.15.16:0/110253771 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55dd9ec1ddc0 0x55dd9ebde880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:32.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.391+0000 7fe173e00640 1 --2- 172.21.15.16:0/110253771 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55dd9ec1d4a0 0x55dd9ebde340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:32.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.391+0000 7fe179c00640 1 -- 172.21.15.16:0/110253771 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55dd9ec1d4a0 msgr2=0x55dd9ebde340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:06:32.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.391+0000 7fe179c00640 1 --2- 172.21.15.16:0/110253771 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55dd9ec1d4a0 0x55dd9ebde340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:32.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.391+0000 7fe179c00640 1 -- 172.21.15.16:0/110253771 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dd9ebd10b0 msgr2=0x55dd9ebdde00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:06:32.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.391+0000 7fe179c00640 1 --2- 172.21.15.16:0/110253771 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dd9ebd10b0 0x55dd9ebdde00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:32.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.391+0000 7fe179c00640 1 -- 172.21.15.16:0/110253771 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({osdmap=0}) -- 0x55dd9e9ed8c0 con 0x55dd9ec1ddc0 2024-03-31T15:06:32.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.391+0000 7fe179200640 1 --2- 172.21.15.16:0/110253771 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dd9ebd10b0 0x55dd9ebdde00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:06:32.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.391+0000 7fe179c00640 1 --2- 172.21.15.16:0/110253771 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55dd9ec1ddc0 0x55dd9ebde880 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fe16c11feb0 tx=0x7fe16c0f15c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=484ed5bf3e2bd6fa server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:06:32.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.391+0000 7fe15a000640 10 client.0 ms_handle_connect on v2:172.21.15.89:3301/0 2024-03-31T15:06:32.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.395+0000 7fe15a000640 1 -- 172.21.15.16:0/110253771 <== mon.2 v2:172.21.15.89:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe16c0f5cc0 con 0x55dd9ec1ddc0 2024-03-31T15:06:32.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.395+0000 7fe15a000640 1 -- 172.21.15.16:0/110253771 <== mon.2 v2:172.21.15.89:3301/0 2 ==== osd_map(56..56 src has 1..56) ==== 5709+0+0 (secure 0 0 0) 0x7fe16c11ae70 con 0x55dd9ec1ddc0 2024-03-31T15:06:32.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.395+0000 7fe17effa000 1 -- 172.21.15.16:0/110253771 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe16801b070 con 0x55dd9ec1ddc0 2024-03-31T15:06:32.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.395+0000 7fe15a000640 1 -- 172.21.15.16:0/110253771 <== mon.2 v2:172.21.15.89:3301/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7fe16c0fa4f0 con 0x55dd9ec1ddc0 2024-03-31T15:06:32.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.395+0000 7fe17effa000 10 client.5660 fetch_fsmap learned FSMap version 34 2024-03-31T15:06:32.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.395+0000 7fe17effa000 1 -- 172.21.15.16:0/110253771 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({config=0+,fsmap=34,monmap=2+}) -- 0x55dd9e9abe80 con 0x55dd9ec1ddc0 2024-03-31T15:06:32.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.395+0000 7fe15a000640 1 -- 172.21.15.16:0/110253771 <== mon.2 v2:172.21.15.89:3301/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe16801b070 con 0x55dd9ec1ddc0 2024-03-31T15:06:32.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.395+0000 7fe15a000640 1 -- 172.21.15.16:0/110253771 <== mon.2 v2:172.21.15.89:3301/0 5 ==== fsmap(e 34) ==== 3494+0+0 (secure 0 0 0) 0x7fe16c0f1c20 con 0x55dd9ec1ddc0 2024-03-31T15:06:32.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.395+0000 7fe17effa000 10 client.5660 fetch_fsmap finished waiting for FSMap version 34 2024-03-31T15:06:32.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.395+0000 7fe17effa000 10 client.5660 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-31T15:06:32.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.395+0000 7fe17effa000 20 client.5660 populate_metadata read hostname 'smithi016' 2024-03-31T15:06:32.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.395+0000 7fe17effa000 1 --2- 172.21.15.16:0/110253771 >> [v2:172.21.15.16:6834/2613117528,v1:172.21.15.16:6835/2613117528] conn(0x55dd9ebdf120 0x55dd9ec2fa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:32.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.395+0000 7fe17effa000 4 client.5660 mds_command: new command op to 6150 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-31T15:06:32.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.395+0000 7fe17effa000 1 -- 172.21.15.16:0/110253771 --> [v2:172.21.15.16:6834/2613117528,v1:172.21.15.16:6835/2613117528] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55dd9e9bebc0 con 0x55dd9ebdf120 2024-03-31T15:06:32.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.395+0000 7fe179200640 1 --2- 172.21.15.16:0/110253771 >> [v2:172.21.15.16:6834/2613117528,v1:172.21.15.16:6835/2613117528] conn(0x55dd9ebdf120 0x55dd9ec2fa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:32.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.395+0000 7fe15a000640 10 client.5660 ms_handle_connect on v2:172.21.15.16:6834/2613117528 2024-03-31T15:06:32.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.395+0000 7fe179200640 1 --2- 172.21.15.16:0/110253771 >> [v2:172.21.15.16:6834/2613117528,v1:172.21.15.16:6835/2613117528] conn(0x55dd9ebdf120 0x55dd9ec2fa20 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=67e64dc8e26458d server_cookie=9aedc87776538381 in_seq=0 out_seq=0 2024-03-31T15:06:32.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.395+0000 7fe15a000640 1 -- 172.21.15.16:0/110253771 <== mds.0 v2:172.21.15.16:6834/2613117528 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55dd9e9bebc0 con 0x55dd9ebdf120 2024-03-31T15:06:32.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.395+0000 7fe15a000640 10 client.5660 handle_command_reply: tid=0 2024-03-31T15:06:32.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.395+0000 7fe17effa000 1 client.5660 shutdown 2024-03-31T15:06:32.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.395+0000 7fe17effa000 1 -- 172.21.15.16:0/110253771 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55dd9ec1ddc0 msgr2=0x55dd9ebde880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:32.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.395+0000 7fe17effa000 1 --2- 172.21.15.16:0/110253771 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55dd9ec1ddc0 0x55dd9ebde880 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fe16c11feb0 tx=0x7fe16c0f15c0 comp rx=0 tx=0).stop 2024-03-31T15:06:32.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.395+0000 7fe17effa000 1 -- 172.21.15.16:0/110253771 shutdown_connections 2024-03-31T15:06:32.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.395+0000 7fe17effa000 1 --2- 172.21.15.16:0/110253771 >> [v2:172.21.15.16:6834/2613117528,v1:172.21.15.16:6835/2613117528] conn(0x55dd9ebdf120 0x55dd9ec2fa20 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:32.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.395+0000 7fe17effa000 1 --2- 172.21.15.16:0/110253771 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55dd9ec1ddc0 0x55dd9ebde880 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:32.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.395+0000 7fe17effa000 1 --2- 172.21.15.16:0/110253771 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55dd9ec1d4a0 0x55dd9ebde340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:32.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.395+0000 7fe17effa000 1 --2- 172.21.15.16:0/110253771 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dd9ebd10b0 0x55dd9ebdde00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:32.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.395+0000 7fe17effa000 1 -- 172.21.15.16:0/110253771 >> 172.21.15.16:0/110253771 conn(0x55dd9eabbb80 msgr2=0x55dd9ebd0270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:06:32.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.395+0000 7fe15a000640 0 client.5660 ms_handle_reset on v2:172.21.15.16:6834/2613117528 2024-03-31T15:06:32.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.395+0000 7fe17effa000 1 -- 172.21.15.16:0/110253771 shutdown_connections 2024-03-31T15:06:32.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.395+0000 7fe17effa000 1 -- 172.21.15.16:0/110253771 wait complete. 2024-03-31T15:06:32.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.395+0000 7fe17effa000 20 client.5660 trim_cache size 0 max 16384 2024-03-31T15:06:32.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.415+0000 7fe17effa000 1 Processor -- start 2024-03-31T15:06:32.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.415+0000 7fe17effa000 1 -- start start 2024-03-31T15:06:32.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.415+0000 7fe17effa000 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55dd9ebd10b0 0x55dd9ebede50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:32.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.415+0000 7fe17effa000 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55dd9ec1d4a0 0x55dd9ebee390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:32.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.415+0000 7fe17effa000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dd9ec1ddc0 0x55dd9ebd02e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:32.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.415+0000 7fe17effa000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fe17415f880 con 0x55dd9ec1ddc0 2024-03-31T15:06:32.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.415+0000 7fe17effa000 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fe17415ebc0 con 0x55dd9ec1d4a0 2024-03-31T15:06:32.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.415+0000 7fe17effa000 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fe17415f210 con 0x55dd9ebd10b0 2024-03-31T15:06:32.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.415+0000 7fe179c00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dd9ec1ddc0 0x55dd9ebd02e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:32.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.415+0000 7fe179c00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dd9ec1ddc0 0x55dd9ebd02e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49104/0 (socket says 172.21.15.16:49104) 2024-03-31T15:06:32.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.415+0000 7fe179c00640 1 -- 172.21.15.16:0/1344411698 learned_addr learned my addr 172.21.15.16:0/1344411698 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:06:32.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.415+0000 7fe173e00640 1 --2- 172.21.15.16:0/1344411698 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55dd9ec1d4a0 0x55dd9ebee390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:32.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.415+0000 7fe179200640 1 --2- 172.21.15.16:0/1344411698 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55dd9ebd10b0 0x55dd9ebede50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:32.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.415+0000 7fe179c00640 1 -- 172.21.15.16:0/1344411698 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55dd9ebd10b0 msgr2=0x55dd9ebede50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:06:32.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.415+0000 7fe179c00640 1 --2- 172.21.15.16:0/1344411698 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55dd9ebd10b0 0x55dd9ebede50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:32.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.415+0000 7fe179c00640 1 -- 172.21.15.16:0/1344411698 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55dd9ec1d4a0 msgr2=0x55dd9ebee390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:06:32.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.415+0000 7fe179c00640 1 --2- 172.21.15.16:0/1344411698 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55dd9ec1d4a0 0x55dd9ebee390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:32.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.415+0000 7fe179c00640 1 -- 172.21.15.16:0/1344411698 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dd9e9de490 con 0x55dd9ec1ddc0 2024-03-31T15:06:32.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.415+0000 7fe179200640 1 --2- 172.21.15.16:0/1344411698 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55dd9ebd10b0 0x55dd9ebede50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:06:32.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.415+0000 7fe173e00640 1 --2- 172.21.15.16:0/1344411698 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55dd9ec1d4a0 0x55dd9ebee390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:06:32.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.415+0000 7fe179c00640 1 --2- 172.21.15.16:0/1344411698 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dd9ec1ddc0 0x55dd9ebd02e0 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7fe16c0ede20 tx=0x7fe16c0c2000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65474133bcf7e384 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:06:32.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.415+0000 7fe164c00640 1 -- 172.21.15.16:0/1344411698 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe16c0f5cc0 con 0x55dd9ec1ddc0 2024-03-31T15:06:32.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.415+0000 7fe164c00640 1 -- 172.21.15.16:0/1344411698 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe16c0f6e50 con 0x55dd9ec1ddc0 2024-03-31T15:06:32.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.415+0000 7fe164c00640 1 -- 172.21.15.16:0/1344411698 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe16c0fa060 con 0x55dd9ec1ddc0 2024-03-31T15:06:32.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.415+0000 7fe17effa000 1 -- 172.21.15.16:0/1344411698 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dd9ec1ddc0 msgr2=0x55dd9ebd02e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:32.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.415+0000 7fe17effa000 1 --2- 172.21.15.16:0/1344411698 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dd9ec1ddc0 0x55dd9ebd02e0 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7fe16c0ede20 tx=0x7fe16c0c2000 comp rx=0 tx=0).stop 2024-03-31T15:06:32.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.415+0000 7fe17effa000 1 -- 172.21.15.16:0/1344411698 shutdown_connections 2024-03-31T15:06:32.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.415+0000 7fe17effa000 1 --2- 172.21.15.16:0/1344411698 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dd9ec1ddc0 0x55dd9ebd02e0 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:32.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.415+0000 7fe17effa000 1 --2- 172.21.15.16:0/1344411698 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55dd9ec1d4a0 0x55dd9ebee390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:32.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.415+0000 7fe17effa000 1 --2- 172.21.15.16:0/1344411698 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55dd9ebd10b0 0x55dd9ebede50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:32.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.415+0000 7fe17effa000 1 -- 172.21.15.16:0/1344411698 >> 172.21.15.16:0/1344411698 conn(0x55dd9eabbb80 msgr2=0x55dd9ec2a870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:06:32.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.415+0000 7fe17effa000 1 -- 172.21.15.16:0/1344411698 shutdown_connections 2024-03-31T15:06:32.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.415+0000 7fe17effa000 1 -- 172.21.15.16:0/1344411698 wait complete. 2024-03-31T15:06:32.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.419+0000 7fe17effa000 1 -- start start 2024-03-31T15:06:32.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.419+0000 7fe17effa000 1 Processor -- start 2024-03-31T15:06:32.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.419+0000 7fe17effa000 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55dd9ebd10b0 0x55dd9ec928b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:32.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.419+0000 7fe17effa000 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55dd9ec1d4a0 0x55dd9ec92df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:32.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.419+0000 7fe17effa000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dd9ec1ddc0 0x55dd9ec8ee90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:32.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.419+0000 7fe17effa000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55dd9e9a7b10 con 0x55dd9ec1ddc0 2024-03-31T15:06:32.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.419+0000 7fe17effa000 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55dd9ea27f80 con 0x55dd9ebd10b0 2024-03-31T15:06:32.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.419+0000 7fe17effa000 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x55dd9ea30060 con 0x55dd9ec1d4a0 2024-03-31T15:06:32.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.419+0000 7fe17effa000 10 client.0 _ll_register_callbacks cb 0x7fe16c0f1790 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T15:06:32.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.419+0000 7fe179c00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dd9ec1ddc0 0x55dd9ec8ee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:32.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.419+0000 7fe179c00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dd9ec1ddc0 0x55dd9ec8ee90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49108/0 (socket says 172.21.15.16:49108) 2024-03-31T15:06:32.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.419+0000 7fe179c00640 1 -- 172.21.15.16:0/1447422208 learned_addr learned my addr 172.21.15.16:0/1447422208 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:06:32.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.419+0000 7fe179200640 1 --2- 172.21.15.16:0/1447422208 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55dd9ebd10b0 0x55dd9ec928b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:32.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.419+0000 7fe173e00640 1 --2- 172.21.15.16:0/1447422208 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55dd9ec1d4a0 0x55dd9ec92df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:32.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.419+0000 7fe179c00640 1 -- 172.21.15.16:0/1447422208 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55dd9ec1d4a0 msgr2=0x55dd9ec92df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:06:32.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.419+0000 7fe179c00640 1 --2- 172.21.15.16:0/1447422208 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55dd9ec1d4a0 0x55dd9ec92df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:32.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.419+0000 7fe179c00640 1 -- 172.21.15.16:0/1447422208 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55dd9ebd10b0 msgr2=0x55dd9ec928b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:06:32.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.419+0000 7fe179c00640 1 --2- 172.21.15.16:0/1447422208 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55dd9ebd10b0 0x55dd9ec928b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:32.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.419+0000 7fe179c00640 1 -- 172.21.15.16:0/1447422208 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dd9e9dce80 con 0x55dd9ec1ddc0 2024-03-31T15:06:32.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.419+0000 7fe179c00640 1 --2- 172.21.15.16:0/1447422208 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dd9ec1ddc0 0x55dd9ec8ee90 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7fe16c019640 tx=0x7fe16c11fd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d2a3f0dfc7a9a94e server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:06:32.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.419+0000 7fe15a000640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-31T15:06:32.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.419+0000 7fe15a000640 1 -- 172.21.15.16:0/1447422208 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe16c0fa060 con 0x55dd9ec1ddc0 2024-03-31T15:06:32.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.419+0000 7fe15a000640 1 -- 172.21.15.16:0/1447422208 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(56..56 src has 1..56) ==== 5709+0+0 (secure 0 0 0) 0x7fe16c0fa200 con 0x55dd9ec1ddc0 2024-03-31T15:06:32.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.419+0000 7fe17effa000 1 -- 172.21.15.16:0/1447422208 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe16c0f6e50 con 0x55dd9ec1ddc0 2024-03-31T15:06:32.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.419+0000 7fe15a000640 1 -- 172.21.15.16:0/1447422208 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7fe16c0f0e10 con 0x55dd9ec1ddc0 2024-03-31T15:06:32.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.419+0000 7fe17effa000 10 client.6288 fetch_fsmap learned FSMap version 34 2024-03-31T15:06:32.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.419+0000 7fe17effa000 1 -- 172.21.15.16:0/1447422208 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=2+}) -- 0x55dd9e9ca270 con 0x55dd9ec1ddc0 2024-03-31T15:06:32.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.419+0000 7fe15a000640 1 -- 172.21.15.16:0/1447422208 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe16c0f6e50 con 0x55dd9ec1ddc0 2024-03-31T15:06:32.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.419+0000 7fe15a000640 1 -- 172.21.15.16:0/1447422208 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 34) ==== 3494+0+0 (secure 0 0 0) 0x7fe16c0f0fb0 con 0x55dd9ec1ddc0 2024-03-31T15:06:32.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.419+0000 7fe17effa000 10 client.6288 fetch_fsmap finished waiting for FSMap version 34 2024-03-31T15:06:32.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.419+0000 7fe17effa000 10 client.6288 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-31T15:06:32.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.419+0000 7fe17effa000 20 client.6288 populate_metadata read hostname 'smithi016' 2024-03-31T15:06:32.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.419+0000 7fe17effa000 1 --2- 172.21.15.16:0/1447422208 >> [v2:172.21.15.16:6834/2613117528,v1:172.21.15.16:6835/2613117528] conn(0x55dd9ec8f3d0 0x55dd9ec8f7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:32.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.419+0000 7fe17effa000 4 client.6288 mds_command: new command op to 6150 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-31T15:06:32.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.419+0000 7fe17effa000 1 -- 172.21.15.16:0/1447422208 --> [v2:172.21.15.16:6834/2613117528,v1:172.21.15.16:6835/2613117528] -- command(tid 0: {"prefix": "status"}) -- 0x55dd9ea30e90 con 0x55dd9ec8f3d0 2024-03-31T15:06:32.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.419+0000 7fe179200640 1 --2- 172.21.15.16:0/1447422208 >> [v2:172.21.15.16:6834/2613117528,v1:172.21.15.16:6835/2613117528] conn(0x55dd9ec8f3d0 0x55dd9ec8f7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:32.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.423+0000 7fe179200640 1 --2- 172.21.15.16:0/1447422208 >> [v2:172.21.15.16:6834/2613117528,v1:172.21.15.16:6835/2613117528] conn(0x55dd9ec8f3d0 0x55dd9ec8f7b0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ff3cfdb6fd79d3d8 server_cookie=7b3f3bc77be024d5 in_seq=0 out_seq=0 2024-03-31T15:06:32.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.423+0000 7fe15a000640 10 client.6288 ms_handle_connect on v2:172.21.15.16:6834/2613117528 2024-03-31T15:06:32.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.423+0000 7fe15a000640 1 -- 172.21.15.16:0/1447422208 <== mds.0 v2:172.21.15.16:6834/2613117528 1 ==== command_reply(tid 0: 0 ) ==== 8+0+325 (crc 0 0 0) 0x55dd9ea30e90 con 0x55dd9ec8f3d0 2024-03-31T15:06:32.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.423+0000 7fe15a000640 10 client.6288 handle_command_reply: tid=0 2024-03-31T15:06:32.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.423+0000 7fe17effa000 1 client.6288 shutdown 2024-03-31T15:06:32.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.423+0000 7fe17effa000 1 -- 172.21.15.16:0/1447422208 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dd9ec1ddc0 msgr2=0x55dd9ec8ee90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:32.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.423+0000 7fe17effa000 1 --2- 172.21.15.16:0/1447422208 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dd9ec1ddc0 0x55dd9ec8ee90 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7fe16c019640 tx=0x7fe16c11fd30 comp rx=0 tx=0).stop 2024-03-31T15:06:32.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.423+0000 7fe17effa000 1 -- 172.21.15.16:0/1447422208 shutdown_connections 2024-03-31T15:06:32.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.423+0000 7fe17effa000 1 --2- 172.21.15.16:0/1447422208 >> [v2:172.21.15.16:6834/2613117528,v1:172.21.15.16:6835/2613117528] conn(0x55dd9ec8f3d0 0x55dd9ec8f7b0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:32.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.423+0000 7fe17effa000 1 --2- 172.21.15.16:0/1447422208 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dd9ec1ddc0 0x55dd9ec8ee90 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:32.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.423+0000 7fe17effa000 1 --2- 172.21.15.16:0/1447422208 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55dd9ec1d4a0 0x55dd9ec92df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:32.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.423+0000 7fe17effa000 1 --2- 172.21.15.16:0/1447422208 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55dd9ebd10b0 0x55dd9ec928b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:32.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.423+0000 7fe17effa000 1 -- 172.21.15.16:0/1447422208 >> 172.21.15.16:0/1447422208 conn(0x55dd9eabbb80 msgr2=0x55dd9ebede50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:06:32.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.423+0000 7fe15a000640 0 client.6288 ms_handle_reset on v2:172.21.15.16:6834/2613117528 2024-03-31T15:06:32.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.423+0000 7fe17effa000 1 -- 172.21.15.16:0/1447422208 shutdown_connections 2024-03-31T15:06:32.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.423+0000 7fe17effa000 1 -- 172.21.15.16:0/1447422208 wait complete. 2024-03-31T15:06:32.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.423+0000 7fe17effa000 20 client.6288 trim_cache size 0 max 16384 2024-03-31T15:06:32.422 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-03-31T15:06:32.422 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_fsid": "286eb5da-fb47-4550-b290-d9519c13b9a5", 2024-03-31T15:06:32.422 INFO:teuthology.orchestra.run.smithi016.stdout: "whoami": 0, 2024-03-31T15:06:32.422 INFO:teuthology.orchestra.run.smithi016.stdout: "id": 6150, 2024-03-31T15:06:32.422 INFO:teuthology.orchestra.run.smithi016.stdout: "want_state": "up:active", 2024-03-31T15:06:32.422 INFO:teuthology.orchestra.run.smithi016.stdout: "state": "up:active", 2024-03-31T15:06:32.423 INFO:teuthology.orchestra.run.smithi016.stdout: "fs_name": "cephfs", 2024-03-31T15:06:32.423 INFO:teuthology.orchestra.run.smithi016.stdout: "rank_uptime": 6.1602115639999999, 2024-03-31T15:06:32.423 INFO:teuthology.orchestra.run.smithi016.stdout: "mdsmap_epoch": 34, 2024-03-31T15:06:32.423 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch": 56, 2024-03-31T15:06:32.423 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch_barrier": 56, 2024-03-31T15:06:32.423 INFO:teuthology.orchestra.run.smithi016.stdout: "uptime": 11.012893402 2024-03-31T15:06:32.423 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-03-31T15:06:32.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.423+0000 7fe17bc00640 1 -- 172.21.15.16:0/2610020467 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fe154068380 msgr2=0x7fe15406a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:32.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.423+0000 7fe17bc00640 1 --2- 172.21.15.16:0/2610020467 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fe154068380 0x7fe15406a840 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fe17415e090 tx=0x7fe168009190 comp rx=0 tx=0).stop 2024-03-31T15:06:32.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.423+0000 7fe17bc00640 1 -- 172.21.15.16:0/2610020467 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe174145e40 msgr2=0x7fe17413c130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:32.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.423+0000 7fe17bc00640 1 --2- 172.21.15.16:0/2610020467 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe174145e40 0x7fe17413c130 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7fe16c0af040 tx=0x7fe16c0197d0 comp rx=0 tx=0).stop 2024-03-31T15:06:32.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.423+0000 7fe17bc00640 1 -- 172.21.15.16:0/2610020467 shutdown_connections 2024-03-31T15:06:32.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.423+0000 7fe17bc00640 1 --2- 172.21.15.16:0/2610020467 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fe154068380 0x7fe15406a840 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:32.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.423+0000 7fe17bc00640 1 --2- 172.21.15.16:0/2610020467 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe174145e40 0x7fe17413c130 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:32.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.423+0000 7fe17bc00640 1 --2- 172.21.15.16:0/2610020467 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fe1740ccc80 0x7fe174140190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:32.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.423+0000 7fe17bc00640 1 --2- 172.21.15.16:0/2610020467 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe174070030 0x7fe17413e400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:32.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.423+0000 7fe17bc00640 1 -- 172.21.15.16:0/2610020467 >> 172.21.15.16:0/2610020467 conn(0x7fe174067330 msgr2=0x7fe1740cff60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:06:32.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.423+0000 7fe17bc00640 1 -- 172.21.15.16:0/2610020467 shutdown_connections 2024-03-31T15:06:32.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.423+0000 7fe17bc00640 1 -- 172.21.15.16:0/2610020467 wait complete. 2024-03-31T15:06:32.436 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-31T15:06:32.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.583+0000 7fae45400640 1 Processor -- start 2024-03-31T15:06:32.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.583+0000 7fae45400640 1 -- start start 2024-03-31T15:06:32.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.583+0000 7fae45400640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fae4006fd80 0x7fae400d2720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:32.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.583+0000 7fae45400640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fae400d0920 con 0x7fae40145e40 2024-03-31T15:06:32.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.583+0000 7fae45400640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fae400d0c00 con 0x7fae4006f920 2024-03-31T15:06:32.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.583+0000 7fae45400640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7fae400d06d0 con 0x7fae4006fd80 2024-03-31T15:06:32.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.587+0000 7fae3e800640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fae40145e40 0x7fae400cc740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:56414/0 (socket says 172.21.15.16:56414) 2024-03-31T15:06:32.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.587+0000 7fae3e800640 1 -- 172.21.15.16:0/1700647652 learned_addr learned my addr 172.21.15.16:0/1700647652 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:06:32.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.587+0000 7fae3f200640 1 --2- 172.21.15.16:0/1700647652 >> v2:172.21.15.16:3300/0 conn(0x7fae4006fd80 0x7fae400d2720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:32.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.587+0000 7fae3de00640 1 -- 172.21.15.16:0/1700647652 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1901794379 0 0) 0x7fae400d0920 con 0x7fae40145e40 2024-03-31T15:06:32.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.587+0000 7fae3f200640 1 -- 172.21.15.16:0/1700647652 >> v1:172.21.15.89:6789/0 conn(0x7fae4006f920 legacy=0x7fae400d2010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:32.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.587+0000 7fae3f200640 1 -- 172.21.15.16:0/1700647652 >> v1:172.21.15.16:6789/0 conn(0x7fae40145e40 legacy=0x7fae400cc740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:32.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.587+0000 7fae3f200640 1 -- 172.21.15.16:0/1700647652 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae400d1810 con 0x7fae4006fd80 2024-03-31T15:06:32.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.587+0000 7fae3f200640 1 --2- 172.21.15.16:0/1700647652 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fae4006fd80 0x7fae400d2720 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7fae380097c0 tx=0x7fae3801c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca019660b46c3917 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:06:32.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.587+0000 7fae3de00640 1 -- 172.21.15.16:0/1700647652 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae380b1070 con 0x7fae4006fd80 2024-03-31T15:06:32.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.587+0000 7fae3de00640 1 -- 172.21.15.16:0/1700647652 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fae380ac5f0 con 0x7fae4006fd80 2024-03-31T15:06:32.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.587+0000 7fae3de00640 1 -- 172.21.15.16:0/1700647652 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae380ac9c0 con 0x7fae4006fd80 2024-03-31T15:06:32.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.591+0000 7fae45400640 1 -- 172.21.15.16:0/1700647652 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fae4006fd80 msgr2=0x7fae400d2720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:32.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.591+0000 7fae45400640 1 --2- 172.21.15.16:0/1700647652 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fae4006fd80 0x7fae400d2720 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7fae380097c0 tx=0x7fae3801c4a0 comp rx=0 tx=0).stop 2024-03-31T15:06:32.585 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.591+0000 7fae45400640 1 -- 172.21.15.16:0/1700647652 shutdown_connections 2024-03-31T15:06:32.585 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.591+0000 7fae45400640 1 --2- 172.21.15.16:0/1700647652 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fae4006fd80 0x7fae400d2720 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:32.585 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.591+0000 7fae45400640 1 -- 172.21.15.16:0/1700647652 >> 172.21.15.16:0/1700647652 conn(0x7fae40067330 msgr2=0x7fae40067740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:06:32.585 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.591+0000 7fae45400640 1 -- 172.21.15.16:0/1700647652 shutdown_connections 2024-03-31T15:06:32.585 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.591+0000 7fae45400640 1 -- 172.21.15.16:0/1700647652 wait complete. 2024-03-31T15:06:32.586 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.591+0000 7fae45400640 1 Processor -- start 2024-03-31T15:06:32.586 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.591+0000 7fae45400640 1 -- start start 2024-03-31T15:06:32.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.591+0000 7fae45400640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fae4006f920 0x7fae4015a5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:32.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.591+0000 7fae45400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fae4006fd80 0x7fae40157a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:32.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.591+0000 7fae45400640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fae40145e40 0x7fae40158a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:32.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.591+0000 7fae45400640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fae400d0ee0 con 0x7fae4006fd80 2024-03-31T15:06:32.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.591+0000 7fae45400640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fae400ce9e0 con 0x7fae40145e40 2024-03-31T15:06:32.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.591+0000 7fae45400640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fae400b50c0 con 0x7fae4006f920 2024-03-31T15:06:32.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.591+0000 7fae35e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fae4006fd80 0x7fae40157a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:32.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.591+0000 7fae35e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fae4006fd80 0x7fae40157a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49128/0 (socket says 172.21.15.16:49128) 2024-03-31T15:06:32.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.591+0000 7fae35e00640 1 -- 172.21.15.16:0/1638252529 learned_addr learned my addr 172.21.15.16:0/1638252529 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:06:32.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.591+0000 7fae3e800640 1 --2- 172.21.15.16:0/1638252529 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fae4006f920 0x7fae4015a5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:32.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.591+0000 7fae3f200640 1 --2- 172.21.15.16:0/1638252529 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fae40145e40 0x7fae40158a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:32.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.591+0000 7fae35e00640 1 -- 172.21.15.16:0/1638252529 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fae4006f920 msgr2=0x7fae4015a5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:32.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.591+0000 7fae35e00640 1 --2- 172.21.15.16:0/1638252529 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fae4006f920 0x7fae4015a5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:32.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.595+0000 7fae35e00640 1 -- 172.21.15.16:0/1638252529 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fae40145e40 msgr2=0x7fae40158a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:32.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.595+0000 7fae35e00640 1 --2- 172.21.15.16:0/1638252529 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fae40145e40 0x7fae40158a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:32.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.595+0000 7fae35e00640 1 -- 172.21.15.16:0/1638252529 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae4006e290 con 0x7fae4006fd80 2024-03-31T15:06:32.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.595+0000 7fae35e00640 1 --2- 172.21.15.16:0/1638252529 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fae4006fd80 0x7fae40157a20 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7fae2800c3a0 tx=0x7fae2800c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:06:32.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.595+0000 7fae37400640 1 -- 172.21.15.16:0/1638252529 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae280052d0 con 0x7fae4006fd80 2024-03-31T15:06:32.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.595+0000 7fae37400640 1 -- 172.21.15.16:0/1638252529 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fae28005470 con 0x7fae4006fd80 2024-03-31T15:06:32.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.595+0000 7fae37400640 1 -- 172.21.15.16:0/1638252529 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae28007260 con 0x7fae4006fd80 2024-03-31T15:06:32.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.595+0000 7fae45400640 1 -- 172.21.15.16:0/1638252529 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fae4015f400 con 0x7fae4006fd80 2024-03-31T15:06:32.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.595+0000 7fae45400640 1 -- 172.21.15.16:0/1638252529 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fae4006e570 con 0x7fae4006fd80 2024-03-31T15:06:32.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.595+0000 7fae45400640 1 -- 172.21.15.16:0/1638252529 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fae40069950 con 0x7fae4006fd80 2024-03-31T15:06:32.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.595+0000 7fae37400640 1 -- 172.21.15.16:0/1638252529 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fae28007400 con 0x7fae4006fd80 2024-03-31T15:06:32.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.595+0000 7fae37400640 1 --2- 172.21.15.16:0/1638252529 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fae1c068380 0x7fae1c06a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:32.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.595+0000 7fae37400640 1 -- 172.21.15.16:0/1638252529 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 5709+0+0 (secure 0 0 0) 0x7fae28085e60 con 0x7fae4006fd80 2024-03-31T15:06:32.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.599+0000 7fae3e800640 1 --2- 172.21.15.16:0/1638252529 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fae1c068380 0x7fae1c06a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:32.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.599+0000 7fae3e800640 1 --2- 172.21.15.16:0/1638252529 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fae1c068380 0x7fae1c06a840 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fae30007fd0 tx=0x7fae30007f20 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:06:32.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.607+0000 7fae37400640 1 -- 172.21.15.16:0/1638252529 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fae28055d30 con 0x7fae4006fd80 2024-03-31T15:06:32.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.807+0000 7fae45400640 1 -- 172.21.15.16:0/1638252529 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7fae400cf3c0 con 0x7fae4006fd80 2024-03-31T15:06:32.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.807+0000 7fae37400640 1 -- 172.21.15.16:0/1638252529 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v34) ==== 53+0+73 (secure 0 0 0) 0x7fae2805a030 con 0x7fae4006fd80 2024-03-31T15:06:32.802 INFO:teuthology.orchestra.run.smithi016.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-31T15:06:32.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.811+0000 7fae45400640 1 -- 172.21.15.16:0/1638252529 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fae1c068380 msgr2=0x7fae1c06a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:32.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.811+0000 7fae45400640 1 --2- 172.21.15.16:0/1638252529 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fae1c068380 0x7fae1c06a840 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fae30007fd0 tx=0x7fae30007f20 comp rx=0 tx=0).stop 2024-03-31T15:06:32.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.811+0000 7fae45400640 1 -- 172.21.15.16:0/1638252529 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fae4006fd80 msgr2=0x7fae40157a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:32.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.811+0000 7fae45400640 1 --2- 172.21.15.16:0/1638252529 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fae4006fd80 0x7fae40157a20 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7fae2800c3a0 tx=0x7fae2800c860 comp rx=0 tx=0).stop 2024-03-31T15:06:32.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.811+0000 7fae45400640 1 -- 172.21.15.16:0/1638252529 shutdown_connections 2024-03-31T15:06:32.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.811+0000 7fae45400640 1 --2- 172.21.15.16:0/1638252529 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fae1c068380 0x7fae1c06a840 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:32.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.811+0000 7fae45400640 1 --2- 172.21.15.16:0/1638252529 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fae40145e40 0x7fae40158a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:32.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.811+0000 7fae45400640 1 --2- 172.21.15.16:0/1638252529 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fae4006fd80 0x7fae40157a20 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:32.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.811+0000 7fae45400640 1 --2- 172.21.15.16:0/1638252529 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fae4006f920 0x7fae4015a5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:32.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.811+0000 7fae45400640 1 -- 172.21.15.16:0/1638252529 >> 172.21.15.16:0/1638252529 conn(0x7fae40067330 msgr2=0x7fae4015b6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:06:32.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.811+0000 7fae45400640 1 -- 172.21.15.16:0/1638252529 shutdown_connections 2024-03-31T15:06:32.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:32.811+0000 7fae45400640 1 -- 172.21.15.16:0/1638252529 wait complete. 2024-03-31T15:06:32.823 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-31T15:06:32.823 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-31T15:06:32.823 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi016.front.sepia.ceph.com 2024-03-31T15:06:32.823 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-31T15:06:32.823 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-31T15:06:32.823 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-31T15:06:32.823 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-31T15:06:32.823 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-31T15:06:32.824 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-31T15:06:32.824 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:06:32.824 DEBUG:teuthology.orchestra.run.smithi016:> ip addr 2024-03-31T15:06:32.834 INFO:teuthology.orchestra.run.smithi016.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-31T15:06:32.834 INFO:teuthology.orchestra.run.smithi016.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-31T15:06:32.834 INFO:teuthology.orchestra.run.smithi016.stdout: inet 127.0.0.1/8 scope host lo 2024-03-31T15:06:32.835 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2024-03-31T15:06:32.835 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 ::1/128 scope host 2024-03-31T15:06:32.835 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2024-03-31T15:06:32.835 INFO:teuthology.orchestra.run.smithi016.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-03-31T15:06:32.835 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:6b:fd:e0 brd ff:ff:ff:ff:ff:ff 2024-03-31T15:06:32.835 INFO:teuthology.orchestra.run.smithi016.stdout: altname enp7s0f0 2024-03-31T15:06:32.835 INFO:teuthology.orchestra.run.smithi016.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-03-31T15:06:32.835 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:6b:fd:e1 brd ff:ff:ff:ff:ff:ff 2024-03-31T15:06:32.835 INFO:teuthology.orchestra.run.smithi016.stdout: altname enp7s0f1 2024-03-31T15:06:32.835 INFO:teuthology.orchestra.run.smithi016.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-31T15:06:32.835 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:bd:15:64 brd ff:ff:ff:ff:ff:ff 2024-03-31T15:06:32.835 INFO:teuthology.orchestra.run.smithi016.stdout: altname enp1s0f0 2024-03-31T15:06:32.835 INFO:teuthology.orchestra.run.smithi016.stdout: inet 172.21.15.16/20 brd 172.21.15.255 scope global dynamic ens1f0 2024-03-31T15:06:32.835 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft 42288sec preferred_lft 42288sec 2024-03-31T15:06:32.836 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 fe80::ec4:7aff:febd:1564/64 scope link 2024-03-31T15:06:32.836 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2024-03-31T15:06:32.836 INFO:teuthology.orchestra.run.smithi016.stdout:5: ens1f1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-03-31T15:06:32.836 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:bd:15:65 brd ff:ff:ff:ff:ff:ff 2024-03-31T15:06:32.836 INFO:teuthology.orchestra.run.smithi016.stdout: altname enp1s0f1 2024-03-31T15:06:32.836 INFO:teuthology.orchestra.run.smithi016.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2024-03-31T15:06:32.836 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 02:42:d4:1f:7d:73 brd ff:ff:ff:ff:ff:ff 2024-03-31T15:06:32.836 INFO:teuthology.orchestra.run.smithi016.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2024-03-31T15:06:32.836 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2024-03-31T15:06:32.836 INFO:teuthology.orchestra.run.smithi016.stdout:7: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-31T15:06:32.836 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 5e:1f:84:6d:79:ae brd ff:ff:ff:ff:ff:ff 2024-03-31T15:06:32.836 INFO:teuthology.orchestra.run.smithi016.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-31T15:06:32.836 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2024-03-31T15:06:32.836 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 fe80::5c1f:84ff:fe6d:79ae/64 scope link 2024-03-31T15:06:32.836 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2024-03-31T15:06:32.837 INFO:teuthology.orchestra.run.smithi016.stdout:8: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-31T15:06:32.837 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0e:5c:f6:1f:82:2f brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-31T15:06:32.837 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 fe80::c5c:f6ff:fe1f:822f/64 scope link 2024-03-31T15:06:32.837 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2024-03-31T15:06:32.837 INFO:teuthology.orchestra.run.smithi016.stdout:9: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-31T15:06:32.837 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether c6:a6:3d:69:d0:db brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-31T15:06:32.837 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 fe80::c4a6:3dff:fe69:d0db/64 scope link 2024-03-31T15:06:32.837 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2024-03-31T15:06:32.837 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:06:32.837 DEBUG:teuthology.orchestra.run.smithi016:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-31T15:06:32.898 INFO:teuthology.orchestra.run.smithi016.stdout:1 2024-03-31T15:06:32.900 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:06:32.900 DEBUG:teuthology.orchestra.run.smithi016:> route 2024-03-31T15:06:32.958 INFO:teuthology.orchestra.run.smithi016.stdout:Kernel IP routing table 2024-03-31T15:06:32.958 INFO:teuthology.orchestra.run.smithi016.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-31T15:06:32.958 INFO:teuthology.orchestra.run.smithi016.stdout:default _gateway 0.0.0.0 UG 0 0 0 ens1f0 2024-03-31T15:06:32.958 INFO:teuthology.orchestra.run.smithi016.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2024-03-31T15:06:32.958 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 0 0 0 ens1f0 2024-03-31T15:06:32.958 INFO:teuthology.orchestra.run.smithi016.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-31T15:06:32.959 DEBUG:teuthology.orchestra.run.smithi016:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-31T15:06:32.959 DEBUG:teuthology.orchestra.run.smithi016:> set -e 2024-03-31T15:06:32.959 DEBUG:teuthology.orchestra.run.smithi016:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-03-31T15:06:32.959 DEBUG:teuthology.orchestra.run.smithi016:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-03-31T15:06:32.959 DEBUG:teuthology.orchestra.run.smithi016:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-03-31T15:06:32.959 DEBUG:teuthology.orchestra.run.smithi016:> ') 2024-03-31T15:06:33.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:33 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-31T15:06:33.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:33 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-31T15:06:33.101 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:06:33.101 DEBUG:teuthology.orchestra.run.smithi016:> ip netns list 2024-03-31T15:06:33.158 INFO:teuthology.orchestra.run.smithi016.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-31T15:06:33.158 INFO:teuthology.orchestra.run.smithi016.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-31T15:06:33.158 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-31T15:06:33.158 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:06:33.158 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip link set brx.0 up 2024-03-31T15:06:33.220 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-03-31T15:06:33.221 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-31T15:06:33.221 DEBUG:teuthology.orchestra.run.smithi016:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-31T15:06:33.273 INFO:teuthology.orchestra.run.smithi016.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-31T15:06:33.273 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-31T15:06:33.273 DEBUG:teuthology.orchestra.run.smithi016:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-31T15:06:33.328 DEBUG:teuthology.orchestra.run.smithi016:> sudo modprobe fuse 2024-03-31T15:06:33.393 DEBUG:teuthology.orchestra.run.smithi016:> cat /proc/self/mounts | awk '{print $2}' 2024-03-31T15:06:33.450 INFO:teuthology.orchestra.run.smithi016.stdout:/sys 2024-03-31T15:06:33.450 INFO:teuthology.orchestra.run.smithi016.stdout:/proc 2024-03-31T15:06:33.450 INFO:teuthology.orchestra.run.smithi016.stdout:/dev 2024-03-31T15:06:33.450 INFO:teuthology.orchestra.run.smithi016.stdout:/dev/pts 2024-03-31T15:06:33.450 INFO:teuthology.orchestra.run.smithi016.stdout:/run 2024-03-31T15:06:33.451 INFO:teuthology.orchestra.run.smithi016.stdout:/ 2024-03-31T15:06:33.451 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/kernel/security 2024-03-31T15:06:33.451 INFO:teuthology.orchestra.run.smithi016.stdout:/dev/shm 2024-03-31T15:06:33.451 INFO:teuthology.orchestra.run.smithi016.stdout:/run/lock 2024-03-31T15:06:33.451 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/cgroup 2024-03-31T15:06:33.451 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/pstore 2024-03-31T15:06:33.451 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/bpf 2024-03-31T15:06:33.451 INFO:teuthology.orchestra.run.smithi016.stdout:/proc/sys/fs/binfmt_misc 2024-03-31T15:06:33.451 INFO:teuthology.orchestra.run.smithi016.stdout:/dev/hugepages 2024-03-31T15:06:33.451 INFO:teuthology.orchestra.run.smithi016.stdout:/dev/mqueue 2024-03-31T15:06:33.451 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/kernel/debug 2024-03-31T15:06:33.451 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/kernel/tracing 2024-03-31T15:06:33.451 INFO:teuthology.orchestra.run.smithi016.stdout:/proc/fs/nfsd 2024-03-31T15:06:33.451 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/fuse/connections 2024-03-31T15:06:33.451 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/kernel/config 2024-03-31T15:06:33.451 INFO:teuthology.orchestra.run.smithi016.stdout:/run/credentials/systemd-sysusers.service 2024-03-31T15:06:33.452 INFO:teuthology.orchestra.run.smithi016.stdout:/run/rpc_pipefs 2024-03-31T15:06:33.452 INFO:teuthology.orchestra.run.smithi016.stdout:/run/user/1000 2024-03-31T15:06:33.452 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/ceph 2024-03-31T15:06:33.452 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/ceph/osd/ceph-0 2024-03-31T15:06:33.452 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/ceph/osd/ceph-1 2024-03-31T15:06:33.452 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/ceph/osd/ceph-2 2024-03-31T15:06:33.452 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/ceph/osd/ceph-3 2024-03-31T15:06:33.452 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns 2024-03-31T15:06:33.452 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-31T15:06:33.452 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-31T15:06:33.452 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-31T15:06:33.452 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-31T15:06:33.453 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:06:33.453 DEBUG:teuthology.orchestra.run.smithi016:> ls /sys/fs/fuse/connections 2024-03-31T15:06:33.504 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-31T15:06:33.505 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-03-31T15:06:33.552 DEBUG:teuthology.orchestra.run.smithi016:> sudo modprobe fuse 2024-03-31T15:06:33.572 DEBUG:teuthology.orchestra.run.smithi016:> cat /proc/self/mounts | awk '{print $2}' 2024-03-31T15:06:33.625 INFO:teuthology.orchestra.run.smithi016.stdout:/sys 2024-03-31T15:06:33.625 INFO:teuthology.orchestra.run.smithi016.stdout:/proc 2024-03-31T15:06:33.625 INFO:teuthology.orchestra.run.smithi016.stdout:/dev 2024-03-31T15:06:33.625 INFO:teuthology.orchestra.run.smithi016.stdout:/dev/pts 2024-03-31T15:06:33.625 INFO:teuthology.orchestra.run.smithi016.stdout:/run 2024-03-31T15:06:33.625 INFO:teuthology.orchestra.run.smithi016.stdout:/ 2024-03-31T15:06:33.625 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/kernel/security 2024-03-31T15:06:33.626 INFO:teuthology.orchestra.run.smithi016.stdout:/dev/shm 2024-03-31T15:06:33.626 INFO:teuthology.orchestra.run.smithi016.stdout:/run/lock 2024-03-31T15:06:33.626 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/cgroup 2024-03-31T15:06:33.626 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/pstore 2024-03-31T15:06:33.626 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/bpf 2024-03-31T15:06:33.626 INFO:teuthology.orchestra.run.smithi016.stdout:/proc/sys/fs/binfmt_misc 2024-03-31T15:06:33.626 INFO:teuthology.orchestra.run.smithi016.stdout:/dev/hugepages 2024-03-31T15:06:33.626 INFO:teuthology.orchestra.run.smithi016.stdout:/dev/mqueue 2024-03-31T15:06:33.626 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/kernel/debug 2024-03-31T15:06:33.626 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/kernel/tracing 2024-03-31T15:06:33.626 INFO:teuthology.orchestra.run.smithi016.stdout:/proc/fs/nfsd 2024-03-31T15:06:33.626 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/fuse/connections 2024-03-31T15:06:33.626 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/kernel/config 2024-03-31T15:06:33.627 INFO:teuthology.orchestra.run.smithi016.stdout:/run/credentials/systemd-sysusers.service 2024-03-31T15:06:33.627 INFO:teuthology.orchestra.run.smithi016.stdout:/run/rpc_pipefs 2024-03-31T15:06:33.627 INFO:teuthology.orchestra.run.smithi016.stdout:/run/user/1000 2024-03-31T15:06:33.627 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/ceph 2024-03-31T15:06:33.627 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/ceph/osd/ceph-0 2024-03-31T15:06:33.627 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/ceph/osd/ceph-1 2024-03-31T15:06:33.627 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/ceph/osd/ceph-2 2024-03-31T15:06:33.627 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/ceph/osd/ceph-3 2024-03-31T15:06:33.627 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns 2024-03-31T15:06:33.627 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-31T15:06:33.627 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-31T15:06:33.628 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-31T15:06:33.628 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-31T15:06:33.629 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:06:33.629 DEBUG:teuthology.orchestra.run.smithi016:> ls /sys/fs/fuse/connections 2024-03-31T15:06:33.647 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi016.stderr:2024-03-31T15:06:33.651+0000 7f001f853580 -1 init, newargv = 0x55836bdb5c20 newargc=15 2024-03-31T15:06:33.647 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi016.stderr:ceph-fuse[39791]: starting ceph client 2024-03-31T15:06:33.655 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi016.stderr:ceph-fuse[39791]: starting fuse 2024-03-31T15:06:33.656 INFO:teuthology.orchestra.run.smithi016.stdout:43 2024-03-31T15:06:33.656 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [43] 2024-03-31T15:06:33.657 DEBUG:teuthology.orchestra.run.smithi016:> sudo stdin-killer -- python3 -c ' 2024-03-31T15:06:33.657 DEBUG:teuthology.orchestra.run.smithi016:> import glob 2024-03-31T15:06:33.657 DEBUG:teuthology.orchestra.run.smithi016:> import re 2024-03-31T15:06:33.657 DEBUG:teuthology.orchestra.run.smithi016:> import os 2024-03-31T15:06:33.657 DEBUG:teuthology.orchestra.run.smithi016:> import subprocess 2024-03-31T15:06:33.657 DEBUG:teuthology.orchestra.run.smithi016:> 2024-03-31T15:06:33.657 DEBUG:teuthology.orchestra.run.smithi016:> def _find_admin_socket(client_name): 2024-03-31T15:06:33.657 DEBUG:teuthology.orchestra.run.smithi016:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-03-31T15:06:33.657 DEBUG:teuthology.orchestra.run.smithi016:> files = glob.glob(asok_path) 2024-03-31T15:06:33.657 DEBUG:teuthology.orchestra.run.smithi016:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-03-31T15:06:33.658 DEBUG:teuthology.orchestra.run.smithi016:> 2024-03-31T15:06:33.658 DEBUG:teuthology.orchestra.run.smithi016:> # Given a non-glob path, it better be there 2024-03-31T15:06:33.658 DEBUG:teuthology.orchestra.run.smithi016:> if "*" not in asok_path: 2024-03-31T15:06:33.658 DEBUG:teuthology.orchestra.run.smithi016:> assert(len(files) == 1) 2024-03-31T15:06:33.658 DEBUG:teuthology.orchestra.run.smithi016:> return files[0] 2024-03-31T15:06:33.658 DEBUG:teuthology.orchestra.run.smithi016:> 2024-03-31T15:06:33.658 DEBUG:teuthology.orchestra.run.smithi016:> for f in files: 2024-03-31T15:06:33.658 DEBUG:teuthology.orchestra.run.smithi016:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-31T15:06:33.658 DEBUG:teuthology.orchestra.run.smithi016:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-31T15:06:33.658 DEBUG:teuthology.orchestra.run.smithi016:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-31T15:06:33.658 DEBUG:teuthology.orchestra.run.smithi016:> contents = proc_f.read() 2024-03-31T15:06:33.658 DEBUG:teuthology.orchestra.run.smithi016:> if mountpoint in contents: 2024-03-31T15:06:33.658 DEBUG:teuthology.orchestra.run.smithi016:> return f 2024-03-31T15:06:33.658 DEBUG:teuthology.orchestra.run.smithi016:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-31T15:06:33.659 DEBUG:teuthology.orchestra.run.smithi016:> 2024-03-31T15:06:33.659 DEBUG:teuthology.orchestra.run.smithi016:> print(_find_admin_socket("client.0")) 2024-03-31T15:06:33.659 DEBUG:teuthology.orchestra.run.smithi016:> ' 2024-03-31T15:06:33.787 INFO:teuthology.orchestra.run.smithi016.stdout:/var/run/ceph/ceph-client.0.39791.asok 2024-03-31T15:06:33.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:33 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-31T15:06:33.799 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.39791.asok 2024-03-31T15:06:33.799 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:06:33.799 DEBUG:teuthology.orchestra.run.smithi016:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.39791.asok status 2024-03-31T15:06:33.972 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-03-31T15:06:33.972 INFO:teuthology.orchestra.run.smithi016.stdout: "metadata": { 2024-03-31T15:06:33.972 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph_sha1": "d2271579d5e735cfd6d436b8349c004f06527cb0", 2024-03-31T15:06:33.972 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph_version": "ceph version 19.0.0-2574-gd2271579 (d2271579d5e735cfd6d436b8349c004f06527cb0) squid (dev)", 2024-03-31T15:06:33.972 INFO:teuthology.orchestra.run.smithi016.stdout: "entity_id": "0", 2024-03-31T15:06:33.972 INFO:teuthology.orchestra.run.smithi016.stdout: "hostname": "smithi016", 2024-03-31T15:06:33.972 INFO:teuthology.orchestra.run.smithi016.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-03-31T15:06:33.972 INFO:teuthology.orchestra.run.smithi016.stdout: "pid": "39791", 2024-03-31T15:06:33.972 INFO:teuthology.orchestra.run.smithi016.stdout: "root": "/" 2024-03-31T15:06:33.973 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-31T15:06:33.973 INFO:teuthology.orchestra.run.smithi016.stdout: "dentry_count": 0, 2024-03-31T15:06:33.973 INFO:teuthology.orchestra.run.smithi016.stdout: "dentry_pinned_count": 0, 2024-03-31T15:06:33.973 INFO:teuthology.orchestra.run.smithi016.stdout: "id": 6303, 2024-03-31T15:06:33.973 INFO:teuthology.orchestra.run.smithi016.stdout: "inst": { 2024-03-31T15:06:33.973 INFO:teuthology.orchestra.run.smithi016.stdout: "name": { 2024-03-31T15:06:33.973 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "client", 2024-03-31T15:06:33.973 INFO:teuthology.orchestra.run.smithi016.stdout: "num": 6303 2024-03-31T15:06:33.973 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-31T15:06:33.973 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": { 2024-03-31T15:06:33.973 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-03-31T15:06:33.973 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "192.168.0.1:0", 2024-03-31T15:06:33.973 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2179632587 2024-03-31T15:06:33.974 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-31T15:06:33.974 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-31T15:06:33.974 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": { 2024-03-31T15:06:33.974 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-03-31T15:06:33.974 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "192.168.0.1:0", 2024-03-31T15:06:33.974 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2179632587 2024-03-31T15:06:33.974 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-31T15:06:33.974 INFO:teuthology.orchestra.run.smithi016.stdout: "inst_str": "client.6303 192.168.0.1:0/2179632587", 2024-03-31T15:06:33.974 INFO:teuthology.orchestra.run.smithi016.stdout: "addr_str": "192.168.0.1:0/2179632587", 2024-03-31T15:06:33.974 INFO:teuthology.orchestra.run.smithi016.stdout: "inode_count": 1, 2024-03-31T15:06:33.974 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_epoch": 34, 2024-03-31T15:06:33.974 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_epoch": 56, 2024-03-31T15:06:33.974 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_epoch_barrier": 0, 2024-03-31T15:06:33.974 INFO:teuthology.orchestra.run.smithi016.stdout: "blocklisted": false, 2024-03-31T15:06:33.975 INFO:teuthology.orchestra.run.smithi016.stdout: "fs_name": "cephfs" 2024-03-31T15:06:33.975 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-03-31T15:06:33.988 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:06:33.988 DEBUG:teuthology.orchestra.run.smithi016:> stat --file-system '--printf=%T 2024-03-31T15:06:33.988 DEBUG:teuthology.orchestra.run.smithi016:> ' -- /home/ubuntu/cephtest/mnt.0 2024-03-31T15:06:33.998 INFO:teuthology.orchestra.run.smithi016.stdout:fuseblk 2024-03-31T15:06:33.998 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-03-31T15:06:33.998 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:06:33.999 DEBUG:teuthology.orchestra.run.smithi016:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-31T15:06:34.059 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:06:34.059 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir dir_test) 2024-03-31T15:06:34.115 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:06:34.115 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec touch dir_test/file) 2024-03-31T15:06:34.168 DEBUG:teuthology.orchestra.run.smithi016:> stdin-killer -- python3 -c ' 2024-03-31T15:06:34.169 DEBUG:teuthology.orchestra.run.smithi016:> import os 2024-03-31T15:06:34.169 DEBUG:teuthology.orchestra.run.smithi016:> import stat 2024-03-31T15:06:34.169 DEBUG:teuthology.orchestra.run.smithi016:> 2024-03-31T15:06:34.169 DEBUG:teuthology.orchestra.run.smithi016:> print(os.stat("/home/ubuntu/cephtest/mnt.0/dir_test/file").st_ino) 2024-03-31T15:06:34.169 DEBUG:teuthology.orchestra.run.smithi016:> ' 2024-03-31T15:06:34.278 INFO:teuthology.orchestra.run.smithi016.stdout:1099511627777 2024-03-31T15:06:34.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:34 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-31T15:06:34.289 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-31T15:06:34.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:34.475+0000 7f8895800640 1 Processor -- start 2024-03-31T15:06:34.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:34.479+0000 7f8895800640 1 -- start start 2024-03-31T15:06:34.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:34.479+0000 7f8895800640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f8890156610 0x7f889015d200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:34.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:34.479+0000 7f8895800640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f889006cd40 0x7f88900ccf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:34.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:34.479+0000 7f8895800640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f88900cd660 con 0x7f88900afb80 2024-03-31T15:06:34.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:34.479+0000 7f8895800640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f88900cd940 con 0x7f8890156610 2024-03-31T15:06:34.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:34.479+0000 7f8895800640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f889015c720 con 0x7f889006cd40 2024-03-31T15:06:34.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:34.479+0000 7f8887e00640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f88900afb80 0x7f889006c630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:56420/0 (socket says 172.21.15.16:56420) 2024-03-31T15:06:34.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:34.479+0000 7f8887e00640 1 -- 172.21.15.16:0/4156624112 learned_addr learned my addr 172.21.15.16:0/4156624112 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:06:34.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:34.479+0000 7f888fe00640 1 --2- 172.21.15.16:0/4156624112 >> v2:172.21.15.89:3300/0 conn(0x7f8890156610 0x7f889015d200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:34.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:34.479+0000 7f8894e00640 1 --2- 172.21.15.16:0/4156624112 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f889006cd40 0x7f88900ccf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:34.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:34.479+0000 7f888f400640 1 -- 172.21.15.16:0/4156624112 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1715052122 0 0) 0x7f88900cd660 con 0x7f88900afb80 2024-03-31T15:06:34.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:34.479+0000 7f888f400640 1 -- 172.21.15.16:0/4156624112 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8874003530 con 0x7f88900afb80 2024-03-31T15:06:34.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:34.479+0000 7f888fe00640 1 -- 172.21.15.16:0/4156624112 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f889006cd40 msgr2=0x7f88900ccf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:06:34.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:34.479+0000 7f888fe00640 1 --2- 172.21.15.16:0/4156624112 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f889006cd40 0x7f88900ccf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:34.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:34.479+0000 7f888fe00640 1 -- 172.21.15.16:0/4156624112 >> v1:172.21.15.16:6789/0 conn(0x7f88900afb80 legacy=0x7f889006c630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:34.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:34.479+0000 7f888fe00640 1 -- 172.21.15.16:0/4156624112 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f889015ca00 con 0x7f8890156610 2024-03-31T15:06:34.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:34.479+0000 7f8894e00640 1 --2- 172.21.15.16:0/4156624112 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f889006cd40 0x7f88900ccf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:06:34.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:34.483+0000 7f888fe00640 1 --2- 172.21.15.16:0/4156624112 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8890156610 0x7f889015d200 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f88800097c0 tx=0x7f888001c4a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=11e43091f0d11435 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:06:34.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:34.483+0000 7f888f400640 1 -- 172.21.15.16:0/4156624112 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88800b1070 con 0x7f8890156610 2024-03-31T15:06:34.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:34.483+0000 7f888f400640 1 -- 172.21.15.16:0/4156624112 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f88800ac410 con 0x7f8890156610 2024-03-31T15:06:34.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:34.483+0000 7f888f400640 1 -- 172.21.15.16:0/4156624112 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88800ac7e0 con 0x7f8890156610 2024-03-31T15:06:34.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:34.483+0000 7f8895800640 1 -- 172.21.15.16:0/4156624112 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8890156610 msgr2=0x7f889015d200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:34.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:34.483+0000 7f8895800640 1 --2- 172.21.15.16:0/4156624112 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8890156610 0x7f889015d200 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f88800097c0 tx=0x7f888001c4a0 comp rx=0 tx=0).stop 2024-03-31T15:06:34.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:34.483+0000 7f8895800640 1 -- 172.21.15.16:0/4156624112 shutdown_connections 2024-03-31T15:06:34.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:34.483+0000 7f8895800640 1 --2- 172.21.15.16:0/4156624112 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f889006cd40 0x7f88900ccf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:34.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:34.483+0000 7f8895800640 1 --2- 172.21.15.16:0/4156624112 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8890156610 0x7f889015d200 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:34.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:34.483+0000 7f8895800640 1 -- 172.21.15.16:0/4156624112 >> 172.21.15.16:0/4156624112 conn(0x7f8890071c20 msgr2=0x7f8890072030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:06:34.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:34.483+0000 7f8895800640 1 -- 172.21.15.16:0/4156624112 shutdown_connections 2024-03-31T15:06:34.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:34.483+0000 7f8895800640 1 -- 172.21.15.16:0/4156624112 wait complete. 2024-03-31T15:06:34.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:34.483+0000 7f8895800640 1 Processor -- start 2024-03-31T15:06:34.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:34.487+0000 7f8895800640 1 -- start start 2024-03-31T15:06:34.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:34.487+0000 7f8895800640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f889006cd40 0x7f889015de00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:34.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:34.487+0000 7f8895800640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f88900afb80 0x7f88901fa5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:34.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:34.487+0000 7f8895800640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8890156610 0x7f88901faaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:34.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:34.487+0000 7f8895800640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f88900cfab0 con 0x7f88900afb80 2024-03-31T15:06:34.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:34.487+0000 7f8895800640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f88900cf930 con 0x7f8890156610 2024-03-31T15:06:34.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:34.487+0000 7f8895800640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f88900cfc30 con 0x7f889006cd40 2024-03-31T15:06:34.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:34.487+0000 7f8887e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f88900afb80 0x7f88901fa5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:34.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:34.487+0000 7f8887e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f88900afb80 0x7f88901fa5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49144/0 (socket says 172.21.15.16:49144) 2024-03-31T15:06:34.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:34.487+0000 7f8887e00640 1 -- 172.21.15.16:0/1541558720 learned_addr learned my addr 172.21.15.16:0/1541558720 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:06:34.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:34.487+0000 7f8894e00640 1 --2- 172.21.15.16:0/1541558720 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8890156610 0x7f88901faaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:34.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:34.487+0000 7f888fe00640 1 --2- 172.21.15.16:0/1541558720 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f889006cd40 0x7f889015de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:34.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:34.487+0000 7f8887e00640 1 -- 172.21.15.16:0/1541558720 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f889006cd40 msgr2=0x7f889015de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:34.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:34.487+0000 7f8887e00640 1 --2- 172.21.15.16:0/1541558720 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f889006cd40 0x7f889015de00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:34.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:34.487+0000 7f8887e00640 1 -- 172.21.15.16:0/1541558720 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8890156610 msgr2=0x7f88901faaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:34.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:34.487+0000 7f8887e00640 1 --2- 172.21.15.16:0/1541558720 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8890156610 0x7f88901faaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:34.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:34.487+0000 7f8887e00640 1 -- 172.21.15.16:0/1541558720 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88900b40a0 con 0x7f88900afb80 2024-03-31T15:06:34.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:34.487+0000 7f8894e00640 1 --2- 172.21.15.16:0/1541558720 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8890156610 0x7f88901faaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:06:34.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:34.487+0000 7f8887e00640 1 --2- 172.21.15.16:0/1541558720 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f88900afb80 0x7f88901fa5b0 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f887800c480 tx=0x7f887800c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:06:34.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:34.487+0000 7f888cc00640 1 -- 172.21.15.16:0/1541558720 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8878017070 con 0x7f88900afb80 2024-03-31T15:06:34.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:34.487+0000 7f888cc00640 1 -- 172.21.15.16:0/1541558720 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8878005cf0 con 0x7f88900afb80 2024-03-31T15:06:34.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:34.487+0000 7f888cc00640 1 -- 172.21.15.16:0/1541558720 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8878007290 con 0x7f88900afb80 2024-03-31T15:06:34.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:34.487+0000 7f8895800640 1 -- 172.21.15.16:0/1541558720 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f88900b4380 con 0x7f88900afb80 2024-03-31T15:06:34.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:34.487+0000 7f8895800640 1 -- 172.21.15.16:0/1541558720 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f88900b4940 con 0x7f88900afb80 2024-03-31T15:06:34.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:34.487+0000 7f8895800640 1 -- 172.21.15.16:0/1541558720 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f889004f8c0 con 0x7f88900afb80 2024-03-31T15:06:34.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:34.491+0000 7f888cc00640 1 -- 172.21.15.16:0/1541558720 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f887801a070 con 0x7f88900afb80 2024-03-31T15:06:34.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:34.491+0000 7f888cc00640 1 --2- 172.21.15.16:0/1541558720 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f886c068380 0x7f886c06a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:34.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:34.491+0000 7f888cc00640 1 -- 172.21.15.16:0/1541558720 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 5709+0+0 (secure 0 0 0) 0x7f8878085d80 con 0x7f88900afb80 2024-03-31T15:06:34.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:34.491+0000 7f888fe00640 1 --2- 172.21.15.16:0/1541558720 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f886c068380 0x7f886c06a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:34.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:34.491+0000 7f888fe00640 1 --2- 172.21.15.16:0/1541558720 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f886c068380 0x7f886c06a840 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f88800ae000 tx=0x7f88800a6040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:06:34.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:34.495+0000 7f888cc00640 1 -- 172.21.15.16:0/1541558720 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f8878055ce0 con 0x7f88900afb80 2024-03-31T15:06:34.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:34.691+0000 7f8895800640 1 -- 172.21.15.16:0/1541558720 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8890152b50 con 0x7f88900afb80 2024-03-31T15:06:34.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:34.691+0000 7f888cc00640 1 -- 172.21.15.16:0/1541558720 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 34 v34) ==== 94+0+5181 (secure 0 0 0) 0x7f8878059fe0 con 0x7f88900afb80 2024-03-31T15:06:34.688 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:06:34.688 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":34,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5579,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6833/3721971977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":3721971977},{"type":"v1","addr":"172.21.15.16:6833","nonce":3721971977}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":29},{"gid":6109,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6836/1166206553","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":1166206553},{"type":"v1","addr":"172.21.15.89:6836","nonce":1166206553}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28},{"gid":6112,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/1826113083","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6835","nonce":1826113083},{"type":"v1","addr":"172.21.15.89:6837","nonce":1826113083}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":29}],"filesystems":[{"mdsmap":{"epoch":34,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-31T15:06:25.665001+0000","modified":"2024-03-31T15:06:28.707528+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6150},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6150":{"gid":6150,"name":"c","rank":0,"incarnation":30,"state":"up:active","state_seq":3,"addr":"172.21.15.16:6835/2613117528","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":2613117528},{"type":"v1","addr":"172.21.15.16:6835","nonce":2613117528}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6150,"qdb_cluster":[6150]},"id":4}]} 2024-03-31T15:06:34.688 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 34 2024-03-31T15:06:34.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:34.695+0000 7f8895800640 1 -- 172.21.15.16:0/1541558720 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f886c068380 msgr2=0x7f886c06a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:34.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:34.695+0000 7f8895800640 1 --2- 172.21.15.16:0/1541558720 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f886c068380 0x7f886c06a840 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f88800ae000 tx=0x7f88800a6040 comp rx=0 tx=0).stop 2024-03-31T15:06:34.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:34.695+0000 7f8895800640 1 -- 172.21.15.16:0/1541558720 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f88900afb80 msgr2=0x7f88901fa5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:34.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:34.695+0000 7f8895800640 1 --2- 172.21.15.16:0/1541558720 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f88900afb80 0x7f88901fa5b0 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f887800c480 tx=0x7f887800c940 comp rx=0 tx=0).stop 2024-03-31T15:06:34.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:34.695+0000 7f8895800640 1 -- 172.21.15.16:0/1541558720 shutdown_connections 2024-03-31T15:06:34.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:34.695+0000 7f8895800640 1 --2- 172.21.15.16:0/1541558720 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f886c068380 0x7f886c06a840 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:34.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:34.695+0000 7f8895800640 1 --2- 172.21.15.16:0/1541558720 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8890156610 0x7f88901faaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:34.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:34.695+0000 7f8895800640 1 --2- 172.21.15.16:0/1541558720 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f88900afb80 0x7f88901fa5b0 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:34.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:34.695+0000 7f8895800640 1 --2- 172.21.15.16:0/1541558720 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f889006cd40 0x7f889015de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:34.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:34.695+0000 7f8895800640 1 -- 172.21.15.16:0/1541558720 >> 172.21.15.16:0/1541558720 conn(0x7f8890071c20 msgr2=0x7f889006b620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:06:34.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:34.695+0000 7f8895800640 1 -- 172.21.15.16:0/1541558720 shutdown_connections 2024-03-31T15:06:34.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:34.695+0000 7f8895800640 1 -- 172.21.15.16:0/1541558720 wait complete. 2024-03-31T15:06:34.712 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json flush journal 2024-03-31T15:06:34.887 INFO:teuthology.orchestra.run.smithi016.stdout:{"message":"","return_code":0} 2024-03-31T15:06:34.887 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2024-03-31T15:06:34.887 DEBUG:teuthology.orchestra.run.smithi016:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_data getxattr 10000000001.00000000 parent 2024-03-31T15:06:34.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:34.939+0000 7f26fe5a0080 1 Processor -- start 2024-03-31T15:06:34.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:34.939+0000 7f26fe5a0080 1 -- start start 2024-03-31T15:06:34.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:34.943+0000 7f26fe5a0080 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d241a02260 0x55d241a92d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:34.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:34.943+0000 7f26fe5a0080 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d241a079f0 0x55d241a8dac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:34.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:34.943+0000 7f26fe5a0080 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55d241a93ab0 0x55d241a913e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:34.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:34.943+0000 7f26fe5a0080 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55d2419ee330 con 0x55d241a079f0 2024-03-31T15:06:34.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:34.943+0000 7f26fe5a0080 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55d2419624b0 con 0x55d241a02260 2024-03-31T15:06:34.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:34.943+0000 7f26fe5a0080 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x55d2419ee4b0 con 0x55d241a93ab0 2024-03-31T15:06:34.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:34.943+0000 7f26fae00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d241a079f0 0x55d241a8dac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:34.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:34.943+0000 7f26fae00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d241a079f0 0x55d241a8dac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49166/0 (socket says 172.21.15.16:49166) 2024-03-31T15:06:34.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:34.943+0000 7f26fae00640 1 -- 172.21.15.16:0/2972673674 learned_addr learned my addr 172.21.15.16:0/2972673674 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:06:34.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:34.943+0000 7f26fc200640 1 --2- 172.21.15.16:0/2972673674 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55d241a93ab0 0x55d241a913e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:34.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:34.943+0000 7f26fb800640 1 --2- 172.21.15.16:0/2972673674 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d241a02260 0x55d241a92d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:34.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:34.943+0000 7f26fae00640 1 -- 172.21.15.16:0/2972673674 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55d241a93ab0 msgr2=0x55d241a913e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:06:34.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:34.943+0000 7f26fae00640 1 --2- 172.21.15.16:0/2972673674 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55d241a93ab0 0x55d241a913e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:34.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:34.943+0000 7f26fae00640 1 -- 172.21.15.16:0/2972673674 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d241a02260 msgr2=0x55d241a92d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:06:34.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:34.943+0000 7f26fae00640 1 --2- 172.21.15.16:0/2972673674 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d241a02260 0x55d241a92d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:34.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:34.943+0000 7f26fae00640 1 -- 172.21.15.16:0/2972673674 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d241a96be0 con 0x55d241a079f0 2024-03-31T15:06:34.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:34.943+0000 7f26fae00640 1 --2- 172.21.15.16:0/2972673674 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d241a079f0 0x55d241a8dac0 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f26e800b2d0 tx=0x7f26e800b790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d03e3192855bb1f1 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:06:34.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:34.943+0000 7f26f9a00640 1 -- 172.21.15.16:0/2972673674 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26e8013030 con 0x55d241a079f0 2024-03-31T15:06:34.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:34.943+0000 7f26f9a00640 1 -- 172.21.15.16:0/2972673674 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f26e800f620 con 0x55d241a079f0 2024-03-31T15:06:34.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:34.943+0000 7f26f9a00640 1 -- 172.21.15.16:0/2972673674 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26e800a040 con 0x55d241a079f0 2024-03-31T15:06:34.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:34.943+0000 7f26fe5a0080 1 -- 172.21.15.16:0/2972673674 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d241a079f0 msgr2=0x55d241a8dac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:34.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:34.943+0000 7f26fe5a0080 1 --2- 172.21.15.16:0/2972673674 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d241a079f0 0x55d241a8dac0 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f26e800b2d0 tx=0x7f26e800b790 comp rx=0 tx=0).stop 2024-03-31T15:06:34.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:34.943+0000 7f26fe5a0080 1 -- 172.21.15.16:0/2972673674 shutdown_connections 2024-03-31T15:06:34.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:34.943+0000 7f26fe5a0080 1 --2- 172.21.15.16:0/2972673674 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55d241a93ab0 0x55d241a913e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:34.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:34.943+0000 7f26fe5a0080 1 --2- 172.21.15.16:0/2972673674 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d241a079f0 0x55d241a8dac0 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:34.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:34.943+0000 7f26fe5a0080 1 --2- 172.21.15.16:0/2972673674 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d241a02260 0x55d241a92d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:34.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:34.943+0000 7f26fe5a0080 1 -- 172.21.15.16:0/2972673674 >> 172.21.15.16:0/2972673674 conn(0x55d241a050b0 msgr2=0x55d241a90870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:06:34.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:34.943+0000 7f26fe5a0080 1 -- 172.21.15.16:0/2972673674 shutdown_connections 2024-03-31T15:06:34.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:34.943+0000 7f26fe5a0080 1 -- 172.21.15.16:0/2972673674 wait complete. 2024-03-31T15:06:34.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:34.943+0000 7f26fe5a0080 1 Processor -- start 2024-03-31T15:06:34.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:34.943+0000 7f26fe5a0080 1 -- start start 2024-03-31T15:06:34.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:34.943+0000 7f26fe5a0080 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55d241a02260 0x55d2419bd520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:34.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:34.947+0000 7f26fe5a0080 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d241a079f0 0x55d2419e8470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:34.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:34.947+0000 7f26fe5a0080 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d241a93ab0 0x55d241a7a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:34.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:34.947+0000 7f26fe5a0080 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55d241a97120 con 0x55d241a079f0 2024-03-31T15:06:34.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:34.947+0000 7f26fe5a0080 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55d241a96e80 con 0x55d241a93ab0 2024-03-31T15:06:34.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:34.947+0000 7f26fe5a0080 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x55d241a973c0 con 0x55d241a02260 2024-03-31T15:06:34.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:34.947+0000 7f26fae00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d241a079f0 0x55d2419e8470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:34.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:34.947+0000 7f26fb800640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55d241a02260 0x55d2419bd520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:34.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:34.947+0000 7f26fae00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d241a079f0 0x55d2419e8470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49176/0 (socket says 172.21.15.16:49176) 2024-03-31T15:06:34.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:34.947+0000 7f26fae00640 1 -- 172.21.15.16:0/4207626682 learned_addr learned my addr 172.21.15.16:0/4207626682 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:06:34.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:34.947+0000 7f26fc200640 1 --2- 172.21.15.16:0/4207626682 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d241a93ab0 0x55d241a7a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:34.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:34.947+0000 7f26fae00640 1 -- 172.21.15.16:0/4207626682 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55d241a02260 msgr2=0x55d2419bd520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:34.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:34.947+0000 7f26fae00640 1 --2- 172.21.15.16:0/4207626682 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55d241a02260 0x55d2419bd520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:34.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:34.947+0000 7f26fae00640 1 -- 172.21.15.16:0/4207626682 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d241a93ab0 msgr2=0x55d241a7a8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:34.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:34.947+0000 7f26fae00640 1 --2- 172.21.15.16:0/4207626682 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d241a93ab0 0x55d241a7a8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:34.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:34.947+0000 7f26fae00640 1 -- 172.21.15.16:0/4207626682 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d241a8ebb0 con 0x55d241a079f0 2024-03-31T15:06:34.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:34.947+0000 7f26fae00640 1 --2- 172.21.15.16:0/4207626682 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d241a079f0 0x55d2419e8470 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f26e800fd80 tx=0x7f26e8001ff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:06:34.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:34.947+0000 7f26f3e00640 1 -- 172.21.15.16:0/4207626682 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26e8017810 con 0x55d241a079f0 2024-03-31T15:06:34.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:34.947+0000 7f26f3e00640 1 -- 172.21.15.16:0/4207626682 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f26e80179b0 con 0x55d241a079f0 2024-03-31T15:06:34.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:34.947+0000 7f26f3e00640 1 -- 172.21.15.16:0/4207626682 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26e8017da0 con 0x55d241a079f0 2024-03-31T15:06:34.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:34.947+0000 7f26fe5a0080 1 -- 172.21.15.16:0/4207626682 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55d241a8ee50 con 0x55d241a079f0 2024-03-31T15:06:34.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:34.947+0000 7f26fe5a0080 1 -- 172.21.15.16:0/4207626682 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d241a8f390 con 0x55d241a079f0 2024-03-31T15:06:34.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:34.951+0000 7f26f3e00640 1 -- 172.21.15.16:0/4207626682 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f26e8016070 con 0x55d241a079f0 2024-03-31T15:06:34.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:34.951+0000 7f26f3e00640 1 --2- 172.21.15.16:0/4207626682 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f26dc066d60 0x7f26dc069220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:34.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:34.951+0000 7f26f3e00640 1 -- 172.21.15.16:0/4207626682 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 5709+0+0 (secure 0 0 0) 0x7f26e8086200 con 0x55d241a079f0 2024-03-31T15:06:34.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:34.951+0000 7f26fe5a0080 1 --2- 172.21.15.16:0/4207626682 >> [v2:172.21.15.89:6810/50395527,v1:172.21.15.89:6814/50395527] conn(0x55d2419e89b0 0x55d241a8ffe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:34.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:34.951+0000 7f26fe5a0080 1 -- 172.21.15.16:0/4207626682 --> [v2:172.21.15.89:6810/50395527,v1:172.21.15.89:6814/50395527] -- osd_op(unknown.0.0:1 9.7 9:e2e07930:::10000000001.00000000:head [getxattr parent in=6b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e56) -- 0x55d241a90520 con 0x55d2419e89b0 2024-03-31T15:06:34.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:34.951+0000 7f26fb800640 1 --2- 172.21.15.16:0/4207626682 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f26dc066d60 0x7f26dc069220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:34.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:34.951+0000 7f26fc200640 1 --2- 172.21.15.16:0/4207626682 >> [v2:172.21.15.89:6810/50395527,v1:172.21.15.89:6814/50395527] conn(0x55d2419e89b0 0x55d241a8ffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:34.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:34.951+0000 7f26fc200640 1 --2- 172.21.15.16:0/4207626682 >> [v2:172.21.15.89:6810/50395527,v1:172.21.15.89:6814/50395527] conn(0x55d2419e89b0 0x55d241a8ffe0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:06:34.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:34.951+0000 7f26fb800640 1 --2- 172.21.15.16:0/4207626682 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f26dc066d60 0x7f26dc069220 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f26ec0ae000 tx=0x7f26ec003030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:06:34.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:34.951+0000 7f26fc200640 1 -- 172.21.15.16:0/4207626682 <== osd.5 v2:172.21.15.89:6810/50395527 1 ==== osd_op_reply(1 10000000001.00000000 [getxattr (94) out=94b] v0'0 uv1 ondisk = 0) ==== 164+0+94 (crc 0 0 0) 0x7f26f400cb40 con 0x55d2419e89b0 2024-03-31T15:06:34.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:34.951+0000 7f26fe5a0080 1 -- 172.21.15.16:0/4207626682 >> [v2:172.21.15.89:6810/50395527,v1:172.21.15.89:6814/50395527] conn(0x55d2419e89b0 msgr2=0x55d241a8ffe0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:34.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:34.951+0000 7f26fe5a0080 1 --2- 172.21.15.16:0/4207626682 >> [v2:172.21.15.89:6810/50395527,v1:172.21.15.89:6814/50395527] conn(0x55d2419e89b0 0x55d241a8ffe0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:34.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:34.951+0000 7f26fe5a0080 1 -- 172.21.15.16:0/4207626682 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f26dc066d60 msgr2=0x7f26dc069220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:34.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:34.951+0000 7f26fe5a0080 1 --2- 172.21.15.16:0/4207626682 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f26dc066d60 0x7f26dc069220 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f26ec0ae000 tx=0x7f26ec003030 comp rx=0 tx=0).stop 2024-03-31T15:06:34.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:34.951+0000 7f26fe5a0080 1 -- 172.21.15.16:0/4207626682 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d241a079f0 msgr2=0x55d2419e8470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:34.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:34.951+0000 7f26fe5a0080 1 --2- 172.21.15.16:0/4207626682 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d241a079f0 0x55d2419e8470 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f26e800fd80 tx=0x7f26e8001ff0 comp rx=0 tx=0).stop 2024-03-31T15:06:34.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:34.951+0000 7f26fc200640 1 -- 172.21.15.16:0/4207626682 reap_dead start 2024-03-31T15:06:34.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:34.951+0000 7f26fe5a0080 1 -- 172.21.15.16:0/4207626682 shutdown_connections 2024-03-31T15:06:34.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:34.951+0000 7f26fe5a0080 1 -- 172.21.15.16:0/4207626682 >> 172.21.15.16:0/4207626682 conn(0x55d241a050b0 msgr2=0x55d241a070a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:06:34.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:34.951+0000 7f26fe5a0080 1 -- 172.21.15.16:0/4207626682 shutdown_connections 2024-03-31T15:06:34.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:34.951+0000 7f26fe5a0080 1 -- 172.21.15.16:0/4207626682 wait complete. 2024-03-31T15:06:34.952 INFO:teuthology.orchestra.run.smithi016.stdout:Xfiledir_test 2024-03-31T15:06:34.952 DEBUG:teuthology.orchestra.run.smithi016:> ceph-dencoder type inode_backtrace_t import - decode dump_json 2024-03-31T15:06:35.040 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-03-31T15:06:35.040 INFO:teuthology.orchestra.run.smithi016.stdout: "ino": 1099511627777, 2024-03-31T15:06:35.040 INFO:teuthology.orchestra.run.smithi016.stdout: "ancestors": [ 2024-03-31T15:06:35.040 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-31T15:06:35.040 INFO:teuthology.orchestra.run.smithi016.stdout: "dirino": 1099511627776, 2024-03-31T15:06:35.040 INFO:teuthology.orchestra.run.smithi016.stdout: "dname": "file", 2024-03-31T15:06:35.041 INFO:teuthology.orchestra.run.smithi016.stdout: "version": 2 2024-03-31T15:06:35.041 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-31T15:06:35.041 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-31T15:06:35.041 INFO:teuthology.orchestra.run.smithi016.stdout: "dirino": 1, 2024-03-31T15:06:35.041 INFO:teuthology.orchestra.run.smithi016.stdout: "dname": "dir_test", 2024-03-31T15:06:35.041 INFO:teuthology.orchestra.run.smithi016.stdout: "version": 4 2024-03-31T15:06:35.041 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-31T15:06:35.041 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-03-31T15:06:35.041 INFO:teuthology.orchestra.run.smithi016.stdout: "pool": 9, 2024-03-31T15:06:35.041 INFO:teuthology.orchestra.run.smithi016.stdout: "old_pools": [] 2024-03-31T15:06:35.041 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-03-31T15:06:35.041 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:06:35.045 DEBUG:teuthology.orchestra.run.smithi016:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_data setxattr 10000000001.00000000 parent 'The backtrace is corrupted' 2024-03-31T15:06:35.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.139+0000 7f3342b66080 1 Processor -- start 2024-03-31T15:06:35.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.139+0000 7f3342b66080 1 -- start start 2024-03-31T15:06:35.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.139+0000 7f3342b66080 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563746585350 0x563746565d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:35.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.139+0000 7f3342b66080 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x563746585730 0x563746613720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:35.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.139+0000 7f3342b66080 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5637465f59a0 0x563746582ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:35.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.139+0000 7f3342b66080 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5637465f6d10 con 0x563746585350 2024-03-31T15:06:35.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.139+0000 7f3342b66080 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x5637464e14b0 con 0x5637465f59a0 2024-03-31T15:06:35.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.139+0000 7f3342b66080 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x5637465f6e90 con 0x563746585730 2024-03-31T15:06:35.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.143+0000 7f333fe00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563746585350 0x563746565d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:35.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.143+0000 7f333fe00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563746585350 0x563746565d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49196/0 (socket says 172.21.15.16:49196) 2024-03-31T15:06:35.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.143+0000 7f333fe00640 1 -- 172.21.15.16:0/3202723521 learned_addr learned my addr 172.21.15.16:0/3202723521 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:06:35.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.143+0000 7f333f400640 1 --2- 172.21.15.16:0/3202723521 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x563746585730 0x563746613720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:35.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.143+0000 7f3340800640 1 --2- 172.21.15.16:0/3202723521 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5637465f59a0 0x563746582ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:35.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.143+0000 7f333fe00640 1 -- 172.21.15.16:0/3202723521 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x563746585730 msgr2=0x563746613720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:06:35.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.143+0000 7f333fe00640 1 --2- 172.21.15.16:0/3202723521 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x563746585730 0x563746613720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:35.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.143+0000 7f333fe00640 1 -- 172.21.15.16:0/3202723521 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5637465f59a0 msgr2=0x563746582ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:06:35.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.143+0000 7f333fe00640 1 --2- 172.21.15.16:0/3202723521 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5637465f59a0 0x563746582ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:35.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.143+0000 7f333fe00640 1 -- 172.21.15.16:0/3202723521 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56374656a130 con 0x563746585350 2024-03-31T15:06:35.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.143+0000 7f3340800640 1 --2- 172.21.15.16:0/3202723521 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5637465f59a0 0x563746582ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:06:35.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.143+0000 7f333fe00640 1 --2- 172.21.15.16:0/3202723521 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563746585350 0x563746565d20 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f333000a750 tx=0x7f333000ac10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e08d71b34a152f36 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:06:35.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.143+0000 7f333e000640 1 -- 172.21.15.16:0/3202723521 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f333000ae60 con 0x563746585350 2024-03-31T15:06:35.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.143+0000 7f333e000640 1 -- 172.21.15.16:0/3202723521 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3330008ba0 con 0x563746585350 2024-03-31T15:06:35.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.143+0000 7f333e000640 1 -- 172.21.15.16:0/3202723521 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33300133f0 con 0x563746585350 2024-03-31T15:06:35.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.143+0000 7f3342b66080 1 -- 172.21.15.16:0/3202723521 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563746585350 msgr2=0x563746565d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:35.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.143+0000 7f3342b66080 1 --2- 172.21.15.16:0/3202723521 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563746585350 0x563746565d20 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f333000a750 tx=0x7f333000ac10 comp rx=0 tx=0).stop 2024-03-31T15:06:35.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.143+0000 7f3342b66080 1 -- 172.21.15.16:0/3202723521 shutdown_connections 2024-03-31T15:06:35.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.143+0000 7f3342b66080 1 --2- 172.21.15.16:0/3202723521 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5637465f59a0 0x563746582ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:35.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.143+0000 7f3342b66080 1 --2- 172.21.15.16:0/3202723521 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x563746585730 0x563746613720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:35.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.143+0000 7f3342b66080 1 --2- 172.21.15.16:0/3202723521 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563746585350 0x563746565d20 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:35.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.143+0000 7f3342b66080 1 -- 172.21.15.16:0/3202723521 >> 172.21.15.16:0/3202723521 conn(0x563746567050 msgr2=0x56374656bae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:06:35.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.143+0000 7f3342b66080 1 -- 172.21.15.16:0/3202723521 shutdown_connections 2024-03-31T15:06:35.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.143+0000 7f3342b66080 1 -- 172.21.15.16:0/3202723521 wait complete. 2024-03-31T15:06:35.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.143+0000 7f3342b66080 1 Processor -- start 2024-03-31T15:06:35.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.143+0000 7f3342b66080 1 -- start start 2024-03-31T15:06:35.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.143+0000 7f3342b66080 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563746585350 0x5637466117a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:35.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.143+0000 7f3342b66080 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563746585730 0x5637466153b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:35.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.143+0000 7f3342b66080 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5637465f59a0 0x56374656b690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:35.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.143+0000 7f3342b66080 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x56374656a670 con 0x563746585730 2024-03-31T15:06:35.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.143+0000 7f3342b66080 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x56374656a3d0 con 0x563746585350 2024-03-31T15:06:35.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.143+0000 7f3342b66080 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x56374656a910 con 0x5637465f59a0 2024-03-31T15:06:35.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.143+0000 7f333f400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563746585730 0x5637466153b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:35.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.143+0000 7f333f400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563746585730 0x5637466153b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49202/0 (socket says 172.21.15.16:49202) 2024-03-31T15:06:35.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.143+0000 7f333f400640 1 -- 172.21.15.16:0/4170356751 learned_addr learned my addr 172.21.15.16:0/4170356751 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:06:35.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.147+0000 7f333fe00640 1 --2- 172.21.15.16:0/4170356751 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563746585350 0x5637466117a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:35.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.147+0000 7f3340800640 1 --2- 172.21.15.16:0/4170356751 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5637465f59a0 0x56374656b690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:35.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.147+0000 7f333f400640 1 -- 172.21.15.16:0/4170356751 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5637465f59a0 msgr2=0x56374656b690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:35.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.147+0000 7f333f400640 1 --2- 172.21.15.16:0/4170356751 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5637465f59a0 0x56374656b690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:35.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.147+0000 7f333f400640 1 -- 172.21.15.16:0/4170356751 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563746585350 msgr2=0x5637466117a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:35.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.147+0000 7f333f400640 1 --2- 172.21.15.16:0/4170356751 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563746585350 0x5637466117a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:35.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.147+0000 7f333f400640 1 -- 172.21.15.16:0/4170356751 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563746615d30 con 0x563746585730 2024-03-31T15:06:35.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.147+0000 7f333f400640 1 --2- 172.21.15.16:0/4170356751 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563746585730 0x5637466153b0 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f332c00c5f0 tx=0x7f332c00cab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:06:35.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.147+0000 7f333cc00640 1 -- 172.21.15.16:0/4170356751 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f332c0106c0 con 0x563746585730 2024-03-31T15:06:35.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.147+0000 7f333cc00640 1 -- 172.21.15.16:0/4170356751 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f332c015020 con 0x563746585730 2024-03-31T15:06:35.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.147+0000 7f333cc00640 1 -- 172.21.15.16:0/4170356751 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f332c010a80 con 0x563746585730 2024-03-31T15:06:35.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.147+0000 7f3342b66080 1 -- 172.21.15.16:0/4170356751 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x563746615fd0 con 0x563746585730 2024-03-31T15:06:35.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.147+0000 7f3342b66080 1 -- 172.21.15.16:0/4170356751 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563746618fd0 con 0x563746585730 2024-03-31T15:06:35.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.147+0000 7f333cc00640 1 -- 172.21.15.16:0/4170356751 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f332c010c20 con 0x563746585730 2024-03-31T15:06:35.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.147+0000 7f333cc00640 1 --2- 172.21.15.16:0/4170356751 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f3320066d90 0x7f3320069250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:35.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.151+0000 7f333cc00640 1 -- 172.21.15.16:0/4170356751 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 5709+0+0 (secure 0 0 0) 0x7f332c085fa0 con 0x563746585730 2024-03-31T15:06:35.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.151+0000 7f3342b66080 1 --2- 172.21.15.16:0/4170356751 >> [v2:172.21.15.89:6810/50395527,v1:172.21.15.89:6814/50395527] conn(0x56374661a480 0x5637466b7d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:35.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.151+0000 7f333fe00640 1 --2- 172.21.15.16:0/4170356751 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f3320066d90 0x7f3320069250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:35.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.151+0000 7f3342b66080 1 -- 172.21.15.16:0/4170356751 --> [v2:172.21.15.89:6810/50395527,v1:172.21.15.89:6814/50395527] -- osd_op(unknown.0.0:1 9.7 9:e2e07930:::10000000001.00000000:head [setxattr parent (26) in=32b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e56) -- 0x5637466158f0 con 0x56374661a480 2024-03-31T15:06:35.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.151+0000 7f3340800640 1 --2- 172.21.15.16:0/4170356751 >> [v2:172.21.15.89:6810/50395527,v1:172.21.15.89:6814/50395527] conn(0x56374661a480 0x5637466b7d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:35.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.151+0000 7f333fe00640 1 --2- 172.21.15.16:0/4170356751 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f3320066d90 0x7f3320069250 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f333000a6c0 tx=0x7f333000c000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:06:35.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.151+0000 7f3340800640 1 --2- 172.21.15.16:0/4170356751 >> [v2:172.21.15.89:6810/50395527,v1:172.21.15.89:6814/50395527] conn(0x56374661a480 0x5637466b7d40 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:06:35.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.151+0000 7f3340800640 1 -- 172.21.15.16:0/4170356751 <== osd.5 v2:172.21.15.89:6810/50395527 1 ==== osd_op_reply(1 10000000001.00000000 [setxattr (26)] v56'2 uv2 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f33380a8070 con 0x56374661a480 2024-03-31T15:06:35.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.151+0000 7f3342b66080 1 -- 172.21.15.16:0/4170356751 >> [v2:172.21.15.89:6810/50395527,v1:172.21.15.89:6814/50395527] conn(0x56374661a480 msgr2=0x5637466b7d40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:35.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.151+0000 7f3342b66080 1 --2- 172.21.15.16:0/4170356751 >> [v2:172.21.15.89:6810/50395527,v1:172.21.15.89:6814/50395527] conn(0x56374661a480 0x5637466b7d40 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:35.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.151+0000 7f3342b66080 1 -- 172.21.15.16:0/4170356751 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f3320066d90 msgr2=0x7f3320069250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:35.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.151+0000 7f3342b66080 1 --2- 172.21.15.16:0/4170356751 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f3320066d90 0x7f3320069250 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f333000a6c0 tx=0x7f333000c000 comp rx=0 tx=0).stop 2024-03-31T15:06:35.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.151+0000 7f3342b66080 1 -- 172.21.15.16:0/4170356751 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563746585730 msgr2=0x5637466153b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:35.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.151+0000 7f3342b66080 1 --2- 172.21.15.16:0/4170356751 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563746585730 0x5637466153b0 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f332c00c5f0 tx=0x7f332c00cab0 comp rx=0 tx=0).stop 2024-03-31T15:06:35.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.151+0000 7f3340800640 1 -- 172.21.15.16:0/4170356751 reap_dead start 2024-03-31T15:06:35.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.151+0000 7f3342b66080 1 -- 172.21.15.16:0/4170356751 shutdown_connections 2024-03-31T15:06:35.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.151+0000 7f3342b66080 1 -- 172.21.15.16:0/4170356751 >> 172.21.15.16:0/4170356751 conn(0x563746567050 msgr2=0x5637465f9df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:06:35.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.151+0000 7f3342b66080 1 -- 172.21.15.16:0/4170356751 shutdown_connections 2024-03-31T15:06:35.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.151+0000 7f3342b66080 1 -- 172.21.15.16:0/4170356751 wait complete. 2024-03-31T15:06:35.152 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:06:35.152 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.4:0 scrub start / recursive 2024-03-31T15:06:35.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.303+0000 7f8402600640 1 Processor -- start 2024-03-31T15:06:35.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.303+0000 7f8402600640 1 -- start start 2024-03-31T15:06:35.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.303+0000 7f8402600640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f83fc15e540 0x7f83fc0d12b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:35.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.303+0000 7f8402600640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f83fc0d17f0 0x7f83fc0d4000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:35.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.303+0000 7f8402600640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f83fc15e920 con 0x7f83fc0b9870 2024-03-31T15:06:35.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.303+0000 7f8402600640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f83fc0b73f0 con 0x7f83fc15e540 2024-03-31T15:06:35.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.303+0000 7f8402600640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f83fc0d5230 con 0x7f83fc0d17f0 2024-03-31T15:06:35.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.303+0000 7f83fb400640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f83fc15e540 0x7f83fc0d12b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:35.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.303+0000 7f83fb400640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f83fc15e540 0x7f83fc0d12b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49208/0 (socket says 172.21.15.16:49208) 2024-03-31T15:06:35.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.303+0000 7f83fb400640 1 -- 172.21.15.16:0/3876889444 learned_addr learned my addr 172.21.15.16:0/3876889444 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:06:35.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.303+0000 7f83fbe00640 1 --2- 172.21.15.16:0/3876889444 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f83fc0d17f0 0x7f83fc0d4000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:35.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.303+0000 7f83fb400640 1 -- 172.21.15.16:0/3876889444 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f83fc0d17f0 msgr2=0x7f83fc0d4000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:06:35.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.303+0000 7f83fb400640 1 --2- 172.21.15.16:0/3876889444 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f83fc0d17f0 0x7f83fc0d4000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:35.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.303+0000 7f83fb400640 1 -- 172.21.15.16:0/3876889444 >> v1:172.21.15.16:6789/0 conn(0x7f83fc0b9870 legacy=0x7f83fc0d38f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:35.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.303+0000 7f83fb400640 1 -- 172.21.15.16:0/3876889444 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83fc0d46b0 con 0x7f83fc15e540 2024-03-31T15:06:35.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.307+0000 7f83fb400640 1 --2- 172.21.15.16:0/3876889444 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f83fc15e540 0x7f83fc0d12b0 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f83ec009870 tx=0x7f83ec01c560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=529364bd4b297294 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:06:35.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.307+0000 7f83faa00640 1 -- 172.21.15.16:0/3876889444 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83ec0ab650 con 0x7f83fc15e540 2024-03-31T15:06:35.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.307+0000 7f83faa00640 1 -- 172.21.15.16:0/3876889444 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f83ec0b0070 con 0x7f83fc15e540 2024-03-31T15:06:35.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.307+0000 7f83faa00640 1 -- 172.21.15.16:0/3876889444 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83ec0aba10 con 0x7f83fc15e540 2024-03-31T15:06:35.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.307+0000 7f8402600640 1 -- 172.21.15.16:0/3876889444 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f83fc15e540 msgr2=0x7f83fc0d12b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:35.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.307+0000 7f8402600640 1 --2- 172.21.15.16:0/3876889444 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f83fc15e540 0x7f83fc0d12b0 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f83ec009870 tx=0x7f83ec01c560 comp rx=0 tx=0).stop 2024-03-31T15:06:35.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.307+0000 7f8402600640 1 -- 172.21.15.16:0/3876889444 shutdown_connections 2024-03-31T15:06:35.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.307+0000 7f8402600640 1 --2- 172.21.15.16:0/3876889444 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f83fc0d17f0 0x7f83fc0d4000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:35.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.307+0000 7f8402600640 1 --2- 172.21.15.16:0/3876889444 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f83fc15e540 0x7f83fc0d12b0 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:35.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.307+0000 7f8402600640 1 -- 172.21.15.16:0/3876889444 >> 172.21.15.16:0/3876889444 conn(0x7f83fc067730 msgr2=0x7f83fc067b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:06:35.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.307+0000 7f8402600640 1 -- 172.21.15.16:0/3876889444 shutdown_connections 2024-03-31T15:06:35.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.307+0000 7f8402600640 1 -- 172.21.15.16:0/3876889444 wait complete. 2024-03-31T15:06:35.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.311+0000 7f8402600640 1 Processor -- start 2024-03-31T15:06:35.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.311+0000 7f8402600640 1 -- start start 2024-03-31T15:06:35.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.311+0000 7f8402600640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f83fc0b9870 0x7f83fc0cd130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:35.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.311+0000 7f8402600640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f83fc0d17f0 0x7f83fc1546a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:35.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.311+0000 7f8402600640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f83fc15e540 0x7f83fc1559f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:35.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.311+0000 7f8402600640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f83fc16aa10 con 0x7f83fc0d17f0 2024-03-31T15:06:35.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.311+0000 7f8402600640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f83fc16a890 con 0x7f83fc0b9870 2024-03-31T15:06:35.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.311+0000 7f8402600640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f83fc16ab90 con 0x7f83fc15e540 2024-03-31T15:06:35.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.311+0000 7f83f2a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f83fc0d17f0 0x7f83fc1546a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:35.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.311+0000 7f83fb400640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f83fc0b9870 0x7f83fc0cd130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:35.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.311+0000 7f83f2a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f83fc0d17f0 0x7f83fc1546a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49212/0 (socket says 172.21.15.16:49212) 2024-03-31T15:06:35.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.311+0000 7f83f2a00640 1 -- 172.21.15.16:0/3512590277 learned_addr learned my addr 172.21.15.16:0/3512590277 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:06:35.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.311+0000 7f83fbe00640 1 --2- 172.21.15.16:0/3512590277 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f83fc15e540 0x7f83fc1559f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:35.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.311+0000 7f83fbe00640 1 -- 172.21.15.16:0/3512590277 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f83fc0b9870 msgr2=0x7f83fc0cd130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:35.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.311+0000 7f83fbe00640 1 --2- 172.21.15.16:0/3512590277 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f83fc0b9870 0x7f83fc0cd130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:35.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.311+0000 7f83fbe00640 1 -- 172.21.15.16:0/3512590277 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f83fc0d17f0 msgr2=0x7f83fc1546a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:35.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.311+0000 7f83fbe00640 1 --2- 172.21.15.16:0/3512590277 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f83fc0d17f0 0x7f83fc1546a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:35.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.311+0000 7f83fbe00640 1 -- 172.21.15.16:0/3512590277 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83fc0cc0b0 con 0x7f83fc15e540 2024-03-31T15:06:35.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.311+0000 7f83f2a00640 1 --2- 172.21.15.16:0/3512590277 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f83fc0d17f0 0x7f83fc1546a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:06:35.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.311+0000 7f83fb400640 1 --2- 172.21.15.16:0/3512590277 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f83fc0b9870 0x7f83fc0cd130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:06:35.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.311+0000 7f83fbe00640 1 --2- 172.21.15.16:0/3512590277 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f83fc15e540 0x7f83fc1559f0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f83f400efc0 tx=0x7f83f400c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:06:35.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.315+0000 7f83f3e00640 1 -- 172.21.15.16:0/3512590277 <== mon.2 v2:172.21.15.89:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83f4018020 con 0x7f83fc15e540 2024-03-31T15:06:35.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.315+0000 7f83f3e00640 1 -- 172.21.15.16:0/3512590277 <== mon.2 v2:172.21.15.89:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f83f4014480 con 0x7f83fc15e540 2024-03-31T15:06:35.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.315+0000 7f8402600640 1 -- 172.21.15.16:0/3512590277 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f83fc0cc390 con 0x7f83fc15e540 2024-03-31T15:06:35.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.315+0000 7f83f3e00640 1 -- 172.21.15.16:0/3512590277 <== mon.2 v2:172.21.15.89:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83f4014870 con 0x7f83fc15e540 2024-03-31T15:06:35.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.315+0000 7f8402600640 1 -- 172.21.15.16:0/3512590277 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f83fc0cc950 con 0x7f83fc15e540 2024-03-31T15:06:35.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.315+0000 7f83f3e00640 1 -- 172.21.15.16:0/3512590277 <== mon.2 v2:172.21.15.89:3301/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f83f401b020 con 0x7f83fc15e540 2024-03-31T15:06:35.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.315+0000 7f83f3e00640 1 --2- 172.21.15.16:0/3512590277 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f83d8068380 0x7f83d806a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:35.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.315+0000 7f83f3e00640 1 -- 172.21.15.16:0/3512590277 <== mon.2 v2:172.21.15.89:3301/0 5 ==== osd_map(56..56 src has 1..56) ==== 5709+0+0 (secure 0 0 0) 0x7f83f4087760 con 0x7f83fc15e540 2024-03-31T15:06:35.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.315+0000 7f83fb400640 1 --2- 172.21.15.16:0/3512590277 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f83d8068380 0x7f83d806a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:35.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.319+0000 7f83fb400640 1 --2- 172.21.15.16:0/3512590277 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f83d8068380 0x7f83d806a840 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f83ec0af000 tx=0x7f83ec0a7000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:06:35.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.319+0000 7f8405ad7000 1 Processor -- start 2024-03-31T15:06:35.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.319+0000 7f8405ad7000 1 -- start start 2024-03-31T15:06:35.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.319+0000 7f8405ad7000 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56321c957210 0x56321c9575f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:35.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.319+0000 7f8405ad7000 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x56321c957b30 0x56321c90d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:35.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.319+0000 7f8405ad7000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56321c90dc30 0x56321c90e010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:35.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.319+0000 7f8405ad7000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x56321c75bc60 con 0x56321c90dc30 2024-03-31T15:06:35.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.319+0000 7f8405ad7000 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x56321c6f8b80 con 0x56321c957210 2024-03-31T15:06:35.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.319+0000 7f8405ad7000 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x56321c757c10 con 0x56321c957b30 2024-03-31T15:06:35.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.319+0000 7f83fbe00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56321c90dc30 0x56321c90e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:35.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.319+0000 7f83fbe00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56321c90dc30 0x56321c90e010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49222/0 (socket says 172.21.15.16:49222) 2024-03-31T15:06:35.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.319+0000 7f83fb400640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56321c957210 0x56321c9575f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:35.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.319+0000 7f83fbe00640 1 -- 172.21.15.16:0/1990193809 learned_addr learned my addr 172.21.15.16:0/1990193809 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:06:35.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.319+0000 7f83f2a00640 1 --2- 172.21.15.16:0/1990193809 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x56321c957b30 0x56321c90d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:35.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.323+0000 7f83fbe00640 1 -- 172.21.15.16:0/1990193809 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x56321c957b30 msgr2=0x56321c90d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:06:35.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.323+0000 7f83fbe00640 1 --2- 172.21.15.16:0/1990193809 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x56321c957b30 0x56321c90d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:35.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.323+0000 7f83fbe00640 1 -- 172.21.15.16:0/1990193809 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56321c957210 msgr2=0x56321c9575f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:06:35.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.323+0000 7f83fbe00640 1 --2- 172.21.15.16:0/1990193809 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56321c957210 0x56321c9575f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:35.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.323+0000 7f83fbe00640 1 -- 172.21.15.16:0/1990193809 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56321c7358d0 con 0x56321c90dc30 2024-03-31T15:06:35.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.323+0000 7f83fbe00640 1 --2- 172.21.15.16:0/1990193809 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56321c90dc30 0x56321c90e010 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f83f400c990 tx=0x7f83f4057770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e881ceb0224d8f81 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:06:35.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.323+0000 7f83ebe00640 1 -- 172.21.15.16:0/1990193809 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83f4053ca0 con 0x56321c90dc30 2024-03-31T15:06:35.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.323+0000 7f83ebe00640 1 -- 172.21.15.16:0/1990193809 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f83f4056a80 con 0x56321c90dc30 2024-03-31T15:06:35.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.323+0000 7f83ebe00640 1 -- 172.21.15.16:0/1990193809 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83f4053380 con 0x56321c90dc30 2024-03-31T15:06:35.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.323+0000 7f8405ad7000 1 -- 172.21.15.16:0/1990193809 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56321c90dc30 msgr2=0x56321c90e010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:35.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.323+0000 7f8405ad7000 1 --2- 172.21.15.16:0/1990193809 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56321c90dc30 0x56321c90e010 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f83f400c990 tx=0x7f83f4057770 comp rx=0 tx=0).stop 2024-03-31T15:06:35.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.323+0000 7f8405ad7000 1 -- 172.21.15.16:0/1990193809 shutdown_connections 2024-03-31T15:06:35.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.323+0000 7f8405ad7000 1 --2- 172.21.15.16:0/1990193809 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56321c90dc30 0x56321c90e010 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:35.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.323+0000 7f8405ad7000 1 --2- 172.21.15.16:0/1990193809 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x56321c957b30 0x56321c90d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:35.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.323+0000 7f8405ad7000 1 --2- 172.21.15.16:0/1990193809 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56321c957210 0x56321c9575f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:35.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.323+0000 7f8405ad7000 1 -- 172.21.15.16:0/1990193809 >> 172.21.15.16:0/1990193809 conn(0x56321c909be0 msgr2=0x56321c90bfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:06:35.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.323+0000 7f8405ad7000 1 -- 172.21.15.16:0/1990193809 shutdown_connections 2024-03-31T15:06:35.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.323+0000 7f8405ad7000 1 -- 172.21.15.16:0/1990193809 wait complete. 2024-03-31T15:06:35.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.323+0000 7f8405ad7000 1 -- start start 2024-03-31T15:06:35.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.323+0000 7f8405ad7000 1 Processor -- start 2024-03-31T15:06:35.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.323+0000 7f8405ad7000 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x56321c90dc30 0x56321c913120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:35.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.323+0000 7f8405ad7000 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56321c957210 0x56321c93c450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:35.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.323+0000 7f8405ad7000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56321c957b30 0x56321c93c990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:35.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.323+0000 7f8405ad7000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x56321c7192a0 con 0x56321c957b30 2024-03-31T15:06:35.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.323+0000 7f8405ad7000 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x56321c721370 con 0x56321c957210 2024-03-31T15:06:35.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.323+0000 7f8405ad7000 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x56321c760290 con 0x56321c90dc30 2024-03-31T15:06:35.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.323+0000 7f8405ad7000 10 client.0 _ll_register_callbacks cb 0x56321c8f8af0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T15:06:35.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.323+0000 7f83fbe00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56321c957b30 0x56321c93c990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:35.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.323+0000 7f83fbe00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56321c957b30 0x56321c93c990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49226/0 (socket says 172.21.15.16:49226) 2024-03-31T15:06:35.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.323+0000 7f83fbe00640 1 -- 172.21.15.16:0/1602824862 learned_addr learned my addr 172.21.15.16:0/1602824862 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:06:35.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.327+0000 7f83f2a00640 1 --2- 172.21.15.16:0/1602824862 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56321c957210 0x56321c93c450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:35.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.327+0000 7f83fb400640 1 --2- 172.21.15.16:0/1602824862 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x56321c90dc30 0x56321c913120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:35.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.327+0000 7f83fbe00640 1 -- 172.21.15.16:0/1602824862 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x56321c90dc30 msgr2=0x56321c913120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:06:35.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.327+0000 7f83fbe00640 1 --2- 172.21.15.16:0/1602824862 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x56321c90dc30 0x56321c913120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:35.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.327+0000 7f83fbe00640 1 -- 172.21.15.16:0/1602824862 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56321c957210 msgr2=0x56321c93c450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:06:35.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.327+0000 7f83fbe00640 1 --2- 172.21.15.16:0/1602824862 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56321c957210 0x56321c93c450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:35.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.327+0000 7f83fbe00640 1 -- 172.21.15.16:0/1602824862 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56321c7a10d0 con 0x56321c957b30 2024-03-31T15:06:35.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.327+0000 7f83fbe00640 1 --2- 172.21.15.16:0/1602824862 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56321c957b30 0x56321c93c990 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f83f4057330 tx=0x7f83f405a070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9499e57d34edf2c0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:06:35.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.327+0000 7f83e8c00640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-31T15:06:35.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.327+0000 7f83e8c00640 1 -- 172.21.15.16:0/1602824862 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83f4074060 con 0x56321c957b30 2024-03-31T15:06:35.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.327+0000 7f83e8c00640 1 -- 172.21.15.16:0/1602824862 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(56..56 src has 1..56) ==== 5709+0+0 (secure 0 0 0) 0x7f83f4053580 con 0x56321c957b30 2024-03-31T15:06:35.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.327+0000 7f8405ad7000 1 -- 172.21.15.16:0/1602824862 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f83f4056a80 con 0x56321c957b30 2024-03-31T15:06:35.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.327+0000 7f83e8c00640 1 -- 172.21.15.16:0/1602824862 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f83f405cac0 con 0x56321c957b30 2024-03-31T15:06:35.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.327+0000 7f8405ad7000 10 client.6345 fetch_fsmap learned FSMap version 34 2024-03-31T15:06:35.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.327+0000 7f8405ad7000 1 -- 172.21.15.16:0/1602824862 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=2+}) -- 0x56321c6e14e0 con 0x56321c957b30 2024-03-31T15:06:35.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.327+0000 7f83e8c00640 1 -- 172.21.15.16:0/1602824862 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f83f4056a80 con 0x56321c957b30 2024-03-31T15:06:35.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.327+0000 7f83e8c00640 1 -- 172.21.15.16:0/1602824862 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 34) ==== 3494+0+0 (secure 0 0 0) 0x7f83f405cc60 con 0x56321c957b30 2024-03-31T15:06:35.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.327+0000 7f8405ad7000 10 client.6345 fetch_fsmap finished waiting for FSMap version 34 2024-03-31T15:06:35.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.327+0000 7f8405ad7000 10 client.6345 resolve_mds: resolved 4:0 to role '4:0' aka daemon mds.c 2024-03-31T15:06:35.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.327+0000 7f8405ad7000 20 client.6345 populate_metadata read hostname 'smithi016' 2024-03-31T15:06:35.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.327+0000 7f8405ad7000 1 --2- 172.21.15.16:0/1602824862 >> [v2:172.21.15.16:6834/2613117528,v1:172.21.15.16:6835/2613117528] conn(0x56321c95e650 0x56321c95ea60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:35.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.327+0000 7f8405ad7000 4 client.6345 mds_command: new command op to 6150 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-31T15:06:35.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.327+0000 7f8405ad7000 1 -- 172.21.15.16:0/1602824862 --> [v2:172.21.15.16:6834/2613117528,v1:172.21.15.16:6835/2613117528] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56321c6f6f70 con 0x56321c95e650 2024-03-31T15:06:35.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.327+0000 7f83fb400640 1 --2- 172.21.15.16:0/1602824862 >> [v2:172.21.15.16:6834/2613117528,v1:172.21.15.16:6835/2613117528] conn(0x56321c95e650 0x56321c95ea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:35.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.327+0000 7f83fb400640 1 --2- 172.21.15.16:0/1602824862 >> [v2:172.21.15.16:6834/2613117528,v1:172.21.15.16:6835/2613117528] conn(0x56321c95e650 0x56321c95ea60 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1af0f09d14c68351 server_cookie=11c15535907d55d4 in_seq=0 out_seq=0 2024-03-31T15:06:35.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.327+0000 7f83e8c00640 10 client.6345 ms_handle_connect on v2:172.21.15.16:6834/2613117528 2024-03-31T15:06:35.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.327+0000 7f83e8c00640 1 -- 172.21.15.16:0/1602824862 <== mds.0 v2:172.21.15.16:6834/2613117528 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56321c6f6f70 con 0x56321c95e650 2024-03-31T15:06:35.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.327+0000 7f83e8c00640 10 client.6345 handle_command_reply: tid=0 2024-03-31T15:06:35.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.327+0000 7f8405ad7000 1 client.6345 shutdown 2024-03-31T15:06:35.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.327+0000 7f8405ad7000 1 -- 172.21.15.16:0/1602824862 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56321c957b30 msgr2=0x56321c93c990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:35.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.327+0000 7f8405ad7000 1 --2- 172.21.15.16:0/1602824862 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56321c957b30 0x56321c93c990 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f83f4057330 tx=0x7f83f405a070 comp rx=0 tx=0).stop 2024-03-31T15:06:35.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.327+0000 7f8405ad7000 1 -- 172.21.15.16:0/1602824862 shutdown_connections 2024-03-31T15:06:35.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.327+0000 7f8405ad7000 1 --2- 172.21.15.16:0/1602824862 >> [v2:172.21.15.16:6834/2613117528,v1:172.21.15.16:6835/2613117528] conn(0x56321c95e650 0x56321c95ea60 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:35.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.327+0000 7f8405ad7000 1 --2- 172.21.15.16:0/1602824862 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56321c957b30 0x56321c93c990 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:35.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.327+0000 7f8405ad7000 1 --2- 172.21.15.16:0/1602824862 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56321c957210 0x56321c93c450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:35.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.327+0000 7f8405ad7000 1 --2- 172.21.15.16:0/1602824862 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x56321c90dc30 0x56321c913120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:35.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.327+0000 7f8405ad7000 1 -- 172.21.15.16:0/1602824862 >> 172.21.15.16:0/1602824862 conn(0x56321c909be0 msgr2=0x56321c90c860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:06:35.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.327+0000 7f83e8c00640 0 client.6345 ms_handle_reset on v2:172.21.15.16:6834/2613117528 2024-03-31T15:06:35.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.327+0000 7f8405ad7000 1 -- 172.21.15.16:0/1602824862 shutdown_connections 2024-03-31T15:06:35.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.327+0000 7f8405ad7000 1 -- 172.21.15.16:0/1602824862 wait complete. 2024-03-31T15:06:35.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.327+0000 7f8405ad7000 20 client.6345 trim_cache size 0 max 16384 2024-03-31T15:06:35.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.351+0000 7f8405ad7000 1 Processor -- start 2024-03-31T15:06:35.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.351+0000 7f8405ad7000 1 -- start start 2024-03-31T15:06:35.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.351+0000 7f8405ad7000 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x56321c90dc30 0x56321c956300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:35.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.351+0000 7f8405ad7000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56321c957210 0x56321c956840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:35.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.351+0000 7f8405ad7000 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56321c957b30 0x56321c90bfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:35.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.351+0000 7f8405ad7000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f83fc203820 con 0x56321c957210 2024-03-31T15:06:35.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.351+0000 7f8405ad7000 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f83fc15a300 con 0x56321c957b30 2024-03-31T15:06:35.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.351+0000 7f8405ad7000 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f83fc159660 con 0x56321c90dc30 2024-03-31T15:06:35.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.355+0000 7f83f2a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56321c957210 0x56321c956840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:35.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.355+0000 7f83fb400640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x56321c90dc30 0x56321c956300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:35.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.355+0000 7f83f2a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56321c957210 0x56321c956840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49234/0 (socket says 172.21.15.16:49234) 2024-03-31T15:06:35.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.355+0000 7f83f2a00640 1 -- 172.21.15.16:0/3736809966 learned_addr learned my addr 172.21.15.16:0/3736809966 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:06:35.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.355+0000 7f83fbe00640 1 --2- 172.21.15.16:0/3736809966 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56321c957b30 0x56321c90bfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:35.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.355+0000 7f83f2a00640 1 -- 172.21.15.16:0/3736809966 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x56321c90dc30 msgr2=0x56321c956300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:06:35.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.355+0000 7f83f2a00640 1 --2- 172.21.15.16:0/3736809966 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x56321c90dc30 0x56321c956300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:35.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.355+0000 7f83f2a00640 1 -- 172.21.15.16:0/3736809966 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56321c957b30 msgr2=0x56321c90bfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:06:35.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.355+0000 7f83f2a00640 1 --2- 172.21.15.16:0/3736809966 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56321c957b30 0x56321c90bfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:35.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.355+0000 7f83f2a00640 1 -- 172.21.15.16:0/3736809966 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56321c729410 con 0x56321c957210 2024-03-31T15:06:35.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.355+0000 7f83f2a00640 1 --2- 172.21.15.16:0/3736809966 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56321c957210 0x56321c956840 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f83e40054e0 tx=0x7f83e4007f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b7139a65f3af9c77 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:06:35.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.355+0000 7f83eb400640 1 -- 172.21.15.16:0/3736809966 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83e401a070 con 0x56321c957210 2024-03-31T15:06:35.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.355+0000 7f83eb400640 1 -- 172.21.15.16:0/3736809966 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f83e4003b60 con 0x56321c957210 2024-03-31T15:06:35.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.355+0000 7f83eb400640 1 -- 172.21.15.16:0/3736809966 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83e4016420 con 0x56321c957210 2024-03-31T15:06:35.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.355+0000 7f8405ad7000 1 -- 172.21.15.16:0/3736809966 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56321c957210 msgr2=0x56321c956840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:35.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.355+0000 7f8405ad7000 1 --2- 172.21.15.16:0/3736809966 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56321c957210 0x56321c956840 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f83e40054e0 tx=0x7f83e4007f20 comp rx=0 tx=0).stop 2024-03-31T15:06:35.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.355+0000 7f8405ad7000 1 -- 172.21.15.16:0/3736809966 shutdown_connections 2024-03-31T15:06:35.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.355+0000 7f8405ad7000 1 --2- 172.21.15.16:0/3736809966 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56321c957b30 0x56321c90bfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:35.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.355+0000 7f8405ad7000 1 --2- 172.21.15.16:0/3736809966 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56321c957210 0x56321c956840 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:35.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.355+0000 7f8405ad7000 1 --2- 172.21.15.16:0/3736809966 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x56321c90dc30 0x56321c956300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:35.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.355+0000 7f8405ad7000 1 -- 172.21.15.16:0/3736809966 >> 172.21.15.16:0/3736809966 conn(0x56321c909be0 msgr2=0x56321c96df90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:06:35.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.355+0000 7f8405ad7000 1 -- 172.21.15.16:0/3736809966 shutdown_connections 2024-03-31T15:06:35.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.355+0000 7f8405ad7000 1 -- 172.21.15.16:0/3736809966 wait complete. 2024-03-31T15:06:35.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.355+0000 7f8405ad7000 1 -- start start 2024-03-31T15:06:35.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.355+0000 7f8405ad7000 1 Processor -- start 2024-03-31T15:06:35.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.359+0000 7f8405ad7000 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56321c90dc30 0x56321c9c6430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:35.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.359+0000 7f8405ad7000 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x56321c957210 0x56321c956340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:35.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.359+0000 7f8405ad7000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56321c957b30 0x56321c956880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:35.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.359+0000 7f8405ad7000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x56321c7dcd50 con 0x56321c957b30 2024-03-31T15:06:35.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.359+0000 7f8405ad7000 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f83fc159cb0 con 0x56321c90dc30 2024-03-31T15:06:35.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.359+0000 7f8405ad7000 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x56321c760290 con 0x56321c957210 2024-03-31T15:06:35.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.359+0000 7f8405ad7000 10 client.0 _ll_register_callbacks cb 0x7f83ec0096f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T15:06:35.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.359+0000 7f83fbe00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56321c957b30 0x56321c956880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:35.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.359+0000 7f83fbe00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56321c957b30 0x56321c956880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49238/0 (socket says 172.21.15.16:49238) 2024-03-31T15:06:35.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.359+0000 7f83fbe00640 1 -- 172.21.15.16:0/2783565268 learned_addr learned my addr 172.21.15.16:0/2783565268 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:06:35.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.359+0000 7f83fb400640 1 --2- 172.21.15.16:0/2783565268 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56321c90dc30 0x56321c9c6430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:35.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.359+0000 7f83f2a00640 1 --2- 172.21.15.16:0/2783565268 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x56321c957210 0x56321c956340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:35.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.359+0000 7f83fbe00640 1 -- 172.21.15.16:0/2783565268 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x56321c957210 msgr2=0x56321c956340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:06:35.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.359+0000 7f83fbe00640 1 --2- 172.21.15.16:0/2783565268 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x56321c957210 0x56321c956340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:35.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.359+0000 7f83fbe00640 1 -- 172.21.15.16:0/2783565268 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56321c90dc30 msgr2=0x56321c9c6430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:06:35.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.359+0000 7f83fbe00640 1 --2- 172.21.15.16:0/2783565268 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56321c90dc30 0x56321c9c6430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:35.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.359+0000 7f83fbe00640 1 -- 172.21.15.16:0/2783565268 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56321c7dd1e0 con 0x56321c957b30 2024-03-31T15:06:35.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.359+0000 7f83f2a00640 1 --2- 172.21.15.16:0/2783565268 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x56321c957210 0x56321c956340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:06:35.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.359+0000 7f83fbe00640 1 --2- 172.21.15.16:0/2783565268 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56321c957b30 0x56321c956880 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f83f4004b40 tx=0x7f83f40538d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=beb80208ca3e026c server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:06:35.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.359+0000 7f83e8c00640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-31T15:06:35.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.359+0000 7f83e8c00640 1 -- 172.21.15.16:0/2783565268 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83f4056a80 con 0x56321c957b30 2024-03-31T15:06:35.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.359+0000 7f83e8c00640 1 -- 172.21.15.16:0/2783565268 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(56..56 src has 1..56) ==== 5709+0+0 (secure 0 0 0) 0x7f83f4014a10 con 0x56321c957b30 2024-03-31T15:06:35.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.359+0000 7f8405ad7000 1 -- 172.21.15.16:0/2783565268 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f83e4003b60 con 0x56321c957b30 2024-03-31T15:06:35.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.359+0000 7f83e8c00640 1 -- 172.21.15.16:0/2783565268 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f83f4068050 con 0x56321c957b30 2024-03-31T15:06:35.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.359+0000 7f8405ad7000 10 client.6351 fetch_fsmap learned FSMap version 34 2024-03-31T15:06:35.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.359+0000 7f8405ad7000 1 -- 172.21.15.16:0/2783565268 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=2+}) -- 0x56321c4173b0 con 0x56321c957b30 2024-03-31T15:06:35.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.359+0000 7f83e8c00640 1 -- 172.21.15.16:0/2783565268 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f83e4003b60 con 0x56321c957b30 2024-03-31T15:06:35.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.359+0000 7f83e8c00640 1 -- 172.21.15.16:0/2783565268 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 34) ==== 3494+0+0 (secure 0 0 0) 0x7f83f406b020 con 0x56321c957b30 2024-03-31T15:06:35.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.359+0000 7f8405ad7000 10 client.6351 fetch_fsmap finished waiting for FSMap version 34 2024-03-31T15:06:35.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.359+0000 7f8405ad7000 10 client.6351 resolve_mds: resolved 4:0 to role '4:0' aka daemon mds.c 2024-03-31T15:06:35.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.359+0000 7f8405ad7000 20 client.6351 populate_metadata read hostname 'smithi016' 2024-03-31T15:06:35.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.359+0000 7f8405ad7000 1 --2- 172.21.15.16:0/2783565268 >> [v2:172.21.15.16:6834/2613117528,v1:172.21.15.16:6835/2613117528] conn(0x56321c95a3f0 0x56321c95a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:35.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.359+0000 7f8405ad7000 4 client.6351 mds_command: new command op to 6150 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["recursive"]}] 2024-03-31T15:06:35.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.359+0000 7f8405ad7000 1 -- 172.21.15.16:0/2783565268 --> [v2:172.21.15.16:6834/2613117528,v1:172.21.15.16:6835/2613117528] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["recursive"]}) -- 0x56321c7691a0 con 0x56321c95a3f0 2024-03-31T15:06:35.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.359+0000 7f83fb400640 1 --2- 172.21.15.16:0/2783565268 >> [v2:172.21.15.16:6834/2613117528,v1:172.21.15.16:6835/2613117528] conn(0x56321c95a3f0 0x56321c95a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:35.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.359+0000 7f83fb400640 1 --2- 172.21.15.16:0/2783565268 >> [v2:172.21.15.16:6834/2613117528,v1:172.21.15.16:6835/2613117528] conn(0x56321c95a3f0 0x56321c95a7d0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=43d5ab71727fef7d server_cookie=a925fd0b557a80df in_seq=0 out_seq=0 2024-03-31T15:06:35.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.359+0000 7f83e8c00640 10 client.6351 ms_handle_connect on v2:172.21.15.16:6834/2613117528 2024-03-31T15:06:35.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.363+0000 7f83e8c00640 1 -- 172.21.15.16:0/2783565268 <== mds.0 v2:172.21.15.16:6834/2613117528 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x56321c7691a0 con 0x56321c95a3f0 2024-03-31T15:06:35.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.363+0000 7f83e8c00640 10 client.6351 handle_command_reply: tid=0 2024-03-31T15:06:35.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.363+0000 7f8405ad7000 1 client.6351 shutdown 2024-03-31T15:06:35.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.363+0000 7f8405ad7000 1 -- 172.21.15.16:0/2783565268 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56321c957b30 msgr2=0x56321c956880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:35.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.363+0000 7f8405ad7000 1 --2- 172.21.15.16:0/2783565268 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56321c957b30 0x56321c956880 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f83f4004b40 tx=0x7f83f40538d0 comp rx=0 tx=0).stop 2024-03-31T15:06:35.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.363+0000 7f8405ad7000 1 -- 172.21.15.16:0/2783565268 shutdown_connections 2024-03-31T15:06:35.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.363+0000 7f8405ad7000 1 --2- 172.21.15.16:0/2783565268 >> [v2:172.21.15.16:6834/2613117528,v1:172.21.15.16:6835/2613117528] conn(0x56321c95a3f0 0x56321c95a7d0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:35.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.363+0000 7f8405ad7000 1 --2- 172.21.15.16:0/2783565268 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56321c957b30 0x56321c956880 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:35.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.363+0000 7f8405ad7000 1 --2- 172.21.15.16:0/2783565268 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x56321c957210 0x56321c956340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:35.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.363+0000 7f8405ad7000 1 --2- 172.21.15.16:0/2783565268 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56321c90dc30 0x56321c9c6430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:35.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.363+0000 7f8405ad7000 1 -- 172.21.15.16:0/2783565268 >> 172.21.15.16:0/2783565268 conn(0x56321c909be0 msgr2=0x56321c96e6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:06:35.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.363+0000 7f83e8c00640 0 client.6351 ms_handle_reset on v2:172.21.15.16:6834/2613117528 2024-03-31T15:06:35.361 INFO:tasks.ceph.mds.c.smithi016.stderr:2024-03-31T15:06:35.363+0000 7f4ef1400640 -1 mds.0.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000001 [2,head] /dir_test/file auth v2 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={6303=0-4194304@1} caps={6303=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@2},l=6303 | request=0 caps=1 dirtyparent=0 dirty=0 authpin=1 scrubqueue=0 0x564c5b37e580]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(-1)0x0:[]//[]","memoryvalue":"(9)0x10000000001:[<0x10000000000/file v2>,<0x1/dir_test v4>]//[]","error_str":"failed to decode on-disk backtrace (26 bytes)!"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-31T15:06:35.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.363+0000 7f8405ad7000 1 -- 172.21.15.16:0/2783565268 shutdown_connections 2024-03-31T15:06:35.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.363+0000 7f8405ad7000 1 -- 172.21.15.16:0/2783565268 wait complete. 2024-03-31T15:06:35.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.363+0000 7f8405ad7000 20 client.6351 trim_cache size 0 max 16384 2024-03-31T15:06:35.362 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-03-31T15:06:35.362 INFO:teuthology.orchestra.run.smithi016.stdout: "return_code": 0, 2024-03-31T15:06:35.362 INFO:teuthology.orchestra.run.smithi016.stdout: "scrub_tag": "ed29a749-4538-4efa-b094-9d8f540791b3", 2024-03-31T15:06:35.362 INFO:teuthology.orchestra.run.smithi016.stdout: "mode": "asynchronous" 2024-03-31T15:06:35.363 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-03-31T15:06:35.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.363+0000 7f8402600640 1 -- 172.21.15.16:0/3512590277 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f83d8068380 msgr2=0x7f83d806a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:35.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.363+0000 7f8402600640 1 --2- 172.21.15.16:0/3512590277 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f83d8068380 0x7f83d806a840 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f83ec0af000 tx=0x7f83ec0a7000 comp rx=0 tx=0).stop 2024-03-31T15:06:35.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.363+0000 7f8402600640 1 -- 172.21.15.16:0/3512590277 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f83fc15e540 msgr2=0x7f83fc1559f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:35.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.363+0000 7f8402600640 1 --2- 172.21.15.16:0/3512590277 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f83fc15e540 0x7f83fc1559f0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f83f400efc0 tx=0x7f83f400c490 comp rx=0 tx=0).stop 2024-03-31T15:06:35.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.363+0000 7f8402600640 1 -- 172.21.15.16:0/3512590277 shutdown_connections 2024-03-31T15:06:35.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.363+0000 7f8402600640 1 --2- 172.21.15.16:0/3512590277 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f83d8068380 0x7f83d806a840 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:35.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.363+0000 7f8402600640 1 --2- 172.21.15.16:0/3512590277 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f83fc15e540 0x7f83fc1559f0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:35.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.363+0000 7f8402600640 1 --2- 172.21.15.16:0/3512590277 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f83fc0d17f0 0x7f83fc1546a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:35.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.363+0000 7f8402600640 1 --2- 172.21.15.16:0/3512590277 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f83fc0b9870 0x7f83fc0cd130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:35.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.363+0000 7f8402600640 1 -- 172.21.15.16:0/3512590277 >> 172.21.15.16:0/3512590277 conn(0x7f83fc067730 msgr2=0x7f83fc0b5510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:06:35.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.363+0000 7f8402600640 1 -- 172.21.15.16:0/3512590277 shutdown_connections 2024-03-31T15:06:35.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.363+0000 7f8402600640 1 -- 172.21.15.16:0/3512590277 wait complete. 2024-03-31T15:06:35.377 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:06:35.377 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.4:0 scrub status 2024-03-31T15:06:35.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.527+0000 7fe5ef200640 1 Processor -- start 2024-03-31T15:06:35.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.527+0000 7fe5ef200640 1 -- start start 2024-03-31T15:06:35.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.527+0000 7fe5ef200640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fe5e8141590 0x7fe5e8144340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:35.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.527+0000 7fe5ef200640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe5e8144980 con 0x7fe5e81411b0 2024-03-31T15:06:35.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.527+0000 7fe5ef200640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe5e8161fe0 con 0x7fe5e8070030 2024-03-31T15:06:35.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.527+0000 7fe5ef200640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7fe5e815f910 con 0x7fe5e8141590 2024-03-31T15:06:35.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.527+0000 7fe5e7e00640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fe5e81411b0 0x7fe5e806f920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:56440/0 (socket says 172.21.15.16:56440) 2024-03-31T15:06:35.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.527+0000 7fe5ed200640 1 --2- 172.21.15.16:0/3160344674 >> v2:172.21.15.16:3300/0 conn(0x7fe5e8141590 0x7fe5e8144340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:35.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.527+0000 7fe5e7e00640 1 -- 172.21.15.16:0/3160344674 learned_addr learned my addr 172.21.15.16:0/3160344674 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:06:35.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.527+0000 7fe5ed200640 1 -- 172.21.15.16:0/3160344674 >> v1:172.21.15.89:6789/0 conn(0x7fe5e8070030 legacy=0x7fe5e8143c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:35.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.527+0000 7fe5ed200640 1 -- 172.21.15.16:0/3160344674 >> v1:172.21.15.16:6789/0 conn(0x7fe5e81411b0 legacy=0x7fe5e806f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:35.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.527+0000 7fe5ed200640 1 -- 172.21.15.16:0/3160344674 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe5e81622c0 con 0x7fe5e8141590 2024-03-31T15:06:35.525 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.531+0000 7fe5ed200640 1 --2- 172.21.15.16:0/3160344674 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe5e8141590 0x7fe5e8144340 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7fe5e0009d90 tx=0x7fe5e001c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e1e5ced56548826b server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:06:35.525 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.531+0000 7fe5e6a00640 1 -- 172.21.15.16:0/3160344674 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5e00b2070 con 0x7fe5e8141590 2024-03-31T15:06:35.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.531+0000 7fe5e6a00640 1 -- 172.21.15.16:0/3160344674 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe5e00a5940 con 0x7fe5e8141590 2024-03-31T15:06:35.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.531+0000 7fe5e6a00640 1 -- 172.21.15.16:0/3160344674 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5e00a5d10 con 0x7fe5e8141590 2024-03-31T15:06:35.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.531+0000 7fe5ef200640 1 -- 172.21.15.16:0/3160344674 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe5e8141590 msgr2=0x7fe5e8144340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:35.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.531+0000 7fe5ef200640 1 --2- 172.21.15.16:0/3160344674 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe5e8141590 0x7fe5e8144340 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7fe5e0009d90 tx=0x7fe5e001c970 comp rx=0 tx=0).stop 2024-03-31T15:06:35.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.531+0000 7fe5ef200640 1 -- 172.21.15.16:0/3160344674 shutdown_connections 2024-03-31T15:06:35.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.531+0000 7fe5ef200640 1 --2- 172.21.15.16:0/3160344674 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe5e8141590 0x7fe5e8144340 unknown :-1 s=CLOSED pgs=682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:35.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.531+0000 7fe5ef200640 1 -- 172.21.15.16:0/3160344674 >> 172.21.15.16:0/3160344674 conn(0x7fe5e8067930 msgr2=0x7fe5e80b3ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:06:35.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.531+0000 7fe5ef200640 1 -- 172.21.15.16:0/3160344674 shutdown_connections 2024-03-31T15:06:35.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.531+0000 7fe5ef200640 1 -- 172.21.15.16:0/3160344674 wait complete. 2024-03-31T15:06:35.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.531+0000 7fe5ef200640 1 Processor -- start 2024-03-31T15:06:35.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.535+0000 7fe5ef200640 1 -- start start 2024-03-31T15:06:35.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.535+0000 7fe5ef200640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fe5e8070030 0x7fe5e812c870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:35.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.535+0000 7fe5ef200640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe5e81411b0 0x7fe5e815d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:35.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.535+0000 7fe5ef200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe5e8141590 0x7fe5e815d630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:35.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.535+0000 7fe5ef200640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fe5e816be80 con 0x7fe5e8141590 2024-03-31T15:06:35.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.535+0000 7fe5ef200640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fe5e816bd00 con 0x7fe5e81411b0 2024-03-31T15:06:35.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.535+0000 7fe5ef200640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fe5e816c000 con 0x7fe5e8070030 2024-03-31T15:06:35.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.535+0000 7fe5ed200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe5e8141590 0x7fe5e815d630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:35.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.535+0000 7fe5ed200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe5e8141590 0x7fe5e815d630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49256/0 (socket says 172.21.15.16:49256) 2024-03-31T15:06:35.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.535+0000 7fe5ed200640 1 -- 172.21.15.16:0/1531084418 learned_addr learned my addr 172.21.15.16:0/1531084418 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:06:35.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.535+0000 7fe5ed200640 1 -- 172.21.15.16:0/1531084418 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fe5e8070030 msgr2=0x7fe5e812c870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:35.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.535+0000 7fe5e7e00640 1 --2- 172.21.15.16:0/1531084418 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fe5e8070030 0x7fe5e812c870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:35.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.535+0000 7fe5e7400640 1 --2- 172.21.15.16:0/1531084418 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe5e81411b0 0x7fe5e815d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:35.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.535+0000 7fe5ed200640 1 --2- 172.21.15.16:0/1531084418 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fe5e8070030 0x7fe5e812c870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:35.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.535+0000 7fe5ed200640 1 -- 172.21.15.16:0/1531084418 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe5e81411b0 msgr2=0x7fe5e815d0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:35.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.535+0000 7fe5ed200640 1 --2- 172.21.15.16:0/1531084418 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe5e81411b0 0x7fe5e815d0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:35.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.535+0000 7fe5ed200640 1 -- 172.21.15.16:0/1531084418 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe5e80d2de0 con 0x7fe5e8141590 2024-03-31T15:06:35.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.535+0000 7fe5e7400640 1 --2- 172.21.15.16:0/1531084418 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe5e81411b0 0x7fe5e815d0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:06:35.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.535+0000 7fe5e7e00640 1 --2- 172.21.15.16:0/1531084418 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fe5e8070030 0x7fe5e812c870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:06:35.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.535+0000 7fe5ed200640 1 --2- 172.21.15.16:0/1531084418 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe5e8141590 0x7fe5e815d630 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7fe5e001a5a0 tx=0x7fe5e0019710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:06:35.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.535+0000 7fe5dfe00640 1 -- 172.21.15.16:0/1531084418 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5e00b2050 con 0x7fe5e8141590 2024-03-31T15:06:35.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.535+0000 7fe5dfe00640 1 -- 172.21.15.16:0/1531084418 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe5e00be070 con 0x7fe5e8141590 2024-03-31T15:06:35.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.535+0000 7fe5dfe00640 1 -- 172.21.15.16:0/1531084418 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5e00ada50 con 0x7fe5e8141590 2024-03-31T15:06:35.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.535+0000 7fe5ef200640 1 -- 172.21.15.16:0/1531084418 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe5e81586a0 con 0x7fe5e8141590 2024-03-31T15:06:35.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.535+0000 7fe5ef200640 1 -- 172.21.15.16:0/1531084418 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe5e8154280 con 0x7fe5e8141590 2024-03-31T15:06:35.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.539+0000 7fe5dfe00640 1 -- 172.21.15.16:0/1531084418 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fe5e00adbf0 con 0x7fe5e8141590 2024-03-31T15:06:35.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.539+0000 7fe5dfe00640 1 --2- 172.21.15.16:0/1531084418 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fe5c80683b0 0x7fe5c806a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:35.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.539+0000 7fe5dfe00640 1 -- 172.21.15.16:0/1531084418 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 5709+0+0 (secure 0 0 0) 0x7fe5e01218c0 con 0x7fe5e8141590 2024-03-31T15:06:35.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.539+0000 7fe5e7e00640 1 --2- 172.21.15.16:0/1531084418 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fe5c80683b0 0x7fe5c806a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:35.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.539+0000 7fe5e7e00640 1 --2- 172.21.15.16:0/1531084418 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fe5c80683b0 0x7fe5c806a870 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fe5d8005890 tx=0x7fe5d8009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:06:35.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.539+0000 7fe5f26a2000 1 Processor -- start 2024-03-31T15:06:35.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.539+0000 7fe5f26a2000 1 -- start start 2024-03-31T15:06:35.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.543+0000 7fe5f26a2000 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55ea1cbe12d0 0x55ea1cbe16b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:35.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.543+0000 7fe5f26a2000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ea1cbe1bf0 0x55ea1cb94470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:35.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.543+0000 7fe5f26a2000 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ea1cb949b0 0x55ea1cb94d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:35.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.543+0000 7fe5f26a2000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55ea1ca7daf0 con 0x55ea1cbe1bf0 2024-03-31T15:06:35.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.543+0000 7fe5f26a2000 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55ea1c9e6b00 con 0x55ea1cb949b0 2024-03-31T15:06:35.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.543+0000 7fe5f26a2000 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x55ea1c983940 con 0x55ea1cbe12d0 2024-03-31T15:06:35.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.543+0000 7fe5e7400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ea1cbe1bf0 0x55ea1cb94470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:35.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.543+0000 7fe5e7400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ea1cbe1bf0 0x55ea1cb94470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49260/0 (socket says 172.21.15.16:49260) 2024-03-31T15:06:35.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.543+0000 7fe5e7400640 1 -- 172.21.15.16:0/797662954 learned_addr learned my addr 172.21.15.16:0/797662954 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:06:35.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.543+0000 7fe5e7e00640 1 --2- 172.21.15.16:0/797662954 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55ea1cbe12d0 0x55ea1cbe16b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:35.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.543+0000 7fe5ed200640 1 --2- 172.21.15.16:0/797662954 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ea1cb949b0 0x55ea1cb94d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:35.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.543+0000 7fe5e7400640 1 -- 172.21.15.16:0/797662954 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55ea1cbe12d0 msgr2=0x55ea1cbe16b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:06:35.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.543+0000 7fe5e7400640 1 --2- 172.21.15.16:0/797662954 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55ea1cbe12d0 0x55ea1cbe16b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:35.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.543+0000 7fe5e7400640 1 -- 172.21.15.16:0/797662954 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ea1cb949b0 msgr2=0x55ea1cb94d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:06:35.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.543+0000 7fe5e7400640 1 --2- 172.21.15.16:0/797662954 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ea1cb949b0 0x55ea1cb94d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:35.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.543+0000 7fe5e7400640 1 -- 172.21.15.16:0/797662954 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ea1c96c380 con 0x55ea1cbe1bf0 2024-03-31T15:06:35.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.543+0000 7fe5ed200640 1 --2- 172.21.15.16:0/797662954 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ea1cb949b0 0x55ea1cb94d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:06:35.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.543+0000 7fe5e7400640 1 --2- 172.21.15.16:0/797662954 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ea1cbe1bf0 0x55ea1cb94470 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7fe5d4001750 tx=0x7fe5d4001c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f95583d4d54c564 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:06:35.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.543+0000 7fe5dcc00640 1 -- 172.21.15.16:0/797662954 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5d4018070 con 0x55ea1cbe1bf0 2024-03-31T15:06:35.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.543+0000 7fe5dcc00640 1 -- 172.21.15.16:0/797662954 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe5d400d730 con 0x55ea1cbe1bf0 2024-03-31T15:06:35.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.543+0000 7fe5dcc00640 1 -- 172.21.15.16:0/797662954 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5d400dae0 con 0x55ea1cbe1bf0 2024-03-31T15:06:35.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.543+0000 7fe5f26a2000 1 -- 172.21.15.16:0/797662954 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ea1cbe1bf0 msgr2=0x55ea1cb94470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:35.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.543+0000 7fe5f26a2000 1 --2- 172.21.15.16:0/797662954 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ea1cbe1bf0 0x55ea1cb94470 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7fe5d4001750 tx=0x7fe5d4001c10 comp rx=0 tx=0).stop 2024-03-31T15:06:35.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.543+0000 7fe5f26a2000 1 -- 172.21.15.16:0/797662954 shutdown_connections 2024-03-31T15:06:35.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.543+0000 7fe5f26a2000 1 --2- 172.21.15.16:0/797662954 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ea1cb949b0 0x55ea1cb94d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:35.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.543+0000 7fe5f26a2000 1 --2- 172.21.15.16:0/797662954 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ea1cbe1bf0 0x55ea1cb94470 unknown :-1 s=CLOSED pgs=684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:35.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.543+0000 7fe5f26a2000 1 --2- 172.21.15.16:0/797662954 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55ea1cbe12d0 0x55ea1cbe16b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:35.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.543+0000 7fe5f26a2000 1 -- 172.21.15.16:0/797662954 >> 172.21.15.16:0/797662954 conn(0x55ea1cbc2070 msgr2=0x55ea1cbc4460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:06:35.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.543+0000 7fe5f26a2000 1 -- 172.21.15.16:0/797662954 shutdown_connections 2024-03-31T15:06:35.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.543+0000 7fe5f26a2000 1 -- 172.21.15.16:0/797662954 wait complete. 2024-03-31T15:06:35.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.543+0000 7fe5f26a2000 1 -- start start 2024-03-31T15:06:35.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.547+0000 7fe5f26a2000 1 Processor -- start 2024-03-31T15:06:35.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.547+0000 7fe5f26a2000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ea1cb949b0 0x55ea1cbe8420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:35.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.547+0000 7fe5f26a2000 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ea1cbe12d0 0x55ea1cbe8960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:35.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.547+0000 7fe5f26a2000 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55ea1cbe1bf0 0x55ea1cb99460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:35.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.547+0000 7fe5f26a2000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55ea1c9a41e0 con 0x55ea1cb949b0 2024-03-31T15:06:35.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.547+0000 7fe5f26a2000 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55ea1c9e2ab0 con 0x55ea1cbe12d0 2024-03-31T15:06:35.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.547+0000 7fe5f26a2000 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x55ea1cba0000 con 0x55ea1cbe1bf0 2024-03-31T15:06:35.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.547+0000 7fe5f26a2000 10 client.0 _ll_register_callbacks cb 0x55ea1cb82370 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T15:06:35.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.547+0000 7fe5e7e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ea1cb949b0 0x55ea1cbe8420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:35.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.547+0000 7fe5e7e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ea1cb949b0 0x55ea1cbe8420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49268/0 (socket says 172.21.15.16:49268) 2024-03-31T15:06:35.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.547+0000 7fe5e7e00640 1 -- 172.21.15.16:0/1726917771 learned_addr learned my addr 172.21.15.16:0/1726917771 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:06:35.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.547+0000 7fe5ed200640 1 --2- 172.21.15.16:0/1726917771 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55ea1cbe1bf0 0x55ea1cb99460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:35.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.547+0000 7fe5e7400640 1 --2- 172.21.15.16:0/1726917771 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ea1cbe12d0 0x55ea1cbe8960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:35.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.547+0000 7fe5e7e00640 1 -- 172.21.15.16:0/1726917771 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55ea1cbe1bf0 msgr2=0x55ea1cb99460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:06:35.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.547+0000 7fe5e7e00640 1 --2- 172.21.15.16:0/1726917771 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55ea1cbe1bf0 0x55ea1cb99460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:35.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.547+0000 7fe5e7e00640 1 -- 172.21.15.16:0/1726917771 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ea1cbe12d0 msgr2=0x55ea1cbe8960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:06:35.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.547+0000 7fe5e7e00640 1 --2- 172.21.15.16:0/1726917771 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ea1cbe12d0 0x55ea1cbe8960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:35.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.547+0000 7fe5e7e00640 1 -- 172.21.15.16:0/1726917771 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ea1c9a2b70 con 0x55ea1cb949b0 2024-03-31T15:06:35.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.547+0000 7fe5e7e00640 1 --2- 172.21.15.16:0/1726917771 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ea1cb949b0 0x55ea1cbe8420 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7fe5d8007450 tx=0x7fe5d8005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=568f29fa47deb14c server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:06:35.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.547+0000 7fe5d1600640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-31T15:06:35.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.547+0000 7fe5d1600640 1 -- 172.21.15.16:0/1726917771 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5d8027060 con 0x55ea1cb949b0 2024-03-31T15:06:35.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.547+0000 7fe5d1600640 1 -- 172.21.15.16:0/1726917771 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(56..56 src has 1..56) ==== 5709+0+0 (secure 0 0 0) 0x7fe5d8022b00 con 0x55ea1cb949b0 2024-03-31T15:06:35.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.547+0000 7fe5f26a2000 1 -- 172.21.15.16:0/1726917771 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe5d400d730 con 0x55ea1cb949b0 2024-03-31T15:06:35.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.547+0000 7fe5d1600640 1 -- 172.21.15.16:0/1726917771 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7fe5d800dea0 con 0x55ea1cb949b0 2024-03-31T15:06:35.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.547+0000 7fe5f26a2000 10 client.6366 fetch_fsmap learned FSMap version 34 2024-03-31T15:06:35.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.547+0000 7fe5f26a2000 1 -- 172.21.15.16:0/1726917771 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=2+}) -- 0x55ea1c983b90 con 0x55ea1cb949b0 2024-03-31T15:06:35.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.547+0000 7fe5d1600640 1 -- 172.21.15.16:0/1726917771 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe5d400d730 con 0x55ea1cb949b0 2024-03-31T15:06:35.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.547+0000 7fe5d1600640 1 -- 172.21.15.16:0/1726917771 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 34) ==== 3494+0+0 (secure 0 0 0) 0x7fe5d8012620 con 0x55ea1cb949b0 2024-03-31T15:06:35.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.547+0000 7fe5f26a2000 10 client.6366 fetch_fsmap finished waiting for FSMap version 34 2024-03-31T15:06:35.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.547+0000 7fe5f26a2000 10 client.6366 resolve_mds: resolved 4:0 to role '4:0' aka daemon mds.c 2024-03-31T15:06:35.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.547+0000 7fe5f26a2000 20 client.6366 populate_metadata read hostname 'smithi016' 2024-03-31T15:06:35.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.547+0000 7fe5f26a2000 1 --2- 172.21.15.16:0/1726917771 >> [v2:172.21.15.16:6834/2613117528,v1:172.21.15.16:6835/2613117528] conn(0x55ea1cb9a1d0 0x55ea1cb9a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:35.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.547+0000 7fe5f26a2000 4 client.6366 mds_command: new command op to 6150 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-31T15:06:35.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.547+0000 7fe5f26a2000 1 -- 172.21.15.16:0/1726917771 --> [v2:172.21.15.16:6834/2613117528,v1:172.21.15.16:6835/2613117528] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ea1c981dd0 con 0x55ea1cb9a1d0 2024-03-31T15:06:35.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.547+0000 7fe5e7400640 1 --2- 172.21.15.16:0/1726917771 >> [v2:172.21.15.16:6834/2613117528,v1:172.21.15.16:6835/2613117528] conn(0x55ea1cb9a1d0 0x55ea1cb9a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:35.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.551+0000 7fe5e7400640 1 --2- 172.21.15.16:0/1726917771 >> [v2:172.21.15.16:6834/2613117528,v1:172.21.15.16:6835/2613117528] conn(0x55ea1cb9a1d0 0x55ea1cb9a5e0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f58cdee790ccb36d server_cookie=ecc1cc359b882cf4 in_seq=0 out_seq=0 2024-03-31T15:06:35.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.551+0000 7fe5d1600640 10 client.6366 ms_handle_connect on v2:172.21.15.16:6834/2613117528 2024-03-31T15:06:35.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.551+0000 7fe5d1600640 1 -- 172.21.15.16:0/1726917771 <== mds.0 v2:172.21.15.16:6834/2613117528 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55ea1c981dd0 con 0x55ea1cb9a1d0 2024-03-31T15:06:35.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.551+0000 7fe5d1600640 10 client.6366 handle_command_reply: tid=0 2024-03-31T15:06:35.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.551+0000 7fe5f26a2000 1 client.6366 shutdown 2024-03-31T15:06:35.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.551+0000 7fe5f26a2000 1 -- 172.21.15.16:0/1726917771 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ea1cb949b0 msgr2=0x55ea1cbe8420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:35.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.551+0000 7fe5f26a2000 1 --2- 172.21.15.16:0/1726917771 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ea1cb949b0 0x55ea1cbe8420 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7fe5d8007450 tx=0x7fe5d8005800 comp rx=0 tx=0).stop 2024-03-31T15:06:35.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.551+0000 7fe5f26a2000 1 -- 172.21.15.16:0/1726917771 shutdown_connections 2024-03-31T15:06:35.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.551+0000 7fe5f26a2000 1 --2- 172.21.15.16:0/1726917771 >> [v2:172.21.15.16:6834/2613117528,v1:172.21.15.16:6835/2613117528] conn(0x55ea1cb9a1d0 0x55ea1cb9a5e0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:35.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.551+0000 7fe5f26a2000 1 --2- 172.21.15.16:0/1726917771 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55ea1cbe1bf0 0x55ea1cb99460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:35.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.551+0000 7fe5f26a2000 1 --2- 172.21.15.16:0/1726917771 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ea1cbe12d0 0x55ea1cbe8960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:35.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.551+0000 7fe5d1600640 0 client.6366 ms_handle_reset on v2:172.21.15.16:6834/2613117528 2024-03-31T15:06:35.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.551+0000 7fe5f26a2000 1 --2- 172.21.15.16:0/1726917771 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ea1cb949b0 0x55ea1cbe8420 unknown :-1 s=CLOSED pgs=685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:35.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.551+0000 7fe5f26a2000 1 -- 172.21.15.16:0/1726917771 >> 172.21.15.16:0/1726917771 conn(0x55ea1cbc2070 msgr2=0x55ea1cbc4460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:06:35.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.551+0000 7fe5f26a2000 1 -- 172.21.15.16:0/1726917771 shutdown_connections 2024-03-31T15:06:35.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.551+0000 7fe5f26a2000 1 -- 172.21.15.16:0/1726917771 wait complete. 2024-03-31T15:06:35.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.551+0000 7fe5f26a2000 20 client.6366 trim_cache size 0 max 16384 2024-03-31T15:06:35.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.567+0000 7fe5f26a2000 1 Processor -- start 2024-03-31T15:06:35.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.567+0000 7fe5f26a2000 1 -- start start 2024-03-31T15:06:35.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.567+0000 7fe5f26a2000 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55ea1cb949b0 0x55ea1cb97ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:35.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.567+0000 7fe5f26a2000 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ea1cbe12d0 0x55ea1cb98020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:35.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.567+0000 7fe5f26a2000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ea1cbe1bf0 0x55ea1cb98560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:35.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.567+0000 7fe5f26a2000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fe5e8154eb0 con 0x55ea1cbe1bf0 2024-03-31T15:06:35.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.567+0000 7fe5f26a2000 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fe5e8155500 con 0x55ea1cbe12d0 2024-03-31T15:06:35.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.567+0000 7fe5f26a2000 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x55ea1c981940 con 0x55ea1cb949b0 2024-03-31T15:06:35.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.567+0000 7fe5ed200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ea1cbe1bf0 0x55ea1cb98560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:35.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.567+0000 7fe5ed200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ea1cbe1bf0 0x55ea1cb98560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49270/0 (socket says 172.21.15.16:49270) 2024-03-31T15:06:35.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.567+0000 7fe5ed200640 1 -- 172.21.15.16:0/391505816 learned_addr learned my addr 172.21.15.16:0/391505816 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:06:35.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.571+0000 7fe5e7e00640 1 --2- 172.21.15.16:0/391505816 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55ea1cb949b0 0x55ea1cb97ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:35.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.571+0000 7fe5e7400640 1 --2- 172.21.15.16:0/391505816 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ea1cbe12d0 0x55ea1cb98020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:35.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.571+0000 7fe5ed200640 1 -- 172.21.15.16:0/391505816 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55ea1cb949b0 msgr2=0x55ea1cb97ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:06:35.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.571+0000 7fe5ed200640 1 --2- 172.21.15.16:0/391505816 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55ea1cb949b0 0x55ea1cb97ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:35.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.571+0000 7fe5ed200640 1 -- 172.21.15.16:0/391505816 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ea1cbe12d0 msgr2=0x55ea1cb98020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:06:35.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.571+0000 7fe5ed200640 1 --2- 172.21.15.16:0/391505816 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ea1cbe12d0 0x55ea1cb98020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:35.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.571+0000 7fe5ed200640 1 -- 172.21.15.16:0/391505816 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ea1cbe1a20 con 0x55ea1cbe1bf0 2024-03-31T15:06:35.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.571+0000 7fe5ed200640 1 --2- 172.21.15.16:0/391505816 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ea1cbe1bf0 0x55ea1cb98560 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7fe5e00edd40 tx=0x7fe5e00f8040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=19f4a44f30cc89c2 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:06:35.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.571+0000 7fe5d3e00640 1 -- 172.21.15.16:0/391505816 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5e00f5ce0 con 0x55ea1cbe1bf0 2024-03-31T15:06:35.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.571+0000 7fe5d3e00640 1 -- 172.21.15.16:0/391505816 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe5e00b3040 con 0x55ea1cbe1bf0 2024-03-31T15:06:35.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.571+0000 7fe5d3e00640 1 -- 172.21.15.16:0/391505816 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5e00fa510 con 0x55ea1cbe1bf0 2024-03-31T15:06:35.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.571+0000 7fe5f26a2000 1 -- 172.21.15.16:0/391505816 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ea1cbe1bf0 msgr2=0x55ea1cb98560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:35.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.571+0000 7fe5f26a2000 1 --2- 172.21.15.16:0/391505816 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ea1cbe1bf0 0x55ea1cb98560 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7fe5e00edd40 tx=0x7fe5e00f8040 comp rx=0 tx=0).stop 2024-03-31T15:06:35.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.571+0000 7fe5f26a2000 1 -- 172.21.15.16:0/391505816 shutdown_connections 2024-03-31T15:06:35.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.571+0000 7fe5f26a2000 1 --2- 172.21.15.16:0/391505816 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ea1cbe1bf0 0x55ea1cb98560 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:35.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.571+0000 7fe5f26a2000 1 --2- 172.21.15.16:0/391505816 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ea1cbe12d0 0x55ea1cb98020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:35.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.571+0000 7fe5f26a2000 1 --2- 172.21.15.16:0/391505816 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55ea1cb949b0 0x55ea1cb97ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:35.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.571+0000 7fe5f26a2000 1 -- 172.21.15.16:0/391505816 >> 172.21.15.16:0/391505816 conn(0x55ea1cbc2070 msgr2=0x55ea1cbf75b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:06:35.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.571+0000 7fe5f26a2000 1 -- 172.21.15.16:0/391505816 shutdown_connections 2024-03-31T15:06:35.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.571+0000 7fe5f26a2000 1 -- 172.21.15.16:0/391505816 wait complete. 2024-03-31T15:06:35.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.571+0000 7fe5f26a2000 1 -- start start 2024-03-31T15:06:35.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.571+0000 7fe5f26a2000 1 Processor -- start 2024-03-31T15:06:35.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.571+0000 7fe5f26a2000 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ea1cb949b0 0x55ea1cc51c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:35.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.571+0000 7fe5f26a2000 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55ea1cbe12d0 0x55ea1cc541b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:35.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.571+0000 7fe5f26a2000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ea1cbe1bf0 0x55ea1cc56700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:35.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.571+0000 7fe5f26a2000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55ea1cab5420 con 0x55ea1cbe1bf0 2024-03-31T15:06:35.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.571+0000 7fe5f26a2000 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55ea1c99d750 con 0x55ea1cb949b0 2024-03-31T15:06:35.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.571+0000 7fe5f26a2000 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x55ea1caa1250 con 0x55ea1cbe12d0 2024-03-31T15:06:35.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.571+0000 7fe5f26a2000 10 client.0 _ll_register_callbacks cb 0x7fe5d8001dd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T15:06:35.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.571+0000 7fe5ed200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ea1cbe1bf0 0x55ea1cc56700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:35.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.571+0000 7fe5ed200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ea1cbe1bf0 0x55ea1cc56700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49282/0 (socket says 172.21.15.16:49282) 2024-03-31T15:06:35.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.571+0000 7fe5ed200640 1 -- 172.21.15.16:0/4140887173 learned_addr learned my addr 172.21.15.16:0/4140887173 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:06:35.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.571+0000 7fe5e7400640 1 --2- 172.21.15.16:0/4140887173 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55ea1cbe12d0 0x55ea1cc541b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:35.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.571+0000 7fe5e7e00640 1 --2- 172.21.15.16:0/4140887173 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ea1cb949b0 0x55ea1cc51c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:35.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.571+0000 7fe5ed200640 1 -- 172.21.15.16:0/4140887173 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55ea1cbe12d0 msgr2=0x55ea1cc541b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:06:35.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.571+0000 7fe5ed200640 1 --2- 172.21.15.16:0/4140887173 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55ea1cbe12d0 0x55ea1cc541b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:35.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.571+0000 7fe5ed200640 1 -- 172.21.15.16:0/4140887173 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ea1cb949b0 msgr2=0x55ea1cc51c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:06:35.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.571+0000 7fe5ed200640 1 --2- 172.21.15.16:0/4140887173 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ea1cb949b0 0x55ea1cc51c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:35.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.571+0000 7fe5ed200640 1 -- 172.21.15.16:0/4140887173 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ea1ca67140 con 0x55ea1cbe1bf0 2024-03-31T15:06:35.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.571+0000 7fe5e7e00640 1 --2- 172.21.15.16:0/4140887173 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ea1cb949b0 0x55ea1cc51c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:06:35.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.571+0000 7fe5ed200640 1 --2- 172.21.15.16:0/4140887173 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ea1cbe1bf0 0x55ea1cc56700 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7fe5e001cee0 tx=0x7fe5e0107000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7b0a7741ec420da server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:06:35.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.571+0000 7fe5d1600640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-31T15:06:35.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.575+0000 7fe5d1600640 1 -- 172.21.15.16:0/4140887173 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5e00f5ce0 con 0x55ea1cbe1bf0 2024-03-31T15:06:35.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.575+0000 7fe5d1600640 1 -- 172.21.15.16:0/4140887173 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(56..56 src has 1..56) ==== 5709+0+0 (secure 0 0 0) 0x7fe5e00edfa0 con 0x55ea1cbe1bf0 2024-03-31T15:06:35.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.575+0000 7fe5f26a2000 1 -- 172.21.15.16:0/4140887173 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe5e00b3040 con 0x55ea1cbe1bf0 2024-03-31T15:06:35.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.575+0000 7fe5d1600640 1 -- 172.21.15.16:0/4140887173 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7fe5e0106020 con 0x55ea1cbe1bf0 2024-03-31T15:06:35.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.575+0000 7fe5f26a2000 10 client.6372 fetch_fsmap learned FSMap version 34 2024-03-31T15:06:35.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.575+0000 7fe5f26a2000 1 -- 172.21.15.16:0/4140887173 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=2+}) -- 0x55ea1c9751e0 con 0x55ea1cbe1bf0 2024-03-31T15:06:35.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.575+0000 7fe5d1600640 1 -- 172.21.15.16:0/4140887173 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe5e00b3040 con 0x55ea1cbe1bf0 2024-03-31T15:06:35.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.575+0000 7fe5d1600640 1 -- 172.21.15.16:0/4140887173 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 34) ==== 3494+0+0 (secure 0 0 0) 0x7fe5e00fa080 con 0x55ea1cbe1bf0 2024-03-31T15:06:35.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.575+0000 7fe5f26a2000 10 client.6372 fetch_fsmap finished waiting for FSMap version 34 2024-03-31T15:06:35.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.575+0000 7fe5f26a2000 10 client.6372 resolve_mds: resolved 4:0 to role '4:0' aka daemon mds.c 2024-03-31T15:06:35.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.575+0000 7fe5f26a2000 20 client.6372 populate_metadata read hostname 'smithi016' 2024-03-31T15:06:35.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.575+0000 7fe5f26a2000 1 --2- 172.21.15.16:0/4140887173 >> [v2:172.21.15.16:6834/2613117528,v1:172.21.15.16:6835/2613117528] conn(0x55ea1cc56c40 0x55ea1cc59030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:06:35.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.575+0000 7fe5f26a2000 4 client.6372 mds_command: new command op to 6150 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-31T15:06:35.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.575+0000 7fe5f26a2000 1 -- 172.21.15.16:0/4140887173 --> [v2:172.21.15.16:6834/2613117528,v1:172.21.15.16:6835/2613117528] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55ea1c9f4040 con 0x55ea1cc56c40 2024-03-31T15:06:35.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.575+0000 7fe5e7e00640 1 --2- 172.21.15.16:0/4140887173 >> [v2:172.21.15.16:6834/2613117528,v1:172.21.15.16:6835/2613117528] conn(0x55ea1cc56c40 0x55ea1cc59030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:06:35.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.575+0000 7fe5e7e00640 1 --2- 172.21.15.16:0/4140887173 >> [v2:172.21.15.16:6834/2613117528,v1:172.21.15.16:6835/2613117528] conn(0x55ea1cc56c40 0x55ea1cc59030 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=44775215597fc5e5 server_cookie=768fd7f4e1188dae in_seq=0 out_seq=0 2024-03-31T15:06:35.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.575+0000 7fe5d1600640 10 client.6372 ms_handle_connect on v2:172.21.15.16:6834/2613117528 2024-03-31T15:06:35.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.575+0000 7fe5d1600640 1 -- 172.21.15.16:0/4140887173 <== mds.0 v2:172.21.15.16:6834/2613117528 1 ==== command_reply(tid 0: 0 ) ==== 8+0+253 (crc 0 0 0) 0x55ea1c9f4040 con 0x55ea1cc56c40 2024-03-31T15:06:35.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.575+0000 7fe5d1600640 10 client.6372 handle_command_reply: tid=0 2024-03-31T15:06:35.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.575+0000 7fe5f26a2000 1 client.6372 shutdown 2024-03-31T15:06:35.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.575+0000 7fe5f26a2000 1 -- 172.21.15.16:0/4140887173 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ea1cbe1bf0 msgr2=0x55ea1cc56700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:35.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.575+0000 7fe5f26a2000 1 --2- 172.21.15.16:0/4140887173 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ea1cbe1bf0 0x55ea1cc56700 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7fe5e001cee0 tx=0x7fe5e0107000 comp rx=0 tx=0).stop 2024-03-31T15:06:35.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.575+0000 7fe5f26a2000 1 -- 172.21.15.16:0/4140887173 shutdown_connections 2024-03-31T15:06:35.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.575+0000 7fe5f26a2000 1 --2- 172.21.15.16:0/4140887173 >> [v2:172.21.15.16:6834/2613117528,v1:172.21.15.16:6835/2613117528] conn(0x55ea1cc56c40 0x55ea1cc59030 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:35.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.575+0000 7fe5f26a2000 1 --2- 172.21.15.16:0/4140887173 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ea1cbe1bf0 0x55ea1cc56700 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:35.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.575+0000 7fe5f26a2000 1 --2- 172.21.15.16:0/4140887173 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55ea1cbe12d0 0x55ea1cc541b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:35.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.575+0000 7fe5f26a2000 1 --2- 172.21.15.16:0/4140887173 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ea1cb949b0 0x55ea1cc51c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:35.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.575+0000 7fe5d1600640 0 client.6372 ms_handle_reset on v2:172.21.15.16:6834/2613117528 2024-03-31T15:06:35.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.575+0000 7fe5f26a2000 1 -- 172.21.15.16:0/4140887173 >> 172.21.15.16:0/4140887173 conn(0x55ea1cbc2070 msgr2=0x55ea1cbe05a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:06:35.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.575+0000 7fe5f26a2000 1 -- 172.21.15.16:0/4140887173 shutdown_connections 2024-03-31T15:06:35.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.575+0000 7fe5f26a2000 1 -- 172.21.15.16:0/4140887173 wait complete. 2024-03-31T15:06:35.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.575+0000 7fe5f26a2000 20 client.6372 trim_cache size 0 max 16384 2024-03-31T15:06:35.574 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-03-31T15:06:35.574 INFO:teuthology.orchestra.run.smithi016.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-31T15:06:35.574 INFO:teuthology.orchestra.run.smithi016.stdout: "scrubs": { 2024-03-31T15:06:35.574 INFO:teuthology.orchestra.run.smithi016.stdout: "ed29a749-4538-4efa-b094-9d8f540791b3": { 2024-03-31T15:06:35.574 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-03-31T15:06:35.574 INFO:teuthology.orchestra.run.smithi016.stdout: "tag": "ed29a749-4538-4efa-b094-9d8f540791b3", 2024-03-31T15:06:35.575 INFO:teuthology.orchestra.run.smithi016.stdout: "options": "recursive" 2024-03-31T15:06:35.575 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-31T15:06:35.575 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-31T15:06:35.575 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-03-31T15:06:35.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.575+0000 7fe5ef200640 1 -- 172.21.15.16:0/1531084418 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fe5c80683b0 msgr2=0x7fe5c806a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:35.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.575+0000 7fe5ef200640 1 --2- 172.21.15.16:0/1531084418 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fe5c80683b0 0x7fe5c806a870 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fe5d8005890 tx=0x7fe5d8009040 comp rx=0 tx=0).stop 2024-03-31T15:06:35.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.575+0000 7fe5ef200640 1 -- 172.21.15.16:0/1531084418 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe5e8141590 msgr2=0x7fe5e815d630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:06:35.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.575+0000 7fe5ef200640 1 --2- 172.21.15.16:0/1531084418 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe5e8141590 0x7fe5e815d630 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7fe5e001a5a0 tx=0x7fe5e0019710 comp rx=0 tx=0).stop 2024-03-31T15:06:35.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.575+0000 7fe5ef200640 1 -- 172.21.15.16:0/1531084418 shutdown_connections 2024-03-31T15:06:35.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.575+0000 7fe5ef200640 1 --2- 172.21.15.16:0/1531084418 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fe5c80683b0 0x7fe5c806a870 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:35.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.575+0000 7fe5ef200640 1 --2- 172.21.15.16:0/1531084418 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe5e8141590 0x7fe5e815d630 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:35.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.575+0000 7fe5ef200640 1 --2- 172.21.15.16:0/1531084418 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe5e81411b0 0x7fe5e815d0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:35.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.575+0000 7fe5ef200640 1 --2- 172.21.15.16:0/1531084418 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fe5e8070030 0x7fe5e812c870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:06:35.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.575+0000 7fe5ef200640 1 -- 172.21.15.16:0/1531084418 >> 172.21.15.16:0/1531084418 conn(0x7fe5e8067930 msgr2=0x7fe5e815faa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:06:35.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.575+0000 7fe5ef200640 1 -- 172.21.15.16:0/1531084418 shutdown_connections 2024-03-31T15:06:35.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:06:35.575+0000 7fe5ef200640 1 -- 172.21.15.16:0/1531084418 wait complete. 2024-03-31T15:06:35.590 INFO:tasks.cephfs.filesystem:scrub status for tag:ed29a749-4538-4efa-b094-9d8f540791b3 - {'path': '/', 'tag': 'ed29a749-4538-4efa-b094-9d8f540791b3', 'options': 'recursive'} 2024-03-31T15:06:36.906 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-31T15:06:36.909 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-31T15:06:38.307 INFO:tasks.ceph.mon.a.smithi016.stderr:2024-03-31T15:06:38.311+0000 7f7dcee00640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 MDSs report damaged metadata (MDS_DAMAGE) 2024-03-31T15:07:05.591 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:07:05.592 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.4:0 scrub status 2024-03-31T15:07:05.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.754+0000 7f5ab7e00640 1 Processor -- start 2024-03-31T15:07:05.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.754+0000 7f5ab7e00640 1 -- start start 2024-03-31T15:07:05.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.758+0000 7f5ab7e00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f5ab815c300 0x7f5ab80cfb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:05.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.758+0000 7f5ab7e00640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5ab80cca30 con 0x7f5ab80cd7b0 2024-03-31T15:07:05.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.758+0000 7f5ab7e00640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5ab80d0140 con 0x7f5ab80cdc10 2024-03-31T15:07:05.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.758+0000 7f5ab7e00640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f5ab80d2db0 con 0x7f5ab815c300 2024-03-31T15:07:05.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.758+0000 7f5ab6000640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f5ab80cd7b0 0x7f5ab806f500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:33718/0 (socket says 172.21.15.16:33718) 2024-03-31T15:07:05.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.758+0000 7f5ab6000640 1 -- 172.21.15.16:0/10333954 learned_addr learned my addr 172.21.15.16:0/10333954 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:07:05.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.758+0000 7f5ab6a00640 1 --2- 172.21.15.16:0/10333954 >> v2:172.21.15.89:3300/0 conn(0x7f5ab815c300 0x7f5ab80cfb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:05.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.758+0000 7f5ab5600640 1 -- 172.21.15.16:0/10333954 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2923246972 0 0) 0x7f5ab80cca30 con 0x7f5ab80cd7b0 2024-03-31T15:07:05.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.758+0000 7f5ab5600640 1 -- 172.21.15.16:0/10333954 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5a9c003530 con 0x7f5ab80cd7b0 2024-03-31T15:07:05.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.758+0000 7f5ab5600640 1 -- 172.21.15.16:0/10333954 <== mon.1 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2509212395 0 0) 0x7f5ab80d0140 con 0x7f5ab80cdc10 2024-03-31T15:07:05.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.758+0000 7f5ab5600640 1 -- 172.21.15.16:0/10333954 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5ab80cca30 con 0x7f5ab80cdc10 2024-03-31T15:07:05.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.758+0000 7f5ab6a00640 1 -- 172.21.15.16:0/10333954 >> v1:172.21.15.89:6789/0 conn(0x7f5ab80cdc10 legacy=0x7f5ab815fe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:05.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.758+0000 7f5ab6a00640 1 -- 172.21.15.16:0/10333954 >> v1:172.21.15.16:6789/0 conn(0x7f5ab80cd7b0 legacy=0x7f5ab806f500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:05.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.758+0000 7f5ab5600640 1 -- 172.21.15.16:0/10333954 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2320583743 0 0) 0x7f5a9c003530 con 0x7f5ab80cd7b0 2024-03-31T15:07:05.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.758+0000 7f5ab6a00640 1 -- 172.21.15.16:0/10333954 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ab80cccd0 con 0x7f5ab815c300 2024-03-31T15:07:05.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.762+0000 7f5ab6a00640 1 --2- 172.21.15.16:0/10333954 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5ab815c300 0x7f5ab80cfb60 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f5aa80097c0 tx=0x7f5aa801c3c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3f0a8b741e69b124 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:07:05.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.762+0000 7f5ab5600640 1 -- 172.21.15.16:0/10333954 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5aa80b0070 con 0x7f5ab815c300 2024-03-31T15:07:05.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.762+0000 7f5ab5600640 1 -- 172.21.15.16:0/10333954 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5aa80ab400 con 0x7f5ab815c300 2024-03-31T15:07:05.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.762+0000 7f5ab5600640 1 -- 172.21.15.16:0/10333954 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ab8160d90 con 0x7f5ab815c300 2024-03-31T15:07:05.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.762+0000 7f5ab7e00640 1 -- 172.21.15.16:0/10333954 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5ab815c300 msgr2=0x7f5ab80cfb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:05.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.762+0000 7f5ab7e00640 1 --2- 172.21.15.16:0/10333954 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5ab815c300 0x7f5ab80cfb60 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f5aa80097c0 tx=0x7f5aa801c3c0 comp rx=0 tx=0).stop 2024-03-31T15:07:05.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.762+0000 7f5ab7e00640 1 -- 172.21.15.16:0/10333954 shutdown_connections 2024-03-31T15:07:05.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.762+0000 7f5ab7e00640 1 --2- 172.21.15.16:0/10333954 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5ab815c300 0x7f5ab80cfb60 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:05.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.762+0000 7f5ab7e00640 1 -- 172.21.15.16:0/10333954 >> 172.21.15.16:0/10333954 conn(0x7f5ab80d1f90 msgr2=0x7f5ab80d23a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:07:05.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.762+0000 7f5ab7e00640 1 -- 172.21.15.16:0/10333954 shutdown_connections 2024-03-31T15:07:05.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.762+0000 7f5ab7e00640 1 -- 172.21.15.16:0/10333954 wait complete. 2024-03-31T15:07:05.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.766+0000 7f5ab7e00640 1 Processor -- start 2024-03-31T15:07:05.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.766+0000 7f5ab7e00640 1 -- start start 2024-03-31T15:07:05.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.766+0000 7f5ab7e00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f5ab80cd7b0 0x7f5ab806e1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:05.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.766+0000 7f5ab7e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5ab815c300 0x7f5ab815b1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:05.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.766+0000 7f5ab7e00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5ab815b730 0x7f5ab8154db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:05.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.766+0000 7f5ab7e00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f5ab816a550 con 0x7f5ab815c300 2024-03-31T15:07:05.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.766+0000 7f5ab7e00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f5ab816a3d0 con 0x7f5ab815b730 2024-03-31T15:07:05.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.766+0000 7f5ab7e00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f5ab816a6d0 con 0x7f5ab80cd7b0 2024-03-31T15:07:05.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.766+0000 7f5ab6000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5ab815c300 0x7f5ab815b1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:05.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.766+0000 7f5ab6000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5ab815c300 0x7f5ab815b1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:52986/0 (socket says 172.21.15.16:52986) 2024-03-31T15:07:05.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.766+0000 7f5ab6000640 1 -- 172.21.15.16:0/1854764541 learned_addr learned my addr 172.21.15.16:0/1854764541 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:07:05.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.766+0000 7f5ab7400640 1 --2- 172.21.15.16:0/1854764541 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5ab815b730 0x7f5ab8154db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:05.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.766+0000 7f5ab6a00640 1 --2- 172.21.15.16:0/1854764541 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f5ab80cd7b0 0x7f5ab806e1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:05.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.766+0000 7f5ab6000640 1 -- 172.21.15.16:0/1854764541 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f5ab80cd7b0 msgr2=0x7f5ab806e1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:05.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.766+0000 7f5ab6000640 1 --2- 172.21.15.16:0/1854764541 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f5ab80cd7b0 0x7f5ab806e1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:05.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.766+0000 7f5ab6000640 1 -- 172.21.15.16:0/1854764541 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5ab815b730 msgr2=0x7f5ab8154db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:05.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.766+0000 7f5ab6000640 1 --2- 172.21.15.16:0/1854764541 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5ab815b730 0x7f5ab8154db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:05.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.766+0000 7f5ab6000640 1 -- 172.21.15.16:0/1854764541 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ab80b6bd0 con 0x7f5ab815c300 2024-03-31T15:07:05.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.766+0000 7f5ab6000640 1 --2- 172.21.15.16:0/1854764541 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5ab815c300 0x7f5ab815b1f0 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f5aa400cac0 tx=0x7f5aa400cf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:07:05.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.766+0000 7f5aaea00640 1 -- 172.21.15.16:0/1854764541 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5aa4005cf0 con 0x7f5ab815c300 2024-03-31T15:07:05.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.766+0000 7f5aaea00640 1 -- 172.21.15.16:0/1854764541 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5aa4007070 con 0x7f5ab815c300 2024-03-31T15:07:05.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.766+0000 7f5ab7e00640 1 -- 172.21.15.16:0/1854764541 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5ab80b6eb0 con 0x7f5ab815c300 2024-03-31T15:07:05.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.766+0000 7f5aaea00640 1 -- 172.21.15.16:0/1854764541 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5aa40074a0 con 0x7f5ab815c300 2024-03-31T15:07:05.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.766+0000 7f5ab7e00640 1 -- 172.21.15.16:0/1854764541 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5ab80b7470 con 0x7f5ab815c300 2024-03-31T15:07:05.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.770+0000 7f5aaea00640 1 -- 172.21.15.16:0/1854764541 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f5aa400a640 con 0x7f5ab815c300 2024-03-31T15:07:05.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.770+0000 7f5aaea00640 1 --2- 172.21.15.16:0/1854764541 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f5a98068380 0x7f5a9806a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:05.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.770+0000 7f5aaea00640 1 -- 172.21.15.16:0/1854764541 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 5709+0+0 (secure 0 0 0) 0x7f5aa40868f0 con 0x7f5ab815c300 2024-03-31T15:07:05.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.770+0000 7f5ac0b08000 1 Processor -- start 2024-03-31T15:07:05.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.770+0000 7f5ac0b08000 1 -- start start 2024-03-31T15:07:05.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.770+0000 7f5ab6a00640 1 --2- 172.21.15.16:0/1854764541 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f5a98068380 0x7f5a9806a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:05.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.774+0000 7f5ac0b08000 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e5cd1f22d0 0x55e5cd1f26b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:05.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.774+0000 7f5ac0b08000 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55e5cd1f2bf0 0x55e5cd1a5470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:05.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.774+0000 7f5ac0b08000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e5cd1a59b0 0x55e5cd1a5d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:05.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.774+0000 7f5ac0b08000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55e5cd08eaf0 con 0x55e5cd1a59b0 2024-03-31T15:07:05.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.774+0000 7f5ac0b08000 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55e5ccff7b00 con 0x55e5cd1f22d0 2024-03-31T15:07:05.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.774+0000 7f5ac0b08000 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x55e5ccf94940 con 0x55e5cd1f2bf0 2024-03-31T15:07:05.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.774+0000 7f5ab6a00640 1 --2- 172.21.15.16:0/1854764541 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f5a98068380 0x7f5a9806a840 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f5aa80ad000 tx=0x7f5aa80a6000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:07:05.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.774+0000 7f5ab7400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e5cd1a59b0 0x55e5cd1a5d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:05.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.774+0000 7f5ab6000640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55e5cd1f2bf0 0x55e5cd1a5470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:05.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.774+0000 7f5ab6a00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e5cd1f22d0 0x55e5cd1f26b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:05.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.774+0000 7f5ab6000640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55e5cd1f2bf0 0x55e5cd1a5470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3301/0 says I am v2:172.21.15.16:46524/0 (socket says 172.21.15.16:46524) 2024-03-31T15:07:05.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.774+0000 7f5ab6000640 1 -- 172.21.15.16:0/179052604 learned_addr learned my addr 172.21.15.16:0/179052604 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:07:05.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.774+0000 7f5ab7400640 1 -- 172.21.15.16:0/179052604 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55e5cd1f2bf0 msgr2=0x55e5cd1a5470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:07:05.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.774+0000 7f5ab7400640 1 --2- 172.21.15.16:0/179052604 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55e5cd1f2bf0 0x55e5cd1a5470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:05.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.774+0000 7f5ab7400640 1 -- 172.21.15.16:0/179052604 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e5cd1f22d0 msgr2=0x55e5cd1f26b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:07:05.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.774+0000 7f5ab7400640 1 --2- 172.21.15.16:0/179052604 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e5cd1f22d0 0x55e5cd1f26b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:05.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.774+0000 7f5ab7400640 1 -- 172.21.15.16:0/179052604 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e5ccf7d380 con 0x55e5cd1a59b0 2024-03-31T15:07:05.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.774+0000 7f5ab7400640 1 --2- 172.21.15.16:0/179052604 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e5cd1a59b0 0x55e5cd1a5d90 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f5ab80723e0 tx=0x7f5ab0001fe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b96637cce63d236 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:07:05.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.774+0000 7f5aa3400640 1 -- 172.21.15.16:0/179052604 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ab0019070 con 0x55e5cd1a59b0 2024-03-31T15:07:05.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.774+0000 7f5aa3400640 1 -- 172.21.15.16:0/179052604 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5ab0003b90 con 0x55e5cd1a59b0 2024-03-31T15:07:05.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.774+0000 7f5aa3400640 1 -- 172.21.15.16:0/179052604 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ab00050d0 con 0x55e5cd1a59b0 2024-03-31T15:07:05.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.774+0000 7f5ac0b08000 1 -- 172.21.15.16:0/179052604 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e5cd1a59b0 msgr2=0x55e5cd1a5d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:05.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.774+0000 7f5ac0b08000 1 --2- 172.21.15.16:0/179052604 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e5cd1a59b0 0x55e5cd1a5d90 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f5ab80723e0 tx=0x7f5ab0001fe0 comp rx=0 tx=0).stop 2024-03-31T15:07:05.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.774+0000 7f5ac0b08000 1 -- 172.21.15.16:0/179052604 shutdown_connections 2024-03-31T15:07:05.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.774+0000 7f5ac0b08000 1 --2- 172.21.15.16:0/179052604 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e5cd1a59b0 0x55e5cd1a5d90 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:05.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.774+0000 7f5ac0b08000 1 --2- 172.21.15.16:0/179052604 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55e5cd1f2bf0 0x55e5cd1a5470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:05.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.774+0000 7f5ac0b08000 1 --2- 172.21.15.16:0/179052604 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e5cd1f22d0 0x55e5cd1f26b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:05.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.774+0000 7f5ac0b08000 1 -- 172.21.15.16:0/179052604 >> 172.21.15.16:0/179052604 conn(0x55e5cd1d3070 msgr2=0x55e5cd1d5460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:07:05.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.774+0000 7f5ac0b08000 1 -- 172.21.15.16:0/179052604 shutdown_connections 2024-03-31T15:07:05.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.774+0000 7f5ac0b08000 1 -- 172.21.15.16:0/179052604 wait complete. 2024-03-31T15:07:05.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.774+0000 7f5ac0b08000 1 -- start start 2024-03-31T15:07:05.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.778+0000 7f5ac0b08000 1 Processor -- start 2024-03-31T15:07:05.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.778+0000 7f5ac0b08000 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55e5cd1a59b0 0x55e5cd1f9630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:05.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.778+0000 7f5ac0b08000 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e5cd1f22d0 0x55e5cd1f9b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:05.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.778+0000 7f5ac0b08000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e5cd1f2bf0 0x55e5cd1aa280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:05.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.778+0000 7f5ac0b08000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55e5ccfb51e0 con 0x55e5cd1f2bf0 2024-03-31T15:07:05.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.778+0000 7f5ac0b08000 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55e5ccff3ab0 con 0x55e5cd1f22d0 2024-03-31T15:07:05.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.778+0000 7f5ac0b08000 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x55e5cd1b1000 con 0x55e5cd1a59b0 2024-03-31T15:07:05.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.778+0000 7f5ac0b08000 10 client.0 _ll_register_callbacks cb 0x55e5cd193370 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T15:07:05.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.778+0000 7f5ab7400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e5cd1f2bf0 0x55e5cd1aa280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:05.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.778+0000 7f5ab6a00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55e5cd1a59b0 0x55e5cd1f9630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:05.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.778+0000 7f5ab7400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e5cd1f2bf0 0x55e5cd1aa280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53008/0 (socket says 172.21.15.16:53008) 2024-03-31T15:07:05.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.778+0000 7f5ab7400640 1 -- 172.21.15.16:0/3677373567 learned_addr learned my addr 172.21.15.16:0/3677373567 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:07:05.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.778+0000 7f5ab6a00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55e5cd1a59b0 0x55e5cd1f9630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3301/0 says I am v2:172.21.15.16:46528/0 (socket says 172.21.15.16:46528) 2024-03-31T15:07:05.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.778+0000 7f5ab6000640 1 --2- 172.21.15.16:0/3677373567 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e5cd1f22d0 0x55e5cd1f9b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:05.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.778+0000 7f5ab7400640 1 -- 172.21.15.16:0/3677373567 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55e5cd1a59b0 msgr2=0x55e5cd1f9630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:07:05.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.778+0000 7f5ab7400640 1 --2- 172.21.15.16:0/3677373567 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55e5cd1a59b0 0x55e5cd1f9630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:05.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.778+0000 7f5ab7400640 1 -- 172.21.15.16:0/3677373567 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e5cd1f22d0 msgr2=0x55e5cd1f9b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:07:05.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.778+0000 7f5ab7400640 1 --2- 172.21.15.16:0/3677373567 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e5cd1f22d0 0x55e5cd1f9b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:05.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.778+0000 7f5ab7400640 1 -- 172.21.15.16:0/3677373567 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e5ccfb3b70 con 0x55e5cd1f2bf0 2024-03-31T15:07:05.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.778+0000 7f5ab7400640 1 --2- 172.21.15.16:0/3677373567 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e5cd1f2bf0 0x55e5cd1aa280 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f5ab0009010 tx=0x7f5ab0001260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e8f909c7480169a1 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:07:05.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.778+0000 7f5a97e00640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-31T15:07:05.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.778+0000 7f5a97e00640 1 -- 172.21.15.16:0/3677373567 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ab0023020 con 0x55e5cd1f2bf0 2024-03-31T15:07:05.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.778+0000 7f5a97e00640 1 -- 172.21.15.16:0/3677373567 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(56..56 src has 1..56) ==== 5709+0+0 (secure 0 0 0) 0x7f5ab0005480 con 0x55e5cd1f2bf0 2024-03-31T15:07:05.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.778+0000 7f5ac0b08000 1 -- 172.21.15.16:0/3677373567 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5ab0003b90 con 0x55e5cd1f2bf0 2024-03-31T15:07:05.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.778+0000 7f5a97e00640 1 -- 172.21.15.16:0/3677373567 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f5ab0019040 con 0x55e5cd1f2bf0 2024-03-31T15:07:05.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.778+0000 7f5ac0b08000 10 client.6384 fetch_fsmap learned FSMap version 35 2024-03-31T15:07:05.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.778+0000 7f5ac0b08000 1 -- 172.21.15.16:0/3677373567 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=2+}) -- 0x55e5ccf94b90 con 0x55e5cd1f2bf0 2024-03-31T15:07:05.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.778+0000 7f5a97e00640 1 -- 172.21.15.16:0/3677373567 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5ab0003b90 con 0x55e5cd1f2bf0 2024-03-31T15:07:05.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.778+0000 7f5a97e00640 1 -- 172.21.15.16:0/3677373567 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 35) ==== 3494+0+0 (secure 0 0 0) 0x7f5ab0005680 con 0x55e5cd1f2bf0 2024-03-31T15:07:05.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.782+0000 7f5ac0b08000 10 client.6384 fetch_fsmap finished waiting for FSMap version 35 2024-03-31T15:07:05.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.782+0000 7f5ac0b08000 10 client.6384 resolve_mds: resolved 4:0 to role '4:0' aka daemon mds.c 2024-03-31T15:07:05.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.782+0000 7f5ac0b08000 20 client.6384 populate_metadata read hostname 'smithi016' 2024-03-31T15:07:05.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.782+0000 7f5ac0b08000 1 --2- 172.21.15.16:0/3677373567 >> [v2:172.21.15.16:6834/2613117528,v1:172.21.15.16:6835/2613117528] conn(0x55e5cd1aac90 0x55e5cd1ab0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:05.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.782+0000 7f5ac0b08000 4 client.6384 mds_command: new command op to 6150 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-31T15:07:05.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.782+0000 7f5ac0b08000 1 -- 172.21.15.16:0/3677373567 --> [v2:172.21.15.16:6834/2613117528,v1:172.21.15.16:6835/2613117528] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e5ccf92dd0 con 0x55e5cd1aac90 2024-03-31T15:07:05.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.782+0000 7f5ab6a00640 1 --2- 172.21.15.16:0/3677373567 >> [v2:172.21.15.16:6834/2613117528,v1:172.21.15.16:6835/2613117528] conn(0x55e5cd1aac90 0x55e5cd1ab0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:05.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.782+0000 7f5a97e00640 10 client.6384 ms_handle_connect on v2:172.21.15.16:6834/2613117528 2024-03-31T15:07:05.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.782+0000 7f5ab6a00640 1 --2- 172.21.15.16:0/3677373567 >> [v2:172.21.15.16:6834/2613117528,v1:172.21.15.16:6835/2613117528] conn(0x55e5cd1aac90 0x55e5cd1ab0a0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=73b86d3bd337064a server_cookie=308ade859803582e in_seq=0 out_seq=0 2024-03-31T15:07:05.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.782+0000 7f5a97e00640 1 -- 172.21.15.16:0/3677373567 <== mds.0 v2:172.21.15.16:6834/2613117528 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55e5ccf92dd0 con 0x55e5cd1aac90 2024-03-31T15:07:05.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.782+0000 7f5a97e00640 10 client.6384 handle_command_reply: tid=0 2024-03-31T15:07:05.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.782+0000 7f5ac0b08000 1 client.6384 shutdown 2024-03-31T15:07:05.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.782+0000 7f5ac0b08000 1 -- 172.21.15.16:0/3677373567 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e5cd1f2bf0 msgr2=0x55e5cd1aa280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:05.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.782+0000 7f5ac0b08000 1 --2- 172.21.15.16:0/3677373567 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e5cd1f2bf0 0x55e5cd1aa280 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f5ab0009010 tx=0x7f5ab0001260 comp rx=0 tx=0).stop 2024-03-31T15:07:05.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.782+0000 7f5ac0b08000 1 -- 172.21.15.16:0/3677373567 shutdown_connections 2024-03-31T15:07:05.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.782+0000 7f5ac0b08000 1 --2- 172.21.15.16:0/3677373567 >> [v2:172.21.15.16:6834/2613117528,v1:172.21.15.16:6835/2613117528] conn(0x55e5cd1aac90 0x55e5cd1ab0a0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:05.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.782+0000 7f5ac0b08000 1 --2- 172.21.15.16:0/3677373567 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e5cd1f2bf0 0x55e5cd1aa280 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:05.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.782+0000 7f5a97e00640 0 client.6384 ms_handle_reset on v2:172.21.15.16:6834/2613117528 2024-03-31T15:07:05.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.782+0000 7f5ac0b08000 1 --2- 172.21.15.16:0/3677373567 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e5cd1f22d0 0x55e5cd1f9b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:05.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.782+0000 7f5ac0b08000 1 --2- 172.21.15.16:0/3677373567 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55e5cd1a59b0 0x55e5cd1f9630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:05.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.782+0000 7f5ac0b08000 1 -- 172.21.15.16:0/3677373567 >> 172.21.15.16:0/3677373567 conn(0x55e5cd1d3070 msgr2=0x55e5cd1d5460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:07:05.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.782+0000 7f5ac0b08000 1 -- 172.21.15.16:0/3677373567 shutdown_connections 2024-03-31T15:07:05.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.782+0000 7f5ac0b08000 1 -- 172.21.15.16:0/3677373567 wait complete. 2024-03-31T15:07:05.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.782+0000 7f5ac0b08000 20 client.6384 trim_cache size 0 max 16384 2024-03-31T15:07:05.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.818+0000 7f5ac0b08000 1 Processor -- start 2024-03-31T15:07:05.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.818+0000 7f5ac0b08000 1 -- start start 2024-03-31T15:07:05.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.818+0000 7f5ac0b08000 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e5cd1a59b0 0x55e5cd1d1f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:05.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.818+0000 7f5ac0b08000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e5cd1f22d0 0x55e5cd1f0c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:05.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.818+0000 7f5ac0b08000 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55e5cd1f2bf0 0x55e5cd1f1150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:05.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.818+0000 7f5ac0b08000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f5ab806d500 con 0x55e5cd1f22d0 2024-03-31T15:07:05.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.818+0000 7f5ac0b08000 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f5ab8165eb0 con 0x55e5cd1a59b0 2024-03-31T15:07:05.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.818+0000 7f5ac0b08000 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x55e5cd0b2250 con 0x55e5cd1f2bf0 2024-03-31T15:07:05.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.818+0000 7f5ab6000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e5cd1f22d0 0x55e5cd1f0c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:05.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.818+0000 7f5ab6000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e5cd1f22d0 0x55e5cd1f0c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53020/0 (socket says 172.21.15.16:53020) 2024-03-31T15:07:05.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.818+0000 7f5ab6000640 1 -- 172.21.15.16:0/3511772184 learned_addr learned my addr 172.21.15.16:0/3511772184 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:07:05.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.818+0000 7f5ab6a00640 1 --2- 172.21.15.16:0/3511772184 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e5cd1a59b0 0x55e5cd1d1f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:05.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.818+0000 7f5ab7400640 1 --2- 172.21.15.16:0/3511772184 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55e5cd1f2bf0 0x55e5cd1f1150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:05.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.818+0000 7f5ab6000640 1 -- 172.21.15.16:0/3511772184 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55e5cd1f2bf0 msgr2=0x55e5cd1f1150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:07:05.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.818+0000 7f5ab6000640 1 --2- 172.21.15.16:0/3511772184 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55e5cd1f2bf0 0x55e5cd1f1150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:05.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.818+0000 7f5ab6000640 1 -- 172.21.15.16:0/3511772184 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e5cd1a59b0 msgr2=0x55e5cd1d1f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:07:05.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.818+0000 7f5ab6000640 1 --2- 172.21.15.16:0/3511772184 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e5cd1a59b0 0x55e5cd1d1f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:05.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.818+0000 7f5ab6000640 1 -- 172.21.15.16:0/3511772184 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e5cd1f2a20 con 0x55e5cd1f22d0 2024-03-31T15:07:05.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.818+0000 7f5ab7400640 1 --2- 172.21.15.16:0/3511772184 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55e5cd1f2bf0 0x55e5cd1f1150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:07:05.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.822+0000 7f5ab6000640 1 --2- 172.21.15.16:0/3511772184 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e5cd1f22d0 0x55e5cd1f0c10 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f5aa4052ca0 tx=0x7f5aa405d040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4917191af2876460 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:07:05.818 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.822+0000 7f5aa2a00640 1 -- 172.21.15.16:0/3511772184 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5aa4082070 con 0x55e5cd1f22d0 2024-03-31T15:07:05.818 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.822+0000 7f5aa2a00640 1 -- 172.21.15.16:0/3511772184 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5aa405abf0 con 0x55e5cd1f22d0 2024-03-31T15:07:05.818 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.822+0000 7f5aa2a00640 1 -- 172.21.15.16:0/3511772184 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5aa405f420 con 0x55e5cd1f22d0 2024-03-31T15:07:05.818 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.822+0000 7f5ac0b08000 1 -- 172.21.15.16:0/3511772184 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e5cd1f22d0 msgr2=0x55e5cd1f0c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:05.818 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.822+0000 7f5ac0b08000 1 --2- 172.21.15.16:0/3511772184 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e5cd1f22d0 0x55e5cd1f0c10 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f5aa4052ca0 tx=0x7f5aa405d040 comp rx=0 tx=0).stop 2024-03-31T15:07:05.818 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.822+0000 7f5ac0b08000 1 -- 172.21.15.16:0/3511772184 shutdown_connections 2024-03-31T15:07:05.818 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.822+0000 7f5ac0b08000 1 --2- 172.21.15.16:0/3511772184 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55e5cd1f2bf0 0x55e5cd1f1150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:05.818 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.822+0000 7f5ac0b08000 1 --2- 172.21.15.16:0/3511772184 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e5cd1f22d0 0x55e5cd1f0c10 unknown :-1 s=CLOSED pgs=692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:05.818 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.822+0000 7f5ac0b08000 1 --2- 172.21.15.16:0/3511772184 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e5cd1a59b0 0x55e5cd1d1f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:05.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.822+0000 7f5ac0b08000 1 -- 172.21.15.16:0/3511772184 >> 172.21.15.16:0/3511772184 conn(0x55e5cd1d3070 msgr2=0x55e5cd1aa010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:07:05.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.822+0000 7f5ac0b08000 1 -- 172.21.15.16:0/3511772184 shutdown_connections 2024-03-31T15:07:05.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.822+0000 7f5ac0b08000 1 -- 172.21.15.16:0/3511772184 wait complete. 2024-03-31T15:07:05.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.822+0000 7f5ac0b08000 1 -- start start 2024-03-31T15:07:05.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.822+0000 7f5ac0b08000 1 Processor -- start 2024-03-31T15:07:05.822 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.822+0000 7f5ac0b08000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e5cd1a59b0 0x55e5cd1d1d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:05.822 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.822+0000 7f5ac0b08000 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e5cd1f22d0 0x55e5cd1f4fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:05.822 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.822+0000 7f5ac0b08000 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55e5cd1f2bf0 0x55e5cd1f5520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:05.822 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.822+0000 7f5ac0b08000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55e5ccfc3150 con 0x55e5cd1a59b0 2024-03-31T15:07:05.822 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.822+0000 7f5ac0b08000 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55e5cd0c6420 con 0x55e5cd1f22d0 2024-03-31T15:07:05.822 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.822+0000 7f5ac0b08000 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x55e5ccf92940 con 0x55e5cd1f2bf0 2024-03-31T15:07:05.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.822+0000 7f5ac0b08000 10 client.0 _ll_register_callbacks cb 0x7f5ab0017070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T15:07:05.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.822+0000 7f5ab6a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e5cd1a59b0 0x55e5cd1d1d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:05.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.822+0000 7f5ab6a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e5cd1a59b0 0x55e5cd1d1d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53024/0 (socket says 172.21.15.16:53024) 2024-03-31T15:07:05.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.822+0000 7f5ab6a00640 1 -- 172.21.15.16:0/1121057201 learned_addr learned my addr 172.21.15.16:0/1121057201 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:07:05.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.822+0000 7f5ab7400640 1 --2- 172.21.15.16:0/1121057201 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55e5cd1f2bf0 0x55e5cd1f5520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:05.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.822+0000 7f5ab6000640 1 --2- 172.21.15.16:0/1121057201 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e5cd1f22d0 0x55e5cd1f4fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:05.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.822+0000 7f5ab6a00640 1 -- 172.21.15.16:0/1121057201 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55e5cd1f2bf0 msgr2=0x55e5cd1f5520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:07:05.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.822+0000 7f5ab6a00640 1 --2- 172.21.15.16:0/1121057201 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55e5cd1f2bf0 0x55e5cd1f5520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:05.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.822+0000 7f5ab6a00640 1 -- 172.21.15.16:0/1121057201 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e5cd1f22d0 msgr2=0x55e5cd1f4fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:07:05.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.822+0000 7f5ab6a00640 1 --2- 172.21.15.16:0/1121057201 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e5cd1f22d0 0x55e5cd1f4fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:05.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.822+0000 7f5ab6a00640 1 -- 172.21.15.16:0/1121057201 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e5cd078140 con 0x55e5cd1a59b0 2024-03-31T15:07:05.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.822+0000 7f5ab6000640 1 --2- 172.21.15.16:0/1121057201 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e5cd1f22d0 0x55e5cd1f4fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:07:05.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.822+0000 7f5ab6a00640 1 --2- 172.21.15.16:0/1121057201 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e5cd1a59b0 0x55e5cd1d1d70 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f5aa80097c0 tx=0x7f5aa80046d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65425518b6b9d2b1 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:07:05.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.822+0000 7f5a97e00640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-31T15:07:05.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.826+0000 7f5a97e00640 1 -- 172.21.15.16:0/1121057201 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5aa80d1060 con 0x55e5cd1a59b0 2024-03-31T15:07:05.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.826+0000 7f5ac0b08000 1 -- 172.21.15.16:0/1121057201 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5aa405abf0 con 0x55e5cd1a59b0 2024-03-31T15:07:05.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.826+0000 7f5a97e00640 1 -- 172.21.15.16:0/1121057201 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(56..56 src has 1..56) ==== 5709+0+0 (secure 0 0 0) 0x7f5aa80ab8b0 con 0x55e5cd1a59b0 2024-03-31T15:07:05.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.826+0000 7f5a97e00640 1 -- 172.21.15.16:0/1121057201 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f5aa80d7050 con 0x55e5cd1a59b0 2024-03-31T15:07:05.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.826+0000 7f5ac0b08000 10 client.6390 fetch_fsmap learned FSMap version 35 2024-03-31T15:07:05.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.826+0000 7f5ac0b08000 1 -- 172.21.15.16:0/1121057201 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=2+}) -- 0x55e5ccf861e0 con 0x55e5cd1a59b0 2024-03-31T15:07:05.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.826+0000 7f5a97e00640 1 -- 172.21.15.16:0/1121057201 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5aa405abf0 con 0x55e5cd1a59b0 2024-03-31T15:07:05.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.826+0000 7f5a97e00640 1 -- 172.21.15.16:0/1121057201 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 35) ==== 3494+0+0 (secure 0 0 0) 0x7f5aa80da020 con 0x55e5cd1a59b0 2024-03-31T15:07:05.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.826+0000 7f5ac0b08000 10 client.6390 fetch_fsmap finished waiting for FSMap version 35 2024-03-31T15:07:05.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.826+0000 7f5ac0b08000 10 client.6390 resolve_mds: resolved 4:0 to role '4:0' aka daemon mds.c 2024-03-31T15:07:05.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.826+0000 7f5ac0b08000 20 client.6390 populate_metadata read hostname 'smithi016' 2024-03-31T15:07:05.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.826+0000 7f5ac0b08000 1 --2- 172.21.15.16:0/1121057201 >> [v2:172.21.15.16:6834/2613117528,v1:172.21.15.16:6835/2613117528] conn(0x55e5cd1f5a60 0x55e5cd26b130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:05.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.826+0000 7f5ac0b08000 4 client.6390 mds_command: new command op to 6150 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-31T15:07:05.825 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.826+0000 7f5ac0b08000 1 -- 172.21.15.16:0/1121057201 --> [v2:172.21.15.16:6834/2613117528,v1:172.21.15.16:6835/2613117528] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55e5cd005040 con 0x55e5cd1f5a60 2024-03-31T15:07:05.825 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.826+0000 7f5ab6000640 1 --2- 172.21.15.16:0/1121057201 >> [v2:172.21.15.16:6834/2613117528,v1:172.21.15.16:6835/2613117528] conn(0x55e5cd1f5a60 0x55e5cd26b130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:05.825 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.826+0000 7f5ab6000640 1 --2- 172.21.15.16:0/1121057201 >> [v2:172.21.15.16:6834/2613117528,v1:172.21.15.16:6835/2613117528] conn(0x55e5cd1f5a60 0x55e5cd26b130 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8468fa1f7e58bef8 server_cookie=9a0c4cf9d2f497d3 in_seq=0 out_seq=0 2024-03-31T15:07:05.825 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.826+0000 7f5a97e00640 10 client.6390 ms_handle_connect on v2:172.21.15.16:6834/2613117528 2024-03-31T15:07:05.825 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.826+0000 7f5a97e00640 1 -- 172.21.15.16:0/1121057201 <== mds.0 v2:172.21.15.16:6834/2613117528 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55e5cd005040 con 0x55e5cd1f5a60 2024-03-31T15:07:05.825 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.826+0000 7f5a97e00640 10 client.6390 handle_command_reply: tid=0 2024-03-31T15:07:05.825 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.826+0000 7f5ac0b08000 1 client.6390 shutdown 2024-03-31T15:07:05.825 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.826+0000 7f5ac0b08000 1 -- 172.21.15.16:0/1121057201 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e5cd1a59b0 msgr2=0x55e5cd1d1d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:05.825 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.826+0000 7f5ac0b08000 1 --2- 172.21.15.16:0/1121057201 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e5cd1a59b0 0x55e5cd1d1d70 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f5aa80097c0 tx=0x7f5aa80046d0 comp rx=0 tx=0).stop 2024-03-31T15:07:05.825 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.826+0000 7f5ac0b08000 1 -- 172.21.15.16:0/1121057201 shutdown_connections 2024-03-31T15:07:05.825 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.826+0000 7f5ac0b08000 1 --2- 172.21.15.16:0/1121057201 >> [v2:172.21.15.16:6834/2613117528,v1:172.21.15.16:6835/2613117528] conn(0x55e5cd1f5a60 0x55e5cd26b130 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:05.825 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.826+0000 7f5ac0b08000 1 --2- 172.21.15.16:0/1121057201 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55e5cd1f2bf0 0x55e5cd1f5520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:05.825 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.826+0000 7f5ac0b08000 1 --2- 172.21.15.16:0/1121057201 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e5cd1f22d0 0x55e5cd1f4fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:05.825 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.826+0000 7f5a97e00640 0 client.6390 ms_handle_reset on v2:172.21.15.16:6834/2613117528 2024-03-31T15:07:05.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.826+0000 7f5ac0b08000 1 --2- 172.21.15.16:0/1121057201 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e5cd1a59b0 0x55e5cd1d1d70 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:05.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.826+0000 7f5ac0b08000 1 -- 172.21.15.16:0/1121057201 >> 172.21.15.16:0/1121057201 conn(0x55e5cd1d3070 msgr2=0x55e5cd1f0380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:07:05.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.826+0000 7f5ac0b08000 1 -- 172.21.15.16:0/1121057201 shutdown_connections 2024-03-31T15:07:05.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.826+0000 7f5ac0b08000 1 -- 172.21.15.16:0/1121057201 wait complete. 2024-03-31T15:07:05.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.826+0000 7f5ac0b08000 20 client.6390 trim_cache size 0 max 16384 2024-03-31T15:07:05.826 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-03-31T15:07:05.826 INFO:teuthology.orchestra.run.smithi016.stdout: "status": "no active scrubs running", 2024-03-31T15:07:05.826 INFO:teuthology.orchestra.run.smithi016.stdout: "scrubs": {} 2024-03-31T15:07:05.827 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-03-31T15:07:05.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.826+0000 7f5ab7e00640 1 -- 172.21.15.16:0/1854764541 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f5a98068380 msgr2=0x7f5a9806a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:05.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.826+0000 7f5ab7e00640 1 --2- 172.21.15.16:0/1854764541 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f5a98068380 0x7f5a9806a840 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f5aa80ad000 tx=0x7f5aa80a6000 comp rx=0 tx=0).stop 2024-03-31T15:07:05.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.826+0000 7f5ab7e00640 1 -- 172.21.15.16:0/1854764541 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5ab815c300 msgr2=0x7f5ab815b1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:05.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.826+0000 7f5ab7e00640 1 --2- 172.21.15.16:0/1854764541 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5ab815c300 0x7f5ab815b1f0 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f5aa400cac0 tx=0x7f5aa400cf80 comp rx=0 tx=0).stop 2024-03-31T15:07:05.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.826+0000 7f5ab7e00640 1 -- 172.21.15.16:0/1854764541 shutdown_connections 2024-03-31T15:07:05.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.826+0000 7f5ab7e00640 1 --2- 172.21.15.16:0/1854764541 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f5a98068380 0x7f5a9806a840 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:05.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.826+0000 7f5ab7e00640 1 --2- 172.21.15.16:0/1854764541 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5ab815b730 0x7f5ab8154db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:05.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.826+0000 7f5ab7e00640 1 --2- 172.21.15.16:0/1854764541 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5ab815c300 0x7f5ab815b1f0 unknown :-1 s=CLOSED pgs=689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:05.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.826+0000 7f5ab7e00640 1 --2- 172.21.15.16:0/1854764541 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f5ab80cd7b0 0x7f5ab806e1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:05.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.826+0000 7f5ab7e00640 1 -- 172.21.15.16:0/1854764541 >> 172.21.15.16:0/1854764541 conn(0x7f5ab80d1f90 msgr2=0x7f5ab80c5d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:07:05.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.826+0000 7f5ab7e00640 1 -- 172.21.15.16:0/1854764541 shutdown_connections 2024-03-31T15:07:05.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.826+0000 7f5ab7e00640 1 -- 172.21.15.16:0/1854764541 wait complete. 2024-03-31T15:07:05.847 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-31T15:07:05.848 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-31T15:07:05.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.994+0000 7f08b1a00640 1 Processor -- start 2024-03-31T15:07:05.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.994+0000 7f08b1a00640 1 -- start start 2024-03-31T15:07:05.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.994+0000 7f08b1a00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f08ac06bd30 0x7f08ac06e4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:05.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.994+0000 7f08b1a00640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f08ac06ec30 con 0x7f08ac158700 2024-03-31T15:07:05.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.994+0000 7f08b1a00640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f08ac06ef10 con 0x7f08ac0706d0 2024-03-31T15:07:05.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.994+0000 7f08b1a00640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f08ac06ea00 con 0x7f08ac06bd30 2024-03-31T15:07:05.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.994+0000 7f08abe00640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f08ac158700 0x7f08ac15f120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:33728/0 (socket says 172.21.15.16:33728) 2024-03-31T15:07:05.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.994+0000 7f08abe00640 1 -- 172.21.15.16:0/894563753 learned_addr learned my addr 172.21.15.16:0/894563753 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:07:05.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.994+0000 7f08b1000640 1 --2- 172.21.15.16:0/894563753 >> v2:172.21.15.16:3300/0 conn(0x7f08ac06bd30 0x7f08ac06e4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:05.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.994+0000 7f08aaa00640 1 -- 172.21.15.16:0/894563753 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2032452849 0 0) 0x7f08ac06ec30 con 0x7f08ac158700 2024-03-31T15:07:05.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.994+0000 7f08aaa00640 1 -- 172.21.15.16:0/894563753 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0890003530 con 0x7f08ac158700 2024-03-31T15:07:05.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.994+0000 7f08b1000640 1 -- 172.21.15.16:0/894563753 >> v1:172.21.15.89:6789/0 conn(0x7f08ac0706d0 legacy=0x7f08ac06b620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:05.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.994+0000 7f08b1000640 1 -- 172.21.15.16:0/894563753 >> v1:172.21.15.16:6789/0 conn(0x7f08ac158700 legacy=0x7f08ac15f120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:05.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.994+0000 7f08b1000640 1 -- 172.21.15.16:0/894563753 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08ac15e920 con 0x7f08ac06bd30 2024-03-31T15:07:05.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.998+0000 7f08b1000640 1 --2- 172.21.15.16:0/894563753 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f08ac06bd30 0x7f08ac06e4c0 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f08a4009d90 tx=0x7f08a401c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a67f2aa933b2a96 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:07:05.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.998+0000 7f08aaa00640 1 -- 172.21.15.16:0/894563753 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08a40b2070 con 0x7f08ac06bd30 2024-03-31T15:07:05.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.998+0000 7f08aaa00640 1 -- 172.21.15.16:0/894563753 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f08a40a5830 con 0x7f08ac06bd30 2024-03-31T15:07:05.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:05.998+0000 7f08aaa00640 1 -- 172.21.15.16:0/894563753 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08a40a5c00 con 0x7f08ac06bd30 2024-03-31T15:07:05.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.002+0000 7f08b1a00640 1 -- 172.21.15.16:0/894563753 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f08ac06bd30 msgr2=0x7f08ac06e4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:05.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.002+0000 7f08b1a00640 1 --2- 172.21.15.16:0/894563753 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f08ac06bd30 0x7f08ac06e4c0 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f08a4009d90 tx=0x7f08a401c970 comp rx=0 tx=0).stop 2024-03-31T15:07:05.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.002+0000 7f08b1a00640 1 -- 172.21.15.16:0/894563753 shutdown_connections 2024-03-31T15:07:05.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.002+0000 7f08b1a00640 1 --2- 172.21.15.16:0/894563753 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f08ac06bd30 0x7f08ac06e4c0 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:05.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.002+0000 7f08b1a00640 1 -- 172.21.15.16:0/894563753 >> 172.21.15.16:0/894563753 conn(0x7f08ac0b2b30 msgr2=0x7f08ac0b2f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:07:05.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.002+0000 7f08b1a00640 1 -- 172.21.15.16:0/894563753 shutdown_connections 2024-03-31T15:07:05.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.002+0000 7f08b1a00640 1 -- 172.21.15.16:0/894563753 wait complete. 2024-03-31T15:07:05.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.002+0000 7f08b1a00640 1 Processor -- start 2024-03-31T15:07:05.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.002+0000 7f08b1a00640 1 -- start start 2024-03-31T15:07:06.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.002+0000 7f08b1a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f08ac06bd30 0x7f08ac1626e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:06.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.002+0000 7f08b1a00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f08ac0706d0 0x7f08ac15a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:06.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.002+0000 7f08b1a00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f08ac158700 0x7f08ac14e300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:06.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.002+0000 7f08b1a00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f08ac15ec90 con 0x7f08ac06bd30 2024-03-31T15:07:06.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.002+0000 7f08b1a00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f08ac0d0bd0 con 0x7f08ac0706d0 2024-03-31T15:07:06.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.002+0000 7f08b1a00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f08ac072460 con 0x7f08ac158700 2024-03-31T15:07:06.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.002+0000 7f08abe00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f08ac06bd30 0x7f08ac1626e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:06.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.002+0000 7f08abe00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f08ac06bd30 0x7f08ac1626e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53028/0 (socket says 172.21.15.16:53028) 2024-03-31T15:07:06.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.002+0000 7f08abe00640 1 -- 172.21.15.16:0/1069241951 learned_addr learned my addr 172.21.15.16:0/1069241951 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:07:06.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.002+0000 7f08ab400640 1 --2- 172.21.15.16:0/1069241951 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f08ac0706d0 0x7f08ac15a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:06.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.006+0000 7f08b1000640 1 --2- 172.21.15.16:0/1069241951 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f08ac158700 0x7f08ac14e300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:06.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.006+0000 7f08abe00640 1 -- 172.21.15.16:0/1069241951 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f08ac158700 msgr2=0x7f08ac14e300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:06.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.006+0000 7f08abe00640 1 --2- 172.21.15.16:0/1069241951 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f08ac158700 0x7f08ac14e300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:06.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.006+0000 7f08abe00640 1 -- 172.21.15.16:0/1069241951 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f08ac0706d0 msgr2=0x7f08ac15a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:06.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.006+0000 7f08abe00640 1 --2- 172.21.15.16:0/1069241951 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f08ac0706d0 0x7f08ac15a2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:06.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.006+0000 7f08abe00640 1 -- 172.21.15.16:0/1069241951 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08ac15aa50 con 0x7f08ac06bd30 2024-03-31T15:07:06.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.006+0000 7f08abe00640 1 --2- 172.21.15.16:0/1069241951 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f08ac06bd30 0x7f08ac1626e0 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f089c00c420 tx=0x7f089c00c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:07:06.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.006+0000 7f08a3e00640 1 -- 172.21.15.16:0/1069241951 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f089c017070 con 0x7f08ac06bd30 2024-03-31T15:07:06.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.006+0000 7f08a3e00640 1 -- 172.21.15.16:0/1069241951 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f089c005cf0 con 0x7f08ac06bd30 2024-03-31T15:07:06.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.006+0000 7f08a3e00640 1 -- 172.21.15.16:0/1069241951 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f089c0072d0 con 0x7f08ac06bd30 2024-03-31T15:07:06.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.006+0000 7f08b1a00640 1 -- 172.21.15.16:0/1069241951 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f08ac15ad30 con 0x7f08ac06bd30 2024-03-31T15:07:06.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.006+0000 7f08b1a00640 1 -- 172.21.15.16:0/1069241951 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f08ac15b2f0 con 0x7f08ac06bd30 2024-03-31T15:07:06.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.006+0000 7f08b1a00640 1 -- 172.21.15.16:0/1069241951 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f08ac04f8c0 con 0x7f08ac06bd30 2024-03-31T15:07:06.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.006+0000 7f08a3e00640 1 -- 172.21.15.16:0/1069241951 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f089c01a070 con 0x7f08ac06bd30 2024-03-31T15:07:06.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.006+0000 7f08a3e00640 1 --2- 172.21.15.16:0/1069241951 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f088c068380 0x7f088c06a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:06.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.006+0000 7f08a3e00640 1 -- 172.21.15.16:0/1069241951 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 5709+0+0 (secure 0 0 0) 0x7f089c085cb0 con 0x7f08ac06bd30 2024-03-31T15:07:06.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.010+0000 7f08ab400640 1 --2- 172.21.15.16:0/1069241951 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f088c068380 0x7f088c06a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:06.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.010+0000 7f08ab400640 1 --2- 172.21.15.16:0/1069241951 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f088c068380 0x7f088c06a840 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f08ac0d0140 tx=0x7f0898009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:07:06.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.018+0000 7f08a3e00640 1 -- 172.21.15.16:0/1069241951 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f089c055c10 con 0x7f08ac06bd30 2024-03-31T15:07:06.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.218+0000 7f08b1a00640 1 -- 172.21.15.16:0/1069241951 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f08ac0af800 con 0x7f08ac06bd30 2024-03-31T15:07:06.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.218+0000 7f08a3e00640 1 -- 172.21.15.16:0/1069241951 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) ==== 94+0+5181 (secure 0 0 0) 0x7f089c059f10 con 0x7f08ac06bd30 2024-03-31T15:07:06.215 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:07:06.215 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":35,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5579,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6833/3721971977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":3721971977},{"type":"v1","addr":"172.21.15.16:6833","nonce":3721971977}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":29},{"gid":6109,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6836/1166206553","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":1166206553},{"type":"v1","addr":"172.21.15.89:6836","nonce":1166206553}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28},{"gid":6112,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/1826113083","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6835","nonce":1826113083},{"type":"v1","addr":"172.21.15.89:6837","nonce":1826113083}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":29}],"filesystems":[{"mdsmap":{"epoch":35,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-31T15:06:25.665001+0000","modified":"2024-03-31T15:06:38.311664+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6150},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6150":{"gid":6150,"name":"c","rank":0,"incarnation":30,"state":"up:active","state_seq":6,"addr":"172.21.15.16:6835/2613117528","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":2613117528},{"type":"v1","addr":"172.21.15.16:6835","nonce":2613117528}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6150,"qdb_cluster":[6150]},"id":4}]} 2024-03-31T15:07:06.216 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 35 2024-03-31T15:07:06.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.222+0000 7f08b1a00640 1 -- 172.21.15.16:0/1069241951 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f088c068380 msgr2=0x7f088c06a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:06.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.222+0000 7f08b1a00640 1 --2- 172.21.15.16:0/1069241951 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f088c068380 0x7f088c06a840 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f08ac0d0140 tx=0x7f0898009040 comp rx=0 tx=0).stop 2024-03-31T15:07:06.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.222+0000 7f08b1a00640 1 -- 172.21.15.16:0/1069241951 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f08ac06bd30 msgr2=0x7f08ac1626e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:06.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.222+0000 7f08b1a00640 1 --2- 172.21.15.16:0/1069241951 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f08ac06bd30 0x7f08ac1626e0 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f089c00c420 tx=0x7f089c00c8e0 comp rx=0 tx=0).stop 2024-03-31T15:07:06.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.222+0000 7f08b1a00640 1 -- 172.21.15.16:0/1069241951 shutdown_connections 2024-03-31T15:07:06.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.222+0000 7f08b1a00640 1 --2- 172.21.15.16:0/1069241951 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f088c068380 0x7f088c06a840 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:06.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.222+0000 7f08b1a00640 1 --2- 172.21.15.16:0/1069241951 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f08ac158700 0x7f08ac14e300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:06.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.222+0000 7f08b1a00640 1 --2- 172.21.15.16:0/1069241951 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f08ac0706d0 0x7f08ac15a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:06.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.222+0000 7f08b1a00640 1 --2- 172.21.15.16:0/1069241951 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f08ac06bd30 0x7f08ac1626e0 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:06.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.222+0000 7f08b1a00640 1 -- 172.21.15.16:0/1069241951 >> 172.21.15.16:0/1069241951 conn(0x7f08ac0b2b30 msgr2=0x7f08ac0ae890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:07:06.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.222+0000 7f08b1a00640 1 -- 172.21.15.16:0/1069241951 shutdown_connections 2024-03-31T15:07:06.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.222+0000 7f08b1a00640 1 -- 172.21.15.16:0/1069241951 wait complete. 2024-03-31T15:07:06.237 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c damage ls --format=json-pretty 2024-03-31T15:07:06.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.394+0000 7f7e44600640 1 Processor -- start 2024-03-31T15:07:06.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.394+0000 7f7e44600640 1 -- start start 2024-03-31T15:07:06.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.398+0000 7f7e44600640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f7e3c1567e0 0x7f7e3c072c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:06.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.398+0000 7f7e44600640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f7e3c06dbe0 0x7f7e3c071550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:06.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.398+0000 7f7e44600640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7e3c15c7e0 con 0x7f7e3c0731c0 2024-03-31T15:07:06.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.398+0000 7f7e44600640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f7e3c0736b0 con 0x7f7e3c1567e0 2024-03-31T15:07:06.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.398+0000 7f7e44600640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f7e3c15caa0 con 0x7f7e3c06dbe0 2024-03-31T15:07:06.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.398+0000 7f7e41c00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f7e3c1567e0 0x7f7e3c072c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:06.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.398+0000 7f7e41c00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f7e3c1567e0 0x7f7e3c072c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53032/0 (socket says 172.21.15.16:53032) 2024-03-31T15:07:06.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.398+0000 7f7e41c00640 1 -- 172.21.15.16:0/3306947084 learned_addr learned my addr 172.21.15.16:0/3306947084 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:07:06.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.398+0000 7f7e42600640 1 --2- 172.21.15.16:0/3306947084 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f7e3c06dbe0 0x7f7e3c071550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:06.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.398+0000 7f7e41c00640 1 -- 172.21.15.16:0/3306947084 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f7e3c06dbe0 msgr2=0x7f7e3c071550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:07:06.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.398+0000 7f7e41c00640 1 --2- 172.21.15.16:0/3306947084 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f7e3c06dbe0 0x7f7e3c071550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:06.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.398+0000 7f7e41c00640 1 -- 172.21.15.16:0/3306947084 >> v1:172.21.15.89:6789/0 conn(0x7f7e3c0731c0 legacy=0x7f7e3c06d4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:06.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.398+0000 7f7e41c00640 1 -- 172.21.15.16:0/3306947084 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e3c15cd80 con 0x7f7e3c1567e0 2024-03-31T15:07:06.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.402+0000 7f7e41c00640 1 --2- 172.21.15.16:0/3306947084 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7e3c1567e0 0x7f7e3c072c80 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f7e30009870 tx=0x7f7e3001c560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12d620470469fe7f server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:07:06.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.402+0000 7f7e3be00640 1 -- 172.21.15.16:0/3306947084 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e300b0070 con 0x7f7e3c1567e0 2024-03-31T15:07:06.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.402+0000 7f7e3be00640 1 -- 172.21.15.16:0/3306947084 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7e300ab600 con 0x7f7e3c1567e0 2024-03-31T15:07:06.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.402+0000 7f7e3be00640 1 -- 172.21.15.16:0/3306947084 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e300ab9d0 con 0x7f7e3c1567e0 2024-03-31T15:07:06.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.402+0000 7f7e44600640 1 -- 172.21.15.16:0/3306947084 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7e3c1567e0 msgr2=0x7f7e3c072c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:06.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.402+0000 7f7e44600640 1 --2- 172.21.15.16:0/3306947084 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7e3c1567e0 0x7f7e3c072c80 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f7e30009870 tx=0x7f7e3001c560 comp rx=0 tx=0).stop 2024-03-31T15:07:06.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.402+0000 7f7e44600640 1 -- 172.21.15.16:0/3306947084 shutdown_connections 2024-03-31T15:07:06.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.402+0000 7f7e44600640 1 --2- 172.21.15.16:0/3306947084 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f7e3c06dbe0 0x7f7e3c071550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:06.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.402+0000 7f7e44600640 1 --2- 172.21.15.16:0/3306947084 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7e3c1567e0 0x7f7e3c072c80 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:06.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.402+0000 7f7e44600640 1 -- 172.21.15.16:0/3306947084 >> 172.21.15.16:0/3306947084 conn(0x7f7e3c06ee40 msgr2=0x7f7e3c06b620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:07:06.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.402+0000 7f7e44600640 1 -- 172.21.15.16:0/3306947084 shutdown_connections 2024-03-31T15:07:06.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.402+0000 7f7e44600640 1 -- 172.21.15.16:0/3306947084 wait complete. 2024-03-31T15:07:06.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.402+0000 7f7e44600640 1 Processor -- start 2024-03-31T15:07:06.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.402+0000 7f7e44600640 1 -- start start 2024-03-31T15:07:06.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.406+0000 7f7e44600640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f7e3c06dbe0 0x7f7e3c15f2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:06.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.406+0000 7f7e44600640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7e3c0731c0 0x7f7e3c147830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:06.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.406+0000 7f7e44600640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7e3c1567e0 0x7f7e3c1534f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:06.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.406+0000 7f7e44600640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f7e3c1692f0 con 0x7f7e3c0731c0 2024-03-31T15:07:06.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.406+0000 7f7e44600640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f7e3c169170 con 0x7f7e3c1567e0 2024-03-31T15:07:06.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.406+0000 7f7e44600640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f7e3c169470 con 0x7f7e3c06dbe0 2024-03-31T15:07:06.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.406+0000 7f7e41200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7e3c0731c0 0x7f7e3c147830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:06.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.406+0000 7f7e41200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7e3c0731c0 0x7f7e3c147830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53044/0 (socket says 172.21.15.16:53044) 2024-03-31T15:07:06.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.406+0000 7f7e41c00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f7e3c06dbe0 0x7f7e3c15f2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:06.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.406+0000 7f7e41200640 1 -- 172.21.15.16:0/1188988204 learned_addr learned my addr 172.21.15.16:0/1188988204 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:07:06.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.406+0000 7f7e42600640 1 --2- 172.21.15.16:0/1188988204 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7e3c1567e0 0x7f7e3c1534f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:06.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.406+0000 7f7e41200640 1 -- 172.21.15.16:0/1188988204 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f7e3c06dbe0 msgr2=0x7f7e3c15f2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:06.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.406+0000 7f7e41200640 1 --2- 172.21.15.16:0/1188988204 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f7e3c06dbe0 0x7f7e3c15f2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:06.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.406+0000 7f7e41200640 1 -- 172.21.15.16:0/1188988204 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7e3c1567e0 msgr2=0x7f7e3c1534f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:06.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.406+0000 7f7e41200640 1 --2- 172.21.15.16:0/1188988204 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7e3c1567e0 0x7f7e3c1534f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:06.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.406+0000 7f7e41200640 1 -- 172.21.15.16:0/1188988204 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e3c1509e0 con 0x7f7e3c0731c0 2024-03-31T15:07:06.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.406+0000 7f7e41200640 1 --2- 172.21.15.16:0/1188988204 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7e3c0731c0 0x7f7e3c147830 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f7e2800c4d0 tx=0x7f7e2800c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:07:06.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.406+0000 7f7e39600640 1 -- 172.21.15.16:0/1188988204 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e280052d0 con 0x7f7e3c0731c0 2024-03-31T15:07:06.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.406+0000 7f7e39600640 1 -- 172.21.15.16:0/1188988204 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7e28005470 con 0x7f7e3c0731c0 2024-03-31T15:07:06.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.406+0000 7f7e39600640 1 -- 172.21.15.16:0/1188988204 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e28012830 con 0x7f7e3c0731c0 2024-03-31T15:07:06.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.406+0000 7f7e44600640 1 -- 172.21.15.16:0/1188988204 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7e3c150cc0 con 0x7f7e3c0731c0 2024-03-31T15:07:06.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.406+0000 7f7e44600640 1 -- 172.21.15.16:0/1188988204 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7e3c147d70 con 0x7f7e3c0731c0 2024-03-31T15:07:06.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.410+0000 7f7e39600640 1 -- 172.21.15.16:0/1188988204 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f7e280129d0 con 0x7f7e3c0731c0 2024-03-31T15:07:06.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.410+0000 7f7e479c4000 1 Processor -- start 2024-03-31T15:07:06.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.410+0000 7f7e479c4000 1 -- start start 2024-03-31T15:07:06.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.410+0000 7f7e39600640 1 --2- 172.21.15.16:0/1188988204 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f7e1c068380 0x7f7e1c06a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:06.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.410+0000 7f7e39600640 1 -- 172.21.15.16:0/1188988204 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 5709+0+0 (secure 0 0 0) 0x7f7e28086550 con 0x7f7e3c0731c0 2024-03-31T15:07:06.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.410+0000 7f7e42600640 1 --2- 172.21.15.16:0/1188988204 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f7e1c068380 0x7f7e1c06a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:06.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.410+0000 7f7e479c4000 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55e9b4947980 0x55e9b4947d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:06.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.410+0000 7f7e479c4000 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e9b49482a0 0x55e9b48fadd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:06.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.410+0000 7f7e479c4000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e9b48fb310 0x55e9b48fb6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:06.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.410+0000 7f7e479c4000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55e9b474bc50 con 0x55e9b48fb310 2024-03-31T15:07:06.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.410+0000 7f7e479c4000 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55e9b46e88a0 con 0x55e9b49482a0 2024-03-31T15:07:06.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.410+0000 7f7e479c4000 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x55e9b4747c00 con 0x55e9b4947980 2024-03-31T15:07:06.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.410+0000 7f7e42600640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e9b48fb310 0x55e9b48fb6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:06.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.410+0000 7f7e41c00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55e9b4947980 0x55e9b4947d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:06.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.410+0000 7f7e42600640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e9b48fb310 0x55e9b48fb6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53052/0 (socket says 172.21.15.16:53052) 2024-03-31T15:07:06.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.410+0000 7f7e41200640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e9b49482a0 0x55e9b48fadd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:06.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.410+0000 7f7e42600640 1 -- 172.21.15.16:0/110343341 learned_addr learned my addr 172.21.15.16:0/110343341 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:07:06.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.410+0000 7f7e41200640 1 -- 172.21.15.16:0/110343341 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55e9b4947980 msgr2=0x55e9b4947d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:07:06.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.410+0000 7f7e41200640 1 --2- 172.21.15.16:0/110343341 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55e9b4947980 0x55e9b4947d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:06.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.410+0000 7f7e41200640 1 -- 172.21.15.16:0/110343341 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e9b48fb310 msgr2=0x55e9b48fb6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:07:06.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.410+0000 7f7e41200640 1 --2- 172.21.15.16:0/110343341 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e9b48fb310 0x55e9b48fb6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:06.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.410+0000 7f7e41200640 1 -- 172.21.15.16:0/110343341 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e9b4725ba0 con 0x55e9b49482a0 2024-03-31T15:07:06.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.410+0000 7f7e42600640 1 --2- 172.21.15.16:0/1188988204 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f7e1c068380 0x7f7e1c06a840 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f7e34003f70 tx=0x7f7e34004410 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:07:06.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.410+0000 7f7e41200640 1 --2- 172.21.15.16:0/110343341 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e9b49482a0 0x55e9b48fadd0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f7e2800ce90 tx=0x7f7e280564a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=56b441804da1102e server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:07:06.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.410+0000 7f7e2e000640 1 -- 172.21.15.16:0/110343341 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e28054720 con 0x55e9b49482a0 2024-03-31T15:07:06.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.410+0000 7f7e2e000640 1 -- 172.21.15.16:0/110343341 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7e28052a90 con 0x55e9b49482a0 2024-03-31T15:07:06.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.410+0000 7f7e2e000640 1 -- 172.21.15.16:0/110343341 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e280860d0 con 0x55e9b49482a0 2024-03-31T15:07:06.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.414+0000 7f7e479c4000 1 -- 172.21.15.16:0/110343341 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e9b49482a0 msgr2=0x55e9b48fadd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:06.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.414+0000 7f7e479c4000 1 --2- 172.21.15.16:0/110343341 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e9b49482a0 0x55e9b48fadd0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f7e2800ce90 tx=0x7f7e280564a0 comp rx=0 tx=0).stop 2024-03-31T15:07:06.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.414+0000 7f7e479c4000 1 -- 172.21.15.16:0/110343341 shutdown_connections 2024-03-31T15:07:06.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.414+0000 7f7e479c4000 1 --2- 172.21.15.16:0/110343341 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e9b48fb310 0x55e9b48fb6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:06.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.414+0000 7f7e479c4000 1 --2- 172.21.15.16:0/110343341 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e9b49482a0 0x55e9b48fadd0 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:06.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.414+0000 7f7e479c4000 1 --2- 172.21.15.16:0/110343341 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55e9b4947980 0x55e9b4947d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:06.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.414+0000 7f7e479c4000 1 -- 172.21.15.16:0/110343341 >> 172.21.15.16:0/110343341 conn(0x55e9b46f89f0 msgr2=0x55e9b492a730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:07:06.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.414+0000 7f7e479c4000 1 -- 172.21.15.16:0/110343341 shutdown_connections 2024-03-31T15:07:06.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.414+0000 7f7e479c4000 1 -- 172.21.15.16:0/110343341 wait complete. 2024-03-31T15:07:06.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.414+0000 7f7e479c4000 1 -- start start 2024-03-31T15:07:06.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.414+0000 7f7e479c4000 1 Processor -- start 2024-03-31T15:07:06.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.414+0000 7f7e479c4000 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e9b48fb310 0x55e9b4900c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:06.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.414+0000 7f7e479c4000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e9b4947980 0x55e9b4901190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:06.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.414+0000 7f7e479c4000 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55e9b49482a0 0x55e9b49016d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:06.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.414+0000 7f7e479c4000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55e9b47090a0 con 0x55e9b4947980 2024-03-31T15:07:06.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.414+0000 7f7e479c4000 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55e9b470c400 con 0x55e9b48fb310 2024-03-31T15:07:06.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.414+0000 7f7e479c4000 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x55e9b4758360 con 0x55e9b49482a0 2024-03-31T15:07:06.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.414+0000 7f7e479c4000 10 client.0 _ll_register_callbacks cb 0x55e9b48e8810 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T15:07:06.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.414+0000 7f7e41200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e9b4947980 0x55e9b4901190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:06.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.414+0000 7f7e41200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e9b4947980 0x55e9b4901190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53058/0 (socket says 172.21.15.16:53058) 2024-03-31T15:07:06.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.414+0000 7f7e41200640 1 -- 172.21.15.16:0/2040871426 learned_addr learned my addr 172.21.15.16:0/2040871426 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:07:06.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.414+0000 7f7e41c00640 1 --2- 172.21.15.16:0/2040871426 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e9b48fb310 0x55e9b4900c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:06.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.414+0000 7f7e42600640 1 --2- 172.21.15.16:0/2040871426 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55e9b49482a0 0x55e9b49016d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:06.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.414+0000 7f7e41200640 1 -- 172.21.15.16:0/2040871426 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55e9b49482a0 msgr2=0x55e9b49016d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:07:06.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.414+0000 7f7e41200640 1 --2- 172.21.15.16:0/2040871426 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55e9b49482a0 0x55e9b49016d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:06.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.414+0000 7f7e41200640 1 -- 172.21.15.16:0/2040871426 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e9b48fb310 msgr2=0x55e9b4900c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:07:06.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.414+0000 7f7e41200640 1 --2- 172.21.15.16:0/2040871426 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e9b48fb310 0x55e9b4900c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:06.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.414+0000 7f7e41200640 1 -- 172.21.15.16:0/2040871426 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e9b4791160 con 0x55e9b4947980 2024-03-31T15:07:06.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.414+0000 7f7e41200640 1 --2- 172.21.15.16:0/2040871426 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e9b4947980 0x55e9b4901190 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f7e28007fd0 tx=0x7f7e28000ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ea761ed837f383f server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:07:06.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.414+0000 7f7e22a00640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-31T15:07:06.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.414+0000 7f7e22a00640 1 -- 172.21.15.16:0/2040871426 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e280860d0 con 0x55e9b4947980 2024-03-31T15:07:06.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.414+0000 7f7e22a00640 1 -- 172.21.15.16:0/2040871426 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(56..56 src has 1..56) ==== 5709+0+0 (secure 0 0 0) 0x7f7e28086270 con 0x55e9b4947980 2024-03-31T15:07:06.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.414+0000 7f7e479c4000 1 -- 172.21.15.16:0/2040871426 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7e28052a90 con 0x55e9b4947980 2024-03-31T15:07:06.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.414+0000 7f7e22a00640 1 -- 172.21.15.16:0/2040871426 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f7e2806eb50 con 0x55e9b4947980 2024-03-31T15:07:06.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.414+0000 7f7e479c4000 10 client.6411 fetch_fsmap learned FSMap version 35 2024-03-31T15:07:06.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.414+0000 7f7e479c4000 1 -- 172.21.15.16:0/2040871426 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=2+}) -- 0x55e9b46e69c0 con 0x55e9b4947980 2024-03-31T15:07:06.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.414+0000 7f7e22a00640 1 -- 172.21.15.16:0/2040871426 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7e28052a90 con 0x55e9b4947980 2024-03-31T15:07:06.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.414+0000 7f7e22a00640 1 -- 172.21.15.16:0/2040871426 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 35) ==== 3494+0+0 (secure 0 0 0) 0x7f7e28055870 con 0x55e9b4947980 2024-03-31T15:07:06.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.414+0000 7f7e479c4000 10 client.6411 fetch_fsmap finished waiting for FSMap version 35 2024-03-31T15:07:06.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.414+0000 7f7e479c4000 10 client.6411 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-31T15:07:06.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.414+0000 7f7e479c4000 20 client.6411 populate_metadata read hostname 'smithi016' 2024-03-31T15:07:06.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.414+0000 7f7e479c4000 1 --2- 172.21.15.16:0/2040871426 >> [v2:172.21.15.16:6834/2613117528,v1:172.21.15.16:6835/2613117528] conn(0x55e9b4901f70 0x55e9b4958ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:06.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.414+0000 7f7e479c4000 4 client.6411 mds_command: new command op to 6150 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-31T15:07:06.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.414+0000 7f7e479c4000 1 -- 172.21.15.16:0/2040871426 --> [v2:172.21.15.16:6834/2613117528,v1:172.21.15.16:6835/2613117528] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e9b46e6e40 con 0x55e9b4901f70 2024-03-31T15:07:06.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.418+0000 7f7e41c00640 1 --2- 172.21.15.16:0/2040871426 >> [v2:172.21.15.16:6834/2613117528,v1:172.21.15.16:6835/2613117528] conn(0x55e9b4901f70 0x55e9b4958ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:06.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.418+0000 7f7e41c00640 1 --2- 172.21.15.16:0/2040871426 >> [v2:172.21.15.16:6834/2613117528,v1:172.21.15.16:6835/2613117528] conn(0x55e9b4901f70 0x55e9b4958ef0 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e4f16b5b37bc5647 server_cookie=65c45c3965f5841b in_seq=0 out_seq=0 2024-03-31T15:07:06.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.418+0000 7f7e22a00640 10 client.6411 ms_handle_connect on v2:172.21.15.16:6834/2613117528 2024-03-31T15:07:06.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.418+0000 7f7e22a00640 1 -- 172.21.15.16:0/2040871426 <== mds.0 v2:172.21.15.16:6834/2613117528 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55e9b46e6e40 con 0x55e9b4901f70 2024-03-31T15:07:06.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.418+0000 7f7e22a00640 10 client.6411 handle_command_reply: tid=0 2024-03-31T15:07:06.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.418+0000 7f7e479c4000 1 client.6411 shutdown 2024-03-31T15:07:06.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.418+0000 7f7e479c4000 1 -- 172.21.15.16:0/2040871426 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e9b4947980 msgr2=0x55e9b4901190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:06.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.418+0000 7f7e479c4000 1 --2- 172.21.15.16:0/2040871426 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e9b4947980 0x55e9b4901190 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f7e28007fd0 tx=0x7f7e28000ee0 comp rx=0 tx=0).stop 2024-03-31T15:07:06.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.418+0000 7f7e479c4000 1 -- 172.21.15.16:0/2040871426 shutdown_connections 2024-03-31T15:07:06.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.418+0000 7f7e479c4000 1 --2- 172.21.15.16:0/2040871426 >> [v2:172.21.15.16:6834/2613117528,v1:172.21.15.16:6835/2613117528] conn(0x55e9b4901f70 0x55e9b4958ef0 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:06.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.418+0000 7f7e479c4000 1 --2- 172.21.15.16:0/2040871426 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55e9b49482a0 0x55e9b49016d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:06.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.418+0000 7f7e479c4000 1 --2- 172.21.15.16:0/2040871426 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e9b4947980 0x55e9b4901190 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:06.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.418+0000 7f7e479c4000 1 --2- 172.21.15.16:0/2040871426 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e9b48fb310 0x55e9b4900c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:06.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.418+0000 7f7e22a00640 0 client.6411 ms_handle_reset on v2:172.21.15.16:6834/2613117528 2024-03-31T15:07:06.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.418+0000 7f7e479c4000 1 -- 172.21.15.16:0/2040871426 >> 172.21.15.16:0/2040871426 conn(0x55e9b46f89f0 msgr2=0x55e9b492a520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:07:06.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.418+0000 7f7e479c4000 1 -- 172.21.15.16:0/2040871426 shutdown_connections 2024-03-31T15:07:06.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.418+0000 7f7e479c4000 1 -- 172.21.15.16:0/2040871426 wait complete. 2024-03-31T15:07:06.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.418+0000 7f7e479c4000 20 client.6411 trim_cache size 0 max 16384 2024-03-31T15:07:06.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.430+0000 7f7e479c4000 1 Processor -- start 2024-03-31T15:07:06.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.430+0000 7f7e479c4000 1 -- start start 2024-03-31T15:07:06.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.434+0000 7f7e479c4000 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e9b48fb310 0x55e9b48f9c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:06.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.434+0000 7f7e479c4000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e9b4947980 0x55e9b48fa180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:06.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.434+0000 7f7e479c4000 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55e9b49482a0 0x55e9b4945a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:06.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.434+0000 7f7e479c4000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f7e3c15f0c0 con 0x55e9b4947980 2024-03-31T15:07:06.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.434+0000 7f7e479c4000 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f7e3c15b0e0 con 0x55e9b48fb310 2024-03-31T15:07:06.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.434+0000 7f7e479c4000 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f7e3c15ea70 con 0x55e9b49482a0 2024-03-31T15:07:06.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.434+0000 7f7e41200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e9b4947980 0x55e9b48fa180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:06.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.434+0000 7f7e41200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e9b4947980 0x55e9b48fa180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53066/0 (socket says 172.21.15.16:53066) 2024-03-31T15:07:06.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.434+0000 7f7e41200640 1 -- 172.21.15.16:0/2291981561 learned_addr learned my addr 172.21.15.16:0/2291981561 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:07:06.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.434+0000 7f7e41c00640 1 --2- 172.21.15.16:0/2291981561 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e9b48fb310 0x55e9b48f9c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:06.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.434+0000 7f7e42600640 1 --2- 172.21.15.16:0/2291981561 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55e9b49482a0 0x55e9b4945a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:06.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.434+0000 7f7e41200640 1 -- 172.21.15.16:0/2291981561 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55e9b49482a0 msgr2=0x55e9b4945a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:07:06.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.434+0000 7f7e41200640 1 --2- 172.21.15.16:0/2291981561 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55e9b49482a0 0x55e9b4945a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:06.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.434+0000 7f7e41200640 1 -- 172.21.15.16:0/2291981561 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e9b48fb310 msgr2=0x55e9b48f9c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:07:06.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.434+0000 7f7e41200640 1 --2- 172.21.15.16:0/2291981561 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e9b48fb310 0x55e9b48f9c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:06.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.434+0000 7f7e41200640 1 -- 172.21.15.16:0/2291981561 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e9b46e8af0 con 0x55e9b4947980 2024-03-31T15:07:06.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.434+0000 7f7e42600640 1 --2- 172.21.15.16:0/2291981561 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55e9b49482a0 0x55e9b4945a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:07:06.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.434+0000 7f7e41c00640 1 --2- 172.21.15.16:0/2291981561 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e9b48fb310 0x55e9b48f9c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:07:06.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.434+0000 7f7e41200640 1 --2- 172.21.15.16:0/2291981561 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e9b4947980 0x55e9b48fa180 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f7e28000fc0 tx=0x7f7e2801d070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83bb5b19f3c8f62f server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:07:06.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.434+0000 7f7e2d600640 1 -- 172.21.15.16:0/2291981561 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e280526c0 con 0x55e9b4947980 2024-03-31T15:07:06.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.434+0000 7f7e2d600640 1 -- 172.21.15.16:0/2291981561 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7e2805a720 con 0x55e9b4947980 2024-03-31T15:07:06.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.434+0000 7f7e2d600640 1 -- 172.21.15.16:0/2291981561 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e28055870 con 0x55e9b4947980 2024-03-31T15:07:06.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.434+0000 7f7e479c4000 1 -- 172.21.15.16:0/2291981561 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e9b4947980 msgr2=0x55e9b48fa180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:06.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.434+0000 7f7e479c4000 1 --2- 172.21.15.16:0/2291981561 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e9b4947980 0x55e9b48fa180 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f7e28000fc0 tx=0x7f7e2801d070 comp rx=0 tx=0).stop 2024-03-31T15:07:06.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.434+0000 7f7e479c4000 1 -- 172.21.15.16:0/2291981561 shutdown_connections 2024-03-31T15:07:06.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.434+0000 7f7e479c4000 1 --2- 172.21.15.16:0/2291981561 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55e9b49482a0 0x55e9b4945a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:06.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.434+0000 7f7e479c4000 1 --2- 172.21.15.16:0/2291981561 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e9b4947980 0x55e9b48fa180 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:06.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.434+0000 7f7e479c4000 1 --2- 172.21.15.16:0/2291981561 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e9b48fb310 0x55e9b48f9c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:06.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.434+0000 7f7e479c4000 1 -- 172.21.15.16:0/2291981561 >> 172.21.15.16:0/2291981561 conn(0x55e9b46f89f0 msgr2=0x55e9b48f9530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:07:06.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.434+0000 7f7e479c4000 1 -- 172.21.15.16:0/2291981561 shutdown_connections 2024-03-31T15:07:06.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.434+0000 7f7e479c4000 1 -- 172.21.15.16:0/2291981561 wait complete. 2024-03-31T15:07:06.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.434+0000 7f7e479c4000 1 -- start start 2024-03-31T15:07:06.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.434+0000 7f7e479c4000 1 Processor -- start 2024-03-31T15:07:06.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.434+0000 7f7e479c4000 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55e9b48fb310 0x55e9b49b78d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:06.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.434+0000 7f7e479c4000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e9b4947980 0x55e9b49b7e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:06.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.434+0000 7f7e479c4000 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e9b49482a0 0x55e9b495a450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:06.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.434+0000 7f7e479c4000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55e9b47090a0 con 0x55e9b4947980 2024-03-31T15:07:06.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.434+0000 7f7e479c4000 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55e9b4758360 con 0x55e9b49482a0 2024-03-31T15:07:06.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.434+0000 7f7e479c4000 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x55e9b470c400 con 0x55e9b48fb310 2024-03-31T15:07:06.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.438+0000 7f7e479c4000 10 client.0 _ll_register_callbacks cb 0x7f7e2806d070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T15:07:06.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.438+0000 7f7e41200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e9b4947980 0x55e9b49b7e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:06.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.438+0000 7f7e41200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e9b4947980 0x55e9b49b7e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53070/0 (socket says 172.21.15.16:53070) 2024-03-31T15:07:06.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.438+0000 7f7e41200640 1 -- 172.21.15.16:0/3421288492 learned_addr learned my addr 172.21.15.16:0/3421288492 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:07:06.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.438+0000 7f7e42600640 1 --2- 172.21.15.16:0/3421288492 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e9b49482a0 0x55e9b495a450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:06.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.438+0000 7f7e41c00640 1 --2- 172.21.15.16:0/3421288492 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55e9b48fb310 0x55e9b49b78d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:06.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.438+0000 7f7e41200640 1 -- 172.21.15.16:0/3421288492 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55e9b48fb310 msgr2=0x55e9b49b78d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:07:06.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.438+0000 7f7e41200640 1 --2- 172.21.15.16:0/3421288492 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55e9b48fb310 0x55e9b49b78d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:06.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.438+0000 7f7e41200640 1 -- 172.21.15.16:0/3421288492 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e9b49482a0 msgr2=0x55e9b495a450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:07:06.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.438+0000 7f7e41200640 1 --2- 172.21.15.16:0/3421288492 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e9b49482a0 0x55e9b495a450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:06.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.438+0000 7f7e41200640 1 -- 172.21.15.16:0/3421288492 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e9b47a4320 con 0x55e9b4947980 2024-03-31T15:07:06.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.438+0000 7f7e41200640 1 --2- 172.21.15.16:0/3421288492 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e9b4947980 0x55e9b49b7e10 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f7e28056340 tx=0x7f7e28001130 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af7f57e664cd1d56 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:07:06.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.438+0000 7f7e22a00640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-31T15:07:06.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.438+0000 7f7e22a00640 1 -- 172.21.15.16:0/3421288492 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e280860d0 con 0x55e9b4947980 2024-03-31T15:07:06.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.438+0000 7f7e22a00640 1 -- 172.21.15.16:0/3421288492 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(56..56 src has 1..56) ==== 5709+0+0 (secure 0 0 0) 0x7f7e28086270 con 0x55e9b4947980 2024-03-31T15:07:06.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.438+0000 7f7e479c4000 1 -- 172.21.15.16:0/3421288492 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7e2805a720 con 0x55e9b4947980 2024-03-31T15:07:06.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.438+0000 7f7e22a00640 1 -- 172.21.15.16:0/3421288492 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f7e28052170 con 0x55e9b4947980 2024-03-31T15:07:06.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.438+0000 7f7e479c4000 10 client.6417 fetch_fsmap learned FSMap version 35 2024-03-31T15:07:06.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.438+0000 7f7e479c4000 1 -- 172.21.15.16:0/3421288492 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=2+}) -- 0x55e9b46dfa80 con 0x55e9b4947980 2024-03-31T15:07:06.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.438+0000 7f7e22a00640 1 -- 172.21.15.16:0/3421288492 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7e2805a720 con 0x55e9b4947980 2024-03-31T15:07:06.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.438+0000 7f7e22a00640 1 -- 172.21.15.16:0/3421288492 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 35) ==== 3494+0+0 (secure 0 0 0) 0x7f7e28052310 con 0x55e9b4947980 2024-03-31T15:07:06.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.438+0000 7f7e479c4000 10 client.6417 fetch_fsmap finished waiting for FSMap version 35 2024-03-31T15:07:06.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.438+0000 7f7e479c4000 10 client.6417 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-31T15:07:06.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.438+0000 7f7e479c4000 20 client.6417 populate_metadata read hostname 'smithi016' 2024-03-31T15:07:06.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.438+0000 7f7e479c4000 1 --2- 172.21.15.16:0/3421288492 >> [v2:172.21.15.16:6834/2613117528,v1:172.21.15.16:6835/2613117528] conn(0x55e9b495ae10 0x55e9b495b1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:06.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.438+0000 7f7e479c4000 4 client.6417 mds_command: new command op to 6150 tid=0 multi_id=0 [{"prefix": "damage ls", "format": "json-pretty"}] 2024-03-31T15:07:06.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.438+0000 7f7e479c4000 1 -- 172.21.15.16:0/3421288492 --> [v2:172.21.15.16:6834/2613117528,v1:172.21.15.16:6835/2613117528] -- command(tid 0: {"prefix": "damage ls", "format": "json-pretty"}) -- 0x55e9b4759190 con 0x55e9b495ae10 2024-03-31T15:07:06.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.438+0000 7f7e41c00640 1 --2- 172.21.15.16:0/3421288492 >> [v2:172.21.15.16:6834/2613117528,v1:172.21.15.16:6835/2613117528] conn(0x55e9b495ae10 0x55e9b495b1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:06.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.438+0000 7f7e41c00640 1 --2- 172.21.15.16:0/3421288492 >> [v2:172.21.15.16:6834/2613117528,v1:172.21.15.16:6835/2613117528] conn(0x55e9b495ae10 0x55e9b495b1f0 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dd9a9cd7acf7523d server_cookie=fcd7f5bd1930d6c5 in_seq=0 out_seq=0 2024-03-31T15:07:06.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.438+0000 7f7e22a00640 10 client.6417 ms_handle_connect on v2:172.21.15.16:6834/2613117528 2024-03-31T15:07:06.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.438+0000 7f7e22a00640 1 -- 172.21.15.16:0/3421288492 <== mds.0 v2:172.21.15.16:6834/2613117528 1 ==== command_reply(tid 0: 0 ) ==== 8+0+141 (crc 0 0 0) 0x55e9b4759190 con 0x55e9b495ae10 2024-03-31T15:07:06.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.438+0000 7f7e22a00640 10 client.6417 handle_command_reply: tid=0 2024-03-31T15:07:06.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.438+0000 7f7e479c4000 1 client.6417 shutdown 2024-03-31T15:07:06.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.438+0000 7f7e479c4000 1 -- 172.21.15.16:0/3421288492 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e9b4947980 msgr2=0x55e9b49b7e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:06.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.438+0000 7f7e479c4000 1 --2- 172.21.15.16:0/3421288492 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e9b4947980 0x55e9b49b7e10 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f7e28056340 tx=0x7f7e28001130 comp rx=0 tx=0).stop 2024-03-31T15:07:06.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.438+0000 7f7e479c4000 1 -- 172.21.15.16:0/3421288492 shutdown_connections 2024-03-31T15:07:06.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.438+0000 7f7e479c4000 1 --2- 172.21.15.16:0/3421288492 >> [v2:172.21.15.16:6834/2613117528,v1:172.21.15.16:6835/2613117528] conn(0x55e9b495ae10 0x55e9b495b1f0 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:06.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.438+0000 7f7e479c4000 1 --2- 172.21.15.16:0/3421288492 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e9b49482a0 0x55e9b495a450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:06.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.438+0000 7f7e479c4000 1 --2- 172.21.15.16:0/3421288492 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e9b4947980 0x55e9b49b7e10 unknown :-1 s=CLOSED pgs=701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:06.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.438+0000 7f7e22a00640 0 client.6417 ms_handle_reset on v2:172.21.15.16:6834/2613117528 2024-03-31T15:07:06.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.438+0000 7f7e479c4000 1 --2- 172.21.15.16:0/3421288492 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55e9b48fb310 0x55e9b49b78d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:06.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.438+0000 7f7e479c4000 1 -- 172.21.15.16:0/3421288492 >> 172.21.15.16:0/3421288492 conn(0x55e9b46f89f0 msgr2=0x55e9b4902be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:07:06.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.438+0000 7f7e479c4000 1 -- 172.21.15.16:0/3421288492 shutdown_connections 2024-03-31T15:07:06.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.438+0000 7f7e479c4000 1 -- 172.21.15.16:0/3421288492 wait complete. 2024-03-31T15:07:06.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.438+0000 7f7e479c4000 20 client.6417 trim_cache size 0 max 16384 2024-03-31T15:07:06.439 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:07:06.439 INFO:teuthology.orchestra.run.smithi016.stdout:[ 2024-03-31T15:07:06.440 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-31T15:07:06.440 INFO:teuthology.orchestra.run.smithi016.stdout: "damage_type": "backtrace", 2024-03-31T15:07:06.440 INFO:teuthology.orchestra.run.smithi016.stdout: "id": 3106237266, 2024-03-31T15:07:06.440 INFO:teuthology.orchestra.run.smithi016.stdout: "ino": 1099511627777, 2024-03-31T15:07:06.440 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/dir_test/file" 2024-03-31T15:07:06.440 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-31T15:07:06.440 INFO:teuthology.orchestra.run.smithi016.stdout:] 2024-03-31T15:07:06.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.438+0000 7f7e44600640 1 -- 172.21.15.16:0/1188988204 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f7e1c068380 msgr2=0x7f7e1c06a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:06.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.438+0000 7f7e44600640 1 --2- 172.21.15.16:0/1188988204 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f7e1c068380 0x7f7e1c06a840 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f7e34003f70 tx=0x7f7e34004410 comp rx=0 tx=0).stop 2024-03-31T15:07:06.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.438+0000 7f7e44600640 1 -- 172.21.15.16:0/1188988204 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7e3c0731c0 msgr2=0x7f7e3c147830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:06.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.438+0000 7f7e44600640 1 --2- 172.21.15.16:0/1188988204 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7e3c0731c0 0x7f7e3c147830 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f7e2800c4d0 tx=0x7f7e2800c990 comp rx=0 tx=0).stop 2024-03-31T15:07:06.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.438+0000 7f7e44600640 1 -- 172.21.15.16:0/1188988204 shutdown_connections 2024-03-31T15:07:06.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.438+0000 7f7e44600640 1 --2- 172.21.15.16:0/1188988204 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f7e1c068380 0x7f7e1c06a840 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:06.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.438+0000 7f7e44600640 1 --2- 172.21.15.16:0/1188988204 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7e3c1567e0 0x7f7e3c1534f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:06.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.438+0000 7f7e44600640 1 --2- 172.21.15.16:0/1188988204 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7e3c0731c0 0x7f7e3c147830 unknown :-1 s=CLOSED pgs=698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:06.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.438+0000 7f7e44600640 1 --2- 172.21.15.16:0/1188988204 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f7e3c06dbe0 0x7f7e3c15f2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:06.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.438+0000 7f7e44600640 1 -- 172.21.15.16:0/1188988204 >> 172.21.15.16:0/1188988204 conn(0x7f7e3c06ee40 msgr2=0x7f7e3c0afd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:07:06.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.442+0000 7f7e44600640 1 -- 172.21.15.16:0/1188988204 shutdown_connections 2024-03-31T15:07:06.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.442+0000 7f7e44600640 1 -- 172.21.15.16:0/1188988204 wait complete. 2024-03-31T15:07:06.455 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-03-31T15:07:06.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.598+0000 7f80d9600640 1 Processor -- start 2024-03-31T15:07:06.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.598+0000 7f80d9600640 1 -- start start 2024-03-31T15:07:06.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.602+0000 7f80d9600640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f80d413dff0 0x7f80d406d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:06.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.602+0000 7f80d9600640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f80d413e3d0 0x7f80d406da10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:06.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.602+0000 7f80d9600640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f80d415fbc0 con 0x7f80d40cc9d0 2024-03-31T15:07:06.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.602+0000 7f80d9600640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f80d415f730 con 0x7f80d413e3d0 2024-03-31T15:07:06.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.602+0000 7f80d9600640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f80d415ee00 con 0x7f80d413dff0 2024-03-31T15:07:06.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.602+0000 7f80cbe00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f80d413e3d0 0x7f80d406da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:06.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.602+0000 7f80cbe00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f80d413e3d0 0x7f80d406da10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53082/0 (socket says 172.21.15.16:53082) 2024-03-31T15:07:06.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.602+0000 7f80cbe00640 1 -- 172.21.15.16:0/1346366329 learned_addr learned my addr 172.21.15.16:0/1346366329 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:07:06.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.602+0000 7f80d2a00640 1 --2- 172.21.15.16:0/1346366329 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f80d413dff0 0x7f80d406d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:06.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.602+0000 7f80cbe00640 1 -- 172.21.15.16:0/1346366329 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f80d413dff0 msgr2=0x7f80d406d4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:07:06.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.602+0000 7f80cbe00640 1 --2- 172.21.15.16:0/1346366329 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f80d413dff0 0x7f80d406d4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:06.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.602+0000 7f80cbe00640 1 -- 172.21.15.16:0/1346366329 >> v1:172.21.15.89:6789/0 conn(0x7f80d40cc9d0 legacy=0x7f80d40cce30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:07:06.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.602+0000 7f80cbe00640 1 -- 172.21.15.16:0/1346366329 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80d415f0e0 con 0x7f80d413e3d0 2024-03-31T15:07:06.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.606+0000 7f80cbe00640 1 --2- 172.21.15.16:0/1346366329 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f80d413e3d0 0x7f80d406da10 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f80bc009870 tx=0x7f80bc01c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4814c14974e9cc48 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:07:06.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.606+0000 7f80d2000640 1 -- 172.21.15.16:0/1346366329 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80bc0b0070 con 0x7f80d413e3d0 2024-03-31T15:07:06.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.606+0000 7f80d2000640 1 -- 172.21.15.16:0/1346366329 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f80bc0ab600 con 0x7f80d413e3d0 2024-03-31T15:07:06.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.606+0000 7f80d2000640 1 -- 172.21.15.16:0/1346366329 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80bc0ab9d0 con 0x7f80d413e3d0 2024-03-31T15:07:06.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.606+0000 7f80d9600640 1 -- 172.21.15.16:0/1346366329 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f80d413e3d0 msgr2=0x7f80d406da10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:06.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.606+0000 7f80d9600640 1 --2- 172.21.15.16:0/1346366329 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f80d413e3d0 0x7f80d406da10 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f80bc009870 tx=0x7f80bc01c490 comp rx=0 tx=0).stop 2024-03-31T15:07:06.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.606+0000 7f80d9600640 1 -- 172.21.15.16:0/1346366329 shutdown_connections 2024-03-31T15:07:06.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.606+0000 7f80d9600640 1 --2- 172.21.15.16:0/1346366329 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f80d413e3d0 0x7f80d406da10 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:06.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.606+0000 7f80d9600640 1 --2- 172.21.15.16:0/1346366329 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f80d413dff0 0x7f80d406d4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:06.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.606+0000 7f80d9600640 1 -- 172.21.15.16:0/1346366329 >> 172.21.15.16:0/1346366329 conn(0x7f80d406ee40 msgr2=0x7f80d406b620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:07:06.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.606+0000 7f80d9600640 1 -- 172.21.15.16:0/1346366329 shutdown_connections 2024-03-31T15:07:06.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.606+0000 7f80d9600640 1 -- 172.21.15.16:0/1346366329 wait complete. 2024-03-31T15:07:06.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.610+0000 7f80d9600640 1 Processor -- start 2024-03-31T15:07:06.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.610+0000 7f80d9600640 1 -- start start 2024-03-31T15:07:06.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.610+0000 7f80d9600640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f80d40cc9d0 0x7f80d413bf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:06.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.610+0000 7f80d9600640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f80d413dff0 0x7f80d4143c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:06.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.610+0000 7f80d9600640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f80d413e3d0 0x7f80d41536f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:06.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.610+0000 7f80d9600640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f80d40720f0 con 0x7f80d413e3d0 2024-03-31T15:07:06.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.610+0000 7f80d9600640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f80d4071f70 con 0x7f80d413dff0 2024-03-31T15:07:06.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.610+0000 7f80d9600640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f80d4072270 con 0x7f80d40cc9d0 2024-03-31T15:07:06.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.610+0000 7f80d3400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f80d413e3d0 0x7f80d41536f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:06.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.610+0000 7f80d3400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f80d413e3d0 0x7f80d41536f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53090/0 (socket says 172.21.15.16:53090) 2024-03-31T15:07:06.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.610+0000 7f80d3400640 1 -- 172.21.15.16:0/607645907 learned_addr learned my addr 172.21.15.16:0/607645907 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:07:06.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.610+0000 7f80cbe00640 1 --2- 172.21.15.16:0/607645907 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f80d413dff0 0x7f80d4143c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:06.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.610+0000 7f80d2a00640 1 --2- 172.21.15.16:0/607645907 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f80d40cc9d0 0x7f80d413bf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:06.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.610+0000 7f80d3400640 1 -- 172.21.15.16:0/607645907 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f80d40cc9d0 msgr2=0x7f80d413bf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:06.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.610+0000 7f80d3400640 1 --2- 172.21.15.16:0/607645907 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f80d40cc9d0 0x7f80d413bf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:06.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.610+0000 7f80d3400640 1 -- 172.21.15.16:0/607645907 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f80d413dff0 msgr2=0x7f80d4143c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:06.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.610+0000 7f80d3400640 1 --2- 172.21.15.16:0/607645907 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f80d413dff0 0x7f80d4143c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:06.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.610+0000 7f80d3400640 1 -- 172.21.15.16:0/607645907 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80d41fb890 con 0x7f80d413e3d0 2024-03-31T15:07:06.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.610+0000 7f80d3400640 1 --2- 172.21.15.16:0/607645907 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f80d413e3d0 0x7f80d41536f0 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f80cc00c780 tx=0x7f80cc00cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:07:06.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.610+0000 7f80caa00640 1 -- 172.21.15.16:0/607645907 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80cc004b70 con 0x7f80d413e3d0 2024-03-31T15:07:06.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.610+0000 7f80caa00640 1 -- 172.21.15.16:0/607645907 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f80cc004d10 con 0x7f80d413e3d0 2024-03-31T15:07:06.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.610+0000 7f80caa00640 1 -- 172.21.15.16:0/607645907 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80cc00aa90 con 0x7f80d413e3d0 2024-03-31T15:07:06.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.610+0000 7f80d9600640 1 -- 172.21.15.16:0/607645907 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f80d41fab10 con 0x7f80d413e3d0 2024-03-31T15:07:06.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.610+0000 7f80d9600640 1 -- 172.21.15.16:0/607645907 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f80d41fadb0 con 0x7f80d413e3d0 2024-03-31T15:07:06.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.614+0000 7f80d9600640 1 -- 172.21.15.16:0/607645907 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f80d4152420 con 0x7f80d413e3d0 2024-03-31T15:07:06.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.614+0000 7f80caa00640 1 -- 172.21.15.16:0/607645907 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f80cc00ac30 con 0x7f80d413e3d0 2024-03-31T15:07:06.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.614+0000 7f80caa00640 1 --2- 172.21.15.16:0/607645907 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f80b0068380 0x7f80b006a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:06.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.614+0000 7f80caa00640 1 -- 172.21.15.16:0/607645907 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 5709+0+0 (secure 0 0 0) 0x7f80cc086b00 con 0x7f80d413e3d0 2024-03-31T15:07:06.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.614+0000 7f80d2a00640 1 --2- 172.21.15.16:0/607645907 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f80b0068380 0x7f80b006a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:06.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.614+0000 7f80d2a00640 1 --2- 172.21.15.16:0/607645907 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f80b0068380 0x7f80b006a840 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f80c4007a60 tx=0x7f80c4008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:07:06.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.618+0000 7f80caa00640 1 -- 172.21.15.16:0/607645907 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f80cc020070 con 0x7f80d413e3d0 2024-03-31T15:07:06.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.818+0000 7f80d9600640 1 -- 172.21.15.16:0/607645907 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f80d4069950 con 0x7f80d413e3d0 2024-03-31T15:07:06.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.818+0000 7f80caa00640 1 -- 172.21.15.16:0/607645907 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+164 (secure 0 0 0) 0x7f80cc056a60 con 0x7f80d413e3d0 2024-03-31T15:07:06.814 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:07:06.814 INFO:teuthology.orchestra.run.smithi016.stdout:{"status":"HEALTH_ERR","checks":{"MDS_DAMAGE":{"severity":"HEALTH_ERR","summary":{"message":"1 MDSs report damaged metadata","count":1},"muted":false}},"mutes":[]} 2024-03-31T15:07:06.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.822+0000 7f80d9600640 1 -- 172.21.15.16:0/607645907 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f80b0068380 msgr2=0x7f80b006a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:06.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.822+0000 7f80d9600640 1 --2- 172.21.15.16:0/607645907 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f80b0068380 0x7f80b006a840 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f80c4007a60 tx=0x7f80c4008040 comp rx=0 tx=0).stop 2024-03-31T15:07:06.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.822+0000 7f80d9600640 1 -- 172.21.15.16:0/607645907 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f80d413e3d0 msgr2=0x7f80d41536f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:06.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.822+0000 7f80d9600640 1 --2- 172.21.15.16:0/607645907 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f80d413e3d0 0x7f80d41536f0 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f80cc00c780 tx=0x7f80cc00cc40 comp rx=0 tx=0).stop 2024-03-31T15:07:06.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.822+0000 7f80d9600640 1 -- 172.21.15.16:0/607645907 shutdown_connections 2024-03-31T15:07:06.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.822+0000 7f80d9600640 1 --2- 172.21.15.16:0/607645907 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f80b0068380 0x7f80b006a840 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:06.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.822+0000 7f80d9600640 1 --2- 172.21.15.16:0/607645907 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f80d413e3d0 0x7f80d41536f0 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:06.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.822+0000 7f80d9600640 1 --2- 172.21.15.16:0/607645907 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f80d413dff0 0x7f80d4143c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:06.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.822+0000 7f80d9600640 1 --2- 172.21.15.16:0/607645907 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f80d40cc9d0 0x7f80d413bf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:06.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.822+0000 7f80d9600640 1 -- 172.21.15.16:0/607645907 >> 172.21.15.16:0/607645907 conn(0x7f80d406ee40 msgr2=0x7f80d413d170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:07:06.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.822+0000 7f80d9600640 1 -- 172.21.15.16:0/607645907 shutdown_connections 2024-03-31T15:07:06.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.822+0000 7f80d9600640 1 -- 172.21.15.16:0/607645907 wait complete. 2024-03-31T15:07:06.835 DEBUG:tasks.ceph_test_case:wait_until_true: success in 0s and 0 retries 2024-03-31T15:07:06.835 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:07:06.835 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.4:0 scrub start / repair,recursive,force 2024-03-31T15:07:06.963 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-31T15:07:06.966 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-31T15:07:06.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.986+0000 7f26ade00640 1 Processor -- start 2024-03-31T15:07:06.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.986+0000 7f26ade00640 1 -- start start 2024-03-31T15:07:06.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.986+0000 7f26ade00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f26a80b44d0 0x7f26a80d1a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:06.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.986+0000 7f26ade00640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f26a80ce2d0 con 0x7f26a80b7900 2024-03-31T15:07:06.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.986+0000 7f26ade00640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f26a80ceb10 con 0x7f26a80cc7b0 2024-03-31T15:07:06.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.986+0000 7f26ade00640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f26a80ce550 con 0x7f26a80b44d0 2024-03-31T15:07:06.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.986+0000 7f26a6a00640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f26a80b7900 0x7f26a8160740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:33734/0 (socket says 172.21.15.16:33734) 2024-03-31T15:07:06.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.986+0000 7f26a6a00640 1 -- 172.21.15.16:0/1802515534 learned_addr learned my addr 172.21.15.16:0/1802515534 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:07:06.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.986+0000 7f26a7400640 1 --2- 172.21.15.16:0/1802515534 >> v2:172.21.15.89:3300/0 conn(0x7f26a80b44d0 0x7f26a80d1a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:06.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.986+0000 7f26a5600640 1 -- 172.21.15.16:0/1802515534 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3139450846 0 0) 0x7f2698001120 con 0x7f26a80b7900 2024-03-31T15:07:06.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.986+0000 7f26a5600640 1 -- 172.21.15.16:0/1802515534 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f268c003530 con 0x7f26a80b7900 2024-03-31T15:07:06.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.986+0000 7f26a5600640 1 -- 172.21.15.16:0/1802515534 <== mon.1 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2266262137 0 0) 0x7f26a80ceb10 con 0x7f26a80cc7b0 2024-03-31T15:07:06.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.986+0000 7f26a5600640 1 -- 172.21.15.16:0/1802515534 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2698001120 con 0x7f26a80cc7b0 2024-03-31T15:07:06.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.986+0000 7f26a5600640 1 -- 172.21.15.16:0/1802515534 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1212626716 0 0) 0x7f268c003530 con 0x7f26a80b7900 2024-03-31T15:07:06.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.986+0000 7f26a5600640 1 -- 172.21.15.16:0/1802515534 >> v2:172.21.15.89:3300/0 conn(0x7f26a80b44d0 msgr2=0x7f26a80d1a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:07:06.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.986+0000 7f26a5600640 1 --2- 172.21.15.16:0/1802515534 >> v2:172.21.15.89:3300/0 conn(0x7f26a80b44d0 0x7f26a80d1a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:06.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.986+0000 7f26a5600640 1 -- 172.21.15.16:0/1802515534 >> v1:172.21.15.89:6789/0 conn(0x7f26a80cc7b0 legacy=0x7f26a80b3dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:06.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.986+0000 7f26a5600640 1 -- 172.21.15.16:0/1802515534 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26a80ce830 con 0x7f26a80b7900 2024-03-31T15:07:06.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.986+0000 7f26a5600640 1 -- 172.21.15.16:0/1802515534 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 214807623 0 0) 0x7f2698001850 con 0x7f26a80b7900 2024-03-31T15:07:06.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.986+0000 7f26a7400640 1 --2- 172.21.15.16:0/1802515534 >> v2:172.21.15.89:3300/0 conn(0x7f26a80b44d0 0x7f26a80d1a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:07:06.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.986+0000 7f26a5600640 1 -- 172.21.15.16:0/1802515534 >> v1:172.21.15.16:6789/0 conn(0x7f26a80b7900 legacy=0x7f26a8160740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:06.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.986+0000 7f26a5600640 1 --2- 172.21.15.16:0/1802515534 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f268c003ba0 0x7f268c003fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:06.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.986+0000 7f26a5600640 1 --2- 172.21.15.16:0/1802515534 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f268c004810 0x7f268c004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:06.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.986+0000 7f26a5600640 1 --2- 172.21.15.16:0/1802515534 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f268c005220 0x7f268c00b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:06.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.986+0000 7f26a5600640 1 -- 172.21.15.16:0/1802515534 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f26a80ce550 con 0x7f268c003ba0 2024-03-31T15:07:06.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.986+0000 7f26a5600640 1 -- 172.21.15.16:0/1802515534 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f268c00c150 con 0x7f268c004810 2024-03-31T15:07:06.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.986+0000 7f26a5600640 1 -- 172.21.15.16:0/1802515534 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f268c00c350 con 0x7f268c005220 2024-03-31T15:07:06.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.986+0000 7f26a6a00640 1 --2- 172.21.15.16:0/1802515534 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f268c003ba0 0x7f268c003fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:06.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.986+0000 7f26a6000640 1 --2- 172.21.15.16:0/1802515534 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f268c004810 0x7f268c004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:06.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.986+0000 7f26a7400640 1 --2- 172.21.15.16:0/1802515534 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f268c005220 0x7f268c00b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:06.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.990+0000 7f26a6a00640 1 -- 172.21.15.16:0/1802515534 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f268c005220 msgr2=0x7f268c00b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:07:06.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.990+0000 7f26a6a00640 1 --2- 172.21.15.16:0/1802515534 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f268c005220 0x7f268c00b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:06.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.990+0000 7f26a6a00640 1 -- 172.21.15.16:0/1802515534 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f268c004810 msgr2=0x7f268c004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:07:06.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.990+0000 7f26a6a00640 1 --2- 172.21.15.16:0/1802515534 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f268c004810 0x7f268c004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:06.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.990+0000 7f26a6a00640 1 -- 172.21.15.16:0/1802515534 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f268c003530 con 0x7f268c003ba0 2024-03-31T15:07:06.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.990+0000 7f26a7400640 1 -- 172.21.15.16:0/1802515534 reap_dead start 2024-03-31T15:07:06.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.994+0000 7f26a6a00640 1 --2- 172.21.15.16:0/1802515534 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f268c003ba0 0x7f268c003fa0 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f269800c320 tx=0x7f269801cb10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=714dfed1a9f29986 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:07:06.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.994+0000 7f26a5600640 1 -- 172.21.15.16:0/1802515534 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2698007dc0 con 0x7f268c003ba0 2024-03-31T15:07:06.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.994+0000 7f26a5600640 1 -- 172.21.15.16:0/1802515534 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f26980a9070 con 0x7f268c003ba0 2024-03-31T15:07:06.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.994+0000 7f26a5600640 1 -- 172.21.15.16:0/1802515534 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2698005d60 con 0x7f268c003ba0 2024-03-31T15:07:06.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.994+0000 7f26ade00640 1 -- 172.21.15.16:0/1802515534 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f268c003ba0 msgr2=0x7f268c003fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:06.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.994+0000 7f26ade00640 1 --2- 172.21.15.16:0/1802515534 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f268c003ba0 0x7f268c003fa0 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f269800c320 tx=0x7f269801cb10 comp rx=0 tx=0).stop 2024-03-31T15:07:06.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.994+0000 7f26ade00640 1 -- 172.21.15.16:0/1802515534 shutdown_connections 2024-03-31T15:07:06.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.994+0000 7f26ade00640 1 --2- 172.21.15.16:0/1802515534 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f268c003ba0 0x7f268c003fa0 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:06.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.994+0000 7f26ade00640 1 -- 172.21.15.16:0/1802515534 >> 172.21.15.16:0/1802515534 conn(0x7f26a806b7f0 msgr2=0x7f26a8067330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:07:06.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.994+0000 7f26ade00640 1 -- 172.21.15.16:0/1802515534 shutdown_connections 2024-03-31T15:07:06.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.994+0000 7f26ade00640 1 -- 172.21.15.16:0/1802515534 wait complete. 2024-03-31T15:07:06.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.994+0000 7f26ade00640 1 Processor -- start 2024-03-31T15:07:06.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.994+0000 7f26ade00640 1 -- start start 2024-03-31T15:07:06.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.998+0000 7f26ade00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f268c003ba0 0x7f26a8155270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:06.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.998+0000 7f26ade00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f26a80b6e70 0x7f26a8152430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:06.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.998+0000 7f26ade00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f26a8140a90 0x7f26a8140f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:06.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.998+0000 7f26ade00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f26a8160cd0 con 0x7f268c003ba0 2024-03-31T15:07:06.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.998+0000 7f26ade00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f26a8160b50 con 0x7f26a8140a90 2024-03-31T15:07:06.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.998+0000 7f26ade00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f26a80b41d0 con 0x7f26a80b6e70 2024-03-31T15:07:06.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.998+0000 7f26a6a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f268c003ba0 0x7f26a8155270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:06.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.998+0000 7f26a6a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f268c003ba0 0x7f26a8155270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53114/0 (socket says 172.21.15.16:53114) 2024-03-31T15:07:06.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.998+0000 7f26a6a00640 1 -- 172.21.15.16:0/3307951858 learned_addr learned my addr 172.21.15.16:0/3307951858 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:07:06.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.998+0000 7f26a7400640 1 --2- 172.21.15.16:0/3307951858 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f26a8140a90 0x7f26a8140f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:06.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.998+0000 7f26a6000640 1 --2- 172.21.15.16:0/3307951858 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f26a80b6e70 0x7f26a8152430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:06.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.998+0000 7f26a6a00640 1 -- 172.21.15.16:0/3307951858 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f26a80b6e70 msgr2=0x7f26a8152430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:06.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.998+0000 7f26a6a00640 1 --2- 172.21.15.16:0/3307951858 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f26a80b6e70 0x7f26a8152430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:06.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.998+0000 7f26a6a00640 1 -- 172.21.15.16:0/3307951858 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f26a8140a90 msgr2=0x7f26a8140f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:06.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.998+0000 7f26a6a00640 1 --2- 172.21.15.16:0/3307951858 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f26a8140a90 0x7f26a8140f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:06.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.998+0000 7f26a6a00640 1 -- 172.21.15.16:0/3307951858 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26a8154210 con 0x7f268c003ba0 2024-03-31T15:07:06.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.998+0000 7f26a6a00640 1 --2- 172.21.15.16:0/3307951858 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f268c003ba0 0x7f26a8155270 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f2698000c10 tx=0x7f268c005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:07:06.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.998+0000 7f269f400640 1 -- 172.21.15.16:0/3307951858 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26980bc070 con 0x7f268c003ba0 2024-03-31T15:07:06.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.998+0000 7f269f400640 1 -- 172.21.15.16:0/3307951858 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2698001650 con 0x7f268c003ba0 2024-03-31T15:07:06.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.998+0000 7f269f400640 1 -- 172.21.15.16:0/3307951858 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2698007dc0 con 0x7f268c003ba0 2024-03-31T15:07:06.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.998+0000 7f26ade00640 1 -- 172.21.15.16:0/3307951858 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f26a81544b0 con 0x7f268c003ba0 2024-03-31T15:07:06.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:06.998+0000 7f26ade00640 1 -- 172.21.15.16:0/3307951858 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f26a8154a70 con 0x7f268c003ba0 2024-03-31T15:07:06.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.002+0000 7f269f400640 1 -- 172.21.15.16:0/3307951858 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f26980bf070 con 0x7f268c003ba0 2024-03-31T15:07:06.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.002+0000 7f269f400640 1 --2- 172.21.15.16:0/3307951858 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f26880683b0 0x7f268806a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:06.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.002+0000 7f269f400640 1 -- 172.21.15.16:0/3307951858 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 5709+0+0 (secure 0 0 0) 0x7f269811fd20 con 0x7f268c003ba0 2024-03-31T15:07:06.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.002+0000 7f26b1358000 1 Processor -- start 2024-03-31T15:07:06.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.002+0000 7f26a6000640 1 --2- 172.21.15.16:0/3307951858 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f26880683b0 0x7f268806a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:06.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.002+0000 7f26b1358000 1 -- start start 2024-03-31T15:07:06.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.002+0000 7f26a6000640 1 --2- 172.21.15.16:0/3307951858 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f26880683b0 0x7f268806a870 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f26a8154000 tx=0x7f2694009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:07:06.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.002+0000 7f26b1358000 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5560b9a46f10 0x5560b9a472f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:06.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.002+0000 7f26b1358000 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5560b9a47830 0x5560b99fd800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:06.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.002+0000 7f26b1358000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5560b99fdd40 0x5560b99fe120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:06.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.002+0000 7f26b1358000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5560b9848960 con 0x5560b99fdd40 2024-03-31T15:07:06.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.002+0000 7f26b1358000 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x5560b97e59f0 con 0x5560b9a46f10 2024-03-31T15:07:06.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.002+0000 7f26b1358000 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x5560b9844910 con 0x5560b9a47830 2024-03-31T15:07:06.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.002+0000 7f26a7400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5560b99fdd40 0x5560b99fe120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:06.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.002+0000 7f26a6a00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5560b9a46f10 0x5560b9a472f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:07.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.002+0000 7f26a6a00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5560b9a46f10 0x5560b9a472f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.16:58716/0 (socket says 172.21.15.16:58716) 2024-03-31T15:07:07.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.002+0000 7f26a6a00640 1 -- 172.21.15.16:0/2378246508 learned_addr learned my addr 172.21.15.16:0/2378246508 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:07:07.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.002+0000 7f26a6000640 1 --2- 172.21.15.16:0/2378246508 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5560b9a47830 0x5560b99fd800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:07.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.002+0000 7f26a6a00640 1 -- 172.21.15.16:0/2378246508 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5560b9a47830 msgr2=0x5560b99fd800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:07:07.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.002+0000 7f26a6a00640 1 --2- 172.21.15.16:0/2378246508 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5560b9a47830 0x5560b99fd800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:07.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.002+0000 7f26a6a00640 1 -- 172.21.15.16:0/2378246508 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5560b99fdd40 msgr2=0x5560b99fe120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:07:07.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.002+0000 7f26a6a00640 1 --2- 172.21.15.16:0/2378246508 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5560b99fdd40 0x5560b99fe120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:07.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.002+0000 7f26a6a00640 1 -- 172.21.15.16:0/2378246508 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5560b988d6e0 con 0x5560b9a46f10 2024-03-31T15:07:07.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.002+0000 7f26a6a00640 1 --2- 172.21.15.16:0/2378246508 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5560b9a46f10 0x5560b9a472f0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f26980ec050 tx=0x7f2698000f80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a16648957eb82e76 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:07:07.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.002+0000 7f2693e00640 1 -- 172.21.15.16:0/2378246508 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26980f3ef0 con 0x5560b9a46f10 2024-03-31T15:07:07.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.002+0000 7f2693e00640 1 -- 172.21.15.16:0/2378246508 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f26980a9030 con 0x5560b9a46f10 2024-03-31T15:07:07.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.002+0000 7f2693e00640 1 -- 172.21.15.16:0/2378246508 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26980f8720 con 0x5560b9a46f10 2024-03-31T15:07:07.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.006+0000 7f26b1358000 1 -- 172.21.15.16:0/2378246508 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5560b9a46f10 msgr2=0x5560b9a472f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:07.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.006+0000 7f26b1358000 1 --2- 172.21.15.16:0/2378246508 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5560b9a46f10 0x5560b9a472f0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f26980ec050 tx=0x7f2698000f80 comp rx=0 tx=0).stop 2024-03-31T15:07:07.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.006+0000 7f26b1358000 1 -- 172.21.15.16:0/2378246508 shutdown_connections 2024-03-31T15:07:07.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.006+0000 7f26b1358000 1 --2- 172.21.15.16:0/2378246508 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5560b99fdd40 0x5560b99fe120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:07.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.006+0000 7f26b1358000 1 --2- 172.21.15.16:0/2378246508 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5560b9a47830 0x5560b99fd800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:07.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.006+0000 7f26b1358000 1 --2- 172.21.15.16:0/2378246508 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5560b9a46f10 0x5560b9a472f0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:07.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.006+0000 7f26b1358000 1 -- 172.21.15.16:0/2378246508 >> 172.21.15.16:0/2378246508 conn(0x5560b98e0db0 msgr2=0x5560b99f7cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:07:07.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.006+0000 7f26b1358000 1 -- 172.21.15.16:0/2378246508 shutdown_connections 2024-03-31T15:07:07.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.006+0000 7f26b1358000 1 -- 172.21.15.16:0/2378246508 wait complete. 2024-03-31T15:07:07.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.006+0000 7f26b1358000 1 -- start start 2024-03-31T15:07:07.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.006+0000 7f26b1358000 1 Processor -- start 2024-03-31T15:07:07.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.006+0000 7f26b1358000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5560b99fdd40 0x5560b9a4e060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:07.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.006+0000 7f26b1358000 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5560b9a46f10 0x5560b9a4e5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:07.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.006+0000 7f26b1358000 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5560b9a47830 0x5560b9a34860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:07.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.006+0000 7f26b1358000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5560b97cc560 con 0x5560b99fdd40 2024-03-31T15:07:07.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.006+0000 7f26b1358000 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x5560b9806440 con 0x5560b9a47830 2024-03-31T15:07:07.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.006+0000 7f26b1358000 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x5560b97e59f0 con 0x5560b9a46f10 2024-03-31T15:07:07.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.006+0000 7f26b1358000 10 client.0 _ll_register_callbacks cb 0x5560b99e4bd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T15:07:07.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.006+0000 7f26a6a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5560b99fdd40 0x5560b9a4e060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:07.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.006+0000 7f26a6a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5560b99fdd40 0x5560b9a4e060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53146/0 (socket says 172.21.15.16:53146) 2024-03-31T15:07:07.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.006+0000 7f26a6a00640 1 -- 172.21.15.16:0/1904311043 learned_addr learned my addr 172.21.15.16:0/1904311043 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:07:07.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.006+0000 7f26a7400640 1 --2- 172.21.15.16:0/1904311043 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5560b9a47830 0x5560b9a34860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:07.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.006+0000 7f26a6000640 1 --2- 172.21.15.16:0/1904311043 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5560b9a46f10 0x5560b9a4e5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:07.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.006+0000 7f26a6a00640 1 -- 172.21.15.16:0/1904311043 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5560b9a46f10 msgr2=0x5560b9a4e5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:07:07.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.006+0000 7f26a6a00640 1 --2- 172.21.15.16:0/1904311043 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5560b9a46f10 0x5560b9a4e5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:07.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.006+0000 7f26a6a00640 1 -- 172.21.15.16:0/1904311043 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5560b9a47830 msgr2=0x5560b9a34860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:07:07.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.006+0000 7f26a6a00640 1 --2- 172.21.15.16:0/1904311043 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5560b9a47830 0x5560b9a34860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:07.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.006+0000 7f26a6a00640 1 -- 172.21.15.16:0/1904311043 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5560b98034f0 con 0x5560b99fdd40 2024-03-31T15:07:07.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.006+0000 7f26a6a00640 1 --2- 172.21.15.16:0/1904311043 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5560b99fdd40 0x5560b9a4e060 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f26980ef8f0 tx=0x7f26980b4000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d898241df6793955 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:07:07.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.006+0000 7f2690c00640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-31T15:07:07.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.006+0000 7f2690c00640 1 -- 172.21.15.16:0/1904311043 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2698001800 con 0x5560b99fdd40 2024-03-31T15:07:07.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.006+0000 7f2690c00640 1 -- 172.21.15.16:0/1904311043 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(56..56 src has 1..56) ==== 5709+0+0 (secure 0 0 0) 0x7f26980ef040 con 0x5560b99fdd40 2024-03-31T15:07:07.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.006+0000 7f26b1358000 1 -- 172.21.15.16:0/1904311043 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f26980a9030 con 0x5560b99fdd40 2024-03-31T15:07:07.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.010+0000 7f2690c00640 1 -- 172.21.15.16:0/1904311043 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f26980a6e80 con 0x5560b99fdd40 2024-03-31T15:07:07.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.010+0000 7f26b1358000 10 client.6435 fetch_fsmap learned FSMap version 35 2024-03-31T15:07:07.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.010+0000 7f26b1358000 1 -- 172.21.15.16:0/1904311043 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=2+}) -- 0x5560b98539b0 con 0x5560b99fdd40 2024-03-31T15:07:07.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.010+0000 7f2690c00640 1 -- 172.21.15.16:0/1904311043 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f26980a9030 con 0x5560b99fdd40 2024-03-31T15:07:07.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.010+0000 7f2690c00640 1 -- 172.21.15.16:0/1904311043 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 35) ==== 3494+0+0 (secure 0 0 0) 0x7f2698117b10 con 0x5560b99fdd40 2024-03-31T15:07:07.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.010+0000 7f26b1358000 10 client.6435 fetch_fsmap finished waiting for FSMap version 35 2024-03-31T15:07:07.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.010+0000 7f26b1358000 10 client.6435 resolve_mds: resolved 4:0 to role '4:0' aka daemon mds.c 2024-03-31T15:07:07.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.010+0000 7f26b1358000 20 client.6435 populate_metadata read hostname 'smithi016' 2024-03-31T15:07:07.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.010+0000 7f26b1358000 1 --2- 172.21.15.16:0/1904311043 >> [v2:172.21.15.16:6834/2613117528,v1:172.21.15.16:6835/2613117528] conn(0x5560b9a35600 0x5560b9a359e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:07.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.010+0000 7f26b1358000 4 client.6435 mds_command: new command op to 6150 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-31T15:07:07.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.010+0000 7f26b1358000 1 -- 172.21.15.16:0/1904311043 --> [v2:172.21.15.16:6834/2613117528,v1:172.21.15.16:6835/2613117528] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5560b97e3b40 con 0x5560b9a35600 2024-03-31T15:07:07.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.010+0000 7f26a6000640 1 --2- 172.21.15.16:0/1904311043 >> [v2:172.21.15.16:6834/2613117528,v1:172.21.15.16:6835/2613117528] conn(0x5560b9a35600 0x5560b9a359e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:07.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.010+0000 7f26a6000640 1 --2- 172.21.15.16:0/1904311043 >> [v2:172.21.15.16:6834/2613117528,v1:172.21.15.16:6835/2613117528] conn(0x5560b9a35600 0x5560b9a359e0 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ef6b20bf06bcfbee server_cookie=ecdf026116b21825 in_seq=0 out_seq=0 2024-03-31T15:07:07.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.010+0000 7f2690c00640 10 client.6435 ms_handle_connect on v2:172.21.15.16:6834/2613117528 2024-03-31T15:07:07.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.010+0000 7f2690c00640 1 -- 172.21.15.16:0/1904311043 <== mds.0 v2:172.21.15.16:6834/2613117528 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5560b97e3b40 con 0x5560b9a35600 2024-03-31T15:07:07.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.010+0000 7f2690c00640 10 client.6435 handle_command_reply: tid=0 2024-03-31T15:07:07.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.010+0000 7f26b1358000 1 client.6435 shutdown 2024-03-31T15:07:07.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.010+0000 7f26b1358000 1 -- 172.21.15.16:0/1904311043 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5560b99fdd40 msgr2=0x5560b9a4e060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:07.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.010+0000 7f26b1358000 1 --2- 172.21.15.16:0/1904311043 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5560b99fdd40 0x5560b9a4e060 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f26980ef8f0 tx=0x7f26980b4000 comp rx=0 tx=0).stop 2024-03-31T15:07:07.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.010+0000 7f26b1358000 1 -- 172.21.15.16:0/1904311043 shutdown_connections 2024-03-31T15:07:07.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.010+0000 7f26b1358000 1 --2- 172.21.15.16:0/1904311043 >> [v2:172.21.15.16:6834/2613117528,v1:172.21.15.16:6835/2613117528] conn(0x5560b9a35600 0x5560b9a359e0 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:07.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.010+0000 7f26b1358000 1 --2- 172.21.15.16:0/1904311043 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5560b9a47830 0x5560b9a34860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:07.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.010+0000 7f26b1358000 1 --2- 172.21.15.16:0/1904311043 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5560b9a46f10 0x5560b9a4e5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:07.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.010+0000 7f26b1358000 1 --2- 172.21.15.16:0/1904311043 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5560b99fdd40 0x5560b9a4e060 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:07.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.010+0000 7f2690c00640 0 client.6435 ms_handle_reset on v2:172.21.15.16:6834/2613117528 2024-03-31T15:07:07.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.010+0000 7f26b1358000 1 -- 172.21.15.16:0/1904311043 >> 172.21.15.16:0/1904311043 conn(0x5560b98e0db0 msgr2=0x5560b99ff0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:07:07.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.010+0000 7f26b1358000 1 -- 172.21.15.16:0/1904311043 shutdown_connections 2024-03-31T15:07:07.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.010+0000 7f26b1358000 1 -- 172.21.15.16:0/1904311043 wait complete. 2024-03-31T15:07:07.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.010+0000 7f26b1358000 20 client.6435 trim_cache size 0 max 16384 2024-03-31T15:07:07.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.030+0000 7f26b1358000 1 Processor -- start 2024-03-31T15:07:07.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.030+0000 7f26b1358000 1 -- start start 2024-03-31T15:07:07.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.030+0000 7f26b1358000 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5560b99fdd40 0x5560b99ff070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:07.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.030+0000 7f26b1358000 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5560b9a46f10 0x5560b99fe120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:07.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.030+0000 7f26b1358000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5560b9a47830 0x5560b99fe660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:07.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.030+0000 7f26b1358000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f26a8200230 con 0x5560b9a47830 2024-03-31T15:07:07.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.030+0000 7f26b1358000 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f26a81ff290 con 0x5560b9a46f10 2024-03-31T15:07:07.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.030+0000 7f26b1358000 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f26a806e980 con 0x5560b99fdd40 2024-03-31T15:07:07.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.030+0000 7f26a7400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5560b9a47830 0x5560b99fe660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:07.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.030+0000 7f26a7400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5560b9a47830 0x5560b99fe660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53158/0 (socket says 172.21.15.16:53158) 2024-03-31T15:07:07.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.030+0000 7f26a7400640 1 -- 172.21.15.16:0/4163452846 learned_addr learned my addr 172.21.15.16:0/4163452846 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:07:07.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.030+0000 7f26a6000640 1 --2- 172.21.15.16:0/4163452846 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5560b9a46f10 0x5560b99fe120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:07.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.030+0000 7f26a6a00640 1 --2- 172.21.15.16:0/4163452846 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5560b99fdd40 0x5560b99ff070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:07.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.030+0000 7f26a7400640 1 -- 172.21.15.16:0/4163452846 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5560b99fdd40 msgr2=0x5560b99ff070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:07:07.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.030+0000 7f26a7400640 1 --2- 172.21.15.16:0/4163452846 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5560b99fdd40 0x5560b99ff070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:07.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.030+0000 7f26a7400640 1 -- 172.21.15.16:0/4163452846 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5560b9a46f10 msgr2=0x5560b99fe120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:07:07.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.030+0000 7f26a7400640 1 --2- 172.21.15.16:0/4163452846 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5560b9a46f10 0x5560b99fe120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:07.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.030+0000 7f26a7400640 1 -- 172.21.15.16:0/4163452846 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5560b9a47660 con 0x5560b9a47830 2024-03-31T15:07:07.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.030+0000 7f26a6000640 1 --2- 172.21.15.16:0/4163452846 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5560b9a46f10 0x5560b99fe120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:07:07.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.030+0000 7f26a7400640 1 --2- 172.21.15.16:0/4163452846 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5560b9a47830 0x5560b99fe660 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f26a80b8220 tx=0x7f26a0009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e18f3db78003079 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:07:07.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.030+0000 7f2693400640 1 -- 172.21.15.16:0/4163452846 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26a00180d0 con 0x5560b9a47830 2024-03-31T15:07:07.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.030+0000 7f2693400640 1 -- 172.21.15.16:0/4163452846 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f26a0003bb0 con 0x5560b9a47830 2024-03-31T15:07:07.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.030+0000 7f2693400640 1 -- 172.21.15.16:0/4163452846 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26a0004030 con 0x5560b9a47830 2024-03-31T15:07:07.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.030+0000 7f26b1358000 1 -- 172.21.15.16:0/4163452846 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5560b9a47830 msgr2=0x5560b99fe660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:07.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.030+0000 7f26b1358000 1 --2- 172.21.15.16:0/4163452846 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5560b9a47830 0x5560b99fe660 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f26a80b8220 tx=0x7f26a0009750 comp rx=0 tx=0).stop 2024-03-31T15:07:07.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.030+0000 7f26b1358000 1 -- 172.21.15.16:0/4163452846 shutdown_connections 2024-03-31T15:07:07.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.030+0000 7f26b1358000 1 --2- 172.21.15.16:0/4163452846 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5560b9a47830 0x5560b99fe660 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:07.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.030+0000 7f26b1358000 1 --2- 172.21.15.16:0/4163452846 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5560b9a46f10 0x5560b99fe120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:07.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.030+0000 7f26b1358000 1 --2- 172.21.15.16:0/4163452846 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5560b99fdd40 0x5560b99ff070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:07.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.030+0000 7f26b1358000 1 -- 172.21.15.16:0/4163452846 >> 172.21.15.16:0/4163452846 conn(0x5560b98e0db0 msgr2=0x5560b9a462f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:07:07.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.030+0000 7f26b1358000 1 -- 172.21.15.16:0/4163452846 shutdown_connections 2024-03-31T15:07:07.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.030+0000 7f26b1358000 1 -- 172.21.15.16:0/4163452846 wait complete. 2024-03-31T15:07:07.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.034+0000 7f26b1358000 1 -- start start 2024-03-31T15:07:07.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.034+0000 7f26b1358000 1 Processor -- start 2024-03-31T15:07:07.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.034+0000 7f26b1358000 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5560b99fdd40 0x5560b9ab7750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:07.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.034+0000 7f26b1358000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5560b9a46f10 0x5560b9ab7c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:07.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.034+0000 7f26b1358000 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5560b9a47830 0x5560b9a4ac70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:07.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.034+0000 7f26b1358000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5560b9812a90 con 0x5560b9a46f10 2024-03-31T15:07:07.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.034+0000 7f26b1358000 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x5560b9903c30 con 0x5560b9a47830 2024-03-31T15:07:07.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.034+0000 7f26b1358000 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x5560b9844910 con 0x5560b99fdd40 2024-03-31T15:07:07.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.034+0000 7f26b1358000 10 client.0 _ll_register_callbacks cb 0x7f26980f6050 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T15:07:07.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.034+0000 7f26a6000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5560b9a46f10 0x5560b9ab7c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:07.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.034+0000 7f26a6a00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5560b99fdd40 0x5560b9ab7750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:07.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.034+0000 7f26a6000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5560b9a46f10 0x5560b9ab7c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53172/0 (socket says 172.21.15.16:53172) 2024-03-31T15:07:07.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.034+0000 7f26a6000640 1 -- 172.21.15.16:0/1791000321 learned_addr learned my addr 172.21.15.16:0/1791000321 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:07:07.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.034+0000 7f26a7400640 1 --2- 172.21.15.16:0/1791000321 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5560b9a47830 0x5560b9a4ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:07.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.034+0000 7f26a6000640 1 -- 172.21.15.16:0/1791000321 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5560b99fdd40 msgr2=0x5560b9ab7750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:07:07.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.034+0000 7f26a6000640 1 --2- 172.21.15.16:0/1791000321 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5560b99fdd40 0x5560b9ab7750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:07.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.034+0000 7f26a6000640 1 -- 172.21.15.16:0/1791000321 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5560b9a47830 msgr2=0x5560b9a4ac70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:07:07.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.034+0000 7f26a6000640 1 --2- 172.21.15.16:0/1791000321 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5560b9a47830 0x5560b9a4ac70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:07.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.034+0000 7f26a6000640 1 -- 172.21.15.16:0/1791000321 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5560b98c9820 con 0x5560b9a46f10 2024-03-31T15:07:07.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.034+0000 7f26a6000640 1 --2- 172.21.15.16:0/1791000321 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5560b9a46f10 0x5560b9ab7c90 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f26a80b1710 tx=0x7f2694007cd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=64e64edd04ef92a0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:07:07.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.034+0000 7f2690c00640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-31T15:07:07.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.034+0000 7f2690c00640 1 -- 172.21.15.16:0/1791000321 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2694019040 con 0x5560b9a46f10 2024-03-31T15:07:07.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.034+0000 7f2690c00640 1 -- 172.21.15.16:0/1791000321 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(56..56 src has 1..56) ==== 5709+0+0 (secure 0 0 0) 0x7f2694015650 con 0x5560b9a46f10 2024-03-31T15:07:07.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.034+0000 7f26b1358000 1 -- 172.21.15.16:0/1791000321 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f26a0003bb0 con 0x5560b9a46f10 2024-03-31T15:07:07.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.034+0000 7f2690c00640 1 -- 172.21.15.16:0/1791000321 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f269403b050 con 0x5560b9a46f10 2024-03-31T15:07:07.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.034+0000 7f26b1358000 10 client.6441 fetch_fsmap learned FSMap version 35 2024-03-31T15:07:07.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.034+0000 7f26b1358000 1 -- 172.21.15.16:0/1791000321 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=2+}) -- 0x5560b97d35f0 con 0x5560b9a46f10 2024-03-31T15:07:07.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.034+0000 7f2690c00640 1 -- 172.21.15.16:0/1791000321 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f26a0003bb0 con 0x5560b9a46f10 2024-03-31T15:07:07.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.034+0000 7f2690c00640 1 -- 172.21.15.16:0/1791000321 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 35) ==== 3494+0+0 (secure 0 0 0) 0x7f269403e020 con 0x5560b9a46f10 2024-03-31T15:07:07.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.034+0000 7f26b1358000 10 client.6441 fetch_fsmap finished waiting for FSMap version 35 2024-03-31T15:07:07.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.034+0000 7f26b1358000 10 client.6441 resolve_mds: resolved 4:0 to role '4:0' aka daemon mds.c 2024-03-31T15:07:07.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.034+0000 7f26b1358000 20 client.6441 populate_metadata read hostname 'smithi016' 2024-03-31T15:07:07.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.034+0000 7f26b1358000 1 --2- 172.21.15.16:0/1791000321 >> [v2:172.21.15.16:6834/2613117528,v1:172.21.15.16:6835/2613117528] conn(0x5560b9a4b1b0 0x5560b9a4b5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:07.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.034+0000 7f26b1358000 4 client.6441 mds_command: new command op to 6150 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["repair", "recursive", "force"]}] 2024-03-31T15:07:07.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.034+0000 7f26b1358000 1 -- 172.21.15.16:0/1791000321 --> [v2:172.21.15.16:6834/2613117528,v1:172.21.15.16:6835/2613117528] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["repair", "recursive", "force"]}) -- 0x5560b9855ea0 con 0x5560b9a4b1b0 2024-03-31T15:07:07.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.034+0000 7f26a6a00640 1 --2- 172.21.15.16:0/1791000321 >> [v2:172.21.15.16:6834/2613117528,v1:172.21.15.16:6835/2613117528] conn(0x5560b9a4b1b0 0x5560b9a4b5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:07.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.038+0000 7f2690c00640 10 client.6441 ms_handle_connect on v2:172.21.15.16:6834/2613117528 2024-03-31T15:07:07.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.038+0000 7f26a6a00640 1 --2- 172.21.15.16:0/1791000321 >> [v2:172.21.15.16:6834/2613117528,v1:172.21.15.16:6835/2613117528] conn(0x5560b9a4b1b0 0x5560b9a4b5c0 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c0ce9b39752a5fbc server_cookie=78e68f5fee055602 in_seq=0 out_seq=0 2024-03-31T15:07:07.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.038+0000 7f2690c00640 1 -- 172.21.15.16:0/1791000321 <== mds.0 v2:172.21.15.16:6834/2613117528 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x5560b9855ea0 con 0x5560b9a4b1b0 2024-03-31T15:07:07.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.038+0000 7f2690c00640 10 client.6441 handle_command_reply: tid=0 2024-03-31T15:07:07.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.038+0000 7f26b1358000 1 client.6441 shutdown 2024-03-31T15:07:07.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.038+0000 7f26b1358000 1 -- 172.21.15.16:0/1791000321 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5560b9a46f10 msgr2=0x5560b9ab7c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:07.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.038+0000 7f26b1358000 1 --2- 172.21.15.16:0/1791000321 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5560b9a46f10 0x5560b9ab7c90 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f26a80b1710 tx=0x7f2694007cd0 comp rx=0 tx=0).stop 2024-03-31T15:07:07.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.038+0000 7f26b1358000 1 -- 172.21.15.16:0/1791000321 shutdown_connections 2024-03-31T15:07:07.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.038+0000 7f26b1358000 1 --2- 172.21.15.16:0/1791000321 >> [v2:172.21.15.16:6834/2613117528,v1:172.21.15.16:6835/2613117528] conn(0x5560b9a4b1b0 0x5560b9a4b5c0 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:07.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.038+0000 7f26b1358000 1 --2- 172.21.15.16:0/1791000321 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5560b9a47830 0x5560b9a4ac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:07.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.038+0000 7f26b1358000 1 --2- 172.21.15.16:0/1791000321 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5560b9a46f10 0x5560b9ab7c90 unknown :-1 s=CLOSED pgs=709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:07.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.038+0000 7f26b1358000 1 --2- 172.21.15.16:0/1791000321 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5560b99fdd40 0x5560b9ab7750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:07.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.038+0000 7f2690c00640 0 client.6441 ms_handle_reset on v2:172.21.15.16:6834/2613117528 2024-03-31T15:07:07.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.038+0000 7f26b1358000 1 -- 172.21.15.16:0/1791000321 >> 172.21.15.16:0/1791000321 conn(0x5560b98e0db0 msgr2=0x5560b99f7440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:07:07.038 INFO:tasks.ceph.mds.c.smithi016.stderr:2024-03-31T15:07:07.038+0000 7f4ef1400640 -1 mds.0.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000001 [2,head] /dir_test/file auth v4 ap=1 DIRTYPARENT s=0 n(v0 rc2024-03-31T15:06:34.177259+0000 1=1+0) caps={6303=pAsLsXsFs/-@2},l=6303(-1) | ptrwaiter=0 request=0 lock=0 caps=1 dirtyparent=1 dirty=1 authpin=1 scrubqueue=0 0x564c5b37e580]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(-1)0x0:[]//[]","memoryvalue":"(9)0x10000000001:[<0x10000000000/file v4>,<0x1/dir_test v7>]//[]","error_str":"failed to decode on-disk backtrace (26 bytes)!"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-31T15:07:07.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.038+0000 7f26b1358000 1 -- 172.21.15.16:0/1791000321 shutdown_connections 2024-03-31T15:07:07.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.038+0000 7f26b1358000 1 -- 172.21.15.16:0/1791000321 wait complete. 2024-03-31T15:07:07.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.038+0000 7f26b1358000 20 client.6441 trim_cache size 0 max 16384 2024-03-31T15:07:07.039 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-03-31T15:07:07.039 INFO:teuthology.orchestra.run.smithi016.stdout: "return_code": 0, 2024-03-31T15:07:07.039 INFO:teuthology.orchestra.run.smithi016.stdout: "scrub_tag": "de39ecf3-9b54-49b9-a421-967d384cc9f3", 2024-03-31T15:07:07.039 INFO:teuthology.orchestra.run.smithi016.stdout: "mode": "asynchronous" 2024-03-31T15:07:07.039 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-03-31T15:07:07.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.038+0000 7f26ade00640 1 -- 172.21.15.16:0/3307951858 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f26880683b0 msgr2=0x7f268806a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:07.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.038+0000 7f26ade00640 1 --2- 172.21.15.16:0/3307951858 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f26880683b0 0x7f268806a870 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f26a8154000 tx=0x7f2694009040 comp rx=0 tx=0).stop 2024-03-31T15:07:07.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.038+0000 7f26ade00640 1 -- 172.21.15.16:0/3307951858 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f268c003ba0 msgr2=0x7f26a8155270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:07.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.038+0000 7f26ade00640 1 --2- 172.21.15.16:0/3307951858 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f268c003ba0 0x7f26a8155270 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f2698000c10 tx=0x7f268c005840 comp rx=0 tx=0).stop 2024-03-31T15:07:07.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.038+0000 7f26ade00640 1 -- 172.21.15.16:0/3307951858 shutdown_connections 2024-03-31T15:07:07.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.038+0000 7f26ade00640 1 --2- 172.21.15.16:0/3307951858 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f26880683b0 0x7f268806a870 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:07.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.038+0000 7f26ade00640 1 --2- 172.21.15.16:0/3307951858 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f26a8140a90 0x7f26a8140f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:07.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.038+0000 7f26ade00640 1 --2- 172.21.15.16:0/3307951858 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f26a80b6e70 0x7f26a8152430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:07.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.038+0000 7f26ade00640 1 --2- 172.21.15.16:0/3307951858 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f268c003ba0 0x7f26a8155270 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:07.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.038+0000 7f26ade00640 1 -- 172.21.15.16:0/3307951858 >> 172.21.15.16:0/3307951858 conn(0x7f26a806b7f0 msgr2=0x7f26a80b1da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:07:07.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.038+0000 7f26ade00640 1 -- 172.21.15.16:0/3307951858 shutdown_connections 2024-03-31T15:07:07.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.038+0000 7f26ade00640 1 -- 172.21.15.16:0/3307951858 wait complete. 2024-03-31T15:07:07.053 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:07:07.053 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.4:0 scrub status 2024-03-31T15:07:07.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.242+0000 7f13b9a00640 1 Processor -- start 2024-03-31T15:07:07.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.242+0000 7f13b9a00640 1 -- start start 2024-03-31T15:07:07.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.242+0000 7f13b9a00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f13b41567e0 0x7f13b4072c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:07.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.242+0000 7f13b9a00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f13b40731c0 0x7f13b406d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:07.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.242+0000 7f13b9a00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f13b406da10 0x7f13b415fdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:07.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.242+0000 7f13b9a00640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f13b406de80 con 0x7f13b40731c0 2024-03-31T15:07:07.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.242+0000 7f13b9a00640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f13b415c7e0 con 0x7f13b406da10 2024-03-31T15:07:07.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.242+0000 7f13b9a00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f13b40736b0 con 0x7f13b41567e0 2024-03-31T15:07:07.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.242+0000 7f13abe00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f13b40731c0 0x7f13b406d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:07.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.242+0000 7f13abe00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f13b40731c0 0x7f13b406d4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57254/0 (socket says 172.21.15.16:57254) 2024-03-31T15:07:07.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.242+0000 7f13abe00640 1 -- 172.21.15.16:0/2542934346 learned_addr learned my addr 172.21.15.16:0/2542934346 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:07:07.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.242+0000 7f13b3400640 1 --2- 172.21.15.16:0/2542934346 >> v2:172.21.15.89:3300/0 conn(0x7f13b406da10 0x7f13b415fdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:07.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.242+0000 7f13b2a00640 1 --2- 172.21.15.16:0/2542934346 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f13b41567e0 0x7f13b4072c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:07.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.242+0000 7f13abe00640 1 -- 172.21.15.16:0/2542934346 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f13b41567e0 msgr2=0x7f13b4072c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:07:07.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.242+0000 7f13abe00640 1 --2- 172.21.15.16:0/2542934346 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f13b41567e0 0x7f13b4072c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:07.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.242+0000 7f13abe00640 1 -- 172.21.15.16:0/2542934346 >> v2:172.21.15.89:3300/0 conn(0x7f13b406da10 msgr2=0x7f13b415fdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:07:07.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.242+0000 7f13abe00640 1 --2- 172.21.15.16:0/2542934346 >> v2:172.21.15.89:3300/0 conn(0x7f13b406da10 0x7f13b415fdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:07.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.242+0000 7f13abe00640 1 -- 172.21.15.16:0/2542934346 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13b415caa0 con 0x7f13b40731c0 2024-03-31T15:07:07.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.242+0000 7f13b3400640 1 --2- 172.21.15.16:0/2542934346 >> v2:172.21.15.89:3300/0 conn(0x7f13b406da10 0x7f13b415fdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:07:07.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.246+0000 7f13abe00640 1 --2- 172.21.15.16:0/2542934346 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f13b40731c0 0x7f13b406d4d0 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f13b40714e0 tx=0x7f139c01c260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=303bee229c2f323f server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:07:07.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.250+0000 7f13b2000640 1 -- 172.21.15.16:0/2542934346 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f139c0b0070 con 0x7f13b40731c0 2024-03-31T15:07:07.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.250+0000 7f13b2000640 1 -- 172.21.15.16:0/2542934346 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f139c0ab600 con 0x7f13b40731c0 2024-03-31T15:07:07.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.250+0000 7f13b2000640 1 -- 172.21.15.16:0/2542934346 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f139c0ab9d0 con 0x7f13b40731c0 2024-03-31T15:07:07.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.250+0000 7f13b9a00640 1 -- 172.21.15.16:0/2542934346 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f13b40731c0 msgr2=0x7f13b406d4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:07.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.250+0000 7f13b9a00640 1 --2- 172.21.15.16:0/2542934346 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f13b40731c0 0x7f13b406d4d0 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f13b40714e0 tx=0x7f139c01c260 comp rx=0 tx=0).stop 2024-03-31T15:07:07.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.250+0000 7f13b9a00640 1 -- 172.21.15.16:0/2542934346 shutdown_connections 2024-03-31T15:07:07.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.250+0000 7f13b9a00640 1 --2- 172.21.15.16:0/2542934346 >> v2:172.21.15.89:3300/0 conn(0x7f13b406da10 0x7f13b415fdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:07.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.250+0000 7f13b9a00640 1 --2- 172.21.15.16:0/2542934346 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f13b40731c0 0x7f13b406d4d0 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:07.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.250+0000 7f13b9a00640 1 --2- 172.21.15.16:0/2542934346 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f13b41567e0 0x7f13b4072c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:07.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.250+0000 7f13b9a00640 1 -- 172.21.15.16:0/2542934346 >> 172.21.15.16:0/2542934346 conn(0x7f13b406ee40 msgr2=0x7f13b406b620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:07:07.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.250+0000 7f13b9a00640 1 -- 172.21.15.16:0/2542934346 shutdown_connections 2024-03-31T15:07:07.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.250+0000 7f13b9a00640 1 -- 172.21.15.16:0/2542934346 wait complete. 2024-03-31T15:07:07.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.250+0000 7f13b9a00640 1 Processor -- start 2024-03-31T15:07:07.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.250+0000 7f13b9a00640 1 -- start start 2024-03-31T15:07:07.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.250+0000 7f13b9a00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f13b406da10 0x7f13b4153ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:07.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.250+0000 7f13b9a00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f13b40731c0 0x7f13b415abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:07.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.250+0000 7f13b9a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f13b41567e0 0x7f13b40b0040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:07.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.250+0000 7f13b9a00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f13b415d6e0 con 0x7f13b41567e0 2024-03-31T15:07:07.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.250+0000 7f13b9a00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f13b415d560 con 0x7f13b40731c0 2024-03-31T15:07:07.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.250+0000 7f13b9a00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f13b4071550 con 0x7f13b406da10 2024-03-31T15:07:07.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.250+0000 7f13b3400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f13b41567e0 0x7f13b40b0040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:07.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.250+0000 7f13b3400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f13b41567e0 0x7f13b40b0040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57270/0 (socket says 172.21.15.16:57270) 2024-03-31T15:07:07.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.250+0000 7f13b3400640 1 -- 172.21.15.16:0/1686426483 learned_addr learned my addr 172.21.15.16:0/1686426483 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:07:07.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.254+0000 7f13b2a00640 1 --2- 172.21.15.16:0/1686426483 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f13b406da10 0x7f13b4153ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:07.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.254+0000 7f13abe00640 1 --2- 172.21.15.16:0/1686426483 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f13b40731c0 0x7f13b415abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:07.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.254+0000 7f13b3400640 1 -- 172.21.15.16:0/1686426483 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f13b406da10 msgr2=0x7f13b4153ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:07.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.254+0000 7f13b3400640 1 --2- 172.21.15.16:0/1686426483 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f13b406da10 0x7f13b4153ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:07.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.254+0000 7f13b3400640 1 -- 172.21.15.16:0/1686426483 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f13b40731c0 msgr2=0x7f13b415abf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:07.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.254+0000 7f13b3400640 1 --2- 172.21.15.16:0/1686426483 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f13b40731c0 0x7f13b415abf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:07.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.254+0000 7f13b3400640 1 -- 172.21.15.16:0/1686426483 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13b41540a0 con 0x7f13b41567e0 2024-03-31T15:07:07.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.254+0000 7f13b3400640 1 --2- 172.21.15.16:0/1686426483 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f13b41567e0 0x7f13b40b0040 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f13ac007e10 tx=0x7f13ac00d4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:07:07.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.254+0000 7f13aaa00640 1 -- 172.21.15.16:0/1686426483 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13ac01a070 con 0x7f13b41567e0 2024-03-31T15:07:07.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.254+0000 7f13aaa00640 1 -- 172.21.15.16:0/1686426483 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f13ac003ba0 con 0x7f13b41567e0 2024-03-31T15:07:07.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.254+0000 7f13b9a00640 1 -- 172.21.15.16:0/1686426483 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f13b4154380 con 0x7f13b41567e0 2024-03-31T15:07:07.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.254+0000 7f13aaa00640 1 -- 172.21.15.16:0/1686426483 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13ac004030 con 0x7f13b41567e0 2024-03-31T15:07:07.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.254+0000 7f13b9a00640 1 -- 172.21.15.16:0/1686426483 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f13b4154940 con 0x7f13b41567e0 2024-03-31T15:07:07.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.258+0000 7f13aaa00640 1 -- 172.21.15.16:0/1686426483 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f13ac01d070 con 0x7f13b41567e0 2024-03-31T15:07:07.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.258+0000 7f13aaa00640 1 --2- 172.21.15.16:0/1686426483 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f1390068380 0x7f139006a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:07.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.258+0000 7f13aaa00640 1 -- 172.21.15.16:0/1686426483 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 5709+0+0 (secure 0 0 0) 0x7f13ac086be0 con 0x7f13b41567e0 2024-03-31T15:07:07.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.258+0000 7f13bcf66000 1 Processor -- start 2024-03-31T15:07:07.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.258+0000 7f13bcf66000 1 -- start start 2024-03-31T15:07:07.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.258+0000 7f13b2a00640 1 --2- 172.21.15.16:0/1686426483 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f1390068380 0x7f139006a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:07.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.258+0000 7f13bcf66000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558df5cac2d0 0x558df5cac6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:07.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.258+0000 7f13bcf66000 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558df5cacbf0 0x558df5c5f470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:07.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.258+0000 7f13bcf66000 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x558df5c5f9b0 0x558df5c5fd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:07.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.258+0000 7f13bcf66000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x558df5b48af0 con 0x558df5cac2d0 2024-03-31T15:07:07.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.258+0000 7f13bcf66000 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x558df5ab1b00 con 0x558df5cacbf0 2024-03-31T15:07:07.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.258+0000 7f13bcf66000 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x558df5a4e940 con 0x558df5c5f9b0 2024-03-31T15:07:07.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.258+0000 7f13b2a00640 1 --2- 172.21.15.16:0/1686426483 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f1390068380 0x7f139006a840 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f13a40079c0 tx=0x7f13a4008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:07:07.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.258+0000 7f13b2a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558df5cac2d0 0x558df5cac6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:07.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.258+0000 7f13b2a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558df5cac2d0 0x558df5cac6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57286/0 (socket says 172.21.15.16:57286) 2024-03-31T15:07:07.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.258+0000 7f13b2a00640 1 -- 172.21.15.16:0/2922518310 learned_addr learned my addr 172.21.15.16:0/2922518310 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:07:07.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.258+0000 7f13abe00640 1 --2- 172.21.15.16:0/2922518310 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558df5cacbf0 0x558df5c5f470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:07.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.258+0000 7f13b3400640 1 --2- 172.21.15.16:0/2922518310 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x558df5c5f9b0 0x558df5c5fd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:07.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.258+0000 7f13b2a00640 1 -- 172.21.15.16:0/2922518310 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x558df5c5f9b0 msgr2=0x558df5c5fd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:07:07.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.258+0000 7f13b2a00640 1 --2- 172.21.15.16:0/2922518310 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x558df5c5f9b0 0x558df5c5fd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:07.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.258+0000 7f13b2a00640 1 -- 172.21.15.16:0/2922518310 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558df5cacbf0 msgr2=0x558df5c5f470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:07:07.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.258+0000 7f13b2a00640 1 --2- 172.21.15.16:0/2922518310 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558df5cacbf0 0x558df5c5f470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:07.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.258+0000 7f13b2a00640 1 -- 172.21.15.16:0/2922518310 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558df5a37380 con 0x558df5cac2d0 2024-03-31T15:07:07.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.258+0000 7f13b3400640 1 --2- 172.21.15.16:0/2922518310 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x558df5c5f9b0 0x558df5c5fd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:07:07.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.258+0000 7f13b2a00640 1 --2- 172.21.15.16:0/2922518310 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558df5cac2d0 0x558df5cac6b0 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f13a4010af0 tx=0x7f13a4013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fdb0972e326c554d server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:07:07.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.258+0000 7f13a3400640 1 -- 172.21.15.16:0/2922518310 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13a401c050 con 0x558df5cac2d0 2024-03-31T15:07:07.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.258+0000 7f13a3400640 1 -- 172.21.15.16:0/2922518310 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f13a4026070 con 0x558df5cac2d0 2024-03-31T15:07:07.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.258+0000 7f13a3400640 1 -- 172.21.15.16:0/2922518310 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13a4021440 con 0x558df5cac2d0 2024-03-31T15:07:07.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.258+0000 7f13bcf66000 1 -- 172.21.15.16:0/2922518310 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558df5cac2d0 msgr2=0x558df5cac6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:07.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.258+0000 7f13bcf66000 1 --2- 172.21.15.16:0/2922518310 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558df5cac2d0 0x558df5cac6b0 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f13a4010af0 tx=0x7f13a4013000 comp rx=0 tx=0).stop 2024-03-31T15:07:07.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.258+0000 7f13bcf66000 1 -- 172.21.15.16:0/2922518310 shutdown_connections 2024-03-31T15:07:07.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.258+0000 7f13bcf66000 1 --2- 172.21.15.16:0/2922518310 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x558df5c5f9b0 0x558df5c5fd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:07.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.258+0000 7f13bcf66000 1 --2- 172.21.15.16:0/2922518310 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558df5cacbf0 0x558df5c5f470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:07.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.258+0000 7f13bcf66000 1 --2- 172.21.15.16:0/2922518310 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558df5cac2d0 0x558df5cac6b0 unknown :-1 s=CLOSED pgs=712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:07.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.262+0000 7f13bcf66000 1 -- 172.21.15.16:0/2922518310 >> 172.21.15.16:0/2922518310 conn(0x558df5c8d070 msgr2=0x558df5c8f460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:07:07.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.262+0000 7f13bcf66000 1 -- 172.21.15.16:0/2922518310 shutdown_connections 2024-03-31T15:07:07.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.262+0000 7f13bcf66000 1 -- 172.21.15.16:0/2922518310 wait complete. 2024-03-31T15:07:07.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.262+0000 7f13bcf66000 1 -- start start 2024-03-31T15:07:07.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.262+0000 7f13bcf66000 1 Processor -- start 2024-03-31T15:07:07.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.262+0000 7f13bcf66000 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558df5c5f9b0 0x558df5cb3420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:07.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.262+0000 7f13bcf66000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558df5cac2d0 0x558df5cb3960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:07.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.262+0000 7f13bcf66000 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x558df5cacbf0 0x558df5c644e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:07.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.262+0000 7f13bcf66000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x558df5a6f1e0 con 0x558df5cac2d0 2024-03-31T15:07:07.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.262+0000 7f13bcf66000 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x558df5aadab0 con 0x558df5c5f9b0 2024-03-31T15:07:07.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.262+0000 7f13bcf66000 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x558df5c6b000 con 0x558df5cacbf0 2024-03-31T15:07:07.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.262+0000 7f13bcf66000 10 client.0 _ll_register_callbacks cb 0x558df5c4d370 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T15:07:07.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.262+0000 7f13abe00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558df5cac2d0 0x558df5cb3960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:07.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.262+0000 7f13b2a00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558df5c5f9b0 0x558df5cb3420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:07.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.262+0000 7f13b2a00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558df5c5f9b0 0x558df5cb3420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.16:58244/0 (socket says 172.21.15.16:58244) 2024-03-31T15:07:07.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.262+0000 7f13b2a00640 1 -- 172.21.15.16:0/2821091294 learned_addr learned my addr 172.21.15.16:0/2821091294 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:07:07.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.262+0000 7f13b3400640 1 --2- 172.21.15.16:0/2821091294 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x558df5cacbf0 0x558df5c644e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:07.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.262+0000 7f13b2a00640 1 -- 172.21.15.16:0/2821091294 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x558df5cacbf0 msgr2=0x558df5c644e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:07:07.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.262+0000 7f13b2a00640 1 --2- 172.21.15.16:0/2821091294 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x558df5cacbf0 0x558df5c644e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:07.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.262+0000 7f13b2a00640 1 -- 172.21.15.16:0/2821091294 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558df5cac2d0 msgr2=0x558df5cb3960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:07:07.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.262+0000 7f13b2a00640 1 --2- 172.21.15.16:0/2821091294 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558df5cac2d0 0x558df5cb3960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:07.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.262+0000 7f13b2a00640 1 -- 172.21.15.16:0/2821091294 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558df5a6db70 con 0x558df5c5f9b0 2024-03-31T15:07:07.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.262+0000 7f13abe00640 1 --2- 172.21.15.16:0/2821091294 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558df5cac2d0 0x558df5cb3960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:07:07.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.262+0000 7f13b2a00640 1 --2- 172.21.15.16:0/2821091294 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558df5c5f9b0 0x558df5cb3420 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x558df5ab7ae0 tx=0x7f13a4004740 comp rx=0 tx=0).ready entity=mon.1 client_cookie=457c9f95dbb0ef96 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:07:07.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.262+0000 7f1397e00640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-03-31T15:07:07.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.262+0000 7f1397e00640 1 -- 172.21.15.16:0/2821091294 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13a4021930 con 0x558df5c5f9b0 2024-03-31T15:07:07.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.262+0000 7f1397e00640 1 -- 172.21.15.16:0/2821091294 <== mon.1 v2:172.21.15.89:3300/0 2 ==== osd_map(56..56 src has 1..56) ==== 5709+0+0 (secure 0 0 0) 0x7f13a4036ae0 con 0x558df5c5f9b0 2024-03-31T15:07:07.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.262+0000 7f13bcf66000 1 -- 172.21.15.16:0/2821091294 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f13a4026070 con 0x558df5c5f9b0 2024-03-31T15:07:07.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.262+0000 7f1397e00640 1 -- 172.21.15.16:0/2821091294 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f13a401c020 con 0x558df5c5f9b0 2024-03-31T15:07:07.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.262+0000 7f13bcf66000 10 client.6376 fetch_fsmap learned FSMap version 35 2024-03-31T15:07:07.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.262+0000 7f13bcf66000 1 -- 172.21.15.16:0/2821091294 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=2+}) -- 0x558df5a4eb90 con 0x558df5c5f9b0 2024-03-31T15:07:07.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.266+0000 7f1397e00640 1 -- 172.21.15.16:0/2821091294 <== mon.1 v2:172.21.15.89:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f13a4026070 con 0x558df5c5f9b0 2024-03-31T15:07:07.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.266+0000 7f1397e00640 1 -- 172.21.15.16:0/2821091294 <== mon.1 v2:172.21.15.89:3300/0 5 ==== fsmap(e 35) ==== 3494+0+0 (secure 0 0 0) 0x7f13a4021ad0 con 0x558df5c5f9b0 2024-03-31T15:07:07.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.266+0000 7f13bcf66000 10 client.6376 fetch_fsmap finished waiting for FSMap version 35 2024-03-31T15:07:07.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.266+0000 7f13bcf66000 10 client.6376 resolve_mds: resolved 4:0 to role '4:0' aka daemon mds.c 2024-03-31T15:07:07.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.266+0000 7f13bcf66000 20 client.6376 populate_metadata read hostname 'smithi016' 2024-03-31T15:07:07.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.266+0000 7f13bcf66000 1 --2- 172.21.15.16:0/2821091294 >> [v2:172.21.15.16:6834/2613117528,v1:172.21.15.16:6835/2613117528] conn(0x558df5c65280 0x558df5c65660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:07.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.266+0000 7f13bcf66000 4 client.6376 mds_command: new command op to 6150 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-31T15:07:07.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.266+0000 7f13bcf66000 1 -- 172.21.15.16:0/2821091294 --> [v2:172.21.15.16:6834/2613117528,v1:172.21.15.16:6835/2613117528] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558df5a4cdd0 con 0x558df5c65280 2024-03-31T15:07:07.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.266+0000 7f13abe00640 1 --2- 172.21.15.16:0/2821091294 >> [v2:172.21.15.16:6834/2613117528,v1:172.21.15.16:6835/2613117528] conn(0x558df5c65280 0x558df5c65660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:07.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.266+0000 7f1397e00640 10 client.6376 ms_handle_connect on v2:172.21.15.16:6834/2613117528 2024-03-31T15:07:07.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.266+0000 7f13abe00640 1 --2- 172.21.15.16:0/2821091294 >> [v2:172.21.15.16:6834/2613117528,v1:172.21.15.16:6835/2613117528] conn(0x558df5c65280 0x558df5c65660 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=68f61b5bc7256edb server_cookie=ae89a0e76121b9a4 in_seq=0 out_seq=0 2024-03-31T15:07:07.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.266+0000 7f1397e00640 1 -- 172.21.15.16:0/2821091294 <== mds.0 v2:172.21.15.16:6834/2613117528 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x558df5a4cdd0 con 0x558df5c65280 2024-03-31T15:07:07.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.266+0000 7f1397e00640 10 client.6376 handle_command_reply: tid=0 2024-03-31T15:07:07.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.266+0000 7f13bcf66000 1 client.6376 shutdown 2024-03-31T15:07:07.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.266+0000 7f13bcf66000 1 -- 172.21.15.16:0/2821091294 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558df5c5f9b0 msgr2=0x558df5cb3420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:07.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.266+0000 7f13bcf66000 1 --2- 172.21.15.16:0/2821091294 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558df5c5f9b0 0x558df5cb3420 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x558df5ab7ae0 tx=0x7f13a4004740 comp rx=0 tx=0).stop 2024-03-31T15:07:07.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.266+0000 7f13bcf66000 1 -- 172.21.15.16:0/2821091294 shutdown_connections 2024-03-31T15:07:07.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.266+0000 7f13bcf66000 1 --2- 172.21.15.16:0/2821091294 >> [v2:172.21.15.16:6834/2613117528,v1:172.21.15.16:6835/2613117528] conn(0x558df5c65280 0x558df5c65660 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:07.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.266+0000 7f13bcf66000 1 --2- 172.21.15.16:0/2821091294 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x558df5cacbf0 0x558df5c644e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:07.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.266+0000 7f13bcf66000 1 --2- 172.21.15.16:0/2821091294 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558df5cac2d0 0x558df5cb3960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:07.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.266+0000 7f13bcf66000 1 --2- 172.21.15.16:0/2821091294 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558df5c5f9b0 0x558df5cb3420 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:07.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.266+0000 7f1397e00640 0 client.6376 ms_handle_reset on v2:172.21.15.16:6834/2613117528 2024-03-31T15:07:07.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.266+0000 7f13bcf66000 1 -- 172.21.15.16:0/2821091294 >> 172.21.15.16:0/2821091294 conn(0x558df5c8d070 msgr2=0x558df5c8f460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:07:07.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.266+0000 7f13bcf66000 1 -- 172.21.15.16:0/2821091294 shutdown_connections 2024-03-31T15:07:07.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.266+0000 7f13bcf66000 1 -- 172.21.15.16:0/2821091294 wait complete. 2024-03-31T15:07:07.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.266+0000 7f13bcf66000 20 client.6376 trim_cache size 0 max 16384 2024-03-31T15:07:07.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.282+0000 7f13bcf66000 1 Processor -- start 2024-03-31T15:07:07.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.286+0000 7f13bcf66000 1 -- start start 2024-03-31T15:07:07.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.286+0000 7f13bcf66000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558df5c5f9b0 0x558df5c5dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:07.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.286+0000 7f13bcf66000 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x558df5cac2d0 0x558df5c5e110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:07.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.286+0000 7f13bcf66000 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558df5cacbf0 0x558df5c5e910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:07.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.286+0000 7f13bcf66000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f13b4203960 con 0x558df5c5f9b0 2024-03-31T15:07:07.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.286+0000 7f13bcf66000 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f13b414eae0 con 0x558df5cacbf0 2024-03-31T15:07:07.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.286+0000 7f13bcf66000 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f13b4155a20 con 0x558df5cac2d0 2024-03-31T15:07:07.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.286+0000 7f13b2a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558df5c5f9b0 0x558df5c5dbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:07.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.286+0000 7f13b2a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558df5c5f9b0 0x558df5c5dbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57318/0 (socket says 172.21.15.16:57318) 2024-03-31T15:07:07.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.286+0000 7f13b2a00640 1 -- 172.21.15.16:0/195687735 learned_addr learned my addr 172.21.15.16:0/195687735 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:07:07.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.286+0000 7f13b3400640 1 --2- 172.21.15.16:0/195687735 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558df5cacbf0 0x558df5c5e910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:07.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.286+0000 7f13abe00640 1 --2- 172.21.15.16:0/195687735 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x558df5cac2d0 0x558df5c5e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:07.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.286+0000 7f13b2a00640 1 -- 172.21.15.16:0/195687735 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x558df5cac2d0 msgr2=0x558df5c5e110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:07:07.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.286+0000 7f13b2a00640 1 --2- 172.21.15.16:0/195687735 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x558df5cac2d0 0x558df5c5e110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:07.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.286+0000 7f13b2a00640 1 -- 172.21.15.16:0/195687735 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558df5cacbf0 msgr2=0x558df5c5e910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:07:07.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.286+0000 7f13b2a00640 1 --2- 172.21.15.16:0/195687735 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558df5cacbf0 0x558df5c5e910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:07.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.286+0000 7f13b2a00640 1 -- 172.21.15.16:0/195687735 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558df5b32140 con 0x558df5c5f9b0 2024-03-31T15:07:07.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.286+0000 7f13abe00640 1 --2- 172.21.15.16:0/195687735 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x558df5cac2d0 0x558df5c5e110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:07:07.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.286+0000 7f13b3400640 1 --2- 172.21.15.16:0/195687735 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558df5cacbf0 0x558df5c5e910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:07:07.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.286+0000 7f13b2a00640 1 --2- 172.21.15.16:0/195687735 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558df5c5f9b0 0x558df5c5dbd0 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f13a4010870 tx=0x7f13a402e070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4807acdf3faf65f1 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:07:07.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.286+0000 7f13a2a00640 1 -- 172.21.15.16:0/195687735 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13a4036e60 con 0x558df5c5f9b0 2024-03-31T15:07:07.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.286+0000 7f13a2a00640 1 -- 172.21.15.16:0/195687735 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f13a4046070 con 0x558df5c5f9b0 2024-03-31T15:07:07.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.286+0000 7f13a2a00640 1 -- 172.21.15.16:0/195687735 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13a4021930 con 0x558df5c5f9b0 2024-03-31T15:07:07.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.286+0000 7f13bcf66000 1 -- 172.21.15.16:0/195687735 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558df5c5f9b0 msgr2=0x558df5c5dbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:07.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.286+0000 7f13bcf66000 1 --2- 172.21.15.16:0/195687735 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558df5c5f9b0 0x558df5c5dbd0 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f13a4010870 tx=0x7f13a402e070 comp rx=0 tx=0).stop 2024-03-31T15:07:07.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.286+0000 7f13bcf66000 1 -- 172.21.15.16:0/195687735 shutdown_connections 2024-03-31T15:07:07.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.286+0000 7f13bcf66000 1 --2- 172.21.15.16:0/195687735 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558df5cacbf0 0x558df5c5e910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:07.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.286+0000 7f13bcf66000 1 --2- 172.21.15.16:0/195687735 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x558df5cac2d0 0x558df5c5e110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:07.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.286+0000 7f13bcf66000 1 --2- 172.21.15.16:0/195687735 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558df5c5f9b0 0x558df5c5dbd0 unknown :-1 s=CLOSED pgs=713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:07.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.286+0000 7f13bcf66000 1 -- 172.21.15.16:0/195687735 >> 172.21.15.16:0/195687735 conn(0x558df5c8d070 msgr2=0x558df5cc25b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:07:07.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.286+0000 7f13bcf66000 1 -- 172.21.15.16:0/195687735 shutdown_connections 2024-03-31T15:07:07.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.286+0000 7f13bcf66000 1 -- 172.21.15.16:0/195687735 wait complete. 2024-03-31T15:07:07.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.286+0000 7f13bcf66000 1 -- start start 2024-03-31T15:07:07.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.286+0000 7f13bcf66000 1 Processor -- start 2024-03-31T15:07:07.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.290+0000 7f13bcf66000 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x558df5c5f9b0 0x558df5d17b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:07.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.290+0000 7f13bcf66000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558df5cac2d0 0x558df5cab290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:07.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.290+0000 7f13bcf66000 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558df5cacbf0 0x558df5cab7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:07.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.290+0000 7f13bcf66000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x558df5a68750 con 0x558df5cac2d0 2024-03-31T15:07:07.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.290+0000 7f13bcf66000 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f13b414e490 con 0x558df5cacbf0 2024-03-31T15:07:07.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.290+0000 7f13bcf66000 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x558df5b80420 con 0x558df5c5f9b0 2024-03-31T15:07:07.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.290+0000 7f13bcf66000 10 client.0 _ll_register_callbacks cb 0x7f139c0096f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T15:07:07.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.290+0000 7f13abe00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558df5cac2d0 0x558df5cab290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:07.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.290+0000 7f13abe00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558df5cac2d0 0x558df5cab290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57322/0 (socket says 172.21.15.16:57322) 2024-03-31T15:07:07.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.290+0000 7f13abe00640 1 -- 172.21.15.16:0/3217441746 learned_addr learned my addr 172.21.15.16:0/3217441746 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:07:07.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.290+0000 7f13b2a00640 1 --2- 172.21.15.16:0/3217441746 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x558df5c5f9b0 0x558df5d17b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:07.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.290+0000 7f13b3400640 1 --2- 172.21.15.16:0/3217441746 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558df5cacbf0 0x558df5cab7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:07.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.290+0000 7f13abe00640 1 -- 172.21.15.16:0/3217441746 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x558df5c5f9b0 msgr2=0x558df5d17b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:07:07.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.290+0000 7f13abe00640 1 --2- 172.21.15.16:0/3217441746 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x558df5c5f9b0 0x558df5d17b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:07.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.290+0000 7f13abe00640 1 -- 172.21.15.16:0/3217441746 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558df5cacbf0 msgr2=0x558df5cab7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:07:07.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.290+0000 7f13abe00640 1 --2- 172.21.15.16:0/3217441746 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558df5cacbf0 0x558df5cab7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:07.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.290+0000 7f13abe00640 1 -- 172.21.15.16:0/3217441746 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558df5a401e0 con 0x558df5cac2d0 2024-03-31T15:07:07.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.290+0000 7f13abe00640 1 --2- 172.21.15.16:0/3217441746 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558df5cac2d0 0x558df5cab290 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f139c01a910 tx=0x7f139c0046d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d6c2e2564e2b1d49 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:07:07.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.290+0000 7f1397e00640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-31T15:07:07.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.290+0000 7f1397e00640 1 -- 172.21.15.16:0/3217441746 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f139c0b0040 con 0x558df5cac2d0 2024-03-31T15:07:07.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.290+0000 7f1397e00640 1 -- 172.21.15.16:0/3217441746 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(56..56 src has 1..56) ==== 5709+0+0 (secure 0 0 0) 0x7f139c004120 con 0x558df5cac2d0 2024-03-31T15:07:07.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.290+0000 7f13bcf66000 1 -- 172.21.15.16:0/3217441746 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f13a4046070 con 0x558df5cac2d0 2024-03-31T15:07:07.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.290+0000 7f1397e00640 1 -- 172.21.15.16:0/3217441746 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f139c004320 con 0x558df5cac2d0 2024-03-31T15:07:07.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.290+0000 7f13bcf66000 10 client.6459 fetch_fsmap learned FSMap version 35 2024-03-31T15:07:07.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.290+0000 7f13bcf66000 1 -- 172.21.15.16:0/3217441746 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=2+}) -- 0x558df5a457b0 con 0x558df5cac2d0 2024-03-31T15:07:07.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.290+0000 7f1397e00640 1 -- 172.21.15.16:0/3217441746 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f13a4046070 con 0x558df5cac2d0 2024-03-31T15:07:07.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.290+0000 7f1397e00640 1 -- 172.21.15.16:0/3217441746 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 35) ==== 3494+0+0 (secure 0 0 0) 0x7f139c0059a0 con 0x558df5cac2d0 2024-03-31T15:07:07.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.290+0000 7f13bcf66000 10 client.6459 fetch_fsmap finished waiting for FSMap version 35 2024-03-31T15:07:07.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.290+0000 7f13bcf66000 10 client.6459 resolve_mds: resolved 4:0 to role '4:0' aka daemon mds.c 2024-03-31T15:07:07.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.290+0000 7f13bcf66000 20 client.6459 populate_metadata read hostname 'smithi016' 2024-03-31T15:07:07.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.290+0000 7f13bcf66000 1 --2- 172.21.15.16:0/3217441746 >> [v2:172.21.15.16:6834/2613117528,v1:172.21.15.16:6835/2613117528] conn(0x558df5cb0030 0x558df5cb0440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:07.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.290+0000 7f13bcf66000 4 client.6459 mds_command: new command op to 6150 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-31T15:07:07.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.290+0000 7f13bcf66000 1 -- 172.21.15.16:0/3217441746 --> [v2:172.21.15.16:6834/2613117528,v1:172.21.15.16:6835/2613117528] -- command(tid 0: {"prefix": "scrub status"}) -- 0x558df5abf040 con 0x558df5cb0030 2024-03-31T15:07:07.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.290+0000 7f13b2a00640 1 --2- 172.21.15.16:0/3217441746 >> [v2:172.21.15.16:6834/2613117528,v1:172.21.15.16:6835/2613117528] conn(0x558df5cb0030 0x558df5cb0440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:07.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.290+0000 7f1397e00640 10 client.6459 ms_handle_connect on v2:172.21.15.16:6834/2613117528 2024-03-31T15:07:07.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.290+0000 7f13b2a00640 1 --2- 172.21.15.16:0/3217441746 >> [v2:172.21.15.16:6834/2613117528,v1:172.21.15.16:6835/2613117528] conn(0x558df5cb0030 0x558df5cb0440 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=48423310c34584fc server_cookie=4701cec373551e72 in_seq=0 out_seq=0 2024-03-31T15:07:07.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.290+0000 7f1397e00640 1 -- 172.21.15.16:0/3217441746 <== mds.0 v2:172.21.15.16:6834/2613117528 1 ==== command_reply(tid 0: 0 ) ==== 8+0+266 (crc 0 0 0) 0x558df5abf040 con 0x558df5cb0030 2024-03-31T15:07:07.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.290+0000 7f1397e00640 10 client.6459 handle_command_reply: tid=0 2024-03-31T15:07:07.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.290+0000 7f13bcf66000 1 client.6459 shutdown 2024-03-31T15:07:07.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.290+0000 7f13bcf66000 1 -- 172.21.15.16:0/3217441746 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558df5cac2d0 msgr2=0x558df5cab290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:07.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.290+0000 7f13bcf66000 1 --2- 172.21.15.16:0/3217441746 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558df5cac2d0 0x558df5cab290 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f139c01a910 tx=0x7f139c0046d0 comp rx=0 tx=0).stop 2024-03-31T15:07:07.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.290+0000 7f13bcf66000 1 -- 172.21.15.16:0/3217441746 shutdown_connections 2024-03-31T15:07:07.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.290+0000 7f13bcf66000 1 --2- 172.21.15.16:0/3217441746 >> [v2:172.21.15.16:6834/2613117528,v1:172.21.15.16:6835/2613117528] conn(0x558df5cb0030 0x558df5cb0440 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:07.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.290+0000 7f13bcf66000 1 --2- 172.21.15.16:0/3217441746 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558df5cacbf0 0x558df5cab7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:07.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.290+0000 7f13bcf66000 1 --2- 172.21.15.16:0/3217441746 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558df5cac2d0 0x558df5cab290 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:07.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.290+0000 7f1397e00640 0 client.6459 ms_handle_reset on v2:172.21.15.16:6834/2613117528 2024-03-31T15:07:07.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.290+0000 7f13bcf66000 1 --2- 172.21.15.16:0/3217441746 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x558df5c5f9b0 0x558df5d17b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:07.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.290+0000 7f13bcf66000 1 -- 172.21.15.16:0/3217441746 >> 172.21.15.16:0/3217441746 conn(0x558df5c8d070 msgr2=0x558df5c64060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:07:07.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.294+0000 7f13bcf66000 1 -- 172.21.15.16:0/3217441746 shutdown_connections 2024-03-31T15:07:07.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.294+0000 7f13bcf66000 1 -- 172.21.15.16:0/3217441746 wait complete. 2024-03-31T15:07:07.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.294+0000 7f13bcf66000 20 client.6459 trim_cache size 0 max 16384 2024-03-31T15:07:07.292 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-03-31T15:07:07.292 INFO:teuthology.orchestra.run.smithi016.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-31T15:07:07.292 INFO:teuthology.orchestra.run.smithi016.stdout: "scrubs": { 2024-03-31T15:07:07.293 INFO:teuthology.orchestra.run.smithi016.stdout: "de39ecf3-9b54-49b9-a421-967d384cc9f3": { 2024-03-31T15:07:07.293 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-03-31T15:07:07.293 INFO:teuthology.orchestra.run.smithi016.stdout: "tag": "de39ecf3-9b54-49b9-a421-967d384cc9f3", 2024-03-31T15:07:07.293 INFO:teuthology.orchestra.run.smithi016.stdout: "options": "recursive,repair,force" 2024-03-31T15:07:07.293 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-31T15:07:07.293 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-31T15:07:07.293 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-03-31T15:07:07.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.294+0000 7f13b9a00640 1 -- 172.21.15.16:0/1686426483 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f1390068380 msgr2=0x7f139006a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:07.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.294+0000 7f13b9a00640 1 --2- 172.21.15.16:0/1686426483 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f1390068380 0x7f139006a840 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f13a40079c0 tx=0x7f13a4008040 comp rx=0 tx=0).stop 2024-03-31T15:07:07.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.294+0000 7f13b9a00640 1 -- 172.21.15.16:0/1686426483 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f13b41567e0 msgr2=0x7f13b40b0040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:07.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.294+0000 7f13b9a00640 1 --2- 172.21.15.16:0/1686426483 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f13b41567e0 0x7f13b40b0040 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f13ac007e10 tx=0x7f13ac00d4a0 comp rx=0 tx=0).stop 2024-03-31T15:07:07.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.294+0000 7f13b9a00640 1 -- 172.21.15.16:0/1686426483 shutdown_connections 2024-03-31T15:07:07.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.294+0000 7f13b9a00640 1 --2- 172.21.15.16:0/1686426483 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f1390068380 0x7f139006a840 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:07.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.294+0000 7f13b9a00640 1 --2- 172.21.15.16:0/1686426483 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f13b41567e0 0x7f13b40b0040 unknown :-1 s=CLOSED pgs=711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:07.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.294+0000 7f13b9a00640 1 --2- 172.21.15.16:0/1686426483 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f13b40731c0 0x7f13b415abf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:07.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.294+0000 7f13b9a00640 1 --2- 172.21.15.16:0/1686426483 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f13b406da10 0x7f13b4153ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:07.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.294+0000 7f13b9a00640 1 -- 172.21.15.16:0/1686426483 >> 172.21.15.16:0/1686426483 conn(0x7f13b406ee40 msgr2=0x7f13b415fbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:07:07.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.294+0000 7f13b9a00640 1 -- 172.21.15.16:0/1686426483 shutdown_connections 2024-03-31T15:07:07.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:07.294+0000 7f13b9a00640 1 -- 172.21.15.16:0/1686426483 wait complete. 2024-03-31T15:07:07.311 INFO:tasks.cephfs.filesystem:scrub status for tag:de39ecf3-9b54-49b9-a421-967d384cc9f3 - {'path': '/', 'tag': 'de39ecf3-9b54-49b9-a421-967d384cc9f3', 'options': 'recursive,repair,force'} 2024-03-31T15:07:37.029 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-31T15:07:37.034 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-31T15:07:37.313 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:07:37.313 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.4:0 scrub status 2024-03-31T15:07:37.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.462+0000 7f70a0400640 1 Processor -- start 2024-03-31T15:07:37.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.462+0000 7f70a0400640 1 -- start start 2024-03-31T15:07:37.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.462+0000 7f70a0400640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f709806b620 0x7f709806ba00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:37.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.462+0000 7f70a0400640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f70980b60a0 0x7f709806fc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:37.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.462+0000 7f70a0400640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f709806bf40 con 0x7f70980d0c00 2024-03-31T15:07:37.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.462+0000 7f70a0400640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f709806f820 con 0x7f709806b620 2024-03-31T15:07:37.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.462+0000 7f70a0400640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f70980b4a40 con 0x7f70980b60a0 2024-03-31T15:07:37.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.462+0000 7f709da00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f709806b620 0x7f709806ba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:37.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.462+0000 7f709da00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f709806b620 0x7f709806ba00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47268/0 (socket says 172.21.15.16:47268) 2024-03-31T15:07:37.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.462+0000 7f709da00640 1 -- 172.21.15.16:0/3139657485 learned_addr learned my addr 172.21.15.16:0/3139657485 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:07:37.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.462+0000 7f709e400640 1 --1- 172.21.15.16:0/3139657485 >> v1:172.21.15.16:6789/0 conn(0x7f70980d0c00 0x7f7098070180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:53748/0 (socket says 172.21.15.16:53748) 2024-03-31T15:07:37.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.462+0000 7f709d000640 1 --2- 172.21.15.16:0/3139657485 >> v2:172.21.15.89:3300/0 conn(0x7f70980b60a0 0x7f709806fc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:37.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.462+0000 7f709da00640 1 -- 172.21.15.16:0/3139657485 >> v2:172.21.15.89:3300/0 conn(0x7f70980b60a0 msgr2=0x7f709806fc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:07:37.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.462+0000 7f709da00640 1 --2- 172.21.15.16:0/3139657485 >> v2:172.21.15.89:3300/0 conn(0x7f70980b60a0 0x7f709806fc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:37.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.462+0000 7f709da00640 1 -- 172.21.15.16:0/3139657485 >> v1:172.21.15.16:6789/0 conn(0x7f70980d0c00 legacy=0x7f7098070180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:37.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.462+0000 7f709da00640 1 -- 172.21.15.16:0/3139657485 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f709806f540 con 0x7f709806b620 2024-03-31T15:07:37.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.466+0000 7f709da00640 1 --2- 172.21.15.16:0/3139657485 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f709806b620 0x7f709806ba00 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f709804ffb0 tx=0x7f708c01c460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40abe5eaa858be38 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:07:37.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.466+0000 7f7097e00640 1 -- 172.21.15.16:0/3139657485 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f708c0b0070 con 0x7f709806b620 2024-03-31T15:07:37.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.466+0000 7f7097e00640 1 -- 172.21.15.16:0/3139657485 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f708c0ab600 con 0x7f709806b620 2024-03-31T15:07:37.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.466+0000 7f7097e00640 1 -- 172.21.15.16:0/3139657485 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f708c0ab9d0 con 0x7f709806b620 2024-03-31T15:07:37.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.466+0000 7f70a0400640 1 -- 172.21.15.16:0/3139657485 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f709806b620 msgr2=0x7f709806ba00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:37.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.466+0000 7f70a0400640 1 --2- 172.21.15.16:0/3139657485 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f709806b620 0x7f709806ba00 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f709804ffb0 tx=0x7f708c01c460 comp rx=0 tx=0).stop 2024-03-31T15:07:37.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.466+0000 7f70a0400640 1 -- 172.21.15.16:0/3139657485 shutdown_connections 2024-03-31T15:07:37.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.466+0000 7f70a0400640 1 --2- 172.21.15.16:0/3139657485 >> v2:172.21.15.89:3300/0 conn(0x7f70980b60a0 0x7f709806fc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:37.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.466+0000 7f70a0400640 1 --2- 172.21.15.16:0/3139657485 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f709806b620 0x7f709806ba00 unknown :-1 s=CLOSED pgs=716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:37.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.466+0000 7f70a0400640 1 -- 172.21.15.16:0/3139657485 >> 172.21.15.16:0/3139657485 conn(0x7f709806e670 msgr2=0x7f709806ea50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:07:37.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.466+0000 7f70a0400640 1 -- 172.21.15.16:0/3139657485 shutdown_connections 2024-03-31T15:07:37.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.466+0000 7f70a0400640 1 -- 172.21.15.16:0/3139657485 wait complete. 2024-03-31T15:07:37.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.470+0000 7f70a0400640 1 Processor -- start 2024-03-31T15:07:37.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.470+0000 7f70a0400640 1 -- start start 2024-03-31T15:07:37.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.470+0000 7f70a0400640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f709806b620 0x7f7098141ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:37.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.470+0000 7f70a0400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f70980b60a0 0x7f7098152740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:37.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.470+0000 7f70a0400640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f70980d0c00 0x7f7098152c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:37.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.470+0000 7f70a0400640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f709815e8d0 con 0x7f70980b60a0 2024-03-31T15:07:37.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.470+0000 7f70a0400640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f709815e750 con 0x7f70980d0c00 2024-03-31T15:07:37.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.470+0000 7f70a0400640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f709806f160 con 0x7f709806b620 2024-03-31T15:07:37.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.470+0000 7f709d000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f70980b60a0 0x7f7098152740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:37.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.470+0000 7f709d000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f70980b60a0 0x7f7098152740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47270/0 (socket says 172.21.15.16:47270) 2024-03-31T15:07:37.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.470+0000 7f709d000640 1 -- 172.21.15.16:0/4250770970 learned_addr learned my addr 172.21.15.16:0/4250770970 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:07:37.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.470+0000 7f709e400640 1 --2- 172.21.15.16:0/4250770970 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f70980d0c00 0x7f7098152c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:37.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.470+0000 7f709da00640 1 --2- 172.21.15.16:0/4250770970 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f709806b620 0x7f7098141ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:37.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.470+0000 7f709d000640 1 -- 172.21.15.16:0/4250770970 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f709806b620 msgr2=0x7f7098141ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:37.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.470+0000 7f709d000640 1 --2- 172.21.15.16:0/4250770970 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f709806b620 0x7f7098141ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:37.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.470+0000 7f709d000640 1 -- 172.21.15.16:0/4250770970 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f70980d0c00 msgr2=0x7f7098152c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:37.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.470+0000 7f709d000640 1 --2- 172.21.15.16:0/4250770970 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f70980d0c00 0x7f7098152c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:37.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.470+0000 7f709d000640 1 -- 172.21.15.16:0/4250770970 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f709815f8f0 con 0x7f70980b60a0 2024-03-31T15:07:37.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.470+0000 7f709d000640 1 --2- 172.21.15.16:0/4250770970 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f70980b60a0 0x7f7098152740 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f708400e870 tx=0x7f708400ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:07:37.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.470+0000 7f7095600640 1 -- 172.21.15.16:0/4250770970 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7084017070 con 0x7f70980b60a0 2024-03-31T15:07:37.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.470+0000 7f7095600640 1 -- 172.21.15.16:0/4250770970 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f708400c940 con 0x7f70980b60a0 2024-03-31T15:07:37.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.470+0000 7f70a0400640 1 -- 172.21.15.16:0/4250770970 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f709815fbd0 con 0x7f70980b60a0 2024-03-31T15:07:37.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.470+0000 7f7095600640 1 -- 172.21.15.16:0/4250770970 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f708400ccf0 con 0x7f70980b60a0 2024-03-31T15:07:37.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.470+0000 7f70a0400640 1 -- 172.21.15.16:0/4250770970 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f709815feb0 con 0x7f70980b60a0 2024-03-31T15:07:37.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.474+0000 7f7095600640 1 -- 172.21.15.16:0/4250770970 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f708401a070 con 0x7f70980b60a0 2024-03-31T15:07:37.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.474+0000 7f7095600640 1 --2- 172.21.15.16:0/4250770970 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f7078068380 0x7f707806a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:37.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.474+0000 7f7095600640 1 -- 172.21.15.16:0/4250770970 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 5709+0+0 (secure 0 0 0) 0x7f7084086ba0 con 0x7f70980b60a0 2024-03-31T15:07:37.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.474+0000 7f70a3944000 1 Processor -- start 2024-03-31T15:07:37.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.474+0000 7f70a3944000 1 -- start start 2024-03-31T15:07:37.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.474+0000 7f709da00640 1 --2- 172.21.15.16:0/4250770970 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f7078068380 0x7f707806a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:37.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.474+0000 7f70a3944000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5587ed9a62d0 0x5587ed9a66b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:37.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.474+0000 7f70a3944000 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5587ed9a6bf0 0x5587ed959470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:37.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.474+0000 7f70a3944000 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5587ed9599b0 0x5587ed959d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:37.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.474+0000 7f70a3944000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5587ed842af0 con 0x5587ed9a62d0 2024-03-31T15:07:37.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.474+0000 7f70a3944000 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x5587ed7abb00 con 0x5587ed9a6bf0 2024-03-31T15:07:37.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.474+0000 7f70a3944000 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x5587ed748940 con 0x5587ed9599b0 2024-03-31T15:07:37.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.478+0000 7f709da00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5587ed9a62d0 0x5587ed9a66b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:37.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.478+0000 7f709da00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5587ed9a62d0 0x5587ed9a66b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47278/0 (socket says 172.21.15.16:47278) 2024-03-31T15:07:37.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.478+0000 7f709da00640 1 -- 172.21.15.16:0/3045105325 learned_addr learned my addr 172.21.15.16:0/3045105325 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:07:37.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.478+0000 7f709da00640 1 --2- 172.21.15.16:0/4250770970 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f7078068380 0x7f707806a840 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f708c0ad000 tx=0x7f708c0a6000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:07:37.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.478+0000 7f709e400640 1 --2- 172.21.15.16:0/3045105325 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5587ed9599b0 0x5587ed959d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:37.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.478+0000 7f709d000640 1 --2- 172.21.15.16:0/3045105325 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5587ed9a6bf0 0x5587ed959470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:37.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.478+0000 7f709da00640 1 -- 172.21.15.16:0/3045105325 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5587ed9599b0 msgr2=0x5587ed959d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:07:37.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.478+0000 7f709da00640 1 --2- 172.21.15.16:0/3045105325 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5587ed9599b0 0x5587ed959d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:37.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.478+0000 7f709da00640 1 -- 172.21.15.16:0/3045105325 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5587ed9a6bf0 msgr2=0x5587ed959470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:07:37.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.478+0000 7f709da00640 1 --2- 172.21.15.16:0/3045105325 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5587ed9a6bf0 0x5587ed959470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:37.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.478+0000 7f709da00640 1 -- 172.21.15.16:0/3045105325 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5587ed731380 con 0x5587ed9a62d0 2024-03-31T15:07:37.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.478+0000 7f709da00640 1 --2- 172.21.15.16:0/3045105325 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5587ed9a62d0 0x5587ed9a66b0 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f708c01c940 tx=0x7f708c009240 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a9f75174b033c41 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:07:37.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.478+0000 7f708a000640 1 -- 172.21.15.16:0/3045105325 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f708c0b0040 con 0x5587ed9a62d0 2024-03-31T15:07:37.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.478+0000 7f708a000640 1 -- 172.21.15.16:0/3045105325 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f708c0ab8c0 con 0x5587ed9a62d0 2024-03-31T15:07:37.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.478+0000 7f708a000640 1 -- 172.21.15.16:0/3045105325 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f708c0abc70 con 0x5587ed9a62d0 2024-03-31T15:07:37.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.478+0000 7f70a3944000 1 -- 172.21.15.16:0/3045105325 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5587ed9a62d0 msgr2=0x5587ed9a66b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:37.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.478+0000 7f70a3944000 1 --2- 172.21.15.16:0/3045105325 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5587ed9a62d0 0x5587ed9a66b0 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f708c01c940 tx=0x7f708c009240 comp rx=0 tx=0).stop 2024-03-31T15:07:37.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.478+0000 7f70a3944000 1 -- 172.21.15.16:0/3045105325 shutdown_connections 2024-03-31T15:07:37.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.478+0000 7f70a3944000 1 --2- 172.21.15.16:0/3045105325 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5587ed9599b0 0x5587ed959d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:37.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.478+0000 7f70a3944000 1 --2- 172.21.15.16:0/3045105325 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5587ed9a6bf0 0x5587ed959470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:37.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.478+0000 7f70a3944000 1 --2- 172.21.15.16:0/3045105325 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5587ed9a62d0 0x5587ed9a66b0 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:37.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.478+0000 7f70a3944000 1 -- 172.21.15.16:0/3045105325 >> 172.21.15.16:0/3045105325 conn(0x5587ed987070 msgr2=0x5587ed989460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:07:37.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.478+0000 7f70a3944000 1 -- 172.21.15.16:0/3045105325 shutdown_connections 2024-03-31T15:07:37.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.478+0000 7f70a3944000 1 -- 172.21.15.16:0/3045105325 wait complete. 2024-03-31T15:07:37.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.478+0000 7f70a3944000 1 -- start start 2024-03-31T15:07:37.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.478+0000 7f70a3944000 1 Processor -- start 2024-03-31T15:07:37.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.478+0000 7f70a3944000 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5587ed9599b0 0x5587ed95e950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:37.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.478+0000 7f70a3944000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5587ed9a62d0 0x5587ed95ee90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:37.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.478+0000 7f70a3944000 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5587ed9a6bf0 0x5587ed95f3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:37.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.478+0000 7f70a3944000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5587ed7691e0 con 0x5587ed9a62d0 2024-03-31T15:07:37.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.478+0000 7f70a3944000 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x5587ed7a7ab0 con 0x5587ed9599b0 2024-03-31T15:07:37.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.478+0000 7f70a3944000 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x5587ed762750 con 0x5587ed9a6bf0 2024-03-31T15:07:37.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.478+0000 7f70a3944000 10 client.0 _ll_register_callbacks cb 0x5587ed947370 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T15:07:37.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.482+0000 7f709d000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5587ed9a62d0 0x5587ed95ee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:37.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.482+0000 7f709e400640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5587ed9a6bf0 0x5587ed95f3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:37.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.482+0000 7f709da00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5587ed9599b0 0x5587ed95e950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:37.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.482+0000 7f709e400640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5587ed9a6bf0 0x5587ed95f3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3301/0 says I am v2:172.21.15.16:54820/0 (socket says 172.21.15.16:54820) 2024-03-31T15:07:37.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.482+0000 7f709e400640 1 -- 172.21.15.16:0/1782029752 learned_addr learned my addr 172.21.15.16:0/1782029752 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:07:37.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.482+0000 7f709d000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5587ed9a62d0 0x5587ed95ee90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47294/0 (socket says 172.21.15.16:47294) 2024-03-31T15:07:37.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.482+0000 7f709e400640 1 -- 172.21.15.16:0/1782029752 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5587ed9599b0 msgr2=0x5587ed95e950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:07:37.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.482+0000 7f709e400640 1 --2- 172.21.15.16:0/1782029752 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5587ed9599b0 0x5587ed95e950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:37.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.482+0000 7f709e400640 1 -- 172.21.15.16:0/1782029752 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5587ed9a62d0 msgr2=0x5587ed95ee90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:07:37.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.482+0000 7f709e400640 1 --2- 172.21.15.16:0/1782029752 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5587ed9a62d0 0x5587ed95ee90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:37.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.482+0000 7f709e400640 1 -- 172.21.15.16:0/1782029752 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({osdmap=0}) -- 0x5587ed767b70 con 0x5587ed9a6bf0 2024-03-31T15:07:37.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.482+0000 7f709d000640 1 --2- 172.21.15.16:0/1782029752 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5587ed9a62d0 0x5587ed95ee90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:07:37.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.482+0000 7f709e400640 1 --2- 172.21.15.16:0/1782029752 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5587ed9a6bf0 0x5587ed95f3d0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f7090004fd0 tx=0x7f70900074f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=95a36a7660b9ceca server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:07:37.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.482+0000 7f707ea00640 10 client.0 ms_handle_connect on v2:172.21.15.89:3301/0 2024-03-31T15:07:37.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.482+0000 7f707ea00640 1 -- 172.21.15.16:0/1782029752 <== mon.2 v2:172.21.15.89:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7090013600 con 0x5587ed9a6bf0 2024-03-31T15:07:37.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.482+0000 7f707ea00640 1 -- 172.21.15.16:0/1782029752 <== mon.2 v2:172.21.15.89:3301/0 2 ==== osd_map(56..56 src has 1..56) ==== 5709+0+0 (secure 0 0 0) 0x7f70900137a0 con 0x5587ed9a6bf0 2024-03-31T15:07:37.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.482+0000 7f70a3944000 1 -- 172.21.15.16:0/1782029752 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f708c0ab8c0 con 0x5587ed9a6bf0 2024-03-31T15:07:37.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.482+0000 7f707ea00640 1 -- 172.21.15.16:0/1782029752 <== mon.2 v2:172.21.15.89:3301/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f70900139e0 con 0x5587ed9a6bf0 2024-03-31T15:07:37.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.482+0000 7f70a3944000 10 client.5798 fetch_fsmap learned FSMap version 36 2024-03-31T15:07:37.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.482+0000 7f70a3944000 1 -- 172.21.15.16:0/1782029752 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({config=0+,fsmap=36,monmap=2+}) -- 0x5587ed748b90 con 0x5587ed9a6bf0 2024-03-31T15:07:37.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.482+0000 7f707ea00640 1 -- 172.21.15.16:0/1782029752 <== mon.2 v2:172.21.15.89:3301/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f708c0ab8c0 con 0x5587ed9a6bf0 2024-03-31T15:07:37.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.482+0000 7f707ea00640 1 -- 172.21.15.16:0/1782029752 <== mon.2 v2:172.21.15.89:3301/0 5 ==== fsmap(e 36) ==== 3494+0+0 (secure 0 0 0) 0x7f7090013b80 con 0x5587ed9a6bf0 2024-03-31T15:07:37.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.482+0000 7f70a3944000 10 client.5798 fetch_fsmap finished waiting for FSMap version 36 2024-03-31T15:07:37.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.482+0000 7f70a3944000 10 client.5798 resolve_mds: resolved 4:0 to role '4:0' aka daemon mds.c 2024-03-31T15:07:37.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.482+0000 7f70a3944000 20 client.5798 populate_metadata read hostname 'smithi016' 2024-03-31T15:07:37.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.482+0000 7f70a3944000 1 --2- 172.21.15.16:0/1782029752 >> [v2:172.21.15.16:6834/2613117528,v1:172.21.15.16:6835/2613117528] conn(0x5587ed95fc70 0x5587ed9b7a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:37.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.482+0000 7f70a3944000 4 client.5798 mds_command: new command op to 6150 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-31T15:07:37.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.482+0000 7f70a3944000 1 -- 172.21.15.16:0/1782029752 --> [v2:172.21.15.16:6834/2613117528,v1:172.21.15.16:6835/2613117528] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5587ed746dd0 con 0x5587ed95fc70 2024-03-31T15:07:37.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.482+0000 7f709da00640 1 --2- 172.21.15.16:0/1782029752 >> [v2:172.21.15.16:6834/2613117528,v1:172.21.15.16:6835/2613117528] conn(0x5587ed95fc70 0x5587ed9b7a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:37.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.482+0000 7f707ea00640 10 client.5798 ms_handle_connect on v2:172.21.15.16:6834/2613117528 2024-03-31T15:07:37.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.482+0000 7f709da00640 1 --2- 172.21.15.16:0/1782029752 >> [v2:172.21.15.16:6834/2613117528,v1:172.21.15.16:6835/2613117528] conn(0x5587ed95fc70 0x5587ed9b7a50 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7aae39541db49603 server_cookie=495c391f65e4f374 in_seq=0 out_seq=0 2024-03-31T15:07:37.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.482+0000 7f707ea00640 1 -- 172.21.15.16:0/1782029752 <== mds.0 v2:172.21.15.16:6834/2613117528 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5587ed746dd0 con 0x5587ed95fc70 2024-03-31T15:07:37.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.482+0000 7f707ea00640 10 client.5798 handle_command_reply: tid=0 2024-03-31T15:07:37.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.482+0000 7f70a3944000 1 client.5798 shutdown 2024-03-31T15:07:37.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.482+0000 7f70a3944000 1 -- 172.21.15.16:0/1782029752 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5587ed9a6bf0 msgr2=0x5587ed95f3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:37.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.482+0000 7f70a3944000 1 --2- 172.21.15.16:0/1782029752 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5587ed9a6bf0 0x5587ed95f3d0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f7090004fd0 tx=0x7f70900074f0 comp rx=0 tx=0).stop 2024-03-31T15:07:37.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.482+0000 7f70a3944000 1 -- 172.21.15.16:0/1782029752 shutdown_connections 2024-03-31T15:07:37.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.482+0000 7f70a3944000 1 --2- 172.21.15.16:0/1782029752 >> [v2:172.21.15.16:6834/2613117528,v1:172.21.15.16:6835/2613117528] conn(0x5587ed95fc70 0x5587ed9b7a50 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:37.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.482+0000 7f70a3944000 1 --2- 172.21.15.16:0/1782029752 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5587ed9a6bf0 0x5587ed95f3d0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:37.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.482+0000 7f70a3944000 1 --2- 172.21.15.16:0/1782029752 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5587ed9a62d0 0x5587ed95ee90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:37.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.482+0000 7f707ea00640 0 client.5798 ms_handle_reset on v2:172.21.15.16:6834/2613117528 2024-03-31T15:07:37.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.482+0000 7f70a3944000 1 --2- 172.21.15.16:0/1782029752 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5587ed9599b0 0x5587ed95e950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:37.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.482+0000 7f70a3944000 1 -- 172.21.15.16:0/1782029752 >> 172.21.15.16:0/1782029752 conn(0x5587ed987070 msgr2=0x5587ed989460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:07:37.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.482+0000 7f70a3944000 1 -- 172.21.15.16:0/1782029752 shutdown_connections 2024-03-31T15:07:37.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.482+0000 7f70a3944000 1 -- 172.21.15.16:0/1782029752 wait complete. 2024-03-31T15:07:37.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.486+0000 7f70a3944000 20 client.5798 trim_cache size 0 max 16384 2024-03-31T15:07:37.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.502+0000 7f70a3944000 1 Processor -- start 2024-03-31T15:07:37.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.502+0000 7f70a3944000 1 -- start start 2024-03-31T15:07:37.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.502+0000 7f70a3944000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5587ed9599b0 0x5587ed9a59c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:37.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.502+0000 7f70a3944000 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5587ed9a62d0 0x5587ed9a4380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:37.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.502+0000 7f70a3944000 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5587ed9a6bf0 0x5587ed9a48c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:37.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.502+0000 7f70a3944000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f7098203c90 con 0x5587ed9599b0 2024-03-31T15:07:37.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.502+0000 7f70a3944000 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f7098202cd0 con 0x5587ed9a62d0 2024-03-31T15:07:37.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.502+0000 7f70a3944000 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f7098202130 con 0x5587ed9a6bf0 2024-03-31T15:07:37.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.502+0000 7f709da00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5587ed9599b0 0x5587ed9a59c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:37.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.502+0000 7f709da00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5587ed9599b0 0x5587ed9a59c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47306/0 (socket says 172.21.15.16:47306) 2024-03-31T15:07:37.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.502+0000 7f709da00640 1 -- 172.21.15.16:0/832070952 learned_addr learned my addr 172.21.15.16:0/832070952 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:07:37.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.502+0000 7f709e400640 1 --2- 172.21.15.16:0/832070952 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5587ed9a6bf0 0x5587ed9a48c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:37.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.502+0000 7f709d000640 1 --2- 172.21.15.16:0/832070952 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5587ed9a62d0 0x5587ed9a4380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:37.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.502+0000 7f709da00640 1 -- 172.21.15.16:0/832070952 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5587ed9a6bf0 msgr2=0x5587ed9a48c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:07:37.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.502+0000 7f709da00640 1 --2- 172.21.15.16:0/832070952 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5587ed9a6bf0 0x5587ed9a48c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:37.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.502+0000 7f709da00640 1 -- 172.21.15.16:0/832070952 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5587ed9a62d0 msgr2=0x5587ed9a4380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:07:37.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.502+0000 7f709da00640 1 --2- 172.21.15.16:0/832070952 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5587ed9a62d0 0x5587ed9a4380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:37.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.502+0000 7f709da00640 1 -- 172.21.15.16:0/832070952 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5587ed82c140 con 0x5587ed9599b0 2024-03-31T15:07:37.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.502+0000 7f709da00640 1 --2- 172.21.15.16:0/832070952 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5587ed9599b0 0x5587ed9a59c0 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f708c01c940 tx=0x7f708c004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ebd79056fb300233 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:07:37.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.502+0000 7f7089600640 1 -- 172.21.15.16:0/832070952 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f708c0b0040 con 0x5587ed9599b0 2024-03-31T15:07:37.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.502+0000 7f7089600640 1 -- 172.21.15.16:0/832070952 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f708c0abd70 con 0x5587ed9599b0 2024-03-31T15:07:37.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.502+0000 7f7089600640 1 -- 172.21.15.16:0/832070952 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f708c004240 con 0x5587ed9599b0 2024-03-31T15:07:37.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.506+0000 7f70a3944000 1 -- 172.21.15.16:0/832070952 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5587ed9599b0 msgr2=0x5587ed9a59c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:37.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.506+0000 7f70a3944000 1 --2- 172.21.15.16:0/832070952 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5587ed9599b0 0x5587ed9a59c0 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f708c01c940 tx=0x7f708c004740 comp rx=0 tx=0).stop 2024-03-31T15:07:37.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.506+0000 7f70a3944000 1 -- 172.21.15.16:0/832070952 shutdown_connections 2024-03-31T15:07:37.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.506+0000 7f70a3944000 1 --2- 172.21.15.16:0/832070952 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5587ed9a6bf0 0x5587ed9a48c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:37.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.506+0000 7f70a3944000 1 --2- 172.21.15.16:0/832070952 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5587ed9a62d0 0x5587ed9a4380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:37.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.506+0000 7f70a3944000 1 --2- 172.21.15.16:0/832070952 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5587ed9599b0 0x5587ed9a59c0 unknown :-1 s=CLOSED pgs=719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:37.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.506+0000 7f70a3944000 1 -- 172.21.15.16:0/832070952 >> 172.21.15.16:0/832070952 conn(0x5587ed987070 msgr2=0x5587ed95d9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:07:37.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.506+0000 7f70a3944000 1 -- 172.21.15.16:0/832070952 shutdown_connections 2024-03-31T15:07:37.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.506+0000 7f70a3944000 1 -- 172.21.15.16:0/832070952 wait complete. 2024-03-31T15:07:37.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.506+0000 7f70a3944000 1 -- start start 2024-03-31T15:07:37.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.506+0000 7f70a3944000 1 Processor -- start 2024-03-31T15:07:37.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.506+0000 7f70a3944000 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5587ed9599b0 0x5587ed95dc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:37.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.506+0000 7f70a3944000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5587ed9a62d0 0x5587ed9b8aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:37.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.506+0000 7f70a3944000 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5587ed9a6bf0 0x5587ed9b8fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:37.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.506+0000 7f70a3944000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5587ed746940 con 0x5587ed9a62d0 2024-03-31T15:07:37.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.506+0000 7f70a3944000 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f7098202680 con 0x5587ed9599b0 2024-03-31T15:07:37.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.506+0000 7f70a3944000 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x5587ed87a420 con 0x5587ed9a6bf0 2024-03-31T15:07:37.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.506+0000 7f70a3944000 10 client.0 _ll_register_callbacks cb 0x7f7090004d10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T15:07:37.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.506+0000 7f709d000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5587ed9a62d0 0x5587ed9b8aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:37.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.506+0000 7f709da00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5587ed9599b0 0x5587ed95dc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:37.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.506+0000 7f709d000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5587ed9a62d0 0x5587ed9b8aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47318/0 (socket says 172.21.15.16:47318) 2024-03-31T15:07:37.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.506+0000 7f709d000640 1 -- 172.21.15.16:0/2569707969 learned_addr learned my addr 172.21.15.16:0/2569707969 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:07:37.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.506+0000 7f709e400640 1 --2- 172.21.15.16:0/2569707969 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5587ed9a6bf0 0x5587ed9b8fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:37.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.506+0000 7f709d000640 1 -- 172.21.15.16:0/2569707969 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5587ed9a6bf0 msgr2=0x5587ed9b8fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:07:37.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.506+0000 7f709d000640 1 --2- 172.21.15.16:0/2569707969 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5587ed9a6bf0 0x5587ed9b8fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:37.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.506+0000 7f709d000640 1 -- 172.21.15.16:0/2569707969 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5587ed9599b0 msgr2=0x5587ed95dc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:07:37.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.506+0000 7f709d000640 1 --2- 172.21.15.16:0/2569707969 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5587ed9599b0 0x5587ed95dc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:37.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.506+0000 7f709d000640 1 -- 172.21.15.16:0/2569707969 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5587ed73a1e0 con 0x5587ed9a62d0 2024-03-31T15:07:37.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.506+0000 7f709d000640 1 --2- 172.21.15.16:0/2569707969 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5587ed9a62d0 0x5587ed9b8aa0 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f708400c520 tx=0x7f7084056520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1333518e4f91d40a server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:07:37.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.506+0000 7f707ea00640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-31T15:07:37.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.506+0000 7f707ea00640 1 -- 172.21.15.16:0/2569707969 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7084082070 con 0x5587ed9a62d0 2024-03-31T15:07:37.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.506+0000 7f707ea00640 1 -- 172.21.15.16:0/2569707969 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(56..56 src has 1..56) ==== 5709+0+0 (secure 0 0 0) 0x7f7084056150 con 0x5587ed9a62d0 2024-03-31T15:07:37.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.506+0000 7f70a3944000 1 -- 172.21.15.16:0/2569707969 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f708c0abd70 con 0x5587ed9a62d0 2024-03-31T15:07:37.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.506+0000 7f707ea00640 1 -- 172.21.15.16:0/2569707969 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f7084021040 con 0x5587ed9a62d0 2024-03-31T15:07:37.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.506+0000 7f70a3944000 10 client.6480 fetch_fsmap learned FSMap version 36 2024-03-31T15:07:37.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.506+0000 7f70a3944000 1 -- 172.21.15.16:0/2569707969 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=2+}) -- 0x5587ed73f7b0 con 0x5587ed9a62d0 2024-03-31T15:07:37.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.506+0000 7f707ea00640 1 -- 172.21.15.16:0/2569707969 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f708c0abd70 con 0x5587ed9a62d0 2024-03-31T15:07:37.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.506+0000 7f707ea00640 1 -- 172.21.15.16:0/2569707969 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 36) ==== 3494+0+0 (secure 0 0 0) 0x7f7084052850 con 0x5587ed9a62d0 2024-03-31T15:07:37.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.506+0000 7f70a3944000 10 client.6480 fetch_fsmap finished waiting for FSMap version 36 2024-03-31T15:07:37.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.506+0000 7f70a3944000 10 client.6480 resolve_mds: resolved 4:0 to role '4:0' aka daemon mds.c 2024-03-31T15:07:37.507 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.506+0000 7f70a3944000 20 client.6480 populate_metadata read hostname 'smithi016' 2024-03-31T15:07:37.507 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.506+0000 7f70a3944000 1 --2- 172.21.15.16:0/2569707969 >> [v2:172.21.15.16:6834/2613117528,v1:172.21.15.16:6835/2613117528] conn(0x5587ed9b9520 0x5587eda15a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:37.507 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.506+0000 7f70a3944000 4 client.6480 mds_command: new command op to 6150 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-31T15:07:37.507 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.506+0000 7f70a3944000 1 -- 172.21.15.16:0/2569707969 --> [v2:172.21.15.16:6834/2613117528,v1:172.21.15.16:6835/2613117528] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5587ed7b9040 con 0x5587ed9b9520 2024-03-31T15:07:37.507 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.510+0000 7f709da00640 1 --2- 172.21.15.16:0/2569707969 >> [v2:172.21.15.16:6834/2613117528,v1:172.21.15.16:6835/2613117528] conn(0x5587ed9b9520 0x5587eda15a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:37.507 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.510+0000 7f707ea00640 10 client.6480 ms_handle_connect on v2:172.21.15.16:6834/2613117528 2024-03-31T15:07:37.507 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.510+0000 7f709da00640 1 --2- 172.21.15.16:0/2569707969 >> [v2:172.21.15.16:6834/2613117528,v1:172.21.15.16:6835/2613117528] conn(0x5587ed9b9520 0x5587eda15a50 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4526560c9b1e92a4 server_cookie=43efdf26e28880e9 in_seq=0 out_seq=0 2024-03-31T15:07:37.508 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.510+0000 7f707ea00640 1 -- 172.21.15.16:0/2569707969 <== mds.0 v2:172.21.15.16:6834/2613117528 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x5587ed7b9040 con 0x5587ed9b9520 2024-03-31T15:07:37.508 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.510+0000 7f707ea00640 10 client.6480 handle_command_reply: tid=0 2024-03-31T15:07:37.508 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.510+0000 7f70a3944000 1 client.6480 shutdown 2024-03-31T15:07:37.508 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.510+0000 7f70a3944000 1 -- 172.21.15.16:0/2569707969 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5587ed9a62d0 msgr2=0x5587ed9b8aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:37.508 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.510+0000 7f70a3944000 1 --2- 172.21.15.16:0/2569707969 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5587ed9a62d0 0x5587ed9b8aa0 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f708400c520 tx=0x7f7084056520 comp rx=0 tx=0).stop 2024-03-31T15:07:37.508 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.510+0000 7f70a3944000 1 -- 172.21.15.16:0/2569707969 shutdown_connections 2024-03-31T15:07:37.508 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.510+0000 7f70a3944000 1 --2- 172.21.15.16:0/2569707969 >> [v2:172.21.15.16:6834/2613117528,v1:172.21.15.16:6835/2613117528] conn(0x5587ed9b9520 0x5587eda15a50 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:37.508 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.510+0000 7f70a3944000 1 --2- 172.21.15.16:0/2569707969 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5587ed9a6bf0 0x5587ed9b8fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:37.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.510+0000 7f70a3944000 1 --2- 172.21.15.16:0/2569707969 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5587ed9a62d0 0x5587ed9b8aa0 unknown :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:37.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.510+0000 7f70a3944000 1 --2- 172.21.15.16:0/2569707969 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5587ed9599b0 0x5587ed95dc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:37.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.510+0000 7f707ea00640 0 client.6480 ms_handle_reset on v2:172.21.15.16:6834/2613117528 2024-03-31T15:07:37.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.510+0000 7f70a3944000 1 -- 172.21.15.16:0/2569707969 >> 172.21.15.16:0/2569707969 conn(0x5587ed987070 msgr2=0x5587ed95d120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:07:37.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.510+0000 7f70a3944000 1 -- 172.21.15.16:0/2569707969 shutdown_connections 2024-03-31T15:07:37.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.510+0000 7f70a3944000 1 -- 172.21.15.16:0/2569707969 wait complete. 2024-03-31T15:07:37.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.510+0000 7f70a3944000 20 client.6480 trim_cache size 0 max 16384 2024-03-31T15:07:37.509 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-03-31T15:07:37.509 INFO:teuthology.orchestra.run.smithi016.stdout: "status": "no active scrubs running", 2024-03-31T15:07:37.509 INFO:teuthology.orchestra.run.smithi016.stdout: "scrubs": {} 2024-03-31T15:07:37.510 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-03-31T15:07:37.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.510+0000 7f70a0400640 1 -- 172.21.15.16:0/4250770970 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f7078068380 msgr2=0x7f707806a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:37.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.510+0000 7f70a0400640 1 --2- 172.21.15.16:0/4250770970 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f7078068380 0x7f707806a840 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f708c0ad000 tx=0x7f708c0a6000 comp rx=0 tx=0).stop 2024-03-31T15:07:37.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.510+0000 7f70a0400640 1 -- 172.21.15.16:0/4250770970 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f70980b60a0 msgr2=0x7f7098152740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:37.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.510+0000 7f70a0400640 1 --2- 172.21.15.16:0/4250770970 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f70980b60a0 0x7f7098152740 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f708400e870 tx=0x7f708400ec70 comp rx=0 tx=0).stop 2024-03-31T15:07:37.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.510+0000 7f70a0400640 1 -- 172.21.15.16:0/4250770970 shutdown_connections 2024-03-31T15:07:37.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.510+0000 7f70a0400640 1 --2- 172.21.15.16:0/4250770970 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f7078068380 0x7f707806a840 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:37.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.510+0000 7f70a0400640 1 --2- 172.21.15.16:0/4250770970 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f70980d0c00 0x7f7098152c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:37.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.510+0000 7f70a0400640 1 --2- 172.21.15.16:0/4250770970 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f70980b60a0 0x7f7098152740 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:37.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.510+0000 7f70a0400640 1 --2- 172.21.15.16:0/4250770970 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f709806b620 0x7f7098141ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:37.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.510+0000 7f70a0400640 1 -- 172.21.15.16:0/4250770970 >> 172.21.15.16:0/4250770970 conn(0x7f709806e670 msgr2=0x7f709806fc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:07:37.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.510+0000 7f70a0400640 1 -- 172.21.15.16:0/4250770970 shutdown_connections 2024-03-31T15:07:37.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.510+0000 7f70a0400640 1 -- 172.21.15.16:0/4250770970 wait complete. 2024-03-31T15:07:37.525 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-31T15:07:37.525 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-31T15:07:37.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.670+0000 7f4e3a200640 1 Processor -- start 2024-03-31T15:07:37.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.670+0000 7f4e3a200640 1 -- start start 2024-03-31T15:07:37.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.670+0000 7f4e3a200640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f4e3406dbc0 0x7f4e3415f650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:37.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.670+0000 7f4e3a200640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f4e340cb440 0x7f4e3406f920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:37.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.670+0000 7f4e3a200640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4e340cbe90 con 0x7f4e340cb8c0 2024-03-31T15:07:37.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.670+0000 7f4e3a200640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f4e3415fb90 con 0x7f4e3406dbc0 2024-03-31T15:07:37.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.670+0000 7f4e3a200640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f4e3415f1c0 con 0x7f4e340cb440 2024-03-31T15:07:37.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.670+0000 7f4e33400640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f4e3406dbc0 0x7f4e3415f650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:37.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.670+0000 7f4e33e00640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f4e340cb8c0 0x7f4e34162180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:53750/0 (socket says 172.21.15.16:53750) 2024-03-31T15:07:37.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.670+0000 7f4e33400640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f4e3406dbc0 0x7f4e3415f650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47332/0 (socket says 172.21.15.16:47332) 2024-03-31T15:07:37.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.670+0000 7f4e33e00640 1 -- 172.21.15.16:0/3774238101 learned_addr learned my addr 172.21.15.16:0/3774238101 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:07:37.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.670+0000 7f4e2be00640 1 --2- 172.21.15.16:0/3774238101 >> v2:172.21.15.89:3300/0 conn(0x7f4e340cb440 0x7f4e3406f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:37.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.670+0000 7f4e33400640 1 -- 172.21.15.16:0/3774238101 >> v2:172.21.15.89:3300/0 conn(0x7f4e340cb440 msgr2=0x7f4e3406f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:07:37.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.670+0000 7f4e33400640 1 --2- 172.21.15.16:0/3774238101 >> v2:172.21.15.89:3300/0 conn(0x7f4e340cb440 0x7f4e3406f920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:37.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.670+0000 7f4e33400640 1 -- 172.21.15.16:0/3774238101 >> v1:172.21.15.16:6789/0 conn(0x7f4e340cb8c0 legacy=0x7f4e34162180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:37.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.670+0000 7f4e33400640 1 -- 172.21.15.16:0/3774238101 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e340cc170 con 0x7f4e3406dbc0 2024-03-31T15:07:37.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.674+0000 7f4e33400640 1 --2- 172.21.15.16:0/3774238101 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4e3406dbc0 0x7f4e3415f650 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f4e340cf0b0 tx=0x7f4e2401c3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=783c967dc287be00 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:07:37.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.678+0000 7f4e32a00640 1 -- 172.21.15.16:0/3774238101 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e240b1070 con 0x7f4e3406dbc0 2024-03-31T15:07:37.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.678+0000 7f4e32a00640 1 -- 172.21.15.16:0/3774238101 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4e240a8050 con 0x7f4e3406dbc0 2024-03-31T15:07:37.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.678+0000 7f4e32a00640 1 -- 172.21.15.16:0/3774238101 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e240ac860 con 0x7f4e3406dbc0 2024-03-31T15:07:37.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.678+0000 7f4e3a200640 1 -- 172.21.15.16:0/3774238101 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4e3406dbc0 msgr2=0x7f4e3415f650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:37.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.678+0000 7f4e3a200640 1 --2- 172.21.15.16:0/3774238101 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4e3406dbc0 0x7f4e3415f650 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f4e340cf0b0 tx=0x7f4e2401c3c0 comp rx=0 tx=0).stop 2024-03-31T15:07:37.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.678+0000 7f4e3a200640 1 -- 172.21.15.16:0/3774238101 shutdown_connections 2024-03-31T15:07:37.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.678+0000 7f4e3a200640 1 --2- 172.21.15.16:0/3774238101 >> v2:172.21.15.89:3300/0 conn(0x7f4e340cb440 0x7f4e3406f920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:37.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.678+0000 7f4e3a200640 1 --2- 172.21.15.16:0/3774238101 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4e3406dbc0 0x7f4e3415f650 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:37.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.678+0000 7f4e3a200640 1 -- 172.21.15.16:0/3774238101 >> 172.21.15.16:0/3774238101 conn(0x7f4e34067330 msgr2=0x7f4e34067740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:07:37.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.678+0000 7f4e3a200640 1 -- 172.21.15.16:0/3774238101 shutdown_connections 2024-03-31T15:07:37.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.678+0000 7f4e3a200640 1 -- 172.21.15.16:0/3774238101 wait complete. 2024-03-31T15:07:37.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.678+0000 7f4e3a200640 1 Processor -- start 2024-03-31T15:07:37.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.678+0000 7f4e3a200640 1 -- start start 2024-03-31T15:07:37.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.682+0000 7f4e3a200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4e3406dbc0 0x7f4e34163930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:37.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.682+0000 7f4e3a200640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4e340cb440 0x7f4e34156a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:37.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.682+0000 7f4e3a200640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f4e340cb8c0 0x7f4e34160d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:37.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.682+0000 7f4e3a200640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f4e34162ca0 con 0x7f4e3406dbc0 2024-03-31T15:07:37.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.682+0000 7f4e3a200640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f4e34070270 con 0x7f4e340cb440 2024-03-31T15:07:37.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.682+0000 7f4e3a200640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f4e3406c950 con 0x7f4e340cb8c0 2024-03-31T15:07:37.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.682+0000 7f4e33400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4e3406dbc0 0x7f4e34163930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:37.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.682+0000 7f4e33400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4e3406dbc0 0x7f4e34163930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47340/0 (socket says 172.21.15.16:47340) 2024-03-31T15:07:37.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.682+0000 7f4e33400640 1 -- 172.21.15.16:0/958089420 learned_addr learned my addr 172.21.15.16:0/958089420 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:07:37.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.682+0000 7f4e2be00640 1 --2- 172.21.15.16:0/958089420 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4e340cb440 0x7f4e34156a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:37.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.682+0000 7f4e33e00640 1 --2- 172.21.15.16:0/958089420 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f4e340cb8c0 0x7f4e34160d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:37.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.682+0000 7f4e33400640 1 -- 172.21.15.16:0/958089420 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f4e340cb8c0 msgr2=0x7f4e34160d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:37.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.682+0000 7f4e33400640 1 --2- 172.21.15.16:0/958089420 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f4e340cb8c0 0x7f4e34160d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:37.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.682+0000 7f4e33400640 1 -- 172.21.15.16:0/958089420 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4e340cb440 msgr2=0x7f4e34156a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:37.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.682+0000 7f4e33400640 1 --2- 172.21.15.16:0/958089420 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4e340cb440 0x7f4e34156a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:37.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.682+0000 7f4e33400640 1 -- 172.21.15.16:0/958089420 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e3415a910 con 0x7f4e3406dbc0 2024-03-31T15:07:37.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.682+0000 7f4e33e00640 1 --2- 172.21.15.16:0/958089420 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f4e340cb8c0 0x7f4e34160d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:07:37.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.682+0000 7f4e33400640 1 --2- 172.21.15.16:0/958089420 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4e3406dbc0 0x7f4e34163930 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f4e24018fd0 tx=0x7f4e2401ac00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:07:37.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.682+0000 7f4e2b400640 1 -- 172.21.15.16:0/958089420 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e240bb020 con 0x7f4e3406dbc0 2024-03-31T15:07:37.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.682+0000 7f4e2b400640 1 -- 172.21.15.16:0/958089420 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4e240b7420 con 0x7f4e3406dbc0 2024-03-31T15:07:37.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.682+0000 7f4e3a200640 1 -- 172.21.15.16:0/958089420 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4e341641c0 con 0x7f4e3406dbc0 2024-03-31T15:07:37.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.682+0000 7f4e2b400640 1 -- 172.21.15.16:0/958089420 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e240b1040 con 0x7f4e3406dbc0 2024-03-31T15:07:37.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.682+0000 7f4e3a200640 1 -- 172.21.15.16:0/958089420 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4e3415abd0 con 0x7f4e3406dbc0 2024-03-31T15:07:37.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.682+0000 7f4e3a200640 1 -- 172.21.15.16:0/958089420 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4e34069950 con 0x7f4e3406dbc0 2024-03-31T15:07:37.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.686+0000 7f4e2b400640 1 -- 172.21.15.16:0/958089420 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f4e240be020 con 0x7f4e3406dbc0 2024-03-31T15:07:37.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.686+0000 7f4e2b400640 1 --2- 172.21.15.16:0/958089420 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f4e100683b0 0x7f4e1006a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:37.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.686+0000 7f4e2b400640 1 -- 172.21.15.16:0/958089420 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 5709+0+0 (secure 0 0 0) 0x7f4e24120a40 con 0x7f4e3406dbc0 2024-03-31T15:07:37.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.686+0000 7f4e2be00640 1 --2- 172.21.15.16:0/958089420 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f4e100683b0 0x7f4e1006a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:37.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.690+0000 7f4e2be00640 1 --2- 172.21.15.16:0/958089420 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f4e100683b0 0x7f4e1006a870 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f4e340b4dc0 tx=0x7f4e1c008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:07:37.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.690+0000 7f4e2b400640 1 -- 172.21.15.16:0/958089420 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f4e240c5070 con 0x7f4e3406dbc0 2024-03-31T15:07:37.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.890+0000 7f4e3a200640 1 -- 172.21.15.16:0/958089420 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4e340cf380 con 0x7f4e3406dbc0 2024-03-31T15:07:37.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.890+0000 7f4e2b400640 1 -- 172.21.15.16:0/958089420 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 36 v36) ==== 94+0+5182 (secure 0 0 0) 0x7f4e24016070 con 0x7f4e3406dbc0 2024-03-31T15:07:37.887 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:07:37.887 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":36,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5579,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6833/3721971977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":3721971977},{"type":"v1","addr":"172.21.15.16:6833","nonce":3721971977}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":29},{"gid":6109,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6836/1166206553","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":1166206553},{"type":"v1","addr":"172.21.15.89:6836","nonce":1166206553}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28},{"gid":6112,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/1826113083","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6835","nonce":1826113083},{"type":"v1","addr":"172.21.15.89:6837","nonce":1826113083}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":29}],"filesystems":[{"mdsmap":{"epoch":36,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-31T15:06:25.665001+0000","modified":"2024-03-31T15:07:14.315988+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6150},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6150":{"gid":6150,"name":"c","rank":0,"incarnation":30,"state":"up:active","state_seq":15,"addr":"172.21.15.16:6835/2613117528","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":2613117528},{"type":"v1","addr":"172.21.15.16:6835","nonce":2613117528}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6150,"qdb_cluster":[6150]},"id":4}]} 2024-03-31T15:07:37.887 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 36 2024-03-31T15:07:37.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.894+0000 7f4e3a200640 1 -- 172.21.15.16:0/958089420 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f4e100683b0 msgr2=0x7f4e1006a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:37.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.894+0000 7f4e3a200640 1 --2- 172.21.15.16:0/958089420 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f4e100683b0 0x7f4e1006a870 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f4e340b4dc0 tx=0x7f4e1c008040 comp rx=0 tx=0).stop 2024-03-31T15:07:37.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.894+0000 7f4e3a200640 1 -- 172.21.15.16:0/958089420 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4e3406dbc0 msgr2=0x7f4e34163930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:37.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.894+0000 7f4e3a200640 1 --2- 172.21.15.16:0/958089420 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4e3406dbc0 0x7f4e34163930 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f4e24018fd0 tx=0x7f4e2401ac00 comp rx=0 tx=0).stop 2024-03-31T15:07:37.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.894+0000 7f4e3a200640 1 -- 172.21.15.16:0/958089420 shutdown_connections 2024-03-31T15:07:37.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.894+0000 7f4e3a200640 1 --2- 172.21.15.16:0/958089420 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f4e100683b0 0x7f4e1006a870 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:37.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.894+0000 7f4e3a200640 1 --2- 172.21.15.16:0/958089420 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f4e340cb8c0 0x7f4e34160d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:37.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.894+0000 7f4e3a200640 1 --2- 172.21.15.16:0/958089420 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4e340cb440 0x7f4e34156a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:37.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.894+0000 7f4e3a200640 1 --2- 172.21.15.16:0/958089420 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4e3406dbc0 0x7f4e34163930 unknown :-1 s=CLOSED pgs=723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:37.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.894+0000 7f4e3a200640 1 -- 172.21.15.16:0/958089420 >> 172.21.15.16:0/958089420 conn(0x7f4e34067330 msgr2=0x7f4e340d1880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:07:37.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.894+0000 7f4e3a200640 1 -- 172.21.15.16:0/958089420 shutdown_connections 2024-03-31T15:07:37.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:37.894+0000 7f4e3a200640 1 -- 172.21.15.16:0/958089420 wait complete. 2024-03-31T15:07:37.908 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c damage ls --format=json-pretty 2024-03-31T15:07:38.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.054+0000 7f3540800640 1 Processor -- start 2024-03-31T15:07:38.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.054+0000 7f3540800640 1 -- start start 2024-03-31T15:07:38.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.054+0000 7f3540800640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f35381411b0 0x7f353806f920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:38.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.054+0000 7f3540800640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f3538141590 0x7f353806fe60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:38.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.054+0000 7f3540800640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f3538143c30 0x7f35381440a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:38.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.054+0000 7f3540800640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f3538162910 con 0x7f3538141590 2024-03-31T15:07:38.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.054+0000 7f3540800640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f3538144980 con 0x7f35381411b0 2024-03-31T15:07:38.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.054+0000 7f3540800640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f3538161fe0 con 0x7f3538143c30 2024-03-31T15:07:38.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.054+0000 7f353d400640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f3538141590 0x7f353806fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:38.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.054+0000 7f353d400640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f3538141590 0x7f353806fe60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47344/0 (socket says 172.21.15.16:47344) 2024-03-31T15:07:38.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.054+0000 7f353d400640 1 -- 172.21.15.16:0/3270271524 learned_addr learned my addr 172.21.15.16:0/3270271524 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:07:38.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.054+0000 7f353de00640 1 --2- 172.21.15.16:0/3270271524 >> v2:172.21.15.89:3300/0 conn(0x7f35381411b0 0x7f353806f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:38.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.054+0000 7f353e800640 1 --2- 172.21.15.16:0/3270271524 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f3538143c30 0x7f35381440a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:38.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.054+0000 7f353d400640 1 -- 172.21.15.16:0/3270271524 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f3538143c30 msgr2=0x7f35381440a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:07:38.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.054+0000 7f353d400640 1 --2- 172.21.15.16:0/3270271524 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f3538143c30 0x7f35381440a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:38.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.054+0000 7f353d400640 1 -- 172.21.15.16:0/3270271524 >> v2:172.21.15.89:3300/0 conn(0x7f35381411b0 msgr2=0x7f353806f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:07:38.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.054+0000 7f353d400640 1 --2- 172.21.15.16:0/3270271524 >> v2:172.21.15.89:3300/0 conn(0x7f35381411b0 0x7f353806f920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:38.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.054+0000 7f353d400640 1 -- 172.21.15.16:0/3270271524 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35381622c0 con 0x7f3538141590 2024-03-31T15:07:38.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.054+0000 7f353de00640 1 --2- 172.21.15.16:0/3270271524 >> v2:172.21.15.89:3300/0 conn(0x7f35381411b0 0x7f353806f920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:07:38.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.058+0000 7f353d400640 1 --2- 172.21.15.16:0/3270271524 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3538141590 0x7f353806fe60 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f35380d3120 tx=0x7f352801c4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba8417d682647403 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:07:38.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.058+0000 7f353ca00640 1 -- 172.21.15.16:0/3270271524 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35280aa430 con 0x7f3538141590 2024-03-31T15:07:38.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.058+0000 7f353ca00640 1 -- 172.21.15.16:0/3270271524 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f35280af070 con 0x7f3538141590 2024-03-31T15:07:38.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.058+0000 7f353ca00640 1 -- 172.21.15.16:0/3270271524 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35280aa7f0 con 0x7f3538141590 2024-03-31T15:07:38.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.058+0000 7f3540800640 1 -- 172.21.15.16:0/3270271524 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3538141590 msgr2=0x7f353806fe60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:38.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.058+0000 7f3540800640 1 --2- 172.21.15.16:0/3270271524 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3538141590 0x7f353806fe60 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f35380d3120 tx=0x7f352801c4f0 comp rx=0 tx=0).stop 2024-03-31T15:07:38.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.058+0000 7f3540800640 1 -- 172.21.15.16:0/3270271524 shutdown_connections 2024-03-31T15:07:38.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.058+0000 7f3540800640 1 --2- 172.21.15.16:0/3270271524 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f3538143c30 0x7f35381440a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:38.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.058+0000 7f3540800640 1 --2- 172.21.15.16:0/3270271524 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3538141590 0x7f353806fe60 unknown :-1 s=CLOSED pgs=724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:38.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.058+0000 7f3540800640 1 --2- 172.21.15.16:0/3270271524 >> v2:172.21.15.89:3300/0 conn(0x7f35381411b0 0x7f353806f920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:38.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.058+0000 7f3540800640 1 -- 172.21.15.16:0/3270271524 >> 172.21.15.16:0/3270271524 conn(0x7f3538067930 msgr2=0x7f35380b3ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:07:38.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.058+0000 7f3540800640 1 -- 172.21.15.16:0/3270271524 shutdown_connections 2024-03-31T15:07:38.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.062+0000 7f3540800640 1 -- 172.21.15.16:0/3270271524 wait complete. 2024-03-31T15:07:38.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.062+0000 7f3540800640 1 Processor -- start 2024-03-31T15:07:38.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.062+0000 7f3540800640 1 -- start start 2024-03-31T15:07:38.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.062+0000 7f3540800640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f35381411b0 0x7f35380b38e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:38.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.062+0000 7f3540800640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f3538141590 0x7f35380d2ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:38.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.062+0000 7f3540800640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3538143c30 0x7f3538140450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:38.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.062+0000 7f3540800640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f35380717f0 con 0x7f35381411b0 2024-03-31T15:07:38.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.062+0000 7f3540800640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f3538071670 con 0x7f3538143c30 2024-03-31T15:07:38.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.062+0000 7f3540800640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f3538071970 con 0x7f3538141590 2024-03-31T15:07:38.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.062+0000 7f353de00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f35381411b0 0x7f35380b38e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:38.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.062+0000 7f353de00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f35381411b0 0x7f35380b38e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47348/0 (socket says 172.21.15.16:47348) 2024-03-31T15:07:38.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.062+0000 7f353de00640 1 -- 172.21.15.16:0/1100413067 learned_addr learned my addr 172.21.15.16:0/1100413067 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:07:38.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.062+0000 7f353d400640 1 --2- 172.21.15.16:0/1100413067 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f3538141590 0x7f35380d2ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:38.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.062+0000 7f353e800640 1 --2- 172.21.15.16:0/1100413067 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3538143c30 0x7f3538140450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:38.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.062+0000 7f353de00640 1 -- 172.21.15.16:0/1100413067 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f3538141590 msgr2=0x7f35380d2ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:38.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.062+0000 7f353de00640 1 --2- 172.21.15.16:0/1100413067 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f3538141590 0x7f35380d2ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:38.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.062+0000 7f353de00640 1 -- 172.21.15.16:0/1100413067 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3538143c30 msgr2=0x7f3538140450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:38.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.062+0000 7f353de00640 1 --2- 172.21.15.16:0/1100413067 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3538143c30 0x7f3538140450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:38.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.062+0000 7f353de00640 1 -- 172.21.15.16:0/1100413067 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3538140990 con 0x7f35381411b0 2024-03-31T15:07:38.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.062+0000 7f353e800640 1 --2- 172.21.15.16:0/1100413067 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3538143c30 0x7f3538140450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:07:38.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.062+0000 7f353de00640 1 --2- 172.21.15.16:0/1100413067 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f35381411b0 0x7f35380b38e0 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7f352c00e840 tx=0x7f352c00ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:07:38.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.062+0000 7f3532000640 1 -- 172.21.15.16:0/1100413067 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f352c017070 con 0x7f35381411b0 2024-03-31T15:07:38.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.062+0000 7f3532000640 1 -- 172.21.15.16:0/1100413067 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f352c00c8d0 con 0x7f35381411b0 2024-03-31T15:07:38.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.062+0000 7f3532000640 1 -- 172.21.15.16:0/1100413067 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f352c00f040 con 0x7f35381411b0 2024-03-31T15:07:38.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.062+0000 7f3540800640 1 -- 172.21.15.16:0/1100413067 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f35380b3ec0 con 0x7f35381411b0 2024-03-31T15:07:38.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.066+0000 7f3540800640 1 -- 172.21.15.16:0/1100413067 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f35381fa870 con 0x7f35381411b0 2024-03-31T15:07:38.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.066+0000 7f3532000640 1 -- 172.21.15.16:0/1100413067 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f352c01a070 con 0x7f35381411b0 2024-03-31T15:07:38.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.066+0000 7f3532000640 1 --2- 172.21.15.16:0/1100413067 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f351c068380 0x7f351c06a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:38.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.066+0000 7f3532000640 1 -- 172.21.15.16:0/1100413067 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 5709+0+0 (secure 0 0 0) 0x7f352c086560 con 0x7f35381411b0 2024-03-31T15:07:38.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.066+0000 7f3543d97000 1 Processor -- start 2024-03-31T15:07:38.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.066+0000 7f3543d97000 1 -- start start 2024-03-31T15:07:38.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.066+0000 7f353d400640 1 --2- 172.21.15.16:0/1100413067 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f351c068380 0x7f351c06a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:38.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.070+0000 7f353d400640 1 --2- 172.21.15.16:0/1100413067 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f351c068380 0x7f351c06a840 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f35280ae000 tx=0x7f35280a7000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:07:38.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.070+0000 7f3543d97000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5592776a5980 0x5592776a5d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:38.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.070+0000 7f3543d97000 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5592776a62a0 0x559277658dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:38.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.070+0000 7f3543d97000 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559277659310 0x5592776596f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:38.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.070+0000 7f3543d97000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5592774a9c50 con 0x5592776a5980 2024-03-31T15:07:38.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.070+0000 7f3543d97000 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x5592774468a0 con 0x559277659310 2024-03-31T15:07:38.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.070+0000 7f3543d97000 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x5592774a5c00 con 0x5592776a62a0 2024-03-31T15:07:38.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.070+0000 7f353de00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5592776a5980 0x5592776a5d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:38.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.070+0000 7f353de00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5592776a5980 0x5592776a5d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47356/0 (socket says 172.21.15.16:47356) 2024-03-31T15:07:38.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.070+0000 7f353de00640 1 -- 172.21.15.16:0/2172690412 learned_addr learned my addr 172.21.15.16:0/2172690412 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:07:38.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.070+0000 7f353d400640 1 --2- 172.21.15.16:0/2172690412 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5592776a62a0 0x559277658dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:38.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.070+0000 7f353e800640 1 --2- 172.21.15.16:0/2172690412 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559277659310 0x5592776596f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:38.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.070+0000 7f353de00640 1 -- 172.21.15.16:0/2172690412 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5592776a62a0 msgr2=0x559277658dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:07:38.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.070+0000 7f353de00640 1 --2- 172.21.15.16:0/2172690412 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5592776a62a0 0x559277658dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:38.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.070+0000 7f353de00640 1 -- 172.21.15.16:0/2172690412 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559277659310 msgr2=0x5592776596f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:07:38.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.070+0000 7f353de00640 1 --2- 172.21.15.16:0/2172690412 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559277659310 0x5592776596f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:38.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.070+0000 7f353de00640 1 -- 172.21.15.16:0/2172690412 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559277483ba0 con 0x5592776a5980 2024-03-31T15:07:38.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.070+0000 7f353e800640 1 --2- 172.21.15.16:0/2172690412 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559277659310 0x5592776596f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:07:38.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.070+0000 7f353de00640 1 --2- 172.21.15.16:0/2172690412 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5592776a5980 0x5592776a5d60 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f352c0561c0 tx=0x7f352c056580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4bca3c42e54d08b5 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:07:38.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.070+0000 7f3526a00640 1 -- 172.21.15.16:0/2172690412 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f352c021040 con 0x5592776a5980 2024-03-31T15:07:38.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.070+0000 7f3526a00640 1 -- 172.21.15.16:0/2172690412 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f352c084e50 con 0x5592776a5980 2024-03-31T15:07:38.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.070+0000 7f3526a00640 1 -- 172.21.15.16:0/2172690412 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f352c052b30 con 0x5592776a5980 2024-03-31T15:07:38.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.070+0000 7f3543d97000 1 -- 172.21.15.16:0/2172690412 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5592776a5980 msgr2=0x5592776a5d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:38.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.070+0000 7f3543d97000 1 --2- 172.21.15.16:0/2172690412 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5592776a5980 0x5592776a5d60 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f352c0561c0 tx=0x7f352c056580 comp rx=0 tx=0).stop 2024-03-31T15:07:38.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.070+0000 7f3543d97000 1 -- 172.21.15.16:0/2172690412 shutdown_connections 2024-03-31T15:07:38.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.070+0000 7f3543d97000 1 --2- 172.21.15.16:0/2172690412 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559277659310 0x5592776596f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:38.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.070+0000 7f3543d97000 1 --2- 172.21.15.16:0/2172690412 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5592776a62a0 0x559277658dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:38.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.070+0000 7f3543d97000 1 --2- 172.21.15.16:0/2172690412 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5592776a5980 0x5592776a5d60 unknown :-1 s=CLOSED pgs=726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:38.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.070+0000 7f3543d97000 1 -- 172.21.15.16:0/2172690412 >> 172.21.15.16:0/2172690412 conn(0x5592774569f0 msgr2=0x559277688730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:07:38.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.070+0000 7f3543d97000 1 -- 172.21.15.16:0/2172690412 shutdown_connections 2024-03-31T15:07:38.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.070+0000 7f3543d97000 1 -- 172.21.15.16:0/2172690412 wait complete. 2024-03-31T15:07:38.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.070+0000 7f3543d97000 1 -- start start 2024-03-31T15:07:38.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.070+0000 7f3543d97000 1 Processor -- start 2024-03-31T15:07:38.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.074+0000 7f3543d97000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559277659310 0x5592776a5110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:38.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.074+0000 7f3543d97000 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5592776a5980 0x55927765ea80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:38.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.074+0000 7f3543d97000 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5592776a62a0 0x55927765efc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:38.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.074+0000 7f3543d97000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5592774670a0 con 0x559277659310 2024-03-31T15:07:38.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.074+0000 7f3543d97000 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x559277476730 con 0x5592776a5980 2024-03-31T15:07:38.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.074+0000 7f3543d97000 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x55927755ce90 con 0x5592776a62a0 2024-03-31T15:07:38.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.074+0000 7f3543d97000 10 client.0 _ll_register_callbacks cb 0x559277646810 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T15:07:38.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.074+0000 7f353de00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559277659310 0x5592776a5110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:38.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.074+0000 7f353de00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559277659310 0x5592776a5110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47364/0 (socket says 172.21.15.16:47364) 2024-03-31T15:07:38.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.074+0000 7f353de00640 1 -- 172.21.15.16:0/1073918421 learned_addr learned my addr 172.21.15.16:0/1073918421 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:07:38.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.074+0000 7f353d400640 1 --2- 172.21.15.16:0/1073918421 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5592776a5980 0x55927765ea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:38.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.074+0000 7f353e800640 1 --2- 172.21.15.16:0/1073918421 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5592776a62a0 0x55927765efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:38.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.074+0000 7f353de00640 1 -- 172.21.15.16:0/1073918421 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5592776a62a0 msgr2=0x55927765efc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:07:38.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.074+0000 7f353de00640 1 --2- 172.21.15.16:0/1073918421 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5592776a62a0 0x55927765efc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:38.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.074+0000 7f353de00640 1 -- 172.21.15.16:0/1073918421 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5592776a5980 msgr2=0x55927765ea80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:07:38.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.074+0000 7f353de00640 1 --2- 172.21.15.16:0/1073918421 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5592776a5980 0x55927765ea80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:38.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.074+0000 7f353de00640 1 -- 172.21.15.16:0/1073918421 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5592774ef160 con 0x559277659310 2024-03-31T15:07:38.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.074+0000 7f353de00640 1 --2- 172.21.15.16:0/1073918421 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559277659310 0x5592776a5110 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f352c05eac0 tx=0x7f352c000bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=738824198fd53a9f server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:07:38.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.074+0000 7f351b400640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-31T15:07:38.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.074+0000 7f351b400640 1 -- 172.21.15.16:0/1073918421 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f352c021040 con 0x559277659310 2024-03-31T15:07:38.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.074+0000 7f351b400640 1 -- 172.21.15.16:0/1073918421 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(56..56 src has 1..56) ==== 5709+0+0 (secure 0 0 0) 0x7f352c0862c0 con 0x559277659310 2024-03-31T15:07:38.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.074+0000 7f3543d97000 1 -- 172.21.15.16:0/1073918421 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f352c084e50 con 0x559277659310 2024-03-31T15:07:38.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.074+0000 7f351b400640 1 -- 172.21.15.16:0/1073918421 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f352c054880 con 0x559277659310 2024-03-31T15:07:38.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.074+0000 7f3543d97000 10 client.6501 fetch_fsmap learned FSMap version 36 2024-03-31T15:07:38.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.074+0000 7f3543d97000 1 -- 172.21.15.16:0/1073918421 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=2+}) -- 0x5592774449c0 con 0x559277659310 2024-03-31T15:07:38.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.074+0000 7f351b400640 1 -- 172.21.15.16:0/1073918421 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f352c084e50 con 0x559277659310 2024-03-31T15:07:38.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.074+0000 7f351b400640 1 -- 172.21.15.16:0/1073918421 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 36) ==== 3494+0+0 (secure 0 0 0) 0x7f352c012050 con 0x559277659310 2024-03-31T15:07:38.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.074+0000 7f3543d97000 10 client.6501 fetch_fsmap finished waiting for FSMap version 36 2024-03-31T15:07:38.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.074+0000 7f3543d97000 10 client.6501 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-31T15:07:38.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.074+0000 7f3543d97000 20 client.6501 populate_metadata read hostname 'smithi016' 2024-03-31T15:07:38.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.074+0000 7f3543d97000 1 --2- 172.21.15.16:0/1073918421 >> [v2:172.21.15.16:6834/2613117528,v1:172.21.15.16:6835/2613117528] conn(0x5592776acdc0 0x5592776ad1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:38.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.074+0000 7f3543d97000 4 client.6501 mds_command: new command op to 6150 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-31T15:07:38.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.074+0000 7f3543d97000 1 -- 172.21.15.16:0/1073918421 --> [v2:172.21.15.16:6834/2613117528,v1:172.21.15.16:6835/2613117528] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559277444e40 con 0x5592776acdc0 2024-03-31T15:07:38.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.074+0000 7f353d400640 1 --2- 172.21.15.16:0/1073918421 >> [v2:172.21.15.16:6834/2613117528,v1:172.21.15.16:6835/2613117528] conn(0x5592776acdc0 0x5592776ad1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:38.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.074+0000 7f353d400640 1 --2- 172.21.15.16:0/1073918421 >> [v2:172.21.15.16:6834/2613117528,v1:172.21.15.16:6835/2613117528] conn(0x5592776acdc0 0x5592776ad1a0 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b1ea64645f23363d server_cookie=e98c435ac906895b in_seq=0 out_seq=0 2024-03-31T15:07:38.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.074+0000 7f351b400640 10 client.6501 ms_handle_connect on v2:172.21.15.16:6834/2613117528 2024-03-31T15:07:38.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.074+0000 7f351b400640 1 -- 172.21.15.16:0/1073918421 <== mds.0 v2:172.21.15.16:6834/2613117528 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x559277444e40 con 0x5592776acdc0 2024-03-31T15:07:38.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.074+0000 7f351b400640 10 client.6501 handle_command_reply: tid=0 2024-03-31T15:07:38.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.074+0000 7f3543d97000 1 client.6501 shutdown 2024-03-31T15:07:38.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.074+0000 7f3543d97000 1 -- 172.21.15.16:0/1073918421 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559277659310 msgr2=0x5592776a5110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:38.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.074+0000 7f3543d97000 1 --2- 172.21.15.16:0/1073918421 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559277659310 0x5592776a5110 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f352c05eac0 tx=0x7f352c000bd0 comp rx=0 tx=0).stop 2024-03-31T15:07:38.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.074+0000 7f3543d97000 1 -- 172.21.15.16:0/1073918421 shutdown_connections 2024-03-31T15:07:38.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.074+0000 7f3543d97000 1 --2- 172.21.15.16:0/1073918421 >> [v2:172.21.15.16:6834/2613117528,v1:172.21.15.16:6835/2613117528] conn(0x5592776acdc0 0x5592776ad1a0 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:38.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.074+0000 7f3543d97000 1 --2- 172.21.15.16:0/1073918421 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5592776a62a0 0x55927765efc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:38.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.074+0000 7f3543d97000 1 --2- 172.21.15.16:0/1073918421 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5592776a5980 0x55927765ea80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:38.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.074+0000 7f3543d97000 1 --2- 172.21.15.16:0/1073918421 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559277659310 0x5592776a5110 unknown :-1 s=CLOSED pgs=727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:38.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.074+0000 7f351b400640 0 client.6501 ms_handle_reset on v2:172.21.15.16:6834/2613117528 2024-03-31T15:07:38.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.074+0000 7f3543d97000 1 -- 172.21.15.16:0/1073918421 >> 172.21.15.16:0/1073918421 conn(0x5592774569f0 msgr2=0x559277688520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:07:38.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.078+0000 7f3543d97000 1 -- 172.21.15.16:0/1073918421 shutdown_connections 2024-03-31T15:07:38.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.078+0000 7f3543d97000 1 -- 172.21.15.16:0/1073918421 wait complete. 2024-03-31T15:07:38.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.078+0000 7f3543d97000 20 client.6501 trim_cache size 0 max 16384 2024-03-31T15:07:38.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.090+0000 7f3543d97000 1 Processor -- start 2024-03-31T15:07:38.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.090+0000 7f3543d97000 1 -- start start 2024-03-31T15:07:38.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.090+0000 7f3543d97000 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559277659310 0x5592776a4140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:38.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.090+0000 7f3543d97000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5592776a5980 0x5592776a4680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:38.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.090+0000 7f3543d97000 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5592776a62a0 0x5592776608a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:38.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.090+0000 7f3543d97000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f3538202680 con 0x5592776a5980 2024-03-31T15:07:38.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.090+0000 7f3543d97000 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f3538201a10 con 0x559277659310 2024-03-31T15:07:38.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.090+0000 7f3543d97000 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f3538202030 con 0x5592776a62a0 2024-03-31T15:07:38.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.090+0000 7f353d400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5592776a5980 0x5592776a4680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:38.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.090+0000 7f353d400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5592776a5980 0x5592776a4680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47374/0 (socket says 172.21.15.16:47374) 2024-03-31T15:07:38.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.090+0000 7f353d400640 1 -- 172.21.15.16:0/475987790 learned_addr learned my addr 172.21.15.16:0/475987790 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:07:38.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.090+0000 7f353d400640 1 -- 172.21.15.16:0/475987790 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5592776a62a0 msgr2=0x5592776608a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:07:38.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.090+0000 7f353d400640 1 --2- 172.21.15.16:0/475987790 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5592776a62a0 0x5592776608a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:38.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.090+0000 7f353d400640 1 -- 172.21.15.16:0/475987790 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559277659310 msgr2=0x5592776a4140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:07:38.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.090+0000 7f353d400640 1 --2- 172.21.15.16:0/475987790 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559277659310 0x5592776a4140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:38.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.090+0000 7f353d400640 1 -- 172.21.15.16:0/475987790 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559277446af0 con 0x5592776a5980 2024-03-31T15:07:38.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.090+0000 7f353d400640 1 --2- 172.21.15.16:0/475987790 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5592776a5980 0x5592776a4680 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f35280ac040 tx=0x7f3528009240 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6959a2de5242627a server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:07:38.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.090+0000 7f3526000640 1 -- 172.21.15.16:0/475987790 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35280cc3d0 con 0x5592776a5980 2024-03-31T15:07:38.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.094+0000 7f3526000640 1 -- 172.21.15.16:0/475987790 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f35280d2070 con 0x5592776a5980 2024-03-31T15:07:38.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.094+0000 7f3526000640 1 -- 172.21.15.16:0/475987790 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35280cc770 con 0x5592776a5980 2024-03-31T15:07:38.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.094+0000 7f3543d97000 1 -- 172.21.15.16:0/475987790 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5592776a5980 msgr2=0x5592776a4680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:38.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.094+0000 7f3543d97000 1 --2- 172.21.15.16:0/475987790 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5592776a5980 0x5592776a4680 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f35280ac040 tx=0x7f3528009240 comp rx=0 tx=0).stop 2024-03-31T15:07:38.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.094+0000 7f3543d97000 1 -- 172.21.15.16:0/475987790 shutdown_connections 2024-03-31T15:07:38.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.094+0000 7f3543d97000 1 --2- 172.21.15.16:0/475987790 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5592776a62a0 0x5592776608a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:38.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.094+0000 7f3543d97000 1 --2- 172.21.15.16:0/475987790 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5592776a5980 0x5592776a4680 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:38.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.094+0000 7f3543d97000 1 --2- 172.21.15.16:0/475987790 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559277659310 0x5592776a4140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:38.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.094+0000 7f3543d97000 1 -- 172.21.15.16:0/475987790 >> 172.21.15.16:0/475987790 conn(0x5592774569f0 msgr2=0x5592776a3a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:07:38.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.094+0000 7f3543d97000 1 -- 172.21.15.16:0/475987790 shutdown_connections 2024-03-31T15:07:38.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.094+0000 7f3543d97000 1 -- 172.21.15.16:0/475987790 wait complete. 2024-03-31T15:07:38.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.094+0000 7f3543d97000 1 -- start start 2024-03-31T15:07:38.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.094+0000 7f3543d97000 1 Processor -- start 2024-03-31T15:07:38.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.094+0000 7f3543d97000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559277659310 0x559277715fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:38.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.094+0000 7f3543d97000 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5592776a5980 0x5592777164e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:38.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.094+0000 7f3543d97000 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5592776a62a0 0x5592776a8040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:38.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.094+0000 7f3543d97000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x559277477040 con 0x559277659310 2024-03-31T15:07:38.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.094+0000 7f3543d97000 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x5592774670a0 con 0x5592776a5980 2024-03-31T15:07:38.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.094+0000 7f3543d97000 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x5592776617e0 con 0x5592776a62a0 2024-03-31T15:07:38.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.094+0000 7f3543d97000 10 client.0 _ll_register_callbacks cb 0x7f352c06d070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T15:07:38.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.094+0000 7f353de00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559277659310 0x559277715fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:38.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.094+0000 7f353de00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559277659310 0x559277715fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47376/0 (socket says 172.21.15.16:47376) 2024-03-31T15:07:38.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.094+0000 7f353de00640 1 -- 172.21.15.16:0/240933219 learned_addr learned my addr 172.21.15.16:0/240933219 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:07:38.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.098+0000 7f353d400640 1 --2- 172.21.15.16:0/240933219 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5592776a5980 0x5592777164e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:38.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.098+0000 7f353e800640 1 --2- 172.21.15.16:0/240933219 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5592776a62a0 0x5592776a8040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:38.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.098+0000 7f353de00640 1 -- 172.21.15.16:0/240933219 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5592776a62a0 msgr2=0x5592776a8040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:07:38.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.098+0000 7f353de00640 1 --2- 172.21.15.16:0/240933219 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5592776a62a0 0x5592776a8040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:38.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.098+0000 7f353de00640 1 -- 172.21.15.16:0/240933219 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5592776a5980 msgr2=0x5592777164e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:07:38.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.098+0000 7f353de00640 1 --2- 172.21.15.16:0/240933219 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5592776a5980 0x5592777164e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:38.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.098+0000 7f353de00640 1 -- 172.21.15.16:0/240933219 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559277502320 con 0x559277659310 2024-03-31T15:07:38.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.098+0000 7f353e800640 1 --2- 172.21.15.16:0/240933219 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5592776a62a0 0x5592776a8040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:07:38.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.098+0000 7f353de00640 1 --2- 172.21.15.16:0/240933219 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559277659310 0x559277715fa0 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f352c052920 tx=0x7f352c084950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7310059cf582b4a7 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:07:38.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.098+0000 7f351b400640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-31T15:07:38.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.098+0000 7f351b400640 1 -- 172.21.15.16:0/240933219 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f352c052260 con 0x559277659310 2024-03-31T15:07:38.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.098+0000 7f351b400640 1 -- 172.21.15.16:0/240933219 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(56..56 src has 1..56) ==== 5709+0+0 (secure 0 0 0) 0x7f352c052400 con 0x559277659310 2024-03-31T15:07:38.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.098+0000 7f3543d97000 1 -- 172.21.15.16:0/240933219 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f35280d2070 con 0x559277659310 2024-03-31T15:07:38.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.098+0000 7f351b400640 1 -- 172.21.15.16:0/240933219 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f352c068050 con 0x559277659310 2024-03-31T15:07:38.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.098+0000 7f3543d97000 10 client.6507 fetch_fsmap learned FSMap version 36 2024-03-31T15:07:38.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.098+0000 7f3543d97000 1 -- 172.21.15.16:0/240933219 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=2+}) -- 0x55927743da80 con 0x559277659310 2024-03-31T15:07:38.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.098+0000 7f351b400640 1 -- 172.21.15.16:0/240933219 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f35280d2070 con 0x559277659310 2024-03-31T15:07:38.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.098+0000 7f351b400640 1 -- 172.21.15.16:0/240933219 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 36) ==== 3494+0+0 (secure 0 0 0) 0x7f352c012070 con 0x559277659310 2024-03-31T15:07:38.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.098+0000 7f3543d97000 10 client.6507 fetch_fsmap finished waiting for FSMap version 36 2024-03-31T15:07:38.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.098+0000 7f3543d97000 10 client.6507 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-31T15:07:38.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.098+0000 7f3543d97000 20 client.6507 populate_metadata read hostname 'smithi016' 2024-03-31T15:07:38.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.098+0000 7f3543d97000 1 --2- 172.21.15.16:0/240933219 >> [v2:172.21.15.16:6834/2613117528,v1:172.21.15.16:6835/2613117528] conn(0x5592776a8b20 0x5592776a8f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:38.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.098+0000 7f3543d97000 4 client.6507 mds_command: new command op to 6150 tid=0 multi_id=0 [{"prefix": "damage ls", "format": "json-pretty"}] 2024-03-31T15:07:38.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.098+0000 7f3543d97000 1 -- 172.21.15.16:0/240933219 --> [v2:172.21.15.16:6834/2613117528,v1:172.21.15.16:6835/2613117528] -- command(tid 0: {"prefix": "damage ls", "format": "json-pretty"}) -- 0x5592774b7190 con 0x5592776a8b20 2024-03-31T15:07:38.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.098+0000 7f353d400640 1 --2- 172.21.15.16:0/240933219 >> [v2:172.21.15.16:6834/2613117528,v1:172.21.15.16:6835/2613117528] conn(0x5592776a8b20 0x5592776a8f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:38.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.098+0000 7f353d400640 1 --2- 172.21.15.16:0/240933219 >> [v2:172.21.15.16:6834/2613117528,v1:172.21.15.16:6835/2613117528] conn(0x5592776a8b20 0x5592776a8f00 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5300bb425e0a8476 server_cookie=4f41f0cfa74b2260 in_seq=0 out_seq=0 2024-03-31T15:07:38.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.098+0000 7f351b400640 10 client.6507 ms_handle_connect on v2:172.21.15.16:6834/2613117528 2024-03-31T15:07:38.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.098+0000 7f351b400640 1 -- 172.21.15.16:0/240933219 <== mds.0 v2:172.21.15.16:6834/2613117528 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5592774b7190 con 0x5592776a8b20 2024-03-31T15:07:38.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.098+0000 7f351b400640 10 client.6507 handle_command_reply: tid=0 2024-03-31T15:07:38.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.098+0000 7f3543d97000 1 client.6507 shutdown 2024-03-31T15:07:38.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.098+0000 7f3543d97000 1 -- 172.21.15.16:0/240933219 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559277659310 msgr2=0x559277715fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:38.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.098+0000 7f3543d97000 1 --2- 172.21.15.16:0/240933219 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559277659310 0x559277715fa0 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f352c052920 tx=0x7f352c084950 comp rx=0 tx=0).stop 2024-03-31T15:07:38.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.098+0000 7f3543d97000 1 -- 172.21.15.16:0/240933219 shutdown_connections 2024-03-31T15:07:38.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.098+0000 7f3543d97000 1 --2- 172.21.15.16:0/240933219 >> [v2:172.21.15.16:6834/2613117528,v1:172.21.15.16:6835/2613117528] conn(0x5592776a8b20 0x5592776a8f00 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:38.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.098+0000 7f3543d97000 1 --2- 172.21.15.16:0/240933219 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5592776a62a0 0x5592776a8040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:38.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.098+0000 7f3543d97000 1 --2- 172.21.15.16:0/240933219 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5592776a5980 0x5592777164e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:38.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.098+0000 7f3543d97000 1 --2- 172.21.15.16:0/240933219 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559277659310 0x559277715fa0 unknown :-1 s=CLOSED pgs=729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:38.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.098+0000 7f351b400640 0 client.6507 ms_handle_reset on v2:172.21.15.16:6834/2613117528 2024-03-31T15:07:38.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.098+0000 7f3543d97000 1 -- 172.21.15.16:0/240933219 >> 172.21.15.16:0/240933219 conn(0x5592774569f0 msgr2=0x5592776a42c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:07:38.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.098+0000 7f3543d97000 1 -- 172.21.15.16:0/240933219 shutdown_connections 2024-03-31T15:07:38.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.098+0000 7f3543d97000 1 -- 172.21.15.16:0/240933219 wait complete. 2024-03-31T15:07:38.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.098+0000 7f3543d97000 20 client.6507 trim_cache size 0 max 16384 2024-03-31T15:07:38.098 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:07:38.099 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-03-31T15:07:38.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.098+0000 7f3540800640 1 -- 172.21.15.16:0/1100413067 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f351c068380 msgr2=0x7f351c06a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:38.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.098+0000 7f3540800640 1 --2- 172.21.15.16:0/1100413067 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f351c068380 0x7f351c06a840 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f35280ae000 tx=0x7f35280a7000 comp rx=0 tx=0).stop 2024-03-31T15:07:38.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.098+0000 7f3540800640 1 -- 172.21.15.16:0/1100413067 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f35381411b0 msgr2=0x7f35380b38e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:38.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.098+0000 7f3540800640 1 --2- 172.21.15.16:0/1100413067 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f35381411b0 0x7f35380b38e0 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7f352c00e840 tx=0x7f352c00ed00 comp rx=0 tx=0).stop 2024-03-31T15:07:38.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.102+0000 7f3540800640 1 -- 172.21.15.16:0/1100413067 shutdown_connections 2024-03-31T15:07:38.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.102+0000 7f3540800640 1 --2- 172.21.15.16:0/1100413067 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f351c068380 0x7f351c06a840 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:38.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.102+0000 7f3540800640 1 --2- 172.21.15.16:0/1100413067 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3538143c30 0x7f3538140450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:38.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.102+0000 7f3540800640 1 --2- 172.21.15.16:0/1100413067 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f3538141590 0x7f35380d2ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:38.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.102+0000 7f3540800640 1 --2- 172.21.15.16:0/1100413067 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f35381411b0 0x7f35380b38e0 unknown :-1 s=CLOSED pgs=725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:38.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.102+0000 7f3540800640 1 -- 172.21.15.16:0/1100413067 >> 172.21.15.16:0/1100413067 conn(0x7f3538067930 msgr2=0x7f35381426e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:07:38.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.102+0000 7f3540800640 1 -- 172.21.15.16:0/1100413067 shutdown_connections 2024-03-31T15:07:38.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.102+0000 7f3540800640 1 -- 172.21.15.16:0/1100413067 wait complete. 2024-03-31T15:07:38.116 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-03-31T15:07:38.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.258+0000 7ff2c4200640 1 Processor -- start 2024-03-31T15:07:38.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.258+0000 7ff2c4200640 1 -- start start 2024-03-31T15:07:38.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.262+0000 7ff2c4200640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff2bc147d60 0x7ff2bc0b3130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:38.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.262+0000 7ff2c4200640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7ff2bc073310 0x7ff2bc06d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:38.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.262+0000 7ff2c4200640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff2bc0737c0 con 0x7ff2bc06da10 2024-03-31T15:07:38.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.262+0000 7ff2c4200640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7ff2bc148140 con 0x7ff2bc147d60 2024-03-31T15:07:38.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.262+0000 7ff2c4200640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7ff2bc06de10 con 0x7ff2bc073310 2024-03-31T15:07:38.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.262+0000 7ff2c1800640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff2bc147d60 0x7ff2bc0b3130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:38.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.262+0000 7ff2c1800640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff2bc147d60 0x7ff2bc0b3130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47380/0 (socket says 172.21.15.16:47380) 2024-03-31T15:07:38.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.262+0000 7ff2c1800640 1 -- 172.21.15.16:0/711262190 learned_addr learned my addr 172.21.15.16:0/711262190 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:07:38.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.262+0000 7ff2c0e00640 1 --2- 172.21.15.16:0/711262190 >> v2:172.21.15.89:3300/0 conn(0x7ff2bc073310 0x7ff2bc06d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:38.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.262+0000 7ff2c1800640 1 -- 172.21.15.16:0/711262190 >> v2:172.21.15.89:3300/0 conn(0x7ff2bc073310 msgr2=0x7ff2bc06d4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:07:38.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.262+0000 7ff2c1800640 1 --2- 172.21.15.16:0/711262190 >> v2:172.21.15.89:3300/0 conn(0x7ff2bc073310 0x7ff2bc06d4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:38.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.262+0000 7ff2c1800640 1 -- 172.21.15.16:0/711262190 >> v1:172.21.15.89:6789/0 conn(0x7ff2bc06da10 legacy=0x7ff2bc161f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:38.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.262+0000 7ff2c1800640 1 -- 172.21.15.16:0/711262190 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2bc0b37c0 con 0x7ff2bc147d60 2024-03-31T15:07:38.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.266+0000 7ff2c1800640 1 --2- 172.21.15.16:0/711262190 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff2bc147d60 0x7ff2bc0b3130 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7ff2bc06e7b0 tx=0x7ff2b001c4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb6df25df0d0f69e server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:07:38.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.266+0000 7ff2bbe00640 1 -- 172.21.15.16:0/711262190 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2b00aa650 con 0x7ff2bc147d60 2024-03-31T15:07:38.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.266+0000 7ff2bbe00640 1 -- 172.21.15.16:0/711262190 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff2b00af070 con 0x7ff2bc147d60 2024-03-31T15:07:38.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.266+0000 7ff2bbe00640 1 -- 172.21.15.16:0/711262190 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2b00aaa10 con 0x7ff2bc147d60 2024-03-31T15:07:38.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.266+0000 7ff2c4200640 1 -- 172.21.15.16:0/711262190 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff2bc147d60 msgr2=0x7ff2bc0b3130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:38.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.266+0000 7ff2c4200640 1 --2- 172.21.15.16:0/711262190 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff2bc147d60 0x7ff2bc0b3130 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7ff2bc06e7b0 tx=0x7ff2b001c4f0 comp rx=0 tx=0).stop 2024-03-31T15:07:38.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.266+0000 7ff2c4200640 1 -- 172.21.15.16:0/711262190 shutdown_connections 2024-03-31T15:07:38.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.266+0000 7ff2c4200640 1 --2- 172.21.15.16:0/711262190 >> v2:172.21.15.89:3300/0 conn(0x7ff2bc073310 0x7ff2bc06d4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:38.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.266+0000 7ff2c4200640 1 --2- 172.21.15.16:0/711262190 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff2bc147d60 0x7ff2bc0b3130 unknown :-1 s=CLOSED pgs=730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:38.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.266+0000 7ff2c4200640 1 -- 172.21.15.16:0/711262190 >> 172.21.15.16:0/711262190 conn(0x7ff2bc06ee40 msgr2=0x7ff2bc06b620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:07:38.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.266+0000 7ff2c4200640 1 -- 172.21.15.16:0/711262190 shutdown_connections 2024-03-31T15:07:38.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.266+0000 7ff2c4200640 1 -- 172.21.15.16:0/711262190 wait complete. 2024-03-31T15:07:38.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.266+0000 7ff2c4200640 1 Processor -- start 2024-03-31T15:07:38.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.266+0000 7ff2c4200640 1 -- start start 2024-03-31T15:07:38.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.270+0000 7ff2c4200640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff2bc06da10 0x7ff2bc168c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:38.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.270+0000 7ff2c4200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff2bc073310 0x7ff2bc13f950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:38.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.270+0000 7ff2c4200640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff2bc147d60 0x7ff2bc0aeef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:38.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.270+0000 7ff2c4200640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7ff2bc16b2e0 con 0x7ff2bc073310 2024-03-31T15:07:38.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.270+0000 7ff2c4200640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7ff2bc0b4340 con 0x7ff2bc147d60 2024-03-31T15:07:38.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.270+0000 7ff2c4200640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7ff2bc16b460 con 0x7ff2bc06da10 2024-03-31T15:07:38.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.270+0000 7ff2c0e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff2bc073310 0x7ff2bc13f950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:38.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.270+0000 7ff2c0e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff2bc073310 0x7ff2bc13f950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47388/0 (socket says 172.21.15.16:47388) 2024-03-31T15:07:38.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.270+0000 7ff2c0e00640 1 -- 172.21.15.16:0/3568302703 learned_addr learned my addr 172.21.15.16:0/3568302703 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:07:38.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.270+0000 7ff2c1800640 1 --2- 172.21.15.16:0/3568302703 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff2bc06da10 0x7ff2bc168c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:38.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.270+0000 7ff2c2200640 1 --2- 172.21.15.16:0/3568302703 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff2bc147d60 0x7ff2bc0aeef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:38.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.270+0000 7ff2c0e00640 1 -- 172.21.15.16:0/3568302703 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff2bc06da10 msgr2=0x7ff2bc168c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:38.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.270+0000 7ff2c0e00640 1 --2- 172.21.15.16:0/3568302703 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff2bc06da10 0x7ff2bc168c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:38.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.270+0000 7ff2c0e00640 1 -- 172.21.15.16:0/3568302703 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff2bc147d60 msgr2=0x7ff2bc0aeef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:38.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.270+0000 7ff2c0e00640 1 --2- 172.21.15.16:0/3568302703 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff2bc147d60 0x7ff2bc0aeef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:38.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.270+0000 7ff2c0e00640 1 -- 172.21.15.16:0/3568302703 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2bc13fe90 con 0x7ff2bc073310 2024-03-31T15:07:38.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.270+0000 7ff2c0e00640 1 --2- 172.21.15.16:0/3568302703 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff2bc073310 0x7ff2bc13f950 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7ff2a800e930 tx=0x7ff2a800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:07:38.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.270+0000 7ff2b9600640 1 -- 172.21.15.16:0/3568302703 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2a8017070 con 0x7ff2bc073310 2024-03-31T15:07:38.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.270+0000 7ff2b9600640 1 -- 172.21.15.16:0/3568302703 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff2a800ca00 con 0x7ff2bc073310 2024-03-31T15:07:38.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.270+0000 7ff2c4200640 1 -- 172.21.15.16:0/3568302703 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff2bc160cb0 con 0x7ff2bc073310 2024-03-31T15:07:38.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.270+0000 7ff2b9600640 1 -- 172.21.15.16:0/3568302703 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2a800cdb0 con 0x7ff2bc073310 2024-03-31T15:07:38.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.270+0000 7ff2c4200640 1 -- 172.21.15.16:0/3568302703 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff2bc161270 con 0x7ff2bc073310 2024-03-31T15:07:38.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.270+0000 7ff2c4200640 1 -- 172.21.15.16:0/3568302703 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff2bc069950 con 0x7ff2bc073310 2024-03-31T15:07:38.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.270+0000 7ff2b9600640 1 -- 172.21.15.16:0/3568302703 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7ff2a801a070 con 0x7ff2bc073310 2024-03-31T15:07:38.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.274+0000 7ff2b9600640 1 --2- 172.21.15.16:0/3568302703 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7ff29c068380 0x7ff29c06a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:38.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.274+0000 7ff2b9600640 1 -- 172.21.15.16:0/3568302703 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 5709+0+0 (secure 0 0 0) 0x7ff2a80869c0 con 0x7ff2bc073310 2024-03-31T15:07:38.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.274+0000 7ff2c1800640 1 --2- 172.21.15.16:0/3568302703 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7ff29c068380 0x7ff29c06a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:38.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.274+0000 7ff2c1800640 1 --2- 172.21.15.16:0/3568302703 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7ff29c068380 0x7ff29c06a840 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7ff2b00ae000 tx=0x7ff2b00a7000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:07:38.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.278+0000 7ff2b9600640 1 -- 172.21.15.16:0/3568302703 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7ff2a8056950 con 0x7ff2bc073310 2024-03-31T15:07:38.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.482+0000 7ff2c4200640 1 -- 172.21.15.16:0/3568302703 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7ff2bc146f00 con 0x7ff2bc073310 2024-03-31T15:07:38.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.482+0000 7ff2b9600640 1 -- 172.21.15.16:0/3568302703 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+46 (secure 0 0 0) 0x7ff2a805ac50 con 0x7ff2bc073310 2024-03-31T15:07:38.478 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:07:38.478 INFO:teuthology.orchestra.run.smithi016.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-03-31T15:07:38.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.486+0000 7ff2c4200640 1 -- 172.21.15.16:0/3568302703 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7ff29c068380 msgr2=0x7ff29c06a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:38.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.486+0000 7ff2c4200640 1 --2- 172.21.15.16:0/3568302703 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7ff29c068380 0x7ff29c06a840 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7ff2b00ae000 tx=0x7ff2b00a7000 comp rx=0 tx=0).stop 2024-03-31T15:07:38.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.486+0000 7ff2c4200640 1 -- 172.21.15.16:0/3568302703 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff2bc073310 msgr2=0x7ff2bc13f950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:38.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.486+0000 7ff2c4200640 1 --2- 172.21.15.16:0/3568302703 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff2bc073310 0x7ff2bc13f950 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7ff2a800e930 tx=0x7ff2a800edf0 comp rx=0 tx=0).stop 2024-03-31T15:07:38.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.486+0000 7ff2c4200640 1 -- 172.21.15.16:0/3568302703 shutdown_connections 2024-03-31T15:07:38.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.486+0000 7ff2c4200640 1 --2- 172.21.15.16:0/3568302703 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7ff29c068380 0x7ff29c06a840 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:38.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.486+0000 7ff2c4200640 1 --2- 172.21.15.16:0/3568302703 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff2bc147d60 0x7ff2bc0aeef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:38.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.486+0000 7ff2c4200640 1 --2- 172.21.15.16:0/3568302703 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff2bc073310 0x7ff2bc13f950 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:38.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.486+0000 7ff2c4200640 1 --2- 172.21.15.16:0/3568302703 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff2bc06da10 0x7ff2bc168c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:38.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.486+0000 7ff2c4200640 1 -- 172.21.15.16:0/3568302703 >> 172.21.15.16:0/3568302703 conn(0x7ff2bc06ee40 msgr2=0x7ff2bc0744f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:07:38.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.486+0000 7ff2c4200640 1 -- 172.21.15.16:0/3568302703 shutdown_connections 2024-03-31T15:07:38.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.486+0000 7ff2c4200640 1 -- 172.21.15.16:0/3568302703 wait complete. 2024-03-31T15:07:38.499 DEBUG:tasks.ceph_test_case:wait_until_true: success in 0s and 0 retries 2024-03-31T15:07:38.500 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-31T15:07:38.500 DEBUG:teuthology.orchestra.run.smithi016:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-31T15:07:38.566 DEBUG:teuthology.orchestra.run.smithi089:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-31T15:07:38.645 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-31T15:07:38.645 INFO:tasks.cephfs.mount:Terminating background process 2024-03-31T15:07:38.646 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-03-31T15:07:38.646 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-31T15:07:38.656 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-31T15:07:38.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.842+0000 7faff1800640 1 Processor -- start 2024-03-31T15:07:38.839 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.842+0000 7faff1800640 1 -- start start 2024-03-31T15:07:38.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.846+0000 7faff1800640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fafec06dbc0 0x7fafec15f650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:38.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.846+0000 7faff1800640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fafec0cb440 0x7fafec06f920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:38.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.846+0000 7faff1800640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fafec0cb8c0 0x7fafec06fe60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:38.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.846+0000 7faff1800640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7fafec15fb90 con 0x7fafec0cb8c0 2024-03-31T15:07:38.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.846+0000 7faff1800640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7fafec15f1c0 con 0x7fafec0cb440 2024-03-31T15:07:38.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.846+0000 7faff1800640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fafec0cbe90 con 0x7fafec06dbc0 2024-03-31T15:07:38.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.846+0000 7fafeb400640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fafec0cb8c0 0x7fafec06fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:38.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.846+0000 7fafeb400640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fafec0cb8c0 0x7fafec06fe60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47390/0 (socket says 172.21.15.16:47390) 2024-03-31T15:07:38.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.846+0000 7fafeb400640 1 -- 172.21.15.16:0/2926072189 learned_addr learned my addr 172.21.15.16:0/2926072189 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:07:38.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.846+0000 7fafeb400640 1 -- 172.21.15.16:0/2926072189 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fafec06dbc0 msgr2=0x7fafec15f650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:07:38.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.846+0000 7fafeb400640 1 --2- 172.21.15.16:0/2926072189 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fafec06dbc0 0x7fafec15f650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:38.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.846+0000 7fafeb400640 1 -- 172.21.15.16:0/2926072189 >> v2:172.21.15.89:3300/0 conn(0x7fafec0cb440 msgr2=0x7fafec06f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:07:38.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.846+0000 7fafe2000640 1 --2- 172.21.15.16:0/2926072189 >> v2:172.21.15.89:3300/0 conn(0x7fafec0cb440 0x7fafec06f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:38.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.846+0000 7fafeb400640 1 --2- 172.21.15.16:0/2926072189 >> v2:172.21.15.89:3300/0 conn(0x7fafec0cb440 0x7fafec06f920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:38.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.846+0000 7fafeb400640 1 -- 172.21.15.16:0/2926072189 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fafec0cc170 con 0x7fafec0cb8c0 2024-03-31T15:07:38.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.846+0000 7fafe2000640 1 --2- 172.21.15.16:0/2926072189 >> v2:172.21.15.89:3300/0 conn(0x7fafec0cb440 0x7fafec06f920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:07:38.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.846+0000 7fafeb400640 1 --2- 172.21.15.16:0/2926072189 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fafec0cb8c0 0x7fafec06fe60 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7fafec0d31b0 tx=0x7fafe401c460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d44052bfc4f395b5 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:07:38.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.850+0000 7fafea000640 1 -- 172.21.15.16:0/2926072189 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafe40b1070 con 0x7fafec0cb8c0 2024-03-31T15:07:38.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.850+0000 7fafea000640 1 -- 172.21.15.16:0/2926072189 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fafe40a8050 con 0x7fafec0cb8c0 2024-03-31T15:07:38.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.850+0000 7fafea000640 1 -- 172.21.15.16:0/2926072189 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafe40ac860 con 0x7fafec0cb8c0 2024-03-31T15:07:38.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.850+0000 7faff1800640 1 -- 172.21.15.16:0/2926072189 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fafec0cb8c0 msgr2=0x7fafec06fe60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:38.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.850+0000 7faff1800640 1 --2- 172.21.15.16:0/2926072189 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fafec0cb8c0 0x7fafec06fe60 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7fafec0d31b0 tx=0x7fafe401c460 comp rx=0 tx=0).stop 2024-03-31T15:07:38.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.850+0000 7faff1800640 1 -- 172.21.15.16:0/2926072189 shutdown_connections 2024-03-31T15:07:38.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.850+0000 7faff1800640 1 --2- 172.21.15.16:0/2926072189 >> v2:172.21.15.89:3300/0 conn(0x7fafec0cb440 0x7fafec06f920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:38.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.850+0000 7faff1800640 1 --2- 172.21.15.16:0/2926072189 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fafec0cb8c0 0x7fafec06fe60 unknown :-1 s=CLOSED pgs=732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:38.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.850+0000 7faff1800640 1 --2- 172.21.15.16:0/2926072189 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fafec06dbc0 0x7fafec15f650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:38.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.850+0000 7faff1800640 1 -- 172.21.15.16:0/2926072189 >> 172.21.15.16:0/2926072189 conn(0x7fafec067330 msgr2=0x7fafec067740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:07:38.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.850+0000 7faff1800640 1 -- 172.21.15.16:0/2926072189 shutdown_connections 2024-03-31T15:07:38.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.850+0000 7faff1800640 1 -- 172.21.15.16:0/2926072189 wait complete. 2024-03-31T15:07:38.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.850+0000 7faff1800640 1 Processor -- start 2024-03-31T15:07:38.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.850+0000 7faff1800640 1 -- start start 2024-03-31T15:07:38.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.850+0000 7faff1800640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fafec06dbc0 0x7fafec13d890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:38.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.850+0000 7faff1800640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fafec0cb440 0x7fafec1581b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:38.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.850+0000 7faff1800640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fafec0cb8c0 0x7fafec1404f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:38.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.850+0000 7faff1800640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fafec06cad0 con 0x7fafec06dbc0 2024-03-31T15:07:38.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.850+0000 7faff1800640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fafec06c950 con 0x7fafec0cb8c0 2024-03-31T15:07:38.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.850+0000 7faff1800640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fafec162990 con 0x7fafec0cb440 2024-03-31T15:07:38.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.850+0000 7fafeaa00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fafec06dbc0 0x7fafec13d890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:38.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.850+0000 7fafeaa00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fafec06dbc0 0x7fafec13d890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47392/0 (socket says 172.21.15.16:47392) 2024-03-31T15:07:38.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.850+0000 7fafeaa00640 1 -- 172.21.15.16:0/1720545279 learned_addr learned my addr 172.21.15.16:0/1720545279 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:07:38.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.850+0000 7fafe2000640 1 --2- 172.21.15.16:0/1720545279 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fafec0cb440 0x7fafec1581b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:38.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.850+0000 7fafeb400640 1 --2- 172.21.15.16:0/1720545279 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fafec0cb8c0 0x7fafec1404f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:38.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.854+0000 7fafeaa00640 1 -- 172.21.15.16:0/1720545279 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fafec0cb440 msgr2=0x7fafec1581b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:38.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.854+0000 7fafeaa00640 1 --2- 172.21.15.16:0/1720545279 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fafec0cb440 0x7fafec1581b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:38.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.854+0000 7fafeaa00640 1 -- 172.21.15.16:0/1720545279 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fafec0cb8c0 msgr2=0x7fafec1404f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:38.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.854+0000 7fafeaa00640 1 --2- 172.21.15.16:0/1720545279 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fafec0cb8c0 0x7fafec1404f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:38.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.854+0000 7fafeaa00640 1 -- 172.21.15.16:0/1720545279 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fafec13df70 con 0x7fafec06dbc0 2024-03-31T15:07:38.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.854+0000 7fafeb400640 1 --2- 172.21.15.16:0/1720545279 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fafec0cb8c0 0x7fafec1404f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:07:38.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.854+0000 7fafe2000640 1 --2- 172.21.15.16:0/1720545279 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fafec0cb440 0x7fafec1581b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:07:38.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.854+0000 7fafeaa00640 1 --2- 172.21.15.16:0/1720545279 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fafec06dbc0 0x7fafec13d890 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7fafdc00ba50 tx=0x7fafdc00bf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:07:38.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.854+0000 7fafe3400640 1 -- 172.21.15.16:0/1720545279 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafdc019070 con 0x7fafec06dbc0 2024-03-31T15:07:38.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.854+0000 7fafe3400640 1 -- 172.21.15.16:0/1720545279 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fafdc00cc00 con 0x7fafec06dbc0 2024-03-31T15:07:38.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.854+0000 7faff1800640 1 -- 172.21.15.16:0/1720545279 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fafec13c6b0 con 0x7fafec06dbc0 2024-03-31T15:07:38.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.854+0000 7fafe3400640 1 -- 172.21.15.16:0/1720545279 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafdc014440 con 0x7fafec06dbc0 2024-03-31T15:07:38.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.854+0000 7faff1800640 1 -- 172.21.15.16:0/1720545279 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fafec13cc70 con 0x7fafec06dbc0 2024-03-31T15:07:38.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.854+0000 7faff1800640 1 -- 172.21.15.16:0/1720545279 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fafec069950 con 0x7fafec06dbc0 2024-03-31T15:07:38.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.854+0000 7fafe3400640 1 -- 172.21.15.16:0/1720545279 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fafdc01c070 con 0x7fafec06dbc0 2024-03-31T15:07:38.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.854+0000 7fafe3400640 1 --2- 172.21.15.16:0/1720545279 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fafc8068380 0x7fafc806a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:38.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.854+0000 7fafe3400640 1 -- 172.21.15.16:0/1720545279 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 5709+0+0 (secure 0 0 0) 0x7fafdc086760 con 0x7fafec06dbc0 2024-03-31T15:07:38.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.854+0000 7fafe2000640 1 --2- 172.21.15.16:0/1720545279 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fafc8068380 0x7fafc806a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:38.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.858+0000 7fafe2000640 1 --2- 172.21.15.16:0/1720545279 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fafc8068380 0x7fafc806a840 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fafd4006fd0 tx=0x7fafd4008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:07:38.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:38.858+0000 7fafe3400640 1 -- 172.21.15.16:0/1720545279 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fafdc056d10 con 0x7fafec06dbc0 2024-03-31T15:07:39.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:39.166+0000 7faff1800640 1 -- 172.21.15.16:0/1720545279 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fafec162d50 con 0x7fafec06dbc0 2024-03-31T15:07:39.164 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:39.170+0000 7fafe3400640 1 -- 172.21.15.16:0/1720545279 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v56) ==== 74+0+16593 (secure 0 0 0) 0x7fafdc056610 con 0x7fafec06dbc0 2024-03-31T15:07:39.165 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:07:39.165 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":56,"fsid":"286eb5da-fb47-4550-b290-d9519c13b9a5","created":"2024-03-31T15:03:11.438952+0000","modified":"2024-03-31T15:06:27.274379+0000","last_up_change":"2024-03-31T15:03:22.885535+0000","last_in_change":"2024-03-31T15:03:13.941667+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-31T15:03:22.550787+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-03-31T15:06:24.043886+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"55","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-03-31T15:06:24.613108+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"56","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"92a66450-ef19-40d8-ac3d-d760b23b0bda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6801","nonce":932712244}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6803","nonce":932712244}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6807","nonce":932712244}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6805","nonce":932712244}]},"public_addr":"172.21.15.16:6801/932712244","cluster_addr":"172.21.15.16:6803/932712244","heartbeat_back_addr":"172.21.15.16:6807/932712244","heartbeat_front_addr":"172.21.15.16:6805/932712244","state":["exists","up"]},{"osd":1,"uuid":"da9d1f2b-1968-4959-b843-faa231988e92","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6810","nonce":1398737533}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6811","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6813","nonce":1398737533}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6821","nonce":1398737533}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6815","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6816","nonce":1398737533}]},"public_addr":"172.21.15.16:6810/1398737533","cluster_addr":"172.21.15.16:6813/1398737533","heartbeat_back_addr":"172.21.15.16:6821/1398737533","heartbeat_front_addr":"172.21.15.16:6816/1398737533","state":["exists","up"]},{"osd":2,"uuid":"01a73662-9a21-4b1f-bcbb-42b9636a4638","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6819","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6822","nonce":3018526646}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6825","nonce":3018526646}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6831","nonce":3018526646}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6827","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6829","nonce":3018526646}]},"public_addr":"172.21.15.16:6822/3018526646","cluster_addr":"172.21.15.16:6825/3018526646","heartbeat_back_addr":"172.21.15.16:6831/3018526646","heartbeat_front_addr":"172.21.15.16:6829/3018526646","state":["exists","up"]},{"osd":3,"uuid":"c7771fd3-e18e-4500-ac38-23cf10ebcd75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6809","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6812","nonce":3360352974}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6817","nonce":3360352974}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6828","nonce":3360352974}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6823","nonce":3360352974}]},"public_addr":"172.21.15.16:6812/3360352974","cluster_addr":"172.21.15.16:6817/3360352974","heartbeat_back_addr":"172.21.15.16:6828/3360352974","heartbeat_front_addr":"172.21.15.16:6823/3360352974","state":["exists","up"]},{"osd":4,"uuid":"88260730-1482-4fbd-a40f-cb1124e9df4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6801","nonce":1856844856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6803","nonce":1856844856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6807","nonce":1856844856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6805","nonce":1856844856}]},"public_addr":"172.21.15.89:6801/1856844856","cluster_addr":"172.21.15.89:6803/1856844856","heartbeat_back_addr":"172.21.15.89:6807/1856844856","heartbeat_front_addr":"172.21.15.89:6805/1856844856","state":["exists","up"]},{"osd":5,"uuid":"4cb2c00c-fae0-4dd8-9d27-ec4f0093ccf7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6814","nonce":50395527}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6817","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6820","nonce":50395527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6831","nonce":50395527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6823","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6826","nonce":50395527}]},"public_addr":"172.21.15.89:6814/50395527","cluster_addr":"172.21.15.89:6820/50395527","heartbeat_back_addr":"172.21.15.89:6831/50395527","heartbeat_front_addr":"172.21.15.89:6826/50395527","state":["exists","up"]},{"osd":6,"uuid":"81ce0c19-2554-4cae-acbe-66a05e9e9b9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6809","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6812","nonce":3470471752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6815","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6818","nonce":3470471752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6827","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6829","nonce":3470471752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6821","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6824","nonce":3470471752}]},"public_addr":"172.21.15.89:6812/3470471752","cluster_addr":"172.21.15.89:6818/3470471752","heartbeat_back_addr":"172.21.15.89:6829/3470471752","heartbeat_front_addr":"172.21.15.89:6824/3470471752","state":["exists","up"]},{"osd":7,"uuid":"85e4313b-d37d-442e-8042-153bf20a847d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6811","nonce":1796778630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6813","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6816","nonce":1796778630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6825","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6828","nonce":1796778630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6819","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6822","nonce":1796778630}]},"public_addr":"172.21.15.89:6811/1796778630","cluster_addr":"172.21.15.89:6816/1796778630","heartbeat_back_addr":"172.21.15.89:6828/1796778630","heartbeat_front_addr":"172.21.15.89:6822/1796778630","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.439088+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.529970+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.680512+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:21.044161+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.582111+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.536558+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.409522+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.590229+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-31T15:07:39.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:39.170+0000 7faff1800640 1 -- 172.21.15.16:0/1720545279 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fafc8068380 msgr2=0x7fafc806a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:39.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:39.170+0000 7faff1800640 1 --2- 172.21.15.16:0/1720545279 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fafc8068380 0x7fafc806a840 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fafd4006fd0 tx=0x7fafd4008040 comp rx=0 tx=0).stop 2024-03-31T15:07:39.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:39.170+0000 7faff1800640 1 -- 172.21.15.16:0/1720545279 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fafec06dbc0 msgr2=0x7fafec13d890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:39.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:39.170+0000 7faff1800640 1 --2- 172.21.15.16:0/1720545279 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fafec06dbc0 0x7fafec13d890 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7fafdc00ba50 tx=0x7fafdc00bf10 comp rx=0 tx=0).stop 2024-03-31T15:07:39.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:39.170+0000 7faff1800640 1 -- 172.21.15.16:0/1720545279 shutdown_connections 2024-03-31T15:07:39.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:39.170+0000 7faff1800640 1 --2- 172.21.15.16:0/1720545279 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fafc8068380 0x7fafc806a840 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:39.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:39.170+0000 7faff1800640 1 --2- 172.21.15.16:0/1720545279 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fafec0cb8c0 0x7fafec1404f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:39.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:39.170+0000 7faff1800640 1 --2- 172.21.15.16:0/1720545279 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fafec0cb440 0x7fafec1581b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:39.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:39.170+0000 7faff1800640 1 --2- 172.21.15.16:0/1720545279 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fafec06dbc0 0x7fafec13d890 unknown :-1 s=CLOSED pgs=733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:39.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:39.170+0000 7faff1800640 1 -- 172.21.15.16:0/1720545279 >> 172.21.15.16:0/1720545279 conn(0x7fafec067330 msgr2=0x7fafec06f920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:07:39.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:39.170+0000 7faff1800640 1 -- 172.21.15.16:0/1720545279 shutdown_connections 2024-03-31T15:07:39.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:39.170+0000 7faff1800640 1 -- 172.21.15.16:0/1720545279 wait complete. 2024-03-31T15:07:39.187 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-31T15:03:22.550787+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 8, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-31T15:06:24.043886+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '55', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 9, 'pool_name': 'cephfs_data', 'create_time': '2024-03-31T15:06:24.613108+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '56', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-31T15:07:39.187 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-31T15:07:39.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:39.334+0000 7efefd800640 1 Processor -- start 2024-03-31T15:07:39.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:39.338+0000 7efefd800640 1 -- start start 2024-03-31T15:07:39.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:39.338+0000 7efefd800640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7efef80d0a60 0x7efef80cea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:39.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:39.338+0000 7efefd800640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7efef80cef80 0x7efef81419c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:39.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:39.338+0000 7efefd800640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efef815ca70 con 0x7efef806f920 2024-03-31T15:07:39.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:39.338+0000 7efefd800640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7efef813f670 con 0x7efef80cef80 2024-03-31T15:07:39.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:39.338+0000 7efefd800640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7efef815cd50 con 0x7efef80d0a60 2024-03-31T15:07:39.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:39.338+0000 7efef7400640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7efef80cef80 0x7efef81419c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:39.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:39.338+0000 7efef6a00640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7efef806f920 0x7efef806fd00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:53762/0 (socket says 172.21.15.16:53762) 2024-03-31T15:07:39.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:39.338+0000 7efef6a00640 1 -- 172.21.15.16:0/2691038118 learned_addr learned my addr 172.21.15.16:0/2691038118 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:07:39.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:39.338+0000 7efef6000640 1 --2- 172.21.15.16:0/2691038118 >> v2:172.21.15.89:3300/0 conn(0x7efef80d0a60 0x7efef80cea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:39.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:39.338+0000 7efef7400640 1 -- 172.21.15.16:0/2691038118 >> v2:172.21.15.89:3300/0 conn(0x7efef80d0a60 msgr2=0x7efef80cea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:07:39.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:39.338+0000 7efef7400640 1 --2- 172.21.15.16:0/2691038118 >> v2:172.21.15.89:3300/0 conn(0x7efef80d0a60 0x7efef80cea40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:39.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:39.338+0000 7efef7400640 1 -- 172.21.15.16:0/2691038118 >> v1:172.21.15.16:6789/0 conn(0x7efef806f920 legacy=0x7efef806fd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:39.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:39.338+0000 7efef7400640 1 -- 172.21.15.16:0/2691038118 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efef815d030 con 0x7efef80cef80 2024-03-31T15:07:39.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:39.338+0000 7efef5600640 1 -- 172.21.15.16:0/2691038118 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1065034789 0 0) 0x7efef815ca70 con 0x7efef806f920 2024-03-31T15:07:39.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:39.342+0000 7efef7400640 1 --2- 172.21.15.16:0/2691038118 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efef80cef80 0x7efef81419c0 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7efef80b5070 tx=0x7efef001c930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7650d0b10e28a7c5 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:07:39.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:39.342+0000 7efef5600640 1 -- 172.21.15.16:0/2691038118 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efef00b2070 con 0x7efef80cef80 2024-03-31T15:07:39.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:39.342+0000 7efef5600640 1 -- 172.21.15.16:0/2691038118 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efef00a57f0 con 0x7efef80cef80 2024-03-31T15:07:39.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:39.342+0000 7efef5600640 1 -- 172.21.15.16:0/2691038118 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efef00a5bc0 con 0x7efef80cef80 2024-03-31T15:07:39.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:39.342+0000 7efefd800640 1 -- 172.21.15.16:0/2691038118 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efef80cef80 msgr2=0x7efef81419c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:39.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:39.342+0000 7efefd800640 1 --2- 172.21.15.16:0/2691038118 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efef80cef80 0x7efef81419c0 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7efef80b5070 tx=0x7efef001c930 comp rx=0 tx=0).stop 2024-03-31T15:07:39.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:39.342+0000 7efefd800640 1 -- 172.21.15.16:0/2691038118 shutdown_connections 2024-03-31T15:07:39.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:39.342+0000 7efefd800640 1 --2- 172.21.15.16:0/2691038118 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efef80cef80 0x7efef81419c0 unknown :-1 s=CLOSED pgs=735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:39.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:39.342+0000 7efefd800640 1 --2- 172.21.15.16:0/2691038118 >> v2:172.21.15.89:3300/0 conn(0x7efef80d0a60 0x7efef80cea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:39.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:39.342+0000 7efefd800640 1 -- 172.21.15.16:0/2691038118 >> 172.21.15.16:0/2691038118 conn(0x7efef8067930 msgr2=0x7efef80b3ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:07:39.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:39.342+0000 7efefd800640 1 -- 172.21.15.16:0/2691038118 shutdown_connections 2024-03-31T15:07:39.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:39.342+0000 7efefd800640 1 -- 172.21.15.16:0/2691038118 wait complete. 2024-03-31T15:07:39.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:39.346+0000 7efefd800640 1 Processor -- start 2024-03-31T15:07:39.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:39.346+0000 7efefd800640 1 -- start start 2024-03-31T15:07:39.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:39.346+0000 7efefd800640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efef806f920 0x7efef813ec20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:39.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:39.346+0000 7efefd800640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7efef80cef80 0x7efef815e040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:39.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:39.346+0000 7efefd800640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efef80d0a60 0x7efef815fbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:39.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:39.346+0000 7efefd800640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7efef8169230 con 0x7efef806f920 2024-03-31T15:07:39.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:39.346+0000 7efefd800640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7efef81690b0 con 0x7efef80d0a60 2024-03-31T15:07:39.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:39.346+0000 7efefd800640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7efef81693b0 con 0x7efef80cef80 2024-03-31T15:07:39.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:39.346+0000 7efef6a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efef806f920 0x7efef813ec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:39.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:39.346+0000 7efef6a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efef806f920 0x7efef813ec20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47410/0 (socket says 172.21.15.16:47410) 2024-03-31T15:07:39.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:39.346+0000 7efef6a00640 1 -- 172.21.15.16:0/1410625993 learned_addr learned my addr 172.21.15.16:0/1410625993 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:07:39.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:39.346+0000 7efef6a00640 1 -- 172.21.15.16:0/1410625993 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7efef80cef80 msgr2=0x7efef815e040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:39.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:39.346+0000 7efef6000640 1 --2- 172.21.15.16:0/1410625993 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7efef80cef80 0x7efef815e040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:39.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:39.346+0000 7efef7400640 1 --2- 172.21.15.16:0/1410625993 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efef80d0a60 0x7efef815fbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:39.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:39.346+0000 7efef6a00640 1 --2- 172.21.15.16:0/1410625993 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7efef80cef80 0x7efef815e040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:39.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:39.346+0000 7efef6a00640 1 -- 172.21.15.16:0/1410625993 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efef80d0a60 msgr2=0x7efef815fbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:39.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:39.346+0000 7efef6a00640 1 --2- 172.21.15.16:0/1410625993 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efef80d0a60 0x7efef815fbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:39.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:39.346+0000 7efef6a00640 1 -- 172.21.15.16:0/1410625993 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efef80b1a90 con 0x7efef806f920 2024-03-31T15:07:39.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:39.346+0000 7efef6000640 1 --2- 172.21.15.16:0/1410625993 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7efef80cef80 0x7efef815e040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:07:39.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:39.346+0000 7efef7400640 1 --2- 172.21.15.16:0/1410625993 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efef80d0a60 0x7efef815fbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:07:39.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:39.346+0000 7efef6a00640 1 --2- 172.21.15.16:0/1410625993 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efef806f920 0x7efef813ec20 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7efee800c520 tx=0x7efee800c9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:07:39.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:39.346+0000 7efeeea00640 1 -- 172.21.15.16:0/1410625993 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efee8018070 con 0x7efef806f920 2024-03-31T15:07:39.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:39.346+0000 7efeeea00640 1 -- 172.21.15.16:0/1410625993 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efee80074e0 con 0x7efef806f920 2024-03-31T15:07:39.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:39.346+0000 7efeeea00640 1 -- 172.21.15.16:0/1410625993 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efee80049f0 con 0x7efef806f920 2024-03-31T15:07:39.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:39.346+0000 7efefd800640 1 -- 172.21.15.16:0/1410625993 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efef80b1d70 con 0x7efef806f920 2024-03-31T15:07:39.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:39.346+0000 7efefd800640 1 -- 172.21.15.16:0/1410625993 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efef80b2330 con 0x7efef806f920 2024-03-31T15:07:39.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:39.350+0000 7efefd800640 1 -- 172.21.15.16:0/1410625993 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efef8069950 con 0x7efef806f920 2024-03-31T15:07:39.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:39.350+0000 7efeeea00640 1 -- 172.21.15.16:0/1410625993 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7efee801b070 con 0x7efef806f920 2024-03-31T15:07:39.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:39.350+0000 7efeeea00640 1 --2- 172.21.15.16:0/1410625993 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7efed8068380 0x7efed806a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:39.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:39.350+0000 7efeeea00640 1 -- 172.21.15.16:0/1410625993 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 5709+0+0 (secure 0 0 0) 0x7efee8085da0 con 0x7efef806f920 2024-03-31T15:07:39.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:39.350+0000 7efef6000640 1 --2- 172.21.15.16:0/1410625993 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7efed8068380 0x7efed806a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:39.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:39.350+0000 7efef6000640 1 --2- 172.21.15.16:0/1410625993 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7efed8068380 0x7efed806a840 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7efef806e790 tx=0x7efee4008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:07:39.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:39.354+0000 7efeeea00640 1 -- 172.21.15.16:0/1410625993 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7efee80563f0 con 0x7efef806f920 2024-03-31T15:07:39.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:39.538+0000 7efefd800640 1 -- 172.21.15.16:0/1410625993 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7efef80b4a20 con 0x7efef806f920 2024-03-31T15:07:39.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:39.538+0000 7efeeea00640 1 -- 172.21.15.16:0/1410625993 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v56) ==== 93+0+10 (secure 0 0 0) 0x7efee8055cf0 con 0x7efef806f920 2024-03-31T15:07:39.535 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2024-03-31T15:07:39.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:39.542+0000 7efefd800640 1 -- 172.21.15.16:0/1410625993 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7efed8068380 msgr2=0x7efed806a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:39.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:39.542+0000 7efefd800640 1 --2- 172.21.15.16:0/1410625993 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7efed8068380 0x7efed806a840 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7efef806e790 tx=0x7efee4008040 comp rx=0 tx=0).stop 2024-03-31T15:07:39.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:39.542+0000 7efefd800640 1 -- 172.21.15.16:0/1410625993 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efef806f920 msgr2=0x7efef813ec20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:39.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:39.542+0000 7efefd800640 1 --2- 172.21.15.16:0/1410625993 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efef806f920 0x7efef813ec20 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7efee800c520 tx=0x7efee800c9e0 comp rx=0 tx=0).stop 2024-03-31T15:07:39.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:39.542+0000 7efefd800640 1 -- 172.21.15.16:0/1410625993 shutdown_connections 2024-03-31T15:07:39.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:39.542+0000 7efefd800640 1 --2- 172.21.15.16:0/1410625993 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7efed8068380 0x7efed806a840 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:39.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:39.542+0000 7efefd800640 1 --2- 172.21.15.16:0/1410625993 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efef80d0a60 0x7efef815fbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:39.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:39.542+0000 7efefd800640 1 --2- 172.21.15.16:0/1410625993 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7efef80cef80 0x7efef815e040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:39.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:39.542+0000 7efefd800640 1 --2- 172.21.15.16:0/1410625993 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efef806f920 0x7efef813ec20 unknown :-1 s=CLOSED pgs=736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:39.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:39.542+0000 7efefd800640 1 -- 172.21.15.16:0/1410625993 >> 172.21.15.16:0/1410625993 conn(0x7efef8067930 msgr2=0x7efef80b5300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:07:39.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:39.542+0000 7efefd800640 1 -- 172.21.15.16:0/1410625993 shutdown_connections 2024-03-31T15:07:39.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:39.542+0000 7efefd800640 1 -- 172.21.15.16:0/1410625993 wait complete. 2024-03-31T15:07:39.557 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-31T15:07:39.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:39.706+0000 7ff6fa400640 1 Processor -- start 2024-03-31T15:07:39.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:39.706+0000 7ff6fa400640 1 -- start start 2024-03-31T15:07:39.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:39.706+0000 7ff6fa400640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7ff6f413dff0 0x7ff6f406d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:39.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:39.706+0000 7ff6fa400640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff6f413e3d0 0x7ff6f406da10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:39.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:39.706+0000 7ff6fa400640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff6f415fbc0 con 0x7ff6f40cc9d0 2024-03-31T15:07:39.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:39.706+0000 7ff6fa400640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7ff6f415f730 con 0x7ff6f413e3d0 2024-03-31T15:07:39.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:39.706+0000 7ff6fa400640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7ff6f415ee00 con 0x7ff6f413dff0 2024-03-31T15:07:39.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:39.706+0000 7ff6eaa00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff6f413e3d0 0x7ff6f406da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:39.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:39.706+0000 7ff6f3e00640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7ff6f40cc9d0 0x7ff6f40cce60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:53774/0 (socket says 172.21.15.16:53774) 2024-03-31T15:07:39.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:39.706+0000 7ff6f3e00640 1 -- 172.21.15.16:0/2502246605 learned_addr learned my addr 172.21.15.16:0/2502246605 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:07:39.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:39.706+0000 7ff6f3400640 1 --2- 172.21.15.16:0/2502246605 >> v2:172.21.15.89:3300/0 conn(0x7ff6f413dff0 0x7ff6f406d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:39.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:39.706+0000 7ff6eaa00640 1 -- 172.21.15.16:0/2502246605 >> v2:172.21.15.89:3300/0 conn(0x7ff6f413dff0 msgr2=0x7ff6f406d4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:07:39.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:39.706+0000 7ff6eaa00640 1 --2- 172.21.15.16:0/2502246605 >> v2:172.21.15.89:3300/0 conn(0x7ff6f413dff0 0x7ff6f406d4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:39.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:39.706+0000 7ff6eaa00640 1 -- 172.21.15.16:0/2502246605 >> v1:172.21.15.16:6789/0 conn(0x7ff6f40cc9d0 legacy=0x7ff6f40cce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:39.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:39.706+0000 7ff6eaa00640 1 -- 172.21.15.16:0/2502246605 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6f415f0e0 con 0x7ff6f413e3d0 2024-03-31T15:07:39.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:39.706+0000 7ff6f3400640 1 --2- 172.21.15.16:0/2502246605 >> v2:172.21.15.89:3300/0 conn(0x7ff6f413dff0 0x7ff6f406d4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:07:39.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:39.710+0000 7ff6eaa00640 1 --2- 172.21.15.16:0/2502246605 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff6f413e3d0 0x7ff6f406da10 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7ff6f406fa10 tx=0x7ff6dc01c3f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37243fb4fd14c52e server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:07:39.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:39.710+0000 7ff6f2a00640 1 -- 172.21.15.16:0/2502246605 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6dc0b0070 con 0x7ff6f413e3d0 2024-03-31T15:07:39.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:39.710+0000 7ff6f2a00640 1 -- 172.21.15.16:0/2502246605 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff6dc0ab600 con 0x7ff6f413e3d0 2024-03-31T15:07:39.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:39.710+0000 7ff6f2a00640 1 -- 172.21.15.16:0/2502246605 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6dc0ab9d0 con 0x7ff6f413e3d0 2024-03-31T15:07:39.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:39.710+0000 7ff6fa400640 1 -- 172.21.15.16:0/2502246605 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff6f413e3d0 msgr2=0x7ff6f406da10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:39.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:39.710+0000 7ff6fa400640 1 --2- 172.21.15.16:0/2502246605 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff6f413e3d0 0x7ff6f406da10 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7ff6f406fa10 tx=0x7ff6dc01c3f0 comp rx=0 tx=0).stop 2024-03-31T15:07:39.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:39.710+0000 7ff6fa400640 1 -- 172.21.15.16:0/2502246605 shutdown_connections 2024-03-31T15:07:39.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:39.710+0000 7ff6fa400640 1 --2- 172.21.15.16:0/2502246605 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff6f413e3d0 0x7ff6f406da10 unknown :-1 s=CLOSED pgs=738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:39.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:39.710+0000 7ff6fa400640 1 --2- 172.21.15.16:0/2502246605 >> v2:172.21.15.89:3300/0 conn(0x7ff6f413dff0 0x7ff6f406d4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:39.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:39.710+0000 7ff6fa400640 1 -- 172.21.15.16:0/2502246605 >> 172.21.15.16:0/2502246605 conn(0x7ff6f406ee40 msgr2=0x7ff6f406b620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:07:39.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:39.710+0000 7ff6fa400640 1 -- 172.21.15.16:0/2502246605 shutdown_connections 2024-03-31T15:07:39.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:39.710+0000 7ff6fa400640 1 -- 172.21.15.16:0/2502246605 wait complete. 2024-03-31T15:07:39.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:39.714+0000 7ff6fa400640 1 Processor -- start 2024-03-31T15:07:39.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:39.714+0000 7ff6fa400640 1 -- start start 2024-03-31T15:07:39.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:39.714+0000 7ff6fa400640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff6f40cc9d0 0x7ff6f4138690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:39.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:39.714+0000 7ff6fa400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff6f413dff0 0x7ff6f413a3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:39.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:39.714+0000 7ff6fa400640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff6f413e3d0 0x7ff6f4154150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:39.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:39.714+0000 7ff6fa400640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7ff6f40cf750 con 0x7ff6f413dff0 2024-03-31T15:07:39.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:39.714+0000 7ff6fa400640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7ff6f40cf5d0 con 0x7ff6f413e3d0 2024-03-31T15:07:39.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:39.714+0000 7ff6fa400640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7ff6f40cf8d0 con 0x7ff6f40cc9d0 2024-03-31T15:07:39.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:39.714+0000 7ff6eaa00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff6f413dff0 0x7ff6f413a3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:39.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:39.714+0000 7ff6eaa00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff6f413dff0 0x7ff6f413a3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47434/0 (socket says 172.21.15.16:47434) 2024-03-31T15:07:39.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:39.714+0000 7ff6eaa00640 1 -- 172.21.15.16:0/1393332727 learned_addr learned my addr 172.21.15.16:0/1393332727 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:07:39.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:39.714+0000 7ff6f3e00640 1 --2- 172.21.15.16:0/1393332727 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff6f413e3d0 0x7ff6f4154150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:39.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:39.714+0000 7ff6f3400640 1 --2- 172.21.15.16:0/1393332727 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff6f40cc9d0 0x7ff6f4138690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:39.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:39.714+0000 7ff6eaa00640 1 -- 172.21.15.16:0/1393332727 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff6f40cc9d0 msgr2=0x7ff6f4138690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:39.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:39.714+0000 7ff6eaa00640 1 --2- 172.21.15.16:0/1393332727 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff6f40cc9d0 0x7ff6f4138690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:39.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:39.714+0000 7ff6eaa00640 1 -- 172.21.15.16:0/1393332727 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff6f413e3d0 msgr2=0x7ff6f4154150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:39.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:39.714+0000 7ff6eaa00640 1 --2- 172.21.15.16:0/1393332727 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff6f413e3d0 0x7ff6f4154150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:39.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:39.714+0000 7ff6eaa00640 1 -- 172.21.15.16:0/1393332727 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6f4155740 con 0x7ff6f413dff0 2024-03-31T15:07:39.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:39.714+0000 7ff6f3400640 1 --2- 172.21.15.16:0/1393332727 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff6f40cc9d0 0x7ff6f4138690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:07:39.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:39.714+0000 7ff6eaa00640 1 --2- 172.21.15.16:0/1393332727 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff6f413dff0 0x7ff6f413a3e0 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7ff6dc0ae040 tx=0x7ff6dc019190 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:07:39.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:39.714+0000 7ff6ebe00640 1 -- 172.21.15.16:0/1393332727 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6dc0b8020 con 0x7ff6f413dff0 2024-03-31T15:07:39.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:39.714+0000 7ff6ebe00640 1 -- 172.21.15.16:0/1393332727 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff6dc0b4610 con 0x7ff6f413dff0 2024-03-31T15:07:39.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:39.714+0000 7ff6ebe00640 1 -- 172.21.15.16:0/1393332727 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6dc0b0040 con 0x7ff6f413dff0 2024-03-31T15:07:39.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:39.714+0000 7ff6fa400640 1 -- 172.21.15.16:0/1393332727 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff6f413c630 con 0x7ff6f413dff0 2024-03-31T15:07:39.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:39.714+0000 7ff6fa400640 1 -- 172.21.15.16:0/1393332727 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff6f413c350 con 0x7ff6f413dff0 2024-03-31T15:07:39.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:39.714+0000 7ff6fa400640 1 -- 172.21.15.16:0/1393332727 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff6f4069950 con 0x7ff6f413dff0 2024-03-31T15:07:39.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:39.718+0000 7ff6ebe00640 1 -- 172.21.15.16:0/1393332727 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7ff6dc0bb020 con 0x7ff6f413dff0 2024-03-31T15:07:39.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:39.718+0000 7ff6ebe00640 1 --2- 172.21.15.16:0/1393332727 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7ff6d0068380 0x7ff6d006a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:39.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:39.718+0000 7ff6ebe00640 1 -- 172.21.15.16:0/1393332727 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 5709+0+0 (secure 0 0 0) 0x7ff6dc120a60 con 0x7ff6f413dff0 2024-03-31T15:07:39.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:39.718+0000 7ff6f3400640 1 --2- 172.21.15.16:0/1393332727 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7ff6d0068380 0x7ff6d006a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:39.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:39.718+0000 7ff6f3400640 1 --2- 172.21.15.16:0/1393332727 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7ff6d0068380 0x7ff6d006a840 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7ff6f41551a0 tx=0x7ff6e4008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:07:39.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:39.722+0000 7ff6ebe00640 1 -- 172.21.15.16:0/1393332727 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7ff6dc0c2030 con 0x7ff6f413dff0 2024-03-31T15:07:39.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:39.886+0000 7ff6fa400640 1 -- 172.21.15.16:0/1393332727 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7ff6f406e4e0 con 0x7ff6f413dff0 2024-03-31T15:07:39.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:39.886+0000 7ff6ebe00640 1 -- 172.21.15.16:0/1393332727 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v56) ==== 104+0+11 (secure 0 0 0) 0x7ff6dc016070 con 0x7ff6f413dff0 2024-03-31T15:07:39.883 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2024-03-31T15:07:39.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:39.890+0000 7ff6fa400640 1 -- 172.21.15.16:0/1393332727 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7ff6d0068380 msgr2=0x7ff6d006a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:39.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:39.890+0000 7ff6fa400640 1 --2- 172.21.15.16:0/1393332727 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7ff6d0068380 0x7ff6d006a840 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7ff6f41551a0 tx=0x7ff6e4008040 comp rx=0 tx=0).stop 2024-03-31T15:07:39.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:39.890+0000 7ff6fa400640 1 -- 172.21.15.16:0/1393332727 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff6f413dff0 msgr2=0x7ff6f413a3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:39.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:39.890+0000 7ff6fa400640 1 --2- 172.21.15.16:0/1393332727 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff6f413dff0 0x7ff6f413a3e0 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7ff6dc0ae040 tx=0x7ff6dc019190 comp rx=0 tx=0).stop 2024-03-31T15:07:39.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:39.890+0000 7ff6fa400640 1 -- 172.21.15.16:0/1393332727 shutdown_connections 2024-03-31T15:07:39.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:39.890+0000 7ff6fa400640 1 --2- 172.21.15.16:0/1393332727 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7ff6d0068380 0x7ff6d006a840 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:39.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:39.890+0000 7ff6fa400640 1 --2- 172.21.15.16:0/1393332727 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff6f413e3d0 0x7ff6f4154150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:39.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:39.890+0000 7ff6fa400640 1 --2- 172.21.15.16:0/1393332727 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff6f413dff0 0x7ff6f413a3e0 unknown :-1 s=CLOSED pgs=739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:39.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:39.890+0000 7ff6fa400640 1 --2- 172.21.15.16:0/1393332727 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff6f40cc9d0 0x7ff6f4138690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:39.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:39.890+0000 7ff6fa400640 1 -- 172.21.15.16:0/1393332727 >> 172.21.15.16:0/1393332727 conn(0x7ff6f406ee40 msgr2=0x7ff6f415b8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:07:39.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:39.890+0000 7ff6fa400640 1 -- 172.21.15.16:0/1393332727 shutdown_connections 2024-03-31T15:07:39.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:39.890+0000 7ff6fa400640 1 -- 172.21.15.16:0/1393332727 wait complete. 2024-03-31T15:07:39.903 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-31T15:07:40.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:40.050+0000 7f985da00640 1 Processor -- start 2024-03-31T15:07:40.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:40.050+0000 7f985da00640 1 -- start start 2024-03-31T15:07:40.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:40.054+0000 7f985da00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f985806b620 0x7f985806bab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:40.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:40.054+0000 7f985da00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f98580af660 0x7f98580afa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:40.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:40.054+0000 7f985da00640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f98580ae5c0 con 0x7f9858145a80 2024-03-31T15:07:40.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:40.054+0000 7f985da00640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f98580ae8a0 con 0x7f985806b620 2024-03-31T15:07:40.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:40.054+0000 7f985da00640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f98580aeb80 con 0x7f98580af660 2024-03-31T15:07:40.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:40.054+0000 7f984fe00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f985806b620 0x7f985806bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:40.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:40.054+0000 7f9857e00640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f9858145a80 0x7f985806e320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:53790/0 (socket says 172.21.15.16:53790) 2024-03-31T15:07:40.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:40.054+0000 7f9857e00640 1 -- 172.21.15.16:0/3192157107 learned_addr learned my addr 172.21.15.16:0/3192157107 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:07:40.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:40.054+0000 7f985d000640 1 --2- 172.21.15.16:0/3192157107 >> v2:172.21.15.89:3300/0 conn(0x7f98580af660 0x7f98580afa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:40.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:40.054+0000 7f984fe00640 1 -- 172.21.15.16:0/3192157107 >> v2:172.21.15.89:3300/0 conn(0x7f98580af660 msgr2=0x7f98580afa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:07:40.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:40.054+0000 7f984fe00640 1 --2- 172.21.15.16:0/3192157107 >> v2:172.21.15.89:3300/0 conn(0x7f98580af660 0x7f98580afa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:40.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:40.054+0000 7f984fe00640 1 -- 172.21.15.16:0/3192157107 >> v1:172.21.15.16:6789/0 conn(0x7f9858145a80 legacy=0x7f985806e320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:40.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:40.054+0000 7f984fe00640 1 -- 172.21.15.16:0/3192157107 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f98580aee60 con 0x7f985806b620 2024-03-31T15:07:40.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:40.054+0000 7f9857400640 1 -- 172.21.15.16:0/3192157107 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2395062137 0 0) 0x7f98580ae5c0 con 0x7f9858145a80 2024-03-31T15:07:40.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:40.054+0000 7f985d000640 1 --2- 172.21.15.16:0/3192157107 >> v2:172.21.15.89:3300/0 conn(0x7f98580af660 0x7f98580afa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:07:40.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:40.058+0000 7f984fe00640 1 --2- 172.21.15.16:0/3192157107 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f985806b620 0x7f985806bab0 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7f98580d0df0 tx=0x7f984001c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f881f3b6d54ae8f1 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:07:40.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:40.058+0000 7f9857400640 1 -- 172.21.15.16:0/3192157107 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98400b1070 con 0x7f985806b620 2024-03-31T15:07:40.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:40.058+0000 7f9857400640 1 -- 172.21.15.16:0/3192157107 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f98400ac5f0 con 0x7f985806b620 2024-03-31T15:07:40.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:40.058+0000 7f9857400640 1 -- 172.21.15.16:0/3192157107 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98400ac9c0 con 0x7f985806b620 2024-03-31T15:07:40.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:40.058+0000 7f985da00640 1 -- 172.21.15.16:0/3192157107 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f985806b620 msgr2=0x7f985806bab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:40.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:40.058+0000 7f985da00640 1 --2- 172.21.15.16:0/3192157107 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f985806b620 0x7f985806bab0 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7f98580d0df0 tx=0x7f984001c480 comp rx=0 tx=0).stop 2024-03-31T15:07:40.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:40.058+0000 7f985da00640 1 -- 172.21.15.16:0/3192157107 shutdown_connections 2024-03-31T15:07:40.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:40.058+0000 7f985da00640 1 --2- 172.21.15.16:0/3192157107 >> v2:172.21.15.89:3300/0 conn(0x7f98580af660 0x7f98580afa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:40.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:40.058+0000 7f985da00640 1 --2- 172.21.15.16:0/3192157107 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f985806b620 0x7f985806bab0 unknown :-1 s=CLOSED pgs=741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:40.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:40.058+0000 7f985da00640 1 -- 172.21.15.16:0/3192157107 >> 172.21.15.16:0/3192157107 conn(0x7f98580b2b30 msgr2=0x7f98580b2f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:07:40.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:40.058+0000 7f985da00640 1 -- 172.21.15.16:0/3192157107 shutdown_connections 2024-03-31T15:07:40.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:40.058+0000 7f985da00640 1 -- 172.21.15.16:0/3192157107 wait complete. 2024-03-31T15:07:40.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:40.062+0000 7f985da00640 1 Processor -- start 2024-03-31T15:07:40.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:40.062+0000 7f985da00640 1 -- start start 2024-03-31T15:07:40.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:40.062+0000 7f985da00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f985806b620 0x7f98581f6b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:40.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:40.062+0000 7f985da00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f98580af660 0x7f985815e920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:40.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:40.062+0000 7f985da00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9858145a80 0x7f9858165850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:40.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:40.062+0000 7f985da00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f9858168df0 con 0x7f9858145a80 2024-03-31T15:07:40.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:40.062+0000 7f985da00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f9858168c70 con 0x7f985806b620 2024-03-31T15:07:40.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:40.062+0000 7f985da00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f9858168f70 con 0x7f98580af660 2024-03-31T15:07:40.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:40.062+0000 7f985d000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9858145a80 0x7f9858165850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:40.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:40.062+0000 7f985d000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9858145a80 0x7f9858165850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47454/0 (socket says 172.21.15.16:47454) 2024-03-31T15:07:40.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:40.062+0000 7f985d000640 1 -- 172.21.15.16:0/568302110 learned_addr learned my addr 172.21.15.16:0/568302110 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:07:40.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:40.062+0000 7f9857e00640 1 --2- 172.21.15.16:0/568302110 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f985806b620 0x7f98581f6b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:40.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:40.062+0000 7f984fe00640 1 --2- 172.21.15.16:0/568302110 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f98580af660 0x7f985815e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:40.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:40.062+0000 7f985d000640 1 -- 172.21.15.16:0/568302110 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f98580af660 msgr2=0x7f985815e920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:40.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:40.062+0000 7f985d000640 1 --2- 172.21.15.16:0/568302110 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f98580af660 0x7f985815e920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:40.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:40.062+0000 7f985d000640 1 -- 172.21.15.16:0/568302110 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f985806b620 msgr2=0x7f98581f6b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:40.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:40.062+0000 7f985d000640 1 --2- 172.21.15.16:0/568302110 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f985806b620 0x7f98581f6b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:40.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:40.062+0000 7f985d000640 1 -- 172.21.15.16:0/568302110 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f985809bcb0 con 0x7f9858145a80 2024-03-31T15:07:40.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:40.062+0000 7f985d000640 1 --2- 172.21.15.16:0/568302110 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9858145a80 0x7f9858165850 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f9850007ef0 tx=0x7f985000d4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:07:40.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:40.062+0000 7f9854c00640 1 -- 172.21.15.16:0/568302110 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f985001a070 con 0x7f9858145a80 2024-03-31T15:07:40.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:40.062+0000 7f9854c00640 1 -- 172.21.15.16:0/568302110 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9850003ba0 con 0x7f9858145a80 2024-03-31T15:07:40.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:40.062+0000 7f9854c00640 1 -- 172.21.15.16:0/568302110 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9850004030 con 0x7f9858145a80 2024-03-31T15:07:40.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:40.062+0000 7f985da00640 1 -- 172.21.15.16:0/568302110 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f985809bf90 con 0x7f9858145a80 2024-03-31T15:07:40.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:40.062+0000 7f985da00640 1 -- 172.21.15.16:0/568302110 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f985809c550 con 0x7f9858145a80 2024-03-31T15:07:40.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:40.066+0000 7f985da00640 1 -- 172.21.15.16:0/568302110 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f985804f8c0 con 0x7f9858145a80 2024-03-31T15:07:40.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:40.066+0000 7f9854c00640 1 -- 172.21.15.16:0/568302110 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f985001d070 con 0x7f9858145a80 2024-03-31T15:07:40.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:40.066+0000 7f9854c00640 1 --2- 172.21.15.16:0/568302110 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f9834068380 0x7f983406a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:40.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:40.066+0000 7f9854c00640 1 -- 172.21.15.16:0/568302110 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 5709+0+0 (secure 0 0 0) 0x7f9850086fa0 con 0x7f9858145a80 2024-03-31T15:07:40.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:40.066+0000 7f9857e00640 1 --2- 172.21.15.16:0/568302110 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f9834068380 0x7f983406a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:40.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:40.070+0000 7f9857e00640 1 --2- 172.21.15.16:0/568302110 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f9834068380 0x7f983406a840 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f9848000f70 tx=0x7f9848007c10 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:07:40.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:40.070+0000 7f9854c00640 1 -- 172.21.15.16:0/568302110 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f9850010070 con 0x7f9858145a80 2024-03-31T15:07:40.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:40.234+0000 7f985da00640 1 -- 172.21.15.16:0/568302110 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f9858075260 con 0x7f9858145a80 2024-03-31T15:07:40.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:40.234+0000 7f9854c00640 1 -- 172.21.15.16:0/568302110 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v56) ==== 100+0+11 (secure 0 0 0) 0x7f9850056ff0 con 0x7f9858145a80 2024-03-31T15:07:40.231 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 64 2024-03-31T15:07:40.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:40.238+0000 7f985da00640 1 -- 172.21.15.16:0/568302110 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f9834068380 msgr2=0x7f983406a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:40.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:40.238+0000 7f985da00640 1 --2- 172.21.15.16:0/568302110 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f9834068380 0x7f983406a840 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f9848000f70 tx=0x7f9848007c10 comp rx=0 tx=0).stop 2024-03-31T15:07:40.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:40.238+0000 7f985da00640 1 -- 172.21.15.16:0/568302110 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9858145a80 msgr2=0x7f9858165850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:40.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:40.238+0000 7f985da00640 1 --2- 172.21.15.16:0/568302110 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9858145a80 0x7f9858165850 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f9850007ef0 tx=0x7f985000d4a0 comp rx=0 tx=0).stop 2024-03-31T15:07:40.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:40.238+0000 7f985da00640 1 -- 172.21.15.16:0/568302110 shutdown_connections 2024-03-31T15:07:40.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:40.238+0000 7f985da00640 1 --2- 172.21.15.16:0/568302110 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f9834068380 0x7f983406a840 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:40.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:40.238+0000 7f985da00640 1 --2- 172.21.15.16:0/568302110 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9858145a80 0x7f9858165850 unknown :-1 s=CLOSED pgs=742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:40.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:40.238+0000 7f985da00640 1 --2- 172.21.15.16:0/568302110 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f98580af660 0x7f985815e920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:40.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:40.238+0000 7f985da00640 1 --2- 172.21.15.16:0/568302110 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f985806b620 0x7f98581f6b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:40.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:40.238+0000 7f985da00640 1 -- 172.21.15.16:0/568302110 >> 172.21.15.16:0/568302110 conn(0x7f98580b2b30 msgr2=0x7f98580731b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:07:40.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:40.238+0000 7f985da00640 1 -- 172.21.15.16:0/568302110 shutdown_connections 2024-03-31T15:07:40.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:40.238+0000 7f985da00640 1 -- 172.21.15.16:0/568302110 wait complete. 2024-03-31T15:07:40.253 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-31T15:07:40.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:40.398+0000 7fc6af800640 1 Processor -- start 2024-03-31T15:07:40.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:40.398+0000 7fc6af800640 1 -- start start 2024-03-31T15:07:40.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:40.402+0000 7fc6af800640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fc6a806dbc0 0x7fc6a815f650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:40.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:40.402+0000 7fc6af800640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fc6a80cb440 0x7fc6a806f920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:40.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:40.402+0000 7fc6af800640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc6a80cbe90 con 0x7fc6a80cb8c0 2024-03-31T15:07:40.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:40.402+0000 7fc6af800640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7fc6a815fb90 con 0x7fc6a806dbc0 2024-03-31T15:07:40.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:40.402+0000 7fc6af800640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7fc6a815f1c0 con 0x7fc6a80cb440 2024-03-31T15:07:40.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:40.402+0000 7fc6ace00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fc6a806dbc0 0x7fc6a815f650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:40.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:40.402+0000 7fc6ad800640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fc6a80cb8c0 0x7fc6a8162180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:53798/0 (socket says 172.21.15.16:53798) 2024-03-31T15:07:40.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:40.402+0000 7fc6ad800640 1 -- 172.21.15.16:0/4118392835 learned_addr learned my addr 172.21.15.16:0/4118392835 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:07:40.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:40.402+0000 7fc69fe00640 1 --2- 172.21.15.16:0/4118392835 >> v2:172.21.15.89:3300/0 conn(0x7fc6a80cb440 0x7fc6a806f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:40.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:40.402+0000 7fc6ace00640 1 -- 172.21.15.16:0/4118392835 >> v2:172.21.15.89:3300/0 conn(0x7fc6a80cb440 msgr2=0x7fc6a806f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:07:40.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:40.402+0000 7fc6ace00640 1 --2- 172.21.15.16:0/4118392835 >> v2:172.21.15.89:3300/0 conn(0x7fc6a80cb440 0x7fc6a806f920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:40.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:40.402+0000 7fc6a7e00640 1 -- 172.21.15.16:0/4118392835 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4159585011 0 0) 0x7fc6a80cbe90 con 0x7fc6a80cb8c0 2024-03-31T15:07:40.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:40.402+0000 7fc6ace00640 1 -- 172.21.15.16:0/4118392835 >> v1:172.21.15.16:6789/0 conn(0x7fc6a80cb8c0 legacy=0x7fc6a8162180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:40.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:40.402+0000 7fc6ace00640 1 -- 172.21.15.16:0/4118392835 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6a80cc170 con 0x7fc6a806dbc0 2024-03-31T15:07:40.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:40.406+0000 7fc6ace00640 1 --2- 172.21.15.16:0/4118392835 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc6a806dbc0 0x7fc6a815f650 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7fc6a80cf0b0 tx=0x7fc69801c3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a90367bf9939dee server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:07:40.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:40.406+0000 7fc6a7e00640 1 -- 172.21.15.16:0/4118392835 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6980b0070 con 0x7fc6a806dbc0 2024-03-31T15:07:40.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:40.406+0000 7fc6a7e00640 1 -- 172.21.15.16:0/4118392835 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc6980ab600 con 0x7fc6a806dbc0 2024-03-31T15:07:40.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:40.406+0000 7fc6a7e00640 1 -- 172.21.15.16:0/4118392835 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6980ab9d0 con 0x7fc6a806dbc0 2024-03-31T15:07:40.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:40.406+0000 7fc6af800640 1 -- 172.21.15.16:0/4118392835 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc6a806dbc0 msgr2=0x7fc6a815f650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:40.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:40.406+0000 7fc6af800640 1 --2- 172.21.15.16:0/4118392835 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc6a806dbc0 0x7fc6a815f650 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7fc6a80cf0b0 tx=0x7fc69801c3c0 comp rx=0 tx=0).stop 2024-03-31T15:07:40.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:40.406+0000 7fc6af800640 1 -- 172.21.15.16:0/4118392835 shutdown_connections 2024-03-31T15:07:40.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:40.406+0000 7fc6af800640 1 --2- 172.21.15.16:0/4118392835 >> v2:172.21.15.89:3300/0 conn(0x7fc6a80cb440 0x7fc6a806f920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:40.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:40.406+0000 7fc6af800640 1 --2- 172.21.15.16:0/4118392835 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc6a806dbc0 0x7fc6a815f650 unknown :-1 s=CLOSED pgs=744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:40.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:40.406+0000 7fc6af800640 1 -- 172.21.15.16:0/4118392835 >> 172.21.15.16:0/4118392835 conn(0x7fc6a8067330 msgr2=0x7fc6a8067740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:07:40.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:40.406+0000 7fc6af800640 1 -- 172.21.15.16:0/4118392835 shutdown_connections 2024-03-31T15:07:40.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:40.406+0000 7fc6af800640 1 -- 172.21.15.16:0/4118392835 wait complete. 2024-03-31T15:07:40.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:40.406+0000 7fc6af800640 1 Processor -- start 2024-03-31T15:07:40.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:40.406+0000 7fc6af800640 1 -- start start 2024-03-31T15:07:40.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:40.410+0000 7fc6af800640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc6a806dbc0 0x7fc6a813d1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:40.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:40.410+0000 7fc6af800640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fc6a80cb440 0x7fc6a8157120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:40.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:40.410+0000 7fc6af800640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc6a80cb8c0 0x7fc6a8156310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:40.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:40.410+0000 7fc6af800640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fc6a806c950 con 0x7fc6a80cb8c0 2024-03-31T15:07:40.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:40.410+0000 7fc6af800640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fc6a8162ca0 con 0x7fc6a806dbc0 2024-03-31T15:07:40.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:40.410+0000 7fc6af800640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fc6a806cad0 con 0x7fc6a80cb440 2024-03-31T15:07:40.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:40.410+0000 7fc6ad800640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc6a80cb8c0 0x7fc6a8156310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:40.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:40.410+0000 7fc6ad800640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc6a80cb8c0 0x7fc6a8156310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47470/0 (socket says 172.21.15.16:47470) 2024-03-31T15:07:40.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:40.410+0000 7fc6ad800640 1 -- 172.21.15.16:0/3707961374 learned_addr learned my addr 172.21.15.16:0/3707961374 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:07:40.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:40.410+0000 7fc6ace00640 1 --2- 172.21.15.16:0/3707961374 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc6a806dbc0 0x7fc6a813d1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:40.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:40.410+0000 7fc6ad800640 1 -- 172.21.15.16:0/3707961374 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fc6a80cb440 msgr2=0x7fc6a8157120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:40.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:40.410+0000 7fc69fe00640 1 --2- 172.21.15.16:0/3707961374 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fc6a80cb440 0x7fc6a8157120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:40.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:40.410+0000 7fc6ad800640 1 --2- 172.21.15.16:0/3707961374 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fc6a80cb440 0x7fc6a8157120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:40.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:40.410+0000 7fc6ad800640 1 -- 172.21.15.16:0/3707961374 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc6a806dbc0 msgr2=0x7fc6a813d1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:40.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:40.410+0000 7fc6ad800640 1 --2- 172.21.15.16:0/3707961374 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc6a806dbc0 0x7fc6a813d1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:40.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:40.410+0000 7fc6ad800640 1 -- 172.21.15.16:0/3707961374 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6a8143090 con 0x7fc6a80cb8c0 2024-03-31T15:07:40.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:40.410+0000 7fc6ace00640 1 --2- 172.21.15.16:0/3707961374 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc6a806dbc0 0x7fc6a813d1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:07:40.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:40.410+0000 7fc69fe00640 1 --2- 172.21.15.16:0/3707961374 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fc6a80cb440 0x7fc6a8157120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:07:40.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:40.410+0000 7fc6ad800640 1 --2- 172.21.15.16:0/3707961374 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc6a80cb8c0 0x7fc6a8156310 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7fc6a000c700 tx=0x7fc6a000cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:07:40.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:40.410+0000 7fc6a5600640 1 -- 172.21.15.16:0/3707961374 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6a0018070 con 0x7fc6a80cb8c0 2024-03-31T15:07:40.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:40.410+0000 7fc6a5600640 1 -- 172.21.15.16:0/3707961374 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc6a000ce10 con 0x7fc6a80cb8c0 2024-03-31T15:07:40.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:40.410+0000 7fc6a5600640 1 -- 172.21.15.16:0/3707961374 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6a000a860 con 0x7fc6a80cb8c0 2024-03-31T15:07:40.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:40.410+0000 7fc6af800640 1 -- 172.21.15.16:0/3707961374 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc6a8141070 con 0x7fc6a80cb8c0 2024-03-31T15:07:40.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:40.410+0000 7fc6af800640 1 -- 172.21.15.16:0/3707961374 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc6a8141630 con 0x7fc6a80cb8c0 2024-03-31T15:07:40.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:40.410+0000 7fc6af800640 1 -- 172.21.15.16:0/3707961374 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc6a8069950 con 0x7fc6a80cb8c0 2024-03-31T15:07:40.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:40.410+0000 7fc6a5600640 1 -- 172.21.15.16:0/3707961374 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fc6a000aa00 con 0x7fc6a80cb8c0 2024-03-31T15:07:40.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:40.414+0000 7fc6a5600640 1 --2- 172.21.15.16:0/3707961374 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fc684068380 0x7fc68406a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:40.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:40.414+0000 7fc6a5600640 1 -- 172.21.15.16:0/3707961374 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 5709+0+0 (secure 0 0 0) 0x7fc6a00860f0 con 0x7fc6a80cb8c0 2024-03-31T15:07:40.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:40.414+0000 7fc6ace00640 1 --2- 172.21.15.16:0/3707961374 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fc684068380 0x7fc68406a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:40.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:40.414+0000 7fc6ace00640 1 --2- 172.21.15.16:0/3707961374 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fc684068380 0x7fc68406a840 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fc6980ad000 tx=0x7fc698008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:07:40.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:40.418+0000 7fc6a5600640 1 -- 172.21.15.16:0/3707961374 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fc6a0056580 con 0x7fc6a80cb8c0 2024-03-31T15:07:40.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:40.662+0000 7fc6af800640 1 -- 172.21.15.16:0/3707961374 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fc6a80cf050 con 0x7fc6a80cb8c0 2024-03-31T15:07:40.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:40.666+0000 7fc6a5600640 1 -- 172.21.15.16:0/3707961374 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v56) ==== 80+0+0 (secure 0 0 0) 0x7fc6a0055e80 con 0x7fc6a80cb8c0 2024-03-31T15:07:40.660 INFO:teuthology.orchestra.run.smithi016.stderr:listed 0 entries 2024-03-31T15:07:40.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:40.666+0000 7fc6af800640 1 -- 172.21.15.16:0/3707961374 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fc684068380 msgr2=0x7fc68406a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:40.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:40.666+0000 7fc6af800640 1 --2- 172.21.15.16:0/3707961374 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fc684068380 0x7fc68406a840 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fc6980ad000 tx=0x7fc698008040 comp rx=0 tx=0).stop 2024-03-31T15:07:40.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:40.666+0000 7fc6af800640 1 -- 172.21.15.16:0/3707961374 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc6a80cb8c0 msgr2=0x7fc6a8156310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:40.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:40.666+0000 7fc6af800640 1 --2- 172.21.15.16:0/3707961374 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc6a80cb8c0 0x7fc6a8156310 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7fc6a000c700 tx=0x7fc6a000cbc0 comp rx=0 tx=0).stop 2024-03-31T15:07:40.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:40.666+0000 7fc6af800640 1 -- 172.21.15.16:0/3707961374 shutdown_connections 2024-03-31T15:07:40.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:40.666+0000 7fc6af800640 1 --2- 172.21.15.16:0/3707961374 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fc684068380 0x7fc68406a840 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:40.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:40.666+0000 7fc6af800640 1 --2- 172.21.15.16:0/3707961374 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc6a80cb8c0 0x7fc6a8156310 unknown :-1 s=CLOSED pgs=745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:40.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:40.666+0000 7fc6af800640 1 --2- 172.21.15.16:0/3707961374 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fc6a80cb440 0x7fc6a8157120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:40.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:40.666+0000 7fc6af800640 1 --2- 172.21.15.16:0/3707961374 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc6a806dbc0 0x7fc6a813d1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:40.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:40.666+0000 7fc6af800640 1 -- 172.21.15.16:0/3707961374 >> 172.21.15.16:0/3707961374 conn(0x7fc6a8067330 msgr2=0x7fc6a80d0890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:07:40.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:40.666+0000 7fc6af800640 1 -- 172.21.15.16:0/3707961374 shutdown_connections 2024-03-31T15:07:40.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:40.666+0000 7fc6af800640 1 -- 172.21.15.16:0/3707961374 wait complete. 2024-03-31T15:07:40.681 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi016.front.sepia.ceph.com... 2024-03-31T15:07:40.681 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:07:40.681 DEBUG:teuthology.orchestra.run.smithi016:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-03-31T15:07:40.701 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi016.stderr:ceph-fuse[39791]: fuse finished with error 0 and tester_r 0 2024-03-31T15:07:40.703 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi016.front.sepia.ceph.com 2024-03-31T15:07:40.704 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:07:40.704 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-31T15:07:40.877 INFO:tasks.cephfs.mount:Terminating background process 2024-03-31T15:07:40.877 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-03-31T15:07:40.877 DEBUG:teuthology.orchestra.run.smithi089:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-31T15:07:40.888 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16354452k,nr_inodes=4088613,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275344k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=901 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275340k,nr_inodes=818835,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275344k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-31T15:07:40.889 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi089.front.sepia.ceph.com 2024-03-31T15:07:40.889 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:07:40.889 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-03-31T15:07:40.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-31T15:07:40.940 INFO:teuthology.orchestra.run.smithi089.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.1': No such file or directory 2024-03-31T15:07:40.940 INFO:tasks.cephfs.mount:Terminating background process 2024-03-31T15:07:40.941 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-03-31T15:07:40.941 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-31T15:07:40.951 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16357708k,nr_inodes=4089427,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275996k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=6539 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275992k,nr_inodes=818998,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275996k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-31T15:07:40.951 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi016.front.sepia.ceph.com 2024-03-31T15:07:40.952 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:07:40.952 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-03-31T15:07:41.003 INFO:teuthology.orchestra.run.smithi016.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2024-03-31T15:07:41.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-31T15:07:41.004 INFO:tasks.cephfs.mount:Terminating background process 2024-03-31T15:07:41.005 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-03-31T15:07:41.005 DEBUG:teuthology.orchestra.run.smithi089:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-31T15:07:41.016 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16354452k,nr_inodes=4088613,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275344k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=901 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275340k,nr_inodes=818835,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275344k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-31T15:07:41.017 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi089.front.sepia.ceph.com 2024-03-31T15:07:41.017 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:07:41.017 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-03-31T15:07:41.071 INFO:teuthology.orchestra.run.smithi089.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2024-03-31T15:07:41.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-31T15:07:41.072 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-31T15:07:41.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:41.218+0000 7f5325e00640 1 Processor -- start 2024-03-31T15:07:41.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:41.218+0000 7f5325e00640 1 -- start start 2024-03-31T15:07:41.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:41.218+0000 7f5325e00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f532006f920 0x7f532006fd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:41.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:41.218+0000 7f5325e00640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f53200d0920 con 0x7f53200d2010 2024-03-31T15:07:41.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:41.218+0000 7f5325e00640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f53200d0c00 con 0x7f5320145e40 2024-03-31T15:07:41.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:41.218+0000 7f5325e00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f53200702c0 con 0x7f532006f920 2024-03-31T15:07:41.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:41.218+0000 7f531f400640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f53200d2010 0x7f53200d24a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:53810/0 (socket says 172.21.15.16:53810) 2024-03-31T15:07:41.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:41.218+0000 7f531f400640 1 -- 172.21.15.16:0/1429116448 learned_addr learned my addr 172.21.15.16:0/1429116448 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:07:41.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:41.218+0000 7f531e000640 1 --2- 172.21.15.16:0/1429116448 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f532006f920 0x7f532006fd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:41.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:41.218+0000 7f531d600640 1 -- 172.21.15.16:0/1429116448 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1406494775 0 0) 0x7f53200d0920 con 0x7f53200d2010 2024-03-31T15:07:41.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:41.218+0000 7f531d600640 1 -- 172.21.15.16:0/1429116448 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5304003530 con 0x7f53200d2010 2024-03-31T15:07:41.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:41.218+0000 7f531d600640 1 -- 172.21.15.16:0/1429116448 <== mon.1 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 702710713 0 0) 0x7f53200d0c00 con 0x7f5320145e40 2024-03-31T15:07:41.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:41.218+0000 7f531d600640 1 -- 172.21.15.16:0/1429116448 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f53200d0920 con 0x7f5320145e40 2024-03-31T15:07:41.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:41.218+0000 7f531e000640 1 -- 172.21.15.16:0/1429116448 >> v1:172.21.15.89:6789/0 conn(0x7f5320145e40 legacy=0x7f53200cc740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:41.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:41.218+0000 7f531e000640 1 -- 172.21.15.16:0/1429116448 >> v1:172.21.15.16:6789/0 conn(0x7f53200d2010 legacy=0x7f53200d24a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:41.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:41.218+0000 7f531e000640 1 -- 172.21.15.16:0/1429116448 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53200d1810 con 0x7f532006f920 2024-03-31T15:07:41.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:41.222+0000 7f531e000640 1 --2- 172.21.15.16:0/1429116448 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f532006f920 0x7f532006fd80 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f530c0097f0 tx=0x7f530c01c4e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9c1fd34aea54e05e server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:07:41.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:41.222+0000 7f531d600640 1 -- 172.21.15.16:0/1429116448 <== mon.2 v2:172.21.15.89:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f530c0b0070 con 0x7f532006f920 2024-03-31T15:07:41.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:41.222+0000 7f531d600640 1 -- 172.21.15.16:0/1429116448 <== mon.2 v2:172.21.15.89:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f530c0ab600 con 0x7f532006f920 2024-03-31T15:07:41.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:41.222+0000 7f531d600640 1 -- 172.21.15.16:0/1429116448 <== mon.2 v2:172.21.15.89:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f530c0ab9d0 con 0x7f532006f920 2024-03-31T15:07:41.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:41.222+0000 7f5325e00640 1 -- 172.21.15.16:0/1429116448 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f532006f920 msgr2=0x7f532006fd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:41.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:41.222+0000 7f5325e00640 1 --2- 172.21.15.16:0/1429116448 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f532006f920 0x7f532006fd80 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f530c0097f0 tx=0x7f530c01c4e0 comp rx=0 tx=0).stop 2024-03-31T15:07:41.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:41.222+0000 7f5325e00640 1 -- 172.21.15.16:0/1429116448 shutdown_connections 2024-03-31T15:07:41.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:41.222+0000 7f5325e00640 1 --2- 172.21.15.16:0/1429116448 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f532006f920 0x7f532006fd80 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:41.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:41.222+0000 7f5325e00640 1 -- 172.21.15.16:0/1429116448 >> 172.21.15.16:0/1429116448 conn(0x7f5320067330 msgr2=0x7f5320067740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:07:41.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:41.222+0000 7f5325e00640 1 -- 172.21.15.16:0/1429116448 shutdown_connections 2024-03-31T15:07:41.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:41.222+0000 7f5325e00640 1 -- 172.21.15.16:0/1429116448 wait complete. 2024-03-31T15:07:41.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:41.226+0000 7f5325e00640 1 Processor -- start 2024-03-31T15:07:41.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:41.226+0000 7f5325e00640 1 -- start start 2024-03-31T15:07:41.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:41.226+0000 7f5325e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f532006f920 0x7f53200b66b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:41.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:41.226+0000 7f5325e00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f53200d2010 0x7f53201426d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:41.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:41.226+0000 7f5325e00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f5320145e40 0x7f5320142c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:41.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:41.226+0000 7f5325e00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f53200d0ee0 con 0x7f532006f920 2024-03-31T15:07:41.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:41.226+0000 7f5325e00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f53200d06d0 con 0x7f53200d2010 2024-03-31T15:07:41.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:41.226+0000 7f5325e00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f53200b50c0 con 0x7f5320145e40 2024-03-31T15:07:41.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:41.226+0000 7f531ea00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f532006f920 0x7f53200b66b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:41.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:41.226+0000 7f531ea00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f532006f920 0x7f53200b66b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47482/0 (socket says 172.21.15.16:47482) 2024-03-31T15:07:41.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:41.226+0000 7f531ea00640 1 -- 172.21.15.16:0/1361613739 learned_addr learned my addr 172.21.15.16:0/1361613739 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:07:41.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:41.226+0000 7f531e000640 1 --2- 172.21.15.16:0/1361613739 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f53200d2010 0x7f53201426d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:41.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:41.226+0000 7f531f400640 1 --2- 172.21.15.16:0/1361613739 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f5320145e40 0x7f5320142c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:41.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:41.226+0000 7f531ea00640 1 -- 172.21.15.16:0/1361613739 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f5320145e40 msgr2=0x7f5320142c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:41.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:41.226+0000 7f531ea00640 1 --2- 172.21.15.16:0/1361613739 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f5320145e40 0x7f5320142c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:41.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:41.226+0000 7f531ea00640 1 -- 172.21.15.16:0/1361613739 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f53200d2010 msgr2=0x7f53201426d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:41.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:41.226+0000 7f531ea00640 1 --2- 172.21.15.16:0/1361613739 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f53200d2010 0x7f53201426d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:41.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:41.226+0000 7f531ea00640 1 -- 172.21.15.16:0/1361613739 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f532004f9f0 con 0x7f532006f920 2024-03-31T15:07:41.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:41.226+0000 7f531ea00640 1 --2- 172.21.15.16:0/1361613739 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f532006f920 0x7f53200b66b0 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f531000c490 tx=0x7f531000c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:07:41.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:41.226+0000 7f5317400640 1 -- 172.21.15.16:0/1361613739 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53100052d0 con 0x7f532006f920 2024-03-31T15:07:41.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:41.226+0000 7f5317400640 1 -- 172.21.15.16:0/1361613739 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5310005470 con 0x7f532006f920 2024-03-31T15:07:41.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:41.226+0000 7f5317400640 1 -- 172.21.15.16:0/1361613739 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5310012870 con 0x7f532006f920 2024-03-31T15:07:41.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:41.226+0000 7f5325e00640 1 -- 172.21.15.16:0/1361613739 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5320158a00 con 0x7f532006f920 2024-03-31T15:07:41.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:41.226+0000 7f5325e00640 1 -- 172.21.15.16:0/1361613739 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5320157c40 con 0x7f532006f920 2024-03-31T15:07:41.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:41.226+0000 7f5325e00640 1 -- 172.21.15.16:0/1361613739 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5320069950 con 0x7f532006f920 2024-03-31T15:07:41.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:41.230+0000 7f5317400640 1 -- 172.21.15.16:0/1361613739 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f5310012a10 con 0x7f532006f920 2024-03-31T15:07:41.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:41.230+0000 7f5317400640 1 --2- 172.21.15.16:0/1361613739 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f5300068380 0x7f530006a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:41.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:41.230+0000 7f5317400640 1 -- 172.21.15.16:0/1361613739 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 5709+0+0 (secure 0 0 0) 0x7f5310086710 con 0x7f532006f920 2024-03-31T15:07:41.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:41.230+0000 7f531e000640 1 --2- 172.21.15.16:0/1361613739 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f5300068380 0x7f530006a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:41.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:41.230+0000 7f531e000640 1 --2- 172.21.15.16:0/1361613739 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f5300068380 0x7f530006a840 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f530c0ad000 tx=0x7f530c008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:07:41.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:41.234+0000 7f5317400640 1 -- 172.21.15.16:0/1361613739 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f5320158a00 con 0x7f532006f920 2024-03-31T15:07:41.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:41.434+0000 7f5325e00640 1 -- 172.21.15.16:0/1361613739 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f53200b7850 con 0x7f532006f920 2024-03-31T15:07:41.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:41.434+0000 7f5317400640 1 -- 172.21.15.16:0/1361613739 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 36 v36) ==== 94+0+5182 (secure 0 0 0) 0x7f5310056c30 con 0x7f532006f920 2024-03-31T15:07:41.430 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:07:41.431 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":36,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5579,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6833/3721971977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":3721971977},{"type":"v1","addr":"172.21.15.16:6833","nonce":3721971977}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":29},{"gid":6109,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6836/1166206553","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":1166206553},{"type":"v1","addr":"172.21.15.89:6836","nonce":1166206553}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28},{"gid":6112,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/1826113083","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6835","nonce":1826113083},{"type":"v1","addr":"172.21.15.89:6837","nonce":1826113083}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":29}],"filesystems":[{"mdsmap":{"epoch":36,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-31T15:06:25.665001+0000","modified":"2024-03-31T15:07:14.315988+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6150},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6150":{"gid":6150,"name":"c","rank":0,"incarnation":30,"state":"up:active","state_seq":15,"addr":"172.21.15.16:6835/2613117528","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":2613117528},{"type":"v1","addr":"172.21.15.16:6835","nonce":2613117528}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6150,"qdb_cluster":[6150]},"id":4}]} 2024-03-31T15:07:41.431 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 36 2024-03-31T15:07:41.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:41.438+0000 7f5325e00640 1 -- 172.21.15.16:0/1361613739 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f5300068380 msgr2=0x7f530006a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:41.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:41.438+0000 7f5325e00640 1 --2- 172.21.15.16:0/1361613739 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f5300068380 0x7f530006a840 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f530c0ad000 tx=0x7f530c008040 comp rx=0 tx=0).stop 2024-03-31T15:07:41.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:41.438+0000 7f5325e00640 1 -- 172.21.15.16:0/1361613739 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f532006f920 msgr2=0x7f53200b66b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:41.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:41.438+0000 7f5325e00640 1 --2- 172.21.15.16:0/1361613739 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f532006f920 0x7f53200b66b0 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f531000c490 tx=0x7f531000c950 comp rx=0 tx=0).stop 2024-03-31T15:07:41.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:41.438+0000 7f5325e00640 1 -- 172.21.15.16:0/1361613739 shutdown_connections 2024-03-31T15:07:41.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:41.438+0000 7f5325e00640 1 --2- 172.21.15.16:0/1361613739 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f5300068380 0x7f530006a840 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:41.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:41.438+0000 7f5325e00640 1 --2- 172.21.15.16:0/1361613739 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f5320145e40 0x7f5320142c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:41.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:41.438+0000 7f5325e00640 1 --2- 172.21.15.16:0/1361613739 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f53200d2010 0x7f53201426d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:41.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:41.438+0000 7f5325e00640 1 --2- 172.21.15.16:0/1361613739 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f532006f920 0x7f53200b66b0 unknown :-1 s=CLOSED pgs=747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:41.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:41.438+0000 7f5325e00640 1 -- 172.21.15.16:0/1361613739 >> 172.21.15.16:0/1361613739 conn(0x7f5320067330 msgr2=0x7f532015fd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:07:41.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:41.438+0000 7f5325e00640 1 -- 172.21.15.16:0/1361613739 shutdown_connections 2024-03-31T15:07:41.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:41.438+0000 7f5325e00640 1 -- 172.21.15.16:0/1361613739 wait complete. 2024-03-31T15:07:41.451 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-31T15:07:41.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:41.598+0000 7f8c56c00640 1 Processor -- start 2024-03-31T15:07:41.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:41.598+0000 7f8c56c00640 1 -- start start 2024-03-31T15:07:41.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:41.598+0000 7f8c56c00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f8c500728b0 0x7f8c500b1cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:41.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:41.598+0000 7f8c56c00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f8c500b2200 0x7f8c5015cd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:41.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:41.598+0000 7f8c56c00640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8c50160e70 con 0x7f8c50145d10 2024-03-31T15:07:41.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:41.598+0000 7f8c56c00640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f8c501602f0 con 0x7f8c500b2200 2024-03-31T15:07:41.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:41.598+0000 7f8c56c00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f8c501605d0 con 0x7f8c500728b0 2024-03-31T15:07:41.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:41.598+0000 7f8c54c00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f8c500b2200 0x7f8c5015cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:41.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:41.598+0000 7f8c54c00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f8c500b2200 0x7f8c5015cd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47488/0 (socket says 172.21.15.16:47488) 2024-03-31T15:07:41.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:41.598+0000 7f8c54c00640 1 -- 172.21.15.16:0/1288471384 learned_addr learned my addr 172.21.15.16:0/1288471384 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:07:41.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:41.598+0000 7f8c47e00640 1 --2- 172.21.15.16:0/1288471384 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f8c500728b0 0x7f8c500b1cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:41.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:41.598+0000 7f8c54c00640 1 -- 172.21.15.16:0/1288471384 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f8c500728b0 msgr2=0x7f8c500b1cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:07:41.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:41.598+0000 7f8c54c00640 1 --2- 172.21.15.16:0/1288471384 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f8c500728b0 0x7f8c500b1cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:41.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:41.598+0000 7f8c54c00640 1 -- 172.21.15.16:0/1288471384 >> v1:172.21.15.89:6789/0 conn(0x7f8c50145d10 legacy=0x7f8c500cf790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:07:41.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:41.598+0000 7f8c54c00640 1 -- 172.21.15.16:0/1288471384 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c501608b0 con 0x7f8c500b2200 2024-03-31T15:07:41.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:41.602+0000 7f8c54c00640 1 --2- 172.21.15.16:0/1288471384 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8c500b2200 0x7f8c5015cd50 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f8c48009e50 tx=0x7f8c4801c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62216b8877b19bdc server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:07:41.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:41.606+0000 7f8c4f400640 1 -- 172.21.15.16:0/1288471384 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c480a5670 con 0x7f8c500b2200 2024-03-31T15:07:41.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:41.606+0000 7f8c4f400640 1 -- 172.21.15.16:0/1288471384 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8c480a5810 con 0x7f8c500b2200 2024-03-31T15:07:41.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:41.606+0000 7f8c4f400640 1 -- 172.21.15.16:0/1288471384 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c480a5be0 con 0x7f8c500b2200 2024-03-31T15:07:41.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:41.606+0000 7f8c56c00640 1 -- 172.21.15.16:0/1288471384 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8c500b2200 msgr2=0x7f8c5015cd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:41.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:41.606+0000 7f8c56c00640 1 --2- 172.21.15.16:0/1288471384 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8c500b2200 0x7f8c5015cd50 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f8c48009e50 tx=0x7f8c4801c9f0 comp rx=0 tx=0).stop 2024-03-31T15:07:41.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:41.606+0000 7f8c56c00640 1 -- 172.21.15.16:0/1288471384 shutdown_connections 2024-03-31T15:07:41.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:41.606+0000 7f8c56c00640 1 --2- 172.21.15.16:0/1288471384 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8c500b2200 0x7f8c5015cd50 unknown :-1 s=CLOSED pgs=748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:41.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:41.606+0000 7f8c56c00640 1 --2- 172.21.15.16:0/1288471384 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f8c500728b0 0x7f8c500b1cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:41.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:41.606+0000 7f8c56c00640 1 -- 172.21.15.16:0/1288471384 >> 172.21.15.16:0/1288471384 conn(0x7f8c5006b620 msgr2=0x7f8c5006ba30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:07:41.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:41.606+0000 7f8c56c00640 1 -- 172.21.15.16:0/1288471384 shutdown_connections 2024-03-31T15:07:41.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:41.606+0000 7f8c56c00640 1 -- 172.21.15.16:0/1288471384 wait complete. 2024-03-31T15:07:41.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:41.606+0000 7f8c56c00640 1 Processor -- start 2024-03-31T15:07:41.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:41.606+0000 7f8c56c00640 1 -- start start 2024-03-31T15:07:41.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:41.606+0000 7f8c56c00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8c500728b0 0x7f8c501402a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:41.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:41.606+0000 7f8c56c00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8c500b2200 0x7f8c5015b8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:41.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:41.606+0000 7f8c56c00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f8c50145d10 0x7f8c501386c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:41.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:41.606+0000 7f8c56c00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f8c50072e10 con 0x7f8c500b2200 2024-03-31T15:07:41.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:41.606+0000 7f8c56c00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f8c50072c90 con 0x7f8c500728b0 2024-03-31T15:07:41.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:41.606+0000 7f8c56c00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f8c501690f0 con 0x7f8c50145d10 2024-03-31T15:07:41.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:41.606+0000 7f8c47e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8c500b2200 0x7f8c5015b8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:41.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:41.606+0000 7f8c47e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8c500b2200 0x7f8c5015b8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47496/0 (socket says 172.21.15.16:47496) 2024-03-31T15:07:41.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:41.606+0000 7f8c47e00640 1 -- 172.21.15.16:0/3153308672 learned_addr learned my addr 172.21.15.16:0/3153308672 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:07:41.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:41.610+0000 7f8c4fe00640 1 --2- 172.21.15.16:0/3153308672 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8c500728b0 0x7f8c501402a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:41.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:41.610+0000 7f8c54c00640 1 --2- 172.21.15.16:0/3153308672 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f8c50145d10 0x7f8c501386c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:41.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:41.610+0000 7f8c47e00640 1 -- 172.21.15.16:0/3153308672 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f8c50145d10 msgr2=0x7f8c501386c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:41.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:41.610+0000 7f8c47e00640 1 --2- 172.21.15.16:0/3153308672 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f8c50145d10 0x7f8c501386c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:41.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:41.610+0000 7f8c47e00640 1 -- 172.21.15.16:0/3153308672 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8c500728b0 msgr2=0x7f8c501402a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:41.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:41.610+0000 7f8c47e00640 1 --2- 172.21.15.16:0/3153308672 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8c500728b0 0x7f8c501402a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:41.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:41.610+0000 7f8c47e00640 1 -- 172.21.15.16:0/3153308672 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c50158f00 con 0x7f8c500b2200 2024-03-31T15:07:41.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:41.610+0000 7f8c54c00640 1 --2- 172.21.15.16:0/3153308672 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f8c50145d10 0x7f8c501386c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:07:41.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:41.610+0000 7f8c4fe00640 1 --2- 172.21.15.16:0/3153308672 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8c500728b0 0x7f8c501402a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:07:41.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:41.610+0000 7f8c47e00640 1 --2- 172.21.15.16:0/3153308672 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8c500b2200 0x7f8c5015b8f0 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f8c3800d830 tx=0x7f8c3800dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:07:41.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:41.610+0000 7f8c4cc00640 1 -- 172.21.15.16:0/3153308672 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c3800b6c0 con 0x7f8c500b2200 2024-03-31T15:07:41.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:41.610+0000 7f8c4cc00640 1 -- 172.21.15.16:0/3153308672 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8c38019070 con 0x7f8c500b2200 2024-03-31T15:07:41.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:41.610+0000 7f8c4cc00640 1 -- 172.21.15.16:0/3153308672 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c3800ba60 con 0x7f8c500b2200 2024-03-31T15:07:41.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:41.610+0000 7f8c56c00640 1 -- 172.21.15.16:0/3153308672 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8c5015ad30 con 0x7f8c500b2200 2024-03-31T15:07:41.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:41.610+0000 7f8c56c00640 1 -- 172.21.15.16:0/3153308672 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8c5015b2f0 con 0x7f8c500b2200 2024-03-31T15:07:41.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:41.610+0000 7f8c56c00640 1 -- 172.21.15.16:0/3153308672 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8c50069950 con 0x7f8c500b2200 2024-03-31T15:07:41.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:41.610+0000 7f8c4cc00640 1 -- 172.21.15.16:0/3153308672 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f8c3801e460 con 0x7f8c500b2200 2024-03-31T15:07:41.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:41.610+0000 7f8c4cc00640 1 --2- 172.21.15.16:0/3153308672 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f8c2c068380 0x7f8c2c06a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:41.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:41.610+0000 7f8c4cc00640 1 -- 172.21.15.16:0/3153308672 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 5709+0+0 (secure 0 0 0) 0x7f8c38087090 con 0x7f8c500b2200 2024-03-31T15:07:41.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:41.614+0000 7f8c4fe00640 1 --2- 172.21.15.16:0/3153308672 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f8c2c068380 0x7f8c2c06a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:41.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:41.614+0000 7f8c4fe00640 1 --2- 172.21.15.16:0/3153308672 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f8c2c068380 0x7f8c2c06a840 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f8c5015b620 tx=0x7f8c40008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:07:41.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:41.622+0000 7f8c4cc00640 1 -- 172.21.15.16:0/3153308672 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f8c38057640 con 0x7f8c500b2200 2024-03-31T15:07:41.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:41.786+0000 7f8c56c00640 1 -- 172.21.15.16:0/3153308672 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f8c5015cc50 con 0x7f8c500b2200 2024-03-31T15:07:41.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:41.790+0000 7f8c4cc00640 1 -- 172.21.15.16:0/3153308672 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v56) ==== 74+0+16593 (secure 0 0 0) 0x7f8c38056f40 con 0x7f8c500b2200 2024-03-31T15:07:41.784 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:07:41.785 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":56,"fsid":"286eb5da-fb47-4550-b290-d9519c13b9a5","created":"2024-03-31T15:03:11.438952+0000","modified":"2024-03-31T15:06:27.274379+0000","last_up_change":"2024-03-31T15:03:22.885535+0000","last_in_change":"2024-03-31T15:03:13.941667+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-31T15:03:22.550787+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-03-31T15:06:24.043886+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"55","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-03-31T15:06:24.613108+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"56","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"92a66450-ef19-40d8-ac3d-d760b23b0bda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6801","nonce":932712244}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6803","nonce":932712244}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6807","nonce":932712244}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6805","nonce":932712244}]},"public_addr":"172.21.15.16:6801/932712244","cluster_addr":"172.21.15.16:6803/932712244","heartbeat_back_addr":"172.21.15.16:6807/932712244","heartbeat_front_addr":"172.21.15.16:6805/932712244","state":["exists","up"]},{"osd":1,"uuid":"da9d1f2b-1968-4959-b843-faa231988e92","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6810","nonce":1398737533}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6811","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6813","nonce":1398737533}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6821","nonce":1398737533}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6815","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6816","nonce":1398737533}]},"public_addr":"172.21.15.16:6810/1398737533","cluster_addr":"172.21.15.16:6813/1398737533","heartbeat_back_addr":"172.21.15.16:6821/1398737533","heartbeat_front_addr":"172.21.15.16:6816/1398737533","state":["exists","up"]},{"osd":2,"uuid":"01a73662-9a21-4b1f-bcbb-42b9636a4638","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6819","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6822","nonce":3018526646}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6825","nonce":3018526646}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6831","nonce":3018526646}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6827","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6829","nonce":3018526646}]},"public_addr":"172.21.15.16:6822/3018526646","cluster_addr":"172.21.15.16:6825/3018526646","heartbeat_back_addr":"172.21.15.16:6831/3018526646","heartbeat_front_addr":"172.21.15.16:6829/3018526646","state":["exists","up"]},{"osd":3,"uuid":"c7771fd3-e18e-4500-ac38-23cf10ebcd75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6809","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6812","nonce":3360352974}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6817","nonce":3360352974}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6828","nonce":3360352974}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6823","nonce":3360352974}]},"public_addr":"172.21.15.16:6812/3360352974","cluster_addr":"172.21.15.16:6817/3360352974","heartbeat_back_addr":"172.21.15.16:6828/3360352974","heartbeat_front_addr":"172.21.15.16:6823/3360352974","state":["exists","up"]},{"osd":4,"uuid":"88260730-1482-4fbd-a40f-cb1124e9df4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6801","nonce":1856844856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6803","nonce":1856844856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6807","nonce":1856844856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6805","nonce":1856844856}]},"public_addr":"172.21.15.89:6801/1856844856","cluster_addr":"172.21.15.89:6803/1856844856","heartbeat_back_addr":"172.21.15.89:6807/1856844856","heartbeat_front_addr":"172.21.15.89:6805/1856844856","state":["exists","up"]},{"osd":5,"uuid":"4cb2c00c-fae0-4dd8-9d27-ec4f0093ccf7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6814","nonce":50395527}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6817","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6820","nonce":50395527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6831","nonce":50395527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6823","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6826","nonce":50395527}]},"public_addr":"172.21.15.89:6814/50395527","cluster_addr":"172.21.15.89:6820/50395527","heartbeat_back_addr":"172.21.15.89:6831/50395527","heartbeat_front_addr":"172.21.15.89:6826/50395527","state":["exists","up"]},{"osd":6,"uuid":"81ce0c19-2554-4cae-acbe-66a05e9e9b9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6809","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6812","nonce":3470471752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6815","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6818","nonce":3470471752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6827","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6829","nonce":3470471752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6821","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6824","nonce":3470471752}]},"public_addr":"172.21.15.89:6812/3470471752","cluster_addr":"172.21.15.89:6818/3470471752","heartbeat_back_addr":"172.21.15.89:6829/3470471752","heartbeat_front_addr":"172.21.15.89:6824/3470471752","state":["exists","up"]},{"osd":7,"uuid":"85e4313b-d37d-442e-8042-153bf20a847d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6811","nonce":1796778630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6813","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6816","nonce":1796778630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6825","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6828","nonce":1796778630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6819","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6822","nonce":1796778630}]},"public_addr":"172.21.15.89:6811/1796778630","cluster_addr":"172.21.15.89:6816/1796778630","heartbeat_back_addr":"172.21.15.89:6828/1796778630","heartbeat_front_addr":"172.21.15.89:6822/1796778630","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.439088+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.529970+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.680512+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:21.044161+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.582111+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.536558+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.409522+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.590229+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-31T15:07:41.787 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:41.790+0000 7f8c56c00640 1 -- 172.21.15.16:0/3153308672 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f8c2c068380 msgr2=0x7f8c2c06a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:41.787 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:41.790+0000 7f8c56c00640 1 --2- 172.21.15.16:0/3153308672 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f8c2c068380 0x7f8c2c06a840 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f8c5015b620 tx=0x7f8c40008040 comp rx=0 tx=0).stop 2024-03-31T15:07:41.787 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:41.790+0000 7f8c56c00640 1 -- 172.21.15.16:0/3153308672 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8c500b2200 msgr2=0x7f8c5015b8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:41.787 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:41.790+0000 7f8c56c00640 1 --2- 172.21.15.16:0/3153308672 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8c500b2200 0x7f8c5015b8f0 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f8c3800d830 tx=0x7f8c3800dcf0 comp rx=0 tx=0).stop 2024-03-31T15:07:41.787 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:41.790+0000 7f8c56c00640 1 -- 172.21.15.16:0/3153308672 shutdown_connections 2024-03-31T15:07:41.787 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:41.790+0000 7f8c56c00640 1 --2- 172.21.15.16:0/3153308672 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f8c2c068380 0x7f8c2c06a840 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:41.787 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:41.790+0000 7f8c56c00640 1 --2- 172.21.15.16:0/3153308672 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f8c50145d10 0x7f8c501386c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:41.787 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:41.790+0000 7f8c56c00640 1 --2- 172.21.15.16:0/3153308672 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8c500b2200 0x7f8c5015b8f0 unknown :-1 s=CLOSED pgs=749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:41.787 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:41.790+0000 7f8c56c00640 1 --2- 172.21.15.16:0/3153308672 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8c500728b0 0x7f8c501402a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:41.787 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:41.790+0000 7f8c56c00640 1 -- 172.21.15.16:0/3153308672 >> 172.21.15.16:0/3153308672 conn(0x7f8c5006b620 msgr2=0x7f8c500ce9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:07:41.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:41.790+0000 7f8c56c00640 1 -- 172.21.15.16:0/3153308672 shutdown_connections 2024-03-31T15:07:41.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:41.790+0000 7f8c56c00640 1 -- 172.21.15.16:0/3153308672 wait complete. 2024-03-31T15:07:41.806 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-31T15:03:22.550787+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 8, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-31T15:06:24.043886+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '55', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 9, 'pool_name': 'cephfs_data', 'create_time': '2024-03-31T15:06:24.613108+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '56', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-31T15:07:41.807 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-31T15:07:41.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:41.954+0000 7f47dbe00640 1 Processor -- start 2024-03-31T15:07:41.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:41.954+0000 7f47dbe00640 1 -- start start 2024-03-31T15:07:41.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:41.958+0000 7f47dbe00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f47dc0d03e0 0x7f47dc166ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:41.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:41.958+0000 7f47dbe00640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f47dc0b57e0 con 0x7f47dc15c330 2024-03-31T15:07:41.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:41.958+0000 7f47dbe00640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f47dc0b5ac0 con 0x7f47dc0b5330 2024-03-31T15:07:41.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:41.958+0000 7f47dbe00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f47dc0adda0 con 0x7f47dc0d03e0 2024-03-31T15:07:41.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:41.958+0000 7f47daa00640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f47dc15c330 0x7f47dc0b4c20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:53816/0 (socket says 172.21.15.16:53816) 2024-03-31T15:07:41.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:41.958+0000 7f47daa00640 1 -- 172.21.15.16:0/1400244332 learned_addr learned my addr 172.21.15.16:0/1400244332 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:07:41.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:41.958+0000 7f47db400640 1 --2- 172.21.15.16:0/1400244332 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f47dc0d03e0 0x7f47dc166ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:41.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:41.958+0000 7f47da000640 1 -- 172.21.15.16:0/1400244332 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 934870264 0 0) 0x7f47dc0b57e0 con 0x7f47dc15c330 2024-03-31T15:07:41.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:41.958+0000 7f47da000640 1 -- 172.21.15.16:0/1400244332 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f47c0003530 con 0x7f47dc15c330 2024-03-31T15:07:41.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:41.958+0000 7f47da000640 1 -- 172.21.15.16:0/1400244332 <== mon.1 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1019484257 0 0) 0x7f47dc0b5ac0 con 0x7f47dc0b5330 2024-03-31T15:07:41.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:41.958+0000 7f47db400640 1 -- 172.21.15.16:0/1400244332 >> v1:172.21.15.89:6789/0 conn(0x7f47dc0b5330 legacy=0x7f47dc06c960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:41.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:41.958+0000 7f47db400640 1 -- 172.21.15.16:0/1400244332 >> v1:172.21.15.16:6789/0 conn(0x7f47dc15c330 legacy=0x7f47dc0b4c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:41.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:41.958+0000 7f47db400640 1 -- 172.21.15.16:0/1400244332 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47dc0b5da0 con 0x7f47dc0d03e0 2024-03-31T15:07:41.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:41.962+0000 7f47db400640 1 --2- 172.21.15.16:0/1400244332 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f47dc0d03e0 0x7f47dc166ef0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f47d4009dc0 tx=0x7f47d401c990 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c6927a255adb99de server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:07:41.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:41.962+0000 7f47da000640 1 -- 172.21.15.16:0/1400244332 <== mon.2 v2:172.21.15.89:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47d40b2070 con 0x7f47dc0d03e0 2024-03-31T15:07:41.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:41.962+0000 7f47da000640 1 -- 172.21.15.16:0/1400244332 <== mon.2 v2:172.21.15.89:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f47d40a58d0 con 0x7f47dc0d03e0 2024-03-31T15:07:41.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:41.962+0000 7f47da000640 1 -- 172.21.15.16:0/1400244332 <== mon.2 v2:172.21.15.89:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47d40a5ca0 con 0x7f47dc0d03e0 2024-03-31T15:07:41.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:41.962+0000 7f47dbe00640 1 -- 172.21.15.16:0/1400244332 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f47dc0d03e0 msgr2=0x7f47dc166ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:41.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:41.962+0000 7f47dbe00640 1 --2- 172.21.15.16:0/1400244332 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f47dc0d03e0 0x7f47dc166ef0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f47d4009dc0 tx=0x7f47d401c990 comp rx=0 tx=0).stop 2024-03-31T15:07:41.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:41.962+0000 7f47dbe00640 1 -- 172.21.15.16:0/1400244332 shutdown_connections 2024-03-31T15:07:41.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:41.962+0000 7f47dbe00640 1 --2- 172.21.15.16:0/1400244332 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f47dc0d03e0 0x7f47dc166ef0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:41.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:41.962+0000 7f47dbe00640 1 -- 172.21.15.16:0/1400244332 >> 172.21.15.16:0/1400244332 conn(0x7f47dc0b2c90 msgr2=0x7f47dc0b30a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:07:41.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:41.962+0000 7f47dbe00640 1 -- 172.21.15.16:0/1400244332 shutdown_connections 2024-03-31T15:07:41.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:41.962+0000 7f47dbe00640 1 -- 172.21.15.16:0/1400244332 wait complete. 2024-03-31T15:07:41.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:41.962+0000 7f47dbe00640 1 Processor -- start 2024-03-31T15:07:41.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:41.962+0000 7f47dbe00640 1 -- start start 2024-03-31T15:07:41.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:41.966+0000 7f47dbe00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f47dc0b5330 0x7f47dc142700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:41.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:41.966+0000 7f47dbe00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f47dc0d03e0 0x7f47dc15c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:41.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:41.966+0000 7f47dbe00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f47dc15c330 0x7f47dc158540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:41.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:41.966+0000 7f47dbe00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f47dc06d1f0 con 0x7f47dc0d03e0 2024-03-31T15:07:41.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:41.966+0000 7f47dbe00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f47dc06d070 con 0x7f47dc0b5330 2024-03-31T15:07:41.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:41.966+0000 7f47dbe00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f47dc169c50 con 0x7f47dc15c330 2024-03-31T15:07:41.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:41.966+0000 7f47d3e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f47dc0d03e0 0x7f47dc15c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:41.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:41.966+0000 7f47d3e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f47dc0d03e0 0x7f47dc15c9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47502/0 (socket says 172.21.15.16:47502) 2024-03-31T15:07:41.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:41.966+0000 7f47d3e00640 1 -- 172.21.15.16:0/2435280665 learned_addr learned my addr 172.21.15.16:0/2435280665 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:07:41.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:41.966+0000 7f47db400640 1 --2- 172.21.15.16:0/2435280665 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f47dc15c330 0x7f47dc158540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:41.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:41.966+0000 7f47daa00640 1 --2- 172.21.15.16:0/2435280665 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f47dc0b5330 0x7f47dc142700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:41.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:41.966+0000 7f47d3e00640 1 -- 172.21.15.16:0/2435280665 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f47dc15c330 msgr2=0x7f47dc158540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:41.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:41.966+0000 7f47d3e00640 1 --2- 172.21.15.16:0/2435280665 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f47dc15c330 0x7f47dc158540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:41.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:41.966+0000 7f47d3e00640 1 -- 172.21.15.16:0/2435280665 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f47dc0b5330 msgr2=0x7f47dc142700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:41.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:41.966+0000 7f47d3e00640 1 --2- 172.21.15.16:0/2435280665 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f47dc0b5330 0x7f47dc142700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:41.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:41.966+0000 7f47d3e00640 1 -- 172.21.15.16:0/2435280665 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47dc06e670 con 0x7f47dc0d03e0 2024-03-31T15:07:41.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:41.966+0000 7f47db400640 1 --2- 172.21.15.16:0/2435280665 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f47dc15c330 0x7f47dc158540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:07:41.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:41.966+0000 7f47daa00640 1 --2- 172.21.15.16:0/2435280665 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f47dc0b5330 0x7f47dc142700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:07:41.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:41.966+0000 7f47d3e00640 1 --2- 172.21.15.16:0/2435280665 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f47dc0d03e0 0x7f47dc15c9b0 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f47c400c550 tx=0x7f47c400ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:07:41.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:41.966+0000 7f47d3400640 1 -- 172.21.15.16:0/2435280665 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47c4017070 con 0x7f47dc0d03e0 2024-03-31T15:07:41.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:41.966+0000 7f47d3400640 1 -- 172.21.15.16:0/2435280665 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f47c4003600 con 0x7f47dc0d03e0 2024-03-31T15:07:41.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:41.966+0000 7f47d3400640 1 -- 172.21.15.16:0/2435280665 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47c40039f0 con 0x7f47dc0d03e0 2024-03-31T15:07:41.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:41.966+0000 7f47dbe00640 1 -- 172.21.15.16:0/2435280665 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f47dc06e950 con 0x7f47dc0d03e0 2024-03-31T15:07:41.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:41.966+0000 7f47dbe00640 1 -- 172.21.15.16:0/2435280665 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f47dc166f80 con 0x7f47dc0d03e0 2024-03-31T15:07:41.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:41.966+0000 7f47dbe00640 1 -- 172.21.15.16:0/2435280665 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f47dc04f8c0 con 0x7f47dc0d03e0 2024-03-31T15:07:41.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:41.966+0000 7f47d3400640 1 -- 172.21.15.16:0/2435280665 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f47c401a070 con 0x7f47dc0d03e0 2024-03-31T15:07:41.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:41.970+0000 7f47d3400640 1 --2- 172.21.15.16:0/2435280665 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f47b8068380 0x7f47b806a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:41.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:41.970+0000 7f47d3400640 1 -- 172.21.15.16:0/2435280665 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 5709+0+0 (secure 0 0 0) 0x7f47c4085db0 con 0x7f47dc0d03e0 2024-03-31T15:07:41.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:41.970+0000 7f47daa00640 1 --2- 172.21.15.16:0/2435280665 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f47b8068380 0x7f47b806a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:41.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:41.970+0000 7f47daa00640 1 --2- 172.21.15.16:0/2435280665 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f47b8068380 0x7f47b806a840 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f47cc005890 tx=0x7f47cc009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:07:41.968 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:41.974+0000 7f47d3400640 1 -- 172.21.15.16:0/2435280665 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f47c408a070 con 0x7f47dc0d03e0 2024-03-31T15:07:42.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:42.134+0000 7f47dbe00640 1 -- 172.21.15.16:0/2435280665 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f47dc06cce0 con 0x7f47dc0d03e0 2024-03-31T15:07:42.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:42.138+0000 7f47d3400640 1 -- 172.21.15.16:0/2435280665 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v56) ==== 93+0+10 (secure 0 0 0) 0x7f47c4056360 con 0x7f47dc0d03e0 2024-03-31T15:07:42.132 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2024-03-31T15:07:42.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:42.138+0000 7f47dbe00640 1 -- 172.21.15.16:0/2435280665 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f47b8068380 msgr2=0x7f47b806a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:42.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:42.138+0000 7f47dbe00640 1 --2- 172.21.15.16:0/2435280665 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f47b8068380 0x7f47b806a840 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f47cc005890 tx=0x7f47cc009040 comp rx=0 tx=0).stop 2024-03-31T15:07:42.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:42.138+0000 7f47dbe00640 1 -- 172.21.15.16:0/2435280665 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f47dc0d03e0 msgr2=0x7f47dc15c9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:42.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:42.138+0000 7f47dbe00640 1 --2- 172.21.15.16:0/2435280665 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f47dc0d03e0 0x7f47dc15c9b0 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f47c400c550 tx=0x7f47c400ca10 comp rx=0 tx=0).stop 2024-03-31T15:07:42.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:42.138+0000 7f47dbe00640 1 -- 172.21.15.16:0/2435280665 shutdown_connections 2024-03-31T15:07:42.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:42.138+0000 7f47dbe00640 1 --2- 172.21.15.16:0/2435280665 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f47b8068380 0x7f47b806a840 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:42.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:42.138+0000 7f47dbe00640 1 --2- 172.21.15.16:0/2435280665 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f47dc15c330 0x7f47dc158540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:42.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:42.138+0000 7f47dbe00640 1 --2- 172.21.15.16:0/2435280665 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f47dc0d03e0 0x7f47dc15c9b0 unknown :-1 s=CLOSED pgs=751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:42.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:42.138+0000 7f47dbe00640 1 --2- 172.21.15.16:0/2435280665 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f47dc0b5330 0x7f47dc142700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:42.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:42.138+0000 7f47dbe00640 1 -- 172.21.15.16:0/2435280665 >> 172.21.15.16:0/2435280665 conn(0x7f47dc0b2c90 msgr2=0x7f47dc0afb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:07:42.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:42.138+0000 7f47dbe00640 1 -- 172.21.15.16:0/2435280665 shutdown_connections 2024-03-31T15:07:42.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:42.138+0000 7f47dbe00640 1 -- 172.21.15.16:0/2435280665 wait complete. 2024-03-31T15:07:42.152 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-31T15:07:42.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:42.298+0000 7ffba1c00640 1 Processor -- start 2024-03-31T15:07:42.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:42.298+0000 7ffba1c00640 1 -- start start 2024-03-31T15:07:42.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:42.302+0000 7ffba1c00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ffb9c070030 0x7ffb9c143c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:42.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:42.302+0000 7ffba1c00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7ffb9c141590 0x7ffb9c144170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:42.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:42.302+0000 7ffba1c00640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffb9c1446b0 con 0x7ffb9c1411b0 2024-03-31T15:07:42.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:42.302+0000 7ffba1c00640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7ffb9c15f910 con 0x7ffb9c070030 2024-03-31T15:07:42.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:42.302+0000 7ffba1c00640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7ffb9c162910 con 0x7ffb9c141590 2024-03-31T15:07:42.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:42.302+0000 7ffb92000640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ffb9c070030 0x7ffb9c143c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:42.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:42.302+0000 7ffb92000640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ffb9c070030 0x7ffb9c143c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47514/0 (socket says 172.21.15.16:47514) 2024-03-31T15:07:42.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:42.302+0000 7ffb92000640 1 -- 172.21.15.16:0/2385102240 learned_addr learned my addr 172.21.15.16:0/2385102240 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:07:42.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:42.302+0000 7ffb9b400640 1 --2- 172.21.15.16:0/2385102240 >> v2:172.21.15.89:3300/0 conn(0x7ffb9c141590 0x7ffb9c144170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:42.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:42.302+0000 7ffb92000640 1 -- 172.21.15.16:0/2385102240 >> v2:172.21.15.89:3300/0 conn(0x7ffb9c141590 msgr2=0x7ffb9c144170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:07:42.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:42.302+0000 7ffb92000640 1 --2- 172.21.15.16:0/2385102240 >> v2:172.21.15.89:3300/0 conn(0x7ffb9c141590 0x7ffb9c144170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:42.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:42.302+0000 7ffb92000640 1 -- 172.21.15.16:0/2385102240 >> v1:172.21.15.16:6789/0 conn(0x7ffb9c1411b0 legacy=0x7ffb9c06f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:42.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:42.302+0000 7ffb92000640 1 -- 172.21.15.16:0/2385102240 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffb9c144980 con 0x7ffb9c070030 2024-03-31T15:07:42.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:42.302+0000 7ffb9a000640 1 -- 172.21.15.16:0/2385102240 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 76747457 0 0) 0x7ffb9c1446b0 con 0x7ffb9c1411b0 2024-03-31T15:07:42.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:42.306+0000 7ffb92000640 1 --2- 172.21.15.16:0/2385102240 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffb9c070030 0x7ffb9c143c30 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7ffb84009870 tx=0x7ffb8401c4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4880535e1fc28423 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:07:42.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:42.306+0000 7ffb9a000640 1 -- 172.21.15.16:0/2385102240 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb840aa650 con 0x7ffb9c070030 2024-03-31T15:07:42.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:42.306+0000 7ffb9a000640 1 -- 172.21.15.16:0/2385102240 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffb840af070 con 0x7ffb9c070030 2024-03-31T15:07:42.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:42.306+0000 7ffb9a000640 1 -- 172.21.15.16:0/2385102240 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb840aaa10 con 0x7ffb9c070030 2024-03-31T15:07:42.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:42.306+0000 7ffba1c00640 1 -- 172.21.15.16:0/2385102240 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffb9c070030 msgr2=0x7ffb9c143c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:42.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:42.306+0000 7ffba1c00640 1 --2- 172.21.15.16:0/2385102240 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffb9c070030 0x7ffb9c143c30 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7ffb84009870 tx=0x7ffb8401c4f0 comp rx=0 tx=0).stop 2024-03-31T15:07:42.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:42.306+0000 7ffba1c00640 1 -- 172.21.15.16:0/2385102240 shutdown_connections 2024-03-31T15:07:42.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:42.306+0000 7ffba1c00640 1 --2- 172.21.15.16:0/2385102240 >> v2:172.21.15.89:3300/0 conn(0x7ffb9c141590 0x7ffb9c144170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:42.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:42.306+0000 7ffba1c00640 1 --2- 172.21.15.16:0/2385102240 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffb9c070030 0x7ffb9c143c30 unknown :-1 s=CLOSED pgs=753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:42.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:42.306+0000 7ffba1c00640 1 -- 172.21.15.16:0/2385102240 >> 172.21.15.16:0/2385102240 conn(0x7ffb9c067930 msgr2=0x7ffb9c0b3ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:07:42.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:42.306+0000 7ffba1c00640 1 -- 172.21.15.16:0/2385102240 shutdown_connections 2024-03-31T15:07:42.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:42.306+0000 7ffba1c00640 1 -- 172.21.15.16:0/2385102240 wait complete. 2024-03-31T15:07:42.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:42.310+0000 7ffba1c00640 1 Processor -- start 2024-03-31T15:07:42.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:42.310+0000 7ffba1c00640 1 -- start start 2024-03-31T15:07:42.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:42.310+0000 7ffba1c00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ffb9c070030 0x7ffb9c1551b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:42.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:42.310+0000 7ffba1c00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffb9c1411b0 0x7ffb9c13e040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:42.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:42.310+0000 7ffba1c00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffb9c141590 0x7ffb9c13e580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:42.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:42.310+0000 7ffba1c00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7ffb9c15f110 con 0x7ffb9c1411b0 2024-03-31T15:07:42.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:42.310+0000 7ffba1c00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7ffb9c162da0 con 0x7ffb9c141590 2024-03-31T15:07:42.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:42.310+0000 7ffba1c00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7ffb9c15f290 con 0x7ffb9c070030 2024-03-31T15:07:42.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:42.310+0000 7ffb92000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffb9c1411b0 0x7ffb9c13e040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:42.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:42.310+0000 7ffb92000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffb9c1411b0 0x7ffb9c13e040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47516/0 (socket says 172.21.15.16:47516) 2024-03-31T15:07:42.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:42.310+0000 7ffb92000640 1 -- 172.21.15.16:0/2553821142 learned_addr learned my addr 172.21.15.16:0/2553821142 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:07:42.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:42.310+0000 7ffb9b400640 1 --2- 172.21.15.16:0/2553821142 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffb9c141590 0x7ffb9c13e580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:42.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:42.310+0000 7ffb9aa00640 1 --2- 172.21.15.16:0/2553821142 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ffb9c070030 0x7ffb9c1551b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:42.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:42.310+0000 7ffb92000640 1 -- 172.21.15.16:0/2553821142 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ffb9c070030 msgr2=0x7ffb9c1551b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:42.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:42.310+0000 7ffb92000640 1 --2- 172.21.15.16:0/2553821142 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ffb9c070030 0x7ffb9c1551b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:42.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:42.310+0000 7ffb92000640 1 -- 172.21.15.16:0/2553821142 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffb9c141590 msgr2=0x7ffb9c13e580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:42.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:42.310+0000 7ffb92000640 1 --2- 172.21.15.16:0/2553821142 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffb9c141590 0x7ffb9c13e580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:42.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:42.310+0000 7ffb92000640 1 -- 172.21.15.16:0/2553821142 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffb9c13ec00 con 0x7ffb9c1411b0 2024-03-31T15:07:42.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:42.310+0000 7ffb92000640 1 --2- 172.21.15.16:0/2553821142 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffb9c1411b0 0x7ffb9c13e040 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7ffb840ac040 tx=0x7ffb8401ad30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:07:42.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:42.310+0000 7ffb93400640 1 -- 172.21.15.16:0/2553821142 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb840bd5f0 con 0x7ffb9c1411b0 2024-03-31T15:07:42.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:42.310+0000 7ffb93400640 1 -- 172.21.15.16:0/2553821142 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffb840a6040 con 0x7ffb9c1411b0 2024-03-31T15:07:42.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:42.310+0000 7ffb93400640 1 -- 172.21.15.16:0/2553821142 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb840bd9f0 con 0x7ffb9c1411b0 2024-03-31T15:07:42.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:42.310+0000 7ffba1c00640 1 -- 172.21.15.16:0/2553821142 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffb9c15aac0 con 0x7ffb9c1411b0 2024-03-31T15:07:42.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:42.310+0000 7ffba1c00640 1 -- 172.21.15.16:0/2553821142 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffb9c15b080 con 0x7ffb9c1411b0 2024-03-31T15:07:42.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:42.314+0000 7ffba1c00640 1 -- 172.21.15.16:0/2553821142 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffb9c069950 con 0x7ffb9c1411b0 2024-03-31T15:07:42.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:42.314+0000 7ffb93400640 1 -- 172.21.15.16:0/2553821142 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7ffb840b33f0 con 0x7ffb9c1411b0 2024-03-31T15:07:42.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:42.314+0000 7ffb93400640 1 --2- 172.21.15.16:0/2553821142 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7ffb78068380 0x7ffb7806a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:42.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:42.314+0000 7ffb93400640 1 -- 172.21.15.16:0/2553821142 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 5709+0+0 (secure 0 0 0) 0x7ffb84120d90 con 0x7ffb9c1411b0 2024-03-31T15:07:42.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:42.314+0000 7ffb9aa00640 1 --2- 172.21.15.16:0/2553821142 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7ffb78068380 0x7ffb7806a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:42.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:42.314+0000 7ffb9aa00640 1 --2- 172.21.15.16:0/2553821142 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7ffb78068380 0x7ffb7806a840 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7ffb8c0012f0 tx=0x7ffb8c009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:07:42.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:42.318+0000 7ffb93400640 1 -- 172.21.15.16:0/2553821142 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7ffb84016110 con 0x7ffb9c1411b0 2024-03-31T15:07:42.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:42.486+0000 7ffba1c00640 1 -- 172.21.15.16:0/2553821142 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7ffb9c0b4a20 con 0x7ffb9c1411b0 2024-03-31T15:07:42.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:42.486+0000 7ffb93400640 1 -- 172.21.15.16:0/2553821142 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v56) ==== 104+0+11 (secure 0 0 0) 0x7ffb840f0cf0 con 0x7ffb9c1411b0 2024-03-31T15:07:42.483 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2024-03-31T15:07:42.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:42.490+0000 7ffba1c00640 1 -- 172.21.15.16:0/2553821142 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7ffb78068380 msgr2=0x7ffb7806a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:42.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:42.490+0000 7ffba1c00640 1 --2- 172.21.15.16:0/2553821142 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7ffb78068380 0x7ffb7806a840 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7ffb8c0012f0 tx=0x7ffb8c009040 comp rx=0 tx=0).stop 2024-03-31T15:07:42.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:42.490+0000 7ffba1c00640 1 -- 172.21.15.16:0/2553821142 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffb9c1411b0 msgr2=0x7ffb9c13e040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:42.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:42.490+0000 7ffba1c00640 1 --2- 172.21.15.16:0/2553821142 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffb9c1411b0 0x7ffb9c13e040 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7ffb840ac040 tx=0x7ffb8401ad30 comp rx=0 tx=0).stop 2024-03-31T15:07:42.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:42.490+0000 7ffba1c00640 1 -- 172.21.15.16:0/2553821142 shutdown_connections 2024-03-31T15:07:42.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:42.490+0000 7ffba1c00640 1 --2- 172.21.15.16:0/2553821142 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7ffb78068380 0x7ffb7806a840 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:42.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:42.490+0000 7ffba1c00640 1 --2- 172.21.15.16:0/2553821142 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffb9c141590 0x7ffb9c13e580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:42.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:42.490+0000 7ffba1c00640 1 --2- 172.21.15.16:0/2553821142 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffb9c1411b0 0x7ffb9c13e040 unknown :-1 s=CLOSED pgs=754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:42.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:42.490+0000 7ffba1c00640 1 --2- 172.21.15.16:0/2553821142 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ffb9c070030 0x7ffb9c1551b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:42.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:42.490+0000 7ffba1c00640 1 -- 172.21.15.16:0/2553821142 >> 172.21.15.16:0/2553821142 conn(0x7ffb9c067930 msgr2=0x7ffb9c15fb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:07:42.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:42.490+0000 7ffba1c00640 1 -- 172.21.15.16:0/2553821142 shutdown_connections 2024-03-31T15:07:42.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:42.490+0000 7ffba1c00640 1 -- 172.21.15.16:0/2553821142 wait complete. 2024-03-31T15:07:42.503 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-31T15:07:42.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:42.654+0000 7f5c72000640 1 Processor -- start 2024-03-31T15:07:42.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:42.654+0000 7f5c72000640 1 -- start start 2024-03-31T15:07:42.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:42.654+0000 7f5c72000640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f5c6c06b620 0x7f5c6c06ba00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:42.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:42.654+0000 7f5c72000640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f5c6c0b0580 0x7f5c6c06e320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:42.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:42.654+0000 7f5c72000640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5c6c06bf40 con 0x7f5c6c072480 2024-03-31T15:07:42.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:42.654+0000 7f5c72000640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f5c6c0ae2e0 con 0x7f5c6c0b0580 2024-03-31T15:07:42.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:42.654+0000 7f5c72000640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f5c6c0ae5c0 con 0x7f5c6c06b620 2024-03-31T15:07:42.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:42.654+0000 7f5c70c00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f5c6c06b620 0x7f5c6c06ba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:42.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:42.654+0000 7f5c6be00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f5c6c0b0580 0x7f5c6c06e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:42.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:42.654+0000 7f5c70c00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f5c6c06b620 0x7f5c6c06ba00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3301/0 says I am v2:172.21.15.16:55042/0 (socket says 172.21.15.16:55042) 2024-03-31T15:07:42.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:42.654+0000 7f5c70c00640 1 -- 172.21.15.16:0/1812360240 learned_addr learned my addr 172.21.15.16:0/1812360240 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:07:42.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:42.654+0000 7f5c70c00640 1 -- 172.21.15.16:0/1812360240 >> v2:172.21.15.89:3300/0 conn(0x7f5c6c0b0580 msgr2=0x7f5c6c06e320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:07:42.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:42.654+0000 7f5c70c00640 1 --2- 172.21.15.16:0/1812360240 >> v2:172.21.15.89:3300/0 conn(0x7f5c6c0b0580 0x7f5c6c06e320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:42.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:42.654+0000 7f5c70c00640 1 -- 172.21.15.16:0/1812360240 >> v1:172.21.15.89:6789/0 conn(0x7f5c6c072480 legacy=0x7f5c6c072910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:42.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:42.654+0000 7f5c70c00640 1 -- 172.21.15.16:0/1812360240 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c6c0ae8a0 con 0x7f5c6c06b620 2024-03-31T15:07:42.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:42.658+0000 7f5c70c00640 1 --2- 172.21.15.16:0/1812360240 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f5c6c06b620 0x7f5c6c06ba00 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f5c60009870 tx=0x7f5c6001c4a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=614de3097bb3294d server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:07:42.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:42.658+0000 7f5c6b400640 1 -- 172.21.15.16:0/1812360240 <== mon.2 v2:172.21.15.89:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c600ab660 con 0x7f5c6c06b620 2024-03-31T15:07:42.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:42.658+0000 7f5c6b400640 1 -- 172.21.15.16:0/1812360240 <== mon.2 v2:172.21.15.89:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5c600b0070 con 0x7f5c6c06b620 2024-03-31T15:07:42.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:42.658+0000 7f5c6b400640 1 -- 172.21.15.16:0/1812360240 <== mon.2 v2:172.21.15.89:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c600aba20 con 0x7f5c6c06b620 2024-03-31T15:07:42.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:42.658+0000 7f5c72000640 1 -- 172.21.15.16:0/1812360240 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f5c6c06b620 msgr2=0x7f5c6c06ba00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:42.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:42.658+0000 7f5c72000640 1 --2- 172.21.15.16:0/1812360240 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f5c6c06b620 0x7f5c6c06ba00 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f5c60009870 tx=0x7f5c6001c4a0 comp rx=0 tx=0).stop 2024-03-31T15:07:42.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:42.658+0000 7f5c72000640 1 -- 172.21.15.16:0/1812360240 shutdown_connections 2024-03-31T15:07:42.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:42.658+0000 7f5c72000640 1 --2- 172.21.15.16:0/1812360240 >> v2:172.21.15.89:3300/0 conn(0x7f5c6c0b0580 0x7f5c6c06e320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:42.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:42.658+0000 7f5c72000640 1 --2- 172.21.15.16:0/1812360240 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f5c6c06b620 0x7f5c6c06ba00 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:42.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:42.658+0000 7f5c72000640 1 -- 172.21.15.16:0/1812360240 >> 172.21.15.16:0/1812360240 conn(0x7f5c6c0b2b30 msgr2=0x7f5c6c0b2f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:07:42.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:42.658+0000 7f5c72000640 1 -- 172.21.15.16:0/1812360240 shutdown_connections 2024-03-31T15:07:42.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:42.658+0000 7f5c72000640 1 -- 172.21.15.16:0/1812360240 wait complete. 2024-03-31T15:07:42.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:42.662+0000 7f5c72000640 1 Processor -- start 2024-03-31T15:07:42.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:42.662+0000 7f5c72000640 1 -- start start 2024-03-31T15:07:42.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:42.662+0000 7f5c72000640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f5c6c06b620 0x7f5c6c0b5630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:42.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:42.662+0000 7f5c72000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5c6c072480 0x7f5c6c0b5b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:42.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:42.662+0000 7f5c72000640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5c6c0b0580 0x7f5c6c0ce9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:42.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:42.662+0000 7f5c72000640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f5c6c145cb0 con 0x7f5c6c072480 2024-03-31T15:07:42.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:42.662+0000 7f5c72000640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f5c6c169470 con 0x7f5c6c0b0580 2024-03-31T15:07:42.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:42.662+0000 7f5c72000640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f5c6c140b20 con 0x7f5c6c06b620 2024-03-31T15:07:42.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:42.662+0000 7f5c6be00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5c6c072480 0x7f5c6c0b5b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:42.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:42.662+0000 7f5c6be00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5c6c072480 0x7f5c6c0b5b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47532/0 (socket says 172.21.15.16:47532) 2024-03-31T15:07:42.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:42.662+0000 7f5c6be00640 1 -- 172.21.15.16:0/3317939154 learned_addr learned my addr 172.21.15.16:0/3317939154 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:07:42.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:42.662+0000 7f5c70c00640 1 --2- 172.21.15.16:0/3317939154 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f5c6c06b620 0x7f5c6c0b5630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:42.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:42.662+0000 7f5c71600640 1 --2- 172.21.15.16:0/3317939154 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5c6c0b0580 0x7f5c6c0ce9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:42.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:42.662+0000 7f5c6be00640 1 -- 172.21.15.16:0/3317939154 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f5c6c06b620 msgr2=0x7f5c6c0b5630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:42.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:42.662+0000 7f5c6be00640 1 --2- 172.21.15.16:0/3317939154 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f5c6c06b620 0x7f5c6c0b5630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:42.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:42.662+0000 7f5c6be00640 1 -- 172.21.15.16:0/3317939154 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5c6c0b0580 msgr2=0x7f5c6c0ce9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:42.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:42.662+0000 7f5c6be00640 1 --2- 172.21.15.16:0/3317939154 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5c6c0b0580 0x7f5c6c0ce9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:42.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:42.662+0000 7f5c6be00640 1 -- 172.21.15.16:0/3317939154 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c6c140d70 con 0x7f5c6c072480 2024-03-31T15:07:42.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:42.662+0000 7f5c6be00640 1 --2- 172.21.15.16:0/3317939154 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5c6c072480 0x7f5c6c0b5b70 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7f5c5800e870 tx=0x7f5c5800ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:07:42.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:42.662+0000 7f5c68c00640 1 -- 172.21.15.16:0/3317939154 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c58017070 con 0x7f5c6c072480 2024-03-31T15:07:42.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:42.662+0000 7f5c68c00640 1 -- 172.21.15.16:0/3317939154 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5c5800c940 con 0x7f5c6c072480 2024-03-31T15:07:42.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:42.662+0000 7f5c72000640 1 -- 172.21.15.16:0/3317939154 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5c6c141050 con 0x7f5c6c072480 2024-03-31T15:07:42.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:42.662+0000 7f5c68c00640 1 -- 172.21.15.16:0/3317939154 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c5800ccf0 con 0x7f5c6c072480 2024-03-31T15:07:42.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:42.662+0000 7f5c72000640 1 -- 172.21.15.16:0/3317939154 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5c6c141610 con 0x7f5c6c072480 2024-03-31T15:07:42.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:42.666+0000 7f5c72000640 1 -- 172.21.15.16:0/3317939154 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5c6c04f8c0 con 0x7f5c6c072480 2024-03-31T15:07:42.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:42.666+0000 7f5c68c00640 1 -- 172.21.15.16:0/3317939154 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f5c5801a070 con 0x7f5c6c072480 2024-03-31T15:07:42.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:42.666+0000 7f5c68c00640 1 --2- 172.21.15.16:0/3317939154 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f5c4c068380 0x7f5c4c06a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:42.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:42.666+0000 7f5c68c00640 1 -- 172.21.15.16:0/3317939154 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 5709+0+0 (secure 0 0 0) 0x7f5c58086ae0 con 0x7f5c6c072480 2024-03-31T15:07:42.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:42.666+0000 7f5c70c00640 1 --2- 172.21.15.16:0/3317939154 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f5c4c068380 0x7f5c4c06a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:42.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:42.666+0000 7f5c70c00640 1 --2- 172.21.15.16:0/3317939154 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f5c4c068380 0x7f5c4c06a840 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f5c600af000 tx=0x7f5c600a7000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:07:42.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:42.670+0000 7f5c68c00640 1 -- 172.21.15.16:0/3317939154 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f5c58056ae0 con 0x7f5c6c072480 2024-03-31T15:07:42.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:42.834+0000 7f5c72000640 1 -- 172.21.15.16:0/3317939154 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f5c6c0b1b10 con 0x7f5c6c072480 2024-03-31T15:07:42.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:42.834+0000 7f5c68c00640 1 -- 172.21.15.16:0/3317939154 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v56) ==== 100+0+11 (secure 0 0 0) 0x7f5c5805ade0 con 0x7f5c6c072480 2024-03-31T15:07:42.832 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 64 2024-03-31T15:07:42.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:42.838+0000 7f5c72000640 1 -- 172.21.15.16:0/3317939154 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f5c4c068380 msgr2=0x7f5c4c06a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:42.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:42.838+0000 7f5c72000640 1 --2- 172.21.15.16:0/3317939154 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f5c4c068380 0x7f5c4c06a840 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f5c600af000 tx=0x7f5c600a7000 comp rx=0 tx=0).stop 2024-03-31T15:07:42.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:42.838+0000 7f5c72000640 1 -- 172.21.15.16:0/3317939154 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5c6c072480 msgr2=0x7f5c6c0b5b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:42.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:42.838+0000 7f5c72000640 1 --2- 172.21.15.16:0/3317939154 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5c6c072480 0x7f5c6c0b5b70 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7f5c5800e870 tx=0x7f5c5800ec70 comp rx=0 tx=0).stop 2024-03-31T15:07:42.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:42.838+0000 7f5c72000640 1 -- 172.21.15.16:0/3317939154 shutdown_connections 2024-03-31T15:07:42.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:42.838+0000 7f5c72000640 1 --2- 172.21.15.16:0/3317939154 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f5c4c068380 0x7f5c4c06a840 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:42.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:42.838+0000 7f5c72000640 1 --2- 172.21.15.16:0/3317939154 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5c6c0b0580 0x7f5c6c0ce9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:42.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:42.838+0000 7f5c72000640 1 --2- 172.21.15.16:0/3317939154 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5c6c072480 0x7f5c6c0b5b70 unknown :-1 s=CLOSED pgs=755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:42.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:42.838+0000 7f5c72000640 1 --2- 172.21.15.16:0/3317939154 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f5c6c06b620 0x7f5c6c0b5630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:42.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:42.838+0000 7f5c72000640 1 -- 172.21.15.16:0/3317939154 >> 172.21.15.16:0/3317939154 conn(0x7f5c6c0b2b30 msgr2=0x7f5c6c0b2f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:07:42.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:42.838+0000 7f5c72000640 1 -- 172.21.15.16:0/3317939154 shutdown_connections 2024-03-31T15:07:42.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:42.838+0000 7f5c72000640 1 -- 172.21.15.16:0/3317939154 wait complete. 2024-03-31T15:07:42.855 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-31T15:07:42.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:42.998+0000 7f5b15e00640 1 Processor -- start 2024-03-31T15:07:42.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:42.998+0000 7f5b15e00640 1 -- start start 2024-03-31T15:07:42.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:43.002+0000 7f5b15e00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f5b1015c1d0 0x7f5b100b44e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:42.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:43.002+0000 7f5b15e00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f5b100cd4f0 0x7f5b1006f5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:42.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:43.002+0000 7f5b15e00640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5b100cd980 con 0x7f5b1006fb10 2024-03-31T15:07:42.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:43.002+0000 7f5b15e00640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f5b100cfaa0 con 0x7f5b100cd4f0 2024-03-31T15:07:42.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:43.002+0000 7f5b15e00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f5b100cdc60 con 0x7f5b1015c1d0 2024-03-31T15:07:42.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:43.002+0000 7f5b14a00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f5b1015c1d0 0x7f5b100b44e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:42.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:43.002+0000 7f5b07e00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f5b100cd4f0 0x7f5b1006f5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:42.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:43.002+0000 7f5b14a00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f5b1015c1d0 0x7f5b100b44e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3301/0 says I am v2:172.21.15.16:55058/0 (socket says 172.21.15.16:55058) 2024-03-31T15:07:42.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:43.002+0000 7f5b14a00640 1 -- 172.21.15.16:0/3317602989 learned_addr learned my addr 172.21.15.16:0/3317602989 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:07:42.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:43.002+0000 7f5b14a00640 1 -- 172.21.15.16:0/3317602989 >> v2:172.21.15.89:3300/0 conn(0x7f5b100cd4f0 msgr2=0x7f5b1006f5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:07:42.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:43.002+0000 7f5b14a00640 1 --2- 172.21.15.16:0/3317602989 >> v2:172.21.15.89:3300/0 conn(0x7f5b100cd4f0 0x7f5b1006f5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:42.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:43.002+0000 7f5b14a00640 1 -- 172.21.15.16:0/3317602989 >> v1:172.21.15.89:6789/0 conn(0x7f5b1006fb10 legacy=0x7f5b1015fd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:42.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:43.002+0000 7f5b14a00640 1 -- 172.21.15.16:0/3317602989 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b100cfcf0 con 0x7f5b1015c1d0 2024-03-31T15:07:43.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:43.006+0000 7f5b14a00640 1 --2- 172.21.15.16:0/3317602989 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f5b1015c1d0 0x7f5b100b44e0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f5b00009870 tx=0x7f5b0001c530 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b446a194411c7889 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:07:43.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:43.006+0000 7f5b0fe00640 1 -- 172.21.15.16:0/3317602989 <== mon.2 v2:172.21.15.89:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b000aa430 con 0x7f5b1015c1d0 2024-03-31T15:07:43.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:43.006+0000 7f5b0fe00640 1 -- 172.21.15.16:0/3317602989 <== mon.2 v2:172.21.15.89:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5b000af070 con 0x7f5b1015c1d0 2024-03-31T15:07:43.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:43.006+0000 7f5b0fe00640 1 -- 172.21.15.16:0/3317602989 <== mon.2 v2:172.21.15.89:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b000aa7f0 con 0x7f5b1015c1d0 2024-03-31T15:07:43.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:43.006+0000 7f5b15e00640 1 -- 172.21.15.16:0/3317602989 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f5b1015c1d0 msgr2=0x7f5b100b44e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:43.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:43.006+0000 7f5b15e00640 1 --2- 172.21.15.16:0/3317602989 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f5b1015c1d0 0x7f5b100b44e0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f5b00009870 tx=0x7f5b0001c530 comp rx=0 tx=0).stop 2024-03-31T15:07:43.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:43.006+0000 7f5b15e00640 1 -- 172.21.15.16:0/3317602989 shutdown_connections 2024-03-31T15:07:43.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:43.006+0000 7f5b15e00640 1 --2- 172.21.15.16:0/3317602989 >> v2:172.21.15.89:3300/0 conn(0x7f5b100cd4f0 0x7f5b1006f5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:43.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:43.006+0000 7f5b15e00640 1 --2- 172.21.15.16:0/3317602989 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f5b1015c1d0 0x7f5b100b44e0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:43.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:43.006+0000 7f5b15e00640 1 -- 172.21.15.16:0/3317602989 >> 172.21.15.16:0/3317602989 conn(0x7f5b100b39c0 msgr2=0x7f5b100b3dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:07:43.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:43.006+0000 7f5b15e00640 1 -- 172.21.15.16:0/3317602989 shutdown_connections 2024-03-31T15:07:43.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:43.006+0000 7f5b15e00640 1 -- 172.21.15.16:0/3317602989 wait complete. 2024-03-31T15:07:43.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:43.010+0000 7f5b15e00640 1 Processor -- start 2024-03-31T15:07:43.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:43.010+0000 7f5b15e00640 1 -- start start 2024-03-31T15:07:43.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:43.010+0000 7f5b15e00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f5b1006fb10 0x7f5b1015f230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:43.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:43.010+0000 7f5b15e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5b1015c1d0 0x7f5b100c8fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:43.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:43.010+0000 7f5b15e00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5b10157280 0x7f5b100b5230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:43.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:43.010+0000 7f5b15e00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f5b100d0e80 con 0x7f5b1015c1d0 2024-03-31T15:07:43.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:43.010+0000 7f5b15e00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f5b100d0d00 con 0x7f5b10157280 2024-03-31T15:07:43.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:43.010+0000 7f5b15e00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f5b10160450 con 0x7f5b1006fb10 2024-03-31T15:07:43.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:43.010+0000 7f5b07e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5b1015c1d0 0x7f5b100c8fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:43.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:43.010+0000 7f5b07e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5b1015c1d0 0x7f5b100c8fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47540/0 (socket says 172.21.15.16:47540) 2024-03-31T15:07:43.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:43.010+0000 7f5b07e00640 1 -- 172.21.15.16:0/850680531 learned_addr learned my addr 172.21.15.16:0/850680531 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:07:43.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:43.010+0000 7f5b14a00640 1 --2- 172.21.15.16:0/850680531 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f5b1006fb10 0x7f5b1015f230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:43.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:43.010+0000 7f5b15400640 1 --2- 172.21.15.16:0/850680531 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5b10157280 0x7f5b100b5230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:43.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:43.010+0000 7f5b07e00640 1 -- 172.21.15.16:0/850680531 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f5b1006fb10 msgr2=0x7f5b1015f230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:43.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:43.010+0000 7f5b07e00640 1 --2- 172.21.15.16:0/850680531 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f5b1006fb10 0x7f5b1015f230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:43.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:43.010+0000 7f5b07e00640 1 -- 172.21.15.16:0/850680531 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5b10157280 msgr2=0x7f5b100b5230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:43.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:43.010+0000 7f5b07e00640 1 --2- 172.21.15.16:0/850680531 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5b10157280 0x7f5b100b5230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:43.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:43.010+0000 7f5b07e00640 1 -- 172.21.15.16:0/850680531 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b100c9710 con 0x7f5b1015c1d0 2024-03-31T15:07:43.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:43.010+0000 7f5b07e00640 1 --2- 172.21.15.16:0/850680531 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5b1015c1d0 0x7f5b100c8fa0 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f5af800e930 tx=0x7f5af800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:07:43.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:43.010+0000 7f5b0d600640 1 -- 172.21.15.16:0/850680531 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5af8017070 con 0x7f5b1015c1d0 2024-03-31T15:07:43.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:43.010+0000 7f5b0d600640 1 -- 172.21.15.16:0/850680531 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5af800ca00 con 0x7f5b1015c1d0 2024-03-31T15:07:43.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:43.010+0000 7f5b0d600640 1 -- 172.21.15.16:0/850680531 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5af800cdb0 con 0x7f5b1015c1d0 2024-03-31T15:07:43.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:43.010+0000 7f5b15e00640 1 -- 172.21.15.16:0/850680531 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5b100c99f0 con 0x7f5b1015c1d0 2024-03-31T15:07:43.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:43.014+0000 7f5b15e00640 1 -- 172.21.15.16:0/850680531 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5b100c9fb0 con 0x7f5b1015c1d0 2024-03-31T15:07:43.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:43.014+0000 7f5b15e00640 1 -- 172.21.15.16:0/850680531 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5b1004f8c0 con 0x7f5b1015c1d0 2024-03-31T15:07:43.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:43.014+0000 7f5b0d600640 1 -- 172.21.15.16:0/850680531 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f5af801a070 con 0x7f5b1015c1d0 2024-03-31T15:07:43.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:43.014+0000 7f5b0d600640 1 --2- 172.21.15.16:0/850680531 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f5aec068380 0x7f5aec06a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:43.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:43.014+0000 7f5b0d600640 1 -- 172.21.15.16:0/850680531 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 5709+0+0 (secure 0 0 0) 0x7f5af8086d60 con 0x7f5b1015c1d0 2024-03-31T15:07:43.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:43.014+0000 7f5b14a00640 1 --2- 172.21.15.16:0/850680531 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f5aec068380 0x7f5aec06a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:43.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:43.014+0000 7f5b14a00640 1 --2- 172.21.15.16:0/850680531 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f5aec068380 0x7f5aec06a840 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f5b000ae000 tx=0x7f5b000a7000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:07:43.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:43.018+0000 7f5b0d600640 1 -- 172.21.15.16:0/850680531 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f5af8056cf0 con 0x7f5b1015c1d0 2024-03-31T15:07:43.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:43.218+0000 7f5b15e00640 1 -- 172.21.15.16:0/850680531 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5b10068ae0 con 0x7f5b1015c1d0 2024-03-31T15:07:43.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:43.218+0000 7f5b0d600640 1 -- 172.21.15.16:0/850680531 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 36 v36) ==== 94+0+5182 (secure 0 0 0) 0x7f5af805aff0 con 0x7f5b1015c1d0 2024-03-31T15:07:43.214 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:07:43.214 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":36,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5579,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6833/3721971977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":3721971977},{"type":"v1","addr":"172.21.15.16:6833","nonce":3721971977}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":29},{"gid":6109,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6836/1166206553","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":1166206553},{"type":"v1","addr":"172.21.15.89:6836","nonce":1166206553}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28},{"gid":6112,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/1826113083","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6835","nonce":1826113083},{"type":"v1","addr":"172.21.15.89:6837","nonce":1826113083}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":29}],"filesystems":[{"mdsmap":{"epoch":36,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-31T15:06:25.665001+0000","modified":"2024-03-31T15:07:14.315988+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6150},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6150":{"gid":6150,"name":"c","rank":0,"incarnation":30,"state":"up:active","state_seq":15,"addr":"172.21.15.16:6835/2613117528","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":2613117528},{"type":"v1","addr":"172.21.15.16:6835","nonce":2613117528}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6150,"qdb_cluster":[6150]},"id":4}]} 2024-03-31T15:07:43.214 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 36 2024-03-31T15:07:43.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:43.222+0000 7f5b15e00640 1 -- 172.21.15.16:0/850680531 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f5aec068380 msgr2=0x7f5aec06a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:43.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:43.222+0000 7f5b15e00640 1 --2- 172.21.15.16:0/850680531 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f5aec068380 0x7f5aec06a840 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f5b000ae000 tx=0x7f5b000a7000 comp rx=0 tx=0).stop 2024-03-31T15:07:43.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:43.222+0000 7f5b15e00640 1 -- 172.21.15.16:0/850680531 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5b1015c1d0 msgr2=0x7f5b100c8fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:43.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:43.222+0000 7f5b15e00640 1 --2- 172.21.15.16:0/850680531 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5b1015c1d0 0x7f5b100c8fa0 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f5af800e930 tx=0x7f5af800edf0 comp rx=0 tx=0).stop 2024-03-31T15:07:43.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:43.222+0000 7f5b15e00640 1 -- 172.21.15.16:0/850680531 shutdown_connections 2024-03-31T15:07:43.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:43.222+0000 7f5b15e00640 1 --2- 172.21.15.16:0/850680531 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f5aec068380 0x7f5aec06a840 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:43.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:43.222+0000 7f5b15e00640 1 --2- 172.21.15.16:0/850680531 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5b10157280 0x7f5b100b5230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:43.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:43.222+0000 7f5b15e00640 1 --2- 172.21.15.16:0/850680531 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5b1015c1d0 0x7f5b100c8fa0 unknown :-1 s=CLOSED pgs=756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:43.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:43.222+0000 7f5b15e00640 1 --2- 172.21.15.16:0/850680531 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f5b1006fb10 0x7f5b1015f230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:43.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:43.222+0000 7f5b15e00640 1 -- 172.21.15.16:0/850680531 >> 172.21.15.16:0/850680531 conn(0x7f5b100b39c0 msgr2=0x7f5b100ce950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:07:43.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:43.222+0000 7f5b15e00640 1 -- 172.21.15.16:0/850680531 shutdown_connections 2024-03-31T15:07:43.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:43.222+0000 7f5b15e00640 1 -- 172.21.15.16:0/850680531 wait complete. 2024-03-31T15:07:43.235 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-31T15:07:43.375 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:43.378+0000 7f621ae00640 1 Processor -- start 2024-03-31T15:07:43.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:43.378+0000 7f621ae00640 1 -- start start 2024-03-31T15:07:43.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:43.382+0000 7f621ae00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f6214147dd0 0x7f621406c730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:43.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:43.382+0000 7f621ae00640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6214147660 con 0x7f6214147280 2024-03-31T15:07:43.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:43.382+0000 7f621ae00640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6214162340 con 0x7f6214070030 2024-03-31T15:07:43.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:43.382+0000 7f621ae00640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f62141620f0 con 0x7f6214147dd0 2024-03-31T15:07:43.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:43.382+0000 7f6213e00640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f6214147280 0x7f621406f920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:53838/0 (socket says 172.21.15.16:53838) 2024-03-31T15:07:43.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:43.382+0000 7f6213e00640 1 -- 172.21.15.16:0/2780641216 learned_addr learned my addr 172.21.15.16:0/2780641216 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:07:43.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:43.382+0000 7f6218e00640 1 --2- 172.21.15.16:0/2780641216 >> v2:172.21.15.16:3300/0 conn(0x7f6214147dd0 0x7f621406c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:43.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:43.382+0000 7f6218e00640 1 -- 172.21.15.16:0/2780641216 >> v1:172.21.15.89:6789/0 conn(0x7f6214070030 legacy=0x7f62140cc8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:43.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:43.382+0000 7f6213400640 1 -- 172.21.15.16:0/2780641216 <== mon.1 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 834480251 0 0) 0x7f6214162340 con 0x7f6214070030 2024-03-31T15:07:43.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:43.382+0000 7f6218e00640 1 -- 172.21.15.16:0/2780641216 >> v1:172.21.15.16:6789/0 conn(0x7f6214147280 legacy=0x7f621406f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:43.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:43.382+0000 7f6218e00640 1 -- 172.21.15.16:0/2780641216 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6214162620 con 0x7f6214147dd0 2024-03-31T15:07:43.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:43.386+0000 7f6218e00640 1 --2- 172.21.15.16:0/2780641216 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6214147dd0 0x7f621406c730 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f620c0097c0 tx=0x7f620c01c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7c8d5646543ecda server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:07:43.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:43.386+0000 7f6213400640 1 -- 172.21.15.16:0/2780641216 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f620c0b1070 con 0x7f6214147dd0 2024-03-31T15:07:43.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:43.386+0000 7f6213400640 1 -- 172.21.15.16:0/2780641216 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f620c0ac5f0 con 0x7f6214147dd0 2024-03-31T15:07:43.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:43.386+0000 7f6213400640 1 -- 172.21.15.16:0/2780641216 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f620c0ac9c0 con 0x7f6214147dd0 2024-03-31T15:07:43.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:43.386+0000 7f621ae00640 1 -- 172.21.15.16:0/2780641216 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6214147dd0 msgr2=0x7f621406c730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:43.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:43.386+0000 7f621ae00640 1 --2- 172.21.15.16:0/2780641216 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6214147dd0 0x7f621406c730 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f620c0097c0 tx=0x7f620c01c4a0 comp rx=0 tx=0).stop 2024-03-31T15:07:43.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:43.386+0000 7f621ae00640 1 -- 172.21.15.16:0/2780641216 shutdown_connections 2024-03-31T15:07:43.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:43.386+0000 7f621ae00640 1 --2- 172.21.15.16:0/2780641216 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6214147dd0 0x7f621406c730 unknown :-1 s=CLOSED pgs=758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:43.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:43.386+0000 7f621ae00640 1 -- 172.21.15.16:0/2780641216 >> 172.21.15.16:0/2780641216 conn(0x7f6214067330 msgr2=0x7f6214067740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:07:43.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:43.386+0000 7f621ae00640 1 -- 172.21.15.16:0/2780641216 shutdown_connections 2024-03-31T15:07:43.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:43.386+0000 7f621ae00640 1 -- 172.21.15.16:0/2780641216 wait complete. 2024-03-31T15:07:43.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:43.386+0000 7f621ae00640 1 Processor -- start 2024-03-31T15:07:43.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:43.386+0000 7f621ae00640 1 -- start start 2024-03-31T15:07:43.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:43.390+0000 7f621ae00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f6214070030 0x7f621415def0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:43.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:43.390+0000 7f621ae00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6214147280 0x7f62140b5ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:43.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:43.390+0000 7f621ae00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6214147dd0 0x7f62140b6020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:43.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:43.390+0000 7f621ae00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f62140cdc20 con 0x7f6214147280 2024-03-31T15:07:43.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:43.390+0000 7f621ae00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f62140cdaa0 con 0x7f6214147dd0 2024-03-31T15:07:43.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:43.390+0000 7f621ae00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f621416c4a0 con 0x7f6214070030 2024-03-31T15:07:43.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:43.390+0000 7f620b400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6214147280 0x7f62140b5ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:43.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:43.390+0000 7f620b400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6214147280 0x7f62140b5ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47550/0 (socket says 172.21.15.16:47550) 2024-03-31T15:07:43.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:43.390+0000 7f620b400640 1 -- 172.21.15.16:0/791166833 learned_addr learned my addr 172.21.15.16:0/791166833 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:07:43.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:43.390+0000 7f6218e00640 1 --2- 172.21.15.16:0/791166833 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6214147dd0 0x7f62140b6020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:43.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:43.390+0000 7f6213e00640 1 --2- 172.21.15.16:0/791166833 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f6214070030 0x7f621415def0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:43.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:43.390+0000 7f620b400640 1 -- 172.21.15.16:0/791166833 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f6214070030 msgr2=0x7f621415def0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:43.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:43.390+0000 7f620b400640 1 --2- 172.21.15.16:0/791166833 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f6214070030 0x7f621415def0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:43.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:43.390+0000 7f620b400640 1 -- 172.21.15.16:0/791166833 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6214147dd0 msgr2=0x7f62140b6020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:43.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:43.390+0000 7f620b400640 1 --2- 172.21.15.16:0/791166833 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6214147dd0 0x7f62140b6020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:43.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:43.390+0000 7f620b400640 1 -- 172.21.15.16:0/791166833 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6214141e30 con 0x7f6214147280 2024-03-31T15:07:43.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:43.390+0000 7f620b400640 1 --2- 172.21.15.16:0/791166833 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6214147280 0x7f62140b5ae0 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f61fc00c440 tx=0x7f61fc00c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:07:43.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:43.390+0000 7f6210c00640 1 -- 172.21.15.16:0/791166833 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61fc005480 con 0x7f6214147280 2024-03-31T15:07:43.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:43.390+0000 7f6210c00640 1 -- 172.21.15.16:0/791166833 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f61fc003600 con 0x7f6214147280 2024-03-31T15:07:43.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:43.390+0000 7f6210c00640 1 -- 172.21.15.16:0/791166833 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61fc0039f0 con 0x7f6214147280 2024-03-31T15:07:43.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:43.390+0000 7f621ae00640 1 -- 172.21.15.16:0/791166833 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f62140b6650 con 0x7f6214147280 2024-03-31T15:07:43.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:43.390+0000 7f621ae00640 1 -- 172.21.15.16:0/791166833 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f62140b6c10 con 0x7f6214147280 2024-03-31T15:07:43.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:43.390+0000 7f621ae00640 1 -- 172.21.15.16:0/791166833 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6214069950 con 0x7f6214147280 2024-03-31T15:07:43.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:43.390+0000 7f6210c00640 1 -- 172.21.15.16:0/791166833 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f61fc003b90 con 0x7f6214147280 2024-03-31T15:07:43.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:43.394+0000 7f6210c00640 1 --2- 172.21.15.16:0/791166833 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f61f0068380 0x7f61f006a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:43.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:43.394+0000 7f6210c00640 1 -- 172.21.15.16:0/791166833 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 5709+0+0 (secure 0 0 0) 0x7f61fc0864a0 con 0x7f6214147280 2024-03-31T15:07:43.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:43.394+0000 7f6213e00640 1 --2- 172.21.15.16:0/791166833 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f61f0068380 0x7f61f006a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:43.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:43.394+0000 7f6213e00640 1 --2- 172.21.15.16:0/791166833 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f61f0068380 0x7f61f006a840 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f62040012f0 tx=0x7f6204001a40 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:07:43.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:43.398+0000 7f6210c00640 1 -- 172.21.15.16:0/791166833 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f61fc056370 con 0x7f6214147280 2024-03-31T15:07:43.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:43.658+0000 7f621ae00640 1 -- 172.21.15.16:0/791166833 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f62140cf3e0 con 0x7f6214147280 2024-03-31T15:07:43.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:43.662+0000 7f6210c00640 1 -- 172.21.15.16:0/791166833 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v56) ==== 74+0+16593 (secure 0 0 0) 0x7f61fc05a670 con 0x7f6214147280 2024-03-31T15:07:43.656 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:07:43.656 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":56,"fsid":"286eb5da-fb47-4550-b290-d9519c13b9a5","created":"2024-03-31T15:03:11.438952+0000","modified":"2024-03-31T15:06:27.274379+0000","last_up_change":"2024-03-31T15:03:22.885535+0000","last_in_change":"2024-03-31T15:03:13.941667+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-31T15:03:22.550787+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-03-31T15:06:24.043886+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"55","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-03-31T15:06:24.613108+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"56","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"92a66450-ef19-40d8-ac3d-d760b23b0bda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6801","nonce":932712244}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6803","nonce":932712244}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6807","nonce":932712244}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6805","nonce":932712244}]},"public_addr":"172.21.15.16:6801/932712244","cluster_addr":"172.21.15.16:6803/932712244","heartbeat_back_addr":"172.21.15.16:6807/932712244","heartbeat_front_addr":"172.21.15.16:6805/932712244","state":["exists","up"]},{"osd":1,"uuid":"da9d1f2b-1968-4959-b843-faa231988e92","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6810","nonce":1398737533}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6811","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6813","nonce":1398737533}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6821","nonce":1398737533}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6815","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6816","nonce":1398737533}]},"public_addr":"172.21.15.16:6810/1398737533","cluster_addr":"172.21.15.16:6813/1398737533","heartbeat_back_addr":"172.21.15.16:6821/1398737533","heartbeat_front_addr":"172.21.15.16:6816/1398737533","state":["exists","up"]},{"osd":2,"uuid":"01a73662-9a21-4b1f-bcbb-42b9636a4638","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6819","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6822","nonce":3018526646}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6825","nonce":3018526646}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6831","nonce":3018526646}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6827","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6829","nonce":3018526646}]},"public_addr":"172.21.15.16:6822/3018526646","cluster_addr":"172.21.15.16:6825/3018526646","heartbeat_back_addr":"172.21.15.16:6831/3018526646","heartbeat_front_addr":"172.21.15.16:6829/3018526646","state":["exists","up"]},{"osd":3,"uuid":"c7771fd3-e18e-4500-ac38-23cf10ebcd75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6809","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6812","nonce":3360352974}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6817","nonce":3360352974}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6828","nonce":3360352974}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6823","nonce":3360352974}]},"public_addr":"172.21.15.16:6812/3360352974","cluster_addr":"172.21.15.16:6817/3360352974","heartbeat_back_addr":"172.21.15.16:6828/3360352974","heartbeat_front_addr":"172.21.15.16:6823/3360352974","state":["exists","up"]},{"osd":4,"uuid":"88260730-1482-4fbd-a40f-cb1124e9df4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6801","nonce":1856844856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6803","nonce":1856844856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6807","nonce":1856844856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6805","nonce":1856844856}]},"public_addr":"172.21.15.89:6801/1856844856","cluster_addr":"172.21.15.89:6803/1856844856","heartbeat_back_addr":"172.21.15.89:6807/1856844856","heartbeat_front_addr":"172.21.15.89:6805/1856844856","state":["exists","up"]},{"osd":5,"uuid":"4cb2c00c-fae0-4dd8-9d27-ec4f0093ccf7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6814","nonce":50395527}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6817","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6820","nonce":50395527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6831","nonce":50395527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6823","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6826","nonce":50395527}]},"public_addr":"172.21.15.89:6814/50395527","cluster_addr":"172.21.15.89:6820/50395527","heartbeat_back_addr":"172.21.15.89:6831/50395527","heartbeat_front_addr":"172.21.15.89:6826/50395527","state":["exists","up"]},{"osd":6,"uuid":"81ce0c19-2554-4cae-acbe-66a05e9e9b9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6809","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6812","nonce":3470471752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6815","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6818","nonce":3470471752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6827","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6829","nonce":3470471752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6821","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6824","nonce":3470471752}]},"public_addr":"172.21.15.89:6812/3470471752","cluster_addr":"172.21.15.89:6818/3470471752","heartbeat_back_addr":"172.21.15.89:6829/3470471752","heartbeat_front_addr":"172.21.15.89:6824/3470471752","state":["exists","up"]},{"osd":7,"uuid":"85e4313b-d37d-442e-8042-153bf20a847d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6811","nonce":1796778630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6813","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6816","nonce":1796778630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6825","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6828","nonce":1796778630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6819","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6822","nonce":1796778630}]},"public_addr":"172.21.15.89:6811/1796778630","cluster_addr":"172.21.15.89:6816/1796778630","heartbeat_back_addr":"172.21.15.89:6828/1796778630","heartbeat_front_addr":"172.21.15.89:6822/1796778630","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.439088+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.529970+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.680512+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:21.044161+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.582111+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.536558+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.409522+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.590229+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-31T15:07:43.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:43.662+0000 7f621ae00640 1 -- 172.21.15.16:0/791166833 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f61f0068380 msgr2=0x7f61f006a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:43.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:43.662+0000 7f621ae00640 1 --2- 172.21.15.16:0/791166833 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f61f0068380 0x7f61f006a840 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f62040012f0 tx=0x7f6204001a40 comp rx=0 tx=0).stop 2024-03-31T15:07:43.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:43.662+0000 7f621ae00640 1 -- 172.21.15.16:0/791166833 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6214147280 msgr2=0x7f62140b5ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:43.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:43.662+0000 7f621ae00640 1 --2- 172.21.15.16:0/791166833 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6214147280 0x7f62140b5ae0 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f61fc00c440 tx=0x7f61fc00c900 comp rx=0 tx=0).stop 2024-03-31T15:07:43.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:43.662+0000 7f621ae00640 1 -- 172.21.15.16:0/791166833 shutdown_connections 2024-03-31T15:07:43.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:43.662+0000 7f621ae00640 1 --2- 172.21.15.16:0/791166833 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f61f0068380 0x7f61f006a840 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:43.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:43.662+0000 7f621ae00640 1 --2- 172.21.15.16:0/791166833 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6214147dd0 0x7f62140b6020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:43.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:43.662+0000 7f621ae00640 1 --2- 172.21.15.16:0/791166833 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6214147280 0x7f62140b5ae0 unknown :-1 s=CLOSED pgs=759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:43.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:43.662+0000 7f621ae00640 1 --2- 172.21.15.16:0/791166833 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f6214070030 0x7f621415def0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:43.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:43.662+0000 7f621ae00640 1 -- 172.21.15.16:0/791166833 >> 172.21.15.16:0/791166833 conn(0x7f6214067330 msgr2=0x7f6214142ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:07:43.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:43.662+0000 7f621ae00640 1 -- 172.21.15.16:0/791166833 shutdown_connections 2024-03-31T15:07:43.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:43.662+0000 7f621ae00640 1 -- 172.21.15.16:0/791166833 wait complete. 2024-03-31T15:07:43.685 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-31T15:07:43.685 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-31T15:07:43.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:43.830+0000 7f443be00640 1 Processor -- start 2024-03-31T15:07:43.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:43.830+0000 7f443be00640 1 -- start start 2024-03-31T15:07:43.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:43.830+0000 7f443be00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f44340b23d0 0x7f4434071d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:43.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:43.830+0000 7f443be00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f443415eee0 0x7f4434072250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:43.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:43.830+0000 7f443be00640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f443415f3c0 con 0x7f443415b3d0 2024-03-31T15:07:43.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:43.830+0000 7f443be00640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f443415b7b0 con 0x7f443415eee0 2024-03-31T15:07:43.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:43.830+0000 7f443be00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f443415f6a0 con 0x7f44340b23d0 2024-03-31T15:07:43.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:43.830+0000 7f4439e00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f443415eee0 0x7f4434072250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:43.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:43.830+0000 7f4439e00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f443415eee0 0x7f4434072250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47566/0 (socket says 172.21.15.16:47566) 2024-03-31T15:07:43.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:43.830+0000 7f4439e00640 1 -- 172.21.15.16:0/773052447 learned_addr learned my addr 172.21.15.16:0/773052447 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:07:43.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:43.830+0000 7f4438a00640 1 --2- 172.21.15.16:0/773052447 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f44340b23d0 0x7f4434071d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:43.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:43.830+0000 7f4439e00640 1 -- 172.21.15.16:0/773052447 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f44340b23d0 msgr2=0x7f4434071d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:07:43.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:43.830+0000 7f4439e00640 1 --2- 172.21.15.16:0/773052447 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f44340b23d0 0x7f4434071d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:43.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:43.830+0000 7f4439e00640 1 -- 172.21.15.16:0/773052447 >> v1:172.21.15.89:6789/0 conn(0x7f443415b3d0 legacy=0x7f44340b1cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:07:43.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:43.830+0000 7f4439e00640 1 -- 172.21.15.16:0/773052447 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f443415f980 con 0x7f443415eee0 2024-03-31T15:07:43.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:43.830+0000 7f4438a00640 1 --2- 172.21.15.16:0/773052447 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f44340b23d0 0x7f4434071d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:07:43.829 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:43.834+0000 7f4439e00640 1 --2- 172.21.15.16:0/773052447 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f443415eee0 0x7f4434072250 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f442c009e50 tx=0x7f442c01c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5ffb2d0746e303d server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:07:43.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:43.834+0000 7f4433e00640 1 -- 172.21.15.16:0/773052447 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f442c0a5800 con 0x7f443415eee0 2024-03-31T15:07:43.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:43.834+0000 7f4433e00640 1 -- 172.21.15.16:0/773052447 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f442c0a59a0 con 0x7f443415eee0 2024-03-31T15:07:43.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:43.834+0000 7f4433e00640 1 -- 172.21.15.16:0/773052447 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f442c0a5d70 con 0x7f443415eee0 2024-03-31T15:07:43.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:43.834+0000 7f443be00640 1 -- 172.21.15.16:0/773052447 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f443415eee0 msgr2=0x7f4434072250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:43.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:43.834+0000 7f443be00640 1 --2- 172.21.15.16:0/773052447 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f443415eee0 0x7f4434072250 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f442c009e50 tx=0x7f442c01c9c0 comp rx=0 tx=0).stop 2024-03-31T15:07:43.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:43.834+0000 7f443be00640 1 -- 172.21.15.16:0/773052447 shutdown_connections 2024-03-31T15:07:43.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:43.834+0000 7f443be00640 1 --2- 172.21.15.16:0/773052447 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f443415eee0 0x7f4434072250 unknown :-1 s=CLOSED pgs=760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:43.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:43.834+0000 7f443be00640 1 --2- 172.21.15.16:0/773052447 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f44340b23d0 0x7f4434071d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:43.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:43.834+0000 7f443be00640 1 -- 172.21.15.16:0/773052447 >> 172.21.15.16:0/773052447 conn(0x7f443406b620 msgr2=0x7f443406ba30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:07:43.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:43.834+0000 7f443be00640 1 -- 172.21.15.16:0/773052447 shutdown_connections 2024-03-31T15:07:43.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:43.834+0000 7f443be00640 1 -- 172.21.15.16:0/773052447 wait complete. 2024-03-31T15:07:43.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:43.838+0000 7f443be00640 1 Processor -- start 2024-03-31T15:07:43.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:43.838+0000 7f443be00640 1 -- start start 2024-03-31T15:07:43.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:43.838+0000 7f443be00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f44340b23d0 0x7f443415a770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:43.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:43.838+0000 7f443be00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f443415b3d0 0x7f4434145fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:43.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:43.838+0000 7f443be00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f443415eee0 0x7f443414e270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:43.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:43.838+0000 7f443be00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f4434073910 con 0x7f443415b3d0 2024-03-31T15:07:43.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:43.838+0000 7f443be00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f4434073790 con 0x7f443415eee0 2024-03-31T15:07:43.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:43.838+0000 7f443be00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f4434072790 con 0x7f44340b23d0 2024-03-31T15:07:43.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:43.838+0000 7f4438a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f443415b3d0 0x7f4434145fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:43.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:43.838+0000 7f4438a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f443415b3d0 0x7f4434145fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47570/0 (socket says 172.21.15.16:47570) 2024-03-31T15:07:43.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:43.838+0000 7f4438a00640 1 -- 172.21.15.16:0/1622342114 learned_addr learned my addr 172.21.15.16:0/1622342114 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:07:43.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:43.838+0000 7f4439e00640 1 --2- 172.21.15.16:0/1622342114 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f443415eee0 0x7f443414e270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:43.835 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:43.838+0000 7f4439400640 1 --2- 172.21.15.16:0/1622342114 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f44340b23d0 0x7f443415a770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:43.835 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:43.838+0000 7f4438a00640 1 -- 172.21.15.16:0/1622342114 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f44340b23d0 msgr2=0x7f443415a770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:43.835 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:43.838+0000 7f4438a00640 1 --2- 172.21.15.16:0/1622342114 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f44340b23d0 0x7f443415a770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:43.835 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:43.838+0000 7f4438a00640 1 -- 172.21.15.16:0/1622342114 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f443415eee0 msgr2=0x7f443414e270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:43.835 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:43.838+0000 7f4438a00640 1 --2- 172.21.15.16:0/1622342114 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f443415eee0 0x7f443414e270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:43.835 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:43.838+0000 7f4438a00640 1 -- 172.21.15.16:0/1622342114 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44341464f0 con 0x7f443415b3d0 2024-03-31T15:07:43.836 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:43.838+0000 7f4438a00640 1 --2- 172.21.15.16:0/1622342114 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f443415b3d0 0x7f4434145fb0 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f442000c7c0 tx=0x7f442000cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:07:43.836 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:43.838+0000 7f4431600640 1 -- 172.21.15.16:0/1622342114 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4420007a30 con 0x7f443415b3d0 2024-03-31T15:07:43.836 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:43.838+0000 7f4431600640 1 -- 172.21.15.16:0/1622342114 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4420007bd0 con 0x7f443415b3d0 2024-03-31T15:07:43.836 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:43.838+0000 7f443be00640 1 -- 172.21.15.16:0/1622342114 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f44341590e0 con 0x7f443415b3d0 2024-03-31T15:07:43.836 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:43.838+0000 7f4431600640 1 -- 172.21.15.16:0/1622342114 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44200113f0 con 0x7f443415b3d0 2024-03-31T15:07:43.836 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:43.838+0000 7f443be00640 1 -- 172.21.15.16:0/1622342114 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f44341596a0 con 0x7f443415b3d0 2024-03-31T15:07:43.836 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:43.842+0000 7f443be00640 1 -- 172.21.15.16:0/1622342114 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4434069950 con 0x7f443415b3d0 2024-03-31T15:07:43.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:43.842+0000 7f4431600640 1 -- 172.21.15.16:0/1622342114 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f4420011590 con 0x7f443415b3d0 2024-03-31T15:07:43.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:43.842+0000 7f4431600640 1 --2- 172.21.15.16:0/1622342114 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f4414068380 0x7f441406a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:43.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:43.842+0000 7f4431600640 1 -- 172.21.15.16:0/1622342114 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 5709+0+0 (secure 0 0 0) 0x7f4420086be0 con 0x7f443415b3d0 2024-03-31T15:07:43.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:43.842+0000 7f4439400640 1 --2- 172.21.15.16:0/1622342114 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f4414068380 0x7f441406a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:43.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:43.842+0000 7f4439400640 1 --2- 172.21.15.16:0/1622342114 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f4414068380 0x7f441406a840 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f4428007fd0 tx=0x7f4428007e40 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:07:43.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:43.846+0000 7f4431600640 1 -- 172.21.15.16:0/1622342114 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f4420015020 con 0x7f443415b3d0 2024-03-31T15:07:44.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:44.042+0000 7f443be00640 1 -- 172.21.15.16:0/1622342114 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f44340cdea0 con 0x7f443415b3d0 2024-03-31T15:07:44.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:44.042+0000 7f4431600640 1 -- 172.21.15.16:0/1622342114 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 36 v36) ==== 94+0+5182 (secure 0 0 0) 0x7f4420056ab0 con 0x7f443415b3d0 2024-03-31T15:07:44.039 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:07:44.039 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":36,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5579,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6833/3721971977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":3721971977},{"type":"v1","addr":"172.21.15.16:6833","nonce":3721971977}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":29},{"gid":6109,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6836/1166206553","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":1166206553},{"type":"v1","addr":"172.21.15.89:6836","nonce":1166206553}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28},{"gid":6112,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/1826113083","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6835","nonce":1826113083},{"type":"v1","addr":"172.21.15.89:6837","nonce":1826113083}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":29}],"filesystems":[{"mdsmap":{"epoch":36,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-31T15:06:25.665001+0000","modified":"2024-03-31T15:07:14.315988+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6150},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6150":{"gid":6150,"name":"c","rank":0,"incarnation":30,"state":"up:active","state_seq":15,"addr":"172.21.15.16:6835/2613117528","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":2613117528},{"type":"v1","addr":"172.21.15.16:6835","nonce":2613117528}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6150,"qdb_cluster":[6150]},"id":4}]} 2024-03-31T15:07:44.039 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 36 2024-03-31T15:07:44.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:44.046+0000 7f443be00640 1 -- 172.21.15.16:0/1622342114 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f4414068380 msgr2=0x7f441406a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:44.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:44.046+0000 7f443be00640 1 --2- 172.21.15.16:0/1622342114 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f4414068380 0x7f441406a840 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f4428007fd0 tx=0x7f4428007e40 comp rx=0 tx=0).stop 2024-03-31T15:07:44.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:44.046+0000 7f443be00640 1 -- 172.21.15.16:0/1622342114 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f443415b3d0 msgr2=0x7f4434145fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:44.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:44.046+0000 7f443be00640 1 --2- 172.21.15.16:0/1622342114 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f443415b3d0 0x7f4434145fb0 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f442000c7c0 tx=0x7f442000cc80 comp rx=0 tx=0).stop 2024-03-31T15:07:44.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:44.046+0000 7f443be00640 1 -- 172.21.15.16:0/1622342114 shutdown_connections 2024-03-31T15:07:44.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:44.046+0000 7f443be00640 1 --2- 172.21.15.16:0/1622342114 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f4414068380 0x7f441406a840 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:44.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:44.046+0000 7f443be00640 1 --2- 172.21.15.16:0/1622342114 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f443415eee0 0x7f443414e270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:44.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:44.046+0000 7f443be00640 1 --2- 172.21.15.16:0/1622342114 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f443415b3d0 0x7f4434145fb0 unknown :-1 s=CLOSED pgs=761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:44.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:44.046+0000 7f443be00640 1 --2- 172.21.15.16:0/1622342114 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f44340b23d0 0x7f443415a770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:44.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:44.046+0000 7f443be00640 1 -- 172.21.15.16:0/1622342114 >> 172.21.15.16:0/1622342114 conn(0x7f443406b620 msgr2=0x7f44340cef10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:07:44.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:44.046+0000 7f443be00640 1 -- 172.21.15.16:0/1622342114 shutdown_connections 2024-03-31T15:07:44.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:44.046+0000 7f443be00640 1 -- 172.21.15.16:0/1622342114 wait complete. 2024-03-31T15:07:44.068 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-31T15:07:44.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:44.214+0000 7f665ce00640 1 Processor -- start 2024-03-31T15:07:44.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:44.214+0000 7f665ce00640 1 -- start start 2024-03-31T15:07:44.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:44.214+0000 7f665ce00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f6658147280 0x7f665806c730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:44.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:44.214+0000 7f665ce00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f6658147dd0 0x7f665806f920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:44.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:44.214+0000 7f665ce00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f665806fe60 0x7f66580cb440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:44.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:44.214+0000 7f665ce00640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f66581620f0 con 0x7f6658147dd0 2024-03-31T15:07:44.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:44.214+0000 7f665ce00640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f6658147660 con 0x7f665806fe60 2024-03-31T15:07:44.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:44.214+0000 7f665ce00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f6658162340 con 0x7f6658147280 2024-03-31T15:07:44.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:44.214+0000 7f664fe00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f6658147dd0 0x7f665806f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:44.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:44.214+0000 7f664fe00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f6658147dd0 0x7f665806f920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47574/0 (socket says 172.21.15.16:47574) 2024-03-31T15:07:44.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:44.214+0000 7f664fe00640 1 -- 172.21.15.16:0/1805736440 learned_addr learned my addr 172.21.15.16:0/1805736440 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:07:44.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:44.214+0000 7f6656800640 1 --2- 172.21.15.16:0/1805736440 >> v2:172.21.15.89:3300/0 conn(0x7f665806fe60 0x7f66580cb440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:44.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:44.214+0000 7f6655e00640 1 --2- 172.21.15.16:0/1805736440 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f6658147280 0x7f665806c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:44.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:44.214+0000 7f664fe00640 1 -- 172.21.15.16:0/1805736440 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f6658147280 msgr2=0x7f665806c730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:07:44.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:44.214+0000 7f664fe00640 1 --2- 172.21.15.16:0/1805736440 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f6658147280 0x7f665806c730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:44.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:44.214+0000 7f664fe00640 1 -- 172.21.15.16:0/1805736440 >> v2:172.21.15.89:3300/0 conn(0x7f665806fe60 msgr2=0x7f66580cb440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:07:44.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:44.214+0000 7f664fe00640 1 --2- 172.21.15.16:0/1805736440 >> v2:172.21.15.89:3300/0 conn(0x7f665806fe60 0x7f66580cb440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:44.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:44.214+0000 7f664fe00640 1 -- 172.21.15.16:0/1805736440 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6658162620 con 0x7f6658147dd0 2024-03-31T15:07:44.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:44.218+0000 7f664fe00640 1 --2- 172.21.15.16:0/1805736440 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6658147dd0 0x7f665806f920 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f665806e8d0 tx=0x7f664001c260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14934e2553921e19 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:07:44.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:44.218+0000 7f6655400640 1 -- 172.21.15.16:0/1805736440 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66400b0070 con 0x7f6658147dd0 2024-03-31T15:07:44.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:44.218+0000 7f6655400640 1 -- 172.21.15.16:0/1805736440 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f66400ab600 con 0x7f6658147dd0 2024-03-31T15:07:44.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:44.218+0000 7f6655400640 1 -- 172.21.15.16:0/1805736440 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66400ab9d0 con 0x7f6658147dd0 2024-03-31T15:07:44.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:44.218+0000 7f665ce00640 1 -- 172.21.15.16:0/1805736440 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6658147dd0 msgr2=0x7f665806f920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:44.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:44.218+0000 7f665ce00640 1 --2- 172.21.15.16:0/1805736440 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6658147dd0 0x7f665806f920 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f665806e8d0 tx=0x7f664001c260 comp rx=0 tx=0).stop 2024-03-31T15:07:44.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:44.218+0000 7f665ce00640 1 -- 172.21.15.16:0/1805736440 shutdown_connections 2024-03-31T15:07:44.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:44.218+0000 7f665ce00640 1 --2- 172.21.15.16:0/1805736440 >> v2:172.21.15.89:3300/0 conn(0x7f665806fe60 0x7f66580cb440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:44.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:44.218+0000 7f665ce00640 1 --2- 172.21.15.16:0/1805736440 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6658147dd0 0x7f665806f920 unknown :-1 s=CLOSED pgs=762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:44.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:44.218+0000 7f665ce00640 1 --2- 172.21.15.16:0/1805736440 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f6658147280 0x7f665806c730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:44.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:44.218+0000 7f665ce00640 1 -- 172.21.15.16:0/1805736440 >> 172.21.15.16:0/1805736440 conn(0x7f6658067330 msgr2=0x7f6658067740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:07:44.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:44.218+0000 7f665ce00640 1 -- 172.21.15.16:0/1805736440 shutdown_connections 2024-03-31T15:07:44.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:44.218+0000 7f665ce00640 1 -- 172.21.15.16:0/1805736440 wait complete. 2024-03-31T15:07:44.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:44.222+0000 7f665ce00640 1 Processor -- start 2024-03-31T15:07:44.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:44.222+0000 7f665ce00640 1 -- start start 2024-03-31T15:07:44.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:44.222+0000 7f665ce00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f665806fe60 0x7f66580b47b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:44.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:44.222+0000 7f665ce00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6658147280 0x7f6658144ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:44.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:44.222+0000 7f665ce00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f6658147dd0 0x7f665815d7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:44.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:44.222+0000 7f665ce00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f66580ccca0 con 0x7f6658147280 2024-03-31T15:07:44.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:44.222+0000 7f665ce00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f66580ccb20 con 0x7f665806fe60 2024-03-31T15:07:44.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:44.222+0000 7f665ce00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f66580cce20 con 0x7f6658147dd0 2024-03-31T15:07:44.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:44.222+0000 7f664fe00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6658147280 0x7f6658144ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:44.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:44.222+0000 7f664fe00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6658147280 0x7f6658144ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47582/0 (socket says 172.21.15.16:47582) 2024-03-31T15:07:44.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:44.222+0000 7f664fe00640 1 -- 172.21.15.16:0/4272330813 learned_addr learned my addr 172.21.15.16:0/4272330813 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:07:44.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:44.222+0000 7f6655e00640 1 --2- 172.21.15.16:0/4272330813 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f665806fe60 0x7f66580b47b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:44.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:44.222+0000 7f6656800640 1 --2- 172.21.15.16:0/4272330813 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f6658147dd0 0x7f665815d7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:44.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:44.222+0000 7f664fe00640 1 -- 172.21.15.16:0/4272330813 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f6658147dd0 msgr2=0x7f665815d7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:44.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:44.222+0000 7f664fe00640 1 --2- 172.21.15.16:0/4272330813 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f6658147dd0 0x7f665815d7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:44.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:44.222+0000 7f664fe00640 1 -- 172.21.15.16:0/4272330813 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f665806fe60 msgr2=0x7f66580b47b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:44.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:44.222+0000 7f664fe00640 1 --2- 172.21.15.16:0/4272330813 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f665806fe60 0x7f66580b47b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:44.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:44.222+0000 7f664fe00640 1 -- 172.21.15.16:0/4272330813 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6658144160 con 0x7f6658147280 2024-03-31T15:07:44.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:44.222+0000 7f664fe00640 1 --2- 172.21.15.16:0/4272330813 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6658147280 0x7f6658144ac0 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7f66400ae040 tx=0x7f6640019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:07:44.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:44.222+0000 7f664e000640 1 -- 172.21.15.16:0/4272330813 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66400b9070 con 0x7f6658147280 2024-03-31T15:07:44.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:44.222+0000 7f664e000640 1 -- 172.21.15.16:0/4272330813 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f66400b4400 con 0x7f6658147280 2024-03-31T15:07:44.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:44.222+0000 7f664e000640 1 -- 172.21.15.16:0/4272330813 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66400b0040 con 0x7f6658147280 2024-03-31T15:07:44.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:44.222+0000 7f665ce00640 1 -- 172.21.15.16:0/4272330813 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6658144440 con 0x7f6658147280 2024-03-31T15:07:44.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:44.222+0000 7f665ce00640 1 -- 172.21.15.16:0/4272330813 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f665815e3d0 con 0x7f6658147280 2024-03-31T15:07:44.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:44.226+0000 7f665ce00640 1 -- 172.21.15.16:0/4272330813 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6658069950 con 0x7f6658147280 2024-03-31T15:07:44.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:44.226+0000 7f664e000640 1 -- 172.21.15.16:0/4272330813 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f66400b33a0 con 0x7f6658147280 2024-03-31T15:07:44.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:44.226+0000 7f664e000640 1 --2- 172.21.15.16:0/4272330813 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f6634068380 0x7f663406a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:44.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:44.226+0000 7f664e000640 1 -- 172.21.15.16:0/4272330813 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 5709+0+0 (secure 0 0 0) 0x7f664011fbe0 con 0x7f6658147280 2024-03-31T15:07:44.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:44.226+0000 7f6655e00640 1 --2- 172.21.15.16:0/4272330813 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f6634068380 0x7f663406a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:44.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:44.226+0000 7f6655e00640 1 --2- 172.21.15.16:0/4272330813 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f6634068380 0x7f663406a840 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f66581fa750 tx=0x7f6648008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:07:44.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:44.230+0000 7f664e000640 1 -- 172.21.15.16:0/4272330813 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f6640016110 con 0x7f6658147280 2024-03-31T15:07:44.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:44.430+0000 7f665ce00640 1 -- 172.21.15.16:0/4272330813 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f66580b6f30 con 0x7f6658147280 2024-03-31T15:07:44.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:44.430+0000 7f664e000640 1 -- 172.21.15.16:0/4272330813 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 36 v36) ==== 94+0+5182 (secure 0 0 0) 0x7f66400efab0 con 0x7f6658147280 2024-03-31T15:07:44.427 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:07:44.427 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":36,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5579,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6833/3721971977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":3721971977},{"type":"v1","addr":"172.21.15.16:6833","nonce":3721971977}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":29},{"gid":6109,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6836/1166206553","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":1166206553},{"type":"v1","addr":"172.21.15.89:6836","nonce":1166206553}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28},{"gid":6112,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/1826113083","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6835","nonce":1826113083},{"type":"v1","addr":"172.21.15.89:6837","nonce":1826113083}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":29}],"filesystems":[{"mdsmap":{"epoch":36,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-31T15:06:25.665001+0000","modified":"2024-03-31T15:07:14.315988+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6150},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6150":{"gid":6150,"name":"c","rank":0,"incarnation":30,"state":"up:active","state_seq":15,"addr":"172.21.15.16:6835/2613117528","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":2613117528},{"type":"v1","addr":"172.21.15.16:6835","nonce":2613117528}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6150,"qdb_cluster":[6150]},"id":4}]} 2024-03-31T15:07:44.427 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 36 2024-03-31T15:07:44.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:44.434+0000 7f665ce00640 1 -- 172.21.15.16:0/4272330813 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f6634068380 msgr2=0x7f663406a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:44.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:44.434+0000 7f665ce00640 1 --2- 172.21.15.16:0/4272330813 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f6634068380 0x7f663406a840 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f66581fa750 tx=0x7f6648008040 comp rx=0 tx=0).stop 2024-03-31T15:07:44.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:44.434+0000 7f665ce00640 1 -- 172.21.15.16:0/4272330813 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6658147280 msgr2=0x7f6658144ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:44.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:44.434+0000 7f665ce00640 1 --2- 172.21.15.16:0/4272330813 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6658147280 0x7f6658144ac0 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7f66400ae040 tx=0x7f6640019000 comp rx=0 tx=0).stop 2024-03-31T15:07:44.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:44.434+0000 7f665ce00640 1 -- 172.21.15.16:0/4272330813 shutdown_connections 2024-03-31T15:07:44.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:44.434+0000 7f665ce00640 1 --2- 172.21.15.16:0/4272330813 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f6634068380 0x7f663406a840 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:44.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:44.434+0000 7f665ce00640 1 --2- 172.21.15.16:0/4272330813 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f6658147dd0 0x7f665815d7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:44.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:44.434+0000 7f665ce00640 1 --2- 172.21.15.16:0/4272330813 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6658147280 0x7f6658144ac0 unknown :-1 s=CLOSED pgs=763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:44.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:44.434+0000 7f665ce00640 1 --2- 172.21.15.16:0/4272330813 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f665806fe60 0x7f66580b47b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:44.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:44.434+0000 7f665ce00640 1 -- 172.21.15.16:0/4272330813 >> 172.21.15.16:0/4272330813 conn(0x7f6658067330 msgr2=0x7f6658160a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:07:44.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:44.434+0000 7f665ce00640 1 -- 172.21.15.16:0/4272330813 shutdown_connections 2024-03-31T15:07:44.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:44.434+0000 7f665ce00640 1 -- 172.21.15.16:0/4272330813 wait complete. 2024-03-31T15:07:44.448 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-31T15:07:44.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:44.594+0000 7ff822600640 1 Processor -- start 2024-03-31T15:07:44.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:44.594+0000 7ff822600640 1 -- start start 2024-03-31T15:07:44.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:44.594+0000 7ff822600640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7ff81c0ae2e0 0x7ff81c06dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:44.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:44.594+0000 7ff822600640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff81c0ae740 0x7ff81c0b4f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:44.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:44.594+0000 7ff822600640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff81c0d0390 con 0x7ff81c06c630 2024-03-31T15:07:44.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:44.594+0000 7ff822600640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7ff81c0b41b0 con 0x7ff81c0ae740 2024-03-31T15:07:44.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:44.594+0000 7ff822600640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7ff81c0b4490 con 0x7ff81c0ae2e0 2024-03-31T15:07:44.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:44.594+0000 7ff821200640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7ff81c06c630 0x7ff81c06ca10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:53852/0 (socket says 172.21.15.16:53852) 2024-03-31T15:07:44.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:44.594+0000 7ff821200640 1 -- 172.21.15.16:0/1997650260 learned_addr learned my addr 172.21.15.16:0/1997650260 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:07:44.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:44.594+0000 7ff821c00640 1 --2- 172.21.15.16:0/1997650260 >> v2:172.21.15.16:3300/0 conn(0x7ff81c0ae740 0x7ff81c0b4f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:44.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:44.594+0000 7ff81be00640 1 --2- 172.21.15.16:0/1997650260 >> v2:172.21.15.89:3300/0 conn(0x7ff81c0ae2e0 0x7ff81c06dbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:44.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:44.594+0000 7ff821c00640 1 -- 172.21.15.16:0/1997650260 >> v2:172.21.15.89:3300/0 conn(0x7ff81c0ae2e0 msgr2=0x7ff81c06dbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:07:44.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:44.594+0000 7ff821c00640 1 --2- 172.21.15.16:0/1997650260 >> v2:172.21.15.89:3300/0 conn(0x7ff81c0ae2e0 0x7ff81c06dbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:44.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:44.594+0000 7ff821c00640 1 -- 172.21.15.16:0/1997650260 >> v1:172.21.15.16:6789/0 conn(0x7ff81c06c630 legacy=0x7ff81c06ca10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:44.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:44.594+0000 7ff821c00640 1 -- 172.21.15.16:0/1997650260 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff81c0b4770 con 0x7ff81c0ae740 2024-03-31T15:07:44.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:44.594+0000 7ff81b400640 1 -- 172.21.15.16:0/1997650260 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2896688048 0 0) 0x7ff81c0d0390 con 0x7ff81c06c630 2024-03-31T15:07:44.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:44.594+0000 7ff81be00640 1 --2- 172.21.15.16:0/1997650260 >> v2:172.21.15.89:3300/0 conn(0x7ff81c0ae2e0 0x7ff81c06dbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:07:44.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:44.598+0000 7ff821c00640 1 --2- 172.21.15.16:0/1997650260 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff81c0ae740 0x7ff81c0b4f70 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7ff81c0728f0 tx=0x7ff81401c930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a777078c516d7f7 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:07:44.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:44.598+0000 7ff81b400640 1 -- 172.21.15.16:0/1997650260 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8140b2070 con 0x7ff81c0ae740 2024-03-31T15:07:44.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:44.598+0000 7ff81b400640 1 -- 172.21.15.16:0/1997650260 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff8140a57f0 con 0x7ff81c0ae740 2024-03-31T15:07:44.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:44.598+0000 7ff81b400640 1 -- 172.21.15.16:0/1997650260 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8140a5bc0 con 0x7ff81c0ae740 2024-03-31T15:07:44.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:44.598+0000 7ff822600640 1 -- 172.21.15.16:0/1997650260 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff81c0ae740 msgr2=0x7ff81c0b4f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:44.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:44.598+0000 7ff822600640 1 --2- 172.21.15.16:0/1997650260 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff81c0ae740 0x7ff81c0b4f70 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7ff81c0728f0 tx=0x7ff81401c930 comp rx=0 tx=0).stop 2024-03-31T15:07:44.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:44.598+0000 7ff822600640 1 -- 172.21.15.16:0/1997650260 shutdown_connections 2024-03-31T15:07:44.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:44.598+0000 7ff822600640 1 --2- 172.21.15.16:0/1997650260 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff81c0ae740 0x7ff81c0b4f70 unknown :-1 s=CLOSED pgs=765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:44.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:44.598+0000 7ff822600640 1 --2- 172.21.15.16:0/1997650260 >> v2:172.21.15.89:3300/0 conn(0x7ff81c0ae2e0 0x7ff81c06dbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:44.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:44.598+0000 7ff822600640 1 -- 172.21.15.16:0/1997650260 >> 172.21.15.16:0/1997650260 conn(0x7ff81c071c40 msgr2=0x7ff81c072050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:07:44.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:44.598+0000 7ff822600640 1 -- 172.21.15.16:0/1997650260 shutdown_connections 2024-03-31T15:07:44.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:44.598+0000 7ff822600640 1 -- 172.21.15.16:0/1997650260 wait complete. 2024-03-31T15:07:44.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:44.598+0000 7ff822600640 1 Processor -- start 2024-03-31T15:07:44.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:44.602+0000 7ff822600640 1 -- start start 2024-03-31T15:07:44.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:44.602+0000 7ff822600640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff81c06c630 0x7ff81c148130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:44.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:44.602+0000 7ff822600640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff81c0ae2e0 0x7ff81c0b16d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:44.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:44.602+0000 7ff822600640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff81c0ae740 0x7ff81c145930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:44.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:44.602+0000 7ff822600640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7ff81c16bbe0 con 0x7ff81c0ae2e0 2024-03-31T15:07:44.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:44.602+0000 7ff822600640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7ff81c0b58c0 con 0x7ff81c0ae740 2024-03-31T15:07:44.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:44.602+0000 7ff822600640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7ff81c16bd60 con 0x7ff81c06c630 2024-03-31T15:07:44.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:44.602+0000 7ff81be00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff81c0ae2e0 0x7ff81c0b16d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:44.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:44.602+0000 7ff81be00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff81c0ae2e0 0x7ff81c0b16d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47600/0 (socket says 172.21.15.16:47600) 2024-03-31T15:07:44.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:44.602+0000 7ff81be00640 1 -- 172.21.15.16:0/1021046797 learned_addr learned my addr 172.21.15.16:0/1021046797 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:07:44.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:44.602+0000 7ff821200640 1 --2- 172.21.15.16:0/1021046797 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff81c06c630 0x7ff81c148130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:44.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:44.602+0000 7ff821c00640 1 --2- 172.21.15.16:0/1021046797 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff81c0ae740 0x7ff81c145930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:44.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:44.602+0000 7ff81be00640 1 -- 172.21.15.16:0/1021046797 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff81c06c630 msgr2=0x7ff81c148130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:44.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:44.602+0000 7ff81be00640 1 --2- 172.21.15.16:0/1021046797 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff81c06c630 0x7ff81c148130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:44.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:44.602+0000 7ff81be00640 1 -- 172.21.15.16:0/1021046797 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff81c0ae740 msgr2=0x7ff81c145930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:44.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:44.602+0000 7ff81be00640 1 --2- 172.21.15.16:0/1021046797 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff81c0ae740 0x7ff81c145930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:44.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:44.602+0000 7ff81be00640 1 -- 172.21.15.16:0/1021046797 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff81c14a6e0 con 0x7ff81c0ae2e0 2024-03-31T15:07:44.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:44.602+0000 7ff81be00640 1 --2- 172.21.15.16:0/1021046797 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff81c0ae2e0 0x7ff81c0b16d0 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7ff80800c7f0 tx=0x7ff80800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:07:44.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:44.602+0000 7ff818c00640 1 -- 172.21.15.16:0/1021046797 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff808015070 con 0x7ff81c0ae2e0 2024-03-31T15:07:44.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:44.602+0000 7ff818c00640 1 -- 172.21.15.16:0/1021046797 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff808007820 con 0x7ff81c0ae2e0 2024-03-31T15:07:44.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:44.602+0000 7ff822600640 1 -- 172.21.15.16:0/1021046797 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff81c14a9c0 con 0x7ff81c0ae2e0 2024-03-31T15:07:44.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:44.602+0000 7ff818c00640 1 -- 172.21.15.16:0/1021046797 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff808007c10 con 0x7ff81c0ae2e0 2024-03-31T15:07:44.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:44.602+0000 7ff822600640 1 -- 172.21.15.16:0/1021046797 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff81c149890 con 0x7ff81c0ae2e0 2024-03-31T15:07:44.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:44.602+0000 7ff822600640 1 -- 172.21.15.16:0/1021046797 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff81c04f8c0 con 0x7ff81c0ae2e0 2024-03-31T15:07:44.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:44.606+0000 7ff818c00640 1 -- 172.21.15.16:0/1021046797 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7ff808018070 con 0x7ff81c0ae2e0 2024-03-31T15:07:44.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:44.606+0000 7ff818c00640 1 --2- 172.21.15.16:0/1021046797 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7ff7fc068380 0x7ff7fc06a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:44.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:44.606+0000 7ff818c00640 1 -- 172.21.15.16:0/1021046797 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 5709+0+0 (secure 0 0 0) 0x7ff808086a30 con 0x7ff81c0ae2e0 2024-03-31T15:07:44.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:44.606+0000 7ff821200640 1 --2- 172.21.15.16:0/1021046797 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7ff7fc068380 0x7ff7fc06a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:44.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:44.606+0000 7ff821200640 1 --2- 172.21.15.16:0/1021046797 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7ff7fc068380 0x7ff7fc06a840 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7ff810000f70 tx=0x7ff810007df0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:07:44.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:44.610+0000 7ff818c00640 1 -- 172.21.15.16:0/1021046797 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7ff808056990 con 0x7ff81c0ae2e0 2024-03-31T15:07:44.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:44.774+0000 7ff822600640 1 -- 172.21.15.16:0/1021046797 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ff81c161b90 con 0x7ff81c0ae2e0 2024-03-31T15:07:44.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:44.774+0000 7ff818c00640 1 -- 172.21.15.16:0/1021046797 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v56) ==== 74+0+16593 (secure 0 0 0) 0x7ff80805be20 con 0x7ff81c0ae2e0 2024-03-31T15:07:44.771 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:07:44.771 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":56,"fsid":"286eb5da-fb47-4550-b290-d9519c13b9a5","created":"2024-03-31T15:03:11.438952+0000","modified":"2024-03-31T15:06:27.274379+0000","last_up_change":"2024-03-31T15:03:22.885535+0000","last_in_change":"2024-03-31T15:03:13.941667+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-31T15:03:22.550787+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-03-31T15:06:24.043886+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"55","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-03-31T15:06:24.613108+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"56","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"92a66450-ef19-40d8-ac3d-d760b23b0bda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6801","nonce":932712244}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6803","nonce":932712244}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6807","nonce":932712244}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6805","nonce":932712244}]},"public_addr":"172.21.15.16:6801/932712244","cluster_addr":"172.21.15.16:6803/932712244","heartbeat_back_addr":"172.21.15.16:6807/932712244","heartbeat_front_addr":"172.21.15.16:6805/932712244","state":["exists","up"]},{"osd":1,"uuid":"da9d1f2b-1968-4959-b843-faa231988e92","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6810","nonce":1398737533}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6811","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6813","nonce":1398737533}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6821","nonce":1398737533}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6815","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6816","nonce":1398737533}]},"public_addr":"172.21.15.16:6810/1398737533","cluster_addr":"172.21.15.16:6813/1398737533","heartbeat_back_addr":"172.21.15.16:6821/1398737533","heartbeat_front_addr":"172.21.15.16:6816/1398737533","state":["exists","up"]},{"osd":2,"uuid":"01a73662-9a21-4b1f-bcbb-42b9636a4638","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6819","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6822","nonce":3018526646}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6825","nonce":3018526646}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6831","nonce":3018526646}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6827","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6829","nonce":3018526646}]},"public_addr":"172.21.15.16:6822/3018526646","cluster_addr":"172.21.15.16:6825/3018526646","heartbeat_back_addr":"172.21.15.16:6831/3018526646","heartbeat_front_addr":"172.21.15.16:6829/3018526646","state":["exists","up"]},{"osd":3,"uuid":"c7771fd3-e18e-4500-ac38-23cf10ebcd75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6809","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6812","nonce":3360352974}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6817","nonce":3360352974}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6828","nonce":3360352974}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6823","nonce":3360352974}]},"public_addr":"172.21.15.16:6812/3360352974","cluster_addr":"172.21.15.16:6817/3360352974","heartbeat_back_addr":"172.21.15.16:6828/3360352974","heartbeat_front_addr":"172.21.15.16:6823/3360352974","state":["exists","up"]},{"osd":4,"uuid":"88260730-1482-4fbd-a40f-cb1124e9df4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6801","nonce":1856844856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6803","nonce":1856844856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6807","nonce":1856844856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6805","nonce":1856844856}]},"public_addr":"172.21.15.89:6801/1856844856","cluster_addr":"172.21.15.89:6803/1856844856","heartbeat_back_addr":"172.21.15.89:6807/1856844856","heartbeat_front_addr":"172.21.15.89:6805/1856844856","state":["exists","up"]},{"osd":5,"uuid":"4cb2c00c-fae0-4dd8-9d27-ec4f0093ccf7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6814","nonce":50395527}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6817","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6820","nonce":50395527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6831","nonce":50395527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6823","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6826","nonce":50395527}]},"public_addr":"172.21.15.89:6814/50395527","cluster_addr":"172.21.15.89:6820/50395527","heartbeat_back_addr":"172.21.15.89:6831/50395527","heartbeat_front_addr":"172.21.15.89:6826/50395527","state":["exists","up"]},{"osd":6,"uuid":"81ce0c19-2554-4cae-acbe-66a05e9e9b9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6809","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6812","nonce":3470471752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6815","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6818","nonce":3470471752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6827","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6829","nonce":3470471752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6821","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6824","nonce":3470471752}]},"public_addr":"172.21.15.89:6812/3470471752","cluster_addr":"172.21.15.89:6818/3470471752","heartbeat_back_addr":"172.21.15.89:6829/3470471752","heartbeat_front_addr":"172.21.15.89:6824/3470471752","state":["exists","up"]},{"osd":7,"uuid":"85e4313b-d37d-442e-8042-153bf20a847d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6811","nonce":1796778630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6813","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6816","nonce":1796778630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6825","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6828","nonce":1796778630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6819","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6822","nonce":1796778630}]},"public_addr":"172.21.15.89:6811/1796778630","cluster_addr":"172.21.15.89:6816/1796778630","heartbeat_back_addr":"172.21.15.89:6828/1796778630","heartbeat_front_addr":"172.21.15.89:6822/1796778630","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.439088+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.529970+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.680512+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:21.044161+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.582111+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.536558+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.409522+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.590229+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-31T15:07:44.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:44.778+0000 7ff822600640 1 -- 172.21.15.16:0/1021046797 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7ff7fc068380 msgr2=0x7ff7fc06a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:44.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:44.778+0000 7ff822600640 1 --2- 172.21.15.16:0/1021046797 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7ff7fc068380 0x7ff7fc06a840 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7ff810000f70 tx=0x7ff810007df0 comp rx=0 tx=0).stop 2024-03-31T15:07:44.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:44.778+0000 7ff822600640 1 -- 172.21.15.16:0/1021046797 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff81c0ae2e0 msgr2=0x7ff81c0b16d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:44.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:44.778+0000 7ff822600640 1 --2- 172.21.15.16:0/1021046797 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff81c0ae2e0 0x7ff81c0b16d0 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7ff80800c7f0 tx=0x7ff80800ccb0 comp rx=0 tx=0).stop 2024-03-31T15:07:44.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:44.778+0000 7ff822600640 1 -- 172.21.15.16:0/1021046797 shutdown_connections 2024-03-31T15:07:44.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:44.778+0000 7ff822600640 1 --2- 172.21.15.16:0/1021046797 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7ff7fc068380 0x7ff7fc06a840 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:44.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:44.778+0000 7ff822600640 1 --2- 172.21.15.16:0/1021046797 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff81c0ae740 0x7ff81c145930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:44.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:44.778+0000 7ff822600640 1 --2- 172.21.15.16:0/1021046797 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff81c0ae2e0 0x7ff81c0b16d0 unknown :-1 s=CLOSED pgs=766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:44.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:44.778+0000 7ff822600640 1 --2- 172.21.15.16:0/1021046797 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff81c06c630 0x7ff81c148130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:44.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:44.778+0000 7ff822600640 1 -- 172.21.15.16:0/1021046797 >> 172.21.15.16:0/1021046797 conn(0x7ff81c071c40 msgr2=0x7ff81c06ef80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:07:44.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:44.778+0000 7ff822600640 1 -- 172.21.15.16:0/1021046797 shutdown_connections 2024-03-31T15:07:44.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:44.778+0000 7ff822600640 1 -- 172.21.15.16:0/1021046797 wait complete. 2024-03-31T15:07:44.794 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-31T15:07:44.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:44.942+0000 7fc9b2600640 1 Processor -- start 2024-03-31T15:07:44.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:44.942+0000 7fc9b2600640 1 -- start start 2024-03-31T15:07:44.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:44.942+0000 7fc9b2600640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fc9ac06f920 0x7fc9ac06fd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:44.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:44.942+0000 7fc9b2600640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fc9ac06d490 0x7fc9ac06d8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:44.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:44.942+0000 7fc9b2600640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc9ac162010 con 0x7fc9ac0cb440 2024-03-31T15:07:44.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:44.942+0000 7fc9b2600640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7fc9ac162be0 con 0x7fc9ac06f920 2024-03-31T15:07:44.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:44.942+0000 7fc9b2600640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fc9ac070240 con 0x7fc9ac06d490 2024-03-31T15:07:44.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:44.942+0000 7fc9ab400640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fc9ac06f920 0x7fc9ac06fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:44.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:44.942+0000 7fc9a3e00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fc9ac06d490 0x7fc9ac06d8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:44.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:44.942+0000 7fc9ab400640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fc9ac06f920 0x7fc9ac06fd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.16:47600/0 (socket says 172.21.15.16:47600) 2024-03-31T15:07:44.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:44.942+0000 7fc9ab400640 1 -- 172.21.15.16:0/2785166993 learned_addr learned my addr 172.21.15.16:0/2785166993 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:07:44.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:44.942+0000 7fc9ab400640 1 -- 172.21.15.16:0/2785166993 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fc9ac06d490 msgr2=0x7fc9ac06d8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:07:44.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:44.942+0000 7fc9ab400640 1 --2- 172.21.15.16:0/2785166993 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fc9ac06d490 0x7fc9ac06d8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:44.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:44.942+0000 7fc9ab400640 1 -- 172.21.15.16:0/2785166993 >> v1:172.21.15.89:6789/0 conn(0x7fc9ac0cb440 legacy=0x7fc9ac0cb8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:44.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:44.942+0000 7fc9ab400640 1 -- 172.21.15.16:0/2785166993 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9ac1622b0 con 0x7fc9ac06f920 2024-03-31T15:07:44.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:44.946+0000 7fc9ab400640 1 --2- 172.21.15.16:0/2785166993 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc9ac06f920 0x7fc9ac06fd00 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fc9ac162d60 tx=0x7fc99c01c510 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dd41daf528185ee9 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:07:44.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:44.946+0000 7fc9aaa00640 1 -- 172.21.15.16:0/2785166993 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc99c0aa670 con 0x7fc9ac06f920 2024-03-31T15:07:44.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:44.946+0000 7fc9aaa00640 1 -- 172.21.15.16:0/2785166993 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc99c0af070 con 0x7fc9ac06f920 2024-03-31T15:07:44.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:44.946+0000 7fc9aaa00640 1 -- 172.21.15.16:0/2785166993 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc99c0aaa30 con 0x7fc9ac06f920 2024-03-31T15:07:44.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:44.946+0000 7fc9b2600640 1 -- 172.21.15.16:0/2785166993 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc9ac06f920 msgr2=0x7fc9ac06fd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:44.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:44.946+0000 7fc9b2600640 1 --2- 172.21.15.16:0/2785166993 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc9ac06f920 0x7fc9ac06fd00 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fc9ac162d60 tx=0x7fc99c01c510 comp rx=0 tx=0).stop 2024-03-31T15:07:44.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:44.946+0000 7fc9b2600640 1 -- 172.21.15.16:0/2785166993 shutdown_connections 2024-03-31T15:07:44.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:44.946+0000 7fc9b2600640 1 --2- 172.21.15.16:0/2785166993 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fc9ac06d490 0x7fc9ac06d8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:44.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:44.946+0000 7fc9b2600640 1 --2- 172.21.15.16:0/2785166993 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc9ac06f920 0x7fc9ac06fd00 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:44.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:44.946+0000 7fc9b2600640 1 -- 172.21.15.16:0/2785166993 >> 172.21.15.16:0/2785166993 conn(0x7fc9ac067330 msgr2=0x7fc9ac067740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:07:44.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:44.946+0000 7fc9b2600640 1 -- 172.21.15.16:0/2785166993 shutdown_connections 2024-03-31T15:07:44.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:44.946+0000 7fc9b2600640 1 -- 172.21.15.16:0/2785166993 wait complete. 2024-03-31T15:07:44.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:44.950+0000 7fc9b2600640 1 Processor -- start 2024-03-31T15:07:44.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:44.950+0000 7fc9b2600640 1 -- start start 2024-03-31T15:07:44.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:44.950+0000 7fc9b2600640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fc9ac06d490 0x7fc9ac145350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:44.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:44.950+0000 7fc9b2600640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc9ac06f920 0x7fc9ac163220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:44.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:44.950+0000 7fc9b2600640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc9ac0cb440 0x7fc9ac1640a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:44.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:44.950+0000 7fc9b2600640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fc9ac06dfb0 con 0x7fc9ac06f920 2024-03-31T15:07:44.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:44.950+0000 7fc9b2600640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fc9ac06de30 con 0x7fc9ac0cb440 2024-03-31T15:07:44.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:44.950+0000 7fc9b2600640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fc9ac0cd750 con 0x7fc9ac06d490 2024-03-31T15:07:44.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:44.950+0000 7fc9a3e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc9ac06f920 0x7fc9ac163220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:44.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:44.950+0000 7fc9a3e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc9ac06f920 0x7fc9ac163220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47610/0 (socket says 172.21.15.16:47610) 2024-03-31T15:07:44.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:44.950+0000 7fc9a3e00640 1 -- 172.21.15.16:0/1720302583 learned_addr learned my addr 172.21.15.16:0/1720302583 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:07:44.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:44.950+0000 7fc9ab400640 1 --2- 172.21.15.16:0/1720302583 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fc9ac06d490 0x7fc9ac145350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:44.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:44.950+0000 7fc9abe00640 1 --2- 172.21.15.16:0/1720302583 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc9ac0cb440 0x7fc9ac1640a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:44.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:44.950+0000 7fc9a3e00640 1 -- 172.21.15.16:0/1720302583 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fc9ac06d490 msgr2=0x7fc9ac145350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:44.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:44.950+0000 7fc9a3e00640 1 --2- 172.21.15.16:0/1720302583 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fc9ac06d490 0x7fc9ac145350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:44.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:44.950+0000 7fc9a3e00640 1 -- 172.21.15.16:0/1720302583 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc9ac0cb440 msgr2=0x7fc9ac1640a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:44.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:44.950+0000 7fc9a3e00640 1 --2- 172.21.15.16:0/1720302583 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc9ac0cb440 0x7fc9ac1640a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:44.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:44.950+0000 7fc9a3e00640 1 -- 172.21.15.16:0/1720302583 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9ac1645e0 con 0x7fc9ac06f920 2024-03-31T15:07:44.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:44.950+0000 7fc9a3e00640 1 --2- 172.21.15.16:0/1720302583 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc9ac06f920 0x7fc9ac163220 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7fc99400e930 tx=0x7fc99400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:07:44.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:44.950+0000 7fc9a3400640 1 -- 172.21.15.16:0/1720302583 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc994017070 con 0x7fc9ac06f920 2024-03-31T15:07:44.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:44.950+0000 7fc9a3400640 1 -- 172.21.15.16:0/1720302583 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc99400ca00 con 0x7fc9ac06f920 2024-03-31T15:07:44.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:44.950+0000 7fc9a3400640 1 -- 172.21.15.16:0/1720302583 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc99400cdb0 con 0x7fc9ac06f920 2024-03-31T15:07:44.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:44.950+0000 7fc9b2600640 1 -- 172.21.15.16:0/1720302583 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc9ac145a30 con 0x7fc9ac06f920 2024-03-31T15:07:44.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:44.950+0000 7fc9b2600640 1 -- 172.21.15.16:0/1720302583 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc9ac145ff0 con 0x7fc9ac06f920 2024-03-31T15:07:44.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:44.954+0000 7fc9b2600640 1 -- 172.21.15.16:0/1720302583 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc9ac069950 con 0x7fc9ac06f920 2024-03-31T15:07:44.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:44.954+0000 7fc9a3400640 1 -- 172.21.15.16:0/1720302583 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fc99401a070 con 0x7fc9ac06f920 2024-03-31T15:07:44.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:44.954+0000 7fc9a3400640 1 --2- 172.21.15.16:0/1720302583 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fc988068380 0x7fc98806a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:44.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:44.954+0000 7fc9a3400640 1 -- 172.21.15.16:0/1720302583 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 5709+0+0 (secure 0 0 0) 0x7fc994086980 con 0x7fc9ac06f920 2024-03-31T15:07:44.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:44.954+0000 7fc9ab400640 1 --2- 172.21.15.16:0/1720302583 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fc988068380 0x7fc98806a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:44.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:44.954+0000 7fc9ab400640 1 --2- 172.21.15.16:0/1720302583 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fc988068380 0x7fc98806a840 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fc99c0ae000 tx=0x7fc99c0a7000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:07:44.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:44.958+0000 7fc9a3400640 1 -- 172.21.15.16:0/1720302583 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fc9940569a0 con 0x7fc9ac06f920 2024-03-31T15:07:45.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:45.202+0000 7fc9b2600640 1 -- 172.21.15.16:0/1720302583 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7fc9ac0cf3f0 con 0x7fc9ac06f920 2024-03-31T15:07:45.199 INFO:tasks.ceph.mon.a.smithi016.stderr:2024-03-31T15:07:45.206+0000 7f7dcbc00640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-31T15:07:45.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:45.206+0000 7fc9a3400640 1 -- 172.21.15.16:0/1720302583 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v37) ==== 161+0+0 (secure 0 0 0) 0x7fc99405aca0 con 0x7fc9ac06f920 2024-03-31T15:07:45.202 INFO:teuthology.orchestra.run.smithi016.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-31T15:07:45.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:45.210+0000 7fc9b2600640 1 -- 172.21.15.16:0/1720302583 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fc988068380 msgr2=0x7fc98806a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:45.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:45.210+0000 7fc9b2600640 1 --2- 172.21.15.16:0/1720302583 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fc988068380 0x7fc98806a840 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fc99c0ae000 tx=0x7fc99c0a7000 comp rx=0 tx=0).stop 2024-03-31T15:07:45.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:45.210+0000 7fc9b2600640 1 -- 172.21.15.16:0/1720302583 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc9ac06f920 msgr2=0x7fc9ac163220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:45.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:45.210+0000 7fc9b2600640 1 --2- 172.21.15.16:0/1720302583 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc9ac06f920 0x7fc9ac163220 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7fc99400e930 tx=0x7fc99400edf0 comp rx=0 tx=0).stop 2024-03-31T15:07:45.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:45.210+0000 7fc9b2600640 1 -- 172.21.15.16:0/1720302583 shutdown_connections 2024-03-31T15:07:45.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:45.210+0000 7fc9b2600640 1 --2- 172.21.15.16:0/1720302583 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fc988068380 0x7fc98806a840 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:45.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:45.210+0000 7fc9b2600640 1 --2- 172.21.15.16:0/1720302583 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc9ac0cb440 0x7fc9ac1640a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:45.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:45.210+0000 7fc9b2600640 1 --2- 172.21.15.16:0/1720302583 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc9ac06f920 0x7fc9ac163220 unknown :-1 s=CLOSED pgs=767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:45.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:45.210+0000 7fc9b2600640 1 --2- 172.21.15.16:0/1720302583 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fc9ac06d490 0x7fc9ac145350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:45.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:45.210+0000 7fc9b2600640 1 -- 172.21.15.16:0/1720302583 >> 172.21.15.16:0/1720302583 conn(0x7fc9ac067330 msgr2=0x7fc9ac0d1b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:07:45.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:45.210+0000 7fc9b2600640 1 -- 172.21.15.16:0/1720302583 shutdown_connections 2024-03-31T15:07:45.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:45.210+0000 7fc9b2600640 1 -- 172.21.15.16:0/1720302583 wait complete. 2024-03-31T15:07:45.224 INFO:tasks.ceph.mds.c.smithi016.stderr: -6684> 2024-03-31T15:06:28.235+0000 7f4ef6400640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-31T15:07:45.225 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-31T15:07:45.230 INFO:tasks.ceph.mds.c.smithi016.stderr: -4881> 2024-03-31T15:06:35.363+0000 7f4ef1400640 -1 mds.0.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000001 [2,head] /dir_test/file auth v2 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={6303=0-4194304@1} caps={6303=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@2},l=6303 | request=0 caps=1 dirtyparent=0 dirty=0 authpin=1 scrubqueue=0 0x564c5b37e580]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(-1)0x0:[]//[]","memoryvalue":"(9)0x10000000001:[<0x10000000000/file v2>,<0x1/dir_test v4>]//[]","error_str":"failed to decode on-disk backtrace (26 bytes)!"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-31T15:07:45.231 INFO:tasks.ceph.mds.c.smithi016.stderr: -2924> 2024-03-31T15:07:07.038+0000 7f4ef1400640 -1 mds.0.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000001 [2,head] /dir_test/file auth v4 ap=1 DIRTYPARENT s=0 n(v0 rc2024-03-31T15:06:34.177259+0000 1=1+0) caps={6303=pAsLsXsFs/-@2},l=6303(-1) | ptrwaiter=0 request=0 lock=0 caps=1 dirtyparent=1 dirty=1 authpin=1 scrubqueue=0 0x564c5b37e580]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(-1)0x0:[]//[]","memoryvalue":"(9)0x10000000001:[<0x10000000000/file v4>,<0x1/dir_test v7>]//[]","error_str":"failed to decode on-disk backtrace (26 bytes)!"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-31T15:07:45.284 INFO:tasks.ceph.mds.c.smithi016.stdout:starting mds.c at 2024-03-31T15:07:45.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:45.378+0000 7f1596c00640 1 Processor -- start 2024-03-31T15:07:45.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:45.378+0000 7f1596c00640 1 -- start start 2024-03-31T15:07:45.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:45.378+0000 7f1596c00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f1590147d60 0x7f15900b3130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:45.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:45.378+0000 7f1596c00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f1590073310 0x7f159006d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:45.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:45.378+0000 7f1596c00640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f15900737c0 con 0x7f159006da10 2024-03-31T15:07:45.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:45.378+0000 7f1596c00640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f1590148140 con 0x7f1590073310 2024-03-31T15:07:45.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:45.378+0000 7f1596c00640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f159006de10 con 0x7f1590147d60 2024-03-31T15:07:45.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:45.378+0000 7f1587400640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f1590073310 0x7f159006d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:45.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:45.382+0000 7f1587400640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f1590073310 0x7f159006d4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47644/0 (socket says 172.21.15.16:47644) 2024-03-31T15:07:45.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:45.382+0000 7f1587400640 1 -- 172.21.15.16:0/19415891 learned_addr learned my addr 172.21.15.16:0/19415891 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:07:45.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:45.382+0000 7f158fe00640 1 --2- 172.21.15.16:0/19415891 >> v2:172.21.15.89:3300/0 conn(0x7f1590147d60 0x7f15900b3130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:45.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:45.382+0000 7f1587400640 1 -- 172.21.15.16:0/19415891 >> v2:172.21.15.89:3300/0 conn(0x7f1590147d60 msgr2=0x7f15900b3130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:07:45.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:45.382+0000 7f1587400640 1 --2- 172.21.15.16:0/19415891 >> v2:172.21.15.89:3300/0 conn(0x7f1590147d60 0x7f15900b3130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:45.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:45.382+0000 7f1587400640 1 -- 172.21.15.16:0/19415891 >> v1:172.21.15.89:6789/0 conn(0x7f159006da10 legacy=0x7f1590161f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:45.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:45.382+0000 7f1587400640 1 -- 172.21.15.16:0/19415891 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15900b37c0 con 0x7f1590073310 2024-03-31T15:07:45.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:45.382+0000 7f1587400640 1 --2- 172.21.15.16:0/19415891 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1590073310 0x7f159006d4d0 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f15900b5590 tx=0x7f157801c4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c7f42ad9be911a7 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:07:45.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:45.382+0000 7f158f400640 1 -- 172.21.15.16:0/19415891 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15780aa650 con 0x7f1590073310 2024-03-31T15:07:45.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:45.382+0000 7f158f400640 1 -- 172.21.15.16:0/19415891 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f15780af070 con 0x7f1590073310 2024-03-31T15:07:45.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:45.382+0000 7f158f400640 1 -- 172.21.15.16:0/19415891 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15780aaa10 con 0x7f1590073310 2024-03-31T15:07:45.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:45.386+0000 7f1596c00640 1 -- 172.21.15.16:0/19415891 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1590073310 msgr2=0x7f159006d4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:45.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:45.386+0000 7f1596c00640 1 --2- 172.21.15.16:0/19415891 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1590073310 0x7f159006d4d0 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f15900b5590 tx=0x7f157801c4f0 comp rx=0 tx=0).stop 2024-03-31T15:07:45.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:45.386+0000 7f1596c00640 1 -- 172.21.15.16:0/19415891 shutdown_connections 2024-03-31T15:07:45.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:45.386+0000 7f1596c00640 1 --2- 172.21.15.16:0/19415891 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1590073310 0x7f159006d4d0 unknown :-1 s=CLOSED pgs=771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:45.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:45.386+0000 7f1596c00640 1 --2- 172.21.15.16:0/19415891 >> v2:172.21.15.89:3300/0 conn(0x7f1590147d60 0x7f15900b3130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:45.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:45.386+0000 7f1596c00640 1 -- 172.21.15.16:0/19415891 >> 172.21.15.16:0/19415891 conn(0x7f159006ee40 msgr2=0x7f159006b620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:07:45.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:45.386+0000 7f1596c00640 1 -- 172.21.15.16:0/19415891 shutdown_connections 2024-03-31T15:07:45.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:45.386+0000 7f1596c00640 1 -- 172.21.15.16:0/19415891 wait complete. 2024-03-31T15:07:45.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:45.386+0000 7f1596c00640 1 Processor -- start 2024-03-31T15:07:45.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:45.386+0000 7f1596c00640 1 -- start start 2024-03-31T15:07:45.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:45.386+0000 7f1596c00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f159006da10 0x7f159015f100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:45.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:45.386+0000 7f1596c00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f1590073310 0x7f15900b00b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:45.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:45.386+0000 7f1596c00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1590147d60 0x7f15900b05f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:45.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:45.386+0000 7f1596c00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f159016b2e0 con 0x7f159006da10 2024-03-31T15:07:45.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:45.386+0000 7f1596c00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f15900b4340 con 0x7f1590147d60 2024-03-31T15:07:45.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:45.386+0000 7f1596c00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f159016b460 con 0x7f1590073310 2024-03-31T15:07:45.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:45.386+0000 7f158fe00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f159006da10 0x7f159015f100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:45.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:45.386+0000 7f158fe00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f159006da10 0x7f159015f100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47658/0 (socket says 172.21.15.16:47658) 2024-03-31T15:07:45.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:45.386+0000 7f158fe00640 1 -- 172.21.15.16:0/3906602017 learned_addr learned my addr 172.21.15.16:0/3906602017 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:07:45.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:45.386+0000 7f1594c00640 1 --2- 172.21.15.16:0/3906602017 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1590147d60 0x7f15900b05f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:45.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:45.386+0000 7f1587400640 1 --2- 172.21.15.16:0/3906602017 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f1590073310 0x7f15900b00b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:45.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:45.386+0000 7f158fe00640 1 -- 172.21.15.16:0/3906602017 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f1590073310 msgr2=0x7f15900b00b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:45.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:45.386+0000 7f158fe00640 1 --2- 172.21.15.16:0/3906602017 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f1590073310 0x7f15900b00b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:45.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:45.386+0000 7f158fe00640 1 -- 172.21.15.16:0/3906602017 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1590147d60 msgr2=0x7f15900b05f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:45.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:45.386+0000 7f158fe00640 1 --2- 172.21.15.16:0/3906602017 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1590147d60 0x7f15900b05f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:45.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:45.386+0000 7f158fe00640 1 -- 172.21.15.16:0/3906602017 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1590142a10 con 0x7f159006da10 2024-03-31T15:07:45.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:45.386+0000 7f158fe00640 1 --2- 172.21.15.16:0/3906602017 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f159006da10 0x7f159015f100 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f1580007e40 tx=0x7f158000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:07:45.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:45.390+0000 7f158cc00640 1 -- 172.21.15.16:0/3906602017 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1580014620 con 0x7f159006da10 2024-03-31T15:07:45.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:45.390+0000 7f158cc00640 1 -- 172.21.15.16:0/3906602017 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1580019070 con 0x7f159006da10 2024-03-31T15:07:45.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:45.390+0000 7f158cc00640 1 -- 172.21.15.16:0/3906602017 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1580014a00 con 0x7f159006da10 2024-03-31T15:07:45.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:45.390+0000 7f1596c00640 1 -- 172.21.15.16:0/3906602017 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1590142cf0 con 0x7f159006da10 2024-03-31T15:07:45.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:45.390+0000 7f1596c00640 1 -- 172.21.15.16:0/3906602017 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f15901432b0 con 0x7f159006da10 2024-03-31T15:07:45.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:45.390+0000 7f1596c00640 1 -- 172.21.15.16:0/3906602017 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1590069950 con 0x7f159006da10 2024-03-31T15:07:45.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:45.390+0000 7f158cc00640 1 -- 172.21.15.16:0/3906602017 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f1580014ba0 con 0x7f159006da10 2024-03-31T15:07:45.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:45.390+0000 7f158cc00640 1 --2- 172.21.15.16:0/3906602017 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f156c068380 0x7f156c06a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:45.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:45.390+0000 7f158cc00640 1 -- 172.21.15.16:0/3906602017 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 5795+0+0 (secure 0 0 0) 0x7f1580087210 con 0x7f159006da10 2024-03-31T15:07:45.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:45.390+0000 7f1587400640 1 --2- 172.21.15.16:0/3906602017 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f156c068380 0x7f156c06a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:45.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:45.394+0000 7f1587400640 1 --2- 172.21.15.16:0/3906602017 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f156c068380 0x7f156c06a840 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f15780ae000 tx=0x7f15780a7000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:07:45.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:45.394+0000 7f158cc00640 1 -- 172.21.15.16:0/3906602017 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f1580057080 con 0x7f159006da10 2024-03-31T15:07:45.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:45.598+0000 7f1596c00640 1 -- 172.21.15.16:0/3906602017 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7f159006e470 con 0x7f159006da10 2024-03-31T15:07:46.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:46.210+0000 7f158cc00640 1 -- 172.21.15.16:0/3906602017 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v38) ==== 104+0+0 (secure 0 0 0) 0x7f158005b380 con 0x7f159006da10 2024-03-31T15:07:46.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:46.210+0000 7f1596c00640 1 -- 172.21.15.16:0/3906602017 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f156c068380 msgr2=0x7f156c06a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:46.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:46.210+0000 7f1596c00640 1 --2- 172.21.15.16:0/3906602017 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f156c068380 0x7f156c06a840 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f15780ae000 tx=0x7f15780a7000 comp rx=0 tx=0).stop 2024-03-31T15:07:46.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:46.210+0000 7f1596c00640 1 -- 172.21.15.16:0/3906602017 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f159006da10 msgr2=0x7f159015f100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:46.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:46.210+0000 7f1596c00640 1 --2- 172.21.15.16:0/3906602017 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f159006da10 0x7f159015f100 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f1580007e40 tx=0x7f158000e4b0 comp rx=0 tx=0).stop 2024-03-31T15:07:46.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:46.210+0000 7f1596c00640 1 -- 172.21.15.16:0/3906602017 shutdown_connections 2024-03-31T15:07:46.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:46.210+0000 7f1596c00640 1 --2- 172.21.15.16:0/3906602017 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f156c068380 0x7f156c06a840 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:46.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:46.210+0000 7f1596c00640 1 --2- 172.21.15.16:0/3906602017 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1590147d60 0x7f15900b05f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:46.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:46.210+0000 7f1596c00640 1 --2- 172.21.15.16:0/3906602017 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f1590073310 0x7f15900b00b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:46.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:46.210+0000 7f1596c00640 1 --2- 172.21.15.16:0/3906602017 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f159006da10 0x7f159015f100 unknown :-1 s=CLOSED pgs=772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:46.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:46.210+0000 7f1596c00640 1 -- 172.21.15.16:0/3906602017 >> 172.21.15.16:0/3906602017 conn(0x7f159006ee40 msgr2=0x7f15900b0f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:07:46.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:46.210+0000 7f1596c00640 1 -- 172.21.15.16:0/3906602017 shutdown_connections 2024-03-31T15:07:46.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:46.210+0000 7f1596c00640 1 -- 172.21.15.16:0/3906602017 wait complete. 2024-03-31T15:07:46.231 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-31T15:07:46.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:46.390+0000 7fd5da000640 1 Processor -- start 2024-03-31T15:07:46.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:46.390+0000 7fd5da000640 1 -- start start 2024-03-31T15:07:46.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:46.390+0000 7fd5da000640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fd5d40730c0 0x7fd5d40b45d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:46.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:46.390+0000 7fd5da000640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd5d40734f0 con 0x7fd5d4143f40 2024-03-31T15:07:46.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:46.390+0000 7fd5da000640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd5d4144320 con 0x7fd5d40b22f0 2024-03-31T15:07:46.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:46.390+0000 7fd5da000640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7fd5d415ef80 con 0x7fd5d40730c0 2024-03-31T15:07:46.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:46.390+0000 7fd5d3400640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fd5d4143f40 0x7fd5d40b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:53874/0 (socket says 172.21.15.16:53874) 2024-03-31T15:07:46.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:46.390+0000 7fd5d3400640 1 -- 172.21.15.16:0/939862146 learned_addr learned my addr 172.21.15.16:0/939862146 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:07:46.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:46.390+0000 7fd5d3e00640 1 --2- 172.21.15.16:0/939862146 >> v2:172.21.15.16:3300/0 conn(0x7fd5d40730c0 0x7fd5d40b45d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:46.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:46.390+0000 7fd5d3e00640 1 -- 172.21.15.16:0/939862146 >> v1:172.21.15.89:6789/0 conn(0x7fd5d40b22f0 legacy=0x7fd5d40b2f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:46.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:46.390+0000 7fd5d3e00640 1 -- 172.21.15.16:0/939862146 >> v1:172.21.15.16:6789/0 conn(0x7fd5d4143f40 legacy=0x7fd5d40b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:46.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:46.390+0000 7fd5d3e00640 1 -- 172.21.15.16:0/939862146 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5d415f260 con 0x7fd5d40730c0 2024-03-31T15:07:46.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:46.394+0000 7fd5d3e00640 1 --2- 172.21.15.16:0/939862146 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd5d40730c0 0x7fd5d40b45d0 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7fd5cc009d90 tx=0x7fd5cc01c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d8081f6a9a80ad9 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:07:46.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:46.394+0000 7fd5d2a00640 1 -- 172.21.15.16:0/939862146 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5cc0b2070 con 0x7fd5d40730c0 2024-03-31T15:07:46.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:46.394+0000 7fd5d2a00640 1 -- 172.21.15.16:0/939862146 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd5cc0a5830 con 0x7fd5d40730c0 2024-03-31T15:07:46.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:46.394+0000 7fd5d2a00640 1 -- 172.21.15.16:0/939862146 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5cc0a5c00 con 0x7fd5d40730c0 2024-03-31T15:07:46.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:46.394+0000 7fd5da000640 1 -- 172.21.15.16:0/939862146 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd5d40730c0 msgr2=0x7fd5d40b45d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:46.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:46.394+0000 7fd5da000640 1 --2- 172.21.15.16:0/939862146 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd5d40730c0 0x7fd5d40b45d0 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7fd5cc009d90 tx=0x7fd5cc01c970 comp rx=0 tx=0).stop 2024-03-31T15:07:46.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:46.394+0000 7fd5da000640 1 -- 172.21.15.16:0/939862146 shutdown_connections 2024-03-31T15:07:46.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:46.394+0000 7fd5da000640 1 --2- 172.21.15.16:0/939862146 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd5d40730c0 0x7fd5d40b45d0 unknown :-1 s=CLOSED pgs=774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:46.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:46.394+0000 7fd5da000640 1 -- 172.21.15.16:0/939862146 >> 172.21.15.16:0/939862146 conn(0x7fd5d406b620 msgr2=0x7fd5d406ba30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:07:46.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:46.394+0000 7fd5da000640 1 -- 172.21.15.16:0/939862146 shutdown_connections 2024-03-31T15:07:46.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:46.394+0000 7fd5da000640 1 -- 172.21.15.16:0/939862146 wait complete. 2024-03-31T15:07:46.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:46.398+0000 7fd5da000640 1 Processor -- start 2024-03-31T15:07:46.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:46.398+0000 7fd5da000640 1 -- start start 2024-03-31T15:07:46.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:46.398+0000 7fd5da000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd5d40730c0 0x7fd5d415bdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:46.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:46.398+0000 7fd5da000640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd5d40b22f0 0x7fd5d4071f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:46.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:46.398+0000 7fd5da000640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fd5d4143f40 0x7fd5d4157870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:46.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:46.398+0000 7fd5da000640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fd5d4169240 con 0x7fd5d40730c0 2024-03-31T15:07:46.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:46.398+0000 7fd5da000640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fd5d41690c0 con 0x7fd5d40b22f0 2024-03-31T15:07:46.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:46.398+0000 7fd5da000640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fd5d41693c0 con 0x7fd5d4143f40 2024-03-31T15:07:46.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:46.398+0000 7fd5d3400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd5d40730c0 0x7fd5d415bdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:46.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:46.398+0000 7fd5d3400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd5d40730c0 0x7fd5d415bdc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47672/0 (socket says 172.21.15.16:47672) 2024-03-31T15:07:46.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:46.398+0000 7fd5d3400640 1 -- 172.21.15.16:0/2884762447 learned_addr learned my addr 172.21.15.16:0/2884762447 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:07:46.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:46.398+0000 7fd5caa00640 1 --2- 172.21.15.16:0/2884762447 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd5d40b22f0 0x7fd5d4071f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:46.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:46.398+0000 7fd5d3e00640 1 --2- 172.21.15.16:0/2884762447 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fd5d4143f40 0x7fd5d4157870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:46.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:46.398+0000 7fd5d3400640 1 -- 172.21.15.16:0/2884762447 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fd5d4143f40 msgr2=0x7fd5d4157870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:46.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:46.398+0000 7fd5d3400640 1 --2- 172.21.15.16:0/2884762447 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fd5d4143f40 0x7fd5d4157870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:46.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:46.398+0000 7fd5d3400640 1 -- 172.21.15.16:0/2884762447 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd5d40b22f0 msgr2=0x7fd5d4071f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:46.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:46.398+0000 7fd5d3400640 1 --2- 172.21.15.16:0/2884762447 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd5d40b22f0 0x7fd5d4071f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:46.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:46.398+0000 7fd5d3400640 1 -- 172.21.15.16:0/2884762447 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5d41409e0 con 0x7fd5d40730c0 2024-03-31T15:07:46.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:46.398+0000 7fd5caa00640 1 --2- 172.21.15.16:0/2884762447 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd5d40b22f0 0x7fd5d4071f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:07:46.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:46.398+0000 7fd5d3400640 1 --2- 172.21.15.16:0/2884762447 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd5d40730c0 0x7fd5d415bdc0 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7fd5c400c500 tx=0x7fd5c400c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:07:46.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:46.398+0000 7fd5cbe00640 1 -- 172.21.15.16:0/2884762447 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5c4017070 con 0x7fd5d40730c0 2024-03-31T15:07:46.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:46.398+0000 7fd5cbe00640 1 -- 172.21.15.16:0/2884762447 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd5c4005cf0 con 0x7fd5d40730c0 2024-03-31T15:07:46.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:46.398+0000 7fd5cbe00640 1 -- 172.21.15.16:0/2884762447 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5c40072f0 con 0x7fd5d40730c0 2024-03-31T15:07:46.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:46.398+0000 7fd5da000640 1 -- 172.21.15.16:0/2884762447 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd5d415b000 con 0x7fd5d40730c0 2024-03-31T15:07:46.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:46.398+0000 7fd5da000640 1 -- 172.21.15.16:0/2884762447 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd5d415b5c0 con 0x7fd5d40730c0 2024-03-31T15:07:46.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:46.398+0000 7fd5da000640 1 -- 172.21.15.16:0/2884762447 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd5d4069950 con 0x7fd5d40730c0 2024-03-31T15:07:46.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:46.402+0000 7fd5cbe00640 1 -- 172.21.15.16:0/2884762447 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fd5c401a070 con 0x7fd5d40730c0 2024-03-31T15:07:46.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:46.402+0000 7fd5cbe00640 1 --2- 172.21.15.16:0/2884762447 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fd5b0068380 0x7fd5b006a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:46.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:46.402+0000 7fd5cbe00640 1 -- 172.21.15.16:0/2884762447 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 5795+0+0 (secure 0 0 0) 0x7fd5c4085d00 con 0x7fd5d40730c0 2024-03-31T15:07:46.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:46.402+0000 7fd5caa00640 1 --2- 172.21.15.16:0/2884762447 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fd5b0068380 0x7fd5b006a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:46.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:46.402+0000 7fd5caa00640 1 --2- 172.21.15.16:0/2884762447 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fd5b0068380 0x7fd5b006a840 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fd5d4143630 tx=0x7fd5bc009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:07:46.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:46.406+0000 7fd5cbe00640 1 -- 172.21.15.16:0/2884762447 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fd5d40734f0 con 0x7fd5d40730c0 2024-03-31T15:07:46.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:46.574+0000 7fd5da000640 1 -- 172.21.15.16:0/2884762447 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7fd5d40b4060 con 0x7fd5d40730c0 2024-03-31T15:07:47.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:47.466+0000 7fd5cbe00640 1 -- 172.21.15.16:0/2884762447 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v58) ==== 181+0+0 (secure 0 0 0) 0x7fd5c4055c00 con 0x7fd5d40730c0 2024-03-31T15:07:47.461 INFO:teuthology.orchestra.run.smithi016.stderr:pool 'cephfs_metadata' removed 2024-03-31T15:07:47.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:47.470+0000 7fd5da000640 1 -- 172.21.15.16:0/2884762447 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fd5b0068380 msgr2=0x7fd5b006a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:47.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:47.470+0000 7fd5da000640 1 --2- 172.21.15.16:0/2884762447 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fd5b0068380 0x7fd5b006a840 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fd5d4143630 tx=0x7fd5bc009040 comp rx=0 tx=0).stop 2024-03-31T15:07:47.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:47.470+0000 7fd5da000640 1 -- 172.21.15.16:0/2884762447 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd5d40730c0 msgr2=0x7fd5d415bdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:47.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:47.470+0000 7fd5da000640 1 --2- 172.21.15.16:0/2884762447 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd5d40730c0 0x7fd5d415bdc0 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7fd5c400c500 tx=0x7fd5c400c9c0 comp rx=0 tx=0).stop 2024-03-31T15:07:47.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:47.470+0000 7fd5da000640 1 -- 172.21.15.16:0/2884762447 shutdown_connections 2024-03-31T15:07:47.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:47.470+0000 7fd5da000640 1 --2- 172.21.15.16:0/2884762447 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fd5b0068380 0x7fd5b006a840 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:47.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:47.470+0000 7fd5da000640 1 --2- 172.21.15.16:0/2884762447 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fd5d4143f40 0x7fd5d4157870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:47.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:47.470+0000 7fd5da000640 1 --2- 172.21.15.16:0/2884762447 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd5d40b22f0 0x7fd5d4071f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:47.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:47.470+0000 7fd5da000640 1 --2- 172.21.15.16:0/2884762447 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd5d40730c0 0x7fd5d415bdc0 unknown :-1 s=CLOSED pgs=775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:47.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:47.470+0000 7fd5da000640 1 -- 172.21.15.16:0/2884762447 >> 172.21.15.16:0/2884762447 conn(0x7fd5d406b620 msgr2=0x7fd5d40adc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:07:47.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:47.470+0000 7fd5da000640 1 -- 172.21.15.16:0/2884762447 shutdown_connections 2024-03-31T15:07:47.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:47.470+0000 7fd5da000640 1 -- 172.21.15.16:0/2884762447 wait complete. 2024-03-31T15:07:47.490 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-31T15:07:47.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:47.642+0000 7f268ae00640 1 Processor -- start 2024-03-31T15:07:47.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:47.642+0000 7f268ae00640 1 -- start start 2024-03-31T15:07:47.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:47.646+0000 7f268ae00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f26840b22f0 0x7f26840730c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:47.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:47.646+0000 7f268ae00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f26840b45d0 0x7f26840b2f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:47.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:47.646+0000 7f268ae00640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f26840b4a00 con 0x7f2684143f40 2024-03-31T15:07:47.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:47.646+0000 7f268ae00640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f2684144320 con 0x7f26840b22f0 2024-03-31T15:07:47.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:47.646+0000 7f268ae00640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f268415ef80 con 0x7f26840b45d0 2024-03-31T15:07:47.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:47.646+0000 7f267be00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f26840b22f0 0x7f26840730c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:47.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:47.646+0000 7f267be00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f26840b22f0 0x7f26840730c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38538/0 (socket says 172.21.15.16:38538) 2024-03-31T15:07:47.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:47.646+0000 7f267be00640 1 -- 172.21.15.16:0/3399017629 learned_addr learned my addr 172.21.15.16:0/3399017629 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:07:47.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:47.646+0000 7f2688e00640 1 --2- 172.21.15.16:0/3399017629 >> v2:172.21.15.89:3300/0 conn(0x7f26840b45d0 0x7f26840b2f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:47.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:47.646+0000 7f267be00640 1 -- 172.21.15.16:0/3399017629 >> v2:172.21.15.89:3300/0 conn(0x7f26840b45d0 msgr2=0x7f26840b2f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:07:47.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:47.646+0000 7f267be00640 1 --2- 172.21.15.16:0/3399017629 >> v2:172.21.15.89:3300/0 conn(0x7f26840b45d0 0x7f26840b2f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:47.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:47.646+0000 7f267be00640 1 -- 172.21.15.16:0/3399017629 >> v1:172.21.15.89:6789/0 conn(0x7f2684143f40 legacy=0x7f26840b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:47.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:47.646+0000 7f267be00640 1 -- 172.21.15.16:0/3399017629 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f268415f260 con 0x7f26840b22f0 2024-03-31T15:07:47.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:47.646+0000 7f2688e00640 1 --2- 172.21.15.16:0/3399017629 >> v2:172.21.15.89:3300/0 conn(0x7f26840b45d0 0x7f26840b2f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:07:47.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:47.646+0000 7f267be00640 1 --2- 172.21.15.16:0/3399017629 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f26840b22f0 0x7f26840730c0 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7f26840b4f40 tx=0x7f266c01c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27a6140a7243d681 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:07:47.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:47.646+0000 7f2683400640 1 -- 172.21.15.16:0/3399017629 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f266c0b0070 con 0x7f26840b22f0 2024-03-31T15:07:47.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:47.650+0000 7f2683400640 1 -- 172.21.15.16:0/3399017629 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f266c0ab600 con 0x7f26840b22f0 2024-03-31T15:07:47.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:47.650+0000 7f2683400640 1 -- 172.21.15.16:0/3399017629 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f266c0ab9d0 con 0x7f26840b22f0 2024-03-31T15:07:47.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:47.650+0000 7f268ae00640 1 -- 172.21.15.16:0/3399017629 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f26840b22f0 msgr2=0x7f26840730c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:47.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:47.650+0000 7f268ae00640 1 --2- 172.21.15.16:0/3399017629 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f26840b22f0 0x7f26840730c0 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7f26840b4f40 tx=0x7f266c01c520 comp rx=0 tx=0).stop 2024-03-31T15:07:47.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:47.650+0000 7f268ae00640 1 -- 172.21.15.16:0/3399017629 shutdown_connections 2024-03-31T15:07:47.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:47.650+0000 7f268ae00640 1 --2- 172.21.15.16:0/3399017629 >> v2:172.21.15.89:3300/0 conn(0x7f26840b45d0 0x7f26840b2f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:47.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:47.650+0000 7f268ae00640 1 --2- 172.21.15.16:0/3399017629 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f26840b22f0 0x7f26840730c0 unknown :-1 s=CLOSED pgs=776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:47.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:47.650+0000 7f268ae00640 1 -- 172.21.15.16:0/3399017629 >> 172.21.15.16:0/3399017629 conn(0x7f268406b620 msgr2=0x7f268406ba30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:07:47.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:47.650+0000 7f268ae00640 1 -- 172.21.15.16:0/3399017629 shutdown_connections 2024-03-31T15:07:47.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:47.650+0000 7f268ae00640 1 -- 172.21.15.16:0/3399017629 wait complete. 2024-03-31T15:07:47.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:47.650+0000 7f268ae00640 1 Processor -- start 2024-03-31T15:07:47.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:47.650+0000 7f268ae00640 1 -- start start 2024-03-31T15:07:47.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:47.650+0000 7f268ae00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f26840b22f0 0x7f26841fb380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:47.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:47.650+0000 7f268ae00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f26840b45d0 0x7f268413ae00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:47.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:47.650+0000 7f268ae00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f2684143f40 0x7f26840b0a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:47.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:47.650+0000 7f268ae00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f26840d0020 con 0x7f26840b22f0 2024-03-31T15:07:47.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:47.650+0000 7f268ae00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f26840cfea0 con 0x7f26840b45d0 2024-03-31T15:07:47.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:47.650+0000 7f268ae00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f26840d01a0 con 0x7f2684143f40 2024-03-31T15:07:47.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:47.650+0000 7f2683e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f26840b22f0 0x7f26841fb380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:47.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:47.650+0000 7f2683e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f26840b22f0 0x7f26841fb380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38544/0 (socket says 172.21.15.16:38544) 2024-03-31T15:07:47.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:47.650+0000 7f2683e00640 1 -- 172.21.15.16:0/3595744746 learned_addr learned my addr 172.21.15.16:0/3595744746 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:07:47.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:47.650+0000 7f267be00640 1 --2- 172.21.15.16:0/3595744746 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f26840b45d0 0x7f268413ae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:47.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:47.654+0000 7f2688e00640 1 --2- 172.21.15.16:0/3595744746 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f2684143f40 0x7f26840b0a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:47.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:47.654+0000 7f2683e00640 1 -- 172.21.15.16:0/3595744746 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f2684143f40 msgr2=0x7f26840b0a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:47.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:47.654+0000 7f2683e00640 1 --2- 172.21.15.16:0/3595744746 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f2684143f40 0x7f26840b0a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:47.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:47.654+0000 7f2683e00640 1 -- 172.21.15.16:0/3595744746 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f26840b45d0 msgr2=0x7f268413ae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:47.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:47.654+0000 7f2683e00640 1 --2- 172.21.15.16:0/3595744746 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f26840b45d0 0x7f268413ae00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:47.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:47.654+0000 7f2683e00640 1 -- 172.21.15.16:0/3595744746 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f268413b340 con 0x7f26840b22f0 2024-03-31T15:07:47.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:47.654+0000 7f2683e00640 1 --2- 172.21.15.16:0/3595744746 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f26840b22f0 0x7f26841fb380 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7f267400c3c0 tx=0x7f267400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:07:47.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:47.654+0000 7f2680c00640 1 -- 172.21.15.16:0/3595744746 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2674017070 con 0x7f26840b22f0 2024-03-31T15:07:47.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:47.654+0000 7f2680c00640 1 -- 172.21.15.16:0/3595744746 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f26740052d0 con 0x7f26840b22f0 2024-03-31T15:07:47.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:47.654+0000 7f2680c00640 1 -- 172.21.15.16:0/3595744746 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2674003840 con 0x7f26840b22f0 2024-03-31T15:07:47.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:47.654+0000 7f268ae00640 1 -- 172.21.15.16:0/3595744746 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f26841422d0 con 0x7f26840b22f0 2024-03-31T15:07:47.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:47.654+0000 7f268ae00640 1 -- 172.21.15.16:0/3595744746 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f268415a9f0 con 0x7f26840b22f0 2024-03-31T15:07:47.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:47.654+0000 7f268ae00640 1 -- 172.21.15.16:0/3595744746 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2684069950 con 0x7f26840b22f0 2024-03-31T15:07:47.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:47.654+0000 7f2680c00640 1 -- 172.21.15.16:0/3595744746 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f267401a070 con 0x7f26840b22f0 2024-03-31T15:07:47.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:47.654+0000 7f2680c00640 1 --2- 172.21.15.16:0/3595744746 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f2660068380 0x7f266006a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:47.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:47.654+0000 7f2680c00640 1 -- 172.21.15.16:0/3595744746 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 5357+0+0 (secure 0 0 0) 0x7f2674085bb0 con 0x7f26840b22f0 2024-03-31T15:07:47.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:47.658+0000 7f267be00640 1 --2- 172.21.15.16:0/3595744746 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f2660068380 0x7f266006a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:47.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:47.658+0000 7f267be00640 1 --2- 172.21.15.16:0/3595744746 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f2660068380 0x7f266006a840 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f266c0ad000 tx=0x7f266c008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:07:47.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:47.658+0000 7f2680c00640 1 -- 172.21.15.16:0/3595744746 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f268413b340 con 0x7f26840b22f0 2024-03-31T15:07:47.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:47.822+0000 7f268ae00640 1 -- 172.21.15.16:0/3595744746 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7f26840cdd60 con 0x7f26840b22f0 2024-03-31T15:07:48.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:48.474+0000 7f2680c00640 1 -- 172.21.15.16:0/3595744746 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v59) ==== 169+0+0 (secure 0 0 0) 0x7f2674005480 con 0x7f26840b22f0 2024-03-31T15:07:48.471 INFO:teuthology.orchestra.run.smithi016.stderr:pool 'cephfs_data' removed 2024-03-31T15:07:48.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:48.478+0000 7f268ae00640 1 -- 172.21.15.16:0/3595744746 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f2660068380 msgr2=0x7f266006a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:48.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:48.478+0000 7f268ae00640 1 --2- 172.21.15.16:0/3595744746 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f2660068380 0x7f266006a840 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f266c0ad000 tx=0x7f266c008040 comp rx=0 tx=0).stop 2024-03-31T15:07:48.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:48.478+0000 7f268ae00640 1 -- 172.21.15.16:0/3595744746 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f26840b22f0 msgr2=0x7f26841fb380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:48.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:48.478+0000 7f268ae00640 1 --2- 172.21.15.16:0/3595744746 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f26840b22f0 0x7f26841fb380 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7f267400c3c0 tx=0x7f267400c880 comp rx=0 tx=0).stop 2024-03-31T15:07:48.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:48.478+0000 7f268ae00640 1 -- 172.21.15.16:0/3595744746 shutdown_connections 2024-03-31T15:07:48.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:48.478+0000 7f268ae00640 1 --2- 172.21.15.16:0/3595744746 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f2660068380 0x7f266006a840 secure :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f266c0ad000 tx=0x7f266c008040 comp rx=0 tx=0).stop 2024-03-31T15:07:48.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:48.478+0000 7f268ae00640 1 --2- 172.21.15.16:0/3595744746 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f2684143f40 0x7f26840b0a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:48.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:48.478+0000 7f268ae00640 1 --2- 172.21.15.16:0/3595744746 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f26840b45d0 0x7f268413ae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:48.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:48.478+0000 7f268ae00640 1 --2- 172.21.15.16:0/3595744746 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f26840b22f0 0x7f26841fb380 secure :-1 s=CLOSED pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7f267400c3c0 tx=0x7f267400c880 comp rx=0 tx=0).stop 2024-03-31T15:07:48.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:48.478+0000 7f268ae00640 1 -- 172.21.15.16:0/3595744746 >> 172.21.15.16:0/3595744746 conn(0x7f268406b620 msgr2=0x7f26840cf770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:07:48.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:48.482+0000 7f268ae00640 1 -- 172.21.15.16:0/3595744746 shutdown_connections 2024-03-31T15:07:48.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:48.482+0000 7f268ae00640 1 -- 172.21.15.16:0/3595744746 wait complete. 2024-03-31T15:07:48.499 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-03-31T15:07:48.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:48.650+0000 7ff23f600640 1 Processor -- start 2024-03-31T15:07:48.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:48.650+0000 7ff23f600640 1 -- start start 2024-03-31T15:07:48.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:48.654+0000 7ff23f600640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff238147d60 0x7ff2380b3130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:48.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:48.654+0000 7ff23f600640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7ff238073310 0x7ff23806d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:48.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:48.654+0000 7ff23f600640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff238073790 con 0x7ff23806da10 2024-03-31T15:07:48.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:48.654+0000 7ff23f600640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7ff238148140 con 0x7ff238073310 2024-03-31T15:07:48.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:48.654+0000 7ff23f600640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7ff23806de80 con 0x7ff238147d60 2024-03-31T15:07:48.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:48.654+0000 7ff23cc00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff238147d60 0x7ff2380b3130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:48.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:48.654+0000 7ff22fe00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7ff238073310 0x7ff23806d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:48.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:48.654+0000 7ff23cc00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff238147d60 0x7ff2380b3130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3301/0 says I am v2:172.21.15.16:57064/0 (socket says 172.21.15.16:57064) 2024-03-31T15:07:48.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:48.654+0000 7ff23cc00640 1 -- 172.21.15.16:0/3485793866 learned_addr learned my addr 172.21.15.16:0/3485793866 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:07:48.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:48.654+0000 7ff23cc00640 1 -- 172.21.15.16:0/3485793866 >> v2:172.21.15.89:3300/0 conn(0x7ff238073310 msgr2=0x7ff23806d4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:07:48.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:48.654+0000 7ff23cc00640 1 --2- 172.21.15.16:0/3485793866 >> v2:172.21.15.89:3300/0 conn(0x7ff238073310 0x7ff23806d4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:48.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:48.654+0000 7ff23cc00640 1 -- 172.21.15.16:0/3485793866 >> v1:172.21.15.89:6789/0 conn(0x7ff23806da10 legacy=0x7ff238161f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:48.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:48.654+0000 7ff23cc00640 1 -- 172.21.15.16:0/3485793866 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2380b37c0 con 0x7ff238147d60 2024-03-31T15:07:48.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:48.654+0000 7ff237e00640 1 -- 172.21.15.16:0/3485793866 <== mon.1 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1396108523 0 0) 0x7ff238073790 con 0x7ff23806da10 2024-03-31T15:07:48.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:48.654+0000 7ff22fe00640 1 --2- 172.21.15.16:0/3485793866 >> v2:172.21.15.89:3300/0 conn(0x7ff238073310 0x7ff23806d4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:07:48.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:48.658+0000 7ff23cc00640 1 --2- 172.21.15.16:0/3485793866 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff238147d60 0x7ff2380b3130 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7ff228009870 tx=0x7ff22801c530 comp rx=0 tx=0).ready entity=mon.2 client_cookie=89cd64be6feef376 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:07:48.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:48.658+0000 7ff237e00640 1 -- 172.21.15.16:0/3485793866 <== mon.2 v2:172.21.15.89:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2280a95f0 con 0x7ff238147d60 2024-03-31T15:07:48.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:48.658+0000 7ff237e00640 1 -- 172.21.15.16:0/3485793866 <== mon.2 v2:172.21.15.89:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff2280a9790 con 0x7ff238147d60 2024-03-31T15:07:48.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:48.658+0000 7ff237e00640 1 -- 172.21.15.16:0/3485793866 <== mon.2 v2:172.21.15.89:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2280a9b60 con 0x7ff238147d60 2024-03-31T15:07:48.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:48.658+0000 7ff23f600640 1 -- 172.21.15.16:0/3485793866 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff238147d60 msgr2=0x7ff2380b3130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:48.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:48.658+0000 7ff23f600640 1 --2- 172.21.15.16:0/3485793866 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff238147d60 0x7ff2380b3130 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7ff228009870 tx=0x7ff22801c530 comp rx=0 tx=0).stop 2024-03-31T15:07:48.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:48.658+0000 7ff23f600640 1 -- 172.21.15.16:0/3485793866 shutdown_connections 2024-03-31T15:07:48.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:48.658+0000 7ff23f600640 1 --2- 172.21.15.16:0/3485793866 >> v2:172.21.15.89:3300/0 conn(0x7ff238073310 0x7ff23806d4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:48.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:48.658+0000 7ff23f600640 1 --2- 172.21.15.16:0/3485793866 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff238147d60 0x7ff2380b3130 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:48.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:48.658+0000 7ff23f600640 1 -- 172.21.15.16:0/3485793866 >> 172.21.15.16:0/3485793866 conn(0x7ff23806ee40 msgr2=0x7ff23806b620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:07:48.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:48.658+0000 7ff23f600640 1 -- 172.21.15.16:0/3485793866 shutdown_connections 2024-03-31T15:07:48.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:48.658+0000 7ff23f600640 1 -- 172.21.15.16:0/3485793866 wait complete. 2024-03-31T15:07:48.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:48.658+0000 7ff23f600640 1 Processor -- start 2024-03-31T15:07:48.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:48.658+0000 7ff23f600640 1 -- start start 2024-03-31T15:07:48.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:48.662+0000 7ff23f600640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff23806da10 0x7ff23813fa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:48.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:48.662+0000 7ff23f600640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff238073310 0x7ff2380b0360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:48.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:48.662+0000 7ff23f600640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff238147d60 0x7ff238144620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:48.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:48.662+0000 7ff23f600640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7ff23816b2e0 con 0x7ff238073310 2024-03-31T15:07:48.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:48.662+0000 7ff23f600640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7ff2380b4340 con 0x7ff238147d60 2024-03-31T15:07:48.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:48.662+0000 7ff23f600640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7ff23816b460 con 0x7ff23806da10 2024-03-31T15:07:48.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:48.662+0000 7ff22fe00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff238073310 0x7ff2380b0360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:48.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:48.662+0000 7ff22fe00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff238073310 0x7ff2380b0360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38548/0 (socket says 172.21.15.16:38548) 2024-03-31T15:07:48.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:48.662+0000 7ff22fe00640 1 -- 172.21.15.16:0/1911949030 learned_addr learned my addr 172.21.15.16:0/1911949030 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:07:48.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:48.662+0000 7ff23cc00640 1 --2- 172.21.15.16:0/1911949030 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff23806da10 0x7ff23813fa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:48.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:48.662+0000 7ff23d600640 1 --2- 172.21.15.16:0/1911949030 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff238147d60 0x7ff238144620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:48.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:48.662+0000 7ff22fe00640 1 -- 172.21.15.16:0/1911949030 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff23806da10 msgr2=0x7ff23813fa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:48.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:48.662+0000 7ff22fe00640 1 --2- 172.21.15.16:0/1911949030 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff23806da10 0x7ff23813fa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:48.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:48.662+0000 7ff22fe00640 1 -- 172.21.15.16:0/1911949030 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff238147d60 msgr2=0x7ff238144620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:48.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:48.662+0000 7ff22fe00640 1 --2- 172.21.15.16:0/1911949030 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff238147d60 0x7ff238144620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:48.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:48.662+0000 7ff22fe00640 1 -- 172.21.15.16:0/1911949030 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2380b1730 con 0x7ff238073310 2024-03-31T15:07:48.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:48.662+0000 7ff22fe00640 1 --2- 172.21.15.16:0/1911949030 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff238073310 0x7ff2380b0360 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7ff22000ebe0 tx=0x7ff22000c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:07:48.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:48.662+0000 7ff235600640 1 -- 172.21.15.16:0/1911949030 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff220017070 con 0x7ff238073310 2024-03-31T15:07:48.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:48.662+0000 7ff235600640 1 -- 172.21.15.16:0/1911949030 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff22000cc30 con 0x7ff238073310 2024-03-31T15:07:48.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:48.662+0000 7ff235600640 1 -- 172.21.15.16:0/1911949030 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff220013470 con 0x7ff238073310 2024-03-31T15:07:48.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:48.662+0000 7ff23f600640 1 -- 172.21.15.16:0/1911949030 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff23813ff90 con 0x7ff238073310 2024-03-31T15:07:48.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:48.662+0000 7ff23f600640 1 -- 172.21.15.16:0/1911949030 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff2381651b0 con 0x7ff238073310 2024-03-31T15:07:48.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:48.662+0000 7ff23f600640 1 -- 172.21.15.16:0/1911949030 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff238069950 con 0x7ff238073310 2024-03-31T15:07:48.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:48.666+0000 7ff235600640 1 -- 172.21.15.16:0/1911949030 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7ff22001a070 con 0x7ff238073310 2024-03-31T15:07:48.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:48.666+0000 7ff235600640 1 --2- 172.21.15.16:0/1911949030 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7ff214068380 0x7ff21406a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:48.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:48.666+0000 7ff235600640 1 -- 172.21.15.16:0/1911949030 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(59..59 src has 1..59) ==== 4943+0+0 (secure 0 0 0) 0x7ff2200865a0 con 0x7ff238073310 2024-03-31T15:07:48.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:48.666+0000 7ff23cc00640 1 --2- 172.21.15.16:0/1911949030 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7ff214068380 0x7ff21406a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:48.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:48.666+0000 7ff23cc00640 1 --2- 172.21.15.16:0/1911949030 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7ff214068380 0x7ff21406a840 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7ff2280ae000 tx=0x7ff2280a7000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:07:48.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:48.670+0000 7ff235600640 1 -- 172.21.15.16:0/1911949030 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7ff220056800 con 0x7ff238073310 2024-03-31T15:07:48.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:48.834+0000 7ff23f600640 1 -- 172.21.15.16:0/1911949030 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) -- 0x7ff23806e470 con 0x7ff238073310 2024-03-31T15:07:48.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:48.838+0000 7ff235600640 1 -- 172.21.15.16:0/1911949030 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v15) ==== 104+0+0 (secure 0 0 0) 0x7ff22005ab00 con 0x7ff238073310 2024-03-31T15:07:48.836 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:48.842+0000 7ff23f600640 1 -- 172.21.15.16:0/1911949030 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7ff214068380 msgr2=0x7ff21406a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:48.836 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:48.842+0000 7ff23f600640 1 --2- 172.21.15.16:0/1911949030 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7ff214068380 0x7ff21406a840 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7ff2280ae000 tx=0x7ff2280a7000 comp rx=0 tx=0).stop 2024-03-31T15:07:48.836 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:48.842+0000 7ff23f600640 1 -- 172.21.15.16:0/1911949030 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff238073310 msgr2=0x7ff2380b0360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:48.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:48.842+0000 7ff23f600640 1 --2- 172.21.15.16:0/1911949030 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff238073310 0x7ff2380b0360 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7ff22000ebe0 tx=0x7ff22000c520 comp rx=0 tx=0).stop 2024-03-31T15:07:48.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:48.842+0000 7ff23f600640 1 -- 172.21.15.16:0/1911949030 shutdown_connections 2024-03-31T15:07:48.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:48.842+0000 7ff23f600640 1 --2- 172.21.15.16:0/1911949030 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7ff214068380 0x7ff21406a840 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:48.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:48.842+0000 7ff23f600640 1 --2- 172.21.15.16:0/1911949030 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff238147d60 0x7ff238144620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:48.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:48.842+0000 7ff23f600640 1 --2- 172.21.15.16:0/1911949030 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff238073310 0x7ff2380b0360 unknown :-1 s=CLOSED pgs=778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:48.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:48.842+0000 7ff23f600640 1 --2- 172.21.15.16:0/1911949030 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff23806da10 0x7ff23813fa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:48.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:48.842+0000 7ff23f600640 1 -- 172.21.15.16:0/1911949030 >> 172.21.15.16:0/1911949030 conn(0x7ff23806ee40 msgr2=0x7ff238145690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:07:48.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:48.842+0000 7ff23f600640 1 -- 172.21.15.16:0/1911949030 shutdown_connections 2024-03-31T15:07:48.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:48.842+0000 7ff23f600640 1 -- 172.21.15.16:0/1911949030 wait complete. 2024-03-31T15:07:48.855 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_health_status_after_backtrace_repair' 2024-03-31T15:07:48.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:49.002+0000 7f608e600640 1 Processor -- start 2024-03-31T15:07:48.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:49.002+0000 7f608e600640 1 -- start start 2024-03-31T15:07:48.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:49.002+0000 7f608e600640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f608815b3d0 0x7f60880b1cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:48.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:49.002+0000 7f608e600640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f60880ce2f0 0x7f60880b2200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:48.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:49.002+0000 7f608e600640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f608815f3c0 con 0x7f608815eee0 2024-03-31T15:07:48.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:49.002+0000 7f608e600640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f608815b7b0 con 0x7f60880ce2f0 2024-03-31T15:07:48.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:49.002+0000 7f608e600640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f608815f6a0 con 0x7f608815b3d0 2024-03-31T15:07:48.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:49.002+0000 7f6087e00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f60880ce2f0 0x7f60880b2200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:48.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:49.002+0000 7f6087e00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f60880ce2f0 0x7f60880b2200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38564/0 (socket says 172.21.15.16:38564) 2024-03-31T15:07:48.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:49.002+0000 7f607fe00640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f608815eee0 0x7f60880cc9c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:49292/0 (socket says 172.21.15.16:49292) 2024-03-31T15:07:48.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:49.002+0000 7f6087e00640 1 -- 172.21.15.16:0/2881436226 learned_addr learned my addr 172.21.15.16:0/2881436226 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:07:48.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:49.002+0000 7f6087400640 1 --2- 172.21.15.16:0/2881436226 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f608815b3d0 0x7f60880b1cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:48.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:49.002+0000 7f6087e00640 1 -- 172.21.15.16:0/2881436226 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f608815b3d0 msgr2=0x7f60880b1cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:07:48.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:49.002+0000 7f6087e00640 1 --2- 172.21.15.16:0/2881436226 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f608815b3d0 0x7f60880b1cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:48.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:49.002+0000 7f6086a00640 1 -- 172.21.15.16:0/2881436226 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4276973551 0 0) 0x7f608815f3c0 con 0x7f608815eee0 2024-03-31T15:07:48.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:49.002+0000 7f6087e00640 1 -- 172.21.15.16:0/2881436226 >> v1:172.21.15.16:6789/0 conn(0x7f608815eee0 legacy=0x7f60880cc9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:48.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:49.002+0000 7f6087e00640 1 -- 172.21.15.16:0/2881436226 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f608815f980 con 0x7f60880ce2f0 2024-03-31T15:07:49.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:49.006+0000 7f6087e00640 1 --2- 172.21.15.16:0/2881436226 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f60880ce2f0 0x7f60880b2200 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7f6080009e50 tx=0x7f608001ca00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=edb4d01abcc57d1f server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:07:49.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:49.006+0000 7f6086a00640 1 -- 172.21.15.16:0/2881436226 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60800a5890 con 0x7f60880ce2f0 2024-03-31T15:07:49.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:49.006+0000 7f6086a00640 1 -- 172.21.15.16:0/2881436226 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f60800a5a30 con 0x7f60880ce2f0 2024-03-31T15:07:49.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:49.006+0000 7f6086a00640 1 -- 172.21.15.16:0/2881436226 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60800a5e00 con 0x7f60880ce2f0 2024-03-31T15:07:49.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:49.006+0000 7f608e600640 1 -- 172.21.15.16:0/2881436226 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f60880ce2f0 msgr2=0x7f60880b2200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:49.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:49.006+0000 7f608e600640 1 --2- 172.21.15.16:0/2881436226 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f60880ce2f0 0x7f60880b2200 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7f6080009e50 tx=0x7f608001ca00 comp rx=0 tx=0).stop 2024-03-31T15:07:49.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:49.006+0000 7f608e600640 1 -- 172.21.15.16:0/2881436226 shutdown_connections 2024-03-31T15:07:49.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:49.006+0000 7f608e600640 1 --2- 172.21.15.16:0/2881436226 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f60880ce2f0 0x7f60880b2200 unknown :-1 s=CLOSED pgs=780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:49.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:49.006+0000 7f608e600640 1 --2- 172.21.15.16:0/2881436226 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f608815b3d0 0x7f60880b1cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:49.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:49.006+0000 7f608e600640 1 -- 172.21.15.16:0/2881436226 >> 172.21.15.16:0/2881436226 conn(0x7f608806b620 msgr2=0x7f608806ba30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:07:49.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:49.006+0000 7f608e600640 1 -- 172.21.15.16:0/2881436226 shutdown_connections 2024-03-31T15:07:49.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:49.006+0000 7f608e600640 1 -- 172.21.15.16:0/2881436226 wait complete. 2024-03-31T15:07:49.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:49.010+0000 7f608e600640 1 Processor -- start 2024-03-31T15:07:49.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:49.010+0000 7f608e600640 1 -- start start 2024-03-31T15:07:49.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:49.010+0000 7f608e600640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f60880ce2f0 0x7f6088138940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:49.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:49.010+0000 7f608e600640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f608815b3d0 0x7f60880cf8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:49.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:49.010+0000 7f608e600640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f608815eee0 0x7f60880cfde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:49.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:49.010+0000 7f608e600640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f6088168610 con 0x7f608815b3d0 2024-03-31T15:07:49.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:49.010+0000 7f608e600640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f6088168490 con 0x7f60880ce2f0 2024-03-31T15:07:49.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:49.010+0000 7f608e600640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f6088168790 con 0x7f608815eee0 2024-03-31T15:07:49.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:49.010+0000 7f607fe00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f608815b3d0 0x7f60880cf8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:49.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:49.010+0000 7f607fe00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f608815b3d0 0x7f60880cf8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38570/0 (socket says 172.21.15.16:38570) 2024-03-31T15:07:49.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:49.010+0000 7f607fe00640 1 -- 172.21.15.16:0/4002709194 learned_addr learned my addr 172.21.15.16:0/4002709194 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:07:49.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:49.010+0000 7f6087400640 1 --2- 172.21.15.16:0/4002709194 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f60880ce2f0 0x7f6088138940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:49.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:49.010+0000 7f6087e00640 1 --2- 172.21.15.16:0/4002709194 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f608815eee0 0x7f60880cfde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:49.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:49.010+0000 7f607fe00640 1 -- 172.21.15.16:0/4002709194 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f608815eee0 msgr2=0x7f60880cfde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:49.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:49.010+0000 7f607fe00640 1 --2- 172.21.15.16:0/4002709194 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f608815eee0 0x7f60880cfde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:49.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:49.010+0000 7f607fe00640 1 -- 172.21.15.16:0/4002709194 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f60880ce2f0 msgr2=0x7f6088138940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:49.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:49.010+0000 7f607fe00640 1 --2- 172.21.15.16:0/4002709194 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f60880ce2f0 0x7f6088138940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:49.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:49.010+0000 7f607fe00640 1 -- 172.21.15.16:0/4002709194 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60880aa2a0 con 0x7f608815b3d0 2024-03-31T15:07:49.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:49.010+0000 7f607fe00640 1 --2- 172.21.15.16:0/4002709194 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f608815b3d0 0x7f60880cf8a0 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f60700011f0 tx=0x7f607000c9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:07:49.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:49.010+0000 7f607f400640 1 -- 172.21.15.16:0/4002709194 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6070018070 con 0x7f608815b3d0 2024-03-31T15:07:49.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:49.010+0000 7f607f400640 1 -- 172.21.15.16:0/4002709194 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f60700074e0 con 0x7f608815b3d0 2024-03-31T15:07:49.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:49.010+0000 7f608e600640 1 -- 172.21.15.16:0/4002709194 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f60880aa580 con 0x7f608815b3d0 2024-03-31T15:07:49.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:49.010+0000 7f607f400640 1 -- 172.21.15.16:0/4002709194 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6070004a10 con 0x7f608815b3d0 2024-03-31T15:07:49.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:49.010+0000 7f608e600640 1 -- 172.21.15.16:0/4002709194 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f60880aab40 con 0x7f608815b3d0 2024-03-31T15:07:49.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:49.010+0000 7f608e600640 1 -- 172.21.15.16:0/4002709194 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6088069950 con 0x7f608815b3d0 2024-03-31T15:07:49.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:49.014+0000 7f607f400640 1 -- 172.21.15.16:0/4002709194 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f607001b070 con 0x7f608815b3d0 2024-03-31T15:07:49.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:49.014+0000 7f607f400640 1 --2- 172.21.15.16:0/4002709194 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f6064068380 0x7f606406a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:49.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:49.014+0000 7f607f400640 1 -- 172.21.15.16:0/4002709194 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(59..59 src has 1..59) ==== 4943+0+0 (secure 0 0 0) 0x7f6070085a70 con 0x7f608815b3d0 2024-03-31T15:07:49.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:49.014+0000 7f6087400640 1 --2- 172.21.15.16:0/4002709194 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f6064068380 0x7f606406a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:49.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:49.014+0000 7f6087400640 1 --2- 172.21.15.16:0/4002709194 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f6064068380 0x7f606406a840 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f607800b930 tx=0x7f6078008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:07:49.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:49.018+0000 7f607f400640 1 -- 172.21.15.16:0/4002709194 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f6070056290 con 0x7f608815b3d0 2024-03-31T15:07:49.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:49.242+0000 7f608e600640 1 -- 172.21.15.16:0/4002709194 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_health_status_after_backtrace_repair"]} v 0) -- 0x7f608815ab30 con 0x7f608815b3d0 2024-03-31T15:07:49.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:49.482+0000 7f607f400640 1 -- 172.21.15.16:0/4002709194 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_health_status_after_backtrace_repair"]}]=0 v221) ==== 169+0+0 (secure 0 0 0) 0x7f6070055b90 con 0x7f608815b3d0 2024-03-31T15:07:49.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:49.486+0000 7f608e600640 1 -- 172.21.15.16:0/4002709194 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f6064068380 msgr2=0x7f606406a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:49.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:49.486+0000 7f608e600640 1 --2- 172.21.15.16:0/4002709194 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f6064068380 0x7f606406a840 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f607800b930 tx=0x7f6078008040 comp rx=0 tx=0).stop 2024-03-31T15:07:49.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:49.486+0000 7f608e600640 1 -- 172.21.15.16:0/4002709194 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f608815b3d0 msgr2=0x7f60880cf8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:49.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:49.486+0000 7f608e600640 1 --2- 172.21.15.16:0/4002709194 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f608815b3d0 0x7f60880cf8a0 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f60700011f0 tx=0x7f607000c9b0 comp rx=0 tx=0).stop 2024-03-31T15:07:49.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:49.486+0000 7f608e600640 1 -- 172.21.15.16:0/4002709194 shutdown_connections 2024-03-31T15:07:49.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:49.486+0000 7f608e600640 1 --2- 172.21.15.16:0/4002709194 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f6064068380 0x7f606406a840 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:49.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:49.486+0000 7f608e600640 1 --2- 172.21.15.16:0/4002709194 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f608815eee0 0x7f60880cfde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:49.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:49.486+0000 7f608e600640 1 --2- 172.21.15.16:0/4002709194 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f608815b3d0 0x7f60880cf8a0 unknown :-1 s=CLOSED pgs=781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:49.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:49.486+0000 7f608e600640 1 --2- 172.21.15.16:0/4002709194 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f60880ce2f0 0x7f6088138940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:49.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:49.486+0000 7f608e600640 1 -- 172.21.15.16:0/4002709194 >> 172.21.15.16:0/4002709194 conn(0x7f608806b620 msgr2=0x7f60880b1b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:07:49.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:49.486+0000 7f608e600640 1 -- 172.21.15.16:0/4002709194 shutdown_connections 2024-03-31T15:07:49.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:49.486+0000 7f608e600640 1 -- 172.21.15.16:0/4002709194 wait complete. 2024-03-31T15:07:49.502 INFO:tasks.cephfs_test_runner:Test that the damage health status is cleared ... ok 2024-03-31T15:07:49.502 INFO:tasks.cephfs_test_runner:Starting test: test_health_status_after_dentry_repair (tasks.cephfs.test_forward_scrub.TestForwardScrub) Test that the damage health status is cleared 2024-03-31T15:07:49.502 INFO:tasks.cephfs_test_runner:test_health_status_after_dentry_repair (tasks.cephfs.test_forward_scrub.TestForwardScrub) 2024-03-31T15:07:49.503 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-31T15:07:49.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:49.646+0000 7fbd46200640 1 Processor -- start 2024-03-31T15:07:49.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:49.646+0000 7fbd46200640 1 -- start start 2024-03-31T15:07:49.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:49.650+0000 7fbd46200640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fbd40145d10 0x7fbd400cf790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:49.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:49.650+0000 7fbd46200640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fbd400b23d0 0x7fbd4015cd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:49.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:49.650+0000 7fbd46200640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbd40160e70 con 0x7fbd400728b0 2024-03-31T15:07:49.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:49.650+0000 7fbd46200640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7fbd401602f0 con 0x7fbd400b23d0 2024-03-31T15:07:49.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:49.650+0000 7fbd46200640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fbd401605d0 con 0x7fbd40145d10 2024-03-31T15:07:49.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:49.650+0000 7fbd3fe00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fbd400b23d0 0x7fbd4015cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:49.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:49.650+0000 7fbd3fe00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fbd400b23d0 0x7fbd4015cd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38574/0 (socket says 172.21.15.16:38574) 2024-03-31T15:07:49.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:49.650+0000 7fbd3fe00640 1 -- 172.21.15.16:0/643814446 learned_addr learned my addr 172.21.15.16:0/643814446 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:07:49.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:49.650+0000 7fbd3f400640 1 --2- 172.21.15.16:0/643814446 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fbd40145d10 0x7fbd400cf790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:49.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:49.650+0000 7fbd3fe00640 1 -- 172.21.15.16:0/643814446 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fbd40145d10 msgr2=0x7fbd400cf790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:07:49.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:49.650+0000 7fbd3fe00640 1 --2- 172.21.15.16:0/643814446 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fbd40145d10 0x7fbd400cf790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:49.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:49.650+0000 7fbd3fe00640 1 -- 172.21.15.16:0/643814446 >> v1:172.21.15.89:6789/0 conn(0x7fbd400728b0 legacy=0x7fbd400b1cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:49.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:49.650+0000 7fbd3fe00640 1 -- 172.21.15.16:0/643814446 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd400cfcd0 con 0x7fbd400b23d0 2024-03-31T15:07:49.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:49.654+0000 7fbd3fe00640 1 --2- 172.21.15.16:0/643814446 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbd400b23d0 0x7fbd4015cd50 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7fbd38009e50 tx=0x7fbd3801ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54b70506a47b24df server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:07:49.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:49.654+0000 7fbd3ea00640 1 -- 172.21.15.16:0/643814446 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd380a5700 con 0x7fbd400b23d0 2024-03-31T15:07:49.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:49.654+0000 7fbd3ea00640 1 -- 172.21.15.16:0/643814446 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbd380a58a0 con 0x7fbd400b23d0 2024-03-31T15:07:49.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:49.654+0000 7fbd3ea00640 1 -- 172.21.15.16:0/643814446 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd380a5c70 con 0x7fbd400b23d0 2024-03-31T15:07:49.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:49.654+0000 7fbd46200640 1 -- 172.21.15.16:0/643814446 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbd400b23d0 msgr2=0x7fbd4015cd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:49.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:49.654+0000 7fbd46200640 1 --2- 172.21.15.16:0/643814446 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbd400b23d0 0x7fbd4015cd50 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7fbd38009e50 tx=0x7fbd3801ca30 comp rx=0 tx=0).stop 2024-03-31T15:07:49.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:49.654+0000 7fbd46200640 1 -- 172.21.15.16:0/643814446 shutdown_connections 2024-03-31T15:07:49.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:49.654+0000 7fbd46200640 1 --2- 172.21.15.16:0/643814446 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbd400b23d0 0x7fbd4015cd50 unknown :-1 s=CLOSED pgs=782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:49.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:49.654+0000 7fbd46200640 1 --2- 172.21.15.16:0/643814446 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fbd40145d10 0x7fbd400cf790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:49.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:49.654+0000 7fbd46200640 1 -- 172.21.15.16:0/643814446 >> 172.21.15.16:0/643814446 conn(0x7fbd4006b620 msgr2=0x7fbd4006ba30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:07:49.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:49.654+0000 7fbd46200640 1 -- 172.21.15.16:0/643814446 shutdown_connections 2024-03-31T15:07:49.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:49.654+0000 7fbd46200640 1 -- 172.21.15.16:0/643814446 wait complete. 2024-03-31T15:07:49.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:49.654+0000 7fbd46200640 1 Processor -- start 2024-03-31T15:07:49.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:49.654+0000 7fbd46200640 1 -- start start 2024-03-31T15:07:49.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:49.654+0000 7fbd46200640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fbd400728b0 0x7fbd4013bad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:49.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:49.654+0000 7fbd46200640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbd400b23d0 0x7fbd40144490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:49.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:49.654+0000 7fbd46200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbd40145d10 0x7fbd401449d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:49.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:49.658+0000 7fbd46200640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fbd400cf4d0 con 0x7fbd40145d10 2024-03-31T15:07:49.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:49.658+0000 7fbd46200640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fbd400cf350 con 0x7fbd400b23d0 2024-03-31T15:07:49.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:49.658+0000 7fbd46200640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fbd40072c90 con 0x7fbd400728b0 2024-03-31T15:07:49.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:49.658+0000 7fbd3fe00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbd40145d10 0x7fbd401449d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:49.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:49.658+0000 7fbd3fe00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbd40145d10 0x7fbd401449d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38590/0 (socket says 172.21.15.16:38590) 2024-03-31T15:07:49.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:49.658+0000 7fbd3fe00640 1 -- 172.21.15.16:0/2417251005 learned_addr learned my addr 172.21.15.16:0/2417251005 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:07:49.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:49.658+0000 7fbd3f400640 1 --2- 172.21.15.16:0/2417251005 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fbd400728b0 0x7fbd4013bad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:49.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:49.658+0000 7fbd36a00640 1 --2- 172.21.15.16:0/2417251005 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbd400b23d0 0x7fbd40144490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:49.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:49.658+0000 7fbd3fe00640 1 -- 172.21.15.16:0/2417251005 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fbd400728b0 msgr2=0x7fbd4013bad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:49.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:49.658+0000 7fbd3fe00640 1 --2- 172.21.15.16:0/2417251005 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fbd400728b0 0x7fbd4013bad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:49.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:49.658+0000 7fbd3fe00640 1 -- 172.21.15.16:0/2417251005 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbd400b23d0 msgr2=0x7fbd40144490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:49.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:49.658+0000 7fbd3fe00640 1 --2- 172.21.15.16:0/2417251005 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbd400b23d0 0x7fbd40144490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:49.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:49.658+0000 7fbd3fe00640 1 -- 172.21.15.16:0/2417251005 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd40140a30 con 0x7fbd40145d10 2024-03-31T15:07:49.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:49.658+0000 7fbd3fe00640 1 --2- 172.21.15.16:0/2417251005 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbd40145d10 0x7fbd401449d0 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7fbd3801a660 tx=0x7fbd380197d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:07:49.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:49.658+0000 7fbd37e00640 1 -- 172.21.15.16:0/2417251005 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd380bc020 con 0x7fbd40145d10 2024-03-31T15:07:49.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:49.658+0000 7fbd37e00640 1 -- 172.21.15.16:0/2417251005 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbd380b8670 con 0x7fbd40145d10 2024-03-31T15:07:49.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:49.658+0000 7fbd37e00640 1 -- 172.21.15.16:0/2417251005 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd38009510 con 0x7fbd40145d10 2024-03-31T15:07:49.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:49.658+0000 7fbd46200640 1 -- 172.21.15.16:0/2417251005 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbd4013cd10 con 0x7fbd40145d10 2024-03-31T15:07:49.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:49.658+0000 7fbd46200640 1 -- 172.21.15.16:0/2417251005 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbd4013d2d0 con 0x7fbd40145d10 2024-03-31T15:07:49.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:49.658+0000 7fbd46200640 1 -- 172.21.15.16:0/2417251005 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbd40069950 con 0x7fbd40145d10 2024-03-31T15:07:49.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:49.658+0000 7fbd37e00640 1 -- 172.21.15.16:0/2417251005 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fbd380bf020 con 0x7fbd40145d10 2024-03-31T15:07:49.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:49.658+0000 7fbd37e00640 1 --2- 172.21.15.16:0/2417251005 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fbd1c068380 0x7fbd1c06a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:49.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:49.662+0000 7fbd37e00640 1 -- 172.21.15.16:0/2417251005 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(59..59 src has 1..59) ==== 4943+0+0 (secure 0 0 0) 0x7fbd381216c0 con 0x7fbd40145d10 2024-03-31T15:07:49.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:49.662+0000 7fbd3f400640 1 --2- 172.21.15.16:0/2417251005 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fbd1c068380 0x7fbd1c06a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:49.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:49.662+0000 7fbd3f400640 1 --2- 172.21.15.16:0/2417251005 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fbd1c068380 0x7fbd1c06a840 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7fbd40143170 tx=0x7fbd30008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:07:49.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:49.662+0000 7fbd37e00640 1 -- 172.21.15.16:0/2417251005 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fbd380c6030 con 0x7fbd40145d10 2024-03-31T15:07:49.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:49.830+0000 7fbd46200640 1 -- 172.21.15.16:0/2417251005 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fbd4013e460 con 0x7fbd40145d10 2024-03-31T15:07:49.825 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:49.830+0000 7fbd37e00640 1 -- 172.21.15.16:0/2417251005 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v59) ==== 74+0+12948 (secure 0 0 0) 0x7fbd40140a30 con 0x7fbd40145d10 2024-03-31T15:07:49.825 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:07:49.825 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":59,"fsid":"286eb5da-fb47-4550-b290-d9519c13b9a5","created":"2024-03-31T15:03:11.438952+0000","modified":"2024-03-31T15:07:48.471162+0000","last_up_change":"2024-03-31T15:03:22.885535+0000","last_in_change":"2024-03-31T15:03:13.941667+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-31T15:03:22.550787+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"92a66450-ef19-40d8-ac3d-d760b23b0bda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6801","nonce":932712244}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6803","nonce":932712244}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6807","nonce":932712244}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6805","nonce":932712244}]},"public_addr":"172.21.15.16:6801/932712244","cluster_addr":"172.21.15.16:6803/932712244","heartbeat_back_addr":"172.21.15.16:6807/932712244","heartbeat_front_addr":"172.21.15.16:6805/932712244","state":["exists","up"]},{"osd":1,"uuid":"da9d1f2b-1968-4959-b843-faa231988e92","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6810","nonce":1398737533}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6811","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6813","nonce":1398737533}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6821","nonce":1398737533}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6815","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6816","nonce":1398737533}]},"public_addr":"172.21.15.16:6810/1398737533","cluster_addr":"172.21.15.16:6813/1398737533","heartbeat_back_addr":"172.21.15.16:6821/1398737533","heartbeat_front_addr":"172.21.15.16:6816/1398737533","state":["exists","up"]},{"osd":2,"uuid":"01a73662-9a21-4b1f-bcbb-42b9636a4638","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6819","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6822","nonce":3018526646}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6825","nonce":3018526646}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6831","nonce":3018526646}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6827","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6829","nonce":3018526646}]},"public_addr":"172.21.15.16:6822/3018526646","cluster_addr":"172.21.15.16:6825/3018526646","heartbeat_back_addr":"172.21.15.16:6831/3018526646","heartbeat_front_addr":"172.21.15.16:6829/3018526646","state":["exists","up"]},{"osd":3,"uuid":"c7771fd3-e18e-4500-ac38-23cf10ebcd75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6809","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6812","nonce":3360352974}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6817","nonce":3360352974}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6828","nonce":3360352974}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6823","nonce":3360352974}]},"public_addr":"172.21.15.16:6812/3360352974","cluster_addr":"172.21.15.16:6817/3360352974","heartbeat_back_addr":"172.21.15.16:6828/3360352974","heartbeat_front_addr":"172.21.15.16:6823/3360352974","state":["exists","up"]},{"osd":4,"uuid":"88260730-1482-4fbd-a40f-cb1124e9df4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6801","nonce":1856844856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6803","nonce":1856844856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6807","nonce":1856844856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6805","nonce":1856844856}]},"public_addr":"172.21.15.89:6801/1856844856","cluster_addr":"172.21.15.89:6803/1856844856","heartbeat_back_addr":"172.21.15.89:6807/1856844856","heartbeat_front_addr":"172.21.15.89:6805/1856844856","state":["exists","up"]},{"osd":5,"uuid":"4cb2c00c-fae0-4dd8-9d27-ec4f0093ccf7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6814","nonce":50395527}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6817","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6820","nonce":50395527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6831","nonce":50395527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6823","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6826","nonce":50395527}]},"public_addr":"172.21.15.89:6814/50395527","cluster_addr":"172.21.15.89:6820/50395527","heartbeat_back_addr":"172.21.15.89:6831/50395527","heartbeat_front_addr":"172.21.15.89:6826/50395527","state":["exists","up"]},{"osd":6,"uuid":"81ce0c19-2554-4cae-acbe-66a05e9e9b9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6809","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6812","nonce":3470471752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6815","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6818","nonce":3470471752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6827","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6829","nonce":3470471752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6821","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6824","nonce":3470471752}]},"public_addr":"172.21.15.89:6812/3470471752","cluster_addr":"172.21.15.89:6818/3470471752","heartbeat_back_addr":"172.21.15.89:6829/3470471752","heartbeat_front_addr":"172.21.15.89:6824/3470471752","state":["exists","up"]},{"osd":7,"uuid":"85e4313b-d37d-442e-8042-153bf20a847d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6811","nonce":1796778630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6813","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6816","nonce":1796778630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6825","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6828","nonce":1796778630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6819","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6822","nonce":1796778630}]},"public_addr":"172.21.15.89:6811/1796778630","cluster_addr":"172.21.15.89:6816/1796778630","heartbeat_back_addr":"172.21.15.89:6828/1796778630","heartbeat_front_addr":"172.21.15.89:6822/1796778630","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.439088+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.529970+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.680512+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:21.044161+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.582111+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.536558+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.409522+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.590229+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:6835/2613117528":"2024-04-01T15:07:45.209490+0000","172.21.15.16:6834/2613117528":"2024-04-01T15:07:45.209490+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-31T15:07:49.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:49.830+0000 7fbd46200640 1 -- 172.21.15.16:0/2417251005 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fbd1c068380 msgr2=0x7fbd1c06a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:49.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:49.830+0000 7fbd46200640 1 --2- 172.21.15.16:0/2417251005 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fbd1c068380 0x7fbd1c06a840 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7fbd40143170 tx=0x7fbd30008040 comp rx=0 tx=0).stop 2024-03-31T15:07:49.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:49.830+0000 7fbd46200640 1 -- 172.21.15.16:0/2417251005 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbd40145d10 msgr2=0x7fbd401449d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:49.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:49.830+0000 7fbd46200640 1 --2- 172.21.15.16:0/2417251005 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbd40145d10 0x7fbd401449d0 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7fbd3801a660 tx=0x7fbd380197d0 comp rx=0 tx=0).stop 2024-03-31T15:07:49.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:49.830+0000 7fbd46200640 1 -- 172.21.15.16:0/2417251005 shutdown_connections 2024-03-31T15:07:49.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:49.830+0000 7fbd46200640 1 --2- 172.21.15.16:0/2417251005 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fbd1c068380 0x7fbd1c06a840 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:49.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:49.830+0000 7fbd46200640 1 --2- 172.21.15.16:0/2417251005 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbd40145d10 0x7fbd401449d0 unknown :-1 s=CLOSED pgs=783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:49.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:49.830+0000 7fbd46200640 1 --2- 172.21.15.16:0/2417251005 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbd400b23d0 0x7fbd40144490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:49.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:49.830+0000 7fbd46200640 1 --2- 172.21.15.16:0/2417251005 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fbd400728b0 0x7fbd4013bad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:49.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:49.830+0000 7fbd46200640 1 -- 172.21.15.16:0/2417251005 >> 172.21.15.16:0/2417251005 conn(0x7fbd4006b620 msgr2=0x7fbd4015eae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:07:49.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:49.830+0000 7fbd46200640 1 -- 172.21.15.16:0/2417251005 shutdown_connections 2024-03-31T15:07:49.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:49.830+0000 7fbd46200640 1 -- 172.21.15.16:0/2417251005 wait complete. 2024-03-31T15:07:49.846 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-31T15:03:22.550787+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-31T15:07:49.847 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-31T15:07:49.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:49.994+0000 7f77bc400640 1 Processor -- start 2024-03-31T15:07:49.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:49.994+0000 7f77bc400640 1 -- start start 2024-03-31T15:07:49.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:49.994+0000 7f77bc400640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f77b413e3d0 0x7f77b40cc9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:49.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:49.994+0000 7f77bc400640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f77b406dbe0 0x7f77b40ccf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:49.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:49.994+0000 7f77bc400640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f77b415fbc0 con 0x7f77b413dff0 2024-03-31T15:07:49.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:49.994+0000 7f77bc400640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f77b40cd450 con 0x7f77b413e3d0 2024-03-31T15:07:49.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:49.994+0000 7f77bc400640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f77b415f730 con 0x7f77b406dbe0 2024-03-31T15:07:49.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:49.994+0000 7f77b9000640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f77b413e3d0 0x7f77b40cc9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:49.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:49.998+0000 7f77b9000640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f77b413e3d0 0x7f77b40cc9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38602/0 (socket says 172.21.15.16:38602) 2024-03-31T15:07:49.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:49.998+0000 7f77b9000640 1 -- 172.21.15.16:0/2825686291 learned_addr learned my addr 172.21.15.16:0/2825686291 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:07:49.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:49.998+0000 7f77ba400640 1 --2- 172.21.15.16:0/2825686291 >> v2:172.21.15.89:3300/0 conn(0x7f77b406dbe0 0x7f77b40ccf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:49.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:49.998+0000 7f77b9000640 1 -- 172.21.15.16:0/2825686291 >> v2:172.21.15.89:3300/0 conn(0x7f77b406dbe0 msgr2=0x7f77b40ccf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:07:49.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:49.998+0000 7f77b9000640 1 --2- 172.21.15.16:0/2825686291 >> v2:172.21.15.89:3300/0 conn(0x7f77b406dbe0 0x7f77b40ccf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:49.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:49.998+0000 7f77b9000640 1 -- 172.21.15.16:0/2825686291 >> v1:172.21.15.89:6789/0 conn(0x7f77b413dff0 legacy=0x7f77b406d4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:49.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:49.998+0000 7f77b9000640 1 -- 172.21.15.16:0/2825686291 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f77b415ee00 con 0x7f77b413e3d0 2024-03-31T15:07:49.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:49.998+0000 7f77b9000640 1 --2- 172.21.15.16:0/2825686291 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f77b413e3d0 0x7f77b40cc9d0 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7f77b415b6e0 tx=0x7f77a001c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=461be2d009214ea3 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:07:49.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:49.998+0000 7f77b3e00640 1 -- 172.21.15.16:0/2825686291 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77a00b2070 con 0x7f77b413e3d0 2024-03-31T15:07:49.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:49.998+0000 7f77b3e00640 1 -- 172.21.15.16:0/2825686291 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f77a00ad680 con 0x7f77b413e3d0 2024-03-31T15:07:49.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:49.998+0000 7f77b3e00640 1 -- 172.21.15.16:0/2825686291 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77a00ada50 con 0x7f77b413e3d0 2024-03-31T15:07:49.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:50.002+0000 7f77bc400640 1 -- 172.21.15.16:0/2825686291 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f77b413e3d0 msgr2=0x7f77b40cc9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:49.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:50.002+0000 7f77bc400640 1 --2- 172.21.15.16:0/2825686291 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f77b413e3d0 0x7f77b40cc9d0 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7f77b415b6e0 tx=0x7f77a001c490 comp rx=0 tx=0).stop 2024-03-31T15:07:49.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:50.002+0000 7f77bc400640 1 -- 172.21.15.16:0/2825686291 shutdown_connections 2024-03-31T15:07:49.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:50.002+0000 7f77bc400640 1 --2- 172.21.15.16:0/2825686291 >> v2:172.21.15.89:3300/0 conn(0x7f77b406dbe0 0x7f77b40ccf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:49.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:50.002+0000 7f77bc400640 1 --2- 172.21.15.16:0/2825686291 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f77b413e3d0 0x7f77b40cc9d0 unknown :-1 s=CLOSED pgs=784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:49.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:50.002+0000 7f77bc400640 1 -- 172.21.15.16:0/2825686291 >> 172.21.15.16:0/2825686291 conn(0x7f77b406ee40 msgr2=0x7f77b406b620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:07:49.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:50.002+0000 7f77bc400640 1 -- 172.21.15.16:0/2825686291 shutdown_connections 2024-03-31T15:07:49.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:50.002+0000 7f77bc400640 1 -- 172.21.15.16:0/2825686291 wait complete. 2024-03-31T15:07:49.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:50.002+0000 7f77bc400640 1 Processor -- start 2024-03-31T15:07:49.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:50.002+0000 7f77bc400640 1 -- start start 2024-03-31T15:07:50.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:50.002+0000 7f77bc400640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f77b406dbe0 0x7f77b415a3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:50.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:50.002+0000 7f77bc400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f77b413dff0 0x7f77b41443e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:50.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:50.002+0000 7f77bc400640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f77b413e3d0 0x7f77b4140540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:50.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:50.002+0000 7f77bc400640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f77b40cf710 con 0x7f77b413dff0 2024-03-31T15:07:50.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:50.002+0000 7f77bc400640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f77b40cf590 con 0x7f77b413e3d0 2024-03-31T15:07:50.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:50.002+0000 7f77bc400640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f77b40cf890 con 0x7f77b406dbe0 2024-03-31T15:07:50.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:50.002+0000 7f77b9000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f77b413dff0 0x7f77b41443e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:50.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:50.002+0000 7f77b9000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f77b413dff0 0x7f77b41443e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38606/0 (socket says 172.21.15.16:38606) 2024-03-31T15:07:50.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:50.002+0000 7f77b9000640 1 -- 172.21.15.16:0/142437127 learned_addr learned my addr 172.21.15.16:0/142437127 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:07:50.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:50.002+0000 7f77ba400640 1 --2- 172.21.15.16:0/142437127 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f77b413e3d0 0x7f77b4140540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:50.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:50.002+0000 7f77b9a00640 1 --2- 172.21.15.16:0/142437127 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f77b406dbe0 0x7f77b415a3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:50.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:50.002+0000 7f77b9000640 1 -- 172.21.15.16:0/142437127 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f77b406dbe0 msgr2=0x7f77b415a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:50.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:50.002+0000 7f77b9000640 1 --2- 172.21.15.16:0/142437127 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f77b406dbe0 0x7f77b415a3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:50.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:50.002+0000 7f77b9000640 1 -- 172.21.15.16:0/142437127 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f77b413e3d0 msgr2=0x7f77b4140540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:50.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:50.002+0000 7f77b9000640 1 --2- 172.21.15.16:0/142437127 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f77b413e3d0 0x7f77b4140540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:50.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:50.002+0000 7f77b9000640 1 -- 172.21.15.16:0/142437127 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f77b4144920 con 0x7f77b413dff0 2024-03-31T15:07:50.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:50.002+0000 7f77ba400640 1 --2- 172.21.15.16:0/142437127 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f77b413e3d0 0x7f77b4140540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:07:50.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:50.006+0000 7f77b9000640 1 --2- 172.21.15.16:0/142437127 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f77b413dff0 0x7f77b41443e0 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f77a001a0c0 tx=0x7f77a001acd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:07:50.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:50.006+0000 7f77b1600640 1 -- 172.21.15.16:0/142437127 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77a00b2040 con 0x7f77b413dff0 2024-03-31T15:07:50.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:50.006+0000 7f77b1600640 1 -- 172.21.15.16:0/142437127 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f77a00ad9b0 con 0x7f77b413dff0 2024-03-31T15:07:50.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:50.006+0000 7f77b1600640 1 -- 172.21.15.16:0/142437127 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77a00add80 con 0x7f77b413dff0 2024-03-31T15:07:50.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:50.006+0000 7f77bc400640 1 -- 172.21.15.16:0/142437127 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f77b4159310 con 0x7f77b413dff0 2024-03-31T15:07:50.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:50.006+0000 7f77bc400640 1 -- 172.21.15.16:0/142437127 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f77b41598d0 con 0x7f77b413dff0 2024-03-31T15:07:50.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:50.006+0000 7f77bc400640 1 -- 172.21.15.16:0/142437127 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f77b4069950 con 0x7f77b413dff0 2024-03-31T15:07:50.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:50.006+0000 7f77b1600640 1 -- 172.21.15.16:0/142437127 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f77a00bf030 con 0x7f77b413dff0 2024-03-31T15:07:50.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:50.006+0000 7f77b1600640 1 --2- 172.21.15.16:0/142437127 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f7794068380 0x7f779406a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:50.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:50.006+0000 7f77b1600640 1 -- 172.21.15.16:0/142437127 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(59..59 src has 1..59) ==== 4943+0+0 (secure 0 0 0) 0x7f77a011fcc0 con 0x7f77b413dff0 2024-03-31T15:07:50.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:50.006+0000 7f77b9a00640 1 --2- 172.21.15.16:0/142437127 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f7794068380 0x7f779406a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:50.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:50.010+0000 7f77b9a00640 1 --2- 172.21.15.16:0/142437127 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f7794068380 0x7f779406a840 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f77a8001650 tx=0x7f77a8009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:07:50.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:50.010+0000 7f77b1600640 1 -- 172.21.15.16:0/142437127 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f77a0126080 con 0x7f77b413dff0 2024-03-31T15:07:50.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:50.222+0000 7f77bc400640 1 -- 172.21.15.16:0/142437127 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f77b40b0680 con 0x7f77b413dff0 2024-03-31T15:07:50.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:50.222+0000 7f77b1600640 1 -- 172.21.15.16:0/142437127 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v59) ==== 93+0+10 (secure 0 0 0) 0x7f77a0016110 con 0x7f77b413dff0 2024-03-31T15:07:50.217 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2024-03-31T15:07:50.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:50.222+0000 7f77bc400640 1 -- 172.21.15.16:0/142437127 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f7794068380 msgr2=0x7f779406a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:50.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:50.222+0000 7f77bc400640 1 --2- 172.21.15.16:0/142437127 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f7794068380 0x7f779406a840 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f77a8001650 tx=0x7f77a8009040 comp rx=0 tx=0).stop 2024-03-31T15:07:50.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:50.222+0000 7f77bc400640 1 -- 172.21.15.16:0/142437127 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f77b413dff0 msgr2=0x7f77b41443e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:50.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:50.222+0000 7f77bc400640 1 --2- 172.21.15.16:0/142437127 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f77b413dff0 0x7f77b41443e0 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f77a001a0c0 tx=0x7f77a001acd0 comp rx=0 tx=0).stop 2024-03-31T15:07:50.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:50.222+0000 7f77bc400640 1 -- 172.21.15.16:0/142437127 shutdown_connections 2024-03-31T15:07:50.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:50.222+0000 7f77bc400640 1 --2- 172.21.15.16:0/142437127 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f7794068380 0x7f779406a840 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:50.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:50.222+0000 7f77bc400640 1 --2- 172.21.15.16:0/142437127 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f77b413e3d0 0x7f77b4140540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:50.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:50.222+0000 7f77bc400640 1 --2- 172.21.15.16:0/142437127 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f77b413dff0 0x7f77b41443e0 unknown :-1 s=CLOSED pgs=785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:50.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:50.222+0000 7f77bc400640 1 --2- 172.21.15.16:0/142437127 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f77b406dbe0 0x7f77b415a3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:50.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:50.222+0000 7f77bc400640 1 -- 172.21.15.16:0/142437127 >> 172.21.15.16:0/142437127 conn(0x7f77b406ee40 msgr2=0x7f77b40b3fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:07:50.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:50.222+0000 7f77bc400640 1 -- 172.21.15.16:0/142437127 shutdown_connections 2024-03-31T15:07:50.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:50.222+0000 7f77bc400640 1 -- 172.21.15.16:0/142437127 wait complete. 2024-03-31T15:07:50.240 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_health_status_after_dentry_repair' 2024-03-31T15:07:50.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:50.386+0000 7f756f200640 1 Processor -- start 2024-03-31T15:07:50.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:50.386+0000 7f756f200640 1 -- start start 2024-03-31T15:07:50.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:50.386+0000 7f756f200640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f75700b4e10 0x7f75700d1230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:50.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:50.386+0000 7f756f200640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f75700d1d00 0x7f75700d2150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:50.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:50.386+0000 7f756f200640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f757015e8a0 con 0x7f757015e4c0 2024-03-31T15:07:50.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:50.386+0000 7f756f200640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f75700d0220 con 0x7f75700b4e10 2024-03-31T15:07:50.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:50.386+0000 7f756f200640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f75700b3770 con 0x7f75700d1d00 2024-03-31T15:07:50.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:50.386+0000 7f756de00640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f757015e4c0 0x7f75700d0b20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:49302/0 (socket says 172.21.15.16:49302) 2024-03-31T15:07:50.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:50.386+0000 7f756de00640 1 -- 172.21.15.16:0/1802621249 learned_addr learned my addr 172.21.15.16:0/1802621249 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:07:50.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:50.386+0000 7f7565400640 1 --2- 172.21.15.16:0/1802621249 >> v2:172.21.15.89:3300/0 conn(0x7f75700b4e10 0x7f75700d1230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:50.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:50.386+0000 7f756e800640 1 --2- 172.21.15.16:0/1802621249 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f75700d1d00 0x7f75700d2150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:50.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:50.386+0000 7f756d400640 1 -- 172.21.15.16:0/1802621249 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1118884230 0 0) 0x7f757015e8a0 con 0x7f757015e4c0 2024-03-31T15:07:50.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:50.386+0000 7f756d400640 1 -- 172.21.15.16:0/1802621249 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7554003530 con 0x7f757015e4c0 2024-03-31T15:07:50.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:50.386+0000 7f756e800640 1 -- 172.21.15.16:0/1802621249 >> v2:172.21.15.89:3300/0 conn(0x7f75700b4e10 msgr2=0x7f75700d1230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:07:50.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:50.386+0000 7f756e800640 1 --2- 172.21.15.16:0/1802621249 >> v2:172.21.15.89:3300/0 conn(0x7f75700b4e10 0x7f75700d1230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:50.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:50.386+0000 7f756e800640 1 -- 172.21.15.16:0/1802621249 >> v1:172.21.15.16:6789/0 conn(0x7f757015e4c0 legacy=0x7f75700d0b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:50.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:50.386+0000 7f756d400640 1 -- 172.21.15.16:0/1802621249 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2973011535 0 0) 0x7f7554003530 con 0x7f757015e4c0 2024-03-31T15:07:50.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:50.386+0000 7f756e800640 1 -- 172.21.15.16:0/1802621249 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75700cf980 con 0x7f75700d1d00 2024-03-31T15:07:50.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:50.390+0000 7f756e800640 1 --2- 172.21.15.16:0/1802621249 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f75700d1d00 0x7f75700d2150 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f7568009d90 tx=0x7f756801c860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ede7f693783e874e server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:07:50.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:50.390+0000 7f756d400640 1 -- 172.21.15.16:0/1802621249 <== mon.2 v2:172.21.15.89:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75680b2070 con 0x7f75700d1d00 2024-03-31T15:07:50.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:50.390+0000 7f756d400640 1 -- 172.21.15.16:0/1802621249 <== mon.2 v2:172.21.15.89:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f75680a5660 con 0x7f75700d1d00 2024-03-31T15:07:50.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:50.390+0000 7f756d400640 1 -- 172.21.15.16:0/1802621249 <== mon.2 v2:172.21.15.89:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7560001680 con 0x7f75700d1d00 2024-03-31T15:07:50.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:50.390+0000 7f756f200640 1 -- 172.21.15.16:0/1802621249 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f75700d1d00 msgr2=0x7f75700d2150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:50.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:50.390+0000 7f756f200640 1 --2- 172.21.15.16:0/1802621249 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f75700d1d00 0x7f75700d2150 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f7568009d90 tx=0x7f756801c860 comp rx=0 tx=0).stop 2024-03-31T15:07:50.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:50.390+0000 7f756f200640 1 -- 172.21.15.16:0/1802621249 shutdown_connections 2024-03-31T15:07:50.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:50.390+0000 7f756f200640 1 --2- 172.21.15.16:0/1802621249 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f75700d1d00 0x7f75700d2150 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:50.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:50.390+0000 7f756f200640 1 --2- 172.21.15.16:0/1802621249 >> v2:172.21.15.89:3300/0 conn(0x7f75700b4e10 0x7f75700d1230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:50.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:50.390+0000 7f756f200640 1 -- 172.21.15.16:0/1802621249 >> 172.21.15.16:0/1802621249 conn(0x7f757009d470 msgr2=0x7f75700d4130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:07:50.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:50.390+0000 7f756f200640 1 -- 172.21.15.16:0/1802621249 shutdown_connections 2024-03-31T15:07:50.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:50.390+0000 7f756f200640 1 -- 172.21.15.16:0/1802621249 wait complete. 2024-03-31T15:07:50.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:50.394+0000 7f756f200640 1 Processor -- start 2024-03-31T15:07:50.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:50.394+0000 7f756f200640 1 -- start start 2024-03-31T15:07:50.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:50.394+0000 7f756f200640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f75700b4e10 0x7f75701607d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:50.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:50.394+0000 7f756f200640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f75700d1d00 0x7f757015cec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:50.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:50.394+0000 7f756f200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f757015e4c0 0x7f757015d400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:50.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:50.394+0000 7f756f200640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f75700d2ea0 con 0x7f757015e4c0 2024-03-31T15:07:50.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:50.394+0000 7f756f200640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f75700d3020 con 0x7f75700b4e10 2024-03-31T15:07:50.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:50.394+0000 7f756f200640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f75700d31a0 con 0x7f75700d1d00 2024-03-31T15:07:50.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:50.394+0000 7f756e800640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f757015e4c0 0x7f757015d400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:50.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:50.394+0000 7f756e800640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f757015e4c0 0x7f757015d400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38614/0 (socket says 172.21.15.16:38614) 2024-03-31T15:07:50.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:50.394+0000 7f756e800640 1 -- 172.21.15.16:0/2817881638 learned_addr learned my addr 172.21.15.16:0/2817881638 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:07:50.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:50.394+0000 7f7565400640 1 --2- 172.21.15.16:0/2817881638 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f75700d1d00 0x7f757015cec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:50.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:50.394+0000 7f756de00640 1 --2- 172.21.15.16:0/2817881638 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f75700b4e10 0x7f75701607d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:50.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:50.394+0000 7f756e800640 1 -- 172.21.15.16:0/2817881638 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f75700d1d00 msgr2=0x7f757015cec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:50.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:50.394+0000 7f756e800640 1 --2- 172.21.15.16:0/2817881638 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f75700d1d00 0x7f757015cec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:50.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:50.394+0000 7f756e800640 1 -- 172.21.15.16:0/2817881638 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f75700b4e10 msgr2=0x7f75701607d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:50.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:50.394+0000 7f756e800640 1 --2- 172.21.15.16:0/2817881638 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f75700b4e10 0x7f75701607d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:50.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:50.394+0000 7f756e800640 1 -- 172.21.15.16:0/2817881638 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f757015ec80 con 0x7f757015e4c0 2024-03-31T15:07:50.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:50.394+0000 7f7565400640 1 --2- 172.21.15.16:0/2817881638 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f75700d1d00 0x7f757015cec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:07:50.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:50.394+0000 7f756e800640 1 --2- 172.21.15.16:0/2817881638 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f757015e4c0 0x7f757015d400 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7f75680b0040 tx=0x7f7568019600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:07:50.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:50.394+0000 7f7566a00640 1 -- 172.21.15.16:0/2817881638 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75680b2040 con 0x7f757015e4c0 2024-03-31T15:07:50.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:50.394+0000 7f7566a00640 1 -- 172.21.15.16:0/2817881638 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f75680bf070 con 0x7f757015e4c0 2024-03-31T15:07:50.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:50.394+0000 7f7566a00640 1 -- 172.21.15.16:0/2817881638 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7568009450 con 0x7f757015e4c0 2024-03-31T15:07:50.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:50.394+0000 7f756f200640 1 -- 172.21.15.16:0/2817881638 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f757015ef60 con 0x7f757015e4c0 2024-03-31T15:07:50.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:50.394+0000 7f756f200640 1 -- 172.21.15.16:0/2817881638 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f757015f520 con 0x7f757015e4c0 2024-03-31T15:07:50.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:50.398+0000 7f756f200640 1 -- 172.21.15.16:0/2817881638 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f757004f8c0 con 0x7f757015e4c0 2024-03-31T15:07:50.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:50.398+0000 7f7566a00640 1 -- 172.21.15.16:0/2817881638 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f75680b88b0 con 0x7f757015e4c0 2024-03-31T15:07:50.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:50.398+0000 7f7566a00640 1 --2- 172.21.15.16:0/2817881638 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f754c068380 0x7f754c06a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:50.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:50.398+0000 7f7566a00640 1 -- 172.21.15.16:0/2817881638 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(59..59 src has 1..59) ==== 4943+0+0 (secure 0 0 0) 0x7f7568121830 con 0x7f757015e4c0 2024-03-31T15:07:50.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:50.398+0000 7f756de00640 1 --2- 172.21.15.16:0/2817881638 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f754c068380 0x7f754c06a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:50.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:50.398+0000 7f756de00640 1 --2- 172.21.15.16:0/2817881638 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f754c068380 0x7f754c06a840 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f75600011f0 tx=0x7f75600016e0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:07:50.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:50.402+0000 7f7566a00640 1 -- 172.21.15.16:0/2817881638 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f7568016480 con 0x7f757015e4c0 2024-03-31T15:07:50.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:50.630+0000 7f756f200640 1 -- 172.21.15.16:0/2817881638 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_health_status_after_dentry_repair"]} v 0) -- 0x7f7570068ae0 con 0x7f757015e4c0 2024-03-31T15:07:51.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:51.490+0000 7f7566a00640 1 -- 172.21.15.16:0/2817881638 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_health_status_after_dentry_repair"]}]=0 v223) ==== 169+0+0 (secure 0 0 0) 0x7f75680f2170 con 0x7f757015e4c0 2024-03-31T15:07:51.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:51.494+0000 7f756f200640 1 -- 172.21.15.16:0/2817881638 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f754c068380 msgr2=0x7f754c06a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:51.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:51.494+0000 7f756f200640 1 --2- 172.21.15.16:0/2817881638 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f754c068380 0x7f754c06a840 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f75600011f0 tx=0x7f75600016e0 comp rx=0 tx=0).stop 2024-03-31T15:07:51.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:51.494+0000 7f756f200640 1 -- 172.21.15.16:0/2817881638 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f757015e4c0 msgr2=0x7f757015d400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:51.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:51.494+0000 7f756f200640 1 --2- 172.21.15.16:0/2817881638 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f757015e4c0 0x7f757015d400 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7f75680b0040 tx=0x7f7568019600 comp rx=0 tx=0).stop 2024-03-31T15:07:51.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:51.494+0000 7f756f200640 1 -- 172.21.15.16:0/2817881638 shutdown_connections 2024-03-31T15:07:51.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:51.494+0000 7f756f200640 1 --2- 172.21.15.16:0/2817881638 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f754c068380 0x7f754c06a840 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:51.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:51.494+0000 7f756f200640 1 --2- 172.21.15.16:0/2817881638 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f757015e4c0 0x7f757015d400 unknown :-1 s=CLOSED pgs=787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:51.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:51.494+0000 7f756f200640 1 --2- 172.21.15.16:0/2817881638 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f75700d1d00 0x7f757015cec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:51.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:51.494+0000 7f756f200640 1 --2- 172.21.15.16:0/2817881638 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f75700b4e10 0x7f75701607d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:51.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:51.494+0000 7f756f200640 1 -- 172.21.15.16:0/2817881638 >> 172.21.15.16:0/2817881638 conn(0x7f757009d470 msgr2=0x7f75700b1520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:07:51.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:51.494+0000 7f756f200640 1 -- 172.21.15.16:0/2817881638 shutdown_connections 2024-03-31T15:07:51.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:51.494+0000 7f756f200640 1 -- 172.21.15.16:0/2817881638 wait complete. 2024-03-31T15:07:51.514 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-03-31T15:07:51.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:51.670+0000 7f1ca7800640 1 Processor -- start 2024-03-31T15:07:51.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:51.670+0000 7f1ca7800640 1 -- start start 2024-03-31T15:07:51.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:51.670+0000 7f1ca7800640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f1ca01567e0 0x7f1ca0072c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:51.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:51.670+0000 7f1ca7800640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f1ca00731c0 0x7f1ca006d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:51.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:51.670+0000 7f1ca7800640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f1ca006da10 0x7f1ca015fdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:51.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:51.670+0000 7f1ca7800640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f1ca006de60 con 0x7f1ca01567e0 2024-03-31T15:07:51.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:51.670+0000 7f1ca7800640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f1ca015c7e0 con 0x7f1ca00731c0 2024-03-31T15:07:51.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:51.670+0000 7f1ca7800640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f1ca00736b0 con 0x7f1ca006da10 2024-03-31T15:07:51.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:51.670+0000 7f1ca4e00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f1ca01567e0 0x7f1ca0072c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:51.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:51.670+0000 7f1ca4e00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f1ca01567e0 0x7f1ca0072c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38628/0 (socket says 172.21.15.16:38628) 2024-03-31T15:07:51.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:51.670+0000 7f1ca4e00640 1 -- 172.21.15.16:0/73835673 learned_addr learned my addr 172.21.15.16:0/73835673 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:07:51.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:51.670+0000 7f1ca5800640 1 --2- 172.21.15.16:0/73835673 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f1ca006da10 0x7f1ca015fdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:51.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:51.670+0000 7f1c9fe00640 1 --2- 172.21.15.16:0/73835673 >> v2:172.21.15.89:3300/0 conn(0x7f1ca00731c0 0x7f1ca006d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:51.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:51.670+0000 7f1ca4e00640 1 -- 172.21.15.16:0/73835673 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f1ca006da10 msgr2=0x7f1ca015fdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:07:51.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:51.670+0000 7f1ca4e00640 1 --2- 172.21.15.16:0/73835673 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f1ca006da10 0x7f1ca015fdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:51.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:51.670+0000 7f1ca4e00640 1 -- 172.21.15.16:0/73835673 >> v2:172.21.15.89:3300/0 conn(0x7f1ca00731c0 msgr2=0x7f1ca006d4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:07:51.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:51.670+0000 7f1ca4e00640 1 --2- 172.21.15.16:0/73835673 >> v2:172.21.15.89:3300/0 conn(0x7f1ca00731c0 0x7f1ca006d4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:51.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:51.670+0000 7f1ca4e00640 1 -- 172.21.15.16:0/73835673 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ca015caa0 con 0x7f1ca01567e0 2024-03-31T15:07:51.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:51.670+0000 7f1ca5800640 1 --2- 172.21.15.16:0/73835673 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f1ca006da10 0x7f1ca015fdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:07:51.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:51.674+0000 7f1ca4e00640 1 --2- 172.21.15.16:0/73835673 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1ca01567e0 0x7f1ca0072c80 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7f1ca006c2e0 tx=0x7f1c9401c260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72b888b6974ea62c server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:07:51.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:51.674+0000 7f1c9f400640 1 -- 172.21.15.16:0/73835673 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c940b0070 con 0x7f1ca01567e0 2024-03-31T15:07:51.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:51.674+0000 7f1c9f400640 1 -- 172.21.15.16:0/73835673 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1c940ab600 con 0x7f1ca01567e0 2024-03-31T15:07:51.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:51.674+0000 7f1c9f400640 1 -- 172.21.15.16:0/73835673 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c940ab9d0 con 0x7f1ca01567e0 2024-03-31T15:07:51.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:51.674+0000 7f1ca7800640 1 -- 172.21.15.16:0/73835673 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1ca01567e0 msgr2=0x7f1ca0072c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:51.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:51.674+0000 7f1ca7800640 1 --2- 172.21.15.16:0/73835673 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1ca01567e0 0x7f1ca0072c80 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7f1ca006c2e0 tx=0x7f1c9401c260 comp rx=0 tx=0).stop 2024-03-31T15:07:51.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:51.674+0000 7f1ca7800640 1 -- 172.21.15.16:0/73835673 shutdown_connections 2024-03-31T15:07:51.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:51.674+0000 7f1ca7800640 1 --2- 172.21.15.16:0/73835673 >> v2:172.21.15.89:3300/0 conn(0x7f1ca00731c0 0x7f1ca006d4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:51.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:51.674+0000 7f1ca7800640 1 --2- 172.21.15.16:0/73835673 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f1ca006da10 0x7f1ca015fdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:51.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:51.674+0000 7f1ca7800640 1 --2- 172.21.15.16:0/73835673 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1ca01567e0 0x7f1ca0072c80 unknown :-1 s=CLOSED pgs=788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:51.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:51.674+0000 7f1ca7800640 1 -- 172.21.15.16:0/73835673 >> 172.21.15.16:0/73835673 conn(0x7f1ca006ee40 msgr2=0x7f1ca006b620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:07:51.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:51.674+0000 7f1ca7800640 1 -- 172.21.15.16:0/73835673 shutdown_connections 2024-03-31T15:07:51.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:51.674+0000 7f1ca7800640 1 -- 172.21.15.16:0/73835673 wait complete. 2024-03-31T15:07:51.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:51.678+0000 7f1ca7800640 1 Processor -- start 2024-03-31T15:07:51.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:51.678+0000 7f1ca7800640 1 -- start start 2024-03-31T15:07:51.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:51.678+0000 7f1ca7800640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1ca006da10 0x7f1ca0150480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:51.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:51.678+0000 7f1ca7800640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f1ca00731c0 0x7f1ca0144320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:51.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:51.678+0000 7f1ca7800640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1ca01567e0 0x7f1ca0156ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:51.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:51.678+0000 7f1ca7800640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f1ca015d700 con 0x7f1ca006da10 2024-03-31T15:07:51.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:51.678+0000 7f1ca7800640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f1ca015d580 con 0x7f1ca01567e0 2024-03-31T15:07:51.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:51.678+0000 7f1ca7800640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f1ca0071550 con 0x7f1ca00731c0 2024-03-31T15:07:51.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:51.678+0000 7f1ca4e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1ca006da10 0x7f1ca0150480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:51.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:51.678+0000 7f1ca4e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1ca006da10 0x7f1ca0150480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38642/0 (socket says 172.21.15.16:38642) 2024-03-31T15:07:51.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:51.678+0000 7f1ca4e00640 1 -- 172.21.15.16:0/1425462988 learned_addr learned my addr 172.21.15.16:0/1425462988 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:07:51.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:51.678+0000 7f1c9fe00640 1 --2- 172.21.15.16:0/1425462988 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f1ca00731c0 0x7f1ca0144320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:51.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:51.678+0000 7f1ca5800640 1 --2- 172.21.15.16:0/1425462988 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1ca01567e0 0x7f1ca0156ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:51.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:51.678+0000 7f1ca4e00640 1 -- 172.21.15.16:0/1425462988 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f1ca00731c0 msgr2=0x7f1ca0144320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:51.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:51.678+0000 7f1ca4e00640 1 --2- 172.21.15.16:0/1425462988 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f1ca00731c0 0x7f1ca0144320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:51.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:51.678+0000 7f1ca4e00640 1 -- 172.21.15.16:0/1425462988 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1ca01567e0 msgr2=0x7f1ca0156ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:51.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:51.678+0000 7f1ca4e00640 1 --2- 172.21.15.16:0/1425462988 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1ca01567e0 0x7f1ca0156ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:51.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:51.678+0000 7f1ca4e00640 1 -- 172.21.15.16:0/1425462988 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ca01458e0 con 0x7f1ca006da10 2024-03-31T15:07:51.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:51.678+0000 7f1ca4e00640 1 --2- 172.21.15.16:0/1425462988 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1ca006da10 0x7f1ca0150480 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7f1c940ae040 tx=0x7f1c94019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:07:51.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:51.678+0000 7f1c9cc00640 1 -- 172.21.15.16:0/1425462988 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c940b0040 con 0x7f1ca006da10 2024-03-31T15:07:51.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:51.678+0000 7f1c9cc00640 1 -- 172.21.15.16:0/1425462988 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1c940b9070 con 0x7f1ca006da10 2024-03-31T15:07:51.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:51.678+0000 7f1c9cc00640 1 -- 172.21.15.16:0/1425462988 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c940b46a0 con 0x7f1ca006da10 2024-03-31T15:07:51.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:51.678+0000 7f1ca7800640 1 -- 172.21.15.16:0/1425462988 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1ca0145bc0 con 0x7f1ca006da10 2024-03-31T15:07:51.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:51.678+0000 7f1ca7800640 1 -- 172.21.15.16:0/1425462988 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1ca0152410 con 0x7f1ca006da10 2024-03-31T15:07:51.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:51.678+0000 7f1ca7800640 1 -- 172.21.15.16:0/1425462988 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1ca0069950 con 0x7f1ca006da10 2024-03-31T15:07:51.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:51.682+0000 7f1c9cc00640 1 -- 172.21.15.16:0/1425462988 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f1c940b4840 con 0x7f1ca006da10 2024-03-31T15:07:51.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:51.682+0000 7f1c9cc00640 1 --2- 172.21.15.16:0/1425462988 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f1c800683b0 0x7f1c8006a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:51.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:51.682+0000 7f1c9cc00640 1 -- 172.21.15.16:0/1425462988 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(59..59 src has 1..59) ==== 4943+0+0 (secure 0 0 0) 0x7f1c941201f0 con 0x7f1ca006da10 2024-03-31T15:07:51.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:51.682+0000 7f1c9fe00640 1 --2- 172.21.15.16:0/1425462988 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f1c800683b0 0x7f1c8006a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:51.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:51.682+0000 7f1c9fe00640 1 --2- 172.21.15.16:0/1425462988 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f1c800683b0 0x7f1c8006a870 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f1ca01f8aa0 tx=0x7f1c8c008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:07:51.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:51.686+0000 7f1c9cc00640 1 -- 172.21.15.16:0/1425462988 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f1c94016110 con 0x7f1ca006da10 2024-03-31T15:07:51.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:51.850+0000 7f1ca7800640 1 -- 172.21.15.16:0/1425462988 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7f1ca00af520 con 0x7f1ca006da10 2024-03-31T15:07:51.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:51.854+0000 7f1c9cc00640 1 -- 172.21.15.16:0/1425462988 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v16)=0 v16) ==== 122+0+0 (secure 0 0 0) 0x7f1c940f0450 con 0x7f1ca006da10 2024-03-31T15:07:51.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:51.858+0000 7f1ca7800640 1 -- 172.21.15.16:0/1425462988 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f1c800683b0 msgr2=0x7f1c8006a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:51.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:51.858+0000 7f1ca7800640 1 --2- 172.21.15.16:0/1425462988 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f1c800683b0 0x7f1c8006a870 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f1ca01f8aa0 tx=0x7f1c8c008040 comp rx=0 tx=0).stop 2024-03-31T15:07:51.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:51.858+0000 7f1ca7800640 1 -- 172.21.15.16:0/1425462988 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1ca006da10 msgr2=0x7f1ca0150480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:51.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:51.858+0000 7f1ca7800640 1 --2- 172.21.15.16:0/1425462988 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1ca006da10 0x7f1ca0150480 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7f1c940ae040 tx=0x7f1c94019000 comp rx=0 tx=0).stop 2024-03-31T15:07:51.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:51.858+0000 7f1ca7800640 1 -- 172.21.15.16:0/1425462988 shutdown_connections 2024-03-31T15:07:51.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:51.858+0000 7f1ca7800640 1 --2- 172.21.15.16:0/1425462988 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f1c800683b0 0x7f1c8006a870 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:51.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:51.858+0000 7f1ca7800640 1 --2- 172.21.15.16:0/1425462988 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1ca01567e0 0x7f1ca0156ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:51.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:51.858+0000 7f1ca7800640 1 --2- 172.21.15.16:0/1425462988 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f1ca00731c0 0x7f1ca0144320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:51.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:51.858+0000 7f1ca7800640 1 --2- 172.21.15.16:0/1425462988 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1ca006da10 0x7f1ca0150480 unknown :-1 s=CLOSED pgs=789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:51.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:51.858+0000 7f1ca7800640 1 -- 172.21.15.16:0/1425462988 >> 172.21.15.16:0/1425462988 conn(0x7f1ca006ee40 msgr2=0x7f1ca00b1e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:07:51.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:51.858+0000 7f1ca7800640 1 -- 172.21.15.16:0/1425462988 shutdown_connections 2024-03-31T15:07:51.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:51.858+0000 7f1ca7800640 1 -- 172.21.15.16:0/1425462988 wait complete. 2024-03-31T15:07:51.871 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-31T15:07:51.871 DEBUG:teuthology.orchestra.run.smithi016:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-31T15:07:51.914 DEBUG:teuthology.orchestra.run.smithi089:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-31T15:07:51.966 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-31T15:07:51.966 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-03-31T15:07:51.966 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-31T15:07:51.975 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16357708k,nr_inodes=4089427,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275996k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=6539 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275992k,nr_inodes=818998,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275996k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-31T15:07:51.975 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-03-31T15:07:51.975 DEBUG:teuthology.orchestra.run.smithi089:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-31T15:07:51.984 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16354452k,nr_inodes=4088613,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275344k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=901 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275340k,nr_inodes=818835,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275344k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-31T15:07:51.984 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-03-31T15:07:51.985 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-31T15:07:52.027 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16357708k,nr_inodes=4089427,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275996k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=6539 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275992k,nr_inodes=818998,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275996k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-31T15:07:52.027 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-03-31T15:07:52.028 DEBUG:teuthology.orchestra.run.smithi089:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-31T15:07:52.038 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16354452k,nr_inodes=4088613,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275344k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=901 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275340k,nr_inodes=818835,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275344k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-31T15:07:52.038 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-03-31T15:07:52.039 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-31T15:07:52.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:52.214+0000 7f6588800640 1 Processor -- start 2024-03-31T15:07:52.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:52.214+0000 7f6588800640 1 -- start start 2024-03-31T15:07:52.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:52.214+0000 7f6588800640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f6580145d10 0x7f65800cf790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:52.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:52.214+0000 7f6588800640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f65800b23d0 0x7f658015cd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:52.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:52.214+0000 7f6588800640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6580160e70 con 0x7f65800728b0 2024-03-31T15:07:52.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:52.214+0000 7f6588800640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f65801602f0 con 0x7f6580145d10 2024-03-31T15:07:52.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:52.214+0000 7f6588800640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f65801605d0 con 0x7f65800b23d0 2024-03-31T15:07:52.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:52.214+0000 7f6585e00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f6580145d10 0x7f65800cf790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:52.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:52.214+0000 7f6585e00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f6580145d10 0x7f65800cf790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38650/0 (socket says 172.21.15.16:38650) 2024-03-31T15:07:52.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:52.214+0000 7f6585e00640 1 -- 172.21.15.16:0/51157280 learned_addr learned my addr 172.21.15.16:0/51157280 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:07:52.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:52.214+0000 7f6586800640 1 --2- 172.21.15.16:0/51157280 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f65800b23d0 0x7f658015cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:52.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:52.214+0000 7f6585e00640 1 -- 172.21.15.16:0/51157280 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f65800b23d0 msgr2=0x7f658015cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:07:52.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:52.214+0000 7f6585e00640 1 --2- 172.21.15.16:0/51157280 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f65800b23d0 0x7f658015cd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:52.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:52.214+0000 7f6585e00640 1 -- 172.21.15.16:0/51157280 >> v1:172.21.15.16:6789/0 conn(0x7f65800728b0 legacy=0x7f65800b1cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:52.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:52.218+0000 7f6585e00640 1 -- 172.21.15.16:0/51157280 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65800cfcd0 con 0x7f6580145d10 2024-03-31T15:07:52.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:52.218+0000 7f6586800640 1 --2- 172.21.15.16:0/51157280 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f65800b23d0 0x7f658015cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:07:52.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:52.218+0000 7f6585e00640 1 --2- 172.21.15.16:0/51157280 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6580145d10 0x7f65800cf790 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f6574009870 tx=0x7f657401c560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ac9adce123e5d54 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:07:52.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:52.218+0000 7f6584a00640 1 -- 172.21.15.16:0/51157280 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65740b2070 con 0x7f6580145d10 2024-03-31T15:07:52.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:52.218+0000 7f6584a00640 1 -- 172.21.15.16:0/51157280 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f65740a9070 con 0x7f6580145d10 2024-03-31T15:07:52.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:52.218+0000 7f6584a00640 1 -- 172.21.15.16:0/51157280 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65740ad880 con 0x7f6580145d10 2024-03-31T15:07:52.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:52.218+0000 7f6588800640 1 -- 172.21.15.16:0/51157280 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6580145d10 msgr2=0x7f65800cf790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:52.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:52.218+0000 7f6588800640 1 --2- 172.21.15.16:0/51157280 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6580145d10 0x7f65800cf790 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f6574009870 tx=0x7f657401c560 comp rx=0 tx=0).stop 2024-03-31T15:07:52.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:52.218+0000 7f6588800640 1 -- 172.21.15.16:0/51157280 shutdown_connections 2024-03-31T15:07:52.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:52.218+0000 7f6588800640 1 --2- 172.21.15.16:0/51157280 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f65800b23d0 0x7f658015cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:52.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:52.218+0000 7f6588800640 1 --2- 172.21.15.16:0/51157280 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6580145d10 0x7f65800cf790 unknown :-1 s=CLOSED pgs=791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:52.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:52.218+0000 7f6588800640 1 -- 172.21.15.16:0/51157280 >> 172.21.15.16:0/51157280 conn(0x7f658006b620 msgr2=0x7f658006ba30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:07:52.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:52.218+0000 7f6588800640 1 -- 172.21.15.16:0/51157280 shutdown_connections 2024-03-31T15:07:52.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:52.218+0000 7f6588800640 1 -- 172.21.15.16:0/51157280 wait complete. 2024-03-31T15:07:52.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:52.222+0000 7f6588800640 1 Processor -- start 2024-03-31T15:07:52.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:52.222+0000 7f6588800640 1 -- start start 2024-03-31T15:07:52.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:52.222+0000 7f6588800640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f65800728b0 0x7f6580161090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:52.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:52.222+0000 7f6588800640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f65800b23d0 0x7f6580159170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:52.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:52.222+0000 7f6588800640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6580145d10 0x7f6580162710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:52.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:52.222+0000 7f6588800640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f65800cf4d0 con 0x7f65800b23d0 2024-03-31T15:07:52.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:52.222+0000 7f6588800640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f65800cf350 con 0x7f6580145d10 2024-03-31T15:07:52.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:52.222+0000 7f6588800640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f6580072cc0 con 0x7f65800728b0 2024-03-31T15:07:52.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:52.222+0000 7f6585400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f65800b23d0 0x7f6580159170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:52.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:52.222+0000 7f6585400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f65800b23d0 0x7f6580159170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38664/0 (socket says 172.21.15.16:38664) 2024-03-31T15:07:52.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:52.222+0000 7f6585400640 1 -- 172.21.15.16:0/2607372311 learned_addr learned my addr 172.21.15.16:0/2607372311 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:07:52.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:52.222+0000 7f6585e00640 1 --2- 172.21.15.16:0/2607372311 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f65800728b0 0x7f6580161090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:52.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:52.222+0000 7f6586800640 1 --2- 172.21.15.16:0/2607372311 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6580145d10 0x7f6580162710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:52.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:52.222+0000 7f6585400640 1 -- 172.21.15.16:0/2607372311 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f65800728b0 msgr2=0x7f6580161090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:52.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:52.222+0000 7f6585400640 1 --2- 172.21.15.16:0/2607372311 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f65800728b0 0x7f6580161090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:52.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:52.222+0000 7f6585400640 1 -- 172.21.15.16:0/2607372311 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6580145d10 msgr2=0x7f6580162710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:52.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:52.222+0000 7f6585400640 1 --2- 172.21.15.16:0/2607372311 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6580145d10 0x7f6580162710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:52.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:52.222+0000 7f6585400640 1 -- 172.21.15.16:0/2607372311 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6580161670 con 0x7f65800b23d0 2024-03-31T15:07:52.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:52.222+0000 7f6586800640 1 --2- 172.21.15.16:0/2607372311 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6580145d10 0x7f6580162710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:07:52.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:52.222+0000 7f6585400640 1 --2- 172.21.15.16:0/2607372311 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f65800b23d0 0x7f6580159170 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f657000c370 tx=0x7f657000c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:07:52.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:52.222+0000 7f657a000640 1 -- 172.21.15.16:0/2607372311 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65700052d0 con 0x7f65800b23d0 2024-03-31T15:07:52.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:52.222+0000 7f657a000640 1 -- 172.21.15.16:0/2607372311 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6570005470 con 0x7f65800b23d0 2024-03-31T15:07:52.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:52.222+0000 7f657a000640 1 -- 172.21.15.16:0/2607372311 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6570007260 con 0x7f65800b23d0 2024-03-31T15:07:52.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:52.222+0000 7f6588800640 1 -- 172.21.15.16:0/2607372311 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6580161950 con 0x7f65800b23d0 2024-03-31T15:07:52.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:52.222+0000 7f6588800640 1 -- 172.21.15.16:0/2607372311 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6580161f10 con 0x7f65800b23d0 2024-03-31T15:07:52.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:52.226+0000 7f6588800640 1 -- 172.21.15.16:0/2607372311 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6580069950 con 0x7f65800b23d0 2024-03-31T15:07:52.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:52.226+0000 7f657a000640 1 -- 172.21.15.16:0/2607372311 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f6570007400 con 0x7f65800b23d0 2024-03-31T15:07:52.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:52.226+0000 7f657a000640 1 --2- 172.21.15.16:0/2607372311 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f6564068380 0x7f656406a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:52.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:52.226+0000 7f657a000640 1 -- 172.21.15.16:0/2607372311 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(59..59 src has 1..59) ==== 4943+0+0 (secure 0 0 0) 0x7f6570085ac0 con 0x7f65800b23d0 2024-03-31T15:07:52.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:52.226+0000 7f6585e00640 1 --2- 172.21.15.16:0/2607372311 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f6564068380 0x7f656406a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:52.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:52.226+0000 7f6585e00640 1 --2- 172.21.15.16:0/2607372311 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f6564068380 0x7f656406a840 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f65740af000 tx=0x7f6574008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:07:52.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:52.230+0000 7f657a000640 1 -- 172.21.15.16:0/2607372311 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f6570055d20 con 0x7f65800b23d0 2024-03-31T15:07:52.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:52.430+0000 7f6588800640 1 -- 172.21.15.16:0/2607372311 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f65800cdb50 con 0x7f65800b23d0 2024-03-31T15:07:52.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:52.430+0000 7f657a000640 1 -- 172.21.15.16:0/2607372311 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 38 v38) ==== 94+0+3869 (secure 0 0 0) 0x7f657005c030 con 0x7f65800b23d0 2024-03-31T15:07:52.426 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:07:52.426 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":38,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5579,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6833/3721971977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":3721971977},{"type":"v1","addr":"172.21.15.16:6833","nonce":3721971977}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":29},{"gid":6109,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6836/1166206553","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":1166206553},{"type":"v1","addr":"172.21.15.89:6836","nonce":1166206553}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28},{"gid":6112,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/1826113083","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6835","nonce":1826113083},{"type":"v1","addr":"172.21.15.89:6837","nonce":1826113083}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":29},{"gid":6630,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/1904937871","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":1904937871},{"type":"v1","addr":"172.21.15.16:6835","nonce":1904937871}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":38}],"filesystems":[]} 2024-03-31T15:07:52.426 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 38 2024-03-31T15:07:52.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:52.434+0000 7f6588800640 1 -- 172.21.15.16:0/2607372311 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f6564068380 msgr2=0x7f656406a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:52.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:52.434+0000 7f6588800640 1 --2- 172.21.15.16:0/2607372311 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f6564068380 0x7f656406a840 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f65740af000 tx=0x7f6574008040 comp rx=0 tx=0).stop 2024-03-31T15:07:52.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:52.434+0000 7f6588800640 1 -- 172.21.15.16:0/2607372311 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f65800b23d0 msgr2=0x7f6580159170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:52.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:52.434+0000 7f6588800640 1 --2- 172.21.15.16:0/2607372311 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f65800b23d0 0x7f6580159170 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f657000c370 tx=0x7f657000c830 comp rx=0 tx=0).stop 2024-03-31T15:07:52.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:52.434+0000 7f6588800640 1 -- 172.21.15.16:0/2607372311 shutdown_connections 2024-03-31T15:07:52.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:52.434+0000 7f6588800640 1 --2- 172.21.15.16:0/2607372311 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f6564068380 0x7f656406a840 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:52.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:52.434+0000 7f6588800640 1 --2- 172.21.15.16:0/2607372311 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6580145d10 0x7f6580162710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:52.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:52.434+0000 7f6588800640 1 --2- 172.21.15.16:0/2607372311 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f65800b23d0 0x7f6580159170 unknown :-1 s=CLOSED pgs=792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:52.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:52.434+0000 7f6588800640 1 --2- 172.21.15.16:0/2607372311 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f65800728b0 0x7f6580161090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:52.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:52.434+0000 7f6588800640 1 -- 172.21.15.16:0/2607372311 >> 172.21.15.16:0/2607372311 conn(0x7f658006b620 msgr2=0x7f65800afba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:07:52.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:52.434+0000 7f6588800640 1 -- 172.21.15.16:0/2607372311 shutdown_connections 2024-03-31T15:07:52.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:52.434+0000 7f6588800640 1 -- 172.21.15.16:0/2607372311 wait complete. 2024-03-31T15:07:52.448 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-31T15:07:52.448 INFO:tasks.ceph.mds.a:Stopping old one... 2024-03-31T15:07:52.448 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-31T15:07:52.448 INFO:teuthology.orchestra.run:waiting for 300 2024-03-31T15:07:52.449 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-31T15:07:52.449 INFO:tasks.ceph.mds.c:Stopping old one... 2024-03-31T15:07:52.449 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-31T15:07:52.449 INFO:teuthology.orchestra.run:waiting for 300 2024-03-31T15:07:52.449 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-31T15:07:52.449 INFO:tasks.ceph.mds.b:Stopping old one... 2024-03-31T15:07:52.449 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-31T15:07:52.449 INFO:teuthology.orchestra.run:waiting for 300 2024-03-31T15:07:52.450 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-31T15:07:52.450 INFO:tasks.ceph.mds.d:Stopping old one... 2024-03-31T15:07:52.450 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-31T15:07:52.450 INFO:teuthology.orchestra.run:waiting for 300 2024-03-31T15:07:52.460 INFO:tasks.ceph.mds.a:Stopped 2024-03-31T15:07:52.461 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-31T15:07:52.462 INFO:tasks.ceph.mds.c:Stopped 2024-03-31T15:07:52.462 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-31T15:07:52.465 INFO:tasks.ceph.mds.b:Stopped 2024-03-31T15:07:52.465 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-31T15:07:52.466 INFO:tasks.ceph.mds.d:Stopped 2024-03-31T15:07:52.466 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-31T15:07:52.505 INFO:tasks.ceph.mds.a:Started 2024-03-31T15:07:52.506 DEBUG:teuthology.parallel:result is None 2024-03-31T15:07:52.507 INFO:tasks.ceph.mds.c:Started 2024-03-31T15:07:52.508 DEBUG:teuthology.parallel:result is None 2024-03-31T15:07:52.513 INFO:tasks.ceph.mds.b:Started 2024-03-31T15:07:52.513 DEBUG:teuthology.parallel:result is None 2024-03-31T15:07:52.515 INFO:tasks.ceph.mds.d:Started 2024-03-31T15:07:52.515 DEBUG:teuthology.parallel:result is None 2024-03-31T15:07:52.515 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-03-31T15:07:52.591 INFO:tasks.ceph.mds.c.smithi016.stdout:starting mds.c at 2024-03-31T15:07:52.593 INFO:tasks.ceph.mds.a.smithi016.stdout:starting mds.a at 2024-03-31T15:07:52.603 INFO:tasks.ceph.mds.b.smithi089.stdout:starting mds.b at 2024-03-31T15:07:52.604 INFO:tasks.ceph.mds.d.smithi089.stdout:starting mds.d at 2024-03-31T15:07:52.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:52.666+0000 7fcbd6600640 1 Processor -- start 2024-03-31T15:07:52.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:52.666+0000 7fcbd6600640 1 -- start start 2024-03-31T15:07:52.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:52.670+0000 7fcbd6600640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fcbd0145e40 0x7fcbd00cc740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:52.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:52.670+0000 7fcbd6600640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fcbd0070030 0x7fcbd00d2010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:52.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:52.670+0000 7fcbd6600640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcbd00d0920 con 0x7fcbd00ccc80 2024-03-31T15:07:52.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:52.670+0000 7fcbd6600640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7fcbd00d06d0 con 0x7fcbd0070030 2024-03-31T15:07:52.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:52.670+0000 7fcbd6600640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7fcbd00d0c00 con 0x7fcbd0145e40 2024-03-31T15:07:52.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:52.670+0000 7fcbcfe00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fcbd0070030 0x7fcbd00d2010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:52.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:52.670+0000 7fcbc7e00640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fcbd00ccc80 0x7fcbd006f920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:49330/0 (socket says 172.21.15.16:49330) 2024-03-31T15:07:52.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:52.670+0000 7fcbc7e00640 1 -- 172.21.15.16:0/59830636 learned_addr learned my addr 172.21.15.16:0/59830636 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:07:52.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:52.670+0000 7fcbcf400640 1 --2- 172.21.15.16:0/59830636 >> v2:172.21.15.89:3300/0 conn(0x7fcbd0145e40 0x7fcbd00cc740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:52.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:52.670+0000 7fcbcea00640 1 -- 172.21.15.16:0/59830636 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2642405796 0 0) 0x7fcbd00d0920 con 0x7fcbd00ccc80 2024-03-31T15:07:52.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:52.670+0000 7fcbcfe00640 1 -- 172.21.15.16:0/59830636 >> v2:172.21.15.89:3300/0 conn(0x7fcbd0145e40 msgr2=0x7fcbd00cc740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:07:52.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:52.670+0000 7fcbcfe00640 1 --2- 172.21.15.16:0/59830636 >> v2:172.21.15.89:3300/0 conn(0x7fcbd0145e40 0x7fcbd00cc740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:52.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:52.670+0000 7fcbcfe00640 1 -- 172.21.15.16:0/59830636 >> v1:172.21.15.16:6789/0 conn(0x7fcbd00ccc80 legacy=0x7fcbd006f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:52.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:52.670+0000 7fcbcfe00640 1 -- 172.21.15.16:0/59830636 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcbd00d1810 con 0x7fcbd0070030 2024-03-31T15:07:52.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:52.674+0000 7fcbcfe00640 1 --2- 172.21.15.16:0/59830636 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcbd0070030 0x7fcbd00d2010 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7fcbd0072e60 tx=0x7fcbc801c930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f243bedfb24555be server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:07:52.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:52.674+0000 7fcbcea00640 1 -- 172.21.15.16:0/59830636 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbc80b2070 con 0x7fcbd0070030 2024-03-31T15:07:52.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:52.674+0000 7fcbcea00640 1 -- 172.21.15.16:0/59830636 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcbc80a5900 con 0x7fcbd0070030 2024-03-31T15:07:52.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:52.674+0000 7fcbcea00640 1 -- 172.21.15.16:0/59830636 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbc80a5cd0 con 0x7fcbd0070030 2024-03-31T15:07:52.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:52.674+0000 7fcbd6600640 1 -- 172.21.15.16:0/59830636 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcbd0070030 msgr2=0x7fcbd00d2010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:52.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:52.674+0000 7fcbd6600640 1 --2- 172.21.15.16:0/59830636 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcbd0070030 0x7fcbd00d2010 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7fcbd0072e60 tx=0x7fcbc801c930 comp rx=0 tx=0).stop 2024-03-31T15:07:52.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:52.674+0000 7fcbd6600640 1 -- 172.21.15.16:0/59830636 shutdown_connections 2024-03-31T15:07:52.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:52.674+0000 7fcbd6600640 1 --2- 172.21.15.16:0/59830636 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcbd0070030 0x7fcbd00d2010 unknown :-1 s=CLOSED pgs=800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:52.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:52.674+0000 7fcbd6600640 1 --2- 172.21.15.16:0/59830636 >> v2:172.21.15.89:3300/0 conn(0x7fcbd0145e40 0x7fcbd00cc740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:52.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:52.674+0000 7fcbd6600640 1 -- 172.21.15.16:0/59830636 >> 172.21.15.16:0/59830636 conn(0x7fcbd0067330 msgr2=0x7fcbd0067740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:07:52.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:52.674+0000 7fcbd6600640 1 -- 172.21.15.16:0/59830636 shutdown_connections 2024-03-31T15:07:52.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:52.674+0000 7fcbd6600640 1 -- 172.21.15.16:0/59830636 wait complete. 2024-03-31T15:07:52.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:52.674+0000 7fcbd6600640 1 Processor -- start 2024-03-31T15:07:52.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:52.674+0000 7fcbd6600640 1 -- start start 2024-03-31T15:07:52.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:52.678+0000 7fcbd6600640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcbd0070030 0x7fcbd0142470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:52.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:52.678+0000 7fcbd6600640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcbd00ccc80 0x7fcbd00c7820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:52.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:52.678+0000 7fcbd6600640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fcbd0145e40 0x7fcbd00c7d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:52.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:52.678+0000 7fcbd6600640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fcbd00b5240 con 0x7fcbd00ccc80 2024-03-31T15:07:52.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:52.678+0000 7fcbd6600640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fcbd00b50c0 con 0x7fcbd0070030 2024-03-31T15:07:52.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:52.678+0000 7fcbd6600640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fcbd0169190 con 0x7fcbd0145e40 2024-03-31T15:07:52.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:52.678+0000 7fcbc7e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcbd00ccc80 0x7fcbd00c7820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:52.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:52.678+0000 7fcbc7e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcbd00ccc80 0x7fcbd00c7820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38726/0 (socket says 172.21.15.16:38726) 2024-03-31T15:07:52.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:52.678+0000 7fcbc7e00640 1 -- 172.21.15.16:0/3035337663 learned_addr learned my addr 172.21.15.16:0/3035337663 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:07:52.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:52.678+0000 7fcbcf400640 1 --2- 172.21.15.16:0/3035337663 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcbd0070030 0x7fcbd0142470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:52.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:52.678+0000 7fcbcfe00640 1 --2- 172.21.15.16:0/3035337663 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fcbd0145e40 0x7fcbd00c7d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:52.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:52.678+0000 7fcbc7e00640 1 -- 172.21.15.16:0/3035337663 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fcbd0145e40 msgr2=0x7fcbd00c7d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:52.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:52.678+0000 7fcbc7e00640 1 --2- 172.21.15.16:0/3035337663 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fcbd0145e40 0x7fcbd00c7d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:52.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:52.678+0000 7fcbc7e00640 1 -- 172.21.15.16:0/3035337663 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcbd0070030 msgr2=0x7fcbd0142470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:52.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:52.678+0000 7fcbc7e00640 1 --2- 172.21.15.16:0/3035337663 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcbd0070030 0x7fcbd0142470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:52.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:52.678+0000 7fcbc7e00640 1 -- 172.21.15.16:0/3035337663 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcbd006df00 con 0x7fcbd00ccc80 2024-03-31T15:07:52.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:52.678+0000 7fcbcf400640 1 --2- 172.21.15.16:0/3035337663 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcbd0070030 0x7fcbd0142470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:07:52.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:52.678+0000 7fcbc7e00640 1 --2- 172.21.15.16:0/3035337663 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcbd00ccc80 0x7fcbd00c7820 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7fcbb80011f0 tx=0x7fcbb800c9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:07:52.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:52.678+0000 7fcbc7400640 1 -- 172.21.15.16:0/3035337663 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbb8018070 con 0x7fcbd00ccc80 2024-03-31T15:07:52.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:52.678+0000 7fcbc7400640 1 -- 172.21.15.16:0/3035337663 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcbb80074e0 con 0x7fcbd00ccc80 2024-03-31T15:07:52.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:52.678+0000 7fcbd6600640 1 -- 172.21.15.16:0/3035337663 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcbd0143740 con 0x7fcbd00ccc80 2024-03-31T15:07:52.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:52.678+0000 7fcbc7400640 1 -- 172.21.15.16:0/3035337663 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbb8004a10 con 0x7fcbd00ccc80 2024-03-31T15:07:52.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:52.678+0000 7fcbd6600640 1 -- 172.21.15.16:0/3035337663 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcbd0143d00 con 0x7fcbd00ccc80 2024-03-31T15:07:52.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:52.678+0000 7fcbd6600640 1 -- 172.21.15.16:0/3035337663 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcbd0143550 con 0x7fcbd00ccc80 2024-03-31T15:07:52.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:52.678+0000 7fcbc7400640 1 -- 172.21.15.16:0/3035337663 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fcbb801b070 con 0x7fcbd00ccc80 2024-03-31T15:07:52.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:52.682+0000 7fcbc7400640 1 --2- 172.21.15.16:0/3035337663 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fcbac068380 0x7fcbac06a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:52.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:52.682+0000 7fcbc7400640 1 -- 172.21.15.16:0/3035337663 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(59..59 src has 1..59) ==== 4943+0+0 (secure 0 0 0) 0x7fcbb80859a0 con 0x7fcbd00ccc80 2024-03-31T15:07:52.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:52.682+0000 7fcbcf400640 1 --2- 172.21.15.16:0/3035337663 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fcbac068380 0x7fcbac06a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:52.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:52.682+0000 7fcbcf400640 1 --2- 172.21.15.16:0/3035337663 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fcbac068380 0x7fcbac06a840 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7fcbc0009eb0 tx=0x7fcbc0009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:07:52.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:52.686+0000 7fcbc7400640 1 -- 172.21.15.16:0/3035337663 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fcbb8055c00 con 0x7fcbd00ccc80 2024-03-31T15:07:52.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:52.866+0000 7fcbd6600640 1 -- 172.21.15.16:0/3035337663 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) -- 0x7fcbd0069950 con 0x7fcbd00ccc80 2024-03-31T15:07:53.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:53.506+0000 7fcbc7400640 1 -- 172.21.15.16:0/3035337663 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v61) ==== 97+0+0 (secure 0 0 0) 0x7fcbb8059f00 con 0x7fcbd00ccc80 2024-03-31T15:07:53.501 INFO:teuthology.orchestra.run.smithi016.stderr: removed all blocklist entries 2024-03-31T15:07:53.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:53.510+0000 7fcbd6600640 1 -- 172.21.15.16:0/3035337663 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fcbac068380 msgr2=0x7fcbac06a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:53.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:53.510+0000 7fcbd6600640 1 --2- 172.21.15.16:0/3035337663 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fcbac068380 0x7fcbac06a840 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7fcbc0009eb0 tx=0x7fcbc0009190 comp rx=0 tx=0).stop 2024-03-31T15:07:53.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:53.510+0000 7fcbd6600640 1 -- 172.21.15.16:0/3035337663 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcbd00ccc80 msgr2=0x7fcbd00c7820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:53.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:53.510+0000 7fcbd6600640 1 --2- 172.21.15.16:0/3035337663 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcbd00ccc80 0x7fcbd00c7820 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7fcbb80011f0 tx=0x7fcbb800c9b0 comp rx=0 tx=0).stop 2024-03-31T15:07:53.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:53.510+0000 7fcbd6600640 1 -- 172.21.15.16:0/3035337663 shutdown_connections 2024-03-31T15:07:53.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:53.510+0000 7fcbd6600640 1 --2- 172.21.15.16:0/3035337663 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fcbac068380 0x7fcbac06a840 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:53.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:53.510+0000 7fcbd6600640 1 --2- 172.21.15.16:0/3035337663 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fcbd0145e40 0x7fcbd00c7d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:53.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:53.510+0000 7fcbd6600640 1 --2- 172.21.15.16:0/3035337663 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcbd00ccc80 0x7fcbd00c7820 unknown :-1 s=CLOSED pgs=801 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:53.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:53.510+0000 7fcbd6600640 1 --2- 172.21.15.16:0/3035337663 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcbd0070030 0x7fcbd0142470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:53.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:53.510+0000 7fcbd6600640 1 -- 172.21.15.16:0/3035337663 >> 172.21.15.16:0/3035337663 conn(0x7fcbd0067330 msgr2=0x7fcbd015fd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:07:53.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:53.510+0000 7fcbd6600640 1 -- 172.21.15.16:0/3035337663 shutdown_connections 2024-03-31T15:07:53.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:53.510+0000 7fcbd6600640 1 -- 172.21.15.16:0/3035337663 wait complete. 2024-03-31T15:07:53.532 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-03-31T15:07:53.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:53.678+0000 7f658d200640 1 Processor -- start 2024-03-31T15:07:53.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:53.678+0000 7f658d200640 1 -- start start 2024-03-31T15:07:53.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:53.682+0000 7f658d200640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f6588145d10 0x7f65880cf790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:53.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:53.682+0000 7f658d200640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f65880728b0 0x7f65880b1cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:53.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:53.682+0000 7f658d200640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f65880b2200 0x7f658815cd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:53.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:53.682+0000 7f658d200640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f6588160e70 con 0x7f65880728b0 2024-03-31T15:07:53.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:53.682+0000 7f658d200640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f65880cfcd0 con 0x7f65880b2200 2024-03-31T15:07:53.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:53.682+0000 7f658d200640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f65881602f0 con 0x7f6588145d10 2024-03-31T15:07:53.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:53.682+0000 7f657d400640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f65880728b0 0x7f65880b1cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:53.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:53.682+0000 7f657d400640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f65880728b0 0x7f65880b1cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38734/0 (socket says 172.21.15.16:38734) 2024-03-31T15:07:53.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:53.682+0000 7f657d400640 1 -- 172.21.15.16:0/3937724645 learned_addr learned my addr 172.21.15.16:0/3937724645 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:07:53.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:53.682+0000 7f6586800640 1 --2- 172.21.15.16:0/3937724645 >> v2:172.21.15.89:3300/0 conn(0x7f65880b2200 0x7f658815cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:53.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:53.682+0000 7f6585e00640 1 --2- 172.21.15.16:0/3937724645 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f6588145d10 0x7f65880cf790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:53.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:53.682+0000 7f657d400640 1 -- 172.21.15.16:0/3937724645 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f6588145d10 msgr2=0x7f65880cf790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:07:53.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:53.682+0000 7f657d400640 1 --2- 172.21.15.16:0/3937724645 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f6588145d10 0x7f65880cf790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:53.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:53.682+0000 7f657d400640 1 -- 172.21.15.16:0/3937724645 >> v2:172.21.15.89:3300/0 conn(0x7f65880b2200 msgr2=0x7f658815cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:07:53.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:53.682+0000 7f657d400640 1 --2- 172.21.15.16:0/3937724645 >> v2:172.21.15.89:3300/0 conn(0x7f65880b2200 0x7f658815cd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:53.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:53.682+0000 7f657d400640 1 -- 172.21.15.16:0/3937724645 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65881605d0 con 0x7f65880728b0 2024-03-31T15:07:53.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:53.682+0000 7f6586800640 1 --2- 172.21.15.16:0/3937724645 >> v2:172.21.15.89:3300/0 conn(0x7f65880b2200 0x7f658815cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:07:53.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:53.686+0000 7f657d400640 1 --2- 172.21.15.16:0/3937724645 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f65880728b0 0x7f65880b1cc0 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f658806c280 tx=0x7f657001c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a036a3cbc59f2cda server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:07:53.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:53.686+0000 7f6585400640 1 -- 172.21.15.16:0/3937724645 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65700a5670 con 0x7f65880728b0 2024-03-31T15:07:53.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:53.686+0000 7f6585400640 1 -- 172.21.15.16:0/3937724645 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f65700a5810 con 0x7f65880728b0 2024-03-31T15:07:53.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:53.686+0000 7f6585400640 1 -- 172.21.15.16:0/3937724645 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65700a5be0 con 0x7f65880728b0 2024-03-31T15:07:53.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:53.686+0000 7f658d200640 1 -- 172.21.15.16:0/3937724645 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f65880728b0 msgr2=0x7f65880b1cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:53.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:53.686+0000 7f658d200640 1 --2- 172.21.15.16:0/3937724645 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f65880728b0 0x7f65880b1cc0 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f658806c280 tx=0x7f657001c9f0 comp rx=0 tx=0).stop 2024-03-31T15:07:53.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:53.686+0000 7f658d200640 1 -- 172.21.15.16:0/3937724645 shutdown_connections 2024-03-31T15:07:53.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:53.686+0000 7f658d200640 1 --2- 172.21.15.16:0/3937724645 >> v2:172.21.15.89:3300/0 conn(0x7f65880b2200 0x7f658815cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:53.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:53.686+0000 7f658d200640 1 --2- 172.21.15.16:0/3937724645 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f65880728b0 0x7f65880b1cc0 unknown :-1 s=CLOSED pgs=802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:53.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:53.686+0000 7f658d200640 1 --2- 172.21.15.16:0/3937724645 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f6588145d10 0x7f65880cf790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:53.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:53.686+0000 7f658d200640 1 -- 172.21.15.16:0/3937724645 >> 172.21.15.16:0/3937724645 conn(0x7f658806b620 msgr2=0x7f658806ba30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:07:53.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:53.686+0000 7f658d200640 1 -- 172.21.15.16:0/3937724645 shutdown_connections 2024-03-31T15:07:53.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:53.686+0000 7f658d200640 1 -- 172.21.15.16:0/3937724645 wait complete. 2024-03-31T15:07:53.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:53.686+0000 7f658d200640 1 Processor -- start 2024-03-31T15:07:53.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:53.686+0000 7f658d200640 1 -- start start 2024-03-31T15:07:53.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:53.690+0000 7f658d200640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f65880728b0 0x7f658815c110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:53.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:53.690+0000 7f658d200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f65880b2200 0x7f65880a94c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:53.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:53.690+0000 7f658d200640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f6588145d10 0x7f658815b950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:53.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:53.690+0000 7f658d200640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f65880cf350 con 0x7f65880b2200 2024-03-31T15:07:53.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:53.690+0000 7f658d200640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f65880b2610 con 0x7f65880728b0 2024-03-31T15:07:53.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:53.690+0000 7f658d200640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f65880cf4d0 con 0x7f6588145d10 2024-03-31T15:07:53.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:53.690+0000 7f657d400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f65880b2200 0x7f65880a94c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:53.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:53.690+0000 7f657d400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f65880b2200 0x7f65880a94c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38746/0 (socket says 172.21.15.16:38746) 2024-03-31T15:07:53.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:53.690+0000 7f657d400640 1 -- 172.21.15.16:0/167775483 learned_addr learned my addr 172.21.15.16:0/167775483 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:07:53.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:53.690+0000 7f6585e00640 1 --2- 172.21.15.16:0/167775483 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f65880728b0 0x7f658815c110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:53.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:53.690+0000 7f6586800640 1 --2- 172.21.15.16:0/167775483 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f6588145d10 0x7f658815b950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:53.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:53.690+0000 7f657d400640 1 -- 172.21.15.16:0/167775483 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f6588145d10 msgr2=0x7f658815b950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:53.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:53.690+0000 7f657d400640 1 --2- 172.21.15.16:0/167775483 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f6588145d10 0x7f658815b950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:53.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:53.690+0000 7f657d400640 1 -- 172.21.15.16:0/167775483 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f65880728b0 msgr2=0x7f658815c110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:53.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:53.690+0000 7f657d400640 1 --2- 172.21.15.16:0/167775483 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f65880728b0 0x7f658815c110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:53.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:53.690+0000 7f657d400640 1 -- 172.21.15.16:0/167775483 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65880b4d50 con 0x7f65880b2200 2024-03-31T15:07:53.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:53.690+0000 7f6585e00640 1 --2- 172.21.15.16:0/167775483 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f65880728b0 0x7f658815c110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:07:53.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:53.690+0000 7f657d400640 1 --2- 172.21.15.16:0/167775483 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f65880b2200 0x7f65880a94c0 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7f65700af040 tx=0x7f6570019790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:07:53.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:53.690+0000 7f657ea00640 1 -- 172.21.15.16:0/167775483 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6570009510 con 0x7f65880b2200 2024-03-31T15:07:53.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:53.690+0000 7f657ea00640 1 -- 172.21.15.16:0/167775483 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f65700be070 con 0x7f65880b2200 2024-03-31T15:07:53.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:53.690+0000 7f657ea00640 1 -- 172.21.15.16:0/167775483 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65700b88b0 con 0x7f65880b2200 2024-03-31T15:07:53.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:53.690+0000 7f658d200640 1 -- 172.21.15.16:0/167775483 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f65880b5030 con 0x7f65880b2200 2024-03-31T15:07:53.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:53.690+0000 7f658d200640 1 -- 172.21.15.16:0/167775483 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6588141c10 con 0x7f65880b2200 2024-03-31T15:07:53.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:53.690+0000 7f658d200640 1 -- 172.21.15.16:0/167775483 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6588069950 con 0x7f65880b2200 2024-03-31T15:07:53.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:53.690+0000 7f657ea00640 1 -- 172.21.15.16:0/167775483 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f65700b8a50 con 0x7f65880b2200 2024-03-31T15:07:53.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:53.694+0000 7f657ea00640 1 --2- 172.21.15.16:0/167775483 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f6564068400 0x7f656406a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:53.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:53.694+0000 7f657ea00640 1 -- 172.21.15.16:0/167775483 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(63..63 src has 1..63) ==== 4857+0+0 (secure 0 0 0) 0x7f65701215f0 con 0x7f65880b2200 2024-03-31T15:07:53.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:53.694+0000 7f6585e00640 1 --2- 172.21.15.16:0/167775483 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f6564068400 0x7f656406a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:53.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:53.694+0000 7f6585e00640 1 --2- 172.21.15.16:0/167775483 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f6564068400 0x7f656406a8c0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f6578007af0 tx=0x7f6578008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:07:53.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:53.698+0000 7f657ea00640 1 -- 172.21.15.16:0/167775483 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f65700165e0 con 0x7f65880b2200 2024-03-31T15:07:53.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:53.918+0000 7f658d200640 1 -- 172.21.15.16:0/167775483 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) -- 0x7f658813f820 con 0x7f65880b2200 2024-03-31T15:07:53.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:53.918+0000 7f657ea00640 1 -- 172.21.15.16:0/167775483 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v15) ==== 80+0+5304 (secure 0 0 0) 0x7f65700f2000 con 0x7f65880b2200 2024-03-31T15:07:53.912 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:07:53.912 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-03-31T15:07:53.913 INFO:teuthology.orchestra.run.smithi016.stdout: "auth_dump": [ 2024-03-31T15:07:53.913 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-31T15:07:53.913 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "mds.a", 2024-03-31T15:07:53.913 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAOewlmhp1ANRAAYS9zo+p4tLU1YK0ySOZE/Q==", 2024-03-31T15:07:53.913 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2024-03-31T15:07:53.913 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2024-03-31T15:07:53.913 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow *", 2024-03-31T15:07:53.913 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow *", 2024-03-31T15:07:53.913 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2024-03-31T15:07:53.913 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-31T15:07:53.913 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-31T15:07:53.913 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-31T15:07:53.913 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "mds.b", 2024-03-31T15:07:53.913 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAPewlmXxupBhAAtpkt93aJOFqhc7GXLIlkBg==", 2024-03-31T15:07:53.914 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2024-03-31T15:07:53.914 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2024-03-31T15:07:53.914 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow *", 2024-03-31T15:07:53.914 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow *", 2024-03-31T15:07:53.914 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2024-03-31T15:07:53.914 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-31T15:07:53.914 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-31T15:07:53.914 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-31T15:07:53.914 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "mds.c", 2024-03-31T15:07:53.914 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAPewlmjaieARAA32teg02vE/Q8RtqLtnArMw==", 2024-03-31T15:07:53.914 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2024-03-31T15:07:53.914 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2024-03-31T15:07:53.914 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow *", 2024-03-31T15:07:53.914 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow *", 2024-03-31T15:07:53.915 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2024-03-31T15:07:53.915 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-31T15:07:53.915 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-31T15:07:53.915 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-31T15:07:53.915 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "mds.d", 2024-03-31T15:07:53.915 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAPewlmo+4PDhAAmJD0siZB3LjWkvdMq1d8aw==", 2024-03-31T15:07:53.915 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2024-03-31T15:07:53.915 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2024-03-31T15:07:53.915 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow *", 2024-03-31T15:07:53.915 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow *", 2024-03-31T15:07:53.915 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2024-03-31T15:07:53.915 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-31T15:07:53.915 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-31T15:07:53.915 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-31T15:07:53.916 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.0", 2024-03-31T15:07:53.916 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAQewlmVP6kLhAAwFr4jwQgz1N3oIUZuJvmzw==", 2024-03-31T15:07:53.916 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2024-03-31T15:07:53.916 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2024-03-31T15:07:53.916 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2024-03-31T15:07:53.916 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2024-03-31T15:07:53.916 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-31T15:07:53.916 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-31T15:07:53.916 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-31T15:07:53.916 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.1", 2024-03-31T15:07:53.916 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQATewlmBh0gNhAAR6LiuFypwA7d3SwKlWuI/Q==", 2024-03-31T15:07:53.916 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2024-03-31T15:07:53.916 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2024-03-31T15:07:53.916 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2024-03-31T15:07:53.916 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2024-03-31T15:07:53.917 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-31T15:07:53.917 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-31T15:07:53.917 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-31T15:07:53.917 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.2", 2024-03-31T15:07:53.917 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAXewlmtQwhARAACbzc7Woo9BoVFPfK3Pns+g==", 2024-03-31T15:07:53.917 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2024-03-31T15:07:53.917 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2024-03-31T15:07:53.917 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2024-03-31T15:07:53.917 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2024-03-31T15:07:53.917 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-31T15:07:53.917 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-31T15:07:53.917 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-31T15:07:53.917 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.3", 2024-03-31T15:07:53.917 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAaewlmmjWABhAA/hwvARNb51nd5vsnYVD/sg==", 2024-03-31T15:07:53.917 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2024-03-31T15:07:53.918 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2024-03-31T15:07:53.918 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2024-03-31T15:07:53.918 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2024-03-31T15:07:53.918 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-31T15:07:53.918 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-31T15:07:53.918 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-31T15:07:53.918 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.4", 2024-03-31T15:07:53.918 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAgewlmby07CxAAChxE/qPuPboIlg3fDxSOeA==", 2024-03-31T15:07:53.918 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2024-03-31T15:07:53.918 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2024-03-31T15:07:53.918 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2024-03-31T15:07:53.918 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2024-03-31T15:07:53.918 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-31T15:07:53.919 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-31T15:07:53.919 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-31T15:07:53.919 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.5", 2024-03-31T15:07:53.919 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAjewlmlD8yExAA9pgP0heBywrgqc39iU8rpQ==", 2024-03-31T15:07:53.919 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2024-03-31T15:07:53.919 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2024-03-31T15:07:53.919 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2024-03-31T15:07:53.919 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2024-03-31T15:07:53.919 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-31T15:07:53.919 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-31T15:07:53.919 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-31T15:07:53.919 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.6", 2024-03-31T15:07:53.919 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAmewlmzAsTGBAALMjVcQ08sIiTDFaBcq8L6A==", 2024-03-31T15:07:53.919 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2024-03-31T15:07:53.919 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2024-03-31T15:07:53.920 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2024-03-31T15:07:53.920 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2024-03-31T15:07:53.920 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-31T15:07:53.920 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-31T15:07:53.920 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-31T15:07:53.920 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.7", 2024-03-31T15:07:53.920 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQApewlmb+CEHRAAXE+iBumDsb2c4zH3C8f6YA==", 2024-03-31T15:07:53.920 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2024-03-31T15:07:53.920 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2024-03-31T15:07:53.920 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2024-03-31T15:07:53.920 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2024-03-31T15:07:53.920 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-31T15:07:53.920 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-31T15:07:53.920 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-31T15:07:53.920 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "client.0", 2024-03-31T15:07:53.921 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAPewlmlMIDEhAAF0Ll1HcrWHmH6xDKq2Qq9A==", 2024-03-31T15:07:53.921 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2024-03-31T15:07:53.921 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2024-03-31T15:07:53.921 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow r", 2024-03-31T15:07:53.921 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-03-31T15:07:53.921 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-31T15:07:53.921 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-31T15:07:53.921 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-31T15:07:53.921 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "client.1", 2024-03-31T15:07:53.921 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAPewlmdsvqGxAAaVdwWwri0IE/2sg7psilWw==", 2024-03-31T15:07:53.921 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2024-03-31T15:07:53.921 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2024-03-31T15:07:53.921 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow r", 2024-03-31T15:07:53.921 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow rw", 2024-03-31T15:07:53.921 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow rwx" 2024-03-31T15:07:53.922 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-31T15:07:53.922 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-31T15:07:53.922 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-31T15:07:53.922 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "client.2", 2024-03-31T15:07:53.922 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAPewlmQapPGBAAbwXSlCi+zyQ9118WyAJmrQ==", 2024-03-31T15:07:53.922 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2024-03-31T15:07:53.922 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2024-03-31T15:07:53.922 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow r", 2024-03-31T15:07:53.922 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow rw", 2024-03-31T15:07:53.922 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow rwx" 2024-03-31T15:07:53.922 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-31T15:07:53.922 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-31T15:07:53.922 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-31T15:07:53.922 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "client.3", 2024-03-31T15:07:53.923 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAPewlmCw7qIRAAEeIjF85JbyfJ6d3FtMOVhQ==", 2024-03-31T15:07:53.923 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2024-03-31T15:07:53.923 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2024-03-31T15:07:53.923 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow r", 2024-03-31T15:07:53.923 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow rw", 2024-03-31T15:07:53.923 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow rwx" 2024-03-31T15:07:53.923 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-31T15:07:53.923 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-31T15:07:53.923 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-31T15:07:53.923 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "client.admin", 2024-03-31T15:07:53.923 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAOewlmXyPZGRAAtZOheiYhzGxdeJGqqyGxTw==", 2024-03-31T15:07:53.923 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2024-03-31T15:07:53.923 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow *", 2024-03-31T15:07:53.923 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow *", 2024-03-31T15:07:53.924 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow *", 2024-03-31T15:07:53.924 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2024-03-31T15:07:53.924 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-31T15:07:53.924 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-31T15:07:53.924 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-31T15:07:53.924 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "mgr.x", 2024-03-31T15:07:53.924 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAOewlm7UkKMRAA8hqgeV86mTCkRcxdXNdWmw==", 2024-03-31T15:07:53.924 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2024-03-31T15:07:53.924 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow *", 2024-03-31T15:07:53.924 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile mgr", 2024-03-31T15:07:53.924 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2024-03-31T15:07:53.924 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-31T15:07:53.924 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-31T15:07:53.924 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-31T15:07:53.924 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "mgr.y", 2024-03-31T15:07:53.925 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAOewlmq0cpLRAAKTgh04tRaHrpMhxdwgvXYA==", 2024-03-31T15:07:53.925 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2024-03-31T15:07:53.925 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow *", 2024-03-31T15:07:53.925 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile mgr", 2024-03-31T15:07:53.925 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2024-03-31T15:07:53.925 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-31T15:07:53.925 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-31T15:07:53.925 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-03-31T15:07:53.925 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-03-31T15:07:53.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:53.918+0000 7f658d200640 1 -- 172.21.15.16:0/167775483 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f6564068400 msgr2=0x7f656406a8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:53.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:53.918+0000 7f658d200640 1 --2- 172.21.15.16:0/167775483 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f6564068400 0x7f656406a8c0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f6578007af0 tx=0x7f6578008040 comp rx=0 tx=0).stop 2024-03-31T15:07:53.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:53.918+0000 7f658d200640 1 -- 172.21.15.16:0/167775483 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f65880b2200 msgr2=0x7f65880a94c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:53.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:53.918+0000 7f658d200640 1 --2- 172.21.15.16:0/167775483 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f65880b2200 0x7f65880a94c0 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7f65700af040 tx=0x7f6570019790 comp rx=0 tx=0).stop 2024-03-31T15:07:53.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:53.918+0000 7f658d200640 1 -- 172.21.15.16:0/167775483 shutdown_connections 2024-03-31T15:07:53.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:53.918+0000 7f658d200640 1 --2- 172.21.15.16:0/167775483 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f6564068400 0x7f656406a8c0 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:53.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:53.918+0000 7f658d200640 1 --2- 172.21.15.16:0/167775483 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f6588145d10 0x7f658815b950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:53.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:53.918+0000 7f658d200640 1 --2- 172.21.15.16:0/167775483 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f65880b2200 0x7f65880a94c0 unknown :-1 s=CLOSED pgs=803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:53.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:53.918+0000 7f658d200640 1 --2- 172.21.15.16:0/167775483 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f65880728b0 0x7f658815c110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:53.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:53.918+0000 7f658d200640 1 -- 172.21.15.16:0/167775483 >> 172.21.15.16:0/167775483 conn(0x7f658806b620 msgr2=0x7f65880cff50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:07:53.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:53.918+0000 7f658d200640 1 -- 172.21.15.16:0/167775483 shutdown_connections 2024-03-31T15:07:53.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:53.918+0000 7f658d200640 1 -- 172.21.15.16:0/167775483 wait complete. 2024-03-31T15:07:53.934 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-31T15:07:54.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:54.078+0000 7fa770000640 1 Processor -- start 2024-03-31T15:07:54.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:54.078+0000 7fa770000640 1 -- start start 2024-03-31T15:07:54.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:54.078+0000 7fa770000640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fa76806dbc0 0x7fa76815f650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:54.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:54.078+0000 7fa770000640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fa768070030 0x7fa768162180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:54.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:54.078+0000 7fa770000640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa7680cbe90 con 0x7fa7680cb440 2024-03-31T15:07:54.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:54.078+0000 7fa770000640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7fa76815fb90 con 0x7fa76806dbc0 2024-03-31T15:07:54.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:54.078+0000 7fa770000640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fa76815f1c0 con 0x7fa768070030 2024-03-31T15:07:54.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:54.082+0000 7fa76e000640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fa768070030 0x7fa768162180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:54.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:54.078+0000 7fa76d600640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fa76806dbc0 0x7fa76815f650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:54.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:54.082+0000 7fa76cc00640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fa7680cb440 0x7fa76806f920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.16:34370/0 (socket says 172.21.15.16:34370) 2024-03-31T15:07:54.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:54.082+0000 7fa76cc00640 1 -- 172.21.15.16:0/448429198 learned_addr learned my addr 172.21.15.16:0/448429198 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:07:54.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:54.082+0000 7fa767e00640 1 -- 172.21.15.16:0/448429198 <== mon.1 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1448969250 0 0) 0x7fa7680cbe90 con 0x7fa7680cb440 2024-03-31T15:07:54.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:54.082+0000 7fa76e000640 1 -- 172.21.15.16:0/448429198 >> v2:172.21.15.89:3300/0 conn(0x7fa76806dbc0 msgr2=0x7fa76815f650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:07:54.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:54.082+0000 7fa76e000640 1 --2- 172.21.15.16:0/448429198 >> v2:172.21.15.89:3300/0 conn(0x7fa76806dbc0 0x7fa76815f650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:54.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:54.082+0000 7fa76e000640 1 -- 172.21.15.16:0/448429198 >> v1:172.21.15.89:6789/0 conn(0x7fa7680cb440 legacy=0x7fa76806f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:54.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:54.082+0000 7fa76e000640 1 -- 172.21.15.16:0/448429198 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7680cc170 con 0x7fa768070030 2024-03-31T15:07:54.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:54.082+0000 7fa76e000640 1 --2- 172.21.15.16:0/448429198 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fa768070030 0x7fa768162180 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fa760009d90 tx=0x7fa76001c8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1f09f72412597f8b server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:07:54.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:54.082+0000 7fa767e00640 1 -- 172.21.15.16:0/448429198 <== mon.2 v2:172.21.15.89:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7600b2070 con 0x7fa768070030 2024-03-31T15:07:54.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:54.082+0000 7fa767e00640 1 -- 172.21.15.16:0/448429198 <== mon.2 v2:172.21.15.89:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa7600a5880 con 0x7fa768070030 2024-03-31T15:07:54.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:54.082+0000 7fa767e00640 1 -- 172.21.15.16:0/448429198 <== mon.2 v2:172.21.15.89:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7600a5c50 con 0x7fa768070030 2024-03-31T15:07:54.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:54.086+0000 7fa770000640 1 -- 172.21.15.16:0/448429198 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fa768070030 msgr2=0x7fa768162180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:54.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:54.086+0000 7fa770000640 1 --2- 172.21.15.16:0/448429198 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fa768070030 0x7fa768162180 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fa760009d90 tx=0x7fa76001c8b0 comp rx=0 tx=0).stop 2024-03-31T15:07:54.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:54.086+0000 7fa770000640 1 -- 172.21.15.16:0/448429198 shutdown_connections 2024-03-31T15:07:54.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:54.086+0000 7fa770000640 1 --2- 172.21.15.16:0/448429198 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fa768070030 0x7fa768162180 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:54.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:54.086+0000 7fa770000640 1 --2- 172.21.15.16:0/448429198 >> v2:172.21.15.89:3300/0 conn(0x7fa76806dbc0 0x7fa76815f650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:54.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:54.086+0000 7fa770000640 1 -- 172.21.15.16:0/448429198 >> 172.21.15.16:0/448429198 conn(0x7fa768067330 msgr2=0x7fa768067740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:07:54.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:54.086+0000 7fa770000640 1 -- 172.21.15.16:0/448429198 shutdown_connections 2024-03-31T15:07:54.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:54.086+0000 7fa770000640 1 -- 172.21.15.16:0/448429198 wait complete. 2024-03-31T15:07:54.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:54.086+0000 7fa770000640 1 Processor -- start 2024-03-31T15:07:54.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:54.086+0000 7fa770000640 1 -- start start 2024-03-31T15:07:54.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:54.086+0000 7fa770000640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa76806dbc0 0x7fa7681426c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:54.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:54.086+0000 7fa770000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa768070030 0x7fa768143bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:54.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:54.086+0000 7fa770000640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fa7680cb440 0x7fa768163350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:54.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:54.086+0000 7fa770000640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fa76806cad0 con 0x7fa768070030 2024-03-31T15:07:54.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:54.086+0000 7fa770000640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fa76806c950 con 0x7fa76806dbc0 2024-03-31T15:07:54.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:54.086+0000 7fa770000640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fa76816b520 con 0x7fa7680cb440 2024-03-31T15:07:54.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:54.086+0000 7fa76cc00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa768070030 0x7fa768143bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:54.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:54.086+0000 7fa76cc00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa768070030 0x7fa768143bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38750/0 (socket says 172.21.15.16:38750) 2024-03-31T15:07:54.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:54.086+0000 7fa76cc00640 1 -- 172.21.15.16:0/508979636 learned_addr learned my addr 172.21.15.16:0/508979636 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:07:54.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:54.086+0000 7fa76e000640 1 --2- 172.21.15.16:0/508979636 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fa7680cb440 0x7fa768163350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:54.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:54.086+0000 7fa76d600640 1 --2- 172.21.15.16:0/508979636 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa76806dbc0 0x7fa7681426c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:54.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:54.086+0000 7fa76cc00640 1 -- 172.21.15.16:0/508979636 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fa7680cb440 msgr2=0x7fa768163350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:54.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:54.086+0000 7fa76cc00640 1 --2- 172.21.15.16:0/508979636 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fa7680cb440 0x7fa768163350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:54.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:54.086+0000 7fa76cc00640 1 -- 172.21.15.16:0/508979636 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa76806dbc0 msgr2=0x7fa7681426c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:54.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:54.086+0000 7fa76cc00640 1 --2- 172.21.15.16:0/508979636 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa76806dbc0 0x7fa7681426c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:54.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:54.086+0000 7fa76cc00640 1 -- 172.21.15.16:0/508979636 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa768163c80 con 0x7fa768070030 2024-03-31T15:07:54.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:54.090+0000 7fa76cc00640 1 --2- 172.21.15.16:0/508979636 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa768070030 0x7fa768143bb0 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7fa7540011f0 tx=0x7fa75400c9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:07:54.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:54.090+0000 7fa765600640 1 -- 172.21.15.16:0/508979636 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa754018070 con 0x7fa768070030 2024-03-31T15:07:54.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:54.090+0000 7fa765600640 1 -- 172.21.15.16:0/508979636 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa7540074e0 con 0x7fa768070030 2024-03-31T15:07:54.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:54.090+0000 7fa770000640 1 -- 172.21.15.16:0/508979636 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa7681430d0 con 0x7fa768070030 2024-03-31T15:07:54.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:54.090+0000 7fa765600640 1 -- 172.21.15.16:0/508979636 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa754004a10 con 0x7fa768070030 2024-03-31T15:07:54.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:54.090+0000 7fa770000640 1 -- 172.21.15.16:0/508979636 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa768143690 con 0x7fa768070030 2024-03-31T15:07:54.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:54.090+0000 7fa770000640 1 -- 172.21.15.16:0/508979636 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa768143890 con 0x7fa768070030 2024-03-31T15:07:54.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:54.090+0000 7fa765600640 1 -- 172.21.15.16:0/508979636 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fa75401b070 con 0x7fa768070030 2024-03-31T15:07:54.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:54.090+0000 7fa765600640 1 --2- 172.21.15.16:0/508979636 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fa748068380 0x7fa74806a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:54.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:54.090+0000 7fa765600640 1 -- 172.21.15.16:0/508979636 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(63..63 src has 1..63) ==== 4857+0+0 (secure 0 0 0) 0x7fa754085920 con 0x7fa768070030 2024-03-31T15:07:54.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:54.094+0000 7fa76d600640 1 --2- 172.21.15.16:0/508979636 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fa748068380 0x7fa74806a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:54.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:54.094+0000 7fa76d600640 1 --2- 172.21.15.16:0/508979636 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fa748068380 0x7fa74806a840 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fa75c009ac0 tx=0x7fa75c009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:07:54.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:54.098+0000 7fa765600640 1 -- 172.21.15.16:0/508979636 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fa768143890 con 0x7fa768070030 2024-03-31T15:07:54.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:54.266+0000 7fa770000640 1 -- 172.21.15.16:0/508979636 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fa7680cf380 con 0x7fa768070030 2024-03-31T15:07:54.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:54.266+0000 7fa765600640 1 -- 172.21.15.16:0/508979636 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v63) ==== 74+0+12819 (secure 0 0 0) 0x7fa754055bd0 con 0x7fa768070030 2024-03-31T15:07:54.261 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:07:54.261 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":63,"fsid":"286eb5da-fb47-4550-b290-d9519c13b9a5","created":"2024-03-31T15:03:11.438952+0000","modified":"2024-03-31T15:07:53.520484+0000","last_up_change":"2024-03-31T15:03:22.885535+0000","last_in_change":"2024-03-31T15:03:13.941667+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-31T15:03:22.550787+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"92a66450-ef19-40d8-ac3d-d760b23b0bda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6801","nonce":932712244}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6803","nonce":932712244}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6807","nonce":932712244}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6805","nonce":932712244}]},"public_addr":"172.21.15.16:6801/932712244","cluster_addr":"172.21.15.16:6803/932712244","heartbeat_back_addr":"172.21.15.16:6807/932712244","heartbeat_front_addr":"172.21.15.16:6805/932712244","state":["exists","up"]},{"osd":1,"uuid":"da9d1f2b-1968-4959-b843-faa231988e92","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6810","nonce":1398737533}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6811","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6813","nonce":1398737533}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6821","nonce":1398737533}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6815","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6816","nonce":1398737533}]},"public_addr":"172.21.15.16:6810/1398737533","cluster_addr":"172.21.15.16:6813/1398737533","heartbeat_back_addr":"172.21.15.16:6821/1398737533","heartbeat_front_addr":"172.21.15.16:6816/1398737533","state":["exists","up"]},{"osd":2,"uuid":"01a73662-9a21-4b1f-bcbb-42b9636a4638","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6819","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6822","nonce":3018526646}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6825","nonce":3018526646}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6831","nonce":3018526646}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6827","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6829","nonce":3018526646}]},"public_addr":"172.21.15.16:6822/3018526646","cluster_addr":"172.21.15.16:6825/3018526646","heartbeat_back_addr":"172.21.15.16:6831/3018526646","heartbeat_front_addr":"172.21.15.16:6829/3018526646","state":["exists","up"]},{"osd":3,"uuid":"c7771fd3-e18e-4500-ac38-23cf10ebcd75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6809","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6812","nonce":3360352974}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6817","nonce":3360352974}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6828","nonce":3360352974}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6823","nonce":3360352974}]},"public_addr":"172.21.15.16:6812/3360352974","cluster_addr":"172.21.15.16:6817/3360352974","heartbeat_back_addr":"172.21.15.16:6828/3360352974","heartbeat_front_addr":"172.21.15.16:6823/3360352974","state":["exists","up"]},{"osd":4,"uuid":"88260730-1482-4fbd-a40f-cb1124e9df4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6801","nonce":1856844856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6803","nonce":1856844856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6807","nonce":1856844856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6805","nonce":1856844856}]},"public_addr":"172.21.15.89:6801/1856844856","cluster_addr":"172.21.15.89:6803/1856844856","heartbeat_back_addr":"172.21.15.89:6807/1856844856","heartbeat_front_addr":"172.21.15.89:6805/1856844856","state":["exists","up"]},{"osd":5,"uuid":"4cb2c00c-fae0-4dd8-9d27-ec4f0093ccf7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6814","nonce":50395527}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6817","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6820","nonce":50395527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6831","nonce":50395527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6823","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6826","nonce":50395527}]},"public_addr":"172.21.15.89:6814/50395527","cluster_addr":"172.21.15.89:6820/50395527","heartbeat_back_addr":"172.21.15.89:6831/50395527","heartbeat_front_addr":"172.21.15.89:6826/50395527","state":["exists","up"]},{"osd":6,"uuid":"81ce0c19-2554-4cae-acbe-66a05e9e9b9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6809","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6812","nonce":3470471752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6815","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6818","nonce":3470471752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6827","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6829","nonce":3470471752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6821","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6824","nonce":3470471752}]},"public_addr":"172.21.15.89:6812/3470471752","cluster_addr":"172.21.15.89:6818/3470471752","heartbeat_back_addr":"172.21.15.89:6829/3470471752","heartbeat_front_addr":"172.21.15.89:6824/3470471752","state":["exists","up"]},{"osd":7,"uuid":"85e4313b-d37d-442e-8042-153bf20a847d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6811","nonce":1796778630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6813","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6816","nonce":1796778630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6825","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6828","nonce":1796778630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6819","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6822","nonce":1796778630}]},"public_addr":"172.21.15.89:6811/1796778630","cluster_addr":"172.21.15.89:6816/1796778630","heartbeat_back_addr":"172.21.15.89:6828/1796778630","heartbeat_front_addr":"172.21.15.89:6822/1796778630","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.439088+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.529970+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.680512+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:21.044161+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.582111+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.536558+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.409522+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.590229+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-31T15:07:54.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:54.266+0000 7fa770000640 1 -- 172.21.15.16:0/508979636 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fa748068380 msgr2=0x7fa74806a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:54.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:54.266+0000 7fa770000640 1 --2- 172.21.15.16:0/508979636 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fa748068380 0x7fa74806a840 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fa75c009ac0 tx=0x7fa75c009190 comp rx=0 tx=0).stop 2024-03-31T15:07:54.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:54.266+0000 7fa770000640 1 -- 172.21.15.16:0/508979636 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa768070030 msgr2=0x7fa768143bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:54.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:54.266+0000 7fa770000640 1 --2- 172.21.15.16:0/508979636 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa768070030 0x7fa768143bb0 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7fa7540011f0 tx=0x7fa75400c9b0 comp rx=0 tx=0).stop 2024-03-31T15:07:54.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:54.266+0000 7fa770000640 1 -- 172.21.15.16:0/508979636 shutdown_connections 2024-03-31T15:07:54.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:54.266+0000 7fa770000640 1 --2- 172.21.15.16:0/508979636 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fa748068380 0x7fa74806a840 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:54.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:54.266+0000 7fa770000640 1 --2- 172.21.15.16:0/508979636 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fa7680cb440 0x7fa768163350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:54.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:54.266+0000 7fa770000640 1 --2- 172.21.15.16:0/508979636 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa768070030 0x7fa768143bb0 unknown :-1 s=CLOSED pgs=804 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:54.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:54.266+0000 7fa770000640 1 --2- 172.21.15.16:0/508979636 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa76806dbc0 0x7fa7681426c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:54.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:54.266+0000 7fa770000640 1 -- 172.21.15.16:0/508979636 >> 172.21.15.16:0/508979636 conn(0x7fa768067330 msgr2=0x7fa76815c7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:07:54.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:54.266+0000 7fa770000640 1 -- 172.21.15.16:0/508979636 shutdown_connections 2024-03-31T15:07:54.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:54.266+0000 7fa770000640 1 -- 172.21.15.16:0/508979636 wait complete. 2024-03-31T15:07:54.282 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-31T15:03:22.550787+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-31T15:07:54.282 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-31T15:07:54.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:54.430+0000 7f6b91600640 1 Processor -- start 2024-03-31T15:07:54.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:54.430+0000 7f6b91600640 1 -- start start 2024-03-31T15:07:54.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:54.430+0000 7f6b91600640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f6b8c15c390 0x7f6b8c0cfb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:54.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:54.430+0000 7f6b91600640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f6b8c06fbc0 0x7f6b8c070070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:54.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:54.430+0000 7f6b91600640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6b8c0cd750 con 0x7f6b8c0d2b40 2024-03-31T15:07:54.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:54.430+0000 7f6b91600640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f6b8c15c770 con 0x7f6b8c15c390 2024-03-31T15:07:54.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:54.430+0000 7f6b91600640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f6b8c0cda30 con 0x7f6b8c06fbc0 2024-03-31T15:07:54.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:54.430+0000 7f6b8be00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f6b8c15c390 0x7f6b8c0cfb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:54.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:54.430+0000 7f6b8be00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f6b8c15c390 0x7f6b8c0cfb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38752/0 (socket says 172.21.15.16:38752) 2024-03-31T15:07:54.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:54.430+0000 7f6b8be00640 1 -- 172.21.15.16:0/3616683223 learned_addr learned my addr 172.21.15.16:0/3616683223 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:07:54.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:54.430+0000 7f6b90c00640 1 --2- 172.21.15.16:0/3616683223 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f6b8c06fbc0 0x7f6b8c070070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:54.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:54.430+0000 7f6b8be00640 1 -- 172.21.15.16:0/3616683223 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f6b8c06fbc0 msgr2=0x7f6b8c070070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:07:54.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:54.430+0000 7f6b8be00640 1 --2- 172.21.15.16:0/3616683223 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f6b8c06fbc0 0x7f6b8c070070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:54.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:54.430+0000 7f6b8be00640 1 -- 172.21.15.16:0/3616683223 >> v1:172.21.15.89:6789/0 conn(0x7f6b8c0d2b40 legacy=0x7f6b8c0d00a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:07:54.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:54.430+0000 7f6b8be00640 1 -- 172.21.15.16:0/3616683223 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b8c0cdd10 con 0x7f6b8c15c390 2024-03-31T15:07:54.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:54.434+0000 7f6b8be00640 1 --2- 172.21.15.16:0/3616683223 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6b8c15c390 0x7f6b8c0cfb60 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7f6b7c009e50 tx=0x7f6b7c01ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7cac2414203ab38e server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:07:54.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:54.434+0000 7f6b8b400640 1 -- 172.21.15.16:0/3616683223 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b7c0a5700 con 0x7f6b8c15c390 2024-03-31T15:07:54.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:54.434+0000 7f6b8b400640 1 -- 172.21.15.16:0/3616683223 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6b7c0a58a0 con 0x7f6b8c15c390 2024-03-31T15:07:54.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:54.434+0000 7f6b8b400640 1 -- 172.21.15.16:0/3616683223 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b7c0a5c70 con 0x7f6b8c15c390 2024-03-31T15:07:54.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:54.434+0000 7f6b91600640 1 -- 172.21.15.16:0/3616683223 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6b8c15c390 msgr2=0x7f6b8c0cfb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:54.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:54.434+0000 7f6b91600640 1 --2- 172.21.15.16:0/3616683223 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6b8c15c390 0x7f6b8c0cfb60 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7f6b7c009e50 tx=0x7f6b7c01ca30 comp rx=0 tx=0).stop 2024-03-31T15:07:54.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:54.434+0000 7f6b91600640 1 -- 172.21.15.16:0/3616683223 shutdown_connections 2024-03-31T15:07:54.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:54.434+0000 7f6b91600640 1 --2- 172.21.15.16:0/3616683223 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f6b8c06fbc0 0x7f6b8c070070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:54.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:54.434+0000 7f6b91600640 1 --2- 172.21.15.16:0/3616683223 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6b8c15c390 0x7f6b8c0cfb60 unknown :-1 s=CLOSED pgs=805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:54.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:54.434+0000 7f6b91600640 1 -- 172.21.15.16:0/3616683223 >> 172.21.15.16:0/3616683223 conn(0x7f6b8c0d1f90 msgr2=0x7f6b8c0d23a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:07:54.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:54.434+0000 7f6b91600640 1 -- 172.21.15.16:0/3616683223 shutdown_connections 2024-03-31T15:07:54.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:54.434+0000 7f6b91600640 1 -- 172.21.15.16:0/3616683223 wait complete. 2024-03-31T15:07:54.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:54.438+0000 7f6b91600640 1 Processor -- start 2024-03-31T15:07:54.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:54.438+0000 7f6b91600640 1 -- start start 2024-03-31T15:07:54.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:54.438+0000 7f6b91600640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6b8c06fbc0 0x7f6b8c1599b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:54.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:54.438+0000 7f6b91600640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f6b8c0d2b40 0x7f6b8c15b220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:54.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:54.438+0000 7f6b91600640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6b8c15c390 0x7f6b8c0aec30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:54.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:54.438+0000 7f6b91600640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f6b8c168fd0 con 0x7f6b8c06fbc0 2024-03-31T15:07:54.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:54.438+0000 7f6b91600640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f6b8c168e50 con 0x7f6b8c15c390 2024-03-31T15:07:54.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:54.438+0000 7f6b91600640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f6b8c169150 con 0x7f6b8c0d2b40 2024-03-31T15:07:54.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:54.438+0000 7f6b8be00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6b8c06fbc0 0x7f6b8c1599b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:54.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:54.438+0000 7f6b8be00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6b8c06fbc0 0x7f6b8c1599b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38762/0 (socket says 172.21.15.16:38762) 2024-03-31T15:07:54.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:54.438+0000 7f6b8be00640 1 -- 172.21.15.16:0/4007405128 learned_addr learned my addr 172.21.15.16:0/4007405128 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:07:54.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:54.438+0000 7f6b83400640 1 --2- 172.21.15.16:0/4007405128 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f6b8c0d2b40 0x7f6b8c15b220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:54.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:54.438+0000 7f6b90c00640 1 --2- 172.21.15.16:0/4007405128 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6b8c15c390 0x7f6b8c0aec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:54.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:54.438+0000 7f6b8be00640 1 -- 172.21.15.16:0/4007405128 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f6b8c0d2b40 msgr2=0x7f6b8c15b220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:54.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:54.438+0000 7f6b8be00640 1 --2- 172.21.15.16:0/4007405128 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f6b8c0d2b40 0x7f6b8c15b220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:54.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:54.438+0000 7f6b8be00640 1 -- 172.21.15.16:0/4007405128 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6b8c15c390 msgr2=0x7f6b8c0aec30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:54.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:54.438+0000 7f6b8be00640 1 --2- 172.21.15.16:0/4007405128 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6b8c15c390 0x7f6b8c0aec30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:54.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:54.438+0000 7f6b8be00640 1 -- 172.21.15.16:0/4007405128 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b8c0b5e00 con 0x7f6b8c06fbc0 2024-03-31T15:07:54.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:54.438+0000 7f6b8be00640 1 --2- 172.21.15.16:0/4007405128 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6b8c06fbc0 0x7f6b8c1599b0 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7f6b7c01a660 tx=0x7f6b7c0197d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:07:54.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:54.438+0000 7f6b88c00640 1 -- 172.21.15.16:0/4007405128 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b7c0bc020 con 0x7f6b8c06fbc0 2024-03-31T15:07:54.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:54.438+0000 7f6b88c00640 1 -- 172.21.15.16:0/4007405128 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6b7c0b8670 con 0x7f6b8c06fbc0 2024-03-31T15:07:54.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:54.438+0000 7f6b88c00640 1 -- 172.21.15.16:0/4007405128 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b7c0b8a40 con 0x7f6b8c06fbc0 2024-03-31T15:07:54.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:54.438+0000 7f6b91600640 1 -- 172.21.15.16:0/4007405128 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6b8c0b60e0 con 0x7f6b8c06fbc0 2024-03-31T15:07:54.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:54.438+0000 7f6b91600640 1 -- 172.21.15.16:0/4007405128 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6b8c0b66a0 con 0x7f6b8c06fbc0 2024-03-31T15:07:54.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:54.442+0000 7f6b91600640 1 -- 172.21.15.16:0/4007405128 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6b8c04f8c0 con 0x7f6b8c06fbc0 2024-03-31T15:07:54.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:54.442+0000 7f6b88c00640 1 -- 172.21.15.16:0/4007405128 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f6b7c0bf020 con 0x7f6b8c06fbc0 2024-03-31T15:07:54.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:54.442+0000 7f6b88c00640 1 --2- 172.21.15.16:0/4007405128 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f6b68068380 0x7f6b6806a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:54.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:54.442+0000 7f6b88c00640 1 -- 172.21.15.16:0/4007405128 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(63..63 src has 1..63) ==== 4857+0+0 (secure 0 0 0) 0x7f6b7c121640 con 0x7f6b8c06fbc0 2024-03-31T15:07:54.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:54.442+0000 7f6b83400640 1 --2- 172.21.15.16:0/4007405128 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f6b68068380 0x7f6b6806a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:54.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:54.442+0000 7f6b83400640 1 --2- 172.21.15.16:0/4007405128 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f6b68068380 0x7f6b6806a840 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f6b8c1fa930 tx=0x7f6b74009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:07:54.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:54.446+0000 7f6b88c00640 1 -- 172.21.15.16:0/4007405128 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f6b7c0c6030 con 0x7f6b8c06fbc0 2024-03-31T15:07:54.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:54.614+0000 7f6b91600640 1 -- 172.21.15.16:0/4007405128 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f6b8c0b75f0 con 0x7f6b8c06fbc0 2024-03-31T15:07:54.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:54.614+0000 7f6b88c00640 1 -- 172.21.15.16:0/4007405128 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v63) ==== 93+0+10 (secure 0 0 0) 0x7f6b7c016620 con 0x7f6b8c06fbc0 2024-03-31T15:07:54.610 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2024-03-31T15:07:54.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:54.618+0000 7f6b91600640 1 -- 172.21.15.16:0/4007405128 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f6b68068380 msgr2=0x7f6b6806a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:54.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:54.618+0000 7f6b91600640 1 --2- 172.21.15.16:0/4007405128 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f6b68068380 0x7f6b6806a840 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f6b8c1fa930 tx=0x7f6b74009040 comp rx=0 tx=0).stop 2024-03-31T15:07:54.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:54.618+0000 7f6b91600640 1 -- 172.21.15.16:0/4007405128 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6b8c06fbc0 msgr2=0x7f6b8c1599b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:54.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:54.618+0000 7f6b91600640 1 --2- 172.21.15.16:0/4007405128 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6b8c06fbc0 0x7f6b8c1599b0 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7f6b7c01a660 tx=0x7f6b7c0197d0 comp rx=0 tx=0).stop 2024-03-31T15:07:54.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:54.618+0000 7f6b91600640 1 -- 172.21.15.16:0/4007405128 shutdown_connections 2024-03-31T15:07:54.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:54.618+0000 7f6b91600640 1 --2- 172.21.15.16:0/4007405128 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f6b68068380 0x7f6b6806a840 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:54.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:54.618+0000 7f6b91600640 1 --2- 172.21.15.16:0/4007405128 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6b8c15c390 0x7f6b8c0aec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:54.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:54.618+0000 7f6b91600640 1 --2- 172.21.15.16:0/4007405128 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f6b8c0d2b40 0x7f6b8c15b220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:54.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:54.618+0000 7f6b91600640 1 --2- 172.21.15.16:0/4007405128 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6b8c06fbc0 0x7f6b8c1599b0 unknown :-1 s=CLOSED pgs=806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:54.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:54.618+0000 7f6b91600640 1 -- 172.21.15.16:0/4007405128 >> 172.21.15.16:0/4007405128 conn(0x7f6b8c0d1f90 msgr2=0x7f6b8c1556a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:07:54.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:54.618+0000 7f6b91600640 1 -- 172.21.15.16:0/4007405128 shutdown_connections 2024-03-31T15:07:54.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:54.618+0000 7f6b91600640 1 -- 172.21.15.16:0/4007405128 wait complete. 2024-03-31T15:07:54.631 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-31T15:07:54.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:54.778+0000 7fe131800640 1 Processor -- start 2024-03-31T15:07:54.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:54.778+0000 7fe131800640 1 -- start start 2024-03-31T15:07:54.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:54.778+0000 7fe131800640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fe12c15c1d0 0x7fe12c0b44e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:54.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:54.778+0000 7fe131800640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fe12c15fd40 0x7fe12c160190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:54.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:54.778+0000 7fe131800640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe12c0cd980 con 0x7fe12c0cd4f0 2024-03-31T15:07:54.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:54.778+0000 7fe131800640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7fe12c1606d0 con 0x7fe12c15c1d0 2024-03-31T15:07:54.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:54.778+0000 7fe131800640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fe12c0cfaa0 con 0x7fe12c15fd40 2024-03-31T15:07:54.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:54.778+0000 7fe123e00640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fe12c0cd4f0 0x7fe12c06f5d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:49338/0 (socket says 172.21.15.16:49338) 2024-03-31T15:07:54.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:54.778+0000 7fe12be00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fe12c15c1d0 0x7fe12c0b44e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:54.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:54.778+0000 7fe123e00640 1 -- 172.21.15.16:0/1120987851 learned_addr learned my addr 172.21.15.16:0/1120987851 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:07:54.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:54.778+0000 7fe130e00640 1 --2- 172.21.15.16:0/1120987851 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fe12c15fd40 0x7fe12c160190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:54.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:54.778+0000 7fe12be00640 1 -- 172.21.15.16:0/1120987851 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fe12c15fd40 msgr2=0x7fe12c160190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:07:54.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:54.778+0000 7fe12be00640 1 --2- 172.21.15.16:0/1120987851 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fe12c15fd40 0x7fe12c160190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:54.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:54.778+0000 7fe12be00640 1 -- 172.21.15.16:0/1120987851 >> v1:172.21.15.16:6789/0 conn(0x7fe12c0cd4f0 legacy=0x7fe12c06f5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:54.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:54.778+0000 7fe12be00640 1 -- 172.21.15.16:0/1120987851 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe12c0cdc60 con 0x7fe12c15c1d0 2024-03-31T15:07:54.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:54.782+0000 7fe12be00640 1 --2- 172.21.15.16:0/1120987851 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe12c15c1d0 0x7fe12c0b44e0 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7fe11c0097c0 tx=0x7fe11c01c470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14e10414ae224424 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:07:54.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:54.782+0000 7fe12b400640 1 -- 172.21.15.16:0/1120987851 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe11c0a7070 con 0x7fe12c15c1d0 2024-03-31T15:07:54.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:54.782+0000 7fe12b400640 1 -- 172.21.15.16:0/1120987851 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe11c0b0070 con 0x7fe12c15c1d0 2024-03-31T15:07:54.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:54.782+0000 7fe12b400640 1 -- 172.21.15.16:0/1120987851 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe11c0ab890 con 0x7fe12c15c1d0 2024-03-31T15:07:54.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:54.782+0000 7fe131800640 1 -- 172.21.15.16:0/1120987851 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe12c15c1d0 msgr2=0x7fe12c0b44e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:54.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:54.782+0000 7fe131800640 1 --2- 172.21.15.16:0/1120987851 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe12c15c1d0 0x7fe12c0b44e0 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7fe11c0097c0 tx=0x7fe11c01c470 comp rx=0 tx=0).stop 2024-03-31T15:07:54.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:54.782+0000 7fe131800640 1 -- 172.21.15.16:0/1120987851 shutdown_connections 2024-03-31T15:07:54.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:54.782+0000 7fe131800640 1 --2- 172.21.15.16:0/1120987851 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fe12c15fd40 0x7fe12c160190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:54.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:54.782+0000 7fe131800640 1 --2- 172.21.15.16:0/1120987851 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe12c15c1d0 0x7fe12c0b44e0 unknown :-1 s=CLOSED pgs=808 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:54.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:54.782+0000 7fe131800640 1 -- 172.21.15.16:0/1120987851 >> 172.21.15.16:0/1120987851 conn(0x7fe12c0b39c0 msgr2=0x7fe12c0b3dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:07:54.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:54.782+0000 7fe131800640 1 -- 172.21.15.16:0/1120987851 shutdown_connections 2024-03-31T15:07:54.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:54.782+0000 7fe131800640 1 -- 172.21.15.16:0/1120987851 wait complete. 2024-03-31T15:07:54.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:54.782+0000 7fe131800640 1 Processor -- start 2024-03-31T15:07:54.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:54.782+0000 7fe131800640 1 -- start start 2024-03-31T15:07:54.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:54.786+0000 7fe131800640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fe12c15c1d0 0x7fe12c0c8fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:54.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:54.786+0000 7fe131800640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe12c15fd40 0x7fe12c14fef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:54.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:54.786+0000 7fe131800640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe12c165f30 0x7fe12c155740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:54.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:54.786+0000 7fe131800640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fe12c06fce0 con 0x7fe12c15fd40 2024-03-31T15:07:54.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:54.786+0000 7fe131800640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fe12c0b1420 con 0x7fe12c165f30 2024-03-31T15:07:54.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:54.786+0000 7fe131800640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fe12c06cb10 con 0x7fe12c15c1d0 2024-03-31T15:07:54.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:54.786+0000 7fe123e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe12c15fd40 0x7fe12c14fef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:54.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:54.786+0000 7fe123e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe12c15fd40 0x7fe12c14fef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38770/0 (socket says 172.21.15.16:38770) 2024-03-31T15:07:54.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:54.786+0000 7fe123e00640 1 -- 172.21.15.16:0/1040112031 learned_addr learned my addr 172.21.15.16:0/1040112031 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:07:54.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:54.786+0000 7fe12be00640 1 --2- 172.21.15.16:0/1040112031 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fe12c15c1d0 0x7fe12c0c8fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:54.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:54.786+0000 7fe130e00640 1 --2- 172.21.15.16:0/1040112031 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe12c165f30 0x7fe12c155740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:54.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:54.786+0000 7fe123e00640 1 -- 172.21.15.16:0/1040112031 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fe12c15c1d0 msgr2=0x7fe12c0c8fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:54.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:54.786+0000 7fe123e00640 1 --2- 172.21.15.16:0/1040112031 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fe12c15c1d0 0x7fe12c0c8fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:54.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:54.786+0000 7fe123e00640 1 -- 172.21.15.16:0/1040112031 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe12c165f30 msgr2=0x7fe12c155740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:54.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:54.786+0000 7fe123e00640 1 --2- 172.21.15.16:0/1040112031 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe12c165f30 0x7fe12c155740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:54.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:54.786+0000 7fe123e00640 1 -- 172.21.15.16:0/1040112031 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe12c06cdf0 con 0x7fe12c15fd40 2024-03-31T15:07:54.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:54.786+0000 7fe123e00640 1 --2- 172.21.15.16:0/1040112031 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe12c15fd40 0x7fe12c14fef0 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7fe11400c420 tx=0x7fe11400c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:07:54.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:54.786+0000 7fe128c00640 1 -- 172.21.15.16:0/1040112031 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe114017070 con 0x7fe12c15fd40 2024-03-31T15:07:54.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:54.786+0000 7fe128c00640 1 -- 172.21.15.16:0/1040112031 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe114005cf0 con 0x7fe12c15fd40 2024-03-31T15:07:54.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:54.786+0000 7fe131800640 1 -- 172.21.15.16:0/1040112031 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe12c06d0d0 con 0x7fe12c15fd40 2024-03-31T15:07:54.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:54.786+0000 7fe128c00640 1 -- 172.21.15.16:0/1040112031 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1140072f0 con 0x7fe12c15fd40 2024-03-31T15:07:54.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:54.786+0000 7fe131800640 1 -- 172.21.15.16:0/1040112031 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe12c06d690 con 0x7fe12c15fd40 2024-03-31T15:07:54.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:54.786+0000 7fe131800640 1 -- 172.21.15.16:0/1040112031 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe12c06d890 con 0x7fe12c15fd40 2024-03-31T15:07:54.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:54.790+0000 7fe128c00640 1 -- 172.21.15.16:0/1040112031 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fe11401a070 con 0x7fe12c15fd40 2024-03-31T15:07:54.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:54.790+0000 7fe128c00640 1 --2- 172.21.15.16:0/1040112031 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fe108068380 0x7fe10806a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:54.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:54.790+0000 7fe128c00640 1 -- 172.21.15.16:0/1040112031 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(63..63 src has 1..63) ==== 4857+0+0 (secure 0 0 0) 0x7fe114085a20 con 0x7fe12c15fd40 2024-03-31T15:07:54.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:54.790+0000 7fe12be00640 1 --2- 172.21.15.16:0/1040112031 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fe108068380 0x7fe10806a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:54.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:54.790+0000 7fe12be00640 1 --2- 172.21.15.16:0/1040112031 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fe108068380 0x7fe10806a840 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fe11c0af000 tx=0x7fe11c0a7000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:07:54.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:54.794+0000 7fe128c00640 1 -- 172.21.15.16:0/1040112031 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fe12c06d890 con 0x7fe12c15fd40 2024-03-31T15:07:54.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:54.962+0000 7fe131800640 1 -- 172.21.15.16:0/1040112031 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7fe12c04f8c0 con 0x7fe12c15fd40 2024-03-31T15:07:54.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:54.962+0000 7fe128c00640 1 -- 172.21.15.16:0/1040112031 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v63) ==== 84+0+65 (secure 0 0 0) 0x7fe114056330 con 0x7fe12c15fd40 2024-03-31T15:07:54.959 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:07:54.959 INFO:teuthology.orchestra.run.smithi016.stdout:[ 2024-03-31T15:07:54.959 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-31T15:07:54.959 INFO:teuthology.orchestra.run.smithi016.stdout: "poolnum": 1, 2024-03-31T15:07:54.959 INFO:teuthology.orchestra.run.smithi016.stdout: "poolname": ".mgr" 2024-03-31T15:07:54.959 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-31T15:07:54.959 INFO:teuthology.orchestra.run.smithi016.stdout:] 2024-03-31T15:07:54.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:54.966+0000 7fe131800640 1 -- 172.21.15.16:0/1040112031 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fe108068380 msgr2=0x7fe10806a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:54.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:54.966+0000 7fe131800640 1 --2- 172.21.15.16:0/1040112031 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fe108068380 0x7fe10806a840 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fe11c0af000 tx=0x7fe11c0a7000 comp rx=0 tx=0).stop 2024-03-31T15:07:54.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:54.966+0000 7fe131800640 1 -- 172.21.15.16:0/1040112031 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe12c15fd40 msgr2=0x7fe12c14fef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:54.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:54.966+0000 7fe131800640 1 --2- 172.21.15.16:0/1040112031 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe12c15fd40 0x7fe12c14fef0 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7fe11400c420 tx=0x7fe11400c8e0 comp rx=0 tx=0).stop 2024-03-31T15:07:54.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:54.966+0000 7fe131800640 1 -- 172.21.15.16:0/1040112031 shutdown_connections 2024-03-31T15:07:54.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:54.966+0000 7fe131800640 1 --2- 172.21.15.16:0/1040112031 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fe108068380 0x7fe10806a840 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:54.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:54.966+0000 7fe131800640 1 --2- 172.21.15.16:0/1040112031 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe12c165f30 0x7fe12c155740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:54.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:54.966+0000 7fe131800640 1 --2- 172.21.15.16:0/1040112031 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe12c15fd40 0x7fe12c14fef0 unknown :-1 s=CLOSED pgs=809 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:54.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:54.966+0000 7fe131800640 1 --2- 172.21.15.16:0/1040112031 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fe12c15c1d0 0x7fe12c0c8fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:54.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:54.966+0000 7fe131800640 1 -- 172.21.15.16:0/1040112031 >> 172.21.15.16:0/1040112031 conn(0x7fe12c0b39c0 msgr2=0x7fe12c06fff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:07:54.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:54.966+0000 7fe131800640 1 -- 172.21.15.16:0/1040112031 shutdown_connections 2024-03-31T15:07:54.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:54.966+0000 7fe131800640 1 -- 172.21.15.16:0/1040112031 wait complete. 2024-03-31T15:07:54.979 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-31T15:07:54.979 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-31T15:07:55.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:55.130+0000 7f314e000640 1 Processor -- start 2024-03-31T15:07:55.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:55.130+0000 7f314e000640 1 -- start start 2024-03-31T15:07:55.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:55.130+0000 7f314e000640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f31480b22f0 0x7f31480730c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:55.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:55.130+0000 7f314e000640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f31480b45d0 0x7f31480b2f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:55.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:55.130+0000 7f314e000640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3148144320 con 0x7f3148143f40 2024-03-31T15:07:55.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:55.130+0000 7f314e000640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f31480b4a60 con 0x7f31480b45d0 2024-03-31T15:07:55.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:55.130+0000 7f314e000640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f314815ef80 con 0x7f31480b22f0 2024-03-31T15:07:55.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:55.130+0000 7f3147e00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f31480b45d0 0x7f31480b2f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:55.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:55.130+0000 7f3147400640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f3148143f40 0x7f31480b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:49340/0 (socket says 172.21.15.16:49340) 2024-03-31T15:07:55.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:55.130+0000 7f3147400640 1 -- 172.21.15.16:0/3255175856 learned_addr learned my addr 172.21.15.16:0/3255175856 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:07:55.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:55.130+0000 7f3146a00640 1 --2- 172.21.15.16:0/3255175856 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f31480b22f0 0x7f31480730c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:55.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:55.130+0000 7f3147e00640 1 -- 172.21.15.16:0/3255175856 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f31480b22f0 msgr2=0x7f31480730c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:07:55.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:55.130+0000 7f3147e00640 1 --2- 172.21.15.16:0/3255175856 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f31480b22f0 0x7f31480730c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:55.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:55.130+0000 7f3147e00640 1 -- 172.21.15.16:0/3255175856 >> v1:172.21.15.16:6789/0 conn(0x7f3148143f40 legacy=0x7f31480b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:55.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:55.130+0000 7f3147e00640 1 -- 172.21.15.16:0/3255175856 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f314815f260 con 0x7f31480b45d0 2024-03-31T15:07:55.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:55.130+0000 7f3146000640 1 -- 172.21.15.16:0/3255175856 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 814249854 0 0) 0x7f3148144320 con 0x7f3148143f40 2024-03-31T15:07:55.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:55.134+0000 7f3147e00640 1 --2- 172.21.15.16:0/3255175856 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f31480b45d0 0x7f31480b2f50 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7f3140009d90 tx=0x7f314001c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f8b1835cb9a0a36 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:07:55.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:55.134+0000 7f3146000640 1 -- 172.21.15.16:0/3255175856 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31400b2070 con 0x7f31480b45d0 2024-03-31T15:07:55.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:55.134+0000 7f3146000640 1 -- 172.21.15.16:0/3255175856 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f31400a5830 con 0x7f31480b45d0 2024-03-31T15:07:55.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:55.134+0000 7f3146000640 1 -- 172.21.15.16:0/3255175856 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31400a5c00 con 0x7f31480b45d0 2024-03-31T15:07:55.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:55.134+0000 7f314e000640 1 -- 172.21.15.16:0/3255175856 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f31480b45d0 msgr2=0x7f31480b2f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:55.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:55.134+0000 7f314e000640 1 --2- 172.21.15.16:0/3255175856 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f31480b45d0 0x7f31480b2f50 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7f3140009d90 tx=0x7f314001c970 comp rx=0 tx=0).stop 2024-03-31T15:07:55.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:55.134+0000 7f314e000640 1 -- 172.21.15.16:0/3255175856 shutdown_connections 2024-03-31T15:07:55.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:55.134+0000 7f314e000640 1 --2- 172.21.15.16:0/3255175856 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f31480b45d0 0x7f31480b2f50 unknown :-1 s=CLOSED pgs=811 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:55.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:55.134+0000 7f314e000640 1 --2- 172.21.15.16:0/3255175856 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f31480b22f0 0x7f31480730c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:55.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:55.134+0000 7f314e000640 1 -- 172.21.15.16:0/3255175856 >> 172.21.15.16:0/3255175856 conn(0x7f314806b620 msgr2=0x7f314806ba30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:07:55.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:55.134+0000 7f314e000640 1 -- 172.21.15.16:0/3255175856 shutdown_connections 2024-03-31T15:07:55.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:55.134+0000 7f314e000640 1 -- 172.21.15.16:0/3255175856 wait complete. 2024-03-31T15:07:55.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:55.134+0000 7f314e000640 1 Processor -- start 2024-03-31T15:07:55.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:55.138+0000 7f314e000640 1 -- start start 2024-03-31T15:07:55.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:55.138+0000 7f314e000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f31480b22f0 0x7f314815a730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:55.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:55.138+0000 7f314e000640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f31480b45d0 0x7f31480aeca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:55.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:55.138+0000 7f314e000640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f3148143f40 0x7f314813fd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:55.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:55.138+0000 7f314e000640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f31481680b0 con 0x7f31480b22f0 2024-03-31T15:07:55.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:55.138+0000 7f314e000640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f31480cd9a0 con 0x7f31480b45d0 2024-03-31T15:07:55.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:55.138+0000 7f314e000640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f3148168230 con 0x7f3148143f40 2024-03-31T15:07:55.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:55.138+0000 7f3147400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f31480b22f0 0x7f314815a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:55.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:55.138+0000 7f3147400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f31480b22f0 0x7f314815a730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38792/0 (socket says 172.21.15.16:38792) 2024-03-31T15:07:55.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:55.138+0000 7f3147400640 1 -- 172.21.15.16:0/2593209174 learned_addr learned my addr 172.21.15.16:0/2593209174 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:07:55.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:55.138+0000 7f3146a00640 1 --2- 172.21.15.16:0/2593209174 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f31480b45d0 0x7f31480aeca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:55.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:55.138+0000 7f3147e00640 1 --2- 172.21.15.16:0/2593209174 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f3148143f40 0x7f314813fd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:55.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:55.138+0000 7f3147400640 1 -- 172.21.15.16:0/2593209174 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f3148143f40 msgr2=0x7f314813fd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:55.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:55.138+0000 7f3147400640 1 --2- 172.21.15.16:0/2593209174 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f3148143f40 0x7f314813fd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:55.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:55.138+0000 7f3147400640 1 -- 172.21.15.16:0/2593209174 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f31480b45d0 msgr2=0x7f31480aeca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:55.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:55.138+0000 7f3147400640 1 --2- 172.21.15.16:0/2593209174 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f31480b45d0 0x7f31480aeca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:55.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:55.138+0000 7f3147400640 1 -- 172.21.15.16:0/2593209174 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f314813e110 con 0x7f31480b22f0 2024-03-31T15:07:55.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:55.138+0000 7f3146a00640 1 --2- 172.21.15.16:0/2593209174 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f31480b45d0 0x7f31480aeca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:07:55.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:55.138+0000 7f3147400640 1 --2- 172.21.15.16:0/2593209174 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f31480b22f0 0x7f314815a730 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7f31380011f0 tx=0x7f313800c9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:07:55.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:55.138+0000 7f313f400640 1 -- 172.21.15.16:0/2593209174 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3138018070 con 0x7f31480b22f0 2024-03-31T15:07:55.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:55.138+0000 7f313f400640 1 -- 172.21.15.16:0/2593209174 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f31380074e0 con 0x7f31480b22f0 2024-03-31T15:07:55.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:55.138+0000 7f313f400640 1 -- 172.21.15.16:0/2593209174 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3138004a10 con 0x7f31480b22f0 2024-03-31T15:07:55.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:55.138+0000 7f314e000640 1 -- 172.21.15.16:0/2593209174 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f314813b610 con 0x7f31480b22f0 2024-03-31T15:07:55.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:55.138+0000 7f314e000640 1 -- 172.21.15.16:0/2593209174 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f314813e3b0 con 0x7f31480b22f0 2024-03-31T15:07:55.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:55.138+0000 7f314e000640 1 -- 172.21.15.16:0/2593209174 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3148069950 con 0x7f31480b22f0 2024-03-31T15:07:55.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:55.142+0000 7f313f400640 1 -- 172.21.15.16:0/2593209174 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f313801b070 con 0x7f31480b22f0 2024-03-31T15:07:55.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:55.142+0000 7f313f400640 1 --2- 172.21.15.16:0/2593209174 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f3128068380 0x7f312806a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:55.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:55.142+0000 7f313f400640 1 -- 172.21.15.16:0/2593209174 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(63..63 src has 1..63) ==== 4857+0+0 (secure 0 0 0) 0x7f31380859a0 con 0x7f31480b22f0 2024-03-31T15:07:55.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:55.142+0000 7f3146a00640 1 --2- 172.21.15.16:0/2593209174 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f3128068380 0x7f312806a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:55.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:55.142+0000 7f3146a00640 1 --2- 172.21.15.16:0/2593209174 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f3128068380 0x7f312806a840 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f314813dda0 tx=0x7f3134008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:07:55.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:55.146+0000 7f313f400640 1 -- 172.21.15.16:0/2593209174 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f3138056290 con 0x7f31480b22f0 2024-03-31T15:07:55.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:55.314+0000 7f314e000640 1 -- 172.21.15.16:0/2593209174 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7f31480b3210 con 0x7f31480b22f0 2024-03-31T15:07:55.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:55.514+0000 7f313f400640 1 -- 172.21.15.16:0/2593209174 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v64) ==== 138+0+0 (secure 0 0 0) 0x7f3138055b90 con 0x7f31480b22f0 2024-03-31T15:07:55.509 INFO:teuthology.orchestra.run.smithi016.stderr:pool 'cephfs_metadata' created 2024-03-31T15:07:55.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:55.514+0000 7f314e000640 1 -- 172.21.15.16:0/2593209174 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f3128068380 msgr2=0x7f312806a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:55.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:55.514+0000 7f314e000640 1 --2- 172.21.15.16:0/2593209174 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f3128068380 0x7f312806a840 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f314813dda0 tx=0x7f3134008040 comp rx=0 tx=0).stop 2024-03-31T15:07:55.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:55.514+0000 7f314e000640 1 -- 172.21.15.16:0/2593209174 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f31480b22f0 msgr2=0x7f314815a730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:55.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:55.514+0000 7f314e000640 1 --2- 172.21.15.16:0/2593209174 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f31480b22f0 0x7f314815a730 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7f31380011f0 tx=0x7f313800c9b0 comp rx=0 tx=0).stop 2024-03-31T15:07:55.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:55.514+0000 7f314e000640 1 -- 172.21.15.16:0/2593209174 shutdown_connections 2024-03-31T15:07:55.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:55.514+0000 7f314e000640 1 --2- 172.21.15.16:0/2593209174 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f3128068380 0x7f312806a840 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:55.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:55.514+0000 7f314e000640 1 --2- 172.21.15.16:0/2593209174 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f3148143f40 0x7f314813fd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:55.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:55.514+0000 7f314e000640 1 --2- 172.21.15.16:0/2593209174 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f31480b45d0 0x7f31480aeca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:55.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:55.514+0000 7f314e000640 1 --2- 172.21.15.16:0/2593209174 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f31480b22f0 0x7f314815a730 unknown :-1 s=CLOSED pgs=812 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:55.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:55.514+0000 7f314e000640 1 -- 172.21.15.16:0/2593209174 >> 172.21.15.16:0/2593209174 conn(0x7f314806b620 msgr2=0x7f31481419e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:07:55.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:55.514+0000 7f314e000640 1 -- 172.21.15.16:0/2593209174 shutdown_connections 2024-03-31T15:07:55.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:55.514+0000 7f314e000640 1 -- 172.21.15.16:0/2593209174 wait complete. 2024-03-31T15:07:55.535 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-31T15:07:55.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:55.686+0000 7f9f0f200640 1 Processor -- start 2024-03-31T15:07:55.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:55.686+0000 7f9f0f200640 1 -- start start 2024-03-31T15:07:55.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:55.690+0000 7f9f0f200640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f9f1006c960 0x7f9f100d00a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:55.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:55.690+0000 7f9f0f200640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9f10162410 con 0x7f9f1006ecf0 2024-03-31T15:07:55.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:55.690+0000 7f9f0f200640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9f101626f0 con 0x7f9f10161e70 2024-03-31T15:07:55.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:55.690+0000 7f9f0f200640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f9f101629d0 con 0x7f9f1006c960 2024-03-31T15:07:55.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:55.690+0000 7f9f0e800640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f9f1006ecf0 0x7f9f10072910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:49342/0 (socket says 172.21.15.16:49342) 2024-03-31T15:07:55.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:55.690+0000 7f9f0e800640 1 -- 172.21.15.16:0/1696224880 learned_addr learned my addr 172.21.15.16:0/1696224880 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:07:55.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:55.690+0000 7f9f0de00640 1 --2- 172.21.15.16:0/1696224880 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f9f1006c960 0x7f9f100d00a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:55.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:55.690+0000 7f9f0d400640 1 -- 172.21.15.16:0/1696224880 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3447582915 0 0) 0x7f9f10162410 con 0x7f9f1006ecf0 2024-03-31T15:07:55.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:55.690+0000 7f9f0d400640 1 -- 172.21.15.16:0/1696224880 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9ef4003530 con 0x7f9f1006ecf0 2024-03-31T15:07:55.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:55.690+0000 7f9f0d400640 1 -- 172.21.15.16:0/1696224880 <== mon.1 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 693632245 0 0) 0x7f9f101626f0 con 0x7f9f10161e70 2024-03-31T15:07:55.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:55.690+0000 7f9f0d400640 1 -- 172.21.15.16:0/1696224880 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9f10162410 con 0x7f9f10161e70 2024-03-31T15:07:55.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:55.690+0000 7f9f0de00640 1 -- 172.21.15.16:0/1696224880 >> v1:172.21.15.89:6789/0 conn(0x7f9f10161e70 legacy=0x7f9f1006cd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:55.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:55.690+0000 7f9f0de00640 1 -- 172.21.15.16:0/1696224880 >> v1:172.21.15.16:6789/0 conn(0x7f9f1006ecf0 legacy=0x7f9f10072910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:55.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:55.690+0000 7f9f0de00640 1 -- 172.21.15.16:0/1696224880 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f10162cb0 con 0x7f9f1006c960 2024-03-31T15:07:55.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:55.694+0000 7f9f0de00640 1 --2- 172.21.15.16:0/1696224880 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f9f1006c960 0x7f9f100d00a0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f9f000097f0 tx=0x7f9f0001c4b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2d2df62c14425e5c server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:07:55.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:55.694+0000 7f9f0d400640 1 -- 172.21.15.16:0/1696224880 <== mon.2 v2:172.21.15.89:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f000aa430 con 0x7f9f1006c960 2024-03-31T15:07:55.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:55.694+0000 7f9f0d400640 1 -- 172.21.15.16:0/1696224880 <== mon.2 v2:172.21.15.89:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9f000af070 con 0x7f9f1006c960 2024-03-31T15:07:55.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:55.694+0000 7f9f0d400640 1 -- 172.21.15.16:0/1696224880 <== mon.2 v2:172.21.15.89:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f000aa7f0 con 0x7f9f1006c960 2024-03-31T15:07:55.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:55.694+0000 7f9f0f200640 1 -- 172.21.15.16:0/1696224880 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f9f1006c960 msgr2=0x7f9f100d00a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:55.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:55.694+0000 7f9f0f200640 1 --2- 172.21.15.16:0/1696224880 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f9f1006c960 0x7f9f100d00a0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f9f000097f0 tx=0x7f9f0001c4b0 comp rx=0 tx=0).stop 2024-03-31T15:07:55.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:55.694+0000 7f9f0f200640 1 -- 172.21.15.16:0/1696224880 shutdown_connections 2024-03-31T15:07:55.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:55.694+0000 7f9f0f200640 1 --2- 172.21.15.16:0/1696224880 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f9f1006c960 0x7f9f100d00a0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:55.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:55.694+0000 7f9f0f200640 1 -- 172.21.15.16:0/1696224880 >> 172.21.15.16:0/1696224880 conn(0x7f9f100b2c90 msgr2=0x7f9f100b30a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:07:55.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:55.694+0000 7f9f0f200640 1 -- 172.21.15.16:0/1696224880 shutdown_connections 2024-03-31T15:07:55.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:55.694+0000 7f9f0f200640 1 -- 172.21.15.16:0/1696224880 wait complete. 2024-03-31T15:07:55.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:55.694+0000 7f9f0f200640 1 Processor -- start 2024-03-31T15:07:55.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:55.694+0000 7f9f0f200640 1 -- start start 2024-03-31T15:07:55.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:55.698+0000 7f9f0f200640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9f1006c960 0x7f9f10163900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:55.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:55.698+0000 7f9f0f200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9f1006ecf0 0x7f9f10169a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:55.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:55.698+0000 7f9f0f200640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f9f10161e70 0x7f9f1015e7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:55.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:55.698+0000 7f9f0f200640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f9f1016ca30 con 0x7f9f1006ecf0 2024-03-31T15:07:55.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:55.698+0000 7f9f0f200640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f9f10073430 con 0x7f9f1006c960 2024-03-31T15:07:55.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:55.698+0000 7f9f0f200640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f9f1016cbb0 con 0x7f9f10161e70 2024-03-31T15:07:55.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:55.698+0000 7f9f05400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9f1006ecf0 0x7f9f10169a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:55.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:55.698+0000 7f9f05400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9f1006ecf0 0x7f9f10169a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38802/0 (socket says 172.21.15.16:38802) 2024-03-31T15:07:55.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:55.698+0000 7f9f05400640 1 -- 172.21.15.16:0/1504990059 learned_addr learned my addr 172.21.15.16:0/1504990059 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:07:55.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:55.698+0000 7f9f0de00640 1 --2- 172.21.15.16:0/1504990059 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9f1006c960 0x7f9f10163900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:55.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:55.698+0000 7f9f0e800640 1 --2- 172.21.15.16:0/1504990059 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f9f10161e70 0x7f9f1015e7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:55.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:55.698+0000 7f9f05400640 1 -- 172.21.15.16:0/1504990059 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f9f10161e70 msgr2=0x7f9f1015e7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:55.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:55.698+0000 7f9f05400640 1 --2- 172.21.15.16:0/1504990059 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f9f10161e70 0x7f9f1015e7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:55.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:55.698+0000 7f9f05400640 1 -- 172.21.15.16:0/1504990059 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9f1006c960 msgr2=0x7f9f10163900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:55.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:55.698+0000 7f9f05400640 1 --2- 172.21.15.16:0/1504990059 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9f1006c960 0x7f9f10163900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:55.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:55.698+0000 7f9f05400640 1 -- 172.21.15.16:0/1504990059 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f10143670 con 0x7f9f1006ecf0 2024-03-31T15:07:55.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:55.698+0000 7f9f0de00640 1 --2- 172.21.15.16:0/1504990059 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9f1006c960 0x7f9f10163900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:07:55.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:55.698+0000 7f9f05400640 1 --2- 172.21.15.16:0/1504990059 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9f1006ecf0 0x7f9f10169a30 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7f9ef800c430 tx=0x7f9ef800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:07:55.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:55.698+0000 7f9f06a00640 1 -- 172.21.15.16:0/1504990059 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ef8017070 con 0x7f9f1006ecf0 2024-03-31T15:07:55.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:55.698+0000 7f9f06a00640 1 -- 172.21.15.16:0/1504990059 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9ef80052f0 con 0x7f9f1006ecf0 2024-03-31T15:07:55.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:55.698+0000 7f9f0f200640 1 -- 172.21.15.16:0/1504990059 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9f10143950 con 0x7f9f1006ecf0 2024-03-31T15:07:55.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:55.698+0000 7f9f06a00640 1 -- 172.21.15.16:0/1504990059 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ef8003840 con 0x7f9f1006ecf0 2024-03-31T15:07:55.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:55.698+0000 7f9f0f200640 1 -- 172.21.15.16:0/1504990059 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9f10143f10 con 0x7f9f1006ecf0 2024-03-31T15:07:55.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:55.698+0000 7f9f0f200640 1 -- 172.21.15.16:0/1504990059 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9f1004f8c0 con 0x7f9f1006ecf0 2024-03-31T15:07:55.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:55.702+0000 7f9f06a00640 1 -- 172.21.15.16:0/1504990059 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f9ef801a070 con 0x7f9f1006ecf0 2024-03-31T15:07:55.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:55.702+0000 7f9f06a00640 1 --2- 172.21.15.16:0/1504990059 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f9eec068380 0x7f9eec06a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:55.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:55.702+0000 7f9f06a00640 1 -- 172.21.15.16:0/1504990059 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(64..64 src has 1..64) ==== 5227+0+0 (secure 0 0 0) 0x7f9ef8085c90 con 0x7f9f1006ecf0 2024-03-31T15:07:55.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:55.702+0000 7f9f0de00640 1 --2- 172.21.15.16:0/1504990059 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f9eec068380 0x7f9eec06a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:55.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:55.702+0000 7f9f0de00640 1 --2- 172.21.15.16:0/1504990059 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f9eec068380 0x7f9eec06a840 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f9f000ae000 tx=0x7f9f000a7000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:07:55.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:55.706+0000 7f9f06a00640 1 -- 172.21.15.16:0/1504990059 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f9ef8056390 con 0x7f9f1006ecf0 2024-03-31T15:07:55.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:55.870+0000 7f9f0f200640 1 -- 172.21.15.16:0/1504990059 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7f9f100ce410 con 0x7f9f1006ecf0 2024-03-31T15:07:56.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:56.522+0000 7f9f06a00640 1 -- 172.21.15.16:0/1504990059 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v65) ==== 170+0+0 (secure 0 0 0) 0x7f9ef8055c90 con 0x7f9f1006ecf0 2024-03-31T15:07:56.516 INFO:teuthology.orchestra.run.smithi016.stderr:pool 'cephfs_data' created 2024-03-31T15:07:56.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:56.522+0000 7f9f0f200640 1 -- 172.21.15.16:0/1504990059 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f9eec068380 msgr2=0x7f9eec06a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:56.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:56.522+0000 7f9f0f200640 1 --2- 172.21.15.16:0/1504990059 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f9eec068380 0x7f9eec06a840 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f9f000ae000 tx=0x7f9f000a7000 comp rx=0 tx=0).stop 2024-03-31T15:07:56.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:56.522+0000 7f9f0f200640 1 -- 172.21.15.16:0/1504990059 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9f1006ecf0 msgr2=0x7f9f10169a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:56.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:56.522+0000 7f9f0f200640 1 --2- 172.21.15.16:0/1504990059 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9f1006ecf0 0x7f9f10169a30 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7f9ef800c430 tx=0x7f9ef800c8f0 comp rx=0 tx=0).stop 2024-03-31T15:07:56.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:56.522+0000 7f9f0f200640 1 -- 172.21.15.16:0/1504990059 shutdown_connections 2024-03-31T15:07:56.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:56.522+0000 7f9f0f200640 1 --2- 172.21.15.16:0/1504990059 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f9eec068380 0x7f9eec06a840 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:56.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:56.522+0000 7f9f0f200640 1 --2- 172.21.15.16:0/1504990059 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f9f10161e70 0x7f9f1015e7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:56.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:56.522+0000 7f9f0f200640 1 --2- 172.21.15.16:0/1504990059 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9f1006ecf0 0x7f9f10169a30 unknown :-1 s=CLOSED pgs=814 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:56.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:56.522+0000 7f9f0f200640 1 --2- 172.21.15.16:0/1504990059 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9f1006c960 0x7f9f10163900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:56.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:56.522+0000 7f9f0f200640 1 -- 172.21.15.16:0/1504990059 >> 172.21.15.16:0/1504990059 conn(0x7f9f100b2c90 msgr2=0x7f9f1015db20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:07:56.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:56.526+0000 7f9f0f200640 1 -- 172.21.15.16:0/1504990059 shutdown_connections 2024-03-31T15:07:56.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:56.526+0000 7f9f0f200640 1 -- 172.21.15.16:0/1504990059 wait complete. 2024-03-31T15:07:56.557 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-31T15:07:56.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:56.706+0000 7fe882e00640 1 Processor -- start 2024-03-31T15:07:56.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:56.710+0000 7fe882e00640 1 -- start start 2024-03-31T15:07:56.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:56.710+0000 7fe882e00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fe87c15e6e0 0x7fe87c0b4f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:56.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:56.710+0000 7fe882e00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fe87c15eac0 0x7fe87c1513c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:56.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:56.710+0000 7fe882e00640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe87c0b34d0 con 0x7fe87c151900 2024-03-31T15:07:56.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:56.710+0000 7fe882e00640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7fe87c15f300 con 0x7fe87c15e6e0 2024-03-31T15:07:56.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:56.710+0000 7fe882e00640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7fe87c0b40e0 con 0x7fe87c15eac0 2024-03-31T15:07:56.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:56.710+0000 7fe87be00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fe87c15e6e0 0x7fe87c0b4f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:56.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:56.710+0000 7fe87be00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fe87c15e6e0 0x7fe87c0b4f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38810/0 (socket says 172.21.15.16:38810) 2024-03-31T15:07:56.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:56.710+0000 7fe87be00640 1 -- 172.21.15.16:0/1169246727 learned_addr learned my addr 172.21.15.16:0/1169246727 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:07:56.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:56.710+0000 7fe873e00640 1 --2- 172.21.15.16:0/1169246727 >> v2:172.21.15.89:3300/0 conn(0x7fe87c15eac0 0x7fe87c1513c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:56.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:56.710+0000 7fe87be00640 1 -- 172.21.15.16:0/1169246727 >> v2:172.21.15.89:3300/0 conn(0x7fe87c15eac0 msgr2=0x7fe87c1513c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:07:56.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:56.710+0000 7fe87be00640 1 --2- 172.21.15.16:0/1169246727 >> v2:172.21.15.89:3300/0 conn(0x7fe87c15eac0 0x7fe87c1513c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:56.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:56.710+0000 7fe87be00640 1 -- 172.21.15.16:0/1169246727 >> v1:172.21.15.89:6789/0 conn(0x7fe87c151900 legacy=0x7fe87c0cf2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:56.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:56.710+0000 7fe87be00640 1 -- 172.21.15.16:0/1169246727 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe87c0b37b0 con 0x7fe87c15e6e0 2024-03-31T15:07:56.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:56.714+0000 7fe87be00640 1 --2- 172.21.15.16:0/1169246727 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe87c15e6e0 0x7fe87c0b4f10 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7fe87c073a00 tx=0x7fe86c01c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e557ff2d53078764 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:07:56.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:56.714+0000 7fe87b400640 1 -- 172.21.15.16:0/1169246727 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe86c0b0070 con 0x7fe87c15e6e0 2024-03-31T15:07:56.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:56.714+0000 7fe87b400640 1 -- 172.21.15.16:0/1169246727 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe86c0ab600 con 0x7fe87c15e6e0 2024-03-31T15:07:56.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:56.714+0000 7fe87b400640 1 -- 172.21.15.16:0/1169246727 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe86c0ab9d0 con 0x7fe87c15e6e0 2024-03-31T15:07:56.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:56.714+0000 7fe882e00640 1 -- 172.21.15.16:0/1169246727 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe87c15e6e0 msgr2=0x7fe87c0b4f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:56.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:56.714+0000 7fe882e00640 1 --2- 172.21.15.16:0/1169246727 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe87c15e6e0 0x7fe87c0b4f10 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7fe87c073a00 tx=0x7fe86c01c520 comp rx=0 tx=0).stop 2024-03-31T15:07:56.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:56.714+0000 7fe882e00640 1 -- 172.21.15.16:0/1169246727 shutdown_connections 2024-03-31T15:07:56.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:56.714+0000 7fe882e00640 1 --2- 172.21.15.16:0/1169246727 >> v2:172.21.15.89:3300/0 conn(0x7fe87c15eac0 0x7fe87c1513c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:56.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:56.714+0000 7fe882e00640 1 --2- 172.21.15.16:0/1169246727 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe87c15e6e0 0x7fe87c0b4f10 unknown :-1 s=CLOSED pgs=815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:56.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:56.714+0000 7fe882e00640 1 -- 172.21.15.16:0/1169246727 >> 172.21.15.16:0/1169246727 conn(0x7fe87c06ee40 msgr2=0x7fe87c06b620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:07:56.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:56.714+0000 7fe882e00640 1 -- 172.21.15.16:0/1169246727 shutdown_connections 2024-03-31T15:07:56.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:56.714+0000 7fe882e00640 1 -- 172.21.15.16:0/1169246727 wait complete. 2024-03-31T15:07:56.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:56.714+0000 7fe882e00640 1 Processor -- start 2024-03-31T15:07:56.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:56.714+0000 7fe882e00640 1 -- start start 2024-03-31T15:07:56.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:56.718+0000 7fe882e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe87c151900 0x7fe87c14e270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:56.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:56.718+0000 7fe882e00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe87c15e6e0 0x7fe87c143c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:56.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:56.718+0000 7fe882e00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fe87c15eac0 0x7fe87c1555e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:56.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:56.718+0000 7fe882e00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fe87c0b32b0 con 0x7fe87c151900 2024-03-31T15:07:56.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:56.718+0000 7fe882e00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fe87c0b3130 con 0x7fe87c15e6e0 2024-03-31T15:07:56.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:56.718+0000 7fe882e00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fe87c0b5450 con 0x7fe87c15eac0 2024-03-31T15:07:56.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:56.718+0000 7fe87be00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe87c151900 0x7fe87c14e270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:56.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:56.718+0000 7fe87be00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe87c151900 0x7fe87c14e270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38820/0 (socket says 172.21.15.16:38820) 2024-03-31T15:07:56.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:56.718+0000 7fe87be00640 1 -- 172.21.15.16:0/1317496456 learned_addr learned my addr 172.21.15.16:0/1317496456 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:07:56.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:56.718+0000 7fe873e00640 1 --2- 172.21.15.16:0/1317496456 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe87c15e6e0 0x7fe87c143c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:56.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:56.718+0000 7fe880e00640 1 --2- 172.21.15.16:0/1317496456 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fe87c15eac0 0x7fe87c1555e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:56.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:56.718+0000 7fe87be00640 1 -- 172.21.15.16:0/1317496456 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fe87c15eac0 msgr2=0x7fe87c1555e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:56.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:56.718+0000 7fe87be00640 1 --2- 172.21.15.16:0/1317496456 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fe87c15eac0 0x7fe87c1555e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:56.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:56.718+0000 7fe87be00640 1 -- 172.21.15.16:0/1317496456 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe87c15e6e0 msgr2=0x7fe87c143c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:56.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:56.718+0000 7fe87be00640 1 --2- 172.21.15.16:0/1317496456 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe87c15e6e0 0x7fe87c143c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:56.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:56.718+0000 7fe87be00640 1 -- 172.21.15.16:0/1317496456 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe87c130340 con 0x7fe87c151900 2024-03-31T15:07:56.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:56.718+0000 7fe873e00640 1 --2- 172.21.15.16:0/1317496456 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe87c15e6e0 0x7fe87c143c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:07:56.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:56.718+0000 7fe87be00640 1 --2- 172.21.15.16:0/1317496456 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe87c151900 0x7fe87c14e270 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7fe86c0ae040 tx=0x7fe86c01ad60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:07:56.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:56.718+0000 7fe878c00640 1 -- 172.21.15.16:0/1317496456 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe86c0b0040 con 0x7fe87c151900 2024-03-31T15:07:56.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:56.718+0000 7fe878c00640 1 -- 172.21.15.16:0/1317496456 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe86c0bc070 con 0x7fe87c151900 2024-03-31T15:07:56.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:56.718+0000 7fe878c00640 1 -- 172.21.15.16:0/1317496456 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe86c0b6880 con 0x7fe87c151900 2024-03-31T15:07:56.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:56.718+0000 7fe882e00640 1 -- 172.21.15.16:0/1317496456 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe87c130620 con 0x7fe87c151900 2024-03-31T15:07:56.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:56.718+0000 7fe882e00640 1 -- 172.21.15.16:0/1317496456 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe87c130be0 con 0x7fe87c151900 2024-03-31T15:07:56.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:56.718+0000 7fe882e00640 1 -- 172.21.15.16:0/1317496456 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe87c069950 con 0x7fe87c151900 2024-03-31T15:07:56.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:56.722+0000 7fe878c00640 1 -- 172.21.15.16:0/1317496456 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fe86c0ab4c0 con 0x7fe87c151900 2024-03-31T15:07:56.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:56.722+0000 7fe878c00640 1 --2- 172.21.15.16:0/1317496456 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fe858068380 0x7fe85806a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:56.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:56.722+0000 7fe878c00640 1 -- 172.21.15.16:0/1317496456 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(65..65 src has 1..65) ==== 5609+0+0 (secure 0 0 0) 0x7fe86c120da0 con 0x7fe87c151900 2024-03-31T15:07:56.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:56.722+0000 7fe873e00640 1 --2- 172.21.15.16:0/1317496456 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fe858068380 0x7fe85806a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:56.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:56.722+0000 7fe873e00640 1 --2- 172.21.15.16:0/1317496456 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fe858068380 0x7fe85806a840 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fe87c149160 tx=0x7fe864008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:07:56.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:56.726+0000 7fe878c00640 1 -- 172.21.15.16:0/1317496456 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fe86c016110 con 0x7fe87c151900 2024-03-31T15:07:56.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:56.930+0000 7fe882e00640 1 -- 172.21.15.16:0/1317496456 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7fe87c0b47c0 con 0x7fe87c151900 2024-03-31T15:07:57.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:57.534+0000 7fe878c00640 1 -- 172.21.15.16:0/1317496456 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 10 and data pool 11 v41) ==== 174+0+0 (secure 0 0 0) 0x7fe86c0f1320 con 0x7fe87c151900 2024-03-31T15:07:57.532 INFO:teuthology.orchestra.run.smithi016.stderr:new fs with metadata pool 10 and data pool 11 2024-03-31T15:07:57.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:57.546+0000 7fe882e00640 1 -- 172.21.15.16:0/1317496456 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fe858068380 msgr2=0x7fe85806a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:57.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:57.546+0000 7fe882e00640 1 --2- 172.21.15.16:0/1317496456 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fe858068380 0x7fe85806a840 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fe87c149160 tx=0x7fe864008040 comp rx=0 tx=0).stop 2024-03-31T15:07:57.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:57.546+0000 7fe882e00640 1 -- 172.21.15.16:0/1317496456 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe87c151900 msgr2=0x7fe87c14e270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:57.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:57.546+0000 7fe882e00640 1 --2- 172.21.15.16:0/1317496456 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe87c151900 0x7fe87c14e270 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7fe86c0ae040 tx=0x7fe86c01ad60 comp rx=0 tx=0).stop 2024-03-31T15:07:57.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:57.546+0000 7fe882e00640 1 -- 172.21.15.16:0/1317496456 shutdown_connections 2024-03-31T15:07:57.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:57.546+0000 7fe882e00640 1 --2- 172.21.15.16:0/1317496456 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fe858068380 0x7fe85806a840 secure :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fe87c149160 tx=0x7fe864008040 comp rx=0 tx=0).stop 2024-03-31T15:07:57.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:57.546+0000 7fe882e00640 1 --2- 172.21.15.16:0/1317496456 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fe87c15eac0 0x7fe87c1555e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:57.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:57.546+0000 7fe882e00640 1 --2- 172.21.15.16:0/1317496456 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe87c15e6e0 0x7fe87c143c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:57.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:57.546+0000 7fe882e00640 1 --2- 172.21.15.16:0/1317496456 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe87c151900 0x7fe87c14e270 secure :-1 s=CLOSED pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7fe86c0ae040 tx=0x7fe86c01ad60 comp rx=0 tx=0).stop 2024-03-31T15:07:57.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:57.546+0000 7fe882e00640 1 -- 172.21.15.16:0/1317496456 >> 172.21.15.16:0/1317496456 conn(0x7fe87c06ee40 msgr2=0x7fe87c15d960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:07:57.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:57.546+0000 7fe882e00640 1 -- 172.21.15.16:0/1317496456 shutdown_connections 2024-03-31T15:07:57.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:57.546+0000 7fe882e00640 1 -- 172.21.15.16:0/1317496456 wait complete. 2024-03-31T15:07:57.576 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-31T15:07:57.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:57.722+0000 7f7092e00640 1 Processor -- start 2024-03-31T15:07:57.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:57.722+0000 7f7092e00640 1 -- start start 2024-03-31T15:07:57.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:57.722+0000 7f7092e00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f708c145d10 0x7f708c0cf790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:57.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:57.722+0000 7f7092e00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f708c0728b0 0x7f708c0b1cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:57.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:57.722+0000 7f7092e00640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f708c160e70 con 0x7f708c0b2200 2024-03-31T15:07:57.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:57.722+0000 7f7092e00640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f708c1602f0 con 0x7f708c0728b0 2024-03-31T15:07:57.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:57.722+0000 7f7092e00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f708c1605d0 con 0x7f708c145d10 2024-03-31T15:07:57.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:57.726+0000 7f7090e00640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f708c0b2200 0x7f708c15cd50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:34182/0 (socket says 172.21.15.16:34182) 2024-03-31T15:07:57.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:57.726+0000 7f7090e00640 1 -- 172.21.15.16:0/3582771244 learned_addr learned my addr 172.21.15.16:0/3582771244 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:07:57.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:57.726+0000 7f708be00640 1 --2- 172.21.15.16:0/3582771244 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f708c145d10 0x7f708c0cf790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:57.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:57.726+0000 7f7083400640 1 --2- 172.21.15.16:0/3582771244 >> v2:172.21.15.89:3300/0 conn(0x7f708c0728b0 0x7f708c0b1cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:57.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:57.726+0000 7f708b400640 1 -- 172.21.15.16:0/3582771244 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3222705396 0 0) 0x7f708c160e70 con 0x7f708c0b2200 2024-03-31T15:07:57.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:57.726+0000 7f708b400640 1 -- 172.21.15.16:0/3582771244 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7070003530 con 0x7f708c0b2200 2024-03-31T15:07:57.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:57.726+0000 7f708b400640 1 -- 172.21.15.16:0/3582771244 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3527939991 0 0) 0x7f7070003530 con 0x7f708c0b2200 2024-03-31T15:07:57.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:57.726+0000 7f708b400640 1 -- 172.21.15.16:0/3582771244 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f708c145d10 msgr2=0x7f708c0cf790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:07:57.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:57.726+0000 7f708b400640 1 --2- 172.21.15.16:0/3582771244 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f708c145d10 0x7f708c0cf790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:57.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:57.726+0000 7f708b400640 1 -- 172.21.15.16:0/3582771244 >> v2:172.21.15.89:3300/0 conn(0x7f708c0728b0 msgr2=0x7f708c0b1cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:07:57.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:57.726+0000 7f708b400640 1 --2- 172.21.15.16:0/3582771244 >> v2:172.21.15.89:3300/0 conn(0x7f708c0728b0 0x7f708c0b1cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:57.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:57.726+0000 7f708b400640 1 -- 172.21.15.16:0/3582771244 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f708c0cfcd0 con 0x7f708c0b2200 2024-03-31T15:07:57.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:57.726+0000 7f708b400640 1 -- 172.21.15.16:0/3582771244 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 214807623 0 0) 0x7f7084001a00 con 0x7f708c0b2200 2024-03-31T15:07:57.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:57.726+0000 7f708b400640 1 -- 172.21.15.16:0/3582771244 >> v1:172.21.15.16:6789/0 conn(0x7f708c0b2200 legacy=0x7f708c15cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:57.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:57.726+0000 7f708b400640 1 --2- 172.21.15.16:0/3582771244 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f70700039c0 0x7f7070003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:57.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:57.726+0000 7f708b400640 1 --2- 172.21.15.16:0/3582771244 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7070004810 0x7f7070004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:57.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:57.726+0000 7f708b400640 1 --2- 172.21.15.16:0/3582771244 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f708c145d10 0x7f70700051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:57.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:57.726+0000 7f708b400640 1 -- 172.21.15.16:0/3582771244 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f708c1602f0 con 0x7f70700039c0 2024-03-31T15:07:57.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:57.726+0000 7f708b400640 1 -- 172.21.15.16:0/3582771244 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f708c1605d0 con 0x7f7070004810 2024-03-31T15:07:57.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:57.726+0000 7f708b400640 1 -- 172.21.15.16:0/3582771244 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f707000b9d0 con 0x7f708c145d10 2024-03-31T15:07:57.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:57.726+0000 7f708be00640 1 --2- 172.21.15.16:0/3582771244 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f70700039c0 0x7f7070003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:57.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:57.726+0000 7f7083400640 1 --2- 172.21.15.16:0/3582771244 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7070004810 0x7f7070004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:57.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:57.726+0000 7f7090e00640 1 --2- 172.21.15.16:0/3582771244 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f708c145d10 0x7f70700051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:57.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:57.726+0000 7f708be00640 1 -- 172.21.15.16:0/3582771244 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f708c145d10 msgr2=0x7f70700051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:07:57.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:57.726+0000 7f708be00640 1 --2- 172.21.15.16:0/3582771244 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f708c145d10 0x7f70700051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:57.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:57.726+0000 7f708be00640 1 -- 172.21.15.16:0/3582771244 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7070004810 msgr2=0x7f7070004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:07:57.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:57.726+0000 7f708be00640 1 --2- 172.21.15.16:0/3582771244 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7070004810 0x7f7070004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:57.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:57.726+0000 7f708be00640 1 -- 172.21.15.16:0/3582771244 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7070003530 con 0x7f70700039c0 2024-03-31T15:07:57.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:57.726+0000 7f7090e00640 1 --2- 172.21.15.16:0/3582771244 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f708c145d10 0x7f70700051d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:07:57.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:57.730+0000 7f708be00640 1 --2- 172.21.15.16:0/3582771244 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f70700039c0 0x7f7070003de0 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7f707c009ae0 tx=0x7f707c01ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9746a93c630947b4 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:07:57.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:57.730+0000 7f708b400640 1 -- 172.21.15.16:0/3582771244 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f707c0096b0 con 0x7f70700039c0 2024-03-31T15:07:57.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:57.730+0000 7f708b400640 1 -- 172.21.15.16:0/3582771244 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f707c0a9470 con 0x7f70700039c0 2024-03-31T15:07:57.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:57.730+0000 7f708b400640 1 -- 172.21.15.16:0/3582771244 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f707c0a9860 con 0x7f70700039c0 2024-03-31T15:07:57.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:57.730+0000 7f7092e00640 1 -- 172.21.15.16:0/3582771244 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f70700039c0 msgr2=0x7f7070003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:57.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:57.730+0000 7f7092e00640 1 --2- 172.21.15.16:0/3582771244 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f70700039c0 0x7f7070003de0 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7f707c009ae0 tx=0x7f707c01ca10 comp rx=0 tx=0).stop 2024-03-31T15:07:57.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:57.730+0000 7f7090e00640 1 -- 172.21.15.16:0/3582771244 reap_dead start 2024-03-31T15:07:57.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:57.730+0000 7f7092e00640 1 -- 172.21.15.16:0/3582771244 shutdown_connections 2024-03-31T15:07:57.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:57.730+0000 7f7092e00640 1 -- 172.21.15.16:0/3582771244 >> 172.21.15.16:0/3582771244 conn(0x7f708c06b620 msgr2=0x7f708c06ba30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:07:57.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:57.730+0000 7f7092e00640 1 -- 172.21.15.16:0/3582771244 shutdown_connections 2024-03-31T15:07:57.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:57.730+0000 7f7092e00640 1 -- 172.21.15.16:0/3582771244 wait complete. 2024-03-31T15:07:57.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:57.730+0000 7f7092e00640 1 Processor -- start 2024-03-31T15:07:57.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:57.730+0000 7f7092e00640 1 -- start start 2024-03-31T15:07:57.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:57.734+0000 7f7092e00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f70700039c0 0x7f708c138a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:57.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:57.734+0000 7f7092e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f708c1630c0 0x7f708c163520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:57.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:57.734+0000 7f7092e00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f708c140e10 0x7f708c141290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:57.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:57.734+0000 7f7092e00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f708c15d870 con 0x7f708c1630c0 2024-03-31T15:07:57.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:57.734+0000 7f7092e00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f708c0b2610 con 0x7f708c140e10 2024-03-31T15:07:57.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:57.734+0000 7f7092e00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f708c0cf350 con 0x7f70700039c0 2024-03-31T15:07:57.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:57.734+0000 7f7083400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f708c1630c0 0x7f708c163520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:57.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:57.734+0000 7f7083400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f708c1630c0 0x7f708c163520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51712/0 (socket says 172.21.15.16:51712) 2024-03-31T15:07:57.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:57.734+0000 7f7083400640 1 -- 172.21.15.16:0/1278279978 learned_addr learned my addr 172.21.15.16:0/1278279978 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:07:57.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:57.734+0000 7f708be00640 1 --2- 172.21.15.16:0/1278279978 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f70700039c0 0x7f708c138a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:57.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:57.734+0000 7f7090e00640 1 --2- 172.21.15.16:0/1278279978 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f708c140e10 0x7f708c141290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:57.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:57.734+0000 7f7083400640 1 -- 172.21.15.16:0/1278279978 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f70700039c0 msgr2=0x7f708c138a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:57.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:57.734+0000 7f7083400640 1 --2- 172.21.15.16:0/1278279978 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f70700039c0 0x7f708c138a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:57.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:57.734+0000 7f7083400640 1 -- 172.21.15.16:0/1278279978 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f708c140e10 msgr2=0x7f708c141290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:57.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:57.734+0000 7f7083400640 1 --2- 172.21.15.16:0/1278279978 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f708c140e10 0x7f708c141290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:57.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:57.734+0000 7f7083400640 1 -- 172.21.15.16:0/1278279978 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f708c164be0 con 0x7f708c1630c0 2024-03-31T15:07:57.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:57.734+0000 7f7090e00640 1 --2- 172.21.15.16:0/1278279978 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f708c140e10 0x7f708c141290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:07:57.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:57.734+0000 7f7083400640 1 --2- 172.21.15.16:0/1278279978 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f708c1630c0 0x7f708c163520 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7f7074011ad0 tx=0x7f7074011f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:07:57.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:57.734+0000 7f7088c00640 1 -- 172.21.15.16:0/1278279978 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f707400fbb0 con 0x7f708c1630c0 2024-03-31T15:07:57.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:57.734+0000 7f7088c00640 1 -- 172.21.15.16:0/1278279978 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f707400fd50 con 0x7f708c1630c0 2024-03-31T15:07:57.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:57.734+0000 7f7092e00640 1 -- 172.21.15.16:0/1278279978 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f708c163b40 con 0x7f708c1630c0 2024-03-31T15:07:57.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:57.734+0000 7f7088c00640 1 -- 172.21.15.16:0/1278279978 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7074014680 con 0x7f708c1630c0 2024-03-31T15:07:57.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:57.734+0000 7f7092e00640 1 -- 172.21.15.16:0/1278279978 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f708c164100 con 0x7f708c1630c0 2024-03-31T15:07:57.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:57.734+0000 7f7092e00640 1 -- 172.21.15.16:0/1278279978 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f708c069950 con 0x7f708c1630c0 2024-03-31T15:07:57.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:57.734+0000 7f7088c00640 1 -- 172.21.15.16:0/1278279978 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f7074012040 con 0x7f708c1630c0 2024-03-31T15:07:57.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:57.738+0000 7f7088c00640 1 --2- 172.21.15.16:0/1278279978 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f7068068380 0x7f706806a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:57.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:57.738+0000 7f7088c00640 1 -- 172.21.15.16:0/1278279978 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(66..66 src has 1..66) ==== 5709+0+0 (secure 0 0 0) 0x7f70740870b0 con 0x7f708c1630c0 2024-03-31T15:07:57.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:57.738+0000 7f708be00640 1 --2- 172.21.15.16:0/1278279978 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f7068068380 0x7f706806a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:57.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:57.738+0000 7f708be00640 1 --2- 172.21.15.16:0/1278279978 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f7068068380 0x7f706806a840 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f707c0ae000 tx=0x7f707c0046d0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:07:57.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:57.742+0000 7f7088c00640 1 -- 172.21.15.16:0/1278279978 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f7074019030 con 0x7f708c1630c0 2024-03-31T15:07:57.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:57.906+0000 7f7092e00640 1 -- 172.21.15.16:0/1278279978 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f708c0aecb0 con 0x7f708c1630c0 2024-03-31T15:07:57.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:57.906+0000 7f7088c00640 1 -- 172.21.15.16:0/1278279978 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v66) ==== 74+0+16609 (secure 0 0 0) 0x7f70740575e0 con 0x7f708c1630c0 2024-03-31T15:07:57.901 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:07:57.901 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":66,"fsid":"286eb5da-fb47-4550-b290-d9519c13b9a5","created":"2024-03-31T15:03:11.438952+0000","modified":"2024-03-31T15:07:56.937824+0000","last_up_change":"2024-03-31T15:03:22.885535+0000","last_in_change":"2024-03-31T15:03:13.941667+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-31T15:03:22.550787+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-03-31T15:07:55.329096+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"66","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-03-31T15:07:55.885919+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"66","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"92a66450-ef19-40d8-ac3d-d760b23b0bda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6801","nonce":932712244}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6803","nonce":932712244}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6807","nonce":932712244}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6805","nonce":932712244}]},"public_addr":"172.21.15.16:6801/932712244","cluster_addr":"172.21.15.16:6803/932712244","heartbeat_back_addr":"172.21.15.16:6807/932712244","heartbeat_front_addr":"172.21.15.16:6805/932712244","state":["exists","up"]},{"osd":1,"uuid":"da9d1f2b-1968-4959-b843-faa231988e92","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6810","nonce":1398737533}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6811","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6813","nonce":1398737533}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6821","nonce":1398737533}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6815","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6816","nonce":1398737533}]},"public_addr":"172.21.15.16:6810/1398737533","cluster_addr":"172.21.15.16:6813/1398737533","heartbeat_back_addr":"172.21.15.16:6821/1398737533","heartbeat_front_addr":"172.21.15.16:6816/1398737533","state":["exists","up"]},{"osd":2,"uuid":"01a73662-9a21-4b1f-bcbb-42b9636a4638","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6819","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6822","nonce":3018526646}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6825","nonce":3018526646}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6831","nonce":3018526646}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6827","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6829","nonce":3018526646}]},"public_addr":"172.21.15.16:6822/3018526646","cluster_addr":"172.21.15.16:6825/3018526646","heartbeat_back_addr":"172.21.15.16:6831/3018526646","heartbeat_front_addr":"172.21.15.16:6829/3018526646","state":["exists","up"]},{"osd":3,"uuid":"c7771fd3-e18e-4500-ac38-23cf10ebcd75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6809","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6812","nonce":3360352974}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6817","nonce":3360352974}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6828","nonce":3360352974}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6823","nonce":3360352974}]},"public_addr":"172.21.15.16:6812/3360352974","cluster_addr":"172.21.15.16:6817/3360352974","heartbeat_back_addr":"172.21.15.16:6828/3360352974","heartbeat_front_addr":"172.21.15.16:6823/3360352974","state":["exists","up"]},{"osd":4,"uuid":"88260730-1482-4fbd-a40f-cb1124e9df4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6801","nonce":1856844856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6803","nonce":1856844856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6807","nonce":1856844856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6805","nonce":1856844856}]},"public_addr":"172.21.15.89:6801/1856844856","cluster_addr":"172.21.15.89:6803/1856844856","heartbeat_back_addr":"172.21.15.89:6807/1856844856","heartbeat_front_addr":"172.21.15.89:6805/1856844856","state":["exists","up"]},{"osd":5,"uuid":"4cb2c00c-fae0-4dd8-9d27-ec4f0093ccf7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6814","nonce":50395527}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6817","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6820","nonce":50395527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6831","nonce":50395527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6823","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6826","nonce":50395527}]},"public_addr":"172.21.15.89:6814/50395527","cluster_addr":"172.21.15.89:6820/50395527","heartbeat_back_addr":"172.21.15.89:6831/50395527","heartbeat_front_addr":"172.21.15.89:6826/50395527","state":["exists","up"]},{"osd":6,"uuid":"81ce0c19-2554-4cae-acbe-66a05e9e9b9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6809","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6812","nonce":3470471752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6815","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6818","nonce":3470471752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6827","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6829","nonce":3470471752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6821","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6824","nonce":3470471752}]},"public_addr":"172.21.15.89:6812/3470471752","cluster_addr":"172.21.15.89:6818/3470471752","heartbeat_back_addr":"172.21.15.89:6829/3470471752","heartbeat_front_addr":"172.21.15.89:6824/3470471752","state":["exists","up"]},{"osd":7,"uuid":"85e4313b-d37d-442e-8042-153bf20a847d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6811","nonce":1796778630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6813","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6816","nonce":1796778630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6825","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6828","nonce":1796778630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6819","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6822","nonce":1796778630}]},"public_addr":"172.21.15.89:6811/1796778630","cluster_addr":"172.21.15.89:6816/1796778630","heartbeat_back_addr":"172.21.15.89:6828/1796778630","heartbeat_front_addr":"172.21.15.89:6822/1796778630","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.439088+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.529970+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.680512+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:21.044161+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.582111+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.536558+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.409522+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.590229+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-31T15:07:57.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:57.906+0000 7f7092e00640 1 -- 172.21.15.16:0/1278279978 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f7068068380 msgr2=0x7f706806a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:57.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:57.906+0000 7f7092e00640 1 --2- 172.21.15.16:0/1278279978 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f7068068380 0x7f706806a840 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f707c0ae000 tx=0x7f707c0046d0 comp rx=0 tx=0).stop 2024-03-31T15:07:57.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:57.906+0000 7f7092e00640 1 -- 172.21.15.16:0/1278279978 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f708c1630c0 msgr2=0x7f708c163520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:57.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:57.906+0000 7f7092e00640 1 --2- 172.21.15.16:0/1278279978 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f708c1630c0 0x7f708c163520 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7f7074011ad0 tx=0x7f7074011f90 comp rx=0 tx=0).stop 2024-03-31T15:07:57.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:57.906+0000 7f7092e00640 1 -- 172.21.15.16:0/1278279978 shutdown_connections 2024-03-31T15:07:57.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:57.906+0000 7f7092e00640 1 --2- 172.21.15.16:0/1278279978 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f7068068380 0x7f706806a840 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:57.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:57.906+0000 7f7092e00640 1 --2- 172.21.15.16:0/1278279978 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f708c140e10 0x7f708c141290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:57.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:57.906+0000 7f7092e00640 1 --2- 172.21.15.16:0/1278279978 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f708c1630c0 0x7f708c163520 unknown :-1 s=CLOSED pgs=819 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:57.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:57.910+0000 7f7092e00640 1 --2- 172.21.15.16:0/1278279978 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f70700039c0 0x7f708c138a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:57.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:57.910+0000 7f7092e00640 1 -- 172.21.15.16:0/1278279978 >> 172.21.15.16:0/1278279978 conn(0x7f708c06b620 msgr2=0x7f708c0ad8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:07:57.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:57.910+0000 7f7092e00640 1 -- 172.21.15.16:0/1278279978 shutdown_connections 2024-03-31T15:07:57.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:57.910+0000 7f7092e00640 1 -- 172.21.15.16:0/1278279978 wait complete. 2024-03-31T15:07:57.923 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-31T15:07:58.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:58.066+0000 7f519a000640 1 Processor -- start 2024-03-31T15:07:58.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:58.066+0000 7f519a000640 1 -- start start 2024-03-31T15:07:58.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:58.070+0000 7f519a000640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f519415b3d0 0x7f51940b1cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:58.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:58.070+0000 7f519a000640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f519415eee0 0x7f51940b2200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:58.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:58.070+0000 7f519a000640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f519415f3c0 con 0x7f51940ce2f0 2024-03-31T15:07:58.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:58.070+0000 7f519a000640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f519415b7b0 con 0x7f519415b3d0 2024-03-31T15:07:58.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:58.070+0000 7f519a000640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f519415f6a0 con 0x7f519415eee0 2024-03-31T15:07:58.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:58.070+0000 7f5193e00640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f51940ce2f0 0x7f51940cc9c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:34186/0 (socket says 172.21.15.16:34186) 2024-03-31T15:07:58.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:58.070+0000 7f5193e00640 1 -- 172.21.15.16:0/2119569320 learned_addr learned my addr 172.21.15.16:0/2119569320 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:07:58.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:58.070+0000 7f5193400640 1 --2- 172.21.15.16:0/2119569320 >> v2:172.21.15.89:3300/0 conn(0x7f519415b3d0 0x7f51940b1cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:58.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:58.070+0000 7f518be00640 1 --2- 172.21.15.16:0/2119569320 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f519415eee0 0x7f51940b2200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:58.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:58.070+0000 7f5192a00640 1 -- 172.21.15.16:0/2119569320 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1424771019 0 0) 0x7f519415f3c0 con 0x7f51940ce2f0 2024-03-31T15:07:58.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:58.070+0000 7f5192a00640 1 -- 172.21.15.16:0/2119569320 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5178003530 con 0x7f51940ce2f0 2024-03-31T15:07:58.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:58.070+0000 7f5193400640 1 -- 172.21.15.16:0/2119569320 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f519415eee0 msgr2=0x7f51940b2200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:07:58.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:58.070+0000 7f5193400640 1 --2- 172.21.15.16:0/2119569320 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f519415eee0 0x7f51940b2200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:58.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:58.070+0000 7f5193400640 1 -- 172.21.15.16:0/2119569320 >> v1:172.21.15.16:6789/0 conn(0x7f51940ce2f0 legacy=0x7f51940cc9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:58.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:58.070+0000 7f5193400640 1 -- 172.21.15.16:0/2119569320 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f519415f980 con 0x7f519415b3d0 2024-03-31T15:07:58.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:58.070+0000 7f5193400640 1 --2- 172.21.15.16:0/2119569320 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f519415b3d0 0x7f51940b1cc0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f51840097c0 tx=0x7f518401c470 comp rx=0 tx=0).ready entity=mon.1 client_cookie=956aaf4d488db9b6 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:07:58.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:58.074+0000 7f5192a00640 1 -- 172.21.15.16:0/2119569320 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51840ab430 con 0x7f519415b3d0 2024-03-31T15:07:58.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:58.074+0000 7f5192a00640 1 -- 172.21.15.16:0/2119569320 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f51840b0070 con 0x7f519415b3d0 2024-03-31T15:07:58.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:58.074+0000 7f5192a00640 1 -- 172.21.15.16:0/2119569320 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51840ab7f0 con 0x7f519415b3d0 2024-03-31T15:07:58.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:58.074+0000 7f519a000640 1 -- 172.21.15.16:0/2119569320 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f519415b3d0 msgr2=0x7f51940b1cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:58.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:58.074+0000 7f519a000640 1 --2- 172.21.15.16:0/2119569320 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f519415b3d0 0x7f51940b1cc0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f51840097c0 tx=0x7f518401c470 comp rx=0 tx=0).stop 2024-03-31T15:07:58.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:58.074+0000 7f519a000640 1 -- 172.21.15.16:0/2119569320 shutdown_connections 2024-03-31T15:07:58.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:58.074+0000 7f519a000640 1 --2- 172.21.15.16:0/2119569320 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f519415eee0 0x7f51940b2200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:58.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:58.074+0000 7f519a000640 1 --2- 172.21.15.16:0/2119569320 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f519415b3d0 0x7f51940b1cc0 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:58.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:58.074+0000 7f519a000640 1 -- 172.21.15.16:0/2119569320 >> 172.21.15.16:0/2119569320 conn(0x7f519406b620 msgr2=0x7f519406ba30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:07:58.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:58.074+0000 7f519a000640 1 -- 172.21.15.16:0/2119569320 shutdown_connections 2024-03-31T15:07:58.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:58.074+0000 7f519a000640 1 -- 172.21.15.16:0/2119569320 wait complete. 2024-03-31T15:07:58.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:58.074+0000 7f519a000640 1 Processor -- start 2024-03-31T15:07:58.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:58.074+0000 7f519a000640 1 -- start start 2024-03-31T15:07:58.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:58.074+0000 7f519a000640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f51940ce2f0 0x7f5194151b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:58.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:58.074+0000 7f519a000640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f519415b3d0 0x7f51940d09a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:58.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:58.074+0000 7f519a000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f519415eee0 0x7f5194167280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:58.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:58.074+0000 7f519a000640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f5194166600 con 0x7f519415eee0 2024-03-31T15:07:58.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:58.074+0000 7f519a000640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f5194166480 con 0x7f519415b3d0 2024-03-31T15:07:58.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:58.074+0000 7f519a000640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f5194166780 con 0x7f51940ce2f0 2024-03-31T15:07:58.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:58.074+0000 7f5193e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f519415eee0 0x7f5194167280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:58.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:58.074+0000 7f5193e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f519415eee0 0x7f5194167280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51728/0 (socket says 172.21.15.16:51728) 2024-03-31T15:07:58.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:58.074+0000 7f5193e00640 1 -- 172.21.15.16:0/1474229323 learned_addr learned my addr 172.21.15.16:0/1474229323 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:07:58.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:58.078+0000 7f5193400640 1 --2- 172.21.15.16:0/1474229323 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f51940ce2f0 0x7f5194151b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:58.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:58.078+0000 7f518be00640 1 --2- 172.21.15.16:0/1474229323 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f519415b3d0 0x7f51940d09a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:58.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:58.078+0000 7f5193e00640 1 -- 172.21.15.16:0/1474229323 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f51940ce2f0 msgr2=0x7f5194151b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:58.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:58.078+0000 7f5193e00640 1 --2- 172.21.15.16:0/1474229323 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f51940ce2f0 0x7f5194151b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:58.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:58.078+0000 7f5193e00640 1 -- 172.21.15.16:0/1474229323 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f519415b3d0 msgr2=0x7f51940d09a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:58.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:58.078+0000 7f5193e00640 1 --2- 172.21.15.16:0/1474229323 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f519415b3d0 0x7f51940d09a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:58.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:58.078+0000 7f5193e00640 1 -- 172.21.15.16:0/1474229323 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5194152820 con 0x7f519415eee0 2024-03-31T15:07:58.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:58.078+0000 7f518be00640 1 --2- 172.21.15.16:0/1474229323 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f519415b3d0 0x7f51940d09a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:07:58.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:58.078+0000 7f5193e00640 1 --2- 172.21.15.16:0/1474229323 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f519415eee0 0x7f5194167280 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7f518c00cbe0 tx=0x7f518c00a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:07:58.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:58.078+0000 7f518b400640 1 -- 172.21.15.16:0/1474229323 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f518c017070 con 0x7f519415eee0 2024-03-31T15:07:58.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:58.078+0000 7f518b400640 1 -- 172.21.15.16:0/1474229323 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f518c00ad40 con 0x7f519415eee0 2024-03-31T15:07:58.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:58.078+0000 7f518b400640 1 -- 172.21.15.16:0/1474229323 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f518c012600 con 0x7f519415eee0 2024-03-31T15:07:58.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:58.078+0000 7f519a000640 1 -- 172.21.15.16:0/1474229323 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f51940d0ee0 con 0x7f519415eee0 2024-03-31T15:07:58.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:58.078+0000 7f519a000640 1 -- 172.21.15.16:0/1474229323 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5194152540 con 0x7f519415eee0 2024-03-31T15:07:58.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:58.078+0000 7f519a000640 1 -- 172.21.15.16:0/1474229323 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5194069950 con 0x7f519415eee0 2024-03-31T15:07:58.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:58.078+0000 7f518b400640 1 -- 172.21.15.16:0/1474229323 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f518c01a070 con 0x7f519415eee0 2024-03-31T15:07:58.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:58.078+0000 7f518b400640 1 --2- 172.21.15.16:0/1474229323 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f5170068380 0x7f517006a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:58.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:58.078+0000 7f518b400640 1 -- 172.21.15.16:0/1474229323 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(66..66 src has 1..66) ==== 5709+0+0 (secure 0 0 0) 0x7f518c0866d0 con 0x7f519415eee0 2024-03-31T15:07:58.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:58.082+0000 7f5193400640 1 --2- 172.21.15.16:0/1474229323 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f5170068380 0x7f517006a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:58.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:58.082+0000 7f5193400640 1 --2- 172.21.15.16:0/1474229323 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f5170068380 0x7f517006a840 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f51840af000 tx=0x7f5184008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:07:58.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:58.082+0000 7f518b400640 1 -- 172.21.15.16:0/1474229323 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f5194152820 con 0x7f519415eee0 2024-03-31T15:07:58.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:58.250+0000 7f519a000640 1 -- 172.21.15.16:0/1474229323 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f51940cdea0 con 0x7f519415eee0 2024-03-31T15:07:58.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:58.250+0000 7f518b400640 1 -- 172.21.15.16:0/1474229323 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v66) ==== 74+0+16609 (secure 0 0 0) 0x7f518c0565a0 con 0x7f519415eee0 2024-03-31T15:07:58.245 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:07:58.245 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":66,"fsid":"286eb5da-fb47-4550-b290-d9519c13b9a5","created":"2024-03-31T15:03:11.438952+0000","modified":"2024-03-31T15:07:56.937824+0000","last_up_change":"2024-03-31T15:03:22.885535+0000","last_in_change":"2024-03-31T15:03:13.941667+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-31T15:03:22.550787+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-03-31T15:07:55.329096+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"66","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-03-31T15:07:55.885919+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"66","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"92a66450-ef19-40d8-ac3d-d760b23b0bda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6801","nonce":932712244}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6803","nonce":932712244}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6807","nonce":932712244}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6805","nonce":932712244}]},"public_addr":"172.21.15.16:6801/932712244","cluster_addr":"172.21.15.16:6803/932712244","heartbeat_back_addr":"172.21.15.16:6807/932712244","heartbeat_front_addr":"172.21.15.16:6805/932712244","state":["exists","up"]},{"osd":1,"uuid":"da9d1f2b-1968-4959-b843-faa231988e92","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6810","nonce":1398737533}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6811","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6813","nonce":1398737533}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6821","nonce":1398737533}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6815","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6816","nonce":1398737533}]},"public_addr":"172.21.15.16:6810/1398737533","cluster_addr":"172.21.15.16:6813/1398737533","heartbeat_back_addr":"172.21.15.16:6821/1398737533","heartbeat_front_addr":"172.21.15.16:6816/1398737533","state":["exists","up"]},{"osd":2,"uuid":"01a73662-9a21-4b1f-bcbb-42b9636a4638","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6819","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6822","nonce":3018526646}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6825","nonce":3018526646}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6831","nonce":3018526646}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6827","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6829","nonce":3018526646}]},"public_addr":"172.21.15.16:6822/3018526646","cluster_addr":"172.21.15.16:6825/3018526646","heartbeat_back_addr":"172.21.15.16:6831/3018526646","heartbeat_front_addr":"172.21.15.16:6829/3018526646","state":["exists","up"]},{"osd":3,"uuid":"c7771fd3-e18e-4500-ac38-23cf10ebcd75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6809","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6812","nonce":3360352974}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6817","nonce":3360352974}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6828","nonce":3360352974}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6823","nonce":3360352974}]},"public_addr":"172.21.15.16:6812/3360352974","cluster_addr":"172.21.15.16:6817/3360352974","heartbeat_back_addr":"172.21.15.16:6828/3360352974","heartbeat_front_addr":"172.21.15.16:6823/3360352974","state":["exists","up"]},{"osd":4,"uuid":"88260730-1482-4fbd-a40f-cb1124e9df4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6801","nonce":1856844856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6803","nonce":1856844856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6807","nonce":1856844856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6805","nonce":1856844856}]},"public_addr":"172.21.15.89:6801/1856844856","cluster_addr":"172.21.15.89:6803/1856844856","heartbeat_back_addr":"172.21.15.89:6807/1856844856","heartbeat_front_addr":"172.21.15.89:6805/1856844856","state":["exists","up"]},{"osd":5,"uuid":"4cb2c00c-fae0-4dd8-9d27-ec4f0093ccf7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6814","nonce":50395527}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6817","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6820","nonce":50395527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6831","nonce":50395527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6823","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6826","nonce":50395527}]},"public_addr":"172.21.15.89:6814/50395527","cluster_addr":"172.21.15.89:6820/50395527","heartbeat_back_addr":"172.21.15.89:6831/50395527","heartbeat_front_addr":"172.21.15.89:6826/50395527","state":["exists","up"]},{"osd":6,"uuid":"81ce0c19-2554-4cae-acbe-66a05e9e9b9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6809","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6812","nonce":3470471752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6815","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6818","nonce":3470471752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6827","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6829","nonce":3470471752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6821","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6824","nonce":3470471752}]},"public_addr":"172.21.15.89:6812/3470471752","cluster_addr":"172.21.15.89:6818/3470471752","heartbeat_back_addr":"172.21.15.89:6829/3470471752","heartbeat_front_addr":"172.21.15.89:6824/3470471752","state":["exists","up"]},{"osd":7,"uuid":"85e4313b-d37d-442e-8042-153bf20a847d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6811","nonce":1796778630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6813","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6816","nonce":1796778630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6825","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6828","nonce":1796778630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6819","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6822","nonce":1796778630}]},"public_addr":"172.21.15.89:6811/1796778630","cluster_addr":"172.21.15.89:6816/1796778630","heartbeat_back_addr":"172.21.15.89:6828/1796778630","heartbeat_front_addr":"172.21.15.89:6822/1796778630","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.439088+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.529970+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.680512+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:21.044161+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.582111+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.536558+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.409522+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.590229+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-31T15:07:58.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:58.250+0000 7f519a000640 1 -- 172.21.15.16:0/1474229323 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f5170068380 msgr2=0x7f517006a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:58.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:58.250+0000 7f519a000640 1 --2- 172.21.15.16:0/1474229323 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f5170068380 0x7f517006a840 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f51840af000 tx=0x7f5184008040 comp rx=0 tx=0).stop 2024-03-31T15:07:58.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:58.250+0000 7f519a000640 1 -- 172.21.15.16:0/1474229323 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f519415eee0 msgr2=0x7f5194167280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:58.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:58.250+0000 7f519a000640 1 --2- 172.21.15.16:0/1474229323 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f519415eee0 0x7f5194167280 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7f518c00cbe0 tx=0x7f518c00a430 comp rx=0 tx=0).stop 2024-03-31T15:07:58.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:58.250+0000 7f519a000640 1 -- 172.21.15.16:0/1474229323 shutdown_connections 2024-03-31T15:07:58.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:58.250+0000 7f519a000640 1 --2- 172.21.15.16:0/1474229323 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f5170068380 0x7f517006a840 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:58.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:58.250+0000 7f519a000640 1 --2- 172.21.15.16:0/1474229323 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f519415eee0 0x7f5194167280 unknown :-1 s=CLOSED pgs=821 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:58.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:58.250+0000 7f519a000640 1 --2- 172.21.15.16:0/1474229323 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f519415b3d0 0x7f51940d09a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:58.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:58.250+0000 7f519a000640 1 --2- 172.21.15.16:0/1474229323 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f51940ce2f0 0x7f5194151b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:58.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:58.250+0000 7f519a000640 1 -- 172.21.15.16:0/1474229323 >> 172.21.15.16:0/1474229323 conn(0x7f519406b620 msgr2=0x7f519414f4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:07:58.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:58.250+0000 7f519a000640 1 -- 172.21.15.16:0/1474229323 shutdown_connections 2024-03-31T15:07:58.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:58.250+0000 7f519a000640 1 -- 172.21.15.16:0/1474229323 wait complete. 2024-03-31T15:07:58.267 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-31T15:07:58.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:58.414+0000 7f6ed7200640 1 Processor -- start 2024-03-31T15:07:58.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:58.414+0000 7f6ed7200640 1 -- start start 2024-03-31T15:07:58.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:58.418+0000 7f6ed7200640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f6ed006d4d0 0x7f6ed006d930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:58.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:58.418+0000 7f6ed7200640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f6ed0071550 0x7f6ed0071980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:58.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:58.418+0000 7f6ed7200640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6ed006de70 con 0x7f6ed01567e0 2024-03-31T15:07:58.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:58.418+0000 7f6ed7200640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f6ed015c7e0 con 0x7f6ed006d4d0 2024-03-31T15:07:58.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:58.418+0000 7f6ed7200640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f6ed00736b0 con 0x7f6ed0071550 2024-03-31T15:07:58.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:58.418+0000 7f6ec7400640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f6ed006d4d0 0x7f6ed006d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:58.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:58.418+0000 7f6ec7400640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f6ed006d4d0 0x7f6ed006d930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51736/0 (socket says 172.21.15.16:51736) 2024-03-31T15:07:58.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:58.418+0000 7f6ec7400640 1 -- 172.21.15.16:0/3068987249 learned_addr learned my addr 172.21.15.16:0/3068987249 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:07:58.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:58.418+0000 7f6ed5200640 1 --2- 172.21.15.16:0/3068987249 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f6ed0071550 0x7f6ed0071980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:58.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:58.418+0000 7f6ec7400640 1 -- 172.21.15.16:0/3068987249 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f6ed0071550 msgr2=0x7f6ed0071980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:07:58.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:58.418+0000 7f6ec7400640 1 --2- 172.21.15.16:0/3068987249 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f6ed0071550 0x7f6ed0071980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:58.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:58.418+0000 7f6ec7400640 1 -- 172.21.15.16:0/3068987249 >> v1:172.21.15.89:6789/0 conn(0x7f6ed01567e0 legacy=0x7f6ed0072c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:58.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:58.418+0000 7f6ec7400640 1 -- 172.21.15.16:0/3068987249 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ed015caa0 con 0x7f6ed006d4d0 2024-03-31T15:07:58.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:58.418+0000 7f6ec7400640 1 --2- 172.21.15.16:0/3068987249 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6ed006d4d0 0x7f6ed006d930 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7f6eb8009870 tx=0x7f6eb801cfc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e091f68a9b5ddc06 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:07:58.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:58.418+0000 7f6ecf400640 1 -- 172.21.15.16:0/3068987249 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6eb80a7070 con 0x7f6ed006d4d0 2024-03-31T15:07:58.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:58.422+0000 7f6ecf400640 1 -- 172.21.15.16:0/3068987249 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6eb80b0070 con 0x7f6ed006d4d0 2024-03-31T15:07:58.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:58.422+0000 7f6ecf400640 1 -- 172.21.15.16:0/3068987249 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6eb80ab890 con 0x7f6ed006d4d0 2024-03-31T15:07:58.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:58.422+0000 7f6ed7200640 1 -- 172.21.15.16:0/3068987249 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6ed006d4d0 msgr2=0x7f6ed006d930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:58.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:58.422+0000 7f6ed7200640 1 --2- 172.21.15.16:0/3068987249 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6ed006d4d0 0x7f6ed006d930 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7f6eb8009870 tx=0x7f6eb801cfc0 comp rx=0 tx=0).stop 2024-03-31T15:07:58.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:58.422+0000 7f6ed7200640 1 -- 172.21.15.16:0/3068987249 shutdown_connections 2024-03-31T15:07:58.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:58.422+0000 7f6ed7200640 1 --2- 172.21.15.16:0/3068987249 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f6ed0071550 0x7f6ed0071980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:58.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:58.422+0000 7f6ed7200640 1 --2- 172.21.15.16:0/3068987249 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6ed006d4d0 0x7f6ed006d930 unknown :-1 s=CLOSED pgs=822 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:58.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:58.422+0000 7f6ed7200640 1 -- 172.21.15.16:0/3068987249 >> 172.21.15.16:0/3068987249 conn(0x7f6ed006ee40 msgr2=0x7f6ed006b620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:07:58.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:58.422+0000 7f6ed7200640 1 -- 172.21.15.16:0/3068987249 shutdown_connections 2024-03-31T15:07:58.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:58.422+0000 7f6ed7200640 1 -- 172.21.15.16:0/3068987249 wait complete. 2024-03-31T15:07:58.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:58.422+0000 7f6ed7200640 1 Processor -- start 2024-03-31T15:07:58.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:58.422+0000 7f6ed7200640 1 -- start start 2024-03-31T15:07:58.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:58.422+0000 7f6ed7200640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f6ed006d4d0 0x7f6ed015e2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:58.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:58.422+0000 7f6ed7200640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6ed0071550 0x7f6ed0147f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:58.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:58.422+0000 7f6ed7200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6ed01567e0 0x7f6ed01484b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:58.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:58.422+0000 7f6ed7200640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f6ed00afbb0 con 0x7f6ed01567e0 2024-03-31T15:07:58.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:58.422+0000 7f6ed7200640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f6ed00afa30 con 0x7f6ed0071550 2024-03-31T15:07:58.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:58.422+0000 7f6ed7200640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f6ed00722d0 con 0x7f6ed006d4d0 2024-03-31T15:07:58.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:58.422+0000 7f6ed5200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6ed01567e0 0x7f6ed01484b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:58.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:58.422+0000 7f6ed5200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6ed01567e0 0x7f6ed01484b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51752/0 (socket says 172.21.15.16:51752) 2024-03-31T15:07:58.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:58.422+0000 7f6ed5200640 1 -- 172.21.15.16:0/1726268498 learned_addr learned my addr 172.21.15.16:0/1726268498 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:07:58.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:58.422+0000 7f6ecfe00640 1 --2- 172.21.15.16:0/1726268498 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f6ed006d4d0 0x7f6ed015e2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:58.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:58.422+0000 7f6ec7400640 1 --2- 172.21.15.16:0/1726268498 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6ed0071550 0x7f6ed0147f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:58.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:58.426+0000 7f6ed5200640 1 -- 172.21.15.16:0/1726268498 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f6ed006d4d0 msgr2=0x7f6ed015e2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:58.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:58.426+0000 7f6ed5200640 1 --2- 172.21.15.16:0/1726268498 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f6ed006d4d0 0x7f6ed015e2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:58.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:58.426+0000 7f6ed5200640 1 -- 172.21.15.16:0/1726268498 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6ed0071550 msgr2=0x7f6ed0147f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:58.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:58.426+0000 7f6ed5200640 1 --2- 172.21.15.16:0/1726268498 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6ed0071550 0x7f6ed0147f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:58.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:58.426+0000 7f6ed5200640 1 -- 172.21.15.16:0/1726268498 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ed01661e0 con 0x7f6ed01567e0 2024-03-31T15:07:58.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:58.426+0000 7f6ed5200640 1 --2- 172.21.15.16:0/1726268498 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6ed01567e0 0x7f6ed01484b0 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7f6ec8007e90 tx=0x7f6ec800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:07:58.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:58.426+0000 7f6eccc00640 1 -- 172.21.15.16:0/1726268498 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ec8014420 con 0x7f6ed01567e0 2024-03-31T15:07:58.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:58.426+0000 7f6eccc00640 1 -- 172.21.15.16:0/1726268498 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6ec8019070 con 0x7f6ed01567e0 2024-03-31T15:07:58.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:58.426+0000 7f6eccc00640 1 -- 172.21.15.16:0/1726268498 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ec8014800 con 0x7f6ed01567e0 2024-03-31T15:07:58.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:58.426+0000 7f6ed7200640 1 -- 172.21.15.16:0/1726268498 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6ed01664c0 con 0x7f6ed01567e0 2024-03-31T15:07:58.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:58.426+0000 7f6ed7200640 1 -- 172.21.15.16:0/1726268498 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6ed0166a80 con 0x7f6ed01567e0 2024-03-31T15:07:58.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:58.426+0000 7f6ed7200640 1 -- 172.21.15.16:0/1726268498 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6ed0069950 con 0x7f6ed01567e0 2024-03-31T15:07:58.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:58.426+0000 7f6eccc00640 1 -- 172.21.15.16:0/1726268498 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f6ec80149a0 con 0x7f6ed01567e0 2024-03-31T15:07:58.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:58.426+0000 7f6eccc00640 1 --2- 172.21.15.16:0/1726268498 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f6eac0683b0 0x7f6eac06a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:58.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:58.426+0000 7f6eccc00640 1 -- 172.21.15.16:0/1726268498 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(66..66 src has 1..66) ==== 5709+0+0 (secure 0 0 0) 0x7f6ec80870e0 con 0x7f6ed01567e0 2024-03-31T15:07:58.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:58.426+0000 7f6ecfe00640 1 --2- 172.21.15.16:0/1726268498 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f6eac0683b0 0x7f6eac06a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:58.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:58.430+0000 7f6ecfe00640 1 --2- 172.21.15.16:0/1726268498 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f6eac0683b0 0x7f6eac06a870 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f6ec0001810 tx=0x7f6ec0009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:07:58.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:58.430+0000 7f6eccc00640 1 -- 172.21.15.16:0/1726268498 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f6ec8056fc0 con 0x7f6ed01567e0 2024-03-31T15:07:58.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:58.634+0000 7f6ed7200640 1 -- 172.21.15.16:0/1726268498 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f6ed006e470 con 0x7f6ed01567e0 2024-03-31T15:07:59.506 INFO:tasks.ceph.mds.a.smithi016.stderr:2024-03-31T15:07:59.510+0000 7f3a3d400640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-31T15:07:59.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:59.550+0000 7f6eccc00640 1 -- 172.21.15.16:0/1726268498 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v44) ==== 118+0+0 (secure 0 0 0) 0x7f6ec805b2c0 con 0x7f6ed01567e0 2024-03-31T15:07:59.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:59.550+0000 7f6ed7200640 1 -- 172.21.15.16:0/1726268498 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f6eac0683b0 msgr2=0x7f6eac06a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:59.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:59.550+0000 7f6ed7200640 1 --2- 172.21.15.16:0/1726268498 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f6eac0683b0 0x7f6eac06a870 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f6ec0001810 tx=0x7f6ec0009040 comp rx=0 tx=0).stop 2024-03-31T15:07:59.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:59.550+0000 7f6ed7200640 1 -- 172.21.15.16:0/1726268498 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6ed01567e0 msgr2=0x7f6ed01484b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:59.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:59.550+0000 7f6ed7200640 1 --2- 172.21.15.16:0/1726268498 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6ed01567e0 0x7f6ed01484b0 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7f6ec8007e90 tx=0x7f6ec800e4b0 comp rx=0 tx=0).stop 2024-03-31T15:07:59.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:59.550+0000 7f6ed7200640 1 -- 172.21.15.16:0/1726268498 shutdown_connections 2024-03-31T15:07:59.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:59.550+0000 7f6ed7200640 1 --2- 172.21.15.16:0/1726268498 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f6eac0683b0 0x7f6eac06a870 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:59.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:59.550+0000 7f6ed7200640 1 --2- 172.21.15.16:0/1726268498 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6ed01567e0 0x7f6ed01484b0 unknown :-1 s=CLOSED pgs=823 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:59.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:59.550+0000 7f6ed7200640 1 --2- 172.21.15.16:0/1726268498 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6ed0071550 0x7f6ed0147f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:59.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:59.550+0000 7f6ed7200640 1 --2- 172.21.15.16:0/1726268498 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f6ed006d4d0 0x7f6ed015e2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:59.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:59.550+0000 7f6ed7200640 1 -- 172.21.15.16:0/1726268498 >> 172.21.15.16:0/1726268498 conn(0x7f6ed006ee40 msgr2=0x7f6ed00ce2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:07:59.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:59.554+0000 7f6ed7200640 1 -- 172.21.15.16:0/1726268498 shutdown_connections 2024-03-31T15:07:59.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:59.554+0000 7f6ed7200640 1 -- 172.21.15.16:0/1726268498 wait complete. 2024-03-31T15:07:59.567 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-03-31T15:07:59.567 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-31T15:07:59.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:59.714+0000 7fdd9c600640 1 Processor -- start 2024-03-31T15:07:59.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:59.714+0000 7fdd9c600640 1 -- start start 2024-03-31T15:07:59.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:59.714+0000 7fdd9c600640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fdd9406f920 0x7fdd9406fd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:59.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:59.714+0000 7fdd9c600640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fdd940d0a60 0x7fdd940cea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:59.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:59.714+0000 7fdd9c600640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdd94070240 con 0x7fdd940cef80 2024-03-31T15:07:59.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:59.714+0000 7fdd9c600640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7fdd9413f670 con 0x7fdd9406f920 2024-03-31T15:07:59.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:59.714+0000 7fdd9c600640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fdd9415ca70 con 0x7fdd940d0a60 2024-03-31T15:07:59.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:59.718+0000 7fdd99c00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fdd9406f920 0x7fdd9406fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:59.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:59.718+0000 7fdd99c00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fdd9406f920 0x7fdd9406fd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51754/0 (socket says 172.21.15.16:51754) 2024-03-31T15:07:59.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:59.718+0000 7fdd99c00640 1 -- 172.21.15.16:0/2438032759 learned_addr learned my addr 172.21.15.16:0/2438032759 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:07:59.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:59.718+0000 7fdd99200640 1 --2- 172.21.15.16:0/2438032759 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fdd940d0a60 0x7fdd940cea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:59.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:59.718+0000 7fdd99c00640 1 -- 172.21.15.16:0/2438032759 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fdd940d0a60 msgr2=0x7fdd940cea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:07:59.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:59.718+0000 7fdd99c00640 1 --2- 172.21.15.16:0/2438032759 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fdd940d0a60 0x7fdd940cea40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:59.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:59.718+0000 7fdd99c00640 1 -- 172.21.15.16:0/2438032759 >> v1:172.21.15.89:6789/0 conn(0x7fdd940cef80 legacy=0x7fdd94070e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:59.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:59.718+0000 7fdd99c00640 1 -- 172.21.15.16:0/2438032759 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd9415cd50 con 0x7fdd9406f920 2024-03-31T15:07:59.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:59.718+0000 7fdd99200640 1 --2- 172.21.15.16:0/2438032759 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fdd940d0a60 0x7fdd940cea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:07:59.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:59.722+0000 7fdd99c00640 1 --2- 172.21.15.16:0/2438032759 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdd9406f920 0x7fdd9406fd00 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7fdd88009870 tx=0x7fdd8801c560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f221cfc64b1c6d62 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:07:59.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:59.722+0000 7fdd93e00640 1 -- 172.21.15.16:0/2438032759 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd880b0070 con 0x7fdd9406f920 2024-03-31T15:07:59.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:59.722+0000 7fdd93e00640 1 -- 172.21.15.16:0/2438032759 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdd880ab600 con 0x7fdd9406f920 2024-03-31T15:07:59.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:59.722+0000 7fdd93e00640 1 -- 172.21.15.16:0/2438032759 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd880ab9d0 con 0x7fdd9406f920 2024-03-31T15:07:59.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:59.722+0000 7fdd9c600640 1 -- 172.21.15.16:0/2438032759 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdd9406f920 msgr2=0x7fdd9406fd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:59.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:59.722+0000 7fdd9c600640 1 --2- 172.21.15.16:0/2438032759 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdd9406f920 0x7fdd9406fd00 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7fdd88009870 tx=0x7fdd8801c560 comp rx=0 tx=0).stop 2024-03-31T15:07:59.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:59.722+0000 7fdd9c600640 1 -- 172.21.15.16:0/2438032759 shutdown_connections 2024-03-31T15:07:59.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:59.722+0000 7fdd9c600640 1 --2- 172.21.15.16:0/2438032759 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fdd940d0a60 0x7fdd940cea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:59.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:59.722+0000 7fdd9c600640 1 --2- 172.21.15.16:0/2438032759 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdd9406f920 0x7fdd9406fd00 unknown :-1 s=CLOSED pgs=824 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:59.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:59.722+0000 7fdd9c600640 1 -- 172.21.15.16:0/2438032759 >> 172.21.15.16:0/2438032759 conn(0x7fdd94067930 msgr2=0x7fdd940b3ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:07:59.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:59.722+0000 7fdd9c600640 1 -- 172.21.15.16:0/2438032759 shutdown_connections 2024-03-31T15:07:59.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:59.722+0000 7fdd9c600640 1 -- 172.21.15.16:0/2438032759 wait complete. 2024-03-31T15:07:59.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:59.726+0000 7fdd9c600640 1 Processor -- start 2024-03-31T15:07:59.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:59.726+0000 7fdd9c600640 1 -- start start 2024-03-31T15:07:59.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:59.726+0000 7fdd9c600640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fdd9406f920 0x7fdd9406c730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:59.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:59.726+0000 7fdd9c600640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdd940cef80 0x7fdd9413edd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:59.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:59.726+0000 7fdd9c600640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdd940d0a60 0x7fdd9406e2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:59.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:59.726+0000 7fdd9c600640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fdd941690b0 con 0x7fdd940d0a60 2024-03-31T15:07:59.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:59.726+0000 7fdd9c600640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fdd940cf390 con 0x7fdd940cef80 2024-03-31T15:07:59.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:59.726+0000 7fdd9c600640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fdd94169230 con 0x7fdd9406f920 2024-03-31T15:07:59.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:59.726+0000 7fdd9a600640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdd940d0a60 0x7fdd9406e2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:59.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:59.726+0000 7fdd9a600640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdd940d0a60 0x7fdd9406e2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51758/0 (socket says 172.21.15.16:51758) 2024-03-31T15:07:59.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:59.726+0000 7fdd9a600640 1 -- 172.21.15.16:0/3930297899 learned_addr learned my addr 172.21.15.16:0/3930297899 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:07:59.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:59.726+0000 7fdd99c00640 1 --2- 172.21.15.16:0/3930297899 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fdd9406f920 0x7fdd9406c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:59.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:59.726+0000 7fdd99200640 1 --2- 172.21.15.16:0/3930297899 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdd940cef80 0x7fdd9413edd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:59.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:59.730+0000 7fdd9a600640 1 -- 172.21.15.16:0/3930297899 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fdd9406f920 msgr2=0x7fdd9406c730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:59.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:59.730+0000 7fdd9a600640 1 --2- 172.21.15.16:0/3930297899 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fdd9406f920 0x7fdd9406c730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:59.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:59.730+0000 7fdd9a600640 1 -- 172.21.15.16:0/3930297899 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdd940cef80 msgr2=0x7fdd9413edd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:07:59.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:59.730+0000 7fdd9a600640 1 --2- 172.21.15.16:0/3930297899 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdd940cef80 0x7fdd9413edd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:07:59.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:59.730+0000 7fdd9a600640 1 -- 172.21.15.16:0/3930297899 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd9406d6b0 con 0x7fdd940d0a60 2024-03-31T15:07:59.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:59.730+0000 7fdd9a600640 1 --2- 172.21.15.16:0/3930297899 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdd940d0a60 0x7fdd9406e2b0 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7fdd8c00c7a0 tx=0x7fdd8c00cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:07:59.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:59.730+0000 7fdd91600640 1 -- 172.21.15.16:0/3930297899 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd8c003b80 con 0x7fdd940d0a60 2024-03-31T15:07:59.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:59.730+0000 7fdd91600640 1 -- 172.21.15.16:0/3930297899 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdd8c003d20 con 0x7fdd940d0a60 2024-03-31T15:07:59.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:59.730+0000 7fdd91600640 1 -- 172.21.15.16:0/3930297899 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd8c00aa90 con 0x7fdd940d0a60 2024-03-31T15:07:59.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:59.730+0000 7fdd9c600640 1 -- 172.21.15.16:0/3930297899 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdd9406ce10 con 0x7fdd940d0a60 2024-03-31T15:07:59.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:59.730+0000 7fdd9c600640 1 -- 172.21.15.16:0/3930297899 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdd9406d3d0 con 0x7fdd940d0a60 2024-03-31T15:07:59.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:59.730+0000 7fdd9c600640 1 -- 172.21.15.16:0/3930297899 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdd94069950 con 0x7fdd940d0a60 2024-03-31T15:07:59.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:59.730+0000 7fdd91600640 1 -- 172.21.15.16:0/3930297899 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fdd8c00ac30 con 0x7fdd940d0a60 2024-03-31T15:07:59.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:59.734+0000 7fdd91600640 1 --2- 172.21.15.16:0/3930297899 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fdd740683b0 0x7fdd7406a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:07:59.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:59.734+0000 7fdd91600640 1 -- 172.21.15.16:0/3930297899 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(67..67 src has 1..67) ==== 5709+0+0 (secure 0 0 0) 0x7fdd8c086800 con 0x7fdd940d0a60 2024-03-31T15:07:59.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:59.734+0000 7fdd99c00640 1 --2- 172.21.15.16:0/3930297899 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fdd740683b0 0x7fdd7406a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:07:59.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:59.734+0000 7fdd99c00640 1 --2- 172.21.15.16:0/3930297899 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fdd740683b0 0x7fdd7406a870 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fdd880ad000 tx=0x7fdd880a6000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:07:59.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:59.734+0000 7fdd91600640 1 -- 172.21.15.16:0/3930297899 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fdd8c0566d0 con 0x7fdd940d0a60 2024-03-31T15:07:59.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:07:59.934+0000 7fdd9c600640 1 -- 172.21.15.16:0/3930297899 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7fdd940ce7f0 con 0x7fdd940d0a60 2024-03-31T15:08:00.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:00.554+0000 7fdd91600640 1 -- 172.21.15.16:0/3930297899 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v45) ==== 122+0+0 (secure 0 0 0) 0x7fdd8c05a9d0 con 0x7fdd940d0a60 2024-03-31T15:08:00.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:00.558+0000 7fdd9c600640 1 -- 172.21.15.16:0/3930297899 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fdd740683b0 msgr2=0x7fdd7406a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:00.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:00.558+0000 7fdd9c600640 1 --2- 172.21.15.16:0/3930297899 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fdd740683b0 0x7fdd7406a870 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fdd880ad000 tx=0x7fdd880a6000 comp rx=0 tx=0).stop 2024-03-31T15:08:00.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:00.558+0000 7fdd9c600640 1 -- 172.21.15.16:0/3930297899 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdd940d0a60 msgr2=0x7fdd9406e2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:00.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:00.558+0000 7fdd9c600640 1 --2- 172.21.15.16:0/3930297899 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdd940d0a60 0x7fdd9406e2b0 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7fdd8c00c7a0 tx=0x7fdd8c00cc60 comp rx=0 tx=0).stop 2024-03-31T15:08:00.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:00.558+0000 7fdd9c600640 1 -- 172.21.15.16:0/3930297899 shutdown_connections 2024-03-31T15:08:00.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:00.558+0000 7fdd9c600640 1 --2- 172.21.15.16:0/3930297899 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fdd740683b0 0x7fdd7406a870 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:00.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:00.558+0000 7fdd9c600640 1 --2- 172.21.15.16:0/3930297899 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdd940d0a60 0x7fdd9406e2b0 unknown :-1 s=CLOSED pgs=825 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:00.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:00.558+0000 7fdd9c600640 1 --2- 172.21.15.16:0/3930297899 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdd940cef80 0x7fdd9413edd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:00.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:00.558+0000 7fdd9c600640 1 --2- 172.21.15.16:0/3930297899 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fdd9406f920 0x7fdd9406c730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:00.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:00.558+0000 7fdd9c600640 1 -- 172.21.15.16:0/3930297899 >> 172.21.15.16:0/3930297899 conn(0x7fdd94067930 msgr2=0x7fdd94154e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:08:00.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:00.558+0000 7fdd9c600640 1 -- 172.21.15.16:0/3930297899 shutdown_connections 2024-03-31T15:08:00.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:00.558+0000 7fdd9c600640 1 -- 172.21.15.16:0/3930297899 wait complete. 2024-03-31T15:08:00.572 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-31T15:08:00.572 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-31T15:08:00.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:00.718+0000 7f24eb000640 1 Processor -- start 2024-03-31T15:08:00.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:00.718+0000 7f24eb000640 1 -- start start 2024-03-31T15:08:00.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:00.718+0000 7f24eb000640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f24e4145e40 0x7f24e40cc740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:00.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:00.718+0000 7f24eb000640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f24e4070030 0x7f24e40d2010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:00.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:00.718+0000 7f24eb000640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f24e40d0920 con 0x7f24e40ccc80 2024-03-31T15:08:00.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:00.718+0000 7f24eb000640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f24e40d06d0 con 0x7f24e4070030 2024-03-31T15:08:00.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:00.718+0000 7f24eb000640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f24e40d0c00 con 0x7f24e4145e40 2024-03-31T15:08:00.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:00.718+0000 7f24db400640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f24e40ccc80 0x7f24e406f920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:34202/0 (socket says 172.21.15.16:34202) 2024-03-31T15:08:00.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:00.718+0000 7f24db400640 1 -- 172.21.15.16:0/1738995874 learned_addr learned my addr 172.21.15.16:0/1738995874 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:08:00.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:00.718+0000 7f24e9000640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f24e4070030 0x7f24e40d2010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:00.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:00.718+0000 7f24e3e00640 1 --2- 172.21.15.16:0/1738995874 >> v2:172.21.15.89:3300/0 conn(0x7f24e4145e40 0x7f24e40cc740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:00.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:00.722+0000 7f24e9000640 1 -- 172.21.15.16:0/1738995874 >> v2:172.21.15.89:3300/0 conn(0x7f24e4145e40 msgr2=0x7f24e40cc740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:08:00.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:00.722+0000 7f24e9000640 1 --2- 172.21.15.16:0/1738995874 >> v2:172.21.15.89:3300/0 conn(0x7f24e4145e40 0x7f24e40cc740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:00.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:00.722+0000 7f24e9000640 1 -- 172.21.15.16:0/1738995874 >> v1:172.21.15.16:6789/0 conn(0x7f24e40ccc80 legacy=0x7f24e406f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:00.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:00.722+0000 7f24e9000640 1 -- 172.21.15.16:0/1738995874 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24e40d1810 con 0x7f24e4070030 2024-03-31T15:08:00.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:00.722+0000 7f24e3e00640 1 --2- 172.21.15.16:0/1738995874 >> v2:172.21.15.89:3300/0 conn(0x7f24e4145e40 0x7f24e40cc740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:08:00.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:00.722+0000 7f24e9000640 1 --2- 172.21.15.16:0/1738995874 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f24e4070030 0x7f24e40d2010 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7f24e4072e60 tx=0x7f24dc01c930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58c6a3004ed5c14e server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:08:00.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:00.722+0000 7f24e3400640 1 -- 172.21.15.16:0/1738995874 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24dc0b2070 con 0x7f24e4070030 2024-03-31T15:08:00.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:00.722+0000 7f24e3400640 1 -- 172.21.15.16:0/1738995874 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f24dc0a5900 con 0x7f24e4070030 2024-03-31T15:08:00.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:00.722+0000 7f24e3400640 1 -- 172.21.15.16:0/1738995874 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24dc0a5cd0 con 0x7f24e4070030 2024-03-31T15:08:00.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:00.722+0000 7f24eb000640 1 -- 172.21.15.16:0/1738995874 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f24e4070030 msgr2=0x7f24e40d2010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:00.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:00.722+0000 7f24eb000640 1 --2- 172.21.15.16:0/1738995874 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f24e4070030 0x7f24e40d2010 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7f24e4072e60 tx=0x7f24dc01c930 comp rx=0 tx=0).stop 2024-03-31T15:08:00.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:00.726+0000 7f24eb000640 1 -- 172.21.15.16:0/1738995874 shutdown_connections 2024-03-31T15:08:00.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:00.726+0000 7f24eb000640 1 --2- 172.21.15.16:0/1738995874 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f24e4070030 0x7f24e40d2010 unknown :-1 s=CLOSED pgs=827 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:00.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:00.726+0000 7f24eb000640 1 --2- 172.21.15.16:0/1738995874 >> v2:172.21.15.89:3300/0 conn(0x7f24e4145e40 0x7f24e40cc740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:00.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:00.726+0000 7f24eb000640 1 -- 172.21.15.16:0/1738995874 >> 172.21.15.16:0/1738995874 conn(0x7f24e4067330 msgr2=0x7f24e4067740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:08:00.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:00.726+0000 7f24eb000640 1 -- 172.21.15.16:0/1738995874 shutdown_connections 2024-03-31T15:08:00.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:00.726+0000 7f24eb000640 1 -- 172.21.15.16:0/1738995874 wait complete. 2024-03-31T15:08:00.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:00.726+0000 7f24eb000640 1 Processor -- start 2024-03-31T15:08:00.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:00.726+0000 7f24eb000640 1 -- start start 2024-03-31T15:08:00.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:00.726+0000 7f24eb000640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f24e4070030 0x7f24e4143440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:00.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:00.726+0000 7f24eb000640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f24e40ccc80 0x7f24e4144610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:00.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:00.726+0000 7f24eb000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f24e4145e40 0x7f24e4154af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:00.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:00.726+0000 7f24eb000640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f24e40b5240 con 0x7f24e4145e40 2024-03-31T15:08:00.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:00.726+0000 7f24eb000640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f24e40b50c0 con 0x7f24e4070030 2024-03-31T15:08:00.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:00.726+0000 7f24eb000640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f24e4169190 con 0x7f24e40ccc80 2024-03-31T15:08:00.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:00.726+0000 7f24e9000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f24e4145e40 0x7f24e4154af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:00.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:00.726+0000 7f24e3e00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f24e4070030 0x7f24e4143440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:00.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:00.726+0000 7f24e9000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f24e4145e40 0x7f24e4154af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51774/0 (socket says 172.21.15.16:51774) 2024-03-31T15:08:00.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:00.726+0000 7f24e9000640 1 -- 172.21.15.16:0/864979448 learned_addr learned my addr 172.21.15.16:0/864979448 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:08:00.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:00.726+0000 7f24db400640 1 --2- 172.21.15.16:0/864979448 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f24e40ccc80 0x7f24e4144610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:00.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:00.726+0000 7f24e9000640 1 -- 172.21.15.16:0/864979448 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f24e40ccc80 msgr2=0x7f24e4144610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:00.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:00.726+0000 7f24e9000640 1 --2- 172.21.15.16:0/864979448 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f24e40ccc80 0x7f24e4144610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:00.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:00.726+0000 7f24e9000640 1 -- 172.21.15.16:0/864979448 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f24e4070030 msgr2=0x7f24e4143440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:00.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:00.726+0000 7f24e9000640 1 --2- 172.21.15.16:0/864979448 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f24e4070030 0x7f24e4143440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:00.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:00.726+0000 7f24e9000640 1 -- 172.21.15.16:0/864979448 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24e406de70 con 0x7f24e4145e40 2024-03-31T15:08:00.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:00.726+0000 7f24e3e00640 1 --2- 172.21.15.16:0/864979448 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f24e4070030 0x7f24e4143440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:08:00.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:00.726+0000 7f24e9000640 1 --2- 172.21.15.16:0/864979448 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f24e4145e40 0x7f24e4154af0 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7f24dc0b0040 tx=0x7f24dc0196d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:08:00.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:00.730+0000 7f24e0c00640 1 -- 172.21.15.16:0/864979448 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24dc0b2050 con 0x7f24e4145e40 2024-03-31T15:08:00.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:00.730+0000 7f24e0c00640 1 -- 172.21.15.16:0/864979448 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f24dc0be070 con 0x7f24e4145e40 2024-03-31T15:08:00.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:00.730+0000 7f24e0c00640 1 -- 172.21.15.16:0/864979448 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24dc0ada90 con 0x7f24e4145e40 2024-03-31T15:08:00.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:00.730+0000 7f24eb000640 1 -- 172.21.15.16:0/864979448 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f24e406e150 con 0x7f24e4145e40 2024-03-31T15:08:00.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:00.730+0000 7f24eb000640 1 -- 172.21.15.16:0/864979448 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f24e4144b50 con 0x7f24e4145e40 2024-03-31T15:08:00.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:00.730+0000 7f24eb000640 1 -- 172.21.15.16:0/864979448 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f24e4069950 con 0x7f24e4145e40 2024-03-31T15:08:00.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:00.730+0000 7f24e0c00640 1 -- 172.21.15.16:0/864979448 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f24dc0adc30 con 0x7f24e4145e40 2024-03-31T15:08:00.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:00.730+0000 7f24e0c00640 1 --2- 172.21.15.16:0/864979448 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f24c00683b0 0x7f24c006a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:00.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:00.730+0000 7f24e0c00640 1 -- 172.21.15.16:0/864979448 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(67..67 src has 1..67) ==== 5709+0+0 (secure 0 0 0) 0x7f24dc121440 con 0x7f24e4145e40 2024-03-31T15:08:00.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:00.730+0000 7f24e3e00640 1 --2- 172.21.15.16:0/864979448 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f24c00683b0 0x7f24c006a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:00.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:00.730+0000 7f24e3e00640 1 --2- 172.21.15.16:0/864979448 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f24c00683b0 0x7f24c006a870 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f24e40b7510 tx=0x7f24d4008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:08:00.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:00.734+0000 7f24e0c00640 1 -- 172.21.15.16:0/864979448 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f24dc0c7020 con 0x7f24e4145e40 2024-03-31T15:08:00.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:00.934+0000 7f24eb000640 1 -- 172.21.15.16:0/864979448 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f24e40cf3c0 con 0x7f24e4145e40 2024-03-31T15:08:00.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:00.934+0000 7f24e0c00640 1 -- 172.21.15.16:0/864979448 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 45 v45) ==== 94+0+5183 (secure 0 0 0) 0x7f24dc016520 con 0x7f24e4145e40 2024-03-31T15:08:00.930 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:08:00.931 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":45,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6836/4188793506","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":4188793506},{"type":"v1","addr":"172.21.15.89:6836","nonce":4188793506}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":40},{"gid":5942,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/4105657676","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6835","nonce":4105657676},{"type":"v1","addr":"172.21.15.89:6837","nonce":4105657676}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":40},{"gid":6592,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6833/3986752851","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":3986752851},{"type":"v1","addr":"172.21.15.16:6833","nonce":3986752851}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":39}],"filesystems":[{"mdsmap":{"epoch":45,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-31T15:07:56.938140+0000","modified":"2024-03-31T15:07:59.939362+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6714},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6714":{"gid":6714,"name":"a","rank":0,"incarnation":41,"state":"up:active","state_seq":3,"addr":"172.21.15.16:6835/2555930820","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":2555930820},{"type":"v1","addr":"172.21.15.16:6835","nonce":2555930820}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6714,"qdb_cluster":[6714]},"id":5}]} 2024-03-31T15:08:00.931 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 45 2024-03-31T15:08:00.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:00.938+0000 7f24eb000640 1 -- 172.21.15.16:0/864979448 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f24c00683b0 msgr2=0x7f24c006a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:00.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:00.938+0000 7f24eb000640 1 --2- 172.21.15.16:0/864979448 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f24c00683b0 0x7f24c006a870 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f24e40b7510 tx=0x7f24d4008040 comp rx=0 tx=0).stop 2024-03-31T15:08:00.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:00.938+0000 7f24eb000640 1 -- 172.21.15.16:0/864979448 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f24e4145e40 msgr2=0x7f24e4154af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:00.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:00.938+0000 7f24eb000640 1 --2- 172.21.15.16:0/864979448 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f24e4145e40 0x7f24e4154af0 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7f24dc0b0040 tx=0x7f24dc0196d0 comp rx=0 tx=0).stop 2024-03-31T15:08:00.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:00.938+0000 7f24eb000640 1 -- 172.21.15.16:0/864979448 shutdown_connections 2024-03-31T15:08:00.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:00.938+0000 7f24eb000640 1 --2- 172.21.15.16:0/864979448 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f24c00683b0 0x7f24c006a870 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:00.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:00.938+0000 7f24eb000640 1 --2- 172.21.15.16:0/864979448 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f24e4145e40 0x7f24e4154af0 unknown :-1 s=CLOSED pgs=828 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:00.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:00.938+0000 7f24eb000640 1 --2- 172.21.15.16:0/864979448 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f24e40ccc80 0x7f24e4144610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:00.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:00.938+0000 7f24eb000640 1 --2- 172.21.15.16:0/864979448 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f24e4070030 0x7f24e4143440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:00.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:00.938+0000 7f24eb000640 1 -- 172.21.15.16:0/864979448 >> 172.21.15.16:0/864979448 conn(0x7f24e4067330 msgr2=0x7f24e40cb440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:08:00.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:00.938+0000 7f24eb000640 1 -- 172.21.15.16:0/864979448 shutdown_connections 2024-03-31T15:08:00.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:00.938+0000 7f24eb000640 1 -- 172.21.15.16:0/864979448 wait complete. 2024-03-31T15:08:00.952 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-31T15:08:01.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:01.098+0000 7f9a24c00640 1 Processor -- start 2024-03-31T15:08:01.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:01.098+0000 7f9a24c00640 1 -- start start 2024-03-31T15:08:01.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:01.098+0000 7f9a24c00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f9a20145e40 0x7f9a200cc740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:01.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:01.098+0000 7f9a24c00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f9a20070030 0x7f9a200d2010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:01.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:01.098+0000 7f9a24c00640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9a200d0920 con 0x7f9a200ccc80 2024-03-31T15:08:01.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:01.098+0000 7f9a24c00640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f9a200d06d0 con 0x7f9a20070030 2024-03-31T15:08:01.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:01.098+0000 7f9a24c00640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f9a200d0c00 con 0x7f9a20145e40 2024-03-31T15:08:01.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:01.102+0000 7f9a1e800640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f9a20070030 0x7f9a200d2010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:01.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:01.102+0000 7f9a1e800640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f9a20070030 0x7f9a200d2010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51776/0 (socket says 172.21.15.16:51776) 2024-03-31T15:08:01.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:01.102+0000 7f9a1e800640 1 -- 172.21.15.16:0/3100695954 learned_addr learned my addr 172.21.15.16:0/3100695954 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:08:01.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:01.102+0000 7f9a1de00640 1 --2- 172.21.15.16:0/3100695954 >> v2:172.21.15.89:3300/0 conn(0x7f9a20145e40 0x7f9a200cc740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:01.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:01.102+0000 7f9a1e800640 1 -- 172.21.15.16:0/3100695954 >> v2:172.21.15.89:3300/0 conn(0x7f9a20145e40 msgr2=0x7f9a200cc740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:08:01.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:01.102+0000 7f9a1e800640 1 --2- 172.21.15.16:0/3100695954 >> v2:172.21.15.89:3300/0 conn(0x7f9a20145e40 0x7f9a200cc740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:01.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:01.102+0000 7f9a1e800640 1 -- 172.21.15.16:0/3100695954 >> v1:172.21.15.89:6789/0 conn(0x7f9a200ccc80 legacy=0x7f9a2006f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:01.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:01.102+0000 7f9a1e800640 1 -- 172.21.15.16:0/3100695954 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a200d1810 con 0x7f9a20070030 2024-03-31T15:08:01.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:01.102+0000 7f9a1de00640 1 --2- 172.21.15.16:0/3100695954 >> v2:172.21.15.89:3300/0 conn(0x7f9a20145e40 0x7f9a200cc740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:08:01.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:01.106+0000 7f9a1e800640 1 --2- 172.21.15.16:0/3100695954 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9a20070030 0x7f9a200d2010 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7f9a20072e60 tx=0x7f9a1801c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f272708ca4b038f0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:08:01.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:01.106+0000 7f9a1d400640 1 -- 172.21.15.16:0/3100695954 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a180a57d0 con 0x7f9a20070030 2024-03-31T15:08:01.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:01.106+0000 7f9a1d400640 1 -- 172.21.15.16:0/3100695954 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9a180a5970 con 0x7f9a20070030 2024-03-31T15:08:01.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:01.106+0000 7f9a1d400640 1 -- 172.21.15.16:0/3100695954 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a180a5d40 con 0x7f9a20070030 2024-03-31T15:08:01.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:01.106+0000 7f9a24c00640 1 -- 172.21.15.16:0/3100695954 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9a20070030 msgr2=0x7f9a200d2010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:01.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:01.106+0000 7f9a24c00640 1 --2- 172.21.15.16:0/3100695954 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9a20070030 0x7f9a200d2010 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7f9a20072e60 tx=0x7f9a1801c9f0 comp rx=0 tx=0).stop 2024-03-31T15:08:01.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:01.106+0000 7f9a24c00640 1 -- 172.21.15.16:0/3100695954 shutdown_connections 2024-03-31T15:08:01.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:01.106+0000 7f9a24c00640 1 --2- 172.21.15.16:0/3100695954 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9a20070030 0x7f9a200d2010 unknown :-1 s=CLOSED pgs=829 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:01.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:01.106+0000 7f9a24c00640 1 --2- 172.21.15.16:0/3100695954 >> v2:172.21.15.89:3300/0 conn(0x7f9a20145e40 0x7f9a200cc740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:01.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:01.106+0000 7f9a24c00640 1 -- 172.21.15.16:0/3100695954 >> 172.21.15.16:0/3100695954 conn(0x7f9a20067330 msgr2=0x7f9a20067740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:08:01.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:01.106+0000 7f9a24c00640 1 -- 172.21.15.16:0/3100695954 shutdown_connections 2024-03-31T15:08:01.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:01.106+0000 7f9a24c00640 1 -- 172.21.15.16:0/3100695954 wait complete. 2024-03-31T15:08:01.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:01.106+0000 7f9a24c00640 1 Processor -- start 2024-03-31T15:08:01.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:01.110+0000 7f9a24c00640 1 -- start start 2024-03-31T15:08:01.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:01.110+0000 7f9a24c00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9a20070030 0x7f9a2013ac90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:01.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:01.110+0000 7f9a24c00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f9a200ccc80 0x7f9a2006d250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:01.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:01.110+0000 7f9a24c00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9a20145e40 0x7f9a2013ddd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:01.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:01.110+0000 7f9a24c00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f9a200b5240 con 0x7f9a20145e40 2024-03-31T15:08:01.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:01.110+0000 7f9a24c00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f9a200b50c0 con 0x7f9a20070030 2024-03-31T15:08:01.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:01.110+0000 7f9a24c00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f9a20169190 con 0x7f9a200ccc80 2024-03-31T15:08:01.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:01.110+0000 7f9a1e800640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9a20145e40 0x7f9a2013ddd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:01.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:01.110+0000 7f9a1e800640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9a20145e40 0x7f9a2013ddd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51782/0 (socket says 172.21.15.16:51782) 2024-03-31T15:08:01.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:01.110+0000 7f9a1e800640 1 -- 172.21.15.16:0/2280130861 learned_addr learned my addr 172.21.15.16:0/2280130861 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:08:01.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:01.110+0000 7f9a1de00640 1 --2- 172.21.15.16:0/2280130861 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9a20070030 0x7f9a2013ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:01.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:01.110+0000 7f9a17e00640 1 --2- 172.21.15.16:0/2280130861 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f9a200ccc80 0x7f9a2006d250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:01.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:01.110+0000 7f9a1e800640 1 -- 172.21.15.16:0/2280130861 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f9a200ccc80 msgr2=0x7f9a2006d250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:01.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:01.110+0000 7f9a1e800640 1 --2- 172.21.15.16:0/2280130861 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f9a200ccc80 0x7f9a2006d250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:01.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:01.110+0000 7f9a1e800640 1 -- 172.21.15.16:0/2280130861 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9a20070030 msgr2=0x7f9a2013ac90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:01.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:01.110+0000 7f9a1e800640 1 --2- 172.21.15.16:0/2280130861 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9a20070030 0x7f9a2013ac90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:01.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:01.110+0000 7f9a1e800640 1 -- 172.21.15.16:0/2280130861 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a2013b1d0 con 0x7f9a20145e40 2024-03-31T15:08:01.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:01.110+0000 7f9a17e00640 1 --2- 172.21.15.16:0/2280130861 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f9a200ccc80 0x7f9a2006d250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:08:01.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:01.110+0000 7f9a1e800640 1 --2- 172.21.15.16:0/2280130861 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9a20145e40 0x7f9a2013ddd0 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7f9a18019600 tx=0x7f9a18019790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:08:01.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:01.110+0000 7f9a16000640 1 -- 172.21.15.16:0/2280130861 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a180b8670 con 0x7f9a20145e40 2024-03-31T15:08:01.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:01.110+0000 7f9a16000640 1 -- 172.21.15.16:0/2280130861 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9a180be070 con 0x7f9a20145e40 2024-03-31T15:08:01.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:01.110+0000 7f9a24c00640 1 -- 172.21.15.16:0/2280130861 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9a2013cf30 con 0x7f9a20145e40 2024-03-31T15:08:01.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:01.110+0000 7f9a16000640 1 -- 172.21.15.16:0/2280130861 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a180b8a50 con 0x7f9a20145e40 2024-03-31T15:08:01.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:01.110+0000 7f9a24c00640 1 -- 172.21.15.16:0/2280130861 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9a2013d1d0 con 0x7f9a20145e40 2024-03-31T15:08:01.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:01.110+0000 7f9a24c00640 1 -- 172.21.15.16:0/2280130861 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9a20069950 con 0x7f9a20145e40 2024-03-31T15:08:01.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:01.114+0000 7f9a16000640 1 -- 172.21.15.16:0/2280130861 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f9a180b8bf0 con 0x7f9a20145e40 2024-03-31T15:08:01.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:01.114+0000 7f9a16000640 1 --2- 172.21.15.16:0/2280130861 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f99fc068380 0x7f99fc06a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:01.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:01.114+0000 7f9a16000640 1 -- 172.21.15.16:0/2280130861 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(67..67 src has 1..67) ==== 5709+0+0 (secure 0 0 0) 0x7f9a18121ca0 con 0x7f9a20145e40 2024-03-31T15:08:01.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:01.114+0000 7f9a1de00640 1 --2- 172.21.15.16:0/2280130861 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f99fc068380 0x7f99fc06a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:01.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:01.114+0000 7f9a1de00640 1 --2- 172.21.15.16:0/2280130861 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f99fc068380 0x7f99fc06a840 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f9a10006e90 tx=0x7f9a10008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:08:01.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:01.118+0000 7f9a16000640 1 -- 172.21.15.16:0/2280130861 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f9a180165e0 con 0x7f9a20145e40 2024-03-31T15:08:01.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:01.294+0000 7f9a24c00640 1 -- 172.21.15.16:0/2280130861 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f9a2006d9c0 con 0x7f9a20145e40 2024-03-31T15:08:01.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:01.294+0000 7f9a16000640 1 -- 172.21.15.16:0/2280130861 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v67) ==== 74+0+16596 (secure 0 0 0) 0x7f9a180f2380 con 0x7f9a20145e40 2024-03-31T15:08:01.290 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:08:01.291 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":67,"fsid":"286eb5da-fb47-4550-b290-d9519c13b9a5","created":"2024-03-31T15:03:11.438952+0000","modified":"2024-03-31T15:07:58.527959+0000","last_up_change":"2024-03-31T15:03:22.885535+0000","last_in_change":"2024-03-31T15:03:13.941667+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-31T15:03:22.550787+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-03-31T15:07:55.329096+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"66","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-03-31T15:07:55.885919+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"67","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"92a66450-ef19-40d8-ac3d-d760b23b0bda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6801","nonce":932712244}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6803","nonce":932712244}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6807","nonce":932712244}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6805","nonce":932712244}]},"public_addr":"172.21.15.16:6801/932712244","cluster_addr":"172.21.15.16:6803/932712244","heartbeat_back_addr":"172.21.15.16:6807/932712244","heartbeat_front_addr":"172.21.15.16:6805/932712244","state":["exists","up"]},{"osd":1,"uuid":"da9d1f2b-1968-4959-b843-faa231988e92","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6810","nonce":1398737533}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6811","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6813","nonce":1398737533}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6821","nonce":1398737533}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6815","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6816","nonce":1398737533}]},"public_addr":"172.21.15.16:6810/1398737533","cluster_addr":"172.21.15.16:6813/1398737533","heartbeat_back_addr":"172.21.15.16:6821/1398737533","heartbeat_front_addr":"172.21.15.16:6816/1398737533","state":["exists","up"]},{"osd":2,"uuid":"01a73662-9a21-4b1f-bcbb-42b9636a4638","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6819","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6822","nonce":3018526646}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6825","nonce":3018526646}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6831","nonce":3018526646}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6827","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6829","nonce":3018526646}]},"public_addr":"172.21.15.16:6822/3018526646","cluster_addr":"172.21.15.16:6825/3018526646","heartbeat_back_addr":"172.21.15.16:6831/3018526646","heartbeat_front_addr":"172.21.15.16:6829/3018526646","state":["exists","up"]},{"osd":3,"uuid":"c7771fd3-e18e-4500-ac38-23cf10ebcd75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6809","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6812","nonce":3360352974}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6817","nonce":3360352974}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6828","nonce":3360352974}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6823","nonce":3360352974}]},"public_addr":"172.21.15.16:6812/3360352974","cluster_addr":"172.21.15.16:6817/3360352974","heartbeat_back_addr":"172.21.15.16:6828/3360352974","heartbeat_front_addr":"172.21.15.16:6823/3360352974","state":["exists","up"]},{"osd":4,"uuid":"88260730-1482-4fbd-a40f-cb1124e9df4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6801","nonce":1856844856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6803","nonce":1856844856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6807","nonce":1856844856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6805","nonce":1856844856}]},"public_addr":"172.21.15.89:6801/1856844856","cluster_addr":"172.21.15.89:6803/1856844856","heartbeat_back_addr":"172.21.15.89:6807/1856844856","heartbeat_front_addr":"172.21.15.89:6805/1856844856","state":["exists","up"]},{"osd":5,"uuid":"4cb2c00c-fae0-4dd8-9d27-ec4f0093ccf7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6814","nonce":50395527}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6817","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6820","nonce":50395527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6831","nonce":50395527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6823","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6826","nonce":50395527}]},"public_addr":"172.21.15.89:6814/50395527","cluster_addr":"172.21.15.89:6820/50395527","heartbeat_back_addr":"172.21.15.89:6831/50395527","heartbeat_front_addr":"172.21.15.89:6826/50395527","state":["exists","up"]},{"osd":6,"uuid":"81ce0c19-2554-4cae-acbe-66a05e9e9b9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6809","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6812","nonce":3470471752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6815","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6818","nonce":3470471752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6827","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6829","nonce":3470471752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6821","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6824","nonce":3470471752}]},"public_addr":"172.21.15.89:6812/3470471752","cluster_addr":"172.21.15.89:6818/3470471752","heartbeat_back_addr":"172.21.15.89:6829/3470471752","heartbeat_front_addr":"172.21.15.89:6824/3470471752","state":["exists","up"]},{"osd":7,"uuid":"85e4313b-d37d-442e-8042-153bf20a847d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6811","nonce":1796778630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6813","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6816","nonce":1796778630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6825","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6828","nonce":1796778630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6819","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6822","nonce":1796778630}]},"public_addr":"172.21.15.89:6811/1796778630","cluster_addr":"172.21.15.89:6816/1796778630","heartbeat_back_addr":"172.21.15.89:6828/1796778630","heartbeat_front_addr":"172.21.15.89:6822/1796778630","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.439088+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.529970+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.680512+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:21.044161+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.582111+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.536558+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.409522+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.590229+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-31T15:08:01.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:01.298+0000 7f9a24c00640 1 -- 172.21.15.16:0/2280130861 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f99fc068380 msgr2=0x7f99fc06a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:01.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:01.298+0000 7f9a24c00640 1 --2- 172.21.15.16:0/2280130861 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f99fc068380 0x7f99fc06a840 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f9a10006e90 tx=0x7f9a10008040 comp rx=0 tx=0).stop 2024-03-31T15:08:01.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:01.298+0000 7f9a24c00640 1 -- 172.21.15.16:0/2280130861 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9a20145e40 msgr2=0x7f9a2013ddd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:01.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:01.298+0000 7f9a24c00640 1 --2- 172.21.15.16:0/2280130861 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9a20145e40 0x7f9a2013ddd0 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7f9a18019600 tx=0x7f9a18019790 comp rx=0 tx=0).stop 2024-03-31T15:08:01.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:01.298+0000 7f9a24c00640 1 -- 172.21.15.16:0/2280130861 shutdown_connections 2024-03-31T15:08:01.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:01.298+0000 7f9a24c00640 1 --2- 172.21.15.16:0/2280130861 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f99fc068380 0x7f99fc06a840 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:01.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:01.298+0000 7f9a24c00640 1 --2- 172.21.15.16:0/2280130861 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9a20145e40 0x7f9a2013ddd0 unknown :-1 s=CLOSED pgs=830 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:01.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:01.298+0000 7f9a24c00640 1 --2- 172.21.15.16:0/2280130861 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f9a200ccc80 0x7f9a2006d250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:01.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:01.298+0000 7f9a24c00640 1 --2- 172.21.15.16:0/2280130861 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9a20070030 0x7f9a2013ac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:01.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:01.298+0000 7f9a24c00640 1 -- 172.21.15.16:0/2280130861 >> 172.21.15.16:0/2280130861 conn(0x7f9a20067330 msgr2=0x7f9a200cff60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:08:01.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:01.298+0000 7f9a24c00640 1 -- 172.21.15.16:0/2280130861 shutdown_connections 2024-03-31T15:08:01.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:01.298+0000 7f9a24c00640 1 -- 172.21.15.16:0/2280130861 wait complete. 2024-03-31T15:08:01.312 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-31T15:08:01.312 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-31T15:08:01.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:01.458+0000 7f684a000640 1 Processor -- start 2024-03-31T15:08:01.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:01.458+0000 7f684a000640 1 -- start start 2024-03-31T15:08:01.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:01.458+0000 7f684a000640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f6844145e40 0x7f68440cc740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:01.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:01.458+0000 7f684a000640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f68440d0920 con 0x7f6844070030 2024-03-31T15:08:01.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:01.458+0000 7f684a000640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f68440d0c00 con 0x7f68440ccc80 2024-03-31T15:08:01.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:01.458+0000 7f684a000640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f68440d06d0 con 0x7f6844145e40 2024-03-31T15:08:01.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:01.458+0000 7f6843e00640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f6844070030 0x7f68440d2010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:34214/0 (socket says 172.21.15.16:34214) 2024-03-31T15:08:01.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:01.458+0000 7f6843e00640 1 -- 172.21.15.16:0/547449424 learned_addr learned my addr 172.21.15.16:0/547449424 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:08:01.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:01.462+0000 7f6843400640 1 --2- 172.21.15.16:0/547449424 >> v2:172.21.15.89:3300/0 conn(0x7f6844145e40 0x7f68440cc740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:01.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:01.462+0000 7f6842000640 1 -- 172.21.15.16:0/547449424 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 86899200 0 0) 0x7f68440d0920 con 0x7f6844070030 2024-03-31T15:08:01.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:01.462+0000 7f6842000640 1 -- 172.21.15.16:0/547449424 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6828003530 con 0x7f6844070030 2024-03-31T15:08:01.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:01.462+0000 7f6842000640 1 -- 172.21.15.16:0/547449424 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2346616729 0 0) 0x7f6828003530 con 0x7f6844070030 2024-03-31T15:08:01.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:01.462+0000 7f6842000640 1 -- 172.21.15.16:0/547449424 >> v2:172.21.15.89:3300/0 conn(0x7f6844145e40 msgr2=0x7f68440cc740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:08:01.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:01.462+0000 7f6842000640 1 --2- 172.21.15.16:0/547449424 >> v2:172.21.15.89:3300/0 conn(0x7f6844145e40 0x7f68440cc740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:01.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:01.462+0000 7f6842000640 1 -- 172.21.15.16:0/547449424 >> v1:172.21.15.89:6789/0 conn(0x7f68440ccc80 legacy=0x7f684406f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:01.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:01.462+0000 7f6842000640 1 -- 172.21.15.16:0/547449424 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68440d1810 con 0x7f6844070030 2024-03-31T15:08:01.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:01.462+0000 7f6842000640 1 -- 172.21.15.16:0/547449424 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 214807623 0 0) 0x7f683c001a00 con 0x7f6844070030 2024-03-31T15:08:01.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:01.462+0000 7f6842000640 1 -- 172.21.15.16:0/547449424 >> v1:172.21.15.16:6789/0 conn(0x7f6844070030 legacy=0x7f68440d2010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:01.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:01.462+0000 7f6842000640 1 --2- 172.21.15.16:0/547449424 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f68280039c0 0x7f6828003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:01.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:01.462+0000 7f6842000640 1 --2- 172.21.15.16:0/547449424 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6828004810 0x7f6828004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:01.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:01.462+0000 7f6842000640 1 --2- 172.21.15.16:0/547449424 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f68280051d0 0x7f682800b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:01.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:01.462+0000 7f6842000640 1 -- 172.21.15.16:0/547449424 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f68440d06d0 con 0x7f6828004810 2024-03-31T15:08:01.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:01.462+0000 7f6842000640 1 -- 172.21.15.16:0/547449424 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f682800c020 con 0x7f68280051d0 2024-03-31T15:08:01.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:01.462+0000 7f6842000640 1 -- 172.21.15.16:0/547449424 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f682800c220 con 0x7f68280039c0 2024-03-31T15:08:01.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:01.462+0000 7f6842a00640 1 --2- 172.21.15.16:0/547449424 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6828004810 0x7f6828004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:01.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:01.462+0000 7f6843400640 1 --2- 172.21.15.16:0/547449424 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f68280039c0 0x7f6828003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:01.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:01.462+0000 7f6843e00640 1 --2- 172.21.15.16:0/547449424 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f68280051d0 0x7f682800b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:01.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:01.462+0000 7f6843400640 1 -- 172.21.15.16:0/547449424 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f68280051d0 msgr2=0x7f682800b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:08:01.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:01.462+0000 7f6843400640 1 --2- 172.21.15.16:0/547449424 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f68280051d0 0x7f682800b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:01.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:01.462+0000 7f6843400640 1 -- 172.21.15.16:0/547449424 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6828004810 msgr2=0x7f6828004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:08:01.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:01.462+0000 7f6843400640 1 --2- 172.21.15.16:0/547449424 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6828004810 0x7f6828004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:01.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:01.462+0000 7f6843400640 1 -- 172.21.15.16:0/547449424 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6828003530 con 0x7f68280039c0 2024-03-31T15:08:01.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:01.462+0000 7f6842a00640 1 --2- 172.21.15.16:0/547449424 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6828004810 0x7f6828004c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:08:01.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:01.462+0000 7f6843e00640 1 -- 172.21.15.16:0/547449424 reap_dead start 2024-03-31T15:08:01.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:01.466+0000 7f6843400640 1 --2- 172.21.15.16:0/547449424 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f68280039c0 0x7f6828003da0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f6834009ae0 tx=0x7f683401ca70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a97a521b9562897b server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:08:01.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:01.466+0000 7f6842000640 1 -- 172.21.15.16:0/547449424 <== mon.2 v2:172.21.15.89:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68340096b0 con 0x7f68280039c0 2024-03-31T15:08:01.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:01.466+0000 7f6842000640 1 -- 172.21.15.16:0/547449424 <== mon.2 v2:172.21.15.89:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f68340a6420 con 0x7f68280039c0 2024-03-31T15:08:01.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:01.466+0000 7f6842000640 1 -- 172.21.15.16:0/547449424 <== mon.2 v2:172.21.15.89:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68340a6810 con 0x7f68280039c0 2024-03-31T15:08:01.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:01.466+0000 7f684a000640 1 -- 172.21.15.16:0/547449424 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f68280039c0 msgr2=0x7f6828003da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:01.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:01.466+0000 7f684a000640 1 --2- 172.21.15.16:0/547449424 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f68280039c0 0x7f6828003da0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f6834009ae0 tx=0x7f683401ca70 comp rx=0 tx=0).stop 2024-03-31T15:08:01.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:01.466+0000 7f684a000640 1 -- 172.21.15.16:0/547449424 shutdown_connections 2024-03-31T15:08:01.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:01.466+0000 7f684a000640 1 --2- 172.21.15.16:0/547449424 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f68280039c0 0x7f6828003da0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:01.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:01.466+0000 7f684a000640 1 -- 172.21.15.16:0/547449424 >> 172.21.15.16:0/547449424 conn(0x7f6844067330 msgr2=0x7f6844067740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:08:01.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:01.466+0000 7f684a000640 1 -- 172.21.15.16:0/547449424 shutdown_connections 2024-03-31T15:08:01.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:01.466+0000 7f684a000640 1 -- 172.21.15.16:0/547449424 wait complete. 2024-03-31T15:08:01.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:01.466+0000 7f684a000640 1 Processor -- start 2024-03-31T15:08:01.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:01.466+0000 7f684a000640 1 -- start start 2024-03-31T15:08:01.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:01.470+0000 7f684a000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f68280039c0 0x7f6844155bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:01.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:01.470+0000 7f684a000640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f68441422a0 0x7f6844135520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:01.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:01.470+0000 7f684a000640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f6844142700 0x7f684415c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:01.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:01.470+0000 7f684a000640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f684406feb0 con 0x7f68280039c0 2024-03-31T15:08:01.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:01.470+0000 7f684a000640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f684406fd30 con 0x7f68441422a0 2024-03-31T15:08:01.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:01.470+0000 7f684a000640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f68440b50c0 con 0x7f6844142700 2024-03-31T15:08:01.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:01.470+0000 7f6843400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f68280039c0 0x7f6844155bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:01.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:01.470+0000 7f6843400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f68280039c0 0x7f6844155bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51800/0 (socket says 172.21.15.16:51800) 2024-03-31T15:08:01.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:01.470+0000 7f6843400640 1 -- 172.21.15.16:0/1827405806 learned_addr learned my addr 172.21.15.16:0/1827405806 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:08:01.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:01.470+0000 7f6842a00640 1 --2- 172.21.15.16:0/1827405806 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f68441422a0 0x7f6844135520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:01.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:01.470+0000 7f6843e00640 1 --2- 172.21.15.16:0/1827405806 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f6844142700 0x7f684415c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:01.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:01.470+0000 7f6843400640 1 -- 172.21.15.16:0/1827405806 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f6844142700 msgr2=0x7f684415c540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:01.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:01.470+0000 7f6843400640 1 --2- 172.21.15.16:0/1827405806 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f6844142700 0x7f684415c540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:01.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:01.470+0000 7f6843400640 1 -- 172.21.15.16:0/1827405806 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f68441422a0 msgr2=0x7f6844135520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:01.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:01.470+0000 7f6843400640 1 --2- 172.21.15.16:0/1827405806 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f68441422a0 0x7f6844135520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:01.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:01.470+0000 7f6843400640 1 -- 172.21.15.16:0/1827405806 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68440b3b80 con 0x7f68280039c0 2024-03-31T15:08:01.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:01.470+0000 7f6843400640 1 --2- 172.21.15.16:0/1827405806 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f68280039c0 0x7f6844155bb0 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7f6834009ae0 tx=0x7f6834019810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:08:01.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:01.470+0000 7f683b400640 1 -- 172.21.15.16:0/1827405806 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68340be070 con 0x7f68280039c0 2024-03-31T15:08:01.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:01.470+0000 7f683b400640 1 -- 172.21.15.16:0/1827405806 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f683400f000 con 0x7f68280039c0 2024-03-31T15:08:01.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:01.470+0000 7f684a000640 1 -- 172.21.15.16:0/1827405806 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f68441451d0 con 0x7f68280039c0 2024-03-31T15:08:01.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:01.470+0000 7f683b400640 1 -- 172.21.15.16:0/1827405806 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68340b9410 con 0x7f68280039c0 2024-03-31T15:08:01.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:01.470+0000 7f684a000640 1 -- 172.21.15.16:0/1827405806 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f684415b780 con 0x7f68280039c0 2024-03-31T15:08:01.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:01.470+0000 7f684a000640 1 -- 172.21.15.16:0/1827405806 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6844069950 con 0x7f68280039c0 2024-03-31T15:08:01.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:01.474+0000 7f683b400640 1 -- 172.21.15.16:0/1827405806 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f68340c1070 con 0x7f68280039c0 2024-03-31T15:08:01.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:01.474+0000 7f683b400640 1 --2- 172.21.15.16:0/1827405806 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f6824068380 0x7f682406a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:01.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:01.474+0000 7f683b400640 1 -- 172.21.15.16:0/1827405806 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(67..67 src has 1..67) ==== 5709+0+0 (secure 0 0 0) 0x7f6834120120 con 0x7f68280039c0 2024-03-31T15:08:01.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:01.474+0000 7f6842a00640 1 --2- 172.21.15.16:0/1827405806 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f6824068380 0x7f682406a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:01.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:01.474+0000 7f6842a00640 1 --2- 172.21.15.16:0/1827405806 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f6824068380 0x7f682406a840 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f68441f8bc0 tx=0x7f6830009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:08:01.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:01.478+0000 7f683b400640 1 -- 172.21.15.16:0/1827405806 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f68340ac020 con 0x7f68280039c0 2024-03-31T15:08:01.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:01.642+0000 7f684a000640 1 -- 172.21.15.16:0/1827405806 --> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f684415b250 con 0x7f6824068380 2024-03-31T15:08:01.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:01.650+0000 7f683b400640 1 -- 172.21.15.16:0/1827405806 <== mgr.4100 v2:172.21.15.89:6832/1549407273 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252025 (secure 0 0 0) 0x7f684415b250 con 0x7f6824068380 2024-03-31T15:08:01.648 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:08:01.650 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-03-31T15:08:01.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:01.654+0000 7f684a000640 1 -- 172.21.15.16:0/1827405806 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f6824068380 msgr2=0x7f682406a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:01.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:01.654+0000 7f684a000640 1 --2- 172.21.15.16:0/1827405806 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f6824068380 0x7f682406a840 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f68441f8bc0 tx=0x7f6830009040 comp rx=0 tx=0).stop 2024-03-31T15:08:01.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:01.654+0000 7f684a000640 1 -- 172.21.15.16:0/1827405806 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f68280039c0 msgr2=0x7f6844155bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:01.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:01.654+0000 7f684a000640 1 --2- 172.21.15.16:0/1827405806 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f68280039c0 0x7f6844155bb0 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7f6834009ae0 tx=0x7f6834019810 comp rx=0 tx=0).stop 2024-03-31T15:08:01.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:01.654+0000 7f684a000640 1 -- 172.21.15.16:0/1827405806 shutdown_connections 2024-03-31T15:08:01.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:01.654+0000 7f684a000640 1 --2- 172.21.15.16:0/1827405806 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f6824068380 0x7f682406a840 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:01.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:01.654+0000 7f684a000640 1 --2- 172.21.15.16:0/1827405806 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f6844142700 0x7f684415c540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:01.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:01.654+0000 7f684a000640 1 --2- 172.21.15.16:0/1827405806 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f68441422a0 0x7f6844135520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:01.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:01.654+0000 7f684a000640 1 --2- 172.21.15.16:0/1827405806 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f68280039c0 0x7f6844155bb0 unknown :-1 s=CLOSED pgs=832 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:01.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:01.654+0000 7f684a000640 1 -- 172.21.15.16:0/1827405806 >> 172.21.15.16:0/1827405806 conn(0x7f6844067330 msgr2=0x7f68440b5d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:08:01.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:01.654+0000 7f684a000640 1 -- 172.21.15.16:0/1827405806 shutdown_connections 2024-03-31T15:08:01.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:01.654+0000 7f684a000640 1 -- 172.21.15.16:0/1827405806 wait complete. 2024-03-31T15:08:01.668 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":199,"stamp":"2024-03-31T15:08:00.592877+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":248520,"kb_used_data":7348,"kb_used_omap":12,"kb_used_meta":241011,"kb_avail":754726200,"statfs":{"total":773094113280,"available":772839628800,"internally_reserved":0,"allocated":7524352,"data_stored":3810848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":246795824},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":32,"apply_latency_ms":32,"commit_latency_ns":32000000,"apply_latency_ns":32000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.072820"},"pg_stats":[{"pgid":"11.3f","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-31T15:07:57.569441+0000","last_change":"2024-03-31T15:07:57.563260+0000","last_active":"2024-03-31T15:07:57.569441+0000","last_peered":"2024-03-31T15:07:57.569441+0000","last_clean":"2024-03-31T15:07:57.569441+0000","last_became_active":"2024-03-31T15:07:57.562984+0000","last_became_peered":"2024-03-31T15:07:57.562984+0000","last_unstale":"2024-03-31T15:07:57.569441+0000","last_undegraded":"2024-03-31T15:07:57.569441+0000","last_fullsized":"2024-03-31T15:07:57.569441+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_clean_scrub_stamp":"2024-03-31T15:07:56.519376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:00:13.108819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.3e","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-31T15:07:57.546774+0000","last_change":"2024-03-31T15:07:57.542905+0000","last_active":"2024-03-31T15:07:57.546774+0000","last_peered":"2024-03-31T15:07:57.546774+0000","last_clean":"2024-03-31T15:07:57.546774+0000","last_became_active":"2024-03-31T15:07:57.542535+0000","last_became_peered":"2024-03-31T15:07:57.542535+0000","last_unstale":"2024-03-31T15:07:57.546774+0000","last_undegraded":"2024-03-31T15:07:57.546774+0000","last_fullsized":"2024-03-31T15:07:57.546774+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_clean_scrub_stamp":"2024-03-31T15:07:56.519376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:52:54.417267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.3d","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-31T15:07:57.563538+0000","last_change":"2024-03-31T15:07:57.559201+0000","last_active":"2024-03-31T15:07:57.563538+0000","last_peered":"2024-03-31T15:07:57.563538+0000","last_clean":"2024-03-31T15:07:57.563538+0000","last_became_active":"2024-03-31T15:07:57.556878+0000","last_became_peered":"2024-03-31T15:07:57.556878+0000","last_unstale":"2024-03-31T15:07:57.563538+0000","last_undegraded":"2024-03-31T15:07:57.563538+0000","last_fullsized":"2024-03-31T15:07:57.563538+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_clean_scrub_stamp":"2024-03-31T15:07:56.519376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:30:23.703431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.3c","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-31T15:07:57.559981+0000","last_change":"2024-03-31T15:07:57.556153+0000","last_active":"2024-03-31T15:07:57.559981+0000","last_peered":"2024-03-31T15:07:57.559981+0000","last_clean":"2024-03-31T15:07:57.559981+0000","last_became_active":"2024-03-31T15:07:57.555803+0000","last_became_peered":"2024-03-31T15:07:57.555803+0000","last_unstale":"2024-03-31T15:07:57.559981+0000","last_undegraded":"2024-03-31T15:07:57.559981+0000","last_fullsized":"2024-03-31T15:07:57.559981+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_clean_scrub_stamp":"2024-03-31T15:07:56.519376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T17:48:15.273565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.3b","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-31T15:07:58.532607+0000","last_change":"2024-03-31T15:07:57.552827+0000","last_active":"2024-03-31T15:07:58.532607+0000","last_peered":"2024-03-31T15:07:58.532607+0000","last_clean":"2024-03-31T15:07:58.532607+0000","last_became_active":"2024-03-31T15:07:57.552556+0000","last_became_peered":"2024-03-31T15:07:57.552556+0000","last_unstale":"2024-03-31T15:07:58.532607+0000","last_undegraded":"2024-03-31T15:07:58.532607+0000","last_fullsized":"2024-03-31T15:07:58.532607+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_clean_scrub_stamp":"2024-03-31T15:07:56.519376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T16:43:03.773114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3a","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-31T15:07:57.553594+0000","last_change":"2024-03-31T15:07:57.549624+0000","last_active":"2024-03-31T15:07:57.553594+0000","last_peered":"2024-03-31T15:07:57.553594+0000","last_clean":"2024-03-31T15:07:57.553594+0000","last_became_active":"2024-03-31T15:07:57.549335+0000","last_became_peered":"2024-03-31T15:07:57.549335+0000","last_unstale":"2024-03-31T15:07:57.553594+0000","last_undegraded":"2024-03-31T15:07:57.553594+0000","last_fullsized":"2024-03-31T15:07:57.553594+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_clean_scrub_stamp":"2024-03-31T15:07:56.519376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T16:05:52.009841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.39","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-31T15:07:58.555680+0000","last_change":"2024-03-31T15:07:57.560980+0000","last_active":"2024-03-31T15:07:58.555680+0000","last_peered":"2024-03-31T15:07:58.555680+0000","last_clean":"2024-03-31T15:07:58.555680+0000","last_became_active":"2024-03-31T15:07:57.560568+0000","last_became_peered":"2024-03-31T15:07:57.560568+0000","last_unstale":"2024-03-31T15:07:58.555680+0000","last_undegraded":"2024-03-31T15:07:58.555680+0000","last_fullsized":"2024-03-31T15:07:58.555680+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_clean_scrub_stamp":"2024-03-31T15:07:56.519376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:06:47.401497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.38","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-31T15:07:57.566663+0000","last_change":"2024-03-31T15:07:57.563352+0000","last_active":"2024-03-31T15:07:57.566663+0000","last_peered":"2024-03-31T15:07:57.566663+0000","last_clean":"2024-03-31T15:07:57.566663+0000","last_became_active":"2024-03-31T15:07:57.562814+0000","last_became_peered":"2024-03-31T15:07:57.562814+0000","last_unstale":"2024-03-31T15:07:57.566663+0000","last_undegraded":"2024-03-31T15:07:57.566663+0000","last_fullsized":"2024-03-31T15:07:57.566663+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_clean_scrub_stamp":"2024-03-31T15:07:56.519376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:20:41.809192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.37","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-31T15:07:58.532740+0000","last_change":"2024-03-31T15:07:57.550433+0000","last_active":"2024-03-31T15:07:58.532740+0000","last_peered":"2024-03-31T15:07:58.532740+0000","last_clean":"2024-03-31T15:07:58.532740+0000","last_became_active":"2024-03-31T15:07:57.549516+0000","last_became_peered":"2024-03-31T15:07:57.549516+0000","last_unstale":"2024-03-31T15:07:58.532740+0000","last_undegraded":"2024-03-31T15:07:58.532740+0000","last_fullsized":"2024-03-31T15:07:58.532740+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_clean_scrub_stamp":"2024-03-31T15:07:56.519376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:02:17.075008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.36","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-31T15:07:58.539664+0000","last_change":"2024-03-31T15:07:57.556497+0000","last_active":"2024-03-31T15:07:58.539664+0000","last_peered":"2024-03-31T15:07:58.539664+0000","last_clean":"2024-03-31T15:07:58.539664+0000","last_became_active":"2024-03-31T15:07:57.556238+0000","last_became_peered":"2024-03-31T15:07:57.556238+0000","last_unstale":"2024-03-31T15:07:58.539664+0000","last_undegraded":"2024-03-31T15:07:58.539664+0000","last_fullsized":"2024-03-31T15:07:58.539664+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_clean_scrub_stamp":"2024-03-31T15:07:56.519376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T16:08:00.122222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.35","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-31T15:07:57.569462+0000","last_change":"2024-03-31T15:07:57.564158+0000","last_active":"2024-03-31T15:07:57.569462+0000","last_peered":"2024-03-31T15:07:57.569462+0000","last_clean":"2024-03-31T15:07:57.569462+0000","last_became_active":"2024-03-31T15:07:57.563865+0000","last_became_peered":"2024-03-31T15:07:57.563865+0000","last_unstale":"2024-03-31T15:07:57.569462+0000","last_undegraded":"2024-03-31T15:07:57.569462+0000","last_fullsized":"2024-03-31T15:07:57.569462+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_clean_scrub_stamp":"2024-03-31T15:07:56.519376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T16:17:18.266469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.34","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-31T15:07:57.560026+0000","last_change":"2024-03-31T15:07:57.557329+0000","last_active":"2024-03-31T15:07:57.560026+0000","last_peered":"2024-03-31T15:07:57.560026+0000","last_clean":"2024-03-31T15:07:57.560026+0000","last_became_active":"2024-03-31T15:07:57.556873+0000","last_became_peered":"2024-03-31T15:07:57.556873+0000","last_unstale":"2024-03-31T15:07:57.560026+0000","last_undegraded":"2024-03-31T15:07:57.560026+0000","last_fullsized":"2024-03-31T15:07:57.560026+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_clean_scrub_stamp":"2024-03-31T15:07:56.519376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:40:17.597369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.33","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-31T15:07:57.551892+0000","last_change":"2024-03-31T15:07:57.547221+0000","last_active":"2024-03-31T15:07:57.551892+0000","last_peered":"2024-03-31T15:07:57.551892+0000","last_clean":"2024-03-31T15:07:57.551892+0000","last_became_active":"2024-03-31T15:07:57.546609+0000","last_became_peered":"2024-03-31T15:07:57.546609+0000","last_unstale":"2024-03-31T15:07:57.551892+0000","last_undegraded":"2024-03-31T15:07:57.551892+0000","last_fullsized":"2024-03-31T15:07:57.551892+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_clean_scrub_stamp":"2024-03-31T15:07:56.519376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:12:37.424035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.32","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-31T15:07:58.532459+0000","last_change":"2024-03-31T15:07:57.553238+0000","last_active":"2024-03-31T15:07:58.532459+0000","last_peered":"2024-03-31T15:07:58.532459+0000","last_clean":"2024-03-31T15:07:58.532459+0000","last_became_active":"2024-03-31T15:07:57.552991+0000","last_became_peered":"2024-03-31T15:07:57.552991+0000","last_unstale":"2024-03-31T15:07:58.532459+0000","last_undegraded":"2024-03-31T15:07:58.532459+0000","last_fullsized":"2024-03-31T15:07:58.532459+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_clean_scrub_stamp":"2024-03-31T15:07:56.519376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T17:32:16.673534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.31","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-31T15:07:57.550186+0000","last_change":"2024-03-31T15:07:57.544771+0000","last_active":"2024-03-31T15:07:57.550186+0000","last_peered":"2024-03-31T15:07:57.550186+0000","last_clean":"2024-03-31T15:07:57.550186+0000","last_became_active":"2024-03-31T15:07:57.544485+0000","last_became_peered":"2024-03-31T15:07:57.544485+0000","last_unstale":"2024-03-31T15:07:57.550186+0000","last_undegraded":"2024-03-31T15:07:57.550186+0000","last_fullsized":"2024-03-31T15:07:57.550186+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_clean_scrub_stamp":"2024-03-31T15:07:56.519376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:43:57.698889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.30","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-31T15:07:58.538552+0000","last_change":"2024-03-31T15:07:57.548359+0000","last_active":"2024-03-31T15:07:58.538552+0000","last_peered":"2024-03-31T15:07:58.538552+0000","last_clean":"2024-03-31T15:07:58.538552+0000","last_became_active":"2024-03-31T15:07:57.546175+0000","last_became_peered":"2024-03-31T15:07:57.546175+0000","last_unstale":"2024-03-31T15:07:58.538552+0000","last_undegraded":"2024-03-31T15:07:58.538552+0000","last_fullsized":"2024-03-31T15:07:58.538552+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_clean_scrub_stamp":"2024-03-31T15:07:56.519376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:43:39.968084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2f","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-31T15:07:58.532732+0000","last_change":"2024-03-31T15:07:57.550632+0000","last_active":"2024-03-31T15:07:58.532732+0000","last_peered":"2024-03-31T15:07:58.532732+0000","last_clean":"2024-03-31T15:07:58.532732+0000","last_became_active":"2024-03-31T15:07:57.549878+0000","last_became_peered":"2024-03-31T15:07:57.549878+0000","last_unstale":"2024-03-31T15:07:58.532732+0000","last_undegraded":"2024-03-31T15:07:58.532732+0000","last_fullsized":"2024-03-31T15:07:58.532732+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_clean_scrub_stamp":"2024-03-31T15:07:56.519376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:54:20.274859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.2e","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-31T15:07:57.559876+0000","last_change":"2024-03-31T15:07:57.555611+0000","last_active":"2024-03-31T15:07:57.559876+0000","last_peered":"2024-03-31T15:07:57.559876+0000","last_clean":"2024-03-31T15:07:57.559876+0000","last_became_active":"2024-03-31T15:07:57.555321+0000","last_became_peered":"2024-03-31T15:07:57.555321+0000","last_unstale":"2024-03-31T15:07:57.559876+0000","last_undegraded":"2024-03-31T15:07:57.559876+0000","last_fullsized":"2024-03-31T15:07:57.559876+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_clean_scrub_stamp":"2024-03-31T15:07:56.519376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T16:43:51.777380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.2d","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-31T15:07:57.564413+0000","last_change":"2024-03-31T15:07:57.559406+0000","last_active":"2024-03-31T15:07:57.564413+0000","last_peered":"2024-03-31T15:07:57.564413+0000","last_clean":"2024-03-31T15:07:57.564413+0000","last_became_active":"2024-03-31T15:07:57.556888+0000","last_became_peered":"2024-03-31T15:07:57.556888+0000","last_unstale":"2024-03-31T15:07:57.564413+0000","last_undegraded":"2024-03-31T15:07:57.564413+0000","last_fullsized":"2024-03-31T15:07:57.564413+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_clean_scrub_stamp":"2024-03-31T15:07:56.519376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:27:24.034975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.2c","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-31T15:07:57.551847+0000","last_change":"2024-03-31T15:07:57.546466+0000","last_active":"2024-03-31T15:07:57.551847+0000","last_peered":"2024-03-31T15:07:57.551847+0000","last_clean":"2024-03-31T15:07:57.551847+0000","last_became_active":"2024-03-31T15:07:57.546141+0000","last_became_peered":"2024-03-31T15:07:57.546141+0000","last_unstale":"2024-03-31T15:07:57.551847+0000","last_undegraded":"2024-03-31T15:07:57.551847+0000","last_fullsized":"2024-03-31T15:07:57.551847+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_clean_scrub_stamp":"2024-03-31T15:07:56.519376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T15:20:13.726100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2b","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-31T15:07:58.539818+0000","last_change":"2024-03-31T15:07:57.551433+0000","last_active":"2024-03-31T15:07:58.539818+0000","last_peered":"2024-03-31T15:07:58.539818+0000","last_clean":"2024-03-31T15:07:58.539818+0000","last_became_active":"2024-03-31T15:07:57.551139+0000","last_became_peered":"2024-03-31T15:07:57.551139+0000","last_unstale":"2024-03-31T15:07:58.539818+0000","last_undegraded":"2024-03-31T15:07:58.539818+0000","last_fullsized":"2024-03-31T15:07:58.539818+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_clean_scrub_stamp":"2024-03-31T15:07:56.519376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:05:26.269858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2a","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-31T15:07:58.538204+0000","last_change":"2024-03-31T15:07:57.555794+0000","last_active":"2024-03-31T15:07:58.538204+0000","last_peered":"2024-03-31T15:07:58.538204+0000","last_clean":"2024-03-31T15:07:58.538204+0000","last_became_active":"2024-03-31T15:07:57.555443+0000","last_became_peered":"2024-03-31T15:07:57.555443+0000","last_unstale":"2024-03-31T15:07:58.538204+0000","last_undegraded":"2024-03-31T15:07:58.538204+0000","last_fullsized":"2024-03-31T15:07:58.538204+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_clean_scrub_stamp":"2024-03-31T15:07:56.519376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T15:53:23.385556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.29","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-31T15:07:57.564070+0000","last_change":"2024-03-31T15:07:57.557824+0000","last_active":"2024-03-31T15:07:57.564070+0000","last_peered":"2024-03-31T15:07:57.564070+0000","last_clean":"2024-03-31T15:07:57.564070+0000","last_became_active":"2024-03-31T15:07:57.557441+0000","last_became_peered":"2024-03-31T15:07:57.557441+0000","last_unstale":"2024-03-31T15:07:57.564070+0000","last_undegraded":"2024-03-31T15:07:57.564070+0000","last_fullsized":"2024-03-31T15:07:57.564070+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_clean_scrub_stamp":"2024-03-31T15:07:56.519376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:20:29.793652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.28","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-31T15:07:57.562393+0000","last_change":"2024-03-31T15:07:57.555429+0000","last_active":"2024-03-31T15:07:57.562393+0000","last_peered":"2024-03-31T15:07:57.562393+0000","last_clean":"2024-03-31T15:07:57.562393+0000","last_became_active":"2024-03-31T15:07:57.555146+0000","last_became_peered":"2024-03-31T15:07:57.555146+0000","last_unstale":"2024-03-31T15:07:57.562393+0000","last_undegraded":"2024-03-31T15:07:57.562393+0000","last_fullsized":"2024-03-31T15:07:57.562393+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_clean_scrub_stamp":"2024-03-31T15:07:56.519376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T19:32:15.203110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.27","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-31T15:07:58.539623+0000","last_change":"2024-03-31T15:07:57.555518+0000","last_active":"2024-03-31T15:07:58.539623+0000","last_peered":"2024-03-31T15:07:58.539623+0000","last_clean":"2024-03-31T15:07:58.539623+0000","last_became_active":"2024-03-31T15:07:57.555013+0000","last_became_peered":"2024-03-31T15:07:57.555013+0000","last_unstale":"2024-03-31T15:07:58.539623+0000","last_undegraded":"2024-03-31T15:07:58.539623+0000","last_fullsized":"2024-03-31T15:07:58.539623+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_clean_scrub_stamp":"2024-03-31T15:07:56.519376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T16:29:14.898390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.26","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-31T15:07:58.555719+0000","last_change":"2024-03-31T15:07:57.546819+0000","last_active":"2024-03-31T15:07:58.555719+0000","last_peered":"2024-03-31T15:07:58.555719+0000","last_clean":"2024-03-31T15:07:58.555719+0000","last_became_active":"2024-03-31T15:07:57.546514+0000","last_became_peered":"2024-03-31T15:07:57.546514+0000","last_unstale":"2024-03-31T15:07:58.555719+0000","last_undegraded":"2024-03-31T15:07:58.555719+0000","last_fullsized":"2024-03-31T15:07:58.555719+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_clean_scrub_stamp":"2024-03-31T15:07:56.519376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T15:54:31.665794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.25","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-31T15:07:57.564531+0000","last_change":"2024-03-31T15:07:57.560074+0000","last_active":"2024-03-31T15:07:57.564531+0000","last_peered":"2024-03-31T15:07:57.564531+0000","last_clean":"2024-03-31T15:07:57.564531+0000","last_became_active":"2024-03-31T15:07:57.559235+0000","last_became_peered":"2024-03-31T15:07:57.559235+0000","last_unstale":"2024-03-31T15:07:57.564531+0000","last_undegraded":"2024-03-31T15:07:57.564531+0000","last_fullsized":"2024-03-31T15:07:57.564531+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_clean_scrub_stamp":"2024-03-31T15:07:56.519376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T15:51:10.452582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.24","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-31T15:07:58.539610+0000","last_change":"2024-03-31T15:07:57.546722+0000","last_active":"2024-03-31T15:07:58.539610+0000","last_peered":"2024-03-31T15:07:58.539610+0000","last_clean":"2024-03-31T15:07:58.539610+0000","last_became_active":"2024-03-31T15:07:57.546352+0000","last_became_peered":"2024-03-31T15:07:57.546352+0000","last_unstale":"2024-03-31T15:07:58.539610+0000","last_undegraded":"2024-03-31T15:07:58.539610+0000","last_fullsized":"2024-03-31T15:07:58.539610+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_clean_scrub_stamp":"2024-03-31T15:07:56.519376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:37:04.228530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.23","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-31T15:07:57.558314+0000","last_change":"2024-03-31T15:07:57.549491+0000","last_active":"2024-03-31T15:07:57.558314+0000","last_peered":"2024-03-31T15:07:57.558314+0000","last_clean":"2024-03-31T15:07:57.558314+0000","last_became_active":"2024-03-31T15:07:57.549125+0000","last_became_peered":"2024-03-31T15:07:57.549125+0000","last_unstale":"2024-03-31T15:07:57.558314+0000","last_undegraded":"2024-03-31T15:07:57.558314+0000","last_fullsized":"2024-03-31T15:07:57.558314+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_clean_scrub_stamp":"2024-03-31T15:07:56.519376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:02:11.074846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.22","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-31T15:07:58.555330+0000","last_change":"2024-03-31T15:07:57.543561+0000","last_active":"2024-03-31T15:07:58.555330+0000","last_peered":"2024-03-31T15:07:58.555330+0000","last_clean":"2024-03-31T15:07:58.555330+0000","last_became_active":"2024-03-31T15:07:57.542432+0000","last_became_peered":"2024-03-31T15:07:57.542432+0000","last_unstale":"2024-03-31T15:07:58.555330+0000","last_undegraded":"2024-03-31T15:07:58.555330+0000","last_fullsized":"2024-03-31T15:07:58.555330+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_clean_scrub_stamp":"2024-03-31T15:07:56.519376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:00:13.108819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.21","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-31T15:07:58.555431+0000","last_change":"2024-03-31T15:07:57.544243+0000","last_active":"2024-03-31T15:07:58.555431+0000","last_peered":"2024-03-31T15:07:58.555431+0000","last_clean":"2024-03-31T15:07:58.555431+0000","last_became_active":"2024-03-31T15:07:57.542605+0000","last_became_peered":"2024-03-31T15:07:57.542605+0000","last_unstale":"2024-03-31T15:07:58.555431+0000","last_undegraded":"2024-03-31T15:07:58.555431+0000","last_fullsized":"2024-03-31T15:07:58.555431+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_clean_scrub_stamp":"2024-03-31T15:07:56.519376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:57:34.405235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.20","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-31T15:07:57.551850+0000","last_change":"2024-03-31T15:07:57.545760+0000","last_active":"2024-03-31T15:07:57.551850+0000","last_peered":"2024-03-31T15:07:57.551850+0000","last_clean":"2024-03-31T15:07:57.551850+0000","last_became_active":"2024-03-31T15:07:57.545479+0000","last_became_peered":"2024-03-31T15:07:57.545479+0000","last_unstale":"2024-03-31T15:07:57.551850+0000","last_undegraded":"2024-03-31T15:07:57.551850+0000","last_fullsized":"2024-03-31T15:07:57.551850+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_clean_scrub_stamp":"2024-03-31T15:07:56.519376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:06:58.534902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.1e","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-31T15:07:57.571297+0000","last_change":"2024-03-31T15:07:57.567648+0000","last_active":"2024-03-31T15:07:57.571297+0000","last_peered":"2024-03-31T15:07:57.571297+0000","last_clean":"2024-03-31T15:07:57.571297+0000","last_became_active":"2024-03-31T15:07:57.565622+0000","last_became_peered":"2024-03-31T15:07:57.565622+0000","last_unstale":"2024-03-31T15:07:57.571297+0000","last_undegraded":"2024-03-31T15:07:57.571297+0000","last_fullsized":"2024-03-31T15:07:57.571297+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_clean_scrub_stamp":"2024-03-31T15:07:56.519376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:18:05.654205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1f","version":"66'4","reported_seq":16,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-31T15:07:57.568905+0000","last_change":"2024-03-31T15:07:56.553471+0000","last_active":"2024-03-31T15:07:57.568905+0000","last_peered":"2024-03-31T15:07:57.568905+0000","last_clean":"2024-03-31T15:07:57.568905+0000","last_became_active":"2024-03-31T15:07:56.553009+0000","last_became_peered":"2024-03-31T15:07:56.553009+0000","last_unstale":"2024-03-31T15:07:57.568905+0000","last_undegraded":"2024-03-31T15:07:57.568905+0000","last_fullsized":"2024-03-31T15:07:57.568905+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:07:55.513096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:07:55.513096+0000","last_clean_scrub_stamp":"2024-03-31T15:07:55.513096+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:18:23.348865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.1f","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-31T15:07:58.532807+0000","last_change":"2024-03-31T15:07:57.550188+0000","last_active":"2024-03-31T15:07:58.532807+0000","last_peered":"2024-03-31T15:07:58.532807+0000","last_clean":"2024-03-31T15:07:58.532807+0000","last_became_active":"2024-03-31T15:07:57.549188+0000","last_became_peered":"2024-03-31T15:07:57.549188+0000","last_unstale":"2024-03-31T15:07:58.532807+0000","last_undegraded":"2024-03-31T15:07:58.532807+0000","last_fullsized":"2024-03-31T15:07:58.532807+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_clean_scrub_stamp":"2024-03-31T15:07:56.519376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T16:57:04.346527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.1e","version":"66'1","reported_seq":13,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-31T15:07:57.562594+0000","last_change":"2024-03-31T15:07:56.548168+0000","last_active":"2024-03-31T15:07:57.562594+0000","last_peered":"2024-03-31T15:07:57.562594+0000","last_clean":"2024-03-31T15:07:57.562594+0000","last_became_active":"2024-03-31T15:07:56.547846+0000","last_became_peered":"2024-03-31T15:07:56.547846+0000","last_unstale":"2024-03-31T15:07:57.562594+0000","last_undegraded":"2024-03-31T15:07:57.562594+0000","last_fullsized":"2024-03-31T15:07:57.562594+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:07:55.513096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:07:55.513096+0000","last_clean_scrub_stamp":"2024-03-31T15:07:55.513096+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T19:24:36.128081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1c","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-31T15:07:57.554456+0000","last_change":"2024-03-31T15:07:57.551178+0000","last_active":"2024-03-31T15:07:57.554456+0000","last_peered":"2024-03-31T15:07:57.554456+0000","last_clean":"2024-03-31T15:07:57.554456+0000","last_became_active":"2024-03-31T15:07:57.550523+0000","last_became_peered":"2024-03-31T15:07:57.550523+0000","last_unstale":"2024-03-31T15:07:57.554456+0000","last_undegraded":"2024-03-31T15:07:57.554456+0000","last_fullsized":"2024-03-31T15:07:57.554456+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_clean_scrub_stamp":"2024-03-31T15:07:56.519376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:29:53.308887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.1d","version":"66'2","reported_seq":16,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-31T15:07:58.539470+0000","last_change":"2024-03-31T15:07:56.545113+0000","last_active":"2024-03-31T15:07:58.539470+0000","last_peered":"2024-03-31T15:07:58.539470+0000","last_clean":"2024-03-31T15:07:58.539470+0000","last_became_active":"2024-03-31T15:07:56.542451+0000","last_became_peered":"2024-03-31T15:07:56.542451+0000","last_unstale":"2024-03-31T15:07:58.539470+0000","last_undegraded":"2024-03-31T15:07:58.539470+0000","last_fullsized":"2024-03-31T15:07:58.539470+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:07:55.513096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:07:55.513096+0000","last_clean_scrub_stamp":"2024-03-31T15:07:55.513096+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:08:42.900813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1d","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-31T15:07:57.571246+0000","last_change":"2024-03-31T15:07:57.565338+0000","last_active":"2024-03-31T15:07:57.571246+0000","last_peered":"2024-03-31T15:07:57.571246+0000","last_clean":"2024-03-31T15:07:57.571246+0000","last_became_active":"2024-03-31T15:07:57.564994+0000","last_became_peered":"2024-03-31T15:07:57.564994+0000","last_unstale":"2024-03-31T15:07:57.571246+0000","last_undegraded":"2024-03-31T15:07:57.571246+0000","last_fullsized":"2024-03-31T15:07:57.571246+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_clean_scrub_stamp":"2024-03-31T15:07:56.519376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:12:19.344476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1c","version":"66'2","reported_seq":14,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-31T15:07:57.573594+0000","last_change":"2024-03-31T15:07:56.546907+0000","last_active":"2024-03-31T15:07:57.573594+0000","last_peered":"2024-03-31T15:07:57.573594+0000","last_clean":"2024-03-31T15:07:57.573594+0000","last_became_active":"2024-03-31T15:07:56.543724+0000","last_became_peered":"2024-03-31T15:07:56.543724+0000","last_unstale":"2024-03-31T15:07:57.573594+0000","last_undegraded":"2024-03-31T15:07:57.573594+0000","last_fullsized":"2024-03-31T15:07:57.573594+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:07:55.513096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:07:55.513096+0000","last_clean_scrub_stamp":"2024-03-31T15:07:55.513096+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T19:36:35.238243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.1a","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-31T15:07:58.555881+0000","last_change":"2024-03-31T15:07:57.546007+0000","last_active":"2024-03-31T15:07:58.555881+0000","last_peered":"2024-03-31T15:07:58.555881+0000","last_clean":"2024-03-31T15:07:58.555881+0000","last_became_active":"2024-03-31T15:07:57.545744+0000","last_became_peered":"2024-03-31T15:07:57.545744+0000","last_unstale":"2024-03-31T15:07:58.555881+0000","last_undegraded":"2024-03-31T15:07:58.555881+0000","last_fullsized":"2024-03-31T15:07:58.555881+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_clean_scrub_stamp":"2024-03-31T15:07:56.519376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:10:54.127697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.1b","version":"66'1","reported_seq":13,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-31T15:07:57.569537+0000","last_change":"2024-03-31T15:07:56.544488+0000","last_active":"2024-03-31T15:07:57.569537+0000","last_peered":"2024-03-31T15:07:57.569537+0000","last_clean":"2024-03-31T15:07:57.569537+0000","last_became_active":"2024-03-31T15:07:56.544088+0000","last_became_peered":"2024-03-31T15:07:56.544088+0000","last_unstale":"2024-03-31T15:07:57.569537+0000","last_undegraded":"2024-03-31T15:07:57.569537+0000","last_fullsized":"2024-03-31T15:07:57.569537+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:07:55.513096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:07:55.513096+0000","last_clean_scrub_stamp":"2024-03-31T15:07:55.513096+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T16:21:08.149844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.1b","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-31T15:07:58.539958+0000","last_change":"2024-03-31T15:07:57.554226+0000","last_active":"2024-03-31T15:07:58.539958+0000","last_peered":"2024-03-31T15:07:58.539958+0000","last_clean":"2024-03-31T15:07:58.539958+0000","last_became_active":"2024-03-31T15:07:57.553938+0000","last_became_peered":"2024-03-31T15:07:57.553938+0000","last_unstale":"2024-03-31T15:07:58.539958+0000","last_undegraded":"2024-03-31T15:07:58.539958+0000","last_fullsized":"2024-03-31T15:07:58.539958+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_clean_scrub_stamp":"2024-03-31T15:07:56.519376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T17:43:07.375959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.1a","version":"66'1","reported_seq":13,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-31T15:07:57.573594+0000","last_change":"2024-03-31T15:07:56.547321+0000","last_active":"2024-03-31T15:07:57.573594+0000","last_peered":"2024-03-31T15:07:57.573594+0000","last_clean":"2024-03-31T15:07:57.573594+0000","last_became_active":"2024-03-31T15:07:56.546434+0000","last_became_peered":"2024-03-31T15:07:56.546434+0000","last_unstale":"2024-03-31T15:07:57.573594+0000","last_undegraded":"2024-03-31T15:07:57.573594+0000","last_fullsized":"2024-03-31T15:07:57.573594+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:07:55.513096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:07:55.513096+0000","last_clean_scrub_stamp":"2024-03-31T15:07:55.513096+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:07:50.050709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-31T15:07:58.532032+0000","last_change":"2024-03-31T15:07:57.543472+0000","last_active":"2024-03-31T15:07:58.532032+0000","last_peered":"2024-03-31T15:07:58.532032+0000","last_clean":"2024-03-31T15:07:58.532032+0000","last_became_active":"2024-03-31T15:07:57.543161+0000","last_became_peered":"2024-03-31T15:07:57.543161+0000","last_unstale":"2024-03-31T15:07:58.532032+0000","last_undegraded":"2024-03-31T15:07:58.532032+0000","last_fullsized":"2024-03-31T15:07:58.532032+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_clean_scrub_stamp":"2024-03-31T15:07:56.519376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:25:16.321644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.19","version":"0'0","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-31T15:07:57.539053+0000","last_change":"2024-03-31T15:07:56.540446+0000","last_active":"2024-03-31T15:07:57.539053+0000","last_peered":"2024-03-31T15:07:57.539053+0000","last_clean":"2024-03-31T15:07:57.539053+0000","last_became_active":"2024-03-31T15:07:56.540082+0000","last_became_peered":"2024-03-31T15:07:56.540082+0000","last_unstale":"2024-03-31T15:07:57.539053+0000","last_undegraded":"2024-03-31T15:07:57.539053+0000","last_fullsized":"2024-03-31T15:07:57.539053+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:07:55.513096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:07:55.513096+0000","last_clean_scrub_stamp":"2024-03-31T15:07:55.513096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:27:16.077645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-31T15:07:57.552198+0000","last_change":"2024-03-31T15:07:57.551029+0000","last_active":"2024-03-31T15:07:57.552198+0000","last_peered":"2024-03-31T15:07:57.552198+0000","last_clean":"2024-03-31T15:07:57.552198+0000","last_became_active":"2024-03-31T15:07:57.550575+0000","last_became_peered":"2024-03-31T15:07:57.550575+0000","last_unstale":"2024-03-31T15:07:57.552198+0000","last_undegraded":"2024-03-31T15:07:57.552198+0000","last_fullsized":"2024-03-31T15:07:57.552198+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_clean_scrub_stamp":"2024-03-31T15:07:56.519376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T16:14:04.493893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.18","version":"0'0","reported_seq":14,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-31T15:07:58.532035+0000","last_change":"2024-03-31T15:07:56.543341+0000","last_active":"2024-03-31T15:07:58.532035+0000","last_peered":"2024-03-31T15:07:58.532035+0000","last_clean":"2024-03-31T15:07:58.532035+0000","last_became_active":"2024-03-31T15:07:56.541685+0000","last_became_peered":"2024-03-31T15:07:56.541685+0000","last_unstale":"2024-03-31T15:07:58.532035+0000","last_undegraded":"2024-03-31T15:07:58.532035+0000","last_fullsized":"2024-03-31T15:07:58.532035+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:07:55.513096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:07:55.513096+0000","last_clean_scrub_stamp":"2024-03-31T15:07:55.513096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:25:20.169134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-31T15:07:58.555489+0000","last_change":"2024-03-31T15:07:57.549590+0000","last_active":"2024-03-31T15:07:58.555489+0000","last_peered":"2024-03-31T15:07:58.555489+0000","last_clean":"2024-03-31T15:07:58.555489+0000","last_became_active":"2024-03-31T15:07:57.549224+0000","last_became_peered":"2024-03-31T15:07:57.549224+0000","last_unstale":"2024-03-31T15:07:58.555489+0000","last_undegraded":"2024-03-31T15:07:58.555489+0000","last_fullsized":"2024-03-31T15:07:58.555489+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_clean_scrub_stamp":"2024-03-31T15:07:56.519376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:12:19.344476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.17","version":"0'0","reported_seq":14,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-31T15:07:58.539743+0000","last_change":"2024-03-31T15:07:56.546737+0000","last_active":"2024-03-31T15:07:58.539743+0000","last_peered":"2024-03-31T15:07:58.539743+0000","last_clean":"2024-03-31T15:07:58.539743+0000","last_became_active":"2024-03-31T15:07:56.546153+0000","last_became_peered":"2024-03-31T15:07:56.546153+0000","last_unstale":"2024-03-31T15:07:58.539743+0000","last_undegraded":"2024-03-31T15:07:58.539743+0000","last_fullsized":"2024-03-31T15:07:58.539743+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:07:55.513096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:07:55.513096+0000","last_clean_scrub_stamp":"2024-03-31T15:07:55.513096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:17:08.509358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-31T15:07:57.571223+0000","last_change":"2024-03-31T15:07:57.565169+0000","last_active":"2024-03-31T15:07:57.571223+0000","last_peered":"2024-03-31T15:07:57.571223+0000","last_clean":"2024-03-31T15:07:57.571223+0000","last_became_active":"2024-03-31T15:07:57.564779+0000","last_became_peered":"2024-03-31T15:07:57.564779+0000","last_unstale":"2024-03-31T15:07:57.571223+0000","last_undegraded":"2024-03-31T15:07:57.571223+0000","last_fullsized":"2024-03-31T15:07:57.571223+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_clean_scrub_stamp":"2024-03-31T15:07:56.519376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:57:34.405235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.16","version":"0'0","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-31T15:07:57.539998+0000","last_change":"2024-03-31T15:07:56.546801+0000","last_active":"2024-03-31T15:07:57.539998+0000","last_peered":"2024-03-31T15:07:57.539998+0000","last_clean":"2024-03-31T15:07:57.539998+0000","last_became_active":"2024-03-31T15:07:56.544142+0000","last_became_peered":"2024-03-31T15:07:56.544142+0000","last_unstale":"2024-03-31T15:07:57.539998+0000","last_undegraded":"2024-03-31T15:07:57.539998+0000","last_fullsized":"2024-03-31T15:07:57.539998+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:07:55.513096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:07:55.513096+0000","last_clean_scrub_stamp":"2024-03-31T15:07:55.513096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:55:12.611335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.14","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-31T15:07:57.553745+0000","last_change":"2024-03-31T15:07:57.549734+0000","last_active":"2024-03-31T15:07:57.553745+0000","last_peered":"2024-03-31T15:07:57.553745+0000","last_clean":"2024-03-31T15:07:57.553745+0000","last_became_active":"2024-03-31T15:07:57.549441+0000","last_became_peered":"2024-03-31T15:07:57.549441+0000","last_unstale":"2024-03-31T15:07:57.553745+0000","last_undegraded":"2024-03-31T15:07:57.553745+0000","last_fullsized":"2024-03-31T15:07:57.553745+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_clean_scrub_stamp":"2024-03-31T15:07:56.519376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T17:20:07.399517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.15","version":"66'3","reported_seq":15,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-31T15:07:57.570571+0000","last_change":"2024-03-31T15:07:56.547818+0000","last_active":"2024-03-31T15:07:57.570571+0000","last_peered":"2024-03-31T15:07:57.570571+0000","last_clean":"2024-03-31T15:07:57.570571+0000","last_became_active":"2024-03-31T15:07:56.546707+0000","last_became_peered":"2024-03-31T15:07:56.546707+0000","last_unstale":"2024-03-31T15:07:57.570571+0000","last_undegraded":"2024-03-31T15:07:57.570571+0000","last_fullsized":"2024-03-31T15:07:57.570571+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:07:55.513096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:07:55.513096+0000","last_clean_scrub_stamp":"2024-03-31T15:07:55.513096+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:14:52.090549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.15","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-31T15:07:58.532417+0000","last_change":"2024-03-31T15:07:57.557344+0000","last_active":"2024-03-31T15:07:58.532417+0000","last_peered":"2024-03-31T15:07:58.532417+0000","last_clean":"2024-03-31T15:07:58.532417+0000","last_became_active":"2024-03-31T15:07:57.556402+0000","last_became_peered":"2024-03-31T15:07:57.556402+0000","last_unstale":"2024-03-31T15:07:58.532417+0000","last_undegraded":"2024-03-31T15:07:58.532417+0000","last_fullsized":"2024-03-31T15:07:58.532417+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_clean_scrub_stamp":"2024-03-31T15:07:56.519376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T15:43:48.032189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.14","version":"66'6","reported_seq":18,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-31T15:07:57.572398+0000","last_change":"2024-03-31T15:07:56.545850+0000","last_active":"2024-03-31T15:07:57.572398+0000","last_peered":"2024-03-31T15:07:57.572398+0000","last_clean":"2024-03-31T15:07:57.572398+0000","last_became_active":"2024-03-31T15:07:56.545561+0000","last_became_peered":"2024-03-31T15:07:56.545561+0000","last_unstale":"2024-03-31T15:07:57.572398+0000","last_undegraded":"2024-03-31T15:07:57.572398+0000","last_fullsized":"2024-03-31T15:07:57.572398+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:07:55.513096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:07:55.513096+0000","last_clean_scrub_stamp":"2024-03-31T15:07:55.513096+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T15:25:09.572264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-31T15:07:57.565295+0000","last_change":"2024-03-31T15:07:57.560749+0000","last_active":"2024-03-31T15:07:57.565295+0000","last_peered":"2024-03-31T15:07:57.565295+0000","last_clean":"2024-03-31T15:07:57.565295+0000","last_became_active":"2024-03-31T15:07:57.560377+0000","last_became_peered":"2024-03-31T15:07:57.560377+0000","last_unstale":"2024-03-31T15:07:57.565295+0000","last_undegraded":"2024-03-31T15:07:57.565295+0000","last_fullsized":"2024-03-31T15:07:57.565295+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_clean_scrub_stamp":"2024-03-31T15:07:56.519376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:10:44.422861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.13","version":"66'2","reported_seq":14,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-31T15:07:57.574397+0000","last_change":"2024-03-31T15:07:56.551058+0000","last_active":"2024-03-31T15:07:57.574397+0000","last_peered":"2024-03-31T15:07:57.574397+0000","last_clean":"2024-03-31T15:07:57.574397+0000","last_became_active":"2024-03-31T15:07:56.550793+0000","last_became_peered":"2024-03-31T15:07:56.550793+0000","last_unstale":"2024-03-31T15:07:57.574397+0000","last_undegraded":"2024-03-31T15:07:57.574397+0000","last_fullsized":"2024-03-31T15:07:57.574397+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:07:55.513096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:07:55.513096+0000","last_clean_scrub_stamp":"2024-03-31T15:07:55.513096+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T19:36:35.238243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.13","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-31T15:07:58.555415+0000","last_change":"2024-03-31T15:07:57.550422+0000","last_active":"2024-03-31T15:07:58.555415+0000","last_peered":"2024-03-31T15:07:58.555415+0000","last_clean":"2024-03-31T15:07:58.555415+0000","last_became_active":"2024-03-31T15:07:57.550127+0000","last_became_peered":"2024-03-31T15:07:57.550127+0000","last_unstale":"2024-03-31T15:07:58.555415+0000","last_undegraded":"2024-03-31T15:07:58.555415+0000","last_fullsized":"2024-03-31T15:07:58.555415+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_clean_scrub_stamp":"2024-03-31T15:07:56.519376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T16:17:18.266469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.12","version":"0'0","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-31T15:07:57.539280+0000","last_change":"2024-03-31T15:07:56.547420+0000","last_active":"2024-03-31T15:07:57.539280+0000","last_peered":"2024-03-31T15:07:57.539280+0000","last_clean":"2024-03-31T15:07:57.539280+0000","last_became_active":"2024-03-31T15:07:56.547137+0000","last_became_peered":"2024-03-31T15:07:56.547137+0000","last_unstale":"2024-03-31T15:07:57.539280+0000","last_undegraded":"2024-03-31T15:07:57.539280+0000","last_fullsized":"2024-03-31T15:07:57.539280+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:07:55.513096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:07:55.513096+0000","last_clean_scrub_stamp":"2024-03-31T15:07:55.513096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T17:48:14.267285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.10","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-31T15:07:57.551763+0000","last_change":"2024-03-31T15:07:57.546040+0000","last_active":"2024-03-31T15:07:57.551763+0000","last_peered":"2024-03-31T15:07:57.551763+0000","last_clean":"2024-03-31T15:07:57.551763+0000","last_became_active":"2024-03-31T15:07:57.545755+0000","last_became_peered":"2024-03-31T15:07:57.545755+0000","last_unstale":"2024-03-31T15:07:57.551763+0000","last_undegraded":"2024-03-31T15:07:57.551763+0000","last_fullsized":"2024-03-31T15:07:57.551763+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_clean_scrub_stamp":"2024-03-31T15:07:56.519376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:14:09.956300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.11","version":"0'0","reported_seq":14,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-31T15:07:58.555257+0000","last_change":"2024-03-31T15:07:56.545516+0000","last_active":"2024-03-31T15:07:58.555257+0000","last_peered":"2024-03-31T15:07:58.555257+0000","last_clean":"2024-03-31T15:07:58.555257+0000","last_became_active":"2024-03-31T15:07:56.545177+0000","last_became_peered":"2024-03-31T15:07:56.545177+0000","last_unstale":"2024-03-31T15:07:58.555257+0000","last_undegraded":"2024-03-31T15:07:58.555257+0000","last_fullsized":"2024-03-31T15:07:58.555257+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:07:55.513096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:07:55.513096+0000","last_clean_scrub_stamp":"2024-03-31T15:07:55.513096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:05:25.263578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-31T15:07:58.532317+0000","last_change":"2024-03-31T15:07:57.540440+0000","last_active":"2024-03-31T15:07:58.532317+0000","last_peered":"2024-03-31T15:07:58.532317+0000","last_clean":"2024-03-31T15:07:58.532317+0000","last_became_active":"2024-03-31T15:07:57.540194+0000","last_became_peered":"2024-03-31T15:07:57.540194+0000","last_unstale":"2024-03-31T15:07:58.532317+0000","last_undegraded":"2024-03-31T15:07:58.532317+0000","last_fullsized":"2024-03-31T15:07:58.532317+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_clean_scrub_stamp":"2024-03-31T15:07:56.519376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:54:31.087964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.10","version":"66'8","reported_seq":22,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-31T15:07:58.538133+0000","last_change":"2024-03-31T15:07:56.550974+0000","last_active":"2024-03-31T15:07:58.538133+0000","last_peered":"2024-03-31T15:07:58.538133+0000","last_clean":"2024-03-31T15:07:58.538133+0000","last_became_active":"2024-03-31T15:07:56.550627+0000","last_became_peered":"2024-03-31T15:07:56.550627+0000","last_unstale":"2024-03-31T15:07:58.538133+0000","last_undegraded":"2024-03-31T15:07:58.538133+0000","last_fullsized":"2024-03-31T15:07:58.538133+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:07:55.513096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:07:55.513096+0000","last_clean_scrub_stamp":"2024-03-31T15:07:55.513096+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:20:01.748754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.e","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-31T15:07:58.539831+0000","last_change":"2024-03-31T15:07:57.556445+0000","last_active":"2024-03-31T15:07:58.539831+0000","last_peered":"2024-03-31T15:07:58.539831+0000","last_clean":"2024-03-31T15:07:58.539831+0000","last_became_active":"2024-03-31T15:07:57.555786+0000","last_became_peered":"2024-03-31T15:07:57.555786+0000","last_unstale":"2024-03-31T15:07:58.539831+0000","last_undegraded":"2024-03-31T15:07:58.539831+0000","last_fullsized":"2024-03-31T15:07:58.539831+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_clean_scrub_stamp":"2024-03-31T15:07:56.519376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:09:37.513942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.f","version":"66'1","reported_seq":13,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-31T15:07:57.573739+0000","last_change":"2024-03-31T15:07:56.549846+0000","last_active":"2024-03-31T15:07:57.573739+0000","last_peered":"2024-03-31T15:07:57.573739+0000","last_clean":"2024-03-31T15:07:57.573739+0000","last_became_active":"2024-03-31T15:07:56.549578+0000","last_became_peered":"2024-03-31T15:07:56.549578+0000","last_unstale":"2024-03-31T15:07:57.573739+0000","last_undegraded":"2024-03-31T15:07:57.573739+0000","last_fullsized":"2024-03-31T15:07:57.573739+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:07:55.513096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:07:55.513096+0000","last_clean_scrub_stamp":"2024-03-31T15:07:55.513096+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:03:52.350530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.f","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-31T15:07:57.569420+0000","last_change":"2024-03-31T15:07:57.563223+0000","last_active":"2024-03-31T15:07:57.569420+0000","last_peered":"2024-03-31T15:07:57.569420+0000","last_clean":"2024-03-31T15:07:57.569420+0000","last_became_active":"2024-03-31T15:07:57.562916+0000","last_became_peered":"2024-03-31T15:07:57.562916+0000","last_unstale":"2024-03-31T15:07:57.569420+0000","last_undegraded":"2024-03-31T15:07:57.569420+0000","last_fullsized":"2024-03-31T15:07:57.569420+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_clean_scrub_stamp":"2024-03-31T15:07:56.519376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:36:07.095425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.e","version":"66'2","reported_seq":16,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-31T15:07:58.539973+0000","last_change":"2024-03-31T15:07:56.545147+0000","last_active":"2024-03-31T15:07:58.539973+0000","last_peered":"2024-03-31T15:07:58.539973+0000","last_clean":"2024-03-31T15:07:58.539973+0000","last_became_active":"2024-03-31T15:07:56.544872+0000","last_became_peered":"2024-03-31T15:07:56.544872+0000","last_unstale":"2024-03-31T15:07:58.539973+0000","last_undegraded":"2024-03-31T15:07:58.539973+0000","last_fullsized":"2024-03-31T15:07:58.539973+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:07:55.513096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:07:55.513096+0000","last_clean_scrub_stamp":"2024-03-31T15:07:55.513096+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:29:22.822927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-31T15:07:57.560005+0000","last_change":"2024-03-31T15:07:57.557119+0000","last_active":"2024-03-31T15:07:57.560005+0000","last_peered":"2024-03-31T15:07:57.560005+0000","last_clean":"2024-03-31T15:07:57.560005+0000","last_became_active":"2024-03-31T15:07:57.556477+0000","last_became_peered":"2024-03-31T15:07:57.556477+0000","last_unstale":"2024-03-31T15:07:57.560005+0000","last_undegraded":"2024-03-31T15:07:57.560005+0000","last_fullsized":"2024-03-31T15:07:57.560005+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_clean_scrub_stamp":"2024-03-31T15:07:56.519376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:28:56.174571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.d","version":"66'1","reported_seq":13,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-31T15:07:57.574433+0000","last_change":"2024-03-31T15:07:56.545706+0000","last_active":"2024-03-31T15:07:57.574433+0000","last_peered":"2024-03-31T15:07:57.574433+0000","last_clean":"2024-03-31T15:07:57.574433+0000","last_became_active":"2024-03-31T15:07:56.545279+0000","last_became_peered":"2024-03-31T15:07:56.545279+0000","last_unstale":"2024-03-31T15:07:57.574433+0000","last_undegraded":"2024-03-31T15:07:57.574433+0000","last_fullsized":"2024-03-31T15:07:57.574433+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:07:55.513096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:07:55.513096+0000","last_clean_scrub_stamp":"2024-03-31T15:07:55.513096+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:06:27.558750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.d","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-31T15:07:58.532496+0000","last_change":"2024-03-31T15:07:57.543174+0000","last_active":"2024-03-31T15:07:58.532496+0000","last_peered":"2024-03-31T15:07:58.532496+0000","last_clean":"2024-03-31T15:07:58.532496+0000","last_became_active":"2024-03-31T15:07:57.542863+0000","last_became_peered":"2024-03-31T15:07:57.542863+0000","last_unstale":"2024-03-31T15:07:58.532496+0000","last_undegraded":"2024-03-31T15:07:58.532496+0000","last_fullsized":"2024-03-31T15:07:58.532496+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_clean_scrub_stamp":"2024-03-31T15:07:56.519376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:36:18.560703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.c","version":"0'0","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-31T15:07:57.550504+0000","last_change":"2024-03-31T15:07:56.553530+0000","last_active":"2024-03-31T15:07:57.550504+0000","last_peered":"2024-03-31T15:07:57.550504+0000","last_clean":"2024-03-31T15:07:57.550504+0000","last_became_active":"2024-03-31T15:07:56.553020+0000","last_became_peered":"2024-03-31T15:07:56.553020+0000","last_unstale":"2024-03-31T15:07:57.550504+0000","last_undegraded":"2024-03-31T15:07:57.550504+0000","last_fullsized":"2024-03-31T15:07:57.550504+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:07:55.513096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:07:55.513096+0000","last_clean_scrub_stamp":"2024-03-31T15:07:55.513096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:03:57.063293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.b","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-31T15:07:57.563690+0000","last_change":"2024-03-31T15:07:57.558081+0000","last_active":"2024-03-31T15:07:57.563690+0000","last_peered":"2024-03-31T15:07:57.563690+0000","last_clean":"2024-03-31T15:07:57.563690+0000","last_became_active":"2024-03-31T15:07:57.557812+0000","last_became_peered":"2024-03-31T15:07:57.557812+0000","last_unstale":"2024-03-31T15:07:57.563690+0000","last_undegraded":"2024-03-31T15:07:57.563690+0000","last_fullsized":"2024-03-31T15:07:57.563690+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_clean_scrub_stamp":"2024-03-31T15:07:56.519376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:51:45.557002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.a","version":"0'0","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-31T15:07:57.539353+0000","last_change":"2024-03-31T15:07:56.547001+0000","last_active":"2024-03-31T15:07:57.539353+0000","last_peered":"2024-03-31T15:07:57.539353+0000","last_clean":"2024-03-31T15:07:57.539353+0000","last_became_active":"2024-03-31T15:07:56.544110+0000","last_became_peered":"2024-03-31T15:07:56.544110+0000","last_unstale":"2024-03-31T15:07:57.539353+0000","last_undegraded":"2024-03-31T15:07:57.539353+0000","last_fullsized":"2024-03-31T15:07:57.539353+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:07:55.513096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:07:55.513096+0000","last_clean_scrub_stamp":"2024-03-31T15:07:55.513096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:28:55.168291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-31T15:07:57.558674+0000","last_change":"2024-03-31T15:07:57.545669+0000","last_active":"2024-03-31T15:07:57.558674+0000","last_peered":"2024-03-31T15:07:57.558674+0000","last_clean":"2024-03-31T15:07:57.558674+0000","last_became_active":"2024-03-31T15:07:57.545289+0000","last_became_peered":"2024-03-31T15:07:57.545289+0000","last_unstale":"2024-03-31T15:07:57.558674+0000","last_undegraded":"2024-03-31T15:07:57.558674+0000","last_fullsized":"2024-03-31T15:07:57.558674+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_clean_scrub_stamp":"2024-03-31T15:07:56.519376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:13:43.088446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.9","version":"0'0","reported_seq":14,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-31T15:07:58.555558+0000","last_change":"2024-03-31T15:07:56.545941+0000","last_active":"2024-03-31T15:07:58.555558+0000","last_peered":"2024-03-31T15:07:58.555558+0000","last_clean":"2024-03-31T15:07:58.555558+0000","last_became_active":"2024-03-31T15:07:56.545659+0000","last_became_peered":"2024-03-31T15:07:56.545659+0000","last_unstale":"2024-03-31T15:07:58.555558+0000","last_undegraded":"2024-03-31T15:07:58.555558+0000","last_fullsized":"2024-03-31T15:07:58.555558+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:07:55.513096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:07:55.513096+0000","last_clean_scrub_stamp":"2024-03-31T15:07:55.513096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:09:36.507662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-31T15:07:57.557081+0000","last_change":"2024-03-31T15:07:57.542386+0000","last_active":"2024-03-31T15:07:57.557081+0000","last_peered":"2024-03-31T15:07:57.557081+0000","last_clean":"2024-03-31T15:07:57.557081+0000","last_became_active":"2024-03-31T15:07:57.541876+0000","last_became_peered":"2024-03-31T15:07:57.541876+0000","last_unstale":"2024-03-31T15:07:57.557081+0000","last_undegraded":"2024-03-31T15:07:57.557081+0000","last_fullsized":"2024-03-31T15:07:57.557081+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_clean_scrub_stamp":"2024-03-31T15:07:56.519376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T17:18:01.667911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.8","version":"0'0","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-31T15:07:57.534186+0000","last_change":"2024-03-31T15:07:56.547442+0000","last_active":"2024-03-31T15:07:57.534186+0000","last_peered":"2024-03-31T15:07:57.534186+0000","last_clean":"2024-03-31T15:07:57.534186+0000","last_became_active":"2024-03-31T15:07:56.546937+0000","last_became_peered":"2024-03-31T15:07:56.546937+0000","last_unstale":"2024-03-31T15:07:57.534186+0000","last_undegraded":"2024-03-31T15:07:57.534186+0000","last_fullsized":"2024-03-31T15:07:57.534186+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:07:55.513096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:07:55.513096+0000","last_clean_scrub_stamp":"2024-03-31T15:07:55.513096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:41:39.402187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.6","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-31T15:07:57.550003+0000","last_change":"2024-03-31T15:07:57.538061+0000","last_active":"2024-03-31T15:07:57.550003+0000","last_peered":"2024-03-31T15:07:57.550003+0000","last_clean":"2024-03-31T15:07:57.550003+0000","last_became_active":"2024-03-31T15:07:57.537575+0000","last_became_peered":"2024-03-31T15:07:57.537575+0000","last_unstale":"2024-03-31T15:07:57.550003+0000","last_undegraded":"2024-03-31T15:07:57.550003+0000","last_fullsized":"2024-03-31T15:07:57.550003+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_clean_scrub_stamp":"2024-03-31T15:07:56.519376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:03:08.318569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.7","version":"66'2","reported_seq":14,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-31T15:07:57.573699+0000","last_change":"2024-03-31T15:07:56.549633+0000","last_active":"2024-03-31T15:07:57.573699+0000","last_peered":"2024-03-31T15:07:57.573699+0000","last_clean":"2024-03-31T15:07:57.573699+0000","last_became_active":"2024-03-31T15:07:56.549111+0000","last_became_peered":"2024-03-31T15:07:56.549111+0000","last_unstale":"2024-03-31T15:07:57.573699+0000","last_undegraded":"2024-03-31T15:07:57.573699+0000","last_fullsized":"2024-03-31T15:07:57.573699+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:07:55.513096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:07:55.513096+0000","last_clean_scrub_stamp":"2024-03-31T15:07:55.513096+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:00:33.868348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.7","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-31T15:07:58.555592+0000","last_change":"2024-03-31T15:07:57.559609+0000","last_active":"2024-03-31T15:07:58.555592+0000","last_peered":"2024-03-31T15:07:58.555592+0000","last_clean":"2024-03-31T15:07:58.555592+0000","last_became_active":"2024-03-31T15:07:57.559315+0000","last_became_peered":"2024-03-31T15:07:57.559315+0000","last_unstale":"2024-03-31T15:07:58.555592+0000","last_undegraded":"2024-03-31T15:07:58.555592+0000","last_fullsized":"2024-03-31T15:07:58.555592+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_clean_scrub_stamp":"2024-03-31T15:07:56.519376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:18:05.654205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.6","version":"67'1","reported_seq":15,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-31T15:07:58.557493+0000","last_change":"2024-03-31T15:07:56.544027+0000","last_active":"2024-03-31T15:07:58.557493+0000","last_peered":"2024-03-31T15:07:58.557493+0000","last_clean":"2024-03-31T15:07:58.557493+0000","last_became_active":"2024-03-31T15:07:56.543741+0000","last_became_peered":"2024-03-31T15:07:56.543741+0000","last_unstale":"2024-03-31T15:07:58.557493+0000","last_undegraded":"2024-03-31T15:07:58.557493+0000","last_fullsized":"2024-03-31T15:07:58.557493+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:07:55.513096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:07:55.513096+0000","last_clean_scrub_stamp":"2024-03-31T15:07:55.513096+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T17:43:06.369679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-31T15:07:57.564282+0000","last_change":"2024-03-31T15:07:57.557984+0000","last_active":"2024-03-31T15:07:57.564282+0000","last_peered":"2024-03-31T15:07:57.564282+0000","last_clean":"2024-03-31T15:07:57.564282+0000","last_became_active":"2024-03-31T15:07:57.556918+0000","last_became_peered":"2024-03-31T15:07:57.556918+0000","last_unstale":"2024-03-31T15:07:57.564282+0000","last_undegraded":"2024-03-31T15:07:57.564282+0000","last_fullsized":"2024-03-31T15:07:57.564282+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_clean_scrub_stamp":"2024-03-31T15:07:56.519376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:05:31.529218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":14,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-31T15:07:58.532526+0000","last_change":"2024-03-31T15:07:56.544111+0000","last_active":"2024-03-31T15:07:58.532526+0000","last_peered":"2024-03-31T15:07:58.532526+0000","last_clean":"2024-03-31T15:07:58.532526+0000","last_became_active":"2024-03-31T15:07:56.543835+0000","last_became_peered":"2024-03-31T15:07:56.543835+0000","last_unstale":"2024-03-31T15:07:58.532526+0000","last_undegraded":"2024-03-31T15:07:58.532526+0000","last_fullsized":"2024-03-31T15:07:58.532526+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:07:55.513096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:07:55.513096+0000","last_clean_scrub_stamp":"2024-03-31T15:07:55.513096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:55:41.199547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-31T15:07:57.574009+0000","last_change":"2024-03-31T15:07:57.569085+0000","last_active":"2024-03-31T15:07:57.574009+0000","last_peered":"2024-03-31T15:07:57.574009+0000","last_clean":"2024-03-31T15:07:57.574009+0000","last_became_active":"2024-03-31T15:07:57.568239+0000","last_became_peered":"2024-03-31T15:07:57.568239+0000","last_unstale":"2024-03-31T15:07:57.574009+0000","last_undegraded":"2024-03-31T15:07:57.574009+0000","last_fullsized":"2024-03-31T15:07:57.574009+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_clean_scrub_stamp":"2024-03-31T15:07:56.519376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:06:47.401497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.4","version":"0'0","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-31T15:07:57.534781+0000","last_change":"2024-03-31T15:07:56.543969+0000","last_active":"2024-03-31T15:07:57.534781+0000","last_peered":"2024-03-31T15:07:57.534781+0000","last_clean":"2024-03-31T15:07:57.534781+0000","last_became_active":"2024-03-31T15:07:56.543704+0000","last_became_peered":"2024-03-31T15:07:56.543704+0000","last_unstale":"2024-03-31T15:07:57.534781+0000","last_undegraded":"2024-03-31T15:07:57.534781+0000","last_fullsized":"2024-03-31T15:07:57.534781+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:07:55.513096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:07:55.513096+0000","last_clean_scrub_stamp":"2024-03-31T15:07:55.513096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:56:54.890833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-31T15:07:58.532598+0000","last_change":"2024-03-31T15:07:57.548152+0000","last_active":"2024-03-31T15:07:58.532598+0000","last_peered":"2024-03-31T15:07:58.532598+0000","last_clean":"2024-03-31T15:07:58.532598+0000","last_became_active":"2024-03-31T15:07:57.547873+0000","last_became_peered":"2024-03-31T15:07:57.547873+0000","last_unstale":"2024-03-31T15:07:58.532598+0000","last_undegraded":"2024-03-31T15:07:58.532598+0000","last_fullsized":"2024-03-31T15:07:58.532598+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_clean_scrub_stamp":"2024-03-31T15:07:56.519376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:48:19.134928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":14,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-31T15:07:58.532476+0000","last_change":"2024-03-31T15:07:56.547123+0000","last_active":"2024-03-31T15:07:58.532476+0000","last_peered":"2024-03-31T15:07:58.532476+0000","last_clean":"2024-03-31T15:07:58.532476+0000","last_became_active":"2024-03-31T15:07:56.546742+0000","last_became_peered":"2024-03-31T15:07:56.546742+0000","last_unstale":"2024-03-31T15:07:58.532476+0000","last_undegraded":"2024-03-31T15:07:58.532476+0000","last_fullsized":"2024-03-31T15:07:58.532476+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:07:55.513096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:07:55.513096+0000","last_clean_scrub_stamp":"2024-03-31T15:07:55.513096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:02:09.057185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-31T15:07:57.554567+0000","last_change":"2024-03-31T15:07:57.552085+0000","last_active":"2024-03-31T15:07:57.554567+0000","last_peered":"2024-03-31T15:07:57.554567+0000","last_clean":"2024-03-31T15:07:57.554567+0000","last_became_active":"2024-03-31T15:07:57.551730+0000","last_became_peered":"2024-03-31T15:07:57.551730+0000","last_unstale":"2024-03-31T15:07:57.554567+0000","last_undegraded":"2024-03-31T15:07:57.554567+0000","last_fullsized":"2024-03-31T15:07:57.554567+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_clean_scrub_stamp":"2024-03-31T15:07:56.519376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T16:38:25.730848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-31T15:07:57.540852+0000","last_change":"2024-03-31T15:07:56.539633+0000","last_active":"2024-03-31T15:07:57.540852+0000","last_peered":"2024-03-31T15:07:57.540852+0000","last_clean":"2024-03-31T15:07:57.540852+0000","last_became_active":"2024-03-31T15:07:56.539277+0000","last_became_peered":"2024-03-31T15:07:56.539277+0000","last_unstale":"2024-03-31T15:07:57.540852+0000","last_undegraded":"2024-03-31T15:07:57.540852+0000","last_fullsized":"2024-03-31T15:07:57.540852+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:07:55.513096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:07:55.513096+0000","last_clean_scrub_stamp":"2024-03-31T15:07:55.513096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T17:25:00.920648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-31T15:07:57.560759+0000","last_change":"2024-03-31T15:07:57.555020+0000","last_active":"2024-03-31T15:07:57.560759+0000","last_peered":"2024-03-31T15:07:57.560759+0000","last_clean":"2024-03-31T15:07:57.560759+0000","last_became_active":"2024-03-31T15:07:57.553141+0000","last_became_peered":"2024-03-31T15:07:57.553141+0000","last_unstale":"2024-03-31T15:07:57.560759+0000","last_undegraded":"2024-03-31T15:07:57.560759+0000","last_fullsized":"2024-03-31T15:07:57.560759+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_clean_scrub_stamp":"2024-03-31T15:07:56.519376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:07:51.056989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":14,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-31T15:07:58.538130+0000","last_change":"2024-03-31T15:07:56.543971+0000","last_active":"2024-03-31T15:07:58.538130+0000","last_peered":"2024-03-31T15:07:58.538130+0000","last_clean":"2024-03-31T15:07:58.538130+0000","last_became_active":"2024-03-31T15:07:56.542879+0000","last_became_peered":"2024-03-31T15:07:56.542879+0000","last_unstale":"2024-03-31T15:07:58.538130+0000","last_undegraded":"2024-03-31T15:07:58.538130+0000","last_fullsized":"2024-03-31T15:07:58.538130+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:07:55.513096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:07:55.513096+0000","last_clean_scrub_stamp":"2024-03-31T15:07:55.513096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T16:50:07.834414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-31T15:07:57.549081+0000","last_change":"2024-03-31T15:07:57.544716+0000","last_active":"2024-03-31T15:07:57.549081+0000","last_peered":"2024-03-31T15:07:57.549081+0000","last_clean":"2024-03-31T15:07:57.549081+0000","last_became_active":"2024-03-31T15:07:57.544436+0000","last_became_peered":"2024-03-31T15:07:57.544436+0000","last_unstale":"2024-03-31T15:07:57.549081+0000","last_undegraded":"2024-03-31T15:07:57.549081+0000","last_fullsized":"2024-03-31T15:07:57.549081+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_clean_scrub_stamp":"2024-03-31T15:07:56.519376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T17:34:10.953702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-31T15:07:57.539163+0000","last_change":"2024-03-31T15:07:56.545932+0000","last_active":"2024-03-31T15:07:57.539163+0000","last_peered":"2024-03-31T15:07:57.539163+0000","last_clean":"2024-03-31T15:07:57.539163+0000","last_became_active":"2024-03-31T15:07:56.544251+0000","last_became_peered":"2024-03-31T15:07:56.544251+0000","last_unstale":"2024-03-31T15:07:57.539163+0000","last_undegraded":"2024-03-31T15:07:57.539163+0000","last_fullsized":"2024-03-31T15:07:57.539163+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:07:55.513096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:07:55.513096+0000","last_clean_scrub_stamp":"2024-03-31T15:07:55.513096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T16:43:50.771100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-31T15:07:57.556979+0000","last_change":"2024-03-31T15:07:57.542062+0000","last_active":"2024-03-31T15:07:57.556979+0000","last_peered":"2024-03-31T15:07:57.556979+0000","last_clean":"2024-03-31T15:07:57.556979+0000","last_became_active":"2024-03-31T15:07:57.541772+0000","last_became_peered":"2024-03-31T15:07:57.541772+0000","last_unstale":"2024-03-31T15:07:57.556979+0000","last_undegraded":"2024-03-31T15:07:57.556979+0000","last_fullsized":"2024-03-31T15:07:57.556979+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_clean_scrub_stamp":"2024-03-31T15:07:56.519376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:13:01.409654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.b","version":"66'2","reported_seq":14,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-31T15:07:57.573675+0000","last_change":"2024-03-31T15:07:56.546172+0000","last_active":"2024-03-31T15:07:57.573675+0000","last_peered":"2024-03-31T15:07:57.573675+0000","last_clean":"2024-03-31T15:07:57.573675+0000","last_became_active":"2024-03-31T15:07:56.545826+0000","last_became_peered":"2024-03-31T15:07:56.545826+0000","last_unstale":"2024-03-31T15:07:57.573675+0000","last_undegraded":"2024-03-31T15:07:57.573675+0000","last_fullsized":"2024-03-31T15:07:57.573675+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:07:55.513096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:07:55.513096+0000","last_clean_scrub_stamp":"2024-03-31T15:07:55.513096+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:40:16.591089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":252,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-31T15:07:57.547654+0000","last_change":"2024-03-31T15:03:24.329890+0000","last_active":"2024-03-31T15:07:57.547654+0000","last_peered":"2024-03-31T15:07:57.547654+0000","last_clean":"2024-03-31T15:07:57.547654+0000","last_became_active":"2024-03-31T15:03:24.328882+0000","last_became_peered":"2024-03-31T15:03:24.328882+0000","last_unstale":"2024-03-31T15:07:57.547654+0000","last_undegraded":"2024-03-31T15:07:57.547654+0000","last_fullsized":"2024-03-31T15:07:57.547654+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:22.885535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:22.885535+0000","last_clean_scrub_stamp":"2024-03-31T15:03:22.885535+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:18:29.004296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":10,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509500,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32548,"kb_used_data":1364,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339292,"statfs":{"total":96636764160,"available":96603435008,"internally_reserved":0,"allocated":1396736,"data_stored":922837,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509499,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":780,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":798720,"data_stored":331943,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":15,"seq":64424509498,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27812,"kb_used_data":780,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344028,"statfs":{"total":96636764160,"available":96608284672,"internally_reserved":0,"allocated":798720,"data_stored":331923,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509497,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31976,"kb_used_data":788,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339864,"statfs":{"total":96636764160,"available":96604020736,"internally_reserved":0,"allocated":806912,"data_stored":333465,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542203,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31948,"kb_used_data":764,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339892,"statfs":{"total":96636764160,"available":96604049408,"internally_reserved":0,"allocated":782336,"data_stored":325078,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542203,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31952,"kb_used_data":764,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339888,"statfs":{"total":96636764160,"available":96604045312,"internally_reserved":0,"allocated":782336,"data_stored":325078,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542203,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31952,"kb_used_data":764,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339888,"statfs":{"total":96636764160,"available":96604045312,"internally_reserved":0,"allocated":782336,"data_stored":325078,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542203,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32524,"kb_used_data":1344,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339316,"statfs":{"total":96636764160,"available":96603459584,"internally_reserved":0,"allocated":1376256,"data_stored":915446,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-31T15:08:01.672 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-31T15:08:01.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:01.902+0000 7f3897200640 1 Processor -- start 2024-03-31T15:08:01.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:01.902+0000 7f3897200640 1 -- start start 2024-03-31T15:08:01.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:01.906+0000 7f3897200640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f38900cd4f0 0x7f38900b44e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:01.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:01.906+0000 7f3897200640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f389015fd40 0x7f3890160190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:01.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:01.906+0000 7f3897200640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f38900cd980 con 0x7f389015c1d0 2024-03-31T15:08:01.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:01.906+0000 7f3897200640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f38901606d0 con 0x7f38900cd4f0 2024-03-31T15:08:01.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:01.906+0000 7f3897200640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f38900cfaa0 con 0x7f389015fd40 2024-03-31T15:08:01.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:01.906+0000 7f3895400640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f38900cd4f0 0x7f38900b44e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:01.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:01.906+0000 7f3896800640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f389015fd40 0x7f3890160190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:01.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:01.906+0000 7f3895e00640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f389015c1d0 0x7f389006f5d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.16:51956/0 (socket says 172.21.15.16:51956) 2024-03-31T15:08:01.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:01.906+0000 7f3895e00640 1 -- 172.21.15.16:0/2431274451 learned_addr learned my addr 172.21.15.16:0/2431274451 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:08:01.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:01.906+0000 7f3895400640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f38900cd4f0 0x7f38900b44e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.16:48040/0 (socket says 172.21.15.16:48040) 2024-03-31T15:08:01.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:01.906+0000 7f3895400640 1 -- 172.21.15.16:0/2431274451 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f389015fd40 msgr2=0x7f3890160190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:08:01.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:01.906+0000 7f3895400640 1 --2- 172.21.15.16:0/2431274451 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f389015fd40 0x7f3890160190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:01.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:01.906+0000 7f3895400640 1 -- 172.21.15.16:0/2431274451 >> v1:172.21.15.89:6789/0 conn(0x7f389015c1d0 legacy=0x7f389006f5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:01.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:01.906+0000 7f3894a00640 1 -- 172.21.15.16:0/2431274451 <== mon.1 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 317805176 0 0) 0x7f38900cd980 con 0x7f389015c1d0 2024-03-31T15:08:01.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:01.906+0000 7f3895400640 1 -- 172.21.15.16:0/2431274451 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38900cdc60 con 0x7f38900cd4f0 2024-03-31T15:08:01.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:01.910+0000 7f3895400640 1 --2- 172.21.15.16:0/2431274451 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f38900cd4f0 0x7f38900b44e0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f38800097c0 tx=0x7f388001c470 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3c047f6c26821bba server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:08:01.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:01.910+0000 7f3894a00640 1 -- 172.21.15.16:0/2431274451 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38800a7070 con 0x7f38900cd4f0 2024-03-31T15:08:01.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:01.910+0000 7f3894a00640 1 -- 172.21.15.16:0/2431274451 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f38800b0070 con 0x7f38900cd4f0 2024-03-31T15:08:01.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:01.910+0000 7f3894a00640 1 -- 172.21.15.16:0/2431274451 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38800ab890 con 0x7f38900cd4f0 2024-03-31T15:08:01.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:01.910+0000 7f3897200640 1 -- 172.21.15.16:0/2431274451 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f38900cd4f0 msgr2=0x7f38900b44e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:01.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:01.910+0000 7f3897200640 1 --2- 172.21.15.16:0/2431274451 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f38900cd4f0 0x7f38900b44e0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f38800097c0 tx=0x7f388001c470 comp rx=0 tx=0).stop 2024-03-31T15:08:01.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:01.910+0000 7f3897200640 1 -- 172.21.15.16:0/2431274451 shutdown_connections 2024-03-31T15:08:01.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:01.910+0000 7f3897200640 1 --2- 172.21.15.16:0/2431274451 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f38900cd4f0 0x7f38900b44e0 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:01.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:01.910+0000 7f3897200640 1 --2- 172.21.15.16:0/2431274451 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f389015fd40 0x7f3890160190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:01.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:01.910+0000 7f3897200640 1 -- 172.21.15.16:0/2431274451 >> 172.21.15.16:0/2431274451 conn(0x7f38900b39c0 msgr2=0x7f38900b3dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:08:01.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:01.910+0000 7f3897200640 1 -- 172.21.15.16:0/2431274451 shutdown_connections 2024-03-31T15:08:01.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:01.910+0000 7f3897200640 1 -- 172.21.15.16:0/2431274451 wait complete. 2024-03-31T15:08:01.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:01.914+0000 7f3897200640 1 Processor -- start 2024-03-31T15:08:01.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:01.914+0000 7f3897200640 1 -- start start 2024-03-31T15:08:01.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:01.914+0000 7f3897200640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f389015c1d0 0x7f38900c8fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:01.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:01.914+0000 7f3897200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f389015fd40 0x7f3890157fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:01.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:01.914+0000 7f3897200640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f38900c94e0 0x7f38901584e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:01.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:01.914+0000 7f3897200640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f389006fce0 con 0x7f389015fd40 2024-03-31T15:08:01.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:01.914+0000 7f3897200640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f38900b1420 con 0x7f389015c1d0 2024-03-31T15:08:01.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:01.914+0000 7f3897200640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f3890165400 con 0x7f38900c94e0 2024-03-31T15:08:01.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:01.914+0000 7f3895400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f389015fd40 0x7f3890157fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:01.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:01.914+0000 7f3895400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f389015fd40 0x7f3890157fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51816/0 (socket says 172.21.15.16:51816) 2024-03-31T15:08:01.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:01.914+0000 7f3895400640 1 -- 172.21.15.16:0/410850262 learned_addr learned my addr 172.21.15.16:0/410850262 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:08:01.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:01.914+0000 7f3895e00640 1 --2- 172.21.15.16:0/410850262 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f389015c1d0 0x7f38900c8fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:01.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:01.914+0000 7f3896800640 1 --2- 172.21.15.16:0/410850262 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f38900c94e0 0x7f38901584e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:01.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:01.914+0000 7f3895400640 1 -- 172.21.15.16:0/410850262 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f38900c94e0 msgr2=0x7f38901584e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:01.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:01.914+0000 7f3895400640 1 --2- 172.21.15.16:0/410850262 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f38900c94e0 0x7f38901584e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:01.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:01.914+0000 7f3895400640 1 -- 172.21.15.16:0/410850262 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f389015c1d0 msgr2=0x7f38900c8fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:01.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:01.914+0000 7f3895400640 1 --2- 172.21.15.16:0/410850262 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f389015c1d0 0x7f38900c8fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:01.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:01.914+0000 7f3895400640 1 -- 172.21.15.16:0/410850262 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38901650d0 con 0x7f389015fd40 2024-03-31T15:08:01.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:01.914+0000 7f3895e00640 1 --2- 172.21.15.16:0/410850262 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f389015c1d0 0x7f38900c8fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:08:01.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:01.914+0000 7f3895400640 1 --2- 172.21.15.16:0/410850262 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f389015fd40 0x7f3890157fa0 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7f38800ad040 tx=0x7f388001acb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:08:01.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:01.914+0000 7f388a000640 1 -- 172.21.15.16:0/410850262 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3880006e20 con 0x7f389015fd40 2024-03-31T15:08:01.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:01.914+0000 7f388a000640 1 -- 172.21.15.16:0/410850262 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3880004230 con 0x7f389015fd40 2024-03-31T15:08:01.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:01.914+0000 7f388a000640 1 -- 172.21.15.16:0/410850262 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38800abe60 con 0x7f389015fd40 2024-03-31T15:08:01.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:01.914+0000 7f3897200640 1 -- 172.21.15.16:0/410850262 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f389015b5f0 con 0x7f389015fd40 2024-03-31T15:08:01.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:01.914+0000 7f3897200640 1 -- 172.21.15.16:0/410850262 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f389015b2a0 con 0x7f389015fd40 2024-03-31T15:08:01.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:01.918+0000 7f3897200640 1 -- 172.21.15.16:0/410850262 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f389004f8c0 con 0x7f389015fd40 2024-03-31T15:08:01.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:01.918+0000 7f388a000640 1 -- 172.21.15.16:0/410850262 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f3880003620 con 0x7f389015fd40 2024-03-31T15:08:01.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:01.918+0000 7f388a000640 1 --2- 172.21.15.16:0/410850262 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f38740683b0 0x7f387406a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:01.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:01.918+0000 7f388a000640 1 -- 172.21.15.16:0/410850262 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(67..67 src has 1..67) ==== 5709+0+0 (secure 0 0 0) 0x7f38801210e0 con 0x7f389015fd40 2024-03-31T15:08:01.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:01.922+0000 7f3895e00640 1 --2- 172.21.15.16:0/410850262 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f38740683b0 0x7f387406a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:01.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:01.922+0000 7f3895e00640 1 --2- 172.21.15.16:0/410850262 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f38740683b0 0x7f387406a870 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f3884000f70 tx=0x7f3884007c60 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:08:01.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:01.926+0000 7f388a000640 1 -- 172.21.15.16:0/410850262 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f38800c6030 con 0x7f389015fd40 2024-03-31T15:08:02.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:02.086+0000 7f3897200640 1 -- 172.21.15.16:0/410850262 --> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f3890068ae0 con 0x7f38740683b0 2024-03-31T15:08:02.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:02.094+0000 7f388a000640 1 -- 172.21.15.16:0/410850262 <== mgr.4100 v2:172.21.15.89:6832/1549407273 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252025 (secure 0 0 0) 0x7f3890068ae0 con 0x7f38740683b0 2024-03-31T15:08:02.092 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:08:02.093 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-03-31T15:08:02.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:02.098+0000 7f3897200640 1 -- 172.21.15.16:0/410850262 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f38740683b0 msgr2=0x7f387406a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:02.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:02.098+0000 7f3897200640 1 --2- 172.21.15.16:0/410850262 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f38740683b0 0x7f387406a870 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f3884000f70 tx=0x7f3884007c60 comp rx=0 tx=0).stop 2024-03-31T15:08:02.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:02.098+0000 7f3897200640 1 -- 172.21.15.16:0/410850262 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f389015fd40 msgr2=0x7f3890157fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:02.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:02.098+0000 7f3897200640 1 --2- 172.21.15.16:0/410850262 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f389015fd40 0x7f3890157fa0 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7f38800ad040 tx=0x7f388001acb0 comp rx=0 tx=0).stop 2024-03-31T15:08:02.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:02.098+0000 7f3897200640 1 -- 172.21.15.16:0/410850262 shutdown_connections 2024-03-31T15:08:02.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:02.098+0000 7f3897200640 1 --2- 172.21.15.16:0/410850262 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f38740683b0 0x7f387406a870 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:02.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:02.098+0000 7f3897200640 1 --2- 172.21.15.16:0/410850262 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f38900c94e0 0x7f38901584e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:02.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:02.098+0000 7f3897200640 1 --2- 172.21.15.16:0/410850262 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f389015fd40 0x7f3890157fa0 unknown :-1 s=CLOSED pgs=833 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:02.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:02.098+0000 7f3897200640 1 --2- 172.21.15.16:0/410850262 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f389015c1d0 0x7f38900c8fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:02.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:02.098+0000 7f3897200640 1 -- 172.21.15.16:0/410850262 >> 172.21.15.16:0/410850262 conn(0x7f38900b39c0 msgr2=0x7f389006f5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:08:02.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:02.098+0000 7f3897200640 1 -- 172.21.15.16:0/410850262 shutdown_connections 2024-03-31T15:08:02.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:02.098+0000 7f3897200640 1 -- 172.21.15.16:0/410850262 wait complete. 2024-03-31T15:08:02.113 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":199,"stamp":"2024-03-31T15:08:00.592877+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":248520,"kb_used_data":7348,"kb_used_omap":12,"kb_used_meta":241011,"kb_avail":754726200,"statfs":{"total":773094113280,"available":772839628800,"internally_reserved":0,"allocated":7524352,"data_stored":3810848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":246795824},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":32,"apply_latency_ms":32,"commit_latency_ns":32000000,"apply_latency_ns":32000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.072820"},"pg_stats":[{"pgid":"11.3f","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-31T15:07:57.569441+0000","last_change":"2024-03-31T15:07:57.563260+0000","last_active":"2024-03-31T15:07:57.569441+0000","last_peered":"2024-03-31T15:07:57.569441+0000","last_clean":"2024-03-31T15:07:57.569441+0000","last_became_active":"2024-03-31T15:07:57.562984+0000","last_became_peered":"2024-03-31T15:07:57.562984+0000","last_unstale":"2024-03-31T15:07:57.569441+0000","last_undegraded":"2024-03-31T15:07:57.569441+0000","last_fullsized":"2024-03-31T15:07:57.569441+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_clean_scrub_stamp":"2024-03-31T15:07:56.519376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:00:13.108819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.3e","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-31T15:07:57.546774+0000","last_change":"2024-03-31T15:07:57.542905+0000","last_active":"2024-03-31T15:07:57.546774+0000","last_peered":"2024-03-31T15:07:57.546774+0000","last_clean":"2024-03-31T15:07:57.546774+0000","last_became_active":"2024-03-31T15:07:57.542535+0000","last_became_peered":"2024-03-31T15:07:57.542535+0000","last_unstale":"2024-03-31T15:07:57.546774+0000","last_undegraded":"2024-03-31T15:07:57.546774+0000","last_fullsized":"2024-03-31T15:07:57.546774+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_clean_scrub_stamp":"2024-03-31T15:07:56.519376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:52:54.417267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.3d","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-31T15:07:57.563538+0000","last_change":"2024-03-31T15:07:57.559201+0000","last_active":"2024-03-31T15:07:57.563538+0000","last_peered":"2024-03-31T15:07:57.563538+0000","last_clean":"2024-03-31T15:07:57.563538+0000","last_became_active":"2024-03-31T15:07:57.556878+0000","last_became_peered":"2024-03-31T15:07:57.556878+0000","last_unstale":"2024-03-31T15:07:57.563538+0000","last_undegraded":"2024-03-31T15:07:57.563538+0000","last_fullsized":"2024-03-31T15:07:57.563538+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_clean_scrub_stamp":"2024-03-31T15:07:56.519376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:30:23.703431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.3c","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-31T15:07:57.559981+0000","last_change":"2024-03-31T15:07:57.556153+0000","last_active":"2024-03-31T15:07:57.559981+0000","last_peered":"2024-03-31T15:07:57.559981+0000","last_clean":"2024-03-31T15:07:57.559981+0000","last_became_active":"2024-03-31T15:07:57.555803+0000","last_became_peered":"2024-03-31T15:07:57.555803+0000","last_unstale":"2024-03-31T15:07:57.559981+0000","last_undegraded":"2024-03-31T15:07:57.559981+0000","last_fullsized":"2024-03-31T15:07:57.559981+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_clean_scrub_stamp":"2024-03-31T15:07:56.519376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T17:48:15.273565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.3b","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-31T15:07:58.532607+0000","last_change":"2024-03-31T15:07:57.552827+0000","last_active":"2024-03-31T15:07:58.532607+0000","last_peered":"2024-03-31T15:07:58.532607+0000","last_clean":"2024-03-31T15:07:58.532607+0000","last_became_active":"2024-03-31T15:07:57.552556+0000","last_became_peered":"2024-03-31T15:07:57.552556+0000","last_unstale":"2024-03-31T15:07:58.532607+0000","last_undegraded":"2024-03-31T15:07:58.532607+0000","last_fullsized":"2024-03-31T15:07:58.532607+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_clean_scrub_stamp":"2024-03-31T15:07:56.519376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T16:43:03.773114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3a","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-31T15:07:57.553594+0000","last_change":"2024-03-31T15:07:57.549624+0000","last_active":"2024-03-31T15:07:57.553594+0000","last_peered":"2024-03-31T15:07:57.553594+0000","last_clean":"2024-03-31T15:07:57.553594+0000","last_became_active":"2024-03-31T15:07:57.549335+0000","last_became_peered":"2024-03-31T15:07:57.549335+0000","last_unstale":"2024-03-31T15:07:57.553594+0000","last_undegraded":"2024-03-31T15:07:57.553594+0000","last_fullsized":"2024-03-31T15:07:57.553594+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_clean_scrub_stamp":"2024-03-31T15:07:56.519376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T16:05:52.009841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.39","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-31T15:07:58.555680+0000","last_change":"2024-03-31T15:07:57.560980+0000","last_active":"2024-03-31T15:07:58.555680+0000","last_peered":"2024-03-31T15:07:58.555680+0000","last_clean":"2024-03-31T15:07:58.555680+0000","last_became_active":"2024-03-31T15:07:57.560568+0000","last_became_peered":"2024-03-31T15:07:57.560568+0000","last_unstale":"2024-03-31T15:07:58.555680+0000","last_undegraded":"2024-03-31T15:07:58.555680+0000","last_fullsized":"2024-03-31T15:07:58.555680+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_clean_scrub_stamp":"2024-03-31T15:07:56.519376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:06:47.401497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.38","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-31T15:07:57.566663+0000","last_change":"2024-03-31T15:07:57.563352+0000","last_active":"2024-03-31T15:07:57.566663+0000","last_peered":"2024-03-31T15:07:57.566663+0000","last_clean":"2024-03-31T15:07:57.566663+0000","last_became_active":"2024-03-31T15:07:57.562814+0000","last_became_peered":"2024-03-31T15:07:57.562814+0000","last_unstale":"2024-03-31T15:07:57.566663+0000","last_undegraded":"2024-03-31T15:07:57.566663+0000","last_fullsized":"2024-03-31T15:07:57.566663+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_clean_scrub_stamp":"2024-03-31T15:07:56.519376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:20:41.809192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.37","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-31T15:07:58.532740+0000","last_change":"2024-03-31T15:07:57.550433+0000","last_active":"2024-03-31T15:07:58.532740+0000","last_peered":"2024-03-31T15:07:58.532740+0000","last_clean":"2024-03-31T15:07:58.532740+0000","last_became_active":"2024-03-31T15:07:57.549516+0000","last_became_peered":"2024-03-31T15:07:57.549516+0000","last_unstale":"2024-03-31T15:07:58.532740+0000","last_undegraded":"2024-03-31T15:07:58.532740+0000","last_fullsized":"2024-03-31T15:07:58.532740+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_clean_scrub_stamp":"2024-03-31T15:07:56.519376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:02:17.075008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.36","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-31T15:07:58.539664+0000","last_change":"2024-03-31T15:07:57.556497+0000","last_active":"2024-03-31T15:07:58.539664+0000","last_peered":"2024-03-31T15:07:58.539664+0000","last_clean":"2024-03-31T15:07:58.539664+0000","last_became_active":"2024-03-31T15:07:57.556238+0000","last_became_peered":"2024-03-31T15:07:57.556238+0000","last_unstale":"2024-03-31T15:07:58.539664+0000","last_undegraded":"2024-03-31T15:07:58.539664+0000","last_fullsized":"2024-03-31T15:07:58.539664+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_clean_scrub_stamp":"2024-03-31T15:07:56.519376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T16:08:00.122222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.35","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-31T15:07:57.569462+0000","last_change":"2024-03-31T15:07:57.564158+0000","last_active":"2024-03-31T15:07:57.569462+0000","last_peered":"2024-03-31T15:07:57.569462+0000","last_clean":"2024-03-31T15:07:57.569462+0000","last_became_active":"2024-03-31T15:07:57.563865+0000","last_became_peered":"2024-03-31T15:07:57.563865+0000","last_unstale":"2024-03-31T15:07:57.569462+0000","last_undegraded":"2024-03-31T15:07:57.569462+0000","last_fullsized":"2024-03-31T15:07:57.569462+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_clean_scrub_stamp":"2024-03-31T15:07:56.519376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T16:17:18.266469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.34","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-31T15:07:57.560026+0000","last_change":"2024-03-31T15:07:57.557329+0000","last_active":"2024-03-31T15:07:57.560026+0000","last_peered":"2024-03-31T15:07:57.560026+0000","last_clean":"2024-03-31T15:07:57.560026+0000","last_became_active":"2024-03-31T15:07:57.556873+0000","last_became_peered":"2024-03-31T15:07:57.556873+0000","last_unstale":"2024-03-31T15:07:57.560026+0000","last_undegraded":"2024-03-31T15:07:57.560026+0000","last_fullsized":"2024-03-31T15:07:57.560026+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_clean_scrub_stamp":"2024-03-31T15:07:56.519376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:40:17.597369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.33","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-31T15:07:57.551892+0000","last_change":"2024-03-31T15:07:57.547221+0000","last_active":"2024-03-31T15:07:57.551892+0000","last_peered":"2024-03-31T15:07:57.551892+0000","last_clean":"2024-03-31T15:07:57.551892+0000","last_became_active":"2024-03-31T15:07:57.546609+0000","last_became_peered":"2024-03-31T15:07:57.546609+0000","last_unstale":"2024-03-31T15:07:57.551892+0000","last_undegraded":"2024-03-31T15:07:57.551892+0000","last_fullsized":"2024-03-31T15:07:57.551892+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_clean_scrub_stamp":"2024-03-31T15:07:56.519376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:12:37.424035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.32","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-31T15:07:58.532459+0000","last_change":"2024-03-31T15:07:57.553238+0000","last_active":"2024-03-31T15:07:58.532459+0000","last_peered":"2024-03-31T15:07:58.532459+0000","last_clean":"2024-03-31T15:07:58.532459+0000","last_became_active":"2024-03-31T15:07:57.552991+0000","last_became_peered":"2024-03-31T15:07:57.552991+0000","last_unstale":"2024-03-31T15:07:58.532459+0000","last_undegraded":"2024-03-31T15:07:58.532459+0000","last_fullsized":"2024-03-31T15:07:58.532459+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_clean_scrub_stamp":"2024-03-31T15:07:56.519376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T17:32:16.673534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.31","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-31T15:07:57.550186+0000","last_change":"2024-03-31T15:07:57.544771+0000","last_active":"2024-03-31T15:07:57.550186+0000","last_peered":"2024-03-31T15:07:57.550186+0000","last_clean":"2024-03-31T15:07:57.550186+0000","last_became_active":"2024-03-31T15:07:57.544485+0000","last_became_peered":"2024-03-31T15:07:57.544485+0000","last_unstale":"2024-03-31T15:07:57.550186+0000","last_undegraded":"2024-03-31T15:07:57.550186+0000","last_fullsized":"2024-03-31T15:07:57.550186+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_clean_scrub_stamp":"2024-03-31T15:07:56.519376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:43:57.698889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.30","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-31T15:07:58.538552+0000","last_change":"2024-03-31T15:07:57.548359+0000","last_active":"2024-03-31T15:07:58.538552+0000","last_peered":"2024-03-31T15:07:58.538552+0000","last_clean":"2024-03-31T15:07:58.538552+0000","last_became_active":"2024-03-31T15:07:57.546175+0000","last_became_peered":"2024-03-31T15:07:57.546175+0000","last_unstale":"2024-03-31T15:07:58.538552+0000","last_undegraded":"2024-03-31T15:07:58.538552+0000","last_fullsized":"2024-03-31T15:07:58.538552+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_clean_scrub_stamp":"2024-03-31T15:07:56.519376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:43:39.968084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2f","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-31T15:07:58.532732+0000","last_change":"2024-03-31T15:07:57.550632+0000","last_active":"2024-03-31T15:07:58.532732+0000","last_peered":"2024-03-31T15:07:58.532732+0000","last_clean":"2024-03-31T15:07:58.532732+0000","last_became_active":"2024-03-31T15:07:57.549878+0000","last_became_peered":"2024-03-31T15:07:57.549878+0000","last_unstale":"2024-03-31T15:07:58.532732+0000","last_undegraded":"2024-03-31T15:07:58.532732+0000","last_fullsized":"2024-03-31T15:07:58.532732+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_clean_scrub_stamp":"2024-03-31T15:07:56.519376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:54:20.274859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.2e","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-31T15:07:57.559876+0000","last_change":"2024-03-31T15:07:57.555611+0000","last_active":"2024-03-31T15:07:57.559876+0000","last_peered":"2024-03-31T15:07:57.559876+0000","last_clean":"2024-03-31T15:07:57.559876+0000","last_became_active":"2024-03-31T15:07:57.555321+0000","last_became_peered":"2024-03-31T15:07:57.555321+0000","last_unstale":"2024-03-31T15:07:57.559876+0000","last_undegraded":"2024-03-31T15:07:57.559876+0000","last_fullsized":"2024-03-31T15:07:57.559876+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_clean_scrub_stamp":"2024-03-31T15:07:56.519376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T16:43:51.777380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.2d","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-31T15:07:57.564413+0000","last_change":"2024-03-31T15:07:57.559406+0000","last_active":"2024-03-31T15:07:57.564413+0000","last_peered":"2024-03-31T15:07:57.564413+0000","last_clean":"2024-03-31T15:07:57.564413+0000","last_became_active":"2024-03-31T15:07:57.556888+0000","last_became_peered":"2024-03-31T15:07:57.556888+0000","last_unstale":"2024-03-31T15:07:57.564413+0000","last_undegraded":"2024-03-31T15:07:57.564413+0000","last_fullsized":"2024-03-31T15:07:57.564413+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_clean_scrub_stamp":"2024-03-31T15:07:56.519376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:27:24.034975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.2c","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-31T15:07:57.551847+0000","last_change":"2024-03-31T15:07:57.546466+0000","last_active":"2024-03-31T15:07:57.551847+0000","last_peered":"2024-03-31T15:07:57.551847+0000","last_clean":"2024-03-31T15:07:57.551847+0000","last_became_active":"2024-03-31T15:07:57.546141+0000","last_became_peered":"2024-03-31T15:07:57.546141+0000","last_unstale":"2024-03-31T15:07:57.551847+0000","last_undegraded":"2024-03-31T15:07:57.551847+0000","last_fullsized":"2024-03-31T15:07:57.551847+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_clean_scrub_stamp":"2024-03-31T15:07:56.519376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T15:20:13.726100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2b","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-31T15:07:58.539818+0000","last_change":"2024-03-31T15:07:57.551433+0000","last_active":"2024-03-31T15:07:58.539818+0000","last_peered":"2024-03-31T15:07:58.539818+0000","last_clean":"2024-03-31T15:07:58.539818+0000","last_became_active":"2024-03-31T15:07:57.551139+0000","last_became_peered":"2024-03-31T15:07:57.551139+0000","last_unstale":"2024-03-31T15:07:58.539818+0000","last_undegraded":"2024-03-31T15:07:58.539818+0000","last_fullsized":"2024-03-31T15:07:58.539818+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_clean_scrub_stamp":"2024-03-31T15:07:56.519376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:05:26.269858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2a","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-31T15:07:58.538204+0000","last_change":"2024-03-31T15:07:57.555794+0000","last_active":"2024-03-31T15:07:58.538204+0000","last_peered":"2024-03-31T15:07:58.538204+0000","last_clean":"2024-03-31T15:07:58.538204+0000","last_became_active":"2024-03-31T15:07:57.555443+0000","last_became_peered":"2024-03-31T15:07:57.555443+0000","last_unstale":"2024-03-31T15:07:58.538204+0000","last_undegraded":"2024-03-31T15:07:58.538204+0000","last_fullsized":"2024-03-31T15:07:58.538204+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_clean_scrub_stamp":"2024-03-31T15:07:56.519376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T15:53:23.385556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.29","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-31T15:07:57.564070+0000","last_change":"2024-03-31T15:07:57.557824+0000","last_active":"2024-03-31T15:07:57.564070+0000","last_peered":"2024-03-31T15:07:57.564070+0000","last_clean":"2024-03-31T15:07:57.564070+0000","last_became_active":"2024-03-31T15:07:57.557441+0000","last_became_peered":"2024-03-31T15:07:57.557441+0000","last_unstale":"2024-03-31T15:07:57.564070+0000","last_undegraded":"2024-03-31T15:07:57.564070+0000","last_fullsized":"2024-03-31T15:07:57.564070+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_clean_scrub_stamp":"2024-03-31T15:07:56.519376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:20:29.793652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.28","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-31T15:07:57.562393+0000","last_change":"2024-03-31T15:07:57.555429+0000","last_active":"2024-03-31T15:07:57.562393+0000","last_peered":"2024-03-31T15:07:57.562393+0000","last_clean":"2024-03-31T15:07:57.562393+0000","last_became_active":"2024-03-31T15:07:57.555146+0000","last_became_peered":"2024-03-31T15:07:57.555146+0000","last_unstale":"2024-03-31T15:07:57.562393+0000","last_undegraded":"2024-03-31T15:07:57.562393+0000","last_fullsized":"2024-03-31T15:07:57.562393+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_clean_scrub_stamp":"2024-03-31T15:07:56.519376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T19:32:15.203110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.27","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-31T15:07:58.539623+0000","last_change":"2024-03-31T15:07:57.555518+0000","last_active":"2024-03-31T15:07:58.539623+0000","last_peered":"2024-03-31T15:07:58.539623+0000","last_clean":"2024-03-31T15:07:58.539623+0000","last_became_active":"2024-03-31T15:07:57.555013+0000","last_became_peered":"2024-03-31T15:07:57.555013+0000","last_unstale":"2024-03-31T15:07:58.539623+0000","last_undegraded":"2024-03-31T15:07:58.539623+0000","last_fullsized":"2024-03-31T15:07:58.539623+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_clean_scrub_stamp":"2024-03-31T15:07:56.519376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T16:29:14.898390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.26","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-31T15:07:58.555719+0000","last_change":"2024-03-31T15:07:57.546819+0000","last_active":"2024-03-31T15:07:58.555719+0000","last_peered":"2024-03-31T15:07:58.555719+0000","last_clean":"2024-03-31T15:07:58.555719+0000","last_became_active":"2024-03-31T15:07:57.546514+0000","last_became_peered":"2024-03-31T15:07:57.546514+0000","last_unstale":"2024-03-31T15:07:58.555719+0000","last_undegraded":"2024-03-31T15:07:58.555719+0000","last_fullsized":"2024-03-31T15:07:58.555719+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_clean_scrub_stamp":"2024-03-31T15:07:56.519376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T15:54:31.665794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.25","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-31T15:07:57.564531+0000","last_change":"2024-03-31T15:07:57.560074+0000","last_active":"2024-03-31T15:07:57.564531+0000","last_peered":"2024-03-31T15:07:57.564531+0000","last_clean":"2024-03-31T15:07:57.564531+0000","last_became_active":"2024-03-31T15:07:57.559235+0000","last_became_peered":"2024-03-31T15:07:57.559235+0000","last_unstale":"2024-03-31T15:07:57.564531+0000","last_undegraded":"2024-03-31T15:07:57.564531+0000","last_fullsized":"2024-03-31T15:07:57.564531+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_clean_scrub_stamp":"2024-03-31T15:07:56.519376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T15:51:10.452582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.24","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-31T15:07:58.539610+0000","last_change":"2024-03-31T15:07:57.546722+0000","last_active":"2024-03-31T15:07:58.539610+0000","last_peered":"2024-03-31T15:07:58.539610+0000","last_clean":"2024-03-31T15:07:58.539610+0000","last_became_active":"2024-03-31T15:07:57.546352+0000","last_became_peered":"2024-03-31T15:07:57.546352+0000","last_unstale":"2024-03-31T15:07:58.539610+0000","last_undegraded":"2024-03-31T15:07:58.539610+0000","last_fullsized":"2024-03-31T15:07:58.539610+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_clean_scrub_stamp":"2024-03-31T15:07:56.519376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:37:04.228530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.23","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-31T15:07:57.558314+0000","last_change":"2024-03-31T15:07:57.549491+0000","last_active":"2024-03-31T15:07:57.558314+0000","last_peered":"2024-03-31T15:07:57.558314+0000","last_clean":"2024-03-31T15:07:57.558314+0000","last_became_active":"2024-03-31T15:07:57.549125+0000","last_became_peered":"2024-03-31T15:07:57.549125+0000","last_unstale":"2024-03-31T15:07:57.558314+0000","last_undegraded":"2024-03-31T15:07:57.558314+0000","last_fullsized":"2024-03-31T15:07:57.558314+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_clean_scrub_stamp":"2024-03-31T15:07:56.519376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:02:11.074846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.22","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-31T15:07:58.555330+0000","last_change":"2024-03-31T15:07:57.543561+0000","last_active":"2024-03-31T15:07:58.555330+0000","last_peered":"2024-03-31T15:07:58.555330+0000","last_clean":"2024-03-31T15:07:58.555330+0000","last_became_active":"2024-03-31T15:07:57.542432+0000","last_became_peered":"2024-03-31T15:07:57.542432+0000","last_unstale":"2024-03-31T15:07:58.555330+0000","last_undegraded":"2024-03-31T15:07:58.555330+0000","last_fullsized":"2024-03-31T15:07:58.555330+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_clean_scrub_stamp":"2024-03-31T15:07:56.519376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:00:13.108819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.21","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-31T15:07:58.555431+0000","last_change":"2024-03-31T15:07:57.544243+0000","last_active":"2024-03-31T15:07:58.555431+0000","last_peered":"2024-03-31T15:07:58.555431+0000","last_clean":"2024-03-31T15:07:58.555431+0000","last_became_active":"2024-03-31T15:07:57.542605+0000","last_became_peered":"2024-03-31T15:07:57.542605+0000","last_unstale":"2024-03-31T15:07:58.555431+0000","last_undegraded":"2024-03-31T15:07:58.555431+0000","last_fullsized":"2024-03-31T15:07:58.555431+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_clean_scrub_stamp":"2024-03-31T15:07:56.519376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:57:34.405235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.20","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-31T15:07:57.551850+0000","last_change":"2024-03-31T15:07:57.545760+0000","last_active":"2024-03-31T15:07:57.551850+0000","last_peered":"2024-03-31T15:07:57.551850+0000","last_clean":"2024-03-31T15:07:57.551850+0000","last_became_active":"2024-03-31T15:07:57.545479+0000","last_became_peered":"2024-03-31T15:07:57.545479+0000","last_unstale":"2024-03-31T15:07:57.551850+0000","last_undegraded":"2024-03-31T15:07:57.551850+0000","last_fullsized":"2024-03-31T15:07:57.551850+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_clean_scrub_stamp":"2024-03-31T15:07:56.519376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:06:58.534902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.1e","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-31T15:07:57.571297+0000","last_change":"2024-03-31T15:07:57.567648+0000","last_active":"2024-03-31T15:07:57.571297+0000","last_peered":"2024-03-31T15:07:57.571297+0000","last_clean":"2024-03-31T15:07:57.571297+0000","last_became_active":"2024-03-31T15:07:57.565622+0000","last_became_peered":"2024-03-31T15:07:57.565622+0000","last_unstale":"2024-03-31T15:07:57.571297+0000","last_undegraded":"2024-03-31T15:07:57.571297+0000","last_fullsized":"2024-03-31T15:07:57.571297+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_clean_scrub_stamp":"2024-03-31T15:07:56.519376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:18:05.654205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1f","version":"66'4","reported_seq":16,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-31T15:07:57.568905+0000","last_change":"2024-03-31T15:07:56.553471+0000","last_active":"2024-03-31T15:07:57.568905+0000","last_peered":"2024-03-31T15:07:57.568905+0000","last_clean":"2024-03-31T15:07:57.568905+0000","last_became_active":"2024-03-31T15:07:56.553009+0000","last_became_peered":"2024-03-31T15:07:56.553009+0000","last_unstale":"2024-03-31T15:07:57.568905+0000","last_undegraded":"2024-03-31T15:07:57.568905+0000","last_fullsized":"2024-03-31T15:07:57.568905+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:07:55.513096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:07:55.513096+0000","last_clean_scrub_stamp":"2024-03-31T15:07:55.513096+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:18:23.348865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.1f","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-31T15:07:58.532807+0000","last_change":"2024-03-31T15:07:57.550188+0000","last_active":"2024-03-31T15:07:58.532807+0000","last_peered":"2024-03-31T15:07:58.532807+0000","last_clean":"2024-03-31T15:07:58.532807+0000","last_became_active":"2024-03-31T15:07:57.549188+0000","last_became_peered":"2024-03-31T15:07:57.549188+0000","last_unstale":"2024-03-31T15:07:58.532807+0000","last_undegraded":"2024-03-31T15:07:58.532807+0000","last_fullsized":"2024-03-31T15:07:58.532807+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_clean_scrub_stamp":"2024-03-31T15:07:56.519376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T16:57:04.346527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.1e","version":"66'1","reported_seq":13,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-31T15:07:57.562594+0000","last_change":"2024-03-31T15:07:56.548168+0000","last_active":"2024-03-31T15:07:57.562594+0000","last_peered":"2024-03-31T15:07:57.562594+0000","last_clean":"2024-03-31T15:07:57.562594+0000","last_became_active":"2024-03-31T15:07:56.547846+0000","last_became_peered":"2024-03-31T15:07:56.547846+0000","last_unstale":"2024-03-31T15:07:57.562594+0000","last_undegraded":"2024-03-31T15:07:57.562594+0000","last_fullsized":"2024-03-31T15:07:57.562594+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:07:55.513096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:07:55.513096+0000","last_clean_scrub_stamp":"2024-03-31T15:07:55.513096+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T19:24:36.128081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1c","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-31T15:07:57.554456+0000","last_change":"2024-03-31T15:07:57.551178+0000","last_active":"2024-03-31T15:07:57.554456+0000","last_peered":"2024-03-31T15:07:57.554456+0000","last_clean":"2024-03-31T15:07:57.554456+0000","last_became_active":"2024-03-31T15:07:57.550523+0000","last_became_peered":"2024-03-31T15:07:57.550523+0000","last_unstale":"2024-03-31T15:07:57.554456+0000","last_undegraded":"2024-03-31T15:07:57.554456+0000","last_fullsized":"2024-03-31T15:07:57.554456+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_clean_scrub_stamp":"2024-03-31T15:07:56.519376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:29:53.308887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.1d","version":"66'2","reported_seq":16,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-31T15:07:58.539470+0000","last_change":"2024-03-31T15:07:56.545113+0000","last_active":"2024-03-31T15:07:58.539470+0000","last_peered":"2024-03-31T15:07:58.539470+0000","last_clean":"2024-03-31T15:07:58.539470+0000","last_became_active":"2024-03-31T15:07:56.542451+0000","last_became_peered":"2024-03-31T15:07:56.542451+0000","last_unstale":"2024-03-31T15:07:58.539470+0000","last_undegraded":"2024-03-31T15:07:58.539470+0000","last_fullsized":"2024-03-31T15:07:58.539470+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:07:55.513096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:07:55.513096+0000","last_clean_scrub_stamp":"2024-03-31T15:07:55.513096+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:08:42.900813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1d","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-31T15:07:57.571246+0000","last_change":"2024-03-31T15:07:57.565338+0000","last_active":"2024-03-31T15:07:57.571246+0000","last_peered":"2024-03-31T15:07:57.571246+0000","last_clean":"2024-03-31T15:07:57.571246+0000","last_became_active":"2024-03-31T15:07:57.564994+0000","last_became_peered":"2024-03-31T15:07:57.564994+0000","last_unstale":"2024-03-31T15:07:57.571246+0000","last_undegraded":"2024-03-31T15:07:57.571246+0000","last_fullsized":"2024-03-31T15:07:57.571246+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_clean_scrub_stamp":"2024-03-31T15:07:56.519376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:12:19.344476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1c","version":"66'2","reported_seq":14,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-31T15:07:57.573594+0000","last_change":"2024-03-31T15:07:56.546907+0000","last_active":"2024-03-31T15:07:57.573594+0000","last_peered":"2024-03-31T15:07:57.573594+0000","last_clean":"2024-03-31T15:07:57.573594+0000","last_became_active":"2024-03-31T15:07:56.543724+0000","last_became_peered":"2024-03-31T15:07:56.543724+0000","last_unstale":"2024-03-31T15:07:57.573594+0000","last_undegraded":"2024-03-31T15:07:57.573594+0000","last_fullsized":"2024-03-31T15:07:57.573594+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:07:55.513096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:07:55.513096+0000","last_clean_scrub_stamp":"2024-03-31T15:07:55.513096+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T19:36:35.238243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.1a","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-31T15:07:58.555881+0000","last_change":"2024-03-31T15:07:57.546007+0000","last_active":"2024-03-31T15:07:58.555881+0000","last_peered":"2024-03-31T15:07:58.555881+0000","last_clean":"2024-03-31T15:07:58.555881+0000","last_became_active":"2024-03-31T15:07:57.545744+0000","last_became_peered":"2024-03-31T15:07:57.545744+0000","last_unstale":"2024-03-31T15:07:58.555881+0000","last_undegraded":"2024-03-31T15:07:58.555881+0000","last_fullsized":"2024-03-31T15:07:58.555881+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_clean_scrub_stamp":"2024-03-31T15:07:56.519376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:10:54.127697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.1b","version":"66'1","reported_seq":13,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-31T15:07:57.569537+0000","last_change":"2024-03-31T15:07:56.544488+0000","last_active":"2024-03-31T15:07:57.569537+0000","last_peered":"2024-03-31T15:07:57.569537+0000","last_clean":"2024-03-31T15:07:57.569537+0000","last_became_active":"2024-03-31T15:07:56.544088+0000","last_became_peered":"2024-03-31T15:07:56.544088+0000","last_unstale":"2024-03-31T15:07:57.569537+0000","last_undegraded":"2024-03-31T15:07:57.569537+0000","last_fullsized":"2024-03-31T15:07:57.569537+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:07:55.513096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:07:55.513096+0000","last_clean_scrub_stamp":"2024-03-31T15:07:55.513096+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T16:21:08.149844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.1b","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-31T15:07:58.539958+0000","last_change":"2024-03-31T15:07:57.554226+0000","last_active":"2024-03-31T15:07:58.539958+0000","last_peered":"2024-03-31T15:07:58.539958+0000","last_clean":"2024-03-31T15:07:58.539958+0000","last_became_active":"2024-03-31T15:07:57.553938+0000","last_became_peered":"2024-03-31T15:07:57.553938+0000","last_unstale":"2024-03-31T15:07:58.539958+0000","last_undegraded":"2024-03-31T15:07:58.539958+0000","last_fullsized":"2024-03-31T15:07:58.539958+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_clean_scrub_stamp":"2024-03-31T15:07:56.519376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T17:43:07.375959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.1a","version":"66'1","reported_seq":13,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-31T15:07:57.573594+0000","last_change":"2024-03-31T15:07:56.547321+0000","last_active":"2024-03-31T15:07:57.573594+0000","last_peered":"2024-03-31T15:07:57.573594+0000","last_clean":"2024-03-31T15:07:57.573594+0000","last_became_active":"2024-03-31T15:07:56.546434+0000","last_became_peered":"2024-03-31T15:07:56.546434+0000","last_unstale":"2024-03-31T15:07:57.573594+0000","last_undegraded":"2024-03-31T15:07:57.573594+0000","last_fullsized":"2024-03-31T15:07:57.573594+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:07:55.513096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:07:55.513096+0000","last_clean_scrub_stamp":"2024-03-31T15:07:55.513096+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:07:50.050709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-31T15:07:58.532032+0000","last_change":"2024-03-31T15:07:57.543472+0000","last_active":"2024-03-31T15:07:58.532032+0000","last_peered":"2024-03-31T15:07:58.532032+0000","last_clean":"2024-03-31T15:07:58.532032+0000","last_became_active":"2024-03-31T15:07:57.543161+0000","last_became_peered":"2024-03-31T15:07:57.543161+0000","last_unstale":"2024-03-31T15:07:58.532032+0000","last_undegraded":"2024-03-31T15:07:58.532032+0000","last_fullsized":"2024-03-31T15:07:58.532032+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_clean_scrub_stamp":"2024-03-31T15:07:56.519376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:25:16.321644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.19","version":"0'0","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-31T15:07:57.539053+0000","last_change":"2024-03-31T15:07:56.540446+0000","last_active":"2024-03-31T15:07:57.539053+0000","last_peered":"2024-03-31T15:07:57.539053+0000","last_clean":"2024-03-31T15:07:57.539053+0000","last_became_active":"2024-03-31T15:07:56.540082+0000","last_became_peered":"2024-03-31T15:07:56.540082+0000","last_unstale":"2024-03-31T15:07:57.539053+0000","last_undegraded":"2024-03-31T15:07:57.539053+0000","last_fullsized":"2024-03-31T15:07:57.539053+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:07:55.513096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:07:55.513096+0000","last_clean_scrub_stamp":"2024-03-31T15:07:55.513096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:27:16.077645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-31T15:07:57.552198+0000","last_change":"2024-03-31T15:07:57.551029+0000","last_active":"2024-03-31T15:07:57.552198+0000","last_peered":"2024-03-31T15:07:57.552198+0000","last_clean":"2024-03-31T15:07:57.552198+0000","last_became_active":"2024-03-31T15:07:57.550575+0000","last_became_peered":"2024-03-31T15:07:57.550575+0000","last_unstale":"2024-03-31T15:07:57.552198+0000","last_undegraded":"2024-03-31T15:07:57.552198+0000","last_fullsized":"2024-03-31T15:07:57.552198+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_clean_scrub_stamp":"2024-03-31T15:07:56.519376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T16:14:04.493893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.18","version":"0'0","reported_seq":14,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-31T15:07:58.532035+0000","last_change":"2024-03-31T15:07:56.543341+0000","last_active":"2024-03-31T15:07:58.532035+0000","last_peered":"2024-03-31T15:07:58.532035+0000","last_clean":"2024-03-31T15:07:58.532035+0000","last_became_active":"2024-03-31T15:07:56.541685+0000","last_became_peered":"2024-03-31T15:07:56.541685+0000","last_unstale":"2024-03-31T15:07:58.532035+0000","last_undegraded":"2024-03-31T15:07:58.532035+0000","last_fullsized":"2024-03-31T15:07:58.532035+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:07:55.513096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:07:55.513096+0000","last_clean_scrub_stamp":"2024-03-31T15:07:55.513096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:25:20.169134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-31T15:07:58.555489+0000","last_change":"2024-03-31T15:07:57.549590+0000","last_active":"2024-03-31T15:07:58.555489+0000","last_peered":"2024-03-31T15:07:58.555489+0000","last_clean":"2024-03-31T15:07:58.555489+0000","last_became_active":"2024-03-31T15:07:57.549224+0000","last_became_peered":"2024-03-31T15:07:57.549224+0000","last_unstale":"2024-03-31T15:07:58.555489+0000","last_undegraded":"2024-03-31T15:07:58.555489+0000","last_fullsized":"2024-03-31T15:07:58.555489+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_clean_scrub_stamp":"2024-03-31T15:07:56.519376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:12:19.344476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.17","version":"0'0","reported_seq":14,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-31T15:07:58.539743+0000","last_change":"2024-03-31T15:07:56.546737+0000","last_active":"2024-03-31T15:07:58.539743+0000","last_peered":"2024-03-31T15:07:58.539743+0000","last_clean":"2024-03-31T15:07:58.539743+0000","last_became_active":"2024-03-31T15:07:56.546153+0000","last_became_peered":"2024-03-31T15:07:56.546153+0000","last_unstale":"2024-03-31T15:07:58.539743+0000","last_undegraded":"2024-03-31T15:07:58.539743+0000","last_fullsized":"2024-03-31T15:07:58.539743+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:07:55.513096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:07:55.513096+0000","last_clean_scrub_stamp":"2024-03-31T15:07:55.513096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:17:08.509358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-31T15:07:57.571223+0000","last_change":"2024-03-31T15:07:57.565169+0000","last_active":"2024-03-31T15:07:57.571223+0000","last_peered":"2024-03-31T15:07:57.571223+0000","last_clean":"2024-03-31T15:07:57.571223+0000","last_became_active":"2024-03-31T15:07:57.564779+0000","last_became_peered":"2024-03-31T15:07:57.564779+0000","last_unstale":"2024-03-31T15:07:57.571223+0000","last_undegraded":"2024-03-31T15:07:57.571223+0000","last_fullsized":"2024-03-31T15:07:57.571223+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_clean_scrub_stamp":"2024-03-31T15:07:56.519376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:57:34.405235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.16","version":"0'0","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-31T15:07:57.539998+0000","last_change":"2024-03-31T15:07:56.546801+0000","last_active":"2024-03-31T15:07:57.539998+0000","last_peered":"2024-03-31T15:07:57.539998+0000","last_clean":"2024-03-31T15:07:57.539998+0000","last_became_active":"2024-03-31T15:07:56.544142+0000","last_became_peered":"2024-03-31T15:07:56.544142+0000","last_unstale":"2024-03-31T15:07:57.539998+0000","last_undegraded":"2024-03-31T15:07:57.539998+0000","last_fullsized":"2024-03-31T15:07:57.539998+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:07:55.513096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:07:55.513096+0000","last_clean_scrub_stamp":"2024-03-31T15:07:55.513096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:55:12.611335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.14","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-31T15:07:57.553745+0000","last_change":"2024-03-31T15:07:57.549734+0000","last_active":"2024-03-31T15:07:57.553745+0000","last_peered":"2024-03-31T15:07:57.553745+0000","last_clean":"2024-03-31T15:07:57.553745+0000","last_became_active":"2024-03-31T15:07:57.549441+0000","last_became_peered":"2024-03-31T15:07:57.549441+0000","last_unstale":"2024-03-31T15:07:57.553745+0000","last_undegraded":"2024-03-31T15:07:57.553745+0000","last_fullsized":"2024-03-31T15:07:57.553745+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_clean_scrub_stamp":"2024-03-31T15:07:56.519376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T17:20:07.399517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.15","version":"66'3","reported_seq":15,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-31T15:07:57.570571+0000","last_change":"2024-03-31T15:07:56.547818+0000","last_active":"2024-03-31T15:07:57.570571+0000","last_peered":"2024-03-31T15:07:57.570571+0000","last_clean":"2024-03-31T15:07:57.570571+0000","last_became_active":"2024-03-31T15:07:56.546707+0000","last_became_peered":"2024-03-31T15:07:56.546707+0000","last_unstale":"2024-03-31T15:07:57.570571+0000","last_undegraded":"2024-03-31T15:07:57.570571+0000","last_fullsized":"2024-03-31T15:07:57.570571+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:07:55.513096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:07:55.513096+0000","last_clean_scrub_stamp":"2024-03-31T15:07:55.513096+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:14:52.090549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.15","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-31T15:07:58.532417+0000","last_change":"2024-03-31T15:07:57.557344+0000","last_active":"2024-03-31T15:07:58.532417+0000","last_peered":"2024-03-31T15:07:58.532417+0000","last_clean":"2024-03-31T15:07:58.532417+0000","last_became_active":"2024-03-31T15:07:57.556402+0000","last_became_peered":"2024-03-31T15:07:57.556402+0000","last_unstale":"2024-03-31T15:07:58.532417+0000","last_undegraded":"2024-03-31T15:07:58.532417+0000","last_fullsized":"2024-03-31T15:07:58.532417+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_clean_scrub_stamp":"2024-03-31T15:07:56.519376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T15:43:48.032189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.14","version":"66'6","reported_seq":18,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-31T15:07:57.572398+0000","last_change":"2024-03-31T15:07:56.545850+0000","last_active":"2024-03-31T15:07:57.572398+0000","last_peered":"2024-03-31T15:07:57.572398+0000","last_clean":"2024-03-31T15:07:57.572398+0000","last_became_active":"2024-03-31T15:07:56.545561+0000","last_became_peered":"2024-03-31T15:07:56.545561+0000","last_unstale":"2024-03-31T15:07:57.572398+0000","last_undegraded":"2024-03-31T15:07:57.572398+0000","last_fullsized":"2024-03-31T15:07:57.572398+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:07:55.513096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:07:55.513096+0000","last_clean_scrub_stamp":"2024-03-31T15:07:55.513096+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T15:25:09.572264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-31T15:07:57.565295+0000","last_change":"2024-03-31T15:07:57.560749+0000","last_active":"2024-03-31T15:07:57.565295+0000","last_peered":"2024-03-31T15:07:57.565295+0000","last_clean":"2024-03-31T15:07:57.565295+0000","last_became_active":"2024-03-31T15:07:57.560377+0000","last_became_peered":"2024-03-31T15:07:57.560377+0000","last_unstale":"2024-03-31T15:07:57.565295+0000","last_undegraded":"2024-03-31T15:07:57.565295+0000","last_fullsized":"2024-03-31T15:07:57.565295+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_clean_scrub_stamp":"2024-03-31T15:07:56.519376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:10:44.422861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.13","version":"66'2","reported_seq":14,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-31T15:07:57.574397+0000","last_change":"2024-03-31T15:07:56.551058+0000","last_active":"2024-03-31T15:07:57.574397+0000","last_peered":"2024-03-31T15:07:57.574397+0000","last_clean":"2024-03-31T15:07:57.574397+0000","last_became_active":"2024-03-31T15:07:56.550793+0000","last_became_peered":"2024-03-31T15:07:56.550793+0000","last_unstale":"2024-03-31T15:07:57.574397+0000","last_undegraded":"2024-03-31T15:07:57.574397+0000","last_fullsized":"2024-03-31T15:07:57.574397+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:07:55.513096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:07:55.513096+0000","last_clean_scrub_stamp":"2024-03-31T15:07:55.513096+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T19:36:35.238243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.13","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-31T15:07:58.555415+0000","last_change":"2024-03-31T15:07:57.550422+0000","last_active":"2024-03-31T15:07:58.555415+0000","last_peered":"2024-03-31T15:07:58.555415+0000","last_clean":"2024-03-31T15:07:58.555415+0000","last_became_active":"2024-03-31T15:07:57.550127+0000","last_became_peered":"2024-03-31T15:07:57.550127+0000","last_unstale":"2024-03-31T15:07:58.555415+0000","last_undegraded":"2024-03-31T15:07:58.555415+0000","last_fullsized":"2024-03-31T15:07:58.555415+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_clean_scrub_stamp":"2024-03-31T15:07:56.519376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T16:17:18.266469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.12","version":"0'0","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-31T15:07:57.539280+0000","last_change":"2024-03-31T15:07:56.547420+0000","last_active":"2024-03-31T15:07:57.539280+0000","last_peered":"2024-03-31T15:07:57.539280+0000","last_clean":"2024-03-31T15:07:57.539280+0000","last_became_active":"2024-03-31T15:07:56.547137+0000","last_became_peered":"2024-03-31T15:07:56.547137+0000","last_unstale":"2024-03-31T15:07:57.539280+0000","last_undegraded":"2024-03-31T15:07:57.539280+0000","last_fullsized":"2024-03-31T15:07:57.539280+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:07:55.513096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:07:55.513096+0000","last_clean_scrub_stamp":"2024-03-31T15:07:55.513096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T17:48:14.267285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.10","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-31T15:07:57.551763+0000","last_change":"2024-03-31T15:07:57.546040+0000","last_active":"2024-03-31T15:07:57.551763+0000","last_peered":"2024-03-31T15:07:57.551763+0000","last_clean":"2024-03-31T15:07:57.551763+0000","last_became_active":"2024-03-31T15:07:57.545755+0000","last_became_peered":"2024-03-31T15:07:57.545755+0000","last_unstale":"2024-03-31T15:07:57.551763+0000","last_undegraded":"2024-03-31T15:07:57.551763+0000","last_fullsized":"2024-03-31T15:07:57.551763+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_clean_scrub_stamp":"2024-03-31T15:07:56.519376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:14:09.956300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.11","version":"0'0","reported_seq":14,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-31T15:07:58.555257+0000","last_change":"2024-03-31T15:07:56.545516+0000","last_active":"2024-03-31T15:07:58.555257+0000","last_peered":"2024-03-31T15:07:58.555257+0000","last_clean":"2024-03-31T15:07:58.555257+0000","last_became_active":"2024-03-31T15:07:56.545177+0000","last_became_peered":"2024-03-31T15:07:56.545177+0000","last_unstale":"2024-03-31T15:07:58.555257+0000","last_undegraded":"2024-03-31T15:07:58.555257+0000","last_fullsized":"2024-03-31T15:07:58.555257+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:07:55.513096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:07:55.513096+0000","last_clean_scrub_stamp":"2024-03-31T15:07:55.513096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:05:25.263578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-31T15:07:58.532317+0000","last_change":"2024-03-31T15:07:57.540440+0000","last_active":"2024-03-31T15:07:58.532317+0000","last_peered":"2024-03-31T15:07:58.532317+0000","last_clean":"2024-03-31T15:07:58.532317+0000","last_became_active":"2024-03-31T15:07:57.540194+0000","last_became_peered":"2024-03-31T15:07:57.540194+0000","last_unstale":"2024-03-31T15:07:58.532317+0000","last_undegraded":"2024-03-31T15:07:58.532317+0000","last_fullsized":"2024-03-31T15:07:58.532317+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_clean_scrub_stamp":"2024-03-31T15:07:56.519376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:54:31.087964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.10","version":"66'8","reported_seq":22,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-31T15:07:58.538133+0000","last_change":"2024-03-31T15:07:56.550974+0000","last_active":"2024-03-31T15:07:58.538133+0000","last_peered":"2024-03-31T15:07:58.538133+0000","last_clean":"2024-03-31T15:07:58.538133+0000","last_became_active":"2024-03-31T15:07:56.550627+0000","last_became_peered":"2024-03-31T15:07:56.550627+0000","last_unstale":"2024-03-31T15:07:58.538133+0000","last_undegraded":"2024-03-31T15:07:58.538133+0000","last_fullsized":"2024-03-31T15:07:58.538133+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:07:55.513096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:07:55.513096+0000","last_clean_scrub_stamp":"2024-03-31T15:07:55.513096+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:20:01.748754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.e","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-31T15:07:58.539831+0000","last_change":"2024-03-31T15:07:57.556445+0000","last_active":"2024-03-31T15:07:58.539831+0000","last_peered":"2024-03-31T15:07:58.539831+0000","last_clean":"2024-03-31T15:07:58.539831+0000","last_became_active":"2024-03-31T15:07:57.555786+0000","last_became_peered":"2024-03-31T15:07:57.555786+0000","last_unstale":"2024-03-31T15:07:58.539831+0000","last_undegraded":"2024-03-31T15:07:58.539831+0000","last_fullsized":"2024-03-31T15:07:58.539831+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_clean_scrub_stamp":"2024-03-31T15:07:56.519376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:09:37.513942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.f","version":"66'1","reported_seq":13,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-31T15:07:57.573739+0000","last_change":"2024-03-31T15:07:56.549846+0000","last_active":"2024-03-31T15:07:57.573739+0000","last_peered":"2024-03-31T15:07:57.573739+0000","last_clean":"2024-03-31T15:07:57.573739+0000","last_became_active":"2024-03-31T15:07:56.549578+0000","last_became_peered":"2024-03-31T15:07:56.549578+0000","last_unstale":"2024-03-31T15:07:57.573739+0000","last_undegraded":"2024-03-31T15:07:57.573739+0000","last_fullsized":"2024-03-31T15:07:57.573739+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:07:55.513096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:07:55.513096+0000","last_clean_scrub_stamp":"2024-03-31T15:07:55.513096+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:03:52.350530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.f","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-31T15:07:57.569420+0000","last_change":"2024-03-31T15:07:57.563223+0000","last_active":"2024-03-31T15:07:57.569420+0000","last_peered":"2024-03-31T15:07:57.569420+0000","last_clean":"2024-03-31T15:07:57.569420+0000","last_became_active":"2024-03-31T15:07:57.562916+0000","last_became_peered":"2024-03-31T15:07:57.562916+0000","last_unstale":"2024-03-31T15:07:57.569420+0000","last_undegraded":"2024-03-31T15:07:57.569420+0000","last_fullsized":"2024-03-31T15:07:57.569420+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_clean_scrub_stamp":"2024-03-31T15:07:56.519376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:36:07.095425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.e","version":"66'2","reported_seq":16,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-31T15:07:58.539973+0000","last_change":"2024-03-31T15:07:56.545147+0000","last_active":"2024-03-31T15:07:58.539973+0000","last_peered":"2024-03-31T15:07:58.539973+0000","last_clean":"2024-03-31T15:07:58.539973+0000","last_became_active":"2024-03-31T15:07:56.544872+0000","last_became_peered":"2024-03-31T15:07:56.544872+0000","last_unstale":"2024-03-31T15:07:58.539973+0000","last_undegraded":"2024-03-31T15:07:58.539973+0000","last_fullsized":"2024-03-31T15:07:58.539973+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:07:55.513096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:07:55.513096+0000","last_clean_scrub_stamp":"2024-03-31T15:07:55.513096+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:29:22.822927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-31T15:07:57.560005+0000","last_change":"2024-03-31T15:07:57.557119+0000","last_active":"2024-03-31T15:07:57.560005+0000","last_peered":"2024-03-31T15:07:57.560005+0000","last_clean":"2024-03-31T15:07:57.560005+0000","last_became_active":"2024-03-31T15:07:57.556477+0000","last_became_peered":"2024-03-31T15:07:57.556477+0000","last_unstale":"2024-03-31T15:07:57.560005+0000","last_undegraded":"2024-03-31T15:07:57.560005+0000","last_fullsized":"2024-03-31T15:07:57.560005+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_clean_scrub_stamp":"2024-03-31T15:07:56.519376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:28:56.174571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.d","version":"66'1","reported_seq":13,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-31T15:07:57.574433+0000","last_change":"2024-03-31T15:07:56.545706+0000","last_active":"2024-03-31T15:07:57.574433+0000","last_peered":"2024-03-31T15:07:57.574433+0000","last_clean":"2024-03-31T15:07:57.574433+0000","last_became_active":"2024-03-31T15:07:56.545279+0000","last_became_peered":"2024-03-31T15:07:56.545279+0000","last_unstale":"2024-03-31T15:07:57.574433+0000","last_undegraded":"2024-03-31T15:07:57.574433+0000","last_fullsized":"2024-03-31T15:07:57.574433+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:07:55.513096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:07:55.513096+0000","last_clean_scrub_stamp":"2024-03-31T15:07:55.513096+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:06:27.558750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.d","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-31T15:07:58.532496+0000","last_change":"2024-03-31T15:07:57.543174+0000","last_active":"2024-03-31T15:07:58.532496+0000","last_peered":"2024-03-31T15:07:58.532496+0000","last_clean":"2024-03-31T15:07:58.532496+0000","last_became_active":"2024-03-31T15:07:57.542863+0000","last_became_peered":"2024-03-31T15:07:57.542863+0000","last_unstale":"2024-03-31T15:07:58.532496+0000","last_undegraded":"2024-03-31T15:07:58.532496+0000","last_fullsized":"2024-03-31T15:07:58.532496+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_clean_scrub_stamp":"2024-03-31T15:07:56.519376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:36:18.560703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.c","version":"0'0","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-31T15:07:57.550504+0000","last_change":"2024-03-31T15:07:56.553530+0000","last_active":"2024-03-31T15:07:57.550504+0000","last_peered":"2024-03-31T15:07:57.550504+0000","last_clean":"2024-03-31T15:07:57.550504+0000","last_became_active":"2024-03-31T15:07:56.553020+0000","last_became_peered":"2024-03-31T15:07:56.553020+0000","last_unstale":"2024-03-31T15:07:57.550504+0000","last_undegraded":"2024-03-31T15:07:57.550504+0000","last_fullsized":"2024-03-31T15:07:57.550504+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:07:55.513096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:07:55.513096+0000","last_clean_scrub_stamp":"2024-03-31T15:07:55.513096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:03:57.063293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.b","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-31T15:07:57.563690+0000","last_change":"2024-03-31T15:07:57.558081+0000","last_active":"2024-03-31T15:07:57.563690+0000","last_peered":"2024-03-31T15:07:57.563690+0000","last_clean":"2024-03-31T15:07:57.563690+0000","last_became_active":"2024-03-31T15:07:57.557812+0000","last_became_peered":"2024-03-31T15:07:57.557812+0000","last_unstale":"2024-03-31T15:07:57.563690+0000","last_undegraded":"2024-03-31T15:07:57.563690+0000","last_fullsized":"2024-03-31T15:07:57.563690+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_clean_scrub_stamp":"2024-03-31T15:07:56.519376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:51:45.557002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.a","version":"0'0","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-31T15:07:57.539353+0000","last_change":"2024-03-31T15:07:56.547001+0000","last_active":"2024-03-31T15:07:57.539353+0000","last_peered":"2024-03-31T15:07:57.539353+0000","last_clean":"2024-03-31T15:07:57.539353+0000","last_became_active":"2024-03-31T15:07:56.544110+0000","last_became_peered":"2024-03-31T15:07:56.544110+0000","last_unstale":"2024-03-31T15:07:57.539353+0000","last_undegraded":"2024-03-31T15:07:57.539353+0000","last_fullsized":"2024-03-31T15:07:57.539353+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:07:55.513096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:07:55.513096+0000","last_clean_scrub_stamp":"2024-03-31T15:07:55.513096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:28:55.168291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-31T15:07:57.558674+0000","last_change":"2024-03-31T15:07:57.545669+0000","last_active":"2024-03-31T15:07:57.558674+0000","last_peered":"2024-03-31T15:07:57.558674+0000","last_clean":"2024-03-31T15:07:57.558674+0000","last_became_active":"2024-03-31T15:07:57.545289+0000","last_became_peered":"2024-03-31T15:07:57.545289+0000","last_unstale":"2024-03-31T15:07:57.558674+0000","last_undegraded":"2024-03-31T15:07:57.558674+0000","last_fullsized":"2024-03-31T15:07:57.558674+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_clean_scrub_stamp":"2024-03-31T15:07:56.519376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:13:43.088446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.9","version":"0'0","reported_seq":14,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-31T15:07:58.555558+0000","last_change":"2024-03-31T15:07:56.545941+0000","last_active":"2024-03-31T15:07:58.555558+0000","last_peered":"2024-03-31T15:07:58.555558+0000","last_clean":"2024-03-31T15:07:58.555558+0000","last_became_active":"2024-03-31T15:07:56.545659+0000","last_became_peered":"2024-03-31T15:07:56.545659+0000","last_unstale":"2024-03-31T15:07:58.555558+0000","last_undegraded":"2024-03-31T15:07:58.555558+0000","last_fullsized":"2024-03-31T15:07:58.555558+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:07:55.513096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:07:55.513096+0000","last_clean_scrub_stamp":"2024-03-31T15:07:55.513096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:09:36.507662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-31T15:07:57.557081+0000","last_change":"2024-03-31T15:07:57.542386+0000","last_active":"2024-03-31T15:07:57.557081+0000","last_peered":"2024-03-31T15:07:57.557081+0000","last_clean":"2024-03-31T15:07:57.557081+0000","last_became_active":"2024-03-31T15:07:57.541876+0000","last_became_peered":"2024-03-31T15:07:57.541876+0000","last_unstale":"2024-03-31T15:07:57.557081+0000","last_undegraded":"2024-03-31T15:07:57.557081+0000","last_fullsized":"2024-03-31T15:07:57.557081+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_clean_scrub_stamp":"2024-03-31T15:07:56.519376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T17:18:01.667911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.8","version":"0'0","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-31T15:07:57.534186+0000","last_change":"2024-03-31T15:07:56.547442+0000","last_active":"2024-03-31T15:07:57.534186+0000","last_peered":"2024-03-31T15:07:57.534186+0000","last_clean":"2024-03-31T15:07:57.534186+0000","last_became_active":"2024-03-31T15:07:56.546937+0000","last_became_peered":"2024-03-31T15:07:56.546937+0000","last_unstale":"2024-03-31T15:07:57.534186+0000","last_undegraded":"2024-03-31T15:07:57.534186+0000","last_fullsized":"2024-03-31T15:07:57.534186+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:07:55.513096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:07:55.513096+0000","last_clean_scrub_stamp":"2024-03-31T15:07:55.513096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:41:39.402187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.6","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-31T15:07:57.550003+0000","last_change":"2024-03-31T15:07:57.538061+0000","last_active":"2024-03-31T15:07:57.550003+0000","last_peered":"2024-03-31T15:07:57.550003+0000","last_clean":"2024-03-31T15:07:57.550003+0000","last_became_active":"2024-03-31T15:07:57.537575+0000","last_became_peered":"2024-03-31T15:07:57.537575+0000","last_unstale":"2024-03-31T15:07:57.550003+0000","last_undegraded":"2024-03-31T15:07:57.550003+0000","last_fullsized":"2024-03-31T15:07:57.550003+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_clean_scrub_stamp":"2024-03-31T15:07:56.519376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:03:08.318569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.7","version":"66'2","reported_seq":14,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-31T15:07:57.573699+0000","last_change":"2024-03-31T15:07:56.549633+0000","last_active":"2024-03-31T15:07:57.573699+0000","last_peered":"2024-03-31T15:07:57.573699+0000","last_clean":"2024-03-31T15:07:57.573699+0000","last_became_active":"2024-03-31T15:07:56.549111+0000","last_became_peered":"2024-03-31T15:07:56.549111+0000","last_unstale":"2024-03-31T15:07:57.573699+0000","last_undegraded":"2024-03-31T15:07:57.573699+0000","last_fullsized":"2024-03-31T15:07:57.573699+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:07:55.513096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:07:55.513096+0000","last_clean_scrub_stamp":"2024-03-31T15:07:55.513096+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:00:33.868348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.7","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-31T15:07:58.555592+0000","last_change":"2024-03-31T15:07:57.559609+0000","last_active":"2024-03-31T15:07:58.555592+0000","last_peered":"2024-03-31T15:07:58.555592+0000","last_clean":"2024-03-31T15:07:58.555592+0000","last_became_active":"2024-03-31T15:07:57.559315+0000","last_became_peered":"2024-03-31T15:07:57.559315+0000","last_unstale":"2024-03-31T15:07:58.555592+0000","last_undegraded":"2024-03-31T15:07:58.555592+0000","last_fullsized":"2024-03-31T15:07:58.555592+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_clean_scrub_stamp":"2024-03-31T15:07:56.519376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:18:05.654205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.6","version":"67'1","reported_seq":15,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-31T15:07:58.557493+0000","last_change":"2024-03-31T15:07:56.544027+0000","last_active":"2024-03-31T15:07:58.557493+0000","last_peered":"2024-03-31T15:07:58.557493+0000","last_clean":"2024-03-31T15:07:58.557493+0000","last_became_active":"2024-03-31T15:07:56.543741+0000","last_became_peered":"2024-03-31T15:07:56.543741+0000","last_unstale":"2024-03-31T15:07:58.557493+0000","last_undegraded":"2024-03-31T15:07:58.557493+0000","last_fullsized":"2024-03-31T15:07:58.557493+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:07:55.513096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:07:55.513096+0000","last_clean_scrub_stamp":"2024-03-31T15:07:55.513096+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T17:43:06.369679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-31T15:07:57.564282+0000","last_change":"2024-03-31T15:07:57.557984+0000","last_active":"2024-03-31T15:07:57.564282+0000","last_peered":"2024-03-31T15:07:57.564282+0000","last_clean":"2024-03-31T15:07:57.564282+0000","last_became_active":"2024-03-31T15:07:57.556918+0000","last_became_peered":"2024-03-31T15:07:57.556918+0000","last_unstale":"2024-03-31T15:07:57.564282+0000","last_undegraded":"2024-03-31T15:07:57.564282+0000","last_fullsized":"2024-03-31T15:07:57.564282+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_clean_scrub_stamp":"2024-03-31T15:07:56.519376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:05:31.529218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":14,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-31T15:07:58.532526+0000","last_change":"2024-03-31T15:07:56.544111+0000","last_active":"2024-03-31T15:07:58.532526+0000","last_peered":"2024-03-31T15:07:58.532526+0000","last_clean":"2024-03-31T15:07:58.532526+0000","last_became_active":"2024-03-31T15:07:56.543835+0000","last_became_peered":"2024-03-31T15:07:56.543835+0000","last_unstale":"2024-03-31T15:07:58.532526+0000","last_undegraded":"2024-03-31T15:07:58.532526+0000","last_fullsized":"2024-03-31T15:07:58.532526+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:07:55.513096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:07:55.513096+0000","last_clean_scrub_stamp":"2024-03-31T15:07:55.513096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:55:41.199547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-31T15:07:57.574009+0000","last_change":"2024-03-31T15:07:57.569085+0000","last_active":"2024-03-31T15:07:57.574009+0000","last_peered":"2024-03-31T15:07:57.574009+0000","last_clean":"2024-03-31T15:07:57.574009+0000","last_became_active":"2024-03-31T15:07:57.568239+0000","last_became_peered":"2024-03-31T15:07:57.568239+0000","last_unstale":"2024-03-31T15:07:57.574009+0000","last_undegraded":"2024-03-31T15:07:57.574009+0000","last_fullsized":"2024-03-31T15:07:57.574009+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_clean_scrub_stamp":"2024-03-31T15:07:56.519376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:06:47.401497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.4","version":"0'0","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-31T15:07:57.534781+0000","last_change":"2024-03-31T15:07:56.543969+0000","last_active":"2024-03-31T15:07:57.534781+0000","last_peered":"2024-03-31T15:07:57.534781+0000","last_clean":"2024-03-31T15:07:57.534781+0000","last_became_active":"2024-03-31T15:07:56.543704+0000","last_became_peered":"2024-03-31T15:07:56.543704+0000","last_unstale":"2024-03-31T15:07:57.534781+0000","last_undegraded":"2024-03-31T15:07:57.534781+0000","last_fullsized":"2024-03-31T15:07:57.534781+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:07:55.513096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:07:55.513096+0000","last_clean_scrub_stamp":"2024-03-31T15:07:55.513096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:56:54.890833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-31T15:07:58.532598+0000","last_change":"2024-03-31T15:07:57.548152+0000","last_active":"2024-03-31T15:07:58.532598+0000","last_peered":"2024-03-31T15:07:58.532598+0000","last_clean":"2024-03-31T15:07:58.532598+0000","last_became_active":"2024-03-31T15:07:57.547873+0000","last_became_peered":"2024-03-31T15:07:57.547873+0000","last_unstale":"2024-03-31T15:07:58.532598+0000","last_undegraded":"2024-03-31T15:07:58.532598+0000","last_fullsized":"2024-03-31T15:07:58.532598+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_clean_scrub_stamp":"2024-03-31T15:07:56.519376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:48:19.134928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":14,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-31T15:07:58.532476+0000","last_change":"2024-03-31T15:07:56.547123+0000","last_active":"2024-03-31T15:07:58.532476+0000","last_peered":"2024-03-31T15:07:58.532476+0000","last_clean":"2024-03-31T15:07:58.532476+0000","last_became_active":"2024-03-31T15:07:56.546742+0000","last_became_peered":"2024-03-31T15:07:56.546742+0000","last_unstale":"2024-03-31T15:07:58.532476+0000","last_undegraded":"2024-03-31T15:07:58.532476+0000","last_fullsized":"2024-03-31T15:07:58.532476+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:07:55.513096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:07:55.513096+0000","last_clean_scrub_stamp":"2024-03-31T15:07:55.513096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:02:09.057185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-31T15:07:57.554567+0000","last_change":"2024-03-31T15:07:57.552085+0000","last_active":"2024-03-31T15:07:57.554567+0000","last_peered":"2024-03-31T15:07:57.554567+0000","last_clean":"2024-03-31T15:07:57.554567+0000","last_became_active":"2024-03-31T15:07:57.551730+0000","last_became_peered":"2024-03-31T15:07:57.551730+0000","last_unstale":"2024-03-31T15:07:57.554567+0000","last_undegraded":"2024-03-31T15:07:57.554567+0000","last_fullsized":"2024-03-31T15:07:57.554567+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_clean_scrub_stamp":"2024-03-31T15:07:56.519376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T16:38:25.730848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-31T15:07:57.540852+0000","last_change":"2024-03-31T15:07:56.539633+0000","last_active":"2024-03-31T15:07:57.540852+0000","last_peered":"2024-03-31T15:07:57.540852+0000","last_clean":"2024-03-31T15:07:57.540852+0000","last_became_active":"2024-03-31T15:07:56.539277+0000","last_became_peered":"2024-03-31T15:07:56.539277+0000","last_unstale":"2024-03-31T15:07:57.540852+0000","last_undegraded":"2024-03-31T15:07:57.540852+0000","last_fullsized":"2024-03-31T15:07:57.540852+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:07:55.513096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:07:55.513096+0000","last_clean_scrub_stamp":"2024-03-31T15:07:55.513096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T17:25:00.920648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-31T15:07:57.560759+0000","last_change":"2024-03-31T15:07:57.555020+0000","last_active":"2024-03-31T15:07:57.560759+0000","last_peered":"2024-03-31T15:07:57.560759+0000","last_clean":"2024-03-31T15:07:57.560759+0000","last_became_active":"2024-03-31T15:07:57.553141+0000","last_became_peered":"2024-03-31T15:07:57.553141+0000","last_unstale":"2024-03-31T15:07:57.560759+0000","last_undegraded":"2024-03-31T15:07:57.560759+0000","last_fullsized":"2024-03-31T15:07:57.560759+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_clean_scrub_stamp":"2024-03-31T15:07:56.519376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:07:51.056989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":14,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-31T15:07:58.538130+0000","last_change":"2024-03-31T15:07:56.543971+0000","last_active":"2024-03-31T15:07:58.538130+0000","last_peered":"2024-03-31T15:07:58.538130+0000","last_clean":"2024-03-31T15:07:58.538130+0000","last_became_active":"2024-03-31T15:07:56.542879+0000","last_became_peered":"2024-03-31T15:07:56.542879+0000","last_unstale":"2024-03-31T15:07:58.538130+0000","last_undegraded":"2024-03-31T15:07:58.538130+0000","last_fullsized":"2024-03-31T15:07:58.538130+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:07:55.513096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:07:55.513096+0000","last_clean_scrub_stamp":"2024-03-31T15:07:55.513096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T16:50:07.834414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-31T15:07:57.549081+0000","last_change":"2024-03-31T15:07:57.544716+0000","last_active":"2024-03-31T15:07:57.549081+0000","last_peered":"2024-03-31T15:07:57.549081+0000","last_clean":"2024-03-31T15:07:57.549081+0000","last_became_active":"2024-03-31T15:07:57.544436+0000","last_became_peered":"2024-03-31T15:07:57.544436+0000","last_unstale":"2024-03-31T15:07:57.549081+0000","last_undegraded":"2024-03-31T15:07:57.549081+0000","last_fullsized":"2024-03-31T15:07:57.549081+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_clean_scrub_stamp":"2024-03-31T15:07:56.519376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T17:34:10.953702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-31T15:07:57.539163+0000","last_change":"2024-03-31T15:07:56.545932+0000","last_active":"2024-03-31T15:07:57.539163+0000","last_peered":"2024-03-31T15:07:57.539163+0000","last_clean":"2024-03-31T15:07:57.539163+0000","last_became_active":"2024-03-31T15:07:56.544251+0000","last_became_peered":"2024-03-31T15:07:56.544251+0000","last_unstale":"2024-03-31T15:07:57.539163+0000","last_undegraded":"2024-03-31T15:07:57.539163+0000","last_fullsized":"2024-03-31T15:07:57.539163+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:07:55.513096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:07:55.513096+0000","last_clean_scrub_stamp":"2024-03-31T15:07:55.513096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T16:43:50.771100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-31T15:07:57.556979+0000","last_change":"2024-03-31T15:07:57.542062+0000","last_active":"2024-03-31T15:07:57.556979+0000","last_peered":"2024-03-31T15:07:57.556979+0000","last_clean":"2024-03-31T15:07:57.556979+0000","last_became_active":"2024-03-31T15:07:57.541772+0000","last_became_peered":"2024-03-31T15:07:57.541772+0000","last_unstale":"2024-03-31T15:07:57.556979+0000","last_undegraded":"2024-03-31T15:07:57.556979+0000","last_fullsized":"2024-03-31T15:07:57.556979+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:07:56.519376+0000","last_clean_scrub_stamp":"2024-03-31T15:07:56.519376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:13:01.409654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.b","version":"66'2","reported_seq":14,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-31T15:07:57.573675+0000","last_change":"2024-03-31T15:07:56.546172+0000","last_active":"2024-03-31T15:07:57.573675+0000","last_peered":"2024-03-31T15:07:57.573675+0000","last_clean":"2024-03-31T15:07:57.573675+0000","last_became_active":"2024-03-31T15:07:56.545826+0000","last_became_peered":"2024-03-31T15:07:56.545826+0000","last_unstale":"2024-03-31T15:07:57.573675+0000","last_undegraded":"2024-03-31T15:07:57.573675+0000","last_fullsized":"2024-03-31T15:07:57.573675+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:07:55.513096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:07:55.513096+0000","last_clean_scrub_stamp":"2024-03-31T15:07:55.513096+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:40:16.591089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":252,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-31T15:07:57.547654+0000","last_change":"2024-03-31T15:03:24.329890+0000","last_active":"2024-03-31T15:07:57.547654+0000","last_peered":"2024-03-31T15:07:57.547654+0000","last_clean":"2024-03-31T15:07:57.547654+0000","last_became_active":"2024-03-31T15:03:24.328882+0000","last_became_peered":"2024-03-31T15:03:24.328882+0000","last_unstale":"2024-03-31T15:07:57.547654+0000","last_undegraded":"2024-03-31T15:07:57.547654+0000","last_fullsized":"2024-03-31T15:07:57.547654+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:22.885535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:22.885535+0000","last_clean_scrub_stamp":"2024-03-31T15:03:22.885535+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:18:29.004296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":10,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509500,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32548,"kb_used_data":1364,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339292,"statfs":{"total":96636764160,"available":96603435008,"internally_reserved":0,"allocated":1396736,"data_stored":922837,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509499,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":780,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":798720,"data_stored":331943,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":15,"seq":64424509498,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27812,"kb_used_data":780,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344028,"statfs":{"total":96636764160,"available":96608284672,"internally_reserved":0,"allocated":798720,"data_stored":331923,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509497,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31976,"kb_used_data":788,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339864,"statfs":{"total":96636764160,"available":96604020736,"internally_reserved":0,"allocated":806912,"data_stored":333465,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542203,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31948,"kb_used_data":764,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339892,"statfs":{"total":96636764160,"available":96604049408,"internally_reserved":0,"allocated":782336,"data_stored":325078,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542203,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31952,"kb_used_data":764,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339888,"statfs":{"total":96636764160,"available":96604045312,"internally_reserved":0,"allocated":782336,"data_stored":325078,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542203,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31952,"kb_used_data":764,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339888,"statfs":{"total":96636764160,"available":96604045312,"internally_reserved":0,"allocated":782336,"data_stored":325078,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542203,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32524,"kb_used_data":1344,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339316,"statfs":{"total":96636764160,"available":96603459584,"internally_reserved":0,"allocated":1376256,"data_stored":915446,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-31T15:08:02.118 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-31T15:08:02.118 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-03-31T15:08:02.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:02.306+0000 7f1853c00640 1 Processor -- start 2024-03-31T15:08:02.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:02.306+0000 7f1853c00640 1 -- start start 2024-03-31T15:08:02.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:02.310+0000 7f1853c00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f184c0cf150 0x7f184c1670a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:02.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:02.310+0000 7f1853c00640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f184c15ca70 con 0x7f184c0d0a60 2024-03-31T15:08:02.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:02.310+0000 7f1853c00640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f184c15cd50 con 0x7f184c06f920 2024-03-31T15:08:02.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:02.310+0000 7f1853c00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f184c13f670 con 0x7f184c0cf150 2024-03-31T15:08:02.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:02.310+0000 7f184be00640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f184c0d0a60 0x7f184c0cea40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:34224/0 (socket says 172.21.15.16:34224) 2024-03-31T15:08:02.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:02.310+0000 7f184be00640 1 -- 172.21.15.16:0/959393759 learned_addr learned my addr 172.21.15.16:0/959393759 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:08:02.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:02.310+0000 7f184b400640 1 -- 172.21.15.16:0/959393759 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 850403454 0 0) 0x7f184c15ca70 con 0x7f184c0d0a60 2024-03-31T15:08:02.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:02.310+0000 7f1851c00640 1 --2- 172.21.15.16:0/959393759 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f184c0cf150 0x7f184c1670a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:02.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:02.310+0000 7f184b400640 1 -- 172.21.15.16:0/959393759 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1834003530 con 0x7f184c0d0a60 2024-03-31T15:08:02.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:02.310+0000 7f184b400640 1 -- 172.21.15.16:0/959393759 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3361609455 0 0) 0x7f1834003530 con 0x7f184c0d0a60 2024-03-31T15:08:02.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:02.310+0000 7f184b400640 1 -- 172.21.15.16:0/959393759 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f184c0cf150 msgr2=0x7f184c1670a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:08:02.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:02.310+0000 7f184b400640 1 --2- 172.21.15.16:0/959393759 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f184c0cf150 0x7f184c1670a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:02.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:02.310+0000 7f184b400640 1 -- 172.21.15.16:0/959393759 >> v1:172.21.15.89:6789/0 conn(0x7f184c06f920 legacy=0x7f184c06fd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:02.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:02.310+0000 7f184b400640 1 -- 172.21.15.16:0/959393759 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f184c15d030 con 0x7f184c0d0a60 2024-03-31T15:08:02.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:02.310+0000 7f184b400640 1 -- 172.21.15.16:0/959393759 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 214807623 0 0) 0x7f1838001680 con 0x7f184c0d0a60 2024-03-31T15:08:02.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:02.310+0000 7f184b400640 1 -- 172.21.15.16:0/959393759 >> v1:172.21.15.16:6789/0 conn(0x7f184c0d0a60 legacy=0x7f184c0cea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:02.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:02.310+0000 7f184b400640 1 --2- 172.21.15.16:0/959393759 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f18340039f0 0x7f1834003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:02.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:02.310+0000 7f184b400640 1 --2- 172.21.15.16:0/959393759 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f184c0cf150 0x7f1834004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:02.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:02.310+0000 7f184b400640 1 --2- 172.21.15.16:0/959393759 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1834004d50 0x7f18340051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:02.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:02.310+0000 7f184b400640 1 -- 172.21.15.16:0/959393759 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f184c13f670 con 0x7f18340039f0 2024-03-31T15:08:02.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:02.310+0000 7f184b400640 1 -- 172.21.15.16:0/959393759 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f183400bb60 con 0x7f1834004d50 2024-03-31T15:08:02.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:02.310+0000 7f184b400640 1 -- 172.21.15.16:0/959393759 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f183400bd60 con 0x7f184c0cf150 2024-03-31T15:08:02.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:02.310+0000 7f184be00640 1 --2- 172.21.15.16:0/959393759 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f184c0cf150 0x7f1834004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:02.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:02.310+0000 7f1851200640 1 --2- 172.21.15.16:0/959393759 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f18340039f0 0x7f1834003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:02.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:02.310+0000 7f1851c00640 1 --2- 172.21.15.16:0/959393759 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1834004d50 0x7f18340051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:02.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:02.310+0000 7f184be00640 1 -- 172.21.15.16:0/959393759 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1834004d50 msgr2=0x7f18340051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:08:02.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:02.310+0000 7f184be00640 1 --2- 172.21.15.16:0/959393759 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1834004d50 0x7f18340051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:02.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:02.310+0000 7f184be00640 1 -- 172.21.15.16:0/959393759 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f18340039f0 msgr2=0x7f1834003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:08:02.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:02.310+0000 7f184be00640 1 --2- 172.21.15.16:0/959393759 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f18340039f0 0x7f1834003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:02.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:02.310+0000 7f184be00640 1 -- 172.21.15.16:0/959393759 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1834003530 con 0x7f184c0cf150 2024-03-31T15:08:02.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:02.310+0000 7f1851200640 1 --2- 172.21.15.16:0/959393759 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f18340039f0 0x7f1834003e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:08:02.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:02.314+0000 7f184be00640 1 --2- 172.21.15.16:0/959393759 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f184c0cf150 0x7f1834004810 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f183800c160 tx=0x7f183801cba0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=60c44d70ec609f85 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:08:02.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:02.314+0000 7f184b400640 1 -- 172.21.15.16:0/959393759 <== mon.2 v2:172.21.15.89:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1838007c70 con 0x7f184c0cf150 2024-03-31T15:08:02.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:02.314+0000 7f184b400640 1 -- 172.21.15.16:0/959393759 <== mon.2 v2:172.21.15.89:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f18380a9070 con 0x7f184c0cf150 2024-03-31T15:08:02.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:02.314+0000 7f184b400640 1 -- 172.21.15.16:0/959393759 <== mon.2 v2:172.21.15.89:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1838005d60 con 0x7f184c0cf150 2024-03-31T15:08:02.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:02.314+0000 7f1853c00640 1 -- 172.21.15.16:0/959393759 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f184c0cf150 msgr2=0x7f1834004810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:02.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:02.314+0000 7f1853c00640 1 --2- 172.21.15.16:0/959393759 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f184c0cf150 0x7f1834004810 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f183800c160 tx=0x7f183801cba0 comp rx=0 tx=0).stop 2024-03-31T15:08:02.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:02.314+0000 7f1851c00640 1 -- 172.21.15.16:0/959393759 reap_dead start 2024-03-31T15:08:02.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:02.314+0000 7f1853c00640 1 -- 172.21.15.16:0/959393759 shutdown_connections 2024-03-31T15:08:02.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:02.314+0000 7f1853c00640 1 -- 172.21.15.16:0/959393759 >> 172.21.15.16:0/959393759 conn(0x7f184c067930 msgr2=0x7f184c0b3ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:08:02.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:02.314+0000 7f1853c00640 1 -- 172.21.15.16:0/959393759 shutdown_connections 2024-03-31T15:08:02.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:02.314+0000 7f1853c00640 1 -- 172.21.15.16:0/959393759 wait complete. 2024-03-31T15:08:02.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:02.318+0000 7f1853c00640 1 Processor -- start 2024-03-31T15:08:02.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:02.318+0000 7f1853c00640 1 -- start start 2024-03-31T15:08:02.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:02.318+0000 7f1853c00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f184c0cf150 0x7f184c13ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:02.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:02.318+0000 7f1853c00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f184c15ad50 0x7f184c157c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:02.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:02.318+0000 7f1853c00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f184c06d660 0x7f184c06db10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:02.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:02.318+0000 7f1853c00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f184c167c40 con 0x7f184c15ad50 2024-03-31T15:08:02.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:02.318+0000 7f1853c00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f184c167ac0 con 0x7f184c0cf150 2024-03-31T15:08:02.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:02.318+0000 7f1853c00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f184c167dc0 con 0x7f184c06d660 2024-03-31T15:08:02.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:02.318+0000 7f184be00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f184c15ad50 0x7f184c157c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:02.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:02.318+0000 7f184be00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f184c15ad50 0x7f184c157c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51826/0 (socket says 172.21.15.16:51826) 2024-03-31T15:08:02.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:02.318+0000 7f184be00640 1 -- 172.21.15.16:0/2392904734 learned_addr learned my addr 172.21.15.16:0/2392904734 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:08:02.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:02.318+0000 7f1851c00640 1 --2- 172.21.15.16:0/2392904734 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f184c06d660 0x7f184c06db10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:02.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:02.318+0000 7f1851200640 1 --2- 172.21.15.16:0/2392904734 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f184c0cf150 0x7f184c13ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:02.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:02.318+0000 7f184be00640 1 -- 172.21.15.16:0/2392904734 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f184c06d660 msgr2=0x7f184c06db10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:02.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:02.318+0000 7f184be00640 1 --2- 172.21.15.16:0/2392904734 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f184c06d660 0x7f184c06db10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:02.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:02.318+0000 7f184be00640 1 -- 172.21.15.16:0/2392904734 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f184c0cf150 msgr2=0x7f184c13ecd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:02.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:02.318+0000 7f184be00640 1 --2- 172.21.15.16:0/2392904734 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f184c0cf150 0x7f184c13ecd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:02.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:02.318+0000 7f184be00640 1 -- 172.21.15.16:0/2392904734 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f184c13f210 con 0x7f184c15ad50 2024-03-31T15:08:02.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:02.318+0000 7f184be00640 1 --2- 172.21.15.16:0/2392904734 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f184c15ad50 0x7f184c157c80 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7f183801c9d0 tx=0x7f1834005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:08:02.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:02.318+0000 7f1848c00640 1 -- 172.21.15.16:0/2392904734 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18380bd070 con 0x7f184c15ad50 2024-03-31T15:08:02.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:02.318+0000 7f1848c00640 1 -- 172.21.15.16:0/2392904734 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1838005d40 con 0x7f184c15ad50 2024-03-31T15:08:02.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:02.318+0000 7f1853c00640 1 -- 172.21.15.16:0/2392904734 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f184c1584d0 con 0x7f184c15ad50 2024-03-31T15:08:02.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:02.318+0000 7f1848c00640 1 -- 172.21.15.16:0/2392904734 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18380015e0 con 0x7f184c15ad50 2024-03-31T15:08:02.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:02.318+0000 7f1853c00640 1 -- 172.21.15.16:0/2392904734 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f184c158a90 con 0x7f184c15ad50 2024-03-31T15:08:02.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:02.322+0000 7f1853c00640 1 -- 172.21.15.16:0/2392904734 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f184c069950 con 0x7f184c15ad50 2024-03-31T15:08:02.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:02.322+0000 7f1848c00640 1 -- 172.21.15.16:0/2392904734 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f18380c0070 con 0x7f184c15ad50 2024-03-31T15:08:02.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:02.322+0000 7f1848c00640 1 --2- 172.21.15.16:0/2392904734 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f182c068380 0x7f182c06a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:02.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:02.322+0000 7f1848c00640 1 -- 172.21.15.16:0/2392904734 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(67..67 src has 1..67) ==== 5709+0+0 (secure 0 0 0) 0x7f183811fb70 con 0x7f184c15ad50 2024-03-31T15:08:02.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:02.322+0000 7f1851200640 1 --2- 172.21.15.16:0/2392904734 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f182c068380 0x7f182c06a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:02.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:02.322+0000 7f1851200640 1 --2- 172.21.15.16:0/2392904734 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f182c068380 0x7f182c06a840 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f1840001370 tx=0x7f1840009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:08:02.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:02.326+0000 7f1848c00640 1 -- 172.21.15.16:0/2392904734 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f18380168b0 con 0x7f184c15ad50 2024-03-31T15:08:02.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:02.602+0000 7f1853c00640 1 -- 172.21.15.16:0/2392904734 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) -- 0x7f184c0b50e0 con 0x7f184c15ad50 2024-03-31T15:08:02.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:02.606+0000 7f1848c00640 1 -- 172.21.15.16:0/2392904734 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v16) ==== 190+0+147 (secure 0 0 0) 0x7f18380efad0 con 0x7f184c15ad50 2024-03-31T15:08:02.601 INFO:teuthology.orchestra.run.smithi016.stdout:[client.0] 2024-03-31T15:08:02.601 INFO:teuthology.orchestra.run.smithi016.stdout: key = AQAPewlmlMIDEhAAF0Ll1HcrWHmH6xDKq2Qq9A== 2024-03-31T15:08:02.601 INFO:teuthology.orchestra.run.smithi016.stdout: caps mds = "allow" 2024-03-31T15:08:02.601 INFO:teuthology.orchestra.run.smithi016.stdout: caps mon = "allow r" 2024-03-31T15:08:02.601 INFO:teuthology.orchestra.run.smithi016.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-03-31T15:08:02.601 INFO:teuthology.orchestra.run.smithi016.stderr:updated caps for client.0 2024-03-31T15:08:02.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:02.606+0000 7f1853c00640 1 -- 172.21.15.16:0/2392904734 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f182c068380 msgr2=0x7f182c06a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:02.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:02.606+0000 7f1853c00640 1 --2- 172.21.15.16:0/2392904734 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f182c068380 0x7f182c06a840 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f1840001370 tx=0x7f1840009040 comp rx=0 tx=0).stop 2024-03-31T15:08:02.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:02.606+0000 7f1853c00640 1 -- 172.21.15.16:0/2392904734 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f184c15ad50 msgr2=0x7f184c157c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:02.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:02.606+0000 7f1853c00640 1 --2- 172.21.15.16:0/2392904734 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f184c15ad50 0x7f184c157c80 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7f183801c9d0 tx=0x7f1834005840 comp rx=0 tx=0).stop 2024-03-31T15:08:02.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:02.606+0000 7f1853c00640 1 -- 172.21.15.16:0/2392904734 shutdown_connections 2024-03-31T15:08:02.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:02.606+0000 7f1853c00640 1 --2- 172.21.15.16:0/2392904734 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f182c068380 0x7f182c06a840 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:02.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:02.606+0000 7f1853c00640 1 --2- 172.21.15.16:0/2392904734 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f184c06d660 0x7f184c06db10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:02.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:02.606+0000 7f1853c00640 1 --2- 172.21.15.16:0/2392904734 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f184c15ad50 0x7f184c157c80 unknown :-1 s=CLOSED pgs=835 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:02.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:02.606+0000 7f1853c00640 1 --2- 172.21.15.16:0/2392904734 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f184c0cf150 0x7f184c13ecd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:02.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:02.606+0000 7f1853c00640 1 -- 172.21.15.16:0/2392904734 >> 172.21.15.16:0/2392904734 conn(0x7f184c067930 msgr2=0x7f184c0716b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:08:02.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:02.606+0000 7f1853c00640 1 -- 172.21.15.16:0/2392904734 shutdown_connections 2024-03-31T15:08:02.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:02.606+0000 7f1853c00640 1 -- 172.21.15.16:0/2392904734 wait complete. 2024-03-31T15:08:02.623 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-31T15:08:02.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:02.770+0000 7ff9c0600640 1 Processor -- start 2024-03-31T15:08:02.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:02.770+0000 7ff9c0600640 1 -- start start 2024-03-31T15:08:02.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:02.770+0000 7ff9c0600640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7ff9b806f920 0x7ff9b806fd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:02.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:02.770+0000 7ff9c0600640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff9b80d2010 0x7ff9b80d24a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:02.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:02.770+0000 7ff9c0600640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff9b80d0920 con 0x7ff9b8145e40 2024-03-31T15:08:02.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:02.770+0000 7ff9c0600640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7ff9b80702c0 con 0x7ff9b806f920 2024-03-31T15:08:02.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:02.770+0000 7ff9c0600640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7ff9b80d06d0 con 0x7ff9b80d2010 2024-03-31T15:08:02.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:02.770+0000 7ff9bdc00640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7ff9b8145e40 0x7ff9b80cc740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:34230/0 (socket says 172.21.15.16:34230) 2024-03-31T15:08:02.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:02.770+0000 7ff9bdc00640 1 -- 172.21.15.16:0/1199789389 learned_addr learned my addr 172.21.15.16:0/1199789389 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:08:02.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:02.770+0000 7ff9bd200640 1 --2- 172.21.15.16:0/1199789389 >> v2:172.21.15.89:3300/0 conn(0x7ff9b806f920 0x7ff9b806fd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:02.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:02.770+0000 7ff9be600640 1 --2- 172.21.15.16:0/1199789389 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff9b80d2010 0x7ff9b80d24a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:02.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:02.770+0000 7ff9b7e00640 1 -- 172.21.15.16:0/1199789389 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1879757261 0 0) 0x7ff9b80d0920 con 0x7ff9b8145e40 2024-03-31T15:08:02.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:02.770+0000 7ff9b7e00640 1 -- 172.21.15.16:0/1199789389 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff9a0003530 con 0x7ff9b8145e40 2024-03-31T15:08:02.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:02.770+0000 7ff9bd200640 1 -- 172.21.15.16:0/1199789389 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff9b80d2010 msgr2=0x7ff9b80d24a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:08:02.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:02.770+0000 7ff9bd200640 1 --2- 172.21.15.16:0/1199789389 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff9b80d2010 0x7ff9b80d24a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:02.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:02.770+0000 7ff9bd200640 1 -- 172.21.15.16:0/1199789389 >> v1:172.21.15.16:6789/0 conn(0x7ff9b8145e40 legacy=0x7ff9b80cc740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:02.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:02.770+0000 7ff9b7e00640 1 -- 172.21.15.16:0/1199789389 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2643319610 0 0) 0x7ff9a0003530 con 0x7ff9b8145e40 2024-03-31T15:08:02.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:02.770+0000 7ff9bd200640 1 -- 172.21.15.16:0/1199789389 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9b80d0c00 con 0x7ff9b806f920 2024-03-31T15:08:02.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:02.770+0000 7ff9be600640 1 --2- 172.21.15.16:0/1199789389 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff9b80d2010 0x7ff9b80d24a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:08:02.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:02.774+0000 7ff9bd200640 1 --2- 172.21.15.16:0/1199789389 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff9b806f920 0x7ff9b806fd80 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7ff9a40097c0 tx=0x7ff9a401c3c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=53836b2063cff528 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:08:02.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:02.774+0000 7ff9b7e00640 1 -- 172.21.15.16:0/1199789389 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9a40b0070 con 0x7ff9b806f920 2024-03-31T15:08:02.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:02.774+0000 7ff9b7e00640 1 -- 172.21.15.16:0/1199789389 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff9a40ab400 con 0x7ff9b806f920 2024-03-31T15:08:02.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:02.774+0000 7ff9b7e00640 1 -- 172.21.15.16:0/1199789389 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9ac001680 con 0x7ff9b806f920 2024-03-31T15:08:02.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:02.774+0000 7ff9c0600640 1 -- 172.21.15.16:0/1199789389 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff9b806f920 msgr2=0x7ff9b806fd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:02.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:02.774+0000 7ff9c0600640 1 --2- 172.21.15.16:0/1199789389 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff9b806f920 0x7ff9b806fd80 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7ff9a40097c0 tx=0x7ff9a401c3c0 comp rx=0 tx=0).stop 2024-03-31T15:08:02.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:02.774+0000 7ff9c0600640 1 -- 172.21.15.16:0/1199789389 shutdown_connections 2024-03-31T15:08:02.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:02.774+0000 7ff9c0600640 1 --2- 172.21.15.16:0/1199789389 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff9b80d2010 0x7ff9b80d24a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:02.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:02.774+0000 7ff9c0600640 1 --2- 172.21.15.16:0/1199789389 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff9b806f920 0x7ff9b806fd80 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:02.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:02.774+0000 7ff9c0600640 1 -- 172.21.15.16:0/1199789389 >> 172.21.15.16:0/1199789389 conn(0x7ff9b8067330 msgr2=0x7ff9b8067740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:08:02.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:02.774+0000 7ff9c0600640 1 -- 172.21.15.16:0/1199789389 shutdown_connections 2024-03-31T15:08:02.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:02.774+0000 7ff9c0600640 1 -- 172.21.15.16:0/1199789389 wait complete. 2024-03-31T15:08:02.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:02.778+0000 7ff9c0600640 1 Processor -- start 2024-03-31T15:08:02.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:02.778+0000 7ff9c0600640 1 -- start start 2024-03-31T15:08:02.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:02.778+0000 7ff9c0600640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff9b806f920 0x7ff9b8157df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:02.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:02.778+0000 7ff9c0600640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff9b80d2010 0x7ff9b8156500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:02.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:02.778+0000 7ff9c0600640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff9b8145e40 0x7ff9b815e180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:02.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:02.778+0000 7ff9c0600640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7ff9b80d0ee0 con 0x7ff9b8145e40 2024-03-31T15:08:02.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:02.778+0000 7ff9c0600640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7ff9b80ce9e0 con 0x7ff9b80d2010 2024-03-31T15:08:02.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:02.778+0000 7ff9c0600640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7ff9b80d2df0 con 0x7ff9b806f920 2024-03-31T15:08:02.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:02.778+0000 7ff9be600640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff9b8145e40 0x7ff9b815e180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:02.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:02.778+0000 7ff9be600640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff9b8145e40 0x7ff9b815e180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51842/0 (socket says 172.21.15.16:51842) 2024-03-31T15:08:02.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:02.778+0000 7ff9be600640 1 -- 172.21.15.16:0/2856832739 learned_addr learned my addr 172.21.15.16:0/2856832739 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:08:02.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:02.778+0000 7ff9bd200640 1 --2- 172.21.15.16:0/2856832739 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff9b80d2010 0x7ff9b8156500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:02.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:02.778+0000 7ff9bdc00640 1 --2- 172.21.15.16:0/2856832739 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff9b806f920 0x7ff9b8157df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:02.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:02.778+0000 7ff9be600640 1 -- 172.21.15.16:0/2856832739 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff9b806f920 msgr2=0x7ff9b8157df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:02.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:02.778+0000 7ff9be600640 1 --2- 172.21.15.16:0/2856832739 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff9b806f920 0x7ff9b8157df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:02.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:02.778+0000 7ff9be600640 1 -- 172.21.15.16:0/2856832739 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff9b80d2010 msgr2=0x7ff9b8156500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:02.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:02.778+0000 7ff9be600640 1 --2- 172.21.15.16:0/2856832739 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff9b80d2010 0x7ff9b8156500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:02.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:02.778+0000 7ff9be600640 1 -- 172.21.15.16:0/2856832739 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9b8156d50 con 0x7ff9b8145e40 2024-03-31T15:08:02.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:02.778+0000 7ff9bd200640 1 --2- 172.21.15.16:0/2856832739 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff9b80d2010 0x7ff9b8156500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:08:02.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:02.778+0000 7ff9be600640 1 --2- 172.21.15.16:0/2856832739 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff9b8145e40 0x7ff9b815e180 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7ff9b000efc0 tx=0x7ff9b0009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:08:02.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:02.778+0000 7ff9b5600640 1 -- 172.21.15.16:0/2856832739 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9b0018030 con 0x7ff9b8145e40 2024-03-31T15:08:02.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:02.778+0000 7ff9b5600640 1 -- 172.21.15.16:0/2856832739 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff9b000cde0 con 0x7ff9b8145e40 2024-03-31T15:08:02.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:02.778+0000 7ff9c0600640 1 -- 172.21.15.16:0/2856832739 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff9b8157030 con 0x7ff9b8145e40 2024-03-31T15:08:02.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:02.778+0000 7ff9b5600640 1 -- 172.21.15.16:0/2856832739 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9b0003bc0 con 0x7ff9b8145e40 2024-03-31T15:08:02.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:02.778+0000 7ff9c0600640 1 -- 172.21.15.16:0/2856832739 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff9b81575f0 con 0x7ff9b8145e40 2024-03-31T15:08:02.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:02.782+0000 7ff9c0600640 1 -- 172.21.15.16:0/2856832739 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff9b8069950 con 0x7ff9b8145e40 2024-03-31T15:08:02.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:02.782+0000 7ff9b5600640 1 -- 172.21.15.16:0/2856832739 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7ff9b001b070 con 0x7ff9b8145e40 2024-03-31T15:08:02.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:02.782+0000 7ff9b5600640 1 --2- 172.21.15.16:0/2856832739 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7ff998068380 0x7ff99806a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:02.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:02.782+0000 7ff9b5600640 1 -- 172.21.15.16:0/2856832739 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(67..67 src has 1..67) ==== 5709+0+0 (secure 0 0 0) 0x7ff9b0087570 con 0x7ff9b8145e40 2024-03-31T15:08:02.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:02.782+0000 7ff9bdc00640 1 --2- 172.21.15.16:0/2856832739 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7ff998068380 0x7ff99806a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:02.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:02.782+0000 7ff9bdc00640 1 --2- 172.21.15.16:0/2856832739 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7ff998068380 0x7ff99806a840 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7ff9ac007a10 tx=0x7ff9ac009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:08:02.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:02.786+0000 7ff9b5600640 1 -- 172.21.15.16:0/2856832739 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7ff9b0010040 con 0x7ff9b8145e40 2024-03-31T15:08:02.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:02.986+0000 7ff9c0600640 1 -- 172.21.15.16:0/2856832739 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff9b80cf3c0 con 0x7ff9b8145e40 2024-03-31T15:08:02.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:02.986+0000 7ff9b5600640 1 -- 172.21.15.16:0/2856832739 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 45 v45) ==== 94+0+5183 (secure 0 0 0) 0x7ff9b0085e20 con 0x7ff9b8145e40 2024-03-31T15:08:02.982 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:08:02.982 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":45,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6836/4188793506","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":4188793506},{"type":"v1","addr":"172.21.15.89:6836","nonce":4188793506}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":40},{"gid":5942,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/4105657676","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6835","nonce":4105657676},{"type":"v1","addr":"172.21.15.89:6837","nonce":4105657676}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":40},{"gid":6592,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6833/3986752851","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":3986752851},{"type":"v1","addr":"172.21.15.16:6833","nonce":3986752851}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":39}],"filesystems":[{"mdsmap":{"epoch":45,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-31T15:07:56.938140+0000","modified":"2024-03-31T15:07:59.939362+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6714},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6714":{"gid":6714,"name":"a","rank":0,"incarnation":41,"state":"up:active","state_seq":3,"addr":"172.21.15.16:6835/2555930820","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":2555930820},{"type":"v1","addr":"172.21.15.16:6835","nonce":2555930820}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6714,"qdb_cluster":[6714]},"id":5}]} 2024-03-31T15:08:02.982 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 45 2024-03-31T15:08:02.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:02.990+0000 7ff9c0600640 1 -- 172.21.15.16:0/2856832739 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7ff998068380 msgr2=0x7ff99806a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:02.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:02.990+0000 7ff9c0600640 1 --2- 172.21.15.16:0/2856832739 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7ff998068380 0x7ff99806a840 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7ff9ac007a10 tx=0x7ff9ac009040 comp rx=0 tx=0).stop 2024-03-31T15:08:02.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:02.990+0000 7ff9c0600640 1 -- 172.21.15.16:0/2856832739 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff9b8145e40 msgr2=0x7ff9b815e180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:02.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:02.990+0000 7ff9c0600640 1 --2- 172.21.15.16:0/2856832739 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff9b8145e40 0x7ff9b815e180 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7ff9b000efc0 tx=0x7ff9b0009f90 comp rx=0 tx=0).stop 2024-03-31T15:08:02.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:02.990+0000 7ff9c0600640 1 -- 172.21.15.16:0/2856832739 shutdown_connections 2024-03-31T15:08:02.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:02.990+0000 7ff9c0600640 1 --2- 172.21.15.16:0/2856832739 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7ff998068380 0x7ff99806a840 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:02.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:02.990+0000 7ff9c0600640 1 --2- 172.21.15.16:0/2856832739 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff9b8145e40 0x7ff9b815e180 unknown :-1 s=CLOSED pgs=837 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:02.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:02.990+0000 7ff9c0600640 1 --2- 172.21.15.16:0/2856832739 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff9b80d2010 0x7ff9b8156500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:02.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:02.990+0000 7ff9c0600640 1 --2- 172.21.15.16:0/2856832739 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff9b806f920 0x7ff9b8157df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:02.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:02.990+0000 7ff9c0600640 1 -- 172.21.15.16:0/2856832739 >> 172.21.15.16:0/2856832739 conn(0x7ff9b8067330 msgr2=0x7ff9b815bd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:08:02.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:02.990+0000 7ff9c0600640 1 -- 172.21.15.16:0/2856832739 shutdown_connections 2024-03-31T15:08:02.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:02.990+0000 7ff9c0600640 1 -- 172.21.15.16:0/2856832739 wait complete. 2024-03-31T15:08:03.406 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 45, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-31T15:07:56.938140+0000', 'modified': '2024-03-31T15:07:59.939362+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6714}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6714': {'gid': 6714, 'name': 'a', 'rank': 0, 'incarnation': 41, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.16:6835/2555930820', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6834', 'nonce': 2555930820}, {'type': 'v1', 'addr': '172.21.15.16:6835', 'nonce': 2555930820}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [11], 'metadata_pool': 10, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 6714, 'qdb_cluster': [6714]} 2024-03-31T15:08:03.407 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-31T15:08:03.407 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-03-31T15:08:03.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.550+0000 7fd8e7400640 1 Processor -- start 2024-03-31T15:08:03.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.554+0000 7fd8e7400640 1 -- start start 2024-03-31T15:08:03.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.554+0000 7fd8e7400640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fd8e0147280 0x7fd8e006c730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:03.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.554+0000 7fd8e7400640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fd8e0147dd0 0x7fd8e006f920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:03.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.554+0000 7fd8e7400640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd8e0147660 con 0x7fd8e006fe60 2024-03-31T15:08:03.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.554+0000 7fd8e7400640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7fd8e01620f0 con 0x7fd8e0147280 2024-03-31T15:08:03.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.554+0000 7fd8e7400640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7fd8e0162340 con 0x7fd8e0147dd0 2024-03-31T15:08:03.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.554+0000 7fd8e4a00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fd8e0147280 0x7fd8e006c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:03.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.554+0000 7fd8e4a00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fd8e0147280 0x7fd8e006c730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51856/0 (socket says 172.21.15.16:51856) 2024-03-31T15:08:03.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.554+0000 7fd8e4a00640 1 -- 172.21.15.16:0/1464408607 learned_addr learned my addr 172.21.15.16:0/1464408607 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:08:03.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.554+0000 7fd8dfe00640 1 --2- 172.21.15.16:0/1464408607 >> v2:172.21.15.89:3300/0 conn(0x7fd8e0147dd0 0x7fd8e006f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:03.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.554+0000 7fd8e4a00640 1 -- 172.21.15.16:0/1464408607 >> v2:172.21.15.89:3300/0 conn(0x7fd8e0147dd0 msgr2=0x7fd8e006f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:08:03.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.554+0000 7fd8e4a00640 1 --2- 172.21.15.16:0/1464408607 >> v2:172.21.15.89:3300/0 conn(0x7fd8e0147dd0 0x7fd8e006f920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:03.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.554+0000 7fd8e4a00640 1 -- 172.21.15.16:0/1464408607 >> v1:172.21.15.89:6789/0 conn(0x7fd8e006fe60 legacy=0x7fd8e00cb440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:03.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.554+0000 7fd8e4a00640 1 -- 172.21.15.16:0/1464408607 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8e0162620 con 0x7fd8e0147280 2024-03-31T15:08:03.554 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.558+0000 7fd8e4a00640 1 --2- 172.21.15.16:0/1464408607 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd8e0147280 0x7fd8e006c730 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7fd8e00cfc70 tx=0x7fd8d401c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc78324f7c810e96 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:08:03.554 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.558+0000 7fd8df400640 1 -- 172.21.15.16:0/1464408607 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8d40b0070 con 0x7fd8e0147280 2024-03-31T15:08:03.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.558+0000 7fd8df400640 1 -- 172.21.15.16:0/1464408607 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd8d40ab600 con 0x7fd8e0147280 2024-03-31T15:08:03.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.558+0000 7fd8df400640 1 -- 172.21.15.16:0/1464408607 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8d40ab9d0 con 0x7fd8e0147280 2024-03-31T15:08:03.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.562+0000 7fd8e7400640 1 -- 172.21.15.16:0/1464408607 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd8e0147280 msgr2=0x7fd8e006c730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:03.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.562+0000 7fd8e7400640 1 --2- 172.21.15.16:0/1464408607 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd8e0147280 0x7fd8e006c730 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7fd8e00cfc70 tx=0x7fd8d401c520 comp rx=0 tx=0).stop 2024-03-31T15:08:03.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.562+0000 7fd8e7400640 1 -- 172.21.15.16:0/1464408607 shutdown_connections 2024-03-31T15:08:03.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.562+0000 7fd8e7400640 1 --2- 172.21.15.16:0/1464408607 >> v2:172.21.15.89:3300/0 conn(0x7fd8e0147dd0 0x7fd8e006f920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:03.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.562+0000 7fd8e7400640 1 --2- 172.21.15.16:0/1464408607 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd8e0147280 0x7fd8e006c730 unknown :-1 s=CLOSED pgs=838 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:03.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.562+0000 7fd8e7400640 1 -- 172.21.15.16:0/1464408607 >> 172.21.15.16:0/1464408607 conn(0x7fd8e0067330 msgr2=0x7fd8e0067740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:08:03.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.562+0000 7fd8e7400640 1 -- 172.21.15.16:0/1464408607 shutdown_connections 2024-03-31T15:08:03.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.562+0000 7fd8e7400640 1 -- 172.21.15.16:0/1464408607 wait complete. 2024-03-31T15:08:03.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.562+0000 7fd8e7400640 1 Processor -- start 2024-03-31T15:08:03.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.562+0000 7fd8e7400640 1 -- start start 2024-03-31T15:08:03.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.566+0000 7fd8e7400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd8e006fe60 0x7fd8e013fd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:03.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.566+0000 7fd8e7400640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fd8e0147280 0x7fd8e00b53b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:03.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.566+0000 7fd8e7400640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd8e0147dd0 0x7fd8e0142730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:03.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.566+0000 7fd8e7400640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fd8e016b6a0 con 0x7fd8e006fe60 2024-03-31T15:08:03.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.566+0000 7fd8e7400640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fd8e0070270 con 0x7fd8e0147dd0 2024-03-31T15:08:03.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.566+0000 7fd8e7400640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fd8e016b820 con 0x7fd8e0147280 2024-03-31T15:08:03.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.566+0000 7fd8e4a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd8e006fe60 0x7fd8e013fd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:03.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.566+0000 7fd8e4a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd8e006fe60 0x7fd8e013fd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51864/0 (socket says 172.21.15.16:51864) 2024-03-31T15:08:03.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.566+0000 7fd8e4a00640 1 -- 172.21.15.16:0/4259179391 learned_addr learned my addr 172.21.15.16:0/4259179391 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:08:03.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.566+0000 7fd8dfe00640 1 --2- 172.21.15.16:0/4259179391 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fd8e0147280 0x7fd8e00b53b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:03.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.566+0000 7fd8e5400640 1 --2- 172.21.15.16:0/4259179391 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd8e0147dd0 0x7fd8e0142730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:03.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.566+0000 7fd8e4a00640 1 -- 172.21.15.16:0/4259179391 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fd8e0147280 msgr2=0x7fd8e00b53b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:03.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.566+0000 7fd8e4a00640 1 --2- 172.21.15.16:0/4259179391 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fd8e0147280 0x7fd8e00b53b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:03.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.566+0000 7fd8e4a00640 1 -- 172.21.15.16:0/4259179391 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd8e0147dd0 msgr2=0x7fd8e0142730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:03.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.566+0000 7fd8e4a00640 1 --2- 172.21.15.16:0/4259179391 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd8e0147dd0 0x7fd8e0142730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:03.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.566+0000 7fd8e4a00640 1 -- 172.21.15.16:0/4259179391 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8e0160ce0 con 0x7fd8e006fe60 2024-03-31T15:08:03.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.566+0000 7fd8e4a00640 1 --2- 172.21.15.16:0/4259179391 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd8e006fe60 0x7fd8e013fd70 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7fd8d401abd0 tx=0x7fd8d401ad60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:08:03.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.566+0000 7fd8dcc00640 1 -- 172.21.15.16:0/4259179391 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8d40b0040 con 0x7fd8e006fe60 2024-03-31T15:08:03.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.566+0000 7fd8dcc00640 1 -- 172.21.15.16:0/4259179391 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd8d40ab400 con 0x7fd8e006fe60 2024-03-31T15:08:03.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.566+0000 7fd8dcc00640 1 -- 172.21.15.16:0/4259179391 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8d40b6640 con 0x7fd8e006fe60 2024-03-31T15:08:03.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.566+0000 7fd8e7400640 1 -- 172.21.15.16:0/4259179391 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd8e0160fc0 con 0x7fd8e006fe60 2024-03-31T15:08:03.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.566+0000 7fd8e7400640 1 -- 172.21.15.16:0/4259179391 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd8e015f570 con 0x7fd8e006fe60 2024-03-31T15:08:03.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.570+0000 7fd8ea842000 1 Processor -- start 2024-03-31T15:08:03.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.570+0000 7fd8ea842000 1 -- start start 2024-03-31T15:08:03.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.570+0000 7fd8dcc00640 1 -- 172.21.15.16:0/4259179391 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fd8d40be030 con 0x7fd8e006fe60 2024-03-31T15:08:03.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.570+0000 7fd8ea842000 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55607cc4e4a0 0x55607cc4e880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:03.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.570+0000 7fd8ea842000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55607cc4edc0 0x55607cc01b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:03.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.570+0000 7fd8ea842000 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55607cc020b0 0x55607cc02490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:03.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.570+0000 7fd8ea842000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55607ca54950 con 0x55607cc4edc0 2024-03-31T15:08:03.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.570+0000 7fd8ea842000 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55607c9f1990 con 0x55607cc020b0 2024-03-31T15:08:03.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.570+0000 7fd8ea842000 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x55607ca50900 con 0x55607cc4e4a0 2024-03-31T15:08:03.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.574+0000 7fd8e5400640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55607cc4e4a0 0x55607cc4e880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:03.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.574+0000 7fd8e4a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55607cc4edc0 0x55607cc01b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:03.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.574+0000 7fd8e5400640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55607cc4e4a0 0x55607cc4e880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3301/0 says I am v2:172.21.15.16:59792/0 (socket says 172.21.15.16:59792) 2024-03-31T15:08:03.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.574+0000 7fd8e5400640 1 -- 172.21.15.16:0/85892243 learned_addr learned my addr 172.21.15.16:0/85892243 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:08:03.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.574+0000 7fd8dfe00640 1 --2- 172.21.15.16:0/85892243 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55607cc020b0 0x55607cc02490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:03.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.574+0000 7fd8dcc00640 1 --2- 172.21.15.16:0/4259179391 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fd8c0068380 0x7fd8c006a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:03.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.574+0000 7fd8e4a00640 1 -- 172.21.15.16:0/85892243 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55607cc4e4a0 msgr2=0x55607cc4e880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:08:03.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.574+0000 7fd8e4a00640 1 --2- 172.21.15.16:0/85892243 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55607cc4e4a0 0x55607cc4e880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:03.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.574+0000 7fd8e4a00640 1 -- 172.21.15.16:0/85892243 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55607cc020b0 msgr2=0x55607cc02490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:08:03.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.574+0000 7fd8e4a00640 1 --2- 172.21.15.16:0/85892243 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55607cc020b0 0x55607cc02490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:03.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.574+0000 7fd8dcc00640 1 -- 172.21.15.16:0/4259179391 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(67..67 src has 1..67) ==== 5709+0+0 (secure 0 0 0) 0x7fd8d41200f0 con 0x7fd8e006fe60 2024-03-31T15:08:03.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.574+0000 7fd8e4a00640 1 -- 172.21.15.16:0/85892243 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55607ca99100 con 0x55607cc4edc0 2024-03-31T15:08:03.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.574+0000 7fd8e5400640 1 --2- 172.21.15.16:0/85892243 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55607cc4e4a0 0x55607cc4e880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:08:03.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.574+0000 7fd8dfe00640 1 --2- 172.21.15.16:0/85892243 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55607cc020b0 0x55607cc02490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:08:03.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.574+0000 7fd8e5400640 1 --2- 172.21.15.16:0/4259179391 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fd8c0068380 0x7fd8c006a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:03.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.574+0000 7fd8e4a00640 1 --2- 172.21.15.16:0/85892243 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55607cc4edc0 0x55607cc01b70 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7fd8d411e620 tx=0x7fd8d40192c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=51f19e12d3bcc525 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:08:03.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.574+0000 7fd8d1600640 1 -- 172.21.15.16:0/85892243 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8d40ec390 con 0x55607cc4edc0 2024-03-31T15:08:03.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.574+0000 7fd8d1600640 1 -- 172.21.15.16:0/85892243 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd8d40dddf0 con 0x55607cc4edc0 2024-03-31T15:08:03.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.574+0000 7fd8d1600640 1 -- 172.21.15.16:0/85892243 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8d40efa50 con 0x55607cc4edc0 2024-03-31T15:08:03.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.574+0000 7fd8ea842000 1 -- 172.21.15.16:0/85892243 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55607cc4edc0 msgr2=0x55607cc01b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:03.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.574+0000 7fd8e5400640 1 --2- 172.21.15.16:0/4259179391 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fd8c0068380 0x7fd8c006a840 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7fd8d8007be0 tx=0x7fd8d8009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:08:03.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.574+0000 7fd8ea842000 1 --2- 172.21.15.16:0/85892243 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55607cc4edc0 0x55607cc01b70 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7fd8d411e620 tx=0x7fd8d40192c0 comp rx=0 tx=0).stop 2024-03-31T15:08:03.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.574+0000 7fd8ea842000 1 -- 172.21.15.16:0/85892243 shutdown_connections 2024-03-31T15:08:03.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.574+0000 7fd8ea842000 1 --2- 172.21.15.16:0/85892243 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55607cc020b0 0x55607cc02490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:03.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.574+0000 7fd8ea842000 1 --2- 172.21.15.16:0/85892243 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55607cc4edc0 0x55607cc01b70 unknown :-1 s=CLOSED pgs=840 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:03.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.574+0000 7fd8ea842000 1 --2- 172.21.15.16:0/85892243 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55607cc4e4a0 0x55607cc4e880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:03.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.574+0000 7fd8ea842000 1 -- 172.21.15.16:0/85892243 >> 172.21.15.16:0/85892243 conn(0x55607caecb80 msgr2=0x55607cc31460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:08:03.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.574+0000 7fd8ea842000 1 -- 172.21.15.16:0/85892243 shutdown_connections 2024-03-31T15:08:03.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.574+0000 7fd8ea842000 1 -- 172.21.15.16:0/85892243 wait complete. 2024-03-31T15:08:03.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.574+0000 7fd8ea842000 1 -- start start 2024-03-31T15:08:03.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.574+0000 7fd8ea842000 1 Processor -- start 2024-03-31T15:08:03.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.574+0000 7fd8ea842000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55607cc020b0 0x55607cc56420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:03.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.574+0000 7fd8ea842000 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55607cc4e4a0 0x55607cc56960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:03.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.574+0000 7fd8ea842000 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55607cc4edc0 0x55607cc0ea10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:03.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.574+0000 7fd8ea842000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55607ca53c60 con 0x55607cc020b0 2024-03-31T15:08:03.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.574+0000 7fd8ea842000 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55607ca123e0 con 0x55607cc4edc0 2024-03-31T15:08:03.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.574+0000 7fd8ea842000 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x55607ca54950 con 0x55607cc4e4a0 2024-03-31T15:08:03.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.578+0000 7fd8ea842000 10 client.0 _ll_register_callbacks cb 0x55607cab50f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T15:08:03.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.578+0000 7fd8e4a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55607cc020b0 0x55607cc56420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:03.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.578+0000 7fd8e4a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55607cc020b0 0x55607cc56420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51882/0 (socket says 172.21.15.16:51882) 2024-03-31T15:08:03.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.578+0000 7fd8e4a00640 1 -- 172.21.15.16:0/3081335050 learned_addr learned my addr 172.21.15.16:0/3081335050 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:08:03.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.578+0000 7fd8e5400640 1 --2- 172.21.15.16:0/3081335050 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55607cc4edc0 0x55607cc0ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:03.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.578+0000 7fd8dfe00640 1 --2- 172.21.15.16:0/3081335050 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55607cc4e4a0 0x55607cc56960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:03.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.578+0000 7fd8e4a00640 1 -- 172.21.15.16:0/3081335050 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55607cc4e4a0 msgr2=0x55607cc56960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:08:03.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.578+0000 7fd8e4a00640 1 --2- 172.21.15.16:0/3081335050 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55607cc4e4a0 0x55607cc56960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:03.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.578+0000 7fd8e4a00640 1 -- 172.21.15.16:0/3081335050 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55607cc4edc0 msgr2=0x55607cc0ea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:08:03.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.578+0000 7fd8e4a00640 1 --2- 172.21.15.16:0/3081335050 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55607cc4edc0 0x55607cc0ea10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:03.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.578+0000 7fd8e4a00640 1 -- 172.21.15.16:0/3081335050 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55607ca1e8c0 con 0x55607cc020b0 2024-03-31T15:08:03.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.578+0000 7fd8e4a00640 1 --2- 172.21.15.16:0/3081335050 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55607cc020b0 0x55607cc56420 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7fd8d401c350 tx=0x7fd8d4122620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=57d0a05ca45f528f server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:08:03.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.578+0000 7fd8c6000640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-31T15:08:03.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.578+0000 7fd8c6000640 1 -- 172.21.15.16:0/3081335050 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8d40efc30 con 0x55607cc020b0 2024-03-31T15:08:03.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.578+0000 7fd8ea842000 1 -- 172.21.15.16:0/3081335050 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd8d40dddf0 con 0x55607cc020b0 2024-03-31T15:08:03.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.578+0000 7fd8c6000640 1 -- 172.21.15.16:0/3081335050 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(67..67 src has 1..67) ==== 5709+0+0 (secure 0 0 0) 0x7fd8d40ee3f0 con 0x55607cc020b0 2024-03-31T15:08:03.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.578+0000 7fd8c6000640 1 -- 172.21.15.16:0/3081335050 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=45) ==== 24+0+0 (secure 0 0 0) 0x7fd8d40efec0 con 0x55607cc020b0 2024-03-31T15:08:03.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.578+0000 7fd8ea842000 10 client.6852 fetch_fsmap learned FSMap version 45 2024-03-31T15:08:03.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.578+0000 7fd8ea842000 1 -- 172.21.15.16:0/3081335050 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=45,monmap=2+}) -- 0x55607c9dce80 con 0x55607cc020b0 2024-03-31T15:08:03.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.578+0000 7fd8c6000640 1 -- 172.21.15.16:0/3081335050 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd8d40dddf0 con 0x55607cc020b0 2024-03-31T15:08:03.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.578+0000 7fd8c6000640 1 -- 172.21.15.16:0/3081335050 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 45) ==== 3494+0+0 (secure 0 0 0) 0x7fd8d40b3030 con 0x55607cc020b0 2024-03-31T15:08:03.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.578+0000 7fd8ea842000 10 client.6852 fetch_fsmap finished waiting for FSMap version 45 2024-03-31T15:08:03.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.578+0000 7fd8ea842000 10 client.6852 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-31T15:08:03.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.578+0000 7fd8ea842000 20 client.6852 populate_metadata read hostname 'smithi016' 2024-03-31T15:08:03.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.578+0000 7fd8ea842000 1 --2- 172.21.15.16:0/3081335050 >> [v2:172.21.15.16:6834/2555930820,v1:172.21.15.16:6835/2555930820] conn(0x55607cc0f420 0x55607cc0f800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:03.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.578+0000 7fd8ea842000 4 client.6852 mds_command: new command op to 6714 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-31T15:08:03.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.578+0000 7fd8ea842000 1 -- 172.21.15.16:0/3081335050 --> [v2:172.21.15.16:6834/2555930820,v1:172.21.15.16:6835/2555930820] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55607c9efbc0 con 0x55607cc0f420 2024-03-31T15:08:03.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.578+0000 7fd8dfe00640 1 --2- 172.21.15.16:0/3081335050 >> [v2:172.21.15.16:6834/2555930820,v1:172.21.15.16:6835/2555930820] conn(0x55607cc0f420 0x55607cc0f800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:03.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.578+0000 7fd8dfe00640 1 --2- 172.21.15.16:0/3081335050 >> [v2:172.21.15.16:6834/2555930820,v1:172.21.15.16:6835/2555930820] conn(0x55607cc0f420 0x55607cc0f800 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=16c1fd175896884a server_cookie=143cc99174b80ef8 in_seq=0 out_seq=0 2024-03-31T15:08:03.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.578+0000 7fd8c6000640 10 client.6852 ms_handle_connect on v2:172.21.15.16:6834/2555930820 2024-03-31T15:08:03.577 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.578+0000 7fd8c6000640 1 -- 172.21.15.16:0/3081335050 <== mds.0 v2:172.21.15.16:6834/2555930820 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55607c9efbc0 con 0x55607cc0f420 2024-03-31T15:08:03.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.578+0000 7fd8c6000640 10 client.6852 handle_command_reply: tid=0 2024-03-31T15:08:03.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.578+0000 7fd8ea842000 1 client.6852 shutdown 2024-03-31T15:08:03.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.578+0000 7fd8ea842000 1 -- 172.21.15.16:0/3081335050 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55607cc020b0 msgr2=0x55607cc56420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:03.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.578+0000 7fd8ea842000 1 --2- 172.21.15.16:0/3081335050 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55607cc020b0 0x55607cc56420 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7fd8d401c350 tx=0x7fd8d4122620 comp rx=0 tx=0).stop 2024-03-31T15:08:03.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.578+0000 7fd8ea842000 1 -- 172.21.15.16:0/3081335050 shutdown_connections 2024-03-31T15:08:03.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.578+0000 7fd8ea842000 1 --2- 172.21.15.16:0/3081335050 >> [v2:172.21.15.16:6834/2555930820,v1:172.21.15.16:6835/2555930820] conn(0x55607cc0f420 0x55607cc0f800 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:03.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.578+0000 7fd8ea842000 1 --2- 172.21.15.16:0/3081335050 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55607cc4edc0 0x55607cc0ea10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:03.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.578+0000 7fd8ea842000 1 --2- 172.21.15.16:0/3081335050 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55607cc4e4a0 0x55607cc56960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:03.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.578+0000 7fd8ea842000 1 --2- 172.21.15.16:0/3081335050 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55607cc020b0 0x55607cc56420 unknown :-1 s=CLOSED pgs=841 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:03.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.578+0000 7fd8c6000640 0 client.6852 ms_handle_reset on v2:172.21.15.16:6834/2555930820 2024-03-31T15:08:03.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.578+0000 7fd8ea842000 1 -- 172.21.15.16:0/3081335050 >> 172.21.15.16:0/3081335050 conn(0x55607caecb80 msgr2=0x55607cc01270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:08:03.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.582+0000 7fd8ea842000 1 -- 172.21.15.16:0/3081335050 shutdown_connections 2024-03-31T15:08:03.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.582+0000 7fd8ea842000 1 -- 172.21.15.16:0/3081335050 wait complete. 2024-03-31T15:08:03.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.582+0000 7fd8ea842000 20 client.6852 trim_cache size 0 max 16384 2024-03-31T15:08:03.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.598+0000 7fd8ea842000 1 Processor -- start 2024-03-31T15:08:03.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.598+0000 7fd8ea842000 1 -- start start 2024-03-31T15:08:03.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.598+0000 7fd8ea842000 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55607cc020b0 0x55607cc099a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:03.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.598+0000 7fd8ea842000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55607cc4e4a0 0x55607cc09ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:03.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.598+0000 7fd8ea842000 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55607cc4edc0 0x55607cc4d770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:03.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.598+0000 7fd8ea842000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fd8e0144e00 con 0x55607cc4e4a0 2024-03-31T15:08:03.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.598+0000 7fd8ea842000 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fd8e0145450 con 0x55607cc4edc0 2024-03-31T15:08:03.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.598+0000 7fd8ea842000 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x55607ca61060 con 0x55607cc020b0 2024-03-31T15:08:03.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.598+0000 7fd8dfe00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55607cc4e4a0 0x55607cc09ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:03.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.598+0000 7fd8dfe00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55607cc4e4a0 0x55607cc09ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51890/0 (socket says 172.21.15.16:51890) 2024-03-31T15:08:03.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.598+0000 7fd8dfe00640 1 -- 172.21.15.16:0/4010748740 learned_addr learned my addr 172.21.15.16:0/4010748740 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:08:03.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.598+0000 7fd8e5400640 1 --2- 172.21.15.16:0/4010748740 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55607cc4edc0 0x55607cc4d770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:03.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.598+0000 7fd8e4a00640 1 --2- 172.21.15.16:0/4010748740 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55607cc020b0 0x55607cc099a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:03.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.598+0000 7fd8dfe00640 1 -- 172.21.15.16:0/4010748740 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55607cc020b0 msgr2=0x55607cc099a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:08:03.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.598+0000 7fd8dfe00640 1 --2- 172.21.15.16:0/4010748740 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55607cc020b0 0x55607cc099a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:03.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.598+0000 7fd8dfe00640 1 -- 172.21.15.16:0/4010748740 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55607cc4edc0 msgr2=0x55607cc4d770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:08:03.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.598+0000 7fd8dfe00640 1 --2- 172.21.15.16:0/4010748740 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55607cc4edc0 0x55607cc4d770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:03.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.598+0000 7fd8dfe00640 1 -- 172.21.15.16:0/4010748740 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55607ca0f490 con 0x55607cc4e4a0 2024-03-31T15:08:03.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.598+0000 7fd8e4a00640 1 --2- 172.21.15.16:0/4010748740 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55607cc020b0 0x55607cc099a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:08:03.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.598+0000 7fd8dfe00640 1 --2- 172.21.15.16:0/4010748740 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55607cc4e4a0 0x55607cc09ee0 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7fd8e01437d0 tx=0x7fd8cc008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c4a04aeadcc5d43 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:08:03.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.602+0000 7fd8d0c00640 1 -- 172.21.15.16:0/4010748740 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8cc0181d0 con 0x55607cc4e4a0 2024-03-31T15:08:03.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.602+0000 7fd8d0c00640 1 -- 172.21.15.16:0/4010748740 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd8cc025070 con 0x55607cc4e4a0 2024-03-31T15:08:03.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.602+0000 7fd8d0c00640 1 -- 172.21.15.16:0/4010748740 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8cc004250 con 0x55607cc4e4a0 2024-03-31T15:08:03.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.602+0000 7fd8ea842000 1 -- 172.21.15.16:0/4010748740 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55607cc4e4a0 msgr2=0x55607cc09ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:03.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.602+0000 7fd8ea842000 1 --2- 172.21.15.16:0/4010748740 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55607cc4e4a0 0x55607cc09ee0 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7fd8e01437d0 tx=0x7fd8cc008040 comp rx=0 tx=0).stop 2024-03-31T15:08:03.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.602+0000 7fd8ea842000 1 -- 172.21.15.16:0/4010748740 shutdown_connections 2024-03-31T15:08:03.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.602+0000 7fd8ea842000 1 --2- 172.21.15.16:0/4010748740 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55607cc4edc0 0x55607cc4d770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:03.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.602+0000 7fd8ea842000 1 --2- 172.21.15.16:0/4010748740 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55607cc4e4a0 0x55607cc09ee0 unknown :-1 s=CLOSED pgs=842 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:03.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.602+0000 7fd8ea842000 1 --2- 172.21.15.16:0/4010748740 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55607cc020b0 0x55607cc099a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:03.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.602+0000 7fd8ea842000 1 -- 172.21.15.16:0/4010748740 >> 172.21.15.16:0/4010748740 conn(0x55607caecb80 msgr2=0x55607cc09290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:08:03.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.602+0000 7fd8ea842000 1 -- 172.21.15.16:0/4010748740 shutdown_connections 2024-03-31T15:08:03.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.602+0000 7fd8ea842000 1 -- 172.21.15.16:0/4010748740 wait complete. 2024-03-31T15:08:03.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.602+0000 7fd8ea842000 1 -- start start 2024-03-31T15:08:03.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.602+0000 7fd8ea842000 1 Processor -- start 2024-03-31T15:08:03.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.602+0000 7fd8ea842000 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55607cc020b0 0x55607ccbef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:03.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.602+0000 7fd8ea842000 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55607cc4e4a0 0x55607ccbf460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:03.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.602+0000 7fd8ea842000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55607cc4edc0 0x55607cc002d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:03.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.602+0000 7fd8ea842000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55607ca54950 con 0x55607cc4edc0 2024-03-31T15:08:03.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.602+0000 7fd8ea842000 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55607ca58f80 con 0x55607cc020b0 2024-03-31T15:08:03.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.602+0000 7fd8ea842000 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x55607ca53c60 con 0x55607cc4e4a0 2024-03-31T15:08:03.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.602+0000 7fd8ea842000 10 client.0 _ll_register_callbacks cb 0x7fd8cc009730 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T15:08:03.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.602+0000 7fd8e5400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55607cc4edc0 0x55607cc002d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:03.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.602+0000 7fd8dfe00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55607cc4e4a0 0x55607ccbf460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:03.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.602+0000 7fd8e4a00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55607cc020b0 0x55607ccbef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:03.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.602+0000 7fd8e5400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55607cc4edc0 0x55607cc002d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51894/0 (socket says 172.21.15.16:51894) 2024-03-31T15:08:03.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.602+0000 7fd8e5400640 1 -- 172.21.15.16:0/4002393447 learned_addr learned my addr 172.21.15.16:0/4002393447 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:08:03.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.602+0000 7fd8dfe00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55607cc4e4a0 0x55607ccbf460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3301/0 says I am v2:172.21.15.16:59812/0 (socket says 172.21.15.16:59812) 2024-03-31T15:08:03.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.602+0000 7fd8e5400640 1 -- 172.21.15.16:0/4002393447 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55607cc4e4a0 msgr2=0x55607ccbf460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:08:03.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.602+0000 7fd8e5400640 1 --2- 172.21.15.16:0/4002393447 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55607cc4e4a0 0x55607ccbf460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:03.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.602+0000 7fd8e5400640 1 -- 172.21.15.16:0/4002393447 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55607cc020b0 msgr2=0x55607ccbef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:08:03.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.602+0000 7fd8e5400640 1 --2- 172.21.15.16:0/4002393447 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55607cc020b0 0x55607ccbef20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:03.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.602+0000 7fd8e5400640 1 -- 172.21.15.16:0/4002393447 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55607ca0de80 con 0x55607cc4edc0 2024-03-31T15:08:03.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.602+0000 7fd8dfe00640 1 --2- 172.21.15.16:0/4002393447 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55607cc4e4a0 0x55607ccbf460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:08:03.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.602+0000 7fd8e5400640 1 --2- 172.21.15.16:0/4002393447 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55607cc4edc0 0x55607cc002d0 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7fd8d8015e40 tx=0x7fd8d8000f50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21f88773ff29b3a server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:08:03.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.602+0000 7fd8c6000640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-31T15:08:03.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.606+0000 7fd8c6000640 1 -- 172.21.15.16:0/4002393447 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8d8019070 con 0x55607cc4edc0 2024-03-31T15:08:03.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.606+0000 7fd8c6000640 1 -- 172.21.15.16:0/4002393447 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(67..67 src has 1..67) ==== 5709+0+0 (secure 0 0 0) 0x7fd8d8014b30 con 0x55607cc4edc0 2024-03-31T15:08:03.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.606+0000 7fd8ea842000 1 -- 172.21.15.16:0/4002393447 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd8cc025070 con 0x55607cc4edc0 2024-03-31T15:08:03.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.606+0000 7fd8c6000640 1 -- 172.21.15.16:0/4002393447 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=45) ==== 24+0+0 (secure 0 0 0) 0x7fd8d8033050 con 0x55607cc4edc0 2024-03-31T15:08:03.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.606+0000 7fd8ea842000 10 client.6858 fetch_fsmap learned FSMap version 45 2024-03-31T15:08:03.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.606+0000 7fd8ea842000 1 -- 172.21.15.16:0/4002393447 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=45,monmap=2+}) -- 0x55607c9fb270 con 0x55607cc4edc0 2024-03-31T15:08:03.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.606+0000 7fd8c6000640 1 -- 172.21.15.16:0/4002393447 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd8cc025070 con 0x55607cc4edc0 2024-03-31T15:08:03.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.606+0000 7fd8c6000640 1 -- 172.21.15.16:0/4002393447 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 45) ==== 3494+0+0 (secure 0 0 0) 0x7fd8d8036020 con 0x55607cc4edc0 2024-03-31T15:08:03.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.606+0000 7fd8ea842000 10 client.6858 fetch_fsmap finished waiting for FSMap version 45 2024-03-31T15:08:03.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.606+0000 7fd8ea842000 10 client.6858 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-31T15:08:03.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.606+0000 7fd8ea842000 20 client.6858 populate_metadata read hostname 'smithi016' 2024-03-31T15:08:03.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.606+0000 7fd8ea842000 1 --2- 172.21.15.16:0/4002393447 >> [v2:172.21.15.16:6834/2555930820,v1:172.21.15.16:6835/2555930820] conn(0x55607cc00810 0x55607cc00bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:03.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.606+0000 7fd8ea842000 4 client.6858 mds_command: new command op to 6714 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-31T15:08:03.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.606+0000 7fd8ea842000 1 -- 172.21.15.16:0/4002393447 --> [v2:172.21.15.16:6834/2555930820,v1:172.21.15.16:6835/2555930820] -- command(tid 0: {"prefix": "status"}) -- 0x55607ca61e90 con 0x55607cc00810 2024-03-31T15:08:03.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.606+0000 7fd8e4a00640 1 --2- 172.21.15.16:0/4002393447 >> [v2:172.21.15.16:6834/2555930820,v1:172.21.15.16:6835/2555930820] conn(0x55607cc00810 0x55607cc00bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:03.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.606+0000 7fd8e4a00640 1 --2- 172.21.15.16:0/4002393447 >> [v2:172.21.15.16:6834/2555930820,v1:172.21.15.16:6835/2555930820] conn(0x55607cc00810 0x55607cc00bf0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d13ac3158cd3cbd2 server_cookie=62afe17f073cd92f in_seq=0 out_seq=0 2024-03-31T15:08:03.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.606+0000 7fd8c6000640 10 client.6858 ms_handle_connect on v2:172.21.15.16:6834/2555930820 2024-03-31T15:08:03.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.606+0000 7fd8c6000640 1 -- 172.21.15.16:0/4002393447 <== mds.0 v2:172.21.15.16:6834/2555930820 1 ==== command_reply(tid 0: 0 ) ==== 8+0+318 (crc 0 0 0) 0x55607ca61e90 con 0x55607cc00810 2024-03-31T15:08:03.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.606+0000 7fd8c6000640 10 client.6858 handle_command_reply: tid=0 2024-03-31T15:08:03.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.606+0000 7fd8ea842000 1 client.6858 shutdown 2024-03-31T15:08:03.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.606+0000 7fd8ea842000 1 -- 172.21.15.16:0/4002393447 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55607cc4edc0 msgr2=0x55607cc002d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:03.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.606+0000 7fd8ea842000 1 --2- 172.21.15.16:0/4002393447 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55607cc4edc0 0x55607cc002d0 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7fd8d8015e40 tx=0x7fd8d8000f50 comp rx=0 tx=0).stop 2024-03-31T15:08:03.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.606+0000 7fd8ea842000 1 -- 172.21.15.16:0/4002393447 shutdown_connections 2024-03-31T15:08:03.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.606+0000 7fd8ea842000 1 --2- 172.21.15.16:0/4002393447 >> [v2:172.21.15.16:6834/2555930820,v1:172.21.15.16:6835/2555930820] conn(0x55607cc00810 0x55607cc00bf0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:03.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.606+0000 7fd8c6000640 0 client.6858 ms_handle_reset on v2:172.21.15.16:6834/2555930820 2024-03-31T15:08:03.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.606+0000 7fd8ea842000 1 --2- 172.21.15.16:0/4002393447 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55607cc4edc0 0x55607cc002d0 unknown :-1 s=CLOSED pgs=843 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:03.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.606+0000 7fd8ea842000 1 --2- 172.21.15.16:0/4002393447 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55607cc4e4a0 0x55607ccbf460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:03.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.606+0000 7fd8ea842000 1 --2- 172.21.15.16:0/4002393447 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55607cc020b0 0x55607ccbef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:03.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.606+0000 7fd8ea842000 1 -- 172.21.15.16:0/4002393447 >> 172.21.15.16:0/4002393447 conn(0x55607caecb80 msgr2=0x55607cc09b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:08:03.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.606+0000 7fd8ea842000 1 -- 172.21.15.16:0/4002393447 shutdown_connections 2024-03-31T15:08:03.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.606+0000 7fd8ea842000 1 -- 172.21.15.16:0/4002393447 wait complete. 2024-03-31T15:08:03.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.606+0000 7fd8ea842000 20 client.6858 trim_cache size 0 max 16384 2024-03-31T15:08:03.607 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-03-31T15:08:03.607 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_fsid": "286eb5da-fb47-4550-b290-d9519c13b9a5", 2024-03-31T15:08:03.607 INFO:teuthology.orchestra.run.smithi016.stdout: "whoami": 0, 2024-03-31T15:08:03.607 INFO:teuthology.orchestra.run.smithi016.stdout: "id": 6714, 2024-03-31T15:08:03.607 INFO:teuthology.orchestra.run.smithi016.stdout: "want_state": "up:active", 2024-03-31T15:08:03.607 INFO:teuthology.orchestra.run.smithi016.stdout: "state": "up:active", 2024-03-31T15:08:03.607 INFO:teuthology.orchestra.run.smithi016.stdout: "fs_name": "cephfs", 2024-03-31T15:08:03.607 INFO:teuthology.orchestra.run.smithi016.stdout: "rank_uptime": 6.069769354, 2024-03-31T15:08:03.607 INFO:teuthology.orchestra.run.smithi016.stdout: "mdsmap_epoch": 45, 2024-03-31T15:08:03.607 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch": 67, 2024-03-31T15:08:03.608 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch_barrier": 67, 2024-03-31T15:08:03.608 INFO:teuthology.orchestra.run.smithi016.stdout: "uptime": 11.003658798 2024-03-31T15:08:03.608 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-03-31T15:08:03.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.606+0000 7fd8e7400640 1 -- 172.21.15.16:0/4259179391 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fd8c0068380 msgr2=0x7fd8c006a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:03.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.606+0000 7fd8e7400640 1 --2- 172.21.15.16:0/4259179391 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fd8c0068380 0x7fd8c006a840 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7fd8d8007be0 tx=0x7fd8d8009040 comp rx=0 tx=0).stop 2024-03-31T15:08:03.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.606+0000 7fd8e7400640 1 -- 172.21.15.16:0/4259179391 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd8e006fe60 msgr2=0x7fd8e013fd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:03.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.606+0000 7fd8e7400640 1 --2- 172.21.15.16:0/4259179391 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd8e006fe60 0x7fd8e013fd70 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7fd8d401abd0 tx=0x7fd8d401ad60 comp rx=0 tx=0).stop 2024-03-31T15:08:03.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.606+0000 7fd8e7400640 1 -- 172.21.15.16:0/4259179391 shutdown_connections 2024-03-31T15:08:03.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.606+0000 7fd8e7400640 1 --2- 172.21.15.16:0/4259179391 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fd8c0068380 0x7fd8c006a840 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:03.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.606+0000 7fd8e7400640 1 --2- 172.21.15.16:0/4259179391 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd8e0147dd0 0x7fd8e0142730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:03.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.606+0000 7fd8e7400640 1 --2- 172.21.15.16:0/4259179391 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fd8e0147280 0x7fd8e00b53b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:03.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.606+0000 7fd8e7400640 1 --2- 172.21.15.16:0/4259179391 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd8e006fe60 0x7fd8e013fd70 unknown :-1 s=CLOSED pgs=839 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:03.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.606+0000 7fd8e7400640 1 -- 172.21.15.16:0/4259179391 >> 172.21.15.16:0/4259179391 conn(0x7fd8e0067330 msgr2=0x7fd8e0067710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:08:03.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.606+0000 7fd8e7400640 1 -- 172.21.15.16:0/4259179391 shutdown_connections 2024-03-31T15:08:03.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.606+0000 7fd8e7400640 1 -- 172.21.15.16:0/4259179391 wait complete. 2024-03-31T15:08:03.621 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-31T15:08:03.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.766+0000 7f55c8400640 1 Processor -- start 2024-03-31T15:08:03.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.766+0000 7f55c8400640 1 -- start start 2024-03-31T15:08:03.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.766+0000 7f55c8400640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f55c0145e40 0x7f55c00cc740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:03.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.766+0000 7f55c8400640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f55c00ccc80 0x7f55c006f920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:03.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.766+0000 7f55c8400640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f55c006fe60 0x7f55c015fdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:03.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.766+0000 7f55c8400640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f55c00d06d0 con 0x7f55c006fe60 2024-03-31T15:08:03.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.766+0000 7f55c8400640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f55c00d0920 con 0x7f55c00ccc80 2024-03-31T15:08:03.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.766+0000 7f55c8400640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f55c00d0c00 con 0x7f55c0145e40 2024-03-31T15:08:03.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.770+0000 7f55c6400640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f55c006fe60 0x7f55c015fdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:03.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.770+0000 7f55c6400640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f55c006fe60 0x7f55c015fdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51896/0 (socket says 172.21.15.16:51896) 2024-03-31T15:08:03.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.770+0000 7f55c6400640 1 -- 172.21.15.16:0/1420929507 learned_addr learned my addr 172.21.15.16:0/1420929507 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:08:03.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.770+0000 7f55c5000640 1 --2- 172.21.15.16:0/1420929507 >> v2:172.21.15.89:3300/0 conn(0x7f55c00ccc80 0x7f55c006f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:03.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.770+0000 7f55c5a00640 1 --2- 172.21.15.16:0/1420929507 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f55c0145e40 0x7f55c00cc740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:03.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.770+0000 7f55c6400640 1 -- 172.21.15.16:0/1420929507 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f55c0145e40 msgr2=0x7f55c00cc740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:08:03.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.770+0000 7f55c6400640 1 --2- 172.21.15.16:0/1420929507 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f55c0145e40 0x7f55c00cc740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:03.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.770+0000 7f55c6400640 1 -- 172.21.15.16:0/1420929507 >> v2:172.21.15.89:3300/0 conn(0x7f55c00ccc80 msgr2=0x7f55c006f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:08:03.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.770+0000 7f55c6400640 1 --2- 172.21.15.16:0/1420929507 >> v2:172.21.15.89:3300/0 conn(0x7f55c00ccc80 0x7f55c006f920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:03.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.770+0000 7f55c6400640 1 -- 172.21.15.16:0/1420929507 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55c00d1810 con 0x7f55c006fe60 2024-03-31T15:08:03.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.770+0000 7f55c6400640 1 --2- 172.21.15.16:0/1420929507 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f55c006fe60 0x7f55c015fdf0 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7f55c0068280 tx=0x7f55b801c730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed4caae553a09c63 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:08:03.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.770+0000 7f55bfe00640 1 -- 172.21.15.16:0/1420929507 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55b80ac070 con 0x7f55c006fe60 2024-03-31T15:08:03.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.774+0000 7f55bfe00640 1 -- 172.21.15.16:0/1420929507 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f55b80a8420 con 0x7f55c006fe60 2024-03-31T15:08:03.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.774+0000 7f55bfe00640 1 -- 172.21.15.16:0/1420929507 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55b80a87f0 con 0x7f55c006fe60 2024-03-31T15:08:03.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.774+0000 7f55c8400640 1 -- 172.21.15.16:0/1420929507 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f55c006fe60 msgr2=0x7f55c015fdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:03.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.774+0000 7f55c8400640 1 --2- 172.21.15.16:0/1420929507 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f55c006fe60 0x7f55c015fdf0 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7f55c0068280 tx=0x7f55b801c730 comp rx=0 tx=0).stop 2024-03-31T15:08:03.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.774+0000 7f55c8400640 1 -- 172.21.15.16:0/1420929507 shutdown_connections 2024-03-31T15:08:03.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.774+0000 7f55c8400640 1 --2- 172.21.15.16:0/1420929507 >> v2:172.21.15.89:3300/0 conn(0x7f55c00ccc80 0x7f55c006f920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:03.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.774+0000 7f55c8400640 1 --2- 172.21.15.16:0/1420929507 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f55c006fe60 0x7f55c015fdf0 unknown :-1 s=CLOSED pgs=844 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:03.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.774+0000 7f55c8400640 1 --2- 172.21.15.16:0/1420929507 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f55c0145e40 0x7f55c00cc740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:03.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.774+0000 7f55c8400640 1 -- 172.21.15.16:0/1420929507 >> 172.21.15.16:0/1420929507 conn(0x7f55c0067330 msgr2=0x7f55c0067740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:08:03.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.774+0000 7f55c8400640 1 -- 172.21.15.16:0/1420929507 shutdown_connections 2024-03-31T15:08:03.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.774+0000 7f55c8400640 1 -- 172.21.15.16:0/1420929507 wait complete. 2024-03-31T15:08:03.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.774+0000 7f55c8400640 1 Processor -- start 2024-03-31T15:08:03.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.774+0000 7f55c8400640 1 -- start start 2024-03-31T15:08:03.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.774+0000 7f55c8400640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f55c006fe60 0x7f55c0159f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:03.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.774+0000 7f55c8400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f55c00ccc80 0x7f55c015a450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:03.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.774+0000 7f55c8400640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f55c0145e40 0x7f55c0131aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:03.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.774+0000 7f55c8400640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f55c00b5240 con 0x7f55c00ccc80 2024-03-31T15:08:03.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.774+0000 7f55c8400640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f55c00b50c0 con 0x7f55c006fe60 2024-03-31T15:08:03.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.774+0000 7f55c8400640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f55c00d2010 con 0x7f55c0145e40 2024-03-31T15:08:03.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.774+0000 7f55c5000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f55c00ccc80 0x7f55c015a450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:03.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.774+0000 7f55c5000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f55c00ccc80 0x7f55c015a450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51904/0 (socket says 172.21.15.16:51904) 2024-03-31T15:08:03.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.774+0000 7f55c5000640 1 -- 172.21.15.16:0/3000274113 learned_addr learned my addr 172.21.15.16:0/3000274113 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:08:03.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.774+0000 7f55c5a00640 1 --2- 172.21.15.16:0/3000274113 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f55c006fe60 0x7f55c0159f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:03.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.774+0000 7f55c6400640 1 --2- 172.21.15.16:0/3000274113 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f55c0145e40 0x7f55c0131aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:03.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.778+0000 7f55c5000640 1 -- 172.21.15.16:0/3000274113 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f55c0145e40 msgr2=0x7f55c0131aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:03.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.778+0000 7f55c5000640 1 --2- 172.21.15.16:0/3000274113 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f55c0145e40 0x7f55c0131aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:03.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.778+0000 7f55c5000640 1 -- 172.21.15.16:0/3000274113 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f55c006fe60 msgr2=0x7f55c0159f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:03.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.778+0000 7f55c5000640 1 --2- 172.21.15.16:0/3000274113 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f55c006fe60 0x7f55c0159f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:03.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.778+0000 7f55c5000640 1 -- 172.21.15.16:0/3000274113 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55c0153d10 con 0x7f55c00ccc80 2024-03-31T15:08:03.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.778+0000 7f55c5000640 1 --2- 172.21.15.16:0/3000274113 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f55c00ccc80 0x7f55c015a450 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7f55ac00e9e0 tx=0x7f55ac00eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:08:03.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.778+0000 7f55bd600640 1 -- 172.21.15.16:0/3000274113 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55ac017070 con 0x7f55c00ccc80 2024-03-31T15:08:03.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.778+0000 7f55bd600640 1 -- 172.21.15.16:0/3000274113 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f55ac00c9d0 con 0x7f55c00ccc80 2024-03-31T15:08:03.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.778+0000 7f55c8400640 1 -- 172.21.15.16:0/3000274113 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f55c0131030 con 0x7f55c00ccc80 2024-03-31T15:08:03.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.778+0000 7f55bd600640 1 -- 172.21.15.16:0/3000274113 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55ac00cda0 con 0x7f55c00ccc80 2024-03-31T15:08:03.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.778+0000 7f55c8400640 1 -- 172.21.15.16:0/3000274113 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f55c01315f0 con 0x7f55c00ccc80 2024-03-31T15:08:03.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.778+0000 7f55c8400640 1 -- 172.21.15.16:0/3000274113 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f55c01523c0 con 0x7f55c00ccc80 2024-03-31T15:08:03.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.778+0000 7f55bd600640 1 -- 172.21.15.16:0/3000274113 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f55ac01a070 con 0x7f55c00ccc80 2024-03-31T15:08:03.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.778+0000 7f55bd600640 1 --2- 172.21.15.16:0/3000274113 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f55a0068380 0x7f55a006a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:03.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.778+0000 7f55bd600640 1 -- 172.21.15.16:0/3000274113 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(67..67 src has 1..67) ==== 5709+0+0 (secure 0 0 0) 0x7f55ac0869f0 con 0x7f55c00ccc80 2024-03-31T15:08:03.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.782+0000 7f55c5a00640 1 --2- 172.21.15.16:0/3000274113 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f55a0068380 0x7f55a006a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:03.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.782+0000 7f55c5a00640 1 --2- 172.21.15.16:0/3000274113 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f55a0068380 0x7f55a006a840 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f55b4009a40 tx=0x7f55b4009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:08:03.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.790+0000 7f55bd600640 1 -- 172.21.15.16:0/3000274113 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f55ac0568c0 con 0x7f55c00ccc80 2024-03-31T15:08:03.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.998+0000 7f55c8400640 1 -- 172.21.15.16:0/3000274113 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f55c0069950 con 0x7f55c00ccc80 2024-03-31T15:08:03.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:03.998+0000 7f55bd600640 1 -- 172.21.15.16:0/3000274113 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v45) ==== 53+0+73 (secure 0 0 0) 0x7f55ac05abc0 con 0x7f55c00ccc80 2024-03-31T15:08:03.995 INFO:teuthology.orchestra.run.smithi016.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-31T15:08:03.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:04.002+0000 7f55c8400640 1 -- 172.21.15.16:0/3000274113 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f55a0068380 msgr2=0x7f55a006a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:03.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:04.002+0000 7f55c8400640 1 --2- 172.21.15.16:0/3000274113 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f55a0068380 0x7f55a006a840 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f55b4009a40 tx=0x7f55b4009190 comp rx=0 tx=0).stop 2024-03-31T15:08:03.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:04.002+0000 7f55c8400640 1 -- 172.21.15.16:0/3000274113 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f55c00ccc80 msgr2=0x7f55c015a450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:03.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:04.002+0000 7f55c8400640 1 --2- 172.21.15.16:0/3000274113 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f55c00ccc80 0x7f55c015a450 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7f55ac00e9e0 tx=0x7f55ac00eea0 comp rx=0 tx=0).stop 2024-03-31T15:08:03.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:04.002+0000 7f55c8400640 1 -- 172.21.15.16:0/3000274113 shutdown_connections 2024-03-31T15:08:03.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:04.002+0000 7f55c8400640 1 --2- 172.21.15.16:0/3000274113 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f55a0068380 0x7f55a006a840 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:03.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:04.002+0000 7f55c8400640 1 --2- 172.21.15.16:0/3000274113 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f55c0145e40 0x7f55c0131aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:03.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:04.002+0000 7f55c8400640 1 --2- 172.21.15.16:0/3000274113 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f55c00ccc80 0x7f55c015a450 unknown :-1 s=CLOSED pgs=845 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:03.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:04.002+0000 7f55c8400640 1 --2- 172.21.15.16:0/3000274113 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f55c006fe60 0x7f55c0159f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:03.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:04.002+0000 7f55c8400640 1 -- 172.21.15.16:0/3000274113 >> 172.21.15.16:0/3000274113 conn(0x7f55c0067330 msgr2=0x7f55c00cba30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:08:03.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:04.002+0000 7f55c8400640 1 -- 172.21.15.16:0/3000274113 shutdown_connections 2024-03-31T15:08:03.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:04.002+0000 7f55c8400640 1 -- 172.21.15.16:0/3000274113 wait complete. 2024-03-31T15:08:04.016 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-31T15:08:04.016 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-31T15:08:04.016 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi016.front.sepia.ceph.com 2024-03-31T15:08:04.016 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-31T15:08:04.016 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-31T15:08:04.016 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-31T15:08:04.016 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-31T15:08:04.016 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-31T15:08:04.016 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-31T15:08:04.016 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:08:04.016 DEBUG:teuthology.orchestra.run.smithi016:> ip addr 2024-03-31T15:08:04.026 INFO:teuthology.orchestra.run.smithi016.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-31T15:08:04.026 INFO:teuthology.orchestra.run.smithi016.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-31T15:08:04.026 INFO:teuthology.orchestra.run.smithi016.stdout: inet 127.0.0.1/8 scope host lo 2024-03-31T15:08:04.026 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2024-03-31T15:08:04.026 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 ::1/128 scope host 2024-03-31T15:08:04.026 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2024-03-31T15:08:04.026 INFO:teuthology.orchestra.run.smithi016.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-03-31T15:08:04.026 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:6b:fd:e0 brd ff:ff:ff:ff:ff:ff 2024-03-31T15:08:04.026 INFO:teuthology.orchestra.run.smithi016.stdout: altname enp7s0f0 2024-03-31T15:08:04.026 INFO:teuthology.orchestra.run.smithi016.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-03-31T15:08:04.027 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:6b:fd:e1 brd ff:ff:ff:ff:ff:ff 2024-03-31T15:08:04.027 INFO:teuthology.orchestra.run.smithi016.stdout: altname enp7s0f1 2024-03-31T15:08:04.027 INFO:teuthology.orchestra.run.smithi016.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-31T15:08:04.027 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:bd:15:64 brd ff:ff:ff:ff:ff:ff 2024-03-31T15:08:04.027 INFO:teuthology.orchestra.run.smithi016.stdout: altname enp1s0f0 2024-03-31T15:08:04.027 INFO:teuthology.orchestra.run.smithi016.stdout: inet 172.21.15.16/20 brd 172.21.15.255 scope global dynamic ens1f0 2024-03-31T15:08:04.027 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft 42197sec preferred_lft 42197sec 2024-03-31T15:08:04.027 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 fe80::ec4:7aff:febd:1564/64 scope link 2024-03-31T15:08:04.027 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2024-03-31T15:08:04.027 INFO:teuthology.orchestra.run.smithi016.stdout:5: ens1f1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-03-31T15:08:04.027 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:bd:15:65 brd ff:ff:ff:ff:ff:ff 2024-03-31T15:08:04.027 INFO:teuthology.orchestra.run.smithi016.stdout: altname enp1s0f1 2024-03-31T15:08:04.027 INFO:teuthology.orchestra.run.smithi016.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2024-03-31T15:08:04.027 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 02:42:d4:1f:7d:73 brd ff:ff:ff:ff:ff:ff 2024-03-31T15:08:04.028 INFO:teuthology.orchestra.run.smithi016.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2024-03-31T15:08:04.028 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2024-03-31T15:08:04.028 INFO:teuthology.orchestra.run.smithi016.stdout:7: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-31T15:08:04.028 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 5e:1f:84:6d:79:ae brd ff:ff:ff:ff:ff:ff 2024-03-31T15:08:04.028 INFO:teuthology.orchestra.run.smithi016.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-31T15:08:04.028 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2024-03-31T15:08:04.028 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 fe80::5c1f:84ff:fe6d:79ae/64 scope link 2024-03-31T15:08:04.028 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2024-03-31T15:08:04.028 INFO:teuthology.orchestra.run.smithi016.stdout:8: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-31T15:08:04.028 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0e:5c:f6:1f:82:2f brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-31T15:08:04.028 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 fe80::c5c:f6ff:fe1f:822f/64 scope link 2024-03-31T15:08:04.028 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2024-03-31T15:08:04.028 INFO:teuthology.orchestra.run.smithi016.stdout:9: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-31T15:08:04.028 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether c6:a6:3d:69:d0:db brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-31T15:08:04.029 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 fe80::c4a6:3dff:fe69:d0db/64 scope link 2024-03-31T15:08:04.029 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2024-03-31T15:08:04.029 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:08:04.029 DEBUG:teuthology.orchestra.run.smithi016:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-31T15:08:04.085 INFO:teuthology.orchestra.run.smithi016.stdout:1 2024-03-31T15:08:04.087 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:08:04.087 DEBUG:teuthology.orchestra.run.smithi016:> route 2024-03-31T15:08:04.140 INFO:teuthology.orchestra.run.smithi016.stdout:Kernel IP routing table 2024-03-31T15:08:04.140 INFO:teuthology.orchestra.run.smithi016.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-31T15:08:04.140 INFO:teuthology.orchestra.run.smithi016.stdout:default _gateway 0.0.0.0 UG 0 0 0 ens1f0 2024-03-31T15:08:04.140 INFO:teuthology.orchestra.run.smithi016.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2024-03-31T15:08:04.140 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 0 0 0 ens1f0 2024-03-31T15:08:04.140 INFO:teuthology.orchestra.run.smithi016.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-31T15:08:04.140 DEBUG:teuthology.orchestra.run.smithi016:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-31T15:08:04.140 DEBUG:teuthology.orchestra.run.smithi016:> set -e 2024-03-31T15:08:04.141 DEBUG:teuthology.orchestra.run.smithi016:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-03-31T15:08:04.141 DEBUG:teuthology.orchestra.run.smithi016:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-03-31T15:08:04.141 DEBUG:teuthology.orchestra.run.smithi016:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-03-31T15:08:04.141 DEBUG:teuthology.orchestra.run.smithi016:> ') 2024-03-31T15:08:04.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:04 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-31T15:08:04.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:04 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-31T15:08:04.279 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:08:04.279 DEBUG:teuthology.orchestra.run.smithi016:> ip netns list 2024-03-31T15:08:04.288 INFO:teuthology.orchestra.run.smithi016.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-31T15:08:04.288 INFO:teuthology.orchestra.run.smithi016.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-31T15:08:04.289 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-31T15:08:04.289 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:08:04.289 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip link set brx.0 up 2024-03-31T15:08:04.350 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-03-31T15:08:04.350 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-31T15:08:04.350 DEBUG:teuthology.orchestra.run.smithi016:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-31T15:08:04.399 INFO:teuthology.orchestra.run.smithi016.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-31T15:08:04.399 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-31T15:08:04.399 DEBUG:teuthology.orchestra.run.smithi016:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-31T15:08:04.450 DEBUG:teuthology.orchestra.run.smithi016:> sudo modprobe fuse 2024-03-31T15:08:04.516 DEBUG:teuthology.orchestra.run.smithi016:> cat /proc/self/mounts | awk '{print $2}' 2024-03-31T15:08:04.569 INFO:teuthology.orchestra.run.smithi016.stdout:/sys 2024-03-31T15:08:04.570 INFO:teuthology.orchestra.run.smithi016.stdout:/proc 2024-03-31T15:08:04.570 INFO:teuthology.orchestra.run.smithi016.stdout:/dev 2024-03-31T15:08:04.570 INFO:teuthology.orchestra.run.smithi016.stdout:/dev/pts 2024-03-31T15:08:04.570 INFO:teuthology.orchestra.run.smithi016.stdout:/run 2024-03-31T15:08:04.570 INFO:teuthology.orchestra.run.smithi016.stdout:/ 2024-03-31T15:08:04.570 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/kernel/security 2024-03-31T15:08:04.570 INFO:teuthology.orchestra.run.smithi016.stdout:/dev/shm 2024-03-31T15:08:04.570 INFO:teuthology.orchestra.run.smithi016.stdout:/run/lock 2024-03-31T15:08:04.570 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/cgroup 2024-03-31T15:08:04.570 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/pstore 2024-03-31T15:08:04.570 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/bpf 2024-03-31T15:08:04.571 INFO:teuthology.orchestra.run.smithi016.stdout:/proc/sys/fs/binfmt_misc 2024-03-31T15:08:04.571 INFO:teuthology.orchestra.run.smithi016.stdout:/dev/hugepages 2024-03-31T15:08:04.571 INFO:teuthology.orchestra.run.smithi016.stdout:/dev/mqueue 2024-03-31T15:08:04.571 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/kernel/debug 2024-03-31T15:08:04.571 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/kernel/tracing 2024-03-31T15:08:04.571 INFO:teuthology.orchestra.run.smithi016.stdout:/proc/fs/nfsd 2024-03-31T15:08:04.571 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/fuse/connections 2024-03-31T15:08:04.571 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/kernel/config 2024-03-31T15:08:04.571 INFO:teuthology.orchestra.run.smithi016.stdout:/run/credentials/systemd-sysusers.service 2024-03-31T15:08:04.571 INFO:teuthology.orchestra.run.smithi016.stdout:/run/rpc_pipefs 2024-03-31T15:08:04.571 INFO:teuthology.orchestra.run.smithi016.stdout:/run/user/1000 2024-03-31T15:08:04.571 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/ceph 2024-03-31T15:08:04.571 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/ceph/osd/ceph-0 2024-03-31T15:08:04.571 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/ceph/osd/ceph-1 2024-03-31T15:08:04.571 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/ceph/osd/ceph-2 2024-03-31T15:08:04.571 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/ceph/osd/ceph-3 2024-03-31T15:08:04.571 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns 2024-03-31T15:08:04.572 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-31T15:08:04.572 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-31T15:08:04.572 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-31T15:08:04.572 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-31T15:08:04.572 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:08:04.573 DEBUG:teuthology.orchestra.run.smithi016:> ls /sys/fs/fuse/connections 2024-03-31T15:08:04.623 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-31T15:08:04.623 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-03-31T15:08:04.669 DEBUG:teuthology.orchestra.run.smithi016:> sudo modprobe fuse 2024-03-31T15:08:04.689 DEBUG:teuthology.orchestra.run.smithi016:> cat /proc/self/mounts | awk '{print $2}' 2024-03-31T15:08:04.740 INFO:teuthology.orchestra.run.smithi016.stdout:/sys 2024-03-31T15:08:04.740 INFO:teuthology.orchestra.run.smithi016.stdout:/proc 2024-03-31T15:08:04.740 INFO:teuthology.orchestra.run.smithi016.stdout:/dev 2024-03-31T15:08:04.740 INFO:teuthology.orchestra.run.smithi016.stdout:/dev/pts 2024-03-31T15:08:04.740 INFO:teuthology.orchestra.run.smithi016.stdout:/run 2024-03-31T15:08:04.740 INFO:teuthology.orchestra.run.smithi016.stdout:/ 2024-03-31T15:08:04.741 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/kernel/security 2024-03-31T15:08:04.741 INFO:teuthology.orchestra.run.smithi016.stdout:/dev/shm 2024-03-31T15:08:04.741 INFO:teuthology.orchestra.run.smithi016.stdout:/run/lock 2024-03-31T15:08:04.741 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/cgroup 2024-03-31T15:08:04.741 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/pstore 2024-03-31T15:08:04.741 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/bpf 2024-03-31T15:08:04.741 INFO:teuthology.orchestra.run.smithi016.stdout:/proc/sys/fs/binfmt_misc 2024-03-31T15:08:04.741 INFO:teuthology.orchestra.run.smithi016.stdout:/dev/hugepages 2024-03-31T15:08:04.741 INFO:teuthology.orchestra.run.smithi016.stdout:/dev/mqueue 2024-03-31T15:08:04.741 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/kernel/debug 2024-03-31T15:08:04.741 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/kernel/tracing 2024-03-31T15:08:04.741 INFO:teuthology.orchestra.run.smithi016.stdout:/proc/fs/nfsd 2024-03-31T15:08:04.741 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/fuse/connections 2024-03-31T15:08:04.741 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/kernel/config 2024-03-31T15:08:04.741 INFO:teuthology.orchestra.run.smithi016.stdout:/run/credentials/systemd-sysusers.service 2024-03-31T15:08:04.742 INFO:teuthology.orchestra.run.smithi016.stdout:/run/rpc_pipefs 2024-03-31T15:08:04.742 INFO:teuthology.orchestra.run.smithi016.stdout:/run/user/1000 2024-03-31T15:08:04.742 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/ceph 2024-03-31T15:08:04.742 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/ceph/osd/ceph-0 2024-03-31T15:08:04.742 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/ceph/osd/ceph-1 2024-03-31T15:08:04.742 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/ceph/osd/ceph-2 2024-03-31T15:08:04.742 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/ceph/osd/ceph-3 2024-03-31T15:08:04.742 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns 2024-03-31T15:08:04.742 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-31T15:08:04.742 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-31T15:08:04.742 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-31T15:08:04.742 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-31T15:08:04.743 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:08:04.743 DEBUG:teuthology.orchestra.run.smithi016:> ls /sys/fs/fuse/connections 2024-03-31T15:08:04.771 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi016.stderr:2024-03-31T15:08:04.774+0000 7f7275c33580 -1 init, newargv = 0x558f1086fa50 newargc=15 2024-03-31T15:08:04.771 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi016.stderr:ceph-fuse[41790]: starting ceph client 2024-03-31T15:08:04.779 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi016.stderr:ceph-fuse[41790]: starting fuse 2024-03-31T15:08:04.781 INFO:teuthology.orchestra.run.smithi016.stdout:43 2024-03-31T15:08:04.781 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [43] 2024-03-31T15:08:04.781 DEBUG:teuthology.orchestra.run.smithi016:> sudo stdin-killer -- python3 -c ' 2024-03-31T15:08:04.781 DEBUG:teuthology.orchestra.run.smithi016:> import glob 2024-03-31T15:08:04.781 DEBUG:teuthology.orchestra.run.smithi016:> import re 2024-03-31T15:08:04.781 DEBUG:teuthology.orchestra.run.smithi016:> import os 2024-03-31T15:08:04.782 DEBUG:teuthology.orchestra.run.smithi016:> import subprocess 2024-03-31T15:08:04.782 DEBUG:teuthology.orchestra.run.smithi016:> 2024-03-31T15:08:04.782 DEBUG:teuthology.orchestra.run.smithi016:> def _find_admin_socket(client_name): 2024-03-31T15:08:04.782 DEBUG:teuthology.orchestra.run.smithi016:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-03-31T15:08:04.782 DEBUG:teuthology.orchestra.run.smithi016:> files = glob.glob(asok_path) 2024-03-31T15:08:04.782 DEBUG:teuthology.orchestra.run.smithi016:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-03-31T15:08:04.782 DEBUG:teuthology.orchestra.run.smithi016:> 2024-03-31T15:08:04.782 DEBUG:teuthology.orchestra.run.smithi016:> # Given a non-glob path, it better be there 2024-03-31T15:08:04.782 DEBUG:teuthology.orchestra.run.smithi016:> if "*" not in asok_path: 2024-03-31T15:08:04.782 DEBUG:teuthology.orchestra.run.smithi016:> assert(len(files) == 1) 2024-03-31T15:08:04.782 DEBUG:teuthology.orchestra.run.smithi016:> return files[0] 2024-03-31T15:08:04.782 DEBUG:teuthology.orchestra.run.smithi016:> 2024-03-31T15:08:04.782 DEBUG:teuthology.orchestra.run.smithi016:> for f in files: 2024-03-31T15:08:04.782 DEBUG:teuthology.orchestra.run.smithi016:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-31T15:08:04.783 DEBUG:teuthology.orchestra.run.smithi016:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-31T15:08:04.783 DEBUG:teuthology.orchestra.run.smithi016:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-31T15:08:04.783 DEBUG:teuthology.orchestra.run.smithi016:> contents = proc_f.read() 2024-03-31T15:08:04.783 DEBUG:teuthology.orchestra.run.smithi016:> if mountpoint in contents: 2024-03-31T15:08:04.783 DEBUG:teuthology.orchestra.run.smithi016:> return f 2024-03-31T15:08:04.783 DEBUG:teuthology.orchestra.run.smithi016:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-31T15:08:04.783 DEBUG:teuthology.orchestra.run.smithi016:> 2024-03-31T15:08:04.783 DEBUG:teuthology.orchestra.run.smithi016:> print(_find_admin_socket("client.0")) 2024-03-31T15:08:04.783 DEBUG:teuthology.orchestra.run.smithi016:> ' 2024-03-31T15:08:04.905 INFO:teuthology.orchestra.run.smithi016.stdout:/var/run/ceph/ceph-client.0.41790.asok 2024-03-31T15:08:04.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:04 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-31T15:08:04.916 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.41790.asok 2024-03-31T15:08:04.916 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:08:04.916 DEBUG:teuthology.orchestra.run.smithi016:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.41790.asok status 2024-03-31T15:08:05.090 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-03-31T15:08:05.090 INFO:teuthology.orchestra.run.smithi016.stdout: "metadata": { 2024-03-31T15:08:05.090 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph_sha1": "d2271579d5e735cfd6d436b8349c004f06527cb0", 2024-03-31T15:08:05.090 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph_version": "ceph version 19.0.0-2574-gd2271579 (d2271579d5e735cfd6d436b8349c004f06527cb0) squid (dev)", 2024-03-31T15:08:05.090 INFO:teuthology.orchestra.run.smithi016.stdout: "entity_id": "0", 2024-03-31T15:08:05.090 INFO:teuthology.orchestra.run.smithi016.stdout: "hostname": "smithi016", 2024-03-31T15:08:05.090 INFO:teuthology.orchestra.run.smithi016.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-03-31T15:08:05.090 INFO:teuthology.orchestra.run.smithi016.stdout: "pid": "41790", 2024-03-31T15:08:05.090 INFO:teuthology.orchestra.run.smithi016.stdout: "root": "/" 2024-03-31T15:08:05.090 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-31T15:08:05.090 INFO:teuthology.orchestra.run.smithi016.stdout: "dentry_count": 0, 2024-03-31T15:08:05.091 INFO:teuthology.orchestra.run.smithi016.stdout: "dentry_pinned_count": 0, 2024-03-31T15:08:05.091 INFO:teuthology.orchestra.run.smithi016.stdout: "id": 6870, 2024-03-31T15:08:05.091 INFO:teuthology.orchestra.run.smithi016.stdout: "inst": { 2024-03-31T15:08:05.091 INFO:teuthology.orchestra.run.smithi016.stdout: "name": { 2024-03-31T15:08:05.091 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "client", 2024-03-31T15:08:05.091 INFO:teuthology.orchestra.run.smithi016.stdout: "num": 6870 2024-03-31T15:08:05.091 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-31T15:08:05.091 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": { 2024-03-31T15:08:05.091 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-03-31T15:08:05.091 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "192.168.0.1:0", 2024-03-31T15:08:05.091 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 1031456110 2024-03-31T15:08:05.091 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-31T15:08:05.091 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-31T15:08:05.091 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": { 2024-03-31T15:08:05.091 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-03-31T15:08:05.091 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "192.168.0.1:0", 2024-03-31T15:08:05.092 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 1031456110 2024-03-31T15:08:05.092 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-31T15:08:05.092 INFO:teuthology.orchestra.run.smithi016.stdout: "inst_str": "client.6870 192.168.0.1:0/1031456110", 2024-03-31T15:08:05.092 INFO:teuthology.orchestra.run.smithi016.stdout: "addr_str": "192.168.0.1:0/1031456110", 2024-03-31T15:08:05.092 INFO:teuthology.orchestra.run.smithi016.stdout: "inode_count": 1, 2024-03-31T15:08:05.092 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_epoch": 45, 2024-03-31T15:08:05.092 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_epoch": 67, 2024-03-31T15:08:05.092 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_epoch_barrier": 0, 2024-03-31T15:08:05.092 INFO:teuthology.orchestra.run.smithi016.stdout: "blocklisted": false, 2024-03-31T15:08:05.092 INFO:teuthology.orchestra.run.smithi016.stdout: "fs_name": "cephfs" 2024-03-31T15:08:05.092 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-03-31T15:08:05.105 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:08:05.105 DEBUG:teuthology.orchestra.run.smithi016:> stat --file-system '--printf=%T 2024-03-31T15:08:05.105 DEBUG:teuthology.orchestra.run.smithi016:> ' -- /home/ubuntu/cephtest/mnt.0 2024-03-31T15:08:05.114 INFO:teuthology.orchestra.run.smithi016.stdout:fuseblk 2024-03-31T15:08:05.114 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-03-31T15:08:05.114 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:08:05.114 DEBUG:teuthology.orchestra.run.smithi016:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-31T15:08:05.175 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:08:05.175 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir subdir/) 2024-03-31T15:08:05.238 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:08:05.239 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec touch subdir/file_undamaged) 2024-03-31T15:08:05.291 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:08:05.291 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec touch subdir/file_to_be_damaged) 2024-03-31T15:08:05.350 DEBUG:teuthology.orchestra.run.smithi016:> stdin-killer -- python3 -c ' 2024-03-31T15:08:05.350 DEBUG:teuthology.orchestra.run.smithi016:> import os 2024-03-31T15:08:05.350 DEBUG:teuthology.orchestra.run.smithi016:> import stat 2024-03-31T15:08:05.350 DEBUG:teuthology.orchestra.run.smithi016:> 2024-03-31T15:08:05.350 DEBUG:teuthology.orchestra.run.smithi016:> print(os.stat("/home/ubuntu/cephtest/mnt.0/subdir").st_ino) 2024-03-31T15:08:05.350 DEBUG:teuthology.orchestra.run.smithi016:> ' 2024-03-31T15:08:05.457 INFO:teuthology.orchestra.run.smithi016.stdout:1099511627776 2024-03-31T15:08:05.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:05 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-31T15:08:05.466 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-03-31T15:08:05.466 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-31T15:08:05.516 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-31T15:08:05.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:05.710+0000 7f359f200640 1 Processor -- start 2024-03-31T15:08:05.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:05.710+0000 7f359f200640 1 -- start start 2024-03-31T15:08:05.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:05.710+0000 7f359f200640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f35a015e3b0 0x7f35a009aa60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:05.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:05.710+0000 7f359f200640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f35a00cdba0 0x7f35a009afa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:05.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:05.710+0000 7f359f200640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f35a00d0800 con 0x7f35a00b1360 2024-03-31T15:08:05.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:05.710+0000 7f359f200640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f35a015e790 con 0x7f35a015e3b0 2024-03-31T15:08:05.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:05.710+0000 7f359f200640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f35a006f5d0 con 0x7f35a00cdba0 2024-03-31T15:08:05.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:05.710+0000 7f359de00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f35a015e3b0 0x7f35a009aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:05.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:05.710+0000 7f359e800640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f35a00cdba0 0x7f35a009afa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:05.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:05.710+0000 7f359d400640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f35a00b1360 0x7f35a006fd40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.16:20468/0 (socket says 172.21.15.16:51992) 2024-03-31T15:08:05.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:05.710+0000 7f359d400640 1 -- 172.21.15.16:0/1237126175 learned_addr learned my addr 172.21.15.16:0/1237126175 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:08:05.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:05.710+0000 7f359de00640 1 -- 172.21.15.16:0/1237126175 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f35a00cdba0 msgr2=0x7f35a009afa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:08:05.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:05.710+0000 7f359de00640 1 --2- 172.21.15.16:0/1237126175 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f35a00cdba0 0x7f35a009afa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:05.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:05.710+0000 7f359de00640 1 -- 172.21.15.16:0/1237126175 >> v1:172.21.15.89:6789/0 conn(0x7f35a00b1360 legacy=0x7f35a006fd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:05.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:05.710+0000 7f359de00640 1 -- 172.21.15.16:0/1237126175 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35a00d0ae0 con 0x7f35a015e3b0 2024-03-31T15:08:05.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:05.710+0000 7f359e800640 1 --2- 172.21.15.16:0/1237126175 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f35a00cdba0 0x7f35a009afa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:08:05.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:05.710+0000 7f359ca00640 1 -- 172.21.15.16:0/1237126175 <== mon.1 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2137457733 0 0) 0x7f35a00d0800 con 0x7f35a00b1360 2024-03-31T15:08:05.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:05.714+0000 7f359de00640 1 --2- 172.21.15.16:0/1237126175 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f35a015e3b0 0x7f35a009aa60 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f3590009d90 tx=0x7f359001c970 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5b6f34fad64325f1 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:08:05.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:05.714+0000 7f359ca00640 1 -- 172.21.15.16:0/1237126175 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35900b2070 con 0x7f35a015e3b0 2024-03-31T15:08:05.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:05.714+0000 7f359ca00640 1 -- 172.21.15.16:0/1237126175 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f35900a5920 con 0x7f35a015e3b0 2024-03-31T15:08:05.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:05.714+0000 7f359ca00640 1 -- 172.21.15.16:0/1237126175 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35900a5cf0 con 0x7f35a015e3b0 2024-03-31T15:08:05.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:05.714+0000 7f359f200640 1 -- 172.21.15.16:0/1237126175 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f35a015e3b0 msgr2=0x7f35a009aa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:05.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:05.714+0000 7f359f200640 1 --2- 172.21.15.16:0/1237126175 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f35a015e3b0 0x7f35a009aa60 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f3590009d90 tx=0x7f359001c970 comp rx=0 tx=0).stop 2024-03-31T15:08:05.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:05.714+0000 7f359f200640 1 -- 172.21.15.16:0/1237126175 shutdown_connections 2024-03-31T15:08:05.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:05.714+0000 7f359f200640 1 --2- 172.21.15.16:0/1237126175 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f35a00cdba0 0x7f35a009afa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:05.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:05.714+0000 7f359f200640 1 --2- 172.21.15.16:0/1237126175 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f35a015e3b0 0x7f35a009aa60 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:05.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:05.714+0000 7f359f200640 1 -- 172.21.15.16:0/1237126175 >> 172.21.15.16:0/1237126175 conn(0x7f35a00b39c0 msgr2=0x7f35a00b3dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:08:05.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:05.714+0000 7f359f200640 1 -- 172.21.15.16:0/1237126175 shutdown_connections 2024-03-31T15:08:05.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:05.714+0000 7f359f200640 1 -- 172.21.15.16:0/1237126175 wait complete. 2024-03-31T15:08:05.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:05.714+0000 7f359f200640 1 Processor -- start 2024-03-31T15:08:05.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:05.718+0000 7f359f200640 1 -- start start 2024-03-31T15:08:05.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:05.718+0000 7f359f200640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f35a00b1360 0x7f35a0153f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:05.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:05.718+0000 7f359f200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f35a00cdba0 0x7f35a015d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:05.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:05.718+0000 7f359f200640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f35a015e3b0 0x7f35a0155b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:05.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:05.718+0000 7f359f200640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f35a0070de0 con 0x7f35a00cdba0 2024-03-31T15:08:05.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:05.718+0000 7f359f200640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f35a0070c60 con 0x7f35a015e3b0 2024-03-31T15:08:05.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:05.718+0000 7f359f200640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f35a0070f60 con 0x7f35a00b1360 2024-03-31T15:08:05.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:05.718+0000 7f359d400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f35a00cdba0 0x7f35a015d4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:05.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:05.718+0000 7f359d400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f35a00cdba0 0x7f35a015d4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51912/0 (socket says 172.21.15.16:51912) 2024-03-31T15:08:05.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:05.718+0000 7f359d400640 1 -- 172.21.15.16:0/3699479164 learned_addr learned my addr 172.21.15.16:0/3699479164 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:08:05.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:05.718+0000 7f359de00640 1 --2- 172.21.15.16:0/3699479164 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f35a00b1360 0x7f35a0153f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:05.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:05.718+0000 7f359e800640 1 --2- 172.21.15.16:0/3699479164 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f35a015e3b0 0x7f35a0155b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:05.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:05.718+0000 7f359de00640 1 -- 172.21.15.16:0/3699479164 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f35a015e3b0 msgr2=0x7f35a0155b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:05.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:05.718+0000 7f359de00640 1 --2- 172.21.15.16:0/3699479164 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f35a015e3b0 0x7f35a0155b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:05.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:05.718+0000 7f359de00640 1 -- 172.21.15.16:0/3699479164 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f35a00cdba0 msgr2=0x7f35a015d4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:05.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:05.718+0000 7f359de00640 1 --2- 172.21.15.16:0/3699479164 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f35a00cdba0 0x7f35a015d4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:05.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:05.718+0000 7f359de00640 1 -- 172.21.15.16:0/3699479164 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35a015b7c0 con 0x7f35a00b1360 2024-03-31T15:08:05.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:05.718+0000 7f359d400640 1 --2- 172.21.15.16:0/3699479164 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f35a00cdba0 0x7f35a015d4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:08:05.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:05.718+0000 7f359de00640 1 --2- 172.21.15.16:0/3699479164 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f35a00b1360 0x7f35a0153f30 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f35900ac000 tx=0x7f3590019710 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:08:05.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:05.718+0000 7f3596000640 1 -- 172.21.15.16:0/3699479164 <== mon.2 v2:172.21.15.89:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35900bd030 con 0x7f35a00b1360 2024-03-31T15:08:05.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:05.718+0000 7f3596000640 1 -- 172.21.15.16:0/3699479164 <== mon.2 v2:172.21.15.89:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f35900b8470 con 0x7f35a00b1360 2024-03-31T15:08:05.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:05.718+0000 7f359f200640 1 -- 172.21.15.16:0/3699479164 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f35a015a840 con 0x7f35a00b1360 2024-03-31T15:08:05.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:05.718+0000 7f3596000640 1 -- 172.21.15.16:0/3699479164 <== mon.2 v2:172.21.15.89:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35900b2040 con 0x7f35a00b1360 2024-03-31T15:08:05.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:05.718+0000 7f359f200640 1 -- 172.21.15.16:0/3699479164 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f35a015bd60 con 0x7f35a00b1360 2024-03-31T15:08:05.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:05.722+0000 7f359f200640 1 -- 172.21.15.16:0/3699479164 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f35a004f8c0 con 0x7f35a00b1360 2024-03-31T15:08:05.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:05.722+0000 7f3596000640 1 -- 172.21.15.16:0/3699479164 <== mon.2 v2:172.21.15.89:3301/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f35900c0070 con 0x7f35a00b1360 2024-03-31T15:08:05.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:05.722+0000 7f3596000640 1 --2- 172.21.15.16:0/3699479164 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f35800683b0 0x7f358006a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:05.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:05.722+0000 7f3596000640 1 -- 172.21.15.16:0/3699479164 <== mon.2 v2:172.21.15.89:3301/0 5 ==== osd_map(67..67 src has 1..67) ==== 5709+0+0 (secure 0 0 0) 0x7f3590121230 con 0x7f35a00b1360 2024-03-31T15:08:05.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:05.722+0000 7f359d400640 1 --2- 172.21.15.16:0/3699479164 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f35800683b0 0x7f358006a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:05.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:05.722+0000 7f359d400640 1 --2- 172.21.15.16:0/3699479164 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f35800683b0 0x7f358006a870 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f35a00ceed0 tx=0x7f358c007eb0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:08:05.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:05.726+0000 7f3596000640 1 -- 172.21.15.16:0/3699479164 <== mon.2 v2:172.21.15.89:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f3590016560 con 0x7f35a00b1360 2024-03-31T15:08:05.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:05.902+0000 7f359f200640 1 -- 172.21.15.16:0/3699479164 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f35a0161b70 con 0x7f35a00b1360 2024-03-31T15:08:05.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:05.902+0000 7f3596000640 1 -- 172.21.15.16:0/3699479164 <== mon.2 v2:172.21.15.89:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v67) ==== 74+0+16596 (secure 0 0 0) 0x7f35900f17e0 con 0x7f35a00b1360 2024-03-31T15:08:05.899 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:08:05.899 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":67,"fsid":"286eb5da-fb47-4550-b290-d9519c13b9a5","created":"2024-03-31T15:03:11.438952+0000","modified":"2024-03-31T15:07:58.527959+0000","last_up_change":"2024-03-31T15:03:22.885535+0000","last_in_change":"2024-03-31T15:03:13.941667+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-31T15:03:22.550787+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-03-31T15:07:55.329096+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"66","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-03-31T15:07:55.885919+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"67","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"92a66450-ef19-40d8-ac3d-d760b23b0bda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6801","nonce":932712244}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6803","nonce":932712244}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6807","nonce":932712244}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6805","nonce":932712244}]},"public_addr":"172.21.15.16:6801/932712244","cluster_addr":"172.21.15.16:6803/932712244","heartbeat_back_addr":"172.21.15.16:6807/932712244","heartbeat_front_addr":"172.21.15.16:6805/932712244","state":["exists","up"]},{"osd":1,"uuid":"da9d1f2b-1968-4959-b843-faa231988e92","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6810","nonce":1398737533}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6811","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6813","nonce":1398737533}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6821","nonce":1398737533}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6815","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6816","nonce":1398737533}]},"public_addr":"172.21.15.16:6810/1398737533","cluster_addr":"172.21.15.16:6813/1398737533","heartbeat_back_addr":"172.21.15.16:6821/1398737533","heartbeat_front_addr":"172.21.15.16:6816/1398737533","state":["exists","up"]},{"osd":2,"uuid":"01a73662-9a21-4b1f-bcbb-42b9636a4638","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6819","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6822","nonce":3018526646}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6825","nonce":3018526646}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6831","nonce":3018526646}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6827","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6829","nonce":3018526646}]},"public_addr":"172.21.15.16:6822/3018526646","cluster_addr":"172.21.15.16:6825/3018526646","heartbeat_back_addr":"172.21.15.16:6831/3018526646","heartbeat_front_addr":"172.21.15.16:6829/3018526646","state":["exists","up"]},{"osd":3,"uuid":"c7771fd3-e18e-4500-ac38-23cf10ebcd75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6809","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6812","nonce":3360352974}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6817","nonce":3360352974}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6828","nonce":3360352974}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6823","nonce":3360352974}]},"public_addr":"172.21.15.16:6812/3360352974","cluster_addr":"172.21.15.16:6817/3360352974","heartbeat_back_addr":"172.21.15.16:6828/3360352974","heartbeat_front_addr":"172.21.15.16:6823/3360352974","state":["exists","up"]},{"osd":4,"uuid":"88260730-1482-4fbd-a40f-cb1124e9df4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6801","nonce":1856844856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6803","nonce":1856844856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6807","nonce":1856844856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6805","nonce":1856844856}]},"public_addr":"172.21.15.89:6801/1856844856","cluster_addr":"172.21.15.89:6803/1856844856","heartbeat_back_addr":"172.21.15.89:6807/1856844856","heartbeat_front_addr":"172.21.15.89:6805/1856844856","state":["exists","up"]},{"osd":5,"uuid":"4cb2c00c-fae0-4dd8-9d27-ec4f0093ccf7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6814","nonce":50395527}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6817","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6820","nonce":50395527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6831","nonce":50395527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6823","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6826","nonce":50395527}]},"public_addr":"172.21.15.89:6814/50395527","cluster_addr":"172.21.15.89:6820/50395527","heartbeat_back_addr":"172.21.15.89:6831/50395527","heartbeat_front_addr":"172.21.15.89:6826/50395527","state":["exists","up"]},{"osd":6,"uuid":"81ce0c19-2554-4cae-acbe-66a05e9e9b9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6809","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6812","nonce":3470471752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6815","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6818","nonce":3470471752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6827","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6829","nonce":3470471752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6821","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6824","nonce":3470471752}]},"public_addr":"172.21.15.89:6812/3470471752","cluster_addr":"172.21.15.89:6818/3470471752","heartbeat_back_addr":"172.21.15.89:6829/3470471752","heartbeat_front_addr":"172.21.15.89:6824/3470471752","state":["exists","up"]},{"osd":7,"uuid":"85e4313b-d37d-442e-8042-153bf20a847d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6811","nonce":1796778630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6813","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6816","nonce":1796778630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6825","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6828","nonce":1796778630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6819","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6822","nonce":1796778630}]},"public_addr":"172.21.15.89:6811/1796778630","cluster_addr":"172.21.15.89:6816/1796778630","heartbeat_back_addr":"172.21.15.89:6828/1796778630","heartbeat_front_addr":"172.21.15.89:6822/1796778630","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.439088+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.529970+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.680512+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:21.044161+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.582111+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.536558+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.409522+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.590229+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-31T15:08:05.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:05.906+0000 7f359f200640 1 -- 172.21.15.16:0/3699479164 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f35800683b0 msgr2=0x7f358006a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:05.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:05.906+0000 7f359f200640 1 --2- 172.21.15.16:0/3699479164 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f35800683b0 0x7f358006a870 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f35a00ceed0 tx=0x7f358c007eb0 comp rx=0 tx=0).stop 2024-03-31T15:08:05.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:05.906+0000 7f359f200640 1 -- 172.21.15.16:0/3699479164 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f35a00b1360 msgr2=0x7f35a0153f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:05.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:05.906+0000 7f359f200640 1 --2- 172.21.15.16:0/3699479164 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f35a00b1360 0x7f35a0153f30 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f35900ac000 tx=0x7f3590019710 comp rx=0 tx=0).stop 2024-03-31T15:08:05.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:05.906+0000 7f359f200640 1 -- 172.21.15.16:0/3699479164 shutdown_connections 2024-03-31T15:08:05.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:05.906+0000 7f359f200640 1 --2- 172.21.15.16:0/3699479164 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f35800683b0 0x7f358006a870 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:05.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:05.906+0000 7f359f200640 1 --2- 172.21.15.16:0/3699479164 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f35a015e3b0 0x7f35a0155b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:05.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:05.906+0000 7f359f200640 1 --2- 172.21.15.16:0/3699479164 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f35a00cdba0 0x7f35a015d4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:05.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:05.906+0000 7f359f200640 1 --2- 172.21.15.16:0/3699479164 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f35a00b1360 0x7f35a0153f30 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:05.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:05.906+0000 7f359f200640 1 -- 172.21.15.16:0/3699479164 >> 172.21.15.16:0/3699479164 conn(0x7f35a00b39c0 msgr2=0x7f35a00b2410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:08:05.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:05.906+0000 7f359f200640 1 -- 172.21.15.16:0/3699479164 shutdown_connections 2024-03-31T15:08:05.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:05.906+0000 7f359f200640 1 -- 172.21.15.16:0/3699479164 wait complete. 2024-03-31T15:08:05.920 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-31T15:03:22.550787+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 10, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-31T15:07:55.329096+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '66', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.5, 'score_stable': 2.5, 'optimal_score': 1, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 11, 'pool_name': 'cephfs_data', 'create_time': '2024-03-31T15:07:55.885919+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '67', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-31T15:08:05.921 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-31T15:08:06.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:06.066+0000 7f9c37000640 1 Processor -- start 2024-03-31T15:08:06.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:06.070+0000 7f9c37000640 1 -- start start 2024-03-31T15:08:06.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:06.070+0000 7f9c37000640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f9c301411b0 0x7f9c3006f920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:06.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:06.070+0000 7f9c37000640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9c30144980 con 0x7f9c30141590 2024-03-31T15:08:06.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:06.070+0000 7f9c37000640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9c30161fe0 con 0x7f9c30144340 2024-03-31T15:08:06.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:06.070+0000 7f9c37000640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f9c30162910 con 0x7f9c301411b0 2024-03-31T15:08:06.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:06.070+0000 7f9c27e00640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f9c30141590 0x7f9c30143c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:34244/0 (socket says 172.21.15.16:34244) 2024-03-31T15:08:06.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:06.070+0000 7f9c27e00640 1 -- 172.21.15.16:0/2995692104 learned_addr learned my addr 172.21.15.16:0/2995692104 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:08:06.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:06.070+0000 7f9c2fe00640 1 --2- 172.21.15.16:0/2995692104 >> v2:172.21.15.89:3300/0 conn(0x7f9c301411b0 0x7f9c3006f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:06.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:06.070+0000 7f9c2f400640 1 -- 172.21.15.16:0/2995692104 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 262758716 0 0) 0x7f9c30144980 con 0x7f9c30141590 2024-03-31T15:08:06.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:06.070+0000 7f9c2f400640 1 -- 172.21.15.16:0/2995692104 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9c14003530 con 0x7f9c30141590 2024-03-31T15:08:06.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:06.070+0000 7f9c2fe00640 1 -- 172.21.15.16:0/2995692104 >> v1:172.21.15.89:6789/0 conn(0x7f9c30144340 legacy=0x7f9c3015e900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:06.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:06.070+0000 7f9c2f400640 1 -- 172.21.15.16:0/2995692104 <== mon.1 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2011702597 0 0) 0x7f9c30161fe0 con 0x7f9c30144340 2024-03-31T15:08:06.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:06.070+0000 7f9c2fe00640 1 -- 172.21.15.16:0/2995692104 >> v1:172.21.15.16:6789/0 conn(0x7f9c30141590 legacy=0x7f9c30143c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:06.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:06.070+0000 7f9c2fe00640 1 -- 172.21.15.16:0/2995692104 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9c301622c0 con 0x7f9c301411b0 2024-03-31T15:08:06.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:06.074+0000 7f9c2fe00640 1 --2- 172.21.15.16:0/2995692104 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9c301411b0 0x7f9c3006f920 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f9c200097c0 tx=0x7f9c2001c4a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1acb0cc3a63a59cb server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:08:06.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:06.074+0000 7f9c2f400640 1 -- 172.21.15.16:0/2995692104 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c200b0070 con 0x7f9c301411b0 2024-03-31T15:08:06.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:06.074+0000 7f9c2f400640 1 -- 172.21.15.16:0/2995692104 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9c200ab600 con 0x7f9c301411b0 2024-03-31T15:08:06.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:06.074+0000 7f9c2f400640 1 -- 172.21.15.16:0/2995692104 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c200ab9d0 con 0x7f9c301411b0 2024-03-31T15:08:06.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:06.078+0000 7f9c37000640 1 -- 172.21.15.16:0/2995692104 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9c301411b0 msgr2=0x7f9c3006f920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:06.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:06.078+0000 7f9c37000640 1 --2- 172.21.15.16:0/2995692104 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9c301411b0 0x7f9c3006f920 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f9c200097c0 tx=0x7f9c2001c4a0 comp rx=0 tx=0).stop 2024-03-31T15:08:06.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:06.078+0000 7f9c37000640 1 -- 172.21.15.16:0/2995692104 shutdown_connections 2024-03-31T15:08:06.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:06.078+0000 7f9c37000640 1 --2- 172.21.15.16:0/2995692104 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9c301411b0 0x7f9c3006f920 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:06.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:06.078+0000 7f9c37000640 1 -- 172.21.15.16:0/2995692104 >> 172.21.15.16:0/2995692104 conn(0x7f9c30067930 msgr2=0x7f9c300b3ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:08:06.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:06.078+0000 7f9c37000640 1 -- 172.21.15.16:0/2995692104 shutdown_connections 2024-03-31T15:08:06.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:06.078+0000 7f9c37000640 1 -- 172.21.15.16:0/2995692104 wait complete. 2024-03-31T15:08:06.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:06.078+0000 7f9c37000640 1 Processor -- start 2024-03-31T15:08:06.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:06.078+0000 7f9c37000640 1 -- start start 2024-03-31T15:08:06.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:06.078+0000 7f9c37000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9c301411b0 0x7f9c300b1c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:06.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:06.078+0000 7f9c37000640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f9c30141590 0x7f9c3013fb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:06.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:06.078+0000 7f9c37000640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9c30144340 0x7f9c301400a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:06.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:06.078+0000 7f9c37000640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f9c3016c680 con 0x7f9c301411b0 2024-03-31T15:08:06.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:06.078+0000 7f9c37000640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f9c3016c500 con 0x7f9c30144340 2024-03-31T15:08:06.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:06.078+0000 7f9c37000640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f9c3016c800 con 0x7f9c30141590 2024-03-31T15:08:06.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:06.078+0000 7f9c2fe00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9c301411b0 0x7f9c300b1c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:06.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:06.078+0000 7f9c2fe00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9c301411b0 0x7f9c300b1c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51922/0 (socket says 172.21.15.16:51922) 2024-03-31T15:08:06.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:06.078+0000 7f9c2fe00640 1 -- 172.21.15.16:0/1898340681 learned_addr learned my addr 172.21.15.16:0/1898340681 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:08:06.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:06.078+0000 7f9c27e00640 1 --2- 172.21.15.16:0/1898340681 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f9c30141590 0x7f9c3013fb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:06.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:06.078+0000 7f9c35000640 1 --2- 172.21.15.16:0/1898340681 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9c30144340 0x7f9c301400a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:06.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:06.078+0000 7f9c2fe00640 1 -- 172.21.15.16:0/1898340681 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f9c30141590 msgr2=0x7f9c3013fb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:06.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:06.078+0000 7f9c2fe00640 1 --2- 172.21.15.16:0/1898340681 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f9c30141590 0x7f9c3013fb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:06.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:06.078+0000 7f9c2fe00640 1 -- 172.21.15.16:0/1898340681 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9c30144340 msgr2=0x7f9c301400a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:06.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:06.078+0000 7f9c2fe00640 1 --2- 172.21.15.16:0/1898340681 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9c30144340 0x7f9c301400a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:06.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:06.078+0000 7f9c2fe00640 1 -- 172.21.15.16:0/1898340681 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9c300b2190 con 0x7f9c301411b0 2024-03-31T15:08:06.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:06.082+0000 7f9c2fe00640 1 --2- 172.21.15.16:0/1898340681 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9c301411b0 0x7f9c300b1c50 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7f9c200097c0 tx=0x7f9c2001ace0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:08:06.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:06.082+0000 7f9c2cc00640 1 -- 172.21.15.16:0/1898340681 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c200bb020 con 0x7f9c301411b0 2024-03-31T15:08:06.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:06.082+0000 7f9c2cc00640 1 -- 172.21.15.16:0/1898340681 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9c200b1040 con 0x7f9c301411b0 2024-03-31T15:08:06.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:06.082+0000 7f9c2cc00640 1 -- 172.21.15.16:0/1898340681 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c200b0050 con 0x7f9c301411b0 2024-03-31T15:08:06.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:06.082+0000 7f9c37000640 1 -- 172.21.15.16:0/1898340681 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9c300b2430 con 0x7f9c301411b0 2024-03-31T15:08:06.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:06.082+0000 7f9c37000640 1 -- 172.21.15.16:0/1898340681 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9c300b29f0 con 0x7f9c301411b0 2024-03-31T15:08:06.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:06.082+0000 7f9c37000640 1 -- 172.21.15.16:0/1898340681 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9c30069950 con 0x7f9c301411b0 2024-03-31T15:08:06.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:06.082+0000 7f9c2cc00640 1 -- 172.21.15.16:0/1898340681 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f9c200be070 con 0x7f9c301411b0 2024-03-31T15:08:06.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:06.082+0000 7f9c2cc00640 1 --2- 172.21.15.16:0/1898340681 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f9c0c068380 0x7f9c0c06a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:06.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:06.082+0000 7f9c2cc00640 1 -- 172.21.15.16:0/1898340681 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(67..67 src has 1..67) ==== 5709+0+0 (secure 0 0 0) 0x7f9c2011fff0 con 0x7f9c301411b0 2024-03-31T15:08:06.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:06.086+0000 7f9c27e00640 1 --2- 172.21.15.16:0/1898340681 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f9c0c068380 0x7f9c0c06a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:06.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:06.086+0000 7f9c27e00640 1 --2- 172.21.15.16:0/1898340681 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f9c0c068380 0x7f9c0c06a840 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f9c18007fd0 tx=0x7f9c18009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:08:06.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:06.086+0000 7f9c2cc00640 1 -- 172.21.15.16:0/1898340681 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f9c20016090 con 0x7f9c301411b0 2024-03-31T15:08:06.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:06.318+0000 7f9c37000640 1 -- 172.21.15.16:0/1898340681 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f9c300b4a20 con 0x7f9c301411b0 2024-03-31T15:08:06.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:06.318+0000 7f9c2cc00640 1 -- 172.21.15.16:0/1898340681 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v67) ==== 93+0+10 (secure 0 0 0) 0x7f9c200efec0 con 0x7f9c301411b0 2024-03-31T15:08:06.312 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2024-03-31T15:08:06.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:06.318+0000 7f9c37000640 1 -- 172.21.15.16:0/1898340681 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f9c0c068380 msgr2=0x7f9c0c06a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:06.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:06.322+0000 7f9c37000640 1 --2- 172.21.15.16:0/1898340681 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f9c0c068380 0x7f9c0c06a840 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f9c18007fd0 tx=0x7f9c18009040 comp rx=0 tx=0).stop 2024-03-31T15:08:06.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:06.322+0000 7f9c37000640 1 -- 172.21.15.16:0/1898340681 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9c301411b0 msgr2=0x7f9c300b1c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:06.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:06.322+0000 7f9c37000640 1 --2- 172.21.15.16:0/1898340681 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9c301411b0 0x7f9c300b1c50 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7f9c200097c0 tx=0x7f9c2001ace0 comp rx=0 tx=0).stop 2024-03-31T15:08:06.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:06.322+0000 7f9c37000640 1 -- 172.21.15.16:0/1898340681 shutdown_connections 2024-03-31T15:08:06.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:06.322+0000 7f9c37000640 1 --2- 172.21.15.16:0/1898340681 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f9c0c068380 0x7f9c0c06a840 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:06.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:06.322+0000 7f9c37000640 1 --2- 172.21.15.16:0/1898340681 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9c30144340 0x7f9c301400a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:06.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:06.322+0000 7f9c37000640 1 --2- 172.21.15.16:0/1898340681 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f9c30141590 0x7f9c3013fb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:06.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:06.322+0000 7f9c37000640 1 --2- 172.21.15.16:0/1898340681 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9c301411b0 0x7f9c300b1c50 unknown :-1 s=CLOSED pgs=849 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:06.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:06.322+0000 7f9c37000640 1 -- 172.21.15.16:0/1898340681 >> 172.21.15.16:0/1898340681 conn(0x7f9c30067930 msgr2=0x7f9c30141cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:08:06.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:06.322+0000 7f9c37000640 1 -- 172.21.15.16:0/1898340681 shutdown_connections 2024-03-31T15:08:06.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:06.322+0000 7f9c37000640 1 -- 172.21.15.16:0/1898340681 wait complete. 2024-03-31T15:08:06.335 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-31T15:08:06.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:06.482+0000 7f0fa9000640 1 Processor -- start 2024-03-31T15:08:06.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:06.486+0000 7f0fa9000640 1 -- start start 2024-03-31T15:08:06.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:06.486+0000 7f0fa9000640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f0fa40afef0 0x7f0fa40b0350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:06.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:06.486+0000 7f0fa9000640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0fa406e570 con 0x7f0fa415c000 2024-03-31T15:08:06.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:06.486+0000 7f0fa9000640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0fa406e850 con 0x7f0fa406f610 2024-03-31T15:08:06.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:06.486+0000 7f0fa9000640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f0fa406f180 con 0x7f0fa40afef0 2024-03-31T15:08:06.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:06.486+0000 7f0fa3400640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f0fa415c000 0x7f0fa40cecf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:34254/0 (socket says 172.21.15.16:34254) 2024-03-31T15:08:06.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:06.486+0000 7f0fa3400640 1 -- 172.21.15.16:0/171696562 learned_addr learned my addr 172.21.15.16:0/171696562 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:08:06.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:06.486+0000 7f0f9be00640 1 --2- 172.21.15.16:0/171696562 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f0fa40afef0 0x7f0fa40b0350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:06.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:06.486+0000 7f0fa2a00640 1 -- 172.21.15.16:0/171696562 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1786604953 0 0) 0x7f0fa406e570 con 0x7f0fa415c000 2024-03-31T15:08:06.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:06.486+0000 7f0fa2a00640 1 -- 172.21.15.16:0/171696562 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0f88003530 con 0x7f0fa415c000 2024-03-31T15:08:06.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:06.486+0000 7f0f9be00640 1 -- 172.21.15.16:0/171696562 >> v1:172.21.15.89:6789/0 conn(0x7f0fa406f610 legacy=0x7f0fa406b620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:06.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:06.486+0000 7f0fa2a00640 1 -- 172.21.15.16:0/171696562 <== mon.1 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1398098665 0 0) 0x7f0fa406e850 con 0x7f0fa406f610 2024-03-31T15:08:06.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:06.486+0000 7f0f9be00640 1 -- 172.21.15.16:0/171696562 >> v1:172.21.15.16:6789/0 conn(0x7f0fa415c000 legacy=0x7f0fa40cecf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:06.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:06.486+0000 7f0f9be00640 1 -- 172.21.15.16:0/171696562 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0fa406eb30 con 0x7f0fa40afef0 2024-03-31T15:08:06.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:06.490+0000 7f0f9be00640 1 --2- 172.21.15.16:0/171696562 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f0fa40afef0 0x7f0fa40b0350 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f0f8c0097f0 tx=0x7f0f8c01c450 comp rx=0 tx=0).ready entity=mon.2 client_cookie=83be93881a978ffe server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:08:06.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:06.490+0000 7f0fa2a00640 1 -- 172.21.15.16:0/171696562 <== mon.2 v2:172.21.15.89:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f8c0b0070 con 0x7f0fa40afef0 2024-03-31T15:08:06.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:06.490+0000 7f0fa2a00640 1 -- 172.21.15.16:0/171696562 <== mon.2 v2:172.21.15.89:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0f8c0ab600 con 0x7f0fa40afef0 2024-03-31T15:08:06.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:06.490+0000 7f0fa2a00640 1 -- 172.21.15.16:0/171696562 <== mon.2 v2:172.21.15.89:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f8c0ab9d0 con 0x7f0fa40afef0 2024-03-31T15:08:06.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:06.490+0000 7f0fa9000640 1 -- 172.21.15.16:0/171696562 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f0fa40afef0 msgr2=0x7f0fa40b0350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:06.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:06.490+0000 7f0fa9000640 1 --2- 172.21.15.16:0/171696562 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f0fa40afef0 0x7f0fa40b0350 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f0f8c0097f0 tx=0x7f0f8c01c450 comp rx=0 tx=0).stop 2024-03-31T15:08:06.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:06.490+0000 7f0fa9000640 1 -- 172.21.15.16:0/171696562 shutdown_connections 2024-03-31T15:08:06.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:06.490+0000 7f0fa9000640 1 --2- 172.21.15.16:0/171696562 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f0fa40afef0 0x7f0fa40b0350 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:06.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:06.490+0000 7f0fa9000640 1 -- 172.21.15.16:0/171696562 >> 172.21.15.16:0/171696562 conn(0x7f0fa40d0df0 msgr2=0x7f0fa40d1200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:08:06.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:06.490+0000 7f0fa9000640 1 -- 172.21.15.16:0/171696562 shutdown_connections 2024-03-31T15:08:06.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:06.490+0000 7f0fa9000640 1 -- 172.21.15.16:0/171696562 wait complete. 2024-03-31T15:08:06.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:06.490+0000 7f0fa9000640 1 Processor -- start 2024-03-31T15:08:06.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:06.490+0000 7f0fa9000640 1 -- start start 2024-03-31T15:08:06.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:06.494+0000 7f0fa9000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0fa406f610 0x7f0fa406c890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:06.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:06.494+0000 7f0fa9000640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0fa40afef0 0x7f0fa40c7590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:06.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:06.494+0000 7f0fa9000640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f0fa415c000 0x7f0fa41550b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:06.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:06.494+0000 7f0fa9000640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f0fa40cf400 con 0x7f0fa406f610 2024-03-31T15:08:06.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:06.494+0000 7f0fa9000640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f0fa406bd30 con 0x7f0fa40afef0 2024-03-31T15:08:06.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:06.494+0000 7f0fa9000640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f0fa40c5a90 con 0x7f0fa415c000 2024-03-31T15:08:06.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:06.494+0000 7f0fa3400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0fa406f610 0x7f0fa406c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:06.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:06.494+0000 7f0fa3400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0fa406f610 0x7f0fa406c890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51924/0 (socket says 172.21.15.16:51924) 2024-03-31T15:08:06.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:06.494+0000 7f0fa3400640 1 -- 172.21.15.16:0/2731090019 learned_addr learned my addr 172.21.15.16:0/2731090019 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:08:06.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:06.494+0000 7f0fa3e00640 1 --2- 172.21.15.16:0/2731090019 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f0fa415c000 0x7f0fa41550b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:06.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:06.494+0000 7f0f9be00640 1 --2- 172.21.15.16:0/2731090019 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0fa40afef0 0x7f0fa40c7590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:06.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:06.494+0000 7f0fa3400640 1 -- 172.21.15.16:0/2731090019 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f0fa415c000 msgr2=0x7f0fa41550b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:06.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:06.494+0000 7f0fa3400640 1 --2- 172.21.15.16:0/2731090019 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f0fa415c000 0x7f0fa41550b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:06.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:06.494+0000 7f0fa3400640 1 -- 172.21.15.16:0/2731090019 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0fa40afef0 msgr2=0x7f0fa40c7590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:06.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:06.494+0000 7f0fa3400640 1 --2- 172.21.15.16:0/2731090019 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0fa40afef0 0x7f0fa40c7590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:06.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:06.494+0000 7f0fa3400640 1 -- 172.21.15.16:0/2731090019 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0fa40c5d30 con 0x7f0fa406f610 2024-03-31T15:08:06.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:06.494+0000 7f0fa3400640 1 --2- 172.21.15.16:0/2731090019 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0fa406f610 0x7f0fa406c890 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7f0f9400c3c0 tx=0x7f0f9400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:08:06.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:06.494+0000 7f0f9b400640 1 -- 172.21.15.16:0/2731090019 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f94017070 con 0x7f0fa406f610 2024-03-31T15:08:06.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:06.494+0000 7f0f9b400640 1 -- 172.21.15.16:0/2731090019 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0f94005cf0 con 0x7f0fa406f610 2024-03-31T15:08:06.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:06.494+0000 7f0fa9000640 1 -- 172.21.15.16:0/2731090019 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0fa40c6010 con 0x7f0fa406f610 2024-03-31T15:08:06.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:06.494+0000 7f0f9b400640 1 -- 172.21.15.16:0/2731090019 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f940072f0 con 0x7f0fa406f610 2024-03-31T15:08:06.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:06.494+0000 7f0fa9000640 1 -- 172.21.15.16:0/2731090019 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0fa40c65d0 con 0x7f0fa406f610 2024-03-31T15:08:06.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:06.494+0000 7f0fa9000640 1 -- 172.21.15.16:0/2731090019 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0fa404f8c0 con 0x7f0fa406f610 2024-03-31T15:08:06.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:06.498+0000 7f0f9b400640 1 -- 172.21.15.16:0/2731090019 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f0f9401a070 con 0x7f0fa406f610 2024-03-31T15:08:06.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:06.498+0000 7f0f9b400640 1 --2- 172.21.15.16:0/2731090019 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f0f80068380 0x7f0f8006a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:06.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:06.498+0000 7f0f9b400640 1 -- 172.21.15.16:0/2731090019 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(67..67 src has 1..67) ==== 5709+0+0 (secure 0 0 0) 0x7f0f94085d60 con 0x7f0fa406f610 2024-03-31T15:08:06.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:06.498+0000 7f0f9be00640 1 --2- 172.21.15.16:0/2731090019 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f0f80068380 0x7f0f8006a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:06.494 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:06.498+0000 7f0f9be00640 1 --2- 172.21.15.16:0/2731090019 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f0f80068380 0x7f0f8006a840 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f0f8c0ad000 tx=0x7f0f8c0a6000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:08:06.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:06.502+0000 7f0f9b400640 1 -- 172.21.15.16:0/2731090019 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f0f94055c30 con 0x7f0fa406f610 2024-03-31T15:08:06.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:06.670+0000 7f0fa9000640 1 -- 172.21.15.16:0/2731090019 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f0fa40d0780 con 0x7f0fa406f610 2024-03-31T15:08:06.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:06.670+0000 7f0f9b400640 1 -- 172.21.15.16:0/2731090019 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v67) ==== 104+0+11 (secure 0 0 0) 0x7f0f94059f30 con 0x7f0fa406f610 2024-03-31T15:08:06.665 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2024-03-31T15:08:06.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:06.670+0000 7f0fa9000640 1 -- 172.21.15.16:0/2731090019 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f0f80068380 msgr2=0x7f0f8006a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:06.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:06.670+0000 7f0fa9000640 1 --2- 172.21.15.16:0/2731090019 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f0f80068380 0x7f0f8006a840 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f0f8c0ad000 tx=0x7f0f8c0a6000 comp rx=0 tx=0).stop 2024-03-31T15:08:06.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:06.670+0000 7f0fa9000640 1 -- 172.21.15.16:0/2731090019 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0fa406f610 msgr2=0x7f0fa406c890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:06.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:06.670+0000 7f0fa9000640 1 --2- 172.21.15.16:0/2731090019 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0fa406f610 0x7f0fa406c890 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7f0f9400c3c0 tx=0x7f0f9400c880 comp rx=0 tx=0).stop 2024-03-31T15:08:06.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:06.674+0000 7f0fa9000640 1 -- 172.21.15.16:0/2731090019 shutdown_connections 2024-03-31T15:08:06.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:06.674+0000 7f0fa9000640 1 --2- 172.21.15.16:0/2731090019 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f0f80068380 0x7f0f8006a840 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:06.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:06.674+0000 7f0fa9000640 1 --2- 172.21.15.16:0/2731090019 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f0fa415c000 0x7f0fa41550b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:06.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:06.674+0000 7f0fa9000640 1 --2- 172.21.15.16:0/2731090019 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0fa40afef0 0x7f0fa40c7590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:06.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:06.674+0000 7f0fa9000640 1 --2- 172.21.15.16:0/2731090019 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0fa406f610 0x7f0fa406c890 unknown :-1 s=CLOSED pgs=851 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:06.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:06.674+0000 7f0fa9000640 1 -- 172.21.15.16:0/2731090019 >> 172.21.15.16:0/2731090019 conn(0x7f0fa40d0df0 msgr2=0x7f0fa40b21f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:08:06.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:06.674+0000 7f0fa9000640 1 -- 172.21.15.16:0/2731090019 shutdown_connections 2024-03-31T15:08:06.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:06.674+0000 7f0fa9000640 1 -- 172.21.15.16:0/2731090019 wait complete. 2024-03-31T15:08:06.687 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-31T15:08:06.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:06.834+0000 7f167ae00640 1 Processor -- start 2024-03-31T15:08:06.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:06.838+0000 7f167ae00640 1 -- start start 2024-03-31T15:08:06.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:06.838+0000 7f167ae00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f167415c390 0x7f16740cfb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:06.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:06.838+0000 7f167ae00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f16740d2b40 0x7f16740d00a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:06.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:06.838+0000 7f167ae00640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f16740cd750 con 0x7f16740d05e0 2024-03-31T15:08:06.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:06.838+0000 7f167ae00640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f167415c770 con 0x7f167415c390 2024-03-31T15:08:06.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:06.838+0000 7f167ae00640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f16740cda30 con 0x7f16740d2b40 2024-03-31T15:08:06.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:06.838+0000 7f1679a00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f167415c390 0x7f16740cfb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:06.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:06.838+0000 7f167a400640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f16740d05e0 0x7f16740ce860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:34256/0 (socket says 172.21.15.16:34256) 2024-03-31T15:08:06.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:06.838+0000 7f1679a00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f167415c390 0x7f16740cfb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51934/0 (socket says 172.21.15.16:51934) 2024-03-31T15:08:06.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:06.838+0000 7f167a400640 1 -- 172.21.15.16:0/689792791 learned_addr learned my addr 172.21.15.16:0/689792791 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:08:06.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:06.838+0000 7f1679000640 1 --2- 172.21.15.16:0/689792791 >> v2:172.21.15.89:3300/0 conn(0x7f16740d2b40 0x7f16740d00a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:06.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:06.838+0000 7f1679a00640 1 -- 172.21.15.16:0/689792791 >> v2:172.21.15.89:3300/0 conn(0x7f16740d2b40 msgr2=0x7f16740d00a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:08:06.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:06.838+0000 7f1679a00640 1 --2- 172.21.15.16:0/689792791 >> v2:172.21.15.89:3300/0 conn(0x7f16740d2b40 0x7f16740d00a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:06.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:06.838+0000 7f1679a00640 1 -- 172.21.15.16:0/689792791 >> v1:172.21.15.16:6789/0 conn(0x7f16740d05e0 legacy=0x7f16740ce860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:06.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:06.838+0000 7f1679a00640 1 -- 172.21.15.16:0/689792791 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16740cdd10 con 0x7f167415c390 2024-03-31T15:08:06.836 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:06.842+0000 7f1679a00640 1 --2- 172.21.15.16:0/689792791 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f167415c390 0x7f16740cfb60 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7f16740d1b90 tx=0x7f166801c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d3e39d603957493b server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:08:06.836 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:06.842+0000 7f1673e00640 1 -- 172.21.15.16:0/689792791 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16680b0070 con 0x7f167415c390 2024-03-31T15:08:06.836 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:06.842+0000 7f1673e00640 1 -- 172.21.15.16:0/689792791 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f16680ab600 con 0x7f167415c390 2024-03-31T15:08:06.836 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:06.842+0000 7f1673e00640 1 -- 172.21.15.16:0/689792791 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16680ab9d0 con 0x7f167415c390 2024-03-31T15:08:06.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:06.842+0000 7f167ae00640 1 -- 172.21.15.16:0/689792791 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f167415c390 msgr2=0x7f16740cfb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:06.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:06.842+0000 7f167ae00640 1 --2- 172.21.15.16:0/689792791 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f167415c390 0x7f16740cfb60 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7f16740d1b90 tx=0x7f166801c480 comp rx=0 tx=0).stop 2024-03-31T15:08:06.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:06.842+0000 7f167ae00640 1 -- 172.21.15.16:0/689792791 shutdown_connections 2024-03-31T15:08:06.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:06.842+0000 7f167ae00640 1 --2- 172.21.15.16:0/689792791 >> v2:172.21.15.89:3300/0 conn(0x7f16740d2b40 0x7f16740d00a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:06.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:06.842+0000 7f167ae00640 1 --2- 172.21.15.16:0/689792791 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f167415c390 0x7f16740cfb60 unknown :-1 s=CLOSED pgs=853 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:06.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:06.842+0000 7f167ae00640 1 -- 172.21.15.16:0/689792791 >> 172.21.15.16:0/689792791 conn(0x7f16740d1f90 msgr2=0x7f16740d23a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:08:06.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:06.842+0000 7f167ae00640 1 -- 172.21.15.16:0/689792791 shutdown_connections 2024-03-31T15:08:06.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:06.842+0000 7f167ae00640 1 -- 172.21.15.16:0/689792791 wait complete. 2024-03-31T15:08:06.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:06.842+0000 7f167ae00640 1 Processor -- start 2024-03-31T15:08:06.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:06.842+0000 7f167ae00640 1 -- start start 2024-03-31T15:08:06.839 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:06.846+0000 7f167ae00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f16740d05e0 0x7f16741470b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:06.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:06.846+0000 7f167ae00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f16740d2b40 0x7f16740af7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:06.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:06.846+0000 7f167ae00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f167415c390 0x7f16740afd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:06.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:06.846+0000 7f167ae00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f16741691e0 con 0x7f167415c390 2024-03-31T15:08:06.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:06.846+0000 7f167ae00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f1674169060 con 0x7f16740d2b40 2024-03-31T15:08:06.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:06.846+0000 7f167ae00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f1674169360 con 0x7f16740d05e0 2024-03-31T15:08:06.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:06.846+0000 7f167a400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f167415c390 0x7f16740afd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:06.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:06.846+0000 7f1679a00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f16740d05e0 0x7f16741470b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:06.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:06.846+0000 7f1679a00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f16740d05e0 0x7f16741470b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3301/0 says I am v2:172.21.15.16:59866/0 (socket says 172.21.15.16:59866) 2024-03-31T15:08:06.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:06.846+0000 7f1679a00640 1 -- 172.21.15.16:0/1101426557 learned_addr learned my addr 172.21.15.16:0/1101426557 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:08:06.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:06.846+0000 7f1679000640 1 --2- 172.21.15.16:0/1101426557 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f16740d2b40 0x7f16740af7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:06.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:06.846+0000 7f1679a00640 1 -- 172.21.15.16:0/1101426557 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f16740d2b40 msgr2=0x7f16740af7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:06.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:06.846+0000 7f1679a00640 1 --2- 172.21.15.16:0/1101426557 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f16740d2b40 0x7f16740af7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:06.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:06.846+0000 7f1679a00640 1 -- 172.21.15.16:0/1101426557 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f167415c390 msgr2=0x7f16740afd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:06.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:06.846+0000 7f1679a00640 1 --2- 172.21.15.16:0/1101426557 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f167415c390 0x7f16740afd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:06.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:06.846+0000 7f1679a00640 1 -- 172.21.15.16:0/1101426557 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f167415a0c0 con 0x7f16740d05e0 2024-03-31T15:08:06.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:06.846+0000 7f1679a00640 1 --2- 172.21.15.16:0/1101426557 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f16740d05e0 0x7f16741470b0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f16680ae040 tx=0x7f1668019220 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:08:06.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:06.846+0000 7f1671600640 1 -- 172.21.15.16:0/1101426557 <== mon.2 v2:172.21.15.89:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16680b0040 con 0x7f16740d05e0 2024-03-31T15:08:06.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:06.846+0000 7f1671600640 1 -- 172.21.15.16:0/1101426557 <== mon.2 v2:172.21.15.89:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f16680ba070 con 0x7f16740d05e0 2024-03-31T15:08:06.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:06.846+0000 7f1671600640 1 -- 172.21.15.16:0/1101426557 <== mon.2 v2:172.21.15.89:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16680b4810 con 0x7f16740d05e0 2024-03-31T15:08:06.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:06.846+0000 7f167ae00640 1 -- 172.21.15.16:0/1101426557 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f167415a3a0 con 0x7f16740d05e0 2024-03-31T15:08:06.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:06.846+0000 7f167ae00640 1 -- 172.21.15.16:0/1101426557 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f167415a960 con 0x7f16740d05e0 2024-03-31T15:08:06.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:06.846+0000 7f167ae00640 1 -- 172.21.15.16:0/1101426557 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f167404f8c0 con 0x7f16740d05e0 2024-03-31T15:08:06.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:06.850+0000 7f1671600640 1 -- 172.21.15.16:0/1101426557 <== mon.2 v2:172.21.15.89:3301/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f16680b49b0 con 0x7f16740d05e0 2024-03-31T15:08:06.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:06.850+0000 7f1671600640 1 --2- 172.21.15.16:0/1101426557 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f1654068380 0x7f165406a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:06.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:06.850+0000 7f1671600640 1 -- 172.21.15.16:0/1101426557 <== mon.2 v2:172.21.15.89:3301/0 5 ==== osd_map(67..67 src has 1..67) ==== 5709+0+0 (secure 0 0 0) 0x7f16681208a0 con 0x7f16740d05e0 2024-03-31T15:08:06.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:06.850+0000 7f1679000640 1 --2- 172.21.15.16:0/1101426557 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f1654068380 0x7f165406a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:06.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:06.850+0000 7f1679000640 1 --2- 172.21.15.16:0/1101426557 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f1654068380 0x7f165406a840 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f16741543a0 tx=0x7f1660008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:08:06.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:06.854+0000 7f1671600640 1 -- 172.21.15.16:0/1101426557 <== mon.2 v2:172.21.15.89:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f1668016070 con 0x7f16740d05e0 2024-03-31T15:08:07.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:07.018+0000 7f167ae00640 1 -- 172.21.15.16:0/1101426557 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f1674071070 con 0x7f16740d05e0 2024-03-31T15:08:07.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:07.018+0000 7f1671600640 1 -- 172.21.15.16:0/1101426557 <== mon.2 v2:172.21.15.89:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v67) ==== 100+0+11 (secure 0 0 0) 0x7f16680f0770 con 0x7f16740d05e0 2024-03-31T15:08:07.013 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 64 2024-03-31T15:08:07.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:07.022+0000 7f167ae00640 1 -- 172.21.15.16:0/1101426557 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f1654068380 msgr2=0x7f165406a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:07.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:07.022+0000 7f167ae00640 1 --2- 172.21.15.16:0/1101426557 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f1654068380 0x7f165406a840 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f16741543a0 tx=0x7f1660008040 comp rx=0 tx=0).stop 2024-03-31T15:08:07.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:07.022+0000 7f167ae00640 1 -- 172.21.15.16:0/1101426557 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f16740d05e0 msgr2=0x7f16741470b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:07.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:07.022+0000 7f167ae00640 1 --2- 172.21.15.16:0/1101426557 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f16740d05e0 0x7f16741470b0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f16680ae040 tx=0x7f1668019220 comp rx=0 tx=0).stop 2024-03-31T15:08:07.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:07.022+0000 7f167ae00640 1 -- 172.21.15.16:0/1101426557 shutdown_connections 2024-03-31T15:08:07.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:07.022+0000 7f167ae00640 1 --2- 172.21.15.16:0/1101426557 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f1654068380 0x7f165406a840 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:07.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:07.022+0000 7f167ae00640 1 --2- 172.21.15.16:0/1101426557 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f167415c390 0x7f16740afd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:07.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:07.022+0000 7f167ae00640 1 --2- 172.21.15.16:0/1101426557 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f16740d2b40 0x7f16740af7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:07.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:07.022+0000 7f167ae00640 1 --2- 172.21.15.16:0/1101426557 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f16740d05e0 0x7f16741470b0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:07.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:07.022+0000 7f167ae00640 1 -- 172.21.15.16:0/1101426557 >> 172.21.15.16:0/1101426557 conn(0x7f16740d1f90 msgr2=0x7f16740ce860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:08:07.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:07.022+0000 7f167ae00640 1 -- 172.21.15.16:0/1101426557 shutdown_connections 2024-03-31T15:08:07.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:07.022+0000 7f167ae00640 1 -- 172.21.15.16:0/1101426557 wait complete. 2024-03-31T15:08:07.045 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-31T15:08:07.083 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-31T15:08:07.093 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-31T15:08:07.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:07.246+0000 7f205c400640 1 Processor -- start 2024-03-31T15:08:07.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:07.246+0000 7f205c400640 1 -- start start 2024-03-31T15:08:07.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:07.250+0000 7f205c400640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f2054145e40 0x7f20540cc740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:07.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:07.250+0000 7f205c400640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f20540ccc80 0x7f205406f920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:07.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:07.250+0000 7f205c400640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f20540d0920 con 0x7f205406fe60 2024-03-31T15:08:07.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:07.250+0000 7f205c400640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f20540d06d0 con 0x7f20540ccc80 2024-03-31T15:08:07.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:07.250+0000 7f205c400640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f20540d0c00 con 0x7f2054145e40 2024-03-31T15:08:07.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:07.250+0000 7f2059000640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f20540ccc80 0x7f205406f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:07.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:07.250+0000 7f2059000640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f20540ccc80 0x7f205406f920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51684/0 (socket says 172.21.15.16:51684) 2024-03-31T15:08:07.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:07.250+0000 7f205a400640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f205406fe60 0x7f205415fdf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:55494/0 (socket says 172.21.15.16:55494) 2024-03-31T15:08:07.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:07.250+0000 7f2059000640 1 -- 172.21.15.16:0/1008454227 learned_addr learned my addr 172.21.15.16:0/1008454227 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:08:07.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:07.250+0000 7f2059a00640 1 --2- 172.21.15.16:0/1008454227 >> v2:172.21.15.89:3300/0 conn(0x7f2054145e40 0x7f20540cc740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:07.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:07.250+0000 7f2059000640 1 -- 172.21.15.16:0/1008454227 >> v2:172.21.15.89:3300/0 conn(0x7f2054145e40 msgr2=0x7f20540cc740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:08:07.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:07.250+0000 7f2059000640 1 --2- 172.21.15.16:0/1008454227 >> v2:172.21.15.89:3300/0 conn(0x7f2054145e40 0x7f20540cc740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:07.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:07.250+0000 7f2059000640 1 -- 172.21.15.16:0/1008454227 >> v1:172.21.15.16:6789/0 conn(0x7f205406fe60 legacy=0x7f205415fdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:07.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:07.250+0000 7f2059000640 1 -- 172.21.15.16:0/1008454227 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20540d1810 con 0x7f20540ccc80 2024-03-31T15:08:07.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:07.254+0000 7f2059000640 1 --2- 172.21.15.16:0/1008454227 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f20540ccc80 0x7f205406f920 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7f2054072e60 tx=0x7f204001c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=903115044bf3c040 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:08:07.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:07.254+0000 7f2053e00640 1 -- 172.21.15.16:0/1008454227 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20400b0070 con 0x7f20540ccc80 2024-03-31T15:08:07.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:07.254+0000 7f2053e00640 1 -- 172.21.15.16:0/1008454227 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f20400ab600 con 0x7f20540ccc80 2024-03-31T15:08:07.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:07.254+0000 7f2053e00640 1 -- 172.21.15.16:0/1008454227 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20400ab9d0 con 0x7f20540ccc80 2024-03-31T15:08:07.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:07.254+0000 7f205c400640 1 -- 172.21.15.16:0/1008454227 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f20540ccc80 msgr2=0x7f205406f920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:07.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:07.254+0000 7f205c400640 1 --2- 172.21.15.16:0/1008454227 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f20540ccc80 0x7f205406f920 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7f2054072e60 tx=0x7f204001c520 comp rx=0 tx=0).stop 2024-03-31T15:08:07.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:07.254+0000 7f205c400640 1 -- 172.21.15.16:0/1008454227 shutdown_connections 2024-03-31T15:08:07.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:07.254+0000 7f205c400640 1 --2- 172.21.15.16:0/1008454227 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f20540ccc80 0x7f205406f920 unknown :-1 s=CLOSED pgs=855 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:07.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:07.254+0000 7f205c400640 1 --2- 172.21.15.16:0/1008454227 >> v2:172.21.15.89:3300/0 conn(0x7f2054145e40 0x7f20540cc740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:07.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:07.254+0000 7f205c400640 1 -- 172.21.15.16:0/1008454227 >> 172.21.15.16:0/1008454227 conn(0x7f2054067330 msgr2=0x7f2054067740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:08:07.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:07.254+0000 7f205c400640 1 -- 172.21.15.16:0/1008454227 shutdown_connections 2024-03-31T15:08:07.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:07.254+0000 7f205c400640 1 -- 172.21.15.16:0/1008454227 wait complete. 2024-03-31T15:08:07.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:07.254+0000 7f205c400640 1 Processor -- start 2024-03-31T15:08:07.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:07.254+0000 7f205c400640 1 -- start start 2024-03-31T15:08:07.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:07.254+0000 7f205c400640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f205406fe60 0x7f2054144fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:07.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:07.254+0000 7f205c400640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f20540ccc80 0x7f20540cb440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:07.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:07.254+0000 7f205c400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2054145e40 0x7f205415d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:07.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:07.254+0000 7f205c400640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f20540b5240 con 0x7f2054145e40 2024-03-31T15:08:07.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:07.254+0000 7f205c400640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f20540b50c0 con 0x7f205406fe60 2024-03-31T15:08:07.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:07.254+0000 7f205c400640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f2054169190 con 0x7f20540ccc80 2024-03-31T15:08:07.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:07.254+0000 7f205a400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2054145e40 0x7f205415d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:07.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:07.254+0000 7f205a400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2054145e40 0x7f205415d330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51686/0 (socket says 172.21.15.16:51686) 2024-03-31T15:08:07.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:07.254+0000 7f205a400640 1 -- 172.21.15.16:0/1240485862 learned_addr learned my addr 172.21.15.16:0/1240485862 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:08:07.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:07.258+0000 7f2059a00640 1 --2- 172.21.15.16:0/1240485862 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f205406fe60 0x7f2054144fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:07.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:07.258+0000 7f2059000640 1 --2- 172.21.15.16:0/1240485862 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f20540ccc80 0x7f20540cb440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:07.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:07.258+0000 7f205a400640 1 -- 172.21.15.16:0/1240485862 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f20540ccc80 msgr2=0x7f20540cb440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:07.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:07.258+0000 7f205a400640 1 --2- 172.21.15.16:0/1240485862 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f20540ccc80 0x7f20540cb440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:07.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:07.258+0000 7f205a400640 1 -- 172.21.15.16:0/1240485862 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f205406fe60 msgr2=0x7f2054144fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:07.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:07.258+0000 7f205a400640 1 --2- 172.21.15.16:0/1240485862 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f205406fe60 0x7f2054144fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:07.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:07.258+0000 7f205a400640 1 -- 172.21.15.16:0/1240485862 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20540cbc90 con 0x7f2054145e40 2024-03-31T15:08:07.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:07.258+0000 7f2059000640 1 --2- 172.21.15.16:0/1240485862 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f20540ccc80 0x7f20540cb440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:08:07.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:07.258+0000 7f205a400640 1 --2- 172.21.15.16:0/1240485862 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2054145e40 0x7f205415d330 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7f204c00c7a0 tx=0x7f204c00cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:08:07.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:07.258+0000 7f2051600640 1 -- 172.21.15.16:0/1240485862 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f204c017070 con 0x7f2054145e40 2024-03-31T15:08:07.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:07.258+0000 7f2051600640 1 -- 172.21.15.16:0/1240485862 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f204c00a640 con 0x7f2054145e40 2024-03-31T15:08:07.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:07.258+0000 7f2051600640 1 -- 172.21.15.16:0/1240485862 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f204c00a9f0 con 0x7f2054145e40 2024-03-31T15:08:07.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:07.258+0000 7f205c400640 1 -- 172.21.15.16:0/1240485862 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f20540cbf70 con 0x7f2054145e40 2024-03-31T15:08:07.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:07.258+0000 7f205c400640 1 -- 172.21.15.16:0/1240485862 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2054156d60 con 0x7f2054145e40 2024-03-31T15:08:07.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:07.258+0000 7f205c400640 1 -- 172.21.15.16:0/1240485862 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2054069950 con 0x7f2054145e40 2024-03-31T15:08:07.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:07.258+0000 7f2051600640 1 -- 172.21.15.16:0/1240485862 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f204c01a070 con 0x7f2054145e40 2024-03-31T15:08:07.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:07.258+0000 7f2051600640 1 --2- 172.21.15.16:0/1240485862 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f20340683b0 0x7f203406a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:07.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:07.262+0000 7f2051600640 1 -- 172.21.15.16:0/1240485862 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(67..67 src has 1..67) ==== 5709+0+0 (secure 0 0 0) 0x7f204c085f40 con 0x7f2054145e40 2024-03-31T15:08:07.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:07.262+0000 7f2059a00640 1 --2- 172.21.15.16:0/1240485862 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f20340683b0 0x7f203406a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:07.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:07.262+0000 7f2059a00640 1 --2- 172.21.15.16:0/1240485862 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f20340683b0 0x7f203406a870 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f2048009ac0 tx=0x7f2048009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:08:07.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:07.266+0000 7f2051600640 1 -- 172.21.15.16:0/1240485862 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f204c055eb0 con 0x7f2054145e40 2024-03-31T15:08:07.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:07.442+0000 7f205c400640 1 -- 172.21.15.16:0/1240485862 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f2054143090 con 0x7f2054145e40 2024-03-31T15:08:07.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:07.442+0000 7f2051600640 1 -- 172.21.15.16:0/1240485862 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v67) ==== 80+0+0 (secure 0 0 0) 0x7f204c05a1b0 con 0x7f2054145e40 2024-03-31T15:08:07.437 INFO:teuthology.orchestra.run.smithi016.stderr:listed 0 entries 2024-03-31T15:08:07.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:07.442+0000 7f205c400640 1 -- 172.21.15.16:0/1240485862 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f20340683b0 msgr2=0x7f203406a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:07.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:07.442+0000 7f205c400640 1 --2- 172.21.15.16:0/1240485862 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f20340683b0 0x7f203406a870 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f2048009ac0 tx=0x7f2048009190 comp rx=0 tx=0).stop 2024-03-31T15:08:07.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:07.442+0000 7f205c400640 1 -- 172.21.15.16:0/1240485862 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2054145e40 msgr2=0x7f205415d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:07.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:07.442+0000 7f205c400640 1 --2- 172.21.15.16:0/1240485862 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2054145e40 0x7f205415d330 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7f204c00c7a0 tx=0x7f204c00cc60 comp rx=0 tx=0).stop 2024-03-31T15:08:07.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:07.442+0000 7f205c400640 1 -- 172.21.15.16:0/1240485862 shutdown_connections 2024-03-31T15:08:07.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:07.442+0000 7f205c400640 1 --2- 172.21.15.16:0/1240485862 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f20340683b0 0x7f203406a870 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:07.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:07.442+0000 7f205c400640 1 --2- 172.21.15.16:0/1240485862 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2054145e40 0x7f205415d330 unknown :-1 s=CLOSED pgs=856 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:07.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:07.442+0000 7f205c400640 1 --2- 172.21.15.16:0/1240485862 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f20540ccc80 0x7f20540cb440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:07.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:07.442+0000 7f205c400640 1 --2- 172.21.15.16:0/1240485862 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f205406fe60 0x7f2054144fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:07.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:07.442+0000 7f205c400640 1 -- 172.21.15.16:0/1240485862 >> 172.21.15.16:0/1240485862 conn(0x7f2054067330 msgr2=0x7f20540b2b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:08:07.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:07.442+0000 7f205c400640 1 -- 172.21.15.16:0/1240485862 shutdown_connections 2024-03-31T15:08:07.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:07.442+0000 7f205c400640 1 -- 172.21.15.16:0/1240485862 wait complete. 2024-03-31T15:08:07.457 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-03-31T15:08:07.457 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-31T15:08:07.466 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-31T15:08:07.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:07.674+0000 7f015ac00640 1 Processor -- start 2024-03-31T15:08:07.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:07.674+0000 7f015ac00640 1 -- start start 2024-03-31T15:08:07.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:07.678+0000 7f015ac00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f0154145e40 0x7f01540cc740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:07.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:07.678+0000 7f015ac00640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f01540d0920 con 0x7f01540ccc80 2024-03-31T15:08:07.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:07.678+0000 7f015ac00640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f01540d0c00 con 0x7f0154070030 2024-03-31T15:08:07.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:07.678+0000 7f015ac00640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f01540d06d0 con 0x7f0154145e40 2024-03-31T15:08:07.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:07.678+0000 7f0153e00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f0154145e40 0x7f01540cc740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:07.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:07.678+0000 7f014b400640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f01540ccc80 0x7f015406f920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:55496/0 (socket says 172.21.15.16:55496) 2024-03-31T15:08:07.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:07.678+0000 7f014b400640 1 -- 172.21.15.16:0/2001802127 learned_addr learned my addr 172.21.15.16:0/2001802127 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:08:07.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:07.678+0000 7f0153e00640 1 -- 172.21.15.16:0/2001802127 >> v1:172.21.15.89:6789/0 conn(0x7f0154070030 legacy=0x7f01540d2010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:07.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:07.678+0000 7f0153e00640 1 -- 172.21.15.16:0/2001802127 >> v1:172.21.15.16:6789/0 conn(0x7f01540ccc80 legacy=0x7f015406f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:07.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:07.678+0000 7f0153400640 1 -- 172.21.15.16:0/2001802127 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3973996006 0 0) 0x7f01540d0920 con 0x7f01540ccc80 2024-03-31T15:08:07.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:07.678+0000 7f0153e00640 1 -- 172.21.15.16:0/2001802127 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01540d1810 con 0x7f0154145e40 2024-03-31T15:08:07.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:07.682+0000 7f0153e00640 1 --2- 172.21.15.16:0/2001802127 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0154145e40 0x7f01540cc740 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7f0144009d90 tx=0x7f014401c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=741aca0dc5255163 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:08:07.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:07.682+0000 7f0153400640 1 -- 172.21.15.16:0/2001802127 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01440b2070 con 0x7f0154145e40 2024-03-31T15:08:07.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:07.682+0000 7f0153400640 1 -- 172.21.15.16:0/2001802127 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f01440a5940 con 0x7f0154145e40 2024-03-31T15:08:07.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:07.682+0000 7f0153400640 1 -- 172.21.15.16:0/2001802127 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01440a5d10 con 0x7f0154145e40 2024-03-31T15:08:07.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:07.682+0000 7f015ac00640 1 -- 172.21.15.16:0/2001802127 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0154145e40 msgr2=0x7f01540cc740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:07.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:07.682+0000 7f015ac00640 1 --2- 172.21.15.16:0/2001802127 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0154145e40 0x7f01540cc740 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7f0144009d90 tx=0x7f014401c970 comp rx=0 tx=0).stop 2024-03-31T15:08:07.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:07.682+0000 7f015ac00640 1 -- 172.21.15.16:0/2001802127 shutdown_connections 2024-03-31T15:08:07.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:07.682+0000 7f015ac00640 1 --2- 172.21.15.16:0/2001802127 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0154145e40 0x7f01540cc740 unknown :-1 s=CLOSED pgs=858 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:07.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:07.682+0000 7f015ac00640 1 -- 172.21.15.16:0/2001802127 >> 172.21.15.16:0/2001802127 conn(0x7f0154067330 msgr2=0x7f0154067740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:08:07.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:07.682+0000 7f015ac00640 1 -- 172.21.15.16:0/2001802127 shutdown_connections 2024-03-31T15:08:07.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:07.682+0000 7f015ac00640 1 -- 172.21.15.16:0/2001802127 wait complete. 2024-03-31T15:08:07.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:07.686+0000 7f015ac00640 1 Processor -- start 2024-03-31T15:08:07.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:07.686+0000 7f015ac00640 1 -- start start 2024-03-31T15:08:07.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:07.686+0000 7f015ac00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0154070030 0x7f015406d0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:07.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:07.686+0000 7f015ac00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f01540ccc80 0x7f0154158000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:07.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:07.686+0000 7f015ac00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0154145e40 0x7f01541445f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:07.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:07.686+0000 7f015ac00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f01540b5240 con 0x7f0154070030 2024-03-31T15:08:07.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:07.686+0000 7f015ac00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f01540b50c0 con 0x7f0154145e40 2024-03-31T15:08:07.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:07.686+0000 7f015ac00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f015416a1a0 con 0x7f01540ccc80 2024-03-31T15:08:07.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:07.686+0000 7f0153e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0154070030 0x7f015406d0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:07.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:07.686+0000 7f0153e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0154070030 0x7f015406d0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51704/0 (socket says 172.21.15.16:51704) 2024-03-31T15:08:07.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:07.686+0000 7f0153e00640 1 -- 172.21.15.16:0/1211316746 learned_addr learned my addr 172.21.15.16:0/1211316746 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:08:07.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:07.686+0000 7f014b400640 1 --2- 172.21.15.16:0/1211316746 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f01540ccc80 0x7f0154158000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:07.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:07.686+0000 7f0158c00640 1 --2- 172.21.15.16:0/1211316746 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0154145e40 0x7f01541445f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:07.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:07.686+0000 7f0153e00640 1 -- 172.21.15.16:0/1211316746 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f01540ccc80 msgr2=0x7f0154158000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:07.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:07.686+0000 7f0153e00640 1 --2- 172.21.15.16:0/1211316746 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f01540ccc80 0x7f0154158000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:07.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:07.686+0000 7f0153e00640 1 -- 172.21.15.16:0/1211316746 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0154145e40 msgr2=0x7f01541445f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:07.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:07.686+0000 7f0153e00640 1 --2- 172.21.15.16:0/1211316746 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0154145e40 0x7f01541445f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:07.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:07.686+0000 7f0153e00640 1 -- 172.21.15.16:0/1211316746 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f015415c9e0 con 0x7f0154070030 2024-03-31T15:08:07.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:07.686+0000 7f0153e00640 1 --2- 172.21.15.16:0/1211316746 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0154070030 0x7f015406d0b0 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7f0144019580 tx=0x7f0144019710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:08:07.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:07.686+0000 7f0150c00640 1 -- 172.21.15.16:0/1211316746 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01440bd020 con 0x7f0154070030 2024-03-31T15:08:07.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:07.686+0000 7f0150c00640 1 -- 172.21.15.16:0/1211316746 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f01440ad760 con 0x7f0154070030 2024-03-31T15:08:07.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:07.686+0000 7f0150c00640 1 -- 172.21.15.16:0/1211316746 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01440b2040 con 0x7f0154070030 2024-03-31T15:08:07.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:07.686+0000 7f015ac00640 1 -- 172.21.15.16:0/1211316746 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f015415ccc0 con 0x7f0154070030 2024-03-31T15:08:07.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:07.686+0000 7f015ac00640 1 -- 172.21.15.16:0/1211316746 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f015415d280 con 0x7f0154070030 2024-03-31T15:08:07.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:07.686+0000 7f015ac00640 1 -- 172.21.15.16:0/1211316746 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0154069950 con 0x7f0154070030 2024-03-31T15:08:07.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:07.690+0000 7f0150c00640 1 -- 172.21.15.16:0/1211316746 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f01440c0020 con 0x7f0154070030 2024-03-31T15:08:07.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:07.690+0000 7f0150c00640 1 --2- 172.21.15.16:0/1211316746 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f01300683d0 0x7f013006a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:07.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:07.690+0000 7f0150c00640 1 -- 172.21.15.16:0/1211316746 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(67..67 src has 1..67) ==== 5709+0+0 (secure 0 0 0) 0x7f0144121b10 con 0x7f0154070030 2024-03-31T15:08:07.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:07.690+0000 7f014b400640 1 --2- 172.21.15.16:0/1211316746 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f01300683d0 0x7f013006a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:07.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:07.690+0000 7f014b400640 1 --2- 172.21.15.16:0/1211316746 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f01300683d0 0x7f013006a890 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f015415f540 tx=0x7f013c007ba0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:08:07.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:07.694+0000 7f0150c00640 1 -- 172.21.15.16:0/1211316746 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f01440c7030 con 0x7f0154070030 2024-03-31T15:08:07.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:07.858+0000 7f015ac00640 1 -- 172.21.15.16:0/1211316746 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f01540cf3c0 con 0x7f0154070030 2024-03-31T15:08:07.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:07.862+0000 7f0150c00640 1 -- 172.21.15.16:0/1211316746 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v67) ==== 74+0+16596 (secure 0 0 0) 0x7f0144016560 con 0x7f0154070030 2024-03-31T15:08:07.856 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:08:07.857 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":67,"fsid":"286eb5da-fb47-4550-b290-d9519c13b9a5","created":"2024-03-31T15:03:11.438952+0000","modified":"2024-03-31T15:07:58.527959+0000","last_up_change":"2024-03-31T15:03:22.885535+0000","last_in_change":"2024-03-31T15:03:13.941667+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-31T15:03:22.550787+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-03-31T15:07:55.329096+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"66","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-03-31T15:07:55.885919+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"67","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"92a66450-ef19-40d8-ac3d-d760b23b0bda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6801","nonce":932712244}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6803","nonce":932712244}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6807","nonce":932712244}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6805","nonce":932712244}]},"public_addr":"172.21.15.16:6801/932712244","cluster_addr":"172.21.15.16:6803/932712244","heartbeat_back_addr":"172.21.15.16:6807/932712244","heartbeat_front_addr":"172.21.15.16:6805/932712244","state":["exists","up"]},{"osd":1,"uuid":"da9d1f2b-1968-4959-b843-faa231988e92","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6810","nonce":1398737533}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6811","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6813","nonce":1398737533}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6821","nonce":1398737533}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6815","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6816","nonce":1398737533}]},"public_addr":"172.21.15.16:6810/1398737533","cluster_addr":"172.21.15.16:6813/1398737533","heartbeat_back_addr":"172.21.15.16:6821/1398737533","heartbeat_front_addr":"172.21.15.16:6816/1398737533","state":["exists","up"]},{"osd":2,"uuid":"01a73662-9a21-4b1f-bcbb-42b9636a4638","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6819","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6822","nonce":3018526646}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6825","nonce":3018526646}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6831","nonce":3018526646}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6827","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6829","nonce":3018526646}]},"public_addr":"172.21.15.16:6822/3018526646","cluster_addr":"172.21.15.16:6825/3018526646","heartbeat_back_addr":"172.21.15.16:6831/3018526646","heartbeat_front_addr":"172.21.15.16:6829/3018526646","state":["exists","up"]},{"osd":3,"uuid":"c7771fd3-e18e-4500-ac38-23cf10ebcd75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6809","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6812","nonce":3360352974}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6817","nonce":3360352974}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6828","nonce":3360352974}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6823","nonce":3360352974}]},"public_addr":"172.21.15.16:6812/3360352974","cluster_addr":"172.21.15.16:6817/3360352974","heartbeat_back_addr":"172.21.15.16:6828/3360352974","heartbeat_front_addr":"172.21.15.16:6823/3360352974","state":["exists","up"]},{"osd":4,"uuid":"88260730-1482-4fbd-a40f-cb1124e9df4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6801","nonce":1856844856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6803","nonce":1856844856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6807","nonce":1856844856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6805","nonce":1856844856}]},"public_addr":"172.21.15.89:6801/1856844856","cluster_addr":"172.21.15.89:6803/1856844856","heartbeat_back_addr":"172.21.15.89:6807/1856844856","heartbeat_front_addr":"172.21.15.89:6805/1856844856","state":["exists","up"]},{"osd":5,"uuid":"4cb2c00c-fae0-4dd8-9d27-ec4f0093ccf7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6814","nonce":50395527}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6817","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6820","nonce":50395527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6831","nonce":50395527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6823","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6826","nonce":50395527}]},"public_addr":"172.21.15.89:6814/50395527","cluster_addr":"172.21.15.89:6820/50395527","heartbeat_back_addr":"172.21.15.89:6831/50395527","heartbeat_front_addr":"172.21.15.89:6826/50395527","state":["exists","up"]},{"osd":6,"uuid":"81ce0c19-2554-4cae-acbe-66a05e9e9b9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6809","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6812","nonce":3470471752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6815","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6818","nonce":3470471752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6827","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6829","nonce":3470471752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6821","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6824","nonce":3470471752}]},"public_addr":"172.21.15.89:6812/3470471752","cluster_addr":"172.21.15.89:6818/3470471752","heartbeat_back_addr":"172.21.15.89:6829/3470471752","heartbeat_front_addr":"172.21.15.89:6824/3470471752","state":["exists","up"]},{"osd":7,"uuid":"85e4313b-d37d-442e-8042-153bf20a847d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6811","nonce":1796778630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6813","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6816","nonce":1796778630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6825","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6828","nonce":1796778630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6819","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6822","nonce":1796778630}]},"public_addr":"172.21.15.89:6811/1796778630","cluster_addr":"172.21.15.89:6816/1796778630","heartbeat_back_addr":"172.21.15.89:6828/1796778630","heartbeat_front_addr":"172.21.15.89:6822/1796778630","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.439088+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.529970+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.680512+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:21.044161+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.582111+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.536558+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.409522+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.590229+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-31T15:08:07.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:07.862+0000 7f015ac00640 1 -- 172.21.15.16:0/1211316746 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f01300683d0 msgr2=0x7f013006a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:07.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:07.862+0000 7f015ac00640 1 --2- 172.21.15.16:0/1211316746 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f01300683d0 0x7f013006a890 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f015415f540 tx=0x7f013c007ba0 comp rx=0 tx=0).stop 2024-03-31T15:08:07.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:07.862+0000 7f015ac00640 1 -- 172.21.15.16:0/1211316746 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0154070030 msgr2=0x7f015406d0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:07.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:07.862+0000 7f015ac00640 1 --2- 172.21.15.16:0/1211316746 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0154070030 0x7f015406d0b0 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7f0144019580 tx=0x7f0144019710 comp rx=0 tx=0).stop 2024-03-31T15:08:07.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:07.862+0000 7f015ac00640 1 -- 172.21.15.16:0/1211316746 shutdown_connections 2024-03-31T15:08:07.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:07.862+0000 7f015ac00640 1 --2- 172.21.15.16:0/1211316746 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f01300683d0 0x7f013006a890 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:07.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:07.862+0000 7f015ac00640 1 --2- 172.21.15.16:0/1211316746 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0154145e40 0x7f01541445f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:07.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:07.862+0000 7f015ac00640 1 --2- 172.21.15.16:0/1211316746 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f01540ccc80 0x7f0154158000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:07.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:07.862+0000 7f015ac00640 1 --2- 172.21.15.16:0/1211316746 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0154070030 0x7f015406d0b0 unknown :-1 s=CLOSED pgs=859 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:07.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:07.862+0000 7f015ac00640 1 -- 172.21.15.16:0/1211316746 >> 172.21.15.16:0/1211316746 conn(0x7f0154067330 msgr2=0x7f015415fd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:08:07.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:07.862+0000 7f015ac00640 1 -- 172.21.15.16:0/1211316746 shutdown_connections 2024-03-31T15:08:07.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:07.862+0000 7f015ac00640 1 -- 172.21.15.16:0/1211316746 wait complete. 2024-03-31T15:08:07.877 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-31T15:03:22.550787+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 10, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-31T15:07:55.329096+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '66', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.5, 'score_stable': 2.5, 'optimal_score': 1, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 11, 'pool_name': 'cephfs_data', 'create_time': '2024-03-31T15:07:55.885919+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '67', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-31T15:08:07.878 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-31T15:08:08.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:08.026+0000 7fc7eb000640 1 Processor -- start 2024-03-31T15:08:08.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:08.026+0000 7fc7eb000640 1 -- start start 2024-03-31T15:08:08.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:08.030+0000 7fc7eb000640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fc7e406f920 0x7fc7e406fd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:08.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:08.030+0000 7fc7eb000640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fc7e40cf150 0x7fc7e41419c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:08.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:08.030+0000 7fc7eb000640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc7e4070240 con 0x7fc7e40d0a60 2024-03-31T15:08:08.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:08.030+0000 7fc7eb000640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7fc7e413f670 con 0x7fc7e40cf150 2024-03-31T15:08:08.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:08.030+0000 7fc7eb000640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fc7e415ca70 con 0x7fc7e406f920 2024-03-31T15:08:08.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:08.030+0000 7fc7dbe00640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fc7e40d0a60 0x7fc7e40cea40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:55500/0 (socket says 172.21.15.16:55500) 2024-03-31T15:08:08.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:08.030+0000 7fc7dbe00640 1 -- 172.21.15.16:0/1286940274 learned_addr learned my addr 172.21.15.16:0/1286940274 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:08:08.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:08.030+0000 7fc7e9000640 1 --2- 172.21.15.16:0/1286940274 >> v2:172.21.15.89:3300/0 conn(0x7fc7e40cf150 0x7fc7e41419c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:08.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:08.030+0000 7fc7e3e00640 1 --2- 172.21.15.16:0/1286940274 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fc7e406f920 0x7fc7e406fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:08.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:08.030+0000 7fc7e3400640 1 -- 172.21.15.16:0/1286940274 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1592551290 0 0) 0x7fc7e4070240 con 0x7fc7e40d0a60 2024-03-31T15:08:08.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:08.030+0000 7fc7e3400640 1 -- 172.21.15.16:0/1286940274 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc7c8003530 con 0x7fc7e40d0a60 2024-03-31T15:08:08.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:08.030+0000 7fc7e3e00640 1 -- 172.21.15.16:0/1286940274 >> v2:172.21.15.89:3300/0 conn(0x7fc7e40cf150 msgr2=0x7fc7e41419c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:08:08.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:08.030+0000 7fc7e3e00640 1 --2- 172.21.15.16:0/1286940274 >> v2:172.21.15.89:3300/0 conn(0x7fc7e40cf150 0x7fc7e41419c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:08.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:08.030+0000 7fc7e3e00640 1 -- 172.21.15.16:0/1286940274 >> v1:172.21.15.16:6789/0 conn(0x7fc7e40d0a60 legacy=0x7fc7e40cea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:08.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:08.030+0000 7fc7e3e00640 1 -- 172.21.15.16:0/1286940274 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc7e415cd50 con 0x7fc7e406f920 2024-03-31T15:08:08.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:08.034+0000 7fc7e3e00640 1 --2- 172.21.15.16:0/1286940274 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fc7e406f920 0x7fc7e406fd00 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fc7d40097c0 tx=0x7fc7d401c4a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c4e6f60f9ab06971 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:08:08.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:08.034+0000 7fc7e3400640 1 -- 172.21.15.16:0/1286940274 <== mon.2 v2:172.21.15.89:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7d40b0070 con 0x7fc7e406f920 2024-03-31T15:08:08.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:08.034+0000 7fc7e3400640 1 -- 172.21.15.16:0/1286940274 <== mon.2 v2:172.21.15.89:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc7d40ab600 con 0x7fc7e406f920 2024-03-31T15:08:08.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:08.034+0000 7fc7e3400640 1 -- 172.21.15.16:0/1286940274 <== mon.2 v2:172.21.15.89:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7d40ab9b0 con 0x7fc7e406f920 2024-03-31T15:08:08.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:08.034+0000 7fc7eb000640 1 -- 172.21.15.16:0/1286940274 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fc7e406f920 msgr2=0x7fc7e406fd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:08.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:08.034+0000 7fc7eb000640 1 --2- 172.21.15.16:0/1286940274 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fc7e406f920 0x7fc7e406fd00 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fc7d40097c0 tx=0x7fc7d401c4a0 comp rx=0 tx=0).stop 2024-03-31T15:08:08.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:08.034+0000 7fc7eb000640 1 -- 172.21.15.16:0/1286940274 shutdown_connections 2024-03-31T15:08:08.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:08.034+0000 7fc7eb000640 1 --2- 172.21.15.16:0/1286940274 >> v2:172.21.15.89:3300/0 conn(0x7fc7e40cf150 0x7fc7e41419c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:08.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:08.034+0000 7fc7eb000640 1 --2- 172.21.15.16:0/1286940274 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fc7e406f920 0x7fc7e406fd00 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:08.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:08.034+0000 7fc7eb000640 1 -- 172.21.15.16:0/1286940274 >> 172.21.15.16:0/1286940274 conn(0x7fc7e4067930 msgr2=0x7fc7e40b3ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:08:08.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:08.034+0000 7fc7eb000640 1 -- 172.21.15.16:0/1286940274 shutdown_connections 2024-03-31T15:08:08.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:08.034+0000 7fc7eb000640 1 -- 172.21.15.16:0/1286940274 wait complete. 2024-03-31T15:08:08.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:08.034+0000 7fc7eb000640 1 Processor -- start 2024-03-31T15:08:08.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:08.034+0000 7fc7eb000640 1 -- start start 2024-03-31T15:08:08.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:08.034+0000 7fc7eb000640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc7e406f920 0x7fc7e406cbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:08.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:08.034+0000 7fc7eb000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc7e40cf150 0x7fc7e413e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:08.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:08.034+0000 7fc7eb000640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fc7e40d0a60 0x7fc7e41549b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:08.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:08.034+0000 7fc7eb000640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fc7e4169230 con 0x7fc7e40cf150 2024-03-31T15:08:08.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:08.034+0000 7fc7eb000640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fc7e41690b0 con 0x7fc7e406f920 2024-03-31T15:08:08.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:08.034+0000 7fc7eb000640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fc7e41693b0 con 0x7fc7e40d0a60 2024-03-31T15:08:08.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:08.038+0000 7fc7dbe00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc7e40cf150 0x7fc7e413e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:08.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:08.038+0000 7fc7dbe00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc7e40cf150 0x7fc7e413e080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51718/0 (socket says 172.21.15.16:51718) 2024-03-31T15:08:08.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:08.038+0000 7fc7dbe00640 1 -- 172.21.15.16:0/1077889791 learned_addr learned my addr 172.21.15.16:0/1077889791 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:08:08.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:08.038+0000 7fc7e3e00640 1 --2- 172.21.15.16:0/1077889791 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc7e406f920 0x7fc7e406cbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:08.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:08.038+0000 7fc7e9000640 1 --2- 172.21.15.16:0/1077889791 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fc7e40d0a60 0x7fc7e41549b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:08.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:08.038+0000 7fc7dbe00640 1 -- 172.21.15.16:0/1077889791 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fc7e40d0a60 msgr2=0x7fc7e41549b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:08.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:08.038+0000 7fc7dbe00640 1 --2- 172.21.15.16:0/1077889791 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fc7e40d0a60 0x7fc7e41549b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:08.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:08.038+0000 7fc7dbe00640 1 -- 172.21.15.16:0/1077889791 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc7e406f920 msgr2=0x7fc7e406cbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:08.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:08.038+0000 7fc7dbe00640 1 --2- 172.21.15.16:0/1077889791 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc7e406f920 0x7fc7e406cbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:08.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:08.038+0000 7fc7dbe00640 1 -- 172.21.15.16:0/1077889791 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc7e406d140 con 0x7fc7e40cf150 2024-03-31T15:08:08.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:08.038+0000 7fc7e9000640 1 --2- 172.21.15.16:0/1077889791 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fc7e40d0a60 0x7fc7e41549b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:08:08.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:08.038+0000 7fc7dbe00640 1 --2- 172.21.15.16:0/1077889791 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc7e40cf150 0x7fc7e413e080 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7fc7cc00c3f0 tx=0x7fc7cc00c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:08:08.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:08.038+0000 7fc7e0c00640 1 -- 172.21.15.16:0/1077889791 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7cc017070 con 0x7fc7e40cf150 2024-03-31T15:08:08.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:08.038+0000 7fc7e0c00640 1 -- 172.21.15.16:0/1077889791 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc7cc005cf0 con 0x7fc7e40cf150 2024-03-31T15:08:08.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:08.038+0000 7fc7e0c00640 1 -- 172.21.15.16:0/1077889791 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7cc0072f0 con 0x7fc7e40cf150 2024-03-31T15:08:08.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:08.038+0000 7fc7eb000640 1 -- 172.21.15.16:0/1077889791 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc7e406d420 con 0x7fc7e40cf150 2024-03-31T15:08:08.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:08.038+0000 7fc7eb000640 1 -- 172.21.15.16:0/1077889791 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc7e40b5eb0 con 0x7fc7e40cf150 2024-03-31T15:08:08.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:08.038+0000 7fc7eb000640 1 -- 172.21.15.16:0/1077889791 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc7e4069950 con 0x7fc7e40cf150 2024-03-31T15:08:08.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:08.038+0000 7fc7e0c00640 1 -- 172.21.15.16:0/1077889791 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fc7cc01a070 con 0x7fc7e40cf150 2024-03-31T15:08:08.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:08.042+0000 7fc7e0c00640 1 --2- 172.21.15.16:0/1077889791 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fc7c0068380 0x7fc7c006a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:08.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:08.042+0000 7fc7e0c00640 1 -- 172.21.15.16:0/1077889791 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(67..67 src has 1..67) ==== 5709+0+0 (secure 0 0 0) 0x7fc7cc085dc0 con 0x7fc7e40cf150 2024-03-31T15:08:08.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:08.042+0000 7fc7e3e00640 1 --2- 172.21.15.16:0/1077889791 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fc7c0068380 0x7fc7c006a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:08.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:08.042+0000 7fc7e3e00640 1 --2- 172.21.15.16:0/1077889791 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fc7c0068380 0x7fc7c006a840 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7fc7d40ad000 tx=0x7fc7d40a6000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:08:08.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:08.050+0000 7fc7e0c00640 1 -- 172.21.15.16:0/1077889791 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fc7cc055ca0 con 0x7fc7e40cf150 2024-03-31T15:08:08.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:08.226+0000 7fc7eb000640 1 -- 172.21.15.16:0/1077889791 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fc7e40b50e0 con 0x7fc7e40cf150 2024-03-31T15:08:08.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:08.226+0000 7fc7e0c00640 1 -- 172.21.15.16:0/1077889791 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v67) ==== 93+0+10 (secure 0 0 0) 0x7fc7cc059fa0 con 0x7fc7e40cf150 2024-03-31T15:08:08.220 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2024-03-31T15:08:08.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:08.226+0000 7fc7eb000640 1 -- 172.21.15.16:0/1077889791 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fc7c0068380 msgr2=0x7fc7c006a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:08.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:08.226+0000 7fc7eb000640 1 --2- 172.21.15.16:0/1077889791 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fc7c0068380 0x7fc7c006a840 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7fc7d40ad000 tx=0x7fc7d40a6000 comp rx=0 tx=0).stop 2024-03-31T15:08:08.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:08.226+0000 7fc7eb000640 1 -- 172.21.15.16:0/1077889791 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc7e40cf150 msgr2=0x7fc7e413e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:08.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:08.226+0000 7fc7eb000640 1 --2- 172.21.15.16:0/1077889791 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc7e40cf150 0x7fc7e413e080 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7fc7cc00c3f0 tx=0x7fc7cc00c8b0 comp rx=0 tx=0).stop 2024-03-31T15:08:08.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:08.226+0000 7fc7eb000640 1 -- 172.21.15.16:0/1077889791 shutdown_connections 2024-03-31T15:08:08.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:08.226+0000 7fc7eb000640 1 --2- 172.21.15.16:0/1077889791 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fc7c0068380 0x7fc7c006a840 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:08.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:08.226+0000 7fc7eb000640 1 --2- 172.21.15.16:0/1077889791 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fc7e40d0a60 0x7fc7e41549b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:08.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:08.226+0000 7fc7eb000640 1 --2- 172.21.15.16:0/1077889791 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc7e40cf150 0x7fc7e413e080 unknown :-1 s=CLOSED pgs=861 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:08.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:08.226+0000 7fc7eb000640 1 --2- 172.21.15.16:0/1077889791 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc7e406f920 0x7fc7e406cbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:08.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:08.226+0000 7fc7eb000640 1 -- 172.21.15.16:0/1077889791 >> 172.21.15.16:0/1077889791 conn(0x7fc7e4067930 msgr2=0x7fc7e4152e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:08:08.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:08.226+0000 7fc7eb000640 1 -- 172.21.15.16:0/1077889791 shutdown_connections 2024-03-31T15:08:08.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:08.226+0000 7fc7eb000640 1 -- 172.21.15.16:0/1077889791 wait complete. 2024-03-31T15:08:08.242 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-31T15:08:08.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:08.390+0000 7fd20be00640 1 Processor -- start 2024-03-31T15:08:08.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:08.390+0000 7fd20be00640 1 -- start start 2024-03-31T15:08:08.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:08.390+0000 7fd20be00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fd20c15f090 0x7fd20c15f470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:08.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:08.390+0000 7fd20be00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fd20c072f50 0x7fd20c0cb440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:08.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:08.390+0000 7fd20be00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fd20c06b620 0x7fd20c06ba90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:08.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:08.390+0000 7fd20be00640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7fd20c06bfd0 con 0x7fd20c06b620 2024-03-31T15:08:08.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:08.390+0000 7fd20be00640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7fd20c0733d0 con 0x7fd20c15f090 2024-03-31T15:08:08.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:08.390+0000 7fd20be00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fd20c0cbd20 con 0x7fd20c072f50 2024-03-31T15:08:08.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:08.390+0000 7fd20b400640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fd20c06b620 0x7fd20c06ba90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:08.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:08.390+0000 7fd20b400640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fd20c06b620 0x7fd20c06ba90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51726/0 (socket says 172.21.15.16:51726) 2024-03-31T15:08:08.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:08.394+0000 7fd20b400640 1 -- 172.21.15.16:0/118034070 learned_addr learned my addr 172.21.15.16:0/118034070 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:08:08.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:08.394+0000 7fd20aa00640 1 --2- 172.21.15.16:0/118034070 >> v2:172.21.15.89:3300/0 conn(0x7fd20c15f090 0x7fd20c15f470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:08.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:08.394+0000 7fd203e00640 1 --2- 172.21.15.16:0/118034070 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fd20c072f50 0x7fd20c0cb440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:08.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:08.394+0000 7fd20b400640 1 -- 172.21.15.16:0/118034070 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fd20c072f50 msgr2=0x7fd20c0cb440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:08:08.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:08.394+0000 7fd20b400640 1 --2- 172.21.15.16:0/118034070 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fd20c072f50 0x7fd20c0cb440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:08.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:08.394+0000 7fd20b400640 1 -- 172.21.15.16:0/118034070 >> v2:172.21.15.89:3300/0 conn(0x7fd20c15f090 msgr2=0x7fd20c15f470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:08:08.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:08.394+0000 7fd20b400640 1 --2- 172.21.15.16:0/118034070 >> v2:172.21.15.89:3300/0 conn(0x7fd20c15f090 0x7fd20c15f470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:08.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:08.394+0000 7fd20b400640 1 -- 172.21.15.16:0/118034070 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd20c0cc000 con 0x7fd20c06b620 2024-03-31T15:08:08.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:08.394+0000 7fd20b400640 1 --2- 172.21.15.16:0/118034070 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd20c06b620 0x7fd20c06ba90 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7fd20c06d740 tx=0x7fd20401c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aea251f30addc990 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:08:08.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:08.394+0000 7fd20a000640 1 -- 172.21.15.16:0/118034070 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2040a5840 con 0x7fd20c06b620 2024-03-31T15:08:08.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:08.394+0000 7fd20a000640 1 -- 172.21.15.16:0/118034070 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd2040a59e0 con 0x7fd20c06b620 2024-03-31T15:08:08.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:08.394+0000 7fd20a000640 1 -- 172.21.15.16:0/118034070 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2040a5db0 con 0x7fd20c06b620 2024-03-31T15:08:08.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:08.398+0000 7fd20be00640 1 -- 172.21.15.16:0/118034070 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd20c06b620 msgr2=0x7fd20c06ba90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:08.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:08.398+0000 7fd20be00640 1 --2- 172.21.15.16:0/118034070 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd20c06b620 0x7fd20c06ba90 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7fd20c06d740 tx=0x7fd20401c9c0 comp rx=0 tx=0).stop 2024-03-31T15:08:08.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:08.398+0000 7fd20be00640 1 -- 172.21.15.16:0/118034070 shutdown_connections 2024-03-31T15:08:08.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:08.398+0000 7fd20be00640 1 --2- 172.21.15.16:0/118034070 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd20c06b620 0x7fd20c06ba90 unknown :-1 s=CLOSED pgs=862 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:08.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:08.398+0000 7fd20be00640 1 --2- 172.21.15.16:0/118034070 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fd20c072f50 0x7fd20c0cb440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:08.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:08.398+0000 7fd20be00640 1 --2- 172.21.15.16:0/118034070 >> v2:172.21.15.89:3300/0 conn(0x7fd20c15f090 0x7fd20c15f470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:08.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:08.398+0000 7fd20be00640 1 -- 172.21.15.16:0/118034070 >> 172.21.15.16:0/118034070 conn(0x7fd20c0d0df0 msgr2=0x7fd20c0d1200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:08:08.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:08.398+0000 7fd20be00640 1 -- 172.21.15.16:0/118034070 shutdown_connections 2024-03-31T15:08:08.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:08.398+0000 7fd20be00640 1 -- 172.21.15.16:0/118034070 wait complete. 2024-03-31T15:08:08.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:08.398+0000 7fd20be00640 1 Processor -- start 2024-03-31T15:08:08.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:08.398+0000 7fd20be00640 1 -- start start 2024-03-31T15:08:08.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:08.398+0000 7fd20be00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fd20c06b620 0x7fd20c0b4b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:08.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:08.398+0000 7fd20be00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd20c072f50 0x7fd20c156a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:08.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:08.398+0000 7fd20be00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd20c15f090 0x7fd20c157dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:08.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:08.398+0000 7fd20be00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fd20c0cbb00 con 0x7fd20c072f50 2024-03-31T15:08:08.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:08.398+0000 7fd20be00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fd20c0cb980 con 0x7fd20c15f090 2024-03-31T15:08:08.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:08.398+0000 7fd20be00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fd20c06eb70 con 0x7fd20c06b620 2024-03-31T15:08:08.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:08.398+0000 7fd203e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd20c072f50 0x7fd20c156a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:08.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:08.398+0000 7fd203e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd20c072f50 0x7fd20c156a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51730/0 (socket says 172.21.15.16:51730) 2024-03-31T15:08:08.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:08.398+0000 7fd203e00640 1 -- 172.21.15.16:0/1582128054 learned_addr learned my addr 172.21.15.16:0/1582128054 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:08:08.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:08.398+0000 7fd20aa00640 1 --2- 172.21.15.16:0/1582128054 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fd20c06b620 0x7fd20c0b4b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:08.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:08.398+0000 7fd20b400640 1 --2- 172.21.15.16:0/1582128054 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd20c15f090 0x7fd20c157dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:08.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:08.398+0000 7fd203e00640 1 -- 172.21.15.16:0/1582128054 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fd20c06b620 msgr2=0x7fd20c0b4b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:08.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:08.398+0000 7fd203e00640 1 --2- 172.21.15.16:0/1582128054 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fd20c06b620 0x7fd20c0b4b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:08.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:08.398+0000 7fd203e00640 1 -- 172.21.15.16:0/1582128054 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd20c15f090 msgr2=0x7fd20c157dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:08.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:08.398+0000 7fd203e00640 1 --2- 172.21.15.16:0/1582128054 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd20c15f090 0x7fd20c157dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:08.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:08.398+0000 7fd203e00640 1 -- 172.21.15.16:0/1582128054 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd20c156f40 con 0x7fd20c072f50 2024-03-31T15:08:08.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:08.402+0000 7fd203e00640 1 --2- 172.21.15.16:0/1582128054 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd20c072f50 0x7fd20c156a00 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7fd1f400e930 tx=0x7fd1f400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:08:08.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:08.402+0000 7fd202a00640 1 -- 172.21.15.16:0/1582128054 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1f4017070 con 0x7fd20c072f50 2024-03-31T15:08:08.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:08.402+0000 7fd202a00640 1 -- 172.21.15.16:0/1582128054 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd1f400ca00 con 0x7fd20c072f50 2024-03-31T15:08:08.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:08.402+0000 7fd20be00640 1 -- 172.21.15.16:0/1582128054 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd20c0c66b0 con 0x7fd20c072f50 2024-03-31T15:08:08.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:08.402+0000 7fd202a00640 1 -- 172.21.15.16:0/1582128054 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1f400cdb0 con 0x7fd20c072f50 2024-03-31T15:08:08.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:08.402+0000 7fd20be00640 1 -- 172.21.15.16:0/1582128054 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd20c0c6c70 con 0x7fd20c072f50 2024-03-31T15:08:08.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:08.402+0000 7fd20be00640 1 -- 172.21.15.16:0/1582128054 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd20c04f8c0 con 0x7fd20c072f50 2024-03-31T15:08:08.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:08.402+0000 7fd202a00640 1 -- 172.21.15.16:0/1582128054 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fd1f401a070 con 0x7fd20c072f50 2024-03-31T15:08:08.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:08.402+0000 7fd202a00640 1 --2- 172.21.15.16:0/1582128054 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fd1e8068380 0x7fd1e806a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:08.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:08.402+0000 7fd202a00640 1 -- 172.21.15.16:0/1582128054 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(67..67 src has 1..67) ==== 5709+0+0 (secure 0 0 0) 0x7fd1f4086970 con 0x7fd20c072f50 2024-03-31T15:08:08.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:08.402+0000 7fd20aa00640 1 --2- 172.21.15.16:0/1582128054 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fd1e8068380 0x7fd1e806a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:08.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:08.406+0000 7fd20aa00640 1 --2- 172.21.15.16:0/1582128054 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fd1e8068380 0x7fd1e806a840 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7fd1fc009a40 tx=0x7fd1fc009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:08:08.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:08.406+0000 7fd202a00640 1 -- 172.21.15.16:0/1582128054 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fd1f40569a0 con 0x7fd20c072f50 2024-03-31T15:08:08.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:08.674+0000 7fd20be00640 1 -- 172.21.15.16:0/1582128054 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fd20c0cb760 con 0x7fd20c072f50 2024-03-31T15:08:08.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:08.674+0000 7fd202a00640 1 -- 172.21.15.16:0/1582128054 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v67) ==== 104+0+11 (secure 0 0 0) 0x7fd1f405aca0 con 0x7fd20c072f50 2024-03-31T15:08:08.669 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2024-03-31T15:08:08.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:08.674+0000 7fd20be00640 1 -- 172.21.15.16:0/1582128054 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fd1e8068380 msgr2=0x7fd1e806a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:08.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:08.674+0000 7fd20be00640 1 --2- 172.21.15.16:0/1582128054 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fd1e8068380 0x7fd1e806a840 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7fd1fc009a40 tx=0x7fd1fc009190 comp rx=0 tx=0).stop 2024-03-31T15:08:08.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:08.674+0000 7fd20be00640 1 -- 172.21.15.16:0/1582128054 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd20c072f50 msgr2=0x7fd20c156a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:08.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:08.674+0000 7fd20be00640 1 --2- 172.21.15.16:0/1582128054 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd20c072f50 0x7fd20c156a00 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7fd1f400e930 tx=0x7fd1f400edf0 comp rx=0 tx=0).stop 2024-03-31T15:08:08.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:08.674+0000 7fd20be00640 1 -- 172.21.15.16:0/1582128054 shutdown_connections 2024-03-31T15:08:08.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:08.674+0000 7fd20be00640 1 --2- 172.21.15.16:0/1582128054 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fd1e8068380 0x7fd1e806a840 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:08.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:08.674+0000 7fd20be00640 1 --2- 172.21.15.16:0/1582128054 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd20c15f090 0x7fd20c157dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:08.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:08.674+0000 7fd20be00640 1 --2- 172.21.15.16:0/1582128054 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd20c072f50 0x7fd20c156a00 unknown :-1 s=CLOSED pgs=863 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:08.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:08.674+0000 7fd20be00640 1 --2- 172.21.15.16:0/1582128054 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fd20c06b620 0x7fd20c0b4b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:08.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:08.674+0000 7fd20be00640 1 -- 172.21.15.16:0/1582128054 >> 172.21.15.16:0/1582128054 conn(0x7fd20c0d0df0 msgr2=0x7fd20c06f7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:08:08.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:08.678+0000 7fd20be00640 1 -- 172.21.15.16:0/1582128054 shutdown_connections 2024-03-31T15:08:08.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:08.678+0000 7fd20be00640 1 -- 172.21.15.16:0/1582128054 wait complete. 2024-03-31T15:08:08.691 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-31T15:08:08.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:08.838+0000 7f3aace00640 1 Processor -- start 2024-03-31T15:08:08.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:08.838+0000 7f3aace00640 1 -- start start 2024-03-31T15:08:08.836 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:08.842+0000 7f3aace00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f3aa81411b0 0x7f3aa806f920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:08.836 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:08.842+0000 7f3aace00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f3aa8141590 0x7f3aa806fe60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:08.836 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:08.842+0000 7f3aace00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f3aa8143c30 0x7f3aa81440a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:08.836 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:08.842+0000 7f3aace00640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f3aa8162910 con 0x7f3aa8141590 2024-03-31T15:08:08.836 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:08.842+0000 7f3aace00640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f3aa8144980 con 0x7f3aa8143c30 2024-03-31T15:08:08.836 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:08.842+0000 7f3aace00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f3aa8161fe0 con 0x7f3aa81411b0 2024-03-31T15:08:08.836 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:08.842+0000 7f3a9fe00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f3aa8141590 0x7f3aa806fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:08.836 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:08.842+0000 7f3a9fe00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f3aa8141590 0x7f3aa806fe60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51732/0 (socket says 172.21.15.16:51732) 2024-03-31T15:08:08.836 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:08.842+0000 7f3a9fe00640 1 -- 172.21.15.16:0/1590250380 learned_addr learned my addr 172.21.15.16:0/1590250380 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:08:08.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:08.842+0000 7f3aa5e00640 1 --2- 172.21.15.16:0/1590250380 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f3aa81411b0 0x7f3aa806f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:08.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:08.842+0000 7f3aa6800640 1 --2- 172.21.15.16:0/1590250380 >> v2:172.21.15.89:3300/0 conn(0x7f3aa8143c30 0x7f3aa81440a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:08.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:08.842+0000 7f3a9fe00640 1 -- 172.21.15.16:0/1590250380 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f3aa81411b0 msgr2=0x7f3aa806f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:08:08.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:08.842+0000 7f3a9fe00640 1 --2- 172.21.15.16:0/1590250380 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f3aa81411b0 0x7f3aa806f920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:08.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:08.842+0000 7f3a9fe00640 1 -- 172.21.15.16:0/1590250380 >> v2:172.21.15.89:3300/0 conn(0x7f3aa8143c30 msgr2=0x7f3aa81440a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:08:08.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:08.842+0000 7f3a9fe00640 1 --2- 172.21.15.16:0/1590250380 >> v2:172.21.15.89:3300/0 conn(0x7f3aa8143c30 0x7f3aa81440a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:08.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:08.842+0000 7f3a9fe00640 1 -- 172.21.15.16:0/1590250380 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3aa81622c0 con 0x7f3aa8141590 2024-03-31T15:08:08.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:08.842+0000 7f3aa6800640 1 --2- 172.21.15.16:0/1590250380 >> v2:172.21.15.89:3300/0 conn(0x7f3aa8143c30 0x7f3aa81440a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:08:08.839 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:08.842+0000 7f3a9fe00640 1 --2- 172.21.15.16:0/1590250380 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3aa8141590 0x7f3aa806fe60 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7f3aa80b1c00 tx=0x7f3a9001c4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd58a891b9e691ae server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:08:08.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:08.846+0000 7f3aa5400640 1 -- 172.21.15.16:0/1590250380 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a900aa430 con 0x7f3aa8141590 2024-03-31T15:08:08.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:08.846+0000 7f3aa5400640 1 -- 172.21.15.16:0/1590250380 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3a900af070 con 0x7f3aa8141590 2024-03-31T15:08:08.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:08.846+0000 7f3aa5400640 1 -- 172.21.15.16:0/1590250380 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a900aa7f0 con 0x7f3aa8141590 2024-03-31T15:08:08.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:08.846+0000 7f3aace00640 1 -- 172.21.15.16:0/1590250380 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3aa8141590 msgr2=0x7f3aa806fe60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:08.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:08.846+0000 7f3aace00640 1 --2- 172.21.15.16:0/1590250380 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3aa8141590 0x7f3aa806fe60 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7f3aa80b1c00 tx=0x7f3a9001c4f0 comp rx=0 tx=0).stop 2024-03-31T15:08:08.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:08.846+0000 7f3aace00640 1 -- 172.21.15.16:0/1590250380 shutdown_connections 2024-03-31T15:08:08.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:08.846+0000 7f3aace00640 1 --2- 172.21.15.16:0/1590250380 >> v2:172.21.15.89:3300/0 conn(0x7f3aa8143c30 0x7f3aa81440a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:08.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:08.846+0000 7f3aace00640 1 --2- 172.21.15.16:0/1590250380 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3aa8141590 0x7f3aa806fe60 unknown :-1 s=CLOSED pgs=864 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:08.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:08.846+0000 7f3aace00640 1 --2- 172.21.15.16:0/1590250380 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f3aa81411b0 0x7f3aa806f920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:08.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:08.846+0000 7f3aace00640 1 -- 172.21.15.16:0/1590250380 >> 172.21.15.16:0/1590250380 conn(0x7f3aa8067930 msgr2=0x7f3aa80b3ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:08:08.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:08.846+0000 7f3aace00640 1 -- 172.21.15.16:0/1590250380 shutdown_connections 2024-03-31T15:08:08.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:08.846+0000 7f3aace00640 1 -- 172.21.15.16:0/1590250380 wait complete. 2024-03-31T15:08:08.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:08.846+0000 7f3aace00640 1 Processor -- start 2024-03-31T15:08:08.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:08.846+0000 7f3aace00640 1 -- start start 2024-03-31T15:08:08.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:08.846+0000 7f3aace00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3aa81411b0 0x7f3aa8156970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:08.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:08.846+0000 7f3aace00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3aa8141590 0x7f3aa8156330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:08.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:08.846+0000 7f3aace00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f3aa8143c30 0x7f3aa813f4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:08.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:08.846+0000 7f3aace00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f3aa80717f0 con 0x7f3aa81411b0 2024-03-31T15:08:08.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:08.846+0000 7f3aace00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f3aa8071670 con 0x7f3aa8141590 2024-03-31T15:08:08.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:08.846+0000 7f3aace00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f3aa8071970 con 0x7f3aa8143c30 2024-03-31T15:08:08.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:08.846+0000 7f3aa5e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3aa81411b0 0x7f3aa8156970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:08.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:08.846+0000 7f3aa5e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3aa81411b0 0x7f3aa8156970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51738/0 (socket says 172.21.15.16:51738) 2024-03-31T15:08:08.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:08.846+0000 7f3aa5e00640 1 -- 172.21.15.16:0/2517279288 learned_addr learned my addr 172.21.15.16:0/2517279288 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:08:08.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:08.846+0000 7f3a9fe00640 1 --2- 172.21.15.16:0/2517279288 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3aa8141590 0x7f3aa8156330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:08.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:08.850+0000 7f3aa6800640 1 --2- 172.21.15.16:0/2517279288 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f3aa8143c30 0x7f3aa813f4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:08.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:08.850+0000 7f3aa5e00640 1 -- 172.21.15.16:0/2517279288 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f3aa8143c30 msgr2=0x7f3aa813f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:08.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:08.850+0000 7f3aa5e00640 1 --2- 172.21.15.16:0/2517279288 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f3aa8143c30 0x7f3aa813f4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:08.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:08.850+0000 7f3aa5e00640 1 -- 172.21.15.16:0/2517279288 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3aa8141590 msgr2=0x7f3aa8156330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:08.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:08.850+0000 7f3aa5e00640 1 --2- 172.21.15.16:0/2517279288 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3aa8141590 0x7f3aa8156330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:08.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:08.850+0000 7f3aa5e00640 1 -- 172.21.15.16:0/2517279288 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3aa8157000 con 0x7f3aa81411b0 2024-03-31T15:08:08.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:08.850+0000 7f3aa6800640 1 --2- 172.21.15.16:0/2517279288 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f3aa8143c30 0x7f3aa813f4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:08:08.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:08.850+0000 7f3aa5e00640 1 --2- 172.21.15.16:0/2517279288 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3aa81411b0 0x7f3aa8156970 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7f3a9800d800 tx=0x7f3a9800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:08:08.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:08.850+0000 7f3a9e000640 1 -- 172.21.15.16:0/2517279288 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a9800b6c0 con 0x7f3aa81411b0 2024-03-31T15:08:08.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:08.850+0000 7f3a9e000640 1 -- 172.21.15.16:0/2517279288 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3a98019070 con 0x7f3aa81411b0 2024-03-31T15:08:08.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:08.850+0000 7f3a9e000640 1 -- 172.21.15.16:0/2517279288 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a9800ba60 con 0x7f3aa81411b0 2024-03-31T15:08:08.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:08.850+0000 7f3aace00640 1 -- 172.21.15.16:0/2517279288 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3aa81572e0 con 0x7f3aa81411b0 2024-03-31T15:08:08.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:08.850+0000 7f3aace00640 1 -- 172.21.15.16:0/2517279288 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3aa81578a0 con 0x7f3aa81411b0 2024-03-31T15:08:08.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:08.850+0000 7f3aace00640 1 -- 172.21.15.16:0/2517279288 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3aa8069950 con 0x7f3aa81411b0 2024-03-31T15:08:08.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:08.850+0000 7f3a9e000640 1 -- 172.21.15.16:0/2517279288 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f3a9800bc00 con 0x7f3aa81411b0 2024-03-31T15:08:08.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:08.854+0000 7f3a9e000640 1 --2- 172.21.15.16:0/2517279288 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f3a84068380 0x7f3a8406a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:08.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:08.854+0000 7f3a9e000640 1 -- 172.21.15.16:0/2517279288 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(67..67 src has 1..67) ==== 5709+0+0 (secure 0 0 0) 0x7f3a98086eb0 con 0x7f3aa81411b0 2024-03-31T15:08:08.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:08.854+0000 7f3a9fe00640 1 --2- 172.21.15.16:0/2517279288 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f3a84068380 0x7f3a8406a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:08.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:08.854+0000 7f3a9fe00640 1 --2- 172.21.15.16:0/2517279288 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f3a84068380 0x7f3a8406a840 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f3a900ae000 tx=0x7f3a900a7000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:08:08.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:08.854+0000 7f3a9e000640 1 -- 172.21.15.16:0/2517279288 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f3a980573d0 con 0x7f3aa81411b0 2024-03-31T15:08:09.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:09.018+0000 7f3aace00640 1 -- 172.21.15.16:0/2517279288 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f3aa80b4a20 con 0x7f3aa81411b0 2024-03-31T15:08:09.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:09.018+0000 7f3a9e000640 1 -- 172.21.15.16:0/2517279288 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v67) ==== 100+0+11 (secure 0 0 0) 0x7f3a98056cd0 con 0x7f3aa81411b0 2024-03-31T15:08:09.015 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 64 2024-03-31T15:08:09.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:09.022+0000 7f3aace00640 1 -- 172.21.15.16:0/2517279288 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f3a84068380 msgr2=0x7f3a8406a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:09.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:09.022+0000 7f3aace00640 1 --2- 172.21.15.16:0/2517279288 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f3a84068380 0x7f3a8406a840 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f3a900ae000 tx=0x7f3a900a7000 comp rx=0 tx=0).stop 2024-03-31T15:08:09.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:09.022+0000 7f3aace00640 1 -- 172.21.15.16:0/2517279288 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3aa81411b0 msgr2=0x7f3aa8156970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:09.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:09.022+0000 7f3aace00640 1 --2- 172.21.15.16:0/2517279288 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3aa81411b0 0x7f3aa8156970 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7f3a9800d800 tx=0x7f3a9800dcc0 comp rx=0 tx=0).stop 2024-03-31T15:08:09.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:09.022+0000 7f3aace00640 1 -- 172.21.15.16:0/2517279288 shutdown_connections 2024-03-31T15:08:09.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:09.022+0000 7f3aace00640 1 --2- 172.21.15.16:0/2517279288 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f3a84068380 0x7f3a8406a840 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:09.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:09.022+0000 7f3aace00640 1 --2- 172.21.15.16:0/2517279288 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f3aa8143c30 0x7f3aa813f4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:09.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:09.022+0000 7f3aace00640 1 --2- 172.21.15.16:0/2517279288 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3aa8141590 0x7f3aa8156330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:09.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:09.022+0000 7f3aace00640 1 --2- 172.21.15.16:0/2517279288 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3aa81411b0 0x7f3aa8156970 unknown :-1 s=CLOSED pgs=865 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:09.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:09.022+0000 7f3aace00640 1 -- 172.21.15.16:0/2517279288 >> 172.21.15.16:0/2517279288 conn(0x7f3aa8067930 msgr2=0x7f3aa8161780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:08:09.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:09.022+0000 7f3aace00640 1 -- 172.21.15.16:0/2517279288 shutdown_connections 2024-03-31T15:08:09.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:09.022+0000 7f3aace00640 1 -- 172.21.15.16:0/2517279288 wait complete. 2024-03-31T15:08:09.044 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-31T15:08:09.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:09.190+0000 7f0426a00640 1 Processor -- start 2024-03-31T15:08:09.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:09.190+0000 7f0426a00640 1 -- start start 2024-03-31T15:08:09.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:09.190+0000 7f0426a00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f042006bd30 0x7f0420070660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:09.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:09.190+0000 7f0426a00640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0420072460 con 0x7f042015e370 2024-03-31T15:08:09.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:09.190+0000 7f0426a00640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0420073350 con 0x7f042006e4c0 2024-03-31T15:08:09.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:09.190+0000 7f0426a00640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f04200727b0 con 0x7f042006bd30 2024-03-31T15:08:09.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:09.190+0000 7f0426000640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f042006bd30 0x7f0420070660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:09.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:09.190+0000 7f0425600640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f042015e370 0x7f0420072c40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:55516/0 (socket says 172.21.15.16:55516) 2024-03-31T15:08:09.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:09.190+0000 7f0425600640 1 -- 172.21.15.16:0/629283350 learned_addr learned my addr 172.21.15.16:0/629283350 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:08:09.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:09.190+0000 7f0426000640 1 -- 172.21.15.16:0/629283350 >> v1:172.21.15.89:6789/0 conn(0x7f042006e4c0 legacy=0x7f042006b620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:09.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:09.194+0000 7f0426000640 1 -- 172.21.15.16:0/629283350 >> v1:172.21.15.16:6789/0 conn(0x7f042015e370 legacy=0x7f0420072c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:09.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:09.194+0000 7f0426000640 1 -- 172.21.15.16:0/629283350 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f042006ea80 con 0x7f042006bd30 2024-03-31T15:08:09.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:09.194+0000 7f0426000640 1 --2- 172.21.15.16:0/629283350 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f042006bd30 0x7f0420070660 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7f0418009d90 tx=0x7f041801c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=714df5a8253fb707 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:08:09.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:09.194+0000 7f041fe00640 1 -- 172.21.15.16:0/629283350 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04180b2070 con 0x7f042006bd30 2024-03-31T15:08:09.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:09.194+0000 7f041fe00640 1 -- 172.21.15.16:0/629283350 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f04180a5910 con 0x7f042006bd30 2024-03-31T15:08:09.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:09.194+0000 7f041fe00640 1 -- 172.21.15.16:0/629283350 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04180a5ce0 con 0x7f042006bd30 2024-03-31T15:08:09.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:09.194+0000 7f0426a00640 1 -- 172.21.15.16:0/629283350 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f042006bd30 msgr2=0x7f0420070660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:09.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:09.194+0000 7f0426a00640 1 --2- 172.21.15.16:0/629283350 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f042006bd30 0x7f0420070660 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7f0418009d90 tx=0x7f041801c940 comp rx=0 tx=0).stop 2024-03-31T15:08:09.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:09.194+0000 7f0426a00640 1 -- 172.21.15.16:0/629283350 shutdown_connections 2024-03-31T15:08:09.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:09.194+0000 7f0426a00640 1 --2- 172.21.15.16:0/629283350 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f042006bd30 0x7f0420070660 unknown :-1 s=CLOSED pgs=867 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:09.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:09.194+0000 7f0426a00640 1 -- 172.21.15.16:0/629283350 >> 172.21.15.16:0/629283350 conn(0x7f04200b2b30 msgr2=0x7f04200b2f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:08:09.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:09.194+0000 7f0426a00640 1 -- 172.21.15.16:0/629283350 shutdown_connections 2024-03-31T15:08:09.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:09.194+0000 7f0426a00640 1 -- 172.21.15.16:0/629283350 wait complete. 2024-03-31T15:08:09.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:09.198+0000 7f0426a00640 1 Processor -- start 2024-03-31T15:08:09.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:09.198+0000 7f0426a00640 1 -- start start 2024-03-31T15:08:09.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:09.198+0000 7f0426a00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f042006bd30 0x7f04200cd850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:09.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:09.198+0000 7f0426a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f042006e4c0 0x7f0420155990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:09.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:09.198+0000 7f0426a00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f042015e370 0x7f0420152bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:09.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:09.198+0000 7f0426a00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f042016c2e0 con 0x7f042006e4c0 2024-03-31T15:08:09.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:09.198+0000 7f0426a00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f042016c160 con 0x7f042006bd30 2024-03-31T15:08:09.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:09.198+0000 7f0426a00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f042016c460 con 0x7f042015e370 2024-03-31T15:08:09.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:09.198+0000 7f0424c00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f042006e4c0 0x7f0420155990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:09.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:09.198+0000 7f0424c00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f042006e4c0 0x7f0420155990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51770/0 (socket says 172.21.15.16:51770) 2024-03-31T15:08:09.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:09.198+0000 7f0424c00640 1 -- 172.21.15.16:0/1874037354 learned_addr learned my addr 172.21.15.16:0/1874037354 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:08:09.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:09.198+0000 7f0425600640 1 --2- 172.21.15.16:0/1874037354 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f042006bd30 0x7f04200cd850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:09.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:09.198+0000 7f0426000640 1 --2- 172.21.15.16:0/1874037354 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f042015e370 0x7f0420152bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:09.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:09.198+0000 7f0424c00640 1 -- 172.21.15.16:0/1874037354 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f042015e370 msgr2=0x7f0420152bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:09.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:09.198+0000 7f0424c00640 1 --2- 172.21.15.16:0/1874037354 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f042015e370 0x7f0420152bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:09.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:09.198+0000 7f0424c00640 1 -- 172.21.15.16:0/1874037354 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f042006bd30 msgr2=0x7f04200cd850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:09.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:09.198+0000 7f0424c00640 1 --2- 172.21.15.16:0/1874037354 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f042006bd30 0x7f04200cd850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:09.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:09.198+0000 7f0424c00640 1 -- 172.21.15.16:0/1874037354 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04200cde30 con 0x7f042006e4c0 2024-03-31T15:08:09.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:09.198+0000 7f0424c00640 1 --2- 172.21.15.16:0/1874037354 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f042006e4c0 0x7f0420155990 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7f040c00c420 tx=0x7f040c00c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:08:09.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:09.198+0000 7f041d600640 1 -- 172.21.15.16:0/1874037354 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f040c0052d0 con 0x7f042006e4c0 2024-03-31T15:08:09.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:09.198+0000 7f041d600640 1 -- 172.21.15.16:0/1874037354 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f040c005470 con 0x7f042006e4c0 2024-03-31T15:08:09.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:09.198+0000 7f0426a00640 1 -- 172.21.15.16:0/1874037354 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f04200ce110 con 0x7f042006e4c0 2024-03-31T15:08:09.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:09.198+0000 7f041d600640 1 -- 172.21.15.16:0/1874037354 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f040c007260 con 0x7f042006e4c0 2024-03-31T15:08:09.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:09.198+0000 7f0426a00640 1 -- 172.21.15.16:0/1874037354 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f04200ce9b0 con 0x7f042006e4c0 2024-03-31T15:08:09.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:09.202+0000 7f0426a00640 1 -- 172.21.15.16:0/1874037354 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f04200cebb0 con 0x7f042006e4c0 2024-03-31T15:08:09.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:09.202+0000 7f041d600640 1 -- 172.21.15.16:0/1874037354 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f040c007400 con 0x7f042006e4c0 2024-03-31T15:08:09.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:09.202+0000 7f041d600640 1 --2- 172.21.15.16:0/1874037354 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f04000683b0 0x7f040006a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:09.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:09.202+0000 7f041d600640 1 -- 172.21.15.16:0/1874037354 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(67..67 src has 1..67) ==== 5709+0+0 (secure 0 0 0) 0x7f040c085d40 con 0x7f042006e4c0 2024-03-31T15:08:09.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:09.202+0000 7f0425600640 1 --2- 172.21.15.16:0/1874037354 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f04000683b0 0x7f040006a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:09.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:09.202+0000 7f0425600640 1 --2- 172.21.15.16:0/1874037354 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f04000683b0 0x7f040006a870 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f0414005890 tx=0x7f0414009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:08:09.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:09.206+0000 7f041d600640 1 -- 172.21.15.16:0/1874037354 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f04200cebb0 con 0x7f042006e4c0 2024-03-31T15:08:09.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:09.442+0000 7f0426a00640 1 -- 172.21.15.16:0/1874037354 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f042004f8c0 con 0x7f042006e4c0 2024-03-31T15:08:09.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:09.442+0000 7f041d600640 1 -- 172.21.15.16:0/1874037354 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v67) ==== 80+0+0 (secure 0 0 0) 0x7f040c055d40 con 0x7f042006e4c0 2024-03-31T15:08:09.439 INFO:teuthology.orchestra.run.smithi016.stderr:listed 0 entries 2024-03-31T15:08:09.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:09.446+0000 7f0426a00640 1 -- 172.21.15.16:0/1874037354 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f04000683b0 msgr2=0x7f040006a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:09.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:09.446+0000 7f0426a00640 1 --2- 172.21.15.16:0/1874037354 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f04000683b0 0x7f040006a870 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f0414005890 tx=0x7f0414009040 comp rx=0 tx=0).stop 2024-03-31T15:08:09.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:09.446+0000 7f0426a00640 1 -- 172.21.15.16:0/1874037354 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f042006e4c0 msgr2=0x7f0420155990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:09.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:09.446+0000 7f0426a00640 1 --2- 172.21.15.16:0/1874037354 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f042006e4c0 0x7f0420155990 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7f040c00c420 tx=0x7f040c00c8e0 comp rx=0 tx=0).stop 2024-03-31T15:08:09.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:09.446+0000 7f0426a00640 1 -- 172.21.15.16:0/1874037354 shutdown_connections 2024-03-31T15:08:09.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:09.446+0000 7f0426a00640 1 --2- 172.21.15.16:0/1874037354 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f04000683b0 0x7f040006a870 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:09.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:09.446+0000 7f0426a00640 1 --2- 172.21.15.16:0/1874037354 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f042015e370 0x7f0420152bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:09.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:09.446+0000 7f0426a00640 1 --2- 172.21.15.16:0/1874037354 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f042006e4c0 0x7f0420155990 unknown :-1 s=CLOSED pgs=868 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:09.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:09.446+0000 7f0426a00640 1 --2- 172.21.15.16:0/1874037354 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f042006bd30 0x7f04200cd850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:09.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:09.446+0000 7f0426a00640 1 -- 172.21.15.16:0/1874037354 >> 172.21.15.16:0/1874037354 conn(0x7f04200b2b30 msgr2=0x7f04200ae1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:08:09.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:09.446+0000 7f0426a00640 1 -- 172.21.15.16:0/1874037354 shutdown_connections 2024-03-31T15:08:09.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:09.446+0000 7f0426a00640 1 -- 172.21.15.16:0/1874037354 wait complete. 2024-03-31T15:08:09.470 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi016.front.sepia.ceph.com... 2024-03-31T15:08:09.470 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:08:09.470 DEBUG:teuthology.orchestra.run.smithi016:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-03-31T15:08:09.488 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi016.stderr:ceph-fuse[41790]: fuse finished with error 0 and tester_r 0 2024-03-31T15:08:09.491 INFO:teuthology.orchestra.run:waiting for 300 2024-03-31T15:08:09.558 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi016.front.sepia.ceph.com 2024-03-31T15:08:09.559 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:08:09.559 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-31T15:08:09.570 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-31T15:08:09.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:09.758+0000 7f48c5000640 1 Processor -- start 2024-03-31T15:08:09.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:09.758+0000 7f48c5000640 1 -- start start 2024-03-31T15:08:09.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:09.762+0000 7f48c5000640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f48c00d0c50 0x7f48c0072fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:09.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:09.762+0000 7f48c5000640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f48c0073510 0x7f48c015eb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:09.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:09.762+0000 7f48c5000640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f48c006bd30 con 0x7f48c0147e70 2024-03-31T15:08:09.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:09.762+0000 7f48c5000640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f48c0073980 con 0x7f48c0073510 2024-03-31T15:08:09.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:09.762+0000 7f48c5000640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f48c01621b0 con 0x7f48c00d0c50 2024-03-31T15:08:09.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:09.762+0000 7f48be800640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f48c0073510 0x7f48c015eb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:09.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:09.762+0000 7f48bde00640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f48c0147e70 0x7f48c006b620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:55524/0 (socket says 172.21.15.16:55524) 2024-03-31T15:08:09.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:09.762+0000 7f48bde00640 1 -- 172.21.15.16:0/2090067566 learned_addr learned my addr 172.21.15.16:0/2090067566 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:08:09.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:09.762+0000 7f48b5400640 1 --2- 172.21.15.16:0/2090067566 >> v2:172.21.15.89:3300/0 conn(0x7f48c00d0c50 0x7f48c0072fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:09.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:09.762+0000 7f48be800640 1 -- 172.21.15.16:0/2090067566 >> v2:172.21.15.89:3300/0 conn(0x7f48c00d0c50 msgr2=0x7f48c0072fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:08:09.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:09.762+0000 7f48be800640 1 --2- 172.21.15.16:0/2090067566 >> v2:172.21.15.89:3300/0 conn(0x7f48c00d0c50 0x7f48c0072fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:09.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:09.762+0000 7f48be800640 1 -- 172.21.15.16:0/2090067566 >> v1:172.21.15.16:6789/0 conn(0x7f48c0147e70 legacy=0x7f48c006b620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:09.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:09.762+0000 7f48be800640 1 -- 172.21.15.16:0/2090067566 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48c0162490 con 0x7f48c0073510 2024-03-31T15:08:09.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:09.762+0000 7f48bd400640 1 -- 172.21.15.16:0/2090067566 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 266692645 0 0) 0x7f48c006bd30 con 0x7f48c0147e70 2024-03-31T15:08:09.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:09.766+0000 7f48be800640 1 --2- 172.21.15.16:0/2090067566 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f48c0073510 0x7f48c015eb20 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7f48c00b7310 tx=0x7f48b801c930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd70f4c68fec6ca1 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:08:09.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:09.766+0000 7f48bd400640 1 -- 172.21.15.16:0/2090067566 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48b80b2070 con 0x7f48c0073510 2024-03-31T15:08:09.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:09.766+0000 7f48bd400640 1 -- 172.21.15.16:0/2090067566 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f48b80a58e0 con 0x7f48c0073510 2024-03-31T15:08:09.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:09.766+0000 7f48bd400640 1 -- 172.21.15.16:0/2090067566 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48b80a5cb0 con 0x7f48c0073510 2024-03-31T15:08:09.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:09.766+0000 7f48c5000640 1 -- 172.21.15.16:0/2090067566 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f48c0073510 msgr2=0x7f48c015eb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:09.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:09.766+0000 7f48c5000640 1 --2- 172.21.15.16:0/2090067566 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f48c0073510 0x7f48c015eb20 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7f48c00b7310 tx=0x7f48b801c930 comp rx=0 tx=0).stop 2024-03-31T15:08:09.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:09.766+0000 7f48c5000640 1 -- 172.21.15.16:0/2090067566 shutdown_connections 2024-03-31T15:08:09.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:09.766+0000 7f48c5000640 1 --2- 172.21.15.16:0/2090067566 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f48c0073510 0x7f48c015eb20 unknown :-1 s=CLOSED pgs=870 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:09.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:09.766+0000 7f48c5000640 1 --2- 172.21.15.16:0/2090067566 >> v2:172.21.15.89:3300/0 conn(0x7f48c00d0c50 0x7f48c0072fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:09.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:09.766+0000 7f48c5000640 1 -- 172.21.15.16:0/2090067566 >> 172.21.15.16:0/2090067566 conn(0x7f48c00d1060 msgr2=0x7f48c00d1470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:08:09.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:09.766+0000 7f48c5000640 1 -- 172.21.15.16:0/2090067566 shutdown_connections 2024-03-31T15:08:09.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:09.766+0000 7f48c5000640 1 -- 172.21.15.16:0/2090067566 wait complete. 2024-03-31T15:08:09.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:09.766+0000 7f48c5000640 1 Processor -- start 2024-03-31T15:08:09.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:09.766+0000 7f48c5000640 1 -- start start 2024-03-31T15:08:09.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:09.766+0000 7f48c5000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f48c0073510 0x7f48c00ce2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:09.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:09.766+0000 7f48c5000640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f48c00d0c50 0x7f48c015b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:09.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:09.766+0000 7f48c5000640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f48c0147e70 0x7f48c015bc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:09.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:09.766+0000 7f48c5000640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f48c015f870 con 0x7f48c0073510 2024-03-31T15:08:09.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:09.766+0000 7f48c5000640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f48c0162f70 con 0x7f48c00d0c50 2024-03-31T15:08:09.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:09.766+0000 7f48c5000640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f48c015f9f0 con 0x7f48c0147e70 2024-03-31T15:08:09.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:09.770+0000 7f48bde00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f48c0073510 0x7f48c00ce2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:09.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:09.770+0000 7f48bde00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f48c0073510 0x7f48c00ce2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51796/0 (socket says 172.21.15.16:51796) 2024-03-31T15:08:09.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:09.770+0000 7f48bde00640 1 -- 172.21.15.16:0/751625528 learned_addr learned my addr 172.21.15.16:0/751625528 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:08:09.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:09.770+0000 7f48be800640 1 --2- 172.21.15.16:0/751625528 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f48c0147e70 0x7f48c015bc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:09.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:09.770+0000 7f48b5400640 1 --2- 172.21.15.16:0/751625528 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f48c00d0c50 0x7f48c015b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:09.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:09.770+0000 7f48bde00640 1 -- 172.21.15.16:0/751625528 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f48c0147e70 msgr2=0x7f48c015bc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:09.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:09.770+0000 7f48bde00640 1 --2- 172.21.15.16:0/751625528 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f48c0147e70 0x7f48c015bc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:09.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:09.770+0000 7f48bde00640 1 -- 172.21.15.16:0/751625528 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f48c00d0c50 msgr2=0x7f48c015b6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:09.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:09.770+0000 7f48bde00640 1 --2- 172.21.15.16:0/751625528 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f48c00d0c50 0x7f48c015b6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:09.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:09.770+0000 7f48bde00640 1 -- 172.21.15.16:0/751625528 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48c0147400 con 0x7f48c0073510 2024-03-31T15:08:09.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:09.770+0000 7f48bde00640 1 --2- 172.21.15.16:0/751625528 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f48c0073510 0x7f48c00ce2f0 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7f48b00011f0 tx=0x7f48b000c9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:08:09.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:09.770+0000 7f48b6a00640 1 -- 172.21.15.16:0/751625528 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48b0018070 con 0x7f48c0073510 2024-03-31T15:08:09.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:09.770+0000 7f48b6a00640 1 -- 172.21.15.16:0/751625528 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f48b00074e0 con 0x7f48c0073510 2024-03-31T15:08:09.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:09.770+0000 7f48c5000640 1 -- 172.21.15.16:0/751625528 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f48c00b10a0 con 0x7f48c0073510 2024-03-31T15:08:09.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:09.770+0000 7f48b6a00640 1 -- 172.21.15.16:0/751625528 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48b0004a10 con 0x7f48c0073510 2024-03-31T15:08:09.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:09.770+0000 7f48c5000640 1 -- 172.21.15.16:0/751625528 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f48c00ce830 con 0x7f48c0073510 2024-03-31T15:08:09.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:09.770+0000 7f48c5000640 1 -- 172.21.15.16:0/751625528 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f48c0069950 con 0x7f48c0073510 2024-03-31T15:08:09.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:09.770+0000 7f48b6a00640 1 -- 172.21.15.16:0/751625528 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f48b001b070 con 0x7f48c0073510 2024-03-31T15:08:09.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:09.774+0000 7f48b6a00640 1 --2- 172.21.15.16:0/751625528 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f489c068380 0x7f489c06a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:09.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:09.774+0000 7f48b6a00640 1 -- 172.21.15.16:0/751625528 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(67..67 src has 1..67) ==== 5709+0+0 (secure 0 0 0) 0x7f48b0085de0 con 0x7f48c0073510 2024-03-31T15:08:09.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:09.774+0000 7f48b5400640 1 --2- 172.21.15.16:0/751625528 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f489c068380 0x7f489c06a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:09.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:09.774+0000 7f48b5400640 1 --2- 172.21.15.16:0/751625528 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f489c068380 0x7f489c06a840 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f48c015dc90 tx=0x7f48a8008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:08:09.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:09.778+0000 7f48b6a00640 1 -- 172.21.15.16:0/751625528 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f48b0056300 con 0x7f48c0073510 2024-03-31T15:08:09.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:09.978+0000 7f48c5000640 1 -- 172.21.15.16:0/751625528 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f48c00d1bf0 con 0x7f48c0073510 2024-03-31T15:08:09.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:09.978+0000 7f48b6a00640 1 -- 172.21.15.16:0/751625528 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 45 v45) ==== 94+0+5183 (secure 0 0 0) 0x7f48b0055c00 con 0x7f48c0073510 2024-03-31T15:08:09.973 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:08:09.974 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":45,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6836/4188793506","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":4188793506},{"type":"v1","addr":"172.21.15.89:6836","nonce":4188793506}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":40},{"gid":5942,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/4105657676","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6835","nonce":4105657676},{"type":"v1","addr":"172.21.15.89:6837","nonce":4105657676}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":40},{"gid":6592,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6833/3986752851","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":3986752851},{"type":"v1","addr":"172.21.15.16:6833","nonce":3986752851}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":39}],"filesystems":[{"mdsmap":{"epoch":45,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-31T15:07:56.938140+0000","modified":"2024-03-31T15:07:59.939362+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6714},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6714":{"gid":6714,"name":"a","rank":0,"incarnation":41,"state":"up:active","state_seq":3,"addr":"172.21.15.16:6835/2555930820","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":2555930820},{"type":"v1","addr":"172.21.15.16:6835","nonce":2555930820}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6714,"qdb_cluster":[6714]},"id":5}]} 2024-03-31T15:08:09.974 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 45 2024-03-31T15:08:09.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:09.978+0000 7f48c5000640 1 -- 172.21.15.16:0/751625528 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f489c068380 msgr2=0x7f489c06a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:09.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:09.978+0000 7f48c5000640 1 --2- 172.21.15.16:0/751625528 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f489c068380 0x7f489c06a840 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f48c015dc90 tx=0x7f48a8008040 comp rx=0 tx=0).stop 2024-03-31T15:08:09.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:09.978+0000 7f48c5000640 1 -- 172.21.15.16:0/751625528 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f48c0073510 msgr2=0x7f48c00ce2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:09.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:09.978+0000 7f48c5000640 1 --2- 172.21.15.16:0/751625528 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f48c0073510 0x7f48c00ce2f0 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7f48b00011f0 tx=0x7f48b000c9b0 comp rx=0 tx=0).stop 2024-03-31T15:08:09.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:09.982+0000 7f48c5000640 1 -- 172.21.15.16:0/751625528 shutdown_connections 2024-03-31T15:08:09.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:09.982+0000 7f48c5000640 1 --2- 172.21.15.16:0/751625528 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f489c068380 0x7f489c06a840 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:09.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:09.982+0000 7f48c5000640 1 --2- 172.21.15.16:0/751625528 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f48c0147e70 0x7f48c015bc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:09.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:09.982+0000 7f48c5000640 1 --2- 172.21.15.16:0/751625528 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f48c00d0c50 0x7f48c015b6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:09.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:09.982+0000 7f48c5000640 1 --2- 172.21.15.16:0/751625528 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f48c0073510 0x7f48c00ce2f0 unknown :-1 s=CLOSED pgs=871 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:09.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:09.982+0000 7f48c5000640 1 -- 172.21.15.16:0/751625528 >> 172.21.15.16:0/751625528 conn(0x7f48c00d1060 msgr2=0x7f48c006b770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:08:09.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:09.982+0000 7f48c5000640 1 -- 172.21.15.16:0/751625528 shutdown_connections 2024-03-31T15:08:09.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:09.982+0000 7f48c5000640 1 -- 172.21.15.16:0/751625528 wait complete. 2024-03-31T15:08:09.994 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json flush journal 2024-03-31T15:08:10.167 INFO:teuthology.orchestra.run.smithi016.stdout:{"message":"","return_code":0} 2024-03-31T15:08:10.167 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2024-03-31T15:08:10.168 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-31T15:08:10.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:10.314+0000 7fd8c8400640 1 Processor -- start 2024-03-31T15:08:10.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:10.314+0000 7fd8c8400640 1 -- start start 2024-03-31T15:08:10.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:10.314+0000 7fd8c8400640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fd8c006dbc0 0x7fd8c015f650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:10.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:10.314+0000 7fd8c8400640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fd8c00cb440 0x7fd8c006f920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:10.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:10.314+0000 7fd8c8400640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd8c00cbe90 con 0x7fd8c00cb8c0 2024-03-31T15:08:10.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:10.314+0000 7fd8c8400640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7fd8c015fb90 con 0x7fd8c00cb440 2024-03-31T15:08:10.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:10.314+0000 7fd8c8400640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7fd8c015f1c0 con 0x7fd8c006dbc0 2024-03-31T15:08:10.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:10.318+0000 7fd8c5000640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fd8c00cb440 0x7fd8c006f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:10.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:10.318+0000 7fd8c6400640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fd8c00cb8c0 0x7fd8c0162180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:55532/0 (socket says 172.21.15.16:55532) 2024-03-31T15:08:10.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:10.318+0000 7fd8c6400640 1 -- 172.21.15.16:0/3556688287 learned_addr learned my addr 172.21.15.16:0/3556688287 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:08:10.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:10.318+0000 7fd8c5a00640 1 --2- 172.21.15.16:0/3556688287 >> v2:172.21.15.89:3300/0 conn(0x7fd8c006dbc0 0x7fd8c015f650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:10.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:10.318+0000 7fd8c5000640 1 -- 172.21.15.16:0/3556688287 >> v2:172.21.15.89:3300/0 conn(0x7fd8c006dbc0 msgr2=0x7fd8c015f650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:08:10.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:10.318+0000 7fd8c5000640 1 --2- 172.21.15.16:0/3556688287 >> v2:172.21.15.89:3300/0 conn(0x7fd8c006dbc0 0x7fd8c015f650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:10.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:10.318+0000 7fd8c5000640 1 -- 172.21.15.16:0/3556688287 >> v1:172.21.15.16:6789/0 conn(0x7fd8c00cb8c0 legacy=0x7fd8c0162180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:10.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:10.318+0000 7fd8bfe00640 1 -- 172.21.15.16:0/3556688287 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1290137926 0 0) 0x7fd8c00cbe90 con 0x7fd8c00cb8c0 2024-03-31T15:08:10.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:10.318+0000 7fd8c5000640 1 -- 172.21.15.16:0/3556688287 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8c00cc170 con 0x7fd8c00cb440 2024-03-31T15:08:10.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:10.318+0000 7fd8c5000640 1 --2- 172.21.15.16:0/3556688287 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd8c00cb440 0x7fd8c006f920 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7fd8c00b8380 tx=0x7fd8ac01c3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=99b416ac95ab3c91 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:08:10.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:10.318+0000 7fd8bfe00640 1 -- 172.21.15.16:0/3556688287 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8ac0b0070 con 0x7fd8c00cb440 2024-03-31T15:08:10.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:10.318+0000 7fd8bfe00640 1 -- 172.21.15.16:0/3556688287 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd8ac0ab600 con 0x7fd8c00cb440 2024-03-31T15:08:10.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:10.318+0000 7fd8bfe00640 1 -- 172.21.15.16:0/3556688287 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8ac0ab9d0 con 0x7fd8c00cb440 2024-03-31T15:08:10.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:10.322+0000 7fd8c8400640 1 -- 172.21.15.16:0/3556688287 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd8c00cb440 msgr2=0x7fd8c006f920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:10.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:10.322+0000 7fd8c8400640 1 --2- 172.21.15.16:0/3556688287 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd8c00cb440 0x7fd8c006f920 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7fd8c00b8380 tx=0x7fd8ac01c3c0 comp rx=0 tx=0).stop 2024-03-31T15:08:10.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:10.322+0000 7fd8c8400640 1 -- 172.21.15.16:0/3556688287 shutdown_connections 2024-03-31T15:08:10.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:10.322+0000 7fd8c8400640 1 --2- 172.21.15.16:0/3556688287 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd8c00cb440 0x7fd8c006f920 unknown :-1 s=CLOSED pgs=873 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:10.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:10.322+0000 7fd8c8400640 1 --2- 172.21.15.16:0/3556688287 >> v2:172.21.15.89:3300/0 conn(0x7fd8c006dbc0 0x7fd8c015f650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:10.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:10.322+0000 7fd8c8400640 1 -- 172.21.15.16:0/3556688287 >> 172.21.15.16:0/3556688287 conn(0x7fd8c0067330 msgr2=0x7fd8c0067740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:08:10.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:10.322+0000 7fd8c8400640 1 -- 172.21.15.16:0/3556688287 shutdown_connections 2024-03-31T15:08:10.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:10.322+0000 7fd8c8400640 1 -- 172.21.15.16:0/3556688287 wait complete. 2024-03-31T15:08:10.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:10.322+0000 7fd8c8400640 1 Processor -- start 2024-03-31T15:08:10.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:10.322+0000 7fd8c8400640 1 -- start start 2024-03-31T15:08:10.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:10.322+0000 7fd8c8400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd8c006dbc0 0x7fd8c013dc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:10.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:10.322+0000 7fd8c8400640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fd8c00cb440 0x7fd8c0143c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:10.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:10.322+0000 7fd8c8400640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd8c00cb8c0 0x7fd8c00b5f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:10.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:10.322+0000 7fd8c8400640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fd8c006c950 con 0x7fd8c006dbc0 2024-03-31T15:08:10.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:10.322+0000 7fd8c8400640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fd8c0162ca0 con 0x7fd8c00cb8c0 2024-03-31T15:08:10.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:10.322+0000 7fd8c8400640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fd8c006cad0 con 0x7fd8c00cb440 2024-03-31T15:08:10.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:10.322+0000 7fd8c5a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd8c006dbc0 0x7fd8c013dc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:10.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:10.322+0000 7fd8c5a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd8c006dbc0 0x7fd8c013dc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51810/0 (socket says 172.21.15.16:51810) 2024-03-31T15:08:10.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:10.322+0000 7fd8c5a00640 1 -- 172.21.15.16:0/3860850172 learned_addr learned my addr 172.21.15.16:0/3860850172 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:08:10.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:10.322+0000 7fd8c6400640 1 --2- 172.21.15.16:0/3860850172 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd8c00cb8c0 0x7fd8c00b5f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:10.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:10.322+0000 7fd8c5000640 1 --2- 172.21.15.16:0/3860850172 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fd8c00cb440 0x7fd8c0143c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:10.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:10.322+0000 7fd8c5a00640 1 -- 172.21.15.16:0/3860850172 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fd8c00cb440 msgr2=0x7fd8c0143c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:10.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:10.322+0000 7fd8c5a00640 1 --2- 172.21.15.16:0/3860850172 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fd8c00cb440 0x7fd8c0143c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:10.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:10.322+0000 7fd8c5a00640 1 -- 172.21.15.16:0/3860850172 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd8c00cb8c0 msgr2=0x7fd8c00b5f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:10.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:10.322+0000 7fd8c5a00640 1 --2- 172.21.15.16:0/3860850172 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd8c00cb8c0 0x7fd8c00b5f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:10.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:10.322+0000 7fd8c5a00640 1 -- 172.21.15.16:0/3860850172 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8c0140800 con 0x7fd8c006dbc0 2024-03-31T15:08:10.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:10.326+0000 7fd8c5a00640 1 --2- 172.21.15.16:0/3860850172 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd8c006dbc0 0x7fd8c013dc40 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7fd8b400d5f0 tx=0x7fd8b400dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:08:10.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:10.326+0000 7fd8bd600640 1 -- 172.21.15.16:0/3860850172 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8b40125f0 con 0x7fd8c006dbc0 2024-03-31T15:08:10.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:10.326+0000 7fd8bd600640 1 -- 172.21.15.16:0/3860850172 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd8b4019070 con 0x7fd8c006dbc0 2024-03-31T15:08:10.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:10.326+0000 7fd8bd600640 1 -- 172.21.15.16:0/3860850172 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8b40129b0 con 0x7fd8c006dbc0 2024-03-31T15:08:10.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:10.326+0000 7fd8c8400640 1 -- 172.21.15.16:0/3860850172 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd8c0140aa0 con 0x7fd8c006dbc0 2024-03-31T15:08:10.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:10.326+0000 7fd8c8400640 1 -- 172.21.15.16:0/3860850172 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd8c0141060 con 0x7fd8c006dbc0 2024-03-31T15:08:10.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:10.326+0000 7fd8c8400640 1 -- 172.21.15.16:0/3860850172 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd8c0069950 con 0x7fd8c006dbc0 2024-03-31T15:08:10.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:10.326+0000 7fd8bd600640 1 -- 172.21.15.16:0/3860850172 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fd8b4012b50 con 0x7fd8c006dbc0 2024-03-31T15:08:10.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:10.326+0000 7fd8bd600640 1 --2- 172.21.15.16:0/3860850172 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fd8a0068380 0x7fd8a006a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:10.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:10.326+0000 7fd8bd600640 1 -- 172.21.15.16:0/3860850172 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(67..67 src has 1..67) ==== 5709+0+0 (secure 0 0 0) 0x7fd8b4086e90 con 0x7fd8c006dbc0 2024-03-31T15:08:10.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:10.326+0000 7fd8c5000640 1 --2- 172.21.15.16:0/3860850172 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fd8a0068380 0x7fd8a006a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:10.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:10.330+0000 7fd8c5000640 1 --2- 172.21.15.16:0/3860850172 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fd8a0068380 0x7fd8a006a840 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7fd8ac0ad000 tx=0x7fd8ac0a6000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:08:10.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:10.330+0000 7fd8bd600640 1 -- 172.21.15.16:0/3860850172 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fd8b40573c0 con 0x7fd8c006dbc0 2024-03-31T15:08:10.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:10.530+0000 7fd8c8400640 1 -- 172.21.15.16:0/3860850172 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7fd8c00cf030 con 0x7fd8c006dbc0 2024-03-31T15:08:10.528 INFO:tasks.ceph.mon.a.smithi016.stderr:2024-03-31T15:08:10.534+0000 7f7dcbc00640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-31T15:08:10.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:10.534+0000 7fd8bd600640 1 -- 172.21.15.16:0/3860850172 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v46) ==== 161+0+0 (secure 0 0 0) 0x7fd8b4056cc0 con 0x7fd8c006dbc0 2024-03-31T15:08:10.529 INFO:teuthology.orchestra.run.smithi016.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-31T15:08:10.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:10.538+0000 7fd8c8400640 1 -- 172.21.15.16:0/3860850172 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fd8a0068380 msgr2=0x7fd8a006a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:10.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:10.538+0000 7fd8c8400640 1 --2- 172.21.15.16:0/3860850172 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fd8a0068380 0x7fd8a006a840 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7fd8ac0ad000 tx=0x7fd8ac0a6000 comp rx=0 tx=0).stop 2024-03-31T15:08:10.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:10.538+0000 7fd8c8400640 1 -- 172.21.15.16:0/3860850172 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd8c006dbc0 msgr2=0x7fd8c013dc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:10.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:10.538+0000 7fd8c8400640 1 --2- 172.21.15.16:0/3860850172 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd8c006dbc0 0x7fd8c013dc40 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7fd8b400d5f0 tx=0x7fd8b400dab0 comp rx=0 tx=0).stop 2024-03-31T15:08:10.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:10.538+0000 7fd8c8400640 1 -- 172.21.15.16:0/3860850172 shutdown_connections 2024-03-31T15:08:10.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:10.538+0000 7fd8c8400640 1 --2- 172.21.15.16:0/3860850172 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fd8a0068380 0x7fd8a006a840 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:10.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:10.538+0000 7fd8c8400640 1 --2- 172.21.15.16:0/3860850172 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd8c00cb8c0 0x7fd8c00b5f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:10.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:10.538+0000 7fd8c8400640 1 --2- 172.21.15.16:0/3860850172 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fd8c00cb440 0x7fd8c0143c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:10.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:10.538+0000 7fd8c8400640 1 --2- 172.21.15.16:0/3860850172 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd8c006dbc0 0x7fd8c013dc40 secure :-1 s=CLOSED pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7fd8b400d5f0 tx=0x7fd8b400dab0 comp rx=0 tx=0).stop 2024-03-31T15:08:10.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:10.538+0000 7fd8c8400640 1 -- 172.21.15.16:0/3860850172 >> 172.21.15.16:0/3860850172 conn(0x7fd8c0067330 msgr2=0x7fd8c00cdc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:08:10.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:10.538+0000 7fd8c8400640 1 -- 172.21.15.16:0/3860850172 shutdown_connections 2024-03-31T15:08:10.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:10.538+0000 7fd8c8400640 1 -- 172.21.15.16:0/3860850172 wait complete. 2024-03-31T15:08:10.540 INFO:tasks.ceph.mds.a.smithi016.stderr: -3480> 2024-03-31T15:07:59.510+0000 7f3a3d400640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-31T15:08:10.553 DEBUG:teuthology.orchestra.run.smithi016:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata setomapval 10000000000.00000000 file_to_be_damaged_head deadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef 2024-03-31T15:08:10.591 INFO:tasks.ceph.mds.a.smithi016.stdout:starting mds.a at 2024-03-31T15:08:10.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:10.610+0000 7f30c8e13080 1 Processor -- start 2024-03-31T15:08:10.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:10.610+0000 7f30c8e13080 1 -- start start 2024-03-31T15:08:10.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:10.610+0000 7f30c8e13080 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f30ac0022e0 0x55877a36aac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:10.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:10.610+0000 7f30c8e13080 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55877a36e6c0 0x55877a346b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:10.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:10.610+0000 7f30c8e13080 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55877a352ec0 0x55877a324700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:10.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:10.610+0000 7f30c8e13080 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55877a2c94b0 con 0x55877a36e6c0 2024-03-31T15:08:10.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:10.610+0000 7f30c8e13080 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55877a2cd500 con 0x7f30ac0022e0 2024-03-31T15:08:10.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:10.610+0000 7f30c8e13080 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x55877a3fb750 con 0x55877a352ec0 2024-03-31T15:08:10.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:10.610+0000 7f30c5600640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55877a36e6c0 0x55877a346b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:10.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:10.610+0000 7f30c5600640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55877a36e6c0 0x55877a346b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51824/0 (socket says 172.21.15.16:51824) 2024-03-31T15:08:10.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:10.610+0000 7f30c5600640 1 -- 172.21.15.16:0/232022394 learned_addr learned my addr 172.21.15.16:0/232022394 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:08:10.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:10.610+0000 7f30c6a00640 1 --2- 172.21.15.16:0/232022394 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55877a352ec0 0x55877a324700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:10.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:10.610+0000 7f30c6000640 1 --2- 172.21.15.16:0/232022394 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f30ac0022e0 0x55877a36aac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:10.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:10.614+0000 7f30c5600640 1 -- 172.21.15.16:0/232022394 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55877a352ec0 msgr2=0x55877a324700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:08:10.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:10.614+0000 7f30c5600640 1 --2- 172.21.15.16:0/232022394 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55877a352ec0 0x55877a324700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:10.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:10.614+0000 7f30c5600640 1 -- 172.21.15.16:0/232022394 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f30ac0022e0 msgr2=0x55877a36aac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:08:10.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:10.614+0000 7f30c5600640 1 --2- 172.21.15.16:0/232022394 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f30ac0022e0 0x55877a36aac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:10.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:10.614+0000 7f30c5600640 1 -- 172.21.15.16:0/232022394 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55877a3fb9f0 con 0x55877a36e6c0 2024-03-31T15:08:10.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:10.614+0000 7f30c5600640 1 --2- 172.21.15.16:0/232022394 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55877a36e6c0 0x55877a346b40 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7f30b4009690 tx=0x7f30b4009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8dc43538621eb09 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:08:10.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:10.614+0000 7f30bfe00640 1 -- 172.21.15.16:0/232022394 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30b400eb50 con 0x55877a36e6c0 2024-03-31T15:08:10.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:10.614+0000 7f30bfe00640 1 -- 172.21.15.16:0/232022394 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f30b400ecf0 con 0x55877a36e6c0 2024-03-31T15:08:10.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:10.614+0000 7f30bfe00640 1 -- 172.21.15.16:0/232022394 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30b4018640 con 0x55877a36e6c0 2024-03-31T15:08:10.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:10.614+0000 7f30c8e13080 1 -- 172.21.15.16:0/232022394 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55877a36e6c0 msgr2=0x55877a346b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:10.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:10.614+0000 7f30c8e13080 1 --2- 172.21.15.16:0/232022394 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55877a36e6c0 0x55877a346b40 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7f30b4009690 tx=0x7f30b4009b50 comp rx=0 tx=0).stop 2024-03-31T15:08:10.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:10.614+0000 7f30c8e13080 1 -- 172.21.15.16:0/232022394 shutdown_connections 2024-03-31T15:08:10.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:10.614+0000 7f30c8e13080 1 --2- 172.21.15.16:0/232022394 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55877a352ec0 0x55877a324700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:10.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:10.614+0000 7f30c8e13080 1 --2- 172.21.15.16:0/232022394 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55877a36e6c0 0x55877a346b40 unknown :-1 s=CLOSED pgs=876 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:10.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:10.614+0000 7f30c8e13080 1 --2- 172.21.15.16:0/232022394 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f30ac0022e0 0x55877a36aac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:10.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:10.614+0000 7f30c8e13080 1 -- 172.21.15.16:0/232022394 >> 172.21.15.16:0/232022394 conn(0x55877a36c0b0 msgr2=0x55877a3f8040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:08:10.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:10.614+0000 7f30c8e13080 1 -- 172.21.15.16:0/232022394 shutdown_connections 2024-03-31T15:08:10.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:10.614+0000 7f30c8e13080 1 -- 172.21.15.16:0/232022394 wait complete. 2024-03-31T15:08:10.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:10.614+0000 7f30c8e13080 1 Processor -- start 2024-03-31T15:08:10.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:10.614+0000 7f30c8e13080 1 -- start start 2024-03-31T15:08:10.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:10.614+0000 7f30c8e13080 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55877a352ec0 0x55877a323390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:10.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:10.614+0000 7f30c8e13080 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55877a36e6c0 0x55877a3dd8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:10.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:10.614+0000 7f30c8e13080 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f30ac0022e0 0x55877a3dde10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:10.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:10.614+0000 7f30c8e13080 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55877a3fbf30 con 0x7f30ac0022e0 2024-03-31T15:08:10.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:10.614+0000 7f30c8e13080 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55877a3fbc90 con 0x55877a36e6c0 2024-03-31T15:08:10.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:10.614+0000 7f30c8e13080 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x55877a347080 con 0x55877a352ec0 2024-03-31T15:08:10.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:10.614+0000 7f30c6a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f30ac0022e0 0x55877a3dde10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:10.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:10.614+0000 7f30c5600640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55877a36e6c0 0x55877a3dd8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:10.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:10.614+0000 7f30c6000640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55877a352ec0 0x55877a323390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:10.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:10.614+0000 7f30c6a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f30ac0022e0 0x55877a3dde10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51838/0 (socket says 172.21.15.16:51838) 2024-03-31T15:08:10.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:10.614+0000 7f30c5600640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55877a36e6c0 0x55877a3dd8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.16:45614/0 (socket says 172.21.15.16:45614) 2024-03-31T15:08:10.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:10.614+0000 7f30c6000640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55877a352ec0 0x55877a323390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3301/0 says I am v2:172.21.15.16:56136/0 (socket says 172.21.15.16:56136) 2024-03-31T15:08:10.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:10.614+0000 7f30c6a00640 1 -- 172.21.15.16:0/566124587 learned_addr learned my addr 172.21.15.16:0/566124587 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:08:10.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:10.614+0000 7f30c6a00640 1 -- 172.21.15.16:0/566124587 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55877a352ec0 msgr2=0x55877a323390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:10.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:10.614+0000 7f30c6a00640 1 --2- 172.21.15.16:0/566124587 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55877a352ec0 0x55877a323390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:10.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:10.614+0000 7f30c6a00640 1 -- 172.21.15.16:0/566124587 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55877a36e6c0 msgr2=0x55877a3dd8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:10.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:10.614+0000 7f30c6a00640 1 --2- 172.21.15.16:0/566124587 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55877a36e6c0 0x55877a3dd8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:10.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:10.614+0000 7f30c6a00640 1 -- 172.21.15.16:0/566124587 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55877a3de520 con 0x7f30ac0022e0 2024-03-31T15:08:10.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:10.618+0000 7f30c6a00640 1 --2- 172.21.15.16:0/566124587 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f30ac0022e0 0x55877a3dde10 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x55877a36ee00 tx=0x7f30c0010c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:08:10.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:10.618+0000 7f30bea00640 1 -- 172.21.15.16:0/566124587 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30c0006d60 con 0x7f30ac0022e0 2024-03-31T15:08:10.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:10.618+0000 7f30bea00640 1 -- 172.21.15.16:0/566124587 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f30c000e730 con 0x7f30ac0022e0 2024-03-31T15:08:10.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:10.618+0000 7f30bea00640 1 -- 172.21.15.16:0/566124587 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30c000eb20 con 0x7f30ac0022e0 2024-03-31T15:08:10.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:10.618+0000 7f30c8e13080 1 -- 172.21.15.16:0/566124587 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55877a347560 con 0x7f30ac0022e0 2024-03-31T15:08:10.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:10.618+0000 7f30c8e13080 1 -- 172.21.15.16:0/566124587 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55877a3de7c0 con 0x7f30ac0022e0 2024-03-31T15:08:10.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:10.618+0000 7f30bea00640 1 -- 172.21.15.16:0/566124587 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f30c0012040 con 0x7f30ac0022e0 2024-03-31T15:08:10.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:10.618+0000 7f30bea00640 1 --2- 172.21.15.16:0/566124587 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f30a8066d60 0x7f30a8069220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:10.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:10.618+0000 7f30bea00640 1 -- 172.21.15.16:0/566124587 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(68..68 src has 1..68) ==== 5795+0+0 (secure 0 0 0) 0x7f30c0085710 con 0x7f30ac0022e0 2024-03-31T15:08:10.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:10.618+0000 7f30c8e13080 1 --2- 172.21.15.16:0/566124587 >> [v2:172.21.15.89:6800/1856844856,v1:172.21.15.89:6801/1856844856] conn(0x55877a3238d0 0x55877a4950d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:10.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:10.618+0000 7f30c6000640 1 --2- 172.21.15.16:0/566124587 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f30a8066d60 0x7f30a8069220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:10.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:10.618+0000 7f30c8e13080 1 -- 172.21.15.16:0/566124587 --> [v2:172.21.15.89:6800/1856844856,v1:172.21.15.89:6801/1856844856] -- osd_op(unknown.0.0:1 10.10 10:0cf6ad0f:::10000000000.00000000:head [omap-set-vals in=115b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e68) -- 0x55877a347ca0 con 0x55877a3238d0 2024-03-31T15:08:10.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:10.618+0000 7f30c5600640 1 --2- 172.21.15.16:0/566124587 >> [v2:172.21.15.89:6800/1856844856,v1:172.21.15.89:6801/1856844856] conn(0x55877a3238d0 0x55877a4950d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:10.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:10.622+0000 7f30c6000640 1 --2- 172.21.15.16:0/566124587 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f30a8066d60 0x7f30a8069220 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f30b80ad000 tx=0x7f30b8003030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:08:10.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:10.622+0000 7f30c5600640 1 --2- 172.21.15.16:0/566124587 >> [v2:172.21.15.89:6800/1856844856,v1:172.21.15.89:6801/1856844856] conn(0x55877a3238d0 0x55877a4950d0 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:08:10.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:10.622+0000 7f30c5600640 1 -- 172.21.15.16:0/566124587 <== osd.4 v2:172.21.15.89:6800/1856844856 1 ==== osd_op_reply(1 10000000000.00000000 [omap-set-vals] v68'11 uv11 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f30b4015020 con 0x55877a3238d0 2024-03-31T15:08:10.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:10.622+0000 7f30c8e13080 1 -- 172.21.15.16:0/566124587 >> [v2:172.21.15.89:6800/1856844856,v1:172.21.15.89:6801/1856844856] conn(0x55877a3238d0 msgr2=0x55877a4950d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:10.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:10.622+0000 7f30c8e13080 1 --2- 172.21.15.16:0/566124587 >> [v2:172.21.15.89:6800/1856844856,v1:172.21.15.89:6801/1856844856] conn(0x55877a3238d0 0x55877a4950d0 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:10.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:10.622+0000 7f30c8e13080 1 -- 172.21.15.16:0/566124587 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f30a8066d60 msgr2=0x7f30a8069220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:10.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:10.622+0000 7f30c8e13080 1 --2- 172.21.15.16:0/566124587 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f30a8066d60 0x7f30a8069220 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f30b80ad000 tx=0x7f30b8003030 comp rx=0 tx=0).stop 2024-03-31T15:08:10.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:10.622+0000 7f30c8e13080 1 -- 172.21.15.16:0/566124587 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f30ac0022e0 msgr2=0x55877a3dde10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:10.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:10.622+0000 7f30c8e13080 1 --2- 172.21.15.16:0/566124587 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f30ac0022e0 0x55877a3dde10 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x55877a36ee00 tx=0x7f30c0010c80 comp rx=0 tx=0).stop 2024-03-31T15:08:10.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:10.622+0000 7f30c6a00640 1 -- 172.21.15.16:0/566124587 reap_dead start 2024-03-31T15:08:10.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:10.622+0000 7f30c8e13080 1 -- 172.21.15.16:0/566124587 shutdown_connections 2024-03-31T15:08:10.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:10.622+0000 7f30c8e13080 1 -- 172.21.15.16:0/566124587 >> 172.21.15.16:0/566124587 conn(0x55877a36c0b0 msgr2=0x55877a3f9510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:08:10.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:10.622+0000 7f30c8e13080 1 -- 172.21.15.16:0/566124587 shutdown_connections 2024-03-31T15:08:10.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:10.622+0000 7f30c8e13080 1 -- 172.21.15.16:0/566124587 wait complete. 2024-03-31T15:08:10.622 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs joinable true 2024-03-31T15:08:10.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:10.814+0000 7f610d600640 1 Processor -- start 2024-03-31T15:08:10.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:10.814+0000 7f610d600640 1 -- start start 2024-03-31T15:08:10.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:10.818+0000 7f610d600640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f610806c300 0x7f610815c8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:10.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:10.818+0000 7f610d600640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f610815ce20 0x7f61080cda50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:10.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:10.818+0000 7f610d600640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f610815d2c0 con 0x7f610813f290 2024-03-31T15:08:10.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:10.818+0000 7f610d600640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f610815d880 con 0x7f610806c300 2024-03-31T15:08:10.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:10.818+0000 7f610d600640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f610815d5a0 con 0x7f610815ce20 2024-03-31T15:08:10.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:10.818+0000 7f6106a00640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f610813f290 0x7f6108160800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:55548/0 (socket says 172.21.15.16:55548) 2024-03-31T15:08:10.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:10.818+0000 7f6106a00640 1 -- 172.21.15.16:0/4254257108 learned_addr learned my addr 172.21.15.16:0/4254257108 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:08:10.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:10.818+0000 7f60fe000640 1 --2- 172.21.15.16:0/4254257108 >> v2:172.21.15.16:3300/0 conn(0x7f610806c300 0x7f610815c8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:10.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:10.818+0000 7f6107400640 1 --2- 172.21.15.16:0/4254257108 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f610815ce20 0x7f61080cda50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:10.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:10.818+0000 7f60fe000640 1 -- 172.21.15.16:0/4254257108 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f610815ce20 msgr2=0x7f61080cda50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:08:10.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:10.818+0000 7f60fe000640 1 --2- 172.21.15.16:0/4254257108 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f610815ce20 0x7f61080cda50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:10.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:10.818+0000 7f60fe000640 1 -- 172.21.15.16:0/4254257108 >> v1:172.21.15.16:6789/0 conn(0x7f610813f290 legacy=0x7f6108160800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:10.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:10.818+0000 7f60fe000640 1 -- 172.21.15.16:0/4254257108 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6108160000 con 0x7f610806c300 2024-03-31T15:08:10.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:10.818+0000 7f6106000640 1 -- 172.21.15.16:0/4254257108 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1997514124 0 0) 0x7f610815d2c0 con 0x7f610813f290 2024-03-31T15:08:10.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:10.822+0000 7f60fe000640 1 --2- 172.21.15.16:0/4254257108 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f610806c300 0x7f610815c8e0 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7f60f00097c0 tx=0x7f60f001c410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed4cb111d6cb2ac0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:08:10.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:10.822+0000 7f6106000640 1 -- 172.21.15.16:0/4254257108 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60f00b0070 con 0x7f610806c300 2024-03-31T15:08:10.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:10.822+0000 7f6106000640 1 -- 172.21.15.16:0/4254257108 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f60f00ab600 con 0x7f610806c300 2024-03-31T15:08:10.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:10.822+0000 7f6106000640 1 -- 172.21.15.16:0/4254257108 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60f00ab9d0 con 0x7f610806c300 2024-03-31T15:08:10.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:10.822+0000 7f610d600640 1 -- 172.21.15.16:0/4254257108 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f610806c300 msgr2=0x7f610815c8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:10.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:10.822+0000 7f610d600640 1 --2- 172.21.15.16:0/4254257108 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f610806c300 0x7f610815c8e0 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7f60f00097c0 tx=0x7f60f001c410 comp rx=0 tx=0).stop 2024-03-31T15:08:10.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:10.822+0000 7f610d600640 1 -- 172.21.15.16:0/4254257108 shutdown_connections 2024-03-31T15:08:10.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:10.822+0000 7f610d600640 1 --2- 172.21.15.16:0/4254257108 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f610815ce20 0x7f61080cda50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:10.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:10.822+0000 7f610d600640 1 --2- 172.21.15.16:0/4254257108 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f610806c300 0x7f610815c8e0 unknown :-1 s=CLOSED pgs=879 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:10.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:10.822+0000 7f610d600640 1 -- 172.21.15.16:0/4254257108 >> 172.21.15.16:0/4254257108 conn(0x7f61080b2db0 msgr2=0x7f61080cd340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:08:10.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:10.822+0000 7f610d600640 1 -- 172.21.15.16:0/4254257108 shutdown_connections 2024-03-31T15:08:10.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:10.822+0000 7f610d600640 1 -- 172.21.15.16:0/4254257108 wait complete. 2024-03-31T15:08:10.818 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:10.822+0000 7f610d600640 1 Processor -- start 2024-03-31T15:08:10.818 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:10.822+0000 7f610d600640 1 -- start start 2024-03-31T15:08:10.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:10.822+0000 7f610d600640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f610806c300 0x7f61081574a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:10.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:10.826+0000 7f610d600640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f610813f290 0x7f610813b710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:10.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:10.826+0000 7f610d600640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f610815ce20 0x7f610815b510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:10.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:10.826+0000 7f610d600640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f610813f880 con 0x7f610813f290 2024-03-31T15:08:10.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:10.826+0000 7f610d600640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f61080cf160 con 0x7f610806c300 2024-03-31T15:08:10.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:10.826+0000 7f610d600640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f6108169270 con 0x7f610815ce20 2024-03-31T15:08:10.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:10.826+0000 7f60fe000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f610813f290 0x7f610813b710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:10.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:10.826+0000 7f60fe000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f610813f290 0x7f610813b710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51844/0 (socket says 172.21.15.16:51844) 2024-03-31T15:08:10.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:10.826+0000 7f60fe000640 1 -- 172.21.15.16:0/1029056406 learned_addr learned my addr 172.21.15.16:0/1029056406 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:08:10.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:10.826+0000 7f6107400640 1 --2- 172.21.15.16:0/1029056406 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f610815ce20 0x7f610815b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:10.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:10.826+0000 7f6106a00640 1 --2- 172.21.15.16:0/1029056406 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f610806c300 0x7f61081574a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:10.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:10.826+0000 7f60fe000640 1 -- 172.21.15.16:0/1029056406 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f610815ce20 msgr2=0x7f610815b510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:10.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:10.826+0000 7f60fe000640 1 --2- 172.21.15.16:0/1029056406 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f610815ce20 0x7f610815b510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:10.822 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:10.826+0000 7f60fe000640 1 -- 172.21.15.16:0/1029056406 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f610806c300 msgr2=0x7f61081574a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:10.822 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:10.826+0000 7f60fe000640 1 --2- 172.21.15.16:0/1029056406 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f610806c300 0x7f61081574a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:10.822 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:10.826+0000 7f60fe000640 1 -- 172.21.15.16:0/1029056406 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f610813bc50 con 0x7f610813f290 2024-03-31T15:08:10.822 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:10.826+0000 7f6106a00640 1 --2- 172.21.15.16:0/1029056406 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f610806c300 0x7f61081574a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:08:10.822 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:10.826+0000 7f60fe000640 1 --2- 172.21.15.16:0/1029056406 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f610813f290 0x7f610813b710 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7f60f001a040 tx=0x7f60f00191b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:08:10.822 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:10.826+0000 7f60ff400640 1 -- 172.21.15.16:0/1029056406 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60f00b8020 con 0x7f610813f290 2024-03-31T15:08:10.822 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:10.826+0000 7f60ff400640 1 -- 172.21.15.16:0/1029056406 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f60f00b4410 con 0x7f610813f290 2024-03-31T15:08:10.822 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:10.826+0000 7f60ff400640 1 -- 172.21.15.16:0/1029056406 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60f00b0040 con 0x7f610813f290 2024-03-31T15:08:10.822 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:10.826+0000 7f610d600640 1 -- 172.21.15.16:0/1029056406 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6108158460 con 0x7f610813f290 2024-03-31T15:08:10.822 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:10.826+0000 7f610d600640 1 -- 172.21.15.16:0/1029056406 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f61081569c0 con 0x7f610813f290 2024-03-31T15:08:10.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:10.826+0000 7f610d600640 1 -- 172.21.15.16:0/1029056406 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6108069950 con 0x7f610813f290 2024-03-31T15:08:10.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:10.826+0000 7f60ff400640 1 -- 172.21.15.16:0/1029056406 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f60f00bb020 con 0x7f610813f290 2024-03-31T15:08:10.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:10.830+0000 7f60ff400640 1 --2- 172.21.15.16:0/1029056406 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f60e4068380 0x7f60e406a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:10.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:10.830+0000 7f60ff400640 1 -- 172.21.15.16:0/1029056406 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(68..68 src has 1..68) ==== 5795+0+0 (secure 0 0 0) 0x7f60f01205d0 con 0x7f610813f290 2024-03-31T15:08:10.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:10.830+0000 7f6106a00640 1 --2- 172.21.15.16:0/1029056406 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f60e4068380 0x7f60e406a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:10.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:10.830+0000 7f6106a00640 1 --2- 172.21.15.16:0/1029056406 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f60e4068380 0x7f60e406a840 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f60f8000f70 tx=0x7f60f8007c40 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:08:10.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:10.834+0000 7f60ff400640 1 -- 172.21.15.16:0/1029056406 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f60f00c2030 con 0x7f610813f290 2024-03-31T15:08:11.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:11.034+0000 7f610d600640 1 -- 172.21.15.16:0/1029056406 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"} v 0) -- 0x7f610806c030 con 0x7f610813f290 2024-03-31T15:08:11.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:11.254+0000 7f60ff400640 1 -- 172.21.15.16:0/1029056406 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]=0 cephfs marked joinable; MDS may join as newly active. v47) ==== 162+0+0 (secure 0 0 0) 0x7f60f0016090 con 0x7f610813f290 2024-03-31T15:08:11.249 INFO:teuthology.orchestra.run.smithi016.stderr:cephfs marked joinable; MDS may join as newly active. 2024-03-31T15:08:11.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:11.258+0000 7f610d600640 1 -- 172.21.15.16:0/1029056406 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f60e4068380 msgr2=0x7f60e406a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:11.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:11.258+0000 7f610d600640 1 --2- 172.21.15.16:0/1029056406 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f60e4068380 0x7f60e406a840 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f60f8000f70 tx=0x7f60f8007c40 comp rx=0 tx=0).stop 2024-03-31T15:08:11.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:11.258+0000 7f610d600640 1 -- 172.21.15.16:0/1029056406 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f610813f290 msgr2=0x7f610813b710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:11.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:11.258+0000 7f610d600640 1 --2- 172.21.15.16:0/1029056406 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f610813f290 0x7f610813b710 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7f60f001a040 tx=0x7f60f00191b0 comp rx=0 tx=0).stop 2024-03-31T15:08:11.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:11.258+0000 7f610d600640 1 -- 172.21.15.16:0/1029056406 shutdown_connections 2024-03-31T15:08:11.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:11.258+0000 7f610d600640 1 --2- 172.21.15.16:0/1029056406 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f60e4068380 0x7f60e406a840 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:11.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:11.258+0000 7f610d600640 1 --2- 172.21.15.16:0/1029056406 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f610815ce20 0x7f610815b510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:11.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:11.258+0000 7f610d600640 1 --2- 172.21.15.16:0/1029056406 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f610813f290 0x7f610813b710 unknown :-1 s=CLOSED pgs=880 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:11.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:11.258+0000 7f610d600640 1 --2- 172.21.15.16:0/1029056406 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f610806c300 0x7f61081574a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:11.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:11.258+0000 7f610d600640 1 -- 172.21.15.16:0/1029056406 >> 172.21.15.16:0/1029056406 conn(0x7f61080b2db0 msgr2=0x7f6108165b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:08:11.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:11.258+0000 7f610d600640 1 -- 172.21.15.16:0/1029056406 shutdown_connections 2024-03-31T15:08:11.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:11.258+0000 7f610d600640 1 -- 172.21.15.16:0/1029056406 wait complete. 2024-03-31T15:08:11.276 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-31T15:08:11.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:11.422+0000 7fbed1600640 1 Processor -- start 2024-03-31T15:08:11.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:11.422+0000 7fbed1600640 1 -- start start 2024-03-31T15:08:11.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:11.422+0000 7fbed1600640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fbecc147e70 0x7fbecc06b620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:11.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:11.422+0000 7fbed1600640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fbecc0d0c50 0x7fbecc072fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:11.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:11.422+0000 7fbed1600640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbecc1621b0 con 0x7fbecc073510 2024-03-31T15:08:11.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:11.422+0000 7fbed1600640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7fbecc0739a0 con 0x7fbecc0d0c50 2024-03-31T15:08:11.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:11.422+0000 7fbed1600640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fbecc162490 con 0x7fbecc147e70 2024-03-31T15:08:11.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:11.422+0000 7fbecb400640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fbecc073510 0x7fbecc0cdab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:55550/0 (socket says 172.21.15.16:55550) 2024-03-31T15:08:11.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:11.422+0000 7fbecb400640 1 -- 172.21.15.16:0/3994397883 learned_addr learned my addr 172.21.15.16:0/3994397883 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:08:11.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:11.422+0000 7fbeca000640 1 --2- 172.21.15.16:0/3994397883 >> v2:172.21.15.89:3300/0 conn(0x7fbecc0d0c50 0x7fbecc072fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:11.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:11.422+0000 7fbecaa00640 1 --2- 172.21.15.16:0/3994397883 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fbecc147e70 0x7fbecc06b620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:11.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:11.422+0000 7fbec9600640 1 -- 172.21.15.16:0/3994397883 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2977096380 0 0) 0x7fbecc1621b0 con 0x7fbecc073510 2024-03-31T15:08:11.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:11.422+0000 7fbec9600640 1 -- 172.21.15.16:0/3994397883 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbeb0003530 con 0x7fbecc073510 2024-03-31T15:08:11.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:11.422+0000 7fbec9600640 1 -- 172.21.15.16:0/3994397883 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2872844882 0 0) 0x7fbeb0003530 con 0x7fbecc073510 2024-03-31T15:08:11.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:11.422+0000 7fbec9600640 1 -- 172.21.15.16:0/3994397883 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fbecc147e70 msgr2=0x7fbecc06b620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:08:11.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:11.422+0000 7fbec9600640 1 --2- 172.21.15.16:0/3994397883 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fbecc147e70 0x7fbecc06b620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:11.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:11.422+0000 7fbec9600640 1 -- 172.21.15.16:0/3994397883 >> v2:172.21.15.89:3300/0 conn(0x7fbecc0d0c50 msgr2=0x7fbecc072fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:08:11.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:11.422+0000 7fbec9600640 1 --2- 172.21.15.16:0/3994397883 >> v2:172.21.15.89:3300/0 conn(0x7fbecc0d0c50 0x7fbecc072fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:11.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:11.422+0000 7fbec9600640 1 -- 172.21.15.16:0/3994397883 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbecc162770 con 0x7fbecc073510 2024-03-31T15:08:11.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:11.422+0000 7fbec9600640 1 -- 172.21.15.16:0/3994397883 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 214807623 0 0) 0x7fbec4001a00 con 0x7fbecc073510 2024-03-31T15:08:11.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:11.422+0000 7fbec9600640 1 -- 172.21.15.16:0/3994397883 >> v1:172.21.15.16:6789/0 conn(0x7fbecc073510 legacy=0x7fbecc0cdab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:11.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:11.422+0000 7fbec9600640 1 --2- 172.21.15.16:0/3994397883 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbeb00039c0 0x7fbeb0003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:11.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:11.422+0000 7fbec9600640 1 --2- 172.21.15.16:0/3994397883 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbeb0004810 0x7fbeb0004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:11.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:11.422+0000 7fbec9600640 1 --2- 172.21.15.16:0/3994397883 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fbecc147e70 0x7fbeb00051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:11.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:11.422+0000 7fbec9600640 1 -- 172.21.15.16:0/3994397883 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fbecc0739a0 con 0x7fbeb00039c0 2024-03-31T15:08:11.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:11.422+0000 7fbec9600640 1 -- 172.21.15.16:0/3994397883 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fbecc162490 con 0x7fbeb0004810 2024-03-31T15:08:11.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:11.422+0000 7fbec9600640 1 -- 172.21.15.16:0/3994397883 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fbeb000b9d0 con 0x7fbecc147e70 2024-03-31T15:08:11.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:11.422+0000 7fbecaa00640 1 --2- 172.21.15.16:0/3994397883 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbeb00039c0 0x7fbeb0003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:11.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:11.422+0000 7fbeca000640 1 --2- 172.21.15.16:0/3994397883 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbeb0004810 0x7fbeb0004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:11.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:11.422+0000 7fbecb400640 1 --2- 172.21.15.16:0/3994397883 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fbecc147e70 0x7fbeb00051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:11.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:11.422+0000 7fbecaa00640 1 -- 172.21.15.16:0/3994397883 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fbecc147e70 msgr2=0x7fbeb00051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:08:11.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:11.422+0000 7fbecaa00640 1 --2- 172.21.15.16:0/3994397883 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fbecc147e70 0x7fbeb00051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:11.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:11.422+0000 7fbecaa00640 1 -- 172.21.15.16:0/3994397883 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbeb0004810 msgr2=0x7fbeb0004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:08:11.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:11.422+0000 7fbecaa00640 1 --2- 172.21.15.16:0/3994397883 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbeb0004810 0x7fbeb0004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:11.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:11.422+0000 7fbecaa00640 1 -- 172.21.15.16:0/3994397883 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbeb0003530 con 0x7fbeb00039c0 2024-03-31T15:08:11.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:11.426+0000 7fbecaa00640 1 --2- 172.21.15.16:0/3994397883 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbeb00039c0 0x7fbeb0003de0 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7fbebc009ae0 tx=0x7fbebc01ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e8230aff08c83d2 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:08:11.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:11.426+0000 7fbec9600640 1 -- 172.21.15.16:0/3994397883 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbebc0096b0 con 0x7fbeb00039c0 2024-03-31T15:08:11.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:11.426+0000 7fbec9600640 1 -- 172.21.15.16:0/3994397883 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbebc0a9470 con 0x7fbeb00039c0 2024-03-31T15:08:11.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:11.426+0000 7fbec9600640 1 -- 172.21.15.16:0/3994397883 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbebc0a9860 con 0x7fbeb00039c0 2024-03-31T15:08:11.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:11.426+0000 7fbed1600640 1 -- 172.21.15.16:0/3994397883 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbeb00039c0 msgr2=0x7fbeb0003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:11.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:11.426+0000 7fbed1600640 1 --2- 172.21.15.16:0/3994397883 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbeb00039c0 0x7fbeb0003de0 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7fbebc009ae0 tx=0x7fbebc01ca10 comp rx=0 tx=0).stop 2024-03-31T15:08:11.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:11.426+0000 7fbecb400640 1 -- 172.21.15.16:0/3994397883 reap_dead start 2024-03-31T15:08:11.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:11.426+0000 7fbed1600640 1 -- 172.21.15.16:0/3994397883 shutdown_connections 2024-03-31T15:08:11.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:11.426+0000 7fbed1600640 1 -- 172.21.15.16:0/3994397883 >> 172.21.15.16:0/3994397883 conn(0x7fbecc0d1060 msgr2=0x7fbecc0d1470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:08:11.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:11.426+0000 7fbed1600640 1 -- 172.21.15.16:0/3994397883 shutdown_connections 2024-03-31T15:08:11.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:11.426+0000 7fbed1600640 1 -- 172.21.15.16:0/3994397883 wait complete. 2024-03-31T15:08:11.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:11.430+0000 7fbed1600640 1 Processor -- start 2024-03-31T15:08:11.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:11.430+0000 7fbed1600640 1 -- start start 2024-03-31T15:08:11.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:11.430+0000 7fbed1600640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbeb00039c0 0x7fbecc15dab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:11.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:11.430+0000 7fbed1600640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fbecc0b3d10 0x7fbecc0b4190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:11.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:11.430+0000 7fbed1600640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbecc0afbd0 0x7fbecc0b0030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:11.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:11.430+0000 7fbed1600640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fbecc0ce340 con 0x7fbeb00039c0 2024-03-31T15:08:11.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:11.430+0000 7fbed1600640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fbecc0ce1c0 con 0x7fbecc0afbd0 2024-03-31T15:08:11.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:11.430+0000 7fbed1600640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fbecc0ce4c0 con 0x7fbecc0b3d10 2024-03-31T15:08:11.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:11.430+0000 7fbecaa00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbeb00039c0 0x7fbecc15dab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:11.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:11.430+0000 7fbecaa00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbeb00039c0 0x7fbecc15dab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51866/0 (socket says 172.21.15.16:51866) 2024-03-31T15:08:11.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:11.430+0000 7fbecaa00640 1 -- 172.21.15.16:0/2822933962 learned_addr learned my addr 172.21.15.16:0/2822933962 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:08:11.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:11.430+0000 7fbecb400640 1 --2- 172.21.15.16:0/2822933962 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbecc0afbd0 0x7fbecc0b0030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:11.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:11.430+0000 7fbeca000640 1 --2- 172.21.15.16:0/2822933962 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fbecc0b3d10 0x7fbecc0b4190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:11.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:11.430+0000 7fbecaa00640 1 -- 172.21.15.16:0/2822933962 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fbecc0b3d10 msgr2=0x7fbecc0b4190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:11.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:11.430+0000 7fbecaa00640 1 --2- 172.21.15.16:0/2822933962 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fbecc0b3d10 0x7fbecc0b4190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:11.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:11.430+0000 7fbecaa00640 1 -- 172.21.15.16:0/2822933962 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbecc0afbd0 msgr2=0x7fbecc0b0030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:11.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:11.430+0000 7fbecaa00640 1 --2- 172.21.15.16:0/2822933962 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbecc0afbd0 0x7fbecc0b0030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:11.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:11.430+0000 7fbecaa00640 1 -- 172.21.15.16:0/2822933962 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbecc0b4790 con 0x7fbeb00039c0 2024-03-31T15:08:11.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:11.430+0000 7fbecaa00640 1 --2- 172.21.15.16:0/2822933962 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbeb00039c0 0x7fbecc15dab0 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7fbebc01d2b0 tx=0x7fbeb0005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:08:11.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:11.430+0000 7fbec2a00640 1 -- 172.21.15.16:0/2822933962 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbebc0096b0 con 0x7fbeb00039c0 2024-03-31T15:08:11.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:11.430+0000 7fbec2a00640 1 -- 172.21.15.16:0/2822933962 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbebc00f000 con 0x7fbeb00039c0 2024-03-31T15:08:11.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:11.430+0000 7fbed1600640 1 -- 172.21.15.16:0/2822933962 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbecc0b4a70 con 0x7fbeb00039c0 2024-03-31T15:08:11.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:11.430+0000 7fbec2a00640 1 -- 172.21.15.16:0/2822933962 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbebc0a9e00 con 0x7fbeb00039c0 2024-03-31T15:08:11.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:11.430+0000 7fbed1600640 1 -- 172.21.15.16:0/2822933962 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbecc0b5030 con 0x7fbeb00039c0 2024-03-31T15:08:11.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:11.434+0000 7fbed1600640 1 -- 172.21.15.16:0/2822933962 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbecc069950 con 0x7fbeb00039c0 2024-03-31T15:08:11.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:11.434+0000 7fbec2a00640 1 -- 172.21.15.16:0/2822933962 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fbebc0c3400 con 0x7fbeb00039c0 2024-03-31T15:08:11.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:11.434+0000 7fbec2a00640 1 --2- 172.21.15.16:0/2822933962 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fbeac068380 0x7fbeac06a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:11.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:11.434+0000 7fbec2a00640 1 -- 172.21.15.16:0/2822933962 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(68..68 src has 1..68) ==== 5795+0+0 (secure 0 0 0) 0x7fbebc120d00 con 0x7fbeb00039c0 2024-03-31T15:08:11.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:11.434+0000 7fbeca000640 1 --2- 172.21.15.16:0/2822933962 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fbeac068380 0x7fbeac06a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:11.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:11.434+0000 7fbeca000640 1 --2- 172.21.15.16:0/2822933962 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fbeac068380 0x7fbeac06a840 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fbecc0d1030 tx=0x7fbeb8009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:08:11.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:11.438+0000 7fbec2a00640 1 -- 172.21.15.16:0/2822933962 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fbebc0c8060 con 0x7fbeb00039c0 2024-03-31T15:08:11.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:11.638+0000 7fbed1600640 1 -- 172.21.15.16:0/2822933962 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbecc0d1bf0 con 0x7fbeb00039c0 2024-03-31T15:08:11.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:11.638+0000 7fbec2a00640 1 -- 172.21.15.16:0/2822933962 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 47 v47) ==== 94+0+5177 (secure 0 0 0) 0x7fbebc0ad020 con 0x7fbeb00039c0 2024-03-31T15:08:11.635 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:08:11.635 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":47,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6836/4188793506","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":4188793506},{"type":"v1","addr":"172.21.15.89:6836","nonce":4188793506}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":40},{"gid":5942,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/4105657676","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6835","nonce":4105657676},{"type":"v1","addr":"172.21.15.89:6837","nonce":4105657676}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":40},{"gid":6592,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6833/3986752851","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":3986752851},{"type":"v1","addr":"172.21.15.16:6833","nonce":3986752851}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":39}],"filesystems":[{"mdsmap":{"epoch":47,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-31T15:07:56.938140+0000","modified":"2024-03-31T15:08:11.257593+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":68,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6829},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6829":{"gid":6829,"name":"a","rank":0,"incarnation":47,"state":"up:replay","state_seq":1,"addr":"172.21.15.16:6835/2645389320","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":2645389320},{"type":"v1","addr":"172.21.15.16:6835","nonce":2645389320}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":5}]} 2024-03-31T15:08:11.636 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 47 2024-03-31T15:08:11.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:11.642+0000 7fbed1600640 1 -- 172.21.15.16:0/2822933962 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fbeac068380 msgr2=0x7fbeac06a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:11.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:11.642+0000 7fbed1600640 1 --2- 172.21.15.16:0/2822933962 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fbeac068380 0x7fbeac06a840 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fbecc0d1030 tx=0x7fbeb8009190 comp rx=0 tx=0).stop 2024-03-31T15:08:11.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:11.642+0000 7fbed1600640 1 -- 172.21.15.16:0/2822933962 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbeb00039c0 msgr2=0x7fbecc15dab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:11.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:11.642+0000 7fbed1600640 1 --2- 172.21.15.16:0/2822933962 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbeb00039c0 0x7fbecc15dab0 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7fbebc01d2b0 tx=0x7fbeb0005840 comp rx=0 tx=0).stop 2024-03-31T15:08:11.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:11.642+0000 7fbed1600640 1 -- 172.21.15.16:0/2822933962 shutdown_connections 2024-03-31T15:08:11.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:11.642+0000 7fbed1600640 1 --2- 172.21.15.16:0/2822933962 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fbeac068380 0x7fbeac06a840 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:11.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:11.642+0000 7fbed1600640 1 --2- 172.21.15.16:0/2822933962 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbecc0afbd0 0x7fbecc0b0030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:11.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:11.642+0000 7fbed1600640 1 --2- 172.21.15.16:0/2822933962 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fbecc0b3d10 0x7fbecc0b4190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:11.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:11.642+0000 7fbed1600640 1 --2- 172.21.15.16:0/2822933962 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbeb00039c0 0x7fbecc15dab0 unknown :-1 s=CLOSED pgs=883 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:11.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:11.642+0000 7fbed1600640 1 -- 172.21.15.16:0/2822933962 >> 172.21.15.16:0/2822933962 conn(0x7fbecc0d1060 msgr2=0x7fbecc070760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:08:11.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:11.642+0000 7fbed1600640 1 -- 172.21.15.16:0/2822933962 shutdown_connections 2024-03-31T15:08:11.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:11.642+0000 7fbed1600640 1 -- 172.21.15.16:0/2822933962 wait complete. 2024-03-31T15:08:12.059 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 47, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-31T15:07:56.938140+0000', 'modified': '2024-03-31T15:08:11.257593+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 68, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6829}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6829': {'gid': 6829, 'name': 'a', 'rank': 0, 'incarnation': 47, 'state': 'up:replay', 'state_seq': 1, 'addr': '172.21.15.16:6835/2645389320', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6834', 'nonce': 2645389320}, {'type': 'v1', 'addr': '172.21.15.16:6835', 'nonce': 2645389320}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [11], 'metadata_pool': 10, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-03-31T15:08:12.060 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6829:up:replay 2024-03-31T15:08:13.060 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-31T15:08:13.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:13.218+0000 7f0c6b200640 1 Processor -- start 2024-03-31T15:08:13.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:13.218+0000 7f0c6b200640 1 -- start start 2024-03-31T15:08:13.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:13.218+0000 7f0c6b200640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f0c6415b3d0 0x7f0c640b1cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:13.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:13.218+0000 7f0c6b200640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f0c640ce2f0 0x7f0c640b2200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:13.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:13.218+0000 7f0c6b200640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0c6415f3c0 con 0x7f0c6415eee0 2024-03-31T15:08:13.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:13.218+0000 7f0c6b200640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f0c6415b7b0 con 0x7f0c6415b3d0 2024-03-31T15:08:13.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:13.218+0000 7f0c6b200640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f0c6415f6a0 con 0x7f0c640ce2f0 2024-03-31T15:08:13.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:13.218+0000 7f0c63e00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f0c6415b3d0 0x7f0c640b1cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:13.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:13.218+0000 7f0c5b400640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f0c6415eee0 0x7f0c640cc9c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:55554/0 (socket says 172.21.15.16:55554) 2024-03-31T15:08:13.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:13.218+0000 7f0c5b400640 1 -- 172.21.15.16:0/261235984 learned_addr learned my addr 172.21.15.16:0/261235984 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:08:13.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:13.218+0000 7f0c69200640 1 --2- 172.21.15.16:0/261235984 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f0c640ce2f0 0x7f0c640b2200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:13.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:13.218+0000 7f0c63e00640 1 -- 172.21.15.16:0/261235984 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f0c640ce2f0 msgr2=0x7f0c640b2200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:08:13.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:13.218+0000 7f0c63e00640 1 --2- 172.21.15.16:0/261235984 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f0c640ce2f0 0x7f0c640b2200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:13.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:13.218+0000 7f0c63e00640 1 -- 172.21.15.16:0/261235984 >> v1:172.21.15.16:6789/0 conn(0x7f0c6415eee0 legacy=0x7f0c640cc9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:13.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:13.218+0000 7f0c63e00640 1 -- 172.21.15.16:0/261235984 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c6415f980 con 0x7f0c6415b3d0 2024-03-31T15:08:13.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:13.222+0000 7f0c63e00640 1 --2- 172.21.15.16:0/261235984 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0c6415b3d0 0x7f0c640b1cc0 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7f0c540097c0 tx=0x7f0c5401c470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7760af92fc0461f server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:08:13.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:13.222+0000 7f0c63400640 1 -- 172.21.15.16:0/261235984 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c540ab430 con 0x7f0c6415b3d0 2024-03-31T15:08:13.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:13.222+0000 7f0c63400640 1 -- 172.21.15.16:0/261235984 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0c540b0070 con 0x7f0c6415b3d0 2024-03-31T15:08:13.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:13.222+0000 7f0c63400640 1 -- 172.21.15.16:0/261235984 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c540ab7f0 con 0x7f0c6415b3d0 2024-03-31T15:08:13.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:13.222+0000 7f0c6b200640 1 -- 172.21.15.16:0/261235984 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0c6415b3d0 msgr2=0x7f0c640b1cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:13.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:13.222+0000 7f0c6b200640 1 --2- 172.21.15.16:0/261235984 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0c6415b3d0 0x7f0c640b1cc0 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7f0c540097c0 tx=0x7f0c5401c470 comp rx=0 tx=0).stop 2024-03-31T15:08:13.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:13.222+0000 7f0c6b200640 1 -- 172.21.15.16:0/261235984 shutdown_connections 2024-03-31T15:08:13.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:13.222+0000 7f0c6b200640 1 --2- 172.21.15.16:0/261235984 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f0c640ce2f0 0x7f0c640b2200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:13.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:13.222+0000 7f0c6b200640 1 --2- 172.21.15.16:0/261235984 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0c6415b3d0 0x7f0c640b1cc0 unknown :-1 s=CLOSED pgs=885 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:13.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:13.222+0000 7f0c6b200640 1 -- 172.21.15.16:0/261235984 >> 172.21.15.16:0/261235984 conn(0x7f0c6406b620 msgr2=0x7f0c6406ba30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:08:13.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:13.222+0000 7f0c6b200640 1 -- 172.21.15.16:0/261235984 shutdown_connections 2024-03-31T15:08:13.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:13.222+0000 7f0c6b200640 1 -- 172.21.15.16:0/261235984 wait complete. 2024-03-31T15:08:13.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:13.226+0000 7f0c6b200640 1 Processor -- start 2024-03-31T15:08:13.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:13.226+0000 7f0c6b200640 1 -- start start 2024-03-31T15:08:13.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:13.226+0000 7f0c6b200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0c640ce2f0 0x7f0c64147320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:13.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:13.226+0000 7f0c6b200640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f0c6415b3d0 0x7f0c640b0160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:13.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:13.226+0000 7f0c6b200640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0c6415eee0 0x7f0c640b3700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:13.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:13.226+0000 7f0c6b200640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f0c64168610 con 0x7f0c640ce2f0 2024-03-31T15:08:13.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:13.226+0000 7f0c6b200640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f0c64168490 con 0x7f0c6415eee0 2024-03-31T15:08:13.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:13.226+0000 7f0c6b200640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f0c64168790 con 0x7f0c6415b3d0 2024-03-31T15:08:13.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:13.226+0000 7f0c63e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0c640ce2f0 0x7f0c64147320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:13.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:13.226+0000 7f0c63e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0c640ce2f0 0x7f0c64147320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51892/0 (socket says 172.21.15.16:51892) 2024-03-31T15:08:13.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:13.226+0000 7f0c63e00640 1 -- 172.21.15.16:0/699273530 learned_addr learned my addr 172.21.15.16:0/699273530 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:08:13.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:13.226+0000 7f0c63e00640 1 -- 172.21.15.16:0/699273530 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f0c6415b3d0 msgr2=0x7f0c640b0160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:13.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:13.226+0000 7f0c5b400640 1 --2- 172.21.15.16:0/699273530 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f0c6415b3d0 0x7f0c640b0160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:13.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:13.226+0000 7f0c69200640 1 --2- 172.21.15.16:0/699273530 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0c6415eee0 0x7f0c640b3700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:13.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:13.226+0000 7f0c63e00640 1 --2- 172.21.15.16:0/699273530 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f0c6415b3d0 0x7f0c640b0160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:13.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:13.226+0000 7f0c63e00640 1 -- 172.21.15.16:0/699273530 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0c6415eee0 msgr2=0x7f0c640b3700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:13.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:13.226+0000 7f0c63e00640 1 --2- 172.21.15.16:0/699273530 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0c6415eee0 0x7f0c640b3700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:13.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:13.226+0000 7f0c63e00640 1 -- 172.21.15.16:0/699273530 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c640b2c20 con 0x7f0c640ce2f0 2024-03-31T15:08:13.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:13.226+0000 7f0c5b400640 1 --2- 172.21.15.16:0/699273530 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f0c6415b3d0 0x7f0c640b0160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:08:13.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:13.226+0000 7f0c69200640 1 --2- 172.21.15.16:0/699273530 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0c6415eee0 0x7f0c640b3700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:08:13.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:13.226+0000 7f0c63e00640 1 --2- 172.21.15.16:0/699273530 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0c640ce2f0 0x7f0c64147320 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7f0c540ad040 tx=0x7f0c54019210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:08:13.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:13.226+0000 7f0c60c00640 1 -- 172.21.15.16:0/699273530 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c540b43f0 con 0x7f0c640ce2f0 2024-03-31T15:08:13.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:13.226+0000 7f0c60c00640 1 -- 172.21.15.16:0/699273530 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0c540bd070 con 0x7f0c640ce2f0 2024-03-31T15:08:13.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:13.226+0000 7f0c60c00640 1 -- 172.21.15.16:0/699273530 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c540b4810 con 0x7f0c640ce2f0 2024-03-31T15:08:13.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:13.226+0000 7f0c6b200640 1 -- 172.21.15.16:0/699273530 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0c64147860 con 0x7f0c640ce2f0 2024-03-31T15:08:13.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:13.226+0000 7f0c6b200640 1 -- 172.21.15.16:0/699273530 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0c64159d30 con 0x7f0c640ce2f0 2024-03-31T15:08:13.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:13.230+0000 7f0c6b200640 1 -- 172.21.15.16:0/699273530 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0c64069950 con 0x7f0c640ce2f0 2024-03-31T15:08:13.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:13.230+0000 7f0c60c00640 1 -- 172.21.15.16:0/699273530 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f0c540b49b0 con 0x7f0c640ce2f0 2024-03-31T15:08:13.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:13.230+0000 7f0c60c00640 1 --2- 172.21.15.16:0/699273530 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f0c40068380 0x7f0c4006a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:13.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:13.230+0000 7f0c60c00640 1 -- 172.21.15.16:0/699273530 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(68..68 src has 1..68) ==== 5795+0+0 (secure 0 0 0) 0x7f0c541209c0 con 0x7f0c640ce2f0 2024-03-31T15:08:13.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:13.230+0000 7f0c5b400640 1 --2- 172.21.15.16:0/699273530 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f0c40068380 0x7f0c4006a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:13.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:13.230+0000 7f0c5b400640 1 --2- 172.21.15.16:0/699273530 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f0c40068380 0x7f0c4006a840 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f0c64165530 tx=0x7f0c4c009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:08:13.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:13.234+0000 7f0c60c00640 1 -- 172.21.15.16:0/699273530 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f0c54122e20 con 0x7f0c640ce2f0 2024-03-31T15:08:13.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:13.434+0000 7f0c6b200640 1 -- 172.21.15.16:0/699273530 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0c640cdea0 con 0x7f0c640ce2f0 2024-03-31T15:08:13.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:13.434+0000 7f0c60c00640 1 -- 172.21.15.16:0/699273530 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) ==== 94+0+5177 (secure 0 0 0) 0x7f0c540c7020 con 0x7f0c640ce2f0 2024-03-31T15:08:13.431 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:08:13.431 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":49,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6836/4188793506","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":4188793506},{"type":"v1","addr":"172.21.15.89:6836","nonce":4188793506}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":40},{"gid":5942,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/4105657676","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6835","nonce":4105657676},{"type":"v1","addr":"172.21.15.89:6837","nonce":4105657676}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":40},{"gid":6592,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6833/3986752851","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":3986752851},{"type":"v1","addr":"172.21.15.16:6833","nonce":3986752851}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":39}],"filesystems":[{"mdsmap":{"epoch":49,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-31T15:07:56.938140+0000","modified":"2024-03-31T15:08:12.264563+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":68,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6829},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6829":{"gid":6829,"name":"a","rank":0,"incarnation":47,"state":"up:rejoin","state_seq":3,"addr":"172.21.15.16:6835/2645389320","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":2645389320},{"type":"v1","addr":"172.21.15.16:6835","nonce":2645389320}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":5}]} 2024-03-31T15:08:13.431 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 49 2024-03-31T15:08:13.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:13.438+0000 7f0c6b200640 1 -- 172.21.15.16:0/699273530 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f0c40068380 msgr2=0x7f0c4006a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:13.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:13.438+0000 7f0c6b200640 1 --2- 172.21.15.16:0/699273530 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f0c40068380 0x7f0c4006a840 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f0c64165530 tx=0x7f0c4c009040 comp rx=0 tx=0).stop 2024-03-31T15:08:13.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:13.438+0000 7f0c6b200640 1 -- 172.21.15.16:0/699273530 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0c640ce2f0 msgr2=0x7f0c64147320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:13.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:13.438+0000 7f0c6b200640 1 --2- 172.21.15.16:0/699273530 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0c640ce2f0 0x7f0c64147320 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7f0c540ad040 tx=0x7f0c54019210 comp rx=0 tx=0).stop 2024-03-31T15:08:13.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:13.438+0000 7f0c6b200640 1 -- 172.21.15.16:0/699273530 shutdown_connections 2024-03-31T15:08:13.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:13.438+0000 7f0c6b200640 1 --2- 172.21.15.16:0/699273530 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f0c40068380 0x7f0c4006a840 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:13.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:13.438+0000 7f0c6b200640 1 --2- 172.21.15.16:0/699273530 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0c6415eee0 0x7f0c640b3700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:13.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:13.438+0000 7f0c6b200640 1 --2- 172.21.15.16:0/699273530 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f0c6415b3d0 0x7f0c640b0160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:13.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:13.438+0000 7f0c6b200640 1 --2- 172.21.15.16:0/699273530 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0c640ce2f0 0x7f0c64147320 unknown :-1 s=CLOSED pgs=886 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:13.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:13.438+0000 7f0c6b200640 1 -- 172.21.15.16:0/699273530 >> 172.21.15.16:0/699273530 conn(0x7f0c6406b620 msgr2=0x7f0c64151a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:08:13.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:13.438+0000 7f0c6b200640 1 -- 172.21.15.16:0/699273530 shutdown_connections 2024-03-31T15:08:13.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:13.438+0000 7f0c6b200640 1 -- 172.21.15.16:0/699273530 wait complete. 2024-03-31T15:08:13.853 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 49, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-31T15:07:56.938140+0000', 'modified': '2024-03-31T15:08:12.264563+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 68, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6829}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6829': {'gid': 6829, 'name': 'a', 'rank': 0, 'incarnation': 47, 'state': 'up:rejoin', 'state_seq': 3, 'addr': '172.21.15.16:6835/2645389320', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6834', 'nonce': 2645389320}, {'type': 'v1', 'addr': '172.21.15.16:6835', 'nonce': 2645389320}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [11], 'metadata_pool': 10, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-03-31T15:08:13.854 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6829:up:rejoin 2024-03-31T15:08:14.854 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-31T15:08:14.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.002+0000 7fa54f000640 1 Processor -- start 2024-03-31T15:08:14.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.002+0000 7fa54f000640 1 -- start start 2024-03-31T15:08:15.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.006+0000 7fa54f000640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fa54809b170 0x7fa5480ccb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:15.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.006+0000 7fa54f000640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fa548070d50 0x7fa5480cd0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:15.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.006+0000 7fa54f000640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa5480d0800 con 0x7fa54815e3b0 2024-03-31T15:08:15.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.006+0000 7fa54f000640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7fa54815e790 con 0x7fa548070d50 2024-03-31T15:08:15.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.006+0000 7fa54f000640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fa54806f5d0 con 0x7fa54809b170 2024-03-31T15:08:15.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.006+0000 7fa54dc00640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fa54815e3b0 0x7fa54809aa60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:55562/0 (socket says 172.21.15.16:55562) 2024-03-31T15:08:15.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.006+0000 7fa54dc00640 1 -- 172.21.15.16:0/1989092331 learned_addr learned my addr 172.21.15.16:0/1989092331 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:08:15.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.006+0000 7fa54e600640 1 --2- 172.21.15.16:0/1989092331 >> v2:172.21.15.16:3300/0 conn(0x7fa548070d50 0x7fa5480cd0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:15.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.006+0000 7fa54d200640 1 --2- 172.21.15.16:0/1989092331 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fa54809b170 0x7fa5480ccb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:15.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.006+0000 7fa547e00640 1 -- 172.21.15.16:0/1989092331 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4217137709 0 0) 0x7fa5480d0800 con 0x7fa54815e3b0 2024-03-31T15:08:15.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.006+0000 7fa547e00640 1 -- 172.21.15.16:0/1989092331 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa530003530 con 0x7fa54815e3b0 2024-03-31T15:08:15.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.006+0000 7fa54e600640 1 -- 172.21.15.16:0/1989092331 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fa54809b170 msgr2=0x7fa5480ccb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:08:15.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.006+0000 7fa54e600640 1 --2- 172.21.15.16:0/1989092331 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fa54809b170 0x7fa5480ccb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:15.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.006+0000 7fa54e600640 1 -- 172.21.15.16:0/1989092331 >> v1:172.21.15.16:6789/0 conn(0x7fa54815e3b0 legacy=0x7fa54809aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:15.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.006+0000 7fa54e600640 1 -- 172.21.15.16:0/1989092331 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5480d0ae0 con 0x7fa548070d50 2024-03-31T15:08:15.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.010+0000 7fa54e600640 1 --2- 172.21.15.16:0/1989092331 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa548070d50 0x7fa5480cd0d0 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7fa540009d90 tx=0x7fa54001c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b24076108318500 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:08:15.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.010+0000 7fa547e00640 1 -- 172.21.15.16:0/1989092331 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5400b2070 con 0x7fa548070d50 2024-03-31T15:08:15.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.010+0000 7fa547e00640 1 -- 172.21.15.16:0/1989092331 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa5400a5a00 con 0x7fa548070d50 2024-03-31T15:08:15.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.010+0000 7fa547e00640 1 -- 172.21.15.16:0/1989092331 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5400a5dd0 con 0x7fa548070d50 2024-03-31T15:08:15.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.010+0000 7fa54f000640 1 -- 172.21.15.16:0/1989092331 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa548070d50 msgr2=0x7fa5480cd0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:15.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.010+0000 7fa54f000640 1 --2- 172.21.15.16:0/1989092331 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa548070d50 0x7fa5480cd0d0 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7fa540009d90 tx=0x7fa54001c940 comp rx=0 tx=0).stop 2024-03-31T15:08:15.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.010+0000 7fa54f000640 1 -- 172.21.15.16:0/1989092331 shutdown_connections 2024-03-31T15:08:15.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.010+0000 7fa54f000640 1 --2- 172.21.15.16:0/1989092331 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa548070d50 0x7fa5480cd0d0 unknown :-1 s=CLOSED pgs=888 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:15.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.010+0000 7fa54f000640 1 --2- 172.21.15.16:0/1989092331 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fa54809b170 0x7fa5480ccb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:15.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.010+0000 7fa54f000640 1 -- 172.21.15.16:0/1989092331 >> 172.21.15.16:0/1989092331 conn(0x7fa5480b39c0 msgr2=0x7fa5480b3dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:08:15.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.010+0000 7fa54f000640 1 -- 172.21.15.16:0/1989092331 shutdown_connections 2024-03-31T15:08:15.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.010+0000 7fa54f000640 1 -- 172.21.15.16:0/1989092331 wait complete. 2024-03-31T15:08:15.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.010+0000 7fa54f000640 1 Processor -- start 2024-03-31T15:08:15.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.010+0000 7fa54f000640 1 -- start start 2024-03-31T15:08:15.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.010+0000 7fa54f000640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa548070d50 0x7fa5480c7680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:15.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.010+0000 7fa54f000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa54809b170 0x7fa548153240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:15.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.010+0000 7fa54f000640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fa54815e3b0 0x7fa54815c2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:15.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.010+0000 7fa54f000640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fa54816b010 con 0x7fa54809b170 2024-03-31T15:08:15.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.010+0000 7fa54f000640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fa54816ae90 con 0x7fa548070d50 2024-03-31T15:08:15.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.010+0000 7fa54f000640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fa54816b190 con 0x7fa54815e3b0 2024-03-31T15:08:15.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.014+0000 7fa54d200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa54809b170 0x7fa548153240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:15.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.014+0000 7fa54d200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa54809b170 0x7fa548153240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51910/0 (socket says 172.21.15.16:51910) 2024-03-31T15:08:15.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.014+0000 7fa54d200640 1 -- 172.21.15.16:0/182260079 learned_addr learned my addr 172.21.15.16:0/182260079 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:08:15.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.014+0000 7fa54e600640 1 --2- 172.21.15.16:0/182260079 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fa54815e3b0 0x7fa54815c2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:15.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.014+0000 7fa54dc00640 1 --2- 172.21.15.16:0/182260079 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa548070d50 0x7fa5480c7680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:15.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.014+0000 7fa54d200640 1 -- 172.21.15.16:0/182260079 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fa54815e3b0 msgr2=0x7fa54815c2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:15.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.014+0000 7fa54d200640 1 --2- 172.21.15.16:0/182260079 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fa54815e3b0 0x7fa54815c2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:15.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.014+0000 7fa54d200640 1 -- 172.21.15.16:0/182260079 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa548070d50 msgr2=0x7fa5480c7680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:15.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.014+0000 7fa54d200640 1 --2- 172.21.15.16:0/182260079 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa548070d50 0x7fa5480c7680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:15.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.014+0000 7fa54d200640 1 -- 172.21.15.16:0/182260079 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5480c87e0 con 0x7fa54809b170 2024-03-31T15:08:15.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.014+0000 7fa54d200640 1 --2- 172.21.15.16:0/182260079 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa54809b170 0x7fa548153240 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7fa53400e930 tx=0x7fa53400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:08:15.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.014+0000 7fa545600640 1 -- 172.21.15.16:0/182260079 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa534017070 con 0x7fa54809b170 2024-03-31T15:08:15.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.014+0000 7fa545600640 1 -- 172.21.15.16:0/182260079 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa53400ca00 con 0x7fa54809b170 2024-03-31T15:08:15.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.014+0000 7fa545600640 1 -- 172.21.15.16:0/182260079 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa53400cdb0 con 0x7fa54809b170 2024-03-31T15:08:15.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.014+0000 7fa54f000640 1 -- 172.21.15.16:0/182260079 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa5480c7c60 con 0x7fa54809b170 2024-03-31T15:08:15.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.014+0000 7fa54f000640 1 -- 172.21.15.16:0/182260079 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa5480c8220 con 0x7fa54809b170 2024-03-31T15:08:15.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.014+0000 7fa54f000640 1 -- 172.21.15.16:0/182260079 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa54804f8c0 con 0x7fa54809b170 2024-03-31T15:08:15.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.018+0000 7fa545600640 1 -- 172.21.15.16:0/182260079 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fa53401a070 con 0x7fa54809b170 2024-03-31T15:08:15.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.018+0000 7fa545600640 1 --2- 172.21.15.16:0/182260079 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fa528068380 0x7fa52806a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:15.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.018+0000 7fa545600640 1 -- 172.21.15.16:0/182260079 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(68..68 src has 1..68) ==== 5795+0+0 (secure 0 0 0) 0x7fa534086e90 con 0x7fa54809b170 2024-03-31T15:08:15.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.018+0000 7fa54dc00640 1 --2- 172.21.15.16:0/182260079 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fa528068380 0x7fa52806a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:15.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.018+0000 7fa54dc00640 1 --2- 172.21.15.16:0/182260079 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fa528068380 0x7fa52806a840 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fa53c005890 tx=0x7fa53c009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:08:15.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.026+0000 7fa545600640 1 -- 172.21.15.16:0/182260079 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fa534057410 con 0x7fa54809b170 2024-03-31T15:08:15.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.222+0000 7fa54f000640 1 -- 172.21.15.16:0/182260079 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa548068ae0 con 0x7fa54809b170 2024-03-31T15:08:15.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.226+0000 7fa545600640 1 -- 172.21.15.16:0/182260079 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 50 v50) ==== 94+0+5184 (secure 0 0 0) 0x7fa534056d10 con 0x7fa54809b170 2024-03-31T15:08:15.220 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:08:15.220 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":50,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6836/4188793506","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":4188793506},{"type":"v1","addr":"172.21.15.89:6836","nonce":4188793506}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":40},{"gid":5942,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/4105657676","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6835","nonce":4105657676},{"type":"v1","addr":"172.21.15.89:6837","nonce":4105657676}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":40},{"gid":6592,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6833/3986752851","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":3986752851},{"type":"v1","addr":"172.21.15.16:6833","nonce":3986752851}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":39}],"filesystems":[{"mdsmap":{"epoch":50,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-31T15:07:56.938140+0000","modified":"2024-03-31T15:08:14.267148+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":68,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6829},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6829":{"gid":6829,"name":"a","rank":0,"incarnation":47,"state":"up:active","state_seq":4,"addr":"172.21.15.16:6835/2645389320","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":2645389320},{"type":"v1","addr":"172.21.15.16:6835","nonce":2645389320}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6829,"qdb_cluster":[6829]},"id":5}]} 2024-03-31T15:08:15.220 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 50 2024-03-31T15:08:15.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.226+0000 7fa54f000640 1 -- 172.21.15.16:0/182260079 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fa528068380 msgr2=0x7fa52806a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:15.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.226+0000 7fa54f000640 1 --2- 172.21.15.16:0/182260079 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fa528068380 0x7fa52806a840 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fa53c005890 tx=0x7fa53c009040 comp rx=0 tx=0).stop 2024-03-31T15:08:15.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.226+0000 7fa54f000640 1 -- 172.21.15.16:0/182260079 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa54809b170 msgr2=0x7fa548153240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:15.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.226+0000 7fa54f000640 1 --2- 172.21.15.16:0/182260079 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa54809b170 0x7fa548153240 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7fa53400e930 tx=0x7fa53400edf0 comp rx=0 tx=0).stop 2024-03-31T15:08:15.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.226+0000 7fa54f000640 1 -- 172.21.15.16:0/182260079 shutdown_connections 2024-03-31T15:08:15.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.226+0000 7fa54f000640 1 --2- 172.21.15.16:0/182260079 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fa528068380 0x7fa52806a840 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:15.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.226+0000 7fa54f000640 1 --2- 172.21.15.16:0/182260079 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fa54815e3b0 0x7fa54815c2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:15.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.226+0000 7fa54f000640 1 --2- 172.21.15.16:0/182260079 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa54809b170 0x7fa548153240 unknown :-1 s=CLOSED pgs=889 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:15.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.226+0000 7fa54f000640 1 --2- 172.21.15.16:0/182260079 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa548070d50 0x7fa5480c7680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:15.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.226+0000 7fa54f000640 1 -- 172.21.15.16:0/182260079 >> 172.21.15.16:0/182260079 conn(0x7fa5480b39c0 msgr2=0x7fa548158360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:08:15.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.226+0000 7fa54f000640 1 -- 172.21.15.16:0/182260079 shutdown_connections 2024-03-31T15:08:15.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.226+0000 7fa54f000640 1 -- 172.21.15.16:0/182260079 wait complete. 2024-03-31T15:08:15.643 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 50, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-31T15:07:56.938140+0000', 'modified': '2024-03-31T15:08:14.267148+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 68, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6829}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6829': {'gid': 6829, 'name': 'a', 'rank': 0, 'incarnation': 47, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.16:6835/2645389320', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6834', 'nonce': 2645389320}, {'type': 'v1', 'addr': '172.21.15.16:6835', 'nonce': 2645389320}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [11], 'metadata_pool': 10, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 6829, 'qdb_cluster': [6829]} 2024-03-31T15:08:15.643 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-31T15:08:15.643 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-03-31T15:08:15.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.790+0000 7ffb62600640 1 Processor -- start 2024-03-31T15:08:15.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.790+0000 7ffb62600640 1 -- start start 2024-03-31T15:08:15.787 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.790+0000 7ffb62600640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ffb5c06dbc0 0x7ffb5c15f650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:15.787 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.790+0000 7ffb62600640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7ffb5c0cb440 0x7ffb5c06f920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:15.787 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.790+0000 7ffb62600640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ffb5c0cb8c0 0x7ffb5c06fe60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:15.787 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.790+0000 7ffb62600640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7ffb5c15fb90 con 0x7ffb5c0cb8c0 2024-03-31T15:08:15.787 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.790+0000 7ffb62600640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7ffb5c15f1c0 con 0x7ffb5c0cb440 2024-03-31T15:08:15.787 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.790+0000 7ffb62600640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7ffb5c0cbe90 con 0x7ffb5c06dbc0 2024-03-31T15:08:15.787 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.790+0000 7ffb5be00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ffb5c0cb8c0 0x7ffb5c06fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:15.787 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.794+0000 7ffb5be00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ffb5c0cb8c0 0x7ffb5c06fe60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51916/0 (socket says 172.21.15.16:51916) 2024-03-31T15:08:15.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.794+0000 7ffb5be00640 1 -- 172.21.15.16:0/3244321089 learned_addr learned my addr 172.21.15.16:0/3244321089 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:08:15.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.794+0000 7ffb53e00640 1 --2- 172.21.15.16:0/3244321089 >> v2:172.21.15.89:3300/0 conn(0x7ffb5c0cb440 0x7ffb5c06f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:15.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.794+0000 7ffb5b400640 1 --2- 172.21.15.16:0/3244321089 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ffb5c06dbc0 0x7ffb5c15f650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:15.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.794+0000 7ffb5be00640 1 -- 172.21.15.16:0/3244321089 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ffb5c06dbc0 msgr2=0x7ffb5c15f650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:08:15.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.794+0000 7ffb5be00640 1 --2- 172.21.15.16:0/3244321089 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ffb5c06dbc0 0x7ffb5c15f650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:15.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.794+0000 7ffb5be00640 1 -- 172.21.15.16:0/3244321089 >> v2:172.21.15.89:3300/0 conn(0x7ffb5c0cb440 msgr2=0x7ffb5c06f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:08:15.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.794+0000 7ffb5be00640 1 --2- 172.21.15.16:0/3244321089 >> v2:172.21.15.89:3300/0 conn(0x7ffb5c0cb440 0x7ffb5c06f920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:15.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.794+0000 7ffb5be00640 1 -- 172.21.15.16:0/3244321089 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffb5c0cc170 con 0x7ffb5c0cb8c0 2024-03-31T15:08:15.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.798+0000 7ffb5be00640 1 --2- 172.21.15.16:0/3244321089 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffb5c0cb8c0 0x7ffb5c06fe60 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7ffb5c0d31b0 tx=0x7ffb5401c930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d45c92a641d101f3 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:08:15.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.798+0000 7ffb5aa00640 1 -- 172.21.15.16:0/3244321089 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb540a5770 con 0x7ffb5c0cb8c0 2024-03-31T15:08:15.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.798+0000 7ffb5aa00640 1 -- 172.21.15.16:0/3244321089 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffb540a5910 con 0x7ffb5c0cb8c0 2024-03-31T15:08:15.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.798+0000 7ffb5aa00640 1 -- 172.21.15.16:0/3244321089 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb540a5ce0 con 0x7ffb5c0cb8c0 2024-03-31T15:08:15.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.798+0000 7ffb62600640 1 -- 172.21.15.16:0/3244321089 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffb5c0cb8c0 msgr2=0x7ffb5c06fe60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:15.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.798+0000 7ffb62600640 1 --2- 172.21.15.16:0/3244321089 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffb5c0cb8c0 0x7ffb5c06fe60 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7ffb5c0d31b0 tx=0x7ffb5401c930 comp rx=0 tx=0).stop 2024-03-31T15:08:15.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.798+0000 7ffb62600640 1 -- 172.21.15.16:0/3244321089 shutdown_connections 2024-03-31T15:08:15.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.798+0000 7ffb62600640 1 --2- 172.21.15.16:0/3244321089 >> v2:172.21.15.89:3300/0 conn(0x7ffb5c0cb440 0x7ffb5c06f920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:15.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.798+0000 7ffb62600640 1 --2- 172.21.15.16:0/3244321089 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffb5c0cb8c0 0x7ffb5c06fe60 unknown :-1 s=CLOSED pgs=890 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:15.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.798+0000 7ffb62600640 1 --2- 172.21.15.16:0/3244321089 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ffb5c06dbc0 0x7ffb5c15f650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:15.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.798+0000 7ffb62600640 1 -- 172.21.15.16:0/3244321089 >> 172.21.15.16:0/3244321089 conn(0x7ffb5c067330 msgr2=0x7ffb5c067740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:08:15.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.798+0000 7ffb62600640 1 -- 172.21.15.16:0/3244321089 shutdown_connections 2024-03-31T15:08:15.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.798+0000 7ffb62600640 1 -- 172.21.15.16:0/3244321089 wait complete. 2024-03-31T15:08:15.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.802+0000 7ffb62600640 1 Processor -- start 2024-03-31T15:08:15.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.802+0000 7ffb62600640 1 -- start start 2024-03-31T15:08:15.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.802+0000 7ffb62600640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffb5c06dbc0 0x7ffb5c1424c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:15.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.802+0000 7ffb62600640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ffb5c0cb440 0x7ffb5c141c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:15.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.802+0000 7ffb62600640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffb5c0cb8c0 0x7ffb5c15b240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:15.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.802+0000 7ffb62600640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7ffb5c06cad0 con 0x7ffb5c06dbc0 2024-03-31T15:08:15.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.802+0000 7ffb62600640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7ffb5c06c950 con 0x7ffb5c0cb8c0 2024-03-31T15:08:15.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.802+0000 7ffb62600640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7ffb5c162990 con 0x7ffb5c0cb440 2024-03-31T15:08:15.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.802+0000 7ffb5b400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffb5c06dbc0 0x7ffb5c1424c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:15.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.802+0000 7ffb5b400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffb5c06dbc0 0x7ffb5c1424c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51926/0 (socket says 172.21.15.16:51926) 2024-03-31T15:08:15.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.802+0000 7ffb5b400640 1 -- 172.21.15.16:0/1006630428 learned_addr learned my addr 172.21.15.16:0/1006630428 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:08:15.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.802+0000 7ffb5be00640 1 --2- 172.21.15.16:0/1006630428 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffb5c0cb8c0 0x7ffb5c15b240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:15.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.802+0000 7ffb53e00640 1 --2- 172.21.15.16:0/1006630428 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ffb5c0cb440 0x7ffb5c141c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:15.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.802+0000 7ffb5b400640 1 -- 172.21.15.16:0/1006630428 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ffb5c0cb440 msgr2=0x7ffb5c141c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:15.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.802+0000 7ffb5b400640 1 --2- 172.21.15.16:0/1006630428 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ffb5c0cb440 0x7ffb5c141c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:15.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.802+0000 7ffb5b400640 1 -- 172.21.15.16:0/1006630428 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffb5c0cb8c0 msgr2=0x7ffb5c15b240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:15.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.802+0000 7ffb5b400640 1 --2- 172.21.15.16:0/1006630428 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffb5c0cb8c0 0x7ffb5c15b240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:15.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.802+0000 7ffb5b400640 1 -- 172.21.15.16:0/1006630428 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffb5c142dc0 con 0x7ffb5c06dbc0 2024-03-31T15:08:15.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.802+0000 7ffb5be00640 1 --2- 172.21.15.16:0/1006630428 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffb5c0cb8c0 0x7ffb5c15b240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:08:15.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.802+0000 7ffb5b400640 1 --2- 172.21.15.16:0/1006630428 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffb5c06dbc0 0x7ffb5c1424c0 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7ffb4c00e8b0 tx=0x7ffb4c00ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:08:15.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.802+0000 7ffb53400640 1 -- 172.21.15.16:0/1006630428 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb4c017070 con 0x7ffb5c06dbc0 2024-03-31T15:08:15.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.802+0000 7ffb53400640 1 -- 172.21.15.16:0/1006630428 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffb4c00c940 con 0x7ffb5c06dbc0 2024-03-31T15:08:15.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.802+0000 7ffb53400640 1 -- 172.21.15.16:0/1006630428 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb4c00cd10 con 0x7ffb5c06dbc0 2024-03-31T15:08:15.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.802+0000 7ffb62600640 1 -- 172.21.15.16:0/1006630428 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffb5c1430a0 con 0x7ffb5c06dbc0 2024-03-31T15:08:15.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.802+0000 7ffb62600640 1 -- 172.21.15.16:0/1006630428 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffb5c143660 con 0x7ffb5c06dbc0 2024-03-31T15:08:15.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.806+0000 7ffb53400640 1 -- 172.21.15.16:0/1006630428 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7ffb4c01a070 con 0x7ffb5c06dbc0 2024-03-31T15:08:15.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.806+0000 7ffb53400640 1 --2- 172.21.15.16:0/1006630428 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7ffb38068380 0x7ffb3806a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:15.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.806+0000 7ffb53400640 1 -- 172.21.15.16:0/1006630428 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(68..68 src has 1..68) ==== 5795+0+0 (secure 0 0 0) 0x7ffb4c086e60 con 0x7ffb5c06dbc0 2024-03-31T15:08:15.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.806+0000 7ffb53e00640 1 --2- 172.21.15.16:0/1006630428 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7ffb38068380 0x7ffb3806a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:15.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.806+0000 7ffb65b53000 1 Processor -- start 2024-03-31T15:08:15.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.806+0000 7ffb65b53000 1 -- start start 2024-03-31T15:08:15.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.806+0000 7ffb53e00640 1 --2- 172.21.15.16:0/1006630428 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7ffb38068380 0x7ffb3806a840 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7ffb5c15be80 tx=0x7ffb44008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:08:15.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.806+0000 7ffb65b53000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555867e204a0 0x555867e20880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:15.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.806+0000 7ffb65b53000 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x555867e20dc0 0x555867dd3b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:15.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.806+0000 7ffb65b53000 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555867dd40b0 0x555867dd4490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:15.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.806+0000 7ffb65b53000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x555867c26950 con 0x555867e204a0 2024-03-31T15:08:15.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.806+0000 7ffb65b53000 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x555867bc3990 con 0x555867dd40b0 2024-03-31T15:08:15.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.806+0000 7ffb65b53000 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x555867c22900 con 0x555867e20dc0 2024-03-31T15:08:15.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.806+0000 7ffb5b400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555867e204a0 0x555867e20880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:15.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.806+0000 7ffb5b400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555867e204a0 0x555867e20880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51930/0 (socket says 172.21.15.16:51930) 2024-03-31T15:08:15.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.806+0000 7ffb5b400640 1 -- 172.21.15.16:0/1618093457 learned_addr learned my addr 172.21.15.16:0/1618093457 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:08:15.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.806+0000 7ffb5be00640 1 --2- 172.21.15.16:0/1618093457 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555867dd40b0 0x555867dd4490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:15.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.806+0000 7ffb53e00640 1 --2- 172.21.15.16:0/1618093457 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x555867e20dc0 0x555867dd3b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:15.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.810+0000 7ffb5b400640 1 -- 172.21.15.16:0/1618093457 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x555867e20dc0 msgr2=0x555867dd3b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:08:15.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.810+0000 7ffb5b400640 1 --2- 172.21.15.16:0/1618093457 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x555867e20dc0 0x555867dd3b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:15.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.810+0000 7ffb5b400640 1 -- 172.21.15.16:0/1618093457 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555867dd40b0 msgr2=0x555867dd4490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:08:15.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.810+0000 7ffb5b400640 1 --2- 172.21.15.16:0/1618093457 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555867dd40b0 0x555867dd4490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:15.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.810+0000 7ffb5b400640 1 -- 172.21.15.16:0/1618093457 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555867c6b100 con 0x555867e204a0 2024-03-31T15:08:15.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.810+0000 7ffb53e00640 1 --2- 172.21.15.16:0/1618093457 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x555867e20dc0 0x555867dd3b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:08:15.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.810+0000 7ffb5b400640 1 --2- 172.21.15.16:0/1618093457 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555867e204a0 0x555867e20880 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7ffb4c0851d0 tx=0x7ffb4c056640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=33c84dfef7d0c120 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:08:15.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.810+0000 7ffb4be00640 1 -- 172.21.15.16:0/1618093457 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb4c021040 con 0x555867e204a0 2024-03-31T15:08:15.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.810+0000 7ffb4be00640 1 -- 172.21.15.16:0/1618093457 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffb4c05f800 con 0x555867e204a0 2024-03-31T15:08:15.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.810+0000 7ffb4be00640 1 -- 172.21.15.16:0/1618093457 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb4c054f20 con 0x555867e204a0 2024-03-31T15:08:15.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.810+0000 7ffb65b53000 1 -- 172.21.15.16:0/1618093457 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555867e204a0 msgr2=0x555867e20880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:15.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.810+0000 7ffb65b53000 1 --2- 172.21.15.16:0/1618093457 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555867e204a0 0x555867e20880 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7ffb4c0851d0 tx=0x7ffb4c056640 comp rx=0 tx=0).stop 2024-03-31T15:08:15.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.810+0000 7ffb65b53000 1 -- 172.21.15.16:0/1618093457 shutdown_connections 2024-03-31T15:08:15.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.810+0000 7ffb65b53000 1 --2- 172.21.15.16:0/1618093457 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555867dd40b0 0x555867dd4490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:15.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.810+0000 7ffb65b53000 1 --2- 172.21.15.16:0/1618093457 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x555867e20dc0 0x555867dd3b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:15.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.810+0000 7ffb65b53000 1 --2- 172.21.15.16:0/1618093457 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555867e204a0 0x555867e20880 unknown :-1 s=CLOSED pgs=892 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:15.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.810+0000 7ffb65b53000 1 -- 172.21.15.16:0/1618093457 >> 172.21.15.16:0/1618093457 conn(0x555867cbeb80 msgr2=0x555867e03460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:08:15.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.810+0000 7ffb65b53000 1 -- 172.21.15.16:0/1618093457 shutdown_connections 2024-03-31T15:08:15.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.810+0000 7ffb65b53000 1 -- 172.21.15.16:0/1618093457 wait complete. 2024-03-31T15:08:15.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.810+0000 7ffb65b53000 1 -- start start 2024-03-31T15:08:15.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.810+0000 7ffb65b53000 1 Processor -- start 2024-03-31T15:08:15.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.810+0000 7ffb65b53000 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555867dd40b0 0x555867de0e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:15.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.810+0000 7ffb65b53000 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x555867e204a0 0x555867de1340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:15.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.810+0000 7ffb65b53000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555867e20dc0 0x555867de1880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:15.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.810+0000 7ffb65b53000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x555867c25c60 con 0x555867e20dc0 2024-03-31T15:08:15.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.810+0000 7ffb65b53000 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x555867be43e0 con 0x555867dd40b0 2024-03-31T15:08:15.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.810+0000 7ffb65b53000 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x555867c33060 con 0x555867e204a0 2024-03-31T15:08:15.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.810+0000 7ffb65b53000 10 client.0 _ll_register_callbacks cb 0x555867c870f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T15:08:15.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.810+0000 7ffb5be00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555867e20dc0 0x555867de1880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:15.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.810+0000 7ffb5be00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555867e20dc0 0x555867de1880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51946/0 (socket says 172.21.15.16:51946) 2024-03-31T15:08:15.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.810+0000 7ffb5be00640 1 -- 172.21.15.16:0/4228584430 learned_addr learned my addr 172.21.15.16:0/4228584430 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:08:15.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.810+0000 7ffb5b400640 1 --2- 172.21.15.16:0/4228584430 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555867dd40b0 0x555867de0e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:15.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.814+0000 7ffb53e00640 1 --2- 172.21.15.16:0/4228584430 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x555867e204a0 0x555867de1340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:15.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.814+0000 7ffb5be00640 1 -- 172.21.15.16:0/4228584430 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x555867e204a0 msgr2=0x555867de1340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:08:15.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.814+0000 7ffb5be00640 1 --2- 172.21.15.16:0/4228584430 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x555867e204a0 0x555867de1340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:15.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.814+0000 7ffb5be00640 1 -- 172.21.15.16:0/4228584430 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555867dd40b0 msgr2=0x555867de0e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:08:15.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.814+0000 7ffb5be00640 1 --2- 172.21.15.16:0/4228584430 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555867dd40b0 0x555867de0e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:15.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.814+0000 7ffb5be00640 1 -- 172.21.15.16:0/4228584430 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555867bf08c0 con 0x555867e20dc0 2024-03-31T15:08:15.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.814+0000 7ffb53e00640 1 --2- 172.21.15.16:0/4228584430 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x555867e204a0 0x555867de1340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:08:15.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.814+0000 7ffb5be00640 1 --2- 172.21.15.16:0/4228584430 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555867e20dc0 0x555867de1880 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7ffb540af040 tx=0x7ffb5401b170 comp rx=0 tx=0).ready entity=mon.0 client_cookie=79340812a92dd310 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:08:15.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.814+0000 7ffb48c00640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-31T15:08:15.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.814+0000 7ffb48c00640 1 -- 172.21.15.16:0/4228584430 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb540a5a30 con 0x555867e20dc0 2024-03-31T15:08:15.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.814+0000 7ffb65b53000 1 -- 172.21.15.16:0/4228584430 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ffb4c05f800 con 0x555867e20dc0 2024-03-31T15:08:15.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.814+0000 7ffb48c00640 1 -- 172.21.15.16:0/4228584430 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(68..68 src has 1..68) ==== 5795+0+0 (secure 0 0 0) 0x7ffb540a5bd0 con 0x555867e20dc0 2024-03-31T15:08:15.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.814+0000 7ffb48c00640 1 -- 172.21.15.16:0/4228584430 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=50) ==== 24+0+0 (secure 0 0 0) 0x7ffb540c9050 con 0x555867e20dc0 2024-03-31T15:08:15.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.814+0000 7ffb65b53000 10 client.7014 fetch_fsmap learned FSMap version 50 2024-03-31T15:08:15.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.814+0000 7ffb65b53000 1 -- 172.21.15.16:0/4228584430 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=50,monmap=2+}) -- 0x555867baee80 con 0x555867e20dc0 2024-03-31T15:08:15.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.814+0000 7ffb48c00640 1 -- 172.21.15.16:0/4228584430 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffb4c05f800 con 0x555867e20dc0 2024-03-31T15:08:15.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.814+0000 7ffb48c00640 1 -- 172.21.15.16:0/4228584430 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 50) ==== 3494+0+0 (secure 0 0 0) 0x7ffb540cc020 con 0x555867e20dc0 2024-03-31T15:08:15.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.814+0000 7ffb65b53000 10 client.7014 fetch_fsmap finished waiting for FSMap version 50 2024-03-31T15:08:15.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.814+0000 7ffb65b53000 10 client.7014 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-31T15:08:15.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.814+0000 7ffb65b53000 20 client.7014 populate_metadata read hostname 'smithi016' 2024-03-31T15:08:15.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.814+0000 7ffb65b53000 1 --2- 172.21.15.16:0/4228584430 >> [v2:172.21.15.16:6834/2645389320,v1:172.21.15.16:6835/2645389320] conn(0x555867de2120 0x555867e32a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:15.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.814+0000 7ffb65b53000 4 client.7014 mds_command: new command op to 6829 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-31T15:08:15.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.814+0000 7ffb65b53000 1 -- 172.21.15.16:0/4228584430 --> [v2:172.21.15.16:6834/2645389320,v1:172.21.15.16:6835/2645389320] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555867bc1bc0 con 0x555867de2120 2024-03-31T15:08:15.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.814+0000 7ffb5b400640 1 --2- 172.21.15.16:0/4228584430 >> [v2:172.21.15.16:6834/2645389320,v1:172.21.15.16:6835/2645389320] conn(0x555867de2120 0x555867e32a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:15.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.814+0000 7ffb5b400640 1 --2- 172.21.15.16:0/4228584430 >> [v2:172.21.15.16:6834/2645389320,v1:172.21.15.16:6835/2645389320] conn(0x555867de2120 0x555867e32a20 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=263f86f4906ccab5 server_cookie=6b5a248328bf4d3a in_seq=0 out_seq=0 2024-03-31T15:08:15.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.814+0000 7ffb48c00640 10 client.7014 ms_handle_connect on v2:172.21.15.16:6834/2645389320 2024-03-31T15:08:15.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.814+0000 7ffb48c00640 1 -- 172.21.15.16:0/4228584430 <== mds.0 v2:172.21.15.16:6834/2645389320 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x555867bc1bc0 con 0x555867de2120 2024-03-31T15:08:15.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.814+0000 7ffb48c00640 10 client.7014 handle_command_reply: tid=0 2024-03-31T15:08:15.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.814+0000 7ffb65b53000 1 client.7014 shutdown 2024-03-31T15:08:15.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.814+0000 7ffb65b53000 1 -- 172.21.15.16:0/4228584430 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555867e20dc0 msgr2=0x555867de1880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:15.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.814+0000 7ffb65b53000 1 --2- 172.21.15.16:0/4228584430 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555867e20dc0 0x555867de1880 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7ffb540af040 tx=0x7ffb5401b170 comp rx=0 tx=0).stop 2024-03-31T15:08:15.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.814+0000 7ffb65b53000 1 -- 172.21.15.16:0/4228584430 shutdown_connections 2024-03-31T15:08:15.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.814+0000 7ffb65b53000 1 --2- 172.21.15.16:0/4228584430 >> [v2:172.21.15.16:6834/2645389320,v1:172.21.15.16:6835/2645389320] conn(0x555867de2120 0x555867e32a20 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:15.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.814+0000 7ffb65b53000 1 --2- 172.21.15.16:0/4228584430 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555867e20dc0 0x555867de1880 unknown :-1 s=CLOSED pgs=893 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:15.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.814+0000 7ffb48c00640 0 client.7014 ms_handle_reset on v2:172.21.15.16:6834/2645389320 2024-03-31T15:08:15.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.814+0000 7ffb65b53000 1 --2- 172.21.15.16:0/4228584430 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x555867e204a0 0x555867de1340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:15.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.814+0000 7ffb65b53000 1 --2- 172.21.15.16:0/4228584430 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555867dd40b0 0x555867de0e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:15.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.814+0000 7ffb65b53000 1 -- 172.21.15.16:0/4228584430 >> 172.21.15.16:0/4228584430 conn(0x555867cbeb80 msgr2=0x555867dd3270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:08:15.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.814+0000 7ffb65b53000 1 -- 172.21.15.16:0/4228584430 shutdown_connections 2024-03-31T15:08:15.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.814+0000 7ffb65b53000 1 -- 172.21.15.16:0/4228584430 wait complete. 2024-03-31T15:08:15.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.814+0000 7ffb65b53000 20 client.7014 trim_cache size 0 max 16384 2024-03-31T15:08:15.829 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.834+0000 7ffb65b53000 1 Processor -- start 2024-03-31T15:08:15.829 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.834+0000 7ffb65b53000 1 -- start start 2024-03-31T15:08:15.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.834+0000 7ffb65b53000 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x555867cbeb80 0x555867dd3910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:15.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.834+0000 7ffb65b53000 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555867dd40b0 0x555867e1f770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:15.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.834+0000 7ffb65b53000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555867e204a0 0x555867e1fcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:15.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.834+0000 7ffb65b53000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7ffb5c203640 con 0x555867e204a0 2024-03-31T15:08:15.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.834+0000 7ffb65b53000 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7ffb5c0b7700 con 0x555867dd40b0 2024-03-31T15:08:15.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.834+0000 7ffb65b53000 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7ffb5c1450c0 con 0x555867cbeb80 2024-03-31T15:08:15.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.834+0000 7ffb5be00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555867e204a0 0x555867e1fcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:15.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.834+0000 7ffb5be00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555867e204a0 0x555867e1fcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51956/0 (socket says 172.21.15.16:51956) 2024-03-31T15:08:15.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.834+0000 7ffb5be00640 1 -- 172.21.15.16:0/2941652468 learned_addr learned my addr 172.21.15.16:0/2941652468 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:08:15.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.834+0000 7ffb5b400640 1 --2- 172.21.15.16:0/2941652468 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x555867cbeb80 0x555867dd3910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:15.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.834+0000 7ffb53e00640 1 --2- 172.21.15.16:0/2941652468 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555867dd40b0 0x555867e1f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:15.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.834+0000 7ffb5be00640 1 -- 172.21.15.16:0/2941652468 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x555867cbeb80 msgr2=0x555867dd3910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:08:15.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.834+0000 7ffb5be00640 1 --2- 172.21.15.16:0/2941652468 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x555867cbeb80 0x555867dd3910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:15.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.834+0000 7ffb5be00640 1 -- 172.21.15.16:0/2941652468 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555867dd40b0 msgr2=0x555867e1f770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:08:15.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.834+0000 7ffb5be00640 1 --2- 172.21.15.16:0/2941652468 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555867dd40b0 0x555867e1f770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:15.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.834+0000 7ffb5be00640 1 -- 172.21.15.16:0/2941652468 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555867be1490 con 0x555867e204a0 2024-03-31T15:08:15.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.834+0000 7ffb53e00640 1 --2- 172.21.15.16:0/2941652468 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555867dd40b0 0x555867e1f770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:08:15.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.834+0000 7ffb5be00640 1 --2- 172.21.15.16:0/2941652468 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555867e204a0 0x555867e1fcb0 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7ffb5401afe0 tx=0x7ffb540b8040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=24937f2a0cb5e8f5 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:08:15.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.838+0000 7ffb4b400640 1 -- 172.21.15.16:0/2941652468 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb540c1540 con 0x555867e204a0 2024-03-31T15:08:15.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.838+0000 7ffb4b400640 1 -- 172.21.15.16:0/2941652468 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffb540039a0 con 0x555867e204a0 2024-03-31T15:08:15.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.838+0000 7ffb4b400640 1 -- 172.21.15.16:0/2941652468 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb54003d50 con 0x555867e204a0 2024-03-31T15:08:15.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.838+0000 7ffb65b53000 1 -- 172.21.15.16:0/2941652468 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555867e204a0 msgr2=0x555867e1fcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:15.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.838+0000 7ffb65b53000 1 --2- 172.21.15.16:0/2941652468 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555867e204a0 0x555867e1fcb0 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7ffb5401afe0 tx=0x7ffb540b8040 comp rx=0 tx=0).stop 2024-03-31T15:08:15.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.838+0000 7ffb65b53000 1 -- 172.21.15.16:0/2941652468 shutdown_connections 2024-03-31T15:08:15.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.838+0000 7ffb65b53000 1 --2- 172.21.15.16:0/2941652468 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555867e204a0 0x555867e1fcb0 unknown :-1 s=CLOSED pgs=894 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:15.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.838+0000 7ffb65b53000 1 --2- 172.21.15.16:0/2941652468 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555867dd40b0 0x555867e1f770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:15.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.838+0000 7ffb65b53000 1 --2- 172.21.15.16:0/2941652468 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x555867cbeb80 0x555867dd3910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:15.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.838+0000 7ffb65b53000 1 -- 172.21.15.16:0/2941652468 >> 172.21.15.16:0/2941652468 conn(0x555867dd8820 msgr2=0x555867ddb290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:08:15.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.838+0000 7ffb65b53000 1 -- 172.21.15.16:0/2941652468 shutdown_connections 2024-03-31T15:08:15.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.838+0000 7ffb65b53000 1 -- 172.21.15.16:0/2941652468 wait complete. 2024-03-31T15:08:15.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.838+0000 7ffb65b53000 1 -- start start 2024-03-31T15:08:15.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.838+0000 7ffb65b53000 1 Processor -- start 2024-03-31T15:08:15.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.838+0000 7ffb65b53000 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555867cbeb80 0x555867ddbf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:15.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.838+0000 7ffb65b53000 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x555867dd40b0 0x555867e77d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:15.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.838+0000 7ffb65b53000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555867e204a0 0x555867e78260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:15.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.838+0000 7ffb65b53000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x555867c2af80 con 0x555867e204a0 2024-03-31T15:08:15.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.838+0000 7ffb65b53000 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7ffb5c0b70b0 con 0x555867cbeb80 2024-03-31T15:08:15.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.838+0000 7ffb65b53000 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x555867be8080 con 0x555867dd40b0 2024-03-31T15:08:15.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.838+0000 7ffb65b53000 10 client.0 _ll_register_callbacks cb 0x7ffb4c009060 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T15:08:15.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.838+0000 7ffb5be00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555867e204a0 0x555867e78260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:15.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.838+0000 7ffb5be00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555867e204a0 0x555867e78260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51958/0 (socket says 172.21.15.16:51958) 2024-03-31T15:08:15.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.838+0000 7ffb5be00640 1 -- 172.21.15.16:0/1244933682 learned_addr learned my addr 172.21.15.16:0/1244933682 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:08:15.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.838+0000 7ffb5b400640 1 --2- 172.21.15.16:0/1244933682 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555867cbeb80 0x555867ddbf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:15.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.838+0000 7ffb53e00640 1 --2- 172.21.15.16:0/1244933682 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x555867dd40b0 0x555867e77d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:15.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.838+0000 7ffb5be00640 1 -- 172.21.15.16:0/1244933682 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x555867dd40b0 msgr2=0x555867e77d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:08:15.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.838+0000 7ffb5be00640 1 --2- 172.21.15.16:0/1244933682 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x555867dd40b0 0x555867e77d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:15.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.838+0000 7ffb5be00640 1 -- 172.21.15.16:0/1244933682 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555867cbeb80 msgr2=0x555867ddbf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:08:15.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.838+0000 7ffb5be00640 1 --2- 172.21.15.16:0/1244933682 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555867cbeb80 0x555867ddbf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:15.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.838+0000 7ffb5be00640 1 -- 172.21.15.16:0/1244933682 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555867bdfe80 con 0x555867e204a0 2024-03-31T15:08:15.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.838+0000 7ffb5be00640 1 --2- 172.21.15.16:0/1244933682 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555867e204a0 0x555867e78260 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7ffb5401ce10 tx=0x7ffb54009810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=648a30d4c6f08a4c server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:08:15.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.838+0000 7ffb48c00640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-31T15:08:15.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.838+0000 7ffb48c00640 1 -- 172.21.15.16:0/1244933682 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb540a5830 con 0x555867e204a0 2024-03-31T15:08:15.839 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.838+0000 7ffb48c00640 1 -- 172.21.15.16:0/1244933682 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(68..68 src has 1..68) ==== 5795+0+0 (secure 0 0 0) 0x7ffb540d7b40 con 0x555867e204a0 2024-03-31T15:08:15.839 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.838+0000 7ffb65b53000 1 -- 172.21.15.16:0/1244933682 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ffb540039a0 con 0x555867e204a0 2024-03-31T15:08:15.839 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.838+0000 7ffb48c00640 1 -- 172.21.15.16:0/1244933682 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=50) ==== 24+0+0 (secure 0 0 0) 0x7ffb540d7d40 con 0x555867e204a0 2024-03-31T15:08:15.839 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.838+0000 7ffb65b53000 10 client.7020 fetch_fsmap learned FSMap version 50 2024-03-31T15:08:15.839 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.838+0000 7ffb65b53000 1 -- 172.21.15.16:0/1244933682 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=50,monmap=2+}) -- 0x555867bcd270 con 0x555867e204a0 2024-03-31T15:08:15.839 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.842+0000 7ffb48c00640 1 -- 172.21.15.16:0/1244933682 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffb540039a0 con 0x555867e204a0 2024-03-31T15:08:15.839 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.842+0000 7ffb48c00640 1 -- 172.21.15.16:0/1244933682 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 50) ==== 3494+0+0 (secure 0 0 0) 0x7ffb540cc070 con 0x555867e204a0 2024-03-31T15:08:15.839 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.842+0000 7ffb65b53000 10 client.7020 fetch_fsmap finished waiting for FSMap version 50 2024-03-31T15:08:15.839 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.842+0000 7ffb65b53000 10 client.7020 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-31T15:08:15.839 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.842+0000 7ffb65b53000 20 client.7020 populate_metadata read hostname 'smithi016' 2024-03-31T15:08:15.839 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.842+0000 7ffb65b53000 1 --2- 172.21.15.16:0/1244933682 >> [v2:172.21.15.16:6834/2645389320,v1:172.21.15.16:6835/2645389320] conn(0x555867e20dc0 0x555867e33a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:15.839 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.842+0000 7ffb65b53000 4 client.7020 mds_command: new command op to 6829 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-31T15:08:15.839 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.842+0000 7ffb65b53000 1 -- 172.21.15.16:0/1244933682 --> [v2:172.21.15.16:6834/2645389320,v1:172.21.15.16:6835/2645389320] -- command(tid 0: {"prefix": "status"}) -- 0x555867c33e90 con 0x555867e20dc0 2024-03-31T15:08:15.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.842+0000 7ffb5b400640 1 --2- 172.21.15.16:0/1244933682 >> [v2:172.21.15.16:6834/2645389320,v1:172.21.15.16:6835/2645389320] conn(0x555867e20dc0 0x555867e33a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:15.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.842+0000 7ffb5b400640 1 --2- 172.21.15.16:0/1244933682 >> [v2:172.21.15.16:6834/2645389320,v1:172.21.15.16:6835/2645389320] conn(0x555867e20dc0 0x555867e33a70 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7032fcb5060f66a7 server_cookie=cc7ece5f0045a6b6 in_seq=0 out_seq=0 2024-03-31T15:08:15.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.842+0000 7ffb48c00640 10 client.7020 ms_handle_connect on v2:172.21.15.16:6834/2645389320 2024-03-31T15:08:15.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.842+0000 7ffb48c00640 1 -- 172.21.15.16:0/1244933682 <== mds.0 v2:172.21.15.16:6834/2645389320 1 ==== command_reply(tid 0: 0 ) ==== 8+0+331 (crc 0 0 0) 0x555867c33e90 con 0x555867e20dc0 2024-03-31T15:08:15.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.842+0000 7ffb48c00640 10 client.7020 handle_command_reply: tid=0 2024-03-31T15:08:15.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.842+0000 7ffb65b53000 1 client.7020 shutdown 2024-03-31T15:08:15.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.842+0000 7ffb65b53000 1 -- 172.21.15.16:0/1244933682 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555867e204a0 msgr2=0x555867e78260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:15.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.842+0000 7ffb65b53000 1 --2- 172.21.15.16:0/1244933682 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555867e204a0 0x555867e78260 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7ffb5401ce10 tx=0x7ffb54009810 comp rx=0 tx=0).stop 2024-03-31T15:08:15.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.842+0000 7ffb65b53000 1 -- 172.21.15.16:0/1244933682 shutdown_connections 2024-03-31T15:08:15.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.842+0000 7ffb65b53000 1 --2- 172.21.15.16:0/1244933682 >> [v2:172.21.15.16:6834/2645389320,v1:172.21.15.16:6835/2645389320] conn(0x555867e20dc0 0x555867e33a70 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:15.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.842+0000 7ffb65b53000 1 --2- 172.21.15.16:0/1244933682 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555867e204a0 0x555867e78260 unknown :-1 s=CLOSED pgs=895 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:15.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.842+0000 7ffb65b53000 1 --2- 172.21.15.16:0/1244933682 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x555867dd40b0 0x555867e77d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:15.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.842+0000 7ffb65b53000 1 --2- 172.21.15.16:0/1244933682 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555867cbeb80 0x555867ddbf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:15.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.842+0000 7ffb65b53000 1 -- 172.21.15.16:0/1244933682 >> 172.21.15.16:0/1244933682 conn(0x555867dd8820 msgr2=0x555867e02d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:08:15.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.842+0000 7ffb48c00640 0 client.7020 ms_handle_reset on v2:172.21.15.16:6834/2645389320 2024-03-31T15:08:15.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.842+0000 7ffb65b53000 1 -- 172.21.15.16:0/1244933682 shutdown_connections 2024-03-31T15:08:15.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.842+0000 7ffb65b53000 1 -- 172.21.15.16:0/1244933682 wait complete. 2024-03-31T15:08:15.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.842+0000 7ffb65b53000 20 client.7020 trim_cache size 0 max 16384 2024-03-31T15:08:15.841 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-03-31T15:08:15.841 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_fsid": "286eb5da-fb47-4550-b290-d9519c13b9a5", 2024-03-31T15:08:15.841 INFO:teuthology.orchestra.run.smithi016.stdout: "whoami": 0, 2024-03-31T15:08:15.841 INFO:teuthology.orchestra.run.smithi016.stdout: "id": 6829, 2024-03-31T15:08:15.841 INFO:teuthology.orchestra.run.smithi016.stdout: "want_state": "up:active", 2024-03-31T15:08:15.842 INFO:teuthology.orchestra.run.smithi016.stdout: "state": "up:active", 2024-03-31T15:08:15.842 INFO:teuthology.orchestra.run.smithi016.stdout: "fs_name": "cephfs", 2024-03-31T15:08:15.842 INFO:teuthology.orchestra.run.smithi016.stdout: "rank_uptime": 4.5858422760000002, 2024-03-31T15:08:15.842 INFO:teuthology.orchestra.run.smithi016.stdout: "mdsmap_epoch": 50, 2024-03-31T15:08:15.842 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch": 68, 2024-03-31T15:08:15.842 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch_barrier": 68, 2024-03-31T15:08:15.842 INFO:teuthology.orchestra.run.smithi016.stdout: "uptime": 5.2423100399999996 2024-03-31T15:08:15.842 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-03-31T15:08:15.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.842+0000 7ffb62600640 1 -- 172.21.15.16:0/1006630428 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7ffb38068380 msgr2=0x7ffb3806a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:15.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.842+0000 7ffb62600640 1 --2- 172.21.15.16:0/1006630428 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7ffb38068380 0x7ffb3806a840 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7ffb5c15be80 tx=0x7ffb44008040 comp rx=0 tx=0).stop 2024-03-31T15:08:15.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.842+0000 7ffb62600640 1 -- 172.21.15.16:0/1006630428 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffb5c06dbc0 msgr2=0x7ffb5c1424c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:15.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.842+0000 7ffb62600640 1 --2- 172.21.15.16:0/1006630428 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffb5c06dbc0 0x7ffb5c1424c0 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7ffb4c00e8b0 tx=0x7ffb4c00ed70 comp rx=0 tx=0).stop 2024-03-31T15:08:15.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.842+0000 7ffb62600640 1 -- 172.21.15.16:0/1006630428 shutdown_connections 2024-03-31T15:08:15.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.842+0000 7ffb62600640 1 --2- 172.21.15.16:0/1006630428 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7ffb38068380 0x7ffb3806a840 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:15.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.842+0000 7ffb62600640 1 --2- 172.21.15.16:0/1006630428 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffb5c0cb8c0 0x7ffb5c15b240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:15.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.842+0000 7ffb62600640 1 --2- 172.21.15.16:0/1006630428 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ffb5c0cb440 0x7ffb5c141c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:15.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.842+0000 7ffb62600640 1 --2- 172.21.15.16:0/1006630428 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffb5c06dbc0 0x7ffb5c1424c0 unknown :-1 s=CLOSED pgs=891 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:15.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.842+0000 7ffb62600640 1 -- 172.21.15.16:0/1006630428 >> 172.21.15.16:0/1006630428 conn(0x7ffb5c067330 msgr2=0x7ffb5c15d9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:08:15.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.842+0000 7ffb62600640 1 -- 172.21.15.16:0/1006630428 shutdown_connections 2024-03-31T15:08:15.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:15.842+0000 7ffb62600640 1 -- 172.21.15.16:0/1006630428 wait complete. 2024-03-31T15:08:15.857 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-31T15:08:15.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:16.002+0000 7f2a6c400640 1 Processor -- start 2024-03-31T15:08:15.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:16.002+0000 7f2a6c400640 1 -- start start 2024-03-31T15:08:15.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:16.002+0000 7f2a6c400640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f2a640cd750 0x7f2a641693a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:15.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:16.002+0000 7f2a6c400640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2a64162010 con 0x7f2a6406d490 2024-03-31T15:08:15.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:16.002+0000 7f2a6c400640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2a641622b0 con 0x7f2a6406f920 2024-03-31T15:08:15.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:16.002+0000 7f2a6c400640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f2a64162be0 con 0x7f2a640cd750 2024-03-31T15:08:15.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:16.002+0000 7f2a69000640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f2a6406d490 0x7f2a6406d8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:55578/0 (socket says 172.21.15.16:55578) 2024-03-31T15:08:15.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:16.002+0000 7f2a69000640 1 -- 172.21.15.16:0/3655902493 learned_addr learned my addr 172.21.15.16:0/3655902493 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:08:16.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:16.002+0000 7f2a6a400640 1 --2- 172.21.15.16:0/3655902493 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f2a640cd750 0x7f2a641693a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:16.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:16.002+0000 7f2a63e00640 1 -- 172.21.15.16:0/3655902493 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 242237028 0 0) 0x7f2a64162010 con 0x7f2a6406d490 2024-03-31T15:08:16.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:16.002+0000 7f2a63e00640 1 -- 172.21.15.16:0/3655902493 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2a4c003530 con 0x7f2a6406d490 2024-03-31T15:08:16.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:16.002+0000 7f2a63e00640 1 -- 172.21.15.16:0/3655902493 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2939259278 0 0) 0x7f2a4c003530 con 0x7f2a6406d490 2024-03-31T15:08:16.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:16.002+0000 7f2a63e00640 1 -- 172.21.15.16:0/3655902493 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f2a640cd750 msgr2=0x7f2a641693a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:08:16.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:16.002+0000 7f2a63e00640 1 --2- 172.21.15.16:0/3655902493 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f2a640cd750 0x7f2a641693a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:16.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:16.002+0000 7f2a63e00640 1 -- 172.21.15.16:0/3655902493 >> v1:172.21.15.89:6789/0 conn(0x7f2a6406f920 legacy=0x7f2a6406fd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:16.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:16.002+0000 7f2a63e00640 1 -- 172.21.15.16:0/3655902493 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a64162590 con 0x7f2a6406d490 2024-03-31T15:08:16.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:16.002+0000 7f2a63e00640 1 -- 172.21.15.16:0/3655902493 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 214807623 0 0) 0x7f2a50001680 con 0x7f2a6406d490 2024-03-31T15:08:16.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:16.006+0000 7f2a63e00640 1 -- 172.21.15.16:0/3655902493 >> v1:172.21.15.16:6789/0 conn(0x7f2a6406d490 legacy=0x7f2a6406d8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:16.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:16.006+0000 7f2a63e00640 1 --2- 172.21.15.16:0/3655902493 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2a4c0039f0 0x7f2a4c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:16.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:16.006+0000 7f2a63e00640 1 --2- 172.21.15.16:0/3655902493 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2a4c0043a0 0x7f2a4c004870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:16.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:16.006+0000 7f2a63e00640 1 --2- 172.21.15.16:0/3655902493 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f2a640cd750 0x7f2a4c0050d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:16.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:16.006+0000 7f2a63e00640 1 -- 172.21.15.16:0/3655902493 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f2a64162be0 con 0x7f2a4c0043a0 2024-03-31T15:08:16.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:16.006+0000 7f2a63e00640 1 -- 172.21.15.16:0/3655902493 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f2a4c00ba30 con 0x7f2a4c0039f0 2024-03-31T15:08:16.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:16.006+0000 7f2a63e00640 1 -- 172.21.15.16:0/3655902493 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f2a4c00bc30 con 0x7f2a640cd750 2024-03-31T15:08:16.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:16.006+0000 7f2a69000640 1 --2- 172.21.15.16:0/3655902493 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2a4c0043a0 0x7f2a4c004870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:16.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:16.006+0000 7f2a69a00640 1 --2- 172.21.15.16:0/3655902493 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2a4c0039f0 0x7f2a4c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:16.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:16.006+0000 7f2a6a400640 1 --2- 172.21.15.16:0/3655902493 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f2a640cd750 0x7f2a4c0050d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:16.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:16.006+0000 7f2a69000640 1 -- 172.21.15.16:0/3655902493 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f2a640cd750 msgr2=0x7f2a4c0050d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:08:16.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:16.006+0000 7f2a69000640 1 --2- 172.21.15.16:0/3655902493 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f2a640cd750 0x7f2a4c0050d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:16.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:16.006+0000 7f2a69000640 1 -- 172.21.15.16:0/3655902493 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2a4c0039f0 msgr2=0x7f2a4c003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:08:16.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:16.006+0000 7f2a69000640 1 --2- 172.21.15.16:0/3655902493 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2a4c0039f0 0x7f2a4c003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:16.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:16.006+0000 7f2a69000640 1 -- 172.21.15.16:0/3655902493 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a4c003530 con 0x7f2a4c0043a0 2024-03-31T15:08:16.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:16.006+0000 7f2a69a00640 1 --2- 172.21.15.16:0/3655902493 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2a4c0039f0 0x7f2a4c003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:08:16.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:16.006+0000 7f2a6a400640 1 --2- 172.21.15.16:0/3655902493 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f2a640cd750 0x7f2a4c0050d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:08:16.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:16.006+0000 7f2a69000640 1 --2- 172.21.15.16:0/3655902493 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2a4c0043a0 0x7f2a4c004870 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7f2a5000c210 tx=0x7f2a5001cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=987a8c6dc94b49bf server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:08:16.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:16.006+0000 7f2a63e00640 1 -- 172.21.15.16:0/3655902493 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a50007d20 con 0x7f2a4c0043a0 2024-03-31T15:08:16.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:16.006+0000 7f2a63e00640 1 -- 172.21.15.16:0/3655902493 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2a500a9020 con 0x7f2a4c0043a0 2024-03-31T15:08:16.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:16.006+0000 7f2a63e00640 1 -- 172.21.15.16:0/3655902493 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a500a5b40 con 0x7f2a4c0043a0 2024-03-31T15:08:16.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:16.010+0000 7f2a6c400640 1 -- 172.21.15.16:0/3655902493 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2a4c0043a0 msgr2=0x7f2a4c004870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:16.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:16.010+0000 7f2a6c400640 1 --2- 172.21.15.16:0/3655902493 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2a4c0043a0 0x7f2a4c004870 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7f2a5000c210 tx=0x7f2a5001cc20 comp rx=0 tx=0).stop 2024-03-31T15:08:16.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:16.010+0000 7f2a6a400640 1 -- 172.21.15.16:0/3655902493 reap_dead start 2024-03-31T15:08:16.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:16.010+0000 7f2a6c400640 1 -- 172.21.15.16:0/3655902493 shutdown_connections 2024-03-31T15:08:16.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:16.010+0000 7f2a6c400640 1 -- 172.21.15.16:0/3655902493 >> 172.21.15.16:0/3655902493 conn(0x7f2a64067330 msgr2=0x7f2a64067740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:08:16.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:16.010+0000 7f2a6c400640 1 -- 172.21.15.16:0/3655902493 shutdown_connections 2024-03-31T15:08:16.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:16.010+0000 7f2a6c400640 1 -- 172.21.15.16:0/3655902493 wait complete. 2024-03-31T15:08:16.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:16.010+0000 7f2a6c400640 1 Processor -- start 2024-03-31T15:08:16.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:16.010+0000 7f2a6c400640 1 -- start start 2024-03-31T15:08:16.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:16.010+0000 7f2a6c400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2a4c0043a0 0x7f2a64159890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:16.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:16.010+0000 7f2a6c400640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f2a6413fb30 0x7f2a64145bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:16.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:16.010+0000 7f2a6c400640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2a64146130 0x7f2a6415ea90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:16.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:16.010+0000 7f2a6c400640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f2a640ce770 con 0x7f2a4c0043a0 2024-03-31T15:08:16.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:16.010+0000 7f2a6c400640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f2a64167390 con 0x7f2a64146130 2024-03-31T15:08:16.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:16.010+0000 7f2a6c400640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f2a641601b0 con 0x7f2a6413fb30 2024-03-31T15:08:16.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:16.010+0000 7f2a69a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2a4c0043a0 0x7f2a64159890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:16.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:16.010+0000 7f2a69a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2a4c0043a0 0x7f2a64159890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51972/0 (socket says 172.21.15.16:51972) 2024-03-31T15:08:16.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:16.010+0000 7f2a69a00640 1 -- 172.21.15.16:0/1772171504 learned_addr learned my addr 172.21.15.16:0/1772171504 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:08:16.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:16.010+0000 7f2a69000640 1 --2- 172.21.15.16:0/1772171504 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f2a6413fb30 0x7f2a64145bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:16.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:16.010+0000 7f2a6a400640 1 --2- 172.21.15.16:0/1772171504 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2a64146130 0x7f2a6415ea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:16.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:16.010+0000 7f2a69a00640 1 -- 172.21.15.16:0/1772171504 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f2a6413fb30 msgr2=0x7f2a64145bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:16.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:16.010+0000 7f2a69a00640 1 --2- 172.21.15.16:0/1772171504 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f2a6413fb30 0x7f2a64145bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:16.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:16.010+0000 7f2a69a00640 1 -- 172.21.15.16:0/1772171504 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2a64146130 msgr2=0x7f2a6415ea90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:16.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:16.010+0000 7f2a69a00640 1 --2- 172.21.15.16:0/1772171504 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2a64146130 0x7f2a6415ea90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:16.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:16.010+0000 7f2a69a00640 1 -- 172.21.15.16:0/1772171504 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a64157c90 con 0x7f2a4c0043a0 2024-03-31T15:08:16.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:16.010+0000 7f2a6a400640 1 --2- 172.21.15.16:0/1772171504 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2a64146130 0x7f2a6415ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:08:16.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:16.014+0000 7f2a69a00640 1 --2- 172.21.15.16:0/1772171504 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2a4c0043a0 0x7f2a64159890 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7f2a58010f60 tx=0x7f2a58010f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:08:16.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:16.014+0000 7f2a61600640 1 -- 172.21.15.16:0/1772171504 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a58018020 con 0x7f2a4c0043a0 2024-03-31T15:08:16.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:16.014+0000 7f2a61600640 1 -- 172.21.15.16:0/1772171504 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2a58005070 con 0x7f2a4c0043a0 2024-03-31T15:08:16.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:16.014+0000 7f2a61600640 1 -- 172.21.15.16:0/1772171504 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a58005420 con 0x7f2a4c0043a0 2024-03-31T15:08:16.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:16.014+0000 7f2a6c400640 1 -- 172.21.15.16:0/1772171504 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2a64157f70 con 0x7f2a4c0043a0 2024-03-31T15:08:16.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:16.014+0000 7f2a6c400640 1 -- 172.21.15.16:0/1772171504 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2a64160450 con 0x7f2a4c0043a0 2024-03-31T15:08:16.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:16.014+0000 7f2a6c400640 1 -- 172.21.15.16:0/1772171504 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2a64069950 con 0x7f2a4c0043a0 2024-03-31T15:08:16.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:16.014+0000 7f2a61600640 1 -- 172.21.15.16:0/1772171504 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f2a5801b020 con 0x7f2a4c0043a0 2024-03-31T15:08:16.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:16.018+0000 7f2a61600640 1 --2- 172.21.15.16:0/1772171504 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f2a440683d0 0x7f2a4406a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:16.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:16.018+0000 7f2a61600640 1 -- 172.21.15.16:0/1772171504 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(68..68 src has 1..68) ==== 5795+0+0 (secure 0 0 0) 0x7f2a5808a110 con 0x7f2a4c0043a0 2024-03-31T15:08:16.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:16.018+0000 7f2a69000640 1 --2- 172.21.15.16:0/1772171504 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f2a440683d0 0x7f2a4406a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:16.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:16.018+0000 7f2a69000640 1 --2- 172.21.15.16:0/1772171504 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f2a440683d0 0x7f2a4406a890 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f2a500aa000 tx=0x7f2a6415ea20 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:08:16.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:16.018+0000 7f2a61600640 1 -- 172.21.15.16:0/1772171504 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f2a58022020 con 0x7f2a4c0043a0 2024-03-31T15:08:16.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:16.222+0000 7f2a6c400640 1 -- 172.21.15.16:0/1772171504 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f2a64146510 con 0x7f2a4c0043a0 2024-03-31T15:08:16.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:16.222+0000 7f2a61600640 1 -- 172.21.15.16:0/1772171504 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v50) ==== 53+0+73 (secure 0 0 0) 0x7f2a5805a650 con 0x7f2a4c0043a0 2024-03-31T15:08:16.217 INFO:teuthology.orchestra.run.smithi016.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-31T15:08:16.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:16.222+0000 7f2a6c400640 1 -- 172.21.15.16:0/1772171504 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f2a440683d0 msgr2=0x7f2a4406a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:16.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:16.222+0000 7f2a6c400640 1 --2- 172.21.15.16:0/1772171504 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f2a440683d0 0x7f2a4406a890 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f2a500aa000 tx=0x7f2a6415ea20 comp rx=0 tx=0).stop 2024-03-31T15:08:16.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:16.222+0000 7f2a6c400640 1 -- 172.21.15.16:0/1772171504 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2a4c0043a0 msgr2=0x7f2a64159890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:16.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:16.222+0000 7f2a6c400640 1 --2- 172.21.15.16:0/1772171504 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2a4c0043a0 0x7f2a64159890 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7f2a58010f60 tx=0x7f2a58010f90 comp rx=0 tx=0).stop 2024-03-31T15:08:16.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:16.222+0000 7f2a6c400640 1 -- 172.21.15.16:0/1772171504 shutdown_connections 2024-03-31T15:08:16.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:16.222+0000 7f2a6c400640 1 --2- 172.21.15.16:0/1772171504 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f2a440683d0 0x7f2a4406a890 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:16.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:16.222+0000 7f2a6c400640 1 --2- 172.21.15.16:0/1772171504 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2a64146130 0x7f2a6415ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:16.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:16.222+0000 7f2a6c400640 1 --2- 172.21.15.16:0/1772171504 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f2a6413fb30 0x7f2a64145bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:16.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:16.222+0000 7f2a6c400640 1 --2- 172.21.15.16:0/1772171504 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2a4c0043a0 0x7f2a64159890 unknown :-1 s=CLOSED pgs=898 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:16.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:16.222+0000 7f2a6c400640 1 -- 172.21.15.16:0/1772171504 >> 172.21.15.16:0/1772171504 conn(0x7f2a64067330 msgr2=0x7f2a640cdd20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:08:16.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:16.222+0000 7f2a6c400640 1 -- 172.21.15.16:0/1772171504 shutdown_connections 2024-03-31T15:08:16.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:16.222+0000 7f2a6c400640 1 -- 172.21.15.16:0/1772171504 wait complete. 2024-03-31T15:08:16.238 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-31T15:08:16.238 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-31T15:08:16.238 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi016.front.sepia.ceph.com 2024-03-31T15:08:16.238 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-31T15:08:16.238 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-31T15:08:16.238 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-31T15:08:16.238 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-31T15:08:16.238 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-31T15:08:16.238 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-31T15:08:16.239 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:08:16.239 DEBUG:teuthology.orchestra.run.smithi016:> ip addr 2024-03-31T15:08:16.248 INFO:teuthology.orchestra.run.smithi016.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-31T15:08:16.248 INFO:teuthology.orchestra.run.smithi016.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-31T15:08:16.248 INFO:teuthology.orchestra.run.smithi016.stdout: inet 127.0.0.1/8 scope host lo 2024-03-31T15:08:16.248 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2024-03-31T15:08:16.248 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 ::1/128 scope host 2024-03-31T15:08:16.248 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2024-03-31T15:08:16.248 INFO:teuthology.orchestra.run.smithi016.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-03-31T15:08:16.249 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:6b:fd:e0 brd ff:ff:ff:ff:ff:ff 2024-03-31T15:08:16.249 INFO:teuthology.orchestra.run.smithi016.stdout: altname enp7s0f0 2024-03-31T15:08:16.249 INFO:teuthology.orchestra.run.smithi016.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-03-31T15:08:16.249 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:6b:fd:e1 brd ff:ff:ff:ff:ff:ff 2024-03-31T15:08:16.249 INFO:teuthology.orchestra.run.smithi016.stdout: altname enp7s0f1 2024-03-31T15:08:16.249 INFO:teuthology.orchestra.run.smithi016.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-31T15:08:16.249 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:bd:15:64 brd ff:ff:ff:ff:ff:ff 2024-03-31T15:08:16.249 INFO:teuthology.orchestra.run.smithi016.stdout: altname enp1s0f0 2024-03-31T15:08:16.249 INFO:teuthology.orchestra.run.smithi016.stdout: inet 172.21.15.16/20 brd 172.21.15.255 scope global dynamic ens1f0 2024-03-31T15:08:16.249 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft 42185sec preferred_lft 42185sec 2024-03-31T15:08:16.249 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 fe80::ec4:7aff:febd:1564/64 scope link 2024-03-31T15:08:16.249 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2024-03-31T15:08:16.249 INFO:teuthology.orchestra.run.smithi016.stdout:5: ens1f1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-03-31T15:08:16.249 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:bd:15:65 brd ff:ff:ff:ff:ff:ff 2024-03-31T15:08:16.250 INFO:teuthology.orchestra.run.smithi016.stdout: altname enp1s0f1 2024-03-31T15:08:16.250 INFO:teuthology.orchestra.run.smithi016.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2024-03-31T15:08:16.250 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 02:42:d4:1f:7d:73 brd ff:ff:ff:ff:ff:ff 2024-03-31T15:08:16.250 INFO:teuthology.orchestra.run.smithi016.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2024-03-31T15:08:16.250 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2024-03-31T15:08:16.250 INFO:teuthology.orchestra.run.smithi016.stdout:7: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-31T15:08:16.250 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 5e:1f:84:6d:79:ae brd ff:ff:ff:ff:ff:ff 2024-03-31T15:08:16.250 INFO:teuthology.orchestra.run.smithi016.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-31T15:08:16.250 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2024-03-31T15:08:16.250 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 fe80::5c1f:84ff:fe6d:79ae/64 scope link 2024-03-31T15:08:16.250 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2024-03-31T15:08:16.250 INFO:teuthology.orchestra.run.smithi016.stdout:8: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-31T15:08:16.250 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0e:5c:f6:1f:82:2f brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-31T15:08:16.251 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 fe80::c5c:f6ff:fe1f:822f/64 scope link 2024-03-31T15:08:16.251 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2024-03-31T15:08:16.251 INFO:teuthology.orchestra.run.smithi016.stdout:9: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-31T15:08:16.251 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether c6:a6:3d:69:d0:db brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-31T15:08:16.251 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 fe80::c4a6:3dff:fe69:d0db/64 scope link 2024-03-31T15:08:16.251 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2024-03-31T15:08:16.251 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:08:16.251 DEBUG:teuthology.orchestra.run.smithi016:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-31T15:08:16.258 INFO:tasks.ceph.mds.a.smithi016.stderr:2024-03-31T15:08:16.262+0000 7f86baa00640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-31T15:08:16.276 INFO:teuthology.orchestra.run.smithi016.stdout:1 2024-03-31T15:08:16.278 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:08:16.278 DEBUG:teuthology.orchestra.run.smithi016:> route 2024-03-31T15:08:16.332 INFO:teuthology.orchestra.run.smithi016.stdout:Kernel IP routing table 2024-03-31T15:08:16.332 INFO:teuthology.orchestra.run.smithi016.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-31T15:08:16.332 INFO:teuthology.orchestra.run.smithi016.stdout:default _gateway 0.0.0.0 UG 0 0 0 ens1f0 2024-03-31T15:08:16.332 INFO:teuthology.orchestra.run.smithi016.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2024-03-31T15:08:16.332 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 0 0 0 ens1f0 2024-03-31T15:08:16.332 INFO:teuthology.orchestra.run.smithi016.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-31T15:08:16.333 DEBUG:teuthology.orchestra.run.smithi016:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-31T15:08:16.333 DEBUG:teuthology.orchestra.run.smithi016:> set -e 2024-03-31T15:08:16.333 DEBUG:teuthology.orchestra.run.smithi016:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-03-31T15:08:16.333 DEBUG:teuthology.orchestra.run.smithi016:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-03-31T15:08:16.333 DEBUG:teuthology.orchestra.run.smithi016:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-03-31T15:08:16.333 DEBUG:teuthology.orchestra.run.smithi016:> ') 2024-03-31T15:08:16.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:16 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-31T15:08:16.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:16 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-31T15:08:16.469 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:08:16.470 DEBUG:teuthology.orchestra.run.smithi016:> ip netns list 2024-03-31T15:08:16.479 INFO:teuthology.orchestra.run.smithi016.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-31T15:08:16.479 INFO:teuthology.orchestra.run.smithi016.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-31T15:08:16.479 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-31T15:08:16.479 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:08:16.479 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip link set brx.0 up 2024-03-31T15:08:16.542 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-03-31T15:08:16.543 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-31T15:08:16.543 DEBUG:teuthology.orchestra.run.smithi016:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-31T15:08:16.594 INFO:teuthology.orchestra.run.smithi016.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-31T15:08:16.595 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-31T15:08:16.595 DEBUG:teuthology.orchestra.run.smithi016:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-31T15:08:16.646 DEBUG:teuthology.orchestra.run.smithi016:> sudo modprobe fuse 2024-03-31T15:08:16.706 DEBUG:teuthology.orchestra.run.smithi016:> cat /proc/self/mounts | awk '{print $2}' 2024-03-31T15:08:16.756 INFO:teuthology.orchestra.run.smithi016.stdout:/sys 2024-03-31T15:08:16.756 INFO:teuthology.orchestra.run.smithi016.stdout:/proc 2024-03-31T15:08:16.756 INFO:teuthology.orchestra.run.smithi016.stdout:/dev 2024-03-31T15:08:16.756 INFO:teuthology.orchestra.run.smithi016.stdout:/dev/pts 2024-03-31T15:08:16.756 INFO:teuthology.orchestra.run.smithi016.stdout:/run 2024-03-31T15:08:16.756 INFO:teuthology.orchestra.run.smithi016.stdout:/ 2024-03-31T15:08:16.756 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/kernel/security 2024-03-31T15:08:16.756 INFO:teuthology.orchestra.run.smithi016.stdout:/dev/shm 2024-03-31T15:08:16.757 INFO:teuthology.orchestra.run.smithi016.stdout:/run/lock 2024-03-31T15:08:16.757 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/cgroup 2024-03-31T15:08:16.757 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/pstore 2024-03-31T15:08:16.757 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/bpf 2024-03-31T15:08:16.757 INFO:teuthology.orchestra.run.smithi016.stdout:/proc/sys/fs/binfmt_misc 2024-03-31T15:08:16.757 INFO:teuthology.orchestra.run.smithi016.stdout:/dev/hugepages 2024-03-31T15:08:16.757 INFO:teuthology.orchestra.run.smithi016.stdout:/dev/mqueue 2024-03-31T15:08:16.757 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/kernel/debug 2024-03-31T15:08:16.757 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/kernel/tracing 2024-03-31T15:08:16.757 INFO:teuthology.orchestra.run.smithi016.stdout:/proc/fs/nfsd 2024-03-31T15:08:16.757 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/fuse/connections 2024-03-31T15:08:16.757 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/kernel/config 2024-03-31T15:08:16.757 INFO:teuthology.orchestra.run.smithi016.stdout:/run/credentials/systemd-sysusers.service 2024-03-31T15:08:16.757 INFO:teuthology.orchestra.run.smithi016.stdout:/run/rpc_pipefs 2024-03-31T15:08:16.757 INFO:teuthology.orchestra.run.smithi016.stdout:/run/user/1000 2024-03-31T15:08:16.757 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/ceph 2024-03-31T15:08:16.758 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/ceph/osd/ceph-0 2024-03-31T15:08:16.758 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/ceph/osd/ceph-1 2024-03-31T15:08:16.758 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/ceph/osd/ceph-2 2024-03-31T15:08:16.758 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/ceph/osd/ceph-3 2024-03-31T15:08:16.758 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns 2024-03-31T15:08:16.758 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-31T15:08:16.758 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-31T15:08:16.758 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-31T15:08:16.758 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-31T15:08:16.759 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:08:16.759 DEBUG:teuthology.orchestra.run.smithi016:> ls /sys/fs/fuse/connections 2024-03-31T15:08:16.810 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-31T15:08:16.810 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-03-31T15:08:16.853 DEBUG:teuthology.orchestra.run.smithi016:> sudo modprobe fuse 2024-03-31T15:08:16.874 DEBUG:teuthology.orchestra.run.smithi016:> cat /proc/self/mounts | awk '{print $2}' 2024-03-31T15:08:16.925 INFO:teuthology.orchestra.run.smithi016.stdout:/sys 2024-03-31T15:08:16.925 INFO:teuthology.orchestra.run.smithi016.stdout:/proc 2024-03-31T15:08:16.925 INFO:teuthology.orchestra.run.smithi016.stdout:/dev 2024-03-31T15:08:16.925 INFO:teuthology.orchestra.run.smithi016.stdout:/dev/pts 2024-03-31T15:08:16.925 INFO:teuthology.orchestra.run.smithi016.stdout:/run 2024-03-31T15:08:16.925 INFO:teuthology.orchestra.run.smithi016.stdout:/ 2024-03-31T15:08:16.926 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/kernel/security 2024-03-31T15:08:16.926 INFO:teuthology.orchestra.run.smithi016.stdout:/dev/shm 2024-03-31T15:08:16.926 INFO:teuthology.orchestra.run.smithi016.stdout:/run/lock 2024-03-31T15:08:16.926 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/cgroup 2024-03-31T15:08:16.926 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/pstore 2024-03-31T15:08:16.926 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/bpf 2024-03-31T15:08:16.926 INFO:teuthology.orchestra.run.smithi016.stdout:/proc/sys/fs/binfmt_misc 2024-03-31T15:08:16.926 INFO:teuthology.orchestra.run.smithi016.stdout:/dev/hugepages 2024-03-31T15:08:16.927 INFO:teuthology.orchestra.run.smithi016.stdout:/dev/mqueue 2024-03-31T15:08:16.927 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/kernel/debug 2024-03-31T15:08:16.927 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/kernel/tracing 2024-03-31T15:08:16.927 INFO:teuthology.orchestra.run.smithi016.stdout:/proc/fs/nfsd 2024-03-31T15:08:16.927 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/fuse/connections 2024-03-31T15:08:16.927 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/kernel/config 2024-03-31T15:08:16.927 INFO:teuthology.orchestra.run.smithi016.stdout:/run/credentials/systemd-sysusers.service 2024-03-31T15:08:16.927 INFO:teuthology.orchestra.run.smithi016.stdout:/run/rpc_pipefs 2024-03-31T15:08:16.927 INFO:teuthology.orchestra.run.smithi016.stdout:/run/user/1000 2024-03-31T15:08:16.927 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/ceph 2024-03-31T15:08:16.928 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/ceph/osd/ceph-0 2024-03-31T15:08:16.928 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/ceph/osd/ceph-1 2024-03-31T15:08:16.928 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/ceph/osd/ceph-2 2024-03-31T15:08:16.928 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/ceph/osd/ceph-3 2024-03-31T15:08:16.928 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns 2024-03-31T15:08:16.928 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-31T15:08:16.928 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-31T15:08:16.928 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-31T15:08:16.928 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-31T15:08:16.929 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:08:16.930 DEBUG:teuthology.orchestra.run.smithi016:> ls /sys/fs/fuse/connections 2024-03-31T15:08:16.951 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi016.stderr:2024-03-31T15:08:16.954+0000 7f534ed7e580 -1 init, newargv = 0x5647e6e474b0 newargc=15 2024-03-31T15:08:16.951 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi016.stderr:ceph-fuse[42435]: starting ceph client 2024-03-31T15:08:16.959 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi016.stderr:ceph-fuse[42435]: starting fuse 2024-03-31T15:08:16.961 INFO:teuthology.orchestra.run.smithi016.stdout:43 2024-03-31T15:08:16.961 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [43] 2024-03-31T15:08:16.961 DEBUG:teuthology.orchestra.run.smithi016:> sudo stdin-killer -- python3 -c ' 2024-03-31T15:08:16.962 DEBUG:teuthology.orchestra.run.smithi016:> import glob 2024-03-31T15:08:16.962 DEBUG:teuthology.orchestra.run.smithi016:> import re 2024-03-31T15:08:16.962 DEBUG:teuthology.orchestra.run.smithi016:> import os 2024-03-31T15:08:16.962 DEBUG:teuthology.orchestra.run.smithi016:> import subprocess 2024-03-31T15:08:16.962 DEBUG:teuthology.orchestra.run.smithi016:> 2024-03-31T15:08:16.962 DEBUG:teuthology.orchestra.run.smithi016:> def _find_admin_socket(client_name): 2024-03-31T15:08:16.962 DEBUG:teuthology.orchestra.run.smithi016:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-03-31T15:08:16.962 DEBUG:teuthology.orchestra.run.smithi016:> files = glob.glob(asok_path) 2024-03-31T15:08:16.962 DEBUG:teuthology.orchestra.run.smithi016:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-03-31T15:08:16.962 DEBUG:teuthology.orchestra.run.smithi016:> 2024-03-31T15:08:16.962 DEBUG:teuthology.orchestra.run.smithi016:> # Given a non-glob path, it better be there 2024-03-31T15:08:16.962 DEBUG:teuthology.orchestra.run.smithi016:> if "*" not in asok_path: 2024-03-31T15:08:16.962 DEBUG:teuthology.orchestra.run.smithi016:> assert(len(files) == 1) 2024-03-31T15:08:16.962 DEBUG:teuthology.orchestra.run.smithi016:> return files[0] 2024-03-31T15:08:16.962 DEBUG:teuthology.orchestra.run.smithi016:> 2024-03-31T15:08:16.963 DEBUG:teuthology.orchestra.run.smithi016:> for f in files: 2024-03-31T15:08:16.963 DEBUG:teuthology.orchestra.run.smithi016:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-31T15:08:16.963 DEBUG:teuthology.orchestra.run.smithi016:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-31T15:08:16.963 DEBUG:teuthology.orchestra.run.smithi016:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-31T15:08:16.963 DEBUG:teuthology.orchestra.run.smithi016:> contents = proc_f.read() 2024-03-31T15:08:16.963 DEBUG:teuthology.orchestra.run.smithi016:> if mountpoint in contents: 2024-03-31T15:08:16.963 DEBUG:teuthology.orchestra.run.smithi016:> return f 2024-03-31T15:08:16.963 DEBUG:teuthology.orchestra.run.smithi016:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-31T15:08:16.963 DEBUG:teuthology.orchestra.run.smithi016:> 2024-03-31T15:08:16.963 DEBUG:teuthology.orchestra.run.smithi016:> print(_find_admin_socket("client.0")) 2024-03-31T15:08:16.963 DEBUG:teuthology.orchestra.run.smithi016:> ' 2024-03-31T15:08:17.085 INFO:teuthology.orchestra.run.smithi016.stdout:/var/run/ceph/ceph-client.0.42435.asok 2024-03-31T15:08:17.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:17 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-31T15:08:17.096 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.42435.asok 2024-03-31T15:08:17.096 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:08:17.096 DEBUG:teuthology.orchestra.run.smithi016:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.42435.asok status 2024-03-31T15:08:17.271 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-03-31T15:08:17.271 INFO:teuthology.orchestra.run.smithi016.stdout: "metadata": { 2024-03-31T15:08:17.271 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph_sha1": "d2271579d5e735cfd6d436b8349c004f06527cb0", 2024-03-31T15:08:17.271 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph_version": "ceph version 19.0.0-2574-gd2271579 (d2271579d5e735cfd6d436b8349c004f06527cb0) squid (dev)", 2024-03-31T15:08:17.272 INFO:teuthology.orchestra.run.smithi016.stdout: "entity_id": "0", 2024-03-31T15:08:17.272 INFO:teuthology.orchestra.run.smithi016.stdout: "hostname": "smithi016", 2024-03-31T15:08:17.272 INFO:teuthology.orchestra.run.smithi016.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-03-31T15:08:17.272 INFO:teuthology.orchestra.run.smithi016.stdout: "pid": "42435", 2024-03-31T15:08:17.272 INFO:teuthology.orchestra.run.smithi016.stdout: "root": "/" 2024-03-31T15:08:17.272 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-31T15:08:17.272 INFO:teuthology.orchestra.run.smithi016.stdout: "dentry_count": 0, 2024-03-31T15:08:17.272 INFO:teuthology.orchestra.run.smithi016.stdout: "dentry_pinned_count": 0, 2024-03-31T15:08:17.272 INFO:teuthology.orchestra.run.smithi016.stdout: "id": 7032, 2024-03-31T15:08:17.272 INFO:teuthology.orchestra.run.smithi016.stdout: "inst": { 2024-03-31T15:08:17.273 INFO:teuthology.orchestra.run.smithi016.stdout: "name": { 2024-03-31T15:08:17.273 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "client", 2024-03-31T15:08:17.273 INFO:teuthology.orchestra.run.smithi016.stdout: "num": 7032 2024-03-31T15:08:17.273 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-31T15:08:17.273 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": { 2024-03-31T15:08:17.273 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-03-31T15:08:17.273 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "192.168.0.1:0", 2024-03-31T15:08:17.273 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 1732960588 2024-03-31T15:08:17.273 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-31T15:08:17.273 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-31T15:08:17.273 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": { 2024-03-31T15:08:17.273 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-03-31T15:08:17.273 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "192.168.0.1:0", 2024-03-31T15:08:17.273 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 1732960588 2024-03-31T15:08:17.273 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-31T15:08:17.273 INFO:teuthology.orchestra.run.smithi016.stdout: "inst_str": "client.7032 192.168.0.1:0/1732960588", 2024-03-31T15:08:17.274 INFO:teuthology.orchestra.run.smithi016.stdout: "addr_str": "192.168.0.1:0/1732960588", 2024-03-31T15:08:17.274 INFO:teuthology.orchestra.run.smithi016.stdout: "inode_count": 1, 2024-03-31T15:08:17.274 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_epoch": 50, 2024-03-31T15:08:17.274 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_epoch": 68, 2024-03-31T15:08:17.274 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_epoch_barrier": 0, 2024-03-31T15:08:17.274 INFO:teuthology.orchestra.run.smithi016.stdout: "blocklisted": false, 2024-03-31T15:08:17.274 INFO:teuthology.orchestra.run.smithi016.stdout: "fs_name": "cephfs" 2024-03-31T15:08:17.274 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-03-31T15:08:17.290 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:08:17.290 DEBUG:teuthology.orchestra.run.smithi016:> stat --file-system '--printf=%T 2024-03-31T15:08:17.290 DEBUG:teuthology.orchestra.run.smithi016:> ' -- /home/ubuntu/cephtest/mnt.0 2024-03-31T15:08:17.299 INFO:teuthology.orchestra.run.smithi016.stdout:fuseblk 2024-03-31T15:08:17.299 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-03-31T15:08:17.299 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:08:17.299 DEBUG:teuthology.orchestra.run.smithi016:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-31T15:08:17.361 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:08:17.361 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls subdir/) 2024-03-31T15:08:17.416 INFO:teuthology.orchestra.run.smithi016.stdout:file_undamaged 2024-03-31T15:08:17.418 DEBUG:teuthology.orchestra.run.smithi016:> stdin-killer -- python3 -c ' 2024-03-31T15:08:17.418 DEBUG:teuthology.orchestra.run.smithi016:> import os 2024-03-31T15:08:17.418 DEBUG:teuthology.orchestra.run.smithi016:> import stat 2024-03-31T15:08:17.418 DEBUG:teuthology.orchestra.run.smithi016:> import json 2024-03-31T15:08:17.418 DEBUG:teuthology.orchestra.run.smithi016:> import sys 2024-03-31T15:08:17.418 DEBUG:teuthology.orchestra.run.smithi016:> 2024-03-31T15:08:17.419 DEBUG:teuthology.orchestra.run.smithi016:> try: 2024-03-31T15:08:17.419 DEBUG:teuthology.orchestra.run.smithi016:> s = os.stat('"'"'/home/ubuntu/cephtest/mnt.0/subdir/file_to_be_damaged'"'"') 2024-03-31T15:08:17.419 DEBUG:teuthology.orchestra.run.smithi016:> except OSError as e: 2024-03-31T15:08:17.419 DEBUG:teuthology.orchestra.run.smithi016:> sys.exit(e.errno) 2024-03-31T15:08:17.419 DEBUG:teuthology.orchestra.run.smithi016:> 2024-03-31T15:08:17.419 DEBUG:teuthology.orchestra.run.smithi016:> attrs = ["st_mode", "st_ino", "st_dev", "st_nlink", "st_uid", "st_gid", "st_size", "st_atime", "st_mtime", "st_ctime"] 2024-03-31T15:08:17.419 DEBUG:teuthology.orchestra.run.smithi016:> print(json.dumps( 2024-03-31T15:08:17.419 DEBUG:teuthology.orchestra.run.smithi016:> dict([(a, getattr(s, a)) for a in attrs]), 2024-03-31T15:08:17.419 DEBUG:teuthology.orchestra.run.smithi016:> indent=2)) 2024-03-31T15:08:17.419 DEBUG:teuthology.orchestra.run.smithi016:> ' 2024-03-31T15:08:17.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:17 stdin-killer INFO: command exited with status 2: exiting normally with same code! 2024-03-31T15:08:17.538 DEBUG:teuthology.orchestra.run:got remote process result: 2 2024-03-31T15:08:17.539 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:08:17.539 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr --only-values -n ceph.dir.files ./subdir) 2024-03-31T15:08:17.598 INFO:teuthology.orchestra.run.smithi016.stdout:2 2024-03-31T15:08:17.598 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-03-31T15:08:17.598 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-31T15:08:17.647 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-31T15:08:17.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:17.834+0000 7f6056200640 1 Processor -- start 2024-03-31T15:08:17.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:17.834+0000 7f6056200640 1 -- start start 2024-03-31T15:08:17.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:17.838+0000 7f6056200640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f605015e4c0 0x7f60500d0b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:17.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:17.838+0000 7f6056200640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f60500d2d10 0x7f60500d14a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:17.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:17.838+0000 7f6056200640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f605015e8a0 con 0x7f60500d1060 2024-03-31T15:08:17.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:17.838+0000 7f6056200640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f60500d0220 con 0x7f60500d2d10 2024-03-31T15:08:17.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:17.838+0000 7f6056200640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f60500d3180 con 0x7f605015e4c0 2024-03-31T15:08:17.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:17.838+0000 7f6055800640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f60500d2d10 0x7f60500d14a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:17.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:17.838+0000 7f6054e00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f605015e4c0 0x7f60500d0b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:17.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:17.838+0000 7f6055800640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f60500d2d10 0x7f60500d14a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.16:49922/0 (socket says 172.21.15.16:49922) 2024-03-31T15:08:17.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:17.838+0000 7f6055800640 1 -- 172.21.15.16:0/1934538798 learned_addr learned my addr 172.21.15.16:0/1934538798 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:08:17.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:17.838+0000 7f6055800640 1 -- 172.21.15.16:0/1934538798 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f605015e4c0 msgr2=0x7f60500d0b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:08:17.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:17.838+0000 7f6055800640 1 --2- 172.21.15.16:0/1934538798 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f605015e4c0 0x7f60500d0b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:17.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:17.838+0000 7f6055800640 1 -- 172.21.15.16:0/1934538798 >> v1:172.21.15.89:6789/0 conn(0x7f60500d1060 legacy=0x7f60500b3e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:17.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:17.838+0000 7f6055800640 1 -- 172.21.15.16:0/1934538798 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60500b3770 con 0x7f60500d2d10 2024-03-31T15:08:17.835 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:17.842+0000 7f6055800640 1 --2- 172.21.15.16:0/1934538798 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f60500d2d10 0x7f60500d14a0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f6048009e50 tx=0x7f604801ca30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6e3d62994736d37 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:08:17.836 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:17.842+0000 7f604fe00640 1 -- 172.21.15.16:0/1934538798 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60480a57f0 con 0x7f60500d2d10 2024-03-31T15:08:17.836 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:17.842+0000 7f604fe00640 1 -- 172.21.15.16:0/1934538798 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f60480a5990 con 0x7f60500d2d10 2024-03-31T15:08:17.836 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:17.842+0000 7f604fe00640 1 -- 172.21.15.16:0/1934538798 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60480a5d60 con 0x7f60500d2d10 2024-03-31T15:08:17.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:17.842+0000 7f6056200640 1 -- 172.21.15.16:0/1934538798 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f60500d2d10 msgr2=0x7f60500d14a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:17.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:17.842+0000 7f6056200640 1 --2- 172.21.15.16:0/1934538798 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f60500d2d10 0x7f60500d14a0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f6048009e50 tx=0x7f604801ca30 comp rx=0 tx=0).stop 2024-03-31T15:08:17.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:17.842+0000 7f6056200640 1 -- 172.21.15.16:0/1934538798 shutdown_connections 2024-03-31T15:08:17.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:17.842+0000 7f6056200640 1 --2- 172.21.15.16:0/1934538798 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f60500d2d10 0x7f60500d14a0 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:17.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:17.842+0000 7f6056200640 1 --2- 172.21.15.16:0/1934538798 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f605015e4c0 0x7f60500d0b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:17.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:17.842+0000 7f6056200640 1 -- 172.21.15.16:0/1934538798 >> 172.21.15.16:0/1934538798 conn(0x7f605009d470 msgr2=0x7f60500d4130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:08:17.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:17.842+0000 7f6056200640 1 -- 172.21.15.16:0/1934538798 shutdown_connections 2024-03-31T15:08:17.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:17.842+0000 7f6056200640 1 -- 172.21.15.16:0/1934538798 wait complete. 2024-03-31T15:08:17.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:17.842+0000 7f6056200640 1 Processor -- start 2024-03-31T15:08:17.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:17.842+0000 7f6056200640 1 -- start start 2024-03-31T15:08:17.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:17.842+0000 7f6056200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f60500d1060 0x7f6050155620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:17.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:17.842+0000 7f6056200640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f60500d2d10 0x7f6050158640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:17.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:17.842+0000 7f6056200640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f605015e4c0 0x7f605015a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:17.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:17.842+0000 7f6056200640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f60500b4ea0 con 0x7f60500d1060 2024-03-31T15:08:17.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:17.842+0000 7f6056200640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f60500b4d20 con 0x7f60500d2d10 2024-03-31T15:08:17.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:17.842+0000 7f6056200640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f60500b5020 con 0x7f605015e4c0 2024-03-31T15:08:17.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:17.846+0000 7f6054e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f60500d1060 0x7f6050155620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:17.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:17.846+0000 7f6054e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f60500d1060 0x7f6050155620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38572/0 (socket says 172.21.15.16:38572) 2024-03-31T15:08:17.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:17.846+0000 7f6054e00640 1 -- 172.21.15.16:0/2540704646 learned_addr learned my addr 172.21.15.16:0/2540704646 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:08:17.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:17.846+0000 7f6055800640 1 --2- 172.21.15.16:0/2540704646 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f605015e4c0 0x7f605015a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:17.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:17.846+0000 7f6047e00640 1 --2- 172.21.15.16:0/2540704646 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f60500d2d10 0x7f6050158640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:17.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:17.846+0000 7f6054e00640 1 -- 172.21.15.16:0/2540704646 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f605015e4c0 msgr2=0x7f605015a8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:17.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:17.846+0000 7f6054e00640 1 --2- 172.21.15.16:0/2540704646 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f605015e4c0 0x7f605015a8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:17.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:17.846+0000 7f6054e00640 1 -- 172.21.15.16:0/2540704646 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f60500d2d10 msgr2=0x7f6050158640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:17.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:17.846+0000 7f6054e00640 1 --2- 172.21.15.16:0/2540704646 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f60500d2d10 0x7f6050158640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:17.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:17.846+0000 7f6054e00640 1 -- 172.21.15.16:0/2540704646 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6050155b60 con 0x7f60500d1060 2024-03-31T15:08:17.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:17.846+0000 7f6054e00640 1 --2- 172.21.15.16:0/2540704646 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f60500d1060 0x7f6050155620 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7f604000e870 tx=0x7f604000ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:08:17.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:17.846+0000 7f604d600640 1 -- 172.21.15.16:0/2540704646 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6040017070 con 0x7f60500d1060 2024-03-31T15:08:17.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:17.846+0000 7f604d600640 1 -- 172.21.15.16:0/2540704646 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f604000c940 con 0x7f60500d1060 2024-03-31T15:08:17.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:17.846+0000 7f604d600640 1 -- 172.21.15.16:0/2540704646 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f604000ccf0 con 0x7f60500d1060 2024-03-31T15:08:17.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:17.846+0000 7f6056200640 1 -- 172.21.15.16:0/2540704646 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6050158fb0 con 0x7f60500d1060 2024-03-31T15:08:17.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:17.846+0000 7f6056200640 1 -- 172.21.15.16:0/2540704646 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6050159570 con 0x7f60500d1060 2024-03-31T15:08:17.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:17.846+0000 7f6056200640 1 -- 172.21.15.16:0/2540704646 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f605004f8c0 con 0x7f60500d1060 2024-03-31T15:08:17.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:17.846+0000 7f604d600640 1 -- 172.21.15.16:0/2540704646 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f604001a070 con 0x7f60500d1060 2024-03-31T15:08:17.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:17.850+0000 7f604d600640 1 --2- 172.21.15.16:0/2540704646 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f602c068380 0x7f602c06a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:17.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:17.850+0000 7f604d600640 1 -- 172.21.15.16:0/2540704646 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(69..69 src has 1..69) ==== 5882+0+0 (secure 0 0 0) 0x7f6040086890 con 0x7f60500d1060 2024-03-31T15:08:17.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:17.850+0000 7f6047e00640 1 --2- 172.21.15.16:0/2540704646 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f602c068380 0x7f602c06a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:17.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:17.850+0000 7f6047e00640 1 --2- 172.21.15.16:0/2540704646 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f602c068380 0x7f602c06a840 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f60500ce1c0 tx=0x7f6038009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:08:17.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:17.858+0000 7f604d600640 1 -- 172.21.15.16:0/2540704646 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f6040056740 con 0x7f60500d1060 2024-03-31T15:08:18.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:18.026+0000 7f6056200640 1 -- 172.21.15.16:0/2540704646 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f605008dbe0 con 0x7f60500d1060 2024-03-31T15:08:18.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:18.026+0000 7f604d600640 1 -- 172.21.15.16:0/2540704646 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v69) ==== 74+0+16870 (secure 0 0 0) 0x7f604005aa40 con 0x7f60500d1060 2024-03-31T15:08:18.023 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:08:18.023 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":69,"fsid":"286eb5da-fb47-4550-b290-d9519c13b9a5","created":"2024-03-31T15:03:11.438952+0000","modified":"2024-03-31T15:08:17.281541+0000","last_up_change":"2024-03-31T15:03:22.885535+0000","last_in_change":"2024-03-31T15:03:13.941667+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-31T15:03:22.550787+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-03-31T15:07:55.329096+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"66","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-03-31T15:07:55.885919+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"67","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"92a66450-ef19-40d8-ac3d-d760b23b0bda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6801","nonce":932712244}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6803","nonce":932712244}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6807","nonce":932712244}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6805","nonce":932712244}]},"public_addr":"172.21.15.16:6801/932712244","cluster_addr":"172.21.15.16:6803/932712244","heartbeat_back_addr":"172.21.15.16:6807/932712244","heartbeat_front_addr":"172.21.15.16:6805/932712244","state":["exists","up"]},{"osd":1,"uuid":"da9d1f2b-1968-4959-b843-faa231988e92","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6810","nonce":1398737533}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6811","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6813","nonce":1398737533}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6821","nonce":1398737533}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6815","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6816","nonce":1398737533}]},"public_addr":"172.21.15.16:6810/1398737533","cluster_addr":"172.21.15.16:6813/1398737533","heartbeat_back_addr":"172.21.15.16:6821/1398737533","heartbeat_front_addr":"172.21.15.16:6816/1398737533","state":["exists","up"]},{"osd":2,"uuid":"01a73662-9a21-4b1f-bcbb-42b9636a4638","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6819","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6822","nonce":3018526646}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6825","nonce":3018526646}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6831","nonce":3018526646}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6827","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6829","nonce":3018526646}]},"public_addr":"172.21.15.16:6822/3018526646","cluster_addr":"172.21.15.16:6825/3018526646","heartbeat_back_addr":"172.21.15.16:6831/3018526646","heartbeat_front_addr":"172.21.15.16:6829/3018526646","state":["exists","up"]},{"osd":3,"uuid":"c7771fd3-e18e-4500-ac38-23cf10ebcd75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6809","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6812","nonce":3360352974}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6817","nonce":3360352974}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6828","nonce":3360352974}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6823","nonce":3360352974}]},"public_addr":"172.21.15.16:6812/3360352974","cluster_addr":"172.21.15.16:6817/3360352974","heartbeat_back_addr":"172.21.15.16:6828/3360352974","heartbeat_front_addr":"172.21.15.16:6823/3360352974","state":["exists","up"]},{"osd":4,"uuid":"88260730-1482-4fbd-a40f-cb1124e9df4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6801","nonce":1856844856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6803","nonce":1856844856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6807","nonce":1856844856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6805","nonce":1856844856}]},"public_addr":"172.21.15.89:6801/1856844856","cluster_addr":"172.21.15.89:6803/1856844856","heartbeat_back_addr":"172.21.15.89:6807/1856844856","heartbeat_front_addr":"172.21.15.89:6805/1856844856","state":["exists","up"]},{"osd":5,"uuid":"4cb2c00c-fae0-4dd8-9d27-ec4f0093ccf7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6814","nonce":50395527}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6817","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6820","nonce":50395527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6831","nonce":50395527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6823","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6826","nonce":50395527}]},"public_addr":"172.21.15.89:6814/50395527","cluster_addr":"172.21.15.89:6820/50395527","heartbeat_back_addr":"172.21.15.89:6831/50395527","heartbeat_front_addr":"172.21.15.89:6826/50395527","state":["exists","up"]},{"osd":6,"uuid":"81ce0c19-2554-4cae-acbe-66a05e9e9b9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6809","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6812","nonce":3470471752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6815","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6818","nonce":3470471752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6827","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6829","nonce":3470471752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6821","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6824","nonce":3470471752}]},"public_addr":"172.21.15.89:6812/3470471752","cluster_addr":"172.21.15.89:6818/3470471752","heartbeat_back_addr":"172.21.15.89:6829/3470471752","heartbeat_front_addr":"172.21.15.89:6824/3470471752","state":["exists","up"]},{"osd":7,"uuid":"85e4313b-d37d-442e-8042-153bf20a847d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6811","nonce":1796778630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6813","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6816","nonce":1796778630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6825","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6828","nonce":1796778630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6819","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6822","nonce":1796778630}]},"public_addr":"172.21.15.89:6811/1796778630","cluster_addr":"172.21.15.89:6816/1796778630","heartbeat_back_addr":"172.21.15.89:6828/1796778630","heartbeat_front_addr":"172.21.15.89:6822/1796778630","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.439088+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.529970+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.680512+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:21.044161+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.582111+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.536558+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.409522+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.590229+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"11.4","mappings":[{"from":6,"to":0}]},{"pgid":"11.10","mappings":[{"from":6,"to":1}]},{"pgid":"11.3e","mappings":[{"from":6,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:6835/2555930820":"2024-04-01T15:08:10.537891+0000","172.21.15.16:6834/2555930820":"2024-04-01T15:08:10.537891+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-31T15:08:18.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:18.030+0000 7f6056200640 1 -- 172.21.15.16:0/2540704646 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f602c068380 msgr2=0x7f602c06a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:18.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:18.030+0000 7f6056200640 1 --2- 172.21.15.16:0/2540704646 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f602c068380 0x7f602c06a840 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f60500ce1c0 tx=0x7f6038009040 comp rx=0 tx=0).stop 2024-03-31T15:08:18.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:18.030+0000 7f6056200640 1 -- 172.21.15.16:0/2540704646 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f60500d1060 msgr2=0x7f6050155620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:18.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:18.030+0000 7f6056200640 1 --2- 172.21.15.16:0/2540704646 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f60500d1060 0x7f6050155620 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7f604000e870 tx=0x7f604000ec70 comp rx=0 tx=0).stop 2024-03-31T15:08:18.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:18.030+0000 7f6056200640 1 -- 172.21.15.16:0/2540704646 shutdown_connections 2024-03-31T15:08:18.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:18.030+0000 7f6056200640 1 --2- 172.21.15.16:0/2540704646 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f602c068380 0x7f602c06a840 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:18.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:18.030+0000 7f6056200640 1 --2- 172.21.15.16:0/2540704646 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f605015e4c0 0x7f605015a8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:18.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:18.030+0000 7f6056200640 1 --2- 172.21.15.16:0/2540704646 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f60500d2d10 0x7f6050158640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:18.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:18.030+0000 7f6056200640 1 --2- 172.21.15.16:0/2540704646 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f60500d1060 0x7f6050155620 unknown :-1 s=CLOSED pgs=901 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:18.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:18.030+0000 7f6056200640 1 -- 172.21.15.16:0/2540704646 >> 172.21.15.16:0/2540704646 conn(0x7f605009d470 msgr2=0x7f60500d03a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:08:18.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:18.030+0000 7f6056200640 1 -- 172.21.15.16:0/2540704646 shutdown_connections 2024-03-31T15:08:18.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:18.030+0000 7f6056200640 1 -- 172.21.15.16:0/2540704646 wait complete. 2024-03-31T15:08:18.045 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-31T15:03:22.550787+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 10, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-31T15:07:55.329096+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '66', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.5, 'score_stable': 2.5, 'optimal_score': 1, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 11, 'pool_name': 'cephfs_data', 'create_time': '2024-03-31T15:07:55.885919+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '67', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-31T15:08:18.045 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-31T15:08:18.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:18.194+0000 7f9aa1800640 1 Processor -- start 2024-03-31T15:08:18.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:18.194+0000 7f9aa1800640 1 -- start start 2024-03-31T15:08:18.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:18.194+0000 7f9aa1800640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f9a9c0d0a60 0x7f9a9c0cea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:18.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:18.194+0000 7f9aa1800640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f9a9c0cef80 0x7f9a9c1419c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:18.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:18.194+0000 7f9aa1800640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9a9c0cf430 con 0x7f9a9c06f920 2024-03-31T15:08:18.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:18.194+0000 7f9aa1800640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f9a9c13f670 con 0x7f9a9c0cef80 2024-03-31T15:08:18.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:18.194+0000 7f9aa1800640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f9a9c15ca70 con 0x7f9a9c0d0a60 2024-03-31T15:08:18.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:18.194+0000 7f9a9aa00640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f9a9c06f920 0x7f9a9c06fd00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:59972/0 (socket says 172.21.15.16:59972) 2024-03-31T15:08:18.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:18.194+0000 7f9a9aa00640 1 -- 172.21.15.16:0/2802270661 learned_addr learned my addr 172.21.15.16:0/2802270661 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:08:18.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:18.194+0000 7f9a9a000640 1 --2- 172.21.15.16:0/2802270661 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f9a9c0d0a60 0x7f9a9c0cea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:18.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:18.194+0000 7f9a99600640 1 -- 172.21.15.16:0/2802270661 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2591355182 0 0) 0x7f9a9c0cf430 con 0x7f9a9c06f920 2024-03-31T15:08:18.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:18.194+0000 7f9a9b400640 1 --2- 172.21.15.16:0/2802270661 >> v2:172.21.15.89:3300/0 conn(0x7f9a9c0cef80 0x7f9a9c1419c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:18.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:18.194+0000 7f9a99600640 1 -- 172.21.15.16:0/2802270661 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9a80003530 con 0x7f9a9c06f920 2024-03-31T15:08:18.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:18.194+0000 7f9a99600640 1 -- 172.21.15.16:0/2802270661 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2806967063 0 0) 0x7f9a80003530 con 0x7f9a9c06f920 2024-03-31T15:08:18.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:18.194+0000 7f9a99600640 1 -- 172.21.15.16:0/2802270661 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f9a9c0d0a60 msgr2=0x7f9a9c0cea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:08:18.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:18.194+0000 7f9a99600640 1 --2- 172.21.15.16:0/2802270661 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f9a9c0d0a60 0x7f9a9c0cea40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:18.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:18.194+0000 7f9a99600640 1 -- 172.21.15.16:0/2802270661 >> v2:172.21.15.89:3300/0 conn(0x7f9a9c0cef80 msgr2=0x7f9a9c1419c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:08:18.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:18.194+0000 7f9a99600640 1 --2- 172.21.15.16:0/2802270661 >> v2:172.21.15.89:3300/0 conn(0x7f9a9c0cef80 0x7f9a9c1419c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:18.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:18.194+0000 7f9a99600640 1 -- 172.21.15.16:0/2802270661 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a9c15cd50 con 0x7f9a9c06f920 2024-03-31T15:08:18.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:18.194+0000 7f9a99600640 1 -- 172.21.15.16:0/2802270661 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 214807623 0 0) 0x7f9a8c001680 con 0x7f9a9c06f920 2024-03-31T15:08:18.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:18.194+0000 7f9a99600640 1 -- 172.21.15.16:0/2802270661 >> v1:172.21.15.16:6789/0 conn(0x7f9a9c06f920 legacy=0x7f9a9c06fd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:18.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:18.194+0000 7f9a99600640 1 --2- 172.21.15.16:0/2802270661 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9a800039c0 0x7f9a80003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:18.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:18.194+0000 7f9a99600640 1 --2- 172.21.15.16:0/2802270661 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f9a9c0d0a60 0x7f9a80004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:18.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:18.194+0000 7f9a99600640 1 --2- 172.21.15.16:0/2802270661 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9a80004d50 0x7f9a800051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:18.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:18.194+0000 7f9a99600640 1 -- 172.21.15.16:0/2802270661 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f9a9c13f670 con 0x7f9a80004d50 2024-03-31T15:08:18.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:18.194+0000 7f9a99600640 1 -- 172.21.15.16:0/2802270661 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f9a9c15ca70 con 0x7f9a800039c0 2024-03-31T15:08:18.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:18.194+0000 7f9a99600640 1 -- 172.21.15.16:0/2802270661 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f9a8000b9d0 con 0x7f9a9c0d0a60 2024-03-31T15:08:18.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:18.194+0000 7f9a9b400640 1 --2- 172.21.15.16:0/2802270661 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9a80004d50 0x7f9a800051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:18.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:18.194+0000 7f9a9a000640 1 --2- 172.21.15.16:0/2802270661 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f9a9c0d0a60 0x7f9a80004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:18.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:18.194+0000 7f9a9aa00640 1 --2- 172.21.15.16:0/2802270661 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9a800039c0 0x7f9a80003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:18.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:18.194+0000 7f9a9b400640 1 -- 172.21.15.16:0/2802270661 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f9a9c0d0a60 msgr2=0x7f9a80004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:08:18.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:18.194+0000 7f9a9b400640 1 --2- 172.21.15.16:0/2802270661 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f9a9c0d0a60 0x7f9a80004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:18.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:18.194+0000 7f9a9b400640 1 -- 172.21.15.16:0/2802270661 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9a800039c0 msgr2=0x7f9a80003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:08:18.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:18.194+0000 7f9a9b400640 1 --2- 172.21.15.16:0/2802270661 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9a800039c0 0x7f9a80003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:18.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:18.194+0000 7f9a9b400640 1 -- 172.21.15.16:0/2802270661 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a80003530 con 0x7f9a80004d50 2024-03-31T15:08:18.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:18.198+0000 7f9a9b400640 1 --2- 172.21.15.16:0/2802270661 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9a80004d50 0x7f9a800051d0 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7f9a9400e170 tx=0x7f9a9401cca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=63fd0c156455ebe0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:08:18.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:18.198+0000 7f9a99600640 1 -- 172.21.15.16:0/2802270661 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a940b2070 con 0x7f9a80004d50 2024-03-31T15:08:18.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:18.198+0000 7f9a99600640 1 -- 172.21.15.16:0/2802270661 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9a940a5cb0 con 0x7f9a80004d50 2024-03-31T15:08:18.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:18.198+0000 7f9a99600640 1 -- 172.21.15.16:0/2802270661 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a94009c10 con 0x7f9a80004d50 2024-03-31T15:08:18.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:18.198+0000 7f9aa1800640 1 -- 172.21.15.16:0/2802270661 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9a80004d50 msgr2=0x7f9a800051d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:18.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:18.198+0000 7f9aa1800640 1 --2- 172.21.15.16:0/2802270661 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9a80004d50 0x7f9a800051d0 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7f9a9400e170 tx=0x7f9a9401cca0 comp rx=0 tx=0).stop 2024-03-31T15:08:18.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:18.198+0000 7f9a9b400640 1 -- 172.21.15.16:0/2802270661 reap_dead start 2024-03-31T15:08:18.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:18.198+0000 7f9aa1800640 1 -- 172.21.15.16:0/2802270661 shutdown_connections 2024-03-31T15:08:18.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:18.198+0000 7f9aa1800640 1 -- 172.21.15.16:0/2802270661 >> 172.21.15.16:0/2802270661 conn(0x7f9a9c067930 msgr2=0x7f9a9c0b3ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:08:18.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:18.198+0000 7f9aa1800640 1 -- 172.21.15.16:0/2802270661 shutdown_connections 2024-03-31T15:08:18.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:18.198+0000 7f9aa1800640 1 -- 172.21.15.16:0/2802270661 wait complete. 2024-03-31T15:08:18.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:18.202+0000 7f9aa1800640 1 Processor -- start 2024-03-31T15:08:18.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:18.202+0000 7f9aa1800640 1 -- start start 2024-03-31T15:08:18.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:18.202+0000 7f9aa1800640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9a80004d50 0x7f9a9c0b3880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:18.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:18.202+0000 7f9aa1800640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f9a9c1f5a30 0x7f9a9c1f5e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:18.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:18.202+0000 7f9aa1800640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9a9c134b50 0x7f9a9c134fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:18.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:18.202+0000 7f9aa1800640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f9a9c15c720 con 0x7f9a9c134b50 2024-03-31T15:08:18.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:18.202+0000 7f9aa1800640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f9a9c15d310 con 0x7f9a80004d50 2024-03-31T15:08:18.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:18.202+0000 7f9aa1800640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f9a9c0cea40 con 0x7f9a9c1f5a30 2024-03-31T15:08:18.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:18.202+0000 7f9a9b400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9a9c134b50 0x7f9a9c134fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:18.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:18.202+0000 7f9a9b400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9a9c134b50 0x7f9a9c134fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38588/0 (socket says 172.21.15.16:38588) 2024-03-31T15:08:18.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:18.202+0000 7f9a9b400640 1 -- 172.21.15.16:0/2419227339 learned_addr learned my addr 172.21.15.16:0/2419227339 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:08:18.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:18.202+0000 7f9a9aa00640 1 --2- 172.21.15.16:0/2419227339 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9a80004d50 0x7f9a9c0b3880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:18.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:18.202+0000 7f9a9a000640 1 --2- 172.21.15.16:0/2419227339 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f9a9c1f5a30 0x7f9a9c1f5e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:18.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:18.202+0000 7f9a9b400640 1 -- 172.21.15.16:0/2419227339 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f9a9c1f5a30 msgr2=0x7f9a9c1f5e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:18.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:18.202+0000 7f9a9b400640 1 --2- 172.21.15.16:0/2419227339 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f9a9c1f5a30 0x7f9a9c1f5e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:18.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:18.202+0000 7f9a9b400640 1 -- 172.21.15.16:0/2419227339 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9a80004d50 msgr2=0x7f9a9c0b3880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:18.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:18.202+0000 7f9a9b400640 1 --2- 172.21.15.16:0/2419227339 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9a80004d50 0x7f9a9c0b3880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:18.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:18.202+0000 7f9a9b400640 1 -- 172.21.15.16:0/2419227339 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a9c135510 con 0x7f9a9c134b50 2024-03-31T15:08:18.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:18.202+0000 7f9a9a000640 1 --2- 172.21.15.16:0/2419227339 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f9a9c1f5a30 0x7f9a9c1f5e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:08:18.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:18.202+0000 7f9a9b400640 1 --2- 172.21.15.16:0/2419227339 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9a9c134b50 0x7f9a9c134fd0 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7f9a9400e170 tx=0x7f9a94009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:08:18.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:18.202+0000 7f9a92a00640 1 -- 172.21.15.16:0/2419227339 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a940bd070 con 0x7f9a9c134b50 2024-03-31T15:08:18.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:18.202+0000 7f9a92a00640 1 -- 172.21.15.16:0/2419227339 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9a940a5ab0 con 0x7f9a9c134b50 2024-03-31T15:08:18.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:18.202+0000 7f9aa1800640 1 -- 172.21.15.16:0/2419227339 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9a9c1f6480 con 0x7f9a9c134b50 2024-03-31T15:08:18.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:18.202+0000 7f9a92a00640 1 -- 172.21.15.16:0/2419227339 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a940b2040 con 0x7f9a9c134b50 2024-03-31T15:08:18.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:18.202+0000 7f9aa1800640 1 -- 172.21.15.16:0/2419227339 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9a9c152d90 con 0x7f9a9c134b50 2024-03-31T15:08:18.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:18.206+0000 7f9aa1800640 1 -- 172.21.15.16:0/2419227339 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9a9c069950 con 0x7f9a9c134b50 2024-03-31T15:08:18.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:18.206+0000 7f9a92a00640 1 -- 172.21.15.16:0/2419227339 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f9a940c0070 con 0x7f9a9c134b50 2024-03-31T15:08:18.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:18.206+0000 7f9a92a00640 1 --2- 172.21.15.16:0/2419227339 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f9a7c068380 0x7f9a7c06a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:18.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:18.206+0000 7f9a92a00640 1 -- 172.21.15.16:0/2419227339 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(69..69 src has 1..69) ==== 5882+0+0 (secure 0 0 0) 0x7f9a94120640 con 0x7f9a9c134b50 2024-03-31T15:08:18.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:18.206+0000 7f9a9aa00640 1 --2- 172.21.15.16:0/2419227339 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f9a7c068380 0x7f9a7c06a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:18.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:18.206+0000 7f9a9aa00640 1 --2- 172.21.15.16:0/2419227339 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f9a7c068380 0x7f9a7c06a840 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f9a8c000fc0 tx=0x7f9a8c009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:08:18.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:18.214+0000 7f9a92a00640 1 -- 172.21.15.16:0/2419227339 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f9a94016a30 con 0x7f9a9c134b50 2024-03-31T15:08:18.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:18.386+0000 7f9aa1800640 1 -- 172.21.15.16:0/2419227339 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f9a9c0b50e0 con 0x7f9a9c134b50 2024-03-31T15:08:18.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:18.386+0000 7f9a92a00640 1 -- 172.21.15.16:0/2419227339 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v70) ==== 93+0+10 (secure 0 0 0) 0x7f9a940f03d0 con 0x7f9a9c134b50 2024-03-31T15:08:18.380 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2024-03-31T15:08:18.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:18.386+0000 7f9aa1800640 1 -- 172.21.15.16:0/2419227339 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f9a7c068380 msgr2=0x7f9a7c06a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:18.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:18.386+0000 7f9aa1800640 1 --2- 172.21.15.16:0/2419227339 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f9a7c068380 0x7f9a7c06a840 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f9a8c000fc0 tx=0x7f9a8c009040 comp rx=0 tx=0).stop 2024-03-31T15:08:18.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:18.386+0000 7f9aa1800640 1 -- 172.21.15.16:0/2419227339 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9a9c134b50 msgr2=0x7f9a9c134fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:18.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:18.386+0000 7f9aa1800640 1 --2- 172.21.15.16:0/2419227339 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9a9c134b50 0x7f9a9c134fd0 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7f9a9400e170 tx=0x7f9a94009750 comp rx=0 tx=0).stop 2024-03-31T15:08:18.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:18.386+0000 7f9aa1800640 1 -- 172.21.15.16:0/2419227339 shutdown_connections 2024-03-31T15:08:18.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:18.386+0000 7f9aa1800640 1 --2- 172.21.15.16:0/2419227339 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f9a7c068380 0x7f9a7c06a840 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:18.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:18.386+0000 7f9aa1800640 1 --2- 172.21.15.16:0/2419227339 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9a9c134b50 0x7f9a9c134fd0 unknown :-1 s=CLOSED pgs=904 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:18.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:18.386+0000 7f9aa1800640 1 --2- 172.21.15.16:0/2419227339 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f9a9c1f5a30 0x7f9a9c1f5e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:18.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:18.386+0000 7f9aa1800640 1 --2- 172.21.15.16:0/2419227339 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9a80004d50 0x7f9a9c0b3880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:18.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:18.386+0000 7f9aa1800640 1 -- 172.21.15.16:0/2419227339 >> 172.21.15.16:0/2419227339 conn(0x7f9a9c067930 msgr2=0x7f9a9c157b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:08:18.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:18.386+0000 7f9aa1800640 1 -- 172.21.15.16:0/2419227339 shutdown_connections 2024-03-31T15:08:18.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:18.386+0000 7f9aa1800640 1 -- 172.21.15.16:0/2419227339 wait complete. 2024-03-31T15:08:18.401 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-31T15:08:18.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:18.610+0000 7ff9d5c00640 1 Processor -- start 2024-03-31T15:08:18.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:18.614+0000 7ff9d5c00640 1 -- start start 2024-03-31T15:08:18.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:18.614+0000 7ff9d5c00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff9d006bd30 0x7ff9d00ae290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:18.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:18.614+0000 7ff9d5c00640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff9d015e920 con 0x7ff9d015e140 2024-03-31T15:08:18.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:18.614+0000 7ff9d5c00640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff9d015ec00 con 0x7ff9d00ae7d0 2024-03-31T15:08:18.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:18.614+0000 7ff9d5c00640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7ff9d015eee0 con 0x7ff9d006bd30 2024-03-31T15:08:18.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:18.614+0000 7ff9c7400640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff9d006bd30 0x7ff9d00ae290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:18.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:18.614+0000 7ff9cfe00640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7ff9d015e140 0x7ff9d006b620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:59984/0 (socket says 172.21.15.16:59984) 2024-03-31T15:08:18.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:18.614+0000 7ff9cfe00640 1 -- 172.21.15.16:0/2231298793 learned_addr learned my addr 172.21.15.16:0/2231298793 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:08:18.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:18.614+0000 7ff9c7400640 1 -- 172.21.15.16:0/2231298793 >> v1:172.21.15.89:6789/0 conn(0x7ff9d00ae7d0 legacy=0x7ff9d0072480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:08:18.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:18.614+0000 7ff9c7400640 1 -- 172.21.15.16:0/2231298793 >> v1:172.21.15.16:6789/0 conn(0x7ff9d015e140 legacy=0x7ff9d006b620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:18.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:18.614+0000 7ff9c7400640 1 -- 172.21.15.16:0/2231298793 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9d015f1c0 con 0x7ff9d006bd30 2024-03-31T15:08:18.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:18.614+0000 7ff9cf400640 1 -- 172.21.15.16:0/2231298793 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1295178634 0 0) 0x7ff9d015e920 con 0x7ff9d015e140 2024-03-31T15:08:18.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:18.618+0000 7ff9c7400640 1 --2- 172.21.15.16:0/2231298793 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff9d006bd30 0x7ff9d00ae290 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7ff9b8009d90 tx=0x7ff9b801c930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=68d7bd183421a4b9 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:08:18.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:18.618+0000 7ff9cf400640 1 -- 172.21.15.16:0/2231298793 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9b80b2070 con 0x7ff9d006bd30 2024-03-31T15:08:18.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:18.618+0000 7ff9cf400640 1 -- 172.21.15.16:0/2231298793 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff9b80a5900 con 0x7ff9d006bd30 2024-03-31T15:08:18.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:18.618+0000 7ff9cf400640 1 -- 172.21.15.16:0/2231298793 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9b80a5cd0 con 0x7ff9d006bd30 2024-03-31T15:08:18.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:18.618+0000 7ff9d5c00640 1 -- 172.21.15.16:0/2231298793 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff9d006bd30 msgr2=0x7ff9d00ae290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:18.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:18.618+0000 7ff9d5c00640 1 --2- 172.21.15.16:0/2231298793 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff9d006bd30 0x7ff9d00ae290 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7ff9b8009d90 tx=0x7ff9b801c930 comp rx=0 tx=0).stop 2024-03-31T15:08:18.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:18.618+0000 7ff9d5c00640 1 -- 172.21.15.16:0/2231298793 shutdown_connections 2024-03-31T15:08:18.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:18.618+0000 7ff9d5c00640 1 --2- 172.21.15.16:0/2231298793 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff9d006bd30 0x7ff9d00ae290 unknown :-1 s=CLOSED pgs=906 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:18.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:18.618+0000 7ff9d5c00640 1 -- 172.21.15.16:0/2231298793 >> 172.21.15.16:0/2231298793 conn(0x7ff9d00b2b30 msgr2=0x7ff9d00b2f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:08:18.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:18.618+0000 7ff9d5c00640 1 -- 172.21.15.16:0/2231298793 shutdown_connections 2024-03-31T15:08:18.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:18.618+0000 7ff9d5c00640 1 -- 172.21.15.16:0/2231298793 wait complete. 2024-03-31T15:08:18.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:18.618+0000 7ff9d5c00640 1 Processor -- start 2024-03-31T15:08:18.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:18.618+0000 7ff9d5c00640 1 -- start start 2024-03-31T15:08:18.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:18.622+0000 7ff9d5c00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff9d006bd30 0x7ff9d014e5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:18.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:18.622+0000 7ff9d5c00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff9d00ae7d0 0x7ff9d00b19f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:18.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:18.622+0000 7ff9d5c00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff9d015e140 0x7ff9d00b1f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:18.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:18.622+0000 7ff9d5c00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7ff9d016c120 con 0x7ff9d00ae7d0 2024-03-31T15:08:18.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:18.622+0000 7ff9d5c00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7ff9d016bfa0 con 0x7ff9d006bd30 2024-03-31T15:08:18.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:18.622+0000 7ff9d5c00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7ff9d016c2a0 con 0x7ff9d015e140 2024-03-31T15:08:18.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:18.622+0000 7ff9c7400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff9d00ae7d0 0x7ff9d00b19f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:18.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:18.622+0000 7ff9c7400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff9d00ae7d0 0x7ff9d00b19f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38608/0 (socket says 172.21.15.16:38608) 2024-03-31T15:08:18.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:18.622+0000 7ff9c7400640 1 -- 172.21.15.16:0/4052179474 learned_addr learned my addr 172.21.15.16:0/4052179474 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:08:18.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:18.622+0000 7ff9cfe00640 1 --2- 172.21.15.16:0/4052179474 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff9d006bd30 0x7ff9d014e5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:18.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:18.622+0000 7ff9d5200640 1 --2- 172.21.15.16:0/4052179474 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff9d015e140 0x7ff9d00b1f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:18.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:18.622+0000 7ff9c7400640 1 -- 172.21.15.16:0/4052179474 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff9d015e140 msgr2=0x7ff9d00b1f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:18.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:18.622+0000 7ff9c7400640 1 --2- 172.21.15.16:0/4052179474 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff9d015e140 0x7ff9d00b1f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:18.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:18.622+0000 7ff9c7400640 1 -- 172.21.15.16:0/4052179474 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff9d006bd30 msgr2=0x7ff9d014e5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:18.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:18.622+0000 7ff9c7400640 1 --2- 172.21.15.16:0/4052179474 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff9d006bd30 0x7ff9d014e5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:18.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:18.622+0000 7ff9c7400640 1 -- 172.21.15.16:0/4052179474 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9d0138f40 con 0x7ff9d00ae7d0 2024-03-31T15:08:18.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:18.622+0000 7ff9cfe00640 1 --2- 172.21.15.16:0/4052179474 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff9d006bd30 0x7ff9d014e5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:08:18.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:18.622+0000 7ff9d5200640 1 --2- 172.21.15.16:0/4052179474 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff9d015e140 0x7ff9d00b1f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:08:18.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:18.622+0000 7ff9c7400640 1 --2- 172.21.15.16:0/4052179474 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff9d00ae7d0 0x7ff9d00b19f0 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7ff9b80b0040 tx=0x7ff9b80196d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:08:18.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:18.622+0000 7ff9ccc00640 1 -- 172.21.15.16:0/4052179474 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9b80b2040 con 0x7ff9d00ae7d0 2024-03-31T15:08:18.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:18.622+0000 7ff9ccc00640 1 -- 172.21.15.16:0/4052179474 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff9b80bf070 con 0x7ff9d00ae7d0 2024-03-31T15:08:18.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:18.622+0000 7ff9ccc00640 1 -- 172.21.15.16:0/4052179474 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9b80ad8e0 con 0x7ff9d00ae7d0 2024-03-31T15:08:18.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:18.622+0000 7ff9d5c00640 1 -- 172.21.15.16:0/4052179474 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff9d00cd8f0 con 0x7ff9d00ae7d0 2024-03-31T15:08:18.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:18.622+0000 7ff9d5c00640 1 -- 172.21.15.16:0/4052179474 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff9d00cdeb0 con 0x7ff9d00ae7d0 2024-03-31T15:08:18.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:18.622+0000 7ff9d5c00640 1 -- 172.21.15.16:0/4052179474 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff9d004f8c0 con 0x7ff9d00ae7d0 2024-03-31T15:08:18.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:18.622+0000 7ff9ccc00640 1 -- 172.21.15.16:0/4052179474 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7ff9b80ada80 con 0x7ff9d00ae7d0 2024-03-31T15:08:18.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:18.626+0000 7ff9ccc00640 1 --2- 172.21.15.16:0/4052179474 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7ff9ac068380 0x7ff9ac06a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:18.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:18.626+0000 7ff9ccc00640 1 -- 172.21.15.16:0/4052179474 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(70..70 src has 1..70) ==== 5882+0+0 (secure 0 0 0) 0x7ff9b8121cf0 con 0x7ff9d00ae7d0 2024-03-31T15:08:18.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:18.626+0000 7ff9cfe00640 1 --2- 172.21.15.16:0/4052179474 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7ff9ac068380 0x7ff9ac06a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:18.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:18.626+0000 7ff9cfe00640 1 --2- 172.21.15.16:0/4052179474 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7ff9ac068380 0x7ff9ac06a840 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7ff9c0007fd0 tx=0x7ff9c0007c20 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:08:18.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:18.634+0000 7ff9ccc00640 1 -- 172.21.15.16:0/4052179474 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7ff9b8016520 con 0x7ff9d00ae7d0 2024-03-31T15:08:18.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:18.798+0000 7ff9d5c00640 1 -- 172.21.15.16:0/4052179474 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7ff9d015a310 con 0x7ff9d00ae7d0 2024-03-31T15:08:18.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:18.798+0000 7ff9ccc00640 1 -- 172.21.15.16:0/4052179474 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v70) ==== 104+0+11 (secure 0 0 0) 0x7ff9b80f2170 con 0x7ff9d00ae7d0 2024-03-31T15:08:18.794 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2024-03-31T15:08:18.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:18.802+0000 7ff9d5c00640 1 -- 172.21.15.16:0/4052179474 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7ff9ac068380 msgr2=0x7ff9ac06a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:18.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:18.802+0000 7ff9d5c00640 1 --2- 172.21.15.16:0/4052179474 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7ff9ac068380 0x7ff9ac06a840 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7ff9c0007fd0 tx=0x7ff9c0007c20 comp rx=0 tx=0).stop 2024-03-31T15:08:18.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:18.802+0000 7ff9d5c00640 1 -- 172.21.15.16:0/4052179474 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff9d00ae7d0 msgr2=0x7ff9d00b19f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:18.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:18.802+0000 7ff9d5c00640 1 --2- 172.21.15.16:0/4052179474 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff9d00ae7d0 0x7ff9d00b19f0 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7ff9b80b0040 tx=0x7ff9b80196d0 comp rx=0 tx=0).stop 2024-03-31T15:08:18.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:18.802+0000 7ff9d5c00640 1 -- 172.21.15.16:0/4052179474 shutdown_connections 2024-03-31T15:08:18.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:18.802+0000 7ff9d5c00640 1 --2- 172.21.15.16:0/4052179474 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7ff9ac068380 0x7ff9ac06a840 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:18.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:18.802+0000 7ff9d5c00640 1 --2- 172.21.15.16:0/4052179474 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff9d015e140 0x7ff9d00b1f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:18.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:18.802+0000 7ff9d5c00640 1 --2- 172.21.15.16:0/4052179474 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff9d00ae7d0 0x7ff9d00b19f0 unknown :-1 s=CLOSED pgs=907 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:18.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:18.802+0000 7ff9d5c00640 1 --2- 172.21.15.16:0/4052179474 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff9d006bd30 0x7ff9d014e5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:18.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:18.802+0000 7ff9d5c00640 1 -- 172.21.15.16:0/4052179474 >> 172.21.15.16:0/4052179474 conn(0x7ff9d00b2b30 msgr2=0x7ff9d00b4160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:08:18.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:18.802+0000 7ff9d5c00640 1 -- 172.21.15.16:0/4052179474 shutdown_connections 2024-03-31T15:08:18.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:18.802+0000 7ff9d5c00640 1 -- 172.21.15.16:0/4052179474 wait complete. 2024-03-31T15:08:18.815 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-31T15:08:18.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:18.962+0000 7f4ba9e00640 1 Processor -- start 2024-03-31T15:08:18.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:18.962+0000 7f4ba9e00640 1 -- start start 2024-03-31T15:08:18.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:18.962+0000 7f4ba9e00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f4ba41411b0 0x7f4ba406f920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:18.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:18.962+0000 7f4ba9e00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f4ba4141590 0x7f4ba406fe60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:18.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:18.962+0000 7f4ba9e00640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4ba4144980 con 0x7f4ba4143c30 2024-03-31T15:08:18.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:18.962+0000 7f4ba9e00640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f4ba415f910 con 0x7f4ba41411b0 2024-03-31T15:08:18.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:18.962+0000 7f4ba9e00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f4ba4162910 con 0x7f4ba4141590 2024-03-31T15:08:18.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:18.962+0000 7f4b9a000640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f4ba4141590 0x7f4ba406fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:18.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:18.962+0000 7f4ba2a00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f4ba41411b0 0x7f4ba406f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:18.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:18.962+0000 7f4b9a000640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f4ba4141590 0x7f4ba406fe60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3301/0 says I am v2:172.21.15.16:51470/0 (socket says 172.21.15.16:51470) 2024-03-31T15:08:18.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:18.962+0000 7f4b9a000640 1 -- 172.21.15.16:0/618898384 learned_addr learned my addr 172.21.15.16:0/618898384 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:08:18.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:18.966+0000 7f4b9a000640 1 -- 172.21.15.16:0/618898384 >> v2:172.21.15.89:3300/0 conn(0x7f4ba41411b0 msgr2=0x7f4ba406f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:08:18.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:18.966+0000 7f4b9a000640 1 --2- 172.21.15.16:0/618898384 >> v2:172.21.15.89:3300/0 conn(0x7f4ba41411b0 0x7f4ba406f920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:18.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:18.966+0000 7f4b9a000640 1 -- 172.21.15.16:0/618898384 >> v1:172.21.15.89:6789/0 conn(0x7f4ba4143c30 legacy=0x7f4ba41440a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:18.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:18.966+0000 7f4b9a000640 1 -- 172.21.15.16:0/618898384 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ba4161fe0 con 0x7f4ba4141590 2024-03-31T15:08:18.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:18.966+0000 7f4ba2a00640 1 --2- 172.21.15.16:0/618898384 >> v2:172.21.15.89:3300/0 conn(0x7f4ba41411b0 0x7f4ba406f920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:08:18.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:18.966+0000 7f4b9a000640 1 --2- 172.21.15.16:0/618898384 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f4ba4141590 0x7f4ba406fe60 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f4b8c009870 tx=0x7f4b8c01c560 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2165aa4e2d89ca1a server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:08:18.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:18.966+0000 7f4ba2000640 1 -- 172.21.15.16:0/618898384 <== mon.2 v2:172.21.15.89:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b8c0ab650 con 0x7f4ba4141590 2024-03-31T15:08:18.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:18.966+0000 7f4ba2000640 1 -- 172.21.15.16:0/618898384 <== mon.2 v2:172.21.15.89:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4b8c0b0070 con 0x7f4ba4141590 2024-03-31T15:08:18.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:18.966+0000 7f4ba2000640 1 -- 172.21.15.16:0/618898384 <== mon.2 v2:172.21.15.89:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b8c0aba10 con 0x7f4ba4141590 2024-03-31T15:08:18.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:18.966+0000 7f4ba9e00640 1 -- 172.21.15.16:0/618898384 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f4ba4141590 msgr2=0x7f4ba406fe60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:18.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:18.966+0000 7f4ba9e00640 1 --2- 172.21.15.16:0/618898384 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f4ba4141590 0x7f4ba406fe60 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f4b8c009870 tx=0x7f4b8c01c560 comp rx=0 tx=0).stop 2024-03-31T15:08:18.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:18.966+0000 7f4ba9e00640 1 -- 172.21.15.16:0/618898384 shutdown_connections 2024-03-31T15:08:18.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:18.966+0000 7f4ba9e00640 1 --2- 172.21.15.16:0/618898384 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f4ba4141590 0x7f4ba406fe60 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:18.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:18.966+0000 7f4ba9e00640 1 --2- 172.21.15.16:0/618898384 >> v2:172.21.15.89:3300/0 conn(0x7f4ba41411b0 0x7f4ba406f920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:18.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:18.966+0000 7f4ba9e00640 1 -- 172.21.15.16:0/618898384 >> 172.21.15.16:0/618898384 conn(0x7f4ba4067930 msgr2=0x7f4ba40b3ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:08:18.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:18.970+0000 7f4ba9e00640 1 -- 172.21.15.16:0/618898384 shutdown_connections 2024-03-31T15:08:18.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:18.970+0000 7f4ba9e00640 1 -- 172.21.15.16:0/618898384 wait complete. 2024-03-31T15:08:18.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:18.970+0000 7f4ba9e00640 1 Processor -- start 2024-03-31T15:08:18.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:18.970+0000 7f4ba9e00640 1 -- start start 2024-03-31T15:08:18.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:18.970+0000 7f4ba9e00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f4ba41411b0 0x7f4ba415a710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:18.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:18.970+0000 7f4ba9e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4ba4141590 0x7f4ba413ebd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:18.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:18.970+0000 7f4ba9e00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4ba4143c30 0x7f4ba4154930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:18.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:18.970+0000 7f4ba9e00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f4ba415f290 con 0x7f4ba4141590 2024-03-31T15:08:18.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:18.970+0000 7f4ba9e00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f4ba415f110 con 0x7f4ba4143c30 2024-03-31T15:08:18.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:18.970+0000 7f4ba9e00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f4ba415f410 con 0x7f4ba41411b0 2024-03-31T15:08:18.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:18.970+0000 7f4b9a000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4ba4141590 0x7f4ba413ebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:18.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:18.970+0000 7f4b9a000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4ba4141590 0x7f4ba413ebd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38610/0 (socket says 172.21.15.16:38610) 2024-03-31T15:08:18.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:18.970+0000 7f4b9a000640 1 -- 172.21.15.16:0/579457421 learned_addr learned my addr 172.21.15.16:0/579457421 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:08:18.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:18.970+0000 7f4ba2a00640 1 --2- 172.21.15.16:0/579457421 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f4ba41411b0 0x7f4ba415a710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:18.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:18.970+0000 7f4ba3400640 1 --2- 172.21.15.16:0/579457421 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4ba4143c30 0x7f4ba4154930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:18.968 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:18.970+0000 7f4b9a000640 1 -- 172.21.15.16:0/579457421 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f4ba41411b0 msgr2=0x7f4ba415a710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:18.968 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:18.970+0000 7f4b9a000640 1 --2- 172.21.15.16:0/579457421 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f4ba41411b0 0x7f4ba415a710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:18.968 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:18.970+0000 7f4b9a000640 1 -- 172.21.15.16:0/579457421 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4ba4143c30 msgr2=0x7f4ba4154930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:18.968 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:18.970+0000 7f4b9a000640 1 --2- 172.21.15.16:0/579457421 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4ba4143c30 0x7f4ba4154930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:18.968 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:18.970+0000 7f4b9a000640 1 -- 172.21.15.16:0/579457421 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ba413f340 con 0x7f4ba4141590 2024-03-31T15:08:18.968 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:18.970+0000 7f4b9a000640 1 --2- 172.21.15.16:0/579457421 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4ba4141590 0x7f4ba413ebd0 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7f4b8c0ad040 tx=0x7f4b8c01ada0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:08:18.968 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:18.970+0000 7f4b9be00640 1 -- 172.21.15.16:0/579457421 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b8c0be5f0 con 0x7f4ba4141590 2024-03-31T15:08:18.968 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:18.970+0000 7f4b9be00640 1 -- 172.21.15.16:0/579457421 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4b8c0be7b0 con 0x7f4ba4141590 2024-03-31T15:08:18.968 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:18.970+0000 7f4ba9e00640 1 -- 172.21.15.16:0/579457421 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4ba413f620 con 0x7f4ba4141590 2024-03-31T15:08:18.968 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:18.970+0000 7f4b9be00640 1 -- 172.21.15.16:0/579457421 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b8c0beba0 con 0x7f4ba4141590 2024-03-31T15:08:18.968 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:18.974+0000 7f4ba9e00640 1 -- 172.21.15.16:0/579457421 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4ba413fbe0 con 0x7f4ba4141590 2024-03-31T15:08:18.969 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:18.974+0000 7f4ba9e00640 1 -- 172.21.15.16:0/579457421 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4ba4069950 con 0x7f4ba4141590 2024-03-31T15:08:18.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:18.974+0000 7f4b9be00640 1 -- 172.21.15.16:0/579457421 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f4b8c0b43f0 con 0x7f4ba4141590 2024-03-31T15:08:18.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:18.974+0000 7f4b9be00640 1 --2- 172.21.15.16:0/579457421 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f4b80068380 0x7f4b8006a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:18.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:18.974+0000 7f4b9be00640 1 -- 172.21.15.16:0/579457421 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(70..70 src has 1..70) ==== 5882+0+0 (secure 0 0 0) 0x7f4b8c1217e0 con 0x7f4ba4141590 2024-03-31T15:08:18.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:18.978+0000 7f4ba2a00640 1 --2- 172.21.15.16:0/579457421 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f4b80068380 0x7f4b8006a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:18.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:18.978+0000 7f4ba2a00640 1 --2- 172.21.15.16:0/579457421 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f4b80068380 0x7f4b8006a840 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f4b94009580 tx=0x7f4b94009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:08:18.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:18.978+0000 7f4b9be00640 1 -- 172.21.15.16:0/579457421 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f4b8c0c4060 con 0x7f4ba4141590 2024-03-31T15:08:19.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:19.142+0000 7f4ba9e00640 1 -- 172.21.15.16:0/579457421 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f4ba4141d30 con 0x7f4ba4141590 2024-03-31T15:08:19.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:19.146+0000 7f4b9be00640 1 -- 172.21.15.16:0/579457421 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v70) ==== 100+0+11 (secure 0 0 0) 0x7f4b8c016150 con 0x7f4ba4141590 2024-03-31T15:08:19.140 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 64 2024-03-31T15:08:19.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:19.146+0000 7f4ba9e00640 1 -- 172.21.15.16:0/579457421 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f4b80068380 msgr2=0x7f4b8006a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:19.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:19.146+0000 7f4ba9e00640 1 --2- 172.21.15.16:0/579457421 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f4b80068380 0x7f4b8006a840 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f4b94009580 tx=0x7f4b94009190 comp rx=0 tx=0).stop 2024-03-31T15:08:19.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:19.146+0000 7f4ba9e00640 1 -- 172.21.15.16:0/579457421 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4ba4141590 msgr2=0x7f4ba413ebd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:19.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:19.146+0000 7f4ba9e00640 1 --2- 172.21.15.16:0/579457421 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4ba4141590 0x7f4ba413ebd0 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7f4b8c0ad040 tx=0x7f4b8c01ada0 comp rx=0 tx=0).stop 2024-03-31T15:08:19.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:19.146+0000 7f4ba9e00640 1 -- 172.21.15.16:0/579457421 shutdown_connections 2024-03-31T15:08:19.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:19.146+0000 7f4ba9e00640 1 --2- 172.21.15.16:0/579457421 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f4b80068380 0x7f4b8006a840 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:19.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:19.146+0000 7f4ba9e00640 1 --2- 172.21.15.16:0/579457421 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4ba4143c30 0x7f4ba4154930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:19.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:19.146+0000 7f4ba9e00640 1 --2- 172.21.15.16:0/579457421 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4ba4141590 0x7f4ba413ebd0 unknown :-1 s=CLOSED pgs=908 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:19.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:19.146+0000 7f4ba9e00640 1 --2- 172.21.15.16:0/579457421 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f4ba41411b0 0x7f4ba415a710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:19.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:19.146+0000 7f4ba9e00640 1 -- 172.21.15.16:0/579457421 >> 172.21.15.16:0/579457421 conn(0x7f4ba4067930 msgr2=0x7f4ba40704e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:08:19.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:19.146+0000 7f4ba9e00640 1 -- 172.21.15.16:0/579457421 shutdown_connections 2024-03-31T15:08:19.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:19.146+0000 7f4ba9e00640 1 -- 172.21.15.16:0/579457421 wait complete. 2024-03-31T15:08:19.162 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-31T15:08:19.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:19.306+0000 7f83cce00640 1 Processor -- start 2024-03-31T15:08:19.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:19.310+0000 7f83cce00640 1 -- start start 2024-03-31T15:08:19.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:19.310+0000 7f83cce00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f83c806f920 0x7f83c806fd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:19.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:19.310+0000 7f83cce00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f83c806d490 0x7f83c806d8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:19.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:19.310+0000 7f83cce00640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f83c8162010 con 0x7f83c80cb440 2024-03-31T15:08:19.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:19.310+0000 7f83cce00640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f83c8162be0 con 0x7f83c806d490 2024-03-31T15:08:19.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:19.310+0000 7f83cce00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f83c8070240 con 0x7f83c806f920 2024-03-31T15:08:19.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:19.310+0000 7f83bfe00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f83c806d490 0x7f83c806d8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:19.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:19.310+0000 7f83bfe00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f83c806d490 0x7f83c806d8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38626/0 (socket says 172.21.15.16:38626) 2024-03-31T15:08:19.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:19.310+0000 7f83bfe00640 1 -- 172.21.15.16:0/840208258 learned_addr learned my addr 172.21.15.16:0/840208258 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:08:19.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:19.310+0000 7f83c5e00640 1 --2- 172.21.15.16:0/840208258 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f83c806f920 0x7f83c806fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:19.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:19.310+0000 7f83bfe00640 1 -- 172.21.15.16:0/840208258 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f83c806f920 msgr2=0x7f83c806fd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:08:19.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:19.310+0000 7f83bfe00640 1 --2- 172.21.15.16:0/840208258 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f83c806f920 0x7f83c806fd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:19.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:19.310+0000 7f83bfe00640 1 -- 172.21.15.16:0/840208258 >> v1:172.21.15.16:6789/0 conn(0x7f83c80cb440 legacy=0x7f83c80cb8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:19.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:19.310+0000 7f83bfe00640 1 -- 172.21.15.16:0/840208258 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83c81622b0 con 0x7f83c806d490 2024-03-31T15:08:19.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:19.314+0000 7f83bfe00640 1 --2- 172.21.15.16:0/840208258 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f83c806d490 0x7f83c806d8f0 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7f83c8162d60 tx=0x7f83b001c510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4a4274837903af1 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:08:19.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:19.314+0000 7f83c5400640 1 -- 172.21.15.16:0/840208258 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83b00ab620 con 0x7f83c806d490 2024-03-31T15:08:19.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:19.314+0000 7f83c5400640 1 -- 172.21.15.16:0/840208258 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f83b00b0070 con 0x7f83c806d490 2024-03-31T15:08:19.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:19.314+0000 7f83c5400640 1 -- 172.21.15.16:0/840208258 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83b00ab9e0 con 0x7f83c806d490 2024-03-31T15:08:19.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:19.314+0000 7f83cce00640 1 -- 172.21.15.16:0/840208258 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f83c806d490 msgr2=0x7f83c806d8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:19.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:19.314+0000 7f83cce00640 1 --2- 172.21.15.16:0/840208258 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f83c806d490 0x7f83c806d8f0 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7f83c8162d60 tx=0x7f83b001c510 comp rx=0 tx=0).stop 2024-03-31T15:08:19.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:19.314+0000 7f83cce00640 1 -- 172.21.15.16:0/840208258 shutdown_connections 2024-03-31T15:08:19.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:19.314+0000 7f83cce00640 1 --2- 172.21.15.16:0/840208258 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f83c806d490 0x7f83c806d8f0 unknown :-1 s=CLOSED pgs=910 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:19.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:19.314+0000 7f83cce00640 1 --2- 172.21.15.16:0/840208258 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f83c806f920 0x7f83c806fd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:19.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:19.314+0000 7f83cce00640 1 -- 172.21.15.16:0/840208258 >> 172.21.15.16:0/840208258 conn(0x7f83c8067330 msgr2=0x7f83c8067740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:08:19.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:19.314+0000 7f83cce00640 1 -- 172.21.15.16:0/840208258 shutdown_connections 2024-03-31T15:08:19.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:19.314+0000 7f83cce00640 1 -- 172.21.15.16:0/840208258 wait complete. 2024-03-31T15:08:19.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:19.314+0000 7f83cce00640 1 Processor -- start 2024-03-31T15:08:19.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:19.314+0000 7f83cce00640 1 -- start start 2024-03-31T15:08:19.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:19.318+0000 7f83cce00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f83c806d490 0x7f83c8160750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:19.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:19.318+0000 7f83cce00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f83c806f920 0x7f83c813adb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:19.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:19.318+0000 7f83cce00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f83c80cb440 0x7f83c80b7430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:19.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:19.318+0000 7f83cce00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f83c806dfb0 con 0x7f83c806d490 2024-03-31T15:08:19.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:19.318+0000 7f83cce00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f83c806de30 con 0x7f83c806f920 2024-03-31T15:08:19.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:19.318+0000 7f83cce00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f83c80cd750 con 0x7f83c80cb440 2024-03-31T15:08:19.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:19.318+0000 7f83c5e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f83c806d490 0x7f83c8160750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:19.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:19.318+0000 7f83c5e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f83c806d490 0x7f83c8160750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38632/0 (socket says 172.21.15.16:38632) 2024-03-31T15:08:19.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:19.318+0000 7f83c5e00640 1 -- 172.21.15.16:0/1972556274 learned_addr learned my addr 172.21.15.16:0/1972556274 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:08:19.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:19.318+0000 7f83bfe00640 1 --2- 172.21.15.16:0/1972556274 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f83c806f920 0x7f83c813adb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:19.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:19.318+0000 7f83c6800640 1 --2- 172.21.15.16:0/1972556274 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f83c80cb440 0x7f83c80b7430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:19.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:19.318+0000 7f83c5e00640 1 -- 172.21.15.16:0/1972556274 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f83c80cb440 msgr2=0x7f83c80b7430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:19.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:19.318+0000 7f83c5e00640 1 --2- 172.21.15.16:0/1972556274 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f83c80cb440 0x7f83c80b7430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:19.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:19.318+0000 7f83c5e00640 1 -- 172.21.15.16:0/1972556274 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f83c806f920 msgr2=0x7f83c813adb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:19.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:19.318+0000 7f83c5e00640 1 --2- 172.21.15.16:0/1972556274 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f83c806f920 0x7f83c813adb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:19.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:19.318+0000 7f83c5e00640 1 -- 172.21.15.16:0/1972556274 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83c8070040 con 0x7f83c806d490 2024-03-31T15:08:19.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:19.318+0000 7f83c5e00640 1 --2- 172.21.15.16:0/1972556274 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f83c806d490 0x7f83c8160750 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7f83b800e930 tx=0x7f83b800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:08:19.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:19.318+0000 7f83be000640 1 -- 172.21.15.16:0/1972556274 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83b8017070 con 0x7f83c806d490 2024-03-31T15:08:19.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:19.318+0000 7f83be000640 1 -- 172.21.15.16:0/1972556274 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f83b800ca00 con 0x7f83c806d490 2024-03-31T15:08:19.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:19.318+0000 7f83be000640 1 -- 172.21.15.16:0/1972556274 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83b800cdb0 con 0x7f83c806d490 2024-03-31T15:08:19.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:19.318+0000 7f83cce00640 1 -- 172.21.15.16:0/1972556274 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f83c8160c90 con 0x7f83c806d490 2024-03-31T15:08:19.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:19.318+0000 7f83cce00640 1 -- 172.21.15.16:0/1972556274 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f83c80b6670 con 0x7f83c806d490 2024-03-31T15:08:19.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:19.318+0000 7f83cce00640 1 -- 172.21.15.16:0/1972556274 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f83c8069950 con 0x7f83c806d490 2024-03-31T15:08:19.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:19.322+0000 7f83be000640 1 -- 172.21.15.16:0/1972556274 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f83b801a070 con 0x7f83c806d490 2024-03-31T15:08:19.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:19.322+0000 7f83be000640 1 --2- 172.21.15.16:0/1972556274 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f83a4068380 0x7f83a406a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:19.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:19.322+0000 7f83be000640 1 -- 172.21.15.16:0/1972556274 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(70..70 src has 1..70) ==== 5882+0+0 (secure 0 0 0) 0x7f83b8086ab0 con 0x7f83c806d490 2024-03-31T15:08:19.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:19.322+0000 7f83bfe00640 1 --2- 172.21.15.16:0/1972556274 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f83a4068380 0x7f83a406a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:19.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:19.322+0000 7f83bfe00640 1 --2- 172.21.15.16:0/1972556274 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f83a4068380 0x7f83a406a840 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f83b00af000 tx=0x7f83b00a7000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:08:19.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:19.326+0000 7f83be000640 1 -- 172.21.15.16:0/1972556274 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f83b8057070 con 0x7f83c806d490 2024-03-31T15:08:19.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:19.502+0000 7f83cce00640 1 -- 172.21.15.16:0/1972556274 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f83c80cf3f0 con 0x7f83c806d490 2024-03-31T15:08:19.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:19.502+0000 7f83be000640 1 -- 172.21.15.16:0/1972556274 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 2 entries v70) ==== 80+0+122 (secure 0 0 0) 0x7f83b8056970 con 0x7f83c806d490 2024-03-31T15:08:19.497 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:6835/2555930820 2024-04-01T15:08:10.537891+0000 2024-03-31T15:08:19.498 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:6834/2555930820 2024-04-01T15:08:10.537891+0000 2024-03-31T15:08:19.498 INFO:teuthology.orchestra.run.smithi016.stderr:listed 2 entries 2024-03-31T15:08:19.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:19.506+0000 7f83cce00640 1 -- 172.21.15.16:0/1972556274 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f83a4068380 msgr2=0x7f83a406a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:19.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:19.506+0000 7f83cce00640 1 --2- 172.21.15.16:0/1972556274 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f83a4068380 0x7f83a406a840 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f83b00af000 tx=0x7f83b00a7000 comp rx=0 tx=0).stop 2024-03-31T15:08:19.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:19.506+0000 7f83cce00640 1 -- 172.21.15.16:0/1972556274 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f83c806d490 msgr2=0x7f83c8160750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:19.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:19.506+0000 7f83cce00640 1 --2- 172.21.15.16:0/1972556274 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f83c806d490 0x7f83c8160750 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7f83b800e930 tx=0x7f83b800edf0 comp rx=0 tx=0).stop 2024-03-31T15:08:19.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:19.506+0000 7f83cce00640 1 -- 172.21.15.16:0/1972556274 shutdown_connections 2024-03-31T15:08:19.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:19.506+0000 7f83cce00640 1 --2- 172.21.15.16:0/1972556274 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f83a4068380 0x7f83a406a840 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:19.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:19.506+0000 7f83cce00640 1 --2- 172.21.15.16:0/1972556274 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f83c80cb440 0x7f83c80b7430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:19.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:19.506+0000 7f83cce00640 1 --2- 172.21.15.16:0/1972556274 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f83c806f920 0x7f83c813adb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:19.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:19.506+0000 7f83cce00640 1 --2- 172.21.15.16:0/1972556274 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f83c806d490 0x7f83c8160750 unknown :-1 s=CLOSED pgs=911 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:19.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:19.506+0000 7f83cce00640 1 -- 172.21.15.16:0/1972556274 >> 172.21.15.16:0/1972556274 conn(0x7f83c8067330 msgr2=0x7f83c80cb8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:08:19.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:19.506+0000 7f83cce00640 1 -- 172.21.15.16:0/1972556274 shutdown_connections 2024-03-31T15:08:19.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:19.506+0000 7f83cce00640 1 -- 172.21.15.16:0/1972556274 wait complete. 2024-03-31T15:08:19.519 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-03-31T15:08:19.519 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-31T15:08:19.529 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-31T15:08:19.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:19.714+0000 7f5e59a00640 1 Processor -- start 2024-03-31T15:08:19.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:19.714+0000 7f5e59a00640 1 -- start start 2024-03-31T15:08:19.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:19.718+0000 7f5e59a00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f5e5415f650 0x7f5e5406f920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:19.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:19.718+0000 7f5e59a00640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5e540cbe90 con 0x7f5e5406dbc0 2024-03-31T15:08:19.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:19.718+0000 7f5e59a00640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5e540cc170 con 0x7f5e5406fe60 2024-03-31T15:08:19.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:19.718+0000 7f5e59a00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f5e5415f1c0 con 0x7f5e5415f650 2024-03-31T15:08:19.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:19.718+0000 7f5e52a00640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f5e5406dbc0 0x7f5e540cb440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:60000/0 (socket says 172.21.15.16:60000) 2024-03-31T15:08:19.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:19.718+0000 7f5e52a00640 1 -- 172.21.15.16:0/3080386908 learned_addr learned my addr 172.21.15.16:0/3080386908 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:08:19.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:19.718+0000 7f5e4be00640 1 --2- 172.21.15.16:0/3080386908 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f5e5415f650 0x7f5e5406f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:19.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:19.718+0000 7f5e52000640 1 -- 172.21.15.16:0/3080386908 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3490667742 0 0) 0x7f5e540cbe90 con 0x7f5e5406dbc0 2024-03-31T15:08:19.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:19.718+0000 7f5e52000640 1 -- 172.21.15.16:0/3080386908 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5e38003530 con 0x7f5e5406dbc0 2024-03-31T15:08:19.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:19.718+0000 7f5e52000640 1 -- 172.21.15.16:0/3080386908 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3364671765 0 0) 0x7f5e38003530 con 0x7f5e5406dbc0 2024-03-31T15:08:19.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:19.718+0000 7f5e52000640 1 -- 172.21.15.16:0/3080386908 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f5e5415f650 msgr2=0x7f5e5406f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:08:19.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:19.718+0000 7f5e52000640 1 --2- 172.21.15.16:0/3080386908 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f5e5415f650 0x7f5e5406f920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:19.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:19.718+0000 7f5e52000640 1 -- 172.21.15.16:0/3080386908 >> v1:172.21.15.89:6789/0 conn(0x7f5e5406fe60 legacy=0x7f5e54162180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:19.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:19.718+0000 7f5e52000640 1 -- 172.21.15.16:0/3080386908 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e5415ee50 con 0x7f5e5406dbc0 2024-03-31T15:08:19.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:19.718+0000 7f5e52000640 1 -- 172.21.15.16:0/3080386908 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 214807623 0 0) 0x7f5e44001680 con 0x7f5e5406dbc0 2024-03-31T15:08:19.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:19.718+0000 7f5e52000640 1 -- 172.21.15.16:0/3080386908 >> v1:172.21.15.16:6789/0 conn(0x7f5e5406dbc0 legacy=0x7f5e540cb440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:19.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:19.718+0000 7f5e52000640 1 --2- 172.21.15.16:0/3080386908 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5e380039f0 0x7f5e38003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:19.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:19.718+0000 7f5e52000640 1 --2- 172.21.15.16:0/3080386908 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5e380043a0 0x7f5e38004870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:19.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:19.718+0000 7f5e52000640 1 --2- 172.21.15.16:0/3080386908 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f5e5415f650 0x7f5e38005170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:19.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:19.718+0000 7f5e52000640 1 -- 172.21.15.16:0/3080386908 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f5e5415f1c0 con 0x7f5e380043a0 2024-03-31T15:08:19.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:19.718+0000 7f5e52000640 1 -- 172.21.15.16:0/3080386908 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f5e3800ba80 con 0x7f5e380039f0 2024-03-31T15:08:19.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:19.718+0000 7f5e52000640 1 -- 172.21.15.16:0/3080386908 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f5e3800bc80 con 0x7f5e5415f650 2024-03-31T15:08:19.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:19.718+0000 7f5e4be00640 1 --2- 172.21.15.16:0/3080386908 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5e380043a0 0x7f5e38004870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:19.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:19.718+0000 7f5e52a00640 1 --2- 172.21.15.16:0/3080386908 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5e380039f0 0x7f5e38003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:19.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:19.718+0000 7f5e53400640 1 --2- 172.21.15.16:0/3080386908 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f5e5415f650 0x7f5e38005170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:19.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:19.718+0000 7f5e4be00640 1 -- 172.21.15.16:0/3080386908 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f5e5415f650 msgr2=0x7f5e38005170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:08:19.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:19.718+0000 7f5e4be00640 1 --2- 172.21.15.16:0/3080386908 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f5e5415f650 0x7f5e38005170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:19.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:19.718+0000 7f5e4be00640 1 -- 172.21.15.16:0/3080386908 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5e380039f0 msgr2=0x7f5e38003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:08:19.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:19.718+0000 7f5e4be00640 1 --2- 172.21.15.16:0/3080386908 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5e380039f0 0x7f5e38003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:19.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:19.718+0000 7f5e4be00640 1 -- 172.21.15.16:0/3080386908 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e38003530 con 0x7f5e380043a0 2024-03-31T15:08:19.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:19.722+0000 7f5e4be00640 1 --2- 172.21.15.16:0/3080386908 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5e380043a0 0x7f5e38004870 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7f5e3c009ae0 tx=0x7f5e3c01ca70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da087be95de0832 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:08:19.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:19.722+0000 7f5e52000640 1 -- 172.21.15.16:0/3080386908 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e3c0b2070 con 0x7f5e380043a0 2024-03-31T15:08:19.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:19.722+0000 7f5e52000640 1 -- 172.21.15.16:0/3080386908 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5e3c0a5660 con 0x7f5e380043a0 2024-03-31T15:08:19.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:19.722+0000 7f5e52000640 1 -- 172.21.15.16:0/3080386908 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e3c0096b0 con 0x7f5e380043a0 2024-03-31T15:08:19.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:19.722+0000 7f5e59a00640 1 -- 172.21.15.16:0/3080386908 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5e380043a0 msgr2=0x7f5e38004870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:19.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:19.722+0000 7f5e59a00640 1 --2- 172.21.15.16:0/3080386908 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5e380043a0 0x7f5e38004870 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7f5e3c009ae0 tx=0x7f5e3c01ca70 comp rx=0 tx=0).stop 2024-03-31T15:08:19.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:19.722+0000 7f5e53400640 1 -- 172.21.15.16:0/3080386908 reap_dead start 2024-03-31T15:08:19.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:19.722+0000 7f5e59a00640 1 -- 172.21.15.16:0/3080386908 shutdown_connections 2024-03-31T15:08:19.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:19.722+0000 7f5e59a00640 1 -- 172.21.15.16:0/3080386908 >> 172.21.15.16:0/3080386908 conn(0x7f5e54067330 msgr2=0x7f5e54067740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:08:19.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:19.722+0000 7f5e59a00640 1 -- 172.21.15.16:0/3080386908 shutdown_connections 2024-03-31T15:08:19.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:19.722+0000 7f5e59a00640 1 -- 172.21.15.16:0/3080386908 wait complete. 2024-03-31T15:08:19.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:19.726+0000 7f5e59a00640 1 Processor -- start 2024-03-31T15:08:19.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:19.726+0000 7f5e59a00640 1 -- start start 2024-03-31T15:08:19.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:19.726+0000 7f5e59a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5e380043a0 0x7f5e540b4500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:19.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:19.726+0000 7f5e59a00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f5e5415c340 0x7f5e5415d260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:19.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:19.726+0000 7f5e59a00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5e5415afb0 0x7f5e5415b460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:19.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:19.726+0000 7f5e59a00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f5e540cbb50 con 0x7f5e380043a0 2024-03-31T15:08:19.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:19.726+0000 7f5e59a00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f5e5415eb40 con 0x7f5e5415afb0 2024-03-31T15:08:19.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:19.726+0000 7f5e59a00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f5e5406dfa0 con 0x7f5e5415c340 2024-03-31T15:08:19.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:19.726+0000 7f5e52a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5e380043a0 0x7f5e540b4500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:19.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:19.726+0000 7f5e52a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5e380043a0 0x7f5e540b4500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38640/0 (socket says 172.21.15.16:38640) 2024-03-31T15:08:19.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:19.726+0000 7f5e52a00640 1 -- 172.21.15.16:0/3850531892 learned_addr learned my addr 172.21.15.16:0/3850531892 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:08:19.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:19.726+0000 7f5e53400640 1 --2- 172.21.15.16:0/3850531892 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5e5415afb0 0x7f5e5415b460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:19.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:19.726+0000 7f5e4be00640 1 --2- 172.21.15.16:0/3850531892 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f5e5415c340 0x7f5e5415d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:19.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:19.726+0000 7f5e52a00640 1 -- 172.21.15.16:0/3850531892 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f5e5415c340 msgr2=0x7f5e5415d260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:19.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:19.726+0000 7f5e52a00640 1 --2- 172.21.15.16:0/3850531892 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f5e5415c340 0x7f5e5415d260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:19.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:19.726+0000 7f5e52a00640 1 -- 172.21.15.16:0/3850531892 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5e5415afb0 msgr2=0x7f5e5415b460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:19.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:19.726+0000 7f5e52a00640 1 --2- 172.21.15.16:0/3850531892 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5e5415afb0 0x7f5e5415b460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:19.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:19.726+0000 7f5e52a00640 1 -- 172.21.15.16:0/3850531892 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e5415d9d0 con 0x7f5e380043a0 2024-03-31T15:08:19.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:19.726+0000 7f5e52a00640 1 --2- 172.21.15.16:0/3850531892 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5e380043a0 0x7f5e540b4500 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7f5e44001d00 tx=0x7f5e44007430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:08:19.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:19.726+0000 7f5e4aa00640 1 -- 172.21.15.16:0/3850531892 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e4400f750 con 0x7f5e380043a0 2024-03-31T15:08:19.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:19.726+0000 7f5e4aa00640 1 -- 172.21.15.16:0/3850531892 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5e44018070 con 0x7f5e380043a0 2024-03-31T15:08:19.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:19.726+0000 7f5e4aa00640 1 -- 172.21.15.16:0/3850531892 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e4400fb50 con 0x7f5e380043a0 2024-03-31T15:08:19.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:19.726+0000 7f5e59a00640 1 -- 172.21.15.16:0/3850531892 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5e5415a4d0 con 0x7f5e380043a0 2024-03-31T15:08:19.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:19.726+0000 7f5e59a00640 1 -- 172.21.15.16:0/3850531892 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5e5415aa90 con 0x7f5e380043a0 2024-03-31T15:08:19.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:19.726+0000 7f5e59a00640 1 -- 172.21.15.16:0/3850531892 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5e5415ac90 con 0x7f5e380043a0 2024-03-31T15:08:19.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:19.730+0000 7f5e4aa00640 1 -- 172.21.15.16:0/3850531892 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f5e4400fcf0 con 0x7f5e380043a0 2024-03-31T15:08:19.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:19.730+0000 7f5e4aa00640 1 --2- 172.21.15.16:0/3850531892 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f5e30068380 0x7f5e3006a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:19.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:19.730+0000 7f5e4aa00640 1 -- 172.21.15.16:0/3850531892 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(70..70 src has 1..70) ==== 5882+0+0 (secure 0 0 0) 0x7f5e44086990 con 0x7f5e380043a0 2024-03-31T15:08:19.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:19.730+0000 7f5e4be00640 1 --2- 172.21.15.16:0/3850531892 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f5e30068380 0x7f5e3006a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:19.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:19.730+0000 7f5e4be00640 1 --2- 172.21.15.16:0/3850531892 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f5e30068380 0x7f5e3006a840 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f5e3c0af000 tx=0x7f5e3c0046d0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:08:19.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:19.734+0000 7f5e4aa00640 1 -- 172.21.15.16:0/3850531892 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f5e5415ac90 con 0x7f5e380043a0 2024-03-31T15:08:19.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:19.898+0000 7f5e59a00640 1 -- 172.21.15.16:0/3850531892 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f5e54069950 con 0x7f5e380043a0 2024-03-31T15:08:19.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:19.898+0000 7f5e4aa00640 1 -- 172.21.15.16:0/3850531892 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v70) ==== 74+0+16870 (secure 0 0 0) 0x7f5e44056e90 con 0x7f5e380043a0 2024-03-31T15:08:19.895 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:08:19.895 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":70,"fsid":"286eb5da-fb47-4550-b290-d9519c13b9a5","created":"2024-03-31T15:03:11.438952+0000","modified":"2024-03-31T15:08:18.287907+0000","last_up_change":"2024-03-31T15:03:22.885535+0000","last_in_change":"2024-03-31T15:03:13.941667+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-31T15:03:22.550787+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-03-31T15:07:55.329096+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"66","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-03-31T15:07:55.885919+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"67","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"92a66450-ef19-40d8-ac3d-d760b23b0bda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6801","nonce":932712244}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6803","nonce":932712244}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6807","nonce":932712244}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6805","nonce":932712244}]},"public_addr":"172.21.15.16:6801/932712244","cluster_addr":"172.21.15.16:6803/932712244","heartbeat_back_addr":"172.21.15.16:6807/932712244","heartbeat_front_addr":"172.21.15.16:6805/932712244","state":["exists","up"]},{"osd":1,"uuid":"da9d1f2b-1968-4959-b843-faa231988e92","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6810","nonce":1398737533}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6811","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6813","nonce":1398737533}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6821","nonce":1398737533}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6815","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6816","nonce":1398737533}]},"public_addr":"172.21.15.16:6810/1398737533","cluster_addr":"172.21.15.16:6813/1398737533","heartbeat_back_addr":"172.21.15.16:6821/1398737533","heartbeat_front_addr":"172.21.15.16:6816/1398737533","state":["exists","up"]},{"osd":2,"uuid":"01a73662-9a21-4b1f-bcbb-42b9636a4638","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6819","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6822","nonce":3018526646}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6825","nonce":3018526646}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6831","nonce":3018526646}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6827","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6829","nonce":3018526646}]},"public_addr":"172.21.15.16:6822/3018526646","cluster_addr":"172.21.15.16:6825/3018526646","heartbeat_back_addr":"172.21.15.16:6831/3018526646","heartbeat_front_addr":"172.21.15.16:6829/3018526646","state":["exists","up"]},{"osd":3,"uuid":"c7771fd3-e18e-4500-ac38-23cf10ebcd75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6809","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6812","nonce":3360352974}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6817","nonce":3360352974}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6828","nonce":3360352974}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6823","nonce":3360352974}]},"public_addr":"172.21.15.16:6812/3360352974","cluster_addr":"172.21.15.16:6817/3360352974","heartbeat_back_addr":"172.21.15.16:6828/3360352974","heartbeat_front_addr":"172.21.15.16:6823/3360352974","state":["exists","up"]},{"osd":4,"uuid":"88260730-1482-4fbd-a40f-cb1124e9df4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6801","nonce":1856844856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6803","nonce":1856844856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6807","nonce":1856844856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6805","nonce":1856844856}]},"public_addr":"172.21.15.89:6801/1856844856","cluster_addr":"172.21.15.89:6803/1856844856","heartbeat_back_addr":"172.21.15.89:6807/1856844856","heartbeat_front_addr":"172.21.15.89:6805/1856844856","state":["exists","up"]},{"osd":5,"uuid":"4cb2c00c-fae0-4dd8-9d27-ec4f0093ccf7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6814","nonce":50395527}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6817","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6820","nonce":50395527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6831","nonce":50395527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6823","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6826","nonce":50395527}]},"public_addr":"172.21.15.89:6814/50395527","cluster_addr":"172.21.15.89:6820/50395527","heartbeat_back_addr":"172.21.15.89:6831/50395527","heartbeat_front_addr":"172.21.15.89:6826/50395527","state":["exists","up"]},{"osd":6,"uuid":"81ce0c19-2554-4cae-acbe-66a05e9e9b9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6809","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6812","nonce":3470471752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6815","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6818","nonce":3470471752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6827","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6829","nonce":3470471752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6821","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6824","nonce":3470471752}]},"public_addr":"172.21.15.89:6812/3470471752","cluster_addr":"172.21.15.89:6818/3470471752","heartbeat_back_addr":"172.21.15.89:6829/3470471752","heartbeat_front_addr":"172.21.15.89:6824/3470471752","state":["exists","up"]},{"osd":7,"uuid":"85e4313b-d37d-442e-8042-153bf20a847d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6811","nonce":1796778630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6813","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6816","nonce":1796778630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6825","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6828","nonce":1796778630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6819","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6822","nonce":1796778630}]},"public_addr":"172.21.15.89:6811/1796778630","cluster_addr":"172.21.15.89:6816/1796778630","heartbeat_back_addr":"172.21.15.89:6828/1796778630","heartbeat_front_addr":"172.21.15.89:6822/1796778630","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.439088+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.529970+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.680512+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:21.044161+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.582111+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.536558+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.409522+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.590229+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"11.4","mappings":[{"from":6,"to":0}]},{"pgid":"11.10","mappings":[{"from":6,"to":1}]},{"pgid":"11.3e","mappings":[{"from":6,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:6835/2555930820":"2024-04-01T15:08:10.537891+0000","172.21.15.16:6834/2555930820":"2024-04-01T15:08:10.537891+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-31T15:08:19.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:19.902+0000 7f5e59a00640 1 -- 172.21.15.16:0/3850531892 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f5e30068380 msgr2=0x7f5e3006a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:19.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:19.902+0000 7f5e59a00640 1 --2- 172.21.15.16:0/3850531892 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f5e30068380 0x7f5e3006a840 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f5e3c0af000 tx=0x7f5e3c0046d0 comp rx=0 tx=0).stop 2024-03-31T15:08:19.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:19.902+0000 7f5e59a00640 1 -- 172.21.15.16:0/3850531892 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5e380043a0 msgr2=0x7f5e540b4500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:19.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:19.902+0000 7f5e59a00640 1 --2- 172.21.15.16:0/3850531892 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5e380043a0 0x7f5e540b4500 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7f5e44001d00 tx=0x7f5e44007430 comp rx=0 tx=0).stop 2024-03-31T15:08:19.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:19.902+0000 7f5e59a00640 1 -- 172.21.15.16:0/3850531892 shutdown_connections 2024-03-31T15:08:19.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:19.902+0000 7f5e59a00640 1 --2- 172.21.15.16:0/3850531892 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f5e30068380 0x7f5e3006a840 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:19.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:19.902+0000 7f5e59a00640 1 --2- 172.21.15.16:0/3850531892 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5e5415afb0 0x7f5e5415b460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:19.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:19.902+0000 7f5e59a00640 1 --2- 172.21.15.16:0/3850531892 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f5e5415c340 0x7f5e5415d260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:19.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:19.902+0000 7f5e59a00640 1 --2- 172.21.15.16:0/3850531892 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5e380043a0 0x7f5e540b4500 unknown :-1 s=CLOSED pgs=914 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:19.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:19.902+0000 7f5e59a00640 1 -- 172.21.15.16:0/3850531892 >> 172.21.15.16:0/3850531892 conn(0x7f5e54067330 msgr2=0x7f5e54161160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:08:19.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:19.902+0000 7f5e59a00640 1 -- 172.21.15.16:0/3850531892 shutdown_connections 2024-03-31T15:08:19.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:19.902+0000 7f5e59a00640 1 -- 172.21.15.16:0/3850531892 wait complete. 2024-03-31T15:08:19.917 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-31T15:03:22.550787+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 10, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-31T15:07:55.329096+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '66', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.5, 'score_stable': 2.5, 'optimal_score': 1, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 11, 'pool_name': 'cephfs_data', 'create_time': '2024-03-31T15:07:55.885919+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '67', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-31T15:08:19.917 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-31T15:08:20.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:20.066+0000 7fd6ede00640 1 Processor -- start 2024-03-31T15:08:20.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:20.066+0000 7fd6ede00640 1 -- start start 2024-03-31T15:08:20.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:20.066+0000 7fd6ede00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fd6e806f920 0x7fd6e806fd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:20.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:20.066+0000 7fd6ede00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fd6e80d0a60 0x7fd6e80cea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:20.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:20.066+0000 7fd6ede00640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd6e8070240 con 0x7fd6e80cef80 2024-03-31T15:08:20.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:20.066+0000 7fd6ede00640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7fd6e813f670 con 0x7fd6e806f920 2024-03-31T15:08:20.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:20.066+0000 7fd6ede00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fd6e815ca70 con 0x7fd6e80d0a60 2024-03-31T15:08:20.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:20.066+0000 7fd6e7400640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fd6e80cef80 0x7fd6e8070e60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:60002/0 (socket says 172.21.15.16:60002) 2024-03-31T15:08:20.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:20.066+0000 7fd6e7400640 1 -- 172.21.15.16:0/434885987 learned_addr learned my addr 172.21.15.16:0/434885987 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:08:20.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:20.066+0000 7fd6dfe00640 1 --2- 172.21.15.16:0/434885987 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fd6e80d0a60 0x7fd6e80cea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:20.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:20.066+0000 7fd6e6a00640 1 --2- 172.21.15.16:0/434885987 >> v2:172.21.15.89:3300/0 conn(0x7fd6e806f920 0x7fd6e806fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:20.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:20.066+0000 7fd6e6000640 1 -- 172.21.15.16:0/434885987 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3876460888 0 0) 0x7fd6e8070240 con 0x7fd6e80cef80 2024-03-31T15:08:20.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:20.066+0000 7fd6e6000640 1 -- 172.21.15.16:0/434885987 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd6cc003530 con 0x7fd6e80cef80 2024-03-31T15:08:20.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:20.066+0000 7fd6dfe00640 1 -- 172.21.15.16:0/434885987 >> v2:172.21.15.89:3300/0 conn(0x7fd6e806f920 msgr2=0x7fd6e806fd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:08:20.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:20.066+0000 7fd6dfe00640 1 --2- 172.21.15.16:0/434885987 >> v2:172.21.15.89:3300/0 conn(0x7fd6e806f920 0x7fd6e806fd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:20.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:20.066+0000 7fd6dfe00640 1 -- 172.21.15.16:0/434885987 >> v1:172.21.15.16:6789/0 conn(0x7fd6e80cef80 legacy=0x7fd6e8070e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:20.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:20.066+0000 7fd6e6000640 1 -- 172.21.15.16:0/434885987 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1611166637 0 0) 0x7fd6cc003530 con 0x7fd6e80cef80 2024-03-31T15:08:20.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:20.066+0000 7fd6dfe00640 1 -- 172.21.15.16:0/434885987 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6e815cd50 con 0x7fd6e80d0a60 2024-03-31T15:08:20.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:20.066+0000 7fd6e6a00640 1 --2- 172.21.15.16:0/434885987 >> v2:172.21.15.89:3300/0 conn(0x7fd6e806f920 0x7fd6e806fd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:08:20.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:20.070+0000 7fd6dfe00640 1 --2- 172.21.15.16:0/434885987 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fd6e80d0a60 0x7fd6e80cea40 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fd6d00097c0 tx=0x7fd6d001c4a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6f568f027a3558ac server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:08:20.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:20.070+0000 7fd6e6000640 1 -- 172.21.15.16:0/434885987 <== mon.2 v2:172.21.15.89:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6d00b0070 con 0x7fd6e80d0a60 2024-03-31T15:08:20.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:20.070+0000 7fd6e6000640 1 -- 172.21.15.16:0/434885987 <== mon.2 v2:172.21.15.89:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd6d00a4e50 con 0x7fd6e80d0a60 2024-03-31T15:08:20.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:20.070+0000 7fd6e6000640 1 -- 172.21.15.16:0/434885987 <== mon.2 v2:172.21.15.89:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6d00ab800 con 0x7fd6e80d0a60 2024-03-31T15:08:20.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:20.070+0000 7fd6ede00640 1 -- 172.21.15.16:0/434885987 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fd6e80d0a60 msgr2=0x7fd6e80cea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:20.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:20.070+0000 7fd6ede00640 1 --2- 172.21.15.16:0/434885987 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fd6e80d0a60 0x7fd6e80cea40 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fd6d00097c0 tx=0x7fd6d001c4a0 comp rx=0 tx=0).stop 2024-03-31T15:08:20.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:20.070+0000 7fd6ede00640 1 -- 172.21.15.16:0/434885987 shutdown_connections 2024-03-31T15:08:20.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:20.070+0000 7fd6ede00640 1 --2- 172.21.15.16:0/434885987 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fd6e80d0a60 0x7fd6e80cea40 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:20.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:20.070+0000 7fd6ede00640 1 --2- 172.21.15.16:0/434885987 >> v2:172.21.15.89:3300/0 conn(0x7fd6e806f920 0x7fd6e806fd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:20.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:20.070+0000 7fd6ede00640 1 -- 172.21.15.16:0/434885987 >> 172.21.15.16:0/434885987 conn(0x7fd6e8067930 msgr2=0x7fd6e80b3ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:08:20.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:20.070+0000 7fd6ede00640 1 -- 172.21.15.16:0/434885987 shutdown_connections 2024-03-31T15:08:20.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:20.070+0000 7fd6ede00640 1 -- 172.21.15.16:0/434885987 wait complete. 2024-03-31T15:08:20.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:20.074+0000 7fd6ede00640 1 Processor -- start 2024-03-31T15:08:20.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:20.074+0000 7fd6ede00640 1 -- start start 2024-03-31T15:08:20.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:20.074+0000 7fd6ede00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fd6e806f920 0x7fd6e8148c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:20.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:20.074+0000 7fd6ede00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd6e80cef80 0x7fd6e813e960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:20.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:20.074+0000 7fd6ede00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd6e80d0a60 0x7fd6e80c8de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:20.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:20.074+0000 7fd6ede00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fd6e81690b0 con 0x7fd6e80d0a60 2024-03-31T15:08:20.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:20.074+0000 7fd6ede00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fd6e815c720 con 0x7fd6e80cef80 2024-03-31T15:08:20.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:20.074+0000 7fd6ede00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fd6e8169230 con 0x7fd6e806f920 2024-03-31T15:08:20.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:20.074+0000 7fd6e7400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd6e80d0a60 0x7fd6e80c8de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:20.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:20.074+0000 7fd6e7400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd6e80d0a60 0x7fd6e80c8de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38644/0 (socket says 172.21.15.16:38644) 2024-03-31T15:08:20.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:20.074+0000 7fd6e7400640 1 -- 172.21.15.16:0/4203227777 learned_addr learned my addr 172.21.15.16:0/4203227777 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:08:20.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:20.074+0000 7fd6e6a00640 1 --2- 172.21.15.16:0/4203227777 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fd6e806f920 0x7fd6e8148c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:20.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:20.074+0000 7fd6dfe00640 1 --2- 172.21.15.16:0/4203227777 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd6e80cef80 0x7fd6e813e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:20.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:20.074+0000 7fd6e7400640 1 -- 172.21.15.16:0/4203227777 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fd6e806f920 msgr2=0x7fd6e8148c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:20.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:20.074+0000 7fd6e7400640 1 --2- 172.21.15.16:0/4203227777 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fd6e806f920 0x7fd6e8148c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:20.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:20.074+0000 7fd6e7400640 1 -- 172.21.15.16:0/4203227777 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd6e80cef80 msgr2=0x7fd6e813e960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:20.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:20.074+0000 7fd6e7400640 1 --2- 172.21.15.16:0/4203227777 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd6e80cef80 0x7fd6e813e960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:20.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:20.074+0000 7fd6e7400640 1 -- 172.21.15.16:0/4203227777 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6e80c6ff0 con 0x7fd6e80d0a60 2024-03-31T15:08:20.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:20.074+0000 7fd6e7400640 1 --2- 172.21.15.16:0/4203227777 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd6e80d0a60 0x7fd6e80c8de0 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7fd6e000cb40 tx=0x7fd6e000a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:08:20.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:20.074+0000 7fd6df400640 1 -- 172.21.15.16:0/4203227777 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6e000aca0 con 0x7fd6e80d0a60 2024-03-31T15:08:20.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:20.074+0000 7fd6df400640 1 -- 172.21.15.16:0/4203227777 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd6e0012400 con 0x7fd6e80d0a60 2024-03-31T15:08:20.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:20.074+0000 7fd6ede00640 1 -- 172.21.15.16:0/4203227777 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd6e80c72d0 con 0x7fd6e80d0a60 2024-03-31T15:08:20.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:20.074+0000 7fd6df400640 1 -- 172.21.15.16:0/4203227777 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6e000ae60 con 0x7fd6e80d0a60 2024-03-31T15:08:20.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:20.074+0000 7fd6ede00640 1 -- 172.21.15.16:0/4203227777 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd6e80c7890 con 0x7fd6e80d0a60 2024-03-31T15:08:20.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:20.074+0000 7fd6ede00640 1 -- 172.21.15.16:0/4203227777 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd6e8159070 con 0x7fd6e80d0a60 2024-03-31T15:08:20.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:20.078+0000 7fd6df400640 1 -- 172.21.15.16:0/4203227777 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fd6e00127f0 con 0x7fd6e80d0a60 2024-03-31T15:08:20.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:20.078+0000 7fd6df400640 1 --2- 172.21.15.16:0/4203227777 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fd6c40683b0 0x7fd6c406a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:20.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:20.078+0000 7fd6df400640 1 -- 172.21.15.16:0/4203227777 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(70..70 src has 1..70) ==== 5882+0+0 (secure 0 0 0) 0x7fd6e00871a0 con 0x7fd6e80d0a60 2024-03-31T15:08:20.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:20.078+0000 7fd6e6a00640 1 --2- 172.21.15.16:0/4203227777 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fd6c40683b0 0x7fd6c406a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:20.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:20.078+0000 7fd6e6a00640 1 --2- 172.21.15.16:0/4203227777 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fd6c40683b0 0x7fd6c406a870 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7fd6d8009580 tx=0x7fd6d8008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:08:20.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:20.082+0000 7fd6df400640 1 -- 172.21.15.16:0/4203227777 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fd6e0057610 con 0x7fd6e80d0a60 2024-03-31T15:08:20.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:20.250+0000 7fd6ede00640 1 -- 172.21.15.16:0/4203227777 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fd6e8069950 con 0x7fd6e80d0a60 2024-03-31T15:08:20.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:20.250+0000 7fd6df400640 1 -- 172.21.15.16:0/4203227777 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v70) ==== 93+0+10 (secure 0 0 0) 0x7fd6e0056f10 con 0x7fd6e80d0a60 2024-03-31T15:08:20.245 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2024-03-31T15:08:20.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:20.250+0000 7fd6ede00640 1 -- 172.21.15.16:0/4203227777 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fd6c40683b0 msgr2=0x7fd6c406a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:20.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:20.250+0000 7fd6ede00640 1 --2- 172.21.15.16:0/4203227777 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fd6c40683b0 0x7fd6c406a870 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7fd6d8009580 tx=0x7fd6d8008040 comp rx=0 tx=0).stop 2024-03-31T15:08:20.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:20.250+0000 7fd6ede00640 1 -- 172.21.15.16:0/4203227777 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd6e80d0a60 msgr2=0x7fd6e80c8de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:20.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:20.250+0000 7fd6ede00640 1 --2- 172.21.15.16:0/4203227777 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd6e80d0a60 0x7fd6e80c8de0 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7fd6e000cb40 tx=0x7fd6e000a060 comp rx=0 tx=0).stop 2024-03-31T15:08:20.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:20.250+0000 7fd6ede00640 1 -- 172.21.15.16:0/4203227777 shutdown_connections 2024-03-31T15:08:20.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:20.250+0000 7fd6ede00640 1 --2- 172.21.15.16:0/4203227777 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fd6c40683b0 0x7fd6c406a870 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:20.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:20.250+0000 7fd6ede00640 1 --2- 172.21.15.16:0/4203227777 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd6e80d0a60 0x7fd6e80c8de0 unknown :-1 s=CLOSED pgs=916 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:20.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:20.250+0000 7fd6ede00640 1 --2- 172.21.15.16:0/4203227777 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd6e80cef80 0x7fd6e813e960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:20.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:20.250+0000 7fd6ede00640 1 --2- 172.21.15.16:0/4203227777 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fd6e806f920 0x7fd6e8148c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:20.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:20.250+0000 7fd6ede00640 1 -- 172.21.15.16:0/4203227777 >> 172.21.15.16:0/4203227777 conn(0x7fd6e8067930 msgr2=0x7fd6e80b6770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:08:20.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:20.250+0000 7fd6ede00640 1 -- 172.21.15.16:0/4203227777 shutdown_connections 2024-03-31T15:08:20.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:20.250+0000 7fd6ede00640 1 -- 172.21.15.16:0/4203227777 wait complete. 2024-03-31T15:08:20.266 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-31T15:08:20.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:20.414+0000 7f2482600640 1 Processor -- start 2024-03-31T15:08:20.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:20.414+0000 7f2482600640 1 -- start start 2024-03-31T15:08:20.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:20.414+0000 7f2482600640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f247c0cf150 0x7f247c1670a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:20.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:20.414+0000 7f2482600640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f247c15ca70 con 0x7f247c0d0a60 2024-03-31T15:08:20.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:20.414+0000 7f2482600640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f247c15cd50 con 0x7f247c06f920 2024-03-31T15:08:20.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:20.414+0000 7f2482600640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f247c13f670 con 0x7f247c0cf150 2024-03-31T15:08:20.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:20.414+0000 7f2473e00640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f247c0d0a60 0x7f247c0cea40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:60014/0 (socket says 172.21.15.16:60014) 2024-03-31T15:08:20.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:20.414+0000 7f2473e00640 1 -- 172.21.15.16:0/1969670920 learned_addr learned my addr 172.21.15.16:0/1969670920 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:08:20.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:20.414+0000 7f247be00640 1 --2- 172.21.15.16:0/1969670920 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f247c0cf150 0x7f247c1670a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:20.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:20.414+0000 7f247aa00640 1 -- 172.21.15.16:0/1969670920 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2949870100 0 0) 0x7f247c15ca70 con 0x7f247c0d0a60 2024-03-31T15:08:20.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:20.414+0000 7f247aa00640 1 -- 172.21.15.16:0/1969670920 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2460003530 con 0x7f247c0d0a60 2024-03-31T15:08:20.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:20.414+0000 7f247aa00640 1 -- 172.21.15.16:0/1969670920 <== mon.1 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2065104845 0 0) 0x7f247c15cd50 con 0x7f247c06f920 2024-03-31T15:08:20.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:20.414+0000 7f247aa00640 1 -- 172.21.15.16:0/1969670920 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f247c15ca70 con 0x7f247c06f920 2024-03-31T15:08:20.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:20.414+0000 7f247aa00640 1 -- 172.21.15.16:0/1969670920 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3154565696 0 0) 0x7f2460003530 con 0x7f247c0d0a60 2024-03-31T15:08:20.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:20.414+0000 7f247aa00640 1 -- 172.21.15.16:0/1969670920 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f247c0cf150 msgr2=0x7f247c1670a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:08:20.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:20.414+0000 7f247aa00640 1 --2- 172.21.15.16:0/1969670920 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f247c0cf150 0x7f247c1670a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:20.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:20.414+0000 7f247aa00640 1 -- 172.21.15.16:0/1969670920 >> v1:172.21.15.89:6789/0 conn(0x7f247c06f920 legacy=0x7f247c06fd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:20.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:20.414+0000 7f247aa00640 1 -- 172.21.15.16:0/1969670920 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f247c15d030 con 0x7f247c0d0a60 2024-03-31T15:08:20.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:20.414+0000 7f247aa00640 1 -- 172.21.15.16:0/1969670920 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 214807623 0 0) 0x7f2464001680 con 0x7f247c0d0a60 2024-03-31T15:08:20.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:20.418+0000 7f247aa00640 1 -- 172.21.15.16:0/1969670920 >> v1:172.21.15.16:6789/0 conn(0x7f247c0d0a60 legacy=0x7f247c0cea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:20.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:20.418+0000 7f247aa00640 1 --2- 172.21.15.16:0/1969670920 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2460003b80 0x7f2460003f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:20.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:20.418+0000 7f247aa00640 1 --2- 172.21.15.16:0/1969670920 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2460004810 0x7f2460004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:20.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:20.418+0000 7f247aa00640 1 --2- 172.21.15.16:0/1969670920 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f247c0cf150 0x7f2460005230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:20.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:20.418+0000 7f247aa00640 1 -- 172.21.15.16:0/1969670920 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f247c13f670 con 0x7f2460004810 2024-03-31T15:08:20.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:20.418+0000 7f247aa00640 1 -- 172.21.15.16:0/1969670920 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f246000bb60 con 0x7f2460003b80 2024-03-31T15:08:20.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:20.418+0000 7f247aa00640 1 -- 172.21.15.16:0/1969670920 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f246000bd30 con 0x7f247c0cf150 2024-03-31T15:08:20.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:20.418+0000 7f2473e00640 1 --2- 172.21.15.16:0/1969670920 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2460004810 0x7f2460004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:20.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:20.418+0000 7f247be00640 1 --2- 172.21.15.16:0/1969670920 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f247c0cf150 0x7f2460005230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:20.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:20.418+0000 7f247b400640 1 --2- 172.21.15.16:0/1969670920 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2460003b80 0x7f2460003f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:20.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:20.418+0000 7f2473e00640 1 -- 172.21.15.16:0/1969670920 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f247c0cf150 msgr2=0x7f2460005230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:08:20.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:20.418+0000 7f2473e00640 1 --2- 172.21.15.16:0/1969670920 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f247c0cf150 0x7f2460005230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:20.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:20.418+0000 7f2473e00640 1 -- 172.21.15.16:0/1969670920 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2460003b80 msgr2=0x7f2460003f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:08:20.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:20.418+0000 7f2473e00640 1 --2- 172.21.15.16:0/1969670920 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2460003b80 0x7f2460003f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:20.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:20.418+0000 7f2473e00640 1 -- 172.21.15.16:0/1969670920 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2460003530 con 0x7f2460004810 2024-03-31T15:08:20.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:20.418+0000 7f247be00640 1 --2- 172.21.15.16:0/1969670920 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f247c0cf150 0x7f2460005230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:08:20.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:20.418+0000 7f2473e00640 1 --2- 172.21.15.16:0/1969670920 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2460004810 0x7f2460004c90 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7f246400c0e0 tx=0x7f246401cb00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d53b0b43d9d878de server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:08:20.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:20.418+0000 7f247aa00640 1 -- 172.21.15.16:0/1969670920 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2464007c50 con 0x7f2460004810 2024-03-31T15:08:20.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:20.418+0000 7f247aa00640 1 -- 172.21.15.16:0/1969670920 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f24640a9070 con 0x7f2460004810 2024-03-31T15:08:20.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:20.418+0000 7f247aa00640 1 -- 172.21.15.16:0/1969670920 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2464005d60 con 0x7f2460004810 2024-03-31T15:08:20.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:20.422+0000 7f2482600640 1 -- 172.21.15.16:0/1969670920 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2460004810 msgr2=0x7f2460004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:20.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:20.422+0000 7f2482600640 1 --2- 172.21.15.16:0/1969670920 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2460004810 0x7f2460004c90 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7f246400c0e0 tx=0x7f246401cb00 comp rx=0 tx=0).stop 2024-03-31T15:08:20.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:20.422+0000 7f247be00640 1 -- 172.21.15.16:0/1969670920 reap_dead start 2024-03-31T15:08:20.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:20.422+0000 7f2482600640 1 -- 172.21.15.16:0/1969670920 shutdown_connections 2024-03-31T15:08:20.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:20.422+0000 7f2482600640 1 -- 172.21.15.16:0/1969670920 >> 172.21.15.16:0/1969670920 conn(0x7f247c067930 msgr2=0x7f247c0b3ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:08:20.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:20.422+0000 7f2482600640 1 -- 172.21.15.16:0/1969670920 shutdown_connections 2024-03-31T15:08:20.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:20.422+0000 7f2482600640 1 -- 172.21.15.16:0/1969670920 wait complete. 2024-03-31T15:08:20.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:20.422+0000 7f2482600640 1 Processor -- start 2024-03-31T15:08:20.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:20.422+0000 7f2482600640 1 -- start start 2024-03-31T15:08:20.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:20.422+0000 7f2482600640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f2460004810 0x7f247c06d7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:20.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:20.422+0000 7f2482600640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f247c06e330 0x7f247c06dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:20.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:20.422+0000 7f2482600640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f247c152650 0x7f247c15dc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:20.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:20.422+0000 7f2482600640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f247c167c40 con 0x7f247c152650 2024-03-31T15:08:20.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:20.422+0000 7f2482600640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f247c167ac0 con 0x7f247c06e330 2024-03-31T15:08:20.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:20.422+0000 7f2482600640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f247c167dc0 con 0x7f2460004810 2024-03-31T15:08:20.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:20.422+0000 7f247be00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f247c152650 0x7f247c15dc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:20.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:20.422+0000 7f247be00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f247c152650 0x7f247c15dc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38666/0 (socket says 172.21.15.16:38666) 2024-03-31T15:08:20.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:20.422+0000 7f247be00640 1 -- 172.21.15.16:0/131391765 learned_addr learned my addr 172.21.15.16:0/131391765 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:08:20.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:20.422+0000 7f247b400640 1 --2- 172.21.15.16:0/131391765 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f2460004810 0x7f247c06d7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:20.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:20.422+0000 7f2473e00640 1 --2- 172.21.15.16:0/131391765 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f247c06e330 0x7f247c06dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:20.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:20.422+0000 7f247be00640 1 -- 172.21.15.16:0/131391765 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f2460004810 msgr2=0x7f247c06d7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:20.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:20.422+0000 7f247be00640 1 --2- 172.21.15.16:0/131391765 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f2460004810 0x7f247c06d7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:20.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:20.422+0000 7f247be00640 1 -- 172.21.15.16:0/131391765 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f247c06e330 msgr2=0x7f247c06dd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:20.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:20.422+0000 7f247be00640 1 --2- 172.21.15.16:0/131391765 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f247c06e330 0x7f247c06dd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:20.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:20.422+0000 7f247be00640 1 -- 172.21.15.16:0/131391765 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f247c06ca20 con 0x7f247c152650 2024-03-31T15:08:20.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:20.426+0000 7f247be00640 1 --2- 172.21.15.16:0/131391765 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f247c152650 0x7f247c15dc70 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7f247c0d3150 tx=0x7f2474009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:08:20.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:20.426+0000 7f2473400640 1 -- 172.21.15.16:0/131391765 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f247401a030 con 0x7f247c152650 2024-03-31T15:08:20.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:20.426+0000 7f2473400640 1 -- 172.21.15.16:0/131391765 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2474010e20 con 0x7f247c152650 2024-03-31T15:08:20.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:20.426+0000 7f2482600640 1 -- 172.21.15.16:0/131391765 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f247c06cd00 con 0x7f247c152650 2024-03-31T15:08:20.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:20.426+0000 7f2473400640 1 -- 172.21.15.16:0/131391765 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2474003be0 con 0x7f247c152650 2024-03-31T15:08:20.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:20.426+0000 7f2482600640 1 -- 172.21.15.16:0/131391765 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f247c06d2c0 con 0x7f247c152650 2024-03-31T15:08:20.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:20.426+0000 7f2482600640 1 -- 172.21.15.16:0/131391765 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f247c069950 con 0x7f247c152650 2024-03-31T15:08:20.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:20.426+0000 7f2473400640 1 -- 172.21.15.16:0/131391765 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f247401d070 con 0x7f247c152650 2024-03-31T15:08:20.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:20.426+0000 7f2473400640 1 --2- 172.21.15.16:0/131391765 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f2458068380 0x7f245806a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:20.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:20.426+0000 7f2473400640 1 -- 172.21.15.16:0/131391765 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(70..70 src has 1..70) ==== 5882+0+0 (secure 0 0 0) 0x7f2474086f00 con 0x7f247c152650 2024-03-31T15:08:20.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:20.430+0000 7f247b400640 1 --2- 172.21.15.16:0/131391765 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f2458068380 0x7f245806a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:20.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:20.430+0000 7f247b400640 1 --2- 172.21.15.16:0/131391765 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f2458068380 0x7f245806a840 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f246c001760 tx=0x7f246c009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:08:20.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:20.434+0000 7f2473400640 1 -- 172.21.15.16:0/131391765 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f2474056db0 con 0x7f247c152650 2024-03-31T15:08:20.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:20.682+0000 7f2482600640 1 -- 172.21.15.16:0/131391765 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f247c0b4a20 con 0x7f247c152650 2024-03-31T15:08:20.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:20.682+0000 7f2473400640 1 -- 172.21.15.16:0/131391765 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v70) ==== 104+0+11 (secure 0 0 0) 0x7f247405b0b0 con 0x7f247c152650 2024-03-31T15:08:20.678 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2024-03-31T15:08:20.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:20.686+0000 7f2482600640 1 -- 172.21.15.16:0/131391765 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f2458068380 msgr2=0x7f245806a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:20.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:20.686+0000 7f2482600640 1 --2- 172.21.15.16:0/131391765 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f2458068380 0x7f245806a840 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f246c001760 tx=0x7f246c009040 comp rx=0 tx=0).stop 2024-03-31T15:08:20.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:20.686+0000 7f2482600640 1 -- 172.21.15.16:0/131391765 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f247c152650 msgr2=0x7f247c15dc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:20.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:20.686+0000 7f2482600640 1 --2- 172.21.15.16:0/131391765 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f247c152650 0x7f247c15dc70 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7f247c0d3150 tx=0x7f2474009f90 comp rx=0 tx=0).stop 2024-03-31T15:08:20.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:20.686+0000 7f2482600640 1 -- 172.21.15.16:0/131391765 shutdown_connections 2024-03-31T15:08:20.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:20.686+0000 7f2482600640 1 --2- 172.21.15.16:0/131391765 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f2458068380 0x7f245806a840 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:20.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:20.686+0000 7f2482600640 1 --2- 172.21.15.16:0/131391765 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f247c152650 0x7f247c15dc70 unknown :-1 s=CLOSED pgs=919 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:20.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:20.686+0000 7f2482600640 1 --2- 172.21.15.16:0/131391765 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f247c06e330 0x7f247c06dd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:20.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:20.686+0000 7f2482600640 1 --2- 172.21.15.16:0/131391765 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f2460004810 0x7f247c06d7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:20.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:20.686+0000 7f2482600640 1 -- 172.21.15.16:0/131391765 >> 172.21.15.16:0/131391765 conn(0x7f247c067930 msgr2=0x7f247c06fd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:08:20.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:20.686+0000 7f2482600640 1 -- 172.21.15.16:0/131391765 shutdown_connections 2024-03-31T15:08:20.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:20.686+0000 7f2482600640 1 -- 172.21.15.16:0/131391765 wait complete. 2024-03-31T15:08:20.700 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-31T15:08:20.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:20.850+0000 7f962ee00640 1 Processor -- start 2024-03-31T15:08:20.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:20.850+0000 7f962ee00640 1 -- start start 2024-03-31T15:08:20.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:20.850+0000 7f962ee00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f9628141590 0x7f9628144340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:20.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:20.850+0000 7f962ee00640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9628144980 con 0x7f9628070030 2024-03-31T15:08:20.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:20.850+0000 7f962ee00640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9628161fe0 con 0x7f96281411b0 2024-03-31T15:08:20.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:20.850+0000 7f962ee00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f962815f910 con 0x7f9628141590 2024-03-31T15:08:20.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:20.850+0000 7f961fe00640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f9628070030 0x7f9628143c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:60030/0 (socket says 172.21.15.16:60030) 2024-03-31T15:08:20.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:20.850+0000 7f961fe00640 1 -- 172.21.15.16:0/3647944738 learned_addr learned my addr 172.21.15.16:0/3647944738 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:08:20.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:20.850+0000 7f962ce00640 1 --2- 172.21.15.16:0/3647944738 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f9628141590 0x7f9628144340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:20.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:20.854+0000 7f9627400640 1 -- 172.21.15.16:0/3647944738 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4037917631 0 0) 0x7f9628144980 con 0x7f9628070030 2024-03-31T15:08:20.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:20.854+0000 7f9627400640 1 -- 172.21.15.16:0/3647944738 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f960c003530 con 0x7f9628070030 2024-03-31T15:08:20.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:20.854+0000 7f9627400640 1 -- 172.21.15.16:0/3647944738 <== mon.1 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3102508949 0 0) 0x7f9628161fe0 con 0x7f96281411b0 2024-03-31T15:08:20.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:20.854+0000 7f9627400640 1 -- 172.21.15.16:0/3647944738 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9628144980 con 0x7f96281411b0 2024-03-31T15:08:20.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:20.854+0000 7f962ce00640 1 -- 172.21.15.16:0/3647944738 >> v1:172.21.15.89:6789/0 conn(0x7f96281411b0 legacy=0x7f962806f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:20.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:20.854+0000 7f962ce00640 1 -- 172.21.15.16:0/3647944738 >> v1:172.21.15.16:6789/0 conn(0x7f9628070030 legacy=0x7f9628143c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:20.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:20.854+0000 7f962ce00640 1 -- 172.21.15.16:0/3647944738 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96281622c0 con 0x7f9628141590 2024-03-31T15:08:20.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:20.854+0000 7f962ce00640 1 --2- 172.21.15.16:0/3647944738 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f9628141590 0x7f9628144340 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f9620009dc0 tx=0x7f962001c990 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c50f6957595f52c5 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:08:20.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:20.854+0000 7f9627400640 1 -- 172.21.15.16:0/3647944738 <== mon.2 v2:172.21.15.89:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96200b2070 con 0x7f9628141590 2024-03-31T15:08:20.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:20.854+0000 7f9627400640 1 -- 172.21.15.16:0/3647944738 <== mon.2 v2:172.21.15.89:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f96200a5960 con 0x7f9628141590 2024-03-31T15:08:20.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:20.854+0000 7f9627400640 1 -- 172.21.15.16:0/3647944738 <== mon.2 v2:172.21.15.89:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96200a5d10 con 0x7f9628141590 2024-03-31T15:08:20.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:20.858+0000 7f962ee00640 1 -- 172.21.15.16:0/3647944738 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f9628141590 msgr2=0x7f9628144340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:20.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:20.858+0000 7f962ee00640 1 --2- 172.21.15.16:0/3647944738 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f9628141590 0x7f9628144340 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f9620009dc0 tx=0x7f962001c990 comp rx=0 tx=0).stop 2024-03-31T15:08:20.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:20.858+0000 7f962ee00640 1 -- 172.21.15.16:0/3647944738 shutdown_connections 2024-03-31T15:08:20.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:20.858+0000 7f962ee00640 1 --2- 172.21.15.16:0/3647944738 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f9628141590 0x7f9628144340 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:20.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:20.858+0000 7f962ee00640 1 -- 172.21.15.16:0/3647944738 >> 172.21.15.16:0/3647944738 conn(0x7f9628067930 msgr2=0x7f96280b3ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:08:20.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:20.858+0000 7f962ee00640 1 -- 172.21.15.16:0/3647944738 shutdown_connections 2024-03-31T15:08:20.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:20.858+0000 7f962ee00640 1 -- 172.21.15.16:0/3647944738 wait complete. 2024-03-31T15:08:20.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:20.858+0000 7f962ee00640 1 Processor -- start 2024-03-31T15:08:20.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:20.858+0000 7f962ee00640 1 -- start start 2024-03-31T15:08:20.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:20.858+0000 7f962ee00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9628070030 0x7f9628154e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:20.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:20.858+0000 7f962ee00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f96281411b0 0x7f96281373c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:20.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:20.858+0000 7f962ee00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9628141590 0x7f9628152080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:20.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:20.858+0000 7f962ee00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f962816be80 con 0x7f9628070030 2024-03-31T15:08:20.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:20.858+0000 7f962ee00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f962816bd00 con 0x7f9628141590 2024-03-31T15:08:20.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:20.858+0000 7f962ee00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f962816c000 con 0x7f96281411b0 2024-03-31T15:08:20.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:20.858+0000 7f9627e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9628070030 0x7f9628154e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:20.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:20.858+0000 7f9627e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9628070030 0x7f9628154e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38670/0 (socket says 172.21.15.16:38670) 2024-03-31T15:08:20.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:20.858+0000 7f9627e00640 1 -- 172.21.15.16:0/35392503 learned_addr learned my addr 172.21.15.16:0/35392503 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:08:20.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:20.858+0000 7f961fe00640 1 --2- 172.21.15.16:0/35392503 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f96281411b0 0x7f96281373c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:20.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:20.858+0000 7f962ce00640 1 --2- 172.21.15.16:0/35392503 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9628141590 0x7f9628152080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:20.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:20.858+0000 7f9627e00640 1 -- 172.21.15.16:0/35392503 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f96281411b0 msgr2=0x7f96281373c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:20.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:20.858+0000 7f9627e00640 1 --2- 172.21.15.16:0/35392503 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f96281411b0 0x7f96281373c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:20.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:20.858+0000 7f9627e00640 1 -- 172.21.15.16:0/35392503 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9628141590 msgr2=0x7f9628152080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:20.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:20.858+0000 7f9627e00640 1 --2- 172.21.15.16:0/35392503 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9628141590 0x7f9628152080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:20.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:20.858+0000 7f9627e00640 1 -- 172.21.15.16:0/35392503 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96281539d0 con 0x7f9628070030 2024-03-31T15:08:20.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:20.858+0000 7f961fe00640 1 --2- 172.21.15.16:0/35392503 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f96281411b0 0x7f96281373c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:08:20.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:20.858+0000 7f962ce00640 1 --2- 172.21.15.16:0/35392503 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9628141590 0x7f9628152080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:08:20.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:20.862+0000 7f9627e00640 1 --2- 172.21.15.16:0/35392503 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9628070030 0x7f9628154e80 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7f961800c500 tx=0x7f961800c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:08:20.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:20.862+0000 7f9624c00640 1 -- 172.21.15.16:0/35392503 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96180052d0 con 0x7f9628070030 2024-03-31T15:08:20.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:20.862+0000 7f9624c00640 1 -- 172.21.15.16:0/35392503 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9618005490 con 0x7f9628070030 2024-03-31T15:08:20.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:20.862+0000 7f962ee00640 1 -- 172.21.15.16:0/35392503 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9628155510 con 0x7f9628070030 2024-03-31T15:08:20.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:20.862+0000 7f9624c00640 1 -- 172.21.15.16:0/35392503 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9618012850 con 0x7f9628070030 2024-03-31T15:08:20.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:20.862+0000 7f962ee00640 1 -- 172.21.15.16:0/35392503 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9628155ad0 con 0x7f9628070030 2024-03-31T15:08:20.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:20.862+0000 7f962ee00640 1 -- 172.21.15.16:0/35392503 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9628069950 con 0x7f9628070030 2024-03-31T15:08:20.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:20.862+0000 7f9624c00640 1 -- 172.21.15.16:0/35392503 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f96180129f0 con 0x7f9628070030 2024-03-31T15:08:20.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:20.862+0000 7f9624c00640 1 --2- 172.21.15.16:0/35392503 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f9604068380 0x7f960406a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:20.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:20.862+0000 7f9624c00640 1 -- 172.21.15.16:0/35392503 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(70..70 src has 1..70) ==== 5882+0+0 (secure 0 0 0) 0x7f9618086790 con 0x7f9628070030 2024-03-31T15:08:20.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:20.866+0000 7f961fe00640 1 --2- 172.21.15.16:0/35392503 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f9604068380 0x7f960406a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:20.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:20.866+0000 7f961fe00640 1 --2- 172.21.15.16:0/35392503 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f9604068380 0x7f960406a840 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f96281faba0 tx=0x7f9610009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:08:20.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:20.870+0000 7f9624c00640 1 -- 172.21.15.16:0/35392503 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f96180565c0 con 0x7f9628070030 2024-03-31T15:08:21.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:21.034+0000 7f962ee00640 1 -- 172.21.15.16:0/35392503 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f96280b4a20 con 0x7f9628070030 2024-03-31T15:08:21.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:21.034+0000 7f9624c00640 1 -- 172.21.15.16:0/35392503 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v70) ==== 100+0+11 (secure 0 0 0) 0x7f961805a8c0 con 0x7f9628070030 2024-03-31T15:08:21.029 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 64 2024-03-31T15:08:21.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:21.034+0000 7f962ee00640 1 -- 172.21.15.16:0/35392503 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f9604068380 msgr2=0x7f960406a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:21.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:21.034+0000 7f962ee00640 1 --2- 172.21.15.16:0/35392503 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f9604068380 0x7f960406a840 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f96281faba0 tx=0x7f9610009040 comp rx=0 tx=0).stop 2024-03-31T15:08:21.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:21.034+0000 7f962ee00640 1 -- 172.21.15.16:0/35392503 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9628070030 msgr2=0x7f9628154e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:21.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:21.034+0000 7f962ee00640 1 --2- 172.21.15.16:0/35392503 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9628070030 0x7f9628154e80 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7f961800c500 tx=0x7f961800c9c0 comp rx=0 tx=0).stop 2024-03-31T15:08:21.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:21.034+0000 7f962ee00640 1 -- 172.21.15.16:0/35392503 shutdown_connections 2024-03-31T15:08:21.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:21.034+0000 7f962ee00640 1 --2- 172.21.15.16:0/35392503 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f9604068380 0x7f960406a840 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:21.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:21.034+0000 7f962ee00640 1 --2- 172.21.15.16:0/35392503 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9628141590 0x7f9628152080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:21.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:21.034+0000 7f962ee00640 1 --2- 172.21.15.16:0/35392503 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f96281411b0 0x7f96281373c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:21.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:21.034+0000 7f962ee00640 1 --2- 172.21.15.16:0/35392503 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9628070030 0x7f9628154e80 unknown :-1 s=CLOSED pgs=921 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:21.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:21.034+0000 7f962ee00640 1 -- 172.21.15.16:0/35392503 >> 172.21.15.16:0/35392503 conn(0x7f9628067930 msgr2=0x7f9628155d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:08:21.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:21.034+0000 7f962ee00640 1 -- 172.21.15.16:0/35392503 shutdown_connections 2024-03-31T15:08:21.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:21.034+0000 7f962ee00640 1 -- 172.21.15.16:0/35392503 wait complete. 2024-03-31T15:08:21.050 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-31T15:08:21.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:21.194+0000 7f3442a00640 1 Processor -- start 2024-03-31T15:08:21.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:21.194+0000 7f3442a00640 1 -- start start 2024-03-31T15:08:21.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:21.198+0000 7f3442a00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f343c06f920 0x7f343c06fd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:21.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:21.198+0000 7f3442a00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f343c0d2010 0x7f343c0d24a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:21.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:21.198+0000 7f3442a00640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f343c0d0920 con 0x7f343c145e40 2024-03-31T15:08:21.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:21.198+0000 7f3442a00640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f343c0702c0 con 0x7f343c0d2010 2024-03-31T15:08:21.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:21.198+0000 7f3442a00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f343c0d06d0 con 0x7f343c06f920 2024-03-31T15:08:21.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:21.198+0000 7f3440a00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f343c0d2010 0x7f343c0d24a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:21.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:21.198+0000 7f3433e00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f343c06f920 0x7f343c06fd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:21.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:21.198+0000 7f3440a00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f343c0d2010 0x7f343c0d24a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.16:50044/0 (socket says 172.21.15.16:50044) 2024-03-31T15:08:21.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:21.198+0000 7f3440a00640 1 -- 172.21.15.16:0/102268427 learned_addr learned my addr 172.21.15.16:0/102268427 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:08:21.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:21.198+0000 7f3440a00640 1 -- 172.21.15.16:0/102268427 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f343c06f920 msgr2=0x7f343c06fd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:08:21.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:21.198+0000 7f3440a00640 1 --2- 172.21.15.16:0/102268427 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f343c06f920 0x7f343c06fd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:21.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:21.198+0000 7f3440a00640 1 -- 172.21.15.16:0/102268427 >> v1:172.21.15.89:6789/0 conn(0x7f343c145e40 legacy=0x7f343c0cc740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:21.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:21.198+0000 7f3440a00640 1 -- 172.21.15.16:0/102268427 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f343c0d0c00 con 0x7f343c0d2010 2024-03-31T15:08:21.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:21.202+0000 7f3440a00640 1 --2- 172.21.15.16:0/102268427 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f343c0d2010 0x7f343c0d24a0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f3434009e50 tx=0x7f343401ca30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c9afe4d5cef4c595 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:08:21.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:21.202+0000 7f343b400640 1 -- 172.21.15.16:0/102268427 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34340a57f0 con 0x7f343c0d2010 2024-03-31T15:08:21.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:21.202+0000 7f343b400640 1 -- 172.21.15.16:0/102268427 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f34340a5990 con 0x7f343c0d2010 2024-03-31T15:08:21.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:21.202+0000 7f343b400640 1 -- 172.21.15.16:0/102268427 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34340a5d60 con 0x7f343c0d2010 2024-03-31T15:08:21.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:21.202+0000 7f3442a00640 1 -- 172.21.15.16:0/102268427 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f343c0d2010 msgr2=0x7f343c0d24a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:21.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:21.202+0000 7f3442a00640 1 --2- 172.21.15.16:0/102268427 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f343c0d2010 0x7f343c0d24a0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f3434009e50 tx=0x7f343401ca30 comp rx=0 tx=0).stop 2024-03-31T15:08:21.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:21.202+0000 7f3442a00640 1 -- 172.21.15.16:0/102268427 shutdown_connections 2024-03-31T15:08:21.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:21.202+0000 7f3442a00640 1 --2- 172.21.15.16:0/102268427 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f343c0d2010 0x7f343c0d24a0 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:21.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:21.202+0000 7f3442a00640 1 --2- 172.21.15.16:0/102268427 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f343c06f920 0x7f343c06fd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:21.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:21.202+0000 7f3442a00640 1 -- 172.21.15.16:0/102268427 >> 172.21.15.16:0/102268427 conn(0x7f343c067330 msgr2=0x7f343c067740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:08:21.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:21.202+0000 7f3442a00640 1 -- 172.21.15.16:0/102268427 shutdown_connections 2024-03-31T15:08:21.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:21.202+0000 7f3442a00640 1 -- 172.21.15.16:0/102268427 wait complete. 2024-03-31T15:08:21.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:21.202+0000 7f3442a00640 1 Processor -- start 2024-03-31T15:08:21.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:21.202+0000 7f3442a00640 1 -- start start 2024-03-31T15:08:21.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:21.202+0000 7f3442a00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f343c06f920 0x7f343c0b1c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:21.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:21.202+0000 7f3442a00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f343c0d2010 0x7f343c0b6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:21.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:21.202+0000 7f3442a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f343c145e40 0x7f343c158950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:21.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:21.202+0000 7f3442a00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f343c0d0ee0 con 0x7f343c145e40 2024-03-31T15:08:21.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:21.202+0000 7f3442a00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f343c0ce9e0 con 0x7f343c06f920 2024-03-31T15:08:21.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:21.202+0000 7f3442a00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f343c0d2df0 con 0x7f343c0d2010 2024-03-31T15:08:21.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:21.202+0000 7f3440a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f343c145e40 0x7f343c158950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:21.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:21.202+0000 7f343be00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f343c06f920 0x7f343c0b1c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:21.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:21.206+0000 7f3433e00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f343c0d2010 0x7f343c0b6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:21.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:21.206+0000 7f343be00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f343c06f920 0x7f343c0b1c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.16:50050/0 (socket says 172.21.15.16:50050) 2024-03-31T15:08:21.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:21.206+0000 7f3440a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f343c145e40 0x7f343c158950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38672/0 (socket says 172.21.15.16:38672) 2024-03-31T15:08:21.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:21.206+0000 7f3433e00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f343c0d2010 0x7f343c0b6070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3301/0 says I am v2:172.21.15.16:51638/0 (socket says 172.21.15.16:51638) 2024-03-31T15:08:21.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:21.206+0000 7f343be00640 1 -- 172.21.15.16:0/2625306473 learned_addr learned my addr 172.21.15.16:0/2625306473 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:08:21.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:21.206+0000 7f3440a00640 1 -- 172.21.15.16:0/2625306473 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f343c0d2010 msgr2=0x7f343c0b6070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:21.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:21.206+0000 7f3440a00640 1 --2- 172.21.15.16:0/2625306473 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f343c0d2010 0x7f343c0b6070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:21.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:21.206+0000 7f3440a00640 1 -- 172.21.15.16:0/2625306473 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f343c06f920 msgr2=0x7f343c0b1c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:21.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:21.206+0000 7f3440a00640 1 --2- 172.21.15.16:0/2625306473 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f343c06f920 0x7f343c0b1c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:21.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:21.206+0000 7f3440a00640 1 -- 172.21.15.16:0/2625306473 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f343c156b50 con 0x7f343c145e40 2024-03-31T15:08:21.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:21.206+0000 7f3440a00640 1 --2- 172.21.15.16:0/2625306473 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f343c145e40 0x7f343c158950 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7f34340ab000 tx=0x7f34340197d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:08:21.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:21.206+0000 7f3438c00640 1 -- 172.21.15.16:0/2625306473 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34340b8470 con 0x7f343c145e40 2024-03-31T15:08:21.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:21.206+0000 7f3438c00640 1 -- 172.21.15.16:0/2625306473 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f34340b8610 con 0x7f343c145e40 2024-03-31T15:08:21.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:21.206+0000 7f3442a00640 1 -- 172.21.15.16:0/2625306473 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f343c156e30 con 0x7f343c145e40 2024-03-31T15:08:21.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:21.206+0000 7f3438c00640 1 -- 172.21.15.16:0/2625306473 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34340b89e0 con 0x7f343c145e40 2024-03-31T15:08:21.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:21.206+0000 7f3442a00640 1 -- 172.21.15.16:0/2625306473 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f343c1573f0 con 0x7f343c145e40 2024-03-31T15:08:21.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:21.206+0000 7f3442a00640 1 -- 172.21.15.16:0/2625306473 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f343c069950 con 0x7f343c145e40 2024-03-31T15:08:21.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:21.206+0000 7f3438c00640 1 -- 172.21.15.16:0/2625306473 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f34340c0020 con 0x7f343c145e40 2024-03-31T15:08:21.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:21.206+0000 7f3438c00640 1 --2- 172.21.15.16:0/2625306473 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f3418068380 0x7f341806a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:21.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:21.210+0000 7f3438c00640 1 -- 172.21.15.16:0/2625306473 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(70..70 src has 1..70) ==== 5882+0+0 (secure 0 0 0) 0x7f3434121410 con 0x7f343c145e40 2024-03-31T15:08:21.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:21.210+0000 7f343be00640 1 --2- 172.21.15.16:0/2625306473 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f3418068380 0x7f341806a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:21.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:21.210+0000 7f343be00640 1 --2- 172.21.15.16:0/2625306473 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f3418068380 0x7f341806a840 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f343c0cc350 tx=0x7f342c009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:08:21.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:21.218+0000 7f3438c00640 1 -- 172.21.15.16:0/2625306473 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f3434016620 con 0x7f343c145e40 2024-03-31T15:08:21.294 INFO:tasks.ceph.mon.a.smithi016.stderr:2024-03-31T15:08:21.298+0000 7f7dcee00640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 MDSs report damaged metadata (MDS_DAMAGE) 2024-03-31T15:08:21.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:21.394+0000 7f3442a00640 1 -- 172.21.15.16:0/2625306473 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f343c0b7e90 con 0x7f343c145e40 2024-03-31T15:08:21.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:21.394+0000 7f3438c00640 1 -- 172.21.15.16:0/2625306473 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 2 entries v70) ==== 80+0+122 (secure 0 0 0) 0x7f34340f1360 con 0x7f343c145e40 2024-03-31T15:08:21.391 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:6835/2555930820 2024-04-01T15:08:10.537891+0000 2024-03-31T15:08:21.391 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:6834/2555930820 2024-04-01T15:08:10.537891+0000 2024-03-31T15:08:21.391 INFO:teuthology.orchestra.run.smithi016.stderr:listed 2 entries 2024-03-31T15:08:21.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:21.398+0000 7f3442a00640 1 -- 172.21.15.16:0/2625306473 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f3418068380 msgr2=0x7f341806a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:21.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:21.398+0000 7f3442a00640 1 --2- 172.21.15.16:0/2625306473 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f3418068380 0x7f341806a840 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f343c0cc350 tx=0x7f342c009040 comp rx=0 tx=0).stop 2024-03-31T15:08:21.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:21.398+0000 7f3442a00640 1 -- 172.21.15.16:0/2625306473 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f343c145e40 msgr2=0x7f343c158950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:21.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:21.398+0000 7f3442a00640 1 --2- 172.21.15.16:0/2625306473 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f343c145e40 0x7f343c158950 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7f34340ab000 tx=0x7f34340197d0 comp rx=0 tx=0).stop 2024-03-31T15:08:21.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:21.398+0000 7f3442a00640 1 -- 172.21.15.16:0/2625306473 shutdown_connections 2024-03-31T15:08:21.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:21.398+0000 7f3442a00640 1 --2- 172.21.15.16:0/2625306473 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f3418068380 0x7f341806a840 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:21.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:21.398+0000 7f3442a00640 1 --2- 172.21.15.16:0/2625306473 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f343c145e40 0x7f343c158950 unknown :-1 s=CLOSED pgs=922 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:21.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:21.398+0000 7f3442a00640 1 --2- 172.21.15.16:0/2625306473 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f343c0d2010 0x7f343c0b6070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:21.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:21.398+0000 7f3442a00640 1 --2- 172.21.15.16:0/2625306473 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f343c06f920 0x7f343c0b1c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:21.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:21.398+0000 7f3442a00640 1 -- 172.21.15.16:0/2625306473 >> 172.21.15.16:0/2625306473 conn(0x7f343c067330 msgr2=0x7f343c067710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:08:21.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:21.398+0000 7f3442a00640 1 -- 172.21.15.16:0/2625306473 shutdown_connections 2024-03-31T15:08:21.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:21.398+0000 7f3442a00640 1 -- 172.21.15.16:0/2625306473 wait complete. 2024-03-31T15:08:21.412 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi016.front.sepia.ceph.com... 2024-03-31T15:08:21.412 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:08:21.412 DEBUG:teuthology.orchestra.run.smithi016:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-03-31T15:08:21.430 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi016.stderr:ceph-fuse[42435]: fuse finished with error 0 and tester_r 0 2024-03-31T15:08:21.433 INFO:teuthology.orchestra.run:waiting for 300 2024-03-31T15:08:27.534 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi016.front.sepia.ceph.com 2024-03-31T15:08:27.534 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:08:27.534 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-31T15:08:27.544 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:08:27.544 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.5:0 scrub start /subdir recursive 2024-03-31T15:08:27.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.733+0000 7fd77c600640 1 Processor -- start 2024-03-31T15:08:27.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.733+0000 7fd77c600640 1 -- start start 2024-03-31T15:08:27.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.733+0000 7fd77c600640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fd7740b7900 0x7fd774160740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:27.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.733+0000 7fd77c600640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fd7740b44d0 0x7fd7740d1a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:27.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.733+0000 7fd77c600640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd7740ce2d0 con 0x7fd7740cc7b0 2024-03-31T15:08:27.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.737+0000 7fd77c600640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7fd7740ceb10 con 0x7fd7740b44d0 2024-03-31T15:08:27.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.737+0000 7fd77c600640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fd7740ce550 con 0x7fd7740b7900 2024-03-31T15:08:27.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.737+0000 7fd77a600640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fd7740b44d0 0x7fd7740d1a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:27.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.737+0000 7fd779200640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fd7740cc7b0 0x7fd7740b3dc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:34964/0 (socket says 172.21.15.16:34964) 2024-03-31T15:08:27.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.737+0000 7fd779200640 1 -- 172.21.15.16:0/4060569970 learned_addr learned my addr 172.21.15.16:0/4060569970 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:08:27.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.737+0000 7fd779c00640 1 --2- 172.21.15.16:0/4060569970 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fd7740b7900 0x7fd774160740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:27.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.737+0000 7fd77a600640 1 -- 172.21.15.16:0/4060569970 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fd7740b7900 msgr2=0x7fd774160740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:08:27.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.737+0000 7fd77a600640 1 --2- 172.21.15.16:0/4060569970 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fd7740b7900 0x7fd774160740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:27.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.737+0000 7fd77a600640 1 -- 172.21.15.16:0/4060569970 >> v1:172.21.15.16:6789/0 conn(0x7fd7740cc7b0 legacy=0x7fd7740b3dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:27.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.737+0000 7fd77a600640 1 -- 172.21.15.16:0/4060569970 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7740ce830 con 0x7fd7740b44d0 2024-03-31T15:08:27.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.737+0000 7fd779c00640 1 --2- 172.21.15.16:0/4060569970 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fd7740b7900 0x7fd774160740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:08:27.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.737+0000 7fd77a600640 1 --2- 172.21.15.16:0/4060569970 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd7740b44d0 0x7fd7740d1a80 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7fd76c009d90 tx=0x7fd76c01c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=310f54fe86ed82b0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:08:27.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.737+0000 7fd773e00640 1 -- 172.21.15.16:0/4060569970 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd76c0b2070 con 0x7fd7740b44d0 2024-03-31T15:08:27.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.737+0000 7fd773e00640 1 -- 172.21.15.16:0/4060569970 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd76c0a5970 con 0x7fd7740b44d0 2024-03-31T15:08:27.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.737+0000 7fd773e00640 1 -- 172.21.15.16:0/4060569970 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd76c0a5d40 con 0x7fd7740b44d0 2024-03-31T15:08:27.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.741+0000 7fd77c600640 1 -- 172.21.15.16:0/4060569970 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd7740b44d0 msgr2=0x7fd7740d1a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:27.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.741+0000 7fd77c600640 1 --2- 172.21.15.16:0/4060569970 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd7740b44d0 0x7fd7740d1a80 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7fd76c009d90 tx=0x7fd76c01c8b0 comp rx=0 tx=0).stop 2024-03-31T15:08:27.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.741+0000 7fd77c600640 1 -- 172.21.15.16:0/4060569970 shutdown_connections 2024-03-31T15:08:27.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.741+0000 7fd77c600640 1 --2- 172.21.15.16:0/4060569970 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd7740b44d0 0x7fd7740d1a80 unknown :-1 s=CLOSED pgs=924 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:27.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.741+0000 7fd77c600640 1 --2- 172.21.15.16:0/4060569970 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fd7740b7900 0x7fd774160740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:27.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.741+0000 7fd77c600640 1 -- 172.21.15.16:0/4060569970 >> 172.21.15.16:0/4060569970 conn(0x7fd77406b7f0 msgr2=0x7fd774067330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:08:27.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.741+0000 7fd77c600640 1 -- 172.21.15.16:0/4060569970 shutdown_connections 2024-03-31T15:08:27.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.741+0000 7fd77c600640 1 -- 172.21.15.16:0/4060569970 wait complete. 2024-03-31T15:08:27.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.741+0000 7fd77c600640 1 Processor -- start 2024-03-31T15:08:27.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.741+0000 7fd77c600640 1 -- start start 2024-03-31T15:08:27.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.741+0000 7fd77c600640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fd7740b44d0 0x7fd7741404b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:27.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.741+0000 7fd77c600640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd7740b7900 0x7fd774158960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:27.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.741+0000 7fd77c600640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd7740cc7b0 0x7fd7740cb4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:27.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.741+0000 7fd77c600640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fd77415d540 con 0x7fd7740cc7b0 2024-03-31T15:08:27.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.741+0000 7fd77c600640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fd7740b7ce0 con 0x7fd7740b7900 2024-03-31T15:08:27.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.741+0000 7fd77c600640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fd77415d6c0 con 0x7fd7740b44d0 2024-03-31T15:08:27.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.741+0000 7fd77a600640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd7740cc7b0 0x7fd7740cb4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:27.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.741+0000 7fd77a600640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd7740cc7b0 0x7fd7740cb4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34566/0 (socket says 172.21.15.16:34566) 2024-03-31T15:08:27.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.741+0000 7fd77a600640 1 -- 172.21.15.16:0/524429024 learned_addr learned my addr 172.21.15.16:0/524429024 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:08:27.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.741+0000 7fd779c00640 1 --2- 172.21.15.16:0/524429024 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fd7740b44d0 0x7fd7741404b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:27.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.741+0000 7fd779200640 1 --2- 172.21.15.16:0/524429024 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd7740b7900 0x7fd774158960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:27.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.741+0000 7fd77a600640 1 -- 172.21.15.16:0/524429024 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fd7740b44d0 msgr2=0x7fd7741404b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:27.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.741+0000 7fd77a600640 1 --2- 172.21.15.16:0/524429024 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fd7740b44d0 0x7fd7741404b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:27.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.741+0000 7fd77a600640 1 -- 172.21.15.16:0/524429024 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd7740b7900 msgr2=0x7fd774158960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:27.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.741+0000 7fd77a600640 1 --2- 172.21.15.16:0/524429024 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd7740b7900 0x7fd774158960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:27.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.741+0000 7fd77a600640 1 -- 172.21.15.16:0/524429024 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd774157be0 con 0x7fd7740cc7b0 2024-03-31T15:08:27.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.741+0000 7fd779200640 1 --2- 172.21.15.16:0/524429024 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd7740b7900 0x7fd774158960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:08:27.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.745+0000 7fd77a600640 1 --2- 172.21.15.16:0/524429024 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd7740cc7b0 0x7fd7740cb4b0 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7fd76c0b0040 tx=0x7fd7740d27d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:08:27.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.745+0000 7fd771600640 1 -- 172.21.15.16:0/524429024 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd76c0bc030 con 0x7fd7740cc7b0 2024-03-31T15:08:27.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.745+0000 7fd771600640 1 -- 172.21.15.16:0/524429024 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd76c0ad440 con 0x7fd7740cc7b0 2024-03-31T15:08:27.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.745+0000 7fd771600640 1 -- 172.21.15.16:0/524429024 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd76c0b2040 con 0x7fd7740cc7b0 2024-03-31T15:08:27.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.745+0000 7fd77c600640 1 -- 172.21.15.16:0/524429024 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd774143c30 con 0x7fd7740cc7b0 2024-03-31T15:08:27.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.745+0000 7fd77c600640 1 -- 172.21.15.16:0/524429024 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd774158160 con 0x7fd7740cc7b0 2024-03-31T15:08:27.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.745+0000 7fd771600640 1 -- 172.21.15.16:0/524429024 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fd76c0bf070 con 0x7fd7740cc7b0 2024-03-31T15:08:27.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.749+0000 7fd77fa25000 1 Processor -- start 2024-03-31T15:08:27.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.749+0000 7fd77fa25000 1 -- start start 2024-03-31T15:08:27.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.749+0000 7fd771600640 1 --2- 172.21.15.16:0/524429024 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fd7540683b0 0x7fd75406a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:27.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.749+0000 7fd771600640 1 -- 172.21.15.16:0/524429024 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(70..70 src has 1..70) ==== 5882+0+0 (secure 0 0 0) 0x7fd76c1210c0 con 0x7fd7740cc7b0 2024-03-31T15:08:27.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.749+0000 7fd779200640 1 --2- 172.21.15.16:0/524429024 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fd7540683b0 0x7fd75406a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:27.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.749+0000 7fd77fa25000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cf51fd8210 0x55cf51fd85f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:27.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.749+0000 7fd77fa25000 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cf51fd8b30 0x55cf51f8e740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:27.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.749+0000 7fd77fa25000 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55cf51f8ec80 0x55cf51f8f060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:27.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.749+0000 7fd77fa25000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55cf51ddda20 con 0x55cf51fd8210 2024-03-31T15:08:27.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.749+0000 7fd77fa25000 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55cf51d7aa60 con 0x55cf51fd8b30 2024-03-31T15:08:27.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.749+0000 7fd77fa25000 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x55cf51dd99d0 con 0x55cf51f8ec80 2024-03-31T15:08:27.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.749+0000 7fd779c00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cf51fd8210 0x55cf51fd85f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:27.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.749+0000 7fd779c00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cf51fd8210 0x55cf51fd85f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34582/0 (socket says 172.21.15.16:34582) 2024-03-31T15:08:27.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.749+0000 7fd779c00640 1 -- 172.21.15.16:0/2639663712 learned_addr learned my addr 172.21.15.16:0/2639663712 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:08:27.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.749+0000 7fd779200640 1 --2- 172.21.15.16:0/2639663712 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cf51fd8b30 0x55cf51f8e740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:27.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.749+0000 7fd77a600640 1 --2- 172.21.15.16:0/2639663712 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55cf51f8ec80 0x55cf51f8f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:27.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.749+0000 7fd779200640 1 --2- 172.21.15.16:0/524429024 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fd7540683b0 0x7fd75406a870 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7fd7740d1b00 tx=0x7fd760009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:08:27.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.749+0000 7fd779c00640 1 -- 172.21.15.16:0/2639663712 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55cf51f8ec80 msgr2=0x55cf51f8f060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:08:27.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.749+0000 7fd779c00640 1 --2- 172.21.15.16:0/2639663712 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55cf51f8ec80 0x55cf51f8f060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:27.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.749+0000 7fd779c00640 1 -- 172.21.15.16:0/2639663712 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cf51fd8b30 msgr2=0x55cf51f8e740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:08:27.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.749+0000 7fd779c00640 1 --2- 172.21.15.16:0/2639663712 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cf51fd8b30 0x55cf51f8e740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:27.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.749+0000 7fd779c00640 1 -- 172.21.15.16:0/2639663712 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cf51e220f0 con 0x55cf51fd8210 2024-03-31T15:08:27.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.749+0000 7fd779c00640 1 --2- 172.21.15.16:0/2639663712 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cf51fd8210 0x55cf51fd85f0 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7fd768007910 tx=0x7fd768007dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a7d4c3ac63d691 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:08:27.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.749+0000 7fd766000640 1 -- 172.21.15.16:0/2639663712 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd768018070 con 0x55cf51fd8210 2024-03-31T15:08:27.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.749+0000 7fd766000640 1 -- 172.21.15.16:0/2639663712 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd768010b00 con 0x55cf51fd8210 2024-03-31T15:08:27.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.749+0000 7fd766000640 1 -- 172.21.15.16:0/2639663712 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd768014430 con 0x55cf51fd8210 2024-03-31T15:08:27.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.749+0000 7fd77fa25000 1 -- 172.21.15.16:0/2639663712 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cf51fd8210 msgr2=0x55cf51fd85f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:27.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.749+0000 7fd77fa25000 1 --2- 172.21.15.16:0/2639663712 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cf51fd8210 0x55cf51fd85f0 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7fd768007910 tx=0x7fd768007dd0 comp rx=0 tx=0).stop 2024-03-31T15:08:27.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.749+0000 7fd77fa25000 1 -- 172.21.15.16:0/2639663712 shutdown_connections 2024-03-31T15:08:27.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.749+0000 7fd77fa25000 1 --2- 172.21.15.16:0/2639663712 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55cf51f8ec80 0x55cf51f8f060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:27.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.749+0000 7fd77fa25000 1 --2- 172.21.15.16:0/2639663712 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cf51fd8b30 0x55cf51f8e740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:27.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.749+0000 7fd77fa25000 1 --2- 172.21.15.16:0/2639663712 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cf51fd8210 0x55cf51fd85f0 unknown :-1 s=CLOSED pgs=926 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:27.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.749+0000 7fd77fa25000 1 -- 172.21.15.16:0/2639663712 >> 172.21.15.16:0/2639663712 conn(0x55cf51f8ac30 msgr2=0x55cf51f8d020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:08:27.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.749+0000 7fd77fa25000 1 -- 172.21.15.16:0/2639663712 shutdown_connections 2024-03-31T15:08:27.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.749+0000 7fd77fa25000 1 -- 172.21.15.16:0/2639663712 wait complete. 2024-03-31T15:08:27.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.753+0000 7fd77fa25000 1 -- start start 2024-03-31T15:08:27.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.753+0000 7fd77fa25000 1 Processor -- start 2024-03-31T15:08:27.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.753+0000 7fd77fa25000 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55cf51f8ec80 0x55cf51fbb3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:27.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.753+0000 7fd77fa25000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cf51fd8210 0x55cf51fbb900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:27.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.753+0000 7fd77fa25000 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cf51fd8b30 0x55cf51fd8f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:27.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.753+0000 7fd77fa25000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55cf51d9b0a0 con 0x55cf51fd8210 2024-03-31T15:08:27.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.753+0000 7fd77fa25000 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55cf51de2050 con 0x55cf51fd8b30 2024-03-31T15:08:27.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.753+0000 7fd77fa25000 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x55cf51f8aab0 con 0x55cf51f8ec80 2024-03-31T15:08:27.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.753+0000 7fd77fa25000 10 client.0 _ll_register_callbacks cb 0x55cf51f798e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T15:08:27.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.753+0000 7fd779200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cf51fd8210 0x55cf51fbb900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:27.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.753+0000 7fd779200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cf51fd8210 0x55cf51fbb900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34588/0 (socket says 172.21.15.16:34588) 2024-03-31T15:08:27.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.753+0000 7fd779200640 1 -- 172.21.15.16:0/2273466780 learned_addr learned my addr 172.21.15.16:0/2273466780 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:08:27.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.753+0000 7fd779c00640 1 --2- 172.21.15.16:0/2273466780 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55cf51f8ec80 0x55cf51fbb3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:27.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.753+0000 7fd77a600640 1 --2- 172.21.15.16:0/2273466780 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cf51fd8b30 0x55cf51fd8f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:27.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.753+0000 7fd779200640 1 -- 172.21.15.16:0/2273466780 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55cf51f8ec80 msgr2=0x55cf51fbb3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:08:27.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.753+0000 7fd779200640 1 --2- 172.21.15.16:0/2273466780 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55cf51f8ec80 0x55cf51fbb3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:27.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.753+0000 7fd779200640 1 -- 172.21.15.16:0/2273466780 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cf51fd8b30 msgr2=0x55cf51fd8f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:08:27.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.753+0000 7fd779200640 1 --2- 172.21.15.16:0/2273466780 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cf51fd8b30 0x55cf51fd8f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:27.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.753+0000 7fd779200640 1 -- 172.21.15.16:0/2273466780 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cf51da52a0 con 0x55cf51fd8210 2024-03-31T15:08:27.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.753+0000 7fd779200640 1 --2- 172.21.15.16:0/2273466780 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cf51fd8210 0x55cf51fbb900 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7fd76000ef30 tx=0x7fd760000bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b69f9db3f7d97a8c server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:08:27.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.753+0000 7fd75aa00640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-31T15:08:27.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.753+0000 7fd75aa00640 1 -- 172.21.15.16:0/2273466780 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd76001a070 con 0x55cf51fd8210 2024-03-31T15:08:27.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.753+0000 7fd77fa25000 1 -- 172.21.15.16:0/2273466780 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd768010b00 con 0x55cf51fd8210 2024-03-31T15:08:27.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.753+0000 7fd75aa00640 1 -- 172.21.15.16:0/2273466780 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(70..70 src has 1..70) ==== 5882+0+0 (secure 0 0 0) 0x7fd760005ab0 con 0x55cf51fd8210 2024-03-31T15:08:27.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.753+0000 7fd75aa00640 1 -- 172.21.15.16:0/2273466780 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=51) ==== 24+0+0 (secure 0 0 0) 0x7fd76002d050 con 0x55cf51fd8210 2024-03-31T15:08:27.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.753+0000 7fd77fa25000 10 client.7104 fetch_fsmap learned FSMap version 51 2024-03-31T15:08:27.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.753+0000 7fd77fa25000 1 -- 172.21.15.16:0/2273466780 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=51,monmap=2+}) -- 0x55cf51d7acb0 con 0x55cf51fd8210 2024-03-31T15:08:27.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.753+0000 7fd75aa00640 1 -- 172.21.15.16:0/2273466780 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd768010b00 con 0x55cf51fd8210 2024-03-31T15:08:27.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.753+0000 7fd75aa00640 1 -- 172.21.15.16:0/2273466780 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 51) ==== 3494+0+0 (secure 0 0 0) 0x7fd760030020 con 0x55cf51fd8210 2024-03-31T15:08:27.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.753+0000 7fd77fa25000 10 client.7104 fetch_fsmap finished waiting for FSMap version 51 2024-03-31T15:08:27.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.753+0000 7fd77fa25000 10 client.7104 resolve_mds: resolved 5:0 to role '5:0' aka daemon mds.a 2024-03-31T15:08:27.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.753+0000 7fd77fa25000 20 client.7104 populate_metadata read hostname 'smithi016' 2024-03-31T15:08:27.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.753+0000 7fd77fa25000 1 --2- 172.21.15.16:0/2273466780 >> [v2:172.21.15.16:6834/2645389320,v1:172.21.15.16:6835/2645389320] conn(0x55cf51fd9cb0 0x55cf51fda0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:27.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.753+0000 7fd77fa25000 4 client.7104 mds_command: new command op to 6829 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-31T15:08:27.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.753+0000 7fd77fa25000 1 -- 172.21.15.16:0/2273466780 --> [v2:172.21.15.16:6834/2645389320,v1:172.21.15.16:6835/2645389320] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55cf51d78df0 con 0x55cf51fd9cb0 2024-03-31T15:08:27.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.757+0000 7fd779c00640 1 --2- 172.21.15.16:0/2273466780 >> [v2:172.21.15.16:6834/2645389320,v1:172.21.15.16:6835/2645389320] conn(0x55cf51fd9cb0 0x55cf51fda0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:27.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.757+0000 7fd779c00640 1 --2- 172.21.15.16:0/2273466780 >> [v2:172.21.15.16:6834/2645389320,v1:172.21.15.16:6835/2645389320] conn(0x55cf51fd9cb0 0x55cf51fda0c0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f96476167f7b77f4 server_cookie=1d5968b73c6e8b4d in_seq=0 out_seq=0 2024-03-31T15:08:27.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.757+0000 7fd75aa00640 10 client.7104 ms_handle_connect on v2:172.21.15.16:6834/2645389320 2024-03-31T15:08:27.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.757+0000 7fd75aa00640 1 -- 172.21.15.16:0/2273466780 <== mds.0 v2:172.21.15.16:6834/2645389320 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55cf51d78df0 con 0x55cf51fd9cb0 2024-03-31T15:08:27.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.757+0000 7fd75aa00640 10 client.7104 handle_command_reply: tid=0 2024-03-31T15:08:27.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.757+0000 7fd77fa25000 1 client.7104 shutdown 2024-03-31T15:08:27.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.757+0000 7fd77fa25000 1 -- 172.21.15.16:0/2273466780 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cf51fd8210 msgr2=0x55cf51fbb900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:27.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.757+0000 7fd77fa25000 1 --2- 172.21.15.16:0/2273466780 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cf51fd8210 0x55cf51fbb900 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7fd76000ef30 tx=0x7fd760000bd0 comp rx=0 tx=0).stop 2024-03-31T15:08:27.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.757+0000 7fd77fa25000 1 -- 172.21.15.16:0/2273466780 shutdown_connections 2024-03-31T15:08:27.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.757+0000 7fd77fa25000 1 --2- 172.21.15.16:0/2273466780 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cf51fd8b30 0x55cf51fd8f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:27.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.757+0000 7fd77fa25000 1 --2- 172.21.15.16:0/2273466780 >> [v2:172.21.15.16:6834/2645389320,v1:172.21.15.16:6835/2645389320] conn(0x55cf51fd9cb0 0x55cf51fda0c0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:27.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.757+0000 7fd77fa25000 1 --2- 172.21.15.16:0/2273466780 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cf51fd8210 0x55cf51fbb900 unknown :-1 s=CLOSED pgs=927 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:27.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.757+0000 7fd77fa25000 1 --2- 172.21.15.16:0/2273466780 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55cf51f8ec80 0x55cf51fbb3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:27.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.757+0000 7fd77fa25000 1 -- 172.21.15.16:0/2273466780 >> 172.21.15.16:0/2273466780 conn(0x55cf51f8ac30 msgr2=0x55cf51f8d8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:08:27.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.757+0000 7fd75aa00640 0 client.7104 ms_handle_reset on v2:172.21.15.16:6834/2645389320 2024-03-31T15:08:27.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.757+0000 7fd77fa25000 1 -- 172.21.15.16:0/2273466780 shutdown_connections 2024-03-31T15:08:27.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.757+0000 7fd77fa25000 1 -- 172.21.15.16:0/2273466780 wait complete. 2024-03-31T15:08:27.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.757+0000 7fd77fa25000 20 client.7104 trim_cache size 0 max 16384 2024-03-31T15:08:27.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.773+0000 7fd77fa25000 1 Processor -- start 2024-03-31T15:08:27.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.773+0000 7fd77fa25000 1 -- start start 2024-03-31T15:08:27.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.773+0000 7fd77fa25000 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cf51f8ec80 0x55cf51fdf4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:27.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.773+0000 7fd77fa25000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cf51fd8210 0x55cf51f8d730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:27.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.777+0000 7fd77fa25000 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55cf51fd8b30 0x55cf51f8dc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:27.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.777+0000 7fd77fa25000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fd774203940 con 0x55cf51fd8210 2024-03-31T15:08:27.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.777+0000 7fd77fa25000 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fd7742029a0 con 0x55cf51f8ec80 2024-03-31T15:08:27.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.777+0000 7fd77fa25000 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fd77406cf90 con 0x55cf51fd8b30 2024-03-31T15:08:27.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.777+0000 7fd779200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cf51fd8210 0x55cf51f8d730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:27.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.777+0000 7fd779200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cf51fd8210 0x55cf51f8d730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34600/0 (socket says 172.21.15.16:34600) 2024-03-31T15:08:27.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.777+0000 7fd779200640 1 -- 172.21.15.16:0/3009199092 learned_addr learned my addr 172.21.15.16:0/3009199092 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:08:27.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.777+0000 7fd779c00640 1 --2- 172.21.15.16:0/3009199092 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cf51f8ec80 0x55cf51fdf4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:27.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.777+0000 7fd77a600640 1 --2- 172.21.15.16:0/3009199092 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55cf51fd8b30 0x55cf51f8dc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:27.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.777+0000 7fd779200640 1 -- 172.21.15.16:0/3009199092 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55cf51fd8b30 msgr2=0x55cf51f8dc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:08:27.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.777+0000 7fd779200640 1 --2- 172.21.15.16:0/3009199092 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55cf51fd8b30 0x55cf51f8dc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:27.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.777+0000 7fd779200640 1 -- 172.21.15.16:0/3009199092 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cf51f8ec80 msgr2=0x55cf51fdf4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:08:27.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.777+0000 7fd779200640 1 --2- 172.21.15.16:0/3009199092 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cf51f8ec80 0x55cf51fdf4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:27.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.777+0000 7fd779200640 1 -- 172.21.15.16:0/3009199092 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cf51e5e2e0 con 0x55cf51fd8210 2024-03-31T15:08:27.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.777+0000 7fd779200640 1 --2- 172.21.15.16:0/3009199092 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cf51fd8210 0x55cf51f8d730 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x7fd760014fd0 tx=0x7fd760020000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=134e99e04d76b1da server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:08:27.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.777+0000 7fd765600640 1 -- 172.21.15.16:0/3009199092 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd760005ab0 con 0x55cf51fd8210 2024-03-31T15:08:27.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.777+0000 7fd765600640 1 -- 172.21.15.16:0/3009199092 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd760036070 con 0x55cf51fd8210 2024-03-31T15:08:27.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.777+0000 7fd765600640 1 -- 172.21.15.16:0/3009199092 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7600071e0 con 0x55cf51fd8210 2024-03-31T15:08:27.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.777+0000 7fd77fa25000 1 -- 172.21.15.16:0/3009199092 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cf51fd8210 msgr2=0x55cf51f8d730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:27.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.777+0000 7fd77fa25000 1 --2- 172.21.15.16:0/3009199092 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cf51fd8210 0x55cf51f8d730 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x7fd760014fd0 tx=0x7fd760020000 comp rx=0 tx=0).stop 2024-03-31T15:08:27.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.777+0000 7fd77fa25000 1 -- 172.21.15.16:0/3009199092 shutdown_connections 2024-03-31T15:08:27.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.777+0000 7fd77fa25000 1 --2- 172.21.15.16:0/3009199092 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55cf51fd8b30 0x55cf51f8dc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:27.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.777+0000 7fd77fa25000 1 --2- 172.21.15.16:0/3009199092 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cf51fd8210 0x55cf51f8d730 unknown :-1 s=CLOSED pgs=928 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:27.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.777+0000 7fd77fa25000 1 --2- 172.21.15.16:0/3009199092 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cf51f8ec80 0x55cf51fdf4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:27.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.777+0000 7fd77fa25000 1 -- 172.21.15.16:0/3009199092 >> 172.21.15.16:0/3009199092 conn(0x55cf51f8ac30 msgr2=0x55cf51f8d020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:08:27.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.777+0000 7fd77fa25000 1 -- 172.21.15.16:0/3009199092 shutdown_connections 2024-03-31T15:08:27.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.777+0000 7fd77fa25000 1 -- 172.21.15.16:0/3009199092 wait complete. 2024-03-31T15:08:27.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.777+0000 7fd77fa25000 1 -- start start 2024-03-31T15:08:27.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.777+0000 7fd77fa25000 1 Processor -- start 2024-03-31T15:08:27.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.777+0000 7fd77fa25000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cf51f8ec80 0x55cf51fd7980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:27.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.777+0000 7fd77fa25000 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55cf51fd8210 0x55cf51fdd0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:27.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.777+0000 7fd77fa25000 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cf51fd8b30 0x55cf51fdd630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:27.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.777+0000 7fd77fa25000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55cf51de2050 con 0x55cf51f8ec80 2024-03-31T15:08:27.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.777+0000 7fd77fa25000 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fd7742025e0 con 0x55cf51fd8b30 2024-03-31T15:08:27.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.777+0000 7fd77fa25000 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x55cf51f86080 con 0x55cf51fd8210 2024-03-31T15:08:27.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.777+0000 7fd77fa25000 10 client.0 _ll_register_callbacks cb 0x7fd760001410 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T15:08:27.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.777+0000 7fd779c00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cf51f8ec80 0x55cf51fd7980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:27.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.777+0000 7fd779c00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cf51f8ec80 0x55cf51fd7980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34606/0 (socket says 172.21.15.16:34606) 2024-03-31T15:08:27.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.777+0000 7fd779c00640 1 -- 172.21.15.16:0/269684448 learned_addr learned my addr 172.21.15.16:0/269684448 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:08:27.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.781+0000 7fd779c00640 1 -- 172.21.15.16:0/269684448 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55cf51fd8210 msgr2=0x55cf51fdd0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:08:27.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.781+0000 7fd77a600640 1 --2- 172.21.15.16:0/269684448 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cf51fd8b30 0x55cf51fdd630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:27.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.781+0000 7fd779200640 1 --2- 172.21.15.16:0/269684448 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55cf51fd8210 0x55cf51fdd0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:27.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.781+0000 7fd779c00640 1 --2- 172.21.15.16:0/269684448 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55cf51fd8210 0x55cf51fdd0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:27.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.781+0000 7fd779c00640 1 -- 172.21.15.16:0/269684448 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cf51fd8b30 msgr2=0x55cf51fdd630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:08:27.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.781+0000 7fd779c00640 1 --2- 172.21.15.16:0/269684448 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cf51fd8b30 0x55cf51fdd630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:27.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.781+0000 7fd779c00640 1 -- 172.21.15.16:0/269684448 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cf51d8dba0 con 0x55cf51f8ec80 2024-03-31T15:08:27.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.781+0000 7fd77a600640 1 --2- 172.21.15.16:0/269684448 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cf51fd8b30 0x55cf51fdd630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:08:27.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.781+0000 7fd779200640 1 --2- 172.21.15.16:0/269684448 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55cf51fd8210 0x55cf51fdd0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:08:27.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.781+0000 7fd779c00640 1 --2- 172.21.15.16:0/269684448 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cf51f8ec80 0x55cf51fd7980 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7fd768000c10 tx=0x7fd768004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e4f75d76c45864c0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:08:27.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.781+0000 7fd75aa00640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-31T15:08:27.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.781+0000 7fd75aa00640 1 -- 172.21.15.16:0/269684448 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd768018040 con 0x55cf51f8ec80 2024-03-31T15:08:27.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.781+0000 7fd75aa00640 1 -- 172.21.15.16:0/269684448 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(70..70 src has 1..70) ==== 5882+0+0 (secure 0 0 0) 0x7fd768029420 con 0x55cf51f8ec80 2024-03-31T15:08:27.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.781+0000 7fd77fa25000 1 -- 172.21.15.16:0/269684448 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd760036070 con 0x55cf51f8ec80 2024-03-31T15:08:27.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.781+0000 7fd75aa00640 1 -- 172.21.15.16:0/269684448 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=51) ==== 24+0+0 (secure 0 0 0) 0x7fd768037050 con 0x55cf51f8ec80 2024-03-31T15:08:27.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.781+0000 7fd77fa25000 10 client.7110 fetch_fsmap learned FSMap version 51 2024-03-31T15:08:27.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.781+0000 7fd77fa25000 1 -- 172.21.15.16:0/269684448 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=51,monmap=2+}) -- 0x55cf51d71890 con 0x55cf51f8ec80 2024-03-31T15:08:27.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.781+0000 7fd75aa00640 1 -- 172.21.15.16:0/269684448 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd760036070 con 0x55cf51f8ec80 2024-03-31T15:08:27.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.781+0000 7fd75aa00640 1 -- 172.21.15.16:0/269684448 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 51) ==== 3494+0+0 (secure 0 0 0) 0x7fd76803a020 con 0x55cf51f8ec80 2024-03-31T15:08:27.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.781+0000 7fd77fa25000 10 client.7110 fetch_fsmap finished waiting for FSMap version 51 2024-03-31T15:08:27.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.781+0000 7fd77fa25000 10 client.7110 resolve_mds: resolved 5:0 to role '5:0' aka daemon mds.a 2024-03-31T15:08:27.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.781+0000 7fd77fa25000 20 client.7110 populate_metadata read hostname 'smithi016' 2024-03-31T15:08:27.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.781+0000 7fd77fa25000 1 --2- 172.21.15.16:0/269684448 >> [v2:172.21.15.16:6834/2645389320,v1:172.21.15.16:6835/2645389320] conn(0x55cf51fddb70 0x55cf52048510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:27.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.781+0000 7fd77fa25000 4 client.7110 mds_command: new command op to 6829 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/subdir", "scrubops": ["recursive"]}] 2024-03-31T15:08:27.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.781+0000 7fd77fa25000 1 -- 172.21.15.16:0/269684448 --> [v2:172.21.15.16:6834/2645389320,v1:172.21.15.16:6835/2645389320] -- command(tid 0: {"prefix": "scrub start", "path": "/subdir", "scrubops": ["recursive"]}) -- 0x55cf51deaf60 con 0x55cf51fddb70 2024-03-31T15:08:27.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.781+0000 7fd779200640 1 --2- 172.21.15.16:0/269684448 >> [v2:172.21.15.16:6834/2645389320,v1:172.21.15.16:6835/2645389320] conn(0x55cf51fddb70 0x55cf52048510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:27.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.781+0000 7fd779200640 1 --2- 172.21.15.16:0/269684448 >> [v2:172.21.15.16:6834/2645389320,v1:172.21.15.16:6835/2645389320] conn(0x55cf51fddb70 0x55cf52048510 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=db71438d645260fc server_cookie=e4da8ff07ed969e7 in_seq=0 out_seq=0 2024-03-31T15:08:27.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.781+0000 7fd75aa00640 10 client.7110 ms_handle_connect on v2:172.21.15.16:6834/2645389320 2024-03-31T15:08:27.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.781+0000 7fd75aa00640 1 -- 172.21.15.16:0/269684448 <== mds.0 v2:172.21.15.16:6834/2645389320 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55cf51deaf60 con 0x55cf51fddb70 2024-03-31T15:08:27.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.781+0000 7fd75aa00640 10 client.7110 handle_command_reply: tid=0 2024-03-31T15:08:27.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.781+0000 7fd77fa25000 1 client.7110 shutdown 2024-03-31T15:08:27.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.781+0000 7fd77fa25000 1 -- 172.21.15.16:0/269684448 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cf51f8ec80 msgr2=0x55cf51fd7980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:27.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.781+0000 7fd77fa25000 1 --2- 172.21.15.16:0/269684448 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cf51f8ec80 0x55cf51fd7980 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7fd768000c10 tx=0x7fd768004740 comp rx=0 tx=0).stop 2024-03-31T15:08:27.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.781+0000 7fd77fa25000 1 -- 172.21.15.16:0/269684448 shutdown_connections 2024-03-31T15:08:27.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.781+0000 7fd77fa25000 1 --2- 172.21.15.16:0/269684448 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cf51fd8b30 0x55cf51fdd630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:27.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.781+0000 7fd77fa25000 1 --2- 172.21.15.16:0/269684448 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55cf51fd8210 0x55cf51fdd0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:27.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.781+0000 7fd77fa25000 1 --2- 172.21.15.16:0/269684448 >> [v2:172.21.15.16:6834/2645389320,v1:172.21.15.16:6835/2645389320] conn(0x55cf51fddb70 0x55cf52048510 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:27.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.781+0000 7fd77fa25000 1 --2- 172.21.15.16:0/269684448 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cf51f8ec80 0x55cf51fd7980 unknown :-1 s=CLOSED pgs=929 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:27.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.781+0000 7fd77fa25000 1 -- 172.21.15.16:0/269684448 >> 172.21.15.16:0/269684448 conn(0x55cf51f8ac30 msgr2=0x55cf51f8d8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:08:27.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.781+0000 7fd75aa00640 0 client.7110 ms_handle_reset on v2:172.21.15.16:6834/2645389320 2024-03-31T15:08:27.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.781+0000 7fd77fa25000 1 -- 172.21.15.16:0/269684448 shutdown_connections 2024-03-31T15:08:27.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.781+0000 7fd77fa25000 1 -- 172.21.15.16:0/269684448 wait complete. 2024-03-31T15:08:27.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.781+0000 7fd77fa25000 20 client.7110 trim_cache size 0 max 16384 2024-03-31T15:08:27.783 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-03-31T15:08:27.783 INFO:teuthology.orchestra.run.smithi016.stdout: "return_code": 0, 2024-03-31T15:08:27.783 INFO:teuthology.orchestra.run.smithi016.stdout: "scrub_tag": "4bab04b3-dab7-4217-af07-ee67fc7d3176", 2024-03-31T15:08:27.783 INFO:teuthology.orchestra.run.smithi016.stdout: "mode": "asynchronous" 2024-03-31T15:08:27.783 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-03-31T15:08:27.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.785+0000 7fd77c600640 1 -- 172.21.15.16:0/524429024 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fd7540683b0 msgr2=0x7fd75406a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:27.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.785+0000 7fd77c600640 1 --2- 172.21.15.16:0/524429024 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fd7540683b0 0x7fd75406a870 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7fd7740d1b00 tx=0x7fd760009040 comp rx=0 tx=0).stop 2024-03-31T15:08:27.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.785+0000 7fd77c600640 1 -- 172.21.15.16:0/524429024 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd7740cc7b0 msgr2=0x7fd7740cb4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:27.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.785+0000 7fd77c600640 1 --2- 172.21.15.16:0/524429024 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd7740cc7b0 0x7fd7740cb4b0 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7fd76c0b0040 tx=0x7fd7740d27d0 comp rx=0 tx=0).stop 2024-03-31T15:08:27.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.785+0000 7fd77c600640 1 -- 172.21.15.16:0/524429024 shutdown_connections 2024-03-31T15:08:27.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.785+0000 7fd77c600640 1 --2- 172.21.15.16:0/524429024 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fd7540683b0 0x7fd75406a870 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:27.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.785+0000 7fd77c600640 1 --2- 172.21.15.16:0/524429024 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd7740cc7b0 0x7fd7740cb4b0 unknown :-1 s=CLOSED pgs=925 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:27.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.785+0000 7fd77c600640 1 --2- 172.21.15.16:0/524429024 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd7740b7900 0x7fd774158960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:27.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.785+0000 7fd77c600640 1 --2- 172.21.15.16:0/524429024 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fd7740b44d0 0x7fd7741404b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:27.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.785+0000 7fd77c600640 1 -- 172.21.15.16:0/524429024 >> 172.21.15.16:0/524429024 conn(0x7fd77406b7f0 msgr2=0x7fd77415f340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:08:27.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.785+0000 7fd77c600640 1 -- 172.21.15.16:0/524429024 shutdown_connections 2024-03-31T15:08:27.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.785+0000 7fd77c600640 1 -- 172.21.15.16:0/524429024 wait complete. 2024-03-31T15:08:27.806 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:08:27.806 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.5:0 scrub status 2024-03-31T15:08:27.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.953+0000 7f841b200640 1 Processor -- start 2024-03-31T15:08:27.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.953+0000 7f841b200640 1 -- start start 2024-03-31T15:08:27.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.953+0000 7f841b200640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f84141411b0 0x7f841406f920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:27.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.953+0000 7f841b200640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f8414141590 0x7f841406fe60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:27.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.953+0000 7f841b200640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8414144980 con 0x7f8414143c30 2024-03-31T15:08:27.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.953+0000 7f841b200640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f841415f910 con 0x7f84141411b0 2024-03-31T15:08:27.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.953+0000 7f841b200640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f8414162910 con 0x7f8414141590 2024-03-31T15:08:27.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.953+0000 7f8419200640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f8414143c30 0x7f84141440a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:34972/0 (socket says 172.21.15.16:34972) 2024-03-31T15:08:27.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.953+0000 7f8413e00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f84141411b0 0x7f841406f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:27.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.953+0000 7f8419200640 1 -- 172.21.15.16:0/2708299351 learned_addr learned my addr 172.21.15.16:0/2708299351 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:08:27.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.953+0000 7f8413400640 1 --2- 172.21.15.16:0/2708299351 >> v2:172.21.15.89:3300/0 conn(0x7f8414141590 0x7f841406fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:27.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.953+0000 7f8413e00640 1 -- 172.21.15.16:0/2708299351 >> v2:172.21.15.89:3300/0 conn(0x7f8414141590 msgr2=0x7f841406fe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:08:27.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.953+0000 7f8413e00640 1 --2- 172.21.15.16:0/2708299351 >> v2:172.21.15.89:3300/0 conn(0x7f8414141590 0x7f841406fe60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:27.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.953+0000 7f8413e00640 1 -- 172.21.15.16:0/2708299351 >> v1:172.21.15.16:6789/0 conn(0x7f8414143c30 legacy=0x7f84141440a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:27.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.953+0000 7f8413e00640 1 -- 172.21.15.16:0/2708299351 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8414161fe0 con 0x7f84141411b0 2024-03-31T15:08:27.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.953+0000 7f8412a00640 1 -- 172.21.15.16:0/2708299351 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2534780601 0 0) 0x7f8414144980 con 0x7f8414143c30 2024-03-31T15:08:27.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.957+0000 7f8413e00640 1 --2- 172.21.15.16:0/2708299351 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f84141411b0 0x7f841406f920 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7f8414158fe0 tx=0x7f840401c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a0d09e44bd4d7f72 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:08:27.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.957+0000 7f8412a00640 1 -- 172.21.15.16:0/2708299351 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84040b0070 con 0x7f84141411b0 2024-03-31T15:08:27.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.957+0000 7f8412a00640 1 -- 172.21.15.16:0/2708299351 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f84040ab600 con 0x7f84141411b0 2024-03-31T15:08:27.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.957+0000 7f8412a00640 1 -- 172.21.15.16:0/2708299351 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84040ab9d0 con 0x7f84141411b0 2024-03-31T15:08:27.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.957+0000 7f841b200640 1 -- 172.21.15.16:0/2708299351 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f84141411b0 msgr2=0x7f841406f920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:27.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.957+0000 7f841b200640 1 --2- 172.21.15.16:0/2708299351 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f84141411b0 0x7f841406f920 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7f8414158fe0 tx=0x7f840401c480 comp rx=0 tx=0).stop 2024-03-31T15:08:27.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.957+0000 7f841b200640 1 -- 172.21.15.16:0/2708299351 shutdown_connections 2024-03-31T15:08:27.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.957+0000 7f841b200640 1 --2- 172.21.15.16:0/2708299351 >> v2:172.21.15.89:3300/0 conn(0x7f8414141590 0x7f841406fe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:27.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.957+0000 7f841b200640 1 --2- 172.21.15.16:0/2708299351 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f84141411b0 0x7f841406f920 unknown :-1 s=CLOSED pgs=931 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:27.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.957+0000 7f841b200640 1 -- 172.21.15.16:0/2708299351 >> 172.21.15.16:0/2708299351 conn(0x7f8414067930 msgr2=0x7f84140b3ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:08:27.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.957+0000 7f841b200640 1 -- 172.21.15.16:0/2708299351 shutdown_connections 2024-03-31T15:08:27.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.957+0000 7f841b200640 1 -- 172.21.15.16:0/2708299351 wait complete. 2024-03-31T15:08:27.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.957+0000 7f841b200640 1 Processor -- start 2024-03-31T15:08:27.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.961+0000 7f841b200640 1 -- start start 2024-03-31T15:08:27.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.961+0000 7f841b200640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f84141411b0 0x7f8414156330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:27.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.961+0000 7f841b200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8414141590 0x7f8414154aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:27.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.961+0000 7f841b200640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f8414143c30 0x7f841413eb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:27.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.961+0000 7f841b200640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f841415f290 con 0x7f8414141590 2024-03-31T15:08:27.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.961+0000 7f841b200640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f841415f110 con 0x7f84141411b0 2024-03-31T15:08:27.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.961+0000 7f841b200640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f841415f410 con 0x7f8414143c30 2024-03-31T15:08:27.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.961+0000 7f8413400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8414141590 0x7f8414154aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:27.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.961+0000 7f8413400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8414141590 0x7f8414154aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34624/0 (socket says 172.21.15.16:34624) 2024-03-31T15:08:27.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.961+0000 7f8413400640 1 -- 172.21.15.16:0/3991526783 learned_addr learned my addr 172.21.15.16:0/3991526783 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:08:27.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.961+0000 7f8419200640 1 --2- 172.21.15.16:0/3991526783 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f8414143c30 0x7f841413eb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:27.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.961+0000 7f8413e00640 1 --2- 172.21.15.16:0/3991526783 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f84141411b0 0x7f8414156330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:27.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.961+0000 7f8413400640 1 -- 172.21.15.16:0/3991526783 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f8414143c30 msgr2=0x7f841413eb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:27.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.961+0000 7f8413400640 1 --2- 172.21.15.16:0/3991526783 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f8414143c30 0x7f841413eb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:27.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.961+0000 7f8413400640 1 -- 172.21.15.16:0/3991526783 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f84141411b0 msgr2=0x7f8414156330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:27.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.961+0000 7f8413400640 1 --2- 172.21.15.16:0/3991526783 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f84141411b0 0x7f8414156330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:27.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.961+0000 7f8413400640 1 -- 172.21.15.16:0/3991526783 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f841413f0d0 con 0x7f8414141590 2024-03-31T15:08:27.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.961+0000 7f8413400640 1 --2- 172.21.15.16:0/3991526783 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8414141590 0x7f8414154aa0 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7f840000e900 tx=0x7f840000edc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:08:27.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.961+0000 7f840be00640 1 -- 172.21.15.16:0/3991526783 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8400017070 con 0x7f8414141590 2024-03-31T15:08:27.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.961+0000 7f840be00640 1 -- 172.21.15.16:0/3991526783 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f840000c9a0 con 0x7f8414141590 2024-03-31T15:08:27.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.961+0000 7f840be00640 1 -- 172.21.15.16:0/3991526783 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f840000cd70 con 0x7f8414141590 2024-03-31T15:08:27.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.961+0000 7f841b200640 1 -- 172.21.15.16:0/3991526783 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8414156a30 con 0x7f8414141590 2024-03-31T15:08:27.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.961+0000 7f841b200640 1 -- 172.21.15.16:0/3991526783 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8414156ff0 con 0x7f8414141590 2024-03-31T15:08:27.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.965+0000 7f840be00640 1 -- 172.21.15.16:0/3991526783 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f840001a070 con 0x7f8414141590 2024-03-31T15:08:27.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.965+0000 7f840be00640 1 --2- 172.21.15.16:0/3991526783 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f83f4068380 0x7f83f406a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:27.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.965+0000 7f840be00640 1 -- 172.21.15.16:0/3991526783 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(70..70 src has 1..70) ==== 5882+0+0 (secure 0 0 0) 0x7f8400086a60 con 0x7f8414141590 2024-03-31T15:08:27.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.965+0000 7f841e729000 1 Processor -- start 2024-03-31T15:08:27.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.965+0000 7f8413e00640 1 --2- 172.21.15.16:0/3991526783 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f83f4068380 0x7f83f406a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:27.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.965+0000 7f841e729000 1 -- start start 2024-03-31T15:08:27.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.965+0000 7f8413e00640 1 --2- 172.21.15.16:0/3991526783 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f83f4068380 0x7f83f406a840 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f84040ad000 tx=0x7f84040a6000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:08:27.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.965+0000 7f841e729000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55acae3c52d0 0x55acae3c56b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:27.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.965+0000 7f841e729000 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55acae3c5bf0 0x55acae378470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:27.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.965+0000 7f841e729000 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55acae3789b0 0x55acae378d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:27.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.965+0000 7f841e729000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55acae261af0 con 0x55acae3c52d0 2024-03-31T15:08:27.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.965+0000 7f841e729000 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55acae1cab00 con 0x55acae3c5bf0 2024-03-31T15:08:27.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.965+0000 7f841e729000 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x55acae167940 con 0x55acae3789b0 2024-03-31T15:08:27.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.965+0000 7f8413e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55acae3c52d0 0x55acae3c56b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:27.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.965+0000 7f8413e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55acae3c52d0 0x55acae3c56b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34628/0 (socket says 172.21.15.16:34628) 2024-03-31T15:08:27.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.965+0000 7f8413e00640 1 -- 172.21.15.16:0/1267664629 learned_addr learned my addr 172.21.15.16:0/1267664629 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:08:27.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.965+0000 7f8413400640 1 --2- 172.21.15.16:0/1267664629 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55acae3c5bf0 0x55acae378470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:27.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.965+0000 7f8419200640 1 --2- 172.21.15.16:0/1267664629 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55acae3789b0 0x55acae378d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:27.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.965+0000 7f8413e00640 1 -- 172.21.15.16:0/1267664629 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55acae3789b0 msgr2=0x55acae378d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:08:27.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.965+0000 7f8413e00640 1 --2- 172.21.15.16:0/1267664629 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55acae3789b0 0x55acae378d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:27.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.965+0000 7f8413e00640 1 -- 172.21.15.16:0/1267664629 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55acae3c5bf0 msgr2=0x55acae378470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:08:27.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.965+0000 7f8413e00640 1 --2- 172.21.15.16:0/1267664629 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55acae3c5bf0 0x55acae378470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:27.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.965+0000 7f8413e00640 1 -- 172.21.15.16:0/1267664629 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55acae150380 con 0x55acae3c52d0 2024-03-31T15:08:27.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.969+0000 7f8413e00640 1 --2- 172.21.15.16:0/1267664629 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55acae3c52d0 0x55acae3c56b0 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7f84040ae040 tx=0x7f8404004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d0aea4e3b0df7ace server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:08:27.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.969+0000 7f8408c00640 1 -- 172.21.15.16:0/1267664629 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84040b0040 con 0x55acae3c52d0 2024-03-31T15:08:27.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.969+0000 7f8408c00640 1 -- 172.21.15.16:0/1267664629 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f84040c6070 con 0x55acae3c52d0 2024-03-31T15:08:27.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.969+0000 7f8408c00640 1 -- 172.21.15.16:0/1267664629 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84040b6660 con 0x55acae3c52d0 2024-03-31T15:08:27.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.969+0000 7f841e729000 1 -- 172.21.15.16:0/1267664629 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55acae3c52d0 msgr2=0x55acae3c56b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:27.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.969+0000 7f841e729000 1 --2- 172.21.15.16:0/1267664629 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55acae3c52d0 0x55acae3c56b0 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7f84040ae040 tx=0x7f8404004740 comp rx=0 tx=0).stop 2024-03-31T15:08:27.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.969+0000 7f841e729000 1 -- 172.21.15.16:0/1267664629 shutdown_connections 2024-03-31T15:08:27.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.969+0000 7f841e729000 1 --2- 172.21.15.16:0/1267664629 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55acae3789b0 0x55acae378d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:27.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.969+0000 7f841e729000 1 --2- 172.21.15.16:0/1267664629 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55acae3c5bf0 0x55acae378470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:27.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.969+0000 7f841e729000 1 --2- 172.21.15.16:0/1267664629 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55acae3c52d0 0x55acae3c56b0 unknown :-1 s=CLOSED pgs=933 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:27.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.969+0000 7f841e729000 1 -- 172.21.15.16:0/1267664629 >> 172.21.15.16:0/1267664629 conn(0x55acae3a6070 msgr2=0x55acae3a8460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:08:27.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.969+0000 7f841e729000 1 -- 172.21.15.16:0/1267664629 shutdown_connections 2024-03-31T15:08:27.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.969+0000 7f841e729000 1 -- 172.21.15.16:0/1267664629 wait complete. 2024-03-31T15:08:27.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.969+0000 7f841e729000 1 -- start start 2024-03-31T15:08:27.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.969+0000 7f841e729000 1 Processor -- start 2024-03-31T15:08:27.969 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.969+0000 7f841e729000 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55acae3789b0 0x55acae37d930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:27.969 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.969+0000 7f841e729000 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55acae3c52d0 0x55acae37de70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:27.969 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.969+0000 7f841e729000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55acae3c5bf0 0x55acae37e3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:27.969 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.969+0000 7f841e729000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55acae1881e0 con 0x55acae3c5bf0 2024-03-31T15:08:27.969 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.969+0000 7f841e729000 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55acae1c6ab0 con 0x55acae3c52d0 2024-03-31T15:08:27.969 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.969+0000 7f841e729000 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x55acae299420 con 0x55acae3789b0 2024-03-31T15:08:27.969 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.969+0000 7f841e729000 10 client.0 _ll_register_callbacks cb 0x55acae366370 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T15:08:27.969 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.969+0000 7f8419200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55acae3c5bf0 0x55acae37e3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:27.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.969+0000 7f8419200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55acae3c5bf0 0x55acae37e3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34644/0 (socket says 172.21.15.16:34644) 2024-03-31T15:08:27.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.969+0000 7f8419200640 1 -- 172.21.15.16:0/2499620066 learned_addr learned my addr 172.21.15.16:0/2499620066 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:08:27.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.969+0000 7f8413400640 1 --2- 172.21.15.16:0/2499620066 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55acae3c52d0 0x55acae37de70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:27.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.969+0000 7f8413e00640 1 --2- 172.21.15.16:0/2499620066 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55acae3789b0 0x55acae37d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:27.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.969+0000 7f8419200640 1 -- 172.21.15.16:0/2499620066 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55acae3789b0 msgr2=0x55acae37d930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:08:27.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.969+0000 7f8419200640 1 --2- 172.21.15.16:0/2499620066 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55acae3789b0 0x55acae37d930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:27.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.969+0000 7f8419200640 1 -- 172.21.15.16:0/2499620066 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55acae3c52d0 msgr2=0x55acae37de70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:08:27.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.969+0000 7f8419200640 1 --2- 172.21.15.16:0/2499620066 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55acae3c52d0 0x55acae37de70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:27.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.969+0000 7f8419200640 1 -- 172.21.15.16:0/2499620066 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55acae186b70 con 0x55acae3c5bf0 2024-03-31T15:08:27.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.969+0000 7f8419200640 1 --2- 172.21.15.16:0/2499620066 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55acae3c5bf0 0x55acae37e3b0 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7f840c007b40 tx=0x7f840c007b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ecd724bd8e015c5a server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:08:27.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.969+0000 7f83fd600640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-31T15:08:27.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.973+0000 7f83fd600640 1 -- 172.21.15.16:0/2499620066 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f840c0020c0 con 0x55acae3c5bf0 2024-03-31T15:08:27.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.973+0000 7f83fd600640 1 -- 172.21.15.16:0/2499620066 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(70..70 src has 1..70) ==== 5882+0+0 (secure 0 0 0) 0x7f840c013570 con 0x55acae3c5bf0 2024-03-31T15:08:27.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.973+0000 7f841e729000 1 -- 172.21.15.16:0/2499620066 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f84040c6070 con 0x55acae3c5bf0 2024-03-31T15:08:27.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.973+0000 7f83fd600640 1 -- 172.21.15.16:0/2499620066 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=51) ==== 24+0+0 (secure 0 0 0) 0x7f840c021050 con 0x55acae3c5bf0 2024-03-31T15:08:27.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.973+0000 7f841e729000 10 client.7125 fetch_fsmap learned FSMap version 51 2024-03-31T15:08:27.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.973+0000 7f841e729000 1 -- 172.21.15.16:0/2499620066 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=51,monmap=2+}) -- 0x55acae167b90 con 0x55acae3c5bf0 2024-03-31T15:08:27.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.973+0000 7f83fd600640 1 -- 172.21.15.16:0/2499620066 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f84040c6070 con 0x55acae3c5bf0 2024-03-31T15:08:27.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.973+0000 7f83fd600640 1 -- 172.21.15.16:0/2499620066 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 51) ==== 3494+0+0 (secure 0 0 0) 0x7f840c024020 con 0x55acae3c5bf0 2024-03-31T15:08:27.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.973+0000 7f841e729000 10 client.7125 fetch_fsmap finished waiting for FSMap version 51 2024-03-31T15:08:27.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.973+0000 7f841e729000 10 client.7125 resolve_mds: resolved 5:0 to role '5:0' aka daemon mds.a 2024-03-31T15:08:27.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.973+0000 7f841e729000 20 client.7125 populate_metadata read hostname 'smithi016' 2024-03-31T15:08:27.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.973+0000 7f841e729000 1 --2- 172.21.15.16:0/2499620066 >> [v2:172.21.15.16:6834/2645389320,v1:172.21.15.16:6835/2645389320] conn(0x55acae37ec50 0x55acae3d6a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:27.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.973+0000 7f841e729000 4 client.7125 mds_command: new command op to 6829 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-31T15:08:27.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.973+0000 7f841e729000 1 -- 172.21.15.16:0/2499620066 --> [v2:172.21.15.16:6834/2645389320,v1:172.21.15.16:6835/2645389320] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55acae165dd0 con 0x55acae37ec50 2024-03-31T15:08:27.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.973+0000 7f8413e00640 1 --2- 172.21.15.16:0/2499620066 >> [v2:172.21.15.16:6834/2645389320,v1:172.21.15.16:6835/2645389320] conn(0x55acae37ec50 0x55acae3d6a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:27.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.973+0000 7f8413e00640 1 --2- 172.21.15.16:0/2499620066 >> [v2:172.21.15.16:6834/2645389320,v1:172.21.15.16:6835/2645389320] conn(0x55acae37ec50 0x55acae3d6a50 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7a033392e852e6e4 server_cookie=2bc3a10241f140c4 in_seq=0 out_seq=0 2024-03-31T15:08:27.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.973+0000 7f83fd600640 10 client.7125 ms_handle_connect on v2:172.21.15.16:6834/2645389320 2024-03-31T15:08:27.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.973+0000 7f83fd600640 1 -- 172.21.15.16:0/2499620066 <== mds.0 v2:172.21.15.16:6834/2645389320 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55acae165dd0 con 0x55acae37ec50 2024-03-31T15:08:27.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.973+0000 7f83fd600640 10 client.7125 handle_command_reply: tid=0 2024-03-31T15:08:27.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.973+0000 7f841e729000 1 client.7125 shutdown 2024-03-31T15:08:27.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.973+0000 7f841e729000 1 -- 172.21.15.16:0/2499620066 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55acae3c5bf0 msgr2=0x55acae37e3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:27.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.973+0000 7f841e729000 1 --2- 172.21.15.16:0/2499620066 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55acae3c5bf0 0x55acae37e3b0 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7f840c007b40 tx=0x7f840c007b70 comp rx=0 tx=0).stop 2024-03-31T15:08:27.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.973+0000 7f841e729000 1 -- 172.21.15.16:0/2499620066 shutdown_connections 2024-03-31T15:08:27.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.973+0000 7f841e729000 1 --2- 172.21.15.16:0/2499620066 >> [v2:172.21.15.16:6834/2645389320,v1:172.21.15.16:6835/2645389320] conn(0x55acae37ec50 0x55acae3d6a50 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:27.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.973+0000 7f841e729000 1 --2- 172.21.15.16:0/2499620066 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55acae3c5bf0 0x55acae37e3b0 unknown :-1 s=CLOSED pgs=934 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:27.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.973+0000 7f841e729000 1 --2- 172.21.15.16:0/2499620066 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55acae3c52d0 0x55acae37de70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:27.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.973+0000 7f841e729000 1 --2- 172.21.15.16:0/2499620066 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55acae3789b0 0x55acae37d930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:27.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.973+0000 7f841e729000 1 -- 172.21.15.16:0/2499620066 >> 172.21.15.16:0/2499620066 conn(0x55acae3a6070 msgr2=0x55acae3a8460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:08:27.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.973+0000 7f83fd600640 0 client.7125 ms_handle_reset on v2:172.21.15.16:6834/2645389320 2024-03-31T15:08:27.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.973+0000 7f841e729000 1 -- 172.21.15.16:0/2499620066 shutdown_connections 2024-03-31T15:08:27.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.973+0000 7f841e729000 1 -- 172.21.15.16:0/2499620066 wait complete. 2024-03-31T15:08:27.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.973+0000 7f841e729000 20 client.7125 trim_cache size 0 max 16384 2024-03-31T15:08:27.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.993+0000 7f841e729000 1 Processor -- start 2024-03-31T15:08:27.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.993+0000 7f841e729000 1 -- start start 2024-03-31T15:08:27.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.993+0000 7f841e729000 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55acae3789b0 0x55acae380090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:27.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.993+0000 7f841e729000 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55acae3c52d0 0x55acae3805d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:27.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.993+0000 7f841e729000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55acae3c5bf0 0x55acae3c3380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:27.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.993+0000 7f841e729000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f8414203760 con 0x55acae3c5bf0 2024-03-31T15:08:27.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.993+0000 7f841e729000 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f84141f88b0 con 0x55acae3789b0 2024-03-31T15:08:27.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.993+0000 7f841e729000 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f8414153970 con 0x55acae3c52d0 2024-03-31T15:08:27.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.993+0000 7f8419200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55acae3c5bf0 0x55acae3c3380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:27.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.993+0000 7f8419200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55acae3c5bf0 0x55acae3c3380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34656/0 (socket says 172.21.15.16:34656) 2024-03-31T15:08:27.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.993+0000 7f8419200640 1 -- 172.21.15.16:0/2245973625 learned_addr learned my addr 172.21.15.16:0/2245973625 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:08:27.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.993+0000 7f8413400640 1 --2- 172.21.15.16:0/2245973625 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55acae3c52d0 0x55acae3805d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:27.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.993+0000 7f8413e00640 1 --2- 172.21.15.16:0/2245973625 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55acae3789b0 0x55acae380090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:27.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.993+0000 7f8419200640 1 -- 172.21.15.16:0/2245973625 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55acae3c52d0 msgr2=0x55acae3805d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:08:27.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.993+0000 7f8419200640 1 --2- 172.21.15.16:0/2245973625 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55acae3c52d0 0x55acae3805d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:27.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.993+0000 7f8419200640 1 -- 172.21.15.16:0/2245973625 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55acae3789b0 msgr2=0x55acae380090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:08:27.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.993+0000 7f8419200640 1 --2- 172.21.15.16:0/2245973625 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55acae3789b0 0x55acae380090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:27.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.993+0000 7f8419200640 1 -- 172.21.15.16:0/2245973625 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55acae24b140 con 0x55acae3c5bf0 2024-03-31T15:08:27.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.993+0000 7f8413400640 1 --2- 172.21.15.16:0/2245973625 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55acae3c52d0 0x55acae3805d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:08:27.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.993+0000 7f8413e00640 1 --2- 172.21.15.16:0/2245973625 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55acae3789b0 0x55acae380090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:08:27.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.993+0000 7f8419200640 1 --2- 172.21.15.16:0/2245973625 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55acae3c5bf0 0x55acae3c3380 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7f840c001350 tx=0x7f840c00e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12f8ac923c312ef6 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:08:27.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.993+0000 7f83ffe00640 1 -- 172.21.15.16:0/2245973625 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f840c001d90 con 0x55acae3c5bf0 2024-03-31T15:08:27.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.993+0000 7f83ffe00640 1 -- 172.21.15.16:0/2245973625 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f840c019070 con 0x55acae3c5bf0 2024-03-31T15:08:27.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.993+0000 7f83ffe00640 1 -- 172.21.15.16:0/2245973625 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f840c013860 con 0x55acae3c5bf0 2024-03-31T15:08:27.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.993+0000 7f841e729000 1 -- 172.21.15.16:0/2245973625 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55acae3c5bf0 msgr2=0x55acae3c3380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:27.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.993+0000 7f841e729000 1 --2- 172.21.15.16:0/2245973625 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55acae3c5bf0 0x55acae3c3380 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7f840c001350 tx=0x7f840c00e000 comp rx=0 tx=0).stop 2024-03-31T15:08:27.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.993+0000 7f841e729000 1 -- 172.21.15.16:0/2245973625 shutdown_connections 2024-03-31T15:08:27.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.993+0000 7f841e729000 1 --2- 172.21.15.16:0/2245973625 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55acae3c5bf0 0x55acae3c3380 unknown :-1 s=CLOSED pgs=935 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:27.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.993+0000 7f841e729000 1 --2- 172.21.15.16:0/2245973625 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55acae3c52d0 0x55acae3805d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:27.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.993+0000 7f841e729000 1 --2- 172.21.15.16:0/2245973625 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55acae3789b0 0x55acae380090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:27.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.993+0000 7f841e729000 1 -- 172.21.15.16:0/2245973625 >> 172.21.15.16:0/2245973625 conn(0x55acae3a6070 msgr2=0x55acae37c990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:08:27.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.993+0000 7f841e729000 1 -- 172.21.15.16:0/2245973625 shutdown_connections 2024-03-31T15:08:27.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.993+0000 7f841e729000 1 -- 172.21.15.16:0/2245973625 wait complete. 2024-03-31T15:08:27.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.993+0000 7f841e729000 1 -- start start 2024-03-31T15:08:27.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.997+0000 7f841e729000 1 Processor -- start 2024-03-31T15:08:27.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.997+0000 7f841e729000 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55acae3789b0 0x55acae434c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:27.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.997+0000 7f841e729000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55acae3c52d0 0x55acae435150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:27.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.997+0000 7f841e729000 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55acae3c5bf0 0x55acae418a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:27.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.997+0000 7f841e729000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55acae165940 con 0x55acae3c52d0 2024-03-31T15:08:27.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.997+0000 7f841e729000 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f84141f8260 con 0x55acae3c5bf0 2024-03-31T15:08:27.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.997+0000 7f841e729000 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x55acae384000 con 0x55acae3789b0 2024-03-31T15:08:27.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.997+0000 7f841e729000 10 client.0 _ll_register_callbacks cb 0x7f840c005e80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T15:08:27.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.997+0000 7f8413400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55acae3c52d0 0x55acae435150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:27.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.997+0000 7f8419200640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55acae3c5bf0 0x55acae418a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:27.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.997+0000 7f8413400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55acae3c52d0 0x55acae435150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34672/0 (socket says 172.21.15.16:34672) 2024-03-31T15:08:27.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.997+0000 7f8413400640 1 -- 172.21.15.16:0/1355933450 learned_addr learned my addr 172.21.15.16:0/1355933450 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:08:27.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.997+0000 7f8413e00640 1 --2- 172.21.15.16:0/1355933450 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55acae3789b0 0x55acae434c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:27.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.997+0000 7f8413400640 1 -- 172.21.15.16:0/1355933450 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55acae3789b0 msgr2=0x55acae434c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:08:27.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.997+0000 7f8413400640 1 --2- 172.21.15.16:0/1355933450 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55acae3789b0 0x55acae434c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:27.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.997+0000 7f8413400640 1 -- 172.21.15.16:0/1355933450 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55acae3c5bf0 msgr2=0x55acae418a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:08:27.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.997+0000 7f8413400640 1 --2- 172.21.15.16:0/1355933450 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55acae3c5bf0 0x55acae418a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:27.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.997+0000 7f8413400640 1 -- 172.21.15.16:0/1355933450 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55acae1591e0 con 0x55acae3c52d0 2024-03-31T15:08:27.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.997+0000 7f8413400640 1 --2- 172.21.15.16:0/1355933450 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55acae3c52d0 0x55acae435150 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7f840000c520 tx=0x7f840005d170 comp rx=0 tx=0).ready entity=mon.0 client_cookie=779d0b5e703867c1 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:08:27.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.997+0000 7f83fd600640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-31T15:08:27.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.997+0000 7f83fd600640 1 -- 172.21.15.16:0/1355933450 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8400082020 con 0x55acae3c52d0 2024-03-31T15:08:27.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.997+0000 7f83fd600640 1 -- 172.21.15.16:0/1355933450 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(70..70 src has 1..70) ==== 5882+0+0 (secure 0 0 0) 0x7f8400055c20 con 0x55acae3c52d0 2024-03-31T15:08:27.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.997+0000 7f841e729000 1 -- 172.21.15.16:0/1355933450 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f840c019070 con 0x55acae3c52d0 2024-03-31T15:08:27.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.997+0000 7f83fd600640 1 -- 172.21.15.16:0/1355933450 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=51) ==== 24+0+0 (secure 0 0 0) 0x7f8400021040 con 0x55acae3c52d0 2024-03-31T15:08:27.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.997+0000 7f841e729000 10 client.7131 fetch_fsmap learned FSMap version 51 2024-03-31T15:08:27.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.997+0000 7f841e729000 1 -- 172.21.15.16:0/1355933450 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=51,monmap=2+}) -- 0x55acae15e7b0 con 0x55acae3c52d0 2024-03-31T15:08:27.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.997+0000 7f83fd600640 1 -- 172.21.15.16:0/1355933450 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f840c019070 con 0x55acae3c52d0 2024-03-31T15:08:27.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.997+0000 7f83fd600640 1 -- 172.21.15.16:0/1355933450 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 51) ==== 3494+0+0 (secure 0 0 0) 0x7f8400024060 con 0x55acae3c52d0 2024-03-31T15:08:27.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.997+0000 7f841e729000 10 client.7131 fetch_fsmap finished waiting for FSMap version 51 2024-03-31T15:08:27.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.997+0000 7f841e729000 10 client.7131 resolve_mds: resolved 5:0 to role '5:0' aka daemon mds.a 2024-03-31T15:08:27.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.997+0000 7f841e729000 20 client.7131 populate_metadata read hostname 'smithi016' 2024-03-31T15:08:27.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.997+0000 7f841e729000 1 --2- 172.21.15.16:0/1355933450 >> [v2:172.21.15.16:6834/2645389320,v1:172.21.15.16:6835/2645389320] conn(0x55acae418fa0 0x55acae4193b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:27.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.997+0000 7f841e729000 4 client.7131 mds_command: new command op to 6829 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-31T15:08:27.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.997+0000 7f841e729000 1 -- 172.21.15.16:0/1355933450 --> [v2:172.21.15.16:6834/2645389320,v1:172.21.15.16:6835/2645389320] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55acae1d8040 con 0x55acae418fa0 2024-03-31T15:08:27.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.997+0000 7f8413e00640 1 --2- 172.21.15.16:0/1355933450 >> [v2:172.21.15.16:6834/2645389320,v1:172.21.15.16:6835/2645389320] conn(0x55acae418fa0 0x55acae4193b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:27.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.997+0000 7f8413e00640 1 --2- 172.21.15.16:0/1355933450 >> [v2:172.21.15.16:6834/2645389320,v1:172.21.15.16:6835/2645389320] conn(0x55acae418fa0 0x55acae4193b0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=76a5d7c3a95b6c9b server_cookie=63380513d8cc2027 in_seq=0 out_seq=0 2024-03-31T15:08:27.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.997+0000 7f83fd600640 10 client.7131 ms_handle_connect on v2:172.21.15.16:6834/2645389320 2024-03-31T15:08:27.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.997+0000 7f83fd600640 1 -- 172.21.15.16:0/1355933450 <== mds.0 v2:172.21.15.16:6834/2645389320 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55acae1d8040 con 0x55acae418fa0 2024-03-31T15:08:27.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.997+0000 7f83fd600640 10 client.7131 handle_command_reply: tid=0 2024-03-31T15:08:27.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:27.997+0000 7f841e729000 1 client.7131 shutdown 2024-03-31T15:08:27.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:28.001+0000 7f841e729000 1 -- 172.21.15.16:0/1355933450 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55acae3c52d0 msgr2=0x55acae435150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:27.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:28.001+0000 7f841e729000 1 --2- 172.21.15.16:0/1355933450 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55acae3c52d0 0x55acae435150 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7f840000c520 tx=0x7f840005d170 comp rx=0 tx=0).stop 2024-03-31T15:08:27.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:28.001+0000 7f841e729000 1 -- 172.21.15.16:0/1355933450 shutdown_connections 2024-03-31T15:08:27.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:28.001+0000 7f841e729000 1 --2- 172.21.15.16:0/1355933450 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55acae3c5bf0 0x55acae418a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:27.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:28.001+0000 7f841e729000 1 --2- 172.21.15.16:0/1355933450 >> [v2:172.21.15.16:6834/2645389320,v1:172.21.15.16:6835/2645389320] conn(0x55acae418fa0 0x55acae4193b0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:27.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:28.001+0000 7f841e729000 1 --2- 172.21.15.16:0/1355933450 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55acae3c52d0 0x55acae435150 unknown :-1 s=CLOSED pgs=936 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:27.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:28.001+0000 7f841e729000 1 --2- 172.21.15.16:0/1355933450 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55acae3789b0 0x55acae434c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:27.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:28.001+0000 7f83fd600640 0 client.7131 ms_handle_reset on v2:172.21.15.16:6834/2645389320 2024-03-31T15:08:27.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:28.001+0000 7f841e729000 1 -- 172.21.15.16:0/1355933450 >> 172.21.15.16:0/1355933450 conn(0x55acae3a6070 msgr2=0x55acae376bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:08:27.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:28.001+0000 7f841e729000 1 -- 172.21.15.16:0/1355933450 shutdown_connections 2024-03-31T15:08:27.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:28.001+0000 7f841e729000 1 -- 172.21.15.16:0/1355933450 wait complete. 2024-03-31T15:08:27.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:28.001+0000 7f841e729000 20 client.7131 trim_cache size 0 max 16384 2024-03-31T15:08:28.000 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-03-31T15:08:28.000 INFO:teuthology.orchestra.run.smithi016.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-31T15:08:28.000 INFO:teuthology.orchestra.run.smithi016.stdout: "scrubs": { 2024-03-31T15:08:28.000 INFO:teuthology.orchestra.run.smithi016.stdout: "4bab04b3-dab7-4217-af07-ee67fc7d3176": { 2024-03-31T15:08:28.000 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/subdir", 2024-03-31T15:08:28.000 INFO:teuthology.orchestra.run.smithi016.stdout: "tag": "4bab04b3-dab7-4217-af07-ee67fc7d3176", 2024-03-31T15:08:28.000 INFO:teuthology.orchestra.run.smithi016.stdout: "options": "recursive" 2024-03-31T15:08:28.000 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-31T15:08:28.000 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-31T15:08:28.000 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-03-31T15:08:28.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:28.001+0000 7f841b200640 1 -- 172.21.15.16:0/3991526783 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f83f4068380 msgr2=0x7f83f406a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:28.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:28.001+0000 7f841b200640 1 --2- 172.21.15.16:0/3991526783 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f83f4068380 0x7f83f406a840 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f84040ad000 tx=0x7f84040a6000 comp rx=0 tx=0).stop 2024-03-31T15:08:28.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:28.001+0000 7f841b200640 1 -- 172.21.15.16:0/3991526783 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8414141590 msgr2=0x7f8414154aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:28.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:28.001+0000 7f841b200640 1 --2- 172.21.15.16:0/3991526783 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8414141590 0x7f8414154aa0 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7f840000e900 tx=0x7f840000edc0 comp rx=0 tx=0).stop 2024-03-31T15:08:28.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:28.001+0000 7f841b200640 1 -- 172.21.15.16:0/3991526783 shutdown_connections 2024-03-31T15:08:28.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:28.001+0000 7f841b200640 1 --2- 172.21.15.16:0/3991526783 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f83f4068380 0x7f83f406a840 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:28.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:28.001+0000 7f841b200640 1 --2- 172.21.15.16:0/3991526783 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f8414143c30 0x7f841413eb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:28.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:28.001+0000 7f841b200640 1 --2- 172.21.15.16:0/3991526783 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8414141590 0x7f8414154aa0 unknown :-1 s=CLOSED pgs=932 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:28.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:28.001+0000 7f841b200640 1 --2- 172.21.15.16:0/3991526783 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f84141411b0 0x7f8414156330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:28.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:28.001+0000 7f841b200640 1 -- 172.21.15.16:0/3991526783 >> 172.21.15.16:0/3991526783 conn(0x7f8414067930 msgr2=0x7f84141432a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:08:28.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:28.001+0000 7f841b200640 1 -- 172.21.15.16:0/3991526783 shutdown_connections 2024-03-31T15:08:28.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:28.001+0000 7f841b200640 1 -- 172.21.15.16:0/3991526783 wait complete. 2024-03-31T15:08:28.015 INFO:tasks.cephfs.filesystem:scrub status for tag:4bab04b3-dab7-4217-af07-ee67fc7d3176 - {'path': '/subdir', 'tag': '4bab04b3-dab7-4217-af07-ee67fc7d3176', 'options': 'recursive'} 2024-03-31T15:08:37.148 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-31T15:08:37.151 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-31T15:08:58.016 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:08:58.017 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.5:0 scrub status 2024-03-31T15:08:58.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.177+0000 7f1bb9200640 1 Processor -- start 2024-03-31T15:08:58.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.177+0000 7f1bb9200640 1 -- start start 2024-03-31T15:08:58.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.181+0000 7f1bb9200640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f1bb41411b0 0x7f1bb406f920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:58.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.181+0000 7f1bb9200640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f1bb4144340 0x7f1bb406fe60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:58.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.181+0000 7f1bb9200640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1bb4144980 con 0x7f1bb4141590 2024-03-31T15:08:58.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.181+0000 7f1bb9200640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f1bb415f910 con 0x7f1bb41411b0 2024-03-31T15:08:58.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.181+0000 7f1bb9200640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f1bb4162910 con 0x7f1bb4144340 2024-03-31T15:08:58.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.181+0000 7f1bb1e00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f1bb41411b0 0x7f1bb406f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:58.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.181+0000 7f1bb1e00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f1bb41411b0 0x7f1bb406f920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42770/0 (socket says 172.21.15.16:42770) 2024-03-31T15:08:58.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.181+0000 7f1bb1e00640 1 -- 172.21.15.16:0/597064427 learned_addr learned my addr 172.21.15.16:0/597064427 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:08:58.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.181+0000 7f1bb2800640 1 --2- 172.21.15.16:0/597064427 >> v2:172.21.15.89:3300/0 conn(0x7f1bb4144340 0x7f1bb406fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:58.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.181+0000 7f1bb1e00640 1 -- 172.21.15.16:0/597064427 >> v2:172.21.15.89:3300/0 conn(0x7f1bb4144340 msgr2=0x7f1bb406fe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:08:58.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.181+0000 7f1bb1e00640 1 --2- 172.21.15.16:0/597064427 >> v2:172.21.15.89:3300/0 conn(0x7f1bb4144340 0x7f1bb406fe60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:58.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.181+0000 7f1bb1e00640 1 -- 172.21.15.16:0/597064427 >> v1:172.21.15.89:6789/0 conn(0x7f1bb4141590 legacy=0x7f1bb4143c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:58.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.181+0000 7f1bb1e00640 1 -- 172.21.15.16:0/597064427 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1bb4161fe0 con 0x7f1bb41411b0 2024-03-31T15:08:58.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.185+0000 7f1bb1e00640 1 --2- 172.21.15.16:0/597064427 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1bb41411b0 0x7f1bb406f920 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7f1bb40b6c30 tx=0x7f1ba401c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92bb01b91bf28c87 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:08:58.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.185+0000 7f1bb1400640 1 -- 172.21.15.16:0/597064427 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ba40ab610 con 0x7f1bb41411b0 2024-03-31T15:08:58.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.185+0000 7f1bb1400640 1 -- 172.21.15.16:0/597064427 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1ba40b0070 con 0x7f1bb41411b0 2024-03-31T15:08:58.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.185+0000 7f1bb1400640 1 -- 172.21.15.16:0/597064427 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ba40ab9d0 con 0x7f1bb41411b0 2024-03-31T15:08:58.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.185+0000 7f1bb9200640 1 -- 172.21.15.16:0/597064427 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1bb41411b0 msgr2=0x7f1bb406f920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:58.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.185+0000 7f1bb9200640 1 --2- 172.21.15.16:0/597064427 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1bb41411b0 0x7f1bb406f920 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7f1bb40b6c30 tx=0x7f1ba401c520 comp rx=0 tx=0).stop 2024-03-31T15:08:58.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.185+0000 7f1bb9200640 1 -- 172.21.15.16:0/597064427 shutdown_connections 2024-03-31T15:08:58.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.185+0000 7f1bb9200640 1 --2- 172.21.15.16:0/597064427 >> v2:172.21.15.89:3300/0 conn(0x7f1bb4144340 0x7f1bb406fe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:58.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.185+0000 7f1bb9200640 1 --2- 172.21.15.16:0/597064427 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1bb41411b0 0x7f1bb406f920 unknown :-1 s=CLOSED pgs=937 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:58.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.185+0000 7f1bb9200640 1 -- 172.21.15.16:0/597064427 >> 172.21.15.16:0/597064427 conn(0x7f1bb4067930 msgr2=0x7f1bb40b3ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:08:58.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.185+0000 7f1bb9200640 1 -- 172.21.15.16:0/597064427 shutdown_connections 2024-03-31T15:08:58.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.185+0000 7f1bb9200640 1 -- 172.21.15.16:0/597064427 wait complete. 2024-03-31T15:08:58.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.185+0000 7f1bb9200640 1 Processor -- start 2024-03-31T15:08:58.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.185+0000 7f1bb9200640 1 -- start start 2024-03-31T15:08:58.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.189+0000 7f1bb9200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1bb41411b0 0x7f1bb415d9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:58.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.189+0000 7f1bb9200640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f1bb4141590 0x7f1bb40b2660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:58.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.189+0000 7f1bb9200640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1bb4144340 0x7f1bb4159010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:58.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.189+0000 7f1bb9200640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f1bb415f290 con 0x7f1bb41411b0 2024-03-31T15:08:58.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.189+0000 7f1bb9200640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f1bb415f110 con 0x7f1bb4144340 2024-03-31T15:08:58.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.189+0000 7f1bb9200640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f1bb415f410 con 0x7f1bb4141590 2024-03-31T15:08:58.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.189+0000 7f1bb1e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1bb41411b0 0x7f1bb415d9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:58.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.189+0000 7f1bb1e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1bb41411b0 0x7f1bb415d9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42782/0 (socket says 172.21.15.16:42782) 2024-03-31T15:08:58.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.189+0000 7f1bb1e00640 1 -- 172.21.15.16:0/4105255821 learned_addr learned my addr 172.21.15.16:0/4105255821 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:08:58.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.189+0000 7f1bb2800640 1 --2- 172.21.15.16:0/4105255821 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1bb4144340 0x7f1bb4159010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:58.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.189+0000 7f1babe00640 1 --2- 172.21.15.16:0/4105255821 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f1bb4141590 0x7f1bb40b2660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:58.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.189+0000 7f1bb1e00640 1 -- 172.21.15.16:0/4105255821 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f1bb4141590 msgr2=0x7f1bb40b2660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:58.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.189+0000 7f1bb1e00640 1 --2- 172.21.15.16:0/4105255821 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f1bb4141590 0x7f1bb40b2660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:58.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.189+0000 7f1bb1e00640 1 -- 172.21.15.16:0/4105255821 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1bb4144340 msgr2=0x7f1bb4159010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:58.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.189+0000 7f1bb1e00640 1 --2- 172.21.15.16:0/4105255821 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1bb4144340 0x7f1bb4159010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:58.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.189+0000 7f1bb1e00640 1 -- 172.21.15.16:0/4105255821 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1bb415def0 con 0x7f1bb41411b0 2024-03-31T15:08:58.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.189+0000 7f1babe00640 1 --2- 172.21.15.16:0/4105255821 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f1bb4141590 0x7f1bb40b2660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:08:58.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.189+0000 7f1bb1e00640 1 --2- 172.21.15.16:0/4105255821 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1bb41411b0 0x7f1bb415d9b0 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7f1ba40ad040 tx=0x7f1ba401ad60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:08:58.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.189+0000 7f1baa000640 1 -- 172.21.15.16:0/4105255821 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ba40be5f0 con 0x7f1bb41411b0 2024-03-31T15:08:58.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.189+0000 7f1baa000640 1 -- 172.21.15.16:0/4105255821 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1ba40be790 con 0x7f1bb41411b0 2024-03-31T15:08:58.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.189+0000 7f1bb9200640 1 -- 172.21.15.16:0/4105255821 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1bb40b3100 con 0x7f1bb41411b0 2024-03-31T15:08:58.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.189+0000 7f1baa000640 1 -- 172.21.15.16:0/4105255821 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ba40beb60 con 0x7f1bb41411b0 2024-03-31T15:08:58.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.189+0000 7f1bb9200640 1 -- 172.21.15.16:0/4105255821 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1bb4152900 con 0x7f1bb41411b0 2024-03-31T15:08:58.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.189+0000 7f1baa000640 1 -- 172.21.15.16:0/4105255821 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f1ba40b43f0 con 0x7f1bb41411b0 2024-03-31T15:08:58.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.193+0000 7f1baa000640 1 --2- 172.21.15.16:0/4105255821 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f1b90068380 0x7f1b9006a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:58.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.193+0000 7f1baa000640 1 -- 172.21.15.16:0/4105255821 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(70..70 src has 1..70) ==== 5882+0+0 (secure 0 0 0) 0x7f1ba4121820 con 0x7f1bb41411b0 2024-03-31T15:08:58.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.193+0000 7f1babe00640 1 --2- 172.21.15.16:0/4105255821 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f1b90068380 0x7f1b9006a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:58.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.193+0000 7f1bbc748000 1 Processor -- start 2024-03-31T15:08:58.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.193+0000 7f1bbc748000 1 -- start start 2024-03-31T15:08:58.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.193+0000 7f1babe00640 1 --2- 172.21.15.16:0/4105255821 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f1b90068380 0x7f1b9006a840 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f1bb406c7a0 tx=0x7f1b9c009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:08:58.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.193+0000 7f1bbc748000 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56179eca62d0 0x56179eca66b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:58.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.193+0000 7f1bbc748000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56179eca6bf0 0x56179ec59470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:58.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.193+0000 7f1bbc748000 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x56179ec599b0 0x56179ec59d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:58.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.193+0000 7f1bbc748000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x56179eb42af0 con 0x56179eca6bf0 2024-03-31T15:08:58.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.193+0000 7f1bbc748000 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x56179eaabb00 con 0x56179eca62d0 2024-03-31T15:08:58.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.193+0000 7f1bbc748000 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x56179ea48940 con 0x56179ec599b0 2024-03-31T15:08:58.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.193+0000 7f1babe00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56179eca6bf0 0x56179ec59470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:58.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.193+0000 7f1babe00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56179eca6bf0 0x56179ec59470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42796/0 (socket says 172.21.15.16:42796) 2024-03-31T15:08:58.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.193+0000 7f1babe00640 1 -- 172.21.15.16:0/34091843 learned_addr learned my addr 172.21.15.16:0/34091843 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:08:58.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.193+0000 7f1babe00640 1 -- 172.21.15.16:0/34091843 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x56179ec599b0 msgr2=0x56179ec59d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:08:58.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.193+0000 7f1babe00640 1 --2- 172.21.15.16:0/34091843 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x56179ec599b0 0x56179ec59d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:58.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.193+0000 7f1babe00640 1 -- 172.21.15.16:0/34091843 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56179eca62d0 msgr2=0x56179eca66b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:08:58.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.193+0000 7f1babe00640 1 --2- 172.21.15.16:0/34091843 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56179eca62d0 0x56179eca66b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:58.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.193+0000 7f1babe00640 1 -- 172.21.15.16:0/34091843 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56179ea31380 con 0x56179eca6bf0 2024-03-31T15:08:58.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.193+0000 7f1babe00640 1 --2- 172.21.15.16:0/34091843 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56179eca6bf0 0x56179ec59470 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7f1b9c001ea0 tx=0x7f1b9c011000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d9aeacbc4ee5d3c4 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:08:58.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.193+0000 7f1ba2a00640 1 -- 172.21.15.16:0/34091843 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b9c003600 con 0x56179eca6bf0 2024-03-31T15:08:58.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.193+0000 7f1ba2a00640 1 -- 172.21.15.16:0/34091843 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1b9c0037a0 con 0x56179eca6bf0 2024-03-31T15:08:58.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.193+0000 7f1ba2a00640 1 -- 172.21.15.16:0/34091843 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b9c003b50 con 0x56179eca6bf0 2024-03-31T15:08:58.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.197+0000 7f1bbc748000 1 -- 172.21.15.16:0/34091843 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56179eca6bf0 msgr2=0x56179ec59470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:58.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.197+0000 7f1bbc748000 1 --2- 172.21.15.16:0/34091843 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56179eca6bf0 0x56179ec59470 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7f1b9c001ea0 tx=0x7f1b9c011000 comp rx=0 tx=0).stop 2024-03-31T15:08:58.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.197+0000 7f1bbc748000 1 -- 172.21.15.16:0/34091843 shutdown_connections 2024-03-31T15:08:58.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.197+0000 7f1bbc748000 1 --2- 172.21.15.16:0/34091843 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x56179ec599b0 0x56179ec59d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:58.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.197+0000 7f1bbc748000 1 --2- 172.21.15.16:0/34091843 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56179eca6bf0 0x56179ec59470 unknown :-1 s=CLOSED pgs=939 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:58.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.197+0000 7f1bbc748000 1 --2- 172.21.15.16:0/34091843 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56179eca62d0 0x56179eca66b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:58.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.197+0000 7f1bbc748000 1 -- 172.21.15.16:0/34091843 >> 172.21.15.16:0/34091843 conn(0x56179ec87070 msgr2=0x56179ec89460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:08:58.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.197+0000 7f1bbc748000 1 -- 172.21.15.16:0/34091843 shutdown_connections 2024-03-31T15:08:58.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.197+0000 7f1bbc748000 1 -- 172.21.15.16:0/34091843 wait complete. 2024-03-31T15:08:58.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.197+0000 7f1bbc748000 1 -- start start 2024-03-31T15:08:58.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.197+0000 7f1bbc748000 1 Processor -- start 2024-03-31T15:08:58.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.197+0000 7f1bbc748000 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56179ec599b0 0x56179ecad630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:58.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.197+0000 7f1bbc748000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56179eca62d0 0x56179ecadb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:58.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.197+0000 7f1bbc748000 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x56179eca6bf0 0x56179ec5e560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:58.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.197+0000 7f1bbc748000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x56179ea691e0 con 0x56179eca62d0 2024-03-31T15:08:58.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.197+0000 7f1bbc748000 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x56179eaa7ab0 con 0x56179ec599b0 2024-03-31T15:08:58.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.197+0000 7f1bbc748000 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x56179ec65000 con 0x56179eca6bf0 2024-03-31T15:08:58.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.197+0000 7f1bbc748000 10 client.0 _ll_register_callbacks cb 0x56179ec47370 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T15:08:58.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.197+0000 7f1babe00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56179eca62d0 0x56179ecadb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:58.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.197+0000 7f1babe00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56179eca62d0 0x56179ecadb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42804/0 (socket says 172.21.15.16:42804) 2024-03-31T15:08:58.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.197+0000 7f1babe00640 1 -- 172.21.15.16:0/983943233 learned_addr learned my addr 172.21.15.16:0/983943233 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:08:58.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.197+0000 7f1bb1e00640 1 --2- 172.21.15.16:0/983943233 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56179ec599b0 0x56179ecad630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:58.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.197+0000 7f1bb2800640 1 --2- 172.21.15.16:0/983943233 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x56179eca6bf0 0x56179ec5e560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:58.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.197+0000 7f1bb1e00640 1 -- 172.21.15.16:0/983943233 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x56179eca6bf0 msgr2=0x56179ec5e560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:08:58.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.197+0000 7f1bb1e00640 1 --2- 172.21.15.16:0/983943233 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x56179eca6bf0 0x56179ec5e560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:58.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.197+0000 7f1bb1e00640 1 -- 172.21.15.16:0/983943233 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56179eca62d0 msgr2=0x56179ecadb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:08:58.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.197+0000 7f1bb1e00640 1 --2- 172.21.15.16:0/983943233 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56179eca62d0 0x56179ecadb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:58.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.197+0000 7f1bb1e00640 1 -- 172.21.15.16:0/983943233 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56179ea67b70 con 0x56179ec599b0 2024-03-31T15:08:58.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.197+0000 7f1babe00640 1 --2- 172.21.15.16:0/983943233 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56179eca62d0 0x56179ecadb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:08:58.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.197+0000 7f1bb1e00640 1 --2- 172.21.15.16:0/983943233 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56179ec599b0 0x56179ecad630 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f1ba4097ee0 tx=0x7f1ba40f1030 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d8dcfa2cc3e8c6f2 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:08:58.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.197+0000 7f1b97400640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-03-31T15:08:58.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.197+0000 7f1b97400640 1 -- 172.21.15.16:0/983943233 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ba40f6d60 con 0x56179ec599b0 2024-03-31T15:08:58.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.197+0000 7f1b97400640 1 -- 172.21.15.16:0/983943233 <== mon.1 v2:172.21.15.89:3300/0 2 ==== osd_map(70..70 src has 1..70) ==== 5882+0+0 (secure 0 0 0) 0x7f1ba40f0b20 con 0x56179ec599b0 2024-03-31T15:08:58.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.197+0000 7f1bbc748000 1 -- 172.21.15.16:0/983943233 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1b9c0037a0 con 0x56179ec599b0 2024-03-31T15:08:58.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.197+0000 7f1b97400640 1 -- 172.21.15.16:0/983943233 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=51) ==== 24+0+0 (secure 0 0 0) 0x7f1ba410e050 con 0x56179ec599b0 2024-03-31T15:08:58.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.197+0000 7f1bbc748000 10 client.6976 fetch_fsmap learned FSMap version 51 2024-03-31T15:08:58.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.197+0000 7f1bbc748000 1 -- 172.21.15.16:0/983943233 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=51,monmap=2+}) -- 0x56179ea48b90 con 0x56179ec599b0 2024-03-31T15:08:58.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.201+0000 7f1b97400640 1 -- 172.21.15.16:0/983943233 <== mon.1 v2:172.21.15.89:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1b9c0037a0 con 0x56179ec599b0 2024-03-31T15:08:58.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.201+0000 7f1b97400640 1 -- 172.21.15.16:0/983943233 <== mon.1 v2:172.21.15.89:3300/0 5 ==== fsmap(e 51) ==== 3494+0+0 (secure 0 0 0) 0x7f1ba40c7070 con 0x56179ec599b0 2024-03-31T15:08:58.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.201+0000 7f1bbc748000 10 client.6976 fetch_fsmap finished waiting for FSMap version 51 2024-03-31T15:08:58.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.201+0000 7f1bbc748000 10 client.6976 resolve_mds: resolved 5:0 to role '5:0' aka daemon mds.a 2024-03-31T15:08:58.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.201+0000 7f1bbc748000 20 client.6976 populate_metadata read hostname 'smithi016' 2024-03-31T15:08:58.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.201+0000 7f1bbc748000 1 --2- 172.21.15.16:0/983943233 >> [v2:172.21.15.16:6834/2645389320,v1:172.21.15.16:6835/2645389320] conn(0x56179ec5f300 0x56179ec5f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:58.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.201+0000 7f1bbc748000 4 client.6976 mds_command: new command op to 6829 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-31T15:08:58.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.201+0000 7f1bbc748000 1 -- 172.21.15.16:0/983943233 --> [v2:172.21.15.16:6834/2645389320,v1:172.21.15.16:6835/2645389320] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56179ea46dd0 con 0x56179ec5f300 2024-03-31T15:08:58.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.201+0000 7f1babe00640 1 --2- 172.21.15.16:0/983943233 >> [v2:172.21.15.16:6834/2645389320,v1:172.21.15.16:6835/2645389320] conn(0x56179ec5f300 0x56179ec5f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:58.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.201+0000 7f1babe00640 1 --2- 172.21.15.16:0/983943233 >> [v2:172.21.15.16:6834/2645389320,v1:172.21.15.16:6835/2645389320] conn(0x56179ec5f300 0x56179ec5f6e0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=75c69c34ad5a0760 server_cookie=2b997b88a4953efa in_seq=0 out_seq=0 2024-03-31T15:08:58.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.201+0000 7f1b97400640 10 client.6976 ms_handle_connect on v2:172.21.15.16:6834/2645389320 2024-03-31T15:08:58.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.201+0000 7f1b97400640 1 -- 172.21.15.16:0/983943233 <== mds.0 v2:172.21.15.16:6834/2645389320 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56179ea46dd0 con 0x56179ec5f300 2024-03-31T15:08:58.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.201+0000 7f1b97400640 10 client.6976 handle_command_reply: tid=0 2024-03-31T15:08:58.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.201+0000 7f1bbc748000 1 client.6976 shutdown 2024-03-31T15:08:58.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.201+0000 7f1bbc748000 1 -- 172.21.15.16:0/983943233 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56179ec599b0 msgr2=0x56179ecad630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:58.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.201+0000 7f1bbc748000 1 --2- 172.21.15.16:0/983943233 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56179ec599b0 0x56179ecad630 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f1ba4097ee0 tx=0x7f1ba40f1030 comp rx=0 tx=0).stop 2024-03-31T15:08:58.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.201+0000 7f1bbc748000 1 -- 172.21.15.16:0/983943233 shutdown_connections 2024-03-31T15:08:58.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.201+0000 7f1bbc748000 1 --2- 172.21.15.16:0/983943233 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x56179eca6bf0 0x56179ec5e560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:58.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.201+0000 7f1bbc748000 1 --2- 172.21.15.16:0/983943233 >> [v2:172.21.15.16:6834/2645389320,v1:172.21.15.16:6835/2645389320] conn(0x56179ec5f300 0x56179ec5f6e0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:58.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.201+0000 7f1bbc748000 1 --2- 172.21.15.16:0/983943233 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56179eca62d0 0x56179ecadb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:58.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.201+0000 7f1bbc748000 1 --2- 172.21.15.16:0/983943233 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56179ec599b0 0x56179ecad630 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:58.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.201+0000 7f1b97400640 0 client.6976 ms_handle_reset on v2:172.21.15.16:6834/2645389320 2024-03-31T15:08:58.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.201+0000 7f1bbc748000 1 -- 172.21.15.16:0/983943233 >> 172.21.15.16:0/983943233 conn(0x56179ec87070 msgr2=0x56179ec89460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:08:58.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.201+0000 7f1bbc748000 1 -- 172.21.15.16:0/983943233 shutdown_connections 2024-03-31T15:08:58.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.201+0000 7f1bbc748000 1 -- 172.21.15.16:0/983943233 wait complete. 2024-03-31T15:08:58.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.201+0000 7f1bbc748000 20 client.6976 trim_cache size 0 max 16384 2024-03-31T15:08:58.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.221+0000 7f1bbc748000 1 Processor -- start 2024-03-31T15:08:58.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.221+0000 7f1bbc748000 1 -- start start 2024-03-31T15:08:58.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.221+0000 7f1bbc748000 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x56179ec87070 0x56179eca55a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:58.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.221+0000 7f1bbc748000 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56179ec599b0 0x56179eca5ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:58.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.221+0000 7f1bbc748000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56179eca62d0 0x56179eca4380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:58.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.221+0000 7f1bbc748000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f1bb4203820 con 0x56179eca62d0 2024-03-31T15:08:58.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.221+0000 7f1bbc748000 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f1bb4202840 con 0x56179ec599b0 2024-03-31T15:08:58.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.221+0000 7f1bbc748000 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f1bb4201bd0 con 0x56179ec87070 2024-03-31T15:08:58.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.221+0000 7f1bb2800640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56179eca62d0 0x56179eca4380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:58.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.221+0000 7f1bb2800640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56179eca62d0 0x56179eca4380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42816/0 (socket says 172.21.15.16:42816) 2024-03-31T15:08:58.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.221+0000 7f1bb2800640 1 -- 172.21.15.16:0/30307746 learned_addr learned my addr 172.21.15.16:0/30307746 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:08:58.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.221+0000 7f1bb1e00640 1 --2- 172.21.15.16:0/30307746 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x56179ec87070 0x56179eca55a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:58.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.221+0000 7f1babe00640 1 --2- 172.21.15.16:0/30307746 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56179ec599b0 0x56179eca5ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:58.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.221+0000 7f1bb2800640 1 -- 172.21.15.16:0/30307746 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x56179ec87070 msgr2=0x56179eca55a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:08:58.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.221+0000 7f1bb2800640 1 --2- 172.21.15.16:0/30307746 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x56179ec87070 0x56179eca55a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:58.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.221+0000 7f1bb2800640 1 -- 172.21.15.16:0/30307746 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56179ec599b0 msgr2=0x56179eca5ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:08:58.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.221+0000 7f1bb2800640 1 --2- 172.21.15.16:0/30307746 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56179ec599b0 0x56179eca5ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:58.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.221+0000 7f1bb2800640 1 -- 172.21.15.16:0/30307746 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56179eca6a20 con 0x56179eca62d0 2024-03-31T15:08:58.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.221+0000 7f1babe00640 1 --2- 172.21.15.16:0/30307746 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56179ec599b0 0x56179eca5ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:08:58.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.221+0000 7f1bb2800640 1 --2- 172.21.15.16:0/30307746 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56179eca62d0 0x56179eca4380 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7f1bac009fa0 tx=0x7f1bac003e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b82ed6a3356e0a1 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:08:58.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.221+0000 7f1ba2000640 1 -- 172.21.15.16:0/30307746 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bac01b020 con 0x56179eca62d0 2024-03-31T15:08:58.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.221+0000 7f1ba2000640 1 -- 172.21.15.16:0/30307746 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1bac004840 con 0x56179eca62d0 2024-03-31T15:08:58.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.221+0000 7f1ba2000640 1 -- 172.21.15.16:0/30307746 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bac017690 con 0x56179eca62d0 2024-03-31T15:08:58.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.221+0000 7f1bbc748000 1 -- 172.21.15.16:0/30307746 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56179eca62d0 msgr2=0x56179eca4380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:58.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.221+0000 7f1bbc748000 1 --2- 172.21.15.16:0/30307746 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56179eca62d0 0x56179eca4380 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7f1bac009fa0 tx=0x7f1bac003e30 comp rx=0 tx=0).stop 2024-03-31T15:08:58.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.221+0000 7f1bbc748000 1 -- 172.21.15.16:0/30307746 shutdown_connections 2024-03-31T15:08:58.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.221+0000 7f1bbc748000 1 --2- 172.21.15.16:0/30307746 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56179eca62d0 0x56179eca4380 unknown :-1 s=CLOSED pgs=940 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:58.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.221+0000 7f1bbc748000 1 --2- 172.21.15.16:0/30307746 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56179ec599b0 0x56179eca5ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:58.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.221+0000 7f1bbc748000 1 --2- 172.21.15.16:0/30307746 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x56179ec87070 0x56179eca55a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:58.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.221+0000 7f1bbc748000 1 -- 172.21.15.16:0/30307746 >> 172.21.15.16:0/30307746 conn(0x56179ecb40b0 msgr2=0x56179ecbc7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:08:58.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.221+0000 7f1bbc748000 1 -- 172.21.15.16:0/30307746 shutdown_connections 2024-03-31T15:08:58.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.221+0000 7f1bbc748000 1 -- 172.21.15.16:0/30307746 wait complete. 2024-03-31T15:08:58.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.221+0000 7f1bbc748000 1 -- start start 2024-03-31T15:08:58.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.221+0000 7f1bbc748000 1 Processor -- start 2024-03-31T15:08:58.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.225+0000 7f1bbc748000 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x56179ec599b0 0x56179ed167b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:58.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.225+0000 7f1bbc748000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56179ec87070 0x56179ed16cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:58.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.225+0000 7f1bbc748000 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56179eca62d0 0x56179ecaa030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:58.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.225+0000 7f1bbc748000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x56179eb7a420 con 0x56179ec87070 2024-03-31T15:08:58.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.225+0000 7f1bbc748000 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f1bb42021f0 con 0x56179eca62d0 2024-03-31T15:08:58.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.225+0000 7f1bbc748000 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x56179ea46940 con 0x56179ec599b0 2024-03-31T15:08:58.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.225+0000 7f1bbc748000 10 client.0 _ll_register_callbacks cb 0x7f1ba4006d00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T15:08:58.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.225+0000 7f1babe00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56179ec87070 0x56179ed16cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:58.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.225+0000 7f1babe00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56179ec87070 0x56179ed16cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42830/0 (socket says 172.21.15.16:42830) 2024-03-31T15:08:58.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.225+0000 7f1babe00640 1 -- 172.21.15.16:0/3490266540 learned_addr learned my addr 172.21.15.16:0/3490266540 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:08:58.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.225+0000 7f1bb1e00640 1 --2- 172.21.15.16:0/3490266540 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x56179ec599b0 0x56179ed167b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:58.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.225+0000 7f1bb2800640 1 --2- 172.21.15.16:0/3490266540 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56179eca62d0 0x56179ecaa030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:58.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.225+0000 7f1babe00640 1 -- 172.21.15.16:0/3490266540 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x56179ec599b0 msgr2=0x56179ed167b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:08:58.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.225+0000 7f1babe00640 1 --2- 172.21.15.16:0/3490266540 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x56179ec599b0 0x56179ed167b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:58.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.225+0000 7f1babe00640 1 -- 172.21.15.16:0/3490266540 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56179eca62d0 msgr2=0x56179ecaa030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:08:58.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.225+0000 7f1babe00640 1 --2- 172.21.15.16:0/3490266540 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56179eca62d0 0x56179ecaa030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:58.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.225+0000 7f1babe00640 1 -- 172.21.15.16:0/3490266540 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56179eb2c140 con 0x56179ec87070 2024-03-31T15:08:58.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.225+0000 7f1babe00640 1 --2- 172.21.15.16:0/3490266540 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56179ec87070 0x56179ed16cf0 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x56179eab1ae0 tx=0x7f1b9c000bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c926a2056a1704e9 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:08:58.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.225+0000 7f1b97400640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-31T15:08:58.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.225+0000 7f1b97400640 1 -- 172.21.15.16:0/3490266540 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b9c003d00 con 0x56179ec87070 2024-03-31T15:08:58.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.225+0000 7f1b97400640 1 -- 172.21.15.16:0/3490266540 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(70..70 src has 1..70) ==== 5882+0+0 (secure 0 0 0) 0x7f1b9c02f030 con 0x56179ec87070 2024-03-31T15:08:58.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.225+0000 7f1bbc748000 1 -- 172.21.15.16:0/3490266540 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1bac004840 con 0x56179ec87070 2024-03-31T15:08:58.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.225+0000 7f1b97400640 1 -- 172.21.15.16:0/3490266540 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=51) ==== 24+0+0 (secure 0 0 0) 0x7f1b9c038050 con 0x56179ec87070 2024-03-31T15:08:58.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.225+0000 7f1bbc748000 10 client.7149 fetch_fsmap learned FSMap version 51 2024-03-31T15:08:58.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.225+0000 7f1bbc748000 1 -- 172.21.15.16:0/3490266540 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=51,monmap=2+}) -- 0x56179ea3a1e0 con 0x56179ec87070 2024-03-31T15:08:58.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.225+0000 7f1b97400640 1 -- 172.21.15.16:0/3490266540 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1bac004840 con 0x56179ec87070 2024-03-31T15:08:58.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.225+0000 7f1b97400640 1 -- 172.21.15.16:0/3490266540 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 51) ==== 3494+0+0 (secure 0 0 0) 0x7f1b9c03b020 con 0x56179ec87070 2024-03-31T15:08:58.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.225+0000 7f1bbc748000 10 client.7149 fetch_fsmap finished waiting for FSMap version 51 2024-03-31T15:08:58.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.225+0000 7f1bbc748000 10 client.7149 resolve_mds: resolved 5:0 to role '5:0' aka daemon mds.a 2024-03-31T15:08:58.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.225+0000 7f1bbc748000 20 client.7149 populate_metadata read hostname 'smithi016' 2024-03-31T15:08:58.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.225+0000 7f1bbc748000 1 --2- 172.21.15.16:0/3490266540 >> [v2:172.21.15.16:6834/2645389320,v1:172.21.15.16:6835/2645389320] conn(0x56179eca6bf0 0x56179ecaa570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:58.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.225+0000 7f1bbc748000 4 client.7149 mds_command: new command op to 6829 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-31T15:08:58.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.225+0000 7f1bbc748000 1 -- 172.21.15.16:0/3490266540 --> [v2:172.21.15.16:6834/2645389320,v1:172.21.15.16:6835/2645389320] -- command(tid 0: {"prefix": "scrub status"}) -- 0x56179eab9040 con 0x56179eca6bf0 2024-03-31T15:08:58.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.225+0000 7f1bb1e00640 1 --2- 172.21.15.16:0/3490266540 >> [v2:172.21.15.16:6834/2645389320,v1:172.21.15.16:6835/2645389320] conn(0x56179eca6bf0 0x56179ecaa570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:58.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.225+0000 7f1b97400640 10 client.7149 ms_handle_connect on v2:172.21.15.16:6834/2645389320 2024-03-31T15:08:58.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.225+0000 7f1bb1e00640 1 --2- 172.21.15.16:0/3490266540 >> [v2:172.21.15.16:6834/2645389320,v1:172.21.15.16:6835/2645389320] conn(0x56179eca6bf0 0x56179ecaa570 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6ecf0681782ddfd7 server_cookie=ddcbff3080910984 in_seq=0 out_seq=0 2024-03-31T15:08:58.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.225+0000 7f1b97400640 1 -- 172.21.15.16:0/3490266540 <== mds.0 v2:172.21.15.16:6834/2645389320 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x56179eab9040 con 0x56179eca6bf0 2024-03-31T15:08:58.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.225+0000 7f1b97400640 10 client.7149 handle_command_reply: tid=0 2024-03-31T15:08:58.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.225+0000 7f1bbc748000 1 client.7149 shutdown 2024-03-31T15:08:58.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.225+0000 7f1bbc748000 1 -- 172.21.15.16:0/3490266540 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56179ec87070 msgr2=0x56179ed16cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:58.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.225+0000 7f1bbc748000 1 --2- 172.21.15.16:0/3490266540 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56179ec87070 0x56179ed16cf0 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x56179eab1ae0 tx=0x7f1b9c000bd0 comp rx=0 tx=0).stop 2024-03-31T15:08:58.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.225+0000 7f1bbc748000 1 -- 172.21.15.16:0/3490266540 shutdown_connections 2024-03-31T15:08:58.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.225+0000 7f1bbc748000 1 --2- 172.21.15.16:0/3490266540 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56179eca62d0 0x56179ecaa030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:58.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.225+0000 7f1bbc748000 1 --2- 172.21.15.16:0/3490266540 >> [v2:172.21.15.16:6834/2645389320,v1:172.21.15.16:6835/2645389320] conn(0x56179eca6bf0 0x56179ecaa570 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:58.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.225+0000 7f1bbc748000 1 --2- 172.21.15.16:0/3490266540 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56179ec87070 0x56179ed16cf0 unknown :-1 s=CLOSED pgs=941 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:58.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.225+0000 7f1bbc748000 1 --2- 172.21.15.16:0/3490266540 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x56179ec599b0 0x56179ed167b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:58.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.225+0000 7f1b97400640 0 client.7149 ms_handle_reset on v2:172.21.15.16:6834/2645389320 2024-03-31T15:08:58.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.225+0000 7f1bbc748000 1 -- 172.21.15.16:0/3490266540 >> 172.21.15.16:0/3490266540 conn(0x56179ecb40b0 msgr2=0x56179ec57bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:08:58.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.229+0000 7f1bbc748000 1 -- 172.21.15.16:0/3490266540 shutdown_connections 2024-03-31T15:08:58.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.229+0000 7f1bbc748000 1 -- 172.21.15.16:0/3490266540 wait complete. 2024-03-31T15:08:58.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.229+0000 7f1bbc748000 20 client.7149 trim_cache size 0 max 16384 2024-03-31T15:08:58.223 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-03-31T15:08:58.224 INFO:teuthology.orchestra.run.smithi016.stdout: "status": "no active scrubs running", 2024-03-31T15:08:58.224 INFO:teuthology.orchestra.run.smithi016.stdout: "scrubs": {} 2024-03-31T15:08:58.224 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-03-31T15:08:58.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.229+0000 7f1bb9200640 1 -- 172.21.15.16:0/4105255821 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f1b90068380 msgr2=0x7f1b9006a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:58.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.229+0000 7f1bb9200640 1 --2- 172.21.15.16:0/4105255821 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f1b90068380 0x7f1b9006a840 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f1bb406c7a0 tx=0x7f1b9c009040 comp rx=0 tx=0).stop 2024-03-31T15:08:58.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.229+0000 7f1bb9200640 1 -- 172.21.15.16:0/4105255821 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1bb41411b0 msgr2=0x7f1bb415d9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:58.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.229+0000 7f1bb9200640 1 --2- 172.21.15.16:0/4105255821 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1bb41411b0 0x7f1bb415d9b0 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7f1ba40ad040 tx=0x7f1ba401ad60 comp rx=0 tx=0).stop 2024-03-31T15:08:58.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.229+0000 7f1bb9200640 1 -- 172.21.15.16:0/4105255821 shutdown_connections 2024-03-31T15:08:58.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.229+0000 7f1bb9200640 1 --2- 172.21.15.16:0/4105255821 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f1b90068380 0x7f1b9006a840 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:58.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.229+0000 7f1bb9200640 1 --2- 172.21.15.16:0/4105255821 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1bb4144340 0x7f1bb4159010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:58.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.229+0000 7f1bb9200640 1 --2- 172.21.15.16:0/4105255821 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f1bb4141590 0x7f1bb40b2660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:58.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.229+0000 7f1bb9200640 1 --2- 172.21.15.16:0/4105255821 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1bb41411b0 0x7f1bb415d9b0 unknown :-1 s=CLOSED pgs=938 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:58.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.229+0000 7f1bb9200640 1 -- 172.21.15.16:0/4105255821 >> 172.21.15.16:0/4105255821 conn(0x7f1bb4067930 msgr2=0x7f1bb40b5660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:08:58.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.229+0000 7f1bb9200640 1 -- 172.21.15.16:0/4105255821 shutdown_connections 2024-03-31T15:08:58.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.229+0000 7f1bb9200640 1 -- 172.21.15.16:0/4105255821 wait complete. 2024-03-31T15:08:58.238 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-31T15:08:58.238 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-31T15:08:58.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.389+0000 7fb926a00640 1 Processor -- start 2024-03-31T15:08:58.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.389+0000 7fb926a00640 1 -- start start 2024-03-31T15:08:58.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.389+0000 7fb926a00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fb920145e40 0x7fb9200cc740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:58.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.389+0000 7fb926a00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb9200ccc80 0x7fb92006f920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:58.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.389+0000 7fb926a00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb92006fe60 0x7fb92015fdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:58.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.389+0000 7fb926a00640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7fb9200d06d0 con 0x7fb9200ccc80 2024-03-31T15:08:58.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.389+0000 7fb926a00640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7fb9200d0920 con 0x7fb920145e40 2024-03-31T15:08:58.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.389+0000 7fb926a00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fb9200d0c00 con 0x7fb92006fe60 2024-03-31T15:08:58.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.389+0000 7fb917e00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb9200ccc80 0x7fb92006f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:58.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.389+0000 7fb917e00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb9200ccc80 0x7fb92006f920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42840/0 (socket says 172.21.15.16:42840) 2024-03-31T15:08:58.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.389+0000 7fb917e00640 1 -- 172.21.15.16:0/1369634543 learned_addr learned my addr 172.21.15.16:0/1369634543 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:08:58.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.389+0000 7fb91fe00640 1 --2- 172.21.15.16:0/1369634543 >> v2:172.21.15.89:3300/0 conn(0x7fb920145e40 0x7fb9200cc740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:58.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.389+0000 7fb924a00640 1 --2- 172.21.15.16:0/1369634543 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb92006fe60 0x7fb92015fdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:58.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.389+0000 7fb917e00640 1 -- 172.21.15.16:0/1369634543 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb92006fe60 msgr2=0x7fb92015fdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:08:58.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.389+0000 7fb917e00640 1 --2- 172.21.15.16:0/1369634543 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb92006fe60 0x7fb92015fdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:58.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.389+0000 7fb917e00640 1 -- 172.21.15.16:0/1369634543 >> v2:172.21.15.89:3300/0 conn(0x7fb920145e40 msgr2=0x7fb9200cc740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:08:58.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.389+0000 7fb917e00640 1 --2- 172.21.15.16:0/1369634543 >> v2:172.21.15.89:3300/0 conn(0x7fb920145e40 0x7fb9200cc740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:58.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.389+0000 7fb917e00640 1 -- 172.21.15.16:0/1369634543 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb9200d1810 con 0x7fb9200ccc80 2024-03-31T15:08:58.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.389+0000 7fb91fe00640 1 --2- 172.21.15.16:0/1369634543 >> v2:172.21.15.89:3300/0 conn(0x7fb920145e40 0x7fb9200cc740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:08:58.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.397+0000 7fb917e00640 1 --2- 172.21.15.16:0/1369634543 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb9200ccc80 0x7fb92006f920 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7fb920072e60 tx=0x7fb90801c260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=91883c8d48604a3a server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:08:58.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.397+0000 7fb91f400640 1 -- 172.21.15.16:0/1369634543 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9080b0070 con 0x7fb9200ccc80 2024-03-31T15:08:58.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.397+0000 7fb91f400640 1 -- 172.21.15.16:0/1369634543 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb9080ab600 con 0x7fb9200ccc80 2024-03-31T15:08:58.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.397+0000 7fb91f400640 1 -- 172.21.15.16:0/1369634543 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9080ab9d0 con 0x7fb9200ccc80 2024-03-31T15:08:58.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.397+0000 7fb926a00640 1 -- 172.21.15.16:0/1369634543 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb9200ccc80 msgr2=0x7fb92006f920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:58.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.397+0000 7fb926a00640 1 --2- 172.21.15.16:0/1369634543 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb9200ccc80 0x7fb92006f920 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7fb920072e60 tx=0x7fb90801c260 comp rx=0 tx=0).stop 2024-03-31T15:08:58.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.397+0000 7fb926a00640 1 -- 172.21.15.16:0/1369634543 shutdown_connections 2024-03-31T15:08:58.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.397+0000 7fb926a00640 1 --2- 172.21.15.16:0/1369634543 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb92006fe60 0x7fb92015fdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:58.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.397+0000 7fb926a00640 1 --2- 172.21.15.16:0/1369634543 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb9200ccc80 0x7fb92006f920 unknown :-1 s=CLOSED pgs=942 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:58.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.397+0000 7fb926a00640 1 --2- 172.21.15.16:0/1369634543 >> v2:172.21.15.89:3300/0 conn(0x7fb920145e40 0x7fb9200cc740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:58.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.397+0000 7fb926a00640 1 -- 172.21.15.16:0/1369634543 >> 172.21.15.16:0/1369634543 conn(0x7fb920067330 msgr2=0x7fb920067740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:08:58.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.397+0000 7fb926a00640 1 -- 172.21.15.16:0/1369634543 shutdown_connections 2024-03-31T15:08:58.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.397+0000 7fb926a00640 1 -- 172.21.15.16:0/1369634543 wait complete. 2024-03-31T15:08:58.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.397+0000 7fb926a00640 1 Processor -- start 2024-03-31T15:08:58.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.397+0000 7fb926a00640 1 -- start start 2024-03-31T15:08:58.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.401+0000 7fb926a00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb92006fe60 0x7fb920159c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:58.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.401+0000 7fb926a00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb9200ccc80 0x7fb92015e170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:58.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.401+0000 7fb926a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb920145e40 0x7fb92015e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:58.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.401+0000 7fb926a00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fb9200b5240 con 0x7fb920145e40 2024-03-31T15:08:58.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.401+0000 7fb926a00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fb9200b50c0 con 0x7fb92006fe60 2024-03-31T15:08:58.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.401+0000 7fb926a00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fb9200d2010 con 0x7fb9200ccc80 2024-03-31T15:08:58.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.401+0000 7fb924a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb920145e40 0x7fb92015e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:58.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.401+0000 7fb91fe00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb92006fe60 0x7fb920159c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:58.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.401+0000 7fb91fe00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb92006fe60 0x7fb920159c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.16:43198/0 (socket says 172.21.15.16:43198) 2024-03-31T15:08:58.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.401+0000 7fb91fe00640 1 -- 172.21.15.16:0/4173599597 learned_addr learned my addr 172.21.15.16:0/4173599597 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:08:58.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.401+0000 7fb917e00640 1 --2- 172.21.15.16:0/4173599597 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb9200ccc80 0x7fb92015e170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:58.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.401+0000 7fb91fe00640 1 -- 172.21.15.16:0/4173599597 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb9200ccc80 msgr2=0x7fb92015e170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:58.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.401+0000 7fb91fe00640 1 --2- 172.21.15.16:0/4173599597 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb9200ccc80 0x7fb92015e170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:58.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.401+0000 7fb91fe00640 1 -- 172.21.15.16:0/4173599597 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb920145e40 msgr2=0x7fb92015e6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:58.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.401+0000 7fb91fe00640 1 --2- 172.21.15.16:0/4173599597 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb920145e40 0x7fb92015e6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:58.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.401+0000 7fb91fe00640 1 -- 172.21.15.16:0/4173599597 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb92015cc70 con 0x7fb92006fe60 2024-03-31T15:08:58.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.401+0000 7fb917e00640 1 --2- 172.21.15.16:0/4173599597 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb9200ccc80 0x7fb92015e170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:08:58.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.401+0000 7fb924a00640 1 --2- 172.21.15.16:0/4173599597 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb920145e40 0x7fb92015e6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:08:58.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.401+0000 7fb91fe00640 1 --2- 172.21.15.16:0/4173599597 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb92006fe60 0x7fb920159c80 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fb91000d800 tx=0x7fb91000dcc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:08:58.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.401+0000 7fb91cc00640 1 -- 172.21.15.16:0/4173599597 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb91000b6c0 con 0x7fb92006fe60 2024-03-31T15:08:58.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.401+0000 7fb91cc00640 1 -- 172.21.15.16:0/4173599597 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb910019070 con 0x7fb92006fe60 2024-03-31T15:08:58.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.401+0000 7fb91cc00640 1 -- 172.21.15.16:0/4173599597 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb91000baa0 con 0x7fb92006fe60 2024-03-31T15:08:58.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.401+0000 7fb926a00640 1 -- 172.21.15.16:0/4173599597 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb92015cf50 con 0x7fb92006fe60 2024-03-31T15:08:58.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.401+0000 7fb926a00640 1 -- 172.21.15.16:0/4173599597 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb92015d510 con 0x7fb92006fe60 2024-03-31T15:08:58.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.401+0000 7fb926a00640 1 -- 172.21.15.16:0/4173599597 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb920069950 con 0x7fb92006fe60 2024-03-31T15:08:58.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.405+0000 7fb91cc00640 1 -- 172.21.15.16:0/4173599597 <== mon.1 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fb91000bc40 con 0x7fb92006fe60 2024-03-31T15:08:58.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.405+0000 7fb91cc00640 1 --2- 172.21.15.16:0/4173599597 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fb8fc0683b0 0x7fb8fc06a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:58.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.405+0000 7fb91cc00640 1 -- 172.21.15.16:0/4173599597 <== mon.1 v2:172.21.15.89:3300/0 5 ==== osd_map(70..70 src has 1..70) ==== 5882+0+0 (secure 0 0 0) 0x7fb910086fe0 con 0x7fb92006fe60 2024-03-31T15:08:58.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.405+0000 7fb917e00640 1 --2- 172.21.15.16:0/4173599597 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fb8fc0683b0 0x7fb8fc06a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:58.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.405+0000 7fb917e00640 1 --2- 172.21.15.16:0/4173599597 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fb8fc0683b0 0x7fb8fc06a870 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7fb9080ad000 tx=0x7fb908008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:08:58.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.409+0000 7fb91cc00640 1 -- 172.21.15.16:0/4173599597 <== mon.1 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fb910056ea0 con 0x7fb92006fe60 2024-03-31T15:08:58.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.609+0000 7fb926a00640 1 -- 172.21.15.16:0/4173599597 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb9200cf3c0 con 0x7fb92006fe60 2024-03-31T15:08:58.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.609+0000 7fb91cc00640 1 -- 172.21.15.16:0/4173599597 <== mon.1 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 51 v51) ==== 94+0+5184 (secure 0 0 0) 0x7fb91005b1a0 con 0x7fb92006fe60 2024-03-31T15:08:58.602 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:08:58.602 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":51,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6836/4188793506","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":4188793506},{"type":"v1","addr":"172.21.15.89:6836","nonce":4188793506}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":40},{"gid":5942,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/4105657676","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6835","nonce":4105657676},{"type":"v1","addr":"172.21.15.89:6837","nonce":4105657676}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":40},{"gid":6592,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6833/3986752851","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":3986752851},{"type":"v1","addr":"172.21.15.16:6833","nonce":3986752851}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":39}],"filesystems":[{"mdsmap":{"epoch":51,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-31T15:07:56.938140+0000","modified":"2024-03-31T15:08:21.266866+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":68,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6829},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6829":{"gid":6829,"name":"a","rank":0,"incarnation":47,"state":"up:active","state_seq":6,"addr":"172.21.15.16:6835/2645389320","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":2645389320},{"type":"v1","addr":"172.21.15.16:6835","nonce":2645389320}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6829,"qdb_cluster":[6829]},"id":5}]} 2024-03-31T15:08:58.602 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 51 2024-03-31T15:08:58.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.613+0000 7fb926a00640 1 -- 172.21.15.16:0/4173599597 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fb8fc0683b0 msgr2=0x7fb8fc06a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:58.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.613+0000 7fb926a00640 1 --2- 172.21.15.16:0/4173599597 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fb8fc0683b0 0x7fb8fc06a870 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7fb9080ad000 tx=0x7fb908008040 comp rx=0 tx=0).stop 2024-03-31T15:08:58.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.613+0000 7fb926a00640 1 -- 172.21.15.16:0/4173599597 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb92006fe60 msgr2=0x7fb920159c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:58.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.613+0000 7fb926a00640 1 --2- 172.21.15.16:0/4173599597 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb92006fe60 0x7fb920159c80 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fb91000d800 tx=0x7fb91000dcc0 comp rx=0 tx=0).stop 2024-03-31T15:08:58.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.613+0000 7fb926a00640 1 -- 172.21.15.16:0/4173599597 shutdown_connections 2024-03-31T15:08:58.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.613+0000 7fb926a00640 1 --2- 172.21.15.16:0/4173599597 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fb8fc0683b0 0x7fb8fc06a870 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:58.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.613+0000 7fb926a00640 1 --2- 172.21.15.16:0/4173599597 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb920145e40 0x7fb92015e6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:58.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.613+0000 7fb926a00640 1 --2- 172.21.15.16:0/4173599597 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb9200ccc80 0x7fb92015e170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:58.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.613+0000 7fb926a00640 1 --2- 172.21.15.16:0/4173599597 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb92006fe60 0x7fb920159c80 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:58.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.613+0000 7fb926a00640 1 -- 172.21.15.16:0/4173599597 >> 172.21.15.16:0/4173599597 conn(0x7fb920067330 msgr2=0x7fb9200b34c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:08:58.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.613+0000 7fb926a00640 1 -- 172.21.15.16:0/4173599597 shutdown_connections 2024-03-31T15:08:58.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.613+0000 7fb926a00640 1 -- 172.21.15.16:0/4173599597 wait complete. 2024-03-31T15:08:58.624 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a damage ls --format=json-pretty 2024-03-31T15:08:58.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.773+0000 7f1c21c00640 1 Processor -- start 2024-03-31T15:08:58.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.773+0000 7f1c21c00640 1 -- start start 2024-03-31T15:08:58.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.777+0000 7f1c21c00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f1c1c1411b0 0x7f1c1c06f920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:58.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.777+0000 7f1c21c00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f1c1c141590 0x7f1c1c06fe60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:58.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.777+0000 7f1c21c00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f1c1c143c30 0x7f1c1c1440a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:58.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.777+0000 7f1c21c00640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f1c1c162910 con 0x7f1c1c1411b0 2024-03-31T15:08:58.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.777+0000 7f1c21c00640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f1c1c144980 con 0x7f1c1c143c30 2024-03-31T15:08:58.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.777+0000 7f1c21c00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f1c1c161fe0 con 0x7f1c1c141590 2024-03-31T15:08:58.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.777+0000 7f1c1aa00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f1c1c1411b0 0x7f1c1c06f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:58.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.777+0000 7f1c1aa00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f1c1c1411b0 0x7f1c1c06f920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42852/0 (socket says 172.21.15.16:42852) 2024-03-31T15:08:58.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.777+0000 7f1c1aa00640 1 -- 172.21.15.16:0/4267838250 learned_addr learned my addr 172.21.15.16:0/4267838250 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:08:58.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.777+0000 7f1c12000640 1 --2- 172.21.15.16:0/4267838250 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f1c1c141590 0x7f1c1c06fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:58.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.777+0000 7f1c1aa00640 1 -- 172.21.15.16:0/4267838250 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f1c1c141590 msgr2=0x7f1c1c06fe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:08:58.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.777+0000 7f1c1b400640 1 --2- 172.21.15.16:0/4267838250 >> v2:172.21.15.89:3300/0 conn(0x7f1c1c143c30 0x7f1c1c1440a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:58.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.777+0000 7f1c1aa00640 1 --2- 172.21.15.16:0/4267838250 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f1c1c141590 0x7f1c1c06fe60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:58.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.777+0000 7f1c1aa00640 1 -- 172.21.15.16:0/4267838250 >> v2:172.21.15.89:3300/0 conn(0x7f1c1c143c30 msgr2=0x7f1c1c1440a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:08:58.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.777+0000 7f1c1aa00640 1 --2- 172.21.15.16:0/4267838250 >> v2:172.21.15.89:3300/0 conn(0x7f1c1c143c30 0x7f1c1c1440a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:58.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.777+0000 7f1c1aa00640 1 -- 172.21.15.16:0/4267838250 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c1c1622c0 con 0x7f1c1c1411b0 2024-03-31T15:08:58.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.777+0000 7f1c12000640 1 --2- 172.21.15.16:0/4267838250 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f1c1c141590 0x7f1c1c06fe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:08:58.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.777+0000 7f1c1b400640 1 --2- 172.21.15.16:0/4267838250 >> v2:172.21.15.89:3300/0 conn(0x7f1c1c143c30 0x7f1c1c1440a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:08:58.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.781+0000 7f1c1aa00640 1 --2- 172.21.15.16:0/4267838250 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c1c1411b0 0x7f1c1c06f920 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7f1c1c0b1c00 tx=0x7f1c0c01c4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b3e8effa342553a5 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:08:58.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.781+0000 7f1c1a000640 1 -- 172.21.15.16:0/4267838250 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c0c0ab640 con 0x7f1c1c1411b0 2024-03-31T15:08:58.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.781+0000 7f1c1a000640 1 -- 172.21.15.16:0/4267838250 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1c0c0b0070 con 0x7f1c1c1411b0 2024-03-31T15:08:58.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.781+0000 7f1c1a000640 1 -- 172.21.15.16:0/4267838250 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c0c0aba00 con 0x7f1c1c1411b0 2024-03-31T15:08:58.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.781+0000 7f1c21c00640 1 -- 172.21.15.16:0/4267838250 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c1c1411b0 msgr2=0x7f1c1c06f920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:58.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.781+0000 7f1c21c00640 1 --2- 172.21.15.16:0/4267838250 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c1c1411b0 0x7f1c1c06f920 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7f1c1c0b1c00 tx=0x7f1c0c01c4f0 comp rx=0 tx=0).stop 2024-03-31T15:08:58.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.781+0000 7f1c21c00640 1 -- 172.21.15.16:0/4267838250 shutdown_connections 2024-03-31T15:08:58.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.781+0000 7f1c21c00640 1 --2- 172.21.15.16:0/4267838250 >> v2:172.21.15.89:3300/0 conn(0x7f1c1c143c30 0x7f1c1c1440a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:58.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.781+0000 7f1c21c00640 1 --2- 172.21.15.16:0/4267838250 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f1c1c141590 0x7f1c1c06fe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:58.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.781+0000 7f1c21c00640 1 --2- 172.21.15.16:0/4267838250 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c1c1411b0 0x7f1c1c06f920 unknown :-1 s=CLOSED pgs=943 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:58.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.781+0000 7f1c21c00640 1 -- 172.21.15.16:0/4267838250 >> 172.21.15.16:0/4267838250 conn(0x7f1c1c067930 msgr2=0x7f1c1c0b3ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:08:58.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.781+0000 7f1c21c00640 1 -- 172.21.15.16:0/4267838250 shutdown_connections 2024-03-31T15:08:58.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.781+0000 7f1c21c00640 1 -- 172.21.15.16:0/4267838250 wait complete. 2024-03-31T15:08:58.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.781+0000 7f1c21c00640 1 Processor -- start 2024-03-31T15:08:58.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.781+0000 7f1c21c00640 1 -- start start 2024-03-31T15:08:58.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.785+0000 7f1c21c00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1c1c1411b0 0x7f1c1c156430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:58.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.785+0000 7f1c21c00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f1c1c141590 0x7f1c1c13e660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:58.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.785+0000 7f1c21c00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c1c143c30 0x7f1c1c157240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:58.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.785+0000 7f1c21c00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f1c1c0717f0 con 0x7f1c1c143c30 2024-03-31T15:08:58.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.785+0000 7f1c21c00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f1c1c071670 con 0x7f1c1c1411b0 2024-03-31T15:08:58.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.785+0000 7f1c21c00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f1c1c071970 con 0x7f1c1c141590 2024-03-31T15:08:58.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.785+0000 7f1c1b400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c1c143c30 0x7f1c1c157240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:58.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.785+0000 7f1c1aa00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1c1c1411b0 0x7f1c1c156430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:58.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.785+0000 7f1c12000640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f1c1c141590 0x7f1c1c13e660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:58.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.785+0000 7f1c1aa00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1c1c1411b0 0x7f1c1c156430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.16:43214/0 (socket says 172.21.15.16:43214) 2024-03-31T15:08:58.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.785+0000 7f1c1aa00640 1 -- 172.21.15.16:0/1237348100 learned_addr learned my addr 172.21.15.16:0/1237348100 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:08:58.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.785+0000 7f1c1b400640 1 -- 172.21.15.16:0/1237348100 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f1c1c141590 msgr2=0x7f1c1c13e660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:58.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.785+0000 7f1c1b400640 1 --2- 172.21.15.16:0/1237348100 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f1c1c141590 0x7f1c1c13e660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:58.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.785+0000 7f1c1b400640 1 -- 172.21.15.16:0/1237348100 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1c1c1411b0 msgr2=0x7f1c1c156430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:58.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.785+0000 7f1c1b400640 1 --2- 172.21.15.16:0/1237348100 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1c1c1411b0 0x7f1c1c156430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:58.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.785+0000 7f1c1b400640 1 -- 172.21.15.16:0/1237348100 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c1c1596a0 con 0x7f1c1c143c30 2024-03-31T15:08:58.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.785+0000 7f1c1aa00640 1 --2- 172.21.15.16:0/1237348100 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1c1c1411b0 0x7f1c1c156430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:08:58.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.785+0000 7f1c1b400640 1 --2- 172.21.15.16:0/1237348100 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c1c143c30 0x7f1c1c157240 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7f1c1400cc90 tx=0x7f1c1400a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:08:58.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.785+0000 7f1c13400640 1 -- 172.21.15.16:0/1237348100 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c1400f680 con 0x7f1c1c143c30 2024-03-31T15:08:58.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.785+0000 7f1c13400640 1 -- 172.21.15.16:0/1237348100 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1c1400f840 con 0x7f1c1c143c30 2024-03-31T15:08:58.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.785+0000 7f1c13400640 1 -- 172.21.15.16:0/1237348100 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c1400fc30 con 0x7f1c1c143c30 2024-03-31T15:08:58.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.785+0000 7f1c21c00640 1 -- 172.21.15.16:0/1237348100 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1c1c159980 con 0x7f1c1c143c30 2024-03-31T15:08:58.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.785+0000 7f1c21c00640 1 -- 172.21.15.16:0/1237348100 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1c1c13eba0 con 0x7f1c1c143c30 2024-03-31T15:08:58.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.785+0000 7f1c13400640 1 -- 172.21.15.16:0/1237348100 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f1c140213f0 con 0x7f1c1c143c30 2024-03-31T15:08:58.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.789+0000 7f1c13400640 1 --2- 172.21.15.16:0/1237348100 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f1bf8068380 0x7f1bf806a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:58.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.789+0000 7f1c13400640 1 -- 172.21.15.16:0/1237348100 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(70..70 src has 1..70) ==== 5882+0+0 (secure 0 0 0) 0x7f1c14087570 con 0x7f1c1c143c30 2024-03-31T15:08:58.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.789+0000 7f1c2518b000 1 Processor -- start 2024-03-31T15:08:58.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.789+0000 7f1c1aa00640 1 --2- 172.21.15.16:0/1237348100 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f1bf8068380 0x7f1bf806a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:58.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.789+0000 7f1c2518b000 1 -- start start 2024-03-31T15:08:58.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.789+0000 7f1c1aa00640 1 --2- 172.21.15.16:0/1237348100 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f1bf8068380 0x7f1bf806a840 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f1c0c01c530 tx=0x7f1c0c0a7000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:08:58.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.789+0000 7f1c2518b000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5561f353a980 0x5561f353ad60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:58.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.789+0000 7f1c2518b000 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5561f353b2a0 0x5561f34eddd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:58.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.789+0000 7f1c2518b000 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5561f34ee310 0x5561f34ee6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:58.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.789+0000 7f1c2518b000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5561f333ec50 con 0x5561f353a980 2024-03-31T15:08:58.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.789+0000 7f1c2518b000 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x5561f32db8a0 con 0x5561f353b2a0 2024-03-31T15:08:58.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.789+0000 7f1c2518b000 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x5561f333ac00 con 0x5561f34ee310 2024-03-31T15:08:58.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.789+0000 7f1c1aa00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5561f353a980 0x5561f353ad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:58.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.789+0000 7f1c1aa00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5561f353a980 0x5561f353ad60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42874/0 (socket says 172.21.15.16:42874) 2024-03-31T15:08:58.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.789+0000 7f1c1aa00640 1 -- 172.21.15.16:0/96116800 learned_addr learned my addr 172.21.15.16:0/96116800 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:08:58.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.789+0000 7f1c12000640 1 --2- 172.21.15.16:0/96116800 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5561f353b2a0 0x5561f34eddd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:58.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.789+0000 7f1c1b400640 1 --2- 172.21.15.16:0/96116800 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5561f34ee310 0x5561f34ee6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:58.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.789+0000 7f1c1aa00640 1 -- 172.21.15.16:0/96116800 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5561f34ee310 msgr2=0x5561f34ee6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:08:58.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.789+0000 7f1c1aa00640 1 --2- 172.21.15.16:0/96116800 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5561f34ee310 0x5561f34ee6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:58.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.789+0000 7f1c1aa00640 1 -- 172.21.15.16:0/96116800 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5561f353b2a0 msgr2=0x5561f34eddd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:08:58.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.789+0000 7f1c1aa00640 1 --2- 172.21.15.16:0/96116800 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5561f353b2a0 0x5561f34eddd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:58.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.789+0000 7f1c1aa00640 1 -- 172.21.15.16:0/96116800 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5561f3318ba0 con 0x5561f353a980 2024-03-31T15:08:58.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.789+0000 7f1c12000640 1 --2- 172.21.15.16:0/96116800 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5561f353b2a0 0x5561f34eddd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:08:58.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.789+0000 7f1c1aa00640 1 --2- 172.21.15.16:0/96116800 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5561f353a980 0x5561f353ad60 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7f1c0c097f10 tx=0x7f1c0c004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=61d058fde3c51e5a server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:08:58.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.789+0000 7f1c0b400640 1 -- 172.21.15.16:0/96116800 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c0c0c9630 con 0x5561f353a980 2024-03-31T15:08:58.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.789+0000 7f1c0b400640 1 -- 172.21.15.16:0/96116800 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1c0c0b0040 con 0x5561f353a980 2024-03-31T15:08:58.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.789+0000 7f1c0b400640 1 -- 172.21.15.16:0/96116800 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c0c0c99d0 con 0x5561f353a980 2024-03-31T15:08:58.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.793+0000 7f1c2518b000 1 -- 172.21.15.16:0/96116800 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5561f353a980 msgr2=0x5561f353ad60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:58.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.793+0000 7f1c2518b000 1 --2- 172.21.15.16:0/96116800 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5561f353a980 0x5561f353ad60 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7f1c0c097f10 tx=0x7f1c0c004740 comp rx=0 tx=0).stop 2024-03-31T15:08:58.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.793+0000 7f1c2518b000 1 -- 172.21.15.16:0/96116800 shutdown_connections 2024-03-31T15:08:58.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.793+0000 7f1c2518b000 1 --2- 172.21.15.16:0/96116800 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5561f34ee310 0x5561f34ee6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:58.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.793+0000 7f1c2518b000 1 --2- 172.21.15.16:0/96116800 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5561f353b2a0 0x5561f34eddd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:58.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.793+0000 7f1c2518b000 1 --2- 172.21.15.16:0/96116800 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5561f353a980 0x5561f353ad60 unknown :-1 s=CLOSED pgs=945 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:58.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.793+0000 7f1c2518b000 1 -- 172.21.15.16:0/96116800 >> 172.21.15.16:0/96116800 conn(0x5561f32eb9f0 msgr2=0x5561f351d730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:08:58.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.793+0000 7f1c2518b000 1 -- 172.21.15.16:0/96116800 shutdown_connections 2024-03-31T15:08:58.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.793+0000 7f1c2518b000 1 -- 172.21.15.16:0/96116800 wait complete. 2024-03-31T15:08:58.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.793+0000 7f1c2518b000 1 -- start start 2024-03-31T15:08:58.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.793+0000 7f1c2518b000 1 Processor -- start 2024-03-31T15:08:58.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.793+0000 7f1c2518b000 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5561f34ee310 0x5561f353a110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:58.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.793+0000 7f1c2518b000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5561f353a980 0x5561f34f3a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:58.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.793+0000 7f1c2518b000 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5561f353b2a0 0x5561f34f3fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:58.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.793+0000 7f1c2518b000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5561f32fc0a0 con 0x5561f353a980 2024-03-31T15:08:58.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.793+0000 7f1c2518b000 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x5561f334b360 con 0x5561f353b2a0 2024-03-31T15:08:58.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.793+0000 7f1c2518b000 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x5561f33f1e90 con 0x5561f34ee310 2024-03-31T15:08:58.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.793+0000 7f1c2518b000 10 client.0 _ll_register_callbacks cb 0x5561f34db810 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T15:08:58.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.793+0000 7f1c12000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5561f353a980 0x5561f34f3a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:58.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.793+0000 7f1c12000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5561f353a980 0x5561f34f3a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42878/0 (socket says 172.21.15.16:42878) 2024-03-31T15:08:58.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.793+0000 7f1c12000640 1 -- 172.21.15.16:0/3933991767 learned_addr learned my addr 172.21.15.16:0/3933991767 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:08:58.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.793+0000 7f1c1aa00640 1 --2- 172.21.15.16:0/3933991767 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5561f34ee310 0x5561f353a110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:58.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.793+0000 7f1c1b400640 1 --2- 172.21.15.16:0/3933991767 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5561f353b2a0 0x5561f34f3fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:58.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.793+0000 7f1c12000640 1 -- 172.21.15.16:0/3933991767 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5561f34ee310 msgr2=0x5561f353a110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:08:58.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.793+0000 7f1c12000640 1 --2- 172.21.15.16:0/3933991767 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5561f34ee310 0x5561f353a110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:58.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.793+0000 7f1c12000640 1 -- 172.21.15.16:0/3933991767 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5561f353b2a0 msgr2=0x5561f34f3fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:08:58.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.793+0000 7f1c12000640 1 --2- 172.21.15.16:0/3933991767 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5561f353b2a0 0x5561f34f3fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:58.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.793+0000 7f1c12000640 1 -- 172.21.15.16:0/3933991767 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5561f3384160 con 0x5561f353a980 2024-03-31T15:08:58.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.793+0000 7f1c12000640 1 --2- 172.21.15.16:0/3933991767 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5561f353a980 0x5561f34f3a90 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7f1c04012780 tx=0x7f1c04012c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6103deb18614df91 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:08:58.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.793+0000 7f1bffe00640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-31T15:08:58.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.793+0000 7f1bffe00640 1 -- 172.21.15.16:0/3933991767 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c04010ca0 con 0x5561f353a980 2024-03-31T15:08:58.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.793+0000 7f1bffe00640 1 -- 172.21.15.16:0/3933991767 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(70..70 src has 1..70) ==== 5882+0+0 (secure 0 0 0) 0x7f1c04010e40 con 0x5561f353a980 2024-03-31T15:08:58.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.793+0000 7f1c2518b000 1 -- 172.21.15.16:0/3933991767 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1c0c0b0040 con 0x5561f353a980 2024-03-31T15:08:58.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.793+0000 7f1bffe00640 1 -- 172.21.15.16:0/3933991767 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=51) ==== 24+0+0 (secure 0 0 0) 0x7f1c04004020 con 0x5561f353a980 2024-03-31T15:08:58.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.793+0000 7f1c2518b000 10 client.7167 fetch_fsmap learned FSMap version 51 2024-03-31T15:08:58.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.793+0000 7f1c2518b000 1 -- 172.21.15.16:0/3933991767 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=51,monmap=2+}) -- 0x5561f32d99c0 con 0x5561f353a980 2024-03-31T15:08:58.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.793+0000 7f1bffe00640 1 -- 172.21.15.16:0/3933991767 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1c0c0b0040 con 0x5561f353a980 2024-03-31T15:08:58.790 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.793+0000 7f1bffe00640 1 -- 172.21.15.16:0/3933991767 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 51) ==== 3494+0+0 (secure 0 0 0) 0x7f1c04024050 con 0x5561f353a980 2024-03-31T15:08:58.790 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.793+0000 7f1c2518b000 10 client.7167 fetch_fsmap finished waiting for FSMap version 51 2024-03-31T15:08:58.790 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.793+0000 7f1c2518b000 10 client.7167 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-31T15:08:58.790 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.793+0000 7f1c2518b000 20 client.7167 populate_metadata read hostname 'smithi016' 2024-03-31T15:08:58.790 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.793+0000 7f1c2518b000 1 --2- 172.21.15.16:0/3933991767 >> [v2:172.21.15.16:6834/2645389320,v1:172.21.15.16:6835/2645389320] conn(0x5561f3541dc0 0x5561f35421a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:58.790 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.793+0000 7f1c2518b000 4 client.7167 mds_command: new command op to 6829 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-31T15:08:58.790 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.793+0000 7f1c2518b000 1 -- 172.21.15.16:0/3933991767 --> [v2:172.21.15.16:6834/2645389320,v1:172.21.15.16:6835/2645389320] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5561f32d9e40 con 0x5561f3541dc0 2024-03-31T15:08:58.790 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.797+0000 7f1c1aa00640 1 --2- 172.21.15.16:0/3933991767 >> [v2:172.21.15.16:6834/2645389320,v1:172.21.15.16:6835/2645389320] conn(0x5561f3541dc0 0x5561f35421a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:58.790 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.797+0000 7f1c1aa00640 1 --2- 172.21.15.16:0/3933991767 >> [v2:172.21.15.16:6834/2645389320,v1:172.21.15.16:6835/2645389320] conn(0x5561f3541dc0 0x5561f35421a0 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bf405028b19ef838 server_cookie=78390ae87113e445 in_seq=0 out_seq=0 2024-03-31T15:08:58.790 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.797+0000 7f1bffe00640 10 client.7167 ms_handle_connect on v2:172.21.15.16:6834/2645389320 2024-03-31T15:08:58.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.797+0000 7f1bffe00640 1 -- 172.21.15.16:0/3933991767 <== mds.0 v2:172.21.15.16:6834/2645389320 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5561f32d9e40 con 0x5561f3541dc0 2024-03-31T15:08:58.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.797+0000 7f1bffe00640 10 client.7167 handle_command_reply: tid=0 2024-03-31T15:08:58.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.797+0000 7f1c2518b000 1 client.7167 shutdown 2024-03-31T15:08:58.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.797+0000 7f1c2518b000 1 -- 172.21.15.16:0/3933991767 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5561f353a980 msgr2=0x5561f34f3a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:58.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.797+0000 7f1c2518b000 1 --2- 172.21.15.16:0/3933991767 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5561f353a980 0x5561f34f3a90 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7f1c04012780 tx=0x7f1c04012c40 comp rx=0 tx=0).stop 2024-03-31T15:08:58.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.797+0000 7f1c2518b000 1 -- 172.21.15.16:0/3933991767 shutdown_connections 2024-03-31T15:08:58.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.797+0000 7f1c2518b000 1 --2- 172.21.15.16:0/3933991767 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5561f353b2a0 0x5561f34f3fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:58.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.797+0000 7f1c2518b000 1 --2- 172.21.15.16:0/3933991767 >> [v2:172.21.15.16:6834/2645389320,v1:172.21.15.16:6835/2645389320] conn(0x5561f3541dc0 0x5561f35421a0 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:58.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.797+0000 7f1c2518b000 1 --2- 172.21.15.16:0/3933991767 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5561f353a980 0x5561f34f3a90 unknown :-1 s=CLOSED pgs=946 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:58.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.797+0000 7f1c2518b000 1 --2- 172.21.15.16:0/3933991767 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5561f34ee310 0x5561f353a110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:58.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.797+0000 7f1bffe00640 0 client.7167 ms_handle_reset on v2:172.21.15.16:6834/2645389320 2024-03-31T15:08:58.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.797+0000 7f1c2518b000 1 -- 172.21.15.16:0/3933991767 >> 172.21.15.16:0/3933991767 conn(0x5561f32eb9f0 msgr2=0x5561f351d520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:08:58.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.797+0000 7f1c2518b000 1 -- 172.21.15.16:0/3933991767 shutdown_connections 2024-03-31T15:08:58.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.797+0000 7f1c2518b000 1 -- 172.21.15.16:0/3933991767 wait complete. 2024-03-31T15:08:58.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.797+0000 7f1c2518b000 20 client.7167 trim_cache size 0 max 16384 2024-03-31T15:08:58.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.809+0000 7f1c2518b000 1 Processor -- start 2024-03-31T15:08:58.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.809+0000 7f1c2518b000 1 -- start start 2024-03-31T15:08:58.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.813+0000 7f1c2518b000 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5561f34ee310 0x5561f3539140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:58.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.813+0000 7f1c2518b000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5561f353a980 0x5561f3539680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:58.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.813+0000 7f1c2518b000 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5561f353b2a0 0x5561f34f58b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:58.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.813+0000 7f1c2518b000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f1c1c15f600 con 0x5561f353a980 2024-03-31T15:08:58.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.813+0000 7f1c2518b000 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f1c1c15e960 con 0x5561f353b2a0 2024-03-31T15:08:58.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.813+0000 7f1c2518b000 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f1c1c15efb0 con 0x5561f34ee310 2024-03-31T15:08:58.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.813+0000 7f1c12000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5561f353a980 0x5561f3539680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:58.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.813+0000 7f1c12000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5561f353a980 0x5561f3539680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42894/0 (socket says 172.21.15.16:42894) 2024-03-31T15:08:58.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.813+0000 7f1c12000640 1 -- 172.21.15.16:0/2898731671 learned_addr learned my addr 172.21.15.16:0/2898731671 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:08:58.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.813+0000 7f1c1aa00640 1 --2- 172.21.15.16:0/2898731671 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5561f34ee310 0x5561f3539140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:58.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.813+0000 7f1c1b400640 1 --2- 172.21.15.16:0/2898731671 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5561f353b2a0 0x5561f34f58b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:58.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.813+0000 7f1c12000640 1 -- 172.21.15.16:0/2898731671 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5561f34ee310 msgr2=0x5561f3539140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:08:58.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.813+0000 7f1c12000640 1 --2- 172.21.15.16:0/2898731671 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5561f34ee310 0x5561f3539140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:58.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.813+0000 7f1c12000640 1 -- 172.21.15.16:0/2898731671 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5561f353b2a0 msgr2=0x5561f34f58b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:08:58.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.813+0000 7f1c12000640 1 --2- 172.21.15.16:0/2898731671 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5561f353b2a0 0x5561f34f58b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:58.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.813+0000 7f1c12000640 1 -- 172.21.15.16:0/2898731671 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5561f32dbaf0 con 0x5561f353a980 2024-03-31T15:08:58.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.813+0000 7f1c1b400640 1 --2- 172.21.15.16:0/2898731671 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5561f353b2a0 0x5561f34f58b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:08:58.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.813+0000 7f1c12000640 1 --2- 172.21.15.16:0/2898731671 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5561f353a980 0x5561f3539680 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7f1c0400acf0 tx=0x7f1c04017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d4214f515d3b0c24 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:08:58.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.813+0000 7f1c0aa00640 1 -- 172.21.15.16:0/2898731671 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c0402b030 con 0x5561f353a980 2024-03-31T15:08:58.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.813+0000 7f1c0aa00640 1 -- 172.21.15.16:0/2898731671 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1c04000e50 con 0x5561f353a980 2024-03-31T15:08:58.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.813+0000 7f1c0aa00640 1 -- 172.21.15.16:0/2898731671 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c04010680 con 0x5561f353a980 2024-03-31T15:08:58.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.813+0000 7f1c2518b000 1 -- 172.21.15.16:0/2898731671 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5561f353a980 msgr2=0x5561f3539680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:58.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.813+0000 7f1c2518b000 1 --2- 172.21.15.16:0/2898731671 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5561f353a980 0x5561f3539680 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7f1c0400acf0 tx=0x7f1c04017000 comp rx=0 tx=0).stop 2024-03-31T15:08:58.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.813+0000 7f1c2518b000 1 -- 172.21.15.16:0/2898731671 shutdown_connections 2024-03-31T15:08:58.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.813+0000 7f1c2518b000 1 --2- 172.21.15.16:0/2898731671 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5561f353b2a0 0x5561f34f58b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:58.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.813+0000 7f1c2518b000 1 --2- 172.21.15.16:0/2898731671 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5561f353a980 0x5561f3539680 unknown :-1 s=CLOSED pgs=947 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:58.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.813+0000 7f1c2518b000 1 --2- 172.21.15.16:0/2898731671 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5561f34ee310 0x5561f3539140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:58.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.813+0000 7f1c2518b000 1 -- 172.21.15.16:0/2898731671 >> 172.21.15.16:0/2898731671 conn(0x5561f32eb9f0 msgr2=0x5561f3538a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:08:58.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.813+0000 7f1c2518b000 1 -- 172.21.15.16:0/2898731671 shutdown_connections 2024-03-31T15:08:58.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.813+0000 7f1c2518b000 1 -- 172.21.15.16:0/2898731671 wait complete. 2024-03-31T15:08:58.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.813+0000 7f1c2518b000 1 -- start start 2024-03-31T15:08:58.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.813+0000 7f1c2518b000 1 Processor -- start 2024-03-31T15:08:58.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.817+0000 7f1c2518b000 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5561f34ee310 0x5561f35a7cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:58.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.817+0000 7f1c2518b000 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5561f353a980 0x5561f35a8230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:58.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.817+0000 7f1c2518b000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5561f353b2a0 0x5561f35a8770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:58.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.817+0000 7f1c2518b000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5561f32e0ea0 con 0x5561f353b2a0 2024-03-31T15:08:58.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.817+0000 7f1c2518b000 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x5561f32f3250 con 0x5561f353a980 2024-03-31T15:08:58.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.817+0000 7f1c2518b000 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x5561f3340ca0 con 0x5561f34ee310 2024-03-31T15:08:58.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.817+0000 7f1c2518b000 10 client.0 _ll_register_callbacks cb 0x7f1c0400a780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T15:08:58.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.817+0000 7f1c1b400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5561f353b2a0 0x5561f35a8770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:58.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.817+0000 7f1c1aa00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5561f34ee310 0x5561f35a7cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:58.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.817+0000 7f1c1b400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5561f353b2a0 0x5561f35a8770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42904/0 (socket says 172.21.15.16:42904) 2024-03-31T15:08:58.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.817+0000 7f1c1b400640 1 -- 172.21.15.16:0/3579530618 learned_addr learned my addr 172.21.15.16:0/3579530618 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:08:58.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.817+0000 7f1c12000640 1 --2- 172.21.15.16:0/3579530618 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5561f353a980 0x5561f35a8230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:58.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.817+0000 7f1c1b400640 1 -- 172.21.15.16:0/3579530618 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5561f34ee310 msgr2=0x5561f35a7cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:08:58.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.817+0000 7f1c1b400640 1 --2- 172.21.15.16:0/3579530618 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5561f34ee310 0x5561f35a7cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:58.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.817+0000 7f1c1b400640 1 -- 172.21.15.16:0/3579530618 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5561f353a980 msgr2=0x5561f35a8230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:08:58.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.817+0000 7f1c1b400640 1 --2- 172.21.15.16:0/3579530618 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5561f353a980 0x5561f35a8230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:58.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.817+0000 7f1c1b400640 1 -- 172.21.15.16:0/3579530618 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5561f3397320 con 0x5561f353b2a0 2024-03-31T15:08:58.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.817+0000 7f1c1b400640 1 --2- 172.21.15.16:0/3579530618 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5561f353b2a0 0x5561f35a8770 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7f1c14053880 tx=0x7f1c1400cea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=398e6aa483bf07b server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:08:58.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.817+0000 7f1bffe00640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-31T15:08:58.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.817+0000 7f1bffe00640 1 -- 172.21.15.16:0/3579530618 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c1405fcc0 con 0x5561f353b2a0 2024-03-31T15:08:58.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.817+0000 7f1bffe00640 1 -- 172.21.15.16:0/3579530618 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(70..70 src has 1..70) ==== 5882+0+0 (secure 0 0 0) 0x7f1c14082be0 con 0x5561f353b2a0 2024-03-31T15:08:58.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.817+0000 7f1c2518b000 1 -- 172.21.15.16:0/3579530618 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1c04000e50 con 0x5561f353b2a0 2024-03-31T15:08:58.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.817+0000 7f1bffe00640 1 -- 172.21.15.16:0/3579530618 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=51) ==== 24+0+0 (secure 0 0 0) 0x7f1c14082de0 con 0x5561f353b2a0 2024-03-31T15:08:58.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.817+0000 7f1c2518b000 10 client.7173 fetch_fsmap learned FSMap version 51 2024-03-31T15:08:58.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.817+0000 7f1c2518b000 1 -- 172.21.15.16:0/3579530618 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=51,monmap=2+}) -- 0x5561f32d2a80 con 0x5561f353b2a0 2024-03-31T15:08:58.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.817+0000 7f1bffe00640 1 -- 172.21.15.16:0/3579530618 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1c04000e50 con 0x5561f353b2a0 2024-03-31T15:08:58.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.817+0000 7f1bffe00640 1 -- 172.21.15.16:0/3579530618 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 51) ==== 3494+0+0 (secure 0 0 0) 0x7f1c140870b0 con 0x5561f353b2a0 2024-03-31T15:08:58.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.817+0000 7f1c2518b000 10 client.7173 fetch_fsmap finished waiting for FSMap version 51 2024-03-31T15:08:58.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.817+0000 7f1c2518b000 10 client.7173 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-31T15:08:58.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.817+0000 7f1c2518b000 20 client.7173 populate_metadata read hostname 'smithi016' 2024-03-31T15:08:58.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.817+0000 7f1c2518b000 1 --2- 172.21.15.16:0/3579530618 >> [v2:172.21.15.16:6834/2645389320,v1:172.21.15.16:6835/2645389320] conn(0x5561f35a8cb0 0x5561f35a9090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:58.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.817+0000 7f1c2518b000 4 client.7173 mds_command: new command op to 6829 tid=0 multi_id=0 [{"prefix": "damage ls", "format": "json-pretty"}] 2024-03-31T15:08:58.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.817+0000 7f1c2518b000 1 -- 172.21.15.16:0/3579530618 --> [v2:172.21.15.16:6834/2645389320,v1:172.21.15.16:6835/2645389320] -- command(tid 0: {"prefix": "damage ls", "format": "json-pretty"}) -- 0x5561f334c190 con 0x5561f35a8cb0 2024-03-31T15:08:58.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.817+0000 7f1c1aa00640 1 --2- 172.21.15.16:0/3579530618 >> [v2:172.21.15.16:6834/2645389320,v1:172.21.15.16:6835/2645389320] conn(0x5561f35a8cb0 0x5561f35a9090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:58.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.817+0000 7f1c1aa00640 1 --2- 172.21.15.16:0/3579530618 >> [v2:172.21.15.16:6834/2645389320,v1:172.21.15.16:6835/2645389320] conn(0x5561f35a8cb0 0x5561f35a9090 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=972ecf5727a41c6a server_cookie=56cd8852442cfefc in_seq=0 out_seq=0 2024-03-31T15:08:58.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.817+0000 7f1bffe00640 10 client.7173 ms_handle_connect on v2:172.21.15.16:6834/2645389320 2024-03-31T15:08:58.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.817+0000 7f1bffe00640 1 -- 172.21.15.16:0/3579530618 <== mds.0 v2:172.21.15.16:6834/2645389320 1 ==== command_reply(tid 0: 0 ) ==== 8+0+237 (crc 0 0 0) 0x5561f334c190 con 0x5561f35a8cb0 2024-03-31T15:08:58.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.817+0000 7f1bffe00640 10 client.7173 handle_command_reply: tid=0 2024-03-31T15:08:58.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.817+0000 7f1c2518b000 1 client.7173 shutdown 2024-03-31T15:08:58.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.817+0000 7f1c2518b000 1 -- 172.21.15.16:0/3579530618 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5561f353b2a0 msgr2=0x5561f35a8770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:58.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.817+0000 7f1c2518b000 1 --2- 172.21.15.16:0/3579530618 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5561f353b2a0 0x5561f35a8770 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7f1c14053880 tx=0x7f1c1400cea0 comp rx=0 tx=0).stop 2024-03-31T15:08:58.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.817+0000 7f1c2518b000 1 -- 172.21.15.16:0/3579530618 shutdown_connections 2024-03-31T15:08:58.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.817+0000 7f1c2518b000 1 --2- 172.21.15.16:0/3579530618 >> [v2:172.21.15.16:6834/2645389320,v1:172.21.15.16:6835/2645389320] conn(0x5561f35a8cb0 0x5561f35a9090 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:58.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.817+0000 7f1c2518b000 1 --2- 172.21.15.16:0/3579530618 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5561f353b2a0 0x5561f35a8770 unknown :-1 s=CLOSED pgs=948 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:58.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.817+0000 7f1c2518b000 1 --2- 172.21.15.16:0/3579530618 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5561f353a980 0x5561f35a8230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:58.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.817+0000 7f1c2518b000 1 --2- 172.21.15.16:0/3579530618 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5561f34ee310 0x5561f35a7cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:58.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.817+0000 7f1c2518b000 1 -- 172.21.15.16:0/3579530618 >> 172.21.15.16:0/3579530618 conn(0x5561f32eb9f0 msgr2=0x5561f35392c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:08:58.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.817+0000 7f1bffe00640 0 client.7173 ms_handle_reset on v2:172.21.15.16:6834/2645389320 2024-03-31T15:08:58.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.817+0000 7f1c2518b000 1 -- 172.21.15.16:0/3579530618 shutdown_connections 2024-03-31T15:08:58.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.817+0000 7f1c2518b000 1 -- 172.21.15.16:0/3579530618 wait complete. 2024-03-31T15:08:58.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.821+0000 7f1c2518b000 20 client.7173 trim_cache size 0 max 16384 2024-03-31T15:08:58.815 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:08:58.815 INFO:teuthology.orchestra.run.smithi016.stdout:[ 2024-03-31T15:08:58.815 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-31T15:08:58.815 INFO:teuthology.orchestra.run.smithi016.stdout: "damage_type": "dentry", 2024-03-31T15:08:58.815 INFO:teuthology.orchestra.run.smithi016.stdout: "id": 3946515147, 2024-03-31T15:08:58.815 INFO:teuthology.orchestra.run.smithi016.stdout: "ino": 1099511627776, 2024-03-31T15:08:58.815 INFO:teuthology.orchestra.run.smithi016.stdout: "frag": "*", 2024-03-31T15:08:58.815 INFO:teuthology.orchestra.run.smithi016.stdout: "dname": "file_to_be_damaged", 2024-03-31T15:08:58.816 INFO:teuthology.orchestra.run.smithi016.stdout: "snap_id": "head", 2024-03-31T15:08:58.816 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/subdir/file_to_be_damaged" 2024-03-31T15:08:58.816 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-31T15:08:58.816 INFO:teuthology.orchestra.run.smithi016.stdout:] 2024-03-31T15:08:58.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.821+0000 7f1c21c00640 1 -- 172.21.15.16:0/1237348100 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f1bf8068380 msgr2=0x7f1bf806a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:58.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.821+0000 7f1c21c00640 1 --2- 172.21.15.16:0/1237348100 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f1bf8068380 0x7f1bf806a840 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f1c0c01c530 tx=0x7f1c0c0a7000 comp rx=0 tx=0).stop 2024-03-31T15:08:58.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.821+0000 7f1c21c00640 1 -- 172.21.15.16:0/1237348100 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c1c143c30 msgr2=0x7f1c1c157240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:58.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.821+0000 7f1c21c00640 1 --2- 172.21.15.16:0/1237348100 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c1c143c30 0x7f1c1c157240 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7f1c1400cc90 tx=0x7f1c1400a650 comp rx=0 tx=0).stop 2024-03-31T15:08:58.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.821+0000 7f1c21c00640 1 -- 172.21.15.16:0/1237348100 shutdown_connections 2024-03-31T15:08:58.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.821+0000 7f1c21c00640 1 --2- 172.21.15.16:0/1237348100 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f1bf8068380 0x7f1bf806a840 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:58.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.821+0000 7f1c21c00640 1 --2- 172.21.15.16:0/1237348100 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c1c143c30 0x7f1c1c157240 unknown :-1 s=CLOSED pgs=944 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:58.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.821+0000 7f1c21c00640 1 --2- 172.21.15.16:0/1237348100 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f1c1c141590 0x7f1c1c13e660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:58.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.821+0000 7f1c21c00640 1 --2- 172.21.15.16:0/1237348100 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1c1c1411b0 0x7f1c1c156430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:58.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.821+0000 7f1c21c00640 1 -- 172.21.15.16:0/1237348100 >> 172.21.15.16:0/1237348100 conn(0x7f1c1c067930 msgr2=0x7f1c1c0b2c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:08:58.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.821+0000 7f1c21c00640 1 -- 172.21.15.16:0/1237348100 shutdown_connections 2024-03-31T15:08:58.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.821+0000 7f1c21c00640 1 -- 172.21.15.16:0/1237348100 wait complete. 2024-03-31T15:08:58.831 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-03-31T15:08:58.969 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.977+0000 7fb8bc200640 1 Processor -- start 2024-03-31T15:08:58.969 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.977+0000 7fb8bc200640 1 -- start start 2024-03-31T15:08:58.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.977+0000 7fb8bc200640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb8b406d4d0 0x7fb8b406d930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:58.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.977+0000 7fb8bc200640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb8b406de70 con 0x7fb8b41567e0 2024-03-31T15:08:58.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.977+0000 7fb8bc200640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb8b415c7e0 con 0x7fb8b4071550 2024-03-31T15:08:58.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.977+0000 7fb8bc200640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fb8b40736b0 con 0x7fb8b406d4d0 2024-03-31T15:08:58.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.977+0000 7fb8b9800640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fb8b41567e0 0x7fb8b4072c80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:53490/0 (socket says 172.21.15.16:53490) 2024-03-31T15:08:58.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.977+0000 7fb8b9800640 1 -- 172.21.15.16:0/346160371 learned_addr learned my addr 172.21.15.16:0/346160371 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:08:58.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.981+0000 7fb8b8e00640 1 --2- 172.21.15.16:0/346160371 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb8b406d4d0 0x7fb8b406d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:58.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.981+0000 7fb8b3e00640 1 -- 172.21.15.16:0/346160371 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2514599638 0 0) 0x7fb8b406de70 con 0x7fb8b41567e0 2024-03-31T15:08:58.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.981+0000 7fb8b3e00640 1 -- 172.21.15.16:0/346160371 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb89c003530 con 0x7fb8b41567e0 2024-03-31T15:08:58.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.981+0000 7fb8b3e00640 1 -- 172.21.15.16:0/346160371 <== mon.1 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 135879276 0 0) 0x7fb8b415c7e0 con 0x7fb8b4071550 2024-03-31T15:08:58.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.981+0000 7fb8b3e00640 1 -- 172.21.15.16:0/346160371 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb8b406de70 con 0x7fb8b4071550 2024-03-31T15:08:58.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.981+0000 7fb8b8e00640 1 -- 172.21.15.16:0/346160371 >> v1:172.21.15.89:6789/0 conn(0x7fb8b4071550 legacy=0x7fb8b40719a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:58.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.981+0000 7fb8b8e00640 1 -- 172.21.15.16:0/346160371 >> v1:172.21.15.16:6789/0 conn(0x7fb8b41567e0 legacy=0x7fb8b4072c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:58.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.981+0000 7fb8b8e00640 1 -- 172.21.15.16:0/346160371 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb8b415caa0 con 0x7fb8b406d4d0 2024-03-31T15:08:58.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.985+0000 7fb8b8e00640 1 --2- 172.21.15.16:0/346160371 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb8b406d4d0 0x7fb8b406d930 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fb8a00097f0 tx=0x7fb8a001c4e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=82794921477d57ea server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:08:58.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.985+0000 7fb8b3e00640 1 -- 172.21.15.16:0/346160371 <== mon.2 v2:172.21.15.89:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8a00b0070 con 0x7fb8b406d4d0 2024-03-31T15:08:58.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.985+0000 7fb8b3e00640 1 -- 172.21.15.16:0/346160371 <== mon.2 v2:172.21.15.89:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb8a00ab600 con 0x7fb8b406d4d0 2024-03-31T15:08:58.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.985+0000 7fb8b3e00640 1 -- 172.21.15.16:0/346160371 <== mon.2 v2:172.21.15.89:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8a00ab9d0 con 0x7fb8b406d4d0 2024-03-31T15:08:58.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.985+0000 7fb8bc200640 1 -- 172.21.15.16:0/346160371 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb8b406d4d0 msgr2=0x7fb8b406d930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:58.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.985+0000 7fb8bc200640 1 --2- 172.21.15.16:0/346160371 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb8b406d4d0 0x7fb8b406d930 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fb8a00097f0 tx=0x7fb8a001c4e0 comp rx=0 tx=0).stop 2024-03-31T15:08:58.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.985+0000 7fb8bc200640 1 -- 172.21.15.16:0/346160371 shutdown_connections 2024-03-31T15:08:58.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.985+0000 7fb8bc200640 1 --2- 172.21.15.16:0/346160371 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb8b406d4d0 0x7fb8b406d930 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:58.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.985+0000 7fb8bc200640 1 -- 172.21.15.16:0/346160371 >> 172.21.15.16:0/346160371 conn(0x7fb8b406ee40 msgr2=0x7fb8b406b620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:08:58.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.985+0000 7fb8bc200640 1 -- 172.21.15.16:0/346160371 shutdown_connections 2024-03-31T15:08:58.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.985+0000 7fb8bc200640 1 -- 172.21.15.16:0/346160371 wait complete. 2024-03-31T15:08:58.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.989+0000 7fb8bc200640 1 Processor -- start 2024-03-31T15:08:58.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.989+0000 7fb8bc200640 1 -- start start 2024-03-31T15:08:58.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.989+0000 7fb8bc200640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb8b406d4d0 0x7fb8b414f420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:58.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.989+0000 7fb8bc200640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb8b4071550 0x7fb8b40ae060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:58.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.989+0000 7fb8bc200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb8b41567e0 0x7fb8b415a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:58.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.989+0000 7fb8bc200640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fb8b40afbb0 con 0x7fb8b41567e0 2024-03-31T15:08:58.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.989+0000 7fb8bc200640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fb8b40afa30 con 0x7fb8b406d4d0 2024-03-31T15:08:58.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.989+0000 7fb8bc200640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fb8b415d560 con 0x7fb8b4071550 2024-03-31T15:08:58.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.989+0000 7fb8ba200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb8b41567e0 0x7fb8b415a400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:58.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.989+0000 7fb8ba200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb8b41567e0 0x7fb8b415a400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42918/0 (socket says 172.21.15.16:42918) 2024-03-31T15:08:58.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.989+0000 7fb8ba200640 1 -- 172.21.15.16:0/2468310400 learned_addr learned my addr 172.21.15.16:0/2468310400 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:08:58.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.989+0000 7fb8b9800640 1 --2- 172.21.15.16:0/2468310400 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb8b406d4d0 0x7fb8b414f420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:58.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.989+0000 7fb8b8e00640 1 --2- 172.21.15.16:0/2468310400 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb8b4071550 0x7fb8b40ae060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:58.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.989+0000 7fb8ba200640 1 -- 172.21.15.16:0/2468310400 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb8b4071550 msgr2=0x7fb8b40ae060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:58.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.989+0000 7fb8ba200640 1 --2- 172.21.15.16:0/2468310400 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb8b4071550 0x7fb8b40ae060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:58.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.989+0000 7fb8ba200640 1 -- 172.21.15.16:0/2468310400 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb8b406d4d0 msgr2=0x7fb8b414f420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:58.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.989+0000 7fb8ba200640 1 --2- 172.21.15.16:0/2468310400 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb8b406d4d0 0x7fb8b414f420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:58.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.989+0000 7fb8ba200640 1 -- 172.21.15.16:0/2468310400 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb8b4150d30 con 0x7fb8b41567e0 2024-03-31T15:08:58.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.989+0000 7fb8ba200640 1 --2- 172.21.15.16:0/2468310400 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb8b41567e0 0x7fb8b415a400 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7fb8ac00c7c0 tx=0x7fb8ac00cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:08:58.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.989+0000 7fb8b1600640 1 -- 172.21.15.16:0/2468310400 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8ac003b80 con 0x7fb8b41567e0 2024-03-31T15:08:58.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.989+0000 7fb8b1600640 1 -- 172.21.15.16:0/2468310400 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb8ac003d20 con 0x7fb8b41567e0 2024-03-31T15:08:58.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.989+0000 7fb8b1600640 1 -- 172.21.15.16:0/2468310400 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8ac00aa50 con 0x7fb8b41567e0 2024-03-31T15:08:58.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.989+0000 7fb8bc200640 1 -- 172.21.15.16:0/2468310400 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb8b41379b0 con 0x7fb8b41567e0 2024-03-31T15:08:58.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.989+0000 7fb8bc200640 1 -- 172.21.15.16:0/2468310400 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb8b4137f70 con 0x7fb8b41567e0 2024-03-31T15:08:58.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.989+0000 7fb8bc200640 1 -- 172.21.15.16:0/2468310400 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb8b4069950 con 0x7fb8b41567e0 2024-03-31T15:08:58.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.993+0000 7fb8b1600640 1 -- 172.21.15.16:0/2468310400 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fb8ac00abf0 con 0x7fb8b41567e0 2024-03-31T15:08:58.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.993+0000 7fb8b1600640 1 --2- 172.21.15.16:0/2468310400 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fb8940683b0 0x7fb89406a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:58.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.993+0000 7fb8b1600640 1 -- 172.21.15.16:0/2468310400 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(70..70 src has 1..70) ==== 5882+0+0 (secure 0 0 0) 0x7fb8ac086880 con 0x7fb8b41567e0 2024-03-31T15:08:58.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.993+0000 7fb8b9800640 1 --2- 172.21.15.16:0/2468310400 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fb8940683b0 0x7fb89406a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:58.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.993+0000 7fb8b9800640 1 --2- 172.21.15.16:0/2468310400 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fb8940683b0 0x7fb89406a870 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7fb8a80011f0 tx=0x7fb8a8007d40 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:08:58.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:58.997+0000 7fb8b1600640 1 -- 172.21.15.16:0/2468310400 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fb8ac056730 con 0x7fb8b41567e0 2024-03-31T15:08:59.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.197+0000 7fb8bc200640 1 -- 172.21.15.16:0/2468310400 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7fb8b406e470 con 0x7fb8b41567e0 2024-03-31T15:08:59.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.197+0000 7fb8b1600640 1 -- 172.21.15.16:0/2468310400 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+164 (secure 0 0 0) 0x7fb8ac05aa30 con 0x7fb8b41567e0 2024-03-31T15:08:59.188 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:08:59.188 INFO:teuthology.orchestra.run.smithi016.stdout:{"status":"HEALTH_ERR","checks":{"MDS_DAMAGE":{"severity":"HEALTH_ERR","summary":{"message":"1 MDSs report damaged metadata","count":1},"muted":false}},"mutes":[]} 2024-03-31T15:08:59.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.197+0000 7fb8bc200640 1 -- 172.21.15.16:0/2468310400 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fb8940683b0 msgr2=0x7fb89406a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:59.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.197+0000 7fb8bc200640 1 --2- 172.21.15.16:0/2468310400 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fb8940683b0 0x7fb89406a870 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7fb8a80011f0 tx=0x7fb8a8007d40 comp rx=0 tx=0).stop 2024-03-31T15:08:59.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.197+0000 7fb8bc200640 1 -- 172.21.15.16:0/2468310400 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb8b41567e0 msgr2=0x7fb8b415a400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:59.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.197+0000 7fb8bc200640 1 --2- 172.21.15.16:0/2468310400 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb8b41567e0 0x7fb8b415a400 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7fb8ac00c7c0 tx=0x7fb8ac00cc80 comp rx=0 tx=0).stop 2024-03-31T15:08:59.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.197+0000 7fb8bc200640 1 -- 172.21.15.16:0/2468310400 shutdown_connections 2024-03-31T15:08:59.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.197+0000 7fb8bc200640 1 --2- 172.21.15.16:0/2468310400 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fb8940683b0 0x7fb89406a870 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:59.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.197+0000 7fb8bc200640 1 --2- 172.21.15.16:0/2468310400 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb8b41567e0 0x7fb8b415a400 unknown :-1 s=CLOSED pgs=950 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:59.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.197+0000 7fb8bc200640 1 --2- 172.21.15.16:0/2468310400 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb8b4071550 0x7fb8b40ae060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:59.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.197+0000 7fb8bc200640 1 --2- 172.21.15.16:0/2468310400 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb8b406d4d0 0x7fb8b414f420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:59.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.197+0000 7fb8bc200640 1 -- 172.21.15.16:0/2468310400 >> 172.21.15.16:0/2468310400 conn(0x7fb8b406ee40 msgr2=0x7fb8b40b0300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:08:59.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.197+0000 7fb8bc200640 1 -- 172.21.15.16:0/2468310400 shutdown_connections 2024-03-31T15:08:59.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.197+0000 7fb8bc200640 1 -- 172.21.15.16:0/2468310400 wait complete. 2024-03-31T15:08:59.217 DEBUG:tasks.ceph_test_case:wait_until_true: success in 0s and 0 retries 2024-03-31T15:08:59.218 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:08:59.218 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.5:0 scrub start /subdir repair,recursive 2024-03-31T15:08:59.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.369+0000 7f2e55400640 1 Processor -- start 2024-03-31T15:08:59.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.369+0000 7f2e55400640 1 -- start start 2024-03-31T15:08:59.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.373+0000 7f2e55400640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f2e50147230 0x7f2e500b3dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:59.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.373+0000 7f2e55400640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f2e500d1040 0x7f2e500cb4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:59.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.373+0000 7f2e55400640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2e500d14d0 con 0x7f2e500b4300 2024-03-31T15:08:59.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.373+0000 7f2e55400640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f2e500b4760 con 0x7f2e50147230 2024-03-31T15:08:59.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.373+0000 7f2e55400640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f2e500d2c30 con 0x7f2e500d1040 2024-03-31T15:08:59.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.373+0000 7f2e4e800640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f2e50147230 0x7f2e500b3dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:59.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.373+0000 7f2e4e800640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f2e50147230 0x7f2e500b3dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42928/0 (socket says 172.21.15.16:42928) 2024-03-31T15:08:59.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.373+0000 7f2e4e800640 1 -- 172.21.15.16:0/61801363 learned_addr learned my addr 172.21.15.16:0/61801363 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:08:59.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.373+0000 7f2e4f200640 1 --2- 172.21.15.16:0/61801363 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f2e500d1040 0x7f2e500cb4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:59.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.373+0000 7f2e4de00640 1 --1- 172.21.15.16:0/61801363 >> v1:172.21.15.89:6789/0 conn(0x7f2e500b4300 0x7f2e500d0930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.16:51284/0 (socket says 172.21.15.16:51284) 2024-03-31T15:08:59.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.373+0000 7f2e4e800640 1 -- 172.21.15.16:0/61801363 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f2e500d1040 msgr2=0x7f2e500cb4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:08:59.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.373+0000 7f2e4e800640 1 --2- 172.21.15.16:0/61801363 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f2e500d1040 0x7f2e500cb4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:59.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.373+0000 7f2e4e800640 1 -- 172.21.15.16:0/61801363 >> v1:172.21.15.89:6789/0 conn(0x7f2e500b4300 legacy=0x7f2e500d0930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:59.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.373+0000 7f2e4e800640 1 -- 172.21.15.16:0/61801363 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e500d2020 con 0x7f2e50147230 2024-03-31T15:08:59.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.377+0000 7f2e4e800640 1 --2- 172.21.15.16:0/61801363 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2e50147230 0x7f2e500b3dc0 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7f2e40009870 tx=0x7f2e4001c560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92778d5603e43ea4 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:08:59.368 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.377+0000 7f2e4d400640 1 -- 172.21.15.16:0/61801363 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e400ab650 con 0x7f2e50147230 2024-03-31T15:08:59.368 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.377+0000 7f2e4d400640 1 -- 172.21.15.16:0/61801363 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2e400b0070 con 0x7f2e50147230 2024-03-31T15:08:59.368 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.377+0000 7f2e4d400640 1 -- 172.21.15.16:0/61801363 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e400aba10 con 0x7f2e50147230 2024-03-31T15:08:59.369 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.377+0000 7f2e55400640 1 -- 172.21.15.16:0/61801363 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2e50147230 msgr2=0x7f2e500b3dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:59.369 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.377+0000 7f2e55400640 1 --2- 172.21.15.16:0/61801363 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2e50147230 0x7f2e500b3dc0 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7f2e40009870 tx=0x7f2e4001c560 comp rx=0 tx=0).stop 2024-03-31T15:08:59.369 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.377+0000 7f2e55400640 1 -- 172.21.15.16:0/61801363 shutdown_connections 2024-03-31T15:08:59.369 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.377+0000 7f2e55400640 1 --2- 172.21.15.16:0/61801363 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f2e500d1040 0x7f2e500cb4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:59.369 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.377+0000 7f2e55400640 1 --2- 172.21.15.16:0/61801363 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2e50147230 0x7f2e500b3dc0 unknown :-1 s=CLOSED pgs=951 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:59.369 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.377+0000 7f2e55400640 1 -- 172.21.15.16:0/61801363 >> 172.21.15.16:0/61801363 conn(0x7f2e5006b7f0 msgr2=0x7f2e50067330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:08:59.369 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.377+0000 7f2e55400640 1 -- 172.21.15.16:0/61801363 shutdown_connections 2024-03-31T15:08:59.369 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.377+0000 7f2e55400640 1 -- 172.21.15.16:0/61801363 wait complete. 2024-03-31T15:08:59.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.377+0000 7f2e55400640 1 Processor -- start 2024-03-31T15:08:59.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.377+0000 7f2e55400640 1 -- start start 2024-03-31T15:08:59.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.381+0000 7f2e55400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2e500b4300 0x7f2e500b5900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:59.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.381+0000 7f2e55400640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2e500d1040 0x7f2e5015a9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:59.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.381+0000 7f2e55400640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f2e50147230 0x7f2e50135cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:59.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.381+0000 7f2e55400640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f2e500cc200 con 0x7f2e500b4300 2024-03-31T15:08:59.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.381+0000 7f2e55400640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f2e5006cfd0 con 0x7f2e500d1040 2024-03-31T15:08:59.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.381+0000 7f2e55400640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f2e500cc380 con 0x7f2e50147230 2024-03-31T15:08:59.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.381+0000 7f2e4e800640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2e500b4300 0x7f2e500b5900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:59.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.381+0000 7f2e4e800640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2e500b4300 0x7f2e500b5900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42932/0 (socket says 172.21.15.16:42932) 2024-03-31T15:08:59.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.381+0000 7f2e4e800640 1 -- 172.21.15.16:0/2771320264 learned_addr learned my addr 172.21.15.16:0/2771320264 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:08:59.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.381+0000 7f2e4f200640 1 --2- 172.21.15.16:0/2771320264 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f2e50147230 0x7f2e50135cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:59.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.381+0000 7f2e4de00640 1 --2- 172.21.15.16:0/2771320264 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2e500d1040 0x7f2e5015a9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:59.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.381+0000 7f2e4e800640 1 -- 172.21.15.16:0/2771320264 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f2e50147230 msgr2=0x7f2e50135cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:59.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.381+0000 7f2e4e800640 1 --2- 172.21.15.16:0/2771320264 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f2e50147230 0x7f2e50135cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:59.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.381+0000 7f2e4e800640 1 -- 172.21.15.16:0/2771320264 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2e500d1040 msgr2=0x7f2e5015a9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:59.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.381+0000 7f2e4e800640 1 --2- 172.21.15.16:0/2771320264 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2e500d1040 0x7f2e5015a9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:59.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.381+0000 7f2e4e800640 1 -- 172.21.15.16:0/2771320264 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e5015b0a0 con 0x7f2e500b4300 2024-03-31T15:08:59.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.381+0000 7f2e4f200640 1 --2- 172.21.15.16:0/2771320264 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f2e50147230 0x7f2e50135cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:08:59.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.381+0000 7f2e4e800640 1 --2- 172.21.15.16:0/2771320264 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2e500b4300 0x7f2e500b5900 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 crypto rx=0x7f2e400ad040 tx=0x7f2e4001ada0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:08:59.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.381+0000 7f2e46a00640 1 -- 172.21.15.16:0/2771320264 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e400be5f0 con 0x7f2e500b4300 2024-03-31T15:08:59.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.381+0000 7f2e46a00640 1 -- 172.21.15.16:0/2771320264 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2e400be7b0 con 0x7f2e500b4300 2024-03-31T15:08:59.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.381+0000 7f2e46a00640 1 -- 172.21.15.16:0/2771320264 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e400beba0 con 0x7f2e500b4300 2024-03-31T15:08:59.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.381+0000 7f2e55400640 1 -- 172.21.15.16:0/2771320264 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2e5015b380 con 0x7f2e500b4300 2024-03-31T15:08:59.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.381+0000 7f2e55400640 1 -- 172.21.15.16:0/2771320264 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2e5015b940 con 0x7f2e500b4300 2024-03-31T15:08:59.375 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.385+0000 7f2e46a00640 1 -- 172.21.15.16:0/2771320264 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f2e400b43f0 con 0x7f2e500b4300 2024-03-31T15:08:59.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.385+0000 7f2e58810000 1 Processor -- start 2024-03-31T15:08:59.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.385+0000 7f2e58810000 1 -- start start 2024-03-31T15:08:59.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.385+0000 7f2e46a00640 1 --2- 172.21.15.16:0/2771320264 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f2e30068380 0x7f2e3006a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:59.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.385+0000 7f2e46a00640 1 -- 172.21.15.16:0/2771320264 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(70..70 src has 1..70) ==== 5882+0+0 (secure 0 0 0) 0x7f2e40121a00 con 0x7f2e500b4300 2024-03-31T15:08:59.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.385+0000 7f2e4f200640 1 --2- 172.21.15.16:0/2771320264 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f2e30068380 0x7f2e3006a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:59.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.385+0000 7f2e58810000 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x558b25451eb0 0x558b25452290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:59.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.385+0000 7f2e58810000 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558b254527d0 0x558b2540af20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:59.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.385+0000 7f2e58810000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558b2540b460 0x558b2540b840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:59.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.385+0000 7f2e58810000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x558b25255910 con 0x558b2540b460 2024-03-31T15:08:59.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.385+0000 7f2e58810000 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x558b251f2cd0 con 0x558b254527d0 2024-03-31T15:08:59.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.385+0000 7f2e58810000 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x558b252518c0 con 0x558b25451eb0 2024-03-31T15:08:59.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.385+0000 7f2e4e800640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x558b25451eb0 0x558b25452290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:59.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.385+0000 7f2e4f200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558b2540b460 0x558b2540b840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:59.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.385+0000 7f2e4e800640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x558b25451eb0 0x558b25452290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3301/0 says I am v2:172.21.15.16:44394/0 (socket says 172.21.15.16:44394) 2024-03-31T15:08:59.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.385+0000 7f2e4e800640 1 -- 172.21.15.16:0/3271439510 learned_addr learned my addr 172.21.15.16:0/3271439510 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:08:59.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.385+0000 7f2e4de00640 1 --2- 172.21.15.16:0/3271439510 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558b254527d0 0x558b2540af20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:59.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.385+0000 7f2e4f200640 1 -- 172.21.15.16:0/3271439510 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x558b25451eb0 msgr2=0x558b25452290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:08:59.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.385+0000 7f2e4f200640 1 --2- 172.21.15.16:0/3271439510 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x558b25451eb0 0x558b25452290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:59.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.385+0000 7f2e4f200640 1 -- 172.21.15.16:0/3271439510 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558b254527d0 msgr2=0x558b2540af20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:08:59.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.385+0000 7f2e4f200640 1 --2- 172.21.15.16:0/3271439510 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558b254527d0 0x558b2540af20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:59.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.385+0000 7f2e4f200640 1 -- 172.21.15.16:0/3271439510 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558b253ebc00 con 0x558b2540b460 2024-03-31T15:08:59.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.385+0000 7f2e4e800640 1 --2- 172.21.15.16:0/3271439510 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x558b25451eb0 0x558b25452290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:08:59.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.385+0000 7f2e4de00640 1 --2- 172.21.15.16:0/3271439510 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558b254527d0 0x558b2540af20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:08:59.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.385+0000 7f2e4f200640 1 --2- 172.21.15.16:0/2771320264 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f2e30068380 0x7f2e3006a840 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f2e480077d0 tx=0x7f2e48007bd0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:08:59.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.385+0000 7f2e4f200640 1 --2- 172.21.15.16:0/3271439510 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558b2540b460 0x558b2540b840 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7f2e480186d0 tx=0x7f2e4801f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c88236da97c4930 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:08:59.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.385+0000 7f2e3b400640 1 -- 172.21.15.16:0/3271439510 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e4802a070 con 0x558b2540b460 2024-03-31T15:08:59.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.385+0000 7f2e3b400640 1 -- 172.21.15.16:0/3271439510 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2e4801bc60 con 0x558b2540b460 2024-03-31T15:08:59.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.385+0000 7f2e3b400640 1 -- 172.21.15.16:0/3271439510 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e48025660 con 0x558b2540b460 2024-03-31T15:08:59.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.389+0000 7f2e58810000 1 -- 172.21.15.16:0/3271439510 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558b2540b460 msgr2=0x558b2540b840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:59.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.389+0000 7f2e58810000 1 --2- 172.21.15.16:0/3271439510 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558b2540b460 0x558b2540b840 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7f2e480186d0 tx=0x7f2e4801f000 comp rx=0 tx=0).stop 2024-03-31T15:08:59.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.389+0000 7f2e58810000 1 -- 172.21.15.16:0/3271439510 shutdown_connections 2024-03-31T15:08:59.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.389+0000 7f2e58810000 1 --2- 172.21.15.16:0/3271439510 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558b2540b460 0x558b2540b840 unknown :-1 s=CLOSED pgs=953 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:59.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.389+0000 7f2e58810000 1 --2- 172.21.15.16:0/3271439510 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558b254527d0 0x558b2540af20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:59.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.389+0000 7f2e58810000 1 --2- 172.21.15.16:0/3271439510 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x558b25451eb0 0x558b25452290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:59.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.389+0000 7f2e58810000 1 -- 172.21.15.16:0/3271439510 >> 172.21.15.16:0/3271439510 conn(0x558b25407410 msgr2=0x558b25409800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:08:59.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.389+0000 7f2e58810000 1 -- 172.21.15.16:0/3271439510 shutdown_connections 2024-03-31T15:08:59.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.389+0000 7f2e58810000 1 -- 172.21.15.16:0/3271439510 wait complete. 2024-03-31T15:08:59.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.389+0000 7f2e58810000 1 -- start start 2024-03-31T15:08:59.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.389+0000 7f2e58810000 1 Processor -- start 2024-03-31T15:08:59.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.389+0000 7f2e58810000 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558b2540b460 0x558b254372d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:59.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.389+0000 7f2e58810000 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x558b25451eb0 0x558b25437810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:59.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.389+0000 7f2e58810000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558b254527d0 0x558b25437d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:59.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.389+0000 7f2e58810000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x558b25213710 con 0x558b254527d0 2024-03-31T15:08:59.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.389+0000 7f2e58810000 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x558b25259f40 con 0x558b2540b460 2024-03-31T15:08:59.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.389+0000 7f2e58810000 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x558b25312940 con 0x558b25451eb0 2024-03-31T15:08:59.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.389+0000 7f2e58810000 10 client.0 _ll_register_callbacks cb 0x558b253f39c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T15:08:59.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.389+0000 7f2e4f200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558b254527d0 0x558b25437d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:59.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.389+0000 7f2e4f200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558b254527d0 0x558b25437d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42952/0 (socket says 172.21.15.16:42952) 2024-03-31T15:08:59.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.389+0000 7f2e4f200640 1 -- 172.21.15.16:0/3138503754 learned_addr learned my addr 172.21.15.16:0/3138503754 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:08:59.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.389+0000 7f2e4e800640 1 --2- 172.21.15.16:0/3138503754 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558b2540b460 0x558b254372d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:59.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.389+0000 7f2e4de00640 1 --2- 172.21.15.16:0/3138503754 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x558b25451eb0 0x558b25437810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:59.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.389+0000 7f2e4f200640 1 -- 172.21.15.16:0/3138503754 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x558b25451eb0 msgr2=0x558b25437810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:08:59.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.389+0000 7f2e4f200640 1 --2- 172.21.15.16:0/3138503754 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x558b25451eb0 0x558b25437810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:59.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.389+0000 7f2e4f200640 1 -- 172.21.15.16:0/3138503754 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558b2540b460 msgr2=0x558b254372d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:08:59.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.389+0000 7f2e4f200640 1 --2- 172.21.15.16:0/3138503754 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558b2540b460 0x558b254372d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:59.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.389+0000 7f2e4f200640 1 -- 172.21.15.16:0/3138503754 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558b253022b0 con 0x558b254527d0 2024-03-31T15:08:59.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.389+0000 7f2e4de00640 1 --2- 172.21.15.16:0/3138503754 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x558b25451eb0 0x558b25437810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:08:59.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.389+0000 7f2e4f200640 1 --2- 172.21.15.16:0/3138503754 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558b254527d0 0x558b25437d50 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x558b24f189b0 tx=0x7f2e48004ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe633a48cf860b2d server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:08:59.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.389+0000 7f2e2fe00640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-31T15:08:59.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.389+0000 7f2e2fe00640 1 -- 172.21.15.16:0/3138503754 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e48034020 con 0x558b254527d0 2024-03-31T15:08:59.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.389+0000 7f2e58810000 1 -- 172.21.15.16:0/3138503754 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2e4801bc60 con 0x558b254527d0 2024-03-31T15:08:59.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.393+0000 7f2e2fe00640 1 -- 172.21.15.16:0/3138503754 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(70..70 src has 1..70) ==== 5882+0+0 (secure 0 0 0) 0x7f2e48025710 con 0x558b254527d0 2024-03-31T15:08:59.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.393+0000 7f2e2fe00640 1 -- 172.21.15.16:0/3138503754 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=51) ==== 24+0+0 (secure 0 0 0) 0x7f2e4803e050 con 0x558b254527d0 2024-03-31T15:08:59.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.393+0000 7f2e58810000 10 client.7191 fetch_fsmap learned FSMap version 51 2024-03-31T15:08:59.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.393+0000 7f2e58810000 1 -- 172.21.15.16:0/3138503754 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=51,monmap=2+}) -- 0x558b252fe260 con 0x558b254527d0 2024-03-31T15:08:59.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.393+0000 7f2e2fe00640 1 -- 172.21.15.16:0/3138503754 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2e4801bc60 con 0x558b254527d0 2024-03-31T15:08:59.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.393+0000 7f2e2fe00640 1 -- 172.21.15.16:0/3138503754 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 51) ==== 3494+0+0 (secure 0 0 0) 0x7f2e48041020 con 0x558b254527d0 2024-03-31T15:08:59.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.393+0000 7f2e58810000 10 client.7191 fetch_fsmap finished waiting for FSMap version 51 2024-03-31T15:08:59.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.393+0000 7f2e58810000 10 client.7191 resolve_mds: resolved 5:0 to role '5:0' aka daemon mds.a 2024-03-31T15:08:59.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.393+0000 7f2e58810000 20 client.7191 populate_metadata read hostname 'smithi016' 2024-03-31T15:08:59.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.393+0000 7f2e58810000 1 --2- 172.21.15.16:0/3138503754 >> [v2:172.21.15.16:6834/2645389320,v1:172.21.15.16:6835/2645389320] conn(0x558b254385f0 0x558b25464430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:59.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.393+0000 7f2e58810000 4 client.7191 mds_command: new command op to 6829 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-31T15:08:59.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.393+0000 7f2e58810000 1 -- 172.21.15.16:0/3138503754 --> [v2:172.21.15.16:6834/2645389320,v1:172.21.15.16:6835/2645389320] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558b251f0cc0 con 0x558b254385f0 2024-03-31T15:08:59.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.393+0000 7f2e4e800640 1 --2- 172.21.15.16:0/3138503754 >> [v2:172.21.15.16:6834/2645389320,v1:172.21.15.16:6835/2645389320] conn(0x558b254385f0 0x558b25464430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:59.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.393+0000 7f2e4e800640 1 --2- 172.21.15.16:0/3138503754 >> [v2:172.21.15.16:6834/2645389320,v1:172.21.15.16:6835/2645389320] conn(0x558b254385f0 0x558b25464430 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a048a66302277338 server_cookie=51cf8aedda174d50 in_seq=0 out_seq=0 2024-03-31T15:08:59.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.393+0000 7f2e2fe00640 10 client.7191 ms_handle_connect on v2:172.21.15.16:6834/2645389320 2024-03-31T15:08:59.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.393+0000 7f2e2fe00640 1 -- 172.21.15.16:0/3138503754 <== mds.0 v2:172.21.15.16:6834/2645389320 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x558b251f0cc0 con 0x558b254385f0 2024-03-31T15:08:59.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.393+0000 7f2e2fe00640 10 client.7191 handle_command_reply: tid=0 2024-03-31T15:08:59.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.393+0000 7f2e58810000 1 client.7191 shutdown 2024-03-31T15:08:59.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.393+0000 7f2e58810000 1 -- 172.21.15.16:0/3138503754 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558b254527d0 msgr2=0x558b25437d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:59.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.393+0000 7f2e58810000 1 --2- 172.21.15.16:0/3138503754 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558b254527d0 0x558b25437d50 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x558b24f189b0 tx=0x7f2e48004ac0 comp rx=0 tx=0).stop 2024-03-31T15:08:59.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.393+0000 7f2e58810000 1 -- 172.21.15.16:0/3138503754 shutdown_connections 2024-03-31T15:08:59.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.393+0000 7f2e58810000 1 --2- 172.21.15.16:0/3138503754 >> [v2:172.21.15.16:6834/2645389320,v1:172.21.15.16:6835/2645389320] conn(0x558b254385f0 0x558b25464430 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:59.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.393+0000 7f2e58810000 1 --2- 172.21.15.16:0/3138503754 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558b254527d0 0x558b25437d50 unknown :-1 s=CLOSED pgs=954 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:59.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.393+0000 7f2e58810000 1 --2- 172.21.15.16:0/3138503754 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x558b25451eb0 0x558b25437810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:59.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.393+0000 7f2e58810000 1 --2- 172.21.15.16:0/3138503754 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558b2540b460 0x558b254372d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:59.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.393+0000 7f2e58810000 1 -- 172.21.15.16:0/3138503754 >> 172.21.15.16:0/3138503754 conn(0x558b25407410 msgr2=0x558b2540a090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:08:59.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.393+0000 7f2e2fe00640 0 client.7191 ms_handle_reset on v2:172.21.15.16:6834/2645389320 2024-03-31T15:08:59.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.393+0000 7f2e58810000 1 -- 172.21.15.16:0/3138503754 shutdown_connections 2024-03-31T15:08:59.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.393+0000 7f2e58810000 1 -- 172.21.15.16:0/3138503754 wait complete. 2024-03-31T15:08:59.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.393+0000 7f2e58810000 20 client.7191 trim_cache size 0 max 16384 2024-03-31T15:08:59.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.409+0000 7f2e58810000 1 Processor -- start 2024-03-31T15:08:59.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.413+0000 7f2e58810000 1 -- start start 2024-03-31T15:08:59.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.413+0000 7f2e58810000 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558b2540b460 0x558b25467460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:59.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.413+0000 7f2e58810000 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x558b25451eb0 0x558b254679a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:59.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.413+0000 7f2e58810000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558b254527d0 0x558b25450ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:59.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.413+0000 7f2e58810000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f2e50202930 con 0x558b254527d0 2024-03-31T15:08:59.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.413+0000 7f2e58810000 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f2e500b28d0 con 0x558b2540b460 2024-03-31T15:08:59.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.413+0000 7f2e58810000 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f2e500b2f20 con 0x558b25451eb0 2024-03-31T15:08:59.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.413+0000 7f2e4f200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558b254527d0 0x558b25450ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:59.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.413+0000 7f2e4f200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558b254527d0 0x558b25450ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42958/0 (socket says 172.21.15.16:42958) 2024-03-31T15:08:59.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.413+0000 7f2e4f200640 1 -- 172.21.15.16:0/1363209270 learned_addr learned my addr 172.21.15.16:0/1363209270 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:08:59.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.413+0000 7f2e4de00640 1 --2- 172.21.15.16:0/1363209270 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x558b25451eb0 0x558b254679a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:59.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.413+0000 7f2e4e800640 1 --2- 172.21.15.16:0/1363209270 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558b2540b460 0x558b25467460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:59.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.413+0000 7f2e4f200640 1 -- 172.21.15.16:0/1363209270 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x558b25451eb0 msgr2=0x558b254679a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:08:59.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.413+0000 7f2e4f200640 1 --2- 172.21.15.16:0/1363209270 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x558b25451eb0 0x558b254679a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:59.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.413+0000 7f2e4f200640 1 -- 172.21.15.16:0/1363209270 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558b2540b460 msgr2=0x558b25467460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:08:59.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.413+0000 7f2e4f200640 1 --2- 172.21.15.16:0/1363209270 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558b2540b460 0x558b25467460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:59.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.413+0000 7f2e4f200640 1 -- 172.21.15.16:0/1363209270 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558b2540eaf0 con 0x558b254527d0 2024-03-31T15:08:59.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.413+0000 7f2e4e800640 1 --2- 172.21.15.16:0/1363209270 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558b2540b460 0x558b25467460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:08:59.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.413+0000 7f2e4f200640 1 --2- 172.21.15.16:0/1363209270 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558b254527d0 0x558b25450ae0 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7f2e48008010 tx=0x7f2e4802e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d651fb7b67665087 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:08:59.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.413+0000 7f2e3aa00640 1 -- 172.21.15.16:0/1363209270 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e48034030 con 0x558b254527d0 2024-03-31T15:08:59.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.413+0000 7f2e3aa00640 1 -- 172.21.15.16:0/1363209270 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2e4801bc80 con 0x558b254527d0 2024-03-31T15:08:59.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.413+0000 7f2e3aa00640 1 -- 172.21.15.16:0/1363209270 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e4802a040 con 0x558b254527d0 2024-03-31T15:08:59.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.413+0000 7f2e58810000 1 -- 172.21.15.16:0/1363209270 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558b254527d0 msgr2=0x558b25450ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:59.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.413+0000 7f2e58810000 1 --2- 172.21.15.16:0/1363209270 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558b254527d0 0x558b25450ae0 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7f2e48008010 tx=0x7f2e4802e000 comp rx=0 tx=0).stop 2024-03-31T15:08:59.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.413+0000 7f2e58810000 1 -- 172.21.15.16:0/1363209270 shutdown_connections 2024-03-31T15:08:59.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.413+0000 7f2e58810000 1 --2- 172.21.15.16:0/1363209270 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558b254527d0 0x558b25450ae0 unknown :-1 s=CLOSED pgs=955 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:59.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.413+0000 7f2e58810000 1 --2- 172.21.15.16:0/1363209270 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x558b25451eb0 0x558b254679a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:59.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.413+0000 7f2e58810000 1 --2- 172.21.15.16:0/1363209270 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558b2540b460 0x558b25467460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:59.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.413+0000 7f2e58810000 1 -- 172.21.15.16:0/1363209270 >> 172.21.15.16:0/1363209270 conn(0x558b25407410 msgr2=0x558b25434b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:08:59.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.413+0000 7f2e58810000 1 -- 172.21.15.16:0/1363209270 shutdown_connections 2024-03-31T15:08:59.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.413+0000 7f2e58810000 1 -- 172.21.15.16:0/1363209270 wait complete. 2024-03-31T15:08:59.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.413+0000 7f2e58810000 1 -- start start 2024-03-31T15:08:59.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.417+0000 7f2e58810000 1 Processor -- start 2024-03-31T15:08:59.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.417+0000 7f2e58810000 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558b2540b460 0x558b2540ab00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:59.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.417+0000 7f2e58810000 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x558b25451eb0 0x558b25465480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:59.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.417+0000 7f2e58810000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558b254527d0 0x558b254659c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:59.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.417+0000 7f2e58810000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x558b251ff9d0 con 0x558b254527d0 2024-03-31T15:08:59.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.417+0000 7f2e58810000 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x558b25297020 con 0x558b2540b460 2024-03-31T15:08:59.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.417+0000 7f2e58810000 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x558b25214b60 con 0x558b25451eb0 2024-03-31T15:08:59.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.417+0000 7f2e58810000 10 client.0 _ll_register_callbacks cb 0x7f2e48028070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T15:08:59.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.417+0000 7f2e4f200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558b254527d0 0x558b254659c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:59.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.417+0000 7f2e4f200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558b254527d0 0x558b254659c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42974/0 (socket says 172.21.15.16:42974) 2024-03-31T15:08:59.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.417+0000 7f2e4f200640 1 -- 172.21.15.16:0/357331401 learned_addr learned my addr 172.21.15.16:0/357331401 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:08:59.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.417+0000 7f2e4e800640 1 --2- 172.21.15.16:0/357331401 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558b2540b460 0x558b2540ab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:59.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.417+0000 7f2e4de00640 1 --2- 172.21.15.16:0/357331401 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x558b25451eb0 0x558b25465480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:59.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.417+0000 7f2e4f200640 1 -- 172.21.15.16:0/357331401 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x558b25451eb0 msgr2=0x558b25465480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:08:59.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.417+0000 7f2e4f200640 1 --2- 172.21.15.16:0/357331401 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x558b25451eb0 0x558b25465480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:59.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.417+0000 7f2e4f200640 1 -- 172.21.15.16:0/357331401 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558b2540b460 msgr2=0x558b2540ab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:08:59.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.417+0000 7f2e4f200640 1 --2- 172.21.15.16:0/357331401 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558b2540b460 0x558b2540ab00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:59.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.417+0000 7f2e4f200640 1 -- 172.21.15.16:0/357331401 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558b2529ba60 con 0x558b254527d0 2024-03-31T15:08:59.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.417+0000 7f2e4f200640 1 --2- 172.21.15.16:0/357331401 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558b254527d0 0x558b254659c0 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7f2e48009da0 tx=0x7f2e48020000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3da0225fc1e1d243 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:08:59.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.417+0000 7f2e2fe00640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-31T15:08:59.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.417+0000 7f2e2fe00640 1 -- 172.21.15.16:0/357331401 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e4802a040 con 0x558b254527d0 2024-03-31T15:08:59.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.417+0000 7f2e2fe00640 1 -- 172.21.15.16:0/357331401 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(70..70 src has 1..70) ==== 5882+0+0 (secure 0 0 0) 0x7f2e48030670 con 0x558b254527d0 2024-03-31T15:08:59.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.417+0000 7f2e58810000 1 -- 172.21.15.16:0/357331401 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2e4801bc80 con 0x558b254527d0 2024-03-31T15:08:59.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.417+0000 7f2e2fe00640 1 -- 172.21.15.16:0/357331401 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=51) ==== 24+0+0 (secure 0 0 0) 0x7f2e48030870 con 0x558b254527d0 2024-03-31T15:08:59.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.417+0000 7f2e58810000 10 client.7197 fetch_fsmap learned FSMap version 51 2024-03-31T15:08:59.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.417+0000 7f2e58810000 1 -- 172.21.15.16:0/357331401 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=51,monmap=2+}) -- 0x558b251faa40 con 0x558b254527d0 2024-03-31T15:08:59.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.417+0000 7f2e2fe00640 1 -- 172.21.15.16:0/357331401 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2e4801bc80 con 0x558b254527d0 2024-03-31T15:08:59.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.417+0000 7f2e2fe00640 1 -- 172.21.15.16:0/357331401 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 51) ==== 3494+0+0 (secure 0 0 0) 0x7f2e48056020 con 0x558b254527d0 2024-03-31T15:08:59.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.417+0000 7f2e58810000 10 client.7197 fetch_fsmap finished waiting for FSMap version 51 2024-03-31T15:08:59.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.417+0000 7f2e58810000 10 client.7197 resolve_mds: resolved 5:0 to role '5:0' aka daemon mds.a 2024-03-31T15:08:59.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.417+0000 7f2e58810000 20 client.7197 populate_metadata read hostname 'smithi016' 2024-03-31T15:08:59.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.417+0000 7f2e58810000 1 --2- 172.21.15.16:0/357331401 >> [v2:172.21.15.16:6834/2645389320,v1:172.21.15.16:6835/2645389320] conn(0x558b254c13e0 0x558b254c17c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:59.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.417+0000 7f2e58810000 4 client.7197 mds_command: new command op to 6829 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/subdir", "scrubops": ["repair", "recursive"]}] 2024-03-31T15:08:59.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.417+0000 7f2e58810000 1 -- 172.21.15.16:0/357331401 --> [v2:172.21.15.16:6834/2645389320,v1:172.21.15.16:6835/2645389320] -- command(tid 0: {"prefix": "scrub start", "path": "/subdir", "scrubops": ["repair", "recursive"]}) -- 0x558b25271c40 con 0x558b254c13e0 2024-03-31T15:08:59.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.417+0000 7f2e4e800640 1 --2- 172.21.15.16:0/357331401 >> [v2:172.21.15.16:6834/2645389320,v1:172.21.15.16:6835/2645389320] conn(0x558b254c13e0 0x558b254c17c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:59.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.421+0000 7f2e4e800640 1 --2- 172.21.15.16:0/357331401 >> [v2:172.21.15.16:6834/2645389320,v1:172.21.15.16:6835/2645389320] conn(0x558b254c13e0 0x558b254c17c0 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=294096267981c83e server_cookie=eb3c2665c70ccd44 in_seq=0 out_seq=0 2024-03-31T15:08:59.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.421+0000 7f2e2fe00640 10 client.7197 ms_handle_connect on v2:172.21.15.16:6834/2645389320 2024-03-31T15:08:59.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.421+0000 7f2e2fe00640 1 -- 172.21.15.16:0/357331401 <== mds.0 v2:172.21.15.16:6834/2645389320 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x558b25271c40 con 0x558b254c13e0 2024-03-31T15:08:59.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.421+0000 7f2e2fe00640 10 client.7197 handle_command_reply: tid=0 2024-03-31T15:08:59.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.421+0000 7f2e58810000 1 client.7197 shutdown 2024-03-31T15:08:59.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.421+0000 7f2e58810000 1 -- 172.21.15.16:0/357331401 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558b254527d0 msgr2=0x558b254659c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:59.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.421+0000 7f2e58810000 1 --2- 172.21.15.16:0/357331401 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558b254527d0 0x558b254659c0 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7f2e48009da0 tx=0x7f2e48020000 comp rx=0 tx=0).stop 2024-03-31T15:08:59.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.421+0000 7f2e58810000 1 -- 172.21.15.16:0/357331401 shutdown_connections 2024-03-31T15:08:59.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.421+0000 7f2e58810000 1 --2- 172.21.15.16:0/357331401 >> [v2:172.21.15.16:6834/2645389320,v1:172.21.15.16:6835/2645389320] conn(0x558b254c13e0 0x558b254c17c0 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:59.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.421+0000 7f2e58810000 1 --2- 172.21.15.16:0/357331401 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558b254527d0 0x558b254659c0 unknown :-1 s=CLOSED pgs=956 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:59.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.421+0000 7f2e58810000 1 --2- 172.21.15.16:0/357331401 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x558b25451eb0 0x558b25465480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:59.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.421+0000 7f2e58810000 1 --2- 172.21.15.16:0/357331401 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558b2540b460 0x558b2540ab00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:59.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.421+0000 7f2e58810000 1 -- 172.21.15.16:0/357331401 >> 172.21.15.16:0/357331401 conn(0x558b25407410 msgr2=0x558b25436720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:08:59.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.421+0000 7f2e2fe00640 0 client.7197 ms_handle_reset on v2:172.21.15.16:6834/2645389320 2024-03-31T15:08:59.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.421+0000 7f2e58810000 1 -- 172.21.15.16:0/357331401 shutdown_connections 2024-03-31T15:08:59.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.421+0000 7f2e58810000 1 -- 172.21.15.16:0/357331401 wait complete. 2024-03-31T15:08:59.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.421+0000 7f2e58810000 20 client.7197 trim_cache size 0 max 16384 2024-03-31T15:08:59.416 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-03-31T15:08:59.416 INFO:teuthology.orchestra.run.smithi016.stdout: "return_code": 0, 2024-03-31T15:08:59.416 INFO:teuthology.orchestra.run.smithi016.stdout: "scrub_tag": "4212950b-f59a-4d0f-8e6a-cfc43e01ca7f", 2024-03-31T15:08:59.416 INFO:teuthology.orchestra.run.smithi016.stdout: "mode": "asynchronous" 2024-03-31T15:08:59.416 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-03-31T15:08:59.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.421+0000 7f2e55400640 1 -- 172.21.15.16:0/2771320264 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f2e30068380 msgr2=0x7f2e3006a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:59.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.421+0000 7f2e55400640 1 --2- 172.21.15.16:0/2771320264 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f2e30068380 0x7f2e3006a840 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f2e480077d0 tx=0x7f2e48007bd0 comp rx=0 tx=0).stop 2024-03-31T15:08:59.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.421+0000 7f2e55400640 1 -- 172.21.15.16:0/2771320264 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2e500b4300 msgr2=0x7f2e500b5900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:59.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.421+0000 7f2e55400640 1 --2- 172.21.15.16:0/2771320264 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2e500b4300 0x7f2e500b5900 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 crypto rx=0x7f2e400ad040 tx=0x7f2e4001ada0 comp rx=0 tx=0).stop 2024-03-31T15:08:59.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.421+0000 7f2e55400640 1 -- 172.21.15.16:0/2771320264 shutdown_connections 2024-03-31T15:08:59.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.421+0000 7f2e55400640 1 --2- 172.21.15.16:0/2771320264 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f2e30068380 0x7f2e3006a840 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:59.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.421+0000 7f2e55400640 1 --2- 172.21.15.16:0/2771320264 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f2e50147230 0x7f2e50135cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:59.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.421+0000 7f2e55400640 1 --2- 172.21.15.16:0/2771320264 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2e500d1040 0x7f2e5015a9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:59.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.421+0000 7f2e55400640 1 --2- 172.21.15.16:0/2771320264 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2e500b4300 0x7f2e500b5900 unknown :-1 s=CLOSED pgs=952 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:59.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.421+0000 7f2e55400640 1 -- 172.21.15.16:0/2771320264 >> 172.21.15.16:0/2771320264 conn(0x7f2e5006b7f0 msgr2=0x7f2e500ce110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:08:59.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.421+0000 7f2e55400640 1 -- 172.21.15.16:0/2771320264 shutdown_connections 2024-03-31T15:08:59.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.421+0000 7f2e55400640 1 -- 172.21.15.16:0/2771320264 wait complete. 2024-03-31T15:08:59.432 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:08:59.432 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.5:0 scrub status 2024-03-31T15:08:59.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.585+0000 7feb5fe00640 1 Processor -- start 2024-03-31T15:08:59.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.585+0000 7feb5fe00640 1 -- start start 2024-03-31T15:08:59.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.585+0000 7feb5fe00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7feb6015e100 0x7feb600c75b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:59.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.585+0000 7feb5fe00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7feb6015e4e0 0x7feb6006b620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:59.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.585+0000 7feb5fe00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7feb6006bb60 0x7feb600723f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:59.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.585+0000 7feb5fe00640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7feb6006bfd0 con 0x7feb6006bb60 2024-03-31T15:08:59.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.585+0000 7feb5fe00640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7feb600c7010 con 0x7feb6015e100 2024-03-31T15:08:59.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.585+0000 7feb5fe00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7feb600c6490 con 0x7feb6015e4e0 2024-03-31T15:08:59.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.585+0000 7feb5f400640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7feb6006bb60 0x7feb600723f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:59.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.585+0000 7feb5f400640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7feb6006bb60 0x7feb600723f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42990/0 (socket says 172.21.15.16:42990) 2024-03-31T15:08:59.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.585+0000 7feb5f400640 1 -- 172.21.15.16:0/2280070291 learned_addr learned my addr 172.21.15.16:0/2280070291 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:08:59.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.585+0000 7feb5ea00640 1 --2- 172.21.15.16:0/2280070291 >> v2:172.21.15.89:3300/0 conn(0x7feb6015e100 0x7feb600c75b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:59.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.585+0000 7feb56000640 1 --2- 172.21.15.16:0/2280070291 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7feb6015e4e0 0x7feb6006b620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:59.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.585+0000 7feb5f400640 1 -- 172.21.15.16:0/2280070291 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7feb6015e4e0 msgr2=0x7feb6006b620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:08:59.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.585+0000 7feb5f400640 1 --2- 172.21.15.16:0/2280070291 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7feb6015e4e0 0x7feb6006b620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:59.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.585+0000 7feb5f400640 1 -- 172.21.15.16:0/2280070291 >> v2:172.21.15.89:3300/0 conn(0x7feb6015e100 msgr2=0x7feb600c75b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:08:59.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.585+0000 7feb5f400640 1 --2- 172.21.15.16:0/2280070291 >> v2:172.21.15.89:3300/0 conn(0x7feb6015e100 0x7feb600c75b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:59.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.585+0000 7feb5f400640 1 -- 172.21.15.16:0/2280070291 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb600c6770 con 0x7feb6006bb60 2024-03-31T15:08:59.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.585+0000 7feb5ea00640 1 --2- 172.21.15.16:0/2280070291 >> v2:172.21.15.89:3300/0 conn(0x7feb6015e100 0x7feb600c75b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:08:59.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.585+0000 7feb56000640 1 --2- 172.21.15.16:0/2280070291 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7feb6015e4e0 0x7feb6006b620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:08:59.580 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.589+0000 7feb5f400640 1 --2- 172.21.15.16:0/2280070291 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feb6006bb60 0x7feb600723f0 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7feb6006e080 tx=0x7feb5801c700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2dc6a553620636e4 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:08:59.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.589+0000 7feb5e000640 1 -- 172.21.15.16:0/2280070291 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb580a5580 con 0x7feb6006bb60 2024-03-31T15:08:59.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.589+0000 7feb5e000640 1 -- 172.21.15.16:0/2280070291 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feb580a5720 con 0x7feb6006bb60 2024-03-31T15:08:59.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.589+0000 7feb5e000640 1 -- 172.21.15.16:0/2280070291 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb580a5af0 con 0x7feb6006bb60 2024-03-31T15:08:59.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.589+0000 7feb5fe00640 1 -- 172.21.15.16:0/2280070291 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feb6006bb60 msgr2=0x7feb600723f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:59.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.589+0000 7feb5fe00640 1 --2- 172.21.15.16:0/2280070291 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feb6006bb60 0x7feb600723f0 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7feb6006e080 tx=0x7feb5801c700 comp rx=0 tx=0).stop 2024-03-31T15:08:59.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.589+0000 7feb5fe00640 1 -- 172.21.15.16:0/2280070291 shutdown_connections 2024-03-31T15:08:59.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.589+0000 7feb5fe00640 1 --2- 172.21.15.16:0/2280070291 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feb6006bb60 0x7feb600723f0 unknown :-1 s=CLOSED pgs=957 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:59.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.589+0000 7feb5fe00640 1 --2- 172.21.15.16:0/2280070291 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7feb6015e4e0 0x7feb6006b620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:59.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.589+0000 7feb5fe00640 1 --2- 172.21.15.16:0/2280070291 >> v2:172.21.15.89:3300/0 conn(0x7feb6015e100 0x7feb600c75b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:59.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.589+0000 7feb5fe00640 1 -- 172.21.15.16:0/2280070291 >> 172.21.15.16:0/2280070291 conn(0x7feb600d0e00 msgr2=0x7feb600d1210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:08:59.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.589+0000 7feb5fe00640 1 -- 172.21.15.16:0/2280070291 shutdown_connections 2024-03-31T15:08:59.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.589+0000 7feb5fe00640 1 -- 172.21.15.16:0/2280070291 wait complete. 2024-03-31T15:08:59.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.593+0000 7feb5fe00640 1 Processor -- start 2024-03-31T15:08:59.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.593+0000 7feb5fe00640 1 -- start start 2024-03-31T15:08:59.587 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.593+0000 7feb5fe00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7feb6006bb60 0x7feb60153090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:59.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.593+0000 7feb5fe00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feb6015e100 0x7feb600b6640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:59.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.593+0000 7feb5fe00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7feb6015e4e0 0x7feb601558d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:59.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.593+0000 7feb5fe00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7feb600c6020 con 0x7feb6015e100 2024-03-31T15:08:59.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.593+0000 7feb5fe00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7feb600c7d00 con 0x7feb6006bb60 2024-03-31T15:08:59.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.593+0000 7feb5fe00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7feb600c61a0 con 0x7feb6015e4e0 2024-03-31T15:08:59.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.593+0000 7feb56000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feb6015e100 0x7feb600b6640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:59.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.593+0000 7feb56000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feb6015e100 0x7feb600b6640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42996/0 (socket says 172.21.15.16:42996) 2024-03-31T15:08:59.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.593+0000 7feb56000640 1 -- 172.21.15.16:0/649237885 learned_addr learned my addr 172.21.15.16:0/649237885 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:08:59.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.593+0000 7feb5ea00640 1 --2- 172.21.15.16:0/649237885 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7feb6006bb60 0x7feb60153090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:59.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.593+0000 7feb5f400640 1 --2- 172.21.15.16:0/649237885 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7feb6015e4e0 0x7feb601558d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:59.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.593+0000 7feb56000640 1 -- 172.21.15.16:0/649237885 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7feb6015e4e0 msgr2=0x7feb601558d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:59.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.593+0000 7feb56000640 1 --2- 172.21.15.16:0/649237885 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7feb6015e4e0 0x7feb601558d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:59.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.593+0000 7feb56000640 1 -- 172.21.15.16:0/649237885 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7feb6006bb60 msgr2=0x7feb60153090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:59.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.593+0000 7feb56000640 1 --2- 172.21.15.16:0/649237885 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7feb6006bb60 0x7feb60153090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:59.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.593+0000 7feb56000640 1 -- 172.21.15.16:0/649237885 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb600ceba0 con 0x7feb6015e100 2024-03-31T15:08:59.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.597+0000 7feb56000640 1 --2- 172.21.15.16:0/649237885 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feb6015e100 0x7feb600b6640 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 crypto rx=0x7feb4800e810 tx=0x7feb4800ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:08:59.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.597+0000 7feb57400640 1 -- 172.21.15.16:0/649237885 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb48017070 con 0x7feb6015e100 2024-03-31T15:08:59.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.597+0000 7feb57400640 1 -- 172.21.15.16:0/649237885 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feb4800c8d0 con 0x7feb6015e100 2024-03-31T15:08:59.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.597+0000 7feb57400640 1 -- 172.21.15.16:0/649237885 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb4800f040 con 0x7feb6015e100 2024-03-31T15:08:59.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.597+0000 7feb5fe00640 1 -- 172.21.15.16:0/649237885 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feb60154b80 con 0x7feb6015e100 2024-03-31T15:08:59.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.597+0000 7feb5fe00640 1 -- 172.21.15.16:0/649237885 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feb600b1a00 con 0x7feb6015e100 2024-03-31T15:08:59.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.597+0000 7feb57400640 1 -- 172.21.15.16:0/649237885 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7feb4801a070 con 0x7feb6015e100 2024-03-31T15:08:59.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.597+0000 7feb57400640 1 --2- 172.21.15.16:0/649237885 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7feb3c068380 0x7feb3c06a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:59.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.597+0000 7feb57400640 1 -- 172.21.15.16:0/649237885 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(70..70 src has 1..70) ==== 5882+0+0 (secure 0 0 0) 0x7feb480866a0 con 0x7feb6015e100 2024-03-31T15:08:59.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.597+0000 7feb68c3b000 1 Processor -- start 2024-03-31T15:08:59.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.601+0000 7feb5ea00640 1 --2- 172.21.15.16:0/649237885 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7feb3c068380 0x7feb3c06a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:59.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.601+0000 7feb68c3b000 1 -- start start 2024-03-31T15:08:59.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.601+0000 7feb68c3b000 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5591c699e2d0 0x5591c699e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:59.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.601+0000 7feb68c3b000 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5591c699ebf0 0x5591c6951470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:59.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.601+0000 7feb68c3b000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5591c69519b0 0x5591c6951d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:59.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.601+0000 7feb68c3b000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5591c683aaf0 con 0x5591c69519b0 2024-03-31T15:08:59.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.601+0000 7feb68c3b000 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x5591c67a3b00 con 0x5591c699e2d0 2024-03-31T15:08:59.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.601+0000 7feb68c3b000 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x5591c6740940 con 0x5591c699ebf0 2024-03-31T15:08:59.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.601+0000 7feb5ea00640 1 --2- 172.21.15.16:0/649237885 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7feb3c068380 0x7feb3c06a840 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7feb500098d0 tx=0x7feb50008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:08:59.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.601+0000 7feb5f400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5591c69519b0 0x5591c6951d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:59.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.601+0000 7feb56000640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5591c699ebf0 0x5591c6951470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:59.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.601+0000 7feb5f400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5591c69519b0 0x5591c6951d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42998/0 (socket says 172.21.15.16:42998) 2024-03-31T15:08:59.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.601+0000 7feb5f400640 1 -- 172.21.15.16:0/3582941318 learned_addr learned my addr 172.21.15.16:0/3582941318 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:08:59.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.601+0000 7feb5ea00640 1 --2- 172.21.15.16:0/3582941318 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5591c699e2d0 0x5591c699e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:59.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.601+0000 7feb5f400640 1 -- 172.21.15.16:0/3582941318 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5591c699ebf0 msgr2=0x5591c6951470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:08:59.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.601+0000 7feb5f400640 1 --2- 172.21.15.16:0/3582941318 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5591c699ebf0 0x5591c6951470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:59.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.601+0000 7feb5f400640 1 -- 172.21.15.16:0/3582941318 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5591c699e2d0 msgr2=0x5591c699e6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:08:59.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.601+0000 7feb5f400640 1 --2- 172.21.15.16:0/3582941318 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5591c699e2d0 0x5591c699e6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:59.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.601+0000 7feb5f400640 1 -- 172.21.15.16:0/3582941318 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5591c6729380 con 0x5591c69519b0 2024-03-31T15:08:59.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.601+0000 7feb5f400640 1 --2- 172.21.15.16:0/3582941318 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5591c69519b0 0x5591c6951d90 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7feb5801adb0 tx=0x7feb58004ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=64a5b7b464e9e583 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:08:59.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.601+0000 7feb4f400640 1 -- 172.21.15.16:0/3582941318 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb580a5580 con 0x5591c69519b0 2024-03-31T15:08:59.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.601+0000 7feb4f400640 1 -- 172.21.15.16:0/3582941318 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feb580a5720 con 0x5591c69519b0 2024-03-31T15:08:59.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.601+0000 7feb4f400640 1 -- 172.21.15.16:0/3582941318 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb580a5ad0 con 0x5591c69519b0 2024-03-31T15:08:59.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.601+0000 7feb68c3b000 1 -- 172.21.15.16:0/3582941318 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5591c69519b0 msgr2=0x5591c6951d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:59.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.601+0000 7feb68c3b000 1 --2- 172.21.15.16:0/3582941318 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5591c69519b0 0x5591c6951d90 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7feb5801adb0 tx=0x7feb58004ac0 comp rx=0 tx=0).stop 2024-03-31T15:08:59.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.601+0000 7feb68c3b000 1 -- 172.21.15.16:0/3582941318 shutdown_connections 2024-03-31T15:08:59.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.601+0000 7feb68c3b000 1 --2- 172.21.15.16:0/3582941318 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5591c69519b0 0x5591c6951d90 unknown :-1 s=CLOSED pgs=959 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:59.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.601+0000 7feb68c3b000 1 --2- 172.21.15.16:0/3582941318 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5591c699ebf0 0x5591c6951470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:59.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.601+0000 7feb68c3b000 1 --2- 172.21.15.16:0/3582941318 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5591c699e2d0 0x5591c699e6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:59.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.601+0000 7feb68c3b000 1 -- 172.21.15.16:0/3582941318 >> 172.21.15.16:0/3582941318 conn(0x5591c697f070 msgr2=0x5591c6981460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:08:59.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.601+0000 7feb68c3b000 1 -- 172.21.15.16:0/3582941318 shutdown_connections 2024-03-31T15:08:59.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.601+0000 7feb68c3b000 1 -- 172.21.15.16:0/3582941318 wait complete. 2024-03-31T15:08:59.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.601+0000 7feb68c3b000 1 -- start start 2024-03-31T15:08:59.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.601+0000 7feb68c3b000 1 Processor -- start 2024-03-31T15:08:59.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.605+0000 7feb68c3b000 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5591c69519b0 0x5591c69a5630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:59.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.605+0000 7feb68c3b000 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5591c699e2d0 0x5591c69a5b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:59.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.605+0000 7feb68c3b000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5591c699ebf0 0x5591c6956260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:59.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.605+0000 7feb68c3b000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5591c67611e0 con 0x5591c699ebf0 2024-03-31T15:08:59.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.605+0000 7feb68c3b000 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x5591c679fab0 con 0x5591c69519b0 2024-03-31T15:08:59.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.605+0000 7feb68c3b000 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x5591c695d000 con 0x5591c699e2d0 2024-03-31T15:08:59.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.605+0000 7feb68c3b000 10 client.0 _ll_register_callbacks cb 0x5591c693f370 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T15:08:59.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.605+0000 7feb5ea00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5591c69519b0 0x5591c69a5630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:59.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.605+0000 7feb5f400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5591c699ebf0 0x5591c6956260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:59.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.605+0000 7feb5ea00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5591c69519b0 0x5591c69a5630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.16:43350/0 (socket says 172.21.15.16:43350) 2024-03-31T15:08:59.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.605+0000 7feb5ea00640 1 -- 172.21.15.16:0/4184252559 learned_addr learned my addr 172.21.15.16:0/4184252559 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:08:59.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.605+0000 7feb56000640 1 --2- 172.21.15.16:0/4184252559 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5591c699e2d0 0x5591c69a5b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:59.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.605+0000 7feb5ea00640 1 -- 172.21.15.16:0/4184252559 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5591c699e2d0 msgr2=0x5591c69a5b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:08:59.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.605+0000 7feb5ea00640 1 --2- 172.21.15.16:0/4184252559 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5591c699e2d0 0x5591c69a5b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:59.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.605+0000 7feb5ea00640 1 -- 172.21.15.16:0/4184252559 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5591c699ebf0 msgr2=0x5591c6956260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:08:59.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.605+0000 7feb5ea00640 1 --2- 172.21.15.16:0/4184252559 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5591c699ebf0 0x5591c6956260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:59.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.605+0000 7feb5ea00640 1 -- 172.21.15.16:0/4184252559 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5591c675fb70 con 0x5591c69519b0 2024-03-31T15:08:59.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.605+0000 7feb5f400640 1 --2- 172.21.15.16:0/4184252559 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5591c699ebf0 0x5591c6956260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:08:59.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.605+0000 7feb5ea00640 1 --2- 172.21.15.16:0/4184252559 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5591c69519b0 0x5591c69a5630 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7feb500156c0 tx=0x7feb500046d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d0f8b64d2d318019 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:08:59.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.605+0000 7feb43e00640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-03-31T15:08:59.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.605+0000 7feb43e00640 1 -- 172.21.15.16:0/4184252559 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb5000f040 con 0x5591c69519b0 2024-03-31T15:08:59.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.605+0000 7feb43e00640 1 -- 172.21.15.16:0/4184252559 <== mon.1 v2:172.21.15.89:3300/0 2 ==== osd_map(70..70 src has 1..70) ==== 5882+0+0 (secure 0 0 0) 0x7feb50014720 con 0x5591c69519b0 2024-03-31T15:08:59.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.605+0000 7feb68c3b000 1 -- 172.21.15.16:0/4184252559 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7feb580a5720 con 0x5591c69519b0 2024-03-31T15:08:59.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.605+0000 7feb43e00640 1 -- 172.21.15.16:0/4184252559 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=51) ==== 24+0+0 (secure 0 0 0) 0x7feb50014960 con 0x5591c69519b0 2024-03-31T15:08:59.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.605+0000 7feb68c3b000 10 client.7036 fetch_fsmap learned FSMap version 51 2024-03-31T15:08:59.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.605+0000 7feb68c3b000 1 -- 172.21.15.16:0/4184252559 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=51,monmap=2+}) -- 0x5591c6740b90 con 0x5591c69519b0 2024-03-31T15:08:59.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.605+0000 7feb43e00640 1 -- 172.21.15.16:0/4184252559 <== mon.1 v2:172.21.15.89:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feb580a5720 con 0x5591c69519b0 2024-03-31T15:08:59.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.605+0000 7feb43e00640 1 -- 172.21.15.16:0/4184252559 <== mon.1 v2:172.21.15.89:3300/0 5 ==== fsmap(e 51) ==== 3494+0+0 (secure 0 0 0) 0x7feb50014b00 con 0x5591c69519b0 2024-03-31T15:08:59.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.605+0000 7feb68c3b000 10 client.7036 fetch_fsmap finished waiting for FSMap version 51 2024-03-31T15:08:59.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.605+0000 7feb68c3b000 10 client.7036 resolve_mds: resolved 5:0 to role '5:0' aka daemon mds.a 2024-03-31T15:08:59.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.605+0000 7feb68c3b000 20 client.7036 populate_metadata read hostname 'smithi016' 2024-03-31T15:08:59.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.605+0000 7feb68c3b000 1 --2- 172.21.15.16:0/4184252559 >> [v2:172.21.15.16:6834/2645389320,v1:172.21.15.16:6835/2645389320] conn(0x5591c6957000 0x5591c6957410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:59.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.605+0000 7feb68c3b000 4 client.7036 mds_command: new command op to 6829 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-31T15:08:59.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.605+0000 7feb68c3b000 1 -- 172.21.15.16:0/4184252559 --> [v2:172.21.15.16:6834/2645389320,v1:172.21.15.16:6835/2645389320] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5591c673edd0 con 0x5591c6957000 2024-03-31T15:08:59.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.605+0000 7feb56000640 1 --2- 172.21.15.16:0/4184252559 >> [v2:172.21.15.16:6834/2645389320,v1:172.21.15.16:6835/2645389320] conn(0x5591c6957000 0x5591c6957410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:59.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.605+0000 7feb56000640 1 --2- 172.21.15.16:0/4184252559 >> [v2:172.21.15.16:6834/2645389320,v1:172.21.15.16:6835/2645389320] conn(0x5591c6957000 0x5591c6957410 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9aa081c554cffdbe server_cookie=18081f2a9a4c1c16 in_seq=0 out_seq=0 2024-03-31T15:08:59.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.605+0000 7feb43e00640 10 client.7036 ms_handle_connect on v2:172.21.15.16:6834/2645389320 2024-03-31T15:08:59.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.609+0000 7feb43e00640 1 -- 172.21.15.16:0/4184252559 <== mds.0 v2:172.21.15.16:6834/2645389320 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5591c673edd0 con 0x5591c6957000 2024-03-31T15:08:59.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.609+0000 7feb43e00640 10 client.7036 handle_command_reply: tid=0 2024-03-31T15:08:59.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.609+0000 7feb68c3b000 1 client.7036 shutdown 2024-03-31T15:08:59.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.609+0000 7feb68c3b000 1 -- 172.21.15.16:0/4184252559 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5591c69519b0 msgr2=0x5591c69a5630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:59.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.609+0000 7feb68c3b000 1 --2- 172.21.15.16:0/4184252559 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5591c69519b0 0x5591c69a5630 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7feb500156c0 tx=0x7feb500046d0 comp rx=0 tx=0).stop 2024-03-31T15:08:59.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.609+0000 7feb68c3b000 1 -- 172.21.15.16:0/4184252559 shutdown_connections 2024-03-31T15:08:59.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.609+0000 7feb68c3b000 1 --2- 172.21.15.16:0/4184252559 >> [v2:172.21.15.16:6834/2645389320,v1:172.21.15.16:6835/2645389320] conn(0x5591c6957000 0x5591c6957410 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:59.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.609+0000 7feb68c3b000 1 --2- 172.21.15.16:0/4184252559 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5591c699ebf0 0x5591c6956260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:59.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.609+0000 7feb43e00640 0 client.7036 ms_handle_reset on v2:172.21.15.16:6834/2645389320 2024-03-31T15:08:59.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.609+0000 7feb68c3b000 1 --2- 172.21.15.16:0/4184252559 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5591c699e2d0 0x5591c69a5b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:59.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.609+0000 7feb68c3b000 1 --2- 172.21.15.16:0/4184252559 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5591c69519b0 0x5591c69a5630 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:59.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.609+0000 7feb68c3b000 1 -- 172.21.15.16:0/4184252559 >> 172.21.15.16:0/4184252559 conn(0x5591c697f070 msgr2=0x5591c6981460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:08:59.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.609+0000 7feb68c3b000 1 -- 172.21.15.16:0/4184252559 shutdown_connections 2024-03-31T15:08:59.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.609+0000 7feb68c3b000 1 -- 172.21.15.16:0/4184252559 wait complete. 2024-03-31T15:08:59.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.609+0000 7feb68c3b000 20 client.7036 trim_cache size 0 max 16384 2024-03-31T15:08:59.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.645+0000 7feb68c3b000 1 Processor -- start 2024-03-31T15:08:59.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.645+0000 7feb68c3b000 1 -- start start 2024-03-31T15:08:59.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.645+0000 7feb68c3b000 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5591c69519b0 0x5591c69b47c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:59.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.645+0000 7feb68c3b000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5591c699e2d0 0x5591c69b4d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:59.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.645+0000 7feb68c3b000 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5591c699ebf0 0x5591c69b5710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:59.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.645+0000 7feb68c3b000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7feb60202850 con 0x5591c699e2d0 2024-03-31T15:08:59.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.645+0000 7feb68c3b000 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7feb60154400 con 0x5591c699ebf0 2024-03-31T15:08:59.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.645+0000 7feb68c3b000 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7feb6015d310 con 0x5591c69519b0 2024-03-31T15:08:59.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.645+0000 7feb56000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5591c699e2d0 0x5591c69b4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:59.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.645+0000 7feb56000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5591c699e2d0 0x5591c69b4d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43020/0 (socket says 172.21.15.16:43020) 2024-03-31T15:08:59.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.645+0000 7feb56000640 1 -- 172.21.15.16:0/4077861272 learned_addr learned my addr 172.21.15.16:0/4077861272 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:08:59.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.645+0000 7feb5f400640 1 --2- 172.21.15.16:0/4077861272 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5591c699ebf0 0x5591c69b5710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:59.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.645+0000 7feb5ea00640 1 --2- 172.21.15.16:0/4077861272 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5591c69519b0 0x5591c69b47c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:59.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.645+0000 7feb56000640 1 -- 172.21.15.16:0/4077861272 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5591c69519b0 msgr2=0x5591c69b47c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:08:59.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.645+0000 7feb56000640 1 --2- 172.21.15.16:0/4077861272 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5591c69519b0 0x5591c69b47c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:59.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.645+0000 7feb56000640 1 -- 172.21.15.16:0/4077861272 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5591c699ebf0 msgr2=0x5591c69b5710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:08:59.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.645+0000 7feb56000640 1 --2- 172.21.15.16:0/4077861272 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5591c699ebf0 0x5591c69b5710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:59.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.645+0000 7feb56000640 1 -- 172.21.15.16:0/4077861272 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5591c699ea20 con 0x5591c699e2d0 2024-03-31T15:08:59.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.645+0000 7feb56000640 1 --2- 172.21.15.16:0/4077861272 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5591c699e2d0 0x5591c69b4d00 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7feb480563d0 tx=0x7feb48009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=647bf339f94cac15 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:08:59.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.645+0000 7feb4ea00640 1 -- 172.21.15.16:0/4077861272 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb48021040 con 0x5591c699e2d0 2024-03-31T15:08:59.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.645+0000 7feb4ea00640 1 -- 172.21.15.16:0/4077861272 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feb4805ef70 con 0x5591c699e2d0 2024-03-31T15:08:59.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.645+0000 7feb4ea00640 1 -- 172.21.15.16:0/4077861272 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb4807e120 con 0x5591c699e2d0 2024-03-31T15:08:59.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.645+0000 7feb68c3b000 1 -- 172.21.15.16:0/4077861272 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5591c699e2d0 msgr2=0x5591c69b4d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:59.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.645+0000 7feb68c3b000 1 --2- 172.21.15.16:0/4077861272 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5591c699e2d0 0x5591c69b4d00 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7feb480563d0 tx=0x7feb48009f90 comp rx=0 tx=0).stop 2024-03-31T15:08:59.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.645+0000 7feb68c3b000 1 -- 172.21.15.16:0/4077861272 shutdown_connections 2024-03-31T15:08:59.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.645+0000 7feb68c3b000 1 --2- 172.21.15.16:0/4077861272 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5591c699ebf0 0x5591c69b5710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:59.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.645+0000 7feb68c3b000 1 --2- 172.21.15.16:0/4077861272 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5591c699e2d0 0x5591c69b4d00 unknown :-1 s=CLOSED pgs=960 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:59.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.645+0000 7feb68c3b000 1 --2- 172.21.15.16:0/4077861272 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5591c69519b0 0x5591c69b47c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:59.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.645+0000 7feb68c3b000 1 -- 172.21.15.16:0/4077861272 >> 172.21.15.16:0/4077861272 conn(0x5591c697f070 msgr2=0x5591c694fbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:08:59.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.645+0000 7feb68c3b000 1 -- 172.21.15.16:0/4077861272 shutdown_connections 2024-03-31T15:08:59.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.645+0000 7feb68c3b000 1 -- 172.21.15.16:0/4077861272 wait complete. 2024-03-31T15:08:59.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.649+0000 7feb68c3b000 1 -- start start 2024-03-31T15:08:59.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.649+0000 7feb68c3b000 1 Processor -- start 2024-03-31T15:08:59.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.649+0000 7feb68c3b000 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5591c69519b0 0x5591c699d650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:59.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.649+0000 7feb68c3b000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5591c699e2d0 0x5591c699db90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:59.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.649+0000 7feb68c3b000 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5591c699ebf0 0x5591c69a2030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:59.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.649+0000 7feb68c3b000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5591c675a750 con 0x5591c699e2d0 2024-03-31T15:08:59.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.649+0000 7feb68c3b000 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x5591c673e940 con 0x5591c69519b0 2024-03-31T15:08:59.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.649+0000 7feb68c3b000 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x5591c6872420 con 0x5591c699ebf0 2024-03-31T15:08:59.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.649+0000 7feb68c3b000 10 client.0 _ll_register_callbacks cb 0x7feb50009f30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T15:08:59.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.649+0000 7feb56000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5591c699e2d0 0x5591c699db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:59.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.649+0000 7feb56000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5591c699e2d0 0x5591c699db90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43028/0 (socket says 172.21.15.16:43028) 2024-03-31T15:08:59.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.649+0000 7feb56000640 1 -- 172.21.15.16:0/443504505 learned_addr learned my addr 172.21.15.16:0/443504505 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:08:59.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.649+0000 7feb5ea00640 1 --2- 172.21.15.16:0/443504505 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5591c69519b0 0x5591c699d650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:59.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.649+0000 7feb5f400640 1 --2- 172.21.15.16:0/443504505 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5591c699ebf0 0x5591c69a2030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:59.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.649+0000 7feb56000640 1 -- 172.21.15.16:0/443504505 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5591c699ebf0 msgr2=0x5591c69a2030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:08:59.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.649+0000 7feb56000640 1 --2- 172.21.15.16:0/443504505 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5591c699ebf0 0x5591c69a2030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:59.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.649+0000 7feb56000640 1 -- 172.21.15.16:0/443504505 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5591c69519b0 msgr2=0x5591c699d650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:08:59.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.649+0000 7feb56000640 1 --2- 172.21.15.16:0/443504505 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5591c69519b0 0x5591c699d650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:59.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.649+0000 7feb56000640 1 -- 172.21.15.16:0/443504505 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5591c6824140 con 0x5591c699e2d0 2024-03-31T15:08:59.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.649+0000 7feb5f400640 1 --2- 172.21.15.16:0/443504505 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5591c699ebf0 0x5591c69a2030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:08:59.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.649+0000 7feb56000640 1 --2- 172.21.15.16:0/443504505 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5591c699e2d0 0x5591c699db90 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7feb48086010 tx=0x7feb48015040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=39431983e374aa31 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:08:59.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.649+0000 7feb43e00640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-31T15:08:59.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.649+0000 7feb43e00640 1 -- 172.21.15.16:0/443504505 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb48021070 con 0x5591c699e2d0 2024-03-31T15:08:59.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.649+0000 7feb43e00640 1 -- 172.21.15.16:0/443504505 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(70..70 src has 1..70) ==== 5882+0+0 (secure 0 0 0) 0x7feb48052ab0 con 0x5591c699e2d0 2024-03-31T15:08:59.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.649+0000 7feb68c3b000 1 -- 172.21.15.16:0/443504505 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7feb4805ef70 con 0x5591c699e2d0 2024-03-31T15:08:59.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.649+0000 7feb43e00640 1 -- 172.21.15.16:0/443504505 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=51) ==== 24+0+0 (secure 0 0 0) 0x7feb4800cca0 con 0x5591c699e2d0 2024-03-31T15:08:59.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.649+0000 7feb68c3b000 10 client.7215 fetch_fsmap learned FSMap version 51 2024-03-31T15:08:59.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.649+0000 7feb68c3b000 1 -- 172.21.15.16:0/443504505 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=51,monmap=2+}) -- 0x5591c67321e0 con 0x5591c699e2d0 2024-03-31T15:08:59.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.649+0000 7feb43e00640 1 -- 172.21.15.16:0/443504505 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feb4805ef70 con 0x5591c699e2d0 2024-03-31T15:08:59.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.649+0000 7feb43e00640 1 -- 172.21.15.16:0/443504505 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 51) ==== 3494+0+0 (secure 0 0 0) 0x7feb48012050 con 0x5591c699e2d0 2024-03-31T15:08:59.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.649+0000 7feb68c3b000 10 client.7215 fetch_fsmap finished waiting for FSMap version 51 2024-03-31T15:08:59.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.649+0000 7feb68c3b000 10 client.7215 resolve_mds: resolved 5:0 to role '5:0' aka daemon mds.a 2024-03-31T15:08:59.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.649+0000 7feb68c3b000 20 client.7215 populate_metadata read hostname 'smithi016' 2024-03-31T15:08:59.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.649+0000 7feb68c3b000 1 --2- 172.21.15.16:0/443504505 >> [v2:172.21.15.16:6834/2645389320,v1:172.21.15.16:6835/2645389320] conn(0x5591c69a2570 0x5591c69a2980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:08:59.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.649+0000 7feb68c3b000 4 client.7215 mds_command: new command op to 6829 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-31T15:08:59.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.649+0000 7feb68c3b000 1 -- 172.21.15.16:0/443504505 --> [v2:172.21.15.16:6834/2645389320,v1:172.21.15.16:6835/2645389320] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5591c67b1040 con 0x5591c69a2570 2024-03-31T15:08:59.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.649+0000 7feb5ea00640 1 --2- 172.21.15.16:0/443504505 >> [v2:172.21.15.16:6834/2645389320,v1:172.21.15.16:6835/2645389320] conn(0x5591c69a2570 0x5591c69a2980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:08:59.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.653+0000 7feb43e00640 10 client.7215 ms_handle_connect on v2:172.21.15.16:6834/2645389320 2024-03-31T15:08:59.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.653+0000 7feb5ea00640 1 --2- 172.21.15.16:0/443504505 >> [v2:172.21.15.16:6834/2645389320,v1:172.21.15.16:6835/2645389320] conn(0x5591c69a2570 0x5591c69a2980 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=664cf21ef228e0a server_cookie=b27036ef808f33f4 in_seq=0 out_seq=0 2024-03-31T15:08:59.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.653+0000 7feb43e00640 1 -- 172.21.15.16:0/443504505 <== mds.0 v2:172.21.15.16:6834/2645389320 1 ==== command_reply(tid 0: 0 ) ==== 8+0+266 (crc 0 0 0) 0x5591c67b1040 con 0x5591c69a2570 2024-03-31T15:08:59.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.653+0000 7feb43e00640 10 client.7215 handle_command_reply: tid=0 2024-03-31T15:08:59.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.653+0000 7feb68c3b000 1 client.7215 shutdown 2024-03-31T15:08:59.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.653+0000 7feb68c3b000 1 -- 172.21.15.16:0/443504505 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5591c699e2d0 msgr2=0x5591c699db90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:59.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.653+0000 7feb68c3b000 1 --2- 172.21.15.16:0/443504505 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5591c699e2d0 0x5591c699db90 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7feb48086010 tx=0x7feb48015040 comp rx=0 tx=0).stop 2024-03-31T15:08:59.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.653+0000 7feb68c3b000 1 -- 172.21.15.16:0/443504505 shutdown_connections 2024-03-31T15:08:59.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.653+0000 7feb68c3b000 1 --2- 172.21.15.16:0/443504505 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5591c699ebf0 0x5591c69a2030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:59.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.653+0000 7feb68c3b000 1 --2- 172.21.15.16:0/443504505 >> [v2:172.21.15.16:6834/2645389320,v1:172.21.15.16:6835/2645389320] conn(0x5591c69a2570 0x5591c69a2980 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:59.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.653+0000 7feb68c3b000 1 --2- 172.21.15.16:0/443504505 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5591c699e2d0 0x5591c699db90 unknown :-1 s=CLOSED pgs=961 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:59.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.653+0000 7feb68c3b000 1 --2- 172.21.15.16:0/443504505 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5591c69519b0 0x5591c699d650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:59.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.653+0000 7feb43e00640 0 client.7215 ms_handle_reset on v2:172.21.15.16:6834/2645389320 2024-03-31T15:08:59.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.653+0000 7feb68c3b000 1 -- 172.21.15.16:0/443504505 >> 172.21.15.16:0/443504505 conn(0x5591c697f070 msgr2=0x5591c69b5050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:08:59.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.653+0000 7feb68c3b000 1 -- 172.21.15.16:0/443504505 shutdown_connections 2024-03-31T15:08:59.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.653+0000 7feb68c3b000 1 -- 172.21.15.16:0/443504505 wait complete. 2024-03-31T15:08:59.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.653+0000 7feb68c3b000 20 client.7215 trim_cache size 0 max 16384 2024-03-31T15:08:59.649 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-03-31T15:08:59.649 INFO:teuthology.orchestra.run.smithi016.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-31T15:08:59.649 INFO:teuthology.orchestra.run.smithi016.stdout: "scrubs": { 2024-03-31T15:08:59.649 INFO:teuthology.orchestra.run.smithi016.stdout: "4212950b-f59a-4d0f-8e6a-cfc43e01ca7f": { 2024-03-31T15:08:59.649 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/subdir", 2024-03-31T15:08:59.649 INFO:teuthology.orchestra.run.smithi016.stdout: "tag": "4212950b-f59a-4d0f-8e6a-cfc43e01ca7f", 2024-03-31T15:08:59.649 INFO:teuthology.orchestra.run.smithi016.stdout: "options": "recursive,repair" 2024-03-31T15:08:59.649 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-31T15:08:59.649 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-31T15:08:59.649 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-03-31T15:08:59.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.653+0000 7feb5fe00640 1 -- 172.21.15.16:0/649237885 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7feb3c068380 msgr2=0x7feb3c06a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:59.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.653+0000 7feb5fe00640 1 --2- 172.21.15.16:0/649237885 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7feb3c068380 0x7feb3c06a840 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7feb500098d0 tx=0x7feb50008040 comp rx=0 tx=0).stop 2024-03-31T15:08:59.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.653+0000 7feb5fe00640 1 -- 172.21.15.16:0/649237885 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feb6015e100 msgr2=0x7feb600b6640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:08:59.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.653+0000 7feb5fe00640 1 --2- 172.21.15.16:0/649237885 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feb6015e100 0x7feb600b6640 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 crypto rx=0x7feb4800e810 tx=0x7feb4800ecd0 comp rx=0 tx=0).stop 2024-03-31T15:08:59.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.653+0000 7feb5fe00640 1 -- 172.21.15.16:0/649237885 shutdown_connections 2024-03-31T15:08:59.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.653+0000 7feb5fe00640 1 --2- 172.21.15.16:0/649237885 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7feb3c068380 0x7feb3c06a840 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:59.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.653+0000 7feb5fe00640 1 --2- 172.21.15.16:0/649237885 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7feb6015e4e0 0x7feb601558d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:59.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.653+0000 7feb5fe00640 1 --2- 172.21.15.16:0/649237885 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feb6015e100 0x7feb600b6640 unknown :-1 s=CLOSED pgs=958 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:59.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.653+0000 7feb5fe00640 1 --2- 172.21.15.16:0/649237885 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7feb6006bb60 0x7feb60153090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:08:59.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.653+0000 7feb5fe00640 1 -- 172.21.15.16:0/649237885 >> 172.21.15.16:0/649237885 conn(0x7feb600d0e00 msgr2=0x7feb600af040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:08:59.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.653+0000 7feb5fe00640 1 -- 172.21.15.16:0/649237885 shutdown_connections 2024-03-31T15:08:59.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:08:59.653+0000 7feb5fe00640 1 -- 172.21.15.16:0/649237885 wait complete. 2024-03-31T15:08:59.663 INFO:tasks.cephfs.filesystem:scrub status for tag:4212950b-f59a-4d0f-8e6a-cfc43e01ca7f - {'path': '/subdir', 'tag': '4212950b-f59a-4d0f-8e6a-cfc43e01ca7f', 'options': 'recursive,repair'} 2024-03-31T15:09:07.201 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-31T15:09:07.204 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-31T15:09:29.664 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:09:29.664 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.5:0 scrub status 2024-03-31T15:09:29.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:29.813+0000 7f67c2e00640 1 Processor -- start 2024-03-31T15:09:29.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:29.813+0000 7f67c2e00640 1 -- start start 2024-03-31T15:09:29.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:29.817+0000 7f67c2e00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f67bc070030 0x7f67bc143c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:29.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:29.817+0000 7f67c2e00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f67bc141590 0x7f67bc144170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:29.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:29.817+0000 7f67c2e00640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f67bc1446b0 con 0x7f67bc1411b0 2024-03-31T15:09:29.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:29.817+0000 7f67c2e00640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f67bc15f910 con 0x7f67bc070030 2024-03-31T15:09:29.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:29.817+0000 7f67c2e00640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f67bc162910 con 0x7f67bc141590 2024-03-31T15:09:29.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:29.817+0000 7f67b3e00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f67bc070030 0x7f67bc143c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:29.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:29.817+0000 7f67b3e00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f67bc070030 0x7f67bc143c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60532/0 (socket says 172.21.15.16:60532) 2024-03-31T15:09:29.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:29.817+0000 7f67b3e00640 1 -- 172.21.15.16:0/3040831659 learned_addr learned my addr 172.21.15.16:0/3040831659 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:09:29.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:29.817+0000 7f67c0e00640 1 --2- 172.21.15.16:0/3040831659 >> v2:172.21.15.89:3300/0 conn(0x7f67bc141590 0x7f67bc144170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:29.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:29.817+0000 7f67b3e00640 1 -- 172.21.15.16:0/3040831659 >> v2:172.21.15.89:3300/0 conn(0x7f67bc141590 msgr2=0x7f67bc144170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:09:29.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:29.817+0000 7f67b3e00640 1 --2- 172.21.15.16:0/3040831659 >> v2:172.21.15.89:3300/0 conn(0x7f67bc141590 0x7f67bc144170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:29.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:29.817+0000 7f67b3e00640 1 -- 172.21.15.16:0/3040831659 >> v1:172.21.15.89:6789/0 conn(0x7f67bc1411b0 legacy=0x7f67bc06f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:09:29.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:29.817+0000 7f67b3e00640 1 -- 172.21.15.16:0/3040831659 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67bc144980 con 0x7f67bc070030 2024-03-31T15:09:29.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:29.817+0000 7f67c0e00640 1 --2- 172.21.15.16:0/3040831659 >> v2:172.21.15.89:3300/0 conn(0x7f67bc141590 0x7f67bc144170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:09:29.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:29.821+0000 7f67b3e00640 1 --2- 172.21.15.16:0/3040831659 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f67bc070030 0x7f67bc143c30 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7f67a4009870 tx=0x7f67a401c4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16bdf933496fbc5e server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:09:29.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:29.821+0000 7f67bb400640 1 -- 172.21.15.16:0/3040831659 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67a40aa610 con 0x7f67bc070030 2024-03-31T15:09:29.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:29.821+0000 7f67bb400640 1 -- 172.21.15.16:0/3040831659 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f67a40af070 con 0x7f67bc070030 2024-03-31T15:09:29.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:29.821+0000 7f67bb400640 1 -- 172.21.15.16:0/3040831659 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67a40aa9d0 con 0x7f67bc070030 2024-03-31T15:09:29.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:29.821+0000 7f67c2e00640 1 -- 172.21.15.16:0/3040831659 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f67bc070030 msgr2=0x7f67bc143c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:29.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:29.821+0000 7f67c2e00640 1 --2- 172.21.15.16:0/3040831659 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f67bc070030 0x7f67bc143c30 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7f67a4009870 tx=0x7f67a401c4b0 comp rx=0 tx=0).stop 2024-03-31T15:09:29.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:29.821+0000 7f67c2e00640 1 -- 172.21.15.16:0/3040831659 shutdown_connections 2024-03-31T15:09:29.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:29.821+0000 7f67c2e00640 1 --2- 172.21.15.16:0/3040831659 >> v2:172.21.15.89:3300/0 conn(0x7f67bc141590 0x7f67bc144170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:29.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:29.821+0000 7f67c2e00640 1 --2- 172.21.15.16:0/3040831659 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f67bc070030 0x7f67bc143c30 unknown :-1 s=CLOSED pgs=962 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:29.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:29.821+0000 7f67c2e00640 1 -- 172.21.15.16:0/3040831659 >> 172.21.15.16:0/3040831659 conn(0x7f67bc067930 msgr2=0x7f67bc0b3ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:09:29.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:29.821+0000 7f67c2e00640 1 -- 172.21.15.16:0/3040831659 shutdown_connections 2024-03-31T15:09:29.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:29.821+0000 7f67c2e00640 1 -- 172.21.15.16:0/3040831659 wait complete. 2024-03-31T15:09:29.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:29.821+0000 7f67c2e00640 1 Processor -- start 2024-03-31T15:09:29.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:29.821+0000 7f67c2e00640 1 -- start start 2024-03-31T15:09:29.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:29.825+0000 7f67c2e00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f67bc070030 0x7f67bc15bdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:29.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:29.825+0000 7f67c2e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f67bc1411b0 0x7f67bc1f7970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:29.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:29.825+0000 7f67c2e00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f67bc141590 0x7f67bc1f7eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:29.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:29.825+0000 7f67c2e00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f67bc15f110 con 0x7f67bc1411b0 2024-03-31T15:09:29.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:29.825+0000 7f67c2e00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f67bc162da0 con 0x7f67bc141590 2024-03-31T15:09:29.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:29.825+0000 7f67c2e00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f67bc15f290 con 0x7f67bc070030 2024-03-31T15:09:29.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:29.825+0000 7f67b3e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f67bc1411b0 0x7f67bc1f7970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:29.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:29.825+0000 7f67b3e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f67bc1411b0 0x7f67bc1f7970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60534/0 (socket says 172.21.15.16:60534) 2024-03-31T15:09:29.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:29.825+0000 7f67b3e00640 1 -- 172.21.15.16:0/1993598770 learned_addr learned my addr 172.21.15.16:0/1993598770 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:09:29.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:29.825+0000 7f67c0e00640 1 --2- 172.21.15.16:0/1993598770 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f67bc141590 0x7f67bc1f7eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:29.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:29.825+0000 7f67bbe00640 1 --2- 172.21.15.16:0/1993598770 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f67bc070030 0x7f67bc15bdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:29.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:29.825+0000 7f67b3e00640 1 -- 172.21.15.16:0/1993598770 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f67bc070030 msgr2=0x7f67bc15bdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:29.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:29.825+0000 7f67b3e00640 1 --2- 172.21.15.16:0/1993598770 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f67bc070030 0x7f67bc15bdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:29.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:29.825+0000 7f67b3e00640 1 -- 172.21.15.16:0/1993598770 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f67bc141590 msgr2=0x7f67bc1f7eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:29.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:29.825+0000 7f67b3e00640 1 --2- 172.21.15.16:0/1993598770 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f67bc141590 0x7f67bc1f7eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:29.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:29.825+0000 7f67b3e00640 1 -- 172.21.15.16:0/1993598770 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67bc15c330 con 0x7f67bc1411b0 2024-03-31T15:09:29.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:29.825+0000 7f67c0e00640 1 --2- 172.21.15.16:0/1993598770 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f67bc141590 0x7f67bc1f7eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:09:29.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:29.825+0000 7f67bbe00640 1 --2- 172.21.15.16:0/1993598770 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f67bc070030 0x7f67bc15bdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:09:29.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:29.825+0000 7f67b3e00640 1 --2- 172.21.15.16:0/1993598770 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f67bc1411b0 0x7f67bc1f7970 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7f67a40ac040 tx=0x7f67a401acf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:09:29.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:29.825+0000 7f67b8c00640 1 -- 172.21.15.16:0/1993598770 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67a40b3600 con 0x7f67bc1411b0 2024-03-31T15:09:29.818 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:29.825+0000 7f67b8c00640 1 -- 172.21.15.16:0/1993598770 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f67a40b37c0 con 0x7f67bc1411b0 2024-03-31T15:09:29.818 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:29.825+0000 7f67b8c00640 1 -- 172.21.15.16:0/1993598770 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67a40b3bb0 con 0x7f67bc1411b0 2024-03-31T15:09:29.818 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:29.825+0000 7f67c2e00640 1 -- 172.21.15.16:0/1993598770 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f67bc143c50 con 0x7f67bc1411b0 2024-03-31T15:09:29.818 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:29.825+0000 7f67c2e00640 1 -- 172.21.15.16:0/1993598770 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f67bc1f8bd0 con 0x7f67bc1411b0 2024-03-31T15:09:29.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:29.829+0000 7f67b8c00640 1 -- 172.21.15.16:0/1993598770 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f67a40b2380 con 0x7f67bc1411b0 2024-03-31T15:09:29.820 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:29.829+0000 7f67b8c00640 1 --2- 172.21.15.16:0/1993598770 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f67980683b0 0x7f679806a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:29.820 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:29.829+0000 7f67b8c00640 1 -- 172.21.15.16:0/1993598770 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(70..70 src has 1..70) ==== 5882+0+0 (secure 0 0 0) 0x7f67a4120df0 con 0x7f67bc1411b0 2024-03-31T15:09:29.820 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:29.829+0000 7f67c61e3000 1 Processor -- start 2024-03-31T15:09:29.820 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:29.829+0000 7f67bbe00640 1 --2- 172.21.15.16:0/1993598770 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f67980683b0 0x7f679806a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:29.820 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:29.829+0000 7f67c61e3000 1 -- start start 2024-03-31T15:09:29.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:29.829+0000 7f67bbe00640 1 --2- 172.21.15.16:0/1993598770 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f67980683b0 0x7f679806a870 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f67ac003e40 tx=0x7f67ac009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:09:29.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:29.829+0000 7f67c61e3000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d2589492d0 0x55d2589496b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:29.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:29.829+0000 7f67c61e3000 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55d258949bf0 0x55d2588fc470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:29.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:29.829+0000 7f67c61e3000 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d2588fc9b0 0x55d2588fcd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:29.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:29.829+0000 7f67c61e3000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55d2587e5af0 con 0x55d2589492d0 2024-03-31T15:09:29.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:29.829+0000 7f67c61e3000 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55d25874eb00 con 0x55d2588fc9b0 2024-03-31T15:09:29.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:29.829+0000 7f67c61e3000 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x55d2586eb940 con 0x55d258949bf0 2024-03-31T15:09:29.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:29.829+0000 7f67bbe00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d2589492d0 0x55d2589496b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:29.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:29.829+0000 7f67bbe00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d2589492d0 0x55d2589496b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60536/0 (socket says 172.21.15.16:60536) 2024-03-31T15:09:29.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:29.829+0000 7f67bbe00640 1 -- 172.21.15.16:0/2297677702 learned_addr learned my addr 172.21.15.16:0/2297677702 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:09:29.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:29.829+0000 7f67b3e00640 1 --2- 172.21.15.16:0/2297677702 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55d258949bf0 0x55d2588fc470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:29.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:29.829+0000 7f67c0e00640 1 --2- 172.21.15.16:0/2297677702 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d2588fc9b0 0x55d2588fcd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:29.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:29.829+0000 7f67bbe00640 1 -- 172.21.15.16:0/2297677702 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55d258949bf0 msgr2=0x55d2588fc470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:09:29.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:29.829+0000 7f67bbe00640 1 --2- 172.21.15.16:0/2297677702 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55d258949bf0 0x55d2588fc470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:29.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:29.829+0000 7f67bbe00640 1 -- 172.21.15.16:0/2297677702 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d2588fc9b0 msgr2=0x55d2588fcd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:09:29.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:29.829+0000 7f67bbe00640 1 --2- 172.21.15.16:0/2297677702 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d2588fc9b0 0x55d2588fcd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:29.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:29.829+0000 7f67bbe00640 1 -- 172.21.15.16:0/2297677702 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d2586d4380 con 0x55d2589492d0 2024-03-31T15:09:29.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:29.829+0000 7f67b3e00640 1 --2- 172.21.15.16:0/2297677702 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55d258949bf0 0x55d2588fc470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:09:29.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:29.833+0000 7f67bbe00640 1 --2- 172.21.15.16:0/2297677702 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d2589492d0 0x55d2589496b0 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 crypto rx=0x7f67ac003c20 tx=0x7f67ac010000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=adb7368cf7e2ab23 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:09:29.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:29.833+0000 7f67b0c00640 1 -- 172.21.15.16:0/2297677702 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67ac024070 con 0x55d2589492d0 2024-03-31T15:09:29.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:29.833+0000 7f67b0c00640 1 -- 172.21.15.16:0/2297677702 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f67ac014db0 con 0x55d2589492d0 2024-03-31T15:09:29.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:29.833+0000 7f67b0c00640 1 -- 172.21.15.16:0/2297677702 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67ac01f620 con 0x55d2589492d0 2024-03-31T15:09:29.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:29.833+0000 7f67c61e3000 1 -- 172.21.15.16:0/2297677702 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d2589492d0 msgr2=0x55d2589496b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:29.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:29.833+0000 7f67c61e3000 1 --2- 172.21.15.16:0/2297677702 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d2589492d0 0x55d2589496b0 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 crypto rx=0x7f67ac003c20 tx=0x7f67ac010000 comp rx=0 tx=0).stop 2024-03-31T15:09:29.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:29.833+0000 7f67c61e3000 1 -- 172.21.15.16:0/2297677702 shutdown_connections 2024-03-31T15:09:29.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:29.833+0000 7f67c61e3000 1 --2- 172.21.15.16:0/2297677702 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d2588fc9b0 0x55d2588fcd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:29.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:29.833+0000 7f67c61e3000 1 --2- 172.21.15.16:0/2297677702 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55d258949bf0 0x55d2588fc470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:29.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:29.833+0000 7f67c61e3000 1 --2- 172.21.15.16:0/2297677702 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d2589492d0 0x55d2589496b0 unknown :-1 s=CLOSED pgs=964 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:29.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:29.833+0000 7f67c61e3000 1 -- 172.21.15.16:0/2297677702 >> 172.21.15.16:0/2297677702 conn(0x55d25892a070 msgr2=0x55d25892c460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:09:29.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:29.833+0000 7f67c61e3000 1 -- 172.21.15.16:0/2297677702 shutdown_connections 2024-03-31T15:09:29.829 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:29.833+0000 7f67c61e3000 1 -- 172.21.15.16:0/2297677702 wait complete. 2024-03-31T15:09:29.829 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:29.833+0000 7f67c61e3000 1 -- start start 2024-03-31T15:09:29.829 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:29.833+0000 7f67c61e3000 1 Processor -- start 2024-03-31T15:09:29.829 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:29.833+0000 7f67c61e3000 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d2588fc9b0 0x55d258901670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:29.829 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:29.833+0000 7f67c61e3000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d2589492d0 0x55d258901bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:29.829 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:29.833+0000 7f67c61e3000 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55d258949bf0 0x55d2589020f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:29.829 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:29.833+0000 7f67c61e3000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55d25870c1e0 con 0x55d2589492d0 2024-03-31T15:09:29.829 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:29.833+0000 7f67c61e3000 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55d25874aab0 con 0x55d2588fc9b0 2024-03-31T15:09:29.829 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:29.833+0000 7f67c61e3000 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x55d258809250 con 0x55d258949bf0 2024-03-31T15:09:29.829 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:29.833+0000 7f67c61e3000 10 client.0 _ll_register_callbacks cb 0x55d2588ea370 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T15:09:29.829 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:29.833+0000 7f67b3e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d2589492d0 0x55d258901bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:29.829 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:29.833+0000 7f67b3e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d2589492d0 0x55d258901bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60538/0 (socket says 172.21.15.16:60538) 2024-03-31T15:09:29.829 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:29.833+0000 7f67b3e00640 1 -- 172.21.15.16:0/3353248432 learned_addr learned my addr 172.21.15.16:0/3353248432 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:09:29.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:29.833+0000 7f67bbe00640 1 --2- 172.21.15.16:0/3353248432 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d2588fc9b0 0x55d258901670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:29.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:29.833+0000 7f67c0e00640 1 --2- 172.21.15.16:0/3353248432 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55d258949bf0 0x55d2589020f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:29.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:29.833+0000 7f67b3e00640 1 -- 172.21.15.16:0/3353248432 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55d258949bf0 msgr2=0x55d2589020f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:09:29.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:29.833+0000 7f67b3e00640 1 --2- 172.21.15.16:0/3353248432 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55d258949bf0 0x55d2589020f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:29.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:29.833+0000 7f67b3e00640 1 -- 172.21.15.16:0/3353248432 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d2588fc9b0 msgr2=0x55d258901670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:09:29.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:29.833+0000 7f67b3e00640 1 --2- 172.21.15.16:0/3353248432 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d2588fc9b0 0x55d258901670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:29.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:29.833+0000 7f67b3e00640 1 -- 172.21.15.16:0/3353248432 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d25870ab70 con 0x55d2589492d0 2024-03-31T15:09:29.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:29.833+0000 7f67bbe00640 1 --2- 172.21.15.16:0/3353248432 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d2588fc9b0 0x55d258901670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:09:29.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:29.833+0000 7f67b3e00640 1 --2- 172.21.15.16:0/3353248432 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d2589492d0 0x55d258901bb0 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7f67a401c990 tx=0x7f67a40f5050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca9ec306cfb413b4 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:09:29.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:29.833+0000 7f67a9600640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-31T15:09:29.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:29.833+0000 7f67a9600640 1 -- 172.21.15.16:0/3353248432 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67a40c3040 con 0x55d2589492d0 2024-03-31T15:09:29.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:29.833+0000 7f67c61e3000 1 -- 172.21.15.16:0/3353248432 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f67ac014db0 con 0x55d2589492d0 2024-03-31T15:09:29.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:29.833+0000 7f67a9600640 1 -- 172.21.15.16:0/3353248432 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(70..70 src has 1..70) ==== 5882+0+0 (secure 0 0 0) 0x7f67a40f7110 con 0x55d2589492d0 2024-03-31T15:09:29.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:29.837+0000 7f67a9600640 1 -- 172.21.15.16:0/3353248432 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=52) ==== 24+0+0 (secure 0 0 0) 0x7f67a4109120 con 0x55d2589492d0 2024-03-31T15:09:29.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:29.837+0000 7f67c61e3000 10 client.7227 fetch_fsmap learned FSMap version 52 2024-03-31T15:09:29.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:29.837+0000 7f67c61e3000 1 -- 172.21.15.16:0/3353248432 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=52,monmap=2+}) -- 0x55d2586ebb90 con 0x55d2589492d0 2024-03-31T15:09:29.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:29.837+0000 7f67a9600640 1 -- 172.21.15.16:0/3353248432 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f67ac014db0 con 0x55d2589492d0 2024-03-31T15:09:29.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:29.837+0000 7f67a9600640 1 -- 172.21.15.16:0/3353248432 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 52) ==== 3494+0+0 (secure 0 0 0) 0x7f67a4118b30 con 0x55d2589492d0 2024-03-31T15:09:29.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:29.837+0000 7f67c61e3000 10 client.7227 fetch_fsmap finished waiting for FSMap version 52 2024-03-31T15:09:29.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:29.837+0000 7f67c61e3000 10 client.7227 resolve_mds: resolved 5:0 to role '5:0' aka daemon mds.a 2024-03-31T15:09:29.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:29.837+0000 7f67c61e3000 20 client.7227 populate_metadata read hostname 'smithi016' 2024-03-31T15:09:29.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:29.837+0000 7f67c61e3000 1 --2- 172.21.15.16:0/3353248432 >> [v2:172.21.15.16:6834/2645389320,v1:172.21.15.16:6835/2645389320] conn(0x55d258902990 0x55d25895aa50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:29.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:29.837+0000 7f67c61e3000 4 client.7227 mds_command: new command op to 6829 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-31T15:09:29.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:29.837+0000 7f67c61e3000 1 -- 172.21.15.16:0/3353248432 --> [v2:172.21.15.16:6834/2645389320,v1:172.21.15.16:6835/2645389320] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d2586e9dd0 con 0x55d258902990 2024-03-31T15:09:29.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:29.837+0000 7f67bbe00640 1 --2- 172.21.15.16:0/3353248432 >> [v2:172.21.15.16:6834/2645389320,v1:172.21.15.16:6835/2645389320] conn(0x55d258902990 0x55d25895aa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:29.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:29.837+0000 7f67bbe00640 1 --2- 172.21.15.16:0/3353248432 >> [v2:172.21.15.16:6834/2645389320,v1:172.21.15.16:6835/2645389320] conn(0x55d258902990 0x55d25895aa50 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7de05c57b21d82cd server_cookie=c4eee4fcac3ec0fb in_seq=0 out_seq=0 2024-03-31T15:09:29.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:29.837+0000 7f67a9600640 10 client.7227 ms_handle_connect on v2:172.21.15.16:6834/2645389320 2024-03-31T15:09:29.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:29.837+0000 7f67a9600640 1 -- 172.21.15.16:0/3353248432 <== mds.0 v2:172.21.15.16:6834/2645389320 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55d2586e9dd0 con 0x55d258902990 2024-03-31T15:09:29.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:29.837+0000 7f67a9600640 10 client.7227 handle_command_reply: tid=0 2024-03-31T15:09:29.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:29.837+0000 7f67c61e3000 1 client.7227 shutdown 2024-03-31T15:09:29.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:29.837+0000 7f67c61e3000 1 -- 172.21.15.16:0/3353248432 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d2589492d0 msgr2=0x55d258901bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:29.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:29.837+0000 7f67c61e3000 1 --2- 172.21.15.16:0/3353248432 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d2589492d0 0x55d258901bb0 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7f67a401c990 tx=0x7f67a40f5050 comp rx=0 tx=0).stop 2024-03-31T15:09:29.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:29.837+0000 7f67c61e3000 1 -- 172.21.15.16:0/3353248432 shutdown_connections 2024-03-31T15:09:29.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:29.837+0000 7f67c61e3000 1 --2- 172.21.15.16:0/3353248432 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55d258949bf0 0x55d2589020f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:29.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:29.837+0000 7f67c61e3000 1 --2- 172.21.15.16:0/3353248432 >> [v2:172.21.15.16:6834/2645389320,v1:172.21.15.16:6835/2645389320] conn(0x55d258902990 0x55d25895aa50 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:29.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:29.837+0000 7f67c61e3000 1 --2- 172.21.15.16:0/3353248432 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d2589492d0 0x55d258901bb0 unknown :-1 s=CLOSED pgs=965 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:29.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:29.837+0000 7f67c61e3000 1 --2- 172.21.15.16:0/3353248432 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d2588fc9b0 0x55d258901670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:29.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:29.837+0000 7f67c61e3000 1 -- 172.21.15.16:0/3353248432 >> 172.21.15.16:0/3353248432 conn(0x55d25892a070 msgr2=0x55d25892c460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:09:29.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:29.837+0000 7f67a9600640 0 client.7227 ms_handle_reset on v2:172.21.15.16:6834/2645389320 2024-03-31T15:09:29.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:29.837+0000 7f67c61e3000 1 -- 172.21.15.16:0/3353248432 shutdown_connections 2024-03-31T15:09:29.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:29.837+0000 7f67c61e3000 1 -- 172.21.15.16:0/3353248432 wait complete. 2024-03-31T15:09:29.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:29.837+0000 7f67c61e3000 20 client.7227 trim_cache size 0 max 16384 2024-03-31T15:09:29.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:29.853+0000 7f67c61e3000 1 Processor -- start 2024-03-31T15:09:29.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:29.857+0000 7f67c61e3000 1 -- start start 2024-03-31T15:09:29.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:29.857+0000 7f67c61e3000 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d2588fc9b0 0x55d2588fbd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:29.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:29.857+0000 7f67c61e3000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d2589492d0 0x55d2588fabd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:29.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:29.857+0000 7f67c61e3000 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55d258949bf0 0x55d2588fb110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:29.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:29.857+0000 7f67c61e3000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f67bc1fa750 con 0x55d2589492d0 2024-03-31T15:09:29.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:29.857+0000 7f67c61e3000 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f67bc202020 con 0x55d2588fc9b0 2024-03-31T15:09:29.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:29.857+0000 7f67c61e3000 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x55d2586e9940 con 0x55d258949bf0 2024-03-31T15:09:29.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:29.857+0000 7f67b3e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d2589492d0 0x55d2588fabd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:29.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:29.857+0000 7f67b3e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d2589492d0 0x55d2588fabd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60548/0 (socket says 172.21.15.16:60548) 2024-03-31T15:09:29.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:29.857+0000 7f67b3e00640 1 -- 172.21.15.16:0/2228608282 learned_addr learned my addr 172.21.15.16:0/2228608282 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:09:29.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:29.857+0000 7f67bbe00640 1 --2- 172.21.15.16:0/2228608282 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d2588fc9b0 0x55d2588fbd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:29.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:29.857+0000 7f67c0e00640 1 --2- 172.21.15.16:0/2228608282 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55d258949bf0 0x55d2588fb110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:29.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:29.857+0000 7f67b3e00640 1 -- 172.21.15.16:0/2228608282 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55d258949bf0 msgr2=0x55d2588fb110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:09:29.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:29.857+0000 7f67b3e00640 1 --2- 172.21.15.16:0/2228608282 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55d258949bf0 0x55d2588fb110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:29.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:29.857+0000 7f67b3e00640 1 -- 172.21.15.16:0/2228608282 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d2588fc9b0 msgr2=0x55d2588fbd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:09:29.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:29.857+0000 7f67b3e00640 1 --2- 172.21.15.16:0/2228608282 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d2588fc9b0 0x55d2588fbd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:29.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:29.857+0000 7f67b3e00640 1 -- 172.21.15.16:0/2228608282 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d258949a20 con 0x55d2589492d0 2024-03-31T15:09:29.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:29.857+0000 7f67b3e00640 1 --2- 172.21.15.16:0/2228608282 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d2589492d0 0x55d2588fabd0 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 crypto rx=0x7f67a40eeec0 tx=0x7f67a40f3000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fec2c0634cc25eb8 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:09:29.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:29.857+0000 7f67abe00640 1 -- 172.21.15.16:0/2228608282 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67a40c3070 con 0x55d2589492d0 2024-03-31T15:09:29.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:29.857+0000 7f67abe00640 1 -- 172.21.15.16:0/2228608282 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f67a40af040 con 0x55d2589492d0 2024-03-31T15:09:29.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:29.857+0000 7f67abe00640 1 -- 172.21.15.16:0/2228608282 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67a40f7310 con 0x55d2589492d0 2024-03-31T15:09:29.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:29.857+0000 7f67c61e3000 1 -- 172.21.15.16:0/2228608282 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d2589492d0 msgr2=0x55d2588fabd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:29.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:29.857+0000 7f67c61e3000 1 --2- 172.21.15.16:0/2228608282 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d2589492d0 0x55d2588fabd0 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 crypto rx=0x7f67a40eeec0 tx=0x7f67a40f3000 comp rx=0 tx=0).stop 2024-03-31T15:09:29.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:29.857+0000 7f67c61e3000 1 -- 172.21.15.16:0/2228608282 shutdown_connections 2024-03-31T15:09:29.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:29.857+0000 7f67c61e3000 1 --2- 172.21.15.16:0/2228608282 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55d258949bf0 0x55d2588fb110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:29.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:29.857+0000 7f67c61e3000 1 --2- 172.21.15.16:0/2228608282 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d2589492d0 0x55d2588fabd0 unknown :-1 s=CLOSED pgs=966 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:29.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:29.857+0000 7f67c61e3000 1 --2- 172.21.15.16:0/2228608282 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d2588fc9b0 0x55d2588fbd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:29.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:29.857+0000 7f67c61e3000 1 -- 172.21.15.16:0/2228608282 >> 172.21.15.16:0/2228608282 conn(0x55d25892a070 msgr2=0x55d2589489c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:09:29.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:29.857+0000 7f67c61e3000 1 -- 172.21.15.16:0/2228608282 shutdown_connections 2024-03-31T15:09:29.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:29.857+0000 7f67c61e3000 1 -- 172.21.15.16:0/2228608282 wait complete. 2024-03-31T15:09:29.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:29.857+0000 7f67c61e3000 1 -- start start 2024-03-31T15:09:29.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:29.857+0000 7f67c61e3000 1 Processor -- start 2024-03-31T15:09:29.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:29.861+0000 7f67c61e3000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d2588fc9b0 0x55d25895d160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:29.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:29.861+0000 7f67c61e3000 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55d2589492d0 0x55d2589b8be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:29.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:29.861+0000 7f67c61e3000 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d258949bf0 0x55d2589b9120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:29.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:29.861+0000 7f67c61e3000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55d25881d420 con 0x55d2588fc9b0 2024-03-31T15:09:29.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:29.861+0000 7f67c61e3000 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55d258705750 con 0x55d258949bf0 2024-03-31T15:09:29.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:29.861+0000 7f67c61e3000 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x55d25871a150 con 0x55d2589492d0 2024-03-31T15:09:29.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:29.861+0000 7f67c61e3000 10 client.0 _ll_register_callbacks cb 0x7f67a40f0750 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T15:09:29.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:29.861+0000 7f67bbe00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d2588fc9b0 0x55d25895d160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:29.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:29.861+0000 7f67bbe00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d2588fc9b0 0x55d25895d160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60562/0 (socket says 172.21.15.16:60562) 2024-03-31T15:09:29.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:29.861+0000 7f67bbe00640 1 -- 172.21.15.16:0/3078412520 learned_addr learned my addr 172.21.15.16:0/3078412520 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:09:29.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:29.861+0000 7f67bbe00640 1 -- 172.21.15.16:0/3078412520 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55d2589492d0 msgr2=0x55d2589b8be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:09:29.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:29.861+0000 7f67bbe00640 1 --2- 172.21.15.16:0/3078412520 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55d2589492d0 0x55d2589b8be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:29.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:29.861+0000 7f67bbe00640 1 -- 172.21.15.16:0/3078412520 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d258949bf0 msgr2=0x55d2589b9120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:09:29.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:29.861+0000 7f67c0e00640 1 --2- 172.21.15.16:0/3078412520 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d258949bf0 0x55d2589b9120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:29.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:29.861+0000 7f67bbe00640 1 --2- 172.21.15.16:0/3078412520 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d258949bf0 0x55d2589b9120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:29.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:29.861+0000 7f67bbe00640 1 -- 172.21.15.16:0/3078412520 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d2587cf140 con 0x55d2588fc9b0 2024-03-31T15:09:29.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:29.861+0000 7f67bbe00640 1 --2- 172.21.15.16:0/3078412520 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d2588fc9b0 0x55d25895d160 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7f67ac022040 tx=0x7f67ac003700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=119c099270010f90 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:09:29.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:29.861+0000 7f67a9600640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-31T15:09:29.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:29.861+0000 7f67a9600640 1 -- 172.21.15.16:0/3078412520 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67ac024040 con 0x55d2588fc9b0 2024-03-31T15:09:29.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:29.861+0000 7f67a9600640 1 -- 172.21.15.16:0/3078412520 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(70..70 src has 1..70) ==== 5882+0+0 (secure 0 0 0) 0x7f67ac030330 con 0x55d2588fc9b0 2024-03-31T15:09:29.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:29.861+0000 7f67c61e3000 1 -- 172.21.15.16:0/3078412520 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f67a40af040 con 0x55d2588fc9b0 2024-03-31T15:09:29.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:29.861+0000 7f67a9600640 1 -- 172.21.15.16:0/3078412520 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=52) ==== 24+0+0 (secure 0 0 0) 0x7f67ac03e050 con 0x55d2588fc9b0 2024-03-31T15:09:29.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:29.861+0000 7f67c61e3000 10 client.7233 fetch_fsmap learned FSMap version 52 2024-03-31T15:09:29.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:29.861+0000 7f67c61e3000 1 -- 172.21.15.16:0/3078412520 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=52,monmap=2+}) -- 0x55d2586dd1e0 con 0x55d2588fc9b0 2024-03-31T15:09:29.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:29.861+0000 7f67a9600640 1 -- 172.21.15.16:0/3078412520 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f67a40af040 con 0x55d2588fc9b0 2024-03-31T15:09:29.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:29.861+0000 7f67a9600640 1 -- 172.21.15.16:0/3078412520 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 52) ==== 3494+0+0 (secure 0 0 0) 0x7f67ac041020 con 0x55d2588fc9b0 2024-03-31T15:09:29.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:29.861+0000 7f67c61e3000 10 client.7233 fetch_fsmap finished waiting for FSMap version 52 2024-03-31T15:09:29.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:29.861+0000 7f67c61e3000 10 client.7233 resolve_mds: resolved 5:0 to role '5:0' aka daemon mds.a 2024-03-31T15:09:29.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:29.861+0000 7f67c61e3000 20 client.7233 populate_metadata read hostname 'smithi016' 2024-03-31T15:09:29.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:29.861+0000 7f67c61e3000 1 --2- 172.21.15.16:0/3078412520 >> [v2:172.21.15.16:6834/2645389320,v1:172.21.15.16:6835/2645389320] conn(0x55d2589b9660 0x55d2589b5600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:29.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:29.861+0000 7f67c61e3000 4 client.7233 mds_command: new command op to 6829 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-31T15:09:29.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:29.861+0000 7f67c61e3000 1 -- 172.21.15.16:0/3078412520 --> [v2:172.21.15.16:6834/2645389320,v1:172.21.15.16:6835/2645389320] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55d25875c040 con 0x55d2589b9660 2024-03-31T15:09:29.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:29.861+0000 7f67b3e00640 1 --2- 172.21.15.16:0/3078412520 >> [v2:172.21.15.16:6834/2645389320,v1:172.21.15.16:6835/2645389320] conn(0x55d2589b9660 0x55d2589b5600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:29.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:29.861+0000 7f67a9600640 10 client.7233 ms_handle_connect on v2:172.21.15.16:6834/2645389320 2024-03-31T15:09:29.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:29.861+0000 7f67b3e00640 1 --2- 172.21.15.16:0/3078412520 >> [v2:172.21.15.16:6834/2645389320,v1:172.21.15.16:6835/2645389320] conn(0x55d2589b9660 0x55d2589b5600 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7f15c782e1657f78 server_cookie=893d637d95aa5318 in_seq=0 out_seq=0 2024-03-31T15:09:29.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:29.861+0000 7f67a9600640 1 -- 172.21.15.16:0/3078412520 <== mds.0 v2:172.21.15.16:6834/2645389320 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55d25875c040 con 0x55d2589b9660 2024-03-31T15:09:29.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:29.861+0000 7f67a9600640 10 client.7233 handle_command_reply: tid=0 2024-03-31T15:09:29.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:29.861+0000 7f67c61e3000 1 client.7233 shutdown 2024-03-31T15:09:29.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:29.861+0000 7f67c61e3000 1 -- 172.21.15.16:0/3078412520 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d2588fc9b0 msgr2=0x55d25895d160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:29.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:29.861+0000 7f67c61e3000 1 --2- 172.21.15.16:0/3078412520 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d2588fc9b0 0x55d25895d160 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7f67ac022040 tx=0x7f67ac003700 comp rx=0 tx=0).stop 2024-03-31T15:09:29.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:29.861+0000 7f67c61e3000 1 -- 172.21.15.16:0/3078412520 shutdown_connections 2024-03-31T15:09:29.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:29.861+0000 7f67c61e3000 1 --2- 172.21.15.16:0/3078412520 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d258949bf0 0x55d2589b9120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:29.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:29.861+0000 7f67c61e3000 1 --2- 172.21.15.16:0/3078412520 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55d2589492d0 0x55d2589b8be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:29.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:29.861+0000 7f67c61e3000 1 --2- 172.21.15.16:0/3078412520 >> [v2:172.21.15.16:6834/2645389320,v1:172.21.15.16:6835/2645389320] conn(0x55d2589b9660 0x55d2589b5600 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:29.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:29.861+0000 7f67c61e3000 1 --2- 172.21.15.16:0/3078412520 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d2588fc9b0 0x55d25895d160 unknown :-1 s=CLOSED pgs=967 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:29.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:29.861+0000 7f67c61e3000 1 -- 172.21.15.16:0/3078412520 >> 172.21.15.16:0/3078412520 conn(0x55d25892a070 msgr2=0x55d258947380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:09:29.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:29.861+0000 7f67a9600640 0 client.7233 ms_handle_reset on v2:172.21.15.16:6834/2645389320 2024-03-31T15:09:29.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:29.865+0000 7f67c61e3000 1 -- 172.21.15.16:0/3078412520 shutdown_connections 2024-03-31T15:09:29.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:29.865+0000 7f67c61e3000 1 -- 172.21.15.16:0/3078412520 wait complete. 2024-03-31T15:09:29.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:29.865+0000 7f67c61e3000 20 client.7233 trim_cache size 0 max 16384 2024-03-31T15:09:29.859 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-03-31T15:09:29.859 INFO:teuthology.orchestra.run.smithi016.stdout: "status": "no active scrubs running", 2024-03-31T15:09:29.860 INFO:teuthology.orchestra.run.smithi016.stdout: "scrubs": {} 2024-03-31T15:09:29.860 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-03-31T15:09:29.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:29.865+0000 7f67c2e00640 1 -- 172.21.15.16:0/1993598770 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f67980683b0 msgr2=0x7f679806a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:29.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:29.865+0000 7f67c2e00640 1 --2- 172.21.15.16:0/1993598770 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f67980683b0 0x7f679806a870 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f67ac003e40 tx=0x7f67ac009040 comp rx=0 tx=0).stop 2024-03-31T15:09:29.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:29.865+0000 7f67c2e00640 1 -- 172.21.15.16:0/1993598770 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f67bc1411b0 msgr2=0x7f67bc1f7970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:29.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:29.865+0000 7f67c2e00640 1 --2- 172.21.15.16:0/1993598770 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f67bc1411b0 0x7f67bc1f7970 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7f67a40ac040 tx=0x7f67a401acf0 comp rx=0 tx=0).stop 2024-03-31T15:09:29.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:29.865+0000 7f67c2e00640 1 -- 172.21.15.16:0/1993598770 shutdown_connections 2024-03-31T15:09:29.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:29.865+0000 7f67c2e00640 1 --2- 172.21.15.16:0/1993598770 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f67980683b0 0x7f679806a870 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:29.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:29.865+0000 7f67c2e00640 1 --2- 172.21.15.16:0/1993598770 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f67bc141590 0x7f67bc1f7eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:29.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:29.865+0000 7f67c2e00640 1 --2- 172.21.15.16:0/1993598770 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f67bc1411b0 0x7f67bc1f7970 unknown :-1 s=CLOSED pgs=963 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:29.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:29.865+0000 7f67c2e00640 1 --2- 172.21.15.16:0/1993598770 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f67bc070030 0x7f67bc15bdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:29.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:29.865+0000 7f67c2e00640 1 -- 172.21.15.16:0/1993598770 >> 172.21.15.16:0/1993598770 conn(0x7f67bc067930 msgr2=0x7f67bc1611b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:09:29.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:29.865+0000 7f67c2e00640 1 -- 172.21.15.16:0/1993598770 shutdown_connections 2024-03-31T15:09:29.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:29.865+0000 7f67c2e00640 1 -- 172.21.15.16:0/1993598770 wait complete. 2024-03-31T15:09:29.875 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-31T15:09:29.875 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-31T15:09:30.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.025+0000 7fcc66a00640 1 Processor -- start 2024-03-31T15:09:30.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.025+0000 7fcc66a00640 1 -- start start 2024-03-31T15:09:30.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.029+0000 7fcc66a00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fcc60070030 0x7fcc6006c730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:30.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.029+0000 7fcc66a00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fcc60147dd0 0x7fcc600cc8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:30.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.029+0000 7fcc66a00640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcc60147660 con 0x7fcc60147280 2024-03-31T15:09:30.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.029+0000 7fcc66a00640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7fcc601620f0 con 0x7fcc60147dd0 2024-03-31T15:09:30.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.029+0000 7fcc66a00640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7fcc60162340 con 0x7fcc60070030 2024-03-31T15:09:30.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.029+0000 7fcc64a00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fcc60147dd0 0x7fcc600cc8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:30.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.029+0000 7fcc64a00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fcc60147dd0 0x7fcc600cc8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60568/0 (socket says 172.21.15.16:60568) 2024-03-31T15:09:30.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.029+0000 7fcc64a00640 1 -- 172.21.15.16:0/2848620245 learned_addr learned my addr 172.21.15.16:0/2848620245 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:09:30.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.029+0000 7fcc57e00640 1 --2- 172.21.15.16:0/2848620245 >> v2:172.21.15.89:3300/0 conn(0x7fcc60070030 0x7fcc6006c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:30.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.029+0000 7fcc64a00640 1 -- 172.21.15.16:0/2848620245 >> v2:172.21.15.89:3300/0 conn(0x7fcc60070030 msgr2=0x7fcc6006c730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:09:30.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.029+0000 7fcc64a00640 1 --2- 172.21.15.16:0/2848620245 >> v2:172.21.15.89:3300/0 conn(0x7fcc60070030 0x7fcc6006c730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:30.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.029+0000 7fcc64a00640 1 -- 172.21.15.16:0/2848620245 >> v1:172.21.15.89:6789/0 conn(0x7fcc60147280 legacy=0x7fcc6006f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:30.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.029+0000 7fcc64a00640 1 -- 172.21.15.16:0/2848620245 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc60162620 con 0x7fcc60147dd0 2024-03-31T15:09:30.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.029+0000 7fcc57e00640 1 --2- 172.21.15.16:0/2848620245 >> v2:172.21.15.89:3300/0 conn(0x7fcc60070030 0x7fcc6006c730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:09:30.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.033+0000 7fcc64a00640 1 --2- 172.21.15.16:0/2848620245 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcc60147dd0 0x7fcc600cc8f0 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x7fcc600cfc70 tx=0x7fcc5801c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=123e557babc33c90 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:09:30.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.033+0000 7fcc5f400640 1 -- 172.21.15.16:0/2848620245 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc580a57a0 con 0x7fcc60147dd0 2024-03-31T15:09:30.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.033+0000 7fcc5f400640 1 -- 172.21.15.16:0/2848620245 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcc580a5940 con 0x7fcc60147dd0 2024-03-31T15:09:30.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.033+0000 7fcc5f400640 1 -- 172.21.15.16:0/2848620245 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc580a5d10 con 0x7fcc60147dd0 2024-03-31T15:09:30.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.033+0000 7fcc66a00640 1 -- 172.21.15.16:0/2848620245 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcc60147dd0 msgr2=0x7fcc600cc8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:30.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.033+0000 7fcc66a00640 1 --2- 172.21.15.16:0/2848620245 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcc60147dd0 0x7fcc600cc8f0 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x7fcc600cfc70 tx=0x7fcc5801c9f0 comp rx=0 tx=0).stop 2024-03-31T15:09:30.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.033+0000 7fcc66a00640 1 -- 172.21.15.16:0/2848620245 shutdown_connections 2024-03-31T15:09:30.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.033+0000 7fcc66a00640 1 --2- 172.21.15.16:0/2848620245 >> v2:172.21.15.89:3300/0 conn(0x7fcc60070030 0x7fcc6006c730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:30.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.033+0000 7fcc66a00640 1 --2- 172.21.15.16:0/2848620245 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcc60147dd0 0x7fcc600cc8f0 unknown :-1 s=CLOSED pgs=968 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:30.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.033+0000 7fcc66a00640 1 -- 172.21.15.16:0/2848620245 >> 172.21.15.16:0/2848620245 conn(0x7fcc60067330 msgr2=0x7fcc60067740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:09:30.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.033+0000 7fcc66a00640 1 -- 172.21.15.16:0/2848620245 shutdown_connections 2024-03-31T15:09:30.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.033+0000 7fcc66a00640 1 -- 172.21.15.16:0/2848620245 wait complete. 2024-03-31T15:09:30.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.037+0000 7fcc66a00640 1 Processor -- start 2024-03-31T15:09:30.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.037+0000 7fcc66a00640 1 -- start start 2024-03-31T15:09:30.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.037+0000 7fcc66a00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcc60070030 0x7fcc60140af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:30.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.037+0000 7fcc66a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcc60147280 0x7fcc600b7820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:30.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.037+0000 7fcc66a00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fcc60147dd0 0x7fcc60146510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:30.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.037+0000 7fcc66a00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fcc600cd9d0 con 0x7fcc60147280 2024-03-31T15:09:30.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.037+0000 7fcc66a00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fcc600cd850 con 0x7fcc60070030 2024-03-31T15:09:30.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.037+0000 7fcc66a00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fcc600cdb50 con 0x7fcc60147dd0 2024-03-31T15:09:30.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.037+0000 7fcc57e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcc60147280 0x7fcc600b7820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:30.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.037+0000 7fcc57e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcc60147280 0x7fcc600b7820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60576/0 (socket says 172.21.15.16:60576) 2024-03-31T15:09:30.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.037+0000 7fcc57e00640 1 -- 172.21.15.16:0/1600481417 learned_addr learned my addr 172.21.15.16:0/1600481417 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:09:30.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.037+0000 7fcc64a00640 1 --2- 172.21.15.16:0/1600481417 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fcc60147dd0 0x7fcc60146510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:30.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.037+0000 7fcc5fe00640 1 --2- 172.21.15.16:0/1600481417 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcc60070030 0x7fcc60140af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:30.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.037+0000 7fcc57e00640 1 -- 172.21.15.16:0/1600481417 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fcc60147dd0 msgr2=0x7fcc60146510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:30.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.037+0000 7fcc57e00640 1 --2- 172.21.15.16:0/1600481417 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fcc60147dd0 0x7fcc60146510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:30.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.037+0000 7fcc57e00640 1 -- 172.21.15.16:0/1600481417 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcc60070030 msgr2=0x7fcc60140af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:30.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.037+0000 7fcc57e00640 1 --2- 172.21.15.16:0/1600481417 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcc60070030 0x7fcc60140af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:30.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.037+0000 7fcc57e00640 1 -- 172.21.15.16:0/1600481417 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc600b6bd0 con 0x7fcc60147280 2024-03-31T15:09:30.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.037+0000 7fcc5fe00640 1 --2- 172.21.15.16:0/1600481417 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcc60070030 0x7fcc60140af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:09:30.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.037+0000 7fcc57e00640 1 --2- 172.21.15.16:0/1600481417 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcc60147280 0x7fcc600b7820 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x7fcc4800d800 tx=0x7fcc4800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:09:30.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.037+0000 7fcc5cc00640 1 -- 172.21.15.16:0/1600481417 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc4800b6c0 con 0x7fcc60147280 2024-03-31T15:09:30.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.037+0000 7fcc5cc00640 1 -- 172.21.15.16:0/1600481417 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcc48019070 con 0x7fcc60147280 2024-03-31T15:09:30.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.037+0000 7fcc5cc00640 1 -- 172.21.15.16:0/1600481417 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc4800baa0 con 0x7fcc60147280 2024-03-31T15:09:30.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.037+0000 7fcc66a00640 1 -- 172.21.15.16:0/1600481417 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcc600b6eb0 con 0x7fcc60147280 2024-03-31T15:09:30.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.037+0000 7fcc66a00640 1 -- 172.21.15.16:0/1600481417 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcc60141e60 con 0x7fcc60147280 2024-03-31T15:09:30.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.041+0000 7fcc66a00640 1 -- 172.21.15.16:0/1600481417 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcc60069950 con 0x7fcc60147280 2024-03-31T15:09:30.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.041+0000 7fcc5cc00640 1 -- 172.21.15.16:0/1600481417 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fcc4800bc40 con 0x7fcc60147280 2024-03-31T15:09:30.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.041+0000 7fcc5cc00640 1 --2- 172.21.15.16:0/1600481417 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fcc3c068380 0x7fcc3c06a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:30.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.041+0000 7fcc5cc00640 1 -- 172.21.15.16:0/1600481417 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(70..70 src has 1..70) ==== 5882+0+0 (secure 0 0 0) 0x7fcc48086e60 con 0x7fcc60147280 2024-03-31T15:09:30.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.041+0000 7fcc5fe00640 1 --2- 172.21.15.16:0/1600481417 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fcc3c068380 0x7fcc3c06a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:30.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.041+0000 7fcc5fe00640 1 --2- 172.21.15.16:0/1600481417 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fcc3c068380 0x7fcc3c06a840 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7fcc50001710 tx=0x7fcc50009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:09:30.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.045+0000 7fcc5cc00640 1 -- 172.21.15.16:0/1600481417 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fcc480572d0 con 0x7fcc60147280 2024-03-31T15:09:30.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.389+0000 7fcc66a00640 1 -- 172.21.15.16:0/1600481417 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fcc600cf3e0 con 0x7fcc60147280 2024-03-31T15:09:30.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.389+0000 7fcc5cc00640 1 -- 172.21.15.16:0/1600481417 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 52 v52) ==== 94+0+5185 (secure 0 0 0) 0x7fcc48056bd0 con 0x7fcc60147280 2024-03-31T15:09:30.380 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:09:30.380 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":52,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6836/4188793506","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":4188793506},{"type":"v1","addr":"172.21.15.89:6836","nonce":4188793506}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":40},{"gid":5942,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/4105657676","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6835","nonce":4105657676},{"type":"v1","addr":"172.21.15.89:6837","nonce":4105657676}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":40},{"gid":6592,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6833/3986752851","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":3986752851},{"type":"v1","addr":"172.21.15.16:6833","nonce":3986752851}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":39}],"filesystems":[{"mdsmap":{"epoch":52,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-31T15:07:56.938140+0000","modified":"2024-03-31T15:09:01.267292+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":68,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6829},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6829":{"gid":6829,"name":"a","rank":0,"incarnation":47,"state":"up:active","state_seq":16,"addr":"172.21.15.16:6835/2645389320","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":2645389320},{"type":"v1","addr":"172.21.15.16:6835","nonce":2645389320}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6829,"qdb_cluster":[6829]},"id":5}]} 2024-03-31T15:09:30.380 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 52 2024-03-31T15:09:30.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.389+0000 7fcc66a00640 1 -- 172.21.15.16:0/1600481417 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fcc3c068380 msgr2=0x7fcc3c06a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:30.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.389+0000 7fcc66a00640 1 --2- 172.21.15.16:0/1600481417 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fcc3c068380 0x7fcc3c06a840 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7fcc50001710 tx=0x7fcc50009040 comp rx=0 tx=0).stop 2024-03-31T15:09:30.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.389+0000 7fcc66a00640 1 -- 172.21.15.16:0/1600481417 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcc60147280 msgr2=0x7fcc600b7820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:30.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.389+0000 7fcc66a00640 1 --2- 172.21.15.16:0/1600481417 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcc60147280 0x7fcc600b7820 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x7fcc4800d800 tx=0x7fcc4800dcc0 comp rx=0 tx=0).stop 2024-03-31T15:09:30.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.389+0000 7fcc66a00640 1 -- 172.21.15.16:0/1600481417 shutdown_connections 2024-03-31T15:09:30.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.389+0000 7fcc66a00640 1 --2- 172.21.15.16:0/1600481417 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fcc3c068380 0x7fcc3c06a840 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:30.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.389+0000 7fcc66a00640 1 --2- 172.21.15.16:0/1600481417 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fcc60147dd0 0x7fcc60146510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:30.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.389+0000 7fcc66a00640 1 --2- 172.21.15.16:0/1600481417 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcc60147280 0x7fcc600b7820 unknown :-1 s=CLOSED pgs=969 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:30.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.389+0000 7fcc66a00640 1 --2- 172.21.15.16:0/1600481417 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcc60070030 0x7fcc60140af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:30.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.389+0000 7fcc66a00640 1 -- 172.21.15.16:0/1600481417 >> 172.21.15.16:0/1600481417 conn(0x7fcc60067330 msgr2=0x7fcc6006f920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:09:30.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.389+0000 7fcc66a00640 1 -- 172.21.15.16:0/1600481417 shutdown_connections 2024-03-31T15:09:30.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.389+0000 7fcc66a00640 1 -- 172.21.15.16:0/1600481417 wait complete. 2024-03-31T15:09:30.401 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a damage ls --format=json-pretty 2024-03-31T15:09:30.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.553+0000 7feaa8600640 1 Processor -- start 2024-03-31T15:09:30.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.553+0000 7feaa8600640 1 -- start start 2024-03-31T15:09:30.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.553+0000 7feaa8600640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7feaa0070030 0x7feaa0143c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:30.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.553+0000 7feaa8600640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7feaa0141590 0x7feaa0144170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:30.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.553+0000 7feaa8600640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feaa01446b0 con 0x7feaa01411b0 2024-03-31T15:09:30.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.553+0000 7feaa8600640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7feaa015f910 con 0x7feaa0070030 2024-03-31T15:09:30.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.553+0000 7feaa8600640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7feaa0162910 con 0x7feaa0141590 2024-03-31T15:09:30.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.553+0000 7feaa5c00640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7feaa01411b0 0x7feaa006f920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:46648/0 (socket says 172.21.15.16:46648) 2024-03-31T15:09:30.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.553+0000 7feaa5c00640 1 -- 172.21.15.16:0/2718646998 learned_addr learned my addr 172.21.15.16:0/2718646998 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:09:30.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.553+0000 7feaa5200640 1 --2- 172.21.15.16:0/2718646998 >> v2:172.21.15.89:3300/0 conn(0x7feaa0070030 0x7feaa0143c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:30.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.553+0000 7feaa6600640 1 --2- 172.21.15.16:0/2718646998 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7feaa0141590 0x7feaa0144170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:30.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.553+0000 7fea9fe00640 1 -- 172.21.15.16:0/2718646998 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1772803788 0 0) 0x7feaa01446b0 con 0x7feaa01411b0 2024-03-31T15:09:30.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.553+0000 7fea9fe00640 1 -- 172.21.15.16:0/2718646998 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fea88003530 con 0x7feaa01411b0 2024-03-31T15:09:30.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.553+0000 7fea9fe00640 1 -- 172.21.15.16:0/2718646998 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 449905706 0 0) 0x7fea88003530 con 0x7feaa01411b0 2024-03-31T15:09:30.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.553+0000 7fea9fe00640 1 -- 172.21.15.16:0/2718646998 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7feaa0141590 msgr2=0x7feaa0144170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:09:30.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.553+0000 7fea9fe00640 1 --2- 172.21.15.16:0/2718646998 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7feaa0141590 0x7feaa0144170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:30.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.553+0000 7fea9fe00640 1 -- 172.21.15.16:0/2718646998 >> v2:172.21.15.89:3300/0 conn(0x7feaa0070030 msgr2=0x7feaa0143c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:09:30.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.553+0000 7fea9fe00640 1 --2- 172.21.15.16:0/2718646998 >> v2:172.21.15.89:3300/0 conn(0x7feaa0070030 0x7feaa0143c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:30.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.553+0000 7fea9fe00640 1 -- 172.21.15.16:0/2718646998 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feaa0144980 con 0x7feaa01411b0 2024-03-31T15:09:30.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.553+0000 7fea9fe00640 1 -- 172.21.15.16:0/2718646998 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 214807623 0 0) 0x7fea94001680 con 0x7feaa01411b0 2024-03-31T15:09:30.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.553+0000 7fea9fe00640 1 -- 172.21.15.16:0/2718646998 >> v1:172.21.15.16:6789/0 conn(0x7feaa01411b0 legacy=0x7feaa006f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:30.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.553+0000 7fea9fe00640 1 --2- 172.21.15.16:0/2718646998 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fea880039c0 0x7fea88003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:30.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.553+0000 7fea9fe00640 1 --2- 172.21.15.16:0/2718646998 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fea88004810 0x7fea88004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:30.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.553+0000 7fea9fe00640 1 --2- 172.21.15.16:0/2718646998 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7feaa0141590 0x7fea880051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:30.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.553+0000 7fea9fe00640 1 -- 172.21.15.16:0/2718646998 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7feaa0162910 con 0x7fea880039c0 2024-03-31T15:09:30.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.553+0000 7fea9fe00640 1 -- 172.21.15.16:0/2718646998 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fea8800b9d0 con 0x7fea88004810 2024-03-31T15:09:30.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.553+0000 7fea9fe00640 1 -- 172.21.15.16:0/2718646998 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fea8800bb50 con 0x7feaa0141590 2024-03-31T15:09:30.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.553+0000 7feaa5c00640 1 --2- 172.21.15.16:0/2718646998 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fea880039c0 0x7fea88003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:30.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.553+0000 7feaa5200640 1 --2- 172.21.15.16:0/2718646998 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fea88004810 0x7fea88004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:30.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.553+0000 7feaa6600640 1 --2- 172.21.15.16:0/2718646998 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7feaa0141590 0x7fea880051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:30.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.557+0000 7feaa5200640 1 -- 172.21.15.16:0/2718646998 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7feaa0141590 msgr2=0x7fea880051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:09:30.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.557+0000 7feaa5200640 1 --2- 172.21.15.16:0/2718646998 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7feaa0141590 0x7fea880051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:30.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.557+0000 7feaa5200640 1 -- 172.21.15.16:0/2718646998 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fea880039c0 msgr2=0x7fea88003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:09:30.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.557+0000 7feaa5200640 1 --2- 172.21.15.16:0/2718646998 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fea880039c0 0x7fea88003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:30.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.557+0000 7feaa5200640 1 -- 172.21.15.16:0/2718646998 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea88003530 con 0x7fea88004810 2024-03-31T15:09:30.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.557+0000 7feaa5c00640 1 --2- 172.21.15.16:0/2718646998 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fea880039c0 0x7fea88003de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:09:30.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.557+0000 7feaa5200640 1 --2- 172.21.15.16:0/2718646998 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fea88004810 0x7fea88004c90 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fea8c009ae0 tx=0x7fea8c01ca20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c517ec6a36e7ef4c server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:09:30.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.557+0000 7fea9fe00640 1 -- 172.21.15.16:0/2718646998 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea8c0096b0 con 0x7fea88004810 2024-03-31T15:09:30.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.557+0000 7fea9fe00640 1 -- 172.21.15.16:0/2718646998 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fea8c0a6490 con 0x7fea88004810 2024-03-31T15:09:30.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.557+0000 7fea9fe00640 1 -- 172.21.15.16:0/2718646998 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea8c0a6880 con 0x7fea88004810 2024-03-31T15:09:30.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.557+0000 7feaa8600640 1 -- 172.21.15.16:0/2718646998 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fea88004810 msgr2=0x7fea88004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:30.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.557+0000 7feaa8600640 1 --2- 172.21.15.16:0/2718646998 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fea88004810 0x7fea88004c90 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fea8c009ae0 tx=0x7fea8c01ca20 comp rx=0 tx=0).stop 2024-03-31T15:09:30.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.557+0000 7feaa6600640 1 -- 172.21.15.16:0/2718646998 reap_dead start 2024-03-31T15:09:30.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.561+0000 7feaa8600640 1 -- 172.21.15.16:0/2718646998 shutdown_connections 2024-03-31T15:09:30.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.561+0000 7feaa8600640 1 -- 172.21.15.16:0/2718646998 >> 172.21.15.16:0/2718646998 conn(0x7feaa0067930 msgr2=0x7feaa00b3ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:09:30.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.561+0000 7feaa8600640 1 -- 172.21.15.16:0/2718646998 shutdown_connections 2024-03-31T15:09:30.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.561+0000 7feaa8600640 1 -- 172.21.15.16:0/2718646998 wait complete. 2024-03-31T15:09:30.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.561+0000 7feaa8600640 1 Processor -- start 2024-03-31T15:09:30.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.561+0000 7feaa8600640 1 -- start start 2024-03-31T15:09:30.554 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.561+0000 7feaa8600640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fea88004810 0x7feaa0153650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:30.554 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.561+0000 7feaa8600640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feaa013fa40 0x7feaa0166160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:30.554 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.561+0000 7feaa8600640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7feaa01666a0 0x7feaa0160670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:30.554 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.561+0000 7feaa8600640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7feaa015f110 con 0x7feaa013fa40 2024-03-31T15:09:30.554 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.561+0000 7feaa8600640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7feaa0162d80 con 0x7feaa01666a0 2024-03-31T15:09:30.554 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.561+0000 7feaa8600640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7feaa015f290 con 0x7fea88004810 2024-03-31T15:09:30.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.561+0000 7feaa5200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feaa013fa40 0x7feaa0166160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:30.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.561+0000 7feaa5200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feaa013fa40 0x7feaa0166160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60602/0 (socket says 172.21.15.16:60602) 2024-03-31T15:09:30.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.561+0000 7feaa5200640 1 -- 172.21.15.16:0/2892959433 learned_addr learned my addr 172.21.15.16:0/2892959433 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:09:30.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.561+0000 7feaa5c00640 1 --2- 172.21.15.16:0/2892959433 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fea88004810 0x7feaa0153650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:30.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.561+0000 7feaa6600640 1 --2- 172.21.15.16:0/2892959433 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7feaa01666a0 0x7feaa0160670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:30.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.561+0000 7feaa5200640 1 -- 172.21.15.16:0/2892959433 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fea88004810 msgr2=0x7feaa0153650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:30.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.561+0000 7feaa5200640 1 --2- 172.21.15.16:0/2892959433 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fea88004810 0x7feaa0153650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:30.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.561+0000 7feaa5200640 1 -- 172.21.15.16:0/2892959433 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7feaa01666a0 msgr2=0x7feaa0160670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:30.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.561+0000 7feaa5200640 1 --2- 172.21.15.16:0/2892959433 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7feaa01666a0 0x7feaa0160670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:30.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.561+0000 7feaa5200640 1 -- 172.21.15.16:0/2892959433 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feaa00b3c00 con 0x7feaa013fa40 2024-03-31T15:09:30.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.565+0000 7feaa5200640 1 --2- 172.21.15.16:0/2892959433 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feaa013fa40 0x7feaa0166160 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x7fea8c009ae0 tx=0x7fea88005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:09:30.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.565+0000 7fea9d600640 1 -- 172.21.15.16:0/2892959433 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea8c0a6bc0 con 0x7feaa013fa40 2024-03-31T15:09:30.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.565+0000 7fea9d600640 1 -- 172.21.15.16:0/2892959433 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fea8c0a6d60 con 0x7feaa013fa40 2024-03-31T15:09:30.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.565+0000 7feaa8600640 1 -- 172.21.15.16:0/2892959433 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feaa0166bd0 con 0x7feaa013fa40 2024-03-31T15:09:30.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.565+0000 7fea9d600640 1 -- 172.21.15.16:0/2892959433 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea8c0a7690 con 0x7feaa013fa40 2024-03-31T15:09:30.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.565+0000 7feaa8600640 1 -- 172.21.15.16:0/2892959433 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feaa00b3400 con 0x7feaa013fa40 2024-03-31T15:09:30.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.565+0000 7fea9d600640 1 -- 172.21.15.16:0/2892959433 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fea8c0c2060 con 0x7feaa013fa40 2024-03-31T15:09:30.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.569+0000 7fea9d600640 1 --2- 172.21.15.16:0/2892959433 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fea800683b0 0x7fea8006a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:30.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.569+0000 7fea9d600640 1 -- 172.21.15.16:0/2892959433 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(70..70 src has 1..70) ==== 5882+0+0 (secure 0 0 0) 0x7fea8c1207b0 con 0x7feaa013fa40 2024-03-31T15:09:30.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.569+0000 7feaabb29000 1 Processor -- start 2024-03-31T15:09:30.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.569+0000 7feaabb29000 1 -- start start 2024-03-31T15:09:30.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.569+0000 7feaa5c00640 1 --2- 172.21.15.16:0/2892959433 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fea800683b0 0x7fea8006a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:30.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.569+0000 7feaabb29000 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55d948b3a980 0x55d948b3ad60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:30.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.569+0000 7feaabb29000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d948b3b2a0 0x55d948aeddd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:30.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.569+0000 7feaabb29000 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d948aee310 0x55d948aee6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:30.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.569+0000 7feaabb29000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55d94893ec50 con 0x55d948b3b2a0 2024-03-31T15:09:30.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.569+0000 7feaabb29000 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55d9488db8a0 con 0x55d948aee310 2024-03-31T15:09:30.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.569+0000 7feaabb29000 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x55d94893ac00 con 0x55d948b3a980 2024-03-31T15:09:30.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.569+0000 7feaa5c00640 1 --2- 172.21.15.16:0/2892959433 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fea800683b0 0x7fea8006a870 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7fea94007fd0 tx=0x7fea94001e20 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:09:30.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.569+0000 7feaa5200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d948b3b2a0 0x55d948aeddd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:30.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.569+0000 7feaa6600640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d948aee310 0x55d948aee6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:30.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.569+0000 7feaa5200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d948b3b2a0 0x55d948aeddd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60606/0 (socket says 172.21.15.16:60606) 2024-03-31T15:09:30.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.569+0000 7feaa5200640 1 -- 172.21.15.16:0/682837438 learned_addr learned my addr 172.21.15.16:0/682837438 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:09:30.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.569+0000 7feaa5c00640 1 --2- 172.21.15.16:0/682837438 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55d948b3a980 0x55d948b3ad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:30.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.569+0000 7feaa5200640 1 -- 172.21.15.16:0/682837438 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55d948b3a980 msgr2=0x55d948b3ad60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:09:30.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.569+0000 7feaa5200640 1 --2- 172.21.15.16:0/682837438 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55d948b3a980 0x55d948b3ad60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:30.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.569+0000 7feaa5200640 1 -- 172.21.15.16:0/682837438 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d948aee310 msgr2=0x55d948aee6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:09:30.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.569+0000 7feaa5200640 1 --2- 172.21.15.16:0/682837438 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d948aee310 0x55d948aee6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:30.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.569+0000 7feaa5200640 1 -- 172.21.15.16:0/682837438 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d948918ba0 con 0x55d948b3b2a0 2024-03-31T15:09:30.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.569+0000 7feaa5200640 1 --2- 172.21.15.16:0/682837438 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d948b3b2a0 0x55d948aeddd0 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7fea8c0ecad0 tx=0x7fea8c006f50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d6b63d77522557c6 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:09:30.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.569+0000 7fea92000640 1 -- 172.21.15.16:0/682837438 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea8c0f4970 con 0x55d948b3b2a0 2024-03-31T15:09:30.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.569+0000 7fea92000640 1 -- 172.21.15.16:0/682837438 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fea8c0f91a0 con 0x55d948b3b2a0 2024-03-31T15:09:30.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.569+0000 7fea92000640 1 -- 172.21.15.16:0/682837438 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea8c0ecce0 con 0x55d948b3b2a0 2024-03-31T15:09:30.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.569+0000 7feaabb29000 1 -- 172.21.15.16:0/682837438 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d948b3b2a0 msgr2=0x55d948aeddd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:30.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.569+0000 7feaabb29000 1 --2- 172.21.15.16:0/682837438 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d948b3b2a0 0x55d948aeddd0 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7fea8c0ecad0 tx=0x7fea8c006f50 comp rx=0 tx=0).stop 2024-03-31T15:09:30.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.569+0000 7feaabb29000 1 -- 172.21.15.16:0/682837438 shutdown_connections 2024-03-31T15:09:30.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.569+0000 7feaabb29000 1 --2- 172.21.15.16:0/682837438 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d948aee310 0x55d948aee6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:30.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.569+0000 7feaabb29000 1 --2- 172.21.15.16:0/682837438 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d948b3b2a0 0x55d948aeddd0 unknown :-1 s=CLOSED pgs=972 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:30.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.569+0000 7feaabb29000 1 --2- 172.21.15.16:0/682837438 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55d948b3a980 0x55d948b3ad60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:30.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.569+0000 7feaabb29000 1 -- 172.21.15.16:0/682837438 >> 172.21.15.16:0/682837438 conn(0x55d9488eb9f0 msgr2=0x55d948b1d730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:09:30.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.569+0000 7feaabb29000 1 -- 172.21.15.16:0/682837438 shutdown_connections 2024-03-31T15:09:30.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.569+0000 7feaabb29000 1 -- 172.21.15.16:0/682837438 wait complete. 2024-03-31T15:09:30.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.573+0000 7feaabb29000 1 -- start start 2024-03-31T15:09:30.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.573+0000 7feaabb29000 1 Processor -- start 2024-03-31T15:09:30.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.573+0000 7feaabb29000 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d948aee310 0x55d948af3c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:30.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.573+0000 7feaabb29000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d948b3a980 0x55d948af41b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:30.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.573+0000 7feaabb29000 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55d948b3b2a0 0x55d948af46f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:30.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.573+0000 7feaabb29000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55d94893ec50 con 0x55d948b3a980 2024-03-31T15:09:30.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.573+0000 7feaabb29000 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55d9488fc0a0 con 0x55d948aee310 2024-03-31T15:09:30.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.573+0000 7feaabb29000 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x55d9488db8a0 con 0x55d948b3b2a0 2024-03-31T15:09:30.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.573+0000 7feaabb29000 10 client.0 _ll_register_callbacks cb 0x55d948adb810 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T15:09:30.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.573+0000 7feaa5200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d948b3a980 0x55d948af41b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:30.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.573+0000 7feaa5200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d948b3a980 0x55d948af41b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60612/0 (socket says 172.21.15.16:60612) 2024-03-31T15:09:30.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.573+0000 7feaa5200640 1 -- 172.21.15.16:0/1269626064 learned_addr learned my addr 172.21.15.16:0/1269626064 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:09:30.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.573+0000 7feaa6600640 1 --2- 172.21.15.16:0/1269626064 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55d948b3b2a0 0x55d948af46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:30.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.573+0000 7feaa5c00640 1 --2- 172.21.15.16:0/1269626064 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d948aee310 0x55d948af3c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:30.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.573+0000 7feaa5200640 1 -- 172.21.15.16:0/1269626064 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55d948b3b2a0 msgr2=0x55d948af46f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:09:30.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.573+0000 7feaa5200640 1 --2- 172.21.15.16:0/1269626064 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55d948b3b2a0 0x55d948af46f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:30.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.573+0000 7feaa5200640 1 -- 172.21.15.16:0/1269626064 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d948aee310 msgr2=0x55d948af3c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:09:30.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.573+0000 7feaa5200640 1 --2- 172.21.15.16:0/1269626064 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d948aee310 0x55d948af3c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:30.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.573+0000 7feaa5200640 1 -- 172.21.15.16:0/1269626064 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d948984160 con 0x55d948b3a980 2024-03-31T15:09:30.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.573+0000 7feaa6600640 1 --2- 172.21.15.16:0/1269626064 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55d948b3b2a0 0x55d948af46f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:09:30.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.573+0000 7feaa5200640 1 --2- 172.21.15.16:0/1269626064 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d948b3a980 0x55d948af41b0 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7fea8c11ec80 tx=0x7fea8c01ca20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d61b861df9998ac8 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:09:30.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.573+0000 7fea86a00640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-31T15:09:30.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.573+0000 7fea86a00640 1 -- 172.21.15.16:0/1269626064 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea8c10d060 con 0x55d948b3a980 2024-03-31T15:09:30.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.573+0000 7feaabb29000 1 -- 172.21.15.16:0/1269626064 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fea8c0f91a0 con 0x55d948b3a980 2024-03-31T15:09:30.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.573+0000 7fea86a00640 1 -- 172.21.15.16:0/1269626064 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(70..70 src has 1..70) ==== 5882+0+0 (secure 0 0 0) 0x7fea8c0f8f10 con 0x55d948b3a980 2024-03-31T15:09:30.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.573+0000 7fea86a00640 1 -- 172.21.15.16:0/1269626064 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=52) ==== 24+0+0 (secure 0 0 0) 0x7fea8c114050 con 0x55d948b3a980 2024-03-31T15:09:30.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.573+0000 7feaabb29000 10 client.7251 fetch_fsmap learned FSMap version 52 2024-03-31T15:09:30.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.573+0000 7feaabb29000 1 -- 172.21.15.16:0/1269626064 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=52,monmap=2+}) -- 0x55d9488d99c0 con 0x55d948b3a980 2024-03-31T15:09:30.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.573+0000 7fea86a00640 1 -- 172.21.15.16:0/1269626064 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fea8c0f91a0 con 0x55d948b3a980 2024-03-31T15:09:30.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.573+0000 7fea86a00640 1 -- 172.21.15.16:0/1269626064 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 52) ==== 3494+0+0 (secure 0 0 0) 0x7fea8c0efac0 con 0x55d948b3a980 2024-03-31T15:09:30.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.573+0000 7feaabb29000 10 client.7251 fetch_fsmap finished waiting for FSMap version 52 2024-03-31T15:09:30.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.573+0000 7feaabb29000 10 client.7251 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-31T15:09:30.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.573+0000 7feaabb29000 20 client.7251 populate_metadata read hostname 'smithi016' 2024-03-31T15:09:30.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.573+0000 7feaabb29000 1 --2- 172.21.15.16:0/1269626064 >> [v2:172.21.15.16:6834/2645389320,v1:172.21.15.16:6835/2645389320] conn(0x55d948af4f60 0x55d948b4bef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:30.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.573+0000 7feaabb29000 4 client.7251 mds_command: new command op to 6829 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-31T15:09:30.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.573+0000 7feaabb29000 1 -- 172.21.15.16:0/1269626064 --> [v2:172.21.15.16:6834/2645389320,v1:172.21.15.16:6835/2645389320] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d9488d9e40 con 0x55d948af4f60 2024-03-31T15:09:30.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.573+0000 7feaa5c00640 1 --2- 172.21.15.16:0/1269626064 >> [v2:172.21.15.16:6834/2645389320,v1:172.21.15.16:6835/2645389320] conn(0x55d948af4f60 0x55d948b4bef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:30.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.573+0000 7fea86a00640 10 client.7251 ms_handle_connect on v2:172.21.15.16:6834/2645389320 2024-03-31T15:09:30.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.573+0000 7feaa5c00640 1 --2- 172.21.15.16:0/1269626064 >> [v2:172.21.15.16:6834/2645389320,v1:172.21.15.16:6835/2645389320] conn(0x55d948af4f60 0x55d948b4bef0 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3687650f0bda9ea server_cookie=fccdfe0a7c8ea3fc in_seq=0 out_seq=0 2024-03-31T15:09:30.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.577+0000 7fea86a00640 1 -- 172.21.15.16:0/1269626064 <== mds.0 v2:172.21.15.16:6834/2645389320 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55d9488d9e40 con 0x55d948af4f60 2024-03-31T15:09:30.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.577+0000 7fea86a00640 10 client.7251 handle_command_reply: tid=0 2024-03-31T15:09:30.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.577+0000 7feaabb29000 1 client.7251 shutdown 2024-03-31T15:09:30.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.577+0000 7feaabb29000 1 -- 172.21.15.16:0/1269626064 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d948b3a980 msgr2=0x55d948af41b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:30.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.577+0000 7feaabb29000 1 --2- 172.21.15.16:0/1269626064 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d948b3a980 0x55d948af41b0 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7fea8c11ec80 tx=0x7fea8c01ca20 comp rx=0 tx=0).stop 2024-03-31T15:09:30.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.577+0000 7feaabb29000 1 -- 172.21.15.16:0/1269626064 shutdown_connections 2024-03-31T15:09:30.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.577+0000 7feaabb29000 1 --2- 172.21.15.16:0/1269626064 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55d948b3b2a0 0x55d948af46f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:30.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.577+0000 7feaabb29000 1 --2- 172.21.15.16:0/1269626064 >> [v2:172.21.15.16:6834/2645389320,v1:172.21.15.16:6835/2645389320] conn(0x55d948af4f60 0x55d948b4bef0 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:30.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.577+0000 7feaabb29000 1 --2- 172.21.15.16:0/1269626064 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d948b3a980 0x55d948af41b0 unknown :-1 s=CLOSED pgs=973 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:30.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.577+0000 7feaabb29000 1 --2- 172.21.15.16:0/1269626064 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d948aee310 0x55d948af3c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:30.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.577+0000 7feaabb29000 1 -- 172.21.15.16:0/1269626064 >> 172.21.15.16:0/1269626064 conn(0x55d9488eb9f0 msgr2=0x55d948b1d520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:09:30.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.577+0000 7fea86a00640 0 client.7251 ms_handle_reset on v2:172.21.15.16:6834/2645389320 2024-03-31T15:09:30.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.577+0000 7feaabb29000 1 -- 172.21.15.16:0/1269626064 shutdown_connections 2024-03-31T15:09:30.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.577+0000 7feaabb29000 1 -- 172.21.15.16:0/1269626064 wait complete. 2024-03-31T15:09:30.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.577+0000 7feaabb29000 20 client.7251 trim_cache size 0 max 16384 2024-03-31T15:09:30.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.589+0000 7feaabb29000 1 Processor -- start 2024-03-31T15:09:30.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.589+0000 7feaabb29000 1 -- start start 2024-03-31T15:09:30.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.589+0000 7feaabb29000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d948aee310 0x55d948aecc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:30.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.589+0000 7feaabb29000 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55d948b3a980 0x55d948aed180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:30.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.589+0000 7feaabb29000 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d948b3b2a0 0x55d948b39c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:30.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.589+0000 7feaabb29000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7feaa0163a00 con 0x55d948aee310 2024-03-31T15:09:30.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.589+0000 7feaabb29000 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7feaa0202d00 con 0x55d948b3b2a0 2024-03-31T15:09:30.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.589+0000 7feaabb29000 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x55d94893ac00 con 0x55d948b3a980 2024-03-31T15:09:30.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.589+0000 7feaa5c00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d948aee310 0x55d948aecc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:30.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.593+0000 7feaa5c00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d948aee310 0x55d948aecc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60616/0 (socket says 172.21.15.16:60616) 2024-03-31T15:09:30.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.593+0000 7feaa5c00640 1 -- 172.21.15.16:0/1288229615 learned_addr learned my addr 172.21.15.16:0/1288229615 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:09:30.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.593+0000 7feaa5200640 1 --2- 172.21.15.16:0/1288229615 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55d948b3a980 0x55d948aed180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:30.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.593+0000 7feaa6600640 1 --2- 172.21.15.16:0/1288229615 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d948b3b2a0 0x55d948b39c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:30.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.593+0000 7feaa5c00640 1 -- 172.21.15.16:0/1288229615 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55d948b3a980 msgr2=0x55d948aed180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:09:30.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.593+0000 7feaa5c00640 1 --2- 172.21.15.16:0/1288229615 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55d948b3a980 0x55d948aed180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:30.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.593+0000 7feaa5c00640 1 -- 172.21.15.16:0/1288229615 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d948b3b2a0 msgr2=0x55d948b39c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:09:30.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.593+0000 7feaa5c00640 1 --2- 172.21.15.16:0/1288229615 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d948b3b2a0 0x55d948b39c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:30.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.593+0000 7feaa5c00640 1 -- 172.21.15.16:0/1288229615 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d9488dbaf0 con 0x55d948aee310 2024-03-31T15:09:30.585 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.593+0000 7feaa5c00640 1 --2- 172.21.15.16:0/1288229615 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d948aee310 0x55d948aecc40 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7fea94007c60 tx=0x7fea9400c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=410271e00afbd408 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:09:30.585 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.593+0000 7fea91600640 1 -- 172.21.15.16:0/1288229615 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea9402a2d0 con 0x55d948aee310 2024-03-31T15:09:30.585 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.593+0000 7fea91600640 1 -- 172.21.15.16:0/1288229615 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fea9402a470 con 0x55d948aee310 2024-03-31T15:09:30.585 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.593+0000 7fea91600640 1 -- 172.21.15.16:0/1288229615 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea9402a620 con 0x55d948aee310 2024-03-31T15:09:30.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.593+0000 7feaabb29000 1 -- 172.21.15.16:0/1288229615 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d948aee310 msgr2=0x55d948aecc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:30.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.593+0000 7feaabb29000 1 --2- 172.21.15.16:0/1288229615 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d948aee310 0x55d948aecc40 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7fea94007c60 tx=0x7fea9400c000 comp rx=0 tx=0).stop 2024-03-31T15:09:30.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.593+0000 7feaabb29000 1 -- 172.21.15.16:0/1288229615 shutdown_connections 2024-03-31T15:09:30.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.593+0000 7feaabb29000 1 --2- 172.21.15.16:0/1288229615 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d948b3b2a0 0x55d948b39c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:30.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.593+0000 7feaabb29000 1 --2- 172.21.15.16:0/1288229615 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55d948b3a980 0x55d948aed180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:30.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.593+0000 7feaabb29000 1 --2- 172.21.15.16:0/1288229615 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d948aee310 0x55d948aecc40 unknown :-1 s=CLOSED pgs=974 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:30.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.593+0000 7feaabb29000 1 -- 172.21.15.16:0/1288229615 >> 172.21.15.16:0/1288229615 conn(0x55d9488eb9f0 msgr2=0x55d948aec530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:09:30.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.593+0000 7feaabb29000 1 -- 172.21.15.16:0/1288229615 shutdown_connections 2024-03-31T15:09:30.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.593+0000 7feaabb29000 1 -- 172.21.15.16:0/1288229615 wait complete. 2024-03-31T15:09:30.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.593+0000 7feaabb29000 1 -- start start 2024-03-31T15:09:30.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.593+0000 7feaabb29000 1 Processor -- start 2024-03-31T15:09:30.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.593+0000 7feaabb29000 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55d948aee310 0x55d948bb2ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:30.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.593+0000 7feaabb29000 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d948b3a980 0x55d948bb2ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:30.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.593+0000 7feaabb29000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d948b3b2a0 0x55d948b4cf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:30.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.593+0000 7feaabb29000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55d94893ec50 con 0x55d948b3b2a0 2024-03-31T15:09:30.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.593+0000 7feaabb29000 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55d9488ff400 con 0x55d948b3a980 2024-03-31T15:09:30.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.593+0000 7feaabb29000 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x55d9488fc0a0 con 0x55d948aee310 2024-03-31T15:09:30.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.593+0000 7feaabb29000 10 client.0 _ll_register_callbacks cb 0x7fea98009210 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T15:09:30.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.593+0000 7feaa6600640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d948b3b2a0 0x55d948b4cf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:30.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.593+0000 7feaa5c00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55d948aee310 0x55d948bb2ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:30.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.593+0000 7feaa6600640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d948b3b2a0 0x55d948b4cf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60618/0 (socket says 172.21.15.16:60618) 2024-03-31T15:09:30.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.593+0000 7feaa5c00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55d948aee310 0x55d948bb2ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3301/0 says I am v2:172.21.15.16:55456/0 (socket says 172.21.15.16:55456) 2024-03-31T15:09:30.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.593+0000 7feaa6600640 1 -- 172.21.15.16:0/1454600270 learned_addr learned my addr 172.21.15.16:0/1454600270 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:09:30.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.593+0000 7feaa5200640 1 --2- 172.21.15.16:0/1454600270 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d948b3a980 0x55d948bb2ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:30.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.597+0000 7feaa6600640 1 -- 172.21.15.16:0/1454600270 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55d948aee310 msgr2=0x55d948bb2ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:09:30.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.597+0000 7feaa6600640 1 --2- 172.21.15.16:0/1454600270 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55d948aee310 0x55d948bb2ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:30.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.597+0000 7feaa6600640 1 -- 172.21.15.16:0/1454600270 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d948b3a980 msgr2=0x55d948bb2ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:09:30.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.597+0000 7feaa6600640 1 --2- 172.21.15.16:0/1454600270 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d948b3a980 0x55d948bb2ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:30.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.597+0000 7feaa6600640 1 -- 172.21.15.16:0/1454600270 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d948997320 con 0x55d948b3b2a0 2024-03-31T15:09:30.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.597+0000 7feaa5200640 1 --2- 172.21.15.16:0/1454600270 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d948b3a980 0x55d948bb2ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:09:30.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.597+0000 7feaa6600640 1 --2- 172.21.15.16:0/1454600270 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d948b3b2a0 0x55d948b4cf40 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7feaa0158fb0 tx=0x7fea98015760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=275f5f0b2f34c5dd server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:09:30.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.597+0000 7fea86a00640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-31T15:09:30.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.597+0000 7fea86a00640 1 -- 172.21.15.16:0/1454600270 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea94007c90 con 0x55d948b3b2a0 2024-03-31T15:09:30.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.597+0000 7fea86a00640 1 -- 172.21.15.16:0/1454600270 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(70..70 src has 1..70) ==== 5882+0+0 (secure 0 0 0) 0x7fea98003ba0 con 0x55d948b3b2a0 2024-03-31T15:09:30.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.597+0000 7feaabb29000 1 -- 172.21.15.16:0/1454600270 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fea9402a470 con 0x55d948b3b2a0 2024-03-31T15:09:30.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.597+0000 7fea86a00640 1 -- 172.21.15.16:0/1454600270 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=52) ==== 24+0+0 (secure 0 0 0) 0x7fea9802c050 con 0x55d948b3b2a0 2024-03-31T15:09:30.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.597+0000 7feaabb29000 10 client.7257 fetch_fsmap learned FSMap version 52 2024-03-31T15:09:30.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.597+0000 7feaabb29000 1 -- 172.21.15.16:0/1454600270 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=52,monmap=2+}) -- 0x55d9488d2a80 con 0x55d948b3b2a0 2024-03-31T15:09:30.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.597+0000 7fea86a00640 1 -- 172.21.15.16:0/1454600270 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fea9402a470 con 0x55d948b3b2a0 2024-03-31T15:09:30.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.597+0000 7fea86a00640 1 -- 172.21.15.16:0/1454600270 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 52) ==== 3494+0+0 (secure 0 0 0) 0x7fea9802f020 con 0x55d948b3b2a0 2024-03-31T15:09:30.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.597+0000 7feaabb29000 10 client.7257 fetch_fsmap finished waiting for FSMap version 52 2024-03-31T15:09:30.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.597+0000 7feaabb29000 10 client.7257 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-31T15:09:30.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.597+0000 7feaabb29000 20 client.7257 populate_metadata read hostname 'smithi016' 2024-03-31T15:09:30.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.597+0000 7feaabb29000 1 --2- 172.21.15.16:0/1454600270 >> [v2:172.21.15.16:6834/2645389320,v1:172.21.15.16:6835/2645389320] conn(0x55d948bb9980 0x55d948b4dd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:30.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.597+0000 7feaabb29000 4 client.7257 mds_command: new command op to 6829 tid=0 multi_id=0 [{"prefix": "damage ls", "format": "json-pretty"}] 2024-03-31T15:09:30.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.597+0000 7feaabb29000 1 -- 172.21.15.16:0/1454600270 --> [v2:172.21.15.16:6834/2645389320,v1:172.21.15.16:6835/2645389320] -- command(tid 0: {"prefix": "damage ls", "format": "json-pretty"}) -- 0x55d94894c190 con 0x55d948bb9980 2024-03-31T15:09:30.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.597+0000 7feaa5c00640 1 --2- 172.21.15.16:0/1454600270 >> [v2:172.21.15.16:6834/2645389320,v1:172.21.15.16:6835/2645389320] conn(0x55d948bb9980 0x55d948b4dd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:30.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.597+0000 7feaa5c00640 1 --2- 172.21.15.16:0/1454600270 >> [v2:172.21.15.16:6834/2645389320,v1:172.21.15.16:6835/2645389320] conn(0x55d948bb9980 0x55d948b4dd10 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d70f5c25069afec8 server_cookie=cdc45df976dd3a2d in_seq=0 out_seq=0 2024-03-31T15:09:30.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.597+0000 7fea86a00640 10 client.7257 ms_handle_connect on v2:172.21.15.16:6834/2645389320 2024-03-31T15:09:30.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.597+0000 7fea86a00640 1 -- 172.21.15.16:0/1454600270 <== mds.0 v2:172.21.15.16:6834/2645389320 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55d94894c190 con 0x55d948bb9980 2024-03-31T15:09:30.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.597+0000 7fea86a00640 10 client.7257 handle_command_reply: tid=0 2024-03-31T15:09:30.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.597+0000 7feaabb29000 1 client.7257 shutdown 2024-03-31T15:09:30.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.597+0000 7feaabb29000 1 -- 172.21.15.16:0/1454600270 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d948b3b2a0 msgr2=0x55d948b4cf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:30.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.597+0000 7feaabb29000 1 --2- 172.21.15.16:0/1454600270 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d948b3b2a0 0x55d948b4cf40 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7feaa0158fb0 tx=0x7fea98015760 comp rx=0 tx=0).stop 2024-03-31T15:09:30.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.597+0000 7feaabb29000 1 -- 172.21.15.16:0/1454600270 shutdown_connections 2024-03-31T15:09:30.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.597+0000 7feaabb29000 1 --2- 172.21.15.16:0/1454600270 >> [v2:172.21.15.16:6834/2645389320,v1:172.21.15.16:6835/2645389320] conn(0x55d948bb9980 0x55d948b4dd10 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:30.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.597+0000 7feaabb29000 1 --2- 172.21.15.16:0/1454600270 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d948b3b2a0 0x55d948b4cf40 unknown :-1 s=CLOSED pgs=975 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:30.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.597+0000 7feaabb29000 1 --2- 172.21.15.16:0/1454600270 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d948b3a980 0x55d948bb2ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:30.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.597+0000 7fea86a00640 0 client.7257 ms_handle_reset on v2:172.21.15.16:6834/2645389320 2024-03-31T15:09:30.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.597+0000 7feaabb29000 1 --2- 172.21.15.16:0/1454600270 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55d948aee310 0x55d948bb2ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:30.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.597+0000 7feaabb29000 1 -- 172.21.15.16:0/1454600270 >> 172.21.15.16:0/1454600270 conn(0x55d9488eb9f0 msgr2=0x55d948aecdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:09:30.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.597+0000 7feaabb29000 1 -- 172.21.15.16:0/1454600270 shutdown_connections 2024-03-31T15:09:30.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.597+0000 7feaabb29000 1 -- 172.21.15.16:0/1454600270 wait complete. 2024-03-31T15:09:30.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.597+0000 7feaabb29000 20 client.7257 trim_cache size 0 max 16384 2024-03-31T15:09:30.594 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:09:30.594 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-03-31T15:09:30.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.597+0000 7feaa8600640 1 -- 172.21.15.16:0/2892959433 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fea800683b0 msgr2=0x7fea8006a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:30.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.597+0000 7feaa8600640 1 --2- 172.21.15.16:0/2892959433 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fea800683b0 0x7fea8006a870 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7fea94007fd0 tx=0x7fea94001e20 comp rx=0 tx=0).stop 2024-03-31T15:09:30.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.597+0000 7feaa8600640 1 -- 172.21.15.16:0/2892959433 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feaa013fa40 msgr2=0x7feaa0166160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:30.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.597+0000 7feaa8600640 1 --2- 172.21.15.16:0/2892959433 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feaa013fa40 0x7feaa0166160 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x7fea8c009ae0 tx=0x7fea88005840 comp rx=0 tx=0).stop 2024-03-31T15:09:30.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.597+0000 7feaa8600640 1 -- 172.21.15.16:0/2892959433 shutdown_connections 2024-03-31T15:09:30.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.597+0000 7feaa8600640 1 --2- 172.21.15.16:0/2892959433 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fea800683b0 0x7fea8006a870 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:30.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.597+0000 7feaa8600640 1 --2- 172.21.15.16:0/2892959433 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7feaa01666a0 0x7feaa0160670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:30.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.597+0000 7feaa8600640 1 --2- 172.21.15.16:0/2892959433 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feaa013fa40 0x7feaa0166160 unknown :-1 s=CLOSED pgs=971 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:30.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.597+0000 7feaa8600640 1 --2- 172.21.15.16:0/2892959433 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fea88004810 0x7feaa0153650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:30.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.597+0000 7feaa8600640 1 -- 172.21.15.16:0/2892959433 >> 172.21.15.16:0/2892959433 conn(0x7feaa0067930 msgr2=0x7feaa0141fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:09:30.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.597+0000 7feaa8600640 1 -- 172.21.15.16:0/2892959433 shutdown_connections 2024-03-31T15:09:30.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.597+0000 7feaa8600640 1 -- 172.21.15.16:0/2892959433 wait complete. 2024-03-31T15:09:30.609 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-03-31T15:09:30.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.757+0000 7fa5dc400640 1 Processor -- start 2024-03-31T15:09:30.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.757+0000 7fa5dc400640 1 -- start start 2024-03-31T15:09:30.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.757+0000 7fa5dc400640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fa5d41411b0 0x7fa5d406f920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:30.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.757+0000 7fa5dc400640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fa5d4144340 0x7fa5d406fe60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:30.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.757+0000 7fa5dc400640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa5d4144980 con 0x7fa5d4141590 2024-03-31T15:09:30.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.757+0000 7fa5dc400640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7fa5d415f910 con 0x7fa5d41411b0 2024-03-31T15:09:30.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.757+0000 7fa5dc400640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fa5d4162910 con 0x7fa5d4144340 2024-03-31T15:09:30.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.757+0000 7fa5d9a00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fa5d41411b0 0x7fa5d406f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:30.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.757+0000 7fa5d9000640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fa5d4141590 0x7fa5d4143c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:46664/0 (socket says 172.21.15.16:46664) 2024-03-31T15:09:30.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.757+0000 7fa5d9000640 1 -- 172.21.15.16:0/2709244107 learned_addr learned my addr 172.21.15.16:0/2709244107 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:09:30.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.757+0000 7fa5da400640 1 --2- 172.21.15.16:0/2709244107 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fa5d4144340 0x7fa5d406fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:30.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.761+0000 7fa5d9a00640 1 -- 172.21.15.16:0/2709244107 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fa5d4144340 msgr2=0x7fa5d406fe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:09:30.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.761+0000 7fa5d9a00640 1 --2- 172.21.15.16:0/2709244107 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fa5d4144340 0x7fa5d406fe60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:30.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.761+0000 7fa5d9a00640 1 -- 172.21.15.16:0/2709244107 >> v1:172.21.15.16:6789/0 conn(0x7fa5d4141590 legacy=0x7fa5d4143c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:30.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.761+0000 7fa5d9a00640 1 -- 172.21.15.16:0/2709244107 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5d4161fe0 con 0x7fa5d41411b0 2024-03-31T15:09:30.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.761+0000 7fa5d3e00640 1 -- 172.21.15.16:0/2709244107 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1955624931 0 0) 0x7fa5d4144980 con 0x7fa5d4141590 2024-03-31T15:09:30.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.761+0000 7fa5d9a00640 1 --2- 172.21.15.16:0/2709244107 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa5d41411b0 0x7fa5d406f920 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7fa5c80097c0 tx=0x7fa5c801c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27eff01ad339a36c server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:09:30.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.761+0000 7fa5d3e00640 1 -- 172.21.15.16:0/2709244107 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5c80b2070 con 0x7fa5d41411b0 2024-03-31T15:09:30.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.761+0000 7fa5d3e00640 1 -- 172.21.15.16:0/2709244107 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa5c80ad640 con 0x7fa5d41411b0 2024-03-31T15:09:30.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.761+0000 7fa5d3e00640 1 -- 172.21.15.16:0/2709244107 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5c80ada10 con 0x7fa5d41411b0 2024-03-31T15:09:30.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.761+0000 7fa5dc400640 1 -- 172.21.15.16:0/2709244107 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa5d41411b0 msgr2=0x7fa5d406f920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:30.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.761+0000 7fa5dc400640 1 --2- 172.21.15.16:0/2709244107 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa5d41411b0 0x7fa5d406f920 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7fa5c80097c0 tx=0x7fa5c801c4a0 comp rx=0 tx=0).stop 2024-03-31T15:09:30.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.761+0000 7fa5dc400640 1 -- 172.21.15.16:0/2709244107 shutdown_connections 2024-03-31T15:09:30.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.761+0000 7fa5dc400640 1 --2- 172.21.15.16:0/2709244107 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fa5d4144340 0x7fa5d406fe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:30.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.761+0000 7fa5dc400640 1 --2- 172.21.15.16:0/2709244107 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa5d41411b0 0x7fa5d406f920 unknown :-1 s=CLOSED pgs=977 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:30.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.761+0000 7fa5dc400640 1 -- 172.21.15.16:0/2709244107 >> 172.21.15.16:0/2709244107 conn(0x7fa5d4067930 msgr2=0x7fa5d40b3ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:09:30.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.761+0000 7fa5dc400640 1 -- 172.21.15.16:0/2709244107 shutdown_connections 2024-03-31T15:09:30.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.761+0000 7fa5dc400640 1 -- 172.21.15.16:0/2709244107 wait complete. 2024-03-31T15:09:30.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.765+0000 7fa5dc400640 1 Processor -- start 2024-03-31T15:09:30.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.765+0000 7fa5dc400640 1 -- start start 2024-03-31T15:09:30.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.765+0000 7fa5dc400640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fa5d41411b0 0x7fa5d415a6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:30.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.765+0000 7fa5dc400640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa5d4141590 0x7fa5d415c590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:30.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.765+0000 7fa5dc400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa5d4144340 0x7fa5d40b2ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:30.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.765+0000 7fa5dc400640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fa5d415f290 con 0x7fa5d4144340 2024-03-31T15:09:30.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.765+0000 7fa5dc400640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fa5d415f110 con 0x7fa5d4141590 2024-03-31T15:09:30.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.765+0000 7fa5dc400640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fa5d415f410 con 0x7fa5d41411b0 2024-03-31T15:09:30.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.765+0000 7fa5da400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa5d4144340 0x7fa5d40b2ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:30.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.765+0000 7fa5d9000640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa5d4141590 0x7fa5d415c590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:30.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.765+0000 7fa5d9a00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fa5d41411b0 0x7fa5d415a6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:30.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.765+0000 7fa5da400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa5d4144340 0x7fa5d40b2ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60642/0 (socket says 172.21.15.16:60642) 2024-03-31T15:09:30.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.765+0000 7fa5da400640 1 -- 172.21.15.16:0/1709945819 learned_addr learned my addr 172.21.15.16:0/1709945819 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:09:30.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.765+0000 7fa5da400640 1 -- 172.21.15.16:0/1709945819 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fa5d41411b0 msgr2=0x7fa5d415a6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:30.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.765+0000 7fa5da400640 1 --2- 172.21.15.16:0/1709945819 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fa5d41411b0 0x7fa5d415a6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:30.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.765+0000 7fa5da400640 1 -- 172.21.15.16:0/1709945819 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa5d4141590 msgr2=0x7fa5d415c590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:30.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.765+0000 7fa5da400640 1 --2- 172.21.15.16:0/1709945819 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa5d4141590 0x7fa5d415c590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:30.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.765+0000 7fa5da400640 1 -- 172.21.15.16:0/1709945819 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5d415ad20 con 0x7fa5d4144340 2024-03-31T15:09:30.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.765+0000 7fa5d9000640 1 --2- 172.21.15.16:0/1709945819 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa5d4141590 0x7fa5d415c590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:09:30.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.765+0000 7fa5d9a00640 1 --2- 172.21.15.16:0/1709945819 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fa5d41411b0 0x7fa5d415a6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:09:30.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.765+0000 7fa5da400640 1 --2- 172.21.15.16:0/1709945819 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa5d4144340 0x7fa5d40b2ca0 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7fa5cc007e60 tx=0x7fa5cc00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:09:30.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.765+0000 7fa5d1600640 1 -- 172.21.15.16:0/1709945819 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5cc014420 con 0x7fa5d4144340 2024-03-31T15:09:30.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.765+0000 7fa5d1600640 1 -- 172.21.15.16:0/1709945819 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa5cc019070 con 0x7fa5d4144340 2024-03-31T15:09:30.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.765+0000 7fa5d1600640 1 -- 172.21.15.16:0/1709945819 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5cc014820 con 0x7fa5d4144340 2024-03-31T15:09:30.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.765+0000 7fa5dc400640 1 -- 172.21.15.16:0/1709945819 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa5d415b000 con 0x7fa5d4144340 2024-03-31T15:09:30.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.769+0000 7fa5dc400640 1 -- 172.21.15.16:0/1709945819 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa5d415b5c0 con 0x7fa5d4144340 2024-03-31T15:09:30.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.769+0000 7fa5dc400640 1 -- 172.21.15.16:0/1709945819 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa5d4069950 con 0x7fa5d4144340 2024-03-31T15:09:30.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.769+0000 7fa5d1600640 1 -- 172.21.15.16:0/1709945819 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fa5cc0149c0 con 0x7fa5d4144340 2024-03-31T15:09:30.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.769+0000 7fa5d1600640 1 --2- 172.21.15.16:0/1709945819 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fa5b4068380 0x7fa5b406a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:30.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.769+0000 7fa5d1600640 1 -- 172.21.15.16:0/1709945819 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(70..70 src has 1..70) ==== 5882+0+0 (secure 0 0 0) 0x7fa5cc087190 con 0x7fa5d4144340 2024-03-31T15:09:30.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.773+0000 7fa5d9a00640 1 --2- 172.21.15.16:0/1709945819 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fa5b4068380 0x7fa5b406a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:30.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.773+0000 7fa5d9a00640 1 --2- 172.21.15.16:0/1709945819 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fa5b4068380 0x7fa5b406a840 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7fa5c80af000 tx=0x7fa5c8008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:09:30.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.781+0000 7fa5d1600640 1 -- 172.21.15.16:0/1709945819 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fa5cc057040 con 0x7fa5d4144340 2024-03-31T15:09:30.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.977+0000 7fa5dc400640 1 -- 172.21.15.16:0/1709945819 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7fa5d4161350 con 0x7fa5d4144340 2024-03-31T15:09:30.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.977+0000 7fa5d1600640 1 -- 172.21.15.16:0/1709945819 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+46 (secure 0 0 0) 0x7fa5cc05b340 con 0x7fa5d4144340 2024-03-31T15:09:30.970 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:09:30.970 INFO:teuthology.orchestra.run.smithi016.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-03-31T15:09:30.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.981+0000 7fa5dc400640 1 -- 172.21.15.16:0/1709945819 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fa5b4068380 msgr2=0x7fa5b406a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:30.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.981+0000 7fa5dc400640 1 --2- 172.21.15.16:0/1709945819 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fa5b4068380 0x7fa5b406a840 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7fa5c80af000 tx=0x7fa5c8008040 comp rx=0 tx=0).stop 2024-03-31T15:09:30.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.981+0000 7fa5dc400640 1 -- 172.21.15.16:0/1709945819 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa5d4144340 msgr2=0x7fa5d40b2ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:30.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.981+0000 7fa5dc400640 1 --2- 172.21.15.16:0/1709945819 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa5d4144340 0x7fa5d40b2ca0 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7fa5cc007e60 tx=0x7fa5cc00e4b0 comp rx=0 tx=0).stop 2024-03-31T15:09:30.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.981+0000 7fa5dc400640 1 -- 172.21.15.16:0/1709945819 shutdown_connections 2024-03-31T15:09:30.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.981+0000 7fa5dc400640 1 --2- 172.21.15.16:0/1709945819 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fa5b4068380 0x7fa5b406a840 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:30.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.981+0000 7fa5dc400640 1 --2- 172.21.15.16:0/1709945819 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa5d4144340 0x7fa5d40b2ca0 unknown :-1 s=CLOSED pgs=978 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:30.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.981+0000 7fa5dc400640 1 --2- 172.21.15.16:0/1709945819 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa5d4141590 0x7fa5d415c590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:30.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.981+0000 7fa5dc400640 1 --2- 172.21.15.16:0/1709945819 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fa5d41411b0 0x7fa5d415a6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:30.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.981+0000 7fa5dc400640 1 -- 172.21.15.16:0/1709945819 >> 172.21.15.16:0/1709945819 conn(0x7fa5d4067930 msgr2=0x7fa5d40b56a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:09:30.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.981+0000 7fa5dc400640 1 -- 172.21.15.16:0/1709945819 shutdown_connections 2024-03-31T15:09:30.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:30.981+0000 7fa5dc400640 1 -- 172.21.15.16:0/1709945819 wait complete. 2024-03-31T15:09:30.991 DEBUG:tasks.ceph_test_case:wait_until_true: success in 0s and 0 retries 2024-03-31T15:09:30.992 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-31T15:09:31.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:31.141+0000 7f9803e00640 1 Processor -- start 2024-03-31T15:09:31.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:31.141+0000 7f9803e00640 1 -- start start 2024-03-31T15:09:31.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:31.141+0000 7f9803e00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f97fc070030 0x7f97fc162180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:31.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:31.141+0000 7f9803e00640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f97fc0cbe90 con 0x7f97fc15f650 2024-03-31T15:09:31.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:31.141+0000 7f9803e00640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f97fc0cc170 con 0x7f97fc06dbc0 2024-03-31T15:09:31.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:31.141+0000 7f9803e00640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f97fc15f1c0 con 0x7f97fc070030 2024-03-31T15:09:31.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:31.141+0000 7f9800a00640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f97fc15f650 0x7f97fc06f920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:46668/0 (socket says 172.21.15.16:46668) 2024-03-31T15:09:31.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:31.141+0000 7f9800a00640 1 -- 172.21.15.16:0/3287831752 learned_addr learned my addr 172.21.15.16:0/3287831752 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:09:31.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:31.141+0000 7f9801e00640 1 --2- 172.21.15.16:0/3287831752 >> v2:172.21.15.89:3300/0 conn(0x7f97fc070030 0x7f97fc162180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:31.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:31.141+0000 7f97fbe00640 1 -- 172.21.15.16:0/3287831752 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1870370646 0 0) 0x7f97fc0cbe90 con 0x7f97fc15f650 2024-03-31T15:09:31.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:31.141+0000 7f97fbe00640 1 -- 172.21.15.16:0/3287831752 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f97e4003530 con 0x7f97fc15f650 2024-03-31T15:09:31.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:31.141+0000 7f97fbe00640 1 -- 172.21.15.16:0/3287831752 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1525628114 0 0) 0x7f97e4003530 con 0x7f97fc15f650 2024-03-31T15:09:31.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:31.141+0000 7f9801e00640 1 -- 172.21.15.16:0/3287831752 >> v1:172.21.15.89:6789/0 conn(0x7f97fc06dbc0 legacy=0x7f97fc0cb440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:31.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:31.141+0000 7f9801e00640 1 -- 172.21.15.16:0/3287831752 >> v1:172.21.15.16:6789/0 conn(0x7f97fc15f650 legacy=0x7f97fc06f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:31.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:31.141+0000 7f9801e00640 1 -- 172.21.15.16:0/3287831752 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97fc15ee50 con 0x7f97fc070030 2024-03-31T15:09:31.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:31.145+0000 7f9801e00640 1 --2- 172.21.15.16:0/3287831752 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f97fc070030 0x7f97fc162180 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f97f4009d90 tx=0x7f97f401c800 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b4b6e89855e80c24 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:09:31.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:31.145+0000 7f97fbe00640 1 -- 172.21.15.16:0/3287831752 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97f40b2070 con 0x7f97fc070030 2024-03-31T15:09:31.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:31.145+0000 7f97fbe00640 1 -- 172.21.15.16:0/3287831752 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f97f40a5600 con 0x7f97fc070030 2024-03-31T15:09:31.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:31.145+0000 7f97fbe00640 1 -- 172.21.15.16:0/3287831752 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97e8001680 con 0x7f97fc070030 2024-03-31T15:09:31.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:31.145+0000 7f9803e00640 1 -- 172.21.15.16:0/3287831752 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f97fc070030 msgr2=0x7f97fc162180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:31.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:31.145+0000 7f9803e00640 1 --2- 172.21.15.16:0/3287831752 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f97fc070030 0x7f97fc162180 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f97f4009d90 tx=0x7f97f401c800 comp rx=0 tx=0).stop 2024-03-31T15:09:31.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:31.145+0000 7f9803e00640 1 -- 172.21.15.16:0/3287831752 shutdown_connections 2024-03-31T15:09:31.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:31.145+0000 7f9803e00640 1 --2- 172.21.15.16:0/3287831752 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f97fc070030 0x7f97fc162180 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:31.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:31.145+0000 7f9803e00640 1 -- 172.21.15.16:0/3287831752 >> 172.21.15.16:0/3287831752 conn(0x7f97fc067330 msgr2=0x7f97fc067740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:09:31.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:31.145+0000 7f9803e00640 1 -- 172.21.15.16:0/3287831752 shutdown_connections 2024-03-31T15:09:31.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:31.145+0000 7f9803e00640 1 -- 172.21.15.16:0/3287831752 wait complete. 2024-03-31T15:09:31.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:31.149+0000 7f9803e00640 1 Processor -- start 2024-03-31T15:09:31.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:31.149+0000 7f9803e00640 1 -- start start 2024-03-31T15:09:31.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:31.149+0000 7f9803e00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f97fc06dbc0 0x7f97fc169e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:31.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:31.149+0000 7f9803e00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f97fc070030 0x7f97fc0b6480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:31.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:31.149+0000 7f9803e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f97fc15f650 0x7f97fc0b69c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:31.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:31.149+0000 7f9803e00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f97fc06c950 con 0x7f97fc15f650 2024-03-31T15:09:31.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:31.149+0000 7f9803e00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f97fc06dfa0 con 0x7f97fc070030 2024-03-31T15:09:31.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:31.149+0000 7f9803e00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f97fc06cad0 con 0x7f97fc06dbc0 2024-03-31T15:09:31.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:31.149+0000 7f9801e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f97fc15f650 0x7f97fc0b69c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:31.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:31.149+0000 7f9801e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f97fc15f650 0x7f97fc0b69c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60646/0 (socket says 172.21.15.16:60646) 2024-03-31T15:09:31.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:31.149+0000 7f9801e00640 1 -- 172.21.15.16:0/56273982 learned_addr learned my addr 172.21.15.16:0/56273982 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:09:31.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:31.153+0000 7f9800a00640 1 --2- 172.21.15.16:0/56273982 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f97fc070030 0x7f97fc0b6480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:31.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:31.153+0000 7f9801400640 1 --2- 172.21.15.16:0/56273982 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f97fc06dbc0 0x7f97fc169e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:31.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:31.153+0000 7f9801e00640 1 -- 172.21.15.16:0/56273982 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f97fc06dbc0 msgr2=0x7f97fc169e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:31.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:31.153+0000 7f9801e00640 1 --2- 172.21.15.16:0/56273982 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f97fc06dbc0 0x7f97fc169e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:31.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:31.153+0000 7f9801e00640 1 -- 172.21.15.16:0/56273982 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f97fc070030 msgr2=0x7f97fc0b6480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:31.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:31.153+0000 7f9801e00640 1 --2- 172.21.15.16:0/56273982 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f97fc070030 0x7f97fc0b6480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:31.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:31.153+0000 7f9801e00640 1 -- 172.21.15.16:0/56273982 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97fc12bdf0 con 0x7f97fc15f650 2024-03-31T15:09:31.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:31.153+0000 7f9801e00640 1 --2- 172.21.15.16:0/56273982 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f97fc15f650 0x7f97fc0b69c0 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 crypto rx=0x7f97f40b0040 tx=0x7f97f40195a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:09:31.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:31.153+0000 7f97f9600640 1 -- 172.21.15.16:0/56273982 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97f40bf070 con 0x7f97fc15f650 2024-03-31T15:09:31.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:31.153+0000 7f97f9600640 1 -- 172.21.15.16:0/56273982 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f97f40b3040 con 0x7f97fc15f650 2024-03-31T15:09:31.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:31.153+0000 7f97f9600640 1 -- 172.21.15.16:0/56273982 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97f40b2050 con 0x7f97fc15f650 2024-03-31T15:09:31.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:31.153+0000 7f9803e00640 1 -- 172.21.15.16:0/56273982 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f97fc12c0d0 con 0x7f97fc15f650 2024-03-31T15:09:31.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:31.153+0000 7f9803e00640 1 -- 172.21.15.16:0/56273982 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f97fc12c690 con 0x7f97fc15f650 2024-03-31T15:09:31.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:31.153+0000 7f9803e00640 1 -- 172.21.15.16:0/56273982 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f97fc069950 con 0x7f97fc15f650 2024-03-31T15:09:31.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:31.153+0000 7f97f9600640 1 -- 172.21.15.16:0/56273982 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f97f40ad650 con 0x7f97fc15f650 2024-03-31T15:09:31.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:31.157+0000 7f97f9600640 1 --2- 172.21.15.16:0/56273982 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f97dc068380 0x7f97dc06a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:31.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:31.157+0000 7f97f9600640 1 -- 172.21.15.16:0/56273982 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(70..70 src has 1..70) ==== 5882+0+0 (secure 0 0 0) 0x7f97f4120dd0 con 0x7f97fc15f650 2024-03-31T15:09:31.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:31.157+0000 7f9801400640 1 --2- 172.21.15.16:0/56273982 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f97dc068380 0x7f97dc06a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:31.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:31.157+0000 7f9801400640 1 --2- 172.21.15.16:0/56273982 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f97dc068380 0x7f97dc06a840 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f97fc0679a0 tx=0x7f97f0009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:09:31.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:31.161+0000 7f97f9600640 1 -- 172.21.15.16:0/56273982 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f97f4016480 con 0x7f97fc15f650 2024-03-31T15:09:31.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:31.365+0000 7f9803e00640 1 -- 172.21.15.16:0/56273982 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f97fc0b1a90 con 0x7f97fc15f650 2024-03-31T15:09:31.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:31.365+0000 7f97f9600640 1 -- 172.21.15.16:0/56273982 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v52) ==== 53+0+73 (secure 0 0 0) 0x7f97f40f0cb0 con 0x7f97fc15f650 2024-03-31T15:09:31.356 INFO:teuthology.orchestra.run.smithi016.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-31T15:09:31.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:31.365+0000 7f9803e00640 1 -- 172.21.15.16:0/56273982 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f97dc068380 msgr2=0x7f97dc06a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:31.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:31.365+0000 7f9803e00640 1 --2- 172.21.15.16:0/56273982 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f97dc068380 0x7f97dc06a840 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f97fc0679a0 tx=0x7f97f0009040 comp rx=0 tx=0).stop 2024-03-31T15:09:31.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:31.365+0000 7f9803e00640 1 -- 172.21.15.16:0/56273982 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f97fc15f650 msgr2=0x7f97fc0b69c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:31.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:31.365+0000 7f9803e00640 1 --2- 172.21.15.16:0/56273982 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f97fc15f650 0x7f97fc0b69c0 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 crypto rx=0x7f97f40b0040 tx=0x7f97f40195a0 comp rx=0 tx=0).stop 2024-03-31T15:09:31.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:31.365+0000 7f9803e00640 1 -- 172.21.15.16:0/56273982 shutdown_connections 2024-03-31T15:09:31.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:31.365+0000 7f9803e00640 1 --2- 172.21.15.16:0/56273982 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f97dc068380 0x7f97dc06a840 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:31.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:31.365+0000 7f9803e00640 1 --2- 172.21.15.16:0/56273982 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f97fc15f650 0x7f97fc0b69c0 unknown :-1 s=CLOSED pgs=980 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:31.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:31.365+0000 7f9803e00640 1 --2- 172.21.15.16:0/56273982 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f97fc070030 0x7f97fc0b6480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:31.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:31.365+0000 7f9803e00640 1 --2- 172.21.15.16:0/56273982 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f97fc06dbc0 0x7f97fc169e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:31.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:31.365+0000 7f9803e00640 1 -- 172.21.15.16:0/56273982 >> 172.21.15.16:0/56273982 conn(0x7f97fc067330 msgr2=0x7f97fc0ce200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:09:31.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:31.365+0000 7f9803e00640 1 -- 172.21.15.16:0/56273982 shutdown_connections 2024-03-31T15:09:31.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:31.365+0000 7f9803e00640 1 -- 172.21.15.16:0/56273982 wait complete. 2024-03-31T15:09:31.378 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-31T15:09:31.378 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-31T15:09:31.378 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi016.front.sepia.ceph.com 2024-03-31T15:09:31.378 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-31T15:09:31.379 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-31T15:09:31.379 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-31T15:09:31.379 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-31T15:09:31.379 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-31T15:09:31.379 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-31T15:09:31.379 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:09:31.379 DEBUG:teuthology.orchestra.run.smithi016:> ip addr 2024-03-31T15:09:31.389 INFO:teuthology.orchestra.run.smithi016.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-31T15:09:31.389 INFO:teuthology.orchestra.run.smithi016.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-31T15:09:31.389 INFO:teuthology.orchestra.run.smithi016.stdout: inet 127.0.0.1/8 scope host lo 2024-03-31T15:09:31.389 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2024-03-31T15:09:31.389 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 ::1/128 scope host 2024-03-31T15:09:31.389 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2024-03-31T15:09:31.389 INFO:teuthology.orchestra.run.smithi016.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-03-31T15:09:31.389 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:6b:fd:e0 brd ff:ff:ff:ff:ff:ff 2024-03-31T15:09:31.390 INFO:teuthology.orchestra.run.smithi016.stdout: altname enp7s0f0 2024-03-31T15:09:31.390 INFO:teuthology.orchestra.run.smithi016.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-03-31T15:09:31.390 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:6b:fd:e1 brd ff:ff:ff:ff:ff:ff 2024-03-31T15:09:31.390 INFO:teuthology.orchestra.run.smithi016.stdout: altname enp7s0f1 2024-03-31T15:09:31.390 INFO:teuthology.orchestra.run.smithi016.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-31T15:09:31.390 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:bd:15:64 brd ff:ff:ff:ff:ff:ff 2024-03-31T15:09:31.390 INFO:teuthology.orchestra.run.smithi016.stdout: altname enp1s0f0 2024-03-31T15:09:31.390 INFO:teuthology.orchestra.run.smithi016.stdout: inet 172.21.15.16/20 brd 172.21.15.255 scope global dynamic ens1f0 2024-03-31T15:09:31.390 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft 42109sec preferred_lft 42109sec 2024-03-31T15:09:31.390 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 fe80::ec4:7aff:febd:1564/64 scope link 2024-03-31T15:09:31.390 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2024-03-31T15:09:31.391 INFO:teuthology.orchestra.run.smithi016.stdout:5: ens1f1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-03-31T15:09:31.391 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:bd:15:65 brd ff:ff:ff:ff:ff:ff 2024-03-31T15:09:31.391 INFO:teuthology.orchestra.run.smithi016.stdout: altname enp1s0f1 2024-03-31T15:09:31.391 INFO:teuthology.orchestra.run.smithi016.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2024-03-31T15:09:31.391 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 02:42:d4:1f:7d:73 brd ff:ff:ff:ff:ff:ff 2024-03-31T15:09:31.391 INFO:teuthology.orchestra.run.smithi016.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2024-03-31T15:09:31.391 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2024-03-31T15:09:31.391 INFO:teuthology.orchestra.run.smithi016.stdout:7: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-31T15:09:31.391 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 5e:1f:84:6d:79:ae brd ff:ff:ff:ff:ff:ff 2024-03-31T15:09:31.391 INFO:teuthology.orchestra.run.smithi016.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-31T15:09:31.392 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2024-03-31T15:09:31.392 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 fe80::5c1f:84ff:fe6d:79ae/64 scope link 2024-03-31T15:09:31.392 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2024-03-31T15:09:31.392 INFO:teuthology.orchestra.run.smithi016.stdout:8: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-31T15:09:31.392 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0e:5c:f6:1f:82:2f brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-31T15:09:31.392 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 fe80::c5c:f6ff:fe1f:822f/64 scope link 2024-03-31T15:09:31.392 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2024-03-31T15:09:31.392 INFO:teuthology.orchestra.run.smithi016.stdout:9: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-31T15:09:31.393 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether c6:a6:3d:69:d0:db brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-31T15:09:31.393 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 fe80::c4a6:3dff:fe69:d0db/64 scope link 2024-03-31T15:09:31.393 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2024-03-31T15:09:31.393 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:09:31.393 DEBUG:teuthology.orchestra.run.smithi016:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-31T15:09:31.452 INFO:teuthology.orchestra.run.smithi016.stdout:1 2024-03-31T15:09:31.454 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:09:31.455 DEBUG:teuthology.orchestra.run.smithi016:> route 2024-03-31T15:09:31.507 INFO:teuthology.orchestra.run.smithi016.stdout:Kernel IP routing table 2024-03-31T15:09:31.507 INFO:teuthology.orchestra.run.smithi016.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-31T15:09:31.508 INFO:teuthology.orchestra.run.smithi016.stdout:default _gateway 0.0.0.0 UG 0 0 0 ens1f0 2024-03-31T15:09:31.508 INFO:teuthology.orchestra.run.smithi016.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2024-03-31T15:09:31.508 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 0 0 0 ens1f0 2024-03-31T15:09:31.508 INFO:teuthology.orchestra.run.smithi016.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-31T15:09:31.508 DEBUG:teuthology.orchestra.run.smithi016:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-31T15:09:31.509 DEBUG:teuthology.orchestra.run.smithi016:> set -e 2024-03-31T15:09:31.509 DEBUG:teuthology.orchestra.run.smithi016:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-03-31T15:09:31.509 DEBUG:teuthology.orchestra.run.smithi016:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-03-31T15:09:31.509 DEBUG:teuthology.orchestra.run.smithi016:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-03-31T15:09:31.509 DEBUG:teuthology.orchestra.run.smithi016:> ') 2024-03-31T15:09:31.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:31 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-31T15:09:31.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:31 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-31T15:09:31.646 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:09:31.646 DEBUG:teuthology.orchestra.run.smithi016:> ip netns list 2024-03-31T15:09:31.655 INFO:teuthology.orchestra.run.smithi016.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-31T15:09:31.656 INFO:teuthology.orchestra.run.smithi016.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-31T15:09:31.656 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-31T15:09:31.656 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:09:31.657 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip link set brx.0 up 2024-03-31T15:09:31.724 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-03-31T15:09:31.724 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-31T15:09:31.724 DEBUG:teuthology.orchestra.run.smithi016:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-31T15:09:31.775 INFO:teuthology.orchestra.run.smithi016.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-31T15:09:31.776 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-31T15:09:31.776 DEBUG:teuthology.orchestra.run.smithi016:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-31T15:09:31.831 DEBUG:teuthology.orchestra.run.smithi016:> sudo modprobe fuse 2024-03-31T15:09:31.896 DEBUG:teuthology.orchestra.run.smithi016:> cat /proc/self/mounts | awk '{print $2}' 2024-03-31T15:09:31.948 INFO:teuthology.orchestra.run.smithi016.stdout:/sys 2024-03-31T15:09:31.949 INFO:teuthology.orchestra.run.smithi016.stdout:/proc 2024-03-31T15:09:31.949 INFO:teuthology.orchestra.run.smithi016.stdout:/dev 2024-03-31T15:09:31.949 INFO:teuthology.orchestra.run.smithi016.stdout:/dev/pts 2024-03-31T15:09:31.949 INFO:teuthology.orchestra.run.smithi016.stdout:/run 2024-03-31T15:09:31.949 INFO:teuthology.orchestra.run.smithi016.stdout:/ 2024-03-31T15:09:31.949 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/kernel/security 2024-03-31T15:09:31.949 INFO:teuthology.orchestra.run.smithi016.stdout:/dev/shm 2024-03-31T15:09:31.949 INFO:teuthology.orchestra.run.smithi016.stdout:/run/lock 2024-03-31T15:09:31.949 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/cgroup 2024-03-31T15:09:31.949 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/pstore 2024-03-31T15:09:31.949 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/bpf 2024-03-31T15:09:31.950 INFO:teuthology.orchestra.run.smithi016.stdout:/proc/sys/fs/binfmt_misc 2024-03-31T15:09:31.950 INFO:teuthology.orchestra.run.smithi016.stdout:/dev/hugepages 2024-03-31T15:09:31.950 INFO:teuthology.orchestra.run.smithi016.stdout:/dev/mqueue 2024-03-31T15:09:31.950 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/kernel/debug 2024-03-31T15:09:31.950 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/kernel/tracing 2024-03-31T15:09:31.950 INFO:teuthology.orchestra.run.smithi016.stdout:/proc/fs/nfsd 2024-03-31T15:09:31.950 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/fuse/connections 2024-03-31T15:09:31.950 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/kernel/config 2024-03-31T15:09:31.950 INFO:teuthology.orchestra.run.smithi016.stdout:/run/credentials/systemd-sysusers.service 2024-03-31T15:09:31.950 INFO:teuthology.orchestra.run.smithi016.stdout:/run/rpc_pipefs 2024-03-31T15:09:31.950 INFO:teuthology.orchestra.run.smithi016.stdout:/run/user/1000 2024-03-31T15:09:31.950 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/ceph 2024-03-31T15:09:31.950 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/ceph/osd/ceph-0 2024-03-31T15:09:31.950 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/ceph/osd/ceph-1 2024-03-31T15:09:31.951 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/ceph/osd/ceph-2 2024-03-31T15:09:31.951 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/ceph/osd/ceph-3 2024-03-31T15:09:31.951 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns 2024-03-31T15:09:31.951 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-31T15:09:31.951 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-31T15:09:31.951 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-31T15:09:31.951 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-31T15:09:31.952 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:09:31.952 DEBUG:teuthology.orchestra.run.smithi016:> ls /sys/fs/fuse/connections 2024-03-31T15:09:32.004 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-31T15:09:32.004 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-03-31T15:09:32.050 DEBUG:teuthology.orchestra.run.smithi016:> sudo modprobe fuse 2024-03-31T15:09:32.072 DEBUG:teuthology.orchestra.run.smithi016:> cat /proc/self/mounts | awk '{print $2}' 2024-03-31T15:09:32.125 INFO:teuthology.orchestra.run.smithi016.stdout:/sys 2024-03-31T15:09:32.125 INFO:teuthology.orchestra.run.smithi016.stdout:/proc 2024-03-31T15:09:32.125 INFO:teuthology.orchestra.run.smithi016.stdout:/dev 2024-03-31T15:09:32.125 INFO:teuthology.orchestra.run.smithi016.stdout:/dev/pts 2024-03-31T15:09:32.125 INFO:teuthology.orchestra.run.smithi016.stdout:/run 2024-03-31T15:09:32.125 INFO:teuthology.orchestra.run.smithi016.stdout:/ 2024-03-31T15:09:32.125 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/kernel/security 2024-03-31T15:09:32.125 INFO:teuthology.orchestra.run.smithi016.stdout:/dev/shm 2024-03-31T15:09:32.126 INFO:teuthology.orchestra.run.smithi016.stdout:/run/lock 2024-03-31T15:09:32.126 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/cgroup 2024-03-31T15:09:32.126 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/pstore 2024-03-31T15:09:32.126 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/bpf 2024-03-31T15:09:32.126 INFO:teuthology.orchestra.run.smithi016.stdout:/proc/sys/fs/binfmt_misc 2024-03-31T15:09:32.126 INFO:teuthology.orchestra.run.smithi016.stdout:/dev/hugepages 2024-03-31T15:09:32.126 INFO:teuthology.orchestra.run.smithi016.stdout:/dev/mqueue 2024-03-31T15:09:32.126 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/kernel/debug 2024-03-31T15:09:32.126 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/kernel/tracing 2024-03-31T15:09:32.126 INFO:teuthology.orchestra.run.smithi016.stdout:/proc/fs/nfsd 2024-03-31T15:09:32.126 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/fuse/connections 2024-03-31T15:09:32.126 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/kernel/config 2024-03-31T15:09:32.126 INFO:teuthology.orchestra.run.smithi016.stdout:/run/credentials/systemd-sysusers.service 2024-03-31T15:09:32.126 INFO:teuthology.orchestra.run.smithi016.stdout:/run/rpc_pipefs 2024-03-31T15:09:32.127 INFO:teuthology.orchestra.run.smithi016.stdout:/run/user/1000 2024-03-31T15:09:32.127 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/ceph 2024-03-31T15:09:32.127 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/ceph/osd/ceph-0 2024-03-31T15:09:32.127 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/ceph/osd/ceph-1 2024-03-31T15:09:32.127 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/ceph/osd/ceph-2 2024-03-31T15:09:32.127 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/ceph/osd/ceph-3 2024-03-31T15:09:32.127 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns 2024-03-31T15:09:32.127 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-31T15:09:32.127 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-31T15:09:32.127 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-31T15:09:32.127 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-31T15:09:32.128 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:09:32.128 DEBUG:teuthology.orchestra.run.smithi016:> ls /sys/fs/fuse/connections 2024-03-31T15:09:32.144 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi016.stderr:2024-03-31T15:09:32.153+0000 7f38ce031580 -1 init, newargv = 0x55a5dae6c190 newargc=15 2024-03-31T15:09:32.145 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi016.stderr:ceph-fuse[43322]: starting ceph client 2024-03-31T15:09:32.151 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi016.stderr:ceph-fuse[43322]: starting fuse 2024-03-31T15:09:32.155 INFO:teuthology.orchestra.run.smithi016.stdout:43 2024-03-31T15:09:32.156 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [43] 2024-03-31T15:09:32.156 DEBUG:teuthology.orchestra.run.smithi016:> sudo stdin-killer -- python3 -c ' 2024-03-31T15:09:32.156 DEBUG:teuthology.orchestra.run.smithi016:> import glob 2024-03-31T15:09:32.156 DEBUG:teuthology.orchestra.run.smithi016:> import re 2024-03-31T15:09:32.156 DEBUG:teuthology.orchestra.run.smithi016:> import os 2024-03-31T15:09:32.156 DEBUG:teuthology.orchestra.run.smithi016:> import subprocess 2024-03-31T15:09:32.156 DEBUG:teuthology.orchestra.run.smithi016:> 2024-03-31T15:09:32.156 DEBUG:teuthology.orchestra.run.smithi016:> def _find_admin_socket(client_name): 2024-03-31T15:09:32.156 DEBUG:teuthology.orchestra.run.smithi016:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-03-31T15:09:32.156 DEBUG:teuthology.orchestra.run.smithi016:> files = glob.glob(asok_path) 2024-03-31T15:09:32.156 DEBUG:teuthology.orchestra.run.smithi016:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-03-31T15:09:32.157 DEBUG:teuthology.orchestra.run.smithi016:> 2024-03-31T15:09:32.157 DEBUG:teuthology.orchestra.run.smithi016:> # Given a non-glob path, it better be there 2024-03-31T15:09:32.157 DEBUG:teuthology.orchestra.run.smithi016:> if "*" not in asok_path: 2024-03-31T15:09:32.157 DEBUG:teuthology.orchestra.run.smithi016:> assert(len(files) == 1) 2024-03-31T15:09:32.157 DEBUG:teuthology.orchestra.run.smithi016:> return files[0] 2024-03-31T15:09:32.157 DEBUG:teuthology.orchestra.run.smithi016:> 2024-03-31T15:09:32.157 DEBUG:teuthology.orchestra.run.smithi016:> for f in files: 2024-03-31T15:09:32.157 DEBUG:teuthology.orchestra.run.smithi016:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-31T15:09:32.157 DEBUG:teuthology.orchestra.run.smithi016:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-31T15:09:32.157 DEBUG:teuthology.orchestra.run.smithi016:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-31T15:09:32.157 DEBUG:teuthology.orchestra.run.smithi016:> contents = proc_f.read() 2024-03-31T15:09:32.157 DEBUG:teuthology.orchestra.run.smithi016:> if mountpoint in contents: 2024-03-31T15:09:32.157 DEBUG:teuthology.orchestra.run.smithi016:> return f 2024-03-31T15:09:32.157 DEBUG:teuthology.orchestra.run.smithi016:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-31T15:09:32.157 DEBUG:teuthology.orchestra.run.smithi016:> 2024-03-31T15:09:32.157 DEBUG:teuthology.orchestra.run.smithi016:> print(_find_admin_socket("client.0")) 2024-03-31T15:09:32.157 DEBUG:teuthology.orchestra.run.smithi016:> ' 2024-03-31T15:09:32.280 INFO:teuthology.orchestra.run.smithi016.stdout:/var/run/ceph/ceph-client.0.43322.asok 2024-03-31T15:09:32.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:32 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-31T15:09:32.292 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.43322.asok 2024-03-31T15:09:32.292 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:09:32.293 DEBUG:teuthology.orchestra.run.smithi016:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.43322.asok status 2024-03-31T15:09:32.465 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-03-31T15:09:32.466 INFO:teuthology.orchestra.run.smithi016.stdout: "metadata": { 2024-03-31T15:09:32.466 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph_sha1": "d2271579d5e735cfd6d436b8349c004f06527cb0", 2024-03-31T15:09:32.466 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph_version": "ceph version 19.0.0-2574-gd2271579 (d2271579d5e735cfd6d436b8349c004f06527cb0) squid (dev)", 2024-03-31T15:09:32.466 INFO:teuthology.orchestra.run.smithi016.stdout: "entity_id": "0", 2024-03-31T15:09:32.466 INFO:teuthology.orchestra.run.smithi016.stdout: "hostname": "smithi016", 2024-03-31T15:09:32.466 INFO:teuthology.orchestra.run.smithi016.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-03-31T15:09:32.466 INFO:teuthology.orchestra.run.smithi016.stdout: "pid": "43322", 2024-03-31T15:09:32.466 INFO:teuthology.orchestra.run.smithi016.stdout: "root": "/" 2024-03-31T15:09:32.466 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-31T15:09:32.466 INFO:teuthology.orchestra.run.smithi016.stdout: "dentry_count": 0, 2024-03-31T15:09:32.466 INFO:teuthology.orchestra.run.smithi016.stdout: "dentry_pinned_count": 0, 2024-03-31T15:09:32.466 INFO:teuthology.orchestra.run.smithi016.stdout: "id": 7275, 2024-03-31T15:09:32.467 INFO:teuthology.orchestra.run.smithi016.stdout: "inst": { 2024-03-31T15:09:32.467 INFO:teuthology.orchestra.run.smithi016.stdout: "name": { 2024-03-31T15:09:32.467 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "client", 2024-03-31T15:09:32.467 INFO:teuthology.orchestra.run.smithi016.stdout: "num": 7275 2024-03-31T15:09:32.467 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-31T15:09:32.467 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": { 2024-03-31T15:09:32.467 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-03-31T15:09:32.467 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "192.168.0.1:0", 2024-03-31T15:09:32.467 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 104048160 2024-03-31T15:09:32.467 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-31T15:09:32.467 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-31T15:09:32.467 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": { 2024-03-31T15:09:32.467 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-03-31T15:09:32.467 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "192.168.0.1:0", 2024-03-31T15:09:32.468 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 104048160 2024-03-31T15:09:32.468 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-31T15:09:32.468 INFO:teuthology.orchestra.run.smithi016.stdout: "inst_str": "client.7275 192.168.0.1:0/104048160", 2024-03-31T15:09:32.468 INFO:teuthology.orchestra.run.smithi016.stdout: "addr_str": "192.168.0.1:0/104048160", 2024-03-31T15:09:32.468 INFO:teuthology.orchestra.run.smithi016.stdout: "inode_count": 1, 2024-03-31T15:09:32.468 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_epoch": 52, 2024-03-31T15:09:32.468 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_epoch": 70, 2024-03-31T15:09:32.468 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_epoch_barrier": 0, 2024-03-31T15:09:32.468 INFO:teuthology.orchestra.run.smithi016.stdout: "blocklisted": false, 2024-03-31T15:09:32.468 INFO:teuthology.orchestra.run.smithi016.stdout: "fs_name": "cephfs" 2024-03-31T15:09:32.468 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-03-31T15:09:32.480 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:09:32.480 DEBUG:teuthology.orchestra.run.smithi016:> stat --file-system '--printf=%T 2024-03-31T15:09:32.481 DEBUG:teuthology.orchestra.run.smithi016:> ' -- /home/ubuntu/cephtest/mnt.0 2024-03-31T15:09:32.489 INFO:teuthology.orchestra.run.smithi016.stdout:fuseblk 2024-03-31T15:09:32.490 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-03-31T15:09:32.490 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:09:32.490 DEBUG:teuthology.orchestra.run.smithi016:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-31T15:09:32.551 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:09:32.551 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr --only-values -n ceph.dir.files ./subdir) 2024-03-31T15:09:32.605 INFO:teuthology.orchestra.run.smithi016.stdout:1 2024-03-31T15:09:32.606 DEBUG:teuthology.orchestra.run.smithi016:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata setomapval 10000000000.00000000 file_to_be_damaged_head deadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef 2024-03-31T15:09:32.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:32.705+0000 7f4219060080 1 Processor -- start 2024-03-31T15:09:32.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:32.705+0000 7f4219060080 1 -- start start 2024-03-31T15:09:32.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:32.705+0000 7f4219060080 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f41fc002300 0x5591066e9ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:32.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:32.705+0000 7f4219060080 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5591066d3890 0x5591066e43a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:32.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:32.705+0000 7f4219060080 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5591066e48e0 0x5591067754a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:32.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:32.705+0000 7f4219060080 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5591066464b0 con 0x7f41fc002300 2024-03-31T15:09:32.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:32.705+0000 7f4219060080 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55910664a500 con 0x5591066d3890 2024-03-31T15:09:32.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:32.705+0000 7f4219060080 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x5591067780c0 con 0x5591066e48e0 2024-03-31T15:09:32.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:32.705+0000 7f4216200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f41fc002300 0x5591066e9ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:32.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:32.705+0000 7f4216200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f41fc002300 0x5591066e9ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60664/0 (socket says 172.21.15.16:60664) 2024-03-31T15:09:32.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:32.705+0000 7f4216200640 1 -- 172.21.15.16:0/1521045670 learned_addr learned my addr 172.21.15.16:0/1521045670 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:09:32.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:32.705+0000 7f4215800640 1 --2- 172.21.15.16:0/1521045670 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5591066d3890 0x5591066e43a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:32.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:32.705+0000 7f4216c00640 1 --2- 172.21.15.16:0/1521045670 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5591066e48e0 0x5591067754a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:32.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:32.705+0000 7f4216200640 1 -- 172.21.15.16:0/1521045670 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5591066e48e0 msgr2=0x5591067754a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:09:32.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:32.705+0000 7f4216200640 1 --2- 172.21.15.16:0/1521045670 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5591066e48e0 0x5591067754a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:32.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:32.705+0000 7f4216200640 1 -- 172.21.15.16:0/1521045670 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5591066d3890 msgr2=0x5591066e43a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:09:32.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:32.705+0000 7f4216200640 1 --2- 172.21.15.16:0/1521045670 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5591066d3890 0x5591066e43a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:32.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:32.705+0000 7f4216200640 1 -- 172.21.15.16:0/1521045670 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5591066a0890 con 0x7f41fc002300 2024-03-31T15:09:32.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:32.705+0000 7f4216200640 1 --2- 172.21.15.16:0/1521045670 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f41fc002300 0x5591066e9ee0 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7f420801c000 tx=0x5591066ed860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6261de2a84fba4f2 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:09:32.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:32.705+0000 7f420fe00640 1 -- 172.21.15.16:0/1521045670 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42080a4700 con 0x7f41fc002300 2024-03-31T15:09:32.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:32.709+0000 7f420fe00640 1 -- 172.21.15.16:0/1521045670 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f42080bb070 con 0x7f41fc002300 2024-03-31T15:09:32.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:32.709+0000 7f420fe00640 1 -- 172.21.15.16:0/1521045670 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42080a4ac0 con 0x7f41fc002300 2024-03-31T15:09:32.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:32.709+0000 7f4219060080 1 -- 172.21.15.16:0/1521045670 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f41fc002300 msgr2=0x5591066e9ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:32.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:32.709+0000 7f4219060080 1 --2- 172.21.15.16:0/1521045670 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f41fc002300 0x5591066e9ee0 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7f420801c000 tx=0x5591066ed860 comp rx=0 tx=0).stop 2024-03-31T15:09:32.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:32.709+0000 7f4219060080 1 -- 172.21.15.16:0/1521045670 shutdown_connections 2024-03-31T15:09:32.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:32.709+0000 7f4219060080 1 --2- 172.21.15.16:0/1521045670 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5591066e48e0 0x5591067754a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:32.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:32.709+0000 7f4219060080 1 --2- 172.21.15.16:0/1521045670 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5591066d3890 0x5591066e43a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:32.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:32.709+0000 7f4219060080 1 --2- 172.21.15.16:0/1521045670 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f41fc002300 0x5591066e9ee0 unknown :-1 s=CLOSED pgs=984 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:32.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:32.709+0000 7f4219060080 1 -- 172.21.15.16:0/1521045670 >> 172.21.15.16:0/1521045670 conn(0x5591066ce420 msgr2=0x559106776900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:09:32.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:32.709+0000 7f4219060080 1 -- 172.21.15.16:0/1521045670 shutdown_connections 2024-03-31T15:09:32.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:32.709+0000 7f4219060080 1 -- 172.21.15.16:0/1521045670 wait complete. 2024-03-31T15:09:32.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:32.709+0000 7f4219060080 1 Processor -- start 2024-03-31T15:09:32.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:32.709+0000 7f4219060080 1 -- start start 2024-03-31T15:09:32.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:32.709+0000 7f4219060080 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5591066d3890 0x55910675f910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:32.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:32.709+0000 7f4219060080 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f41fc002300 0x559106764ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:32.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:32.709+0000 7f4219060080 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559106770b80 0x559106765010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:32.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:32.709+0000 7f4219060080 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5591066a0b30 con 0x7f41fc002300 2024-03-31T15:09:32.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:32.709+0000 7f4219060080 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x5591066a0dd0 con 0x559106770b80 2024-03-31T15:09:32.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:32.709+0000 7f4219060080 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x5591066a1070 con 0x5591066d3890 2024-03-31T15:09:32.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:32.709+0000 7f4215800640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f41fc002300 0x559106764ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:32.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:32.713+0000 7f4215800640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f41fc002300 0x559106764ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60668/0 (socket says 172.21.15.16:60668) 2024-03-31T15:09:32.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:32.713+0000 7f4215800640 1 -- 172.21.15.16:0/1112627436 learned_addr learned my addr 172.21.15.16:0/1112627436 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:09:32.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:32.713+0000 7f4216200640 1 --2- 172.21.15.16:0/1112627436 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5591066d3890 0x55910675f910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:32.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:32.713+0000 7f4216c00640 1 --2- 172.21.15.16:0/1112627436 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559106770b80 0x559106765010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:32.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:32.713+0000 7f4215800640 1 -- 172.21.15.16:0/1112627436 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5591066d3890 msgr2=0x55910675f910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:32.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:32.713+0000 7f4215800640 1 --2- 172.21.15.16:0/1112627436 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5591066d3890 0x55910675f910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:32.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:32.713+0000 7f4215800640 1 -- 172.21.15.16:0/1112627436 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559106770b80 msgr2=0x559106765010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:32.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:32.713+0000 7f4215800640 1 --2- 172.21.15.16:0/1112627436 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559106770b80 0x559106765010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:32.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:32.713+0000 7f4215800640 1 -- 172.21.15.16:0/1112627436 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559106760290 con 0x7f41fc002300 2024-03-31T15:09:32.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:32.713+0000 7f4215800640 1 --2- 172.21.15.16:0/1112627436 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f41fc002300 0x559106764ad0 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7f4204008770 tx=0x7f4204008c30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:09:32.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:32.713+0000 7f420ea00640 1 -- 172.21.15.16:0/1112627436 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4204013c90 con 0x7f41fc002300 2024-03-31T15:09:32.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:32.713+0000 7f420ea00640 1 -- 172.21.15.16:0/1112627436 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4204013e30 con 0x7f41fc002300 2024-03-31T15:09:32.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:32.713+0000 7f420ea00640 1 -- 172.21.15.16:0/1112627436 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4204016620 con 0x7f41fc002300 2024-03-31T15:09:32.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:32.713+0000 7f4219060080 1 -- 172.21.15.16:0/1112627436 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x559106760530 con 0x7f41fc002300 2024-03-31T15:09:32.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:32.713+0000 7f4219060080 1 -- 172.21.15.16:0/1112627436 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559106760a70 con 0x7f41fc002300 2024-03-31T15:09:32.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:32.713+0000 7f420ea00640 1 -- 172.21.15.16:0/1112627436 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f42040167c0 con 0x7f41fc002300 2024-03-31T15:09:32.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:32.713+0000 7f420ea00640 1 --2- 172.21.15.16:0/1112627436 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f41f8066d90 0x7f41f8069250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:32.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:32.713+0000 7f420ea00640 1 -- 172.21.15.16:0/1112627436 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(70..70 src has 1..70) ==== 5882+0+0 (secure 0 0 0) 0x7f4204086630 con 0x7f41fc002300 2024-03-31T15:09:32.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:32.717+0000 7f4216200640 1 --2- 172.21.15.16:0/1112627436 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f41f8066d90 0x7f41f8069250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:32.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:32.717+0000 7f4219060080 1 --2- 172.21.15.16:0/1112627436 >> [v2:172.21.15.89:6800/1856844856,v1:172.21.15.89:6801/1856844856] conn(0x559106810650 0x559106810a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:32.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:32.717+0000 7f4219060080 1 -- 172.21.15.16:0/1112627436 --> [v2:172.21.15.89:6800/1856844856,v1:172.21.15.89:6801/1856844856] -- osd_op(unknown.0.0:1 10.10 10:0cf6ad0f:::10000000000.00000000:head [omap-set-vals in=115b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e70) -- 0x55910675fe50 con 0x559106810650 2024-03-31T15:09:32.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:32.717+0000 7f4216c00640 1 --2- 172.21.15.16:0/1112627436 >> [v2:172.21.15.89:6800/1856844856,v1:172.21.15.89:6801/1856844856] conn(0x559106810650 0x559106810a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:32.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:32.717+0000 7f4216200640 1 --2- 172.21.15.16:0/1112627436 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f41f8066d90 0x7f41f8069250 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f42080ba000 tx=0x7f42080c4040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:09:32.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:32.717+0000 7f4216c00640 1 --2- 172.21.15.16:0/1112627436 >> [v2:172.21.15.89:6800/1856844856,v1:172.21.15.89:6801/1856844856] conn(0x559106810650 0x559106810a30 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:09:32.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:32.717+0000 7f4216c00640 1 -- 172.21.15.16:0/1112627436 <== osd.4 v2:172.21.15.89:6800/1856844856 1 ==== osd_op_reply(1 10000000000.00000000 [omap-set-vals] v70'12 uv12 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f4210015420 con 0x559106810650 2024-03-31T15:09:32.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:32.717+0000 7f4219060080 1 -- 172.21.15.16:0/1112627436 >> [v2:172.21.15.89:6800/1856844856,v1:172.21.15.89:6801/1856844856] conn(0x559106810650 msgr2=0x559106810a30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:32.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:32.717+0000 7f4219060080 1 --2- 172.21.15.16:0/1112627436 >> [v2:172.21.15.89:6800/1856844856,v1:172.21.15.89:6801/1856844856] conn(0x559106810650 0x559106810a30 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:32.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:32.717+0000 7f4219060080 1 -- 172.21.15.16:0/1112627436 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f41f8066d90 msgr2=0x7f41f8069250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:32.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:32.717+0000 7f4219060080 1 --2- 172.21.15.16:0/1112627436 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f41f8066d90 0x7f41f8069250 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f42080ba000 tx=0x7f42080c4040 comp rx=0 tx=0).stop 2024-03-31T15:09:32.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:32.717+0000 7f4219060080 1 -- 172.21.15.16:0/1112627436 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f41fc002300 msgr2=0x559106764ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:32.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:32.717+0000 7f4219060080 1 --2- 172.21.15.16:0/1112627436 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f41fc002300 0x559106764ad0 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7f4204008770 tx=0x7f4204008c30 comp rx=0 tx=0).stop 2024-03-31T15:09:32.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:32.717+0000 7f4216c00640 1 -- 172.21.15.16:0/1112627436 reap_dead start 2024-03-31T15:09:32.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:32.717+0000 7f4219060080 1 -- 172.21.15.16:0/1112627436 shutdown_connections 2024-03-31T15:09:32.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:32.717+0000 7f4219060080 1 -- 172.21.15.16:0/1112627436 >> 172.21.15.16:0/1112627436 conn(0x5591066ce420 msgr2=0x5591067768c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:09:32.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:32.717+0000 7f4219060080 1 -- 172.21.15.16:0/1112627436 shutdown_connections 2024-03-31T15:09:32.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:32.717+0000 7f4219060080 1 -- 172.21.15.16:0/1112627436 wait complete. 2024-03-31T15:09:32.717 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-31T15:09:32.717 DEBUG:teuthology.orchestra.run.smithi016:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-31T15:09:32.805 DEBUG:teuthology.orchestra.run.smithi089:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-31T15:09:32.876 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-31T15:09:32.876 INFO:tasks.cephfs.mount:Terminating background process 2024-03-31T15:09:32.876 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-03-31T15:09:32.876 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-31T15:09:32.885 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-31T15:09:33.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:33.073+0000 7f29dba00640 1 Processor -- start 2024-03-31T15:09:33.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:33.073+0000 7f29dba00640 1 -- start start 2024-03-31T15:09:33.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:33.073+0000 7f29dba00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f29d406f920 0x7f29d406fd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:33.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:33.073+0000 7f29dba00640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f29d40d0920 con 0x7f29d4145e40 2024-03-31T15:09:33.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:33.073+0000 7f29dba00640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f29d40d0c00 con 0x7f29d40d2010 2024-03-31T15:09:33.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:33.073+0000 7f29dba00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f29d40702c0 con 0x7f29d406f920 2024-03-31T15:09:33.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:33.073+0000 7f29d9000640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f29d4145e40 0x7f29d40cc740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:46696/0 (socket says 172.21.15.16:46696) 2024-03-31T15:09:33.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:33.073+0000 7f29d9000640 1 -- 172.21.15.16:0/3975288451 learned_addr learned my addr 172.21.15.16:0/3975288451 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:09:33.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:33.073+0000 7f29d3e00640 1 --2- 172.21.15.16:0/3975288451 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f29d406f920 0x7f29d406fd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:33.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:33.077+0000 7f29d3400640 1 -- 172.21.15.16:0/3975288451 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4208379314 0 0) 0x7f29d40d0920 con 0x7f29d4145e40 2024-03-31T15:09:33.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:33.077+0000 7f29d3400640 1 -- 172.21.15.16:0/3975288451 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f29bc003530 con 0x7f29d4145e40 2024-03-31T15:09:33.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:33.077+0000 7f29d3400640 1 -- 172.21.15.16:0/3975288451 <== mon.1 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1091414985 0 0) 0x7f29d40d0c00 con 0x7f29d40d2010 2024-03-31T15:09:33.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:33.077+0000 7f29d3400640 1 -- 172.21.15.16:0/3975288451 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f29d40d0920 con 0x7f29d40d2010 2024-03-31T15:09:33.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:33.077+0000 7f29d3400640 1 -- 172.21.15.16:0/3975288451 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2516652335 0 0) 0x7f29bc003530 con 0x7f29d4145e40 2024-03-31T15:09:33.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:33.077+0000 7f29d3400640 1 -- 172.21.15.16:0/3975288451 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f29d406f920 msgr2=0x7f29d406fd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:09:33.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:33.077+0000 7f29d3400640 1 --2- 172.21.15.16:0/3975288451 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f29d406f920 0x7f29d406fd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:33.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:33.077+0000 7f29d3400640 1 -- 172.21.15.16:0/3975288451 >> v1:172.21.15.89:6789/0 conn(0x7f29d40d2010 legacy=0x7f29d40d24a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:33.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:33.077+0000 7f29d3400640 1 -- 172.21.15.16:0/3975288451 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29d40d1810 con 0x7f29d4145e40 2024-03-31T15:09:33.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:33.077+0000 7f29d3400640 1 -- 172.21.15.16:0/3975288451 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 214807623 0 0) 0x7f29c8001600 con 0x7f29d4145e40 2024-03-31T15:09:33.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:33.077+0000 7f29d3400640 1 -- 172.21.15.16:0/3975288451 >> v1:172.21.15.16:6789/0 conn(0x7f29d4145e40 legacy=0x7f29d40cc740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:33.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:33.077+0000 7f29d3400640 1 --2- 172.21.15.16:0/3975288451 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f29bc003b80 0x7f29bc003f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:33.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:33.077+0000 7f29d3400640 1 --2- 172.21.15.16:0/3975288451 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f29d406f920 0x7f29bc004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:33.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:33.077+0000 7f29d3400640 1 --2- 172.21.15.16:0/3975288451 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f29bc004d50 0x7f29bc0051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:33.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:33.077+0000 7f29d3400640 1 -- 172.21.15.16:0/3975288451 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f29bc00ba80 con 0x7f29bc003b80 2024-03-31T15:09:33.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:33.077+0000 7f29d3400640 1 -- 172.21.15.16:0/3975288451 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f29bc00bc00 con 0x7f29bc004d50 2024-03-31T15:09:33.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:33.077+0000 7f29d3400640 1 -- 172.21.15.16:0/3975288451 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f29bc00bdb0 con 0x7f29d406f920 2024-03-31T15:09:33.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:33.077+0000 7f29d9000640 1 --2- 172.21.15.16:0/3975288451 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f29bc003b80 0x7f29bc003f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:33.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:33.077+0000 7f29d9a00640 1 --2- 172.21.15.16:0/3975288451 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f29bc004d50 0x7f29bc0051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:33.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:33.077+0000 7f29d3e00640 1 --2- 172.21.15.16:0/3975288451 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f29d406f920 0x7f29bc004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:33.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:33.077+0000 7f29d9000640 1 -- 172.21.15.16:0/3975288451 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f29d406f920 msgr2=0x7f29bc004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:09:33.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:33.077+0000 7f29d9000640 1 --2- 172.21.15.16:0/3975288451 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f29d406f920 0x7f29bc004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:33.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:33.077+0000 7f29d9000640 1 -- 172.21.15.16:0/3975288451 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f29bc004d50 msgr2=0x7f29bc0051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:09:33.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:33.077+0000 7f29d9000640 1 --2- 172.21.15.16:0/3975288451 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f29bc004d50 0x7f29bc0051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:33.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:33.077+0000 7f29d9000640 1 -- 172.21.15.16:0/3975288451 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29bc003530 con 0x7f29bc003b80 2024-03-31T15:09:33.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:33.081+0000 7f29d9000640 1 --2- 172.21.15.16:0/3975288451 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f29bc003b80 0x7f29bc003f80 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7f29c800c110 tx=0x7f29c801cba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bfcfb10a39a45e7a server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:09:33.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:33.081+0000 7f29d3400640 1 -- 172.21.15.16:0/3975288451 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29c8007c30 con 0x7f29bc003b80 2024-03-31T15:09:33.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:33.081+0000 7f29d3400640 1 -- 172.21.15.16:0/3975288451 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f29c80a9020 con 0x7f29bc003b80 2024-03-31T15:09:33.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:33.081+0000 7f29d3400640 1 -- 172.21.15.16:0/3975288451 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29c8005b00 con 0x7f29bc003b80 2024-03-31T15:09:33.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:33.081+0000 7f29dba00640 1 -- 172.21.15.16:0/3975288451 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f29bc003b80 msgr2=0x7f29bc003f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:33.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:33.081+0000 7f29dba00640 1 --2- 172.21.15.16:0/3975288451 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f29bc003b80 0x7f29bc003f80 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7f29c800c110 tx=0x7f29c801cba0 comp rx=0 tx=0).stop 2024-03-31T15:09:33.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:33.081+0000 7f29d9a00640 1 -- 172.21.15.16:0/3975288451 reap_dead start 2024-03-31T15:09:33.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:33.081+0000 7f29dba00640 1 -- 172.21.15.16:0/3975288451 shutdown_connections 2024-03-31T15:09:33.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:33.081+0000 7f29dba00640 1 -- 172.21.15.16:0/3975288451 >> 172.21.15.16:0/3975288451 conn(0x7f29d4067330 msgr2=0x7f29d4067740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:09:33.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:33.081+0000 7f29dba00640 1 -- 172.21.15.16:0/3975288451 shutdown_connections 2024-03-31T15:09:33.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:33.081+0000 7f29dba00640 1 -- 172.21.15.16:0/3975288451 wait complete. 2024-03-31T15:09:33.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:33.081+0000 7f29dba00640 1 Processor -- start 2024-03-31T15:09:33.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:33.081+0000 7f29dba00640 1 -- start start 2024-03-31T15:09:33.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:33.081+0000 7f29dba00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f29bc003b80 0x7f29d4165b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:33.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:33.081+0000 7f29dba00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f29d406d7b0 0x7f29d4144b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:33.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:33.081+0000 7f29dba00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f29d40cc290 0x7f29d40cc710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:33.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:33.081+0000 7f29dba00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f29d4070110 con 0x7f29bc003b80 2024-03-31T15:09:33.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:33.081+0000 7f29dba00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f29d40d06d0 con 0x7f29d406d7b0 2024-03-31T15:09:33.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:33.081+0000 7f29dba00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f29d40d0ee0 con 0x7f29d40cc290 2024-03-31T15:09:33.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:33.085+0000 7f29d9000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f29bc003b80 0x7f29d4165b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:33.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:33.085+0000 7f29d9000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f29bc003b80 0x7f29d4165b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60696/0 (socket says 172.21.15.16:60696) 2024-03-31T15:09:33.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:33.085+0000 7f29d9000640 1 -- 172.21.15.16:0/1067364033 learned_addr learned my addr 172.21.15.16:0/1067364033 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:09:33.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:33.085+0000 7f29d9a00640 1 --2- 172.21.15.16:0/1067364033 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f29d40cc290 0x7f29d40cc710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:33.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:33.085+0000 7f29d3e00640 1 --2- 172.21.15.16:0/1067364033 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f29d406d7b0 0x7f29d4144b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:33.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:33.085+0000 7f29d9000640 1 -- 172.21.15.16:0/1067364033 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f29d40cc290 msgr2=0x7f29d40cc710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:33.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:33.085+0000 7f29d9000640 1 --2- 172.21.15.16:0/1067364033 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f29d40cc290 0x7f29d40cc710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:33.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:33.085+0000 7f29d9000640 1 -- 172.21.15.16:0/1067364033 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f29d406d7b0 msgr2=0x7f29d4144b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:33.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:33.085+0000 7f29d9000640 1 --2- 172.21.15.16:0/1067364033 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f29d406d7b0 0x7f29d4144b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:33.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:33.085+0000 7f29d9000640 1 -- 172.21.15.16:0/1067364033 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29d41660c0 con 0x7f29bc003b80 2024-03-31T15:09:33.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:33.085+0000 7f29d9a00640 1 --2- 172.21.15.16:0/1067364033 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f29d40cc290 0x7f29d40cc710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:09:33.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:33.085+0000 7f29d9000640 1 --2- 172.21.15.16:0/1067364033 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f29bc003b80 0x7f29d4165b80 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7f29c80ab040 tx=0x7f29bc005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:09:33.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:33.085+0000 7f29d0c00640 1 -- 172.21.15.16:0/1067364033 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29c8004ab0 con 0x7f29bc003b80 2024-03-31T15:09:33.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:33.085+0000 7f29d0c00640 1 -- 172.21.15.16:0/1067364033 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f29c8004c50 con 0x7f29bc003b80 2024-03-31T15:09:33.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:33.085+0000 7f29d0c00640 1 -- 172.21.15.16:0/1067364033 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29c80070d0 con 0x7f29bc003b80 2024-03-31T15:09:33.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:33.085+0000 7f29dba00640 1 -- 172.21.15.16:0/1067364033 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f29d4166370 con 0x7f29bc003b80 2024-03-31T15:09:33.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:33.085+0000 7f29dba00640 1 -- 172.21.15.16:0/1067364033 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f29d4166930 con 0x7f29bc003b80 2024-03-31T15:09:33.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:33.085+0000 7f29dba00640 1 -- 172.21.15.16:0/1067364033 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f29d4069950 con 0x7f29bc003b80 2024-03-31T15:09:33.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:33.089+0000 7f29d0c00640 1 -- 172.21.15.16:0/1067364033 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f29c8007270 con 0x7f29bc003b80 2024-03-31T15:09:33.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:33.089+0000 7f29d0c00640 1 --2- 172.21.15.16:0/1067364033 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f29b4068380 0x7f29b406a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:33.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:33.089+0000 7f29d0c00640 1 -- 172.21.15.16:0/1067364033 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(70..70 src has 1..70) ==== 5882+0+0 (secure 0 0 0) 0x7f29c8120220 con 0x7f29bc003b80 2024-03-31T15:09:33.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:33.089+0000 7f29d3e00640 1 --2- 172.21.15.16:0/1067364033 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f29b4068380 0x7f29b406a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:33.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:33.089+0000 7f29d3e00640 1 --2- 172.21.15.16:0/1067364033 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f29b4068380 0x7f29b406a840 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f29d40cb610 tx=0x7f29c0008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:09:33.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:33.093+0000 7f29d0c00640 1 -- 172.21.15.16:0/1067364033 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f29c80168b0 con 0x7f29bc003b80 2024-03-31T15:09:33.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:33.257+0000 7f29dba00640 1 -- 172.21.15.16:0/1067364033 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f29d40cf3c0 con 0x7f29bc003b80 2024-03-31T15:09:33.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:33.261+0000 7f29d0c00640 1 -- 172.21.15.16:0/1067364033 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v70) ==== 74+0+16870 (secure 0 0 0) 0x7f29c80f0730 con 0x7f29bc003b80 2024-03-31T15:09:33.251 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:09:33.251 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":70,"fsid":"286eb5da-fb47-4550-b290-d9519c13b9a5","created":"2024-03-31T15:03:11.438952+0000","modified":"2024-03-31T15:08:18.287907+0000","last_up_change":"2024-03-31T15:03:22.885535+0000","last_in_change":"2024-03-31T15:03:13.941667+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-31T15:03:22.550787+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-03-31T15:07:55.329096+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"66","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-03-31T15:07:55.885919+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"67","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"92a66450-ef19-40d8-ac3d-d760b23b0bda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6801","nonce":932712244}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6803","nonce":932712244}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6807","nonce":932712244}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6805","nonce":932712244}]},"public_addr":"172.21.15.16:6801/932712244","cluster_addr":"172.21.15.16:6803/932712244","heartbeat_back_addr":"172.21.15.16:6807/932712244","heartbeat_front_addr":"172.21.15.16:6805/932712244","state":["exists","up"]},{"osd":1,"uuid":"da9d1f2b-1968-4959-b843-faa231988e92","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6810","nonce":1398737533}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6811","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6813","nonce":1398737533}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6821","nonce":1398737533}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6815","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6816","nonce":1398737533}]},"public_addr":"172.21.15.16:6810/1398737533","cluster_addr":"172.21.15.16:6813/1398737533","heartbeat_back_addr":"172.21.15.16:6821/1398737533","heartbeat_front_addr":"172.21.15.16:6816/1398737533","state":["exists","up"]},{"osd":2,"uuid":"01a73662-9a21-4b1f-bcbb-42b9636a4638","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6819","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6822","nonce":3018526646}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6825","nonce":3018526646}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6831","nonce":3018526646}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6827","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6829","nonce":3018526646}]},"public_addr":"172.21.15.16:6822/3018526646","cluster_addr":"172.21.15.16:6825/3018526646","heartbeat_back_addr":"172.21.15.16:6831/3018526646","heartbeat_front_addr":"172.21.15.16:6829/3018526646","state":["exists","up"]},{"osd":3,"uuid":"c7771fd3-e18e-4500-ac38-23cf10ebcd75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6809","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6812","nonce":3360352974}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6817","nonce":3360352974}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6828","nonce":3360352974}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6823","nonce":3360352974}]},"public_addr":"172.21.15.16:6812/3360352974","cluster_addr":"172.21.15.16:6817/3360352974","heartbeat_back_addr":"172.21.15.16:6828/3360352974","heartbeat_front_addr":"172.21.15.16:6823/3360352974","state":["exists","up"]},{"osd":4,"uuid":"88260730-1482-4fbd-a40f-cb1124e9df4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6801","nonce":1856844856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6803","nonce":1856844856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6807","nonce":1856844856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6805","nonce":1856844856}]},"public_addr":"172.21.15.89:6801/1856844856","cluster_addr":"172.21.15.89:6803/1856844856","heartbeat_back_addr":"172.21.15.89:6807/1856844856","heartbeat_front_addr":"172.21.15.89:6805/1856844856","state":["exists","up"]},{"osd":5,"uuid":"4cb2c00c-fae0-4dd8-9d27-ec4f0093ccf7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6814","nonce":50395527}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6817","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6820","nonce":50395527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6831","nonce":50395527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6823","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6826","nonce":50395527}]},"public_addr":"172.21.15.89:6814/50395527","cluster_addr":"172.21.15.89:6820/50395527","heartbeat_back_addr":"172.21.15.89:6831/50395527","heartbeat_front_addr":"172.21.15.89:6826/50395527","state":["exists","up"]},{"osd":6,"uuid":"81ce0c19-2554-4cae-acbe-66a05e9e9b9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6809","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6812","nonce":3470471752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6815","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6818","nonce":3470471752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6827","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6829","nonce":3470471752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6821","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6824","nonce":3470471752}]},"public_addr":"172.21.15.89:6812/3470471752","cluster_addr":"172.21.15.89:6818/3470471752","heartbeat_back_addr":"172.21.15.89:6829/3470471752","heartbeat_front_addr":"172.21.15.89:6824/3470471752","state":["exists","up"]},{"osd":7,"uuid":"85e4313b-d37d-442e-8042-153bf20a847d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6811","nonce":1796778630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6813","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6816","nonce":1796778630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6825","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6828","nonce":1796778630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6819","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6822","nonce":1796778630}]},"public_addr":"172.21.15.89:6811/1796778630","cluster_addr":"172.21.15.89:6816/1796778630","heartbeat_back_addr":"172.21.15.89:6828/1796778630","heartbeat_front_addr":"172.21.15.89:6822/1796778630","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.439088+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.529970+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.680512+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:21.044161+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.582111+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.536558+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.409522+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.590229+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"11.4","mappings":[{"from":6,"to":0}]},{"pgid":"11.10","mappings":[{"from":6,"to":1}]},{"pgid":"11.3e","mappings":[{"from":6,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:6835/2555930820":"2024-04-01T15:08:10.537891+0000","172.21.15.16:6834/2555930820":"2024-04-01T15:08:10.537891+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-31T15:09:33.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:33.261+0000 7f29dba00640 1 -- 172.21.15.16:0/1067364033 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f29b4068380 msgr2=0x7f29b406a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:33.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:33.261+0000 7f29dba00640 1 --2- 172.21.15.16:0/1067364033 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f29b4068380 0x7f29b406a840 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f29d40cb610 tx=0x7f29c0008040 comp rx=0 tx=0).stop 2024-03-31T15:09:33.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:33.261+0000 7f29dba00640 1 -- 172.21.15.16:0/1067364033 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f29bc003b80 msgr2=0x7f29d4165b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:33.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:33.261+0000 7f29dba00640 1 --2- 172.21.15.16:0/1067364033 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f29bc003b80 0x7f29d4165b80 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7f29c80ab040 tx=0x7f29bc005840 comp rx=0 tx=0).stop 2024-03-31T15:09:33.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:33.261+0000 7f29dba00640 1 -- 172.21.15.16:0/1067364033 shutdown_connections 2024-03-31T15:09:33.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:33.261+0000 7f29dba00640 1 --2- 172.21.15.16:0/1067364033 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f29b4068380 0x7f29b406a840 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:33.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:33.261+0000 7f29dba00640 1 --2- 172.21.15.16:0/1067364033 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f29d40cc290 0x7f29d40cc710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:33.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:33.261+0000 7f29dba00640 1 --2- 172.21.15.16:0/1067364033 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f29d406d7b0 0x7f29d4144b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:33.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:33.261+0000 7f29dba00640 1 --2- 172.21.15.16:0/1067364033 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f29bc003b80 0x7f29d4165b80 unknown :-1 s=CLOSED pgs=988 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:33.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:33.261+0000 7f29dba00640 1 -- 172.21.15.16:0/1067364033 >> 172.21.15.16:0/1067364033 conn(0x7f29d4067330 msgr2=0x7f29d415d8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:09:33.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:33.261+0000 7f29dba00640 1 -- 172.21.15.16:0/1067364033 shutdown_connections 2024-03-31T15:09:33.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:33.261+0000 7f29dba00640 1 -- 172.21.15.16:0/1067364033 wait complete. 2024-03-31T15:09:33.273 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-31T15:03:22.550787+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 10, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-31T15:07:55.329096+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '66', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.5, 'score_stable': 2.5, 'optimal_score': 1, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 11, 'pool_name': 'cephfs_data', 'create_time': '2024-03-31T15:07:55.885919+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '67', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-31T15:09:33.273 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-31T15:09:33.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:33.425+0000 7f42d6c00640 1 Processor -- start 2024-03-31T15:09:33.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:33.425+0000 7f42d6c00640 1 -- start start 2024-03-31T15:09:33.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:33.429+0000 7f42d6c00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f42d00cf150 0x7f42d01670a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:33.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:33.429+0000 7f42d6c00640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f42d015ca70 con 0x7f42d006f920 2024-03-31T15:09:33.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:33.429+0000 7f42d6c00640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f42d015cd50 con 0x7f42d00d0a60 2024-03-31T15:09:33.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:33.429+0000 7f42d6c00640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f42d013f670 con 0x7f42d00cf150 2024-03-31T15:09:33.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:33.429+0000 7f42cfe00640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f42d006f920 0x7f42d006fd00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:46706/0 (socket says 172.21.15.16:46706) 2024-03-31T15:09:33.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:33.429+0000 7f42cfe00640 1 -- 172.21.15.16:0/3068154048 learned_addr learned my addr 172.21.15.16:0/3068154048 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:09:33.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:33.429+0000 7f42d4c00640 1 --2- 172.21.15.16:0/3068154048 >> v2:172.21.15.89:3300/0 conn(0x7f42d00cf150 0x7f42d01670a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:33.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:33.429+0000 7f42cf400640 1 -- 172.21.15.16:0/3068154048 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3793540675 0 0) 0x7f42d015ca70 con 0x7f42d006f920 2024-03-31T15:09:33.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:33.429+0000 7f42cf400640 1 -- 172.21.15.16:0/3068154048 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f42b4003530 con 0x7f42d006f920 2024-03-31T15:09:33.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:33.429+0000 7f42cf400640 1 -- 172.21.15.16:0/3068154048 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3967276295 0 0) 0x7f42b4003530 con 0x7f42d006f920 2024-03-31T15:09:33.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:33.429+0000 7f42cf400640 1 -- 172.21.15.16:0/3068154048 >> v2:172.21.15.89:3300/0 conn(0x7f42d00cf150 msgr2=0x7f42d01670a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:09:33.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:33.429+0000 7f42cf400640 1 --2- 172.21.15.16:0/3068154048 >> v2:172.21.15.89:3300/0 conn(0x7f42d00cf150 0x7f42d01670a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:33.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:33.429+0000 7f42cf400640 1 -- 172.21.15.16:0/3068154048 >> v1:172.21.15.89:6789/0 conn(0x7f42d00d0a60 legacy=0x7f42d00cea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:33.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:33.429+0000 7f42cf400640 1 -- 172.21.15.16:0/3068154048 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42d015d030 con 0x7f42d006f920 2024-03-31T15:09:33.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:33.429+0000 7f42cf400640 1 -- 172.21.15.16:0/3068154048 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 214807623 0 0) 0x7f42c0001680 con 0x7f42d006f920 2024-03-31T15:09:33.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:33.429+0000 7f42cf400640 1 -- 172.21.15.16:0/3068154048 >> v1:172.21.15.16:6789/0 conn(0x7f42d006f920 legacy=0x7f42d006fd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:33.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:33.429+0000 7f42cf400640 1 --2- 172.21.15.16:0/3068154048 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f42b4003a10 0x7f42b4003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:33.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:33.429+0000 7f42cf400640 1 --2- 172.21.15.16:0/3068154048 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f42b4004810 0x7f42b4004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:33.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:33.429+0000 7f42cf400640 1 --2- 172.21.15.16:0/3068154048 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f42b40051d0 0x7f42b400b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:33.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:33.429+0000 7f42cf400640 1 -- 172.21.15.16:0/3068154048 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f42d013f670 con 0x7f42b4004810 2024-03-31T15:09:33.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:33.429+0000 7f42cf400640 1 -- 172.21.15.16:0/3068154048 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f42b400c0d0 con 0x7f42b40051d0 2024-03-31T15:09:33.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:33.429+0000 7f42cf400640 1 -- 172.21.15.16:0/3068154048 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f42b400c2d0 con 0x7f42b4003a10 2024-03-31T15:09:33.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:33.429+0000 7f42c7400640 1 --2- 172.21.15.16:0/3068154048 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f42b4004810 0x7f42b4004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:33.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:33.429+0000 7f42cfe00640 1 --2- 172.21.15.16:0/3068154048 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f42b4003a10 0x7f42b4003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:33.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:33.429+0000 7f42d4c00640 1 --2- 172.21.15.16:0/3068154048 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f42b40051d0 0x7f42b400b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:33.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:33.429+0000 7f42c7400640 1 -- 172.21.15.16:0/3068154048 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f42b4003a10 msgr2=0x7f42b4003e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:09:33.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:33.429+0000 7f42c7400640 1 --2- 172.21.15.16:0/3068154048 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f42b4003a10 0x7f42b4003e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:33.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:33.429+0000 7f42c7400640 1 -- 172.21.15.16:0/3068154048 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f42b40051d0 msgr2=0x7f42b400b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:09:33.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:33.429+0000 7f42c7400640 1 --2- 172.21.15.16:0/3068154048 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f42b40051d0 0x7f42b400b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:33.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:33.429+0000 7f42c7400640 1 -- 172.21.15.16:0/3068154048 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42b4003530 con 0x7f42b4004810 2024-03-31T15:09:33.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:33.429+0000 7f42d4c00640 1 -- 172.21.15.16:0/3068154048 reap_dead start 2024-03-31T15:09:33.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:33.433+0000 7f42c7400640 1 --2- 172.21.15.16:0/3068154048 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f42b4004810 0x7f42b4004c90 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x7f42b8001750 tx=0x7f42b801c3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=61a0c3af7c9d0b9e server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:09:33.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:33.433+0000 7f42cf400640 1 -- 172.21.15.16:0/3068154048 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42b80013a0 con 0x7f42b4004810 2024-03-31T15:09:33.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:33.433+0000 7f42cf400640 1 -- 172.21.15.16:0/3068154048 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f42b80052f0 con 0x7f42b4004810 2024-03-31T15:09:33.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:33.433+0000 7f42cf400640 1 -- 172.21.15.16:0/3068154048 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42c0007e60 con 0x7f42b4004810 2024-03-31T15:09:33.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:33.433+0000 7f42d6c00640 1 -- 172.21.15.16:0/3068154048 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f42b4004810 msgr2=0x7f42b4004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:33.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:33.433+0000 7f42d6c00640 1 --2- 172.21.15.16:0/3068154048 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f42b4004810 0x7f42b4004c90 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x7f42b8001750 tx=0x7f42b801c3c0 comp rx=0 tx=0).stop 2024-03-31T15:09:33.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:33.433+0000 7f42d6c00640 1 -- 172.21.15.16:0/3068154048 shutdown_connections 2024-03-31T15:09:33.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:33.433+0000 7f42d6c00640 1 --2- 172.21.15.16:0/3068154048 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f42b4004810 0x7f42b4004c90 unknown :-1 s=CLOSED pgs=990 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:33.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:33.433+0000 7f42d6c00640 1 -- 172.21.15.16:0/3068154048 >> 172.21.15.16:0/3068154048 conn(0x7f42d0067930 msgr2=0x7f42d00b3ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:09:33.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:33.433+0000 7f42d6c00640 1 -- 172.21.15.16:0/3068154048 shutdown_connections 2024-03-31T15:09:33.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:33.433+0000 7f42d6c00640 1 -- 172.21.15.16:0/3068154048 wait complete. 2024-03-31T15:09:33.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:33.437+0000 7f42d6c00640 1 Processor -- start 2024-03-31T15:09:33.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:33.437+0000 7f42d6c00640 1 -- start start 2024-03-31T15:09:33.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:33.437+0000 7f42d6c00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f42b4004810 0x7f42d01e57e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:33.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:33.437+0000 7f42d6c00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f42d015daa0 0x7f42d013e640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:33.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:33.437+0000 7f42d6c00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f42d01e5fa0 0x7f42d01e6450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:33.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:33.437+0000 7f42d6c00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f42d006fd00 con 0x7f42d015daa0 2024-03-31T15:09:33.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:33.437+0000 7f42d6c00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f42d015c720 con 0x7f42d01e5fa0 2024-03-31T15:09:33.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:33.437+0000 7f42d6c00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f42d006fe80 con 0x7f42b4004810 2024-03-31T15:09:33.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:33.437+0000 7f42c7400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f42d015daa0 0x7f42d013e640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:33.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:33.437+0000 7f42c7400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f42d015daa0 0x7f42d013e640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60726/0 (socket says 172.21.15.16:60726) 2024-03-31T15:09:33.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:33.437+0000 7f42c7400640 1 -- 172.21.15.16:0/1966604488 learned_addr learned my addr 172.21.15.16:0/1966604488 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:09:33.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:33.437+0000 7f42d4c00640 1 --2- 172.21.15.16:0/1966604488 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f42d01e5fa0 0x7f42d01e6450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:33.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:33.437+0000 7f42cfe00640 1 --2- 172.21.15.16:0/1966604488 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f42b4004810 0x7f42d01e57e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:33.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:33.441+0000 7f42c7400640 1 -- 172.21.15.16:0/1966604488 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f42b4004810 msgr2=0x7f42d01e57e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:33.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:33.441+0000 7f42c7400640 1 --2- 172.21.15.16:0/1966604488 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f42b4004810 0x7f42d01e57e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:33.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:33.441+0000 7f42c7400640 1 -- 172.21.15.16:0/1966604488 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f42d01e5fa0 msgr2=0x7f42d01e6450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:33.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:33.441+0000 7f42c7400640 1 --2- 172.21.15.16:0/1966604488 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f42d01e5fa0 0x7f42d01e6450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:33.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:33.441+0000 7f42c7400640 1 -- 172.21.15.16:0/1966604488 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42d013eb80 con 0x7f42d015daa0 2024-03-31T15:09:33.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:33.441+0000 7f42c7400640 1 --2- 172.21.15.16:0/1966604488 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f42d015daa0 0x7f42d013e640 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x7f42b801c1f0 tx=0x7f42b4005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:09:33.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:33.441+0000 7f42ccc00640 1 -- 172.21.15.16:0/1966604488 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42b80013a0 con 0x7f42d015daa0 2024-03-31T15:09:33.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:33.441+0000 7f42ccc00640 1 -- 172.21.15.16:0/1966604488 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f42b80ba070 con 0x7f42d015daa0 2024-03-31T15:09:33.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:33.441+0000 7f42ccc00640 1 -- 172.21.15.16:0/1966604488 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42b8007a40 con 0x7f42d015daa0 2024-03-31T15:09:33.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:33.441+0000 7f42d6c00640 1 -- 172.21.15.16:0/1966604488 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f42d01f3380 con 0x7f42d015daa0 2024-03-31T15:09:33.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:33.441+0000 7f42d6c00640 1 -- 172.21.15.16:0/1966604488 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f42d0154700 con 0x7f42d015daa0 2024-03-31T15:09:33.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:33.441+0000 7f42d6c00640 1 -- 172.21.15.16:0/1966604488 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f42d0069950 con 0x7f42d015daa0 2024-03-31T15:09:33.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:33.441+0000 7f42ccc00640 1 -- 172.21.15.16:0/1966604488 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f42b80052f0 con 0x7f42d015daa0 2024-03-31T15:09:33.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:33.441+0000 7f42ccc00640 1 --2- 172.21.15.16:0/1966604488 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f42ac068380 0x7f42ac06a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:33.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:33.441+0000 7f42ccc00640 1 -- 172.21.15.16:0/1966604488 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(70..70 src has 1..70) ==== 5882+0+0 (secure 0 0 0) 0x7f42b8120d00 con 0x7f42d015daa0 2024-03-31T15:09:33.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:33.445+0000 7f42cfe00640 1 --2- 172.21.15.16:0/1966604488 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f42ac068380 0x7f42ac06a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:33.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:33.445+0000 7f42cfe00640 1 --2- 172.21.15.16:0/1966604488 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f42ac068380 0x7f42ac06a840 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f42c0007fd0 tx=0x7f42c0007e60 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:09:33.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:33.445+0000 7f42ccc00640 1 -- 172.21.15.16:0/1966604488 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f42b80160d0 con 0x7f42d015daa0 2024-03-31T15:09:33.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:33.617+0000 7f42d6c00640 1 -- 172.21.15.16:0/1966604488 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f42d01e5d80 con 0x7f42d015daa0 2024-03-31T15:09:33.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:33.617+0000 7f42ccc00640 1 -- 172.21.15.16:0/1966604488 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v70) ==== 93+0+10 (secure 0 0 0) 0x7f42b80f1170 con 0x7f42d015daa0 2024-03-31T15:09:33.609 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2024-03-31T15:09:33.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:33.617+0000 7f42d6c00640 1 -- 172.21.15.16:0/1966604488 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f42ac068380 msgr2=0x7f42ac06a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:33.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:33.617+0000 7f42d6c00640 1 --2- 172.21.15.16:0/1966604488 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f42ac068380 0x7f42ac06a840 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f42c0007fd0 tx=0x7f42c0007e60 comp rx=0 tx=0).stop 2024-03-31T15:09:33.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:33.617+0000 7f42d6c00640 1 -- 172.21.15.16:0/1966604488 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f42d015daa0 msgr2=0x7f42d013e640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:33.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:33.617+0000 7f42d6c00640 1 --2- 172.21.15.16:0/1966604488 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f42d015daa0 0x7f42d013e640 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x7f42b801c1f0 tx=0x7f42b4005840 comp rx=0 tx=0).stop 2024-03-31T15:09:33.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:33.617+0000 7f42d6c00640 1 -- 172.21.15.16:0/1966604488 shutdown_connections 2024-03-31T15:09:33.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:33.617+0000 7f42d6c00640 1 --2- 172.21.15.16:0/1966604488 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f42ac068380 0x7f42ac06a840 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:33.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:33.617+0000 7f42d6c00640 1 --2- 172.21.15.16:0/1966604488 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f42d01e5fa0 0x7f42d01e6450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:33.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:33.617+0000 7f42d6c00640 1 --2- 172.21.15.16:0/1966604488 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f42d015daa0 0x7f42d013e640 unknown :-1 s=CLOSED pgs=991 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:33.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:33.617+0000 7f42d6c00640 1 --2- 172.21.15.16:0/1966604488 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f42b4004810 0x7f42d01e57e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:33.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:33.617+0000 7f42d6c00640 1 -- 172.21.15.16:0/1966604488 >> 172.21.15.16:0/1966604488 conn(0x7f42d0067930 msgr2=0x7f42d00ce8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:09:33.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:33.621+0000 7f42d6c00640 1 -- 172.21.15.16:0/1966604488 shutdown_connections 2024-03-31T15:09:33.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:33.621+0000 7f42d6c00640 1 -- 172.21.15.16:0/1966604488 wait complete. 2024-03-31T15:09:33.630 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-31T15:09:33.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:33.781+0000 7f1d71e00640 1 Processor -- start 2024-03-31T15:09:33.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:33.781+0000 7f1d71e00640 1 -- start start 2024-03-31T15:09:33.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:33.781+0000 7f1d71e00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f1d6c06f920 0x7f1d6c06fd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:33.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:33.781+0000 7f1d71e00640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1d6c070240 con 0x7f1d6c0cf150 2024-03-31T15:09:33.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:33.781+0000 7f1d71e00640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1d6c15ca70 con 0x7f1d6c0d0a60 2024-03-31T15:09:33.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:33.781+0000 7f1d71e00640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f1d6c13f670 con 0x7f1d6c06f920 2024-03-31T15:09:33.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:33.781+0000 7f1d6aa00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f1d6c06f920 0x7f1d6c06fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:33.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:33.781+0000 7f1d6b400640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f1d6c0cf150 0x7f1d6c1419c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:46716/0 (socket says 172.21.15.16:46716) 2024-03-31T15:09:33.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:33.781+0000 7f1d6b400640 1 -- 172.21.15.16:0/2011580448 learned_addr learned my addr 172.21.15.16:0/2011580448 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:09:33.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:33.785+0000 7f1d6aa00640 1 -- 172.21.15.16:0/2011580448 >> v1:172.21.15.89:6789/0 conn(0x7f1d6c0d0a60 legacy=0x7f1d6c0cea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:33.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:33.785+0000 7f1d6aa00640 1 -- 172.21.15.16:0/2011580448 >> v1:172.21.15.16:6789/0 conn(0x7f1d6c0cf150 legacy=0x7f1d6c1419c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:33.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:33.785+0000 7f1d6aa00640 1 -- 172.21.15.16:0/2011580448 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1d6c15cd50 con 0x7f1d6c06f920 2024-03-31T15:09:33.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:33.785+0000 7f1d6aa00640 1 --2- 172.21.15.16:0/2011580448 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1d6c06f920 0x7f1d6c06fd00 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 crypto rx=0x7f1d5c0097c0 tx=0x7f1d5c01c410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e23b0667995d1b2 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:09:33.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:33.785+0000 7f1d6a000640 1 -- 172.21.15.16:0/2011580448 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d5c0b0070 con 0x7f1d6c06f920 2024-03-31T15:09:33.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:33.785+0000 7f1d6a000640 1 -- 172.21.15.16:0/2011580448 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1d5c0ab600 con 0x7f1d6c06f920 2024-03-31T15:09:33.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:33.785+0000 7f1d6a000640 1 -- 172.21.15.16:0/2011580448 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d5c0ab9d0 con 0x7f1d6c06f920 2024-03-31T15:09:33.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:33.789+0000 7f1d71e00640 1 -- 172.21.15.16:0/2011580448 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1d6c06f920 msgr2=0x7f1d6c06fd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:33.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:33.789+0000 7f1d71e00640 1 --2- 172.21.15.16:0/2011580448 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1d6c06f920 0x7f1d6c06fd00 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 crypto rx=0x7f1d5c0097c0 tx=0x7f1d5c01c410 comp rx=0 tx=0).stop 2024-03-31T15:09:33.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:33.789+0000 7f1d71e00640 1 -- 172.21.15.16:0/2011580448 shutdown_connections 2024-03-31T15:09:33.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:33.789+0000 7f1d71e00640 1 --2- 172.21.15.16:0/2011580448 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1d6c06f920 0x7f1d6c06fd00 unknown :-1 s=CLOSED pgs=993 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:33.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:33.789+0000 7f1d71e00640 1 -- 172.21.15.16:0/2011580448 >> 172.21.15.16:0/2011580448 conn(0x7f1d6c067930 msgr2=0x7f1d6c0b3ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:09:33.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:33.789+0000 7f1d71e00640 1 -- 172.21.15.16:0/2011580448 shutdown_connections 2024-03-31T15:09:33.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:33.789+0000 7f1d71e00640 1 -- 172.21.15.16:0/2011580448 wait complete. 2024-03-31T15:09:33.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:33.789+0000 7f1d71e00640 1 Processor -- start 2024-03-31T15:09:33.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:33.789+0000 7f1d71e00640 1 -- start start 2024-03-31T15:09:33.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:33.789+0000 7f1d71e00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f1d6c06f920 0x7f1d6c0b6d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:33.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:33.789+0000 7f1d71e00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1d6c0cf150 0x7f1d6c141270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:33.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:33.789+0000 7f1d71e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1d6c0d0a60 0x7f1d6c1558e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:33.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:33.789+0000 7f1d71e00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f1d6c16a240 con 0x7f1d6c0d0a60 2024-03-31T15:09:33.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:33.789+0000 7f1d71e00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f1d6c16a0c0 con 0x7f1d6c0cf150 2024-03-31T15:09:33.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:33.789+0000 7f1d71e00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f1d6c16a3c0 con 0x7f1d6c06f920 2024-03-31T15:09:33.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:33.789+0000 7f1d6b400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1d6c0d0a60 0x7f1d6c1558e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:33.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:33.789+0000 7f1d6b400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1d6c0d0a60 0x7f1d6c1558e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60746/0 (socket says 172.21.15.16:60746) 2024-03-31T15:09:33.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:33.789+0000 7f1d6b400640 1 -- 172.21.15.16:0/1078250347 learned_addr learned my addr 172.21.15.16:0/1078250347 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:09:33.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:33.789+0000 7f1d6aa00640 1 --2- 172.21.15.16:0/1078250347 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f1d6c06f920 0x7f1d6c0b6d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:33.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:33.789+0000 7f1d63e00640 1 --2- 172.21.15.16:0/1078250347 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1d6c0cf150 0x7f1d6c141270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:33.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:33.789+0000 7f1d6b400640 1 -- 172.21.15.16:0/1078250347 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f1d6c06f920 msgr2=0x7f1d6c0b6d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:33.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:33.789+0000 7f1d6b400640 1 --2- 172.21.15.16:0/1078250347 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f1d6c06f920 0x7f1d6c0b6d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:33.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:33.789+0000 7f1d6b400640 1 -- 172.21.15.16:0/1078250347 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1d6c0cf150 msgr2=0x7f1d6c141270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:33.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:33.789+0000 7f1d6b400640 1 --2- 172.21.15.16:0/1078250347 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1d6c0cf150 0x7f1d6c141270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:33.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:33.789+0000 7f1d6b400640 1 -- 172.21.15.16:0/1078250347 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1d6c1417b0 con 0x7f1d6c0d0a60 2024-03-31T15:09:33.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:33.789+0000 7f1d6b400640 1 --2- 172.21.15.16:0/1078250347 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1d6c0d0a60 0x7f1d6c1558e0 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 crypto rx=0x7f1d6400cbf0 tx=0x7f1d6400a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:09:33.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:33.789+0000 7f1d63400640 1 -- 172.21.15.16:0/1078250347 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d64017070 con 0x7f1d6c0d0a60 2024-03-31T15:09:33.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:33.793+0000 7f1d63400640 1 -- 172.21.15.16:0/1078250347 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1d6400ad40 con 0x7f1d6c0d0a60 2024-03-31T15:09:33.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:33.793+0000 7f1d63400640 1 -- 172.21.15.16:0/1078250347 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d64012600 con 0x7f1d6c0d0a60 2024-03-31T15:09:33.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:33.793+0000 7f1d71e00640 1 -- 172.21.15.16:0/1078250347 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1d6c13f890 con 0x7f1d6c0d0a60 2024-03-31T15:09:33.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:33.793+0000 7f1d71e00640 1 -- 172.21.15.16:0/1078250347 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1d6c1550e0 con 0x7f1d6c0d0a60 2024-03-31T15:09:33.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:33.793+0000 7f1d71e00640 1 -- 172.21.15.16:0/1078250347 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1d6c069950 con 0x7f1d6c0d0a60 2024-03-31T15:09:33.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:33.793+0000 7f1d63400640 1 -- 172.21.15.16:0/1078250347 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f1d6401a070 con 0x7f1d6c0d0a60 2024-03-31T15:09:33.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:33.793+0000 7f1d63400640 1 --2- 172.21.15.16:0/1078250347 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f1d48068380 0x7f1d4806a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:33.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:33.793+0000 7f1d63400640 1 -- 172.21.15.16:0/1078250347 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(70..70 src has 1..70) ==== 5882+0+0 (secure 0 0 0) 0x7f1d640866d0 con 0x7f1d6c0d0a60 2024-03-31T15:09:33.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:33.793+0000 7f1d6aa00640 1 --2- 172.21.15.16:0/1078250347 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f1d48068380 0x7f1d4806a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:33.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:33.797+0000 7f1d6aa00640 1 --2- 172.21.15.16:0/1078250347 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f1d48068380 0x7f1d4806a840 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f1d5c0ad000 tx=0x7f1d5c008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:09:33.790 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:33.797+0000 7f1d63400640 1 -- 172.21.15.16:0/1078250347 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f1d64056580 con 0x7f1d6c0d0a60 2024-03-31T15:09:33.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:33.961+0000 7f1d71e00640 1 -- 172.21.15.16:0/1078250347 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f1d6c0b4a20 con 0x7f1d6c0d0a60 2024-03-31T15:09:33.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:33.961+0000 7f1d63400640 1 -- 172.21.15.16:0/1078250347 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v70) ==== 104+0+11 (secure 0 0 0) 0x7f1d6405a880 con 0x7f1d6c0d0a60 2024-03-31T15:09:33.954 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2024-03-31T15:09:33.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:33.965+0000 7f1d71e00640 1 -- 172.21.15.16:0/1078250347 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f1d48068380 msgr2=0x7f1d4806a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:33.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:33.965+0000 7f1d71e00640 1 --2- 172.21.15.16:0/1078250347 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f1d48068380 0x7f1d4806a840 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f1d5c0ad000 tx=0x7f1d5c008040 comp rx=0 tx=0).stop 2024-03-31T15:09:33.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:33.965+0000 7f1d71e00640 1 -- 172.21.15.16:0/1078250347 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1d6c0d0a60 msgr2=0x7f1d6c1558e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:33.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:33.965+0000 7f1d71e00640 1 --2- 172.21.15.16:0/1078250347 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1d6c0d0a60 0x7f1d6c1558e0 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 crypto rx=0x7f1d6400cbf0 tx=0x7f1d6400a430 comp rx=0 tx=0).stop 2024-03-31T15:09:33.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:33.965+0000 7f1d71e00640 1 -- 172.21.15.16:0/1078250347 shutdown_connections 2024-03-31T15:09:33.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:33.965+0000 7f1d71e00640 1 --2- 172.21.15.16:0/1078250347 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f1d48068380 0x7f1d4806a840 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:33.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:33.965+0000 7f1d71e00640 1 --2- 172.21.15.16:0/1078250347 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1d6c0d0a60 0x7f1d6c1558e0 unknown :-1 s=CLOSED pgs=994 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:33.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:33.965+0000 7f1d71e00640 1 --2- 172.21.15.16:0/1078250347 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1d6c0cf150 0x7f1d6c141270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:33.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:33.965+0000 7f1d71e00640 1 --2- 172.21.15.16:0/1078250347 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f1d6c06f920 0x7f1d6c0b6d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:33.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:33.965+0000 7f1d71e00640 1 -- 172.21.15.16:0/1078250347 >> 172.21.15.16:0/1078250347 conn(0x7f1d6c067930 msgr2=0x7f1d6c15a2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:09:33.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:33.965+0000 7f1d71e00640 1 -- 172.21.15.16:0/1078250347 shutdown_connections 2024-03-31T15:09:33.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:33.965+0000 7f1d71e00640 1 -- 172.21.15.16:0/1078250347 wait complete. 2024-03-31T15:09:33.974 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-31T15:09:34.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:34.125+0000 7fd6ccc00640 1 Processor -- start 2024-03-31T15:09:34.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:34.125+0000 7fd6ccc00640 1 -- start start 2024-03-31T15:09:34.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:34.125+0000 7fd6ccc00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fd6c806f920 0x7fd6c806fd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:34.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:34.125+0000 7fd6ccc00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fd6c80cf150 0x7fd6c81419c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:34.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:34.125+0000 7fd6ccc00640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd6c8070240 con 0x7fd6c80d0a60 2024-03-31T15:09:34.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:34.125+0000 7fd6ccc00640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7fd6c813f670 con 0x7fd6c806f920 2024-03-31T15:09:34.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:34.125+0000 7fd6ccc00640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7fd6c815ca70 con 0x7fd6c80cf150 2024-03-31T15:09:34.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:34.129+0000 7fd6c5e00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fd6c806f920 0x7fd6c806fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:34.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:34.129+0000 7fd6c5e00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fd6c806f920 0x7fd6c806fd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60750/0 (socket says 172.21.15.16:60750) 2024-03-31T15:09:34.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:34.129+0000 7fd6c5e00640 1 -- 172.21.15.16:0/2583328903 learned_addr learned my addr 172.21.15.16:0/2583328903 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:09:34.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:34.129+0000 7fd6c6800640 1 --2- 172.21.15.16:0/2583328903 >> v2:172.21.15.89:3300/0 conn(0x7fd6c80cf150 0x7fd6c81419c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:34.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:34.129+0000 7fd6c5e00640 1 -- 172.21.15.16:0/2583328903 >> v2:172.21.15.89:3300/0 conn(0x7fd6c80cf150 msgr2=0x7fd6c81419c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:09:34.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:34.129+0000 7fd6c5e00640 1 --2- 172.21.15.16:0/2583328903 >> v2:172.21.15.89:3300/0 conn(0x7fd6c80cf150 0x7fd6c81419c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:34.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:34.129+0000 7fd6c5e00640 1 -- 172.21.15.16:0/2583328903 >> v1:172.21.15.16:6789/0 conn(0x7fd6c80d0a60 legacy=0x7fd6c80cea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:34.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:34.129+0000 7fd6c5e00640 1 -- 172.21.15.16:0/2583328903 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6c815cd50 con 0x7fd6c806f920 2024-03-31T15:09:34.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:34.129+0000 7fd6c5e00640 1 --2- 172.21.15.16:0/2583328903 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd6c806f920 0x7fd6c806fd00 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 crypto rx=0x7fd6c80d2080 tx=0x7fd6b801c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f2843a5ac4193a34 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:09:34.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:34.129+0000 7fd6c5400640 1 -- 172.21.15.16:0/2583328903 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6b80b0070 con 0x7fd6c806f920 2024-03-31T15:09:34.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:34.129+0000 7fd6c5400640 1 -- 172.21.15.16:0/2583328903 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd6b80ab600 con 0x7fd6c806f920 2024-03-31T15:09:34.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:34.129+0000 7fd6c5400640 1 -- 172.21.15.16:0/2583328903 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6b80ab9d0 con 0x7fd6c806f920 2024-03-31T15:09:34.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:34.133+0000 7fd6ccc00640 1 -- 172.21.15.16:0/2583328903 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd6c806f920 msgr2=0x7fd6c806fd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:34.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:34.133+0000 7fd6ccc00640 1 --2- 172.21.15.16:0/2583328903 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd6c806f920 0x7fd6c806fd00 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 crypto rx=0x7fd6c80d2080 tx=0x7fd6b801c490 comp rx=0 tx=0).stop 2024-03-31T15:09:34.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:34.133+0000 7fd6ccc00640 1 -- 172.21.15.16:0/2583328903 shutdown_connections 2024-03-31T15:09:34.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:34.133+0000 7fd6ccc00640 1 --2- 172.21.15.16:0/2583328903 >> v2:172.21.15.89:3300/0 conn(0x7fd6c80cf150 0x7fd6c81419c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:34.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:34.133+0000 7fd6ccc00640 1 --2- 172.21.15.16:0/2583328903 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd6c806f920 0x7fd6c806fd00 unknown :-1 s=CLOSED pgs=996 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:34.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:34.133+0000 7fd6ccc00640 1 -- 172.21.15.16:0/2583328903 >> 172.21.15.16:0/2583328903 conn(0x7fd6c8067930 msgr2=0x7fd6c80b3ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:09:34.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:34.133+0000 7fd6ccc00640 1 -- 172.21.15.16:0/2583328903 shutdown_connections 2024-03-31T15:09:34.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:34.133+0000 7fd6ccc00640 1 -- 172.21.15.16:0/2583328903 wait complete. 2024-03-31T15:09:34.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:34.133+0000 7fd6ccc00640 1 Processor -- start 2024-03-31T15:09:34.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:34.133+0000 7fd6ccc00640 1 -- start start 2024-03-31T15:09:34.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:34.137+0000 7fd6ccc00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd6c806f920 0x7fd6c813d6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:34.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:34.137+0000 7fd6ccc00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd6c80cf150 0x7fd6c813e890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:34.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:34.137+0000 7fd6ccc00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fd6c80d0a60 0x7fd6c813edd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:34.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:34.137+0000 7fd6ccc00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fd6c8169230 con 0x7fd6c80cf150 2024-03-31T15:09:34.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:34.137+0000 7fd6ccc00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fd6c81690b0 con 0x7fd6c806f920 2024-03-31T15:09:34.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:34.137+0000 7fd6ccc00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fd6c81693b0 con 0x7fd6c80d0a60 2024-03-31T15:09:34.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:34.137+0000 7fd6bfe00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd6c80cf150 0x7fd6c813e890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:34.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:34.137+0000 7fd6bfe00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd6c80cf150 0x7fd6c813e890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60766/0 (socket says 172.21.15.16:60766) 2024-03-31T15:09:34.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:34.137+0000 7fd6bfe00640 1 -- 172.21.15.16:0/3335954382 learned_addr learned my addr 172.21.15.16:0/3335954382 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:09:34.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:34.137+0000 7fd6c5e00640 1 --2- 172.21.15.16:0/3335954382 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd6c806f920 0x7fd6c813d6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:34.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:34.137+0000 7fd6c6800640 1 --2- 172.21.15.16:0/3335954382 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fd6c80d0a60 0x7fd6c813edd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:34.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:34.137+0000 7fd6bfe00640 1 -- 172.21.15.16:0/3335954382 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fd6c80d0a60 msgr2=0x7fd6c813edd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:34.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:34.137+0000 7fd6bfe00640 1 --2- 172.21.15.16:0/3335954382 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fd6c80d0a60 0x7fd6c813edd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:34.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:34.137+0000 7fd6bfe00640 1 -- 172.21.15.16:0/3335954382 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd6c806f920 msgr2=0x7fd6c813d6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:34.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:34.137+0000 7fd6bfe00640 1 --2- 172.21.15.16:0/3335954382 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd6c806f920 0x7fd6c813d6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:34.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:34.137+0000 7fd6bfe00640 1 -- 172.21.15.16:0/3335954382 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6c815d9d0 con 0x7fd6c80cf150 2024-03-31T15:09:34.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:34.137+0000 7fd6c6800640 1 --2- 172.21.15.16:0/3335954382 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fd6c80d0a60 0x7fd6c813edd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:09:34.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:34.137+0000 7fd6bfe00640 1 --2- 172.21.15.16:0/3335954382 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd6c80cf150 0x7fd6c813e890 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7fd6b000c340 tx=0x7fd6b000c800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:09:34.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:34.137+0000 7fd6be000640 1 -- 172.21.15.16:0/3335954382 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6b00052d0 con 0x7fd6c80cf150 2024-03-31T15:09:34.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:34.137+0000 7fd6be000640 1 -- 172.21.15.16:0/3335954382 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd6b0005470 con 0x7fd6c80cf150 2024-03-31T15:09:34.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:34.137+0000 7fd6be000640 1 -- 172.21.15.16:0/3335954382 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6b0007260 con 0x7fd6c80cf150 2024-03-31T15:09:34.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:34.137+0000 7fd6ccc00640 1 -- 172.21.15.16:0/3335954382 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd6c815dc90 con 0x7fd6c80cf150 2024-03-31T15:09:34.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:34.137+0000 7fd6ccc00640 1 -- 172.21.15.16:0/3335954382 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd6c815e250 con 0x7fd6c80cf150 2024-03-31T15:09:34.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:34.137+0000 7fd6ccc00640 1 -- 172.21.15.16:0/3335954382 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd6c8069950 con 0x7fd6c80cf150 2024-03-31T15:09:34.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:34.141+0000 7fd6be000640 1 -- 172.21.15.16:0/3335954382 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fd6b0007400 con 0x7fd6c80cf150 2024-03-31T15:09:34.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:34.141+0000 7fd6be000640 1 --2- 172.21.15.16:0/3335954382 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fd6a4068380 0x7fd6a406a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:34.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:34.141+0000 7fd6be000640 1 -- 172.21.15.16:0/3335954382 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(70..70 src has 1..70) ==== 5882+0+0 (secure 0 0 0) 0x7fd6b0085e60 con 0x7fd6c80cf150 2024-03-31T15:09:34.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:34.141+0000 7fd6c5e00640 1 --2- 172.21.15.16:0/3335954382 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fd6a4068380 0x7fd6a406a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:34.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:34.141+0000 7fd6c5e00640 1 --2- 172.21.15.16:0/3335954382 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fd6a4068380 0x7fd6a406a840 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7fd6b80ad000 tx=0x7fd6b8008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:09:34.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:34.145+0000 7fd6be000640 1 -- 172.21.15.16:0/3335954382 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fd6b0055d10 con 0x7fd6c80cf150 2024-03-31T15:09:34.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:34.309+0000 7fd6ccc00640 1 -- 172.21.15.16:0/3335954382 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fd6c80b7170 con 0x7fd6c80cf150 2024-03-31T15:09:34.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:34.309+0000 7fd6be000640 1 -- 172.21.15.16:0/3335954382 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v70) ==== 100+0+11 (secure 0 0 0) 0x7fd6b005a010 con 0x7fd6c80cf150 2024-03-31T15:09:34.300 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 64 2024-03-31T15:09:34.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:34.309+0000 7fd6ccc00640 1 -- 172.21.15.16:0/3335954382 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fd6a4068380 msgr2=0x7fd6a406a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:34.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:34.309+0000 7fd6ccc00640 1 --2- 172.21.15.16:0/3335954382 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fd6a4068380 0x7fd6a406a840 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7fd6b80ad000 tx=0x7fd6b8008040 comp rx=0 tx=0).stop 2024-03-31T15:09:34.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:34.309+0000 7fd6ccc00640 1 -- 172.21.15.16:0/3335954382 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd6c80cf150 msgr2=0x7fd6c813e890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:34.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:34.309+0000 7fd6ccc00640 1 --2- 172.21.15.16:0/3335954382 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd6c80cf150 0x7fd6c813e890 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7fd6b000c340 tx=0x7fd6b000c800 comp rx=0 tx=0).stop 2024-03-31T15:09:34.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:34.309+0000 7fd6ccc00640 1 -- 172.21.15.16:0/3335954382 shutdown_connections 2024-03-31T15:09:34.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:34.309+0000 7fd6ccc00640 1 --2- 172.21.15.16:0/3335954382 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fd6a4068380 0x7fd6a406a840 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:34.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:34.309+0000 7fd6ccc00640 1 --2- 172.21.15.16:0/3335954382 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fd6c80d0a60 0x7fd6c813edd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:34.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:34.309+0000 7fd6ccc00640 1 --2- 172.21.15.16:0/3335954382 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd6c80cf150 0x7fd6c813e890 unknown :-1 s=CLOSED pgs=997 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:34.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:34.309+0000 7fd6ccc00640 1 --2- 172.21.15.16:0/3335954382 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd6c806f920 0x7fd6c813d6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:34.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:34.309+0000 7fd6ccc00640 1 -- 172.21.15.16:0/3335954382 >> 172.21.15.16:0/3335954382 conn(0x7fd6c8067930 msgr2=0x7fd6c8141550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:09:34.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:34.309+0000 7fd6ccc00640 1 -- 172.21.15.16:0/3335954382 shutdown_connections 2024-03-31T15:09:34.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:34.309+0000 7fd6ccc00640 1 -- 172.21.15.16:0/3335954382 wait complete. 2024-03-31T15:09:34.332 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-31T15:09:34.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:34.485+0000 7f915f600640 1 Processor -- start 2024-03-31T15:09:34.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:34.485+0000 7f915f600640 1 -- start start 2024-03-31T15:09:34.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:34.485+0000 7f915f600640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f915806f920 0x7f915806fd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:34.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:34.485+0000 7f915f600640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f91580cc740 0x7f91580ccbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:34.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:34.485+0000 7f915f600640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9158162010 con 0x7f915806d490 2024-03-31T15:09:34.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:34.485+0000 7f915f600640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f9158162be0 con 0x7f915806f920 2024-03-31T15:09:34.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:34.485+0000 7f915f600640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f9158070240 con 0x7f91580cc740 2024-03-31T15:09:34.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:34.485+0000 7f915cc00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f915806f920 0x7f915806fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:34.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:34.485+0000 7f915d600640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f91580cc740 0x7f91580ccbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:34.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:34.485+0000 7f915cc00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f915806f920 0x7f915806fd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.16:33954/0 (socket says 172.21.15.16:33954) 2024-03-31T15:09:34.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:34.485+0000 7f915cc00640 1 -- 172.21.15.16:0/2879786666 learned_addr learned my addr 172.21.15.16:0/2879786666 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:09:34.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:34.485+0000 7f915cc00640 1 -- 172.21.15.16:0/2879786666 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f91580cc740 msgr2=0x7f91580ccbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:09:34.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:34.485+0000 7f915cc00640 1 --2- 172.21.15.16:0/2879786666 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f91580cc740 0x7f91580ccbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:34.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:34.485+0000 7f915cc00640 1 -- 172.21.15.16:0/2879786666 >> v1:172.21.15.89:6789/0 conn(0x7f915806d490 legacy=0x7f915806d8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:34.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:34.485+0000 7f915cc00640 1 -- 172.21.15.16:0/2879786666 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f91581622b0 con 0x7f915806f920 2024-03-31T15:09:34.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:34.485+0000 7f915d600640 1 --2- 172.21.15.16:0/2879786666 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f91580cc740 0x7f91580ccbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:09:34.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:34.489+0000 7f915cc00640 1 --2- 172.21.15.16:0/2879786666 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f915806f920 0x7f915806fd00 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f9158162d60 tx=0x7f914c01c540 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bd297d07e8fc0b89 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:09:34.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:34.489+0000 7f9157400640 1 -- 172.21.15.16:0/2879786666 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f914c0ab430 con 0x7f915806f920 2024-03-31T15:09:34.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:34.489+0000 7f9157400640 1 -- 172.21.15.16:0/2879786666 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f914c0b0070 con 0x7f915806f920 2024-03-31T15:09:34.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:34.489+0000 7f9157400640 1 -- 172.21.15.16:0/2879786666 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f914c0ab7f0 con 0x7f915806f920 2024-03-31T15:09:34.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:34.489+0000 7f915f600640 1 -- 172.21.15.16:0/2879786666 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f915806f920 msgr2=0x7f915806fd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:34.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:34.489+0000 7f915f600640 1 --2- 172.21.15.16:0/2879786666 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f915806f920 0x7f915806fd00 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f9158162d60 tx=0x7f914c01c540 comp rx=0 tx=0).stop 2024-03-31T15:09:34.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:34.489+0000 7f915f600640 1 -- 172.21.15.16:0/2879786666 shutdown_connections 2024-03-31T15:09:34.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:34.489+0000 7f915f600640 1 --2- 172.21.15.16:0/2879786666 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f91580cc740 0x7f91580ccbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:34.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:34.489+0000 7f915f600640 1 --2- 172.21.15.16:0/2879786666 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f915806f920 0x7f915806fd00 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:34.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:34.489+0000 7f915f600640 1 -- 172.21.15.16:0/2879786666 >> 172.21.15.16:0/2879786666 conn(0x7f9158067330 msgr2=0x7f9158067740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:09:34.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:34.489+0000 7f915f600640 1 -- 172.21.15.16:0/2879786666 shutdown_connections 2024-03-31T15:09:34.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:34.489+0000 7f915f600640 1 -- 172.21.15.16:0/2879786666 wait complete. 2024-03-31T15:09:34.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:34.493+0000 7f915f600640 1 Processor -- start 2024-03-31T15:09:34.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:34.493+0000 7f915f600640 1 -- start start 2024-03-31T15:09:34.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:34.493+0000 7f915f600640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f915806d490 0x7f915815eb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:34.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:34.493+0000 7f915f600640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f915806f920 0x7f9158145170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:34.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:34.493+0000 7f915f600640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f91580cc740 0x7f915813f460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:34.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:34.493+0000 7f915f600640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f915816b530 con 0x7f915806f920 2024-03-31T15:09:34.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:34.493+0000 7f915f600640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f915816b3b0 con 0x7f915806d490 2024-03-31T15:09:34.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:34.493+0000 7f915f600640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f915816b6b0 con 0x7f91580cc740 2024-03-31T15:09:34.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:34.493+0000 7f9157e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f915806f920 0x7f9158145170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:34.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:34.493+0000 7f915cc00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f915806d490 0x7f915815eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:34.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:34.493+0000 7f915cc00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f915806d490 0x7f915815eb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.16:33958/0 (socket says 172.21.15.16:33958) 2024-03-31T15:09:34.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:34.493+0000 7f915cc00640 1 -- 172.21.15.16:0/3277501115 learned_addr learned my addr 172.21.15.16:0/3277501115 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:09:34.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:34.493+0000 7f915d600640 1 --2- 172.21.15.16:0/3277501115 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f91580cc740 0x7f915813f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:34.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:34.497+0000 7f915cc00640 1 -- 172.21.15.16:0/3277501115 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f91580cc740 msgr2=0x7f915813f460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:34.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:34.497+0000 7f915cc00640 1 --2- 172.21.15.16:0/3277501115 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f91580cc740 0x7f915813f460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:34.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:34.497+0000 7f915cc00640 1 -- 172.21.15.16:0/3277501115 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f915806f920 msgr2=0x7f9158145170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:34.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:34.497+0000 7f915cc00640 1 --2- 172.21.15.16:0/3277501115 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f915806f920 0x7f9158145170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:34.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:34.497+0000 7f915cc00640 1 -- 172.21.15.16:0/3277501115 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f915815f080 con 0x7f915806d490 2024-03-31T15:09:34.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:34.497+0000 7f9157e00640 1 --2- 172.21.15.16:0/3277501115 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f915806f920 0x7f9158145170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:09:34.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:34.497+0000 7f915cc00640 1 --2- 172.21.15.16:0/3277501115 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f915806d490 0x7f915815eb40 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f914c0ad040 tx=0x7f914c0192e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:09:34.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:34.497+0000 7f9154c00640 1 -- 172.21.15.16:0/3277501115 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f914c0abe00 con 0x7f915806d490 2024-03-31T15:09:34.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:34.497+0000 7f9154c00640 1 -- 172.21.15.16:0/3277501115 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f914c0bd070 con 0x7f915806d490 2024-03-31T15:09:34.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:34.497+0000 7f9154c00640 1 -- 172.21.15.16:0/3277501115 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f914c0b4660 con 0x7f915806d490 2024-03-31T15:09:34.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:34.497+0000 7f915f600640 1 -- 172.21.15.16:0/3277501115 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9158144d00 con 0x7f915806d490 2024-03-31T15:09:34.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:34.497+0000 7f915f600640 1 -- 172.21.15.16:0/3277501115 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f915813ec60 con 0x7f915806d490 2024-03-31T15:09:34.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:34.497+0000 7f915f600640 1 -- 172.21.15.16:0/3277501115 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9158069950 con 0x7f915806d490 2024-03-31T15:09:34.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:34.501+0000 7f9154c00640 1 -- 172.21.15.16:0/3277501115 <== mon.1 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f914c0b4800 con 0x7f915806d490 2024-03-31T15:09:34.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:34.501+0000 7f9154c00640 1 --2- 172.21.15.16:0/3277501115 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f9138068380 0x7f913806a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:34.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:34.501+0000 7f9154c00640 1 -- 172.21.15.16:0/3277501115 <== mon.1 v2:172.21.15.89:3300/0 5 ==== osd_map(70..70 src has 1..70) ==== 5882+0+0 (secure 0 0 0) 0x7f914c120720 con 0x7f915806d490 2024-03-31T15:09:34.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:34.501+0000 7f9157e00640 1 --2- 172.21.15.16:0/3277501115 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f9138068380 0x7f913806a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:34.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:34.501+0000 7f9157e00640 1 --2- 172.21.15.16:0/3277501115 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f9138068380 0x7f913806a840 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f91581604b0 tx=0x7f9144009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:09:34.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:34.505+0000 7f9154c00640 1 -- 172.21.15.16:0/3277501115 <== mon.1 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f914c0c7020 con 0x7f915806d490 2024-03-31T15:09:34.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:34.681+0000 7f915f600640 1 -- 172.21.15.16:0/3277501115 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f91580cf3f0 con 0x7f915806d490 2024-03-31T15:09:34.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:34.681+0000 7f9154c00640 1 -- 172.21.15.16:0/3277501115 <== mon.1 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 2 entries v70) ==== 80+0+122 (secure 0 0 0) 0x7f914c016130 con 0x7f915806d490 2024-03-31T15:09:34.672 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:6835/2555930820 2024-04-01T15:08:10.537891+0000 2024-03-31T15:09:34.672 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:6834/2555930820 2024-04-01T15:08:10.537891+0000 2024-03-31T15:09:34.672 INFO:teuthology.orchestra.run.smithi016.stderr:listed 2 entries 2024-03-31T15:09:34.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:34.681+0000 7f915f600640 1 -- 172.21.15.16:0/3277501115 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f9138068380 msgr2=0x7f913806a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:34.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:34.681+0000 7f915f600640 1 --2- 172.21.15.16:0/3277501115 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f9138068380 0x7f913806a840 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f91581604b0 tx=0x7f9144009040 comp rx=0 tx=0).stop 2024-03-31T15:09:34.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:34.681+0000 7f915f600640 1 -- 172.21.15.16:0/3277501115 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f915806d490 msgr2=0x7f915815eb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:34.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:34.681+0000 7f915f600640 1 --2- 172.21.15.16:0/3277501115 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f915806d490 0x7f915815eb40 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f914c0ad040 tx=0x7f914c0192e0 comp rx=0 tx=0).stop 2024-03-31T15:09:34.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:34.681+0000 7f915f600640 1 -- 172.21.15.16:0/3277501115 shutdown_connections 2024-03-31T15:09:34.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:34.681+0000 7f915f600640 1 --2- 172.21.15.16:0/3277501115 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f9138068380 0x7f913806a840 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:34.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:34.681+0000 7f915f600640 1 --2- 172.21.15.16:0/3277501115 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f91580cc740 0x7f915813f460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:34.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:34.681+0000 7f915f600640 1 --2- 172.21.15.16:0/3277501115 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f915806f920 0x7f9158145170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:34.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:34.681+0000 7f915f600640 1 --2- 172.21.15.16:0/3277501115 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f915806d490 0x7f915815eb40 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:34.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:34.681+0000 7f915f600640 1 -- 172.21.15.16:0/3277501115 >> 172.21.15.16:0/3277501115 conn(0x7f9158067330 msgr2=0x7f91580d08b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:09:34.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:34.681+0000 7f915f600640 1 -- 172.21.15.16:0/3277501115 shutdown_connections 2024-03-31T15:09:34.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:34.681+0000 7f915f600640 1 -- 172.21.15.16:0/3277501115 wait complete. 2024-03-31T15:09:34.694 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi016.front.sepia.ceph.com... 2024-03-31T15:09:34.694 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:09:34.694 DEBUG:teuthology.orchestra.run.smithi016:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-03-31T15:09:34.712 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi016.stderr:ceph-fuse[43322]: fuse finished with error 0 and tester_r 0 2024-03-31T15:09:34.716 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi016.front.sepia.ceph.com 2024-03-31T15:09:34.716 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:09:34.716 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-31T15:09:34.879 INFO:tasks.cephfs.mount:Terminating background process 2024-03-31T15:09:34.879 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-03-31T15:09:34.879 DEBUG:teuthology.orchestra.run.smithi089:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-31T15:09:34.889 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16354452k,nr_inodes=4088613,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275344k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=901 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275340k,nr_inodes=818835,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275344k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-31T15:09:34.889 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi089.front.sepia.ceph.com 2024-03-31T15:09:34.889 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:09:34.889 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-03-31T15:09:34.941 INFO:teuthology.orchestra.run.smithi089.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.1': No such file or directory 2024-03-31T15:09:34.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-31T15:09:34.942 INFO:tasks.cephfs.mount:Terminating background process 2024-03-31T15:09:34.942 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-03-31T15:09:34.942 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-31T15:09:34.951 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16357708k,nr_inodes=4089427,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275996k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=6539 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275992k,nr_inodes=818998,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275996k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-31T15:09:34.951 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi016.front.sepia.ceph.com 2024-03-31T15:09:34.951 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:09:34.952 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-03-31T15:09:35.002 INFO:teuthology.orchestra.run.smithi016.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2024-03-31T15:09:35.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-31T15:09:35.003 INFO:tasks.cephfs.mount:Terminating background process 2024-03-31T15:09:35.003 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-03-31T15:09:35.003 DEBUG:teuthology.orchestra.run.smithi089:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-31T15:09:35.012 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16354452k,nr_inodes=4088613,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275344k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=901 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275340k,nr_inodes=818835,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275344k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-31T15:09:35.013 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi089.front.sepia.ceph.com 2024-03-31T15:09:35.013 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:09:35.013 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-03-31T15:09:35.061 INFO:teuthology.orchestra.run.smithi089.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2024-03-31T15:09:35.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-31T15:09:35.062 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-31T15:09:35.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:35.209+0000 7f9aa0200640 1 Processor -- start 2024-03-31T15:09:35.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:35.209+0000 7f9aa0200640 1 -- start start 2024-03-31T15:09:35.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:35.213+0000 7f9aa0200640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f9a9806f920 0x7f9a9806fd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:35.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:35.213+0000 7f9aa0200640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f9a9806d490 0x7f9a9806d8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:35.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:35.213+0000 7f9aa0200640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9a98162010 con 0x7f9a980cb440 2024-03-31T15:09:35.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:35.213+0000 7f9aa0200640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f9a98162be0 con 0x7f9a9806d490 2024-03-31T15:09:35.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:35.213+0000 7f9aa0200640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f9a98070240 con 0x7f9a9806f920 2024-03-31T15:09:35.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:35.213+0000 7f9a9e200640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f9a980cb440 0x7f9a980cb8a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:46738/0 (socket says 172.21.15.16:46738) 2024-03-31T15:09:35.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:35.213+0000 7f9a9e200640 1 -- 172.21.15.16:0/3221341308 learned_addr learned my addr 172.21.15.16:0/3221341308 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:09:35.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:35.213+0000 7f9a9d800640 1 --2- 172.21.15.16:0/3221341308 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f9a9806f920 0x7f9a9806fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:35.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:35.213+0000 7f9a9ce00640 1 --2- 172.21.15.16:0/3221341308 >> v2:172.21.15.89:3300/0 conn(0x7f9a9806d490 0x7f9a9806d8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:35.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:35.213+0000 7f9a97e00640 1 -- 172.21.15.16:0/3221341308 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4173706857 0 0) 0x7f9a98162010 con 0x7f9a980cb440 2024-03-31T15:09:35.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:35.213+0000 7f9a97e00640 1 -- 172.21.15.16:0/3221341308 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9a80003530 con 0x7f9a980cb440 2024-03-31T15:09:35.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:35.213+0000 7f9a9ce00640 1 -- 172.21.15.16:0/3221341308 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f9a9806f920 msgr2=0x7f9a9806fd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:09:35.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:35.213+0000 7f9a9ce00640 1 --2- 172.21.15.16:0/3221341308 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f9a9806f920 0x7f9a9806fd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:35.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:35.213+0000 7f9a9ce00640 1 -- 172.21.15.16:0/3221341308 >> v1:172.21.15.16:6789/0 conn(0x7f9a980cb440 legacy=0x7f9a980cb8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:35.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:35.213+0000 7f9a9ce00640 1 -- 172.21.15.16:0/3221341308 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a981622b0 con 0x7f9a9806d490 2024-03-31T15:09:35.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:35.213+0000 7f9a9d800640 1 --2- 172.21.15.16:0/3221341308 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f9a9806f920 0x7f9a9806fd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:09:35.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:35.217+0000 7f9a9ce00640 1 --2- 172.21.15.16:0/3221341308 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9a9806d490 0x7f9a9806d8f0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f9a98162d60 tx=0x7f9a8401c470 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ef54ddfc9c24281e server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:09:35.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:35.217+0000 7f9a97e00640 1 -- 172.21.15.16:0/3221341308 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a840b2070 con 0x7f9a9806d490 2024-03-31T15:09:35.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:35.217+0000 7f9a97e00640 1 -- 172.21.15.16:0/3221341308 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9a840ad680 con 0x7f9a9806d490 2024-03-31T15:09:35.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:35.217+0000 7f9a97e00640 1 -- 172.21.15.16:0/3221341308 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a840ada30 con 0x7f9a9806d490 2024-03-31T15:09:35.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:35.217+0000 7f9aa0200640 1 -- 172.21.15.16:0/3221341308 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9a9806d490 msgr2=0x7f9a9806d8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:35.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:35.217+0000 7f9aa0200640 1 --2- 172.21.15.16:0/3221341308 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9a9806d490 0x7f9a9806d8f0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f9a98162d60 tx=0x7f9a8401c470 comp rx=0 tx=0).stop 2024-03-31T15:09:35.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:35.217+0000 7f9aa0200640 1 -- 172.21.15.16:0/3221341308 shutdown_connections 2024-03-31T15:09:35.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:35.217+0000 7f9aa0200640 1 --2- 172.21.15.16:0/3221341308 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9a9806d490 0x7f9a9806d8f0 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:35.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:35.217+0000 7f9aa0200640 1 --2- 172.21.15.16:0/3221341308 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f9a9806f920 0x7f9a9806fd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:35.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:35.217+0000 7f9aa0200640 1 -- 172.21.15.16:0/3221341308 >> 172.21.15.16:0/3221341308 conn(0x7f9a98067330 msgr2=0x7f9a98067740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:09:35.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:35.217+0000 7f9aa0200640 1 -- 172.21.15.16:0/3221341308 shutdown_connections 2024-03-31T15:09:35.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:35.217+0000 7f9aa0200640 1 -- 172.21.15.16:0/3221341308 wait complete. 2024-03-31T15:09:35.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:35.217+0000 7f9aa0200640 1 Processor -- start 2024-03-31T15:09:35.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:35.217+0000 7f9aa0200640 1 -- start start 2024-03-31T15:09:35.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:35.217+0000 7f9aa0200640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9a9806d490 0x7f9a98159120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:35.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:35.217+0000 7f9aa0200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9a9806f920 0x7f9a98142780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:35.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:35.217+0000 7f9aa0200640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f9a980cb440 0x7f9a9815c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:35.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:35.217+0000 7f9aa0200640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f9a9806dfb0 con 0x7f9a9806f920 2024-03-31T15:09:35.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:35.217+0000 7f9aa0200640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f9a9806de30 con 0x7f9a9806d490 2024-03-31T15:09:35.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:35.217+0000 7f9aa0200640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f9a980cd750 con 0x7f9a980cb440 2024-03-31T15:09:35.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:35.221+0000 7f9a9ce00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9a9806f920 0x7f9a98142780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:35.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:35.221+0000 7f9a9ce00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9a9806f920 0x7f9a98142780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60794/0 (socket says 172.21.15.16:60794) 2024-03-31T15:09:35.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:35.221+0000 7f9a9ce00640 1 -- 172.21.15.16:0/1282971024 learned_addr learned my addr 172.21.15.16:0/1282971024 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:09:35.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:35.221+0000 7f9a9e200640 1 --2- 172.21.15.16:0/1282971024 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f9a980cb440 0x7f9a9815c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:35.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:35.221+0000 7f9a9d800640 1 --2- 172.21.15.16:0/1282971024 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9a9806d490 0x7f9a98159120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:35.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:35.221+0000 7f9a9ce00640 1 -- 172.21.15.16:0/1282971024 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f9a980cb440 msgr2=0x7f9a9815c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:35.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:35.221+0000 7f9a9ce00640 1 --2- 172.21.15.16:0/1282971024 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f9a980cb440 0x7f9a9815c9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:35.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:35.221+0000 7f9a9ce00640 1 -- 172.21.15.16:0/1282971024 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9a9806d490 msgr2=0x7f9a98159120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:35.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:35.221+0000 7f9a9ce00640 1 --2- 172.21.15.16:0/1282971024 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9a9806d490 0x7f9a98159120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:35.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:35.221+0000 7f9a9ce00640 1 -- 172.21.15.16:0/1282971024 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a98159960 con 0x7f9a9806f920 2024-03-31T15:09:35.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:35.221+0000 7f9a9ce00640 1 --2- 172.21.15.16:0/1282971024 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9a9806f920 0x7f9a98142780 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7f9a8401ab20 tx=0x7f9a84019210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:09:35.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:35.221+0000 7f9a95600640 1 -- 172.21.15.16:0/1282971024 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a840bd070 con 0x7f9a9806f920 2024-03-31T15:09:35.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:35.221+0000 7f9a95600640 1 -- 172.21.15.16:0/1282971024 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9a84004230 con 0x7f9a9806f920 2024-03-31T15:09:35.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:35.221+0000 7f9a95600640 1 -- 172.21.15.16:0/1282971024 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a840b2040 con 0x7f9a9806f920 2024-03-31T15:09:35.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:35.221+0000 7f9aa0200640 1 -- 172.21.15.16:0/1282971024 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9a98161380 con 0x7f9a9806f920 2024-03-31T15:09:35.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:35.221+0000 7f9aa0200640 1 -- 172.21.15.16:0/1282971024 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9a98159c00 con 0x7f9a9806f920 2024-03-31T15:09:35.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:35.221+0000 7f9aa0200640 1 -- 172.21.15.16:0/1282971024 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9a98069950 con 0x7f9a9806f920 2024-03-31T15:09:35.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:35.221+0000 7f9a95600640 1 -- 172.21.15.16:0/1282971024 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f9a840ad680 con 0x7f9a9806f920 2024-03-31T15:09:35.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:35.225+0000 7f9a95600640 1 --2- 172.21.15.16:0/1282971024 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f9a78068380 0x7f9a7806a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:35.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:35.225+0000 7f9a95600640 1 -- 172.21.15.16:0/1282971024 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(70..70 src has 1..70) ==== 5882+0+0 (secure 0 0 0) 0x7f9a8411fb60 con 0x7f9a9806f920 2024-03-31T15:09:35.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:35.225+0000 7f9a9d800640 1 --2- 172.21.15.16:0/1282971024 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f9a78068380 0x7f9a7806a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:35.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:35.225+0000 7f9a9d800640 1 --2- 172.21.15.16:0/1282971024 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f9a78068380 0x7f9a7806a840 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f9a9815edd0 tx=0x7f9a8c008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:09:35.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:35.229+0000 7f9a95600640 1 -- 172.21.15.16:0/1282971024 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f9a84016090 con 0x7f9a9806f920 2024-03-31T15:09:35.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:35.425+0000 7f9aa0200640 1 -- 172.21.15.16:0/1282971024 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9a981618a0 con 0x7f9a9806f920 2024-03-31T15:09:35.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:35.425+0000 7f9a95600640 1 -- 172.21.15.16:0/1282971024 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 52 v52) ==== 94+0+5185 (secure 0 0 0) 0x7f9a840ef980 con 0x7f9a9806f920 2024-03-31T15:09:35.418 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:09:35.418 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":52,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6836/4188793506","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":4188793506},{"type":"v1","addr":"172.21.15.89:6836","nonce":4188793506}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":40},{"gid":5942,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/4105657676","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6835","nonce":4105657676},{"type":"v1","addr":"172.21.15.89:6837","nonce":4105657676}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":40},{"gid":6592,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6833/3986752851","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":3986752851},{"type":"v1","addr":"172.21.15.16:6833","nonce":3986752851}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":39}],"filesystems":[{"mdsmap":{"epoch":52,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-31T15:07:56.938140+0000","modified":"2024-03-31T15:09:01.267292+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":68,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6829},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6829":{"gid":6829,"name":"a","rank":0,"incarnation":47,"state":"up:active","state_seq":16,"addr":"172.21.15.16:6835/2645389320","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":2645389320},{"type":"v1","addr":"172.21.15.16:6835","nonce":2645389320}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6829,"qdb_cluster":[6829]},"id":5}]} 2024-03-31T15:09:35.418 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 52 2024-03-31T15:09:35.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:35.429+0000 7f9aa0200640 1 -- 172.21.15.16:0/1282971024 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f9a78068380 msgr2=0x7f9a7806a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:35.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:35.429+0000 7f9aa0200640 1 --2- 172.21.15.16:0/1282971024 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f9a78068380 0x7f9a7806a840 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f9a9815edd0 tx=0x7f9a8c008040 comp rx=0 tx=0).stop 2024-03-31T15:09:35.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:35.429+0000 7f9aa0200640 1 -- 172.21.15.16:0/1282971024 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9a9806f920 msgr2=0x7f9a98142780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:35.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:35.429+0000 7f9aa0200640 1 --2- 172.21.15.16:0/1282971024 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9a9806f920 0x7f9a98142780 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7f9a8401ab20 tx=0x7f9a84019210 comp rx=0 tx=0).stop 2024-03-31T15:09:35.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:35.429+0000 7f9aa0200640 1 -- 172.21.15.16:0/1282971024 shutdown_connections 2024-03-31T15:09:35.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:35.429+0000 7f9aa0200640 1 --2- 172.21.15.16:0/1282971024 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f9a78068380 0x7f9a7806a840 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:35.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:35.429+0000 7f9aa0200640 1 --2- 172.21.15.16:0/1282971024 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f9a980cb440 0x7f9a9815c9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:35.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:35.429+0000 7f9aa0200640 1 --2- 172.21.15.16:0/1282971024 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9a9806f920 0x7f9a98142780 unknown :-1 s=CLOSED pgs=999 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:35.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:35.429+0000 7f9aa0200640 1 --2- 172.21.15.16:0/1282971024 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9a9806d490 0x7f9a98159120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:35.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:35.429+0000 7f9aa0200640 1 -- 172.21.15.16:0/1282971024 >> 172.21.15.16:0/1282971024 conn(0x7f9a98067330 msgr2=0x7f9a98067710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:09:35.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:35.429+0000 7f9aa0200640 1 -- 172.21.15.16:0/1282971024 shutdown_connections 2024-03-31T15:09:35.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:35.429+0000 7f9aa0200640 1 -- 172.21.15.16:0/1282971024 wait complete. 2024-03-31T15:09:35.445 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-31T15:09:35.586 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:35.593+0000 7fb231600640 1 Processor -- start 2024-03-31T15:09:35.587 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:35.593+0000 7fb231600640 1 -- start start 2024-03-31T15:09:35.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:35.597+0000 7fb231600640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb22c06dbc0 0x7fb22c15f650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:35.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:35.597+0000 7fb231600640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb22c070030 0x7fb22c162180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:35.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:35.597+0000 7fb231600640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb22c0cbe90 con 0x7fb22c0cb440 2024-03-31T15:09:35.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:35.597+0000 7fb231600640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7fb22c15fb90 con 0x7fb22c070030 2024-03-31T15:09:35.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:35.597+0000 7fb231600640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fb22c15f1c0 con 0x7fb22c06dbc0 2024-03-31T15:09:35.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:35.597+0000 7fb22b400640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb22c070030 0x7fb22c162180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:35.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:35.597+0000 7fb22b400640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb22c070030 0x7fb22c162180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60800/0 (socket says 172.21.15.16:60800) 2024-03-31T15:09:35.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:35.597+0000 7fb22b400640 1 -- 172.21.15.16:0/1260373085 learned_addr learned my addr 172.21.15.16:0/1260373085 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:09:35.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:35.597+0000 7fb22aa00640 1 --2- 172.21.15.16:0/1260373085 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb22c06dbc0 0x7fb22c15f650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:35.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:35.597+0000 7fb22b400640 1 -- 172.21.15.16:0/1260373085 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb22c06dbc0 msgr2=0x7fb22c15f650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:09:35.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:35.597+0000 7fb22b400640 1 --2- 172.21.15.16:0/1260373085 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb22c06dbc0 0x7fb22c15f650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:35.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:35.597+0000 7fb22b400640 1 -- 172.21.15.16:0/1260373085 >> v1:172.21.15.89:6789/0 conn(0x7fb22c0cb440 legacy=0x7fb22c06f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:35.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:35.597+0000 7fb22b400640 1 -- 172.21.15.16:0/1260373085 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb22c0cc170 con 0x7fb22c070030 2024-03-31T15:09:35.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:35.601+0000 7fb22b400640 1 --2- 172.21.15.16:0/1260373085 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb22c070030 0x7fb22c162180 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7fb224009870 tx=0x7fb22401c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3861b8b8cdc6cd9c server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:09:35.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:35.601+0000 7fb22a000640 1 -- 172.21.15.16:0/1260373085 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2240b1070 con 0x7fb22c070030 2024-03-31T15:09:35.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:35.601+0000 7fb22a000640 1 -- 172.21.15.16:0/1260373085 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb2240ac410 con 0x7fb22c070030 2024-03-31T15:09:35.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:35.601+0000 7fb22a000640 1 -- 172.21.15.16:0/1260373085 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2240ac7e0 con 0x7fb22c070030 2024-03-31T15:09:35.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:35.601+0000 7fb231600640 1 -- 172.21.15.16:0/1260373085 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb22c070030 msgr2=0x7fb22c162180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:35.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:35.601+0000 7fb231600640 1 --2- 172.21.15.16:0/1260373085 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb22c070030 0x7fb22c162180 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7fb224009870 tx=0x7fb22401c4a0 comp rx=0 tx=0).stop 2024-03-31T15:09:35.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:35.601+0000 7fb231600640 1 -- 172.21.15.16:0/1260373085 shutdown_connections 2024-03-31T15:09:35.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:35.601+0000 7fb231600640 1 --2- 172.21.15.16:0/1260373085 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb22c070030 0x7fb22c162180 unknown :-1 s=CLOSED pgs=1000 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:35.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:35.601+0000 7fb231600640 1 --2- 172.21.15.16:0/1260373085 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb22c06dbc0 0x7fb22c15f650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:35.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:35.601+0000 7fb231600640 1 -- 172.21.15.16:0/1260373085 >> 172.21.15.16:0/1260373085 conn(0x7fb22c067330 msgr2=0x7fb22c067740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:09:35.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:35.601+0000 7fb231600640 1 -- 172.21.15.16:0/1260373085 shutdown_connections 2024-03-31T15:09:35.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:35.601+0000 7fb231600640 1 -- 172.21.15.16:0/1260373085 wait complete. 2024-03-31T15:09:35.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:35.601+0000 7fb231600640 1 Processor -- start 2024-03-31T15:09:35.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:35.601+0000 7fb231600640 1 -- start start 2024-03-31T15:09:35.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:35.601+0000 7fb231600640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb22c06dbc0 0x7fb22c14cc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:35.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:35.601+0000 7fb231600640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb22c070030 0x7fb22c150980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:35.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:35.601+0000 7fb231600640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb22c0cb440 0x7fb22c0b70b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:35.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:35.601+0000 7fb231600640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fb22c06cad0 con 0x7fb22c06dbc0 2024-03-31T15:09:35.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:35.601+0000 7fb231600640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fb22c06c950 con 0x7fb22c0cb440 2024-03-31T15:09:35.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:35.601+0000 7fb231600640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fb22c16b520 con 0x7fb22c070030 2024-03-31T15:09:35.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:35.605+0000 7fb22aa00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb22c06dbc0 0x7fb22c14cc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:35.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:35.605+0000 7fb22aa00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb22c06dbc0 0x7fb22c14cc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60816/0 (socket says 172.21.15.16:60816) 2024-03-31T15:09:35.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:35.605+0000 7fb22aa00640 1 -- 172.21.15.16:0/875098970 learned_addr learned my addr 172.21.15.16:0/875098970 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:09:35.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:35.605+0000 7fb222000640 1 --2- 172.21.15.16:0/875098970 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb22c070030 0x7fb22c150980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:35.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:35.605+0000 7fb22b400640 1 --2- 172.21.15.16:0/875098970 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb22c0cb440 0x7fb22c0b70b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:35.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:35.605+0000 7fb22aa00640 1 -- 172.21.15.16:0/875098970 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb22c070030 msgr2=0x7fb22c150980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:35.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:35.605+0000 7fb22aa00640 1 --2- 172.21.15.16:0/875098970 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb22c070030 0x7fb22c150980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:35.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:35.605+0000 7fb22aa00640 1 -- 172.21.15.16:0/875098970 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb22c0cb440 msgr2=0x7fb22c0b70b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:35.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:35.605+0000 7fb22aa00640 1 --2- 172.21.15.16:0/875098970 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb22c0cb440 0x7fb22c0b70b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:35.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:35.605+0000 7fb22aa00640 1 -- 172.21.15.16:0/875098970 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb22c159d70 con 0x7fb22c06dbc0 2024-03-31T15:09:35.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:35.605+0000 7fb22aa00640 1 --2- 172.21.15.16:0/875098970 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb22c06dbc0 0x7fb22c14cc40 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 crypto rx=0x7fb21c00d5f0 tx=0x7fb21c00dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:09:35.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:35.605+0000 7fb223400640 1 -- 172.21.15.16:0/875098970 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb21c0125f0 con 0x7fb22c06dbc0 2024-03-31T15:09:35.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:35.605+0000 7fb223400640 1 -- 172.21.15.16:0/875098970 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb21c019070 con 0x7fb22c06dbc0 2024-03-31T15:09:35.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:35.605+0000 7fb223400640 1 -- 172.21.15.16:0/875098970 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb21c0129b0 con 0x7fb22c06dbc0 2024-03-31T15:09:35.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:35.605+0000 7fb231600640 1 -- 172.21.15.16:0/875098970 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb22c15a050 con 0x7fb22c06dbc0 2024-03-31T15:09:35.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:35.605+0000 7fb231600640 1 -- 172.21.15.16:0/875098970 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb22c15a610 con 0x7fb22c06dbc0 2024-03-31T15:09:35.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:35.605+0000 7fb231600640 1 -- 172.21.15.16:0/875098970 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb22c069950 con 0x7fb22c06dbc0 2024-03-31T15:09:35.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:35.605+0000 7fb223400640 1 -- 172.21.15.16:0/875098970 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fb21c012b50 con 0x7fb22c06dbc0 2024-03-31T15:09:35.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:35.609+0000 7fb223400640 1 --2- 172.21.15.16:0/875098970 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fb208068380 0x7fb20806a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:35.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:35.609+0000 7fb223400640 1 -- 172.21.15.16:0/875098970 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(70..70 src has 1..70) ==== 5882+0+0 (secure 0 0 0) 0x7fb21c087830 con 0x7fb22c06dbc0 2024-03-31T15:09:35.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:35.609+0000 7fb222000640 1 --2- 172.21.15.16:0/875098970 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fb208068380 0x7fb20806a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:35.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:35.609+0000 7fb222000640 1 --2- 172.21.15.16:0/875098970 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fb208068380 0x7fb20806a840 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7fb22c1623a0 tx=0x7fb214009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:09:35.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:35.609+0000 7fb223400640 1 -- 172.21.15.16:0/875098970 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fb21c057760 con 0x7fb22c06dbc0 2024-03-31T15:09:35.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:35.777+0000 7fb231600640 1 -- 172.21.15.16:0/875098970 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fb22c0cf380 con 0x7fb22c06dbc0 2024-03-31T15:09:35.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:35.777+0000 7fb223400640 1 -- 172.21.15.16:0/875098970 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v70) ==== 74+0+16870 (secure 0 0 0) 0x7fb21c05ba60 con 0x7fb22c06dbc0 2024-03-31T15:09:35.769 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:09:35.769 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":70,"fsid":"286eb5da-fb47-4550-b290-d9519c13b9a5","created":"2024-03-31T15:03:11.438952+0000","modified":"2024-03-31T15:08:18.287907+0000","last_up_change":"2024-03-31T15:03:22.885535+0000","last_in_change":"2024-03-31T15:03:13.941667+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-31T15:03:22.550787+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-03-31T15:07:55.329096+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"66","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-03-31T15:07:55.885919+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"67","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"92a66450-ef19-40d8-ac3d-d760b23b0bda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6801","nonce":932712244}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6803","nonce":932712244}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6807","nonce":932712244}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6805","nonce":932712244}]},"public_addr":"172.21.15.16:6801/932712244","cluster_addr":"172.21.15.16:6803/932712244","heartbeat_back_addr":"172.21.15.16:6807/932712244","heartbeat_front_addr":"172.21.15.16:6805/932712244","state":["exists","up"]},{"osd":1,"uuid":"da9d1f2b-1968-4959-b843-faa231988e92","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6810","nonce":1398737533}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6811","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6813","nonce":1398737533}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6821","nonce":1398737533}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6815","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6816","nonce":1398737533}]},"public_addr":"172.21.15.16:6810/1398737533","cluster_addr":"172.21.15.16:6813/1398737533","heartbeat_back_addr":"172.21.15.16:6821/1398737533","heartbeat_front_addr":"172.21.15.16:6816/1398737533","state":["exists","up"]},{"osd":2,"uuid":"01a73662-9a21-4b1f-bcbb-42b9636a4638","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6819","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6822","nonce":3018526646}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6825","nonce":3018526646}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6831","nonce":3018526646}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6827","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6829","nonce":3018526646}]},"public_addr":"172.21.15.16:6822/3018526646","cluster_addr":"172.21.15.16:6825/3018526646","heartbeat_back_addr":"172.21.15.16:6831/3018526646","heartbeat_front_addr":"172.21.15.16:6829/3018526646","state":["exists","up"]},{"osd":3,"uuid":"c7771fd3-e18e-4500-ac38-23cf10ebcd75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6809","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6812","nonce":3360352974}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6817","nonce":3360352974}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6828","nonce":3360352974}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6823","nonce":3360352974}]},"public_addr":"172.21.15.16:6812/3360352974","cluster_addr":"172.21.15.16:6817/3360352974","heartbeat_back_addr":"172.21.15.16:6828/3360352974","heartbeat_front_addr":"172.21.15.16:6823/3360352974","state":["exists","up"]},{"osd":4,"uuid":"88260730-1482-4fbd-a40f-cb1124e9df4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6801","nonce":1856844856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6803","nonce":1856844856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6807","nonce":1856844856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6805","nonce":1856844856}]},"public_addr":"172.21.15.89:6801/1856844856","cluster_addr":"172.21.15.89:6803/1856844856","heartbeat_back_addr":"172.21.15.89:6807/1856844856","heartbeat_front_addr":"172.21.15.89:6805/1856844856","state":["exists","up"]},{"osd":5,"uuid":"4cb2c00c-fae0-4dd8-9d27-ec4f0093ccf7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6814","nonce":50395527}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6817","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6820","nonce":50395527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6831","nonce":50395527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6823","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6826","nonce":50395527}]},"public_addr":"172.21.15.89:6814/50395527","cluster_addr":"172.21.15.89:6820/50395527","heartbeat_back_addr":"172.21.15.89:6831/50395527","heartbeat_front_addr":"172.21.15.89:6826/50395527","state":["exists","up"]},{"osd":6,"uuid":"81ce0c19-2554-4cae-acbe-66a05e9e9b9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6809","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6812","nonce":3470471752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6815","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6818","nonce":3470471752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6827","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6829","nonce":3470471752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6821","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6824","nonce":3470471752}]},"public_addr":"172.21.15.89:6812/3470471752","cluster_addr":"172.21.15.89:6818/3470471752","heartbeat_back_addr":"172.21.15.89:6829/3470471752","heartbeat_front_addr":"172.21.15.89:6824/3470471752","state":["exists","up"]},{"osd":7,"uuid":"85e4313b-d37d-442e-8042-153bf20a847d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6811","nonce":1796778630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6813","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6816","nonce":1796778630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6825","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6828","nonce":1796778630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6819","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6822","nonce":1796778630}]},"public_addr":"172.21.15.89:6811/1796778630","cluster_addr":"172.21.15.89:6816/1796778630","heartbeat_back_addr":"172.21.15.89:6828/1796778630","heartbeat_front_addr":"172.21.15.89:6822/1796778630","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.439088+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.529970+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.680512+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:21.044161+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.582111+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.536558+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.409522+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.590229+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"11.4","mappings":[{"from":6,"to":0}]},{"pgid":"11.10","mappings":[{"from":6,"to":1}]},{"pgid":"11.3e","mappings":[{"from":6,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:6835/2555930820":"2024-04-01T15:08:10.537891+0000","172.21.15.16:6834/2555930820":"2024-04-01T15:08:10.537891+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-31T15:09:35.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:35.777+0000 7fb231600640 1 -- 172.21.15.16:0/875098970 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fb208068380 msgr2=0x7fb20806a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:35.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:35.777+0000 7fb231600640 1 --2- 172.21.15.16:0/875098970 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fb208068380 0x7fb20806a840 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7fb22c1623a0 tx=0x7fb214009040 comp rx=0 tx=0).stop 2024-03-31T15:09:35.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:35.777+0000 7fb231600640 1 -- 172.21.15.16:0/875098970 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb22c06dbc0 msgr2=0x7fb22c14cc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:35.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:35.777+0000 7fb231600640 1 --2- 172.21.15.16:0/875098970 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb22c06dbc0 0x7fb22c14cc40 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 crypto rx=0x7fb21c00d5f0 tx=0x7fb21c00dab0 comp rx=0 tx=0).stop 2024-03-31T15:09:35.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:35.777+0000 7fb231600640 1 -- 172.21.15.16:0/875098970 shutdown_connections 2024-03-31T15:09:35.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:35.777+0000 7fb231600640 1 --2- 172.21.15.16:0/875098970 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fb208068380 0x7fb20806a840 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:35.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:35.777+0000 7fb231600640 1 --2- 172.21.15.16:0/875098970 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb22c0cb440 0x7fb22c0b70b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:35.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:35.777+0000 7fb231600640 1 --2- 172.21.15.16:0/875098970 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb22c070030 0x7fb22c150980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:35.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:35.777+0000 7fb231600640 1 --2- 172.21.15.16:0/875098970 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb22c06dbc0 0x7fb22c14cc40 unknown :-1 s=CLOSED pgs=1001 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:35.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:35.777+0000 7fb231600640 1 -- 172.21.15.16:0/875098970 >> 172.21.15.16:0/875098970 conn(0x7fb22c067330 msgr2=0x7fb22c161650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:09:35.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:35.777+0000 7fb231600640 1 -- 172.21.15.16:0/875098970 shutdown_connections 2024-03-31T15:09:35.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:35.777+0000 7fb231600640 1 -- 172.21.15.16:0/875098970 wait complete. 2024-03-31T15:09:35.791 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-31T15:03:22.550787+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 10, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-31T15:07:55.329096+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '66', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.5, 'score_stable': 2.5, 'optimal_score': 1, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 11, 'pool_name': 'cephfs_data', 'create_time': '2024-03-31T15:07:55.885919+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '67', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-31T15:09:35.791 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-31T15:09:35.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:35.945+0000 7f8fdb200640 1 Processor -- start 2024-03-31T15:09:35.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:35.945+0000 7f8fdb200640 1 -- start start 2024-03-31T15:09:35.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:35.945+0000 7f8fdb200640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f8fd406f920 0x7f8fd406fd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:35.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:35.945+0000 7f8fdb200640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f8fd40d0a60 0x7f8fd40cea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:35.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:35.945+0000 7f8fdb200640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f8fd40cef80 0x7f8fd4070e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:35.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:35.945+0000 7f8fdb200640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f8fd413f670 con 0x7f8fd40cef80 2024-03-31T15:09:35.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:35.945+0000 7f8fdb200640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f8fd4070240 con 0x7f8fd406f920 2024-03-31T15:09:35.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:35.945+0000 7f8fdb200640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f8fd415ca70 con 0x7f8fd40d0a60 2024-03-31T15:09:35.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:35.945+0000 7f8fd9200640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f8fd40cef80 0x7f8fd4070e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:35.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:35.945+0000 7f8fd9200640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f8fd40cef80 0x7f8fd4070e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60826/0 (socket says 172.21.15.16:60826) 2024-03-31T15:09:35.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:35.945+0000 7f8fd9200640 1 -- 172.21.15.16:0/575589715 learned_addr learned my addr 172.21.15.16:0/575589715 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:09:35.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:35.945+0000 7f8fd3400640 1 --2- 172.21.15.16:0/575589715 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f8fd40d0a60 0x7f8fd40cea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:35.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:35.945+0000 7f8fd3e00640 1 --2- 172.21.15.16:0/575589715 >> v2:172.21.15.89:3300/0 conn(0x7f8fd406f920 0x7f8fd406fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:35.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:35.945+0000 7f8fd9200640 1 -- 172.21.15.16:0/575589715 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f8fd40d0a60 msgr2=0x7f8fd40cea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:09:35.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:35.945+0000 7f8fd9200640 1 --2- 172.21.15.16:0/575589715 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f8fd40d0a60 0x7f8fd40cea40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:35.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:35.945+0000 7f8fd9200640 1 -- 172.21.15.16:0/575589715 >> v2:172.21.15.89:3300/0 conn(0x7f8fd406f920 msgr2=0x7f8fd406fd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:09:35.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:35.945+0000 7f8fd9200640 1 --2- 172.21.15.16:0/575589715 >> v2:172.21.15.89:3300/0 conn(0x7f8fd406f920 0x7f8fd406fd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:35.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:35.945+0000 7f8fd9200640 1 -- 172.21.15.16:0/575589715 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8fd415cd50 con 0x7f8fd40cef80 2024-03-31T15:09:35.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:35.949+0000 7f8fd9200640 1 --2- 172.21.15.16:0/575589715 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8fd40cef80 0x7f8fd4070e60 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7f8fd40708e0 tx=0x7f8fcc01c730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c7b82624f6e739e server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:09:35.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:35.949+0000 7f8fd2a00640 1 -- 172.21.15.16:0/575589715 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fcc0ac070 con 0x7f8fd40cef80 2024-03-31T15:09:35.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:35.949+0000 7f8fd2a00640 1 -- 172.21.15.16:0/575589715 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8fcc0a8420 con 0x7f8fd40cef80 2024-03-31T15:09:35.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:35.949+0000 7f8fd2a00640 1 -- 172.21.15.16:0/575589715 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fcc0a87f0 con 0x7f8fd40cef80 2024-03-31T15:09:35.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:35.949+0000 7f8fdb200640 1 -- 172.21.15.16:0/575589715 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8fd40cef80 msgr2=0x7f8fd4070e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:35.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:35.949+0000 7f8fdb200640 1 --2- 172.21.15.16:0/575589715 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8fd40cef80 0x7f8fd4070e60 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7f8fd40708e0 tx=0x7f8fcc01c730 comp rx=0 tx=0).stop 2024-03-31T15:09:35.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:35.949+0000 7f8fdb200640 1 -- 172.21.15.16:0/575589715 shutdown_connections 2024-03-31T15:09:35.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:35.949+0000 7f8fdb200640 1 --2- 172.21.15.16:0/575589715 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8fd40cef80 0x7f8fd4070e60 unknown :-1 s=CLOSED pgs=1002 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:35.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:35.949+0000 7f8fdb200640 1 --2- 172.21.15.16:0/575589715 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f8fd40d0a60 0x7f8fd40cea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:35.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:35.949+0000 7f8fdb200640 1 --2- 172.21.15.16:0/575589715 >> v2:172.21.15.89:3300/0 conn(0x7f8fd406f920 0x7f8fd406fd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:35.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:35.949+0000 7f8fdb200640 1 -- 172.21.15.16:0/575589715 >> 172.21.15.16:0/575589715 conn(0x7f8fd4067930 msgr2=0x7f8fd40b3ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:09:35.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:35.949+0000 7f8fdb200640 1 -- 172.21.15.16:0/575589715 shutdown_connections 2024-03-31T15:09:35.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:35.949+0000 7f8fdb200640 1 -- 172.21.15.16:0/575589715 wait complete. 2024-03-31T15:09:35.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:35.953+0000 7f8fdb200640 1 Processor -- start 2024-03-31T15:09:35.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:35.953+0000 7f8fdb200640 1 -- start start 2024-03-31T15:09:35.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:35.953+0000 7f8fdb200640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f8fd406f920 0x7f8fd413ca90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:35.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:35.953+0000 7f8fdb200640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8fd40cef80 0x7f8fd413f830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:35.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:35.953+0000 7f8fdb200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8fd40d0a60 0x7f8fd415ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:35.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:35.953+0000 7f8fdb200640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f8fd41419c0 con 0x7f8fd40d0a60 2024-03-31T15:09:35.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:35.953+0000 7f8fdb200640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f8fd415c720 con 0x7f8fd40cef80 2024-03-31T15:09:35.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:35.953+0000 7f8fdb200640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f8fd4141b40 con 0x7f8fd406f920 2024-03-31T15:09:35.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:35.953+0000 7f8fd9200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8fd40d0a60 0x7f8fd415ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:35.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:35.953+0000 7f8fd9200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8fd40d0a60 0x7f8fd415ecd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60840/0 (socket says 172.21.15.16:60840) 2024-03-31T15:09:35.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:35.953+0000 7f8fd9200640 1 -- 172.21.15.16:0/3266516810 learned_addr learned my addr 172.21.15.16:0/3266516810 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:09:35.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:35.953+0000 7f8fd3400640 1 --2- 172.21.15.16:0/3266516810 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8fd40cef80 0x7f8fd413f830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:35.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:35.953+0000 7f8fd3e00640 1 --2- 172.21.15.16:0/3266516810 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f8fd406f920 0x7f8fd413ca90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:35.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:35.953+0000 7f8fd9200640 1 -- 172.21.15.16:0/3266516810 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f8fd406f920 msgr2=0x7f8fd413ca90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:35.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:35.953+0000 7f8fd9200640 1 --2- 172.21.15.16:0/3266516810 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f8fd406f920 0x7f8fd413ca90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:35.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:35.953+0000 7f8fd9200640 1 -- 172.21.15.16:0/3266516810 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8fd40cef80 msgr2=0x7f8fd413f830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:35.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:35.953+0000 7f8fd9200640 1 --2- 172.21.15.16:0/3266516810 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8fd40cef80 0x7f8fd413f830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:35.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:35.953+0000 7f8fd9200640 1 -- 172.21.15.16:0/3266516810 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8fd415e580 con 0x7f8fd40d0a60 2024-03-31T15:09:35.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:35.953+0000 7f8fd3400640 1 --2- 172.21.15.16:0/3266516810 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8fd40cef80 0x7f8fd413f830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:09:35.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:35.953+0000 7f8fd9200640 1 --2- 172.21.15.16:0/3266516810 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8fd40d0a60 0x7f8fd415ecd0 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 crypto rx=0x7f8fcc01ade0 tx=0x7f8fcc0194d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:09:35.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:35.957+0000 7f8fcbe00640 1 -- 172.21.15.16:0/3266516810 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fcc0ac040 con 0x7f8fd40d0a60 2024-03-31T15:09:35.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:35.957+0000 7f8fcbe00640 1 -- 172.21.15.16:0/3266516810 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8fcc0bd450 con 0x7f8fd40d0a60 2024-03-31T15:09:35.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:35.957+0000 7f8fdb200640 1 -- 172.21.15.16:0/3266516810 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8fd415e860 con 0x7f8fd40d0a60 2024-03-31T15:09:35.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:35.957+0000 7f8fcbe00640 1 -- 172.21.15.16:0/3266516810 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fcc009510 con 0x7f8fd40d0a60 2024-03-31T15:09:35.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:35.957+0000 7f8fdb200640 1 -- 172.21.15.16:0/3266516810 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8fd413cfd0 con 0x7f8fd40d0a60 2024-03-31T15:09:35.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:35.957+0000 7f8fdb200640 1 -- 172.21.15.16:0/3266516810 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8fd4069950 con 0x7f8fd40d0a60 2024-03-31T15:09:35.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:35.957+0000 7f8fcbe00640 1 -- 172.21.15.16:0/3266516810 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f8fcc0af070 con 0x7f8fd40d0a60 2024-03-31T15:09:35.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:35.957+0000 7f8fcbe00640 1 --2- 172.21.15.16:0/3266516810 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f8fb4068380 0x7f8fb406a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:35.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:35.957+0000 7f8fcbe00640 1 -- 172.21.15.16:0/3266516810 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(70..70 src has 1..70) ==== 5882+0+0 (secure 0 0 0) 0x7f8fcc121260 con 0x7f8fd40d0a60 2024-03-31T15:09:35.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:35.961+0000 7f8fd3e00640 1 --2- 172.21.15.16:0/3266516810 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f8fb4068380 0x7f8fb406a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:35.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:35.961+0000 7f8fd3e00640 1 --2- 172.21.15.16:0/3266516810 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f8fb4068380 0x7f8fb406a840 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f8fd4158070 tx=0x7f8fc4008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:09:35.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:35.961+0000 7f8fcbe00640 1 -- 172.21.15.16:0/3266516810 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f8fcc0165e0 con 0x7f8fd40d0a60 2024-03-31T15:09:36.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:36.129+0000 7f8fdb200640 1 -- 172.21.15.16:0/3266516810 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f8fd40b50e0 con 0x7f8fd40d0a60 2024-03-31T15:09:36.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:36.133+0000 7f8fcbe00640 1 -- 172.21.15.16:0/3266516810 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v70) ==== 93+0+10 (secure 0 0 0) 0x7f8fcc0f1760 con 0x7f8fd40d0a60 2024-03-31T15:09:36.123 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2024-03-31T15:09:36.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:36.133+0000 7f8fdb200640 1 -- 172.21.15.16:0/3266516810 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f8fb4068380 msgr2=0x7f8fb406a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:36.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:36.133+0000 7f8fdb200640 1 --2- 172.21.15.16:0/3266516810 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f8fb4068380 0x7f8fb406a840 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f8fd4158070 tx=0x7f8fc4008040 comp rx=0 tx=0).stop 2024-03-31T15:09:36.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:36.133+0000 7f8fdb200640 1 -- 172.21.15.16:0/3266516810 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8fd40d0a60 msgr2=0x7f8fd415ecd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:36.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:36.133+0000 7f8fdb200640 1 --2- 172.21.15.16:0/3266516810 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8fd40d0a60 0x7f8fd415ecd0 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 crypto rx=0x7f8fcc01ade0 tx=0x7f8fcc0194d0 comp rx=0 tx=0).stop 2024-03-31T15:09:36.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:36.133+0000 7f8fdb200640 1 -- 172.21.15.16:0/3266516810 shutdown_connections 2024-03-31T15:09:36.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:36.133+0000 7f8fdb200640 1 --2- 172.21.15.16:0/3266516810 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f8fb4068380 0x7f8fb406a840 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:36.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:36.133+0000 7f8fdb200640 1 --2- 172.21.15.16:0/3266516810 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8fd40d0a60 0x7f8fd415ecd0 unknown :-1 s=CLOSED pgs=1003 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:36.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:36.133+0000 7f8fdb200640 1 --2- 172.21.15.16:0/3266516810 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8fd40cef80 0x7f8fd413f830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:36.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:36.133+0000 7f8fdb200640 1 --2- 172.21.15.16:0/3266516810 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f8fd406f920 0x7f8fd413ca90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:36.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:36.133+0000 7f8fdb200640 1 -- 172.21.15.16:0/3266516810 >> 172.21.15.16:0/3266516810 conn(0x7f8fd4067930 msgr2=0x7f8fd40d15a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:09:36.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:36.133+0000 7f8fdb200640 1 -- 172.21.15.16:0/3266516810 shutdown_connections 2024-03-31T15:09:36.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:36.133+0000 7f8fdb200640 1 -- 172.21.15.16:0/3266516810 wait complete. 2024-03-31T15:09:36.145 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-31T15:09:36.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:36.297+0000 7f694ea00640 1 Processor -- start 2024-03-31T15:09:36.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:36.297+0000 7f694ea00640 1 -- start start 2024-03-31T15:09:36.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:36.297+0000 7f694ea00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f694806ba80 0x7f69480ce6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:36.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:36.297+0000 7f694ea00640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6948160230 con 0x7f6948145190 2024-03-31T15:09:36.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:36.297+0000 7f694ea00640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6948160510 con 0x7f694806b620 2024-03-31T15:09:36.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:36.297+0000 7f694ea00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f6948071060 con 0x7f694806ba80 2024-03-31T15:09:36.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:36.297+0000 7f6947e00640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f6948145190 0x7f694806f160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:46744/0 (socket says 172.21.15.16:46744) 2024-03-31T15:09:36.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:36.297+0000 7f6947e00640 1 -- 172.21.15.16:0/2596326272 learned_addr learned my addr 172.21.15.16:0/2596326272 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:09:36.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:36.297+0000 7f694ca00640 1 --2- 172.21.15.16:0/2596326272 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f694806ba80 0x7f69480ce6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:36.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:36.297+0000 7f6947400640 1 -- 172.21.15.16:0/2596326272 <== mon.1 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 217005685 0 0) 0x7f6948160510 con 0x7f694806b620 2024-03-31T15:09:36.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:36.297+0000 7f6947400640 1 -- 172.21.15.16:0/2596326272 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f692c003530 con 0x7f694806b620 2024-03-31T15:09:36.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:36.297+0000 7f6947400640 1 -- 172.21.15.16:0/2596326272 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3446061890 0 0) 0x7f6948160230 con 0x7f6948145190 2024-03-31T15:09:36.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:36.297+0000 7f694ca00640 1 -- 172.21.15.16:0/2596326272 >> v1:172.21.15.89:6789/0 conn(0x7f694806b620 legacy=0x7f69480cdfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:36.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:36.297+0000 7f694ca00640 1 -- 172.21.15.16:0/2596326272 >> v1:172.21.15.16:6789/0 conn(0x7f6948145190 legacy=0x7f694806f160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:36.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:36.297+0000 7f694ca00640 1 -- 172.21.15.16:0/2596326272 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69481607f0 con 0x7f694806ba80 2024-03-31T15:09:36.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:36.301+0000 7f694ca00640 1 --2- 172.21.15.16:0/2596326272 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f694806ba80 0x7f69480ce6b0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f6940009dc0 tx=0x7f694001c870 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a46a983292b48bf0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:09:36.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:36.301+0000 7f6947400640 1 -- 172.21.15.16:0/2596326272 <== mon.2 v2:172.21.15.89:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69400b2070 con 0x7f694806ba80 2024-03-31T15:09:36.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:36.301+0000 7f6947400640 1 -- 172.21.15.16:0/2596326272 <== mon.2 v2:172.21.15.89:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f69400a5730 con 0x7f694806ba80 2024-03-31T15:09:36.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:36.301+0000 7f6947400640 1 -- 172.21.15.16:0/2596326272 <== mon.2 v2:172.21.15.89:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69400a5b00 con 0x7f694806ba80 2024-03-31T15:09:36.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:36.301+0000 7f694ea00640 1 -- 172.21.15.16:0/2596326272 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f694806ba80 msgr2=0x7f69480ce6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:36.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:36.301+0000 7f694ea00640 1 --2- 172.21.15.16:0/2596326272 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f694806ba80 0x7f69480ce6b0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f6940009dc0 tx=0x7f694001c870 comp rx=0 tx=0).stop 2024-03-31T15:09:36.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:36.301+0000 7f694ea00640 1 -- 172.21.15.16:0/2596326272 shutdown_connections 2024-03-31T15:09:36.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:36.301+0000 7f694ea00640 1 --2- 172.21.15.16:0/2596326272 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f694806ba80 0x7f69480ce6b0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:36.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:36.301+0000 7f694ea00640 1 -- 172.21.15.16:0/2596326272 >> 172.21.15.16:0/2596326272 conn(0x7f694806e670 msgr2=0x7f694806ea50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:09:36.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:36.301+0000 7f694ea00640 1 -- 172.21.15.16:0/2596326272 shutdown_connections 2024-03-31T15:09:36.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:36.301+0000 7f694ea00640 1 -- 172.21.15.16:0/2596326272 wait complete. 2024-03-31T15:09:36.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:36.305+0000 7f694ea00640 1 Processor -- start 2024-03-31T15:09:36.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:36.305+0000 7f694ea00640 1 -- start start 2024-03-31T15:09:36.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:36.305+0000 7f694ea00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f694806b620 0x7f6948159130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:36.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:36.305+0000 7f694ea00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f694806ba80 0x7f694815a7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:36.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:36.305+0000 7f694ea00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6948145190 0x7f694813ddd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:36.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:36.305+0000 7f694ea00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f694816a390 con 0x7f694806b620 2024-03-31T15:09:36.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:36.305+0000 7f694ea00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f694816a210 con 0x7f6948145190 2024-03-31T15:09:36.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:36.305+0000 7f694ea00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f694816a510 con 0x7f694806ba80 2024-03-31T15:09:36.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:36.305+0000 7f6947e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f694806b620 0x7f6948159130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:36.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:36.305+0000 7f6947e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f694806b620 0x7f6948159130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60854/0 (socket says 172.21.15.16:60854) 2024-03-31T15:09:36.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:36.305+0000 7f6947e00640 1 -- 172.21.15.16:0/2123329234 learned_addr learned my addr 172.21.15.16:0/2123329234 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:09:36.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:36.305+0000 7f693fe00640 1 --2- 172.21.15.16:0/2123329234 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f694806ba80 0x7f694815a7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:36.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:36.305+0000 7f694ca00640 1 --2- 172.21.15.16:0/2123329234 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6948145190 0x7f694813ddd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:36.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:36.305+0000 7f6947e00640 1 -- 172.21.15.16:0/2123329234 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f694806ba80 msgr2=0x7f694815a7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:36.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:36.305+0000 7f6947e00640 1 --2- 172.21.15.16:0/2123329234 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f694806ba80 0x7f694815a7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:36.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:36.305+0000 7f6947e00640 1 -- 172.21.15.16:0/2123329234 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6948145190 msgr2=0x7f694813ddd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:36.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:36.305+0000 7f6947e00640 1 --2- 172.21.15.16:0/2123329234 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6948145190 0x7f694813ddd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:36.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:36.305+0000 7f6947e00640 1 -- 172.21.15.16:0/2123329234 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6948159710 con 0x7f694806b620 2024-03-31T15:09:36.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:36.305+0000 7f693fe00640 1 --2- 172.21.15.16:0/2123329234 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f694806ba80 0x7f694815a7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:09:36.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:36.305+0000 7f6947e00640 1 --2- 172.21.15.16:0/2123329234 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f694806b620 0x7f6948159130 secure :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 crypto rx=0x7f693800c480 tx=0x7f693800c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:09:36.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:36.305+0000 7f6944c00640 1 -- 172.21.15.16:0/2123329234 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6938018070 con 0x7f694806b620 2024-03-31T15:09:36.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:36.305+0000 7f6944c00640 1 -- 172.21.15.16:0/2123329234 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f69380074e0 con 0x7f694806b620 2024-03-31T15:09:36.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:36.305+0000 7f6944c00640 1 -- 172.21.15.16:0/2123329234 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6938004a30 con 0x7f694806b620 2024-03-31T15:09:36.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:36.305+0000 7f694ea00640 1 -- 172.21.15.16:0/2123329234 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f69481599f0 con 0x7f694806b620 2024-03-31T15:09:36.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:36.305+0000 7f694ea00640 1 -- 172.21.15.16:0/2123329234 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6948159fb0 con 0x7f694806b620 2024-03-31T15:09:36.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:36.309+0000 7f694ea00640 1 -- 172.21.15.16:0/2123329234 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6948069950 con 0x7f694806b620 2024-03-31T15:09:36.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:36.309+0000 7f6944c00640 1 -- 172.21.15.16:0/2123329234 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f693801b070 con 0x7f694806b620 2024-03-31T15:09:36.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:36.309+0000 7f6944c00640 1 --2- 172.21.15.16:0/2123329234 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f6924068380 0x7f692406a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:36.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:36.309+0000 7f6944c00640 1 -- 172.21.15.16:0/2123329234 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(70..70 src has 1..70) ==== 5882+0+0 (secure 0 0 0) 0x7f6938085fc0 con 0x7f694806b620 2024-03-31T15:09:36.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:36.309+0000 7f693fe00640 1 --2- 172.21.15.16:0/2123329234 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f6924068380 0x7f692406a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:36.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:36.309+0000 7f693fe00640 1 --2- 172.21.15.16:0/2123329234 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f6924068380 0x7f692406a840 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f6948156f80 tx=0x7f6930009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:09:36.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:36.313+0000 7f6944c00640 1 -- 172.21.15.16:0/2123329234 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f6938055ff0 con 0x7f694806b620 2024-03-31T15:09:36.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:36.481+0000 7f694ea00640 1 -- 172.21.15.16:0/2123329234 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f69480737b0 con 0x7f694806b620 2024-03-31T15:09:36.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:36.481+0000 7f6944c00640 1 -- 172.21.15.16:0/2123329234 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v70) ==== 104+0+11 (secure 0 0 0) 0x7f693805a2f0 con 0x7f694806b620 2024-03-31T15:09:36.473 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2024-03-31T15:09:36.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:36.485+0000 7f694ea00640 1 -- 172.21.15.16:0/2123329234 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f6924068380 msgr2=0x7f692406a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:36.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:36.485+0000 7f694ea00640 1 --2- 172.21.15.16:0/2123329234 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f6924068380 0x7f692406a840 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f6948156f80 tx=0x7f6930009040 comp rx=0 tx=0).stop 2024-03-31T15:09:36.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:36.485+0000 7f694ea00640 1 -- 172.21.15.16:0/2123329234 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f694806b620 msgr2=0x7f6948159130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:36.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:36.485+0000 7f694ea00640 1 --2- 172.21.15.16:0/2123329234 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f694806b620 0x7f6948159130 secure :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 crypto rx=0x7f693800c480 tx=0x7f693800c940 comp rx=0 tx=0).stop 2024-03-31T15:09:36.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:36.485+0000 7f694ea00640 1 -- 172.21.15.16:0/2123329234 shutdown_connections 2024-03-31T15:09:36.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:36.485+0000 7f694ea00640 1 --2- 172.21.15.16:0/2123329234 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f6924068380 0x7f692406a840 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:36.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:36.485+0000 7f694ea00640 1 --2- 172.21.15.16:0/2123329234 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6948145190 0x7f694813ddd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:36.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:36.485+0000 7f694ea00640 1 --2- 172.21.15.16:0/2123329234 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f694806ba80 0x7f694815a7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:36.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:36.485+0000 7f694ea00640 1 --2- 172.21.15.16:0/2123329234 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f694806b620 0x7f6948159130 unknown :-1 s=CLOSED pgs=1005 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:36.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:36.485+0000 7f694ea00640 1 -- 172.21.15.16:0/2123329234 >> 172.21.15.16:0/2123329234 conn(0x7f694806e670 msgr2=0x7f6948143790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:09:36.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:36.485+0000 7f694ea00640 1 -- 172.21.15.16:0/2123329234 shutdown_connections 2024-03-31T15:09:36.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:36.485+0000 7f694ea00640 1 -- 172.21.15.16:0/2123329234 wait complete. 2024-03-31T15:09:36.504 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-31T15:09:36.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:36.657+0000 7fe69ba00640 1 Processor -- start 2024-03-31T15:09:36.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:36.657+0000 7fe69ba00640 1 -- start start 2024-03-31T15:09:36.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:36.657+0000 7fe69ba00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fe694070030 0x7fe694143c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:36.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:36.657+0000 7fe69ba00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fe694141590 0x7fe694144170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:36.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:36.657+0000 7fe69ba00640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe6941446b0 con 0x7fe6941411b0 2024-03-31T15:09:36.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:36.657+0000 7fe69ba00640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7fe69415f910 con 0x7fe694141590 2024-03-31T15:09:36.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:36.657+0000 7fe69ba00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fe694162910 con 0x7fe694070030 2024-03-31T15:09:36.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:36.657+0000 7fe699a00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fe694141590 0x7fe694144170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:36.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:36.657+0000 7fe693e00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fe694070030 0x7fe694143c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:36.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:36.657+0000 7fe699a00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fe694141590 0x7fe694144170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.16:34034/0 (socket says 172.21.15.16:34034) 2024-03-31T15:09:36.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:36.657+0000 7fe699a00640 1 -- 172.21.15.16:0/2389797707 learned_addr learned my addr 172.21.15.16:0/2389797707 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:09:36.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:36.657+0000 7fe699a00640 1 -- 172.21.15.16:0/2389797707 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fe694070030 msgr2=0x7fe694143c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:09:36.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:36.657+0000 7fe699a00640 1 --2- 172.21.15.16:0/2389797707 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fe694070030 0x7fe694143c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:36.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:36.657+0000 7fe699a00640 1 -- 172.21.15.16:0/2389797707 >> v1:172.21.15.89:6789/0 conn(0x7fe6941411b0 legacy=0x7fe69406f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:36.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:36.657+0000 7fe699a00640 1 -- 172.21.15.16:0/2389797707 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe694144980 con 0x7fe694141590 2024-03-31T15:09:36.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:36.657+0000 7fe693400640 1 -- 172.21.15.16:0/2389797707 <== mon.1 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2017554070 0 0) 0x7fe6941446b0 con 0x7fe6941411b0 2024-03-31T15:09:36.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:36.661+0000 7fe699a00640 1 --2- 172.21.15.16:0/2389797707 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe694141590 0x7fe694144170 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7fe68c009e50 tx=0x7fe68c01ca00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3adc3652b374ecab server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:09:36.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:36.661+0000 7fe693400640 1 -- 172.21.15.16:0/2389797707 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe68c0a58d0 con 0x7fe694141590 2024-03-31T15:09:36.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:36.661+0000 7fe693400640 1 -- 172.21.15.16:0/2389797707 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe68c0a5a70 con 0x7fe694141590 2024-03-31T15:09:36.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:36.661+0000 7fe693400640 1 -- 172.21.15.16:0/2389797707 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe68c0a5e40 con 0x7fe694141590 2024-03-31T15:09:36.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:36.661+0000 7fe69ba00640 1 -- 172.21.15.16:0/2389797707 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe694141590 msgr2=0x7fe694144170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:36.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:36.661+0000 7fe69ba00640 1 --2- 172.21.15.16:0/2389797707 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe694141590 0x7fe694144170 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7fe68c009e50 tx=0x7fe68c01ca00 comp rx=0 tx=0).stop 2024-03-31T15:09:36.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:36.661+0000 7fe69ba00640 1 -- 172.21.15.16:0/2389797707 shutdown_connections 2024-03-31T15:09:36.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:36.661+0000 7fe69ba00640 1 --2- 172.21.15.16:0/2389797707 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe694141590 0x7fe694144170 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:36.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:36.661+0000 7fe69ba00640 1 --2- 172.21.15.16:0/2389797707 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fe694070030 0x7fe694143c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:36.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:36.661+0000 7fe69ba00640 1 -- 172.21.15.16:0/2389797707 >> 172.21.15.16:0/2389797707 conn(0x7fe694067930 msgr2=0x7fe6940b3ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:09:36.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:36.661+0000 7fe69ba00640 1 -- 172.21.15.16:0/2389797707 shutdown_connections 2024-03-31T15:09:36.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:36.661+0000 7fe69ba00640 1 -- 172.21.15.16:0/2389797707 wait complete. 2024-03-31T15:09:36.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:36.665+0000 7fe69ba00640 1 Processor -- start 2024-03-31T15:09:36.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:36.665+0000 7fe69ba00640 1 -- start start 2024-03-31T15:09:36.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:36.665+0000 7fe69ba00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fe694070030 0x7fe69413f2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:36.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:36.665+0000 7fe69ba00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe6941411b0 0x7fe694159010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:36.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:36.665+0000 7fe69ba00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe694141590 0x7fe69415d340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:36.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:36.665+0000 7fe69ba00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fe69415f110 con 0x7fe694141590 2024-03-31T15:09:36.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:36.665+0000 7fe69ba00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fe694162da0 con 0x7fe6941411b0 2024-03-31T15:09:36.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:36.665+0000 7fe69ba00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fe69415f290 con 0x7fe694070030 2024-03-31T15:09:36.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:36.665+0000 7fe699a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe694141590 0x7fe69415d340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:36.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:36.665+0000 7fe699a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe694141590 0x7fe69415d340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60858/0 (socket says 172.21.15.16:60858) 2024-03-31T15:09:36.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:36.665+0000 7fe699a00640 1 -- 172.21.15.16:0/95792336 learned_addr learned my addr 172.21.15.16:0/95792336 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:09:36.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:36.665+0000 7fe693e00640 1 --2- 172.21.15.16:0/95792336 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe6941411b0 0x7fe694159010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:36.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:36.665+0000 7fe699000640 1 --2- 172.21.15.16:0/95792336 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fe694070030 0x7fe69413f2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:36.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:36.665+0000 7fe699a00640 1 -- 172.21.15.16:0/95792336 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fe694070030 msgr2=0x7fe69413f2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:36.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:36.665+0000 7fe699a00640 1 --2- 172.21.15.16:0/95792336 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fe694070030 0x7fe69413f2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:36.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:36.665+0000 7fe699a00640 1 -- 172.21.15.16:0/95792336 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe6941411b0 msgr2=0x7fe694159010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:36.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:36.665+0000 7fe699a00640 1 --2- 172.21.15.16:0/95792336 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe6941411b0 0x7fe694159010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:36.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:36.665+0000 7fe699a00640 1 -- 172.21.15.16:0/95792336 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe694140130 con 0x7fe694141590 2024-03-31T15:09:36.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:36.665+0000 7fe699a00640 1 --2- 172.21.15.16:0/95792336 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe694141590 0x7fe69415d340 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 crypto rx=0x7fe68c009e50 tx=0x7fe68c0197a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:09:36.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:36.665+0000 7fe690c00640 1 -- 172.21.15.16:0/95792336 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe68c0bd020 con 0x7fe694141590 2024-03-31T15:09:36.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:36.665+0000 7fe690c00640 1 -- 172.21.15.16:0/95792336 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe68c007e30 con 0x7fe694141590 2024-03-31T15:09:36.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:36.665+0000 7fe69ba00640 1 -- 172.21.15.16:0/95792336 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe694140410 con 0x7fe694141590 2024-03-31T15:09:36.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:36.665+0000 7fe690c00640 1 -- 172.21.15.16:0/95792336 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe68c0a58d0 con 0x7fe694141590 2024-03-31T15:09:36.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:36.665+0000 7fe69ba00640 1 -- 172.21.15.16:0/95792336 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe69413f7e0 con 0x7fe694141590 2024-03-31T15:09:36.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:36.669+0000 7fe69ba00640 1 -- 172.21.15.16:0/95792336 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe694069950 con 0x7fe694141590 2024-03-31T15:09:36.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:36.669+0000 7fe690c00640 1 -- 172.21.15.16:0/95792336 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fe68c0c0070 con 0x7fe694141590 2024-03-31T15:09:36.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:36.669+0000 7fe690c00640 1 --2- 172.21.15.16:0/95792336 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fe674068380 0x7fe67406a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:36.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:36.669+0000 7fe690c00640 1 -- 172.21.15.16:0/95792336 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(70..70 src has 1..70) ==== 5882+0+0 (secure 0 0 0) 0x7fe68c121440 con 0x7fe694141590 2024-03-31T15:09:36.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:36.669+0000 7fe699000640 1 --2- 172.21.15.16:0/95792336 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fe674068380 0x7fe67406a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:36.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:36.673+0000 7fe699000640 1 --2- 172.21.15.16:0/95792336 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fe674068380 0x7fe67406a840 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7fe688000f70 tx=0x7fe688009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:09:36.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:36.673+0000 7fe690c00640 1 -- 172.21.15.16:0/95792336 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fe68c016620 con 0x7fe694141590 2024-03-31T15:09:36.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:36.849+0000 7fe69ba00640 1 -- 172.21.15.16:0/95792336 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fe6941405e0 con 0x7fe694141590 2024-03-31T15:09:36.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:36.849+0000 7fe690c00640 1 -- 172.21.15.16:0/95792336 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v70) ==== 100+0+11 (secure 0 0 0) 0x7fe68c0f1260 con 0x7fe694141590 2024-03-31T15:09:36.841 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 64 2024-03-31T15:09:36.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:36.853+0000 7fe69ba00640 1 -- 172.21.15.16:0/95792336 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fe674068380 msgr2=0x7fe67406a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:36.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:36.853+0000 7fe69ba00640 1 --2- 172.21.15.16:0/95792336 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fe674068380 0x7fe67406a840 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7fe688000f70 tx=0x7fe688009040 comp rx=0 tx=0).stop 2024-03-31T15:09:36.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:36.853+0000 7fe69ba00640 1 -- 172.21.15.16:0/95792336 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe694141590 msgr2=0x7fe69415d340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:36.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:36.853+0000 7fe69ba00640 1 --2- 172.21.15.16:0/95792336 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe694141590 0x7fe69415d340 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 crypto rx=0x7fe68c009e50 tx=0x7fe68c0197a0 comp rx=0 tx=0).stop 2024-03-31T15:09:36.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:36.853+0000 7fe69ba00640 1 -- 172.21.15.16:0/95792336 shutdown_connections 2024-03-31T15:09:36.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:36.853+0000 7fe69ba00640 1 --2- 172.21.15.16:0/95792336 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fe674068380 0x7fe67406a840 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:36.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:36.853+0000 7fe69ba00640 1 --2- 172.21.15.16:0/95792336 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe694141590 0x7fe69415d340 unknown :-1 s=CLOSED pgs=1006 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:36.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:36.853+0000 7fe69ba00640 1 --2- 172.21.15.16:0/95792336 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe6941411b0 0x7fe694159010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:36.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:36.853+0000 7fe69ba00640 1 --2- 172.21.15.16:0/95792336 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fe694070030 0x7fe69413f2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:36.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:36.853+0000 7fe69ba00640 1 -- 172.21.15.16:0/95792336 >> 172.21.15.16:0/95792336 conn(0x7fe694067930 msgr2=0x7fe69406c730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:09:36.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:36.853+0000 7fe69ba00640 1 -- 172.21.15.16:0/95792336 shutdown_connections 2024-03-31T15:09:36.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:36.853+0000 7fe69ba00640 1 -- 172.21.15.16:0/95792336 wait complete. 2024-03-31T15:09:36.862 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-31T15:09:37.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:37.009+0000 7f3139800640 1 Processor -- start 2024-03-31T15:09:37.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:37.009+0000 7f3139800640 1 -- start start 2024-03-31T15:09:37.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:37.013+0000 7f3139800640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f313406dc20 0x7f313406bd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:37.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:37.013+0000 7f3139800640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f313415e9b0 con 0x7f31341629f0 2024-03-31T15:09:37.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:37.013+0000 7f3139800640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f313415ec70 con 0x7f31340b3f20 2024-03-31T15:09:37.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:37.013+0000 7f3139800640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f313415ef50 con 0x7f313406dc20 2024-03-31T15:09:37.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:37.013+0000 7f3133400640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f313406dc20 0x7f313406bd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:37.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:37.013+0000 7f3132a00640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f31341629f0 0x7f31340b3810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:46752/0 (socket says 172.21.15.16:46752) 2024-03-31T15:09:37.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:37.013+0000 7f3132a00640 1 -- 172.21.15.16:0/3559657592 learned_addr learned my addr 172.21.15.16:0/3559657592 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:09:37.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:37.013+0000 7f3133400640 1 -- 172.21.15.16:0/3559657592 >> v1:172.21.15.89:6789/0 conn(0x7f31340b3f20 legacy=0x7f313406b620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:09:37.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:37.013+0000 7f3133400640 1 -- 172.21.15.16:0/3559657592 >> v1:172.21.15.16:6789/0 conn(0x7f31341629f0 legacy=0x7f31340b3810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:37.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:37.013+0000 7f3133400640 1 -- 172.21.15.16:0/3559657592 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f313415f230 con 0x7f313406dc20 2024-03-31T15:09:37.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:37.013+0000 7f3132000640 1 -- 172.21.15.16:0/3559657592 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3126105523 0 0) 0x7f313415e9b0 con 0x7f31341629f0 2024-03-31T15:09:37.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:37.017+0000 7f3133400640 1 --2- 172.21.15.16:0/3559657592 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f313406dc20 0x7f313406bd30 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 crypto rx=0x7f312c009d90 tx=0x7f313406ca80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=879dd13375ae2a0a server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:09:37.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:37.017+0000 7f3132000640 1 -- 172.21.15.16:0/3559657592 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f312c0b2070 con 0x7f313406dc20 2024-03-31T15:09:37.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:37.017+0000 7f3132000640 1 -- 172.21.15.16:0/3559657592 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f312c0a5940 con 0x7f313406dc20 2024-03-31T15:09:37.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:37.017+0000 7f3132000640 1 -- 172.21.15.16:0/3559657592 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f312c0a5d10 con 0x7f313406dc20 2024-03-31T15:09:37.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:37.017+0000 7f3139800640 1 -- 172.21.15.16:0/3559657592 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f313406dc20 msgr2=0x7f313406bd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:37.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:37.017+0000 7f3139800640 1 --2- 172.21.15.16:0/3559657592 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f313406dc20 0x7f313406bd30 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 crypto rx=0x7f312c009d90 tx=0x7f313406ca80 comp rx=0 tx=0).stop 2024-03-31T15:09:37.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:37.017+0000 7f3139800640 1 -- 172.21.15.16:0/3559657592 shutdown_connections 2024-03-31T15:09:37.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:37.017+0000 7f3139800640 1 --2- 172.21.15.16:0/3559657592 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f313406dc20 0x7f313406bd30 unknown :-1 s=CLOSED pgs=1008 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:37.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:37.017+0000 7f3139800640 1 -- 172.21.15.16:0/3559657592 >> 172.21.15.16:0/3559657592 conn(0x7f31340d0760 msgr2=0x7f31340d0b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:09:37.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:37.017+0000 7f3139800640 1 -- 172.21.15.16:0/3559657592 shutdown_connections 2024-03-31T15:09:37.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:37.017+0000 7f3139800640 1 -- 172.21.15.16:0/3559657592 wait complete. 2024-03-31T15:09:37.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:37.017+0000 7f3139800640 1 Processor -- start 2024-03-31T15:09:37.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:37.017+0000 7f3139800640 1 -- start start 2024-03-31T15:09:37.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:37.021+0000 7f3139800640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f313406dc20 0x7f3134154dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:37.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:37.021+0000 7f3139800640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f31340b3f20 0x7f31341f8b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:37.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:37.021+0000 7f3139800640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f31341629f0 0x7f31341f7f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:37.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:37.021+0000 7f3139800640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f313416c4d0 con 0x7f31340b3f20 2024-03-31T15:09:37.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:37.021+0000 7f3139800640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f313416c350 con 0x7f313406dc20 2024-03-31T15:09:37.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:37.021+0000 7f3139800640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f313416c650 con 0x7f31341629f0 2024-03-31T15:09:37.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:37.021+0000 7f312be00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f31340b3f20 0x7f31341f8b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:37.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:37.021+0000 7f312be00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f31340b3f20 0x7f31341f8b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60862/0 (socket says 172.21.15.16:60862) 2024-03-31T15:09:37.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:37.021+0000 7f312be00640 1 -- 172.21.15.16:0/3044957615 learned_addr learned my addr 172.21.15.16:0/3044957615 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:09:37.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:37.021+0000 7f3132a00640 1 --2- 172.21.15.16:0/3044957615 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f313406dc20 0x7f3134154dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:37.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:37.021+0000 7f3133400640 1 --2- 172.21.15.16:0/3044957615 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f31341629f0 0x7f31341f7f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:37.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:37.021+0000 7f312be00640 1 -- 172.21.15.16:0/3044957615 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f31341629f0 msgr2=0x7f31341f7f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:37.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:37.021+0000 7f312be00640 1 --2- 172.21.15.16:0/3044957615 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f31341629f0 0x7f31341f7f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:37.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:37.021+0000 7f312be00640 1 -- 172.21.15.16:0/3044957615 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f313406dc20 msgr2=0x7f3134154dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:37.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:37.021+0000 7f312be00640 1 --2- 172.21.15.16:0/3044957615 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f313406dc20 0x7f3134154dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:37.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:37.021+0000 7f312be00640 1 -- 172.21.15.16:0/3044957615 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3134155780 con 0x7f31340b3f20 2024-03-31T15:09:37.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:37.021+0000 7f312be00640 1 --2- 172.21.15.16:0/3044957615 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f31340b3f20 0x7f31341f8b20 secure :-1 s=READY pgs=1009 cs=0 l=1 rev1=1 crypto rx=0x7f311c00c500 tx=0x7f311c00c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:09:37.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:37.021+0000 7f312aa00640 1 -- 172.21.15.16:0/3044957615 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f311c016020 con 0x7f31340b3f20 2024-03-31T15:09:37.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:37.021+0000 7f312aa00640 1 -- 172.21.15.16:0/3044957615 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f311c005220 con 0x7f31340b3f20 2024-03-31T15:09:37.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:37.021+0000 7f3139800640 1 -- 172.21.15.16:0/3044957615 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3134155a60 con 0x7f31340b3f20 2024-03-31T15:09:37.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:37.021+0000 7f312aa00640 1 -- 172.21.15.16:0/3044957615 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f311c007030 con 0x7f31340b3f20 2024-03-31T15:09:37.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:37.021+0000 7f3139800640 1 -- 172.21.15.16:0/3044957615 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f31341f76b0 con 0x7f31340b3f20 2024-03-31T15:09:37.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:37.025+0000 7f3139800640 1 -- 172.21.15.16:0/3044957615 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3134069950 con 0x7f31340b3f20 2024-03-31T15:09:37.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:37.025+0000 7f312aa00640 1 -- 172.21.15.16:0/3044957615 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f311c0071f0 con 0x7f31340b3f20 2024-03-31T15:09:37.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:37.025+0000 7f312aa00640 1 --2- 172.21.15.16:0/3044957615 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f3110068380 0x7f311006a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:37.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:37.025+0000 7f312aa00640 1 -- 172.21.15.16:0/3044957615 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(70..70 src has 1..70) ==== 5882+0+0 (secure 0 0 0) 0x7f311c086420 con 0x7f31340b3f20 2024-03-31T15:09:37.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:37.025+0000 7f3132a00640 1 --2- 172.21.15.16:0/3044957615 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f3110068380 0x7f311006a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:37.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:37.025+0000 7f3132a00640 1 --2- 172.21.15.16:0/3044957615 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f3110068380 0x7f311006a840 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f3124000f70 tx=0x7f3124007240 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:09:37.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:37.029+0000 7f312aa00640 1 -- 172.21.15.16:0/3044957615 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f311c020020 con 0x7f31340b3f20 2024-03-31T15:09:37.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:37.229+0000 7f3139800640 1 -- 172.21.15.16:0/3044957615 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f31341627d0 con 0x7f31340b3f20 2024-03-31T15:09:37.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:37.229+0000 7f312aa00640 1 -- 172.21.15.16:0/3044957615 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 52 v52) ==== 94+0+5185 (secure 0 0 0) 0x7f311c056250 con 0x7f31340b3f20 2024-03-31T15:09:37.223 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:09:37.223 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":52,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6836/4188793506","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":4188793506},{"type":"v1","addr":"172.21.15.89:6836","nonce":4188793506}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":40},{"gid":5942,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/4105657676","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6835","nonce":4105657676},{"type":"v1","addr":"172.21.15.89:6837","nonce":4105657676}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":40},{"gid":6592,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6833/3986752851","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":3986752851},{"type":"v1","addr":"172.21.15.16:6833","nonce":3986752851}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":39}],"filesystems":[{"mdsmap":{"epoch":52,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-31T15:07:56.938140+0000","modified":"2024-03-31T15:09:01.267292+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":68,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6829},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6829":{"gid":6829,"name":"a","rank":0,"incarnation":47,"state":"up:active","state_seq":16,"addr":"172.21.15.16:6835/2645389320","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":2645389320},{"type":"v1","addr":"172.21.15.16:6835","nonce":2645389320}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6829,"qdb_cluster":[6829]},"id":5}]} 2024-03-31T15:09:37.223 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 52 2024-03-31T15:09:37.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:37.233+0000 7f3139800640 1 -- 172.21.15.16:0/3044957615 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f3110068380 msgr2=0x7f311006a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:37.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:37.233+0000 7f3139800640 1 --2- 172.21.15.16:0/3044957615 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f3110068380 0x7f311006a840 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f3124000f70 tx=0x7f3124007240 comp rx=0 tx=0).stop 2024-03-31T15:09:37.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:37.233+0000 7f3139800640 1 -- 172.21.15.16:0/3044957615 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f31340b3f20 msgr2=0x7f31341f8b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:37.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:37.233+0000 7f3139800640 1 --2- 172.21.15.16:0/3044957615 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f31340b3f20 0x7f31341f8b20 secure :-1 s=READY pgs=1009 cs=0 l=1 rev1=1 crypto rx=0x7f311c00c500 tx=0x7f311c00c9c0 comp rx=0 tx=0).stop 2024-03-31T15:09:37.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:37.233+0000 7f3139800640 1 -- 172.21.15.16:0/3044957615 shutdown_connections 2024-03-31T15:09:37.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:37.233+0000 7f3139800640 1 --2- 172.21.15.16:0/3044957615 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f3110068380 0x7f311006a840 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:37.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:37.233+0000 7f3139800640 1 --2- 172.21.15.16:0/3044957615 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f31341629f0 0x7f31341f7f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:37.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:37.233+0000 7f3139800640 1 --2- 172.21.15.16:0/3044957615 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f31340b3f20 0x7f31341f8b20 unknown :-1 s=CLOSED pgs=1009 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:37.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:37.233+0000 7f3139800640 1 --2- 172.21.15.16:0/3044957615 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f313406dc20 0x7f3134154dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:37.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:37.233+0000 7f3139800640 1 -- 172.21.15.16:0/3044957615 >> 172.21.15.16:0/3044957615 conn(0x7f31340d0760 msgr2=0x7f313415c540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:09:37.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:37.233+0000 7f3139800640 1 -- 172.21.15.16:0/3044957615 shutdown_connections 2024-03-31T15:09:37.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:37.233+0000 7f3139800640 1 -- 172.21.15.16:0/3044957615 wait complete. 2024-03-31T15:09:37.248 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-31T15:09:37.256 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-31T15:09:37.297 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-31T15:09:37.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:37.401+0000 7f2391800640 1 Processor -- start 2024-03-31T15:09:37.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:37.401+0000 7f2391800640 1 -- start start 2024-03-31T15:09:37.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:37.401+0000 7f2391800640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f238c145d10 0x7f238c0cf790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:37.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:37.401+0000 7f2391800640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f238c0728b0 0x7f238c0b1cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:37.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:37.401+0000 7f2391800640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f238c160e70 con 0x7f238c0b2200 2024-03-31T15:09:37.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:37.401+0000 7f2391800640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f238c1602f0 con 0x7f238c0728b0 2024-03-31T15:09:37.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:37.401+0000 7f2391800640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f238c1605d0 con 0x7f238c145d10 2024-03-31T15:09:37.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:37.401+0000 7f238a000640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f238c0728b0 0x7f238c0b1cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:37.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:37.401+0000 7f238a000640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f238c0728b0 0x7f238c0b1cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59636/0 (socket says 172.21.15.16:59636) 2024-03-31T15:09:37.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:37.401+0000 7f238a000640 1 -- 172.21.15.16:0/3755627737 learned_addr learned my addr 172.21.15.16:0/3755627737 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:09:37.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:37.401+0000 7f238aa00640 1 --2- 172.21.15.16:0/3755627737 >> v2:172.21.15.89:3300/0 conn(0x7f238c145d10 0x7f238c0cf790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:37.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:37.401+0000 7f238a000640 1 -- 172.21.15.16:0/3755627737 >> v2:172.21.15.89:3300/0 conn(0x7f238c145d10 msgr2=0x7f238c0cf790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:09:37.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:37.401+0000 7f238a000640 1 --2- 172.21.15.16:0/3755627737 >> v2:172.21.15.89:3300/0 conn(0x7f238c145d10 0x7f238c0cf790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:37.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:37.401+0000 7f238a000640 1 -- 172.21.15.16:0/3755627737 >> v1:172.21.15.89:6789/0 conn(0x7f238c0b2200 legacy=0x7f238c15cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:37.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:37.401+0000 7f238a000640 1 -- 172.21.15.16:0/3755627737 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f238c0cfcd0 con 0x7f238c0728b0 2024-03-31T15:09:37.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:37.401+0000 7f238aa00640 1 --2- 172.21.15.16:0/3755627737 >> v2:172.21.15.89:3300/0 conn(0x7f238c145d10 0x7f238c0cf790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:09:37.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:37.405+0000 7f238a000640 1 --2- 172.21.15.16:0/3755627737 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f238c0728b0 0x7f238c0b1cc0 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 crypto rx=0x7f238c0d1060 tx=0x7f237801c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97f403314a2da838 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:09:37.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:37.405+0000 7f2389600640 1 -- 172.21.15.16:0/3755627737 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23780b0070 con 0x7f238c0728b0 2024-03-31T15:09:37.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:37.405+0000 7f2389600640 1 -- 172.21.15.16:0/3755627737 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f23780ab600 con 0x7f238c0728b0 2024-03-31T15:09:37.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:37.405+0000 7f2389600640 1 -- 172.21.15.16:0/3755627737 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23780ab9d0 con 0x7f238c0728b0 2024-03-31T15:09:37.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:37.409+0000 7f2391800640 1 -- 172.21.15.16:0/3755627737 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f238c0728b0 msgr2=0x7f238c0b1cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:37.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:37.409+0000 7f2391800640 1 --2- 172.21.15.16:0/3755627737 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f238c0728b0 0x7f238c0b1cc0 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 crypto rx=0x7f238c0d1060 tx=0x7f237801c520 comp rx=0 tx=0).stop 2024-03-31T15:09:37.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:37.409+0000 7f2391800640 1 -- 172.21.15.16:0/3755627737 shutdown_connections 2024-03-31T15:09:37.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:37.409+0000 7f2391800640 1 --2- 172.21.15.16:0/3755627737 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f238c0728b0 0x7f238c0b1cc0 unknown :-1 s=CLOSED pgs=1010 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:37.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:37.409+0000 7f2391800640 1 --2- 172.21.15.16:0/3755627737 >> v2:172.21.15.89:3300/0 conn(0x7f238c145d10 0x7f238c0cf790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:37.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:37.409+0000 7f2391800640 1 -- 172.21.15.16:0/3755627737 >> 172.21.15.16:0/3755627737 conn(0x7f238c06b620 msgr2=0x7f238c06ba30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:09:37.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:37.409+0000 7f2391800640 1 -- 172.21.15.16:0/3755627737 shutdown_connections 2024-03-31T15:09:37.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:37.409+0000 7f2391800640 1 -- 172.21.15.16:0/3755627737 wait complete. 2024-03-31T15:09:37.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:37.409+0000 7f2391800640 1 Processor -- start 2024-03-31T15:09:37.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:37.409+0000 7f2391800640 1 -- start start 2024-03-31T15:09:37.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:37.413+0000 7f2391800640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f238c0728b0 0x7f238c138a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:37.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:37.413+0000 7f2391800640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f238c0b2200 0x7f238c165700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:37.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:37.413+0000 7f2391800640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f238c145d10 0x7f238c143310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:37.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:37.413+0000 7f2391800640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f238c15d870 con 0x7f238c145d10 2024-03-31T15:09:37.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:37.413+0000 7f2391800640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f238c0b2610 con 0x7f238c0b2200 2024-03-31T15:09:37.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:37.413+0000 7f2391800640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f238c0cf350 con 0x7f238c0728b0 2024-03-31T15:09:37.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:37.413+0000 7f238b400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f238c145d10 0x7f238c143310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:37.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:37.413+0000 7f238b400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f238c145d10 0x7f238c143310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59650/0 (socket says 172.21.15.16:59650) 2024-03-31T15:09:37.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:37.413+0000 7f238b400640 1 -- 172.21.15.16:0/4185080494 learned_addr learned my addr 172.21.15.16:0/4185080494 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:09:37.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:37.413+0000 7f238a000640 1 --2- 172.21.15.16:0/4185080494 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f238c0b2200 0x7f238c165700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:37.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:37.413+0000 7f238aa00640 1 --2- 172.21.15.16:0/4185080494 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f238c0728b0 0x7f238c138a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:37.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:37.413+0000 7f238b400640 1 -- 172.21.15.16:0/4185080494 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f238c0728b0 msgr2=0x7f238c138a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:37.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:37.413+0000 7f238b400640 1 --2- 172.21.15.16:0/4185080494 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f238c0728b0 0x7f238c138a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:37.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:37.413+0000 7f238b400640 1 -- 172.21.15.16:0/4185080494 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f238c0b2200 msgr2=0x7f238c165700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:37.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:37.413+0000 7f238b400640 1 --2- 172.21.15.16:0/4185080494 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f238c0b2200 0x7f238c165700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:37.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:37.413+0000 7f238b400640 1 -- 172.21.15.16:0/4185080494 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f238c158430 con 0x7f238c145d10 2024-03-31T15:09:37.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:37.413+0000 7f238aa00640 1 --2- 172.21.15.16:0/4185080494 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f238c0728b0 0x7f238c138a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:09:37.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:37.413+0000 7f238b400640 1 --2- 172.21.15.16:0/4185080494 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f238c145d10 0x7f238c143310 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7f238400c770 tx=0x7f238400cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:09:37.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:37.413+0000 7f2382a00640 1 -- 172.21.15.16:0/4185080494 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2384003b80 con 0x7f238c145d10 2024-03-31T15:09:37.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:37.413+0000 7f2382a00640 1 -- 172.21.15.16:0/4185080494 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2384003d20 con 0x7f238c145d10 2024-03-31T15:09:37.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:37.413+0000 7f2382a00640 1 -- 172.21.15.16:0/4185080494 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f238400aa90 con 0x7f238c145d10 2024-03-31T15:09:37.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:37.413+0000 7f2391800640 1 -- 172.21.15.16:0/4185080494 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f238c158710 con 0x7f238c145d10 2024-03-31T15:09:37.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:37.413+0000 7f2391800640 1 -- 172.21.15.16:0/4185080494 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f238c1589f0 con 0x7f238c145d10 2024-03-31T15:09:37.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:37.413+0000 7f2391800640 1 -- 172.21.15.16:0/4185080494 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f238c0b3310 con 0x7f238c145d10 2024-03-31T15:09:37.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:37.417+0000 7f2382a00640 1 -- 172.21.15.16:0/4185080494 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f238400ac30 con 0x7f238c145d10 2024-03-31T15:09:37.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:37.417+0000 7f2382a00640 1 --2- 172.21.15.16:0/4185080494 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f236c068380 0x7f236c06a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:37.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:37.417+0000 7f2382a00640 1 -- 172.21.15.16:0/4185080494 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(70..70 src has 1..70) ==== 5882+0+0 (secure 0 0 0) 0x7f23840868e0 con 0x7f238c145d10 2024-03-31T15:09:37.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:37.417+0000 7f238aa00640 1 --2- 172.21.15.16:0/4185080494 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f236c068380 0x7f236c06a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:37.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:37.417+0000 7f238aa00640 1 --2- 172.21.15.16:0/4185080494 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f236c068380 0x7f236c06a840 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f237c00b990 tx=0x7f237c008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:09:37.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:37.421+0000 7f2382a00640 1 -- 172.21.15.16:0/4185080494 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f2384056de0 con 0x7f238c145d10 2024-03-31T15:09:37.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:37.589+0000 7f2391800640 1 -- 172.21.15.16:0/4185080494 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f238c069950 con 0x7f238c145d10 2024-03-31T15:09:37.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:37.593+0000 7f2382a00640 1 -- 172.21.15.16:0/4185080494 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v70) ==== 74+0+16870 (secure 0 0 0) 0x7f23840566e0 con 0x7f238c145d10 2024-03-31T15:09:37.584 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:09:37.584 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":70,"fsid":"286eb5da-fb47-4550-b290-d9519c13b9a5","created":"2024-03-31T15:03:11.438952+0000","modified":"2024-03-31T15:08:18.287907+0000","last_up_change":"2024-03-31T15:03:22.885535+0000","last_in_change":"2024-03-31T15:03:13.941667+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-31T15:03:22.550787+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-03-31T15:07:55.329096+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"66","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-03-31T15:07:55.885919+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"67","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"92a66450-ef19-40d8-ac3d-d760b23b0bda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6801","nonce":932712244}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6803","nonce":932712244}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6807","nonce":932712244}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6805","nonce":932712244}]},"public_addr":"172.21.15.16:6801/932712244","cluster_addr":"172.21.15.16:6803/932712244","heartbeat_back_addr":"172.21.15.16:6807/932712244","heartbeat_front_addr":"172.21.15.16:6805/932712244","state":["exists","up"]},{"osd":1,"uuid":"da9d1f2b-1968-4959-b843-faa231988e92","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6810","nonce":1398737533}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6811","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6813","nonce":1398737533}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6821","nonce":1398737533}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6815","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6816","nonce":1398737533}]},"public_addr":"172.21.15.16:6810/1398737533","cluster_addr":"172.21.15.16:6813/1398737533","heartbeat_back_addr":"172.21.15.16:6821/1398737533","heartbeat_front_addr":"172.21.15.16:6816/1398737533","state":["exists","up"]},{"osd":2,"uuid":"01a73662-9a21-4b1f-bcbb-42b9636a4638","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6819","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6822","nonce":3018526646}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6825","nonce":3018526646}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6831","nonce":3018526646}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6827","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6829","nonce":3018526646}]},"public_addr":"172.21.15.16:6822/3018526646","cluster_addr":"172.21.15.16:6825/3018526646","heartbeat_back_addr":"172.21.15.16:6831/3018526646","heartbeat_front_addr":"172.21.15.16:6829/3018526646","state":["exists","up"]},{"osd":3,"uuid":"c7771fd3-e18e-4500-ac38-23cf10ebcd75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6809","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6812","nonce":3360352974}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6817","nonce":3360352974}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6828","nonce":3360352974}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6823","nonce":3360352974}]},"public_addr":"172.21.15.16:6812/3360352974","cluster_addr":"172.21.15.16:6817/3360352974","heartbeat_back_addr":"172.21.15.16:6828/3360352974","heartbeat_front_addr":"172.21.15.16:6823/3360352974","state":["exists","up"]},{"osd":4,"uuid":"88260730-1482-4fbd-a40f-cb1124e9df4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6801","nonce":1856844856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6803","nonce":1856844856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6807","nonce":1856844856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6805","nonce":1856844856}]},"public_addr":"172.21.15.89:6801/1856844856","cluster_addr":"172.21.15.89:6803/1856844856","heartbeat_back_addr":"172.21.15.89:6807/1856844856","heartbeat_front_addr":"172.21.15.89:6805/1856844856","state":["exists","up"]},{"osd":5,"uuid":"4cb2c00c-fae0-4dd8-9d27-ec4f0093ccf7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6814","nonce":50395527}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6817","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6820","nonce":50395527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6831","nonce":50395527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6823","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6826","nonce":50395527}]},"public_addr":"172.21.15.89:6814/50395527","cluster_addr":"172.21.15.89:6820/50395527","heartbeat_back_addr":"172.21.15.89:6831/50395527","heartbeat_front_addr":"172.21.15.89:6826/50395527","state":["exists","up"]},{"osd":6,"uuid":"81ce0c19-2554-4cae-acbe-66a05e9e9b9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6809","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6812","nonce":3470471752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6815","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6818","nonce":3470471752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6827","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6829","nonce":3470471752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6821","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6824","nonce":3470471752}]},"public_addr":"172.21.15.89:6812/3470471752","cluster_addr":"172.21.15.89:6818/3470471752","heartbeat_back_addr":"172.21.15.89:6829/3470471752","heartbeat_front_addr":"172.21.15.89:6824/3470471752","state":["exists","up"]},{"osd":7,"uuid":"85e4313b-d37d-442e-8042-153bf20a847d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6811","nonce":1796778630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6813","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6816","nonce":1796778630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6825","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6828","nonce":1796778630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6819","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6822","nonce":1796778630}]},"public_addr":"172.21.15.89:6811/1796778630","cluster_addr":"172.21.15.89:6816/1796778630","heartbeat_back_addr":"172.21.15.89:6828/1796778630","heartbeat_front_addr":"172.21.15.89:6822/1796778630","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.439088+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.529970+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.680512+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:21.044161+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.582111+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.536558+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.409522+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.590229+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"11.4","mappings":[{"from":6,"to":0}]},{"pgid":"11.10","mappings":[{"from":6,"to":1}]},{"pgid":"11.3e","mappings":[{"from":6,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:6835/2555930820":"2024-04-01T15:08:10.537891+0000","172.21.15.16:6834/2555930820":"2024-04-01T15:08:10.537891+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-31T15:09:37.586 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:37.593+0000 7f2391800640 1 -- 172.21.15.16:0/4185080494 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f236c068380 msgr2=0x7f236c06a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:37.586 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:37.593+0000 7f2391800640 1 --2- 172.21.15.16:0/4185080494 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f236c068380 0x7f236c06a840 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f237c00b990 tx=0x7f237c008040 comp rx=0 tx=0).stop 2024-03-31T15:09:37.586 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:37.593+0000 7f2391800640 1 -- 172.21.15.16:0/4185080494 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f238c145d10 msgr2=0x7f238c143310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:37.586 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:37.593+0000 7f2391800640 1 --2- 172.21.15.16:0/4185080494 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f238c145d10 0x7f238c143310 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7f238400c770 tx=0x7f238400cc30 comp rx=0 tx=0).stop 2024-03-31T15:09:37.586 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:37.593+0000 7f2391800640 1 -- 172.21.15.16:0/4185080494 shutdown_connections 2024-03-31T15:09:37.586 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:37.593+0000 7f2391800640 1 --2- 172.21.15.16:0/4185080494 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f236c068380 0x7f236c06a840 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:37.587 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:37.593+0000 7f2391800640 1 --2- 172.21.15.16:0/4185080494 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f238c145d10 0x7f238c143310 unknown :-1 s=CLOSED pgs=1011 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:37.587 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:37.593+0000 7f2391800640 1 --2- 172.21.15.16:0/4185080494 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f238c0b2200 0x7f238c165700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:37.587 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:37.593+0000 7f2391800640 1 --2- 172.21.15.16:0/4185080494 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f238c0728b0 0x7f238c138a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:37.587 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:37.593+0000 7f2391800640 1 -- 172.21.15.16:0/4185080494 >> 172.21.15.16:0/4185080494 conn(0x7f238c06b620 msgr2=0x7f238c072d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:09:37.587 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:37.593+0000 7f2391800640 1 -- 172.21.15.16:0/4185080494 shutdown_connections 2024-03-31T15:09:37.587 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:37.593+0000 7f2391800640 1 -- 172.21.15.16:0/4185080494 wait complete. 2024-03-31T15:09:37.606 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-31T15:09:37.606 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-31T15:09:37.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:37.753+0000 7ff13ca00640 1 Processor -- start 2024-03-31T15:09:37.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:37.757+0000 7ff13ca00640 1 -- start start 2024-03-31T15:09:37.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:37.757+0000 7ff13ca00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff13806fd80 0x7ff1380d2720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:37.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:37.757+0000 7ff13ca00640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff1380d0920 con 0x7ff13806f920 2024-03-31T15:09:37.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:37.757+0000 7ff13ca00640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff1380d0c00 con 0x7ff138145e40 2024-03-31T15:09:37.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:37.757+0000 7ff13ca00640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7ff1380d06d0 con 0x7ff13806fd80 2024-03-31T15:09:37.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:37.757+0000 7ff12d400640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7ff13806f920 0x7ff1380d2010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:55166/0 (socket says 172.21.15.16:55166) 2024-03-31T15:09:37.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:37.757+0000 7ff12d400640 1 -- 172.21.15.16:0/1497524471 learned_addr learned my addr 172.21.15.16:0/1497524471 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:09:37.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:37.757+0000 7ff136800640 1 --2- 172.21.15.16:0/1497524471 >> v2:172.21.15.16:3300/0 conn(0x7ff13806fd80 0x7ff1380d2720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:37.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:37.757+0000 7ff135400640 1 -- 172.21.15.16:0/1497524471 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2481122607 0 0) 0x7ff1380d0920 con 0x7ff13806f920 2024-03-31T15:09:37.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:37.757+0000 7ff135400640 1 -- 172.21.15.16:0/1497524471 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff11c003530 con 0x7ff13806f920 2024-03-31T15:09:37.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:37.757+0000 7ff136800640 1 -- 172.21.15.16:0/1497524471 >> v1:172.21.15.89:6789/0 conn(0x7ff138145e40 legacy=0x7ff1380cc740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:37.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:37.757+0000 7ff136800640 1 -- 172.21.15.16:0/1497524471 >> v1:172.21.15.16:6789/0 conn(0x7ff13806f920 legacy=0x7ff1380d2010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:37.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:37.757+0000 7ff136800640 1 -- 172.21.15.16:0/1497524471 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1380d1810 con 0x7ff13806fd80 2024-03-31T15:09:37.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:37.761+0000 7ff136800640 1 --2- 172.21.15.16:0/1497524471 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff13806fd80 0x7ff1380d2720 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 crypto rx=0x7ff130009d90 tx=0x7ff13001c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2eebaa6e28564139 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:09:37.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:37.761+0000 7ff135400640 1 -- 172.21.15.16:0/1497524471 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1300b2070 con 0x7ff13806fd80 2024-03-31T15:09:37.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:37.761+0000 7ff135400640 1 -- 172.21.15.16:0/1497524471 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff1300a5920 con 0x7ff13806fd80 2024-03-31T15:09:37.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:37.761+0000 7ff135400640 1 -- 172.21.15.16:0/1497524471 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1300a5cf0 con 0x7ff13806fd80 2024-03-31T15:09:37.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:37.761+0000 7ff13ca00640 1 -- 172.21.15.16:0/1497524471 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff13806fd80 msgr2=0x7ff1380d2720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:37.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:37.761+0000 7ff13ca00640 1 --2- 172.21.15.16:0/1497524471 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff13806fd80 0x7ff1380d2720 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 crypto rx=0x7ff130009d90 tx=0x7ff13001c970 comp rx=0 tx=0).stop 2024-03-31T15:09:37.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:37.761+0000 7ff13ca00640 1 -- 172.21.15.16:0/1497524471 shutdown_connections 2024-03-31T15:09:37.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:37.761+0000 7ff13ca00640 1 --2- 172.21.15.16:0/1497524471 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff13806fd80 0x7ff1380d2720 unknown :-1 s=CLOSED pgs=1013 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:37.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:37.761+0000 7ff13ca00640 1 -- 172.21.15.16:0/1497524471 >> 172.21.15.16:0/1497524471 conn(0x7ff138067330 msgr2=0x7ff138067740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:09:37.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:37.761+0000 7ff13ca00640 1 -- 172.21.15.16:0/1497524471 shutdown_connections 2024-03-31T15:09:37.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:37.761+0000 7ff13ca00640 1 -- 172.21.15.16:0/1497524471 wait complete. 2024-03-31T15:09:37.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:37.765+0000 7ff13ca00640 1 Processor -- start 2024-03-31T15:09:37.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:37.765+0000 7ff13ca00640 1 -- start start 2024-03-31T15:09:37.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:37.765+0000 7ff13ca00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff13806f920 0x7ff1380b5fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:37.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:37.765+0000 7ff13ca00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff13806fd80 0x7ff1380b57b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:37.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:37.765+0000 7ff13ca00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff138145e40 0x7ff13806df80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:37.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:37.765+0000 7ff13ca00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7ff1380d0ee0 con 0x7ff138145e40 2024-03-31T15:09:37.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:37.765+0000 7ff13ca00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7ff1380ce9e0 con 0x7ff13806fd80 2024-03-31T15:09:37.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:37.765+0000 7ff13ca00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7ff1380b50c0 con 0x7ff13806f920 2024-03-31T15:09:37.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:37.765+0000 7ff136800640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff138145e40 0x7ff13806df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:37.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:37.765+0000 7ff136800640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff138145e40 0x7ff13806df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59676/0 (socket says 172.21.15.16:59676) 2024-03-31T15:09:37.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:37.765+0000 7ff136800640 1 -- 172.21.15.16:0/1871037567 learned_addr learned my addr 172.21.15.16:0/1871037567 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:09:37.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:37.765+0000 7ff12d400640 1 --2- 172.21.15.16:0/1871037567 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff13806fd80 0x7ff1380b57b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:37.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:37.765+0000 7ff135e00640 1 --2- 172.21.15.16:0/1871037567 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff13806f920 0x7ff1380b5fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:37.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:37.765+0000 7ff136800640 1 -- 172.21.15.16:0/1871037567 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff13806f920 msgr2=0x7ff1380b5fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:37.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:37.765+0000 7ff136800640 1 --2- 172.21.15.16:0/1871037567 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff13806f920 0x7ff1380b5fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:37.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:37.765+0000 7ff136800640 1 -- 172.21.15.16:0/1871037567 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff13806fd80 msgr2=0x7ff1380b57b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:37.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:37.765+0000 7ff136800640 1 --2- 172.21.15.16:0/1871037567 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff13806fd80 0x7ff1380b57b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:37.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:37.765+0000 7ff136800640 1 -- 172.21.15.16:0/1871037567 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff13806d7c0 con 0x7ff138145e40 2024-03-31T15:09:37.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:37.765+0000 7ff12d400640 1 --2- 172.21.15.16:0/1871037567 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff13806fd80 0x7ff1380b57b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:09:37.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:37.765+0000 7ff135e00640 1 --2- 172.21.15.16:0/1871037567 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff13806f920 0x7ff1380b5fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:09:37.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:37.765+0000 7ff136800640 1 --2- 172.21.15.16:0/1871037567 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff138145e40 0x7ff13806df80 secure :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 crypto rx=0x7ff130009d90 tx=0x7ff130019710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:09:37.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:37.765+0000 7ff12ea00640 1 -- 172.21.15.16:0/1871037567 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1300bd020 con 0x7ff138145e40 2024-03-31T15:09:37.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:37.765+0000 7ff12ea00640 1 -- 172.21.15.16:0/1871037567 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff1300b8670 con 0x7ff138145e40 2024-03-31T15:09:37.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:37.765+0000 7ff13ca00640 1 -- 172.21.15.16:0/1871037567 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff1380b6650 con 0x7ff138145e40 2024-03-31T15:09:37.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:37.765+0000 7ff12ea00640 1 -- 172.21.15.16:0/1871037567 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1300b2040 con 0x7ff138145e40 2024-03-31T15:09:37.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:37.765+0000 7ff13ca00640 1 -- 172.21.15.16:0/1871037567 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff1380b6c10 con 0x7ff138145e40 2024-03-31T15:09:37.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:37.769+0000 7ff13ca00640 1 -- 172.21.15.16:0/1871037567 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff138069950 con 0x7ff138145e40 2024-03-31T15:09:37.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:37.769+0000 7ff12ea00640 1 -- 172.21.15.16:0/1871037567 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7ff1300c0020 con 0x7ff138145e40 2024-03-31T15:09:37.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:37.769+0000 7ff12ea00640 1 --2- 172.21.15.16:0/1871037567 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7ff114068380 0x7ff11406a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:37.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:37.769+0000 7ff12ea00640 1 -- 172.21.15.16:0/1871037567 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(70..70 src has 1..70) ==== 5882+0+0 (secure 0 0 0) 0x7ff1301218d0 con 0x7ff138145e40 2024-03-31T15:09:37.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:37.769+0000 7ff135e00640 1 --2- 172.21.15.16:0/1871037567 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7ff114068380 0x7ff11406a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:37.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:37.769+0000 7ff135e00640 1 --2- 172.21.15.16:0/1871037567 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7ff114068380 0x7ff11406a840 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7ff1280016f0 tx=0x7ff128009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:09:37.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:37.773+0000 7ff12ea00640 1 -- 172.21.15.16:0/1871037567 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7ff1300c7030 con 0x7ff138145e40 2024-03-31T15:09:37.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:37.973+0000 7ff13ca00640 1 -- 172.21.15.16:0/1871037567 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff1380b3b30 con 0x7ff138145e40 2024-03-31T15:09:37.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:37.973+0000 7ff12ea00640 1 -- 172.21.15.16:0/1871037567 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 52 v52) ==== 94+0+5185 (secure 0 0 0) 0x7ff130016560 con 0x7ff138145e40 2024-03-31T15:09:37.965 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:09:37.966 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":52,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6836/4188793506","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":4188793506},{"type":"v1","addr":"172.21.15.89:6836","nonce":4188793506}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":40},{"gid":5942,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/4105657676","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6835","nonce":4105657676},{"type":"v1","addr":"172.21.15.89:6837","nonce":4105657676}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":40},{"gid":6592,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6833/3986752851","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":3986752851},{"type":"v1","addr":"172.21.15.16:6833","nonce":3986752851}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":39}],"filesystems":[{"mdsmap":{"epoch":52,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-31T15:07:56.938140+0000","modified":"2024-03-31T15:09:01.267292+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":68,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6829},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6829":{"gid":6829,"name":"a","rank":0,"incarnation":47,"state":"up:active","state_seq":16,"addr":"172.21.15.16:6835/2645389320","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":2645389320},{"type":"v1","addr":"172.21.15.16:6835","nonce":2645389320}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6829,"qdb_cluster":[6829]},"id":5}]} 2024-03-31T15:09:37.966 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 52 2024-03-31T15:09:37.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:37.977+0000 7ff13ca00640 1 -- 172.21.15.16:0/1871037567 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7ff114068380 msgr2=0x7ff11406a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:37.968 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:37.977+0000 7ff13ca00640 1 --2- 172.21.15.16:0/1871037567 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7ff114068380 0x7ff11406a840 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7ff1280016f0 tx=0x7ff128009040 comp rx=0 tx=0).stop 2024-03-31T15:09:37.968 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:37.977+0000 7ff13ca00640 1 -- 172.21.15.16:0/1871037567 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff138145e40 msgr2=0x7ff13806df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:37.968 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:37.977+0000 7ff13ca00640 1 --2- 172.21.15.16:0/1871037567 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff138145e40 0x7ff13806df80 secure :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 crypto rx=0x7ff130009d90 tx=0x7ff130019710 comp rx=0 tx=0).stop 2024-03-31T15:09:37.968 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:37.977+0000 7ff13ca00640 1 -- 172.21.15.16:0/1871037567 shutdown_connections 2024-03-31T15:09:37.968 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:37.977+0000 7ff13ca00640 1 --2- 172.21.15.16:0/1871037567 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7ff114068380 0x7ff11406a840 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:37.968 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:37.977+0000 7ff13ca00640 1 --2- 172.21.15.16:0/1871037567 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff138145e40 0x7ff13806df80 unknown :-1 s=CLOSED pgs=1014 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:37.968 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:37.977+0000 7ff13ca00640 1 --2- 172.21.15.16:0/1871037567 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff13806fd80 0x7ff1380b57b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:37.968 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:37.977+0000 7ff13ca00640 1 --2- 172.21.15.16:0/1871037567 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff13806f920 0x7ff1380b5fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:37.969 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:37.977+0000 7ff13ca00640 1 -- 172.21.15.16:0/1871037567 >> 172.21.15.16:0/1871037567 conn(0x7ff138067330 msgr2=0x7ff1380cb950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:09:37.969 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:37.977+0000 7ff13ca00640 1 -- 172.21.15.16:0/1871037567 shutdown_connections 2024-03-31T15:09:37.969 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:37.977+0000 7ff13ca00640 1 -- 172.21.15.16:0/1871037567 wait complete. 2024-03-31T15:09:37.987 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-31T15:09:38.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:38.137+0000 7fabd4400640 1 Processor -- start 2024-03-31T15:09:38.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:38.137+0000 7fabd4400640 1 -- start start 2024-03-31T15:09:38.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:38.137+0000 7fabd4400640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fabcc15f650 0x7fabcc06f920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:38.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:38.137+0000 7fabd4400640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fabcc0cbe90 con 0x7fabcc06fe60 2024-03-31T15:09:38.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:38.137+0000 7fabd4400640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fabcc0cc170 con 0x7fabcc06dbc0 2024-03-31T15:09:38.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:38.137+0000 7fabd4400640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7fabcc0702d0 con 0x7fabcc15f650 2024-03-31T15:09:38.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:38.137+0000 7fabd2400640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fabcc06fe60 0x7fabcc162180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:55172/0 (socket says 172.21.15.16:55172) 2024-03-31T15:09:38.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:38.137+0000 7fabd2400640 1 -- 172.21.15.16:0/537847774 learned_addr learned my addr 172.21.15.16:0/537847774 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:09:38.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:38.137+0000 7fabd1000640 1 --2- 172.21.15.16:0/537847774 >> v2:172.21.15.89:3300/0 conn(0x7fabcc15f650 0x7fabcc06f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:38.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:38.137+0000 7fabcbe00640 1 -- 172.21.15.16:0/537847774 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3251689536 0 0) 0x7fabcc0cbe90 con 0x7fabcc06fe60 2024-03-31T15:09:38.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:38.137+0000 7fabcbe00640 1 -- 172.21.15.16:0/537847774 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fabb4003530 con 0x7fabcc06fe60 2024-03-31T15:09:38.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:38.137+0000 7fabcbe00640 1 -- 172.21.15.16:0/537847774 <== mon.1 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1449727818 0 0) 0x7fabcc0cc170 con 0x7fabcc06dbc0 2024-03-31T15:09:38.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:38.137+0000 7fabcbe00640 1 -- 172.21.15.16:0/537847774 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fabcc0cbe90 con 0x7fabcc06dbc0 2024-03-31T15:09:38.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:38.137+0000 7fabd1000640 1 -- 172.21.15.16:0/537847774 >> v1:172.21.15.89:6789/0 conn(0x7fabcc06dbc0 legacy=0x7fabcc0cb440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:38.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:38.137+0000 7fabd1000640 1 -- 172.21.15.16:0/537847774 >> v1:172.21.15.16:6789/0 conn(0x7fabcc06fe60 legacy=0x7fabcc162180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:38.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:38.137+0000 7fabd1000640 1 -- 172.21.15.16:0/537847774 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fabcc15ee50 con 0x7fabcc15f650 2024-03-31T15:09:38.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:38.141+0000 7fabd1000640 1 --2- 172.21.15.16:0/537847774 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fabcc15f650 0x7fabcc06f920 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7fabb80097c0 tx=0x7fabb801c410 comp rx=0 tx=0).ready entity=mon.1 client_cookie=96a1a948445d6e83 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:09:38.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:38.141+0000 7fabcbe00640 1 -- 172.21.15.16:0/537847774 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabb80b0070 con 0x7fabcc15f650 2024-03-31T15:09:38.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:38.141+0000 7fabcbe00640 1 -- 172.21.15.16:0/537847774 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fabb80ab600 con 0x7fabcc15f650 2024-03-31T15:09:38.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:38.141+0000 7fabcbe00640 1 -- 172.21.15.16:0/537847774 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabb80ab9d0 con 0x7fabcc15f650 2024-03-31T15:09:38.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:38.141+0000 7fabd4400640 1 -- 172.21.15.16:0/537847774 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fabcc15f650 msgr2=0x7fabcc06f920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:38.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:38.141+0000 7fabd4400640 1 --2- 172.21.15.16:0/537847774 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fabcc15f650 0x7fabcc06f920 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7fabb80097c0 tx=0x7fabb801c410 comp rx=0 tx=0).stop 2024-03-31T15:09:38.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:38.141+0000 7fabd4400640 1 -- 172.21.15.16:0/537847774 shutdown_connections 2024-03-31T15:09:38.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:38.141+0000 7fabd4400640 1 --2- 172.21.15.16:0/537847774 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fabcc15f650 0x7fabcc06f920 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:38.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:38.141+0000 7fabd4400640 1 -- 172.21.15.16:0/537847774 >> 172.21.15.16:0/537847774 conn(0x7fabcc067330 msgr2=0x7fabcc067740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:09:38.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:38.141+0000 7fabd4400640 1 -- 172.21.15.16:0/537847774 shutdown_connections 2024-03-31T15:09:38.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:38.141+0000 7fabd4400640 1 -- 172.21.15.16:0/537847774 wait complete. 2024-03-31T15:09:38.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:38.145+0000 7fabd4400640 1 Processor -- start 2024-03-31T15:09:38.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:38.145+0000 7fabd4400640 1 -- start start 2024-03-31T15:09:38.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:38.145+0000 7fabd4400640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fabcc06dbc0 0x7fabcc14a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:38.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:38.145+0000 7fabd4400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fabcc06fe60 0x7fabcc157a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:38.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:38.145+0000 7fabd4400640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fabcc15f650 0x7fabcc0b67c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:38.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:38.145+0000 7fabd4400640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fabcc162ca0 con 0x7fabcc06fe60 2024-03-31T15:09:38.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:38.145+0000 7fabd4400640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fabcc06dfa0 con 0x7fabcc15f650 2024-03-31T15:09:38.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:38.145+0000 7fabd4400640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fabcc06c950 con 0x7fabcc06dbc0 2024-03-31T15:09:38.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:38.145+0000 7fabd1000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fabcc06fe60 0x7fabcc157a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:38.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:38.145+0000 7fabd1000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fabcc06fe60 0x7fabcc157a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59692/0 (socket says 172.21.15.16:59692) 2024-03-31T15:09:38.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:38.145+0000 7fabd1000640 1 -- 172.21.15.16:0/19538235 learned_addr learned my addr 172.21.15.16:0/19538235 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:09:38.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:38.145+0000 7fabd1a00640 1 --2- 172.21.15.16:0/19538235 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fabcc06dbc0 0x7fabcc14a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:38.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:38.145+0000 7fabd2400640 1 --2- 172.21.15.16:0/19538235 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fabcc15f650 0x7fabcc0b67c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:38.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:38.145+0000 7fabd1000640 1 -- 172.21.15.16:0/19538235 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fabcc06dbc0 msgr2=0x7fabcc14a7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:38.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:38.145+0000 7fabd1000640 1 --2- 172.21.15.16:0/19538235 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fabcc06dbc0 0x7fabcc14a7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:38.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:38.145+0000 7fabd1000640 1 -- 172.21.15.16:0/19538235 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fabcc15f650 msgr2=0x7fabcc0b67c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:38.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:38.145+0000 7fabd1000640 1 --2- 172.21.15.16:0/19538235 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fabcc15f650 0x7fabcc0b67c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:38.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:38.145+0000 7fabd1000640 1 -- 172.21.15.16:0/19538235 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fabcc156970 con 0x7fabcc06fe60 2024-03-31T15:09:38.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:38.145+0000 7fabd2400640 1 --2- 172.21.15.16:0/19538235 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fabcc15f650 0x7fabcc0b67c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:09:38.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:38.145+0000 7fabd1000640 1 --2- 172.21.15.16:0/19538235 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fabcc06fe60 0x7fabcc157a10 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 crypto rx=0x7fabb801a040 tx=0x7fabb80191b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:09:38.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:38.145+0000 7fabc9600640 1 -- 172.21.15.16:0/19538235 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabb80ba020 con 0x7fabcc06fe60 2024-03-31T15:09:38.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:38.145+0000 7fabc9600640 1 -- 172.21.15.16:0/19538235 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fabb80b5460 con 0x7fabcc06fe60 2024-03-31T15:09:38.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:38.145+0000 7fabd4400640 1 -- 172.21.15.16:0/19538235 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fabcc156c50 con 0x7fabcc06fe60 2024-03-31T15:09:38.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:38.145+0000 7fabc9600640 1 -- 172.21.15.16:0/19538235 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabb80b0040 con 0x7fabcc06fe60 2024-03-31T15:09:38.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:38.145+0000 7fabd4400640 1 -- 172.21.15.16:0/19538235 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fabcc157210 con 0x7fabcc06fe60 2024-03-31T15:09:38.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:38.149+0000 7fabd4400640 1 -- 172.21.15.16:0/19538235 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fabcc069950 con 0x7fabcc06fe60 2024-03-31T15:09:38.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:38.149+0000 7fabc9600640 1 -- 172.21.15.16:0/19538235 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fabb80bd020 con 0x7fabcc06fe60 2024-03-31T15:09:38.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:38.149+0000 7fabc9600640 1 --2- 172.21.15.16:0/19538235 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fabac068380 0x7fabac06a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:38.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:38.149+0000 7fabc9600640 1 -- 172.21.15.16:0/19538235 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(70..70 src has 1..70) ==== 5882+0+0 (secure 0 0 0) 0x7fabb8120c50 con 0x7fabcc06fe60 2024-03-31T15:09:38.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:38.149+0000 7fabd1a00640 1 --2- 172.21.15.16:0/19538235 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fabac068380 0x7fabac06a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:38.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:38.149+0000 7fabd1a00640 1 --2- 172.21.15.16:0/19538235 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fabac068380 0x7fabac06a840 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7fabc0001720 tx=0x7fabc0009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:09:38.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:38.157+0000 7fabc9600640 1 -- 172.21.15.16:0/19538235 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fabb80c4030 con 0x7fabcc06fe60 2024-03-31T15:09:38.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:38.365+0000 7fabd4400640 1 -- 172.21.15.16:0/19538235 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fabcc14ad10 con 0x7fabcc06fe60 2024-03-31T15:09:38.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:38.365+0000 7fabc9600640 1 -- 172.21.15.16:0/19538235 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 52 v52) ==== 94+0+5185 (secure 0 0 0) 0x7fabb8016090 con 0x7fabcc06fe60 2024-03-31T15:09:38.357 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:09:38.357 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":52,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6836/4188793506","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":4188793506},{"type":"v1","addr":"172.21.15.89:6836","nonce":4188793506}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":40},{"gid":5942,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/4105657676","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6835","nonce":4105657676},{"type":"v1","addr":"172.21.15.89:6837","nonce":4105657676}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":40},{"gid":6592,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6833/3986752851","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":3986752851},{"type":"v1","addr":"172.21.15.16:6833","nonce":3986752851}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":39}],"filesystems":[{"mdsmap":{"epoch":52,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-31T15:07:56.938140+0000","modified":"2024-03-31T15:09:01.267292+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":68,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6829},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6829":{"gid":6829,"name":"a","rank":0,"incarnation":47,"state":"up:active","state_seq":16,"addr":"172.21.15.16:6835/2645389320","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":2645389320},{"type":"v1","addr":"172.21.15.16:6835","nonce":2645389320}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6829,"qdb_cluster":[6829]},"id":5}]} 2024-03-31T15:09:38.359 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 52 2024-03-31T15:09:38.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:38.365+0000 7fabd4400640 1 -- 172.21.15.16:0/19538235 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fabac068380 msgr2=0x7fabac06a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:38.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:38.365+0000 7fabd4400640 1 --2- 172.21.15.16:0/19538235 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fabac068380 0x7fabac06a840 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7fabc0001720 tx=0x7fabc0009040 comp rx=0 tx=0).stop 2024-03-31T15:09:38.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:38.365+0000 7fabd4400640 1 -- 172.21.15.16:0/19538235 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fabcc06fe60 msgr2=0x7fabcc157a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:38.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:38.365+0000 7fabd4400640 1 --2- 172.21.15.16:0/19538235 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fabcc06fe60 0x7fabcc157a10 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 crypto rx=0x7fabb801a040 tx=0x7fabb80191b0 comp rx=0 tx=0).stop 2024-03-31T15:09:38.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:38.365+0000 7fabd4400640 1 -- 172.21.15.16:0/19538235 shutdown_connections 2024-03-31T15:09:38.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:38.365+0000 7fabd4400640 1 --2- 172.21.15.16:0/19538235 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fabac068380 0x7fabac06a840 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:38.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:38.365+0000 7fabd4400640 1 --2- 172.21.15.16:0/19538235 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fabcc15f650 0x7fabcc0b67c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:38.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:38.365+0000 7fabd4400640 1 --2- 172.21.15.16:0/19538235 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fabcc06fe60 0x7fabcc157a10 unknown :-1 s=CLOSED pgs=1016 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:38.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:38.365+0000 7fabd4400640 1 --2- 172.21.15.16:0/19538235 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fabcc06dbc0 0x7fabcc14a7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:38.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:38.365+0000 7fabd4400640 1 -- 172.21.15.16:0/19538235 >> 172.21.15.16:0/19538235 conn(0x7fabcc067330 msgr2=0x7fabcc15c7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:09:38.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:38.365+0000 7fabd4400640 1 -- 172.21.15.16:0/19538235 shutdown_connections 2024-03-31T15:09:38.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:38.365+0000 7fabd4400640 1 -- 172.21.15.16:0/19538235 wait complete. 2024-03-31T15:09:38.382 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-31T15:09:38.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:38.529+0000 7f1217c00640 1 Processor -- start 2024-03-31T15:09:38.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:38.529+0000 7f1217c00640 1 -- start start 2024-03-31T15:09:38.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:38.533+0000 7f1217c00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f1210145e40 0x7f12100cc740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:38.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:38.533+0000 7f1217c00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f12100ccc80 0x7f121006f920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:38.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:38.533+0000 7f1217c00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f121006fe60 0x7f121015fdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:38.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:38.533+0000 7f1217c00640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f12100d06d0 con 0x7f1210145e40 2024-03-31T15:09:38.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:38.533+0000 7f1217c00640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f12100d0920 con 0x7f121006fe60 2024-03-31T15:09:38.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:38.533+0000 7f1217c00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f12100d0c00 con 0x7f12100ccc80 2024-03-31T15:09:38.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:38.533+0000 7f1215200640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f1210145e40 0x7f12100cc740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:38.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:38.533+0000 7f1215200640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f1210145e40 0x7f12100cc740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59702/0 (socket says 172.21.15.16:59702) 2024-03-31T15:09:38.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:38.533+0000 7f1215200640 1 -- 172.21.15.16:0/3263903857 learned_addr learned my addr 172.21.15.16:0/3263903857 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:09:38.525 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:38.533+0000 7f120fe00640 1 --2- 172.21.15.16:0/3263903857 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f12100ccc80 0x7f121006f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:38.525 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:38.533+0000 7f1215c00640 1 --2- 172.21.15.16:0/3263903857 >> v2:172.21.15.89:3300/0 conn(0x7f121006fe60 0x7f121015fdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:38.525 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:38.533+0000 7f1215200640 1 -- 172.21.15.16:0/3263903857 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f12100ccc80 msgr2=0x7f121006f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:09:38.525 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:38.533+0000 7f1215200640 1 --2- 172.21.15.16:0/3263903857 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f12100ccc80 0x7f121006f920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:38.525 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:38.533+0000 7f1215200640 1 -- 172.21.15.16:0/3263903857 >> v2:172.21.15.89:3300/0 conn(0x7f121006fe60 msgr2=0x7f121015fdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:09:38.525 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:38.533+0000 7f1215200640 1 --2- 172.21.15.16:0/3263903857 >> v2:172.21.15.89:3300/0 conn(0x7f121006fe60 0x7f121015fdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:38.525 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:38.533+0000 7f1215200640 1 -- 172.21.15.16:0/3263903857 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12100d1810 con 0x7f1210145e40 2024-03-31T15:09:38.525 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:38.533+0000 7f1215c00640 1 --2- 172.21.15.16:0/3263903857 >> v2:172.21.15.89:3300/0 conn(0x7f121006fe60 0x7f121015fdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:09:38.525 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:38.533+0000 7f120fe00640 1 --2- 172.21.15.16:0/3263903857 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f12100ccc80 0x7f121006f920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:09:38.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:38.537+0000 7f1215200640 1 --2- 172.21.15.16:0/3263903857 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1210145e40 0x7f12100cc740 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 crypto rx=0x7f12100d3170 tx=0x7f120401c260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f6012e53b7102ec2 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:09:38.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:38.537+0000 7f120f400640 1 -- 172.21.15.16:0/3263903857 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12040b0070 con 0x7f1210145e40 2024-03-31T15:09:38.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:38.537+0000 7f120f400640 1 -- 172.21.15.16:0/3263903857 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f12040ab600 con 0x7f1210145e40 2024-03-31T15:09:38.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:38.537+0000 7f120f400640 1 -- 172.21.15.16:0/3263903857 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12040ab9d0 con 0x7f1210145e40 2024-03-31T15:09:38.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:38.537+0000 7f1217c00640 1 -- 172.21.15.16:0/3263903857 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1210145e40 msgr2=0x7f12100cc740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:38.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:38.537+0000 7f1217c00640 1 --2- 172.21.15.16:0/3263903857 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1210145e40 0x7f12100cc740 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 crypto rx=0x7f12100d3170 tx=0x7f120401c260 comp rx=0 tx=0).stop 2024-03-31T15:09:38.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:38.537+0000 7f1217c00640 1 -- 172.21.15.16:0/3263903857 shutdown_connections 2024-03-31T15:09:38.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:38.537+0000 7f1217c00640 1 --2- 172.21.15.16:0/3263903857 >> v2:172.21.15.89:3300/0 conn(0x7f121006fe60 0x7f121015fdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:38.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:38.537+0000 7f1217c00640 1 --2- 172.21.15.16:0/3263903857 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f12100ccc80 0x7f121006f920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:38.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:38.537+0000 7f1217c00640 1 --2- 172.21.15.16:0/3263903857 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1210145e40 0x7f12100cc740 unknown :-1 s=CLOSED pgs=1017 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:38.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:38.537+0000 7f1217c00640 1 -- 172.21.15.16:0/3263903857 >> 172.21.15.16:0/3263903857 conn(0x7f1210067330 msgr2=0x7f1210067740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:09:38.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:38.537+0000 7f1217c00640 1 -- 172.21.15.16:0/3263903857 shutdown_connections 2024-03-31T15:09:38.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:38.537+0000 7f1217c00640 1 -- 172.21.15.16:0/3263903857 wait complete. 2024-03-31T15:09:38.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:38.537+0000 7f1217c00640 1 Processor -- start 2024-03-31T15:09:38.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:38.537+0000 7f1217c00640 1 -- start start 2024-03-31T15:09:38.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:38.541+0000 7f1217c00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f121006fe60 0x7f1210159ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:38.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:38.541+0000 7f1217c00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f12100ccc80 0x7f121015df10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:38.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:38.541+0000 7f1217c00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f1210145e40 0x7f121013f570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:38.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:38.541+0000 7f1217c00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f12100b5240 con 0x7f121006fe60 2024-03-31T15:09:38.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:38.541+0000 7f1217c00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f12100b50c0 con 0x7f12100ccc80 2024-03-31T15:09:38.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:38.541+0000 7f1217c00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f12100d2010 con 0x7f1210145e40 2024-03-31T15:09:38.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:38.541+0000 7f1215200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f121006fe60 0x7f1210159ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:38.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:38.541+0000 7f1215200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f121006fe60 0x7f1210159ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59708/0 (socket says 172.21.15.16:59708) 2024-03-31T15:09:38.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:38.541+0000 7f1215200640 1 -- 172.21.15.16:0/2609795414 learned_addr learned my addr 172.21.15.16:0/2609795414 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:09:38.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:38.541+0000 7f120fe00640 1 --2- 172.21.15.16:0/2609795414 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f12100ccc80 0x7f121015df10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:38.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:38.541+0000 7f1215c00640 1 --2- 172.21.15.16:0/2609795414 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f1210145e40 0x7f121013f570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:38.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:38.541+0000 7f1215200640 1 -- 172.21.15.16:0/2609795414 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f1210145e40 msgr2=0x7f121013f570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:38.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:38.541+0000 7f1215200640 1 --2- 172.21.15.16:0/2609795414 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f1210145e40 0x7f121013f570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:38.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:38.541+0000 7f1215200640 1 -- 172.21.15.16:0/2609795414 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f12100ccc80 msgr2=0x7f121015df10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:38.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:38.541+0000 7f1215200640 1 --2- 172.21.15.16:0/2609795414 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f12100ccc80 0x7f121015df10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:38.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:38.541+0000 7f1215200640 1 -- 172.21.15.16:0/2609795414 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f121015e610 con 0x7f121006fe60 2024-03-31T15:09:38.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:38.541+0000 7f1215200640 1 --2- 172.21.15.16:0/2609795414 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f121006fe60 0x7f1210159ac0 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 crypto rx=0x7f12040ae040 tx=0x7f120401aaa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:09:38.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:38.541+0000 7f120cc00640 1 -- 172.21.15.16:0/2609795414 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12040ba020 con 0x7f121006fe60 2024-03-31T15:09:38.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:38.541+0000 7f120cc00640 1 -- 172.21.15.16:0/2609795414 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f12040b6640 con 0x7f121006fe60 2024-03-31T15:09:38.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:38.541+0000 7f120cc00640 1 -- 172.21.15.16:0/2609795414 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12040b0040 con 0x7f121006fe60 2024-03-31T15:09:38.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:38.541+0000 7f1217c00640 1 -- 172.21.15.16:0/2609795414 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f121015a150 con 0x7f121006fe60 2024-03-31T15:09:38.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:38.541+0000 7f1217c00640 1 -- 172.21.15.16:0/2609795414 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f121015e8b0 con 0x7f121006fe60 2024-03-31T15:09:38.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:38.541+0000 7f1217c00640 1 -- 172.21.15.16:0/2609795414 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1210069950 con 0x7f121006fe60 2024-03-31T15:09:38.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:38.545+0000 7f120cc00640 1 -- 172.21.15.16:0/2609795414 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f12040bd020 con 0x7f121006fe60 2024-03-31T15:09:38.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:38.545+0000 7f120cc00640 1 --2- 172.21.15.16:0/2609795414 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f11f0068380 0x7f11f006a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:38.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:38.545+0000 7f120cc00640 1 -- 172.21.15.16:0/2609795414 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(70..70 src has 1..70) ==== 5882+0+0 (secure 0 0 0) 0x7f1204120dd0 con 0x7f121006fe60 2024-03-31T15:09:38.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:38.545+0000 7f120fe00640 1 --2- 172.21.15.16:0/2609795414 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f11f0068380 0x7f11f006a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:38.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:38.545+0000 7f120fe00640 1 --2- 172.21.15.16:0/2609795414 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f11f0068380 0x7f11f006a840 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f11fc006fd0 tx=0x7f11fc008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:09:38.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:38.549+0000 7f120cc00640 1 -- 172.21.15.16:0/2609795414 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f12040c4030 con 0x7f121006fe60 2024-03-31T15:09:38.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:38.713+0000 7f1217c00640 1 -- 172.21.15.16:0/2609795414 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f12100b39e0 con 0x7f121006fe60 2024-03-31T15:09:38.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:38.717+0000 7f120cc00640 1 -- 172.21.15.16:0/2609795414 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v70) ==== 74+0+16870 (secure 0 0 0) 0x7f1204122020 con 0x7f121006fe60 2024-03-31T15:09:38.708 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:09:38.708 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":70,"fsid":"286eb5da-fb47-4550-b290-d9519c13b9a5","created":"2024-03-31T15:03:11.438952+0000","modified":"2024-03-31T15:08:18.287907+0000","last_up_change":"2024-03-31T15:03:22.885535+0000","last_in_change":"2024-03-31T15:03:13.941667+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-31T15:03:22.550787+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-03-31T15:07:55.329096+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"66","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-03-31T15:07:55.885919+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"67","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"92a66450-ef19-40d8-ac3d-d760b23b0bda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6801","nonce":932712244}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6803","nonce":932712244}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6807","nonce":932712244}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6805","nonce":932712244}]},"public_addr":"172.21.15.16:6801/932712244","cluster_addr":"172.21.15.16:6803/932712244","heartbeat_back_addr":"172.21.15.16:6807/932712244","heartbeat_front_addr":"172.21.15.16:6805/932712244","state":["exists","up"]},{"osd":1,"uuid":"da9d1f2b-1968-4959-b843-faa231988e92","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6810","nonce":1398737533}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6811","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6813","nonce":1398737533}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6821","nonce":1398737533}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6815","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6816","nonce":1398737533}]},"public_addr":"172.21.15.16:6810/1398737533","cluster_addr":"172.21.15.16:6813/1398737533","heartbeat_back_addr":"172.21.15.16:6821/1398737533","heartbeat_front_addr":"172.21.15.16:6816/1398737533","state":["exists","up"]},{"osd":2,"uuid":"01a73662-9a21-4b1f-bcbb-42b9636a4638","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6819","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6822","nonce":3018526646}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6825","nonce":3018526646}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6831","nonce":3018526646}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6827","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6829","nonce":3018526646}]},"public_addr":"172.21.15.16:6822/3018526646","cluster_addr":"172.21.15.16:6825/3018526646","heartbeat_back_addr":"172.21.15.16:6831/3018526646","heartbeat_front_addr":"172.21.15.16:6829/3018526646","state":["exists","up"]},{"osd":3,"uuid":"c7771fd3-e18e-4500-ac38-23cf10ebcd75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6809","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6812","nonce":3360352974}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6817","nonce":3360352974}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6828","nonce":3360352974}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6823","nonce":3360352974}]},"public_addr":"172.21.15.16:6812/3360352974","cluster_addr":"172.21.15.16:6817/3360352974","heartbeat_back_addr":"172.21.15.16:6828/3360352974","heartbeat_front_addr":"172.21.15.16:6823/3360352974","state":["exists","up"]},{"osd":4,"uuid":"88260730-1482-4fbd-a40f-cb1124e9df4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6801","nonce":1856844856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6803","nonce":1856844856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6807","nonce":1856844856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6805","nonce":1856844856}]},"public_addr":"172.21.15.89:6801/1856844856","cluster_addr":"172.21.15.89:6803/1856844856","heartbeat_back_addr":"172.21.15.89:6807/1856844856","heartbeat_front_addr":"172.21.15.89:6805/1856844856","state":["exists","up"]},{"osd":5,"uuid":"4cb2c00c-fae0-4dd8-9d27-ec4f0093ccf7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6814","nonce":50395527}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6817","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6820","nonce":50395527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6831","nonce":50395527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6823","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6826","nonce":50395527}]},"public_addr":"172.21.15.89:6814/50395527","cluster_addr":"172.21.15.89:6820/50395527","heartbeat_back_addr":"172.21.15.89:6831/50395527","heartbeat_front_addr":"172.21.15.89:6826/50395527","state":["exists","up"]},{"osd":6,"uuid":"81ce0c19-2554-4cae-acbe-66a05e9e9b9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6809","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6812","nonce":3470471752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6815","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6818","nonce":3470471752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6827","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6829","nonce":3470471752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6821","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6824","nonce":3470471752}]},"public_addr":"172.21.15.89:6812/3470471752","cluster_addr":"172.21.15.89:6818/3470471752","heartbeat_back_addr":"172.21.15.89:6829/3470471752","heartbeat_front_addr":"172.21.15.89:6824/3470471752","state":["exists","up"]},{"osd":7,"uuid":"85e4313b-d37d-442e-8042-153bf20a847d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6811","nonce":1796778630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6813","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6816","nonce":1796778630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6825","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6828","nonce":1796778630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6819","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6822","nonce":1796778630}]},"public_addr":"172.21.15.89:6811/1796778630","cluster_addr":"172.21.15.89:6816/1796778630","heartbeat_back_addr":"172.21.15.89:6828/1796778630","heartbeat_front_addr":"172.21.15.89:6822/1796778630","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.439088+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.529970+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.680512+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:21.044161+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.582111+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.536558+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.409522+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.590229+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"11.4","mappings":[{"from":6,"to":0}]},{"pgid":"11.10","mappings":[{"from":6,"to":1}]},{"pgid":"11.3e","mappings":[{"from":6,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:6835/2555930820":"2024-04-01T15:08:10.537891+0000","172.21.15.16:6834/2555930820":"2024-04-01T15:08:10.537891+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-31T15:09:38.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:38.717+0000 7f1217c00640 1 -- 172.21.15.16:0/2609795414 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f11f0068380 msgr2=0x7f11f006a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:38.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:38.717+0000 7f1217c00640 1 --2- 172.21.15.16:0/2609795414 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f11f0068380 0x7f11f006a840 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f11fc006fd0 tx=0x7f11fc008040 comp rx=0 tx=0).stop 2024-03-31T15:09:38.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:38.717+0000 7f1217c00640 1 -- 172.21.15.16:0/2609795414 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f121006fe60 msgr2=0x7f1210159ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:38.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:38.717+0000 7f1217c00640 1 --2- 172.21.15.16:0/2609795414 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f121006fe60 0x7f1210159ac0 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 crypto rx=0x7f12040ae040 tx=0x7f120401aaa0 comp rx=0 tx=0).stop 2024-03-31T15:09:38.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:38.717+0000 7f1217c00640 1 -- 172.21.15.16:0/2609795414 shutdown_connections 2024-03-31T15:09:38.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:38.717+0000 7f1217c00640 1 --2- 172.21.15.16:0/2609795414 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f11f0068380 0x7f11f006a840 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:38.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:38.717+0000 7f1217c00640 1 --2- 172.21.15.16:0/2609795414 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f1210145e40 0x7f121013f570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:38.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:38.717+0000 7f1217c00640 1 --2- 172.21.15.16:0/2609795414 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f12100ccc80 0x7f121015df10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:38.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:38.717+0000 7f1217c00640 1 --2- 172.21.15.16:0/2609795414 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f121006fe60 0x7f1210159ac0 unknown :-1 s=CLOSED pgs=1018 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:38.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:38.717+0000 7f1217c00640 1 -- 172.21.15.16:0/2609795414 >> 172.21.15.16:0/2609795414 conn(0x7f1210067330 msgr2=0x7f12100b59d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:09:38.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:38.717+0000 7f1217c00640 1 -- 172.21.15.16:0/2609795414 shutdown_connections 2024-03-31T15:09:38.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:38.717+0000 7f1217c00640 1 -- 172.21.15.16:0/2609795414 wait complete. 2024-03-31T15:09:38.729 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-31T15:09:38.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:38.877+0000 7fa535400640 1 Processor -- start 2024-03-31T15:09:38.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:38.877+0000 7fa535400640 1 -- start start 2024-03-31T15:09:38.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:38.881+0000 7fa535400640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fa53006dbc0 0x7fa53015f650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:38.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:38.881+0000 7fa535400640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fa530070030 0x7fa530162180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:38.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:38.881+0000 7fa535400640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa5300cbe90 con 0x7fa5300cb440 2024-03-31T15:09:38.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:38.881+0000 7fa535400640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7fa53015fb90 con 0x7fa530070030 2024-03-31T15:09:38.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:38.881+0000 7fa535400640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fa53015f1c0 con 0x7fa53006dbc0 2024-03-31T15:09:38.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:38.881+0000 7fa52f200640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fa530070030 0x7fa530162180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:38.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:38.881+0000 7fa52f200640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fa530070030 0x7fa530162180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59712/0 (socket says 172.21.15.16:59712) 2024-03-31T15:09:38.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:38.881+0000 7fa52f200640 1 -- 172.21.15.16:0/2112183681 learned_addr learned my addr 172.21.15.16:0/2112183681 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:09:38.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:38.881+0000 7fa52e800640 1 --2- 172.21.15.16:0/2112183681 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fa53006dbc0 0x7fa53015f650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:38.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:38.881+0000 7fa52f200640 1 -- 172.21.15.16:0/2112183681 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fa53006dbc0 msgr2=0x7fa53015f650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:09:38.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:38.881+0000 7fa52f200640 1 --2- 172.21.15.16:0/2112183681 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fa53006dbc0 0x7fa53015f650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:38.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:38.881+0000 7fa52f200640 1 -- 172.21.15.16:0/2112183681 >> v1:172.21.15.89:6789/0 conn(0x7fa5300cb440 legacy=0x7fa53006f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:09:38.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:38.881+0000 7fa52f200640 1 -- 172.21.15.16:0/2112183681 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5300cc170 con 0x7fa530070030 2024-03-31T15:09:38.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:38.881+0000 7fa52e800640 1 --2- 172.21.15.16:0/2112183681 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fa53006dbc0 0x7fa53015f650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:09:38.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:38.885+0000 7fa52f200640 1 --2- 172.21.15.16:0/2112183681 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa530070030 0x7fa530162180 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7fa528009e50 tx=0x7fa52801c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c18e654e51038993 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:09:38.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:38.885+0000 7fa52de00640 1 -- 172.21.15.16:0/2112183681 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5280a5740 con 0x7fa530070030 2024-03-31T15:09:38.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:38.885+0000 7fa52de00640 1 -- 172.21.15.16:0/2112183681 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa5280a58e0 con 0x7fa530070030 2024-03-31T15:09:38.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:38.885+0000 7fa52de00640 1 -- 172.21.15.16:0/2112183681 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5280a5cb0 con 0x7fa530070030 2024-03-31T15:09:38.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:38.885+0000 7fa535400640 1 -- 172.21.15.16:0/2112183681 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa530070030 msgr2=0x7fa530162180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:38.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:38.885+0000 7fa535400640 1 --2- 172.21.15.16:0/2112183681 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa530070030 0x7fa530162180 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7fa528009e50 tx=0x7fa52801c970 comp rx=0 tx=0).stop 2024-03-31T15:09:38.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:38.885+0000 7fa535400640 1 -- 172.21.15.16:0/2112183681 shutdown_connections 2024-03-31T15:09:38.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:38.885+0000 7fa535400640 1 --2- 172.21.15.16:0/2112183681 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa530070030 0x7fa530162180 unknown :-1 s=CLOSED pgs=1019 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:38.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:38.885+0000 7fa535400640 1 --2- 172.21.15.16:0/2112183681 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fa53006dbc0 0x7fa53015f650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:38.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:38.885+0000 7fa535400640 1 -- 172.21.15.16:0/2112183681 >> 172.21.15.16:0/2112183681 conn(0x7fa530067330 msgr2=0x7fa530067740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:09:38.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:38.885+0000 7fa535400640 1 -- 172.21.15.16:0/2112183681 shutdown_connections 2024-03-31T15:09:38.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:38.885+0000 7fa535400640 1 -- 172.21.15.16:0/2112183681 wait complete. 2024-03-31T15:09:38.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:38.885+0000 7fa535400640 1 Processor -- start 2024-03-31T15:09:38.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:38.885+0000 7fa535400640 1 -- start start 2024-03-31T15:09:38.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:38.885+0000 7fa535400640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa53006dbc0 0x7fa53015c240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:38.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:38.885+0000 7fa535400640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fa530070030 0x7fa5300b7690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:38.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:38.889+0000 7fa535400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa5300cb440 0x7fa53014b710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:38.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:38.889+0000 7fa535400640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fa53006cad0 con 0x7fa5300cb440 2024-03-31T15:09:38.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:38.889+0000 7fa535400640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fa53006c950 con 0x7fa53006dbc0 2024-03-31T15:09:38.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:38.889+0000 7fa535400640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fa53016b520 con 0x7fa530070030 2024-03-31T15:09:38.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:38.889+0000 7fa52f200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa5300cb440 0x7fa53014b710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:38.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:38.889+0000 7fa52f200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa5300cb440 0x7fa53014b710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59718/0 (socket says 172.21.15.16:59718) 2024-03-31T15:09:38.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:38.889+0000 7fa52f200640 1 -- 172.21.15.16:0/1133568642 learned_addr learned my addr 172.21.15.16:0/1133568642 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:09:38.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:38.889+0000 7fa525e00640 1 --2- 172.21.15.16:0/1133568642 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fa530070030 0x7fa5300b7690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:38.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:38.889+0000 7fa52e800640 1 --2- 172.21.15.16:0/1133568642 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa53006dbc0 0x7fa53015c240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:38.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:38.889+0000 7fa52f200640 1 -- 172.21.15.16:0/1133568642 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fa530070030 msgr2=0x7fa5300b7690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:38.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:38.889+0000 7fa52f200640 1 --2- 172.21.15.16:0/1133568642 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fa530070030 0x7fa5300b7690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:38.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:38.889+0000 7fa52f200640 1 -- 172.21.15.16:0/1133568642 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa53006dbc0 msgr2=0x7fa53015c240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:38.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:38.889+0000 7fa52f200640 1 --2- 172.21.15.16:0/1133568642 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa53006dbc0 0x7fa53015c240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:38.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:38.889+0000 7fa52f200640 1 -- 172.21.15.16:0/1133568642 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5300b5c80 con 0x7fa5300cb440 2024-03-31T15:09:38.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:38.889+0000 7fa52f200640 1 --2- 172.21.15.16:0/1133568642 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa5300cb440 0x7fa53014b710 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 crypto rx=0x7fa52801a5a0 tx=0x7fa528019710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:09:38.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:38.889+0000 7fa527400640 1 -- 172.21.15.16:0/1133568642 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5280b8670 con 0x7fa5300cb440 2024-03-31T15:09:38.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:38.889+0000 7fa527400640 1 -- 172.21.15.16:0/1133568642 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa5280be070 con 0x7fa5300cb440 2024-03-31T15:09:38.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:38.889+0000 7fa535400640 1 -- 172.21.15.16:0/1133568642 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa5300b53e0 con 0x7fa5300cb440 2024-03-31T15:09:38.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:38.889+0000 7fa527400640 1 -- 172.21.15.16:0/1133568642 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5280b8a50 con 0x7fa5300cb440 2024-03-31T15:09:38.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:38.889+0000 7fa535400640 1 -- 172.21.15.16:0/1133568642 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa5300b59a0 con 0x7fa5300cb440 2024-03-31T15:09:38.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:38.889+0000 7fa535400640 1 -- 172.21.15.16:0/1133568642 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa530069950 con 0x7fa5300cb440 2024-03-31T15:09:38.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:38.889+0000 7fa527400640 1 -- 172.21.15.16:0/1133568642 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fa5280b8bf0 con 0x7fa5300cb440 2024-03-31T15:09:38.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:38.893+0000 7fa527400640 1 --2- 172.21.15.16:0/1133568642 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fa50c068380 0x7fa50c06a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:38.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:38.893+0000 7fa527400640 1 -- 172.21.15.16:0/1133568642 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(70..70 src has 1..70) ==== 5882+0+0 (secure 0 0 0) 0x7fa528121c80 con 0x7fa5300cb440 2024-03-31T15:09:38.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:38.893+0000 7fa52e800640 1 --2- 172.21.15.16:0/1133568642 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fa50c068380 0x7fa50c06a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:38.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:38.893+0000 7fa52e800640 1 --2- 172.21.15.16:0/1133568642 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fa50c068380 0x7fa50c06a840 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7fa520006fd0 tx=0x7fa520008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:09:38.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:38.897+0000 7fa527400640 1 -- 172.21.15.16:0/1133568642 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fa528016620 con 0x7fa5300cb440 2024-03-31T15:09:39.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:39.197+0000 7fa535400640 1 -- 172.21.15.16:0/1133568642 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7fa5300d13c0 con 0x7fa5300cb440 2024-03-31T15:09:39.187 INFO:tasks.ceph.mon.a.smithi016.stderr:2024-03-31T15:09:39.197+0000 7f7dcbc00640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-31T15:09:39.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:39.197+0000 7fa527400640 1 -- 172.21.15.16:0/1133568642 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v53) ==== 161+0+0 (secure 0 0 0) 0x7fa5280f2250 con 0x7fa5300cb440 2024-03-31T15:09:39.189 INFO:teuthology.orchestra.run.smithi016.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-31T15:09:39.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:39.201+0000 7fa535400640 1 -- 172.21.15.16:0/1133568642 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fa50c068380 msgr2=0x7fa50c06a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:39.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:39.201+0000 7fa535400640 1 --2- 172.21.15.16:0/1133568642 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fa50c068380 0x7fa50c06a840 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7fa520006fd0 tx=0x7fa520008040 comp rx=0 tx=0).stop 2024-03-31T15:09:39.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:39.201+0000 7fa535400640 1 -- 172.21.15.16:0/1133568642 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa5300cb440 msgr2=0x7fa53014b710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:39.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:39.201+0000 7fa535400640 1 --2- 172.21.15.16:0/1133568642 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa5300cb440 0x7fa53014b710 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 crypto rx=0x7fa52801a5a0 tx=0x7fa528019710 comp rx=0 tx=0).stop 2024-03-31T15:09:39.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:39.201+0000 7fa535400640 1 -- 172.21.15.16:0/1133568642 shutdown_connections 2024-03-31T15:09:39.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:39.201+0000 7fa535400640 1 --2- 172.21.15.16:0/1133568642 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fa50c068380 0x7fa50c06a840 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:39.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:39.201+0000 7fa535400640 1 --2- 172.21.15.16:0/1133568642 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa5300cb440 0x7fa53014b710 unknown :-1 s=CLOSED pgs=1020 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:39.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:39.201+0000 7fa535400640 1 --2- 172.21.15.16:0/1133568642 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fa530070030 0x7fa5300b7690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:39.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:39.201+0000 7fa535400640 1 --2- 172.21.15.16:0/1133568642 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa53006dbc0 0x7fa53015c240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:39.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:39.201+0000 7fa535400640 1 -- 172.21.15.16:0/1133568642 >> 172.21.15.16:0/1133568642 conn(0x7fa530067330 msgr2=0x7fa5301587e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:09:39.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:39.201+0000 7fa535400640 1 -- 172.21.15.16:0/1133568642 shutdown_connections 2024-03-31T15:09:39.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:39.201+0000 7fa535400640 1 -- 172.21.15.16:0/1133568642 wait complete. 2024-03-31T15:09:39.209 INFO:tasks.ceph.mds.a.smithi016.stderr: -4531> 2024-03-31T15:08:16.262+0000 7f86baa00640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-31T15:09:39.221 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-31T15:09:39.262 INFO:tasks.ceph.mds.a.smithi016.stdout:starting mds.a at 2024-03-31T15:09:39.368 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:39.377+0000 7f7810800640 1 Processor -- start 2024-03-31T15:09:39.368 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:39.377+0000 7f7810800640 1 -- start start 2024-03-31T15:09:39.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:39.377+0000 7f7810800640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f78081567e0 0x7f7808072c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:39.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:39.377+0000 7f7810800640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f78080731c0 0x7f780806d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:39.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:39.377+0000 7f7810800640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f780815c7e0 con 0x7f780806da10 2024-03-31T15:09:39.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:39.377+0000 7f7810800640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f780806de20 con 0x7f78080731c0 2024-03-31T15:09:39.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:39.377+0000 7f7810800640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f78080736b0 con 0x7f78081567e0 2024-03-31T15:09:39.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:39.377+0000 7f780d400640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f78080731c0 0x7f780806d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:39.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:39.377+0000 7f780d400640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f78080731c0 0x7f780806d4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59750/0 (socket says 172.21.15.16:59750) 2024-03-31T15:09:39.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:39.377+0000 7f780d400640 1 -- 172.21.15.16:0/2821576121 learned_addr learned my addr 172.21.15.16:0/2821576121 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:09:39.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:39.377+0000 7f780de00640 1 --2- 172.21.15.16:0/2821576121 >> v2:172.21.15.89:3300/0 conn(0x7f78081567e0 0x7f7808072c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:39.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:39.377+0000 7f780d400640 1 -- 172.21.15.16:0/2821576121 >> v2:172.21.15.89:3300/0 conn(0x7f78081567e0 msgr2=0x7f7808072c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:09:39.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:39.377+0000 7f780d400640 1 --2- 172.21.15.16:0/2821576121 >> v2:172.21.15.89:3300/0 conn(0x7f78081567e0 0x7f7808072c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:39.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:39.377+0000 7f780d400640 1 -- 172.21.15.16:0/2821576121 >> v1:172.21.15.89:6789/0 conn(0x7f780806da10 legacy=0x7f780815fdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:09:39.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:39.377+0000 7f780d400640 1 -- 172.21.15.16:0/2821576121 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f780815caa0 con 0x7f78080731c0 2024-03-31T15:09:39.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:39.381+0000 7f780d400640 1 --2- 172.21.15.16:0/2821576121 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f78080731c0 0x7f780806d4d0 secure :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 crypto rx=0x7f78080cffe0 tx=0x7f77f801c4e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=acd760aa2cbbad9f server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:09:39.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:39.381+0000 7f780ca00640 1 -- 172.21.15.16:0/2821576121 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77f80b0070 con 0x7f78080731c0 2024-03-31T15:09:39.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:39.381+0000 7f780ca00640 1 -- 172.21.15.16:0/2821576121 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f77f80ab600 con 0x7f78080731c0 2024-03-31T15:09:39.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:39.381+0000 7f780ca00640 1 -- 172.21.15.16:0/2821576121 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77f80ab9d0 con 0x7f78080731c0 2024-03-31T15:09:39.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:39.381+0000 7f7810800640 1 -- 172.21.15.16:0/2821576121 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f78080731c0 msgr2=0x7f780806d4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:39.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:39.381+0000 7f7810800640 1 --2- 172.21.15.16:0/2821576121 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f78080731c0 0x7f780806d4d0 secure :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 crypto rx=0x7f78080cffe0 tx=0x7f77f801c4e0 comp rx=0 tx=0).stop 2024-03-31T15:09:39.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:39.381+0000 7f7810800640 1 -- 172.21.15.16:0/2821576121 shutdown_connections 2024-03-31T15:09:39.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:39.381+0000 7f7810800640 1 --2- 172.21.15.16:0/2821576121 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f78080731c0 0x7f780806d4d0 unknown :-1 s=CLOSED pgs=1024 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:39.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:39.381+0000 7f7810800640 1 --2- 172.21.15.16:0/2821576121 >> v2:172.21.15.89:3300/0 conn(0x7f78081567e0 0x7f7808072c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:39.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:39.381+0000 7f7810800640 1 -- 172.21.15.16:0/2821576121 >> 172.21.15.16:0/2821576121 conn(0x7f780806ee40 msgr2=0x7f780806b620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:09:39.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:39.381+0000 7f7810800640 1 -- 172.21.15.16:0/2821576121 shutdown_connections 2024-03-31T15:09:39.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:39.381+0000 7f7810800640 1 -- 172.21.15.16:0/2821576121 wait complete. 2024-03-31T15:09:39.375 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:39.385+0000 7f7810800640 1 Processor -- start 2024-03-31T15:09:39.375 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:39.385+0000 7f7810800640 1 -- start start 2024-03-31T15:09:39.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:39.385+0000 7f7810800640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f780806da10 0x7f7808154420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:39.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:39.385+0000 7f7810800640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f78080731c0 0x7f7808154e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:39.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:39.385+0000 7f7810800640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f78081567e0 0x7f7808146b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:39.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:39.385+0000 7f7810800640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f780815d6e0 con 0x7f78080731c0 2024-03-31T15:09:39.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:39.385+0000 7f7810800640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f780815d560 con 0x7f780806da10 2024-03-31T15:09:39.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:39.385+0000 7f7810800640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f7808169170 con 0x7f78081567e0 2024-03-31T15:09:39.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:39.385+0000 7f780d400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f78080731c0 0x7f7808154e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:39.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:39.385+0000 7f780d400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f78080731c0 0x7f7808154e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59752/0 (socket says 172.21.15.16:59752) 2024-03-31T15:09:39.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:39.385+0000 7f780d400640 1 -- 172.21.15.16:0/3590457866 learned_addr learned my addr 172.21.15.16:0/3590457866 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:09:39.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:39.385+0000 7f780de00640 1 --2- 172.21.15.16:0/3590457866 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f780806da10 0x7f7808154420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:39.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:39.385+0000 7f780e800640 1 --2- 172.21.15.16:0/3590457866 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f78081567e0 0x7f7808146b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:39.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:39.385+0000 7f780d400640 1 -- 172.21.15.16:0/3590457866 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f78081567e0 msgr2=0x7f7808146b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:39.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:39.385+0000 7f780d400640 1 --2- 172.21.15.16:0/3590457866 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f78081567e0 0x7f7808146b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:39.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:39.385+0000 7f780d400640 1 -- 172.21.15.16:0/3590457866 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f780806da10 msgr2=0x7f7808154420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:39.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:39.385+0000 7f780d400640 1 --2- 172.21.15.16:0/3590457866 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f780806da10 0x7f7808154420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:39.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:39.385+0000 7f780d400640 1 -- 172.21.15.16:0/3590457866 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7808153980 con 0x7f78080731c0 2024-03-31T15:09:39.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:39.385+0000 7f780d400640 1 --2- 172.21.15.16:0/3590457866 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f78080731c0 0x7f7808154e30 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 crypto rx=0x7f77f80ae040 tx=0x7f77f8019280 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:09:39.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:39.385+0000 7f7802000640 1 -- 172.21.15.16:0/3590457866 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77f80b0040 con 0x7f78080731c0 2024-03-31T15:09:39.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:39.385+0000 7f7802000640 1 -- 172.21.15.16:0/3590457866 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f77f80b9070 con 0x7f78080731c0 2024-03-31T15:09:39.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:39.385+0000 7f7802000640 1 -- 172.21.15.16:0/3590457866 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77f80b4680 con 0x7f78080731c0 2024-03-31T15:09:39.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:39.385+0000 7f7810800640 1 -- 172.21.15.16:0/3590457866 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f780815b5c0 con 0x7f78080731c0 2024-03-31T15:09:39.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:39.385+0000 7f7810800640 1 -- 172.21.15.16:0/3590457866 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7808153f00 con 0x7f78080731c0 2024-03-31T15:09:39.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:39.385+0000 7f7810800640 1 -- 172.21.15.16:0/3590457866 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7808069950 con 0x7f78080731c0 2024-03-31T15:09:39.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:39.389+0000 7f7802000640 1 -- 172.21.15.16:0/3590457866 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f77f80b4820 con 0x7f78080731c0 2024-03-31T15:09:39.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:39.389+0000 7f7802000640 1 --2- 172.21.15.16:0/3590457866 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f77ec068380 0x7f77ec06a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:39.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:39.389+0000 7f7802000640 1 -- 172.21.15.16:0/3590457866 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(71..71 src has 1..71) ==== 5968+0+0 (secure 0 0 0) 0x7f77f81207d0 con 0x7f78080731c0 2024-03-31T15:09:39.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:39.389+0000 7f780de00640 1 --2- 172.21.15.16:0/3590457866 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f77ec068380 0x7f77ec06a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:39.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:39.389+0000 7f780de00640 1 --2- 172.21.15.16:0/3590457866 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f77ec068380 0x7f77ec06a840 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f780814b9c0 tx=0x7f77fc008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:09:39.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:39.393+0000 7f7802000640 1 -- 172.21.15.16:0/3590457866 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f77f80160d0 con 0x7f78080731c0 2024-03-31T15:09:39.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:39.593+0000 7f7810800640 1 -- 172.21.15.16:0/3590457866 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7f780815f410 con 0x7f78080731c0 2024-03-31T15:09:40.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:40.201+0000 7f7802000640 1 -- 172.21.15.16:0/3590457866 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v54) ==== 104+0+0 (secure 0 0 0) 0x7f77f80f0c90 con 0x7f78080731c0 2024-03-31T15:09:40.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:40.201+0000 7f7810800640 1 -- 172.21.15.16:0/3590457866 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f77ec068380 msgr2=0x7f77ec06a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:40.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:40.201+0000 7f7810800640 1 --2- 172.21.15.16:0/3590457866 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f77ec068380 0x7f77ec06a840 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f780814b9c0 tx=0x7f77fc008040 comp rx=0 tx=0).stop 2024-03-31T15:09:40.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:40.201+0000 7f7810800640 1 -- 172.21.15.16:0/3590457866 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f78080731c0 msgr2=0x7f7808154e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:40.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:40.201+0000 7f7810800640 1 --2- 172.21.15.16:0/3590457866 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f78080731c0 0x7f7808154e30 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 crypto rx=0x7f77f80ae040 tx=0x7f77f8019280 comp rx=0 tx=0).stop 2024-03-31T15:09:40.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:40.201+0000 7f7810800640 1 -- 172.21.15.16:0/3590457866 shutdown_connections 2024-03-31T15:09:40.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:40.201+0000 7f7810800640 1 --2- 172.21.15.16:0/3590457866 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f77ec068380 0x7f77ec06a840 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:40.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:40.201+0000 7f7810800640 1 --2- 172.21.15.16:0/3590457866 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f78081567e0 0x7f7808146b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:40.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:40.201+0000 7f7810800640 1 --2- 172.21.15.16:0/3590457866 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f78080731c0 0x7f7808154e30 unknown :-1 s=CLOSED pgs=1025 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:40.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:40.201+0000 7f7810800640 1 --2- 172.21.15.16:0/3590457866 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f780806da10 0x7f7808154420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:40.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:40.201+0000 7f7810800640 1 -- 172.21.15.16:0/3590457866 >> 172.21.15.16:0/3590457866 conn(0x7f780806ee40 msgr2=0x7f780815e510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:09:40.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:40.205+0000 7f7810800640 1 -- 172.21.15.16:0/3590457866 shutdown_connections 2024-03-31T15:09:40.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:40.205+0000 7f7810800640 1 -- 172.21.15.16:0/3590457866 wait complete. 2024-03-31T15:09:40.218 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-31T15:09:40.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:40.413+0000 7f1090400640 1 Processor -- start 2024-03-31T15:09:40.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:40.413+0000 7f1090400640 1 -- start start 2024-03-31T15:09:40.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:40.413+0000 7f1090400640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f10880b7900 0x7f1088160740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:40.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:40.413+0000 7f1090400640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f10880cc7b0 0x7f10880b3dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:40.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:40.413+0000 7f1090400640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f10880ce2d0 con 0x7f10880b4300 2024-03-31T15:09:40.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:40.413+0000 7f1090400640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f10880b4770 con 0x7f10880cc7b0 2024-03-31T15:09:40.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:40.413+0000 7f1090400640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f10880ceb10 con 0x7f10880b7900 2024-03-31T15:09:40.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:40.413+0000 7f108d000640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f10880cc7b0 0x7f10880b3dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:40.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:40.413+0000 7f108d000640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f10880cc7b0 0x7f10880b3dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59768/0 (socket says 172.21.15.16:59768) 2024-03-31T15:09:40.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:40.413+0000 7f108e400640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f10880b4300 0x7f10880d1a80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:55186/0 (socket says 172.21.15.16:55186) 2024-03-31T15:09:40.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:40.413+0000 7f108d000640 1 -- 172.21.15.16:0/3986770282 learned_addr learned my addr 172.21.15.16:0/3986770282 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:09:40.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:40.417+0000 7f108da00640 1 --2- 172.21.15.16:0/3986770282 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f10880b7900 0x7f1088160740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:40.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:40.417+0000 7f108d000640 1 -- 172.21.15.16:0/3986770282 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f10880b7900 msgr2=0x7f1088160740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:09:40.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:40.417+0000 7f108d000640 1 --2- 172.21.15.16:0/3986770282 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f10880b7900 0x7f1088160740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:40.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:40.417+0000 7f108d000640 1 -- 172.21.15.16:0/3986770282 >> v1:172.21.15.16:6789/0 conn(0x7f10880b4300 legacy=0x7f10880d1a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:40.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:40.417+0000 7f108d000640 1 -- 172.21.15.16:0/3986770282 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10880ce550 con 0x7f10880cc7b0 2024-03-31T15:09:40.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:40.417+0000 7f1087e00640 1 -- 172.21.15.16:0/3986770282 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1337573429 0 0) 0x7f10800014a0 con 0x7f10880b4300 2024-03-31T15:09:40.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:40.417+0000 7f108d000640 1 --2- 172.21.15.16:0/3986770282 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f10880cc7b0 0x7f10880b3dc0 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7f1074009870 tx=0x7f107401c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77d5c69f08708a20 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:09:40.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:40.417+0000 7f1087e00640 1 -- 172.21.15.16:0/3986770282 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10740aa600 con 0x7f10880cc7b0 2024-03-31T15:09:40.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:40.417+0000 7f1087e00640 1 -- 172.21.15.16:0/3986770282 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f10740af070 con 0x7f10880cc7b0 2024-03-31T15:09:40.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:40.417+0000 7f1087e00640 1 -- 172.21.15.16:0/3986770282 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10740aa9c0 con 0x7f10880cc7b0 2024-03-31T15:09:40.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:40.417+0000 7f1090400640 1 -- 172.21.15.16:0/3986770282 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f10880cc7b0 msgr2=0x7f10880b3dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:40.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:40.417+0000 7f1090400640 1 --2- 172.21.15.16:0/3986770282 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f10880cc7b0 0x7f10880b3dc0 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7f1074009870 tx=0x7f107401c4a0 comp rx=0 tx=0).stop 2024-03-31T15:09:40.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:40.421+0000 7f1090400640 1 -- 172.21.15.16:0/3986770282 shutdown_connections 2024-03-31T15:09:40.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:40.421+0000 7f1090400640 1 --2- 172.21.15.16:0/3986770282 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f10880cc7b0 0x7f10880b3dc0 unknown :-1 s=CLOSED pgs=1027 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:40.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:40.421+0000 7f1090400640 1 --2- 172.21.15.16:0/3986770282 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f10880b7900 0x7f1088160740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:40.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:40.421+0000 7f1090400640 1 -- 172.21.15.16:0/3986770282 >> 172.21.15.16:0/3986770282 conn(0x7f108806b7f0 msgr2=0x7f1088067330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:09:40.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:40.421+0000 7f1090400640 1 -- 172.21.15.16:0/3986770282 shutdown_connections 2024-03-31T15:09:40.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:40.421+0000 7f1090400640 1 -- 172.21.15.16:0/3986770282 wait complete. 2024-03-31T15:09:40.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:40.421+0000 7f1090400640 1 Processor -- start 2024-03-31T15:09:40.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:40.421+0000 7f1090400640 1 -- start start 2024-03-31T15:09:40.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:40.421+0000 7f1090400640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f10880b4300 0x7f1088149ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:40.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:40.421+0000 7f1090400640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f10880b7900 0x7f108815ebf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:40.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:40.421+0000 7f1090400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f10880cc7b0 0x7f10880b2880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:40.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:40.421+0000 7f1090400640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f10880b7ce0 con 0x7f10880cc7b0 2024-03-31T15:09:40.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:40.421+0000 7f1090400640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f1088160c80 con 0x7f10880b7900 2024-03-31T15:09:40.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:40.421+0000 7f1090400640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f10880d25a0 con 0x7f10880b4300 2024-03-31T15:09:40.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:40.421+0000 7f108e400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f10880cc7b0 0x7f10880b2880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:40.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:40.421+0000 7f108e400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f10880cc7b0 0x7f10880b2880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59770/0 (socket says 172.21.15.16:59770) 2024-03-31T15:09:40.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:40.421+0000 7f108e400640 1 -- 172.21.15.16:0/818040126 learned_addr learned my addr 172.21.15.16:0/818040126 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:09:40.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:40.421+0000 7f108e400640 1 -- 172.21.15.16:0/818040126 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f10880b4300 msgr2=0x7f1088149ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:40.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:40.421+0000 7f108e400640 1 --2- 172.21.15.16:0/818040126 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f10880b4300 0x7f1088149ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:40.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:40.421+0000 7f108e400640 1 -- 172.21.15.16:0/818040126 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f10880b7900 msgr2=0x7f108815ebf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:40.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:40.421+0000 7f108e400640 1 --2- 172.21.15.16:0/818040126 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f10880b7900 0x7f108815ebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:40.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:40.421+0000 7f108e400640 1 -- 172.21.15.16:0/818040126 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f108815f440 con 0x7f10880cc7b0 2024-03-31T15:09:40.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:40.421+0000 7f108e400640 1 --2- 172.21.15.16:0/818040126 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f10880cc7b0 0x7f10880b2880 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 crypto rx=0x7f108000c7f0 tx=0x7f108000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:09:40.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:40.421+0000 7f1085600640 1 -- 172.21.15.16:0/818040126 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1080018070 con 0x7f10880cc7b0 2024-03-31T15:09:40.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:40.421+0000 7f1085600640 1 -- 172.21.15.16:0/818040126 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f108000a640 con 0x7f10880cc7b0 2024-03-31T15:09:40.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:40.421+0000 7f1085600640 1 -- 172.21.15.16:0/818040126 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f108000aa30 con 0x7f10880cc7b0 2024-03-31T15:09:40.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:40.421+0000 7f1090400640 1 -- 172.21.15.16:0/818040126 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f108815f720 con 0x7f10880cc7b0 2024-03-31T15:09:40.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:40.425+0000 7f1090400640 1 -- 172.21.15.16:0/818040126 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f10880b2080 con 0x7f10880cc7b0 2024-03-31T15:09:40.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:40.425+0000 7f1090400640 1 -- 172.21.15.16:0/818040126 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1088069950 con 0x7f10880cc7b0 2024-03-31T15:09:40.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:40.425+0000 7f1085600640 1 -- 172.21.15.16:0/818040126 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f108001b070 con 0x7f10880cc7b0 2024-03-31T15:09:40.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:40.425+0000 7f1085600640 1 --2- 172.21.15.16:0/818040126 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f10680683d0 0x7f106806a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:40.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:40.425+0000 7f1085600640 1 -- 172.21.15.16:0/818040126 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(71..71 src has 1..71) ==== 5968+0+0 (secure 0 0 0) 0x7f1080086230 con 0x7f10880cc7b0 2024-03-31T15:09:40.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:40.425+0000 7f108da00640 1 --2- 172.21.15.16:0/818040126 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f10680683d0 0x7f106806a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:40.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:40.425+0000 7f108da00640 1 --2- 172.21.15.16:0/818040126 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f10680683d0 0x7f106806a890 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f107c0098f0 tx=0x7f107c008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:09:40.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:40.429+0000 7f1085600640 1 -- 172.21.15.16:0/818040126 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f10880ce2d0 con 0x7f10880cc7b0 2024-03-31T15:09:40.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:40.601+0000 7f1090400640 1 -- 172.21.15.16:0/818040126 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7f10880d19f0 con 0x7f10880cc7b0 2024-03-31T15:09:41.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:41.465+0000 7f1085600640 1 -- 172.21.15.16:0/818040126 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v72) ==== 181+0+0 (secure 0 0 0) 0x7f10800566f0 con 0x7f10880cc7b0 2024-03-31T15:09:41.457 INFO:teuthology.orchestra.run.smithi016.stderr:pool 'cephfs_metadata' removed 2024-03-31T15:09:41.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:41.469+0000 7f1090400640 1 -- 172.21.15.16:0/818040126 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f10680683d0 msgr2=0x7f106806a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:41.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:41.469+0000 7f1090400640 1 --2- 172.21.15.16:0/818040126 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f10680683d0 0x7f106806a890 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f107c0098f0 tx=0x7f107c008040 comp rx=0 tx=0).stop 2024-03-31T15:09:41.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:41.469+0000 7f1090400640 1 -- 172.21.15.16:0/818040126 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f10880cc7b0 msgr2=0x7f10880b2880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:41.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:41.469+0000 7f1090400640 1 --2- 172.21.15.16:0/818040126 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f10880cc7b0 0x7f10880b2880 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 crypto rx=0x7f108000c7f0 tx=0x7f108000ccb0 comp rx=0 tx=0).stop 2024-03-31T15:09:41.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:41.469+0000 7f1090400640 1 -- 172.21.15.16:0/818040126 shutdown_connections 2024-03-31T15:09:41.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:41.469+0000 7f1090400640 1 --2- 172.21.15.16:0/818040126 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f10680683d0 0x7f106806a890 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:41.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:41.469+0000 7f1090400640 1 --2- 172.21.15.16:0/818040126 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f10880cc7b0 0x7f10880b2880 unknown :-1 s=CLOSED pgs=1028 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:41.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:41.469+0000 7f1090400640 1 --2- 172.21.15.16:0/818040126 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f10880b7900 0x7f108815ebf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:41.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:41.469+0000 7f1090400640 1 --2- 172.21.15.16:0/818040126 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f10880b4300 0x7f1088149ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:41.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:41.469+0000 7f1090400640 1 -- 172.21.15.16:0/818040126 >> 172.21.15.16:0/818040126 conn(0x7f108806b7f0 msgr2=0x7f1088067330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:09:41.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:41.469+0000 7f1090400640 1 -- 172.21.15.16:0/818040126 shutdown_connections 2024-03-31T15:09:41.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:41.469+0000 7f1090400640 1 -- 172.21.15.16:0/818040126 wait complete. 2024-03-31T15:09:41.487 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-31T15:09:41.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:41.645+0000 7f7544c00640 1 Processor -- start 2024-03-31T15:09:41.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:41.645+0000 7f7544c00640 1 -- start start 2024-03-31T15:09:41.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:41.649+0000 7f7544c00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f75400b44d0 0x7f75400d0930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:41.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:41.649+0000 7f7544c00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f75400d0e70 0x7f75400cb4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:41.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:41.649+0000 7f7544c00640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f75400d2020 con 0x7f7540147230 2024-03-31T15:09:41.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:41.649+0000 7f7544c00640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f75400d2c30 con 0x7f75400d0e70 2024-03-31T15:09:41.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:41.649+0000 7f7544c00640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f75400d2300 con 0x7f75400b44d0 2024-03-31T15:09:41.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:41.649+0000 7f753de00640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f7540147230 0x7f75400b3dc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:55192/0 (socket says 172.21.15.16:55192) 2024-03-31T15:09:41.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:41.649+0000 7f753de00640 1 -- 172.21.15.16:0/3404727197 learned_addr learned my addr 172.21.15.16:0/3404727197 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:09:41.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:41.649+0000 7f753e800640 1 --2- 172.21.15.16:0/3404727197 >> v2:172.21.15.16:3300/0 conn(0x7f75400d0e70 0x7f75400cb4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:41.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:41.649+0000 7f7537e00640 1 --2- 172.21.15.16:0/3404727197 >> v2:172.21.15.89:3300/0 conn(0x7f75400b44d0 0x7f75400d0930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:41.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:41.649+0000 7f753d400640 1 -- 172.21.15.16:0/3404727197 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 52516630 0 0) 0x7f75400d2020 con 0x7f7540147230 2024-03-31T15:09:41.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:41.649+0000 7f753e800640 1 -- 172.21.15.16:0/3404727197 >> v2:172.21.15.89:3300/0 conn(0x7f75400b44d0 msgr2=0x7f75400d0930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:09:41.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:41.649+0000 7f753e800640 1 --2- 172.21.15.16:0/3404727197 >> v2:172.21.15.89:3300/0 conn(0x7f75400b44d0 0x7f75400d0930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:41.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:41.649+0000 7f753e800640 1 -- 172.21.15.16:0/3404727197 >> v1:172.21.15.16:6789/0 conn(0x7f7540147230 legacy=0x7f75400b3dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:41.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:41.649+0000 7f753e800640 1 -- 172.21.15.16:0/3404727197 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75400d25e0 con 0x7f75400d0e70 2024-03-31T15:09:41.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:41.653+0000 7f753e800640 1 --2- 172.21.15.16:0/3404727197 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f75400d0e70 0x7f75400cb4b0 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 crypto rx=0x7f75400d3200 tx=0x7f753801c930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38159326c08a3b1a server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:09:41.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:41.653+0000 7f753d400640 1 -- 172.21.15.16:0/3404727197 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75380b2070 con 0x7f75400d0e70 2024-03-31T15:09:41.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:41.653+0000 7f753d400640 1 -- 172.21.15.16:0/3404727197 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f75380a58c0 con 0x7f75400d0e70 2024-03-31T15:09:41.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:41.653+0000 7f753d400640 1 -- 172.21.15.16:0/3404727197 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75380a5c90 con 0x7f75400d0e70 2024-03-31T15:09:41.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:41.653+0000 7f7544c00640 1 -- 172.21.15.16:0/3404727197 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f75400d0e70 msgr2=0x7f75400cb4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:41.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:41.653+0000 7f7544c00640 1 --2- 172.21.15.16:0/3404727197 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f75400d0e70 0x7f75400cb4b0 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 crypto rx=0x7f75400d3200 tx=0x7f753801c930 comp rx=0 tx=0).stop 2024-03-31T15:09:41.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:41.653+0000 7f7544c00640 1 -- 172.21.15.16:0/3404727197 shutdown_connections 2024-03-31T15:09:41.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:41.653+0000 7f7544c00640 1 --2- 172.21.15.16:0/3404727197 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f75400d0e70 0x7f75400cb4b0 unknown :-1 s=CLOSED pgs=1030 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:41.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:41.653+0000 7f7544c00640 1 --2- 172.21.15.16:0/3404727197 >> v2:172.21.15.89:3300/0 conn(0x7f75400b44d0 0x7f75400d0930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:41.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:41.653+0000 7f7544c00640 1 -- 172.21.15.16:0/3404727197 >> 172.21.15.16:0/3404727197 conn(0x7f754006b7f0 msgr2=0x7f7540067330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:09:41.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:41.653+0000 7f7544c00640 1 -- 172.21.15.16:0/3404727197 shutdown_connections 2024-03-31T15:09:41.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:41.653+0000 7f7544c00640 1 -- 172.21.15.16:0/3404727197 wait complete. 2024-03-31T15:09:41.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:41.653+0000 7f7544c00640 1 Processor -- start 2024-03-31T15:09:41.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:41.653+0000 7f7544c00640 1 -- start start 2024-03-31T15:09:41.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:41.657+0000 7f7544c00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f75400b44d0 0x7f754015f980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:41.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:41.657+0000 7f7544c00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f75400d0e70 0x7f75400b6500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:41.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:41.657+0000 7f7544c00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7540147230 0x7f754015d730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:41.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:41.657+0000 7f7544c00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f75400cc200 con 0x7f7540147230 2024-03-31T15:09:41.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:41.657+0000 7f7544c00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f75400d1510 con 0x7f75400d0e70 2024-03-31T15:09:41.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:41.657+0000 7f7544c00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f75400cc380 con 0x7f75400b44d0 2024-03-31T15:09:41.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:41.657+0000 7f753e800640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7540147230 0x7f754015d730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:41.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:41.657+0000 7f753e800640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7540147230 0x7f754015d730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59792/0 (socket says 172.21.15.16:59792) 2024-03-31T15:09:41.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:41.657+0000 7f753e800640 1 -- 172.21.15.16:0/829229581 learned_addr learned my addr 172.21.15.16:0/829229581 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:09:41.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:41.657+0000 7f7537e00640 1 --2- 172.21.15.16:0/829229581 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f75400d0e70 0x7f75400b6500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:41.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:41.657+0000 7f753de00640 1 --2- 172.21.15.16:0/829229581 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f75400b44d0 0x7f754015f980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:41.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:41.657+0000 7f753e800640 1 -- 172.21.15.16:0/829229581 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f75400b44d0 msgr2=0x7f754015f980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:41.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:41.657+0000 7f753e800640 1 --2- 172.21.15.16:0/829229581 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f75400b44d0 0x7f754015f980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:41.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:41.657+0000 7f753e800640 1 -- 172.21.15.16:0/829229581 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f75400d0e70 msgr2=0x7f75400b6500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:41.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:41.657+0000 7f753e800640 1 --2- 172.21.15.16:0/829229581 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f75400d0e70 0x7f75400b6500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:41.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:41.657+0000 7f753e800640 1 -- 172.21.15.16:0/829229581 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7540157b90 con 0x7f7540147230 2024-03-31T15:09:41.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:41.657+0000 7f753e800640 1 --2- 172.21.15.16:0/829229581 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7540147230 0x7f754015d730 secure :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 crypto rx=0x7f753801c970 tx=0x7f75380196d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:09:41.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:41.657+0000 7f7536000640 1 -- 172.21.15.16:0/829229581 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75380be020 con 0x7f7540147230 2024-03-31T15:09:41.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:41.657+0000 7f7536000640 1 -- 172.21.15.16:0/829229581 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f75380b8670 con 0x7f7540147230 2024-03-31T15:09:41.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:41.657+0000 7f7536000640 1 -- 172.21.15.16:0/829229581 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75380b2040 con 0x7f7540147230 2024-03-31T15:09:41.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:41.657+0000 7f7544c00640 1 -- 172.21.15.16:0/829229581 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f754015ebc0 con 0x7f7540147230 2024-03-31T15:09:41.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:41.657+0000 7f7544c00640 1 -- 172.21.15.16:0/829229581 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f754015f180 con 0x7f7540147230 2024-03-31T15:09:41.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:41.657+0000 7f7544c00640 1 -- 172.21.15.16:0/829229581 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f754013b430 con 0x7f7540147230 2024-03-31T15:09:41.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:41.657+0000 7f7536000640 1 -- 172.21.15.16:0/829229581 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f75380c1070 con 0x7f7540147230 2024-03-31T15:09:41.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:41.661+0000 7f7536000640 1 --2- 172.21.15.16:0/829229581 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f751c068380 0x7f751c06a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:41.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:41.661+0000 7f7536000640 1 -- 172.21.15.16:0/829229581 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 5530+0+0 (secure 0 0 0) 0x7f7538120e40 con 0x7f7540147230 2024-03-31T15:09:41.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:41.661+0000 7f753de00640 1 --2- 172.21.15.16:0/829229581 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f751c068380 0x7f751c06a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:41.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:41.661+0000 7f753de00640 1 --2- 172.21.15.16:0/829229581 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f751c068380 0x7f751c06a840 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f7530007fd0 tx=0x7f7530007d60 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:09:41.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:41.665+0000 7f7536000640 1 -- 172.21.15.16:0/829229581 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f754013b430 con 0x7f7540147230 2024-03-31T15:09:41.822 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:41.829+0000 7f7544c00640 1 -- 172.21.15.16:0/829229581 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7f75400d0990 con 0x7f7540147230 2024-03-31T15:09:42.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:42.465+0000 7f7536000640 1 -- 172.21.15.16:0/829229581 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v73) ==== 169+0+0 (secure 0 0 0) 0x7f7538016520 con 0x7f7540147230 2024-03-31T15:09:42.459 INFO:teuthology.orchestra.run.smithi016.stderr:pool 'cephfs_data' removed 2024-03-31T15:09:42.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:42.469+0000 7f7544c00640 1 -- 172.21.15.16:0/829229581 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f751c068380 msgr2=0x7f751c06a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:42.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:42.469+0000 7f7544c00640 1 --2- 172.21.15.16:0/829229581 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f751c068380 0x7f751c06a840 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f7530007fd0 tx=0x7f7530007d60 comp rx=0 tx=0).stop 2024-03-31T15:09:42.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:42.469+0000 7f7544c00640 1 -- 172.21.15.16:0/829229581 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7540147230 msgr2=0x7f754015d730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:42.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:42.469+0000 7f7544c00640 1 --2- 172.21.15.16:0/829229581 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7540147230 0x7f754015d730 secure :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 crypto rx=0x7f753801c970 tx=0x7f75380196d0 comp rx=0 tx=0).stop 2024-03-31T15:09:42.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:42.469+0000 7f7544c00640 1 -- 172.21.15.16:0/829229581 shutdown_connections 2024-03-31T15:09:42.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:42.469+0000 7f7544c00640 1 --2- 172.21.15.16:0/829229581 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f751c068380 0x7f751c06a840 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:42.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:42.469+0000 7f7544c00640 1 --2- 172.21.15.16:0/829229581 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7540147230 0x7f754015d730 unknown :-1 s=CLOSED pgs=1031 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:42.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:42.469+0000 7f7544c00640 1 --2- 172.21.15.16:0/829229581 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f75400d0e70 0x7f75400b6500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:42.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:42.469+0000 7f7544c00640 1 --2- 172.21.15.16:0/829229581 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f75400b44d0 0x7f754015f980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:42.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:42.469+0000 7f7544c00640 1 -- 172.21.15.16:0/829229581 >> 172.21.15.16:0/829229581 conn(0x7f754006b7f0 msgr2=0x7f754013f610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:09:42.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:42.473+0000 7f7544c00640 1 -- 172.21.15.16:0/829229581 shutdown_connections 2024-03-31T15:09:42.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:42.473+0000 7f7544c00640 1 -- 172.21.15.16:0/829229581 wait complete. 2024-03-31T15:09:42.490 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-03-31T15:09:42.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:42.641+0000 7f6bee000640 1 Processor -- start 2024-03-31T15:09:42.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:42.641+0000 7f6bee000640 1 -- start start 2024-03-31T15:09:42.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:42.645+0000 7f6bee000640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f6be81567e0 0x7f6be8072c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:42.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:42.645+0000 7f6bee000640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f6be80731c0 0x7f6be806d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:42.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:42.645+0000 7f6bee000640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6be815c7e0 con 0x7f6be806da10 2024-03-31T15:09:42.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:42.645+0000 7f6bee000640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f6be806de20 con 0x7f6be80731c0 2024-03-31T15:09:42.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:42.645+0000 7f6bee000640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f6be80736b0 con 0x7f6be81567e0 2024-03-31T15:09:42.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:42.645+0000 7f6bdfe00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f6be80731c0 0x7f6be806d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:42.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:42.645+0000 7f6be7e00640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f6be806da10 0x7f6be815fdd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:55202/0 (socket says 172.21.15.16:55202) 2024-03-31T15:09:42.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:42.645+0000 7f6be7e00640 1 -- 172.21.15.16:0/3782006343 learned_addr learned my addr 172.21.15.16:0/3782006343 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:09:42.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:42.645+0000 7f6bdfe00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f6be80731c0 0x7f6be806d4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59806/0 (socket says 172.21.15.16:59806) 2024-03-31T15:09:42.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:42.645+0000 7f6be7400640 1 --2- 172.21.15.16:0/3782006343 >> v2:172.21.15.89:3300/0 conn(0x7f6be81567e0 0x7f6be8072c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:42.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:42.645+0000 7f6bdfe00640 1 -- 172.21.15.16:0/3782006343 >> v2:172.21.15.89:3300/0 conn(0x7f6be81567e0 msgr2=0x7f6be8072c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:09:42.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:42.645+0000 7f6bdfe00640 1 --2- 172.21.15.16:0/3782006343 >> v2:172.21.15.89:3300/0 conn(0x7f6be81567e0 0x7f6be8072c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:42.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:42.645+0000 7f6bdfe00640 1 -- 172.21.15.16:0/3782006343 >> v1:172.21.15.16:6789/0 conn(0x7f6be806da10 legacy=0x7f6be815fdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:42.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:42.645+0000 7f6bdfe00640 1 -- 172.21.15.16:0/3782006343 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6be815caa0 con 0x7f6be80731c0 2024-03-31T15:09:42.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:42.645+0000 7f6be6a00640 1 -- 172.21.15.16:0/3782006343 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4249646583 0 0) 0x7f6be815c7e0 con 0x7f6be806da10 2024-03-31T15:09:42.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:42.645+0000 7f6be7400640 1 --2- 172.21.15.16:0/3782006343 >> v2:172.21.15.89:3300/0 conn(0x7f6be81567e0 0x7f6be8072c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:09:42.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:42.645+0000 7f6bdfe00640 1 --2- 172.21.15.16:0/3782006343 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6be80731c0 0x7f6be806d4d0 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 crypto rx=0x7f6be80cffe0 tx=0x7f6bd001c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=677bddc3cc4ad767 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:09:42.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:42.649+0000 7f6be6a00640 1 -- 172.21.15.16:0/3782006343 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bd00b0070 con 0x7f6be80731c0 2024-03-31T15:09:42.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:42.649+0000 7f6be6a00640 1 -- 172.21.15.16:0/3782006343 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6bd00ab600 con 0x7f6be80731c0 2024-03-31T15:09:42.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:42.649+0000 7f6be6a00640 1 -- 172.21.15.16:0/3782006343 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bd00ab9d0 con 0x7f6be80731c0 2024-03-31T15:09:42.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:42.649+0000 7f6bee000640 1 -- 172.21.15.16:0/3782006343 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6be80731c0 msgr2=0x7f6be806d4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:42.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:42.649+0000 7f6bee000640 1 --2- 172.21.15.16:0/3782006343 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6be80731c0 0x7f6be806d4d0 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 crypto rx=0x7f6be80cffe0 tx=0x7f6bd001c480 comp rx=0 tx=0).stop 2024-03-31T15:09:42.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:42.649+0000 7f6bee000640 1 -- 172.21.15.16:0/3782006343 shutdown_connections 2024-03-31T15:09:42.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:42.649+0000 7f6bee000640 1 --2- 172.21.15.16:0/3782006343 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6be80731c0 0x7f6be806d4d0 unknown :-1 s=CLOSED pgs=1033 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:42.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:42.649+0000 7f6bee000640 1 --2- 172.21.15.16:0/3782006343 >> v2:172.21.15.89:3300/0 conn(0x7f6be81567e0 0x7f6be8072c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:42.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:42.649+0000 7f6bee000640 1 -- 172.21.15.16:0/3782006343 >> 172.21.15.16:0/3782006343 conn(0x7f6be806ee40 msgr2=0x7f6be806b620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:09:42.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:42.649+0000 7f6bee000640 1 -- 172.21.15.16:0/3782006343 shutdown_connections 2024-03-31T15:09:42.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:42.649+0000 7f6bee000640 1 -- 172.21.15.16:0/3782006343 wait complete. 2024-03-31T15:09:42.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:42.649+0000 7f6bee000640 1 Processor -- start 2024-03-31T15:09:42.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:42.649+0000 7f6bee000640 1 -- start start 2024-03-31T15:09:42.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:42.649+0000 7f6bee000640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6be806da10 0x7f6be81449a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:42.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:42.649+0000 7f6bee000640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f6be80731c0 0x7f6be815a3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:42.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:42.649+0000 7f6bee000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6be81567e0 0x7f6be815a920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:42.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:42.649+0000 7f6bee000640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f6be815d6e0 con 0x7f6be81567e0 2024-03-31T15:09:42.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:42.649+0000 7f6bee000640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f6be815d560 con 0x7f6be806da10 2024-03-31T15:09:42.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:42.649+0000 7f6bee000640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f6be8169170 con 0x7f6be80731c0 2024-03-31T15:09:42.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:42.649+0000 7f6be7e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6be81567e0 0x7f6be815a920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:42.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:42.649+0000 7f6be7e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6be81567e0 0x7f6be815a920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59818/0 (socket says 172.21.15.16:59818) 2024-03-31T15:09:42.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:42.649+0000 7f6be7e00640 1 -- 172.21.15.16:0/3844255852 learned_addr learned my addr 172.21.15.16:0/3844255852 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:09:42.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:42.649+0000 7f6be7400640 1 --2- 172.21.15.16:0/3844255852 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6be806da10 0x7f6be81449a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:42.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:42.649+0000 7f6bdfe00640 1 --2- 172.21.15.16:0/3844255852 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f6be80731c0 0x7f6be815a3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:42.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:42.649+0000 7f6be7e00640 1 -- 172.21.15.16:0/3844255852 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f6be80731c0 msgr2=0x7f6be815a3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:42.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:42.649+0000 7f6be7e00640 1 --2- 172.21.15.16:0/3844255852 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f6be80731c0 0x7f6be815a3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:42.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:42.649+0000 7f6be7e00640 1 -- 172.21.15.16:0/3844255852 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6be806da10 msgr2=0x7f6be81449a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:42.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:42.649+0000 7f6be7e00640 1 --2- 172.21.15.16:0/3844255852 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6be806da10 0x7f6be81449a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:42.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:42.649+0000 7f6be7e00640 1 -- 172.21.15.16:0/3844255852 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6be80b46a0 con 0x7f6be81567e0 2024-03-31T15:09:42.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:42.653+0000 7f6be7e00640 1 --2- 172.21.15.16:0/3844255852 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6be81567e0 0x7f6be815a920 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 crypto rx=0x7f6be000c7a0 tx=0x7f6be000cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:09:42.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:42.653+0000 7f6bdf400640 1 -- 172.21.15.16:0/3844255852 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6be0018070 con 0x7f6be81567e0 2024-03-31T15:09:42.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:42.653+0000 7f6bdf400640 1 -- 172.21.15.16:0/3844255852 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6be000a640 con 0x7f6be81567e0 2024-03-31T15:09:42.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:42.653+0000 7f6bdf400640 1 -- 172.21.15.16:0/3844255852 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6be000aab0 con 0x7f6be81567e0 2024-03-31T15:09:42.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:42.653+0000 7f6bee000640 1 -- 172.21.15.16:0/3844255852 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6be80b4980 con 0x7f6be81567e0 2024-03-31T15:09:42.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:42.653+0000 7f6bee000640 1 -- 172.21.15.16:0/3844255852 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6be8144fb0 con 0x7f6be81567e0 2024-03-31T15:09:42.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:42.653+0000 7f6bee000640 1 -- 172.21.15.16:0/3844255852 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6be8069950 con 0x7f6be81567e0 2024-03-31T15:09:42.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:42.653+0000 7f6bdf400640 1 -- 172.21.15.16:0/3844255852 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f6be001b070 con 0x7f6be81567e0 2024-03-31T15:09:42.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:42.653+0000 7f6bdf400640 1 --2- 172.21.15.16:0/3844255852 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f6bc4068380 0x7f6bc406a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:42.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:42.653+0000 7f6bdf400640 1 -- 172.21.15.16:0/3844255852 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 5029+0+0 (secure 0 0 0) 0x7f6be0085d80 con 0x7f6be81567e0 2024-03-31T15:09:42.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:42.653+0000 7f6be7400640 1 --2- 172.21.15.16:0/3844255852 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f6bc4068380 0x7f6bc406a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:42.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:42.657+0000 7f6be7400640 1 --2- 172.21.15.16:0/3844255852 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f6bc4068380 0x7f6bc406a840 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f6bd8006fd0 tx=0x7f6bd8008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:09:42.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:42.657+0000 7f6bdf400640 1 -- 172.21.15.16:0/3844255852 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f6be00565d0 con 0x7f6be81567e0 2024-03-31T15:09:42.820 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:42.829+0000 7f6bee000640 1 -- 172.21.15.16:0/3844255852 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) -- 0x7f6be806e470 con 0x7f6be81567e0 2024-03-31T15:09:42.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:42.833+0000 7f6bdf400640 1 -- 172.21.15.16:0/3844255852 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v17) ==== 104+0+0 (secure 0 0 0) 0x7f6be0055ed0 con 0x7f6be81567e0 2024-03-31T15:09:42.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:42.833+0000 7f6bee000640 1 -- 172.21.15.16:0/3844255852 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f6bc4068380 msgr2=0x7f6bc406a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:42.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:42.833+0000 7f6bee000640 1 --2- 172.21.15.16:0/3844255852 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f6bc4068380 0x7f6bc406a840 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f6bd8006fd0 tx=0x7f6bd8008040 comp rx=0 tx=0).stop 2024-03-31T15:09:42.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:42.833+0000 7f6bee000640 1 -- 172.21.15.16:0/3844255852 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6be81567e0 msgr2=0x7f6be815a920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:42.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:42.833+0000 7f6bee000640 1 --2- 172.21.15.16:0/3844255852 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6be81567e0 0x7f6be815a920 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 crypto rx=0x7f6be000c7a0 tx=0x7f6be000cc60 comp rx=0 tx=0).stop 2024-03-31T15:09:42.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:42.833+0000 7f6bee000640 1 -- 172.21.15.16:0/3844255852 shutdown_connections 2024-03-31T15:09:42.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:42.833+0000 7f6bee000640 1 --2- 172.21.15.16:0/3844255852 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f6bc4068380 0x7f6bc406a840 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:42.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:42.833+0000 7f6bee000640 1 --2- 172.21.15.16:0/3844255852 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6be81567e0 0x7f6be815a920 unknown :-1 s=CLOSED pgs=1034 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:42.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:42.833+0000 7f6bee000640 1 --2- 172.21.15.16:0/3844255852 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f6be80731c0 0x7f6be815a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:42.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:42.833+0000 7f6bee000640 1 --2- 172.21.15.16:0/3844255852 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6be806da10 0x7f6be81449a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:42.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:42.833+0000 7f6bee000640 1 -- 172.21.15.16:0/3844255852 >> 172.21.15.16:0/3844255852 conn(0x7f6be806ee40 msgr2=0x7f6be80affb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:09:42.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:42.833+0000 7f6bee000640 1 -- 172.21.15.16:0/3844255852 shutdown_connections 2024-03-31T15:09:42.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:42.833+0000 7f6bee000640 1 -- 172.21.15.16:0/3844255852 wait complete. 2024-03-31T15:09:42.845 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_health_status_after_dentry_repair' 2024-03-31T15:09:42.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:42.993+0000 7f0467e00640 1 Processor -- start 2024-03-31T15:09:42.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:42.993+0000 7f0467e00640 1 -- start start 2024-03-31T15:09:42.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:42.997+0000 7f0467e00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f046015b3d0 0x7f04600b1cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:42.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:42.997+0000 7f0467e00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f04600ce2f0 0x7f04600b2200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:42.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:42.997+0000 7f0467e00640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f046015f3c0 con 0x7f046015eee0 2024-03-31T15:09:42.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:42.997+0000 7f0467e00640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f046015b7b0 con 0x7f04600ce2f0 2024-03-31T15:09:42.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:42.997+0000 7f0467e00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f046015f6a0 con 0x7f046015b3d0 2024-03-31T15:09:42.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:42.997+0000 7f0465e00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f04600ce2f0 0x7f04600b2200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:42.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:42.997+0000 7f0465e00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f04600ce2f0 0x7f04600b2200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59830/0 (socket says 172.21.15.16:59830) 2024-03-31T15:09:42.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:42.997+0000 7f0465e00640 1 -- 172.21.15.16:0/688551727 learned_addr learned my addr 172.21.15.16:0/688551727 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:09:42.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:42.997+0000 7f0465400640 1 --2- 172.21.15.16:0/688551727 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f046015b3d0 0x7f04600b1cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:42.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:42.997+0000 7f0465e00640 1 -- 172.21.15.16:0/688551727 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f046015b3d0 msgr2=0x7f04600b1cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:09:42.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:42.997+0000 7f0465e00640 1 --2- 172.21.15.16:0/688551727 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f046015b3d0 0x7f04600b1cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:42.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:42.997+0000 7f0465e00640 1 -- 172.21.15.16:0/688551727 >> v1:172.21.15.89:6789/0 conn(0x7f046015eee0 legacy=0x7f04600cc9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:42.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:42.997+0000 7f0465e00640 1 -- 172.21.15.16:0/688551727 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f046015f980 con 0x7f04600ce2f0 2024-03-31T15:09:42.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:43.001+0000 7f0465e00640 1 --2- 172.21.15.16:0/688551727 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f04600ce2f0 0x7f04600b2200 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 crypto rx=0x7f0458009e50 tx=0x7f045801ca00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40fcef533795f040 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:09:42.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:43.001+0000 7f045fe00640 1 -- 172.21.15.16:0/688551727 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04580a5890 con 0x7f04600ce2f0 2024-03-31T15:09:42.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:43.001+0000 7f045fe00640 1 -- 172.21.15.16:0/688551727 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f04580a5a30 con 0x7f04600ce2f0 2024-03-31T15:09:42.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:43.001+0000 7f045fe00640 1 -- 172.21.15.16:0/688551727 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04580a5e00 con 0x7f04600ce2f0 2024-03-31T15:09:42.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:43.001+0000 7f0467e00640 1 -- 172.21.15.16:0/688551727 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f04600ce2f0 msgr2=0x7f04600b2200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:42.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:43.001+0000 7f0467e00640 1 --2- 172.21.15.16:0/688551727 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f04600ce2f0 0x7f04600b2200 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 crypto rx=0x7f0458009e50 tx=0x7f045801ca00 comp rx=0 tx=0).stop 2024-03-31T15:09:42.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:43.001+0000 7f0467e00640 1 -- 172.21.15.16:0/688551727 shutdown_connections 2024-03-31T15:09:42.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:43.001+0000 7f0467e00640 1 --2- 172.21.15.16:0/688551727 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f04600ce2f0 0x7f04600b2200 unknown :-1 s=CLOSED pgs=1035 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:42.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:43.001+0000 7f0467e00640 1 --2- 172.21.15.16:0/688551727 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f046015b3d0 0x7f04600b1cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:42.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:43.001+0000 7f0467e00640 1 -- 172.21.15.16:0/688551727 >> 172.21.15.16:0/688551727 conn(0x7f046006b620 msgr2=0x7f046006ba30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:09:42.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:43.001+0000 7f0467e00640 1 -- 172.21.15.16:0/688551727 shutdown_connections 2024-03-31T15:09:42.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:43.001+0000 7f0467e00640 1 -- 172.21.15.16:0/688551727 wait complete. 2024-03-31T15:09:42.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:43.001+0000 7f0467e00640 1 Processor -- start 2024-03-31T15:09:42.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:43.001+0000 7f0467e00640 1 -- start start 2024-03-31T15:09:42.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:43.001+0000 7f0467e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f04600ce2f0 0x7f046015ca30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:42.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:43.001+0000 7f0467e00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f046015b3d0 0x7f04600d0950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:42.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:43.001+0000 7f0467e00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f046015eee0 0x7f04600b12f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:42.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:43.001+0000 7f0467e00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f0460168610 con 0x7f04600ce2f0 2024-03-31T15:09:42.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:43.001+0000 7f0467e00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f0460168490 con 0x7f046015b3d0 2024-03-31T15:09:42.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:43.001+0000 7f0467e00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f0460168790 con 0x7f046015eee0 2024-03-31T15:09:42.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:43.005+0000 7f0465400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f04600ce2f0 0x7f046015ca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:42.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:43.005+0000 7f0465400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f04600ce2f0 0x7f046015ca30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59836/0 (socket says 172.21.15.16:59836) 2024-03-31T15:09:42.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:43.005+0000 7f0465400640 1 -- 172.21.15.16:0/4282403728 learned_addr learned my addr 172.21.15.16:0/4282403728 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:09:42.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:43.005+0000 7f0464a00640 1 --2- 172.21.15.16:0/4282403728 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f046015b3d0 0x7f04600d0950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:42.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:43.005+0000 7f0465e00640 1 --2- 172.21.15.16:0/4282403728 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f046015eee0 0x7f04600b12f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:42.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:43.005+0000 7f0465400640 1 -- 172.21.15.16:0/4282403728 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f046015eee0 msgr2=0x7f04600b12f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:42.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:43.005+0000 7f0465400640 1 --2- 172.21.15.16:0/4282403728 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f046015eee0 0x7f04600b12f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:42.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:43.005+0000 7f0465400640 1 -- 172.21.15.16:0/4282403728 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f046015b3d0 msgr2=0x7f04600d0950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:42.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:43.005+0000 7f0465400640 1 --2- 172.21.15.16:0/4282403728 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f046015b3d0 0x7f04600d0950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:42.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:43.005+0000 7f0465400640 1 -- 172.21.15.16:0/4282403728 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f046015cf70 con 0x7f04600ce2f0 2024-03-31T15:09:42.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:43.005+0000 7f0465400640 1 --2- 172.21.15.16:0/4282403728 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f04600ce2f0 0x7f046015ca30 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 crypto rx=0x7f045400d5f0 tx=0x7f045400dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:09:42.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:43.005+0000 7f045d600640 1 -- 172.21.15.16:0/4282403728 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04540125f0 con 0x7f04600ce2f0 2024-03-31T15:09:42.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:43.005+0000 7f045d600640 1 -- 172.21.15.16:0/4282403728 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0454019070 con 0x7f04600ce2f0 2024-03-31T15:09:42.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:43.005+0000 7f0467e00640 1 -- 172.21.15.16:0/4282403728 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f04600d0e90 con 0x7f04600ce2f0 2024-03-31T15:09:42.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:43.005+0000 7f045d600640 1 -- 172.21.15.16:0/4282403728 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04540129d0 con 0x7f04600ce2f0 2024-03-31T15:09:42.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:43.005+0000 7f0467e00640 1 -- 172.21.15.16:0/4282403728 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0460156840 con 0x7f04600ce2f0 2024-03-31T15:09:42.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:43.005+0000 7f0467e00640 1 -- 172.21.15.16:0/4282403728 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0460069950 con 0x7f04600ce2f0 2024-03-31T15:09:42.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:43.005+0000 7f045d600640 1 -- 172.21.15.16:0/4282403728 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f0454012b70 con 0x7f04600ce2f0 2024-03-31T15:09:43.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:43.005+0000 7f045d600640 1 --2- 172.21.15.16:0/4282403728 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f04400683b0 0x7f044006a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:43.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:43.009+0000 7f045d600640 1 -- 172.21.15.16:0/4282403728 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 5029+0+0 (secure 0 0 0) 0x7f0454086a30 con 0x7f04600ce2f0 2024-03-31T15:09:43.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:43.009+0000 7f0464a00640 1 --2- 172.21.15.16:0/4282403728 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f04400683b0 0x7f044006a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:43.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:43.009+0000 7f0464a00640 1 --2- 172.21.15.16:0/4282403728 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f04400683b0 0x7f044006a870 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f04600cfe00 tx=0x7f044c009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:09:43.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:43.013+0000 7f045d600640 1 -- 172.21.15.16:0/4282403728 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f0454056c40 con 0x7f04600ce2f0 2024-03-31T15:09:43.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:43.241+0000 7f0467e00640 1 -- 172.21.15.16:0/4282403728 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_health_status_after_dentry_repair"]} v 0) -- 0x7f04600ce6d0 con 0x7f04600ce2f0 2024-03-31T15:09:43.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:43.481+0000 7f045d600640 1 -- 172.21.15.16:0/4282403728 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_health_status_after_dentry_repair"]}]=0 v303) ==== 166+0+0 (secure 0 0 0) 0x7f045405af40 con 0x7f04600ce2f0 2024-03-31T15:09:43.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:43.485+0000 7f0467e00640 1 -- 172.21.15.16:0/4282403728 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f04400683b0 msgr2=0x7f044006a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:43.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:43.485+0000 7f0467e00640 1 --2- 172.21.15.16:0/4282403728 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f04400683b0 0x7f044006a870 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f04600cfe00 tx=0x7f044c009040 comp rx=0 tx=0).stop 2024-03-31T15:09:43.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:43.485+0000 7f0467e00640 1 -- 172.21.15.16:0/4282403728 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f04600ce2f0 msgr2=0x7f046015ca30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:43.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:43.485+0000 7f0467e00640 1 --2- 172.21.15.16:0/4282403728 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f04600ce2f0 0x7f046015ca30 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 crypto rx=0x7f045400d5f0 tx=0x7f045400dab0 comp rx=0 tx=0).stop 2024-03-31T15:09:43.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:43.485+0000 7f0467e00640 1 -- 172.21.15.16:0/4282403728 shutdown_connections 2024-03-31T15:09:43.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:43.485+0000 7f0467e00640 1 --2- 172.21.15.16:0/4282403728 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f04400683b0 0x7f044006a870 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:43.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:43.485+0000 7f0467e00640 1 --2- 172.21.15.16:0/4282403728 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f046015eee0 0x7f04600b12f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:43.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:43.485+0000 7f0467e00640 1 --2- 172.21.15.16:0/4282403728 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f046015b3d0 0x7f04600d0950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:43.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:43.485+0000 7f0467e00640 1 --2- 172.21.15.16:0/4282403728 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f04600ce2f0 0x7f046015ca30 unknown :-1 s=CLOSED pgs=1036 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:43.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:43.485+0000 7f0467e00640 1 -- 172.21.15.16:0/4282403728 >> 172.21.15.16:0/4282403728 conn(0x7f046006b620 msgr2=0x7f0460159fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:09:43.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:43.485+0000 7f0467e00640 1 -- 172.21.15.16:0/4282403728 shutdown_connections 2024-03-31T15:09:43.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:43.485+0000 7f0467e00640 1 -- 172.21.15.16:0/4282403728 wait complete. 2024-03-31T15:09:43.495 INFO:tasks.cephfs_test_runner:Test that the damage health status is cleared ... ok 2024-03-31T15:09:43.496 INFO:tasks.cephfs_test_runner:Starting test: test_health_status_after_dirfrag_repair (tasks.cephfs.test_forward_scrub.TestForwardScrub) Test that the damage health status is cleared 2024-03-31T15:09:43.496 INFO:tasks.cephfs_test_runner:test_health_status_after_dirfrag_repair (tasks.cephfs.test_forward_scrub.TestForwardScrub) 2024-03-31T15:09:43.496 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-31T15:09:43.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:43.645+0000 7f83a7e00640 1 Processor -- start 2024-03-31T15:09:43.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:43.645+0000 7f83a7e00640 1 -- start start 2024-03-31T15:09:43.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:43.645+0000 7f83a7e00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f83a815e3b0 0x7f83a809aa60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:43.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:43.645+0000 7f83a7e00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f83a80b1360 0x7f83a809afa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:43.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:43.645+0000 7f83a7e00640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f83a80d0800 con 0x7f83a806fd40 2024-03-31T15:09:43.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:43.645+0000 7f83a7e00640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f83a815e790 con 0x7f83a815e3b0 2024-03-31T15:09:43.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:43.645+0000 7f83a7e00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f83a806f5d0 con 0x7f83a80b1360 2024-03-31T15:09:43.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:43.645+0000 7f83a6a00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f83a815e3b0 0x7f83a809aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:43.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:43.645+0000 7f83a7400640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f83a806fd40 0x7f83a80701d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:55216/0 (socket says 172.21.15.16:55216) 2024-03-31T15:09:43.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:43.649+0000 7f83a7400640 1 -- 172.21.15.16:0/3535281468 learned_addr learned my addr 172.21.15.16:0/3535281468 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:09:43.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:43.649+0000 7f839e000640 1 --2- 172.21.15.16:0/3535281468 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f83a80b1360 0x7f83a809afa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:43.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:43.649+0000 7f83a6a00640 1 -- 172.21.15.16:0/3535281468 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f83a80b1360 msgr2=0x7f83a809afa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:09:43.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:43.649+0000 7f83a6a00640 1 --2- 172.21.15.16:0/3535281468 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f83a80b1360 0x7f83a809afa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:43.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:43.649+0000 7f83a6a00640 1 -- 172.21.15.16:0/3535281468 >> v1:172.21.15.16:6789/0 conn(0x7f83a806fd40 legacy=0x7f83a80701d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:43.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:43.649+0000 7f83a6a00640 1 -- 172.21.15.16:0/3535281468 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83a80d0ae0 con 0x7f83a815e3b0 2024-03-31T15:09:43.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:43.649+0000 7f83a6a00640 1 --2- 172.21.15.16:0/3535281468 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f83a815e3b0 0x7f83a809aa60 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 crypto rx=0x7f83980097c0 tx=0x7f839801c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27b4122d1f5af408 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:09:43.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:43.649+0000 7f83a6000640 1 -- 172.21.15.16:0/3535281468 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83980b0070 con 0x7f83a815e3b0 2024-03-31T15:09:43.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:43.649+0000 7f83a6000640 1 -- 172.21.15.16:0/3535281468 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f83980ab600 con 0x7f83a815e3b0 2024-03-31T15:09:43.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:43.649+0000 7f83a6000640 1 -- 172.21.15.16:0/3535281468 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83980ab9d0 con 0x7f83a815e3b0 2024-03-31T15:09:43.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:43.653+0000 7f83a7e00640 1 -- 172.21.15.16:0/3535281468 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f83a815e3b0 msgr2=0x7f83a809aa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:43.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:43.653+0000 7f83a7e00640 1 --2- 172.21.15.16:0/3535281468 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f83a815e3b0 0x7f83a809aa60 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 crypto rx=0x7f83980097c0 tx=0x7f839801c330 comp rx=0 tx=0).stop 2024-03-31T15:09:43.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:43.653+0000 7f83a7e00640 1 -- 172.21.15.16:0/3535281468 shutdown_connections 2024-03-31T15:09:43.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:43.653+0000 7f83a7e00640 1 --2- 172.21.15.16:0/3535281468 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f83a80b1360 0x7f83a809afa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:43.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:43.653+0000 7f83a7e00640 1 --2- 172.21.15.16:0/3535281468 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f83a815e3b0 0x7f83a809aa60 unknown :-1 s=CLOSED pgs=1038 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:43.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:43.653+0000 7f83a7e00640 1 -- 172.21.15.16:0/3535281468 >> 172.21.15.16:0/3535281468 conn(0x7f83a80b39c0 msgr2=0x7f83a80b3dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:09:43.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:43.653+0000 7f83a7e00640 1 -- 172.21.15.16:0/3535281468 shutdown_connections 2024-03-31T15:09:43.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:43.653+0000 7f83a7e00640 1 -- 172.21.15.16:0/3535281468 wait complete. 2024-03-31T15:09:43.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:43.653+0000 7f83a7e00640 1 Processor -- start 2024-03-31T15:09:43.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:43.653+0000 7f83a7e00640 1 -- start start 2024-03-31T15:09:43.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:43.653+0000 7f83a7e00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f83a806fd40 0x7f83a81585e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:43.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:43.653+0000 7f83a7e00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f83a80b1360 0x7f83a80b4d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:43.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:43.653+0000 7f83a7e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f83a815e3b0 0x7f83a81f76f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:43.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:43.653+0000 7f83a7e00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f83a80d04b0 con 0x7f83a815e3b0 2024-03-31T15:09:43.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:43.653+0000 7f83a7e00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f83a806f820 con 0x7f83a80b1360 2024-03-31T15:09:43.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:43.653+0000 7f83a7e00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f83a80cdba0 con 0x7f83a806fd40 2024-03-31T15:09:43.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:43.653+0000 7f83a7400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f83a815e3b0 0x7f83a81f76f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:43.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:43.653+0000 7f83a7400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f83a815e3b0 0x7f83a81f76f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59852/0 (socket says 172.21.15.16:59852) 2024-03-31T15:09:43.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:43.653+0000 7f83a7400640 1 -- 172.21.15.16:0/3555908147 learned_addr learned my addr 172.21.15.16:0/3555908147 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:09:43.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:43.653+0000 7f83a6a00640 1 --2- 172.21.15.16:0/3555908147 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f83a806fd40 0x7f83a81585e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:43.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:43.653+0000 7f839e000640 1 --2- 172.21.15.16:0/3555908147 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f83a80b1360 0x7f83a80b4d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:43.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:43.653+0000 7f83a7400640 1 -- 172.21.15.16:0/3555908147 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f83a806fd40 msgr2=0x7f83a81585e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:43.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:43.653+0000 7f83a7400640 1 --2- 172.21.15.16:0/3555908147 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f83a806fd40 0x7f83a81585e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:43.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:43.653+0000 7f83a7400640 1 -- 172.21.15.16:0/3555908147 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f83a80b1360 msgr2=0x7f83a80b4d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:43.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:43.653+0000 7f83a7400640 1 --2- 172.21.15.16:0/3555908147 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f83a80b1360 0x7f83a80b4d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:43.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:43.653+0000 7f83a7400640 1 -- 172.21.15.16:0/3555908147 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83a8158b20 con 0x7f83a815e3b0 2024-03-31T15:09:43.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:43.657+0000 7f83a7400640 1 --2- 172.21.15.16:0/3555908147 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f83a815e3b0 0x7f83a81f76f0 secure :-1 s=READY pgs=1039 cs=0 l=1 rev1=1 crypto rx=0x7f83a000e7b0 tx=0x7f83a000ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:09:43.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:43.657+0000 7f839f400640 1 -- 172.21.15.16:0/3555908147 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83a000f930 con 0x7f83a815e3b0 2024-03-31T15:09:43.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:43.657+0000 7f839f400640 1 -- 172.21.15.16:0/3555908147 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f83a0013070 con 0x7f83a815e3b0 2024-03-31T15:09:43.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:43.657+0000 7f83a7e00640 1 -- 172.21.15.16:0/3555908147 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f83a80c6a20 con 0x7f83a815e3b0 2024-03-31T15:09:43.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:43.657+0000 7f839f400640 1 -- 172.21.15.16:0/3555908147 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83a000fcd0 con 0x7f83a815e3b0 2024-03-31T15:09:43.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:43.657+0000 7f83a7e00640 1 -- 172.21.15.16:0/3555908147 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f83a80c6fe0 con 0x7f83a815e3b0 2024-03-31T15:09:43.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:43.657+0000 7f83a7e00640 1 -- 172.21.15.16:0/3555908147 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f83a804f8c0 con 0x7f83a815e3b0 2024-03-31T15:09:43.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:43.657+0000 7f839f400640 1 -- 172.21.15.16:0/3555908147 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f83a0018470 con 0x7f83a815e3b0 2024-03-31T15:09:43.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:43.657+0000 7f839f400640 1 --2- 172.21.15.16:0/3555908147 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f8384068380 0x7f838406a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:43.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:43.657+0000 7f839f400640 1 -- 172.21.15.16:0/3555908147 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 5029+0+0 (secure 0 0 0) 0x7f83a00891e0 con 0x7f83a815e3b0 2024-03-31T15:09:43.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:43.657+0000 7f83a6a00640 1 --2- 172.21.15.16:0/3555908147 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f8384068380 0x7f838406a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:43.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:43.661+0000 7f83a6a00640 1 --2- 172.21.15.16:0/3555908147 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f8384068380 0x7f838406a840 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f83980ad000 tx=0x7f8398008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:09:43.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:43.661+0000 7f839f400640 1 -- 172.21.15.16:0/3555908147 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f83a0059350 con 0x7f83a815e3b0 2024-03-31T15:09:43.820 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:43.829+0000 7f83a7e00640 1 -- 172.21.15.16:0/3555908147 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f83a80b7d50 con 0x7f83a815e3b0 2024-03-31T15:09:43.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:43.829+0000 7f839f400640 1 -- 172.21.15.16:0/3555908147 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v73) ==== 74+0+13079 (secure 0 0 0) 0x7f83a005d650 con 0x7f83a815e3b0 2024-03-31T15:09:43.821 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:09:43.821 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":73,"fsid":"286eb5da-fb47-4550-b290-d9519c13b9a5","created":"2024-03-31T15:03:11.438952+0000","modified":"2024-03-31T15:09:42.467602+0000","last_up_change":"2024-03-31T15:03:22.885535+0000","last_in_change":"2024-03-31T15:03:13.941667+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-31T15:03:22.550787+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"92a66450-ef19-40d8-ac3d-d760b23b0bda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6801","nonce":932712244}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6803","nonce":932712244}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6807","nonce":932712244}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6805","nonce":932712244}]},"public_addr":"172.21.15.16:6801/932712244","cluster_addr":"172.21.15.16:6803/932712244","heartbeat_back_addr":"172.21.15.16:6807/932712244","heartbeat_front_addr":"172.21.15.16:6805/932712244","state":["exists","up"]},{"osd":1,"uuid":"da9d1f2b-1968-4959-b843-faa231988e92","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6810","nonce":1398737533}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6811","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6813","nonce":1398737533}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6821","nonce":1398737533}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6815","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6816","nonce":1398737533}]},"public_addr":"172.21.15.16:6810/1398737533","cluster_addr":"172.21.15.16:6813/1398737533","heartbeat_back_addr":"172.21.15.16:6821/1398737533","heartbeat_front_addr":"172.21.15.16:6816/1398737533","state":["exists","up"]},{"osd":2,"uuid":"01a73662-9a21-4b1f-bcbb-42b9636a4638","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6819","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6822","nonce":3018526646}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6825","nonce":3018526646}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6831","nonce":3018526646}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6827","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6829","nonce":3018526646}]},"public_addr":"172.21.15.16:6822/3018526646","cluster_addr":"172.21.15.16:6825/3018526646","heartbeat_back_addr":"172.21.15.16:6831/3018526646","heartbeat_front_addr":"172.21.15.16:6829/3018526646","state":["exists","up"]},{"osd":3,"uuid":"c7771fd3-e18e-4500-ac38-23cf10ebcd75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6809","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6812","nonce":3360352974}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6817","nonce":3360352974}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6828","nonce":3360352974}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6823","nonce":3360352974}]},"public_addr":"172.21.15.16:6812/3360352974","cluster_addr":"172.21.15.16:6817/3360352974","heartbeat_back_addr":"172.21.15.16:6828/3360352974","heartbeat_front_addr":"172.21.15.16:6823/3360352974","state":["exists","up"]},{"osd":4,"uuid":"88260730-1482-4fbd-a40f-cb1124e9df4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6801","nonce":1856844856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6803","nonce":1856844856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6807","nonce":1856844856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6805","nonce":1856844856}]},"public_addr":"172.21.15.89:6801/1856844856","cluster_addr":"172.21.15.89:6803/1856844856","heartbeat_back_addr":"172.21.15.89:6807/1856844856","heartbeat_front_addr":"172.21.15.89:6805/1856844856","state":["exists","up"]},{"osd":5,"uuid":"4cb2c00c-fae0-4dd8-9d27-ec4f0093ccf7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6814","nonce":50395527}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6817","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6820","nonce":50395527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6831","nonce":50395527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6823","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6826","nonce":50395527}]},"public_addr":"172.21.15.89:6814/50395527","cluster_addr":"172.21.15.89:6820/50395527","heartbeat_back_addr":"172.21.15.89:6831/50395527","heartbeat_front_addr":"172.21.15.89:6826/50395527","state":["exists","up"]},{"osd":6,"uuid":"81ce0c19-2554-4cae-acbe-66a05e9e9b9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6809","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6812","nonce":3470471752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6815","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6818","nonce":3470471752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6827","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6829","nonce":3470471752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6821","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6824","nonce":3470471752}]},"public_addr":"172.21.15.89:6812/3470471752","cluster_addr":"172.21.15.89:6818/3470471752","heartbeat_back_addr":"172.21.15.89:6829/3470471752","heartbeat_front_addr":"172.21.15.89:6824/3470471752","state":["exists","up"]},{"osd":7,"uuid":"85e4313b-d37d-442e-8042-153bf20a847d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6811","nonce":1796778630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6813","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6816","nonce":1796778630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6825","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6828","nonce":1796778630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6819","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6822","nonce":1796778630}]},"public_addr":"172.21.15.89:6811/1796778630","cluster_addr":"172.21.15.89:6816/1796778630","heartbeat_back_addr":"172.21.15.89:6828/1796778630","heartbeat_front_addr":"172.21.15.89:6822/1796778630","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.439088+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.529970+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.680512+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:21.044161+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.582111+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.536558+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.409522+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.590229+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:6835/2645389320":"2024-04-01T15:09:39.198245+0000","172.21.15.16:6834/2645389320":"2024-04-01T15:09:39.198245+0000","172.21.15.16:6835/2555930820":"2024-04-01T15:08:10.537891+0000","172.21.15.16:6834/2555930820":"2024-04-01T15:08:10.537891+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-31T15:09:43.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:43.829+0000 7f83a7e00640 1 -- 172.21.15.16:0/3555908147 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f8384068380 msgr2=0x7f838406a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:43.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:43.829+0000 7f83a7e00640 1 --2- 172.21.15.16:0/3555908147 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f8384068380 0x7f838406a840 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f83980ad000 tx=0x7f8398008040 comp rx=0 tx=0).stop 2024-03-31T15:09:43.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:43.829+0000 7f83a7e00640 1 -- 172.21.15.16:0/3555908147 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f83a815e3b0 msgr2=0x7f83a81f76f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:43.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:43.829+0000 7f83a7e00640 1 --2- 172.21.15.16:0/3555908147 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f83a815e3b0 0x7f83a81f76f0 secure :-1 s=READY pgs=1039 cs=0 l=1 rev1=1 crypto rx=0x7f83a000e7b0 tx=0x7f83a000ec70 comp rx=0 tx=0).stop 2024-03-31T15:09:43.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:43.833+0000 7f83a7e00640 1 -- 172.21.15.16:0/3555908147 shutdown_connections 2024-03-31T15:09:43.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:43.833+0000 7f83a7e00640 1 --2- 172.21.15.16:0/3555908147 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f8384068380 0x7f838406a840 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:43.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:43.833+0000 7f83a7e00640 1 --2- 172.21.15.16:0/3555908147 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f83a815e3b0 0x7f83a81f76f0 unknown :-1 s=CLOSED pgs=1039 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:43.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:43.833+0000 7f83a7e00640 1 --2- 172.21.15.16:0/3555908147 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f83a80b1360 0x7f83a80b4d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:43.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:43.833+0000 7f83a7e00640 1 --2- 172.21.15.16:0/3555908147 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f83a806fd40 0x7f83a81585e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:43.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:43.833+0000 7f83a7e00640 1 -- 172.21.15.16:0/3555908147 >> 172.21.15.16:0/3555908147 conn(0x7f83a80b39c0 msgr2=0x7f83a80b3da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:09:43.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:43.833+0000 7f83a7e00640 1 -- 172.21.15.16:0/3555908147 shutdown_connections 2024-03-31T15:09:43.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:43.833+0000 7f83a7e00640 1 -- 172.21.15.16:0/3555908147 wait complete. 2024-03-31T15:09:43.841 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-31T15:03:22.550787+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-31T15:09:43.842 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-31T15:09:43.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:43.993+0000 7f2035a00640 1 Processor -- start 2024-03-31T15:09:43.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:43.993+0000 7f2035a00640 1 -- start start 2024-03-31T15:09:43.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:43.993+0000 7f2035a00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f20301567e0 0x7f2030072c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:43.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:43.993+0000 7f2035a00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f20300731c0 0x7f203006d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:43.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:43.993+0000 7f2035a00640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f203015c7e0 con 0x7f203006da10 2024-03-31T15:09:43.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:43.993+0000 7f2035a00640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f203006de80 con 0x7f20301567e0 2024-03-31T15:09:43.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:43.993+0000 7f2035a00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f20300736b0 con 0x7f20300731c0 2024-03-31T15:09:43.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:43.993+0000 7f202f400640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f203006da10 0x7f203015fdd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:55226/0 (socket says 172.21.15.16:55226) 2024-03-31T15:09:43.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:43.993+0000 7f202f400640 1 -- 172.21.15.16:0/3197995388 learned_addr learned my addr 172.21.15.16:0/3197995388 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:09:43.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:43.997+0000 7f202ea00640 1 --2- 172.21.15.16:0/3197995388 >> v2:172.21.15.89:3300/0 conn(0x7f20301567e0 0x7f2030072c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:43.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:43.997+0000 7f2027e00640 1 --2- 172.21.15.16:0/3197995388 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f20300731c0 0x7f203006d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:43.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:43.997+0000 7f202e000640 1 -- 172.21.15.16:0/3197995388 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2777154374 0 0) 0x7f203015c7e0 con 0x7f203006da10 2024-03-31T15:09:43.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:43.997+0000 7f202e000640 1 -- 172.21.15.16:0/3197995388 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2014003530 con 0x7f203006da10 2024-03-31T15:09:43.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:43.997+0000 7f2027e00640 1 -- 172.21.15.16:0/3197995388 >> v2:172.21.15.89:3300/0 conn(0x7f20301567e0 msgr2=0x7f2030072c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:09:43.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:43.997+0000 7f2027e00640 1 --2- 172.21.15.16:0/3197995388 >> v2:172.21.15.89:3300/0 conn(0x7f20301567e0 0x7f2030072c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:43.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:43.997+0000 7f2027e00640 1 -- 172.21.15.16:0/3197995388 >> v1:172.21.15.16:6789/0 conn(0x7f203006da10 legacy=0x7f203015fdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:43.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:43.997+0000 7f2027e00640 1 -- 172.21.15.16:0/3197995388 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f203015caa0 con 0x7f20300731c0 2024-03-31T15:09:43.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:43.997+0000 7f2027e00640 1 --2- 172.21.15.16:0/3197995388 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f20300731c0 0x7f203006d4d0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f20180097c0 tx=0x7f201801c4a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=efc95fa75c2c2504 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:09:43.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:43.997+0000 7f202e000640 1 -- 172.21.15.16:0/3197995388 <== mon.2 v2:172.21.15.89:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20180b0070 con 0x7f20300731c0 2024-03-31T15:09:43.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:43.997+0000 7f202e000640 1 -- 172.21.15.16:0/3197995388 <== mon.2 v2:172.21.15.89:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f20180a4e50 con 0x7f20300731c0 2024-03-31T15:09:43.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:43.997+0000 7f202e000640 1 -- 172.21.15.16:0/3197995388 <== mon.2 v2:172.21.15.89:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20180ab800 con 0x7f20300731c0 2024-03-31T15:09:43.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:44.001+0000 7f2035a00640 1 -- 172.21.15.16:0/3197995388 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f20300731c0 msgr2=0x7f203006d4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:43.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:44.001+0000 7f2035a00640 1 --2- 172.21.15.16:0/3197995388 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f20300731c0 0x7f203006d4d0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f20180097c0 tx=0x7f201801c4a0 comp rx=0 tx=0).stop 2024-03-31T15:09:43.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:44.001+0000 7f2035a00640 1 -- 172.21.15.16:0/3197995388 shutdown_connections 2024-03-31T15:09:43.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:44.001+0000 7f2035a00640 1 --2- 172.21.15.16:0/3197995388 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f20300731c0 0x7f203006d4d0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:43.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:44.001+0000 7f2035a00640 1 --2- 172.21.15.16:0/3197995388 >> v2:172.21.15.89:3300/0 conn(0x7f20301567e0 0x7f2030072c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:43.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:44.001+0000 7f2035a00640 1 -- 172.21.15.16:0/3197995388 >> 172.21.15.16:0/3197995388 conn(0x7f203006ee40 msgr2=0x7f203006b620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:09:43.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:44.001+0000 7f2035a00640 1 -- 172.21.15.16:0/3197995388 shutdown_connections 2024-03-31T15:09:43.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:44.001+0000 7f2035a00640 1 -- 172.21.15.16:0/3197995388 wait complete. 2024-03-31T15:09:43.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:44.001+0000 7f2035a00640 1 Processor -- start 2024-03-31T15:09:43.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:44.001+0000 7f2035a00640 1 -- start start 2024-03-31T15:09:43.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:44.001+0000 7f2035a00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f203006da10 0x7f20301472a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:43.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:44.001+0000 7f2035a00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f20300731c0 0x7f20300b3870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:43.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:44.001+0000 7f2035a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f20301567e0 0x7f20300b3db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:43.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:44.001+0000 7f2035a00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f203015d6e0 con 0x7f20301567e0 2024-03-31T15:09:43.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:44.001+0000 7f2035a00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f203015d560 con 0x7f203006da10 2024-03-31T15:09:43.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:44.001+0000 7f2035a00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f2030169170 con 0x7f20300731c0 2024-03-31T15:09:43.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:44.001+0000 7f202f400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f20301567e0 0x7f20300b3db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:43.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:44.001+0000 7f202f400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f20301567e0 0x7f20300b3db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59856/0 (socket says 172.21.15.16:59856) 2024-03-31T15:09:43.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:44.001+0000 7f202f400640 1 -- 172.21.15.16:0/560824656 learned_addr learned my addr 172.21.15.16:0/560824656 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:09:43.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:44.001+0000 7f202ea00640 1 --2- 172.21.15.16:0/560824656 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f203006da10 0x7f20301472a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:43.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:44.001+0000 7f2027e00640 1 --2- 172.21.15.16:0/560824656 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f20300731c0 0x7f20300b3870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:43.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:44.001+0000 7f202f400640 1 -- 172.21.15.16:0/560824656 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f20300731c0 msgr2=0x7f20300b3870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:43.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:44.001+0000 7f202f400640 1 --2- 172.21.15.16:0/560824656 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f20300731c0 0x7f20300b3870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:43.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:44.001+0000 7f202f400640 1 -- 172.21.15.16:0/560824656 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f203006da10 msgr2=0x7f20301472a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:43.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:44.001+0000 7f202f400640 1 --2- 172.21.15.16:0/560824656 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f203006da10 0x7f20301472a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:43.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:44.001+0000 7f202f400640 1 -- 172.21.15.16:0/560824656 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2030147de0 con 0x7f20301567e0 2024-03-31T15:09:43.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:44.005+0000 7f202f400640 1 --2- 172.21.15.16:0/560824656 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f20301567e0 0x7f20300b3db0 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 crypto rx=0x7f202800cad0 tx=0x7f202800cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:09:43.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:44.005+0000 7f2026a00640 1 -- 172.21.15.16:0/560824656 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2028017070 con 0x7f20301567e0 2024-03-31T15:09:43.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:44.005+0000 7f2026a00640 1 -- 172.21.15.16:0/560824656 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f202800ac30 con 0x7f20301567e0 2024-03-31T15:09:43.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:44.005+0000 7f2035a00640 1 -- 172.21.15.16:0/560824656 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f203015b320 con 0x7f20301567e0 2024-03-31T15:09:43.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:44.005+0000 7f2026a00640 1 -- 172.21.15.16:0/560824656 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2028012400 con 0x7f20301567e0 2024-03-31T15:09:43.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:44.005+0000 7f2035a00640 1 -- 172.21.15.16:0/560824656 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f203015b8e0 con 0x7f20301567e0 2024-03-31T15:09:43.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:44.005+0000 7f2035a00640 1 -- 172.21.15.16:0/560824656 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2030069950 con 0x7f20301567e0 2024-03-31T15:09:43.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:44.005+0000 7f2026a00640 1 -- 172.21.15.16:0/560824656 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f202801a070 con 0x7f20301567e0 2024-03-31T15:09:43.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:44.005+0000 7f2026a00640 1 --2- 172.21.15.16:0/560824656 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f200c068380 0x7f200c06a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:43.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:44.005+0000 7f2026a00640 1 -- 172.21.15.16:0/560824656 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 5029+0+0 (secure 0 0 0) 0x7f2028086580 con 0x7f20301567e0 2024-03-31T15:09:43.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:44.009+0000 7f202ea00640 1 --2- 172.21.15.16:0/560824656 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f200c068380 0x7f200c06a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:43.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:44.009+0000 7f202ea00640 1 --2- 172.21.15.16:0/560824656 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f200c068380 0x7f200c06a840 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f2030157310 tx=0x7f2020008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:09:44.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:44.009+0000 7f2026a00640 1 -- 172.21.15.16:0/560824656 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f2030147de0 con 0x7f20301567e0 2024-03-31T15:09:44.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:44.173+0000 7f2035a00640 1 -- 172.21.15.16:0/560824656 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f2030154bc0 con 0x7f20301567e0 2024-03-31T15:09:44.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:44.177+0000 7f2026a00640 1 -- 172.21.15.16:0/560824656 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v73) ==== 93+0+10 (secure 0 0 0) 0x7f2028056d40 con 0x7f20301567e0 2024-03-31T15:09:44.167 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2024-03-31T15:09:44.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:44.177+0000 7f2035a00640 1 -- 172.21.15.16:0/560824656 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f200c068380 msgr2=0x7f200c06a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:44.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:44.177+0000 7f2035a00640 1 --2- 172.21.15.16:0/560824656 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f200c068380 0x7f200c06a840 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f2030157310 tx=0x7f2020008040 comp rx=0 tx=0).stop 2024-03-31T15:09:44.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:44.177+0000 7f2035a00640 1 -- 172.21.15.16:0/560824656 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f20301567e0 msgr2=0x7f20300b3db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:44.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:44.177+0000 7f2035a00640 1 --2- 172.21.15.16:0/560824656 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f20301567e0 0x7f20300b3db0 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 crypto rx=0x7f202800cad0 tx=0x7f202800cf90 comp rx=0 tx=0).stop 2024-03-31T15:09:44.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:44.177+0000 7f2035a00640 1 -- 172.21.15.16:0/560824656 shutdown_connections 2024-03-31T15:09:44.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:44.177+0000 7f2035a00640 1 --2- 172.21.15.16:0/560824656 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f200c068380 0x7f200c06a840 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:44.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:44.177+0000 7f2035a00640 1 --2- 172.21.15.16:0/560824656 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f20301567e0 0x7f20300b3db0 unknown :-1 s=CLOSED pgs=1041 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:44.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:44.177+0000 7f2035a00640 1 --2- 172.21.15.16:0/560824656 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f20300731c0 0x7f20300b3870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:44.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:44.177+0000 7f2035a00640 1 --2- 172.21.15.16:0/560824656 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f203006da10 0x7f20301472a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:44.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:44.177+0000 7f2035a00640 1 -- 172.21.15.16:0/560824656 >> 172.21.15.16:0/560824656 conn(0x7f203006ee40 msgr2=0x7f20300b12d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:09:44.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:44.177+0000 7f2035a00640 1 -- 172.21.15.16:0/560824656 shutdown_connections 2024-03-31T15:09:44.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:44.177+0000 7f2035a00640 1 -- 172.21.15.16:0/560824656 wait complete. 2024-03-31T15:09:44.189 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_health_status_after_dirfrag_repair' 2024-03-31T15:09:44.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:44.337+0000 7efdf2000640 1 Processor -- start 2024-03-31T15:09:44.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:44.337+0000 7efdf2000640 1 -- start start 2024-03-31T15:09:44.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:44.337+0000 7efdf2000640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7efdec15b3d0 0x7efdec0b1cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:44.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:44.337+0000 7efdf2000640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7efdec15eee0 0x7efdec0b2200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:44.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:44.337+0000 7efdf2000640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7efdec0ce2f0 0x7efdec0cc9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:44.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:44.337+0000 7efdf2000640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7efdec15b7b0 con 0x7efdec0ce2f0 2024-03-31T15:09:44.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:44.337+0000 7efdf2000640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7efdec15f3c0 con 0x7efdec15eee0 2024-03-31T15:09:44.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:44.337+0000 7efdf2000640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7efdec15f6a0 con 0x7efdec15b3d0 2024-03-31T15:09:44.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:44.337+0000 7efdebe00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7efdec0ce2f0 0x7efdec0cc9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:44.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:44.337+0000 7efdebe00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7efdec0ce2f0 0x7efdec0cc9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59860/0 (socket says 172.21.15.16:59860) 2024-03-31T15:09:44.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:44.337+0000 7efdebe00640 1 -- 172.21.15.16:0/2277774610 learned_addr learned my addr 172.21.15.16:0/2277774610 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:09:44.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:44.341+0000 7efde2a00640 1 --2- 172.21.15.16:0/2277774610 >> v2:172.21.15.89:3300/0 conn(0x7efdec15eee0 0x7efdec0b2200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:44.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:44.341+0000 7efdeb400640 1 --2- 172.21.15.16:0/2277774610 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7efdec15b3d0 0x7efdec0b1cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:44.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:44.341+0000 7efdebe00640 1 -- 172.21.15.16:0/2277774610 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7efdec15b3d0 msgr2=0x7efdec0b1cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:09:44.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:44.341+0000 7efdebe00640 1 --2- 172.21.15.16:0/2277774610 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7efdec15b3d0 0x7efdec0b1cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:44.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:44.341+0000 7efdebe00640 1 -- 172.21.15.16:0/2277774610 >> v2:172.21.15.89:3300/0 conn(0x7efdec15eee0 msgr2=0x7efdec0b2200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:09:44.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:44.341+0000 7efdebe00640 1 --2- 172.21.15.16:0/2277774610 >> v2:172.21.15.89:3300/0 conn(0x7efdec15eee0 0x7efdec0b2200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:44.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:44.341+0000 7efdebe00640 1 -- 172.21.15.16:0/2277774610 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efdec15f980 con 0x7efdec0ce2f0 2024-03-31T15:09:44.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:44.341+0000 7efde2a00640 1 --2- 172.21.15.16:0/2277774610 >> v2:172.21.15.89:3300/0 conn(0x7efdec15eee0 0x7efdec0b2200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:09:44.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:44.341+0000 7efdeb400640 1 --2- 172.21.15.16:0/2277774610 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7efdec15b3d0 0x7efdec0b1cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:09:44.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:44.341+0000 7efdebe00640 1 --2- 172.21.15.16:0/2277774610 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efdec0ce2f0 0x7efdec0cc9c0 secure :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 crypto rx=0x7efdec0cda90 tx=0x7efde401c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55ce3d52d700919b server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:09:44.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:44.341+0000 7efdeaa00640 1 -- 172.21.15.16:0/2277774610 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efde40a5800 con 0x7efdec0ce2f0 2024-03-31T15:09:44.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:44.341+0000 7efdeaa00640 1 -- 172.21.15.16:0/2277774610 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efde40a59a0 con 0x7efdec0ce2f0 2024-03-31T15:09:44.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:44.341+0000 7efdeaa00640 1 -- 172.21.15.16:0/2277774610 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efde40a5d70 con 0x7efdec0ce2f0 2024-03-31T15:09:44.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:44.341+0000 7efdf2000640 1 -- 172.21.15.16:0/2277774610 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efdec0ce2f0 msgr2=0x7efdec0cc9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:44.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:44.341+0000 7efdf2000640 1 --2- 172.21.15.16:0/2277774610 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efdec0ce2f0 0x7efdec0cc9c0 secure :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 crypto rx=0x7efdec0cda90 tx=0x7efde401c9c0 comp rx=0 tx=0).stop 2024-03-31T15:09:44.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:44.345+0000 7efdf2000640 1 -- 172.21.15.16:0/2277774610 shutdown_connections 2024-03-31T15:09:44.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:44.345+0000 7efdf2000640 1 --2- 172.21.15.16:0/2277774610 >> v2:172.21.15.89:3300/0 conn(0x7efdec15eee0 0x7efdec0b2200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:44.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:44.345+0000 7efdf2000640 1 --2- 172.21.15.16:0/2277774610 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efdec0ce2f0 0x7efdec0cc9c0 unknown :-1 s=CLOSED pgs=1042 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:44.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:44.345+0000 7efdf2000640 1 --2- 172.21.15.16:0/2277774610 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7efdec15b3d0 0x7efdec0b1cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:44.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:44.345+0000 7efdf2000640 1 -- 172.21.15.16:0/2277774610 >> 172.21.15.16:0/2277774610 conn(0x7efdec06b620 msgr2=0x7efdec06ba30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:09:44.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:44.345+0000 7efdf2000640 1 -- 172.21.15.16:0/2277774610 shutdown_connections 2024-03-31T15:09:44.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:44.345+0000 7efdf2000640 1 -- 172.21.15.16:0/2277774610 wait complete. 2024-03-31T15:09:44.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:44.345+0000 7efdf2000640 1 Processor -- start 2024-03-31T15:09:44.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:44.345+0000 7efdf2000640 1 -- start start 2024-03-31T15:09:44.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:44.345+0000 7efdf2000640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efdec0ce2f0 0x7efdec15e0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:44.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:44.345+0000 7efdf2000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efdec15b3d0 0x7efdec1440d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:44.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:44.345+0000 7efdf2000640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7efdec15eee0 0x7efdec0d07f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:44.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:44.345+0000 7efdf2000640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7efdec071e70 con 0x7efdec15b3d0 2024-03-31T15:09:44.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:44.345+0000 7efdf2000640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7efdec071cf0 con 0x7efdec0ce2f0 2024-03-31T15:09:44.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:44.345+0000 7efdf2000640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7efdec071ff0 con 0x7efdec15eee0 2024-03-31T15:09:44.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:44.345+0000 7efde2a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efdec15b3d0 0x7efdec1440d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:44.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:44.345+0000 7efde2a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efdec15b3d0 0x7efdec1440d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59862/0 (socket says 172.21.15.16:59862) 2024-03-31T15:09:44.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:44.345+0000 7efde2a00640 1 -- 172.21.15.16:0/2339917051 learned_addr learned my addr 172.21.15.16:0/2339917051 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:09:44.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:44.345+0000 7efdebe00640 1 --2- 172.21.15.16:0/2339917051 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7efdec15eee0 0x7efdec0d07f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:44.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:44.345+0000 7efdeb400640 1 --2- 172.21.15.16:0/2339917051 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efdec0ce2f0 0x7efdec15e0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:44.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:44.345+0000 7efde2a00640 1 -- 172.21.15.16:0/2339917051 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7efdec15eee0 msgr2=0x7efdec0d07f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:44.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:44.345+0000 7efde2a00640 1 --2- 172.21.15.16:0/2339917051 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7efdec15eee0 0x7efdec0d07f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:44.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:44.345+0000 7efde2a00640 1 -- 172.21.15.16:0/2339917051 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efdec0ce2f0 msgr2=0x7efdec15e0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:44.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:44.345+0000 7efde2a00640 1 --2- 172.21.15.16:0/2339917051 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efdec0ce2f0 0x7efdec15e0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:44.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:44.345+0000 7efde2a00640 1 -- 172.21.15.16:0/2339917051 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efdec15e630 con 0x7efdec15b3d0 2024-03-31T15:09:44.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:44.345+0000 7efdeb400640 1 --2- 172.21.15.16:0/2339917051 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efdec0ce2f0 0x7efdec15e0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:09:44.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:44.345+0000 7efde2a00640 1 --2- 172.21.15.16:0/2339917051 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efdec15b3d0 0x7efdec1440d0 secure :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 crypto rx=0x7efdd400c7f0 tx=0x7efdd400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:09:44.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:44.345+0000 7efde3e00640 1 -- 172.21.15.16:0/2339917051 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdd4015070 con 0x7efdec15b3d0 2024-03-31T15:09:44.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:44.345+0000 7efde3e00640 1 -- 172.21.15.16:0/2339917051 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efdd4007820 con 0x7efdec15b3d0 2024-03-31T15:09:44.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:44.345+0000 7efde3e00640 1 -- 172.21.15.16:0/2339917051 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdd4007c10 con 0x7efdec15b3d0 2024-03-31T15:09:44.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:44.345+0000 7efdf2000640 1 -- 172.21.15.16:0/2339917051 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efdec14b000 con 0x7efdec15b3d0 2024-03-31T15:09:44.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:44.349+0000 7efdf2000640 1 -- 172.21.15.16:0/2339917051 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efdec0d02d0 con 0x7efdec15b3d0 2024-03-31T15:09:44.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:44.349+0000 7efdf2000640 1 -- 172.21.15.16:0/2339917051 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efdec14b3f0 con 0x7efdec15b3d0 2024-03-31T15:09:44.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:44.349+0000 7efde3e00640 1 -- 172.21.15.16:0/2339917051 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7efdd4018070 con 0x7efdec15b3d0 2024-03-31T15:09:44.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:44.349+0000 7efde3e00640 1 --2- 172.21.15.16:0/2339917051 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7efdc8068380 0x7efdc806a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:44.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:44.349+0000 7efde3e00640 1 -- 172.21.15.16:0/2339917051 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 5029+0+0 (secure 0 0 0) 0x7efdd40867d0 con 0x7efdec15b3d0 2024-03-31T15:09:44.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:44.349+0000 7efdeb400640 1 --2- 172.21.15.16:0/2339917051 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7efdc8068380 0x7efdc806a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:44.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:44.349+0000 7efdeb400640 1 --2- 172.21.15.16:0/2339917051 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7efdc8068380 0x7efdc806a840 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7efddc006b20 tx=0x7efddc008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:09:44.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:44.353+0000 7efde3e00640 1 -- 172.21.15.16:0/2339917051 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7efdd40569d0 con 0x7efdec15b3d0 2024-03-31T15:09:44.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:44.577+0000 7efdf2000640 1 -- 172.21.15.16:0/2339917051 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_health_status_after_dirfrag_repair"]} v 0) -- 0x7efdec069950 con 0x7efdec15b3d0 2024-03-31T15:09:45.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:45.489+0000 7efde3e00640 1 -- 172.21.15.16:0/2339917051 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_health_status_after_dirfrag_repair"]}]=0 v305) ==== 170+0+0 (secure 0 0 0) 0x7efdd405acd0 con 0x7efdec15b3d0 2024-03-31T15:09:45.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:45.493+0000 7efdf2000640 1 -- 172.21.15.16:0/2339917051 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7efdc8068380 msgr2=0x7efdc806a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:45.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:45.493+0000 7efdf2000640 1 --2- 172.21.15.16:0/2339917051 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7efdc8068380 0x7efdc806a840 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7efddc006b20 tx=0x7efddc008040 comp rx=0 tx=0).stop 2024-03-31T15:09:45.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:45.493+0000 7efdf2000640 1 -- 172.21.15.16:0/2339917051 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efdec15b3d0 msgr2=0x7efdec1440d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:45.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:45.493+0000 7efdf2000640 1 --2- 172.21.15.16:0/2339917051 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efdec15b3d0 0x7efdec1440d0 secure :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 crypto rx=0x7efdd400c7f0 tx=0x7efdd400ccb0 comp rx=0 tx=0).stop 2024-03-31T15:09:45.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:45.493+0000 7efdf2000640 1 -- 172.21.15.16:0/2339917051 shutdown_connections 2024-03-31T15:09:45.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:45.493+0000 7efdf2000640 1 --2- 172.21.15.16:0/2339917051 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7efdc8068380 0x7efdc806a840 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:45.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:45.493+0000 7efdf2000640 1 --2- 172.21.15.16:0/2339917051 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7efdec15eee0 0x7efdec0d07f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:45.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:45.493+0000 7efdf2000640 1 --2- 172.21.15.16:0/2339917051 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efdec15b3d0 0x7efdec1440d0 unknown :-1 s=CLOSED pgs=1043 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:45.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:45.493+0000 7efdf2000640 1 --2- 172.21.15.16:0/2339917051 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efdec0ce2f0 0x7efdec15e0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:45.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:45.493+0000 7efdf2000640 1 -- 172.21.15.16:0/2339917051 >> 172.21.15.16:0/2339917051 conn(0x7efdec06b620 msgr2=0x7efdec15a2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:09:45.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:45.493+0000 7efdf2000640 1 -- 172.21.15.16:0/2339917051 shutdown_connections 2024-03-31T15:09:45.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:45.493+0000 7efdf2000640 1 -- 172.21.15.16:0/2339917051 wait complete. 2024-03-31T15:09:45.509 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-03-31T15:09:45.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:45.665+0000 7fd5fea00640 1 Processor -- start 2024-03-31T15:09:45.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:45.665+0000 7fd5fea00640 1 -- start start 2024-03-31T15:09:45.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:45.665+0000 7fd5fea00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fd5f806d4d0 0x7fd5f806d930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:45.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:45.665+0000 7fd5fea00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fd5f8071550 0x7fd5f8071960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:45.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:45.665+0000 7fd5fea00640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd5f806de70 con 0x7fd5f81567e0 2024-03-31T15:09:45.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:45.665+0000 7fd5fea00640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7fd5f815c7e0 con 0x7fd5f8071550 2024-03-31T15:09:45.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:45.665+0000 7fd5fea00640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7fd5f80736b0 con 0x7fd5f806d4d0 2024-03-31T15:09:45.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:45.665+0000 7fd5fca00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fd5f8071550 0x7fd5f8071960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:45.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:45.665+0000 7fd5fca00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fd5f8071550 0x7fd5f8071960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59864/0 (socket says 172.21.15.16:59864) 2024-03-31T15:09:45.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:45.665+0000 7fd5fca00640 1 -- 172.21.15.16:0/3744312298 learned_addr learned my addr 172.21.15.16:0/3744312298 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:09:45.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:45.665+0000 7fd5efe00640 1 --2- 172.21.15.16:0/3744312298 >> v2:172.21.15.89:3300/0 conn(0x7fd5f806d4d0 0x7fd5f806d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:45.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:45.665+0000 7fd5fca00640 1 -- 172.21.15.16:0/3744312298 >> v2:172.21.15.89:3300/0 conn(0x7fd5f806d4d0 msgr2=0x7fd5f806d930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:09:45.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:45.665+0000 7fd5fca00640 1 --2- 172.21.15.16:0/3744312298 >> v2:172.21.15.89:3300/0 conn(0x7fd5f806d4d0 0x7fd5f806d930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:45.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:45.665+0000 7fd5fca00640 1 -- 172.21.15.16:0/3744312298 >> v1:172.21.15.89:6789/0 conn(0x7fd5f81567e0 legacy=0x7fd5f8072c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:45.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:45.665+0000 7fd5fca00640 1 -- 172.21.15.16:0/3744312298 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5f815caa0 con 0x7fd5f8071550 2024-03-31T15:09:45.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:45.669+0000 7fd5fca00640 1 --2- 172.21.15.16:0/3744312298 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd5f8071550 0x7fd5f8071960 secure :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 crypto rx=0x7fd5f806c2e0 tx=0x7fd5f001d450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9164b2fc8eb5532b server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:09:45.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:45.669+0000 7fd5f7400640 1 -- 172.21.15.16:0/3744312298 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5f00a57d0 con 0x7fd5f8071550 2024-03-31T15:09:45.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:45.669+0000 7fd5f7400640 1 -- 172.21.15.16:0/3744312298 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd5f00a5970 con 0x7fd5f8071550 2024-03-31T15:09:45.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:45.669+0000 7fd5f7400640 1 -- 172.21.15.16:0/3744312298 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5f00a5d40 con 0x7fd5f8071550 2024-03-31T15:09:45.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:45.669+0000 7fd5fea00640 1 -- 172.21.15.16:0/3744312298 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd5f8071550 msgr2=0x7fd5f8071960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:45.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:45.669+0000 7fd5fea00640 1 --2- 172.21.15.16:0/3744312298 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd5f8071550 0x7fd5f8071960 secure :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 crypto rx=0x7fd5f806c2e0 tx=0x7fd5f001d450 comp rx=0 tx=0).stop 2024-03-31T15:09:45.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:45.669+0000 7fd5fea00640 1 -- 172.21.15.16:0/3744312298 shutdown_connections 2024-03-31T15:09:45.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:45.669+0000 7fd5fea00640 1 --2- 172.21.15.16:0/3744312298 >> v2:172.21.15.89:3300/0 conn(0x7fd5f806d4d0 0x7fd5f806d930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:45.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:45.669+0000 7fd5fea00640 1 --2- 172.21.15.16:0/3744312298 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd5f8071550 0x7fd5f8071960 unknown :-1 s=CLOSED pgs=1044 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:45.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:45.669+0000 7fd5fea00640 1 -- 172.21.15.16:0/3744312298 >> 172.21.15.16:0/3744312298 conn(0x7fd5f806ee40 msgr2=0x7fd5f806b620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:09:45.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:45.669+0000 7fd5fea00640 1 -- 172.21.15.16:0/3744312298 shutdown_connections 2024-03-31T15:09:45.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:45.669+0000 7fd5fea00640 1 -- 172.21.15.16:0/3744312298 wait complete. 2024-03-31T15:09:45.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:45.673+0000 7fd5fea00640 1 Processor -- start 2024-03-31T15:09:45.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:45.673+0000 7fd5fea00640 1 -- start start 2024-03-31T15:09:45.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:45.673+0000 7fd5fea00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd5f806d4d0 0x7fd5f81f7ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:45.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:45.673+0000 7fd5fea00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd5f8071550 0x7fd5f814fed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:45.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:45.673+0000 7fd5fea00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fd5f81567e0 0x7fd5f815e560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:45.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:45.673+0000 7fd5fea00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fd5f80afbb0 con 0x7fd5f8071550 2024-03-31T15:09:45.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:45.673+0000 7fd5fea00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fd5f80afa30 con 0x7fd5f806d4d0 2024-03-31T15:09:45.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:45.673+0000 7fd5fea00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fd5f80722b0 con 0x7fd5f81567e0 2024-03-31T15:09:45.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:45.673+0000 7fd5efe00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd5f8071550 0x7fd5f814fed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:45.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:45.673+0000 7fd5efe00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd5f8071550 0x7fd5f814fed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59866/0 (socket says 172.21.15.16:59866) 2024-03-31T15:09:45.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:45.673+0000 7fd5efe00640 1 -- 172.21.15.16:0/1708287458 learned_addr learned my addr 172.21.15.16:0/1708287458 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:09:45.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:45.673+0000 7fd5f7e00640 1 --2- 172.21.15.16:0/1708287458 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd5f806d4d0 0x7fd5f81f7ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:45.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:45.673+0000 7fd5fca00640 1 --2- 172.21.15.16:0/1708287458 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fd5f81567e0 0x7fd5f815e560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:45.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:45.673+0000 7fd5efe00640 1 -- 172.21.15.16:0/1708287458 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fd5f81567e0 msgr2=0x7fd5f815e560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:45.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:45.673+0000 7fd5efe00640 1 --2- 172.21.15.16:0/1708287458 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fd5f81567e0 0x7fd5f815e560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:45.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:45.673+0000 7fd5efe00640 1 -- 172.21.15.16:0/1708287458 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd5f806d4d0 msgr2=0x7fd5f81f7ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:45.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:45.673+0000 7fd5efe00640 1 --2- 172.21.15.16:0/1708287458 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd5f806d4d0 0x7fd5f81f7ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:45.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:45.673+0000 7fd5efe00640 1 -- 172.21.15.16:0/1708287458 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5f81f8000 con 0x7fd5f8071550 2024-03-31T15:09:45.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:45.673+0000 7fd5efe00640 1 --2- 172.21.15.16:0/1708287458 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd5f8071550 0x7fd5f814fed0 secure :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 crypto rx=0x7fd5e000e930 tx=0x7fd5e000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:09:45.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:45.673+0000 7fd5f4c00640 1 -- 172.21.15.16:0/1708287458 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5e0017070 con 0x7fd5f8071550 2024-03-31T15:09:45.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:45.673+0000 7fd5f4c00640 1 -- 172.21.15.16:0/1708287458 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd5e000ca00 con 0x7fd5f8071550 2024-03-31T15:09:45.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:45.673+0000 7fd5f4c00640 1 -- 172.21.15.16:0/1708287458 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5e000cdb0 con 0x7fd5f8071550 2024-03-31T15:09:45.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:45.673+0000 7fd5fea00640 1 -- 172.21.15.16:0/1708287458 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd5f815f6c0 con 0x7fd5f8071550 2024-03-31T15:09:45.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:45.673+0000 7fd5fea00640 1 -- 172.21.15.16:0/1708287458 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd5f80b35e0 con 0x7fd5f8071550 2024-03-31T15:09:45.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:45.677+0000 7fd5fea00640 1 -- 172.21.15.16:0/1708287458 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd5f8069950 con 0x7fd5f8071550 2024-03-31T15:09:45.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:45.677+0000 7fd5f4c00640 1 -- 172.21.15.16:0/1708287458 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fd5e001a070 con 0x7fd5f8071550 2024-03-31T15:09:45.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:45.677+0000 7fd5f4c00640 1 --2- 172.21.15.16:0/1708287458 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fd5d4068380 0x7fd5d406a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:45.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:45.677+0000 7fd5f4c00640 1 -- 172.21.15.16:0/1708287458 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 5029+0+0 (secure 0 0 0) 0x7fd5e0086740 con 0x7fd5f8071550 2024-03-31T15:09:45.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:45.677+0000 7fd5f7e00640 1 --2- 172.21.15.16:0/1708287458 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fd5d4068380 0x7fd5d406a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:45.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:45.677+0000 7fd5f7e00640 1 --2- 172.21.15.16:0/1708287458 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fd5d4068380 0x7fd5d406a840 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7fd5e8001770 tx=0x7fd5e8009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:09:45.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:45.681+0000 7fd5f4c00640 1 -- 172.21.15.16:0/1708287458 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fd5f81f8000 con 0x7fd5f8071550 2024-03-31T15:09:45.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:45.845+0000 7fd5fea00640 1 -- 172.21.15.16:0/1708287458 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7fd5f81550d0 con 0x7fd5f8071550 2024-03-31T15:09:45.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:45.849+0000 7fd5f4c00640 1 -- 172.21.15.16:0/1708287458 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v18)=0 v18) ==== 122+0+0 (secure 0 0 0) 0x7fd5e0056970 con 0x7fd5f8071550 2024-03-31T15:09:45.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:45.849+0000 7fd5fea00640 1 -- 172.21.15.16:0/1708287458 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fd5d4068380 msgr2=0x7fd5d406a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:45.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:45.849+0000 7fd5fea00640 1 --2- 172.21.15.16:0/1708287458 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fd5d4068380 0x7fd5d406a840 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7fd5e8001770 tx=0x7fd5e8009040 comp rx=0 tx=0).stop 2024-03-31T15:09:45.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:45.849+0000 7fd5fea00640 1 -- 172.21.15.16:0/1708287458 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd5f8071550 msgr2=0x7fd5f814fed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:45.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:45.849+0000 7fd5fea00640 1 --2- 172.21.15.16:0/1708287458 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd5f8071550 0x7fd5f814fed0 secure :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 crypto rx=0x7fd5e000e930 tx=0x7fd5e000edf0 comp rx=0 tx=0).stop 2024-03-31T15:09:45.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:45.849+0000 7fd5fea00640 1 -- 172.21.15.16:0/1708287458 shutdown_connections 2024-03-31T15:09:45.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:45.849+0000 7fd5fea00640 1 --2- 172.21.15.16:0/1708287458 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fd5d4068380 0x7fd5d406a840 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:45.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:45.849+0000 7fd5fea00640 1 --2- 172.21.15.16:0/1708287458 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fd5f81567e0 0x7fd5f815e560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:45.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:45.849+0000 7fd5fea00640 1 --2- 172.21.15.16:0/1708287458 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd5f8071550 0x7fd5f814fed0 unknown :-1 s=CLOSED pgs=1045 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:45.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:45.849+0000 7fd5fea00640 1 --2- 172.21.15.16:0/1708287458 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd5f806d4d0 0x7fd5f81f7ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:45.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:45.849+0000 7fd5fea00640 1 -- 172.21.15.16:0/1708287458 >> 172.21.15.16:0/1708287458 conn(0x7fd5f806ee40 msgr2=0x7fd5f8147760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:09:45.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:45.849+0000 7fd5fea00640 1 -- 172.21.15.16:0/1708287458 shutdown_connections 2024-03-31T15:09:45.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:45.849+0000 7fd5fea00640 1 -- 172.21.15.16:0/1708287458 wait complete. 2024-03-31T15:09:45.862 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-31T15:09:45.862 DEBUG:teuthology.orchestra.run.smithi016:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-31T15:09:45.945 DEBUG:teuthology.orchestra.run.smithi089:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-31T15:09:46.016 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-31T15:09:46.016 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-03-31T15:09:46.016 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-31T15:09:46.025 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16357708k,nr_inodes=4089427,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275996k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=6539 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275992k,nr_inodes=818998,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275996k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-31T15:09:46.025 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-03-31T15:09:46.025 DEBUG:teuthology.orchestra.run.smithi089:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-31T15:09:46.034 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16354452k,nr_inodes=4088613,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275344k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=901 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275340k,nr_inodes=818835,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275344k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-31T15:09:46.035 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-03-31T15:09:46.035 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-31T15:09:46.074 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16357708k,nr_inodes=4089427,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275996k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=6539 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275992k,nr_inodes=818998,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275996k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-31T15:09:46.074 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-03-31T15:09:46.074 DEBUG:teuthology.orchestra.run.smithi089:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-31T15:09:46.085 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16354452k,nr_inodes=4088613,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275344k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=901 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275340k,nr_inodes=818835,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275344k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-31T15:09:46.085 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-03-31T15:09:46.085 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-31T15:09:46.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:46.269+0000 7f79f7e00640 1 Processor -- start 2024-03-31T15:09:46.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:46.269+0000 7f79f7e00640 1 -- start start 2024-03-31T15:09:46.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:46.269+0000 7f79f7e00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f79f0070030 0x7f79f0162180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:46.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:46.269+0000 7f79f7e00640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f79f00cbe90 con 0x7f79f015f650 2024-03-31T15:09:46.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:46.269+0000 7f79f7e00640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f79f00cc170 con 0x7f79f006dbc0 2024-03-31T15:09:46.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:46.269+0000 7f79f7e00640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f79f015f1c0 con 0x7f79f0070030 2024-03-31T15:09:46.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:46.269+0000 7f79f5e00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f79f0070030 0x7f79f0162180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:46.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:46.269+0000 7f79f4a00640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f79f015f650 0x7f79f006f920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:55242/0 (socket says 172.21.15.16:55242) 2024-03-31T15:09:46.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:46.269+0000 7f79f4a00640 1 -- 172.21.15.16:0/953634723 learned_addr learned my addr 172.21.15.16:0/953634723 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:09:46.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:46.269+0000 7f79f5e00640 1 -- 172.21.15.16:0/953634723 >> v1:172.21.15.89:6789/0 conn(0x7f79f006dbc0 legacy=0x7f79f00cb440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:09:46.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:46.269+0000 7f79f5e00640 1 -- 172.21.15.16:0/953634723 >> v1:172.21.15.16:6789/0 conn(0x7f79f015f650 legacy=0x7f79f006f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:46.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:46.269+0000 7f79f5e00640 1 -- 172.21.15.16:0/953634723 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79f015ee50 con 0x7f79f0070030 2024-03-31T15:09:46.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:46.273+0000 7f79f5e00640 1 --2- 172.21.15.16:0/953634723 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f79f0070030 0x7f79f0162180 secure :-1 s=READY pgs=1047 cs=0 l=1 rev1=1 crypto rx=0x7f79e8009d90 tx=0x7f79e801c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55217ab3bd78f1f4 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:09:46.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:46.273+0000 7f79efe00640 1 -- 172.21.15.16:0/953634723 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79e80b2070 con 0x7f79f0070030 2024-03-31T15:09:46.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:46.273+0000 7f79efe00640 1 -- 172.21.15.16:0/953634723 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f79e80a5920 con 0x7f79f0070030 2024-03-31T15:09:46.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:46.273+0000 7f79efe00640 1 -- 172.21.15.16:0/953634723 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79e80a5cf0 con 0x7f79f0070030 2024-03-31T15:09:46.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:46.273+0000 7f79f7e00640 1 -- 172.21.15.16:0/953634723 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f79f0070030 msgr2=0x7f79f0162180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:46.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:46.273+0000 7f79f7e00640 1 --2- 172.21.15.16:0/953634723 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f79f0070030 0x7f79f0162180 secure :-1 s=READY pgs=1047 cs=0 l=1 rev1=1 crypto rx=0x7f79e8009d90 tx=0x7f79e801c8a0 comp rx=0 tx=0).stop 2024-03-31T15:09:46.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:46.273+0000 7f79f7e00640 1 -- 172.21.15.16:0/953634723 shutdown_connections 2024-03-31T15:09:46.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:46.273+0000 7f79f7e00640 1 --2- 172.21.15.16:0/953634723 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f79f0070030 0x7f79f0162180 unknown :-1 s=CLOSED pgs=1047 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:46.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:46.273+0000 7f79f7e00640 1 -- 172.21.15.16:0/953634723 >> 172.21.15.16:0/953634723 conn(0x7f79f0067330 msgr2=0x7f79f0067740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:09:46.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:46.273+0000 7f79f7e00640 1 -- 172.21.15.16:0/953634723 shutdown_connections 2024-03-31T15:09:46.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:46.273+0000 7f79f7e00640 1 -- 172.21.15.16:0/953634723 wait complete. 2024-03-31T15:09:46.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:46.273+0000 7f79f7e00640 1 Processor -- start 2024-03-31T15:09:46.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:46.273+0000 7f79f7e00640 1 -- start start 2024-03-31T15:09:46.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:46.277+0000 7f79f7e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f79f006dbc0 0x7f79f00b32b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:46.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:46.277+0000 7f79f7e00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f79f0070030 0x7f79f01584d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:46.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:46.277+0000 7f79f7e00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f79f015f650 0x7f79f00b6a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:46.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:46.277+0000 7f79f7e00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f79f006c950 con 0x7f79f006dbc0 2024-03-31T15:09:46.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:46.277+0000 7f79f7e00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f79f006dfa0 con 0x7f79f015f650 2024-03-31T15:09:46.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:46.277+0000 7f79f7e00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f79f006cad0 con 0x7f79f0070030 2024-03-31T15:09:46.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:46.277+0000 7f79f5400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f79f006dbc0 0x7f79f00b32b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:46.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:46.277+0000 7f79f5400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f79f006dbc0 0x7f79f00b32b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59890/0 (socket says 172.21.15.16:59890) 2024-03-31T15:09:46.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:46.277+0000 7f79f5400640 1 -- 172.21.15.16:0/3473717197 learned_addr learned my addr 172.21.15.16:0/3473717197 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:09:46.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:46.277+0000 7f79f5400640 1 -- 172.21.15.16:0/3473717197 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f79f0070030 msgr2=0x7f79f01584d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:46.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:46.277+0000 7f79f5400640 1 --2- 172.21.15.16:0/3473717197 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f79f0070030 0x7f79f01584d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:46.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:46.277+0000 7f79f5400640 1 -- 172.21.15.16:0/3473717197 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f79f015f650 msgr2=0x7f79f00b6a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:46.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:46.277+0000 7f79f5400640 1 --2- 172.21.15.16:0/3473717197 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f79f015f650 0x7f79f00b6a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:46.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:46.277+0000 7f79f5400640 1 -- 172.21.15.16:0/3473717197 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79f00b2210 con 0x7f79f006dbc0 2024-03-31T15:09:46.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:46.277+0000 7f79f5400640 1 --2- 172.21.15.16:0/3473717197 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f79f006dbc0 0x7f79f00b32b0 secure :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 crypto rx=0x7f79e400b440 tx=0x7f79e400b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:09:46.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:46.277+0000 7f79ed600640 1 -- 172.21.15.16:0/3473717197 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79e4015020 con 0x7f79f006dbc0 2024-03-31T15:09:46.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:46.277+0000 7f79ed600640 1 -- 172.21.15.16:0/3473717197 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f79e4011890 con 0x7f79f006dbc0 2024-03-31T15:09:46.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:46.277+0000 7f79ed600640 1 -- 172.21.15.16:0/3473717197 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79e4011c80 con 0x7f79f006dbc0 2024-03-31T15:09:46.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:46.277+0000 7f79f7e00640 1 -- 172.21.15.16:0/3473717197 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f79f00b24f0 con 0x7f79f006dbc0 2024-03-31T15:09:46.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:46.277+0000 7f79f7e00640 1 -- 172.21.15.16:0/3473717197 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f79f00b2ab0 con 0x7f79f006dbc0 2024-03-31T15:09:46.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:46.277+0000 7f79f7e00640 1 -- 172.21.15.16:0/3473717197 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f79f0069950 con 0x7f79f006dbc0 2024-03-31T15:09:46.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:46.277+0000 7f79ed600640 1 -- 172.21.15.16:0/3473717197 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f79e4018020 con 0x7f79f006dbc0 2024-03-31T15:09:46.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:46.281+0000 7f79ed600640 1 --2- 172.21.15.16:0/3473717197 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f79d0068380 0x7f79d006a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:46.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:46.281+0000 7f79ed600640 1 -- 172.21.15.16:0/3473717197 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 5029+0+0 (secure 0 0 0) 0x7f79e40865b0 con 0x7f79f006dbc0 2024-03-31T15:09:46.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:46.281+0000 7f79f4a00640 1 --2- 172.21.15.16:0/3473717197 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f79d0068380 0x7f79d006a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:46.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:46.281+0000 7f79f4a00640 1 --2- 172.21.15.16:0/3473717197 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f79d0068380 0x7f79d006a840 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f79dc000f70 tx=0x7f79dc007a90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:09:46.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:46.285+0000 7f79ed600640 1 -- 172.21.15.16:0/3473717197 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f79e401f030 con 0x7f79f006dbc0 2024-03-31T15:09:46.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:46.481+0000 7f79f7e00640 1 -- 172.21.15.16:0/3473717197 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f79f00cf380 con 0x7f79f006dbc0 2024-03-31T15:09:46.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:46.481+0000 7f79ed600640 1 -- 172.21.15.16:0/3473717197 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 54 v54) ==== 94+0+3869 (secure 0 0 0) 0x7f79e400d040 con 0x7f79f006dbc0 2024-03-31T15:09:46.474 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:09:46.475 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":54,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6836/4188793506","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":4188793506},{"type":"v1","addr":"172.21.15.89:6836","nonce":4188793506}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":40},{"gid":5942,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/4105657676","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6835","nonce":4105657676},{"type":"v1","addr":"172.21.15.89:6837","nonce":4105657676}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":40},{"gid":6592,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6833/3986752851","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":3986752851},{"type":"v1","addr":"172.21.15.16:6833","nonce":3986752851}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":39},{"gid":7395,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/2182011577","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":2182011577},{"type":"v1","addr":"172.21.15.16:6835","nonce":2182011577}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":54}],"filesystems":[]} 2024-03-31T15:09:46.475 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 54 2024-03-31T15:09:46.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:46.485+0000 7f79f7e00640 1 -- 172.21.15.16:0/3473717197 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f79d0068380 msgr2=0x7f79d006a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:46.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:46.485+0000 7f79f7e00640 1 --2- 172.21.15.16:0/3473717197 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f79d0068380 0x7f79d006a840 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f79dc000f70 tx=0x7f79dc007a90 comp rx=0 tx=0).stop 2024-03-31T15:09:46.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:46.485+0000 7f79f7e00640 1 -- 172.21.15.16:0/3473717197 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f79f006dbc0 msgr2=0x7f79f00b32b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:46.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:46.485+0000 7f79f7e00640 1 --2- 172.21.15.16:0/3473717197 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f79f006dbc0 0x7f79f00b32b0 secure :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 crypto rx=0x7f79e400b440 tx=0x7f79e400b900 comp rx=0 tx=0).stop 2024-03-31T15:09:46.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:46.485+0000 7f79f7e00640 1 -- 172.21.15.16:0/3473717197 shutdown_connections 2024-03-31T15:09:46.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:46.485+0000 7f79f7e00640 1 --2- 172.21.15.16:0/3473717197 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f79d0068380 0x7f79d006a840 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:46.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:46.485+0000 7f79f7e00640 1 --2- 172.21.15.16:0/3473717197 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f79f015f650 0x7f79f00b6a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:46.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:46.485+0000 7f79f7e00640 1 --2- 172.21.15.16:0/3473717197 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f79f0070030 0x7f79f01584d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:46.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:46.485+0000 7f79f7e00640 1 --2- 172.21.15.16:0/3473717197 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f79f006dbc0 0x7f79f00b32b0 unknown :-1 s=CLOSED pgs=1048 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:46.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:46.485+0000 7f79f7e00640 1 -- 172.21.15.16:0/3473717197 >> 172.21.15.16:0/3473717197 conn(0x7f79f0067330 msgr2=0x7f79f015cb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:09:46.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:46.485+0000 7f79f7e00640 1 -- 172.21.15.16:0/3473717197 shutdown_connections 2024-03-31T15:09:46.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:46.485+0000 7f79f7e00640 1 -- 172.21.15.16:0/3473717197 wait complete. 2024-03-31T15:09:46.496 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-31T15:09:46.497 INFO:tasks.ceph.mds.a:Stopping old one... 2024-03-31T15:09:46.497 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-31T15:09:46.497 INFO:teuthology.orchestra.run:waiting for 300 2024-03-31T15:09:46.497 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-31T15:09:46.497 INFO:tasks.ceph.mds.c:Stopping old one... 2024-03-31T15:09:46.497 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-31T15:09:46.497 INFO:teuthology.orchestra.run:waiting for 300 2024-03-31T15:09:46.497 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-31T15:09:46.498 INFO:tasks.ceph.mds.b:Stopping old one... 2024-03-31T15:09:46.498 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-31T15:09:46.498 INFO:teuthology.orchestra.run:waiting for 300 2024-03-31T15:09:46.498 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-31T15:09:46.498 INFO:tasks.ceph.mds.d:Stopping old one... 2024-03-31T15:09:46.498 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-31T15:09:46.498 INFO:teuthology.orchestra.run:waiting for 300 2024-03-31T15:09:46.509 INFO:tasks.ceph.mds.a:Stopped 2024-03-31T15:09:46.509 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-31T15:09:46.510 INFO:tasks.ceph.mds.c:Stopped 2024-03-31T15:09:46.510 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-31T15:09:46.512 INFO:tasks.ceph.mds.b:Stopped 2024-03-31T15:09:46.512 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-31T15:09:46.512 INFO:tasks.ceph.mds.d:Stopped 2024-03-31T15:09:46.513 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-31T15:09:46.553 INFO:tasks.ceph.mds.a:Started 2024-03-31T15:09:46.553 DEBUG:teuthology.parallel:result is None 2024-03-31T15:09:46.555 INFO:tasks.ceph.mds.c:Started 2024-03-31T15:09:46.555 DEBUG:teuthology.parallel:result is None 2024-03-31T15:09:46.560 INFO:tasks.ceph.mds.b:Started 2024-03-31T15:09:46.560 DEBUG:teuthology.parallel:result is None 2024-03-31T15:09:46.561 INFO:tasks.ceph.mds.d:Started 2024-03-31T15:09:46.562 DEBUG:teuthology.parallel:result is None 2024-03-31T15:09:46.562 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-03-31T15:09:46.641 INFO:tasks.ceph.mds.c.smithi016.stdout:starting mds.c at 2024-03-31T15:09:46.641 INFO:tasks.ceph.mds.a.smithi016.stdout:starting mds.a at 2024-03-31T15:09:46.655 INFO:tasks.ceph.mds.b.smithi089.stdout:starting mds.b at 2024-03-31T15:09:46.655 INFO:tasks.ceph.mds.d.smithi089.stdout:starting mds.d at 2024-03-31T15:09:46.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:46.717+0000 7f79ffa00640 1 Processor -- start 2024-03-31T15:09:46.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:46.717+0000 7f79ffa00640 1 -- start start 2024-03-31T15:09:46.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:46.717+0000 7f79ffa00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f79f815eee0 0x7f79f8072420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:46.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:46.717+0000 7f79ffa00640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f79f815f3c0 con 0x7f79f815b3d0 2024-03-31T15:09:46.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:46.717+0000 7f79ffa00640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f79f815f6a0 con 0x7f79f80b23d0 2024-03-31T15:09:46.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:46.717+0000 7f79ffa00640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f79f815b7b0 con 0x7f79f815eee0 2024-03-31T15:09:46.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:46.717+0000 7f79fd000640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f79f815b3d0 0x7f79f80b1cc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:55272/0 (socket says 172.21.15.16:55272) 2024-03-31T15:09:46.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:46.717+0000 7f79fd000640 1 -- 172.21.15.16:0/3831272638 learned_addr learned my addr 172.21.15.16:0/3831272638 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:09:46.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:46.717+0000 7f79fda00640 1 --2- 172.21.15.16:0/3831272638 >> v2:172.21.15.16:3300/0 conn(0x7f79f815eee0 0x7f79f8072420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:46.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:46.717+0000 7f79fda00640 1 -- 172.21.15.16:0/3831272638 >> v1:172.21.15.89:6789/0 conn(0x7f79f80b23d0 legacy=0x7f79f8071d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:46.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:46.717+0000 7f79f7e00640 1 -- 172.21.15.16:0/3831272638 <== mon.1 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1089368448 0 0) 0x7f79f815f6a0 con 0x7f79f80b23d0 2024-03-31T15:09:46.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:46.717+0000 7f79fda00640 1 -- 172.21.15.16:0/3831272638 >> v1:172.21.15.16:6789/0 conn(0x7f79f815b3d0 legacy=0x7f79f80b1cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:46.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:46.717+0000 7f79fda00640 1 -- 172.21.15.16:0/3831272638 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79f815f980 con 0x7f79f815eee0 2024-03-31T15:09:46.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:46.721+0000 7f79fda00640 1 --2- 172.21.15.16:0/3831272638 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f79f815eee0 0x7f79f8072420 secure :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 crypto rx=0x7f79f0009d90 tx=0x7f79f001c7d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b034abac6315a74c server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:09:46.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:46.721+0000 7f79f7e00640 1 -- 172.21.15.16:0/3831272638 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79f00b2070 con 0x7f79f815eee0 2024-03-31T15:09:46.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:46.721+0000 7f79f7e00640 1 -- 172.21.15.16:0/3831272638 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f79f00a57d0 con 0x7f79f815eee0 2024-03-31T15:09:46.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:46.721+0000 7f79f7e00640 1 -- 172.21.15.16:0/3831272638 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79f00a5ba0 con 0x7f79f815eee0 2024-03-31T15:09:46.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:46.721+0000 7f79ffa00640 1 -- 172.21.15.16:0/3831272638 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f79f815eee0 msgr2=0x7f79f8072420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:46.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:46.721+0000 7f79ffa00640 1 --2- 172.21.15.16:0/3831272638 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f79f815eee0 0x7f79f8072420 secure :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 crypto rx=0x7f79f0009d90 tx=0x7f79f001c7d0 comp rx=0 tx=0).stop 2024-03-31T15:09:46.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:46.721+0000 7f79ffa00640 1 -- 172.21.15.16:0/3831272638 shutdown_connections 2024-03-31T15:09:46.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:46.721+0000 7f79ffa00640 1 --2- 172.21.15.16:0/3831272638 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f79f815eee0 0x7f79f8072420 unknown :-1 s=CLOSED pgs=1055 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:46.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:46.721+0000 7f79ffa00640 1 -- 172.21.15.16:0/3831272638 >> 172.21.15.16:0/3831272638 conn(0x7f79f806b620 msgr2=0x7f79f806ba30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:09:46.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:46.721+0000 7f79ffa00640 1 -- 172.21.15.16:0/3831272638 shutdown_connections 2024-03-31T15:09:46.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:46.721+0000 7f79ffa00640 1 -- 172.21.15.16:0/3831272638 wait complete. 2024-03-31T15:09:46.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:46.725+0000 7f79ffa00640 1 Processor -- start 2024-03-31T15:09:46.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:46.725+0000 7f79ffa00640 1 -- start start 2024-03-31T15:09:46.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:46.725+0000 7f79ffa00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f79f80b23d0 0x7f79f81f7940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:46.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:46.725+0000 7f79ffa00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f79f815b3d0 0x7f79f814edc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:46.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:46.725+0000 7f79ffa00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f79f815eee0 0x7f79f8151580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:46.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:46.725+0000 7f79ffa00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f79f8073580 con 0x7f79f80b23d0 2024-03-31T15:09:46.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:46.725+0000 7f79ffa00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f79f80b4560 con 0x7f79f815eee0 2024-03-31T15:09:46.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:46.725+0000 7f79ffa00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f79f8073700 con 0x7f79f815b3d0 2024-03-31T15:09:46.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:46.725+0000 7f79fd000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f79f80b23d0 0x7f79f81f7940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:46.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:46.725+0000 7f79fd000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f79f80b23d0 0x7f79f81f7940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59916/0 (socket says 172.21.15.16:59916) 2024-03-31T15:09:46.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:46.725+0000 7f79fd000640 1 -- 172.21.15.16:0/1592029184 learned_addr learned my addr 172.21.15.16:0/1592029184 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:09:46.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:46.725+0000 7f79fda00640 1 --2- 172.21.15.16:0/1592029184 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f79f815eee0 0x7f79f8151580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:46.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:46.725+0000 7f79efe00640 1 --2- 172.21.15.16:0/1592029184 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f79f815b3d0 0x7f79f814edc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:46.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:46.725+0000 7f79fd000640 1 -- 172.21.15.16:0/1592029184 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f79f815b3d0 msgr2=0x7f79f814edc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:46.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:46.725+0000 7f79fd000640 1 --2- 172.21.15.16:0/1592029184 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f79f815b3d0 0x7f79f814edc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:46.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:46.725+0000 7f79fd000640 1 -- 172.21.15.16:0/1592029184 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f79f815eee0 msgr2=0x7f79f8151580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:46.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:46.725+0000 7f79fd000640 1 --2- 172.21.15.16:0/1592029184 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f79f815eee0 0x7f79f8151580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:46.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:46.725+0000 7f79fd000640 1 -- 172.21.15.16:0/1592029184 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79f81538a0 con 0x7f79f80b23d0 2024-03-31T15:09:46.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:46.725+0000 7f79efe00640 1 --2- 172.21.15.16:0/1592029184 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f79f815b3d0 0x7f79f814edc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:09:46.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:46.725+0000 7f79fd000640 1 --2- 172.21.15.16:0/1592029184 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f79f80b23d0 0x7f79f81f7940 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 crypto rx=0x7f79e800c780 tx=0x7f79e800cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:09:46.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:46.725+0000 7f79f5600640 1 -- 172.21.15.16:0/1592029184 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79e8018070 con 0x7f79f80b23d0 2024-03-31T15:09:46.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:46.725+0000 7f79f5600640 1 -- 172.21.15.16:0/1592029184 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f79e80072d0 con 0x7f79f80b23d0 2024-03-31T15:09:46.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:46.725+0000 7f79f5600640 1 -- 172.21.15.16:0/1592029184 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79e80047f0 con 0x7f79f80b23d0 2024-03-31T15:09:46.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:46.725+0000 7f79ffa00640 1 -- 172.21.15.16:0/1592029184 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f79f8153b80 con 0x7f79f80b23d0 2024-03-31T15:09:46.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:46.725+0000 7f79ffa00640 1 -- 172.21.15.16:0/1592029184 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f79f815d350 con 0x7f79f80b23d0 2024-03-31T15:09:46.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:46.729+0000 7f79ffa00640 1 -- 172.21.15.16:0/1592029184 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f79f8069950 con 0x7f79f80b23d0 2024-03-31T15:09:46.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:46.729+0000 7f79f5600640 1 -- 172.21.15.16:0/1592029184 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f79e8004990 con 0x7f79f80b23d0 2024-03-31T15:09:46.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:46.729+0000 7f79f5600640 1 --2- 172.21.15.16:0/1592029184 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f79d4068380 0x7f79d406a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:46.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:46.729+0000 7f79f5600640 1 -- 172.21.15.16:0/1592029184 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 5029+0+0 (secure 0 0 0) 0x7f79e8085c40 con 0x7f79f80b23d0 2024-03-31T15:09:46.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:46.729+0000 7f79efe00640 1 --2- 172.21.15.16:0/1592029184 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f79d4068380 0x7f79d406a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:46.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:46.729+0000 7f79efe00640 1 --2- 172.21.15.16:0/1592029184 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f79d4068380 0x7f79d406a840 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f79f8153080 tx=0x7f79e0009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:09:46.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:46.733+0000 7f79f5600640 1 -- 172.21.15.16:0/1592029184 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f79e8056490 con 0x7f79f80b23d0 2024-03-31T15:09:46.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:46.909+0000 7f79ffa00640 1 -- 172.21.15.16:0/1592029184 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) -- 0x7f79f80cdea0 con 0x7f79f80b23d0 2024-03-31T15:09:47.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:47.505+0000 7f79f5600640 1 -- 172.21.15.16:0/1592029184 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v75) ==== 97+0+0 (secure 0 0 0) 0x7f79e8055d90 con 0x7f79f80b23d0 2024-03-31T15:09:47.498 INFO:teuthology.orchestra.run.smithi016.stderr: removed all blocklist entries 2024-03-31T15:09:47.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:47.509+0000 7f79ffa00640 1 -- 172.21.15.16:0/1592029184 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f79d4068380 msgr2=0x7f79d406a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:47.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:47.509+0000 7f79ffa00640 1 --2- 172.21.15.16:0/1592029184 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f79d4068380 0x7f79d406a840 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f79f8153080 tx=0x7f79e0009040 comp rx=0 tx=0).stop 2024-03-31T15:09:47.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:47.509+0000 7f79ffa00640 1 -- 172.21.15.16:0/1592029184 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f79f80b23d0 msgr2=0x7f79f81f7940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:47.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:47.509+0000 7f79ffa00640 1 --2- 172.21.15.16:0/1592029184 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f79f80b23d0 0x7f79f81f7940 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 crypto rx=0x7f79e800c780 tx=0x7f79e800cc40 comp rx=0 tx=0).stop 2024-03-31T15:09:47.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:47.509+0000 7f79ffa00640 1 -- 172.21.15.16:0/1592029184 shutdown_connections 2024-03-31T15:09:47.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:47.509+0000 7f79ffa00640 1 --2- 172.21.15.16:0/1592029184 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f79d4068380 0x7f79d406a840 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:47.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:47.509+0000 7f79ffa00640 1 --2- 172.21.15.16:0/1592029184 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f79f815eee0 0x7f79f8151580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:47.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:47.509+0000 7f79ffa00640 1 --2- 172.21.15.16:0/1592029184 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f79f815b3d0 0x7f79f814edc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:47.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:47.509+0000 7f79ffa00640 1 --2- 172.21.15.16:0/1592029184 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f79f80b23d0 0x7f79f81f7940 unknown :-1 s=CLOSED pgs=1056 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:47.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:47.509+0000 7f79ffa00640 1 -- 172.21.15.16:0/1592029184 >> 172.21.15.16:0/1592029184 conn(0x7f79f806b620 msgr2=0x7f79f80cf0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:09:47.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:47.509+0000 7f79ffa00640 1 -- 172.21.15.16:0/1592029184 shutdown_connections 2024-03-31T15:09:47.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:47.509+0000 7f79ffa00640 1 -- 172.21.15.16:0/1592029184 wait complete. 2024-03-31T15:09:47.528 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-03-31T15:09:47.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:47.676+0000 7f482c000640 1 Processor -- start 2024-03-31T15:09:47.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:47.676+0000 7f482c000640 1 -- start start 2024-03-31T15:09:47.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:47.680+0000 7f482c000640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f482415b3d0 0x7f48240b1cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:47.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:47.680+0000 7f482c000640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f482415eee0 0x7f48240b2200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:47.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:47.680+0000 7f482c000640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f482415f3c0 con 0x7f48240ce2f0 2024-03-31T15:09:47.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:47.680+0000 7f482c000640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f482415b7b0 con 0x7f482415eee0 2024-03-31T15:09:47.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:47.680+0000 7f482c000640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f482415f6a0 con 0x7f482415b3d0 2024-03-31T15:09:47.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:47.680+0000 7f482a000640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f48240ce2f0 0x7f48240cc9c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:48176/0 (socket says 172.21.15.16:48176) 2024-03-31T15:09:47.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:47.680+0000 7f482a000640 1 -- 172.21.15.16:0/4020030978 learned_addr learned my addr 172.21.15.16:0/4020030978 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:09:47.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:47.680+0000 7f4828c00640 1 --2- 172.21.15.16:0/4020030978 >> v2:172.21.15.89:3300/0 conn(0x7f482415eee0 0x7f48240b2200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:47.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:47.680+0000 7f4829600640 1 --2- 172.21.15.16:0/4020030978 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f482415b3d0 0x7f48240b1cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:47.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:47.680+0000 7f4823e00640 1 -- 172.21.15.16:0/4020030978 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3591539746 0 0) 0x7f482415f3c0 con 0x7f48240ce2f0 2024-03-31T15:09:47.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:47.680+0000 7f4823e00640 1 -- 172.21.15.16:0/4020030978 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f480c003530 con 0x7f48240ce2f0 2024-03-31T15:09:47.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:47.680+0000 7f4823e00640 1 -- 172.21.15.16:0/4020030978 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 995547323 0 0) 0x7f480c003530 con 0x7f48240ce2f0 2024-03-31T15:09:47.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:47.680+0000 7f4823e00640 1 -- 172.21.15.16:0/4020030978 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f482415b3d0 msgr2=0x7f48240b1cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:09:47.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:47.680+0000 7f4823e00640 1 --2- 172.21.15.16:0/4020030978 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f482415b3d0 0x7f48240b1cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:47.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:47.680+0000 7f4823e00640 1 -- 172.21.15.16:0/4020030978 >> v2:172.21.15.89:3300/0 conn(0x7f482415eee0 msgr2=0x7f48240b2200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:09:47.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:47.680+0000 7f4823e00640 1 --2- 172.21.15.16:0/4020030978 >> v2:172.21.15.89:3300/0 conn(0x7f482415eee0 0x7f48240b2200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:47.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:47.680+0000 7f4823e00640 1 -- 172.21.15.16:0/4020030978 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f482415f980 con 0x7f48240ce2f0 2024-03-31T15:09:47.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:47.680+0000 7f4823e00640 1 -- 172.21.15.16:0/4020030978 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 214807623 0 0) 0x7f481c001a00 con 0x7f48240ce2f0 2024-03-31T15:09:47.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:47.680+0000 7f4823e00640 1 -- 172.21.15.16:0/4020030978 >> v1:172.21.15.16:6789/0 conn(0x7f48240ce2f0 legacy=0x7f48240cc9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:47.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:47.680+0000 7f4823e00640 1 --2- 172.21.15.16:0/4020030978 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f480c0039c0 0x7f480c003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:47.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:47.680+0000 7f4823e00640 1 --2- 172.21.15.16:0/4020030978 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f480c004810 0x7f480c004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:47.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:47.680+0000 7f4823e00640 1 --2- 172.21.15.16:0/4020030978 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f482415b3d0 0x7f480c0051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:47.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:47.680+0000 7f4823e00640 1 -- 172.21.15.16:0/4020030978 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f482415b7b0 con 0x7f480c004810 2024-03-31T15:09:47.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:47.680+0000 7f4823e00640 1 -- 172.21.15.16:0/4020030978 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f482415f6a0 con 0x7f480c0039c0 2024-03-31T15:09:47.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:47.680+0000 7f4823e00640 1 -- 172.21.15.16:0/4020030978 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f480c00b9d0 con 0x7f482415b3d0 2024-03-31T15:09:47.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:47.680+0000 7f4828c00640 1 --2- 172.21.15.16:0/4020030978 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f480c004810 0x7f480c004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:47.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:47.680+0000 7f4829600640 1 --2- 172.21.15.16:0/4020030978 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f480c0039c0 0x7f480c003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:47.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:47.680+0000 7f482a000640 1 --2- 172.21.15.16:0/4020030978 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f482415b3d0 0x7f480c0051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:47.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:47.680+0000 7f4828c00640 1 -- 172.21.15.16:0/4020030978 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f482415b3d0 msgr2=0x7f480c0051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:09:47.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:47.680+0000 7f4828c00640 1 --2- 172.21.15.16:0/4020030978 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f482415b3d0 0x7f480c0051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:47.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:47.680+0000 7f4828c00640 1 -- 172.21.15.16:0/4020030978 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f480c0039c0 msgr2=0x7f480c003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:09:47.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:47.680+0000 7f4828c00640 1 --2- 172.21.15.16:0/4020030978 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f480c0039c0 0x7f480c003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:47.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:47.680+0000 7f4828c00640 1 -- 172.21.15.16:0/4020030978 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f480c003530 con 0x7f480c004810 2024-03-31T15:09:47.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:47.684+0000 7f4828c00640 1 --2- 172.21.15.16:0/4020030978 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f480c004810 0x7f480c004c90 secure :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 crypto rx=0x7f4810009ae0 tx=0x7f481001ca50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa3c826bac3d7162 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:09:47.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:47.684+0000 7f4823e00640 1 -- 172.21.15.16:0/4020030978 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48100096b0 con 0x7f480c004810 2024-03-31T15:09:47.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:47.684+0000 7f4823e00640 1 -- 172.21.15.16:0/4020030978 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f48100a64c0 con 0x7f480c004810 2024-03-31T15:09:47.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:47.684+0000 7f4823e00640 1 -- 172.21.15.16:0/4020030978 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48100a68b0 con 0x7f480c004810 2024-03-31T15:09:47.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:47.684+0000 7f482c000640 1 -- 172.21.15.16:0/4020030978 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f480c004810 msgr2=0x7f480c004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:47.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:47.684+0000 7f482c000640 1 --2- 172.21.15.16:0/4020030978 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f480c004810 0x7f480c004c90 secure :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 crypto rx=0x7f4810009ae0 tx=0x7f481001ca50 comp rx=0 tx=0).stop 2024-03-31T15:09:47.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:47.684+0000 7f482a000640 1 -- 172.21.15.16:0/4020030978 reap_dead start 2024-03-31T15:09:47.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:47.684+0000 7f482c000640 1 -- 172.21.15.16:0/4020030978 shutdown_connections 2024-03-31T15:09:47.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:47.684+0000 7f482c000640 1 -- 172.21.15.16:0/4020030978 >> 172.21.15.16:0/4020030978 conn(0x7f482406b620 msgr2=0x7f482406ba30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:09:47.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:47.684+0000 7f482c000640 1 -- 172.21.15.16:0/4020030978 shutdown_connections 2024-03-31T15:09:47.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:47.684+0000 7f482c000640 1 -- 172.21.15.16:0/4020030978 wait complete. 2024-03-31T15:09:47.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:47.684+0000 7f482c000640 1 Processor -- start 2024-03-31T15:09:47.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:47.688+0000 7f482c000640 1 -- start start 2024-03-31T15:09:47.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:47.688+0000 7f482c000640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f480c004810 0x7f4824153fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:47.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:47.688+0000 7f482c000640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4824153720 0x7f482415e0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:47.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:47.688+0000 7f482c000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4824155cc0 0x7f4824156150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:47.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:47.688+0000 7f482c000640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f48240b4350 con 0x7f4824155cc0 2024-03-31T15:09:47.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:47.688+0000 7f482c000640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f48240717c0 con 0x7f4824153720 2024-03-31T15:09:47.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:47.688+0000 7f482c000640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f48240b44d0 con 0x7f480c004810 2024-03-31T15:09:47.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:47.688+0000 7f482a000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4824155cc0 0x7f4824156150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:47.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:47.688+0000 7f4829600640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f480c004810 0x7f4824153fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:47.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:47.688+0000 7f482a000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4824155cc0 0x7f4824156150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59352/0 (socket says 172.21.15.16:59352) 2024-03-31T15:09:47.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:47.688+0000 7f482a000640 1 -- 172.21.15.16:0/2359845228 learned_addr learned my addr 172.21.15.16:0/2359845228 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:09:47.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:47.688+0000 7f4829600640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f480c004810 0x7f4824153fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3301/0 says I am v2:172.21.15.16:47578/0 (socket says 172.21.15.16:47578) 2024-03-31T15:09:47.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:47.688+0000 7f4828c00640 1 --2- 172.21.15.16:0/2359845228 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4824153720 0x7f482415e0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:47.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:47.688+0000 7f482a000640 1 -- 172.21.15.16:0/2359845228 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f480c004810 msgr2=0x7f4824153fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:47.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:47.688+0000 7f482a000640 1 --2- 172.21.15.16:0/2359845228 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f480c004810 0x7f4824153fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:47.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:47.688+0000 7f482a000640 1 -- 172.21.15.16:0/2359845228 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4824153720 msgr2=0x7f482415e0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:47.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:47.688+0000 7f482a000640 1 --2- 172.21.15.16:0/2359845228 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4824153720 0x7f482415e0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:47.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:47.688+0000 7f482a000640 1 -- 172.21.15.16:0/2359845228 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48240b08b0 con 0x7f4824155cc0 2024-03-31T15:09:47.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:47.688+0000 7f482a000640 1 --2- 172.21.15.16:0/2359845228 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4824155cc0 0x7f4824156150 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 crypto rx=0x7f48240cc890 tx=0x7f481c0115c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:09:47.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:47.688+0000 7f4821600640 1 -- 172.21.15.16:0/2359845228 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f481c01c070 con 0x7f4824155cc0 2024-03-31T15:09:47.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:47.688+0000 7f4821600640 1 -- 172.21.15.16:0/2359845228 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f481c011e60 con 0x7f4824155cc0 2024-03-31T15:09:47.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:47.688+0000 7f4821600640 1 -- 172.21.15.16:0/2359845228 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f481c010040 con 0x7f4824155cc0 2024-03-31T15:09:47.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:47.688+0000 7f482c000640 1 -- 172.21.15.16:0/2359845228 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f48240b0b50 con 0x7f4824155cc0 2024-03-31T15:09:47.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:47.688+0000 7f482c000640 1 -- 172.21.15.16:0/2359845228 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f48240b1110 con 0x7f4824155cc0 2024-03-31T15:09:47.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:47.688+0000 7f482c000640 1 -- 172.21.15.16:0/2359845228 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4824069950 con 0x7f4824155cc0 2024-03-31T15:09:47.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:47.692+0000 7f4821600640 1 -- 172.21.15.16:0/2359845228 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f481c006070 con 0x7f4824155cc0 2024-03-31T15:09:47.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:47.692+0000 7f4821600640 1 --2- 172.21.15.16:0/2359845228 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f48040682f0 0x7f480406a7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:47.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:47.692+0000 7f4821600640 1 -- 172.21.15.16:0/2359845228 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 4857+0+0 (secure 0 0 0) 0x7f481c089500 con 0x7f4824155cc0 2024-03-31T15:09:47.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:47.692+0000 7f4829600640 1 --2- 172.21.15.16:0/2359845228 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f48040682f0 0x7f480406a7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:47.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:47.692+0000 7f4829600640 1 --2- 172.21.15.16:0/2359845228 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f48040682f0 0x7f480406a7b0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f4818009a80 tx=0x7f4818009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:09:47.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:47.696+0000 7f4821600640 1 -- 172.21.15.16:0/2359845228 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f481c059720 con 0x7f4824155cc0 2024-03-31T15:09:47.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:47.916+0000 7f482c000640 1 -- 172.21.15.16:0/2359845228 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) -- 0x7f48240cdea0 con 0x7f4824155cc0 2024-03-31T15:09:47.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:47.916+0000 7f4821600640 1 -- 172.21.15.16:0/2359845228 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v16) ==== 80+0+5304 (secure 0 0 0) 0x7f481c05da20 con 0x7f4824155cc0 2024-03-31T15:09:47.910 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:09:47.910 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-03-31T15:09:47.910 INFO:teuthology.orchestra.run.smithi016.stdout: "auth_dump": [ 2024-03-31T15:09:47.910 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-31T15:09:47.910 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "mds.a", 2024-03-31T15:09:47.910 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAOewlmhp1ANRAAYS9zo+p4tLU1YK0ySOZE/Q==", 2024-03-31T15:09:47.910 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2024-03-31T15:09:47.910 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2024-03-31T15:09:47.910 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow *", 2024-03-31T15:09:47.911 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow *", 2024-03-31T15:09:47.911 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2024-03-31T15:09:47.911 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-31T15:09:47.911 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-31T15:09:47.911 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-31T15:09:47.911 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "mds.b", 2024-03-31T15:09:47.911 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAPewlmXxupBhAAtpkt93aJOFqhc7GXLIlkBg==", 2024-03-31T15:09:47.911 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2024-03-31T15:09:47.911 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2024-03-31T15:09:47.911 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow *", 2024-03-31T15:09:47.911 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow *", 2024-03-31T15:09:47.911 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2024-03-31T15:09:47.911 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-31T15:09:47.911 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-31T15:09:47.912 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-31T15:09:47.912 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "mds.c", 2024-03-31T15:09:47.912 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAPewlmjaieARAA32teg02vE/Q8RtqLtnArMw==", 2024-03-31T15:09:47.912 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2024-03-31T15:09:47.912 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2024-03-31T15:09:47.912 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow *", 2024-03-31T15:09:47.912 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow *", 2024-03-31T15:09:47.912 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2024-03-31T15:09:47.912 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-31T15:09:47.912 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-31T15:09:47.912 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-31T15:09:47.912 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "mds.d", 2024-03-31T15:09:47.912 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAPewlmo+4PDhAAmJD0siZB3LjWkvdMq1d8aw==", 2024-03-31T15:09:47.912 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2024-03-31T15:09:47.912 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2024-03-31T15:09:47.912 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow *", 2024-03-31T15:09:47.913 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow *", 2024-03-31T15:09:47.913 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2024-03-31T15:09:47.913 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-31T15:09:47.913 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-31T15:09:47.913 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-31T15:09:47.913 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.0", 2024-03-31T15:09:47.913 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAQewlmVP6kLhAAwFr4jwQgz1N3oIUZuJvmzw==", 2024-03-31T15:09:47.913 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2024-03-31T15:09:47.913 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2024-03-31T15:09:47.913 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2024-03-31T15:09:47.913 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2024-03-31T15:09:47.913 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-31T15:09:47.913 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-31T15:09:47.913 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-31T15:09:47.913 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.1", 2024-03-31T15:09:47.913 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQATewlmBh0gNhAAR6LiuFypwA7d3SwKlWuI/Q==", 2024-03-31T15:09:47.914 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2024-03-31T15:09:47.914 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2024-03-31T15:09:47.914 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2024-03-31T15:09:47.914 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2024-03-31T15:09:47.914 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-31T15:09:47.914 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-31T15:09:47.914 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-31T15:09:47.914 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.2", 2024-03-31T15:09:47.914 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAXewlmtQwhARAACbzc7Woo9BoVFPfK3Pns+g==", 2024-03-31T15:09:47.914 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2024-03-31T15:09:47.914 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2024-03-31T15:09:47.914 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2024-03-31T15:09:47.914 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2024-03-31T15:09:47.914 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-31T15:09:47.915 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-31T15:09:47.915 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-31T15:09:47.915 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.3", 2024-03-31T15:09:47.915 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAaewlmmjWABhAA/hwvARNb51nd5vsnYVD/sg==", 2024-03-31T15:09:47.915 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2024-03-31T15:09:47.915 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2024-03-31T15:09:47.915 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2024-03-31T15:09:47.915 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2024-03-31T15:09:47.915 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-31T15:09:47.915 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-31T15:09:47.915 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-31T15:09:47.915 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.4", 2024-03-31T15:09:47.915 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAgewlmby07CxAAChxE/qPuPboIlg3fDxSOeA==", 2024-03-31T15:09:47.915 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2024-03-31T15:09:47.915 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2024-03-31T15:09:47.915 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2024-03-31T15:09:47.916 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2024-03-31T15:09:47.916 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-31T15:09:47.916 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-31T15:09:47.916 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-31T15:09:47.916 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.5", 2024-03-31T15:09:47.916 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAjewlmlD8yExAA9pgP0heBywrgqc39iU8rpQ==", 2024-03-31T15:09:47.916 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2024-03-31T15:09:47.916 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2024-03-31T15:09:47.916 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2024-03-31T15:09:47.916 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2024-03-31T15:09:47.916 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-31T15:09:47.916 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-31T15:09:47.916 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-31T15:09:47.916 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.6", 2024-03-31T15:09:47.916 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAmewlmzAsTGBAALMjVcQ08sIiTDFaBcq8L6A==", 2024-03-31T15:09:47.916 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2024-03-31T15:09:47.917 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2024-03-31T15:09:47.917 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2024-03-31T15:09:47.917 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2024-03-31T15:09:47.917 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-31T15:09:47.917 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-31T15:09:47.917 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-31T15:09:47.917 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.7", 2024-03-31T15:09:47.917 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQApewlmb+CEHRAAXE+iBumDsb2c4zH3C8f6YA==", 2024-03-31T15:09:47.917 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2024-03-31T15:09:47.917 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2024-03-31T15:09:47.917 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2024-03-31T15:09:47.917 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2024-03-31T15:09:47.917 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-31T15:09:47.917 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-31T15:09:47.917 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-31T15:09:47.917 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "client.0", 2024-03-31T15:09:47.918 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAPewlmlMIDEhAAF0Ll1HcrWHmH6xDKq2Qq9A==", 2024-03-31T15:09:47.918 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2024-03-31T15:09:47.918 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2024-03-31T15:09:47.918 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow r", 2024-03-31T15:09:47.918 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-03-31T15:09:47.918 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-31T15:09:47.918 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-31T15:09:47.918 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-31T15:09:47.918 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "client.1", 2024-03-31T15:09:47.918 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAPewlmdsvqGxAAaVdwWwri0IE/2sg7psilWw==", 2024-03-31T15:09:47.918 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2024-03-31T15:09:47.918 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2024-03-31T15:09:47.918 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow r", 2024-03-31T15:09:47.919 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow rw", 2024-03-31T15:09:47.919 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow rwx" 2024-03-31T15:09:47.919 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-31T15:09:47.919 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-31T15:09:47.919 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-31T15:09:47.919 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "client.2", 2024-03-31T15:09:47.919 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAPewlmQapPGBAAbwXSlCi+zyQ9118WyAJmrQ==", 2024-03-31T15:09:47.919 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2024-03-31T15:09:47.919 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2024-03-31T15:09:47.919 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow r", 2024-03-31T15:09:47.919 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow rw", 2024-03-31T15:09:47.919 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow rwx" 2024-03-31T15:09:47.919 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-31T15:09:47.919 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-31T15:09:47.919 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-31T15:09:47.920 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "client.3", 2024-03-31T15:09:47.920 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAPewlmCw7qIRAAEeIjF85JbyfJ6d3FtMOVhQ==", 2024-03-31T15:09:47.920 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2024-03-31T15:09:47.920 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2024-03-31T15:09:47.920 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow r", 2024-03-31T15:09:47.920 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow rw", 2024-03-31T15:09:47.920 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow rwx" 2024-03-31T15:09:47.920 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-31T15:09:47.920 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-31T15:09:47.920 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-31T15:09:47.920 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "client.admin", 2024-03-31T15:09:47.920 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAOewlmXyPZGRAAtZOheiYhzGxdeJGqqyGxTw==", 2024-03-31T15:09:47.920 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2024-03-31T15:09:47.920 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow *", 2024-03-31T15:09:47.920 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow *", 2024-03-31T15:09:47.921 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow *", 2024-03-31T15:09:47.921 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2024-03-31T15:09:47.921 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-31T15:09:47.921 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-31T15:09:47.921 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-31T15:09:47.921 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "mgr.x", 2024-03-31T15:09:47.921 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAOewlm7UkKMRAA8hqgeV86mTCkRcxdXNdWmw==", 2024-03-31T15:09:47.921 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2024-03-31T15:09:47.921 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow *", 2024-03-31T15:09:47.921 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile mgr", 2024-03-31T15:09:47.921 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2024-03-31T15:09:47.921 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-31T15:09:47.921 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-31T15:09:47.921 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-31T15:09:47.922 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "mgr.y", 2024-03-31T15:09:47.922 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAOewlmq0cpLRAAKTgh04tRaHrpMhxdwgvXYA==", 2024-03-31T15:09:47.922 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2024-03-31T15:09:47.922 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow *", 2024-03-31T15:09:47.922 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile mgr", 2024-03-31T15:09:47.922 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2024-03-31T15:09:47.922 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-31T15:09:47.922 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-31T15:09:47.922 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-03-31T15:09:47.922 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-03-31T15:09:47.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:47.920+0000 7f482c000640 1 -- 172.21.15.16:0/2359845228 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f48040682f0 msgr2=0x7f480406a7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:47.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:47.920+0000 7f482c000640 1 --2- 172.21.15.16:0/2359845228 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f48040682f0 0x7f480406a7b0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f4818009a80 tx=0x7f4818009190 comp rx=0 tx=0).stop 2024-03-31T15:09:47.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:47.920+0000 7f482c000640 1 -- 172.21.15.16:0/2359845228 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4824155cc0 msgr2=0x7f4824156150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:47.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:47.920+0000 7f482c000640 1 --2- 172.21.15.16:0/2359845228 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4824155cc0 0x7f4824156150 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 crypto rx=0x7f48240cc890 tx=0x7f481c0115c0 comp rx=0 tx=0).stop 2024-03-31T15:09:47.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:47.920+0000 7f482c000640 1 -- 172.21.15.16:0/2359845228 shutdown_connections 2024-03-31T15:09:47.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:47.920+0000 7f482c000640 1 --2- 172.21.15.16:0/2359845228 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f48040682f0 0x7f480406a7b0 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:47.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:47.920+0000 7f482c000640 1 --2- 172.21.15.16:0/2359845228 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4824155cc0 0x7f4824156150 unknown :-1 s=CLOSED pgs=1059 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:47.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:47.920+0000 7f482c000640 1 --2- 172.21.15.16:0/2359845228 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4824153720 0x7f482415e0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:47.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:47.920+0000 7f482c000640 1 --2- 172.21.15.16:0/2359845228 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f480c004810 0x7f4824153fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:47.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:47.920+0000 7f482c000640 1 -- 172.21.15.16:0/2359845228 >> 172.21.15.16:0/2359845228 conn(0x7f482406b620 msgr2=0x7f48240cc8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:09:47.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:47.920+0000 7f482c000640 1 -- 172.21.15.16:0/2359845228 shutdown_connections 2024-03-31T15:09:47.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:47.920+0000 7f482c000640 1 -- 172.21.15.16:0/2359845228 wait complete. 2024-03-31T15:09:47.931 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-31T15:09:48.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:48.080+0000 7fd1e9e00640 1 Processor -- start 2024-03-31T15:09:48.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:48.080+0000 7fd1e9e00640 1 -- start start 2024-03-31T15:09:48.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:48.080+0000 7fd1e9e00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fd1e406dbc0 0x7fd1e415f650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:48.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:48.080+0000 7fd1e9e00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fd1e40cb440 0x7fd1e406f920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:48.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:48.080+0000 7fd1e9e00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fd1e40cb8c0 0x7fd1e406fe60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:48.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:48.080+0000 7fd1e9e00640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7fd1e415fb90 con 0x7fd1e40cb8c0 2024-03-31T15:09:48.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:48.080+0000 7fd1e9e00640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7fd1e415f1c0 con 0x7fd1e40cb440 2024-03-31T15:09:48.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:48.080+0000 7fd1e9e00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fd1e40cbe90 con 0x7fd1e406dbc0 2024-03-31T15:09:48.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:48.080+0000 7fd1e3400640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fd1e40cb8c0 0x7fd1e406fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:48.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:48.080+0000 7fd1e3400640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fd1e40cb8c0 0x7fd1e406fe60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59362/0 (socket says 172.21.15.16:59362) 2024-03-31T15:09:48.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:48.080+0000 7fd1e3400640 1 -- 172.21.15.16:0/1635082397 learned_addr learned my addr 172.21.15.16:0/1635082397 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:09:48.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:48.080+0000 7fd1e2a00640 1 --2- 172.21.15.16:0/1635082397 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fd1e406dbc0 0x7fd1e415f650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:48.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:48.080+0000 7fd1da000640 1 --2- 172.21.15.16:0/1635082397 >> v2:172.21.15.89:3300/0 conn(0x7fd1e40cb440 0x7fd1e406f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:48.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:48.080+0000 7fd1e3400640 1 -- 172.21.15.16:0/1635082397 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fd1e406dbc0 msgr2=0x7fd1e415f650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:09:48.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:48.080+0000 7fd1e3400640 1 --2- 172.21.15.16:0/1635082397 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fd1e406dbc0 0x7fd1e415f650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:48.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:48.080+0000 7fd1e3400640 1 -- 172.21.15.16:0/1635082397 >> v2:172.21.15.89:3300/0 conn(0x7fd1e40cb440 msgr2=0x7fd1e406f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:09:48.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:48.080+0000 7fd1e3400640 1 --2- 172.21.15.16:0/1635082397 >> v2:172.21.15.89:3300/0 conn(0x7fd1e40cb440 0x7fd1e406f920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:48.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:48.080+0000 7fd1e3400640 1 -- 172.21.15.16:0/1635082397 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1e40cc170 con 0x7fd1e40cb8c0 2024-03-31T15:09:48.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:48.084+0000 7fd1e3400640 1 --2- 172.21.15.16:0/1635082397 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd1e40cb8c0 0x7fd1e406fe60 secure :-1 s=READY pgs=1060 cs=0 l=1 rev1=1 crypto rx=0x7fd1e40d31b0 tx=0x7fd1dc01c930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a0e9f1d13a608ba9 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:09:48.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:48.084+0000 7fd1e2000640 1 -- 172.21.15.16:0/1635082397 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1dc0a5770 con 0x7fd1e40cb8c0 2024-03-31T15:09:48.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:48.084+0000 7fd1e2000640 1 -- 172.21.15.16:0/1635082397 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd1dc0a5910 con 0x7fd1e40cb8c0 2024-03-31T15:09:48.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:48.084+0000 7fd1e2000640 1 -- 172.21.15.16:0/1635082397 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1dc0a5ce0 con 0x7fd1e40cb8c0 2024-03-31T15:09:48.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:48.084+0000 7fd1e9e00640 1 -- 172.21.15.16:0/1635082397 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd1e40cb8c0 msgr2=0x7fd1e406fe60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:48.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:48.084+0000 7fd1e9e00640 1 --2- 172.21.15.16:0/1635082397 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd1e40cb8c0 0x7fd1e406fe60 secure :-1 s=READY pgs=1060 cs=0 l=1 rev1=1 crypto rx=0x7fd1e40d31b0 tx=0x7fd1dc01c930 comp rx=0 tx=0).stop 2024-03-31T15:09:48.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:48.084+0000 7fd1e9e00640 1 -- 172.21.15.16:0/1635082397 shutdown_connections 2024-03-31T15:09:48.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:48.084+0000 7fd1e9e00640 1 --2- 172.21.15.16:0/1635082397 >> v2:172.21.15.89:3300/0 conn(0x7fd1e40cb440 0x7fd1e406f920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:48.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:48.084+0000 7fd1e9e00640 1 --2- 172.21.15.16:0/1635082397 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd1e40cb8c0 0x7fd1e406fe60 unknown :-1 s=CLOSED pgs=1060 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:48.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:48.084+0000 7fd1e9e00640 1 --2- 172.21.15.16:0/1635082397 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fd1e406dbc0 0x7fd1e415f650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:48.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:48.084+0000 7fd1e9e00640 1 -- 172.21.15.16:0/1635082397 >> 172.21.15.16:0/1635082397 conn(0x7fd1e4067330 msgr2=0x7fd1e4067740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:09:48.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:48.084+0000 7fd1e9e00640 1 -- 172.21.15.16:0/1635082397 shutdown_connections 2024-03-31T15:09:48.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:48.084+0000 7fd1e9e00640 1 -- 172.21.15.16:0/1635082397 wait complete. 2024-03-31T15:09:48.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:48.088+0000 7fd1e9e00640 1 Processor -- start 2024-03-31T15:09:48.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:48.088+0000 7fd1e9e00640 1 -- start start 2024-03-31T15:09:48.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:48.088+0000 7fd1e9e00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fd1e406dbc0 0x7fd1e4166d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:48.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:48.088+0000 7fd1e9e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd1e40cb440 0x7fd1e4158070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:48.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:48.088+0000 7fd1e9e00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd1e40cb8c0 0x7fd1e415b890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:48.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:48.088+0000 7fd1e9e00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fd1e406cad0 con 0x7fd1e40cb440 2024-03-31T15:09:48.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:48.088+0000 7fd1e9e00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fd1e406c950 con 0x7fd1e40cb8c0 2024-03-31T15:09:48.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:48.088+0000 7fd1e9e00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fd1e4162990 con 0x7fd1e406dbc0 2024-03-31T15:09:48.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:48.088+0000 7fd1da000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd1e40cb440 0x7fd1e4158070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:48.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:48.088+0000 7fd1da000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd1e40cb440 0x7fd1e4158070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59364/0 (socket says 172.21.15.16:59364) 2024-03-31T15:09:48.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:48.088+0000 7fd1da000640 1 -- 172.21.15.16:0/239175952 learned_addr learned my addr 172.21.15.16:0/239175952 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:09:48.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:48.088+0000 7fd1e2a00640 1 --2- 172.21.15.16:0/239175952 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fd1e406dbc0 0x7fd1e4166d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:48.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:48.088+0000 7fd1e3400640 1 --2- 172.21.15.16:0/239175952 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd1e40cb8c0 0x7fd1e415b890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:48.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:48.088+0000 7fd1da000640 1 -- 172.21.15.16:0/239175952 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fd1e406dbc0 msgr2=0x7fd1e4166d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:48.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:48.088+0000 7fd1da000640 1 --2- 172.21.15.16:0/239175952 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fd1e406dbc0 0x7fd1e4166d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:48.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:48.088+0000 7fd1da000640 1 -- 172.21.15.16:0/239175952 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd1e40cb8c0 msgr2=0x7fd1e415b890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:48.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:48.088+0000 7fd1da000640 1 --2- 172.21.15.16:0/239175952 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd1e40cb8c0 0x7fd1e415b890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:48.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:48.088+0000 7fd1da000640 1 -- 172.21.15.16:0/239175952 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1e415c350 con 0x7fd1e40cb440 2024-03-31T15:09:48.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:48.088+0000 7fd1da000640 1 --2- 172.21.15.16:0/239175952 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd1e40cb440 0x7fd1e4158070 secure :-1 s=READY pgs=1061 cs=0 l=1 rev1=1 crypto rx=0x7fd1cc00e930 tx=0x7fd1cc00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:09:48.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:48.088+0000 7fd1dbe00640 1 -- 172.21.15.16:0/239175952 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1cc017070 con 0x7fd1e40cb440 2024-03-31T15:09:48.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:48.088+0000 7fd1dbe00640 1 -- 172.21.15.16:0/239175952 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd1cc00ca00 con 0x7fd1e40cb440 2024-03-31T15:09:48.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:48.088+0000 7fd1e9e00640 1 -- 172.21.15.16:0/239175952 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd1e41593d0 con 0x7fd1e40cb440 2024-03-31T15:09:48.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:48.088+0000 7fd1dbe00640 1 -- 172.21.15.16:0/239175952 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1cc00cdb0 con 0x7fd1e40cb440 2024-03-31T15:09:48.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:48.088+0000 7fd1e9e00640 1 -- 172.21.15.16:0/239175952 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd1e4159990 con 0x7fd1e40cb440 2024-03-31T15:09:48.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:48.088+0000 7fd1e9e00640 1 -- 172.21.15.16:0/239175952 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd1e4069950 con 0x7fd1e40cb440 2024-03-31T15:09:48.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:48.092+0000 7fd1dbe00640 1 -- 172.21.15.16:0/239175952 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fd1cc01a070 con 0x7fd1e40cb440 2024-03-31T15:09:48.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:48.092+0000 7fd1dbe00640 1 --2- 172.21.15.16:0/239175952 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fd1c0068380 0x7fd1c006a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:48.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:48.092+0000 7fd1dbe00640 1 -- 172.21.15.16:0/239175952 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 4857+0+0 (secure 0 0 0) 0x7fd1cc086a20 con 0x7fd1e40cb440 2024-03-31T15:09:48.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:48.092+0000 7fd1e2a00640 1 --2- 172.21.15.16:0/239175952 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fd1c0068380 0x7fd1c006a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:48.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:48.092+0000 7fd1e2a00640 1 --2- 172.21.15.16:0/239175952 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fd1c0068380 0x7fd1c006a840 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7fd1d4009ae0 tx=0x7fd1d4009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:09:48.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:48.096+0000 7fd1dbe00640 1 -- 172.21.15.16:0/239175952 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fd1cc056d00 con 0x7fd1e40cb440 2024-03-31T15:09:48.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:48.260+0000 7fd1e9e00640 1 -- 172.21.15.16:0/239175952 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fd1e40b83a0 con 0x7fd1e40cb440 2024-03-31T15:09:48.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:48.264+0000 7fd1dbe00640 1 -- 172.21.15.16:0/239175952 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v77) ==== 74+0+12820 (secure 0 0 0) 0x7fd1cc05b000 con 0x7fd1e40cb440 2024-03-31T15:09:48.255 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:09:48.255 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":77,"fsid":"286eb5da-fb47-4550-b290-d9519c13b9a5","created":"2024-03-31T15:03:11.438952+0000","modified":"2024-03-31T15:09:47.516438+0000","last_up_change":"2024-03-31T15:03:22.885535+0000","last_in_change":"2024-03-31T15:03:13.941667+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-31T15:03:22.550787+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"92a66450-ef19-40d8-ac3d-d760b23b0bda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6801","nonce":932712244}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6803","nonce":932712244}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6807","nonce":932712244}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6805","nonce":932712244}]},"public_addr":"172.21.15.16:6801/932712244","cluster_addr":"172.21.15.16:6803/932712244","heartbeat_back_addr":"172.21.15.16:6807/932712244","heartbeat_front_addr":"172.21.15.16:6805/932712244","state":["exists","up"]},{"osd":1,"uuid":"da9d1f2b-1968-4959-b843-faa231988e92","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6810","nonce":1398737533}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6811","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6813","nonce":1398737533}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6821","nonce":1398737533}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6815","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6816","nonce":1398737533}]},"public_addr":"172.21.15.16:6810/1398737533","cluster_addr":"172.21.15.16:6813/1398737533","heartbeat_back_addr":"172.21.15.16:6821/1398737533","heartbeat_front_addr":"172.21.15.16:6816/1398737533","state":["exists","up"]},{"osd":2,"uuid":"01a73662-9a21-4b1f-bcbb-42b9636a4638","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6819","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6822","nonce":3018526646}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6825","nonce":3018526646}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6831","nonce":3018526646}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6827","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6829","nonce":3018526646}]},"public_addr":"172.21.15.16:6822/3018526646","cluster_addr":"172.21.15.16:6825/3018526646","heartbeat_back_addr":"172.21.15.16:6831/3018526646","heartbeat_front_addr":"172.21.15.16:6829/3018526646","state":["exists","up"]},{"osd":3,"uuid":"c7771fd3-e18e-4500-ac38-23cf10ebcd75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6809","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6812","nonce":3360352974}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6817","nonce":3360352974}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6828","nonce":3360352974}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6823","nonce":3360352974}]},"public_addr":"172.21.15.16:6812/3360352974","cluster_addr":"172.21.15.16:6817/3360352974","heartbeat_back_addr":"172.21.15.16:6828/3360352974","heartbeat_front_addr":"172.21.15.16:6823/3360352974","state":["exists","up"]},{"osd":4,"uuid":"88260730-1482-4fbd-a40f-cb1124e9df4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6801","nonce":1856844856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6803","nonce":1856844856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6807","nonce":1856844856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6805","nonce":1856844856}]},"public_addr":"172.21.15.89:6801/1856844856","cluster_addr":"172.21.15.89:6803/1856844856","heartbeat_back_addr":"172.21.15.89:6807/1856844856","heartbeat_front_addr":"172.21.15.89:6805/1856844856","state":["exists","up"]},{"osd":5,"uuid":"4cb2c00c-fae0-4dd8-9d27-ec4f0093ccf7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6814","nonce":50395527}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6817","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6820","nonce":50395527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6831","nonce":50395527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6823","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6826","nonce":50395527}]},"public_addr":"172.21.15.89:6814/50395527","cluster_addr":"172.21.15.89:6820/50395527","heartbeat_back_addr":"172.21.15.89:6831/50395527","heartbeat_front_addr":"172.21.15.89:6826/50395527","state":["exists","up"]},{"osd":6,"uuid":"81ce0c19-2554-4cae-acbe-66a05e9e9b9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6809","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6812","nonce":3470471752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6815","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6818","nonce":3470471752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6827","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6829","nonce":3470471752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6821","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6824","nonce":3470471752}]},"public_addr":"172.21.15.89:6812/3470471752","cluster_addr":"172.21.15.89:6818/3470471752","heartbeat_back_addr":"172.21.15.89:6829/3470471752","heartbeat_front_addr":"172.21.15.89:6824/3470471752","state":["exists","up"]},{"osd":7,"uuid":"85e4313b-d37d-442e-8042-153bf20a847d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6811","nonce":1796778630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6813","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6816","nonce":1796778630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6825","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6828","nonce":1796778630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6819","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6822","nonce":1796778630}]},"public_addr":"172.21.15.89:6811/1796778630","cluster_addr":"172.21.15.89:6816/1796778630","heartbeat_back_addr":"172.21.15.89:6828/1796778630","heartbeat_front_addr":"172.21.15.89:6822/1796778630","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.439088+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.529970+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.680512+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:21.044161+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.582111+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.536558+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.409522+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.590229+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-31T15:09:48.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:48.264+0000 7fd1e9e00640 1 -- 172.21.15.16:0/239175952 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fd1c0068380 msgr2=0x7fd1c006a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:48.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:48.264+0000 7fd1e9e00640 1 --2- 172.21.15.16:0/239175952 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fd1c0068380 0x7fd1c006a840 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7fd1d4009ae0 tx=0x7fd1d4009190 comp rx=0 tx=0).stop 2024-03-31T15:09:48.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:48.264+0000 7fd1e9e00640 1 -- 172.21.15.16:0/239175952 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd1e40cb440 msgr2=0x7fd1e4158070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:48.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:48.264+0000 7fd1e9e00640 1 --2- 172.21.15.16:0/239175952 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd1e40cb440 0x7fd1e4158070 secure :-1 s=READY pgs=1061 cs=0 l=1 rev1=1 crypto rx=0x7fd1cc00e930 tx=0x7fd1cc00edf0 comp rx=0 tx=0).stop 2024-03-31T15:09:48.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:48.264+0000 7fd1e9e00640 1 -- 172.21.15.16:0/239175952 shutdown_connections 2024-03-31T15:09:48.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:48.264+0000 7fd1e9e00640 1 --2- 172.21.15.16:0/239175952 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fd1c0068380 0x7fd1c006a840 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:48.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:48.264+0000 7fd1e9e00640 1 --2- 172.21.15.16:0/239175952 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd1e40cb8c0 0x7fd1e415b890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:48.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:48.264+0000 7fd1e9e00640 1 --2- 172.21.15.16:0/239175952 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd1e40cb440 0x7fd1e4158070 unknown :-1 s=CLOSED pgs=1061 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:48.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:48.264+0000 7fd1e9e00640 1 --2- 172.21.15.16:0/239175952 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fd1e406dbc0 0x7fd1e4166d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:48.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:48.264+0000 7fd1e9e00640 1 -- 172.21.15.16:0/239175952 >> 172.21.15.16:0/239175952 conn(0x7fd1e4067330 msgr2=0x7fd1e40cc6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:09:48.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:48.264+0000 7fd1e9e00640 1 -- 172.21.15.16:0/239175952 shutdown_connections 2024-03-31T15:09:48.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:48.264+0000 7fd1e9e00640 1 -- 172.21.15.16:0/239175952 wait complete. 2024-03-31T15:09:48.276 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-31T15:03:22.550787+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-31T15:09:48.277 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-31T15:09:48.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:48.428+0000 7fcb82800640 1 Processor -- start 2024-03-31T15:09:48.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:48.428+0000 7fcb82800640 1 -- start start 2024-03-31T15:09:48.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:48.428+0000 7fcb82800640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fcb7c0d0a60 0x7fcb7c0cea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:48.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:48.428+0000 7fcb82800640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fcb7c0cef80 0x7fcb7c1419c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:48.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:48.428+0000 7fcb82800640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcb7c15ca70 con 0x7fcb7c06f920 2024-03-31T15:09:48.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:48.428+0000 7fcb82800640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7fcb7c13f670 con 0x7fcb7c0cef80 2024-03-31T15:09:48.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:48.428+0000 7fcb82800640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7fcb7c15cd50 con 0x7fcb7c0d0a60 2024-03-31T15:09:48.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:48.428+0000 7fcb7be00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fcb7c0cef80 0x7fcb7c1419c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:48.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:48.428+0000 7fcb7be00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fcb7c0cef80 0x7fcb7c1419c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59378/0 (socket says 172.21.15.16:59378) 2024-03-31T15:09:48.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:48.428+0000 7fcb7be00640 1 -- 172.21.15.16:0/2569415938 learned_addr learned my addr 172.21.15.16:0/2569415938 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:09:48.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:48.428+0000 7fcb7b400640 1 --1- 172.21.15.16:0/2569415938 >> v1:172.21.15.89:6789/0 conn(0x7fcb7c06f920 0x7fcb7c06fd00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.16:58894/0 (socket says 172.21.15.16:58894) 2024-03-31T15:09:48.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:48.428+0000 7fcb72a00640 1 --2- 172.21.15.16:0/2569415938 >> v2:172.21.15.89:3300/0 conn(0x7fcb7c0d0a60 0x7fcb7c0cea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:48.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:48.428+0000 7fcb7be00640 1 -- 172.21.15.16:0/2569415938 >> v2:172.21.15.89:3300/0 conn(0x7fcb7c0d0a60 msgr2=0x7fcb7c0cea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:09:48.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:48.428+0000 7fcb7be00640 1 --2- 172.21.15.16:0/2569415938 >> v2:172.21.15.89:3300/0 conn(0x7fcb7c0d0a60 0x7fcb7c0cea40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:48.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:48.428+0000 7fcb7be00640 1 -- 172.21.15.16:0/2569415938 >> v1:172.21.15.89:6789/0 conn(0x7fcb7c06f920 legacy=0x7fcb7c06fd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:48.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:48.428+0000 7fcb7be00640 1 -- 172.21.15.16:0/2569415938 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb7c15d030 con 0x7fcb7c0cef80 2024-03-31T15:09:48.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:48.432+0000 7fcb7be00640 1 --2- 172.21.15.16:0/2569415938 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcb7c0cef80 0x7fcb7c1419c0 secure :-1 s=READY pgs=1062 cs=0 l=1 rev1=1 crypto rx=0x7fcb7c0b5070 tx=0x7fcb7401c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cce1ca2e43bd5df1 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:09:48.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:48.432+0000 7fcb7aa00640 1 -- 172.21.15.16:0/2569415938 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb740a56c0 con 0x7fcb7c0cef80 2024-03-31T15:09:48.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:48.432+0000 7fcb7aa00640 1 -- 172.21.15.16:0/2569415938 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcb740a5860 con 0x7fcb7c0cef80 2024-03-31T15:09:48.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:48.432+0000 7fcb7aa00640 1 -- 172.21.15.16:0/2569415938 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb740a5c30 con 0x7fcb7c0cef80 2024-03-31T15:09:48.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:48.432+0000 7fcb82800640 1 -- 172.21.15.16:0/2569415938 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcb7c0cef80 msgr2=0x7fcb7c1419c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:48.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:48.432+0000 7fcb82800640 1 --2- 172.21.15.16:0/2569415938 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcb7c0cef80 0x7fcb7c1419c0 secure :-1 s=READY pgs=1062 cs=0 l=1 rev1=1 crypto rx=0x7fcb7c0b5070 tx=0x7fcb7401c9f0 comp rx=0 tx=0).stop 2024-03-31T15:09:48.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:48.432+0000 7fcb82800640 1 -- 172.21.15.16:0/2569415938 shutdown_connections 2024-03-31T15:09:48.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:48.432+0000 7fcb82800640 1 --2- 172.21.15.16:0/2569415938 >> v2:172.21.15.89:3300/0 conn(0x7fcb7c0d0a60 0x7fcb7c0cea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:48.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:48.432+0000 7fcb82800640 1 --2- 172.21.15.16:0/2569415938 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcb7c0cef80 0x7fcb7c1419c0 unknown :-1 s=CLOSED pgs=1062 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:48.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:48.432+0000 7fcb82800640 1 -- 172.21.15.16:0/2569415938 >> 172.21.15.16:0/2569415938 conn(0x7fcb7c067930 msgr2=0x7fcb7c0b3ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:09:48.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:48.432+0000 7fcb82800640 1 -- 172.21.15.16:0/2569415938 shutdown_connections 2024-03-31T15:09:48.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:48.432+0000 7fcb82800640 1 -- 172.21.15.16:0/2569415938 wait complete. 2024-03-31T15:09:48.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:48.436+0000 7fcb82800640 1 Processor -- start 2024-03-31T15:09:48.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:48.436+0000 7fcb82800640 1 -- start start 2024-03-31T15:09:48.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:48.436+0000 7fcb82800640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcb7c06f920 0x7fcb7c0b3c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:48.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:48.436+0000 7fcb82800640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcb7c0cef80 0x7fcb7c13e7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:48.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:48.436+0000 7fcb82800640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fcb7c0d0a60 0x7fcb7c0d0ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:48.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:48.436+0000 7fcb82800640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fcb7c169230 con 0x7fcb7c06f920 2024-03-31T15:09:48.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:48.436+0000 7fcb82800640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fcb7c1690b0 con 0x7fcb7c0cef80 2024-03-31T15:09:48.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:48.436+0000 7fcb82800640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fcb7c1693b0 con 0x7fcb7c0d0a60 2024-03-31T15:09:48.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:48.436+0000 7fcb7b400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcb7c06f920 0x7fcb7c0b3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:48.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:48.436+0000 7fcb7b400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcb7c06f920 0x7fcb7c0b3c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59388/0 (socket says 172.21.15.16:59388) 2024-03-31T15:09:48.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:48.436+0000 7fcb7b400640 1 -- 172.21.15.16:0/899210583 learned_addr learned my addr 172.21.15.16:0/899210583 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:09:48.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:48.436+0000 7fcb72a00640 1 --2- 172.21.15.16:0/899210583 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcb7c0cef80 0x7fcb7c13e7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:48.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:48.436+0000 7fcb7be00640 1 --2- 172.21.15.16:0/899210583 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fcb7c0d0a60 0x7fcb7c0d0ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:48.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:48.436+0000 7fcb7b400640 1 -- 172.21.15.16:0/899210583 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fcb7c0d0a60 msgr2=0x7fcb7c0d0ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:48.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:48.436+0000 7fcb7b400640 1 --2- 172.21.15.16:0/899210583 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fcb7c0d0a60 0x7fcb7c0d0ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:48.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:48.436+0000 7fcb7b400640 1 -- 172.21.15.16:0/899210583 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcb7c0cef80 msgr2=0x7fcb7c13e7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:48.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:48.436+0000 7fcb7b400640 1 --2- 172.21.15.16:0/899210583 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcb7c0cef80 0x7fcb7c13e7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:48.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:48.436+0000 7fcb7b400640 1 -- 172.21.15.16:0/899210583 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb7c1fb110 con 0x7fcb7c06f920 2024-03-31T15:09:48.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:48.436+0000 7fcb7b400640 1 --2- 172.21.15.16:0/899210583 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcb7c06f920 0x7fcb7c0b3c30 secure :-1 s=READY pgs=1063 cs=0 l=1 rev1=1 crypto rx=0x7fcb6c00c420 tx=0x7fcb6c00c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:09:48.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:48.436+0000 7fcb73e00640 1 -- 172.21.15.16:0/899210583 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb6c0074e0 con 0x7fcb7c06f920 2024-03-31T15:09:48.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:48.436+0000 7fcb73e00640 1 -- 172.21.15.16:0/899210583 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcb6c007680 con 0x7fcb7c06f920 2024-03-31T15:09:48.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:48.436+0000 7fcb82800640 1 -- 172.21.15.16:0/899210583 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcb7c1fb3f0 con 0x7fcb7c06f920 2024-03-31T15:09:48.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:48.436+0000 7fcb73e00640 1 -- 172.21.15.16:0/899210583 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb6c004a30 con 0x7fcb7c06f920 2024-03-31T15:09:48.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:48.436+0000 7fcb82800640 1 -- 172.21.15.16:0/899210583 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcb7c1fb9b0 con 0x7fcb7c06f920 2024-03-31T15:09:48.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:48.440+0000 7fcb82800640 1 -- 172.21.15.16:0/899210583 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcb7c069950 con 0x7fcb7c06f920 2024-03-31T15:09:48.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:48.440+0000 7fcb73e00640 1 -- 172.21.15.16:0/899210583 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fcb6c01a020 con 0x7fcb7c06f920 2024-03-31T15:09:48.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:48.440+0000 7fcb73e00640 1 --2- 172.21.15.16:0/899210583 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fcb58068380 0x7fcb5806a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:48.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:48.440+0000 7fcb73e00640 1 -- 172.21.15.16:0/899210583 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 4857+0+0 (secure 0 0 0) 0x7fcb6c085d80 con 0x7fcb7c06f920 2024-03-31T15:09:48.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:48.440+0000 7fcb72a00640 1 --2- 172.21.15.16:0/899210583 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fcb58068380 0x7fcb5806a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:48.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:48.440+0000 7fcb72a00640 1 --2- 172.21.15.16:0/899210583 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fcb58068380 0x7fcb5806a840 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7fcb7c13f200 tx=0x7fcb64009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:09:48.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:48.444+0000 7fcb73e00640 1 -- 172.21.15.16:0/899210583 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fcb6c056030 con 0x7fcb7c06f920 2024-03-31T15:09:48.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:48.612+0000 7fcb82800640 1 -- 172.21.15.16:0/899210583 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fcb7c0b50e0 con 0x7fcb7c06f920 2024-03-31T15:09:48.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:48.616+0000 7fcb73e00640 1 -- 172.21.15.16:0/899210583 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v77) ==== 93+0+10 (secure 0 0 0) 0x7fcb6c05a330 con 0x7fcb7c06f920 2024-03-31T15:09:48.607 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2024-03-31T15:09:48.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:48.616+0000 7fcb82800640 1 -- 172.21.15.16:0/899210583 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fcb58068380 msgr2=0x7fcb5806a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:48.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:48.616+0000 7fcb82800640 1 --2- 172.21.15.16:0/899210583 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fcb58068380 0x7fcb5806a840 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7fcb7c13f200 tx=0x7fcb64009190 comp rx=0 tx=0).stop 2024-03-31T15:09:48.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:48.616+0000 7fcb82800640 1 -- 172.21.15.16:0/899210583 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcb7c06f920 msgr2=0x7fcb7c0b3c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:48.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:48.616+0000 7fcb82800640 1 --2- 172.21.15.16:0/899210583 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcb7c06f920 0x7fcb7c0b3c30 secure :-1 s=READY pgs=1063 cs=0 l=1 rev1=1 crypto rx=0x7fcb6c00c420 tx=0x7fcb6c00c8e0 comp rx=0 tx=0).stop 2024-03-31T15:09:48.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:48.616+0000 7fcb82800640 1 -- 172.21.15.16:0/899210583 shutdown_connections 2024-03-31T15:09:48.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:48.616+0000 7fcb82800640 1 --2- 172.21.15.16:0/899210583 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fcb58068380 0x7fcb5806a840 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:48.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:48.616+0000 7fcb82800640 1 --2- 172.21.15.16:0/899210583 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fcb7c0d0a60 0x7fcb7c0d0ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:48.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:48.616+0000 7fcb82800640 1 --2- 172.21.15.16:0/899210583 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcb7c0cef80 0x7fcb7c13e7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:48.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:48.616+0000 7fcb82800640 1 --2- 172.21.15.16:0/899210583 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcb7c06f920 0x7fcb7c0b3c30 unknown :-1 s=CLOSED pgs=1063 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:48.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:48.616+0000 7fcb82800640 1 -- 172.21.15.16:0/899210583 >> 172.21.15.16:0/899210583 conn(0x7fcb7c067930 msgr2=0x7fcb7c15a2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:09:48.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:48.616+0000 7fcb82800640 1 -- 172.21.15.16:0/899210583 shutdown_connections 2024-03-31T15:09:48.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:48.616+0000 7fcb82800640 1 -- 172.21.15.16:0/899210583 wait complete. 2024-03-31T15:09:48.628 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-31T15:09:48.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:48.776+0000 7f7e93200640 1 Processor -- start 2024-03-31T15:09:48.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:48.776+0000 7f7e93200640 1 -- start start 2024-03-31T15:09:48.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:48.780+0000 7f7e93200640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f7e940cfbb0 0x7f7e940cdcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:48.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:48.780+0000 7f7e93200640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7e9415d790 con 0x7f7e940cd820 2024-03-31T15:09:48.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:48.780+0000 7f7e93200640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7e94160a80 con 0x7f7e9415d3b0 2024-03-31T15:09:48.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:48.780+0000 7f7e93200640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f7e9415a260 con 0x7f7e940cfbb0 2024-03-31T15:09:48.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:48.780+0000 7f7e89400640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f7e940cd820 0x7f7e940cf4a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:48188/0 (socket says 172.21.15.16:48188) 2024-03-31T15:09:48.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:48.780+0000 7f7e89400640 1 -- 172.21.15.16:0/1773339705 learned_addr learned my addr 172.21.15.16:0/1773339705 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:09:48.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:48.780+0000 7f7e92800640 1 --2- 172.21.15.16:0/1773339705 >> v2:172.21.15.16:3300/0 conn(0x7f7e940cfbb0 0x7f7e940cdcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:48.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:48.780+0000 7f7e91400640 1 -- 172.21.15.16:0/1773339705 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1680270680 0 0) 0x7f7e7c0014a0 con 0x7f7e940cd820 2024-03-31T15:09:48.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:48.780+0000 7f7e91400640 1 -- 172.21.15.16:0/1773339705 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7e78003530 con 0x7f7e940cd820 2024-03-31T15:09:48.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:48.780+0000 7f7e92800640 1 -- 172.21.15.16:0/1773339705 >> v1:172.21.15.89:6789/0 conn(0x7f7e9415d3b0 legacy=0x7f7e940b56e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:09:48.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:48.780+0000 7f7e92800640 1 -- 172.21.15.16:0/1773339705 >> v1:172.21.15.16:6789/0 conn(0x7f7e940cd820 legacy=0x7f7e940cf4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:48.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:48.780+0000 7f7e92800640 1 -- 172.21.15.16:0/1773339705 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e940b4c00 con 0x7f7e940cfbb0 2024-03-31T15:09:48.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:48.784+0000 7f7e92800640 1 --2- 172.21.15.16:0/1773339705 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7e940cfbb0 0x7f7e940cdcb0 secure :-1 s=READY pgs=1065 cs=0 l=1 rev1=1 crypto rx=0x7f7e8c0097c0 tx=0x7f7e8c01c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ecb2055c04376585 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:09:48.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:48.784+0000 7f7e91400640 1 -- 172.21.15.16:0/1773339705 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e8c0b0070 con 0x7f7e940cfbb0 2024-03-31T15:09:48.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:48.784+0000 7f7e91400640 1 -- 172.21.15.16:0/1773339705 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7e8c0ab600 con 0x7f7e940cfbb0 2024-03-31T15:09:48.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:48.784+0000 7f7e91400640 1 -- 172.21.15.16:0/1773339705 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e8c0ab9b0 con 0x7f7e940cfbb0 2024-03-31T15:09:48.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:48.784+0000 7f7e93200640 1 -- 172.21.15.16:0/1773339705 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7e940cfbb0 msgr2=0x7f7e940cdcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:48.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:48.784+0000 7f7e93200640 1 --2- 172.21.15.16:0/1773339705 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7e940cfbb0 0x7f7e940cdcb0 secure :-1 s=READY pgs=1065 cs=0 l=1 rev1=1 crypto rx=0x7f7e8c0097c0 tx=0x7f7e8c01c4a0 comp rx=0 tx=0).stop 2024-03-31T15:09:48.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:48.784+0000 7f7e93200640 1 -- 172.21.15.16:0/1773339705 shutdown_connections 2024-03-31T15:09:48.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:48.784+0000 7f7e93200640 1 --2- 172.21.15.16:0/1773339705 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7e940cfbb0 0x7f7e940cdcb0 unknown :-1 s=CLOSED pgs=1065 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:48.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:48.784+0000 7f7e93200640 1 -- 172.21.15.16:0/1773339705 >> 172.21.15.16:0/1773339705 conn(0x7f7e940b2ac0 msgr2=0x7f7e940b2ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:09:48.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:48.784+0000 7f7e93200640 1 -- 172.21.15.16:0/1773339705 shutdown_connections 2024-03-31T15:09:48.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:48.784+0000 7f7e93200640 1 -- 172.21.15.16:0/1773339705 wait complete. 2024-03-31T15:09:48.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:48.784+0000 7f7e93200640 1 Processor -- start 2024-03-31T15:09:48.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:48.784+0000 7f7e93200640 1 -- start start 2024-03-31T15:09:48.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:48.788+0000 7f7e93200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7e940cd820 0x7f7e9414ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:48.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:48.788+0000 7f7e93200640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7e940cfbb0 0x7f7e9406baf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:48.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:48.788+0000 7f7e93200640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f7e9415d3b0 0x7f7e9415eaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:48.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:48.788+0000 7f7e93200640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f7e94160d60 con 0x7f7e940cd820 2024-03-31T15:09:48.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:48.788+0000 7f7e93200640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f7e940b5df0 con 0x7f7e940cfbb0 2024-03-31T15:09:48.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:48.788+0000 7f7e93200640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f7e9416a380 con 0x7f7e9415d3b0 2024-03-31T15:09:48.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:48.788+0000 7f7e91e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7e940cd820 0x7f7e9414ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:48.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:48.788+0000 7f7e91e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7e940cd820 0x7f7e9414ec80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59408/0 (socket says 172.21.15.16:59408) 2024-03-31T15:09:48.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:48.788+0000 7f7e91e00640 1 -- 172.21.15.16:0/3439660980 learned_addr learned my addr 172.21.15.16:0/3439660980 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:09:48.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:48.788+0000 7f7e89400640 1 --2- 172.21.15.16:0/3439660980 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7e940cfbb0 0x7f7e9406baf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:48.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:48.788+0000 7f7e92800640 1 --2- 172.21.15.16:0/3439660980 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f7e9415d3b0 0x7f7e9415eaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:48.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:48.788+0000 7f7e91e00640 1 -- 172.21.15.16:0/3439660980 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f7e9415d3b0 msgr2=0x7f7e9415eaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:48.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:48.788+0000 7f7e91e00640 1 --2- 172.21.15.16:0/3439660980 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f7e9415d3b0 0x7f7e9415eaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:48.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:48.788+0000 7f7e91e00640 1 -- 172.21.15.16:0/3439660980 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7e940cfbb0 msgr2=0x7f7e9406baf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:48.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:48.788+0000 7f7e91e00640 1 --2- 172.21.15.16:0/3439660980 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7e940cfbb0 0x7f7e9406baf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:48.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:48.788+0000 7f7e91e00640 1 -- 172.21.15.16:0/3439660980 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e9415a520 con 0x7f7e940cd820 2024-03-31T15:09:48.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:48.788+0000 7f7e89400640 1 --2- 172.21.15.16:0/3439660980 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7e940cfbb0 0x7f7e9406baf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:09:48.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:48.788+0000 7f7e92800640 1 --2- 172.21.15.16:0/3439660980 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f7e9415d3b0 0x7f7e9415eaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:09:48.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:48.788+0000 7f7e91e00640 1 --2- 172.21.15.16:0/3439660980 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7e940cd820 0x7f7e9414ec80 secure :-1 s=READY pgs=1066 cs=0 l=1 rev1=1 crypto rx=0x7f7e8400c3c0 tx=0x7f7e8400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:09:48.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:48.788+0000 7f7e8aa00640 1 -- 172.21.15.16:0/3439660980 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e84017070 con 0x7f7e940cd820 2024-03-31T15:09:48.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:48.788+0000 7f7e8aa00640 1 -- 172.21.15.16:0/3439660980 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7e840052d0 con 0x7f7e940cd820 2024-03-31T15:09:48.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:48.788+0000 7f7e8aa00640 1 -- 172.21.15.16:0/3439660980 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e84003840 con 0x7f7e940cd820 2024-03-31T15:09:48.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:48.788+0000 7f7e93200640 1 -- 172.21.15.16:0/3439660980 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7e9415a7c0 con 0x7f7e940cd820 2024-03-31T15:09:48.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:48.788+0000 7f7e93200640 1 -- 172.21.15.16:0/3439660980 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7e9415ad80 con 0x7f7e940cd820 2024-03-31T15:09:48.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:48.788+0000 7f7e93200640 1 -- 172.21.15.16:0/3439660980 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7e9404f8c0 con 0x7f7e940cd820 2024-03-31T15:09:48.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:48.788+0000 7f7e8aa00640 1 -- 172.21.15.16:0/3439660980 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f7e8401a070 con 0x7f7e940cd820 2024-03-31T15:09:48.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:48.792+0000 7f7e8aa00640 1 --2- 172.21.15.16:0/3439660980 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f7e70068380 0x7f7e7006a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:48.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:48.792+0000 7f7e8aa00640 1 -- 172.21.15.16:0/3439660980 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 4857+0+0 (secure 0 0 0) 0x7f7e84085960 con 0x7f7e940cd820 2024-03-31T15:09:48.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:48.792+0000 7f7e89400640 1 --2- 172.21.15.16:0/3439660980 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f7e70068380 0x7f7e7006a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:48.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:48.792+0000 7f7e89400640 1 --2- 172.21.15.16:0/3439660980 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f7e70068380 0x7f7e7006a840 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f7e94168130 tx=0x7f7e7c009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:09:48.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:48.796+0000 7f7e8aa00640 1 -- 172.21.15.16:0/3439660980 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f7e84005480 con 0x7f7e940cd820 2024-03-31T15:09:48.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:48.964+0000 7f7e93200640 1 -- 172.21.15.16:0/3439660980 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7f7e940cf280 con 0x7f7e940cd820 2024-03-31T15:09:48.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:48.964+0000 7f7e8aa00640 1 -- 172.21.15.16:0/3439660980 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v77) ==== 84+0+65 (secure 0 0 0) 0x7f7e84056260 con 0x7f7e940cd820 2024-03-31T15:09:48.955 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:09:48.956 INFO:teuthology.orchestra.run.smithi016.stdout:[ 2024-03-31T15:09:48.956 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-31T15:09:48.956 INFO:teuthology.orchestra.run.smithi016.stdout: "poolnum": 1, 2024-03-31T15:09:48.956 INFO:teuthology.orchestra.run.smithi016.stdout: "poolname": ".mgr" 2024-03-31T15:09:48.956 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-31T15:09:48.956 INFO:teuthology.orchestra.run.smithi016.stdout:] 2024-03-31T15:09:48.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:48.964+0000 7f7e93200640 1 -- 172.21.15.16:0/3439660980 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f7e70068380 msgr2=0x7f7e7006a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:48.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:48.964+0000 7f7e93200640 1 --2- 172.21.15.16:0/3439660980 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f7e70068380 0x7f7e7006a840 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f7e94168130 tx=0x7f7e7c009040 comp rx=0 tx=0).stop 2024-03-31T15:09:48.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:48.964+0000 7f7e93200640 1 -- 172.21.15.16:0/3439660980 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7e940cd820 msgr2=0x7f7e9414ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:48.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:48.964+0000 7f7e93200640 1 --2- 172.21.15.16:0/3439660980 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7e940cd820 0x7f7e9414ec80 secure :-1 s=READY pgs=1066 cs=0 l=1 rev1=1 crypto rx=0x7f7e8400c3c0 tx=0x7f7e8400c880 comp rx=0 tx=0).stop 2024-03-31T15:09:48.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:48.964+0000 7f7e93200640 1 -- 172.21.15.16:0/3439660980 shutdown_connections 2024-03-31T15:09:48.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:48.964+0000 7f7e93200640 1 --2- 172.21.15.16:0/3439660980 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f7e70068380 0x7f7e7006a840 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:48.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:48.964+0000 7f7e93200640 1 --2- 172.21.15.16:0/3439660980 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f7e9415d3b0 0x7f7e9415eaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:48.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:48.964+0000 7f7e93200640 1 --2- 172.21.15.16:0/3439660980 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7e940cfbb0 0x7f7e9406baf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:48.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:48.964+0000 7f7e93200640 1 --2- 172.21.15.16:0/3439660980 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7e940cd820 0x7f7e9414ec80 unknown :-1 s=CLOSED pgs=1066 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:48.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:48.964+0000 7f7e93200640 1 -- 172.21.15.16:0/3439660980 >> 172.21.15.16:0/3439660980 conn(0x7f7e940b2ac0 msgr2=0x7f7e940b2ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:09:48.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:48.964+0000 7f7e93200640 1 -- 172.21.15.16:0/3439660980 shutdown_connections 2024-03-31T15:09:48.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:48.964+0000 7f7e93200640 1 -- 172.21.15.16:0/3439660980 wait complete. 2024-03-31T15:09:48.977 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-31T15:09:48.977 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-31T15:09:49.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:49.128+0000 7ffaa5200640 1 Processor -- start 2024-03-31T15:09:49.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:49.128+0000 7ffaa5200640 1 -- start start 2024-03-31T15:09:49.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:49.132+0000 7ffaa5200640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ffaa0143f40 0x7ffaa00730c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:49.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:49.132+0000 7ffaa5200640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ffaa00b45d0 0x7ffaa00b1be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:49.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:49.132+0000 7ffaa5200640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffaa0144320 con 0x7ffaa00b2120 2024-03-31T15:09:49.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:49.132+0000 7ffaa5200640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7ffaa00b4a30 con 0x7ffaa0143f40 2024-03-31T15:09:49.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:49.132+0000 7ffaa5200640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7ffaa015ef80 con 0x7ffaa00b45d0 2024-03-31T15:09:49.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:49.132+0000 7ffa9de00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ffaa0143f40 0x7ffaa00730c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:49.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:49.132+0000 7ffa9e800640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7ffaa00b2120 0x7ffaa0071490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:48196/0 (socket says 172.21.15.16:48196) 2024-03-31T15:09:49.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:49.132+0000 7ffa9e800640 1 -- 172.21.15.16:0/1191968657 learned_addr learned my addr 172.21.15.16:0/1191968657 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:09:49.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:49.132+0000 7ffa97e00640 1 --2- 172.21.15.16:0/1191968657 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ffaa00b45d0 0x7ffaa00b1be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:49.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:49.132+0000 7ffa9de00640 1 -- 172.21.15.16:0/1191968657 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ffaa00b45d0 msgr2=0x7ffaa00b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:09:49.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:49.132+0000 7ffa9de00640 1 --2- 172.21.15.16:0/1191968657 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ffaa00b45d0 0x7ffaa00b1be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:49.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:49.132+0000 7ffa9de00640 1 -- 172.21.15.16:0/1191968657 >> v1:172.21.15.16:6789/0 conn(0x7ffaa00b2120 legacy=0x7ffaa0071490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:49.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:49.132+0000 7ffa9de00640 1 -- 172.21.15.16:0/1191968657 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffaa015f260 con 0x7ffaa0143f40 2024-03-31T15:09:49.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:49.132+0000 7ffa9d400640 1 -- 172.21.15.16:0/1191968657 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1888212589 0 0) 0x7ffaa0144320 con 0x7ffaa00b2120 2024-03-31T15:09:49.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:49.132+0000 7ffa9de00640 1 --2- 172.21.15.16:0/1191968657 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffaa0143f40 0x7ffaa00730c0 secure :-1 s=READY pgs=1068 cs=0 l=1 rev1=1 crypto rx=0x7ffa900097c0 tx=0x7ffa9001c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d650674e981e7483 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:09:49.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:49.132+0000 7ffa9d400640 1 -- 172.21.15.16:0/1191968657 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa900b0070 con 0x7ffaa0143f40 2024-03-31T15:09:49.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:49.132+0000 7ffa9d400640 1 -- 172.21.15.16:0/1191968657 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffa900a4e50 con 0x7ffaa0143f40 2024-03-31T15:09:49.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:49.132+0000 7ffa9d400640 1 -- 172.21.15.16:0/1191968657 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa900ab820 con 0x7ffaa0143f40 2024-03-31T15:09:49.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:49.136+0000 7ffaa5200640 1 -- 172.21.15.16:0/1191968657 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffaa0143f40 msgr2=0x7ffaa00730c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:49.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:49.136+0000 7ffaa5200640 1 --2- 172.21.15.16:0/1191968657 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffaa0143f40 0x7ffaa00730c0 secure :-1 s=READY pgs=1068 cs=0 l=1 rev1=1 crypto rx=0x7ffa900097c0 tx=0x7ffa9001c4a0 comp rx=0 tx=0).stop 2024-03-31T15:09:49.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:49.136+0000 7ffaa5200640 1 -- 172.21.15.16:0/1191968657 shutdown_connections 2024-03-31T15:09:49.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:49.136+0000 7ffaa5200640 1 --2- 172.21.15.16:0/1191968657 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ffaa00b45d0 0x7ffaa00b1be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:49.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:49.136+0000 7ffaa5200640 1 --2- 172.21.15.16:0/1191968657 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffaa0143f40 0x7ffaa00730c0 unknown :-1 s=CLOSED pgs=1068 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:49.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:49.136+0000 7ffaa5200640 1 -- 172.21.15.16:0/1191968657 >> 172.21.15.16:0/1191968657 conn(0x7ffaa006b620 msgr2=0x7ffaa006ba30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:09:49.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:49.136+0000 7ffaa5200640 1 -- 172.21.15.16:0/1191968657 shutdown_connections 2024-03-31T15:09:49.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:49.136+0000 7ffaa5200640 1 -- 172.21.15.16:0/1191968657 wait complete. 2024-03-31T15:09:49.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:49.136+0000 7ffaa5200640 1 Processor -- start 2024-03-31T15:09:49.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:49.136+0000 7ffaa5200640 1 -- start start 2024-03-31T15:09:49.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:49.136+0000 7ffaa5200640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ffaa00b2120 0x7ffaa013cb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:49.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:49.136+0000 7ffaa5200640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffaa00b45d0 0x7ffaa0151980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:49.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:49.136+0000 7ffaa5200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffaa0143f40 0x7ffaa00d02b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:49.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:49.136+0000 7ffaa5200640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7ffaa00b2570 con 0x7ffaa0143f40 2024-03-31T15:09:49.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:49.136+0000 7ffaa5200640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7ffaa00cd9a0 con 0x7ffaa00b45d0 2024-03-31T15:09:49.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:49.136+0000 7ffaa5200640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7ffaa01680b0 con 0x7ffaa00b2120 2024-03-31T15:09:49.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:49.136+0000 7ffa9e800640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffaa0143f40 0x7ffaa00d02b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:49.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:49.136+0000 7ffa9e800640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffaa0143f40 0x7ffaa00d02b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59434/0 (socket says 172.21.15.16:59434) 2024-03-31T15:09:49.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:49.136+0000 7ffa9e800640 1 -- 172.21.15.16:0/3356115002 learned_addr learned my addr 172.21.15.16:0/3356115002 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:09:49.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:49.136+0000 7ffa9de00640 1 --2- 172.21.15.16:0/3356115002 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ffaa00b2120 0x7ffaa013cb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:49.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:49.136+0000 7ffa97e00640 1 --2- 172.21.15.16:0/3356115002 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffaa00b45d0 0x7ffaa0151980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:49.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:49.136+0000 7ffa9e800640 1 -- 172.21.15.16:0/3356115002 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ffaa00b2120 msgr2=0x7ffaa013cb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:49.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:49.136+0000 7ffa9e800640 1 --2- 172.21.15.16:0/3356115002 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ffaa00b2120 0x7ffaa013cb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:49.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:49.136+0000 7ffa9e800640 1 -- 172.21.15.16:0/3356115002 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffaa00b45d0 msgr2=0x7ffaa0151980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:49.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:49.136+0000 7ffa9e800640 1 --2- 172.21.15.16:0/3356115002 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffaa00b45d0 0x7ffaa0151980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:49.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:49.140+0000 7ffa9e800640 1 -- 172.21.15.16:0/3356115002 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffaa00ce7b0 con 0x7ffaa0143f40 2024-03-31T15:09:49.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:49.140+0000 7ffa97e00640 1 --2- 172.21.15.16:0/3356115002 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffaa00b45d0 0x7ffaa0151980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:09:49.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:49.140+0000 7ffa9e800640 1 --2- 172.21.15.16:0/3356115002 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffaa0143f40 0x7ffaa00d02b0 secure :-1 s=READY pgs=1069 cs=0 l=1 rev1=1 crypto rx=0x7ffa9800c700 tx=0x7ffa9800cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:09:49.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:49.140+0000 7ffa96000640 1 -- 172.21.15.16:0/3356115002 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa98018070 con 0x7ffaa0143f40 2024-03-31T15:09:49.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:49.140+0000 7ffa96000640 1 -- 172.21.15.16:0/3356115002 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffa9800ce10 con 0x7ffaa0143f40 2024-03-31T15:09:49.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:49.140+0000 7ffa96000640 1 -- 172.21.15.16:0/3356115002 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa9800a880 con 0x7ffaa0143f40 2024-03-31T15:09:49.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:49.140+0000 7ffaa5200640 1 -- 172.21.15.16:0/3356115002 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffaa00cea90 con 0x7ffaa0143f40 2024-03-31T15:09:49.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:49.140+0000 7ffaa5200640 1 -- 172.21.15.16:0/3356115002 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffaa00cf050 con 0x7ffaa0143f40 2024-03-31T15:09:49.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:49.140+0000 7ffaa5200640 1 -- 172.21.15.16:0/3356115002 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffaa0069950 con 0x7ffaa0143f40 2024-03-31T15:09:49.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:49.140+0000 7ffa96000640 1 -- 172.21.15.16:0/3356115002 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7ffa9801b070 con 0x7ffaa0143f40 2024-03-31T15:09:49.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:49.140+0000 7ffa96000640 1 --2- 172.21.15.16:0/3356115002 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7ffa7c068380 0x7ffa7c06a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:49.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:49.140+0000 7ffa96000640 1 -- 172.21.15.16:0/3356115002 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 4857+0+0 (secure 0 0 0) 0x7ffa98085bd0 con 0x7ffaa0143f40 2024-03-31T15:09:49.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:49.144+0000 7ffa9de00640 1 --2- 172.21.15.16:0/3356115002 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7ffa7c068380 0x7ffa7c06a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:49.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:49.144+0000 7ffa9de00640 1 --2- 172.21.15.16:0/3356115002 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7ffa7c068380 0x7ffa7c06a840 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7ffa900ad000 tx=0x7ffa900a6000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:09:49.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:49.144+0000 7ffa96000640 1 -- 172.21.15.16:0/3356115002 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7ffa98055e00 con 0x7ffaa0143f40 2024-03-31T15:09:49.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:49.308+0000 7ffaa5200640 1 -- 172.21.15.16:0/3356115002 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7ffaa00ce4f0 con 0x7ffaa0143f40 2024-03-31T15:09:49.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:49.508+0000 7ffa96000640 1 -- 172.21.15.16:0/3356115002 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v78) ==== 138+0+0 (secure 0 0 0) 0x7ffa9805a100 con 0x7ffaa0143f40 2024-03-31T15:09:49.501 INFO:teuthology.orchestra.run.smithi016.stderr:pool 'cephfs_metadata' created 2024-03-31T15:09:49.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:49.512+0000 7ffaa5200640 1 -- 172.21.15.16:0/3356115002 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7ffa7c068380 msgr2=0x7ffa7c06a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:49.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:49.512+0000 7ffaa5200640 1 --2- 172.21.15.16:0/3356115002 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7ffa7c068380 0x7ffa7c06a840 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7ffa900ad000 tx=0x7ffa900a6000 comp rx=0 tx=0).stop 2024-03-31T15:09:49.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:49.512+0000 7ffaa5200640 1 -- 172.21.15.16:0/3356115002 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffaa0143f40 msgr2=0x7ffaa00d02b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:49.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:49.512+0000 7ffaa5200640 1 --2- 172.21.15.16:0/3356115002 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffaa0143f40 0x7ffaa00d02b0 secure :-1 s=READY pgs=1069 cs=0 l=1 rev1=1 crypto rx=0x7ffa9800c700 tx=0x7ffa9800cbc0 comp rx=0 tx=0).stop 2024-03-31T15:09:49.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:49.512+0000 7ffaa5200640 1 -- 172.21.15.16:0/3356115002 shutdown_connections 2024-03-31T15:09:49.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:49.512+0000 7ffaa5200640 1 --2- 172.21.15.16:0/3356115002 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7ffa7c068380 0x7ffa7c06a840 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:49.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:49.512+0000 7ffaa5200640 1 --2- 172.21.15.16:0/3356115002 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffaa0143f40 0x7ffaa00d02b0 unknown :-1 s=CLOSED pgs=1069 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:49.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:49.512+0000 7ffaa5200640 1 --2- 172.21.15.16:0/3356115002 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffaa00b45d0 0x7ffaa0151980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:49.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:49.512+0000 7ffaa5200640 1 --2- 172.21.15.16:0/3356115002 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ffaa00b2120 0x7ffaa013cb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:49.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:49.512+0000 7ffaa5200640 1 -- 172.21.15.16:0/3356115002 >> 172.21.15.16:0/3356115002 conn(0x7ffaa006b620 msgr2=0x7ffaa00cc140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:09:49.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:49.512+0000 7ffaa5200640 1 -- 172.21.15.16:0/3356115002 shutdown_connections 2024-03-31T15:09:49.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:49.512+0000 7ffaa5200640 1 -- 172.21.15.16:0/3356115002 wait complete. 2024-03-31T15:09:49.528 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-31T15:09:49.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:49.684+0000 7f2034200640 1 Processor -- start 2024-03-31T15:09:49.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:49.684+0000 7f2034200640 1 -- start start 2024-03-31T15:09:49.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:49.684+0000 7f2034200640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f202c06f920 0x7f202c06fd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:49.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:49.684+0000 7f2034200640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f202c0d0a60 0x7f202c0cea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:49.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:49.684+0000 7f2034200640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f202c0cef80 0x7f202c070e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:49.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:49.684+0000 7f2034200640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f202c13f670 con 0x7f202c0d0a60 2024-03-31T15:09:49.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:49.684+0000 7f2034200640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f202c070240 con 0x7f202c0cef80 2024-03-31T15:09:49.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:49.684+0000 7f2034200640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f202c15ca70 con 0x7f202c06f920 2024-03-31T15:09:49.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:49.684+0000 7f2030e00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f202c0d0a60 0x7f202c0cea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:49.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:49.684+0000 7f2030e00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f202c0d0a60 0x7f202c0cea40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59440/0 (socket says 172.21.15.16:59440) 2024-03-31T15:09:49.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:49.684+0000 7f2030e00640 1 -- 172.21.15.16:0/985003731 learned_addr learned my addr 172.21.15.16:0/985003731 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:09:49.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:49.684+0000 7f2031800640 1 --2- 172.21.15.16:0/985003731 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f202c06f920 0x7f202c06fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:49.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:49.684+0000 7f2032200640 1 --2- 172.21.15.16:0/985003731 >> v2:172.21.15.89:3300/0 conn(0x7f202c0cef80 0x7f202c070e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:49.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:49.684+0000 7f2030e00640 1 -- 172.21.15.16:0/985003731 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f202c06f920 msgr2=0x7f202c06fd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:09:49.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:49.684+0000 7f2030e00640 1 --2- 172.21.15.16:0/985003731 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f202c06f920 0x7f202c06fd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:49.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:49.684+0000 7f2030e00640 1 -- 172.21.15.16:0/985003731 >> v2:172.21.15.89:3300/0 conn(0x7f202c0cef80 msgr2=0x7f202c070e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:09:49.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:49.684+0000 7f2030e00640 1 --2- 172.21.15.16:0/985003731 >> v2:172.21.15.89:3300/0 conn(0x7f202c0cef80 0x7f202c070e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:49.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:49.684+0000 7f2030e00640 1 -- 172.21.15.16:0/985003731 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f202c15cd50 con 0x7f202c0d0a60 2024-03-31T15:09:49.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:49.688+0000 7f2030e00640 1 --2- 172.21.15.16:0/985003731 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f202c0d0a60 0x7f202c0cea40 secure :-1 s=READY pgs=1070 cs=0 l=1 rev1=1 crypto rx=0x7f202c0b4cd0 tx=0x7f201801c260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f41ca6373983cb server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:09:49.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:49.688+0000 7f202be00640 1 -- 172.21.15.16:0/985003731 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20180b0070 con 0x7f202c0d0a60 2024-03-31T15:09:49.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:49.688+0000 7f202be00640 1 -- 172.21.15.16:0/985003731 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f20180ab600 con 0x7f202c0d0a60 2024-03-31T15:09:49.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:49.688+0000 7f202be00640 1 -- 172.21.15.16:0/985003731 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20180ab9d0 con 0x7f202c0d0a60 2024-03-31T15:09:49.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:49.688+0000 7f2034200640 1 -- 172.21.15.16:0/985003731 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f202c0d0a60 msgr2=0x7f202c0cea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:49.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:49.688+0000 7f2034200640 1 --2- 172.21.15.16:0/985003731 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f202c0d0a60 0x7f202c0cea40 secure :-1 s=READY pgs=1070 cs=0 l=1 rev1=1 crypto rx=0x7f202c0b4cd0 tx=0x7f201801c260 comp rx=0 tx=0).stop 2024-03-31T15:09:49.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:49.688+0000 7f2034200640 1 -- 172.21.15.16:0/985003731 shutdown_connections 2024-03-31T15:09:49.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:49.688+0000 7f2034200640 1 --2- 172.21.15.16:0/985003731 >> v2:172.21.15.89:3300/0 conn(0x7f202c0cef80 0x7f202c070e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:49.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:49.688+0000 7f2034200640 1 --2- 172.21.15.16:0/985003731 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f202c0d0a60 0x7f202c0cea40 unknown :-1 s=CLOSED pgs=1070 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:49.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:49.688+0000 7f2034200640 1 --2- 172.21.15.16:0/985003731 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f202c06f920 0x7f202c06fd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:49.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:49.688+0000 7f2034200640 1 -- 172.21.15.16:0/985003731 >> 172.21.15.16:0/985003731 conn(0x7f202c067930 msgr2=0x7f202c0b3ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:09:49.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:49.688+0000 7f2034200640 1 -- 172.21.15.16:0/985003731 shutdown_connections 2024-03-31T15:09:49.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:49.688+0000 7f2034200640 1 -- 172.21.15.16:0/985003731 wait complete. 2024-03-31T15:09:49.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:49.692+0000 7f2034200640 1 Processor -- start 2024-03-31T15:09:49.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:49.692+0000 7f2034200640 1 -- start start 2024-03-31T15:09:49.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:49.692+0000 7f2034200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f202c06f920 0x7f202c155ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:49.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:49.692+0000 7f2034200640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f202c0cef80 0x7f202c166680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:49.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:49.692+0000 7f2034200640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f202c0d0a60 0x7f202c15a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:49.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:49.692+0000 7f2034200640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f202c1419c0 con 0x7f202c06f920 2024-03-31T15:09:49.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:49.692+0000 7f2034200640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f202c0cf390 con 0x7f202c0d0a60 2024-03-31T15:09:49.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:49.692+0000 7f2034200640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f202c141b40 con 0x7f202c0cef80 2024-03-31T15:09:49.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:49.692+0000 7f2031800640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f202c06f920 0x7f202c155ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:49.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:49.692+0000 7f2031800640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f202c06f920 0x7f202c155ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59456/0 (socket says 172.21.15.16:59456) 2024-03-31T15:09:49.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:49.692+0000 7f2031800640 1 -- 172.21.15.16:0/2654993274 learned_addr learned my addr 172.21.15.16:0/2654993274 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:09:49.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:49.692+0000 7f2030e00640 1 --2- 172.21.15.16:0/2654993274 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f202c0cef80 0x7f202c166680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:49.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:49.692+0000 7f2032200640 1 --2- 172.21.15.16:0/2654993274 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f202c0d0a60 0x7f202c15a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:49.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:49.692+0000 7f2031800640 1 -- 172.21.15.16:0/2654993274 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f202c0cef80 msgr2=0x7f202c166680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:49.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:49.692+0000 7f2031800640 1 --2- 172.21.15.16:0/2654993274 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f202c0cef80 0x7f202c166680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:49.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:49.692+0000 7f2031800640 1 -- 172.21.15.16:0/2654993274 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f202c0d0a60 msgr2=0x7f202c15a2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:49.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:49.692+0000 7f2031800640 1 --2- 172.21.15.16:0/2654993274 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f202c0d0a60 0x7f202c15a2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:49.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:49.692+0000 7f2031800640 1 -- 172.21.15.16:0/2654993274 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f202c156410 con 0x7f202c06f920 2024-03-31T15:09:49.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:49.692+0000 7f2032200640 1 --2- 172.21.15.16:0/2654993274 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f202c0d0a60 0x7f202c15a2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:09:49.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:49.692+0000 7f2031800640 1 --2- 172.21.15.16:0/2654993274 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f202c06f920 0x7f202c155ed0 secure :-1 s=READY pgs=1071 cs=0 l=1 rev1=1 crypto rx=0x7f202000e960 tx=0x7f202000ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:09:49.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:49.692+0000 7f2029600640 1 -- 172.21.15.16:0/2654993274 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2020017070 con 0x7f202c06f920 2024-03-31T15:09:49.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:49.692+0000 7f2029600640 1 -- 172.21.15.16:0/2654993274 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f202000ca00 con 0x7f202c06f920 2024-03-31T15:09:49.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:49.692+0000 7f2029600640 1 -- 172.21.15.16:0/2654993274 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f202000cdb0 con 0x7f202c06f920 2024-03-31T15:09:49.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:49.692+0000 7f2034200640 1 -- 172.21.15.16:0/2654993274 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f202c165710 con 0x7f202c06f920 2024-03-31T15:09:49.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:49.692+0000 7f2034200640 1 -- 172.21.15.16:0/2654993274 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f202c165cd0 con 0x7f202c06f920 2024-03-31T15:09:49.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:49.692+0000 7f2034200640 1 -- 172.21.15.16:0/2654993274 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f202c069950 con 0x7f202c06f920 2024-03-31T15:09:49.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:49.696+0000 7f2029600640 1 -- 172.21.15.16:0/2654993274 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f202001a070 con 0x7f202c06f920 2024-03-31T15:09:49.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:49.696+0000 7f2029600640 1 --2- 172.21.15.16:0/2654993274 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f200c068380 0x7f200c06a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:49.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:49.696+0000 7f2029600640 1 -- 172.21.15.16:0/2654993274 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 5227+0+0 (secure 0 0 0) 0x7f2020086ef0 con 0x7f202c06f920 2024-03-31T15:09:49.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:49.696+0000 7f2030e00640 1 --2- 172.21.15.16:0/2654993274 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f200c068380 0x7f200c06a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:49.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:49.696+0000 7f2030e00640 1 --2- 172.21.15.16:0/2654993274 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f200c068380 0x7f200c06a840 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f20180ad000 tx=0x7f20180a6000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:09:49.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:49.700+0000 7f2029600640 1 -- 172.21.15.16:0/2654993274 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f202c156410 con 0x7f202c06f920 2024-03-31T15:09:49.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:49.864+0000 7f2034200640 1 -- 172.21.15.16:0/2654993274 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7f202c06c730 con 0x7f202c06f920 2024-03-31T15:09:50.508 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:50.516+0000 7f2029600640 1 -- 172.21.15.16:0/2654993274 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v79) ==== 170+0+0 (secure 0 0 0) 0x7f2020057120 con 0x7f202c06f920 2024-03-31T15:09:50.508 INFO:teuthology.orchestra.run.smithi016.stderr:pool 'cephfs_data' created 2024-03-31T15:09:50.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:50.516+0000 7f2034200640 1 -- 172.21.15.16:0/2654993274 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f200c068380 msgr2=0x7f200c06a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:50.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:50.516+0000 7f2034200640 1 --2- 172.21.15.16:0/2654993274 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f200c068380 0x7f200c06a840 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f20180ad000 tx=0x7f20180a6000 comp rx=0 tx=0).stop 2024-03-31T15:09:50.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:50.516+0000 7f2034200640 1 -- 172.21.15.16:0/2654993274 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f202c06f920 msgr2=0x7f202c155ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:50.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:50.516+0000 7f2034200640 1 --2- 172.21.15.16:0/2654993274 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f202c06f920 0x7f202c155ed0 secure :-1 s=READY pgs=1071 cs=0 l=1 rev1=1 crypto rx=0x7f202000e960 tx=0x7f202000ee20 comp rx=0 tx=0).stop 2024-03-31T15:09:50.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:50.520+0000 7f2034200640 1 -- 172.21.15.16:0/2654993274 shutdown_connections 2024-03-31T15:09:50.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:50.520+0000 7f2034200640 1 --2- 172.21.15.16:0/2654993274 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f200c068380 0x7f200c06a840 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:50.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:50.520+0000 7f2034200640 1 --2- 172.21.15.16:0/2654993274 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f202c0d0a60 0x7f202c15a2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:50.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:50.520+0000 7f2034200640 1 --2- 172.21.15.16:0/2654993274 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f202c0cef80 0x7f202c166680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:50.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:50.520+0000 7f2034200640 1 --2- 172.21.15.16:0/2654993274 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f202c06f920 0x7f202c155ed0 unknown :-1 s=CLOSED pgs=1071 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:50.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:50.520+0000 7f2034200640 1 -- 172.21.15.16:0/2654993274 >> 172.21.15.16:0/2654993274 conn(0x7f202c067930 msgr2=0x7f202c140b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:09:50.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:50.520+0000 7f2034200640 1 -- 172.21.15.16:0/2654993274 shutdown_connections 2024-03-31T15:09:50.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:50.520+0000 7f2034200640 1 -- 172.21.15.16:0/2654993274 wait complete. 2024-03-31T15:09:50.550 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-31T15:09:50.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:50.700+0000 7f18f9400640 1 Processor -- start 2024-03-31T15:09:50.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:50.700+0000 7f18f9400640 1 -- start start 2024-03-31T15:09:50.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:50.704+0000 7f18f9400640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f18f413dff0 0x7f18f406d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:50.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:50.704+0000 7f18f9400640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f18f40cd0e0 0x7f18f406da10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:50.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:50.704+0000 7f18f9400640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f18f415fbc0 con 0x7f18f413e3d0 2024-03-31T15:09:50.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:50.704+0000 7f18f9400640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f18f415f730 con 0x7f18f413dff0 2024-03-31T15:09:50.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:50.704+0000 7f18f9400640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f18f415ee00 con 0x7f18f40cd0e0 2024-03-31T15:09:50.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:50.704+0000 7f18f2800640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f18f413dff0 0x7f18f406d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:50.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:50.704+0000 7f18f2800640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f18f413dff0 0x7f18f406d4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59466/0 (socket says 172.21.15.16:59466) 2024-03-31T15:09:50.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:50.704+0000 7f18f2800640 1 -- 172.21.15.16:0/2921937919 learned_addr learned my addr 172.21.15.16:0/2921937919 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:09:50.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:50.704+0000 7f18f3200640 1 --2- 172.21.15.16:0/2921937919 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f18f40cd0e0 0x7f18f406da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:50.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:50.704+0000 7f18f2800640 1 -- 172.21.15.16:0/2921937919 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f18f40cd0e0 msgr2=0x7f18f406da10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:09:50.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:50.704+0000 7f18f2800640 1 --2- 172.21.15.16:0/2921937919 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f18f40cd0e0 0x7f18f406da10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:50.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:50.704+0000 7f18f2800640 1 -- 172.21.15.16:0/2921937919 >> v1:172.21.15.89:6789/0 conn(0x7f18f413e3d0 legacy=0x7f18f40cc9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:50.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:50.704+0000 7f18f2800640 1 -- 172.21.15.16:0/2921937919 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18f415f0e0 con 0x7f18f413dff0 2024-03-31T15:09:50.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:50.708+0000 7f18f2800640 1 --2- 172.21.15.16:0/2921937919 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f18f413dff0 0x7f18f406d4d0 secure :-1 s=READY pgs=1072 cs=0 l=1 rev1=1 crypto rx=0x7f18e400bb10 tx=0x7f18e401eba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ed14468de81df64 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:09:50.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:50.708+0000 7f18f1e00640 1 -- 172.21.15.16:0/2921937919 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18e40b4070 con 0x7f18f413dff0 2024-03-31T15:09:50.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:50.708+0000 7f18f1e00640 1 -- 172.21.15.16:0/2921937919 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f18e40a7a80 con 0x7f18f413dff0 2024-03-31T15:09:50.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:50.708+0000 7f18f1e00640 1 -- 172.21.15.16:0/2921937919 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18e40a7e50 con 0x7f18f413dff0 2024-03-31T15:09:50.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:50.708+0000 7f18f9400640 1 -- 172.21.15.16:0/2921937919 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f18f413dff0 msgr2=0x7f18f406d4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:50.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:50.708+0000 7f18f9400640 1 --2- 172.21.15.16:0/2921937919 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f18f413dff0 0x7f18f406d4d0 secure :-1 s=READY pgs=1072 cs=0 l=1 rev1=1 crypto rx=0x7f18e400bb10 tx=0x7f18e401eba0 comp rx=0 tx=0).stop 2024-03-31T15:09:50.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:50.708+0000 7f18f9400640 1 -- 172.21.15.16:0/2921937919 shutdown_connections 2024-03-31T15:09:50.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:50.708+0000 7f18f9400640 1 --2- 172.21.15.16:0/2921937919 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f18f40cd0e0 0x7f18f406da10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:50.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:50.708+0000 7f18f9400640 1 --2- 172.21.15.16:0/2921937919 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f18f413dff0 0x7f18f406d4d0 unknown :-1 s=CLOSED pgs=1072 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:50.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:50.708+0000 7f18f9400640 1 -- 172.21.15.16:0/2921937919 >> 172.21.15.16:0/2921937919 conn(0x7f18f406ee40 msgr2=0x7f18f406b620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:09:50.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:50.708+0000 7f18f9400640 1 -- 172.21.15.16:0/2921937919 shutdown_connections 2024-03-31T15:09:50.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:50.708+0000 7f18f9400640 1 -- 172.21.15.16:0/2921937919 wait complete. 2024-03-31T15:09:50.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:50.708+0000 7f18f9400640 1 Processor -- start 2024-03-31T15:09:50.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:50.708+0000 7f18f9400640 1 -- start start 2024-03-31T15:09:50.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:50.712+0000 7f18f9400640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f18f40cd0e0 0x7f18f4163170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:50.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:50.712+0000 7f18f9400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f18f413dff0 0x7f18f413c110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:50.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:50.712+0000 7f18f9400640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f18f413e3d0 0x7f18f415deb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:50.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:50.712+0000 7f18f9400640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f18f40720f0 con 0x7f18f413dff0 2024-03-31T15:09:50.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:50.712+0000 7f18f9400640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f18f4071f70 con 0x7f18f413e3d0 2024-03-31T15:09:50.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:50.712+0000 7f18f9400640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f18f4072270 con 0x7f18f40cd0e0 2024-03-31T15:09:50.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:50.712+0000 7f18e9e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f18f413dff0 0x7f18f413c110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:50.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:50.712+0000 7f18e9e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f18f413dff0 0x7f18f413c110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59482/0 (socket says 172.21.15.16:59482) 2024-03-31T15:09:50.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:50.712+0000 7f18e9e00640 1 -- 172.21.15.16:0/448113827 learned_addr learned my addr 172.21.15.16:0/448113827 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:09:50.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:50.712+0000 7f18f2800640 1 --2- 172.21.15.16:0/448113827 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f18f40cd0e0 0x7f18f4163170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:50.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:50.712+0000 7f18f3200640 1 --2- 172.21.15.16:0/448113827 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f18f413e3d0 0x7f18f415deb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:50.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:50.712+0000 7f18f2800640 1 -- 172.21.15.16:0/448113827 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f18f413e3d0 msgr2=0x7f18f415deb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:50.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:50.712+0000 7f18f2800640 1 --2- 172.21.15.16:0/448113827 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f18f413e3d0 0x7f18f415deb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:50.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:50.712+0000 7f18f2800640 1 -- 172.21.15.16:0/448113827 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f18f413dff0 msgr2=0x7f18f413c110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:50.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:50.712+0000 7f18f2800640 1 --2- 172.21.15.16:0/448113827 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f18f413dff0 0x7f18f413c110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:50.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:50.712+0000 7f18f2800640 1 -- 172.21.15.16:0/448113827 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18f4160000 con 0x7f18f40cd0e0 2024-03-31T15:09:50.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:50.712+0000 7f18e9e00640 1 --2- 172.21.15.16:0/448113827 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f18f413dff0 0x7f18f413c110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:09:50.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:50.712+0000 7f18f2800640 1 --2- 172.21.15.16:0/448113827 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f18f40cd0e0 0x7f18f4163170 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f18e40ae000 tx=0x7f18e401d3e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:09:50.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:50.712+0000 7f18eb400640 1 -- 172.21.15.16:0/448113827 <== mon.2 v2:172.21.15.89:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18e40bf070 con 0x7f18f40cd0e0 2024-03-31T15:09:50.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:50.712+0000 7f18eb400640 1 -- 172.21.15.16:0/448113827 <== mon.2 v2:172.21.15.89:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f18e40b8550 con 0x7f18f40cd0e0 2024-03-31T15:09:50.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:50.712+0000 7f18eb400640 1 -- 172.21.15.16:0/448113827 <== mon.2 v2:172.21.15.89:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18e40b4070 con 0x7f18f40cd0e0 2024-03-31T15:09:50.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:50.712+0000 7f18f9400640 1 -- 172.21.15.16:0/448113827 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f18f41602e0 con 0x7f18f40cd0e0 2024-03-31T15:09:50.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:50.712+0000 7f18f9400640 1 -- 172.21.15.16:0/448113827 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f18f41608a0 con 0x7f18f40cd0e0 2024-03-31T15:09:50.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:50.712+0000 7f18f9400640 1 -- 172.21.15.16:0/448113827 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f18f4069950 con 0x7f18f40cd0e0 2024-03-31T15:09:50.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:50.712+0000 7f18eb400640 1 -- 172.21.15.16:0/448113827 <== mon.2 v2:172.21.15.89:3301/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f18e40a7940 con 0x7f18f40cd0e0 2024-03-31T15:09:50.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:50.716+0000 7f18eb400640 1 --2- 172.21.15.16:0/448113827 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f18d0068380 0x7f18d006a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:50.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:50.716+0000 7f18eb400640 1 -- 172.21.15.16:0/448113827 <== mon.2 v2:172.21.15.89:3301/0 5 ==== osd_map(79..79 src has 1..79) ==== 5609+0+0 (secure 0 0 0) 0x7f18e41210e0 con 0x7f18f40cd0e0 2024-03-31T15:09:50.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:50.716+0000 7f18e9e00640 1 --2- 172.21.15.16:0/448113827 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f18d0068380 0x7f18d006a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:50.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:50.716+0000 7f18e9e00640 1 --2- 172.21.15.16:0/448113827 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f18d0068380 0x7f18d006a840 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f18f41fa570 tx=0x7f18dc009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:09:50.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:50.720+0000 7f18eb400640 1 -- 172.21.15.16:0/448113827 <== mon.2 v2:172.21.15.89:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f18e4018820 con 0x7f18f40cd0e0 2024-03-31T15:09:50.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:50.920+0000 7f18f9400640 1 -- 172.21.15.16:0/448113827 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7f18f406e4e0 con 0x7f18f40cd0e0 2024-03-31T15:09:51.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:51.544+0000 7f18eb400640 1 -- 172.21.15.16:0/448113827 <== mon.2 v2:172.21.15.89:3301/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 12 and data pool 13 v57) ==== 174+0+0 (secure 0 0 0) 0x7f18e40f10a0 con 0x7f18f40cd0e0 2024-03-31T15:09:51.540 INFO:teuthology.orchestra.run.smithi016.stderr:new fs with metadata pool 12 and data pool 13 2024-03-31T15:09:51.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:51.552+0000 7f18f9400640 1 -- 172.21.15.16:0/448113827 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f18d0068380 msgr2=0x7f18d006a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:51.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:51.552+0000 7f18f9400640 1 --2- 172.21.15.16:0/448113827 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f18d0068380 0x7f18d006a840 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f18f41fa570 tx=0x7f18dc009040 comp rx=0 tx=0).stop 2024-03-31T15:09:51.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:51.552+0000 7f18f9400640 1 -- 172.21.15.16:0/448113827 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f18f40cd0e0 msgr2=0x7f18f4163170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:51.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:51.552+0000 7f18f9400640 1 --2- 172.21.15.16:0/448113827 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f18f40cd0e0 0x7f18f4163170 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f18e40ae000 tx=0x7f18e401d3e0 comp rx=0 tx=0).stop 2024-03-31T15:09:51.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:51.552+0000 7f18f9400640 1 -- 172.21.15.16:0/448113827 shutdown_connections 2024-03-31T15:09:51.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:51.552+0000 7f18f9400640 1 --2- 172.21.15.16:0/448113827 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f18d0068380 0x7f18d006a840 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:51.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:51.552+0000 7f18f9400640 1 --2- 172.21.15.16:0/448113827 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f18f413e3d0 0x7f18f415deb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:51.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:51.552+0000 7f18f9400640 1 --2- 172.21.15.16:0/448113827 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f18f413dff0 0x7f18f413c110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:51.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:51.552+0000 7f18f9400640 1 --2- 172.21.15.16:0/448113827 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f18f40cd0e0 0x7f18f4163170 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:51.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:51.552+0000 7f18f9400640 1 -- 172.21.15.16:0/448113827 >> 172.21.15.16:0/448113827 conn(0x7f18f406ee40 msgr2=0x7f18f40ad8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:09:51.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:51.556+0000 7f18f9400640 1 -- 172.21.15.16:0/448113827 shutdown_connections 2024-03-31T15:09:51.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:51.556+0000 7f18f9400640 1 -- 172.21.15.16:0/448113827 wait complete. 2024-03-31T15:09:51.577 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-31T15:09:51.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:51.728+0000 7fb573200640 1 Processor -- start 2024-03-31T15:09:51.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:51.728+0000 7fb573200640 1 -- start start 2024-03-31T15:09:51.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:51.728+0000 7fb573200640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb56c06f920 0x7fb56c06fd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:51.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:51.728+0000 7fb573200640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb56c0d2010 0x7fb56c0d24a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:51.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:51.728+0000 7fb573200640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb56c0d0920 con 0x7fb56c145e40 2024-03-31T15:09:51.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:51.728+0000 7fb573200640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7fb56c0702c0 con 0x7fb56c06f920 2024-03-31T15:09:51.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:51.728+0000 7fb573200640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fb56c0d06d0 con 0x7fb56c0d2010 2024-03-31T15:09:51.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:51.728+0000 7fb56be00640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fb56c145e40 0x7fb56c0cc740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:48210/0 (socket says 172.21.15.16:48210) 2024-03-31T15:09:51.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:51.728+0000 7fb56be00640 1 -- 172.21.15.16:0/3356135859 learned_addr learned my addr 172.21.15.16:0/3356135859 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:09:51.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:51.728+0000 7fb563400640 1 --2- 172.21.15.16:0/3356135859 >> v2:172.21.15.16:3300/0 conn(0x7fb56c06f920 0x7fb56c06fd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:51.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:51.728+0000 7fb571200640 1 --2- 172.21.15.16:0/3356135859 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb56c0d2010 0x7fb56c0d24a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:51.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:51.728+0000 7fb56b400640 1 -- 172.21.15.16:0/3356135859 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4185745268 0 0) 0x7fb56c0d0920 con 0x7fb56c145e40 2024-03-31T15:09:51.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:51.728+0000 7fb563400640 1 -- 172.21.15.16:0/3356135859 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb56c0d2010 msgr2=0x7fb56c0d24a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:09:51.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:51.728+0000 7fb563400640 1 --2- 172.21.15.16:0/3356135859 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb56c0d2010 0x7fb56c0d24a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:51.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:51.728+0000 7fb563400640 1 -- 172.21.15.16:0/3356135859 >> v1:172.21.15.16:6789/0 conn(0x7fb56c145e40 legacy=0x7fb56c0cc740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:51.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:51.728+0000 7fb563400640 1 -- 172.21.15.16:0/3356135859 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb56c0d0c00 con 0x7fb56c06f920 2024-03-31T15:09:51.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:51.732+0000 7fb563400640 1 --2- 172.21.15.16:0/3356135859 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb56c06f920 0x7fb56c06fd80 secure :-1 s=READY pgs=1074 cs=0 l=1 rev1=1 crypto rx=0x7fb5540097c0 tx=0x7fb55401c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c45fcbd2c900fc44 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:09:51.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:51.732+0000 7fb56b400640 1 -- 172.21.15.16:0/3356135859 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5540b2070 con 0x7fb56c06f920 2024-03-31T15:09:51.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:51.732+0000 7fb56b400640 1 -- 172.21.15.16:0/3356135859 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb5540ad640 con 0x7fb56c06f920 2024-03-31T15:09:51.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:51.732+0000 7fb56b400640 1 -- 172.21.15.16:0/3356135859 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5540ada10 con 0x7fb56c06f920 2024-03-31T15:09:51.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:51.732+0000 7fb573200640 1 -- 172.21.15.16:0/3356135859 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb56c06f920 msgr2=0x7fb56c06fd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:51.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:51.732+0000 7fb573200640 1 --2- 172.21.15.16:0/3356135859 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb56c06f920 0x7fb56c06fd80 secure :-1 s=READY pgs=1074 cs=0 l=1 rev1=1 crypto rx=0x7fb5540097c0 tx=0x7fb55401c4a0 comp rx=0 tx=0).stop 2024-03-31T15:09:51.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:51.732+0000 7fb573200640 1 -- 172.21.15.16:0/3356135859 shutdown_connections 2024-03-31T15:09:51.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:51.732+0000 7fb573200640 1 --2- 172.21.15.16:0/3356135859 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb56c0d2010 0x7fb56c0d24a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:51.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:51.732+0000 7fb573200640 1 --2- 172.21.15.16:0/3356135859 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb56c06f920 0x7fb56c06fd80 unknown :-1 s=CLOSED pgs=1074 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:51.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:51.732+0000 7fb573200640 1 -- 172.21.15.16:0/3356135859 >> 172.21.15.16:0/3356135859 conn(0x7fb56c067330 msgr2=0x7fb56c067740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:09:51.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:51.732+0000 7fb573200640 1 -- 172.21.15.16:0/3356135859 shutdown_connections 2024-03-31T15:09:51.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:51.732+0000 7fb573200640 1 -- 172.21.15.16:0/3356135859 wait complete. 2024-03-31T15:09:51.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:51.736+0000 7fb573200640 1 Processor -- start 2024-03-31T15:09:51.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:51.736+0000 7fb573200640 1 -- start start 2024-03-31T15:09:51.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:51.736+0000 7fb573200640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb56c06f920 0x7fb56c15ec60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:51.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:51.736+0000 7fb573200640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb56c0d2010 0x7fb56c13f690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:51.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:51.736+0000 7fb573200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb56c145e40 0x7fb56c06cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:51.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:51.736+0000 7fb573200640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fb56c0d0ee0 con 0x7fb56c145e40 2024-03-31T15:09:51.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:51.736+0000 7fb573200640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fb56c0ce9e0 con 0x7fb56c0d2010 2024-03-31T15:09:51.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:51.736+0000 7fb573200640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fb56c0d2df0 con 0x7fb56c06f920 2024-03-31T15:09:51.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:51.736+0000 7fb571200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb56c145e40 0x7fb56c06cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:51.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:51.736+0000 7fb56be00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb56c06f920 0x7fb56c15ec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:51.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:51.736+0000 7fb571200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb56c145e40 0x7fb56c06cfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59502/0 (socket says 172.21.15.16:59502) 2024-03-31T15:09:51.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:51.736+0000 7fb571200640 1 -- 172.21.15.16:0/1055542104 learned_addr learned my addr 172.21.15.16:0/1055542104 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:09:51.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:51.736+0000 7fb563400640 1 --2- 172.21.15.16:0/1055542104 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb56c0d2010 0x7fb56c13f690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:51.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:51.736+0000 7fb571200640 1 -- 172.21.15.16:0/1055542104 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb56c06f920 msgr2=0x7fb56c15ec60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:51.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:51.736+0000 7fb571200640 1 --2- 172.21.15.16:0/1055542104 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb56c06f920 0x7fb56c15ec60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:51.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:51.736+0000 7fb571200640 1 -- 172.21.15.16:0/1055542104 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb56c0d2010 msgr2=0x7fb56c13f690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:51.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:51.736+0000 7fb571200640 1 --2- 172.21.15.16:0/1055542104 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb56c0d2010 0x7fb56c13f690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:51.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:51.736+0000 7fb571200640 1 -- 172.21.15.16:0/1055542104 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb56c0cf1f0 con 0x7fb56c145e40 2024-03-31T15:09:51.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:51.736+0000 7fb571200640 1 --2- 172.21.15.16:0/1055542104 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb56c145e40 0x7fb56c06cfd0 secure :-1 s=READY pgs=1075 cs=0 l=1 rev1=1 crypto rx=0x7fb564007e70 tx=0x7fb56400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:09:51.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:51.736+0000 7fb568c00640 1 -- 172.21.15.16:0/1055542104 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb564014620 con 0x7fb56c145e40 2024-03-31T15:09:51.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:51.736+0000 7fb568c00640 1 -- 172.21.15.16:0/1055542104 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb564019070 con 0x7fb56c145e40 2024-03-31T15:09:51.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:51.736+0000 7fb573200640 1 -- 172.21.15.16:0/1055542104 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb56c06e120 con 0x7fb56c145e40 2024-03-31T15:09:51.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:51.736+0000 7fb568c00640 1 -- 172.21.15.16:0/1055542104 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb564014a20 con 0x7fb56c145e40 2024-03-31T15:09:51.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:51.736+0000 7fb573200640 1 -- 172.21.15.16:0/1055542104 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb56c06de40 con 0x7fb56c145e40 2024-03-31T15:09:51.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:51.740+0000 7fb573200640 1 -- 172.21.15.16:0/1055542104 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb56c069950 con 0x7fb56c145e40 2024-03-31T15:09:51.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:51.740+0000 7fb568c00640 1 -- 172.21.15.16:0/1055542104 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fb564014bc0 con 0x7fb56c145e40 2024-03-31T15:09:51.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:51.740+0000 7fb568c00640 1 --2- 172.21.15.16:0/1055542104 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fb548068380 0x7fb54806a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:51.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:51.740+0000 7fb568c00640 1 -- 172.21.15.16:0/1055542104 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 5709+0+0 (secure 0 0 0) 0x7fb564087180 con 0x7fb56c145e40 2024-03-31T15:09:51.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:51.740+0000 7fb56be00640 1 --2- 172.21.15.16:0/1055542104 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fb548068380 0x7fb54806a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:51.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:51.740+0000 7fb56be00640 1 --2- 172.21.15.16:0/1055542104 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fb548068380 0x7fb54806a840 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7fb55c007fd0 tx=0x7fb55c007d60 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:09:51.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:51.744+0000 7fb568c00640 1 -- 172.21.15.16:0/1055542104 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fb5640570f0 con 0x7fb56c145e40 2024-03-31T15:09:51.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:51.912+0000 7fb573200640 1 -- 172.21.15.16:0/1055542104 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fb56c1596b0 con 0x7fb56c145e40 2024-03-31T15:09:51.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:51.916+0000 7fb568c00640 1 -- 172.21.15.16:0/1055542104 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v80) ==== 74+0+16601 (secure 0 0 0) 0x7fb56405b3f0 con 0x7fb56c145e40 2024-03-31T15:09:51.907 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:09:51.908 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":80,"fsid":"286eb5da-fb47-4550-b290-d9519c13b9a5","created":"2024-03-31T15:03:11.438952+0000","modified":"2024-03-31T15:09:50.925850+0000","last_up_change":"2024-03-31T15:03:22.885535+0000","last_in_change":"2024-03-31T15:03:13.941667+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-31T15:03:22.550787+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-03-31T15:09:49.322620+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-03-31T15:09:49.877778+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"92a66450-ef19-40d8-ac3d-d760b23b0bda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6801","nonce":932712244}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6803","nonce":932712244}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6807","nonce":932712244}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6805","nonce":932712244}]},"public_addr":"172.21.15.16:6801/932712244","cluster_addr":"172.21.15.16:6803/932712244","heartbeat_back_addr":"172.21.15.16:6807/932712244","heartbeat_front_addr":"172.21.15.16:6805/932712244","state":["exists","up"]},{"osd":1,"uuid":"da9d1f2b-1968-4959-b843-faa231988e92","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6810","nonce":1398737533}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6811","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6813","nonce":1398737533}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6821","nonce":1398737533}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6815","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6816","nonce":1398737533}]},"public_addr":"172.21.15.16:6810/1398737533","cluster_addr":"172.21.15.16:6813/1398737533","heartbeat_back_addr":"172.21.15.16:6821/1398737533","heartbeat_front_addr":"172.21.15.16:6816/1398737533","state":["exists","up"]},{"osd":2,"uuid":"01a73662-9a21-4b1f-bcbb-42b9636a4638","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6819","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6822","nonce":3018526646}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6825","nonce":3018526646}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6831","nonce":3018526646}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6827","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6829","nonce":3018526646}]},"public_addr":"172.21.15.16:6822/3018526646","cluster_addr":"172.21.15.16:6825/3018526646","heartbeat_back_addr":"172.21.15.16:6831/3018526646","heartbeat_front_addr":"172.21.15.16:6829/3018526646","state":["exists","up"]},{"osd":3,"uuid":"c7771fd3-e18e-4500-ac38-23cf10ebcd75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6809","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6812","nonce":3360352974}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6817","nonce":3360352974}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6828","nonce":3360352974}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6823","nonce":3360352974}]},"public_addr":"172.21.15.16:6812/3360352974","cluster_addr":"172.21.15.16:6817/3360352974","heartbeat_back_addr":"172.21.15.16:6828/3360352974","heartbeat_front_addr":"172.21.15.16:6823/3360352974","state":["exists","up"]},{"osd":4,"uuid":"88260730-1482-4fbd-a40f-cb1124e9df4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6801","nonce":1856844856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6803","nonce":1856844856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6807","nonce":1856844856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6805","nonce":1856844856}]},"public_addr":"172.21.15.89:6801/1856844856","cluster_addr":"172.21.15.89:6803/1856844856","heartbeat_back_addr":"172.21.15.89:6807/1856844856","heartbeat_front_addr":"172.21.15.89:6805/1856844856","state":["exists","up"]},{"osd":5,"uuid":"4cb2c00c-fae0-4dd8-9d27-ec4f0093ccf7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6814","nonce":50395527}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6817","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6820","nonce":50395527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6831","nonce":50395527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6823","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6826","nonce":50395527}]},"public_addr":"172.21.15.89:6814/50395527","cluster_addr":"172.21.15.89:6820/50395527","heartbeat_back_addr":"172.21.15.89:6831/50395527","heartbeat_front_addr":"172.21.15.89:6826/50395527","state":["exists","up"]},{"osd":6,"uuid":"81ce0c19-2554-4cae-acbe-66a05e9e9b9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6809","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6812","nonce":3470471752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6815","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6818","nonce":3470471752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6827","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6829","nonce":3470471752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6821","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6824","nonce":3470471752}]},"public_addr":"172.21.15.89:6812/3470471752","cluster_addr":"172.21.15.89:6818/3470471752","heartbeat_back_addr":"172.21.15.89:6829/3470471752","heartbeat_front_addr":"172.21.15.89:6824/3470471752","state":["exists","up"]},{"osd":7,"uuid":"85e4313b-d37d-442e-8042-153bf20a847d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6811","nonce":1796778630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6813","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6816","nonce":1796778630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6825","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6828","nonce":1796778630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6819","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6822","nonce":1796778630}]},"public_addr":"172.21.15.89:6811/1796778630","cluster_addr":"172.21.15.89:6816/1796778630","heartbeat_back_addr":"172.21.15.89:6828/1796778630","heartbeat_front_addr":"172.21.15.89:6822/1796778630","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.439088+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.529970+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.680512+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:21.044161+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.582111+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.536558+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.409522+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.590229+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-31T15:09:51.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:51.916+0000 7fb573200640 1 -- 172.21.15.16:0/1055542104 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fb548068380 msgr2=0x7fb54806a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:51.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:51.916+0000 7fb573200640 1 --2- 172.21.15.16:0/1055542104 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fb548068380 0x7fb54806a840 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7fb55c007fd0 tx=0x7fb55c007d60 comp rx=0 tx=0).stop 2024-03-31T15:09:51.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:51.916+0000 7fb573200640 1 -- 172.21.15.16:0/1055542104 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb56c145e40 msgr2=0x7fb56c06cfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:51.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:51.916+0000 7fb573200640 1 --2- 172.21.15.16:0/1055542104 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb56c145e40 0x7fb56c06cfd0 secure :-1 s=READY pgs=1075 cs=0 l=1 rev1=1 crypto rx=0x7fb564007e70 tx=0x7fb56400e4b0 comp rx=0 tx=0).stop 2024-03-31T15:09:51.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:51.916+0000 7fb573200640 1 -- 172.21.15.16:0/1055542104 shutdown_connections 2024-03-31T15:09:51.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:51.916+0000 7fb573200640 1 --2- 172.21.15.16:0/1055542104 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fb548068380 0x7fb54806a840 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:51.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:51.916+0000 7fb573200640 1 --2- 172.21.15.16:0/1055542104 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb56c145e40 0x7fb56c06cfd0 unknown :-1 s=CLOSED pgs=1075 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:51.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:51.916+0000 7fb573200640 1 --2- 172.21.15.16:0/1055542104 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb56c0d2010 0x7fb56c13f690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:51.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:51.916+0000 7fb573200640 1 --2- 172.21.15.16:0/1055542104 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb56c06f920 0x7fb56c15ec60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:51.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:51.916+0000 7fb573200640 1 -- 172.21.15.16:0/1055542104 >> 172.21.15.16:0/1055542104 conn(0x7fb56c067330 msgr2=0x7fb56c158a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:09:51.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:51.916+0000 7fb573200640 1 -- 172.21.15.16:0/1055542104 shutdown_connections 2024-03-31T15:09:51.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:51.916+0000 7fb573200640 1 -- 172.21.15.16:0/1055542104 wait complete. 2024-03-31T15:09:51.928 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-31T15:09:52.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:52.076+0000 7f7835600640 1 Processor -- start 2024-03-31T15:09:52.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:52.076+0000 7f7835600640 1 -- start start 2024-03-31T15:09:52.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:52.076+0000 7f7835600640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f783015b3d0 0x7f78300b1cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:52.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:52.076+0000 7f7835600640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f783015eee0 0x7f78300b2200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:52.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:52.076+0000 7f7835600640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f783015f3c0 con 0x7f78300ce2f0 2024-03-31T15:09:52.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:52.076+0000 7f7835600640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f783015b7b0 con 0x7f783015eee0 2024-03-31T15:09:52.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:52.076+0000 7f7835600640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f783015f6a0 con 0x7f783015b3d0 2024-03-31T15:09:52.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:52.076+0000 7f7827e00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f783015eee0 0x7f78300b2200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:52.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:52.076+0000 7f7827e00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f783015eee0 0x7f78300b2200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59512/0 (socket says 172.21.15.16:59512) 2024-03-31T15:09:52.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:52.076+0000 7f7827e00640 1 -- 172.21.15.16:0/3177402260 learned_addr learned my addr 172.21.15.16:0/3177402260 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:09:52.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:52.076+0000 7f782f400640 1 --1- 172.21.15.16:0/3177402260 >> v1:172.21.15.16:6789/0 conn(0x7f78300ce2f0 0x7f78300cc9c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:48220/0 (socket says 172.21.15.16:48220) 2024-03-31T15:09:52.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:52.076+0000 7f782ea00640 1 --2- 172.21.15.16:0/3177402260 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f783015b3d0 0x7f78300b1cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:52.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:52.076+0000 7f7827e00640 1 -- 172.21.15.16:0/3177402260 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f783015b3d0 msgr2=0x7f78300b1cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:09:52.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:52.076+0000 7f7827e00640 1 --2- 172.21.15.16:0/3177402260 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f783015b3d0 0x7f78300b1cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:52.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:52.076+0000 7f7827e00640 1 -- 172.21.15.16:0/3177402260 >> v1:172.21.15.16:6789/0 conn(0x7f78300ce2f0 legacy=0x7f78300cc9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:52.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:52.076+0000 7f782e000640 1 -- 172.21.15.16:0/3177402260 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4260656615 0 0) 0x7f783015f3c0 con 0x7f78300ce2f0 2024-03-31T15:09:52.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:52.080+0000 7f7827e00640 1 -- 172.21.15.16:0/3177402260 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f783015f980 con 0x7f783015eee0 2024-03-31T15:09:52.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:52.080+0000 7f782ea00640 1 --2- 172.21.15.16:0/3177402260 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f783015b3d0 0x7f78300b1cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:09:52.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:52.080+0000 7f7827e00640 1 --2- 172.21.15.16:0/3177402260 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f783015eee0 0x7f78300b2200 secure :-1 s=READY pgs=1077 cs=0 l=1 rev1=1 crypto rx=0x7f7818009870 tx=0x7f781801c530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1da832d124a04522 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:09:52.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:52.080+0000 7f782e000640 1 -- 172.21.15.16:0/3177402260 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78180aa650 con 0x7f783015eee0 2024-03-31T15:09:52.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:52.080+0000 7f782e000640 1 -- 172.21.15.16:0/3177402260 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f78180af070 con 0x7f783015eee0 2024-03-31T15:09:52.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:52.080+0000 7f782e000640 1 -- 172.21.15.16:0/3177402260 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78180aaa10 con 0x7f783015eee0 2024-03-31T15:09:52.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:52.080+0000 7f7835600640 1 -- 172.21.15.16:0/3177402260 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f783015eee0 msgr2=0x7f78300b2200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:52.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:52.080+0000 7f7835600640 1 --2- 172.21.15.16:0/3177402260 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f783015eee0 0x7f78300b2200 secure :-1 s=READY pgs=1077 cs=0 l=1 rev1=1 crypto rx=0x7f7818009870 tx=0x7f781801c530 comp rx=0 tx=0).stop 2024-03-31T15:09:52.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:52.080+0000 7f7835600640 1 -- 172.21.15.16:0/3177402260 shutdown_connections 2024-03-31T15:09:52.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:52.080+0000 7f7835600640 1 --2- 172.21.15.16:0/3177402260 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f783015eee0 0x7f78300b2200 unknown :-1 s=CLOSED pgs=1077 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:52.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:52.080+0000 7f7835600640 1 --2- 172.21.15.16:0/3177402260 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f783015b3d0 0x7f78300b1cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:52.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:52.080+0000 7f7835600640 1 -- 172.21.15.16:0/3177402260 >> 172.21.15.16:0/3177402260 conn(0x7f783006b620 msgr2=0x7f783006ba30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:09:52.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:52.080+0000 7f7835600640 1 -- 172.21.15.16:0/3177402260 shutdown_connections 2024-03-31T15:09:52.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:52.080+0000 7f7835600640 1 -- 172.21.15.16:0/3177402260 wait complete. 2024-03-31T15:09:52.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:52.084+0000 7f7835600640 1 Processor -- start 2024-03-31T15:09:52.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:52.084+0000 7f7835600640 1 -- start start 2024-03-31T15:09:52.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:52.084+0000 7f7835600640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f78300ce2f0 0x7f7830157af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:52.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:52.084+0000 7f7835600640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f783015b3d0 0x7f7830159d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:52.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:52.084+0000 7f7835600640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f783015eee0 0x7f783015a290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:52.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:52.084+0000 7f7835600640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f7830168610 con 0x7f783015eee0 2024-03-31T15:09:52.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:52.084+0000 7f7835600640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f7830168490 con 0x7f78300ce2f0 2024-03-31T15:09:52.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:52.084+0000 7f7835600640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f7830168790 con 0x7f783015b3d0 2024-03-31T15:09:52.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:52.084+0000 7f782f400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f783015eee0 0x7f783015a290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:52.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:52.084+0000 7f782f400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f783015eee0 0x7f783015a290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59522/0 (socket says 172.21.15.16:59522) 2024-03-31T15:09:52.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:52.084+0000 7f782f400640 1 -- 172.21.15.16:0/2407950828 learned_addr learned my addr 172.21.15.16:0/2407950828 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:09:52.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:52.084+0000 7f7827e00640 1 --2- 172.21.15.16:0/2407950828 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f783015b3d0 0x7f7830159d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:52.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:52.084+0000 7f782ea00640 1 --2- 172.21.15.16:0/2407950828 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f78300ce2f0 0x7f7830157af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:52.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:52.084+0000 7f782f400640 1 -- 172.21.15.16:0/2407950828 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f783015b3d0 msgr2=0x7f7830159d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:52.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:52.084+0000 7f782f400640 1 --2- 172.21.15.16:0/2407950828 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f783015b3d0 0x7f7830159d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:52.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:52.084+0000 7f782f400640 1 -- 172.21.15.16:0/2407950828 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f78300ce2f0 msgr2=0x7f7830157af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:52.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:52.084+0000 7f782f400640 1 --2- 172.21.15.16:0/2407950828 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f78300ce2f0 0x7f7830157af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:52.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:52.084+0000 7f782f400640 1 -- 172.21.15.16:0/2407950828 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7830156740 con 0x7f783015eee0 2024-03-31T15:09:52.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:52.084+0000 7f782ea00640 1 --2- 172.21.15.16:0/2407950828 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f78300ce2f0 0x7f7830157af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:09:52.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:52.084+0000 7f782f400640 1 --2- 172.21.15.16:0/2407950828 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f783015eee0 0x7f783015a290 secure :-1 s=READY pgs=1078 cs=0 l=1 rev1=1 crypto rx=0x7f782800c7b0 tx=0x7f782800cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:09:52.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:52.084+0000 7f7826a00640 1 -- 172.21.15.16:0/2407950828 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7828018070 con 0x7f783015eee0 2024-03-31T15:09:52.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:52.084+0000 7f7826a00640 1 -- 172.21.15.16:0/2407950828 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f782800a640 con 0x7f783015eee0 2024-03-31T15:09:52.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:52.084+0000 7f7835600640 1 -- 172.21.15.16:0/2407950828 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7830156a20 con 0x7f783015eee0 2024-03-31T15:09:52.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:52.084+0000 7f7826a00640 1 -- 172.21.15.16:0/2407950828 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f782800aa70 con 0x7f783015eee0 2024-03-31T15:09:52.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:52.084+0000 7f7835600640 1 -- 172.21.15.16:0/2407950828 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7830156fe0 con 0x7f783015eee0 2024-03-31T15:09:52.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:52.088+0000 7f7835600640 1 -- 172.21.15.16:0/2407950828 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7830069950 con 0x7f783015eee0 2024-03-31T15:09:52.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:52.088+0000 7f7826a00640 1 -- 172.21.15.16:0/2407950828 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f782801b070 con 0x7f783015eee0 2024-03-31T15:09:52.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:52.088+0000 7f7826a00640 1 --2- 172.21.15.16:0/2407950828 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f780c0683b0 0x7f780c06a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:52.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:52.088+0000 7f7826a00640 1 -- 172.21.15.16:0/2407950828 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 5709+0+0 (secure 0 0 0) 0x7f7828085fa0 con 0x7f783015eee0 2024-03-31T15:09:52.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:52.088+0000 7f782ea00640 1 --2- 172.21.15.16:0/2407950828 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f780c0683b0 0x7f780c06a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:52.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:52.088+0000 7f782ea00640 1 --2- 172.21.15.16:0/2407950828 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f780c0683b0 0x7f780c06a870 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f7820006fd0 tx=0x7f7820008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:09:52.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:52.092+0000 7f7826a00640 1 -- 172.21.15.16:0/2407950828 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f7828055e70 con 0x7f783015eee0 2024-03-31T15:09:52.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:52.256+0000 7f7835600640 1 -- 172.21.15.16:0/2407950828 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f7830148690 con 0x7f783015eee0 2024-03-31T15:09:52.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:52.260+0000 7f7826a00640 1 -- 172.21.15.16:0/2407950828 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v80) ==== 74+0+16601 (secure 0 0 0) 0x7f782805a170 con 0x7f783015eee0 2024-03-31T15:09:52.252 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:09:52.252 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":80,"fsid":"286eb5da-fb47-4550-b290-d9519c13b9a5","created":"2024-03-31T15:03:11.438952+0000","modified":"2024-03-31T15:09:50.925850+0000","last_up_change":"2024-03-31T15:03:22.885535+0000","last_in_change":"2024-03-31T15:03:13.941667+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-31T15:03:22.550787+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-03-31T15:09:49.322620+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-03-31T15:09:49.877778+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"92a66450-ef19-40d8-ac3d-d760b23b0bda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6801","nonce":932712244}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6803","nonce":932712244}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6807","nonce":932712244}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6805","nonce":932712244}]},"public_addr":"172.21.15.16:6801/932712244","cluster_addr":"172.21.15.16:6803/932712244","heartbeat_back_addr":"172.21.15.16:6807/932712244","heartbeat_front_addr":"172.21.15.16:6805/932712244","state":["exists","up"]},{"osd":1,"uuid":"da9d1f2b-1968-4959-b843-faa231988e92","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6810","nonce":1398737533}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6811","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6813","nonce":1398737533}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6821","nonce":1398737533}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6815","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6816","nonce":1398737533}]},"public_addr":"172.21.15.16:6810/1398737533","cluster_addr":"172.21.15.16:6813/1398737533","heartbeat_back_addr":"172.21.15.16:6821/1398737533","heartbeat_front_addr":"172.21.15.16:6816/1398737533","state":["exists","up"]},{"osd":2,"uuid":"01a73662-9a21-4b1f-bcbb-42b9636a4638","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6819","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6822","nonce":3018526646}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6825","nonce":3018526646}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6831","nonce":3018526646}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6827","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6829","nonce":3018526646}]},"public_addr":"172.21.15.16:6822/3018526646","cluster_addr":"172.21.15.16:6825/3018526646","heartbeat_back_addr":"172.21.15.16:6831/3018526646","heartbeat_front_addr":"172.21.15.16:6829/3018526646","state":["exists","up"]},{"osd":3,"uuid":"c7771fd3-e18e-4500-ac38-23cf10ebcd75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6809","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6812","nonce":3360352974}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6817","nonce":3360352974}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6828","nonce":3360352974}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6823","nonce":3360352974}]},"public_addr":"172.21.15.16:6812/3360352974","cluster_addr":"172.21.15.16:6817/3360352974","heartbeat_back_addr":"172.21.15.16:6828/3360352974","heartbeat_front_addr":"172.21.15.16:6823/3360352974","state":["exists","up"]},{"osd":4,"uuid":"88260730-1482-4fbd-a40f-cb1124e9df4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6801","nonce":1856844856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6803","nonce":1856844856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6807","nonce":1856844856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6805","nonce":1856844856}]},"public_addr":"172.21.15.89:6801/1856844856","cluster_addr":"172.21.15.89:6803/1856844856","heartbeat_back_addr":"172.21.15.89:6807/1856844856","heartbeat_front_addr":"172.21.15.89:6805/1856844856","state":["exists","up"]},{"osd":5,"uuid":"4cb2c00c-fae0-4dd8-9d27-ec4f0093ccf7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6814","nonce":50395527}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6817","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6820","nonce":50395527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6831","nonce":50395527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6823","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6826","nonce":50395527}]},"public_addr":"172.21.15.89:6814/50395527","cluster_addr":"172.21.15.89:6820/50395527","heartbeat_back_addr":"172.21.15.89:6831/50395527","heartbeat_front_addr":"172.21.15.89:6826/50395527","state":["exists","up"]},{"osd":6,"uuid":"81ce0c19-2554-4cae-acbe-66a05e9e9b9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6809","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6812","nonce":3470471752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6815","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6818","nonce":3470471752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6827","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6829","nonce":3470471752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6821","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6824","nonce":3470471752}]},"public_addr":"172.21.15.89:6812/3470471752","cluster_addr":"172.21.15.89:6818/3470471752","heartbeat_back_addr":"172.21.15.89:6829/3470471752","heartbeat_front_addr":"172.21.15.89:6824/3470471752","state":["exists","up"]},{"osd":7,"uuid":"85e4313b-d37d-442e-8042-153bf20a847d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6811","nonce":1796778630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6813","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6816","nonce":1796778630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6825","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6828","nonce":1796778630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6819","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6822","nonce":1796778630}]},"public_addr":"172.21.15.89:6811/1796778630","cluster_addr":"172.21.15.89:6816/1796778630","heartbeat_back_addr":"172.21.15.89:6828/1796778630","heartbeat_front_addr":"172.21.15.89:6822/1796778630","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.439088+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.529970+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.680512+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:21.044161+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.582111+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.536558+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.409522+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.590229+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-31T15:09:52.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:52.260+0000 7f7835600640 1 -- 172.21.15.16:0/2407950828 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f780c0683b0 msgr2=0x7f780c06a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:52.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:52.260+0000 7f7835600640 1 --2- 172.21.15.16:0/2407950828 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f780c0683b0 0x7f780c06a870 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f7820006fd0 tx=0x7f7820008040 comp rx=0 tx=0).stop 2024-03-31T15:09:52.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:52.260+0000 7f7835600640 1 -- 172.21.15.16:0/2407950828 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f783015eee0 msgr2=0x7f783015a290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:52.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:52.260+0000 7f7835600640 1 --2- 172.21.15.16:0/2407950828 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f783015eee0 0x7f783015a290 secure :-1 s=READY pgs=1078 cs=0 l=1 rev1=1 crypto rx=0x7f782800c7b0 tx=0x7f782800cc70 comp rx=0 tx=0).stop 2024-03-31T15:09:52.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:52.260+0000 7f7835600640 1 -- 172.21.15.16:0/2407950828 shutdown_connections 2024-03-31T15:09:52.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:52.260+0000 7f7835600640 1 --2- 172.21.15.16:0/2407950828 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f780c0683b0 0x7f780c06a870 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:52.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:52.260+0000 7f7835600640 1 --2- 172.21.15.16:0/2407950828 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f783015eee0 0x7f783015a290 unknown :-1 s=CLOSED pgs=1078 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:52.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:52.260+0000 7f7835600640 1 --2- 172.21.15.16:0/2407950828 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f783015b3d0 0x7f7830159d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:52.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:52.260+0000 7f7835600640 1 --2- 172.21.15.16:0/2407950828 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f78300ce2f0 0x7f7830157af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:52.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:52.260+0000 7f7835600640 1 -- 172.21.15.16:0/2407950828 >> 172.21.15.16:0/2407950828 conn(0x7f783006b620 msgr2=0x7f78300cc9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:09:52.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:52.260+0000 7f7835600640 1 -- 172.21.15.16:0/2407950828 shutdown_connections 2024-03-31T15:09:52.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:52.260+0000 7f7835600640 1 -- 172.21.15.16:0/2407950828 wait complete. 2024-03-31T15:09:52.273 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-31T15:09:52.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:52.424+0000 7fc839600640 1 Processor -- start 2024-03-31T15:09:52.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:52.424+0000 7fc839600640 1 -- start start 2024-03-31T15:09:52.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:52.424+0000 7fc839600640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fc8341411b0 0x7fc83406f920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:52.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:52.424+0000 7fc839600640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fc834141590 0x7fc83406fe60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:52.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:52.424+0000 7fc839600640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc834144980 con 0x7fc834143c30 2024-03-31T15:09:52.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:52.424+0000 7fc839600640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7fc83415f910 con 0x7fc834141590 2024-03-31T15:09:52.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:52.424+0000 7fc839600640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fc834162910 con 0x7fc8341411b0 2024-03-31T15:09:52.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:52.424+0000 7fc82be00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fc834141590 0x7fc83406fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:52.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:52.424+0000 7fc832a00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fc8341411b0 0x7fc83406f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:52.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:52.424+0000 7fc82be00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fc834141590 0x7fc83406fe60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.16:39764/0 (socket says 172.21.15.16:39764) 2024-03-31T15:09:52.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:52.424+0000 7fc82be00640 1 -- 172.21.15.16:0/1847703285 learned_addr learned my addr 172.21.15.16:0/1847703285 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:09:52.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:52.424+0000 7fc82be00640 1 -- 172.21.15.16:0/1847703285 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fc8341411b0 msgr2=0x7fc83406f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:09:52.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:52.424+0000 7fc82be00640 1 --2- 172.21.15.16:0/1847703285 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fc8341411b0 0x7fc83406f920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:52.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:52.424+0000 7fc82be00640 1 -- 172.21.15.16:0/1847703285 >> v1:172.21.15.89:6789/0 conn(0x7fc834143c30 legacy=0x7fc8341440a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:52.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:52.424+0000 7fc82be00640 1 -- 172.21.15.16:0/1847703285 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc834161fe0 con 0x7fc834141590 2024-03-31T15:09:52.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:52.424+0000 7fc832a00640 1 --2- 172.21.15.16:0/1847703285 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fc8341411b0 0x7fc83406f920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:09:52.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:52.428+0000 7fc82be00640 1 --2- 172.21.15.16:0/1847703285 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc834141590 0x7fc83406fe60 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7fc81c009870 tx=0x7fc81c01c560 comp rx=0 tx=0).ready entity=mon.1 client_cookie=53245f2f2539ec77 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:09:52.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:52.428+0000 7fc832000640 1 -- 172.21.15.16:0/1847703285 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc81c0ab650 con 0x7fc834141590 2024-03-31T15:09:52.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:52.428+0000 7fc832000640 1 -- 172.21.15.16:0/1847703285 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc81c0b0070 con 0x7fc834141590 2024-03-31T15:09:52.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:52.428+0000 7fc832000640 1 -- 172.21.15.16:0/1847703285 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc81c0aba10 con 0x7fc834141590 2024-03-31T15:09:52.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:52.428+0000 7fc839600640 1 -- 172.21.15.16:0/1847703285 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc834141590 msgr2=0x7fc83406fe60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:52.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:52.428+0000 7fc839600640 1 --2- 172.21.15.16:0/1847703285 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc834141590 0x7fc83406fe60 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7fc81c009870 tx=0x7fc81c01c560 comp rx=0 tx=0).stop 2024-03-31T15:09:52.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:52.428+0000 7fc839600640 1 -- 172.21.15.16:0/1847703285 shutdown_connections 2024-03-31T15:09:52.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:52.428+0000 7fc839600640 1 --2- 172.21.15.16:0/1847703285 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc834141590 0x7fc83406fe60 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:52.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:52.428+0000 7fc839600640 1 --2- 172.21.15.16:0/1847703285 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fc8341411b0 0x7fc83406f920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:52.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:52.428+0000 7fc839600640 1 -- 172.21.15.16:0/1847703285 >> 172.21.15.16:0/1847703285 conn(0x7fc834067930 msgr2=0x7fc8340b3ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:09:52.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:52.428+0000 7fc839600640 1 -- 172.21.15.16:0/1847703285 shutdown_connections 2024-03-31T15:09:52.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:52.428+0000 7fc839600640 1 -- 172.21.15.16:0/1847703285 wait complete. 2024-03-31T15:09:52.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:52.432+0000 7fc839600640 1 Processor -- start 2024-03-31T15:09:52.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:52.432+0000 7fc839600640 1 -- start start 2024-03-31T15:09:52.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:52.432+0000 7fc839600640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fc8341411b0 0x7fc8340b2050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:52.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:52.432+0000 7fc839600640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc834141590 0x7fc834154710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:52.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:52.432+0000 7fc839600640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc834143c30 0x7fc83406d4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:52.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:52.432+0000 7fc839600640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fc83415f290 con 0x7fc834143c30 2024-03-31T15:09:52.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:52.432+0000 7fc839600640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fc83415f110 con 0x7fc834141590 2024-03-31T15:09:52.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:52.432+0000 7fc839600640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fc83415f410 con 0x7fc8341411b0 2024-03-31T15:09:52.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:52.432+0000 7fc833400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc834143c30 0x7fc83406d4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:52.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:52.432+0000 7fc832a00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fc8341411b0 0x7fc8340b2050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:52.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:52.432+0000 7fc833400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc834143c30 0x7fc83406d4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59524/0 (socket says 172.21.15.16:59524) 2024-03-31T15:09:52.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:52.432+0000 7fc833400640 1 -- 172.21.15.16:0/3686006089 learned_addr learned my addr 172.21.15.16:0/3686006089 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:09:52.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:52.432+0000 7fc82be00640 1 --2- 172.21.15.16:0/3686006089 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc834141590 0x7fc834154710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:52.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:52.432+0000 7fc833400640 1 -- 172.21.15.16:0/3686006089 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fc8341411b0 msgr2=0x7fc8340b2050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:52.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:52.432+0000 7fc833400640 1 --2- 172.21.15.16:0/3686006089 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fc8341411b0 0x7fc8340b2050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:52.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:52.432+0000 7fc833400640 1 -- 172.21.15.16:0/3686006089 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc834141590 msgr2=0x7fc834154710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:52.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:52.432+0000 7fc833400640 1 --2- 172.21.15.16:0/3686006089 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc834141590 0x7fc834154710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:52.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:52.432+0000 7fc833400640 1 -- 172.21.15.16:0/3686006089 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8340b1da0 con 0x7fc834143c30 2024-03-31T15:09:52.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:52.432+0000 7fc833400640 1 --2- 172.21.15.16:0/3686006089 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc834143c30 0x7fc83406d4f0 secure :-1 s=READY pgs=1079 cs=0 l=1 rev1=1 crypto rx=0x7fc82c00c760 tx=0x7fc82c00cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:09:52.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:52.432+0000 7fc82aa00640 1 -- 172.21.15.16:0/3686006089 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc82c003b80 con 0x7fc834143c30 2024-03-31T15:09:52.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:52.432+0000 7fc82aa00640 1 -- 172.21.15.16:0/3686006089 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc82c003d20 con 0x7fc834143c30 2024-03-31T15:09:52.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:52.432+0000 7fc82aa00640 1 -- 172.21.15.16:0/3686006089 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc82c00aa30 con 0x7fc834143c30 2024-03-31T15:09:52.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:52.432+0000 7fc839600640 1 -- 172.21.15.16:0/3686006089 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc8340b2630 con 0x7fc834143c30 2024-03-31T15:09:52.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:52.432+0000 7fc839600640 1 -- 172.21.15.16:0/3686006089 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc8340b2bf0 con 0x7fc834143c30 2024-03-31T15:09:52.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:52.436+0000 7fc839600640 1 -- 172.21.15.16:0/3686006089 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc834069950 con 0x7fc834143c30 2024-03-31T15:09:52.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:52.436+0000 7fc82aa00640 1 -- 172.21.15.16:0/3686006089 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fc82c00abd0 con 0x7fc834143c30 2024-03-31T15:09:52.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:52.436+0000 7fc82aa00640 1 --2- 172.21.15.16:0/3686006089 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fc8100683b0 0x7fc81006a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:52.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:52.436+0000 7fc82aa00640 1 -- 172.21.15.16:0/3686006089 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 5709+0+0 (secure 0 0 0) 0x7fc82c086750 con 0x7fc834143c30 2024-03-31T15:09:52.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:52.436+0000 7fc832a00640 1 --2- 172.21.15.16:0/3686006089 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fc8100683b0 0x7fc81006a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:52.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:52.436+0000 7fc832a00640 1 --2- 172.21.15.16:0/3686006089 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fc8100683b0 0x7fc81006a870 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7fc824009580 tx=0x7fc824009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:09:52.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:52.440+0000 7fc82aa00640 1 -- 172.21.15.16:0/3686006089 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fc82c056620 con 0x7fc834143c30 2024-03-31T15:09:52.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:52.680+0000 7fc839600640 1 -- 172.21.15.16:0/3686006089 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7fc83415e490 con 0x7fc834143c30 2024-03-31T15:09:53.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:53.544+0000 7fc82aa00640 1 -- 172.21.15.16:0/3686006089 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v60) ==== 118+0+0 (secure 0 0 0) 0x7fc82c05a920 con 0x7fc834143c30 2024-03-31T15:09:53.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:53.544+0000 7fc839600640 1 -- 172.21.15.16:0/3686006089 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fc8100683b0 msgr2=0x7fc81006a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:53.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:53.544+0000 7fc839600640 1 --2- 172.21.15.16:0/3686006089 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fc8100683b0 0x7fc81006a870 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7fc824009580 tx=0x7fc824009190 comp rx=0 tx=0).stop 2024-03-31T15:09:53.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:53.544+0000 7fc839600640 1 -- 172.21.15.16:0/3686006089 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc834143c30 msgr2=0x7fc83406d4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:53.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:53.544+0000 7fc839600640 1 --2- 172.21.15.16:0/3686006089 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc834143c30 0x7fc83406d4f0 secure :-1 s=READY pgs=1079 cs=0 l=1 rev1=1 crypto rx=0x7fc82c00c760 tx=0x7fc82c00cc20 comp rx=0 tx=0).stop 2024-03-31T15:09:53.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:53.544+0000 7fc839600640 1 -- 172.21.15.16:0/3686006089 shutdown_connections 2024-03-31T15:09:53.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:53.544+0000 7fc839600640 1 --2- 172.21.15.16:0/3686006089 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fc8100683b0 0x7fc81006a870 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:53.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:53.544+0000 7fc839600640 1 --2- 172.21.15.16:0/3686006089 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc834143c30 0x7fc83406d4f0 unknown :-1 s=CLOSED pgs=1079 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:53.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:53.544+0000 7fc839600640 1 --2- 172.21.15.16:0/3686006089 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc834141590 0x7fc834154710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:53.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:53.544+0000 7fc839600640 1 --2- 172.21.15.16:0/3686006089 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fc8341411b0 0x7fc8340b2050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:53.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:53.544+0000 7fc839600640 1 -- 172.21.15.16:0/3686006089 >> 172.21.15.16:0/3686006089 conn(0x7fc834067930 msgr2=0x7fc83406c730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:09:53.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:53.544+0000 7fc839600640 1 -- 172.21.15.16:0/3686006089 shutdown_connections 2024-03-31T15:09:53.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:53.544+0000 7fc839600640 1 -- 172.21.15.16:0/3686006089 wait complete. 2024-03-31T15:09:53.557 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-03-31T15:09:53.557 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-31T15:09:53.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:53.708+0000 7f39c7400640 1 Processor -- start 2024-03-31T15:09:53.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:53.708+0000 7f39c7400640 1 -- start start 2024-03-31T15:09:53.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:53.712+0000 7f39c7400640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f39c00d0a60 0x7f39c00cea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:53.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:53.712+0000 7f39c7400640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f39c015ca70 con 0x7f39c006f920 2024-03-31T15:09:53.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:53.712+0000 7f39c7400640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f39c015cd50 con 0x7f39c00cef80 2024-03-31T15:09:53.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:53.712+0000 7f39c7400640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f39c013f670 con 0x7f39c00d0a60 2024-03-31T15:09:53.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:53.712+0000 7f39c4a00640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f39c006f920 0x7f39c006fd00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:48236/0 (socket says 172.21.15.16:48236) 2024-03-31T15:09:53.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:53.712+0000 7f39c4a00640 1 -- 172.21.15.16:0/3121983014 learned_addr learned my addr 172.21.15.16:0/3121983014 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:09:53.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:53.712+0000 7f39bfe00640 1 --2- 172.21.15.16:0/3121983014 >> v2:172.21.15.16:3300/0 conn(0x7f39c00d0a60 0x7f39c00cea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:53.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:53.712+0000 7f39bfe00640 1 -- 172.21.15.16:0/3121983014 >> v1:172.21.15.89:6789/0 conn(0x7f39c00cef80 legacy=0x7f39c01419c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:53.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:53.712+0000 7f39bfe00640 1 -- 172.21.15.16:0/3121983014 >> v1:172.21.15.16:6789/0 conn(0x7f39c006f920 legacy=0x7f39c006fd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:53.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:53.712+0000 7f39bfe00640 1 -- 172.21.15.16:0/3121983014 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39c015d030 con 0x7f39c00d0a60 2024-03-31T15:09:53.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:53.712+0000 7f39bf400640 1 -- 172.21.15.16:0/3121983014 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 147240158 0 0) 0x7f39c015ca70 con 0x7f39c006f920 2024-03-31T15:09:53.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:53.716+0000 7f39bfe00640 1 --2- 172.21.15.16:0/3121983014 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f39c00d0a60 0x7f39c00cea40 secure :-1 s=READY pgs=1081 cs=0 l=1 rev1=1 crypto rx=0x7f39ac0097c0 tx=0x7f39ac01c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5242e30901849694 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:09:53.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:53.716+0000 7f39bf400640 1 -- 172.21.15.16:0/3121983014 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39ac0b0070 con 0x7f39c00d0a60 2024-03-31T15:09:53.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:53.716+0000 7f39bf400640 1 -- 172.21.15.16:0/3121983014 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f39ac0ab600 con 0x7f39c00d0a60 2024-03-31T15:09:53.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:53.716+0000 7f39bf400640 1 -- 172.21.15.16:0/3121983014 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39ac0ab9d0 con 0x7f39c00d0a60 2024-03-31T15:09:53.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:53.716+0000 7f39c7400640 1 -- 172.21.15.16:0/3121983014 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f39c00d0a60 msgr2=0x7f39c00cea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:53.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:53.716+0000 7f39c7400640 1 --2- 172.21.15.16:0/3121983014 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f39c00d0a60 0x7f39c00cea40 secure :-1 s=READY pgs=1081 cs=0 l=1 rev1=1 crypto rx=0x7f39ac0097c0 tx=0x7f39ac01c4a0 comp rx=0 tx=0).stop 2024-03-31T15:09:53.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:53.716+0000 7f39c7400640 1 -- 172.21.15.16:0/3121983014 shutdown_connections 2024-03-31T15:09:53.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:53.716+0000 7f39c7400640 1 --2- 172.21.15.16:0/3121983014 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f39c00d0a60 0x7f39c00cea40 unknown :-1 s=CLOSED pgs=1081 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:53.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:53.716+0000 7f39c7400640 1 -- 172.21.15.16:0/3121983014 >> 172.21.15.16:0/3121983014 conn(0x7f39c0067930 msgr2=0x7f39c00b3ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:09:53.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:53.716+0000 7f39c7400640 1 -- 172.21.15.16:0/3121983014 shutdown_connections 2024-03-31T15:09:53.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:53.716+0000 7f39c7400640 1 -- 172.21.15.16:0/3121983014 wait complete. 2024-03-31T15:09:53.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:53.720+0000 7f39c7400640 1 Processor -- start 2024-03-31T15:09:53.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:53.720+0000 7f39c7400640 1 -- start start 2024-03-31T15:09:53.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:53.720+0000 7f39c7400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f39c006f920 0x7f39c015be50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:53.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:53.720+0000 7f39c7400640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f39c00cef80 0x7f39c0150c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:53.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:53.720+0000 7f39c7400640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f39c00d0a60 0x7f39c0167820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:53.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:53.720+0000 7f39c7400640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f39c016a240 con 0x7f39c006f920 2024-03-31T15:09:53.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:53.720+0000 7f39c7400640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f39c016a0c0 con 0x7f39c00cef80 2024-03-31T15:09:53.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:53.720+0000 7f39c7400640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f39c016a3c0 con 0x7f39c00d0a60 2024-03-31T15:09:53.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:53.720+0000 7f39c4a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f39c006f920 0x7f39c015be50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:53.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:53.720+0000 7f39c4a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f39c006f920 0x7f39c015be50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59544/0 (socket says 172.21.15.16:59544) 2024-03-31T15:09:53.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:53.720+0000 7f39c4a00640 1 -- 172.21.15.16:0/1866662338 learned_addr learned my addr 172.21.15.16:0/1866662338 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:09:53.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:53.720+0000 7f39c4a00640 1 -- 172.21.15.16:0/1866662338 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f39c00d0a60 msgr2=0x7f39c0167820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:53.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:53.720+0000 7f39c4a00640 1 --2- 172.21.15.16:0/1866662338 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f39c00d0a60 0x7f39c0167820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:53.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:53.720+0000 7f39c4a00640 1 -- 172.21.15.16:0/1866662338 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f39c00cef80 msgr2=0x7f39c0150c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:53.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:53.720+0000 7f39c4a00640 1 --2- 172.21.15.16:0/1866662338 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f39c00cef80 0x7f39c0150c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:53.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:53.720+0000 7f39c4a00640 1 -- 172.21.15.16:0/1866662338 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39c015ad70 con 0x7f39c006f920 2024-03-31T15:09:53.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:53.720+0000 7f39c4a00640 1 --2- 172.21.15.16:0/1866662338 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f39c006f920 0x7f39c015be50 secure :-1 s=READY pgs=1082 cs=0 l=1 rev1=1 crypto rx=0x7f39b400c4d0 tx=0x7f39b400c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:09:53.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:53.720+0000 7f39bcc00640 1 -- 172.21.15.16:0/1866662338 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39b4018070 con 0x7f39c006f920 2024-03-31T15:09:53.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:53.720+0000 7f39bcc00640 1 -- 172.21.15.16:0/1866662338 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f39b40074e0 con 0x7f39c006f920 2024-03-31T15:09:53.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:53.720+0000 7f39c7400640 1 -- 172.21.15.16:0/1866662338 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f39c015b050 con 0x7f39c006f920 2024-03-31T15:09:53.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:53.720+0000 7f39bcc00640 1 -- 172.21.15.16:0/1866662338 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39b4004a30 con 0x7f39c006f920 2024-03-31T15:09:53.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:53.720+0000 7f39c7400640 1 -- 172.21.15.16:0/1866662338 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f39c006c980 con 0x7f39c006f920 2024-03-31T15:09:53.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:53.724+0000 7f39c7400640 1 -- 172.21.15.16:0/1866662338 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f39c006c790 con 0x7f39c006f920 2024-03-31T15:09:53.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:53.724+0000 7f39bcc00640 1 -- 172.21.15.16:0/1866662338 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f39b401b070 con 0x7f39c006f920 2024-03-31T15:09:53.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:53.724+0000 7f39bcc00640 1 --2- 172.21.15.16:0/1866662338 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f39a0068380 0x7f39a006a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:53.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:53.724+0000 7f39bcc00640 1 -- 172.21.15.16:0/1866662338 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 5709+0+0 (secure 0 0 0) 0x7f39b4085d60 con 0x7f39c006f920 2024-03-31T15:09:53.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:53.724+0000 7f39bfe00640 1 --2- 172.21.15.16:0/1866662338 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f39a0068380 0x7f39a006a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:53.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:53.724+0000 7f39bfe00640 1 --2- 172.21.15.16:0/1866662338 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f39a0068380 0x7f39a006a840 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f39ac0ad000 tx=0x7f39ac008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:09:53.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:53.728+0000 7f39bcc00640 1 -- 172.21.15.16:0/1866662338 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f39b4056280 con 0x7f39c006f920 2024-03-31T15:09:54.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:54.008+0000 7f39c7400640 1 -- 172.21.15.16:0/1866662338 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7f39c0069950 con 0x7f39c006f920 2024-03-31T15:09:54.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:54.548+0000 7f39bcc00640 1 -- 172.21.15.16:0/1866662338 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v61) ==== 122+0+0 (secure 0 0 0) 0x7f39b4055b80 con 0x7f39c006f920 2024-03-31T15:09:54.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:54.552+0000 7f39c7400640 1 -- 172.21.15.16:0/1866662338 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f39a0068380 msgr2=0x7f39a006a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:54.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:54.552+0000 7f39c7400640 1 --2- 172.21.15.16:0/1866662338 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f39a0068380 0x7f39a006a840 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f39ac0ad000 tx=0x7f39ac008040 comp rx=0 tx=0).stop 2024-03-31T15:09:54.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:54.552+0000 7f39c7400640 1 -- 172.21.15.16:0/1866662338 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f39c006f920 msgr2=0x7f39c015be50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:54.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:54.552+0000 7f39c7400640 1 --2- 172.21.15.16:0/1866662338 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f39c006f920 0x7f39c015be50 secure :-1 s=READY pgs=1082 cs=0 l=1 rev1=1 crypto rx=0x7f39b400c4d0 tx=0x7f39b400c990 comp rx=0 tx=0).stop 2024-03-31T15:09:54.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:54.552+0000 7f39c7400640 1 -- 172.21.15.16:0/1866662338 shutdown_connections 2024-03-31T15:09:54.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:54.552+0000 7f39c7400640 1 --2- 172.21.15.16:0/1866662338 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f39a0068380 0x7f39a006a840 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:54.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:54.552+0000 7f39c7400640 1 --2- 172.21.15.16:0/1866662338 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f39c00d0a60 0x7f39c0167820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:54.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:54.552+0000 7f39c7400640 1 --2- 172.21.15.16:0/1866662338 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f39c00cef80 0x7f39c0150c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:54.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:54.552+0000 7f39c7400640 1 --2- 172.21.15.16:0/1866662338 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f39c006f920 0x7f39c015be50 unknown :-1 s=CLOSED pgs=1082 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:54.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:54.552+0000 7f39c7400640 1 -- 172.21.15.16:0/1866662338 >> 172.21.15.16:0/1866662338 conn(0x7f39c0067930 msgr2=0x7f39c01552b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:09:54.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:54.552+0000 7f39c7400640 1 -- 172.21.15.16:0/1866662338 shutdown_connections 2024-03-31T15:09:54.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:54.552+0000 7f39c7400640 1 -- 172.21.15.16:0/1866662338 wait complete. 2024-03-31T15:09:54.564 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-31T15:09:54.564 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-31T15:09:54.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:54.712+0000 7f22ab600640 1 Processor -- start 2024-03-31T15:09:54.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:54.712+0000 7f22ab600640 1 -- start start 2024-03-31T15:09:54.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:54.712+0000 7f22ab600640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f22a40b23d0 0x7f22a415cd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:54.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:54.712+0000 7f22ab600640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f22a4160e70 con 0x7f22a4145d10 2024-03-31T15:09:54.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:54.712+0000 7f22ab600640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f22a41602f0 con 0x7f22a40728b0 2024-03-31T15:09:54.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:54.712+0000 7f22ab600640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f22a41605d0 con 0x7f22a40b23d0 2024-03-31T15:09:54.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:54.712+0000 7f22a8c00640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f22a4145d10 0x7f22a40cf790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:48248/0 (socket says 172.21.15.16:48248) 2024-03-31T15:09:54.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:54.712+0000 7f22a8c00640 1 -- 172.21.15.16:0/2335368442 learned_addr learned my addr 172.21.15.16:0/2335368442 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:09:54.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:54.712+0000 7f22a9600640 1 --2- 172.21.15.16:0/2335368442 >> v2:172.21.15.89:3300/0 conn(0x7f22a40b23d0 0x7f22a415cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:54.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:54.712+0000 7f22a3e00640 1 -- 172.21.15.16:0/2335368442 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2662322766 0 0) 0x7f22a4160e70 con 0x7f22a4145d10 2024-03-31T15:09:54.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:54.712+0000 7f22a3e00640 1 -- 172.21.15.16:0/2335368442 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2288003530 con 0x7f22a4145d10 2024-03-31T15:09:54.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:54.716+0000 7f22a9600640 1 -- 172.21.15.16:0/2335368442 >> v1:172.21.15.89:6789/0 conn(0x7f22a40728b0 legacy=0x7f22a40b1cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:54.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:54.716+0000 7f22a9600640 1 -- 172.21.15.16:0/2335368442 >> v1:172.21.15.16:6789/0 conn(0x7f22a4145d10 legacy=0x7f22a40cf790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:54.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:54.716+0000 7f22a3e00640 1 -- 172.21.15.16:0/2335368442 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1124574412 0 0) 0x7f2288003530 con 0x7f22a4145d10 2024-03-31T15:09:54.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:54.716+0000 7f22a9600640 1 -- 172.21.15.16:0/2335368442 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22a41608b0 con 0x7f22a40b23d0 2024-03-31T15:09:54.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:54.716+0000 7f22a9600640 1 --2- 172.21.15.16:0/2335368442 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f22a40b23d0 0x7f22a415cd50 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f229c009d90 tx=0x7f229c01c890 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6ab58344baa5b500 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:09:54.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:54.716+0000 7f22a3e00640 1 -- 172.21.15.16:0/2335368442 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2294001680 con 0x7f22a40b23d0 2024-03-31T15:09:54.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:54.716+0000 7f22a3e00640 1 -- 172.21.15.16:0/2335368442 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f229c0a64e0 con 0x7f22a40b23d0 2024-03-31T15:09:54.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:54.716+0000 7f22a3e00640 1 -- 172.21.15.16:0/2335368442 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f229c0a68b0 con 0x7f22a40b23d0 2024-03-31T15:09:54.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:54.720+0000 7f22ab600640 1 -- 172.21.15.16:0/2335368442 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f22a40b23d0 msgr2=0x7f22a415cd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:54.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:54.720+0000 7f22ab600640 1 --2- 172.21.15.16:0/2335368442 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f22a40b23d0 0x7f22a415cd50 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f229c009d90 tx=0x7f229c01c890 comp rx=0 tx=0).stop 2024-03-31T15:09:54.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:54.720+0000 7f22ab600640 1 -- 172.21.15.16:0/2335368442 shutdown_connections 2024-03-31T15:09:54.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:54.720+0000 7f22ab600640 1 --2- 172.21.15.16:0/2335368442 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f22a40b23d0 0x7f22a415cd50 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:54.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:54.720+0000 7f22ab600640 1 -- 172.21.15.16:0/2335368442 >> 172.21.15.16:0/2335368442 conn(0x7f22a406b620 msgr2=0x7f22a406ba30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:09:54.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:54.720+0000 7f22ab600640 1 -- 172.21.15.16:0/2335368442 shutdown_connections 2024-03-31T15:09:54.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:54.720+0000 7f22ab600640 1 -- 172.21.15.16:0/2335368442 wait complete. 2024-03-31T15:09:54.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:54.720+0000 7f22ab600640 1 Processor -- start 2024-03-31T15:09:54.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:54.720+0000 7f22ab600640 1 -- start start 2024-03-31T15:09:54.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:54.724+0000 7f22ab600640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f22a40728b0 0x7f22a40b0a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:54.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:54.724+0000 7f22ab600640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f22a40b23d0 0x7f22a415b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:54.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:54.724+0000 7f22ab600640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f22a4145d10 0x7f22a40b11d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:54.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:54.724+0000 7f22ab600640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f22a4072e40 con 0x7f22a40728b0 2024-03-31T15:09:54.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:54.724+0000 7f22ab600640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f22a4072cc0 con 0x7f22a40b23d0 2024-03-31T15:09:54.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:54.724+0000 7f22ab600640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f22a416a100 con 0x7f22a4145d10 2024-03-31T15:09:54.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:54.724+0000 7f22a8c00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f22a40728b0 0x7f22a40b0a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:54.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:54.724+0000 7f22a8c00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f22a40728b0 0x7f22a40b0a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59560/0 (socket says 172.21.15.16:59560) 2024-03-31T15:09:54.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:54.724+0000 7f22a8c00640 1 -- 172.21.15.16:0/1557270148 learned_addr learned my addr 172.21.15.16:0/1557270148 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:09:54.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:54.724+0000 7f22a8c00640 1 -- 172.21.15.16:0/1557270148 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f22a4145d10 msgr2=0x7f22a40b11d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:54.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:54.724+0000 7f22a9600640 1 --2- 172.21.15.16:0/1557270148 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f22a4145d10 0x7f22a40b11d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:54.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:54.724+0000 7f229be00640 1 --2- 172.21.15.16:0/1557270148 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f22a40b23d0 0x7f22a415b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:54.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:54.724+0000 7f22a8c00640 1 --2- 172.21.15.16:0/1557270148 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f22a4145d10 0x7f22a40b11d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:54.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:54.724+0000 7f22a8c00640 1 -- 172.21.15.16:0/1557270148 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f22a40b23d0 msgr2=0x7f22a415b750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:54.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:54.724+0000 7f22a8c00640 1 --2- 172.21.15.16:0/1557270148 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f22a40b23d0 0x7f22a415b750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:54.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:54.724+0000 7f22a8c00640 1 -- 172.21.15.16:0/1557270148 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22a40b1710 con 0x7f22a40728b0 2024-03-31T15:09:54.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:54.724+0000 7f22a9600640 1 --2- 172.21.15.16:0/1557270148 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f22a4145d10 0x7f22a40b11d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:09:54.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:54.724+0000 7f229be00640 1 --2- 172.21.15.16:0/1557270148 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f22a40b23d0 0x7f22a415b750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:09:54.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:54.724+0000 7f22a8c00640 1 --2- 172.21.15.16:0/1557270148 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f22a40728b0 0x7f22a40b0a90 secure :-1 s=READY pgs=1084 cs=0 l=1 rev1=1 crypto rx=0x7f229400c520 tx=0x7f229400c9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:09:54.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:54.724+0000 7f22a1600640 1 -- 172.21.15.16:0/1557270148 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2294017070 con 0x7f22a40728b0 2024-03-31T15:09:54.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:54.724+0000 7f22a1600640 1 -- 172.21.15.16:0/1557270148 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2294005d10 con 0x7f22a40728b0 2024-03-31T15:09:54.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:54.724+0000 7f22a1600640 1 -- 172.21.15.16:0/1557270148 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22940072b0 con 0x7f22a40728b0 2024-03-31T15:09:54.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:54.724+0000 7f22ab600640 1 -- 172.21.15.16:0/1557270148 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f22a40b1990 con 0x7f22a40728b0 2024-03-31T15:09:54.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:54.724+0000 7f22ab600640 1 -- 172.21.15.16:0/1557270148 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f22a40b0290 con 0x7f22a40728b0 2024-03-31T15:09:54.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:54.724+0000 7f22ab600640 1 -- 172.21.15.16:0/1557270148 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f22a4069950 con 0x7f22a40728b0 2024-03-31T15:09:54.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:54.728+0000 7f22a1600640 1 -- 172.21.15.16:0/1557270148 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f229401a070 con 0x7f22a40728b0 2024-03-31T15:09:54.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:54.728+0000 7f22a1600640 1 --2- 172.21.15.16:0/1557270148 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f2280068380 0x7f228006a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:54.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:54.728+0000 7f22a1600640 1 -- 172.21.15.16:0/1557270148 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 5709+0+0 (secure 0 0 0) 0x7f2294085e90 con 0x7f22a40728b0 2024-03-31T15:09:54.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:54.728+0000 7f229be00640 1 --2- 172.21.15.16:0/1557270148 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f2280068380 0x7f228006a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:54.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:54.728+0000 7f229be00640 1 --2- 172.21.15.16:0/1557270148 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f2280068380 0x7f228006a840 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f22a40cd0f0 tx=0x7f228c009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:09:54.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:54.732+0000 7f22a1600640 1 -- 172.21.15.16:0/1557270148 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f22a40b1710 con 0x7f22a40728b0 2024-03-31T15:09:54.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:54.928+0000 7f22ab600640 1 -- 172.21.15.16:0/1557270148 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f22a40cd9c0 con 0x7f22a40728b0 2024-03-31T15:09:54.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:54.932+0000 7f22a1600640 1 -- 172.21.15.16:0/1557270148 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 61 v61) ==== 94+0+5183 (secure 0 0 0) 0x7f2294056440 con 0x7f22a40728b0 2024-03-31T15:09:54.923 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:09:54.923 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":61,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6494,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/2445558494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6835","nonce":2445558494},{"type":"v1","addr":"172.21.15.89:6837","nonce":2445558494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":56},{"gid":7270,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/3185913858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6833","nonce":3185913858},{"type":"v1","addr":"172.21.15.16:6835","nonce":3185913858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":56},{"gid":7279,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6836/1599468493","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":1599468493},{"type":"v1","addr":"172.21.15.89:6836","nonce":1599468493}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":56}],"filesystems":[{"mdsmap":{"epoch":61,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-31T15:09:50.926196+0000","modified":"2024-03-31T15:09:54.012516+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7479},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7479":{"gid":7479,"name":"c","rank":0,"incarnation":57,"state":"up:active","state_seq":3,"addr":"172.21.15.16:6834/2478530200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":2478530200},{"type":"v1","addr":"172.21.15.16:6834","nonce":2478530200}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7479,"qdb_cluster":[7479]},"id":6}]} 2024-03-31T15:09:54.924 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 61 2024-03-31T15:09:54.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:54.932+0000 7f22ab600640 1 -- 172.21.15.16:0/1557270148 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f2280068380 msgr2=0x7f228006a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:54.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:54.932+0000 7f22ab600640 1 --2- 172.21.15.16:0/1557270148 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f2280068380 0x7f228006a840 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f22a40cd0f0 tx=0x7f228c009040 comp rx=0 tx=0).stop 2024-03-31T15:09:54.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:54.932+0000 7f22ab600640 1 -- 172.21.15.16:0/1557270148 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f22a40728b0 msgr2=0x7f22a40b0a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:54.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:54.932+0000 7f22ab600640 1 --2- 172.21.15.16:0/1557270148 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f22a40728b0 0x7f22a40b0a90 secure :-1 s=READY pgs=1084 cs=0 l=1 rev1=1 crypto rx=0x7f229400c520 tx=0x7f229400c9e0 comp rx=0 tx=0).stop 2024-03-31T15:09:54.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:54.932+0000 7f22ab600640 1 -- 172.21.15.16:0/1557270148 shutdown_connections 2024-03-31T15:09:54.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:54.932+0000 7f22ab600640 1 --2- 172.21.15.16:0/1557270148 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f2280068380 0x7f228006a840 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:54.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:54.932+0000 7f22ab600640 1 --2- 172.21.15.16:0/1557270148 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f22a4145d10 0x7f22a40b11d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:54.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:54.932+0000 7f22ab600640 1 --2- 172.21.15.16:0/1557270148 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f22a40b23d0 0x7f22a415b750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:54.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:54.932+0000 7f22ab600640 1 --2- 172.21.15.16:0/1557270148 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f22a40728b0 0x7f22a40b0a90 unknown :-1 s=CLOSED pgs=1084 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:54.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:54.932+0000 7f22ab600640 1 -- 172.21.15.16:0/1557270148 >> 172.21.15.16:0/1557270148 conn(0x7f22a406b620 msgr2=0x7f22a40cff50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:09:54.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:54.932+0000 7f22ab600640 1 -- 172.21.15.16:0/1557270148 shutdown_connections 2024-03-31T15:09:54.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:54.932+0000 7f22ab600640 1 -- 172.21.15.16:0/1557270148 wait complete. 2024-03-31T15:09:54.944 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-31T15:09:55.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:55.092+0000 7ff9c8c00640 1 Processor -- start 2024-03-31T15:09:55.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:55.092+0000 7ff9c8c00640 1 -- start start 2024-03-31T15:09:55.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:55.092+0000 7ff9c8c00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff9c4145aa0 0x7ff9c415c8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:55.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:55.092+0000 7ff9c8c00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff9c406c1c0 0x7ff9c4072e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:55.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:55.092+0000 7ff9c8c00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7ff9c406b620 0x7ff9c406bab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:55.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:55.092+0000 7ff9c8c00640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7ff9c406fd50 con 0x7ff9c4145aa0 2024-03-31T15:09:55.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:55.092+0000 7ff9c8c00640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7ff9c40d09f0 con 0x7ff9c406b620 2024-03-31T15:09:55.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:55.092+0000 7ff9c8c00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7ff9c40d0cd0 con 0x7ff9c406c1c0 2024-03-31T15:09:55.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:55.092+0000 7ff9c1e00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff9c4145aa0 0x7ff9c415c8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:55.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:55.092+0000 7ff9c1e00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff9c4145aa0 0x7ff9c415c8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59572/0 (socket says 172.21.15.16:59572) 2024-03-31T15:09:55.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:55.092+0000 7ff9c1e00640 1 -- 172.21.15.16:0/609821762 learned_addr learned my addr 172.21.15.16:0/609821762 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:09:55.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:55.092+0000 7ff9c2800640 1 --2- 172.21.15.16:0/609821762 >> v2:172.21.15.89:3300/0 conn(0x7ff9c406b620 0x7ff9c406bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:55.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:55.092+0000 7ff9b9400640 1 --2- 172.21.15.16:0/609821762 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff9c406c1c0 0x7ff9c4072e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:55.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:55.092+0000 7ff9c1e00640 1 -- 172.21.15.16:0/609821762 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff9c406c1c0 msgr2=0x7ff9c4072e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:09:55.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:55.092+0000 7ff9c1e00640 1 --2- 172.21.15.16:0/609821762 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff9c406c1c0 0x7ff9c4072e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:55.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:55.092+0000 7ff9c1e00640 1 -- 172.21.15.16:0/609821762 >> v2:172.21.15.89:3300/0 conn(0x7ff9c406b620 msgr2=0x7ff9c406bab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:09:55.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:55.092+0000 7ff9c1e00640 1 --2- 172.21.15.16:0/609821762 >> v2:172.21.15.89:3300/0 conn(0x7ff9c406b620 0x7ff9c406bab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:55.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:55.092+0000 7ff9c1e00640 1 -- 172.21.15.16:0/609821762 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9c40d06e0 con 0x7ff9c4145aa0 2024-03-31T15:09:55.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:55.092+0000 7ff9b9400640 1 --2- 172.21.15.16:0/609821762 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff9c406c1c0 0x7ff9c4072e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:09:55.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:55.096+0000 7ff9c1e00640 1 --2- 172.21.15.16:0/609821762 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff9c4145aa0 0x7ff9c415c8e0 secure :-1 s=READY pgs=1085 cs=0 l=1 rev1=1 crypto rx=0x7ff9c40cec70 tx=0x7ff9b401c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c19e891997ff04f7 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:09:55.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:55.096+0000 7ff9c1400640 1 -- 172.21.15.16:0/609821762 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9b40b0070 con 0x7ff9c4145aa0 2024-03-31T15:09:55.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:55.096+0000 7ff9c1400640 1 -- 172.21.15.16:0/609821762 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff9b40ab600 con 0x7ff9c4145aa0 2024-03-31T15:09:55.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:55.096+0000 7ff9c1400640 1 -- 172.21.15.16:0/609821762 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9b40ab9d0 con 0x7ff9c4145aa0 2024-03-31T15:09:55.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:55.096+0000 7ff9c8c00640 1 -- 172.21.15.16:0/609821762 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff9c4145aa0 msgr2=0x7ff9c415c8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:55.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:55.096+0000 7ff9c8c00640 1 --2- 172.21.15.16:0/609821762 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff9c4145aa0 0x7ff9c415c8e0 secure :-1 s=READY pgs=1085 cs=0 l=1 rev1=1 crypto rx=0x7ff9c40cec70 tx=0x7ff9b401c520 comp rx=0 tx=0).stop 2024-03-31T15:09:55.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:55.096+0000 7ff9c8c00640 1 -- 172.21.15.16:0/609821762 shutdown_connections 2024-03-31T15:09:55.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:55.096+0000 7ff9c8c00640 1 --2- 172.21.15.16:0/609821762 >> v2:172.21.15.89:3300/0 conn(0x7ff9c406b620 0x7ff9c406bab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:55.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:55.096+0000 7ff9c8c00640 1 --2- 172.21.15.16:0/609821762 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff9c406c1c0 0x7ff9c4072e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:55.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:55.096+0000 7ff9c8c00640 1 --2- 172.21.15.16:0/609821762 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff9c4145aa0 0x7ff9c415c8e0 unknown :-1 s=CLOSED pgs=1085 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:55.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:55.096+0000 7ff9c8c00640 1 -- 172.21.15.16:0/609821762 >> 172.21.15.16:0/609821762 conn(0x7ff9c40d1060 msgr2=0x7ff9c40d1470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:09:55.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:55.096+0000 7ff9c8c00640 1 -- 172.21.15.16:0/609821762 shutdown_connections 2024-03-31T15:09:55.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:55.096+0000 7ff9c8c00640 1 -- 172.21.15.16:0/609821762 wait complete. 2024-03-31T15:09:55.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:55.100+0000 7ff9c8c00640 1 Processor -- start 2024-03-31T15:09:55.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:55.100+0000 7ff9c8c00640 1 -- start start 2024-03-31T15:09:55.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:55.100+0000 7ff9c8c00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff9c406b620 0x7ff9c40c6a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:55.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:55.100+0000 7ff9c8c00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff9c406c1c0 0x7ff9c4143990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:55.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:55.100+0000 7ff9c8c00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff9c4145aa0 0x7ff9c415b670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:55.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:55.100+0000 7ff9c8c00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7ff9c4070d80 con 0x7ff9c406c1c0 2024-03-31T15:09:55.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:55.100+0000 7ff9c8c00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7ff9c4070c00 con 0x7ff9c4145aa0 2024-03-31T15:09:55.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:55.100+0000 7ff9c8c00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7ff9c4070f00 con 0x7ff9c406b620 2024-03-31T15:09:55.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:55.100+0000 7ff9b9400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff9c406c1c0 0x7ff9c4143990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:55.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:55.100+0000 7ff9b9400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff9c406c1c0 0x7ff9c4143990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59584/0 (socket says 172.21.15.16:59584) 2024-03-31T15:09:55.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:55.100+0000 7ff9b9400640 1 -- 172.21.15.16:0/3009697587 learned_addr learned my addr 172.21.15.16:0/3009697587 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:09:55.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:55.100+0000 7ff9c1e00640 1 --2- 172.21.15.16:0/3009697587 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff9c406b620 0x7ff9c40c6a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:55.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:55.100+0000 7ff9c2800640 1 --2- 172.21.15.16:0/3009697587 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff9c4145aa0 0x7ff9c415b670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:55.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:55.100+0000 7ff9b9400640 1 -- 172.21.15.16:0/3009697587 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff9c406b620 msgr2=0x7ff9c40c6a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:55.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:55.100+0000 7ff9b9400640 1 --2- 172.21.15.16:0/3009697587 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff9c406b620 0x7ff9c40c6a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:55.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:55.100+0000 7ff9b9400640 1 -- 172.21.15.16:0/3009697587 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff9c4145aa0 msgr2=0x7ff9c415b670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:55.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:55.100+0000 7ff9b9400640 1 --2- 172.21.15.16:0/3009697587 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff9c4145aa0 0x7ff9c415b670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:55.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:55.100+0000 7ff9b9400640 1 -- 172.21.15.16:0/3009697587 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9c40c7080 con 0x7ff9c406c1c0 2024-03-31T15:09:55.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:55.100+0000 7ff9c1e00640 1 --2- 172.21.15.16:0/3009697587 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff9c406b620 0x7ff9c40c6a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:09:55.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:55.100+0000 7ff9c2800640 1 --2- 172.21.15.16:0/3009697587 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff9c4145aa0 0x7ff9c415b670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:09:55.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:55.100+0000 7ff9b9400640 1 --2- 172.21.15.16:0/3009697587 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff9c406c1c0 0x7ff9c4143990 secure :-1 s=READY pgs=1086 cs=0 l=1 rev1=1 crypto rx=0x7ff9ac00c7f0 tx=0x7ff9ac00ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:09:55.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:55.100+0000 7ff9baa00640 1 -- 172.21.15.16:0/3009697587 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9ac015070 con 0x7ff9c406c1c0 2024-03-31T15:09:55.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:55.100+0000 7ff9baa00640 1 -- 172.21.15.16:0/3009697587 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff9ac004220 con 0x7ff9c406c1c0 2024-03-31T15:09:55.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:55.100+0000 7ff9c8c00640 1 -- 172.21.15.16:0/3009697587 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff9c40c7360 con 0x7ff9c406c1c0 2024-03-31T15:09:55.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:55.100+0000 7ff9baa00640 1 -- 172.21.15.16:0/3009697587 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9ac003620 con 0x7ff9c406c1c0 2024-03-31T15:09:55.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:55.100+0000 7ff9c8c00640 1 -- 172.21.15.16:0/3009697587 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff9c40c7920 con 0x7ff9c406c1c0 2024-03-31T15:09:55.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:55.104+0000 7ff9c8c00640 1 -- 172.21.15.16:0/3009697587 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff9c4069950 con 0x7ff9c406c1c0 2024-03-31T15:09:55.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:55.104+0000 7ff9baa00640 1 -- 172.21.15.16:0/3009697587 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7ff9ac0039d0 con 0x7ff9c406c1c0 2024-03-31T15:09:55.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:55.104+0000 7ff9baa00640 1 --2- 172.21.15.16:0/3009697587 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7ff9a0068380 0x7ff9a006a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:55.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:55.104+0000 7ff9baa00640 1 -- 172.21.15.16:0/3009697587 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 5709+0+0 (secure 0 0 0) 0x7ff9ac0869e0 con 0x7ff9c406c1c0 2024-03-31T15:09:55.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:55.104+0000 7ff9c1e00640 1 --2- 172.21.15.16:0/3009697587 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7ff9a0068380 0x7ff9a006a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:55.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:55.104+0000 7ff9c1e00640 1 --2- 172.21.15.16:0/3009697587 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7ff9a0068380 0x7ff9a006a840 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7ff9b40ad000 tx=0x7ff9b40a6000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:09:55.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:55.108+0000 7ff9baa00640 1 -- 172.21.15.16:0/3009697587 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7ff9ac056f90 con 0x7ff9c406c1c0 2024-03-31T15:09:55.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:55.276+0000 7ff9c8c00640 1 -- 172.21.15.16:0/3009697587 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ff9c41450c0 con 0x7ff9c406c1c0 2024-03-31T15:09:55.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:55.276+0000 7ff9baa00640 1 -- 172.21.15.16:0/3009697587 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) ==== 74+0+16588 (secure 0 0 0) 0x7ff9ac056890 con 0x7ff9c406c1c0 2024-03-31T15:09:55.269 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:09:55.269 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":81,"fsid":"286eb5da-fb47-4550-b290-d9519c13b9a5","created":"2024-03-31T15:03:11.438952+0000","modified":"2024-03-31T15:09:52.535916+0000","last_up_change":"2024-03-31T15:03:22.885535+0000","last_in_change":"2024-03-31T15:03:13.941667+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-31T15:03:22.550787+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-03-31T15:09:49.322620+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-03-31T15:09:49.877778+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"92a66450-ef19-40d8-ac3d-d760b23b0bda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6801","nonce":932712244}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6803","nonce":932712244}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6807","nonce":932712244}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6805","nonce":932712244}]},"public_addr":"172.21.15.16:6801/932712244","cluster_addr":"172.21.15.16:6803/932712244","heartbeat_back_addr":"172.21.15.16:6807/932712244","heartbeat_front_addr":"172.21.15.16:6805/932712244","state":["exists","up"]},{"osd":1,"uuid":"da9d1f2b-1968-4959-b843-faa231988e92","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6810","nonce":1398737533}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6811","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6813","nonce":1398737533}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6821","nonce":1398737533}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6815","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6816","nonce":1398737533}]},"public_addr":"172.21.15.16:6810/1398737533","cluster_addr":"172.21.15.16:6813/1398737533","heartbeat_back_addr":"172.21.15.16:6821/1398737533","heartbeat_front_addr":"172.21.15.16:6816/1398737533","state":["exists","up"]},{"osd":2,"uuid":"01a73662-9a21-4b1f-bcbb-42b9636a4638","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6819","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6822","nonce":3018526646}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6825","nonce":3018526646}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6831","nonce":3018526646}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6827","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6829","nonce":3018526646}]},"public_addr":"172.21.15.16:6822/3018526646","cluster_addr":"172.21.15.16:6825/3018526646","heartbeat_back_addr":"172.21.15.16:6831/3018526646","heartbeat_front_addr":"172.21.15.16:6829/3018526646","state":["exists","up"]},{"osd":3,"uuid":"c7771fd3-e18e-4500-ac38-23cf10ebcd75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6809","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6812","nonce":3360352974}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6817","nonce":3360352974}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6828","nonce":3360352974}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6823","nonce":3360352974}]},"public_addr":"172.21.15.16:6812/3360352974","cluster_addr":"172.21.15.16:6817/3360352974","heartbeat_back_addr":"172.21.15.16:6828/3360352974","heartbeat_front_addr":"172.21.15.16:6823/3360352974","state":["exists","up"]},{"osd":4,"uuid":"88260730-1482-4fbd-a40f-cb1124e9df4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6801","nonce":1856844856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6803","nonce":1856844856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6807","nonce":1856844856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6805","nonce":1856844856}]},"public_addr":"172.21.15.89:6801/1856844856","cluster_addr":"172.21.15.89:6803/1856844856","heartbeat_back_addr":"172.21.15.89:6807/1856844856","heartbeat_front_addr":"172.21.15.89:6805/1856844856","state":["exists","up"]},{"osd":5,"uuid":"4cb2c00c-fae0-4dd8-9d27-ec4f0093ccf7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6814","nonce":50395527}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6817","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6820","nonce":50395527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6831","nonce":50395527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6823","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6826","nonce":50395527}]},"public_addr":"172.21.15.89:6814/50395527","cluster_addr":"172.21.15.89:6820/50395527","heartbeat_back_addr":"172.21.15.89:6831/50395527","heartbeat_front_addr":"172.21.15.89:6826/50395527","state":["exists","up"]},{"osd":6,"uuid":"81ce0c19-2554-4cae-acbe-66a05e9e9b9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6809","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6812","nonce":3470471752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6815","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6818","nonce":3470471752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6827","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6829","nonce":3470471752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6821","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6824","nonce":3470471752}]},"public_addr":"172.21.15.89:6812/3470471752","cluster_addr":"172.21.15.89:6818/3470471752","heartbeat_back_addr":"172.21.15.89:6829/3470471752","heartbeat_front_addr":"172.21.15.89:6824/3470471752","state":["exists","up"]},{"osd":7,"uuid":"85e4313b-d37d-442e-8042-153bf20a847d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6811","nonce":1796778630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6813","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6816","nonce":1796778630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6825","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6828","nonce":1796778630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6819","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6822","nonce":1796778630}]},"public_addr":"172.21.15.89:6811/1796778630","cluster_addr":"172.21.15.89:6816/1796778630","heartbeat_back_addr":"172.21.15.89:6828/1796778630","heartbeat_front_addr":"172.21.15.89:6822/1796778630","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.439088+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.529970+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.680512+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:21.044161+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.582111+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.536558+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.409522+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.590229+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-31T15:09:55.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:55.280+0000 7ff9c8c00640 1 -- 172.21.15.16:0/3009697587 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7ff9a0068380 msgr2=0x7ff9a006a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:55.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:55.280+0000 7ff9c8c00640 1 --2- 172.21.15.16:0/3009697587 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7ff9a0068380 0x7ff9a006a840 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7ff9b40ad000 tx=0x7ff9b40a6000 comp rx=0 tx=0).stop 2024-03-31T15:09:55.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:55.280+0000 7ff9c8c00640 1 -- 172.21.15.16:0/3009697587 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff9c406c1c0 msgr2=0x7ff9c4143990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:55.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:55.280+0000 7ff9c8c00640 1 --2- 172.21.15.16:0/3009697587 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff9c406c1c0 0x7ff9c4143990 secure :-1 s=READY pgs=1086 cs=0 l=1 rev1=1 crypto rx=0x7ff9ac00c7f0 tx=0x7ff9ac00ccb0 comp rx=0 tx=0).stop 2024-03-31T15:09:55.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:55.280+0000 7ff9c8c00640 1 -- 172.21.15.16:0/3009697587 shutdown_connections 2024-03-31T15:09:55.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:55.280+0000 7ff9c8c00640 1 --2- 172.21.15.16:0/3009697587 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7ff9a0068380 0x7ff9a006a840 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:55.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:55.280+0000 7ff9c8c00640 1 --2- 172.21.15.16:0/3009697587 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff9c4145aa0 0x7ff9c415b670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:55.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:55.280+0000 7ff9c8c00640 1 --2- 172.21.15.16:0/3009697587 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff9c406c1c0 0x7ff9c4143990 unknown :-1 s=CLOSED pgs=1086 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:55.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:55.280+0000 7ff9c8c00640 1 --2- 172.21.15.16:0/3009697587 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff9c406b620 0x7ff9c40c6a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:55.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:55.280+0000 7ff9c8c00640 1 -- 172.21.15.16:0/3009697587 >> 172.21.15.16:0/3009697587 conn(0x7ff9c40d1060 msgr2=0x7ff9c40affd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:09:55.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:55.280+0000 7ff9c8c00640 1 -- 172.21.15.16:0/3009697587 shutdown_connections 2024-03-31T15:09:55.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:55.280+0000 7ff9c8c00640 1 -- 172.21.15.16:0/3009697587 wait complete. 2024-03-31T15:09:55.291 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-31T15:09:55.291 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-31T15:09:55.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:55.436+0000 7f5319c00640 1 Processor -- start 2024-03-31T15:09:55.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:55.436+0000 7f5319c00640 1 -- start start 2024-03-31T15:09:55.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:55.440+0000 7f5319c00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f53140cef40 0x7f53140b5b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:55.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:55.440+0000 7f5319c00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f53140b61e0 0x7f53140b42e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:55.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:55.440+0000 7f5319c00640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f53140ce8a0 con 0x7f531415b4d0 2024-03-31T15:09:55.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:55.440+0000 7f5319c00640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f53140709a0 con 0x7f53140b61e0 2024-03-31T15:09:55.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:55.440+0000 7f5319c00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f531406fbe0 con 0x7f53140cef40 2024-03-31T15:09:55.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:55.440+0000 7f530be00640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f531415b4d0 0x7f53140cf320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:48256/0 (socket says 172.21.15.16:48256) 2024-03-31T15:09:55.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:55.440+0000 7f530be00640 1 -- 172.21.15.16:0/147226821 learned_addr learned my addr 172.21.15.16:0/147226821 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:09:55.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:55.440+0000 7f5313400640 1 --2- 172.21.15.16:0/147226821 >> v2:172.21.15.89:3300/0 conn(0x7f53140b61e0 0x7f53140b42e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:55.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:55.440+0000 7f5312a00640 1 --2- 172.21.15.16:0/147226821 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f53140cef40 0x7f53140b5b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:55.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:55.440+0000 7f5312000640 1 -- 172.21.15.16:0/147226821 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 383616919 0 0) 0x7f53140ce8a0 con 0x7f531415b4d0 2024-03-31T15:09:55.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:55.440+0000 7f5312000640 1 -- 172.21.15.16:0/147226821 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f52f8003530 con 0x7f531415b4d0 2024-03-31T15:09:55.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:55.440+0000 7f5313400640 1 -- 172.21.15.16:0/147226821 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f53140cef40 msgr2=0x7f53140b5b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:09:55.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:55.440+0000 7f5313400640 1 --2- 172.21.15.16:0/147226821 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f53140cef40 0x7f53140b5b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:55.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:55.440+0000 7f5313400640 1 -- 172.21.15.16:0/147226821 >> v1:172.21.15.16:6789/0 conn(0x7f531415b4d0 legacy=0x7f53140cf320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:55.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:55.440+0000 7f5312000640 1 -- 172.21.15.16:0/147226821 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1528901333 0 0) 0x7f52f8003530 con 0x7f531415b4d0 2024-03-31T15:09:55.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:55.440+0000 7f5313400640 1 -- 172.21.15.16:0/147226821 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f531406fec0 con 0x7f53140b61e0 2024-03-31T15:09:55.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:55.444+0000 7f5313400640 1 --2- 172.21.15.16:0/147226821 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f53140b61e0 0x7f53140b42e0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f530c009d90 tx=0x7f530c01c860 comp rx=0 tx=0).ready entity=mon.1 client_cookie=48c3cceb3a76ce2a server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:09:55.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:55.444+0000 7f5312000640 1 -- 172.21.15.16:0/147226821 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f530c0b2070 con 0x7f53140b61e0 2024-03-31T15:09:55.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:55.444+0000 7f5312000640 1 -- 172.21.15.16:0/147226821 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f530c0a5660 con 0x7f53140b61e0 2024-03-31T15:09:55.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:55.444+0000 7f5312000640 1 -- 172.21.15.16:0/147226821 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52fc001680 con 0x7f53140b61e0 2024-03-31T15:09:55.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:55.444+0000 7f5319c00640 1 -- 172.21.15.16:0/147226821 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f53140b61e0 msgr2=0x7f53140b42e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:55.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:55.444+0000 7f5319c00640 1 --2- 172.21.15.16:0/147226821 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f53140b61e0 0x7f53140b42e0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f530c009d90 tx=0x7f530c01c860 comp rx=0 tx=0).stop 2024-03-31T15:09:55.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:55.444+0000 7f5319c00640 1 -- 172.21.15.16:0/147226821 shutdown_connections 2024-03-31T15:09:55.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:55.444+0000 7f5319c00640 1 --2- 172.21.15.16:0/147226821 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f53140b61e0 0x7f53140b42e0 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:55.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:55.444+0000 7f5319c00640 1 --2- 172.21.15.16:0/147226821 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f53140cef40 0x7f53140b5b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:55.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:55.444+0000 7f5319c00640 1 -- 172.21.15.16:0/147226821 >> 172.21.15.16:0/147226821 conn(0x7f53140d1060 msgr2=0x7f53140d1470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:09:55.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:55.444+0000 7f5319c00640 1 -- 172.21.15.16:0/147226821 shutdown_connections 2024-03-31T15:09:55.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:55.444+0000 7f5319c00640 1 -- 172.21.15.16:0/147226821 wait complete. 2024-03-31T15:09:55.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:55.448+0000 7f5319c00640 1 Processor -- start 2024-03-31T15:09:55.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:55.448+0000 7f5319c00640 1 -- start start 2024-03-31T15:09:55.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:55.448+0000 7f5319c00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f53140b61e0 0x7f5314150670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:55.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:55.448+0000 7f5319c00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f53140cef40 0x7f531415e960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:55.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:55.448+0000 7f5319c00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f531415b4d0 0x7f53141559c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:55.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:55.448+0000 7f5319c00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f53140b51b0 con 0x7f531415b4d0 2024-03-31T15:09:55.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:55.448+0000 7f5319c00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f53140b5030 con 0x7f53140b61e0 2024-03-31T15:09:55.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:55.448+0000 7f5319c00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f53140b5330 con 0x7f53140cef40 2024-03-31T15:09:55.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:55.448+0000 7f5313400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f531415b4d0 0x7f53141559c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:55.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:55.448+0000 7f5313400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f531415b4d0 0x7f53141559c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59586/0 (socket says 172.21.15.16:59586) 2024-03-31T15:09:55.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:55.448+0000 7f5313400640 1 -- 172.21.15.16:0/3687245712 learned_addr learned my addr 172.21.15.16:0/3687245712 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:09:55.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:55.448+0000 7f5312a00640 1 --2- 172.21.15.16:0/3687245712 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f53140b61e0 0x7f5314150670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:55.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:55.448+0000 7f530be00640 1 --2- 172.21.15.16:0/3687245712 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f53140cef40 0x7f531415e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:55.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:55.448+0000 7f5313400640 1 -- 172.21.15.16:0/3687245712 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f53140cef40 msgr2=0x7f531415e960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:55.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:55.448+0000 7f5313400640 1 --2- 172.21.15.16:0/3687245712 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f53140cef40 0x7f531415e960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:55.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:55.448+0000 7f5313400640 1 -- 172.21.15.16:0/3687245712 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f53140b61e0 msgr2=0x7f5314150670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:55.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:55.448+0000 7f5313400640 1 --2- 172.21.15.16:0/3687245712 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f53140b61e0 0x7f5314150670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:55.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:55.448+0000 7f5313400640 1 -- 172.21.15.16:0/3687245712 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5314160e50 con 0x7f531415b4d0 2024-03-31T15:09:55.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:55.448+0000 7f5312a00640 1 --2- 172.21.15.16:0/3687245712 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f53140b61e0 0x7f5314150670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:09:55.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:55.448+0000 7f5313400640 1 --2- 172.21.15.16:0/3687245712 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f531415b4d0 0x7f53141559c0 secure :-1 s=READY pgs=1088 cs=0 l=1 rev1=1 crypto rx=0x7f530c0b0040 tx=0x7f530c019600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:09:55.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:55.448+0000 7f530aa00640 1 -- 172.21.15.16:0/3687245712 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f530c0b2050 con 0x7f531415b4d0 2024-03-31T15:09:55.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:55.448+0000 7f530aa00640 1 -- 172.21.15.16:0/3687245712 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f530c0be070 con 0x7f531415b4d0 2024-03-31T15:09:55.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:55.448+0000 7f5319c00640 1 -- 172.21.15.16:0/3687245712 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5314161130 con 0x7f531415b4d0 2024-03-31T15:09:55.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:55.448+0000 7f530aa00640 1 -- 172.21.15.16:0/3687245712 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f530c009450 con 0x7f531415b4d0 2024-03-31T15:09:55.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:55.448+0000 7f5319c00640 1 -- 172.21.15.16:0/3687245712 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f53141616f0 con 0x7f531415b4d0 2024-03-31T15:09:55.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:55.448+0000 7f5319c00640 1 -- 172.21.15.16:0/3687245712 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5314069950 con 0x7f531415b4d0 2024-03-31T15:09:55.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:55.452+0000 7f530aa00640 1 -- 172.21.15.16:0/3687245712 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f530c0ada90 con 0x7f531415b4d0 2024-03-31T15:09:55.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:55.452+0000 7f530aa00640 1 --2- 172.21.15.16:0/3687245712 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f52f00683b0 0x7f52f006a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:55.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:55.452+0000 7f530aa00640 1 -- 172.21.15.16:0/3687245712 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 5709+0+0 (secure 0 0 0) 0x7f530c121560 con 0x7f531415b4d0 2024-03-31T15:09:55.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:55.452+0000 7f5312a00640 1 --2- 172.21.15.16:0/3687245712 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f52f00683b0 0x7f52f006a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:55.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:55.452+0000 7f5312a00640 1 --2- 172.21.15.16:0/3687245712 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f52f00683b0 0x7f52f006a870 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f53141f8690 tx=0x7f5304008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:09:55.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:55.460+0000 7f530aa00640 1 -- 172.21.15.16:0/3687245712 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f530c0c7020 con 0x7f531415b4d0 2024-03-31T15:09:55.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:55.624+0000 7f5319c00640 1 -- 172.21.15.16:0/3687245712 --> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f53140cdf70 con 0x7f52f00683b0 2024-03-31T15:09:55.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:55.632+0000 7f530aa00640 1 -- 172.21.15.16:0/3687245712 <== mgr.4100 v2:172.21.15.89:6832/1549407273 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252069 (secure 0 0 0) 0x7f53140cdf70 con 0x7f52f00683b0 2024-03-31T15:09:55.628 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:09:55.630 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-03-31T15:09:55.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:55.636+0000 7f5319c00640 1 -- 172.21.15.16:0/3687245712 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f52f00683b0 msgr2=0x7f52f006a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:55.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:55.636+0000 7f5319c00640 1 --2- 172.21.15.16:0/3687245712 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f52f00683b0 0x7f52f006a870 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f53141f8690 tx=0x7f5304008040 comp rx=0 tx=0).stop 2024-03-31T15:09:55.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:55.636+0000 7f5319c00640 1 -- 172.21.15.16:0/3687245712 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f531415b4d0 msgr2=0x7f53141559c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:55.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:55.636+0000 7f5319c00640 1 --2- 172.21.15.16:0/3687245712 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f531415b4d0 0x7f53141559c0 secure :-1 s=READY pgs=1088 cs=0 l=1 rev1=1 crypto rx=0x7f530c0b0040 tx=0x7f530c019600 comp rx=0 tx=0).stop 2024-03-31T15:09:55.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:55.636+0000 7f5319c00640 1 -- 172.21.15.16:0/3687245712 shutdown_connections 2024-03-31T15:09:55.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:55.636+0000 7f5319c00640 1 --2- 172.21.15.16:0/3687245712 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f52f00683b0 0x7f52f006a870 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:55.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:55.636+0000 7f5319c00640 1 --2- 172.21.15.16:0/3687245712 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f531415b4d0 0x7f53141559c0 unknown :-1 s=CLOSED pgs=1088 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:55.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:55.636+0000 7f5319c00640 1 --2- 172.21.15.16:0/3687245712 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f53140cef40 0x7f531415e960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:55.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:55.636+0000 7f5319c00640 1 --2- 172.21.15.16:0/3687245712 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f53140b61e0 0x7f5314150670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:55.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:55.636+0000 7f5319c00640 1 -- 172.21.15.16:0/3687245712 >> 172.21.15.16:0/3687245712 conn(0x7f53140d1060 msgr2=0x7f5314152e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:09:55.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:55.636+0000 7f5319c00640 1 -- 172.21.15.16:0/3687245712 shutdown_connections 2024-03-31T15:09:55.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:55.636+0000 7f5319c00640 1 -- 172.21.15.16:0/3687245712 wait complete. 2024-03-31T15:09:55.649 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":270,"stamp":"2024-03-31T15:09:54.623425+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":124,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":258360,"kb_used_data":8888,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754716360,"statfs":{"total":773094113280,"available":772829552640,"internally_reserved":0,"allocated":9101312,"data_stored":4523076,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":255315504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":35,"apply_latency_ms":35,"commit_latency_ns":35000000,"apply_latency_ns":35000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.109755"},"pg_stats":[{"pgid":"13.3f","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-31T15:09:52.550311+0000","last_change":"2024-03-31T15:09:51.547305+0000","last_active":"2024-03-31T15:09:52.550311+0000","last_peered":"2024-03-31T15:09:52.550311+0000","last_clean":"2024-03-31T15:09:52.550311+0000","last_became_active":"2024-03-31T15:09:51.546999+0000","last_became_peered":"2024-03-31T15:09:51.546999+0000","last_unstale":"2024-03-31T15:09:52.550311+0000","last_undegraded":"2024-03-31T15:09:52.550311+0000","last_fullsized":"2024-03-31T15:09:52.550311+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_clean_scrub_stamp":"2024-03-31T15:09:50.511936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:57:48.201502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.3e","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-31T15:09:52.544886+0000","last_change":"2024-03-31T15:09:51.555327+0000","last_active":"2024-03-31T15:09:52.544886+0000","last_peered":"2024-03-31T15:09:52.544886+0000","last_clean":"2024-03-31T15:09:52.544886+0000","last_became_active":"2024-03-31T15:09:51.555026+0000","last_became_peered":"2024-03-31T15:09:51.555026+0000","last_unstale":"2024-03-31T15:09:52.544886+0000","last_undegraded":"2024-03-31T15:09:52.544886+0000","last_fullsized":"2024-03-31T15:09:52.544886+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_clean_scrub_stamp":"2024-03-31T15:09:50.511936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:18:24.477207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.3d","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-31T15:09:52.667882+0000","last_change":"2024-03-31T15:09:51.544619+0000","last_active":"2024-03-31T15:09:52.667882+0000","last_peered":"2024-03-31T15:09:52.667882+0000","last_clean":"2024-03-31T15:09:52.667882+0000","last_became_active":"2024-03-31T15:09:51.544330+0000","last_became_peered":"2024-03-31T15:09:51.544330+0000","last_unstale":"2024-03-31T15:09:52.667882+0000","last_undegraded":"2024-03-31T15:09:52.667882+0000","last_fullsized":"2024-03-31T15:09:52.667882+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_clean_scrub_stamp":"2024-03-31T15:09:50.511936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:21:07.775538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.3c","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-31T15:09:52.667487+0000","last_change":"2024-03-31T15:09:51.550635+0000","last_active":"2024-03-31T15:09:52.667487+0000","last_peered":"2024-03-31T15:09:52.667487+0000","last_clean":"2024-03-31T15:09:52.667487+0000","last_became_active":"2024-03-31T15:09:51.549526+0000","last_became_peered":"2024-03-31T15:09:51.549526+0000","last_unstale":"2024-03-31T15:09:52.667487+0000","last_undegraded":"2024-03-31T15:09:52.667487+0000","last_fullsized":"2024-03-31T15:09:52.667487+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_clean_scrub_stamp":"2024-03-31T15:09:50.511936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:48:17.779928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.3b","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-31T15:09:52.549662+0000","last_change":"2024-03-31T15:09:51.550825+0000","last_active":"2024-03-31T15:09:52.549662+0000","last_peered":"2024-03-31T15:09:52.549662+0000","last_clean":"2024-03-31T15:09:52.549662+0000","last_became_active":"2024-03-31T15:09:51.550541+0000","last_became_peered":"2024-03-31T15:09:51.550541+0000","last_unstale":"2024-03-31T15:09:52.549662+0000","last_undegraded":"2024-03-31T15:09:52.549662+0000","last_fullsized":"2024-03-31T15:09:52.549662+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_clean_scrub_stamp":"2024-03-31T15:09:50.511936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T16:32:50.574989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.3a","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-31T15:09:52.667031+0000","last_change":"2024-03-31T15:09:51.542684+0000","last_active":"2024-03-31T15:09:52.667031+0000","last_peered":"2024-03-31T15:09:52.667031+0000","last_clean":"2024-03-31T15:09:52.667031+0000","last_became_active":"2024-03-31T15:09:51.542264+0000","last_became_peered":"2024-03-31T15:09:51.542264+0000","last_unstale":"2024-03-31T15:09:52.667031+0000","last_undegraded":"2024-03-31T15:09:52.667031+0000","last_fullsized":"2024-03-31T15:09:52.667031+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_clean_scrub_stamp":"2024-03-31T15:09:50.511936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T17:38:46.806700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.39","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-31T15:09:52.665919+0000","last_change":"2024-03-31T15:09:51.542033+0000","last_active":"2024-03-31T15:09:52.665919+0000","last_peered":"2024-03-31T15:09:52.665919+0000","last_clean":"2024-03-31T15:09:52.665919+0000","last_became_active":"2024-03-31T15:09:51.541720+0000","last_became_peered":"2024-03-31T15:09:51.541720+0000","last_unstale":"2024-03-31T15:09:52.665919+0000","last_undegraded":"2024-03-31T15:09:52.665919+0000","last_fullsized":"2024-03-31T15:09:52.665919+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_clean_scrub_stamp":"2024-03-31T15:09:50.511936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:37:57.748412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.38","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-31T15:09:52.666820+0000","last_change":"2024-03-31T15:09:51.549875+0000","last_active":"2024-03-31T15:09:52.666820+0000","last_peered":"2024-03-31T15:09:52.666820+0000","last_clean":"2024-03-31T15:09:52.666820+0000","last_became_active":"2024-03-31T15:09:51.549573+0000","last_became_peered":"2024-03-31T15:09:51.549573+0000","last_unstale":"2024-03-31T15:09:52.666820+0000","last_undegraded":"2024-03-31T15:09:52.666820+0000","last_fullsized":"2024-03-31T15:09:52.666820+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_clean_scrub_stamp":"2024-03-31T15:09:50.511936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T19:47:03.799495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.37","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-31T15:09:52.667110+0000","last_change":"2024-03-31T15:09:51.546549+0000","last_active":"2024-03-31T15:09:52.667110+0000","last_peered":"2024-03-31T15:09:52.667110+0000","last_clean":"2024-03-31T15:09:52.667110+0000","last_became_active":"2024-03-31T15:09:51.542234+0000","last_became_peered":"2024-03-31T15:09:51.542234+0000","last_unstale":"2024-03-31T15:09:52.667110+0000","last_undegraded":"2024-03-31T15:09:52.667110+0000","last_fullsized":"2024-03-31T15:09:52.667110+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_clean_scrub_stamp":"2024-03-31T15:09:50.511936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:11:51.786356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.36","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-31T15:09:52.549420+0000","last_change":"2024-03-31T15:09:51.543749+0000","last_active":"2024-03-31T15:09:52.549420+0000","last_peered":"2024-03-31T15:09:52.549420+0000","last_clean":"2024-03-31T15:09:52.549420+0000","last_became_active":"2024-03-31T15:09:51.543112+0000","last_became_peered":"2024-03-31T15:09:51.543112+0000","last_unstale":"2024-03-31T15:09:52.549420+0000","last_undegraded":"2024-03-31T15:09:52.549420+0000","last_fullsized":"2024-03-31T15:09:52.549420+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_clean_scrub_stamp":"2024-03-31T15:09:50.511936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:15:50.454538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.35","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-31T15:09:52.544636+0000","last_change":"2024-03-31T15:09:51.546945+0000","last_active":"2024-03-31T15:09:52.544636+0000","last_peered":"2024-03-31T15:09:52.544636+0000","last_clean":"2024-03-31T15:09:52.544636+0000","last_became_active":"2024-03-31T15:09:51.546661+0000","last_became_peered":"2024-03-31T15:09:51.546661+0000","last_unstale":"2024-03-31T15:09:52.544636+0000","last_undegraded":"2024-03-31T15:09:52.544636+0000","last_fullsized":"2024-03-31T15:09:52.544636+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_clean_scrub_stamp":"2024-03-31T15:09:50.511936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T15:52:19.471216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.34","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-31T15:09:52.549808+0000","last_change":"2024-03-31T15:09:51.537351+0000","last_active":"2024-03-31T15:09:52.549808+0000","last_peered":"2024-03-31T15:09:52.549808+0000","last_clean":"2024-03-31T15:09:52.549808+0000","last_became_active":"2024-03-31T15:09:51.537016+0000","last_became_peered":"2024-03-31T15:09:51.537016+0000","last_unstale":"2024-03-31T15:09:52.549808+0000","last_undegraded":"2024-03-31T15:09:52.549808+0000","last_fullsized":"2024-03-31T15:09:52.549808+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_clean_scrub_stamp":"2024-03-31T15:09:50.511936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:14:18.592996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.33","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-31T15:09:52.667438+0000","last_change":"2024-03-31T15:09:51.544698+0000","last_active":"2024-03-31T15:09:52.667438+0000","last_peered":"2024-03-31T15:09:52.667438+0000","last_clean":"2024-03-31T15:09:52.667438+0000","last_became_active":"2024-03-31T15:09:51.544367+0000","last_became_peered":"2024-03-31T15:09:51.544367+0000","last_unstale":"2024-03-31T15:09:52.667438+0000","last_undegraded":"2024-03-31T15:09:52.667438+0000","last_fullsized":"2024-03-31T15:09:52.667438+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_clean_scrub_stamp":"2024-03-31T15:09:50.511936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T19:05:16.947592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.32","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-31T15:09:52.666368+0000","last_change":"2024-03-31T15:09:51.550608+0000","last_active":"2024-03-31T15:09:52.666368+0000","last_peered":"2024-03-31T15:09:52.666368+0000","last_clean":"2024-03-31T15:09:52.666368+0000","last_became_active":"2024-03-31T15:09:51.550159+0000","last_became_peered":"2024-03-31T15:09:51.550159+0000","last_unstale":"2024-03-31T15:09:52.666368+0000","last_undegraded":"2024-03-31T15:09:52.666368+0000","last_fullsized":"2024-03-31T15:09:52.666368+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_clean_scrub_stamp":"2024-03-31T15:09:50.511936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T16:35:00.010577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.31","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-31T15:09:51.553912+0000","last_change":"2024-03-31T15:09:51.550936+0000","last_active":"2024-03-31T15:09:51.553912+0000","last_peered":"2024-03-31T15:09:51.553912+0000","last_clean":"2024-03-31T15:09:51.553912+0000","last_became_active":"2024-03-31T15:09:51.546318+0000","last_became_peered":"2024-03-31T15:09:51.546318+0000","last_unstale":"2024-03-31T15:09:51.553912+0000","last_undegraded":"2024-03-31T15:09:51.553912+0000","last_fullsized":"2024-03-31T15:09:51.553912+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_clean_scrub_stamp":"2024-03-31T15:09:50.511936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:22:32.112780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.30","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-31T15:09:52.549138+0000","last_change":"2024-03-31T15:09:51.535673+0000","last_active":"2024-03-31T15:09:52.549138+0000","last_peered":"2024-03-31T15:09:52.549138+0000","last_clean":"2024-03-31T15:09:52.549138+0000","last_became_active":"2024-03-31T15:09:51.535406+0000","last_became_peered":"2024-03-31T15:09:51.535406+0000","last_unstale":"2024-03-31T15:09:52.549138+0000","last_undegraded":"2024-03-31T15:09:52.549138+0000","last_fullsized":"2024-03-31T15:09:52.549138+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_clean_scrub_stamp":"2024-03-31T15:09:50.511936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:50:05.710178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.2f","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-31T15:09:52.550670+0000","last_change":"2024-03-31T15:09:51.537689+0000","last_active":"2024-03-31T15:09:52.550670+0000","last_peered":"2024-03-31T15:09:52.550670+0000","last_clean":"2024-03-31T15:09:52.550670+0000","last_became_active":"2024-03-31T15:09:51.537397+0000","last_became_peered":"2024-03-31T15:09:51.537397+0000","last_unstale":"2024-03-31T15:09:52.550670+0000","last_undegraded":"2024-03-31T15:09:52.550670+0000","last_fullsized":"2024-03-31T15:09:52.550670+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_clean_scrub_stamp":"2024-03-31T15:09:50.511936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T16:00:33.458197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.2e","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-31T15:09:52.666543+0000","last_change":"2024-03-31T15:09:51.536072+0000","last_active":"2024-03-31T15:09:52.666543+0000","last_peered":"2024-03-31T15:09:52.666543+0000","last_clean":"2024-03-31T15:09:52.666543+0000","last_became_active":"2024-03-31T15:09:51.535757+0000","last_became_peered":"2024-03-31T15:09:51.535757+0000","last_unstale":"2024-03-31T15:09:52.666543+0000","last_undegraded":"2024-03-31T15:09:52.666543+0000","last_fullsized":"2024-03-31T15:09:52.666543+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_clean_scrub_stamp":"2024-03-31T15:09:50.511936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:36:41.005143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2d","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-31T15:09:52.559175+0000","last_change":"2024-03-31T15:09:51.542520+0000","last_active":"2024-03-31T15:09:52.559175+0000","last_peered":"2024-03-31T15:09:52.559175+0000","last_clean":"2024-03-31T15:09:52.559175+0000","last_became_active":"2024-03-31T15:09:51.542251+0000","last_became_peered":"2024-03-31T15:09:51.542251+0000","last_unstale":"2024-03-31T15:09:52.559175+0000","last_undegraded":"2024-03-31T15:09:52.559175+0000","last_fullsized":"2024-03-31T15:09:52.559175+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_clean_scrub_stamp":"2024-03-31T15:09:50.511936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T16:09:39.274428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.2c","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-31T15:09:52.666506+0000","last_change":"2024-03-31T15:09:51.543793+0000","last_active":"2024-03-31T15:09:52.666506+0000","last_peered":"2024-03-31T15:09:52.666506+0000","last_clean":"2024-03-31T15:09:52.666506+0000","last_became_active":"2024-03-31T15:09:51.543080+0000","last_became_peered":"2024-03-31T15:09:51.543080+0000","last_unstale":"2024-03-31T15:09:52.666506+0000","last_undegraded":"2024-03-31T15:09:52.666506+0000","last_fullsized":"2024-03-31T15:09:52.666506+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_clean_scrub_stamp":"2024-03-31T15:09:50.511936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T16:04:24.273984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2b","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-31T15:09:52.666462+0000","last_change":"2024-03-31T15:09:51.543561+0000","last_active":"2024-03-31T15:09:52.666462+0000","last_peered":"2024-03-31T15:09:52.666462+0000","last_clean":"2024-03-31T15:09:52.666462+0000","last_became_active":"2024-03-31T15:09:51.543284+0000","last_became_peered":"2024-03-31T15:09:51.543284+0000","last_unstale":"2024-03-31T15:09:52.666462+0000","last_undegraded":"2024-03-31T15:09:52.666462+0000","last_fullsized":"2024-03-31T15:09:52.666462+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_clean_scrub_stamp":"2024-03-31T15:09:50.511936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:38:01.351257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2a","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-31T15:09:52.544258+0000","last_change":"2024-03-31T15:09:51.549782+0000","last_active":"2024-03-31T15:09:52.544258+0000","last_peered":"2024-03-31T15:09:52.544258+0000","last_clean":"2024-03-31T15:09:52.544258+0000","last_became_active":"2024-03-31T15:09:51.549443+0000","last_became_peered":"2024-03-31T15:09:51.549443+0000","last_unstale":"2024-03-31T15:09:52.544258+0000","last_undegraded":"2024-03-31T15:09:52.544258+0000","last_fullsized":"2024-03-31T15:09:52.544258+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_clean_scrub_stamp":"2024-03-31T15:09:50.511936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:49:21.020185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.29","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-31T15:09:52.548951+0000","last_change":"2024-03-31T15:09:51.537951+0000","last_active":"2024-03-31T15:09:52.548951+0000","last_peered":"2024-03-31T15:09:52.548951+0000","last_clean":"2024-03-31T15:09:52.548951+0000","last_became_active":"2024-03-31T15:09:51.537667+0000","last_became_peered":"2024-03-31T15:09:51.537667+0000","last_unstale":"2024-03-31T15:09:52.548951+0000","last_undegraded":"2024-03-31T15:09:52.548951+0000","last_fullsized":"2024-03-31T15:09:52.548951+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_clean_scrub_stamp":"2024-03-31T15:09:50.511936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T17:06:30.006241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.28","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-31T15:09:52.665915+0000","last_change":"2024-03-31T15:09:51.547098+0000","last_active":"2024-03-31T15:09:52.665915+0000","last_peered":"2024-03-31T15:09:52.665915+0000","last_clean":"2024-03-31T15:09:52.665915+0000","last_became_active":"2024-03-31T15:09:51.542964+0000","last_became_peered":"2024-03-31T15:09:51.542964+0000","last_unstale":"2024-03-31T15:09:52.665915+0000","last_undegraded":"2024-03-31T15:09:52.665915+0000","last_fullsized":"2024-03-31T15:09:52.665915+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_clean_scrub_stamp":"2024-03-31T15:09:50.511936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T15:13:41.631564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.27","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-31T15:09:52.544824+0000","last_change":"2024-03-31T15:09:51.546775+0000","last_active":"2024-03-31T15:09:52.544824+0000","last_peered":"2024-03-31T15:09:52.544824+0000","last_clean":"2024-03-31T15:09:52.544824+0000","last_became_active":"2024-03-31T15:09:51.544322+0000","last_became_peered":"2024-03-31T15:09:51.544322+0000","last_unstale":"2024-03-31T15:09:52.544824+0000","last_undegraded":"2024-03-31T15:09:52.544824+0000","last_fullsized":"2024-03-31T15:09:52.544824+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_clean_scrub_stamp":"2024-03-31T15:09:50.511936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T19:17:45.203823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.26","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-31T15:09:52.667964+0000","last_change":"2024-03-31T15:09:51.559167+0000","last_active":"2024-03-31T15:09:52.667964+0000","last_peered":"2024-03-31T15:09:52.667964+0000","last_clean":"2024-03-31T15:09:52.667964+0000","last_became_active":"2024-03-31T15:09:51.558877+0000","last_became_peered":"2024-03-31T15:09:51.558877+0000","last_unstale":"2024-03-31T15:09:52.667964+0000","last_undegraded":"2024-03-31T15:09:52.667964+0000","last_fullsized":"2024-03-31T15:09:52.667964+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_clean_scrub_stamp":"2024-03-31T15:09:50.511936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:34:04.012928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.25","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-31T15:09:52.550384+0000","last_change":"2024-03-31T15:09:51.541140+0000","last_active":"2024-03-31T15:09:52.550384+0000","last_peered":"2024-03-31T15:09:52.550384+0000","last_clean":"2024-03-31T15:09:52.550384+0000","last_became_active":"2024-03-31T15:09:51.540885+0000","last_became_peered":"2024-03-31T15:09:51.540885+0000","last_unstale":"2024-03-31T15:09:52.550384+0000","last_undegraded":"2024-03-31T15:09:52.550384+0000","last_fullsized":"2024-03-31T15:09:52.550384+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_clean_scrub_stamp":"2024-03-31T15:09:50.511936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T19:47:03.799495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.24","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-31T15:09:52.549532+0000","last_change":"2024-03-31T15:09:51.540836+0000","last_active":"2024-03-31T15:09:52.549532+0000","last_peered":"2024-03-31T15:09:52.549532+0000","last_clean":"2024-03-31T15:09:52.549532+0000","last_became_active":"2024-03-31T15:09:51.540402+0000","last_became_peered":"2024-03-31T15:09:51.540402+0000","last_unstale":"2024-03-31T15:09:52.549532+0000","last_undegraded":"2024-03-31T15:09:52.549532+0000","last_fullsized":"2024-03-31T15:09:52.549532+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_clean_scrub_stamp":"2024-03-31T15:09:50.511936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:34:48.084767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.23","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-31T15:09:52.544390+0000","last_change":"2024-03-31T15:09:51.546167+0000","last_active":"2024-03-31T15:09:52.544390+0000","last_peered":"2024-03-31T15:09:52.544390+0000","last_clean":"2024-03-31T15:09:52.544390+0000","last_became_active":"2024-03-31T15:09:51.543436+0000","last_became_peered":"2024-03-31T15:09:51.543436+0000","last_unstale":"2024-03-31T15:09:52.544390+0000","last_undegraded":"2024-03-31T15:09:52.544390+0000","last_fullsized":"2024-03-31T15:09:52.544390+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_clean_scrub_stamp":"2024-03-31T15:09:50.511936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:57:59.438991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.22","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-31T15:09:52.544397+0000","last_change":"2024-03-31T15:09:51.543951+0000","last_active":"2024-03-31T15:09:52.544397+0000","last_peered":"2024-03-31T15:09:52.544397+0000","last_clean":"2024-03-31T15:09:52.544397+0000","last_became_active":"2024-03-31T15:09:51.542477+0000","last_became_peered":"2024-03-31T15:09:51.542477+0000","last_unstale":"2024-03-31T15:09:52.544397+0000","last_undegraded":"2024-03-31T15:09:52.544397+0000","last_fullsized":"2024-03-31T15:09:52.544397+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_clean_scrub_stamp":"2024-03-31T15:09:50.511936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:32:33.486273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.21","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-31T15:09:52.549525+0000","last_change":"2024-03-31T15:09:51.539017+0000","last_active":"2024-03-31T15:09:52.549525+0000","last_peered":"2024-03-31T15:09:52.549525+0000","last_clean":"2024-03-31T15:09:52.549525+0000","last_became_active":"2024-03-31T15:09:51.538744+0000","last_became_peered":"2024-03-31T15:09:51.538744+0000","last_unstale":"2024-03-31T15:09:52.549525+0000","last_undegraded":"2024-03-31T15:09:52.549525+0000","last_fullsized":"2024-03-31T15:09:52.549525+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_clean_scrub_stamp":"2024-03-31T15:09:50.511936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T15:58:08.695735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.20","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-31T15:09:52.548859+0000","last_change":"2024-03-31T15:09:51.549947+0000","last_active":"2024-03-31T15:09:52.548859+0000","last_peered":"2024-03-31T15:09:52.548859+0000","last_clean":"2024-03-31T15:09:52.548859+0000","last_became_active":"2024-03-31T15:09:51.548914+0000","last_became_peered":"2024-03-31T15:09:51.548914+0000","last_unstale":"2024-03-31T15:09:52.548859+0000","last_undegraded":"2024-03-31T15:09:52.548859+0000","last_fullsized":"2024-03-31T15:09:52.548859+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_clean_scrub_stamp":"2024-03-31T15:09:50.511936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:26:23.084428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1e","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-31T15:09:52.550624+0000","last_change":"2024-03-31T15:09:51.536892+0000","last_active":"2024-03-31T15:09:52.550624+0000","last_peered":"2024-03-31T15:09:52.550624+0000","last_clean":"2024-03-31T15:09:52.550624+0000","last_became_active":"2024-03-31T15:09:51.536645+0000","last_became_peered":"2024-03-31T15:09:51.536645+0000","last_unstale":"2024-03-31T15:09:52.550624+0000","last_undegraded":"2024-03-31T15:09:52.550624+0000","last_fullsized":"2024-03-31T15:09:52.550624+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_clean_scrub_stamp":"2024-03-31T15:09:50.511936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T17:38:46.806700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1f","version":"80'4","reported_seq":18,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-31T15:09:52.550744+0000","last_change":"2024-03-31T15:09:50.545217+0000","last_active":"2024-03-31T15:09:52.550744+0000","last_peered":"2024-03-31T15:09:52.550744+0000","last_clean":"2024-03-31T15:09:52.550744+0000","last_became_active":"2024-03-31T15:09:50.544922+0000","last_became_peered":"2024-03-31T15:09:50.544922+0000","last_unstale":"2024-03-31T15:09:52.550744+0000","last_undegraded":"2024-03-31T15:09:52.550744+0000","last_fullsized":"2024-03-31T15:09:52.550744+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:09:49.505944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:09:49.505944+0000","last_clean_scrub_stamp":"2024-03-31T15:09:49.505944+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:57:02.348097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1f","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-31T15:09:52.550457+0000","last_change":"2024-03-31T15:09:51.542309+0000","last_active":"2024-03-31T15:09:52.550457+0000","last_peered":"2024-03-31T15:09:52.550457+0000","last_clean":"2024-03-31T15:09:52.550457+0000","last_became_active":"2024-03-31T15:09:51.542014+0000","last_became_peered":"2024-03-31T15:09:51.542014+0000","last_unstale":"2024-03-31T15:09:52.550457+0000","last_undegraded":"2024-03-31T15:09:52.550457+0000","last_fullsized":"2024-03-31T15:09:52.550457+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_clean_scrub_stamp":"2024-03-31T15:09:50.511936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:34:08.773480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1e","version":"80'1","reported_seq":15,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-31T15:09:52.550782+0000","last_change":"2024-03-31T15:09:50.543326+0000","last_active":"2024-03-31T15:09:52.550782+0000","last_peered":"2024-03-31T15:09:52.550782+0000","last_clean":"2024-03-31T15:09:52.550782+0000","last_became_active":"2024-03-31T15:09:50.543080+0000","last_became_peered":"2024-03-31T15:09:50.543080+0000","last_unstale":"2024-03-31T15:09:52.550782+0000","last_undegraded":"2024-03-31T15:09:52.550782+0000","last_fullsized":"2024-03-31T15:09:52.550782+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:09:49.505944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:09:49.505944+0000","last_clean_scrub_stamp":"2024-03-31T15:09:49.505944+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T16:14:43.183861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.1c","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-31T15:09:52.549642+0000","last_change":"2024-03-31T15:09:51.548316+0000","last_active":"2024-03-31T15:09:52.549642+0000","last_peered":"2024-03-31T15:09:52.549642+0000","last_clean":"2024-03-31T15:09:52.549642+0000","last_became_active":"2024-03-31T15:09:51.548059+0000","last_became_peered":"2024-03-31T15:09:51.548059+0000","last_unstale":"2024-03-31T15:09:52.549642+0000","last_undegraded":"2024-03-31T15:09:52.549642+0000","last_fullsized":"2024-03-31T15:09:52.549642+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_clean_scrub_stamp":"2024-03-31T15:09:50.511936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T16:09:39.274428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1d","version":"80'2","reported_seq":16,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-31T15:09:52.559150+0000","last_change":"2024-03-31T15:09:50.538148+0000","last_active":"2024-03-31T15:09:52.559150+0000","last_peered":"2024-03-31T15:09:52.559150+0000","last_clean":"2024-03-31T15:09:52.559150+0000","last_became_active":"2024-03-31T15:09:50.537866+0000","last_became_peered":"2024-03-31T15:09:50.537866+0000","last_unstale":"2024-03-31T15:09:52.559150+0000","last_undegraded":"2024-03-31T15:09:52.559150+0000","last_fullsized":"2024-03-31T15:09:52.559150+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:09:49.505944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:09:49.505944+0000","last_clean_scrub_stamp":"2024-03-31T15:09:49.505944+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:48:53.589741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1d","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-31T15:09:52.667612+0000","last_change":"2024-03-31T15:09:51.550578+0000","last_active":"2024-03-31T15:09:52.667612+0000","last_peered":"2024-03-31T15:09:52.667612+0000","last_clean":"2024-03-31T15:09:52.667612+0000","last_became_active":"2024-03-31T15:09:51.550324+0000","last_became_peered":"2024-03-31T15:09:51.550324+0000","last_unstale":"2024-03-31T15:09:52.667612+0000","last_undegraded":"2024-03-31T15:09:52.667612+0000","last_fullsized":"2024-03-31T15:09:52.667612+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_clean_scrub_stamp":"2024-03-31T15:09:50.511936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:52:20.745228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1c","version":"80'2","reported_seq":16,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-31T15:09:52.558962+0000","last_change":"2024-03-31T15:09:50.538287+0000","last_active":"2024-03-31T15:09:52.558962+0000","last_peered":"2024-03-31T15:09:52.558962+0000","last_clean":"2024-03-31T15:09:52.558962+0000","last_became_active":"2024-03-31T15:09:50.538003+0000","last_became_peered":"2024-03-31T15:09:50.538003+0000","last_unstale":"2024-03-31T15:09:52.558962+0000","last_undegraded":"2024-03-31T15:09:52.558962+0000","last_fullsized":"2024-03-31T15:09:52.558962+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:09:49.505944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:09:49.505944+0000","last_clean_scrub_stamp":"2024-03-31T15:09:49.505944+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:43:08.145230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1a","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-31T15:09:52.544551+0000","last_change":"2024-03-31T15:09:51.542775+0000","last_active":"2024-03-31T15:09:52.544551+0000","last_peered":"2024-03-31T15:09:52.544551+0000","last_clean":"2024-03-31T15:09:52.544551+0000","last_became_active":"2024-03-31T15:09:51.542156+0000","last_became_peered":"2024-03-31T15:09:51.542156+0000","last_unstale":"2024-03-31T15:09:52.544551+0000","last_undegraded":"2024-03-31T15:09:52.544551+0000","last_fullsized":"2024-03-31T15:09:52.544551+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_clean_scrub_stamp":"2024-03-31T15:09:50.511936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:38:55.935292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.1b","version":"80'1","reported_seq":15,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-31T15:09:52.550186+0000","last_change":"2024-03-31T15:09:50.541919+0000","last_active":"2024-03-31T15:09:52.550186+0000","last_peered":"2024-03-31T15:09:52.550186+0000","last_clean":"2024-03-31T15:09:52.550186+0000","last_became_active":"2024-03-31T15:09:50.541635+0000","last_became_peered":"2024-03-31T15:09:50.541635+0000","last_unstale":"2024-03-31T15:09:52.550186+0000","last_undegraded":"2024-03-31T15:09:52.550186+0000","last_fullsized":"2024-03-31T15:09:52.550186+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:09:49.505944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:09:49.505944+0000","last_clean_scrub_stamp":"2024-03-31T15:09:49.505944+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T19:24:01.003392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.1b","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-31T15:09:52.550112+0000","last_change":"2024-03-31T15:09:51.538370+0000","last_active":"2024-03-31T15:09:52.550112+0000","last_peered":"2024-03-31T15:09:52.550112+0000","last_clean":"2024-03-31T15:09:52.550112+0000","last_became_active":"2024-03-31T15:09:51.538078+0000","last_became_peered":"2024-03-31T15:09:51.538078+0000","last_unstale":"2024-03-31T15:09:52.550112+0000","last_undegraded":"2024-03-31T15:09:52.550112+0000","last_fullsized":"2024-03-31T15:09:52.550112+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_clean_scrub_stamp":"2024-03-31T15:09:50.511936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:17:46.089835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1a","version":"80'1","reported_seq":15,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-31T15:09:52.549972+0000","last_change":"2024-03-31T15:09:50.540730+0000","last_active":"2024-03-31T15:09:52.549972+0000","last_peered":"2024-03-31T15:09:52.549972+0000","last_clean":"2024-03-31T15:09:52.549972+0000","last_became_active":"2024-03-31T15:09:50.540459+0000","last_became_peered":"2024-03-31T15:09:50.540459+0000","last_unstale":"2024-03-31T15:09:52.549972+0000","last_undegraded":"2024-03-31T15:09:52.549972+0000","last_fullsized":"2024-03-31T15:09:52.549972+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:09:49.505944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:09:49.505944+0000","last_clean_scrub_stamp":"2024-03-31T15:09:49.505944+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:55:42.160711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.18","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-31T15:09:52.666900+0000","last_change":"2024-03-31T15:09:51.548569+0000","last_active":"2024-03-31T15:09:52.666900+0000","last_peered":"2024-03-31T15:09:52.666900+0000","last_clean":"2024-03-31T15:09:52.666900+0000","last_became_active":"2024-03-31T15:09:51.548311+0000","last_became_peered":"2024-03-31T15:09:51.548311+0000","last_unstale":"2024-03-31T15:09:52.666900+0000","last_undegraded":"2024-03-31T15:09:52.666900+0000","last_fullsized":"2024-03-31T15:09:52.666900+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_clean_scrub_stamp":"2024-03-31T15:09:50.511936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:57:10.227550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-31T15:09:52.558639+0000","last_change":"2024-03-31T15:09:50.537083+0000","last_active":"2024-03-31T15:09:52.558639+0000","last_peered":"2024-03-31T15:09:52.558639+0000","last_clean":"2024-03-31T15:09:52.558639+0000","last_became_active":"2024-03-31T15:09:50.536811+0000","last_became_peered":"2024-03-31T15:09:50.536811+0000","last_unstale":"2024-03-31T15:09:52.558639+0000","last_undegraded":"2024-03-31T15:09:52.558639+0000","last_fullsized":"2024-03-31T15:09:52.558639+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:09:49.505944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:09:49.505944+0000","last_clean_scrub_stamp":"2024-03-31T15:09:49.505944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T19:10:34.012335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.19","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-31T15:09:51.553888+0000","last_change":"2024-03-31T15:09:51.550926+0000","last_active":"2024-03-31T15:09:51.553888+0000","last_peered":"2024-03-31T15:09:51.553888+0000","last_clean":"2024-03-31T15:09:51.553888+0000","last_became_active":"2024-03-31T15:09:51.546622+0000","last_became_peered":"2024-03-31T15:09:51.546622+0000","last_unstale":"2024-03-31T15:09:51.553888+0000","last_undegraded":"2024-03-31T15:09:51.553888+0000","last_fullsized":"2024-03-31T15:09:51.553888+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_clean_scrub_stamp":"2024-03-31T15:09:50.511936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:04:17.103002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-31T15:09:52.543883+0000","last_change":"2024-03-31T15:09:50.533668+0000","last_active":"2024-03-31T15:09:52.543883+0000","last_peered":"2024-03-31T15:09:52.543883+0000","last_clean":"2024-03-31T15:09:52.543883+0000","last_became_active":"2024-03-31T15:09:50.533372+0000","last_became_peered":"2024-03-31T15:09:50.533372+0000","last_unstale":"2024-03-31T15:09:52.543883+0000","last_undegraded":"2024-03-31T15:09:52.543883+0000","last_fullsized":"2024-03-31T15:09:52.543883+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:09:49.505944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:09:49.505944+0000","last_clean_scrub_stamp":"2024-03-31T15:09:49.505944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T16:23:02.142692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.16","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-31T15:09:51.555173+0000","last_change":"2024-03-31T15:09:51.553387+0000","last_active":"2024-03-31T15:09:51.555173+0000","last_peered":"2024-03-31T15:09:51.555173+0000","last_clean":"2024-03-31T15:09:51.555173+0000","last_became_active":"2024-03-31T15:09:51.553070+0000","last_became_peered":"2024-03-31T15:09:51.553070+0000","last_unstale":"2024-03-31T15:09:51.555173+0000","last_undegraded":"2024-03-31T15:09:51.555173+0000","last_fullsized":"2024-03-31T15:09:51.555173+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_clean_scrub_stamp":"2024-03-31T15:09:50.511936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:53:42.790342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-31T15:09:52.550683+0000","last_change":"2024-03-31T15:09:50.544934+0000","last_active":"2024-03-31T15:09:52.550683+0000","last_peered":"2024-03-31T15:09:52.550683+0000","last_clean":"2024-03-31T15:09:52.550683+0000","last_became_active":"2024-03-31T15:09:50.544637+0000","last_became_peered":"2024-03-31T15:09:50.544637+0000","last_unstale":"2024-03-31T15:09:52.550683+0000","last_undegraded":"2024-03-31T15:09:52.550683+0000","last_fullsized":"2024-03-31T15:09:52.550683+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:09:49.505944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:09:49.505944+0000","last_clean_scrub_stamp":"2024-03-31T15:09:49.505944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T16:51:49.273095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.17","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-31T15:09:52.667244+0000","last_change":"2024-03-31T15:09:51.542123+0000","last_active":"2024-03-31T15:09:52.667244+0000","last_peered":"2024-03-31T15:09:52.667244+0000","last_clean":"2024-03-31T15:09:52.667244+0000","last_became_active":"2024-03-31T15:09:51.541813+0000","last_became_peered":"2024-03-31T15:09:51.541813+0000","last_unstale":"2024-03-31T15:09:52.667244+0000","last_undegraded":"2024-03-31T15:09:52.667244+0000","last_fullsized":"2024-03-31T15:09:52.667244+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_clean_scrub_stamp":"2024-03-31T15:09:50.511936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T16:39:51.561018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-31T15:09:52.549514+0000","last_change":"2024-03-31T15:09:50.538425+0000","last_active":"2024-03-31T15:09:52.549514+0000","last_peered":"2024-03-31T15:09:52.549514+0000","last_clean":"2024-03-31T15:09:52.549514+0000","last_became_active":"2024-03-31T15:09:50.537990+0000","last_became_peered":"2024-03-31T15:09:50.537990+0000","last_unstale":"2024-03-31T15:09:52.549514+0000","last_undegraded":"2024-03-31T15:09:52.549514+0000","last_fullsized":"2024-03-31T15:09:52.549514+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:09:49.505944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:09:49.505944+0000","last_clean_scrub_stamp":"2024-03-31T15:09:49.505944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T16:38:05.228480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.14","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-31T15:09:52.558988+0000","last_change":"2024-03-31T15:09:51.537871+0000","last_active":"2024-03-31T15:09:52.558988+0000","last_peered":"2024-03-31T15:09:52.558988+0000","last_clean":"2024-03-31T15:09:52.558988+0000","last_became_active":"2024-03-31T15:09:51.536342+0000","last_became_peered":"2024-03-31T15:09:51.536342+0000","last_unstale":"2024-03-31T15:09:52.558988+0000","last_undegraded":"2024-03-31T15:09:52.558988+0000","last_fullsized":"2024-03-31T15:09:52.558988+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_clean_scrub_stamp":"2024-03-31T15:09:50.511936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:34:48.084767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.15","version":"80'3","reported_seq":17,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-31T15:09:52.549095+0000","last_change":"2024-03-31T15:09:50.544034+0000","last_active":"2024-03-31T15:09:52.549095+0000","last_peered":"2024-03-31T15:09:52.549095+0000","last_clean":"2024-03-31T15:09:52.549095+0000","last_became_active":"2024-03-31T15:09:50.543817+0000","last_became_peered":"2024-03-31T15:09:50.543817+0000","last_unstale":"2024-03-31T15:09:52.549095+0000","last_undegraded":"2024-03-31T15:09:52.549095+0000","last_fullsized":"2024-03-31T15:09:52.549095+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:09:49.505944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:09:49.505944+0000","last_clean_scrub_stamp":"2024-03-31T15:09:49.505944+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T15:52:18.465224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.15","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-31T15:09:52.550378+0000","last_change":"2024-03-31T15:09:51.546851+0000","last_active":"2024-03-31T15:09:52.550378+0000","last_peered":"2024-03-31T15:09:52.550378+0000","last_clean":"2024-03-31T15:09:52.550378+0000","last_became_active":"2024-03-31T15:09:51.546090+0000","last_became_peered":"2024-03-31T15:09:51.546090+0000","last_unstale":"2024-03-31T15:09:52.550378+0000","last_undegraded":"2024-03-31T15:09:52.550378+0000","last_fullsized":"2024-03-31T15:09:52.550378+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_clean_scrub_stamp":"2024-03-31T15:09:50.511936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T19:08:15.892648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.14","version":"80'6","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-31T15:09:52.667116+0000","last_change":"2024-03-31T15:09:50.540842+0000","last_active":"2024-03-31T15:09:52.667116+0000","last_peered":"2024-03-31T15:09:52.667116+0000","last_clean":"2024-03-31T15:09:52.667116+0000","last_became_active":"2024-03-31T15:09:50.540533+0000","last_became_peered":"2024-03-31T15:09:50.540533+0000","last_unstale":"2024-03-31T15:09:52.667116+0000","last_undegraded":"2024-03-31T15:09:52.667116+0000","last_fullsized":"2024-03-31T15:09:52.667116+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:09:49.505944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:09:49.505944+0000","last_clean_scrub_stamp":"2024-03-31T15:09:49.505944+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T15:43:07.032387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.12","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-31T15:09:52.558785+0000","last_change":"2024-03-31T15:09:51.550650+0000","last_active":"2024-03-31T15:09:52.558785+0000","last_peered":"2024-03-31T15:09:52.558785+0000","last_clean":"2024-03-31T15:09:52.558785+0000","last_became_active":"2024-03-31T15:09:51.548190+0000","last_became_peered":"2024-03-31T15:09:51.548190+0000","last_unstale":"2024-03-31T15:09:52.558785+0000","last_undegraded":"2024-03-31T15:09:52.558785+0000","last_fullsized":"2024-03-31T15:09:52.558785+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_clean_scrub_stamp":"2024-03-31T15:09:50.511936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T17:06:30.006241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.13","version":"80'2","reported_seq":16,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-31T15:09:52.544600+0000","last_change":"2024-03-31T15:09:50.540625+0000","last_active":"2024-03-31T15:09:52.544600+0000","last_peered":"2024-03-31T15:09:52.544600+0000","last_clean":"2024-03-31T15:09:52.544600+0000","last_became_active":"2024-03-31T15:09:50.540209+0000","last_became_peered":"2024-03-31T15:09:50.540209+0000","last_unstale":"2024-03-31T15:09:52.544600+0000","last_undegraded":"2024-03-31T15:09:52.544600+0000","last_fullsized":"2024-03-31T15:09:52.544600+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:09:49.505944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:09:49.505944+0000","last_clean_scrub_stamp":"2024-03-31T15:09:49.505944+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T15:42:58.090713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.13","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-31T15:09:52.667595+0000","last_change":"2024-03-31T15:09:51.547812+0000","last_active":"2024-03-31T15:09:52.667595+0000","last_peered":"2024-03-31T15:09:52.667595+0000","last_clean":"2024-03-31T15:09:52.667595+0000","last_became_active":"2024-03-31T15:09:51.547531+0000","last_became_peered":"2024-03-31T15:09:51.547531+0000","last_unstale":"2024-03-31T15:09:52.667595+0000","last_undegraded":"2024-03-31T15:09:52.667595+0000","last_fullsized":"2024-03-31T15:09:52.667595+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_clean_scrub_stamp":"2024-03-31T15:09:50.511936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T19:16:50.867364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-31T15:09:51.523817+0000","last_change":"2024-03-31T15:09:50.544023+0000","last_active":"2024-03-31T15:09:51.523817+0000","last_peered":"2024-03-31T15:09:51.523817+0000","last_clean":"2024-03-31T15:09:51.523817+0000","last_became_active":"2024-03-31T15:09:50.543788+0000","last_became_peered":"2024-03-31T15:09:50.543788+0000","last_unstale":"2024-03-31T15:09:51.523817+0000","last_undegraded":"2024-03-31T15:09:51.523817+0000","last_fullsized":"2024-03-31T15:09:51.523817+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:09:49.505944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:09:49.505944+0000","last_clean_scrub_stamp":"2024-03-31T15:09:49.505944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:03:49.568921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.10","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-31T15:09:52.543856+0000","last_change":"2024-03-31T15:09:51.538756+0000","last_active":"2024-03-31T15:09:52.543856+0000","last_peered":"2024-03-31T15:09:52.543856+0000","last_clean":"2024-03-31T15:09:52.543856+0000","last_became_active":"2024-03-31T15:09:51.538469+0000","last_became_peered":"2024-03-31T15:09:51.538469+0000","last_unstale":"2024-03-31T15:09:52.543856+0000","last_undegraded":"2024-03-31T15:09:52.543856+0000","last_fullsized":"2024-03-31T15:09:52.543856+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_clean_scrub_stamp":"2024-03-31T15:09:50.511936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:05:25.856582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-31T15:09:52.667063+0000","last_change":"2024-03-31T15:09:50.533351+0000","last_active":"2024-03-31T15:09:52.667063+0000","last_peered":"2024-03-31T15:09:52.667063+0000","last_clean":"2024-03-31T15:09:52.667063+0000","last_became_active":"2024-03-31T15:09:50.532596+0000","last_became_peered":"2024-03-31T15:09:50.532596+0000","last_unstale":"2024-03-31T15:09:52.667063+0000","last_undegraded":"2024-03-31T15:09:52.667063+0000","last_fullsized":"2024-03-31T15:09:52.667063+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:09:49.505944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:09:49.505944+0000","last_clean_scrub_stamp":"2024-03-31T15:09:49.505944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T19:25:06.659724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.11","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-31T15:09:52.543811+0000","last_change":"2024-03-31T15:09:51.543962+0000","last_active":"2024-03-31T15:09:52.543811+0000","last_peered":"2024-03-31T15:09:52.543811+0000","last_clean":"2024-03-31T15:09:52.543811+0000","last_became_active":"2024-03-31T15:09:51.543666+0000","last_became_peered":"2024-03-31T15:09:51.543666+0000","last_unstale":"2024-03-31T15:09:52.543811+0000","last_undegraded":"2024-03-31T15:09:52.543811+0000","last_fullsized":"2024-03-31T15:09:52.543811+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_clean_scrub_stamp":"2024-03-31T15:09:50.511936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:12:59.780143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.10","version":"80'8","reported_seq":22,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-31T15:09:52.558430+0000","last_change":"2024-03-31T15:09:50.537134+0000","last_active":"2024-03-31T15:09:52.558430+0000","last_peered":"2024-03-31T15:09:52.558430+0000","last_clean":"2024-03-31T15:09:52.558430+0000","last_became_active":"2024-03-31T15:09:50.536875+0000","last_became_peered":"2024-03-31T15:09:50.536875+0000","last_unstale":"2024-03-31T15:09:52.558430+0000","last_undegraded":"2024-03-31T15:09:52.558430+0000","last_fullsized":"2024-03-31T15:09:52.558430+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:09:49.505944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:09:49.505944+0000","last_clean_scrub_stamp":"2024-03-31T15:09:49.505944+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:43:48.699483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.e","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-31T15:09:52.667596+0000","last_change":"2024-03-31T15:09:51.546032+0000","last_active":"2024-03-31T15:09:52.667596+0000","last_peered":"2024-03-31T15:09:52.667596+0000","last_clean":"2024-03-31T15:09:52.667596+0000","last_became_active":"2024-03-31T15:09:51.545449+0000","last_became_peered":"2024-03-31T15:09:51.545449+0000","last_unstale":"2024-03-31T15:09:52.667596+0000","last_undegraded":"2024-03-31T15:09:52.667596+0000","last_fullsized":"2024-03-31T15:09:52.667596+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_clean_scrub_stamp":"2024-03-31T15:09:50.511936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:53:28.586386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.f","version":"80'1","reported_seq":15,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-31T15:09:52.550871+0000","last_change":"2024-03-31T15:09:50.536213+0000","last_active":"2024-03-31T15:09:52.550871+0000","last_peered":"2024-03-31T15:09:52.550871+0000","last_clean":"2024-03-31T15:09:52.550871+0000","last_became_active":"2024-03-31T15:09:50.535763+0000","last_became_peered":"2024-03-31T15:09:50.535763+0000","last_unstale":"2024-03-31T15:09:52.550871+0000","last_undegraded":"2024-03-31T15:09:52.550871+0000","last_fullsized":"2024-03-31T15:09:52.550871+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:09:49.505944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:09:49.505944+0000","last_clean_scrub_stamp":"2024-03-31T15:09:49.505944+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:59:41.721395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.f","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-31T15:09:52.550601+0000","last_change":"2024-03-31T15:09:51.545827+0000","last_active":"2024-03-31T15:09:52.550601+0000","last_peered":"2024-03-31T15:09:52.550601+0000","last_clean":"2024-03-31T15:09:52.550601+0000","last_became_active":"2024-03-31T15:09:51.545521+0000","last_became_peered":"2024-03-31T15:09:51.545521+0000","last_unstale":"2024-03-31T15:09:52.550601+0000","last_undegraded":"2024-03-31T15:09:52.550601+0000","last_fullsized":"2024-03-31T15:09:52.550601+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_clean_scrub_stamp":"2024-03-31T15:09:50.511936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:14:22.643945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.e","version":"80'2","reported_seq":16,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-31T15:09:52.559164+0000","last_change":"2024-03-31T15:09:50.539986+0000","last_active":"2024-03-31T15:09:52.559164+0000","last_peered":"2024-03-31T15:09:52.559164+0000","last_clean":"2024-03-31T15:09:52.559164+0000","last_became_active":"2024-03-31T15:09:50.539712+0000","last_became_peered":"2024-03-31T15:09:50.539712+0000","last_unstale":"2024-03-31T15:09:52.559164+0000","last_undegraded":"2024-03-31T15:09:52.559164+0000","last_fullsized":"2024-03-31T15:09:52.559164+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:09:49.505944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:09:49.505944+0000","last_clean_scrub_stamp":"2024-03-31T15:09:49.505944+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:45:33.452687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.d","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-31T15:09:52.550663+0000","last_change":"2024-03-31T15:09:51.547077+0000","last_active":"2024-03-31T15:09:52.550663+0000","last_peered":"2024-03-31T15:09:52.550663+0000","last_clean":"2024-03-31T15:09:52.550663+0000","last_became_active":"2024-03-31T15:09:51.546544+0000","last_became_peered":"2024-03-31T15:09:51.546544+0000","last_unstale":"2024-03-31T15:09:52.550663+0000","last_undegraded":"2024-03-31T15:09:52.550663+0000","last_fullsized":"2024-03-31T15:09:52.550663+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_clean_scrub_stamp":"2024-03-31T15:09:50.511936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:11:51.786356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-31T15:09:52.559110+0000","last_change":"2024-03-31T15:09:50.535196+0000","last_active":"2024-03-31T15:09:52.559110+0000","last_peered":"2024-03-31T15:09:52.559110+0000","last_clean":"2024-03-31T15:09:52.559110+0000","last_became_active":"2024-03-31T15:09:50.534917+0000","last_became_peered":"2024-03-31T15:09:50.534917+0000","last_unstale":"2024-03-31T15:09:52.559110+0000","last_undegraded":"2024-03-31T15:09:52.559110+0000","last_fullsized":"2024-03-31T15:09:52.559110+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:09:49.505944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:09:49.505944+0000","last_clean_scrub_stamp":"2024-03-31T15:09:49.505944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T19:39:46.883617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.a","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-31T15:09:52.666610+0000","last_change":"2024-03-31T15:09:51.550715+0000","last_active":"2024-03-31T15:09:52.666610+0000","last_peered":"2024-03-31T15:09:52.666610+0000","last_clean":"2024-03-31T15:09:52.666610+0000","last_became_active":"2024-03-31T15:09:51.550400+0000","last_became_peered":"2024-03-31T15:09:51.550400+0000","last_unstale":"2024-03-31T15:09:52.666610+0000","last_undegraded":"2024-03-31T15:09:52.666610+0000","last_fullsized":"2024-03-31T15:09:52.666610+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_clean_scrub_stamp":"2024-03-31T15:09:50.511936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:13:12.207860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.b","version":"80'2","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-31T15:09:51.566105+0000","last_change":"2024-03-31T15:09:50.542249+0000","last_active":"2024-03-31T15:09:51.566105+0000","last_peered":"2024-03-31T15:09:51.566105+0000","last_clean":"2024-03-31T15:09:51.566105+0000","last_became_active":"2024-03-31T15:09:50.541962+0000","last_became_peered":"2024-03-31T15:09:50.541962+0000","last_unstale":"2024-03-31T15:09:51.566105+0000","last_undegraded":"2024-03-31T15:09:51.566105+0000","last_fullsized":"2024-03-31T15:09:51.566105+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:09:49.505944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:09:49.505944+0000","last_clean_scrub_stamp":"2024-03-31T15:09:49.505944+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:35:26.584096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.b","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-31T15:09:52.667221+0000","last_change":"2024-03-31T15:09:51.547631+0000","last_active":"2024-03-31T15:09:52.667221+0000","last_peered":"2024-03-31T15:09:52.667221+0000","last_clean":"2024-03-31T15:09:52.667221+0000","last_became_active":"2024-03-31T15:09:51.547286+0000","last_became_peered":"2024-03-31T15:09:51.547286+0000","last_unstale":"2024-03-31T15:09:52.667221+0000","last_undegraded":"2024-03-31T15:09:52.667221+0000","last_fullsized":"2024-03-31T15:09:52.667221+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_clean_scrub_stamp":"2024-03-31T15:09:50.511936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T16:00:33.458197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-31T15:09:52.666714+0000","last_change":"2024-03-31T15:09:50.540958+0000","last_active":"2024-03-31T15:09:52.666714+0000","last_peered":"2024-03-31T15:09:52.666714+0000","last_clean":"2024-03-31T15:09:52.666714+0000","last_became_active":"2024-03-31T15:09:50.540683+0000","last_became_peered":"2024-03-31T15:09:50.540683+0000","last_unstale":"2024-03-31T15:09:52.666714+0000","last_undegraded":"2024-03-31T15:09:52.666714+0000","last_fullsized":"2024-03-31T15:09:52.666714+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:09:49.505944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:09:49.505944+0000","last_clean_scrub_stamp":"2024-03-31T15:09:49.505944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:14:12.331045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.8","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-31T15:09:52.666065+0000","last_change":"2024-03-31T15:09:51.535262+0000","last_active":"2024-03-31T15:09:52.666065+0000","last_peered":"2024-03-31T15:09:52.666065+0000","last_clean":"2024-03-31T15:09:52.666065+0000","last_became_active":"2024-03-31T15:09:51.534918+0000","last_became_peered":"2024-03-31T15:09:51.534918+0000","last_unstale":"2024-03-31T15:09:52.666065+0000","last_undegraded":"2024-03-31T15:09:52.666065+0000","last_fullsized":"2024-03-31T15:09:52.666065+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_clean_scrub_stamp":"2024-03-31T15:09:50.511936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:06:54.523522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-31T15:09:52.666081+0000","last_change":"2024-03-31T15:09:50.537727+0000","last_active":"2024-03-31T15:09:52.666081+0000","last_peered":"2024-03-31T15:09:52.666081+0000","last_clean":"2024-03-31T15:09:52.666081+0000","last_became_active":"2024-03-31T15:09:50.537405+0000","last_became_peered":"2024-03-31T15:09:50.537405+0000","last_unstale":"2024-03-31T15:09:52.666081+0000","last_undegraded":"2024-03-31T15:09:52.666081+0000","last_fullsized":"2024-03-31T15:09:52.666081+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:09:49.505944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:09:49.505944+0000","last_clean_scrub_stamp":"2024-03-31T15:09:49.505944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:19:58.640774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.9","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-31T15:09:52.666890+0000","last_change":"2024-03-31T15:09:51.542988+0000","last_active":"2024-03-31T15:09:52.666890+0000","last_peered":"2024-03-31T15:09:52.666890+0000","last_clean":"2024-03-31T15:09:52.666890+0000","last_became_active":"2024-03-31T15:09:51.542718+0000","last_became_peered":"2024-03-31T15:09:51.542718+0000","last_unstale":"2024-03-31T15:09:52.666890+0000","last_undegraded":"2024-03-31T15:09:52.666890+0000","last_fullsized":"2024-03-31T15:09:52.666890+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_clean_scrub_stamp":"2024-03-31T15:09:50.511936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T19:08:15.892648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-31T15:09:51.523601+0000","last_change":"2024-03-31T15:09:50.541426+0000","last_active":"2024-03-31T15:09:51.523601+0000","last_peered":"2024-03-31T15:09:51.523601+0000","last_clean":"2024-03-31T15:09:51.523601+0000","last_became_active":"2024-03-31T15:09:50.541091+0000","last_became_peered":"2024-03-31T15:09:50.541091+0000","last_unstale":"2024-03-31T15:09:51.523601+0000","last_undegraded":"2024-03-31T15:09:51.523601+0000","last_fullsized":"2024-03-31T15:09:51.523601+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:09:49.505944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:09:49.505944+0000","last_clean_scrub_stamp":"2024-03-31T15:09:49.505944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:22:16.329860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.6","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-31T15:09:52.550723+0000","last_change":"2024-03-31T15:09:51.540460+0000","last_active":"2024-03-31T15:09:52.550723+0000","last_peered":"2024-03-31T15:09:52.550723+0000","last_clean":"2024-03-31T15:09:52.550723+0000","last_became_active":"2024-03-31T15:09:51.540163+0000","last_became_peered":"2024-03-31T15:09:51.540163+0000","last_unstale":"2024-03-31T15:09:52.550723+0000","last_undegraded":"2024-03-31T15:09:52.550723+0000","last_fullsized":"2024-03-31T15:09:52.550723+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_clean_scrub_stamp":"2024-03-31T15:09:50.511936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T16:39:51.561018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.7","version":"80'2","reported_seq":16,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-31T15:09:52.544842+0000","last_change":"2024-03-31T15:09:50.539307+0000","last_active":"2024-03-31T15:09:52.544842+0000","last_peered":"2024-03-31T15:09:52.544842+0000","last_clean":"2024-03-31T15:09:52.544842+0000","last_became_active":"2024-03-31T15:09:50.538797+0000","last_became_peered":"2024-03-31T15:09:50.538797+0000","last_unstale":"2024-03-31T15:09:52.544842+0000","last_undegraded":"2024-03-31T15:09:52.544842+0000","last_fullsized":"2024-03-31T15:09:52.544842+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:09:49.505944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:09:49.505944+0000","last_clean_scrub_stamp":"2024-03-31T15:09:49.505944+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:15:32.976948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.7","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-31T15:09:52.667536+0000","last_change":"2024-03-31T15:09:51.544940+0000","last_active":"2024-03-31T15:09:52.667536+0000","last_peered":"2024-03-31T15:09:52.667536+0000","last_clean":"2024-03-31T15:09:52.667536+0000","last_became_active":"2024-03-31T15:09:51.544605+0000","last_became_peered":"2024-03-31T15:09:51.544605+0000","last_unstale":"2024-03-31T15:09:52.667536+0000","last_undegraded":"2024-03-31T15:09:52.667536+0000","last_fullsized":"2024-03-31T15:09:52.667536+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_clean_scrub_stamp":"2024-03-31T15:09:50.511936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:02:03.905466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.6","version":"81'1","reported_seq":15,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-31T15:09:52.557804+0000","last_change":"2024-03-31T15:09:50.543092+0000","last_active":"2024-03-31T15:09:52.557804+0000","last_peered":"2024-03-31T15:09:52.557804+0000","last_clean":"2024-03-31T15:09:52.557804+0000","last_became_active":"2024-03-31T15:09:50.541837+0000","last_became_peered":"2024-03-31T15:09:50.541837+0000","last_unstale":"2024-03-31T15:09:52.557804+0000","last_undegraded":"2024-03-31T15:09:52.557804+0000","last_fullsized":"2024-03-31T15:09:52.557804+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:09:49.505944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:09:49.505944+0000","last_clean_scrub_stamp":"2024-03-31T15:09:49.505944+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:16:46.083397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.4","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-31T15:09:52.549725+0000","last_change":"2024-03-31T15:09:51.538888+0000","last_active":"2024-03-31T15:09:52.549725+0000","last_peered":"2024-03-31T15:09:52.549725+0000","last_clean":"2024-03-31T15:09:52.549725+0000","last_became_active":"2024-03-31T15:09:51.538547+0000","last_became_peered":"2024-03-31T15:09:51.538547+0000","last_unstale":"2024-03-31T15:09:52.549725+0000","last_undegraded":"2024-03-31T15:09:52.549725+0000","last_fullsized":"2024-03-31T15:09:52.549725+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_clean_scrub_stamp":"2024-03-31T15:09:50.511936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:11:25.799163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-31T15:09:52.549331+0000","last_change":"2024-03-31T15:09:50.545294+0000","last_active":"2024-03-31T15:09:52.549331+0000","last_peered":"2024-03-31T15:09:52.549331+0000","last_clean":"2024-03-31T15:09:52.549331+0000","last_became_active":"2024-03-31T15:09:50.544876+0000","last_became_peered":"2024-03-31T15:09:50.544876+0000","last_unstale":"2024-03-31T15:09:52.549331+0000","last_undegraded":"2024-03-31T15:09:52.549331+0000","last_fullsized":"2024-03-31T15:09:52.549331+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:09:49.505944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:09:49.505944+0000","last_clean_scrub_stamp":"2024-03-31T15:09:49.505944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T19:17:44.197831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.5","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-31T15:09:52.550572+0000","last_change":"2024-03-31T15:09:51.539287+0000","last_active":"2024-03-31T15:09:52.550572+0000","last_peered":"2024-03-31T15:09:52.550572+0000","last_clean":"2024-03-31T15:09:52.550572+0000","last_became_active":"2024-03-31T15:09:51.539027+0000","last_became_peered":"2024-03-31T15:09:51.539027+0000","last_unstale":"2024-03-31T15:09:52.550572+0000","last_undegraded":"2024-03-31T15:09:52.550572+0000","last_fullsized":"2024-03-31T15:09:52.550572+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_clean_scrub_stamp":"2024-03-31T15:09:50.511936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:43:01.479295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-31T15:09:52.549858+0000","last_change":"2024-03-31T15:09:50.540706+0000","last_active":"2024-03-31T15:09:52.549858+0000","last_peered":"2024-03-31T15:09:52.549858+0000","last_clean":"2024-03-31T15:09:52.549858+0000","last_became_active":"2024-03-31T15:09:50.540404+0000","last_became_peered":"2024-03-31T15:09:50.540404+0000","last_unstale":"2024-03-31T15:09:52.549858+0000","last_undegraded":"2024-03-31T15:09:52.549858+0000","last_fullsized":"2024-03-31T15:09:52.549858+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:09:49.505944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:09:49.505944+0000","last_clean_scrub_stamp":"2024-03-31T15:09:49.505944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:18:23.471215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.2","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-31T15:09:52.667027+0000","last_change":"2024-03-31T15:09:51.547051+0000","last_active":"2024-03-31T15:09:52.667027+0000","last_peered":"2024-03-31T15:09:52.667027+0000","last_clean":"2024-03-31T15:09:52.667027+0000","last_became_active":"2024-03-31T15:09:51.542637+0000","last_became_peered":"2024-03-31T15:09:51.542637+0000","last_unstale":"2024-03-31T15:09:52.667027+0000","last_undegraded":"2024-03-31T15:09:52.667027+0000","last_fullsized":"2024-03-31T15:09:52.667027+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_clean_scrub_stamp":"2024-03-31T15:09:50.511936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:43:01.479295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-31T15:09:52.558925+0000","last_change":"2024-03-31T15:09:50.532994+0000","last_active":"2024-03-31T15:09:52.558925+0000","last_peered":"2024-03-31T15:09:52.558925+0000","last_clean":"2024-03-31T15:09:52.558925+0000","last_became_active":"2024-03-31T15:09:50.532729+0000","last_became_peered":"2024-03-31T15:09:50.532729+0000","last_unstale":"2024-03-31T15:09:52.558925+0000","last_undegraded":"2024-03-31T15:09:52.558925+0000","last_fullsized":"2024-03-31T15:09:52.558925+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:09:49.505944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:09:49.505944+0000","last_clean_scrub_stamp":"2024-03-31T15:09:49.505944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:20:39.160870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.3","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-31T15:09:52.558907+0000","last_change":"2024-03-31T15:09:51.548913+0000","last_active":"2024-03-31T15:09:52.558907+0000","last_peered":"2024-03-31T15:09:52.558907+0000","last_clean":"2024-03-31T15:09:52.558907+0000","last_became_active":"2024-03-31T15:09:51.548157+0000","last_became_peered":"2024-03-31T15:09:51.548157+0000","last_unstale":"2024-03-31T15:09:52.558907+0000","last_undegraded":"2024-03-31T15:09:52.558907+0000","last_fullsized":"2024-03-31T15:09:52.558907+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_clean_scrub_stamp":"2024-03-31T15:09:50.511936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:15:50.454538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-31T15:09:52.550098+0000","last_change":"2024-03-31T15:09:50.540209+0000","last_active":"2024-03-31T15:09:52.550098+0000","last_peered":"2024-03-31T15:09:52.550098+0000","last_clean":"2024-03-31T15:09:52.550098+0000","last_became_active":"2024-03-31T15:09:50.539673+0000","last_became_peered":"2024-03-31T15:09:50.539673+0000","last_unstale":"2024-03-31T15:09:52.550098+0000","last_undegraded":"2024-03-31T15:09:52.550098+0000","last_fullsized":"2024-03-31T15:09:52.550098+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:09:49.505944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:09:49.505944+0000","last_clean_scrub_stamp":"2024-03-31T15:09:49.505944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:58:01.144719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.0","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-31T15:09:52.667255+0000","last_change":"2024-03-31T15:09:51.557263+0000","last_active":"2024-03-31T15:09:52.667255+0000","last_peered":"2024-03-31T15:09:52.667255+0000","last_clean":"2024-03-31T15:09:52.667255+0000","last_became_active":"2024-03-31T15:09:51.556909+0000","last_became_peered":"2024-03-31T15:09:51.556909+0000","last_unstale":"2024-03-31T15:09:52.667255+0000","last_undegraded":"2024-03-31T15:09:52.667255+0000","last_fullsized":"2024-03-31T15:09:52.667255+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_clean_scrub_stamp":"2024-03-31T15:09:50.511936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T19:30:58.212618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1","version":"0'0","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-31T15:09:52.549098+0000","last_change":"2024-03-31T15:09:50.540686+0000","last_active":"2024-03-31T15:09:52.549098+0000","last_peered":"2024-03-31T15:09:52.549098+0000","last_clean":"2024-03-31T15:09:52.549098+0000","last_became_active":"2024-03-31T15:09:50.540380+0000","last_became_peered":"2024-03-31T15:09:50.540380+0000","last_unstale":"2024-03-31T15:09:52.549098+0000","last_undegraded":"2024-03-31T15:09:52.549098+0000","last_fullsized":"2024-03-31T15:09:52.549098+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:09:49.505944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:09:49.505944+0000","last_clean_scrub_stamp":"2024-03-31T15:09:49.505944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:56:54.028656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-31T15:09:52.548876+0000","last_change":"2024-03-31T15:09:51.543391+0000","last_active":"2024-03-31T15:09:52.548876+0000","last_peered":"2024-03-31T15:09:52.548876+0000","last_clean":"2024-03-31T15:09:52.548876+0000","last_became_active":"2024-03-31T15:09:51.541847+0000","last_became_peered":"2024-03-31T15:09:51.541847+0000","last_unstale":"2024-03-31T15:09:52.548876+0000","last_undegraded":"2024-03-31T15:09:52.548876+0000","last_fullsized":"2024-03-31T15:09:52.548876+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_clean_scrub_stamp":"2024-03-31T15:09:50.511936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T16:43:30.041245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-31T15:09:52.550150+0000","last_change":"2024-03-31T15:09:50.538704+0000","last_active":"2024-03-31T15:09:52.550150+0000","last_peered":"2024-03-31T15:09:52.550150+0000","last_clean":"2024-03-31T15:09:52.550150+0000","last_became_active":"2024-03-31T15:09:50.538424+0000","last_became_peered":"2024-03-31T15:09:50.538424+0000","last_unstale":"2024-03-31T15:09:52.550150+0000","last_undegraded":"2024-03-31T15:09:52.550150+0000","last_fullsized":"2024-03-31T15:09:52.550150+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:09:49.505944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:09:49.505944+0000","last_clean_scrub_stamp":"2024-03-31T15:09:49.505944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T17:18:19.456364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.c","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-31T15:09:52.544457+0000","last_change":"2024-03-31T15:09:51.544333+0000","last_active":"2024-03-31T15:09:52.544457+0000","last_peered":"2024-03-31T15:09:52.544457+0000","last_clean":"2024-03-31T15:09:52.544457+0000","last_became_active":"2024-03-31T15:09:51.543352+0000","last_became_peered":"2024-03-31T15:09:51.543352+0000","last_unstale":"2024-03-31T15:09:52.544457+0000","last_undegraded":"2024-03-31T15:09:52.544457+0000","last_fullsized":"2024-03-31T15:09:52.544457+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_clean_scrub_stamp":"2024-03-31T15:09:50.511936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:09:54.562905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.d","version":"80'1","reported_seq":15,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-31T15:09:52.667720+0000","last_change":"2024-03-31T15:09:50.538537+0000","last_active":"2024-03-31T15:09:52.667720+0000","last_peered":"2024-03-31T15:09:52.667720+0000","last_clean":"2024-03-31T15:09:52.667720+0000","last_became_active":"2024-03-31T15:09:50.538201+0000","last_became_peered":"2024-03-31T15:09:50.538201+0000","last_unstale":"2024-03-31T15:09:52.667720+0000","last_undegraded":"2024-03-31T15:09:52.667720+0000","last_fullsized":"2024-03-31T15:09:52.667720+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:09:49.505944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:09:49.505944+0000","last_clean_scrub_stamp":"2024-03-31T15:09:49.505944+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T19:47:02.793503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":282,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-31T15:09:52.667319+0000","last_change":"2024-03-31T15:03:24.329890+0000","last_active":"2024-03-31T15:09:52.667319+0000","last_peered":"2024-03-31T15:09:52.667319+0000","last_clean":"2024-03-31T15:09:52.667319+0000","last_became_active":"2024-03-31T15:03:24.328882+0000","last_became_peered":"2024-03-31T15:03:24.328882+0000","last_unstale":"2024-03-31T15:09:52.667319+0000","last_undegraded":"2024-03-31T15:09:52.667319+0000","last_fullsized":"2024-03-31T15:09:52.667319+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:22.885535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:22.885535+0000","last_clean_scrub_stamp":"2024-03-31T15:03:22.885535+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:18:29.004296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":13,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":12,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509523,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32732,"kb_used_data":1548,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339108,"statfs":{"total":96636764160,"available":96603246592,"internally_reserved":0,"allocated":1585152,"data_stored":1010778,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509522,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32168,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339672,"statfs":{"total":96636764160,"available":96603824128,"internally_reserved":0,"allocated":1003520,"data_stored":421974,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":15,"seq":64424509521,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32152,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339688,"statfs":{"total":96636764160,"available":96603840512,"internally_reserved":0,"allocated":987136,"data_stored":420340,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509520,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32148,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339692,"statfs":{"total":96636764160,"available":96603844608,"internally_reserved":0,"allocated":991232,"data_stored":420922,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542226,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32152,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339688,"statfs":{"total":96636764160,"available":96603840512,"internally_reserved":0,"allocated":987136,"data_stored":414734,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542226,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32144,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339696,"statfs":{"total":96636764160,"available":96603848704,"internally_reserved":0,"allocated":987136,"data_stored":415652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542226,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32144,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339696,"statfs":{"total":96636764160,"available":96603848704,"internally_reserved":0,"allocated":987136,"data_stored":414234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542226,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32720,"kb_used_data":1536,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339120,"statfs":{"total":96636764160,"available":96603258880,"internally_reserved":0,"allocated":1572864,"data_stored":1004442,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-31T15:09:55.654 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-31T15:09:55.829 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:55.836+0000 7f0c95a00640 1 Processor -- start 2024-03-31T15:09:55.829 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:55.836+0000 7f0c95a00640 1 -- start start 2024-03-31T15:09:55.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:55.836+0000 7f0c95a00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f0c9006f920 0x7f0c9006fd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:55.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:55.836+0000 7f0c95a00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f0c900d2010 0x7f0c900d24a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:55.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:55.836+0000 7f0c95a00640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0c900d0920 con 0x7f0c90145e40 2024-03-31T15:09:55.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:55.836+0000 7f0c95a00640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f0c900702c0 con 0x7f0c9006f920 2024-03-31T15:09:55.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:55.836+0000 7f0c95a00640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f0c900d06d0 con 0x7f0c900d2010 2024-03-31T15:09:55.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:55.836+0000 7f0c86000640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f0c9006f920 0x7f0c9006fd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:55.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:55.840+0000 7f0c86000640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f0c9006f920 0x7f0c9006fd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59594/0 (socket says 172.21.15.16:59594) 2024-03-31T15:09:55.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:55.840+0000 7f0c86000640 1 -- 172.21.15.16:0/4130585119 learned_addr learned my addr 172.21.15.16:0/4130585119 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:09:55.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:55.840+0000 7f0c8f400640 1 --2- 172.21.15.16:0/4130585119 >> v2:172.21.15.89:3300/0 conn(0x7f0c900d2010 0x7f0c900d24a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:55.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:55.840+0000 7f0c86000640 1 -- 172.21.15.16:0/4130585119 >> v2:172.21.15.89:3300/0 conn(0x7f0c900d2010 msgr2=0x7f0c900d24a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:09:55.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:55.840+0000 7f0c86000640 1 --2- 172.21.15.16:0/4130585119 >> v2:172.21.15.89:3300/0 conn(0x7f0c900d2010 0x7f0c900d24a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:55.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:55.840+0000 7f0c86000640 1 -- 172.21.15.16:0/4130585119 >> v1:172.21.15.89:6789/0 conn(0x7f0c90145e40 legacy=0x7f0c900cc740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:55.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:55.840+0000 7f0c86000640 1 -- 172.21.15.16:0/4130585119 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c900d0c00 con 0x7f0c9006f920 2024-03-31T15:09:55.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:55.840+0000 7f0c8e000640 1 -- 172.21.15.16:0/4130585119 <== mon.1 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 427818784 0 0) 0x7f0c900d0920 con 0x7f0c90145e40 2024-03-31T15:09:55.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:55.840+0000 7f0c86000640 1 --2- 172.21.15.16:0/4130585119 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0c9006f920 0x7f0c9006fd80 secure :-1 s=READY pgs=1089 cs=0 l=1 rev1=1 crypto rx=0x7f0c90160e90 tx=0x7f0c7801c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6008754a9678b1c9 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:09:55.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:55.840+0000 7f0c8e000640 1 -- 172.21.15.16:0/4130585119 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c780a57a0 con 0x7f0c9006f920 2024-03-31T15:09:55.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:55.840+0000 7f0c8e000640 1 -- 172.21.15.16:0/4130585119 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0c780a5940 con 0x7f0c9006f920 2024-03-31T15:09:55.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:55.840+0000 7f0c8e000640 1 -- 172.21.15.16:0/4130585119 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c780a5d10 con 0x7f0c9006f920 2024-03-31T15:09:55.835 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:55.844+0000 7f0c95a00640 1 -- 172.21.15.16:0/4130585119 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0c9006f920 msgr2=0x7f0c9006fd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:55.835 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:55.844+0000 7f0c95a00640 1 --2- 172.21.15.16:0/4130585119 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0c9006f920 0x7f0c9006fd80 secure :-1 s=READY pgs=1089 cs=0 l=1 rev1=1 crypto rx=0x7f0c90160e90 tx=0x7f0c7801c9f0 comp rx=0 tx=0).stop 2024-03-31T15:09:55.835 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:55.844+0000 7f0c95a00640 1 -- 172.21.15.16:0/4130585119 shutdown_connections 2024-03-31T15:09:55.835 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:55.844+0000 7f0c95a00640 1 --2- 172.21.15.16:0/4130585119 >> v2:172.21.15.89:3300/0 conn(0x7f0c900d2010 0x7f0c900d24a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:55.835 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:55.844+0000 7f0c95a00640 1 --2- 172.21.15.16:0/4130585119 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0c9006f920 0x7f0c9006fd80 unknown :-1 s=CLOSED pgs=1089 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:55.835 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:55.844+0000 7f0c95a00640 1 -- 172.21.15.16:0/4130585119 >> 172.21.15.16:0/4130585119 conn(0x7f0c90067330 msgr2=0x7f0c90067740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:09:55.835 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:55.844+0000 7f0c95a00640 1 -- 172.21.15.16:0/4130585119 shutdown_connections 2024-03-31T15:09:55.835 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:55.844+0000 7f0c95a00640 1 -- 172.21.15.16:0/4130585119 wait complete. 2024-03-31T15:09:55.836 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:55.844+0000 7f0c95a00640 1 Processor -- start 2024-03-31T15:09:55.836 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:55.844+0000 7f0c95a00640 1 -- start start 2024-03-31T15:09:55.839 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:55.844+0000 7f0c95a00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f0c9006f920 0x7f0c9015a300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:55.839 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:55.844+0000 7f0c95a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0c900d2010 0x7f0c90158490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:55.839 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:55.844+0000 7f0c95a00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0c90145e40 0x7f0c900b6470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:55.839 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:55.844+0000 7f0c95a00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f0c900d0ee0 con 0x7f0c900d2010 2024-03-31T15:09:55.839 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:55.844+0000 7f0c95a00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f0c900ce9e0 con 0x7f0c90145e40 2024-03-31T15:09:55.839 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:55.844+0000 7f0c95a00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f0c900d2df0 con 0x7f0c9006f920 2024-03-31T15:09:55.839 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:55.844+0000 7f0c86000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0c900d2010 0x7f0c90158490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:55.839 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:55.844+0000 7f0c86000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0c900d2010 0x7f0c90158490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59598/0 (socket says 172.21.15.16:59598) 2024-03-31T15:09:55.839 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:55.844+0000 7f0c86000640 1 -- 172.21.15.16:0/1302061240 learned_addr learned my addr 172.21.15.16:0/1302061240 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:09:55.839 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:55.844+0000 7f0c8f400640 1 --2- 172.21.15.16:0/1302061240 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0c90145e40 0x7f0c900b6470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:55.839 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:55.844+0000 7f0c8ea00640 1 --2- 172.21.15.16:0/1302061240 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f0c9006f920 0x7f0c9015a300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:55.839 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:55.844+0000 7f0c86000640 1 -- 172.21.15.16:0/1302061240 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f0c9006f920 msgr2=0x7f0c9015a300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:55.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:55.844+0000 7f0c86000640 1 --2- 172.21.15.16:0/1302061240 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f0c9006f920 0x7f0c9015a300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:55.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:55.844+0000 7f0c86000640 1 -- 172.21.15.16:0/1302061240 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0c90145e40 msgr2=0x7f0c900b6470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:55.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:55.844+0000 7f0c86000640 1 --2- 172.21.15.16:0/1302061240 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0c90145e40 0x7f0c900b6470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:55.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:55.844+0000 7f0c86000640 1 -- 172.21.15.16:0/1302061240 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c90158d20 con 0x7f0c900d2010 2024-03-31T15:09:55.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:55.848+0000 7f0c86000640 1 --2- 172.21.15.16:0/1302061240 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0c900d2010 0x7f0c90158490 secure :-1 s=READY pgs=1090 cs=0 l=1 rev1=1 crypto rx=0x7f0c780af040 tx=0x7f0c78019790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:09:55.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:55.848+0000 7f0c87400640 1 -- 172.21.15.16:0/1302061240 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c780b8670 con 0x7f0c900d2010 2024-03-31T15:09:55.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:55.848+0000 7f0c87400640 1 -- 172.21.15.16:0/1302061240 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0c780be070 con 0x7f0c900d2010 2024-03-31T15:09:55.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:55.848+0000 7f0c95a00640 1 -- 172.21.15.16:0/1302061240 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0c90159000 con 0x7f0c900d2010 2024-03-31T15:09:55.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:55.848+0000 7f0c87400640 1 -- 172.21.15.16:0/1302061240 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c780b8a90 con 0x7f0c900d2010 2024-03-31T15:09:55.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:55.848+0000 7f0c95a00640 1 -- 172.21.15.16:0/1302061240 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0c901409b0 con 0x7f0c900d2010 2024-03-31T15:09:55.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:55.848+0000 7f0c95a00640 1 -- 172.21.15.16:0/1302061240 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0c901593f0 con 0x7f0c900d2010 2024-03-31T15:09:55.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:55.848+0000 7f0c87400640 1 -- 172.21.15.16:0/1302061240 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f0c780b8c30 con 0x7f0c900d2010 2024-03-31T15:09:55.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:55.848+0000 7f0c87400640 1 --2- 172.21.15.16:0/1302061240 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f0c6c068380 0x7f0c6c06a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:55.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:55.848+0000 7f0c87400640 1 -- 172.21.15.16:0/1302061240 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 5709+0+0 (secure 0 0 0) 0x7f0c78121b00 con 0x7f0c900d2010 2024-03-31T15:09:55.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:55.848+0000 7f0c8ea00640 1 --2- 172.21.15.16:0/1302061240 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f0c6c068380 0x7f0c6c06a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:55.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:55.852+0000 7f0c8ea00640 1 --2- 172.21.15.16:0/1302061240 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f0c6c068380 0x7f0c6c06a840 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f0c80007fd0 tx=0x7f0c80009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:09:55.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:55.852+0000 7f0c87400640 1 -- 172.21.15.16:0/1302061240 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f0c780165e0 con 0x7f0c900d2010 2024-03-31T15:09:56.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:56.016+0000 7f0c95a00640 1 -- 172.21.15.16:0/1302061240 --> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f0c90069950 con 0x7f0c6c068380 2024-03-31T15:09:56.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:56.024+0000 7f0c87400640 1 -- 172.21.15.16:0/1302061240 <== mgr.4100 v2:172.21.15.89:6832/1549407273 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252069 (secure 0 0 0) 0x7f0c90069950 con 0x7f0c6c068380 2024-03-31T15:09:56.018 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:09:56.020 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-03-31T15:09:56.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:56.028+0000 7f0c95a00640 1 -- 172.21.15.16:0/1302061240 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f0c6c068380 msgr2=0x7f0c6c06a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:56.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:56.028+0000 7f0c95a00640 1 --2- 172.21.15.16:0/1302061240 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f0c6c068380 0x7f0c6c06a840 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f0c80007fd0 tx=0x7f0c80009040 comp rx=0 tx=0).stop 2024-03-31T15:09:56.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:56.028+0000 7f0c95a00640 1 -- 172.21.15.16:0/1302061240 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0c900d2010 msgr2=0x7f0c90158490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:56.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:56.028+0000 7f0c95a00640 1 --2- 172.21.15.16:0/1302061240 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0c900d2010 0x7f0c90158490 secure :-1 s=READY pgs=1090 cs=0 l=1 rev1=1 crypto rx=0x7f0c780af040 tx=0x7f0c78019790 comp rx=0 tx=0).stop 2024-03-31T15:09:56.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:56.028+0000 7f0c95a00640 1 -- 172.21.15.16:0/1302061240 shutdown_connections 2024-03-31T15:09:56.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:56.028+0000 7f0c95a00640 1 --2- 172.21.15.16:0/1302061240 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f0c6c068380 0x7f0c6c06a840 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:56.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:56.028+0000 7f0c95a00640 1 --2- 172.21.15.16:0/1302061240 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0c90145e40 0x7f0c900b6470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:56.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:56.028+0000 7f0c95a00640 1 --2- 172.21.15.16:0/1302061240 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0c900d2010 0x7f0c90158490 unknown :-1 s=CLOSED pgs=1090 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:56.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:56.028+0000 7f0c95a00640 1 --2- 172.21.15.16:0/1302061240 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f0c9006f920 0x7f0c9015a300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:56.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:56.028+0000 7f0c95a00640 1 -- 172.21.15.16:0/1302061240 >> 172.21.15.16:0/1302061240 conn(0x7f0c90067330 msgr2=0x7f0c90142c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:09:56.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:56.028+0000 7f0c95a00640 1 -- 172.21.15.16:0/1302061240 shutdown_connections 2024-03-31T15:09:56.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:56.028+0000 7f0c95a00640 1 -- 172.21.15.16:0/1302061240 wait complete. 2024-03-31T15:09:56.039 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":270,"stamp":"2024-03-31T15:09:54.623425+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":124,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":258360,"kb_used_data":8888,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754716360,"statfs":{"total":773094113280,"available":772829552640,"internally_reserved":0,"allocated":9101312,"data_stored":4523076,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":255315504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":35,"apply_latency_ms":35,"commit_latency_ns":35000000,"apply_latency_ns":35000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.109755"},"pg_stats":[{"pgid":"13.3f","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-31T15:09:52.550311+0000","last_change":"2024-03-31T15:09:51.547305+0000","last_active":"2024-03-31T15:09:52.550311+0000","last_peered":"2024-03-31T15:09:52.550311+0000","last_clean":"2024-03-31T15:09:52.550311+0000","last_became_active":"2024-03-31T15:09:51.546999+0000","last_became_peered":"2024-03-31T15:09:51.546999+0000","last_unstale":"2024-03-31T15:09:52.550311+0000","last_undegraded":"2024-03-31T15:09:52.550311+0000","last_fullsized":"2024-03-31T15:09:52.550311+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_clean_scrub_stamp":"2024-03-31T15:09:50.511936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:57:48.201502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.3e","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-31T15:09:52.544886+0000","last_change":"2024-03-31T15:09:51.555327+0000","last_active":"2024-03-31T15:09:52.544886+0000","last_peered":"2024-03-31T15:09:52.544886+0000","last_clean":"2024-03-31T15:09:52.544886+0000","last_became_active":"2024-03-31T15:09:51.555026+0000","last_became_peered":"2024-03-31T15:09:51.555026+0000","last_unstale":"2024-03-31T15:09:52.544886+0000","last_undegraded":"2024-03-31T15:09:52.544886+0000","last_fullsized":"2024-03-31T15:09:52.544886+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_clean_scrub_stamp":"2024-03-31T15:09:50.511936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:18:24.477207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.3d","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-31T15:09:52.667882+0000","last_change":"2024-03-31T15:09:51.544619+0000","last_active":"2024-03-31T15:09:52.667882+0000","last_peered":"2024-03-31T15:09:52.667882+0000","last_clean":"2024-03-31T15:09:52.667882+0000","last_became_active":"2024-03-31T15:09:51.544330+0000","last_became_peered":"2024-03-31T15:09:51.544330+0000","last_unstale":"2024-03-31T15:09:52.667882+0000","last_undegraded":"2024-03-31T15:09:52.667882+0000","last_fullsized":"2024-03-31T15:09:52.667882+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_clean_scrub_stamp":"2024-03-31T15:09:50.511936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:21:07.775538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.3c","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-31T15:09:52.667487+0000","last_change":"2024-03-31T15:09:51.550635+0000","last_active":"2024-03-31T15:09:52.667487+0000","last_peered":"2024-03-31T15:09:52.667487+0000","last_clean":"2024-03-31T15:09:52.667487+0000","last_became_active":"2024-03-31T15:09:51.549526+0000","last_became_peered":"2024-03-31T15:09:51.549526+0000","last_unstale":"2024-03-31T15:09:52.667487+0000","last_undegraded":"2024-03-31T15:09:52.667487+0000","last_fullsized":"2024-03-31T15:09:52.667487+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_clean_scrub_stamp":"2024-03-31T15:09:50.511936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:48:17.779928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.3b","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-31T15:09:52.549662+0000","last_change":"2024-03-31T15:09:51.550825+0000","last_active":"2024-03-31T15:09:52.549662+0000","last_peered":"2024-03-31T15:09:52.549662+0000","last_clean":"2024-03-31T15:09:52.549662+0000","last_became_active":"2024-03-31T15:09:51.550541+0000","last_became_peered":"2024-03-31T15:09:51.550541+0000","last_unstale":"2024-03-31T15:09:52.549662+0000","last_undegraded":"2024-03-31T15:09:52.549662+0000","last_fullsized":"2024-03-31T15:09:52.549662+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_clean_scrub_stamp":"2024-03-31T15:09:50.511936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T16:32:50.574989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.3a","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-31T15:09:52.667031+0000","last_change":"2024-03-31T15:09:51.542684+0000","last_active":"2024-03-31T15:09:52.667031+0000","last_peered":"2024-03-31T15:09:52.667031+0000","last_clean":"2024-03-31T15:09:52.667031+0000","last_became_active":"2024-03-31T15:09:51.542264+0000","last_became_peered":"2024-03-31T15:09:51.542264+0000","last_unstale":"2024-03-31T15:09:52.667031+0000","last_undegraded":"2024-03-31T15:09:52.667031+0000","last_fullsized":"2024-03-31T15:09:52.667031+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_clean_scrub_stamp":"2024-03-31T15:09:50.511936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T17:38:46.806700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.39","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-31T15:09:52.665919+0000","last_change":"2024-03-31T15:09:51.542033+0000","last_active":"2024-03-31T15:09:52.665919+0000","last_peered":"2024-03-31T15:09:52.665919+0000","last_clean":"2024-03-31T15:09:52.665919+0000","last_became_active":"2024-03-31T15:09:51.541720+0000","last_became_peered":"2024-03-31T15:09:51.541720+0000","last_unstale":"2024-03-31T15:09:52.665919+0000","last_undegraded":"2024-03-31T15:09:52.665919+0000","last_fullsized":"2024-03-31T15:09:52.665919+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_clean_scrub_stamp":"2024-03-31T15:09:50.511936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:37:57.748412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.38","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-31T15:09:52.666820+0000","last_change":"2024-03-31T15:09:51.549875+0000","last_active":"2024-03-31T15:09:52.666820+0000","last_peered":"2024-03-31T15:09:52.666820+0000","last_clean":"2024-03-31T15:09:52.666820+0000","last_became_active":"2024-03-31T15:09:51.549573+0000","last_became_peered":"2024-03-31T15:09:51.549573+0000","last_unstale":"2024-03-31T15:09:52.666820+0000","last_undegraded":"2024-03-31T15:09:52.666820+0000","last_fullsized":"2024-03-31T15:09:52.666820+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_clean_scrub_stamp":"2024-03-31T15:09:50.511936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T19:47:03.799495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.37","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-31T15:09:52.667110+0000","last_change":"2024-03-31T15:09:51.546549+0000","last_active":"2024-03-31T15:09:52.667110+0000","last_peered":"2024-03-31T15:09:52.667110+0000","last_clean":"2024-03-31T15:09:52.667110+0000","last_became_active":"2024-03-31T15:09:51.542234+0000","last_became_peered":"2024-03-31T15:09:51.542234+0000","last_unstale":"2024-03-31T15:09:52.667110+0000","last_undegraded":"2024-03-31T15:09:52.667110+0000","last_fullsized":"2024-03-31T15:09:52.667110+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_clean_scrub_stamp":"2024-03-31T15:09:50.511936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:11:51.786356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.36","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-31T15:09:52.549420+0000","last_change":"2024-03-31T15:09:51.543749+0000","last_active":"2024-03-31T15:09:52.549420+0000","last_peered":"2024-03-31T15:09:52.549420+0000","last_clean":"2024-03-31T15:09:52.549420+0000","last_became_active":"2024-03-31T15:09:51.543112+0000","last_became_peered":"2024-03-31T15:09:51.543112+0000","last_unstale":"2024-03-31T15:09:52.549420+0000","last_undegraded":"2024-03-31T15:09:52.549420+0000","last_fullsized":"2024-03-31T15:09:52.549420+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_clean_scrub_stamp":"2024-03-31T15:09:50.511936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:15:50.454538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.35","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-31T15:09:52.544636+0000","last_change":"2024-03-31T15:09:51.546945+0000","last_active":"2024-03-31T15:09:52.544636+0000","last_peered":"2024-03-31T15:09:52.544636+0000","last_clean":"2024-03-31T15:09:52.544636+0000","last_became_active":"2024-03-31T15:09:51.546661+0000","last_became_peered":"2024-03-31T15:09:51.546661+0000","last_unstale":"2024-03-31T15:09:52.544636+0000","last_undegraded":"2024-03-31T15:09:52.544636+0000","last_fullsized":"2024-03-31T15:09:52.544636+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_clean_scrub_stamp":"2024-03-31T15:09:50.511936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T15:52:19.471216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.34","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-31T15:09:52.549808+0000","last_change":"2024-03-31T15:09:51.537351+0000","last_active":"2024-03-31T15:09:52.549808+0000","last_peered":"2024-03-31T15:09:52.549808+0000","last_clean":"2024-03-31T15:09:52.549808+0000","last_became_active":"2024-03-31T15:09:51.537016+0000","last_became_peered":"2024-03-31T15:09:51.537016+0000","last_unstale":"2024-03-31T15:09:52.549808+0000","last_undegraded":"2024-03-31T15:09:52.549808+0000","last_fullsized":"2024-03-31T15:09:52.549808+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_clean_scrub_stamp":"2024-03-31T15:09:50.511936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:14:18.592996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.33","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-31T15:09:52.667438+0000","last_change":"2024-03-31T15:09:51.544698+0000","last_active":"2024-03-31T15:09:52.667438+0000","last_peered":"2024-03-31T15:09:52.667438+0000","last_clean":"2024-03-31T15:09:52.667438+0000","last_became_active":"2024-03-31T15:09:51.544367+0000","last_became_peered":"2024-03-31T15:09:51.544367+0000","last_unstale":"2024-03-31T15:09:52.667438+0000","last_undegraded":"2024-03-31T15:09:52.667438+0000","last_fullsized":"2024-03-31T15:09:52.667438+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_clean_scrub_stamp":"2024-03-31T15:09:50.511936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T19:05:16.947592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.32","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-31T15:09:52.666368+0000","last_change":"2024-03-31T15:09:51.550608+0000","last_active":"2024-03-31T15:09:52.666368+0000","last_peered":"2024-03-31T15:09:52.666368+0000","last_clean":"2024-03-31T15:09:52.666368+0000","last_became_active":"2024-03-31T15:09:51.550159+0000","last_became_peered":"2024-03-31T15:09:51.550159+0000","last_unstale":"2024-03-31T15:09:52.666368+0000","last_undegraded":"2024-03-31T15:09:52.666368+0000","last_fullsized":"2024-03-31T15:09:52.666368+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_clean_scrub_stamp":"2024-03-31T15:09:50.511936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T16:35:00.010577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.31","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-31T15:09:51.553912+0000","last_change":"2024-03-31T15:09:51.550936+0000","last_active":"2024-03-31T15:09:51.553912+0000","last_peered":"2024-03-31T15:09:51.553912+0000","last_clean":"2024-03-31T15:09:51.553912+0000","last_became_active":"2024-03-31T15:09:51.546318+0000","last_became_peered":"2024-03-31T15:09:51.546318+0000","last_unstale":"2024-03-31T15:09:51.553912+0000","last_undegraded":"2024-03-31T15:09:51.553912+0000","last_fullsized":"2024-03-31T15:09:51.553912+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_clean_scrub_stamp":"2024-03-31T15:09:50.511936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:22:32.112780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.30","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-31T15:09:52.549138+0000","last_change":"2024-03-31T15:09:51.535673+0000","last_active":"2024-03-31T15:09:52.549138+0000","last_peered":"2024-03-31T15:09:52.549138+0000","last_clean":"2024-03-31T15:09:52.549138+0000","last_became_active":"2024-03-31T15:09:51.535406+0000","last_became_peered":"2024-03-31T15:09:51.535406+0000","last_unstale":"2024-03-31T15:09:52.549138+0000","last_undegraded":"2024-03-31T15:09:52.549138+0000","last_fullsized":"2024-03-31T15:09:52.549138+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_clean_scrub_stamp":"2024-03-31T15:09:50.511936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:50:05.710178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.2f","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-31T15:09:52.550670+0000","last_change":"2024-03-31T15:09:51.537689+0000","last_active":"2024-03-31T15:09:52.550670+0000","last_peered":"2024-03-31T15:09:52.550670+0000","last_clean":"2024-03-31T15:09:52.550670+0000","last_became_active":"2024-03-31T15:09:51.537397+0000","last_became_peered":"2024-03-31T15:09:51.537397+0000","last_unstale":"2024-03-31T15:09:52.550670+0000","last_undegraded":"2024-03-31T15:09:52.550670+0000","last_fullsized":"2024-03-31T15:09:52.550670+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_clean_scrub_stamp":"2024-03-31T15:09:50.511936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T16:00:33.458197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.2e","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-31T15:09:52.666543+0000","last_change":"2024-03-31T15:09:51.536072+0000","last_active":"2024-03-31T15:09:52.666543+0000","last_peered":"2024-03-31T15:09:52.666543+0000","last_clean":"2024-03-31T15:09:52.666543+0000","last_became_active":"2024-03-31T15:09:51.535757+0000","last_became_peered":"2024-03-31T15:09:51.535757+0000","last_unstale":"2024-03-31T15:09:52.666543+0000","last_undegraded":"2024-03-31T15:09:52.666543+0000","last_fullsized":"2024-03-31T15:09:52.666543+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_clean_scrub_stamp":"2024-03-31T15:09:50.511936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:36:41.005143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2d","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-31T15:09:52.559175+0000","last_change":"2024-03-31T15:09:51.542520+0000","last_active":"2024-03-31T15:09:52.559175+0000","last_peered":"2024-03-31T15:09:52.559175+0000","last_clean":"2024-03-31T15:09:52.559175+0000","last_became_active":"2024-03-31T15:09:51.542251+0000","last_became_peered":"2024-03-31T15:09:51.542251+0000","last_unstale":"2024-03-31T15:09:52.559175+0000","last_undegraded":"2024-03-31T15:09:52.559175+0000","last_fullsized":"2024-03-31T15:09:52.559175+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_clean_scrub_stamp":"2024-03-31T15:09:50.511936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T16:09:39.274428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.2c","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-31T15:09:52.666506+0000","last_change":"2024-03-31T15:09:51.543793+0000","last_active":"2024-03-31T15:09:52.666506+0000","last_peered":"2024-03-31T15:09:52.666506+0000","last_clean":"2024-03-31T15:09:52.666506+0000","last_became_active":"2024-03-31T15:09:51.543080+0000","last_became_peered":"2024-03-31T15:09:51.543080+0000","last_unstale":"2024-03-31T15:09:52.666506+0000","last_undegraded":"2024-03-31T15:09:52.666506+0000","last_fullsized":"2024-03-31T15:09:52.666506+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_clean_scrub_stamp":"2024-03-31T15:09:50.511936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T16:04:24.273984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2b","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-31T15:09:52.666462+0000","last_change":"2024-03-31T15:09:51.543561+0000","last_active":"2024-03-31T15:09:52.666462+0000","last_peered":"2024-03-31T15:09:52.666462+0000","last_clean":"2024-03-31T15:09:52.666462+0000","last_became_active":"2024-03-31T15:09:51.543284+0000","last_became_peered":"2024-03-31T15:09:51.543284+0000","last_unstale":"2024-03-31T15:09:52.666462+0000","last_undegraded":"2024-03-31T15:09:52.666462+0000","last_fullsized":"2024-03-31T15:09:52.666462+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_clean_scrub_stamp":"2024-03-31T15:09:50.511936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:38:01.351257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2a","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-31T15:09:52.544258+0000","last_change":"2024-03-31T15:09:51.549782+0000","last_active":"2024-03-31T15:09:52.544258+0000","last_peered":"2024-03-31T15:09:52.544258+0000","last_clean":"2024-03-31T15:09:52.544258+0000","last_became_active":"2024-03-31T15:09:51.549443+0000","last_became_peered":"2024-03-31T15:09:51.549443+0000","last_unstale":"2024-03-31T15:09:52.544258+0000","last_undegraded":"2024-03-31T15:09:52.544258+0000","last_fullsized":"2024-03-31T15:09:52.544258+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_clean_scrub_stamp":"2024-03-31T15:09:50.511936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:49:21.020185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.29","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-31T15:09:52.548951+0000","last_change":"2024-03-31T15:09:51.537951+0000","last_active":"2024-03-31T15:09:52.548951+0000","last_peered":"2024-03-31T15:09:52.548951+0000","last_clean":"2024-03-31T15:09:52.548951+0000","last_became_active":"2024-03-31T15:09:51.537667+0000","last_became_peered":"2024-03-31T15:09:51.537667+0000","last_unstale":"2024-03-31T15:09:52.548951+0000","last_undegraded":"2024-03-31T15:09:52.548951+0000","last_fullsized":"2024-03-31T15:09:52.548951+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_clean_scrub_stamp":"2024-03-31T15:09:50.511936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T17:06:30.006241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.28","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-31T15:09:52.665915+0000","last_change":"2024-03-31T15:09:51.547098+0000","last_active":"2024-03-31T15:09:52.665915+0000","last_peered":"2024-03-31T15:09:52.665915+0000","last_clean":"2024-03-31T15:09:52.665915+0000","last_became_active":"2024-03-31T15:09:51.542964+0000","last_became_peered":"2024-03-31T15:09:51.542964+0000","last_unstale":"2024-03-31T15:09:52.665915+0000","last_undegraded":"2024-03-31T15:09:52.665915+0000","last_fullsized":"2024-03-31T15:09:52.665915+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_clean_scrub_stamp":"2024-03-31T15:09:50.511936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T15:13:41.631564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.27","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-31T15:09:52.544824+0000","last_change":"2024-03-31T15:09:51.546775+0000","last_active":"2024-03-31T15:09:52.544824+0000","last_peered":"2024-03-31T15:09:52.544824+0000","last_clean":"2024-03-31T15:09:52.544824+0000","last_became_active":"2024-03-31T15:09:51.544322+0000","last_became_peered":"2024-03-31T15:09:51.544322+0000","last_unstale":"2024-03-31T15:09:52.544824+0000","last_undegraded":"2024-03-31T15:09:52.544824+0000","last_fullsized":"2024-03-31T15:09:52.544824+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_clean_scrub_stamp":"2024-03-31T15:09:50.511936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T19:17:45.203823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.26","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-31T15:09:52.667964+0000","last_change":"2024-03-31T15:09:51.559167+0000","last_active":"2024-03-31T15:09:52.667964+0000","last_peered":"2024-03-31T15:09:52.667964+0000","last_clean":"2024-03-31T15:09:52.667964+0000","last_became_active":"2024-03-31T15:09:51.558877+0000","last_became_peered":"2024-03-31T15:09:51.558877+0000","last_unstale":"2024-03-31T15:09:52.667964+0000","last_undegraded":"2024-03-31T15:09:52.667964+0000","last_fullsized":"2024-03-31T15:09:52.667964+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_clean_scrub_stamp":"2024-03-31T15:09:50.511936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:34:04.012928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.25","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-31T15:09:52.550384+0000","last_change":"2024-03-31T15:09:51.541140+0000","last_active":"2024-03-31T15:09:52.550384+0000","last_peered":"2024-03-31T15:09:52.550384+0000","last_clean":"2024-03-31T15:09:52.550384+0000","last_became_active":"2024-03-31T15:09:51.540885+0000","last_became_peered":"2024-03-31T15:09:51.540885+0000","last_unstale":"2024-03-31T15:09:52.550384+0000","last_undegraded":"2024-03-31T15:09:52.550384+0000","last_fullsized":"2024-03-31T15:09:52.550384+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_clean_scrub_stamp":"2024-03-31T15:09:50.511936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T19:47:03.799495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.24","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-31T15:09:52.549532+0000","last_change":"2024-03-31T15:09:51.540836+0000","last_active":"2024-03-31T15:09:52.549532+0000","last_peered":"2024-03-31T15:09:52.549532+0000","last_clean":"2024-03-31T15:09:52.549532+0000","last_became_active":"2024-03-31T15:09:51.540402+0000","last_became_peered":"2024-03-31T15:09:51.540402+0000","last_unstale":"2024-03-31T15:09:52.549532+0000","last_undegraded":"2024-03-31T15:09:52.549532+0000","last_fullsized":"2024-03-31T15:09:52.549532+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_clean_scrub_stamp":"2024-03-31T15:09:50.511936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:34:48.084767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.23","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-31T15:09:52.544390+0000","last_change":"2024-03-31T15:09:51.546167+0000","last_active":"2024-03-31T15:09:52.544390+0000","last_peered":"2024-03-31T15:09:52.544390+0000","last_clean":"2024-03-31T15:09:52.544390+0000","last_became_active":"2024-03-31T15:09:51.543436+0000","last_became_peered":"2024-03-31T15:09:51.543436+0000","last_unstale":"2024-03-31T15:09:52.544390+0000","last_undegraded":"2024-03-31T15:09:52.544390+0000","last_fullsized":"2024-03-31T15:09:52.544390+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_clean_scrub_stamp":"2024-03-31T15:09:50.511936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:57:59.438991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.22","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-31T15:09:52.544397+0000","last_change":"2024-03-31T15:09:51.543951+0000","last_active":"2024-03-31T15:09:52.544397+0000","last_peered":"2024-03-31T15:09:52.544397+0000","last_clean":"2024-03-31T15:09:52.544397+0000","last_became_active":"2024-03-31T15:09:51.542477+0000","last_became_peered":"2024-03-31T15:09:51.542477+0000","last_unstale":"2024-03-31T15:09:52.544397+0000","last_undegraded":"2024-03-31T15:09:52.544397+0000","last_fullsized":"2024-03-31T15:09:52.544397+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_clean_scrub_stamp":"2024-03-31T15:09:50.511936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:32:33.486273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.21","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-31T15:09:52.549525+0000","last_change":"2024-03-31T15:09:51.539017+0000","last_active":"2024-03-31T15:09:52.549525+0000","last_peered":"2024-03-31T15:09:52.549525+0000","last_clean":"2024-03-31T15:09:52.549525+0000","last_became_active":"2024-03-31T15:09:51.538744+0000","last_became_peered":"2024-03-31T15:09:51.538744+0000","last_unstale":"2024-03-31T15:09:52.549525+0000","last_undegraded":"2024-03-31T15:09:52.549525+0000","last_fullsized":"2024-03-31T15:09:52.549525+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_clean_scrub_stamp":"2024-03-31T15:09:50.511936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T15:58:08.695735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.20","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-31T15:09:52.548859+0000","last_change":"2024-03-31T15:09:51.549947+0000","last_active":"2024-03-31T15:09:52.548859+0000","last_peered":"2024-03-31T15:09:52.548859+0000","last_clean":"2024-03-31T15:09:52.548859+0000","last_became_active":"2024-03-31T15:09:51.548914+0000","last_became_peered":"2024-03-31T15:09:51.548914+0000","last_unstale":"2024-03-31T15:09:52.548859+0000","last_undegraded":"2024-03-31T15:09:52.548859+0000","last_fullsized":"2024-03-31T15:09:52.548859+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_clean_scrub_stamp":"2024-03-31T15:09:50.511936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:26:23.084428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1e","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-31T15:09:52.550624+0000","last_change":"2024-03-31T15:09:51.536892+0000","last_active":"2024-03-31T15:09:52.550624+0000","last_peered":"2024-03-31T15:09:52.550624+0000","last_clean":"2024-03-31T15:09:52.550624+0000","last_became_active":"2024-03-31T15:09:51.536645+0000","last_became_peered":"2024-03-31T15:09:51.536645+0000","last_unstale":"2024-03-31T15:09:52.550624+0000","last_undegraded":"2024-03-31T15:09:52.550624+0000","last_fullsized":"2024-03-31T15:09:52.550624+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_clean_scrub_stamp":"2024-03-31T15:09:50.511936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T17:38:46.806700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1f","version":"80'4","reported_seq":18,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-31T15:09:52.550744+0000","last_change":"2024-03-31T15:09:50.545217+0000","last_active":"2024-03-31T15:09:52.550744+0000","last_peered":"2024-03-31T15:09:52.550744+0000","last_clean":"2024-03-31T15:09:52.550744+0000","last_became_active":"2024-03-31T15:09:50.544922+0000","last_became_peered":"2024-03-31T15:09:50.544922+0000","last_unstale":"2024-03-31T15:09:52.550744+0000","last_undegraded":"2024-03-31T15:09:52.550744+0000","last_fullsized":"2024-03-31T15:09:52.550744+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:09:49.505944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:09:49.505944+0000","last_clean_scrub_stamp":"2024-03-31T15:09:49.505944+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:57:02.348097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1f","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-31T15:09:52.550457+0000","last_change":"2024-03-31T15:09:51.542309+0000","last_active":"2024-03-31T15:09:52.550457+0000","last_peered":"2024-03-31T15:09:52.550457+0000","last_clean":"2024-03-31T15:09:52.550457+0000","last_became_active":"2024-03-31T15:09:51.542014+0000","last_became_peered":"2024-03-31T15:09:51.542014+0000","last_unstale":"2024-03-31T15:09:52.550457+0000","last_undegraded":"2024-03-31T15:09:52.550457+0000","last_fullsized":"2024-03-31T15:09:52.550457+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_clean_scrub_stamp":"2024-03-31T15:09:50.511936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:34:08.773480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1e","version":"80'1","reported_seq":15,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-31T15:09:52.550782+0000","last_change":"2024-03-31T15:09:50.543326+0000","last_active":"2024-03-31T15:09:52.550782+0000","last_peered":"2024-03-31T15:09:52.550782+0000","last_clean":"2024-03-31T15:09:52.550782+0000","last_became_active":"2024-03-31T15:09:50.543080+0000","last_became_peered":"2024-03-31T15:09:50.543080+0000","last_unstale":"2024-03-31T15:09:52.550782+0000","last_undegraded":"2024-03-31T15:09:52.550782+0000","last_fullsized":"2024-03-31T15:09:52.550782+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:09:49.505944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:09:49.505944+0000","last_clean_scrub_stamp":"2024-03-31T15:09:49.505944+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T16:14:43.183861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.1c","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-31T15:09:52.549642+0000","last_change":"2024-03-31T15:09:51.548316+0000","last_active":"2024-03-31T15:09:52.549642+0000","last_peered":"2024-03-31T15:09:52.549642+0000","last_clean":"2024-03-31T15:09:52.549642+0000","last_became_active":"2024-03-31T15:09:51.548059+0000","last_became_peered":"2024-03-31T15:09:51.548059+0000","last_unstale":"2024-03-31T15:09:52.549642+0000","last_undegraded":"2024-03-31T15:09:52.549642+0000","last_fullsized":"2024-03-31T15:09:52.549642+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_clean_scrub_stamp":"2024-03-31T15:09:50.511936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T16:09:39.274428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1d","version":"80'2","reported_seq":16,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-31T15:09:52.559150+0000","last_change":"2024-03-31T15:09:50.538148+0000","last_active":"2024-03-31T15:09:52.559150+0000","last_peered":"2024-03-31T15:09:52.559150+0000","last_clean":"2024-03-31T15:09:52.559150+0000","last_became_active":"2024-03-31T15:09:50.537866+0000","last_became_peered":"2024-03-31T15:09:50.537866+0000","last_unstale":"2024-03-31T15:09:52.559150+0000","last_undegraded":"2024-03-31T15:09:52.559150+0000","last_fullsized":"2024-03-31T15:09:52.559150+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:09:49.505944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:09:49.505944+0000","last_clean_scrub_stamp":"2024-03-31T15:09:49.505944+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:48:53.589741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1d","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-31T15:09:52.667612+0000","last_change":"2024-03-31T15:09:51.550578+0000","last_active":"2024-03-31T15:09:52.667612+0000","last_peered":"2024-03-31T15:09:52.667612+0000","last_clean":"2024-03-31T15:09:52.667612+0000","last_became_active":"2024-03-31T15:09:51.550324+0000","last_became_peered":"2024-03-31T15:09:51.550324+0000","last_unstale":"2024-03-31T15:09:52.667612+0000","last_undegraded":"2024-03-31T15:09:52.667612+0000","last_fullsized":"2024-03-31T15:09:52.667612+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_clean_scrub_stamp":"2024-03-31T15:09:50.511936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:52:20.745228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1c","version":"80'2","reported_seq":16,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-31T15:09:52.558962+0000","last_change":"2024-03-31T15:09:50.538287+0000","last_active":"2024-03-31T15:09:52.558962+0000","last_peered":"2024-03-31T15:09:52.558962+0000","last_clean":"2024-03-31T15:09:52.558962+0000","last_became_active":"2024-03-31T15:09:50.538003+0000","last_became_peered":"2024-03-31T15:09:50.538003+0000","last_unstale":"2024-03-31T15:09:52.558962+0000","last_undegraded":"2024-03-31T15:09:52.558962+0000","last_fullsized":"2024-03-31T15:09:52.558962+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:09:49.505944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:09:49.505944+0000","last_clean_scrub_stamp":"2024-03-31T15:09:49.505944+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:43:08.145230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1a","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-31T15:09:52.544551+0000","last_change":"2024-03-31T15:09:51.542775+0000","last_active":"2024-03-31T15:09:52.544551+0000","last_peered":"2024-03-31T15:09:52.544551+0000","last_clean":"2024-03-31T15:09:52.544551+0000","last_became_active":"2024-03-31T15:09:51.542156+0000","last_became_peered":"2024-03-31T15:09:51.542156+0000","last_unstale":"2024-03-31T15:09:52.544551+0000","last_undegraded":"2024-03-31T15:09:52.544551+0000","last_fullsized":"2024-03-31T15:09:52.544551+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_clean_scrub_stamp":"2024-03-31T15:09:50.511936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:38:55.935292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.1b","version":"80'1","reported_seq":15,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-31T15:09:52.550186+0000","last_change":"2024-03-31T15:09:50.541919+0000","last_active":"2024-03-31T15:09:52.550186+0000","last_peered":"2024-03-31T15:09:52.550186+0000","last_clean":"2024-03-31T15:09:52.550186+0000","last_became_active":"2024-03-31T15:09:50.541635+0000","last_became_peered":"2024-03-31T15:09:50.541635+0000","last_unstale":"2024-03-31T15:09:52.550186+0000","last_undegraded":"2024-03-31T15:09:52.550186+0000","last_fullsized":"2024-03-31T15:09:52.550186+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:09:49.505944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:09:49.505944+0000","last_clean_scrub_stamp":"2024-03-31T15:09:49.505944+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T19:24:01.003392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.1b","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-31T15:09:52.550112+0000","last_change":"2024-03-31T15:09:51.538370+0000","last_active":"2024-03-31T15:09:52.550112+0000","last_peered":"2024-03-31T15:09:52.550112+0000","last_clean":"2024-03-31T15:09:52.550112+0000","last_became_active":"2024-03-31T15:09:51.538078+0000","last_became_peered":"2024-03-31T15:09:51.538078+0000","last_unstale":"2024-03-31T15:09:52.550112+0000","last_undegraded":"2024-03-31T15:09:52.550112+0000","last_fullsized":"2024-03-31T15:09:52.550112+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_clean_scrub_stamp":"2024-03-31T15:09:50.511936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:17:46.089835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1a","version":"80'1","reported_seq":15,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-31T15:09:52.549972+0000","last_change":"2024-03-31T15:09:50.540730+0000","last_active":"2024-03-31T15:09:52.549972+0000","last_peered":"2024-03-31T15:09:52.549972+0000","last_clean":"2024-03-31T15:09:52.549972+0000","last_became_active":"2024-03-31T15:09:50.540459+0000","last_became_peered":"2024-03-31T15:09:50.540459+0000","last_unstale":"2024-03-31T15:09:52.549972+0000","last_undegraded":"2024-03-31T15:09:52.549972+0000","last_fullsized":"2024-03-31T15:09:52.549972+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:09:49.505944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:09:49.505944+0000","last_clean_scrub_stamp":"2024-03-31T15:09:49.505944+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:55:42.160711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.18","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-31T15:09:52.666900+0000","last_change":"2024-03-31T15:09:51.548569+0000","last_active":"2024-03-31T15:09:52.666900+0000","last_peered":"2024-03-31T15:09:52.666900+0000","last_clean":"2024-03-31T15:09:52.666900+0000","last_became_active":"2024-03-31T15:09:51.548311+0000","last_became_peered":"2024-03-31T15:09:51.548311+0000","last_unstale":"2024-03-31T15:09:52.666900+0000","last_undegraded":"2024-03-31T15:09:52.666900+0000","last_fullsized":"2024-03-31T15:09:52.666900+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_clean_scrub_stamp":"2024-03-31T15:09:50.511936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:57:10.227550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-31T15:09:52.558639+0000","last_change":"2024-03-31T15:09:50.537083+0000","last_active":"2024-03-31T15:09:52.558639+0000","last_peered":"2024-03-31T15:09:52.558639+0000","last_clean":"2024-03-31T15:09:52.558639+0000","last_became_active":"2024-03-31T15:09:50.536811+0000","last_became_peered":"2024-03-31T15:09:50.536811+0000","last_unstale":"2024-03-31T15:09:52.558639+0000","last_undegraded":"2024-03-31T15:09:52.558639+0000","last_fullsized":"2024-03-31T15:09:52.558639+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:09:49.505944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:09:49.505944+0000","last_clean_scrub_stamp":"2024-03-31T15:09:49.505944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T19:10:34.012335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.19","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-31T15:09:51.553888+0000","last_change":"2024-03-31T15:09:51.550926+0000","last_active":"2024-03-31T15:09:51.553888+0000","last_peered":"2024-03-31T15:09:51.553888+0000","last_clean":"2024-03-31T15:09:51.553888+0000","last_became_active":"2024-03-31T15:09:51.546622+0000","last_became_peered":"2024-03-31T15:09:51.546622+0000","last_unstale":"2024-03-31T15:09:51.553888+0000","last_undegraded":"2024-03-31T15:09:51.553888+0000","last_fullsized":"2024-03-31T15:09:51.553888+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_clean_scrub_stamp":"2024-03-31T15:09:50.511936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:04:17.103002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-31T15:09:52.543883+0000","last_change":"2024-03-31T15:09:50.533668+0000","last_active":"2024-03-31T15:09:52.543883+0000","last_peered":"2024-03-31T15:09:52.543883+0000","last_clean":"2024-03-31T15:09:52.543883+0000","last_became_active":"2024-03-31T15:09:50.533372+0000","last_became_peered":"2024-03-31T15:09:50.533372+0000","last_unstale":"2024-03-31T15:09:52.543883+0000","last_undegraded":"2024-03-31T15:09:52.543883+0000","last_fullsized":"2024-03-31T15:09:52.543883+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:09:49.505944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:09:49.505944+0000","last_clean_scrub_stamp":"2024-03-31T15:09:49.505944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T16:23:02.142692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.16","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-31T15:09:51.555173+0000","last_change":"2024-03-31T15:09:51.553387+0000","last_active":"2024-03-31T15:09:51.555173+0000","last_peered":"2024-03-31T15:09:51.555173+0000","last_clean":"2024-03-31T15:09:51.555173+0000","last_became_active":"2024-03-31T15:09:51.553070+0000","last_became_peered":"2024-03-31T15:09:51.553070+0000","last_unstale":"2024-03-31T15:09:51.555173+0000","last_undegraded":"2024-03-31T15:09:51.555173+0000","last_fullsized":"2024-03-31T15:09:51.555173+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_clean_scrub_stamp":"2024-03-31T15:09:50.511936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:53:42.790342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-31T15:09:52.550683+0000","last_change":"2024-03-31T15:09:50.544934+0000","last_active":"2024-03-31T15:09:52.550683+0000","last_peered":"2024-03-31T15:09:52.550683+0000","last_clean":"2024-03-31T15:09:52.550683+0000","last_became_active":"2024-03-31T15:09:50.544637+0000","last_became_peered":"2024-03-31T15:09:50.544637+0000","last_unstale":"2024-03-31T15:09:52.550683+0000","last_undegraded":"2024-03-31T15:09:52.550683+0000","last_fullsized":"2024-03-31T15:09:52.550683+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:09:49.505944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:09:49.505944+0000","last_clean_scrub_stamp":"2024-03-31T15:09:49.505944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T16:51:49.273095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.17","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-31T15:09:52.667244+0000","last_change":"2024-03-31T15:09:51.542123+0000","last_active":"2024-03-31T15:09:52.667244+0000","last_peered":"2024-03-31T15:09:52.667244+0000","last_clean":"2024-03-31T15:09:52.667244+0000","last_became_active":"2024-03-31T15:09:51.541813+0000","last_became_peered":"2024-03-31T15:09:51.541813+0000","last_unstale":"2024-03-31T15:09:52.667244+0000","last_undegraded":"2024-03-31T15:09:52.667244+0000","last_fullsized":"2024-03-31T15:09:52.667244+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_clean_scrub_stamp":"2024-03-31T15:09:50.511936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T16:39:51.561018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-31T15:09:52.549514+0000","last_change":"2024-03-31T15:09:50.538425+0000","last_active":"2024-03-31T15:09:52.549514+0000","last_peered":"2024-03-31T15:09:52.549514+0000","last_clean":"2024-03-31T15:09:52.549514+0000","last_became_active":"2024-03-31T15:09:50.537990+0000","last_became_peered":"2024-03-31T15:09:50.537990+0000","last_unstale":"2024-03-31T15:09:52.549514+0000","last_undegraded":"2024-03-31T15:09:52.549514+0000","last_fullsized":"2024-03-31T15:09:52.549514+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:09:49.505944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:09:49.505944+0000","last_clean_scrub_stamp":"2024-03-31T15:09:49.505944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T16:38:05.228480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.14","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-31T15:09:52.558988+0000","last_change":"2024-03-31T15:09:51.537871+0000","last_active":"2024-03-31T15:09:52.558988+0000","last_peered":"2024-03-31T15:09:52.558988+0000","last_clean":"2024-03-31T15:09:52.558988+0000","last_became_active":"2024-03-31T15:09:51.536342+0000","last_became_peered":"2024-03-31T15:09:51.536342+0000","last_unstale":"2024-03-31T15:09:52.558988+0000","last_undegraded":"2024-03-31T15:09:52.558988+0000","last_fullsized":"2024-03-31T15:09:52.558988+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_clean_scrub_stamp":"2024-03-31T15:09:50.511936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:34:48.084767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.15","version":"80'3","reported_seq":17,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-31T15:09:52.549095+0000","last_change":"2024-03-31T15:09:50.544034+0000","last_active":"2024-03-31T15:09:52.549095+0000","last_peered":"2024-03-31T15:09:52.549095+0000","last_clean":"2024-03-31T15:09:52.549095+0000","last_became_active":"2024-03-31T15:09:50.543817+0000","last_became_peered":"2024-03-31T15:09:50.543817+0000","last_unstale":"2024-03-31T15:09:52.549095+0000","last_undegraded":"2024-03-31T15:09:52.549095+0000","last_fullsized":"2024-03-31T15:09:52.549095+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:09:49.505944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:09:49.505944+0000","last_clean_scrub_stamp":"2024-03-31T15:09:49.505944+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T15:52:18.465224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.15","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-31T15:09:52.550378+0000","last_change":"2024-03-31T15:09:51.546851+0000","last_active":"2024-03-31T15:09:52.550378+0000","last_peered":"2024-03-31T15:09:52.550378+0000","last_clean":"2024-03-31T15:09:52.550378+0000","last_became_active":"2024-03-31T15:09:51.546090+0000","last_became_peered":"2024-03-31T15:09:51.546090+0000","last_unstale":"2024-03-31T15:09:52.550378+0000","last_undegraded":"2024-03-31T15:09:52.550378+0000","last_fullsized":"2024-03-31T15:09:52.550378+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_clean_scrub_stamp":"2024-03-31T15:09:50.511936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T19:08:15.892648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.14","version":"80'6","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-31T15:09:52.667116+0000","last_change":"2024-03-31T15:09:50.540842+0000","last_active":"2024-03-31T15:09:52.667116+0000","last_peered":"2024-03-31T15:09:52.667116+0000","last_clean":"2024-03-31T15:09:52.667116+0000","last_became_active":"2024-03-31T15:09:50.540533+0000","last_became_peered":"2024-03-31T15:09:50.540533+0000","last_unstale":"2024-03-31T15:09:52.667116+0000","last_undegraded":"2024-03-31T15:09:52.667116+0000","last_fullsized":"2024-03-31T15:09:52.667116+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:09:49.505944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:09:49.505944+0000","last_clean_scrub_stamp":"2024-03-31T15:09:49.505944+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T15:43:07.032387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.12","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-31T15:09:52.558785+0000","last_change":"2024-03-31T15:09:51.550650+0000","last_active":"2024-03-31T15:09:52.558785+0000","last_peered":"2024-03-31T15:09:52.558785+0000","last_clean":"2024-03-31T15:09:52.558785+0000","last_became_active":"2024-03-31T15:09:51.548190+0000","last_became_peered":"2024-03-31T15:09:51.548190+0000","last_unstale":"2024-03-31T15:09:52.558785+0000","last_undegraded":"2024-03-31T15:09:52.558785+0000","last_fullsized":"2024-03-31T15:09:52.558785+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_clean_scrub_stamp":"2024-03-31T15:09:50.511936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T17:06:30.006241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.13","version":"80'2","reported_seq":16,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-31T15:09:52.544600+0000","last_change":"2024-03-31T15:09:50.540625+0000","last_active":"2024-03-31T15:09:52.544600+0000","last_peered":"2024-03-31T15:09:52.544600+0000","last_clean":"2024-03-31T15:09:52.544600+0000","last_became_active":"2024-03-31T15:09:50.540209+0000","last_became_peered":"2024-03-31T15:09:50.540209+0000","last_unstale":"2024-03-31T15:09:52.544600+0000","last_undegraded":"2024-03-31T15:09:52.544600+0000","last_fullsized":"2024-03-31T15:09:52.544600+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:09:49.505944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:09:49.505944+0000","last_clean_scrub_stamp":"2024-03-31T15:09:49.505944+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T15:42:58.090713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.13","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-31T15:09:52.667595+0000","last_change":"2024-03-31T15:09:51.547812+0000","last_active":"2024-03-31T15:09:52.667595+0000","last_peered":"2024-03-31T15:09:52.667595+0000","last_clean":"2024-03-31T15:09:52.667595+0000","last_became_active":"2024-03-31T15:09:51.547531+0000","last_became_peered":"2024-03-31T15:09:51.547531+0000","last_unstale":"2024-03-31T15:09:52.667595+0000","last_undegraded":"2024-03-31T15:09:52.667595+0000","last_fullsized":"2024-03-31T15:09:52.667595+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_clean_scrub_stamp":"2024-03-31T15:09:50.511936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T19:16:50.867364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-31T15:09:51.523817+0000","last_change":"2024-03-31T15:09:50.544023+0000","last_active":"2024-03-31T15:09:51.523817+0000","last_peered":"2024-03-31T15:09:51.523817+0000","last_clean":"2024-03-31T15:09:51.523817+0000","last_became_active":"2024-03-31T15:09:50.543788+0000","last_became_peered":"2024-03-31T15:09:50.543788+0000","last_unstale":"2024-03-31T15:09:51.523817+0000","last_undegraded":"2024-03-31T15:09:51.523817+0000","last_fullsized":"2024-03-31T15:09:51.523817+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:09:49.505944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:09:49.505944+0000","last_clean_scrub_stamp":"2024-03-31T15:09:49.505944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:03:49.568921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.10","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-31T15:09:52.543856+0000","last_change":"2024-03-31T15:09:51.538756+0000","last_active":"2024-03-31T15:09:52.543856+0000","last_peered":"2024-03-31T15:09:52.543856+0000","last_clean":"2024-03-31T15:09:52.543856+0000","last_became_active":"2024-03-31T15:09:51.538469+0000","last_became_peered":"2024-03-31T15:09:51.538469+0000","last_unstale":"2024-03-31T15:09:52.543856+0000","last_undegraded":"2024-03-31T15:09:52.543856+0000","last_fullsized":"2024-03-31T15:09:52.543856+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_clean_scrub_stamp":"2024-03-31T15:09:50.511936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:05:25.856582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-31T15:09:52.667063+0000","last_change":"2024-03-31T15:09:50.533351+0000","last_active":"2024-03-31T15:09:52.667063+0000","last_peered":"2024-03-31T15:09:52.667063+0000","last_clean":"2024-03-31T15:09:52.667063+0000","last_became_active":"2024-03-31T15:09:50.532596+0000","last_became_peered":"2024-03-31T15:09:50.532596+0000","last_unstale":"2024-03-31T15:09:52.667063+0000","last_undegraded":"2024-03-31T15:09:52.667063+0000","last_fullsized":"2024-03-31T15:09:52.667063+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:09:49.505944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:09:49.505944+0000","last_clean_scrub_stamp":"2024-03-31T15:09:49.505944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T19:25:06.659724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.11","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-31T15:09:52.543811+0000","last_change":"2024-03-31T15:09:51.543962+0000","last_active":"2024-03-31T15:09:52.543811+0000","last_peered":"2024-03-31T15:09:52.543811+0000","last_clean":"2024-03-31T15:09:52.543811+0000","last_became_active":"2024-03-31T15:09:51.543666+0000","last_became_peered":"2024-03-31T15:09:51.543666+0000","last_unstale":"2024-03-31T15:09:52.543811+0000","last_undegraded":"2024-03-31T15:09:52.543811+0000","last_fullsized":"2024-03-31T15:09:52.543811+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_clean_scrub_stamp":"2024-03-31T15:09:50.511936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:12:59.780143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.10","version":"80'8","reported_seq":22,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-31T15:09:52.558430+0000","last_change":"2024-03-31T15:09:50.537134+0000","last_active":"2024-03-31T15:09:52.558430+0000","last_peered":"2024-03-31T15:09:52.558430+0000","last_clean":"2024-03-31T15:09:52.558430+0000","last_became_active":"2024-03-31T15:09:50.536875+0000","last_became_peered":"2024-03-31T15:09:50.536875+0000","last_unstale":"2024-03-31T15:09:52.558430+0000","last_undegraded":"2024-03-31T15:09:52.558430+0000","last_fullsized":"2024-03-31T15:09:52.558430+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:09:49.505944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:09:49.505944+0000","last_clean_scrub_stamp":"2024-03-31T15:09:49.505944+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:43:48.699483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.e","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-31T15:09:52.667596+0000","last_change":"2024-03-31T15:09:51.546032+0000","last_active":"2024-03-31T15:09:52.667596+0000","last_peered":"2024-03-31T15:09:52.667596+0000","last_clean":"2024-03-31T15:09:52.667596+0000","last_became_active":"2024-03-31T15:09:51.545449+0000","last_became_peered":"2024-03-31T15:09:51.545449+0000","last_unstale":"2024-03-31T15:09:52.667596+0000","last_undegraded":"2024-03-31T15:09:52.667596+0000","last_fullsized":"2024-03-31T15:09:52.667596+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_clean_scrub_stamp":"2024-03-31T15:09:50.511936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:53:28.586386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.f","version":"80'1","reported_seq":15,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-31T15:09:52.550871+0000","last_change":"2024-03-31T15:09:50.536213+0000","last_active":"2024-03-31T15:09:52.550871+0000","last_peered":"2024-03-31T15:09:52.550871+0000","last_clean":"2024-03-31T15:09:52.550871+0000","last_became_active":"2024-03-31T15:09:50.535763+0000","last_became_peered":"2024-03-31T15:09:50.535763+0000","last_unstale":"2024-03-31T15:09:52.550871+0000","last_undegraded":"2024-03-31T15:09:52.550871+0000","last_fullsized":"2024-03-31T15:09:52.550871+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:09:49.505944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:09:49.505944+0000","last_clean_scrub_stamp":"2024-03-31T15:09:49.505944+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:59:41.721395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.f","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-31T15:09:52.550601+0000","last_change":"2024-03-31T15:09:51.545827+0000","last_active":"2024-03-31T15:09:52.550601+0000","last_peered":"2024-03-31T15:09:52.550601+0000","last_clean":"2024-03-31T15:09:52.550601+0000","last_became_active":"2024-03-31T15:09:51.545521+0000","last_became_peered":"2024-03-31T15:09:51.545521+0000","last_unstale":"2024-03-31T15:09:52.550601+0000","last_undegraded":"2024-03-31T15:09:52.550601+0000","last_fullsized":"2024-03-31T15:09:52.550601+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_clean_scrub_stamp":"2024-03-31T15:09:50.511936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:14:22.643945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.e","version":"80'2","reported_seq":16,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-31T15:09:52.559164+0000","last_change":"2024-03-31T15:09:50.539986+0000","last_active":"2024-03-31T15:09:52.559164+0000","last_peered":"2024-03-31T15:09:52.559164+0000","last_clean":"2024-03-31T15:09:52.559164+0000","last_became_active":"2024-03-31T15:09:50.539712+0000","last_became_peered":"2024-03-31T15:09:50.539712+0000","last_unstale":"2024-03-31T15:09:52.559164+0000","last_undegraded":"2024-03-31T15:09:52.559164+0000","last_fullsized":"2024-03-31T15:09:52.559164+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:09:49.505944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:09:49.505944+0000","last_clean_scrub_stamp":"2024-03-31T15:09:49.505944+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:45:33.452687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.d","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-31T15:09:52.550663+0000","last_change":"2024-03-31T15:09:51.547077+0000","last_active":"2024-03-31T15:09:52.550663+0000","last_peered":"2024-03-31T15:09:52.550663+0000","last_clean":"2024-03-31T15:09:52.550663+0000","last_became_active":"2024-03-31T15:09:51.546544+0000","last_became_peered":"2024-03-31T15:09:51.546544+0000","last_unstale":"2024-03-31T15:09:52.550663+0000","last_undegraded":"2024-03-31T15:09:52.550663+0000","last_fullsized":"2024-03-31T15:09:52.550663+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_clean_scrub_stamp":"2024-03-31T15:09:50.511936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:11:51.786356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-31T15:09:52.559110+0000","last_change":"2024-03-31T15:09:50.535196+0000","last_active":"2024-03-31T15:09:52.559110+0000","last_peered":"2024-03-31T15:09:52.559110+0000","last_clean":"2024-03-31T15:09:52.559110+0000","last_became_active":"2024-03-31T15:09:50.534917+0000","last_became_peered":"2024-03-31T15:09:50.534917+0000","last_unstale":"2024-03-31T15:09:52.559110+0000","last_undegraded":"2024-03-31T15:09:52.559110+0000","last_fullsized":"2024-03-31T15:09:52.559110+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:09:49.505944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:09:49.505944+0000","last_clean_scrub_stamp":"2024-03-31T15:09:49.505944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T19:39:46.883617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.a","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-31T15:09:52.666610+0000","last_change":"2024-03-31T15:09:51.550715+0000","last_active":"2024-03-31T15:09:52.666610+0000","last_peered":"2024-03-31T15:09:52.666610+0000","last_clean":"2024-03-31T15:09:52.666610+0000","last_became_active":"2024-03-31T15:09:51.550400+0000","last_became_peered":"2024-03-31T15:09:51.550400+0000","last_unstale":"2024-03-31T15:09:52.666610+0000","last_undegraded":"2024-03-31T15:09:52.666610+0000","last_fullsized":"2024-03-31T15:09:52.666610+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_clean_scrub_stamp":"2024-03-31T15:09:50.511936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:13:12.207860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.b","version":"80'2","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-31T15:09:51.566105+0000","last_change":"2024-03-31T15:09:50.542249+0000","last_active":"2024-03-31T15:09:51.566105+0000","last_peered":"2024-03-31T15:09:51.566105+0000","last_clean":"2024-03-31T15:09:51.566105+0000","last_became_active":"2024-03-31T15:09:50.541962+0000","last_became_peered":"2024-03-31T15:09:50.541962+0000","last_unstale":"2024-03-31T15:09:51.566105+0000","last_undegraded":"2024-03-31T15:09:51.566105+0000","last_fullsized":"2024-03-31T15:09:51.566105+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:09:49.505944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:09:49.505944+0000","last_clean_scrub_stamp":"2024-03-31T15:09:49.505944+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:35:26.584096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.b","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-31T15:09:52.667221+0000","last_change":"2024-03-31T15:09:51.547631+0000","last_active":"2024-03-31T15:09:52.667221+0000","last_peered":"2024-03-31T15:09:52.667221+0000","last_clean":"2024-03-31T15:09:52.667221+0000","last_became_active":"2024-03-31T15:09:51.547286+0000","last_became_peered":"2024-03-31T15:09:51.547286+0000","last_unstale":"2024-03-31T15:09:52.667221+0000","last_undegraded":"2024-03-31T15:09:52.667221+0000","last_fullsized":"2024-03-31T15:09:52.667221+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_clean_scrub_stamp":"2024-03-31T15:09:50.511936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T16:00:33.458197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-31T15:09:52.666714+0000","last_change":"2024-03-31T15:09:50.540958+0000","last_active":"2024-03-31T15:09:52.666714+0000","last_peered":"2024-03-31T15:09:52.666714+0000","last_clean":"2024-03-31T15:09:52.666714+0000","last_became_active":"2024-03-31T15:09:50.540683+0000","last_became_peered":"2024-03-31T15:09:50.540683+0000","last_unstale":"2024-03-31T15:09:52.666714+0000","last_undegraded":"2024-03-31T15:09:52.666714+0000","last_fullsized":"2024-03-31T15:09:52.666714+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:09:49.505944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:09:49.505944+0000","last_clean_scrub_stamp":"2024-03-31T15:09:49.505944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:14:12.331045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.8","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-31T15:09:52.666065+0000","last_change":"2024-03-31T15:09:51.535262+0000","last_active":"2024-03-31T15:09:52.666065+0000","last_peered":"2024-03-31T15:09:52.666065+0000","last_clean":"2024-03-31T15:09:52.666065+0000","last_became_active":"2024-03-31T15:09:51.534918+0000","last_became_peered":"2024-03-31T15:09:51.534918+0000","last_unstale":"2024-03-31T15:09:52.666065+0000","last_undegraded":"2024-03-31T15:09:52.666065+0000","last_fullsized":"2024-03-31T15:09:52.666065+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_clean_scrub_stamp":"2024-03-31T15:09:50.511936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:06:54.523522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-31T15:09:52.666081+0000","last_change":"2024-03-31T15:09:50.537727+0000","last_active":"2024-03-31T15:09:52.666081+0000","last_peered":"2024-03-31T15:09:52.666081+0000","last_clean":"2024-03-31T15:09:52.666081+0000","last_became_active":"2024-03-31T15:09:50.537405+0000","last_became_peered":"2024-03-31T15:09:50.537405+0000","last_unstale":"2024-03-31T15:09:52.666081+0000","last_undegraded":"2024-03-31T15:09:52.666081+0000","last_fullsized":"2024-03-31T15:09:52.666081+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:09:49.505944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:09:49.505944+0000","last_clean_scrub_stamp":"2024-03-31T15:09:49.505944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:19:58.640774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.9","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-31T15:09:52.666890+0000","last_change":"2024-03-31T15:09:51.542988+0000","last_active":"2024-03-31T15:09:52.666890+0000","last_peered":"2024-03-31T15:09:52.666890+0000","last_clean":"2024-03-31T15:09:52.666890+0000","last_became_active":"2024-03-31T15:09:51.542718+0000","last_became_peered":"2024-03-31T15:09:51.542718+0000","last_unstale":"2024-03-31T15:09:52.666890+0000","last_undegraded":"2024-03-31T15:09:52.666890+0000","last_fullsized":"2024-03-31T15:09:52.666890+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_clean_scrub_stamp":"2024-03-31T15:09:50.511936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T19:08:15.892648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-31T15:09:51.523601+0000","last_change":"2024-03-31T15:09:50.541426+0000","last_active":"2024-03-31T15:09:51.523601+0000","last_peered":"2024-03-31T15:09:51.523601+0000","last_clean":"2024-03-31T15:09:51.523601+0000","last_became_active":"2024-03-31T15:09:50.541091+0000","last_became_peered":"2024-03-31T15:09:50.541091+0000","last_unstale":"2024-03-31T15:09:51.523601+0000","last_undegraded":"2024-03-31T15:09:51.523601+0000","last_fullsized":"2024-03-31T15:09:51.523601+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:09:49.505944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:09:49.505944+0000","last_clean_scrub_stamp":"2024-03-31T15:09:49.505944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:22:16.329860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.6","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-31T15:09:52.550723+0000","last_change":"2024-03-31T15:09:51.540460+0000","last_active":"2024-03-31T15:09:52.550723+0000","last_peered":"2024-03-31T15:09:52.550723+0000","last_clean":"2024-03-31T15:09:52.550723+0000","last_became_active":"2024-03-31T15:09:51.540163+0000","last_became_peered":"2024-03-31T15:09:51.540163+0000","last_unstale":"2024-03-31T15:09:52.550723+0000","last_undegraded":"2024-03-31T15:09:52.550723+0000","last_fullsized":"2024-03-31T15:09:52.550723+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_clean_scrub_stamp":"2024-03-31T15:09:50.511936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T16:39:51.561018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.7","version":"80'2","reported_seq":16,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-31T15:09:52.544842+0000","last_change":"2024-03-31T15:09:50.539307+0000","last_active":"2024-03-31T15:09:52.544842+0000","last_peered":"2024-03-31T15:09:52.544842+0000","last_clean":"2024-03-31T15:09:52.544842+0000","last_became_active":"2024-03-31T15:09:50.538797+0000","last_became_peered":"2024-03-31T15:09:50.538797+0000","last_unstale":"2024-03-31T15:09:52.544842+0000","last_undegraded":"2024-03-31T15:09:52.544842+0000","last_fullsized":"2024-03-31T15:09:52.544842+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:09:49.505944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:09:49.505944+0000","last_clean_scrub_stamp":"2024-03-31T15:09:49.505944+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:15:32.976948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.7","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-31T15:09:52.667536+0000","last_change":"2024-03-31T15:09:51.544940+0000","last_active":"2024-03-31T15:09:52.667536+0000","last_peered":"2024-03-31T15:09:52.667536+0000","last_clean":"2024-03-31T15:09:52.667536+0000","last_became_active":"2024-03-31T15:09:51.544605+0000","last_became_peered":"2024-03-31T15:09:51.544605+0000","last_unstale":"2024-03-31T15:09:52.667536+0000","last_undegraded":"2024-03-31T15:09:52.667536+0000","last_fullsized":"2024-03-31T15:09:52.667536+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_clean_scrub_stamp":"2024-03-31T15:09:50.511936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:02:03.905466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.6","version":"81'1","reported_seq":15,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-31T15:09:52.557804+0000","last_change":"2024-03-31T15:09:50.543092+0000","last_active":"2024-03-31T15:09:52.557804+0000","last_peered":"2024-03-31T15:09:52.557804+0000","last_clean":"2024-03-31T15:09:52.557804+0000","last_became_active":"2024-03-31T15:09:50.541837+0000","last_became_peered":"2024-03-31T15:09:50.541837+0000","last_unstale":"2024-03-31T15:09:52.557804+0000","last_undegraded":"2024-03-31T15:09:52.557804+0000","last_fullsized":"2024-03-31T15:09:52.557804+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:09:49.505944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:09:49.505944+0000","last_clean_scrub_stamp":"2024-03-31T15:09:49.505944+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:16:46.083397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.4","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-31T15:09:52.549725+0000","last_change":"2024-03-31T15:09:51.538888+0000","last_active":"2024-03-31T15:09:52.549725+0000","last_peered":"2024-03-31T15:09:52.549725+0000","last_clean":"2024-03-31T15:09:52.549725+0000","last_became_active":"2024-03-31T15:09:51.538547+0000","last_became_peered":"2024-03-31T15:09:51.538547+0000","last_unstale":"2024-03-31T15:09:52.549725+0000","last_undegraded":"2024-03-31T15:09:52.549725+0000","last_fullsized":"2024-03-31T15:09:52.549725+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_clean_scrub_stamp":"2024-03-31T15:09:50.511936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:11:25.799163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-31T15:09:52.549331+0000","last_change":"2024-03-31T15:09:50.545294+0000","last_active":"2024-03-31T15:09:52.549331+0000","last_peered":"2024-03-31T15:09:52.549331+0000","last_clean":"2024-03-31T15:09:52.549331+0000","last_became_active":"2024-03-31T15:09:50.544876+0000","last_became_peered":"2024-03-31T15:09:50.544876+0000","last_unstale":"2024-03-31T15:09:52.549331+0000","last_undegraded":"2024-03-31T15:09:52.549331+0000","last_fullsized":"2024-03-31T15:09:52.549331+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:09:49.505944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:09:49.505944+0000","last_clean_scrub_stamp":"2024-03-31T15:09:49.505944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T19:17:44.197831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.5","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-31T15:09:52.550572+0000","last_change":"2024-03-31T15:09:51.539287+0000","last_active":"2024-03-31T15:09:52.550572+0000","last_peered":"2024-03-31T15:09:52.550572+0000","last_clean":"2024-03-31T15:09:52.550572+0000","last_became_active":"2024-03-31T15:09:51.539027+0000","last_became_peered":"2024-03-31T15:09:51.539027+0000","last_unstale":"2024-03-31T15:09:52.550572+0000","last_undegraded":"2024-03-31T15:09:52.550572+0000","last_fullsized":"2024-03-31T15:09:52.550572+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_clean_scrub_stamp":"2024-03-31T15:09:50.511936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:43:01.479295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-31T15:09:52.549858+0000","last_change":"2024-03-31T15:09:50.540706+0000","last_active":"2024-03-31T15:09:52.549858+0000","last_peered":"2024-03-31T15:09:52.549858+0000","last_clean":"2024-03-31T15:09:52.549858+0000","last_became_active":"2024-03-31T15:09:50.540404+0000","last_became_peered":"2024-03-31T15:09:50.540404+0000","last_unstale":"2024-03-31T15:09:52.549858+0000","last_undegraded":"2024-03-31T15:09:52.549858+0000","last_fullsized":"2024-03-31T15:09:52.549858+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:09:49.505944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:09:49.505944+0000","last_clean_scrub_stamp":"2024-03-31T15:09:49.505944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:18:23.471215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.2","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-31T15:09:52.667027+0000","last_change":"2024-03-31T15:09:51.547051+0000","last_active":"2024-03-31T15:09:52.667027+0000","last_peered":"2024-03-31T15:09:52.667027+0000","last_clean":"2024-03-31T15:09:52.667027+0000","last_became_active":"2024-03-31T15:09:51.542637+0000","last_became_peered":"2024-03-31T15:09:51.542637+0000","last_unstale":"2024-03-31T15:09:52.667027+0000","last_undegraded":"2024-03-31T15:09:52.667027+0000","last_fullsized":"2024-03-31T15:09:52.667027+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_clean_scrub_stamp":"2024-03-31T15:09:50.511936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:43:01.479295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-31T15:09:52.558925+0000","last_change":"2024-03-31T15:09:50.532994+0000","last_active":"2024-03-31T15:09:52.558925+0000","last_peered":"2024-03-31T15:09:52.558925+0000","last_clean":"2024-03-31T15:09:52.558925+0000","last_became_active":"2024-03-31T15:09:50.532729+0000","last_became_peered":"2024-03-31T15:09:50.532729+0000","last_unstale":"2024-03-31T15:09:52.558925+0000","last_undegraded":"2024-03-31T15:09:52.558925+0000","last_fullsized":"2024-03-31T15:09:52.558925+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:09:49.505944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:09:49.505944+0000","last_clean_scrub_stamp":"2024-03-31T15:09:49.505944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:20:39.160870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.3","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-31T15:09:52.558907+0000","last_change":"2024-03-31T15:09:51.548913+0000","last_active":"2024-03-31T15:09:52.558907+0000","last_peered":"2024-03-31T15:09:52.558907+0000","last_clean":"2024-03-31T15:09:52.558907+0000","last_became_active":"2024-03-31T15:09:51.548157+0000","last_became_peered":"2024-03-31T15:09:51.548157+0000","last_unstale":"2024-03-31T15:09:52.558907+0000","last_undegraded":"2024-03-31T15:09:52.558907+0000","last_fullsized":"2024-03-31T15:09:52.558907+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_clean_scrub_stamp":"2024-03-31T15:09:50.511936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:15:50.454538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-31T15:09:52.550098+0000","last_change":"2024-03-31T15:09:50.540209+0000","last_active":"2024-03-31T15:09:52.550098+0000","last_peered":"2024-03-31T15:09:52.550098+0000","last_clean":"2024-03-31T15:09:52.550098+0000","last_became_active":"2024-03-31T15:09:50.539673+0000","last_became_peered":"2024-03-31T15:09:50.539673+0000","last_unstale":"2024-03-31T15:09:52.550098+0000","last_undegraded":"2024-03-31T15:09:52.550098+0000","last_fullsized":"2024-03-31T15:09:52.550098+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:09:49.505944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:09:49.505944+0000","last_clean_scrub_stamp":"2024-03-31T15:09:49.505944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:58:01.144719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.0","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-31T15:09:52.667255+0000","last_change":"2024-03-31T15:09:51.557263+0000","last_active":"2024-03-31T15:09:52.667255+0000","last_peered":"2024-03-31T15:09:52.667255+0000","last_clean":"2024-03-31T15:09:52.667255+0000","last_became_active":"2024-03-31T15:09:51.556909+0000","last_became_peered":"2024-03-31T15:09:51.556909+0000","last_unstale":"2024-03-31T15:09:52.667255+0000","last_undegraded":"2024-03-31T15:09:52.667255+0000","last_fullsized":"2024-03-31T15:09:52.667255+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_clean_scrub_stamp":"2024-03-31T15:09:50.511936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T19:30:58.212618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1","version":"0'0","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-31T15:09:52.549098+0000","last_change":"2024-03-31T15:09:50.540686+0000","last_active":"2024-03-31T15:09:52.549098+0000","last_peered":"2024-03-31T15:09:52.549098+0000","last_clean":"2024-03-31T15:09:52.549098+0000","last_became_active":"2024-03-31T15:09:50.540380+0000","last_became_peered":"2024-03-31T15:09:50.540380+0000","last_unstale":"2024-03-31T15:09:52.549098+0000","last_undegraded":"2024-03-31T15:09:52.549098+0000","last_fullsized":"2024-03-31T15:09:52.549098+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:09:49.505944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:09:49.505944+0000","last_clean_scrub_stamp":"2024-03-31T15:09:49.505944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:56:54.028656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-31T15:09:52.548876+0000","last_change":"2024-03-31T15:09:51.543391+0000","last_active":"2024-03-31T15:09:52.548876+0000","last_peered":"2024-03-31T15:09:52.548876+0000","last_clean":"2024-03-31T15:09:52.548876+0000","last_became_active":"2024-03-31T15:09:51.541847+0000","last_became_peered":"2024-03-31T15:09:51.541847+0000","last_unstale":"2024-03-31T15:09:52.548876+0000","last_undegraded":"2024-03-31T15:09:52.548876+0000","last_fullsized":"2024-03-31T15:09:52.548876+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_clean_scrub_stamp":"2024-03-31T15:09:50.511936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T16:43:30.041245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-31T15:09:52.550150+0000","last_change":"2024-03-31T15:09:50.538704+0000","last_active":"2024-03-31T15:09:52.550150+0000","last_peered":"2024-03-31T15:09:52.550150+0000","last_clean":"2024-03-31T15:09:52.550150+0000","last_became_active":"2024-03-31T15:09:50.538424+0000","last_became_peered":"2024-03-31T15:09:50.538424+0000","last_unstale":"2024-03-31T15:09:52.550150+0000","last_undegraded":"2024-03-31T15:09:52.550150+0000","last_fullsized":"2024-03-31T15:09:52.550150+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:09:49.505944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:09:49.505944+0000","last_clean_scrub_stamp":"2024-03-31T15:09:49.505944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T17:18:19.456364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.c","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-31T15:09:52.544457+0000","last_change":"2024-03-31T15:09:51.544333+0000","last_active":"2024-03-31T15:09:52.544457+0000","last_peered":"2024-03-31T15:09:52.544457+0000","last_clean":"2024-03-31T15:09:52.544457+0000","last_became_active":"2024-03-31T15:09:51.543352+0000","last_became_peered":"2024-03-31T15:09:51.543352+0000","last_unstale":"2024-03-31T15:09:52.544457+0000","last_undegraded":"2024-03-31T15:09:52.544457+0000","last_fullsized":"2024-03-31T15:09:52.544457+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:09:50.511936+0000","last_clean_scrub_stamp":"2024-03-31T15:09:50.511936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:09:54.562905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.d","version":"80'1","reported_seq":15,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-31T15:09:52.667720+0000","last_change":"2024-03-31T15:09:50.538537+0000","last_active":"2024-03-31T15:09:52.667720+0000","last_peered":"2024-03-31T15:09:52.667720+0000","last_clean":"2024-03-31T15:09:52.667720+0000","last_became_active":"2024-03-31T15:09:50.538201+0000","last_became_peered":"2024-03-31T15:09:50.538201+0000","last_unstale":"2024-03-31T15:09:52.667720+0000","last_undegraded":"2024-03-31T15:09:52.667720+0000","last_fullsized":"2024-03-31T15:09:52.667720+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:09:49.505944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:09:49.505944+0000","last_clean_scrub_stamp":"2024-03-31T15:09:49.505944+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T19:47:02.793503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":282,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-31T15:09:52.667319+0000","last_change":"2024-03-31T15:03:24.329890+0000","last_active":"2024-03-31T15:09:52.667319+0000","last_peered":"2024-03-31T15:09:52.667319+0000","last_clean":"2024-03-31T15:09:52.667319+0000","last_became_active":"2024-03-31T15:03:24.328882+0000","last_became_peered":"2024-03-31T15:03:24.328882+0000","last_unstale":"2024-03-31T15:09:52.667319+0000","last_undegraded":"2024-03-31T15:09:52.667319+0000","last_fullsized":"2024-03-31T15:09:52.667319+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:22.885535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:22.885535+0000","last_clean_scrub_stamp":"2024-03-31T15:03:22.885535+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:18:29.004296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":13,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":12,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509523,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32732,"kb_used_data":1548,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339108,"statfs":{"total":96636764160,"available":96603246592,"internally_reserved":0,"allocated":1585152,"data_stored":1010778,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509522,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32168,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339672,"statfs":{"total":96636764160,"available":96603824128,"internally_reserved":0,"allocated":1003520,"data_stored":421974,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":15,"seq":64424509521,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32152,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339688,"statfs":{"total":96636764160,"available":96603840512,"internally_reserved":0,"allocated":987136,"data_stored":420340,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509520,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32148,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339692,"statfs":{"total":96636764160,"available":96603844608,"internally_reserved":0,"allocated":991232,"data_stored":420922,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542226,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32152,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339688,"statfs":{"total":96636764160,"available":96603840512,"internally_reserved":0,"allocated":987136,"data_stored":414734,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542226,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32144,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339696,"statfs":{"total":96636764160,"available":96603848704,"internally_reserved":0,"allocated":987136,"data_stored":415652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542226,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32144,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339696,"statfs":{"total":96636764160,"available":96603848704,"internally_reserved":0,"allocated":987136,"data_stored":414234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542226,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32720,"kb_used_data":1536,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339120,"statfs":{"total":96636764160,"available":96603258880,"internally_reserved":0,"allocated":1572864,"data_stored":1004442,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-31T15:09:56.042 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-31T15:09:56.043 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-03-31T15:09:56.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:56.236+0000 7f74a9600640 1 Processor -- start 2024-03-31T15:09:56.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:56.236+0000 7f74a9600640 1 -- start start 2024-03-31T15:09:56.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:56.236+0000 7f74a9600640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f74a41411b0 0x7f74a406f920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:56.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:56.236+0000 7f74a9600640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f74a4141590 0x7f74a406fe60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:56.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:56.236+0000 7f74a9600640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f74a4144980 con 0x7f74a4143c30 2024-03-31T15:09:56.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:56.236+0000 7f74a9600640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f74a415f910 con 0x7f74a41411b0 2024-03-31T15:09:56.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:56.236+0000 7f74a9600640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f74a4162910 con 0x7f74a4141590 2024-03-31T15:09:56.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:56.236+0000 7f74a2a00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f74a41411b0 0x7f74a406f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:56.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:56.236+0000 7f74a3400640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f74a4143c30 0x7f74a41440a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:48270/0 (socket says 172.21.15.16:48270) 2024-03-31T15:09:56.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:56.236+0000 7f74a3400640 1 -- 172.21.15.16:0/1650432941 learned_addr learned my addr 172.21.15.16:0/1650432941 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:09:56.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:56.236+0000 7f749be00640 1 --2- 172.21.15.16:0/1650432941 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f74a4141590 0x7f74a406fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:56.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:56.240+0000 7f74a2a00640 1 -- 172.21.15.16:0/1650432941 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f74a4141590 msgr2=0x7f74a406fe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:09:56.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:56.240+0000 7f74a2a00640 1 --2- 172.21.15.16:0/1650432941 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f74a4141590 0x7f74a406fe60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:56.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:56.240+0000 7f74a2a00640 1 -- 172.21.15.16:0/1650432941 >> v1:172.21.15.16:6789/0 conn(0x7f74a4143c30 legacy=0x7f74a41440a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:56.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:56.240+0000 7f74a2a00640 1 -- 172.21.15.16:0/1650432941 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74a4161fe0 con 0x7f74a41411b0 2024-03-31T15:09:56.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:56.240+0000 7f749be00640 1 --2- 172.21.15.16:0/1650432941 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f74a4141590 0x7f74a406fe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:09:56.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:56.240+0000 7f74a2000640 1 -- 172.21.15.16:0/1650432941 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1194354523 0 0) 0x7f74a4144980 con 0x7f74a4143c30 2024-03-31T15:09:56.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:56.240+0000 7f74a2a00640 1 --2- 172.21.15.16:0/1650432941 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f74a41411b0 0x7f74a406f920 secure :-1 s=READY pgs=1092 cs=0 l=1 rev1=1 crypto rx=0x7f74940097c0 tx=0x7f749401c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=80cd1540ad2a75ce server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:09:56.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:56.240+0000 7f74a2000640 1 -- 172.21.15.16:0/1650432941 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74940b2070 con 0x7f74a41411b0 2024-03-31T15:09:56.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:56.240+0000 7f74a2000640 1 -- 172.21.15.16:0/1650432941 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f74940ad640 con 0x7f74a41411b0 2024-03-31T15:09:56.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:56.240+0000 7f74a2000640 1 -- 172.21.15.16:0/1650432941 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74940ada10 con 0x7f74a41411b0 2024-03-31T15:09:56.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:56.240+0000 7f74a9600640 1 -- 172.21.15.16:0/1650432941 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f74a41411b0 msgr2=0x7f74a406f920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:56.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:56.240+0000 7f74a9600640 1 --2- 172.21.15.16:0/1650432941 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f74a41411b0 0x7f74a406f920 secure :-1 s=READY pgs=1092 cs=0 l=1 rev1=1 crypto rx=0x7f74940097c0 tx=0x7f749401c4a0 comp rx=0 tx=0).stop 2024-03-31T15:09:56.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:56.244+0000 7f74a9600640 1 -- 172.21.15.16:0/1650432941 shutdown_connections 2024-03-31T15:09:56.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:56.244+0000 7f74a9600640 1 --2- 172.21.15.16:0/1650432941 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f74a4141590 0x7f74a406fe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:56.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:56.244+0000 7f74a9600640 1 --2- 172.21.15.16:0/1650432941 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f74a41411b0 0x7f74a406f920 unknown :-1 s=CLOSED pgs=1092 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:56.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:56.244+0000 7f74a9600640 1 -- 172.21.15.16:0/1650432941 >> 172.21.15.16:0/1650432941 conn(0x7f74a4067930 msgr2=0x7f74a40b3ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:09:56.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:56.244+0000 7f74a9600640 1 -- 172.21.15.16:0/1650432941 shutdown_connections 2024-03-31T15:09:56.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:56.244+0000 7f74a9600640 1 -- 172.21.15.16:0/1650432941 wait complete. 2024-03-31T15:09:56.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:56.244+0000 7f74a9600640 1 Processor -- start 2024-03-31T15:09:56.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:56.244+0000 7f74a9600640 1 -- start start 2024-03-31T15:09:56.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:56.244+0000 7f74a9600640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f74a41411b0 0x7f74a4157290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:56.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:56.244+0000 7f74a9600640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f74a4141590 0x7f74a4152260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:56.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:56.244+0000 7f74a9600640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f74a4143c30 0x7f74a41527a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:56.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:56.244+0000 7f74a9600640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f74a415f290 con 0x7f74a4143c30 2024-03-31T15:09:56.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:56.244+0000 7f74a9600640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f74a415f110 con 0x7f74a4141590 2024-03-31T15:09:56.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:56.244+0000 7f74a9600640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f74a415f410 con 0x7f74a41411b0 2024-03-31T15:09:56.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:56.244+0000 7f74a3400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f74a4143c30 0x7f74a41527a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:56.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:56.244+0000 7f74a3400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f74a4143c30 0x7f74a41527a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59620/0 (socket says 172.21.15.16:59620) 2024-03-31T15:09:56.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:56.244+0000 7f74a3400640 1 -- 172.21.15.16:0/1217895440 learned_addr learned my addr 172.21.15.16:0/1217895440 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:09:56.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:56.244+0000 7f749be00640 1 --2- 172.21.15.16:0/1217895440 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f74a4141590 0x7f74a4152260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:56.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:56.244+0000 7f74a2a00640 1 --2- 172.21.15.16:0/1217895440 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f74a41411b0 0x7f74a4157290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:56.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:56.244+0000 7f74a3400640 1 -- 172.21.15.16:0/1217895440 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f74a41411b0 msgr2=0x7f74a4157290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:56.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:56.244+0000 7f74a3400640 1 --2- 172.21.15.16:0/1217895440 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f74a41411b0 0x7f74a4157290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:56.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:56.244+0000 7f74a3400640 1 -- 172.21.15.16:0/1217895440 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f74a4141590 msgr2=0x7f74a4152260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:56.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:56.244+0000 7f74a3400640 1 --2- 172.21.15.16:0/1217895440 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f74a4141590 0x7f74a4152260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:56.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:56.244+0000 7f74a3400640 1 -- 172.21.15.16:0/1217895440 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74a406d6b0 con 0x7f74a4143c30 2024-03-31T15:09:56.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:56.244+0000 7f74a3400640 1 --2- 172.21.15.16:0/1217895440 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f74a4143c30 0x7f74a41527a0 secure :-1 s=READY pgs=1093 cs=0 l=1 rev1=1 crypto rx=0x7f749c00c730 tx=0x7f749c00cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:09:56.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:56.244+0000 7f749aa00640 1 -- 172.21.15.16:0/1217895440 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f749c018070 con 0x7f74a4143c30 2024-03-31T15:09:56.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:56.244+0000 7f749aa00640 1 -- 172.21.15.16:0/1217895440 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f749c00ce40 con 0x7f74a4143c30 2024-03-31T15:09:56.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:56.244+0000 7f749aa00640 1 -- 172.21.15.16:0/1217895440 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f749c00a860 con 0x7f74a4143c30 2024-03-31T15:09:56.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:56.244+0000 7f74a9600640 1 -- 172.21.15.16:0/1217895440 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f74a406d990 con 0x7f74a4143c30 2024-03-31T15:09:56.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:56.248+0000 7f74a9600640 1 -- 172.21.15.16:0/1217895440 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f74a406df50 con 0x7f74a4143c30 2024-03-31T15:09:56.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:56.248+0000 7f74a9600640 1 -- 172.21.15.16:0/1217895440 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f74a4069950 con 0x7f74a4143c30 2024-03-31T15:09:56.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:56.248+0000 7f749aa00640 1 -- 172.21.15.16:0/1217895440 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f749c00aa00 con 0x7f74a4143c30 2024-03-31T15:09:56.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:56.248+0000 7f749aa00640 1 --2- 172.21.15.16:0/1217895440 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f7480068380 0x7f748006a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:56.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:56.248+0000 7f749aa00640 1 -- 172.21.15.16:0/1217895440 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 5709+0+0 (secure 0 0 0) 0x7f749c085f20 con 0x7f74a4143c30 2024-03-31T15:09:56.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:56.248+0000 7f74a2a00640 1 --2- 172.21.15.16:0/1217895440 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f7480068380 0x7f748006a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:56.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:56.252+0000 7f74a2a00640 1 --2- 172.21.15.16:0/1217895440 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f7480068380 0x7f748006a840 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f74940af000 tx=0x7f7494008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:09:56.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:56.252+0000 7f749aa00640 1 -- 172.21.15.16:0/1217895440 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f749c055e80 con 0x7f74a4143c30 2024-03-31T15:09:56.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:56.484+0000 7f74a9600640 1 -- 172.21.15.16:0/1217895440 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) -- 0x7f74a41611c0 con 0x7f74a4143c30 2024-03-31T15:09:56.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:56.488+0000 7f749aa00640 1 -- 172.21.15.16:0/1217895440 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v17) ==== 190+0+147 (secure 0 0 0) 0x7f749c05a180 con 0x7f74a4143c30 2024-03-31T15:09:56.479 INFO:teuthology.orchestra.run.smithi016.stdout:[client.0] 2024-03-31T15:09:56.479 INFO:teuthology.orchestra.run.smithi016.stdout: key = AQAPewlmlMIDEhAAF0Ll1HcrWHmH6xDKq2Qq9A== 2024-03-31T15:09:56.479 INFO:teuthology.orchestra.run.smithi016.stdout: caps mds = "allow" 2024-03-31T15:09:56.479 INFO:teuthology.orchestra.run.smithi016.stdout: caps mon = "allow r" 2024-03-31T15:09:56.479 INFO:teuthology.orchestra.run.smithi016.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-03-31T15:09:56.479 INFO:teuthology.orchestra.run.smithi016.stderr:updated caps for client.0 2024-03-31T15:09:56.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:56.488+0000 7f74a9600640 1 -- 172.21.15.16:0/1217895440 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f7480068380 msgr2=0x7f748006a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:56.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:56.488+0000 7f74a9600640 1 --2- 172.21.15.16:0/1217895440 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f7480068380 0x7f748006a840 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f74940af000 tx=0x7f7494008040 comp rx=0 tx=0).stop 2024-03-31T15:09:56.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:56.488+0000 7f74a9600640 1 -- 172.21.15.16:0/1217895440 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f74a4143c30 msgr2=0x7f74a41527a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:56.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:56.488+0000 7f74a9600640 1 --2- 172.21.15.16:0/1217895440 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f74a4143c30 0x7f74a41527a0 secure :-1 s=READY pgs=1093 cs=0 l=1 rev1=1 crypto rx=0x7f749c00c730 tx=0x7f749c00cbf0 comp rx=0 tx=0).stop 2024-03-31T15:09:56.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:56.488+0000 7f74a9600640 1 -- 172.21.15.16:0/1217895440 shutdown_connections 2024-03-31T15:09:56.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:56.488+0000 7f74a9600640 1 --2- 172.21.15.16:0/1217895440 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f7480068380 0x7f748006a840 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:56.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:56.488+0000 7f74a9600640 1 --2- 172.21.15.16:0/1217895440 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f74a4143c30 0x7f74a41527a0 unknown :-1 s=CLOSED pgs=1093 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:56.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:56.488+0000 7f74a9600640 1 --2- 172.21.15.16:0/1217895440 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f74a4141590 0x7f74a4152260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:56.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:56.488+0000 7f74a9600640 1 --2- 172.21.15.16:0/1217895440 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f74a41411b0 0x7f74a4157290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:56.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:56.488+0000 7f74a9600640 1 -- 172.21.15.16:0/1217895440 >> 172.21.15.16:0/1217895440 conn(0x7f74a4067930 msgr2=0x7f74a4144060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:09:56.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:56.488+0000 7f74a9600640 1 -- 172.21.15.16:0/1217895440 shutdown_connections 2024-03-31T15:09:56.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:56.488+0000 7f74a9600640 1 -- 172.21.15.16:0/1217895440 wait complete. 2024-03-31T15:09:56.502 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-31T15:09:56.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:56.652+0000 7fb784200640 1 Processor -- start 2024-03-31T15:09:56.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:56.652+0000 7fb784200640 1 -- start start 2024-03-31T15:09:56.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:56.652+0000 7fb784200640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb77c0d0c50 0x7fb77c072fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:56.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:56.652+0000 7fb784200640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb77c073510 0x7fb77c15eb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:56.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:56.652+0000 7fb784200640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb77c06bd30 con 0x7fb77c147e70 2024-03-31T15:09:56.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:56.652+0000 7fb784200640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7fb77c0739a0 con 0x7fb77c073510 2024-03-31T15:09:56.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:56.652+0000 7fb784200640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fb77c1621b0 con 0x7fb77c0d0c50 2024-03-31T15:09:56.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:56.652+0000 7fb782200640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb77c073510 0x7fb77c15eb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:56.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:56.652+0000 7fb781800640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fb77c147e70 0x7fb77c06b620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:48284/0 (socket says 172.21.15.16:48284) 2024-03-31T15:09:56.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:56.652+0000 7fb781800640 1 -- 172.21.15.16:0/2338443676 learned_addr learned my addr 172.21.15.16:0/2338443676 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:09:56.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:56.652+0000 7fb780e00640 1 --2- 172.21.15.16:0/2338443676 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb77c0d0c50 0x7fb77c072fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:56.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:56.652+0000 7fb782200640 1 -- 172.21.15.16:0/2338443676 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb77c0d0c50 msgr2=0x7fb77c072fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:09:56.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:56.652+0000 7fb782200640 1 --2- 172.21.15.16:0/2338443676 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb77c0d0c50 0x7fb77c072fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:56.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:56.652+0000 7fb782200640 1 -- 172.21.15.16:0/2338443676 >> v1:172.21.15.16:6789/0 conn(0x7fb77c147e70 legacy=0x7fb77c06b620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:56.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:56.652+0000 7fb782200640 1 -- 172.21.15.16:0/2338443676 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb77c162490 con 0x7fb77c073510 2024-03-31T15:09:56.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:56.652+0000 7fb77be00640 1 -- 172.21.15.16:0/2338443676 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 655025830 0 0) 0x7fb77c06bd30 con 0x7fb77c147e70 2024-03-31T15:09:56.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:56.656+0000 7fb782200640 1 --2- 172.21.15.16:0/2338443676 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb77c073510 0x7fb77c15eb20 secure :-1 s=READY pgs=1095 cs=0 l=1 rev1=1 crypto rx=0x7fb774009d90 tx=0x7fb77401c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a899cf0ef16d4178 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:09:56.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:56.656+0000 7fb77be00640 1 -- 172.21.15.16:0/2338443676 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7740b2070 con 0x7fb77c073510 2024-03-31T15:09:56.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:56.656+0000 7fb77be00640 1 -- 172.21.15.16:0/2338443676 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb7740a5920 con 0x7fb77c073510 2024-03-31T15:09:56.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:56.656+0000 7fb77be00640 1 -- 172.21.15.16:0/2338443676 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7740a5cf0 con 0x7fb77c073510 2024-03-31T15:09:56.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:56.656+0000 7fb784200640 1 -- 172.21.15.16:0/2338443676 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb77c073510 msgr2=0x7fb77c15eb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:56.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:56.656+0000 7fb784200640 1 --2- 172.21.15.16:0/2338443676 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb77c073510 0x7fb77c15eb20 secure :-1 s=READY pgs=1095 cs=0 l=1 rev1=1 crypto rx=0x7fb774009d90 tx=0x7fb77401c970 comp rx=0 tx=0).stop 2024-03-31T15:09:56.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:56.656+0000 7fb784200640 1 -- 172.21.15.16:0/2338443676 shutdown_connections 2024-03-31T15:09:56.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:56.656+0000 7fb784200640 1 --2- 172.21.15.16:0/2338443676 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb77c073510 0x7fb77c15eb20 unknown :-1 s=CLOSED pgs=1095 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:56.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:56.656+0000 7fb784200640 1 --2- 172.21.15.16:0/2338443676 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb77c0d0c50 0x7fb77c072fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:56.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:56.656+0000 7fb784200640 1 -- 172.21.15.16:0/2338443676 >> 172.21.15.16:0/2338443676 conn(0x7fb77c0d1060 msgr2=0x7fb77c0d1470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:09:56.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:56.656+0000 7fb784200640 1 -- 172.21.15.16:0/2338443676 shutdown_connections 2024-03-31T15:09:56.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:56.656+0000 7fb784200640 1 -- 172.21.15.16:0/2338443676 wait complete. 2024-03-31T15:09:56.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:56.660+0000 7fb784200640 1 Processor -- start 2024-03-31T15:09:56.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:56.660+0000 7fb784200640 1 -- start start 2024-03-31T15:09:56.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:56.660+0000 7fb784200640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb77c073510 0x7fb77c15ce10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:56.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:56.660+0000 7fb784200640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb77c0d0c50 0x7fb77c0b4d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:56.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:56.660+0000 7fb784200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb77c147e70 0x7fb77c0b52a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:56.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:56.660+0000 7fb784200640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fb77c15f870 con 0x7fb77c147e70 2024-03-31T15:09:56.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:56.660+0000 7fb784200640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fb77c162f70 con 0x7fb77c073510 2024-03-31T15:09:56.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:56.660+0000 7fb784200640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fb77c15f9f0 con 0x7fb77c0d0c50 2024-03-31T15:09:56.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:56.660+0000 7fb782200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb77c147e70 0x7fb77c0b52a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:56.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:56.660+0000 7fb782200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb77c147e70 0x7fb77c0b52a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59640/0 (socket says 172.21.15.16:59640) 2024-03-31T15:09:56.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:56.660+0000 7fb782200640 1 -- 172.21.15.16:0/1091299676 learned_addr learned my addr 172.21.15.16:0/1091299676 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:09:56.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:56.660+0000 7fb781800640 1 --2- 172.21.15.16:0/1091299676 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb77c073510 0x7fb77c15ce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:56.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:56.660+0000 7fb780e00640 1 --2- 172.21.15.16:0/1091299676 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb77c0d0c50 0x7fb77c0b4d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:56.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:56.660+0000 7fb782200640 1 -- 172.21.15.16:0/1091299676 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb77c0d0c50 msgr2=0x7fb77c0b4d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:56.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:56.660+0000 7fb782200640 1 --2- 172.21.15.16:0/1091299676 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb77c0d0c50 0x7fb77c0b4d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:56.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:56.660+0000 7fb782200640 1 -- 172.21.15.16:0/1091299676 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb77c073510 msgr2=0x7fb77c15ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:56.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:56.660+0000 7fb782200640 1 --2- 172.21.15.16:0/1091299676 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb77c073510 0x7fb77c15ce10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:56.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:56.660+0000 7fb782200640 1 -- 172.21.15.16:0/1091299676 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb77c15d710 con 0x7fb77c147e70 2024-03-31T15:09:56.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:56.664+0000 7fb782200640 1 --2- 172.21.15.16:0/1091299676 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb77c147e70 0x7fb77c0b52a0 secure :-1 s=READY pgs=1096 cs=0 l=1 rev1=1 crypto rx=0x7fb7740b0040 tx=0x7fb774019710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:09:56.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:56.664+0000 7fb779600640 1 -- 172.21.15.16:0/1091299676 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7740bd030 con 0x7fb77c147e70 2024-03-31T15:09:56.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:56.664+0000 7fb779600640 1 -- 172.21.15.16:0/1091299676 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb7740b8470 con 0x7fb77c147e70 2024-03-31T15:09:56.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:56.664+0000 7fb784200640 1 -- 172.21.15.16:0/1091299676 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb77c15d9f0 con 0x7fb77c147e70 2024-03-31T15:09:56.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:56.664+0000 7fb779600640 1 -- 172.21.15.16:0/1091299676 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7740b2040 con 0x7fb77c147e70 2024-03-31T15:09:56.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:56.664+0000 7fb784200640 1 -- 172.21.15.16:0/1091299676 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb77c15dfb0 con 0x7fb77c147e70 2024-03-31T15:09:56.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:56.664+0000 7fb784200640 1 -- 172.21.15.16:0/1091299676 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb77c069950 con 0x7fb77c147e70 2024-03-31T15:09:56.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:56.664+0000 7fb779600640 1 -- 172.21.15.16:0/1091299676 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fb7740c0070 con 0x7fb77c147e70 2024-03-31T15:09:56.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:56.668+0000 7fb779600640 1 --2- 172.21.15.16:0/1091299676 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fb75c0683b0 0x7fb75c06a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:56.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:56.668+0000 7fb779600640 1 -- 172.21.15.16:0/1091299676 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 5709+0+0 (secure 0 0 0) 0x7fb7741211f0 con 0x7fb77c147e70 2024-03-31T15:09:56.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:56.668+0000 7fb781800640 1 --2- 172.21.15.16:0/1091299676 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fb75c0683b0 0x7fb75c06a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:56.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:56.668+0000 7fb781800640 1 --2- 172.21.15.16:0/1091299676 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fb75c0683b0 0x7fb75c06a870 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7fb770007fd0 tx=0x7fb770007d60 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:09:56.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:56.672+0000 7fb779600640 1 -- 172.21.15.16:0/1091299676 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fb774016560 con 0x7fb77c147e70 2024-03-31T15:09:56.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:56.868+0000 7fb784200640 1 -- 172.21.15.16:0/1091299676 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb77c0d1bf0 con 0x7fb77c147e70 2024-03-31T15:09:56.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:56.868+0000 7fb779600640 1 -- 172.21.15.16:0/1091299676 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 61 v61) ==== 94+0+5183 (secure 0 0 0) 0x7fb7740f1150 con 0x7fb77c147e70 2024-03-31T15:09:56.863 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:09:56.863 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":61,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6494,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/2445558494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6835","nonce":2445558494},{"type":"v1","addr":"172.21.15.89:6837","nonce":2445558494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":56},{"gid":7270,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/3185913858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6833","nonce":3185913858},{"type":"v1","addr":"172.21.15.16:6835","nonce":3185913858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":56},{"gid":7279,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6836/1599468493","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":1599468493},{"type":"v1","addr":"172.21.15.89:6836","nonce":1599468493}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":56}],"filesystems":[{"mdsmap":{"epoch":61,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-31T15:09:50.926196+0000","modified":"2024-03-31T15:09:54.012516+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7479},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7479":{"gid":7479,"name":"c","rank":0,"incarnation":57,"state":"up:active","state_seq":3,"addr":"172.21.15.16:6834/2478530200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":2478530200},{"type":"v1","addr":"172.21.15.16:6834","nonce":2478530200}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7479,"qdb_cluster":[7479]},"id":6}]} 2024-03-31T15:09:56.863 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 61 2024-03-31T15:09:56.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:56.872+0000 7fb784200640 1 -- 172.21.15.16:0/1091299676 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fb75c0683b0 msgr2=0x7fb75c06a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:56.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:56.872+0000 7fb784200640 1 --2- 172.21.15.16:0/1091299676 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fb75c0683b0 0x7fb75c06a870 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7fb770007fd0 tx=0x7fb770007d60 comp rx=0 tx=0).stop 2024-03-31T15:09:56.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:56.872+0000 7fb784200640 1 -- 172.21.15.16:0/1091299676 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb77c147e70 msgr2=0x7fb77c0b52a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:56.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:56.872+0000 7fb784200640 1 --2- 172.21.15.16:0/1091299676 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb77c147e70 0x7fb77c0b52a0 secure :-1 s=READY pgs=1096 cs=0 l=1 rev1=1 crypto rx=0x7fb7740b0040 tx=0x7fb774019710 comp rx=0 tx=0).stop 2024-03-31T15:09:56.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:56.872+0000 7fb784200640 1 -- 172.21.15.16:0/1091299676 shutdown_connections 2024-03-31T15:09:56.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:56.872+0000 7fb784200640 1 --2- 172.21.15.16:0/1091299676 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fb75c0683b0 0x7fb75c06a870 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:56.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:56.872+0000 7fb784200640 1 --2- 172.21.15.16:0/1091299676 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb77c147e70 0x7fb77c0b52a0 unknown :-1 s=CLOSED pgs=1096 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:56.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:56.872+0000 7fb784200640 1 --2- 172.21.15.16:0/1091299676 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb77c0d0c50 0x7fb77c0b4d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:56.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:56.872+0000 7fb784200640 1 --2- 172.21.15.16:0/1091299676 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb77c073510 0x7fb77c15ce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:56.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:56.872+0000 7fb784200640 1 -- 172.21.15.16:0/1091299676 >> 172.21.15.16:0/1091299676 conn(0x7fb77c0d1060 msgr2=0x7fb77c13d240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:09:56.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:56.872+0000 7fb784200640 1 -- 172.21.15.16:0/1091299676 shutdown_connections 2024-03-31T15:09:56.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:56.872+0000 7fb784200640 1 -- 172.21.15.16:0/1091299676 wait complete. 2024-03-31T15:09:57.295 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 61, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-31T15:09:50.926196+0000', 'modified': '2024-03-31T15:09:54.012516+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7479}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7479': {'gid': 7479, 'name': 'c', 'rank': 0, 'incarnation': 57, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.16:6834/2478530200', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6832', 'nonce': 2478530200}, {'type': 'v1', 'addr': '172.21.15.16:6834', 'nonce': 2478530200}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [13], 'metadata_pool': 12, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 7479, 'qdb_cluster': [7479]} 2024-03-31T15:09:57.296 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-31T15:09:57.296 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-03-31T15:09:57.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.444+0000 7fc315000640 1 Processor -- start 2024-03-31T15:09:57.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.444+0000 7fc315000640 1 -- start start 2024-03-31T15:09:57.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.448+0000 7fc315000640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fc3100b9c60 0x7fc3100b7470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:57.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.448+0000 7fc315000640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fc3100b79b0 0x7fc3100b7e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:57.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.448+0000 7fc315000640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc3100d3990 con 0x7fc3100d1280 2024-03-31T15:09:57.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.448+0000 7fc315000640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7fc3100d26a0 con 0x7fc3100b79b0 2024-03-31T15:09:57.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.448+0000 7fc315000640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fc3100d28f0 con 0x7fc3100b9c60 2024-03-31T15:09:57.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.448+0000 7fc30e800640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fc3100b79b0 0x7fc3100b7e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:57.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.448+0000 7fc30de00640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fc3100d1280 0x7fc3100d1660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:57688/0 (socket says 172.21.15.16:57688) 2024-03-31T15:09:57.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.448+0000 7fc30de00640 1 -- 172.21.15.16:0/1268897904 learned_addr learned my addr 172.21.15.16:0/1268897904 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:09:57.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.448+0000 7fc305400640 1 --2- 172.21.15.16:0/1268897904 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fc3100b9c60 0x7fc3100b7470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:57.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.448+0000 7fc30e800640 1 -- 172.21.15.16:0/1268897904 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fc3100b9c60 msgr2=0x7fc3100b7470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:09:57.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.448+0000 7fc30e800640 1 --2- 172.21.15.16:0/1268897904 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fc3100b9c60 0x7fc3100b7470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:57.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.448+0000 7fc30e800640 1 -- 172.21.15.16:0/1268897904 >> v1:172.21.15.16:6789/0 conn(0x7fc3100d1280 legacy=0x7fc3100d1660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:57.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.448+0000 7fc30e800640 1 -- 172.21.15.16:0/1268897904 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3100d2bd0 con 0x7fc3100b79b0 2024-03-31T15:09:57.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.448+0000 7fc30d400640 1 -- 172.21.15.16:0/1268897904 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 861031049 0 0) 0x7fc3100d3990 con 0x7fc3100d1280 2024-03-31T15:09:57.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.448+0000 7fc305400640 1 --2- 172.21.15.16:0/1268897904 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fc3100b9c60 0x7fc3100b7470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:09:57.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.448+0000 7fc30e800640 1 --2- 172.21.15.16:0/1268897904 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc3100b79b0 0x7fc3100b7e30 secure :-1 s=READY pgs=1098 cs=0 l=1 rev1=1 crypto rx=0x7fc308009d90 tx=0x7fc30801c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fae1b55a050f8f3 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:09:57.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.452+0000 7fc30d400640 1 -- 172.21.15.16:0/1268897904 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3080b2070 con 0x7fc3100b79b0 2024-03-31T15:09:57.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.452+0000 7fc30d400640 1 -- 172.21.15.16:0/1268897904 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc3080a5890 con 0x7fc3100b79b0 2024-03-31T15:09:57.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.452+0000 7fc30d400640 1 -- 172.21.15.16:0/1268897904 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3080a5c60 con 0x7fc3100b79b0 2024-03-31T15:09:57.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.452+0000 7fc315000640 1 -- 172.21.15.16:0/1268897904 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc3100b79b0 msgr2=0x7fc3100b7e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:57.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.452+0000 7fc315000640 1 --2- 172.21.15.16:0/1268897904 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc3100b79b0 0x7fc3100b7e30 secure :-1 s=READY pgs=1098 cs=0 l=1 rev1=1 crypto rx=0x7fc308009d90 tx=0x7fc30801c8e0 comp rx=0 tx=0).stop 2024-03-31T15:09:57.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.452+0000 7fc315000640 1 -- 172.21.15.16:0/1268897904 shutdown_connections 2024-03-31T15:09:57.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.452+0000 7fc315000640 1 --2- 172.21.15.16:0/1268897904 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc3100b79b0 0x7fc3100b7e30 unknown :-1 s=CLOSED pgs=1098 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:57.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.452+0000 7fc315000640 1 --2- 172.21.15.16:0/1268897904 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fc3100b9c60 0x7fc3100b7470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:57.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.452+0000 7fc315000640 1 -- 172.21.15.16:0/1268897904 >> 172.21.15.16:0/1268897904 conn(0x7fc310067730 msgr2=0x7fc310067b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:09:57.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.452+0000 7fc315000640 1 -- 172.21.15.16:0/1268897904 shutdown_connections 2024-03-31T15:09:57.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.452+0000 7fc315000640 1 -- 172.21.15.16:0/1268897904 wait complete. 2024-03-31T15:09:57.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.452+0000 7fc315000640 1 Processor -- start 2024-03-31T15:09:57.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.452+0000 7fc315000640 1 -- start start 2024-03-31T15:09:57.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.452+0000 7fc315000640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc3100b79b0 0x7fc3101589a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:57.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.452+0000 7fc315000640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fc3100b9c60 0x7fc3100b54f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:57.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.452+0000 7fc315000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc3100d1280 0x7fc3100b5a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:57.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.452+0000 7fc315000640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fc3100d4010 con 0x7fc3100d1280 2024-03-31T15:09:57.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.452+0000 7fc315000640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fc3100d3e90 con 0x7fc3100b79b0 2024-03-31T15:09:57.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.452+0000 7fc315000640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fc31016b8a0 con 0x7fc3100b9c60 2024-03-31T15:09:57.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.452+0000 7fc30e800640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc3100d1280 0x7fc3100b5a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:57.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.452+0000 7fc30e800640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc3100d1280 0x7fc3100b5a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39202/0 (socket says 172.21.15.16:39202) 2024-03-31T15:09:57.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.452+0000 7fc30e800640 1 -- 172.21.15.16:0/1169303847 learned_addr learned my addr 172.21.15.16:0/1169303847 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:09:57.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.452+0000 7fc30de00640 1 --2- 172.21.15.16:0/1169303847 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc3100b79b0 0x7fc3101589a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:57.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.452+0000 7fc305400640 1 --2- 172.21.15.16:0/1169303847 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fc3100b9c60 0x7fc3100b54f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:57.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.456+0000 7fc30e800640 1 -- 172.21.15.16:0/1169303847 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fc3100b9c60 msgr2=0x7fc3100b54f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:57.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.456+0000 7fc30e800640 1 --2- 172.21.15.16:0/1169303847 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fc3100b9c60 0x7fc3100b54f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:57.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.456+0000 7fc30e800640 1 -- 172.21.15.16:0/1169303847 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc3100b79b0 msgr2=0x7fc3101589a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:57.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.456+0000 7fc30e800640 1 --2- 172.21.15.16:0/1169303847 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc3100b79b0 0x7fc3101589a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:57.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.456+0000 7fc30e800640 1 -- 172.21.15.16:0/1169303847 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc31015d400 con 0x7fc3100d1280 2024-03-31T15:09:57.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.456+0000 7fc30e800640 1 --2- 172.21.15.16:0/1169303847 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc3100d1280 0x7fc3100b5a30 secure :-1 s=READY pgs=1099 cs=0 l=1 rev1=1 crypto rx=0x7fc308009d90 tx=0x7fc308019680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:09:57.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.456+0000 7fc306a00640 1 -- 172.21.15.16:0/1169303847 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3080b2050 con 0x7fc3100d1280 2024-03-31T15:09:57.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.456+0000 7fc306a00640 1 -- 172.21.15.16:0/1169303847 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc3080b3040 con 0x7fc3100d1280 2024-03-31T15:09:57.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.456+0000 7fc315000640 1 -- 172.21.15.16:0/1169303847 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc31015d6e0 con 0x7fc3100d1280 2024-03-31T15:09:57.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.456+0000 7fc306a00640 1 -- 172.21.15.16:0/1169303847 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3080ad650 con 0x7fc3100d1280 2024-03-31T15:09:57.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.456+0000 7fc315000640 1 -- 172.21.15.16:0/1169303847 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc31015dca0 con 0x7fc3100d1280 2024-03-31T15:09:57.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.456+0000 7fc306a00640 1 -- 172.21.15.16:0/1169303847 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fc3080c0020 con 0x7fc3100d1280 2024-03-31T15:09:57.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.460+0000 7fc3184bd000 1 Processor -- start 2024-03-31T15:09:57.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.460+0000 7fc3184bd000 1 -- start start 2024-03-31T15:09:57.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.460+0000 7fc306a00640 1 --2- 172.21.15.16:0/1169303847 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fc2ec0683b0 0x7fc2ec06a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:57.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.460+0000 7fc306a00640 1 -- 172.21.15.16:0/1169303847 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 5709+0+0 (secure 0 0 0) 0x7fc3081210f0 con 0x7fc3100d1280 2024-03-31T15:09:57.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.460+0000 7fc305400640 1 --2- 172.21.15.16:0/1169303847 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fc2ec0683b0 0x7fc2ec06a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:57.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.460+0000 7fc3184bd000 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5630591e94a0 0x5630591e9880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:57.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.460+0000 7fc3184bd000 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5630591e9dc0 0x56305919cb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:57.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.460+0000 7fc3184bd000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56305919d0b0 0x56305919d490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:57.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.460+0000 7fc3184bd000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x563058fef950 con 0x56305919d0b0 2024-03-31T15:09:57.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.460+0000 7fc3184bd000 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x563058f8c990 con 0x5630591e9dc0 2024-03-31T15:09:57.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.460+0000 7fc3184bd000 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x563058feb900 con 0x5630591e94a0 2024-03-31T15:09:57.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.460+0000 7fc30e800640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56305919d0b0 0x56305919d490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:57.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.460+0000 7fc30e800640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56305919d0b0 0x56305919d490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39204/0 (socket says 172.21.15.16:39204) 2024-03-31T15:09:57.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.460+0000 7fc30e800640 1 -- 172.21.15.16:0/1170295800 learned_addr learned my addr 172.21.15.16:0/1170295800 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:09:57.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.460+0000 7fc30de00640 1 --2- 172.21.15.16:0/1170295800 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5630591e94a0 0x5630591e9880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:57.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.460+0000 7fc305400640 1 --2- 172.21.15.16:0/1170295800 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5630591e9dc0 0x56305919cb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:57.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.460+0000 7fc305400640 1 --2- 172.21.15.16:0/1169303847 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fc2ec0683b0 0x7fc2ec06a870 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7fc31006d030 tx=0x7fc2f8008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:09:57.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.460+0000 7fc30e800640 1 -- 172.21.15.16:0/1170295800 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5630591e94a0 msgr2=0x5630591e9880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:09:57.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.460+0000 7fc30e800640 1 --2- 172.21.15.16:0/1170295800 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5630591e94a0 0x5630591e9880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:57.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.460+0000 7fc30e800640 1 -- 172.21.15.16:0/1170295800 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5630591e9dc0 msgr2=0x56305919cb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:09:57.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.460+0000 7fc30e800640 1 --2- 172.21.15.16:0/1170295800 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5630591e9dc0 0x56305919cb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:57.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.460+0000 7fc30e800640 1 -- 172.21.15.16:0/1170295800 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563059034100 con 0x56305919d0b0 2024-03-31T15:09:57.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.460+0000 7fc30e800640 1 --2- 172.21.15.16:0/1170295800 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56305919d0b0 0x56305919d490 secure :-1 s=READY pgs=1100 cs=0 l=1 rev1=1 crypto rx=0x7fc3080ed4b0 tx=0x7fc3080f09b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72deb62eab9c3303 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:09:57.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.460+0000 7fc2fea00640 1 -- 172.21.15.16:0/1170295800 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3080f9b80 con 0x56305919d0b0 2024-03-31T15:09:57.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.460+0000 7fc2fea00640 1 -- 172.21.15.16:0/1170295800 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc3080ef350 con 0x56305919d0b0 2024-03-31T15:09:57.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.460+0000 7fc2fea00640 1 -- 172.21.15.16:0/1170295800 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3080f96f0 con 0x56305919d0b0 2024-03-31T15:09:57.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.460+0000 7fc3184bd000 1 -- 172.21.15.16:0/1170295800 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56305919d0b0 msgr2=0x56305919d490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:57.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.460+0000 7fc3184bd000 1 --2- 172.21.15.16:0/1170295800 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56305919d0b0 0x56305919d490 secure :-1 s=READY pgs=1100 cs=0 l=1 rev1=1 crypto rx=0x7fc3080ed4b0 tx=0x7fc3080f09b0 comp rx=0 tx=0).stop 2024-03-31T15:09:57.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.460+0000 7fc3184bd000 1 -- 172.21.15.16:0/1170295800 shutdown_connections 2024-03-31T15:09:57.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.460+0000 7fc3184bd000 1 --2- 172.21.15.16:0/1170295800 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56305919d0b0 0x56305919d490 unknown :-1 s=CLOSED pgs=1100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:57.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.460+0000 7fc3184bd000 1 --2- 172.21.15.16:0/1170295800 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5630591e9dc0 0x56305919cb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:57.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.460+0000 7fc3184bd000 1 --2- 172.21.15.16:0/1170295800 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5630591e94a0 0x5630591e9880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:57.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.460+0000 7fc3184bd000 1 -- 172.21.15.16:0/1170295800 >> 172.21.15.16:0/1170295800 conn(0x563059087b80 msgr2=0x5630591cc460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:09:57.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.460+0000 7fc3184bd000 1 -- 172.21.15.16:0/1170295800 shutdown_connections 2024-03-31T15:09:57.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.460+0000 7fc3184bd000 1 -- 172.21.15.16:0/1170295800 wait complete. 2024-03-31T15:09:57.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.460+0000 7fc3184bd000 1 -- start start 2024-03-31T15:09:57.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.464+0000 7fc3184bd000 1 Processor -- start 2024-03-31T15:09:57.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.464+0000 7fc3184bd000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56305919d0b0 0x5630591a9e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:57.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.464+0000 7fc3184bd000 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5630591e94a0 0x5630591aa340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:57.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.464+0000 7fc3184bd000 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5630591e9dc0 0x5630591aa880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:57.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.464+0000 7fc3184bd000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x563058feec60 con 0x56305919d0b0 2024-03-31T15:09:57.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.464+0000 7fc3184bd000 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x563058fad3e0 con 0x5630591e9dc0 2024-03-31T15:09:57.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.464+0000 7fc3184bd000 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x563058fef950 con 0x5630591e94a0 2024-03-31T15:09:57.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.464+0000 7fc3184bd000 10 client.0 _ll_register_callbacks cb 0x5630590500f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T15:09:57.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.464+0000 7fc30de00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56305919d0b0 0x5630591a9e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:57.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.464+0000 7fc30de00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56305919d0b0 0x5630591a9e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39220/0 (socket says 172.21.15.16:39220) 2024-03-31T15:09:57.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.464+0000 7fc30de00640 1 -- 172.21.15.16:0/548746573 learned_addr learned my addr 172.21.15.16:0/548746573 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:09:57.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.464+0000 7fc305400640 1 --2- 172.21.15.16:0/548746573 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5630591e94a0 0x5630591aa340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:57.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.464+0000 7fc30e800640 1 --2- 172.21.15.16:0/548746573 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5630591e9dc0 0x5630591aa880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:57.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.464+0000 7fc30de00640 1 -- 172.21.15.16:0/548746573 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5630591e94a0 msgr2=0x5630591aa340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:09:57.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.464+0000 7fc30de00640 1 --2- 172.21.15.16:0/548746573 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5630591e94a0 0x5630591aa340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:57.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.464+0000 7fc30de00640 1 -- 172.21.15.16:0/548746573 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5630591e9dc0 msgr2=0x5630591aa880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:09:57.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.464+0000 7fc30de00640 1 --2- 172.21.15.16:0/548746573 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5630591e9dc0 0x5630591aa880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:57.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.464+0000 7fc30de00640 1 -- 172.21.15.16:0/548746573 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563058fb98c0 con 0x56305919d0b0 2024-03-31T15:09:57.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.464+0000 7fc30e800640 1 --2- 172.21.15.16:0/548746573 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5630591e9dc0 0x5630591aa880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:09:57.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.464+0000 7fc30de00640 1 --2- 172.21.15.16:0/548746573 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56305919d0b0 0x5630591a9e00 secure :-1 s=READY pgs=1101 cs=0 l=1 rev1=1 crypto rx=0x7fc300004ee0 tx=0x7fc3000016d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b2295462a6238fe9 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:09:57.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.464+0000 7fc2f3400640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-31T15:09:57.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.464+0000 7fc2f3400640 1 -- 172.21.15.16:0/548746573 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3000145b0 con 0x56305919d0b0 2024-03-31T15:09:57.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.464+0000 7fc2f3400640 1 -- 172.21.15.16:0/548746573 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 5709+0+0 (secure 0 0 0) 0x7fc30000c780 con 0x56305919d0b0 2024-03-31T15:09:57.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.464+0000 7fc3184bd000 1 -- 172.21.15.16:0/548746573 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc3080ef350 con 0x56305919d0b0 2024-03-31T15:09:57.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.464+0000 7fc2f3400640 1 -- 172.21.15.16:0/548746573 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=61) ==== 24+0+0 (secure 0 0 0) 0x7fc30000c980 con 0x56305919d0b0 2024-03-31T15:09:57.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.464+0000 7fc3184bd000 10 client.7641 fetch_fsmap learned FSMap version 61 2024-03-31T15:09:57.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.464+0000 7fc3184bd000 1 -- 172.21.15.16:0/548746573 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=61,monmap=2+}) -- 0x563058f77e80 con 0x56305919d0b0 2024-03-31T15:09:57.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.464+0000 7fc2f3400640 1 -- 172.21.15.16:0/548746573 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc3080ef350 con 0x56305919d0b0 2024-03-31T15:09:57.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.464+0000 7fc2f3400640 1 -- 172.21.15.16:0/548746573 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 61) ==== 3494+0+0 (secure 0 0 0) 0x7fc30000cb20 con 0x56305919d0b0 2024-03-31T15:09:57.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.464+0000 7fc3184bd000 10 client.7641 fetch_fsmap finished waiting for FSMap version 61 2024-03-31T15:09:57.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.464+0000 7fc3184bd000 10 client.7641 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-31T15:09:57.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.464+0000 7fc3184bd000 20 client.7641 populate_metadata read hostname 'smithi016' 2024-03-31T15:09:57.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.464+0000 7fc3184bd000 1 --2- 172.21.15.16:0/548746573 >> [v2:172.21.15.16:6832/2478530200,v1:172.21.15.16:6834/2478530200] conn(0x5630591ab0f0 0x5630591fba20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:57.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.464+0000 7fc3184bd000 4 client.7641 mds_command: new command op to 7479 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-31T15:09:57.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.464+0000 7fc3184bd000 1 -- 172.21.15.16:0/548746573 --> [v2:172.21.15.16:6832/2478530200,v1:172.21.15.16:6834/2478530200] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563058f8abc0 con 0x5630591ab0f0 2024-03-31T15:09:57.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.464+0000 7fc305400640 1 --2- 172.21.15.16:0/548746573 >> [v2:172.21.15.16:6832/2478530200,v1:172.21.15.16:6834/2478530200] conn(0x5630591ab0f0 0x5630591fba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:57.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.464+0000 7fc305400640 1 --2- 172.21.15.16:0/548746573 >> [v2:172.21.15.16:6832/2478530200,v1:172.21.15.16:6834/2478530200] conn(0x5630591ab0f0 0x5630591fba20 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f9846785e14350d0 server_cookie=8f041c60f1b83548 in_seq=0 out_seq=0 2024-03-31T15:09:57.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.464+0000 7fc2f3400640 10 client.7641 ms_handle_connect on v2:172.21.15.16:6832/2478530200 2024-03-31T15:09:57.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.468+0000 7fc2f3400640 1 -- 172.21.15.16:0/548746573 <== mds.0 v2:172.21.15.16:6832/2478530200 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x563058f8abc0 con 0x5630591ab0f0 2024-03-31T15:09:57.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.468+0000 7fc2f3400640 10 client.7641 handle_command_reply: tid=0 2024-03-31T15:09:57.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.468+0000 7fc3184bd000 1 client.7641 shutdown 2024-03-31T15:09:57.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.468+0000 7fc3184bd000 1 -- 172.21.15.16:0/548746573 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56305919d0b0 msgr2=0x5630591a9e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:57.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.468+0000 7fc3184bd000 1 --2- 172.21.15.16:0/548746573 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56305919d0b0 0x5630591a9e00 secure :-1 s=READY pgs=1101 cs=0 l=1 rev1=1 crypto rx=0x7fc300004ee0 tx=0x7fc3000016d0 comp rx=0 tx=0).stop 2024-03-31T15:09:57.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.468+0000 7fc3184bd000 1 -- 172.21.15.16:0/548746573 shutdown_connections 2024-03-31T15:09:57.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.468+0000 7fc3184bd000 1 --2- 172.21.15.16:0/548746573 >> [v2:172.21.15.16:6832/2478530200,v1:172.21.15.16:6834/2478530200] conn(0x5630591ab0f0 0x5630591fba20 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:57.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.468+0000 7fc3184bd000 1 --2- 172.21.15.16:0/548746573 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5630591e9dc0 0x5630591aa880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:57.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.468+0000 7fc3184bd000 1 --2- 172.21.15.16:0/548746573 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5630591e94a0 0x5630591aa340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:57.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.468+0000 7fc3184bd000 1 --2- 172.21.15.16:0/548746573 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56305919d0b0 0x5630591a9e00 unknown :-1 s=CLOSED pgs=1101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:57.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.468+0000 7fc3184bd000 1 -- 172.21.15.16:0/548746573 >> 172.21.15.16:0/548746573 conn(0x563059087b80 msgr2=0x56305919c270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:09:57.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.468+0000 7fc2f3400640 0 client.7641 ms_handle_reset on v2:172.21.15.16:6832/2478530200 2024-03-31T15:09:57.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.468+0000 7fc3184bd000 1 -- 172.21.15.16:0/548746573 shutdown_connections 2024-03-31T15:09:57.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.468+0000 7fc3184bd000 1 -- 172.21.15.16:0/548746573 wait complete. 2024-03-31T15:09:57.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.468+0000 7fc3184bd000 20 client.7641 trim_cache size 0 max 16384 2024-03-31T15:09:57.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.484+0000 7fc3184bd000 1 Processor -- start 2024-03-31T15:09:57.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.484+0000 7fc3184bd000 1 -- start start 2024-03-31T15:09:57.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.488+0000 7fc3184bd000 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56305919d0b0 0x5630591e8980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:57.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.488+0000 7fc3184bd000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5630591e94a0 0x5630591e8ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:57.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.488+0000 7fc3184bd000 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5630591e9dc0 0x5630591cb5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:57.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.488+0000 7fc3184bd000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fc310203ba0 con 0x5630591e94a0 2024-03-31T15:09:57.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.488+0000 7fc3184bd000 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fc310202d10 con 0x56305919d0b0 2024-03-31T15:09:57.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.488+0000 7fc3184bd000 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fc3100cd660 con 0x5630591e9dc0 2024-03-31T15:09:57.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.488+0000 7fc305400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5630591e94a0 0x5630591e8ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:57.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.488+0000 7fc305400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5630591e94a0 0x5630591e8ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39228/0 (socket says 172.21.15.16:39228) 2024-03-31T15:09:57.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.488+0000 7fc305400640 1 -- 172.21.15.16:0/3353195627 learned_addr learned my addr 172.21.15.16:0/3353195627 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:09:57.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.488+0000 7fc30de00640 1 --2- 172.21.15.16:0/3353195627 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56305919d0b0 0x5630591e8980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:57.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.488+0000 7fc30e800640 1 --2- 172.21.15.16:0/3353195627 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5630591e9dc0 0x5630591cb5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:57.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.488+0000 7fc305400640 1 -- 172.21.15.16:0/3353195627 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5630591e9dc0 msgr2=0x5630591cb5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:09:57.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.488+0000 7fc305400640 1 --2- 172.21.15.16:0/3353195627 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5630591e9dc0 0x5630591cb5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:57.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.488+0000 7fc305400640 1 -- 172.21.15.16:0/3353195627 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56305919d0b0 msgr2=0x5630591e8980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:09:57.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.488+0000 7fc305400640 1 --2- 172.21.15.16:0/3353195627 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56305919d0b0 0x5630591e8980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:57.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.488+0000 7fc305400640 1 -- 172.21.15.16:0/3353195627 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563058faa490 con 0x5630591e94a0 2024-03-31T15:09:57.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.488+0000 7fc30de00640 1 --2- 172.21.15.16:0/3353195627 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56305919d0b0 0x5630591e8980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:09:57.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.488+0000 7fc305400640 1 --2- 172.21.15.16:0/3353195627 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5630591e94a0 0x5630591e8ec0 secure :-1 s=READY pgs=1102 cs=0 l=1 rev1=1 crypto rx=0x563058ffbb20 tx=0x7fc2f801c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f2f93d4dc6822ce server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:09:57.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.488+0000 7fc2fe000640 1 -- 172.21.15.16:0/3353195627 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2f8032070 con 0x5630591e94a0 2024-03-31T15:09:57.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.488+0000 7fc2fe000640 1 -- 172.21.15.16:0/3353195627 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc2f80170d0 con 0x5630591e94a0 2024-03-31T15:09:57.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.488+0000 7fc2fe000640 1 -- 172.21.15.16:0/3353195627 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2f800f040 con 0x5630591e94a0 2024-03-31T15:09:57.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.488+0000 7fc3184bd000 1 -- 172.21.15.16:0/3353195627 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5630591e94a0 msgr2=0x5630591e8ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:57.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.488+0000 7fc3184bd000 1 --2- 172.21.15.16:0/3353195627 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5630591e94a0 0x5630591e8ec0 secure :-1 s=READY pgs=1102 cs=0 l=1 rev1=1 crypto rx=0x563058ffbb20 tx=0x7fc2f801c000 comp rx=0 tx=0).stop 2024-03-31T15:09:57.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.488+0000 7fc3184bd000 1 -- 172.21.15.16:0/3353195627 shutdown_connections 2024-03-31T15:09:57.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.488+0000 7fc3184bd000 1 --2- 172.21.15.16:0/3353195627 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5630591e9dc0 0x5630591cb5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:57.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.488+0000 7fc3184bd000 1 --2- 172.21.15.16:0/3353195627 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5630591e94a0 0x5630591e8ec0 unknown :-1 s=CLOSED pgs=1102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:57.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.488+0000 7fc3184bd000 1 --2- 172.21.15.16:0/3353195627 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56305919d0b0 0x5630591e8980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:57.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.488+0000 7fc3184bd000 1 -- 172.21.15.16:0/3353195627 >> 172.21.15.16:0/3353195627 conn(0x563059087b80 msgr2=0x5630591f6a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:09:57.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.488+0000 7fc3184bd000 1 -- 172.21.15.16:0/3353195627 shutdown_connections 2024-03-31T15:09:57.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.488+0000 7fc3184bd000 1 -- 172.21.15.16:0/3353195627 wait complete. 2024-03-31T15:09:57.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.488+0000 7fc3184bd000 1 -- start start 2024-03-31T15:09:57.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.488+0000 7fc3184bd000 1 Processor -- start 2024-03-31T15:09:57.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.488+0000 7fc3184bd000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56305919d0b0 0x5630591fed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:57.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.488+0000 7fc3184bd000 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5630591e94a0 0x563059259920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:57.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.488+0000 7fc3184bd000 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5630591e9dc0 0x563059259e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:57.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.488+0000 7fc3184bd000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x563058ffc060 con 0x56305919d0b0 2024-03-31T15:09:57.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.488+0000 7fc3184bd000 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fc3100cbc00 con 0x5630591e94a0 2024-03-31T15:09:57.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.488+0000 7fc3184bd000 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x563058ff3f80 con 0x5630591e9dc0 2024-03-31T15:09:57.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.488+0000 7fc3184bd000 10 client.0 _ll_register_callbacks cb 0x7fc300004b40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T15:09:57.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.488+0000 7fc30de00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56305919d0b0 0x5630591fed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:57.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.492+0000 7fc30de00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56305919d0b0 0x5630591fed90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39230/0 (socket says 172.21.15.16:39230) 2024-03-31T15:09:57.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.492+0000 7fc30de00640 1 -- 172.21.15.16:0/1740863892 learned_addr learned my addr 172.21.15.16:0/1740863892 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:09:57.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.492+0000 7fc30e800640 1 --2- 172.21.15.16:0/1740863892 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5630591e9dc0 0x563059259e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:57.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.492+0000 7fc305400640 1 --2- 172.21.15.16:0/1740863892 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5630591e94a0 0x563059259920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:57.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.492+0000 7fc30de00640 1 -- 172.21.15.16:0/1740863892 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5630591e9dc0 msgr2=0x563059259e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:09:57.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.492+0000 7fc30de00640 1 --2- 172.21.15.16:0/1740863892 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5630591e9dc0 0x563059259e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:57.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.492+0000 7fc30de00640 1 -- 172.21.15.16:0/1740863892 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5630591e94a0 msgr2=0x563059259920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:09:57.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.492+0000 7fc30de00640 1 --2- 172.21.15.16:0/1740863892 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5630591e94a0 0x563059259920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:57.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.492+0000 7fc30de00640 1 -- 172.21.15.16:0/1740863892 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563058fa8e80 con 0x56305919d0b0 2024-03-31T15:09:57.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.492+0000 7fc30de00640 1 --2- 172.21.15.16:0/1740863892 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56305919d0b0 0x5630591fed90 secure :-1 s=READY pgs=1103 cs=0 l=1 rev1=1 crypto rx=0x7fc300004eb0 tx=0x7fc300015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3992cdfdc3d943c7 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:09:57.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.492+0000 7fc2f3400640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-31T15:09:57.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.492+0000 7fc2f3400640 1 -- 172.21.15.16:0/1740863892 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3000145b0 con 0x56305919d0b0 2024-03-31T15:09:57.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.492+0000 7fc2f3400640 1 -- 172.21.15.16:0/1740863892 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 5709+0+0 (secure 0 0 0) 0x7fc30000c340 con 0x56305919d0b0 2024-03-31T15:09:57.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.492+0000 7fc3184bd000 1 -- 172.21.15.16:0/1740863892 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc2f80170d0 con 0x56305919d0b0 2024-03-31T15:09:57.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.492+0000 7fc2f3400640 1 -- 172.21.15.16:0/1740863892 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=61) ==== 24+0+0 (secure 0 0 0) 0x7fc300036050 con 0x56305919d0b0 2024-03-31T15:09:57.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.492+0000 7fc3184bd000 10 client.7647 fetch_fsmap learned FSMap version 61 2024-03-31T15:09:57.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.492+0000 7fc3184bd000 1 -- 172.21.15.16:0/1740863892 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=61,monmap=2+}) -- 0x563058f96270 con 0x56305919d0b0 2024-03-31T15:09:57.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.492+0000 7fc2f3400640 1 -- 172.21.15.16:0/1740863892 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc2f80170d0 con 0x56305919d0b0 2024-03-31T15:09:57.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.492+0000 7fc2f3400640 1 -- 172.21.15.16:0/1740863892 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 61) ==== 3494+0+0 (secure 0 0 0) 0x7fc300039020 con 0x56305919d0b0 2024-03-31T15:09:57.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.492+0000 7fc3184bd000 10 client.7647 fetch_fsmap finished waiting for FSMap version 61 2024-03-31T15:09:57.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.492+0000 7fc3184bd000 10 client.7647 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-31T15:09:57.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.492+0000 7fc3184bd000 20 client.7647 populate_metadata read hostname 'smithi016' 2024-03-31T15:09:57.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.492+0000 7fc3184bd000 1 --2- 172.21.15.16:0/1740863892 >> [v2:172.21.15.16:6832/2478530200,v1:172.21.15.16:6834/2478530200] conn(0x56305925a3a0 0x5630591fd270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:57.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.492+0000 7fc3184bd000 4 client.7647 mds_command: new command op to 7479 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-31T15:09:57.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.492+0000 7fc3184bd000 1 -- 172.21.15.16:0/1740863892 --> [v2:172.21.15.16:6832/2478530200,v1:172.21.15.16:6834/2478530200] -- command(tid 0: {"prefix": "status"}) -- 0x563058ffce90 con 0x56305925a3a0 2024-03-31T15:09:57.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.492+0000 7fc305400640 1 --2- 172.21.15.16:0/1740863892 >> [v2:172.21.15.16:6832/2478530200,v1:172.21.15.16:6834/2478530200] conn(0x56305925a3a0 0x5630591fd270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:57.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.492+0000 7fc305400640 1 --2- 172.21.15.16:0/1740863892 >> [v2:172.21.15.16:6832/2478530200,v1:172.21.15.16:6834/2478530200] conn(0x56305925a3a0 0x5630591fd270 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9758882f0d49d15f server_cookie=e5776575ed72739f in_seq=0 out_seq=0 2024-03-31T15:09:57.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.492+0000 7fc2f3400640 10 client.7647 ms_handle_connect on v2:172.21.15.16:6832/2478530200 2024-03-31T15:09:57.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.492+0000 7fc2f3400640 1 -- 172.21.15.16:0/1740863892 <== mds.0 v2:172.21.15.16:6832/2478530200 1 ==== command_reply(tid 0: 0 ) ==== 8+0+325 (crc 0 0 0) 0x563058ffce90 con 0x56305925a3a0 2024-03-31T15:09:57.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.492+0000 7fc2f3400640 10 client.7647 handle_command_reply: tid=0 2024-03-31T15:09:57.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.492+0000 7fc3184bd000 1 client.7647 shutdown 2024-03-31T15:09:57.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.492+0000 7fc3184bd000 1 -- 172.21.15.16:0/1740863892 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56305919d0b0 msgr2=0x5630591fed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:57.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.492+0000 7fc3184bd000 1 --2- 172.21.15.16:0/1740863892 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56305919d0b0 0x5630591fed90 secure :-1 s=READY pgs=1103 cs=0 l=1 rev1=1 crypto rx=0x7fc300004eb0 tx=0x7fc300015000 comp rx=0 tx=0).stop 2024-03-31T15:09:57.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.492+0000 7fc3184bd000 1 -- 172.21.15.16:0/1740863892 shutdown_connections 2024-03-31T15:09:57.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.492+0000 7fc3184bd000 1 --2- 172.21.15.16:0/1740863892 >> [v2:172.21.15.16:6832/2478530200,v1:172.21.15.16:6834/2478530200] conn(0x56305925a3a0 0x5630591fd270 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:57.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.492+0000 7fc3184bd000 1 --2- 172.21.15.16:0/1740863892 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5630591e9dc0 0x563059259e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:57.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.492+0000 7fc3184bd000 1 --2- 172.21.15.16:0/1740863892 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5630591e94a0 0x563059259920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:57.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.492+0000 7fc3184bd000 1 --2- 172.21.15.16:0/1740863892 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56305919d0b0 0x5630591fed90 unknown :-1 s=CLOSED pgs=1103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:57.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.492+0000 7fc3184bd000 1 -- 172.21.15.16:0/1740863892 >> 172.21.15.16:0/1740863892 conn(0x563059087b80 msgr2=0x5630591cb5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:09:57.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.492+0000 7fc2f3400640 0 client.7647 ms_handle_reset on v2:172.21.15.16:6832/2478530200 2024-03-31T15:09:57.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.492+0000 7fc3184bd000 1 -- 172.21.15.16:0/1740863892 shutdown_connections 2024-03-31T15:09:57.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.492+0000 7fc3184bd000 1 -- 172.21.15.16:0/1740863892 wait complete. 2024-03-31T15:09:57.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.492+0000 7fc3184bd000 20 client.7647 trim_cache size 0 max 16384 2024-03-31T15:09:57.490 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-03-31T15:09:57.490 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_fsid": "286eb5da-fb47-4550-b290-d9519c13b9a5", 2024-03-31T15:09:57.490 INFO:teuthology.orchestra.run.smithi016.stdout: "whoami": 0, 2024-03-31T15:09:57.490 INFO:teuthology.orchestra.run.smithi016.stdout: "id": 7479, 2024-03-31T15:09:57.490 INFO:teuthology.orchestra.run.smithi016.stdout: "want_state": "up:active", 2024-03-31T15:09:57.490 INFO:teuthology.orchestra.run.smithi016.stdout: "state": "up:active", 2024-03-31T15:09:57.490 INFO:teuthology.orchestra.run.smithi016.stdout: "fs_name": "cephfs", 2024-03-31T15:09:57.490 INFO:teuthology.orchestra.run.smithi016.stdout: "rank_uptime": 5.9571416399999997, 2024-03-31T15:09:57.490 INFO:teuthology.orchestra.run.smithi016.stdout: "mdsmap_epoch": 61, 2024-03-31T15:09:57.491 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch": 81, 2024-03-31T15:09:57.491 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch_barrier": 81, 2024-03-31T15:09:57.491 INFO:teuthology.orchestra.run.smithi016.stdout: "uptime": 10.841698005 2024-03-31T15:09:57.491 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-03-31T15:09:57.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.496+0000 7fc315000640 1 -- 172.21.15.16:0/1169303847 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fc2ec0683b0 msgr2=0x7fc2ec06a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:57.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.496+0000 7fc315000640 1 --2- 172.21.15.16:0/1169303847 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fc2ec0683b0 0x7fc2ec06a870 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7fc31006d030 tx=0x7fc2f8008040 comp rx=0 tx=0).stop 2024-03-31T15:09:57.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.496+0000 7fc315000640 1 -- 172.21.15.16:0/1169303847 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc3100d1280 msgr2=0x7fc3100b5a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:57.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.496+0000 7fc315000640 1 --2- 172.21.15.16:0/1169303847 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc3100d1280 0x7fc3100b5a30 secure :-1 s=READY pgs=1099 cs=0 l=1 rev1=1 crypto rx=0x7fc308009d90 tx=0x7fc308019680 comp rx=0 tx=0).stop 2024-03-31T15:09:57.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.496+0000 7fc315000640 1 -- 172.21.15.16:0/1169303847 shutdown_connections 2024-03-31T15:09:57.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.496+0000 7fc315000640 1 --2- 172.21.15.16:0/1169303847 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fc2ec0683b0 0x7fc2ec06a870 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:57.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.496+0000 7fc315000640 1 --2- 172.21.15.16:0/1169303847 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc3100d1280 0x7fc3100b5a30 unknown :-1 s=CLOSED pgs=1099 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:57.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.496+0000 7fc315000640 1 --2- 172.21.15.16:0/1169303847 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fc3100b9c60 0x7fc3100b54f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:57.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.496+0000 7fc315000640 1 --2- 172.21.15.16:0/1169303847 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc3100b79b0 0x7fc3101589a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:57.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.496+0000 7fc315000640 1 -- 172.21.15.16:0/1169303847 >> 172.21.15.16:0/1169303847 conn(0x7fc310067730 msgr2=0x7fc3100c7cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:09:57.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.496+0000 7fc315000640 1 -- 172.21.15.16:0/1169303847 shutdown_connections 2024-03-31T15:09:57.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.496+0000 7fc315000640 1 -- 172.21.15.16:0/1169303847 wait complete. 2024-03-31T15:09:57.501 INFO:tasks.ceph.mds.c.smithi016.stderr:2024-03-31T15:09:57.508+0000 7f9ed8600640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-31T15:09:57.506 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-31T15:09:57.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.656+0000 7fb67a600640 1 Processor -- start 2024-03-31T15:09:57.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.656+0000 7fb67a600640 1 -- start start 2024-03-31T15:09:57.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.656+0000 7fb67a600640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fb67409b170 0x7fb6740ccb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:57.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.656+0000 7fb67a600640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb6740d0800 con 0x7fb674070d50 2024-03-31T15:09:57.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.656+0000 7fb67a600640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb6740d0ae0 con 0x7fb67415e3b0 2024-03-31T15:09:57.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.656+0000 7fb67a600640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7fb67415e790 con 0x7fb67409b170 2024-03-31T15:09:57.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.656+0000 7fb679c00640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fb674070d50 0x7fb6740cd0d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:57696/0 (socket says 172.21.15.16:57696) 2024-03-31T15:09:57.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.656+0000 7fb679c00640 1 -- 172.21.15.16:0/2684819793 learned_addr learned my addr 172.21.15.16:0/2684819793 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:09:57.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.656+0000 7fb673e00640 1 --2- 172.21.15.16:0/2684819793 >> v2:172.21.15.89:3300/0 conn(0x7fb67409b170 0x7fb6740ccb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:57.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.656+0000 7fb673400640 1 -- 172.21.15.16:0/2684819793 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1489826689 0 0) 0x7fb6740d0800 con 0x7fb674070d50 2024-03-31T15:09:57.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.656+0000 7fb673400640 1 -- 172.21.15.16:0/2684819793 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb65c003530 con 0x7fb674070d50 2024-03-31T15:09:57.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.656+0000 7fb673400640 1 -- 172.21.15.16:0/2684819793 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 938739105 0 0) 0x7fb65c003530 con 0x7fb674070d50 2024-03-31T15:09:57.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.656+0000 7fb673400640 1 -- 172.21.15.16:0/2684819793 >> v2:172.21.15.89:3300/0 conn(0x7fb67409b170 msgr2=0x7fb6740ccb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:09:57.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.656+0000 7fb673400640 1 --2- 172.21.15.16:0/2684819793 >> v2:172.21.15.89:3300/0 conn(0x7fb67409b170 0x7fb6740ccb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:57.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.656+0000 7fb673400640 1 -- 172.21.15.16:0/2684819793 >> v1:172.21.15.89:6789/0 conn(0x7fb67415e3b0 legacy=0x7fb67409aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:57.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.656+0000 7fb673400640 1 -- 172.21.15.16:0/2684819793 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6740d0dc0 con 0x7fb674070d50 2024-03-31T15:09:57.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.656+0000 7fb673400640 1 -- 172.21.15.16:0/2684819793 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 214807623 0 0) 0x7fb66c001a00 con 0x7fb674070d50 2024-03-31T15:09:57.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.656+0000 7fb673400640 1 -- 172.21.15.16:0/2684819793 >> v1:172.21.15.16:6789/0 conn(0x7fb674070d50 legacy=0x7fb6740cd0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:57.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.656+0000 7fb673400640 1 --2- 172.21.15.16:0/2684819793 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb65c0039c0 0x7fb65c003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:57.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.656+0000 7fb673400640 1 --2- 172.21.15.16:0/2684819793 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb65c004810 0x7fb65c004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:57.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.656+0000 7fb673400640 1 --2- 172.21.15.16:0/2684819793 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb65c0051d0 0x7fb65c00b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:57.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.656+0000 7fb673400640 1 -- 172.21.15.16:0/2684819793 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fb67415e790 con 0x7fb65c0039c0 2024-03-31T15:09:57.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.656+0000 7fb673400640 1 -- 172.21.15.16:0/2684819793 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fb65c00c070 con 0x7fb65c004810 2024-03-31T15:09:57.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.656+0000 7fb673400640 1 -- 172.21.15.16:0/2684819793 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fb65c00c270 con 0x7fb65c0051d0 2024-03-31T15:09:57.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.656+0000 7fb679200640 1 --2- 172.21.15.16:0/2684819793 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb65c0039c0 0x7fb65c003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:57.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.656+0000 7fb673e00640 1 --2- 172.21.15.16:0/2684819793 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb65c004810 0x7fb65c004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:57.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.656+0000 7fb679c00640 1 --2- 172.21.15.16:0/2684819793 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb65c0051d0 0x7fb65c00b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:57.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.656+0000 7fb673e00640 1 -- 172.21.15.16:0/2684819793 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb65c0051d0 msgr2=0x7fb65c00b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:09:57.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.656+0000 7fb673e00640 1 --2- 172.21.15.16:0/2684819793 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb65c0051d0 0x7fb65c00b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:57.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.656+0000 7fb673e00640 1 -- 172.21.15.16:0/2684819793 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb65c0039c0 msgr2=0x7fb65c003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:09:57.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.656+0000 7fb673e00640 1 --2- 172.21.15.16:0/2684819793 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb65c0039c0 0x7fb65c003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:57.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.656+0000 7fb673e00640 1 -- 172.21.15.16:0/2684819793 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb65c003530 con 0x7fb65c004810 2024-03-31T15:09:57.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.656+0000 7fb679200640 1 --2- 172.21.15.16:0/2684819793 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb65c0039c0 0x7fb65c003de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:09:57.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.656+0000 7fb679c00640 1 -- 172.21.15.16:0/2684819793 reap_dead start 2024-03-31T15:09:57.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.660+0000 7fb673e00640 1 --2- 172.21.15.16:0/2684819793 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb65c004810 0x7fb65c004c90 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7fb660009ae0 tx=0x7fb66001ca70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7851a5ea8f82b8f2 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:09:57.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.660+0000 7fb673400640 1 -- 172.21.15.16:0/2684819793 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6600b2070 con 0x7fb65c004810 2024-03-31T15:09:57.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.660+0000 7fb673400640 1 -- 172.21.15.16:0/2684819793 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb6600a5550 con 0x7fb65c004810 2024-03-31T15:09:57.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.660+0000 7fb673400640 1 -- 172.21.15.16:0/2684819793 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6600096b0 con 0x7fb65c004810 2024-03-31T15:09:57.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.660+0000 7fb67a600640 1 -- 172.21.15.16:0/2684819793 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb65c004810 msgr2=0x7fb65c004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:57.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.660+0000 7fb67a600640 1 --2- 172.21.15.16:0/2684819793 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb65c004810 0x7fb65c004c90 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7fb660009ae0 tx=0x7fb66001ca70 comp rx=0 tx=0).stop 2024-03-31T15:09:57.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.660+0000 7fb67a600640 1 -- 172.21.15.16:0/2684819793 shutdown_connections 2024-03-31T15:09:57.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.660+0000 7fb67a600640 1 --2- 172.21.15.16:0/2684819793 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb65c004810 0x7fb65c004c90 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:57.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.660+0000 7fb67a600640 1 -- 172.21.15.16:0/2684819793 >> 172.21.15.16:0/2684819793 conn(0x7fb6740b39c0 msgr2=0x7fb6740b3dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:09:57.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.660+0000 7fb67a600640 1 -- 172.21.15.16:0/2684819793 shutdown_connections 2024-03-31T15:09:57.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.660+0000 7fb67a600640 1 -- 172.21.15.16:0/2684819793 wait complete. 2024-03-31T15:09:57.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.664+0000 7fb67a600640 1 Processor -- start 2024-03-31T15:09:57.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.664+0000 7fb67a600640 1 -- start start 2024-03-31T15:09:57.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.668+0000 7fb67a600640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb65c004810 0x7fb6741f7380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:57.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.668+0000 7fb67a600640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb67415bf60 0x7fb6740c61f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:57.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.668+0000 7fb67a600640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb6740c7e90 0x7fb6740c8310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:57.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.668+0000 7fb67a600640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fb6740706d0 con 0x7fb65c004810 2024-03-31T15:09:57.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.668+0000 7fb67a600640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fb674070550 con 0x7fb67415bf60 2024-03-31T15:09:57.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.668+0000 7fb67a600640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fb674070850 con 0x7fb6740c7e90 2024-03-31T15:09:57.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.668+0000 7fb679200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb65c004810 0x7fb6741f7380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:57.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.668+0000 7fb679200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb65c004810 0x7fb6741f7380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39252/0 (socket says 172.21.15.16:39252) 2024-03-31T15:09:57.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.668+0000 7fb673e00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb67415bf60 0x7fb6740c61f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:57.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.668+0000 7fb679200640 1 -- 172.21.15.16:0/3152120120 learned_addr learned my addr 172.21.15.16:0/3152120120 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:09:57.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.668+0000 7fb679c00640 1 --2- 172.21.15.16:0/3152120120 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb6740c7e90 0x7fb6740c8310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:57.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.668+0000 7fb679200640 1 -- 172.21.15.16:0/3152120120 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb6740c7e90 msgr2=0x7fb6740c8310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:57.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.668+0000 7fb679200640 1 --2- 172.21.15.16:0/3152120120 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb6740c7e90 0x7fb6740c8310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:57.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.668+0000 7fb679200640 1 -- 172.21.15.16:0/3152120120 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb67415bf60 msgr2=0x7fb6740c61f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:57.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.668+0000 7fb679200640 1 --2- 172.21.15.16:0/3152120120 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb67415bf60 0x7fb6740c61f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:57.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.668+0000 7fb679200640 1 -- 172.21.15.16:0/3152120120 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6740cfdf0 con 0x7fb65c004810 2024-03-31T15:09:57.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.668+0000 7fb673e00640 1 --2- 172.21.15.16:0/3152120120 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb67415bf60 0x7fb6740c61f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:09:57.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.668+0000 7fb679200640 1 --2- 172.21.15.16:0/3152120120 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb65c004810 0x7fb6741f7380 secure :-1 s=READY pgs=1105 cs=0 l=1 rev1=1 crypto rx=0x7fb66800f700 tx=0x7fb66800fbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:09:57.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.668+0000 7fb670c00640 1 -- 172.21.15.16:0/3152120120 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6680013a0 con 0x7fb65c004810 2024-03-31T15:09:57.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.668+0000 7fb670c00640 1 -- 172.21.15.16:0/3152120120 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb668018070 con 0x7fb65c004810 2024-03-31T15:09:57.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.668+0000 7fb670c00640 1 -- 172.21.15.16:0/3152120120 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6680052f0 con 0x7fb65c004810 2024-03-31T15:09:57.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.668+0000 7fb67a600640 1 -- 172.21.15.16:0/3152120120 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb6740c6a40 con 0x7fb65c004810 2024-03-31T15:09:57.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.668+0000 7fb67a600640 1 -- 172.21.15.16:0/3152120120 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb6741f78c0 con 0x7fb65c004810 2024-03-31T15:09:57.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.668+0000 7fb67a600640 1 -- 172.21.15.16:0/3152120120 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb67404f8c0 con 0x7fb65c004810 2024-03-31T15:09:57.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.668+0000 7fb670c00640 1 -- 172.21.15.16:0/3152120120 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fb66800d730 con 0x7fb65c004810 2024-03-31T15:09:57.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.672+0000 7fb670c00640 1 --2- 172.21.15.16:0/3152120120 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fb6540683d0 0x7fb65406a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:57.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.672+0000 7fb670c00640 1 -- 172.21.15.16:0/3152120120 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 5709+0+0 (secure 0 0 0) 0x7fb668086490 con 0x7fb65c004810 2024-03-31T15:09:57.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.672+0000 7fb673e00640 1 --2- 172.21.15.16:0/3152120120 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fb6540683d0 0x7fb65406a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:57.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.672+0000 7fb673e00640 1 --2- 172.21.15.16:0/3152120120 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fb6540683d0 0x7fb65406a890 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7fb6600af000 tx=0x7fb6600046d0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:09:57.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:57.676+0000 7fb670c00640 1 -- 172.21.15.16:0/3152120120 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fb668056570 con 0x7fb65c004810 2024-03-31T15:09:58.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:58.012+0000 7fb67a600640 1 -- 172.21.15.16:0/3152120120 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7fb6740b2900 con 0x7fb65c004810 2024-03-31T15:09:58.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:58.012+0000 7fb670c00640 1 -- 172.21.15.16:0/3152120120 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v61) ==== 53+0+73 (secure 0 0 0) 0x7fb66805a870 con 0x7fb65c004810 2024-03-31T15:09:58.005 INFO:teuthology.orchestra.run.smithi016.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-31T15:09:58.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:58.016+0000 7fb67a600640 1 -- 172.21.15.16:0/3152120120 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fb6540683d0 msgr2=0x7fb65406a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:58.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:58.016+0000 7fb67a600640 1 --2- 172.21.15.16:0/3152120120 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fb6540683d0 0x7fb65406a890 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7fb6600af000 tx=0x7fb6600046d0 comp rx=0 tx=0).stop 2024-03-31T15:09:58.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:58.016+0000 7fb67a600640 1 -- 172.21.15.16:0/3152120120 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb65c004810 msgr2=0x7fb6741f7380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:58.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:58.016+0000 7fb67a600640 1 --2- 172.21.15.16:0/3152120120 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb65c004810 0x7fb6741f7380 secure :-1 s=READY pgs=1105 cs=0 l=1 rev1=1 crypto rx=0x7fb66800f700 tx=0x7fb66800fbc0 comp rx=0 tx=0).stop 2024-03-31T15:09:58.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:58.016+0000 7fb67a600640 1 -- 172.21.15.16:0/3152120120 shutdown_connections 2024-03-31T15:09:58.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:58.016+0000 7fb67a600640 1 --2- 172.21.15.16:0/3152120120 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fb6540683d0 0x7fb65406a890 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:58.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:58.016+0000 7fb67a600640 1 --2- 172.21.15.16:0/3152120120 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb6740c7e90 0x7fb6740c8310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:58.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:58.016+0000 7fb67a600640 1 --2- 172.21.15.16:0/3152120120 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb67415bf60 0x7fb6740c61f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:58.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:58.016+0000 7fb67a600640 1 --2- 172.21.15.16:0/3152120120 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb65c004810 0x7fb6741f7380 unknown :-1 s=CLOSED pgs=1105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:58.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:58.016+0000 7fb67a600640 1 -- 172.21.15.16:0/3152120120 >> 172.21.15.16:0/3152120120 conn(0x7fb6740b39c0 msgr2=0x7fb6740b2b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:09:58.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:58.016+0000 7fb67a600640 1 -- 172.21.15.16:0/3152120120 shutdown_connections 2024-03-31T15:09:58.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:58.016+0000 7fb67a600640 1 -- 172.21.15.16:0/3152120120 wait complete. 2024-03-31T15:09:58.031 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-31T15:09:58.031 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-31T15:09:58.031 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi016.front.sepia.ceph.com 2024-03-31T15:09:58.032 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-31T15:09:58.032 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-31T15:09:58.032 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-31T15:09:58.032 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-31T15:09:58.032 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-31T15:09:58.032 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-31T15:09:58.032 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:09:58.032 DEBUG:teuthology.orchestra.run.smithi016:> ip addr 2024-03-31T15:09:58.042 INFO:teuthology.orchestra.run.smithi016.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-31T15:09:58.042 INFO:teuthology.orchestra.run.smithi016.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-31T15:09:58.042 INFO:teuthology.orchestra.run.smithi016.stdout: inet 127.0.0.1/8 scope host lo 2024-03-31T15:09:58.042 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2024-03-31T15:09:58.042 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 ::1/128 scope host 2024-03-31T15:09:58.042 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2024-03-31T15:09:58.042 INFO:teuthology.orchestra.run.smithi016.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-03-31T15:09:58.042 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:6b:fd:e0 brd ff:ff:ff:ff:ff:ff 2024-03-31T15:09:58.042 INFO:teuthology.orchestra.run.smithi016.stdout: altname enp7s0f0 2024-03-31T15:09:58.042 INFO:teuthology.orchestra.run.smithi016.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-03-31T15:09:58.042 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:6b:fd:e1 brd ff:ff:ff:ff:ff:ff 2024-03-31T15:09:58.043 INFO:teuthology.orchestra.run.smithi016.stdout: altname enp7s0f1 2024-03-31T15:09:58.043 INFO:teuthology.orchestra.run.smithi016.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-31T15:09:58.043 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:bd:15:64 brd ff:ff:ff:ff:ff:ff 2024-03-31T15:09:58.043 INFO:teuthology.orchestra.run.smithi016.stdout: altname enp1s0f0 2024-03-31T15:09:58.043 INFO:teuthology.orchestra.run.smithi016.stdout: inet 172.21.15.16/20 brd 172.21.15.255 scope global dynamic ens1f0 2024-03-31T15:09:58.043 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft 42083sec preferred_lft 42083sec 2024-03-31T15:09:58.043 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 fe80::ec4:7aff:febd:1564/64 scope link 2024-03-31T15:09:58.043 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2024-03-31T15:09:58.043 INFO:teuthology.orchestra.run.smithi016.stdout:5: ens1f1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-03-31T15:09:58.043 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:bd:15:65 brd ff:ff:ff:ff:ff:ff 2024-03-31T15:09:58.043 INFO:teuthology.orchestra.run.smithi016.stdout: altname enp1s0f1 2024-03-31T15:09:58.043 INFO:teuthology.orchestra.run.smithi016.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2024-03-31T15:09:58.043 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 02:42:d4:1f:7d:73 brd ff:ff:ff:ff:ff:ff 2024-03-31T15:09:58.043 INFO:teuthology.orchestra.run.smithi016.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2024-03-31T15:09:58.043 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2024-03-31T15:09:58.044 INFO:teuthology.orchestra.run.smithi016.stdout:7: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-31T15:09:58.044 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 5e:1f:84:6d:79:ae brd ff:ff:ff:ff:ff:ff 2024-03-31T15:09:58.044 INFO:teuthology.orchestra.run.smithi016.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-31T15:09:58.044 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2024-03-31T15:09:58.044 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 fe80::5c1f:84ff:fe6d:79ae/64 scope link 2024-03-31T15:09:58.044 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2024-03-31T15:09:58.044 INFO:teuthology.orchestra.run.smithi016.stdout:8: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-31T15:09:58.044 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0e:5c:f6:1f:82:2f brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-31T15:09:58.044 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 fe80::c5c:f6ff:fe1f:822f/64 scope link 2024-03-31T15:09:58.044 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2024-03-31T15:09:58.044 INFO:teuthology.orchestra.run.smithi016.stdout:9: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-31T15:09:58.044 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether c6:a6:3d:69:d0:db brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-31T15:09:58.044 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 fe80::c4a6:3dff:fe69:d0db/64 scope link 2024-03-31T15:09:58.044 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2024-03-31T15:09:58.045 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:09:58.045 DEBUG:teuthology.orchestra.run.smithi016:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-31T15:09:58.099 INFO:teuthology.orchestra.run.smithi016.stdout:1 2024-03-31T15:09:58.102 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:09:58.102 DEBUG:teuthology.orchestra.run.smithi016:> route 2024-03-31T15:09:58.151 INFO:teuthology.orchestra.run.smithi016.stdout:Kernel IP routing table 2024-03-31T15:09:58.151 INFO:teuthology.orchestra.run.smithi016.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-31T15:09:58.151 INFO:teuthology.orchestra.run.smithi016.stdout:default _gateway 0.0.0.0 UG 0 0 0 ens1f0 2024-03-31T15:09:58.152 INFO:teuthology.orchestra.run.smithi016.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2024-03-31T15:09:58.152 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 0 0 0 ens1f0 2024-03-31T15:09:58.152 INFO:teuthology.orchestra.run.smithi016.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-31T15:09:58.152 DEBUG:teuthology.orchestra.run.smithi016:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-31T15:09:58.152 DEBUG:teuthology.orchestra.run.smithi016:> set -e 2024-03-31T15:09:58.152 DEBUG:teuthology.orchestra.run.smithi016:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-03-31T15:09:58.152 DEBUG:teuthology.orchestra.run.smithi016:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-03-31T15:09:58.152 DEBUG:teuthology.orchestra.run.smithi016:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-03-31T15:09:58.152 DEBUG:teuthology.orchestra.run.smithi016:> ') 2024-03-31T15:09:58.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:58 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-31T15:09:58.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:58 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-31T15:09:58.296 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:09:58.296 DEBUG:teuthology.orchestra.run.smithi016:> ip netns list 2024-03-31T15:09:58.305 INFO:teuthology.orchestra.run.smithi016.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-31T15:09:58.305 INFO:teuthology.orchestra.run.smithi016.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-31T15:09:58.305 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-31T15:09:58.305 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:09:58.305 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip link set brx.0 up 2024-03-31T15:09:58.370 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-03-31T15:09:58.370 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-31T15:09:58.370 DEBUG:teuthology.orchestra.run.smithi016:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-31T15:09:58.418 INFO:teuthology.orchestra.run.smithi016.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-31T15:09:58.418 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-31T15:09:58.418 DEBUG:teuthology.orchestra.run.smithi016:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-31T15:09:58.470 DEBUG:teuthology.orchestra.run.smithi016:> sudo modprobe fuse 2024-03-31T15:09:58.533 DEBUG:teuthology.orchestra.run.smithi016:> cat /proc/self/mounts | awk '{print $2}' 2024-03-31T15:09:58.584 INFO:teuthology.orchestra.run.smithi016.stdout:/sys 2024-03-31T15:09:58.584 INFO:teuthology.orchestra.run.smithi016.stdout:/proc 2024-03-31T15:09:58.584 INFO:teuthology.orchestra.run.smithi016.stdout:/dev 2024-03-31T15:09:58.584 INFO:teuthology.orchestra.run.smithi016.stdout:/dev/pts 2024-03-31T15:09:58.584 INFO:teuthology.orchestra.run.smithi016.stdout:/run 2024-03-31T15:09:58.584 INFO:teuthology.orchestra.run.smithi016.stdout:/ 2024-03-31T15:09:58.584 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/kernel/security 2024-03-31T15:09:58.585 INFO:teuthology.orchestra.run.smithi016.stdout:/dev/shm 2024-03-31T15:09:58.585 INFO:teuthology.orchestra.run.smithi016.stdout:/run/lock 2024-03-31T15:09:58.585 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/cgroup 2024-03-31T15:09:58.585 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/pstore 2024-03-31T15:09:58.586 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/bpf 2024-03-31T15:09:58.586 INFO:teuthology.orchestra.run.smithi016.stdout:/proc/sys/fs/binfmt_misc 2024-03-31T15:09:58.586 INFO:teuthology.orchestra.run.smithi016.stdout:/dev/hugepages 2024-03-31T15:09:58.586 INFO:teuthology.orchestra.run.smithi016.stdout:/dev/mqueue 2024-03-31T15:09:58.586 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/kernel/debug 2024-03-31T15:09:58.586 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/kernel/tracing 2024-03-31T15:09:58.586 INFO:teuthology.orchestra.run.smithi016.stdout:/proc/fs/nfsd 2024-03-31T15:09:58.586 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/fuse/connections 2024-03-31T15:09:58.586 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/kernel/config 2024-03-31T15:09:58.586 INFO:teuthology.orchestra.run.smithi016.stdout:/run/credentials/systemd-sysusers.service 2024-03-31T15:09:58.586 INFO:teuthology.orchestra.run.smithi016.stdout:/run/rpc_pipefs 2024-03-31T15:09:58.587 INFO:teuthology.orchestra.run.smithi016.stdout:/run/user/1000 2024-03-31T15:09:58.587 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/ceph 2024-03-31T15:09:58.587 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/ceph/osd/ceph-0 2024-03-31T15:09:58.587 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/ceph/osd/ceph-1 2024-03-31T15:09:58.587 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/ceph/osd/ceph-2 2024-03-31T15:09:58.587 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/ceph/osd/ceph-3 2024-03-31T15:09:58.587 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns 2024-03-31T15:09:58.587 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-31T15:09:58.587 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-31T15:09:58.587 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-31T15:09:58.587 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-31T15:09:58.588 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:09:58.588 DEBUG:teuthology.orchestra.run.smithi016:> ls /sys/fs/fuse/connections 2024-03-31T15:09:58.638 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-31T15:09:58.638 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-03-31T15:09:58.685 DEBUG:teuthology.orchestra.run.smithi016:> sudo modprobe fuse 2024-03-31T15:09:58.705 DEBUG:teuthology.orchestra.run.smithi016:> cat /proc/self/mounts | awk '{print $2}' 2024-03-31T15:09:58.756 INFO:teuthology.orchestra.run.smithi016.stdout:/sys 2024-03-31T15:09:58.756 INFO:teuthology.orchestra.run.smithi016.stdout:/proc 2024-03-31T15:09:58.756 INFO:teuthology.orchestra.run.smithi016.stdout:/dev 2024-03-31T15:09:58.756 INFO:teuthology.orchestra.run.smithi016.stdout:/dev/pts 2024-03-31T15:09:58.756 INFO:teuthology.orchestra.run.smithi016.stdout:/run 2024-03-31T15:09:58.756 INFO:teuthology.orchestra.run.smithi016.stdout:/ 2024-03-31T15:09:58.756 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/kernel/security 2024-03-31T15:09:58.756 INFO:teuthology.orchestra.run.smithi016.stdout:/dev/shm 2024-03-31T15:09:58.756 INFO:teuthology.orchestra.run.smithi016.stdout:/run/lock 2024-03-31T15:09:58.757 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/cgroup 2024-03-31T15:09:58.757 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/pstore 2024-03-31T15:09:58.757 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/bpf 2024-03-31T15:09:58.757 INFO:teuthology.orchestra.run.smithi016.stdout:/proc/sys/fs/binfmt_misc 2024-03-31T15:09:58.757 INFO:teuthology.orchestra.run.smithi016.stdout:/dev/hugepages 2024-03-31T15:09:58.757 INFO:teuthology.orchestra.run.smithi016.stdout:/dev/mqueue 2024-03-31T15:09:58.757 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/kernel/debug 2024-03-31T15:09:58.757 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/kernel/tracing 2024-03-31T15:09:58.757 INFO:teuthology.orchestra.run.smithi016.stdout:/proc/fs/nfsd 2024-03-31T15:09:58.757 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/fuse/connections 2024-03-31T15:09:58.757 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/kernel/config 2024-03-31T15:09:58.757 INFO:teuthology.orchestra.run.smithi016.stdout:/run/credentials/systemd-sysusers.service 2024-03-31T15:09:58.757 INFO:teuthology.orchestra.run.smithi016.stdout:/run/rpc_pipefs 2024-03-31T15:09:58.757 INFO:teuthology.orchestra.run.smithi016.stdout:/run/user/1000 2024-03-31T15:09:58.758 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/ceph 2024-03-31T15:09:58.758 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/ceph/osd/ceph-0 2024-03-31T15:09:58.758 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/ceph/osd/ceph-1 2024-03-31T15:09:58.758 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/ceph/osd/ceph-2 2024-03-31T15:09:58.758 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/ceph/osd/ceph-3 2024-03-31T15:09:58.758 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns 2024-03-31T15:09:58.758 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-31T15:09:58.758 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-31T15:09:58.758 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-31T15:09:58.758 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-31T15:09:58.759 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:09:58.759 DEBUG:teuthology.orchestra.run.smithi016:> ls /sys/fs/fuse/connections 2024-03-31T15:09:58.780 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi016.stderr:2024-03-31T15:09:58.788+0000 7fa4cb548580 -1 init, newargv = 0x55c078774430 newargc=15 2024-03-31T15:09:58.780 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi016.stderr:ceph-fuse[44750]: starting ceph client 2024-03-31T15:09:58.787 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi016.stderr:ceph-fuse[44750]: starting fuse 2024-03-31T15:09:58.789 INFO:teuthology.orchestra.run.smithi016.stdout:43 2024-03-31T15:09:58.789 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [43] 2024-03-31T15:09:58.790 DEBUG:teuthology.orchestra.run.smithi016:> sudo stdin-killer -- python3 -c ' 2024-03-31T15:09:58.790 DEBUG:teuthology.orchestra.run.smithi016:> import glob 2024-03-31T15:09:58.790 DEBUG:teuthology.orchestra.run.smithi016:> import re 2024-03-31T15:09:58.790 DEBUG:teuthology.orchestra.run.smithi016:> import os 2024-03-31T15:09:58.790 DEBUG:teuthology.orchestra.run.smithi016:> import subprocess 2024-03-31T15:09:58.790 DEBUG:teuthology.orchestra.run.smithi016:> 2024-03-31T15:09:58.790 DEBUG:teuthology.orchestra.run.smithi016:> def _find_admin_socket(client_name): 2024-03-31T15:09:58.790 DEBUG:teuthology.orchestra.run.smithi016:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-03-31T15:09:58.790 DEBUG:teuthology.orchestra.run.smithi016:> files = glob.glob(asok_path) 2024-03-31T15:09:58.790 DEBUG:teuthology.orchestra.run.smithi016:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-03-31T15:09:58.790 DEBUG:teuthology.orchestra.run.smithi016:> 2024-03-31T15:09:58.791 DEBUG:teuthology.orchestra.run.smithi016:> # Given a non-glob path, it better be there 2024-03-31T15:09:58.791 DEBUG:teuthology.orchestra.run.smithi016:> if "*" not in asok_path: 2024-03-31T15:09:58.791 DEBUG:teuthology.orchestra.run.smithi016:> assert(len(files) == 1) 2024-03-31T15:09:58.791 DEBUG:teuthology.orchestra.run.smithi016:> return files[0] 2024-03-31T15:09:58.791 DEBUG:teuthology.orchestra.run.smithi016:> 2024-03-31T15:09:58.791 DEBUG:teuthology.orchestra.run.smithi016:> for f in files: 2024-03-31T15:09:58.791 DEBUG:teuthology.orchestra.run.smithi016:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-31T15:09:58.791 DEBUG:teuthology.orchestra.run.smithi016:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-31T15:09:58.791 DEBUG:teuthology.orchestra.run.smithi016:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-31T15:09:58.791 DEBUG:teuthology.orchestra.run.smithi016:> contents = proc_f.read() 2024-03-31T15:09:58.791 DEBUG:teuthology.orchestra.run.smithi016:> if mountpoint in contents: 2024-03-31T15:09:58.791 DEBUG:teuthology.orchestra.run.smithi016:> return f 2024-03-31T15:09:58.791 DEBUG:teuthology.orchestra.run.smithi016:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-31T15:09:58.791 DEBUG:teuthology.orchestra.run.smithi016:> 2024-03-31T15:09:58.791 DEBUG:teuthology.orchestra.run.smithi016:> print(_find_admin_socket("client.0")) 2024-03-31T15:09:58.791 DEBUG:teuthology.orchestra.run.smithi016:> ' 2024-03-31T15:09:58.922 INFO:teuthology.orchestra.run.smithi016.stdout:/var/run/ceph/ceph-client.0.44750.asok 2024-03-31T15:09:58.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:58 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-31T15:09:58.935 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.44750.asok 2024-03-31T15:09:58.935 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:09:58.935 DEBUG:teuthology.orchestra.run.smithi016:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.44750.asok status 2024-03-31T15:09:59.112 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-03-31T15:09:59.112 INFO:teuthology.orchestra.run.smithi016.stdout: "metadata": { 2024-03-31T15:09:59.112 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph_sha1": "d2271579d5e735cfd6d436b8349c004f06527cb0", 2024-03-31T15:09:59.112 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph_version": "ceph version 19.0.0-2574-gd2271579 (d2271579d5e735cfd6d436b8349c004f06527cb0) squid (dev)", 2024-03-31T15:09:59.112 INFO:teuthology.orchestra.run.smithi016.stdout: "entity_id": "0", 2024-03-31T15:09:59.113 INFO:teuthology.orchestra.run.smithi016.stdout: "hostname": "smithi016", 2024-03-31T15:09:59.113 INFO:teuthology.orchestra.run.smithi016.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-03-31T15:09:59.113 INFO:teuthology.orchestra.run.smithi016.stdout: "pid": "44750", 2024-03-31T15:09:59.113 INFO:teuthology.orchestra.run.smithi016.stdout: "root": "/" 2024-03-31T15:09:59.113 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-31T15:09:59.113 INFO:teuthology.orchestra.run.smithi016.stdout: "dentry_count": 0, 2024-03-31T15:09:59.113 INFO:teuthology.orchestra.run.smithi016.stdout: "dentry_pinned_count": 0, 2024-03-31T15:09:59.113 INFO:teuthology.orchestra.run.smithi016.stdout: "id": 7659, 2024-03-31T15:09:59.113 INFO:teuthology.orchestra.run.smithi016.stdout: "inst": { 2024-03-31T15:09:59.113 INFO:teuthology.orchestra.run.smithi016.stdout: "name": { 2024-03-31T15:09:59.113 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "client", 2024-03-31T15:09:59.113 INFO:teuthology.orchestra.run.smithi016.stdout: "num": 7659 2024-03-31T15:09:59.113 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-31T15:09:59.113 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": { 2024-03-31T15:09:59.114 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-03-31T15:09:59.114 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "192.168.0.1:0", 2024-03-31T15:09:59.114 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 3862954768 2024-03-31T15:09:59.114 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-31T15:09:59.114 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-31T15:09:59.114 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": { 2024-03-31T15:09:59.114 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-03-31T15:09:59.114 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "192.168.0.1:0", 2024-03-31T15:09:59.114 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 3862954768 2024-03-31T15:09:59.114 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-31T15:09:59.114 INFO:teuthology.orchestra.run.smithi016.stdout: "inst_str": "client.7659 192.168.0.1:0/3862954768", 2024-03-31T15:09:59.114 INFO:teuthology.orchestra.run.smithi016.stdout: "addr_str": "192.168.0.1:0/3862954768", 2024-03-31T15:09:59.114 INFO:teuthology.orchestra.run.smithi016.stdout: "inode_count": 1, 2024-03-31T15:09:59.115 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_epoch": 61, 2024-03-31T15:09:59.115 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_epoch": 81, 2024-03-31T15:09:59.115 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_epoch_barrier": 0, 2024-03-31T15:09:59.115 INFO:teuthology.orchestra.run.smithi016.stdout: "blocklisted": false, 2024-03-31T15:09:59.115 INFO:teuthology.orchestra.run.smithi016.stdout: "fs_name": "cephfs" 2024-03-31T15:09:59.115 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-03-31T15:09:59.127 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:09:59.127 DEBUG:teuthology.orchestra.run.smithi016:> stat --file-system '--printf=%T 2024-03-31T15:09:59.127 DEBUG:teuthology.orchestra.run.smithi016:> ' -- /home/ubuntu/cephtest/mnt.0 2024-03-31T15:09:59.136 INFO:teuthology.orchestra.run.smithi016.stdout:fuseblk 2024-03-31T15:09:59.137 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-03-31T15:09:59.137 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:09:59.137 DEBUG:teuthology.orchestra.run.smithi016:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-31T15:09:59.198 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:09:59.198 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir dir) 2024-03-31T15:09:59.253 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:09:59.253 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec touch dir/file) 2024-03-31T15:09:59.307 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:09:59.307 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir testdir) 2024-03-31T15:09:59.365 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:09:59.365 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ln dir/file testdir/hardlink) 2024-03-31T15:09:59.424 DEBUG:teuthology.orchestra.run.smithi016:> stdin-killer -- python3 -c ' 2024-03-31T15:09:59.425 DEBUG:teuthology.orchestra.run.smithi016:> import os 2024-03-31T15:09:59.425 DEBUG:teuthology.orchestra.run.smithi016:> import stat 2024-03-31T15:09:59.425 DEBUG:teuthology.orchestra.run.smithi016:> 2024-03-31T15:09:59.425 DEBUG:teuthology.orchestra.run.smithi016:> print(os.stat("/home/ubuntu/cephtest/mnt.0/dir").st_ino) 2024-03-31T15:09:59.425 DEBUG:teuthology.orchestra.run.smithi016:> ' 2024-03-31T15:09:59.535 INFO:teuthology.orchestra.run.smithi016.stdout:1099511627776 2024-03-31T15:09:59.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:59 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-31T15:09:59.546 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-03-31T15:09:59.546 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-31T15:09:59.595 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-31T15:09:59.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:59.784+0000 7f4d90c00640 1 Processor -- start 2024-03-31T15:09:59.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:59.784+0000 7f4d90c00640 1 -- start start 2024-03-31T15:09:59.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:59.788+0000 7f4d90c00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f4d8c15b3d0 0x7f4d8c0b1cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:59.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:59.788+0000 7f4d90c00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f4d8c15eee0 0x7f4d8c0b2200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:59.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:59.788+0000 7f4d90c00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f4d8c0ce2f0 0x7f4d8c0cc9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:59.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:59.788+0000 7f4d90c00640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f4d8c15b7b0 con 0x7f4d8c0ce2f0 2024-03-31T15:09:59.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:59.788+0000 7f4d90c00640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f4d8c15f3c0 con 0x7f4d8c15eee0 2024-03-31T15:09:59.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:59.788+0000 7f4d90c00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f4d8c15f6a0 con 0x7f4d8c15b3d0 2024-03-31T15:09:59.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:59.788+0000 7f4d8a800640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f4d8c0ce2f0 0x7f4d8c0cc9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:59.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:59.788+0000 7f4d83e00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f4d8c15eee0 0x7f4d8c0b2200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:59.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:59.788+0000 7f4d89e00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f4d8c15b3d0 0x7f4d8c0b1cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:59.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:59.788+0000 7f4d8a800640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f4d8c0ce2f0 0x7f4d8c0cc9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39258/0 (socket says 172.21.15.16:39258) 2024-03-31T15:09:59.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:59.788+0000 7f4d83e00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f4d8c15eee0 0x7f4d8c0b2200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.16:44124/0 (socket says 172.21.15.16:44124) 2024-03-31T15:09:59.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:59.788+0000 7f4d8a800640 1 -- 172.21.15.16:0/1145188129 learned_addr learned my addr 172.21.15.16:0/1145188129 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:09:59.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:59.788+0000 7f4d8a800640 1 -- 172.21.15.16:0/1145188129 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f4d8c15b3d0 msgr2=0x7f4d8c0b1cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:09:59.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:59.788+0000 7f4d8a800640 1 --2- 172.21.15.16:0/1145188129 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f4d8c15b3d0 0x7f4d8c0b1cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:59.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:59.788+0000 7f4d8a800640 1 -- 172.21.15.16:0/1145188129 >> v2:172.21.15.89:3300/0 conn(0x7f4d8c15eee0 msgr2=0x7f4d8c0b2200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:09:59.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:59.788+0000 7f4d8a800640 1 --2- 172.21.15.16:0/1145188129 >> v2:172.21.15.89:3300/0 conn(0x7f4d8c15eee0 0x7f4d8c0b2200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:59.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:59.788+0000 7f4d8a800640 1 -- 172.21.15.16:0/1145188129 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d8c15f980 con 0x7f4d8c0ce2f0 2024-03-31T15:09:59.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:59.788+0000 7f4d8a800640 1 --2- 172.21.15.16:0/1145188129 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4d8c0ce2f0 0x7f4d8c0cc9c0 secure :-1 s=READY pgs=1108 cs=0 l=1 rev1=1 crypto rx=0x7f4d8c0cda90 tx=0x7f4d8401c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ee4c4703ea71dd5 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:09:59.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:59.792+0000 7f4d89400640 1 -- 172.21.15.16:0/1145188129 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d840a5800 con 0x7f4d8c0ce2f0 2024-03-31T15:09:59.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:59.792+0000 7f4d89400640 1 -- 172.21.15.16:0/1145188129 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4d840a59a0 con 0x7f4d8c0ce2f0 2024-03-31T15:09:59.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:59.792+0000 7f4d89400640 1 -- 172.21.15.16:0/1145188129 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d840a5d70 con 0x7f4d8c0ce2f0 2024-03-31T15:09:59.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:59.792+0000 7f4d90c00640 1 -- 172.21.15.16:0/1145188129 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4d8c0ce2f0 msgr2=0x7f4d8c0cc9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:59.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:59.792+0000 7f4d90c00640 1 --2- 172.21.15.16:0/1145188129 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4d8c0ce2f0 0x7f4d8c0cc9c0 secure :-1 s=READY pgs=1108 cs=0 l=1 rev1=1 crypto rx=0x7f4d8c0cda90 tx=0x7f4d8401c9c0 comp rx=0 tx=0).stop 2024-03-31T15:09:59.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:59.792+0000 7f4d90c00640 1 -- 172.21.15.16:0/1145188129 shutdown_connections 2024-03-31T15:09:59.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:59.792+0000 7f4d90c00640 1 --2- 172.21.15.16:0/1145188129 >> v2:172.21.15.89:3300/0 conn(0x7f4d8c15eee0 0x7f4d8c0b2200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:59.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:59.792+0000 7f4d90c00640 1 --2- 172.21.15.16:0/1145188129 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4d8c0ce2f0 0x7f4d8c0cc9c0 unknown :-1 s=CLOSED pgs=1108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:59.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:59.792+0000 7f4d90c00640 1 --2- 172.21.15.16:0/1145188129 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f4d8c15b3d0 0x7f4d8c0b1cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:59.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:59.792+0000 7f4d90c00640 1 -- 172.21.15.16:0/1145188129 >> 172.21.15.16:0/1145188129 conn(0x7f4d8c06b620 msgr2=0x7f4d8c06ba30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:09:59.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:59.792+0000 7f4d90c00640 1 -- 172.21.15.16:0/1145188129 shutdown_connections 2024-03-31T15:09:59.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:59.792+0000 7f4d90c00640 1 -- 172.21.15.16:0/1145188129 wait complete. 2024-03-31T15:09:59.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:59.792+0000 7f4d90c00640 1 Processor -- start 2024-03-31T15:09:59.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:59.792+0000 7f4d90c00640 1 -- start start 2024-03-31T15:09:59.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:59.792+0000 7f4d90c00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4d8c0ce2f0 0x7f4d8c156b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:59.787 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:59.792+0000 7f4d90c00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4d8c15b3d0 0x7f4d8c0b3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:59.787 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:59.792+0000 7f4d90c00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f4d8c15eee0 0x7f4d8c1435d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:59.787 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:59.792+0000 7f4d90c00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f4d8c071e70 con 0x7f4d8c0ce2f0 2024-03-31T15:09:59.787 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:59.792+0000 7f4d90c00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f4d8c071cf0 con 0x7f4d8c15b3d0 2024-03-31T15:09:59.787 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:59.792+0000 7f4d90c00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f4d8c071ff0 con 0x7f4d8c15eee0 2024-03-31T15:09:59.787 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:59.792+0000 7f4d89e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4d8c0ce2f0 0x7f4d8c156b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:59.787 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:59.792+0000 7f4d89e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4d8c0ce2f0 0x7f4d8c156b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39268/0 (socket says 172.21.15.16:39268) 2024-03-31T15:09:59.787 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:59.792+0000 7f4d89e00640 1 -- 172.21.15.16:0/3979056221 learned_addr learned my addr 172.21.15.16:0/3979056221 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:09:59.787 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:59.796+0000 7f4d8a800640 1 --2- 172.21.15.16:0/3979056221 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f4d8c15eee0 0x7f4d8c1435d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:59.787 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:59.796+0000 7f4d83e00640 1 --2- 172.21.15.16:0/3979056221 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4d8c15b3d0 0x7f4d8c0b3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:59.787 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:59.796+0000 7f4d89e00640 1 -- 172.21.15.16:0/3979056221 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f4d8c15eee0 msgr2=0x7f4d8c1435d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:59.787 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:59.796+0000 7f4d89e00640 1 --2- 172.21.15.16:0/3979056221 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f4d8c15eee0 0x7f4d8c1435d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:59.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:59.796+0000 7f4d89e00640 1 -- 172.21.15.16:0/3979056221 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4d8c15b3d0 msgr2=0x7f4d8c0b3750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:59.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:59.796+0000 7f4d89e00640 1 --2- 172.21.15.16:0/3979056221 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4d8c15b3d0 0x7f4d8c0b3750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:59.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:59.796+0000 7f4d89e00640 1 -- 172.21.15.16:0/3979056221 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d8c1570b0 con 0x7f4d8c0ce2f0 2024-03-31T15:09:59.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:59.796+0000 7f4d89e00640 1 --2- 172.21.15.16:0/3979056221 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4d8c0ce2f0 0x7f4d8c156b70 secure :-1 s=READY pgs=1109 cs=0 l=1 rev1=1 crypto rx=0x7f4d7c00e8e0 tx=0x7f4d7c00eda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:09:59.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:59.796+0000 7f4d82000640 1 -- 172.21.15.16:0/3979056221 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d7c017070 con 0x7f4d8c0ce2f0 2024-03-31T15:09:59.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:59.796+0000 7f4d82000640 1 -- 172.21.15.16:0/3979056221 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4d7c00ca00 con 0x7f4d8c0ce2f0 2024-03-31T15:09:59.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:59.796+0000 7f4d90c00640 1 -- 172.21.15.16:0/3979056221 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4d8c144580 con 0x7f4d8c0ce2f0 2024-03-31T15:09:59.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:59.796+0000 7f4d82000640 1 -- 172.21.15.16:0/3979056221 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d7c00cdd0 con 0x7f4d8c0ce2f0 2024-03-31T15:09:59.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:59.796+0000 7f4d90c00640 1 -- 172.21.15.16:0/3979056221 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4d8c144b40 con 0x7f4d8c0ce2f0 2024-03-31T15:09:59.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:59.796+0000 7f4d90c00640 1 -- 172.21.15.16:0/3979056221 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4d8c069950 con 0x7f4d8c0ce2f0 2024-03-31T15:09:59.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:59.796+0000 7f4d82000640 1 -- 172.21.15.16:0/3979056221 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f4d7c01a070 con 0x7f4d8c0ce2f0 2024-03-31T15:09:59.790 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:59.796+0000 7f4d82000640 1 --2- 172.21.15.16:0/3979056221 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f4d68068380 0x7f4d6806a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:09:59.790 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:59.800+0000 7f4d82000640 1 -- 172.21.15.16:0/3979056221 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 5709+0+0 (secure 0 0 0) 0x7f4d7c086a40 con 0x7f4d8c0ce2f0 2024-03-31T15:09:59.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:59.800+0000 7f4d83e00640 1 --2- 172.21.15.16:0/3979056221 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f4d68068380 0x7f4d6806a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:09:59.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:59.800+0000 7f4d83e00640 1 --2- 172.21.15.16:0/3979056221 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f4d68068380 0x7f4d6806a840 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f4d8c157370 tx=0x7f4d74008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:09:59.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:59.808+0000 7f4d82000640 1 -- 172.21.15.16:0/3979056221 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f4d7c056ff0 con 0x7f4d8c0ce2f0 2024-03-31T15:09:59.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:59.972+0000 7f4d90c00640 1 -- 172.21.15.16:0/3979056221 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f4d8c0cdea0 con 0x7f4d8c0ce2f0 2024-03-31T15:09:59.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:59.972+0000 7f4d82000640 1 -- 172.21.15.16:0/3979056221 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) ==== 74+0+16588 (secure 0 0 0) 0x7f4d7c0568f0 con 0x7f4d8c0ce2f0 2024-03-31T15:09:59.965 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:09:59.965 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":81,"fsid":"286eb5da-fb47-4550-b290-d9519c13b9a5","created":"2024-03-31T15:03:11.438952+0000","modified":"2024-03-31T15:09:52.535916+0000","last_up_change":"2024-03-31T15:03:22.885535+0000","last_in_change":"2024-03-31T15:03:13.941667+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-31T15:03:22.550787+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-03-31T15:09:49.322620+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-03-31T15:09:49.877778+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"92a66450-ef19-40d8-ac3d-d760b23b0bda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6801","nonce":932712244}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6803","nonce":932712244}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6807","nonce":932712244}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6805","nonce":932712244}]},"public_addr":"172.21.15.16:6801/932712244","cluster_addr":"172.21.15.16:6803/932712244","heartbeat_back_addr":"172.21.15.16:6807/932712244","heartbeat_front_addr":"172.21.15.16:6805/932712244","state":["exists","up"]},{"osd":1,"uuid":"da9d1f2b-1968-4959-b843-faa231988e92","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6810","nonce":1398737533}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6811","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6813","nonce":1398737533}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6821","nonce":1398737533}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6815","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6816","nonce":1398737533}]},"public_addr":"172.21.15.16:6810/1398737533","cluster_addr":"172.21.15.16:6813/1398737533","heartbeat_back_addr":"172.21.15.16:6821/1398737533","heartbeat_front_addr":"172.21.15.16:6816/1398737533","state":["exists","up"]},{"osd":2,"uuid":"01a73662-9a21-4b1f-bcbb-42b9636a4638","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6819","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6822","nonce":3018526646}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6825","nonce":3018526646}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6831","nonce":3018526646}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6827","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6829","nonce":3018526646}]},"public_addr":"172.21.15.16:6822/3018526646","cluster_addr":"172.21.15.16:6825/3018526646","heartbeat_back_addr":"172.21.15.16:6831/3018526646","heartbeat_front_addr":"172.21.15.16:6829/3018526646","state":["exists","up"]},{"osd":3,"uuid":"c7771fd3-e18e-4500-ac38-23cf10ebcd75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6809","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6812","nonce":3360352974}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6817","nonce":3360352974}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6828","nonce":3360352974}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6823","nonce":3360352974}]},"public_addr":"172.21.15.16:6812/3360352974","cluster_addr":"172.21.15.16:6817/3360352974","heartbeat_back_addr":"172.21.15.16:6828/3360352974","heartbeat_front_addr":"172.21.15.16:6823/3360352974","state":["exists","up"]},{"osd":4,"uuid":"88260730-1482-4fbd-a40f-cb1124e9df4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6801","nonce":1856844856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6803","nonce":1856844856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6807","nonce":1856844856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6805","nonce":1856844856}]},"public_addr":"172.21.15.89:6801/1856844856","cluster_addr":"172.21.15.89:6803/1856844856","heartbeat_back_addr":"172.21.15.89:6807/1856844856","heartbeat_front_addr":"172.21.15.89:6805/1856844856","state":["exists","up"]},{"osd":5,"uuid":"4cb2c00c-fae0-4dd8-9d27-ec4f0093ccf7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6814","nonce":50395527}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6817","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6820","nonce":50395527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6831","nonce":50395527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6823","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6826","nonce":50395527}]},"public_addr":"172.21.15.89:6814/50395527","cluster_addr":"172.21.15.89:6820/50395527","heartbeat_back_addr":"172.21.15.89:6831/50395527","heartbeat_front_addr":"172.21.15.89:6826/50395527","state":["exists","up"]},{"osd":6,"uuid":"81ce0c19-2554-4cae-acbe-66a05e9e9b9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6809","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6812","nonce":3470471752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6815","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6818","nonce":3470471752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6827","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6829","nonce":3470471752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6821","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6824","nonce":3470471752}]},"public_addr":"172.21.15.89:6812/3470471752","cluster_addr":"172.21.15.89:6818/3470471752","heartbeat_back_addr":"172.21.15.89:6829/3470471752","heartbeat_front_addr":"172.21.15.89:6824/3470471752","state":["exists","up"]},{"osd":7,"uuid":"85e4313b-d37d-442e-8042-153bf20a847d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6811","nonce":1796778630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6813","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6816","nonce":1796778630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6825","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6828","nonce":1796778630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6819","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6822","nonce":1796778630}]},"public_addr":"172.21.15.89:6811/1796778630","cluster_addr":"172.21.15.89:6816/1796778630","heartbeat_back_addr":"172.21.15.89:6828/1796778630","heartbeat_front_addr":"172.21.15.89:6822/1796778630","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.439088+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.529970+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.680512+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:21.044161+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.582111+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.536558+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.409522+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.590229+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-31T15:09:59.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:59.976+0000 7f4d90c00640 1 -- 172.21.15.16:0/3979056221 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f4d68068380 msgr2=0x7f4d6806a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:59.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:59.976+0000 7f4d90c00640 1 --2- 172.21.15.16:0/3979056221 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f4d68068380 0x7f4d6806a840 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f4d8c157370 tx=0x7f4d74008040 comp rx=0 tx=0).stop 2024-03-31T15:09:59.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:59.976+0000 7f4d90c00640 1 -- 172.21.15.16:0/3979056221 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4d8c0ce2f0 msgr2=0x7f4d8c156b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:09:59.968 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:59.976+0000 7f4d90c00640 1 --2- 172.21.15.16:0/3979056221 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4d8c0ce2f0 0x7f4d8c156b70 secure :-1 s=READY pgs=1109 cs=0 l=1 rev1=1 crypto rx=0x7f4d7c00e8e0 tx=0x7f4d7c00eda0 comp rx=0 tx=0).stop 2024-03-31T15:09:59.968 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:59.976+0000 7f4d90c00640 1 -- 172.21.15.16:0/3979056221 shutdown_connections 2024-03-31T15:09:59.968 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:59.976+0000 7f4d90c00640 1 --2- 172.21.15.16:0/3979056221 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f4d68068380 0x7f4d6806a840 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:59.968 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:59.976+0000 7f4d90c00640 1 --2- 172.21.15.16:0/3979056221 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f4d8c15eee0 0x7f4d8c1435d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:59.968 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:59.976+0000 7f4d90c00640 1 --2- 172.21.15.16:0/3979056221 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4d8c15b3d0 0x7f4d8c0b3750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:59.968 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:59.976+0000 7f4d90c00640 1 --2- 172.21.15.16:0/3979056221 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4d8c0ce2f0 0x7f4d8c156b70 unknown :-1 s=CLOSED pgs=1109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:09:59.968 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:59.976+0000 7f4d90c00640 1 -- 172.21.15.16:0/3979056221 >> 172.21.15.16:0/3979056221 conn(0x7f4d8c06b620 msgr2=0x7f4d8c158d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:09:59.968 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:59.976+0000 7f4d90c00640 1 -- 172.21.15.16:0/3979056221 shutdown_connections 2024-03-31T15:09:59.968 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:09:59.976+0000 7f4d90c00640 1 -- 172.21.15.16:0/3979056221 wait complete. 2024-03-31T15:09:59.996 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-31T15:03:22.550787+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 12, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-31T15:09:49.322620+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '80', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 13, 'pool_name': 'cephfs_data', 'create_time': '2024-03-31T15:09:49.877778+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '81', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-31T15:09:59.997 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-31T15:10:00.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:00.148+0000 7f3034e00640 1 Processor -- start 2024-03-31T15:10:00.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:00.148+0000 7f3034e00640 1 -- start start 2024-03-31T15:10:00.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:00.148+0000 7f3034e00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f30301411b0 0x7f303006f920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:00.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:00.148+0000 7f3034e00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f3030144340 0x7f303006fe60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:00.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:00.148+0000 7f3034e00640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3030144980 con 0x7f3030141590 2024-03-31T15:10:00.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:00.148+0000 7f3034e00640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f303015f910 con 0x7f3030144340 2024-03-31T15:10:00.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:00.148+0000 7f3034e00640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f3030162910 con 0x7f30301411b0 2024-03-31T15:10:00.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:00.148+0000 7f302e800640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f3030144340 0x7f303006fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:00.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:00.148+0000 7f302e800640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f3030144340 0x7f303006fe60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39270/0 (socket says 172.21.15.16:39270) 2024-03-31T15:10:00.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:00.148+0000 7f302e800640 1 -- 172.21.15.16:0/1121345048 learned_addr learned my addr 172.21.15.16:0/1121345048 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:10:00.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:00.148+0000 7f302de00640 1 --2- 172.21.15.16:0/1121345048 >> v2:172.21.15.89:3300/0 conn(0x7f30301411b0 0x7f303006f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:00.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:00.148+0000 7f302e800640 1 -- 172.21.15.16:0/1121345048 >> v2:172.21.15.89:3300/0 conn(0x7f30301411b0 msgr2=0x7f303006f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:10:00.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:00.148+0000 7f302e800640 1 --2- 172.21.15.16:0/1121345048 >> v2:172.21.15.89:3300/0 conn(0x7f30301411b0 0x7f303006f920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:00.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:00.148+0000 7f302e800640 1 -- 172.21.15.16:0/1121345048 >> v1:172.21.15.89:6789/0 conn(0x7f3030141590 legacy=0x7f3030143c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:10:00.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:00.148+0000 7f302e800640 1 -- 172.21.15.16:0/1121345048 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3030161fe0 con 0x7f3030144340 2024-03-31T15:10:00.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:00.148+0000 7f302de00640 1 --2- 172.21.15.16:0/1121345048 >> v2:172.21.15.89:3300/0 conn(0x7f30301411b0 0x7f303006f920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:10:00.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:00.152+0000 7f302e800640 1 --2- 172.21.15.16:0/1121345048 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3030144340 0x7f303006fe60 secure :-1 s=READY pgs=1110 cs=0 l=1 rev1=1 crypto rx=0x7f30300d3120 tx=0x7f302801c9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf3e0a36072ba89d server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:10:00.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:00.152+0000 7f302d400640 1 -- 172.21.15.16:0/1121345048 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30280a5760 con 0x7f3030144340 2024-03-31T15:10:00.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:00.152+0000 7f302d400640 1 -- 172.21.15.16:0/1121345048 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f30280a5900 con 0x7f3030144340 2024-03-31T15:10:00.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:00.152+0000 7f302d400640 1 -- 172.21.15.16:0/1121345048 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30280a5cd0 con 0x7f3030144340 2024-03-31T15:10:00.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:00.152+0000 7f3034e00640 1 -- 172.21.15.16:0/1121345048 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3030144340 msgr2=0x7f303006fe60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:10:00.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:00.152+0000 7f3034e00640 1 --2- 172.21.15.16:0/1121345048 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3030144340 0x7f303006fe60 secure :-1 s=READY pgs=1110 cs=0 l=1 rev1=1 crypto rx=0x7f30300d3120 tx=0x7f302801c9b0 comp rx=0 tx=0).stop 2024-03-31T15:10:00.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:00.152+0000 7f3034e00640 1 -- 172.21.15.16:0/1121345048 shutdown_connections 2024-03-31T15:10:00.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:00.152+0000 7f3034e00640 1 --2- 172.21.15.16:0/1121345048 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3030144340 0x7f303006fe60 unknown :-1 s=CLOSED pgs=1110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:00.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:00.152+0000 7f3034e00640 1 --2- 172.21.15.16:0/1121345048 >> v2:172.21.15.89:3300/0 conn(0x7f30301411b0 0x7f303006f920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:00.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:00.152+0000 7f3034e00640 1 -- 172.21.15.16:0/1121345048 >> 172.21.15.16:0/1121345048 conn(0x7f3030067930 msgr2=0x7f30300b3ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:10:00.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:00.152+0000 7f3034e00640 1 -- 172.21.15.16:0/1121345048 shutdown_connections 2024-03-31T15:10:00.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:00.152+0000 7f3034e00640 1 -- 172.21.15.16:0/1121345048 wait complete. 2024-03-31T15:10:00.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:00.156+0000 7f3034e00640 1 Processor -- start 2024-03-31T15:10:00.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:00.156+0000 7f3034e00640 1 -- start start 2024-03-31T15:10:00.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:00.156+0000 7f3034e00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f30301411b0 0x7f303006c730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:00.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:00.156+0000 7f3034e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3030141590 0x7f3030142510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:00.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:00.156+0000 7f3034e00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f3030144340 0x7f3030137390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:00.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:00.156+0000 7f3034e00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f303015f290 con 0x7f3030141590 2024-03-31T15:10:00.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:00.156+0000 7f3034e00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f303015f110 con 0x7f30301411b0 2024-03-31T15:10:00.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:00.156+0000 7f3034e00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f303015f410 con 0x7f3030144340 2024-03-31T15:10:00.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:00.156+0000 7f3025400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3030141590 0x7f3030142510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:00.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:00.156+0000 7f302de00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f30301411b0 0x7f303006c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:00.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:00.156+0000 7f3025400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3030141590 0x7f3030142510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39284/0 (socket says 172.21.15.16:39284) 2024-03-31T15:10:00.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:00.156+0000 7f3025400640 1 -- 172.21.15.16:0/4191383782 learned_addr learned my addr 172.21.15.16:0/4191383782 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:10:00.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:00.156+0000 7f302e800640 1 --2- 172.21.15.16:0/4191383782 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f3030144340 0x7f3030137390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:00.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:00.156+0000 7f3025400640 1 -- 172.21.15.16:0/4191383782 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f3030144340 msgr2=0x7f3030137390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:10:00.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:00.156+0000 7f3025400640 1 --2- 172.21.15.16:0/4191383782 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f3030144340 0x7f3030137390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:00.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:00.156+0000 7f3025400640 1 -- 172.21.15.16:0/4191383782 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f30301411b0 msgr2=0x7f303006c730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:10:00.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:00.156+0000 7f3025400640 1 --2- 172.21.15.16:0/4191383782 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f30301411b0 0x7f303006c730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:00.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:00.156+0000 7f3025400640 1 -- 172.21.15.16:0/4191383782 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f303006d0a0 con 0x7f3030141590 2024-03-31T15:10:00.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:00.156+0000 7f302de00640 1 --2- 172.21.15.16:0/4191383782 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f30301411b0 0x7f303006c730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:10:00.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:00.156+0000 7f3025400640 1 --2- 172.21.15.16:0/4191383782 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3030141590 0x7f3030142510 secure :-1 s=READY pgs=1111 cs=0 l=1 rev1=1 crypto rx=0x7f301800c410 tx=0x7f301800c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:10:00.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:00.156+0000 7f3026a00640 1 -- 172.21.15.16:0/4191383782 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3018011790 con 0x7f3030141590 2024-03-31T15:10:00.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:00.156+0000 7f3026a00640 1 -- 172.21.15.16:0/4191383782 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3018017070 con 0x7f3030141590 2024-03-31T15:10:00.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:00.156+0000 7f3034e00640 1 -- 172.21.15.16:0/4191383782 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f303006d380 con 0x7f3030141590 2024-03-31T15:10:00.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:00.156+0000 7f3026a00640 1 -- 172.21.15.16:0/4191383782 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3018011bb0 con 0x7f3030141590 2024-03-31T15:10:00.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:00.156+0000 7f3034e00640 1 -- 172.21.15.16:0/4191383782 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f303006d940 con 0x7f3030141590 2024-03-31T15:10:00.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:00.156+0000 7f3034e00640 1 -- 172.21.15.16:0/4191383782 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3030069950 con 0x7f3030141590 2024-03-31T15:10:00.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:00.160+0000 7f3026a00640 1 -- 172.21.15.16:0/4191383782 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f301801c460 con 0x7f3030141590 2024-03-31T15:10:00.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:00.160+0000 7f3026a00640 1 --2- 172.21.15.16:0/4191383782 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f300c068380 0x7f300c06a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:00.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:00.160+0000 7f3026a00640 1 -- 172.21.15.16:0/4191383782 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 5709+0+0 (secure 0 0 0) 0x7f3018086540 con 0x7f3030141590 2024-03-31T15:10:00.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:00.160+0000 7f302de00640 1 --2- 172.21.15.16:0/4191383782 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f300c068380 0x7f300c06a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:00.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:00.160+0000 7f302de00640 1 --2- 172.21.15.16:0/4191383782 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f300c068380 0x7f300c06a840 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f302000af30 tx=0x7f3020008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:10:00.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:00.164+0000 7f3026a00640 1 -- 172.21.15.16:0/4191383782 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f3018021020 con 0x7f3030141590 2024-03-31T15:10:00.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:00.328+0000 7f3034e00640 1 -- 172.21.15.16:0/4191383782 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f30300b4a20 con 0x7f3030141590 2024-03-31T15:10:00.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:00.328+0000 7f3026a00640 1 -- 172.21.15.16:0/4191383782 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v81) ==== 93+0+10 (secure 0 0 0) 0x7f3018056420 con 0x7f3030141590 2024-03-31T15:10:00.320 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2024-03-31T15:10:00.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:00.328+0000 7f3034e00640 1 -- 172.21.15.16:0/4191383782 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f300c068380 msgr2=0x7f300c06a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:10:00.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:00.328+0000 7f3034e00640 1 --2- 172.21.15.16:0/4191383782 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f300c068380 0x7f300c06a840 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f302000af30 tx=0x7f3020008040 comp rx=0 tx=0).stop 2024-03-31T15:10:00.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:00.328+0000 7f3034e00640 1 -- 172.21.15.16:0/4191383782 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3030141590 msgr2=0x7f3030142510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:10:00.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:00.328+0000 7f3034e00640 1 --2- 172.21.15.16:0/4191383782 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3030141590 0x7f3030142510 secure :-1 s=READY pgs=1111 cs=0 l=1 rev1=1 crypto rx=0x7f301800c410 tx=0x7f301800c8d0 comp rx=0 tx=0).stop 2024-03-31T15:10:00.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:00.328+0000 7f3034e00640 1 -- 172.21.15.16:0/4191383782 shutdown_connections 2024-03-31T15:10:00.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:00.328+0000 7f3034e00640 1 --2- 172.21.15.16:0/4191383782 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f300c068380 0x7f300c06a840 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:00.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:00.328+0000 7f3034e00640 1 --2- 172.21.15.16:0/4191383782 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f3030144340 0x7f3030137390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:00.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:00.328+0000 7f3034e00640 1 --2- 172.21.15.16:0/4191383782 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3030141590 0x7f3030142510 unknown :-1 s=CLOSED pgs=1111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:00.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:00.328+0000 7f3034e00640 1 --2- 172.21.15.16:0/4191383782 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f30301411b0 0x7f303006c730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:00.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:00.328+0000 7f3034e00640 1 -- 172.21.15.16:0/4191383782 >> 172.21.15.16:0/4191383782 conn(0x7f3030067930 msgr2=0x7f30300b4d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:10:00.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:00.328+0000 7f3034e00640 1 -- 172.21.15.16:0/4191383782 shutdown_connections 2024-03-31T15:10:00.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:00.328+0000 7f3034e00640 1 -- 172.21.15.16:0/4191383782 wait complete. 2024-03-31T15:10:00.342 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-31T15:10:00.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:00.492+0000 7f7af3e00640 1 Processor -- start 2024-03-31T15:10:00.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:00.492+0000 7f7af3e00640 1 -- start start 2024-03-31T15:10:00.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:00.496+0000 7f7af3e00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f7af415e140 0x7f7af406b620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:00.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:00.496+0000 7f7af3e00640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7af415e920 con 0x7f7af406bb60 2024-03-31T15:10:00.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:00.496+0000 7f7af3e00640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7af415ec00 con 0x7f7af40ae9a0 2024-03-31T15:10:00.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:00.496+0000 7f7af3e00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f7af40aee30 con 0x7f7af415e140 2024-03-31T15:10:00.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:00.496+0000 7f7aea000640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f7af406bb60 0x7f7af40ae290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:57712/0 (socket says 172.21.15.16:57712) 2024-03-31T15:10:00.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:00.496+0000 7f7aea000640 1 -- 172.21.15.16:0/493207833 learned_addr learned my addr 172.21.15.16:0/493207833 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:10:00.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:00.496+0000 7f7af2a00640 1 --2- 172.21.15.16:0/493207833 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f7af415e140 0x7f7af406b620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:00.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:00.496+0000 7f7af2000640 1 -- 172.21.15.16:0/493207833 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3591186102 0 0) 0x7f7af415e920 con 0x7f7af406bb60 2024-03-31T15:10:00.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:00.496+0000 7f7af2000640 1 -- 172.21.15.16:0/493207833 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7ad8003530 con 0x7f7af406bb60 2024-03-31T15:10:00.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:00.496+0000 7f7af2000640 1 -- 172.21.15.16:0/493207833 <== mon.1 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2642242201 0 0) 0x7f7af415ec00 con 0x7f7af40ae9a0 2024-03-31T15:10:00.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:00.496+0000 7f7af2a00640 1 -- 172.21.15.16:0/493207833 >> v1:172.21.15.89:6789/0 conn(0x7f7af40ae9a0 legacy=0x7f7af4072480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:10:00.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:00.496+0000 7f7af2a00640 1 -- 172.21.15.16:0/493207833 >> v1:172.21.15.16:6789/0 conn(0x7f7af406bb60 legacy=0x7f7af40ae290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:10:00.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:00.496+0000 7f7af2a00640 1 -- 172.21.15.16:0/493207833 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7af415eee0 con 0x7f7af415e140 2024-03-31T15:10:00.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:00.500+0000 7f7af2a00640 1 --2- 172.21.15.16:0/493207833 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f7af415e140 0x7f7af406b620 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f7ae40097f0 tx=0x7f7ae401c4e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=71d4fee16600ceb1 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:10:00.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:00.500+0000 7f7af2000640 1 -- 172.21.15.16:0/493207833 <== mon.2 v2:172.21.15.89:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ae40b0070 con 0x7f7af415e140 2024-03-31T15:10:00.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:00.500+0000 7f7af2000640 1 -- 172.21.15.16:0/493207833 <== mon.2 v2:172.21.15.89:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7ae40ab600 con 0x7f7af415e140 2024-03-31T15:10:00.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:00.500+0000 7f7af2000640 1 -- 172.21.15.16:0/493207833 <== mon.2 v2:172.21.15.89:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ae40ab9b0 con 0x7f7af415e140 2024-03-31T15:10:00.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:00.500+0000 7f7af3e00640 1 -- 172.21.15.16:0/493207833 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f7af415e140 msgr2=0x7f7af406b620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:10:00.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:00.500+0000 7f7af3e00640 1 --2- 172.21.15.16:0/493207833 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f7af415e140 0x7f7af406b620 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f7ae40097f0 tx=0x7f7ae401c4e0 comp rx=0 tx=0).stop 2024-03-31T15:10:00.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:00.500+0000 7f7af3e00640 1 -- 172.21.15.16:0/493207833 shutdown_connections 2024-03-31T15:10:00.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:00.500+0000 7f7af3e00640 1 --2- 172.21.15.16:0/493207833 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f7af415e140 0x7f7af406b620 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:00.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:00.500+0000 7f7af3e00640 1 -- 172.21.15.16:0/493207833 >> 172.21.15.16:0/493207833 conn(0x7f7af40b2b30 msgr2=0x7f7af40b2f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:10:00.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:00.500+0000 7f7af3e00640 1 -- 172.21.15.16:0/493207833 shutdown_connections 2024-03-31T15:10:00.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:00.500+0000 7f7af3e00640 1 -- 172.21.15.16:0/493207833 wait complete. 2024-03-31T15:10:00.494 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:00.500+0000 7f7af3e00640 1 Processor -- start 2024-03-31T15:10:00.494 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:00.500+0000 7f7af3e00640 1 -- start start 2024-03-31T15:10:00.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:00.504+0000 7f7af3e00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7af406bb60 0x7f7af4151ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:00.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:00.504+0000 7f7af3e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7af40ae9a0 0x7f7af40cd850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:00.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:00.504+0000 7f7af3e00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f7af415e140 0x7f7af4153fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:00.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:00.504+0000 7f7af3e00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f7af416c120 con 0x7f7af40ae9a0 2024-03-31T15:10:00.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:00.504+0000 7f7af3e00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f7af416bfa0 con 0x7f7af406bb60 2024-03-31T15:10:00.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:00.504+0000 7f7af3e00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f7af416c2a0 con 0x7f7af415e140 2024-03-31T15:10:00.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:00.504+0000 7f7aea000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7af40ae9a0 0x7f7af40cd850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:00.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:00.504+0000 7f7aea000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7af40ae9a0 0x7f7af40cd850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39286/0 (socket says 172.21.15.16:39286) 2024-03-31T15:10:00.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:00.504+0000 7f7aea000640 1 -- 172.21.15.16:0/15088048 learned_addr learned my addr 172.21.15.16:0/15088048 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:10:00.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:00.504+0000 7f7af2a00640 1 --2- 172.21.15.16:0/15088048 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7af406bb60 0x7f7af4151ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:00.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:00.504+0000 7f7af3400640 1 --2- 172.21.15.16:0/15088048 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f7af415e140 0x7f7af4153fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:00.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:00.504+0000 7f7aea000640 1 -- 172.21.15.16:0/15088048 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f7af415e140 msgr2=0x7f7af4153fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:10:00.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:00.504+0000 7f7aea000640 1 --2- 172.21.15.16:0/15088048 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f7af415e140 0x7f7af4153fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:00.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:00.504+0000 7f7aea000640 1 -- 172.21.15.16:0/15088048 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7af406bb60 msgr2=0x7f7af4151ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:10:00.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:00.504+0000 7f7aea000640 1 --2- 172.21.15.16:0/15088048 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7af406bb60 0x7f7af4151ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:00.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:00.504+0000 7f7aea000640 1 -- 172.21.15.16:0/15088048 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7af40cdd90 con 0x7f7af40ae9a0 2024-03-31T15:10:00.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:00.504+0000 7f7aea000640 1 --2- 172.21.15.16:0/15088048 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7af40ae9a0 0x7f7af40cd850 secure :-1 s=READY pgs=1113 cs=0 l=1 rev1=1 crypto rx=0x7f7adc00c3c0 tx=0x7f7adc00c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:10:00.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:00.504+0000 7f7aeb400640 1 -- 172.21.15.16:0/15088048 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7adc017070 con 0x7f7af40ae9a0 2024-03-31T15:10:00.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:00.504+0000 7f7aeb400640 1 -- 172.21.15.16:0/15088048 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7adc005cf0 con 0x7f7af40ae9a0 2024-03-31T15:10:00.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:00.504+0000 7f7aeb400640 1 -- 172.21.15.16:0/15088048 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7adc0072f0 con 0x7f7af40ae9a0 2024-03-31T15:10:00.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:00.504+0000 7f7af3e00640 1 -- 172.21.15.16:0/15088048 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7af415a800 con 0x7f7af40ae9a0 2024-03-31T15:10:00.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:00.504+0000 7f7af3e00640 1 -- 172.21.15.16:0/15088048 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7af41537c0 con 0x7f7af40ae9a0 2024-03-31T15:10:00.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:00.504+0000 7f7af3e00640 1 -- 172.21.15.16:0/15088048 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7af404f8c0 con 0x7f7af40ae9a0 2024-03-31T15:10:00.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:00.504+0000 7f7aeb400640 1 -- 172.21.15.16:0/15088048 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f7adc01a070 con 0x7f7af40ae9a0 2024-03-31T15:10:00.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:00.508+0000 7f7aeb400640 1 --2- 172.21.15.16:0/15088048 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f7ad0068380 0x7f7ad006a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:00.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:00.508+0000 7f7aeb400640 1 -- 172.21.15.16:0/15088048 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 5709+0+0 (secure 0 0 0) 0x7f7adc085db0 con 0x7f7af40ae9a0 2024-03-31T15:10:00.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:00.508+0000 7f7af2a00640 1 --2- 172.21.15.16:0/15088048 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f7ad0068380 0x7f7ad006a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:00.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:00.508+0000 7f7af2a00640 1 --2- 172.21.15.16:0/15088048 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f7ad0068380 0x7f7ad006a840 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f7ae40ad000 tx=0x7f7ae40a6000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:10:00.508 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:00.516+0000 7f7aeb400640 1 -- 172.21.15.16:0/15088048 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f7adc08a070 con 0x7f7af40ae9a0 2024-03-31T15:10:00.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:00.680+0000 7f7af3e00640 1 -- 172.21.15.16:0/15088048 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f7af406eea0 con 0x7f7af40ae9a0 2024-03-31T15:10:00.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:00.680+0000 7f7aeb400640 1 -- 172.21.15.16:0/15088048 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v81) ==== 104+0+11 (secure 0 0 0) 0x7f7adc055c90 con 0x7f7af40ae9a0 2024-03-31T15:10:00.674 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2024-03-31T15:10:00.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:00.684+0000 7f7af3e00640 1 -- 172.21.15.16:0/15088048 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f7ad0068380 msgr2=0x7f7ad006a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:10:00.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:00.684+0000 7f7af3e00640 1 --2- 172.21.15.16:0/15088048 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f7ad0068380 0x7f7ad006a840 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f7ae40ad000 tx=0x7f7ae40a6000 comp rx=0 tx=0).stop 2024-03-31T15:10:00.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:00.684+0000 7f7af3e00640 1 -- 172.21.15.16:0/15088048 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7af40ae9a0 msgr2=0x7f7af40cd850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:10:00.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:00.684+0000 7f7af3e00640 1 --2- 172.21.15.16:0/15088048 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7af40ae9a0 0x7f7af40cd850 secure :-1 s=READY pgs=1113 cs=0 l=1 rev1=1 crypto rx=0x7f7adc00c3c0 tx=0x7f7adc00c880 comp rx=0 tx=0).stop 2024-03-31T15:10:00.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:00.684+0000 7f7af3e00640 1 -- 172.21.15.16:0/15088048 shutdown_connections 2024-03-31T15:10:00.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:00.684+0000 7f7af3e00640 1 --2- 172.21.15.16:0/15088048 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f7ad0068380 0x7f7ad006a840 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:00.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:00.684+0000 7f7af3e00640 1 --2- 172.21.15.16:0/15088048 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f7af415e140 0x7f7af4153fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:00.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:00.684+0000 7f7af3e00640 1 --2- 172.21.15.16:0/15088048 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7af40ae9a0 0x7f7af40cd850 unknown :-1 s=CLOSED pgs=1113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:00.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:00.684+0000 7f7af3e00640 1 --2- 172.21.15.16:0/15088048 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7af406bb60 0x7f7af4151ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:00.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:00.684+0000 7f7af3e00640 1 -- 172.21.15.16:0/15088048 >> 172.21.15.16:0/15088048 conn(0x7f7af40b2b30 msgr2=0x7f7af4158f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:10:00.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:00.684+0000 7f7af3e00640 1 -- 172.21.15.16:0/15088048 shutdown_connections 2024-03-31T15:10:00.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:00.684+0000 7f7af3e00640 1 -- 172.21.15.16:0/15088048 wait complete. 2024-03-31T15:10:00.695 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-31T15:10:00.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:00.848+0000 7f16fc400640 1 Processor -- start 2024-03-31T15:10:00.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:00.848+0000 7f16fc400640 1 -- start start 2024-03-31T15:10:00.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:00.848+0000 7f16fc400640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f16f41411b0 0x7f16f406f920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:00.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:00.848+0000 7f16fc400640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f16f4141590 0x7f16f406fe60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:00.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:00.848+0000 7f16fc400640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f16f4143c30 0x7f16f41440a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:00.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:00.848+0000 7f16fc400640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f16f4162910 con 0x7f16f4143c30 2024-03-31T15:10:00.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:00.848+0000 7f16fc400640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f16f4144980 con 0x7f16f4141590 2024-03-31T15:10:00.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:00.848+0000 7f16fc400640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f16f4161fe0 con 0x7f16f41411b0 2024-03-31T15:10:00.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:00.848+0000 7f16fa400640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f16f4143c30 0x7f16f41440a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:00.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:00.848+0000 7f16fa400640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f16f4143c30 0x7f16f41440a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39296/0 (socket says 172.21.15.16:39296) 2024-03-31T15:10:00.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:00.848+0000 7f16fa400640 1 -- 172.21.15.16:0/210714981 learned_addr learned my addr 172.21.15.16:0/210714981 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:10:00.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:00.848+0000 7f16f9000640 1 --2- 172.21.15.16:0/210714981 >> v2:172.21.15.89:3300/0 conn(0x7f16f4141590 0x7f16f406fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:00.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:00.848+0000 7f16f9a00640 1 --2- 172.21.15.16:0/210714981 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f16f41411b0 0x7f16f406f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:00.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:00.848+0000 7f16fa400640 1 -- 172.21.15.16:0/210714981 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f16f41411b0 msgr2=0x7f16f406f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:10:00.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:00.848+0000 7f16fa400640 1 --2- 172.21.15.16:0/210714981 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f16f41411b0 0x7f16f406f920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:00.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:00.848+0000 7f16fa400640 1 -- 172.21.15.16:0/210714981 >> v2:172.21.15.89:3300/0 conn(0x7f16f4141590 msgr2=0x7f16f406fe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:10:00.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:00.848+0000 7f16fa400640 1 --2- 172.21.15.16:0/210714981 >> v2:172.21.15.89:3300/0 conn(0x7f16f4141590 0x7f16f406fe60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:00.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:00.848+0000 7f16fa400640 1 -- 172.21.15.16:0/210714981 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16f41622c0 con 0x7f16f4143c30 2024-03-31T15:10:00.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:00.848+0000 7f16f9a00640 1 --2- 172.21.15.16:0/210714981 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f16f41411b0 0x7f16f406f920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:10:00.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:00.852+0000 7f16fa400640 1 --2- 172.21.15.16:0/210714981 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f16f4143c30 0x7f16f41440a0 secure :-1 s=READY pgs=1114 cs=0 l=1 rev1=1 crypto rx=0x7f16f4071f30 tx=0x7f16ec01c4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff37c6445098353a server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:10:00.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:00.852+0000 7f16f3e00640 1 -- 172.21.15.16:0/210714981 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16ec0aa430 con 0x7f16f4143c30 2024-03-31T15:10:00.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:00.852+0000 7f16f3e00640 1 -- 172.21.15.16:0/210714981 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f16ec0af070 con 0x7f16f4143c30 2024-03-31T15:10:00.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:00.852+0000 7f16f3e00640 1 -- 172.21.15.16:0/210714981 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16ec0aa7f0 con 0x7f16f4143c30 2024-03-31T15:10:00.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:00.852+0000 7f16fc400640 1 -- 172.21.15.16:0/210714981 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f16f4143c30 msgr2=0x7f16f41440a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:10:00.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:00.852+0000 7f16fc400640 1 --2- 172.21.15.16:0/210714981 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f16f4143c30 0x7f16f41440a0 secure :-1 s=READY pgs=1114 cs=0 l=1 rev1=1 crypto rx=0x7f16f4071f30 tx=0x7f16ec01c4f0 comp rx=0 tx=0).stop 2024-03-31T15:10:00.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:00.852+0000 7f16fc400640 1 -- 172.21.15.16:0/210714981 shutdown_connections 2024-03-31T15:10:00.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:00.852+0000 7f16fc400640 1 --2- 172.21.15.16:0/210714981 >> v2:172.21.15.89:3300/0 conn(0x7f16f4141590 0x7f16f406fe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:00.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:00.852+0000 7f16fc400640 1 --2- 172.21.15.16:0/210714981 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f16f4143c30 0x7f16f41440a0 unknown :-1 s=CLOSED pgs=1114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:00.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:00.852+0000 7f16fc400640 1 --2- 172.21.15.16:0/210714981 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f16f41411b0 0x7f16f406f920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:00.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:00.852+0000 7f16fc400640 1 -- 172.21.15.16:0/210714981 >> 172.21.15.16:0/210714981 conn(0x7f16f4067930 msgr2=0x7f16f40b3ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:10:00.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:00.852+0000 7f16fc400640 1 -- 172.21.15.16:0/210714981 shutdown_connections 2024-03-31T15:10:00.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:00.852+0000 7f16fc400640 1 -- 172.21.15.16:0/210714981 wait complete. 2024-03-31T15:10:00.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:00.856+0000 7f16fc400640 1 Processor -- start 2024-03-31T15:10:00.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:00.856+0000 7f16fc400640 1 -- start start 2024-03-31T15:10:00.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:00.856+0000 7f16fc400640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f16f41411b0 0x7f16f413ca20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:00.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:00.856+0000 7f16fc400640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f16f4141590 0x7f16f4154060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:00.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:00.856+0000 7f16fc400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f16f4143c30 0x7f16f415e8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:00.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:00.856+0000 7f16fc400640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f16f40717f0 con 0x7f16f4143c30 2024-03-31T15:10:00.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:00.856+0000 7f16fc400640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f16f4071670 con 0x7f16f4141590 2024-03-31T15:10:00.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:00.856+0000 7f16fc400640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f16f4071970 con 0x7f16f41411b0 2024-03-31T15:10:00.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:00.856+0000 7f16fa400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f16f4143c30 0x7f16f415e8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:00.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:00.860+0000 7f16fa400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f16f4143c30 0x7f16f415e8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39302/0 (socket says 172.21.15.16:39302) 2024-03-31T15:10:00.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:00.860+0000 7f16fa400640 1 -- 172.21.15.16:0/1141067147 learned_addr learned my addr 172.21.15.16:0/1141067147 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:10:00.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:00.860+0000 7f16f9000640 1 --2- 172.21.15.16:0/1141067147 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f16f4141590 0x7f16f4154060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:00.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:00.860+0000 7f16f9a00640 1 --2- 172.21.15.16:0/1141067147 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f16f41411b0 0x7f16f413ca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:00.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:00.860+0000 7f16fa400640 1 -- 172.21.15.16:0/1141067147 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f16f41411b0 msgr2=0x7f16f413ca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:10:00.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:00.860+0000 7f16fa400640 1 --2- 172.21.15.16:0/1141067147 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f16f41411b0 0x7f16f413ca20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:00.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:00.860+0000 7f16fa400640 1 -- 172.21.15.16:0/1141067147 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f16f4141590 msgr2=0x7f16f4154060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:10:00.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:00.860+0000 7f16fa400640 1 --2- 172.21.15.16:0/1141067147 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f16f4141590 0x7f16f4154060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:00.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:00.860+0000 7f16fa400640 1 -- 172.21.15.16:0/1141067147 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16f413cf60 con 0x7f16f4143c30 2024-03-31T15:10:00.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:00.860+0000 7f16f9000640 1 --2- 172.21.15.16:0/1141067147 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f16f4141590 0x7f16f4154060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:10:00.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:00.860+0000 7f16fa400640 1 --2- 172.21.15.16:0/1141067147 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f16f4143c30 0x7f16f415e8e0 secure :-1 s=READY pgs=1115 cs=0 l=1 rev1=1 crypto rx=0x7f16ec01cd90 tx=0x7f16ec019290 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:10:00.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:00.860+0000 7f16f1600640 1 -- 172.21.15.16:0/1141067147 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16ec0be610 con 0x7f16f4143c30 2024-03-31T15:10:00.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:00.860+0000 7f16f1600640 1 -- 172.21.15.16:0/1141067147 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f16ec0be7b0 con 0x7f16f4143c30 2024-03-31T15:10:00.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:00.860+0000 7f16fc400640 1 -- 172.21.15.16:0/1141067147 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f16f413df20 con 0x7f16f4143c30 2024-03-31T15:10:00.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:00.860+0000 7f16f1600640 1 -- 172.21.15.16:0/1141067147 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16ec0beba0 con 0x7f16f4143c30 2024-03-31T15:10:00.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:00.860+0000 7f16fc400640 1 -- 172.21.15.16:0/1141067147 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f16f413e4e0 con 0x7f16f4143c30 2024-03-31T15:10:00.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:00.860+0000 7f16fc400640 1 -- 172.21.15.16:0/1141067147 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f16f4069950 con 0x7f16f4143c30 2024-03-31T15:10:00.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:00.860+0000 7f16f1600640 1 -- 172.21.15.16:0/1141067147 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f16ec0bc020 con 0x7f16f4143c30 2024-03-31T15:10:00.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:00.864+0000 7f16f1600640 1 --2- 172.21.15.16:0/1141067147 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f16d4068380 0x7f16d406a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:00.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:00.864+0000 7f16f1600640 1 -- 172.21.15.16:0/1141067147 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 5709+0+0 (secure 0 0 0) 0x7f16ec1212a0 con 0x7f16f4143c30 2024-03-31T15:10:00.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:00.864+0000 7f16f9a00640 1 --2- 172.21.15.16:0/1141067147 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f16d4068380 0x7f16d406a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:00.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:00.864+0000 7f16f9a00640 1 --2- 172.21.15.16:0/1141067147 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f16d4068380 0x7f16d406a840 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f16e80075b0 tx=0x7f16e8007540 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:10:00.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:00.868+0000 7f16f1600640 1 -- 172.21.15.16:0/1141067147 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f16f413cf60 con 0x7f16f4143c30 2024-03-31T15:10:01.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:01.032+0000 7f16fc400640 1 -- 172.21.15.16:0/1141067147 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f16f4142500 con 0x7f16f4143c30 2024-03-31T15:10:01.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:01.032+0000 7f16f1600640 1 -- 172.21.15.16:0/1141067147 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v81) ==== 100+0+11 (secure 0 0 0) 0x7f16ec016110 con 0x7f16f4143c30 2024-03-31T15:10:01.026 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 64 2024-03-31T15:10:01.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:01.036+0000 7f16fc400640 1 -- 172.21.15.16:0/1141067147 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f16d4068380 msgr2=0x7f16d406a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:10:01.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:01.036+0000 7f16fc400640 1 --2- 172.21.15.16:0/1141067147 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f16d4068380 0x7f16d406a840 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f16e80075b0 tx=0x7f16e8007540 comp rx=0 tx=0).stop 2024-03-31T15:10:01.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:01.036+0000 7f16fc400640 1 -- 172.21.15.16:0/1141067147 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f16f4143c30 msgr2=0x7f16f415e8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:10:01.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:01.036+0000 7f16fc400640 1 --2- 172.21.15.16:0/1141067147 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f16f4143c30 0x7f16f415e8e0 secure :-1 s=READY pgs=1115 cs=0 l=1 rev1=1 crypto rx=0x7f16ec01cd90 tx=0x7f16ec019290 comp rx=0 tx=0).stop 2024-03-31T15:10:01.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:01.036+0000 7f16fc400640 1 -- 172.21.15.16:0/1141067147 shutdown_connections 2024-03-31T15:10:01.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:01.036+0000 7f16fc400640 1 --2- 172.21.15.16:0/1141067147 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f16d4068380 0x7f16d406a840 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:01.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:01.036+0000 7f16fc400640 1 --2- 172.21.15.16:0/1141067147 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f16f4143c30 0x7f16f415e8e0 unknown :-1 s=CLOSED pgs=1115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:01.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:01.036+0000 7f16fc400640 1 --2- 172.21.15.16:0/1141067147 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f16f4141590 0x7f16f4154060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:01.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:01.036+0000 7f16fc400640 1 --2- 172.21.15.16:0/1141067147 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f16f41411b0 0x7f16f413ca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:01.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:01.036+0000 7f16fc400640 1 -- 172.21.15.16:0/1141067147 >> 172.21.15.16:0/1141067147 conn(0x7f16f4067930 msgr2=0x7f16f40b2340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:10:01.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:01.036+0000 7f16fc400640 1 -- 172.21.15.16:0/1141067147 shutdown_connections 2024-03-31T15:10:01.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:01.036+0000 7f16fc400640 1 -- 172.21.15.16:0/1141067147 wait complete. 2024-03-31T15:10:01.056 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-31T15:10:01.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:01.204+0000 7ff2b8400640 1 Processor -- start 2024-03-31T15:10:01.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:01.204+0000 7ff2b8400640 1 -- start start 2024-03-31T15:10:01.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:01.208+0000 7ff2b8400640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7ff2b006f920 0x7ff2b006fd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:01.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:01.208+0000 7ff2b8400640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff2b00cc740 0x7ff2b00ccbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:01.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:01.208+0000 7ff2b8400640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff2b0162010 con 0x7ff2b006d490 2024-03-31T15:10:01.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:01.208+0000 7ff2b8400640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7ff2b0162be0 con 0x7ff2b00cc740 2024-03-31T15:10:01.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:01.208+0000 7ff2b8400640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7ff2b0070240 con 0x7ff2b006f920 2024-03-31T15:10:01.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:01.208+0000 7ff2b6400640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff2b00cc740 0x7ff2b00ccbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:01.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:01.208+0000 7ff2b6400640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff2b00cc740 0x7ff2b00ccbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39318/0 (socket says 172.21.15.16:39318) 2024-03-31T15:10:01.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:01.208+0000 7ff2b6400640 1 -- 172.21.15.16:0/3600322209 learned_addr learned my addr 172.21.15.16:0/3600322209 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:10:01.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:01.208+0000 7ff2b5a00640 1 --2- 172.21.15.16:0/3600322209 >> v2:172.21.15.89:3300/0 conn(0x7ff2b006f920 0x7ff2b006fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:01.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:01.208+0000 7ff2b6400640 1 -- 172.21.15.16:0/3600322209 >> v2:172.21.15.89:3300/0 conn(0x7ff2b006f920 msgr2=0x7ff2b006fd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:10:01.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:01.208+0000 7ff2b6400640 1 --2- 172.21.15.16:0/3600322209 >> v2:172.21.15.89:3300/0 conn(0x7ff2b006f920 0x7ff2b006fd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:01.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:01.208+0000 7ff2b6400640 1 -- 172.21.15.16:0/3600322209 >> v1:172.21.15.89:6789/0 conn(0x7ff2b006d490 legacy=0x7ff2b006d8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:10:01.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:01.208+0000 7ff2b6400640 1 -- 172.21.15.16:0/3600322209 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2b01622b0 con 0x7ff2b00cc740 2024-03-31T15:10:01.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:01.212+0000 7ff2b6400640 1 --2- 172.21.15.16:0/3600322209 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff2b00cc740 0x7ff2b00ccbd0 secure :-1 s=READY pgs=1116 cs=0 l=1 rev1=1 crypto rx=0x7ff2b00ced00 tx=0x7ff2a801c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f51ecbb79d29971c server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:10:01.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:01.212+0000 7ff2afe00640 1 -- 172.21.15.16:0/3600322209 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2a80a56c0 con 0x7ff2b00cc740 2024-03-31T15:10:01.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:01.212+0000 7ff2afe00640 1 -- 172.21.15.16:0/3600322209 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff2a80a5860 con 0x7ff2b00cc740 2024-03-31T15:10:01.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:01.212+0000 7ff2afe00640 1 -- 172.21.15.16:0/3600322209 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2a80a5c30 con 0x7ff2b00cc740 2024-03-31T15:10:01.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:01.212+0000 7ff2b8400640 1 -- 172.21.15.16:0/3600322209 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff2b00cc740 msgr2=0x7ff2b00ccbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:10:01.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:01.212+0000 7ff2b8400640 1 --2- 172.21.15.16:0/3600322209 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff2b00cc740 0x7ff2b00ccbd0 secure :-1 s=READY pgs=1116 cs=0 l=1 rev1=1 crypto rx=0x7ff2b00ced00 tx=0x7ff2a801c9f0 comp rx=0 tx=0).stop 2024-03-31T15:10:01.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:01.212+0000 7ff2b8400640 1 -- 172.21.15.16:0/3600322209 shutdown_connections 2024-03-31T15:10:01.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:01.212+0000 7ff2b8400640 1 --2- 172.21.15.16:0/3600322209 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff2b00cc740 0x7ff2b00ccbd0 unknown :-1 s=CLOSED pgs=1116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:01.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:01.212+0000 7ff2b8400640 1 --2- 172.21.15.16:0/3600322209 >> v2:172.21.15.89:3300/0 conn(0x7ff2b006f920 0x7ff2b006fd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:01.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:01.212+0000 7ff2b8400640 1 -- 172.21.15.16:0/3600322209 >> 172.21.15.16:0/3600322209 conn(0x7ff2b0067330 msgr2=0x7ff2b0067740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:10:01.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:01.212+0000 7ff2b8400640 1 -- 172.21.15.16:0/3600322209 shutdown_connections 2024-03-31T15:10:01.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:01.212+0000 7ff2b8400640 1 -- 172.21.15.16:0/3600322209 wait complete. 2024-03-31T15:10:01.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:01.212+0000 7ff2b8400640 1 Processor -- start 2024-03-31T15:10:01.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:01.212+0000 7ff2b8400640 1 -- start start 2024-03-31T15:10:01.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:01.216+0000 7ff2b8400640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff2b006d490 0x7ff2b0160f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:01.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:01.216+0000 7ff2b8400640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff2b006f920 0x7ff2b00b7b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:01.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:01.216+0000 7ff2b8400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff2b00cc740 0x7ff2b00b58b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:01.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:01.216+0000 7ff2b8400640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7ff2b016b530 con 0x7ff2b00cc740 2024-03-31T15:10:01.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:01.216+0000 7ff2b8400640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7ff2b016b3b0 con 0x7ff2b006d490 2024-03-31T15:10:01.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:01.216+0000 7ff2b8400640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7ff2b016b6b0 con 0x7ff2b006f920 2024-03-31T15:10:01.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:01.216+0000 7ff2b6400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff2b00cc740 0x7ff2b00b58b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:01.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:01.216+0000 7ff2b6400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff2b00cc740 0x7ff2b00b58b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39324/0 (socket says 172.21.15.16:39324) 2024-03-31T15:10:01.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:01.216+0000 7ff2b6400640 1 -- 172.21.15.16:0/3061710854 learned_addr learned my addr 172.21.15.16:0/3061710854 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:10:01.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:01.216+0000 7ff2b5000640 1 --2- 172.21.15.16:0/3061710854 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff2b006f920 0x7ff2b00b7b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:01.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:01.216+0000 7ff2b5a00640 1 --2- 172.21.15.16:0/3061710854 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff2b006d490 0x7ff2b0160f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:01.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:01.216+0000 7ff2b6400640 1 -- 172.21.15.16:0/3061710854 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff2b006f920 msgr2=0x7ff2b00b7b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:10:01.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:01.216+0000 7ff2b6400640 1 --2- 172.21.15.16:0/3061710854 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff2b006f920 0x7ff2b00b7b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:01.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:01.216+0000 7ff2b6400640 1 -- 172.21.15.16:0/3061710854 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff2b006d490 msgr2=0x7ff2b0160f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:10:01.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:01.216+0000 7ff2b6400640 1 --2- 172.21.15.16:0/3061710854 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff2b006d490 0x7ff2b0160f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:01.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:01.216+0000 7ff2b6400640 1 -- 172.21.15.16:0/3061710854 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2b00b8080 con 0x7ff2b00cc740 2024-03-31T15:10:01.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:01.216+0000 7ff2b6400640 1 --2- 172.21.15.16:0/3061710854 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff2b00cc740 0x7ff2b00b58b0 secure :-1 s=READY pgs=1117 cs=0 l=1 rev1=1 crypto rx=0x7ff2a80af040 tx=0x7ff2a8019790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:10:01.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:01.216+0000 7ff2ad600640 1 -- 172.21.15.16:0/3061710854 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2a80b8470 con 0x7ff2b00cc740 2024-03-31T15:10:01.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:01.216+0000 7ff2ad600640 1 -- 172.21.15.16:0/3061710854 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff2a80b8650 con 0x7ff2b00cc740 2024-03-31T15:10:01.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:01.216+0000 7ff2ad600640 1 -- 172.21.15.16:0/3061710854 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2a80b8a40 con 0x7ff2b00cc740 2024-03-31T15:10:01.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:01.216+0000 7ff2b8400640 1 -- 172.21.15.16:0/3061710854 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff2b00b62b0 con 0x7ff2b00cc740 2024-03-31T15:10:01.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:01.216+0000 7ff2b8400640 1 -- 172.21.15.16:0/3061710854 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff2b01459d0 con 0x7ff2b00cc740 2024-03-31T15:10:01.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:01.216+0000 7ff2b8400640 1 -- 172.21.15.16:0/3061710854 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff2b0069950 con 0x7ff2b00cc740 2024-03-31T15:10:01.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:01.220+0000 7ff2ad600640 1 -- 172.21.15.16:0/3061710854 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7ff2a80c0020 con 0x7ff2b00cc740 2024-03-31T15:10:01.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:01.220+0000 7ff2ad600640 1 --2- 172.21.15.16:0/3061710854 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7ff290068380 0x7ff29006a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:01.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:01.220+0000 7ff2ad600640 1 -- 172.21.15.16:0/3061710854 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 5709+0+0 (secure 0 0 0) 0x7ff2a81213f0 con 0x7ff2b00cc740 2024-03-31T15:10:01.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:01.220+0000 7ff2b5a00640 1 --2- 172.21.15.16:0/3061710854 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7ff290068380 0x7ff29006a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:01.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:01.220+0000 7ff2b5a00640 1 --2- 172.21.15.16:0/3061710854 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7ff290068380 0x7ff29006a840 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7ff2a4009ac0 tx=0x7ff2a4009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:10:01.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:01.224+0000 7ff2ad600640 1 -- 172.21.15.16:0/3061710854 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7ff2a80165e0 con 0x7ff2b00cc740 2024-03-31T15:10:01.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:01.400+0000 7ff2b8400640 1 -- 172.21.15.16:0/3061710854 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7ff2b00cf3f0 con 0x7ff2b00cc740 2024-03-31T15:10:01.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:01.400+0000 7ff2ad600640 1 -- 172.21.15.16:0/3061710854 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v81) ==== 80+0+0 (secure 0 0 0) 0x7ff2a80a5830 con 0x7ff2b00cc740 2024-03-31T15:10:01.394 INFO:teuthology.orchestra.run.smithi016.stderr:listed 0 entries 2024-03-31T15:10:01.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:01.404+0000 7ff2b8400640 1 -- 172.21.15.16:0/3061710854 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7ff290068380 msgr2=0x7ff29006a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:10:01.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:01.404+0000 7ff2b8400640 1 --2- 172.21.15.16:0/3061710854 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7ff290068380 0x7ff29006a840 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7ff2a4009ac0 tx=0x7ff2a4009190 comp rx=0 tx=0).stop 2024-03-31T15:10:01.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:01.404+0000 7ff2b8400640 1 -- 172.21.15.16:0/3061710854 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff2b00cc740 msgr2=0x7ff2b00b58b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:10:01.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:01.404+0000 7ff2b8400640 1 --2- 172.21.15.16:0/3061710854 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff2b00cc740 0x7ff2b00b58b0 secure :-1 s=READY pgs=1117 cs=0 l=1 rev1=1 crypto rx=0x7ff2a80af040 tx=0x7ff2a8019790 comp rx=0 tx=0).stop 2024-03-31T15:10:01.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:01.404+0000 7ff2b8400640 1 -- 172.21.15.16:0/3061710854 shutdown_connections 2024-03-31T15:10:01.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:01.404+0000 7ff2b8400640 1 --2- 172.21.15.16:0/3061710854 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7ff290068380 0x7ff29006a840 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:01.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:01.404+0000 7ff2b8400640 1 --2- 172.21.15.16:0/3061710854 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff2b00cc740 0x7ff2b00b58b0 unknown :-1 s=CLOSED pgs=1117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:01.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:01.404+0000 7ff2b8400640 1 --2- 172.21.15.16:0/3061710854 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff2b006f920 0x7ff2b00b7b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:01.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:01.404+0000 7ff2b8400640 1 --2- 172.21.15.16:0/3061710854 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff2b006d490 0x7ff2b0160f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:01.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:01.404+0000 7ff2b8400640 1 -- 172.21.15.16:0/3061710854 >> 172.21.15.16:0/3061710854 conn(0x7ff2b0067330 msgr2=0x7ff2b00cbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:10:01.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:01.404+0000 7ff2b8400640 1 -- 172.21.15.16:0/3061710854 shutdown_connections 2024-03-31T15:10:01.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:01.404+0000 7ff2b8400640 1 -- 172.21.15.16:0/3061710854 wait complete. 2024-03-31T15:10:01.417 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-03-31T15:10:01.417 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-31T15:10:01.426 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-31T15:10:01.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:01.616+0000 7f8145e00640 1 Processor -- start 2024-03-31T15:10:01.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:01.616+0000 7f8145e00640 1 -- start start 2024-03-31T15:10:01.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:01.616+0000 7f8145e00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f8140070030 0x7f814006c730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:01.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:01.616+0000 7f8145e00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f8140147dd0 0x7f81400cc8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:01.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:01.616+0000 7f8145e00640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8140147660 con 0x7f8140147280 2024-03-31T15:10:01.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:01.616+0000 7f8145e00640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f81401620f0 con 0x7f8140147dd0 2024-03-31T15:10:01.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:01.616+0000 7f8145e00640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f8140162340 con 0x7f8140070030 2024-03-31T15:10:01.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:01.616+0000 7f813f400640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f8140147dd0 0x7f81400cc8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:01.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:01.616+0000 7f813ea00640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f8140147280 0x7f814006f920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:57718/0 (socket says 172.21.15.16:57718) 2024-03-31T15:10:01.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:01.616+0000 7f813ea00640 1 -- 172.21.15.16:0/2374227899 learned_addr learned my addr 172.21.15.16:0/2374227899 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:10:01.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:01.616+0000 7f813f400640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f8140147dd0 0x7f81400cc8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39334/0 (socket says 172.21.15.16:39334) 2024-03-31T15:10:01.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:01.616+0000 7f8136000640 1 --2- 172.21.15.16:0/2374227899 >> v2:172.21.15.89:3300/0 conn(0x7f8140070030 0x7f814006c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:01.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:01.616+0000 7f813f400640 1 -- 172.21.15.16:0/2374227899 >> v2:172.21.15.89:3300/0 conn(0x7f8140070030 msgr2=0x7f814006c730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:10:01.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:01.616+0000 7f813f400640 1 --2- 172.21.15.16:0/2374227899 >> v2:172.21.15.89:3300/0 conn(0x7f8140070030 0x7f814006c730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:01.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:01.616+0000 7f813f400640 1 -- 172.21.15.16:0/2374227899 >> v1:172.21.15.16:6789/0 conn(0x7f8140147280 legacy=0x7f814006f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:10:01.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:01.616+0000 7f813f400640 1 -- 172.21.15.16:0/2374227899 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8140162620 con 0x7f8140147dd0 2024-03-31T15:10:01.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:01.616+0000 7f813e000640 1 -- 172.21.15.16:0/2374227899 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1788700422 0 0) 0x7f8140147660 con 0x7f8140147280 2024-03-31T15:10:01.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:01.620+0000 7f813f400640 1 --2- 172.21.15.16:0/2374227899 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8140147dd0 0x7f81400cc8f0 secure :-1 s=READY pgs=1119 cs=0 l=1 rev1=1 crypto rx=0x7f81400cfc70 tx=0x7f813801c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5cf0e17821033c44 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:10:01.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:01.620+0000 7f813e000640 1 -- 172.21.15.16:0/2374227899 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81380b0070 con 0x7f8140147dd0 2024-03-31T15:10:01.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:01.620+0000 7f813e000640 1 -- 172.21.15.16:0/2374227899 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f81380ab600 con 0x7f8140147dd0 2024-03-31T15:10:01.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:01.620+0000 7f813e000640 1 -- 172.21.15.16:0/2374227899 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81380ab9d0 con 0x7f8140147dd0 2024-03-31T15:10:01.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:01.620+0000 7f8145e00640 1 -- 172.21.15.16:0/2374227899 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8140147dd0 msgr2=0x7f81400cc8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:10:01.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:01.620+0000 7f8145e00640 1 --2- 172.21.15.16:0/2374227899 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8140147dd0 0x7f81400cc8f0 secure :-1 s=READY pgs=1119 cs=0 l=1 rev1=1 crypto rx=0x7f81400cfc70 tx=0x7f813801c480 comp rx=0 tx=0).stop 2024-03-31T15:10:01.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:01.620+0000 7f8145e00640 1 -- 172.21.15.16:0/2374227899 shutdown_connections 2024-03-31T15:10:01.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:01.620+0000 7f8145e00640 1 --2- 172.21.15.16:0/2374227899 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8140147dd0 0x7f81400cc8f0 unknown :-1 s=CLOSED pgs=1119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:01.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:01.620+0000 7f8145e00640 1 --2- 172.21.15.16:0/2374227899 >> v2:172.21.15.89:3300/0 conn(0x7f8140070030 0x7f814006c730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:01.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:01.620+0000 7f8145e00640 1 -- 172.21.15.16:0/2374227899 >> 172.21.15.16:0/2374227899 conn(0x7f8140067330 msgr2=0x7f8140067740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:10:01.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:01.620+0000 7f8145e00640 1 -- 172.21.15.16:0/2374227899 shutdown_connections 2024-03-31T15:10:01.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:01.620+0000 7f8145e00640 1 -- 172.21.15.16:0/2374227899 wait complete. 2024-03-31T15:10:01.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:01.624+0000 7f8145e00640 1 Processor -- start 2024-03-31T15:10:01.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:01.624+0000 7f8145e00640 1 -- start start 2024-03-31T15:10:01.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:01.624+0000 7f8145e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8140070030 0x7f81400b77e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:01.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:01.624+0000 7f8145e00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8140147280 0x7f814015be60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:01.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:01.624+0000 7f8145e00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f8140147dd0 0x7f81400d0bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:01.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:01.624+0000 7f8145e00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f81400cd9d0 con 0x7f8140070030 2024-03-31T15:10:01.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:01.624+0000 7f8145e00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f81400cd850 con 0x7f8140147280 2024-03-31T15:10:01.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:01.624+0000 7f8145e00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f81400cdb50 con 0x7f8140147dd0 2024-03-31T15:10:01.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:01.624+0000 7f813ea00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8140070030 0x7f81400b77e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:01.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:01.624+0000 7f813ea00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8140070030 0x7f81400b77e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39340/0 (socket says 172.21.15.16:39340) 2024-03-31T15:10:01.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:01.624+0000 7f813ea00640 1 -- 172.21.15.16:0/546001962 learned_addr learned my addr 172.21.15.16:0/546001962 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:10:01.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:01.624+0000 7f8136000640 1 --2- 172.21.15.16:0/546001962 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8140147280 0x7f814015be60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:01.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:01.624+0000 7f813f400640 1 --2- 172.21.15.16:0/546001962 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f8140147dd0 0x7f81400d0bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:01.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:01.624+0000 7f813ea00640 1 -- 172.21.15.16:0/546001962 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f8140147dd0 msgr2=0x7f81400d0bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:10:01.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:01.624+0000 7f813ea00640 1 --2- 172.21.15.16:0/546001962 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f8140147dd0 0x7f81400d0bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:01.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:01.624+0000 7f813ea00640 1 -- 172.21.15.16:0/546001962 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8140147280 msgr2=0x7f814015be60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:10:01.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:01.624+0000 7f813ea00640 1 --2- 172.21.15.16:0/546001962 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8140147280 0x7f814015be60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:01.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:01.624+0000 7f813ea00640 1 -- 172.21.15.16:0/546001962 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f814013b5f0 con 0x7f8140070030 2024-03-31T15:10:01.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:01.624+0000 7f813f400640 1 --2- 172.21.15.16:0/546001962 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f8140147dd0 0x7f81400d0bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:10:01.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:01.624+0000 7f813ea00640 1 --2- 172.21.15.16:0/546001962 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8140070030 0x7f81400b77e0 secure :-1 s=READY pgs=1120 cs=0 l=1 rev1=1 crypto rx=0x7f8130001570 tx=0x7f813000cd20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:10:01.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:01.624+0000 7f8137e00640 1 -- 172.21.15.16:0/546001962 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8130018070 con 0x7f8140070030 2024-03-31T15:10:01.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:01.624+0000 7f8137e00640 1 -- 172.21.15.16:0/546001962 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f813000a640 con 0x7f8140070030 2024-03-31T15:10:01.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:01.624+0000 7f8137e00640 1 -- 172.21.15.16:0/546001962 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f813000aa30 con 0x7f8140070030 2024-03-31T15:10:01.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:01.624+0000 7f8145e00640 1 -- 172.21.15.16:0/546001962 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f814013ce80 con 0x7f8140070030 2024-03-31T15:10:01.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:01.624+0000 7f8145e00640 1 -- 172.21.15.16:0/546001962 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f814013d440 con 0x7f8140070030 2024-03-31T15:10:01.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:01.628+0000 7f8145e00640 1 -- 172.21.15.16:0/546001962 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8140069950 con 0x7f8140070030 2024-03-31T15:10:01.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:01.628+0000 7f8137e00640 1 -- 172.21.15.16:0/546001962 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f813000abd0 con 0x7f8140070030 2024-03-31T15:10:01.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:01.628+0000 7f8137e00640 1 --2- 172.21.15.16:0/546001962 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f811c0683b0 0x7f811c06a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:01.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:01.628+0000 7f8137e00640 1 -- 172.21.15.16:0/546001962 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 5709+0+0 (secure 0 0 0) 0x7f8130085fd0 con 0x7f8140070030 2024-03-31T15:10:01.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:01.628+0000 7f8136000640 1 --2- 172.21.15.16:0/546001962 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f811c0683b0 0x7f811c06a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:01.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:01.628+0000 7f8136000640 1 --2- 172.21.15.16:0/546001962 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f811c0683b0 0x7f811c06a870 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f81400b7280 tx=0x7f8128009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:10:01.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:01.632+0000 7f8137e00640 1 -- 172.21.15.16:0/546001962 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f8130055ea0 con 0x7f8140070030 2024-03-31T15:10:01.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:01.808+0000 7f8145e00640 1 -- 172.21.15.16:0/546001962 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f814013b7c0 con 0x7f8140070030 2024-03-31T15:10:01.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:01.812+0000 7f8137e00640 1 -- 172.21.15.16:0/546001962 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) ==== 74+0+16588 (secure 0 0 0) 0x7f813005a1a0 con 0x7f8140070030 2024-03-31T15:10:01.803 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:10:01.803 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":81,"fsid":"286eb5da-fb47-4550-b290-d9519c13b9a5","created":"2024-03-31T15:03:11.438952+0000","modified":"2024-03-31T15:09:52.535916+0000","last_up_change":"2024-03-31T15:03:22.885535+0000","last_in_change":"2024-03-31T15:03:13.941667+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-31T15:03:22.550787+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-03-31T15:09:49.322620+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-03-31T15:09:49.877778+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"92a66450-ef19-40d8-ac3d-d760b23b0bda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6801","nonce":932712244}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6803","nonce":932712244}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6807","nonce":932712244}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6805","nonce":932712244}]},"public_addr":"172.21.15.16:6801/932712244","cluster_addr":"172.21.15.16:6803/932712244","heartbeat_back_addr":"172.21.15.16:6807/932712244","heartbeat_front_addr":"172.21.15.16:6805/932712244","state":["exists","up"]},{"osd":1,"uuid":"da9d1f2b-1968-4959-b843-faa231988e92","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6810","nonce":1398737533}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6811","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6813","nonce":1398737533}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6821","nonce":1398737533}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6815","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6816","nonce":1398737533}]},"public_addr":"172.21.15.16:6810/1398737533","cluster_addr":"172.21.15.16:6813/1398737533","heartbeat_back_addr":"172.21.15.16:6821/1398737533","heartbeat_front_addr":"172.21.15.16:6816/1398737533","state":["exists","up"]},{"osd":2,"uuid":"01a73662-9a21-4b1f-bcbb-42b9636a4638","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6819","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6822","nonce":3018526646}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6825","nonce":3018526646}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6831","nonce":3018526646}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6827","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6829","nonce":3018526646}]},"public_addr":"172.21.15.16:6822/3018526646","cluster_addr":"172.21.15.16:6825/3018526646","heartbeat_back_addr":"172.21.15.16:6831/3018526646","heartbeat_front_addr":"172.21.15.16:6829/3018526646","state":["exists","up"]},{"osd":3,"uuid":"c7771fd3-e18e-4500-ac38-23cf10ebcd75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6809","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6812","nonce":3360352974}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6817","nonce":3360352974}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6828","nonce":3360352974}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6823","nonce":3360352974}]},"public_addr":"172.21.15.16:6812/3360352974","cluster_addr":"172.21.15.16:6817/3360352974","heartbeat_back_addr":"172.21.15.16:6828/3360352974","heartbeat_front_addr":"172.21.15.16:6823/3360352974","state":["exists","up"]},{"osd":4,"uuid":"88260730-1482-4fbd-a40f-cb1124e9df4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6801","nonce":1856844856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6803","nonce":1856844856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6807","nonce":1856844856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6805","nonce":1856844856}]},"public_addr":"172.21.15.89:6801/1856844856","cluster_addr":"172.21.15.89:6803/1856844856","heartbeat_back_addr":"172.21.15.89:6807/1856844856","heartbeat_front_addr":"172.21.15.89:6805/1856844856","state":["exists","up"]},{"osd":5,"uuid":"4cb2c00c-fae0-4dd8-9d27-ec4f0093ccf7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6814","nonce":50395527}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6817","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6820","nonce":50395527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6831","nonce":50395527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6823","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6826","nonce":50395527}]},"public_addr":"172.21.15.89:6814/50395527","cluster_addr":"172.21.15.89:6820/50395527","heartbeat_back_addr":"172.21.15.89:6831/50395527","heartbeat_front_addr":"172.21.15.89:6826/50395527","state":["exists","up"]},{"osd":6,"uuid":"81ce0c19-2554-4cae-acbe-66a05e9e9b9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6809","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6812","nonce":3470471752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6815","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6818","nonce":3470471752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6827","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6829","nonce":3470471752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6821","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6824","nonce":3470471752}]},"public_addr":"172.21.15.89:6812/3470471752","cluster_addr":"172.21.15.89:6818/3470471752","heartbeat_back_addr":"172.21.15.89:6829/3470471752","heartbeat_front_addr":"172.21.15.89:6824/3470471752","state":["exists","up"]},{"osd":7,"uuid":"85e4313b-d37d-442e-8042-153bf20a847d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6811","nonce":1796778630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6813","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6816","nonce":1796778630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6825","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6828","nonce":1796778630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6819","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6822","nonce":1796778630}]},"public_addr":"172.21.15.89:6811/1796778630","cluster_addr":"172.21.15.89:6816/1796778630","heartbeat_back_addr":"172.21.15.89:6828/1796778630","heartbeat_front_addr":"172.21.15.89:6822/1796778630","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.439088+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.529970+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.680512+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:21.044161+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.582111+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.536558+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.409522+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.590229+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-31T15:10:01.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:01.812+0000 7f8145e00640 1 -- 172.21.15.16:0/546001962 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f811c0683b0 msgr2=0x7f811c06a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:10:01.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:01.812+0000 7f8145e00640 1 --2- 172.21.15.16:0/546001962 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f811c0683b0 0x7f811c06a870 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f81400b7280 tx=0x7f8128009190 comp rx=0 tx=0).stop 2024-03-31T15:10:01.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:01.812+0000 7f8145e00640 1 -- 172.21.15.16:0/546001962 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8140070030 msgr2=0x7f81400b77e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:10:01.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:01.812+0000 7f8145e00640 1 --2- 172.21.15.16:0/546001962 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8140070030 0x7f81400b77e0 secure :-1 s=READY pgs=1120 cs=0 l=1 rev1=1 crypto rx=0x7f8130001570 tx=0x7f813000cd20 comp rx=0 tx=0).stop 2024-03-31T15:10:01.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:01.812+0000 7f8145e00640 1 -- 172.21.15.16:0/546001962 shutdown_connections 2024-03-31T15:10:01.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:01.812+0000 7f8145e00640 1 --2- 172.21.15.16:0/546001962 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f811c0683b0 0x7f811c06a870 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:01.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:01.812+0000 7f8145e00640 1 --2- 172.21.15.16:0/546001962 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f8140147dd0 0x7f81400d0bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:01.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:01.812+0000 7f8145e00640 1 --2- 172.21.15.16:0/546001962 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8140147280 0x7f814015be60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:01.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:01.812+0000 7f8145e00640 1 --2- 172.21.15.16:0/546001962 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8140070030 0x7f81400b77e0 unknown :-1 s=CLOSED pgs=1120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:01.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:01.812+0000 7f8145e00640 1 -- 172.21.15.16:0/546001962 >> 172.21.15.16:0/546001962 conn(0x7f8140067330 msgr2=0x7f814006f920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:10:01.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:01.812+0000 7f8145e00640 1 -- 172.21.15.16:0/546001962 shutdown_connections 2024-03-31T15:10:01.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:01.812+0000 7f8145e00640 1 -- 172.21.15.16:0/546001962 wait complete. 2024-03-31T15:10:01.824 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-31T15:03:22.550787+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 12, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-31T15:09:49.322620+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '80', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 13, 'pool_name': 'cephfs_data', 'create_time': '2024-03-31T15:09:49.877778+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '81', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-31T15:10:01.825 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-31T15:10:01.968 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:01.976+0000 7fb167000640 1 Processor -- start 2024-03-31T15:10:01.968 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:01.976+0000 7fb167000640 1 -- start start 2024-03-31T15:10:01.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:01.976+0000 7fb167000640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fb1600b60a0 0x7fb16006fc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:01.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:01.976+0000 7fb167000640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb16006f540 con 0x7fb16006b620 2024-03-31T15:10:01.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:01.976+0000 7fb167000640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb1600b43f0 con 0x7fb1600d0c00 2024-03-31T15:10:01.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:01.976+0000 7fb167000640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7fb16006f820 con 0x7fb1600b60a0 2024-03-31T15:10:01.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:01.976+0000 7fb15fe00640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fb16006b620 0x7fb16006ba00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:57724/0 (socket says 172.21.15.16:57724) 2024-03-31T15:10:01.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:01.976+0000 7fb15fe00640 1 -- 172.21.15.16:0/48150 learned_addr learned my addr 172.21.15.16:0/48150 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:10:01.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:01.976+0000 7fb157e00640 1 --2- 172.21.15.16:0/48150 >> v2:172.21.15.89:3300/0 conn(0x7fb1600b60a0 0x7fb16006fc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:01.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:01.976+0000 7fb15f400640 1 -- 172.21.15.16:0/48150 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2933351227 0 0) 0x7fb16006f540 con 0x7fb16006b620 2024-03-31T15:10:01.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:01.976+0000 7fb15f400640 1 -- 172.21.15.16:0/48150 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb144003530 con 0x7fb16006b620 2024-03-31T15:10:01.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:01.976+0000 7fb157e00640 1 -- 172.21.15.16:0/48150 >> v1:172.21.15.89:6789/0 conn(0x7fb1600d0c00 legacy=0x7fb160070180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:10:01.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:01.976+0000 7fb15f400640 1 -- 172.21.15.16:0/48150 <== mon.1 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3030622501 0 0) 0x7fb1600b43f0 con 0x7fb1600d0c00 2024-03-31T15:10:01.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:01.976+0000 7fb157e00640 1 -- 172.21.15.16:0/48150 >> v1:172.21.15.16:6789/0 conn(0x7fb16006b620 legacy=0x7fb16006ba00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:10:01.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:01.976+0000 7fb157e00640 1 -- 172.21.15.16:0/48150 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1600b46d0 con 0x7fb1600b60a0 2024-03-31T15:10:01.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:01.980+0000 7fb157e00640 1 --2- 172.21.15.16:0/48150 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb1600b60a0 0x7fb16006fc40 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7fb1480097c0 tx=0x7fb14801c410 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1683b4d2e79a96bd server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:10:01.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:01.980+0000 7fb15f400640 1 -- 172.21.15.16:0/48150 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1480b0070 con 0x7fb1600b60a0 2024-03-31T15:10:01.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:01.980+0000 7fb15f400640 1 -- 172.21.15.16:0/48150 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb1480ab600 con 0x7fb1600b60a0 2024-03-31T15:10:01.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:01.980+0000 7fb15f400640 1 -- 172.21.15.16:0/48150 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1480ab9b0 con 0x7fb1600b60a0 2024-03-31T15:10:01.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:01.980+0000 7fb167000640 1 -- 172.21.15.16:0/48150 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb1600b60a0 msgr2=0x7fb16006fc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:10:01.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:01.980+0000 7fb167000640 1 --2- 172.21.15.16:0/48150 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb1600b60a0 0x7fb16006fc40 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7fb1480097c0 tx=0x7fb14801c410 comp rx=0 tx=0).stop 2024-03-31T15:10:01.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:01.980+0000 7fb167000640 1 -- 172.21.15.16:0/48150 shutdown_connections 2024-03-31T15:10:01.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:01.980+0000 7fb167000640 1 --2- 172.21.15.16:0/48150 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb1600b60a0 0x7fb16006fc40 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:01.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:01.980+0000 7fb167000640 1 -- 172.21.15.16:0/48150 >> 172.21.15.16:0/48150 conn(0x7fb16006e670 msgr2=0x7fb16006ea50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:10:01.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:01.980+0000 7fb167000640 1 -- 172.21.15.16:0/48150 shutdown_connections 2024-03-31T15:10:01.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:01.980+0000 7fb167000640 1 -- 172.21.15.16:0/48150 wait complete. 2024-03-31T15:10:01.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:01.984+0000 7fb167000640 1 Processor -- start 2024-03-31T15:10:01.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:01.984+0000 7fb167000640 1 -- start start 2024-03-31T15:10:01.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:01.984+0000 7fb167000640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb16006b620 0x7fb16014b350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:01.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:01.984+0000 7fb167000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb1600b60a0 0x7fb16014c170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:01.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:01.984+0000 7fb167000640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb1600d0c00 0x7fb16014d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:01.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:01.984+0000 7fb167000640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fb16016cd10 con 0x7fb1600b60a0 2024-03-31T15:10:01.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:01.984+0000 7fb167000640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fb16016cb90 con 0x7fb1600d0c00 2024-03-31T15:10:01.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:01.984+0000 7fb167000640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fb16016ce90 con 0x7fb16006b620 2024-03-31T15:10:01.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:01.984+0000 7fb157e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb1600b60a0 0x7fb16014c170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:01.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:01.984+0000 7fb157e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb1600b60a0 0x7fb16014c170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39344/0 (socket says 172.21.15.16:39344) 2024-03-31T15:10:01.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:01.984+0000 7fb157e00640 1 -- 172.21.15.16:0/3903672554 learned_addr learned my addr 172.21.15.16:0/3903672554 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:10:01.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:01.984+0000 7fb15fe00640 1 --2- 172.21.15.16:0/3903672554 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb16006b620 0x7fb16014b350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:01.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:01.984+0000 7fb165000640 1 --2- 172.21.15.16:0/3903672554 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb1600d0c00 0x7fb16014d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:01.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:01.984+0000 7fb157e00640 1 -- 172.21.15.16:0/3903672554 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb16006b620 msgr2=0x7fb16014b350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:10:01.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:01.984+0000 7fb157e00640 1 --2- 172.21.15.16:0/3903672554 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb16006b620 0x7fb16014b350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:01.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:01.984+0000 7fb157e00640 1 -- 172.21.15.16:0/3903672554 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb1600d0c00 msgr2=0x7fb16014d3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:10:01.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:01.984+0000 7fb157e00640 1 --2- 172.21.15.16:0/3903672554 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb1600d0c00 0x7fb16014d3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:01.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:01.984+0000 7fb157e00640 1 -- 172.21.15.16:0/3903672554 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb16015abf0 con 0x7fb1600b60a0 2024-03-31T15:10:01.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:01.984+0000 7fb157e00640 1 --2- 172.21.15.16:0/3903672554 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb1600b60a0 0x7fb16014c170 secure :-1 s=READY pgs=1122 cs=0 l=1 rev1=1 crypto rx=0x7fb148019020 tx=0x7fb14801ac50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:10:01.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:01.984+0000 7fb15cc00640 1 -- 172.21.15.16:0/3903672554 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1480b0050 con 0x7fb1600b60a0 2024-03-31T15:10:01.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:01.984+0000 7fb15cc00640 1 -- 172.21.15.16:0/3903672554 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb1480b1040 con 0x7fb1600b60a0 2024-03-31T15:10:01.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:01.984+0000 7fb15cc00640 1 -- 172.21.15.16:0/3903672554 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb148006e20 con 0x7fb1600b60a0 2024-03-31T15:10:01.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:01.984+0000 7fb167000640 1 -- 172.21.15.16:0/3903672554 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb16015a630 con 0x7fb1600b60a0 2024-03-31T15:10:01.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:01.984+0000 7fb167000640 1 -- 172.21.15.16:0/3903672554 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb16014b970 con 0x7fb1600b60a0 2024-03-31T15:10:01.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:01.988+0000 7fb167000640 1 -- 172.21.15.16:0/3903672554 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb160069950 con 0x7fb1600b60a0 2024-03-31T15:10:01.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:01.988+0000 7fb15cc00640 1 -- 172.21.15.16:0/3903672554 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fb1480be030 con 0x7fb1600b60a0 2024-03-31T15:10:01.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:01.988+0000 7fb15cc00640 1 --2- 172.21.15.16:0/3903672554 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fb13c068380 0x7fb13c06a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:01.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:01.988+0000 7fb15cc00640 1 -- 172.21.15.16:0/3903672554 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 5709+0+0 (secure 0 0 0) 0x7fb148121470 con 0x7fb1600b60a0 2024-03-31T15:10:01.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:01.988+0000 7fb15fe00640 1 --2- 172.21.15.16:0/3903672554 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fb13c068380 0x7fb13c06a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:01.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:01.992+0000 7fb15fe00640 1 --2- 172.21.15.16:0/3903672554 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fb13c068380 0x7fb13c06a840 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7fb150007fd0 tx=0x7fb150009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:10:01.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:01.996+0000 7fb15cc00640 1 -- 172.21.15.16:0/3903672554 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fb1480c5020 con 0x7fb1600b60a0 2024-03-31T15:10:02.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:02.164+0000 7fb167000640 1 -- 172.21.15.16:0/3903672554 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fb16006c220 con 0x7fb1600b60a0 2024-03-31T15:10:02.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:02.164+0000 7fb15cc00640 1 -- 172.21.15.16:0/3903672554 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v81) ==== 93+0+10 (secure 0 0 0) 0x7fb148016090 con 0x7fb1600b60a0 2024-03-31T15:10:02.158 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2024-03-31T15:10:02.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:02.168+0000 7fb167000640 1 -- 172.21.15.16:0/3903672554 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fb13c068380 msgr2=0x7fb13c06a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:10:02.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:02.168+0000 7fb167000640 1 --2- 172.21.15.16:0/3903672554 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fb13c068380 0x7fb13c06a840 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7fb150007fd0 tx=0x7fb150009040 comp rx=0 tx=0).stop 2024-03-31T15:10:02.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:02.168+0000 7fb167000640 1 -- 172.21.15.16:0/3903672554 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb1600b60a0 msgr2=0x7fb16014c170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:10:02.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:02.168+0000 7fb167000640 1 --2- 172.21.15.16:0/3903672554 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb1600b60a0 0x7fb16014c170 secure :-1 s=READY pgs=1122 cs=0 l=1 rev1=1 crypto rx=0x7fb148019020 tx=0x7fb14801ac50 comp rx=0 tx=0).stop 2024-03-31T15:10:02.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:02.168+0000 7fb167000640 1 -- 172.21.15.16:0/3903672554 shutdown_connections 2024-03-31T15:10:02.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:02.168+0000 7fb167000640 1 --2- 172.21.15.16:0/3903672554 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fb13c068380 0x7fb13c06a840 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:02.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:02.168+0000 7fb167000640 1 --2- 172.21.15.16:0/3903672554 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb1600d0c00 0x7fb16014d3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:02.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:02.168+0000 7fb167000640 1 --2- 172.21.15.16:0/3903672554 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb1600b60a0 0x7fb16014c170 unknown :-1 s=CLOSED pgs=1122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:02.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:02.168+0000 7fb167000640 1 --2- 172.21.15.16:0/3903672554 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb16006b620 0x7fb16014b350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:02.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:02.168+0000 7fb167000640 1 -- 172.21.15.16:0/3903672554 >> 172.21.15.16:0/3903672554 conn(0x7fb16006e670 msgr2=0x7fb16015da30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:10:02.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:02.168+0000 7fb167000640 1 -- 172.21.15.16:0/3903672554 shutdown_connections 2024-03-31T15:10:02.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:02.168+0000 7fb167000640 1 -- 172.21.15.16:0/3903672554 wait complete. 2024-03-31T15:10:02.179 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-31T15:10:02.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:02.328+0000 7f8346400640 1 Processor -- start 2024-03-31T15:10:02.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:02.328+0000 7f8346400640 1 -- start start 2024-03-31T15:10:02.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:02.328+0000 7f8346400640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f834006f920 0x7f834006fd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:02.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:02.332+0000 7f8346400640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f83400cf150 0x7f83401419c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:02.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:02.332+0000 7f8346400640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8340070240 con 0x7f83400d0a60 2024-03-31T15:10:02.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:02.332+0000 7f8346400640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f834013f670 con 0x7f834006f920 2024-03-31T15:10:02.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:02.332+0000 7f8346400640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f834015ca70 con 0x7f83400cf150 2024-03-31T15:10:02.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:02.332+0000 7f833f400640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f834006f920 0x7f834006fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:02.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:02.332+0000 7f8337e00640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f83400d0a60 0x7f83400cea40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:57736/0 (socket says 172.21.15.16:57736) 2024-03-31T15:10:02.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:02.332+0000 7f8337e00640 1 -- 172.21.15.16:0/1062844413 learned_addr learned my addr 172.21.15.16:0/1062844413 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:10:02.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:02.332+0000 7f833fe00640 1 --2- 172.21.15.16:0/1062844413 >> v2:172.21.15.89:3300/0 conn(0x7f83400cf150 0x7f83401419c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:02.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:02.332+0000 7f833f400640 1 -- 172.21.15.16:0/1062844413 >> v2:172.21.15.89:3300/0 conn(0x7f83400cf150 msgr2=0x7f83401419c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:10:02.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:02.332+0000 7f833f400640 1 --2- 172.21.15.16:0/1062844413 >> v2:172.21.15.89:3300/0 conn(0x7f83400cf150 0x7f83401419c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:02.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:02.332+0000 7f833f400640 1 -- 172.21.15.16:0/1062844413 >> v1:172.21.15.16:6789/0 conn(0x7f83400d0a60 legacy=0x7f83400cea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:10:02.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:02.332+0000 7f833f400640 1 -- 172.21.15.16:0/1062844413 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f834015cd50 con 0x7f834006f920 2024-03-31T15:10:02.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:02.332+0000 7f833fe00640 1 --2- 172.21.15.16:0/1062844413 >> v2:172.21.15.89:3300/0 conn(0x7f83400cf150 0x7f83401419c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:10:02.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:02.332+0000 7f833f400640 1 --2- 172.21.15.16:0/1062844413 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f834006f920 0x7f834006fd00 secure :-1 s=READY pgs=1124 cs=0 l=1 rev1=1 crypto rx=0x7f83400d2080 tx=0x7f833001c3f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cabedecbc731e9c7 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:10:02.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:02.332+0000 7f833ea00640 1 -- 172.21.15.16:0/1062844413 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83300b0070 con 0x7f834006f920 2024-03-31T15:10:02.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:02.332+0000 7f833ea00640 1 -- 172.21.15.16:0/1062844413 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f83300ab600 con 0x7f834006f920 2024-03-31T15:10:02.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:02.332+0000 7f833ea00640 1 -- 172.21.15.16:0/1062844413 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83300ab9d0 con 0x7f834006f920 2024-03-31T15:10:02.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:02.336+0000 7f8346400640 1 -- 172.21.15.16:0/1062844413 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f834006f920 msgr2=0x7f834006fd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:10:02.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:02.336+0000 7f8346400640 1 --2- 172.21.15.16:0/1062844413 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f834006f920 0x7f834006fd00 secure :-1 s=READY pgs=1124 cs=0 l=1 rev1=1 crypto rx=0x7f83400d2080 tx=0x7f833001c3f0 comp rx=0 tx=0).stop 2024-03-31T15:10:02.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:02.336+0000 7f8346400640 1 -- 172.21.15.16:0/1062844413 shutdown_connections 2024-03-31T15:10:02.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:02.336+0000 7f8346400640 1 --2- 172.21.15.16:0/1062844413 >> v2:172.21.15.89:3300/0 conn(0x7f83400cf150 0x7f83401419c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:02.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:02.336+0000 7f8346400640 1 --2- 172.21.15.16:0/1062844413 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f834006f920 0x7f834006fd00 unknown :-1 s=CLOSED pgs=1124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:02.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:02.336+0000 7f8346400640 1 -- 172.21.15.16:0/1062844413 >> 172.21.15.16:0/1062844413 conn(0x7f8340067930 msgr2=0x7f83400b3ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:10:02.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:02.336+0000 7f8346400640 1 -- 172.21.15.16:0/1062844413 shutdown_connections 2024-03-31T15:10:02.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:02.336+0000 7f8346400640 1 -- 172.21.15.16:0/1062844413 wait complete. 2024-03-31T15:10:02.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:02.336+0000 7f8346400640 1 Processor -- start 2024-03-31T15:10:02.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:02.336+0000 7f8346400640 1 -- start start 2024-03-31T15:10:02.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:02.336+0000 7f8346400640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f834006f920 0x7f8340166760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:02.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:02.336+0000 7f8346400640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f83400cf150 0x7f834006e000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:02.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:02.336+0000 7f8346400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f83400d0a60 0x7f8340154f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:02.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:02.336+0000 7f8346400640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f8340169230 con 0x7f83400d0a60 2024-03-31T15:10:02.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:02.336+0000 7f8346400640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f83401690b0 con 0x7f83400cf150 2024-03-31T15:10:02.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:02.336+0000 7f8346400640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f83401693b0 con 0x7f834006f920 2024-03-31T15:10:02.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:02.336+0000 7f833fe00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f83400d0a60 0x7f8340154f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:02.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:02.336+0000 7f833fe00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f83400d0a60 0x7f8340154f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39358/0 (socket says 172.21.15.16:39358) 2024-03-31T15:10:02.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:02.336+0000 7f833fe00640 1 -- 172.21.15.16:0/478949181 learned_addr learned my addr 172.21.15.16:0/478949181 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:10:02.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:02.336+0000 7f833f400640 1 --2- 172.21.15.16:0/478949181 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f834006f920 0x7f8340166760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:02.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:02.336+0000 7f8337e00640 1 --2- 172.21.15.16:0/478949181 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f83400cf150 0x7f834006e000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:02.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:02.340+0000 7f833fe00640 1 -- 172.21.15.16:0/478949181 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f834006f920 msgr2=0x7f8340166760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:10:02.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:02.340+0000 7f833fe00640 1 --2- 172.21.15.16:0/478949181 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f834006f920 0x7f8340166760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:02.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:02.340+0000 7f833fe00640 1 -- 172.21.15.16:0/478949181 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f83400cf150 msgr2=0x7f834006e000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:10:02.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:02.340+0000 7f833fe00640 1 --2- 172.21.15.16:0/478949181 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f83400cf150 0x7f834006e000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:02.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:02.340+0000 7f833fe00640 1 -- 172.21.15.16:0/478949181 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8340150950 con 0x7f83400d0a60 2024-03-31T15:10:02.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:02.340+0000 7f833f400640 1 --2- 172.21.15.16:0/478949181 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f834006f920 0x7f8340166760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:10:02.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:02.340+0000 7f833fe00640 1 --2- 172.21.15.16:0/478949181 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f83400d0a60 0x7f8340154f50 secure :-1 s=READY pgs=1125 cs=0 l=1 rev1=1 crypto rx=0x7f833800cef0 tx=0x7f833800a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:10:02.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:02.340+0000 7f8337400640 1 -- 172.21.15.16:0/478949181 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f833801a070 con 0x7f83400d0a60 2024-03-31T15:10:02.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:02.340+0000 7f8337400640 1 -- 172.21.15.16:0/478949181 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f833800d860 con 0x7f83400d0a60 2024-03-31T15:10:02.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:02.340+0000 7f8337400640 1 -- 172.21.15.16:0/478949181 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f833800dc50 con 0x7f83400d0a60 2024-03-31T15:10:02.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:02.340+0000 7f8346400640 1 -- 172.21.15.16:0/478949181 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f834006d5c0 con 0x7f83400d0a60 2024-03-31T15:10:02.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:02.340+0000 7f8346400640 1 -- 172.21.15.16:0/478949181 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8340150bc0 con 0x7f83400d0a60 2024-03-31T15:10:02.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:02.340+0000 7f8346400640 1 -- 172.21.15.16:0/478949181 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8340069950 con 0x7f83400d0a60 2024-03-31T15:10:02.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:02.340+0000 7f8337400640 1 -- 172.21.15.16:0/478949181 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f833801d070 con 0x7f83400d0a60 2024-03-31T15:10:02.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:02.340+0000 7f8337400640 1 --2- 172.21.15.16:0/478949181 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f831c0683b0 0x7f831c06a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:02.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:02.340+0000 7f8337400640 1 -- 172.21.15.16:0/478949181 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 5709+0+0 (secure 0 0 0) 0x7f83380871f0 con 0x7f83400d0a60 2024-03-31T15:10:02.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:02.344+0000 7f833f400640 1 --2- 172.21.15.16:0/478949181 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f831c0683b0 0x7f831c06a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:02.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:02.344+0000 7f833f400640 1 --2- 172.21.15.16:0/478949181 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f831c0683b0 0x7f831c06a870 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f83300ad000 tx=0x7f8330008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:10:02.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:02.352+0000 7f8337400640 1 -- 172.21.15.16:0/478949181 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f8338057190 con 0x7f83400d0a60 2024-03-31T15:10:02.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:02.516+0000 7f8346400640 1 -- 172.21.15.16:0/478949181 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f834006ffc0 con 0x7f83400d0a60 2024-03-31T15:10:02.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:02.516+0000 7f8337400640 1 -- 172.21.15.16:0/478949181 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v81) ==== 104+0+11 (secure 0 0 0) 0x7f833805b490 con 0x7f83400d0a60 2024-03-31T15:10:02.510 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2024-03-31T15:10:02.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:02.520+0000 7f8346400640 1 -- 172.21.15.16:0/478949181 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f831c0683b0 msgr2=0x7f831c06a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:10:02.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:02.520+0000 7f8346400640 1 --2- 172.21.15.16:0/478949181 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f831c0683b0 0x7f831c06a870 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f83300ad000 tx=0x7f8330008040 comp rx=0 tx=0).stop 2024-03-31T15:10:02.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:02.520+0000 7f8346400640 1 -- 172.21.15.16:0/478949181 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f83400d0a60 msgr2=0x7f8340154f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:10:02.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:02.520+0000 7f8346400640 1 --2- 172.21.15.16:0/478949181 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f83400d0a60 0x7f8340154f50 secure :-1 s=READY pgs=1125 cs=0 l=1 rev1=1 crypto rx=0x7f833800cef0 tx=0x7f833800a710 comp rx=0 tx=0).stop 2024-03-31T15:10:02.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:02.520+0000 7f8346400640 1 -- 172.21.15.16:0/478949181 shutdown_connections 2024-03-31T15:10:02.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:02.520+0000 7f8346400640 1 --2- 172.21.15.16:0/478949181 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f831c0683b0 0x7f831c06a870 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:02.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:02.520+0000 7f8346400640 1 --2- 172.21.15.16:0/478949181 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f83400d0a60 0x7f8340154f50 unknown :-1 s=CLOSED pgs=1125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:02.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:02.520+0000 7f8346400640 1 --2- 172.21.15.16:0/478949181 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f83400cf150 0x7f834006e000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:02.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:02.520+0000 7f8346400640 1 --2- 172.21.15.16:0/478949181 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f834006f920 0x7f8340166760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:02.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:02.520+0000 7f8346400640 1 -- 172.21.15.16:0/478949181 >> 172.21.15.16:0/478949181 conn(0x7f8340067930 msgr2=0x7f8340156b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:10:02.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:02.520+0000 7f8346400640 1 -- 172.21.15.16:0/478949181 shutdown_connections 2024-03-31T15:10:02.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:02.520+0000 7f8346400640 1 -- 172.21.15.16:0/478949181 wait complete. 2024-03-31T15:10:02.531 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-31T15:10:02.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:02.684+0000 7f9a09600640 1 Processor -- start 2024-03-31T15:10:02.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:02.684+0000 7f9a09600640 1 -- start start 2024-03-31T15:10:02.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:02.684+0000 7f9a09600640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f9a0413e3d0 0x7f9a040cc9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:02.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:02.684+0000 7f9a09600640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9a0415fbc0 con 0x7f9a0406dbe0 2024-03-31T15:10:02.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:02.684+0000 7f9a09600640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9a0415ee00 con 0x7f9a0413dff0 2024-03-31T15:10:02.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:02.684+0000 7f9a09600640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f9a0415f730 con 0x7f9a0413e3d0 2024-03-31T15:10:02.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:02.684+0000 7f99fbe00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f9a0413e3d0 0x7f9a040cc9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:02.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:02.684+0000 7f99fbe00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f9a0413e3d0 0x7f9a040cc9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39370/0 (socket says 172.21.15.16:39370) 2024-03-31T15:10:02.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:02.684+0000 7f99fbe00640 1 -- 172.21.15.16:0/3928169540 learned_addr learned my addr 172.21.15.16:0/3928169540 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:10:02.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:02.684+0000 7f99fbe00640 1 -- 172.21.15.16:0/3928169540 >> v1:172.21.15.89:6789/0 conn(0x7f9a0413dff0 legacy=0x7f9a0406d4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:10:02.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:02.684+0000 7f99fbe00640 1 -- 172.21.15.16:0/3928169540 >> v1:172.21.15.16:6789/0 conn(0x7f9a0406dbe0 legacy=0x7f9a040ceb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:10:02.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:02.684+0000 7f99fbe00640 1 -- 172.21.15.16:0/3928169540 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a0415f0e0 con 0x7f9a0413e3d0 2024-03-31T15:10:02.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:02.688+0000 7f99fbe00640 1 --2- 172.21.15.16:0/3928169540 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9a0413e3d0 0x7f9a040cc9d0 secure :-1 s=READY pgs=1127 cs=0 l=1 rev1=1 crypto rx=0x7f99ec009870 tx=0x7f99ec01c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4522a4f7b66fa38e server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:10:02.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:02.688+0000 7f9a02000640 1 -- 172.21.15.16:0/3928169540 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99ec0b0070 con 0x7f9a0413e3d0 2024-03-31T15:10:02.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:02.688+0000 7f9a02000640 1 -- 172.21.15.16:0/3928169540 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f99ec0ab600 con 0x7f9a0413e3d0 2024-03-31T15:10:02.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:02.688+0000 7f9a02000640 1 -- 172.21.15.16:0/3928169540 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99ec0ab9d0 con 0x7f9a0413e3d0 2024-03-31T15:10:02.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:02.688+0000 7f9a09600640 1 -- 172.21.15.16:0/3928169540 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9a0413e3d0 msgr2=0x7f9a040cc9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:10:02.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:02.688+0000 7f9a09600640 1 --2- 172.21.15.16:0/3928169540 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9a0413e3d0 0x7f9a040cc9d0 secure :-1 s=READY pgs=1127 cs=0 l=1 rev1=1 crypto rx=0x7f99ec009870 tx=0x7f99ec01c4d0 comp rx=0 tx=0).stop 2024-03-31T15:10:02.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:02.688+0000 7f9a09600640 1 -- 172.21.15.16:0/3928169540 shutdown_connections 2024-03-31T15:10:02.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:02.688+0000 7f9a09600640 1 --2- 172.21.15.16:0/3928169540 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9a0413e3d0 0x7f9a040cc9d0 unknown :-1 s=CLOSED pgs=1127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:02.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:02.688+0000 7f9a09600640 1 -- 172.21.15.16:0/3928169540 >> 172.21.15.16:0/3928169540 conn(0x7f9a0406ee40 msgr2=0x7f9a0406b620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:10:02.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:02.688+0000 7f9a09600640 1 -- 172.21.15.16:0/3928169540 shutdown_connections 2024-03-31T15:10:02.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:02.688+0000 7f9a09600640 1 -- 172.21.15.16:0/3928169540 wait complete. 2024-03-31T15:10:02.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:02.688+0000 7f9a09600640 1 Processor -- start 2024-03-31T15:10:02.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:02.688+0000 7f9a09600640 1 -- start start 2024-03-31T15:10:02.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:02.692+0000 7f9a09600640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f9a0406dbe0 0x7f9a0413d120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:02.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:02.692+0000 7f9a09600640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9a0413dff0 0x7f9a04159e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:02.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:02.692+0000 7f9a09600640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9a0413e3d0 0x7f9a0415d040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:02.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:02.692+0000 7f9a09600640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f9a04169080 con 0x7f9a0413e3d0 2024-03-31T15:10:02.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:02.692+0000 7f9a09600640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f9a040cfce0 con 0x7f9a0413dff0 2024-03-31T15:10:02.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:02.692+0000 7f9a09600640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f9a04169200 con 0x7f9a0406dbe0 2024-03-31T15:10:02.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:02.692+0000 7f9a03400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9a0413e3d0 0x7f9a0415d040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:02.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:02.692+0000 7f99fbe00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9a0413dff0 0x7f9a04159e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:02.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:02.692+0000 7f9a03400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9a0413e3d0 0x7f9a0415d040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39378/0 (socket says 172.21.15.16:39378) 2024-03-31T15:10:02.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:02.692+0000 7f9a03400640 1 -- 172.21.15.16:0/1854508661 learned_addr learned my addr 172.21.15.16:0/1854508661 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:10:02.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:02.692+0000 7f9a02a00640 1 --2- 172.21.15.16:0/1854508661 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f9a0406dbe0 0x7f9a0413d120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:02.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:02.692+0000 7f9a03400640 1 -- 172.21.15.16:0/1854508661 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f9a0406dbe0 msgr2=0x7f9a0413d120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:10:02.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:02.692+0000 7f9a03400640 1 --2- 172.21.15.16:0/1854508661 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f9a0406dbe0 0x7f9a0413d120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:02.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:02.692+0000 7f9a03400640 1 -- 172.21.15.16:0/1854508661 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9a0413dff0 msgr2=0x7f9a04159e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:10:02.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:02.692+0000 7f9a03400640 1 --2- 172.21.15.16:0/1854508661 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9a0413dff0 0x7f9a04159e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:02.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:02.692+0000 7f9a03400640 1 -- 172.21.15.16:0/1854508661 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a0413d660 con 0x7f9a0413e3d0 2024-03-31T15:10:02.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:02.692+0000 7f99fbe00640 1 --2- 172.21.15.16:0/1854508661 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9a0413dff0 0x7f9a04159e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:10:02.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:02.692+0000 7f9a03400640 1 --2- 172.21.15.16:0/1854508661 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9a0413e3d0 0x7f9a0415d040 secure :-1 s=READY pgs=1128 cs=0 l=1 rev1=1 crypto rx=0x7f99fc00c7a0 tx=0x7f99fc00cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:10:02.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:02.692+0000 7f99faa00640 1 -- 172.21.15.16:0/1854508661 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99fc003b80 con 0x7f9a0413e3d0 2024-03-31T15:10:02.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:02.692+0000 7f99faa00640 1 -- 172.21.15.16:0/1854508661 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f99fc003d20 con 0x7f9a0413e3d0 2024-03-31T15:10:02.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:02.692+0000 7f9a09600640 1 -- 172.21.15.16:0/1854508661 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9a0415a860 con 0x7f9a0413e3d0 2024-03-31T15:10:02.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:02.692+0000 7f99faa00640 1 -- 172.21.15.16:0/1854508661 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99fc00aa30 con 0x7f9a0413e3d0 2024-03-31T15:10:02.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:02.692+0000 7f9a09600640 1 -- 172.21.15.16:0/1854508661 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9a04152680 con 0x7f9a0413e3d0 2024-03-31T15:10:02.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:02.692+0000 7f9a09600640 1 -- 172.21.15.16:0/1854508661 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9a04069950 con 0x7f9a0413e3d0 2024-03-31T15:10:02.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:02.696+0000 7f99faa00640 1 -- 172.21.15.16:0/1854508661 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f99fc00abd0 con 0x7f9a0413e3d0 2024-03-31T15:10:02.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:02.696+0000 7f99faa00640 1 --2- 172.21.15.16:0/1854508661 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f99e0068380 0x7f99e006a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:02.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:02.696+0000 7f99faa00640 1 -- 172.21.15.16:0/1854508661 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 5709+0+0 (secure 0 0 0) 0x7f99fc086880 con 0x7f9a0413e3d0 2024-03-31T15:10:02.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:02.696+0000 7f9a02a00640 1 --2- 172.21.15.16:0/1854508661 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f99e0068380 0x7f99e006a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:02.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:02.696+0000 7f9a02a00640 1 --2- 172.21.15.16:0/1854508661 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f99e0068380 0x7f99e006a840 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f9a040af170 tx=0x7f99f4009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:10:02.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:02.700+0000 7f99faa00640 1 -- 172.21.15.16:0/1854508661 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f99fc0566d0 con 0x7f9a0413e3d0 2024-03-31T15:10:02.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:02.864+0000 7f9a09600640 1 -- 172.21.15.16:0/1854508661 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f9a0406e4e0 con 0x7f9a0413e3d0 2024-03-31T15:10:02.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:02.864+0000 7f99faa00640 1 -- 172.21.15.16:0/1854508661 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v81) ==== 100+0+11 (secure 0 0 0) 0x7f99fc05a9d0 con 0x7f9a0413e3d0 2024-03-31T15:10:02.857 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 64 2024-03-31T15:10:02.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:02.868+0000 7f9a09600640 1 -- 172.21.15.16:0/1854508661 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f99e0068380 msgr2=0x7f99e006a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:10:02.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:02.868+0000 7f9a09600640 1 --2- 172.21.15.16:0/1854508661 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f99e0068380 0x7f99e006a840 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f9a040af170 tx=0x7f99f4009040 comp rx=0 tx=0).stop 2024-03-31T15:10:02.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:02.868+0000 7f9a09600640 1 -- 172.21.15.16:0/1854508661 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9a0413e3d0 msgr2=0x7f9a0415d040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:10:02.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:02.868+0000 7f9a09600640 1 --2- 172.21.15.16:0/1854508661 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9a0413e3d0 0x7f9a0415d040 secure :-1 s=READY pgs=1128 cs=0 l=1 rev1=1 crypto rx=0x7f99fc00c7a0 tx=0x7f99fc00cc60 comp rx=0 tx=0).stop 2024-03-31T15:10:02.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:02.868+0000 7f9a09600640 1 -- 172.21.15.16:0/1854508661 shutdown_connections 2024-03-31T15:10:02.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:02.868+0000 7f9a09600640 1 --2- 172.21.15.16:0/1854508661 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f99e0068380 0x7f99e006a840 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:02.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:02.868+0000 7f9a09600640 1 --2- 172.21.15.16:0/1854508661 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9a0413e3d0 0x7f9a0415d040 unknown :-1 s=CLOSED pgs=1128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:02.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:02.868+0000 7f9a09600640 1 --2- 172.21.15.16:0/1854508661 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9a0413dff0 0x7f9a04159e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:02.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:02.868+0000 7f9a09600640 1 --2- 172.21.15.16:0/1854508661 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f9a0406dbe0 0x7f9a0413d120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:02.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:02.868+0000 7f9a09600640 1 -- 172.21.15.16:0/1854508661 >> 172.21.15.16:0/1854508661 conn(0x7f9a0406ee40 msgr2=0x7f9a04071550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:10:02.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:02.868+0000 7f9a09600640 1 -- 172.21.15.16:0/1854508661 shutdown_connections 2024-03-31T15:10:02.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:02.868+0000 7f9a09600640 1 -- 172.21.15.16:0/1854508661 wait complete. 2024-03-31T15:10:02.878 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-31T15:10:03.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:03.052+0000 7f6570e00640 1 Processor -- start 2024-03-31T15:10:03.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:03.052+0000 7f6570e00640 1 -- start start 2024-03-31T15:10:03.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:03.052+0000 7f6570e00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f656c15b3d0 0x7f656c0b1cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:03.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:03.052+0000 7f6570e00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f656c0ce2f0 0x7f656c0b2200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:03.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:03.052+0000 7f6570e00640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f656c15f3c0 con 0x7f656c15eee0 2024-03-31T15:10:03.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:03.052+0000 7f6570e00640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f656c15b7b0 con 0x7f656c0ce2f0 2024-03-31T15:10:03.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:03.052+0000 7f6570e00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f656c15f6a0 con 0x7f656c15b3d0 2024-03-31T15:10:03.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:03.052+0000 7f6563e00640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f656c15eee0 0x7f656c0cc9c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:57752/0 (socket says 172.21.15.16:57752) 2024-03-31T15:10:03.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:03.052+0000 7f6563e00640 1 -- 172.21.15.16:0/2020951601 learned_addr learned my addr 172.21.15.16:0/2020951601 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:10:03.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:03.052+0000 7f656a800640 1 --2- 172.21.15.16:0/2020951601 >> v2:172.21.15.89:3300/0 conn(0x7f656c0ce2f0 0x7f656c0b2200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:03.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:03.052+0000 7f6569e00640 1 --2- 172.21.15.16:0/2020951601 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f656c15b3d0 0x7f656c0b1cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:03.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:03.052+0000 7f6569400640 1 -- 172.21.15.16:0/2020951601 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1606185361 0 0) 0x7f656c15f3c0 con 0x7f656c15eee0 2024-03-31T15:10:03.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:03.052+0000 7f6569400640 1 -- 172.21.15.16:0/2020951601 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6550003530 con 0x7f656c15eee0 2024-03-31T15:10:03.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:03.052+0000 7f6569400640 1 -- 172.21.15.16:0/2020951601 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2725001767 0 0) 0x7f6550003530 con 0x7f656c15eee0 2024-03-31T15:10:03.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:03.052+0000 7f6569400640 1 -- 172.21.15.16:0/2020951601 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f656c15b3d0 msgr2=0x7f656c0b1cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:10:03.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:03.052+0000 7f6569400640 1 --2- 172.21.15.16:0/2020951601 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f656c15b3d0 0x7f656c0b1cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:03.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:03.052+0000 7f6569400640 1 -- 172.21.15.16:0/2020951601 >> v2:172.21.15.89:3300/0 conn(0x7f656c0ce2f0 msgr2=0x7f656c0b2200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:10:03.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:03.052+0000 7f6569400640 1 --2- 172.21.15.16:0/2020951601 >> v2:172.21.15.89:3300/0 conn(0x7f656c0ce2f0 0x7f656c0b2200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:03.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:03.052+0000 7f6569400640 1 -- 172.21.15.16:0/2020951601 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f656c15f980 con 0x7f656c15eee0 2024-03-31T15:10:03.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:03.052+0000 7f6569400640 1 -- 172.21.15.16:0/2020951601 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 214807623 0 0) 0x7f6554001680 con 0x7f656c15eee0 2024-03-31T15:10:03.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:03.052+0000 7f6569400640 1 -- 172.21.15.16:0/2020951601 >> v1:172.21.15.16:6789/0 conn(0x7f656c15eee0 legacy=0x7f656c0cc9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:10:03.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:03.052+0000 7f6569400640 1 --2- 172.21.15.16:0/2020951601 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f65500039c0 0x7f6550003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:03.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:03.052+0000 7f6569400640 1 --2- 172.21.15.16:0/2020951601 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6550004810 0x7f6550004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:03.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:03.052+0000 7f6569400640 1 --2- 172.21.15.16:0/2020951601 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f656c15b3d0 0x7f65500051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:03.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:03.052+0000 7f6569400640 1 -- 172.21.15.16:0/2020951601 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f656c15b7b0 con 0x7f65500039c0 2024-03-31T15:10:03.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:03.052+0000 7f6569400640 1 -- 172.21.15.16:0/2020951601 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f656c15f6a0 con 0x7f6550004810 2024-03-31T15:10:03.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:03.052+0000 7f6569400640 1 -- 172.21.15.16:0/2020951601 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f655000dbe0 con 0x7f656c15b3d0 2024-03-31T15:10:03.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:03.052+0000 7f656a800640 1 --2- 172.21.15.16:0/2020951601 >> v2:172.21.15.89:3300/0 conn(0x7f656c0ce2f0 0x7f656c0b2200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:10:03.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:03.052+0000 7f6569e00640 1 --2- 172.21.15.16:0/2020951601 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f65500039c0 0x7f6550003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:03.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:03.052+0000 7f6563e00640 1 --2- 172.21.15.16:0/2020951601 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6550004810 0x7f6550004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:03.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:03.052+0000 7f656a800640 1 --2- 172.21.15.16:0/2020951601 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f656c15b3d0 0x7f65500051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:03.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:03.056+0000 7f6569e00640 1 -- 172.21.15.16:0/2020951601 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f656c15b3d0 msgr2=0x7f65500051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:10:03.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:03.056+0000 7f6569e00640 1 --2- 172.21.15.16:0/2020951601 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f656c15b3d0 0x7f65500051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:03.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:03.056+0000 7f6569e00640 1 -- 172.21.15.16:0/2020951601 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6550004810 msgr2=0x7f6550004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:10:03.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:03.056+0000 7f6569e00640 1 --2- 172.21.15.16:0/2020951601 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6550004810 0x7f6550004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:03.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:03.056+0000 7f6569e00640 1 -- 172.21.15.16:0/2020951601 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6550003530 con 0x7f65500039c0 2024-03-31T15:10:03.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:03.056+0000 7f6569e00640 1 --2- 172.21.15.16:0/2020951601 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f65500039c0 0x7f6550003de0 secure :-1 s=READY pgs=1130 cs=0 l=1 rev1=1 crypto rx=0x7f655c009ae0 tx=0x7f655c01ca40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e06c3c30421af68 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:10:03.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:03.056+0000 7f6569400640 1 -- 172.21.15.16:0/2020951601 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f655c0096b0 con 0x7f65500039c0 2024-03-31T15:10:03.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:03.056+0000 7f6569400640 1 -- 172.21.15.16:0/2020951601 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f655c0a6500 con 0x7f65500039c0 2024-03-31T15:10:03.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:03.056+0000 7f6569400640 1 -- 172.21.15.16:0/2020951601 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f655c0a68f0 con 0x7f65500039c0 2024-03-31T15:10:03.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:03.056+0000 7f6570e00640 1 -- 172.21.15.16:0/2020951601 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f65500039c0 msgr2=0x7f6550003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:10:03.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:03.056+0000 7f6570e00640 1 --2- 172.21.15.16:0/2020951601 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f65500039c0 0x7f6550003de0 secure :-1 s=READY pgs=1130 cs=0 l=1 rev1=1 crypto rx=0x7f655c009ae0 tx=0x7f655c01ca40 comp rx=0 tx=0).stop 2024-03-31T15:10:03.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:03.056+0000 7f656a800640 1 -- 172.21.15.16:0/2020951601 reap_dead start 2024-03-31T15:10:03.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:03.056+0000 7f6570e00640 1 -- 172.21.15.16:0/2020951601 shutdown_connections 2024-03-31T15:10:03.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:03.056+0000 7f6570e00640 1 -- 172.21.15.16:0/2020951601 >> 172.21.15.16:0/2020951601 conn(0x7f656c06b620 msgr2=0x7f656c06ba30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:10:03.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:03.056+0000 7f6570e00640 1 -- 172.21.15.16:0/2020951601 shutdown_connections 2024-03-31T15:10:03.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:03.056+0000 7f6570e00640 1 -- 172.21.15.16:0/2020951601 wait complete. 2024-03-31T15:10:03.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:03.060+0000 7f6570e00640 1 Processor -- start 2024-03-31T15:10:03.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:03.060+0000 7f6570e00640 1 -- start start 2024-03-31T15:10:03.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:03.060+0000 7f6570e00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f65500039c0 0x7f656c152f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:03.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:03.060+0000 7f6570e00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f656c1400e0 0x7f656c15c0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:03.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:03.060+0000 7f6570e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f656c1f3540 0x7f656c150490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:03.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:03.060+0000 7f6570e00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f656c0b2050 con 0x7f656c1f3540 2024-03-31T15:10:03.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:03.060+0000 7f6570e00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f656c0b1ed0 con 0x7f65500039c0 2024-03-31T15:10:03.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:03.060+0000 7f6570e00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f656c0cff80 con 0x7f656c1400e0 2024-03-31T15:10:03.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:03.060+0000 7f656a800640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f656c1f3540 0x7f656c150490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:03.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:03.060+0000 7f6569e00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f65500039c0 0x7f656c152f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:03.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:03.060+0000 7f656a800640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f656c1f3540 0x7f656c150490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39402/0 (socket says 172.21.15.16:39402) 2024-03-31T15:10:03.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:03.060+0000 7f656a800640 1 -- 172.21.15.16:0/2320444777 learned_addr learned my addr 172.21.15.16:0/2320444777 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:10:03.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:03.060+0000 7f6563e00640 1 --2- 172.21.15.16:0/2320444777 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f656c1400e0 0x7f656c15c0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:03.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:03.060+0000 7f656a800640 1 -- 172.21.15.16:0/2320444777 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f656c1400e0 msgr2=0x7f656c15c0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:10:03.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:03.060+0000 7f656a800640 1 --2- 172.21.15.16:0/2320444777 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f656c1400e0 0x7f656c15c0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:03.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:03.060+0000 7f656a800640 1 -- 172.21.15.16:0/2320444777 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f65500039c0 msgr2=0x7f656c152f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:10:03.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:03.060+0000 7f656a800640 1 --2- 172.21.15.16:0/2320444777 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f65500039c0 0x7f656c152f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:03.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:03.060+0000 7f656a800640 1 -- 172.21.15.16:0/2320444777 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f656c140560 con 0x7f656c1f3540 2024-03-31T15:10:03.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:03.060+0000 7f6569e00640 1 --2- 172.21.15.16:0/2320444777 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f65500039c0 0x7f656c152f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:10:03.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:03.060+0000 7f656a800640 1 --2- 172.21.15.16:0/2320444777 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f656c1f3540 0x7f656c150490 secure :-1 s=READY pgs=1131 cs=0 l=1 rev1=1 crypto rx=0x7f656c0b5720 tx=0x7f6564011890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:10:03.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:03.060+0000 7f6562000640 1 -- 172.21.15.16:0/2320444777 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65640039a0 con 0x7f656c1f3540 2024-03-31T15:10:03.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:03.064+0000 7f6562000640 1 -- 172.21.15.16:0/2320444777 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f656401b070 con 0x7f656c1f3540 2024-03-31T15:10:03.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:03.064+0000 7f6570e00640 1 -- 172.21.15.16:0/2320444777 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f656c15c5f0 con 0x7f656c1f3540 2024-03-31T15:10:03.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:03.064+0000 7f6562000640 1 -- 172.21.15.16:0/2320444777 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6564003b60 con 0x7f656c1f3540 2024-03-31T15:10:03.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:03.064+0000 7f6570e00640 1 -- 172.21.15.16:0/2320444777 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f656c158ea0 con 0x7f656c1f3540 2024-03-31T15:10:03.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:03.064+0000 7f6570e00640 1 -- 172.21.15.16:0/2320444777 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f656c069950 con 0x7f656c1f3540 2024-03-31T15:10:03.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:03.064+0000 7f6562000640 1 -- 172.21.15.16:0/2320444777 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f6564003d00 con 0x7f656c1f3540 2024-03-31T15:10:03.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:03.068+0000 7f6562000640 1 --2- 172.21.15.16:0/2320444777 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f65480683b0 0x7f654806a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:03.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:03.068+0000 7f6562000640 1 -- 172.21.15.16:0/2320444777 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 5709+0+0 (secure 0 0 0) 0x7f6564087280 con 0x7f656c1f3540 2024-03-31T15:10:03.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:03.068+0000 7f6569e00640 1 --2- 172.21.15.16:0/2320444777 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f65480683b0 0x7f654806a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:03.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:03.068+0000 7f6569e00640 1 --2- 172.21.15.16:0/2320444777 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f65480683b0 0x7f654806a870 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f655c0af000 tx=0x7f655c0046d0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:10:03.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:03.072+0000 7f6562000640 1 -- 172.21.15.16:0/2320444777 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f6564025020 con 0x7f656c1f3540 2024-03-31T15:10:03.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:03.244+0000 7f6570e00640 1 -- 172.21.15.16:0/2320444777 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f656c0cdea0 con 0x7f656c1f3540 2024-03-31T15:10:03.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:03.244+0000 7f6562000640 1 -- 172.21.15.16:0/2320444777 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v81) ==== 80+0+0 (secure 0 0 0) 0x7f656c140560 con 0x7f656c1f3540 2024-03-31T15:10:03.238 INFO:teuthology.orchestra.run.smithi016.stderr:listed 0 entries 2024-03-31T15:10:03.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:03.248+0000 7f6570e00640 1 -- 172.21.15.16:0/2320444777 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f65480683b0 msgr2=0x7f654806a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:10:03.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:03.248+0000 7f6570e00640 1 --2- 172.21.15.16:0/2320444777 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f65480683b0 0x7f654806a870 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f655c0af000 tx=0x7f655c0046d0 comp rx=0 tx=0).stop 2024-03-31T15:10:03.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:03.248+0000 7f6570e00640 1 -- 172.21.15.16:0/2320444777 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f656c1f3540 msgr2=0x7f656c150490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:10:03.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:03.248+0000 7f6570e00640 1 --2- 172.21.15.16:0/2320444777 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f656c1f3540 0x7f656c150490 secure :-1 s=READY pgs=1131 cs=0 l=1 rev1=1 crypto rx=0x7f656c0b5720 tx=0x7f6564011890 comp rx=0 tx=0).stop 2024-03-31T15:10:03.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:03.248+0000 7f6570e00640 1 -- 172.21.15.16:0/2320444777 shutdown_connections 2024-03-31T15:10:03.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:03.248+0000 7f6570e00640 1 --2- 172.21.15.16:0/2320444777 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f65480683b0 0x7f654806a870 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:03.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:03.248+0000 7f6570e00640 1 --2- 172.21.15.16:0/2320444777 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f656c1f3540 0x7f656c150490 unknown :-1 s=CLOSED pgs=1131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:03.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:03.248+0000 7f6570e00640 1 --2- 172.21.15.16:0/2320444777 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f656c1400e0 0x7f656c15c0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:03.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:03.248+0000 7f6570e00640 1 --2- 172.21.15.16:0/2320444777 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f65500039c0 0x7f656c152f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:03.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:03.248+0000 7f6570e00640 1 -- 172.21.15.16:0/2320444777 >> 172.21.15.16:0/2320444777 conn(0x7f656c06b620 msgr2=0x7f656c0b34f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:10:03.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:03.248+0000 7f6570e00640 1 -- 172.21.15.16:0/2320444777 shutdown_connections 2024-03-31T15:10:03.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:03.248+0000 7f6570e00640 1 -- 172.21.15.16:0/2320444777 wait complete. 2024-03-31T15:10:03.259 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi016.front.sepia.ceph.com... 2024-03-31T15:10:03.260 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:10:03.260 DEBUG:teuthology.orchestra.run.smithi016:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-03-31T15:10:03.277 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi016.stderr:ceph-fuse[44750]: fuse finished with error 0 and tester_r 0 2024-03-31T15:10:03.280 INFO:teuthology.orchestra.run:waiting for 300 2024-03-31T15:10:03.369 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi016.front.sepia.ceph.com 2024-03-31T15:10:03.369 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:10:03.369 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-31T15:10:03.378 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-31T15:10:03.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:03.580+0000 7f6582a00640 1 Processor -- start 2024-03-31T15:10:03.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:03.580+0000 7f6582a00640 1 -- start start 2024-03-31T15:10:03.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:03.580+0000 7f6582a00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f657c0b23d0 0x7f657c071d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:03.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:03.580+0000 7f6582a00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f657c15eee0 0x7f657c072250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:03.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:03.580+0000 7f6582a00640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f657c15f3c0 con 0x7f657c15b3d0 2024-03-31T15:10:03.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:03.580+0000 7f6582a00640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f657c15b7b0 con 0x7f657c15eee0 2024-03-31T15:10:03.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:03.580+0000 7f6582a00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f657c15f6a0 con 0x7f657c0b23d0 2024-03-31T15:10:03.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:03.580+0000 7f6580a00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f657c15eee0 0x7f657c072250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:03.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:03.580+0000 7f6580a00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f657c15eee0 0x7f657c072250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39408/0 (socket says 172.21.15.16:39408) 2024-03-31T15:10:03.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:03.580+0000 7f6580a00640 1 -- 172.21.15.16:0/2412086915 learned_addr learned my addr 172.21.15.16:0/2412086915 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:10:03.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:03.580+0000 7f6573e00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f657c0b23d0 0x7f657c071d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:03.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:03.584+0000 7f6580a00640 1 -- 172.21.15.16:0/2412086915 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f657c0b23d0 msgr2=0x7f657c071d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:10:03.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:03.584+0000 7f6580a00640 1 --2- 172.21.15.16:0/2412086915 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f657c0b23d0 0x7f657c071d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:03.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:03.584+0000 7f6580a00640 1 -- 172.21.15.16:0/2412086915 >> v1:172.21.15.89:6789/0 conn(0x7f657c15b3d0 legacy=0x7f657c0b1cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:10:03.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:03.584+0000 7f6580a00640 1 -- 172.21.15.16:0/2412086915 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f657c15f980 con 0x7f657c15eee0 2024-03-31T15:10:03.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:03.588+0000 7f6580a00640 1 --2- 172.21.15.16:0/2412086915 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f657c15eee0 0x7f657c072250 secure :-1 s=READY pgs=1132 cs=0 l=1 rev1=1 crypto rx=0x7f6574009e50 tx=0x7f657401ca00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d2d11328ed40ed96 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:10:03.580 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:03.588+0000 7f657b400640 1 -- 172.21.15.16:0/2412086915 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65740a5890 con 0x7f657c15eee0 2024-03-31T15:10:03.580 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:03.588+0000 7f657b400640 1 -- 172.21.15.16:0/2412086915 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f65740a5a30 con 0x7f657c15eee0 2024-03-31T15:10:03.580 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:03.588+0000 7f657b400640 1 -- 172.21.15.16:0/2412086915 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65740a5e00 con 0x7f657c15eee0 2024-03-31T15:10:03.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:03.588+0000 7f6582a00640 1 -- 172.21.15.16:0/2412086915 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f657c15eee0 msgr2=0x7f657c072250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:10:03.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:03.588+0000 7f6582a00640 1 --2- 172.21.15.16:0/2412086915 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f657c15eee0 0x7f657c072250 secure :-1 s=READY pgs=1132 cs=0 l=1 rev1=1 crypto rx=0x7f6574009e50 tx=0x7f657401ca00 comp rx=0 tx=0).stop 2024-03-31T15:10:03.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:03.588+0000 7f6582a00640 1 -- 172.21.15.16:0/2412086915 shutdown_connections 2024-03-31T15:10:03.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:03.588+0000 7f6582a00640 1 --2- 172.21.15.16:0/2412086915 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f657c15eee0 0x7f657c072250 unknown :-1 s=CLOSED pgs=1132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:03.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:03.588+0000 7f6582a00640 1 --2- 172.21.15.16:0/2412086915 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f657c0b23d0 0x7f657c071d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:03.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:03.588+0000 7f6582a00640 1 -- 172.21.15.16:0/2412086915 >> 172.21.15.16:0/2412086915 conn(0x7f657c06b620 msgr2=0x7f657c06ba30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:10:03.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:03.588+0000 7f6582a00640 1 -- 172.21.15.16:0/2412086915 shutdown_connections 2024-03-31T15:10:03.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:03.588+0000 7f6582a00640 1 -- 172.21.15.16:0/2412086915 wait complete. 2024-03-31T15:10:03.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:03.592+0000 7f6582a00640 1 Processor -- start 2024-03-31T15:10:03.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:03.592+0000 7f6582a00640 1 -- start start 2024-03-31T15:10:03.585 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:03.592+0000 7f6582a00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f657c0b23d0 0x7f657c0b1d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:03.585 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:03.592+0000 7f6582a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f657c15b3d0 0x7f657c158d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:03.585 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:03.592+0000 7f6582a00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f657c15eee0 0x7f657c14d170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:03.585 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:03.592+0000 7f6582a00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f657c073910 con 0x7f657c15b3d0 2024-03-31T15:10:03.585 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:03.592+0000 7f6582a00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f657c073790 con 0x7f657c15eee0 2024-03-31T15:10:03.585 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:03.592+0000 7f6582a00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f657c072790 con 0x7f657c0b23d0 2024-03-31T15:10:03.585 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:03.592+0000 7f6573e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f657c15b3d0 0x7f657c158d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:03.585 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:03.592+0000 7f6573e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f657c15b3d0 0x7f657c158d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39418/0 (socket says 172.21.15.16:39418) 2024-03-31T15:10:03.585 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:03.592+0000 7f6573e00640 1 -- 172.21.15.16:0/653227514 learned_addr learned my addr 172.21.15.16:0/653227514 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:10:03.585 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:03.592+0000 7f657be00640 1 --2- 172.21.15.16:0/653227514 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f657c0b23d0 0x7f657c0b1d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:03.585 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:03.592+0000 7f6580a00640 1 --2- 172.21.15.16:0/653227514 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f657c15eee0 0x7f657c14d170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:03.585 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:03.592+0000 7f6573e00640 1 -- 172.21.15.16:0/653227514 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f657c0b23d0 msgr2=0x7f657c0b1d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:10:03.586 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:03.592+0000 7f6573e00640 1 --2- 172.21.15.16:0/653227514 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f657c0b23d0 0x7f657c0b1d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:03.586 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:03.592+0000 7f6573e00640 1 -- 172.21.15.16:0/653227514 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f657c15eee0 msgr2=0x7f657c14d170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:10:03.586 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:03.592+0000 7f6573e00640 1 --2- 172.21.15.16:0/653227514 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f657c15eee0 0x7f657c14d170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:03.586 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:03.592+0000 7f6573e00640 1 -- 172.21.15.16:0/653227514 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f657c14df60 con 0x7f657c15b3d0 2024-03-31T15:10:03.587 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:03.592+0000 7f6573e00640 1 --2- 172.21.15.16:0/653227514 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f657c15b3d0 0x7f657c158d00 secure :-1 s=READY pgs=1133 cs=0 l=1 rev1=1 crypto rx=0x7f656400e960 tx=0x7f656400ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:10:03.587 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:03.592+0000 7f6578c00640 1 -- 172.21.15.16:0/653227514 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6564017070 con 0x7f657c15b3d0 2024-03-31T15:10:03.587 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:03.592+0000 7f6578c00640 1 -- 172.21.15.16:0/653227514 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f656400ca00 con 0x7f657c15b3d0 2024-03-31T15:10:03.587 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:03.592+0000 7f6578c00640 1 -- 172.21.15.16:0/653227514 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f656400cdb0 con 0x7f657c15b3d0 2024-03-31T15:10:03.587 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:03.592+0000 7f6582a00640 1 -- 172.21.15.16:0/653227514 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f657c14e240 con 0x7f657c15b3d0 2024-03-31T15:10:03.587 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:03.592+0000 7f6582a00640 1 -- 172.21.15.16:0/653227514 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f657c159240 con 0x7f657c15b3d0 2024-03-31T15:10:03.587 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:03.596+0000 7f6582a00640 1 -- 172.21.15.16:0/653227514 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f657c069950 con 0x7f657c15b3d0 2024-03-31T15:10:03.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:03.596+0000 7f6578c00640 1 -- 172.21.15.16:0/653227514 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f656401a070 con 0x7f657c15b3d0 2024-03-31T15:10:03.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:03.596+0000 7f6578c00640 1 --2- 172.21.15.16:0/653227514 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f65580683b0 0x7f655806a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:03.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:03.596+0000 7f6578c00640 1 -- 172.21.15.16:0/653227514 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 5709+0+0 (secure 0 0 0) 0x7f6564086cb0 con 0x7f657c15b3d0 2024-03-31T15:10:03.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:03.596+0000 7f657be00640 1 --2- 172.21.15.16:0/653227514 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f65580683b0 0x7f655806a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:03.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:03.596+0000 7f657be00640 1 --2- 172.21.15.16:0/653227514 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f65580683b0 0x7f655806a870 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f656c007fd0 tx=0x7f656c009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:10:03.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:03.600+0000 7f6578c00640 1 -- 172.21.15.16:0/653227514 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f6564056c10 con 0x7f657c15b3d0 2024-03-31T15:10:03.790 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:03.796+0000 7f6582a00640 1 -- 172.21.15.16:0/653227514 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f657c0cdea0 con 0x7f657c15b3d0 2024-03-31T15:10:03.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:03.800+0000 7f6578c00640 1 -- 172.21.15.16:0/653227514 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 61 v61) ==== 94+0+5183 (secure 0 0 0) 0x7f656405af10 con 0x7f657c15b3d0 2024-03-31T15:10:03.791 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:10:03.791 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":61,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6494,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/2445558494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6835","nonce":2445558494},{"type":"v1","addr":"172.21.15.89:6837","nonce":2445558494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":56},{"gid":7270,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/3185913858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6833","nonce":3185913858},{"type":"v1","addr":"172.21.15.16:6835","nonce":3185913858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":56},{"gid":7279,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6836/1599468493","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":1599468493},{"type":"v1","addr":"172.21.15.89:6836","nonce":1599468493}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":56}],"filesystems":[{"mdsmap":{"epoch":61,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-31T15:09:50.926196+0000","modified":"2024-03-31T15:09:54.012516+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7479},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7479":{"gid":7479,"name":"c","rank":0,"incarnation":57,"state":"up:active","state_seq":3,"addr":"172.21.15.16:6834/2478530200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":2478530200},{"type":"v1","addr":"172.21.15.16:6834","nonce":2478530200}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7479,"qdb_cluster":[7479]},"id":6}]} 2024-03-31T15:10:03.791 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 61 2024-03-31T15:10:03.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:03.800+0000 7f6582a00640 1 -- 172.21.15.16:0/653227514 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f65580683b0 msgr2=0x7f655806a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:10:03.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:03.800+0000 7f6582a00640 1 --2- 172.21.15.16:0/653227514 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f65580683b0 0x7f655806a870 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f656c007fd0 tx=0x7f656c009040 comp rx=0 tx=0).stop 2024-03-31T15:10:03.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:03.800+0000 7f6582a00640 1 -- 172.21.15.16:0/653227514 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f657c15b3d0 msgr2=0x7f657c158d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:10:03.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:03.800+0000 7f6582a00640 1 --2- 172.21.15.16:0/653227514 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f657c15b3d0 0x7f657c158d00 secure :-1 s=READY pgs=1133 cs=0 l=1 rev1=1 crypto rx=0x7f656400e960 tx=0x7f656400ee20 comp rx=0 tx=0).stop 2024-03-31T15:10:03.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:03.800+0000 7f6582a00640 1 -- 172.21.15.16:0/653227514 shutdown_connections 2024-03-31T15:10:03.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:03.800+0000 7f6582a00640 1 --2- 172.21.15.16:0/653227514 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f65580683b0 0x7f655806a870 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:03.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:03.800+0000 7f6582a00640 1 --2- 172.21.15.16:0/653227514 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f657c15eee0 0x7f657c14d170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:03.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:03.800+0000 7f6582a00640 1 --2- 172.21.15.16:0/653227514 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f657c15b3d0 0x7f657c158d00 unknown :-1 s=CLOSED pgs=1133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:03.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:03.800+0000 7f6582a00640 1 --2- 172.21.15.16:0/653227514 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f657c0b23d0 0x7f657c0b1d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:03.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:03.800+0000 7f6582a00640 1 -- 172.21.15.16:0/653227514 >> 172.21.15.16:0/653227514 conn(0x7f657c06b620 msgr2=0x7f657c0ce700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:10:03.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:03.800+0000 7f6582a00640 1 -- 172.21.15.16:0/653227514 shutdown_connections 2024-03-31T15:10:03.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:03.800+0000 7f6582a00640 1 -- 172.21.15.16:0/653227514 wait complete. 2024-03-31T15:10:03.813 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json flush journal 2024-03-31T15:10:03.984 INFO:teuthology.orchestra.run.smithi016.stdout:{"message":"","return_code":0} 2024-03-31T15:10:03.985 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2024-03-31T15:10:03.985 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-31T15:10:04.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.136+0000 7f46e9400640 1 Processor -- start 2024-03-31T15:10:04.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.136+0000 7f46e9400640 1 -- start start 2024-03-31T15:10:04.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.136+0000 7f46e9400640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f46e415c1d0 0x7f46e40b44e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:04.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.136+0000 7f46e9400640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f46e40cd980 con 0x7f46e40cd4f0 2024-03-31T15:10:04.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.136+0000 7f46e9400640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f46e40cdc60 con 0x7f46e415fd40 2024-03-31T15:10:04.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.136+0000 7f46e9400640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f46e40cfaa0 con 0x7f46e415c1d0 2024-03-31T15:10:04.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.136+0000 7f46db400640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f46e40cd4f0 0x7f46e406f5d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:57766/0 (socket says 172.21.15.16:57766) 2024-03-31T15:10:04.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.136+0000 7f46db400640 1 -- 172.21.15.16:0/781166693 learned_addr learned my addr 172.21.15.16:0/781166693 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:10:04.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.136+0000 7f46e3e00640 1 --2- 172.21.15.16:0/781166693 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f46e415c1d0 0x7f46e40b44e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:04.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.136+0000 7f46e3400640 1 -- 172.21.15.16:0/781166693 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2267875628 0 0) 0x7f46e40cd980 con 0x7f46e40cd4f0 2024-03-31T15:10:04.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.136+0000 7f46e3400640 1 -- 172.21.15.16:0/781166693 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f46c8003530 con 0x7f46e40cd4f0 2024-03-31T15:10:04.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.136+0000 7f46e3400640 1 -- 172.21.15.16:0/781166693 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1808279013 0 0) 0x7f46c8003530 con 0x7f46e40cd4f0 2024-03-31T15:10:04.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.136+0000 7f46e3400640 1 -- 172.21.15.16:0/781166693 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f46e415c1d0 msgr2=0x7f46e40b44e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:10:04.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.136+0000 7f46e3400640 1 --2- 172.21.15.16:0/781166693 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f46e415c1d0 0x7f46e40b44e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:04.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.136+0000 7f46e3400640 1 -- 172.21.15.16:0/781166693 >> v1:172.21.15.89:6789/0 conn(0x7f46e415fd40 legacy=0x7f46e40d04f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:10:04.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.136+0000 7f46e3400640 1 -- 172.21.15.16:0/781166693 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46e40cfcf0 con 0x7f46e40cd4f0 2024-03-31T15:10:04.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.136+0000 7f46e3400640 1 -- 172.21.15.16:0/781166693 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 214807623 0 0) 0x7f46cc001a00 con 0x7f46e40cd4f0 2024-03-31T15:10:04.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.136+0000 7f46e3400640 1 -- 172.21.15.16:0/781166693 >> v1:172.21.15.16:6789/0 conn(0x7f46e40cd4f0 legacy=0x7f46e406f5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:10:04.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.136+0000 7f46e3400640 1 --2- 172.21.15.16:0/781166693 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f46e415c1d0 0x7f46c80039f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:04.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.140+0000 7f46e3400640 1 --2- 172.21.15.16:0/781166693 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f46c8003f30 0x7f46c8004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:04.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.140+0000 7f46e3400640 1 --2- 172.21.15.16:0/781166693 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f46c8004d50 0x7f46c80051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:04.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.140+0000 7f46e3400640 1 -- 172.21.15.16:0/781166693 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f46e40cfaa0 con 0x7f46c8003f30 2024-03-31T15:10:04.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.140+0000 7f46e3400640 1 -- 172.21.15.16:0/781166693 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f46c800ba30 con 0x7f46c8004d50 2024-03-31T15:10:04.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.140+0000 7f46e3400640 1 -- 172.21.15.16:0/781166693 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f46c800bc30 con 0x7f46e415c1d0 2024-03-31T15:10:04.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.140+0000 7f46db400640 1 --2- 172.21.15.16:0/781166693 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f46c8003f30 0x7f46c8004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:04.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.140+0000 7f46e3e00640 1 --2- 172.21.15.16:0/781166693 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f46e415c1d0 0x7f46c80039f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:04.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.140+0000 7f46e8a00640 1 --2- 172.21.15.16:0/781166693 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f46c8004d50 0x7f46c80051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:04.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.140+0000 7f46db400640 1 -- 172.21.15.16:0/781166693 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f46e415c1d0 msgr2=0x7f46c80039f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:10:04.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.140+0000 7f46db400640 1 --2- 172.21.15.16:0/781166693 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f46e415c1d0 0x7f46c80039f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:04.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.140+0000 7f46db400640 1 -- 172.21.15.16:0/781166693 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f46c8004d50 msgr2=0x7f46c80051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:10:04.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.140+0000 7f46db400640 1 --2- 172.21.15.16:0/781166693 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f46c8004d50 0x7f46c80051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:04.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.140+0000 7f46db400640 1 -- 172.21.15.16:0/781166693 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46c8003530 con 0x7f46c8003f30 2024-03-31T15:10:04.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.144+0000 7f46db400640 1 --2- 172.21.15.16:0/781166693 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f46c8003f30 0x7f46c8004810 secure :-1 s=READY pgs=1135 cs=0 l=1 rev1=1 crypto rx=0x7f46cc00a810 tx=0x7f46cc01d230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84eecdf0206bab46 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:10:04.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.144+0000 7f46e3400640 1 -- 172.21.15.16:0/781166693 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46cc00a430 con 0x7f46c8003f30 2024-03-31T15:10:04.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.144+0000 7f46e3400640 1 -- 172.21.15.16:0/781166693 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f46cc0ad630 con 0x7f46c8003f30 2024-03-31T15:10:04.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.144+0000 7f46e3400640 1 -- 172.21.15.16:0/781166693 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46cc0ada60 con 0x7f46c8003f30 2024-03-31T15:10:04.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.144+0000 7f46e9400640 1 -- 172.21.15.16:0/781166693 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f46c8003f30 msgr2=0x7f46c8004810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:10:04.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.144+0000 7f46e9400640 1 --2- 172.21.15.16:0/781166693 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f46c8003f30 0x7f46c8004810 secure :-1 s=READY pgs=1135 cs=0 l=1 rev1=1 crypto rx=0x7f46cc00a810 tx=0x7f46cc01d230 comp rx=0 tx=0).stop 2024-03-31T15:10:04.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.144+0000 7f46e8a00640 1 -- 172.21.15.16:0/781166693 reap_dead start 2024-03-31T15:10:04.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.144+0000 7f46e9400640 1 -- 172.21.15.16:0/781166693 shutdown_connections 2024-03-31T15:10:04.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.144+0000 7f46e9400640 1 -- 172.21.15.16:0/781166693 >> 172.21.15.16:0/781166693 conn(0x7f46e40b39c0 msgr2=0x7f46e40b3dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:10:04.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.144+0000 7f46e9400640 1 -- 172.21.15.16:0/781166693 shutdown_connections 2024-03-31T15:10:04.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.144+0000 7f46e9400640 1 -- 172.21.15.16:0/781166693 wait complete. 2024-03-31T15:10:04.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.148+0000 7f46e9400640 1 Processor -- start 2024-03-31T15:10:04.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.148+0000 7f46e9400640 1 -- start start 2024-03-31T15:10:04.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.148+0000 7f46e9400640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f46c8003f30 0x7f46e4159d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:04.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.148+0000 7f46e9400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f46e406c970 0x7f46e40b19c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:04.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.148+0000 7f46e9400640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f46e406cdf0 0x7f46e40b1f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:04.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.148+0000 7f46e9400640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f46e406fce0 con 0x7f46e406c970 2024-03-31T15:10:04.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.148+0000 7f46e9400640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f46e40b48f0 con 0x7f46e406cdf0 2024-03-31T15:10:04.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.148+0000 7f46e9400640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f46e406e460 con 0x7f46c8003f30 2024-03-31T15:10:04.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.148+0000 7f46db400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f46e406c970 0x7f46e40b19c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:04.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.148+0000 7f46db400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f46e406c970 0x7f46e40b19c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39436/0 (socket says 172.21.15.16:39436) 2024-03-31T15:10:04.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.148+0000 7f46db400640 1 -- 172.21.15.16:0/2617056023 learned_addr learned my addr 172.21.15.16:0/2617056023 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:10:04.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.148+0000 7f46e8a00640 1 --2- 172.21.15.16:0/2617056023 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f46e406cdf0 0x7f46e40b1f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:04.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.148+0000 7f46e3e00640 1 --2- 172.21.15.16:0/2617056023 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f46c8003f30 0x7f46e4159d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:04.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.148+0000 7f46db400640 1 -- 172.21.15.16:0/2617056023 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f46c8003f30 msgr2=0x7f46e4159d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:10:04.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.148+0000 7f46db400640 1 --2- 172.21.15.16:0/2617056023 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f46c8003f30 0x7f46e4159d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:04.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.148+0000 7f46db400640 1 -- 172.21.15.16:0/2617056023 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f46e406cdf0 msgr2=0x7f46e40b1f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:10:04.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.148+0000 7f46db400640 1 --2- 172.21.15.16:0/2617056023 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f46e406cdf0 0x7f46e40b1f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:04.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.148+0000 7f46db400640 1 -- 172.21.15.16:0/2617056023 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46e406d3c0 con 0x7f46e406c970 2024-03-31T15:10:04.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.148+0000 7f46db400640 1 --2- 172.21.15.16:0/2617056023 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f46e406c970 0x7f46e40b19c0 secure :-1 s=READY pgs=1136 cs=0 l=1 rev1=1 crypto rx=0x7f46cc01cca0 tx=0x7f46c8005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:10:04.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.148+0000 7f46e0c00640 1 -- 172.21.15.16:0/2617056023 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46cc0017c0 con 0x7f46e406c970 2024-03-31T15:10:04.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.148+0000 7f46e0c00640 1 -- 172.21.15.16:0/2617056023 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f46cc0b3040 con 0x7f46e406c970 2024-03-31T15:10:04.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.148+0000 7f46e0c00640 1 -- 172.21.15.16:0/2617056023 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46cc004b70 con 0x7f46e406c970 2024-03-31T15:10:04.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.148+0000 7f46e9400640 1 -- 172.21.15.16:0/2617056023 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f46e406d6a0 con 0x7f46e406c970 2024-03-31T15:10:04.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.148+0000 7f46e9400640 1 -- 172.21.15.16:0/2617056023 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f46e406dc60 con 0x7f46e406c970 2024-03-31T15:10:04.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.148+0000 7f46e9400640 1 -- 172.21.15.16:0/2617056023 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f46e404f8c0 con 0x7f46e406c970 2024-03-31T15:10:04.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.152+0000 7f46e0c00640 1 -- 172.21.15.16:0/2617056023 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f46cc0c0060 con 0x7f46e406c970 2024-03-31T15:10:04.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.152+0000 7f46e0c00640 1 --2- 172.21.15.16:0/2617056023 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f46c00683b0 0x7f46c006a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:04.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.152+0000 7f46e0c00640 1 -- 172.21.15.16:0/2617056023 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 5709+0+0 (secure 0 0 0) 0x7f46cc1210e0 con 0x7f46e406c970 2024-03-31T15:10:04.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.152+0000 7f46e3e00640 1 --2- 172.21.15.16:0/2617056023 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f46c00683b0 0x7f46c006a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:04.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.152+0000 7f46e3e00640 1 --2- 172.21.15.16:0/2617056023 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f46c00683b0 0x7f46c006a870 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f46d4009be0 tx=0x7f46d4009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:10:04.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.156+0000 7f46e0c00640 1 -- 172.21.15.16:0/2617056023 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f46cc016f70 con 0x7f46e406c970 2024-03-31T15:10:04.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.356+0000 7f46e9400640 1 -- 172.21.15.16:0/2617056023 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f46e4068ae0 con 0x7f46e406c970 2024-03-31T15:10:04.350 INFO:tasks.ceph.mon.a.smithi016.stderr:2024-03-31T15:10:04.356+0000 7f7dcbc00640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-31T15:10:04.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.360+0000 7f46e0c00640 1 -- 172.21.15.16:0/2617056023 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v62) ==== 161+0+0 (secure 0 0 0) 0x7f46cc0f1690 con 0x7f46e406c970 2024-03-31T15:10:04.352 INFO:teuthology.orchestra.run.smithi016.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-31T15:10:04.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.364+0000 7f46e9400640 1 -- 172.21.15.16:0/2617056023 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f46c00683b0 msgr2=0x7f46c006a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:10:04.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.364+0000 7f46e9400640 1 --2- 172.21.15.16:0/2617056023 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f46c00683b0 0x7f46c006a870 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f46d4009be0 tx=0x7f46d4009190 comp rx=0 tx=0).stop 2024-03-31T15:10:04.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.364+0000 7f46e9400640 1 -- 172.21.15.16:0/2617056023 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f46e406c970 msgr2=0x7f46e40b19c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:10:04.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.364+0000 7f46e9400640 1 --2- 172.21.15.16:0/2617056023 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f46e406c970 0x7f46e40b19c0 secure :-1 s=READY pgs=1136 cs=0 l=1 rev1=1 crypto rx=0x7f46cc01cca0 tx=0x7f46c8005840 comp rx=0 tx=0).stop 2024-03-31T15:10:04.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.364+0000 7f46e9400640 1 -- 172.21.15.16:0/2617056023 shutdown_connections 2024-03-31T15:10:04.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.364+0000 7f46e9400640 1 --2- 172.21.15.16:0/2617056023 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f46c00683b0 0x7f46c006a870 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:04.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.364+0000 7f46e9400640 1 --2- 172.21.15.16:0/2617056023 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f46e406cdf0 0x7f46e40b1f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:04.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.364+0000 7f46e9400640 1 --2- 172.21.15.16:0/2617056023 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f46e406c970 0x7f46e40b19c0 unknown :-1 s=CLOSED pgs=1136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:04.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.364+0000 7f46e9400640 1 --2- 172.21.15.16:0/2617056023 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f46c8003f30 0x7f46e4159d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:04.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.364+0000 7f46e9400640 1 -- 172.21.15.16:0/2617056023 >> 172.21.15.16:0/2617056023 conn(0x7f46e40b39c0 msgr2=0x7f46e40b2da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:10:04.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.364+0000 7f46e9400640 1 -- 172.21.15.16:0/2617056023 shutdown_connections 2024-03-31T15:10:04.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.364+0000 7f46e9400640 1 -- 172.21.15.16:0/2617056023 wait complete. 2024-03-31T15:10:04.362 INFO:tasks.ceph.mds.c.smithi016.stderr: -3720> 2024-03-31T15:09:57.508+0000 7f9ed8600640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-31T15:10:04.376 DEBUG:teuthology.orchestra.run.smithi016:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata rm 10000000000.00000000 2024-03-31T15:10:04.412 INFO:tasks.ceph.mds.c.smithi016.stdout:starting mds.c at 2024-03-31T15:10:04.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.432+0000 7faec2a48080 1 Processor -- start 2024-03-31T15:10:04.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.432+0000 7faec2a48080 1 -- start start 2024-03-31T15:10:04.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.432+0000 7faec2a48080 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x558e26951e70 0x558e269f3f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:04.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.432+0000 7faec2a48080 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558e2696fc60 0x558e269f5570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:04.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.432+0000 7faec2a48080 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558e26970040 0x558e269f5ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:04.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.432+0000 7faec2a48080 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x558e26954390 con 0x558e26970040 2024-03-31T15:10:04.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.432+0000 7faec2a48080 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x558e26954210 con 0x558e2696fc60 2024-03-31T15:10:04.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.432+0000 7faec2a48080 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x558e26954510 con 0x558e26951e70 2024-03-31T15:10:04.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.432+0000 7faebfa00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558e26970040 0x558e269f5ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:04.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.432+0000 7faebf000640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x558e26951e70 0x558e269f3f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:04.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.432+0000 7faebe600640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558e2696fc60 0x558e269f5570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:04.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.432+0000 7faebfa00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558e26970040 0x558e269f5ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39468/0 (socket says 172.21.15.16:39468) 2024-03-31T15:10:04.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.432+0000 7faebfa00640 1 -- 172.21.15.16:0/2905855697 learned_addr learned my addr 172.21.15.16:0/2905855697 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:10:04.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.432+0000 7faebfa00640 1 -- 172.21.15.16:0/2905855697 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x558e26951e70 msgr2=0x558e269f3f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:10:04.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.432+0000 7faebfa00640 1 --2- 172.21.15.16:0/2905855697 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x558e26951e70 0x558e269f3f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:04.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.432+0000 7faebfa00640 1 -- 172.21.15.16:0/2905855697 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558e2696fc60 msgr2=0x558e269f5570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:10:04.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.432+0000 7faebfa00640 1 --2- 172.21.15.16:0/2905855697 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558e2696fc60 0x558e269f5570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:04.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.432+0000 7faebfa00640 1 -- 172.21.15.16:0/2905855697 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558e2684e950 con 0x558e26970040 2024-03-31T15:10:04.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.432+0000 7faebe600640 1 --2- 172.21.15.16:0/2905855697 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558e2696fc60 0x558e269f5570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:10:04.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.432+0000 7faebfa00640 1 --2- 172.21.15.16:0/2905855697 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558e26970040 0x558e269f5ab0 secure :-1 s=READY pgs=1141 cs=0 l=1 rev1=1 crypto rx=0x7faeb801b4a0 tx=0x7faeb80033b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d8bbe790cdd9b95 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:10:04.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.432+0000 7faebd200640 1 -- 172.21.15.16:0/2905855697 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faeb80b0040 con 0x558e26970040 2024-03-31T15:10:04.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.432+0000 7faebd200640 1 -- 172.21.15.16:0/2905855697 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faeb80ab440 con 0x558e26970040 2024-03-31T15:10:04.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.432+0000 7faebd200640 1 -- 172.21.15.16:0/2905855697 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faeb80ab7f0 con 0x558e26970040 2024-03-31T15:10:04.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.432+0000 7faec2a48080 1 -- 172.21.15.16:0/2905855697 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558e26970040 msgr2=0x558e269f5ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:10:04.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.432+0000 7faec2a48080 1 --2- 172.21.15.16:0/2905855697 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558e26970040 0x558e269f5ab0 secure :-1 s=READY pgs=1141 cs=0 l=1 rev1=1 crypto rx=0x7faeb801b4a0 tx=0x7faeb80033b0 comp rx=0 tx=0).stop 2024-03-31T15:10:04.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.432+0000 7faec2a48080 1 -- 172.21.15.16:0/2905855697 shutdown_connections 2024-03-31T15:10:04.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.432+0000 7faec2a48080 1 --2- 172.21.15.16:0/2905855697 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558e26970040 0x558e269f5ab0 unknown :-1 s=CLOSED pgs=1141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:04.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.432+0000 7faec2a48080 1 --2- 172.21.15.16:0/2905855697 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558e2696fc60 0x558e269f5570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:04.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.432+0000 7faec2a48080 1 --2- 172.21.15.16:0/2905855697 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x558e26951e70 0x558e269f3f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:04.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.432+0000 7faec2a48080 1 -- 172.21.15.16:0/2905855697 >> 172.21.15.16:0/2905855697 conn(0x558e26950420 msgr2=0x558e26922890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:10:04.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.432+0000 7faec2a48080 1 -- 172.21.15.16:0/2905855697 shutdown_connections 2024-03-31T15:10:04.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.432+0000 7faec2a48080 1 -- 172.21.15.16:0/2905855697 wait complete. 2024-03-31T15:10:04.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.436+0000 7faec2a48080 1 Processor -- start 2024-03-31T15:10:04.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.436+0000 7faec2a48080 1 -- start start 2024-03-31T15:10:04.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.436+0000 7faec2a48080 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558e26951e70 0x558e269d7770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:04.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.436+0000 7faec2a48080 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x558e2696fc60 0x558e269d7cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:04.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.436+0000 7faec2a48080 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558e26970040 0x558e269d8940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:04.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.436+0000 7faec2a48080 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x558e26a8b730 con 0x558e26970040 2024-03-31T15:10:04.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.436+0000 7faec2a48080 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x558e269f6a90 con 0x558e26951e70 2024-03-31T15:10:04.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.436+0000 7faec2a48080 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x558e26a8b9d0 con 0x558e2696fc60 2024-03-31T15:10:04.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.436+0000 7faebfa00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558e26970040 0x558e269d8940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:04.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.436+0000 7faebfa00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558e26970040 0x558e269d8940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39482/0 (socket says 172.21.15.16:39482) 2024-03-31T15:10:04.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.436+0000 7faebfa00640 1 -- 172.21.15.16:0/1873248497 learned_addr learned my addr 172.21.15.16:0/1873248497 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:10:04.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.436+0000 7faebf000640 1 --2- 172.21.15.16:0/1873248497 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558e26951e70 0x558e269d7770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:04.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.436+0000 7faebe600640 1 --2- 172.21.15.16:0/1873248497 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x558e2696fc60 0x558e269d7cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:04.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.436+0000 7faebfa00640 1 -- 172.21.15.16:0/1873248497 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x558e2696fc60 msgr2=0x558e269d7cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:10:04.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.436+0000 7faebfa00640 1 --2- 172.21.15.16:0/1873248497 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x558e2696fc60 0x558e269d7cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:04.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.436+0000 7faebfa00640 1 -- 172.21.15.16:0/1873248497 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558e26951e70 msgr2=0x558e269d7770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:10:04.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.436+0000 7faebfa00640 1 --2- 172.21.15.16:0/1873248497 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558e26951e70 0x558e269d7770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:04.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.436+0000 7faebfa00640 1 -- 172.21.15.16:0/1873248497 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558e269d84a0 con 0x558e26970040 2024-03-31T15:10:04.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.436+0000 7faebfa00640 1 --2- 172.21.15.16:0/1873248497 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558e26970040 0x558e269d8940 secure :-1 s=READY pgs=1142 cs=0 l=1 rev1=1 crypto rx=0x7faeb801b000 tx=0x7faeb8017d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:10:04.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.436+0000 7faeb7400640 1 -- 172.21.15.16:0/1873248497 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faeb80b0040 con 0x558e26970040 2024-03-31T15:10:04.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.436+0000 7faeb7400640 1 -- 172.21.15.16:0/1873248497 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faeb80abbc0 con 0x558e26970040 2024-03-31T15:10:04.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.436+0000 7faeb7400640 1 -- 172.21.15.16:0/1873248497 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faeb80c4420 con 0x558e26970040 2024-03-31T15:10:04.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.436+0000 7faec2a48080 1 -- 172.21.15.16:0/1873248497 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x558e26a8c220 con 0x558e26970040 2024-03-31T15:10:04.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.436+0000 7faec2a48080 1 -- 172.21.15.16:0/1873248497 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558e26a8c760 con 0x558e26970040 2024-03-31T15:10:04.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.440+0000 7faeb7400640 1 -- 172.21.15.16:0/1873248497 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7faeb80c45c0 con 0x558e26970040 2024-03-31T15:10:04.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.440+0000 7faeb7400640 1 --2- 172.21.15.16:0/1873248497 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7faea0066d90 0x7faea0069250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:04.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.440+0000 7faeb7400640 1 -- 172.21.15.16:0/1873248497 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 5795+0+0 (secure 0 0 0) 0x7faeb8120f10 con 0x558e26970040 2024-03-31T15:10:04.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.440+0000 7faec2a48080 1 --2- 172.21.15.16:0/1873248497 >> [v2:172.21.15.16:6800/932712244,v1:172.21.15.16:6801/932712244] conn(0x558e269eebf0 0x558e269ef000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:04.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.440+0000 7faebf000640 1 --2- 172.21.15.16:0/1873248497 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7faea0066d90 0x7faea0069250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:04.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.440+0000 7faec2a48080 1 -- 172.21.15.16:0/1873248497 --> [v2:172.21.15.16:6800/932712244,v1:172.21.15.16:6801/932712244] -- osd_op(unknown.0.0:1 12.10 12:0cf6ad0f:::10000000000.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e82) -- 0x558e269ef540 con 0x558e269eebf0 2024-03-31T15:10:04.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.440+0000 7faebe600640 1 --2- 172.21.15.16:0/1873248497 >> [v2:172.21.15.16:6800/932712244,v1:172.21.15.16:6801/932712244] conn(0x558e269eebf0 0x558e269ef000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:04.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.440+0000 7faebf000640 1 --2- 172.21.15.16:0/1873248497 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7faea0066d90 0x7faea0069250 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7faeb0005fd0 tx=0x7faeb000c040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:10:04.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.440+0000 7faebe600640 1 --2- 172.21.15.16:0/1873248497 >> [v2:172.21.15.16:6800/932712244,v1:172.21.15.16:6801/932712244] conn(0x558e269eebf0 0x558e269ef000 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:10:04.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.444+0000 7faebe600640 1 -- 172.21.15.16:0/1873248497 <== osd.0 v2:172.21.15.16:6800/932712244 1 ==== osd_op_reply(1 10000000000.00000000 [delete] v82'11 uv11 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7faeac00acb0 con 0x558e269eebf0 2024-03-31T15:10:04.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.444+0000 7faec2a48080 1 -- 172.21.15.16:0/1873248497 >> [v2:172.21.15.16:6800/932712244,v1:172.21.15.16:6801/932712244] conn(0x558e269eebf0 msgr2=0x558e269ef000 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:10:04.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.444+0000 7faec2a48080 1 --2- 172.21.15.16:0/1873248497 >> [v2:172.21.15.16:6800/932712244,v1:172.21.15.16:6801/932712244] conn(0x558e269eebf0 0x558e269ef000 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:04.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.444+0000 7faec2a48080 1 -- 172.21.15.16:0/1873248497 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7faea0066d90 msgr2=0x7faea0069250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:10:04.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.444+0000 7faec2a48080 1 --2- 172.21.15.16:0/1873248497 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7faea0066d90 0x7faea0069250 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7faeb0005fd0 tx=0x7faeb000c040 comp rx=0 tx=0).stop 2024-03-31T15:10:04.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.444+0000 7faec2a48080 1 -- 172.21.15.16:0/1873248497 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558e26970040 msgr2=0x558e269d8940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:10:04.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.444+0000 7faec2a48080 1 --2- 172.21.15.16:0/1873248497 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558e26970040 0x558e269d8940 secure :-1 s=READY pgs=1142 cs=0 l=1 rev1=1 crypto rx=0x7faeb801b000 tx=0x7faeb8017d60 comp rx=0 tx=0).stop 2024-03-31T15:10:04.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.444+0000 7faebfa00640 1 -- 172.21.15.16:0/1873248497 reap_dead start 2024-03-31T15:10:04.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.444+0000 7faec2a48080 1 -- 172.21.15.16:0/1873248497 shutdown_connections 2024-03-31T15:10:04.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.444+0000 7faec2a48080 1 -- 172.21.15.16:0/1873248497 >> 172.21.15.16:0/1873248497 conn(0x558e26950420 msgr2=0x558e269f5570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:10:04.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.444+0000 7faec2a48080 1 -- 172.21.15.16:0/1873248497 shutdown_connections 2024-03-31T15:10:04.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.444+0000 7faec2a48080 1 -- 172.21.15.16:0/1873248497 wait complete. 2024-03-31T15:10:04.442 DEBUG:teuthology.orchestra.run.smithi016:> cephfs-journal-tool --debug-mds=20 --debug-ms=1 --debug-objecter=1 --rank cephfs:0 journal reset 2024-03-31T15:10:04.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.536+0000 7f429e418400 1 -- start start 2024-03-31T15:10:04.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.536+0000 7f429e418400 1 Processor -- start 2024-03-31T15:10:04.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.540+0000 7f429e418400 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5595223f1800 0x5595231b2000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:04.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.540+0000 7f429e418400 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5595223f1400 0x5595231b2580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:04.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.540+0000 7f429e418400 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5595223f1000 0x5595231b2b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:04.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.540+0000 7f429e418400 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5595231c6180 con 0x5595223f1400 2024-03-31T15:10:04.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.540+0000 7f429e418400 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x5595231c6300 con 0x5595223f1000 2024-03-31T15:10:04.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.540+0000 7f429e418400 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x5595231c6480 con 0x5595223f1800 2024-03-31T15:10:04.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.540+0000 7f429ba00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5595223f1400 0x5595231b2580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:04.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.540+0000 7f429ba00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5595223f1400 0x5595231b2580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39510/0 (socket says 172.21.15.16:39510) 2024-03-31T15:10:04.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.540+0000 7f429ba00640 1 -- 172.21.15.16:0/1297353038 learned_addr learned my addr 172.21.15.16:0/1297353038 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:10:04.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.540+0000 7f429c400640 1 --2- 172.21.15.16:0/1297353038 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5595223f1800 0x5595231b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:04.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.540+0000 7f429ce00640 1 --2- 172.21.15.16:0/1297353038 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5595223f1000 0x5595231b2b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:04.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.540+0000 7f429ba00640 1 -- 172.21.15.16:0/1297353038 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5595223f1800 msgr2=0x5595231b2000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:10:04.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.540+0000 7f429ba00640 1 --2- 172.21.15.16:0/1297353038 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5595223f1800 0x5595231b2000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:04.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.540+0000 7f429ba00640 1 -- 172.21.15.16:0/1297353038 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5595223f1000 msgr2=0x5595231b2b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:10:04.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.540+0000 7f429ba00640 1 --2- 172.21.15.16:0/1297353038 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5595223f1000 0x5595231b2b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:04.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.540+0000 7f429ba00640 1 -- 172.21.15.16:0/1297353038 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5595231acb40 con 0x5595223f1400 2024-03-31T15:10:04.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.540+0000 7f429ba00640 1 --2- 172.21.15.16:0/1297353038 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5595223f1400 0x5595231b2580 secure :-1 s=READY pgs=1144 cs=0 l=1 rev1=1 crypto rx=0x55952325c960 tx=0x5595231c2000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d900192dc6ed4060 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:10:04.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.540+0000 7f4299200640 1 -- 172.21.15.16:0/1297353038 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x5595224269c0 con 0x5595223f1400 2024-03-31T15:10:04.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.540+0000 7f4299200640 1 -- 172.21.15.16:0/1297353038 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x55952243c1c0 con 0x5595223f1400 2024-03-31T15:10:04.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.540+0000 7f4299200640 1 -- 172.21.15.16:0/1297353038 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x5595224264e0 con 0x5595223f1400 2024-03-31T15:10:04.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.540+0000 7f429e418400 1 -- 172.21.15.16:0/1297353038 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5595231acd20 con 0x5595223f1400 2024-03-31T15:10:04.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.540+0000 7f429e418400 1 -- 172.21.15.16:0/1297353038 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({fsmap=0}) -- 0x5595231acf00 con 0x5595223f1400 2024-03-31T15:10:04.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.540+0000 7f429e418400 4 waiting for MDS map... 2024-03-31T15:10:04.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.540+0000 7f4299200640 1 -- 172.21.15.16:0/1297353038 <== mon.0 v2:172.21.15.16:3300/0 4 ==== osd_map(82..82 src has 1..82) ==== 5795+0+0 (secure 0 0 0) 0x559523308200 con 0x5595223f1400 2024-03-31T15:10:04.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.540+0000 7f4299200640 1 -- 172.21.15.16:0/1297353038 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 62) ==== 2885+0+0 (secure 0 0 0) 0x559522447180 con 0x5595223f1400 2024-03-31T15:10:04.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.540+0000 7f429e418400 4 Got MDS map 62 2024-03-31T15:10:04.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.540+0000 7f429e418400 10 main: JournalTool::main 2024-03-31T15:10:04.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.540+0000 7f429e418400 4 main: JournalTool: connecting to RADOS... 2024-03-31T15:10:04.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.540+0000 7f429e418400 1 Processor -- start 2024-03-31T15:10:04.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.540+0000 7f429e418400 1 -- start start 2024-03-31T15:10:04.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.540+0000 7f429e418400 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5595231cfc00 0x5595231b3080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:04.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.540+0000 7f429e418400 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55952335a000 0x5595231b3600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:04.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.540+0000 7f429e418400 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55952335a400 0x5595231b3b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:04.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.540+0000 7f429e418400 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5595231c6600 con 0x55952335a000 2024-03-31T15:10:04.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.540+0000 7f429e418400 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x5595231c6780 con 0x55952335a400 2024-03-31T15:10:04.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.540+0000 7f429e418400 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x5595231c6900 con 0x5595231cfc00 2024-03-31T15:10:04.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.544+0000 7f429c400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55952335a000 0x5595231b3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:04.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.544+0000 7f429c400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55952335a000 0x5595231b3600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39516/0 (socket says 172.21.15.16:39516) 2024-03-31T15:10:04.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.544+0000 7f429c400640 1 -- 172.21.15.16:0/3699345459 learned_addr learned my addr 172.21.15.16:0/3699345459 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:10:04.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.544+0000 7f429ce00640 1 --2- 172.21.15.16:0/3699345459 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5595231cfc00 0x5595231b3080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:04.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.544+0000 7f429ba00640 1 --2- 172.21.15.16:0/3699345459 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55952335a400 0x5595231b3b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:04.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.544+0000 7f429c400640 1 -- 172.21.15.16:0/3699345459 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5595231cfc00 msgr2=0x5595231b3080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:10:04.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.544+0000 7f429c400640 1 --2- 172.21.15.16:0/3699345459 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5595231cfc00 0x5595231b3080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:04.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.544+0000 7f429c400640 1 -- 172.21.15.16:0/3699345459 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55952335a400 msgr2=0x5595231b3b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:10:04.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.544+0000 7f429c400640 1 --2- 172.21.15.16:0/3699345459 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55952335a400 0x5595231b3b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:04.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.544+0000 7f429c400640 1 -- 172.21.15.16:0/3699345459 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5595231ac1e0 con 0x55952335a000 2024-03-31T15:10:04.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.544+0000 7f429c400640 1 --2- 172.21.15.16:0/3699345459 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55952335a000 0x5595231b3600 secure :-1 s=READY pgs=1145 cs=0 l=1 rev1=1 crypto rx=0x5595233664e0 tx=0x5595231c3ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=49677c282c73ad74 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:10:04.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.544+0000 7f4296a00640 1 -- 172.21.15.16:0/3699345459 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x5595224269c0 con 0x55952335a000 2024-03-31T15:10:04.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.544+0000 7f4296a00640 1 -- 172.21.15.16:0/3699345459 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x55952243ce00 con 0x55952335a000 2024-03-31T15:10:04.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.544+0000 7f4296a00640 1 -- 172.21.15.16:0/3699345459 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x5595224264e0 con 0x55952335a000 2024-03-31T15:10:04.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.544+0000 7f429e418400 1 -- 172.21.15.16:0/3699345459 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55952335a000 msgr2=0x5595231b3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:10:04.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.544+0000 7f429e418400 1 --2- 172.21.15.16:0/3699345459 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55952335a000 0x5595231b3600 secure :-1 s=READY pgs=1145 cs=0 l=1 rev1=1 crypto rx=0x5595233664e0 tx=0x5595231c3ec0 comp rx=0 tx=0).stop 2024-03-31T15:10:04.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.544+0000 7f429e418400 1 -- 172.21.15.16:0/3699345459 shutdown_connections 2024-03-31T15:10:04.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.544+0000 7f429e418400 1 --2- 172.21.15.16:0/3699345459 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55952335a400 0x5595231b3b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:04.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.544+0000 7f429e418400 1 --2- 172.21.15.16:0/3699345459 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55952335a000 0x5595231b3600 unknown :-1 s=CLOSED pgs=1145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:04.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.544+0000 7f429e418400 1 --2- 172.21.15.16:0/3699345459 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5595231cfc00 0x5595231b3080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:04.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.544+0000 7f429e418400 1 -- 172.21.15.16:0/3699345459 >> 172.21.15.16:0/3699345459 conn(0x5595223f0c00 msgr2=0x55952241f800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:10:04.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.544+0000 7f429e418400 1 -- 172.21.15.16:0/3699345459 shutdown_connections 2024-03-31T15:10:04.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.544+0000 7f429e418400 1 -- 172.21.15.16:0/3699345459 wait complete. 2024-03-31T15:10:04.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.544+0000 7f429e418400 1 Processor -- start 2024-03-31T15:10:04.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.544+0000 7f429e418400 1 -- start start 2024-03-31T15:10:04.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.544+0000 7f429e418400 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5595231cfc00 0x5595231b3080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:04.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.544+0000 7f429e418400 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55952335b800 0x5595231b3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:04.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.544+0000 7f429e418400 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55952335b400 0x5595231b3b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:04.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.544+0000 7f429e418400 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5595231c7c80 con 0x55952335b800 2024-03-31T15:10:04.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.544+0000 7f429e418400 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x5595231c7b00 con 0x5595231cfc00 2024-03-31T15:10:04.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.544+0000 7f429e418400 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x5595231c7980 con 0x55952335b400 2024-03-31T15:10:04.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.544+0000 7f429c400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55952335b800 0x5595231b3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:04.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.544+0000 7f429c400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55952335b800 0x5595231b3600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39522/0 (socket says 172.21.15.16:39522) 2024-03-31T15:10:04.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.544+0000 7f429c400640 1 -- 172.21.15.16:0/840018770 learned_addr learned my addr 172.21.15.16:0/840018770 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:10:04.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.544+0000 7f429ba00640 1 --2- 172.21.15.16:0/840018770 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55952335b400 0x5595231b3b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:04.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.544+0000 7f429ce00640 1 --2- 172.21.15.16:0/840018770 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5595231cfc00 0x5595231b3080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:04.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.544+0000 7f429c400640 1 -- 172.21.15.16:0/840018770 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55952335b400 msgr2=0x5595231b3b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:10:04.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.544+0000 7f429c400640 1 --2- 172.21.15.16:0/840018770 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55952335b400 0x5595231b3b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:04.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.544+0000 7f429c400640 1 -- 172.21.15.16:0/840018770 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5595231cfc00 msgr2=0x5595231b3080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:10:04.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.544+0000 7f429c400640 1 --2- 172.21.15.16:0/840018770 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5595231cfc00 0x5595231b3080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:04.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.544+0000 7f429c400640 1 -- 172.21.15.16:0/840018770 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5595231ada40 con 0x55952335b800 2024-03-31T15:10:04.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.548+0000 7f429c400640 1 --2- 172.21.15.16:0/840018770 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55952335b800 0x5595231b3600 secure :-1 s=READY pgs=1146 cs=0 l=1 rev1=1 crypto rx=0x5595233677a0 tx=0x5595224a8cc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:10:04.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.548+0000 7f4294c00640 1 -- 172.21.15.16:0/840018770 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x559522426000 con 0x55952335b800 2024-03-31T15:10:04.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.548+0000 7f4294c00640 1 -- 172.21.15.16:0/840018770 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x55952243cc40 con 0x55952335b800 2024-03-31T15:10:04.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.548+0000 7f4294c00640 1 -- 172.21.15.16:0/840018770 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x5595224269c0 con 0x55952335b800 2024-03-31T15:10:04.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.548+0000 7f429e418400 1 -- 172.21.15.16:0/840018770 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x5595231adc20 con 0x55952335b800 2024-03-31T15:10:04.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.548+0000 7f429e418400 1 -- 172.21.15.16:0/840018770 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5595231ada40 con 0x55952335b800 2024-03-31T15:10:04.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.548+0000 7f429e418400 4 main: JournalTool: resolving pool 12 2024-03-31T15:10:04.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.548+0000 7f4294c00640 1 -- 172.21.15.16:0/840018770 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x559523358c00 con 0x55952335b800 2024-03-31T15:10:04.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.548+0000 7f4294c00640 1 --2- 172.21.15.16:0/840018770 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x559523374800 0x5595231b4100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:04.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.548+0000 7f4294c00640 1 -- 172.21.15.16:0/840018770 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 5795+0+0 (secure 0 0 0) 0x5595226dc400 con 0x55952335b800 2024-03-31T15:10:04.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.548+0000 7f429e418400 4 main: JournalTool: creating IoCtx.. 2024-03-31T15:10:04.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.548+0000 7f429e418400 4 main: Executing for rank 0 2024-03-31T15:10:04.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.548+0000 7f429ce00640 1 --2- 172.21.15.16:0/840018770 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x559523374800 0x5595231b4100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:04.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.552+0000 7f429e418400 1 -- start start 2024-03-31T15:10:04.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.552+0000 7f429e418400 1 Processor -- start 2024-03-31T15:10:04.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.552+0000 7f429ce00640 1 --2- 172.21.15.16:0/840018770 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x559523374800 0x5595231b4100 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x5595224de6f0 tx=0x5595227d6c00 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:10:04.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.552+0000 7f429e418400 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55952335ac00 0x5595231b4680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:04.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.552+0000 7f429e418400 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55952335a800 0x5595231b4c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:04.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.552+0000 7f429e418400 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55952335a400 0x5595231b5180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:04.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.552+0000 7f429e418400 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5595231c7680 con 0x55952335a800 2024-03-31T15:10:04.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.552+0000 7f429e418400 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x5595231c6600 con 0x55952335ac00 2024-03-31T15:10:04.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.552+0000 7f429e418400 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x5595231c6780 con 0x55952335a400 2024-03-31T15:10:04.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.552+0000 7f429c400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55952335a800 0x5595231b4c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:04.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.552+0000 7f429ce00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55952335ac00 0x5595231b4680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:04.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.552+0000 7f429ce00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55952335ac00 0x5595231b4680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.16:44408/0 (socket says 172.21.15.16:44408) 2024-03-31T15:10:04.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.552+0000 7f429ce00640 1 -- 172.21.15.16:0/46483750 learned_addr learned my addr 172.21.15.16:0/46483750 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:10:04.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.552+0000 7f429ba00640 1 --2- 172.21.15.16:0/46483750 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55952335a400 0x5595231b5180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:04.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.552+0000 7f429ce00640 1 -- 172.21.15.16:0/46483750 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55952335a400 msgr2=0x5595231b5180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:10:04.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.552+0000 7f429ce00640 1 --2- 172.21.15.16:0/46483750 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55952335a400 0x5595231b5180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:04.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.552+0000 7f429ce00640 1 -- 172.21.15.16:0/46483750 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55952335a800 msgr2=0x5595231b4c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:10:04.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.552+0000 7f429ce00640 1 --2- 172.21.15.16:0/46483750 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55952335a800 0x5595231b4c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:04.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.552+0000 7f429ce00640 1 -- 172.21.15.16:0/46483750 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559522826780 con 0x55952335ac00 2024-03-31T15:10:04.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.552+0000 7f429ce00640 1 --2- 172.21.15.16:0/46483750 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55952335ac00 0x5595231b4680 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x559522775560 tx=0x559523173020 comp rx=0 tx=0).ready entity=mon.1 client_cookie=57d38772a177eae server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:10:04.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.552+0000 7f429c400640 1 --2- 172.21.15.16:0/46483750 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55952335a800 0x5595231b4c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:10:04.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.552+0000 7f4291000640 1 -- 172.21.15.16:0/46483750 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x559522716d00 con 0x55952335ac00 2024-03-31T15:10:04.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.552+0000 7f4291000640 1 -- 172.21.15.16:0/46483750 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x55952243d340 con 0x55952335ac00 2024-03-31T15:10:04.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.552+0000 7f4291000640 1 -- 172.21.15.16:0/46483750 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x559522718820 con 0x55952335ac00 2024-03-31T15:10:04.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.552+0000 7f429e418400 1 -- 172.21.15.16:0/46483750 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559522826960 con 0x55952335ac00 2024-03-31T15:10:04.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.552+0000 7f429e418400 1 -- 172.21.15.16:0/46483750 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({fsmap=0}) -- 0x559522826b40 con 0x55952335ac00 2024-03-31T15:10:04.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.552+0000 7f429e418400 4 waiting for MDS map... 2024-03-31T15:10:04.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.556+0000 7f4291000640 1 -- 172.21.15.16:0/46483750 <== mon.1 v2:172.21.15.89:3300/0 4 ==== osd_map(82..82 src has 1..82) ==== 5795+0+0 (secure 0 0 0) 0x55952283e600 con 0x55952335ac00 2024-03-31T15:10:04.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.556+0000 7f4291000640 1 -- 172.21.15.16:0/46483750 <== mon.1 v2:172.21.15.89:3300/0 5 ==== fsmap(e 62) ==== 2885+0+0 (secure 0 0 0) 0x559522447500 con 0x55952335ac00 2024-03-31T15:10:04.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.556+0000 7f429e418400 4 Got MDS map 62 2024-03-31T15:10:04.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.556+0000 7f429e418400 1 --2- 172.21.15.16:0/46483750 >> [v2:172.21.15.89:6809/3470471752,v1:172.21.15.89:6812/3470471752] conn(0x55952335a000 0x5595231b5700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:04.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.556+0000 7f429e418400 1 -- 172.21.15.16:0/46483750 --> [v2:172.21.15.89:6809/3470471752,v1:172.21.15.89:6812/3470471752] -- osd_op(unknown.0.0:1 12.f 12:f1f69726:::400.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e82) -- 0x559522841400 con 0x55952335a000 2024-03-31T15:10:04.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.556+0000 7f429c400640 1 --2- 172.21.15.16:0/46483750 >> [v2:172.21.15.89:6809/3470471752,v1:172.21.15.89:6812/3470471752] conn(0x55952335a000 0x5595231b5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:04.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.556+0000 7f429c400640 1 --2- 172.21.15.16:0/46483750 >> [v2:172.21.15.89:6809/3470471752,v1:172.21.15.89:6812/3470471752] conn(0x55952335a000 0x5595231b5700 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=5f9c31f790b9c25c server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:10:04.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.556+0000 7f429c400640 1 -- 172.21.15.16:0/46483750 <== osd.6 v2:172.21.15.89:6809/3470471752 1 ==== osd_op_reply(1 400.00000000 [read 0~22 out=22b] v0'0 uv1 ondisk = 0) ==== 156+0+22 (crc 0 0 0) 0x55952316c000 con 0x55952335a000 2024-03-31T15:10:04.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.556+0000 7f429e418400 1 --2- 172.21.15.16:0/46483750 >> [v2:172.21.15.89:6810/50395527,v1:172.21.15.89:6814/50395527] conn(0x559522841800 0x55952285e000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:04.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.556+0000 7f429e418400 1 -- 172.21.15.16:0/46483750 --> [v2:172.21.15.89:6810/50395527,v1:172.21.15.89:6814/50395527] -- osd_op(unknown.0.0:2 12.14 12:292cf221:::200.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e82) -- 0x559522841c00 con 0x559522841800 2024-03-31T15:10:04.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.556+0000 7f429ba00640 1 --2- 172.21.15.16:0/46483750 >> [v2:172.21.15.89:6810/50395527,v1:172.21.15.89:6814/50395527] conn(0x559522841800 0x55952285e000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:04.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.556+0000 7f429ba00640 1 --2- 172.21.15.16:0/46483750 >> [v2:172.21.15.89:6810/50395527,v1:172.21.15.89:6814/50395527] conn(0x559522841800 0x55952285e000 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=70784c9491f7dc09 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:10:04.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.556+0000 7f429ba00640 1 -- 172.21.15.16:0/46483750 <== osd.5 v2:172.21.15.89:6810/50395527 1 ==== osd_op_reply(2 200.00000000 [read 0~90 out=90b] v0'0 uv19 ondisk = 0) ==== 156+0+90 (crc 0 0 0) 0x55952316c500 con 0x559522841800 2024-03-31T15:10:04.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.556+0000 7f428f200640 1 -- 172.21.15.16:0/46483750 --> [v2:172.21.15.89:6810/50395527,v1:172.21.15.89:6814/50395527] -- osd_op(unknown.0.0:3 12.14 12:2e2fa760:::200.00000001:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e82) -- 0x559522862000 con 0x559522841800 2024-03-31T15:10:04.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.556+0000 7f428f200640 1 --2- 172.21.15.16:0/46483750 >> [v2:172.21.15.89:6800/1856844856,v1:172.21.15.89:6801/1856844856] conn(0x559522862400 0x55952285fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:04.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.556+0000 7f428f200640 1 -- 172.21.15.16:0/46483750 --> [v2:172.21.15.89:6800/1856844856,v1:172.21.15.89:6801/1856844856] -- osd_op(unknown.0.0:4 12.1b 12:ddb4e4d7:::200.00000002:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e82) -- 0x559522862800 con 0x559522862400 2024-03-31T15:10:04.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.556+0000 7f429ba00640 1 -- 172.21.15.16:0/46483750 <== osd.5 v2:172.21.15.89:6810/50395527 2 ==== osd_op_reply(3 200.00000001 [stat out=16b] v0'0 uv16 ondisk = 0) ==== 156+0+16 (crc 0 0 0) 0x55952316c500 con 0x559522841800 2024-03-31T15:10:04.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.556+0000 7f429ce00640 1 --2- 172.21.15.16:0/46483750 >> [v2:172.21.15.89:6800/1856844856,v1:172.21.15.89:6801/1856844856] conn(0x559522862400 0x55952285fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:04.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.560+0000 7f429ce00640 1 --2- 172.21.15.16:0/46483750 >> [v2:172.21.15.89:6800/1856844856,v1:172.21.15.89:6801/1856844856] conn(0x559522862400 0x55952285fb80 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=1cc040fbb0d41bfd server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:10:04.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.560+0000 7f429ce00640 1 -- 172.21.15.16:0/46483750 <== osd.4 v2:172.21.15.89:6800/1856844856 1 ==== osd_op_reply(4 200.00000002 [stat] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 156+0+0 (crc 0 0 0) 0x55952316c780 con 0x559522862400 2024-03-31T15:10:04.552 INFO:teuthology.orchestra.run.smithi016.stdout:old journal was 4208696~2030 2024-03-31T15:10:04.552 INFO:teuthology.orchestra.run.smithi016.stdout:new journal start will be 8388608 (4177882 bytes past old end) 2024-03-31T15:10:04.552 INFO:teuthology.orchestra.run.smithi016.stdout:writing journal head 2024-03-31T15:10:04.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.560+0000 7f429e418400 1 -- 172.21.15.16:0/46483750 --> [v2:172.21.15.89:6810/50395527,v1:172.21.15.89:6814/50395527] -- osd_op(unknown.0.0:5 12.14 12:292cf221:::200.00000000:head [writefull 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e82) -- 0x559522862c00 con 0x559522841800 2024-03-31T15:10:04.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.560+0000 7f429ba00640 1 -- 172.21.15.16:0/46483750 <== osd.5 v2:172.21.15.89:6810/50395527 3 ==== osd_op_reply(5 200.00000000 [writefull 0~90] v82'20 uv20 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x55952316c500 con 0x559522841800 2024-03-31T15:10:04.553 INFO:teuthology.orchestra.run.smithi016.stdout:writing EResetJournal entry 2024-03-31T15:10:04.554 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.560+0000 7f429e418400 1 -- 172.21.15.16:0/46483750 --> [v2:172.21.15.89:6800/1856844856,v1:172.21.15.89:6801/1856844856] -- osd_op(unknown.0.0:6 12.1b 12:ddb4e4d7:::200.00000002:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e82) -- 0x559522863000 con 0x559522862400 2024-03-31T15:10:04.554 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.560+0000 7f429e418400 1 --2- 172.21.15.16:0/46483750 >> [v2:172.21.15.16:6809/3360352974,v1:172.21.15.16:6812/3360352974] conn(0x559522863400 0x55952285e580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:04.554 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.560+0000 7f429e418400 1 -- 172.21.15.16:0/46483750 --> [v2:172.21.15.16:6809/3360352974,v1:172.21.15.16:6812/3360352974] -- osd_op(unknown.0.0:7 12.b 12:d5c7a900:::200.00000003:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e82) -- 0x559522863800 con 0x559522863400 2024-03-31T15:10:04.554 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.560+0000 7f429e418400 1 -- 172.21.15.16:0/46483750 --> [v2:172.21.15.89:6809/3470471752,v1:172.21.15.89:6812/3470471752] -- osd_op(unknown.0.0:8 12.1f 12:f83d65d8:::200.00000004:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e82) -- 0x559522863c00 con 0x55952335a000 2024-03-31T15:10:04.554 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.560+0000 7f429e418400 1 -- 172.21.15.16:0/46483750 --> [v2:172.21.15.89:6800/1856844856,v1:172.21.15.89:6801/1856844856] -- osd_op(unknown.0.0:9 12.1a 12:5e41ab69:::200.00000005:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e82) -- 0x559522870000 con 0x559522862400 2024-03-31T15:10:04.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.560+0000 7f429e418400 1 -- 172.21.15.16:0/46483750 --> [v2:172.21.15.89:6800/1856844856,v1:172.21.15.89:6801/1856844856] -- osd_op(unknown.0.0:10 12.1e 12:7bf7e7b5:::200.00000006:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e82) -- 0x559522870400 con 0x559522862400 2024-03-31T15:10:04.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.560+0000 7f429e418400 1 --2- 172.21.15.16:0/46483750 >> [v2:172.21.15.16:6800/932712244,v1:172.21.15.16:6801/932712244] conn(0x559522870800 0x55952285eb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:04.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.560+0000 7f429e418400 1 -- 172.21.15.16:0/46483750 --> [v2:172.21.15.16:6800/932712244,v1:172.21.15.16:6801/932712244] -- osd_op(unknown.0.0:11 12.1d 12:bec1251c:::200.00000007:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e82) -- 0x559522870c00 con 0x559522870800 2024-03-31T15:10:04.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.560+0000 7f429c400640 1 --2- 172.21.15.16:0/46483750 >> [v2:172.21.15.16:6809/3360352974,v1:172.21.15.16:6812/3360352974] conn(0x559522863400 0x55952285e580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:04.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.560+0000 7f429ba00640 1 --2- 172.21.15.16:0/46483750 >> [v2:172.21.15.16:6800/932712244,v1:172.21.15.16:6801/932712244] conn(0x559522870800 0x55952285eb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:04.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.560+0000 7f429c400640 1 --2- 172.21.15.16:0/46483750 >> [v2:172.21.15.16:6809/3360352974,v1:172.21.15.16:6812/3360352974] conn(0x559522863400 0x55952285e580 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=7eeeaccdd36d27fe server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:10:04.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.560+0000 7f429ba00640 1 --2- 172.21.15.16:0/46483750 >> [v2:172.21.15.16:6800/932712244,v1:172.21.15.16:6801/932712244] conn(0x559522870800 0x55952285eb00 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=de5afd47d5ec8362 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:10:04.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.564+0000 7f429ce00640 1 -- 172.21.15.16:0/46483750 <== osd.4 v2:172.21.15.89:6800/1856844856 2 ==== osd_op_reply(6 200.00000002 [delete] v82'2 uv0 ondisk = -2 ((2) No such file or directory)) ==== 156+0+0 (crc 0 0 0) 0x55952316c780 con 0x559522862400 2024-03-31T15:10:04.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.564+0000 7f429ce00640 1 -- 172.21.15.16:0/46483750 <== osd.4 v2:172.21.15.89:6800/1856844856 3 ==== osd_op_reply(9 200.00000005 [delete] v82'2 uv0 ondisk = -2 ((2) No such file or directory)) ==== 156+0+0 (crc 0 0 0) 0x55952316c780 con 0x559522862400 2024-03-31T15:10:04.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.564+0000 7f429ce00640 1 -- 172.21.15.16:0/46483750 <== osd.4 v2:172.21.15.89:6800/1856844856 4 ==== osd_op_reply(10 200.00000006 [delete] v82'2 uv0 ondisk = -2 ((2) No such file or directory)) ==== 156+0+0 (crc 0 0 0) 0x55952316c780 con 0x559522862400 2024-03-31T15:10:04.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.564+0000 7f429c400640 1 -- 172.21.15.16:0/46483750 <== osd.6 v2:172.21.15.89:6809/3470471752 2 ==== osd_op_reply(8 200.00000004 [delete] v82'7 uv6 ondisk = -2 ((2) No such file or directory)) ==== 156+0+0 (crc 0 0 0) 0x55952316c000 con 0x55952335a000 2024-03-31T15:10:04.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.564+0000 7f429c400640 1 -- 172.21.15.16:0/46483750 <== osd.3 v2:172.21.15.16:6809/3360352974 1 ==== osd_op_reply(7 200.00000003 [delete] v82'4 uv3 ondisk = -2 ((2) No such file or directory)) ==== 156+0+0 (crc 0 0 0) 0x55952316c000 con 0x559522863400 2024-03-31T15:10:04.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.564+0000 7f428f200640 1 -- 172.21.15.16:0/46483750 --> [v2:172.21.15.89:6800/1856844856,v1:172.21.15.89:6801/1856844856] -- osd_op(unknown.0.0:12 12.1b 12:ddb4e4d7:::200.00000002:head [write 0~48 in=48b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e82) -- 0x559522872000 con 0x559522862400 2024-03-31T15:10:04.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.564+0000 7f429ba00640 1 -- 172.21.15.16:0/46483750 <== osd.0 v2:172.21.15.16:6800/932712244 1 ==== osd_op_reply(11 200.00000007 [delete] v82'3 uv2 ondisk = -2 ((2) No such file or directory)) ==== 156+0+0 (crc 0 0 0) 0x55952316c500 con 0x559522870800 2024-03-31T15:10:04.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.564+0000 7f429ce00640 1 -- 172.21.15.16:0/46483750 <== osd.4 v2:172.21.15.89:6800/1856844856 5 ==== osd_op_reply(12 200.00000002 [write 0~48] v82'3 uv3 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x55952316c780 con 0x559522862400 2024-03-31T15:10:04.557 INFO:teuthology.orchestra.run.smithi016.stdout:done 2024-03-31T15:10:04.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.564+0000 7f429e418400 1 -- 172.21.15.16:0/46483750 >> [v2:172.21.15.16:6800/932712244,v1:172.21.15.16:6801/932712244] conn(0x559522870800 msgr2=0x55952285eb00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:10:04.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.564+0000 7f429e418400 1 --2- 172.21.15.16:0/46483750 >> [v2:172.21.15.16:6800/932712244,v1:172.21.15.16:6801/932712244] conn(0x559522870800 0x55952285eb00 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:04.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.564+0000 7f429e418400 1 -- 172.21.15.16:0/46483750 >> [v2:172.21.15.16:6809/3360352974,v1:172.21.15.16:6812/3360352974] conn(0x559522863400 msgr2=0x55952285e580 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:10:04.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.564+0000 7f429e418400 1 --2- 172.21.15.16:0/46483750 >> [v2:172.21.15.16:6809/3360352974,v1:172.21.15.16:6812/3360352974] conn(0x559522863400 0x55952285e580 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:04.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.564+0000 7f429e418400 1 -- 172.21.15.16:0/46483750 >> [v2:172.21.15.89:6800/1856844856,v1:172.21.15.89:6801/1856844856] conn(0x559522862400 msgr2=0x55952285fb80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:10:04.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.564+0000 7f429e418400 1 --2- 172.21.15.16:0/46483750 >> [v2:172.21.15.89:6800/1856844856,v1:172.21.15.89:6801/1856844856] conn(0x559522862400 0x55952285fb80 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:04.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.564+0000 7f429e418400 1 -- 172.21.15.16:0/46483750 >> [v2:172.21.15.89:6810/50395527,v1:172.21.15.89:6814/50395527] conn(0x559522841800 msgr2=0x55952285e000 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:10:04.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.564+0000 7f429e418400 1 --2- 172.21.15.16:0/46483750 >> [v2:172.21.15.89:6810/50395527,v1:172.21.15.89:6814/50395527] conn(0x559522841800 0x55952285e000 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:04.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.564+0000 7f429e418400 1 -- 172.21.15.16:0/46483750 >> [v2:172.21.15.89:6809/3470471752,v1:172.21.15.89:6812/3470471752] conn(0x55952335a000 msgr2=0x5595231b5700 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:10:04.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.564+0000 7f429e418400 1 --2- 172.21.15.16:0/46483750 >> [v2:172.21.15.89:6809/3470471752,v1:172.21.15.89:6812/3470471752] conn(0x55952335a000 0x5595231b5700 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:04.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.564+0000 7f429ba00640 1 -- 172.21.15.16:0/46483750 reap_dead start 2024-03-31T15:10:04.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.564+0000 7f429ba00640 1 -- 172.21.15.16:0/46483750 reap_dead start 2024-03-31T15:10:04.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.564+0000 7f429e418400 1 -- 172.21.15.16:0/46483750 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55952335ac00 msgr2=0x5595231b4680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:10:04.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.564+0000 7f429e418400 1 --2- 172.21.15.16:0/46483750 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55952335ac00 0x5595231b4680 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x559522775560 tx=0x559523173020 comp rx=0 tx=0).stop 2024-03-31T15:10:04.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.564+0000 7f429ba00640 1 -- 172.21.15.16:0/46483750 reap_dead start 2024-03-31T15:10:04.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.564+0000 7f429e418400 1 -- 172.21.15.16:0/46483750 shutdown_connections 2024-03-31T15:10:04.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.564+0000 7f429e418400 1 -- 172.21.15.16:0/46483750 >> 172.21.15.16:0/46483750 conn(0x55952335b000 msgr2=0x559522820000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:10:04.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.564+0000 7f429e418400 1 -- 172.21.15.16:0/46483750 shutdown_connections 2024-03-31T15:10:04.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.564+0000 7f429e418400 1 -- 172.21.15.16:0/46483750 wait complete. 2024-03-31T15:10:04.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.564+0000 7f429e418400 1 -- 172.21.15.16:0/840018770 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x559523374800 msgr2=0x5595231b4100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:10:04.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.564+0000 7f429e418400 1 --2- 172.21.15.16:0/840018770 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x559523374800 0x5595231b4100 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x5595224de6f0 tx=0x5595227d6c00 comp rx=0 tx=0).stop 2024-03-31T15:10:04.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.564+0000 7f429e418400 1 -- 172.21.15.16:0/840018770 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55952335b800 msgr2=0x5595231b3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:10:04.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.564+0000 7f429e418400 1 --2- 172.21.15.16:0/840018770 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55952335b800 0x5595231b3600 secure :-1 s=READY pgs=1146 cs=0 l=1 rev1=1 crypto rx=0x5595233677a0 tx=0x5595224a8cc0 comp rx=0 tx=0).stop 2024-03-31T15:10:04.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.564+0000 7f429e418400 1 -- 172.21.15.16:0/840018770 shutdown_connections 2024-03-31T15:10:04.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.564+0000 7f429e418400 1 --2- 172.21.15.16:0/840018770 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x559523374800 0x5595231b4100 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:04.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.564+0000 7f429e418400 1 --2- 172.21.15.16:0/840018770 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55952335b400 0x5595231b3b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:04.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.564+0000 7f429e418400 1 --2- 172.21.15.16:0/840018770 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55952335b800 0x5595231b3600 unknown :-1 s=CLOSED pgs=1146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:04.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.564+0000 7f429e418400 1 --2- 172.21.15.16:0/840018770 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5595231cfc00 0x5595231b3080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:04.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.564+0000 7f429e418400 1 -- 172.21.15.16:0/840018770 >> 172.21.15.16:0/840018770 conn(0x5595223f0c00 msgr2=0x55952241f800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:10:04.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.564+0000 7f429e418400 1 -- 172.21.15.16:0/840018770 shutdown_connections 2024-03-31T15:10:04.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.564+0000 7f429e418400 1 -- 172.21.15.16:0/840018770 wait complete. 2024-03-31T15:10:04.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.568+0000 7f429e418400 1 -- 172.21.15.16:0/1297353038 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5595223f1400 msgr2=0x5595231b2580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:10:04.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.568+0000 7f429e418400 1 --2- 172.21.15.16:0/1297353038 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5595223f1400 0x5595231b2580 secure :-1 s=READY pgs=1144 cs=0 l=1 rev1=1 crypto rx=0x55952325c960 tx=0x5595231c2000 comp rx=0 tx=0).stop 2024-03-31T15:10:04.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.568+0000 7f429e418400 1 -- 172.21.15.16:0/1297353038 shutdown_connections 2024-03-31T15:10:04.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.568+0000 7f429e418400 1 --2- 172.21.15.16:0/1297353038 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5595223f1000 0x5595231b2b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:04.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.568+0000 7f429e418400 1 --2- 172.21.15.16:0/1297353038 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5595223f1400 0x5595231b2580 unknown :-1 s=CLOSED pgs=1144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:04.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.568+0000 7f429e418400 1 --2- 172.21.15.16:0/1297353038 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5595223f1800 0x5595231b2000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:04.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.568+0000 7f429e418400 1 -- 172.21.15.16:0/1297353038 >> 172.21.15.16:0/1297353038 conn(0x5595223f1c00 msgr2=0x55952241f000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:10:04.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.568+0000 7f429e418400 1 -- 172.21.15.16:0/1297353038 shutdown_connections 2024-03-31T15:10:04.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.568+0000 7f429e418400 1 -- 172.21.15.16:0/1297353038 wait complete. 2024-03-31T15:10:04.563 DEBUG:tasks.cephfs.filesystem:Ran ['cephfs-journal-tool', '--debug-mds=20', '--debug-ms=1', '--debug-objecter=1', '--rank', 'cephfs:0', 'journal', 'reset'] in time 0:00:00.121018, result: old journal was 4208696~2030 new journal start will be 8388608 (4177882 bytes past old end) writing journal head writing EResetJournal entry done 2024-03-31T15:10:04.564 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs joinable true 2024-03-31T15:10:04.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.756+0000 7f4999a00640 1 Processor -- start 2024-03-31T15:10:04.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.756+0000 7f4999a00640 1 -- start start 2024-03-31T15:10:04.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.756+0000 7f4999a00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f49941411b0 0x7f499406f920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:04.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.756+0000 7f4999a00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f4994144340 0x7f499406fe60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:04.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.756+0000 7f4999a00640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4994144980 con 0x7f4994141590 2024-03-31T15:10:04.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.756+0000 7f4999a00640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f499415f910 con 0x7f4994144340 2024-03-31T15:10:04.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.756+0000 7f4999a00640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f4994162910 con 0x7f49941411b0 2024-03-31T15:10:04.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.756+0000 7f4993400640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f4994144340 0x7f499406fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:04.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.756+0000 7f4993400640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f4994144340 0x7f499406fe60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39548/0 (socket says 172.21.15.16:39548) 2024-03-31T15:10:04.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.756+0000 7f4993400640 1 -- 172.21.15.16:0/3784260793 learned_addr learned my addr 172.21.15.16:0/3784260793 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:10:04.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.756+0000 7f498a000640 1 --1- 172.21.15.16:0/3784260793 >> v1:172.21.15.16:6789/0 conn(0x7f4994141590 0x7f4994143c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:57784/0 (socket says 172.21.15.16:57784) 2024-03-31T15:10:04.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.756+0000 7f4992a00640 1 --2- 172.21.15.16:0/3784260793 >> v2:172.21.15.89:3300/0 conn(0x7f49941411b0 0x7f499406f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:04.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.756+0000 7f4993400640 1 -- 172.21.15.16:0/3784260793 >> v2:172.21.15.89:3300/0 conn(0x7f49941411b0 msgr2=0x7f499406f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:10:04.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.756+0000 7f4993400640 1 --2- 172.21.15.16:0/3784260793 >> v2:172.21.15.89:3300/0 conn(0x7f49941411b0 0x7f499406f920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:04.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.756+0000 7f4993400640 1 -- 172.21.15.16:0/3784260793 >> v1:172.21.15.16:6789/0 conn(0x7f4994141590 legacy=0x7f4994143c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:10:04.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.756+0000 7f4993400640 1 -- 172.21.15.16:0/3784260793 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4994161fe0 con 0x7f4994144340 2024-03-31T15:10:04.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.760+0000 7f4993400640 1 --2- 172.21.15.16:0/3784260793 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4994144340 0x7f499406fe60 secure :-1 s=READY pgs=1148 cs=0 l=1 rev1=1 crypto rx=0x7f49940d3120 tx=0x7f498c01c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3455c17582c9c572 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:10:04.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.760+0000 7f4992000640 1 -- 172.21.15.16:0/3784260793 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f498c0ab610 con 0x7f4994144340 2024-03-31T15:10:04.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.760+0000 7f4992000640 1 -- 172.21.15.16:0/3784260793 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f498c0b0070 con 0x7f4994144340 2024-03-31T15:10:04.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.760+0000 7f4992000640 1 -- 172.21.15.16:0/3784260793 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f498c0ab9d0 con 0x7f4994144340 2024-03-31T15:10:04.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.760+0000 7f4999a00640 1 -- 172.21.15.16:0/3784260793 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4994144340 msgr2=0x7f499406fe60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:10:04.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.760+0000 7f4999a00640 1 --2- 172.21.15.16:0/3784260793 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4994144340 0x7f499406fe60 secure :-1 s=READY pgs=1148 cs=0 l=1 rev1=1 crypto rx=0x7f49940d3120 tx=0x7f498c01c520 comp rx=0 tx=0).stop 2024-03-31T15:10:04.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.760+0000 7f4999a00640 1 -- 172.21.15.16:0/3784260793 shutdown_connections 2024-03-31T15:10:04.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.760+0000 7f4999a00640 1 --2- 172.21.15.16:0/3784260793 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4994144340 0x7f499406fe60 unknown :-1 s=CLOSED pgs=1148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:04.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.760+0000 7f4999a00640 1 --2- 172.21.15.16:0/3784260793 >> v2:172.21.15.89:3300/0 conn(0x7f49941411b0 0x7f499406f920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:04.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.760+0000 7f4999a00640 1 -- 172.21.15.16:0/3784260793 >> 172.21.15.16:0/3784260793 conn(0x7f4994067930 msgr2=0x7f49940b3ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:10:04.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.760+0000 7f4999a00640 1 -- 172.21.15.16:0/3784260793 shutdown_connections 2024-03-31T15:10:04.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.760+0000 7f4999a00640 1 -- 172.21.15.16:0/3784260793 wait complete. 2024-03-31T15:10:04.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.764+0000 7f4999a00640 1 Processor -- start 2024-03-31T15:10:04.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.764+0000 7f4999a00640 1 -- start start 2024-03-31T15:10:04.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.764+0000 7f4999a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f49941411b0 0x7f49941509d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:04.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.764+0000 7f4999a00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f4994141590 0x7f4994153d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:04.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.764+0000 7f4999a00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4994144340 0x7f49940d2600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:04.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.764+0000 7f4999a00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f499415f290 con 0x7f49941411b0 2024-03-31T15:10:04.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.764+0000 7f4999a00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f499415f110 con 0x7f4994144340 2024-03-31T15:10:04.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.764+0000 7f4999a00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f499415f410 con 0x7f4994141590 2024-03-31T15:10:04.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.764+0000 7f4992a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f49941411b0 0x7f49941509d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:04.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.764+0000 7f4992a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f49941411b0 0x7f49941509d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39556/0 (socket says 172.21.15.16:39556) 2024-03-31T15:10:04.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.764+0000 7f4992a00640 1 -- 172.21.15.16:0/3216889661 learned_addr learned my addr 172.21.15.16:0/3216889661 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:10:04.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.764+0000 7f498a000640 1 --2- 172.21.15.16:0/3216889661 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f4994141590 0x7f4994153d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:04.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.764+0000 7f4993400640 1 --2- 172.21.15.16:0/3216889661 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4994144340 0x7f49940d2600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:04.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.764+0000 7f4992a00640 1 -- 172.21.15.16:0/3216889661 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f4994141590 msgr2=0x7f4994153d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:10:04.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.764+0000 7f4992a00640 1 --2- 172.21.15.16:0/3216889661 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f4994141590 0x7f4994153d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:04.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.764+0000 7f4992a00640 1 -- 172.21.15.16:0/3216889661 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4994144340 msgr2=0x7f49940d2600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:10:04.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.764+0000 7f4992a00640 1 --2- 172.21.15.16:0/3216889661 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4994144340 0x7f49940d2600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:04.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.764+0000 7f4992a00640 1 -- 172.21.15.16:0/3216889661 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4994154830 con 0x7f49941411b0 2024-03-31T15:10:04.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.764+0000 7f4992a00640 1 --2- 172.21.15.16:0/3216889661 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f49941411b0 0x7f49941509d0 secure :-1 s=READY pgs=1149 cs=0 l=1 rev1=1 crypto rx=0x7f4984007f20 tx=0x7f498400e570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:10:04.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.764+0000 7f498b400640 1 -- 172.21.15.16:0/3216889661 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4984014620 con 0x7f49941411b0 2024-03-31T15:10:04.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.764+0000 7f498b400640 1 -- 172.21.15.16:0/3216889661 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4984019070 con 0x7f49941411b0 2024-03-31T15:10:04.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.764+0000 7f498b400640 1 -- 172.21.15.16:0/3216889661 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4984014a00 con 0x7f49941411b0 2024-03-31T15:10:04.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.764+0000 7f4999a00640 1 -- 172.21.15.16:0/3216889661 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4994154b10 con 0x7f49941411b0 2024-03-31T15:10:04.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.764+0000 7f4999a00640 1 -- 172.21.15.16:0/3216889661 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4994156920 con 0x7f49941411b0 2024-03-31T15:10:04.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.768+0000 7f4999a00640 1 -- 172.21.15.16:0/3216889661 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4994069950 con 0x7f49941411b0 2024-03-31T15:10:04.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.768+0000 7f498b400640 1 -- 172.21.15.16:0/3216889661 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f4984014ba0 con 0x7f49941411b0 2024-03-31T15:10:04.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.768+0000 7f498b400640 1 --2- 172.21.15.16:0/3216889661 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f4970068380 0x7f497006a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:04.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.768+0000 7f498b400640 1 -- 172.21.15.16:0/3216889661 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 5795+0+0 (secure 0 0 0) 0x7f4984087230 con 0x7f49941411b0 2024-03-31T15:10:04.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.768+0000 7f498a000640 1 --2- 172.21.15.16:0/3216889661 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f4970068380 0x7f497006a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:04.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.768+0000 7f498a000640 1 --2- 172.21.15.16:0/3216889661 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f4970068380 0x7f497006a840 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f499414c270 tx=0x7f497c007e90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:10:04.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.772+0000 7f498b400640 1 -- 172.21.15.16:0/3216889661 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f4984057130 con 0x7f49941411b0 2024-03-31T15:10:04.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:04.972+0000 7f4999a00640 1 -- 172.21.15.16:0/3216889661 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"} v 0) -- 0x7f49940b4a20 con 0x7f49941411b0 2024-03-31T15:10:05.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:05.360+0000 7f498b400640 1 -- 172.21.15.16:0/3216889661 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]=0 cephfs marked joinable; MDS may join as newly active. v63) ==== 162+0+0 (secure 0 0 0) 0x7f498405b430 con 0x7f49941411b0 2024-03-31T15:10:05.354 INFO:teuthology.orchestra.run.smithi016.stderr:cephfs marked joinable; MDS may join as newly active. 2024-03-31T15:10:05.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:05.364+0000 7f4999a00640 1 -- 172.21.15.16:0/3216889661 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f4970068380 msgr2=0x7f497006a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:10:05.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:05.364+0000 7f4999a00640 1 --2- 172.21.15.16:0/3216889661 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f4970068380 0x7f497006a840 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f499414c270 tx=0x7f497c007e90 comp rx=0 tx=0).stop 2024-03-31T15:10:05.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:05.364+0000 7f4999a00640 1 -- 172.21.15.16:0/3216889661 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f49941411b0 msgr2=0x7f49941509d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:10:05.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:05.364+0000 7f4999a00640 1 --2- 172.21.15.16:0/3216889661 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f49941411b0 0x7f49941509d0 secure :-1 s=READY pgs=1149 cs=0 l=1 rev1=1 crypto rx=0x7f4984007f20 tx=0x7f498400e570 comp rx=0 tx=0).stop 2024-03-31T15:10:05.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:05.364+0000 7f4999a00640 1 -- 172.21.15.16:0/3216889661 shutdown_connections 2024-03-31T15:10:05.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:05.364+0000 7f4999a00640 1 --2- 172.21.15.16:0/3216889661 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f4970068380 0x7f497006a840 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:05.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:05.364+0000 7f4999a00640 1 --2- 172.21.15.16:0/3216889661 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4994144340 0x7f49940d2600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:05.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:05.364+0000 7f4999a00640 1 --2- 172.21.15.16:0/3216889661 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f4994141590 0x7f4994153d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:05.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:05.364+0000 7f4999a00640 1 --2- 172.21.15.16:0/3216889661 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f49941411b0 0x7f49941509d0 unknown :-1 s=CLOSED pgs=1149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:05.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:05.364+0000 7f4999a00640 1 -- 172.21.15.16:0/3216889661 >> 172.21.15.16:0/3216889661 conn(0x7f4994067930 msgr2=0x7f4994155be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:10:05.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:05.364+0000 7f4999a00640 1 -- 172.21.15.16:0/3216889661 shutdown_connections 2024-03-31T15:10:05.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:05.364+0000 7f4999a00640 1 -- 172.21.15.16:0/3216889661 wait complete. 2024-03-31T15:10:05.381 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-31T15:10:05.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:05.528+0000 7f6342200640 1 Processor -- start 2024-03-31T15:10:05.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:05.532+0000 7f6342200640 1 -- start start 2024-03-31T15:10:05.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:05.532+0000 7f6342200640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f633c15b3d0 0x7f633c0b1cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:05.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:05.532+0000 7f6342200640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f633c15eee0 0x7f633c0b2200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:05.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:05.532+0000 7f6342200640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f633c15f3c0 con 0x7f633c0ce2f0 2024-03-31T15:10:05.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:05.532+0000 7f6342200640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f633c15b7b0 con 0x7f633c15b3d0 2024-03-31T15:10:05.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:05.532+0000 7f6342200640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f633c15f6a0 con 0x7f633c15eee0 2024-03-31T15:10:05.525 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:05.532+0000 7f633be00640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f633c0ce2f0 0x7f633c0cc9c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:57800/0 (socket says 172.21.15.16:57800) 2024-03-31T15:10:05.525 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:05.532+0000 7f633be00640 1 -- 172.21.15.16:0/4187202991 learned_addr learned my addr 172.21.15.16:0/4187202991 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:10:05.525 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:05.532+0000 7f6332a00640 1 --2- 172.21.15.16:0/4187202991 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f633c15eee0 0x7f633c0b2200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:05.525 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:05.532+0000 7f633b400640 1 --2- 172.21.15.16:0/4187202991 >> v2:172.21.15.89:3300/0 conn(0x7f633c15b3d0 0x7f633c0b1cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:05.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:05.532+0000 7f633aa00640 1 -- 172.21.15.16:0/4187202991 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 799776182 0 0) 0x7f633c15f3c0 con 0x7f633c0ce2f0 2024-03-31T15:10:05.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:05.532+0000 7f633aa00640 1 -- 172.21.15.16:0/4187202991 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6320005740 con 0x7f633c0ce2f0 2024-03-31T15:10:05.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:05.532+0000 7f633aa00640 1 -- 172.21.15.16:0/4187202991 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2107618908 0 0) 0x7f6320005740 con 0x7f633c0ce2f0 2024-03-31T15:10:05.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:05.532+0000 7f633aa00640 1 -- 172.21.15.16:0/4187202991 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f633c15eee0 msgr2=0x7f633c0b2200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:10:05.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:05.532+0000 7f633aa00640 1 --2- 172.21.15.16:0/4187202991 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f633c15eee0 0x7f633c0b2200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:05.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:05.532+0000 7f633aa00640 1 -- 172.21.15.16:0/4187202991 >> v2:172.21.15.89:3300/0 conn(0x7f633c15b3d0 msgr2=0x7f633c0b1cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:10:05.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:05.532+0000 7f633aa00640 1 --2- 172.21.15.16:0/4187202991 >> v2:172.21.15.89:3300/0 conn(0x7f633c15b3d0 0x7f633c0b1cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:05.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:05.532+0000 7f633aa00640 1 -- 172.21.15.16:0/4187202991 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f633c15f980 con 0x7f633c0ce2f0 2024-03-31T15:10:05.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:05.532+0000 7f633aa00640 1 -- 172.21.15.16:0/4187202991 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 214807623 0 0) 0x7f6334001a00 con 0x7f633c0ce2f0 2024-03-31T15:10:05.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:05.532+0000 7f633b400640 1 --2- 172.21.15.16:0/4187202991 >> v2:172.21.15.89:3300/0 conn(0x7f633c15b3d0 0x7f633c0b1cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:10:05.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:05.532+0000 7f633aa00640 1 -- 172.21.15.16:0/4187202991 >> v1:172.21.15.16:6789/0 conn(0x7f633c0ce2f0 legacy=0x7f633c0cc9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:10:05.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:05.532+0000 7f633aa00640 1 --2- 172.21.15.16:0/4187202991 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6320005bd0 0x7f6320005ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:05.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:05.532+0000 7f633aa00640 1 --2- 172.21.15.16:0/4187202991 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f633c15eee0 0x7f6320006a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:05.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:05.532+0000 7f633aa00640 1 --2- 172.21.15.16:0/4187202991 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6320006f60 0x7f63200073e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:05.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:05.532+0000 7f633aa00640 1 -- 172.21.15.16:0/4187202991 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f633c15b7b0 con 0x7f6320006f60 2024-03-31T15:10:05.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:05.532+0000 7f633aa00640 1 -- 172.21.15.16:0/4187202991 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f633c15f6a0 con 0x7f6320005bd0 2024-03-31T15:10:05.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:05.532+0000 7f633aa00640 1 -- 172.21.15.16:0/4187202991 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f632000dbe0 con 0x7f633c15eee0 2024-03-31T15:10:05.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:05.532+0000 7f633be00640 1 --2- 172.21.15.16:0/4187202991 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6320006f60 0x7f63200073e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:05.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:05.532+0000 7f633b400640 1 --2- 172.21.15.16:0/4187202991 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6320005bd0 0x7f6320005ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:05.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:05.532+0000 7f6332a00640 1 --2- 172.21.15.16:0/4187202991 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f633c15eee0 0x7f6320006a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:05.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:05.532+0000 7f633be00640 1 -- 172.21.15.16:0/4187202991 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f633c15eee0 msgr2=0x7f6320006a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:10:05.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:05.532+0000 7f633be00640 1 --2- 172.21.15.16:0/4187202991 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f633c15eee0 0x7f6320006a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:05.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:05.532+0000 7f633be00640 1 -- 172.21.15.16:0/4187202991 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6320005bd0 msgr2=0x7f6320005ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:10:05.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:05.532+0000 7f633be00640 1 --2- 172.21.15.16:0/4187202991 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6320005bd0 0x7f6320005ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:05.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:05.532+0000 7f633be00640 1 -- 172.21.15.16:0/4187202991 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6320005740 con 0x7f6320006f60 2024-03-31T15:10:05.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:05.532+0000 7f6332a00640 1 --2- 172.21.15.16:0/4187202991 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f633c15eee0 0x7f6320006a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:10:05.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:05.536+0000 7f633be00640 1 --2- 172.21.15.16:0/4187202991 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6320006f60 0x7f63200073e0 secure :-1 s=READY pgs=1151 cs=0 l=1 rev1=1 crypto rx=0x7f633400a870 tx=0x7f633401d3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=649e42aa11dfbe3 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:10:05.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:05.536+0000 7f633aa00640 1 -- 172.21.15.16:0/4187202991 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f633400a4c0 con 0x7f6320006f60 2024-03-31T15:10:05.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:05.536+0000 7f633aa00640 1 -- 172.21.15.16:0/4187202991 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f63340ad650 con 0x7f6320006f60 2024-03-31T15:10:05.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:05.536+0000 7f633aa00640 1 -- 172.21.15.16:0/4187202991 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63340ada40 con 0x7f6320006f60 2024-03-31T15:10:05.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:05.536+0000 7f6342200640 1 -- 172.21.15.16:0/4187202991 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6320006f60 msgr2=0x7f63200073e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:10:05.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:05.536+0000 7f6342200640 1 --2- 172.21.15.16:0/4187202991 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6320006f60 0x7f63200073e0 secure :-1 s=READY pgs=1151 cs=0 l=1 rev1=1 crypto rx=0x7f633400a870 tx=0x7f633401d3a0 comp rx=0 tx=0).stop 2024-03-31T15:10:05.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:05.536+0000 7f633be00640 1 -- 172.21.15.16:0/4187202991 reap_dead start 2024-03-31T15:10:05.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:05.536+0000 7f6342200640 1 -- 172.21.15.16:0/4187202991 shutdown_connections 2024-03-31T15:10:05.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:05.536+0000 7f6342200640 1 -- 172.21.15.16:0/4187202991 >> 172.21.15.16:0/4187202991 conn(0x7f633c06b620 msgr2=0x7f633c06ba30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:10:05.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:05.536+0000 7f6342200640 1 -- 172.21.15.16:0/4187202991 shutdown_connections 2024-03-31T15:10:05.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:05.536+0000 7f6342200640 1 -- 172.21.15.16:0/4187202991 wait complete. 2024-03-31T15:10:05.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:05.540+0000 7f6342200640 1 Processor -- start 2024-03-31T15:10:05.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:05.540+0000 7f6342200640 1 -- start start 2024-03-31T15:10:05.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:05.540+0000 7f6342200640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6320006f60 0x7f633c144b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:05.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:05.540+0000 7f6342200640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f633c0b10d0 0x7f633c0b0670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:05.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:05.540+0000 7f6342200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f633c15ceb0 0x7f633c15d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:05.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:05.540+0000 7f6342200640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f633c0b4350 con 0x7f633c15ceb0 2024-03-31T15:10:05.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:05.540+0000 7f6342200640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f633c0717c0 con 0x7f6320006f60 2024-03-31T15:10:05.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:05.540+0000 7f6342200640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f633c0b44d0 con 0x7f633c0b10d0 2024-03-31T15:10:05.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:05.540+0000 7f633be00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f633c15ceb0 0x7f633c15d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:05.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:05.540+0000 7f633be00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f633c15ceb0 0x7f633c15d330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39582/0 (socket says 172.21.15.16:39582) 2024-03-31T15:10:05.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:05.540+0000 7f633be00640 1 -- 172.21.15.16:0/44832289 learned_addr learned my addr 172.21.15.16:0/44832289 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:10:05.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:05.540+0000 7f633b400640 1 --2- 172.21.15.16:0/44832289 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6320006f60 0x7f633c144b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:05.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:05.540+0000 7f6332a00640 1 --2- 172.21.15.16:0/44832289 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f633c0b10d0 0x7f633c0b0670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:05.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:05.540+0000 7f633be00640 1 -- 172.21.15.16:0/44832289 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f633c0b10d0 msgr2=0x7f633c0b0670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:10:05.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:05.540+0000 7f633be00640 1 --2- 172.21.15.16:0/44832289 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f633c0b10d0 0x7f633c0b0670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:05.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:05.540+0000 7f633be00640 1 -- 172.21.15.16:0/44832289 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6320006f60 msgr2=0x7f633c144b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:10:05.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:05.540+0000 7f633be00640 1 --2- 172.21.15.16:0/44832289 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6320006f60 0x7f633c144b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:05.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:05.540+0000 7f633be00640 1 -- 172.21.15.16:0/44832289 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f633c0b1570 con 0x7f633c15ceb0 2024-03-31T15:10:05.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:05.540+0000 7f633be00640 1 --2- 172.21.15.16:0/44832289 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f633c15ceb0 0x7f633c15d330 secure :-1 s=READY pgs=1152 cs=0 l=1 rev1=1 crypto rx=0x7f633400a870 tx=0x7f6320007a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:10:05.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:05.540+0000 7f6333e00640 1 -- 172.21.15.16:0/44832289 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63340bc070 con 0x7f633c15ceb0 2024-03-31T15:10:05.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:05.540+0000 7f6333e00640 1 -- 172.21.15.16:0/44832289 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f63340011d0 con 0x7f633c15ceb0 2024-03-31T15:10:05.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:05.540+0000 7f6333e00640 1 -- 172.21.15.16:0/44832289 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63340b2040 con 0x7f633c15ceb0 2024-03-31T15:10:05.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:05.540+0000 7f6342200640 1 -- 172.21.15.16:0/44832289 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f633c0b1850 con 0x7f633c15ceb0 2024-03-31T15:10:05.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:05.540+0000 7f6342200640 1 -- 172.21.15.16:0/44832289 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f633c0b1e10 con 0x7f633c15ceb0 2024-03-31T15:10:05.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:05.540+0000 7f6342200640 1 -- 172.21.15.16:0/44832289 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f633c069950 con 0x7f633c15ceb0 2024-03-31T15:10:05.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:05.544+0000 7f6333e00640 1 -- 172.21.15.16:0/44832289 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f63340bf070 con 0x7f633c15ceb0 2024-03-31T15:10:05.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:05.544+0000 7f6333e00640 1 --2- 172.21.15.16:0/44832289 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f63180683d0 0x7f631806a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:05.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:05.544+0000 7f6333e00640 1 -- 172.21.15.16:0/44832289 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 5795+0+0 (secure 0 0 0) 0x7f6334120920 con 0x7f633c15ceb0 2024-03-31T15:10:05.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:05.544+0000 7f633b400640 1 --2- 172.21.15.16:0/44832289 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f63180683d0 0x7f631806a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:05.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:05.544+0000 7f633b400640 1 --2- 172.21.15.16:0/44832289 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f63180683d0 0x7f631806a890 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f632c0093f0 tx=0x7f632c009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:10:05.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:05.548+0000 7f6333e00640 1 -- 172.21.15.16:0/44832289 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f63340170b0 con 0x7f633c15ceb0 2024-03-31T15:10:05.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:05.748+0000 7f6342200640 1 -- 172.21.15.16:0/44832289 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f633c0cdea0 con 0x7f633c15ceb0 2024-03-31T15:10:05.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:05.748+0000 7f6333e00640 1 -- 172.21.15.16:0/44832289 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 64 v64) ==== 94+0+5174 (secure 0 0 0) 0x7f63340019c0 con 0x7f633c15ceb0 2024-03-31T15:10:05.740 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:10:05.740 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":64,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6494,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/2445558494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6835","nonce":2445558494},{"type":"v1","addr":"172.21.15.89:6837","nonce":2445558494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":56},{"gid":7270,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/3185913858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6833","nonce":3185913858},{"type":"v1","addr":"172.21.15.16:6835","nonce":3185913858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":56},{"gid":7279,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6836/1599468493","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":1599468493},{"type":"v1","addr":"172.21.15.89:6836","nonce":1599468493}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":56}],"filesystems":[{"mdsmap":{"epoch":64,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-31T15:09:50.926196+0000","modified":"2024-03-31T15:10:05.367356+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":82,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7749},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7749":{"gid":7749,"name":"c","rank":0,"incarnation":64,"state":"up:replay","state_seq":1,"addr":"172.21.15.16:6834/818074038","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":818074038},{"type":"v1","addr":"172.21.15.16:6834","nonce":818074038}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-03-31T15:10:05.740 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 64 2024-03-31T15:10:05.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:05.748+0000 7f6342200640 1 -- 172.21.15.16:0/44832289 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f63180683d0 msgr2=0x7f631806a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:10:05.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:05.748+0000 7f6342200640 1 --2- 172.21.15.16:0/44832289 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f63180683d0 0x7f631806a890 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f632c0093f0 tx=0x7f632c009190 comp rx=0 tx=0).stop 2024-03-31T15:10:05.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:05.748+0000 7f6342200640 1 -- 172.21.15.16:0/44832289 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f633c15ceb0 msgr2=0x7f633c15d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:10:05.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:05.748+0000 7f6342200640 1 --2- 172.21.15.16:0/44832289 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f633c15ceb0 0x7f633c15d330 secure :-1 s=READY pgs=1152 cs=0 l=1 rev1=1 crypto rx=0x7f633400a870 tx=0x7f6320007a50 comp rx=0 tx=0).stop 2024-03-31T15:10:05.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:05.748+0000 7f6342200640 1 -- 172.21.15.16:0/44832289 shutdown_connections 2024-03-31T15:10:05.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:05.748+0000 7f6342200640 1 --2- 172.21.15.16:0/44832289 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f63180683d0 0x7f631806a890 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:05.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:05.748+0000 7f6342200640 1 --2- 172.21.15.16:0/44832289 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f633c15ceb0 0x7f633c15d330 unknown :-1 s=CLOSED pgs=1152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:05.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:05.748+0000 7f6342200640 1 --2- 172.21.15.16:0/44832289 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f633c0b10d0 0x7f633c0b0670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:05.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:05.748+0000 7f6342200640 1 --2- 172.21.15.16:0/44832289 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6320006f60 0x7f633c144b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:05.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:05.748+0000 7f6342200640 1 -- 172.21.15.16:0/44832289 >> 172.21.15.16:0/44832289 conn(0x7f633c06b620 msgr2=0x7f633c158110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:10:05.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:05.748+0000 7f6342200640 1 -- 172.21.15.16:0/44832289 shutdown_connections 2024-03-31T15:10:05.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:05.748+0000 7f6342200640 1 -- 172.21.15.16:0/44832289 wait complete. 2024-03-31T15:10:06.164 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 64, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-31T15:09:50.926196+0000', 'modified': '2024-03-31T15:10:05.367356+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 82, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7749}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7749': {'gid': 7749, 'name': 'c', 'rank': 0, 'incarnation': 64, 'state': 'up:replay', 'state_seq': 1, 'addr': '172.21.15.16:6834/818074038', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6832', 'nonce': 818074038}, {'type': 'v1', 'addr': '172.21.15.16:6834', 'nonce': 818074038}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [13], 'metadata_pool': 12, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-03-31T15:10:06.164 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7749:up:replay 2024-03-31T15:10:07.165 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-31T15:10:07.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:07.324+0000 7f7810000640 1 Processor -- start 2024-03-31T15:10:07.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:07.324+0000 7f7810000640 1 -- start start 2024-03-31T15:10:07.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:07.328+0000 7f7810000640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f7808145d10 0x7f78080cf790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:07.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:07.328+0000 7f7810000640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f78080728b0 0x7f78080b1cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:07.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:07.328+0000 7f7810000640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7808160e70 con 0x7f78080b2200 2024-03-31T15:10:07.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:07.328+0000 7f7810000640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f78081602f0 con 0x7f78080728b0 2024-03-31T15:10:07.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:07.328+0000 7f7810000640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f78081605d0 con 0x7f7808145d10 2024-03-31T15:10:07.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:07.328+0000 7f780cc00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f78080728b0 0x7f78080b1cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:07.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:07.328+0000 7f780e000640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f78080b2200 0x7f780815cd50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:40596/0 (socket says 172.21.15.16:40596) 2024-03-31T15:10:07.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:07.328+0000 7f780e000640 1 -- 172.21.15.16:0/3325653282 learned_addr learned my addr 172.21.15.16:0/3325653282 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:10:07.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:07.328+0000 7f780d600640 1 --2- 172.21.15.16:0/3325653282 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f7808145d10 0x7f78080cf790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:07.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:07.328+0000 7f780cc00640 1 -- 172.21.15.16:0/3325653282 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f7808145d10 msgr2=0x7f78080cf790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:10:07.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:07.328+0000 7f780cc00640 1 --2- 172.21.15.16:0/3325653282 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f7808145d10 0x7f78080cf790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:07.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:07.328+0000 7f780cc00640 1 -- 172.21.15.16:0/3325653282 >> v1:172.21.15.16:6789/0 conn(0x7f78080b2200 legacy=0x7f780815cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:10:07.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:07.328+0000 7f780cc00640 1 -- 172.21.15.16:0/3325653282 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78080cfcd0 con 0x7f78080728b0 2024-03-31T15:10:07.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:07.328+0000 7f7807e00640 1 -- 172.21.15.16:0/3325653282 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3734453618 0 0) 0x7f7808160e70 con 0x7f78080b2200 2024-03-31T15:10:07.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:07.328+0000 7f780cc00640 1 --2- 172.21.15.16:0/3325653282 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f78080728b0 0x7f78080b1cc0 secure :-1 s=READY pgs=1154 cs=0 l=1 rev1=1 crypto rx=0x7f77f40097c0 tx=0x7f77f401c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59111223fb390c5d server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:10:07.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:07.332+0000 7f7807e00640 1 -- 172.21.15.16:0/3325653282 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77f40b0070 con 0x7f78080728b0 2024-03-31T15:10:07.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:07.332+0000 7f7807e00640 1 -- 172.21.15.16:0/3325653282 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f77f40ab600 con 0x7f78080728b0 2024-03-31T15:10:07.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:07.332+0000 7f7807e00640 1 -- 172.21.15.16:0/3325653282 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77f40ab9d0 con 0x7f78080728b0 2024-03-31T15:10:07.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:07.332+0000 7f7810000640 1 -- 172.21.15.16:0/3325653282 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f78080728b0 msgr2=0x7f78080b1cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:10:07.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:07.332+0000 7f7810000640 1 --2- 172.21.15.16:0/3325653282 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f78080728b0 0x7f78080b1cc0 secure :-1 s=READY pgs=1154 cs=0 l=1 rev1=1 crypto rx=0x7f77f40097c0 tx=0x7f77f401c4a0 comp rx=0 tx=0).stop 2024-03-31T15:10:07.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:07.332+0000 7f7810000640 1 -- 172.21.15.16:0/3325653282 shutdown_connections 2024-03-31T15:10:07.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:07.332+0000 7f7810000640 1 --2- 172.21.15.16:0/3325653282 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f78080728b0 0x7f78080b1cc0 unknown :-1 s=CLOSED pgs=1154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:07.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:07.332+0000 7f7810000640 1 --2- 172.21.15.16:0/3325653282 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f7808145d10 0x7f78080cf790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:07.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:07.332+0000 7f7810000640 1 -- 172.21.15.16:0/3325653282 >> 172.21.15.16:0/3325653282 conn(0x7f780806b620 msgr2=0x7f780806ba30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:10:07.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:07.332+0000 7f7810000640 1 -- 172.21.15.16:0/3325653282 shutdown_connections 2024-03-31T15:10:07.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:07.332+0000 7f7810000640 1 -- 172.21.15.16:0/3325653282 wait complete. 2024-03-31T15:10:07.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:07.332+0000 7f7810000640 1 Processor -- start 2024-03-31T15:10:07.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:07.332+0000 7f7810000640 1 -- start start 2024-03-31T15:10:07.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:07.332+0000 7f7810000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f78080728b0 0x7f78080b0dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:07.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:07.332+0000 7f7810000640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f78080b2200 0x7f78080b33d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:07.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:07.332+0000 7f7810000640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7808145d10 0x7f78081f6c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:07.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:07.332+0000 7f7810000640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f780815d870 con 0x7f78080728b0 2024-03-31T15:10:07.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:07.332+0000 7f7810000640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f78080b2650 con 0x7f7808145d10 2024-03-31T15:10:07.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:07.332+0000 7f7810000640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f78080cf350 con 0x7f78080b2200 2024-03-31T15:10:07.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:07.332+0000 7f780d600640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f78080728b0 0x7f78080b0dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:07.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:07.336+0000 7f780d600640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f78080728b0 0x7f78080b0dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36504/0 (socket says 172.21.15.16:36504) 2024-03-31T15:10:07.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:07.336+0000 7f780d600640 1 -- 172.21.15.16:0/2835751864 learned_addr learned my addr 172.21.15.16:0/2835751864 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:10:07.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:07.336+0000 7f780e000640 1 --2- 172.21.15.16:0/2835751864 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7808145d10 0x7f78081f6c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:07.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:07.336+0000 7f780d600640 1 -- 172.21.15.16:0/2835751864 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f78080b2200 msgr2=0x7f78080b33d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:10:07.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:07.336+0000 7f780d600640 1 --2- 172.21.15.16:0/2835751864 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f78080b2200 0x7f78080b33d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:07.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:07.336+0000 7f780d600640 1 -- 172.21.15.16:0/2835751864 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7808145d10 msgr2=0x7f78081f6c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:10:07.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:07.336+0000 7f780d600640 1 --2- 172.21.15.16:0/2835751864 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7808145d10 0x7f78081f6c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:07.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:07.336+0000 7f780d600640 1 -- 172.21.15.16:0/2835751864 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78080b3c90 con 0x7f78080728b0 2024-03-31T15:10:07.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:07.336+0000 7f780e000640 1 --2- 172.21.15.16:0/2835751864 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7808145d10 0x7f78081f6c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:10:07.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:07.336+0000 7f780d600640 1 --2- 172.21.15.16:0/2835751864 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f78080728b0 0x7f78080b0dd0 secure :-1 s=READY pgs=1155 cs=0 l=1 rev1=1 crypto rx=0x7f77fc00d800 tx=0x7f77fc00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:10:07.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:07.336+0000 7f7805600640 1 -- 172.21.15.16:0/2835751864 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77fc00b6c0 con 0x7f78080728b0 2024-03-31T15:10:07.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:07.336+0000 7f7805600640 1 -- 172.21.15.16:0/2835751864 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f77fc019070 con 0x7f78080728b0 2024-03-31T15:10:07.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:07.336+0000 7f7805600640 1 -- 172.21.15.16:0/2835751864 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77fc00ba60 con 0x7f78080728b0 2024-03-31T15:10:07.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:07.336+0000 7f7810000640 1 -- 172.21.15.16:0/2835751864 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f78080b3f70 con 0x7f78080728b0 2024-03-31T15:10:07.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:07.336+0000 7f7810000640 1 -- 172.21.15.16:0/2835751864 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f78080b4530 con 0x7f78080728b0 2024-03-31T15:10:07.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:07.336+0000 7f7810000640 1 -- 172.21.15.16:0/2835751864 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7808069950 con 0x7f78080728b0 2024-03-31T15:10:07.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:07.336+0000 7f7805600640 1 -- 172.21.15.16:0/2835751864 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f77fc00bc00 con 0x7f78080728b0 2024-03-31T15:10:07.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:07.336+0000 7f7805600640 1 --2- 172.21.15.16:0/2835751864 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f77e8068380 0x7f77e806a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:07.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:07.336+0000 7f7805600640 1 -- 172.21.15.16:0/2835751864 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 5795+0+0 (secure 0 0 0) 0x7f77fc086f60 con 0x7f78080728b0 2024-03-31T15:10:07.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:07.340+0000 7f780cc00640 1 --2- 172.21.15.16:0/2835751864 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f77e8068380 0x7f77e806a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:07.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:07.340+0000 7f780cc00640 1 --2- 172.21.15.16:0/2835751864 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f77e8068380 0x7f77e806a840 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f77f40ad000 tx=0x7f77f4004740 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:10:07.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:07.344+0000 7f7805600640 1 -- 172.21.15.16:0/2835751864 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f77fc057420 con 0x7f78080728b0 2024-03-31T15:10:07.347 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-31T15:10:07.350 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-31T15:10:07.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:07.544+0000 7f7810000640 1 -- 172.21.15.16:0/2835751864 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f78080ce950 con 0x7f78080728b0 2024-03-31T15:10:07.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:07.544+0000 7f7805600640 1 -- 172.21.15.16:0/2835751864 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 66 v66) ==== 94+0+5174 (secure 0 0 0) 0x7f77fc056d20 con 0x7f78080728b0 2024-03-31T15:10:07.537 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:10:07.537 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":66,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6494,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/2445558494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6835","nonce":2445558494},{"type":"v1","addr":"172.21.15.89:6837","nonce":2445558494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":56},{"gid":7270,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/3185913858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6833","nonce":3185913858},{"type":"v1","addr":"172.21.15.16:6835","nonce":3185913858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":56},{"gid":7279,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6836/1599468493","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":1599468493},{"type":"v1","addr":"172.21.15.89:6836","nonce":1599468493}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":56}],"filesystems":[{"mdsmap":{"epoch":66,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-31T15:09:50.926196+0000","modified":"2024-03-31T15:10:06.373846+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":82,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7749},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7749":{"gid":7749,"name":"c","rank":0,"incarnation":64,"state":"up:rejoin","state_seq":3,"addr":"172.21.15.16:6834/818074038","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":818074038},{"type":"v1","addr":"172.21.15.16:6834","nonce":818074038}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-03-31T15:10:07.537 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 66 2024-03-31T15:10:07.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:07.548+0000 7f7810000640 1 -- 172.21.15.16:0/2835751864 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f77e8068380 msgr2=0x7f77e806a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:10:07.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:07.548+0000 7f7810000640 1 --2- 172.21.15.16:0/2835751864 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f77e8068380 0x7f77e806a840 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f77f40ad000 tx=0x7f77f4004740 comp rx=0 tx=0).stop 2024-03-31T15:10:07.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:07.548+0000 7f7810000640 1 -- 172.21.15.16:0/2835751864 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f78080728b0 msgr2=0x7f78080b0dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:10:07.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:07.548+0000 7f7810000640 1 --2- 172.21.15.16:0/2835751864 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f78080728b0 0x7f78080b0dd0 secure :-1 s=READY pgs=1155 cs=0 l=1 rev1=1 crypto rx=0x7f77fc00d800 tx=0x7f77fc00dcc0 comp rx=0 tx=0).stop 2024-03-31T15:10:07.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:07.548+0000 7f7810000640 1 -- 172.21.15.16:0/2835751864 shutdown_connections 2024-03-31T15:10:07.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:07.548+0000 7f7810000640 1 --2- 172.21.15.16:0/2835751864 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f77e8068380 0x7f77e806a840 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:07.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:07.548+0000 7f7810000640 1 --2- 172.21.15.16:0/2835751864 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7808145d10 0x7f78081f6c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:07.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:07.548+0000 7f7810000640 1 --2- 172.21.15.16:0/2835751864 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f78080b2200 0x7f78080b33d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:07.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:07.548+0000 7f7810000640 1 --2- 172.21.15.16:0/2835751864 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f78080728b0 0x7f78080b0dd0 unknown :-1 s=CLOSED pgs=1155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:07.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:07.548+0000 7f7810000640 1 -- 172.21.15.16:0/2835751864 >> 172.21.15.16:0/2835751864 conn(0x7f780806b620 msgr2=0x7f780815ba50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:10:07.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:07.548+0000 7f7810000640 1 -- 172.21.15.16:0/2835751864 shutdown_connections 2024-03-31T15:10:07.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:07.548+0000 7f7810000640 1 -- 172.21.15.16:0/2835751864 wait complete. 2024-03-31T15:10:07.961 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 66, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-31T15:09:50.926196+0000', 'modified': '2024-03-31T15:10:06.373846+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 82, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7749}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7749': {'gid': 7749, 'name': 'c', 'rank': 0, 'incarnation': 64, 'state': 'up:rejoin', 'state_seq': 3, 'addr': '172.21.15.16:6834/818074038', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6832', 'nonce': 818074038}, {'type': 'v1', 'addr': '172.21.15.16:6834', 'nonce': 818074038}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [13], 'metadata_pool': 12, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-03-31T15:10:07.961 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7749:up:rejoin 2024-03-31T15:10:08.963 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-31T15:10:09.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.112+0000 7fef16c00640 1 Processor -- start 2024-03-31T15:10:09.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.112+0000 7fef16c00640 1 -- start start 2024-03-31T15:10:09.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.112+0000 7fef16c00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fef100cf250 0x7fef100b4630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:09.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.112+0000 7fef16c00640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fef100cf850 con 0x7fef101529c0 2024-03-31T15:10:09.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.112+0000 7fef16c00640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fef100cfb30 con 0x7fef1015fd70 2024-03-31T15:10:09.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.112+0000 7fef16c00640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7fef100d0460 con 0x7fef100cf250 2024-03-31T15:10:09.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.112+0000 7fef14c00640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fef101529c0 0x7fef10152e50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:40604/0 (socket says 172.21.15.16:40604) 2024-03-31T15:10:09.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.112+0000 7fef14c00640 1 -- 172.21.15.16:0/4102895634 learned_addr learned my addr 172.21.15.16:0/4102895634 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:10:09.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.112+0000 7fef07e00640 1 --2- 172.21.15.16:0/4102895634 >> v2:172.21.15.89:3300/0 conn(0x7fef100cf250 0x7fef100b4630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:09.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.112+0000 7fef0f400640 1 -- 172.21.15.16:0/4102895634 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3908364077 0 0) 0x7fef100cf850 con 0x7fef101529c0 2024-03-31T15:10:09.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.112+0000 7fef0f400640 1 -- 172.21.15.16:0/4102895634 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7feef4003530 con 0x7fef101529c0 2024-03-31T15:10:09.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.112+0000 7fef0f400640 1 -- 172.21.15.16:0/4102895634 <== mon.1 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3909177048 0 0) 0x7fef100cfb30 con 0x7fef1015fd70 2024-03-31T15:10:09.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.112+0000 7fef0f400640 1 -- 172.21.15.16:0/4102895634 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fef100cf850 con 0x7fef1015fd70 2024-03-31T15:10:09.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.112+0000 7fef0f400640 1 -- 172.21.15.16:0/4102895634 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3283607874 0 0) 0x7feef4003530 con 0x7fef101529c0 2024-03-31T15:10:09.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.112+0000 7fef07e00640 1 -- 172.21.15.16:0/4102895634 >> v1:172.21.15.89:6789/0 conn(0x7fef1015fd70 legacy=0x7fef10160150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:10:09.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.112+0000 7fef07e00640 1 -- 172.21.15.16:0/4102895634 >> v1:172.21.15.16:6789/0 conn(0x7fef101529c0 legacy=0x7fef10152e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:10:09.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.112+0000 7fef07e00640 1 -- 172.21.15.16:0/4102895634 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef100cfe10 con 0x7fef100cf250 2024-03-31T15:10:09.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.116+0000 7fef07e00640 1 --2- 172.21.15.16:0/4102895634 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fef100cf250 0x7fef100b4630 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7feef80097f0 tx=0x7feef801c3c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=36c1e43b0bd6e585 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:10:09.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.116+0000 7fef0f400640 1 -- 172.21.15.16:0/4102895634 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feef80b0070 con 0x7fef100cf250 2024-03-31T15:10:09.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.116+0000 7fef0f400640 1 -- 172.21.15.16:0/4102895634 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feef80ab400 con 0x7fef100cf250 2024-03-31T15:10:09.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.116+0000 7fef0f400640 1 -- 172.21.15.16:0/4102895634 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef08001a00 con 0x7fef100cf250 2024-03-31T15:10:09.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.116+0000 7fef16c00640 1 -- 172.21.15.16:0/4102895634 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fef100cf250 msgr2=0x7fef100b4630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:10:09.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.116+0000 7fef16c00640 1 --2- 172.21.15.16:0/4102895634 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fef100cf250 0x7fef100b4630 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7feef80097f0 tx=0x7feef801c3c0 comp rx=0 tx=0).stop 2024-03-31T15:10:09.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.116+0000 7fef16c00640 1 -- 172.21.15.16:0/4102895634 shutdown_connections 2024-03-31T15:10:09.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.116+0000 7fef16c00640 1 --2- 172.21.15.16:0/4102895634 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fef100cf250 0x7fef100b4630 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:09.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.116+0000 7fef16c00640 1 -- 172.21.15.16:0/4102895634 >> 172.21.15.16:0/4102895634 conn(0x7fef100d0760 msgr2=0x7fef100d0b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:10:09.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.116+0000 7fef16c00640 1 -- 172.21.15.16:0/4102895634 shutdown_connections 2024-03-31T15:10:09.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.116+0000 7fef16c00640 1 -- 172.21.15.16:0/4102895634 wait complete. 2024-03-31T15:10:09.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.120+0000 7fef16c00640 1 Processor -- start 2024-03-31T15:10:09.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.120+0000 7fef16c00640 1 -- start start 2024-03-31T15:10:09.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.120+0000 7fef16c00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fef100cf250 0x7fef1015f1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:09.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.120+0000 7fef16c00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fef101529c0 0x7fef1014aef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:09.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.120+0000 7fef16c00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fef1015fd70 0x7fef1014cdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:09.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.120+0000 7fef16c00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fef1016a690 con 0x7fef100cf250 2024-03-31T15:10:09.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.120+0000 7fef16c00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fef1016a510 con 0x7fef101529c0 2024-03-31T15:10:09.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.120+0000 7fef16c00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fef1016a810 con 0x7fef1015fd70 2024-03-31T15:10:09.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.120+0000 7fef0fe00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fef100cf250 0x7fef1015f1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:09.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.120+0000 7fef0fe00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fef100cf250 0x7fef1015f1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36506/0 (socket says 172.21.15.16:36506) 2024-03-31T15:10:09.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.120+0000 7fef0fe00640 1 -- 172.21.15.16:0/321062938 learned_addr learned my addr 172.21.15.16:0/321062938 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:10:09.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.120+0000 7fef07e00640 1 --2- 172.21.15.16:0/321062938 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fef101529c0 0x7fef1014aef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:09.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.120+0000 7fef14c00640 1 --2- 172.21.15.16:0/321062938 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fef1015fd70 0x7fef1014cdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:09.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.120+0000 7fef0fe00640 1 -- 172.21.15.16:0/321062938 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fef1015fd70 msgr2=0x7fef1014cdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:10:09.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.120+0000 7fef0fe00640 1 --2- 172.21.15.16:0/321062938 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fef1015fd70 0x7fef1014cdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:09.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.120+0000 7fef0fe00640 1 -- 172.21.15.16:0/321062938 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fef101529c0 msgr2=0x7fef1014aef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:10:09.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.120+0000 7fef0fe00640 1 --2- 172.21.15.16:0/321062938 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fef101529c0 0x7fef1014aef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:09.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.120+0000 7fef0fe00640 1 -- 172.21.15.16:0/321062938 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef1014ed20 con 0x7fef100cf250 2024-03-31T15:10:09.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.120+0000 7fef07e00640 1 --2- 172.21.15.16:0/321062938 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fef101529c0 0x7fef1014aef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:10:09.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.120+0000 7fef0fe00640 1 --2- 172.21.15.16:0/321062938 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fef100cf250 0x7fef1015f1d0 secure :-1 s=READY pgs=1157 cs=0 l=1 rev1=1 crypto rx=0x7fef0000c500 tx=0x7fef0000c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:10:09.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.120+0000 7fef0cc00640 1 -- 172.21.15.16:0/321062938 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef000052d0 con 0x7fef100cf250 2024-03-31T15:10:09.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.120+0000 7fef0cc00640 1 -- 172.21.15.16:0/321062938 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fef00005490 con 0x7fef100cf250 2024-03-31T15:10:09.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.120+0000 7fef0cc00640 1 -- 172.21.15.16:0/321062938 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef00012850 con 0x7fef100cf250 2024-03-31T15:10:09.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.120+0000 7fef16c00640 1 -- 172.21.15.16:0/321062938 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fef1014f210 con 0x7fef100cf250 2024-03-31T15:10:09.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.120+0000 7fef16c00640 1 -- 172.21.15.16:0/321062938 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fef1014f7d0 con 0x7fef100cf250 2024-03-31T15:10:09.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.124+0000 7fef16c00640 1 -- 172.21.15.16:0/321062938 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fef10069950 con 0x7fef100cf250 2024-03-31T15:10:09.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.124+0000 7fef0cc00640 1 -- 172.21.15.16:0/321062938 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fef000129f0 con 0x7fef100cf250 2024-03-31T15:10:09.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.124+0000 7fef0cc00640 1 --2- 172.21.15.16:0/321062938 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7feeec0683b0 0x7feeec06a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:09.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.124+0000 7fef0cc00640 1 -- 172.21.15.16:0/321062938 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 5795+0+0 (secure 0 0 0) 0x7fef00086780 con 0x7fef100cf250 2024-03-31T15:10:09.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.124+0000 7fef07e00640 1 --2- 172.21.15.16:0/321062938 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7feeec0683b0 0x7feeec06a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:09.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.124+0000 7fef07e00640 1 --2- 172.21.15.16:0/321062938 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7feeec0683b0 0x7feeec06a870 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7feef80ad000 tx=0x7feef8008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:10:09.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.128+0000 7fef0cc00640 1 -- 172.21.15.16:0/321062938 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fef00056c50 con 0x7fef100cf250 2024-03-31T15:10:09.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.328+0000 7fef16c00640 1 -- 172.21.15.16:0/321062938 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fef100b3e40 con 0x7fef100cf250 2024-03-31T15:10:09.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.328+0000 7fef0cc00640 1 -- 172.21.15.16:0/321062938 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 67 v67) ==== 94+0+5181 (secure 0 0 0) 0x7fef00056550 con 0x7fef100cf250 2024-03-31T15:10:09.321 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:10:09.322 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":67,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6494,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/2445558494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6835","nonce":2445558494},{"type":"v1","addr":"172.21.15.89:6837","nonce":2445558494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":56},{"gid":7270,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/3185913858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6833","nonce":3185913858},{"type":"v1","addr":"172.21.15.16:6835","nonce":3185913858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":56},{"gid":7279,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6836/1599468493","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":1599468493},{"type":"v1","addr":"172.21.15.89:6836","nonce":1599468493}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":56}],"filesystems":[{"mdsmap":{"epoch":67,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-31T15:09:50.926196+0000","modified":"2024-03-31T15:10:08.373541+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":82,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7749},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7749":{"gid":7749,"name":"c","rank":0,"incarnation":64,"state":"up:active","state_seq":4,"addr":"172.21.15.16:6834/818074038","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":818074038},{"type":"v1","addr":"172.21.15.16:6834","nonce":818074038}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7749,"qdb_cluster":[7749]},"id":6}]} 2024-03-31T15:10:09.322 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 67 2024-03-31T15:10:09.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.332+0000 7fef16c00640 1 -- 172.21.15.16:0/321062938 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7feeec0683b0 msgr2=0x7feeec06a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:10:09.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.332+0000 7fef16c00640 1 --2- 172.21.15.16:0/321062938 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7feeec0683b0 0x7feeec06a870 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7feef80ad000 tx=0x7feef8008040 comp rx=0 tx=0).stop 2024-03-31T15:10:09.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.332+0000 7fef16c00640 1 -- 172.21.15.16:0/321062938 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fef100cf250 msgr2=0x7fef1015f1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:10:09.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.332+0000 7fef16c00640 1 --2- 172.21.15.16:0/321062938 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fef100cf250 0x7fef1015f1d0 secure :-1 s=READY pgs=1157 cs=0 l=1 rev1=1 crypto rx=0x7fef0000c500 tx=0x7fef0000c9c0 comp rx=0 tx=0).stop 2024-03-31T15:10:09.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.332+0000 7fef16c00640 1 -- 172.21.15.16:0/321062938 shutdown_connections 2024-03-31T15:10:09.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.332+0000 7fef16c00640 1 --2- 172.21.15.16:0/321062938 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7feeec0683b0 0x7feeec06a870 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:09.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.332+0000 7fef16c00640 1 --2- 172.21.15.16:0/321062938 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fef1015fd70 0x7fef1014cdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:09.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.332+0000 7fef16c00640 1 --2- 172.21.15.16:0/321062938 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fef101529c0 0x7fef1014aef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:09.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.332+0000 7fef16c00640 1 --2- 172.21.15.16:0/321062938 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fef100cf250 0x7fef1015f1d0 unknown :-1 s=CLOSED pgs=1157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:09.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.332+0000 7fef16c00640 1 -- 172.21.15.16:0/321062938 >> 172.21.15.16:0/321062938 conn(0x7fef100d0760 msgr2=0x7fef1015bbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:10:09.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.332+0000 7fef16c00640 1 -- 172.21.15.16:0/321062938 shutdown_connections 2024-03-31T15:10:09.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.332+0000 7fef16c00640 1 -- 172.21.15.16:0/321062938 wait complete. 2024-03-31T15:10:09.746 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 67, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-31T15:09:50.926196+0000', 'modified': '2024-03-31T15:10:08.373541+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 82, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7749}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7749': {'gid': 7749, 'name': 'c', 'rank': 0, 'incarnation': 64, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.16:6834/818074038', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6832', 'nonce': 818074038}, {'type': 'v1', 'addr': '172.21.15.16:6834', 'nonce': 818074038}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [13], 'metadata_pool': 12, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 7749, 'qdb_cluster': [7749]} 2024-03-31T15:10:09.746 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-31T15:10:09.746 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-03-31T15:10:09.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.892+0000 7f36bce00640 1 Processor -- start 2024-03-31T15:10:09.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.892+0000 7f36bce00640 1 -- start start 2024-03-31T15:10:09.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.896+0000 7f36bce00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f36b815f650 0x7f36b806f920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:09.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.896+0000 7f36bce00640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f36b80cbe90 con 0x7f36b806dbc0 2024-03-31T15:10:09.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.896+0000 7f36bce00640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f36b80cc170 con 0x7f36b806fe60 2024-03-31T15:10:09.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.896+0000 7f36bce00640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f36b80702d0 con 0x7f36b815f650 2024-03-31T15:10:09.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.896+0000 7f36b5e00640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f36b806dbc0 0x7f36b80cb440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:40608/0 (socket says 172.21.15.16:40608) 2024-03-31T15:10:09.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.896+0000 7f36b5e00640 1 -- 172.21.15.16:0/1924321497 learned_addr learned my addr 172.21.15.16:0/1924321497 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:10:09.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.896+0000 7f36ad400640 1 --2- 172.21.15.16:0/1924321497 >> v2:172.21.15.89:3300/0 conn(0x7f36b815f650 0x7f36b806f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:09.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.896+0000 7f36b5400640 1 -- 172.21.15.16:0/1924321497 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1557211002 0 0) 0x7f36b80cbe90 con 0x7f36b806dbc0 2024-03-31T15:10:09.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.896+0000 7f36b5400640 1 -- 172.21.15.16:0/1924321497 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f369c003530 con 0x7f36b806dbc0 2024-03-31T15:10:09.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.896+0000 7f36ad400640 1 -- 172.21.15.16:0/1924321497 >> v1:172.21.15.89:6789/0 conn(0x7f36b806fe60 legacy=0x7f36b8162180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:10:09.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.896+0000 7f36ad400640 1 -- 172.21.15.16:0/1924321497 >> v1:172.21.15.16:6789/0 conn(0x7f36b806dbc0 legacy=0x7f36b80cb440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:10:09.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.896+0000 7f36ad400640 1 -- 172.21.15.16:0/1924321497 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f36b815ee50 con 0x7f36b815f650 2024-03-31T15:10:09.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.900+0000 7f36ad400640 1 --2- 172.21.15.16:0/1924321497 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f36b815f650 0x7f36b806f920 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f36a00097c0 tx=0x7f36a001c410 comp rx=0 tx=0).ready entity=mon.1 client_cookie=54f3796ea586032d server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:10:09.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.900+0000 7f36b5400640 1 -- 172.21.15.16:0/1924321497 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36a00b0070 con 0x7f36b815f650 2024-03-31T15:10:09.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.900+0000 7f36b5400640 1 -- 172.21.15.16:0/1924321497 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f36a00ab600 con 0x7f36b815f650 2024-03-31T15:10:09.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.900+0000 7f36b5400640 1 -- 172.21.15.16:0/1924321497 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36a00ab9d0 con 0x7f36b815f650 2024-03-31T15:10:09.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.900+0000 7f36bce00640 1 -- 172.21.15.16:0/1924321497 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f36b815f650 msgr2=0x7f36b806f920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:10:09.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.900+0000 7f36bce00640 1 --2- 172.21.15.16:0/1924321497 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f36b815f650 0x7f36b806f920 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f36a00097c0 tx=0x7f36a001c410 comp rx=0 tx=0).stop 2024-03-31T15:10:09.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.900+0000 7f36bce00640 1 -- 172.21.15.16:0/1924321497 shutdown_connections 2024-03-31T15:10:09.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.900+0000 7f36bce00640 1 --2- 172.21.15.16:0/1924321497 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f36b815f650 0x7f36b806f920 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:09.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.900+0000 7f36bce00640 1 -- 172.21.15.16:0/1924321497 >> 172.21.15.16:0/1924321497 conn(0x7f36b8067330 msgr2=0x7f36b8067740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:10:09.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.900+0000 7f36bce00640 1 -- 172.21.15.16:0/1924321497 shutdown_connections 2024-03-31T15:10:09.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.900+0000 7f36bce00640 1 -- 172.21.15.16:0/1924321497 wait complete. 2024-03-31T15:10:09.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.900+0000 7f36bce00640 1 Processor -- start 2024-03-31T15:10:09.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.900+0000 7f36bce00640 1 -- start start 2024-03-31T15:10:09.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.904+0000 7f36bce00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f36b806dbc0 0x7f36b8156e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:09.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.904+0000 7f36bce00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f36b806fe60 0x7f36b814bba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:09.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.904+0000 7f36bce00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f36b815f650 0x7f36b8150520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:09.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.904+0000 7f36bce00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f36b8162ca0 con 0x7f36b806dbc0 2024-03-31T15:10:09.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.904+0000 7f36bce00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f36b806dfa0 con 0x7f36b815f650 2024-03-31T15:10:09.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.904+0000 7f36bce00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f36b806c950 con 0x7f36b806fe60 2024-03-31T15:10:09.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.904+0000 7f36b5e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f36b806dbc0 0x7f36b8156e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:09.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.904+0000 7f36b5e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f36b806dbc0 0x7f36b8156e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36514/0 (socket says 172.21.15.16:36514) 2024-03-31T15:10:09.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.904+0000 7f36b5e00640 1 -- 172.21.15.16:0/2510492483 learned_addr learned my addr 172.21.15.16:0/2510492483 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:10:09.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.904+0000 7f36b6800640 1 --2- 172.21.15.16:0/2510492483 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f36b815f650 0x7f36b8150520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:09.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.904+0000 7f36ad400640 1 --2- 172.21.15.16:0/2510492483 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f36b806fe60 0x7f36b814bba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:09.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.904+0000 7f36b5e00640 1 -- 172.21.15.16:0/2510492483 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f36b806fe60 msgr2=0x7f36b814bba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:10:09.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.904+0000 7f36b5e00640 1 --2- 172.21.15.16:0/2510492483 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f36b806fe60 0x7f36b814bba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:09.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.904+0000 7f36b5e00640 1 -- 172.21.15.16:0/2510492483 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f36b815f650 msgr2=0x7f36b8150520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:10:09.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.904+0000 7f36b5e00640 1 --2- 172.21.15.16:0/2510492483 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f36b815f650 0x7f36b8150520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:09.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.904+0000 7f36b5e00640 1 -- 172.21.15.16:0/2510492483 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f36b8157460 con 0x7f36b806dbc0 2024-03-31T15:10:09.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.904+0000 7f36b5e00640 1 --2- 172.21.15.16:0/2510492483 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f36b806dbc0 0x7f36b8156e80 secure :-1 s=READY pgs=1159 cs=0 l=1 rev1=1 crypto rx=0x7f36a800c450 tx=0x7f36a800c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:10:09.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.904+0000 7f36aea00640 1 -- 172.21.15.16:0/2510492483 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36a8017070 con 0x7f36b806dbc0 2024-03-31T15:10:09.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.904+0000 7f36aea00640 1 -- 172.21.15.16:0/2510492483 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f36a8005cf0 con 0x7f36b806dbc0 2024-03-31T15:10:09.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.904+0000 7f36aea00640 1 -- 172.21.15.16:0/2510492483 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36a8007270 con 0x7f36b806dbc0 2024-03-31T15:10:09.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.904+0000 7f36bce00640 1 -- 172.21.15.16:0/2510492483 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f36b8157740 con 0x7f36b806dbc0 2024-03-31T15:10:09.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.904+0000 7f36bce00640 1 -- 172.21.15.16:0/2510492483 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f36b8157d00 con 0x7f36b806dbc0 2024-03-31T15:10:09.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.908+0000 7f36aea00640 1 -- 172.21.15.16:0/2510492483 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f36a801a070 con 0x7f36b806dbc0 2024-03-31T15:10:09.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.908+0000 7f36c0217000 1 Processor -- start 2024-03-31T15:10:09.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.908+0000 7f36c0217000 1 -- start start 2024-03-31T15:10:09.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.908+0000 7f36c0217000 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557bdf1f74a0 0x557bdf1f7880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:09.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.908+0000 7f36c0217000 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x557bdf1f7dc0 0x557bdf1aab70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:09.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.908+0000 7f36c0217000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557bdf1ab0b0 0x557bdf1ab490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:09.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.908+0000 7f36c0217000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x557bdeffd950 con 0x557bdf1ab0b0 2024-03-31T15:10:09.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.908+0000 7f36c0217000 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x557bdef9a990 con 0x557bdf1f74a0 2024-03-31T15:10:09.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.908+0000 7f36c0217000 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x557bdeff9900 con 0x557bdf1f7dc0 2024-03-31T15:10:09.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.908+0000 7f36aea00640 1 --2- 172.21.15.16:0/2510492483 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f3694068380 0x7f369406a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:09.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.908+0000 7f36ad400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557bdf1ab0b0 0x557bdf1ab490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:09.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.908+0000 7f36aea00640 1 -- 172.21.15.16:0/2510492483 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 5795+0+0 (secure 0 0 0) 0x7f36a8085d00 con 0x7f36b806dbc0 2024-03-31T15:10:09.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.908+0000 7f36ad400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557bdf1ab0b0 0x557bdf1ab490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36518/0 (socket says 172.21.15.16:36518) 2024-03-31T15:10:09.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.908+0000 7f36ad400640 1 -- 172.21.15.16:0/3938532861 learned_addr learned my addr 172.21.15.16:0/3938532861 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:10:09.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.908+0000 7f36b6800640 1 --2- 172.21.15.16:0/3938532861 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557bdf1f74a0 0x557bdf1f7880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:09.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.908+0000 7f36b5e00640 1 --2- 172.21.15.16:0/3938532861 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x557bdf1f7dc0 0x557bdf1aab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:09.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.908+0000 7f36b6800640 1 --2- 172.21.15.16:0/2510492483 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f3694068380 0x7f369406a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:09.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.908+0000 7f36ad400640 1 -- 172.21.15.16:0/3938532861 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x557bdf1f7dc0 msgr2=0x557bdf1aab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:10:09.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.908+0000 7f36ad400640 1 --2- 172.21.15.16:0/3938532861 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x557bdf1f7dc0 0x557bdf1aab70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:09.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.908+0000 7f36ad400640 1 -- 172.21.15.16:0/3938532861 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557bdf1f74a0 msgr2=0x557bdf1f7880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:10:09.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.908+0000 7f36ad400640 1 --2- 172.21.15.16:0/3938532861 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557bdf1f74a0 0x557bdf1f7880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:09.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.908+0000 7f36ad400640 1 -- 172.21.15.16:0/3938532861 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557bdf042100 con 0x557bdf1ab0b0 2024-03-31T15:10:09.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.908+0000 7f36b5e00640 1 --2- 172.21.15.16:0/3938532861 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x557bdf1f7dc0 0x557bdf1aab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:10:09.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.908+0000 7f36ad400640 1 --2- 172.21.15.16:0/3938532861 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557bdf1ab0b0 0x557bdf1ab490 secure :-1 s=READY pgs=1160 cs=0 l=1 rev1=1 crypto rx=0x7f36a001c450 tx=0x7f36a0004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=196a1a67242ef424 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:10:09.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.912+0000 7f36b6800640 1 --2- 172.21.15.16:0/2510492483 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f3694068380 0x7f369406a840 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f36b00012f0 tx=0x7f36b0001610 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:10:09.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.912+0000 7f36a6a00640 1 -- 172.21.15.16:0/3938532861 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36a00b0040 con 0x557bdf1ab0b0 2024-03-31T15:10:09.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.912+0000 7f36a6a00640 1 -- 172.21.15.16:0/3938532861 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f36a00ab7c0 con 0x557bdf1ab0b0 2024-03-31T15:10:09.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.912+0000 7f36a6a00640 1 -- 172.21.15.16:0/3938532861 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36a00abb70 con 0x557bdf1ab0b0 2024-03-31T15:10:09.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.912+0000 7f36c0217000 1 -- 172.21.15.16:0/3938532861 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557bdf1ab0b0 msgr2=0x557bdf1ab490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:10:09.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.912+0000 7f36c0217000 1 --2- 172.21.15.16:0/3938532861 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557bdf1ab0b0 0x557bdf1ab490 secure :-1 s=READY pgs=1160 cs=0 l=1 rev1=1 crypto rx=0x7f36a001c450 tx=0x7f36a0004740 comp rx=0 tx=0).stop 2024-03-31T15:10:09.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.912+0000 7f36c0217000 1 -- 172.21.15.16:0/3938532861 shutdown_connections 2024-03-31T15:10:09.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.912+0000 7f36c0217000 1 --2- 172.21.15.16:0/3938532861 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557bdf1ab0b0 0x557bdf1ab490 unknown :-1 s=CLOSED pgs=1160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:09.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.912+0000 7f36c0217000 1 --2- 172.21.15.16:0/3938532861 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x557bdf1f7dc0 0x557bdf1aab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:09.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.912+0000 7f36c0217000 1 --2- 172.21.15.16:0/3938532861 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557bdf1f74a0 0x557bdf1f7880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:09.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.912+0000 7f36c0217000 1 -- 172.21.15.16:0/3938532861 >> 172.21.15.16:0/3938532861 conn(0x557bdf095b80 msgr2=0x557bdf1da460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:10:09.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.912+0000 7f36c0217000 1 -- 172.21.15.16:0/3938532861 shutdown_connections 2024-03-31T15:10:09.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.912+0000 7f36c0217000 1 -- 172.21.15.16:0/3938532861 wait complete. 2024-03-31T15:10:09.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.912+0000 7f36c0217000 1 -- start start 2024-03-31T15:10:09.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.912+0000 7f36c0217000 1 Processor -- start 2024-03-31T15:10:09.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.912+0000 7f36c0217000 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557bdf1ab0b0 0x557bdf1b7e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:09.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.912+0000 7f36c0217000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557bdf1f74a0 0x557bdf1b8340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:09.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.916+0000 7f36c0217000 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x557bdf1f7dc0 0x557bdf1b8880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:09.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.916+0000 7f36c0217000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x557bdeffcc60 con 0x557bdf1f74a0 2024-03-31T15:10:09.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.916+0000 7f36c0217000 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x557bdefbb3e0 con 0x557bdf1ab0b0 2024-03-31T15:10:09.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.916+0000 7f36c0217000 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x557bdf00a060 con 0x557bdf1f7dc0 2024-03-31T15:10:09.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.916+0000 7f36c0217000 10 client.0 _ll_register_callbacks cb 0x557bdf05e0f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T15:10:09.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.916+0000 7f36ad400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557bdf1f74a0 0x557bdf1b8340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:09.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.916+0000 7f36ad400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557bdf1f74a0 0x557bdf1b8340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36528/0 (socket says 172.21.15.16:36528) 2024-03-31T15:10:09.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.916+0000 7f36ad400640 1 -- 172.21.15.16:0/2983305188 learned_addr learned my addr 172.21.15.16:0/2983305188 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:10:09.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.916+0000 7f36b6800640 1 --2- 172.21.15.16:0/2983305188 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x557bdf1f7dc0 0x557bdf1b8880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:09.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.916+0000 7f36b5e00640 1 --2- 172.21.15.16:0/2983305188 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557bdf1ab0b0 0x557bdf1b7e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:09.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.916+0000 7f36ad400640 1 -- 172.21.15.16:0/2983305188 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x557bdf1f7dc0 msgr2=0x557bdf1b8880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:10:09.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.916+0000 7f36ad400640 1 --2- 172.21.15.16:0/2983305188 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x557bdf1f7dc0 0x557bdf1b8880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:09.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.916+0000 7f36ad400640 1 -- 172.21.15.16:0/2983305188 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557bdf1ab0b0 msgr2=0x557bdf1b7e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:10:09.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.916+0000 7f36ad400640 1 --2- 172.21.15.16:0/2983305188 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557bdf1ab0b0 0x557bdf1b7e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:09.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.916+0000 7f36ad400640 1 -- 172.21.15.16:0/2983305188 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557bdefc78c0 con 0x557bdf1f74a0 2024-03-31T15:10:09.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.916+0000 7f36ad400640 1 --2- 172.21.15.16:0/2983305188 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557bdf1f74a0 0x557bdf1b8340 secure :-1 s=READY pgs=1161 cs=0 l=1 rev1=1 crypto rx=0x7f36a001c450 tx=0x7f36a00046d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=127c9ebc9d40ac9a server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:10:09.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.916+0000 7f369b400640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-31T15:10:09.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.916+0000 7f369b400640 1 -- 172.21.15.16:0/2983305188 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36a00cc060 con 0x557bdf1f74a0 2024-03-31T15:10:09.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.916+0000 7f369b400640 1 -- 172.21.15.16:0/2983305188 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 5795+0+0 (secure 0 0 0) 0x7f36a00ab5c0 con 0x557bdf1f74a0 2024-03-31T15:10:09.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.916+0000 7f36c0217000 1 -- 172.21.15.16:0/2983305188 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f36a00ab7c0 con 0x557bdf1f74a0 2024-03-31T15:10:09.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.916+0000 7f369b400640 1 -- 172.21.15.16:0/2983305188 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=67) ==== 24+0+0 (secure 0 0 0) 0x7f36a00d2050 con 0x557bdf1f74a0 2024-03-31T15:10:09.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.916+0000 7f36c0217000 10 client.7815 fetch_fsmap learned FSMap version 67 2024-03-31T15:10:09.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.916+0000 7f36c0217000 1 -- 172.21.15.16:0/2983305188 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=67,monmap=2+}) -- 0x557bdef85e80 con 0x557bdf1f74a0 2024-03-31T15:10:09.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.916+0000 7f369b400640 1 -- 172.21.15.16:0/2983305188 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f36a00ab7c0 con 0x557bdf1f74a0 2024-03-31T15:10:09.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.916+0000 7f369b400640 1 -- 172.21.15.16:0/2983305188 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 67) ==== 3494+0+0 (secure 0 0 0) 0x7f36a00d5020 con 0x557bdf1f74a0 2024-03-31T15:10:09.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.916+0000 7f36c0217000 10 client.7815 fetch_fsmap finished waiting for FSMap version 67 2024-03-31T15:10:09.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.916+0000 7f36c0217000 10 client.7815 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-31T15:10:09.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.916+0000 7f36c0217000 20 client.7815 populate_metadata read hostname 'smithi016' 2024-03-31T15:10:09.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.916+0000 7f36c0217000 1 --2- 172.21.15.16:0/2983305188 >> [v2:172.21.15.16:6832/818074038,v1:172.21.15.16:6834/818074038] conn(0x557bdf1b9090 0x557bdf209a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:09.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.916+0000 7f36c0217000 4 client.7815 mds_command: new command op to 7749 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-31T15:10:09.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.916+0000 7f36c0217000 1 -- 172.21.15.16:0/2983305188 --> [v2:172.21.15.16:6832/818074038,v1:172.21.15.16:6834/818074038] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557bdef98bc0 con 0x557bdf1b9090 2024-03-31T15:10:09.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.916+0000 7f36b5e00640 1 --2- 172.21.15.16:0/2983305188 >> [v2:172.21.15.16:6832/818074038,v1:172.21.15.16:6834/818074038] conn(0x557bdf1b9090 0x557bdf209a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:09.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.916+0000 7f36b5e00640 1 --2- 172.21.15.16:0/2983305188 >> [v2:172.21.15.16:6832/818074038,v1:172.21.15.16:6834/818074038] conn(0x557bdf1b9090 0x557bdf209a20 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f3b5f9f2918523b8 server_cookie=d72619813014111d in_seq=0 out_seq=0 2024-03-31T15:10:09.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.916+0000 7f369b400640 10 client.7815 ms_handle_connect on v2:172.21.15.16:6832/818074038 2024-03-31T15:10:09.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.916+0000 7f369b400640 1 -- 172.21.15.16:0/2983305188 <== mds.0 v2:172.21.15.16:6832/818074038 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x557bdef98bc0 con 0x557bdf1b9090 2024-03-31T15:10:09.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.916+0000 7f369b400640 10 client.7815 handle_command_reply: tid=0 2024-03-31T15:10:09.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.916+0000 7f36c0217000 1 client.7815 shutdown 2024-03-31T15:10:09.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.916+0000 7f36c0217000 1 -- 172.21.15.16:0/2983305188 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557bdf1f74a0 msgr2=0x557bdf1b8340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:10:09.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.916+0000 7f36c0217000 1 --2- 172.21.15.16:0/2983305188 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557bdf1f74a0 0x557bdf1b8340 secure :-1 s=READY pgs=1161 cs=0 l=1 rev1=1 crypto rx=0x7f36a001c450 tx=0x7f36a00046d0 comp rx=0 tx=0).stop 2024-03-31T15:10:09.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.916+0000 7f36c0217000 1 -- 172.21.15.16:0/2983305188 shutdown_connections 2024-03-31T15:10:09.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.916+0000 7f36c0217000 1 --2- 172.21.15.16:0/2983305188 >> [v2:172.21.15.16:6832/818074038,v1:172.21.15.16:6834/818074038] conn(0x557bdf1b9090 0x557bdf209a20 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:09.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.916+0000 7f36c0217000 1 --2- 172.21.15.16:0/2983305188 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x557bdf1f7dc0 0x557bdf1b8880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:09.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.916+0000 7f36c0217000 1 --2- 172.21.15.16:0/2983305188 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557bdf1f74a0 0x557bdf1b8340 unknown :-1 s=CLOSED pgs=1161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:09.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.916+0000 7f369b400640 0 client.7815 ms_handle_reset on v2:172.21.15.16:6832/818074038 2024-03-31T15:10:09.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.916+0000 7f36c0217000 1 --2- 172.21.15.16:0/2983305188 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557bdf1ab0b0 0x557bdf1b7e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:09.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.916+0000 7f36c0217000 1 -- 172.21.15.16:0/2983305188 >> 172.21.15.16:0/2983305188 conn(0x557bdf095b80 msgr2=0x557bdf1aa270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:10:09.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.916+0000 7f36c0217000 1 -- 172.21.15.16:0/2983305188 shutdown_connections 2024-03-31T15:10:09.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.920+0000 7f36c0217000 1 -- 172.21.15.16:0/2983305188 wait complete. 2024-03-31T15:10:09.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.920+0000 7f36c0217000 20 client.7815 trim_cache size 0 max 16384 2024-03-31T15:10:09.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.952+0000 7f36c0217000 1 Processor -- start 2024-03-31T15:10:09.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.952+0000 7f36c0217000 1 -- start start 2024-03-31T15:10:09.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.952+0000 7f36c0217000 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557bdf1ab0b0 0x557bdf1aa2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:09.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.952+0000 7f36c0217000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557bdf1f74a0 0x557bdf1aa820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:09.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.952+0000 7f36c0217000 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x557bdf1f7dc0 0x557bdf1f5760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:09.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.952+0000 7f36c0217000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f36b8203b60 con 0x557bdf1f74a0 2024-03-31T15:10:09.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.952+0000 7f36c0217000 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f36b8202bc0 con 0x557bdf1ab0b0 2024-03-31T15:10:09.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.952+0000 7f36c0217000 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f36b8202550 con 0x557bdf1f7dc0 2024-03-31T15:10:09.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.952+0000 7f36ad400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557bdf1f74a0 0x557bdf1aa820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:09.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.952+0000 7f36ad400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557bdf1f74a0 0x557bdf1aa820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36542/0 (socket says 172.21.15.16:36542) 2024-03-31T15:10:09.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.952+0000 7f36ad400640 1 -- 172.21.15.16:0/1492864104 learned_addr learned my addr 172.21.15.16:0/1492864104 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:10:09.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.952+0000 7f36b6800640 1 --2- 172.21.15.16:0/1492864104 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x557bdf1f7dc0 0x557bdf1f5760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:09.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.952+0000 7f36b5e00640 1 --2- 172.21.15.16:0/1492864104 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557bdf1ab0b0 0x557bdf1aa2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:09.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.952+0000 7f36ad400640 1 -- 172.21.15.16:0/1492864104 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x557bdf1f7dc0 msgr2=0x557bdf1f5760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:10:09.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.952+0000 7f36ad400640 1 --2- 172.21.15.16:0/1492864104 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x557bdf1f7dc0 0x557bdf1f5760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:09.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.952+0000 7f36ad400640 1 -- 172.21.15.16:0/1492864104 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557bdf1ab0b0 msgr2=0x557bdf1aa2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:10:09.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.952+0000 7f36ad400640 1 --2- 172.21.15.16:0/1492864104 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557bdf1ab0b0 0x557bdf1aa2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:09.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.952+0000 7f36ad400640 1 -- 172.21.15.16:0/1492864104 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557bdefb8490 con 0x557bdf1f74a0 2024-03-31T15:10:09.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.952+0000 7f36ad400640 1 --2- 172.21.15.16:0/1492864104 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557bdf1f74a0 0x557bdf1aa820 secure :-1 s=READY pgs=1162 cs=0 l=1 rev1=1 crypto rx=0x7f36a001a040 tx=0x7f36a00b6000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=790335ce1f77fd91 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:10:09.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.952+0000 7f36a6000640 1 -- 172.21.15.16:0/1492864104 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36a00b0050 con 0x557bdf1f74a0 2024-03-31T15:10:09.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.952+0000 7f36a6000640 1 -- 172.21.15.16:0/1492864104 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f36a0000e50 con 0x557bdf1f74a0 2024-03-31T15:10:09.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.952+0000 7f36a6000640 1 -- 172.21.15.16:0/1492864104 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36a00ab400 con 0x557bdf1f74a0 2024-03-31T15:10:09.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.952+0000 7f36c0217000 1 -- 172.21.15.16:0/1492864104 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557bdf1f74a0 msgr2=0x557bdf1aa820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:10:09.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.952+0000 7f36c0217000 1 --2- 172.21.15.16:0/1492864104 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557bdf1f74a0 0x557bdf1aa820 secure :-1 s=READY pgs=1162 cs=0 l=1 rev1=1 crypto rx=0x7f36a001a040 tx=0x7f36a00b6000 comp rx=0 tx=0).stop 2024-03-31T15:10:09.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.952+0000 7f36c0217000 1 -- 172.21.15.16:0/1492864104 shutdown_connections 2024-03-31T15:10:09.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.952+0000 7f36c0217000 1 --2- 172.21.15.16:0/1492864104 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x557bdf1f7dc0 0x557bdf1f5760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:09.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.952+0000 7f36c0217000 1 --2- 172.21.15.16:0/1492864104 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557bdf1f74a0 0x557bdf1aa820 unknown :-1 s=CLOSED pgs=1162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:09.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.952+0000 7f36c0217000 1 --2- 172.21.15.16:0/1492864104 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557bdf1ab0b0 0x557bdf1aa2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:09.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.952+0000 7f36c0217000 1 -- 172.21.15.16:0/1492864104 >> 172.21.15.16:0/1492864104 conn(0x557bdf095b80 msgr2=0x557bdf204200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:10:09.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.952+0000 7f36c0217000 1 -- 172.21.15.16:0/1492864104 shutdown_connections 2024-03-31T15:10:09.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.952+0000 7f36c0217000 1 -- 172.21.15.16:0/1492864104 wait complete. 2024-03-31T15:10:09.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.952+0000 7f36c0217000 1 -- start start 2024-03-31T15:10:09.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.952+0000 7f36c0217000 1 Processor -- start 2024-03-31T15:10:09.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.956+0000 7f36c0217000 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x557bdf1ab0b0 0x557bdf1aa710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:09.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.956+0000 7f36c0217000 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557bdf1f74a0 0x557bdf267ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:09.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.956+0000 7f36c0217000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557bdf1f7dc0 0x557bdf2681e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:09.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.956+0000 7f36c0217000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x557bdefbf080 con 0x557bdf1f7dc0 2024-03-31T15:10:09.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.956+0000 7f36c0217000 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x557bdf001f80 con 0x557bdf1f74a0 2024-03-31T15:10:09.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.956+0000 7f36c0217000 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x557bdeffd950 con 0x557bdf1ab0b0 2024-03-31T15:10:09.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.956+0000 7f36c0217000 10 client.0 _ll_register_callbacks cb 0x7f36a00b9050 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T15:10:09.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.956+0000 7f36b6800640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557bdf1f7dc0 0x557bdf2681e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:09.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.956+0000 7f36b6800640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557bdf1f7dc0 0x557bdf2681e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36554/0 (socket says 172.21.15.16:36554) 2024-03-31T15:10:09.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.956+0000 7f36b6800640 1 -- 172.21.15.16:0/63358558 learned_addr learned my addr 172.21.15.16:0/63358558 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:10:09.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.956+0000 7f36b5e00640 1 --2- 172.21.15.16:0/63358558 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x557bdf1ab0b0 0x557bdf1aa710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:09.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.956+0000 7f36ad400640 1 --2- 172.21.15.16:0/63358558 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557bdf1f74a0 0x557bdf267ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:09.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.956+0000 7f36b6800640 1 -- 172.21.15.16:0/63358558 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x557bdf1ab0b0 msgr2=0x557bdf1aa710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:10:09.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.956+0000 7f36b6800640 1 --2- 172.21.15.16:0/63358558 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x557bdf1ab0b0 0x557bdf1aa710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:09.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.956+0000 7f36b6800640 1 -- 172.21.15.16:0/63358558 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557bdf1f74a0 msgr2=0x557bdf267ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:10:09.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.956+0000 7f36b6800640 1 --2- 172.21.15.16:0/63358558 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557bdf1f74a0 0x557bdf267ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:09.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.956+0000 7f36b6800640 1 -- 172.21.15.16:0/63358558 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557bdefb6e80 con 0x557bdf1f7dc0 2024-03-31T15:10:09.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.956+0000 7f36b6800640 1 --2- 172.21.15.16:0/63358558 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557bdf1f7dc0 0x557bdf2681e0 secure :-1 s=READY pgs=1163 cs=0 l=1 rev1=1 crypto rx=0x7f36b00052d0 tx=0x7f36b0001410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d854d879f350f30 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:10:09.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.956+0000 7f369b400640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-31T15:10:09.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.956+0000 7f369b400640 1 -- 172.21.15.16:0/63358558 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36b002b060 con 0x557bdf1f7dc0 2024-03-31T15:10:09.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.956+0000 7f369b400640 1 -- 172.21.15.16:0/63358558 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 5795+0+0 (secure 0 0 0) 0x7f36b001aa80 con 0x557bdf1f7dc0 2024-03-31T15:10:09.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.956+0000 7f36c0217000 1 -- 172.21.15.16:0/63358558 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f36a0000e50 con 0x557bdf1f7dc0 2024-03-31T15:10:09.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.956+0000 7f369b400640 1 -- 172.21.15.16:0/63358558 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=67) ==== 24+0+0 (secure 0 0 0) 0x7f36b0031050 con 0x557bdf1f7dc0 2024-03-31T15:10:09.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.956+0000 7f36c0217000 10 client.7821 fetch_fsmap learned FSMap version 67 2024-03-31T15:10:09.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.956+0000 7f36c0217000 1 -- 172.21.15.16:0/63358558 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=67,monmap=2+}) -- 0x557bdefa4270 con 0x557bdf1f7dc0 2024-03-31T15:10:09.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.956+0000 7f369b400640 1 -- 172.21.15.16:0/63358558 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f36a0000e50 con 0x557bdf1f7dc0 2024-03-31T15:10:09.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.956+0000 7f369b400640 1 -- 172.21.15.16:0/63358558 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 67) ==== 3494+0+0 (secure 0 0 0) 0x7f36b0034020 con 0x557bdf1f7dc0 2024-03-31T15:10:09.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.956+0000 7f36c0217000 10 client.7821 fetch_fsmap finished waiting for FSMap version 67 2024-03-31T15:10:09.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.956+0000 7f36c0217000 10 client.7821 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-31T15:10:09.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.956+0000 7f36c0217000 20 client.7821 populate_metadata read hostname 'smithi016' 2024-03-31T15:10:09.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.956+0000 7f36c0217000 1 --2- 172.21.15.16:0/63358558 >> [v2:172.21.15.16:6832/818074038,v1:172.21.15.16:6834/818074038] conn(0x557bdf268790 0x557bdf20b780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:09.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.956+0000 7f36c0217000 4 client.7821 mds_command: new command op to 7749 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-31T15:10:09.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.956+0000 7f36c0217000 1 -- 172.21.15.16:0/63358558 --> [v2:172.21.15.16:6832/818074038,v1:172.21.15.16:6834/818074038] -- command(tid 0: {"prefix": "status"}) -- 0x557bdf00ae90 con 0x557bdf268790 2024-03-31T15:10:09.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.956+0000 7f36b5e00640 1 --2- 172.21.15.16:0/63358558 >> [v2:172.21.15.16:6832/818074038,v1:172.21.15.16:6834/818074038] conn(0x557bdf268790 0x557bdf20b780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:09.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.956+0000 7f369b400640 10 client.7821 ms_handle_connect on v2:172.21.15.16:6832/818074038 2024-03-31T15:10:09.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.956+0000 7f36b5e00640 1 --2- 172.21.15.16:0/63358558 >> [v2:172.21.15.16:6832/818074038,v1:172.21.15.16:6834/818074038] conn(0x557bdf268790 0x557bdf20b780 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=44a21699486c2039 server_cookie=a28cf28d1cde747 in_seq=0 out_seq=0 2024-03-31T15:10:09.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.956+0000 7f369b400640 1 -- 172.21.15.16:0/63358558 <== mds.0 v2:172.21.15.16:6832/818074038 1 ==== command_reply(tid 0: 0 ) ==== 8+0+331 (crc 0 0 0) 0x557bdf00ae90 con 0x557bdf268790 2024-03-31T15:10:09.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.956+0000 7f369b400640 10 client.7821 handle_command_reply: tid=0 2024-03-31T15:10:09.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.956+0000 7f36c0217000 1 client.7821 shutdown 2024-03-31T15:10:09.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.956+0000 7f36c0217000 1 -- 172.21.15.16:0/63358558 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557bdf1f7dc0 msgr2=0x557bdf2681e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:10:09.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.956+0000 7f36c0217000 1 --2- 172.21.15.16:0/63358558 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557bdf1f7dc0 0x557bdf2681e0 secure :-1 s=READY pgs=1163 cs=0 l=1 rev1=1 crypto rx=0x7f36b00052d0 tx=0x7f36b0001410 comp rx=0 tx=0).stop 2024-03-31T15:10:09.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.960+0000 7f36c0217000 1 -- 172.21.15.16:0/63358558 shutdown_connections 2024-03-31T15:10:09.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.960+0000 7f36c0217000 1 --2- 172.21.15.16:0/63358558 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557bdf1f7dc0 0x557bdf2681e0 unknown :-1 s=CLOSED pgs=1163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:09.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.960+0000 7f36c0217000 1 --2- 172.21.15.16:0/63358558 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557bdf1f74a0 0x557bdf267ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:09.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.960+0000 7f36c0217000 1 --2- 172.21.15.16:0/63358558 >> [v2:172.21.15.16:6832/818074038,v1:172.21.15.16:6834/818074038] conn(0x557bdf268790 0x557bdf20b780 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:09.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.960+0000 7f36c0217000 1 --2- 172.21.15.16:0/63358558 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x557bdf1ab0b0 0x557bdf1aa710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:09.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.960+0000 7f36c0217000 1 -- 172.21.15.16:0/63358558 >> 172.21.15.16:0/63358558 conn(0x557bdf095b80 msgr2=0x557bdf1d9c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:10:09.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.960+0000 7f369b400640 0 client.7821 ms_handle_reset on v2:172.21.15.16:6832/818074038 2024-03-31T15:10:09.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.960+0000 7f36c0217000 1 -- 172.21.15.16:0/63358558 shutdown_connections 2024-03-31T15:10:09.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.960+0000 7f36c0217000 1 -- 172.21.15.16:0/63358558 wait complete. 2024-03-31T15:10:09.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.960+0000 7f36c0217000 20 client.7821 trim_cache size 0 max 16384 2024-03-31T15:10:09.955 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-03-31T15:10:09.955 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_fsid": "286eb5da-fb47-4550-b290-d9519c13b9a5", 2024-03-31T15:10:09.955 INFO:teuthology.orchestra.run.smithi016.stdout: "whoami": 0, 2024-03-31T15:10:09.955 INFO:teuthology.orchestra.run.smithi016.stdout: "id": 7749, 2024-03-31T15:10:09.955 INFO:teuthology.orchestra.run.smithi016.stdout: "want_state": "up:active", 2024-03-31T15:10:09.955 INFO:teuthology.orchestra.run.smithi016.stdout: "state": "up:active", 2024-03-31T15:10:09.955 INFO:teuthology.orchestra.run.smithi016.stdout: "fs_name": "cephfs", 2024-03-31T15:10:09.956 INFO:teuthology.orchestra.run.smithi016.stdout: "rank_uptime": 4.5876582429999999, 2024-03-31T15:10:09.956 INFO:teuthology.orchestra.run.smithi016.stdout: "mdsmap_epoch": 67, 2024-03-31T15:10:09.956 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch": 82, 2024-03-31T15:10:09.956 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch_barrier": 82, 2024-03-31T15:10:09.956 INFO:teuthology.orchestra.run.smithi016.stdout: "uptime": 5.5349264219999998 2024-03-31T15:10:09.956 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-03-31T15:10:09.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.960+0000 7f36bce00640 1 -- 172.21.15.16:0/2510492483 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f3694068380 msgr2=0x7f369406a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:10:09.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.960+0000 7f36bce00640 1 --2- 172.21.15.16:0/2510492483 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f3694068380 0x7f369406a840 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f36b00012f0 tx=0x7f36b0001610 comp rx=0 tx=0).stop 2024-03-31T15:10:09.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.960+0000 7f36bce00640 1 -- 172.21.15.16:0/2510492483 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f36b806dbc0 msgr2=0x7f36b8156e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:10:09.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.960+0000 7f36bce00640 1 --2- 172.21.15.16:0/2510492483 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f36b806dbc0 0x7f36b8156e80 secure :-1 s=READY pgs=1159 cs=0 l=1 rev1=1 crypto rx=0x7f36a800c450 tx=0x7f36a800c910 comp rx=0 tx=0).stop 2024-03-31T15:10:09.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.960+0000 7f36bce00640 1 -- 172.21.15.16:0/2510492483 shutdown_connections 2024-03-31T15:10:09.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.960+0000 7f36bce00640 1 --2- 172.21.15.16:0/2510492483 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f3694068380 0x7f369406a840 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:09.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.960+0000 7f36bce00640 1 --2- 172.21.15.16:0/2510492483 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f36b815f650 0x7f36b8150520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:09.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.960+0000 7f36bce00640 1 --2- 172.21.15.16:0/2510492483 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f36b806fe60 0x7f36b814bba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:09.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.960+0000 7f36bce00640 1 --2- 172.21.15.16:0/2510492483 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f36b806dbc0 0x7f36b8156e80 unknown :-1 s=CLOSED pgs=1159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:09.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.960+0000 7f36bce00640 1 -- 172.21.15.16:0/2510492483 >> 172.21.15.16:0/2510492483 conn(0x7f36b8067330 msgr2=0x7f36b815c840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:10:09.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.960+0000 7f36bce00640 1 -- 172.21.15.16:0/2510492483 shutdown_connections 2024-03-31T15:10:09.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:09.960+0000 7f36bce00640 1 -- 172.21.15.16:0/2510492483 wait complete. 2024-03-31T15:10:09.971 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-31T15:10:10.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:10.120+0000 7fefab200640 1 Processor -- start 2024-03-31T15:10:10.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:10.120+0000 7fefab200640 1 -- start start 2024-03-31T15:10:10.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:10.120+0000 7fefab200640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fefac158700 0x7fefac0706d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:10.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:10.120+0000 7fefab200640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fefac06e4c0 0x7fefac15f120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:10.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:10.120+0000 7fefab200640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fefac06e970 con 0x7fefac06b620 2024-03-31T15:10:10.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:10.120+0000 7fefab200640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7fefac15ec90 con 0x7fefac158700 2024-03-31T15:10:10.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:10.120+0000 7fefab200640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7fefac06ec30 con 0x7fefac06e4c0 2024-03-31T15:10:10.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:10.120+0000 7fefa9e00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fefac158700 0x7fefac0706d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:10.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:10.120+0000 7fefaa800640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fefac06b620 0x7fefac06ba30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:40624/0 (socket says 172.21.15.16:40624) 2024-03-31T15:10:10.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:10.120+0000 7fefa9e00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fefac158700 0x7fefac0706d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36560/0 (socket says 172.21.15.16:36560) 2024-03-31T15:10:10.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:10.120+0000 7fefaa800640 1 -- 172.21.15.16:0/3378410960 learned_addr learned my addr 172.21.15.16:0/3378410960 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:10:10.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:10.120+0000 7fefa9400640 1 --2- 172.21.15.16:0/3378410960 >> v2:172.21.15.89:3300/0 conn(0x7fefac06e4c0 0x7fefac15f120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:10.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:10.120+0000 7fefa9e00640 1 -- 172.21.15.16:0/3378410960 >> v2:172.21.15.89:3300/0 conn(0x7fefac06e4c0 msgr2=0x7fefac15f120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:10:10.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:10.120+0000 7fefa9e00640 1 --2- 172.21.15.16:0/3378410960 >> v2:172.21.15.89:3300/0 conn(0x7fefac06e4c0 0x7fefac15f120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:10.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:10.120+0000 7fefa9e00640 1 -- 172.21.15.16:0/3378410960 >> v1:172.21.15.16:6789/0 conn(0x7fefac06b620 legacy=0x7fefac06ba30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:10:10.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:10.120+0000 7fefa9e00640 1 -- 172.21.15.16:0/3378410960 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fefac06ef10 con 0x7fefac158700 2024-03-31T15:10:10.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:10.120+0000 7fefa9400640 1 --2- 172.21.15.16:0/3378410960 >> v2:172.21.15.89:3300/0 conn(0x7fefac06e4c0 0x7fefac15f120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:10:10.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:10.124+0000 7fefa9e00640 1 --2- 172.21.15.16:0/3378410960 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fefac158700 0x7fefac0706d0 secure :-1 s=READY pgs=1165 cs=0 l=1 rev1=1 crypto rx=0x7fefac09f140 tx=0x7fef9c01c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea2a145c3898b8ec server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:10:10.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:10.124+0000 7fefa8a00640 1 -- 172.21.15.16:0/3378410960 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef9c0b0070 con 0x7fefac158700 2024-03-31T15:10:10.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:10.124+0000 7fefa8a00640 1 -- 172.21.15.16:0/3378410960 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fef9c0ab600 con 0x7fefac158700 2024-03-31T15:10:10.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:10.124+0000 7fefa8a00640 1 -- 172.21.15.16:0/3378410960 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef9c0ab9d0 con 0x7fefac158700 2024-03-31T15:10:10.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:10.124+0000 7fefab200640 1 -- 172.21.15.16:0/3378410960 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fefac158700 msgr2=0x7fefac0706d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:10:10.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:10.124+0000 7fefab200640 1 --2- 172.21.15.16:0/3378410960 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fefac158700 0x7fefac0706d0 secure :-1 s=READY pgs=1165 cs=0 l=1 rev1=1 crypto rx=0x7fefac09f140 tx=0x7fef9c01c480 comp rx=0 tx=0).stop 2024-03-31T15:10:10.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:10.124+0000 7fefab200640 1 -- 172.21.15.16:0/3378410960 shutdown_connections 2024-03-31T15:10:10.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:10.124+0000 7fefab200640 1 --2- 172.21.15.16:0/3378410960 >> v2:172.21.15.89:3300/0 conn(0x7fefac06e4c0 0x7fefac15f120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:10.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:10.124+0000 7fefab200640 1 --2- 172.21.15.16:0/3378410960 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fefac158700 0x7fefac0706d0 unknown :-1 s=CLOSED pgs=1165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:10.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:10.124+0000 7fefab200640 1 -- 172.21.15.16:0/3378410960 >> 172.21.15.16:0/3378410960 conn(0x7fefac0b2b30 msgr2=0x7fefac0b2f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:10:10.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:10.124+0000 7fefab200640 1 -- 172.21.15.16:0/3378410960 shutdown_connections 2024-03-31T15:10:10.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:10.124+0000 7fefab200640 1 -- 172.21.15.16:0/3378410960 wait complete. 2024-03-31T15:10:10.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:10.124+0000 7fefab200640 1 Processor -- start 2024-03-31T15:10:10.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:10.124+0000 7fefab200640 1 -- start start 2024-03-31T15:10:10.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:10.128+0000 7fefab200640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fefac06b620 0x7fefac14bca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:10.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:10.128+0000 7fefab200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fefac06e4c0 0x7fefac14d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:10.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:10.128+0000 7fefab200640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fefac158700 0x7fefac160bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:10.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:10.128+0000 7fefab200640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fefac0d0b00 con 0x7fefac06e4c0 2024-03-31T15:10:10.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:10.128+0000 7fefab200640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fefac0d0980 con 0x7fefac06b620 2024-03-31T15:10:10.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:10.128+0000 7fefab200640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fefac0d0c80 con 0x7fefac158700 2024-03-31T15:10:10.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:10.128+0000 7fefa9400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fefac06e4c0 0x7fefac14d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:10.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:10.128+0000 7fefa9400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fefac06e4c0 0x7fefac14d4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36572/0 (socket says 172.21.15.16:36572) 2024-03-31T15:10:10.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:10.128+0000 7fefa9400640 1 -- 172.21.15.16:0/1296976357 learned_addr learned my addr 172.21.15.16:0/1296976357 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:10:10.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:10.128+0000 7fefa9e00640 1 --2- 172.21.15.16:0/1296976357 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fefac06b620 0x7fefac14bca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:10.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:10.128+0000 7fefa9400640 1 -- 172.21.15.16:0/1296976357 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fefac158700 msgr2=0x7fefac160bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:10:10.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:10.128+0000 7fefa9400640 1 --2- 172.21.15.16:0/1296976357 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fefac158700 0x7fefac160bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:10.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:10.128+0000 7fefa9400640 1 -- 172.21.15.16:0/1296976357 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fefac06b620 msgr2=0x7fefac14bca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:10:10.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:10.128+0000 7fefa9400640 1 --2- 172.21.15.16:0/1296976357 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fefac06b620 0x7fefac14bca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:10.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:10.128+0000 7fefa9400640 1 -- 172.21.15.16:0/1296976357 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fefac161a70 con 0x7fefac06e4c0 2024-03-31T15:10:10.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:10.128+0000 7fefa9400640 1 --2- 172.21.15.16:0/1296976357 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fefac06e4c0 0x7fefac14d4a0 secure :-1 s=READY pgs=1166 cs=0 l=1 rev1=1 crypto rx=0x7fef9800d830 tx=0x7fef9800dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:10:10.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:10.128+0000 7fefa2000640 1 -- 172.21.15.16:0/1296976357 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef9800b6c0 con 0x7fefac06e4c0 2024-03-31T15:10:10.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:10.128+0000 7fefa2000640 1 -- 172.21.15.16:0/1296976357 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fef98019070 con 0x7fefac06e4c0 2024-03-31T15:10:10.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:10.128+0000 7fefab200640 1 -- 172.21.15.16:0/1296976357 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fefac161d50 con 0x7fefac06e4c0 2024-03-31T15:10:10.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:10.128+0000 7fefa2000640 1 -- 172.21.15.16:0/1296976357 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef9800baa0 con 0x7fefac06e4c0 2024-03-31T15:10:10.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:10.128+0000 7fefab200640 1 -- 172.21.15.16:0/1296976357 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fefac14d9e0 con 0x7fefac06e4c0 2024-03-31T15:10:10.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:10.128+0000 7fefab200640 1 -- 172.21.15.16:0/1296976357 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fefac04f8c0 con 0x7fefac06e4c0 2024-03-31T15:10:10.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:10.132+0000 7fefa2000640 1 -- 172.21.15.16:0/1296976357 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fef9801e460 con 0x7fefac06e4c0 2024-03-31T15:10:10.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:10.132+0000 7fefa2000640 1 --2- 172.21.15.16:0/1296976357 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fef8c068380 0x7fef8c06a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:10.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:10.132+0000 7fefa2000640 1 -- 172.21.15.16:0/1296976357 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 5795+0+0 (secure 0 0 0) 0x7fef98086ff0 con 0x7fefac06e4c0 2024-03-31T15:10:10.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:10.132+0000 7fefa9e00640 1 --2- 172.21.15.16:0/1296976357 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fef8c068380 0x7fef8c06a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:10.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:10.132+0000 7fefa9e00640 1 --2- 172.21.15.16:0/1296976357 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fef8c068380 0x7fef8c06a840 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7fef9c0ad000 tx=0x7fef9c008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:10:10.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:10.136+0000 7fefa2000640 1 -- 172.21.15.16:0/1296976357 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fef98057580 con 0x7fefac06e4c0 2024-03-31T15:10:10.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:10.336+0000 7fefab200640 1 -- 172.21.15.16:0/1296976357 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7fefac06c460 con 0x7fefac06e4c0 2024-03-31T15:10:10.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:10.336+0000 7fefa2000640 1 -- 172.21.15.16:0/1296976357 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v67) ==== 53+0+73 (secure 0 0 0) 0x7fef98056e80 con 0x7fefac06e4c0 2024-03-31T15:10:10.330 INFO:teuthology.orchestra.run.smithi016.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-31T15:10:10.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:10.340+0000 7fefab200640 1 -- 172.21.15.16:0/1296976357 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fef8c068380 msgr2=0x7fef8c06a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:10:10.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:10.340+0000 7fefab200640 1 --2- 172.21.15.16:0/1296976357 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fef8c068380 0x7fef8c06a840 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7fef9c0ad000 tx=0x7fef9c008040 comp rx=0 tx=0).stop 2024-03-31T15:10:10.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:10.340+0000 7fefab200640 1 -- 172.21.15.16:0/1296976357 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fefac06e4c0 msgr2=0x7fefac14d4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:10:10.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:10.340+0000 7fefab200640 1 --2- 172.21.15.16:0/1296976357 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fefac06e4c0 0x7fefac14d4a0 secure :-1 s=READY pgs=1166 cs=0 l=1 rev1=1 crypto rx=0x7fef9800d830 tx=0x7fef9800dcf0 comp rx=0 tx=0).stop 2024-03-31T15:10:10.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:10.340+0000 7fefab200640 1 -- 172.21.15.16:0/1296976357 shutdown_connections 2024-03-31T15:10:10.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:10.340+0000 7fefab200640 1 --2- 172.21.15.16:0/1296976357 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fef8c068380 0x7fef8c06a840 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:10.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:10.340+0000 7fefab200640 1 --2- 172.21.15.16:0/1296976357 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fefac158700 0x7fefac160bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:10.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:10.340+0000 7fefab200640 1 --2- 172.21.15.16:0/1296976357 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fefac06e4c0 0x7fefac14d4a0 unknown :-1 s=CLOSED pgs=1166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:10.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:10.340+0000 7fefab200640 1 --2- 172.21.15.16:0/1296976357 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fefac06b620 0x7fefac14bca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:10.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:10.340+0000 7fefab200640 1 -- 172.21.15.16:0/1296976357 >> 172.21.15.16:0/1296976357 conn(0x7fefac0b2b30 msgr2=0x7fefac0aef10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:10:10.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:10.340+0000 7fefab200640 1 -- 172.21.15.16:0/1296976357 shutdown_connections 2024-03-31T15:10:10.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:10.340+0000 7fefab200640 1 -- 172.21.15.16:0/1296976357 wait complete. 2024-03-31T15:10:10.352 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-31T15:10:10.352 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-31T15:10:10.352 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi016.front.sepia.ceph.com 2024-03-31T15:10:10.352 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-31T15:10:10.352 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-31T15:10:10.352 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-31T15:10:10.352 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-31T15:10:10.352 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-31T15:10:10.352 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-31T15:10:10.352 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:10:10.352 DEBUG:teuthology.orchestra.run.smithi016:> ip addr 2024-03-31T15:10:10.362 INFO:teuthology.orchestra.run.smithi016.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-31T15:10:10.362 INFO:teuthology.orchestra.run.smithi016.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-31T15:10:10.362 INFO:teuthology.orchestra.run.smithi016.stdout: inet 127.0.0.1/8 scope host lo 2024-03-31T15:10:10.362 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2024-03-31T15:10:10.362 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 ::1/128 scope host 2024-03-31T15:10:10.362 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2024-03-31T15:10:10.362 INFO:teuthology.orchestra.run.smithi016.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-03-31T15:10:10.362 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:6b:fd:e0 brd ff:ff:ff:ff:ff:ff 2024-03-31T15:10:10.362 INFO:teuthology.orchestra.run.smithi016.stdout: altname enp7s0f0 2024-03-31T15:10:10.362 INFO:teuthology.orchestra.run.smithi016.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-03-31T15:10:10.363 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:6b:fd:e1 brd ff:ff:ff:ff:ff:ff 2024-03-31T15:10:10.363 INFO:teuthology.orchestra.run.smithi016.stdout: altname enp7s0f1 2024-03-31T15:10:10.363 INFO:teuthology.orchestra.run.smithi016.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-31T15:10:10.363 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:bd:15:64 brd ff:ff:ff:ff:ff:ff 2024-03-31T15:10:10.363 INFO:teuthology.orchestra.run.smithi016.stdout: altname enp1s0f0 2024-03-31T15:10:10.363 INFO:teuthology.orchestra.run.smithi016.stdout: inet 172.21.15.16/20 brd 172.21.15.255 scope global dynamic ens1f0 2024-03-31T15:10:10.363 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft 42070sec preferred_lft 42070sec 2024-03-31T15:10:10.363 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 fe80::ec4:7aff:febd:1564/64 scope link 2024-03-31T15:10:10.363 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2024-03-31T15:10:10.363 INFO:teuthology.orchestra.run.smithi016.stdout:5: ens1f1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-03-31T15:10:10.363 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:bd:15:65 brd ff:ff:ff:ff:ff:ff 2024-03-31T15:10:10.363 INFO:teuthology.orchestra.run.smithi016.stdout: altname enp1s0f1 2024-03-31T15:10:10.363 INFO:teuthology.orchestra.run.smithi016.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2024-03-31T15:10:10.363 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 02:42:d4:1f:7d:73 brd ff:ff:ff:ff:ff:ff 2024-03-31T15:10:10.363 INFO:teuthology.orchestra.run.smithi016.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2024-03-31T15:10:10.364 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2024-03-31T15:10:10.364 INFO:teuthology.orchestra.run.smithi016.stdout:7: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-31T15:10:10.364 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 5e:1f:84:6d:79:ae brd ff:ff:ff:ff:ff:ff 2024-03-31T15:10:10.364 INFO:teuthology.orchestra.run.smithi016.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-31T15:10:10.364 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2024-03-31T15:10:10.364 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 fe80::5c1f:84ff:fe6d:79ae/64 scope link 2024-03-31T15:10:10.364 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2024-03-31T15:10:10.364 INFO:teuthology.orchestra.run.smithi016.stdout:8: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-31T15:10:10.364 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0e:5c:f6:1f:82:2f brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-31T15:10:10.364 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 fe80::c5c:f6ff:fe1f:822f/64 scope link 2024-03-31T15:10:10.364 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2024-03-31T15:10:10.364 INFO:teuthology.orchestra.run.smithi016.stdout:9: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-31T15:10:10.364 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether c6:a6:3d:69:d0:db brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-31T15:10:10.364 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 fe80::c4a6:3dff:fe69:d0db/64 scope link 2024-03-31T15:10:10.365 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2024-03-31T15:10:10.365 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:10:10.365 DEBUG:teuthology.orchestra.run.smithi016:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-31T15:10:10.424 INFO:teuthology.orchestra.run.smithi016.stdout:1 2024-03-31T15:10:10.426 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:10:10.427 DEBUG:teuthology.orchestra.run.smithi016:> route 2024-03-31T15:10:10.483 INFO:teuthology.orchestra.run.smithi016.stdout:Kernel IP routing table 2024-03-31T15:10:10.483 INFO:teuthology.orchestra.run.smithi016.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-31T15:10:10.484 INFO:teuthology.orchestra.run.smithi016.stdout:default _gateway 0.0.0.0 UG 0 0 0 ens1f0 2024-03-31T15:10:10.484 INFO:teuthology.orchestra.run.smithi016.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2024-03-31T15:10:10.484 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 0 0 0 ens1f0 2024-03-31T15:10:10.484 INFO:teuthology.orchestra.run.smithi016.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-31T15:10:10.484 DEBUG:teuthology.orchestra.run.smithi016:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-31T15:10:10.484 DEBUG:teuthology.orchestra.run.smithi016:> set -e 2024-03-31T15:10:10.484 DEBUG:teuthology.orchestra.run.smithi016:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-03-31T15:10:10.484 DEBUG:teuthology.orchestra.run.smithi016:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-03-31T15:10:10.484 DEBUG:teuthology.orchestra.run.smithi016:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-03-31T15:10:10.484 DEBUG:teuthology.orchestra.run.smithi016:> ') 2024-03-31T15:10:10.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:10 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-31T15:10:10.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:10 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-31T15:10:10.627 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:10:10.627 DEBUG:teuthology.orchestra.run.smithi016:> ip netns list 2024-03-31T15:10:10.679 INFO:teuthology.orchestra.run.smithi016.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-31T15:10:10.679 INFO:teuthology.orchestra.run.smithi016.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-31T15:10:10.679 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-31T15:10:10.679 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:10:10.679 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip link set brx.0 up 2024-03-31T15:10:10.741 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-03-31T15:10:10.742 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-31T15:10:10.742 DEBUG:teuthology.orchestra.run.smithi016:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-31T15:10:10.790 INFO:teuthology.orchestra.run.smithi016.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-31T15:10:10.790 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-31T15:10:10.790 DEBUG:teuthology.orchestra.run.smithi016:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-31T15:10:10.841 DEBUG:teuthology.orchestra.run.smithi016:> sudo modprobe fuse 2024-03-31T15:10:10.904 DEBUG:teuthology.orchestra.run.smithi016:> cat /proc/self/mounts | awk '{print $2}' 2024-03-31T15:10:10.960 INFO:teuthology.orchestra.run.smithi016.stdout:/sys 2024-03-31T15:10:10.961 INFO:teuthology.orchestra.run.smithi016.stdout:/proc 2024-03-31T15:10:10.961 INFO:teuthology.orchestra.run.smithi016.stdout:/dev 2024-03-31T15:10:10.961 INFO:teuthology.orchestra.run.smithi016.stdout:/dev/pts 2024-03-31T15:10:10.961 INFO:teuthology.orchestra.run.smithi016.stdout:/run 2024-03-31T15:10:10.961 INFO:teuthology.orchestra.run.smithi016.stdout:/ 2024-03-31T15:10:10.961 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/kernel/security 2024-03-31T15:10:10.961 INFO:teuthology.orchestra.run.smithi016.stdout:/dev/shm 2024-03-31T15:10:10.961 INFO:teuthology.orchestra.run.smithi016.stdout:/run/lock 2024-03-31T15:10:10.961 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/cgroup 2024-03-31T15:10:10.961 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/pstore 2024-03-31T15:10:10.961 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/bpf 2024-03-31T15:10:10.962 INFO:teuthology.orchestra.run.smithi016.stdout:/proc/sys/fs/binfmt_misc 2024-03-31T15:10:10.962 INFO:teuthology.orchestra.run.smithi016.stdout:/dev/hugepages 2024-03-31T15:10:10.962 INFO:teuthology.orchestra.run.smithi016.stdout:/dev/mqueue 2024-03-31T15:10:10.962 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/kernel/debug 2024-03-31T15:10:10.962 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/kernel/tracing 2024-03-31T15:10:10.962 INFO:teuthology.orchestra.run.smithi016.stdout:/proc/fs/nfsd 2024-03-31T15:10:10.962 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/fuse/connections 2024-03-31T15:10:10.962 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/kernel/config 2024-03-31T15:10:10.962 INFO:teuthology.orchestra.run.smithi016.stdout:/run/credentials/systemd-sysusers.service 2024-03-31T15:10:10.962 INFO:teuthology.orchestra.run.smithi016.stdout:/run/rpc_pipefs 2024-03-31T15:10:10.962 INFO:teuthology.orchestra.run.smithi016.stdout:/run/user/1000 2024-03-31T15:10:10.962 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/ceph 2024-03-31T15:10:10.962 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/ceph/osd/ceph-0 2024-03-31T15:10:10.963 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/ceph/osd/ceph-1 2024-03-31T15:10:10.963 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/ceph/osd/ceph-2 2024-03-31T15:10:10.963 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/ceph/osd/ceph-3 2024-03-31T15:10:10.963 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns 2024-03-31T15:10:10.963 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-31T15:10:10.963 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-31T15:10:10.963 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-31T15:10:10.963 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-31T15:10:10.964 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:10:10.964 DEBUG:teuthology.orchestra.run.smithi016:> ls /sys/fs/fuse/connections 2024-03-31T15:10:11.015 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-31T15:10:11.015 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-03-31T15:10:11.066 DEBUG:teuthology.orchestra.run.smithi016:> sudo modprobe fuse 2024-03-31T15:10:11.088 DEBUG:teuthology.orchestra.run.smithi016:> cat /proc/self/mounts | awk '{print $2}' 2024-03-31T15:10:11.141 INFO:teuthology.orchestra.run.smithi016.stdout:/sys 2024-03-31T15:10:11.141 INFO:teuthology.orchestra.run.smithi016.stdout:/proc 2024-03-31T15:10:11.141 INFO:teuthology.orchestra.run.smithi016.stdout:/dev 2024-03-31T15:10:11.141 INFO:teuthology.orchestra.run.smithi016.stdout:/dev/pts 2024-03-31T15:10:11.141 INFO:teuthology.orchestra.run.smithi016.stdout:/run 2024-03-31T15:10:11.141 INFO:teuthology.orchestra.run.smithi016.stdout:/ 2024-03-31T15:10:11.141 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/kernel/security 2024-03-31T15:10:11.141 INFO:teuthology.orchestra.run.smithi016.stdout:/dev/shm 2024-03-31T15:10:11.141 INFO:teuthology.orchestra.run.smithi016.stdout:/run/lock 2024-03-31T15:10:11.141 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/cgroup 2024-03-31T15:10:11.142 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/pstore 2024-03-31T15:10:11.142 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/bpf 2024-03-31T15:10:11.142 INFO:teuthology.orchestra.run.smithi016.stdout:/proc/sys/fs/binfmt_misc 2024-03-31T15:10:11.142 INFO:teuthology.orchestra.run.smithi016.stdout:/dev/hugepages 2024-03-31T15:10:11.142 INFO:teuthology.orchestra.run.smithi016.stdout:/dev/mqueue 2024-03-31T15:10:11.142 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/kernel/debug 2024-03-31T15:10:11.142 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/kernel/tracing 2024-03-31T15:10:11.142 INFO:teuthology.orchestra.run.smithi016.stdout:/proc/fs/nfsd 2024-03-31T15:10:11.142 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/fuse/connections 2024-03-31T15:10:11.142 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/kernel/config 2024-03-31T15:10:11.142 INFO:teuthology.orchestra.run.smithi016.stdout:/run/credentials/systemd-sysusers.service 2024-03-31T15:10:11.142 INFO:teuthology.orchestra.run.smithi016.stdout:/run/rpc_pipefs 2024-03-31T15:10:11.142 INFO:teuthology.orchestra.run.smithi016.stdout:/run/user/1000 2024-03-31T15:10:11.142 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/ceph 2024-03-31T15:10:11.143 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/ceph/osd/ceph-0 2024-03-31T15:10:11.143 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/ceph/osd/ceph-1 2024-03-31T15:10:11.143 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/ceph/osd/ceph-2 2024-03-31T15:10:11.143 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/ceph/osd/ceph-3 2024-03-31T15:10:11.143 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns 2024-03-31T15:10:11.143 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-31T15:10:11.143 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-31T15:10:11.143 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-31T15:10:11.143 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-31T15:10:11.144 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:10:11.144 DEBUG:teuthology.orchestra.run.smithi016:> ls /sys/fs/fuse/connections 2024-03-31T15:10:11.162 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi016.stderr:2024-03-31T15:10:11.168+0000 7f9b0b370580 -1 init, newargv = 0x562dc871c330 newargc=15 2024-03-31T15:10:11.162 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi016.stderr:ceph-fuse[45433]: starting ceph client 2024-03-31T15:10:11.170 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi016.stderr:ceph-fuse[45433]: starting fuse 2024-03-31T15:10:11.172 INFO:teuthology.orchestra.run.smithi016.stdout:43 2024-03-31T15:10:11.172 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [43] 2024-03-31T15:10:11.172 DEBUG:teuthology.orchestra.run.smithi016:> sudo stdin-killer -- python3 -c ' 2024-03-31T15:10:11.172 DEBUG:teuthology.orchestra.run.smithi016:> import glob 2024-03-31T15:10:11.172 DEBUG:teuthology.orchestra.run.smithi016:> import re 2024-03-31T15:10:11.172 DEBUG:teuthology.orchestra.run.smithi016:> import os 2024-03-31T15:10:11.172 DEBUG:teuthology.orchestra.run.smithi016:> import subprocess 2024-03-31T15:10:11.172 DEBUG:teuthology.orchestra.run.smithi016:> 2024-03-31T15:10:11.173 DEBUG:teuthology.orchestra.run.smithi016:> def _find_admin_socket(client_name): 2024-03-31T15:10:11.173 DEBUG:teuthology.orchestra.run.smithi016:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-03-31T15:10:11.173 DEBUG:teuthology.orchestra.run.smithi016:> files = glob.glob(asok_path) 2024-03-31T15:10:11.173 DEBUG:teuthology.orchestra.run.smithi016:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-03-31T15:10:11.173 DEBUG:teuthology.orchestra.run.smithi016:> 2024-03-31T15:10:11.173 DEBUG:teuthology.orchestra.run.smithi016:> # Given a non-glob path, it better be there 2024-03-31T15:10:11.173 DEBUG:teuthology.orchestra.run.smithi016:> if "*" not in asok_path: 2024-03-31T15:10:11.173 DEBUG:teuthology.orchestra.run.smithi016:> assert(len(files) == 1) 2024-03-31T15:10:11.173 DEBUG:teuthology.orchestra.run.smithi016:> return files[0] 2024-03-31T15:10:11.173 DEBUG:teuthology.orchestra.run.smithi016:> 2024-03-31T15:10:11.173 DEBUG:teuthology.orchestra.run.smithi016:> for f in files: 2024-03-31T15:10:11.173 DEBUG:teuthology.orchestra.run.smithi016:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-31T15:10:11.173 DEBUG:teuthology.orchestra.run.smithi016:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-31T15:10:11.173 DEBUG:teuthology.orchestra.run.smithi016:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-31T15:10:11.173 DEBUG:teuthology.orchestra.run.smithi016:> contents = proc_f.read() 2024-03-31T15:10:11.173 DEBUG:teuthology.orchestra.run.smithi016:> if mountpoint in contents: 2024-03-31T15:10:11.173 DEBUG:teuthology.orchestra.run.smithi016:> return f 2024-03-31T15:10:11.173 DEBUG:teuthology.orchestra.run.smithi016:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-31T15:10:11.174 DEBUG:teuthology.orchestra.run.smithi016:> 2024-03-31T15:10:11.174 DEBUG:teuthology.orchestra.run.smithi016:> print(_find_admin_socket("client.0")) 2024-03-31T15:10:11.174 DEBUG:teuthology.orchestra.run.smithi016:> ' 2024-03-31T15:10:11.297 INFO:teuthology.orchestra.run.smithi016.stdout:/var/run/ceph/ceph-client.0.45433.asok 2024-03-31T15:10:11.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:11 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-31T15:10:11.309 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.45433.asok 2024-03-31T15:10:11.309 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:10:11.309 DEBUG:teuthology.orchestra.run.smithi016:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.45433.asok status 2024-03-31T15:10:11.363 INFO:tasks.ceph.mds.c.smithi016.stderr:2024-03-31T15:10:11.372+0000 7f8537000640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-31T15:10:11.483 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-03-31T15:10:11.483 INFO:teuthology.orchestra.run.smithi016.stdout: "metadata": { 2024-03-31T15:10:11.483 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph_sha1": "d2271579d5e735cfd6d436b8349c004f06527cb0", 2024-03-31T15:10:11.483 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph_version": "ceph version 19.0.0-2574-gd2271579 (d2271579d5e735cfd6d436b8349c004f06527cb0) squid (dev)", 2024-03-31T15:10:11.483 INFO:teuthology.orchestra.run.smithi016.stdout: "entity_id": "0", 2024-03-31T15:10:11.483 INFO:teuthology.orchestra.run.smithi016.stdout: "hostname": "smithi016", 2024-03-31T15:10:11.483 INFO:teuthology.orchestra.run.smithi016.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-03-31T15:10:11.484 INFO:teuthology.orchestra.run.smithi016.stdout: "pid": "45433", 2024-03-31T15:10:11.484 INFO:teuthology.orchestra.run.smithi016.stdout: "root": "/" 2024-03-31T15:10:11.484 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-31T15:10:11.484 INFO:teuthology.orchestra.run.smithi016.stdout: "dentry_count": 0, 2024-03-31T15:10:11.484 INFO:teuthology.orchestra.run.smithi016.stdout: "dentry_pinned_count": 0, 2024-03-31T15:10:11.484 INFO:teuthology.orchestra.run.smithi016.stdout: "id": 7839, 2024-03-31T15:10:11.484 INFO:teuthology.orchestra.run.smithi016.stdout: "inst": { 2024-03-31T15:10:11.484 INFO:teuthology.orchestra.run.smithi016.stdout: "name": { 2024-03-31T15:10:11.484 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "client", 2024-03-31T15:10:11.484 INFO:teuthology.orchestra.run.smithi016.stdout: "num": 7839 2024-03-31T15:10:11.484 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-31T15:10:11.484 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": { 2024-03-31T15:10:11.484 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-03-31T15:10:11.484 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "192.168.0.1:0", 2024-03-31T15:10:11.485 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 3556362935 2024-03-31T15:10:11.485 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-31T15:10:11.485 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-31T15:10:11.485 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": { 2024-03-31T15:10:11.485 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-03-31T15:10:11.485 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "192.168.0.1:0", 2024-03-31T15:10:11.485 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 3556362935 2024-03-31T15:10:11.485 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-31T15:10:11.485 INFO:teuthology.orchestra.run.smithi016.stdout: "inst_str": "client.7839 192.168.0.1:0/3556362935", 2024-03-31T15:10:11.485 INFO:teuthology.orchestra.run.smithi016.stdout: "addr_str": "192.168.0.1:0/3556362935", 2024-03-31T15:10:11.485 INFO:teuthology.orchestra.run.smithi016.stdout: "inode_count": 1, 2024-03-31T15:10:11.485 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_epoch": 67, 2024-03-31T15:10:11.485 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_epoch": 82, 2024-03-31T15:10:11.485 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_epoch_barrier": 0, 2024-03-31T15:10:11.486 INFO:teuthology.orchestra.run.smithi016.stdout: "blocklisted": false, 2024-03-31T15:10:11.486 INFO:teuthology.orchestra.run.smithi016.stdout: "fs_name": "cephfs" 2024-03-31T15:10:11.486 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-03-31T15:10:11.498 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:10:11.498 DEBUG:teuthology.orchestra.run.smithi016:> stat --file-system '--printf=%T 2024-03-31T15:10:11.499 DEBUG:teuthology.orchestra.run.smithi016:> ' -- /home/ubuntu/cephtest/mnt.0 2024-03-31T15:10:11.552 INFO:teuthology.orchestra.run.smithi016.stdout:fuseblk 2024-03-31T15:10:11.552 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-03-31T15:10:11.552 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:10:11.552 DEBUG:teuthology.orchestra.run.smithi016:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-31T15:10:11.614 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:10:11.614 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat testdir/hardlink) 2024-03-31T15:10:11.672 INFO:tasks.ceph.mds.c.smithi016.stderr:2024-03-31T15:10:11.680+0000 7f8532000640 -1 log_channel(cluster) log [ERR] : dir 0x10000000000 object missing on disk; some files may be lost (/dir) 2024-03-31T15:10:11.673 INFO:teuthology.orchestra.run.smithi016.stderr:stat: cannot statx 'testdir/hardlink': Input/output error 2024-03-31T15:10:11.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-31T15:10:11.673 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:10:11.674 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.6:0 scrub start /dir recursive 2024-03-31T15:10:11.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:11.864+0000 7f986b400640 1 Processor -- start 2024-03-31T15:10:11.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:11.864+0000 7f986b400640 1 -- start start 2024-03-31T15:10:11.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:11.868+0000 7f986b400640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f9864147230 0x7f98640b3dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:11.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:11.868+0000 7f986b400640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f98640d14b0 con 0x7f98640d1040 2024-03-31T15:10:11.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:11.868+0000 7f986b400640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f98640d2020 con 0x7f98640b4300 2024-03-31T15:10:11.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:11.868+0000 7f986b400640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f98640b4760 con 0x7f9864147230 2024-03-31T15:10:11.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:11.868+0000 7f9868a00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f9864147230 0x7f98640b3dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:11.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:11.868+0000 7f9869400640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f98640d1040 0x7f98640cb4b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:40632/0 (socket says 172.21.15.16:40632) 2024-03-31T15:10:11.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:11.868+0000 7f9869400640 1 -- 172.21.15.16:0/2649080859 learned_addr learned my addr 172.21.15.16:0/2649080859 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:10:11.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:11.868+0000 7f9868a00640 1 -- 172.21.15.16:0/2649080859 >> v1:172.21.15.89:6789/0 conn(0x7f98640b4300 legacy=0x7f98640d0930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:10:11.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:11.868+0000 7f9868a00640 1 -- 172.21.15.16:0/2649080859 >> v1:172.21.15.16:6789/0 conn(0x7f98640d1040 legacy=0x7f98640cb4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:10:11.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:11.868+0000 7f9868a00640 1 -- 172.21.15.16:0/2649080859 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f98640d2300 con 0x7f9864147230 2024-03-31T15:10:11.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:11.868+0000 7f9863e00640 1 -- 172.21.15.16:0/2649080859 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3546550739 0 0) 0x7f98640d14b0 con 0x7f98640d1040 2024-03-31T15:10:11.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:11.872+0000 7f9868a00640 1 --2- 172.21.15.16:0/2649080859 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9864147230 0x7f98640b3dc0 secure :-1 s=READY pgs=1171 cs=0 l=1 rev1=1 crypto rx=0x7f98540097c0 tx=0x7f985401c460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e343096eff384256 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:10:11.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:11.872+0000 7f9863e00640 1 -- 172.21.15.16:0/2649080859 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98540b0070 con 0x7f9864147230 2024-03-31T15:10:11.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:11.872+0000 7f9863e00640 1 -- 172.21.15.16:0/2649080859 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f98540ab600 con 0x7f9864147230 2024-03-31T15:10:11.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:11.872+0000 7f9863e00640 1 -- 172.21.15.16:0/2649080859 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98540ab9d0 con 0x7f9864147230 2024-03-31T15:10:11.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:11.872+0000 7f986b400640 1 -- 172.21.15.16:0/2649080859 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9864147230 msgr2=0x7f98640b3dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:10:11.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:11.872+0000 7f986b400640 1 --2- 172.21.15.16:0/2649080859 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9864147230 0x7f98640b3dc0 secure :-1 s=READY pgs=1171 cs=0 l=1 rev1=1 crypto rx=0x7f98540097c0 tx=0x7f985401c460 comp rx=0 tx=0).stop 2024-03-31T15:10:11.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:11.872+0000 7f986b400640 1 -- 172.21.15.16:0/2649080859 shutdown_connections 2024-03-31T15:10:11.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:11.872+0000 7f986b400640 1 --2- 172.21.15.16:0/2649080859 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9864147230 0x7f98640b3dc0 unknown :-1 s=CLOSED pgs=1171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:11.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:11.872+0000 7f986b400640 1 -- 172.21.15.16:0/2649080859 >> 172.21.15.16:0/2649080859 conn(0x7f986406b7f0 msgr2=0x7f9864067330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:10:11.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:11.872+0000 7f986b400640 1 -- 172.21.15.16:0/2649080859 shutdown_connections 2024-03-31T15:10:11.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:11.872+0000 7f986b400640 1 -- 172.21.15.16:0/2649080859 wait complete. 2024-03-31T15:10:11.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:11.876+0000 7f986b400640 1 Processor -- start 2024-03-31T15:10:11.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:11.876+0000 7f986b400640 1 -- start start 2024-03-31T15:10:11.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:11.876+0000 7f986b400640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f98640b4300 0x7f986415dfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:11.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:11.876+0000 7f986b400640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f98640d1040 0x7f98640b2400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:11.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:11.876+0000 7f986b400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9864147230 0x7f98640b6f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:11.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:11.876+0000 7f986b400640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f986416c410 con 0x7f9864147230 2024-03-31T15:10:11.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:11.876+0000 7f986b400640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f986416c290 con 0x7f98640b4300 2024-03-31T15:10:11.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:11.876+0000 7f986b400640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f986416c590 con 0x7f98640d1040 2024-03-31T15:10:11.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:11.876+0000 7f9869400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9864147230 0x7f98640b6f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:11.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:11.876+0000 7f9869400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9864147230 0x7f98640b6f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36592/0 (socket says 172.21.15.16:36592) 2024-03-31T15:10:11.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:11.876+0000 7f9869400640 1 -- 172.21.15.16:0/921994543 learned_addr learned my addr 172.21.15.16:0/921994543 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:10:11.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:11.876+0000 7f9868a00640 1 --2- 172.21.15.16:0/921994543 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f98640b4300 0x7f986415dfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:11.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:11.876+0000 7f985be00640 1 --2- 172.21.15.16:0/921994543 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f98640d1040 0x7f98640b2400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:11.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:11.876+0000 7f9869400640 1 -- 172.21.15.16:0/921994543 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f98640d1040 msgr2=0x7f98640b2400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:10:11.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:11.876+0000 7f9869400640 1 --2- 172.21.15.16:0/921994543 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f98640d1040 0x7f98640b2400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:11.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:11.876+0000 7f9869400640 1 -- 172.21.15.16:0/921994543 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f98640b4300 msgr2=0x7f986415dfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:10:11.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:11.876+0000 7f9869400640 1 --2- 172.21.15.16:0/921994543 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f98640b4300 0x7f986415dfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:11.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:11.876+0000 7f9869400640 1 -- 172.21.15.16:0/921994543 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9864140270 con 0x7f9864147230 2024-03-31T15:10:11.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:11.876+0000 7f9869400640 1 --2- 172.21.15.16:0/921994543 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9864147230 0x7f98640b6f70 secure :-1 s=READY pgs=1172 cs=0 l=1 rev1=1 crypto rx=0x7f985c00c700 tx=0x7f985c00cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:10:11.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:11.876+0000 7f9861600640 1 -- 172.21.15.16:0/921994543 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f985c018070 con 0x7f9864147230 2024-03-31T15:10:11.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:11.876+0000 7f9861600640 1 -- 172.21.15.16:0/921994543 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f985c00ce10 con 0x7f9864147230 2024-03-31T15:10:11.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:11.876+0000 7f9861600640 1 -- 172.21.15.16:0/921994543 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f985c00a860 con 0x7f9864147230 2024-03-31T15:10:11.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:11.876+0000 7f986b400640 1 -- 172.21.15.16:0/921994543 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9864140550 con 0x7f9864147230 2024-03-31T15:10:11.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:11.876+0000 7f986b400640 1 -- 172.21.15.16:0/921994543 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9864140b10 con 0x7f9864147230 2024-03-31T15:10:11.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:11.880+0000 7f9861600640 1 -- 172.21.15.16:0/921994543 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f985c00aa00 con 0x7f9864147230 2024-03-31T15:10:11.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:11.880+0000 7f9861600640 1 --2- 172.21.15.16:0/921994543 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f9840068380 0x7f984006a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:11.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:11.880+0000 7f9861600640 1 -- 172.21.15.16:0/921994543 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 5795+0+0 (secure 0 0 0) 0x7f985c085f80 con 0x7f9864147230 2024-03-31T15:10:11.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:11.880+0000 7f9868a00640 1 --2- 172.21.15.16:0/921994543 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f9840068380 0x7f984006a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:11.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:11.880+0000 7f986e878000 1 Processor -- start 2024-03-31T15:10:11.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:11.880+0000 7f986e878000 1 -- start start 2024-03-31T15:10:11.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:11.880+0000 7f9868a00640 1 --2- 172.21.15.16:0/921994543 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f9840068380 0x7f984006a840 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f98540ad000 tx=0x7f98540a6000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:10:11.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:11.884+0000 7f986e878000 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ea401c5210 0x55ea401c55f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:11.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:11.884+0000 7f986e878000 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55ea401c5b30 0x55ea4017b6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:11.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:11.884+0000 7f986e878000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ea4017bc30 0x55ea4017c010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:11.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:11.884+0000 7f986e878000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55ea3ffc9c60 con 0x55ea4017bc30 2024-03-31T15:10:11.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:11.884+0000 7f986e878000 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55ea3ff66b80 con 0x55ea401c5210 2024-03-31T15:10:11.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:11.884+0000 7f986e878000 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x55ea3ffc5c10 con 0x55ea401c5b30 2024-03-31T15:10:11.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:11.884+0000 7f9869400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ea4017bc30 0x55ea4017c010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:11.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:11.884+0000 7f9868a00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ea401c5210 0x55ea401c55f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:11.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:11.884+0000 7f9868a00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ea401c5210 0x55ea401c55f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.16:51896/0 (socket says 172.21.15.16:51896) 2024-03-31T15:10:11.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:11.884+0000 7f9868a00640 1 -- 172.21.15.16:0/48947281 learned_addr learned my addr 172.21.15.16:0/48947281 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:10:11.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:11.884+0000 7f985be00640 1 --2- 172.21.15.16:0/48947281 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55ea401c5b30 0x55ea4017b6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:11.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:11.884+0000 7f9868a00640 1 -- 172.21.15.16:0/48947281 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55ea401c5b30 msgr2=0x55ea4017b6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:10:11.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:11.884+0000 7f9868a00640 1 --2- 172.21.15.16:0/48947281 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55ea401c5b30 0x55ea4017b6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:11.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:11.884+0000 7f9868a00640 1 -- 172.21.15.16:0/48947281 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ea4017bc30 msgr2=0x55ea4017c010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:10:11.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:11.884+0000 7f9868a00640 1 --2- 172.21.15.16:0/48947281 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ea4017bc30 0x55ea4017c010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:11.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:11.884+0000 7f9868a00640 1 -- 172.21.15.16:0/48947281 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ea3ffa38d0 con 0x55ea401c5210 2024-03-31T15:10:11.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:11.884+0000 7f9869400640 1 --2- 172.21.15.16:0/48947281 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ea4017bc30 0x55ea4017c010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:10:11.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:11.884+0000 7f9868a00640 1 --2- 172.21.15.16:0/48947281 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ea401c5210 0x55ea401c55f0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f9854000c10 tx=0x7f9854004740 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3e2c8662c1b2a2 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:10:11.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:11.884+0000 7f9859600640 1 -- 172.21.15.16:0/48947281 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98540b0040 con 0x55ea401c5210 2024-03-31T15:10:11.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:11.884+0000 7f9859600640 1 -- 172.21.15.16:0/48947281 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f98540b1040 con 0x55ea401c5210 2024-03-31T15:10:11.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:11.884+0000 7f9859600640 1 -- 172.21.15.16:0/48947281 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98540c2600 con 0x55ea401c5210 2024-03-31T15:10:11.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:11.884+0000 7f986e878000 1 -- 172.21.15.16:0/48947281 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ea401c5210 msgr2=0x55ea401c55f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:10:11.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:11.884+0000 7f986e878000 1 --2- 172.21.15.16:0/48947281 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ea401c5210 0x55ea401c55f0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f9854000c10 tx=0x7f9854004740 comp rx=0 tx=0).stop 2024-03-31T15:10:11.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:11.884+0000 7f986e878000 1 -- 172.21.15.16:0/48947281 shutdown_connections 2024-03-31T15:10:11.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:11.884+0000 7f986e878000 1 --2- 172.21.15.16:0/48947281 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ea4017bc30 0x55ea4017c010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:11.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:11.884+0000 7f986e878000 1 --2- 172.21.15.16:0/48947281 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55ea401c5b30 0x55ea4017b6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:11.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:11.884+0000 7f986e878000 1 --2- 172.21.15.16:0/48947281 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ea401c5210 0x55ea401c55f0 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:11.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:11.884+0000 7f986e878000 1 -- 172.21.15.16:0/48947281 >> 172.21.15.16:0/48947281 conn(0x55ea40177be0 msgr2=0x55ea40179fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:10:11.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:11.884+0000 7f986e878000 1 -- 172.21.15.16:0/48947281 shutdown_connections 2024-03-31T15:10:11.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:11.884+0000 7f986e878000 1 -- 172.21.15.16:0/48947281 wait complete. 2024-03-31T15:10:11.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:11.884+0000 7f986e878000 1 -- start start 2024-03-31T15:10:11.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:11.884+0000 7f986e878000 1 Processor -- start 2024-03-31T15:10:11.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:11.884+0000 7f986e878000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ea4017bc30 0x55ea40181100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:11.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:11.888+0000 7f986e878000 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55ea401c5210 0x55ea401aa450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:11.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:11.888+0000 7f986e878000 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ea401c5b30 0x55ea401aa990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:11.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:11.888+0000 7f986e878000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55ea3ff872a0 con 0x55ea4017bc30 2024-03-31T15:10:11.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:11.888+0000 7f986e878000 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55ea40177a60 con 0x55ea401c5b30 2024-03-31T15:10:11.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:11.888+0000 7f986e878000 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x55ea3ffce290 con 0x55ea401c5210 2024-03-31T15:10:11.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:11.888+0000 7f986e878000 10 client.0 _ll_register_callbacks cb 0x55ea40166af0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T15:10:11.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:11.888+0000 7f9868a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ea4017bc30 0x55ea40181100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:11.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:11.888+0000 7f9868a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ea4017bc30 0x55ea40181100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36608/0 (socket says 172.21.15.16:36608) 2024-03-31T15:10:11.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:11.888+0000 7f9868a00640 1 -- 172.21.15.16:0/1894729177 learned_addr learned my addr 172.21.15.16:0/1894729177 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:10:11.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:11.888+0000 7f985be00640 1 --2- 172.21.15.16:0/1894729177 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55ea401c5210 0x55ea401aa450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:11.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:11.888+0000 7f9869400640 1 --2- 172.21.15.16:0/1894729177 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ea401c5b30 0x55ea401aa990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:11.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:11.888+0000 7f9868a00640 1 -- 172.21.15.16:0/1894729177 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55ea401c5210 msgr2=0x55ea401aa450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:10:11.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:11.888+0000 7f9868a00640 1 --2- 172.21.15.16:0/1894729177 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55ea401c5210 0x55ea401aa450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:11.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:11.888+0000 7f9868a00640 1 -- 172.21.15.16:0/1894729177 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ea401c5b30 msgr2=0x55ea401aa990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:10:11.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:11.888+0000 7f9868a00640 1 --2- 172.21.15.16:0/1894729177 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ea401c5b30 0x55ea401aa990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:11.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:11.888+0000 7f9868a00640 1 -- 172.21.15.16:0/1894729177 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ea4000f0d0 con 0x55ea4017bc30 2024-03-31T15:10:11.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:11.888+0000 7f9868a00640 1 --2- 172.21.15.16:0/1894729177 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ea4017bc30 0x55ea40181100 secure :-1 s=READY pgs=1173 cs=0 l=1 rev1=1 crypto rx=0x7f985401c940 tx=0x7f985401aca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=63b1ef22decf75fa server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:10:11.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:11.888+0000 7f9852000640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-31T15:10:11.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:11.888+0000 7f9852000640 1 -- 172.21.15.16:0/1894729177 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98540b0040 con 0x55ea4017bc30 2024-03-31T15:10:11.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:11.888+0000 7f986e878000 1 -- 172.21.15.16:0/1894729177 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f98540b1040 con 0x55ea4017bc30 2024-03-31T15:10:11.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:11.888+0000 7f9852000640 1 -- 172.21.15.16:0/1894729177 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 5795+0+0 (secure 0 0 0) 0x7f98540c7040 con 0x55ea4017bc30 2024-03-31T15:10:11.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:11.888+0000 7f9852000640 1 -- 172.21.15.16:0/1894729177 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=67) ==== 24+0+0 (secure 0 0 0) 0x7f98540d6050 con 0x55ea4017bc30 2024-03-31T15:10:11.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:11.888+0000 7f986e878000 10 client.7854 fetch_fsmap learned FSMap version 67 2024-03-31T15:10:11.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:11.888+0000 7f986e878000 1 -- 172.21.15.16:0/1894729177 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=67,monmap=2+}) -- 0x55ea3ff4f4e0 con 0x55ea4017bc30 2024-03-31T15:10:11.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:11.888+0000 7f9852000640 1 -- 172.21.15.16:0/1894729177 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f98540b1040 con 0x55ea4017bc30 2024-03-31T15:10:11.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:11.888+0000 7f9852000640 1 -- 172.21.15.16:0/1894729177 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 67) ==== 3494+0+0 (secure 0 0 0) 0x7f98540d9020 con 0x55ea4017bc30 2024-03-31T15:10:11.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:11.888+0000 7f986e878000 10 client.7854 fetch_fsmap finished waiting for FSMap version 67 2024-03-31T15:10:11.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:11.888+0000 7f986e878000 10 client.7854 resolve_mds: resolved 6:0 to role '6:0' aka daemon mds.c 2024-03-31T15:10:11.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:11.888+0000 7f986e878000 20 client.7854 populate_metadata read hostname 'smithi016' 2024-03-31T15:10:11.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:11.888+0000 7f986e878000 1 --2- 172.21.15.16:0/1894729177 >> [v2:172.21.15.16:6832/818074038,v1:172.21.15.16:6834/818074038] conn(0x55ea401cc650 0x55ea401cca60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:11.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:11.888+0000 7f986e878000 4 client.7854 mds_command: new command op to 7749 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-31T15:10:11.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:11.888+0000 7f986e878000 1 -- 172.21.15.16:0/1894729177 --> [v2:172.21.15.16:6832/818074038,v1:172.21.15.16:6834/818074038] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ea3ff64f70 con 0x55ea401cc650 2024-03-31T15:10:11.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:11.888+0000 7f985be00640 1 --2- 172.21.15.16:0/1894729177 >> [v2:172.21.15.16:6832/818074038,v1:172.21.15.16:6834/818074038] conn(0x55ea401cc650 0x55ea401cca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:11.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:11.888+0000 7f985be00640 1 --2- 172.21.15.16:0/1894729177 >> [v2:172.21.15.16:6832/818074038,v1:172.21.15.16:6834/818074038] conn(0x55ea401cc650 0x55ea401cca60 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=db4b87820a302302 server_cookie=f68a20adf2aa4b52 in_seq=0 out_seq=0 2024-03-31T15:10:11.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:11.888+0000 7f9852000640 10 client.7854 ms_handle_connect on v2:172.21.15.16:6832/818074038 2024-03-31T15:10:11.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:11.888+0000 7f9852000640 1 -- 172.21.15.16:0/1894729177 <== mds.0 v2:172.21.15.16:6832/818074038 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55ea3ff64f70 con 0x55ea401cc650 2024-03-31T15:10:11.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:11.888+0000 7f9852000640 10 client.7854 handle_command_reply: tid=0 2024-03-31T15:10:11.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:11.888+0000 7f986e878000 1 client.7854 shutdown 2024-03-31T15:10:11.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:11.888+0000 7f986e878000 1 -- 172.21.15.16:0/1894729177 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ea4017bc30 msgr2=0x55ea40181100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:10:11.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:11.888+0000 7f986e878000 1 --2- 172.21.15.16:0/1894729177 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ea4017bc30 0x55ea40181100 secure :-1 s=READY pgs=1173 cs=0 l=1 rev1=1 crypto rx=0x7f985401c940 tx=0x7f985401aca0 comp rx=0 tx=0).stop 2024-03-31T15:10:11.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:11.888+0000 7f986e878000 1 -- 172.21.15.16:0/1894729177 shutdown_connections 2024-03-31T15:10:11.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:11.888+0000 7f986e878000 1 --2- 172.21.15.16:0/1894729177 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ea401c5b30 0x55ea401aa990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:11.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:11.888+0000 7f986e878000 1 --2- 172.21.15.16:0/1894729177 >> [v2:172.21.15.16:6832/818074038,v1:172.21.15.16:6834/818074038] conn(0x55ea401cc650 0x55ea401cca60 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:11.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:11.888+0000 7f986e878000 1 --2- 172.21.15.16:0/1894729177 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55ea401c5210 0x55ea401aa450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:11.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:11.888+0000 7f986e878000 1 --2- 172.21.15.16:0/1894729177 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ea4017bc30 0x55ea40181100 unknown :-1 s=CLOSED pgs=1173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:11.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:11.888+0000 7f9852000640 0 client.7854 ms_handle_reset on v2:172.21.15.16:6832/818074038 2024-03-31T15:10:11.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:11.888+0000 7f986e878000 1 -- 172.21.15.16:0/1894729177 >> 172.21.15.16:0/1894729177 conn(0x55ea40177be0 msgr2=0x55ea4017a860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:10:11.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:11.888+0000 7f986e878000 1 -- 172.21.15.16:0/1894729177 shutdown_connections 2024-03-31T15:10:11.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:11.888+0000 7f986e878000 1 -- 172.21.15.16:0/1894729177 wait complete. 2024-03-31T15:10:11.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:11.892+0000 7f986e878000 20 client.7854 trim_cache size 0 max 16384 2024-03-31T15:10:11.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:11.908+0000 7f986e878000 1 Processor -- start 2024-03-31T15:10:11.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:11.908+0000 7f986e878000 1 -- start start 2024-03-31T15:10:11.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:11.908+0000 7f986e878000 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ea4017bc30 0x55ea401c4330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:11.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:11.908+0000 7f986e878000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ea401c5210 0x55ea401c4870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:11.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:11.908+0000 7f986e878000 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55ea401c5b30 0x55ea40179fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:11.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:11.908+0000 7f986e878000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f9864203af0 con 0x55ea401c5210 2024-03-31T15:10:11.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:11.908+0000 7f986e878000 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f9864202b70 con 0x55ea4017bc30 2024-03-31T15:10:11.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:11.908+0000 7f986e878000 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f9864201f20 con 0x55ea401c5b30 2024-03-31T15:10:11.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:11.908+0000 7f985be00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ea401c5210 0x55ea401c4870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:11.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:11.908+0000 7f985be00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ea401c5210 0x55ea401c4870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36616/0 (socket says 172.21.15.16:36616) 2024-03-31T15:10:11.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:11.908+0000 7f985be00640 1 -- 172.21.15.16:0/2035291794 learned_addr learned my addr 172.21.15.16:0/2035291794 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:10:11.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:11.908+0000 7f9869400640 1 --2- 172.21.15.16:0/2035291794 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55ea401c5b30 0x55ea40179fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:11.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:11.908+0000 7f9868a00640 1 --2- 172.21.15.16:0/2035291794 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ea4017bc30 0x55ea401c4330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:11.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:11.908+0000 7f985be00640 1 -- 172.21.15.16:0/2035291794 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55ea401c5b30 msgr2=0x55ea40179fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:10:11.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:11.908+0000 7f985be00640 1 --2- 172.21.15.16:0/2035291794 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55ea401c5b30 0x55ea40179fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:11.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:11.908+0000 7f985be00640 1 -- 172.21.15.16:0/2035291794 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ea4017bc30 msgr2=0x55ea401c4330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:10:11.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:11.908+0000 7f985be00640 1 --2- 172.21.15.16:0/2035291794 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ea4017bc30 0x55ea401c4330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:11.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:11.908+0000 7f985be00640 1 -- 172.21.15.16:0/2035291794 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ea3ff97410 con 0x55ea401c5210 2024-03-31T15:10:11.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:11.908+0000 7f985be00640 1 --2- 172.21.15.16:0/2035291794 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ea401c5210 0x55ea401c4870 secure :-1 s=READY pgs=1174 cs=0 l=1 rev1=1 crypto rx=0x7f984c004740 tx=0x7f984c008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=85991b76f75229fb server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:10:11.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:11.908+0000 7f9858c00640 1 -- 172.21.15.16:0/2035291794 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f984c020070 con 0x55ea401c5210 2024-03-31T15:10:11.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:11.908+0000 7f9858c00640 1 -- 172.21.15.16:0/2035291794 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f984c01cc10 con 0x55ea401c5210 2024-03-31T15:10:11.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:11.908+0000 7f9858c00640 1 -- 172.21.15.16:0/2035291794 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f984c0158d0 con 0x55ea401c5210 2024-03-31T15:10:11.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:11.912+0000 7f986e878000 1 -- 172.21.15.16:0/2035291794 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ea401c5210 msgr2=0x55ea401c4870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:10:11.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:11.912+0000 7f986e878000 1 --2- 172.21.15.16:0/2035291794 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ea401c5210 0x55ea401c4870 secure :-1 s=READY pgs=1174 cs=0 l=1 rev1=1 crypto rx=0x7f984c004740 tx=0x7f984c008040 comp rx=0 tx=0).stop 2024-03-31T15:10:11.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:11.912+0000 7f986e878000 1 -- 172.21.15.16:0/2035291794 shutdown_connections 2024-03-31T15:10:11.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:11.912+0000 7f986e878000 1 --2- 172.21.15.16:0/2035291794 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55ea401c5b30 0x55ea40179fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:11.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:11.912+0000 7f986e878000 1 --2- 172.21.15.16:0/2035291794 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ea401c5210 0x55ea401c4870 unknown :-1 s=CLOSED pgs=1174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:11.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:11.912+0000 7f986e878000 1 --2- 172.21.15.16:0/2035291794 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ea4017bc30 0x55ea401c4330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:11.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:11.912+0000 7f986e878000 1 -- 172.21.15.16:0/2035291794 >> 172.21.15.16:0/2035291794 conn(0x55ea40177be0 msgr2=0x55ea401dbf90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:10:11.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:11.912+0000 7f986e878000 1 -- 172.21.15.16:0/2035291794 shutdown_connections 2024-03-31T15:10:11.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:11.912+0000 7f986e878000 1 -- 172.21.15.16:0/2035291794 wait complete. 2024-03-31T15:10:11.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:11.912+0000 7f986e878000 1 -- start start 2024-03-31T15:10:11.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:11.912+0000 7f986e878000 1 Processor -- start 2024-03-31T15:10:11.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:11.912+0000 7f986e878000 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ea4017bc30 0x55ea401c4760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:11.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:11.912+0000 7f986e878000 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55ea401c5210 0x55ea401c7f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:11.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:11.912+0000 7f986e878000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ea401c5b30 0x55ea401c8460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:11.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:11.912+0000 7f986e878000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55ea3ffce290 con 0x55ea401c5b30 2024-03-31T15:10:11.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:11.912+0000 7f986e878000 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f9864202520 con 0x55ea4017bc30 2024-03-31T15:10:11.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:11.912+0000 7f986e878000 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x55ea3ff512c0 con 0x55ea401c5210 2024-03-31T15:10:11.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:11.912+0000 7f986e878000 10 client.0 _ll_register_callbacks cb 0x7f985c052120 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T15:10:11.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:11.912+0000 7f9869400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ea401c5b30 0x55ea401c8460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:11.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:11.912+0000 7f985be00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55ea401c5210 0x55ea401c7f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:11.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:11.912+0000 7f9868a00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ea4017bc30 0x55ea401c4760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:11.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:11.912+0000 7f9869400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ea401c5b30 0x55ea401c8460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36632/0 (socket says 172.21.15.16:36632) 2024-03-31T15:10:11.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:11.912+0000 7f9869400640 1 -- 172.21.15.16:0/3220068470 learned_addr learned my addr 172.21.15.16:0/3220068470 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:10:11.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:11.912+0000 7f985be00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55ea401c5210 0x55ea401c7f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3301/0 says I am v2:172.21.15.16:56850/0 (socket says 172.21.15.16:56850) 2024-03-31T15:10:11.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:11.912+0000 7f985be00640 1 -- 172.21.15.16:0/3220068470 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ea4017bc30 msgr2=0x55ea401c4760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:10:11.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:11.912+0000 7f985be00640 1 --2- 172.21.15.16:0/3220068470 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ea4017bc30 0x55ea401c4760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:11.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:11.912+0000 7f985be00640 1 -- 172.21.15.16:0/3220068470 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ea401c5b30 msgr2=0x55ea401c8460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:10:11.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:11.912+0000 7f985be00640 1 --2- 172.21.15.16:0/3220068470 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ea401c5b30 0x55ea401c8460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:11.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:11.912+0000 7f985be00640 1 -- 172.21.15.16:0/3220068470 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({osdmap=0}) -- 0x55ea4004b1e0 con 0x55ea401c5210 2024-03-31T15:10:11.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:11.912+0000 7f9869400640 1 --2- 172.21.15.16:0/3220068470 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ea401c5b30 0x55ea401c8460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:10:11.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:11.912+0000 7f985be00640 1 --2- 172.21.15.16:0/3220068470 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55ea401c5210 0x55ea401c7f20 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x55ea40044200 tx=0x7f984c01cae0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a9236e3885a78084 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:10:11.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:11.912+0000 7f9852000640 10 client.0 ms_handle_connect on v2:172.21.15.89:3301/0 2024-03-31T15:10:11.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:11.912+0000 7f9852000640 1 -- 172.21.15.16:0/3220068470 <== mon.2 v2:172.21.15.89:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f984c020070 con 0x55ea401c5210 2024-03-31T15:10:11.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:11.912+0000 7f9852000640 1 -- 172.21.15.16:0/3220068470 <== mon.2 v2:172.21.15.89:3301/0 2 ==== osd_map(82..82 src has 1..82) ==== 5795+0+0 (secure 0 0 0) 0x7f984c01c250 con 0x55ea401c5210 2024-03-31T15:10:11.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:11.912+0000 7f986e878000 1 -- 172.21.15.16:0/3220068470 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f984c01cc10 con 0x55ea401c5210 2024-03-31T15:10:11.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:11.912+0000 7f9852000640 1 -- 172.21.15.16:0/3220068470 <== mon.2 v2:172.21.15.89:3301/0 3 ==== mon_get_version_reply(handle=1 version=67) ==== 24+0+0 (secure 0 0 0) 0x7f984c032050 con 0x55ea401c5210 2024-03-31T15:10:11.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:11.912+0000 7f986e878000 10 client.6710 fetch_fsmap learned FSMap version 67 2024-03-31T15:10:11.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:11.912+0000 7f986e878000 1 -- 172.21.15.16:0/3220068470 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({config=0+,fsmap=67,monmap=2+}) -- 0x55ea3fc853b0 con 0x55ea401c5210 2024-03-31T15:10:11.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:11.916+0000 7f9852000640 1 -- 172.21.15.16:0/3220068470 <== mon.2 v2:172.21.15.89:3301/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f984c01cc10 con 0x55ea401c5210 2024-03-31T15:10:11.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:11.916+0000 7f9852000640 1 -- 172.21.15.16:0/3220068470 <== mon.2 v2:172.21.15.89:3301/0 5 ==== fsmap(e 67) ==== 3494+0+0 (secure 0 0 0) 0x7f984c035020 con 0x55ea401c5210 2024-03-31T15:10:11.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:11.916+0000 7f986e878000 10 client.6710 fetch_fsmap finished waiting for FSMap version 67 2024-03-31T15:10:11.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:11.916+0000 7f986e878000 10 client.6710 resolve_mds: resolved 6:0 to role '6:0' aka daemon mds.c 2024-03-31T15:10:11.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:11.916+0000 7f986e878000 20 client.6710 populate_metadata read hostname 'smithi016' 2024-03-31T15:10:11.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:11.916+0000 7f986e878000 1 --2- 172.21.15.16:0/3220068470 >> [v2:172.21.15.16:6832/818074038,v1:172.21.15.16:6834/818074038] conn(0x55ea40235360 0x55ea40235770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:11.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:11.916+0000 7f986e878000 4 client.6710 mds_command: new command op to 7749 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/dir", "scrubops": ["recursive"]}] 2024-03-31T15:10:11.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:11.916+0000 7f986e878000 1 -- 172.21.15.16:0/3220068470 --> [v2:172.21.15.16:6832/818074038,v1:172.21.15.16:6834/818074038] -- command(tid 0: {"prefix": "scrub start", "path": "/dir", "scrubops": ["recursive"]}) -- 0x55ea3ffd71a0 con 0x55ea40235360 2024-03-31T15:10:11.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:11.916+0000 7f9868a00640 1 --2- 172.21.15.16:0/3220068470 >> [v2:172.21.15.16:6832/818074038,v1:172.21.15.16:6834/818074038] conn(0x55ea40235360 0x55ea40235770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:11.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:11.916+0000 7f9852000640 10 client.6710 ms_handle_connect on v2:172.21.15.16:6832/818074038 2024-03-31T15:10:11.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:11.916+0000 7f9868a00640 1 --2- 172.21.15.16:0/3220068470 >> [v2:172.21.15.16:6832/818074038,v1:172.21.15.16:6834/818074038] conn(0x55ea40235360 0x55ea40235770 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a55a99f128f98977 server_cookie=73a831127fbc65ba in_seq=0 out_seq=0 2024-03-31T15:10:11.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:11.916+0000 7f9852000640 1 -- 172.21.15.16:0/3220068470 <== mds.0 v2:172.21.15.16:6832/818074038 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55ea3ffd71a0 con 0x55ea40235360 2024-03-31T15:10:11.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:11.916+0000 7f9852000640 10 client.6710 handle_command_reply: tid=0 2024-03-31T15:10:11.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:11.916+0000 7f986e878000 1 client.6710 shutdown 2024-03-31T15:10:11.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:11.916+0000 7f986e878000 1 -- 172.21.15.16:0/3220068470 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55ea401c5210 msgr2=0x55ea401c7f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:10:11.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:11.916+0000 7f986e878000 1 --2- 172.21.15.16:0/3220068470 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55ea401c5210 0x55ea401c7f20 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x55ea40044200 tx=0x7f984c01cae0 comp rx=0 tx=0).stop 2024-03-31T15:10:11.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:11.916+0000 7f986e878000 1 -- 172.21.15.16:0/3220068470 shutdown_connections 2024-03-31T15:10:11.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:11.916+0000 7f986e878000 1 --2- 172.21.15.16:0/3220068470 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ea401c5b30 0x55ea401c8460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:11.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:11.916+0000 7f986e878000 1 --2- 172.21.15.16:0/3220068470 >> [v2:172.21.15.16:6832/818074038,v1:172.21.15.16:6834/818074038] conn(0x55ea40235360 0x55ea40235770 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:11.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:11.916+0000 7f986e878000 1 --2- 172.21.15.16:0/3220068470 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55ea401c5210 0x55ea401c7f20 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:11.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:11.916+0000 7f986e878000 1 --2- 172.21.15.16:0/3220068470 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ea4017bc30 0x55ea401c4760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:11.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:11.916+0000 7f9852000640 0 client.6710 ms_handle_reset on v2:172.21.15.16:6832/818074038 2024-03-31T15:10:11.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:11.916+0000 7f986e878000 1 -- 172.21.15.16:0/3220068470 >> 172.21.15.16:0/3220068470 conn(0x55ea40177be0 msgr2=0x55ea401dc6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:10:11.911 INFO:tasks.ceph.mds.c.smithi016.stderr:2024-03-31T15:10:11.916+0000 7f8532000640 -1 mds.0.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000000 [...2,head] /dir/ auth v13 ap=1 f(v0 m2024-03-31T15:09:59.315089+0000 1=1+0) n(v0 rc2024-03-31T15:09:59.431625+0000 2=1+1) | dirfrag=0 authpin=1 scrubqueue=0 0x55f996656100]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":-2,"ondisk_value":"(-1)0x0:[]//[]","memoryvalue":"(12)0x10000000000:[<0x1/dir v13>]//[]","error_str":"failed to decode on-disk backtrace (0 bytes)!"},"raw_stats":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n(v0 1=0+1)","memory_value.dirstat":"f(v0 m2024-03-31T15:09:59.315089+0000 1=1+0)","memory_value.rstat":"n(v0 rc2024-03-31T15:09:59.431625+0000 2=1+1)","error_str":"freshly-calculated rstats don't match existing ones"},"return_code":-2} 2024-03-31T15:10:11.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:11.916+0000 7f986e878000 1 -- 172.21.15.16:0/3220068470 shutdown_connections 2024-03-31T15:10:11.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:11.916+0000 7f986e878000 1 -- 172.21.15.16:0/3220068470 wait complete. 2024-03-31T15:10:11.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:11.916+0000 7f986e878000 20 client.6710 trim_cache size 0 max 16384 2024-03-31T15:10:11.912 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-03-31T15:10:11.912 INFO:teuthology.orchestra.run.smithi016.stdout: "return_code": 0, 2024-03-31T15:10:11.912 INFO:teuthology.orchestra.run.smithi016.stdout: "scrub_tag": "73e2fea1-8e96-428d-b985-a1c2091e8679", 2024-03-31T15:10:11.912 INFO:teuthology.orchestra.run.smithi016.stdout: "mode": "asynchronous" 2024-03-31T15:10:11.912 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-03-31T15:10:11.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:11.916+0000 7f986b400640 1 -- 172.21.15.16:0/921994543 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f9840068380 msgr2=0x7f984006a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:10:11.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:11.916+0000 7f986b400640 1 --2- 172.21.15.16:0/921994543 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f9840068380 0x7f984006a840 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f98540ad000 tx=0x7f98540a6000 comp rx=0 tx=0).stop 2024-03-31T15:10:11.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:11.916+0000 7f986b400640 1 -- 172.21.15.16:0/921994543 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9864147230 msgr2=0x7f98640b6f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:10:11.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:11.916+0000 7f986b400640 1 --2- 172.21.15.16:0/921994543 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9864147230 0x7f98640b6f70 secure :-1 s=READY pgs=1172 cs=0 l=1 rev1=1 crypto rx=0x7f985c00c700 tx=0x7f985c00cbc0 comp rx=0 tx=0).stop 2024-03-31T15:10:11.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:11.916+0000 7f986b400640 1 -- 172.21.15.16:0/921994543 shutdown_connections 2024-03-31T15:10:11.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:11.916+0000 7f986b400640 1 --2- 172.21.15.16:0/921994543 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f9840068380 0x7f984006a840 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:11.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:11.916+0000 7f986b400640 1 --2- 172.21.15.16:0/921994543 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9864147230 0x7f98640b6f70 unknown :-1 s=CLOSED pgs=1172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:11.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:11.916+0000 7f986b400640 1 --2- 172.21.15.16:0/921994543 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f98640d1040 0x7f98640b2400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:11.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:11.916+0000 7f986b400640 1 --2- 172.21.15.16:0/921994543 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f98640b4300 0x7f986415dfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:11.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:11.916+0000 7f986b400640 1 -- 172.21.15.16:0/921994543 >> 172.21.15.16:0/921994543 conn(0x7f986406b7f0 msgr2=0x7f986413e880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:10:11.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:11.916+0000 7f986b400640 1 -- 172.21.15.16:0/921994543 shutdown_connections 2024-03-31T15:10:11.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:11.916+0000 7f986b400640 1 -- 172.21.15.16:0/921994543 wait complete. 2024-03-31T15:10:11.928 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:10:11.928 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.6:0 scrub status 2024-03-31T15:10:12.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:12.076+0000 7f524e800640 1 Processor -- start 2024-03-31T15:10:12.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:12.076+0000 7f524e800640 1 -- start start 2024-03-31T15:10:12.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:12.080+0000 7f524e800640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f524806f920 0x7f524806fd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:12.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:12.080+0000 7f524e800640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f52480d0a60 0x7f52480cea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:12.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:12.080+0000 7f524e800640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5248070240 con 0x7f52480cef80 2024-03-31T15:10:12.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:12.080+0000 7f524e800640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f524813f670 con 0x7f52480d0a60 2024-03-31T15:10:12.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:12.080+0000 7f524e800640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f524815ca70 con 0x7f524806f920 2024-03-31T15:10:12.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:12.080+0000 7f523ea00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f52480d0a60 0x7f52480cea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:12.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:12.080+0000 7f523ea00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f52480d0a60 0x7f52480cea40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36646/0 (socket says 172.21.15.16:36646) 2024-03-31T15:10:12.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:12.080+0000 7f523ea00640 1 -- 172.21.15.16:0/1510776787 learned_addr learned my addr 172.21.15.16:0/1510776787 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:10:12.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:12.080+0000 7f5247400640 1 --2- 172.21.15.16:0/1510776787 >> v2:172.21.15.89:3300/0 conn(0x7f524806f920 0x7f524806fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:12.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:12.080+0000 7f523ea00640 1 -- 172.21.15.16:0/1510776787 >> v2:172.21.15.89:3300/0 conn(0x7f524806f920 msgr2=0x7f524806fd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:10:12.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:12.080+0000 7f523ea00640 1 --2- 172.21.15.16:0/1510776787 >> v2:172.21.15.89:3300/0 conn(0x7f524806f920 0x7f524806fd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:12.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:12.080+0000 7f523ea00640 1 -- 172.21.15.16:0/1510776787 >> v1:172.21.15.89:6789/0 conn(0x7f52480cef80 legacy=0x7f5248070e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:10:12.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:12.080+0000 7f523ea00640 1 -- 172.21.15.16:0/1510776787 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f524815cd50 con 0x7f52480d0a60 2024-03-31T15:10:12.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:12.084+0000 7f523ea00640 1 --2- 172.21.15.16:0/1510776787 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f52480d0a60 0x7f52480cea40 secure :-1 s=READY pgs=1175 cs=0 l=1 rev1=1 crypto rx=0x7f52480708e0 tx=0x7f523001c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=495d40b7ed1742e server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:10:12.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:12.084+0000 7f5246a00640 1 -- 172.21.15.16:0/1510776787 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52300b0070 con 0x7f52480d0a60 2024-03-31T15:10:12.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:12.084+0000 7f5246a00640 1 -- 172.21.15.16:0/1510776787 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f52300ab600 con 0x7f52480d0a60 2024-03-31T15:10:12.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:12.084+0000 7f5246a00640 1 -- 172.21.15.16:0/1510776787 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52300ab9d0 con 0x7f52480d0a60 2024-03-31T15:10:12.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:12.084+0000 7f524e800640 1 -- 172.21.15.16:0/1510776787 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f52480d0a60 msgr2=0x7f52480cea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:10:12.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:12.084+0000 7f524e800640 1 --2- 172.21.15.16:0/1510776787 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f52480d0a60 0x7f52480cea40 secure :-1 s=READY pgs=1175 cs=0 l=1 rev1=1 crypto rx=0x7f52480708e0 tx=0x7f523001c490 comp rx=0 tx=0).stop 2024-03-31T15:10:12.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:12.084+0000 7f524e800640 1 -- 172.21.15.16:0/1510776787 shutdown_connections 2024-03-31T15:10:12.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:12.084+0000 7f524e800640 1 --2- 172.21.15.16:0/1510776787 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f52480d0a60 0x7f52480cea40 unknown :-1 s=CLOSED pgs=1175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:12.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:12.084+0000 7f524e800640 1 --2- 172.21.15.16:0/1510776787 >> v2:172.21.15.89:3300/0 conn(0x7f524806f920 0x7f524806fd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:12.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:12.084+0000 7f524e800640 1 -- 172.21.15.16:0/1510776787 >> 172.21.15.16:0/1510776787 conn(0x7f5248067930 msgr2=0x7f52480b3ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:10:12.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:12.084+0000 7f524e800640 1 -- 172.21.15.16:0/1510776787 shutdown_connections 2024-03-31T15:10:12.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:12.084+0000 7f524e800640 1 -- 172.21.15.16:0/1510776787 wait complete. 2024-03-31T15:10:12.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:12.088+0000 7f524e800640 1 Processor -- start 2024-03-31T15:10:12.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:12.088+0000 7f524e800640 1 -- start start 2024-03-31T15:10:12.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:12.088+0000 7f524e800640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f524806f920 0x7f52481543a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:12.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:12.088+0000 7f524e800640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f52480cef80 0x7f524813ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:12.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:12.088+0000 7f524e800640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f52480d0a60 0x7f524815ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:12.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:12.088+0000 7f524e800640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f52481690b0 con 0x7f524806f920 2024-03-31T15:10:12.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:12.088+0000 7f524e800640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f52480cf390 con 0x7f52480cef80 2024-03-31T15:10:12.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:12.088+0000 7f524e800640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f5248169230 con 0x7f52480d0a60 2024-03-31T15:10:12.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:12.088+0000 7f5247400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f524806f920 0x7f52481543a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:12.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:12.088+0000 7f5247400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f524806f920 0x7f52481543a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36662/0 (socket says 172.21.15.16:36662) 2024-03-31T15:10:12.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:12.088+0000 7f5247400640 1 -- 172.21.15.16:0/1820855164 learned_addr learned my addr 172.21.15.16:0/1820855164 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:10:12.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:12.088+0000 7f523ea00640 1 --2- 172.21.15.16:0/1820855164 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f52480cef80 0x7f524813ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:12.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:12.088+0000 7f5247e00640 1 --2- 172.21.15.16:0/1820855164 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f52480d0a60 0x7f524815ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:12.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:12.088+0000 7f5247400640 1 -- 172.21.15.16:0/1820855164 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f52480d0a60 msgr2=0x7f524815ecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:10:12.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:12.088+0000 7f5247400640 1 --2- 172.21.15.16:0/1820855164 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f52480d0a60 0x7f524815ecc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:12.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:12.088+0000 7f5247400640 1 -- 172.21.15.16:0/1820855164 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f52480cef80 msgr2=0x7f524813ecd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:10:12.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:12.088+0000 7f5247400640 1 --2- 172.21.15.16:0/1820855164 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f52480cef80 0x7f524813ecd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:12.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:12.088+0000 7f5247400640 1 -- 172.21.15.16:0/1820855164 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f524815f200 con 0x7f524806f920 2024-03-31T15:10:12.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:12.088+0000 7f523ea00640 1 --2- 172.21.15.16:0/1820855164 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f52480cef80 0x7f524813ecd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:10:12.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:12.088+0000 7f5247400640 1 --2- 172.21.15.16:0/1820855164 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f524806f920 0x7f52481543a0 secure :-1 s=READY pgs=1176 cs=0 l=1 rev1=1 crypto rx=0x7f523800e960 tx=0x7f523800ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:10:12.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:12.088+0000 7f523fe00640 1 -- 172.21.15.16:0/1820855164 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5238017070 con 0x7f524806f920 2024-03-31T15:10:12.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:12.088+0000 7f523fe00640 1 -- 172.21.15.16:0/1820855164 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f523800ca00 con 0x7f524806f920 2024-03-31T15:10:12.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:12.088+0000 7f524e800640 1 -- 172.21.15.16:0/1820855164 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5248161000 con 0x7f524806f920 2024-03-31T15:10:12.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:12.088+0000 7f523fe00640 1 -- 172.21.15.16:0/1820855164 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f523800cdb0 con 0x7f524806f920 2024-03-31T15:10:12.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:12.088+0000 7f524e800640 1 -- 172.21.15.16:0/1820855164 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f52481615c0 con 0x7f524806f920 2024-03-31T15:10:12.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:12.092+0000 7f523fe00640 1 -- 172.21.15.16:0/1820855164 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f523801a070 con 0x7f524806f920 2024-03-31T15:10:12.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:12.092+0000 7f5251db4000 1 Processor -- start 2024-03-31T15:10:12.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:12.092+0000 7f5251db4000 1 -- start start 2024-03-31T15:10:12.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:12.092+0000 7f523fe00640 1 --2- 172.21.15.16:0/1820855164 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f5224068380 0x7f522406a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:12.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:12.092+0000 7f523fe00640 1 -- 172.21.15.16:0/1820855164 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 5795+0+0 (secure 0 0 0) 0x7f5238086ac0 con 0x7f524806f920 2024-03-31T15:10:12.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:12.092+0000 7f5247e00640 1 --2- 172.21.15.16:0/1820855164 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f5224068380 0x7f522406a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:12.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:12.092+0000 7f5251db4000 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56290717a2d0 0x56290717a6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:12.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:12.092+0000 7f5251db4000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56290717abf0 0x56290712d470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:12.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:12.092+0000 7f5251db4000 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x56290712d9b0 0x56290712dd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:12.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:12.092+0000 7f5251db4000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x562907016af0 con 0x56290717abf0 2024-03-31T15:10:12.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:12.092+0000 7f5251db4000 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x562906f7fb00 con 0x56290717a2d0 2024-03-31T15:10:12.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:12.092+0000 7f5251db4000 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x562906f1c940 con 0x56290712d9b0 2024-03-31T15:10:12.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:12.096+0000 7f523ea00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56290717abf0 0x56290712d470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:12.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:12.096+0000 7f5247400640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56290717a2d0 0x56290717a6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:12.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:12.096+0000 7f523ea00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56290717abf0 0x56290712d470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36674/0 (socket says 172.21.15.16:36674) 2024-03-31T15:10:12.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:12.096+0000 7f523ea00640 1 -- 172.21.15.16:0/134996588 learned_addr learned my addr 172.21.15.16:0/134996588 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:10:12.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:12.096+0000 7f5247400640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56290717a2d0 0x56290717a6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.16:51968/0 (socket says 172.21.15.16:51968) 2024-03-31T15:10:12.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:12.096+0000 7f5247e00640 1 --2- 172.21.15.16:0/134996588 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x56290712d9b0 0x56290712dd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:12.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:12.096+0000 7f523ea00640 1 -- 172.21.15.16:0/134996588 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x56290712d9b0 msgr2=0x56290712dd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:10:12.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:12.096+0000 7f523ea00640 1 --2- 172.21.15.16:0/134996588 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x56290712d9b0 0x56290712dd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:12.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:12.096+0000 7f5247e00640 1 --2- 172.21.15.16:0/1820855164 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f5224068380 0x7f522406a840 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f52400012f0 tx=0x7f5240002150 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:10:12.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:12.096+0000 7f523ea00640 1 -- 172.21.15.16:0/134996588 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56290717a2d0 msgr2=0x56290717a6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:10:12.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:12.096+0000 7f523ea00640 1 --2- 172.21.15.16:0/134996588 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56290717a2d0 0x56290717a6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:12.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:12.096+0000 7f523ea00640 1 -- 172.21.15.16:0/134996588 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562906f05380 con 0x56290717abf0 2024-03-31T15:10:12.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:12.096+0000 7f523ea00640 1 --2- 172.21.15.16:0/134996588 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56290717abf0 0x56290712d470 secure :-1 s=READY pgs=1177 cs=0 l=1 rev1=1 crypto rx=0x7f523001a0c0 tx=0x7f52300a6000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=caf5aec263756c66 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:10:12.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:12.096+0000 7f5237e00640 1 -- 172.21.15.16:0/134996588 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52300b0040 con 0x56290717abf0 2024-03-31T15:10:12.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:12.096+0000 7f5237e00640 1 -- 172.21.15.16:0/134996588 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f52300ab690 con 0x56290717abf0 2024-03-31T15:10:12.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:12.096+0000 7f5237e00640 1 -- 172.21.15.16:0/134996588 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52300aba40 con 0x56290717abf0 2024-03-31T15:10:12.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:12.096+0000 7f5251db4000 1 -- 172.21.15.16:0/134996588 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56290717abf0 msgr2=0x56290712d470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:10:12.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:12.096+0000 7f5251db4000 1 --2- 172.21.15.16:0/134996588 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56290717abf0 0x56290712d470 secure :-1 s=READY pgs=1177 cs=0 l=1 rev1=1 crypto rx=0x7f523001a0c0 tx=0x7f52300a6000 comp rx=0 tx=0).stop 2024-03-31T15:10:12.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:12.096+0000 7f5251db4000 1 -- 172.21.15.16:0/134996588 shutdown_connections 2024-03-31T15:10:12.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:12.096+0000 7f5251db4000 1 --2- 172.21.15.16:0/134996588 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x56290712d9b0 0x56290712dd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:12.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:12.096+0000 7f5251db4000 1 --2- 172.21.15.16:0/134996588 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56290717abf0 0x56290712d470 unknown :-1 s=CLOSED pgs=1177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:12.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:12.096+0000 7f5251db4000 1 --2- 172.21.15.16:0/134996588 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56290717a2d0 0x56290717a6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:12.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:12.096+0000 7f5251db4000 1 -- 172.21.15.16:0/134996588 >> 172.21.15.16:0/134996588 conn(0x56290715b070 msgr2=0x56290715d460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:10:12.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:12.096+0000 7f5251db4000 1 -- 172.21.15.16:0/134996588 shutdown_connections 2024-03-31T15:10:12.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:12.096+0000 7f5251db4000 1 -- 172.21.15.16:0/134996588 wait complete. 2024-03-31T15:10:12.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:12.096+0000 7f5251db4000 1 -- start start 2024-03-31T15:10:12.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:12.096+0000 7f5251db4000 1 Processor -- start 2024-03-31T15:10:12.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:12.096+0000 7f5251db4000 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56290712d9b0 0x5629071359e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:12.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:12.096+0000 7f5251db4000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56290717a2d0 0x56290717b1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:12.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:12.096+0000 7f5251db4000 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x56290717abf0 0x56290717b720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:12.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:12.096+0000 7f5251db4000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x562906f3d1e0 con 0x56290717a2d0 2024-03-31T15:10:12.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:12.096+0000 7f5251db4000 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x562906f7bab0 con 0x56290712d9b0 2024-03-31T15:10:12.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:12.096+0000 7f5251db4000 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x562906f4b150 con 0x56290717abf0 2024-03-31T15:10:12.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:12.096+0000 7f5251db4000 10 client.0 _ll_register_callbacks cb 0x56290711b370 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T15:10:12.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:12.096+0000 7f523ea00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56290717a2d0 0x56290717b1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:12.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:12.096+0000 7f523ea00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56290717a2d0 0x56290717b1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36688/0 (socket says 172.21.15.16:36688) 2024-03-31T15:10:12.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:12.096+0000 7f523ea00640 1 -- 172.21.15.16:0/3993803150 learned_addr learned my addr 172.21.15.16:0/3993803150 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:10:12.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:12.100+0000 7f5247e00640 1 --2- 172.21.15.16:0/3993803150 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x56290717abf0 0x56290717b720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:12.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:12.100+0000 7f5247400640 1 --2- 172.21.15.16:0/3993803150 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56290712d9b0 0x5629071359e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:12.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:12.100+0000 7f523ea00640 1 -- 172.21.15.16:0/3993803150 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x56290717abf0 msgr2=0x56290717b720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:10:12.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:12.100+0000 7f523ea00640 1 --2- 172.21.15.16:0/3993803150 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x56290717abf0 0x56290717b720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:12.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:12.100+0000 7f523ea00640 1 -- 172.21.15.16:0/3993803150 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56290712d9b0 msgr2=0x5629071359e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:10:12.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:12.100+0000 7f523ea00640 1 --2- 172.21.15.16:0/3993803150 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56290712d9b0 0x5629071359e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:12.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:12.100+0000 7f523ea00640 1 -- 172.21.15.16:0/3993803150 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562906f3bb70 con 0x56290717a2d0 2024-03-31T15:10:12.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:12.100+0000 7f523ea00640 1 --2- 172.21.15.16:0/3993803150 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56290717a2d0 0x56290717b1e0 secure :-1 s=READY pgs=1178 cs=0 l=1 rev1=1 crypto rx=0x7f5230008010 tx=0x7f52300046d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5de40b80d5cfbd32 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:10:12.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:12.100+0000 7f5234c00640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-31T15:10:12.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:12.100+0000 7f5234c00640 1 -- 172.21.15.16:0/3993803150 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52300b0040 con 0x56290717a2d0 2024-03-31T15:10:12.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:12.100+0000 7f5234c00640 1 -- 172.21.15.16:0/3993803150 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 5795+0+0 (secure 0 0 0) 0x7f52300abcd0 con 0x56290717a2d0 2024-03-31T15:10:12.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:12.100+0000 7f5251db4000 1 -- 172.21.15.16:0/3993803150 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f52300ab690 con 0x56290717a2d0 2024-03-31T15:10:12.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:12.100+0000 7f5234c00640 1 -- 172.21.15.16:0/3993803150 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=67) ==== 24+0+0 (secure 0 0 0) 0x7f52300ce050 con 0x56290717a2d0 2024-03-31T15:10:12.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:12.100+0000 7f5251db4000 10 client.7872 fetch_fsmap learned FSMap version 67 2024-03-31T15:10:12.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:12.100+0000 7f5251db4000 1 -- 172.21.15.16:0/3993803150 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=67,monmap=2+}) -- 0x562906f1cb90 con 0x56290717a2d0 2024-03-31T15:10:12.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:12.100+0000 7f5234c00640 1 -- 172.21.15.16:0/3993803150 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f52300ab690 con 0x56290717a2d0 2024-03-31T15:10:12.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:12.100+0000 7f5234c00640 1 -- 172.21.15.16:0/3993803150 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 67) ==== 3494+0+0 (secure 0 0 0) 0x7f52300d1020 con 0x56290717a2d0 2024-03-31T15:10:12.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:12.100+0000 7f5251db4000 10 client.7872 fetch_fsmap finished waiting for FSMap version 67 2024-03-31T15:10:12.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:12.100+0000 7f5251db4000 10 client.7872 resolve_mds: resolved 6:0 to role '6:0' aka daemon mds.c 2024-03-31T15:10:12.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:12.100+0000 7f5251db4000 20 client.7872 populate_metadata read hostname 'smithi016' 2024-03-31T15:10:12.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:12.100+0000 7f5251db4000 1 --2- 172.21.15.16:0/3993803150 >> [v2:172.21.15.16:6832/818074038,v1:172.21.15.16:6834/818074038] conn(0x56290717c490 0x56290718be40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:12.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:12.100+0000 7f5251db4000 4 client.7872 mds_command: new command op to 7749 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-31T15:10:12.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:12.100+0000 7f5251db4000 1 -- 172.21.15.16:0/3993803150 --> [v2:172.21.15.16:6832/818074038,v1:172.21.15.16:6834/818074038] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562906f1add0 con 0x56290717c490 2024-03-31T15:10:12.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:12.100+0000 7f5247400640 1 --2- 172.21.15.16:0/3993803150 >> [v2:172.21.15.16:6832/818074038,v1:172.21.15.16:6834/818074038] conn(0x56290717c490 0x56290718be40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:12.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:12.100+0000 7f5247400640 1 --2- 172.21.15.16:0/3993803150 >> [v2:172.21.15.16:6832/818074038,v1:172.21.15.16:6834/818074038] conn(0x56290717c490 0x56290718be40 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=895de94ac388e77b server_cookie=97f2d23cadc8640a in_seq=0 out_seq=0 2024-03-31T15:10:12.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:12.100+0000 7f5234c00640 10 client.7872 ms_handle_connect on v2:172.21.15.16:6832/818074038 2024-03-31T15:10:12.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:12.100+0000 7f5234c00640 1 -- 172.21.15.16:0/3993803150 <== mds.0 v2:172.21.15.16:6832/818074038 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x562906f1add0 con 0x56290717c490 2024-03-31T15:10:12.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:12.100+0000 7f5234c00640 10 client.7872 handle_command_reply: tid=0 2024-03-31T15:10:12.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:12.100+0000 7f5251db4000 1 client.7872 shutdown 2024-03-31T15:10:12.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:12.100+0000 7f5251db4000 1 -- 172.21.15.16:0/3993803150 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56290717a2d0 msgr2=0x56290717b1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:10:12.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:12.100+0000 7f5251db4000 1 --2- 172.21.15.16:0/3993803150 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56290717a2d0 0x56290717b1e0 secure :-1 s=READY pgs=1178 cs=0 l=1 rev1=1 crypto rx=0x7f5230008010 tx=0x7f52300046d0 comp rx=0 tx=0).stop 2024-03-31T15:10:12.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:12.100+0000 7f5251db4000 1 -- 172.21.15.16:0/3993803150 shutdown_connections 2024-03-31T15:10:12.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:12.100+0000 7f5251db4000 1 --2- 172.21.15.16:0/3993803150 >> [v2:172.21.15.16:6832/818074038,v1:172.21.15.16:6834/818074038] conn(0x56290717c490 0x56290718be40 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:12.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:12.100+0000 7f5251db4000 1 --2- 172.21.15.16:0/3993803150 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x56290717abf0 0x56290717b720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:12.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:12.100+0000 7f5251db4000 1 --2- 172.21.15.16:0/3993803150 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56290717a2d0 0x56290717b1e0 unknown :-1 s=CLOSED pgs=1178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:12.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:12.100+0000 7f5251db4000 1 --2- 172.21.15.16:0/3993803150 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56290712d9b0 0x5629071359e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:12.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:12.100+0000 7f5251db4000 1 -- 172.21.15.16:0/3993803150 >> 172.21.15.16:0/3993803150 conn(0x56290715b070 msgr2=0x56290715d460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:10:12.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:12.100+0000 7f5234c00640 0 client.7872 ms_handle_reset on v2:172.21.15.16:6832/818074038 2024-03-31T15:10:12.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:12.100+0000 7f5251db4000 1 -- 172.21.15.16:0/3993803150 shutdown_connections 2024-03-31T15:10:12.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:12.100+0000 7f5251db4000 1 -- 172.21.15.16:0/3993803150 wait complete. 2024-03-31T15:10:12.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:12.100+0000 7f5251db4000 20 client.7872 trim_cache size 0 max 16384 2024-03-31T15:10:12.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:12.120+0000 7f5251db4000 1 Processor -- start 2024-03-31T15:10:12.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:12.120+0000 7f5251db4000 1 -- start start 2024-03-31T15:10:12.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:12.120+0000 7f5251db4000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56290715b070 0x5629071914a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:12.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:12.120+0000 7f5251db4000 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x56290712d9b0 0x562907179390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:12.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:12.120+0000 7f5251db4000 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56290717a2d0 0x5629071798d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:12.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:12.120+0000 7f5251db4000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f52482000d0 con 0x56290715b070 2024-03-31T15:10:12.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:12.120+0000 7f5251db4000 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x562906f1a940 con 0x56290717a2d0 2024-03-31T15:10:12.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:12.120+0000 7f5251db4000 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x562906f36750 con 0x56290712d9b0 2024-03-31T15:10:12.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:12.120+0000 7f5247400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56290715b070 0x5629071914a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:12.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:12.120+0000 7f5247400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56290715b070 0x5629071914a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36692/0 (socket says 172.21.15.16:36692) 2024-03-31T15:10:12.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:12.120+0000 7f5247400640 1 -- 172.21.15.16:0/2433270076 learned_addr learned my addr 172.21.15.16:0/2433270076 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:10:12.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:12.120+0000 7f523ea00640 1 --2- 172.21.15.16:0/2433270076 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x56290712d9b0 0x562907179390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:12.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:12.120+0000 7f5247e00640 1 --2- 172.21.15.16:0/2433270076 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56290717a2d0 0x5629071798d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:12.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:12.120+0000 7f5247400640 1 -- 172.21.15.16:0/2433270076 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x56290712d9b0 msgr2=0x562907179390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:10:12.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:12.120+0000 7f5247400640 1 --2- 172.21.15.16:0/2433270076 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x56290712d9b0 0x562907179390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:12.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:12.120+0000 7f5247400640 1 -- 172.21.15.16:0/2433270076 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56290717a2d0 msgr2=0x5629071798d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:10:12.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:12.120+0000 7f5247400640 1 --2- 172.21.15.16:0/2433270076 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56290717a2d0 0x5629071798d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:12.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:12.120+0000 7f5247400640 1 -- 172.21.15.16:0/2433270076 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56290717aa20 con 0x56290715b070 2024-03-31T15:10:12.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:12.120+0000 7f523ea00640 1 --2- 172.21.15.16:0/2433270076 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x56290712d9b0 0x562907179390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:10:12.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:12.120+0000 7f5247400640 1 --2- 172.21.15.16:0/2433270076 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56290715b070 0x5629071914a0 secure :-1 s=READY pgs=1179 cs=0 l=1 rev1=1 crypto rx=0x7f5238056810 tx=0x7f523801e040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27f29729af7a9ac5 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:10:12.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:12.120+0000 7f5237400640 1 -- 172.21.15.16:0/2433270076 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5238021040 con 0x56290715b070 2024-03-31T15:10:12.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:12.120+0000 7f5237400640 1 -- 172.21.15.16:0/2433270076 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5238082070 con 0x56290715b070 2024-03-31T15:10:12.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:12.120+0000 7f5237400640 1 -- 172.21.15.16:0/2433270076 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f523805d0b0 con 0x56290715b070 2024-03-31T15:10:12.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:12.120+0000 7f5251db4000 1 -- 172.21.15.16:0/2433270076 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56290715b070 msgr2=0x5629071914a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:10:12.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:12.120+0000 7f5251db4000 1 --2- 172.21.15.16:0/2433270076 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56290715b070 0x5629071914a0 secure :-1 s=READY pgs=1179 cs=0 l=1 rev1=1 crypto rx=0x7f5238056810 tx=0x7f523801e040 comp rx=0 tx=0).stop 2024-03-31T15:10:12.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:12.124+0000 7f5251db4000 1 -- 172.21.15.16:0/2433270076 shutdown_connections 2024-03-31T15:10:12.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:12.124+0000 7f5251db4000 1 --2- 172.21.15.16:0/2433270076 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56290717a2d0 0x5629071798d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:12.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:12.124+0000 7f5251db4000 1 --2- 172.21.15.16:0/2433270076 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x56290712d9b0 0x562907179390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:12.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:12.124+0000 7f5251db4000 1 --2- 172.21.15.16:0/2433270076 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56290715b070 0x5629071914a0 unknown :-1 s=CLOSED pgs=1179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:12.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:12.124+0000 7f5251db4000 1 -- 172.21.15.16:0/2433270076 >> 172.21.15.16:0/2433270076 conn(0x562907135c40 msgr2=0x56290712cad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:10:12.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:12.124+0000 7f5251db4000 1 -- 172.21.15.16:0/2433270076 shutdown_connections 2024-03-31T15:10:12.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:12.124+0000 7f5251db4000 1 -- 172.21.15.16:0/2433270076 wait complete. 2024-03-31T15:10:12.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:12.124+0000 7f5251db4000 1 -- start start 2024-03-31T15:10:12.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:12.124+0000 7f5251db4000 1 Processor -- start 2024-03-31T15:10:12.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:12.124+0000 7f5251db4000 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56290712d9b0 0x5629071912a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:12.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:12.124+0000 7f5251db4000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56290715b070 0x5629071ce260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:12.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:12.124+0000 7f5251db4000 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x56290717a2d0 0x5629071ce7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:12.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:12.124+0000 7f5251db4000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x56290704e420 con 0x56290715b070 2024-03-31T15:10:12.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:12.124+0000 7f5251db4000 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x56290703a250 con 0x56290712d9b0 2024-03-31T15:10:12.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:12.124+0000 7f5251db4000 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x562906f7ee10 con 0x56290717a2d0 2024-03-31T15:10:12.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:12.124+0000 7f5251db4000 10 client.0 _ll_register_callbacks cb 0x7f5238009710 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T15:10:12.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:12.124+0000 7f523ea00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56290715b070 0x5629071ce260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:12.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:12.124+0000 7f523ea00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56290715b070 0x5629071ce260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36700/0 (socket says 172.21.15.16:36700) 2024-03-31T15:10:12.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:12.124+0000 7f523ea00640 1 -- 172.21.15.16:0/397367112 learned_addr learned my addr 172.21.15.16:0/397367112 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:10:12.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:12.124+0000 7f5247e00640 1 --2- 172.21.15.16:0/397367112 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x56290717a2d0 0x5629071ce7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:12.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:12.124+0000 7f5247400640 1 --2- 172.21.15.16:0/397367112 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56290712d9b0 0x5629071912a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:12.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:12.124+0000 7f523ea00640 1 -- 172.21.15.16:0/397367112 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x56290717a2d0 msgr2=0x5629071ce7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:10:12.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:12.124+0000 7f523ea00640 1 --2- 172.21.15.16:0/397367112 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x56290717a2d0 0x5629071ce7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:12.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:12.124+0000 7f523ea00640 1 -- 172.21.15.16:0/397367112 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56290712d9b0 msgr2=0x5629071912a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:10:12.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:12.124+0000 7f523ea00640 1 --2- 172.21.15.16:0/397367112 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56290712d9b0 0x5629071912a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:12.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:12.124+0000 7f523ea00640 1 -- 172.21.15.16:0/397367112 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562907000140 con 0x56290715b070 2024-03-31T15:10:12.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:12.124+0000 7f523ea00640 1 --2- 172.21.15.16:0/397367112 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56290715b070 0x5629071ce260 secure :-1 s=READY pgs=1180 cs=0 l=1 rev1=1 crypto rx=0x7f523001a0c0 tx=0x7f523001acd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b9977602a72e716e server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:10:12.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:12.124+0000 7f5234c00640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-31T15:10:12.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:12.124+0000 7f5234c00640 1 -- 172.21.15.16:0/397367112 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52300b0040 con 0x56290715b070 2024-03-31T15:10:12.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:12.124+0000 7f5234c00640 1 -- 172.21.15.16:0/397367112 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 5795+0+0 (secure 0 0 0) 0x7f52300d3b00 con 0x56290715b070 2024-03-31T15:10:12.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:12.124+0000 7f5251db4000 1 -- 172.21.15.16:0/397367112 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5238082070 con 0x56290715b070 2024-03-31T15:10:12.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:12.124+0000 7f5234c00640 1 -- 172.21.15.16:0/397367112 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=67) ==== 24+0+0 (secure 0 0 0) 0x7f52300d3d00 con 0x56290715b070 2024-03-31T15:10:12.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:12.124+0000 7f5251db4000 10 client.7878 fetch_fsmap learned FSMap version 67 2024-03-31T15:10:12.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:12.124+0000 7f5251db4000 1 -- 172.21.15.16:0/397367112 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=67,monmap=2+}) -- 0x562906f0e1e0 con 0x56290715b070 2024-03-31T15:10:12.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:12.124+0000 7f5234c00640 1 -- 172.21.15.16:0/397367112 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5238082070 con 0x56290715b070 2024-03-31T15:10:12.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:12.124+0000 7f5234c00640 1 -- 172.21.15.16:0/397367112 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 67) ==== 3494+0+0 (secure 0 0 0) 0x7f52300d1030 con 0x56290715b070 2024-03-31T15:10:12.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:12.124+0000 7f5251db4000 10 client.7878 fetch_fsmap finished waiting for FSMap version 67 2024-03-31T15:10:12.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:12.124+0000 7f5251db4000 10 client.7878 resolve_mds: resolved 6:0 to role '6:0' aka daemon mds.c 2024-03-31T15:10:12.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:12.124+0000 7f5251db4000 20 client.7878 populate_metadata read hostname 'smithi016' 2024-03-31T15:10:12.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:12.124+0000 7f5251db4000 1 --2- 172.21.15.16:0/397367112 >> [v2:172.21.15.16:6832/818074038,v1:172.21.15.16:6834/818074038] conn(0x56290717abf0 0x5629071cece0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:12.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:12.124+0000 7f5251db4000 4 client.7878 mds_command: new command op to 7749 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-31T15:10:12.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:12.124+0000 7f5251db4000 1 -- 172.21.15.16:0/397367112 --> [v2:172.21.15.16:6832/818074038,v1:172.21.15.16:6834/818074038] -- command(tid 0: {"prefix": "scrub status"}) -- 0x562906f8d040 con 0x56290717abf0 2024-03-31T15:10:12.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:12.128+0000 7f5247400640 1 --2- 172.21.15.16:0/397367112 >> [v2:172.21.15.16:6832/818074038,v1:172.21.15.16:6834/818074038] conn(0x56290717abf0 0x5629071cece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:12.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:12.128+0000 7f5247400640 1 --2- 172.21.15.16:0/397367112 >> [v2:172.21.15.16:6832/818074038,v1:172.21.15.16:6834/818074038] conn(0x56290717abf0 0x5629071cece0 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3fe7eebe2837852 server_cookie=e10336af2fed1487 in_seq=0 out_seq=0 2024-03-31T15:10:12.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:12.128+0000 7f5234c00640 10 client.7878 ms_handle_connect on v2:172.21.15.16:6832/818074038 2024-03-31T15:10:12.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:12.128+0000 7f5234c00640 1 -- 172.21.15.16:0/397367112 <== mds.0 v2:172.21.15.16:6832/818074038 1 ==== command_reply(tid 0: 0 ) ==== 8+0+256 (crc 0 0 0) 0x562906f8d040 con 0x56290717abf0 2024-03-31T15:10:12.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:12.128+0000 7f5234c00640 10 client.7878 handle_command_reply: tid=0 2024-03-31T15:10:12.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:12.128+0000 7f5251db4000 1 client.7878 shutdown 2024-03-31T15:10:12.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:12.128+0000 7f5251db4000 1 -- 172.21.15.16:0/397367112 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56290715b070 msgr2=0x5629071ce260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:10:12.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:12.128+0000 7f5251db4000 1 --2- 172.21.15.16:0/397367112 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56290715b070 0x5629071ce260 secure :-1 s=READY pgs=1180 cs=0 l=1 rev1=1 crypto rx=0x7f523001a0c0 tx=0x7f523001acd0 comp rx=0 tx=0).stop 2024-03-31T15:10:12.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:12.128+0000 7f5251db4000 1 -- 172.21.15.16:0/397367112 shutdown_connections 2024-03-31T15:10:12.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:12.128+0000 7f5251db4000 1 --2- 172.21.15.16:0/397367112 >> [v2:172.21.15.16:6832/818074038,v1:172.21.15.16:6834/818074038] conn(0x56290717abf0 0x5629071cece0 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:12.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:12.128+0000 7f5251db4000 1 --2- 172.21.15.16:0/397367112 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x56290717a2d0 0x5629071ce7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:12.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:12.128+0000 7f5251db4000 1 --2- 172.21.15.16:0/397367112 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56290715b070 0x5629071ce260 unknown :-1 s=CLOSED pgs=1180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:12.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:12.128+0000 7f5251db4000 1 --2- 172.21.15.16:0/397367112 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56290712d9b0 0x5629071912a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:12.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:12.128+0000 7f5251db4000 1 -- 172.21.15.16:0/397367112 >> 172.21.15.16:0/397367112 conn(0x562907135c40 msgr2=0x562907178380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:10:12.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:12.128+0000 7f5234c00640 0 client.7878 ms_handle_reset on v2:172.21.15.16:6832/818074038 2024-03-31T15:10:12.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:12.128+0000 7f5251db4000 1 -- 172.21.15.16:0/397367112 shutdown_connections 2024-03-31T15:10:12.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:12.128+0000 7f5251db4000 1 -- 172.21.15.16:0/397367112 wait complete. 2024-03-31T15:10:12.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:12.128+0000 7f5251db4000 20 client.7878 trim_cache size 0 max 16384 2024-03-31T15:10:12.123 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-03-31T15:10:12.124 INFO:teuthology.orchestra.run.smithi016.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-31T15:10:12.124 INFO:teuthology.orchestra.run.smithi016.stdout: "scrubs": { 2024-03-31T15:10:12.124 INFO:teuthology.orchestra.run.smithi016.stdout: "73e2fea1-8e96-428d-b985-a1c2091e8679": { 2024-03-31T15:10:12.124 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/dir", 2024-03-31T15:10:12.124 INFO:teuthology.orchestra.run.smithi016.stdout: "tag": "73e2fea1-8e96-428d-b985-a1c2091e8679", 2024-03-31T15:10:12.124 INFO:teuthology.orchestra.run.smithi016.stdout: "options": "recursive" 2024-03-31T15:10:12.124 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-31T15:10:12.124 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-31T15:10:12.124 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-03-31T15:10:12.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:12.128+0000 7f524e800640 1 -- 172.21.15.16:0/1820855164 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f5224068380 msgr2=0x7f522406a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:10:12.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:12.128+0000 7f524e800640 1 --2- 172.21.15.16:0/1820855164 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f5224068380 0x7f522406a840 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f52400012f0 tx=0x7f5240002150 comp rx=0 tx=0).stop 2024-03-31T15:10:12.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:12.128+0000 7f524e800640 1 -- 172.21.15.16:0/1820855164 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f524806f920 msgr2=0x7f52481543a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:10:12.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:12.128+0000 7f524e800640 1 --2- 172.21.15.16:0/1820855164 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f524806f920 0x7f52481543a0 secure :-1 s=READY pgs=1176 cs=0 l=1 rev1=1 crypto rx=0x7f523800e960 tx=0x7f523800ee20 comp rx=0 tx=0).stop 2024-03-31T15:10:12.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:12.128+0000 7f524e800640 1 -- 172.21.15.16:0/1820855164 shutdown_connections 2024-03-31T15:10:12.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:12.128+0000 7f524e800640 1 --2- 172.21.15.16:0/1820855164 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f5224068380 0x7f522406a840 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:12.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:12.128+0000 7f524e800640 1 --2- 172.21.15.16:0/1820855164 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f52480d0a60 0x7f524815ecc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:12.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:12.128+0000 7f524e800640 1 --2- 172.21.15.16:0/1820855164 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f52480cef80 0x7f524813ecd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:12.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:12.128+0000 7f524e800640 1 --2- 172.21.15.16:0/1820855164 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f524806f920 0x7f52481543a0 unknown :-1 s=CLOSED pgs=1176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:12.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:12.128+0000 7f524e800640 1 -- 172.21.15.16:0/1820855164 >> 172.21.15.16:0/1820855164 conn(0x7f5248067930 msgr2=0x7f5248070e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:10:12.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:12.128+0000 7f524e800640 1 -- 172.21.15.16:0/1820855164 shutdown_connections 2024-03-31T15:10:12.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:12.128+0000 7f524e800640 1 -- 172.21.15.16:0/1820855164 wait complete. 2024-03-31T15:10:12.139 INFO:tasks.cephfs.filesystem:scrub status for tag:73e2fea1-8e96-428d-b985-a1c2091e8679 - {'path': '/dir', 'tag': '73e2fea1-8e96-428d-b985-a1c2091e8679', 'options': 'recursive'} 2024-03-31T15:10:15.382 INFO:tasks.ceph.mon.a.smithi016.stderr:2024-03-31T15:10:15.388+0000 7f7dcee00640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 MDSs report damaged metadata (MDS_DAMAGE) 2024-03-31T15:10:37.402 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-31T15:10:37.405 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-31T15:10:42.139 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:10:42.140 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.6:0 scrub status 2024-03-31T15:10:42.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.288+0000 7f9717800640 1 Processor -- start 2024-03-31T15:10:42.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.288+0000 7f9717800640 1 -- start start 2024-03-31T15:10:42.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.292+0000 7f9717800640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f97101567e0 0x7f9710072c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:42.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.292+0000 7f9717800640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f971006dbe0 0x7f9710071550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:42.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.292+0000 7f9717800640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f971015c7e0 con 0x7f97100731c0 2024-03-31T15:10:42.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.292+0000 7f9717800640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f97100736b0 con 0x7f971006dbe0 2024-03-31T15:10:42.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.292+0000 7f9717800640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f971015caa0 con 0x7f97101567e0 2024-03-31T15:10:42.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.292+0000 7f9715800640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f971006dbe0 0x7f9710071550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:42.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.292+0000 7f9715800640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f971006dbe0 0x7f9710071550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41006/0 (socket says 172.21.15.16:41006) 2024-03-31T15:10:42.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.292+0000 7f9715800640 1 -- 172.21.15.16:0/239790954 learned_addr learned my addr 172.21.15.16:0/239790954 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:10:42.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.292+0000 7f9714e00640 1 --2- 172.21.15.16:0/239790954 >> v2:172.21.15.89:3300/0 conn(0x7f97101567e0 0x7f9710072c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:42.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.292+0000 7f9715800640 1 -- 172.21.15.16:0/239790954 >> v2:172.21.15.89:3300/0 conn(0x7f97101567e0 msgr2=0x7f9710072c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:10:42.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.292+0000 7f9715800640 1 --2- 172.21.15.16:0/239790954 >> v2:172.21.15.89:3300/0 conn(0x7f97101567e0 0x7f9710072c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:42.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.292+0000 7f9715800640 1 -- 172.21.15.16:0/239790954 >> v1:172.21.15.89:6789/0 conn(0x7f97100731c0 legacy=0x7f971006d4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:10:42.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.292+0000 7f9715800640 1 -- 172.21.15.16:0/239790954 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f971015cd80 con 0x7f971006dbe0 2024-03-31T15:10:42.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.292+0000 7f9714e00640 1 --2- 172.21.15.16:0/239790954 >> v2:172.21.15.89:3300/0 conn(0x7f97101567e0 0x7f9710072c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:10:42.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.296+0000 7f9715800640 1 --2- 172.21.15.16:0/239790954 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f971006dbe0 0x7f9710071550 secure :-1 s=READY pgs=1181 cs=0 l=1 rev1=1 crypto rx=0x7f97100cffe0 tx=0x7f970801c9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1dd729ceba976319 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:10:42.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.296+0000 7f970f400640 1 -- 172.21.15.16:0/239790954 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97080aa450 con 0x7f971006dbe0 2024-03-31T15:10:42.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.296+0000 7f970f400640 1 -- 172.21.15.16:0/239790954 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f97080af070 con 0x7f971006dbe0 2024-03-31T15:10:42.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.296+0000 7f970f400640 1 -- 172.21.15.16:0/239790954 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97080aa810 con 0x7f971006dbe0 2024-03-31T15:10:42.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.296+0000 7f9717800640 1 -- 172.21.15.16:0/239790954 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f971006dbe0 msgr2=0x7f9710071550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:10:42.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.296+0000 7f9717800640 1 --2- 172.21.15.16:0/239790954 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f971006dbe0 0x7f9710071550 secure :-1 s=READY pgs=1181 cs=0 l=1 rev1=1 crypto rx=0x7f97100cffe0 tx=0x7f970801c9b0 comp rx=0 tx=0).stop 2024-03-31T15:10:42.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.296+0000 7f9717800640 1 -- 172.21.15.16:0/239790954 shutdown_connections 2024-03-31T15:10:42.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.296+0000 7f9717800640 1 --2- 172.21.15.16:0/239790954 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f971006dbe0 0x7f9710071550 unknown :-1 s=CLOSED pgs=1181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:42.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.296+0000 7f9717800640 1 --2- 172.21.15.16:0/239790954 >> v2:172.21.15.89:3300/0 conn(0x7f97101567e0 0x7f9710072c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:42.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.296+0000 7f9717800640 1 -- 172.21.15.16:0/239790954 >> 172.21.15.16:0/239790954 conn(0x7f971006ee40 msgr2=0x7f971006b620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:10:42.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.296+0000 7f9717800640 1 -- 172.21.15.16:0/239790954 shutdown_connections 2024-03-31T15:10:42.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.296+0000 7f9717800640 1 -- 172.21.15.16:0/239790954 wait complete. 2024-03-31T15:10:42.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.296+0000 7f9717800640 1 Processor -- start 2024-03-31T15:10:42.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.296+0000 7f9717800640 1 -- start start 2024-03-31T15:10:42.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.300+0000 7f9717800640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f971006dbe0 0x7f9710145490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:42.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.300+0000 7f9717800640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f97100731c0 0x7f971014ddc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:42.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.300+0000 7f9717800640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f97101567e0 0x7f971014e300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:42.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.300+0000 7f9717800640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f9710169170 con 0x7f97100731c0 2024-03-31T15:10:42.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.300+0000 7f9717800640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f971015d7b0 con 0x7f971006dbe0 2024-03-31T15:10:42.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.300+0000 7f9717800640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f97101692f0 con 0x7f97101567e0 2024-03-31T15:10:42.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.300+0000 7f970fe00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f97100731c0 0x7f971014ddc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:42.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.300+0000 7f970fe00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f97100731c0 0x7f971014ddc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41014/0 (socket says 172.21.15.16:41014) 2024-03-31T15:10:42.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.300+0000 7f970fe00640 1 -- 172.21.15.16:0/3222452056 learned_addr learned my addr 172.21.15.16:0/3222452056 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:10:42.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.300+0000 7f9714e00640 1 --2- 172.21.15.16:0/3222452056 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f971006dbe0 0x7f9710145490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:42.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.300+0000 7f9715800640 1 --2- 172.21.15.16:0/3222452056 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f97101567e0 0x7f971014e300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:42.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.300+0000 7f970fe00640 1 -- 172.21.15.16:0/3222452056 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f97101567e0 msgr2=0x7f971014e300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:10:42.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.300+0000 7f970fe00640 1 --2- 172.21.15.16:0/3222452056 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f97101567e0 0x7f971014e300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:42.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.300+0000 7f970fe00640 1 -- 172.21.15.16:0/3222452056 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f971006dbe0 msgr2=0x7f9710145490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:10:42.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.300+0000 7f970fe00640 1 --2- 172.21.15.16:0/3222452056 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f971006dbe0 0x7f9710145490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:42.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.300+0000 7f970fe00640 1 -- 172.21.15.16:0/3222452056 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97101554d0 con 0x7f97100731c0 2024-03-31T15:10:42.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.300+0000 7f970fe00640 1 --2- 172.21.15.16:0/3222452056 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f97100731c0 0x7f971014ddc0 secure :-1 s=READY pgs=1182 cs=0 l=1 rev1=1 crypto rx=0x7f96fc00c410 tx=0x7f96fc00c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:10:42.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.300+0000 7f970cc00640 1 -- 172.21.15.16:0/3222452056 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96fc011790 con 0x7f97100731c0 2024-03-31T15:10:42.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.300+0000 7f970cc00640 1 -- 172.21.15.16:0/3222452056 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f96fc017070 con 0x7f97100731c0 2024-03-31T15:10:42.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.300+0000 7f970cc00640 1 -- 172.21.15.16:0/3222452056 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96fc011bb0 con 0x7f97100731c0 2024-03-31T15:10:42.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.300+0000 7f9717800640 1 -- 172.21.15.16:0/3222452056 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f97101557b0 con 0x7f97100731c0 2024-03-31T15:10:42.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.300+0000 7f9717800640 1 -- 172.21.15.16:0/3222452056 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9710155d70 con 0x7f97100731c0 2024-03-31T15:10:42.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.304+0000 7f970cc00640 1 -- 172.21.15.16:0/3222452056 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f96fc01c460 con 0x7f97100731c0 2024-03-31T15:10:42.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.304+0000 7f970cc00640 1 --2- 172.21.15.16:0/3222452056 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f96f0068380 0x7f96f006a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:42.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.304+0000 7f970cc00640 1 -- 172.21.15.16:0/3222452056 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 5824+0+0 (secure 0 0 0) 0x7f96fc0865a0 con 0x7f97100731c0 2024-03-31T15:10:42.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.304+0000 7f9714e00640 1 --2- 172.21.15.16:0/3222452056 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f96f0068380 0x7f96f006a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:42.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.304+0000 7f9714e00640 1 --2- 172.21.15.16:0/3222452056 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f96f0068380 0x7f96f006a840 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f9704006fd0 tx=0x7f9704008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:10:42.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.304+0000 7f971abf9000 1 Processor -- start 2024-03-31T15:10:42.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.304+0000 7f971abf9000 1 -- start start 2024-03-31T15:10:42.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.308+0000 7f971abf9000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564d2de0f520 0x564d2de0f900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:42.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.308+0000 7f971abf9000 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564d2de0fe40 0x564d2ddc2880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:42.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.308+0000 7f971abf9000 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x564d2ddc2dc0 0x564d2ddc31a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:42.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.308+0000 7f971abf9000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x564d2dca7af0 con 0x564d2de0f520 2024-03-31T15:10:42.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.308+0000 7f971abf9000 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x564d2dc10b00 con 0x564d2de0fe40 2024-03-31T15:10:42.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.308+0000 7f971abf9000 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x564d2dbad940 con 0x564d2ddc2dc0 2024-03-31T15:10:42.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.308+0000 7f9714e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564d2de0f520 0x564d2de0f900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:42.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.308+0000 7f9714e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564d2de0f520 0x564d2de0f900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41026/0 (socket says 172.21.15.16:41026) 2024-03-31T15:10:42.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.308+0000 7f9714e00640 1 -- 172.21.15.16:0/4247363977 learned_addr learned my addr 172.21.15.16:0/4247363977 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:10:42.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.308+0000 7f9714e00640 1 -- 172.21.15.16:0/4247363977 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x564d2ddc2dc0 msgr2=0x564d2ddc31a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:10:42.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.308+0000 7f970fe00640 1 --2- 172.21.15.16:0/4247363977 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564d2de0fe40 0x564d2ddc2880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:42.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.308+0000 7f9715800640 1 --2- 172.21.15.16:0/4247363977 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x564d2ddc2dc0 0x564d2ddc31a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:42.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.308+0000 7f9714e00640 1 --2- 172.21.15.16:0/4247363977 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x564d2ddc2dc0 0x564d2ddc31a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:42.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.308+0000 7f9714e00640 1 -- 172.21.15.16:0/4247363977 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564d2de0fe40 msgr2=0x564d2ddc2880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:10:42.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.308+0000 7f9714e00640 1 --2- 172.21.15.16:0/4247363977 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564d2de0fe40 0x564d2ddc2880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:42.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.308+0000 7f9714e00640 1 -- 172.21.15.16:0/4247363977 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564d2db96380 con 0x564d2de0f520 2024-03-31T15:10:42.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.308+0000 7f970fe00640 1 --2- 172.21.15.16:0/4247363977 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564d2de0fe40 0x564d2ddc2880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:10:42.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.308+0000 7f9715800640 1 --2- 172.21.15.16:0/4247363977 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x564d2ddc2dc0 0x564d2ddc31a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:10:42.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.308+0000 7f9714e00640 1 --2- 172.21.15.16:0/4247363977 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564d2de0f520 0x564d2de0f900 secure :-1 s=READY pgs=1183 cs=0 l=1 rev1=1 crypto rx=0x7f970400fd60 tx=0x7f970400af90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec8ae37df2930131 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:10:42.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.308+0000 7f9701600640 1 -- 172.21.15.16:0/4247363977 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9704013620 con 0x564d2de0f520 2024-03-31T15:10:42.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.308+0000 7f9701600640 1 -- 172.21.15.16:0/4247363977 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9704019070 con 0x564d2de0f520 2024-03-31T15:10:42.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.308+0000 7f9701600640 1 -- 172.21.15.16:0/4247363977 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97040139c0 con 0x564d2de0f520 2024-03-31T15:10:42.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.308+0000 7f971abf9000 1 -- 172.21.15.16:0/4247363977 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564d2de0f520 msgr2=0x564d2de0f900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:10:42.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.308+0000 7f971abf9000 1 --2- 172.21.15.16:0/4247363977 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564d2de0f520 0x564d2de0f900 secure :-1 s=READY pgs=1183 cs=0 l=1 rev1=1 crypto rx=0x7f970400fd60 tx=0x7f970400af90 comp rx=0 tx=0).stop 2024-03-31T15:10:42.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.308+0000 7f971abf9000 1 -- 172.21.15.16:0/4247363977 shutdown_connections 2024-03-31T15:10:42.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.308+0000 7f971abf9000 1 --2- 172.21.15.16:0/4247363977 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x564d2ddc2dc0 0x564d2ddc31a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:42.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.308+0000 7f971abf9000 1 --2- 172.21.15.16:0/4247363977 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564d2de0fe40 0x564d2ddc2880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:42.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.308+0000 7f971abf9000 1 --2- 172.21.15.16:0/4247363977 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564d2de0f520 0x564d2de0f900 unknown :-1 s=CLOSED pgs=1183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:42.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.308+0000 7f971abf9000 1 -- 172.21.15.16:0/4247363977 >> 172.21.15.16:0/4247363977 conn(0x564d2ddec070 msgr2=0x564d2ddee460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:10:42.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.308+0000 7f971abf9000 1 -- 172.21.15.16:0/4247363977 shutdown_connections 2024-03-31T15:10:42.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.308+0000 7f971abf9000 1 -- 172.21.15.16:0/4247363977 wait complete. 2024-03-31T15:10:42.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.312+0000 7f971abf9000 1 -- start start 2024-03-31T15:10:42.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.312+0000 7f971abf9000 1 Processor -- start 2024-03-31T15:10:42.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.312+0000 7f971abf9000 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x564d2ddc2dc0 0x564d2de16670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:42.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.312+0000 7f971abf9000 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564d2de0f520 0x564d2de16bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:42.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.312+0000 7f971abf9000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564d2de0fe40 0x564d2ddf0300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:42.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.312+0000 7f971abf9000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x564d2dbce1e0 con 0x564d2de0fe40 2024-03-31T15:10:42.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.312+0000 7f971abf9000 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x564d2dc0cab0 con 0x564d2de0f520 2024-03-31T15:10:42.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.312+0000 7f971abf9000 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x564d2dccb250 con 0x564d2ddc2dc0 2024-03-31T15:10:42.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.312+0000 7f971abf9000 10 client.0 _ll_register_callbacks cb 0x564d2ddac370 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T15:10:42.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.312+0000 7f9715800640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564d2de0fe40 0x564d2ddf0300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:42.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.312+0000 7f9715800640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564d2de0fe40 0x564d2ddf0300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41032/0 (socket says 172.21.15.16:41032) 2024-03-31T15:10:42.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.312+0000 7f9715800640 1 -- 172.21.15.16:0/2787952198 learned_addr learned my addr 172.21.15.16:0/2787952198 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:10:42.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.312+0000 7f970fe00640 1 --2- 172.21.15.16:0/2787952198 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564d2de0f520 0x564d2de16bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:42.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.312+0000 7f9714e00640 1 --2- 172.21.15.16:0/2787952198 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x564d2ddc2dc0 0x564d2de16670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:42.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.312+0000 7f9715800640 1 -- 172.21.15.16:0/2787952198 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x564d2ddc2dc0 msgr2=0x564d2de16670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:10:42.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.312+0000 7f9715800640 1 --2- 172.21.15.16:0/2787952198 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x564d2ddc2dc0 0x564d2de16670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:42.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.312+0000 7f9715800640 1 -- 172.21.15.16:0/2787952198 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564d2de0f520 msgr2=0x564d2de16bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:10:42.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.312+0000 7f9715800640 1 --2- 172.21.15.16:0/2787952198 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564d2de0f520 0x564d2de16bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:42.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.312+0000 7f9715800640 1 -- 172.21.15.16:0/2787952198 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564d2dbccb70 con 0x564d2de0fe40 2024-03-31T15:10:42.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.312+0000 7f9715800640 1 --2- 172.21.15.16:0/2787952198 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564d2de0fe40 0x564d2ddf0300 secure :-1 s=READY pgs=1184 cs=0 l=1 rev1=1 crypto rx=0x7f97080ac040 tx=0x7f9708004a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8938cb5fba7f0067 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:10:42.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.312+0000 7f96f6000640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-31T15:10:42.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.312+0000 7f96f6000640 1 -- 172.21.15.16:0/2787952198 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97080aa940 con 0x564d2de0fe40 2024-03-31T15:10:42.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.312+0000 7f971abf9000 1 -- 172.21.15.16:0/2787952198 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9704019070 con 0x564d2de0fe40 2024-03-31T15:10:42.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.312+0000 7f96f6000640 1 -- 172.21.15.16:0/2787952198 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 5824+0+0 (secure 0 0 0) 0x7f97080bfb60 con 0x564d2de0fe40 2024-03-31T15:10:42.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.312+0000 7f96f6000640 1 -- 172.21.15.16:0/2787952198 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=68) ==== 24+0+0 (secure 0 0 0) 0x7f97080bfda0 con 0x564d2de0fe40 2024-03-31T15:10:42.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.312+0000 7f971abf9000 10 client.7890 fetch_fsmap learned FSMap version 68 2024-03-31T15:10:42.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.312+0000 7f971abf9000 1 -- 172.21.15.16:0/2787952198 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=68,monmap=2+}) -- 0x564d2dbadb90 con 0x564d2de0fe40 2024-03-31T15:10:42.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.312+0000 7f96f6000640 1 -- 172.21.15.16:0/2787952198 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9704019070 con 0x564d2de0fe40 2024-03-31T15:10:42.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.312+0000 7f96f6000640 1 -- 172.21.15.16:0/2787952198 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 68) ==== 3494+0+0 (secure 0 0 0) 0x7f97080c6050 con 0x564d2de0fe40 2024-03-31T15:10:42.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.312+0000 7f971abf9000 10 client.7890 fetch_fsmap finished waiting for FSMap version 68 2024-03-31T15:10:42.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.312+0000 7f971abf9000 10 client.7890 resolve_mds: resolved 6:0 to role '6:0' aka daemon mds.c 2024-03-31T15:10:42.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.312+0000 7f971abf9000 20 client.7890 populate_metadata read hostname 'smithi016' 2024-03-31T15:10:42.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.312+0000 7f971abf9000 1 --2- 172.21.15.16:0/2787952198 >> [v2:172.21.15.16:6832/818074038,v1:172.21.15.16:6834/818074038] conn(0x564d2ddf10a0 0x564d2ddf1480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:42.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.312+0000 7f971abf9000 4 client.7890 mds_command: new command op to 7749 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-31T15:10:42.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.312+0000 7f971abf9000 1 -- 172.21.15.16:0/2787952198 --> [v2:172.21.15.16:6832/818074038,v1:172.21.15.16:6834/818074038] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564d2dbabdd0 con 0x564d2ddf10a0 2024-03-31T15:10:42.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.312+0000 7f9714e00640 1 --2- 172.21.15.16:0/2787952198 >> [v2:172.21.15.16:6832/818074038,v1:172.21.15.16:6834/818074038] conn(0x564d2ddf10a0 0x564d2ddf1480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:42.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.316+0000 7f9714e00640 1 --2- 172.21.15.16:0/2787952198 >> [v2:172.21.15.16:6832/818074038,v1:172.21.15.16:6834/818074038] conn(0x564d2ddf10a0 0x564d2ddf1480 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bd1fc64c6dab46cc server_cookie=d98770742fb58ecc in_seq=0 out_seq=0 2024-03-31T15:10:42.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.316+0000 7f96f6000640 10 client.7890 ms_handle_connect on v2:172.21.15.16:6832/818074038 2024-03-31T15:10:42.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.316+0000 7f96f6000640 1 -- 172.21.15.16:0/2787952198 <== mds.0 v2:172.21.15.16:6832/818074038 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x564d2dbabdd0 con 0x564d2ddf10a0 2024-03-31T15:10:42.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.316+0000 7f96f6000640 10 client.7890 handle_command_reply: tid=0 2024-03-31T15:10:42.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.316+0000 7f971abf9000 1 client.7890 shutdown 2024-03-31T15:10:42.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.316+0000 7f971abf9000 1 -- 172.21.15.16:0/2787952198 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564d2de0fe40 msgr2=0x564d2ddf0300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:10:42.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.316+0000 7f971abf9000 1 --2- 172.21.15.16:0/2787952198 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564d2de0fe40 0x564d2ddf0300 secure :-1 s=READY pgs=1184 cs=0 l=1 rev1=1 crypto rx=0x7f97080ac040 tx=0x7f9708004a50 comp rx=0 tx=0).stop 2024-03-31T15:10:42.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.316+0000 7f971abf9000 1 -- 172.21.15.16:0/2787952198 shutdown_connections 2024-03-31T15:10:42.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.316+0000 7f971abf9000 1 --2- 172.21.15.16:0/2787952198 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564d2de0fe40 0x564d2ddf0300 unknown :-1 s=CLOSED pgs=1184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:42.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.316+0000 7f971abf9000 1 --2- 172.21.15.16:0/2787952198 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564d2de0f520 0x564d2de16bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:42.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.316+0000 7f971abf9000 1 --2- 172.21.15.16:0/2787952198 >> [v2:172.21.15.16:6832/818074038,v1:172.21.15.16:6834/818074038] conn(0x564d2ddf10a0 0x564d2ddf1480 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:42.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.316+0000 7f971abf9000 1 --2- 172.21.15.16:0/2787952198 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x564d2ddc2dc0 0x564d2de16670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:42.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.316+0000 7f971abf9000 1 -- 172.21.15.16:0/2787952198 >> 172.21.15.16:0/2787952198 conn(0x564d2ddec070 msgr2=0x564d2ddee460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:10:42.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.316+0000 7f96f6000640 0 client.7890 ms_handle_reset on v2:172.21.15.16:6832/818074038 2024-03-31T15:10:42.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.316+0000 7f971abf9000 1 -- 172.21.15.16:0/2787952198 shutdown_connections 2024-03-31T15:10:42.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.316+0000 7f971abf9000 1 -- 172.21.15.16:0/2787952198 wait complete. 2024-03-31T15:10:42.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.316+0000 7f971abf9000 20 client.7890 trim_cache size 0 max 16384 2024-03-31T15:10:42.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.332+0000 7f971abf9000 1 Processor -- start 2024-03-31T15:10:42.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.332+0000 7f971abf9000 1 -- start start 2024-03-31T15:10:42.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.332+0000 7f971abf9000 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x564d2ddc2dc0 0x564d2de0eba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:42.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.332+0000 7f971abf9000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564d2de0f520 0x564d2de25800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:42.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.332+0000 7f971abf9000 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564d2de0fe40 0x564d2de25d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:42.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.332+0000 7f971abf9000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f97100b51c0 con 0x564d2de0f520 2024-03-31T15:10:42.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.332+0000 7f971abf9000 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f97100b4200 con 0x564d2de0fe40 2024-03-31T15:10:42.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.332+0000 7f971abf9000 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f97101546c0 con 0x564d2ddc2dc0 2024-03-31T15:10:42.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.332+0000 7f970fe00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564d2de0f520 0x564d2de25800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:42.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.332+0000 7f970fe00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564d2de0f520 0x564d2de25800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41046/0 (socket says 172.21.15.16:41046) 2024-03-31T15:10:42.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.332+0000 7f970fe00640 1 -- 172.21.15.16:0/3365805038 learned_addr learned my addr 172.21.15.16:0/3365805038 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:10:42.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.336+0000 7f970fe00640 1 -- 172.21.15.16:0/3365805038 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x564d2ddc2dc0 msgr2=0x564d2de0eba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:10:42.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.336+0000 7f970fe00640 1 --2- 172.21.15.16:0/3365805038 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x564d2ddc2dc0 0x564d2de0eba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:42.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.336+0000 7f970fe00640 1 -- 172.21.15.16:0/3365805038 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564d2de0fe40 msgr2=0x564d2de25d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:10:42.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.336+0000 7f970fe00640 1 --2- 172.21.15.16:0/3365805038 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564d2de0fe40 0x564d2de25d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:42.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.336+0000 7f970fe00640 1 -- 172.21.15.16:0/3365805038 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564d2dc91140 con 0x564d2de0f520 2024-03-31T15:10:42.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.336+0000 7f970fe00640 1 --2- 172.21.15.16:0/3365805038 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564d2de0f520 0x564d2de25800 secure :-1 s=READY pgs=1185 cs=0 l=1 rev1=1 crypto rx=0x7f96fc0039c0 tx=0x7f96fc01e040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4fadfc69109f94c server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:10:42.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.336+0000 7f9700c00640 1 -- 172.21.15.16:0/3365805038 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96fc06e030 con 0x564d2de0f520 2024-03-31T15:10:42.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.336+0000 7f9700c00640 1 -- 172.21.15.16:0/3365805038 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f96fc011d50 con 0x564d2de0f520 2024-03-31T15:10:42.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.336+0000 7f9700c00640 1 -- 172.21.15.16:0/3365805038 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96fc0860b0 con 0x564d2de0f520 2024-03-31T15:10:42.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.336+0000 7f971abf9000 1 -- 172.21.15.16:0/3365805038 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564d2de0f520 msgr2=0x564d2de25800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:10:42.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.336+0000 7f971abf9000 1 --2- 172.21.15.16:0/3365805038 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564d2de0f520 0x564d2de25800 secure :-1 s=READY pgs=1185 cs=0 l=1 rev1=1 crypto rx=0x7f96fc0039c0 tx=0x7f96fc01e040 comp rx=0 tx=0).stop 2024-03-31T15:10:42.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.336+0000 7f971abf9000 1 -- 172.21.15.16:0/3365805038 shutdown_connections 2024-03-31T15:10:42.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.336+0000 7f971abf9000 1 --2- 172.21.15.16:0/3365805038 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564d2de0fe40 0x564d2de25d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:42.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.336+0000 7f971abf9000 1 --2- 172.21.15.16:0/3365805038 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564d2de0f520 0x564d2de25800 unknown :-1 s=CLOSED pgs=1185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:42.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.336+0000 7f971abf9000 1 --2- 172.21.15.16:0/3365805038 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x564d2ddc2dc0 0x564d2de0eba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:42.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.336+0000 7f971abf9000 1 -- 172.21.15.16:0/3365805038 >> 172.21.15.16:0/3365805038 conn(0x564d2ddec070 msgr2=0x564d2de1cd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:10:42.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.336+0000 7f971abf9000 1 -- 172.21.15.16:0/3365805038 shutdown_connections 2024-03-31T15:10:42.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.336+0000 7f971abf9000 1 -- 172.21.15.16:0/3365805038 wait complete. 2024-03-31T15:10:42.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.336+0000 7f971abf9000 1 -- start start 2024-03-31T15:10:42.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.336+0000 7f971abf9000 1 Processor -- start 2024-03-31T15:10:42.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.336+0000 7f971abf9000 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564d2ddc2dc0 0x564d2de0ec20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:42.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.336+0000 7f971abf9000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564d2de0f520 0x564d2de13280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:42.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.336+0000 7f971abf9000 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x564d2de0fe40 0x564d2de137c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:42.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.336+0000 7f971abf9000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x564d2dbce1e0 con 0x564d2de0f520 2024-03-31T15:10:42.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.336+0000 7f971abf9000 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f97101474e0 con 0x564d2ddc2dc0 2024-03-31T15:10:42.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.336+0000 7f971abf9000 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x564d2dc0fe10 con 0x564d2de0fe40 2024-03-31T15:10:42.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.336+0000 7f971abf9000 10 client.0 _ll_register_callbacks cb 0x7f9708009b30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T15:10:42.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.336+0000 7f970fe00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564d2de0f520 0x564d2de13280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:42.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.336+0000 7f970fe00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564d2de0f520 0x564d2de13280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41052/0 (socket says 172.21.15.16:41052) 2024-03-31T15:10:42.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.336+0000 7f970fe00640 1 -- 172.21.15.16:0/2445583329 learned_addr learned my addr 172.21.15.16:0/2445583329 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:10:42.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.336+0000 7f970fe00640 1 -- 172.21.15.16:0/2445583329 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x564d2de0fe40 msgr2=0x564d2de137c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:10:42.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.336+0000 7f9714e00640 1 --2- 172.21.15.16:0/2445583329 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564d2ddc2dc0 0x564d2de0ec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:42.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.336+0000 7f9715800640 1 --2- 172.21.15.16:0/2445583329 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x564d2de0fe40 0x564d2de137c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:42.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.336+0000 7f970fe00640 1 --2- 172.21.15.16:0/2445583329 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x564d2de0fe40 0x564d2de137c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:42.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.336+0000 7f970fe00640 1 -- 172.21.15.16:0/2445583329 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564d2ddc2dc0 msgr2=0x564d2de0ec20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:10:42.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.336+0000 7f970fe00640 1 --2- 172.21.15.16:0/2445583329 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564d2ddc2dc0 0x564d2de0ec20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:42.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.336+0000 7f970fe00640 1 -- 172.21.15.16:0/2445583329 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564d2db9f1e0 con 0x564d2de0f520 2024-03-31T15:10:42.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.336+0000 7f9714e00640 1 --2- 172.21.15.16:0/2445583329 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564d2ddc2dc0 0x564d2de0ec20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:10:42.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.336+0000 7f9715800640 1 --2- 172.21.15.16:0/2445583329 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x564d2de0fe40 0x564d2de137c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:10:42.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.340+0000 7f970fe00640 1 --2- 172.21.15.16:0/2445583329 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564d2de0f520 0x564d2de13280 secure :-1 s=READY pgs=1186 cs=0 l=1 rev1=1 crypto rx=0x7f96fc0563b0 tx=0x7f96fc086040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4631cecbc7617ac server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:10:42.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.340+0000 7f96f6000640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-31T15:10:42.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.340+0000 7f96f6000640 1 -- 172.21.15.16:0/2445583329 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96fc074020 con 0x564d2de0f520 2024-03-31T15:10:42.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.340+0000 7f96f6000640 1 -- 172.21.15.16:0/2445583329 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 5824+0+0 (secure 0 0 0) 0x7f96fc06ed50 con 0x564d2de0f520 2024-03-31T15:10:42.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.340+0000 7f971abf9000 1 -- 172.21.15.16:0/2445583329 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f96fc011d50 con 0x564d2de0f520 2024-03-31T15:10:42.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.340+0000 7f96f6000640 1 -- 172.21.15.16:0/2445583329 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=68) ==== 24+0+0 (secure 0 0 0) 0x7f96fc06e040 con 0x564d2de0f520 2024-03-31T15:10:42.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.340+0000 7f971abf9000 10 client.7896 fetch_fsmap learned FSMap version 68 2024-03-31T15:10:42.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.340+0000 7f971abf9000 1 -- 172.21.15.16:0/2445583329 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=68,monmap=2+}) -- 0x564d2dba47b0 con 0x564d2de0f520 2024-03-31T15:10:42.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.340+0000 7f96f6000640 1 -- 172.21.15.16:0/2445583329 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f96fc011d50 con 0x564d2de0f520 2024-03-31T15:10:42.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.340+0000 7f96f6000640 1 -- 172.21.15.16:0/2445583329 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 68) ==== 3494+0+0 (secure 0 0 0) 0x7f96fc0860b0 con 0x564d2de0f520 2024-03-31T15:10:42.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.340+0000 7f971abf9000 10 client.7896 fetch_fsmap finished waiting for FSMap version 68 2024-03-31T15:10:42.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.340+0000 7f971abf9000 10 client.7896 resolve_mds: resolved 6:0 to role '6:0' aka daemon mds.c 2024-03-31T15:10:42.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.340+0000 7f971abf9000 20 client.7896 populate_metadata read hostname 'smithi016' 2024-03-31T15:10:42.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.340+0000 7f971abf9000 1 --2- 172.21.15.16:0/2445583329 >> [v2:172.21.15.16:6832/818074038,v1:172.21.15.16:6834/818074038] conn(0x564d2de13d00 0x564d2de7f340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:42.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.340+0000 7f971abf9000 4 client.7896 mds_command: new command op to 7749 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-31T15:10:42.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.340+0000 7f971abf9000 1 -- 172.21.15.16:0/2445583329 --> [v2:172.21.15.16:6832/818074038,v1:172.21.15.16:6834/818074038] -- command(tid 0: {"prefix": "scrub status"}) -- 0x564d2dc1e040 con 0x564d2de13d00 2024-03-31T15:10:42.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.340+0000 7f9714e00640 1 --2- 172.21.15.16:0/2445583329 >> [v2:172.21.15.16:6832/818074038,v1:172.21.15.16:6834/818074038] conn(0x564d2de13d00 0x564d2de7f340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:42.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.340+0000 7f9714e00640 1 --2- 172.21.15.16:0/2445583329 >> [v2:172.21.15.16:6832/818074038,v1:172.21.15.16:6834/818074038] conn(0x564d2de13d00 0x564d2de7f340 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ab9fe7c56fd5ccd3 server_cookie=fe9832b3f9f49b51 in_seq=0 out_seq=0 2024-03-31T15:10:42.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.340+0000 7f96f6000640 10 client.7896 ms_handle_connect on v2:172.21.15.16:6832/818074038 2024-03-31T15:10:42.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.340+0000 7f96f6000640 1 -- 172.21.15.16:0/2445583329 <== mds.0 v2:172.21.15.16:6832/818074038 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x564d2dc1e040 con 0x564d2de13d00 2024-03-31T15:10:42.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.340+0000 7f96f6000640 10 client.7896 handle_command_reply: tid=0 2024-03-31T15:10:42.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.340+0000 7f971abf9000 1 client.7896 shutdown 2024-03-31T15:10:42.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.340+0000 7f971abf9000 1 -- 172.21.15.16:0/2445583329 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564d2de0f520 msgr2=0x564d2de13280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:10:42.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.340+0000 7f971abf9000 1 --2- 172.21.15.16:0/2445583329 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564d2de0f520 0x564d2de13280 secure :-1 s=READY pgs=1186 cs=0 l=1 rev1=1 crypto rx=0x7f96fc0563b0 tx=0x7f96fc086040 comp rx=0 tx=0).stop 2024-03-31T15:10:42.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.340+0000 7f971abf9000 1 -- 172.21.15.16:0/2445583329 shutdown_connections 2024-03-31T15:10:42.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.340+0000 7f971abf9000 1 --2- 172.21.15.16:0/2445583329 >> [v2:172.21.15.16:6832/818074038,v1:172.21.15.16:6834/818074038] conn(0x564d2de13d00 0x564d2de7f340 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:42.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.340+0000 7f971abf9000 1 --2- 172.21.15.16:0/2445583329 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x564d2de0fe40 0x564d2de137c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:42.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.340+0000 7f971abf9000 1 --2- 172.21.15.16:0/2445583329 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564d2de0f520 0x564d2de13280 unknown :-1 s=CLOSED pgs=1186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:42.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.340+0000 7f971abf9000 1 --2- 172.21.15.16:0/2445583329 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564d2ddc2dc0 0x564d2de0ec20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:42.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.340+0000 7f96f6000640 0 client.7896 ms_handle_reset on v2:172.21.15.16:6832/818074038 2024-03-31T15:10:42.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.340+0000 7f971abf9000 1 -- 172.21.15.16:0/2445583329 >> 172.21.15.16:0/2445583329 conn(0x564d2ddec070 msgr2=0x564d2ddef2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:10:42.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.340+0000 7f971abf9000 1 -- 172.21.15.16:0/2445583329 shutdown_connections 2024-03-31T15:10:42.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.340+0000 7f971abf9000 1 -- 172.21.15.16:0/2445583329 wait complete. 2024-03-31T15:10:42.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.340+0000 7f971abf9000 20 client.7896 trim_cache size 0 max 16384 2024-03-31T15:10:42.337 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-03-31T15:10:42.337 INFO:teuthology.orchestra.run.smithi016.stdout: "status": "no active scrubs running", 2024-03-31T15:10:42.337 INFO:teuthology.orchestra.run.smithi016.stdout: "scrubs": {} 2024-03-31T15:10:42.337 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-03-31T15:10:42.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.340+0000 7f9717800640 1 -- 172.21.15.16:0/3222452056 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f96f0068380 msgr2=0x7f96f006a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:10:42.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.340+0000 7f9717800640 1 --2- 172.21.15.16:0/3222452056 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f96f0068380 0x7f96f006a840 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f9704006fd0 tx=0x7f9704008040 comp rx=0 tx=0).stop 2024-03-31T15:10:42.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.340+0000 7f9717800640 1 -- 172.21.15.16:0/3222452056 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f97100731c0 msgr2=0x7f971014ddc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:10:42.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.340+0000 7f9717800640 1 --2- 172.21.15.16:0/3222452056 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f97100731c0 0x7f971014ddc0 secure :-1 s=READY pgs=1182 cs=0 l=1 rev1=1 crypto rx=0x7f96fc00c410 tx=0x7f96fc00c8d0 comp rx=0 tx=0).stop 2024-03-31T15:10:42.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.340+0000 7f9717800640 1 -- 172.21.15.16:0/3222452056 shutdown_connections 2024-03-31T15:10:42.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.340+0000 7f9717800640 1 --2- 172.21.15.16:0/3222452056 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f96f0068380 0x7f96f006a840 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:42.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.340+0000 7f9717800640 1 --2- 172.21.15.16:0/3222452056 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f97101567e0 0x7f971014e300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:42.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.340+0000 7f9717800640 1 --2- 172.21.15.16:0/3222452056 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f97100731c0 0x7f971014ddc0 unknown :-1 s=CLOSED pgs=1182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:42.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.340+0000 7f9717800640 1 --2- 172.21.15.16:0/3222452056 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f971006dbe0 0x7f9710145490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:42.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.340+0000 7f9717800640 1 -- 172.21.15.16:0/3222452056 >> 172.21.15.16:0/3222452056 conn(0x7f971006ee40 msgr2=0x7f971006b620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:10:42.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.340+0000 7f9717800640 1 -- 172.21.15.16:0/3222452056 shutdown_connections 2024-03-31T15:10:42.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.340+0000 7f9717800640 1 -- 172.21.15.16:0/3222452056 wait complete. 2024-03-31T15:10:42.352 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-31T15:10:42.353 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-31T15:10:42.494 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.504+0000 7f9abce00640 1 Processor -- start 2024-03-31T15:10:42.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.504+0000 7f9abce00640 1 -- start start 2024-03-31T15:10:42.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.504+0000 7f9abce00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f9ab8145aa0 0x7f9ab815c8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:42.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.504+0000 7f9abce00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f9ab806c1c0 0x7f9ab8072e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:42.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.504+0000 7f9abce00640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9ab80d09f0 con 0x7f9ab806b620 2024-03-31T15:10:42.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.504+0000 7f9abce00640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f9ab806fd50 con 0x7f9ab8145aa0 2024-03-31T15:10:42.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.504+0000 7f9abce00640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f9ab80d0cd0 con 0x7f9ab806c1c0 2024-03-31T15:10:42.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.504+0000 7f9ab5e00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f9ab8145aa0 0x7f9ab815c8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:42.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.504+0000 7f9ab6800640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f9ab806b620 0x7f9ab80701e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:34538/0 (socket says 172.21.15.16:34538) 2024-03-31T15:10:42.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.504+0000 7f9ab6800640 1 -- 172.21.15.16:0/1956593097 learned_addr learned my addr 172.21.15.16:0/1956593097 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:10:42.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.504+0000 7f9ab5400640 1 --2- 172.21.15.16:0/1956593097 >> v2:172.21.15.89:3300/0 conn(0x7f9ab806c1c0 0x7f9ab8072e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:42.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.504+0000 7f9ab5e00640 1 -- 172.21.15.16:0/1956593097 >> v2:172.21.15.89:3300/0 conn(0x7f9ab806c1c0 msgr2=0x7f9ab8072e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:10:42.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.504+0000 7f9ab5e00640 1 --2- 172.21.15.16:0/1956593097 >> v2:172.21.15.89:3300/0 conn(0x7f9ab806c1c0 0x7f9ab8072e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:42.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.504+0000 7f9ab5e00640 1 -- 172.21.15.16:0/1956593097 >> v1:172.21.15.16:6789/0 conn(0x7f9ab806b620 legacy=0x7f9ab80701e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:10:42.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.504+0000 7f9ab5e00640 1 -- 172.21.15.16:0/1956593097 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ab80d06e0 con 0x7f9ab8145aa0 2024-03-31T15:10:42.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.504+0000 7f9ab4a00640 1 -- 172.21.15.16:0/1956593097 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2286967166 0 0) 0x7f9ab80d09f0 con 0x7f9ab806b620 2024-03-31T15:10:42.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.508+0000 7f9ab5e00640 1 --2- 172.21.15.16:0/1956593097 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9ab8145aa0 0x7f9ab815c8e0 secure :-1 s=READY pgs=1188 cs=0 l=1 rev1=1 crypto rx=0x7f9ab80b7310 tx=0x7f9aa801c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e26ae0a623dd9c5e server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:10:42.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.508+0000 7f9ab4a00640 1 -- 172.21.15.16:0/1956593097 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9aa80b1070 con 0x7f9ab8145aa0 2024-03-31T15:10:42.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.508+0000 7f9ab4a00640 1 -- 172.21.15.16:0/1956593097 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9aa80ac5f0 con 0x7f9ab8145aa0 2024-03-31T15:10:42.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.508+0000 7f9ab4a00640 1 -- 172.21.15.16:0/1956593097 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9aa80ac9c0 con 0x7f9ab8145aa0 2024-03-31T15:10:42.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.508+0000 7f9abce00640 1 -- 172.21.15.16:0/1956593097 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9ab8145aa0 msgr2=0x7f9ab815c8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:10:42.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.508+0000 7f9abce00640 1 --2- 172.21.15.16:0/1956593097 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9ab8145aa0 0x7f9ab815c8e0 secure :-1 s=READY pgs=1188 cs=0 l=1 rev1=1 crypto rx=0x7f9ab80b7310 tx=0x7f9aa801c480 comp rx=0 tx=0).stop 2024-03-31T15:10:42.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.508+0000 7f9abce00640 1 -- 172.21.15.16:0/1956593097 shutdown_connections 2024-03-31T15:10:42.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.508+0000 7f9abce00640 1 --2- 172.21.15.16:0/1956593097 >> v2:172.21.15.89:3300/0 conn(0x7f9ab806c1c0 0x7f9ab8072e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:42.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.508+0000 7f9abce00640 1 --2- 172.21.15.16:0/1956593097 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9ab8145aa0 0x7f9ab815c8e0 unknown :-1 s=CLOSED pgs=1188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:42.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.508+0000 7f9abce00640 1 -- 172.21.15.16:0/1956593097 >> 172.21.15.16:0/1956593097 conn(0x7f9ab80d1060 msgr2=0x7f9ab80d1470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:10:42.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.508+0000 7f9abce00640 1 -- 172.21.15.16:0/1956593097 shutdown_connections 2024-03-31T15:10:42.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.508+0000 7f9abce00640 1 -- 172.21.15.16:0/1956593097 wait complete. 2024-03-31T15:10:42.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.512+0000 7f9abce00640 1 Processor -- start 2024-03-31T15:10:42.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.512+0000 7f9abce00640 1 -- start start 2024-03-31T15:10:42.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.512+0000 7f9abce00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f9ab806b620 0x7f9ab813fbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:42.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.512+0000 7f9abce00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9ab806c1c0 0x7f9ab80c75f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:42.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.512+0000 7f9abce00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9ab8145aa0 0x7f9ab80aedb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:42.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.512+0000 7f9abce00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f9ab8169070 con 0x7f9ab8145aa0 2024-03-31T15:10:42.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.512+0000 7f9abce00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f9ab8168ef0 con 0x7f9ab806c1c0 2024-03-31T15:10:42.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.512+0000 7f9abce00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f9ab81691f0 con 0x7f9ab806b620 2024-03-31T15:10:42.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.512+0000 7f9ab6800640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9ab8145aa0 0x7f9ab80aedb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:42.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.512+0000 7f9ab6800640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9ab8145aa0 0x7f9ab80aedb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41070/0 (socket says 172.21.15.16:41070) 2024-03-31T15:10:42.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.512+0000 7f9ab6800640 1 -- 172.21.15.16:0/2521200876 learned_addr learned my addr 172.21.15.16:0/2521200876 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:10:42.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.512+0000 7f9ab5400640 1 --2- 172.21.15.16:0/2521200876 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9ab806c1c0 0x7f9ab80c75f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:42.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.512+0000 7f9ab5e00640 1 --2- 172.21.15.16:0/2521200876 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f9ab806b620 0x7f9ab813fbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:42.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.512+0000 7f9ab6800640 1 -- 172.21.15.16:0/2521200876 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f9ab806b620 msgr2=0x7f9ab813fbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:10:42.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.512+0000 7f9ab6800640 1 --2- 172.21.15.16:0/2521200876 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f9ab806b620 0x7f9ab813fbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:42.507 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.512+0000 7f9ab6800640 1 -- 172.21.15.16:0/2521200876 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9ab806c1c0 msgr2=0x7f9ab80c75f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:10:42.507 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.512+0000 7f9ab6800640 1 --2- 172.21.15.16:0/2521200876 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9ab806c1c0 0x7f9ab80c75f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:42.507 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.512+0000 7f9ab6800640 1 -- 172.21.15.16:0/2521200876 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ab81fb0b0 con 0x7f9ab8145aa0 2024-03-31T15:10:42.507 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.512+0000 7f9ab6800640 1 --2- 172.21.15.16:0/2521200876 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9ab8145aa0 0x7f9ab80aedb0 secure :-1 s=READY pgs=1189 cs=0 l=1 rev1=1 crypto rx=0x7f9ab000c700 tx=0x7f9ab000cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:10:42.507 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.512+0000 7f9aae000640 1 -- 172.21.15.16:0/2521200876 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ab0018070 con 0x7f9ab8145aa0 2024-03-31T15:10:42.508 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.512+0000 7f9aae000640 1 -- 172.21.15.16:0/2521200876 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9ab000ce10 con 0x7f9ab8145aa0 2024-03-31T15:10:42.508 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.512+0000 7f9abce00640 1 -- 172.21.15.16:0/2521200876 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9ab81fb390 con 0x7f9ab8145aa0 2024-03-31T15:10:42.508 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.512+0000 7f9aae000640 1 -- 172.21.15.16:0/2521200876 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ab000a860 con 0x7f9ab8145aa0 2024-03-31T15:10:42.508 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.512+0000 7f9abce00640 1 -- 172.21.15.16:0/2521200876 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9ab81fb950 con 0x7f9ab8145aa0 2024-03-31T15:10:42.508 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.516+0000 7f9abce00640 1 -- 172.21.15.16:0/2521200876 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9ab81fbd30 con 0x7f9ab8145aa0 2024-03-31T15:10:42.508 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.516+0000 7f9aae000640 1 -- 172.21.15.16:0/2521200876 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f9ab001b070 con 0x7f9ab8145aa0 2024-03-31T15:10:42.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.516+0000 7f9aae000640 1 --2- 172.21.15.16:0/2521200876 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f9a98068380 0x7f9a9806a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:42.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.516+0000 7f9aae000640 1 -- 172.21.15.16:0/2521200876 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 5824+0+0 (secure 0 0 0) 0x7f9ab0086060 con 0x7f9ab8145aa0 2024-03-31T15:10:42.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.516+0000 7f9ab5e00640 1 --2- 172.21.15.16:0/2521200876 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f9a98068380 0x7f9a9806a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:42.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.520+0000 7f9ab5e00640 1 --2- 172.21.15.16:0/2521200876 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f9a98068380 0x7f9a9806a840 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f9aa80ae000 tx=0x7f9aa8008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:10:42.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.520+0000 7f9aae000640 1 -- 172.21.15.16:0/2521200876 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f9ab81fbd30 con 0x7f9ab8145aa0 2024-03-31T15:10:42.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.808+0000 7f9abce00640 1 -- 172.21.15.16:0/2521200876 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9ab8069950 con 0x7f9ab8145aa0 2024-03-31T15:10:42.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.808+0000 7f9aae000640 1 -- 172.21.15.16:0/2521200876 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 68 v68) ==== 94+0+5181 (secure 0 0 0) 0x7f9ab0056530 con 0x7f9ab8145aa0 2024-03-31T15:10:42.799 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:10:42.800 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":68,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6494,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/2445558494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6835","nonce":2445558494},{"type":"v1","addr":"172.21.15.89:6837","nonce":2445558494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":56},{"gid":7270,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/3185913858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6833","nonce":3185913858},{"type":"v1","addr":"172.21.15.16:6835","nonce":3185913858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":56},{"gid":7279,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6836/1599468493","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":1599468493},{"type":"v1","addr":"172.21.15.89:6836","nonce":1599468493}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":56}],"filesystems":[{"mdsmap":{"epoch":68,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-31T15:09:50.926196+0000","modified":"2024-03-31T15:10:15.370488+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":82,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7749},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7749":{"gid":7749,"name":"c","rank":0,"incarnation":64,"state":"up:active","state_seq":6,"addr":"172.21.15.16:6834/818074038","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":818074038},{"type":"v1","addr":"172.21.15.16:6834","nonce":818074038}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7749,"qdb_cluster":[7749]},"id":6}]} 2024-03-31T15:10:42.800 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 68 2024-03-31T15:10:42.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.808+0000 7f9abce00640 1 -- 172.21.15.16:0/2521200876 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f9a98068380 msgr2=0x7f9a9806a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:10:42.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.808+0000 7f9abce00640 1 --2- 172.21.15.16:0/2521200876 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f9a98068380 0x7f9a9806a840 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f9aa80ae000 tx=0x7f9aa8008040 comp rx=0 tx=0).stop 2024-03-31T15:10:42.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.808+0000 7f9abce00640 1 -- 172.21.15.16:0/2521200876 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9ab8145aa0 msgr2=0x7f9ab80aedb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:10:42.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.808+0000 7f9abce00640 1 --2- 172.21.15.16:0/2521200876 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9ab8145aa0 0x7f9ab80aedb0 secure :-1 s=READY pgs=1189 cs=0 l=1 rev1=1 crypto rx=0x7f9ab000c700 tx=0x7f9ab000cbc0 comp rx=0 tx=0).stop 2024-03-31T15:10:42.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.808+0000 7f9abce00640 1 -- 172.21.15.16:0/2521200876 shutdown_connections 2024-03-31T15:10:42.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.808+0000 7f9abce00640 1 --2- 172.21.15.16:0/2521200876 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f9a98068380 0x7f9a9806a840 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:42.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.808+0000 7f9abce00640 1 --2- 172.21.15.16:0/2521200876 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9ab8145aa0 0x7f9ab80aedb0 unknown :-1 s=CLOSED pgs=1189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:42.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.808+0000 7f9abce00640 1 --2- 172.21.15.16:0/2521200876 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9ab806c1c0 0x7f9ab80c75f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:42.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.808+0000 7f9abce00640 1 --2- 172.21.15.16:0/2521200876 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f9ab806b620 0x7f9ab813fbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:42.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.808+0000 7f9abce00640 1 -- 172.21.15.16:0/2521200876 >> 172.21.15.16:0/2521200876 conn(0x7f9ab80d1060 msgr2=0x7f9ab80b5ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:10:42.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.808+0000 7f9abce00640 1 -- 172.21.15.16:0/2521200876 shutdown_connections 2024-03-31T15:10:42.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.808+0000 7f9abce00640 1 -- 172.21.15.16:0/2521200876 wait complete. 2024-03-31T15:10:42.821 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c damage ls --format=json-pretty 2024-03-31T15:10:42.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.972+0000 7f95b2e00640 1 Processor -- start 2024-03-31T15:10:42.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.972+0000 7f95b2e00640 1 -- start start 2024-03-31T15:10:42.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.972+0000 7f95b2e00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f95ac06f920 0x7f95ac06fd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:42.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.972+0000 7f95b2e00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f95ac0d0a60 0x7f95ac0cea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:42.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.972+0000 7f95b2e00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f95ac0cef80 0x7f95ac070e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:42.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.972+0000 7f95b2e00640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f95ac13f670 con 0x7f95ac0cef80 2024-03-31T15:10:42.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.972+0000 7f95b2e00640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f95ac070240 con 0x7f95ac0d0a60 2024-03-31T15:10:42.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.972+0000 7f95b2e00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f95ac15ca70 con 0x7f95ac06f920 2024-03-31T15:10:42.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.972+0000 7f95b0e00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f95ac0cef80 0x7f95ac070e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:42.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.972+0000 7f95b0e00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f95ac0cef80 0x7f95ac070e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41074/0 (socket says 172.21.15.16:41074) 2024-03-31T15:10:42.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.972+0000 7f95b0e00640 1 -- 172.21.15.16:0/2995428415 learned_addr learned my addr 172.21.15.16:0/2995428415 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:10:42.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.972+0000 7f95ab400640 1 --2- 172.21.15.16:0/2995428415 >> v2:172.21.15.89:3300/0 conn(0x7f95ac0d0a60 0x7f95ac0cea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:42.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.972+0000 7f95abe00640 1 --2- 172.21.15.16:0/2995428415 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f95ac06f920 0x7f95ac06fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:42.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.972+0000 7f95b0e00640 1 -- 172.21.15.16:0/2995428415 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f95ac06f920 msgr2=0x7f95ac06fd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:10:42.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.972+0000 7f95b0e00640 1 --2- 172.21.15.16:0/2995428415 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f95ac06f920 0x7f95ac06fd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:42.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.972+0000 7f95b0e00640 1 -- 172.21.15.16:0/2995428415 >> v2:172.21.15.89:3300/0 conn(0x7f95ac0d0a60 msgr2=0x7f95ac0cea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:10:42.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.972+0000 7f95b0e00640 1 --2- 172.21.15.16:0/2995428415 >> v2:172.21.15.89:3300/0 conn(0x7f95ac0d0a60 0x7f95ac0cea40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:42.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.972+0000 7f95b0e00640 1 -- 172.21.15.16:0/2995428415 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95ac15cd50 con 0x7f95ac0cef80 2024-03-31T15:10:42.969 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.976+0000 7f95b0e00640 1 --2- 172.21.15.16:0/2995428415 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f95ac0cef80 0x7f95ac070e60 secure :-1 s=READY pgs=1190 cs=0 l=1 rev1=1 crypto rx=0x7f95ac071f30 tx=0x7f95a401c730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc822050142a5deb server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:10:42.969 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.976+0000 7f95aaa00640 1 -- 172.21.15.16:0/2995428415 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95a40ac070 con 0x7f95ac0cef80 2024-03-31T15:10:42.969 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.976+0000 7f95aaa00640 1 -- 172.21.15.16:0/2995428415 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f95a40a8420 con 0x7f95ac0cef80 2024-03-31T15:10:42.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.976+0000 7f95aaa00640 1 -- 172.21.15.16:0/2995428415 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95a40a87f0 con 0x7f95ac0cef80 2024-03-31T15:10:42.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.976+0000 7f95b2e00640 1 -- 172.21.15.16:0/2995428415 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f95ac0cef80 msgr2=0x7f95ac070e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:10:42.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.976+0000 7f95b2e00640 1 --2- 172.21.15.16:0/2995428415 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f95ac0cef80 0x7f95ac070e60 secure :-1 s=READY pgs=1190 cs=0 l=1 rev1=1 crypto rx=0x7f95ac071f30 tx=0x7f95a401c730 comp rx=0 tx=0).stop 2024-03-31T15:10:42.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.976+0000 7f95b2e00640 1 -- 172.21.15.16:0/2995428415 shutdown_connections 2024-03-31T15:10:42.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.976+0000 7f95b2e00640 1 --2- 172.21.15.16:0/2995428415 >> v2:172.21.15.89:3300/0 conn(0x7f95ac0d0a60 0x7f95ac0cea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:42.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.976+0000 7f95b2e00640 1 --2- 172.21.15.16:0/2995428415 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f95ac0cef80 0x7f95ac070e60 unknown :-1 s=CLOSED pgs=1190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:42.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.976+0000 7f95b2e00640 1 --2- 172.21.15.16:0/2995428415 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f95ac06f920 0x7f95ac06fd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:42.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.976+0000 7f95b2e00640 1 -- 172.21.15.16:0/2995428415 >> 172.21.15.16:0/2995428415 conn(0x7f95ac067930 msgr2=0x7f95ac0b3ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:10:42.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.976+0000 7f95b2e00640 1 -- 172.21.15.16:0/2995428415 shutdown_connections 2024-03-31T15:10:42.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.976+0000 7f95b2e00640 1 -- 172.21.15.16:0/2995428415 wait complete. 2024-03-31T15:10:42.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.980+0000 7f95b2e00640 1 Processor -- start 2024-03-31T15:10:42.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.980+0000 7f95b2e00640 1 -- start start 2024-03-31T15:10:42.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.980+0000 7f95b2e00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f95ac06f920 0x7f95ac15d850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:42.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.980+0000 7f95b2e00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f95ac0cef80 0x7f95ac06e040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:42.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.980+0000 7f95b2e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f95ac0d0a60 0x7f95ac13ebb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:42.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.980+0000 7f95b2e00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f95ac1419c0 con 0x7f95ac0d0a60 2024-03-31T15:10:42.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.980+0000 7f95b2e00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f95ac15c720 con 0x7f95ac0cef80 2024-03-31T15:10:42.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.980+0000 7f95b2e00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f95ac141b40 con 0x7f95ac06f920 2024-03-31T15:10:42.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.980+0000 7f95b0e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f95ac0d0a60 0x7f95ac13ebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:42.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.980+0000 7f95b0e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f95ac0d0a60 0x7f95ac13ebb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41078/0 (socket says 172.21.15.16:41078) 2024-03-31T15:10:42.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.980+0000 7f95b0e00640 1 -- 172.21.15.16:0/1527280309 learned_addr learned my addr 172.21.15.16:0/1527280309 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:10:42.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.980+0000 7f95ab400640 1 --2- 172.21.15.16:0/1527280309 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f95ac0cef80 0x7f95ac06e040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:42.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.980+0000 7f95abe00640 1 --2- 172.21.15.16:0/1527280309 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f95ac06f920 0x7f95ac15d850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:42.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.980+0000 7f95b0e00640 1 -- 172.21.15.16:0/1527280309 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f95ac06f920 msgr2=0x7f95ac15d850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:10:42.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.980+0000 7f95b0e00640 1 --2- 172.21.15.16:0/1527280309 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f95ac06f920 0x7f95ac15d850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:42.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.980+0000 7f95b0e00640 1 -- 172.21.15.16:0/1527280309 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f95ac0cef80 msgr2=0x7f95ac06e040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:10:42.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.980+0000 7f95b0e00640 1 --2- 172.21.15.16:0/1527280309 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f95ac0cef80 0x7f95ac06e040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:42.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.980+0000 7f95b0e00640 1 -- 172.21.15.16:0/1527280309 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95ac15df50 con 0x7f95ac0d0a60 2024-03-31T15:10:42.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.980+0000 7f95b0e00640 1 --2- 172.21.15.16:0/1527280309 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f95ac0d0a60 0x7f95ac13ebb0 secure :-1 s=READY pgs=1191 cs=0 l=1 rev1=1 crypto rx=0x7f95a401ade0 tx=0x7f95a40194d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:10:42.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.984+0000 7f95a3e00640 1 -- 172.21.15.16:0/1527280309 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95a40ac040 con 0x7f95ac0d0a60 2024-03-31T15:10:42.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.984+0000 7f95a3e00640 1 -- 172.21.15.16:0/1527280309 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f95a40bd450 con 0x7f95ac0d0a60 2024-03-31T15:10:42.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.984+0000 7f95b2e00640 1 -- 172.21.15.16:0/1527280309 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f95ac15e230 con 0x7f95ac0d0a60 2024-03-31T15:10:42.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.984+0000 7f95a3e00640 1 -- 172.21.15.16:0/1527280309 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95a4009510 con 0x7f95ac0d0a60 2024-03-31T15:10:42.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.984+0000 7f95b2e00640 1 -- 172.21.15.16:0/1527280309 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f95ac15e7f0 con 0x7f95ac0d0a60 2024-03-31T15:10:42.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.984+0000 7f95a3e00640 1 -- 172.21.15.16:0/1527280309 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f95a40af070 con 0x7f95ac0d0a60 2024-03-31T15:10:42.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.984+0000 7f95b633f000 1 Processor -- start 2024-03-31T15:10:42.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.984+0000 7f95b633f000 1 -- start start 2024-03-31T15:10:42.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.988+0000 7f95a3e00640 1 --2- 172.21.15.16:0/1527280309 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f958c068380 0x7f958c06a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:42.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.988+0000 7f95a3e00640 1 -- 172.21.15.16:0/1527280309 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 5824+0+0 (secure 0 0 0) 0x7f95a4121070 con 0x7f95ac0d0a60 2024-03-31T15:10:42.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.988+0000 7f95ab400640 1 --2- 172.21.15.16:0/1527280309 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f958c068380 0x7f958c06a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:42.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.988+0000 7f95b633f000 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55ad2b4ee980 0x55ad2b4eed60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:42.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.988+0000 7f95b633f000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ad2b4ef2a0 0x55ad2b4a1dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:42.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.988+0000 7f95b633f000 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ad2b4a2310 0x55ad2b4a26f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:42.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.988+0000 7f95b633f000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55ad2b2f2c50 con 0x55ad2b4ef2a0 2024-03-31T15:10:42.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.988+0000 7f95b633f000 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55ad2b28f8a0 con 0x55ad2b4a2310 2024-03-31T15:10:42.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.988+0000 7f95b633f000 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x55ad2b2eec00 con 0x55ad2b4ee980 2024-03-31T15:10:42.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.988+0000 7f95ab400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ad2b4ef2a0 0x55ad2b4a1dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:42.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.988+0000 7f95abe00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55ad2b4ee980 0x55ad2b4eed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:42.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.988+0000 7f95ab400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ad2b4ef2a0 0x55ad2b4a1dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41094/0 (socket says 172.21.15.16:41094) 2024-03-31T15:10:42.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.988+0000 7f95ab400640 1 -- 172.21.15.16:0/3058298214 learned_addr learned my addr 172.21.15.16:0/3058298214 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:10:42.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.988+0000 7f95b0e00640 1 --2- 172.21.15.16:0/3058298214 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ad2b4a2310 0x55ad2b4a26f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:42.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.988+0000 7f95ab400640 1 -- 172.21.15.16:0/3058298214 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55ad2b4ee980 msgr2=0x55ad2b4eed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:10:42.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.988+0000 7f95ab400640 1 --2- 172.21.15.16:0/3058298214 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55ad2b4ee980 0x55ad2b4eed60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:42.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.988+0000 7f95ab400640 1 -- 172.21.15.16:0/3058298214 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ad2b4a2310 msgr2=0x55ad2b4a26f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:10:42.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.988+0000 7f95ab400640 1 --2- 172.21.15.16:0/3058298214 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ad2b4a2310 0x55ad2b4a26f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:42.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.988+0000 7f95ab400640 1 -- 172.21.15.16:0/3058298214 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ad2b2ccba0 con 0x55ad2b4ef2a0 2024-03-31T15:10:42.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.988+0000 7f95ab400640 1 --2- 172.21.15.16:0/3058298214 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ad2b4ef2a0 0x55ad2b4a1dd0 secure :-1 s=READY pgs=1192 cs=0 l=1 rev1=1 crypto rx=0x7f9598018fa0 tx=0x7f959801d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c17470e701722eb3 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:10:42.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.988+0000 7f95ab400640 1 --2- 172.21.15.16:0/1527280309 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f958c068380 0x7f958c06a840 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f95ac06d4b0 tx=0x7f9598009eb0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:10:42.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.988+0000 7f95a0c00640 1 -- 172.21.15.16:0/3058298214 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9598015e30 con 0x55ad2b4ef2a0 2024-03-31T15:10:42.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.988+0000 7f95a0c00640 1 -- 172.21.15.16:0/3058298214 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9598004020 con 0x55ad2b4ef2a0 2024-03-31T15:10:42.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.988+0000 7f95a0c00640 1 -- 172.21.15.16:0/3058298214 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95980041d0 con 0x55ad2b4ef2a0 2024-03-31T15:10:42.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.988+0000 7f95b633f000 1 -- 172.21.15.16:0/3058298214 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ad2b4ef2a0 msgr2=0x55ad2b4a1dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:10:42.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.988+0000 7f95b633f000 1 --2- 172.21.15.16:0/3058298214 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ad2b4ef2a0 0x55ad2b4a1dd0 secure :-1 s=READY pgs=1192 cs=0 l=1 rev1=1 crypto rx=0x7f9598018fa0 tx=0x7f959801d000 comp rx=0 tx=0).stop 2024-03-31T15:10:42.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.988+0000 7f95b633f000 1 -- 172.21.15.16:0/3058298214 shutdown_connections 2024-03-31T15:10:42.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.988+0000 7f95b633f000 1 --2- 172.21.15.16:0/3058298214 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ad2b4a2310 0x55ad2b4a26f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:42.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.988+0000 7f95b633f000 1 --2- 172.21.15.16:0/3058298214 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ad2b4ef2a0 0x55ad2b4a1dd0 unknown :-1 s=CLOSED pgs=1192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:42.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.988+0000 7f95b633f000 1 --2- 172.21.15.16:0/3058298214 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55ad2b4ee980 0x55ad2b4eed60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:42.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.988+0000 7f95b633f000 1 -- 172.21.15.16:0/3058298214 >> 172.21.15.16:0/3058298214 conn(0x55ad2b29f9f0 msgr2=0x55ad2b4d1730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:10:42.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.988+0000 7f95b633f000 1 -- 172.21.15.16:0/3058298214 shutdown_connections 2024-03-31T15:10:42.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.988+0000 7f95b633f000 1 -- 172.21.15.16:0/3058298214 wait complete. 2024-03-31T15:10:42.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.988+0000 7f95b633f000 1 -- start start 2024-03-31T15:10:42.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.988+0000 7f95b633f000 1 Processor -- start 2024-03-31T15:10:42.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.992+0000 7f95b633f000 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ad2b4a2310 0x55ad2b4ee110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:42.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.992+0000 7f95b633f000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ad2b4ee980 0x55ad2b4a8110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:42.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.992+0000 7f95b633f000 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55ad2b4ef2a0 0x55ad2b4a8650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:42.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.992+0000 7f95b633f000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55ad2b2b00a0 con 0x55ad2b4ee980 2024-03-31T15:10:42.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.992+0000 7f95b633f000 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55ad2b2f7280 con 0x55ad2b4a2310 2024-03-31T15:10:42.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.992+0000 7f95b633f000 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x55ad2b2af690 con 0x55ad2b4ef2a0 2024-03-31T15:10:42.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.992+0000 7f95b633f000 10 client.0 _ll_register_callbacks cb 0x55ad2b48f810 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T15:10:42.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.992+0000 7f95ab400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ad2b4ee980 0x55ad2b4a8110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:42.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.992+0000 7f95ab400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ad2b4ee980 0x55ad2b4a8110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41096/0 (socket says 172.21.15.16:41096) 2024-03-31T15:10:42.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.992+0000 7f95ab400640 1 -- 172.21.15.16:0/147591022 learned_addr learned my addr 172.21.15.16:0/147591022 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:10:42.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.992+0000 7f95abe00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ad2b4a2310 0x55ad2b4ee110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:42.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.992+0000 7f95b0e00640 1 --2- 172.21.15.16:0/147591022 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55ad2b4ef2a0 0x55ad2b4a8650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:42.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.992+0000 7f95ab400640 1 -- 172.21.15.16:0/147591022 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55ad2b4ef2a0 msgr2=0x55ad2b4a8650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:10:42.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.992+0000 7f95ab400640 1 --2- 172.21.15.16:0/147591022 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55ad2b4ef2a0 0x55ad2b4a8650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:42.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.992+0000 7f95ab400640 1 -- 172.21.15.16:0/147591022 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ad2b4a2310 msgr2=0x55ad2b4ee110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:10:42.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.992+0000 7f95ab400640 1 --2- 172.21.15.16:0/147591022 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ad2b4a2310 0x55ad2b4ee110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:42.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.992+0000 7f95ab400640 1 -- 172.21.15.16:0/147591022 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ad2b338160 con 0x55ad2b4ee980 2024-03-31T15:10:42.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.992+0000 7f95ab400640 1 --2- 172.21.15.16:0/147591022 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ad2b4ee980 0x55ad2b4a8110 secure :-1 s=READY pgs=1193 cs=0 l=1 rev1=1 crypto rx=0x7f9598018fd0 tx=0x7f9598004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=57f0bb711b826277 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:10:42.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.992+0000 7f9595600640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-31T15:10:42.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.992+0000 7f9595600640 1 -- 172.21.15.16:0/147591022 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9598033070 con 0x55ad2b4ee980 2024-03-31T15:10:42.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.992+0000 7f95b633f000 1 -- 172.21.15.16:0/147591022 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9598004020 con 0x55ad2b4ee980 2024-03-31T15:10:42.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.992+0000 7f9595600640 1 -- 172.21.15.16:0/147591022 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 5824+0+0 (secure 0 0 0) 0x7f959800f040 con 0x55ad2b4ee980 2024-03-31T15:10:42.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.992+0000 7f9595600640 1 -- 172.21.15.16:0/147591022 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=68) ==== 24+0+0 (secure 0 0 0) 0x7f9598009aa0 con 0x55ad2b4ee980 2024-03-31T15:10:42.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.992+0000 7f95b633f000 10 client.7917 fetch_fsmap learned FSMap version 68 2024-03-31T15:10:42.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.992+0000 7f95b633f000 1 -- 172.21.15.16:0/147591022 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=68,monmap=2+}) -- 0x55ad2b28d9c0 con 0x55ad2b4ee980 2024-03-31T15:10:42.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.992+0000 7f9595600640 1 -- 172.21.15.16:0/147591022 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9598004020 con 0x55ad2b4ee980 2024-03-31T15:10:42.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.992+0000 7f9595600640 1 -- 172.21.15.16:0/147591022 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 68) ==== 3494+0+0 (secure 0 0 0) 0x7f959803f080 con 0x55ad2b4ee980 2024-03-31T15:10:42.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.992+0000 7f95b633f000 10 client.7917 fetch_fsmap finished waiting for FSMap version 68 2024-03-31T15:10:42.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.992+0000 7f95b633f000 10 client.7917 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-31T15:10:42.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.992+0000 7f95b633f000 20 client.7917 populate_metadata read hostname 'smithi016' 2024-03-31T15:10:42.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.992+0000 7f95b633f000 1 --2- 172.21.15.16:0/147591022 >> [v2:172.21.15.16:6832/818074038,v1:172.21.15.16:6834/818074038] conn(0x55ad2b4a9060 0x55ad2b4a9440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:42.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.992+0000 7f95b633f000 4 client.7917 mds_command: new command op to 7749 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-31T15:10:42.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.992+0000 7f95b633f000 1 -- 172.21.15.16:0/147591022 --> [v2:172.21.15.16:6832/818074038,v1:172.21.15.16:6834/818074038] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ad2b28de40 con 0x55ad2b4a9060 2024-03-31T15:10:42.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.992+0000 7f95abe00640 1 --2- 172.21.15.16:0/147591022 >> [v2:172.21.15.16:6832/818074038,v1:172.21.15.16:6834/818074038] conn(0x55ad2b4a9060 0x55ad2b4a9440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:42.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.992+0000 7f95abe00640 1 --2- 172.21.15.16:0/147591022 >> [v2:172.21.15.16:6832/818074038,v1:172.21.15.16:6834/818074038] conn(0x55ad2b4a9060 0x55ad2b4a9440 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=958036f6d332ad4b server_cookie=24119cad7044baf3 in_seq=0 out_seq=0 2024-03-31T15:10:42.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.992+0000 7f9595600640 10 client.7917 ms_handle_connect on v2:172.21.15.16:6832/818074038 2024-03-31T15:10:42.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.996+0000 7f9595600640 1 -- 172.21.15.16:0/147591022 <== mds.0 v2:172.21.15.16:6832/818074038 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55ad2b28de40 con 0x55ad2b4a9060 2024-03-31T15:10:42.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.996+0000 7f9595600640 10 client.7917 handle_command_reply: tid=0 2024-03-31T15:10:42.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.996+0000 7f95b633f000 1 client.7917 shutdown 2024-03-31T15:10:42.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.996+0000 7f95b633f000 1 -- 172.21.15.16:0/147591022 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ad2b4ee980 msgr2=0x55ad2b4a8110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:10:42.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.996+0000 7f95b633f000 1 --2- 172.21.15.16:0/147591022 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ad2b4ee980 0x55ad2b4a8110 secure :-1 s=READY pgs=1193 cs=0 l=1 rev1=1 crypto rx=0x7f9598018fd0 tx=0x7f9598004740 comp rx=0 tx=0).stop 2024-03-31T15:10:42.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.996+0000 7f95b633f000 1 -- 172.21.15.16:0/147591022 shutdown_connections 2024-03-31T15:10:42.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.996+0000 7f95b633f000 1 --2- 172.21.15.16:0/147591022 >> [v2:172.21.15.16:6832/818074038,v1:172.21.15.16:6834/818074038] conn(0x55ad2b4a9060 0x55ad2b4a9440 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:42.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.996+0000 7f95b633f000 1 --2- 172.21.15.16:0/147591022 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55ad2b4ef2a0 0x55ad2b4a8650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:42.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.996+0000 7f95b633f000 1 --2- 172.21.15.16:0/147591022 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ad2b4ee980 0x55ad2b4a8110 unknown :-1 s=CLOSED pgs=1193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:42.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.996+0000 7f9595600640 0 client.7917 ms_handle_reset on v2:172.21.15.16:6832/818074038 2024-03-31T15:10:42.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.996+0000 7f95b633f000 1 --2- 172.21.15.16:0/147591022 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ad2b4a2310 0x55ad2b4ee110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:42.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.996+0000 7f95b633f000 1 -- 172.21.15.16:0/147591022 >> 172.21.15.16:0/147591022 conn(0x55ad2b29f9f0 msgr2=0x55ad2b4d1520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:10:42.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.996+0000 7f95b633f000 1 -- 172.21.15.16:0/147591022 shutdown_connections 2024-03-31T15:10:42.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.996+0000 7f95b633f000 1 -- 172.21.15.16:0/147591022 wait complete. 2024-03-31T15:10:42.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:42.996+0000 7f95b633f000 20 client.7917 trim_cache size 0 max 16384 2024-03-31T15:10:43.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.008+0000 7f95b633f000 1 Processor -- start 2024-03-31T15:10:43.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.008+0000 7f95b633f000 1 -- start start 2024-03-31T15:10:43.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.008+0000 7f95b633f000 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55ad2b4a2310 0x55ad2b4a1b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:43.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.008+0000 7f95b633f000 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ad2b4ee980 0x55ad2b4a6eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:43.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.008+0000 7f95b633f000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ad2b4ef2a0 0x55ad2b4a73f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:43.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.008+0000 7f95b633f000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f95ac141520 con 0x55ad2b4ef2a0 2024-03-31T15:10:43.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.008+0000 7f95b633f000 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f95ac140880 con 0x55ad2b4ee980 2024-03-31T15:10:43.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.008+0000 7f95b633f000 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f95ac140ed0 con 0x55ad2b4a2310 2024-03-31T15:10:43.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.012+0000 7f95b0e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ad2b4ef2a0 0x55ad2b4a73f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:43.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.012+0000 7f95b0e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ad2b4ef2a0 0x55ad2b4a73f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41110/0 (socket says 172.21.15.16:41110) 2024-03-31T15:10:43.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.012+0000 7f95b0e00640 1 -- 172.21.15.16:0/2047362872 learned_addr learned my addr 172.21.15.16:0/2047362872 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:10:43.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.012+0000 7f95abe00640 1 --2- 172.21.15.16:0/2047362872 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55ad2b4a2310 0x55ad2b4a1b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:43.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.012+0000 7f95ab400640 1 --2- 172.21.15.16:0/2047362872 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ad2b4ee980 0x55ad2b4a6eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:43.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.012+0000 7f95b0e00640 1 -- 172.21.15.16:0/2047362872 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55ad2b4a2310 msgr2=0x55ad2b4a1b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:10:43.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.012+0000 7f95b0e00640 1 --2- 172.21.15.16:0/2047362872 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55ad2b4a2310 0x55ad2b4a1b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:43.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.012+0000 7f95b0e00640 1 -- 172.21.15.16:0/2047362872 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ad2b4ee980 msgr2=0x55ad2b4a6eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:10:43.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.012+0000 7f95b0e00640 1 --2- 172.21.15.16:0/2047362872 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ad2b4ee980 0x55ad2b4a6eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:43.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.012+0000 7f95b0e00640 1 -- 172.21.15.16:0/2047362872 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ad2b28faf0 con 0x55ad2b4ef2a0 2024-03-31T15:10:43.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.012+0000 7f95ab400640 1 --2- 172.21.15.16:0/2047362872 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ad2b4ee980 0x55ad2b4a6eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:10:43.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.012+0000 7f95b0e00640 1 --2- 172.21.15.16:0/2047362872 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ad2b4ef2a0 0x55ad2b4a73f0 secure :-1 s=READY pgs=1194 cs=0 l=1 rev1=1 crypto rx=0x7f95a401cc10 tx=0x7f95a40ed210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee4c93a711b4e9d8 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:10:43.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.012+0000 7f9597e00640 1 -- 172.21.15.16:0/2047362872 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95a40f5270 con 0x55ad2b4ef2a0 2024-03-31T15:10:43.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.012+0000 7f9597e00640 1 -- 172.21.15.16:0/2047362872 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f95a40f9aa0 con 0x55ad2b4ef2a0 2024-03-31T15:10:43.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.012+0000 7f9597e00640 1 -- 172.21.15.16:0/2047362872 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95a411f170 con 0x55ad2b4ef2a0 2024-03-31T15:10:43.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.012+0000 7f95b633f000 1 -- 172.21.15.16:0/2047362872 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ad2b4ef2a0 msgr2=0x55ad2b4a73f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:10:43.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.012+0000 7f95b633f000 1 --2- 172.21.15.16:0/2047362872 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ad2b4ef2a0 0x55ad2b4a73f0 secure :-1 s=READY pgs=1194 cs=0 l=1 rev1=1 crypto rx=0x7f95a401cc10 tx=0x7f95a40ed210 comp rx=0 tx=0).stop 2024-03-31T15:10:43.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.012+0000 7f95b633f000 1 -- 172.21.15.16:0/2047362872 shutdown_connections 2024-03-31T15:10:43.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.012+0000 7f95b633f000 1 --2- 172.21.15.16:0/2047362872 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ad2b4ef2a0 0x55ad2b4a73f0 unknown :-1 s=CLOSED pgs=1194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:43.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.012+0000 7f95b633f000 1 --2- 172.21.15.16:0/2047362872 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ad2b4ee980 0x55ad2b4a6eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:43.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.012+0000 7f95b633f000 1 --2- 172.21.15.16:0/2047362872 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55ad2b4a2310 0x55ad2b4a1b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:43.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.012+0000 7f95b633f000 1 -- 172.21.15.16:0/2047362872 >> 172.21.15.16:0/2047362872 conn(0x55ad2b29f9f0 msgr2=0x55ad2b4fbf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:10:43.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.012+0000 7f95b633f000 1 -- 172.21.15.16:0/2047362872 shutdown_connections 2024-03-31T15:10:43.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.012+0000 7f95b633f000 1 -- 172.21.15.16:0/2047362872 wait complete. 2024-03-31T15:10:43.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.012+0000 7f95b633f000 1 -- start start 2024-03-31T15:10:43.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.012+0000 7f95b633f000 1 Processor -- start 2024-03-31T15:10:43.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.012+0000 7f95b633f000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ad2b4a2310 0x55ad2b4a7ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:43.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.012+0000 7f95b633f000 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55ad2b4ee980 0x55ad2b4a8400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:43.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.012+0000 7f95b633f000 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ad2b4ef2a0 0x55ad2b55aa80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:43.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.012+0000 7f95b633f000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55ad2b2f9860 con 0x55ad2b4a2310 2024-03-31T15:10:43.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.012+0000 7f95b633f000 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55ad2b2fe7f0 con 0x55ad2b4ef2a0 2024-03-31T15:10:43.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.012+0000 7f95b633f000 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x55ad2b2f4ca0 con 0x55ad2b4ee980 2024-03-31T15:10:43.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.012+0000 7f95b633f000 10 client.0 _ll_register_callbacks cb 0x7f95a40092d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T15:10:43.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.012+0000 7f95abe00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ad2b4a2310 0x55ad2b4a7ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:43.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.012+0000 7f95abe00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ad2b4a2310 0x55ad2b4a7ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41114/0 (socket says 172.21.15.16:41114) 2024-03-31T15:10:43.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.012+0000 7f95abe00640 1 -- 172.21.15.16:0/2129563663 learned_addr learned my addr 172.21.15.16:0/2129563663 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:10:43.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.012+0000 7f95ab400640 1 --2- 172.21.15.16:0/2129563663 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55ad2b4ee980 0x55ad2b4a8400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:43.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.012+0000 7f95b0e00640 1 --2- 172.21.15.16:0/2129563663 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ad2b4ef2a0 0x55ad2b55aa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:43.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.016+0000 7f95abe00640 1 -- 172.21.15.16:0/2129563663 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55ad2b4ee980 msgr2=0x55ad2b4a8400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:10:43.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.016+0000 7f95abe00640 1 --2- 172.21.15.16:0/2129563663 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55ad2b4ee980 0x55ad2b4a8400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:43.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.016+0000 7f95abe00640 1 -- 172.21.15.16:0/2129563663 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ad2b4ef2a0 msgr2=0x55ad2b55aa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:10:43.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.016+0000 7f95abe00640 1 --2- 172.21.15.16:0/2129563663 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ad2b4ef2a0 0x55ad2b55aa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:43.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.016+0000 7f95abe00640 1 -- 172.21.15.16:0/2129563663 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ad2b34b320 con 0x55ad2b4a2310 2024-03-31T15:10:43.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.016+0000 7f95abe00640 1 --2- 172.21.15.16:0/2129563663 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ad2b4a2310 0x55ad2b4a7ec0 secure :-1 s=READY pgs=1195 cs=0 l=1 rev1=1 crypto rx=0x55ad2b2f3500 tx=0x7f959c022680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=98d7b84a91acd40 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:10:43.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.016+0000 7f9595600640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-31T15:10:43.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.016+0000 7f9595600640 1 -- 172.21.15.16:0/2129563663 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f959c017110 con 0x55ad2b4a2310 2024-03-31T15:10:43.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.016+0000 7f9595600640 1 -- 172.21.15.16:0/2129563663 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 5824+0+0 (secure 0 0 0) 0x7f959c004030 con 0x55ad2b4a2310 2024-03-31T15:10:43.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.016+0000 7f95b633f000 1 -- 172.21.15.16:0/2129563663 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f95a40f9aa0 con 0x55ad2b4a2310 2024-03-31T15:10:43.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.016+0000 7f9595600640 1 -- 172.21.15.16:0/2129563663 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=68) ==== 24+0+0 (secure 0 0 0) 0x7f959c033050 con 0x55ad2b4a2310 2024-03-31T15:10:43.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.016+0000 7f95b633f000 10 client.7923 fetch_fsmap learned FSMap version 68 2024-03-31T15:10:43.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.016+0000 7f95b633f000 1 -- 172.21.15.16:0/2129563663 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=68,monmap=2+}) -- 0x55ad2b286a80 con 0x55ad2b4a2310 2024-03-31T15:10:43.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.016+0000 7f9595600640 1 -- 172.21.15.16:0/2129563663 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f95a40f9aa0 con 0x55ad2b4a2310 2024-03-31T15:10:43.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.016+0000 7f9595600640 1 -- 172.21.15.16:0/2129563663 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 68) ==== 3494+0+0 (secure 0 0 0) 0x7f959c036020 con 0x55ad2b4a2310 2024-03-31T15:10:43.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.016+0000 7f95b633f000 10 client.7923 fetch_fsmap finished waiting for FSMap version 68 2024-03-31T15:10:43.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.016+0000 7f95b633f000 10 client.7923 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-31T15:10:43.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.016+0000 7f95b633f000 20 client.7923 populate_metadata read hostname 'smithi016' 2024-03-31T15:10:43.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.016+0000 7f95b633f000 1 --2- 172.21.15.16:0/2129563663 >> [v2:172.21.15.16:6832/818074038,v1:172.21.15.16:6834/818074038] conn(0x55ad2b55afc0 0x55ad2b55b3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:43.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.016+0000 7f95b633f000 4 client.7923 mds_command: new command op to 7749 tid=0 multi_id=0 [{"prefix": "damage ls", "format": "json-pretty"}] 2024-03-31T15:10:43.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.016+0000 7f95b633f000 1 -- 172.21.15.16:0/2129563663 --> [v2:172.21.15.16:6832/818074038,v1:172.21.15.16:6834/818074038] -- command(tid 0: {"prefix": "damage ls", "format": "json-pretty"}) -- 0x55ad2b300190 con 0x55ad2b55afc0 2024-03-31T15:10:43.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.016+0000 7f95ab400640 1 --2- 172.21.15.16:0/2129563663 >> [v2:172.21.15.16:6832/818074038,v1:172.21.15.16:6834/818074038] conn(0x55ad2b55afc0 0x55ad2b55b3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:43.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.016+0000 7f95ab400640 1 --2- 172.21.15.16:0/2129563663 >> [v2:172.21.15.16:6832/818074038,v1:172.21.15.16:6834/818074038] conn(0x55ad2b55afc0 0x55ad2b55b3a0 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bfa5be6bb65d20fe server_cookie=7431854ced88aeb0 in_seq=0 out_seq=0 2024-03-31T15:10:43.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.016+0000 7f9595600640 10 client.7923 ms_handle_connect on v2:172.21.15.16:6832/818074038 2024-03-31T15:10:43.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.016+0000 7f9595600640 1 -- 172.21.15.16:0/2129563663 <== mds.0 v2:172.21.15.16:6832/818074038 1 ==== command_reply(tid 0: 0 ) ==== 8+0+420 (crc 0 0 0) 0x55ad2b300190 con 0x55ad2b55afc0 2024-03-31T15:10:43.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.016+0000 7f9595600640 10 client.7923 handle_command_reply: tid=0 2024-03-31T15:10:43.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.016+0000 7f95b633f000 1 client.7923 shutdown 2024-03-31T15:10:43.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.016+0000 7f95b633f000 1 -- 172.21.15.16:0/2129563663 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ad2b4a2310 msgr2=0x55ad2b4a7ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:10:43.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.016+0000 7f95b633f000 1 --2- 172.21.15.16:0/2129563663 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ad2b4a2310 0x55ad2b4a7ec0 secure :-1 s=READY pgs=1195 cs=0 l=1 rev1=1 crypto rx=0x55ad2b2f3500 tx=0x7f959c022680 comp rx=0 tx=0).stop 2024-03-31T15:10:43.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.016+0000 7f95b633f000 1 -- 172.21.15.16:0/2129563663 shutdown_connections 2024-03-31T15:10:43.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.016+0000 7f95b633f000 1 --2- 172.21.15.16:0/2129563663 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ad2b4ef2a0 0x55ad2b55aa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:43.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.016+0000 7f95b633f000 1 --2- 172.21.15.16:0/2129563663 >> [v2:172.21.15.16:6832/818074038,v1:172.21.15.16:6834/818074038] conn(0x55ad2b55afc0 0x55ad2b55b3a0 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:43.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.016+0000 7f95b633f000 1 --2- 172.21.15.16:0/2129563663 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55ad2b4ee980 0x55ad2b4a8400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:43.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.016+0000 7f95b633f000 1 --2- 172.21.15.16:0/2129563663 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ad2b4a2310 0x55ad2b4a7ec0 unknown :-1 s=CLOSED pgs=1195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:43.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.016+0000 7f9595600640 0 client.7923 ms_handle_reset on v2:172.21.15.16:6832/818074038 2024-03-31T15:10:43.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.016+0000 7f95b633f000 1 -- 172.21.15.16:0/2129563663 >> 172.21.15.16:0/2129563663 conn(0x55ad2b29f9f0 msgr2=0x55ad2b4eca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:10:43.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.016+0000 7f95b633f000 1 -- 172.21.15.16:0/2129563663 shutdown_connections 2024-03-31T15:10:43.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.016+0000 7f95b633f000 1 -- 172.21.15.16:0/2129563663 wait complete. 2024-03-31T15:10:43.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.016+0000 7f95b633f000 20 client.7923 trim_cache size 0 max 16384 2024-03-31T15:10:43.013 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:10:43.013 INFO:teuthology.orchestra.run.smithi016.stdout:[ 2024-03-31T15:10:43.013 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-31T15:10:43.013 INFO:teuthology.orchestra.run.smithi016.stdout: "damage_type": "backtrace", 2024-03-31T15:10:43.014 INFO:teuthology.orchestra.run.smithi016.stdout: "id": 1050930771, 2024-03-31T15:10:43.014 INFO:teuthology.orchestra.run.smithi016.stdout: "ino": 1099511627777, 2024-03-31T15:10:43.014 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/testdir/hardlink" 2024-03-31T15:10:43.014 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-31T15:10:43.014 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-31T15:10:43.014 INFO:teuthology.orchestra.run.smithi016.stdout: "damage_type": "dir_frag", 2024-03-31T15:10:43.014 INFO:teuthology.orchestra.run.smithi016.stdout: "id": 1266504711, 2024-03-31T15:10:43.014 INFO:teuthology.orchestra.run.smithi016.stdout: "ino": 1099511627776, 2024-03-31T15:10:43.014 INFO:teuthology.orchestra.run.smithi016.stdout: "frag": "*", 2024-03-31T15:10:43.014 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/dir" 2024-03-31T15:10:43.014 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-31T15:10:43.015 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-31T15:10:43.015 INFO:teuthology.orchestra.run.smithi016.stdout: "damage_type": "backtrace", 2024-03-31T15:10:43.018 INFO:teuthology.orchestra.run.smithi016.stdout: "id": 2620386182, 2024-03-31T15:10:43.018 INFO:teuthology.orchestra.run.smithi016.stdout: "ino": 1099511627776, 2024-03-31T15:10:43.018 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/dir" 2024-03-31T15:10:43.018 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-31T15:10:43.018 INFO:teuthology.orchestra.run.smithi016.stdout:] 2024-03-31T15:10:43.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.016+0000 7f95b2e00640 1 -- 172.21.15.16:0/1527280309 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f958c068380 msgr2=0x7f958c06a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:10:43.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.016+0000 7f95b2e00640 1 --2- 172.21.15.16:0/1527280309 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f958c068380 0x7f958c06a840 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f95ac06d4b0 tx=0x7f9598009eb0 comp rx=0 tx=0).stop 2024-03-31T15:10:43.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.016+0000 7f95b2e00640 1 -- 172.21.15.16:0/1527280309 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f95ac0d0a60 msgr2=0x7f95ac13ebb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:10:43.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.016+0000 7f95b2e00640 1 --2- 172.21.15.16:0/1527280309 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f95ac0d0a60 0x7f95ac13ebb0 secure :-1 s=READY pgs=1191 cs=0 l=1 rev1=1 crypto rx=0x7f95a401ade0 tx=0x7f95a40194d0 comp rx=0 tx=0).stop 2024-03-31T15:10:43.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.016+0000 7f95b2e00640 1 -- 172.21.15.16:0/1527280309 shutdown_connections 2024-03-31T15:10:43.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.016+0000 7f95b2e00640 1 --2- 172.21.15.16:0/1527280309 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f958c068380 0x7f958c06a840 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:43.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.016+0000 7f95b2e00640 1 --2- 172.21.15.16:0/1527280309 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f95ac0d0a60 0x7f95ac13ebb0 unknown :-1 s=CLOSED pgs=1191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:43.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.016+0000 7f95b2e00640 1 --2- 172.21.15.16:0/1527280309 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f95ac0cef80 0x7f95ac06e040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:43.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.016+0000 7f95b2e00640 1 --2- 172.21.15.16:0/1527280309 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f95ac06f920 0x7f95ac15d850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:43.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.016+0000 7f95b2e00640 1 -- 172.21.15.16:0/1527280309 >> 172.21.15.16:0/1527280309 conn(0x7f95ac067930 msgr2=0x7f95ac0d1810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:10:43.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.016+0000 7f95b2e00640 1 -- 172.21.15.16:0/1527280309 shutdown_connections 2024-03-31T15:10:43.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.016+0000 7f95b2e00640 1 -- 172.21.15.16:0/1527280309 wait complete. 2024-03-31T15:10:43.030 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-03-31T15:10:43.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.176+0000 7f585a200640 1 Processor -- start 2024-03-31T15:10:43.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.176+0000 7f585a200640 1 -- start start 2024-03-31T15:10:43.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.180+0000 7f585a200640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f585415bf40 0x7f58540afbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:43.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.180+0000 7f585a200640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f585406dc80 0x7f5854160420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:43.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.180+0000 7f585a200640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f585415fc60 con 0x7f585406c630 2024-03-31T15:10:43.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.180+0000 7f585a200640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f585415ff90 con 0x7f585415bf40 2024-03-31T15:10:43.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.180+0000 7f585a200640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f585406e110 con 0x7f585406dc80 2024-03-31T15:10:43.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.180+0000 7f5858e00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f585415bf40 0x7f58540afbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:43.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.180+0000 7f5858e00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f585415bf40 0x7f58540afbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41122/0 (socket says 172.21.15.16:41122) 2024-03-31T15:10:43.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.180+0000 7f5858e00640 1 -- 172.21.15.16:0/771125111 learned_addr learned my addr 172.21.15.16:0/771125111 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:10:43.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.180+0000 7f584be00640 1 --2- 172.21.15.16:0/771125111 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f585406dc80 0x7f5854160420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:43.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.180+0000 7f5858e00640 1 -- 172.21.15.16:0/771125111 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f585406dc80 msgr2=0x7f5854160420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:10:43.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.180+0000 7f5858e00640 1 --2- 172.21.15.16:0/771125111 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f585406dc80 0x7f5854160420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:43.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.180+0000 7f5858e00640 1 -- 172.21.15.16:0/771125111 >> v1:172.21.15.89:6789/0 conn(0x7f585406c630 legacy=0x7f58540ce190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:10:43.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.180+0000 7f5858e00640 1 -- 172.21.15.16:0/771125111 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f585406e3f0 con 0x7f585415bf40 2024-03-31T15:10:43.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.180+0000 7f584be00640 1 --2- 172.21.15.16:0/771125111 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f585406dc80 0x7f5854160420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:10:43.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.184+0000 7f5858e00640 1 --2- 172.21.15.16:0/771125111 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f585415bf40 0x7f58540afbf0 secure :-1 s=READY pgs=1196 cs=0 l=1 rev1=1 crypto rx=0x7f5844009870 tx=0x7f584401c460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d52d4beb4cf4dfbb server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:10:43.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.184+0000 7f5853e00640 1 -- 172.21.15.16:0/771125111 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58440b0070 con 0x7f585415bf40 2024-03-31T15:10:43.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.184+0000 7f5853e00640 1 -- 172.21.15.16:0/771125111 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f58440ab600 con 0x7f585415bf40 2024-03-31T15:10:43.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.184+0000 7f5853e00640 1 -- 172.21.15.16:0/771125111 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58440ab9d0 con 0x7f585415bf40 2024-03-31T15:10:43.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.184+0000 7f585a200640 1 -- 172.21.15.16:0/771125111 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f585415bf40 msgr2=0x7f58540afbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:10:43.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.184+0000 7f585a200640 1 --2- 172.21.15.16:0/771125111 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f585415bf40 0x7f58540afbf0 secure :-1 s=READY pgs=1196 cs=0 l=1 rev1=1 crypto rx=0x7f5844009870 tx=0x7f584401c460 comp rx=0 tx=0).stop 2024-03-31T15:10:43.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.184+0000 7f585a200640 1 -- 172.21.15.16:0/771125111 shutdown_connections 2024-03-31T15:10:43.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.184+0000 7f585a200640 1 --2- 172.21.15.16:0/771125111 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f585406dc80 0x7f5854160420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:43.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.184+0000 7f585a200640 1 --2- 172.21.15.16:0/771125111 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f585415bf40 0x7f58540afbf0 unknown :-1 s=CLOSED pgs=1196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:43.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.184+0000 7f585a200640 1 -- 172.21.15.16:0/771125111 >> 172.21.15.16:0/771125111 conn(0x7f5854071c20 msgr2=0x7f5854072030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:10:43.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.184+0000 7f585a200640 1 -- 172.21.15.16:0/771125111 shutdown_connections 2024-03-31T15:10:43.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.184+0000 7f585a200640 1 -- 172.21.15.16:0/771125111 wait complete. 2024-03-31T15:10:43.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.184+0000 7f585a200640 1 Processor -- start 2024-03-31T15:10:43.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.184+0000 7f585a200640 1 -- start start 2024-03-31T15:10:43.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.184+0000 7f585a200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f585406c630 0x7f585414dfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:43.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.184+0000 7f585a200640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f585415bf40 0x7f585414e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:43.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.184+0000 7f585a200640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f585415a9a0 0x7f58540cbd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:43.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.184+0000 7f585a200640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f585406e850 con 0x7f585406c630 2024-03-31T15:10:43.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.184+0000 7f585a200640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f585406e6d0 con 0x7f585415bf40 2024-03-31T15:10:43.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.184+0000 7f585a200640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f585415d330 con 0x7f585415a9a0 2024-03-31T15:10:43.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.184+0000 7f5858e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f585406c630 0x7f585414dfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:43.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.188+0000 7f5858e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f585406c630 0x7f585414dfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41130/0 (socket says 172.21.15.16:41130) 2024-03-31T15:10:43.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.188+0000 7f5858e00640 1 -- 172.21.15.16:0/3979938753 learned_addr learned my addr 172.21.15.16:0/3979938753 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:10:43.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.188+0000 7f584be00640 1 --2- 172.21.15.16:0/3979938753 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f585415bf40 0x7f585414e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:43.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.188+0000 7f5859800640 1 --2- 172.21.15.16:0/3979938753 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f585415a9a0 0x7f58540cbd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:43.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.188+0000 7f5858e00640 1 -- 172.21.15.16:0/3979938753 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f585415a9a0 msgr2=0x7f58540cbd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:10:43.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.188+0000 7f5858e00640 1 --2- 172.21.15.16:0/3979938753 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f585415a9a0 0x7f58540cbd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:43.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.188+0000 7f5858e00640 1 -- 172.21.15.16:0/3979938753 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f585415bf40 msgr2=0x7f585414e4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:10:43.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.188+0000 7f5858e00640 1 --2- 172.21.15.16:0/3979938753 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f585415bf40 0x7f585414e4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:43.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.188+0000 7f5858e00640 1 -- 172.21.15.16:0/3979938753 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5854152720 con 0x7f585406c630 2024-03-31T15:10:43.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.188+0000 7f5858e00640 1 --2- 172.21.15.16:0/3979938753 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f585406c630 0x7f585414dfb0 secure :-1 s=READY pgs=1197 cs=0 l=1 rev1=1 crypto rx=0x7f58440ae040 tx=0x7f584401aca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:10:43.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.188+0000 7f5851600640 1 -- 172.21.15.16:0/3979938753 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58440b0040 con 0x7f585406c630 2024-03-31T15:10:43.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.188+0000 7f5851600640 1 -- 172.21.15.16:0/3979938753 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f58440bc070 con 0x7f585406c630 2024-03-31T15:10:43.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.188+0000 7f5851600640 1 -- 172.21.15.16:0/3979938753 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58440b6880 con 0x7f585406c630 2024-03-31T15:10:43.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.188+0000 7f585a200640 1 -- 172.21.15.16:0/3979938753 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5854152a00 con 0x7f585406c630 2024-03-31T15:10:43.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.188+0000 7f585a200640 1 -- 172.21.15.16:0/3979938753 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5854158240 con 0x7f585406c630 2024-03-31T15:10:43.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.188+0000 7f585a200640 1 -- 172.21.15.16:0/3979938753 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5854158030 con 0x7f585406c630 2024-03-31T15:10:43.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.192+0000 7f5851600640 1 -- 172.21.15.16:0/3979938753 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f58440b6a20 con 0x7f585406c630 2024-03-31T15:10:43.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.192+0000 7f5851600640 1 --2- 172.21.15.16:0/3979938753 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f58300683b0 0x7f583006a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:43.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.192+0000 7f5851600640 1 -- 172.21.15.16:0/3979938753 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 5824+0+0 (secure 0 0 0) 0x7f58441206a0 con 0x7f585406c630 2024-03-31T15:10:43.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.192+0000 7f584be00640 1 --2- 172.21.15.16:0/3979938753 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f58300683b0 0x7f583006a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:43.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.192+0000 7f584be00640 1 --2- 172.21.15.16:0/3979938753 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f58300683b0 0x7f583006a870 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f5854153270 tx=0x7f583c008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:10:43.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.196+0000 7f5851600640 1 -- 172.21.15.16:0/3979938753 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f58440c5020 con 0x7f585406c630 2024-03-31T15:10:43.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.408+0000 7f585a200640 1 -- 172.21.15.16:0/3979938753 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f585404f8c0 con 0x7f585406c630 2024-03-31T15:10:43.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.408+0000 7f5851600640 1 -- 172.21.15.16:0/3979938753 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+164 (secure 0 0 0) 0x7f5844016110 con 0x7f585406c630 2024-03-31T15:10:43.400 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:10:43.400 INFO:teuthology.orchestra.run.smithi016.stdout:{"status":"HEALTH_ERR","checks":{"MDS_DAMAGE":{"severity":"HEALTH_ERR","summary":{"message":"1 MDSs report damaged metadata","count":1},"muted":false}},"mutes":[]} 2024-03-31T15:10:43.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.408+0000 7f585a200640 1 -- 172.21.15.16:0/3979938753 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f58300683b0 msgr2=0x7f583006a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:10:43.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.408+0000 7f585a200640 1 --2- 172.21.15.16:0/3979938753 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f58300683b0 0x7f583006a870 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f5854153270 tx=0x7f583c008040 comp rx=0 tx=0).stop 2024-03-31T15:10:43.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.408+0000 7f585a200640 1 -- 172.21.15.16:0/3979938753 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f585406c630 msgr2=0x7f585414dfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:10:43.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.408+0000 7f585a200640 1 --2- 172.21.15.16:0/3979938753 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f585406c630 0x7f585414dfb0 secure :-1 s=READY pgs=1197 cs=0 l=1 rev1=1 crypto rx=0x7f58440ae040 tx=0x7f584401aca0 comp rx=0 tx=0).stop 2024-03-31T15:10:43.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.408+0000 7f585a200640 1 -- 172.21.15.16:0/3979938753 shutdown_connections 2024-03-31T15:10:43.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.408+0000 7f585a200640 1 --2- 172.21.15.16:0/3979938753 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f58300683b0 0x7f583006a870 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:43.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.408+0000 7f585a200640 1 --2- 172.21.15.16:0/3979938753 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f585415a9a0 0x7f58540cbd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:43.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.408+0000 7f585a200640 1 --2- 172.21.15.16:0/3979938753 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f585415bf40 0x7f585414e4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:43.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.408+0000 7f585a200640 1 --2- 172.21.15.16:0/3979938753 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f585406c630 0x7f585414dfb0 unknown :-1 s=CLOSED pgs=1197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:43.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.408+0000 7f585a200640 1 -- 172.21.15.16:0/3979938753 >> 172.21.15.16:0/3979938753 conn(0x7f5854071c20 msgr2=0x7f585406feb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:10:43.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.408+0000 7f585a200640 1 -- 172.21.15.16:0/3979938753 shutdown_connections 2024-03-31T15:10:43.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.408+0000 7f585a200640 1 -- 172.21.15.16:0/3979938753 wait complete. 2024-03-31T15:10:43.421 DEBUG:tasks.ceph_test_case:wait_until_true: success in 0s and 0 retries 2024-03-31T15:10:43.421 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:10:43.421 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.6:0 scrub start /dir recursive,repair 2024-03-31T15:10:43.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.580+0000 7f88e7c00640 1 Processor -- start 2024-03-31T15:10:43.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.580+0000 7f88e7c00640 1 -- start start 2024-03-31T15:10:43.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.580+0000 7f88e7c00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f88e0147230 0x7f88e00b3dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:43.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.580+0000 7f88e7c00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f88e00d1040 0x7f88e00cb4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:43.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.580+0000 7f88e7c00640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f88e00d14d0 con 0x7f88e00b4300 2024-03-31T15:10:43.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.580+0000 7f88e7c00640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f88e00b4760 con 0x7f88e00d1040 2024-03-31T15:10:43.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.580+0000 7f88e7c00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f88e00d2c30 con 0x7f88e0147230 2024-03-31T15:10:43.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.580+0000 7f88e5c00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f88e00d1040 0x7f88e00cb4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:43.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.580+0000 7f88e5200640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f88e0147230 0x7f88e00b3dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:43.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.580+0000 7f88e5c00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f88e00d1040 0x7f88e00cb4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.16:49142/0 (socket says 172.21.15.16:49142) 2024-03-31T15:10:43.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.580+0000 7f88e5c00640 1 -- 172.21.15.16:0/3995271575 learned_addr learned my addr 172.21.15.16:0/3995271575 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:10:43.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.580+0000 7f88e5c00640 1 -- 172.21.15.16:0/3995271575 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f88e0147230 msgr2=0x7f88e00b3dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:10:43.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.580+0000 7f88e5c00640 1 --2- 172.21.15.16:0/3995271575 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f88e0147230 0x7f88e00b3dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:43.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.584+0000 7f88e5c00640 1 -- 172.21.15.16:0/3995271575 >> v1:172.21.15.89:6789/0 conn(0x7f88e00b4300 legacy=0x7f88e00d0930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:10:43.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.584+0000 7f88e5c00640 1 -- 172.21.15.16:0/3995271575 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88e00d2020 con 0x7f88e00d1040 2024-03-31T15:10:43.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.584+0000 7f88e5200640 1 --2- 172.21.15.16:0/3995271575 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f88e0147230 0x7f88e00b3dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:10:43.577 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.584+0000 7f88e5c00640 1 --2- 172.21.15.16:0/3995271575 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f88e00d1040 0x7f88e00cb4b0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f88d8009e50 tx=0x7f88d801ca30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=15a68e627d04786 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:10:43.577 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.584+0000 7f88dfe00640 1 -- 172.21.15.16:0/3995271575 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88d80a57f0 con 0x7f88e00d1040 2024-03-31T15:10:43.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.584+0000 7f88dfe00640 1 -- 172.21.15.16:0/3995271575 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f88d80a5990 con 0x7f88e00d1040 2024-03-31T15:10:43.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.584+0000 7f88dfe00640 1 -- 172.21.15.16:0/3995271575 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88d80a5d60 con 0x7f88e00d1040 2024-03-31T15:10:43.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.584+0000 7f88e7c00640 1 -- 172.21.15.16:0/3995271575 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f88e00d1040 msgr2=0x7f88e00cb4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:10:43.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.584+0000 7f88e7c00640 1 --2- 172.21.15.16:0/3995271575 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f88e00d1040 0x7f88e00cb4b0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f88d8009e50 tx=0x7f88d801ca30 comp rx=0 tx=0).stop 2024-03-31T15:10:43.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.584+0000 7f88e7c00640 1 -- 172.21.15.16:0/3995271575 shutdown_connections 2024-03-31T15:10:43.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.584+0000 7f88e7c00640 1 --2- 172.21.15.16:0/3995271575 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f88e00d1040 0x7f88e00cb4b0 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:43.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.584+0000 7f88e7c00640 1 --2- 172.21.15.16:0/3995271575 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f88e0147230 0x7f88e00b3dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:43.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.584+0000 7f88e7c00640 1 -- 172.21.15.16:0/3995271575 >> 172.21.15.16:0/3995271575 conn(0x7f88e006b7f0 msgr2=0x7f88e0067330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:10:43.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.588+0000 7f88e7c00640 1 -- 172.21.15.16:0/3995271575 shutdown_connections 2024-03-31T15:10:43.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.588+0000 7f88e7c00640 1 -- 172.21.15.16:0/3995271575 wait complete. 2024-03-31T15:10:43.580 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.588+0000 7f88e7c00640 1 Processor -- start 2024-03-31T15:10:43.580 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.588+0000 7f88e7c00640 1 -- start start 2024-03-31T15:10:43.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.588+0000 7f88e7c00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f88e00b4300 0x7f88e015e000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:43.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.588+0000 7f88e7c00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f88e00d1040 0x7f88e0145490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:43.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.588+0000 7f88e7c00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f88e0147230 0x7f88e00a9ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:43.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.588+0000 7f88e7c00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f88e00cc200 con 0x7f88e00b4300 2024-03-31T15:10:43.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.588+0000 7f88e7c00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f88e006cfd0 con 0x7f88e00d1040 2024-03-31T15:10:43.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.588+0000 7f88e7c00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f88e00cc380 con 0x7f88e0147230 2024-03-31T15:10:43.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.588+0000 7f88e5200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f88e00b4300 0x7f88e015e000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:43.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.588+0000 7f88e5200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f88e00b4300 0x7f88e015e000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41138/0 (socket says 172.21.15.16:41138) 2024-03-31T15:10:43.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.588+0000 7f88e5200640 1 -- 172.21.15.16:0/3044523858 learned_addr learned my addr 172.21.15.16:0/3044523858 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:10:43.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.588+0000 7f88d7e00640 1 --2- 172.21.15.16:0/3044523858 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f88e00d1040 0x7f88e0145490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:43.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.588+0000 7f88e5c00640 1 --2- 172.21.15.16:0/3044523858 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f88e0147230 0x7f88e00a9ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:43.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.588+0000 7f88e5200640 1 -- 172.21.15.16:0/3044523858 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f88e0147230 msgr2=0x7f88e00a9ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:10:43.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.588+0000 7f88e5200640 1 --2- 172.21.15.16:0/3044523858 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f88e0147230 0x7f88e00a9ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:43.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.588+0000 7f88e5200640 1 -- 172.21.15.16:0/3044523858 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f88e00d1040 msgr2=0x7f88e0145490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:10:43.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.588+0000 7f88e5200640 1 --2- 172.21.15.16:0/3044523858 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f88e00d1040 0x7f88e0145490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:43.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.588+0000 7f88e5200640 1 -- 172.21.15.16:0/3044523858 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88e015e540 con 0x7f88e00b4300 2024-03-31T15:10:43.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.588+0000 7f88e5200640 1 --2- 172.21.15.16:0/3044523858 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f88e00b4300 0x7f88e015e000 secure :-1 s=READY pgs=1198 cs=0 l=1 rev1=1 crypto rx=0x7f88d000ebe0 tx=0x7f88d000c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:10:43.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.592+0000 7f88dd600640 1 -- 172.21.15.16:0/3044523858 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88d0017070 con 0x7f88e00b4300 2024-03-31T15:10:43.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.592+0000 7f88dd600640 1 -- 172.21.15.16:0/3044523858 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f88d000cc30 con 0x7f88e00b4300 2024-03-31T15:10:43.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.592+0000 7f88dd600640 1 -- 172.21.15.16:0/3044523858 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88d0013470 con 0x7f88e00b4300 2024-03-31T15:10:43.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.592+0000 7f88e7c00640 1 -- 172.21.15.16:0/3044523858 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f88e00aa9d0 con 0x7f88e00b4300 2024-03-31T15:10:43.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.592+0000 7f88e7c00640 1 -- 172.21.15.16:0/3044523858 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f88e0144c90 con 0x7f88e00b4300 2024-03-31T15:10:43.585 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.592+0000 7f88dd600640 1 -- 172.21.15.16:0/3044523858 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f88d001a070 con 0x7f88e00b4300 2024-03-31T15:10:43.585 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.592+0000 7f88dd600640 1 --2- 172.21.15.16:0/3044523858 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f88bc068380 0x7f88bc06a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:43.585 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.592+0000 7f88dd600640 1 -- 172.21.15.16:0/3044523858 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 5824+0+0 (secure 0 0 0) 0x7f88d0086910 con 0x7f88e00b4300 2024-03-31T15:10:43.586 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.592+0000 7f88d7e00640 1 --2- 172.21.15.16:0/3044523858 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f88bc068380 0x7f88bc06a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:43.586 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.596+0000 7f88eb1b7000 1 Processor -- start 2024-03-31T15:10:43.587 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.596+0000 7f88eb1b7000 1 -- start start 2024-03-31T15:10:43.587 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.596+0000 7f88d7e00640 1 --2- 172.21.15.16:0/3044523858 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f88bc068380 0x7f88bc06a840 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f88e00b7520 tx=0x7f88c8009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:10:43.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.596+0000 7f88eb1b7000 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55f40548a840 0x55f40548ac20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:43.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.596+0000 7f88eb1b7000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f40548b160 0x55f40543dc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:43.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.596+0000 7f88eb1b7000 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f40543e1d0 0x55f40543e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:43.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.596+0000 7f88eb1b7000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55f405328180 con 0x55f40548b160 2024-03-31T15:10:43.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.596+0000 7f88eb1b7000 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55f40528fc90 con 0x55f40543e1d0 2024-03-31T15:10:43.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.596+0000 7f88eb1b7000 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x55f40522ca00 con 0x55f40548a840 2024-03-31T15:10:43.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.596+0000 7f88d7e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f40548b160 0x55f40543dc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:43.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.596+0000 7f88d7e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f40548b160 0x55f40543dc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41146/0 (socket says 172.21.15.16:41146) 2024-03-31T15:10:43.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.596+0000 7f88d7e00640 1 -- 172.21.15.16:0/3108499656 learned_addr learned my addr 172.21.15.16:0/3108499656 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:10:43.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.596+0000 7f88e5c00640 1 --2- 172.21.15.16:0/3108499656 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f40543e1d0 0x55f40543e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:43.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.596+0000 7f88e5200640 1 --2- 172.21.15.16:0/3108499656 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55f40548a840 0x55f40548ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:43.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.596+0000 7f88d7e00640 1 -- 172.21.15.16:0/3108499656 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55f40548a840 msgr2=0x55f40548ac20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:10:43.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.596+0000 7f88d7e00640 1 --2- 172.21.15.16:0/3108499656 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55f40548a840 0x55f40548ac20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:43.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.596+0000 7f88d7e00640 1 -- 172.21.15.16:0/3108499656 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f40543e1d0 msgr2=0x55f40543e5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:10:43.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.596+0000 7f88d7e00640 1 --2- 172.21.15.16:0/3108499656 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f40543e1d0 0x55f40543e5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:43.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.596+0000 7f88d7e00640 1 -- 172.21.15.16:0/3108499656 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f405215510 con 0x55f40548b160 2024-03-31T15:10:43.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.596+0000 7f88d7e00640 1 --2- 172.21.15.16:0/3108499656 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f40548b160 0x55f40543dc90 secure :-1 s=READY pgs=1199 cs=0 l=1 rev1=1 crypto rx=0x7f88c800fb10 tx=0x7f88c8000ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7820968166ac014 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:10:43.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.596+0000 7f88d5600640 1 -- 172.21.15.16:0/3108499656 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88c8025070 con 0x55f40548b160 2024-03-31T15:10:43.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.596+0000 7f88d5600640 1 -- 172.21.15.16:0/3108499656 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f88c8015bf0 con 0x55f40548b160 2024-03-31T15:10:43.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.596+0000 7f88d5600640 1 -- 172.21.15.16:0/3108499656 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88c8020400 con 0x55f40548b160 2024-03-31T15:10:43.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.596+0000 7f88eb1b7000 1 -- 172.21.15.16:0/3108499656 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f40548b160 msgr2=0x55f40543dc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:10:43.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.596+0000 7f88eb1b7000 1 --2- 172.21.15.16:0/3108499656 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f40548b160 0x55f40543dc90 secure :-1 s=READY pgs=1199 cs=0 l=1 rev1=1 crypto rx=0x7f88c800fb10 tx=0x7f88c8000ee0 comp rx=0 tx=0).stop 2024-03-31T15:10:43.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.600+0000 7f88eb1b7000 1 -- 172.21.15.16:0/3108499656 shutdown_connections 2024-03-31T15:10:43.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.600+0000 7f88eb1b7000 1 --2- 172.21.15.16:0/3108499656 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f40543e1d0 0x55f40543e5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:43.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.600+0000 7f88eb1b7000 1 --2- 172.21.15.16:0/3108499656 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f40548b160 0x55f40543dc90 unknown :-1 s=CLOSED pgs=1199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:43.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.600+0000 7f88eb1b7000 1 --2- 172.21.15.16:0/3108499656 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55f40548a840 0x55f40548ac20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:43.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.600+0000 7f88eb1b7000 1 -- 172.21.15.16:0/3108499656 >> 172.21.15.16:0/3108499656 conn(0x55f40546b5e0 msgr2=0x55f40546d9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:10:43.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.600+0000 7f88eb1b7000 1 -- 172.21.15.16:0/3108499656 shutdown_connections 2024-03-31T15:10:43.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.600+0000 7f88eb1b7000 1 -- 172.21.15.16:0/3108499656 wait complete. 2024-03-31T15:10:43.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.600+0000 7f88eb1b7000 1 -- start start 2024-03-31T15:10:43.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.600+0000 7f88eb1b7000 1 Processor -- start 2024-03-31T15:10:43.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.600+0000 7f88eb1b7000 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55f40543e1d0 0x55f405491990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:43.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.600+0000 7f88eb1b7000 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f40548a840 0x55f405491ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:43.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.600+0000 7f88eb1b7000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f40548b160 0x55f405442d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:43.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.600+0000 7f88eb1b7000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55f40524d200 con 0x55f40548b160 2024-03-31T15:10:43.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.600+0000 7f88eb1b7000 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55f40528bc40 con 0x55f40548a840 2024-03-31T15:10:43.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.600+0000 7f88eb1b7000 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x55f40525bb20 con 0x55f40543e1d0 2024-03-31T15:10:43.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.600+0000 7f88eb1b7000 10 client.0 _ll_register_callbacks cb 0x55f40542b930 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T15:10:43.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.600+0000 7f88e5c00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f40548b160 0x55f405442d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:43.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.600+0000 7f88e5c00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f40548b160 0x55f405442d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41150/0 (socket says 172.21.15.16:41150) 2024-03-31T15:10:43.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.600+0000 7f88e5c00640 1 -- 172.21.15.16:0/3920859974 learned_addr learned my addr 172.21.15.16:0/3920859974 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:10:43.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.600+0000 7f88d7e00640 1 --2- 172.21.15.16:0/3920859974 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f40548a840 0x55f405491ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:43.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.600+0000 7f88e5200640 1 --2- 172.21.15.16:0/3920859974 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55f40543e1d0 0x55f405491990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:43.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.600+0000 7f88e5c00640 1 -- 172.21.15.16:0/3920859974 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55f40543e1d0 msgr2=0x55f405491990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:10:43.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.600+0000 7f88e5c00640 1 --2- 172.21.15.16:0/3920859974 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55f40543e1d0 0x55f405491990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:43.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.600+0000 7f88e5c00640 1 -- 172.21.15.16:0/3920859974 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f40548a840 msgr2=0x55f405491ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:10:43.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.600+0000 7f88e5c00640 1 --2- 172.21.15.16:0/3920859974 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f40548a840 0x55f405491ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:43.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.600+0000 7f88e5c00640 1 -- 172.21.15.16:0/3920859974 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f40529ace0 con 0x55f40548b160 2024-03-31T15:10:43.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.600+0000 7f88e5c00640 1 --2- 172.21.15.16:0/3920859974 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f40548b160 0x55f405442d60 secure :-1 s=READY pgs=1200 cs=0 l=1 rev1=1 crypto rx=0x7f88d80af040 tx=0x7f88d8004a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a0f4c8d34988b848 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:10:43.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.600+0000 7f88ce000640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-31T15:10:43.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.600+0000 7f88ce000640 1 -- 172.21.15.16:0/3920859974 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88d80a5a20 con 0x55f40548b160 2024-03-31T15:10:43.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.600+0000 7f88ce000640 1 -- 172.21.15.16:0/3920859974 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 5824+0+0 (secure 0 0 0) 0x7f88d80a5bc0 con 0x55f40548b160 2024-03-31T15:10:43.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.600+0000 7f88eb1b7000 1 -- 172.21.15.16:0/3920859974 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f88c8015bf0 con 0x55f40548b160 2024-03-31T15:10:43.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.600+0000 7f88ce000640 1 -- 172.21.15.16:0/3920859974 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=68) ==== 24+0+0 (secure 0 0 0) 0x7f88d80a5dc0 con 0x55f40548b160 2024-03-31T15:10:43.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.600+0000 7f88eb1b7000 10 client.7938 fetch_fsmap learned FSMap version 68 2024-03-31T15:10:43.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.600+0000 7f88eb1b7000 1 -- 172.21.15.16:0/3920859974 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=68,monmap=2+}) -- 0x55f40524b7c0 con 0x55f40548b160 2024-03-31T15:10:43.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.600+0000 7f88ce000640 1 -- 172.21.15.16:0/3920859974 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f88c8015bf0 con 0x55f40548b160 2024-03-31T15:10:43.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.600+0000 7f88ce000640 1 -- 172.21.15.16:0/3920859974 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 68) ==== 3494+0+0 (secure 0 0 0) 0x7f88d80cc420 con 0x55f40548b160 2024-03-31T15:10:43.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.604+0000 7f88eb1b7000 10 client.7938 fetch_fsmap finished waiting for FSMap version 68 2024-03-31T15:10:43.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.604+0000 7f88eb1b7000 10 client.7938 resolve_mds: resolved 6:0 to role '6:0' aka daemon mds.c 2024-03-31T15:10:43.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.604+0000 7f88eb1b7000 20 client.7938 populate_metadata read hostname 'smithi016' 2024-03-31T15:10:43.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.604+0000 7f88eb1b7000 1 --2- 172.21.15.16:0/3920859974 >> [v2:172.21.15.16:6832/818074038,v1:172.21.15.16:6834/818074038] conn(0x55f405443b00 0x55f405443ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:43.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.604+0000 7f88eb1b7000 4 client.7938 mds_command: new command op to 7749 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-31T15:10:43.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.604+0000 7f88eb1b7000 1 -- 172.21.15.16:0/3920859974 --> [v2:172.21.15.16:6832/818074038,v1:172.21.15.16:6834/818074038] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f40522ac20 con 0x55f405443b00 2024-03-31T15:10:43.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.604+0000 7f88e5200640 1 --2- 172.21.15.16:0/3920859974 >> [v2:172.21.15.16:6832/818074038,v1:172.21.15.16:6834/818074038] conn(0x55f405443b00 0x55f405443ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:43.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.604+0000 7f88e5200640 1 --2- 172.21.15.16:0/3920859974 >> [v2:172.21.15.16:6832/818074038,v1:172.21.15.16:6834/818074038] conn(0x55f405443b00 0x55f405443ee0 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5b7bd200ce429bba server_cookie=772bca105b109cc1 in_seq=0 out_seq=0 2024-03-31T15:10:43.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.604+0000 7f88ce000640 10 client.7938 ms_handle_connect on v2:172.21.15.16:6832/818074038 2024-03-31T15:10:43.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.604+0000 7f88ce000640 1 -- 172.21.15.16:0/3920859974 <== mds.0 v2:172.21.15.16:6832/818074038 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55f40522ac20 con 0x55f405443b00 2024-03-31T15:10:43.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.604+0000 7f88ce000640 10 client.7938 handle_command_reply: tid=0 2024-03-31T15:10:43.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.604+0000 7f88eb1b7000 1 client.7938 shutdown 2024-03-31T15:10:43.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.604+0000 7f88eb1b7000 1 -- 172.21.15.16:0/3920859974 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f40548b160 msgr2=0x55f405442d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:10:43.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.604+0000 7f88eb1b7000 1 --2- 172.21.15.16:0/3920859974 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f40548b160 0x55f405442d60 secure :-1 s=READY pgs=1200 cs=0 l=1 rev1=1 crypto rx=0x7f88d80af040 tx=0x7f88d8004a50 comp rx=0 tx=0).stop 2024-03-31T15:10:43.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.604+0000 7f88eb1b7000 1 -- 172.21.15.16:0/3920859974 shutdown_connections 2024-03-31T15:10:43.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.604+0000 7f88eb1b7000 1 --2- 172.21.15.16:0/3920859974 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f40548b160 0x55f405442d60 unknown :-1 s=CLOSED pgs=1200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:43.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.604+0000 7f88eb1b7000 1 --2- 172.21.15.16:0/3920859974 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f40548a840 0x55f405491ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:43.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.604+0000 7f88eb1b7000 1 --2- 172.21.15.16:0/3920859974 >> [v2:172.21.15.16:6832/818074038,v1:172.21.15.16:6834/818074038] conn(0x55f405443b00 0x55f405443ee0 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:43.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.604+0000 7f88eb1b7000 1 --2- 172.21.15.16:0/3920859974 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55f40543e1d0 0x55f405491990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:43.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.604+0000 7f88eb1b7000 1 -- 172.21.15.16:0/3920859974 >> 172.21.15.16:0/3920859974 conn(0x55f40546b5e0 msgr2=0x55f40546d9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:10:43.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.604+0000 7f88ce000640 0 client.7938 ms_handle_reset on v2:172.21.15.16:6832/818074038 2024-03-31T15:10:43.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.604+0000 7f88eb1b7000 1 -- 172.21.15.16:0/3920859974 shutdown_connections 2024-03-31T15:10:43.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.604+0000 7f88eb1b7000 1 -- 172.21.15.16:0/3920859974 wait complete. 2024-03-31T15:10:43.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.604+0000 7f88eb1b7000 20 client.7938 trim_cache size 0 max 16384 2024-03-31T15:10:43.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.620+0000 7f88eb1b7000 1 Processor -- start 2024-03-31T15:10:43.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.620+0000 7f88eb1b7000 1 -- start start 2024-03-31T15:10:43.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.624+0000 7f88eb1b7000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f40543e1d0 0x55f40543c3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:43.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.624+0000 7f88eb1b7000 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f40548a840 0x55f40543c930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:43.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.624+0000 7f88eb1b7000 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55f40548b160 0x55f40543d130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:43.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.624+0000 7f88eb1b7000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f88e0203800 con 0x55f40543e1d0 2024-03-31T15:10:43.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.624+0000 7f88eb1b7000 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f88e02028b0 con 0x55f40548a840 2024-03-31T15:10:43.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.624+0000 7f88eb1b7000 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f88e00b3430 con 0x55f40548b160 2024-03-31T15:10:43.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.624+0000 7f88e5200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f40543e1d0 0x55f40543c3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:43.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.624+0000 7f88e5200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f40543e1d0 0x55f40543c3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41152/0 (socket says 172.21.15.16:41152) 2024-03-31T15:10:43.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.624+0000 7f88e5200640 1 -- 172.21.15.16:0/1817188739 learned_addr learned my addr 172.21.15.16:0/1817188739 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:10:43.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.624+0000 7f88e5200640 1 -- 172.21.15.16:0/1817188739 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55f40548b160 msgr2=0x55f40543d130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:10:43.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.624+0000 7f88e5200640 1 --2- 172.21.15.16:0/1817188739 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55f40548b160 0x55f40543d130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:43.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.624+0000 7f88d7e00640 1 --2- 172.21.15.16:0/1817188739 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f40548a840 0x55f40543c930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:43.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.624+0000 7f88e5200640 1 -- 172.21.15.16:0/1817188739 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f40548a840 msgr2=0x55f40543c930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:10:43.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.624+0000 7f88e5200640 1 --2- 172.21.15.16:0/1817188739 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f40548a840 0x55f40543c930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:43.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.624+0000 7f88e5200640 1 -- 172.21.15.16:0/1817188739 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f40548af90 con 0x55f40543e1d0 2024-03-31T15:10:43.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.624+0000 7f88d7e00640 1 --2- 172.21.15.16:0/1817188739 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f40548a840 0x55f40543c930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:10:43.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.624+0000 7f88e5200640 1 --2- 172.21.15.16:0/1817188739 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f40543e1d0 0x55f40543c3f0 secure :-1 s=READY pgs=1201 cs=0 l=1 rev1=1 crypto rx=0x7f88d00564a0 tx=0x7f88d0084cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c065dba9bcf9085 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:10:43.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.624+0000 7f88d4c00640 1 -- 172.21.15.16:0/1817188739 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88d006d040 con 0x55f40543e1d0 2024-03-31T15:10:43.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.624+0000 7f88d4c00640 1 -- 172.21.15.16:0/1817188739 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f88d0054aa0 con 0x55f40543e1d0 2024-03-31T15:10:43.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.624+0000 7f88d4c00640 1 -- 172.21.15.16:0/1817188739 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88d0021040 con 0x55f40543e1d0 2024-03-31T15:10:43.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.624+0000 7f88eb1b7000 1 -- 172.21.15.16:0/1817188739 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f40543e1d0 msgr2=0x55f40543c3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:10:43.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.624+0000 7f88eb1b7000 1 --2- 172.21.15.16:0/1817188739 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f40543e1d0 0x55f40543c3f0 secure :-1 s=READY pgs=1201 cs=0 l=1 rev1=1 crypto rx=0x7f88d00564a0 tx=0x7f88d0084cb0 comp rx=0 tx=0).stop 2024-03-31T15:10:43.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.624+0000 7f88eb1b7000 1 -- 172.21.15.16:0/1817188739 shutdown_connections 2024-03-31T15:10:43.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.624+0000 7f88eb1b7000 1 --2- 172.21.15.16:0/1817188739 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55f40548b160 0x55f40543d130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:43.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.624+0000 7f88eb1b7000 1 --2- 172.21.15.16:0/1817188739 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f40548a840 0x55f40543c930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:43.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.624+0000 7f88eb1b7000 1 --2- 172.21.15.16:0/1817188739 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f40543e1d0 0x55f40543c3f0 unknown :-1 s=CLOSED pgs=1201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:43.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.624+0000 7f88eb1b7000 1 -- 172.21.15.16:0/1817188739 >> 172.21.15.16:0/1817188739 conn(0x55f40546b5e0 msgr2=0x55f4054a0b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:10:43.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.624+0000 7f88eb1b7000 1 -- 172.21.15.16:0/1817188739 shutdown_connections 2024-03-31T15:10:43.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.624+0000 7f88eb1b7000 1 -- 172.21.15.16:0/1817188739 wait complete. 2024-03-31T15:10:43.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.624+0000 7f88eb1b7000 1 -- start start 2024-03-31T15:10:43.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.624+0000 7f88eb1b7000 1 Processor -- start 2024-03-31T15:10:43.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.628+0000 7f88eb1b7000 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f40543e1d0 0x55f405489bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:43.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.628+0000 7f88eb1b7000 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55f40548a840 0x55f40548e5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:43.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.628+0000 7f88eb1b7000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f40548b160 0x55f40548eae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:43.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.628+0000 7f88eb1b7000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55f40528bc40 con 0x55f40548b160 2024-03-31T15:10:43.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.628+0000 7f88eb1b7000 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f88e00b3a80 con 0x55f40543e1d0 2024-03-31T15:10:43.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.628+0000 7f88eb1b7000 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x55f405443120 con 0x55f40548a840 2024-03-31T15:10:43.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.628+0000 7f88eb1b7000 10 client.0 _ll_register_callbacks cb 0x7f88d8009c10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T15:10:43.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.628+0000 7f88e5c00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f40548b160 0x55f40548eae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:43.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.628+0000 7f88e5c00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f40548b160 0x55f40548eae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41154/0 (socket says 172.21.15.16:41154) 2024-03-31T15:10:43.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.628+0000 7f88e5c00640 1 -- 172.21.15.16:0/2383858256 learned_addr learned my addr 172.21.15.16:0/2383858256 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:10:43.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.628+0000 7f88e5200640 1 --2- 172.21.15.16:0/2383858256 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f40543e1d0 0x55f405489bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:43.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.628+0000 7f88d7e00640 1 --2- 172.21.15.16:0/2383858256 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55f40548a840 0x55f40548e5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:43.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.628+0000 7f88e5c00640 1 -- 172.21.15.16:0/2383858256 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55f40548a840 msgr2=0x55f40548e5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:10:43.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.628+0000 7f88e5c00640 1 --2- 172.21.15.16:0/2383858256 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55f40548a840 0x55f40548e5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:43.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.628+0000 7f88e5c00640 1 -- 172.21.15.16:0/2383858256 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f40543e1d0 msgr2=0x55f405489bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:10:43.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.628+0000 7f88e5c00640 1 --2- 172.21.15.16:0/2383858256 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f40543e1d0 0x55f405489bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:43.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.628+0000 7f88e5c00640 1 -- 172.21.15.16:0/2383858256 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f40524bc70 con 0x55f40548b160 2024-03-31T15:10:43.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.628+0000 7f88e5c00640 1 --2- 172.21.15.16:0/2383858256 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f40548b160 0x55f40548eae0 secure :-1 s=READY pgs=1202 cs=0 l=1 rev1=1 crypto rx=0x7f88d801c4a0 tx=0x7f88d80197d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e8c28786d3bba18 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:10:43.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.628+0000 7f88ce000640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-31T15:10:43.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.628+0000 7f88ce000640 1 -- 172.21.15.16:0/2383858256 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88d80a59f0 con 0x55f40548b160 2024-03-31T15:10:43.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.628+0000 7f88ce000640 1 -- 172.21.15.16:0/2383858256 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 5824+0+0 (secure 0 0 0) 0x7f88d80a5b90 con 0x55f40548b160 2024-03-31T15:10:43.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.628+0000 7f88eb1b7000 1 -- 172.21.15.16:0/2383858256 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f88d0054aa0 con 0x55f40548b160 2024-03-31T15:10:43.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.628+0000 7f88ce000640 1 -- 172.21.15.16:0/2383858256 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=68) ==== 24+0+0 (secure 0 0 0) 0x7f88d80039a0 con 0x55f40548b160 2024-03-31T15:10:43.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.628+0000 7f88eb1b7000 10 client.7944 fetch_fsmap learned FSMap version 68 2024-03-31T15:10:43.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.628+0000 7f88eb1b7000 1 -- 172.21.15.16:0/2383858256 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=68,monmap=2+}) -- 0x55f40522cc50 con 0x55f40548b160 2024-03-31T15:10:43.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.628+0000 7f88ce000640 1 -- 172.21.15.16:0/2383858256 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f88d0054aa0 con 0x55f40548b160 2024-03-31T15:10:43.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.628+0000 7f88ce000640 1 -- 172.21.15.16:0/2383858256 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 68) ==== 3494+0+0 (secure 0 0 0) 0x7f88d8008040 con 0x55f40548b160 2024-03-31T15:10:43.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.628+0000 7f88eb1b7000 10 client.7944 fetch_fsmap finished waiting for FSMap version 68 2024-03-31T15:10:43.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.628+0000 7f88eb1b7000 10 client.7944 resolve_mds: resolved 6:0 to role '6:0' aka daemon mds.c 2024-03-31T15:10:43.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.628+0000 7f88eb1b7000 20 client.7944 populate_metadata read hostname 'smithi016' 2024-03-31T15:10:43.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.628+0000 7f88eb1b7000 1 --2- 172.21.15.16:0/2383858256 >> [v2:172.21.15.16:6832/818074038,v1:172.21.15.16:6834/818074038] conn(0x55f40548f020 0x55f4054fa740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:43.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.628+0000 7f88eb1b7000 4 client.7944 mds_command: new command op to 7749 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/dir", "scrubops": ["recursive", "repair"]}] 2024-03-31T15:10:43.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.628+0000 7f88eb1b7000 1 -- 172.21.15.16:0/2383858256 --> [v2:172.21.15.16:6832/818074038,v1:172.21.15.16:6834/818074038] -- command(tid 0: {"prefix": "scrub start", "path": "/dir", "scrubops": ["recursive", "repair"]}) -- 0x55f40529d1d0 con 0x55f40548f020 2024-03-31T15:10:43.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.628+0000 7f88e5200640 1 --2- 172.21.15.16:0/2383858256 >> [v2:172.21.15.16:6832/818074038,v1:172.21.15.16:6834/818074038] conn(0x55f40548f020 0x55f4054fa740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:43.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.628+0000 7f88e5200640 1 --2- 172.21.15.16:0/2383858256 >> [v2:172.21.15.16:6832/818074038,v1:172.21.15.16:6834/818074038] conn(0x55f40548f020 0x55f4054fa740 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7f33ff75e0c304d5 server_cookie=1d83e731076f4717 in_seq=0 out_seq=0 2024-03-31T15:10:43.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.628+0000 7f88ce000640 10 client.7944 ms_handle_connect on v2:172.21.15.16:6832/818074038 2024-03-31T15:10:43.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.632+0000 7f88ce000640 1 -- 172.21.15.16:0/2383858256 <== mds.0 v2:172.21.15.16:6832/818074038 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55f40529d1d0 con 0x55f40548f020 2024-03-31T15:10:43.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.632+0000 7f88ce000640 10 client.7944 handle_command_reply: tid=0 2024-03-31T15:10:43.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.632+0000 7f88eb1b7000 1 client.7944 shutdown 2024-03-31T15:10:43.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.632+0000 7f88eb1b7000 1 -- 172.21.15.16:0/2383858256 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f40548b160 msgr2=0x55f40548eae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:10:43.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.632+0000 7f88eb1b7000 1 --2- 172.21.15.16:0/2383858256 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f40548b160 0x55f40548eae0 secure :-1 s=READY pgs=1202 cs=0 l=1 rev1=1 crypto rx=0x7f88d801c4a0 tx=0x7f88d80197d0 comp rx=0 tx=0).stop 2024-03-31T15:10:43.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.632+0000 7f88eb1b7000 1 -- 172.21.15.16:0/2383858256 shutdown_connections 2024-03-31T15:10:43.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.632+0000 7f88eb1b7000 1 --2- 172.21.15.16:0/2383858256 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f40548b160 0x55f40548eae0 unknown :-1 s=CLOSED pgs=1202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:43.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.632+0000 7f88eb1b7000 1 --2- 172.21.15.16:0/2383858256 >> [v2:172.21.15.16:6832/818074038,v1:172.21.15.16:6834/818074038] conn(0x55f40548f020 0x55f4054fa740 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:43.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.632+0000 7f88eb1b7000 1 --2- 172.21.15.16:0/2383858256 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55f40548a840 0x55f40548e5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:43.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.632+0000 7f88eb1b7000 1 --2- 172.21.15.16:0/2383858256 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f40543e1d0 0x55f405489bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:43.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.632+0000 7f88eb1b7000 1 -- 172.21.15.16:0/2383858256 >> 172.21.15.16:0/2383858256 conn(0x55f40546b5e0 msgr2=0x55f4054a0b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:10:43.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.632+0000 7f88ce000640 0 client.7944 ms_handle_reset on v2:172.21.15.16:6832/818074038 2024-03-31T15:10:43.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.632+0000 7f88eb1b7000 1 -- 172.21.15.16:0/2383858256 shutdown_connections 2024-03-31T15:10:43.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.632+0000 7f88eb1b7000 1 -- 172.21.15.16:0/2383858256 wait complete. 2024-03-31T15:10:43.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.632+0000 7f88eb1b7000 20 client.7944 trim_cache size 0 max 16384 2024-03-31T15:10:43.628 INFO:tasks.ceph.mds.c.smithi016.stderr:2024-03-31T15:10:43.632+0000 7f8532000640 -1 mds.0.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000000 [...2,head] /dir/ auth v13 pv16 ap=3 DIRTYPARENT f(v0 m2024-03-31T15:09:59.315089+0000 1=1+0)->f(v1 m2024-03-31T15:09:59.315089+0000) n(v0 rc2024-03-31T15:09:59.431625+0000 2=1+1)->n(v0 rc2024-03-31T15:09:59.431625+0000 2=1+1) (inest lock dirty) (ifile lock->sync w=1 flushing) | dirtyscattered=2 lock=1 dirfrag=1 dirtyrstat=1 dirtyparent=1 waiter=1 authpin=1 scrubqueue=0 0x55f996656100]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":-2,"ondisk_value":"(-1)0x0:[]//[]","memoryvalue":"(12)0x10000000000:[<0x1/dir v13>]//[]","error_str":"failed to decode on-disk backtrace (0 bytes)!"},"raw_stats":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n(v0 1=0+1)","memory_value.dirstat":"f(v0 m2024-03-31T15:09:59.315089+0000 1=1+0)","memory_value.rstat":"n(v0 rc2024-03-31T15:09:59.431625+0000 2=1+1)","error_str":"freshly-calculated rstats don't match existing ones (will be fixed)"},"return_code":-2} 2024-03-31T15:10:43.628 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-03-31T15:10:43.628 INFO:teuthology.orchestra.run.smithi016.stdout: "return_code": 0, 2024-03-31T15:10:43.628 INFO:teuthology.orchestra.run.smithi016.stdout: "scrub_tag": "e29c520e-8750-442e-b654-00f09d3161bd", 2024-03-31T15:10:43.628 INFO:teuthology.orchestra.run.smithi016.stdout: "mode": "asynchronous" 2024-03-31T15:10:43.628 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-03-31T15:10:43.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.632+0000 7f88e7c00640 1 -- 172.21.15.16:0/3044523858 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f88bc068380 msgr2=0x7f88bc06a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:10:43.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.632+0000 7f88e7c00640 1 --2- 172.21.15.16:0/3044523858 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f88bc068380 0x7f88bc06a840 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f88e00b7520 tx=0x7f88c8009040 comp rx=0 tx=0).stop 2024-03-31T15:10:43.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.632+0000 7f88e7c00640 1 -- 172.21.15.16:0/3044523858 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f88e00b4300 msgr2=0x7f88e015e000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:10:43.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.632+0000 7f88e7c00640 1 --2- 172.21.15.16:0/3044523858 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f88e00b4300 0x7f88e015e000 secure :-1 s=READY pgs=1198 cs=0 l=1 rev1=1 crypto rx=0x7f88d000ebe0 tx=0x7f88d000c520 comp rx=0 tx=0).stop 2024-03-31T15:10:43.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.632+0000 7f88e7c00640 1 -- 172.21.15.16:0/3044523858 shutdown_connections 2024-03-31T15:10:43.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.632+0000 7f88e7c00640 1 --2- 172.21.15.16:0/3044523858 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f88bc068380 0x7f88bc06a840 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:43.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.632+0000 7f88e7c00640 1 --2- 172.21.15.16:0/3044523858 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f88e0147230 0x7f88e00a9ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:43.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.632+0000 7f88e7c00640 1 --2- 172.21.15.16:0/3044523858 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f88e00d1040 0x7f88e0145490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:43.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.632+0000 7f88e7c00640 1 --2- 172.21.15.16:0/3044523858 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f88e00b4300 0x7f88e015e000 unknown :-1 s=CLOSED pgs=1198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:43.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.632+0000 7f88e7c00640 1 -- 172.21.15.16:0/3044523858 >> 172.21.15.16:0/3044523858 conn(0x7f88e006b7f0 msgr2=0x7f88e00b1da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:10:43.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.632+0000 7f88e7c00640 1 -- 172.21.15.16:0/3044523858 shutdown_connections 2024-03-31T15:10:43.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.632+0000 7f88e7c00640 1 -- 172.21.15.16:0/3044523858 wait complete. 2024-03-31T15:10:43.644 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:10:43.644 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.6:0 scrub status 2024-03-31T15:10:43.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.792+0000 7f58dd400640 1 Processor -- start 2024-03-31T15:10:43.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.792+0000 7f58dd400640 1 -- start start 2024-03-31T15:10:43.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.796+0000 7f58dd400640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f58d806f920 0x7f58d806fd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:43.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.796+0000 7f58dd400640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f58d80cf150 0x7f58d81419c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:43.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.796+0000 7f58dd400640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f58d8070240 con 0x7f58d80d0a60 2024-03-31T15:10:43.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.796+0000 7f58dd400640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f58d813f670 con 0x7f58d806f920 2024-03-31T15:10:43.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.796+0000 7f58dd400640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f58d815ca70 con 0x7f58d80cf150 2024-03-31T15:10:43.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.796+0000 7f58d6800640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f58d806f920 0x7f58d806fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:43.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.796+0000 7f58d6800640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f58d806f920 0x7f58d806fd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41170/0 (socket says 172.21.15.16:41170) 2024-03-31T15:10:43.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.796+0000 7f58d6800640 1 -- 172.21.15.16:0/632343004 learned_addr learned my addr 172.21.15.16:0/632343004 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:10:43.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.796+0000 7f58d7200640 1 --2- 172.21.15.16:0/632343004 >> v2:172.21.15.89:3300/0 conn(0x7f58d80cf150 0x7f58d81419c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:43.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.796+0000 7f58d6800640 1 -- 172.21.15.16:0/632343004 >> v2:172.21.15.89:3300/0 conn(0x7f58d80cf150 msgr2=0x7f58d81419c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:10:43.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.796+0000 7f58d6800640 1 --2- 172.21.15.16:0/632343004 >> v2:172.21.15.89:3300/0 conn(0x7f58d80cf150 0x7f58d81419c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:43.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.796+0000 7f58d6800640 1 -- 172.21.15.16:0/632343004 >> v1:172.21.15.89:6789/0 conn(0x7f58d80d0a60 legacy=0x7f58d80cea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:10:43.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.796+0000 7f58d6800640 1 -- 172.21.15.16:0/632343004 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58d815cd50 con 0x7f58d806f920 2024-03-31T15:10:43.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.800+0000 7f58d6800640 1 --2- 172.21.15.16:0/632343004 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f58d806f920 0x7f58d806fd00 secure :-1 s=READY pgs=1203 cs=0 l=1 rev1=1 crypto rx=0x7f58d80d2080 tx=0x7f58c801c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af205dafaf4ea40e server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:10:43.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.800+0000 7f58d5e00640 1 -- 172.21.15.16:0/632343004 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58c80b0070 con 0x7f58d806f920 2024-03-31T15:10:43.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.800+0000 7f58d5e00640 1 -- 172.21.15.16:0/632343004 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f58c80ab600 con 0x7f58d806f920 2024-03-31T15:10:43.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.800+0000 7f58d5e00640 1 -- 172.21.15.16:0/632343004 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58c80ab9d0 con 0x7f58d806f920 2024-03-31T15:10:43.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.800+0000 7f58dd400640 1 -- 172.21.15.16:0/632343004 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f58d806f920 msgr2=0x7f58d806fd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:10:43.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.800+0000 7f58dd400640 1 --2- 172.21.15.16:0/632343004 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f58d806f920 0x7f58d806fd00 secure :-1 s=READY pgs=1203 cs=0 l=1 rev1=1 crypto rx=0x7f58d80d2080 tx=0x7f58c801c490 comp rx=0 tx=0).stop 2024-03-31T15:10:43.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.800+0000 7f58dd400640 1 -- 172.21.15.16:0/632343004 shutdown_connections 2024-03-31T15:10:43.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.800+0000 7f58dd400640 1 --2- 172.21.15.16:0/632343004 >> v2:172.21.15.89:3300/0 conn(0x7f58d80cf150 0x7f58d81419c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:43.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.800+0000 7f58dd400640 1 --2- 172.21.15.16:0/632343004 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f58d806f920 0x7f58d806fd00 unknown :-1 s=CLOSED pgs=1203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:43.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.800+0000 7f58dd400640 1 -- 172.21.15.16:0/632343004 >> 172.21.15.16:0/632343004 conn(0x7f58d8067930 msgr2=0x7f58d80b3ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:10:43.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.800+0000 7f58dd400640 1 -- 172.21.15.16:0/632343004 shutdown_connections 2024-03-31T15:10:43.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.800+0000 7f58dd400640 1 -- 172.21.15.16:0/632343004 wait complete. 2024-03-31T15:10:43.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.800+0000 7f58dd400640 1 Processor -- start 2024-03-31T15:10:43.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.800+0000 7f58dd400640 1 -- start start 2024-03-31T15:10:43.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.804+0000 7f58dd400640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f58d806f920 0x7f58d806e360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:43.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.804+0000 7f58dd400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f58d80cf150 0x7f58d813ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:43.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.804+0000 7f58dd400640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f58d80d0a60 0x7f58d815d850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:43.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.804+0000 7f58dd400640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f58d8169230 con 0x7f58d80cf150 2024-03-31T15:10:43.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.804+0000 7f58dd400640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f58d81690b0 con 0x7f58d80d0a60 2024-03-31T15:10:43.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.804+0000 7f58dd400640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f58d81693b0 con 0x7f58d806f920 2024-03-31T15:10:43.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.804+0000 7f58cde00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f58d80cf150 0x7f58d813ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:43.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.804+0000 7f58cde00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f58d80cf150 0x7f58d813ecd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41178/0 (socket says 172.21.15.16:41178) 2024-03-31T15:10:43.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.804+0000 7f58cde00640 1 -- 172.21.15.16:0/3254927488 learned_addr learned my addr 172.21.15.16:0/3254927488 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:10:43.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.804+0000 7f58d7200640 1 --2- 172.21.15.16:0/3254927488 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f58d80d0a60 0x7f58d815d850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:43.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.804+0000 7f58d6800640 1 --2- 172.21.15.16:0/3254927488 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f58d806f920 0x7f58d806e360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:43.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.804+0000 7f58cde00640 1 -- 172.21.15.16:0/3254927488 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f58d806f920 msgr2=0x7f58d806e360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:10:43.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.804+0000 7f58cde00640 1 --2- 172.21.15.16:0/3254927488 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f58d806f920 0x7f58d806e360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:43.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.804+0000 7f58cde00640 1 -- 172.21.15.16:0/3254927488 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f58d80d0a60 msgr2=0x7f58d815d850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:10:43.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.804+0000 7f58cde00640 1 --2- 172.21.15.16:0/3254927488 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f58d80d0a60 0x7f58d815d850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:43.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.804+0000 7f58cde00640 1 -- 172.21.15.16:0/3254927488 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58d8159670 con 0x7f58d80cf150 2024-03-31T15:10:43.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.804+0000 7f58cde00640 1 --2- 172.21.15.16:0/3254927488 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f58d80cf150 0x7f58d813ecd0 secure :-1 s=READY pgs=1204 cs=0 l=1 rev1=1 crypto rx=0x7f58c000c730 tx=0x7f58c000cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:10:43.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.804+0000 7f58cf400640 1 -- 172.21.15.16:0/3254927488 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58c0016020 con 0x7f58d80cf150 2024-03-31T15:10:43.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.804+0000 7f58cf400640 1 -- 172.21.15.16:0/3254927488 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f58c0006d80 con 0x7f58d80cf150 2024-03-31T15:10:43.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.804+0000 7f58dd400640 1 -- 172.21.15.16:0/3254927488 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f58d8166ea0 con 0x7f58d80cf150 2024-03-31T15:10:43.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.804+0000 7f58cf400640 1 -- 172.21.15.16:0/3254927488 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58c0003bc0 con 0x7f58d80cf150 2024-03-31T15:10:43.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.804+0000 7f58dd400640 1 -- 172.21.15.16:0/3254927488 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f58d8166850 con 0x7f58d80cf150 2024-03-31T15:10:43.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.804+0000 7f58cf400640 1 -- 172.21.15.16:0/3254927488 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f58c000a7f0 con 0x7f58d80cf150 2024-03-31T15:10:43.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.808+0000 7f58cf400640 1 --2- 172.21.15.16:0/3254927488 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f58b40683b0 0x7f58b406a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:43.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.808+0000 7f58cf400640 1 -- 172.21.15.16:0/3254927488 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 5824+0+0 (secure 0 0 0) 0x7f58c00864c0 con 0x7f58d80cf150 2024-03-31T15:10:43.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.808+0000 7f58e0930000 1 Processor -- start 2024-03-31T15:10:43.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.808+0000 7f58d6800640 1 --2- 172.21.15.16:0/3254927488 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f58b40683b0 0x7f58b406a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:43.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.808+0000 7f58e0930000 1 -- start start 2024-03-31T15:10:43.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.808+0000 7f58d6800640 1 --2- 172.21.15.16:0/3254927488 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f58b40683b0 0x7f58b406a870 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f58c80ad000 tx=0x7f58c8008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:10:43.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.808+0000 7f58e0930000 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55f804ab92d0 0x55f804ab96b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:43.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.808+0000 7f58e0930000 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f804ab9bf0 0x55f804a6c470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:43.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.808+0000 7f58e0930000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f804a6c9b0 0x55f804a6cd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:43.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.808+0000 7f58e0930000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55f804955af0 con 0x55f804a6c9b0 2024-03-31T15:10:43.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.808+0000 7f58e0930000 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55f8048beb00 con 0x55f804ab9bf0 2024-03-31T15:10:43.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.808+0000 7f58e0930000 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x55f80485b940 con 0x55f804ab92d0 2024-03-31T15:10:43.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.808+0000 7f58d7200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f804a6c9b0 0x55f804a6cd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:43.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.808+0000 7f58d7200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f804a6c9b0 0x55f804a6cd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41184/0 (socket says 172.21.15.16:41184) 2024-03-31T15:10:43.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.808+0000 7f58d7200640 1 -- 172.21.15.16:0/2536172757 learned_addr learned my addr 172.21.15.16:0/2536172757 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:10:43.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.808+0000 7f58d6800640 1 --2- 172.21.15.16:0/2536172757 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55f804ab92d0 0x55f804ab96b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:43.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.808+0000 7f58cde00640 1 --2- 172.21.15.16:0/2536172757 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f804ab9bf0 0x55f804a6c470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:43.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.808+0000 7f58d7200640 1 -- 172.21.15.16:0/2536172757 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55f804ab92d0 msgr2=0x55f804ab96b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:10:43.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.808+0000 7f58d7200640 1 --2- 172.21.15.16:0/2536172757 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55f804ab92d0 0x55f804ab96b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:43.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.808+0000 7f58d7200640 1 -- 172.21.15.16:0/2536172757 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f804ab9bf0 msgr2=0x55f804a6c470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:10:43.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.808+0000 7f58d7200640 1 --2- 172.21.15.16:0/2536172757 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f804ab9bf0 0x55f804a6c470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:43.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.808+0000 7f58d7200640 1 -- 172.21.15.16:0/2536172757 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f804844380 con 0x55f804a6c9b0 2024-03-31T15:10:43.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.808+0000 7f58d7200640 1 --2- 172.21.15.16:0/2536172757 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f804a6c9b0 0x55f804a6cd90 secure :-1 s=READY pgs=1205 cs=0 l=1 rev1=1 crypto rx=0x7f58d0007940 tx=0x7f58d0007e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dffc9a3801f2f414 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:10:43.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.808+0000 7f58c7400640 1 -- 172.21.15.16:0/2536172757 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58d0018070 con 0x55f804a6c9b0 2024-03-31T15:10:43.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.808+0000 7f58c7400640 1 -- 172.21.15.16:0/2536172757 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f58d0010b00 con 0x55f804a6c9b0 2024-03-31T15:10:43.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.808+0000 7f58c7400640 1 -- 172.21.15.16:0/2536172757 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58d0014430 con 0x55f804a6c9b0 2024-03-31T15:10:43.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.808+0000 7f58e0930000 1 -- 172.21.15.16:0/2536172757 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f804a6c9b0 msgr2=0x55f804a6cd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:10:43.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.808+0000 7f58e0930000 1 --2- 172.21.15.16:0/2536172757 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f804a6c9b0 0x55f804a6cd90 secure :-1 s=READY pgs=1205 cs=0 l=1 rev1=1 crypto rx=0x7f58d0007940 tx=0x7f58d0007e00 comp rx=0 tx=0).stop 2024-03-31T15:10:43.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.812+0000 7f58e0930000 1 -- 172.21.15.16:0/2536172757 shutdown_connections 2024-03-31T15:10:43.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.812+0000 7f58e0930000 1 --2- 172.21.15.16:0/2536172757 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f804a6c9b0 0x55f804a6cd90 unknown :-1 s=CLOSED pgs=1205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:43.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.812+0000 7f58e0930000 1 --2- 172.21.15.16:0/2536172757 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f804ab9bf0 0x55f804a6c470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:43.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.812+0000 7f58e0930000 1 --2- 172.21.15.16:0/2536172757 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55f804ab92d0 0x55f804ab96b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:43.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.812+0000 7f58e0930000 1 -- 172.21.15.16:0/2536172757 >> 172.21.15.16:0/2536172757 conn(0x55f804a9a070 msgr2=0x55f804a9c460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:10:43.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.812+0000 7f58e0930000 1 -- 172.21.15.16:0/2536172757 shutdown_connections 2024-03-31T15:10:43.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.812+0000 7f58e0930000 1 -- 172.21.15.16:0/2536172757 wait complete. 2024-03-31T15:10:43.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.812+0000 7f58e0930000 1 -- start start 2024-03-31T15:10:43.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.812+0000 7f58e0930000 1 Processor -- start 2024-03-31T15:10:43.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.812+0000 7f58e0930000 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55f804a6c9b0 0x55f804ac0420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:43.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.812+0000 7f58e0930000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f804ab92d0 0x55f804ac0960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:43.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.812+0000 7f58e0930000 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f804ab9bf0 0x55f804a71520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:43.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.812+0000 7f58e0930000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55f80487c1e0 con 0x55f804ab92d0 2024-03-31T15:10:43.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.812+0000 7f58e0930000 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55f8048baab0 con 0x55f804ab9bf0 2024-03-31T15:10:43.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.812+0000 7f58e0930000 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x55f804a78000 con 0x55f804a6c9b0 2024-03-31T15:10:43.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.812+0000 7f58e0930000 10 client.0 _ll_register_callbacks cb 0x55f804a5a370 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T15:10:43.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.812+0000 7f58cde00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f804ab92d0 0x55f804ac0960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:43.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.812+0000 7f58d6800640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55f804a6c9b0 0x55f804ac0420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:43.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.812+0000 7f58d7200640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f804ab9bf0 0x55f804a71520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:43.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.812+0000 7f58d6800640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55f804a6c9b0 0x55f804ac0420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3301/0 says I am v2:172.21.15.16:57256/0 (socket says 172.21.15.16:57256) 2024-03-31T15:10:43.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.812+0000 7f58d6800640 1 -- 172.21.15.16:0/4141282090 learned_addr learned my addr 172.21.15.16:0/4141282090 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:10:43.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.812+0000 7f58cde00640 1 -- 172.21.15.16:0/4141282090 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55f804a6c9b0 msgr2=0x55f804ac0420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:10:43.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.812+0000 7f58cde00640 1 --2- 172.21.15.16:0/4141282090 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55f804a6c9b0 0x55f804ac0420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:43.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.812+0000 7f58cde00640 1 -- 172.21.15.16:0/4141282090 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f804ab9bf0 msgr2=0x55f804a71520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:10:43.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.812+0000 7f58cde00640 1 --2- 172.21.15.16:0/4141282090 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f804ab9bf0 0x55f804a71520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:43.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.812+0000 7f58cde00640 1 -- 172.21.15.16:0/4141282090 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f80487ab70 con 0x55f804ab92d0 2024-03-31T15:10:43.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.812+0000 7f58cde00640 1 --2- 172.21.15.16:0/4141282090 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f804ab92d0 0x55f804ac0960 secure :-1 s=READY pgs=1206 cs=0 l=1 rev1=1 crypto rx=0x7f58c0084990 tx=0x7f58c005a790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8cf7e8689b9903d9 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:10:43.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.812+0000 7f58bbe00640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-31T15:10:43.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.812+0000 7f58bbe00640 1 -- 172.21.15.16:0/4141282090 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58c00862c0 con 0x55f804ab92d0 2024-03-31T15:10:43.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.812+0000 7f58bbe00640 1 -- 172.21.15.16:0/4141282090 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 5824+0+0 (secure 0 0 0) 0x7f58c0082b60 con 0x55f804ab92d0 2024-03-31T15:10:43.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.812+0000 7f58e0930000 1 -- 172.21.15.16:0/4141282090 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f58d0010b00 con 0x55f804ab92d0 2024-03-31T15:10:43.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.812+0000 7f58bbe00640 1 -- 172.21.15.16:0/4141282090 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=68) ==== 24+0+0 (secure 0 0 0) 0x7f58c0073050 con 0x55f804ab92d0 2024-03-31T15:10:43.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.812+0000 7f58e0930000 10 client.7956 fetch_fsmap learned FSMap version 68 2024-03-31T15:10:43.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.812+0000 7f58e0930000 1 -- 172.21.15.16:0/4141282090 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=68,monmap=2+}) -- 0x55f80485bb90 con 0x55f804ab92d0 2024-03-31T15:10:43.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.812+0000 7f58bbe00640 1 -- 172.21.15.16:0/4141282090 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f58d0010b00 con 0x55f804ab92d0 2024-03-31T15:10:43.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.812+0000 7f58bbe00640 1 -- 172.21.15.16:0/4141282090 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 68) ==== 3494+0+0 (secure 0 0 0) 0x7f58c007f070 con 0x55f804ab92d0 2024-03-31T15:10:43.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.812+0000 7f58e0930000 10 client.7956 fetch_fsmap finished waiting for FSMap version 68 2024-03-31T15:10:43.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.812+0000 7f58e0930000 10 client.7956 resolve_mds: resolved 6:0 to role '6:0' aka daemon mds.c 2024-03-31T15:10:43.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.812+0000 7f58e0930000 20 client.7956 populate_metadata read hostname 'smithi016' 2024-03-31T15:10:43.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.812+0000 7f58e0930000 1 --2- 172.21.15.16:0/4141282090 >> [v2:172.21.15.16:6832/818074038,v1:172.21.15.16:6834/818074038] conn(0x55f804a722c0 0x55f804a726a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:43.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.812+0000 7f58e0930000 4 client.7956 mds_command: new command op to 7749 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-31T15:10:43.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.812+0000 7f58e0930000 1 -- 172.21.15.16:0/4141282090 --> [v2:172.21.15.16:6832/818074038,v1:172.21.15.16:6834/818074038] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f804859dd0 con 0x55f804a722c0 2024-03-31T15:10:43.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.816+0000 7f58d6800640 1 --2- 172.21.15.16:0/4141282090 >> [v2:172.21.15.16:6832/818074038,v1:172.21.15.16:6834/818074038] conn(0x55f804a722c0 0x55f804a726a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:43.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.816+0000 7f58d6800640 1 --2- 172.21.15.16:0/4141282090 >> [v2:172.21.15.16:6832/818074038,v1:172.21.15.16:6834/818074038] conn(0x55f804a722c0 0x55f804a726a0 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e61cc3c53c82241e server_cookie=ed402c9c6b67d41 in_seq=0 out_seq=0 2024-03-31T15:10:43.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.816+0000 7f58bbe00640 10 client.7956 ms_handle_connect on v2:172.21.15.16:6832/818074038 2024-03-31T15:10:43.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.816+0000 7f58bbe00640 1 -- 172.21.15.16:0/4141282090 <== mds.0 v2:172.21.15.16:6832/818074038 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55f804859dd0 con 0x55f804a722c0 2024-03-31T15:10:43.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.816+0000 7f58bbe00640 10 client.7956 handle_command_reply: tid=0 2024-03-31T15:10:43.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.816+0000 7f58e0930000 1 client.7956 shutdown 2024-03-31T15:10:43.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.816+0000 7f58e0930000 1 -- 172.21.15.16:0/4141282090 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f804ab92d0 msgr2=0x55f804ac0960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:10:43.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.816+0000 7f58e0930000 1 --2- 172.21.15.16:0/4141282090 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f804ab92d0 0x55f804ac0960 secure :-1 s=READY pgs=1206 cs=0 l=1 rev1=1 crypto rx=0x7f58c0084990 tx=0x7f58c005a790 comp rx=0 tx=0).stop 2024-03-31T15:10:43.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.816+0000 7f58e0930000 1 -- 172.21.15.16:0/4141282090 shutdown_connections 2024-03-31T15:10:43.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.816+0000 7f58e0930000 1 --2- 172.21.15.16:0/4141282090 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f804ab9bf0 0x55f804a71520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:43.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.816+0000 7f58e0930000 1 --2- 172.21.15.16:0/4141282090 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f804ab92d0 0x55f804ac0960 unknown :-1 s=CLOSED pgs=1206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:43.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.816+0000 7f58e0930000 1 --2- 172.21.15.16:0/4141282090 >> [v2:172.21.15.16:6832/818074038,v1:172.21.15.16:6834/818074038] conn(0x55f804a722c0 0x55f804a726a0 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:43.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.816+0000 7f58e0930000 1 --2- 172.21.15.16:0/4141282090 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55f804a6c9b0 0x55f804ac0420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:43.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.816+0000 7f58bbe00640 0 client.7956 ms_handle_reset on v2:172.21.15.16:6832/818074038 2024-03-31T15:10:43.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.816+0000 7f58e0930000 1 -- 172.21.15.16:0/4141282090 >> 172.21.15.16:0/4141282090 conn(0x55f804a9a070 msgr2=0x55f804a9c460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:10:43.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.816+0000 7f58e0930000 1 -- 172.21.15.16:0/4141282090 shutdown_connections 2024-03-31T15:10:43.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.816+0000 7f58e0930000 1 -- 172.21.15.16:0/4141282090 wait complete. 2024-03-31T15:10:43.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.816+0000 7f58e0930000 20 client.7956 trim_cache size 0 max 16384 2024-03-31T15:10:43.825 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.832+0000 7f58e0930000 1 Processor -- start 2024-03-31T15:10:43.825 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.832+0000 7f58e0930000 1 -- start start 2024-03-31T15:10:43.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.832+0000 7f58e0930000 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f804a6c9b0 0x55f804acf5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:43.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.832+0000 7f58e0930000 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55f804ab92d0 0x55f804acfaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:43.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.832+0000 7f58e0930000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f804ab9bf0 0x55f804ad02f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:43.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.832+0000 7f58e0930000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f58d806c730 con 0x55f804ab9bf0 2024-03-31T15:10:43.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.832+0000 7f58e0930000 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f58d806cd30 con 0x55f804a6c9b0 2024-03-31T15:10:43.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.832+0000 7f58e0930000 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x55f80488a150 con 0x55f804ab92d0 2024-03-31T15:10:43.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.836+0000 7f58d7200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f804ab9bf0 0x55f804ad02f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:43.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.836+0000 7f58d7200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f804ab9bf0 0x55f804ad02f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41204/0 (socket says 172.21.15.16:41204) 2024-03-31T15:10:43.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.836+0000 7f58d7200640 1 -- 172.21.15.16:0/1295602757 learned_addr learned my addr 172.21.15.16:0/1295602757 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:10:43.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.836+0000 7f58d7200640 1 -- 172.21.15.16:0/1295602757 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55f804ab92d0 msgr2=0x55f804acfaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:10:43.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.836+0000 7f58d7200640 1 --2- 172.21.15.16:0/1295602757 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55f804ab92d0 0x55f804acfaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:43.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.836+0000 7f58d7200640 1 -- 172.21.15.16:0/1295602757 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f804a6c9b0 msgr2=0x55f804acf5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:10:43.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.836+0000 7f58d7200640 1 --2- 172.21.15.16:0/1295602757 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f804a6c9b0 0x55f804acf5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:43.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.836+0000 7f58d7200640 1 -- 172.21.15.16:0/1295602757 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f80493f140 con 0x55f804ab9bf0 2024-03-31T15:10:43.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.836+0000 7f58d7200640 1 --2- 172.21.15.16:0/1295602757 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f804ab9bf0 0x55f804ad02f0 secure :-1 s=READY pgs=1207 cs=0 l=1 rev1=1 crypto rx=0x7f58d0017000 tx=0x7f58d0015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a743a575f7569a5 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:10:43.829 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.836+0000 7f58c6a00640 1 -- 172.21.15.16:0/1295602757 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58d002c070 con 0x55f804ab9bf0 2024-03-31T15:10:43.829 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.836+0000 7f58c6a00640 1 -- 172.21.15.16:0/1295602757 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f58d0010d90 con 0x55f804ab9bf0 2024-03-31T15:10:43.829 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.836+0000 7f58c6a00640 1 -- 172.21.15.16:0/1295602757 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58d0018040 con 0x55f804ab9bf0 2024-03-31T15:10:43.829 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.836+0000 7f58e0930000 1 -- 172.21.15.16:0/1295602757 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f804ab9bf0 msgr2=0x55f804ad02f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:10:43.829 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.836+0000 7f58e0930000 1 --2- 172.21.15.16:0/1295602757 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f804ab9bf0 0x55f804ad02f0 secure :-1 s=READY pgs=1207 cs=0 l=1 rev1=1 crypto rx=0x7f58d0017000 tx=0x7f58d0015000 comp rx=0 tx=0).stop 2024-03-31T15:10:43.829 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.836+0000 7f58e0930000 1 -- 172.21.15.16:0/1295602757 shutdown_connections 2024-03-31T15:10:43.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.836+0000 7f58e0930000 1 --2- 172.21.15.16:0/1295602757 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f804ab9bf0 0x55f804ad02f0 unknown :-1 s=CLOSED pgs=1207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:43.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.836+0000 7f58e0930000 1 --2- 172.21.15.16:0/1295602757 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55f804ab92d0 0x55f804acfaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:43.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.836+0000 7f58e0930000 1 --2- 172.21.15.16:0/1295602757 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f804a6c9b0 0x55f804acf5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:43.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.836+0000 7f58e0930000 1 -- 172.21.15.16:0/1295602757 >> 172.21.15.16:0/1295602757 conn(0x55f804a9a070 msgr2=0x55f804a6abd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:10:43.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.836+0000 7f58e0930000 1 -- 172.21.15.16:0/1295602757 shutdown_connections 2024-03-31T15:10:43.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.836+0000 7f58e0930000 1 -- 172.21.15.16:0/1295602757 wait complete. 2024-03-31T15:10:43.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.836+0000 7f58e0930000 1 -- start start 2024-03-31T15:10:43.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.836+0000 7f58e0930000 1 Processor -- start 2024-03-31T15:10:43.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.836+0000 7f58e0930000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f804a6c9b0 0x55f804abf860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:43.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.836+0000 7f58e0930000 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f804ab92d0 0x55f804abd030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:43.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.836+0000 7f58e0930000 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55f804ab9bf0 0x55f804abd570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:43.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.836+0000 7f58e0930000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55f804979250 con 0x55f804a6c9b0 2024-03-31T15:10:43.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.836+0000 7f58e0930000 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55f804859940 con 0x55f804ab92d0 2024-03-31T15:10:43.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.836+0000 7f58e0930000 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x55f80498d420 con 0x55f804ab9bf0 2024-03-31T15:10:43.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.836+0000 7f58e0930000 10 client.0 _ll_register_callbacks cb 0x7f58c0007f50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T15:10:43.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.836+0000 7f58d6800640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f804a6c9b0 0x55f804abf860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:43.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.836+0000 7f58d6800640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f804a6c9b0 0x55f804abf860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41218/0 (socket says 172.21.15.16:41218) 2024-03-31T15:10:43.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.836+0000 7f58d6800640 1 -- 172.21.15.16:0/800887652 learned_addr learned my addr 172.21.15.16:0/800887652 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:10:43.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.836+0000 7f58cde00640 1 --2- 172.21.15.16:0/800887652 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f804ab92d0 0x55f804abd030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:43.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.836+0000 7f58d7200640 1 --2- 172.21.15.16:0/800887652 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55f804ab9bf0 0x55f804abd570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:43.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.840+0000 7f58d6800640 1 -- 172.21.15.16:0/800887652 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55f804ab9bf0 msgr2=0x55f804abd570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:10:43.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.840+0000 7f58d6800640 1 --2- 172.21.15.16:0/800887652 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55f804ab9bf0 0x55f804abd570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:43.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.840+0000 7f58d6800640 1 -- 172.21.15.16:0/800887652 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f804ab92d0 msgr2=0x55f804abd030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:10:43.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.840+0000 7f58d6800640 1 --2- 172.21.15.16:0/800887652 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f804ab92d0 0x55f804abd030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:43.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.840+0000 7f58d6800640 1 -- 172.21.15.16:0/800887652 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f80484d1e0 con 0x55f804a6c9b0 2024-03-31T15:10:43.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.840+0000 7f58cde00640 1 --2- 172.21.15.16:0/800887652 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f804ab92d0 0x55f804abd030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:10:43.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.840+0000 7f58d6800640 1 --2- 172.21.15.16:0/800887652 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f804a6c9b0 0x55f804abf860 secure :-1 s=READY pgs=1208 cs=0 l=1 rev1=1 crypto rx=0x7f58c801c970 tx=0x7f58c8019230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f45504ab21faa730 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:10:43.835 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.840+0000 7f58bbe00640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-31T15:10:43.835 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.840+0000 7f58bbe00640 1 -- 172.21.15.16:0/800887652 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58c80b0040 con 0x55f804a6c9b0 2024-03-31T15:10:43.835 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.840+0000 7f58bbe00640 1 -- 172.21.15.16:0/800887652 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 5824+0+0 (secure 0 0 0) 0x7f58c80c80d0 con 0x55f804a6c9b0 2024-03-31T15:10:43.835 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.840+0000 7f58e0930000 1 -- 172.21.15.16:0/800887652 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f58d0010d90 con 0x55f804a6c9b0 2024-03-31T15:10:43.835 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.840+0000 7f58bbe00640 1 -- 172.21.15.16:0/800887652 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=68) ==== 24+0+0 (secure 0 0 0) 0x7f58c80d7050 con 0x55f804a6c9b0 2024-03-31T15:10:43.835 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.840+0000 7f58e0930000 10 client.7962 fetch_fsmap learned FSMap version 68 2024-03-31T15:10:43.835 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.840+0000 7f58e0930000 1 -- 172.21.15.16:0/800887652 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=68,monmap=2+}) -- 0x55f8048527b0 con 0x55f804a6c9b0 2024-03-31T15:10:43.835 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.840+0000 7f58bbe00640 1 -- 172.21.15.16:0/800887652 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f58d0010d90 con 0x55f804a6c9b0 2024-03-31T15:10:43.835 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.840+0000 7f58bbe00640 1 -- 172.21.15.16:0/800887652 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 68) ==== 3494+0+0 (secure 0 0 0) 0x7f58c80da020 con 0x55f804a6c9b0 2024-03-31T15:10:43.835 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.840+0000 7f58e0930000 10 client.7962 fetch_fsmap finished waiting for FSMap version 68 2024-03-31T15:10:43.835 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.840+0000 7f58e0930000 10 client.7962 resolve_mds: resolved 6:0 to role '6:0' aka daemon mds.c 2024-03-31T15:10:43.835 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.840+0000 7f58e0930000 20 client.7962 populate_metadata read hostname 'smithi016' 2024-03-31T15:10:43.835 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.840+0000 7f58e0930000 1 --2- 172.21.15.16:0/800887652 >> [v2:172.21.15.16:6832/818074038,v1:172.21.15.16:6834/818074038] conn(0x55f804abdab0 0x55f804b291d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:10:43.836 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.840+0000 7f58e0930000 4 client.7962 mds_command: new command op to 7749 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-31T15:10:43.836 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.840+0000 7f58e0930000 1 -- 172.21.15.16:0/800887652 --> [v2:172.21.15.16:6832/818074038,v1:172.21.15.16:6834/818074038] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55f8048cc040 con 0x55f804abdab0 2024-03-31T15:10:43.836 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.840+0000 7f58cde00640 1 --2- 172.21.15.16:0/800887652 >> [v2:172.21.15.16:6832/818074038,v1:172.21.15.16:6834/818074038] conn(0x55f804abdab0 0x55f804b291d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:10:43.836 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.840+0000 7f58bbe00640 10 client.7962 ms_handle_connect on v2:172.21.15.16:6832/818074038 2024-03-31T15:10:43.836 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.840+0000 7f58cde00640 1 --2- 172.21.15.16:0/800887652 >> [v2:172.21.15.16:6832/818074038,v1:172.21.15.16:6834/818074038] conn(0x55f804abdab0 0x55f804b291d0 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4cfaf522abfea4a server_cookie=aa15c2ccccfbb044 in_seq=0 out_seq=0 2024-03-31T15:10:43.836 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.840+0000 7f58bbe00640 1 -- 172.21.15.16:0/800887652 <== mds.0 v2:172.21.15.16:6832/818074038 1 ==== command_reply(tid 0: 0 ) ==== 8+0+263 (crc 0 0 0) 0x55f8048cc040 con 0x55f804abdab0 2024-03-31T15:10:43.836 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.840+0000 7f58bbe00640 10 client.7962 handle_command_reply: tid=0 2024-03-31T15:10:43.836 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.840+0000 7f58e0930000 1 client.7962 shutdown 2024-03-31T15:10:43.836 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.840+0000 7f58e0930000 1 -- 172.21.15.16:0/800887652 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f804a6c9b0 msgr2=0x55f804abf860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:10:43.836 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.840+0000 7f58e0930000 1 --2- 172.21.15.16:0/800887652 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f804a6c9b0 0x55f804abf860 secure :-1 s=READY pgs=1208 cs=0 l=1 rev1=1 crypto rx=0x7f58c801c970 tx=0x7f58c8019230 comp rx=0 tx=0).stop 2024-03-31T15:10:43.836 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.840+0000 7f58e0930000 1 -- 172.21.15.16:0/800887652 shutdown_connections 2024-03-31T15:10:43.836 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.840+0000 7f58e0930000 1 --2- 172.21.15.16:0/800887652 >> [v2:172.21.15.16:6832/818074038,v1:172.21.15.16:6834/818074038] conn(0x55f804abdab0 0x55f804b291d0 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:43.836 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.840+0000 7f58e0930000 1 --2- 172.21.15.16:0/800887652 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55f804ab9bf0 0x55f804abd570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:43.836 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.840+0000 7f58e0930000 1 --2- 172.21.15.16:0/800887652 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f804ab92d0 0x55f804abd030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:43.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.840+0000 7f58bbe00640 0 client.7962 ms_handle_reset on v2:172.21.15.16:6832/818074038 2024-03-31T15:10:43.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.840+0000 7f58e0930000 1 --2- 172.21.15.16:0/800887652 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f804a6c9b0 0x55f804abf860 unknown :-1 s=CLOSED pgs=1208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:43.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.840+0000 7f58e0930000 1 -- 172.21.15.16:0/800887652 >> 172.21.15.16:0/800887652 conn(0x55f804a9a070 msgr2=0x55f804a710a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:10:43.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.840+0000 7f58e0930000 1 -- 172.21.15.16:0/800887652 shutdown_connections 2024-03-31T15:10:43.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.840+0000 7f58e0930000 1 -- 172.21.15.16:0/800887652 wait complete. 2024-03-31T15:10:43.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.840+0000 7f58e0930000 20 client.7962 trim_cache size 0 max 16384 2024-03-31T15:10:43.837 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-03-31T15:10:43.837 INFO:teuthology.orchestra.run.smithi016.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-31T15:10:43.837 INFO:teuthology.orchestra.run.smithi016.stdout: "scrubs": { 2024-03-31T15:10:43.837 INFO:teuthology.orchestra.run.smithi016.stdout: "e29c520e-8750-442e-b654-00f09d3161bd": { 2024-03-31T15:10:43.838 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/dir", 2024-03-31T15:10:43.838 INFO:teuthology.orchestra.run.smithi016.stdout: "tag": "e29c520e-8750-442e-b654-00f09d3161bd", 2024-03-31T15:10:43.838 INFO:teuthology.orchestra.run.smithi016.stdout: "options": "recursive,repair" 2024-03-31T15:10:43.838 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-31T15:10:43.838 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-31T15:10:43.838 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-03-31T15:10:43.839 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.840+0000 7f58dd400640 1 -- 172.21.15.16:0/3254927488 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f58b40683b0 msgr2=0x7f58b406a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:10:43.839 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.840+0000 7f58dd400640 1 --2- 172.21.15.16:0/3254927488 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f58b40683b0 0x7f58b406a870 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f58c80ad000 tx=0x7f58c8008040 comp rx=0 tx=0).stop 2024-03-31T15:10:43.839 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.840+0000 7f58dd400640 1 -- 172.21.15.16:0/3254927488 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f58d80cf150 msgr2=0x7f58d813ecd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:10:43.839 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.840+0000 7f58dd400640 1 --2- 172.21.15.16:0/3254927488 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f58d80cf150 0x7f58d813ecd0 secure :-1 s=READY pgs=1204 cs=0 l=1 rev1=1 crypto rx=0x7f58c000c730 tx=0x7f58c000cbf0 comp rx=0 tx=0).stop 2024-03-31T15:10:43.839 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.840+0000 7f58dd400640 1 -- 172.21.15.16:0/3254927488 shutdown_connections 2024-03-31T15:10:43.839 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.840+0000 7f58dd400640 1 --2- 172.21.15.16:0/3254927488 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f58b40683b0 0x7f58b406a870 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:43.839 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.840+0000 7f58dd400640 1 --2- 172.21.15.16:0/3254927488 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f58d80d0a60 0x7f58d815d850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:43.839 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.840+0000 7f58dd400640 1 --2- 172.21.15.16:0/3254927488 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f58d80cf150 0x7f58d813ecd0 unknown :-1 s=CLOSED pgs=1204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:43.839 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.840+0000 7f58dd400640 1 --2- 172.21.15.16:0/3254927488 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f58d806f920 0x7f58d806e360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:10:43.839 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.840+0000 7f58dd400640 1 -- 172.21.15.16:0/3254927488 >> 172.21.15.16:0/3254927488 conn(0x7f58d8067930 msgr2=0x7f58d815a2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:10:43.839 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.840+0000 7f58dd400640 1 -- 172.21.15.16:0/3254927488 shutdown_connections 2024-03-31T15:10:43.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:10:43.840+0000 7f58dd400640 1 -- 172.21.15.16:0/3254927488 wait complete. 2024-03-31T15:10:43.861 INFO:tasks.cephfs.filesystem:scrub status for tag:e29c520e-8750-442e-b654-00f09d3161bd - {'path': '/dir', 'tag': 'e29c520e-8750-442e-b654-00f09d3161bd', 'options': 'recursive,repair'} 2024-03-31T15:11:07.454 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-31T15:11:07.458 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-31T15:11:13.862 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:11:13.862 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.6:0 scrub status 2024-03-31T15:11:14.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.012+0000 7f3407c00640 1 Processor -- start 2024-03-31T15:11:14.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.012+0000 7f3407c00640 1 -- start start 2024-03-31T15:11:14.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.016+0000 7f3407c00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f3400070030 0x7f3400143c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:14.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.016+0000 7f3407c00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f3400141590 0x7f3400144170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:14.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.016+0000 7f3407c00640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f34001446b0 con 0x7f34001411b0 2024-03-31T15:11:14.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.016+0000 7f3407c00640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f340015f910 con 0x7f3400070030 2024-03-31T15:11:14.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.016+0000 7f3407c00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f3400162910 con 0x7f3400141590 2024-03-31T15:11:14.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.016+0000 7f33f7e00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f3400070030 0x7f3400143c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:14.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.016+0000 7f3405c00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f3400141590 0x7f3400144170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:14.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.016+0000 7f3405c00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f3400141590 0x7f3400144170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3301/0 says I am v2:172.21.15.16:33496/0 (socket says 172.21.15.16:33496) 2024-03-31T15:11:14.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.016+0000 7f3405c00640 1 -- 172.21.15.16:0/2850160177 learned_addr learned my addr 172.21.15.16:0/2850160177 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:11:14.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.016+0000 7f3405c00640 1 -- 172.21.15.16:0/2850160177 >> v2:172.21.15.89:3300/0 conn(0x7f3400070030 msgr2=0x7f3400143c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:11:14.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.016+0000 7f3405c00640 1 --2- 172.21.15.16:0/2850160177 >> v2:172.21.15.89:3300/0 conn(0x7f3400070030 0x7f3400143c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:14.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.016+0000 7f3405c00640 1 -- 172.21.15.16:0/2850160177 >> v1:172.21.15.89:6789/0 conn(0x7f34001411b0 legacy=0x7f340006f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:11:14.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.016+0000 7f3405c00640 1 -- 172.21.15.16:0/2850160177 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3400144980 con 0x7f3400141590 2024-03-31T15:11:14.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.020+0000 7f3405c00640 1 --2- 172.21.15.16:0/2850160177 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f3400141590 0x7f3400144170 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f33f8009e50 tx=0x7f33f801ca00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=962e494218e058ab server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:11:14.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.020+0000 7f33ffe00640 1 -- 172.21.15.16:0/2850160177 <== mon.2 v2:172.21.15.89:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33f80a58d0 con 0x7f3400141590 2024-03-31T15:11:14.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.020+0000 7f33ffe00640 1 -- 172.21.15.16:0/2850160177 <== mon.2 v2:172.21.15.89:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f33f80a5a70 con 0x7f3400141590 2024-03-31T15:11:14.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.020+0000 7f33ffe00640 1 -- 172.21.15.16:0/2850160177 <== mon.2 v2:172.21.15.89:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33f80a5e40 con 0x7f3400141590 2024-03-31T15:11:14.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.020+0000 7f3407c00640 1 -- 172.21.15.16:0/2850160177 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f3400141590 msgr2=0x7f3400144170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:11:14.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.020+0000 7f3407c00640 1 --2- 172.21.15.16:0/2850160177 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f3400141590 0x7f3400144170 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f33f8009e50 tx=0x7f33f801ca00 comp rx=0 tx=0).stop 2024-03-31T15:11:14.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.020+0000 7f3407c00640 1 -- 172.21.15.16:0/2850160177 shutdown_connections 2024-03-31T15:11:14.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.020+0000 7f3407c00640 1 --2- 172.21.15.16:0/2850160177 >> v2:172.21.15.89:3300/0 conn(0x7f3400070030 0x7f3400143c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:14.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.020+0000 7f3407c00640 1 --2- 172.21.15.16:0/2850160177 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f3400141590 0x7f3400144170 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:14.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.020+0000 7f3407c00640 1 -- 172.21.15.16:0/2850160177 >> 172.21.15.16:0/2850160177 conn(0x7f3400067930 msgr2=0x7f34000b3ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:11:14.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.020+0000 7f3407c00640 1 -- 172.21.15.16:0/2850160177 shutdown_connections 2024-03-31T15:11:14.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.020+0000 7f3407c00640 1 -- 172.21.15.16:0/2850160177 wait complete. 2024-03-31T15:11:14.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.020+0000 7f3407c00640 1 Processor -- start 2024-03-31T15:11:14.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.020+0000 7f3407c00640 1 -- start start 2024-03-31T15:11:14.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.024+0000 7f3407c00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f3400070030 0x7f34001686e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:14.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.024+0000 7f3407c00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f34001411b0 0x7f340013fa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:14.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.024+0000 7f3407c00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3400141590 0x7f34001f8640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:14.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.024+0000 7f3407c00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f340015f110 con 0x7f3400141590 2024-03-31T15:11:14.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.024+0000 7f3407c00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f3400162d80 con 0x7f34001411b0 2024-03-31T15:11:14.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.024+0000 7f3407c00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f340015f290 con 0x7f3400070030 2024-03-31T15:11:14.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.024+0000 7f3405c00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3400141590 0x7f34001f8640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:14.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.024+0000 7f3405200640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f3400070030 0x7f34001686e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:14.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.024+0000 7f3405200640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f3400070030 0x7f34001686e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3301/0 says I am v2:172.21.15.16:33504/0 (socket says 172.21.15.16:33504) 2024-03-31T15:11:14.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.024+0000 7f3405200640 1 -- 172.21.15.16:0/3631310433 learned_addr learned my addr 172.21.15.16:0/3631310433 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:11:14.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.024+0000 7f33f7e00640 1 --2- 172.21.15.16:0/3631310433 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f34001411b0 0x7f340013fa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:14.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.024+0000 7f3405200640 1 -- 172.21.15.16:0/3631310433 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f34001411b0 msgr2=0x7f340013fa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:11:14.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.024+0000 7f3405200640 1 --2- 172.21.15.16:0/3631310433 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f34001411b0 0x7f340013fa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:14.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.024+0000 7f3405200640 1 -- 172.21.15.16:0/3631310433 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3400141590 msgr2=0x7f34001f8640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:11:14.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.024+0000 7f3405200640 1 --2- 172.21.15.16:0/3631310433 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3400141590 0x7f34001f8640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:14.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.024+0000 7f3405200640 1 -- 172.21.15.16:0/3631310433 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f340013efc0 con 0x7f3400070030 2024-03-31T15:11:14.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.024+0000 7f3405200640 1 --2- 172.21.15.16:0/3631310433 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f3400070030 0x7f34001686e0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f33f000c450 tx=0x7f33f000c910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:11:14.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.024+0000 7f33fd600640 1 -- 172.21.15.16:0/3631310433 <== mon.2 v2:172.21.15.89:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33f0016020 con 0x7f3400070030 2024-03-31T15:11:14.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.024+0000 7f33fd600640 1 -- 172.21.15.16:0/3631310433 <== mon.2 v2:172.21.15.89:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f33f00052d0 con 0x7f3400070030 2024-03-31T15:11:14.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.024+0000 7f3407c00640 1 -- 172.21.15.16:0/3631310433 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f34001f7fe0 con 0x7f3400070030 2024-03-31T15:11:14.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.024+0000 7f33fd600640 1 -- 172.21.15.16:0/3631310433 <== mon.2 v2:172.21.15.89:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33f0007260 con 0x7f3400070030 2024-03-31T15:11:14.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.024+0000 7f3407c00640 1 -- 172.21.15.16:0/3631310433 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f34001f82c0 con 0x7f3400070030 2024-03-31T15:11:14.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.028+0000 7f33fd600640 1 -- 172.21.15.16:0/3631310433 <== mon.2 v2:172.21.15.89:3301/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f33f0007400 con 0x7f3400070030 2024-03-31T15:11:14.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.028+0000 7f33fd600640 1 --2- 172.21.15.16:0/3631310433 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f33dc068380 0x7f33dc06a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:14.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.028+0000 7f33fd600640 1 -- 172.21.15.16:0/3631310433 <== mon.2 v2:172.21.15.89:3301/0 5 ==== osd_map(84..84 src has 1..84) ==== 5824+0+0 (secure 0 0 0) 0x7f33f00862b0 con 0x7f3400070030 2024-03-31T15:11:14.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.028+0000 7f340afc7000 1 Processor -- start 2024-03-31T15:11:14.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.028+0000 7f340afc7000 1 -- start start 2024-03-31T15:11:14.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.028+0000 7f33f7e00640 1 --2- 172.21.15.16:0/3631310433 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f33dc068380 0x7f33dc06a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:14.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.028+0000 7f33f7e00640 1 --2- 172.21.15.16:0/3631310433 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f33dc068380 0x7f33dc06a840 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f33e8009b60 tx=0x7f33e8009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:11:14.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.028+0000 7f340afc7000 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5592d709f2d0 0x5592d709f6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:14.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.028+0000 7f340afc7000 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5592d709fbf0 0x5592d7052470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:14.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.028+0000 7f340afc7000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5592d70529b0 0x5592d7052d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:14.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.028+0000 7f340afc7000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5592d6f3baf0 con 0x5592d70529b0 2024-03-31T15:11:14.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.028+0000 7f340afc7000 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x5592d6ea4b00 con 0x5592d709f2d0 2024-03-31T15:11:14.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.028+0000 7f340afc7000 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x5592d6e41940 con 0x5592d709fbf0 2024-03-31T15:11:14.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.028+0000 7f3405c00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5592d70529b0 0x5592d7052d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:14.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.028+0000 7f3405200640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5592d709f2d0 0x5592d709f6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:14.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.028+0000 7f3405200640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5592d709f2d0 0x5592d709f6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.16:37786/0 (socket says 172.21.15.16:37786) 2024-03-31T15:11:14.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.028+0000 7f3405200640 1 -- 172.21.15.16:0/2558852925 learned_addr learned my addr 172.21.15.16:0/2558852925 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:11:14.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.028+0000 7f33f7e00640 1 --2- 172.21.15.16:0/2558852925 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5592d709fbf0 0x5592d7052470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:14.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.028+0000 7f3405200640 1 -- 172.21.15.16:0/2558852925 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5592d709fbf0 msgr2=0x5592d7052470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:11:14.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.028+0000 7f3405200640 1 --2- 172.21.15.16:0/2558852925 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5592d709fbf0 0x5592d7052470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:14.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.028+0000 7f3405200640 1 -- 172.21.15.16:0/2558852925 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5592d70529b0 msgr2=0x5592d7052d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:11:14.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.028+0000 7f3405200640 1 --2- 172.21.15.16:0/2558852925 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5592d70529b0 0x5592d7052d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:14.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.028+0000 7f3405200640 1 -- 172.21.15.16:0/2558852925 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5592d6e2a380 con 0x5592d709f2d0 2024-03-31T15:11:14.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.028+0000 7f3405200640 1 --2- 172.21.15.16:0/2558852925 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5592d709f2d0 0x5592d709f6b0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f33f00846c0 tx=0x7f33f0011000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=139e5a0db3db87d3 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:11:14.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.028+0000 7f33f5600640 1 -- 172.21.15.16:0/2558852925 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33f005ecf0 con 0x5592d709f2d0 2024-03-31T15:11:14.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.028+0000 7f33f5600640 1 -- 172.21.15.16:0/2558852925 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f33f00544c0 con 0x5592d709f2d0 2024-03-31T15:11:14.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.028+0000 7f33f5600640 1 -- 172.21.15.16:0/2558852925 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33f0052830 con 0x5592d709f2d0 2024-03-31T15:11:14.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.032+0000 7f340afc7000 1 -- 172.21.15.16:0/2558852925 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5592d709f2d0 msgr2=0x5592d709f6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:11:14.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.032+0000 7f340afc7000 1 --2- 172.21.15.16:0/2558852925 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5592d709f2d0 0x5592d709f6b0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f33f00846c0 tx=0x7f33f0011000 comp rx=0 tx=0).stop 2024-03-31T15:11:14.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.032+0000 7f340afc7000 1 -- 172.21.15.16:0/2558852925 shutdown_connections 2024-03-31T15:11:14.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.032+0000 7f340afc7000 1 --2- 172.21.15.16:0/2558852925 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5592d70529b0 0x5592d7052d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:14.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.032+0000 7f340afc7000 1 --2- 172.21.15.16:0/2558852925 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5592d709fbf0 0x5592d7052470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:14.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.032+0000 7f340afc7000 1 --2- 172.21.15.16:0/2558852925 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5592d709f2d0 0x5592d709f6b0 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:14.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.032+0000 7f340afc7000 1 -- 172.21.15.16:0/2558852925 >> 172.21.15.16:0/2558852925 conn(0x5592d7080070 msgr2=0x5592d7082460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:11:14.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.032+0000 7f340afc7000 1 -- 172.21.15.16:0/2558852925 shutdown_connections 2024-03-31T15:11:14.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.032+0000 7f340afc7000 1 -- 172.21.15.16:0/2558852925 wait complete. 2024-03-31T15:11:14.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.032+0000 7f340afc7000 1 -- start start 2024-03-31T15:11:14.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.032+0000 7f340afc7000 1 Processor -- start 2024-03-31T15:11:14.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.032+0000 7f340afc7000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5592d70529b0 0x5592d705a9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:14.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.032+0000 7f340afc7000 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5592d709f2d0 0x5592d70a01e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:14.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.032+0000 7f340afc7000 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5592d709fbf0 0x5592d70a0720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:14.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.032+0000 7f340afc7000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5592d6e621e0 con 0x5592d70529b0 2024-03-31T15:11:14.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.032+0000 7f340afc7000 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x5592d6ea0ab0 con 0x5592d709f2d0 2024-03-31T15:11:14.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.032+0000 7f340afc7000 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x5592d705e000 con 0x5592d709fbf0 2024-03-31T15:11:14.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.032+0000 7f340afc7000 10 client.0 _ll_register_callbacks cb 0x5592d7040370 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T15:11:14.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.032+0000 7f3405200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5592d70529b0 0x5592d705a9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:14.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.032+0000 7f3405200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5592d70529b0 0x5592d705a9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37872/0 (socket says 172.21.15.16:37872) 2024-03-31T15:11:14.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.032+0000 7f3405200640 1 -- 172.21.15.16:0/2862848141 learned_addr learned my addr 172.21.15.16:0/2862848141 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:11:14.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.032+0000 7f33f7e00640 1 --2- 172.21.15.16:0/2862848141 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5592d709f2d0 0x5592d70a01e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:14.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.032+0000 7f3405c00640 1 --2- 172.21.15.16:0/2862848141 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5592d709fbf0 0x5592d70a0720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:14.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.032+0000 7f3405200640 1 -- 172.21.15.16:0/2862848141 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5592d709fbf0 msgr2=0x5592d70a0720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:11:14.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.032+0000 7f3405200640 1 --2- 172.21.15.16:0/2862848141 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5592d709fbf0 0x5592d70a0720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:14.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.032+0000 7f3405200640 1 -- 172.21.15.16:0/2862848141 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5592d709f2d0 msgr2=0x5592d70a01e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:11:14.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.032+0000 7f3405200640 1 --2- 172.21.15.16:0/2862848141 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5592d709f2d0 0x5592d70a01e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:14.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.032+0000 7f3405200640 1 -- 172.21.15.16:0/2862848141 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5592d6e60b70 con 0x5592d70529b0 2024-03-31T15:11:14.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.032+0000 7f33f7e00640 1 --2- 172.21.15.16:0/2862848141 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5592d709f2d0 0x5592d70a01e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:11:14.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.032+0000 7f3405200640 1 --2- 172.21.15.16:0/2862848141 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5592d70529b0 0x5592d705a9d0 secure :-1 s=READY pgs=1209 cs=0 l=1 rev1=1 crypto rx=0x7f33f000a040 tx=0x7f33f0084650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=230964faef640725 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:11:14.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.032+0000 7f33ee000640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-31T15:11:14.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.032+0000 7f33ee000640 1 -- 172.21.15.16:0/2862848141 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33f0073060 con 0x5592d70529b0 2024-03-31T15:11:14.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.032+0000 7f33ee000640 1 -- 172.21.15.16:0/2862848141 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 5824+0+0 (secure 0 0 0) 0x7f33f005a150 con 0x5592d70529b0 2024-03-31T15:11:14.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.032+0000 7f340afc7000 1 -- 172.21.15.16:0/2862848141 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f33f00544c0 con 0x5592d70529b0 2024-03-31T15:11:14.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.032+0000 7f33ee000640 1 -- 172.21.15.16:0/2862848141 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=68) ==== 24+0+0 (secure 0 0 0) 0x7f33f0079050 con 0x5592d70529b0 2024-03-31T15:11:14.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.032+0000 7f340afc7000 10 client.7971 fetch_fsmap learned FSMap version 68 2024-03-31T15:11:14.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.032+0000 7f340afc7000 1 -- 172.21.15.16:0/2862848141 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=68,monmap=2+}) -- 0x5592d6e41b90 con 0x5592d70529b0 2024-03-31T15:11:14.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.032+0000 7f33ee000640 1 -- 172.21.15.16:0/2862848141 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f33f00544c0 con 0x5592d70529b0 2024-03-31T15:11:14.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.032+0000 7f33ee000640 1 -- 172.21.15.16:0/2862848141 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 68) ==== 3494+0+0 (secure 0 0 0) 0x7f33f0052830 con 0x5592d70529b0 2024-03-31T15:11:14.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.032+0000 7f340afc7000 10 client.7971 fetch_fsmap finished waiting for FSMap version 68 2024-03-31T15:11:14.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.032+0000 7f340afc7000 10 client.7971 resolve_mds: resolved 6:0 to role '6:0' aka daemon mds.c 2024-03-31T15:11:14.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.032+0000 7f340afc7000 20 client.7971 populate_metadata read hostname 'smithi016' 2024-03-31T15:11:14.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.032+0000 7f340afc7000 1 --2- 172.21.15.16:0/2862848141 >> [v2:172.21.15.16:6832/818074038,v1:172.21.15.16:6834/818074038] conn(0x5592d70a14c0 0x5592d70b0eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:14.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.036+0000 7f340afc7000 4 client.7971 mds_command: new command op to 7749 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-31T15:11:14.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.036+0000 7f340afc7000 1 -- 172.21.15.16:0/2862848141 --> [v2:172.21.15.16:6832/818074038,v1:172.21.15.16:6834/818074038] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5592d6e3fdd0 con 0x5592d70a14c0 2024-03-31T15:11:14.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.036+0000 7f33f7e00640 1 --2- 172.21.15.16:0/2862848141 >> [v2:172.21.15.16:6832/818074038,v1:172.21.15.16:6834/818074038] conn(0x5592d70a14c0 0x5592d70b0eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:14.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.036+0000 7f33f7e00640 1 --2- 172.21.15.16:0/2862848141 >> [v2:172.21.15.16:6832/818074038,v1:172.21.15.16:6834/818074038] conn(0x5592d70a14c0 0x5592d70b0eb0 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a288257326876026 server_cookie=3ee61ba26ea8e84a in_seq=0 out_seq=0 2024-03-31T15:11:14.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.036+0000 7f33ee000640 10 client.7971 ms_handle_connect on v2:172.21.15.16:6832/818074038 2024-03-31T15:11:14.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.036+0000 7f33ee000640 1 -- 172.21.15.16:0/2862848141 <== mds.0 v2:172.21.15.16:6832/818074038 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5592d6e3fdd0 con 0x5592d70a14c0 2024-03-31T15:11:14.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.036+0000 7f33ee000640 10 client.7971 handle_command_reply: tid=0 2024-03-31T15:11:14.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.036+0000 7f340afc7000 1 client.7971 shutdown 2024-03-31T15:11:14.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.036+0000 7f340afc7000 1 -- 172.21.15.16:0/2862848141 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5592d70529b0 msgr2=0x5592d705a9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:11:14.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.036+0000 7f340afc7000 1 --2- 172.21.15.16:0/2862848141 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5592d70529b0 0x5592d705a9d0 secure :-1 s=READY pgs=1209 cs=0 l=1 rev1=1 crypto rx=0x7f33f000a040 tx=0x7f33f0084650 comp rx=0 tx=0).stop 2024-03-31T15:11:14.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.036+0000 7f340afc7000 1 -- 172.21.15.16:0/2862848141 shutdown_connections 2024-03-31T15:11:14.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.036+0000 7f340afc7000 1 --2- 172.21.15.16:0/2862848141 >> [v2:172.21.15.16:6832/818074038,v1:172.21.15.16:6834/818074038] conn(0x5592d70a14c0 0x5592d70b0eb0 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:14.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.036+0000 7f340afc7000 1 --2- 172.21.15.16:0/2862848141 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5592d709fbf0 0x5592d70a0720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:14.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.036+0000 7f340afc7000 1 --2- 172.21.15.16:0/2862848141 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5592d709f2d0 0x5592d70a01e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:14.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.036+0000 7f33ee000640 0 client.7971 ms_handle_reset on v2:172.21.15.16:6832/818074038 2024-03-31T15:11:14.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.036+0000 7f340afc7000 1 --2- 172.21.15.16:0/2862848141 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5592d70529b0 0x5592d705a9d0 unknown :-1 s=CLOSED pgs=1209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:14.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.036+0000 7f340afc7000 1 -- 172.21.15.16:0/2862848141 >> 172.21.15.16:0/2862848141 conn(0x5592d7080070 msgr2=0x5592d7082460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:11:14.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.036+0000 7f340afc7000 1 -- 172.21.15.16:0/2862848141 shutdown_connections 2024-03-31T15:11:14.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.036+0000 7f340afc7000 1 -- 172.21.15.16:0/2862848141 wait complete. 2024-03-31T15:11:14.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.036+0000 7f340afc7000 20 client.7971 trim_cache size 0 max 16384 2024-03-31T15:11:14.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.052+0000 7f340afc7000 1 Processor -- start 2024-03-31T15:11:14.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.052+0000 7f340afc7000 1 -- start start 2024-03-31T15:11:14.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.056+0000 7f340afc7000 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5592d70529b0 0x5592d70b6510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:14.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.056+0000 7f340afc7000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5592d709f2d0 0x5592d709e390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:14.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.056+0000 7f340afc7000 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5592d709fbf0 0x5592d709e8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:14.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.056+0000 7f340afc7000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f3400159180 con 0x5592d709f2d0 2024-03-31T15:11:14.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.056+0000 7f340afc7000 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f34001597d0 con 0x5592d709fbf0 2024-03-31T15:11:14.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.056+0000 7f340afc7000 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x5592d6f5f250 con 0x5592d70529b0 2024-03-31T15:11:14.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.056+0000 7f33f7e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5592d709f2d0 0x5592d709e390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:14.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.056+0000 7f33f7e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5592d709f2d0 0x5592d709e390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37876/0 (socket says 172.21.15.16:37876) 2024-03-31T15:11:14.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.056+0000 7f33f7e00640 1 -- 172.21.15.16:0/4194756316 learned_addr learned my addr 172.21.15.16:0/4194756316 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:11:14.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.056+0000 7f3405c00640 1 --2- 172.21.15.16:0/4194756316 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5592d709fbf0 0x5592d709e8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:14.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.056+0000 7f3405200640 1 --2- 172.21.15.16:0/4194756316 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5592d70529b0 0x5592d70b6510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:14.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.056+0000 7f33f7e00640 1 -- 172.21.15.16:0/4194756316 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5592d70529b0 msgr2=0x5592d70b6510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:11:14.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.056+0000 7f33f7e00640 1 --2- 172.21.15.16:0/4194756316 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5592d70529b0 0x5592d70b6510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:14.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.056+0000 7f33f7e00640 1 -- 172.21.15.16:0/4194756316 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5592d709fbf0 msgr2=0x5592d709e8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:11:14.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.056+0000 7f33f7e00640 1 --2- 172.21.15.16:0/4194756316 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5592d709fbf0 0x5592d709e8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:14.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.056+0000 7f33f7e00640 1 -- 172.21.15.16:0/4194756316 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5592d709fa20 con 0x5592d709f2d0 2024-03-31T15:11:14.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.056+0000 7f3405c00640 1 --2- 172.21.15.16:0/4194756316 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5592d709fbf0 0x5592d709e8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:11:14.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.056+0000 7f33f7e00640 1 --2- 172.21.15.16:0/4194756316 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5592d709f2d0 0x5592d709e390 secure :-1 s=READY pgs=1210 cs=0 l=1 rev1=1 crypto rx=0x5592d6e622f0 tx=0x7f33e8018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2caff78576878e3a server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:11:14.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.056+0000 7f33f4c00640 1 -- 172.21.15.16:0/4194756316 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33e802be10 con 0x5592d709f2d0 2024-03-31T15:11:14.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.056+0000 7f33f4c00640 1 -- 172.21.15.16:0/4194756316 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f33e802b340 con 0x5592d709f2d0 2024-03-31T15:11:14.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.056+0000 7f33f4c00640 1 -- 172.21.15.16:0/4194756316 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33e802b6f0 con 0x5592d709f2d0 2024-03-31T15:11:14.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.056+0000 7f340afc7000 1 -- 172.21.15.16:0/4194756316 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5592d709f2d0 msgr2=0x5592d709e390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:11:14.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.056+0000 7f340afc7000 1 --2- 172.21.15.16:0/4194756316 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5592d709f2d0 0x5592d709e390 secure :-1 s=READY pgs=1210 cs=0 l=1 rev1=1 crypto rx=0x5592d6e622f0 tx=0x7f33e8018000 comp rx=0 tx=0).stop 2024-03-31T15:11:14.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.056+0000 7f340afc7000 1 -- 172.21.15.16:0/4194756316 shutdown_connections 2024-03-31T15:11:14.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.056+0000 7f340afc7000 1 --2- 172.21.15.16:0/4194756316 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5592d709fbf0 0x5592d709e8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:14.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.056+0000 7f340afc7000 1 --2- 172.21.15.16:0/4194756316 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5592d709f2d0 0x5592d709e390 unknown :-1 s=CLOSED pgs=1210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:14.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.056+0000 7f340afc7000 1 --2- 172.21.15.16:0/4194756316 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5592d70529b0 0x5592d70b6510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:14.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.056+0000 7f340afc7000 1 -- 172.21.15.16:0/4194756316 >> 172.21.15.16:0/4194756316 conn(0x5592d7080070 msgr2=0x5592d7051ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:11:14.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.056+0000 7f340afc7000 1 -- 172.21.15.16:0/4194756316 shutdown_connections 2024-03-31T15:11:14.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.056+0000 7f340afc7000 1 -- 172.21.15.16:0/4194756316 wait complete. 2024-03-31T15:11:14.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.056+0000 7f340afc7000 1 -- start start 2024-03-31T15:11:14.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.056+0000 7f340afc7000 1 Processor -- start 2024-03-31T15:11:14.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.060+0000 7f340afc7000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5592d70529b0 0x5592d70b6310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:14.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.060+0000 7f340afc7000 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5592d709f2d0 0x5592d70f2eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:14.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.060+0000 7f340afc7000 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5592d709fbf0 0x5592d70f33f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:14.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.060+0000 7f340afc7000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5592d6e70150 con 0x5592d70529b0 2024-03-31T15:11:14.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.060+0000 7f340afc7000 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x5592d6f73420 con 0x5592d709fbf0 2024-03-31T15:11:14.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.060+0000 7f340afc7000 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x5592d6e3f940 con 0x5592d709f2d0 2024-03-31T15:11:14.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.060+0000 7f340afc7000 10 client.0 _ll_register_callbacks cb 0x7f33f8009d50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T15:11:14.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.060+0000 7f3405200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5592d70529b0 0x5592d70b6310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:14.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.060+0000 7f3405200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5592d70529b0 0x5592d70b6310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37888/0 (socket says 172.21.15.16:37888) 2024-03-31T15:11:14.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.060+0000 7f3405200640 1 -- 172.21.15.16:0/2064804209 learned_addr learned my addr 172.21.15.16:0/2064804209 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:11:14.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.060+0000 7f3405c00640 1 --2- 172.21.15.16:0/2064804209 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5592d709fbf0 0x5592d70f33f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:14.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.060+0000 7f33f7e00640 1 --2- 172.21.15.16:0/2064804209 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5592d709f2d0 0x5592d70f2eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:14.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.060+0000 7f3405200640 1 -- 172.21.15.16:0/2064804209 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5592d709f2d0 msgr2=0x5592d70f2eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:11:14.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.060+0000 7f3405200640 1 --2- 172.21.15.16:0/2064804209 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5592d709f2d0 0x5592d70f2eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:14.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.060+0000 7f3405200640 1 -- 172.21.15.16:0/2064804209 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5592d709fbf0 msgr2=0x5592d70f33f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:11:14.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.060+0000 7f3405200640 1 --2- 172.21.15.16:0/2064804209 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5592d709fbf0 0x5592d70f33f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:14.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.060+0000 7f3405200640 1 -- 172.21.15.16:0/2064804209 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5592d6f25140 con 0x5592d70529b0 2024-03-31T15:11:14.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.060+0000 7f3405200640 1 --2- 172.21.15.16:0/2064804209 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5592d70529b0 0x5592d70b6310 secure :-1 s=READY pgs=1211 cs=0 l=1 rev1=1 crypto rx=0x7f33f000ce10 tx=0x7f33f0000ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81a18a68a287cc38 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:11:14.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.060+0000 7f33ee000640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-31T15:11:14.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.060+0000 7f33ee000640 1 -- 172.21.15.16:0/2064804209 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33f0063060 con 0x5592d70529b0 2024-03-31T15:11:14.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.060+0000 7f33ee000640 1 -- 172.21.15.16:0/2064804209 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 5824+0+0 (secure 0 0 0) 0x7f33f0052830 con 0x5592d70529b0 2024-03-31T15:11:14.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.060+0000 7f340afc7000 1 -- 172.21.15.16:0/2064804209 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f33e802b340 con 0x5592d70529b0 2024-03-31T15:11:14.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.060+0000 7f33ee000640 1 -- 172.21.15.16:0/2064804209 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=68) ==== 24+0+0 (secure 0 0 0) 0x7f33f0069050 con 0x5592d70529b0 2024-03-31T15:11:14.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.060+0000 7f340afc7000 10 client.7977 fetch_fsmap learned FSMap version 68 2024-03-31T15:11:14.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.060+0000 7f340afc7000 1 -- 172.21.15.16:0/2064804209 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=68,monmap=2+}) -- 0x5592d6e331e0 con 0x5592d70529b0 2024-03-31T15:11:14.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.060+0000 7f33ee000640 1 -- 172.21.15.16:0/2064804209 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f33e802b340 con 0x5592d70529b0 2024-03-31T15:11:14.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.060+0000 7f33ee000640 1 -- 172.21.15.16:0/2064804209 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 68) ==== 3494+0+0 (secure 0 0 0) 0x7f33f005a070 con 0x5592d70529b0 2024-03-31T15:11:14.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.060+0000 7f340afc7000 10 client.7977 fetch_fsmap finished waiting for FSMap version 68 2024-03-31T15:11:14.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.060+0000 7f340afc7000 10 client.7977 resolve_mds: resolved 6:0 to role '6:0' aka daemon mds.c 2024-03-31T15:11:14.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.060+0000 7f340afc7000 20 client.7977 populate_metadata read hostname 'smithi016' 2024-03-31T15:11:14.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.060+0000 7f340afc7000 1 --2- 172.21.15.16:0/2064804209 >> [v2:172.21.15.16:6832/818074038,v1:172.21.15.16:6834/818074038] conn(0x5592d70f3930 0x5592d710ed70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:14.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.060+0000 7f340afc7000 4 client.7977 mds_command: new command op to 7749 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-31T15:11:14.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.060+0000 7f340afc7000 1 -- 172.21.15.16:0/2064804209 --> [v2:172.21.15.16:6832/818074038,v1:172.21.15.16:6834/818074038] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5592d6eb2040 con 0x5592d70f3930 2024-03-31T15:11:14.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.060+0000 7f33f7e00640 1 --2- 172.21.15.16:0/2064804209 >> [v2:172.21.15.16:6832/818074038,v1:172.21.15.16:6834/818074038] conn(0x5592d70f3930 0x5592d710ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:14.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.064+0000 7f33f7e00640 1 --2- 172.21.15.16:0/2064804209 >> [v2:172.21.15.16:6832/818074038,v1:172.21.15.16:6834/818074038] conn(0x5592d70f3930 0x5592d710ed70 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cf31ccd3e039164d server_cookie=486a1677382ab446 in_seq=0 out_seq=0 2024-03-31T15:11:14.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.064+0000 7f33ee000640 10 client.7977 ms_handle_connect on v2:172.21.15.16:6832/818074038 2024-03-31T15:11:14.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.064+0000 7f33ee000640 1 -- 172.21.15.16:0/2064804209 <== mds.0 v2:172.21.15.16:6832/818074038 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x5592d6eb2040 con 0x5592d70f3930 2024-03-31T15:11:14.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.064+0000 7f33ee000640 10 client.7977 handle_command_reply: tid=0 2024-03-31T15:11:14.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.064+0000 7f340afc7000 1 client.7977 shutdown 2024-03-31T15:11:14.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.064+0000 7f340afc7000 1 -- 172.21.15.16:0/2064804209 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5592d70529b0 msgr2=0x5592d70b6310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:11:14.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.064+0000 7f340afc7000 1 --2- 172.21.15.16:0/2064804209 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5592d70529b0 0x5592d70b6310 secure :-1 s=READY pgs=1211 cs=0 l=1 rev1=1 crypto rx=0x7f33f000ce10 tx=0x7f33f0000ee0 comp rx=0 tx=0).stop 2024-03-31T15:11:14.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.064+0000 7f340afc7000 1 -- 172.21.15.16:0/2064804209 shutdown_connections 2024-03-31T15:11:14.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.064+0000 7f340afc7000 1 --2- 172.21.15.16:0/2064804209 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5592d709fbf0 0x5592d70f33f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:14.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.064+0000 7f340afc7000 1 --2- 172.21.15.16:0/2064804209 >> [v2:172.21.15.16:6832/818074038,v1:172.21.15.16:6834/818074038] conn(0x5592d70f3930 0x5592d710ed70 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:14.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.064+0000 7f340afc7000 1 --2- 172.21.15.16:0/2064804209 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5592d709f2d0 0x5592d70f2eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:14.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.064+0000 7f340afc7000 1 --2- 172.21.15.16:0/2064804209 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5592d70529b0 0x5592d70b6310 unknown :-1 s=CLOSED pgs=1211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:14.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.064+0000 7f33ee000640 0 client.7977 ms_handle_reset on v2:172.21.15.16:6832/818074038 2024-03-31T15:11:14.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.064+0000 7f340afc7000 1 -- 172.21.15.16:0/2064804209 >> 172.21.15.16:0/2064804209 conn(0x5592d7080070 msgr2=0x5592d709d380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:11:14.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.064+0000 7f340afc7000 1 -- 172.21.15.16:0/2064804209 shutdown_connections 2024-03-31T15:11:14.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.064+0000 7f340afc7000 1 -- 172.21.15.16:0/2064804209 wait complete. 2024-03-31T15:11:14.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.064+0000 7f340afc7000 20 client.7977 trim_cache size 0 max 16384 2024-03-31T15:11:14.060 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-03-31T15:11:14.060 INFO:teuthology.orchestra.run.smithi016.stdout: "status": "no active scrubs running", 2024-03-31T15:11:14.060 INFO:teuthology.orchestra.run.smithi016.stdout: "scrubs": {} 2024-03-31T15:11:14.060 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-03-31T15:11:14.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.064+0000 7f3407c00640 1 -- 172.21.15.16:0/3631310433 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f33dc068380 msgr2=0x7f33dc06a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:11:14.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.064+0000 7f3407c00640 1 --2- 172.21.15.16:0/3631310433 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f33dc068380 0x7f33dc06a840 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f33e8009b60 tx=0x7f33e8009190 comp rx=0 tx=0).stop 2024-03-31T15:11:14.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.064+0000 7f3407c00640 1 -- 172.21.15.16:0/3631310433 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f3400070030 msgr2=0x7f34001686e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:11:14.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.064+0000 7f3407c00640 1 --2- 172.21.15.16:0/3631310433 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f3400070030 0x7f34001686e0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f33f000c450 tx=0x7f33f000c910 comp rx=0 tx=0).stop 2024-03-31T15:11:14.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.064+0000 7f3407c00640 1 -- 172.21.15.16:0/3631310433 shutdown_connections 2024-03-31T15:11:14.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.064+0000 7f3407c00640 1 --2- 172.21.15.16:0/3631310433 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f33dc068380 0x7f33dc06a840 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:14.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.064+0000 7f3407c00640 1 --2- 172.21.15.16:0/3631310433 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3400141590 0x7f34001f8640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:14.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.064+0000 7f3407c00640 1 --2- 172.21.15.16:0/3631310433 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f34001411b0 0x7f340013fa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:14.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.064+0000 7f3407c00640 1 --2- 172.21.15.16:0/3631310433 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f3400070030 0x7f34001686e0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:14.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.064+0000 7f3407c00640 1 -- 172.21.15.16:0/3631310433 >> 172.21.15.16:0/3631310433 conn(0x7f3400067930 msgr2=0x7f3400160330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:11:14.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.064+0000 7f3407c00640 1 -- 172.21.15.16:0/3631310433 shutdown_connections 2024-03-31T15:11:14.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.064+0000 7f3407c00640 1 -- 172.21.15.16:0/3631310433 wait complete. 2024-03-31T15:11:14.075 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-31T15:11:14.076 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-31T15:11:14.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.224+0000 7ff037c00640 1 Processor -- start 2024-03-31T15:11:14.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.224+0000 7ff037c00640 1 -- start start 2024-03-31T15:11:14.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.224+0000 7ff037c00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff03006dbc0 0x7ff03015f650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:14.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.224+0000 7ff037c00640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff0300cbe90 con 0x7ff0300cb440 2024-03-31T15:11:14.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.224+0000 7ff037c00640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff0300cc170 con 0x7ff030070030 2024-03-31T15:11:14.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.224+0000 7ff037c00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7ff03015fb90 con 0x7ff03006dbc0 2024-03-31T15:11:14.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.224+0000 7ff027e00640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7ff0300cb440 0x7ff03006f920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:51520/0 (socket says 172.21.15.16:51520) 2024-03-31T15:11:14.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.224+0000 7ff027e00640 1 -- 172.21.15.16:0/3003937452 learned_addr learned my addr 172.21.15.16:0/3003937452 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:11:14.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.224+0000 7ff035200640 1 --2- 172.21.15.16:0/3003937452 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff03006dbc0 0x7ff03015f650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:14.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.224+0000 7ff02fe00640 1 -- 172.21.15.16:0/3003937452 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2750580607 0 0) 0x7ff0300cbe90 con 0x7ff0300cb440 2024-03-31T15:11:14.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.228+0000 7ff02fe00640 1 -- 172.21.15.16:0/3003937452 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff014003530 con 0x7ff0300cb440 2024-03-31T15:11:14.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.228+0000 7ff02fe00640 1 -- 172.21.15.16:0/3003937452 <== mon.1 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3025009002 0 0) 0x7ff0300cc170 con 0x7ff030070030 2024-03-31T15:11:14.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.228+0000 7ff02fe00640 1 -- 172.21.15.16:0/3003937452 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff0300cbe90 con 0x7ff030070030 2024-03-31T15:11:14.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.228+0000 7ff02fe00640 1 -- 172.21.15.16:0/3003937452 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1647781446 0 0) 0x7ff014003530 con 0x7ff0300cb440 2024-03-31T15:11:14.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.228+0000 7ff02fe00640 1 -- 172.21.15.16:0/3003937452 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff03006dbc0 msgr2=0x7ff03015f650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:11:14.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.228+0000 7ff02fe00640 1 --2- 172.21.15.16:0/3003937452 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff03006dbc0 0x7ff03015f650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:14.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.228+0000 7ff02fe00640 1 -- 172.21.15.16:0/3003937452 >> v1:172.21.15.89:6789/0 conn(0x7ff030070030 legacy=0x7ff030162180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:11:14.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.228+0000 7ff02fe00640 1 -- 172.21.15.16:0/3003937452 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff03015ee50 con 0x7ff0300cb440 2024-03-31T15:11:14.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.228+0000 7ff02fe00640 1 -- 172.21.15.16:0/3003937452 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 214807623 0 0) 0x7ff018001600 con 0x7ff0300cb440 2024-03-31T15:11:14.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.228+0000 7ff02fe00640 1 -- 172.21.15.16:0/3003937452 >> v1:172.21.15.16:6789/0 conn(0x7ff0300cb440 legacy=0x7ff03006f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:11:14.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.228+0000 7ff02fe00640 1 --2- 172.21.15.16:0/3003937452 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff014003b80 0x7ff014003f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:14.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.228+0000 7ff02fe00640 1 --2- 172.21.15.16:0/3003937452 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff014004810 0x7ff014004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:14.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.228+0000 7ff02fe00640 1 --2- 172.21.15.16:0/3003937452 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff03006dbc0 0x7ff014005230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:14.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.228+0000 7ff02fe00640 1 -- 172.21.15.16:0/3003937452 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7ff03015fb90 con 0x7ff014004810 2024-03-31T15:11:14.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.228+0000 7ff02fe00640 1 -- 172.21.15.16:0/3003937452 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7ff01400bb60 con 0x7ff014003b80 2024-03-31T15:11:14.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.228+0000 7ff02fe00640 1 -- 172.21.15.16:0/3003937452 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7ff01400bd30 con 0x7ff03006dbc0 2024-03-31T15:11:14.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.228+0000 7ff027e00640 1 --2- 172.21.15.16:0/3003937452 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff014004810 0x7ff014004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:14.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.228+0000 7ff035200640 1 --2- 172.21.15.16:0/3003937452 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff014003b80 0x7ff014003f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:14.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.228+0000 7ff035c00640 1 --2- 172.21.15.16:0/3003937452 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff03006dbc0 0x7ff014005230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:14.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.228+0000 7ff027e00640 1 -- 172.21.15.16:0/3003937452 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff03006dbc0 msgr2=0x7ff014005230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:11:14.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.228+0000 7ff027e00640 1 --2- 172.21.15.16:0/3003937452 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff03006dbc0 0x7ff014005230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:14.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.228+0000 7ff027e00640 1 -- 172.21.15.16:0/3003937452 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff014003b80 msgr2=0x7ff014003f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:11:14.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.228+0000 7ff027e00640 1 --2- 172.21.15.16:0/3003937452 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff014003b80 0x7ff014003f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:14.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.228+0000 7ff027e00640 1 -- 172.21.15.16:0/3003937452 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff014003530 con 0x7ff014004810 2024-03-31T15:11:14.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.232+0000 7ff027e00640 1 --2- 172.21.15.16:0/3003937452 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff014004810 0x7ff014004c90 secure :-1 s=READY pgs=1213 cs=0 l=1 rev1=1 crypto rx=0x7ff018007fa0 tx=0x7ff01801caa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6355d589231682bd server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:11:14.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.232+0000 7ff02fe00640 1 -- 172.21.15.16:0/3003937452 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff018007b90 con 0x7ff014004810 2024-03-31T15:11:14.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.232+0000 7ff02fe00640 1 -- 172.21.15.16:0/3003937452 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff0180a9070 con 0x7ff014004810 2024-03-31T15:11:14.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.232+0000 7ff02fe00640 1 -- 172.21.15.16:0/3003937452 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff018005e40 con 0x7ff014004810 2024-03-31T15:11:14.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.232+0000 7ff037c00640 1 -- 172.21.15.16:0/3003937452 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff014004810 msgr2=0x7ff014004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:11:14.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.232+0000 7ff037c00640 1 --2- 172.21.15.16:0/3003937452 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff014004810 0x7ff014004c90 secure :-1 s=READY pgs=1213 cs=0 l=1 rev1=1 crypto rx=0x7ff018007fa0 tx=0x7ff01801caa0 comp rx=0 tx=0).stop 2024-03-31T15:11:14.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.232+0000 7ff035c00640 1 -- 172.21.15.16:0/3003937452 reap_dead start 2024-03-31T15:11:14.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.232+0000 7ff037c00640 1 -- 172.21.15.16:0/3003937452 shutdown_connections 2024-03-31T15:11:14.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.232+0000 7ff037c00640 1 -- 172.21.15.16:0/3003937452 >> 172.21.15.16:0/3003937452 conn(0x7ff030067330 msgr2=0x7ff030067740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:11:14.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.232+0000 7ff037c00640 1 -- 172.21.15.16:0/3003937452 shutdown_connections 2024-03-31T15:11:14.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.232+0000 7ff037c00640 1 -- 172.21.15.16:0/3003937452 wait complete. 2024-03-31T15:11:14.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.232+0000 7ff037c00640 1 Processor -- start 2024-03-31T15:11:14.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.232+0000 7ff037c00640 1 -- start start 2024-03-31T15:11:14.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.232+0000 7ff037c00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff014004810 0x7ff0301660e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:14.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.232+0000 7ff037c00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff0301f7490 0x7ff0301422d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:14.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.232+0000 7ff037c00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff0301fa1a0 0x7ff030160400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:14.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.232+0000 7ff037c00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7ff030162a10 con 0x7ff0301f7490 2024-03-31T15:11:14.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.232+0000 7ff037c00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7ff030162890 con 0x7ff014004810 2024-03-31T15:11:14.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.232+0000 7ff037c00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7ff030162b90 con 0x7ff0301fa1a0 2024-03-31T15:11:14.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.232+0000 7ff027e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff0301f7490 0x7ff0301422d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:14.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.236+0000 7ff027e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff0301f7490 0x7ff0301422d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37906/0 (socket says 172.21.15.16:37906) 2024-03-31T15:11:14.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.236+0000 7ff027e00640 1 -- 172.21.15.16:0/2110868847 learned_addr learned my addr 172.21.15.16:0/2110868847 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:11:14.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.236+0000 7ff035200640 1 --2- 172.21.15.16:0/2110868847 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff014004810 0x7ff0301660e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:14.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.236+0000 7ff035c00640 1 --2- 172.21.15.16:0/2110868847 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff0301fa1a0 0x7ff030160400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:14.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.236+0000 7ff027e00640 1 -- 172.21.15.16:0/2110868847 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff0301fa1a0 msgr2=0x7ff030160400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:11:14.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.236+0000 7ff027e00640 1 --2- 172.21.15.16:0/2110868847 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff0301fa1a0 0x7ff030160400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:14.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.236+0000 7ff027e00640 1 -- 172.21.15.16:0/2110868847 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff014004810 msgr2=0x7ff0301660e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:11:14.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.236+0000 7ff027e00640 1 --2- 172.21.15.16:0/2110868847 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff014004810 0x7ff0301660e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:14.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.236+0000 7ff027e00640 1 -- 172.21.15.16:0/2110868847 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff030142b00 con 0x7ff0301f7490 2024-03-31T15:11:14.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.236+0000 7ff027e00640 1 --2- 172.21.15.16:0/2110868847 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff0301f7490 0x7ff0301422d0 secure :-1 s=READY pgs=1214 cs=0 l=1 rev1=1 crypto rx=0x7ff01801cf80 tx=0x7ff014005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:11:14.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.236+0000 7ff02d600640 1 -- 172.21.15.16:0/2110868847 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff018007b90 con 0x7ff0301f7490 2024-03-31T15:11:14.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.236+0000 7ff02d600640 1 -- 172.21.15.16:0/2110868847 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff0180070a0 con 0x7ff0301f7490 2024-03-31T15:11:14.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.236+0000 7ff037c00640 1 -- 172.21.15.16:0/2110868847 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff030142de0 con 0x7ff0301f7490 2024-03-31T15:11:14.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.236+0000 7ff02d600640 1 -- 172.21.15.16:0/2110868847 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0180074b0 con 0x7ff0301f7490 2024-03-31T15:11:14.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.236+0000 7ff037c00640 1 -- 172.21.15.16:0/2110868847 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff0301433a0 con 0x7ff0301f7490 2024-03-31T15:11:14.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.236+0000 7ff037c00640 1 -- 172.21.15.16:0/2110868847 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff030069950 con 0x7ff0301f7490 2024-03-31T15:11:14.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.240+0000 7ff02d600640 1 -- 172.21.15.16:0/2110868847 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7ff0180a5570 con 0x7ff0301f7490 2024-03-31T15:11:14.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.240+0000 7ff02d600640 1 --2- 172.21.15.16:0/2110868847 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7ff00c068380 0x7ff00c06a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:14.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.240+0000 7ff02d600640 1 -- 172.21.15.16:0/2110868847 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 5824+0+0 (secure 0 0 0) 0x7ff0181202c0 con 0x7ff0301f7490 2024-03-31T15:11:14.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.240+0000 7ff035200640 1 --2- 172.21.15.16:0/2110868847 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7ff00c068380 0x7ff00c06a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:14.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.240+0000 7ff035200640 1 --2- 172.21.15.16:0/2110868847 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7ff00c068380 0x7ff00c06a840 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7ff020009a80 tx=0x7ff020009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:11:14.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.244+0000 7ff02d600640 1 -- 172.21.15.16:0/2110868847 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7ff0180167b0 con 0x7ff0301f7490 2024-03-31T15:11:14.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.448+0000 7ff037c00640 1 -- 172.21.15.16:0/2110868847 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff0300cf380 con 0x7ff0301f7490 2024-03-31T15:11:14.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.448+0000 7ff02d600640 1 -- 172.21.15.16:0/2110868847 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 68 v68) ==== 94+0+5181 (secure 0 0 0) 0x7ff0180f0790 con 0x7ff0301f7490 2024-03-31T15:11:14.441 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:11:14.441 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":68,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6494,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/2445558494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6835","nonce":2445558494},{"type":"v1","addr":"172.21.15.89:6837","nonce":2445558494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":56},{"gid":7270,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/3185913858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6833","nonce":3185913858},{"type":"v1","addr":"172.21.15.16:6835","nonce":3185913858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":56},{"gid":7279,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6836/1599468493","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":1599468493},{"type":"v1","addr":"172.21.15.89:6836","nonce":1599468493}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":56}],"filesystems":[{"mdsmap":{"epoch":68,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-31T15:09:50.926196+0000","modified":"2024-03-31T15:10:15.370488+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":82,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7749},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7749":{"gid":7749,"name":"c","rank":0,"incarnation":64,"state":"up:active","state_seq":6,"addr":"172.21.15.16:6834/818074038","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":818074038},{"type":"v1","addr":"172.21.15.16:6834","nonce":818074038}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7749,"qdb_cluster":[7749]},"id":6}]} 2024-03-31T15:11:14.441 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 68 2024-03-31T15:11:14.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.452+0000 7ff037c00640 1 -- 172.21.15.16:0/2110868847 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7ff00c068380 msgr2=0x7ff00c06a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:11:14.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.452+0000 7ff037c00640 1 --2- 172.21.15.16:0/2110868847 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7ff00c068380 0x7ff00c06a840 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7ff020009a80 tx=0x7ff020009190 comp rx=0 tx=0).stop 2024-03-31T15:11:14.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.452+0000 7ff037c00640 1 -- 172.21.15.16:0/2110868847 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff0301f7490 msgr2=0x7ff0301422d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:11:14.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.452+0000 7ff037c00640 1 --2- 172.21.15.16:0/2110868847 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff0301f7490 0x7ff0301422d0 secure :-1 s=READY pgs=1214 cs=0 l=1 rev1=1 crypto rx=0x7ff01801cf80 tx=0x7ff014005840 comp rx=0 tx=0).stop 2024-03-31T15:11:14.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.452+0000 7ff037c00640 1 -- 172.21.15.16:0/2110868847 shutdown_connections 2024-03-31T15:11:14.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.452+0000 7ff037c00640 1 --2- 172.21.15.16:0/2110868847 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7ff00c068380 0x7ff00c06a840 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:14.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.452+0000 7ff037c00640 1 --2- 172.21.15.16:0/2110868847 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff0301fa1a0 0x7ff030160400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:14.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.452+0000 7ff037c00640 1 --2- 172.21.15.16:0/2110868847 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff0301f7490 0x7ff0301422d0 unknown :-1 s=CLOSED pgs=1214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:14.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.452+0000 7ff037c00640 1 --2- 172.21.15.16:0/2110868847 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff014004810 0x7ff0301660e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:14.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.452+0000 7ff037c00640 1 -- 172.21.15.16:0/2110868847 >> 172.21.15.16:0/2110868847 conn(0x7ff030067330 msgr2=0x7ff030157fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:11:14.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.452+0000 7ff037c00640 1 -- 172.21.15.16:0/2110868847 shutdown_connections 2024-03-31T15:11:14.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.452+0000 7ff037c00640 1 -- 172.21.15.16:0/2110868847 wait complete. 2024-03-31T15:11:14.463 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c damage ls --format=json-pretty 2024-03-31T15:11:14.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.612+0000 7f4cc0c00640 1 Processor -- start 2024-03-31T15:11:14.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.612+0000 7f4cc0c00640 1 -- start start 2024-03-31T15:11:14.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.616+0000 7f4cc0c00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f4cbc0d0a60 0x7f4cbc0cea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:14.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.616+0000 7f4cc0c00640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4cbc15ca70 con 0x7f4cbc06f920 2024-03-31T15:11:14.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.616+0000 7f4cc0c00640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4cbc15cd50 con 0x7f4cbc0cef80 2024-03-31T15:11:14.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.616+0000 7f4cc0c00640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f4cbc13f670 con 0x7f4cbc0d0a60 2024-03-31T15:11:14.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.616+0000 7f4cb3e00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f4cbc0d0a60 0x7f4cbc0cea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:14.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.616+0000 7f4cb9e00640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f4cbc06f920 0x7f4cbc06fd00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:51532/0 (socket says 172.21.15.16:51532) 2024-03-31T15:11:14.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.616+0000 7f4cb3e00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f4cbc0d0a60 0x7f4cbc0cea40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37922/0 (socket says 172.21.15.16:37922) 2024-03-31T15:11:14.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.616+0000 7f4cb9e00640 1 -- 172.21.15.16:0/1526174303 learned_addr learned my addr 172.21.15.16:0/1526174303 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:11:14.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.616+0000 7f4cb3e00640 1 -- 172.21.15.16:0/1526174303 >> v1:172.21.15.89:6789/0 conn(0x7f4cbc0cef80 legacy=0x7f4cbc1419c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:11:14.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.616+0000 7f4cb3e00640 1 -- 172.21.15.16:0/1526174303 >> v1:172.21.15.16:6789/0 conn(0x7f4cbc06f920 legacy=0x7f4cbc06fd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:11:14.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.616+0000 7f4cb3e00640 1 -- 172.21.15.16:0/1526174303 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4cbc15d030 con 0x7f4cbc0d0a60 2024-03-31T15:11:14.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.620+0000 7f4cb3e00640 1 --2- 172.21.15.16:0/1526174303 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4cbc0d0a60 0x7f4cbc0cea40 secure :-1 s=READY pgs=1216 cs=0 l=1 rev1=1 crypto rx=0x7f4ca40097c0 tx=0x7f4ca401c460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d13bb264878e65a9 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:11:14.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.620+0000 7f4cb9400640 1 -- 172.21.15.16:0/1526174303 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ca40b0070 con 0x7f4cbc0d0a60 2024-03-31T15:11:14.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.620+0000 7f4cb9400640 1 -- 172.21.15.16:0/1526174303 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4ca40ab600 con 0x7f4cbc0d0a60 2024-03-31T15:11:14.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.620+0000 7f4cb9400640 1 -- 172.21.15.16:0/1526174303 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ca40ab9d0 con 0x7f4cbc0d0a60 2024-03-31T15:11:14.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.620+0000 7f4cc0c00640 1 -- 172.21.15.16:0/1526174303 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4cbc0d0a60 msgr2=0x7f4cbc0cea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:11:14.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.620+0000 7f4cc0c00640 1 --2- 172.21.15.16:0/1526174303 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4cbc0d0a60 0x7f4cbc0cea40 secure :-1 s=READY pgs=1216 cs=0 l=1 rev1=1 crypto rx=0x7f4ca40097c0 tx=0x7f4ca401c460 comp rx=0 tx=0).stop 2024-03-31T15:11:14.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.620+0000 7f4cc0c00640 1 -- 172.21.15.16:0/1526174303 shutdown_connections 2024-03-31T15:11:14.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.620+0000 7f4cc0c00640 1 --2- 172.21.15.16:0/1526174303 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4cbc0d0a60 0x7f4cbc0cea40 unknown :-1 s=CLOSED pgs=1216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:14.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.620+0000 7f4cc0c00640 1 -- 172.21.15.16:0/1526174303 >> 172.21.15.16:0/1526174303 conn(0x7f4cbc067930 msgr2=0x7f4cbc0b3ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:11:14.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.620+0000 7f4cc0c00640 1 -- 172.21.15.16:0/1526174303 shutdown_connections 2024-03-31T15:11:14.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.620+0000 7f4cc0c00640 1 -- 172.21.15.16:0/1526174303 wait complete. 2024-03-31T15:11:14.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.620+0000 7f4cc0c00640 1 Processor -- start 2024-03-31T15:11:14.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.620+0000 7f4cc0c00640 1 -- start start 2024-03-31T15:11:14.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.620+0000 7f4cc0c00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4cbc06f920 0x7f4cbc15e260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:14.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.620+0000 7f4cc0c00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f4cbc0cef80 0x7f4cbc13e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:14.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.620+0000 7f4cc0c00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4cbc0d0a60 0x7f4cbc13ebf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:14.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.620+0000 7f4cc0c00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f4cbc16a240 con 0x7f4cbc06f920 2024-03-31T15:11:14.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.620+0000 7f4cc0c00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f4cbc16a0c0 con 0x7f4cbc0d0a60 2024-03-31T15:11:14.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.620+0000 7f4cc0c00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f4cbc16a3c0 con 0x7f4cbc0cef80 2024-03-31T15:11:14.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.620+0000 7f4cb9e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4cbc06f920 0x7f4cbc15e260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:14.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.620+0000 7f4cb9e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4cbc06f920 0x7f4cbc15e260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37928/0 (socket says 172.21.15.16:37928) 2024-03-31T15:11:14.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.620+0000 7f4cb9e00640 1 -- 172.21.15.16:0/4112716075 learned_addr learned my addr 172.21.15.16:0/4112716075 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:11:14.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.620+0000 7f4cba800640 1 --2- 172.21.15.16:0/4112716075 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4cbc0d0a60 0x7f4cbc13ebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:14.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.620+0000 7f4cb3e00640 1 --2- 172.21.15.16:0/4112716075 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f4cbc0cef80 0x7f4cbc13e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:14.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.624+0000 7f4cb9e00640 1 -- 172.21.15.16:0/4112716075 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f4cbc0cef80 msgr2=0x7f4cbc13e6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:11:14.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.624+0000 7f4cb9e00640 1 --2- 172.21.15.16:0/4112716075 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f4cbc0cef80 0x7f4cbc13e6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:14.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.624+0000 7f4cb9e00640 1 -- 172.21.15.16:0/4112716075 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4cbc0d0a60 msgr2=0x7f4cbc13ebf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:11:14.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.624+0000 7f4cb9e00640 1 --2- 172.21.15.16:0/4112716075 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4cbc0d0a60 0x7f4cbc13ebf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:14.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.624+0000 7f4cb9e00640 1 -- 172.21.15.16:0/4112716075 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4cbc15da60 con 0x7f4cbc06f920 2024-03-31T15:11:14.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.624+0000 7f4cb9e00640 1 --2- 172.21.15.16:0/4112716075 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4cbc06f920 0x7f4cbc15e260 secure :-1 s=READY pgs=1217 cs=0 l=1 rev1=1 crypto rx=0x7f4cac00c450 tx=0x7f4cac00c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:11:14.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.624+0000 7f4cb2000640 1 -- 172.21.15.16:0/4112716075 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cac017070 con 0x7f4cbc06f920 2024-03-31T15:11:14.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.624+0000 7f4cb2000640 1 -- 172.21.15.16:0/4112716075 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4cac005cf0 con 0x7f4cbc06f920 2024-03-31T15:11:14.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.624+0000 7f4cb2000640 1 -- 172.21.15.16:0/4112716075 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cac007290 con 0x7f4cbc06f920 2024-03-31T15:11:14.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.624+0000 7f4cc0c00640 1 -- 172.21.15.16:0/4112716075 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4cbc15dd40 con 0x7f4cbc06f920 2024-03-31T15:11:14.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.624+0000 7f4cc0c00640 1 -- 172.21.15.16:0/4112716075 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4cbc06dc70 con 0x7f4cbc06f920 2024-03-31T15:11:14.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.624+0000 7f4cb2000640 1 -- 172.21.15.16:0/4112716075 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f4cac01a070 con 0x7f4cbc06f920 2024-03-31T15:11:14.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.628+0000 7f4cb2000640 1 --2- 172.21.15.16:0/4112716075 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f4c980683b0 0x7f4c9806a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:14.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.628+0000 7f4cb2000640 1 -- 172.21.15.16:0/4112716075 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 5824+0+0 (secure 0 0 0) 0x7f4cac085d30 con 0x7f4cbc06f920 2024-03-31T15:11:14.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.628+0000 7f4cc409a000 1 Processor -- start 2024-03-31T15:11:14.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.628+0000 7f4cc409a000 1 -- start start 2024-03-31T15:11:14.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.628+0000 7f4cb3e00640 1 --2- 172.21.15.16:0/4112716075 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f4c980683b0 0x7f4c9806a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:14.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.628+0000 7f4cb3e00640 1 --2- 172.21.15.16:0/4112716075 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f4c980683b0 0x7f4c9806a870 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f4ca40ad000 tx=0x7f4ca4008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:11:14.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.628+0000 7f4cc409a000 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55bc2e503980 0x55bc2e503d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:14.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.628+0000 7f4cc409a000 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55bc2e5042a0 0x55bc2e4b6dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:14.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.628+0000 7f4cc409a000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bc2e4b7310 0x55bc2e4b76f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:14.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.628+0000 7f4cc409a000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55bc2e307c50 con 0x55bc2e4b7310 2024-03-31T15:11:14.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.628+0000 7f4cc409a000 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55bc2e2a48a0 con 0x55bc2e503980 2024-03-31T15:11:14.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.628+0000 7f4cc409a000 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x55bc2e303c00 con 0x55bc2e5042a0 2024-03-31T15:11:14.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.628+0000 7f4cba800640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bc2e4b7310 0x55bc2e4b76f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:14.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.628+0000 7f4cba800640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bc2e4b7310 0x55bc2e4b76f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37930/0 (socket says 172.21.15.16:37930) 2024-03-31T15:11:14.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.628+0000 7f4cba800640 1 -- 172.21.15.16:0/659570006 learned_addr learned my addr 172.21.15.16:0/659570006 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:11:14.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.628+0000 7f4cb3e00640 1 --2- 172.21.15.16:0/659570006 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55bc2e5042a0 0x55bc2e4b6dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:14.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.628+0000 7f4cb9e00640 1 --2- 172.21.15.16:0/659570006 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55bc2e503980 0x55bc2e503d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:14.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.628+0000 7f4cba800640 1 -- 172.21.15.16:0/659570006 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55bc2e5042a0 msgr2=0x55bc2e4b6dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:11:14.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.628+0000 7f4cba800640 1 --2- 172.21.15.16:0/659570006 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55bc2e5042a0 0x55bc2e4b6dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:14.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.628+0000 7f4cba800640 1 -- 172.21.15.16:0/659570006 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55bc2e503980 msgr2=0x55bc2e503d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:11:14.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.628+0000 7f4cba800640 1 --2- 172.21.15.16:0/659570006 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55bc2e503980 0x55bc2e503d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:14.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.628+0000 7f4cba800640 1 -- 172.21.15.16:0/659570006 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bc2e2e1ba0 con 0x55bc2e4b7310 2024-03-31T15:11:14.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.628+0000 7f4cb3e00640 1 --2- 172.21.15.16:0/659570006 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55bc2e5042a0 0x55bc2e4b6dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:11:14.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.628+0000 7f4cb9e00640 1 --2- 172.21.15.16:0/659570006 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55bc2e503980 0x55bc2e503d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:11:14.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.628+0000 7f4cba800640 1 --2- 172.21.15.16:0/659570006 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bc2e4b7310 0x55bc2e4b76f0 secure :-1 s=READY pgs=1218 cs=0 l=1 rev1=1 crypto rx=0x7f4cb40056d0 tx=0x7f4cb400f5a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e3090381bd6a02f3 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:11:14.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.628+0000 7f4caaa00640 1 -- 172.21.15.16:0/659570006 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cb4018020 con 0x55bc2e4b7310 2024-03-31T15:11:14.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.628+0000 7f4caaa00640 1 -- 172.21.15.16:0/659570006 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4cb400fe60 con 0x55bc2e4b7310 2024-03-31T15:11:14.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.628+0000 7f4caaa00640 1 -- 172.21.15.16:0/659570006 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cb4014690 con 0x55bc2e4b7310 2024-03-31T15:11:14.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.632+0000 7f4cc409a000 1 -- 172.21.15.16:0/659570006 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bc2e4b7310 msgr2=0x55bc2e4b76f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:11:14.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.632+0000 7f4cc409a000 1 --2- 172.21.15.16:0/659570006 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bc2e4b7310 0x55bc2e4b76f0 secure :-1 s=READY pgs=1218 cs=0 l=1 rev1=1 crypto rx=0x7f4cb40056d0 tx=0x7f4cb400f5a0 comp rx=0 tx=0).stop 2024-03-31T15:11:14.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.632+0000 7f4cc409a000 1 -- 172.21.15.16:0/659570006 shutdown_connections 2024-03-31T15:11:14.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.632+0000 7f4cc409a000 1 --2- 172.21.15.16:0/659570006 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bc2e4b7310 0x55bc2e4b76f0 unknown :-1 s=CLOSED pgs=1218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:14.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.632+0000 7f4cc409a000 1 --2- 172.21.15.16:0/659570006 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55bc2e5042a0 0x55bc2e4b6dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:14.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.632+0000 7f4cc409a000 1 --2- 172.21.15.16:0/659570006 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55bc2e503980 0x55bc2e503d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:14.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.632+0000 7f4cc409a000 1 -- 172.21.15.16:0/659570006 >> 172.21.15.16:0/659570006 conn(0x55bc2e2b49f0 msgr2=0x55bc2e4e6730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:11:14.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.632+0000 7f4cc409a000 1 -- 172.21.15.16:0/659570006 shutdown_connections 2024-03-31T15:11:14.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.632+0000 7f4cc409a000 1 -- 172.21.15.16:0/659570006 wait complete. 2024-03-31T15:11:14.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.632+0000 7f4cc409a000 1 -- start start 2024-03-31T15:11:14.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.632+0000 7f4cc409a000 1 Processor -- start 2024-03-31T15:11:14.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.632+0000 7f4cc409a000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bc2e4b7310 0x55bc2e503110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:14.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.632+0000 7f4cc409a000 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55bc2e503980 0x55bc2e4bca80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:14.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.632+0000 7f4cc409a000 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55bc2e5042a0 0x55bc2e4bcfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:14.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.632+0000 7f4cc409a000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55bc2e2c50a0 con 0x55bc2e4b7310 2024-03-31T15:11:14.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.632+0000 7f4cc409a000 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55bc2e2c8400 con 0x55bc2e5042a0 2024-03-31T15:11:14.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.632+0000 7f4cc409a000 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x55bc2e2c3600 con 0x55bc2e503980 2024-03-31T15:11:14.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.632+0000 7f4cc409a000 10 client.0 _ll_register_callbacks cb 0x55bc2e4a4810 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T15:11:14.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.632+0000 7f4cb9e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bc2e4b7310 0x55bc2e503110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:14.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.632+0000 7f4cb9e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bc2e4b7310 0x55bc2e503110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37944/0 (socket says 172.21.15.16:37944) 2024-03-31T15:11:14.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.632+0000 7f4cb9e00640 1 -- 172.21.15.16:0/3397644383 learned_addr learned my addr 172.21.15.16:0/3397644383 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:11:14.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.636+0000 7f4cb3e00640 1 --2- 172.21.15.16:0/3397644383 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55bc2e503980 0x55bc2e4bca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:14.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.636+0000 7f4cba800640 1 --2- 172.21.15.16:0/3397644383 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55bc2e5042a0 0x55bc2e4bcfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:14.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.636+0000 7f4cb9e00640 1 -- 172.21.15.16:0/3397644383 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55bc2e503980 msgr2=0x55bc2e4bca80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:11:14.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.636+0000 7f4cb9e00640 1 --2- 172.21.15.16:0/3397644383 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55bc2e503980 0x55bc2e4bca80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:14.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.636+0000 7f4cb9e00640 1 -- 172.21.15.16:0/3397644383 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55bc2e5042a0 msgr2=0x55bc2e4bcfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:11:14.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.636+0000 7f4cb9e00640 1 --2- 172.21.15.16:0/3397644383 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55bc2e5042a0 0x55bc2e4bcfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:14.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.636+0000 7f4cb9e00640 1 -- 172.21.15.16:0/3397644383 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bc2e34d160 con 0x55bc2e4b7310 2024-03-31T15:11:14.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.636+0000 7f4cb3e00640 1 --2- 172.21.15.16:0/3397644383 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55bc2e503980 0x55bc2e4bca80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:11:14.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.636+0000 7f4cb9e00640 1 --2- 172.21.15.16:0/3397644383 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bc2e4b7310 0x55bc2e503110 secure :-1 s=READY pgs=1219 cs=0 l=1 rev1=1 crypto rx=0x7f4cac052000 tx=0x7f4cac055ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=868fd31afe9c2c60 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:11:14.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.636+0000 7f4c9f400640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-31T15:11:14.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.636+0000 7f4c9f400640 1 -- 172.21.15.16:0/3397644383 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cac082060 con 0x55bc2e4b7310 2024-03-31T15:11:14.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.636+0000 7f4c9f400640 1 -- 172.21.15.16:0/3397644383 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 5824+0+0 (secure 0 0 0) 0x7f4cac05e4d0 con 0x55bc2e4b7310 2024-03-31T15:11:14.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.636+0000 7f4cc409a000 1 -- 172.21.15.16:0/3397644383 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4cb400fe60 con 0x55bc2e4b7310 2024-03-31T15:11:14.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.636+0000 7f4c9f400640 1 -- 172.21.15.16:0/3397644383 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=68) ==== 24+0+0 (secure 0 0 0) 0x7f4cac07db50 con 0x55bc2e4b7310 2024-03-31T15:11:14.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.636+0000 7f4cc409a000 10 client.7998 fetch_fsmap learned FSMap version 68 2024-03-31T15:11:14.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.636+0000 7f4cc409a000 1 -- 172.21.15.16:0/3397644383 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=68,monmap=2+}) -- 0x55bc2e2a29c0 con 0x55bc2e4b7310 2024-03-31T15:11:14.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.636+0000 7f4c9f400640 1 -- 172.21.15.16:0/3397644383 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4cb400fe60 con 0x55bc2e4b7310 2024-03-31T15:11:14.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.636+0000 7f4c9f400640 1 -- 172.21.15.16:0/3397644383 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 68) ==== 3494+0+0 (secure 0 0 0) 0x7f4cac054ff0 con 0x55bc2e4b7310 2024-03-31T15:11:14.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.636+0000 7f4cc409a000 10 client.7998 fetch_fsmap finished waiting for FSMap version 68 2024-03-31T15:11:14.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.636+0000 7f4cc409a000 10 client.7998 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-31T15:11:14.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.636+0000 7f4cc409a000 20 client.7998 populate_metadata read hostname 'smithi016' 2024-03-31T15:11:14.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.636+0000 7f4cc409a000 1 --2- 172.21.15.16:0/3397644383 >> [v2:172.21.15.16:6832/818074038,v1:172.21.15.16:6834/818074038] conn(0x55bc2e50adc0 0x55bc2e50b1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:14.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.636+0000 7f4cc409a000 4 client.7998 mds_command: new command op to 7749 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-31T15:11:14.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.636+0000 7f4cc409a000 1 -- 172.21.15.16:0/3397644383 --> [v2:172.21.15.16:6832/818074038,v1:172.21.15.16:6834/818074038] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55bc2e2a2e40 con 0x55bc2e50adc0 2024-03-31T15:11:14.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.636+0000 7f4cb3e00640 1 --2- 172.21.15.16:0/3397644383 >> [v2:172.21.15.16:6832/818074038,v1:172.21.15.16:6834/818074038] conn(0x55bc2e50adc0 0x55bc2e50b1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:14.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.636+0000 7f4cb3e00640 1 --2- 172.21.15.16:0/3397644383 >> [v2:172.21.15.16:6832/818074038,v1:172.21.15.16:6834/818074038] conn(0x55bc2e50adc0 0x55bc2e50b1a0 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=32da7cdf010eecec server_cookie=503cd2ded17e6a5d in_seq=0 out_seq=0 2024-03-31T15:11:14.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.636+0000 7f4c9f400640 10 client.7998 ms_handle_connect on v2:172.21.15.16:6832/818074038 2024-03-31T15:11:14.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.636+0000 7f4c9f400640 1 -- 172.21.15.16:0/3397644383 <== mds.0 v2:172.21.15.16:6832/818074038 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55bc2e2a2e40 con 0x55bc2e50adc0 2024-03-31T15:11:14.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.636+0000 7f4c9f400640 10 client.7998 handle_command_reply: tid=0 2024-03-31T15:11:14.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.636+0000 7f4cc409a000 1 client.7998 shutdown 2024-03-31T15:11:14.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.636+0000 7f4cc409a000 1 -- 172.21.15.16:0/3397644383 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bc2e4b7310 msgr2=0x55bc2e503110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:11:14.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.636+0000 7f4cc409a000 1 --2- 172.21.15.16:0/3397644383 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bc2e4b7310 0x55bc2e503110 secure :-1 s=READY pgs=1219 cs=0 l=1 rev1=1 crypto rx=0x7f4cac052000 tx=0x7f4cac055ce0 comp rx=0 tx=0).stop 2024-03-31T15:11:14.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.636+0000 7f4cc409a000 1 -- 172.21.15.16:0/3397644383 shutdown_connections 2024-03-31T15:11:14.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.636+0000 7f4cc409a000 1 --2- 172.21.15.16:0/3397644383 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55bc2e5042a0 0x55bc2e4bcfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:14.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.636+0000 7f4cc409a000 1 --2- 172.21.15.16:0/3397644383 >> [v2:172.21.15.16:6832/818074038,v1:172.21.15.16:6834/818074038] conn(0x55bc2e50adc0 0x55bc2e50b1a0 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:14.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.636+0000 7f4cc409a000 1 --2- 172.21.15.16:0/3397644383 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55bc2e503980 0x55bc2e4bca80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:14.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.636+0000 7f4cc409a000 1 --2- 172.21.15.16:0/3397644383 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bc2e4b7310 0x55bc2e503110 unknown :-1 s=CLOSED pgs=1219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:14.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.636+0000 7f4c9f400640 0 client.7998 ms_handle_reset on v2:172.21.15.16:6832/818074038 2024-03-31T15:11:14.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.636+0000 7f4cc409a000 1 -- 172.21.15.16:0/3397644383 >> 172.21.15.16:0/3397644383 conn(0x55bc2e2b49f0 msgr2=0x55bc2e4e6520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:11:14.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.636+0000 7f4cc409a000 1 -- 172.21.15.16:0/3397644383 shutdown_connections 2024-03-31T15:11:14.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.636+0000 7f4cc409a000 1 -- 172.21.15.16:0/3397644383 wait complete. 2024-03-31T15:11:14.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.636+0000 7f4cc409a000 20 client.7998 trim_cache size 0 max 16384 2024-03-31T15:11:14.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.652+0000 7f4cc409a000 1 Processor -- start 2024-03-31T15:11:14.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.652+0000 7f4cc409a000 1 -- start start 2024-03-31T15:11:14.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.652+0000 7f4cc409a000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bc2e4b7310 0x55bc2e4bf0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:14.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.652+0000 7f4cc409a000 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55bc2e503980 0x55bc2e4bb5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:14.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.652+0000 7f4cc409a000 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55bc2e5042a0 0x55bc2e4bbb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:14.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.652+0000 7f4cc409a000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f4cbc202e80 con 0x55bc2e4b7310 2024-03-31T15:11:14.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.652+0000 7f4cc409a000 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f4cbc165740 con 0x55bc2e503980 2024-03-31T15:11:14.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.652+0000 7f4cc409a000 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f4cbc165d90 con 0x55bc2e5042a0 2024-03-31T15:11:14.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.652+0000 7f4cb9e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bc2e4b7310 0x55bc2e4bf0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:14.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.652+0000 7f4cb9e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bc2e4b7310 0x55bc2e4bf0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37952/0 (socket says 172.21.15.16:37952) 2024-03-31T15:11:14.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.652+0000 7f4cb9e00640 1 -- 172.21.15.16:0/926902519 learned_addr learned my addr 172.21.15.16:0/926902519 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:11:14.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.652+0000 7f4cba800640 1 --2- 172.21.15.16:0/926902519 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55bc2e5042a0 0x55bc2e4bbb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:14.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.652+0000 7f4cb3e00640 1 --2- 172.21.15.16:0/926902519 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55bc2e503980 0x55bc2e4bb5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:14.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.652+0000 7f4cb9e00640 1 -- 172.21.15.16:0/926902519 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55bc2e5042a0 msgr2=0x55bc2e4bbb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:11:14.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.652+0000 7f4cb9e00640 1 --2- 172.21.15.16:0/926902519 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55bc2e5042a0 0x55bc2e4bbb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:14.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.652+0000 7f4cb9e00640 1 -- 172.21.15.16:0/926902519 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55bc2e503980 msgr2=0x55bc2e4bb5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:11:14.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.652+0000 7f4cb9e00640 1 --2- 172.21.15.16:0/926902519 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55bc2e503980 0x55bc2e4bb5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:14.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.652+0000 7f4cb9e00640 1 -- 172.21.15.16:0/926902519 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bc2e2a4af0 con 0x55bc2e4b7310 2024-03-31T15:11:14.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.652+0000 7f4cb3e00640 1 --2- 172.21.15.16:0/926902519 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55bc2e503980 0x55bc2e4bb5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:11:14.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.652+0000 7f4cba800640 1 --2- 172.21.15.16:0/926902519 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55bc2e5042a0 0x55bc2e4bbb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:11:14.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.652+0000 7f4cb9e00640 1 --2- 172.21.15.16:0/926902519 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bc2e4b7310 0x55bc2e4bf0b0 secure :-1 s=READY pgs=1220 cs=0 l=1 rev1=1 crypto rx=0x7f4cac000fc0 tx=0x7f4cac01e050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a35f1d3ae576172d server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:11:14.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.652+0000 7f4caa000640 1 -- 172.21.15.16:0/926902519 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cac055500 con 0x55bc2e4b7310 2024-03-31T15:11:14.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.652+0000 7f4caa000640 1 -- 172.21.15.16:0/926902519 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4cac05e2d0 con 0x55bc2e4b7310 2024-03-31T15:11:14.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.652+0000 7f4caa000640 1 -- 172.21.15.16:0/926902519 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cac007430 con 0x55bc2e4b7310 2024-03-31T15:11:14.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.652+0000 7f4cc409a000 1 -- 172.21.15.16:0/926902519 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bc2e4b7310 msgr2=0x55bc2e4bf0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:11:14.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.652+0000 7f4cc409a000 1 --2- 172.21.15.16:0/926902519 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bc2e4b7310 0x55bc2e4bf0b0 secure :-1 s=READY pgs=1220 cs=0 l=1 rev1=1 crypto rx=0x7f4cac000fc0 tx=0x7f4cac01e050 comp rx=0 tx=0).stop 2024-03-31T15:11:14.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.652+0000 7f4cc409a000 1 -- 172.21.15.16:0/926902519 shutdown_connections 2024-03-31T15:11:14.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.652+0000 7f4cc409a000 1 --2- 172.21.15.16:0/926902519 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55bc2e5042a0 0x55bc2e4bbb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:14.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.652+0000 7f4cc409a000 1 --2- 172.21.15.16:0/926902519 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55bc2e503980 0x55bc2e4bb5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:14.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.652+0000 7f4cc409a000 1 --2- 172.21.15.16:0/926902519 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bc2e4b7310 0x55bc2e4bf0b0 unknown :-1 s=CLOSED pgs=1220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:14.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.652+0000 7f4cc409a000 1 -- 172.21.15.16:0/926902519 >> 172.21.15.16:0/926902519 conn(0x55bc2e2b49f0 msgr2=0x55bc2e4b5dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:11:14.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.652+0000 7f4cc409a000 1 -- 172.21.15.16:0/926902519 shutdown_connections 2024-03-31T15:11:14.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.656+0000 7f4cc409a000 1 -- 172.21.15.16:0/926902519 wait complete. 2024-03-31T15:11:14.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.656+0000 7f4cc409a000 1 -- start start 2024-03-31T15:11:14.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.656+0000 7f4cc409a000 1 Processor -- start 2024-03-31T15:11:14.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.656+0000 7f4cc409a000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bc2e4b7310 0x55bc2e51b170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:14.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.656+0000 7f4cc409a000 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55bc2e503980 0x55bc2e506690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:14.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.656+0000 7f4cc409a000 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55bc2e5042a0 0x55bc2e506bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:14.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.656+0000 7f4cc409a000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55bc2e30e860 con 0x55bc2e4b7310 2024-03-31T15:11:14.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.656+0000 7f4cc409a000 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55bc2e4bf7e0 con 0x55bc2e5042a0 2024-03-31T15:11:14.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.656+0000 7f4cc409a000 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x55bc2e309ca0 con 0x55bc2e503980 2024-03-31T15:11:14.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.656+0000 7f4cc409a000 10 client.0 _ll_register_callbacks cb 0x7f4ca40ae070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T15:11:14.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.656+0000 7f4cb9e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bc2e4b7310 0x55bc2e51b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:14.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.656+0000 7f4cb9e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bc2e4b7310 0x55bc2e51b170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37964/0 (socket says 172.21.15.16:37964) 2024-03-31T15:11:14.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.656+0000 7f4cb9e00640 1 -- 172.21.15.16:0/713987249 learned_addr learned my addr 172.21.15.16:0/713987249 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:11:14.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.656+0000 7f4cb3e00640 1 --2- 172.21.15.16:0/713987249 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55bc2e503980 0x55bc2e506690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:14.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.656+0000 7f4cba800640 1 --2- 172.21.15.16:0/713987249 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55bc2e5042a0 0x55bc2e506bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:14.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.656+0000 7f4cb9e00640 1 -- 172.21.15.16:0/713987249 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55bc2e503980 msgr2=0x55bc2e506690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:11:14.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.656+0000 7f4cb9e00640 1 --2- 172.21.15.16:0/713987249 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55bc2e503980 0x55bc2e506690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:14.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.656+0000 7f4cb9e00640 1 -- 172.21.15.16:0/713987249 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55bc2e5042a0 msgr2=0x55bc2e506bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:11:14.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.656+0000 7f4cb9e00640 1 --2- 172.21.15.16:0/713987249 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55bc2e5042a0 0x55bc2e506bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:14.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.656+0000 7f4cb9e00640 1 -- 172.21.15.16:0/713987249 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bc2e360320 con 0x55bc2e4b7310 2024-03-31T15:11:14.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.656+0000 7f4cba800640 1 --2- 172.21.15.16:0/713987249 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55bc2e5042a0 0x55bc2e506bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:11:14.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.656+0000 7f4cb3e00640 1 --2- 172.21.15.16:0/713987249 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55bc2e503980 0x55bc2e506690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:11:14.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.656+0000 7f4cb9e00640 1 --2- 172.21.15.16:0/713987249 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bc2e4b7310 0x55bc2e51b170 secure :-1 s=READY pgs=1221 cs=0 l=1 rev1=1 crypto rx=0x7f4cac00ce10 tx=0x7f4cac0840c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c736f30eb4460a86 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:11:14.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.656+0000 7f4c9f400640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-31T15:11:14.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.656+0000 7f4c9f400640 1 -- 172.21.15.16:0/713987249 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cac078070 con 0x55bc2e4b7310 2024-03-31T15:11:14.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.656+0000 7f4c9f400640 1 -- 172.21.15.16:0/713987249 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 5824+0+0 (secure 0 0 0) 0x7f4cac0518f0 con 0x55bc2e4b7310 2024-03-31T15:11:14.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.656+0000 7f4cc409a000 1 -- 172.21.15.16:0/713987249 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4cac05e2d0 con 0x55bc2e4b7310 2024-03-31T15:11:14.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.656+0000 7f4c9f400640 1 -- 172.21.15.16:0/713987249 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=68) ==== 24+0+0 (secure 0 0 0) 0x7f4cac055500 con 0x55bc2e4b7310 2024-03-31T15:11:14.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.656+0000 7f4cc409a000 10 client.8004 fetch_fsmap learned FSMap version 68 2024-03-31T15:11:14.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.656+0000 7f4cc409a000 1 -- 172.21.15.16:0/713987249 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=68,monmap=2+}) -- 0x55bc2e29ba80 con 0x55bc2e4b7310 2024-03-31T15:11:14.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.656+0000 7f4c9f400640 1 -- 172.21.15.16:0/713987249 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4cac05e2d0 con 0x55bc2e4b7310 2024-03-31T15:11:14.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.656+0000 7f4c9f400640 1 -- 172.21.15.16:0/713987249 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 68) ==== 3494+0+0 (secure 0 0 0) 0x7f4cac07b020 con 0x55bc2e4b7310 2024-03-31T15:11:14.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.656+0000 7f4cc409a000 10 client.8004 fetch_fsmap finished waiting for FSMap version 68 2024-03-31T15:11:14.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.656+0000 7f4cc409a000 10 client.8004 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-31T15:11:14.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.656+0000 7f4cc409a000 20 client.8004 populate_metadata read hostname 'smithi016' 2024-03-31T15:11:14.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.656+0000 7f4cc409a000 1 --2- 172.21.15.16:0/713987249 >> [v2:172.21.15.16:6832/818074038,v1:172.21.15.16:6834/818074038] conn(0x55bc2e507110 0x55bc2e573a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:14.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.656+0000 7f4cc409a000 4 client.8004 mds_command: new command op to 7749 tid=0 multi_id=0 [{"prefix": "damage ls", "format": "json-pretty"}] 2024-03-31T15:11:14.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.656+0000 7f4cc409a000 1 -- 172.21.15.16:0/713987249 --> [v2:172.21.15.16:6832/818074038,v1:172.21.15.16:6834/818074038] -- command(tid 0: {"prefix": "damage ls", "format": "json-pretty"}) -- 0x55bc2e315190 con 0x55bc2e507110 2024-03-31T15:11:14.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.656+0000 7f4cb3e00640 1 --2- 172.21.15.16:0/713987249 >> [v2:172.21.15.16:6832/818074038,v1:172.21.15.16:6834/818074038] conn(0x55bc2e507110 0x55bc2e573a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:14.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.660+0000 7f4cb3e00640 1 --2- 172.21.15.16:0/713987249 >> [v2:172.21.15.16:6832/818074038,v1:172.21.15.16:6834/818074038] conn(0x55bc2e507110 0x55bc2e573a20 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=11a73092282f13f6 server_cookie=987611a0fd28e378 in_seq=0 out_seq=0 2024-03-31T15:11:14.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.660+0000 7f4c9f400640 10 client.8004 ms_handle_connect on v2:172.21.15.16:6832/818074038 2024-03-31T15:11:14.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.660+0000 7f4c9f400640 1 -- 172.21.15.16:0/713987249 <== mds.0 v2:172.21.15.16:6832/818074038 1 ==== command_reply(tid 0: 0 ) ==== 8+0+144 (crc 0 0 0) 0x55bc2e315190 con 0x55bc2e507110 2024-03-31T15:11:14.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.660+0000 7f4c9f400640 10 client.8004 handle_command_reply: tid=0 2024-03-31T15:11:14.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.660+0000 7f4cc409a000 1 client.8004 shutdown 2024-03-31T15:11:14.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.660+0000 7f4cc409a000 1 -- 172.21.15.16:0/713987249 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bc2e4b7310 msgr2=0x55bc2e51b170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:11:14.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.660+0000 7f4cc409a000 1 --2- 172.21.15.16:0/713987249 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bc2e4b7310 0x55bc2e51b170 secure :-1 s=READY pgs=1221 cs=0 l=1 rev1=1 crypto rx=0x7f4cac00ce10 tx=0x7f4cac0840c0 comp rx=0 tx=0).stop 2024-03-31T15:11:14.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.660+0000 7f4cc409a000 1 -- 172.21.15.16:0/713987249 shutdown_connections 2024-03-31T15:11:14.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.660+0000 7f4cc409a000 1 --2- 172.21.15.16:0/713987249 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55bc2e5042a0 0x55bc2e506bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:14.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.660+0000 7f4cc409a000 1 --2- 172.21.15.16:0/713987249 >> [v2:172.21.15.16:6832/818074038,v1:172.21.15.16:6834/818074038] conn(0x55bc2e507110 0x55bc2e573a20 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:14.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.660+0000 7f4cc409a000 1 --2- 172.21.15.16:0/713987249 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55bc2e503980 0x55bc2e506690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:14.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.660+0000 7f4cc409a000 1 --2- 172.21.15.16:0/713987249 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bc2e4b7310 0x55bc2e51b170 unknown :-1 s=CLOSED pgs=1221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:14.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.660+0000 7f4cc409a000 1 -- 172.21.15.16:0/713987249 >> 172.21.15.16:0/713987249 conn(0x55bc2e2b49f0 msgr2=0x55bc2e4b5dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:11:14.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.660+0000 7f4c9f400640 0 client.8004 ms_handle_reset on v2:172.21.15.16:6832/818074038 2024-03-31T15:11:14.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.660+0000 7f4cc409a000 1 -- 172.21.15.16:0/713987249 shutdown_connections 2024-03-31T15:11:14.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.660+0000 7f4cc409a000 1 -- 172.21.15.16:0/713987249 wait complete. 2024-03-31T15:11:14.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.660+0000 7f4cc409a000 20 client.8004 trim_cache size 0 max 16384 2024-03-31T15:11:14.656 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:11:14.656 INFO:teuthology.orchestra.run.smithi016.stdout:[ 2024-03-31T15:11:14.656 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-31T15:11:14.656 INFO:teuthology.orchestra.run.smithi016.stdout: "damage_type": "backtrace", 2024-03-31T15:11:14.656 INFO:teuthology.orchestra.run.smithi016.stdout: "id": 1050930771, 2024-03-31T15:11:14.656 INFO:teuthology.orchestra.run.smithi016.stdout: "ino": 1099511627777, 2024-03-31T15:11:14.656 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/testdir/hardlink" 2024-03-31T15:11:14.657 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-31T15:11:14.657 INFO:teuthology.orchestra.run.smithi016.stdout:] 2024-03-31T15:11:14.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.660+0000 7f4cc0c00640 1 -- 172.21.15.16:0/4112716075 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f4c980683b0 msgr2=0x7f4c9806a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:11:14.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.660+0000 7f4cc0c00640 1 --2- 172.21.15.16:0/4112716075 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f4c980683b0 0x7f4c9806a870 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f4ca40ad000 tx=0x7f4ca4008040 comp rx=0 tx=0).stop 2024-03-31T15:11:14.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.660+0000 7f4cc0c00640 1 -- 172.21.15.16:0/4112716075 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4cbc06f920 msgr2=0x7f4cbc15e260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:11:14.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.660+0000 7f4cc0c00640 1 --2- 172.21.15.16:0/4112716075 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4cbc06f920 0x7f4cbc15e260 secure :-1 s=READY pgs=1217 cs=0 l=1 rev1=1 crypto rx=0x7f4cac00c450 tx=0x7f4cac00c910 comp rx=0 tx=0).stop 2024-03-31T15:11:14.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.660+0000 7f4cc0c00640 1 -- 172.21.15.16:0/4112716075 shutdown_connections 2024-03-31T15:11:14.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.660+0000 7f4cc0c00640 1 --2- 172.21.15.16:0/4112716075 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f4c980683b0 0x7f4c9806a870 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:14.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.660+0000 7f4cc0c00640 1 --2- 172.21.15.16:0/4112716075 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4cbc0d0a60 0x7f4cbc13ebf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:14.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.660+0000 7f4cc0c00640 1 --2- 172.21.15.16:0/4112716075 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f4cbc0cef80 0x7f4cbc13e6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:14.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.660+0000 7f4cc0c00640 1 --2- 172.21.15.16:0/4112716075 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4cbc06f920 0x7f4cbc15e260 unknown :-1 s=CLOSED pgs=1217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:14.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.660+0000 7f4cc0c00640 1 -- 172.21.15.16:0/4112716075 >> 172.21.15.16:0/4112716075 conn(0x7f4cbc067930 msgr2=0x7f4cbc0d1720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:11:14.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.660+0000 7f4cc0c00640 1 -- 172.21.15.16:0/4112716075 shutdown_connections 2024-03-31T15:11:14.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.660+0000 7f4cc0c00640 1 -- 172.21.15.16:0/4112716075 wait complete. 2024-03-31T15:11:14.672 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-03-31T15:11:14.672 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-31T15:11:14.681 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-31T15:11:14.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.868+0000 7efe2f400640 1 Processor -- start 2024-03-31T15:11:14.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.868+0000 7efe2f400640 1 -- start start 2024-03-31T15:11:14.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.868+0000 7efe2f400640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7efe2806f920 0x7efe2806fd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:14.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.868+0000 7efe2f400640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7efe280d2010 0x7efe280d24a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:14.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.868+0000 7efe2f400640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efe280d0920 con 0x7efe28145e40 2024-03-31T15:11:14.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.868+0000 7efe2f400640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7efe280702c0 con 0x7efe2806f920 2024-03-31T15:11:14.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.868+0000 7efe2f400640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7efe280d06d0 con 0x7efe280d2010 2024-03-31T15:11:14.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.872+0000 7efe27e00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7efe2806f920 0x7efe2806fd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:14.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.872+0000 7efe2d400640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7efe280d2010 0x7efe280d24a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:14.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.872+0000 7efe2ca00640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7efe28145e40 0x7efe280cc740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.16:40942/0 (socket says 172.21.15.16:40942) 2024-03-31T15:11:14.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.872+0000 7efe2ca00640 1 -- 172.21.15.16:0/2573782133 learned_addr learned my addr 172.21.15.16:0/2573782133 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:11:14.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.872+0000 7efe27400640 1 -- 172.21.15.16:0/2573782133 <== mon.1 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 515448413 0 0) 0x7efe280d0920 con 0x7efe28145e40 2024-03-31T15:11:14.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.872+0000 7efe27400640 1 -- 172.21.15.16:0/2573782133 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efe10003530 con 0x7efe28145e40 2024-03-31T15:11:14.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.872+0000 7efe27e00640 1 -- 172.21.15.16:0/2573782133 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7efe280d2010 msgr2=0x7efe280d24a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:11:14.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.872+0000 7efe27e00640 1 --2- 172.21.15.16:0/2573782133 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7efe280d2010 0x7efe280d24a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:14.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.872+0000 7efe27e00640 1 -- 172.21.15.16:0/2573782133 >> v1:172.21.15.89:6789/0 conn(0x7efe28145e40 legacy=0x7efe280cc740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:11:14.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.872+0000 7efe27e00640 1 -- 172.21.15.16:0/2573782133 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efe280d0c00 con 0x7efe2806f920 2024-03-31T15:11:14.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.872+0000 7efe27e00640 1 --2- 172.21.15.16:0/2573782133 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efe2806f920 0x7efe2806fd80 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7efe140097c0 tx=0x7efe1401c4a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f7be253fea0a54c2 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:11:14.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.872+0000 7efe27400640 1 -- 172.21.15.16:0/2573782133 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe140b2070 con 0x7efe2806f920 2024-03-31T15:11:14.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.876+0000 7efe27400640 1 -- 172.21.15.16:0/2573782133 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efe140ad640 con 0x7efe2806f920 2024-03-31T15:11:14.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.876+0000 7efe27400640 1 -- 172.21.15.16:0/2573782133 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe140ada10 con 0x7efe2806f920 2024-03-31T15:11:14.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.876+0000 7efe2f400640 1 -- 172.21.15.16:0/2573782133 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efe2806f920 msgr2=0x7efe2806fd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:11:14.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.876+0000 7efe2f400640 1 --2- 172.21.15.16:0/2573782133 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efe2806f920 0x7efe2806fd80 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7efe140097c0 tx=0x7efe1401c4a0 comp rx=0 tx=0).stop 2024-03-31T15:11:14.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.876+0000 7efe2f400640 1 -- 172.21.15.16:0/2573782133 shutdown_connections 2024-03-31T15:11:14.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.876+0000 7efe2f400640 1 --2- 172.21.15.16:0/2573782133 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efe2806f920 0x7efe2806fd80 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:14.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.876+0000 7efe2f400640 1 --2- 172.21.15.16:0/2573782133 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7efe280d2010 0x7efe280d24a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:14.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.876+0000 7efe2f400640 1 -- 172.21.15.16:0/2573782133 >> 172.21.15.16:0/2573782133 conn(0x7efe28067330 msgr2=0x7efe28067740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:11:14.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.876+0000 7efe2f400640 1 -- 172.21.15.16:0/2573782133 shutdown_connections 2024-03-31T15:11:14.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.876+0000 7efe2f400640 1 -- 172.21.15.16:0/2573782133 wait complete. 2024-03-31T15:11:14.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.876+0000 7efe2f400640 1 Processor -- start 2024-03-31T15:11:14.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.876+0000 7efe2f400640 1 -- start start 2024-03-31T15:11:14.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.876+0000 7efe2f400640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7efe2806f920 0x7efe2815bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:14.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.876+0000 7efe2f400640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efe280d2010 0x7efe2815e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:14.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.876+0000 7efe2f400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efe28145e40 0x7efe28157b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:14.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.876+0000 7efe2f400640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7efe280d0ee0 con 0x7efe28145e40 2024-03-31T15:11:14.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.876+0000 7efe2f400640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7efe280ce9e0 con 0x7efe280d2010 2024-03-31T15:11:14.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.876+0000 7efe2f400640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7efe280d2df0 con 0x7efe2806f920 2024-03-31T15:11:14.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.876+0000 7efe2d400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efe28145e40 0x7efe28157b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:14.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.876+0000 7efe2d400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efe28145e40 0x7efe28157b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37968/0 (socket says 172.21.15.16:37968) 2024-03-31T15:11:14.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.876+0000 7efe2d400640 1 -- 172.21.15.16:0/1079340041 learned_addr learned my addr 172.21.15.16:0/1079340041 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:11:14.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.876+0000 7efe2ca00640 1 --2- 172.21.15.16:0/1079340041 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7efe2806f920 0x7efe2815bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:14.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.876+0000 7efe27e00640 1 --2- 172.21.15.16:0/1079340041 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efe280d2010 0x7efe2815e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:14.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.876+0000 7efe2d400640 1 -- 172.21.15.16:0/1079340041 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7efe2806f920 msgr2=0x7efe2815bfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:11:14.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.880+0000 7efe2d400640 1 --2- 172.21.15.16:0/1079340041 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7efe2806f920 0x7efe2815bfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:14.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.880+0000 7efe2d400640 1 -- 172.21.15.16:0/1079340041 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efe280d2010 msgr2=0x7efe2815e350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:11:14.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.880+0000 7efe2d400640 1 --2- 172.21.15.16:0/1079340041 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efe280d2010 0x7efe2815e350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:14.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.880+0000 7efe2d400640 1 -- 172.21.15.16:0/1079340041 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efe280b74c0 con 0x7efe28145e40 2024-03-31T15:11:14.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.880+0000 7efe2d400640 1 --2- 172.21.15.16:0/1079340041 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efe28145e40 0x7efe28157b90 secure :-1 s=READY pgs=1222 cs=0 l=1 rev1=1 crypto rx=0x7efe2000efc0 tx=0x7efe2000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:11:14.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.880+0000 7efe24c00640 1 -- 172.21.15.16:0/1079340041 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe20018020 con 0x7efe28145e40 2024-03-31T15:11:14.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.880+0000 7efe24c00640 1 -- 172.21.15.16:0/1079340041 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efe20014480 con 0x7efe28145e40 2024-03-31T15:11:14.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.880+0000 7efe24c00640 1 -- 172.21.15.16:0/1079340041 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe20014870 con 0x7efe28145e40 2024-03-31T15:11:14.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.880+0000 7efe2f400640 1 -- 172.21.15.16:0/1079340041 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efe28159f20 con 0x7efe28145e40 2024-03-31T15:11:14.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.880+0000 7efe2f400640 1 -- 172.21.15.16:0/1079340041 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efe280b5440 con 0x7efe28145e40 2024-03-31T15:11:14.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.880+0000 7efe2f400640 1 -- 172.21.15.16:0/1079340041 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efe28159ca0 con 0x7efe28145e40 2024-03-31T15:11:14.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.880+0000 7efe24c00640 1 -- 172.21.15.16:0/1079340041 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7efe2001b020 con 0x7efe28145e40 2024-03-31T15:11:14.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.880+0000 7efe24c00640 1 --2- 172.21.15.16:0/1079340041 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7efe08068380 0x7efe0806a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:14.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.880+0000 7efe24c00640 1 -- 172.21.15.16:0/1079340041 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 5824+0+0 (secure 0 0 0) 0x7efe20087420 con 0x7efe28145e40 2024-03-31T15:11:14.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.884+0000 7efe2ca00640 1 --2- 172.21.15.16:0/1079340041 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7efe08068380 0x7efe0806a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:14.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.884+0000 7efe2ca00640 1 --2- 172.21.15.16:0/1079340041 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7efe08068380 0x7efe0806a840 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7efe1c007fd0 tx=0x7efe1c009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:11:14.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:14.884+0000 7efe24c00640 1 -- 172.21.15.16:0/1079340041 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7efe28159ca0 con 0x7efe28145e40 2024-03-31T15:11:15.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:15.052+0000 7efe2f400640 1 -- 172.21.15.16:0/1079340041 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7efe2815c8f0 con 0x7efe28145e40 2024-03-31T15:11:15.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:15.052+0000 7efe24c00640 1 -- 172.21.15.16:0/1079340041 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v84) ==== 74+0+16766 (secure 0 0 0) 0x7efe20022070 con 0x7efe28145e40 2024-03-31T15:11:15.045 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:11:15.045 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":84,"fsid":"286eb5da-fb47-4550-b290-d9519c13b9a5","created":"2024-03-31T15:03:11.438952+0000","modified":"2024-03-31T15:10:18.450885+0000","last_up_change":"2024-03-31T15:03:22.885535+0000","last_in_change":"2024-03-31T15:03:13.941667+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-31T15:03:22.550787+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-03-31T15:09:49.322620+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-03-31T15:09:49.877778+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"92a66450-ef19-40d8-ac3d-d760b23b0bda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6801","nonce":932712244}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6803","nonce":932712244}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6807","nonce":932712244}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6805","nonce":932712244}]},"public_addr":"172.21.15.16:6801/932712244","cluster_addr":"172.21.15.16:6803/932712244","heartbeat_back_addr":"172.21.15.16:6807/932712244","heartbeat_front_addr":"172.21.15.16:6805/932712244","state":["exists","up"]},{"osd":1,"uuid":"da9d1f2b-1968-4959-b843-faa231988e92","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6810","nonce":1398737533}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6811","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6813","nonce":1398737533}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6821","nonce":1398737533}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6815","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6816","nonce":1398737533}]},"public_addr":"172.21.15.16:6810/1398737533","cluster_addr":"172.21.15.16:6813/1398737533","heartbeat_back_addr":"172.21.15.16:6821/1398737533","heartbeat_front_addr":"172.21.15.16:6816/1398737533","state":["exists","up"]},{"osd":2,"uuid":"01a73662-9a21-4b1f-bcbb-42b9636a4638","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6819","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6822","nonce":3018526646}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6825","nonce":3018526646}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6831","nonce":3018526646}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6827","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6829","nonce":3018526646}]},"public_addr":"172.21.15.16:6822/3018526646","cluster_addr":"172.21.15.16:6825/3018526646","heartbeat_back_addr":"172.21.15.16:6831/3018526646","heartbeat_front_addr":"172.21.15.16:6829/3018526646","state":["exists","up"]},{"osd":3,"uuid":"c7771fd3-e18e-4500-ac38-23cf10ebcd75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6809","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6812","nonce":3360352974}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6817","nonce":3360352974}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6828","nonce":3360352974}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6823","nonce":3360352974}]},"public_addr":"172.21.15.16:6812/3360352974","cluster_addr":"172.21.15.16:6817/3360352974","heartbeat_back_addr":"172.21.15.16:6828/3360352974","heartbeat_front_addr":"172.21.15.16:6823/3360352974","state":["exists","up"]},{"osd":4,"uuid":"88260730-1482-4fbd-a40f-cb1124e9df4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6801","nonce":1856844856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6803","nonce":1856844856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6807","nonce":1856844856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6805","nonce":1856844856}]},"public_addr":"172.21.15.89:6801/1856844856","cluster_addr":"172.21.15.89:6803/1856844856","heartbeat_back_addr":"172.21.15.89:6807/1856844856","heartbeat_front_addr":"172.21.15.89:6805/1856844856","state":["exists","up"]},{"osd":5,"uuid":"4cb2c00c-fae0-4dd8-9d27-ec4f0093ccf7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6814","nonce":50395527}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6817","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6820","nonce":50395527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6831","nonce":50395527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6823","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6826","nonce":50395527}]},"public_addr":"172.21.15.89:6814/50395527","cluster_addr":"172.21.15.89:6820/50395527","heartbeat_back_addr":"172.21.15.89:6831/50395527","heartbeat_front_addr":"172.21.15.89:6826/50395527","state":["exists","up"]},{"osd":6,"uuid":"81ce0c19-2554-4cae-acbe-66a05e9e9b9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6809","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6812","nonce":3470471752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6815","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6818","nonce":3470471752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6827","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6829","nonce":3470471752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6821","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6824","nonce":3470471752}]},"public_addr":"172.21.15.89:6812/3470471752","cluster_addr":"172.21.15.89:6818/3470471752","heartbeat_back_addr":"172.21.15.89:6829/3470471752","heartbeat_front_addr":"172.21.15.89:6824/3470471752","state":["exists","up"]},{"osd":7,"uuid":"85e4313b-d37d-442e-8042-153bf20a847d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6811","nonce":1796778630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6813","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6816","nonce":1796778630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6825","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6828","nonce":1796778630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6819","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6822","nonce":1796778630}]},"public_addr":"172.21.15.89:6811/1796778630","cluster_addr":"172.21.15.89:6816/1796778630","heartbeat_back_addr":"172.21.15.89:6828/1796778630","heartbeat_front_addr":"172.21.15.89:6822/1796778630","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.439088+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.529970+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.680512+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:21.044161+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.582111+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.536558+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.409522+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.590229+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"13.d","mappings":[{"from":1,"to":3}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:6834/2478530200":"2024-04-01T15:10:04.360772+0000","172.21.15.16:6832/2478530200":"2024-04-01T15:10:04.360772+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-31T15:11:15.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:15.056+0000 7efe2f400640 1 -- 172.21.15.16:0/1079340041 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7efe08068380 msgr2=0x7efe0806a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:11:15.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:15.056+0000 7efe2f400640 1 --2- 172.21.15.16:0/1079340041 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7efe08068380 0x7efe0806a840 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7efe1c007fd0 tx=0x7efe1c009040 comp rx=0 tx=0).stop 2024-03-31T15:11:15.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:15.056+0000 7efe2f400640 1 -- 172.21.15.16:0/1079340041 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efe28145e40 msgr2=0x7efe28157b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:11:15.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:15.056+0000 7efe2f400640 1 --2- 172.21.15.16:0/1079340041 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efe28145e40 0x7efe28157b90 secure :-1 s=READY pgs=1222 cs=0 l=1 rev1=1 crypto rx=0x7efe2000efc0 tx=0x7efe2000c490 comp rx=0 tx=0).stop 2024-03-31T15:11:15.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:15.056+0000 7efe2f400640 1 -- 172.21.15.16:0/1079340041 shutdown_connections 2024-03-31T15:11:15.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:15.056+0000 7efe2f400640 1 --2- 172.21.15.16:0/1079340041 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7efe08068380 0x7efe0806a840 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:15.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:15.056+0000 7efe2f400640 1 --2- 172.21.15.16:0/1079340041 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efe28145e40 0x7efe28157b90 unknown :-1 s=CLOSED pgs=1222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:15.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:15.056+0000 7efe2f400640 1 --2- 172.21.15.16:0/1079340041 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efe280d2010 0x7efe2815e350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:15.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:15.056+0000 7efe2f400640 1 --2- 172.21.15.16:0/1079340041 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7efe2806f920 0x7efe2815bfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:15.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:15.056+0000 7efe2f400640 1 -- 172.21.15.16:0/1079340041 >> 172.21.15.16:0/1079340041 conn(0x7efe28067330 msgr2=0x7efe28159190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:11:15.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:15.056+0000 7efe2f400640 1 -- 172.21.15.16:0/1079340041 shutdown_connections 2024-03-31T15:11:15.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:15.056+0000 7efe2f400640 1 -- 172.21.15.16:0/1079340041 wait complete. 2024-03-31T15:11:15.067 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-31T15:03:22.550787+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 12, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-31T15:09:49.322620+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '80', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 13, 'pool_name': 'cephfs_data', 'create_time': '2024-03-31T15:09:49.877778+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '81', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-31T15:11:15.067 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-31T15:11:15.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:15.220+0000 7f6a65e00640 1 Processor -- start 2024-03-31T15:11:15.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:15.220+0000 7f6a65e00640 1 -- start start 2024-03-31T15:11:15.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:15.220+0000 7f6a65e00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f6a600d0a60 0x7f6a600cea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:15.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:15.220+0000 7f6a65e00640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6a6015ca70 con 0x7f6a600cef80 2024-03-31T15:11:15.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:15.220+0000 7f6a65e00640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6a6015cd50 con 0x7f6a6006f920 2024-03-31T15:11:15.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:15.220+0000 7f6a65e00640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f6a6013f670 con 0x7f6a600d0a60 2024-03-31T15:11:15.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:15.220+0000 7f6a56000640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f6a600d0a60 0x7f6a600cea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:15.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:15.220+0000 7f6a5f400640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f6a600cef80 0x7f6a601419c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:51540/0 (socket says 172.21.15.16:51540) 2024-03-31T15:11:15.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:15.220+0000 7f6a5f400640 1 -- 172.21.15.16:0/1381040422 learned_addr learned my addr 172.21.15.16:0/1381040422 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:11:15.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:15.220+0000 7f6a5e000640 1 -- 172.21.15.16:0/1381040422 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1630104157 0 0) 0x7f6a6015ca70 con 0x7f6a600cef80 2024-03-31T15:11:15.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:15.220+0000 7f6a5e000640 1 -- 172.21.15.16:0/1381040422 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6a44003530 con 0x7f6a600cef80 2024-03-31T15:11:15.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:15.220+0000 7f6a56000640 1 -- 172.21.15.16:0/1381040422 >> v1:172.21.15.89:6789/0 conn(0x7f6a6006f920 legacy=0x7f6a6006fd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:11:15.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:15.224+0000 7f6a56000640 1 -- 172.21.15.16:0/1381040422 >> v1:172.21.15.16:6789/0 conn(0x7f6a600cef80 legacy=0x7f6a601419c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:11:15.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:15.224+0000 7f6a56000640 1 -- 172.21.15.16:0/1381040422 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a6015d030 con 0x7f6a600d0a60 2024-03-31T15:11:15.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:15.224+0000 7f6a56000640 1 --2- 172.21.15.16:0/1381040422 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6a600d0a60 0x7f6a600cea40 secure :-1 s=READY pgs=1224 cs=0 l=1 rev1=1 crypto rx=0x7f6a480097c0 tx=0x7f6a4801c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=36bbc560e8486c7b server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:11:15.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:15.224+0000 7f6a5e000640 1 -- 172.21.15.16:0/1381040422 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a480b0070 con 0x7f6a600d0a60 2024-03-31T15:11:15.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:15.224+0000 7f6a5e000640 1 -- 172.21.15.16:0/1381040422 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6a480ab600 con 0x7f6a600d0a60 2024-03-31T15:11:15.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:15.224+0000 7f6a5e000640 1 -- 172.21.15.16:0/1381040422 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a480ab9b0 con 0x7f6a600d0a60 2024-03-31T15:11:15.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:15.224+0000 7f6a65e00640 1 -- 172.21.15.16:0/1381040422 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6a600d0a60 msgr2=0x7f6a600cea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:11:15.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:15.224+0000 7f6a65e00640 1 --2- 172.21.15.16:0/1381040422 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6a600d0a60 0x7f6a600cea40 secure :-1 s=READY pgs=1224 cs=0 l=1 rev1=1 crypto rx=0x7f6a480097c0 tx=0x7f6a4801c4a0 comp rx=0 tx=0).stop 2024-03-31T15:11:15.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:15.224+0000 7f6a65e00640 1 -- 172.21.15.16:0/1381040422 shutdown_connections 2024-03-31T15:11:15.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:15.224+0000 7f6a65e00640 1 --2- 172.21.15.16:0/1381040422 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6a600d0a60 0x7f6a600cea40 unknown :-1 s=CLOSED pgs=1224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:15.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:15.224+0000 7f6a65e00640 1 -- 172.21.15.16:0/1381040422 >> 172.21.15.16:0/1381040422 conn(0x7f6a60067930 msgr2=0x7f6a600b3ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:11:15.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:15.224+0000 7f6a65e00640 1 -- 172.21.15.16:0/1381040422 shutdown_connections 2024-03-31T15:11:15.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:15.224+0000 7f6a65e00640 1 -- 172.21.15.16:0/1381040422 wait complete. 2024-03-31T15:11:15.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:15.228+0000 7f6a65e00640 1 Processor -- start 2024-03-31T15:11:15.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:15.228+0000 7f6a65e00640 1 -- start start 2024-03-31T15:11:15.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:15.228+0000 7f6a65e00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f6a6006f920 0x7f6a6013ebb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:15.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:15.228+0000 7f6a65e00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6a600cef80 0x7f6a6013fcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:15.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:15.228+0000 7f6a65e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6a600d0a60 0x7f6a60140210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:15.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:15.228+0000 7f6a65e00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f6a6016a240 con 0x7f6a600d0a60 2024-03-31T15:11:15.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:15.228+0000 7f6a65e00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f6a6016a0c0 con 0x7f6a600cef80 2024-03-31T15:11:15.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:15.228+0000 7f6a65e00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f6a6016a3c0 con 0x7f6a6006f920 2024-03-31T15:11:15.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:15.228+0000 7f6a5f400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6a600d0a60 0x7f6a60140210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:15.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:15.228+0000 7f6a5f400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6a600d0a60 0x7f6a60140210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37988/0 (socket says 172.21.15.16:37988) 2024-03-31T15:11:15.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:15.228+0000 7f6a5f400640 1 -- 172.21.15.16:0/1590077221 learned_addr learned my addr 172.21.15.16:0/1590077221 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:11:15.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:15.228+0000 7f6a56000640 1 --2- 172.21.15.16:0/1590077221 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6a600cef80 0x7f6a6013fcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:15.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:15.228+0000 7f6a5ea00640 1 --2- 172.21.15.16:0/1590077221 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f6a6006f920 0x7f6a6013ebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:15.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:15.228+0000 7f6a5f400640 1 -- 172.21.15.16:0/1590077221 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f6a6006f920 msgr2=0x7f6a6013ebb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:11:15.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:15.228+0000 7f6a5f400640 1 --2- 172.21.15.16:0/1590077221 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f6a6006f920 0x7f6a6013ebb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:15.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:15.228+0000 7f6a5f400640 1 -- 172.21.15.16:0/1590077221 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6a600cef80 msgr2=0x7f6a6013fcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:11:15.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:15.228+0000 7f6a5f400640 1 --2- 172.21.15.16:0/1590077221 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6a600cef80 0x7f6a6013fcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:15.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:15.228+0000 7f6a5f400640 1 -- 172.21.15.16:0/1590077221 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a600b1ed0 con 0x7f6a600d0a60 2024-03-31T15:11:15.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:15.228+0000 7f6a56000640 1 --2- 172.21.15.16:0/1590077221 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6a600cef80 0x7f6a6013fcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:11:15.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:15.228+0000 7f6a5f400640 1 --2- 172.21.15.16:0/1590077221 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6a600d0a60 0x7f6a60140210 secure :-1 s=READY pgs=1225 cs=0 l=1 rev1=1 crypto rx=0x7f6a5800cb40 tx=0x7f6a5800a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:11:15.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:15.228+0000 7f6a57e00640 1 -- 172.21.15.16:0/1590077221 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a58017070 con 0x7f6a600d0a60 2024-03-31T15:11:15.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:15.228+0000 7f6a57e00640 1 -- 172.21.15.16:0/1590077221 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6a5800aca0 con 0x7f6a600d0a60 2024-03-31T15:11:15.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:15.228+0000 7f6a57e00640 1 -- 172.21.15.16:0/1590077221 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a58012620 con 0x7f6a600d0a60 2024-03-31T15:11:15.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:15.228+0000 7f6a65e00640 1 -- 172.21.15.16:0/1590077221 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6a600b21b0 con 0x7f6a600d0a60 2024-03-31T15:11:15.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:15.232+0000 7f6a65e00640 1 -- 172.21.15.16:0/1590077221 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6a600b2770 con 0x7f6a600d0a60 2024-03-31T15:11:15.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:15.232+0000 7f6a65e00640 1 -- 172.21.15.16:0/1590077221 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6a60069950 con 0x7f6a600d0a60 2024-03-31T15:11:15.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:15.232+0000 7f6a57e00640 1 -- 172.21.15.16:0/1590077221 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f6a5801a070 con 0x7f6a600d0a60 2024-03-31T15:11:15.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:15.232+0000 7f6a57e00640 1 --2- 172.21.15.16:0/1590077221 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f6a3c0683b0 0x7f6a3c06a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:15.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:15.232+0000 7f6a57e00640 1 -- 172.21.15.16:0/1590077221 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 5824+0+0 (secure 0 0 0) 0x7f6a58086850 con 0x7f6a600d0a60 2024-03-31T15:11:15.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:15.232+0000 7f6a5ea00640 1 --2- 172.21.15.16:0/1590077221 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f6a3c0683b0 0x7f6a3c06a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:15.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:15.236+0000 7f6a5ea00640 1 --2- 172.21.15.16:0/1590077221 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f6a3c0683b0 0x7f6a3c06a870 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f6a50000f70 tx=0x7f6a50009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:11:15.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:15.236+0000 7f6a57e00640 1 -- 172.21.15.16:0/1590077221 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f6a58056db0 con 0x7f6a600d0a60 2024-03-31T15:11:15.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:15.399+0000 7f6a65e00640 1 -- 172.21.15.16:0/1590077221 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f6a600b50e0 con 0x7f6a600d0a60 2024-03-31T15:11:15.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:15.399+0000 7f6a57e00640 1 -- 172.21.15.16:0/1590077221 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v84) ==== 93+0+10 (secure 0 0 0) 0x7f6a580566b0 con 0x7f6a600d0a60 2024-03-31T15:11:15.394 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2024-03-31T15:11:15.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:15.403+0000 7f6a65e00640 1 -- 172.21.15.16:0/1590077221 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f6a3c0683b0 msgr2=0x7f6a3c06a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:11:15.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:15.403+0000 7f6a65e00640 1 --2- 172.21.15.16:0/1590077221 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f6a3c0683b0 0x7f6a3c06a870 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f6a50000f70 tx=0x7f6a50009040 comp rx=0 tx=0).stop 2024-03-31T15:11:15.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:15.403+0000 7f6a65e00640 1 -- 172.21.15.16:0/1590077221 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6a600d0a60 msgr2=0x7f6a60140210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:11:15.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:15.403+0000 7f6a65e00640 1 --2- 172.21.15.16:0/1590077221 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6a600d0a60 0x7f6a60140210 secure :-1 s=READY pgs=1225 cs=0 l=1 rev1=1 crypto rx=0x7f6a5800cb40 tx=0x7f6a5800a060 comp rx=0 tx=0).stop 2024-03-31T15:11:15.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:15.403+0000 7f6a65e00640 1 -- 172.21.15.16:0/1590077221 shutdown_connections 2024-03-31T15:11:15.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:15.403+0000 7f6a65e00640 1 --2- 172.21.15.16:0/1590077221 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f6a3c0683b0 0x7f6a3c06a870 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:15.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:15.403+0000 7f6a65e00640 1 --2- 172.21.15.16:0/1590077221 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6a600d0a60 0x7f6a60140210 unknown :-1 s=CLOSED pgs=1225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:15.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:15.403+0000 7f6a65e00640 1 --2- 172.21.15.16:0/1590077221 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6a600cef80 0x7f6a6013fcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:15.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:15.403+0000 7f6a65e00640 1 --2- 172.21.15.16:0/1590077221 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f6a6006f920 0x7f6a6013ebb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:15.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:15.403+0000 7f6a65e00640 1 -- 172.21.15.16:0/1590077221 >> 172.21.15.16:0/1590077221 conn(0x7f6a60067930 msgr2=0x7f6a6006c730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:11:15.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:15.403+0000 7f6a65e00640 1 -- 172.21.15.16:0/1590077221 shutdown_connections 2024-03-31T15:11:15.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:15.403+0000 7f6a65e00640 1 -- 172.21.15.16:0/1590077221 wait complete. 2024-03-31T15:11:15.415 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-31T15:11:15.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:15.567+0000 7efedd000640 1 Processor -- start 2024-03-31T15:11:15.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:15.567+0000 7efedd000640 1 -- start start 2024-03-31T15:11:15.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:15.567+0000 7efedd000640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7efed81411b0 0x7efed806f920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:15.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:15.567+0000 7efedd000640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7efed8141590 0x7efed806fe60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:15.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:15.567+0000 7efedd000640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efed8144980 con 0x7efed8143c30 2024-03-31T15:11:15.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:15.567+0000 7efedd000640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7efed815f910 con 0x7efed81411b0 2024-03-31T15:11:15.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:15.567+0000 7efedd000640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7efed8162910 con 0x7efed8141590 2024-03-31T15:11:15.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:15.567+0000 7efed5e00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7efed81411b0 0x7efed806f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:15.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:15.567+0000 7efed5e00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7efed81411b0 0x7efed806f920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37994/0 (socket says 172.21.15.16:37994) 2024-03-31T15:11:15.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:15.567+0000 7efed5e00640 1 -- 172.21.15.16:0/1319321314 learned_addr learned my addr 172.21.15.16:0/1319321314 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:11:15.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:15.567+0000 7efed6800640 1 --1- 172.21.15.16:0/1319321314 >> v1:172.21.15.16:6789/0 conn(0x7efed8143c30 0x7efed81440a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:51546/0 (socket says 172.21.15.16:51546) 2024-03-31T15:11:15.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:15.567+0000 7efed5e00640 1 -- 172.21.15.16:0/1319321314 >> v2:172.21.15.89:3300/0 conn(0x7efed8141590 msgr2=0x7efed806fe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:11:15.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:15.567+0000 7efecd400640 1 --2- 172.21.15.16:0/1319321314 >> v2:172.21.15.89:3300/0 conn(0x7efed8141590 0x7efed806fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:15.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:15.571+0000 7efed5e00640 1 --2- 172.21.15.16:0/1319321314 >> v2:172.21.15.89:3300/0 conn(0x7efed8141590 0x7efed806fe60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:15.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:15.567+0000 7efed5400640 1 -- 172.21.15.16:0/1319321314 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3338864380 0 0) 0x7efed8144980 con 0x7efed8143c30 2024-03-31T15:11:15.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:15.571+0000 7efed5e00640 1 -- 172.21.15.16:0/1319321314 >> v1:172.21.15.16:6789/0 conn(0x7efed8143c30 legacy=0x7efed81440a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:11:15.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:15.571+0000 7efed5e00640 1 -- 172.21.15.16:0/1319321314 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efed8161fe0 con 0x7efed81411b0 2024-03-31T15:11:15.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:15.571+0000 7efecd400640 1 --2- 172.21.15.16:0/1319321314 >> v2:172.21.15.89:3300/0 conn(0x7efed8141590 0x7efed806fe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:11:15.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:15.571+0000 7efed5e00640 1 --2- 172.21.15.16:0/1319321314 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efed81411b0 0x7efed806f920 secure :-1 s=READY pgs=1227 cs=0 l=1 rev1=1 crypto rx=0x7efed8158fe0 tx=0x7efec801c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c7f89ba80589e4b server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:11:15.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:15.571+0000 7efed5400640 1 -- 172.21.15.16:0/1319321314 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efec80ab610 con 0x7efed81411b0 2024-03-31T15:11:15.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:15.571+0000 7efed5400640 1 -- 172.21.15.16:0/1319321314 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efec80b0070 con 0x7efed81411b0 2024-03-31T15:11:15.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:15.571+0000 7efed5400640 1 -- 172.21.15.16:0/1319321314 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efec80ab9d0 con 0x7efed81411b0 2024-03-31T15:11:15.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:15.571+0000 7efedd000640 1 -- 172.21.15.16:0/1319321314 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efed81411b0 msgr2=0x7efed806f920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:11:15.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:15.571+0000 7efedd000640 1 --2- 172.21.15.16:0/1319321314 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efed81411b0 0x7efed806f920 secure :-1 s=READY pgs=1227 cs=0 l=1 rev1=1 crypto rx=0x7efed8158fe0 tx=0x7efec801c520 comp rx=0 tx=0).stop 2024-03-31T15:11:15.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:15.571+0000 7efedd000640 1 -- 172.21.15.16:0/1319321314 shutdown_connections 2024-03-31T15:11:15.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:15.571+0000 7efedd000640 1 --2- 172.21.15.16:0/1319321314 >> v2:172.21.15.89:3300/0 conn(0x7efed8141590 0x7efed806fe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:15.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:15.571+0000 7efedd000640 1 --2- 172.21.15.16:0/1319321314 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efed81411b0 0x7efed806f920 unknown :-1 s=CLOSED pgs=1227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:15.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:15.571+0000 7efedd000640 1 -- 172.21.15.16:0/1319321314 >> 172.21.15.16:0/1319321314 conn(0x7efed8067930 msgr2=0x7efed80b3ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:11:15.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:15.571+0000 7efedd000640 1 -- 172.21.15.16:0/1319321314 shutdown_connections 2024-03-31T15:11:15.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:15.571+0000 7efedd000640 1 -- 172.21.15.16:0/1319321314 wait complete. 2024-03-31T15:11:15.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:15.575+0000 7efedd000640 1 Processor -- start 2024-03-31T15:11:15.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:15.575+0000 7efedd000640 1 -- start start 2024-03-31T15:11:15.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:15.575+0000 7efedd000640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7efed81411b0 0x7efed813af50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:15.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:15.575+0000 7efedd000640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efed8141590 0x7efed813e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:15.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:15.575+0000 7efedd000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efed8143c30 0x7efed813e650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:15.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:15.575+0000 7efedd000640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7efed815f290 con 0x7efed8143c30 2024-03-31T15:11:15.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:15.575+0000 7efedd000640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7efed815f110 con 0x7efed8141590 2024-03-31T15:11:15.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:15.575+0000 7efedd000640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7efed815f410 con 0x7efed81411b0 2024-03-31T15:11:15.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:15.575+0000 7efed6800640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efed8143c30 0x7efed813e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:15.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:15.575+0000 7efed6800640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efed8143c30 0x7efed813e650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38000/0 (socket says 172.21.15.16:38000) 2024-03-31T15:11:15.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:15.575+0000 7efed6800640 1 -- 172.21.15.16:0/498401014 learned_addr learned my addr 172.21.15.16:0/498401014 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:11:15.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:15.575+0000 7efed5e00640 1 --2- 172.21.15.16:0/498401014 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7efed81411b0 0x7efed813af50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:15.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:15.575+0000 7efecd400640 1 --2- 172.21.15.16:0/498401014 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efed8141590 0x7efed813e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:15.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:15.575+0000 7efed6800640 1 -- 172.21.15.16:0/498401014 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7efed81411b0 msgr2=0x7efed813af50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:11:15.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:15.575+0000 7efed6800640 1 --2- 172.21.15.16:0/498401014 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7efed81411b0 0x7efed813af50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:15.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:15.575+0000 7efed6800640 1 -- 172.21.15.16:0/498401014 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efed8141590 msgr2=0x7efed813e110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:11:15.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:15.575+0000 7efed6800640 1 --2- 172.21.15.16:0/498401014 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efed8141590 0x7efed813e110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:15.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:15.575+0000 7efed6800640 1 -- 172.21.15.16:0/498401014 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efed813b490 con 0x7efed8143c30 2024-03-31T15:11:15.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:15.575+0000 7efed6800640 1 --2- 172.21.15.16:0/498401014 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efed8143c30 0x7efed813e650 secure :-1 s=READY pgs=1228 cs=0 l=1 rev1=1 crypto rx=0x7efed000c730 tx=0x7efed000cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:11:15.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:15.575+0000 7efecea00640 1 -- 172.21.15.16:0/498401014 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efed0018070 con 0x7efed8143c30 2024-03-31T15:11:15.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:15.579+0000 7efecea00640 1 -- 172.21.15.16:0/498401014 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efed000ce40 con 0x7efed8143c30 2024-03-31T15:11:15.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:15.579+0000 7efecea00640 1 -- 172.21.15.16:0/498401014 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efed000a860 con 0x7efed8143c30 2024-03-31T15:11:15.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:15.579+0000 7efedd000640 1 -- 172.21.15.16:0/498401014 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efed806d2e0 con 0x7efed8143c30 2024-03-31T15:11:15.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:15.579+0000 7efedd000640 1 -- 172.21.15.16:0/498401014 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efed806d8a0 con 0x7efed8143c30 2024-03-31T15:11:15.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:15.579+0000 7efedd000640 1 -- 172.21.15.16:0/498401014 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efed8069950 con 0x7efed8143c30 2024-03-31T15:11:15.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:15.579+0000 7efecea00640 1 -- 172.21.15.16:0/498401014 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7efed000aa00 con 0x7efed8143c30 2024-03-31T15:11:15.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:15.579+0000 7efecea00640 1 --2- 172.21.15.16:0/498401014 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7efeb4068380 0x7efeb406a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:15.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:15.583+0000 7efecea00640 1 -- 172.21.15.16:0/498401014 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 5824+0+0 (secure 0 0 0) 0x7efed00862d0 con 0x7efed8143c30 2024-03-31T15:11:15.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:15.583+0000 7efed5e00640 1 --2- 172.21.15.16:0/498401014 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7efeb4068380 0x7efeb406a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:15.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:15.583+0000 7efed5e00640 1 --2- 172.21.15.16:0/498401014 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7efeb4068380 0x7efeb406a840 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7efec80af000 tx=0x7efec80a7000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:11:15.577 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:15.583+0000 7efecea00640 1 -- 172.21.15.16:0/498401014 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7efed0021020 con 0x7efed8143c30 2024-03-31T15:11:15.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:15.747+0000 7efedd000640 1 -- 172.21.15.16:0/498401014 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7efed80b5de0 con 0x7efed8143c30 2024-03-31T15:11:15.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:15.747+0000 7efecea00640 1 -- 172.21.15.16:0/498401014 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v84) ==== 104+0+11 (secure 0 0 0) 0x7efed0056260 con 0x7efed8143c30 2024-03-31T15:11:15.739 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2024-03-31T15:11:15.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:15.747+0000 7efedd000640 1 -- 172.21.15.16:0/498401014 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7efeb4068380 msgr2=0x7efeb406a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:11:15.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:15.747+0000 7efedd000640 1 --2- 172.21.15.16:0/498401014 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7efeb4068380 0x7efeb406a840 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7efec80af000 tx=0x7efec80a7000 comp rx=0 tx=0).stop 2024-03-31T15:11:15.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:15.747+0000 7efedd000640 1 -- 172.21.15.16:0/498401014 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efed8143c30 msgr2=0x7efed813e650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:11:15.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:15.747+0000 7efedd000640 1 --2- 172.21.15.16:0/498401014 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efed8143c30 0x7efed813e650 secure :-1 s=READY pgs=1228 cs=0 l=1 rev1=1 crypto rx=0x7efed000c730 tx=0x7efed000cbf0 comp rx=0 tx=0).stop 2024-03-31T15:11:15.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:15.747+0000 7efedd000640 1 -- 172.21.15.16:0/498401014 shutdown_connections 2024-03-31T15:11:15.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:15.747+0000 7efedd000640 1 --2- 172.21.15.16:0/498401014 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7efeb4068380 0x7efeb406a840 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:15.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:15.747+0000 7efedd000640 1 --2- 172.21.15.16:0/498401014 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efed8143c30 0x7efed813e650 unknown :-1 s=CLOSED pgs=1228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:15.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:15.747+0000 7efedd000640 1 --2- 172.21.15.16:0/498401014 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efed8141590 0x7efed813e110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:15.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:15.747+0000 7efedd000640 1 --2- 172.21.15.16:0/498401014 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7efed81411b0 0x7efed813af50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:15.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:15.747+0000 7efedd000640 1 -- 172.21.15.16:0/498401014 >> 172.21.15.16:0/498401014 conn(0x7efed8067930 msgr2=0x7efed8142eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:11:15.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:15.747+0000 7efedd000640 1 -- 172.21.15.16:0/498401014 shutdown_connections 2024-03-31T15:11:15.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:15.747+0000 7efedd000640 1 -- 172.21.15.16:0/498401014 wait complete. 2024-03-31T15:11:15.761 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-31T15:11:15.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:15.911+0000 7ff3c4a00640 1 Processor -- start 2024-03-31T15:11:15.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:15.911+0000 7ff3c4a00640 1 -- start start 2024-03-31T15:11:15.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:15.915+0000 7ff3c4a00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff3c015e140 0x7ff3c006b620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:15.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:15.915+0000 7ff3c4a00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7ff3c00ae9a0 0x7ff3c0072480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:15.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:15.915+0000 7ff3c4a00640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff3c015e920 con 0x7ff3c006bb60 2024-03-31T15:11:15.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:15.915+0000 7ff3c4a00640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7ff3c00aee10 con 0x7ff3c015e140 2024-03-31T15:11:15.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:15.915+0000 7ff3c4a00640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7ff3c015ec00 con 0x7ff3c00ae9a0 2024-03-31T15:11:15.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:15.915+0000 7ff3bf400640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff3c015e140 0x7ff3c006b620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:15.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:15.915+0000 7ff3bf400640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff3c015e140 0x7ff3c006b620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38010/0 (socket says 172.21.15.16:38010) 2024-03-31T15:11:15.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:15.915+0000 7ff3bf400640 1 -- 172.21.15.16:0/4278856744 learned_addr learned my addr 172.21.15.16:0/4278856744 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:11:15.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:15.915+0000 7ff3bfe00640 1 --2- 172.21.15.16:0/4278856744 >> v2:172.21.15.89:3300/0 conn(0x7ff3c00ae9a0 0x7ff3c0072480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:15.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:15.915+0000 7ff3bf400640 1 -- 172.21.15.16:0/4278856744 >> v2:172.21.15.89:3300/0 conn(0x7ff3c00ae9a0 msgr2=0x7ff3c0072480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:11:15.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:15.915+0000 7ff3bf400640 1 --2- 172.21.15.16:0/4278856744 >> v2:172.21.15.89:3300/0 conn(0x7ff3c00ae9a0 0x7ff3c0072480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:15.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:15.915+0000 7ff3bf400640 1 -- 172.21.15.16:0/4278856744 >> v1:172.21.15.89:6789/0 conn(0x7ff3c006bb60 legacy=0x7ff3c00ae290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:11:15.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:15.915+0000 7ff3bf400640 1 -- 172.21.15.16:0/4278856744 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff3c015eee0 con 0x7ff3c015e140 2024-03-31T15:11:15.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:15.919+0000 7ff3bf400640 1 --2- 172.21.15.16:0/4278856744 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff3c015e140 0x7ff3c006b620 secure :-1 s=READY pgs=1229 cs=0 l=1 rev1=1 crypto rx=0x7ff3c00b5ca0 tx=0x7ff3b001c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f653b8120b683a02 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:11:15.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:15.919+0000 7ff3bea00640 1 -- 172.21.15.16:0/4278856744 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3b00b0070 con 0x7ff3c015e140 2024-03-31T15:11:15.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:15.919+0000 7ff3bea00640 1 -- 172.21.15.16:0/4278856744 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff3b00ab600 con 0x7ff3c015e140 2024-03-31T15:11:15.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:15.919+0000 7ff3bea00640 1 -- 172.21.15.16:0/4278856744 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3b00ab9d0 con 0x7ff3c015e140 2024-03-31T15:11:15.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:15.919+0000 7ff3c4a00640 1 -- 172.21.15.16:0/4278856744 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff3c015e140 msgr2=0x7ff3c006b620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:11:15.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:15.919+0000 7ff3c4a00640 1 --2- 172.21.15.16:0/4278856744 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff3c015e140 0x7ff3c006b620 secure :-1 s=READY pgs=1229 cs=0 l=1 rev1=1 crypto rx=0x7ff3c00b5ca0 tx=0x7ff3b001c520 comp rx=0 tx=0).stop 2024-03-31T15:11:15.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:15.919+0000 7ff3c4a00640 1 -- 172.21.15.16:0/4278856744 shutdown_connections 2024-03-31T15:11:15.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:15.919+0000 7ff3c4a00640 1 --2- 172.21.15.16:0/4278856744 >> v2:172.21.15.89:3300/0 conn(0x7ff3c00ae9a0 0x7ff3c0072480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:15.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:15.919+0000 7ff3c4a00640 1 --2- 172.21.15.16:0/4278856744 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff3c015e140 0x7ff3c006b620 unknown :-1 s=CLOSED pgs=1229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:15.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:15.919+0000 7ff3c4a00640 1 -- 172.21.15.16:0/4278856744 >> 172.21.15.16:0/4278856744 conn(0x7ff3c00b2b30 msgr2=0x7ff3c00b2f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:11:15.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:15.919+0000 7ff3c4a00640 1 -- 172.21.15.16:0/4278856744 shutdown_connections 2024-03-31T15:11:15.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:15.919+0000 7ff3c4a00640 1 -- 172.21.15.16:0/4278856744 wait complete. 2024-03-31T15:11:15.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:15.923+0000 7ff3c4a00640 1 Processor -- start 2024-03-31T15:11:15.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:15.923+0000 7ff3c4a00640 1 -- start start 2024-03-31T15:11:15.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:15.923+0000 7ff3c4a00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff3c006bb60 0x7ff3c0150210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:15.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:15.923+0000 7ff3c4a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff3c00ae9a0 0x7ff3c00ce6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:15.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:15.923+0000 7ff3c4a00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff3c015e140 0x7ff3c015d5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:15.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:15.923+0000 7ff3c4a00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7ff3c0073350 con 0x7ff3c00ae9a0 2024-03-31T15:11:15.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:15.923+0000 7ff3c4a00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7ff3c00731d0 con 0x7ff3c006bb60 2024-03-31T15:11:15.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:15.923+0000 7ff3c4a00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7ff3c016af90 con 0x7ff3c015e140 2024-03-31T15:11:15.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:15.923+0000 7ff3b6a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff3c00ae9a0 0x7ff3c00ce6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:15.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:15.923+0000 7ff3bf400640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff3c006bb60 0x7ff3c0150210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:15.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:15.923+0000 7ff3b6a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff3c00ae9a0 0x7ff3c00ce6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38022/0 (socket says 172.21.15.16:38022) 2024-03-31T15:11:15.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:15.923+0000 7ff3b6a00640 1 -- 172.21.15.16:0/1308347858 learned_addr learned my addr 172.21.15.16:0/1308347858 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:11:15.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:15.923+0000 7ff3bfe00640 1 --2- 172.21.15.16:0/1308347858 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff3c015e140 0x7ff3c015d5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:15.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:15.923+0000 7ff3b6a00640 1 -- 172.21.15.16:0/1308347858 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff3c015e140 msgr2=0x7ff3c015d5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:11:15.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:15.923+0000 7ff3b6a00640 1 --2- 172.21.15.16:0/1308347858 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff3c015e140 0x7ff3c015d5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:15.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:15.923+0000 7ff3b6a00640 1 -- 172.21.15.16:0/1308347858 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff3c006bb60 msgr2=0x7ff3c0150210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:11:15.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:15.923+0000 7ff3b6a00640 1 --2- 172.21.15.16:0/1308347858 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff3c006bb60 0x7ff3c0150210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:15.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:15.923+0000 7ff3b6a00640 1 -- 172.21.15.16:0/1308347858 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff3c00cf120 con 0x7ff3c00ae9a0 2024-03-31T15:11:15.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:15.923+0000 7ff3bf400640 1 --2- 172.21.15.16:0/1308347858 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff3c006bb60 0x7ff3c0150210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:11:15.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:15.923+0000 7ff3b6a00640 1 --2- 172.21.15.16:0/1308347858 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff3c00ae9a0 0x7ff3c00ce6b0 secure :-1 s=READY pgs=1230 cs=0 l=1 rev1=1 crypto rx=0x7ff3a800c3a0 tx=0x7ff3a800c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:11:15.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:15.923+0000 7ff3b7e00640 1 -- 172.21.15.16:0/1308347858 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3a80052d0 con 0x7ff3c00ae9a0 2024-03-31T15:11:15.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:15.923+0000 7ff3b7e00640 1 -- 172.21.15.16:0/1308347858 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff3a8005470 con 0x7ff3c00ae9a0 2024-03-31T15:11:15.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:15.923+0000 7ff3c4a00640 1 -- 172.21.15.16:0/1308347858 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff3c00cf400 con 0x7ff3c00ae9a0 2024-03-31T15:11:15.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:15.923+0000 7ff3b7e00640 1 -- 172.21.15.16:0/1308347858 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3a8007260 con 0x7ff3c00ae9a0 2024-03-31T15:11:15.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:15.923+0000 7ff3c4a00640 1 -- 172.21.15.16:0/1308347858 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff3c00cf9c0 con 0x7ff3c00ae9a0 2024-03-31T15:11:15.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:15.923+0000 7ff3c4a00640 1 -- 172.21.15.16:0/1308347858 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff3c004f8c0 con 0x7ff3c00ae9a0 2024-03-31T15:11:15.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:15.927+0000 7ff3b7e00640 1 -- 172.21.15.16:0/1308347858 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7ff3a8007400 con 0x7ff3c00ae9a0 2024-03-31T15:11:15.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:15.927+0000 7ff3b7e00640 1 --2- 172.21.15.16:0/1308347858 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7ff39c068380 0x7ff39c06a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:15.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:15.927+0000 7ff3b7e00640 1 -- 172.21.15.16:0/1308347858 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 5824+0+0 (secure 0 0 0) 0x7ff3a8085e30 con 0x7ff3c00ae9a0 2024-03-31T15:11:15.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:15.927+0000 7ff3bf400640 1 --2- 172.21.15.16:0/1308347858 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7ff39c068380 0x7ff39c06a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:15.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:15.927+0000 7ff3bf400640 1 --2- 172.21.15.16:0/1308347858 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7ff39c068380 0x7ff39c06a840 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7ff3b00ad000 tx=0x7ff3b00a6000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:11:15.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:15.935+0000 7ff3b7e00640 1 -- 172.21.15.16:0/1308347858 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7ff3a8055d30 con 0x7ff3c00ae9a0 2024-03-31T15:11:16.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:16.103+0000 7ff3c4a00640 1 -- 172.21.15.16:0/1308347858 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7ff3c0070be0 con 0x7ff3c00ae9a0 2024-03-31T15:11:16.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:16.103+0000 7ff3b7e00640 1 -- 172.21.15.16:0/1308347858 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v84) ==== 100+0+11 (secure 0 0 0) 0x7ff3a805a030 con 0x7ff3c00ae9a0 2024-03-31T15:11:16.095 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 64 2024-03-31T15:11:16.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:16.103+0000 7ff3c4a00640 1 -- 172.21.15.16:0/1308347858 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7ff39c068380 msgr2=0x7ff39c06a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:11:16.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:16.103+0000 7ff3c4a00640 1 --2- 172.21.15.16:0/1308347858 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7ff39c068380 0x7ff39c06a840 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7ff3b00ad000 tx=0x7ff3b00a6000 comp rx=0 tx=0).stop 2024-03-31T15:11:16.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:16.103+0000 7ff3c4a00640 1 -- 172.21.15.16:0/1308347858 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff3c00ae9a0 msgr2=0x7ff3c00ce6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:11:16.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:16.103+0000 7ff3c4a00640 1 --2- 172.21.15.16:0/1308347858 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff3c00ae9a0 0x7ff3c00ce6b0 secure :-1 s=READY pgs=1230 cs=0 l=1 rev1=1 crypto rx=0x7ff3a800c3a0 tx=0x7ff3a800c860 comp rx=0 tx=0).stop 2024-03-31T15:11:16.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:16.103+0000 7ff3c4a00640 1 -- 172.21.15.16:0/1308347858 shutdown_connections 2024-03-31T15:11:16.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:16.103+0000 7ff3c4a00640 1 --2- 172.21.15.16:0/1308347858 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7ff39c068380 0x7ff39c06a840 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:16.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:16.103+0000 7ff3c4a00640 1 --2- 172.21.15.16:0/1308347858 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff3c015e140 0x7ff3c015d5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:16.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:16.103+0000 7ff3c4a00640 1 --2- 172.21.15.16:0/1308347858 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff3c00ae9a0 0x7ff3c00ce6b0 unknown :-1 s=CLOSED pgs=1230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:16.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:16.103+0000 7ff3c4a00640 1 --2- 172.21.15.16:0/1308347858 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff3c006bb60 0x7ff3c0150210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:16.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:16.103+0000 7ff3c4a00640 1 -- 172.21.15.16:0/1308347858 >> 172.21.15.16:0/1308347858 conn(0x7ff3c00b2b30 msgr2=0x7ff3c015f9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:11:16.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:16.103+0000 7ff3c4a00640 1 -- 172.21.15.16:0/1308347858 shutdown_connections 2024-03-31T15:11:16.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:16.103+0000 7ff3c4a00640 1 -- 172.21.15.16:0/1308347858 wait complete. 2024-03-31T15:11:16.126 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-31T15:11:16.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:16.275+0000 7f0093200640 1 Processor -- start 2024-03-31T15:11:16.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:16.275+0000 7f0093200640 1 -- start start 2024-03-31T15:11:16.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:16.275+0000 7f0093200640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f008c09b170 0x7f008c0ccb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:16.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:16.275+0000 7f0093200640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f008c0d0800 con 0x7f008c15e3b0 2024-03-31T15:11:16.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:16.275+0000 7f0093200640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f008c0d0ae0 con 0x7f008c070d50 2024-03-31T15:11:16.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:16.275+0000 7f0093200640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f008c15e790 con 0x7f008c09b170 2024-03-31T15:11:16.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:16.275+0000 7f0091e00640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f008c15e3b0 0x7f008c09aa60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:51558/0 (socket says 172.21.15.16:51558) 2024-03-31T15:11:16.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:16.275+0000 7f0091e00640 1 -- 172.21.15.16:0/3184925541 learned_addr learned my addr 172.21.15.16:0/3184925541 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:11:16.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:16.275+0000 7f0091400640 1 --2- 172.21.15.16:0/3184925541 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f008c09b170 0x7f008c0ccb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:16.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:16.275+0000 7f0091400640 1 -- 172.21.15.16:0/3184925541 >> v1:172.21.15.89:6789/0 conn(0x7f008c070d50 legacy=0x7f008c0cd0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:11:16.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:16.275+0000 7f0091400640 1 -- 172.21.15.16:0/3184925541 >> v1:172.21.15.16:6789/0 conn(0x7f008c15e3b0 legacy=0x7f008c09aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:11:16.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:16.275+0000 7f0091400640 1 -- 172.21.15.16:0/3184925541 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f008c0d0dc0 con 0x7f008c09b170 2024-03-31T15:11:16.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:16.275+0000 7f0090a00640 1 -- 172.21.15.16:0/3184925541 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2414187542 0 0) 0x7f008c0d0800 con 0x7f008c15e3b0 2024-03-31T15:11:16.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:16.279+0000 7f0091400640 1 --2- 172.21.15.16:0/3184925541 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f008c09b170 0x7f008c0ccb90 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f007c0097f0 tx=0x7f007c01c4e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3e3345d5378f3351 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:11:16.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:16.279+0000 7f0090a00640 1 -- 172.21.15.16:0/3184925541 <== mon.2 v2:172.21.15.89:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f007c0ab5f0 con 0x7f008c09b170 2024-03-31T15:11:16.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:16.279+0000 7f0090a00640 1 -- 172.21.15.16:0/3184925541 <== mon.2 v2:172.21.15.89:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f007c0b0070 con 0x7f008c09b170 2024-03-31T15:11:16.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:16.279+0000 7f0090a00640 1 -- 172.21.15.16:0/3184925541 <== mon.2 v2:172.21.15.89:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f007c0ab9b0 con 0x7f008c09b170 2024-03-31T15:11:16.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:16.279+0000 7f0093200640 1 -- 172.21.15.16:0/3184925541 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f008c09b170 msgr2=0x7f008c0ccb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:11:16.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:16.279+0000 7f0093200640 1 --2- 172.21.15.16:0/3184925541 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f008c09b170 0x7f008c0ccb90 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f007c0097f0 tx=0x7f007c01c4e0 comp rx=0 tx=0).stop 2024-03-31T15:11:16.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:16.279+0000 7f0093200640 1 -- 172.21.15.16:0/3184925541 shutdown_connections 2024-03-31T15:11:16.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:16.279+0000 7f0093200640 1 --2- 172.21.15.16:0/3184925541 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f008c09b170 0x7f008c0ccb90 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:16.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:16.279+0000 7f0093200640 1 -- 172.21.15.16:0/3184925541 >> 172.21.15.16:0/3184925541 conn(0x7f008c0b39c0 msgr2=0x7f008c0b3dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:11:16.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:16.283+0000 7f0093200640 1 -- 172.21.15.16:0/3184925541 shutdown_connections 2024-03-31T15:11:16.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:16.283+0000 7f0093200640 1 -- 172.21.15.16:0/3184925541 wait complete. 2024-03-31T15:11:16.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:16.283+0000 7f0093200640 1 Processor -- start 2024-03-31T15:11:16.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:16.283+0000 7f0093200640 1 -- start start 2024-03-31T15:11:16.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:16.283+0000 7f0093200640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f008c070d50 0x7f008c154a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:16.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:16.283+0000 7f0093200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f008c09b170 0x7f008c0c8fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:16.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:16.283+0000 7f0093200640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f008c15e3b0 0x7f008c0c94e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:16.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:16.283+0000 7f0093200640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f008c16c230 con 0x7f008c09b170 2024-03-31T15:11:16.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:16.283+0000 7f0093200640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f008c16c0b0 con 0x7f008c070d50 2024-03-31T15:11:16.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:16.283+0000 7f0093200640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f008c16c3b0 con 0x7f008c15e3b0 2024-03-31T15:11:16.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:16.283+0000 7f0091400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f008c09b170 0x7f008c0c8fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:16.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:16.283+0000 7f0091400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f008c09b170 0x7f008c0c8fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38038/0 (socket says 172.21.15.16:38038) 2024-03-31T15:11:16.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:16.283+0000 7f0091400640 1 -- 172.21.15.16:0/2235237517 learned_addr learned my addr 172.21.15.16:0/2235237517 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:11:16.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:16.283+0000 7f0091e00640 1 --2- 172.21.15.16:0/2235237517 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f008c070d50 0x7f008c154a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:16.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:16.283+0000 7f0092800640 1 --2- 172.21.15.16:0/2235237517 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f008c15e3b0 0x7f008c0c94e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:16.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:16.283+0000 7f0091400640 1 -- 172.21.15.16:0/2235237517 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f008c15e3b0 msgr2=0x7f008c0c94e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:11:16.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:16.283+0000 7f0091400640 1 --2- 172.21.15.16:0/2235237517 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f008c15e3b0 0x7f008c0c94e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:16.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:16.283+0000 7f0091400640 1 -- 172.21.15.16:0/2235237517 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f008c070d50 msgr2=0x7f008c154a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:11:16.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:16.283+0000 7f0091400640 1 --2- 172.21.15.16:0/2235237517 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f008c070d50 0x7f008c154a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:16.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:16.283+0000 7f0091400640 1 -- 172.21.15.16:0/2235237517 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f008c15ec50 con 0x7f008c09b170 2024-03-31T15:11:16.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:16.283+0000 7f0091400640 1 --2- 172.21.15.16:0/2235237517 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f008c09b170 0x7f008c0c8fa0 secure :-1 s=READY pgs=1232 cs=0 l=1 rev1=1 crypto rx=0x7f007c0190f0 tx=0x7f007c01ad20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:11:16.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:16.287+0000 7f0086000640 1 -- 172.21.15.16:0/2235237517 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f007c0bf610 con 0x7f008c09b170 2024-03-31T15:11:16.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:16.287+0000 7f0086000640 1 -- 172.21.15.16:0/2235237517 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f007c0bf7d0 con 0x7f008c09b170 2024-03-31T15:11:16.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:16.287+0000 7f0086000640 1 -- 172.21.15.16:0/2235237517 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f007c0bfbc0 con 0x7f008c09b170 2024-03-31T15:11:16.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:16.287+0000 7f0093200640 1 -- 172.21.15.16:0/2235237517 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f008c0c9b10 con 0x7f008c09b170 2024-03-31T15:11:16.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:16.287+0000 7f0093200640 1 -- 172.21.15.16:0/2235237517 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f008c0ca0d0 con 0x7f008c09b170 2024-03-31T15:11:16.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:16.287+0000 7f0093200640 1 -- 172.21.15.16:0/2235237517 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f008c04f8c0 con 0x7f008c09b170 2024-03-31T15:11:16.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:16.287+0000 7f0086000640 1 -- 172.21.15.16:0/2235237517 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f007c0bd020 con 0x7f008c09b170 2024-03-31T15:11:16.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:16.291+0000 7f0086000640 1 --2- 172.21.15.16:0/2235237517 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f00700683d0 0x7f007006a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:16.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:16.291+0000 7f0086000640 1 -- 172.21.15.16:0/2235237517 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 5824+0+0 (secure 0 0 0) 0x7f007c121760 con 0x7f008c09b170 2024-03-31T15:11:16.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:16.291+0000 7f0091e00640 1 --2- 172.21.15.16:0/2235237517 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f00700683d0 0x7f007006a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:16.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:16.291+0000 7f0091e00640 1 --2- 172.21.15.16:0/2235237517 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f00700683d0 0x7f007006a890 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f0080007bc0 tx=0x7f0080007b10 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:11:16.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:16.291+0000 7f0086000640 1 -- 172.21.15.16:0/2235237517 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f007c0cd080 con 0x7f008c09b170 2024-03-31T15:11:16.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:16.471+0000 7f0093200640 1 -- 172.21.15.16:0/2235237517 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f008c154fa0 con 0x7f008c09b170 2024-03-31T15:11:16.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:16.471+0000 7f0086000640 1 -- 172.21.15.16:0/2235237517 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 2 entries v84) ==== 80+0+122 (secure 0 0 0) 0x7f007c0160d0 con 0x7f008c09b170 2024-03-31T15:11:16.464 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:6834/2478530200 2024-04-01T15:10:04.360772+0000 2024-03-31T15:11:16.464 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:6832/2478530200 2024-04-01T15:10:04.360772+0000 2024-03-31T15:11:16.465 INFO:teuthology.orchestra.run.smithi016.stderr:listed 2 entries 2024-03-31T15:11:16.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:16.475+0000 7f0093200640 1 -- 172.21.15.16:0/2235237517 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f00700683d0 msgr2=0x7f007006a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:11:16.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:16.475+0000 7f0093200640 1 --2- 172.21.15.16:0/2235237517 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f00700683d0 0x7f007006a890 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f0080007bc0 tx=0x7f0080007b10 comp rx=0 tx=0).stop 2024-03-31T15:11:16.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:16.475+0000 7f0093200640 1 -- 172.21.15.16:0/2235237517 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f008c09b170 msgr2=0x7f008c0c8fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:11:16.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:16.475+0000 7f0093200640 1 --2- 172.21.15.16:0/2235237517 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f008c09b170 0x7f008c0c8fa0 secure :-1 s=READY pgs=1232 cs=0 l=1 rev1=1 crypto rx=0x7f007c0190f0 tx=0x7f007c01ad20 comp rx=0 tx=0).stop 2024-03-31T15:11:16.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:16.475+0000 7f0093200640 1 -- 172.21.15.16:0/2235237517 shutdown_connections 2024-03-31T15:11:16.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:16.475+0000 7f0093200640 1 --2- 172.21.15.16:0/2235237517 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f00700683d0 0x7f007006a890 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:16.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:16.475+0000 7f0093200640 1 --2- 172.21.15.16:0/2235237517 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f008c15e3b0 0x7f008c0c94e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:16.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:16.475+0000 7f0093200640 1 --2- 172.21.15.16:0/2235237517 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f008c09b170 0x7f008c0c8fa0 unknown :-1 s=CLOSED pgs=1232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:16.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:16.475+0000 7f0093200640 1 --2- 172.21.15.16:0/2235237517 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f008c070d50 0x7f008c154a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:16.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:16.475+0000 7f0093200640 1 -- 172.21.15.16:0/2235237517 >> 172.21.15.16:0/2235237517 conn(0x7f008c0b39c0 msgr2=0x7f008c0ce9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:11:16.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:16.475+0000 7f0093200640 1 -- 172.21.15.16:0/2235237517 shutdown_connections 2024-03-31T15:11:16.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:16.475+0000 7f0093200640 1 -- 172.21.15.16:0/2235237517 wait complete. 2024-03-31T15:11:16.486 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-03-31T15:11:16.486 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-31T15:11:16.495 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-31T15:11:16.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:16.687+0000 7fba92e00640 1 Processor -- start 2024-03-31T15:11:16.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:16.687+0000 7fba92e00640 1 -- start start 2024-03-31T15:11:16.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:16.687+0000 7fba92e00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fba8c15b3d0 0x7fba8c0b1cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:16.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:16.687+0000 7fba92e00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fba8c15eee0 0x7fba8c0b2200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:16.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:16.687+0000 7fba92e00640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fba8c15f3c0 con 0x7fba8c0ce2f0 2024-03-31T15:11:16.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:16.687+0000 7fba92e00640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7fba8c15b7b0 con 0x7fba8c15b3d0 2024-03-31T15:11:16.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:16.687+0000 7fba92e00640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7fba8c15f6a0 con 0x7fba8c15eee0 2024-03-31T15:11:16.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:16.687+0000 7fba8be00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fba8c15b3d0 0x7fba8c0b1cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:16.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:16.687+0000 7fba8be00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fba8c15b3d0 0x7fba8c0b1cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38046/0 (socket says 172.21.15.16:38046) 2024-03-31T15:11:16.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:16.687+0000 7fba8be00640 1 -- 172.21.15.16:0/3189580707 learned_addr learned my addr 172.21.15.16:0/3189580707 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:11:16.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:16.687+0000 7fba83e00640 1 --2- 172.21.15.16:0/3189580707 >> v2:172.21.15.89:3300/0 conn(0x7fba8c15eee0 0x7fba8c0b2200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:16.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:16.687+0000 7fba8be00640 1 -- 172.21.15.16:0/3189580707 >> v2:172.21.15.89:3300/0 conn(0x7fba8c15eee0 msgr2=0x7fba8c0b2200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:11:16.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:16.687+0000 7fba8be00640 1 --2- 172.21.15.16:0/3189580707 >> v2:172.21.15.89:3300/0 conn(0x7fba8c15eee0 0x7fba8c0b2200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:16.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:16.687+0000 7fba8be00640 1 -- 172.21.15.16:0/3189580707 >> v1:172.21.15.89:6789/0 conn(0x7fba8c0ce2f0 legacy=0x7fba8c0cc9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:11:16.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:16.687+0000 7fba8be00640 1 -- 172.21.15.16:0/3189580707 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba8c15f980 con 0x7fba8c15b3d0 2024-03-31T15:11:16.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:16.687+0000 7fba83e00640 1 --2- 172.21.15.16:0/3189580707 >> v2:172.21.15.89:3300/0 conn(0x7fba8c15eee0 0x7fba8c0b2200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:11:16.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:16.691+0000 7fba8be00640 1 --2- 172.21.15.16:0/3189580707 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fba8c15b3d0 0x7fba8c0b1cc0 secure :-1 s=READY pgs=1233 cs=0 l=1 rev1=1 crypto rx=0x7fba8c0cdae0 tx=0x7fba7c01c4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f5761caa64f429c server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:11:16.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:16.691+0000 7fba8b400640 1 -- 172.21.15.16:0/3189580707 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba7c0a7070 con 0x7fba8c15b3d0 2024-03-31T15:11:16.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:16.691+0000 7fba8b400640 1 -- 172.21.15.16:0/3189580707 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fba7c0b0070 con 0x7fba8c15b3d0 2024-03-31T15:11:16.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:16.691+0000 7fba8b400640 1 -- 172.21.15.16:0/3189580707 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba7c0ab890 con 0x7fba8c15b3d0 2024-03-31T15:11:16.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:16.691+0000 7fba92e00640 1 -- 172.21.15.16:0/3189580707 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fba8c15b3d0 msgr2=0x7fba8c0b1cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:11:16.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:16.691+0000 7fba92e00640 1 --2- 172.21.15.16:0/3189580707 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fba8c15b3d0 0x7fba8c0b1cc0 secure :-1 s=READY pgs=1233 cs=0 l=1 rev1=1 crypto rx=0x7fba8c0cdae0 tx=0x7fba7c01c4b0 comp rx=0 tx=0).stop 2024-03-31T15:11:16.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:16.691+0000 7fba92e00640 1 -- 172.21.15.16:0/3189580707 shutdown_connections 2024-03-31T15:11:16.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:16.691+0000 7fba92e00640 1 --2- 172.21.15.16:0/3189580707 >> v2:172.21.15.89:3300/0 conn(0x7fba8c15eee0 0x7fba8c0b2200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:16.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:16.691+0000 7fba92e00640 1 --2- 172.21.15.16:0/3189580707 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fba8c15b3d0 0x7fba8c0b1cc0 unknown :-1 s=CLOSED pgs=1233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:16.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:16.691+0000 7fba92e00640 1 -- 172.21.15.16:0/3189580707 >> 172.21.15.16:0/3189580707 conn(0x7fba8c06b620 msgr2=0x7fba8c06ba30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:11:16.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:16.691+0000 7fba92e00640 1 -- 172.21.15.16:0/3189580707 shutdown_connections 2024-03-31T15:11:16.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:16.691+0000 7fba92e00640 1 -- 172.21.15.16:0/3189580707 wait complete. 2024-03-31T15:11:16.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:16.695+0000 7fba92e00640 1 Processor -- start 2024-03-31T15:11:16.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:16.695+0000 7fba92e00640 1 -- start start 2024-03-31T15:11:16.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:16.695+0000 7fba92e00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fba8c0ce2f0 0x7fba8c143810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:16.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:16.695+0000 7fba92e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fba8c15b3d0 0x7fba8c144070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:16.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:16.695+0000 7fba92e00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fba8c15eee0 0x7fba8c1445b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:16.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:16.695+0000 7fba92e00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fba8c166600 con 0x7fba8c15b3d0 2024-03-31T15:11:16.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:16.695+0000 7fba92e00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fba8c166480 con 0x7fba8c0ce2f0 2024-03-31T15:11:16.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:16.695+0000 7fba92e00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fba8c166780 con 0x7fba8c15eee0 2024-03-31T15:11:16.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:16.695+0000 7fba83e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fba8c15b3d0 0x7fba8c144070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:16.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:16.695+0000 7fba8be00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fba8c0ce2f0 0x7fba8c143810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:16.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:16.695+0000 7fba83e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fba8c15b3d0 0x7fba8c144070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38060/0 (socket says 172.21.15.16:38060) 2024-03-31T15:11:16.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:16.695+0000 7fba83e00640 1 -- 172.21.15.16:0/1675112399 learned_addr learned my addr 172.21.15.16:0/1675112399 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:11:16.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:16.695+0000 7fba90e00640 1 --2- 172.21.15.16:0/1675112399 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fba8c15eee0 0x7fba8c1445b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:16.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:16.695+0000 7fba83e00640 1 -- 172.21.15.16:0/1675112399 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fba8c15eee0 msgr2=0x7fba8c1445b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:11:16.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:16.695+0000 7fba83e00640 1 --2- 172.21.15.16:0/1675112399 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fba8c15eee0 0x7fba8c1445b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:16.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:16.695+0000 7fba83e00640 1 -- 172.21.15.16:0/1675112399 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fba8c0ce2f0 msgr2=0x7fba8c143810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:11:16.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:16.695+0000 7fba83e00640 1 --2- 172.21.15.16:0/1675112399 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fba8c0ce2f0 0x7fba8c143810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:16.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:16.695+0000 7fba83e00640 1 -- 172.21.15.16:0/1675112399 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba8c0b0af0 con 0x7fba8c15b3d0 2024-03-31T15:11:16.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:16.695+0000 7fba83e00640 1 --2- 172.21.15.16:0/1675112399 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fba8c15b3d0 0x7fba8c144070 secure :-1 s=READY pgs=1234 cs=0 l=1 rev1=1 crypto rx=0x7fba7400c820 tx=0x7fba7400cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:11:16.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:16.695+0000 7fba88c00640 1 -- 172.21.15.16:0/1675112399 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba74015070 con 0x7fba8c15b3d0 2024-03-31T15:11:16.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:16.695+0000 7fba88c00640 1 -- 172.21.15.16:0/1675112399 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fba74007a30 con 0x7fba8c15b3d0 2024-03-31T15:11:16.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:16.695+0000 7fba88c00640 1 -- 172.21.15.16:0/1675112399 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba74007e00 con 0x7fba8c15b3d0 2024-03-31T15:11:16.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:16.695+0000 7fba92e00640 1 -- 172.21.15.16:0/1675112399 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fba8c0b0dd0 con 0x7fba8c15b3d0 2024-03-31T15:11:16.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:16.695+0000 7fba92e00640 1 -- 172.21.15.16:0/1675112399 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fba8c0b1390 con 0x7fba8c15b3d0 2024-03-31T15:11:16.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:16.695+0000 7fba92e00640 1 -- 172.21.15.16:0/1675112399 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fba8c069950 con 0x7fba8c15b3d0 2024-03-31T15:11:16.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:16.699+0000 7fba88c00640 1 -- 172.21.15.16:0/1675112399 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fba74018070 con 0x7fba8c15b3d0 2024-03-31T15:11:16.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:16.699+0000 7fba88c00640 1 --2- 172.21.15.16:0/1675112399 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fba68068380 0x7fba6806a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:16.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:16.699+0000 7fba88c00640 1 -- 172.21.15.16:0/1675112399 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 5824+0+0 (secure 0 0 0) 0x7fba74086d50 con 0x7fba8c15b3d0 2024-03-31T15:11:16.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:16.699+0000 7fba8be00640 1 --2- 172.21.15.16:0/1675112399 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fba68068380 0x7fba6806a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:16.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:16.699+0000 7fba8be00640 1 --2- 172.21.15.16:0/1675112399 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fba68068380 0x7fba6806a840 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7fba7c0af000 tx=0x7fba7c0a7000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:11:16.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:16.707+0000 7fba88c00640 1 -- 172.21.15.16:0/1675112399 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fba74056c50 con 0x7fba8c15b3d0 2024-03-31T15:11:16.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:16.871+0000 7fba92e00640 1 -- 172.21.15.16:0/1675112399 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fba8c0cdea0 con 0x7fba8c15b3d0 2024-03-31T15:11:16.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:16.875+0000 7fba88c00640 1 -- 172.21.15.16:0/1675112399 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v84) ==== 74+0+16766 (secure 0 0 0) 0x7fba7405af50 con 0x7fba8c15b3d0 2024-03-31T15:11:16.868 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:11:16.868 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":84,"fsid":"286eb5da-fb47-4550-b290-d9519c13b9a5","created":"2024-03-31T15:03:11.438952+0000","modified":"2024-03-31T15:10:18.450885+0000","last_up_change":"2024-03-31T15:03:22.885535+0000","last_in_change":"2024-03-31T15:03:13.941667+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-31T15:03:22.550787+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-03-31T15:09:49.322620+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-03-31T15:09:49.877778+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"92a66450-ef19-40d8-ac3d-d760b23b0bda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6801","nonce":932712244}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6803","nonce":932712244}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6807","nonce":932712244}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6805","nonce":932712244}]},"public_addr":"172.21.15.16:6801/932712244","cluster_addr":"172.21.15.16:6803/932712244","heartbeat_back_addr":"172.21.15.16:6807/932712244","heartbeat_front_addr":"172.21.15.16:6805/932712244","state":["exists","up"]},{"osd":1,"uuid":"da9d1f2b-1968-4959-b843-faa231988e92","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6810","nonce":1398737533}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6811","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6813","nonce":1398737533}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6821","nonce":1398737533}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6815","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6816","nonce":1398737533}]},"public_addr":"172.21.15.16:6810/1398737533","cluster_addr":"172.21.15.16:6813/1398737533","heartbeat_back_addr":"172.21.15.16:6821/1398737533","heartbeat_front_addr":"172.21.15.16:6816/1398737533","state":["exists","up"]},{"osd":2,"uuid":"01a73662-9a21-4b1f-bcbb-42b9636a4638","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6819","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6822","nonce":3018526646}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6825","nonce":3018526646}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6831","nonce":3018526646}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6827","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6829","nonce":3018526646}]},"public_addr":"172.21.15.16:6822/3018526646","cluster_addr":"172.21.15.16:6825/3018526646","heartbeat_back_addr":"172.21.15.16:6831/3018526646","heartbeat_front_addr":"172.21.15.16:6829/3018526646","state":["exists","up"]},{"osd":3,"uuid":"c7771fd3-e18e-4500-ac38-23cf10ebcd75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6809","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6812","nonce":3360352974}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6817","nonce":3360352974}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6828","nonce":3360352974}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6823","nonce":3360352974}]},"public_addr":"172.21.15.16:6812/3360352974","cluster_addr":"172.21.15.16:6817/3360352974","heartbeat_back_addr":"172.21.15.16:6828/3360352974","heartbeat_front_addr":"172.21.15.16:6823/3360352974","state":["exists","up"]},{"osd":4,"uuid":"88260730-1482-4fbd-a40f-cb1124e9df4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6801","nonce":1856844856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6803","nonce":1856844856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6807","nonce":1856844856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6805","nonce":1856844856}]},"public_addr":"172.21.15.89:6801/1856844856","cluster_addr":"172.21.15.89:6803/1856844856","heartbeat_back_addr":"172.21.15.89:6807/1856844856","heartbeat_front_addr":"172.21.15.89:6805/1856844856","state":["exists","up"]},{"osd":5,"uuid":"4cb2c00c-fae0-4dd8-9d27-ec4f0093ccf7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6814","nonce":50395527}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6817","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6820","nonce":50395527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6831","nonce":50395527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6823","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6826","nonce":50395527}]},"public_addr":"172.21.15.89:6814/50395527","cluster_addr":"172.21.15.89:6820/50395527","heartbeat_back_addr":"172.21.15.89:6831/50395527","heartbeat_front_addr":"172.21.15.89:6826/50395527","state":["exists","up"]},{"osd":6,"uuid":"81ce0c19-2554-4cae-acbe-66a05e9e9b9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6809","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6812","nonce":3470471752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6815","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6818","nonce":3470471752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6827","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6829","nonce":3470471752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6821","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6824","nonce":3470471752}]},"public_addr":"172.21.15.89:6812/3470471752","cluster_addr":"172.21.15.89:6818/3470471752","heartbeat_back_addr":"172.21.15.89:6829/3470471752","heartbeat_front_addr":"172.21.15.89:6824/3470471752","state":["exists","up"]},{"osd":7,"uuid":"85e4313b-d37d-442e-8042-153bf20a847d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6811","nonce":1796778630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6813","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6816","nonce":1796778630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6825","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6828","nonce":1796778630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6819","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6822","nonce":1796778630}]},"public_addr":"172.21.15.89:6811/1796778630","cluster_addr":"172.21.15.89:6816/1796778630","heartbeat_back_addr":"172.21.15.89:6828/1796778630","heartbeat_front_addr":"172.21.15.89:6822/1796778630","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.439088+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.529970+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.680512+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:21.044161+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.582111+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.536558+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.409522+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.590229+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"13.d","mappings":[{"from":1,"to":3}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:6834/2478530200":"2024-04-01T15:10:04.360772+0000","172.21.15.16:6832/2478530200":"2024-04-01T15:10:04.360772+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-31T15:11:16.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:16.875+0000 7fba92e00640 1 -- 172.21.15.16:0/1675112399 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fba68068380 msgr2=0x7fba6806a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:11:16.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:16.875+0000 7fba92e00640 1 --2- 172.21.15.16:0/1675112399 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fba68068380 0x7fba6806a840 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7fba7c0af000 tx=0x7fba7c0a7000 comp rx=0 tx=0).stop 2024-03-31T15:11:16.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:16.875+0000 7fba92e00640 1 -- 172.21.15.16:0/1675112399 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fba8c15b3d0 msgr2=0x7fba8c144070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:11:16.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:16.875+0000 7fba92e00640 1 --2- 172.21.15.16:0/1675112399 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fba8c15b3d0 0x7fba8c144070 secure :-1 s=READY pgs=1234 cs=0 l=1 rev1=1 crypto rx=0x7fba7400c820 tx=0x7fba7400cce0 comp rx=0 tx=0).stop 2024-03-31T15:11:16.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:16.875+0000 7fba92e00640 1 -- 172.21.15.16:0/1675112399 shutdown_connections 2024-03-31T15:11:16.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:16.875+0000 7fba92e00640 1 --2- 172.21.15.16:0/1675112399 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fba68068380 0x7fba6806a840 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:16.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:16.875+0000 7fba92e00640 1 --2- 172.21.15.16:0/1675112399 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fba8c15eee0 0x7fba8c1445b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:16.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:16.875+0000 7fba92e00640 1 --2- 172.21.15.16:0/1675112399 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fba8c15b3d0 0x7fba8c144070 unknown :-1 s=CLOSED pgs=1234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:16.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:16.875+0000 7fba92e00640 1 --2- 172.21.15.16:0/1675112399 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fba8c0ce2f0 0x7fba8c143810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:16.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:16.875+0000 7fba92e00640 1 -- 172.21.15.16:0/1675112399 >> 172.21.15.16:0/1675112399 conn(0x7fba8c06b620 msgr2=0x7fba8c0d0960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:11:16.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:16.875+0000 7fba92e00640 1 -- 172.21.15.16:0/1675112399 shutdown_connections 2024-03-31T15:11:16.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:16.875+0000 7fba92e00640 1 -- 172.21.15.16:0/1675112399 wait complete. 2024-03-31T15:11:16.889 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-31T15:03:22.550787+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 12, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-31T15:09:49.322620+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '80', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 13, 'pool_name': 'cephfs_data', 'create_time': '2024-03-31T15:09:49.877778+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '81', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-31T15:11:16.889 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-31T15:11:17.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:17.039+0000 7f36da400640 1 Processor -- start 2024-03-31T15:11:17.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:17.039+0000 7f36da400640 1 -- start start 2024-03-31T15:11:17.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:17.039+0000 7f36da400640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f36d406dbe0 0x7f36d40b3130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:17.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:17.039+0000 7f36da400640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f36d4073310 0x7f36d40ada40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:17.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:17.039+0000 7f36da400640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f36d40737a0 con 0x7f36d4147d60 2024-03-31T15:11:17.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:17.039+0000 7f36da400640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f36d4148140 con 0x7f36d4073310 2024-03-31T15:11:17.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:17.039+0000 7f36da400640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f36d40b37c0 con 0x7f36d406dbe0 2024-03-31T15:11:17.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:17.039+0000 7f36d3e00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f36d4073310 0x7f36d40ada40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:17.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:17.039+0000 7f36d3400640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f36d4147d60 0x7f36d406d4d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:33664/0 (socket says 172.21.15.16:33664) 2024-03-31T15:11:17.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:17.039+0000 7f36d3400640 1 -- 172.21.15.16:0/1693451776 learned_addr learned my addr 172.21.15.16:0/1693451776 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:11:17.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:17.039+0000 7f36d3e00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f36d4073310 0x7f36d40ada40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56096/0 (socket says 172.21.15.16:56096) 2024-03-31T15:11:17.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:17.039+0000 7f36caa00640 1 --2- 172.21.15.16:0/1693451776 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f36d406dbe0 0x7f36d40b3130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:17.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:17.043+0000 7f36d3e00640 1 -- 172.21.15.16:0/1693451776 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f36d406dbe0 msgr2=0x7f36d40b3130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:11:17.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:17.043+0000 7f36d3e00640 1 --2- 172.21.15.16:0/1693451776 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f36d406dbe0 0x7f36d40b3130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:17.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:17.043+0000 7f36d3e00640 1 -- 172.21.15.16:0/1693451776 >> v1:172.21.15.16:6789/0 conn(0x7f36d4147d60 legacy=0x7f36d406d4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:11:17.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:17.043+0000 7f36d3e00640 1 -- 172.21.15.16:0/1693451776 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f36d40b3aa0 con 0x7f36d4073310 2024-03-31T15:11:17.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:17.043+0000 7f36d2a00640 1 -- 172.21.15.16:0/1693451776 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2931278898 0 0) 0x7f36d40737a0 con 0x7f36d4147d60 2024-03-31T15:11:17.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:17.043+0000 7f36d3e00640 1 --2- 172.21.15.16:0/1693451776 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f36d4073310 0x7f36d40ada40 secure :-1 s=READY pgs=1236 cs=0 l=1 rev1=1 crypto rx=0x7f36cc009d90 tx=0x7f36cc01c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=707bd63c626e9e9d server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:11:17.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:17.043+0000 7f36d2a00640 1 -- 172.21.15.16:0/1693451776 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36cc0b2070 con 0x7f36d4073310 2024-03-31T15:11:17.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:17.043+0000 7f36d2a00640 1 -- 172.21.15.16:0/1693451776 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f36cc0a5a00 con 0x7f36d4073310 2024-03-31T15:11:17.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:17.043+0000 7f36d2a00640 1 -- 172.21.15.16:0/1693451776 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36cc0a5dd0 con 0x7f36d4073310 2024-03-31T15:11:17.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:17.043+0000 7f36da400640 1 -- 172.21.15.16:0/1693451776 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f36d4073310 msgr2=0x7f36d40ada40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:11:17.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:17.043+0000 7f36da400640 1 --2- 172.21.15.16:0/1693451776 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f36d4073310 0x7f36d40ada40 secure :-1 s=READY pgs=1236 cs=0 l=1 rev1=1 crypto rx=0x7f36cc009d90 tx=0x7f36cc01c940 comp rx=0 tx=0).stop 2024-03-31T15:11:17.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:17.043+0000 7f36da400640 1 -- 172.21.15.16:0/1693451776 shutdown_connections 2024-03-31T15:11:17.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:17.043+0000 7f36da400640 1 --2- 172.21.15.16:0/1693451776 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f36d4073310 0x7f36d40ada40 unknown :-1 s=CLOSED pgs=1236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:17.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:17.043+0000 7f36da400640 1 --2- 172.21.15.16:0/1693451776 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f36d406dbe0 0x7f36d40b3130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:17.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:17.043+0000 7f36da400640 1 -- 172.21.15.16:0/1693451776 >> 172.21.15.16:0/1693451776 conn(0x7f36d406ee40 msgr2=0x7f36d406b620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:11:17.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:17.047+0000 7f36da400640 1 -- 172.21.15.16:0/1693451776 shutdown_connections 2024-03-31T15:11:17.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:17.047+0000 7f36da400640 1 -- 172.21.15.16:0/1693451776 wait complete. 2024-03-31T15:11:17.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:17.047+0000 7f36da400640 1 Processor -- start 2024-03-31T15:11:17.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:17.047+0000 7f36da400640 1 -- start start 2024-03-31T15:11:17.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:17.047+0000 7f36da400640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f36d406dbe0 0x7f36d4144750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:17.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:17.047+0000 7f36da400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f36d4073310 0x7f36d4145b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:17.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:17.047+0000 7f36da400640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f36d4147d60 0x7f36d41405b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:17.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:17.047+0000 7f36da400640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f36d40ae910 con 0x7f36d4073310 2024-03-31T15:11:17.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:17.047+0000 7f36da400640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f36d40ae790 con 0x7f36d406dbe0 2024-03-31T15:11:17.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:17.047+0000 7f36da400640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f36d40aea90 con 0x7f36d4147d60 2024-03-31T15:11:17.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:17.047+0000 7f36caa00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f36d4073310 0x7f36d4145b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:17.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:17.047+0000 7f36caa00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f36d4073310 0x7f36d4145b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56100/0 (socket says 172.21.15.16:56100) 2024-03-31T15:11:17.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:17.047+0000 7f36caa00640 1 -- 172.21.15.16:0/55369296 learned_addr learned my addr 172.21.15.16:0/55369296 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:11:17.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:17.047+0000 7f36d3e00640 1 --2- 172.21.15.16:0/55369296 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f36d4147d60 0x7f36d41405b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:17.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:17.047+0000 7f36d3400640 1 --2- 172.21.15.16:0/55369296 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f36d406dbe0 0x7f36d4144750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:17.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:17.047+0000 7f36caa00640 1 -- 172.21.15.16:0/55369296 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f36d4147d60 msgr2=0x7f36d41405b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:11:17.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:17.047+0000 7f36caa00640 1 --2- 172.21.15.16:0/55369296 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f36d4147d60 0x7f36d41405b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:17.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:17.047+0000 7f36caa00640 1 -- 172.21.15.16:0/55369296 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f36d406dbe0 msgr2=0x7f36d4144750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:11:17.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:17.047+0000 7f36caa00640 1 --2- 172.21.15.16:0/55369296 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f36d406dbe0 0x7f36d4144750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:17.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:17.047+0000 7f36caa00640 1 -- 172.21.15.16:0/55369296 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f36d41424e0 con 0x7f36d4073310 2024-03-31T15:11:17.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:17.047+0000 7f36caa00640 1 --2- 172.21.15.16:0/55369296 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f36d4073310 0x7f36d4145b40 secure :-1 s=READY pgs=1237 cs=0 l=1 rev1=1 crypto rx=0x7f36bc00e960 tx=0x7f36bc00ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:11:17.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:17.047+0000 7f36cbe00640 1 -- 172.21.15.16:0/55369296 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36bc017070 con 0x7f36d4073310 2024-03-31T15:11:17.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:17.047+0000 7f36cbe00640 1 -- 172.21.15.16:0/55369296 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f36bc00c9a0 con 0x7f36d4073310 2024-03-31T15:11:17.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:17.051+0000 7f36da400640 1 -- 172.21.15.16:0/55369296 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f36d41427c0 con 0x7f36d4073310 2024-03-31T15:11:17.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:17.051+0000 7f36cbe00640 1 -- 172.21.15.16:0/55369296 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36bc00cd50 con 0x7f36d4073310 2024-03-31T15:11:17.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:17.051+0000 7f36da400640 1 -- 172.21.15.16:0/55369296 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f36d4143060 con 0x7f36d4073310 2024-03-31T15:11:17.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:17.051+0000 7f36da400640 1 -- 172.21.15.16:0/55369296 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f36d4069950 con 0x7f36d4073310 2024-03-31T15:11:17.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:17.051+0000 7f36cbe00640 1 -- 172.21.15.16:0/55369296 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f36bc01a070 con 0x7f36d4073310 2024-03-31T15:11:17.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:17.051+0000 7f36cbe00640 1 --2- 172.21.15.16:0/55369296 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f36b0068380 0x7f36b006a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:17.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:17.051+0000 7f36cbe00640 1 -- 172.21.15.16:0/55369296 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 5824+0+0 (secure 0 0 0) 0x7f36bc086d10 con 0x7f36d4073310 2024-03-31T15:11:17.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:17.055+0000 7f36d3400640 1 --2- 172.21.15.16:0/55369296 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f36b0068380 0x7f36b006a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:17.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:17.055+0000 7f36d3400640 1 --2- 172.21.15.16:0/55369296 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f36b0068380 0x7f36b006a840 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f36c4000f70 tx=0x7f36c4007df0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:11:17.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:17.059+0000 7f36cbe00640 1 -- 172.21.15.16:0/55369296 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f36bc08c020 con 0x7f36d4073310 2024-03-31T15:11:17.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:17.223+0000 7f36da400640 1 -- 172.21.15.16:0/55369296 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f36d406e470 con 0x7f36d4073310 2024-03-31T15:11:17.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:17.223+0000 7f36cbe00640 1 -- 172.21.15.16:0/55369296 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v84) ==== 93+0+10 (secure 0 0 0) 0x7f36bc056c20 con 0x7f36d4073310 2024-03-31T15:11:17.215 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2024-03-31T15:11:17.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:17.223+0000 7f36da400640 1 -- 172.21.15.16:0/55369296 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f36b0068380 msgr2=0x7f36b006a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:11:17.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:17.223+0000 7f36da400640 1 --2- 172.21.15.16:0/55369296 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f36b0068380 0x7f36b006a840 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f36c4000f70 tx=0x7f36c4007df0 comp rx=0 tx=0).stop 2024-03-31T15:11:17.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:17.223+0000 7f36da400640 1 -- 172.21.15.16:0/55369296 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f36d4073310 msgr2=0x7f36d4145b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:11:17.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:17.223+0000 7f36da400640 1 --2- 172.21.15.16:0/55369296 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f36d4073310 0x7f36d4145b40 secure :-1 s=READY pgs=1237 cs=0 l=1 rev1=1 crypto rx=0x7f36bc00e960 tx=0x7f36bc00ee20 comp rx=0 tx=0).stop 2024-03-31T15:11:17.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:17.223+0000 7f36da400640 1 -- 172.21.15.16:0/55369296 shutdown_connections 2024-03-31T15:11:17.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:17.223+0000 7f36da400640 1 --2- 172.21.15.16:0/55369296 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f36b0068380 0x7f36b006a840 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:17.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:17.223+0000 7f36da400640 1 --2- 172.21.15.16:0/55369296 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f36d4147d60 0x7f36d41405b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:17.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:17.223+0000 7f36da400640 1 --2- 172.21.15.16:0/55369296 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f36d4073310 0x7f36d4145b40 unknown :-1 s=CLOSED pgs=1237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:17.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:17.223+0000 7f36da400640 1 --2- 172.21.15.16:0/55369296 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f36d406dbe0 0x7f36d4144750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:17.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:17.223+0000 7f36da400640 1 -- 172.21.15.16:0/55369296 >> 172.21.15.16:0/55369296 conn(0x7f36d406ee40 msgr2=0x7f36d406d3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:11:17.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:17.223+0000 7f36da400640 1 -- 172.21.15.16:0/55369296 shutdown_connections 2024-03-31T15:11:17.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:17.223+0000 7f36da400640 1 -- 172.21.15.16:0/55369296 wait complete. 2024-03-31T15:11:17.236 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-31T15:11:17.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:17.387+0000 7f2597000640 1 Processor -- start 2024-03-31T15:11:17.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:17.387+0000 7f2597000640 1 -- start start 2024-03-31T15:11:17.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:17.387+0000 7f2597000640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f2590070030 0x7f2590143c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:17.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:17.387+0000 7f2597000640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f2590141590 0x7f2590144170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:17.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:17.387+0000 7f2597000640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f25901446b0 con 0x7f25901411b0 2024-03-31T15:11:17.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:17.387+0000 7f2597000640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f259015f910 con 0x7f2590141590 2024-03-31T15:11:17.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:17.387+0000 7f2597000640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f2590162910 con 0x7f2590070030 2024-03-31T15:11:17.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:17.387+0000 7f2595000640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f2590141590 0x7f2590144170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:17.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:17.387+0000 7f2595000640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f2590141590 0x7f2590144170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56104/0 (socket says 172.21.15.16:56104) 2024-03-31T15:11:17.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:17.387+0000 7f2595000640 1 -- 172.21.15.16:0/2573734577 learned_addr learned my addr 172.21.15.16:0/2573734577 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:11:17.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:17.387+0000 7f258f400640 1 --2- 172.21.15.16:0/2573734577 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f2590070030 0x7f2590143c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:17.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:17.387+0000 7f2595000640 1 -- 172.21.15.16:0/2573734577 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f2590070030 msgr2=0x7f2590143c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:11:17.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:17.387+0000 7f2595000640 1 --2- 172.21.15.16:0/2573734577 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f2590070030 0x7f2590143c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:17.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:17.387+0000 7f2595000640 1 -- 172.21.15.16:0/2573734577 >> v1:172.21.15.89:6789/0 conn(0x7f25901411b0 legacy=0x7f259006f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:11:17.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:17.387+0000 7f2595000640 1 -- 172.21.15.16:0/2573734577 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2590144980 con 0x7f2590141590 2024-03-31T15:11:17.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:17.391+0000 7f2595000640 1 --2- 172.21.15.16:0/2573734577 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2590141590 0x7f2590144170 secure :-1 s=READY pgs=1238 cs=0 l=1 rev1=1 crypto rx=0x7f2588009e50 tx=0x7f258801c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4839c015edfe190b server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:11:17.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:17.391+0000 7f258ea00640 1 -- 172.21.15.16:0/2573734577 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25880a5840 con 0x7f2590141590 2024-03-31T15:11:17.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:17.391+0000 7f258ea00640 1 -- 172.21.15.16:0/2573734577 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f25880a59e0 con 0x7f2590141590 2024-03-31T15:11:17.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:17.391+0000 7f258ea00640 1 -- 172.21.15.16:0/2573734577 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25880a5db0 con 0x7f2590141590 2024-03-31T15:11:17.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:17.391+0000 7f2597000640 1 -- 172.21.15.16:0/2573734577 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2590141590 msgr2=0x7f2590144170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:11:17.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:17.391+0000 7f2597000640 1 --2- 172.21.15.16:0/2573734577 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2590141590 0x7f2590144170 secure :-1 s=READY pgs=1238 cs=0 l=1 rev1=1 crypto rx=0x7f2588009e50 tx=0x7f258801c9c0 comp rx=0 tx=0).stop 2024-03-31T15:11:17.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:17.391+0000 7f2597000640 1 -- 172.21.15.16:0/2573734577 shutdown_connections 2024-03-31T15:11:17.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:17.391+0000 7f2597000640 1 --2- 172.21.15.16:0/2573734577 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2590141590 0x7f2590144170 unknown :-1 s=CLOSED pgs=1238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:17.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:17.391+0000 7f2597000640 1 --2- 172.21.15.16:0/2573734577 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f2590070030 0x7f2590143c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:17.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:17.391+0000 7f2597000640 1 -- 172.21.15.16:0/2573734577 >> 172.21.15.16:0/2573734577 conn(0x7f2590067930 msgr2=0x7f25900b3ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:11:17.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:17.391+0000 7f2597000640 1 -- 172.21.15.16:0/2573734577 shutdown_connections 2024-03-31T15:11:17.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:17.391+0000 7f2597000640 1 -- 172.21.15.16:0/2573734577 wait complete. 2024-03-31T15:11:17.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:17.395+0000 7f2597000640 1 Processor -- start 2024-03-31T15:11:17.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:17.395+0000 7f2597000640 1 -- start start 2024-03-31T15:11:17.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:17.395+0000 7f2597000640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2590070030 0x7f25901429f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:17.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:17.395+0000 7f2597000640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f25901411b0 0x7f2590142f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:17.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:17.395+0000 7f2597000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2590141590 0x7f259013ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:17.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:17.395+0000 7f2597000640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f259015f110 con 0x7f2590141590 2024-03-31T15:11:17.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:17.395+0000 7f2597000640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f2590162da0 con 0x7f2590070030 2024-03-31T15:11:17.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:17.395+0000 7f2597000640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f259015f290 con 0x7f25901411b0 2024-03-31T15:11:17.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:17.395+0000 7f2595000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2590141590 0x7f259013ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:17.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:17.395+0000 7f2595000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2590141590 0x7f259013ef20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56118/0 (socket says 172.21.15.16:56118) 2024-03-31T15:11:17.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:17.395+0000 7f2595000640 1 -- 172.21.15.16:0/4218737481 learned_addr learned my addr 172.21.15.16:0/4218737481 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:11:17.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:17.395+0000 7f258f400640 1 --2- 172.21.15.16:0/4218737481 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f25901411b0 0x7f2590142f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:17.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:17.395+0000 7f258fe00640 1 --2- 172.21.15.16:0/4218737481 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2590070030 0x7f25901429f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:17.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:17.395+0000 7f2595000640 1 -- 172.21.15.16:0/4218737481 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f25901411b0 msgr2=0x7f2590142f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:11:17.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:17.395+0000 7f2595000640 1 --2- 172.21.15.16:0/4218737481 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f25901411b0 0x7f2590142f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:17.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:17.395+0000 7f2595000640 1 -- 172.21.15.16:0/4218737481 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2590070030 msgr2=0x7f25901429f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:11:17.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:17.395+0000 7f2595000640 1 --2- 172.21.15.16:0/4218737481 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2590070030 0x7f25901429f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:17.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:17.395+0000 7f2595000640 1 -- 172.21.15.16:0/4218737481 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f259013ec20 con 0x7f2590141590 2024-03-31T15:11:17.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:17.395+0000 7f2595000640 1 --2- 172.21.15.16:0/4218737481 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2590141590 0x7f259013ef20 secure :-1 s=READY pgs=1239 cs=0 l=1 rev1=1 crypto rx=0x7f25880af040 tx=0x7f2588019760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:11:17.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:17.395+0000 7f2587e00640 1 -- 172.21.15.16:0/4218737481 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25880bc020 con 0x7f2590141590 2024-03-31T15:11:17.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:17.395+0000 7f2587e00640 1 -- 172.21.15.16:0/4218737481 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f25880b8690 con 0x7f2590141590 2024-03-31T15:11:17.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:17.395+0000 7f2597000640 1 -- 172.21.15.16:0/4218737481 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f259015a8f0 con 0x7f2590141590 2024-03-31T15:11:17.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:17.395+0000 7f2587e00640 1 -- 172.21.15.16:0/4218737481 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25880b8a80 con 0x7f2590141590 2024-03-31T15:11:17.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:17.395+0000 7f2597000640 1 -- 172.21.15.16:0/4218737481 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f259015aeb0 con 0x7f2590141590 2024-03-31T15:11:17.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:17.399+0000 7f2597000640 1 -- 172.21.15.16:0/4218737481 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2590069950 con 0x7f2590141590 2024-03-31T15:11:17.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:17.399+0000 7f2587e00640 1 -- 172.21.15.16:0/4218737481 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f25880bf020 con 0x7f2590141590 2024-03-31T15:11:17.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:17.399+0000 7f2587e00640 1 --2- 172.21.15.16:0/4218737481 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f2570068380 0x7f257006a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:17.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:17.399+0000 7f2587e00640 1 -- 172.21.15.16:0/4218737481 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 5824+0+0 (secure 0 0 0) 0x7f2588121bd0 con 0x7f2590141590 2024-03-31T15:11:17.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:17.399+0000 7f258fe00640 1 --2- 172.21.15.16:0/4218737481 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f2570068380 0x7f257006a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:17.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:17.403+0000 7f258fe00640 1 --2- 172.21.15.16:0/4218737481 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f2570068380 0x7f257006a840 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f2590143660 tx=0x7f2580009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:11:17.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:17.403+0000 7f2587e00640 1 -- 172.21.15.16:0/4218737481 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f25880c6030 con 0x7f2590141590 2024-03-31T15:11:17.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:17.567+0000 7f2597000640 1 -- 172.21.15.16:0/4218737481 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f2590070950 con 0x7f2590141590 2024-03-31T15:11:17.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:17.567+0000 7f2587e00640 1 -- 172.21.15.16:0/4218737481 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v84) ==== 104+0+11 (secure 0 0 0) 0x7f25880165e0 con 0x7f2590141590 2024-03-31T15:11:17.562 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2024-03-31T15:11:17.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:17.571+0000 7f2597000640 1 -- 172.21.15.16:0/4218737481 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f2570068380 msgr2=0x7f257006a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:11:17.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:17.571+0000 7f2597000640 1 --2- 172.21.15.16:0/4218737481 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f2570068380 0x7f257006a840 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f2590143660 tx=0x7f2580009040 comp rx=0 tx=0).stop 2024-03-31T15:11:17.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:17.571+0000 7f2597000640 1 -- 172.21.15.16:0/4218737481 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2590141590 msgr2=0x7f259013ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:11:17.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:17.571+0000 7f2597000640 1 --2- 172.21.15.16:0/4218737481 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2590141590 0x7f259013ef20 secure :-1 s=READY pgs=1239 cs=0 l=1 rev1=1 crypto rx=0x7f25880af040 tx=0x7f2588019760 comp rx=0 tx=0).stop 2024-03-31T15:11:17.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:17.571+0000 7f2597000640 1 -- 172.21.15.16:0/4218737481 shutdown_connections 2024-03-31T15:11:17.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:17.571+0000 7f2597000640 1 --2- 172.21.15.16:0/4218737481 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f2570068380 0x7f257006a840 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:17.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:17.571+0000 7f2597000640 1 --2- 172.21.15.16:0/4218737481 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2590141590 0x7f259013ef20 unknown :-1 s=CLOSED pgs=1239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:17.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:17.571+0000 7f2597000640 1 --2- 172.21.15.16:0/4218737481 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f25901411b0 0x7f2590142f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:17.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:17.571+0000 7f2597000640 1 --2- 172.21.15.16:0/4218737481 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2590070030 0x7f25901429f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:17.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:17.571+0000 7f2597000640 1 -- 172.21.15.16:0/4218737481 >> 172.21.15.16:0/4218737481 conn(0x7f2590067930 msgr2=0x7f2590160330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:11:17.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:17.571+0000 7f2597000640 1 -- 172.21.15.16:0/4218737481 shutdown_connections 2024-03-31T15:11:17.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:17.571+0000 7f2597000640 1 -- 172.21.15.16:0/4218737481 wait complete. 2024-03-31T15:11:17.582 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-31T15:11:17.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:17.735+0000 7f90f4c00640 1 Processor -- start 2024-03-31T15:11:17.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:17.735+0000 7f90f4c00640 1 -- start start 2024-03-31T15:11:17.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:17.735+0000 7f90f4c00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f90f006b620 0x7f90f006ba00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:17.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:17.735+0000 7f90f4c00640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f90f006bf40 con 0x7f90f00b60a0 2024-03-31T15:11:17.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:17.735+0000 7f90f4c00640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f90f006f540 con 0x7f90f00d0c00 2024-03-31T15:11:17.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:17.735+0000 7f90f4c00640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f90f006f820 con 0x7f90f006b620 2024-03-31T15:11:17.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:17.735+0000 7f90ed400640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f90f00b60a0 0x7f90f006fc40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:33666/0 (socket says 172.21.15.16:33666) 2024-03-31T15:11:17.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:17.735+0000 7f90ed400640 1 -- 172.21.15.16:0/3256618492 learned_addr learned my addr 172.21.15.16:0/3256618492 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:11:17.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:17.735+0000 7f90ede00640 1 --2- 172.21.15.16:0/3256618492 >> v2:172.21.15.89:3300/0 conn(0x7f90f006b620 0x7f90f006ba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:17.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:17.735+0000 7f90eca00640 1 -- 172.21.15.16:0/3256618492 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1453499892 0 0) 0x7f90f006bf40 con 0x7f90f00b60a0 2024-03-31T15:11:17.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:17.735+0000 7f90eca00640 1 -- 172.21.15.16:0/3256618492 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f90d4003530 con 0x7f90f00b60a0 2024-03-31T15:11:17.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:17.735+0000 7f90ede00640 1 -- 172.21.15.16:0/3256618492 >> v1:172.21.15.89:6789/0 conn(0x7f90f00d0c00 legacy=0x7f90f0070350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:11:17.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:17.735+0000 7f90ede00640 1 -- 172.21.15.16:0/3256618492 >> v1:172.21.15.16:6789/0 conn(0x7f90f00b60a0 legacy=0x7f90f006fc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:11:17.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:17.735+0000 7f90eca00640 1 -- 172.21.15.16:0/3256618492 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4293275406 0 0) 0x7f90d4003530 con 0x7f90f00b60a0 2024-03-31T15:11:17.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:17.735+0000 7f90ede00640 1 -- 172.21.15.16:0/3256618492 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f90f00b43f0 con 0x7f90f006b620 2024-03-31T15:11:17.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:17.739+0000 7f90ede00640 1 --2- 172.21.15.16:0/3256618492 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f90f006b620 0x7f90f006ba00 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f90e00097c0 tx=0x7f90e001c330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=560c5145ad27af19 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:11:17.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:17.739+0000 7f90eca00640 1 -- 172.21.15.16:0/3256618492 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90e00b0070 con 0x7f90f006b620 2024-03-31T15:11:17.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:17.739+0000 7f90eca00640 1 -- 172.21.15.16:0/3256618492 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f90e00ab400 con 0x7f90f006b620 2024-03-31T15:11:17.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:17.739+0000 7f90eca00640 1 -- 172.21.15.16:0/3256618492 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90dc001680 con 0x7f90f006b620 2024-03-31T15:11:17.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:17.739+0000 7f90f4c00640 1 -- 172.21.15.16:0/3256618492 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f90f006b620 msgr2=0x7f90f006ba00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:11:17.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:17.739+0000 7f90f4c00640 1 --2- 172.21.15.16:0/3256618492 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f90f006b620 0x7f90f006ba00 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f90e00097c0 tx=0x7f90e001c330 comp rx=0 tx=0).stop 2024-03-31T15:11:17.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:17.739+0000 7f90f4c00640 1 -- 172.21.15.16:0/3256618492 shutdown_connections 2024-03-31T15:11:17.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:17.739+0000 7f90f4c00640 1 --2- 172.21.15.16:0/3256618492 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f90f006b620 0x7f90f006ba00 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:17.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:17.739+0000 7f90f4c00640 1 -- 172.21.15.16:0/3256618492 >> 172.21.15.16:0/3256618492 conn(0x7f90f006e670 msgr2=0x7f90f006ea50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:11:17.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:17.739+0000 7f90f4c00640 1 -- 172.21.15.16:0/3256618492 shutdown_connections 2024-03-31T15:11:17.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:17.739+0000 7f90f4c00640 1 -- 172.21.15.16:0/3256618492 wait complete. 2024-03-31T15:11:17.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:17.743+0000 7f90f4c00640 1 Processor -- start 2024-03-31T15:11:17.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:17.743+0000 7f90f4c00640 1 -- start start 2024-03-31T15:11:17.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:17.743+0000 7f90f4c00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f90f006b620 0x7f90f014dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:17.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:17.743+0000 7f90f4c00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f90f00b60a0 0x7f90f0154a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:17.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:17.743+0000 7f90f4c00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f90f00d0c00 0x7f90f0149700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:17.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:17.743+0000 7f90f4c00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f90f016cb90 con 0x7f90f00b60a0 2024-03-31T15:11:17.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:17.743+0000 7f90f4c00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f90f00b4a40 con 0x7f90f00d0c00 2024-03-31T15:11:17.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:17.743+0000 7f90f4c00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f90f016cd10 con 0x7f90f006b620 2024-03-31T15:11:17.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:17.743+0000 7f90ed400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f90f00b60a0 0x7f90f0154a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:17.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:17.743+0000 7f90ed400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f90f00b60a0 0x7f90f0154a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56130/0 (socket says 172.21.15.16:56130) 2024-03-31T15:11:17.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:17.743+0000 7f90ed400640 1 -- 172.21.15.16:0/3440923135 learned_addr learned my addr 172.21.15.16:0/3440923135 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:11:17.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:17.743+0000 7f90ede00640 1 --2- 172.21.15.16:0/3440923135 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f90f006b620 0x7f90f014dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:17.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:17.743+0000 7f90ee800640 1 --2- 172.21.15.16:0/3440923135 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f90f00d0c00 0x7f90f0149700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:17.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:17.743+0000 7f90ed400640 1 -- 172.21.15.16:0/3440923135 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f90f006b620 msgr2=0x7f90f014dcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:11:17.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:17.743+0000 7f90ed400640 1 --2- 172.21.15.16:0/3440923135 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f90f006b620 0x7f90f014dcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:17.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:17.743+0000 7f90ed400640 1 -- 172.21.15.16:0/3440923135 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f90f00d0c00 msgr2=0x7f90f0149700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:11:17.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:17.743+0000 7f90ed400640 1 --2- 172.21.15.16:0/3440923135 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f90f00d0c00 0x7f90f0149700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:17.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:17.743+0000 7f90ed400640 1 -- 172.21.15.16:0/3440923135 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f90f014fbb0 con 0x7f90f00b60a0 2024-03-31T15:11:17.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:17.743+0000 7f90ed400640 1 --2- 172.21.15.16:0/3440923135 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f90f00b60a0 0x7f90f0154a30 secure :-1 s=READY pgs=1241 cs=0 l=1 rev1=1 crypto rx=0x7f90dc00c730 tx=0x7f90dc00cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:11:17.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:17.743+0000 7f90e6000640 1 -- 172.21.15.16:0/3440923135 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90dc017070 con 0x7f90f00b60a0 2024-03-31T15:11:17.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:17.743+0000 7f90e6000640 1 -- 172.21.15.16:0/3440923135 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f90dc005b00 con 0x7f90f00b60a0 2024-03-31T15:11:17.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:17.743+0000 7f90f4c00640 1 -- 172.21.15.16:0/3440923135 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f90f014fef0 con 0x7f90f00b60a0 2024-03-31T15:11:17.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:17.743+0000 7f90e6000640 1 -- 172.21.15.16:0/3440923135 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90dc007070 con 0x7f90f00b60a0 2024-03-31T15:11:17.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:17.743+0000 7f90f4c00640 1 -- 172.21.15.16:0/3440923135 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f90f0148b60 con 0x7f90f00b60a0 2024-03-31T15:11:17.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:17.747+0000 7f90f4c00640 1 -- 172.21.15.16:0/3440923135 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f90f0069950 con 0x7f90f00b60a0 2024-03-31T15:11:17.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:17.747+0000 7f90e6000640 1 -- 172.21.15.16:0/3440923135 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f90dc01a070 con 0x7f90f00b60a0 2024-03-31T15:11:17.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:17.747+0000 7f90e6000640 1 --2- 172.21.15.16:0/3440923135 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f90d0068380 0x7f90d006a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:17.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:17.747+0000 7f90e6000640 1 -- 172.21.15.16:0/3440923135 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 5824+0+0 (secure 0 0 0) 0x7f90dc085f40 con 0x7f90f00b60a0 2024-03-31T15:11:17.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:17.747+0000 7f90ede00640 1 --2- 172.21.15.16:0/3440923135 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f90d0068380 0x7f90d006a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:17.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:17.747+0000 7f90ede00640 1 --2- 172.21.15.16:0/3440923135 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f90d0068380 0x7f90d006a840 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f90e00ad000 tx=0x7f90e00a6000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:11:17.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:17.751+0000 7f90e6000640 1 -- 172.21.15.16:0/3440923135 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f90f014fef0 con 0x7f90f00b60a0 2024-03-31T15:11:17.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:17.919+0000 7f90f4c00640 1 -- 172.21.15.16:0/3440923135 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f90f006c220 con 0x7f90f00b60a0 2024-03-31T15:11:17.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:17.919+0000 7f90e6000640 1 -- 172.21.15.16:0/3440923135 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v84) ==== 100+0+11 (secure 0 0 0) 0x7f90dc055dc0 con 0x7f90f00b60a0 2024-03-31T15:11:17.914 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 64 2024-03-31T15:11:17.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:17.923+0000 7f90f4c00640 1 -- 172.21.15.16:0/3440923135 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f90d0068380 msgr2=0x7f90d006a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:11:17.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:17.923+0000 7f90f4c00640 1 --2- 172.21.15.16:0/3440923135 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f90d0068380 0x7f90d006a840 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f90e00ad000 tx=0x7f90e00a6000 comp rx=0 tx=0).stop 2024-03-31T15:11:17.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:17.923+0000 7f90f4c00640 1 -- 172.21.15.16:0/3440923135 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f90f00b60a0 msgr2=0x7f90f0154a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:11:17.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:17.923+0000 7f90f4c00640 1 --2- 172.21.15.16:0/3440923135 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f90f00b60a0 0x7f90f0154a30 secure :-1 s=READY pgs=1241 cs=0 l=1 rev1=1 crypto rx=0x7f90dc00c730 tx=0x7f90dc00cbf0 comp rx=0 tx=0).stop 2024-03-31T15:11:17.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:17.923+0000 7f90f4c00640 1 -- 172.21.15.16:0/3440923135 shutdown_connections 2024-03-31T15:11:17.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:17.923+0000 7f90f4c00640 1 --2- 172.21.15.16:0/3440923135 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f90d0068380 0x7f90d006a840 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:17.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:17.923+0000 7f90f4c00640 1 --2- 172.21.15.16:0/3440923135 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f90f00d0c00 0x7f90f0149700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:17.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:17.923+0000 7f90f4c00640 1 --2- 172.21.15.16:0/3440923135 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f90f00b60a0 0x7f90f0154a30 unknown :-1 s=CLOSED pgs=1241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:17.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:17.923+0000 7f90f4c00640 1 --2- 172.21.15.16:0/3440923135 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f90f006b620 0x7f90f014dcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:17.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:17.923+0000 7f90f4c00640 1 -- 172.21.15.16:0/3440923135 >> 172.21.15.16:0/3440923135 conn(0x7f90f006e670 msgr2=0x7f90f015d160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:11:17.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:17.923+0000 7f90f4c00640 1 -- 172.21.15.16:0/3440923135 shutdown_connections 2024-03-31T15:11:17.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:17.923+0000 7f90f4c00640 1 -- 172.21.15.16:0/3440923135 wait complete. 2024-03-31T15:11:17.935 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-31T15:11:18.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:18.083+0000 7fa020800640 1 Processor -- start 2024-03-31T15:11:18.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:18.083+0000 7fa020800640 1 -- start start 2024-03-31T15:11:18.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:18.083+0000 7fa020800640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fa018147280 0x7fa01806c730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:18.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:18.083+0000 7fa020800640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fa018070030 0x7fa0180cc8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:18.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:18.083+0000 7fa020800640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa018147660 con 0x7fa018147dd0 2024-03-31T15:11:18.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:18.083+0000 7fa020800640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7fa0181620f0 con 0x7fa018070030 2024-03-31T15:11:18.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:18.083+0000 7fa020800640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7fa018162340 con 0x7fa018147280 2024-03-31T15:11:18.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:18.087+0000 7fa01d400640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fa018147dd0 0x7fa01806f920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:33672/0 (socket says 172.21.15.16:33672) 2024-03-31T15:11:18.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:18.087+0000 7fa01d400640 1 -- 172.21.15.16:0/4280709465 learned_addr learned my addr 172.21.15.16:0/4280709465 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:11:18.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:18.087+0000 7fa01e800640 1 --2- 172.21.15.16:0/4280709465 >> v2:172.21.15.16:3300/0 conn(0x7fa018070030 0x7fa0180cc8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:18.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:18.087+0000 7fa01de00640 1 --2- 172.21.15.16:0/4280709465 >> v2:172.21.15.89:3300/0 conn(0x7fa018147280 0x7fa01806c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:18.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:18.087+0000 7fa01e800640 1 -- 172.21.15.16:0/4280709465 >> v2:172.21.15.89:3300/0 conn(0x7fa018147280 msgr2=0x7fa01806c730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:11:18.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:18.087+0000 7fa01e800640 1 --2- 172.21.15.16:0/4280709465 >> v2:172.21.15.89:3300/0 conn(0x7fa018147280 0x7fa01806c730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:18.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:18.087+0000 7fa01e800640 1 -- 172.21.15.16:0/4280709465 >> v1:172.21.15.16:6789/0 conn(0x7fa018147dd0 legacy=0x7fa01806f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:11:18.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:18.087+0000 7fa01e800640 1 -- 172.21.15.16:0/4280709465 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa018162620 con 0x7fa018070030 2024-03-31T15:11:18.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:18.087+0000 7fa01ca00640 1 -- 172.21.15.16:0/4280709465 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1138928263 0 0) 0x7fa018147660 con 0x7fa018147dd0 2024-03-31T15:11:18.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:18.087+0000 7fa01e800640 1 --2- 172.21.15.16:0/4280709465 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa018070030 0x7fa0180cc8f0 secure :-1 s=READY pgs=1243 cs=0 l=1 rev1=1 crypto rx=0x7fa018073cf0 tx=0x7fa01401c930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9926548ae29431 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:11:18.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:18.087+0000 7fa01ca00640 1 -- 172.21.15.16:0/4280709465 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0140b2070 con 0x7fa018070030 2024-03-31T15:11:18.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:18.087+0000 7fa01ca00640 1 -- 172.21.15.16:0/4280709465 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa0140a57f0 con 0x7fa018070030 2024-03-31T15:11:18.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:18.087+0000 7fa01ca00640 1 -- 172.21.15.16:0/4280709465 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0140a5bc0 con 0x7fa018070030 2024-03-31T15:11:18.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:18.091+0000 7fa020800640 1 -- 172.21.15.16:0/4280709465 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa018070030 msgr2=0x7fa0180cc8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:11:18.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:18.091+0000 7fa020800640 1 --2- 172.21.15.16:0/4280709465 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa018070030 0x7fa0180cc8f0 secure :-1 s=READY pgs=1243 cs=0 l=1 rev1=1 crypto rx=0x7fa018073cf0 tx=0x7fa01401c930 comp rx=0 tx=0).stop 2024-03-31T15:11:18.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:18.091+0000 7fa020800640 1 -- 172.21.15.16:0/4280709465 shutdown_connections 2024-03-31T15:11:18.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:18.091+0000 7fa020800640 1 --2- 172.21.15.16:0/4280709465 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa018070030 0x7fa0180cc8f0 unknown :-1 s=CLOSED pgs=1243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:18.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:18.091+0000 7fa020800640 1 --2- 172.21.15.16:0/4280709465 >> v2:172.21.15.89:3300/0 conn(0x7fa018147280 0x7fa01806c730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:18.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:18.091+0000 7fa020800640 1 -- 172.21.15.16:0/4280709465 >> 172.21.15.16:0/4280709465 conn(0x7fa018067330 msgr2=0x7fa018067740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:11:18.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:18.091+0000 7fa020800640 1 -- 172.21.15.16:0/4280709465 shutdown_connections 2024-03-31T15:11:18.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:18.091+0000 7fa020800640 1 -- 172.21.15.16:0/4280709465 wait complete. 2024-03-31T15:11:18.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:18.091+0000 7fa020800640 1 Processor -- start 2024-03-31T15:11:18.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:18.091+0000 7fa020800640 1 -- start start 2024-03-31T15:11:18.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:18.091+0000 7fa020800640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fa018070030 0x7fa018168c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:18.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:18.091+0000 7fa020800640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa018147280 0x7fa01812d410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:18.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:18.091+0000 7fa020800640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa018147dd0 0x7fa0181456f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:18.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:18.091+0000 7fa020800640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fa0180cd9d0 con 0x7fa018147280 2024-03-31T15:11:18.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:18.091+0000 7fa020800640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fa0180cd850 con 0x7fa018147dd0 2024-03-31T15:11:18.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:18.091+0000 7fa020800640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fa0180cdb50 con 0x7fa018070030 2024-03-31T15:11:18.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:18.091+0000 7fa01d400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa018147280 0x7fa01812d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:18.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:18.091+0000 7fa01d400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa018147280 0x7fa01812d410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56142/0 (socket says 172.21.15.16:56142) 2024-03-31T15:11:18.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:18.091+0000 7fa01d400640 1 -- 172.21.15.16:0/1078535063 learned_addr learned my addr 172.21.15.16:0/1078535063 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:11:18.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:18.091+0000 7fa01de00640 1 --2- 172.21.15.16:0/1078535063 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fa018070030 0x7fa018168c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:18.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:18.091+0000 7fa01e800640 1 --2- 172.21.15.16:0/1078535063 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa018147dd0 0x7fa0181456f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:18.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:18.091+0000 7fa01d400640 1 -- 172.21.15.16:0/1078535063 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fa018070030 msgr2=0x7fa018168c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:11:18.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:18.091+0000 7fa01d400640 1 --2- 172.21.15.16:0/1078535063 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fa018070030 0x7fa018168c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:18.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:18.091+0000 7fa01d400640 1 -- 172.21.15.16:0/1078535063 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa018147dd0 msgr2=0x7fa0181456f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:11:18.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:18.091+0000 7fa01d400640 1 --2- 172.21.15.16:0/1078535063 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa018147dd0 0x7fa0181456f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:18.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:18.091+0000 7fa01d400640 1 -- 172.21.15.16:0/1078535063 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0180b7970 con 0x7fa018147280 2024-03-31T15:11:18.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:18.095+0000 7fa01d400640 1 --2- 172.21.15.16:0/1078535063 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa018147280 0x7fa01812d410 secure :-1 s=READY pgs=1244 cs=0 l=1 rev1=1 crypto rx=0x7fa0080011f0 tx=0x7fa00800c9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:11:18.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:18.095+0000 7fa012000640 1 -- 172.21.15.16:0/1078535063 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa008018070 con 0x7fa018147280 2024-03-31T15:11:18.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:18.095+0000 7fa012000640 1 -- 172.21.15.16:0/1078535063 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa0080074e0 con 0x7fa018147280 2024-03-31T15:11:18.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:18.095+0000 7fa020800640 1 -- 172.21.15.16:0/1078535063 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa0180b7c50 con 0x7fa018147280 2024-03-31T15:11:18.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:18.095+0000 7fa012000640 1 -- 172.21.15.16:0/1078535063 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa008004a10 con 0x7fa018147280 2024-03-31T15:11:18.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:18.095+0000 7fa020800640 1 -- 172.21.15.16:0/1078535063 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa01812be50 con 0x7fa018147280 2024-03-31T15:11:18.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:18.095+0000 7fa020800640 1 -- 172.21.15.16:0/1078535063 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa018069950 con 0x7fa018147280 2024-03-31T15:11:18.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:18.095+0000 7fa012000640 1 -- 172.21.15.16:0/1078535063 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fa00801b070 con 0x7fa018147280 2024-03-31T15:11:18.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:18.095+0000 7fa012000640 1 --2- 172.21.15.16:0/1078535063 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f9ffc068380 0x7f9ffc06a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:18.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:18.095+0000 7fa012000640 1 -- 172.21.15.16:0/1078535063 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 5824+0+0 (secure 0 0 0) 0x7fa008085d80 con 0x7fa018147280 2024-03-31T15:11:18.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:18.095+0000 7fa01de00640 1 --2- 172.21.15.16:0/1078535063 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f9ffc068380 0x7f9ffc06a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:18.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:18.099+0000 7fa01de00640 1 --2- 172.21.15.16:0/1078535063 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f9ffc068380 0x7f9ffc06a840 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7fa018130530 tx=0x7fa00c009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:11:18.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:18.099+0000 7fa012000640 1 -- 172.21.15.16:0/1078535063 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fa008055c00 con 0x7fa018147280 2024-03-31T15:11:18.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:18.299+0000 7fa020800640 1 -- 172.21.15.16:0/1078535063 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fa0180cf3e0 con 0x7fa018147280 2024-03-31T15:11:18.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:18.299+0000 7fa012000640 1 -- 172.21.15.16:0/1078535063 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 2 entries v84) ==== 80+0+122 (secure 0 0 0) 0x7fa008059f00 con 0x7fa018147280 2024-03-31T15:11:18.293 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:6834/2478530200 2024-04-01T15:10:04.360772+0000 2024-03-31T15:11:18.293 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:6832/2478530200 2024-04-01T15:10:04.360772+0000 2024-03-31T15:11:18.293 INFO:teuthology.orchestra.run.smithi016.stderr:listed 2 entries 2024-03-31T15:11:18.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:18.303+0000 7fa020800640 1 -- 172.21.15.16:0/1078535063 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f9ffc068380 msgr2=0x7f9ffc06a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:11:18.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:18.303+0000 7fa020800640 1 --2- 172.21.15.16:0/1078535063 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f9ffc068380 0x7f9ffc06a840 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7fa018130530 tx=0x7fa00c009190 comp rx=0 tx=0).stop 2024-03-31T15:11:18.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:18.303+0000 7fa020800640 1 -- 172.21.15.16:0/1078535063 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa018147280 msgr2=0x7fa01812d410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:11:18.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:18.303+0000 7fa020800640 1 --2- 172.21.15.16:0/1078535063 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa018147280 0x7fa01812d410 secure :-1 s=READY pgs=1244 cs=0 l=1 rev1=1 crypto rx=0x7fa0080011f0 tx=0x7fa00800c9b0 comp rx=0 tx=0).stop 2024-03-31T15:11:18.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:18.303+0000 7fa020800640 1 -- 172.21.15.16:0/1078535063 shutdown_connections 2024-03-31T15:11:18.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:18.303+0000 7fa020800640 1 --2- 172.21.15.16:0/1078535063 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f9ffc068380 0x7f9ffc06a840 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:18.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:18.303+0000 7fa020800640 1 --2- 172.21.15.16:0/1078535063 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa018147dd0 0x7fa0181456f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:18.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:18.303+0000 7fa020800640 1 --2- 172.21.15.16:0/1078535063 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa018147280 0x7fa01812d410 unknown :-1 s=CLOSED pgs=1244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:18.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:18.303+0000 7fa020800640 1 --2- 172.21.15.16:0/1078535063 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fa018070030 0x7fa018168c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:18.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:18.303+0000 7fa020800640 1 -- 172.21.15.16:0/1078535063 >> 172.21.15.16:0/1078535063 conn(0x7fa018067330 msgr2=0x7fa01806f920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:11:18.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:18.303+0000 7fa020800640 1 -- 172.21.15.16:0/1078535063 shutdown_connections 2024-03-31T15:11:18.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:18.303+0000 7fa020800640 1 -- 172.21.15.16:0/1078535063 wait complete. 2024-03-31T15:11:18.313 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi016.front.sepia.ceph.com... 2024-03-31T15:11:18.314 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:11:18.314 DEBUG:teuthology.orchestra.run.smithi016:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-03-31T15:11:18.331 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi016.stderr:ceph-fuse[45433]: fuse finished with error 0 and tester_r 0 2024-03-31T15:11:18.334 INFO:teuthology.orchestra.run:waiting for 300 2024-03-31T15:11:18.418 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi016.front.sepia.ceph.com 2024-03-31T15:11:18.418 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:11:18.418 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-31T15:11:18.427 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-31T15:11:18.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:18.623+0000 7ff037800640 1 Processor -- start 2024-03-31T15:11:18.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:18.623+0000 7ff037800640 1 -- start start 2024-03-31T15:11:18.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:18.623+0000 7ff037800640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff03015f650 0x7ff03006f920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:18.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:18.623+0000 7ff037800640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7ff03006fe60 0x7ff030162180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:18.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:18.623+0000 7ff037800640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff0300cbe90 con 0x7ff03006dbc0 2024-03-31T15:11:18.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:18.623+0000 7ff037800640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7ff03015f1c0 con 0x7ff03006fe60 2024-03-31T15:11:18.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:18.623+0000 7ff037800640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7ff0300cc170 con 0x7ff03015f650 2024-03-31T15:11:18.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:18.623+0000 7ff034e00640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7ff03006dbc0 0x7ff0300cb440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:33680/0 (socket says 172.21.15.16:33680) 2024-03-31T15:11:18.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:18.623+0000 7ff034e00640 1 -- 172.21.15.16:0/2521705209 learned_addr learned my addr 172.21.15.16:0/2521705209 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:11:18.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:18.627+0000 7ff02fe00640 1 --2- 172.21.15.16:0/2521705209 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff03015f650 0x7ff03006f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:18.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:18.627+0000 7ff035800640 1 --2- 172.21.15.16:0/2521705209 >> v2:172.21.15.89:3300/0 conn(0x7ff03006fe60 0x7ff030162180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:18.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:18.627+0000 7ff02f400640 1 -- 172.21.15.16:0/2521705209 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 480104578 0 0) 0x7ff0300cbe90 con 0x7ff03006dbc0 2024-03-31T15:11:18.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:18.627+0000 7ff02f400640 1 -- 172.21.15.16:0/2521705209 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff018003530 con 0x7ff03006dbc0 2024-03-31T15:11:18.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:18.627+0000 7ff02fe00640 1 -- 172.21.15.16:0/2521705209 >> v2:172.21.15.89:3300/0 conn(0x7ff03006fe60 msgr2=0x7ff030162180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:11:18.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:18.627+0000 7ff02fe00640 1 --2- 172.21.15.16:0/2521705209 >> v2:172.21.15.89:3300/0 conn(0x7ff03006fe60 0x7ff030162180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:18.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:18.627+0000 7ff02f400640 1 -- 172.21.15.16:0/2521705209 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 360906442 0 0) 0x7ff018003530 con 0x7ff03006dbc0 2024-03-31T15:11:18.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:18.627+0000 7ff02fe00640 1 -- 172.21.15.16:0/2521705209 >> v1:172.21.15.16:6789/0 conn(0x7ff03006dbc0 legacy=0x7ff0300cb440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:11:18.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:18.627+0000 7ff02fe00640 1 -- 172.21.15.16:0/2521705209 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff03015ee50 con 0x7ff03015f650 2024-03-31T15:11:18.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:18.627+0000 7ff02fe00640 1 --2- 172.21.15.16:0/2521705209 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff03015f650 0x7ff03006f920 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7ff01c0097c0 tx=0x7ff01c01c330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4f1c7a4ff0d3a2df server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:11:18.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:18.631+0000 7ff02f400640 1 -- 172.21.15.16:0/2521705209 <== mon.2 v2:172.21.15.89:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff01c0b0070 con 0x7ff03015f650 2024-03-31T15:11:18.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:18.631+0000 7ff02f400640 1 -- 172.21.15.16:0/2521705209 <== mon.2 v2:172.21.15.89:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff01c0ab400 con 0x7ff03015f650 2024-03-31T15:11:18.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:18.631+0000 7ff02f400640 1 -- 172.21.15.16:0/2521705209 <== mon.2 v2:172.21.15.89:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff024001680 con 0x7ff03015f650 2024-03-31T15:11:18.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:18.631+0000 7ff037800640 1 -- 172.21.15.16:0/2521705209 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff03015f650 msgr2=0x7ff03006f920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:11:18.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:18.631+0000 7ff037800640 1 --2- 172.21.15.16:0/2521705209 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff03015f650 0x7ff03006f920 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7ff01c0097c0 tx=0x7ff01c01c330 comp rx=0 tx=0).stop 2024-03-31T15:11:18.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:18.631+0000 7ff037800640 1 -- 172.21.15.16:0/2521705209 shutdown_connections 2024-03-31T15:11:18.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:18.631+0000 7ff037800640 1 --2- 172.21.15.16:0/2521705209 >> v2:172.21.15.89:3300/0 conn(0x7ff03006fe60 0x7ff030162180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:18.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:18.631+0000 7ff037800640 1 --2- 172.21.15.16:0/2521705209 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff03015f650 0x7ff03006f920 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:18.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:18.631+0000 7ff037800640 1 -- 172.21.15.16:0/2521705209 >> 172.21.15.16:0/2521705209 conn(0x7ff030067330 msgr2=0x7ff030067740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:11:18.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:18.631+0000 7ff037800640 1 -- 172.21.15.16:0/2521705209 shutdown_connections 2024-03-31T15:11:18.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:18.631+0000 7ff037800640 1 -- 172.21.15.16:0/2521705209 wait complete. 2024-03-31T15:11:18.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:18.631+0000 7ff037800640 1 Processor -- start 2024-03-31T15:11:18.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:18.631+0000 7ff037800640 1 -- start start 2024-03-31T15:11:18.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:18.631+0000 7ff037800640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff03006dbc0 0x7ff030140780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:18.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:18.631+0000 7ff037800640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff03006fe60 0x7ff03015bec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:18.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:18.631+0000 7ff037800640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff03015f650 0x7ff030156340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:18.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:18.631+0000 7ff037800640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7ff03006c950 con 0x7ff03006dbc0 2024-03-31T15:11:18.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:18.631+0000 7ff037800640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7ff03006dfa0 con 0x7ff03015f650 2024-03-31T15:11:18.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:18.631+0000 7ff037800640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7ff03006cad0 con 0x7ff03006fe60 2024-03-31T15:11:18.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:18.631+0000 7ff034e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff03006dbc0 0x7ff030140780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:18.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:18.631+0000 7ff034e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff03006dbc0 0x7ff030140780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56152/0 (socket says 172.21.15.16:56152) 2024-03-31T15:11:18.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:18.631+0000 7ff034e00640 1 -- 172.21.15.16:0/2944812934 learned_addr learned my addr 172.21.15.16:0/2944812934 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:11:18.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:18.631+0000 7ff02fe00640 1 --2- 172.21.15.16:0/2944812934 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff03006fe60 0x7ff03015bec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:18.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:18.631+0000 7ff035800640 1 --2- 172.21.15.16:0/2944812934 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff03015f650 0x7ff030156340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:18.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:18.635+0000 7ff034e00640 1 -- 172.21.15.16:0/2944812934 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff03006fe60 msgr2=0x7ff03015bec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:11:18.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:18.635+0000 7ff034e00640 1 --2- 172.21.15.16:0/2944812934 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff03006fe60 0x7ff03015bec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:18.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:18.635+0000 7ff034e00640 1 -- 172.21.15.16:0/2944812934 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff03015f650 msgr2=0x7ff030156340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:11:18.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:18.635+0000 7ff034e00640 1 --2- 172.21.15.16:0/2944812934 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff03015f650 0x7ff030156340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:18.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:18.635+0000 7ff034e00640 1 -- 172.21.15.16:0/2944812934 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff030158300 con 0x7ff03006dbc0 2024-03-31T15:11:18.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:18.635+0000 7ff035800640 1 --2- 172.21.15.16:0/2944812934 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff03015f650 0x7ff030156340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:11:18.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:18.635+0000 7ff034e00640 1 --2- 172.21.15.16:0/2944812934 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff03006dbc0 0x7ff030140780 secure :-1 s=READY pgs=1246 cs=0 l=1 rev1=1 crypto rx=0x7ff02400c700 tx=0x7ff02400cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:11:18.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:18.635+0000 7ff02cc00640 1 -- 172.21.15.16:0/2944812934 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff024017070 con 0x7ff03006dbc0 2024-03-31T15:11:18.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:18.635+0000 7ff02cc00640 1 -- 172.21.15.16:0/2944812934 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff024005ae0 con 0x7ff03006dbc0 2024-03-31T15:11:18.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:18.635+0000 7ff02cc00640 1 -- 172.21.15.16:0/2944812934 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff024007070 con 0x7ff03006dbc0 2024-03-31T15:11:18.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:18.635+0000 7ff037800640 1 -- 172.21.15.16:0/2944812934 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff0301585e0 con 0x7ff03006dbc0 2024-03-31T15:11:18.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:18.635+0000 7ff037800640 1 -- 172.21.15.16:0/2944812934 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff030159890 con 0x7ff03006dbc0 2024-03-31T15:11:18.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:18.635+0000 7ff037800640 1 -- 172.21.15.16:0/2944812934 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff030069950 con 0x7ff03006dbc0 2024-03-31T15:11:18.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:18.635+0000 7ff02cc00640 1 -- 172.21.15.16:0/2944812934 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7ff02401a070 con 0x7ff03006dbc0 2024-03-31T15:11:18.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:18.635+0000 7ff02cc00640 1 --2- 172.21.15.16:0/2944812934 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7ff010068380 0x7ff01006a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:18.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:18.635+0000 7ff02cc00640 1 -- 172.21.15.16:0/2944812934 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 5824+0+0 (secure 0 0 0) 0x7ff024085e30 con 0x7ff03006dbc0 2024-03-31T15:11:18.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:18.639+0000 7ff02fe00640 1 --2- 172.21.15.16:0/2944812934 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7ff010068380 0x7ff01006a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:18.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:18.639+0000 7ff02fe00640 1 --2- 172.21.15.16:0/2944812934 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7ff010068380 0x7ff01006a840 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7ff01c0ad000 tx=0x7ff01c0a6000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:11:18.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:18.643+0000 7ff02cc00640 1 -- 172.21.15.16:0/2944812934 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7ff024055d40 con 0x7ff03006dbc0 2024-03-31T15:11:18.835 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:18.843+0000 7ff037800640 1 -- 172.21.15.16:0/2944812934 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff0300cf380 con 0x7ff03006dbc0 2024-03-31T15:11:18.836 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:18.843+0000 7ff02cc00640 1 -- 172.21.15.16:0/2944812934 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 68 v68) ==== 94+0+5181 (secure 0 0 0) 0x7ff02405a040 con 0x7ff03006dbc0 2024-03-31T15:11:18.836 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:11:18.836 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":68,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6494,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/2445558494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6835","nonce":2445558494},{"type":"v1","addr":"172.21.15.89:6837","nonce":2445558494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":56},{"gid":7270,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/3185913858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6833","nonce":3185913858},{"type":"v1","addr":"172.21.15.16:6835","nonce":3185913858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":56},{"gid":7279,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6836/1599468493","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":1599468493},{"type":"v1","addr":"172.21.15.89:6836","nonce":1599468493}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":56}],"filesystems":[{"mdsmap":{"epoch":68,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-31T15:09:50.926196+0000","modified":"2024-03-31T15:10:15.370488+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":82,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7749},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7749":{"gid":7749,"name":"c","rank":0,"incarnation":64,"state":"up:active","state_seq":6,"addr":"172.21.15.16:6834/818074038","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":818074038},{"type":"v1","addr":"172.21.15.16:6834","nonce":818074038}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7749,"qdb_cluster":[7749]},"id":6}]} 2024-03-31T15:11:18.836 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 68 2024-03-31T15:11:18.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:18.843+0000 7ff037800640 1 -- 172.21.15.16:0/2944812934 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7ff010068380 msgr2=0x7ff01006a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:11:18.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:18.843+0000 7ff037800640 1 --2- 172.21.15.16:0/2944812934 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7ff010068380 0x7ff01006a840 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7ff01c0ad000 tx=0x7ff01c0a6000 comp rx=0 tx=0).stop 2024-03-31T15:11:18.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:18.843+0000 7ff037800640 1 -- 172.21.15.16:0/2944812934 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff03006dbc0 msgr2=0x7ff030140780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:11:18.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:18.843+0000 7ff037800640 1 --2- 172.21.15.16:0/2944812934 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff03006dbc0 0x7ff030140780 secure :-1 s=READY pgs=1246 cs=0 l=1 rev1=1 crypto rx=0x7ff02400c700 tx=0x7ff02400cbc0 comp rx=0 tx=0).stop 2024-03-31T15:11:18.839 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:18.843+0000 7ff037800640 1 -- 172.21.15.16:0/2944812934 shutdown_connections 2024-03-31T15:11:18.839 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:18.843+0000 7ff037800640 1 --2- 172.21.15.16:0/2944812934 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7ff010068380 0x7ff01006a840 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:18.839 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:18.843+0000 7ff037800640 1 --2- 172.21.15.16:0/2944812934 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff03015f650 0x7ff030156340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:18.839 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:18.843+0000 7ff037800640 1 --2- 172.21.15.16:0/2944812934 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff03006fe60 0x7ff03015bec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:18.839 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:18.843+0000 7ff037800640 1 --2- 172.21.15.16:0/2944812934 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff03006dbc0 0x7ff030140780 unknown :-1 s=CLOSED pgs=1246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:18.839 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:18.843+0000 7ff037800640 1 -- 172.21.15.16:0/2944812934 >> 172.21.15.16:0/2944812934 conn(0x7ff030067330 msgr2=0x7ff0300cd310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:11:18.839 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:18.843+0000 7ff037800640 1 -- 172.21.15.16:0/2944812934 shutdown_connections 2024-03-31T15:11:18.839 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:18.847+0000 7ff037800640 1 -- 172.21.15.16:0/2944812934 wait complete. 2024-03-31T15:11:18.860 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json flush journal 2024-03-31T15:11:19.033 INFO:teuthology.orchestra.run.smithi016.stdout:{"message":"","return_code":0} 2024-03-31T15:11:19.033 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2024-03-31T15:11:19.033 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-31T15:11:19.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.179+0000 7f73c8400640 1 Processor -- start 2024-03-31T15:11:19.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.179+0000 7f73c8400640 1 -- start start 2024-03-31T15:11:19.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.183+0000 7f73c8400640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f73c006dbc0 0x7f73c015f650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:19.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.183+0000 7f73c8400640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f73c00cb440 0x7f73c006f920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:19.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.183+0000 7f73c8400640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f73c00cbe90 con 0x7f73c00cb8c0 2024-03-31T15:11:19.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.183+0000 7f73c8400640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f73c015fb90 con 0x7f73c00cb440 2024-03-31T15:11:19.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.183+0000 7f73c8400640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f73c015f1c0 con 0x7f73c006dbc0 2024-03-31T15:11:19.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.183+0000 7f73c5000640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f73c00cb440 0x7f73c006f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:19.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.183+0000 7f73c6400640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f73c00cb8c0 0x7f73c0162180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.16:41374/0 (socket says 172.21.15.16:41374) 2024-03-31T15:11:19.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.183+0000 7f73c6400640 1 -- 172.21.15.16:0/1273446973 learned_addr learned my addr 172.21.15.16:0/1273446973 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:11:19.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.183+0000 7f73c5a00640 1 --2- 172.21.15.16:0/1273446973 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f73c006dbc0 0x7f73c015f650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:19.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.183+0000 7f73bfe00640 1 -- 172.21.15.16:0/1273446973 <== mon.1 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3142993384 0 0) 0x7f73c00cbe90 con 0x7f73c00cb8c0 2024-03-31T15:11:19.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.183+0000 7f73bfe00640 1 -- 172.21.15.16:0/1273446973 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f73a8003530 con 0x7f73c00cb8c0 2024-03-31T15:11:19.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.183+0000 7f73c5000640 1 -- 172.21.15.16:0/1273446973 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f73c006dbc0 msgr2=0x7f73c015f650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:11:19.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.183+0000 7f73c5000640 1 --2- 172.21.15.16:0/1273446973 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f73c006dbc0 0x7f73c015f650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:19.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.183+0000 7f73c5000640 1 -- 172.21.15.16:0/1273446973 >> v1:172.21.15.89:6789/0 conn(0x7f73c00cb8c0 legacy=0x7f73c0162180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:11:19.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.183+0000 7f73c5000640 1 -- 172.21.15.16:0/1273446973 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73c00cc170 con 0x7f73c00cb440 2024-03-31T15:11:19.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.187+0000 7f73c5000640 1 --2- 172.21.15.16:0/1273446973 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f73c00cb440 0x7f73c006f920 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f73ac0097c0 tx=0x7f73ac01c3e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3e0fbb397d5b4e14 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:11:19.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.187+0000 7f73bfe00640 1 -- 172.21.15.16:0/1273446973 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73ac0ab600 con 0x7f73c00cb440 2024-03-31T15:11:19.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.187+0000 7f73bfe00640 1 -- 172.21.15.16:0/1273446973 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f73ac0b0070 con 0x7f73c00cb440 2024-03-31T15:11:19.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.187+0000 7f73bfe00640 1 -- 172.21.15.16:0/1273446973 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73ac0ab9c0 con 0x7f73c00cb440 2024-03-31T15:11:19.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.187+0000 7f73c8400640 1 -- 172.21.15.16:0/1273446973 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f73c00cb440 msgr2=0x7f73c006f920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:11:19.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.187+0000 7f73c8400640 1 --2- 172.21.15.16:0/1273446973 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f73c00cb440 0x7f73c006f920 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f73ac0097c0 tx=0x7f73ac01c3e0 comp rx=0 tx=0).stop 2024-03-31T15:11:19.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.187+0000 7f73c8400640 1 -- 172.21.15.16:0/1273446973 shutdown_connections 2024-03-31T15:11:19.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.187+0000 7f73c8400640 1 --2- 172.21.15.16:0/1273446973 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f73c00cb440 0x7f73c006f920 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:19.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.187+0000 7f73c8400640 1 --2- 172.21.15.16:0/1273446973 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f73c006dbc0 0x7f73c015f650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:19.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.187+0000 7f73c8400640 1 -- 172.21.15.16:0/1273446973 >> 172.21.15.16:0/1273446973 conn(0x7f73c0067330 msgr2=0x7f73c0067740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:11:19.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.187+0000 7f73c8400640 1 -- 172.21.15.16:0/1273446973 shutdown_connections 2024-03-31T15:11:19.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.187+0000 7f73c8400640 1 -- 172.21.15.16:0/1273446973 wait complete. 2024-03-31T15:11:19.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.187+0000 7f73c8400640 1 Processor -- start 2024-03-31T15:11:19.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.187+0000 7f73c8400640 1 -- start start 2024-03-31T15:11:19.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.191+0000 7f73c8400640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f73c006dbc0 0x7f73c00b6c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:19.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.191+0000 7f73c8400640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f73c00cb440 0x7f73c015a4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:19.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.191+0000 7f73c8400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f73c00cb8c0 0x7f73c015aa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:19.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.191+0000 7f73c8400640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f73c0162ca0 con 0x7f73c00cb8c0 2024-03-31T15:11:19.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.191+0000 7f73c8400640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f73c0070270 con 0x7f73c00cb440 2024-03-31T15:11:19.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.191+0000 7f73c8400640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f73c006c950 con 0x7f73c006dbc0 2024-03-31T15:11:19.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.191+0000 7f73c6400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f73c00cb8c0 0x7f73c015aa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:19.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.191+0000 7f73c6400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f73c00cb8c0 0x7f73c015aa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56164/0 (socket says 172.21.15.16:56164) 2024-03-31T15:11:19.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.191+0000 7f73c6400640 1 -- 172.21.15.16:0/1727845348 learned_addr learned my addr 172.21.15.16:0/1727845348 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:11:19.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.191+0000 7f73c5000640 1 --2- 172.21.15.16:0/1727845348 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f73c00cb440 0x7f73c015a4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:19.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.191+0000 7f73c5a00640 1 --2- 172.21.15.16:0/1727845348 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f73c006dbc0 0x7f73c00b6c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:19.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.191+0000 7f73c6400640 1 -- 172.21.15.16:0/1727845348 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f73c006dbc0 msgr2=0x7f73c00b6c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:11:19.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.191+0000 7f73c6400640 1 --2- 172.21.15.16:0/1727845348 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f73c006dbc0 0x7f73c00b6c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:19.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.191+0000 7f73c6400640 1 -- 172.21.15.16:0/1727845348 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f73c00cb440 msgr2=0x7f73c015a4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:11:19.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.191+0000 7f73c6400640 1 --2- 172.21.15.16:0/1727845348 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f73c00cb440 0x7f73c015a4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:19.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.191+0000 7f73c6400640 1 -- 172.21.15.16:0/1727845348 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73c015b180 con 0x7f73c00cb8c0 2024-03-31T15:11:19.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.191+0000 7f73c5000640 1 --2- 172.21.15.16:0/1727845348 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f73c00cb440 0x7f73c015a4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:11:19.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.191+0000 7f73c6400640 1 --2- 172.21.15.16:0/1727845348 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f73c00cb8c0 0x7f73c015aa20 secure :-1 s=READY pgs=1247 cs=0 l=1 rev1=1 crypto rx=0x7f73b800caa0 tx=0x7f73b800cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:11:19.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.191+0000 7f73bd600640 1 -- 172.21.15.16:0/1727845348 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73b8017070 con 0x7f73c00cb8c0 2024-03-31T15:11:19.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.191+0000 7f73bd600640 1 -- 172.21.15.16:0/1727845348 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f73b800ac00 con 0x7f73c00cb8c0 2024-03-31T15:11:19.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.191+0000 7f73bd600640 1 -- 172.21.15.16:0/1727845348 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73b8012400 con 0x7f73c00cb8c0 2024-03-31T15:11:19.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.191+0000 7f73c8400640 1 -- 172.21.15.16:0/1727845348 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f73c015b460 con 0x7f73c00cb8c0 2024-03-31T15:11:19.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.191+0000 7f73c8400640 1 -- 172.21.15.16:0/1727845348 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f73c015ba20 con 0x7f73c00cb8c0 2024-03-31T15:11:19.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.191+0000 7f73c8400640 1 -- 172.21.15.16:0/1727845348 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f73c0069950 con 0x7f73c00cb8c0 2024-03-31T15:11:19.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.195+0000 7f73bd600640 1 -- 172.21.15.16:0/1727845348 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f73b801a070 con 0x7f73c00cb8c0 2024-03-31T15:11:19.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.195+0000 7f73bd600640 1 --2- 172.21.15.16:0/1727845348 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f73a00683b0 0x7f73a006a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:19.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.195+0000 7f73bd600640 1 -- 172.21.15.16:0/1727845348 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 5824+0+0 (secure 0 0 0) 0x7f73b80868d0 con 0x7f73c00cb8c0 2024-03-31T15:11:19.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.195+0000 7f73c5a00640 1 --2- 172.21.15.16:0/1727845348 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f73a00683b0 0x7f73a006a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:19.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.195+0000 7f73c5a00640 1 --2- 172.21.15.16:0/1727845348 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f73a00683b0 0x7f73a006a870 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f73c00cde70 tx=0x7f73b4008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:11:19.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.199+0000 7f73bd600640 1 -- 172.21.15.16:0/1727845348 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f73b8056d20 con 0x7f73c00cb8c0 2024-03-31T15:11:19.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.399+0000 7f73c8400640 1 -- 172.21.15.16:0/1727845348 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f73c00d1700 con 0x7f73c00cb8c0 2024-03-31T15:11:19.391 INFO:tasks.ceph.mon.a.smithi016.stderr:2024-03-31T15:11:19.399+0000 7f7dcbc00640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-31T15:11:19.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.399+0000 7f73bd600640 1 -- 172.21.15.16:0/1727845348 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v69) ==== 161+0+0 (secure 0 0 0) 0x7f73b8056620 con 0x7f73c00cb8c0 2024-03-31T15:11:19.394 INFO:teuthology.orchestra.run.smithi016.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-31T15:11:19.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.403+0000 7f73c8400640 1 -- 172.21.15.16:0/1727845348 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f73a00683b0 msgr2=0x7f73a006a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:11:19.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.403+0000 7f73c8400640 1 --2- 172.21.15.16:0/1727845348 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f73a00683b0 0x7f73a006a870 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f73c00cde70 tx=0x7f73b4008040 comp rx=0 tx=0).stop 2024-03-31T15:11:19.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.403+0000 7f73c8400640 1 -- 172.21.15.16:0/1727845348 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f73c00cb8c0 msgr2=0x7f73c015aa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:11:19.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.403+0000 7f73c8400640 1 --2- 172.21.15.16:0/1727845348 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f73c00cb8c0 0x7f73c015aa20 secure :-1 s=READY pgs=1247 cs=0 l=1 rev1=1 crypto rx=0x7f73b800caa0 tx=0x7f73b800cf60 comp rx=0 tx=0).stop 2024-03-31T15:11:19.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.403+0000 7f73c8400640 1 -- 172.21.15.16:0/1727845348 shutdown_connections 2024-03-31T15:11:19.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.403+0000 7f73c8400640 1 --2- 172.21.15.16:0/1727845348 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f73a00683b0 0x7f73a006a870 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:19.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.403+0000 7f73c8400640 1 --2- 172.21.15.16:0/1727845348 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f73c00cb8c0 0x7f73c015aa20 unknown :-1 s=CLOSED pgs=1247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:19.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.403+0000 7f73c8400640 1 --2- 172.21.15.16:0/1727845348 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f73c00cb440 0x7f73c015a4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:19.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.403+0000 7f73c8400640 1 --2- 172.21.15.16:0/1727845348 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f73c006dbc0 0x7f73c00b6c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:19.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.403+0000 7f73c8400640 1 -- 172.21.15.16:0/1727845348 >> 172.21.15.16:0/1727845348 conn(0x7f73c0067330 msgr2=0x7f73c00d07c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:11:19.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.403+0000 7f73c8400640 1 -- 172.21.15.16:0/1727845348 shutdown_connections 2024-03-31T15:11:19.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.403+0000 7f73c8400640 1 -- 172.21.15.16:0/1727845348 wait complete. 2024-03-31T15:11:19.407 INFO:tasks.ceph.mds.c.smithi016.stderr: -4478> 2024-03-31T15:10:11.372+0000 7f8537000640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-31T15:11:19.407 INFO:tasks.ceph.mds.c.smithi016.stderr: -4275> 2024-03-31T15:10:11.680+0000 7f8532000640 -1 log_channel(cluster) log [ERR] : dir 0x10000000000 object missing on disk; some files may be lost (/dir) 2024-03-31T15:11:19.408 INFO:tasks.ceph.mds.c.smithi016.stderr: -4109> 2024-03-31T15:10:11.916+0000 7f8532000640 -1 mds.0.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000000 [...2,head] /dir/ auth v13 ap=1 f(v0 m2024-03-31T15:09:59.315089+0000 1=1+0) n(v0 rc2024-03-31T15:09:59.431625+0000 2=1+1) | dirfrag=0 authpin=1 scrubqueue=0 0x55f996656100]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":-2,"ondisk_value":"(-1)0x0:[]//[]","memoryvalue":"(12)0x10000000000:[<0x1/dir v13>]//[]","error_str":"failed to decode on-disk backtrace (0 bytes)!"},"raw_stats":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n(v0 1=0+1)","memory_value.dirstat":"f(v0 m2024-03-31T15:09:59.315089+0000 1=1+0)","memory_value.rstat":"n(v0 rc2024-03-31T15:09:59.431625+0000 2=1+1)","error_str":"freshly-calculated rstats don't match existing ones"},"return_code":-2} 2024-03-31T15:11:19.412 INFO:tasks.ceph.mds.c.smithi016.stderr: -2323> 2024-03-31T15:10:43.632+0000 7f8532000640 -1 mds.0.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000000 [...2,head] /dir/ auth v13 pv16 ap=3 DIRTYPARENT f(v0 m2024-03-31T15:09:59.315089+0000 1=1+0)->f(v1 m2024-03-31T15:09:59.315089+0000) n(v0 rc2024-03-31T15:09:59.431625+0000 2=1+1)->n(v0 rc2024-03-31T15:09:59.431625+0000 2=1+1) (inest lock dirty) (ifile lock->sync w=1 flushing) | dirtyscattered=2 lock=1 dirfrag=1 dirtyrstat=1 dirtyparent=1 waiter=1 authpin=1 scrubqueue=0 0x55f996656100]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":-2,"ondisk_value":"(-1)0x0:[]//[]","memoryvalue":"(12)0x10000000000:[<0x1/dir v13>]//[]","error_str":"failed to decode on-disk backtrace (0 bytes)!"},"raw_stats":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n(v0 1=0+1)","memory_value.dirstat":"f(v0 m2024-03-31T15:09:59.315089+0000 1=1+0)","memory_value.rstat":"n(v0 rc2024-03-31T15:09:59.431625+0000 2=1+1)","error_str":"freshly-calculated rstats don't match existing ones (will be fixed)"},"return_code":-2} 2024-03-31T15:11:19.421 DEBUG:teuthology.orchestra.run.smithi016:> cephfs-data-scan --debug-mds=20 --debug-ms=1 --debug-objecter=1 scan_extents cephfs_data 2024-03-31T15:11:19.455 INFO:tasks.ceph.mds.c.smithi016.stdout:starting mds.c at 2024-03-31T15:11:19.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.475+0000 7f586be4b8c0 1 -- start start 2024-03-31T15:11:19.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.475+0000 7f586be4b8c0 1 Processor -- start 2024-03-31T15:11:19.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.475+0000 7f586be4b8c0 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d9d7e55800 0x55d9d8c0c000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:19.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.475+0000 7f586be4b8c0 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d9d7e55400 0x55d9d8c0c580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:19.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.475+0000 7f586be4b8c0 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55d9d7e55000 0x55d9d8c0cb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:19.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.475+0000 7f586be4b8c0 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55d9d8c28180 con 0x55d9d7e55800 2024-03-31T15:11:19.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.475+0000 7f586be4b8c0 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55d9d8c28300 con 0x55d9d7e55400 2024-03-31T15:11:19.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.475+0000 7f586be4b8c0 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x55d9d8c28480 con 0x55d9d7e55000 2024-03-31T15:11:19.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.475+0000 7f586a000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d9d7e55800 0x55d9d8c0c000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:19.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.475+0000 7f586a000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d9d7e55800 0x55d9d8c0c000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56168/0 (socket says 172.21.15.16:56168) 2024-03-31T15:11:19.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.475+0000 7f586a000640 1 -- 172.21.15.16:0/2331561691 learned_addr learned my addr 172.21.15.16:0/2331561691 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:11:19.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.475+0000 7f5869600640 1 --2- 172.21.15.16:0/2331561691 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d9d7e55400 0x55d9d8c0c580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:19.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.475+0000 7f586aa00640 1 --2- 172.21.15.16:0/2331561691 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55d9d7e55000 0x55d9d8c0cb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:19.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.475+0000 7f586a000640 1 -- 172.21.15.16:0/2331561691 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55d9d7e55000 msgr2=0x55d9d8c0cb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:11:19.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.475+0000 7f586a000640 1 --2- 172.21.15.16:0/2331561691 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55d9d7e55000 0x55d9d8c0cb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:19.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.475+0000 7f586a000640 1 -- 172.21.15.16:0/2331561691 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d9d7e55400 msgr2=0x55d9d8c0c580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:11:19.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.475+0000 7f586a000640 1 --2- 172.21.15.16:0/2331561691 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d9d7e55400 0x55d9d8c0c580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:19.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.475+0000 7f586a000640 1 -- 172.21.15.16:0/2331561691 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d9d7e7a1e0 con 0x55d9d7e55800 2024-03-31T15:11:19.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.475+0000 7f586a000640 1 --2- 172.21.15.16:0/2331561691 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d9d7e55800 0x55d9d8c0c000 secure :-1 s=READY pgs=1250 cs=0 l=1 rev1=1 crypto rx=0x55d9d8cbb8f0 tx=0x55d9d8bdca80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7bc51f34a5f4eb86 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:11:19.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.475+0000 7f5866e00640 1 -- 172.21.15.16:0/2331561691 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x55d9d7e8a4e0 con 0x55d9d7e55800 2024-03-31T15:11:19.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.475+0000 7f5866e00640 1 -- 172.21.15.16:0/2331561691 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x55d9d7ea01c0 con 0x55d9d7e55800 2024-03-31T15:11:19.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.475+0000 7f5866e00640 1 -- 172.21.15.16:0/2331561691 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x55d9d7e8ab60 con 0x55d9d7e55800 2024-03-31T15:11:19.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.475+0000 7f586be4b8c0 1 -- 172.21.15.16:0/2331561691 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d9d7e7b4a0 con 0x55d9d7e55800 2024-03-31T15:11:19.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.475+0000 7f586be4b8c0 1 -- 172.21.15.16:0/2331561691 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({fsmap=0}) -- 0x55d9d7e7ba40 con 0x55d9d7e55800 2024-03-31T15:11:19.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.475+0000 7f586be4b8c0 4 waiting for MDS map... 2024-03-31T15:11:19.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.475+0000 7f5866e00640 1 -- 172.21.15.16:0/2331561691 <== mon.0 v2:172.21.15.16:3300/0 4 ==== osd_map(85..85 src has 1..85) ==== 5910+0+0 (secure 0 0 0) 0x55d9d8d78600 con 0x55d9d7e55800 2024-03-31T15:11:19.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.475+0000 7f5866e00640 1 -- 172.21.15.16:0/2331561691 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 69) ==== 2885+0+0 (secure 0 0 0) 0x55d9d7eab180 con 0x55d9d7e55800 2024-03-31T15:11:19.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.475+0000 7f586be4b8c0 4 Got MDS map 69 2024-03-31T15:11:19.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.479+0000 7f586be4b8c0 4 datascan.main: Using metadata pool output 2024-03-31T15:11:19.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.479+0000 7f586be4b8c0 4 datascan.main: connecting to RADOS... 2024-03-31T15:11:19.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.479+0000 7f586be4b8c0 1 Processor -- start 2024-03-31T15:11:19.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.479+0000 7f586be4b8c0 1 -- start start 2024-03-31T15:11:19.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.479+0000 7f586be4b8c0 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d9d8d73000 0x55d9d8c0d080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:19.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.479+0000 7f586be4b8c0 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55d9d8d73400 0x55d9d8c0d600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:19.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.479+0000 7f586be4b8c0 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d9d8d73800 0x55d9d8c0db80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:19.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.479+0000 7f586be4b8c0 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55d9d8c28600 con 0x55d9d8d73800 2024-03-31T15:11:19.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.479+0000 7f586be4b8c0 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55d9d8c28780 con 0x55d9d8d73000 2024-03-31T15:11:19.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.479+0000 7f586be4b8c0 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x55d9d8c28900 con 0x55d9d8d73400 2024-03-31T15:11:19.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.479+0000 7f5869600640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d9d8d73800 0x55d9d8c0db80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:19.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.479+0000 7f586aa00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d9d8d73000 0x55d9d8c0d080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:19.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.479+0000 7f586aa00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d9d8d73000 0x55d9d8c0d080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.16:34700/0 (socket says 172.21.15.16:34700) 2024-03-31T15:11:19.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.479+0000 7f5869600640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d9d8d73800 0x55d9d8c0db80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56180/0 (socket says 172.21.15.16:56180) 2024-03-31T15:11:19.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.479+0000 7f586a000640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55d9d8d73400 0x55d9d8c0d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:19.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.479+0000 7f586aa00640 1 -- 172.21.15.16:0/980142147 learned_addr learned my addr 172.21.15.16:0/980142147 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:11:19.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.479+0000 7f586aa00640 1 -- 172.21.15.16:0/980142147 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55d9d8d73400 msgr2=0x55d9d8c0d600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:11:19.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.479+0000 7f586aa00640 1 --2- 172.21.15.16:0/980142147 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55d9d8d73400 0x55d9d8c0d600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:19.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.479+0000 7f586aa00640 1 -- 172.21.15.16:0/980142147 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d9d8d73800 msgr2=0x55d9d8c0db80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:11:19.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.479+0000 7f586aa00640 1 --2- 172.21.15.16:0/980142147 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d9d8d73800 0x55d9d8c0db80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:19.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.479+0000 7f586aa00640 1 -- 172.21.15.16:0/980142147 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d9d7e7bc20 con 0x55d9d8d73000 2024-03-31T15:11:19.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.479+0000 7f5869600640 1 --2- 172.21.15.16:0/980142147 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d9d8d73800 0x55d9d8c0db80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:11:19.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.479+0000 7f586aa00640 1 --2- 172.21.15.16:0/980142147 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d9d8d73000 0x55d9d8c0d080 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x55d9d8d74c30 tx=0x55d9d814c000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=508664770c434898 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:11:19.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.479+0000 7f5864600640 1 -- 172.21.15.16:0/980142147 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x55d9d7e8a4e0 con 0x55d9d8d73000 2024-03-31T15:11:19.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.479+0000 7f5864600640 1 -- 172.21.15.16:0/980142147 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x55d9d7ea0e00 con 0x55d9d8d73000 2024-03-31T15:11:19.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.479+0000 7f5864600640 1 -- 172.21.15.16:0/980142147 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x55d9d7e8ab60 con 0x55d9d8d73000 2024-03-31T15:11:19.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.479+0000 7f586be4b8c0 1 -- 172.21.15.16:0/980142147 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d9d8d73000 msgr2=0x55d9d8c0d080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:11:19.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.479+0000 7f586be4b8c0 1 --2- 172.21.15.16:0/980142147 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d9d8d73000 0x55d9d8c0d080 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x55d9d8d74c30 tx=0x55d9d814c000 comp rx=0 tx=0).stop 2024-03-31T15:11:19.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.479+0000 7f586be4b8c0 1 -- 172.21.15.16:0/980142147 shutdown_connections 2024-03-31T15:11:19.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.479+0000 7f586be4b8c0 1 --2- 172.21.15.16:0/980142147 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d9d8d73800 0x55d9d8c0db80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:19.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.479+0000 7f586be4b8c0 1 --2- 172.21.15.16:0/980142147 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55d9d8d73400 0x55d9d8c0d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:19.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.479+0000 7f586be4b8c0 1 --2- 172.21.15.16:0/980142147 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d9d8d73000 0x55d9d8c0d080 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:19.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.479+0000 7f586be4b8c0 1 -- 172.21.15.16:0/980142147 >> 172.21.15.16:0/980142147 conn(0x55d9d7e54c00 msgr2=0x55d9d7e83800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:11:19.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.479+0000 7f586be4b8c0 1 -- 172.21.15.16:0/980142147 shutdown_connections 2024-03-31T15:11:19.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.479+0000 7f586be4b8c0 1 -- 172.21.15.16:0/980142147 wait complete. 2024-03-31T15:11:19.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.479+0000 7f586be4b8c0 1 Processor -- start 2024-03-31T15:11:19.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.483+0000 7f586be4b8c0 1 -- start start 2024-03-31T15:11:19.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.483+0000 7f586be4b8c0 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55d9d8d73000 0x55d9d8c0d080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:19.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.483+0000 7f586be4b8c0 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d9d8d73400 0x55d9d8c0d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:19.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.483+0000 7f586be4b8c0 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d9d8d73800 0x55d9d8c0db80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:19.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.483+0000 7f586be4b8c0 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55d9d8c29e00 con 0x55d9d8d73800 2024-03-31T15:11:19.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.483+0000 7f586be4b8c0 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55d9d8c29c80 con 0x55d9d8d73400 2024-03-31T15:11:19.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.483+0000 7f586be4b8c0 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x55d9d8c29b00 con 0x55d9d8d73000 2024-03-31T15:11:19.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.483+0000 7f5869600640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d9d8d73800 0x55d9d8c0db80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:19.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.483+0000 7f5869600640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d9d8d73800 0x55d9d8c0db80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56196/0 (socket says 172.21.15.16:56196) 2024-03-31T15:11:19.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.483+0000 7f586a000640 1 --2- 172.21.15.16:0/2582117395 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d9d8d73400 0x55d9d8c0d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:19.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.483+0000 7f5869600640 1 -- 172.21.15.16:0/2582117395 learned_addr learned my addr 172.21.15.16:0/2582117395 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:11:19.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.483+0000 7f586aa00640 1 --2- 172.21.15.16:0/2582117395 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55d9d8d73000 0x55d9d8c0d080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:19.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.483+0000 7f5869600640 1 -- 172.21.15.16:0/2582117395 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55d9d8d73000 msgr2=0x55d9d8c0d080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:11:19.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.483+0000 7f5869600640 1 --2- 172.21.15.16:0/2582117395 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55d9d8d73000 0x55d9d8c0d080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:19.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.483+0000 7f5869600640 1 -- 172.21.15.16:0/2582117395 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d9d8d73400 msgr2=0x55d9d8c0d600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:11:19.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.483+0000 7f5869600640 1 --2- 172.21.15.16:0/2582117395 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d9d8d73400 0x55d9d8c0d600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:19.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.483+0000 7f5869600640 1 -- 172.21.15.16:0/2582117395 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d9d7e7a1e0 con 0x55d9d8d73800 2024-03-31T15:11:19.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.483+0000 7f586aa00640 1 --2- 172.21.15.16:0/2582117395 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55d9d8d73000 0x55d9d8c0d080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:11:19.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.483+0000 7f586a000640 1 --2- 172.21.15.16:0/2582117395 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d9d8d73400 0x55d9d8c0d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:11:19.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.483+0000 7f5869600640 1 --2- 172.21.15.16:0/2582117395 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d9d8d73800 0x55d9d8c0db80 secure :-1 s=READY pgs=1251 cs=0 l=1 rev1=1 crypto rx=0x55d9d8cbaff0 tx=0x55d9d7fc8000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:11:19.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.483+0000 7f5862800640 1 -- 172.21.15.16:0/2582117395 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x55d9d7e8a9c0 con 0x55d9d8d73800 2024-03-31T15:11:19.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.483+0000 7f5862800640 1 -- 172.21.15.16:0/2582117395 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x55d9d7ea0a80 con 0x55d9d8d73800 2024-03-31T15:11:19.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.483+0000 7f5862800640 1 -- 172.21.15.16:0/2582117395 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x55d9d7e8a4e0 con 0x55d9d8d73800 2024-03-31T15:11:19.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.483+0000 7f586be4b8c0 1 -- 172.21.15.16:0/2582117395 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55d9d7e7b4a0 con 0x55d9d8d73800 2024-03-31T15:11:19.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.483+0000 7f586be4b8c0 1 -- 172.21.15.16:0/2582117395 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d9d7e7be00 con 0x55d9d8d73800 2024-03-31T15:11:19.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.483+0000 7f586be4b8c0 4 datascan.init: resolving metadata pool 12 2024-03-31T15:11:19.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.483+0000 7f5862800640 1 -- 172.21.15.16:0/2582117395 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x55d9d8156c00 con 0x55d9d8d73800 2024-03-31T15:11:19.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.487+0000 7f5862800640 1 --2- 172.21.15.16:0/2582117395 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x55d9d8ddcc00 0x55d9d8c0e100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:19.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.487+0000 7f5862800640 1 -- 172.21.15.16:0/2582117395 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 5910+0+0 (secure 0 0 0) 0x55d9d814ea00 con 0x55d9d8d73800 2024-03-31T15:11:19.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.487+0000 7f586be4b8c0 4 datascan.init: found metadata pool 'cephfs_metadata' 2024-03-31T15:11:19.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.487+0000 7f586be4b8c0 4 datascan.main: data pool 'cephfs_data' has ID 13 2024-03-31T15:11:19.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.487+0000 7f586be4b8c0 4 datascan.main: opening data pool 'cephfs_data' 2024-03-31T15:11:19.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.487+0000 7f586be4b8c0 1 --2- 172.21.15.16:0/2582117395 >> [v2:172.21.15.89:6808/1796778630,v1:172.21.15.89:6811/1796778630] conn(0x55d9d8d73c00 0x55d9d8c0e680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:19.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.487+0000 7f586be4b8c0 1 -- 172.21.15.16:0/2582117395 --> [v2:172.21.15.89:6808/1796778630,v1:172.21.15.89:6811/1796778630] -- osd_op(unknown.0.0:1 13.0 13:00000000::::head [pgnls start_epoch 85 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e85) -- 0x55d9d8ddd000 con 0x55d9d8d73c00 2024-03-31T15:11:19.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.487+0000 7f586aa00640 1 --2- 172.21.15.16:0/2582117395 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x55d9d8ddcc00 0x55d9d8c0e100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:19.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.487+0000 7f586a000640 1 --2- 172.21.15.16:0/2582117395 >> [v2:172.21.15.89:6808/1796778630,v1:172.21.15.89:6811/1796778630] conn(0x55d9d8d73c00 0x55d9d8c0e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:19.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.487+0000 7f586aa00640 1 --2- 172.21.15.16:0/2582117395 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x55d9d8ddcc00 0x55d9d8c0e100 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x55d9d8d74c30 tx=0x55d9d8bdaf60 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:11:19.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.487+0000 7f586a000640 1 --2- 172.21.15.16:0/2582117395 >> [v2:172.21.15.89:6808/1796778630,v1:172.21.15.89:6811/1796778630] conn(0x55d9d8d73c00 0x55d9d8c0e680 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:11:19.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.487+0000 7f586a000640 1 -- 172.21.15.16:0/2582117395 <== osd.7 v2:172.21.15.89:6808/1796778630 1 ==== osd_op_reply(1 [pgnls start_epoch 85 out=49b] v81'8 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55d9d7f2a000 con 0x55d9d8d73c00 2024-03-31T15:11:19.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.487+0000 7f5863c00640 1 --2- 172.21.15.16:0/2582117395 >> [v2:172.21.15.89:6809/3470471752,v1:172.21.15.89:6812/3470471752] conn(0x55d9d8240000 0x55d9d8246000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:19.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.487+0000 7f5863c00640 1 -- 172.21.15.16:0/2582117395 --> [v2:172.21.15.89:6809/3470471752,v1:172.21.15.89:6812/3470471752] -- osd_op(unknown.0.0:2 13.20 13:04000000::::head [pgnls start_epoch 85 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e85) -- 0x55d9d8240400 con 0x55d9d8240000 2024-03-31T15:11:19.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.487+0000 7f586aa00640 1 --2- 172.21.15.16:0/2582117395 >> [v2:172.21.15.89:6809/3470471752,v1:172.21.15.89:6812/3470471752] conn(0x55d9d8240000 0x55d9d8246000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:19.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.487+0000 7f586aa00640 1 --2- 172.21.15.16:0/2582117395 >> [v2:172.21.15.89:6809/3470471752,v1:172.21.15.89:6812/3470471752] conn(0x55d9d8240000 0x55d9d8246000 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:11:19.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.487+0000 7f586aa00640 1 -- 172.21.15.16:0/2582117395 <== osd.6 v2:172.21.15.89:6809/3470471752 1 ==== osd_op_reply(2 [pgnls start_epoch 85 out=49b] v81'9 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55d9d7f2a500 con 0x55d9d8240000 2024-03-31T15:11:19.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.487+0000 7f5864600640 1 --2- 172.21.15.16:0/2582117395 >> [v2:172.21.15.16:6808/1398737533,v1:172.21.15.16:6810/1398737533] conn(0x55d9d8240c00 0x55d9d8246580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:19.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.487+0000 7f5864600640 1 -- 172.21.15.16:0/2582117395 --> [v2:172.21.15.16:6808/1398737533,v1:172.21.15.16:6810/1398737533] -- osd_op(unknown.0.0:3 13.10 13:08000000::::head [pgnls start_epoch 85 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e85) -- 0x55d9d8241000 con 0x55d9d8240c00 2024-03-31T15:11:19.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.487+0000 7f586a000640 1 --2- 172.21.15.16:0/2582117395 >> [v2:172.21.15.16:6808/1398737533,v1:172.21.15.16:6810/1398737533] conn(0x55d9d8240c00 0x55d9d8246580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:19.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.491+0000 7f586a000640 1 --2- 172.21.15.16:0/2582117395 >> [v2:172.21.15.16:6808/1398737533,v1:172.21.15.16:6810/1398737533] conn(0x55d9d8240c00 0x55d9d8246580 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:11:19.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.491+0000 7f586a000640 1 -- 172.21.15.16:0/2582117395 <== osd.1 v2:172.21.15.16:6808/1398737533 1 ==== osd_op_reply(3 [pgnls start_epoch 85 out=49b] v81'6 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55d9d7f2a000 con 0x55d9d8240c00 2024-03-31T15:11:19.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.491+0000 7f5863c00640 1 --2- 172.21.15.16:0/2582117395 >> [v2:172.21.15.89:6800/1856844856,v1:172.21.15.89:6801/1856844856] conn(0x55d9d8240800 0x55d9d8246b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:19.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.491+0000 7f5863c00640 1 -- 172.21.15.16:0/2582117395 --> [v2:172.21.15.89:6800/1856844856,v1:172.21.15.89:6801/1856844856] -- osd_op(unknown.0.0:4 13.30 13:0c000000::::head [pgnls start_epoch 85 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e85) -- 0x55d9d8241800 con 0x55d9d8240800 2024-03-31T15:11:19.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.491+0000 7f5869600640 1 --2- 172.21.15.16:0/2582117395 >> [v2:172.21.15.89:6800/1856844856,v1:172.21.15.89:6801/1856844856] conn(0x55d9d8240800 0x55d9d8246b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:19.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.491+0000 7f5869600640 1 --2- 172.21.15.16:0/2582117395 >> [v2:172.21.15.89:6800/1856844856,v1:172.21.15.89:6801/1856844856] conn(0x55d9d8240800 0x55d9d8246b00 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:11:19.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.495+0000 7f5869600640 1 -- 172.21.15.16:0/2582117395 <== osd.4 v2:172.21.15.89:6800/1856844856 1 ==== osd_op_reply(4 [pgnls start_epoch 85 out=49b] v81'9 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55d9d7f2a780 con 0x55d9d8240800 2024-03-31T15:11:19.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.495+0000 7f5864600640 1 --2- 172.21.15.16:0/2582117395 >> [v2:172.21.15.16:6819/3018526646,v1:172.21.15.16:6822/3018526646] conn(0x55d9d8241400 0x55d9d8247600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:19.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.495+0000 7f5864600640 1 -- 172.21.15.16:0/2582117395 --> [v2:172.21.15.16:6819/3018526646,v1:172.21.15.16:6822/3018526646] -- osd_op(unknown.0.0:5 13.8 13:10000000::::head [pgnls start_epoch 85 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e85) -- 0x55d9d825c000 con 0x55d9d8241400 2024-03-31T15:11:19.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.495+0000 7f586aa00640 1 --2- 172.21.15.16:0/2582117395 >> [v2:172.21.15.16:6819/3018526646,v1:172.21.15.16:6822/3018526646] conn(0x55d9d8241400 0x55d9d8247600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:19.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.495+0000 7f586aa00640 1 --2- 172.21.15.16:0/2582117395 >> [v2:172.21.15.16:6819/3018526646,v1:172.21.15.16:6822/3018526646] conn(0x55d9d8241400 0x55d9d8247600 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:11:19.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.499+0000 7f586aa00640 1 -- 172.21.15.16:0/2582117395 <== osd.2 v2:172.21.15.16:6819/3018526646 1 ==== osd_op_reply(5 [pgnls start_epoch 85 out=49b] v81'9 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55d9d7f2a500 con 0x55d9d8241400 2024-03-31T15:11:19.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.499+0000 7f5863c00640 1 -- 172.21.15.16:0/2582117395 --> [v2:172.21.15.16:6819/3018526646,v1:172.21.15.16:6822/3018526646] -- osd_op(unknown.0.0:6 13.28 13:14000000::::head [pgnls start_epoch 85 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e85) -- 0x55d9d8241c00 con 0x55d9d8241400 2024-03-31T15:11:19.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.499+0000 7f586aa00640 1 -- 172.21.15.16:0/2582117395 <== osd.2 v2:172.21.15.16:6819/3018526646 2 ==== osd_op_reply(6 [pgnls start_epoch 85 out=49b] v81'6 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55d9d7f2a500 con 0x55d9d8241400 2024-03-31T15:11:19.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.499+0000 7f5864600640 1 -- 172.21.15.16:0/2582117395 --> [v2:172.21.15.89:6808/1796778630,v1:172.21.15.89:6811/1796778630] -- osd_op(unknown.0.0:7 13.18 13:18000000::::head [pgnls start_epoch 85 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e85) -- 0x55d9d825c400 con 0x55d9d8d73c00 2024-03-31T15:11:19.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.499+0000 7f586a000640 1 -- 172.21.15.16:0/2582117395 <== osd.7 v2:172.21.15.89:6808/1796778630 2 ==== osd_op_reply(7 [pgnls start_epoch 85 out=49b] v81'8 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55d9d7f2a000 con 0x55d9d8d73c00 2024-03-31T15:11:19.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.499+0000 7f5863c00640 1 --2- 172.21.15.16:0/2582117395 >> [v2:172.21.15.89:6810/50395527,v1:172.21.15.89:6814/50395527] conn(0x55d9d825cc00 0x55d9d8247080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:19.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.499+0000 7f5863c00640 1 -- 172.21.15.16:0/2582117395 --> [v2:172.21.15.89:6810/50395527,v1:172.21.15.89:6814/50395527] -- osd_op(unknown.0.0:8 13.38 13:1c000000::::head [pgnls start_epoch 85 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e85) -- 0x55d9d825d000 con 0x55d9d825cc00 2024-03-31T15:11:19.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.499+0000 7f586a000640 1 --2- 172.21.15.16:0/2582117395 >> [v2:172.21.15.89:6810/50395527,v1:172.21.15.89:6814/50395527] conn(0x55d9d825cc00 0x55d9d8247080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:19.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.499+0000 7f586a000640 1 --2- 172.21.15.16:0/2582117395 >> [v2:172.21.15.89:6810/50395527,v1:172.21.15.89:6814/50395527] conn(0x55d9d825cc00 0x55d9d8247080 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:11:19.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.503+0000 7f586a000640 1 -- 172.21.15.16:0/2582117395 <== osd.5 v2:172.21.15.89:6810/50395527 1 ==== osd_op_reply(8 [pgnls start_epoch 85 out=49b] v81'9 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55d9d7f2a000 con 0x55d9d825cc00 2024-03-31T15:11:19.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.503+0000 7f5864600640 1 -- 172.21.15.16:0/2582117395 --> [v2:172.21.15.89:6809/3470471752,v1:172.21.15.89:6812/3470471752] -- osd_op(unknown.0.0:9 13.4 13:20000000::::head [pgnls start_epoch 85 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e85) -- 0x55d9d825c800 con 0x55d9d8240000 2024-03-31T15:11:19.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.503+0000 7f586aa00640 1 -- 172.21.15.16:0/2582117395 <== osd.6 v2:172.21.15.89:6809/3470471752 2 ==== osd_op_reply(9 [pgnls start_epoch 85 out=49b] v81'9 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55d9d7f2a500 con 0x55d9d8240000 2024-03-31T15:11:19.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.503+0000 7f5863c00640 1 -- 172.21.15.16:0/2582117395 --> [v2:172.21.15.89:6809/3470471752,v1:172.21.15.89:6812/3470471752] -- osd_op(unknown.0.0:10 13.24 13:24000000::::head [pgnls start_epoch 85 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e85) -- 0x55d9d825d400 con 0x55d9d8240000 2024-03-31T15:11:19.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.503+0000 7f586aa00640 1 -- 172.21.15.16:0/2582117395 <== osd.6 v2:172.21.15.89:6809/3470471752 3 ==== osd_op_reply(10 [pgnls start_epoch 85 out=49b] v81'4 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55d9d7f2a500 con 0x55d9d8240000 2024-03-31T15:11:19.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.503+0000 7f5864600640 1 --2- 172.21.15.16:0/2582117395 >> [v2:172.21.15.16:6800/932712244,v1:172.21.15.16:6801/932712244] conn(0x55d9d825c000 0x55d9d8247b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:19.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.503+0000 7f5864600640 1 -- 172.21.15.16:0/2582117395 --> [v2:172.21.15.16:6800/932712244,v1:172.21.15.16:6801/932712244] -- osd_op(unknown.0.0:11 13.14 13:28000000::::head [pgnls start_epoch 85 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e85) -- 0x55d9d825c800 con 0x55d9d825c000 2024-03-31T15:11:19.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.503+0000 7f5869600640 1 --2- 172.21.15.16:0/2582117395 >> [v2:172.21.15.16:6800/932712244,v1:172.21.15.16:6801/932712244] conn(0x55d9d825c000 0x55d9d8247b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:19.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.503+0000 7f5869600640 1 --2- 172.21.15.16:0/2582117395 >> [v2:172.21.15.16:6800/932712244,v1:172.21.15.16:6801/932712244] conn(0x55d9d825c000 0x55d9d8247b80 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:11:19.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.503+0000 7f5869600640 1 -- 172.21.15.16:0/2582117395 <== osd.0 v2:172.21.15.16:6800/932712244 1 ==== osd_op_reply(11 [pgnls start_epoch 85 out=49b] v81'11 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55d9d7f2a780 con 0x55d9d825c000 2024-03-31T15:11:19.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.503+0000 7f5863c00640 1 -- 172.21.15.16:0/2582117395 --> [v2:172.21.15.89:6809/3470471752,v1:172.21.15.89:6812/3470471752] -- osd_op(unknown.0.0:12 13.34 13:2c000000::::head [pgnls start_epoch 85 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e85) -- 0x55d9d825d800 con 0x55d9d8240000 2024-03-31T15:11:19.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.503+0000 7f586aa00640 1 -- 172.21.15.16:0/2582117395 <== osd.6 v2:172.21.15.89:6809/3470471752 4 ==== osd_op_reply(12 [pgnls start_epoch 85 out=49b] v81'7 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55d9d7f2a500 con 0x55d9d8240000 2024-03-31T15:11:19.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.503+0000 7f5864600640 1 -- 172.21.15.16:0/2582117395 --> [v2:172.21.15.16:6808/1398737533,v1:172.21.15.16:6810/1398737533] -- osd_op(unknown.0.0:13 13.c 13:30000000::::head [pgnls start_epoch 85 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e85) -- 0x55d9d8240400 con 0x55d9d8240c00 2024-03-31T15:11:19.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.503+0000 7f586a000640 1 -- 172.21.15.16:0/2582117395 <== osd.1 v2:172.21.15.16:6808/1398737533 2 ==== osd_op_reply(13 [pgnls start_epoch 85 out=49b] v81'6 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55d9d7f2a000 con 0x55d9d8240c00 2024-03-31T15:11:19.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.503+0000 7f5863c00640 1 -- 172.21.15.16:0/2582117395 --> [v2:172.21.15.16:6819/3018526646,v1:172.21.15.16:6822/3018526646] -- osd_op(unknown.0.0:14 13.2c 13:34000000::::head [pgnls start_epoch 85 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e85) -- 0x55d9d825c400 con 0x55d9d8241400 2024-03-31T15:11:19.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.503+0000 7f586aa00640 1 -- 172.21.15.16:0/2582117395 <== osd.2 v2:172.21.15.16:6819/3018526646 3 ==== osd_op_reply(14 [pgnls start_epoch 85 out=49b] v81'8 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55d9d7f2a500 con 0x55d9d8241400 2024-03-31T15:11:19.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.503+0000 7f5864600640 1 -- 172.21.15.16:0/2582117395 --> [v2:172.21.15.89:6809/3470471752,v1:172.21.15.89:6812/3470471752] -- osd_op(unknown.0.0:15 13.1c 13:38000000::::head [pgnls start_epoch 85 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e85) -- 0x55d9d8241c00 con 0x55d9d8240000 2024-03-31T15:11:19.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.507+0000 7f586aa00640 1 -- 172.21.15.16:0/2582117395 <== osd.6 v2:172.21.15.89:6809/3470471752 5 ==== osd_op_reply(15 [pgnls start_epoch 85 out=49b] v81'9 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55d9d7f2a500 con 0x55d9d8240000 2024-03-31T15:11:19.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.507+0000 7f5863c00640 1 -- 172.21.15.16:0/2582117395 --> [v2:172.21.15.16:6819/3018526646,v1:172.21.15.16:6822/3018526646] -- osd_op(unknown.0.0:16 13.3c 13:3c000000::::head [pgnls start_epoch 85 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e85) -- 0x55d9d825d000 con 0x55d9d8241400 2024-03-31T15:11:19.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.507+0000 7f586aa00640 1 -- 172.21.15.16:0/2582117395 <== osd.2 v2:172.21.15.16:6819/3018526646 4 ==== osd_op_reply(16 [pgnls start_epoch 85 out=49b] v81'8 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55d9d7f2a500 con 0x55d9d8241400 2024-03-31T15:11:19.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.507+0000 7f5864600640 1 -- 172.21.15.16:0/2582117395 --> [v2:172.21.15.89:6810/50395527,v1:172.21.15.89:6814/50395527] -- osd_op(unknown.0.0:17 13.2 13:40000000::::head [pgnls start_epoch 85 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e85) -- 0x55d9d825d400 con 0x55d9d825cc00 2024-03-31T15:11:19.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.507+0000 7f586a000640 1 -- 172.21.15.16:0/2582117395 <== osd.5 v2:172.21.15.89:6810/50395527 2 ==== osd_op_reply(17 [pgnls start_epoch 85 out=49b] v81'8 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55d9d7f2a000 con 0x55d9d825cc00 2024-03-31T15:11:19.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.507+0000 7f5863c00640 1 -- 172.21.15.16:0/2582117395 --> [v2:172.21.15.16:6808/1398737533,v1:172.21.15.16:6810/1398737533] -- osd_op(unknown.0.0:18 13.22 13:44000000::::head [pgnls start_epoch 85 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e85) -- 0x55d9d8241000 con 0x55d9d8240c00 2024-03-31T15:11:19.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.507+0000 7f586a000640 1 -- 172.21.15.16:0/2582117395 <== osd.1 v2:172.21.15.16:6808/1398737533 3 ==== osd_op_reply(18 [pgnls start_epoch 85 out=49b] v81'8 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55d9d7f2a000 con 0x55d9d8240c00 2024-03-31T15:11:19.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.507+0000 7f5864600640 1 -- 172.21.15.16:0/2582117395 --> [v2:172.21.15.16:6800/932712244,v1:172.21.15.16:6801/932712244] -- osd_op(unknown.0.0:19 13.12 13:48000000::::head [pgnls start_epoch 85 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e85) -- 0x55d9d825d800 con 0x55d9d825c000 2024-03-31T15:11:19.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.507+0000 7f5869600640 1 -- 172.21.15.16:0/2582117395 <== osd.0 v2:172.21.15.16:6800/932712244 2 ==== osd_op_reply(19 [pgnls start_epoch 85 out=49b] v81'10 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55d9d7f2a780 con 0x55d9d825c000 2024-03-31T15:11:19.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.507+0000 7f5863c00640 1 -- 172.21.15.16:0/2582117395 --> [v2:172.21.15.16:6819/3018526646,v1:172.21.15.16:6822/3018526646] -- osd_op(unknown.0.0:20 13.32 13:4c000000::::head [pgnls start_epoch 85 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e85) -- 0x55d9d8240400 con 0x55d9d8241400 2024-03-31T15:11:19.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.507+0000 7f586aa00640 1 -- 172.21.15.16:0/2582117395 <== osd.2 v2:172.21.15.16:6819/3018526646 5 ==== osd_op_reply(20 [pgnls start_epoch 85 out=49b] v81'8 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55d9d7f2a500 con 0x55d9d8241400 2024-03-31T15:11:19.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.507+0000 7f5864600640 1 -- 172.21.15.16:0/2582117395 --> [v2:172.21.15.16:6819/3018526646,v1:172.21.15.16:6822/3018526646] -- osd_op(unknown.0.0:21 13.a 13:50000000::::head [pgnls start_epoch 85 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e85) -- 0x55d9d825c400 con 0x55d9d8241400 2024-03-31T15:11:19.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.507+0000 7f586aa00640 1 -- 172.21.15.16:0/2582117395 <== osd.2 v2:172.21.15.16:6819/3018526646 6 ==== osd_op_reply(21 [pgnls start_epoch 85 out=49b] v81'5 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55d9d7f2a500 con 0x55d9d8241400 2024-03-31T15:11:19.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.507+0000 7f5863c00640 1 -- 172.21.15.16:0/2582117395 --> [v2:172.21.15.16:6808/1398737533,v1:172.21.15.16:6810/1398737533] -- osd_op(unknown.0.0:22 13.2a 13:54000000::::head [pgnls start_epoch 85 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e85) -- 0x55d9d8ddd000 con 0x55d9d8240c00 2024-03-31T15:11:19.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.507+0000 7f586a000640 1 -- 172.21.15.16:0/2582117395 <== osd.1 v2:172.21.15.16:6808/1398737533 4 ==== osd_op_reply(22 [pgnls start_epoch 85 out=49b] v81'9 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55d9d7f2a000 con 0x55d9d8240c00 2024-03-31T15:11:19.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.507+0000 7f5864600640 1 -- 172.21.15.16:0/2582117395 --> [v2:172.21.15.16:6808/1398737533,v1:172.21.15.16:6810/1398737533] -- osd_op(unknown.0.0:23 13.1a 13:58000000::::head [pgnls start_epoch 85 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e85) -- 0x55d9d825d000 con 0x55d9d8240c00 2024-03-31T15:11:19.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.507+0000 7f586a000640 1 -- 172.21.15.16:0/2582117395 <== osd.1 v2:172.21.15.16:6808/1398737533 5 ==== osd_op_reply(23 [pgnls start_epoch 85 out=49b] v81'7 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55d9d7f2a000 con 0x55d9d8240c00 2024-03-31T15:11:19.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.507+0000 7f5863c00640 1 -- 172.21.15.16:0/2582117395 --> [v2:172.21.15.89:6810/50395527,v1:172.21.15.89:6814/50395527] -- osd_op(unknown.0.0:24 13.3a 13:5c000000::::head [pgnls start_epoch 85 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e85) -- 0x55d9d825dc00 con 0x55d9d825cc00 2024-03-31T15:11:19.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.507+0000 7f586a000640 1 -- 172.21.15.16:0/2582117395 <== osd.5 v2:172.21.15.89:6810/50395527 3 ==== osd_op_reply(24 [pgnls start_epoch 85 out=49b] v81'6 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55d9d7f2a000 con 0x55d9d825cc00 2024-03-31T15:11:19.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.507+0000 7f5864600640 1 -- 172.21.15.16:0/2582117395 --> [v2:172.21.15.89:6800/1856844856,v1:172.21.15.89:6801/1856844856] -- osd_op(unknown.0.0:25 13.6 13:60000000::::head [pgnls start_epoch 85 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e85) -- 0x55d9d8241c00 con 0x55d9d8240800 2024-03-31T15:11:19.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.507+0000 7f5869600640 1 -- 172.21.15.16:0/2582117395 <== osd.4 v2:172.21.15.89:6800/1856844856 2 ==== osd_op_reply(25 [pgnls start_epoch 85 out=49b] v81'7 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55d9d7f2a780 con 0x55d9d8240800 2024-03-31T15:11:19.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.507+0000 7f5863c00640 1 -- 172.21.15.16:0/2582117395 --> [v2:172.21.15.89:6808/1796778630,v1:172.21.15.89:6811/1796778630] -- osd_op(unknown.0.0:26 13.26 13:64000000::::head [pgnls start_epoch 85 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e85) -- 0x55d9d8241800 con 0x55d9d8d73c00 2024-03-31T15:11:19.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.507+0000 7f586a000640 1 -- 172.21.15.16:0/2582117395 <== osd.7 v2:172.21.15.89:6808/1796778630 3 ==== osd_op_reply(26 [pgnls start_epoch 85 out=49b] v81'15 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55d9d7f2a000 con 0x55d9d8d73c00 2024-03-31T15:11:19.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.507+0000 7f5864600640 1 --2- 172.21.15.16:0/2582117395 >> [v2:172.21.15.16:6809/3360352974,v1:172.21.15.16:6812/3360352974] conn(0x55d9d8241000 0x55d9d8248100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:19.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.507+0000 7f5864600640 1 -- 172.21.15.16:0/2582117395 --> [v2:172.21.15.16:6809/3360352974,v1:172.21.15.16:6812/3360352974] -- osd_op(unknown.0.0:27 13.16 13:68000000::::head [pgnls start_epoch 85 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e85) -- 0x55d9d8241800 con 0x55d9d8241000 2024-03-31T15:11:19.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.507+0000 7f586aa00640 1 --2- 172.21.15.16:0/2582117395 >> [v2:172.21.15.16:6809/3360352974,v1:172.21.15.16:6812/3360352974] conn(0x55d9d8241000 0x55d9d8248100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:19.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.511+0000 7f586aa00640 1 --2- 172.21.15.16:0/2582117395 >> [v2:172.21.15.16:6809/3360352974,v1:172.21.15.16:6812/3360352974] conn(0x55d9d8241000 0x55d9d8248100 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:11:19.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.511+0000 7f586aa00640 1 -- 172.21.15.16:0/2582117395 <== osd.3 v2:172.21.15.16:6809/3360352974 1 ==== osd_op_reply(27 [pgnls start_epoch 85 out=49b] v81'10 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55d9d7f2a500 con 0x55d9d8241000 2024-03-31T15:11:19.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.511+0000 7f5863c00640 1 -- 172.21.15.16:0/2582117395 --> [v2:172.21.15.89:6809/3470471752,v1:172.21.15.89:6812/3470471752] -- osd_op(unknown.0.0:28 13.36 13:6c000000::::head [pgnls start_epoch 85 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e85) -- 0x55d9d8241c00 con 0x55d9d8240000 2024-03-31T15:11:19.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.511+0000 7f586aa00640 1 -- 172.21.15.16:0/2582117395 <== osd.6 v2:172.21.15.89:6809/3470471752 6 ==== osd_op_reply(28 [pgnls start_epoch 85 out=49b] v81'15 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55d9d7f2a500 con 0x55d9d8240000 2024-03-31T15:11:19.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.511+0000 7f5864600640 1 -- 172.21.15.16:0/2582117395 --> [v2:172.21.15.89:6808/1796778630,v1:172.21.15.89:6811/1796778630] -- osd_op(unknown.0.0:29 13.e 13:70000000::::head [pgnls start_epoch 85 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e85) -- 0x55d9d825d400 con 0x55d9d8d73c00 2024-03-31T15:11:19.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.511+0000 7f586a000640 1 -- 172.21.15.16:0/2582117395 <== osd.7 v2:172.21.15.89:6808/1796778630 4 ==== osd_op_reply(29 [pgnls start_epoch 85 out=49b] v81'5 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55d9d7f2a000 con 0x55d9d8d73c00 2024-03-31T15:11:19.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.511+0000 7f5863c00640 1 -- 172.21.15.16:0/2582117395 --> [v2:172.21.15.16:6819/3018526646,v1:172.21.15.16:6822/3018526646] -- osd_op(unknown.0.0:30 13.2e 13:74000000::::head [pgnls start_epoch 85 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e85) -- 0x55d9d825c800 con 0x55d9d8241400 2024-03-31T15:11:19.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.511+0000 7f586aa00640 1 -- 172.21.15.16:0/2582117395 <== osd.2 v2:172.21.15.16:6819/3018526646 7 ==== osd_op_reply(30 [pgnls start_epoch 85 out=49b] v81'10 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55d9d7f2a500 con 0x55d9d8241400 2024-03-31T15:11:19.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.511+0000 7f5864600640 1 -- 172.21.15.16:0/2582117395 --> [v2:172.21.15.89:6800/1856844856,v1:172.21.15.89:6801/1856844856] -- osd_op(unknown.0.0:31 13.1e 13:78000000::::head [pgnls start_epoch 85 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e85) -- 0x55d9d825d000 con 0x55d9d8240800 2024-03-31T15:11:19.507 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.515+0000 7f5869600640 1 -- 172.21.15.16:0/2582117395 <== osd.4 v2:172.21.15.89:6800/1856844856 3 ==== osd_op_reply(31 [pgnls start_epoch 85 out=49b] v81'5 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55d9d7f2a780 con 0x55d9d8240800 2024-03-31T15:11:19.507 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.515+0000 7f5863c00640 1 -- 172.21.15.16:0/2582117395 --> [v2:172.21.15.16:6808/1398737533,v1:172.21.15.16:6810/1398737533] -- osd_op(unknown.0.0:32 13.3e 13:7c000000::::head [pgnls start_epoch 85 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e85) -- 0x55d9d825d800 con 0x55d9d8240c00 2024-03-31T15:11:19.507 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.515+0000 7f586a000640 1 -- 172.21.15.16:0/2582117395 <== osd.1 v2:172.21.15.16:6808/1398737533 6 ==== osd_op_reply(32 [pgnls start_epoch 85 out=49b] v81'4 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55d9d7f2a000 con 0x55d9d8240c00 2024-03-31T15:11:19.507 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.515+0000 7f5864600640 1 -- 172.21.15.16:0/2582117395 --> [v2:172.21.15.89:6809/3470471752,v1:172.21.15.89:6812/3470471752] -- osd_op(unknown.0.0:33 13.1 13:80000000::::head [pgnls start_epoch 85 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e85) -- 0x55d9d825dc00 con 0x55d9d8240000 2024-03-31T15:11:19.507 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.515+0000 7f586aa00640 1 -- 172.21.15.16:0/2582117395 <== osd.6 v2:172.21.15.89:6809/3470471752 7 ==== osd_op_reply(33 [pgnls start_epoch 85 out=49b] v81'5 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55d9d7f2a500 con 0x55d9d8240000 2024-03-31T15:11:19.507 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.515+0000 7f5863c00640 1 -- 172.21.15.16:0/2582117395 --> [v2:172.21.15.89:6800/1856844856,v1:172.21.15.89:6801/1856844856] -- osd_op(unknown.0.0:34 13.21 13:84000000::::head [pgnls start_epoch 85 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e85) -- 0x55d9d8c43400 con 0x55d9d8240800 2024-03-31T15:11:19.508 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.515+0000 7f5869600640 1 -- 172.21.15.16:0/2582117395 <== osd.4 v2:172.21.15.89:6800/1856844856 4 ==== osd_op_reply(34 [pgnls start_epoch 85 out=49b] v81'12 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55d9d7f2a780 con 0x55d9d8240800 2024-03-31T15:11:19.508 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.515+0000 7f5864600640 1 -- 172.21.15.16:0/2582117395 --> [v2:172.21.15.16:6808/1398737533,v1:172.21.15.16:6810/1398737533] -- osd_op(unknown.0.0:35 13.11 13:88000000::::head [pgnls start_epoch 85 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e85) -- 0x55d9d8ddd000 con 0x55d9d8240c00 2024-03-31T15:11:19.508 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.515+0000 7f586a000640 1 -- 172.21.15.16:0/2582117395 <== osd.1 v2:172.21.15.16:6808/1398737533 7 ==== osd_op_reply(35 [pgnls start_epoch 85 out=49b] v81'6 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55d9d7f2a000 con 0x55d9d8240c00 2024-03-31T15:11:19.508 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.515+0000 7f5863c00640 1 -- 172.21.15.16:0/2582117395 --> [v2:172.21.15.16:6809/3360352974,v1:172.21.15.16:6812/3360352974] -- osd_op(unknown.0.0:36 13.31 13:8c000000::::head [pgnls start_epoch 85 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e85) -- 0x55d9d825c400 con 0x55d9d8241000 2024-03-31T15:11:19.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.515+0000 7f586aa00640 1 -- 172.21.15.16:0/2582117395 <== osd.3 v2:172.21.15.16:6809/3360352974 2 ==== osd_op_reply(36 [pgnls start_epoch 85 out=49b] v81'7 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55d9d7f2a500 con 0x55d9d8241000 2024-03-31T15:11:19.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.515+0000 7f5864600640 1 -- 172.21.15.16:0/2582117395 --> [v2:172.21.15.89:6810/50395527,v1:172.21.15.89:6814/50395527] -- osd_op(unknown.0.0:37 13.9 13:90000000::::head [pgnls start_epoch 85 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e85) -- 0x55d9d8272000 con 0x55d9d825cc00 2024-03-31T15:11:19.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.515+0000 7f586a000640 1 -- 172.21.15.16:0/2582117395 <== osd.5 v2:172.21.15.89:6810/50395527 4 ==== osd_op_reply(37 [pgnls start_epoch 85 out=49b] v81'2 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55d9d7f2a000 con 0x55d9d825cc00 2024-03-31T15:11:19.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.515+0000 7f5863c00640 1 -- 172.21.15.16:0/2582117395 --> [v2:172.21.15.89:6809/3470471752,v1:172.21.15.89:6812/3470471752] -- osd_op(unknown.0.0:38 13.29 13:94000000::::head [pgnls start_epoch 85 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e85) -- 0x55d9d825c800 con 0x55d9d8240000 2024-03-31T15:11:19.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.515+0000 7f586aa00640 1 -- 172.21.15.16:0/2582117395 <== osd.6 v2:172.21.15.89:6809/3470471752 8 ==== osd_op_reply(38 [pgnls start_epoch 85 out=49b] v81'8 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55d9d7f2a500 con 0x55d9d8240000 2024-03-31T15:11:19.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.515+0000 7f5864600640 1 -- 172.21.15.16:0/2582117395 --> [v2:172.21.15.16:6809/3360352974,v1:172.21.15.16:6812/3360352974] -- osd_op(unknown.0.0:39 13.19 13:98000000::::head [pgnls start_epoch 85 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e85) -- 0x55d9d8272400 con 0x55d9d8241000 2024-03-31T15:11:19.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.515+0000 7f586aa00640 1 -- 172.21.15.16:0/2582117395 <== osd.3 v2:172.21.15.16:6809/3360352974 3 ==== osd_op_reply(39 [pgnls start_epoch 85 out=49b] v81'10 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55d9d7f2a500 con 0x55d9d8241000 2024-03-31T15:11:19.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.515+0000 7f5863c00640 1 -- 172.21.15.16:0/2582117395 --> [v2:172.21.15.16:6819/3018526646,v1:172.21.15.16:6822/3018526646] -- osd_op(unknown.0.0:40 13.39 13:9c000000::::head [pgnls start_epoch 85 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e85) -- 0x55d9d825dc00 con 0x55d9d8241400 2024-03-31T15:11:19.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.515+0000 7f586aa00640 1 -- 172.21.15.16:0/2582117395 <== osd.2 v2:172.21.15.16:6819/3018526646 8 ==== osd_op_reply(40 [pgnls start_epoch 85 out=49b] v81'5 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55d9d7f2a500 con 0x55d9d8241400 2024-03-31T15:11:19.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.515+0000 7f5864600640 1 -- 172.21.15.16:0/2582117395 --> [v2:172.21.15.89:6800/1856844856,v1:172.21.15.89:6801/1856844856] -- osd_op(unknown.0.0:41 13.5 13:a0000000::::head [pgnls start_epoch 85 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e85) -- 0x55d9d8272800 con 0x55d9d8240800 2024-03-31T15:11:19.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.519+0000 7f5869600640 1 -- 172.21.15.16:0/2582117395 <== osd.4 v2:172.21.15.89:6800/1856844856 5 ==== osd_op_reply(41 [pgnls start_epoch 85 out=49b] v81'5 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55d9d7f2a780 con 0x55d9d8240800 2024-03-31T15:11:19.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.519+0000 7f5863c00640 1 -- 172.21.15.16:0/2582117395 --> [v2:172.21.15.89:6800/1856844856,v1:172.21.15.89:6801/1856844856] -- osd_op(unknown.0.0:42 13.25 13:a4000000::::head [pgnls start_epoch 85 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e85) -- 0x55d9d8240400 con 0x55d9d8240800 2024-03-31T15:11:19.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.519+0000 7f5869600640 1 -- 172.21.15.16:0/2582117395 <== osd.4 v2:172.21.15.89:6800/1856844856 6 ==== osd_op_reply(42 [pgnls start_epoch 85 out=49b] v81'6 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55d9d7f2a780 con 0x55d9d8240800 2024-03-31T15:11:19.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.519+0000 7f5864600640 1 -- 172.21.15.16:0/2582117395 --> [v2:172.21.15.89:6800/1856844856,v1:172.21.15.89:6801/1856844856] -- osd_op(unknown.0.0:43 13.15 13:a8000000::::head [pgnls start_epoch 85 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e85) -- 0x55d9d8272c00 con 0x55d9d8240800 2024-03-31T15:11:19.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.519+0000 7f5869600640 1 -- 172.21.15.16:0/2582117395 <== osd.4 v2:172.21.15.89:6800/1856844856 7 ==== osd_op_reply(43 [pgnls start_epoch 85 out=49b] v81'6 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55d9d7f2a780 con 0x55d9d8240800 2024-03-31T15:11:19.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.519+0000 7f5863c00640 1 -- 172.21.15.16:0/2582117395 --> [v2:172.21.15.16:6808/1398737533,v1:172.21.15.16:6810/1398737533] -- osd_op(unknown.0.0:44 13.35 13:ac000000::::head [pgnls start_epoch 85 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e85) -- 0x55d9d8241800 con 0x55d9d8240c00 2024-03-31T15:11:19.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.519+0000 7f586a000640 1 -- 172.21.15.16:0/2582117395 <== osd.1 v2:172.21.15.16:6808/1398737533 8 ==== osd_op_reply(44 [pgnls start_epoch 85 out=49b] v81'6 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55d9d7f2a000 con 0x55d9d8240c00 2024-03-31T15:11:19.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.519+0000 7f5864600640 1 -- 172.21.15.16:0/2582117395 --> [v2:172.21.15.89:6800/1856844856,v1:172.21.15.89:6801/1856844856] -- osd_op(unknown.0.0:45 13.d 13:b0000000::::head [pgnls start_epoch 85 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e85) -- 0x55d9d8273000 con 0x55d9d8240800 2024-03-31T15:11:19.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.519+0000 7f5869600640 1 -- 172.21.15.16:0/2582117395 <== osd.4 v2:172.21.15.89:6800/1856844856 8 ==== osd_op_reply(45 [pgnls start_epoch 85 out=49b] v81'7 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55d9d7f2a780 con 0x55d9d8240800 2024-03-31T15:11:19.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.519+0000 7f5863c00640 1 -- 172.21.15.16:0/2582117395 --> [v2:172.21.15.16:6800/932712244,v1:172.21.15.16:6801/932712244] -- osd_op(unknown.0.0:46 13.2d 13:b4000000::::head [pgnls start_epoch 85 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e85) -- 0x55d9d8241c00 con 0x55d9d825c000 2024-03-31T15:11:19.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.519+0000 7f5869600640 1 -- 172.21.15.16:0/2582117395 <== osd.0 v2:172.21.15.16:6800/932712244 3 ==== osd_op_reply(46 [pgnls start_epoch 85 out=49b] v81'7 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55d9d7f2a780 con 0x55d9d825c000 2024-03-31T15:11:19.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.519+0000 7f5864600640 1 -- 172.21.15.16:0/2582117395 --> [v2:172.21.15.89:6808/1796778630,v1:172.21.15.89:6811/1796778630] -- osd_op(unknown.0.0:47 13.1d 13:b8000000::::head [pgnls start_epoch 85 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e85) -- 0x55d9d8273400 con 0x55d9d8d73c00 2024-03-31T15:11:19.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.519+0000 7f586a000640 1 -- 172.21.15.16:0/2582117395 <== osd.7 v2:172.21.15.89:6808/1796778630 5 ==== osd_op_reply(47 [pgnls start_epoch 85 out=49b] v81'15 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55d9d7f2a000 con 0x55d9d8d73c00 2024-03-31T15:11:19.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.519+0000 7f5863c00640 1 -- 172.21.15.16:0/2582117395 --> [v2:172.21.15.89:6808/1796778630,v1:172.21.15.89:6811/1796778630] -- osd_op(unknown.0.0:48 13.3d 13:bc000000::::head [pgnls start_epoch 85 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e85) -- 0x55d9d8272800 con 0x55d9d8d73c00 2024-03-31T15:11:19.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.519+0000 7f586a000640 1 -- 172.21.15.16:0/2582117395 <== osd.7 v2:172.21.15.89:6808/1796778630 6 ==== osd_op_reply(48 [pgnls start_epoch 85 out=49b] v81'7 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55d9d7f2a000 con 0x55d9d8d73c00 2024-03-31T15:11:19.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.519+0000 7f5864600640 1 -- 172.21.15.16:0/2582117395 --> [v2:172.21.15.16:6800/932712244,v1:172.21.15.16:6801/932712244] -- osd_op(unknown.0.0:49 13.3 13:c0000000::::head [pgnls start_epoch 85 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e85) -- 0x55d9d8273800 con 0x55d9d825c000 2024-03-31T15:11:19.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.519+0000 7f5869600640 1 -- 172.21.15.16:0/2582117395 <== osd.0 v2:172.21.15.16:6800/932712244 4 ==== osd_op_reply(49 [pgnls start_epoch 85 out=49b] v81'7 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55d9d7f2a780 con 0x55d9d825c000 2024-03-31T15:11:19.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.519+0000 7f5863c00640 1 -- 172.21.15.16:0/2582117395 --> [v2:172.21.15.16:6808/1398737533,v1:172.21.15.16:6810/1398737533] -- osd_op(unknown.0.0:50 13.23 13:c4000000::::head [pgnls start_epoch 85 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e85) -- 0x55d9d8272c00 con 0x55d9d8240c00 2024-03-31T15:11:19.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.519+0000 7f586a000640 1 -- 172.21.15.16:0/2582117395 <== osd.1 v2:172.21.15.16:6808/1398737533 9 ==== osd_op_reply(50 [pgnls start_epoch 85 out=49b] v81'7 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55d9d7f2a000 con 0x55d9d8240c00 2024-03-31T15:11:19.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.519+0000 7f5864600640 1 -- 172.21.15.16:0/2582117395 --> [v2:172.21.15.89:6808/1796778630,v1:172.21.15.89:6811/1796778630] -- osd_op(unknown.0.0:51 13.13 13:c8000000::::head [pgnls start_epoch 85 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e85) -- 0x55d9d8272000 con 0x55d9d8d73c00 2024-03-31T15:11:19.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.519+0000 7f586a000640 1 -- 172.21.15.16:0/2582117395 <== osd.7 v2:172.21.15.89:6808/1796778630 7 ==== osd_op_reply(51 [pgnls start_epoch 85 out=49b] v81'15 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55d9d7f2a000 con 0x55d9d8d73c00 2024-03-31T15:11:19.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.519+0000 7f5863c00640 1 -- 172.21.15.16:0/2582117395 --> [v2:172.21.15.89:6808/1796778630,v1:172.21.15.89:6811/1796778630] -- osd_op(unknown.0.0:52 13.33 13:cc000000::::head [pgnls start_epoch 85 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e85) -- 0x55d9d8273000 con 0x55d9d8d73c00 2024-03-31T15:11:19.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.519+0000 7f586a000640 1 -- 172.21.15.16:0/2582117395 <== osd.7 v2:172.21.15.89:6808/1796778630 8 ==== osd_op_reply(52 [pgnls start_epoch 85 out=49b] v81'10 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55d9d7f2a000 con 0x55d9d8d73c00 2024-03-31T15:11:19.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.519+0000 7f5864600640 1 -- 172.21.15.16:0/2582117395 --> [v2:172.21.15.89:6810/50395527,v1:172.21.15.89:6814/50395527] -- osd_op(unknown.0.0:53 13.b 13:d0000000::::head [pgnls start_epoch 85 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e85) -- 0x55d9d8273400 con 0x55d9d825cc00 2024-03-31T15:11:19.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.523+0000 7f586a000640 1 -- 172.21.15.16:0/2582117395 <== osd.5 v2:172.21.15.89:6810/50395527 5 ==== osd_op_reply(53 [pgnls start_epoch 85 out=49b] v81'8 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55d9d7f2a000 con 0x55d9d825cc00 2024-03-31T15:11:19.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.523+0000 7f5863c00640 1 -- 172.21.15.16:0/2582117395 --> [v2:172.21.15.16:6819/3018526646,v1:172.21.15.16:6822/3018526646] -- osd_op(unknown.0.0:54 13.2b 13:d4000000::::head [pgnls start_epoch 85 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e85) -- 0x55d9d8273c00 con 0x55d9d8241400 2024-03-31T15:11:19.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.523+0000 7f586aa00640 1 -- 172.21.15.16:0/2582117395 <== osd.2 v2:172.21.15.16:6819/3018526646 9 ==== osd_op_reply(54 [pgnls start_epoch 85 out=49b] v81'11 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55d9d7f2a500 con 0x55d9d8241400 2024-03-31T15:11:19.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.523+0000 7f5864600640 1 -- 172.21.15.16:0/2582117395 --> [v2:172.21.15.89:6800/1856844856,v1:172.21.15.89:6801/1856844856] -- osd_op(unknown.0.0:55 13.1b 13:d8000000::::head [pgnls start_epoch 85 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e85) -- 0x55d9d8272800 con 0x55d9d8240800 2024-03-31T15:11:19.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.523+0000 7f5869600640 1 -- 172.21.15.16:0/2582117395 <== osd.4 v2:172.21.15.89:6800/1856844856 9 ==== osd_op_reply(55 [pgnls start_epoch 85 out=49b] v81'8 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55d9d7f2a780 con 0x55d9d8240800 2024-03-31T15:11:19.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.523+0000 7f5863c00640 1 -- 172.21.15.16:0/2582117395 --> [v2:172.21.15.89:6800/1856844856,v1:172.21.15.89:6801/1856844856] -- osd_op(unknown.0.0:56 13.3b 13:dc000000::::head [pgnls start_epoch 85 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e85) -- 0x55d9d8240400 con 0x55d9d8240800 2024-03-31T15:11:19.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.523+0000 7f5869600640 1 -- 172.21.15.16:0/2582117395 <== osd.4 v2:172.21.15.89:6800/1856844856 10 ==== osd_op_reply(56 [pgnls start_epoch 85 out=49b] v81'4 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55d9d7f2a780 con 0x55d9d8240800 2024-03-31T15:11:19.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.523+0000 7f5864600640 1 -- 172.21.15.16:0/2582117395 --> [v2:172.21.15.89:6808/1796778630,v1:172.21.15.89:6811/1796778630] -- osd_op(unknown.0.0:57 13.7 13:e0000000::::head [pgnls start_epoch 85 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e85) -- 0x55d9d8241800 con 0x55d9d8d73c00 2024-03-31T15:11:19.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.523+0000 7f586a000640 1 -- 172.21.15.16:0/2582117395 <== osd.7 v2:172.21.15.89:6808/1796778630 9 ==== osd_op_reply(57 [pgnls start_epoch 85 out=81b] v81'6 uv6 ondisk = 1) ==== 144+0+81 (crc 0 0 0) 0x55d9d7f2a000 con 0x55d9d8d73c00 2024-03-31T15:11:19.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.523+0000 7f586be4b8c0 1 -- 172.21.15.16:0/2582117395 --> [v2:172.21.15.89:6808/1796778630,v1:172.21.15.89:6811/1796778630] -- osd_op(unknown.0.0:58 13.7 13:e2e07930:::10000000001.00000000:head [stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e85) -- 0x55d9d8ddd400 con 0x55d9d8d73c00 2024-03-31T15:11:19.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.523+0000 7f586a000640 1 -- 172.21.15.16:0/2582117395 <== osd.7 v2:172.21.15.89:6808/1796778630 10 ==== osd_op_reply(58 10000000001.00000000 [stat out=16b] v0'0 uv6 ondisk = 0) ==== 164+0+16 (crc 0 0 0) 0x55d9d7f2a000 con 0x55d9d8d73c00 2024-03-31T15:11:19.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.523+0000 7f586be4b8c0 10 datascan.operator(): handling object 1099511627777.0 2024-03-31T15:11:19.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.523+0000 7f586be4b8c0 1 -- 172.21.15.16:0/2582117395 --> [v2:172.21.15.89:6808/1796778630,v1:172.21.15.89:6811/1796778630] -- osd_op(unknown.0.0:59 13.7 13:e2e07930:::10000000001.00000000:head [call cephfs.accumulate_inode_metadata in=112b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85) -- 0x55d9d8ddd800 con 0x55d9d8d73c00 2024-03-31T15:11:19.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.523+0000 7f586a000640 1 -- 172.21.15.16:0/2582117395 <== osd.7 v2:172.21.15.89:6808/1796778630 11 ==== osd_op_reply(59 10000000001.00000000 [call] v85'7 uv7 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x55d9d7f2a000 con 0x55d9d8d73c00 2024-03-31T15:11:19.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.523+0000 7f586be4b8c0 1 -- 172.21.15.16:0/2582117395 --> [v2:172.21.15.16:6808/1398737533,v1:172.21.15.16:6810/1398737533] -- osd_op(unknown.0.0:60 13.27 13:e4000000::::head [pgnls start_epoch 85 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e85) -- 0x55d9d8dddc00 con 0x55d9d8240c00 2024-03-31T15:11:19.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.523+0000 7f586a000640 1 -- 172.21.15.16:0/2582117395 <== osd.1 v2:172.21.15.16:6808/1398737533 10 ==== osd_op_reply(60 [pgnls start_epoch 85 out=49b] v81'2 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55d9d7f2a000 con 0x55d9d8240c00 2024-03-31T15:11:19.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.523+0000 7f5864600640 1 -- 172.21.15.16:0/2582117395 --> [v2:172.21.15.89:6810/50395527,v1:172.21.15.89:6814/50395527] -- osd_op(unknown.0.0:61 13.17 13:e8000000::::head [pgnls start_epoch 85 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e85) -- 0x55d9d8ddd000 con 0x55d9d825cc00 2024-03-31T15:11:19.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.523+0000 7f586a000640 1 -- 172.21.15.16:0/2582117395 <== osd.5 v2:172.21.15.89:6810/50395527 6 ==== osd_op_reply(61 [pgnls start_epoch 85 out=49b] v81'9 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55d9d7f2a000 con 0x55d9d825cc00 2024-03-31T15:11:19.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.523+0000 7f5863c00640 1 -- 172.21.15.16:0/2582117395 --> [v2:172.21.15.89:6810/50395527,v1:172.21.15.89:6814/50395527] -- osd_op(unknown.0.0:62 13.37 13:ec000000::::head [pgnls start_epoch 85 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e85) -- 0x55d9d8c43400 con 0x55d9d825cc00 2024-03-31T15:11:19.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.527+0000 7f586a000640 1 -- 172.21.15.16:0/2582117395 <== osd.5 v2:172.21.15.89:6810/50395527 7 ==== osd_op_reply(62 [pgnls start_epoch 85 out=49b] v81'11 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55d9d7f2a000 con 0x55d9d825cc00 2024-03-31T15:11:19.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.527+0000 7f5864600640 1 -- 172.21.15.16:0/2582117395 --> [v2:172.21.15.89:6809/3470471752,v1:172.21.15.89:6812/3470471752] -- osd_op(unknown.0.0:63 13.f 13:f0000000::::head [pgnls start_epoch 85 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e85) -- 0x55d9d825d400 con 0x55d9d8240000 2024-03-31T15:11:19.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.527+0000 7f586aa00640 1 -- 172.21.15.16:0/2582117395 <== osd.6 v2:172.21.15.89:6809/3470471752 9 ==== osd_op_reply(63 [pgnls start_epoch 85 out=49b] v81'9 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55d9d7f2a500 con 0x55d9d8240000 2024-03-31T15:11:19.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.527+0000 7f5863c00640 1 -- 172.21.15.16:0/2582117395 --> [v2:172.21.15.89:6800/1856844856,v1:172.21.15.89:6801/1856844856] -- osd_op(unknown.0.0:64 13.2f 13:f4000000::::head [pgnls start_epoch 85 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e85) -- 0x55d9d825d000 con 0x55d9d8240800 2024-03-31T15:11:19.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.527+0000 7f5869600640 1 -- 172.21.15.16:0/2582117395 <== osd.4 v2:172.21.15.89:6800/1856844856 11 ==== osd_op_reply(64 [pgnls start_epoch 85 out=49b] v81'8 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55d9d7f2a780 con 0x55d9d8240800 2024-03-31T15:11:19.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.527+0000 7f5864600640 1 -- 172.21.15.16:0/2582117395 --> [v2:172.21.15.89:6809/3470471752,v1:172.21.15.89:6812/3470471752] -- osd_op(unknown.0.0:65 13.1f 13:f8000000::::head [pgnls start_epoch 85 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e85) -- 0x55d9d825d800 con 0x55d9d8240000 2024-03-31T15:11:19.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.527+0000 7f586aa00640 1 -- 172.21.15.16:0/2582117395 <== osd.6 v2:172.21.15.89:6809/3470471752 10 ==== osd_op_reply(65 [pgnls start_epoch 85 out=49b] v81'6 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55d9d7f2a500 con 0x55d9d8240000 2024-03-31T15:11:19.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.527+0000 7f5863c00640 1 -- 172.21.15.16:0/2582117395 --> [v2:172.21.15.89:6809/3470471752,v1:172.21.15.89:6812/3470471752] -- osd_op(unknown.0.0:66 13.3f 13:fc000000::::head [pgnls start_epoch 85 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e85) -- 0x55d9d8272400 con 0x55d9d8240000 2024-03-31T15:11:19.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.527+0000 7f586aa00640 1 -- 172.21.15.16:0/2582117395 <== osd.6 v2:172.21.15.89:6809/3470471752 11 ==== osd_op_reply(66 [pgnls start_epoch 85 out=49b] v81'8 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55d9d7f2a500 con 0x55d9d8240000 2024-03-31T15:11:19.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.527+0000 7f586be4b8c0 1 -- 172.21.15.16:0/2582117395 >> [v2:172.21.15.16:6800/932712244,v1:172.21.15.16:6801/932712244] conn(0x55d9d825c000 msgr2=0x55d9d8247b80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:11:19.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.527+0000 7f586be4b8c0 1 --2- 172.21.15.16:0/2582117395 >> [v2:172.21.15.16:6800/932712244,v1:172.21.15.16:6801/932712244] conn(0x55d9d825c000 0x55d9d8247b80 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:19.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.527+0000 7f586be4b8c0 1 -- 172.21.15.16:0/2582117395 >> [v2:172.21.15.16:6808/1398737533,v1:172.21.15.16:6810/1398737533] conn(0x55d9d8240c00 msgr2=0x55d9d8246580 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:11:19.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.527+0000 7f586be4b8c0 1 --2- 172.21.15.16:0/2582117395 >> [v2:172.21.15.16:6808/1398737533,v1:172.21.15.16:6810/1398737533] conn(0x55d9d8240c00 0x55d9d8246580 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:19.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.527+0000 7f586be4b8c0 1 -- 172.21.15.16:0/2582117395 >> [v2:172.21.15.16:6819/3018526646,v1:172.21.15.16:6822/3018526646] conn(0x55d9d8241400 msgr2=0x55d9d8247600 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:11:19.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.527+0000 7f586be4b8c0 1 --2- 172.21.15.16:0/2582117395 >> [v2:172.21.15.16:6819/3018526646,v1:172.21.15.16:6822/3018526646] conn(0x55d9d8241400 0x55d9d8247600 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:19.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.527+0000 7f586be4b8c0 1 -- 172.21.15.16:0/2582117395 >> [v2:172.21.15.16:6809/3360352974,v1:172.21.15.16:6812/3360352974] conn(0x55d9d8241000 msgr2=0x55d9d8248100 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:11:19.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.527+0000 7f586be4b8c0 1 --2- 172.21.15.16:0/2582117395 >> [v2:172.21.15.16:6809/3360352974,v1:172.21.15.16:6812/3360352974] conn(0x55d9d8241000 0x55d9d8248100 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:19.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.527+0000 7f586be4b8c0 1 -- 172.21.15.16:0/2582117395 >> [v2:172.21.15.89:6800/1856844856,v1:172.21.15.89:6801/1856844856] conn(0x55d9d8240800 msgr2=0x55d9d8246b00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:11:19.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.527+0000 7f586be4b8c0 1 --2- 172.21.15.16:0/2582117395 >> [v2:172.21.15.89:6800/1856844856,v1:172.21.15.89:6801/1856844856] conn(0x55d9d8240800 0x55d9d8246b00 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:19.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.527+0000 7f586be4b8c0 1 -- 172.21.15.16:0/2582117395 >> [v2:172.21.15.89:6810/50395527,v1:172.21.15.89:6814/50395527] conn(0x55d9d825cc00 msgr2=0x55d9d8247080 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:11:19.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.527+0000 7f586be4b8c0 1 --2- 172.21.15.16:0/2582117395 >> [v2:172.21.15.89:6810/50395527,v1:172.21.15.89:6814/50395527] conn(0x55d9d825cc00 0x55d9d8247080 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:19.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.527+0000 7f586be4b8c0 1 -- 172.21.15.16:0/2582117395 >> [v2:172.21.15.89:6809/3470471752,v1:172.21.15.89:6812/3470471752] conn(0x55d9d8240000 msgr2=0x55d9d8246000 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:11:19.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.527+0000 7f586be4b8c0 1 --2- 172.21.15.16:0/2582117395 >> [v2:172.21.15.89:6809/3470471752,v1:172.21.15.89:6812/3470471752] conn(0x55d9d8240000 0x55d9d8246000 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:19.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.527+0000 7f586be4b8c0 1 -- 172.21.15.16:0/2582117395 >> [v2:172.21.15.89:6808/1796778630,v1:172.21.15.89:6811/1796778630] conn(0x55d9d8d73c00 msgr2=0x55d9d8c0e680 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:11:19.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.527+0000 7f586be4b8c0 1 --2- 172.21.15.16:0/2582117395 >> [v2:172.21.15.89:6808/1796778630,v1:172.21.15.89:6811/1796778630] conn(0x55d9d8d73c00 0x55d9d8c0e680 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:19.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.527+0000 7f5869600640 1 -- 172.21.15.16:0/2582117395 reap_dead start 2024-03-31T15:11:19.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.527+0000 7f5869600640 1 -- 172.21.15.16:0/2582117395 reap_dead start 2024-03-31T15:11:19.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.527+0000 7f586be4b8c0 1 -- 172.21.15.16:0/2582117395 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x55d9d8ddcc00 msgr2=0x55d9d8c0e100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:11:19.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.527+0000 7f5869600640 1 -- 172.21.15.16:0/2582117395 reap_dead start 2024-03-31T15:11:19.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.527+0000 7f586be4b8c0 1 --2- 172.21.15.16:0/2582117395 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x55d9d8ddcc00 0x55d9d8c0e100 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x55d9d8d74c30 tx=0x55d9d8bdaf60 comp rx=0 tx=0).stop 2024-03-31T15:11:19.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.527+0000 7f5869600640 1 -- 172.21.15.16:0/2582117395 reap_dead start 2024-03-31T15:11:19.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.527+0000 7f586be4b8c0 1 -- 172.21.15.16:0/2582117395 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d9d8d73800 msgr2=0x55d9d8c0db80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:11:19.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.527+0000 7f586be4b8c0 1 --2- 172.21.15.16:0/2582117395 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d9d8d73800 0x55d9d8c0db80 secure :-1 s=READY pgs=1251 cs=0 l=1 rev1=1 crypto rx=0x55d9d8cbaff0 tx=0x55d9d7fc8000 comp rx=0 tx=0).stop 2024-03-31T15:11:19.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.527+0000 7f586be4b8c0 1 -- 172.21.15.16:0/2582117395 shutdown_connections 2024-03-31T15:11:19.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.527+0000 7f586be4b8c0 1 --2- 172.21.15.16:0/2582117395 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x55d9d8ddcc00 0x55d9d8c0e100 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:19.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.527+0000 7f586be4b8c0 1 --2- 172.21.15.16:0/2582117395 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d9d8d73800 0x55d9d8c0db80 unknown :-1 s=CLOSED pgs=1251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:19.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.527+0000 7f586be4b8c0 1 -- 172.21.15.16:0/2582117395 >> 172.21.15.16:0/2582117395 conn(0x55d9d7e54c00 msgr2=0x55d9d7e83800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:11:19.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.527+0000 7f586be4b8c0 1 -- 172.21.15.16:0/2582117395 shutdown_connections 2024-03-31T15:11:19.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.527+0000 7f586be4b8c0 1 -- 172.21.15.16:0/2582117395 wait complete. 2024-03-31T15:11:19.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.527+0000 7f586be4b8c0 1 -- 172.21.15.16:0/2331561691 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d9d7e55800 msgr2=0x55d9d8c0c000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:11:19.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.527+0000 7f586be4b8c0 1 --2- 172.21.15.16:0/2331561691 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d9d7e55800 0x55d9d8c0c000 secure :-1 s=READY pgs=1250 cs=0 l=1 rev1=1 crypto rx=0x55d9d8cbb8f0 tx=0x55d9d8bdca80 comp rx=0 tx=0).stop 2024-03-31T15:11:19.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.527+0000 7f586be4b8c0 1 -- 172.21.15.16:0/2331561691 shutdown_connections 2024-03-31T15:11:19.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.527+0000 7f586be4b8c0 1 --2- 172.21.15.16:0/2331561691 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55d9d7e55000 0x55d9d8c0cb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:19.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.527+0000 7f586be4b8c0 1 --2- 172.21.15.16:0/2331561691 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d9d7e55400 0x55d9d8c0c580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:19.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.527+0000 7f586be4b8c0 1 --2- 172.21.15.16:0/2331561691 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d9d7e55800 0x55d9d8c0c000 unknown :-1 s=CLOSED pgs=1250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:19.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.527+0000 7f586be4b8c0 1 -- 172.21.15.16:0/2331561691 >> 172.21.15.16:0/2331561691 conn(0x55d9d7e55c00 msgr2=0x55d9d7e83000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:11:19.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.527+0000 7f586be4b8c0 1 -- 172.21.15.16:0/2331561691 shutdown_connections 2024-03-31T15:11:19.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.527+0000 7f586be4b8c0 1 -- 172.21.15.16:0/2331561691 wait complete. 2024-03-31T15:11:19.526 DEBUG:tasks.cephfs.filesystem:Ran ['cephfs-data-scan', '--debug-mds=20', '--debug-ms=1', '--debug-objecter=1', 'scan_extents', 'cephfs_data'] in time 0:00:00.105079, result: 2024-03-31T15:11:19.526 DEBUG:teuthology.orchestra.run.smithi016:> cephfs-data-scan --debug-mds=20 --debug-ms=1 --debug-objecter=1 scan_inodes cephfs_data 2024-03-31T15:11:19.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.623+0000 7feb36c488c0 1 -- start start 2024-03-31T15:11:19.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.623+0000 7feb36c488c0 1 Processor -- start 2024-03-31T15:11:19.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.623+0000 7feb36c488c0 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55958c1f5000 0x55958cfc4000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:19.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.623+0000 7feb36c488c0 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55958c1f5400 0x55958cfc4580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:19.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.623+0000 7feb36c488c0 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55958c1f5800 0x55958cfc4b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:19.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.623+0000 7feb36c488c0 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55958cfda600 con 0x55958c1f5000 2024-03-31T15:11:19.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.623+0000 7feb36c488c0 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55958cfda480 con 0x55958c1f5800 2024-03-31T15:11:19.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.623+0000 7feb36c488c0 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x55958cfda300 con 0x55958c1f5400 2024-03-31T15:11:19.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.623+0000 7feb34200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55958c1f5000 0x55958cfc4000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:19.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.623+0000 7feb34200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55958c1f5000 0x55958cfc4000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56198/0 (socket says 172.21.15.16:56198) 2024-03-31T15:11:19.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.623+0000 7feb34200640 1 -- 172.21.15.16:0/1562624489 learned_addr learned my addr 172.21.15.16:0/1562624489 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:11:19.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.623+0000 7feb33800640 1 --2- 172.21.15.16:0/1562624489 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55958c1f5400 0x55958cfc4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:19.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.623+0000 7feb34c00640 1 --2- 172.21.15.16:0/1562624489 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55958c1f5800 0x55958cfc4b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:19.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.627+0000 7feb34200640 1 -- 172.21.15.16:0/1562624489 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55958c1f5400 msgr2=0x55958cfc4580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:11:19.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.627+0000 7feb34200640 1 --2- 172.21.15.16:0/1562624489 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55958c1f5400 0x55958cfc4580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:19.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.627+0000 7feb34200640 1 -- 172.21.15.16:0/1562624489 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55958c1f5800 msgr2=0x55958cfc4b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:11:19.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.627+0000 7feb34200640 1 --2- 172.21.15.16:0/1562624489 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55958c1f5800 0x55958cfc4b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:19.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.627+0000 7feb34200640 1 -- 172.21.15.16:0/1562624489 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55958c21a1e0 con 0x55958c1f5000 2024-03-31T15:11:19.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.627+0000 7feb33800640 1 --2- 172.21.15.16:0/1562624489 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55958c1f5400 0x55958cfc4580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:11:19.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.627+0000 7feb34c00640 1 --2- 172.21.15.16:0/1562624489 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55958c1f5800 0x55958cfc4b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:11:19.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.627+0000 7feb34200640 1 --2- 172.21.15.16:0/1562624489 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55958c1f5000 0x55958cfc4000 secure :-1 s=READY pgs=1253 cs=0 l=1 rev1=1 crypto rx=0x55958d059a70 tx=0x55958cfd4600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d32e9b6de529047 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:11:19.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.627+0000 7feb31000640 1 -- 172.21.15.16:0/1562624489 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x55958c22a4e0 con 0x55958c1f5000 2024-03-31T15:11:19.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.627+0000 7feb31000640 1 -- 172.21.15.16:0/1562624489 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x55958c240c40 con 0x55958c1f5000 2024-03-31T15:11:19.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.627+0000 7feb31000640 1 -- 172.21.15.16:0/1562624489 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x55958c22ab60 con 0x55958c1f5000 2024-03-31T15:11:19.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.627+0000 7feb36c488c0 1 -- 172.21.15.16:0/1562624489 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55958c21b2c0 con 0x55958c1f5000 2024-03-31T15:11:19.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.627+0000 7feb36c488c0 1 -- 172.21.15.16:0/1562624489 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({fsmap=0}) -- 0x55958c21ba40 con 0x55958c1f5000 2024-03-31T15:11:19.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.627+0000 7feb36c488c0 4 waiting for MDS map... 2024-03-31T15:11:19.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.627+0000 7feb31000640 1 -- 172.21.15.16:0/1562624489 <== mon.0 v2:172.21.15.16:3300/0 4 ==== osd_map(85..85 src has 1..85) ==== 5910+0+0 (secure 0 0 0) 0x55958cfbf000 con 0x55958c1f5000 2024-03-31T15:11:19.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.627+0000 7feb31000640 1 -- 172.21.15.16:0/1562624489 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 69) ==== 2885+0+0 (secure 0 0 0) 0x55958c24b180 con 0x55958c1f5000 2024-03-31T15:11:19.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.627+0000 7feb36c488c0 4 Got MDS map 69 2024-03-31T15:11:19.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.627+0000 7feb36c488c0 4 datascan.main: Using metadata pool output 2024-03-31T15:11:19.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.627+0000 7feb36c488c0 4 datascan.main: connecting to RADOS... 2024-03-31T15:11:19.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.627+0000 7feb36c488c0 1 Processor -- start 2024-03-31T15:11:19.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.627+0000 7feb36c488c0 1 -- start start 2024-03-31T15:11:19.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.627+0000 7feb36c488c0 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55958d11f400 0x55958cfc5080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:19.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.627+0000 7feb36c488c0 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55958d11f800 0x55958cfc5600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:19.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.627+0000 7feb36c488c0 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55958d11fc00 0x55958cfc5b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:19.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.627+0000 7feb36c488c0 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55958cfda180 con 0x55958d11f400 2024-03-31T15:11:19.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.627+0000 7feb36c488c0 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55958c254480 con 0x55958d11f800 2024-03-31T15:11:19.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.627+0000 7feb36c488c0 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x55958c254900 con 0x55958d11fc00 2024-03-31T15:11:19.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.627+0000 7feb34c00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55958d11f400 0x55958cfc5080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:19.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.627+0000 7feb34c00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55958d11f400 0x55958cfc5080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56202/0 (socket says 172.21.15.16:56202) 2024-03-31T15:11:19.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.627+0000 7feb34c00640 1 -- 172.21.15.16:0/1985651298 learned_addr learned my addr 172.21.15.16:0/1985651298 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:11:19.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.627+0000 7feb34200640 1 --2- 172.21.15.16:0/1985651298 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55958d11f800 0x55958cfc5600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:19.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.627+0000 7feb33800640 1 --2- 172.21.15.16:0/1985651298 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55958d11fc00 0x55958cfc5b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:19.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.627+0000 7feb34c00640 1 -- 172.21.15.16:0/1985651298 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55958d11fc00 msgr2=0x55958cfc5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:11:19.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.627+0000 7feb34c00640 1 --2- 172.21.15.16:0/1985651298 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55958d11fc00 0x55958cfc5b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:19.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.627+0000 7feb34c00640 1 -- 172.21.15.16:0/1985651298 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55958d11f800 msgr2=0x55958cfc5600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:11:19.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.627+0000 7feb34c00640 1 --2- 172.21.15.16:0/1985651298 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55958d11f800 0x55958cfc5600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:19.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.627+0000 7feb34c00640 1 -- 172.21.15.16:0/1985651298 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55958c21bc20 con 0x55958d11f400 2024-03-31T15:11:19.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.627+0000 7feb34c00640 1 --2- 172.21.15.16:0/1985651298 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55958d11f400 0x55958cfc5080 secure :-1 s=READY pgs=1254 cs=0 l=1 rev1=1 crypto rx=0x55958d126960 tx=0x55958c4e4000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54ed56427fc0546f server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:11:19.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.627+0000 7feb2e800640 1 -- 172.21.15.16:0/1985651298 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x55958c22a4e0 con 0x55958d11f400 2024-03-31T15:11:19.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.627+0000 7feb2e800640 1 -- 172.21.15.16:0/1985651298 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x55958c240a80 con 0x55958d11f400 2024-03-31T15:11:19.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.627+0000 7feb2e800640 1 -- 172.21.15.16:0/1985651298 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x55958c22ab60 con 0x55958d11f400 2024-03-31T15:11:19.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.631+0000 7feb36c488c0 1 -- 172.21.15.16:0/1985651298 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55958d11f400 msgr2=0x55958cfc5080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:11:19.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.631+0000 7feb36c488c0 1 --2- 172.21.15.16:0/1985651298 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55958d11f400 0x55958cfc5080 secure :-1 s=READY pgs=1254 cs=0 l=1 rev1=1 crypto rx=0x55958d126960 tx=0x55958c4e4000 comp rx=0 tx=0).stop 2024-03-31T15:11:19.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.631+0000 7feb36c488c0 1 -- 172.21.15.16:0/1985651298 shutdown_connections 2024-03-31T15:11:19.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.631+0000 7feb36c488c0 1 --2- 172.21.15.16:0/1985651298 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55958d11fc00 0x55958cfc5b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:19.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.631+0000 7feb36c488c0 1 --2- 172.21.15.16:0/1985651298 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55958d11f800 0x55958cfc5600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:19.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.631+0000 7feb36c488c0 1 --2- 172.21.15.16:0/1985651298 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55958d11f400 0x55958cfc5080 unknown :-1 s=CLOSED pgs=1254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:19.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.631+0000 7feb36c488c0 1 -- 172.21.15.16:0/1985651298 >> 172.21.15.16:0/1985651298 conn(0x55958d11f000 msgr2=0x55958c222000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:11:19.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.631+0000 7feb36c488c0 1 -- 172.21.15.16:0/1985651298 shutdown_connections 2024-03-31T15:11:19.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.631+0000 7feb36c488c0 1 -- 172.21.15.16:0/1985651298 wait complete. 2024-03-31T15:11:19.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.631+0000 7feb36c488c0 1 Processor -- start 2024-03-31T15:11:19.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.631+0000 7feb36c488c0 1 -- start start 2024-03-31T15:11:19.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.631+0000 7feb36c488c0 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55958d11f400 0x55958cfc5080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:19.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.631+0000 7feb36c488c0 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55958d11f800 0x55958cfc5600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:19.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.631+0000 7feb36c488c0 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55958d11fc00 0x55958cfc5b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:19.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.631+0000 7feb36c488c0 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55958cfdb380 con 0x55958d11fc00 2024-03-31T15:11:19.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.631+0000 7feb36c488c0 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55958cfda180 con 0x55958d11f800 2024-03-31T15:11:19.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.631+0000 7feb36c488c0 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x55958cfdb500 con 0x55958d11f400 2024-03-31T15:11:19.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.631+0000 7feb33800640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55958d11fc00 0x55958cfc5b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:19.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.631+0000 7feb33800640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55958d11fc00 0x55958cfc5b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56210/0 (socket says 172.21.15.16:56210) 2024-03-31T15:11:19.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.631+0000 7feb33800640 1 -- 172.21.15.16:0/3131452291 learned_addr learned my addr 172.21.15.16:0/3131452291 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:11:19.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.631+0000 7feb34200640 1 --2- 172.21.15.16:0/3131452291 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55958d11f800 0x55958cfc5600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:19.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.631+0000 7feb34c00640 1 --2- 172.21.15.16:0/3131452291 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55958d11f400 0x55958cfc5080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:19.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.631+0000 7feb33800640 1 -- 172.21.15.16:0/3131452291 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55958d11f400 msgr2=0x55958cfc5080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:11:19.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.631+0000 7feb33800640 1 --2- 172.21.15.16:0/3131452291 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55958d11f400 0x55958cfc5080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:19.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.631+0000 7feb33800640 1 -- 172.21.15.16:0/3131452291 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55958d11f800 msgr2=0x55958cfc5600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:11:19.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.631+0000 7feb33800640 1 --2- 172.21.15.16:0/3131452291 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55958d11f800 0x55958cfc5600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:19.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.631+0000 7feb33800640 1 -- 172.21.15.16:0/3131452291 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55958c21a1e0 con 0x55958d11fc00 2024-03-31T15:11:19.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.631+0000 7feb33800640 1 --2- 172.21.15.16:0/3131452291 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55958d11fc00 0x55958cfc5b80 secure :-1 s=READY pgs=1255 cs=0 l=1 rev1=1 crypto rx=0x55958d059020 tx=0x55958cfd4000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:11:19.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.631+0000 7feb2ca00640 1 -- 172.21.15.16:0/3131452291 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x55958c22a9c0 con 0x55958d11fc00 2024-03-31T15:11:19.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.631+0000 7feb2ca00640 1 -- 172.21.15.16:0/3131452291 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x55958c241500 con 0x55958d11fc00 2024-03-31T15:11:19.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.631+0000 7feb36c488c0 1 -- 172.21.15.16:0/3131452291 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55958c21b2c0 con 0x55958d11fc00 2024-03-31T15:11:19.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.631+0000 7feb2ca00640 1 -- 172.21.15.16:0/3131452291 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x55958c22a4e0 con 0x55958d11fc00 2024-03-31T15:11:19.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.631+0000 7feb36c488c0 1 -- 172.21.15.16:0/3131452291 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55958c21ba40 con 0x55958d11fc00 2024-03-31T15:11:19.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.631+0000 7feb36c488c0 4 datascan.init: resolving metadata pool 12 2024-03-31T15:11:19.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.635+0000 7feb2ca00640 1 -- 172.21.15.16:0/3131452291 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x55958c4f4c00 con 0x55958d11fc00 2024-03-31T15:11:19.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.635+0000 7feb2ca00640 1 --2- 172.21.15.16:0/3131452291 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x55958d17c800 0x55958cfc6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:19.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.635+0000 7feb2ca00640 1 -- 172.21.15.16:0/3131452291 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 5910+0+0 (secure 0 0 0) 0x55958c4e6a00 con 0x55958d11fc00 2024-03-31T15:11:19.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.635+0000 7feb36c488c0 4 datascan.init: found metadata pool 'cephfs_metadata' 2024-03-31T15:11:19.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.635+0000 7feb36c488c0 4 datascan.main: data pool 'cephfs_data' has ID 13 2024-03-31T15:11:19.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.635+0000 7feb36c488c0 4 datascan.main: opening data pool 'cephfs_data' 2024-03-31T15:11:19.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.635+0000 7feb36c488c0 1 --2- 172.21.15.16:0/3131452291 >> [v2:172.21.15.89:6810/50395527,v1:172.21.15.89:6814/50395527] conn(0x55958d17cc00 0x55958cfc6680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:19.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.635+0000 7feb34c00640 1 --2- 172.21.15.16:0/3131452291 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x55958d17c800 0x55958cfc6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:19.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.635+0000 7feb36c488c0 1 -- 172.21.15.16:0/3131452291 --> [v2:172.21.15.89:6810/50395527,v1:172.21.15.89:6814/50395527] -- osd_op(unknown.0.0:1 12.14 12:287034c4:::1.00000000.inode:head [stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e85) -- 0x55958d17d000 con 0x55958d17cc00 2024-03-31T15:11:19.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.635+0000 7feb34200640 1 --2- 172.21.15.16:0/3131452291 >> [v2:172.21.15.89:6810/50395527,v1:172.21.15.89:6814/50395527] conn(0x55958d17cc00 0x55958cfc6680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:19.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.635+0000 7feb34c00640 1 --2- 172.21.15.16:0/3131452291 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x55958d17c800 0x55958cfc6100 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x55958d126960 tx=0x55958cfd45a0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:11:19.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.635+0000 7feb34200640 1 --2- 172.21.15.16:0/3131452291 >> [v2:172.21.15.89:6810/50395527,v1:172.21.15.89:6814/50395527] conn(0x55958d17cc00 0x55958cfc6680 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:11:19.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.635+0000 7feb34200640 1 -- 172.21.15.16:0/3131452291 <== osd.5 v2:172.21.15.89:6810/50395527 1 ==== osd_op_reply(1 1.00000000.inode [stat out=16b] v0'0 uv24 ondisk = 0) ==== 160+0+16 (crc 0 0 0) 0x55958c368000 con 0x55958d17cc00 2024-03-31T15:11:19.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.635+0000 7feb36c488c0 1 --2- 172.21.15.16:0/3131452291 >> [v2:172.21.15.89:6809/3470471752,v1:172.21.15.89:6812/3470471752] conn(0x55958d17d400 0x55958cfc6c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:19.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.635+0000 7feb36c488c0 1 -- 172.21.15.16:0/3131452291 --> [v2:172.21.15.89:6809/3470471752,v1:172.21.15.89:6812/3470471752] -- osd_op(unknown.0.0:2 12.1f 12:fe07bba1:::100.00000000.inode:head [stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e85) -- 0x55958d17d800 con 0x55958d17d400 2024-03-31T15:11:19.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.635+0000 7feb34c00640 1 --2- 172.21.15.16:0/3131452291 >> [v2:172.21.15.89:6809/3470471752,v1:172.21.15.89:6812/3470471752] conn(0x55958d17d400 0x55958cfc6c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:19.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.635+0000 7feb34c00640 1 --2- 172.21.15.16:0/3131452291 >> [v2:172.21.15.89:6809/3470471752,v1:172.21.15.89:6812/3470471752] conn(0x55958d17d400 0x55958cfc6c00 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:11:19.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.635+0000 7feb34c00640 1 -- 172.21.15.16:0/3131452291 <== osd.6 v2:172.21.15.89:6809/3470471752 1 ==== osd_op_reply(2 100.00000000.inode [stat out=16b] v0'0 uv3 ondisk = 0) ==== 162+0+16 (crc 0 0 0) 0x55958c368500 con 0x55958d17d400 2024-03-31T15:11:19.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.639+0000 7feb36c488c0 1 --2- 172.21.15.16:0/3131452291 >> [v2:172.21.15.89:6808/1796778630,v1:172.21.15.89:6811/1796778630] conn(0x55958d17dc00 0x55958cfc7180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:19.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.639+0000 7feb36c488c0 1 -- 172.21.15.16:0/3131452291 --> [v2:172.21.15.89:6808/1796778630,v1:172.21.15.89:6811/1796778630] -- osd_op(unknown.0.0:3 13.0 13:00000000::::head [pgnls-filter start_epoch 85 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e85) -- 0x55958c5e2000 con 0x55958d17dc00 2024-03-31T15:11:19.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.639+0000 7feb34200640 1 --2- 172.21.15.16:0/3131452291 >> [v2:172.21.15.89:6808/1796778630,v1:172.21.15.89:6811/1796778630] conn(0x55958d17dc00 0x55958cfc7180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:19.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.639+0000 7feb34200640 1 --2- 172.21.15.16:0/3131452291 >> [v2:172.21.15.89:6808/1796778630,v1:172.21.15.89:6811/1796778630] conn(0x55958d17dc00 0x55958cfc7180 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:11:19.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.639+0000 7feb34200640 1 -- 172.21.15.16:0/3131452291 <== osd.7 v2:172.21.15.89:6808/1796778630 1 ==== osd_op_reply(3 [pgnls-filter start_epoch 85 out=49b] v81'8 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55958c368000 con 0x55958d17dc00 2024-03-31T15:11:19.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.639+0000 7feb2de00640 1 -- 172.21.15.16:0/3131452291 --> [v2:172.21.15.89:6809/3470471752,v1:172.21.15.89:6812/3470471752] -- osd_op(unknown.0.0:4 13.20 13:04000000::::head [pgnls-filter start_epoch 85 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e85) -- 0x55958c5e3800 con 0x55958d17d400 2024-03-31T15:11:19.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.639+0000 7feb34c00640 1 -- 172.21.15.16:0/3131452291 <== osd.6 v2:172.21.15.89:6809/3470471752 2 ==== osd_op_reply(4 [pgnls-filter start_epoch 85 out=49b] v81'9 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55958c368500 con 0x55958d17d400 2024-03-31T15:11:19.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.639+0000 7feb2e800640 1 --2- 172.21.15.16:0/3131452291 >> [v2:172.21.15.16:6808/1398737533,v1:172.21.15.16:6810/1398737533] conn(0x55958c5e3c00 0x55958c5f2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:19.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.639+0000 7feb2e800640 1 -- 172.21.15.16:0/3131452291 --> [v2:172.21.15.16:6808/1398737533,v1:172.21.15.16:6810/1398737533] -- osd_op(unknown.0.0:5 13.10 13:08000000::::head [pgnls-filter start_epoch 85 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e85) -- 0x55958c5f6000 con 0x55958c5e3c00 2024-03-31T15:11:19.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.639+0000 7feb33800640 1 --2- 172.21.15.16:0/3131452291 >> [v2:172.21.15.16:6808/1398737533,v1:172.21.15.16:6810/1398737533] conn(0x55958c5e3c00 0x55958c5f2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:19.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.639+0000 7feb33800640 1 --2- 172.21.15.16:0/3131452291 >> [v2:172.21.15.16:6808/1398737533,v1:172.21.15.16:6810/1398737533] conn(0x55958c5e3c00 0x55958c5f2000 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:11:19.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.639+0000 7feb33800640 1 -- 172.21.15.16:0/3131452291 <== osd.1 v2:172.21.15.16:6808/1398737533 1 ==== osd_op_reply(5 [pgnls-filter start_epoch 85 out=49b] v81'6 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55958c368780 con 0x55958c5e3c00 2024-03-31T15:11:19.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.639+0000 7feb2de00640 1 --2- 172.21.15.16:0/3131452291 >> [v2:172.21.15.89:6800/1856844856,v1:172.21.15.89:6801/1856844856] conn(0x55958c5f6800 0x55958c5f2580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:19.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.639+0000 7feb2de00640 1 -- 172.21.15.16:0/3131452291 --> [v2:172.21.15.89:6800/1856844856,v1:172.21.15.89:6801/1856844856] -- osd_op(unknown.0.0:6 13.30 13:0c000000::::head [pgnls-filter start_epoch 85 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e85) -- 0x55958c5f6c00 con 0x55958c5f6800 2024-03-31T15:11:19.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.639+0000 7feb34c00640 1 --2- 172.21.15.16:0/3131452291 >> [v2:172.21.15.89:6800/1856844856,v1:172.21.15.89:6801/1856844856] conn(0x55958c5f6800 0x55958c5f2580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:19.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.643+0000 7feb34c00640 1 --2- 172.21.15.16:0/3131452291 >> [v2:172.21.15.89:6800/1856844856,v1:172.21.15.89:6801/1856844856] conn(0x55958c5f6800 0x55958c5f2580 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:11:19.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.643+0000 7feb34c00640 1 -- 172.21.15.16:0/3131452291 <== osd.4 v2:172.21.15.89:6800/1856844856 1 ==== osd_op_reply(6 [pgnls-filter start_epoch 85 out=49b] v81'9 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55958c368500 con 0x55958c5f6800 2024-03-31T15:11:19.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.643+0000 7feb2e800640 1 --2- 172.21.15.16:0/3131452291 >> [v2:172.21.15.16:6819/3018526646,v1:172.21.15.16:6822/3018526646] conn(0x55958c5f6400 0x55958c5f2b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:19.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.643+0000 7feb2e800640 1 -- 172.21.15.16:0/3131452291 --> [v2:172.21.15.16:6819/3018526646,v1:172.21.15.16:6822/3018526646] -- osd_op(unknown.0.0:7 13.8 13:10000000::::head [pgnls-filter start_epoch 85 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e85) -- 0x55958c5f7000 con 0x55958c5f6400 2024-03-31T15:11:19.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.643+0000 7feb34200640 1 --2- 172.21.15.16:0/3131452291 >> [v2:172.21.15.16:6819/3018526646,v1:172.21.15.16:6822/3018526646] conn(0x55958c5f6400 0x55958c5f2b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:19.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.643+0000 7feb34200640 1 --2- 172.21.15.16:0/3131452291 >> [v2:172.21.15.16:6819/3018526646,v1:172.21.15.16:6822/3018526646] conn(0x55958c5f6400 0x55958c5f2b00 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:11:19.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.643+0000 7feb34200640 1 -- 172.21.15.16:0/3131452291 <== osd.2 v2:172.21.15.16:6819/3018526646 1 ==== osd_op_reply(7 [pgnls-filter start_epoch 85 out=49b] v81'9 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55958c368000 con 0x55958c5f6400 2024-03-31T15:11:19.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.643+0000 7feb2de00640 1 -- 172.21.15.16:0/3131452291 --> [v2:172.21.15.16:6819/3018526646,v1:172.21.15.16:6822/3018526646] -- osd_op(unknown.0.0:8 13.28 13:14000000::::head [pgnls-filter start_epoch 85 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e85) -- 0x55958c5f7c00 con 0x55958c5f6400 2024-03-31T15:11:19.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.643+0000 7feb34200640 1 -- 172.21.15.16:0/3131452291 <== osd.2 v2:172.21.15.16:6819/3018526646 2 ==== osd_op_reply(8 [pgnls-filter start_epoch 85 out=49b] v81'6 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55958c368000 con 0x55958c5f6400 2024-03-31T15:11:19.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.643+0000 7feb2e800640 1 -- 172.21.15.16:0/3131452291 --> [v2:172.21.15.89:6808/1796778630,v1:172.21.15.89:6811/1796778630] -- osd_op(unknown.0.0:9 13.18 13:18000000::::head [pgnls-filter start_epoch 85 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e85) -- 0x55958c5f7400 con 0x55958d17dc00 2024-03-31T15:11:19.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.643+0000 7feb34200640 1 -- 172.21.15.16:0/3131452291 <== osd.7 v2:172.21.15.89:6808/1796778630 2 ==== osd_op_reply(9 [pgnls-filter start_epoch 85 out=49b] v81'8 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55958c368000 con 0x55958d17dc00 2024-03-31T15:11:19.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.643+0000 7feb2de00640 1 -- 172.21.15.16:0/3131452291 --> [v2:172.21.15.89:6810/50395527,v1:172.21.15.89:6814/50395527] -- osd_op(unknown.0.0:10 13.38 13:1c000000::::head [pgnls-filter start_epoch 85 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e85) -- 0x55958c5f7000 con 0x55958d17cc00 2024-03-31T15:11:19.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.643+0000 7feb34200640 1 -- 172.21.15.16:0/3131452291 <== osd.5 v2:172.21.15.89:6810/50395527 2 ==== osd_op_reply(10 [pgnls-filter start_epoch 85 out=49b] v81'9 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55958c368000 con 0x55958d17cc00 2024-03-31T15:11:19.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.643+0000 7feb2e800640 1 -- 172.21.15.16:0/3131452291 --> [v2:172.21.15.89:6809/3470471752,v1:172.21.15.89:6812/3470471752] -- osd_op(unknown.0.0:11 13.4 13:20000000::::head [pgnls-filter start_epoch 85 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e85) -- 0x55958c5f7800 con 0x55958d17d400 2024-03-31T15:11:19.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.643+0000 7feb34c00640 1 -- 172.21.15.16:0/3131452291 <== osd.6 v2:172.21.15.89:6809/3470471752 3 ==== osd_op_reply(11 [pgnls-filter start_epoch 85 out=49b] v81'9 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55958c368500 con 0x55958d17d400 2024-03-31T15:11:19.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.643+0000 7feb2de00640 1 -- 172.21.15.16:0/3131452291 --> [v2:172.21.15.89:6809/3470471752,v1:172.21.15.89:6812/3470471752] -- osd_op(unknown.0.0:12 13.24 13:24000000::::head [pgnls-filter start_epoch 85 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e85) -- 0x55958c5e2000 con 0x55958d17d400 2024-03-31T15:11:19.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.643+0000 7feb34c00640 1 -- 172.21.15.16:0/3131452291 <== osd.6 v2:172.21.15.89:6809/3470471752 4 ==== osd_op_reply(12 [pgnls-filter start_epoch 85 out=49b] v81'4 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55958c368500 con 0x55958d17d400 2024-03-31T15:11:19.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.643+0000 7feb2e800640 1 --2- 172.21.15.16:0/3131452291 >> [v2:172.21.15.16:6800/932712244,v1:172.21.15.16:6801/932712244] conn(0x55958c5f6c00 0x55958c5f3080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:19.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.643+0000 7feb2e800640 1 -- 172.21.15.16:0/3131452291 --> [v2:172.21.15.16:6800/932712244,v1:172.21.15.16:6801/932712244] -- osd_op(unknown.0.0:13 13.14 13:28000000::::head [pgnls-filter start_epoch 85 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e85) -- 0x55958c5f7800 con 0x55958c5f6c00 2024-03-31T15:11:19.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.643+0000 7feb33800640 1 --2- 172.21.15.16:0/3131452291 >> [v2:172.21.15.16:6800/932712244,v1:172.21.15.16:6801/932712244] conn(0x55958c5f6c00 0x55958c5f3080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:19.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.643+0000 7feb33800640 1 --2- 172.21.15.16:0/3131452291 >> [v2:172.21.15.16:6800/932712244,v1:172.21.15.16:6801/932712244] conn(0x55958c5f6c00 0x55958c5f3080 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:11:19.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.643+0000 7feb33800640 1 -- 172.21.15.16:0/3131452291 <== osd.0 v2:172.21.15.16:6800/932712244 1 ==== osd_op_reply(13 [pgnls-filter start_epoch 85 out=49b] v81'11 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55958c368780 con 0x55958c5f6c00 2024-03-31T15:11:19.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.647+0000 7feb2de00640 1 -- 172.21.15.16:0/3131452291 --> [v2:172.21.15.89:6809/3470471752,v1:172.21.15.89:6812/3470471752] -- osd_op(unknown.0.0:14 13.34 13:2c000000::::head [pgnls-filter start_epoch 85 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e85) -- 0x55958d17d000 con 0x55958d17d400 2024-03-31T15:11:19.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.647+0000 7feb34c00640 1 -- 172.21.15.16:0/3131452291 <== osd.6 v2:172.21.15.89:6809/3470471752 5 ==== osd_op_reply(14 [pgnls-filter start_epoch 85 out=49b] v81'7 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55958c368500 con 0x55958d17d400 2024-03-31T15:11:19.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.647+0000 7feb2e800640 1 -- 172.21.15.16:0/3131452291 --> [v2:172.21.15.16:6808/1398737533,v1:172.21.15.16:6810/1398737533] -- osd_op(unknown.0.0:15 13.c 13:30000000::::head [pgnls-filter start_epoch 85 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e85) -- 0x55958c5e3800 con 0x55958c5e3c00 2024-03-31T15:11:19.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.647+0000 7feb33800640 1 -- 172.21.15.16:0/3131452291 <== osd.1 v2:172.21.15.16:6808/1398737533 2 ==== osd_op_reply(15 [pgnls-filter start_epoch 85 out=49b] v81'6 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55958c368780 con 0x55958c5e3c00 2024-03-31T15:11:19.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.647+0000 7feb2de00640 1 -- 172.21.15.16:0/3131452291 --> [v2:172.21.15.16:6819/3018526646,v1:172.21.15.16:6822/3018526646] -- osd_op(unknown.0.0:16 13.2c 13:34000000::::head [pgnls-filter start_epoch 85 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e85) -- 0x55958d11e800 con 0x55958c5f6400 2024-03-31T15:11:19.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.647+0000 7feb34200640 1 -- 172.21.15.16:0/3131452291 <== osd.2 v2:172.21.15.16:6819/3018526646 3 ==== osd_op_reply(16 [pgnls-filter start_epoch 85 out=49b] v81'8 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55958c368000 con 0x55958c5f6400 2024-03-31T15:11:19.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.647+0000 7feb2e800640 1 -- 172.21.15.16:0/3131452291 --> [v2:172.21.15.89:6809/3470471752,v1:172.21.15.89:6812/3470471752] -- osd_op(unknown.0.0:17 13.1c 13:38000000::::head [pgnls-filter start_epoch 85 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e85) -- 0x55958d17d800 con 0x55958d17d400 2024-03-31T15:11:19.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.647+0000 7feb34c00640 1 -- 172.21.15.16:0/3131452291 <== osd.6 v2:172.21.15.89:6809/3470471752 6 ==== osd_op_reply(17 [pgnls-filter start_epoch 85 out=49b] v81'9 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55958c368500 con 0x55958d17d400 2024-03-31T15:11:19.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.647+0000 7feb2de00640 1 -- 172.21.15.16:0/3131452291 --> [v2:172.21.15.16:6819/3018526646,v1:172.21.15.16:6822/3018526646] -- osd_op(unknown.0.0:18 13.3c 13:3c000000::::head [pgnls-filter start_epoch 85 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e85) -- 0x55958c5f7c00 con 0x55958c5f6400 2024-03-31T15:11:19.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.647+0000 7feb34200640 1 -- 172.21.15.16:0/3131452291 <== osd.2 v2:172.21.15.16:6819/3018526646 4 ==== osd_op_reply(18 [pgnls-filter start_epoch 85 out=49b] v81'8 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55958c368000 con 0x55958c5f6400 2024-03-31T15:11:19.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.647+0000 7feb2e800640 1 -- 172.21.15.16:0/3131452291 --> [v2:172.21.15.89:6810/50395527,v1:172.21.15.89:6814/50395527] -- osd_op(unknown.0.0:19 13.2 13:40000000::::head [pgnls-filter start_epoch 85 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e85) -- 0x55958c608000 con 0x55958d17cc00 2024-03-31T15:11:19.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.647+0000 7feb34200640 1 -- 172.21.15.16:0/3131452291 <== osd.5 v2:172.21.15.89:6810/50395527 3 ==== osd_op_reply(19 [pgnls-filter start_epoch 85 out=49b] v81'8 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55958c368000 con 0x55958d17cc00 2024-03-31T15:11:19.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.647+0000 7feb2de00640 1 -- 172.21.15.16:0/3131452291 --> [v2:172.21.15.16:6808/1398737533,v1:172.21.15.16:6810/1398737533] -- osd_op(unknown.0.0:20 13.22 13:44000000::::head [pgnls-filter start_epoch 85 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e85) -- 0x55958c609400 con 0x55958c5e3c00 2024-03-31T15:11:19.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.647+0000 7feb33800640 1 -- 172.21.15.16:0/3131452291 <== osd.1 v2:172.21.15.16:6808/1398737533 3 ==== osd_op_reply(20 [pgnls-filter start_epoch 85 out=49b] v81'8 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55958c368780 con 0x55958c5e3c00 2024-03-31T15:11:19.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.647+0000 7feb2e800640 1 -- 172.21.15.16:0/3131452291 --> [v2:172.21.15.16:6800/932712244,v1:172.21.15.16:6801/932712244] -- osd_op(unknown.0.0:21 13.12 13:48000000::::head [pgnls-filter start_epoch 85 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e85) -- 0x55958c608400 con 0x55958c5f6c00 2024-03-31T15:11:19.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.647+0000 7feb33800640 1 -- 172.21.15.16:0/3131452291 <== osd.0 v2:172.21.15.16:6800/932712244 2 ==== osd_op_reply(21 [pgnls-filter start_epoch 85 out=49b] v81'10 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55958c368780 con 0x55958c5f6c00 2024-03-31T15:11:19.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.647+0000 7feb2de00640 1 -- 172.21.15.16:0/3131452291 --> [v2:172.21.15.16:6819/3018526646,v1:172.21.15.16:6822/3018526646] -- osd_op(unknown.0.0:22 13.32 13:4c000000::::head [pgnls-filter start_epoch 85 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e85) -- 0x55958c609800 con 0x55958c5f6400 2024-03-31T15:11:19.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.647+0000 7feb34200640 1 -- 172.21.15.16:0/3131452291 <== osd.2 v2:172.21.15.16:6819/3018526646 5 ==== osd_op_reply(22 [pgnls-filter start_epoch 85 out=49b] v81'8 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55958c368000 con 0x55958c5f6400 2024-03-31T15:11:19.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.647+0000 7feb2e800640 1 -- 172.21.15.16:0/3131452291 --> [v2:172.21.15.16:6819/3018526646,v1:172.21.15.16:6822/3018526646] -- osd_op(unknown.0.0:23 13.a 13:50000000::::head [pgnls-filter start_epoch 85 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e85) -- 0x55958c608800 con 0x55958c5f6400 2024-03-31T15:11:19.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.647+0000 7feb34200640 1 -- 172.21.15.16:0/3131452291 <== osd.2 v2:172.21.15.16:6819/3018526646 6 ==== osd_op_reply(23 [pgnls-filter start_epoch 85 out=49b] v81'5 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55958c368000 con 0x55958c5f6400 2024-03-31T15:11:19.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.647+0000 7feb2de00640 1 -- 172.21.15.16:0/3131452291 --> [v2:172.21.15.16:6808/1398737533,v1:172.21.15.16:6810/1398737533] -- osd_op(unknown.0.0:24 13.2a 13:54000000::::head [pgnls-filter start_epoch 85 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e85) -- 0x55958c609c00 con 0x55958c5e3c00 2024-03-31T15:11:19.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.647+0000 7feb33800640 1 -- 172.21.15.16:0/3131452291 <== osd.1 v2:172.21.15.16:6808/1398737533 4 ==== osd_op_reply(24 [pgnls-filter start_epoch 85 out=49b] v81'9 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55958c368780 con 0x55958c5e3c00 2024-03-31T15:11:19.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.647+0000 7feb2e800640 1 -- 172.21.15.16:0/3131452291 --> [v2:172.21.15.16:6808/1398737533,v1:172.21.15.16:6810/1398737533] -- osd_op(unknown.0.0:25 13.1a 13:58000000::::head [pgnls-filter start_epoch 85 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e85) -- 0x55958c608c00 con 0x55958c5e3c00 2024-03-31T15:11:19.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.647+0000 7feb33800640 1 -- 172.21.15.16:0/3131452291 <== osd.1 v2:172.21.15.16:6808/1398737533 5 ==== osd_op_reply(25 [pgnls-filter start_epoch 85 out=49b] v81'7 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55958c368780 con 0x55958c5e3c00 2024-03-31T15:11:19.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.651+0000 7feb2de00640 1 -- 172.21.15.16:0/3131452291 --> [v2:172.21.15.89:6810/50395527,v1:172.21.15.89:6814/50395527] -- osd_op(unknown.0.0:26 13.3a 13:5c000000::::head [pgnls-filter start_epoch 85 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e85) -- 0x55958c5e3800 con 0x55958d17cc00 2024-03-31T15:11:19.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.651+0000 7feb34200640 1 -- 172.21.15.16:0/3131452291 <== osd.5 v2:172.21.15.89:6810/50395527 4 ==== osd_op_reply(26 [pgnls-filter start_epoch 85 out=49b] v81'6 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55958c368000 con 0x55958d17cc00 2024-03-31T15:11:19.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.651+0000 7feb2e800640 1 -- 172.21.15.16:0/3131452291 --> [v2:172.21.15.89:6800/1856844856,v1:172.21.15.89:6801/1856844856] -- osd_op(unknown.0.0:27 13.6 13:60000000::::head [pgnls-filter start_epoch 85 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e85) -- 0x55958c609000 con 0x55958c5f6800 2024-03-31T15:11:19.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.651+0000 7feb34c00640 1 -- 172.21.15.16:0/3131452291 <== osd.4 v2:172.21.15.89:6800/1856844856 2 ==== osd_op_reply(27 [pgnls-filter start_epoch 85 out=49b] v81'7 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55958c368500 con 0x55958c5f6800 2024-03-31T15:11:19.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.651+0000 7feb2de00640 1 -- 172.21.15.16:0/3131452291 --> [v2:172.21.15.89:6808/1796778630,v1:172.21.15.89:6811/1796778630] -- osd_op(unknown.0.0:28 13.26 13:64000000::::head [pgnls-filter start_epoch 85 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e85) -- 0x55958c5f7400 con 0x55958d17dc00 2024-03-31T15:11:19.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.651+0000 7feb34200640 1 -- 172.21.15.16:0/3131452291 <== osd.7 v2:172.21.15.89:6808/1796778630 3 ==== osd_op_reply(28 [pgnls-filter start_epoch 85 out=49b] v81'15 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55958c368000 con 0x55958d17dc00 2024-03-31T15:11:19.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.651+0000 7feb2e800640 1 --2- 172.21.15.16:0/3131452291 >> [v2:172.21.15.16:6809/3360352974,v1:172.21.15.16:6812/3360352974] conn(0x55958c608000 0x55958c5f3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:19.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.651+0000 7feb2e800640 1 -- 172.21.15.16:0/3131452291 --> [v2:172.21.15.16:6809/3360352974,v1:172.21.15.16:6812/3360352974] -- osd_op(unknown.0.0:29 13.16 13:68000000::::head [pgnls-filter start_epoch 85 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e85) -- 0x55958c609800 con 0x55958c608000 2024-03-31T15:11:19.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.651+0000 7feb34c00640 1 --2- 172.21.15.16:0/3131452291 >> [v2:172.21.15.16:6809/3360352974,v1:172.21.15.16:6812/3360352974] conn(0x55958c608000 0x55958c5f3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:19.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.651+0000 7feb34c00640 1 --2- 172.21.15.16:0/3131452291 >> [v2:172.21.15.16:6809/3360352974,v1:172.21.15.16:6812/3360352974] conn(0x55958c608000 0x55958c5f3600 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:11:19.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.651+0000 7feb34c00640 1 -- 172.21.15.16:0/3131452291 <== osd.3 v2:172.21.15.16:6809/3360352974 1 ==== osd_op_reply(29 [pgnls-filter start_epoch 85 out=49b] v81'10 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55958c368500 con 0x55958c608000 2024-03-31T15:11:19.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.651+0000 7feb2de00640 1 -- 172.21.15.16:0/3131452291 --> [v2:172.21.15.89:6809/3470471752,v1:172.21.15.89:6812/3470471752] -- osd_op(unknown.0.0:30 13.36 13:6c000000::::head [pgnls-filter start_epoch 85 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e85) -- 0x55958c5f7000 con 0x55958d17d400 2024-03-31T15:11:19.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.651+0000 7feb34c00640 1 -- 172.21.15.16:0/3131452291 <== osd.6 v2:172.21.15.89:6809/3470471752 7 ==== osd_op_reply(30 [pgnls-filter start_epoch 85 out=49b] v81'15 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55958c368500 con 0x55958d17d400 2024-03-31T15:11:19.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.651+0000 7feb2e800640 1 -- 172.21.15.16:0/3131452291 --> [v2:172.21.15.89:6808/1796778630,v1:172.21.15.89:6811/1796778630] -- osd_op(unknown.0.0:31 13.e 13:70000000::::head [pgnls-filter start_epoch 85 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e85) -- 0x55958c609400 con 0x55958d17dc00 2024-03-31T15:11:19.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.651+0000 7feb34200640 1 -- 172.21.15.16:0/3131452291 <== osd.7 v2:172.21.15.89:6808/1796778630 4 ==== osd_op_reply(31 [pgnls-filter start_epoch 85 out=49b] v81'5 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55958c368000 con 0x55958d17dc00 2024-03-31T15:11:19.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.651+0000 7feb2de00640 1 -- 172.21.15.16:0/3131452291 --> [v2:172.21.15.16:6819/3018526646,v1:172.21.15.16:6822/3018526646] -- osd_op(unknown.0.0:32 13.2e 13:74000000::::head [pgnls-filter start_epoch 85 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e85) -- 0x55958c5f7c00 con 0x55958c5f6400 2024-03-31T15:11:19.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.651+0000 7feb34200640 1 -- 172.21.15.16:0/3131452291 <== osd.2 v2:172.21.15.16:6819/3018526646 7 ==== osd_op_reply(32 [pgnls-filter start_epoch 85 out=49b] v81'10 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55958c368000 con 0x55958c5f6400 2024-03-31T15:11:19.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.651+0000 7feb2e800640 1 -- 172.21.15.16:0/3131452291 --> [v2:172.21.15.89:6800/1856844856,v1:172.21.15.89:6801/1856844856] -- osd_op(unknown.0.0:33 13.1e 13:78000000::::head [pgnls-filter start_epoch 85 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e85) -- 0x55958c608400 con 0x55958c5f6800 2024-03-31T15:11:19.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.651+0000 7feb34c00640 1 -- 172.21.15.16:0/3131452291 <== osd.4 v2:172.21.15.89:6800/1856844856 3 ==== osd_op_reply(33 [pgnls-filter start_epoch 85 out=49b] v81'5 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55958c368500 con 0x55958c5f6800 2024-03-31T15:11:19.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.651+0000 7feb2de00640 1 -- 172.21.15.16:0/3131452291 --> [v2:172.21.15.16:6808/1398737533,v1:172.21.15.16:6810/1398737533] -- osd_op(unknown.0.0:34 13.3e 13:7c000000::::head [pgnls-filter start_epoch 85 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e85) -- 0x55958c5f6000 con 0x55958c5e3c00 2024-03-31T15:11:19.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.651+0000 7feb33800640 1 -- 172.21.15.16:0/3131452291 <== osd.1 v2:172.21.15.16:6808/1398737533 6 ==== osd_op_reply(34 [pgnls-filter start_epoch 85 out=49b] v81'4 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55958c368780 con 0x55958c5e3c00 2024-03-31T15:11:19.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.655+0000 7feb2e800640 1 -- 172.21.15.16:0/3131452291 --> [v2:172.21.15.89:6809/3470471752,v1:172.21.15.89:6812/3470471752] -- osd_op(unknown.0.0:35 13.1 13:80000000::::head [pgnls-filter start_epoch 85 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e85) -- 0x55958c609000 con 0x55958d17d400 2024-03-31T15:11:19.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.655+0000 7feb34c00640 1 -- 172.21.15.16:0/3131452291 <== osd.6 v2:172.21.15.89:6809/3470471752 8 ==== osd_op_reply(35 [pgnls-filter start_epoch 85 out=49b] v81'5 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55958c368500 con 0x55958d17d400 2024-03-31T15:11:19.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.655+0000 7feb2de00640 1 -- 172.21.15.16:0/3131452291 --> [v2:172.21.15.89:6800/1856844856,v1:172.21.15.89:6801/1856844856] -- osd_op(unknown.0.0:36 13.21 13:84000000::::head [pgnls-filter start_epoch 85 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e85) -- 0x55958c5f7800 con 0x55958c5f6800 2024-03-31T15:11:19.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.655+0000 7feb34c00640 1 -- 172.21.15.16:0/3131452291 <== osd.4 v2:172.21.15.89:6800/1856844856 4 ==== osd_op_reply(36 [pgnls-filter start_epoch 85 out=49b] v81'12 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55958c368500 con 0x55958c5f6800 2024-03-31T15:11:19.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.655+0000 7feb2e800640 1 -- 172.21.15.16:0/3131452291 --> [v2:172.21.15.16:6808/1398737533,v1:172.21.15.16:6810/1398737533] -- osd_op(unknown.0.0:37 13.11 13:88000000::::head [pgnls-filter start_epoch 85 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e85) -- 0x55958c609800 con 0x55958c5e3c00 2024-03-31T15:11:19.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.655+0000 7feb33800640 1 -- 172.21.15.16:0/3131452291 <== osd.1 v2:172.21.15.16:6808/1398737533 7 ==== osd_op_reply(37 [pgnls-filter start_epoch 85 out=49b] v81'6 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55958c368780 con 0x55958c5e3c00 2024-03-31T15:11:19.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.655+0000 7feb2de00640 1 -- 172.21.15.16:0/3131452291 --> [v2:172.21.15.16:6809/3360352974,v1:172.21.15.16:6812/3360352974] -- osd_op(unknown.0.0:38 13.31 13:8c000000::::head [pgnls-filter start_epoch 85 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e85) -- 0x55958d11e800 con 0x55958c608000 2024-03-31T15:11:19.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.655+0000 7feb34c00640 1 -- 172.21.15.16:0/3131452291 <== osd.3 v2:172.21.15.16:6809/3360352974 2 ==== osd_op_reply(38 [pgnls-filter start_epoch 85 out=49b] v81'7 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55958c368500 con 0x55958c608000 2024-03-31T15:11:19.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.655+0000 7feb2e800640 1 -- 172.21.15.16:0/3131452291 --> [v2:172.21.15.89:6810/50395527,v1:172.21.15.89:6814/50395527] -- osd_op(unknown.0.0:39 13.9 13:90000000::::head [pgnls-filter start_epoch 85 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e85) -- 0x55958d17d000 con 0x55958d17cc00 2024-03-31T15:11:19.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.655+0000 7feb34200640 1 -- 172.21.15.16:0/3131452291 <== osd.5 v2:172.21.15.89:6810/50395527 5 ==== osd_op_reply(39 [pgnls-filter start_epoch 85 out=49b] v81'2 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55958c368000 con 0x55958d17cc00 2024-03-31T15:11:19.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.655+0000 7feb2de00640 1 -- 172.21.15.16:0/3131452291 --> [v2:172.21.15.89:6809/3470471752,v1:172.21.15.89:6812/3470471752] -- osd_op(unknown.0.0:40 13.29 13:94000000::::head [pgnls-filter start_epoch 85 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e85) -- 0x55958c612000 con 0x55958d17d400 2024-03-31T15:11:19.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.655+0000 7feb34c00640 1 -- 172.21.15.16:0/3131452291 <== osd.6 v2:172.21.15.89:6809/3470471752 9 ==== osd_op_reply(40 [pgnls-filter start_epoch 85 out=49b] v81'8 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55958c368500 con 0x55958d17d400 2024-03-31T15:11:19.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.655+0000 7feb2e800640 1 -- 172.21.15.16:0/3131452291 --> [v2:172.21.15.16:6809/3360352974,v1:172.21.15.16:6812/3360352974] -- osd_op(unknown.0.0:41 13.19 13:98000000::::head [pgnls-filter start_epoch 85 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e85) -- 0x55958d17d800 con 0x55958c608000 2024-03-31T15:11:19.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.655+0000 7feb34c00640 1 -- 172.21.15.16:0/3131452291 <== osd.3 v2:172.21.15.16:6809/3360352974 3 ==== osd_op_reply(41 [pgnls-filter start_epoch 85 out=49b] v81'10 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55958c368500 con 0x55958c608000 2024-03-31T15:11:19.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.655+0000 7feb2de00640 1 -- 172.21.15.16:0/3131452291 --> [v2:172.21.15.16:6819/3018526646,v1:172.21.15.16:6822/3018526646] -- osd_op(unknown.0.0:42 13.39 13:9c000000::::head [pgnls-filter start_epoch 85 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e85) -- 0x55958c612400 con 0x55958c5f6400 2024-03-31T15:11:19.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.655+0000 7feb34200640 1 -- 172.21.15.16:0/3131452291 <== osd.2 v2:172.21.15.16:6819/3018526646 8 ==== osd_op_reply(42 [pgnls-filter start_epoch 85 out=49b] v81'5 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55958c368000 con 0x55958c5f6400 2024-03-31T15:11:19.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.655+0000 7feb2e800640 1 -- 172.21.15.16:0/3131452291 --> [v2:172.21.15.89:6800/1856844856,v1:172.21.15.89:6801/1856844856] -- osd_op(unknown.0.0:43 13.5 13:a0000000::::head [pgnls-filter start_epoch 85 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e85) -- 0x55958c5e2000 con 0x55958c5f6800 2024-03-31T15:11:19.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.655+0000 7feb34c00640 1 -- 172.21.15.16:0/3131452291 <== osd.4 v2:172.21.15.89:6800/1856844856 5 ==== osd_op_reply(43 [pgnls-filter start_epoch 85 out=49b] v81'5 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55958c368500 con 0x55958c5f6800 2024-03-31T15:11:19.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.655+0000 7feb2de00640 1 -- 172.21.15.16:0/3131452291 --> [v2:172.21.15.89:6800/1856844856,v1:172.21.15.89:6801/1856844856] -- osd_op(unknown.0.0:44 13.25 13:a4000000::::head [pgnls-filter start_epoch 85 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e85) -- 0x55958c612800 con 0x55958c5f6800 2024-03-31T15:11:19.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.655+0000 7feb34c00640 1 -- 172.21.15.16:0/3131452291 <== osd.4 v2:172.21.15.89:6800/1856844856 6 ==== osd_op_reply(44 [pgnls-filter start_epoch 85 out=49b] v81'6 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55958c368500 con 0x55958c5f6800 2024-03-31T15:11:19.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.655+0000 7feb2e800640 1 -- 172.21.15.16:0/3131452291 --> [v2:172.21.15.89:6800/1856844856,v1:172.21.15.89:6801/1856844856] -- osd_op(unknown.0.0:45 13.15 13:a8000000::::head [pgnls-filter start_epoch 85 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e85) -- 0x55958c5f7400 con 0x55958c5f6800 2024-03-31T15:11:19.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.655+0000 7feb34c00640 1 -- 172.21.15.16:0/3131452291 <== osd.4 v2:172.21.15.89:6800/1856844856 7 ==== osd_op_reply(45 [pgnls-filter start_epoch 85 out=49b] v81'6 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55958c368500 con 0x55958c5f6800 2024-03-31T15:11:19.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.655+0000 7feb2de00640 1 -- 172.21.15.16:0/3131452291 --> [v2:172.21.15.16:6808/1398737533,v1:172.21.15.16:6810/1398737533] -- osd_op(unknown.0.0:46 13.35 13:ac000000::::head [pgnls-filter start_epoch 85 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e85) -- 0x55958c612c00 con 0x55958c5e3c00 2024-03-31T15:11:19.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.659+0000 7feb33800640 1 -- 172.21.15.16:0/3131452291 <== osd.1 v2:172.21.15.16:6808/1398737533 8 ==== osd_op_reply(46 [pgnls-filter start_epoch 85 out=49b] v81'6 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55958c368780 con 0x55958c5e3c00 2024-03-31T15:11:19.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.659+0000 7feb2e800640 1 -- 172.21.15.16:0/3131452291 --> [v2:172.21.15.89:6800/1856844856,v1:172.21.15.89:6801/1856844856] -- osd_op(unknown.0.0:47 13.d 13:b0000000::::head [pgnls-filter start_epoch 85 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e85) -- 0x55958c5f7c00 con 0x55958c5f6800 2024-03-31T15:11:19.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.659+0000 7feb34c00640 1 -- 172.21.15.16:0/3131452291 <== osd.4 v2:172.21.15.89:6800/1856844856 8 ==== osd_op_reply(47 [pgnls-filter start_epoch 85 out=49b] v81'7 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55958c368500 con 0x55958c5f6800 2024-03-31T15:11:19.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.659+0000 7feb2de00640 1 -- 172.21.15.16:0/3131452291 --> [v2:172.21.15.16:6800/932712244,v1:172.21.15.16:6801/932712244] -- osd_op(unknown.0.0:48 13.2d 13:b4000000::::head [pgnls-filter start_epoch 85 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e85) -- 0x55958c613000 con 0x55958c5f6c00 2024-03-31T15:11:19.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.659+0000 7feb33800640 1 -- 172.21.15.16:0/3131452291 <== osd.0 v2:172.21.15.16:6800/932712244 3 ==== osd_op_reply(48 [pgnls-filter start_epoch 85 out=49b] v81'7 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55958c368780 con 0x55958c5f6c00 2024-03-31T15:11:19.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.659+0000 7feb2e800640 1 -- 172.21.15.16:0/3131452291 --> [v2:172.21.15.89:6808/1796778630,v1:172.21.15.89:6811/1796778630] -- osd_op(unknown.0.0:49 13.1d 13:b8000000::::head [pgnls-filter start_epoch 85 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e85) -- 0x55958c608800 con 0x55958d17dc00 2024-03-31T15:11:19.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.659+0000 7feb34200640 1 -- 172.21.15.16:0/3131452291 <== osd.7 v2:172.21.15.89:6808/1796778630 5 ==== osd_op_reply(49 [pgnls-filter start_epoch 85 out=49b] v81'15 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55958c368000 con 0x55958d17dc00 2024-03-31T15:11:19.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.659+0000 7feb2de00640 1 -- 172.21.15.16:0/3131452291 --> [v2:172.21.15.89:6808/1796778630,v1:172.21.15.89:6811/1796778630] -- osd_op(unknown.0.0:50 13.3d 13:bc000000::::head [pgnls-filter start_epoch 85 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e85) -- 0x55958c613400 con 0x55958d17dc00 2024-03-31T15:11:19.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.659+0000 7feb34200640 1 -- 172.21.15.16:0/3131452291 <== osd.7 v2:172.21.15.89:6808/1796778630 6 ==== osd_op_reply(50 [pgnls-filter start_epoch 85 out=49b] v81'7 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55958c368000 con 0x55958d17dc00 2024-03-31T15:11:19.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.659+0000 7feb2e800640 1 -- 172.21.15.16:0/3131452291 --> [v2:172.21.15.16:6800/932712244,v1:172.21.15.16:6801/932712244] -- osd_op(unknown.0.0:51 13.3 13:c0000000::::head [pgnls-filter start_epoch 85 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e85) -- 0x55958c609400 con 0x55958c5f6c00 2024-03-31T15:11:19.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.659+0000 7feb33800640 1 -- 172.21.15.16:0/3131452291 <== osd.0 v2:172.21.15.16:6800/932712244 4 ==== osd_op_reply(51 [pgnls-filter start_epoch 85 out=49b] v81'7 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55958c368780 con 0x55958c5f6c00 2024-03-31T15:11:19.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.659+0000 7feb2de00640 1 -- 172.21.15.16:0/3131452291 --> [v2:172.21.15.16:6808/1398737533,v1:172.21.15.16:6810/1398737533] -- osd_op(unknown.0.0:52 13.23 13:c4000000::::head [pgnls-filter start_epoch 85 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e85) -- 0x55958c613800 con 0x55958c5e3c00 2024-03-31T15:11:19.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.659+0000 7feb33800640 1 -- 172.21.15.16:0/3131452291 <== osd.1 v2:172.21.15.16:6808/1398737533 9 ==== osd_op_reply(52 [pgnls-filter start_epoch 85 out=49b] v81'7 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55958c368780 con 0x55958c5e3c00 2024-03-31T15:11:19.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.659+0000 7feb2e800640 1 -- 172.21.15.16:0/3131452291 --> [v2:172.21.15.89:6808/1796778630,v1:172.21.15.89:6811/1796778630] -- osd_op(unknown.0.0:53 13.13 13:c8000000::::head [pgnls-filter start_epoch 85 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e85) -- 0x55958c608400 con 0x55958d17dc00 2024-03-31T15:11:19.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.659+0000 7feb34200640 1 -- 172.21.15.16:0/3131452291 <== osd.7 v2:172.21.15.89:6808/1796778630 7 ==== osd_op_reply(53 [pgnls-filter start_epoch 85 out=49b] v81'15 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55958c368000 con 0x55958d17dc00 2024-03-31T15:11:19.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.659+0000 7feb2de00640 1 -- 172.21.15.16:0/3131452291 --> [v2:172.21.15.89:6808/1796778630,v1:172.21.15.89:6811/1796778630] -- osd_op(unknown.0.0:54 13.33 13:cc000000::::head [pgnls-filter start_epoch 85 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e85) -- 0x55958c613c00 con 0x55958d17dc00 2024-03-31T15:11:19.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.659+0000 7feb34200640 1 -- 172.21.15.16:0/3131452291 <== osd.7 v2:172.21.15.89:6808/1796778630 8 ==== osd_op_reply(54 [pgnls-filter start_epoch 85 out=49b] v81'10 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55958c368000 con 0x55958d17dc00 2024-03-31T15:11:19.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.659+0000 7feb2e800640 1 -- 172.21.15.16:0/3131452291 --> [v2:172.21.15.89:6810/50395527,v1:172.21.15.89:6814/50395527] -- osd_op(unknown.0.0:55 13.b 13:d0000000::::head [pgnls-filter start_epoch 85 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e85) -- 0x55958c609000 con 0x55958d17cc00 2024-03-31T15:11:19.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.659+0000 7feb34200640 1 -- 172.21.15.16:0/3131452291 <== osd.5 v2:172.21.15.89:6810/50395527 6 ==== osd_op_reply(55 [pgnls-filter start_epoch 85 out=49b] v81'8 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55958c368000 con 0x55958d17cc00 2024-03-31T15:11:19.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.659+0000 7feb2de00640 1 -- 172.21.15.16:0/3131452291 --> [v2:172.21.15.16:6819/3018526646,v1:172.21.15.16:6822/3018526646] -- osd_op(unknown.0.0:56 13.2b 13:d4000000::::head [pgnls-filter start_epoch 85 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e85) -- 0x55958c612c00 con 0x55958c5f6400 2024-03-31T15:11:19.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.659+0000 7feb34200640 1 -- 172.21.15.16:0/3131452291 <== osd.2 v2:172.21.15.16:6819/3018526646 9 ==== osd_op_reply(56 [pgnls-filter start_epoch 85 out=49b] v81'11 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55958c368000 con 0x55958c5f6400 2024-03-31T15:11:19.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.659+0000 7feb2e800640 1 -- 172.21.15.16:0/3131452291 --> [v2:172.21.15.89:6800/1856844856,v1:172.21.15.89:6801/1856844856] -- osd_op(unknown.0.0:57 13.1b 13:d8000000::::head [pgnls-filter start_epoch 85 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e85) -- 0x55958d17d000 con 0x55958c5f6800 2024-03-31T15:11:19.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.663+0000 7feb34c00640 1 -- 172.21.15.16:0/3131452291 <== osd.4 v2:172.21.15.89:6800/1856844856 9 ==== osd_op_reply(57 [pgnls-filter start_epoch 85 out=49b] v81'8 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55958c368500 con 0x55958c5f6800 2024-03-31T15:11:19.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.663+0000 7feb2de00640 1 -- 172.21.15.16:0/3131452291 --> [v2:172.21.15.89:6800/1856844856,v1:172.21.15.89:6801/1856844856] -- osd_op(unknown.0.0:58 13.3b 13:dc000000::::head [pgnls-filter start_epoch 85 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e85) -- 0x55958c613000 con 0x55958c5f6800 2024-03-31T15:11:19.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.663+0000 7feb34c00640 1 -- 172.21.15.16:0/3131452291 <== osd.4 v2:172.21.15.89:6800/1856844856 10 ==== osd_op_reply(58 [pgnls-filter start_epoch 85 out=49b] v81'4 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55958c368500 con 0x55958c5f6800 2024-03-31T15:11:19.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.663+0000 7feb2e800640 1 -- 172.21.15.16:0/3131452291 --> [v2:172.21.15.89:6808/1796778630,v1:172.21.15.89:6811/1796778630] -- osd_op(unknown.0.0:59 13.7 13:e0000000::::head [pgnls-filter start_epoch 85 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e85) -- 0x55958c5e3800 con 0x55958d17dc00 2024-03-31T15:11:19.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.663+0000 7feb34200640 1 -- 172.21.15.16:0/3131452291 <== osd.7 v2:172.21.15.89:6808/1796778630 9 ==== osd_op_reply(59 [pgnls-filter start_epoch 85 out=81b] v85'7 uv7 ondisk = 1) ==== 144+0+81 (crc 0 0 0) 0x55958c368000 con 0x55958d17dc00 2024-03-31T15:11:19.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.663+0000 7feb36c488c0 1 -- 172.21.15.16:0/3131452291 --> [v2:172.21.15.89:6808/1796778630,v1:172.21.15.89:6811/1796778630] -- osd_op(unknown.0.0:60 13.0 13:00000000::::head [pgnls-filter start_epoch 85 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e85) -- 0x55958c5e2400 con 0x55958d17dc00 2024-03-31T15:11:19.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.663+0000 7feb34200640 1 -- 172.21.15.16:0/3131452291 <== osd.7 v2:172.21.15.89:6808/1796778630 10 ==== osd_op_reply(60 [pgnls-filter start_epoch 85 out=49b] v81'8 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55958c368000 con 0x55958d17dc00 2024-03-31T15:11:19.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.663+0000 7feb2e800640 1 -- 172.21.15.16:0/3131452291 --> [v2:172.21.15.89:6809/3470471752,v1:172.21.15.89:6812/3470471752] -- osd_op(unknown.0.0:61 13.20 13:04000000::::head [pgnls-filter start_epoch 85 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e85) -- 0x55958d17d800 con 0x55958d17d400 2024-03-31T15:11:19.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.663+0000 7feb34c00640 1 -- 172.21.15.16:0/3131452291 <== osd.6 v2:172.21.15.89:6809/3470471752 10 ==== osd_op_reply(61 [pgnls-filter start_epoch 85 out=49b] v81'9 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55958c368500 con 0x55958d17d400 2024-03-31T15:11:19.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.663+0000 7feb2de00640 1 -- 172.21.15.16:0/3131452291 --> [v2:172.21.15.16:6808/1398737533,v1:172.21.15.16:6810/1398737533] -- osd_op(unknown.0.0:62 13.10 13:08000000::::head [pgnls-filter start_epoch 85 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e85) -- 0x55958d11e800 con 0x55958c5e3c00 2024-03-31T15:11:19.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.663+0000 7feb33800640 1 -- 172.21.15.16:0/3131452291 <== osd.1 v2:172.21.15.16:6808/1398737533 10 ==== osd_op_reply(62 [pgnls-filter start_epoch 85 out=49b] v81'6 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55958c368780 con 0x55958c5e3c00 2024-03-31T15:11:19.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.663+0000 7feb2e800640 1 -- 172.21.15.16:0/3131452291 --> [v2:172.21.15.89:6800/1856844856,v1:172.21.15.89:6801/1856844856] -- osd_op(unknown.0.0:63 13.30 13:0c000000::::head [pgnls-filter start_epoch 85 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e85) -- 0x55958c5f7400 con 0x55958c5f6800 2024-03-31T15:11:19.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.663+0000 7feb34c00640 1 -- 172.21.15.16:0/3131452291 <== osd.4 v2:172.21.15.89:6800/1856844856 11 ==== osd_op_reply(63 [pgnls-filter start_epoch 85 out=49b] v81'9 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55958c368500 con 0x55958c5f6800 2024-03-31T15:11:19.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.663+0000 7feb2de00640 1 -- 172.21.15.16:0/3131452291 --> [v2:172.21.15.16:6819/3018526646,v1:172.21.15.16:6822/3018526646] -- osd_op(unknown.0.0:64 13.8 13:10000000::::head [pgnls-filter start_epoch 85 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e85) -- 0x55958c5f6000 con 0x55958c5f6400 2024-03-31T15:11:19.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.663+0000 7feb34200640 1 -- 172.21.15.16:0/3131452291 <== osd.2 v2:172.21.15.16:6819/3018526646 10 ==== osd_op_reply(64 [pgnls-filter start_epoch 85 out=49b] v81'9 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55958c368000 con 0x55958c5f6400 2024-03-31T15:11:19.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.663+0000 7feb2e800640 1 -- 172.21.15.16:0/3131452291 --> [v2:172.21.15.16:6819/3018526646,v1:172.21.15.16:6822/3018526646] -- osd_op(unknown.0.0:65 13.28 13:14000000::::head [pgnls-filter start_epoch 85 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e85) -- 0x55958c5f7c00 con 0x55958c5f6400 2024-03-31T15:11:19.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.663+0000 7feb34200640 1 -- 172.21.15.16:0/3131452291 <== osd.2 v2:172.21.15.16:6819/3018526646 11 ==== osd_op_reply(65 [pgnls-filter start_epoch 85 out=49b] v81'6 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55958c368000 con 0x55958c5f6400 2024-03-31T15:11:19.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.663+0000 7feb2de00640 1 -- 172.21.15.16:0/3131452291 --> [v2:172.21.15.89:6808/1796778630,v1:172.21.15.89:6811/1796778630] -- osd_op(unknown.0.0:66 13.18 13:18000000::::head [pgnls-filter start_epoch 85 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e85) -- 0x55958c5f7000 con 0x55958d17dc00 2024-03-31T15:11:19.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.663+0000 7feb34200640 1 -- 172.21.15.16:0/3131452291 <== osd.7 v2:172.21.15.89:6808/1796778630 11 ==== osd_op_reply(66 [pgnls-filter start_epoch 85 out=49b] v81'8 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55958c368000 con 0x55958d17dc00 2024-03-31T15:11:19.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.663+0000 7feb2e800640 1 -- 172.21.15.16:0/3131452291 --> [v2:172.21.15.89:6810/50395527,v1:172.21.15.89:6814/50395527] -- osd_op(unknown.0.0:67 13.38 13:1c000000::::head [pgnls-filter start_epoch 85 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e85) -- 0x55958c5e3800 con 0x55958d17cc00 2024-03-31T15:11:19.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.663+0000 7feb34200640 1 -- 172.21.15.16:0/3131452291 <== osd.5 v2:172.21.15.89:6810/50395527 7 ==== osd_op_reply(67 [pgnls-filter start_epoch 85 out=49b] v81'9 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55958c368000 con 0x55958d17cc00 2024-03-31T15:11:19.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.663+0000 7feb2de00640 1 -- 172.21.15.16:0/3131452291 --> [v2:172.21.15.89:6809/3470471752,v1:172.21.15.89:6812/3470471752] -- osd_op(unknown.0.0:68 13.4 13:20000000::::head [pgnls-filter start_epoch 85 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e85) -- 0x55958c5f7800 con 0x55958d17d400 2024-03-31T15:11:19.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.667+0000 7feb34c00640 1 -- 172.21.15.16:0/3131452291 <== osd.6 v2:172.21.15.89:6809/3470471752 11 ==== osd_op_reply(68 [pgnls-filter start_epoch 85 out=49b] v81'9 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55958c368500 con 0x55958d17d400 2024-03-31T15:11:19.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.667+0000 7feb2e800640 1 -- 172.21.15.16:0/3131452291 --> [v2:172.21.15.89:6809/3470471752,v1:172.21.15.89:6812/3470471752] -- osd_op(unknown.0.0:69 13.24 13:24000000::::head [pgnls-filter start_epoch 85 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e85) -- 0x55958c5e2400 con 0x55958d17d400 2024-03-31T15:11:19.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.667+0000 7feb34c00640 1 -- 172.21.15.16:0/3131452291 <== osd.6 v2:172.21.15.89:6809/3470471752 12 ==== osd_op_reply(69 [pgnls-filter start_epoch 85 out=49b] v81'4 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55958c368500 con 0x55958d17d400 2024-03-31T15:11:19.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.667+0000 7feb2de00640 1 -- 172.21.15.16:0/3131452291 --> [v2:172.21.15.16:6800/932712244,v1:172.21.15.16:6801/932712244] -- osd_op(unknown.0.0:70 13.14 13:28000000::::head [pgnls-filter start_epoch 85 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e85) -- 0x55958c609c00 con 0x55958c5f6c00 2024-03-31T15:11:19.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.667+0000 7feb33800640 1 -- 172.21.15.16:0/3131452291 <== osd.0 v2:172.21.15.16:6800/932712244 5 ==== osd_op_reply(70 [pgnls-filter start_epoch 85 out=49b] v81'11 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55958c368780 con 0x55958c5f6c00 2024-03-31T15:11:19.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.667+0000 7feb2e800640 1 -- 172.21.15.16:0/3131452291 --> [v2:172.21.15.89:6809/3470471752,v1:172.21.15.89:6812/3470471752] -- osd_op(unknown.0.0:71 13.34 13:2c000000::::head [pgnls-filter start_epoch 85 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e85) -- 0x55958c5e2000 con 0x55958d17d400 2024-03-31T15:11:19.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.667+0000 7feb34c00640 1 -- 172.21.15.16:0/3131452291 <== osd.6 v2:172.21.15.89:6809/3470471752 13 ==== osd_op_reply(71 [pgnls-filter start_epoch 85 out=49b] v81'7 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55958c368500 con 0x55958d17d400 2024-03-31T15:11:19.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.667+0000 7feb2de00640 1 -- 172.21.15.16:0/3131452291 --> [v2:172.21.15.16:6808/1398737533,v1:172.21.15.16:6810/1398737533] -- osd_op(unknown.0.0:72 13.c 13:30000000::::head [pgnls-filter start_epoch 85 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e85) -- 0x55958c608c00 con 0x55958c5e3c00 2024-03-31T15:11:19.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.667+0000 7feb33800640 1 -- 172.21.15.16:0/3131452291 <== osd.1 v2:172.21.15.16:6808/1398737533 11 ==== osd_op_reply(72 [pgnls-filter start_epoch 85 out=49b] v81'6 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55958c368780 con 0x55958c5e3c00 2024-03-31T15:11:19.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.667+0000 7feb2e800640 1 -- 172.21.15.16:0/3131452291 --> [v2:172.21.15.16:6819/3018526646,v1:172.21.15.16:6822/3018526646] -- osd_op(unknown.0.0:73 13.2c 13:34000000::::head [pgnls-filter start_epoch 85 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e85) -- 0x55958c608800 con 0x55958c5f6400 2024-03-31T15:11:19.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.667+0000 7feb34200640 1 -- 172.21.15.16:0/3131452291 <== osd.2 v2:172.21.15.16:6819/3018526646 12 ==== osd_op_reply(73 [pgnls-filter start_epoch 85 out=49b] v81'8 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55958c368000 con 0x55958c5f6400 2024-03-31T15:11:19.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.667+0000 7feb2de00640 1 -- 172.21.15.16:0/3131452291 --> [v2:172.21.15.89:6809/3470471752,v1:172.21.15.89:6812/3470471752] -- osd_op(unknown.0.0:74 13.1c 13:38000000::::head [pgnls-filter start_epoch 85 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e85) -- 0x55958c609800 con 0x55958d17d400 2024-03-31T15:11:19.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.667+0000 7feb34c00640 1 -- 172.21.15.16:0/3131452291 <== osd.6 v2:172.21.15.89:6809/3470471752 14 ==== osd_op_reply(74 [pgnls-filter start_epoch 85 out=49b] v81'9 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55958c368500 con 0x55958d17d400 2024-03-31T15:11:19.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.667+0000 7feb2e800640 1 -- 172.21.15.16:0/3131452291 --> [v2:172.21.15.16:6819/3018526646,v1:172.21.15.16:6822/3018526646] -- osd_op(unknown.0.0:75 13.3c 13:3c000000::::head [pgnls-filter start_epoch 85 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e85) -- 0x55958c608400 con 0x55958c5f6400 2024-03-31T15:11:19.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.667+0000 7feb34200640 1 -- 172.21.15.16:0/3131452291 <== osd.2 v2:172.21.15.16:6819/3018526646 13 ==== osd_op_reply(75 [pgnls-filter start_epoch 85 out=49b] v81'8 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55958c368000 con 0x55958c5f6400 2024-03-31T15:11:19.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.667+0000 7feb2de00640 1 -- 172.21.15.16:0/3131452291 --> [v2:172.21.15.89:6810/50395527,v1:172.21.15.89:6814/50395527] -- osd_op(unknown.0.0:76 13.2 13:40000000::::head [pgnls-filter start_epoch 85 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e85) -- 0x55958c613400 con 0x55958d17cc00 2024-03-31T15:11:19.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.667+0000 7feb34200640 1 -- 172.21.15.16:0/3131452291 <== osd.5 v2:172.21.15.89:6810/50395527 8 ==== osd_op_reply(76 [pgnls-filter start_epoch 85 out=49b] v81'8 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55958c368000 con 0x55958d17cc00 2024-03-31T15:11:19.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.667+0000 7feb2e800640 1 -- 172.21.15.16:0/3131452291 --> [v2:172.21.15.16:6808/1398737533,v1:172.21.15.16:6810/1398737533] -- osd_op(unknown.0.0:77 13.22 13:44000000::::head [pgnls-filter start_epoch 85 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e85) -- 0x55958c609000 con 0x55958c5e3c00 2024-03-31T15:11:19.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.667+0000 7feb33800640 1 -- 172.21.15.16:0/3131452291 <== osd.1 v2:172.21.15.16:6808/1398737533 12 ==== osd_op_reply(77 [pgnls-filter start_epoch 85 out=49b] v81'8 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55958c368780 con 0x55958c5e3c00 2024-03-31T15:11:19.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.667+0000 7feb2de00640 1 -- 172.21.15.16:0/3131452291 --> [v2:172.21.15.16:6800/932712244,v1:172.21.15.16:6801/932712244] -- osd_op(unknown.0.0:78 13.12 13:48000000::::head [pgnls-filter start_epoch 85 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e85) -- 0x55958c613c00 con 0x55958c5f6c00 2024-03-31T15:11:19.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.667+0000 7feb33800640 1 -- 172.21.15.16:0/3131452291 <== osd.0 v2:172.21.15.16:6800/932712244 6 ==== osd_op_reply(78 [pgnls-filter start_epoch 85 out=49b] v81'10 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55958c368780 con 0x55958c5f6c00 2024-03-31T15:11:19.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.667+0000 7feb2e800640 1 -- 172.21.15.16:0/3131452291 --> [v2:172.21.15.16:6819/3018526646,v1:172.21.15.16:6822/3018526646] -- osd_op(unknown.0.0:79 13.32 13:4c000000::::head [pgnls-filter start_epoch 85 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e85) -- 0x55958c613800 con 0x55958c5f6400 2024-03-31T15:11:19.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.667+0000 7feb34200640 1 -- 172.21.15.16:0/3131452291 <== osd.2 v2:172.21.15.16:6819/3018526646 14 ==== osd_op_reply(79 [pgnls-filter start_epoch 85 out=49b] v81'8 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55958c368000 con 0x55958c5f6400 2024-03-31T15:11:19.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.667+0000 7feb2de00640 1 -- 172.21.15.16:0/3131452291 --> [v2:172.21.15.16:6819/3018526646,v1:172.21.15.16:6822/3018526646] -- osd_op(unknown.0.0:80 13.a 13:50000000::::head [pgnls-filter start_epoch 85 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e85) -- 0x55958c612c00 con 0x55958c5f6400 2024-03-31T15:11:19.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.671+0000 7feb34200640 1 -- 172.21.15.16:0/3131452291 <== osd.2 v2:172.21.15.16:6819/3018526646 15 ==== osd_op_reply(80 [pgnls-filter start_epoch 85 out=49b] v81'5 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55958c368000 con 0x55958c5f6400 2024-03-31T15:11:19.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.671+0000 7feb2e800640 1 -- 172.21.15.16:0/3131452291 --> [v2:172.21.15.16:6808/1398737533,v1:172.21.15.16:6810/1398737533] -- osd_op(unknown.0.0:81 13.2a 13:54000000::::head [pgnls-filter start_epoch 85 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e85) -- 0x55958c613c00 con 0x55958c5e3c00 2024-03-31T15:11:19.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.671+0000 7feb33800640 1 -- 172.21.15.16:0/3131452291 <== osd.1 v2:172.21.15.16:6808/1398737533 13 ==== osd_op_reply(81 [pgnls-filter start_epoch 85 out=49b] v81'9 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55958c368780 con 0x55958c5e3c00 2024-03-31T15:11:19.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.671+0000 7feb2de00640 1 -- 172.21.15.16:0/3131452291 --> [v2:172.21.15.16:6808/1398737533,v1:172.21.15.16:6810/1398737533] -- osd_op(unknown.0.0:82 13.1a 13:58000000::::head [pgnls-filter start_epoch 85 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e85) -- 0x55958c612800 con 0x55958c5e3c00 2024-03-31T15:11:19.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.671+0000 7feb33800640 1 -- 172.21.15.16:0/3131452291 <== osd.1 v2:172.21.15.16:6808/1398737533 14 ==== osd_op_reply(82 [pgnls-filter start_epoch 85 out=49b] v81'7 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55958c368780 con 0x55958c5e3c00 2024-03-31T15:11:19.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.671+0000 7feb2e800640 1 -- 172.21.15.16:0/3131452291 --> [v2:172.21.15.89:6810/50395527,v1:172.21.15.89:6814/50395527] -- osd_op(unknown.0.0:83 13.3a 13:5c000000::::head [pgnls-filter start_epoch 85 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e85) -- 0x55958c609400 con 0x55958d17cc00 2024-03-31T15:11:19.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.671+0000 7feb34200640 1 -- 172.21.15.16:0/3131452291 <== osd.5 v2:172.21.15.89:6810/50395527 9 ==== osd_op_reply(83 [pgnls-filter start_epoch 85 out=49b] v81'6 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55958c368000 con 0x55958d17cc00 2024-03-31T15:11:19.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.671+0000 7feb2de00640 1 -- 172.21.15.16:0/3131452291 --> [v2:172.21.15.89:6800/1856844856,v1:172.21.15.89:6801/1856844856] -- osd_op(unknown.0.0:84 13.6 13:60000000::::head [pgnls-filter start_epoch 85 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e85) -- 0x55958c613000 con 0x55958c5f6800 2024-03-31T15:11:19.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.671+0000 7feb34c00640 1 -- 172.21.15.16:0/3131452291 <== osd.4 v2:172.21.15.89:6800/1856844856 12 ==== osd_op_reply(84 [pgnls-filter start_epoch 85 out=49b] v81'7 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55958c368500 con 0x55958c5f6800 2024-03-31T15:11:19.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.671+0000 7feb2e800640 1 -- 172.21.15.16:0/3131452291 --> [v2:172.21.15.89:6808/1796778630,v1:172.21.15.89:6811/1796778630] -- osd_op(unknown.0.0:85 13.26 13:64000000::::head [pgnls-filter start_epoch 85 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e85) -- 0x55958c609c00 con 0x55958d17dc00 2024-03-31T15:11:19.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.671+0000 7feb34200640 1 -- 172.21.15.16:0/3131452291 <== osd.7 v2:172.21.15.89:6808/1796778630 12 ==== osd_op_reply(85 [pgnls-filter start_epoch 85 out=49b] v81'15 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55958c368000 con 0x55958d17dc00 2024-03-31T15:11:19.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.671+0000 7feb2de00640 1 -- 172.21.15.16:0/3131452291 --> [v2:172.21.15.16:6809/3360352974,v1:172.21.15.16:6812/3360352974] -- osd_op(unknown.0.0:86 13.16 13:68000000::::head [pgnls-filter start_epoch 85 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e85) -- 0x55958c612400 con 0x55958c608000 2024-03-31T15:11:19.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.671+0000 7feb34c00640 1 -- 172.21.15.16:0/3131452291 <== osd.3 v2:172.21.15.16:6809/3360352974 4 ==== osd_op_reply(86 [pgnls-filter start_epoch 85 out=49b] v81'10 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55958c368500 con 0x55958c608000 2024-03-31T15:11:19.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.671+0000 7feb2e800640 1 -- 172.21.15.16:0/3131452291 --> [v2:172.21.15.89:6809/3470471752,v1:172.21.15.89:6812/3470471752] -- osd_op(unknown.0.0:87 13.36 13:6c000000::::head [pgnls-filter start_epoch 85 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e85) -- 0x55958c608c00 con 0x55958d17d400 2024-03-31T15:11:19.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.671+0000 7feb34c00640 1 -- 172.21.15.16:0/3131452291 <== osd.6 v2:172.21.15.89:6809/3470471752 15 ==== osd_op_reply(87 [pgnls-filter start_epoch 85 out=49b] v81'15 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55958c368500 con 0x55958d17d400 2024-03-31T15:11:19.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.671+0000 7feb2de00640 1 -- 172.21.15.16:0/3131452291 --> [v2:172.21.15.89:6808/1796778630,v1:172.21.15.89:6811/1796778630] -- osd_op(unknown.0.0:88 13.e 13:70000000::::head [pgnls-filter start_epoch 85 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e85) -- 0x55958c612000 con 0x55958d17dc00 2024-03-31T15:11:19.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.671+0000 7feb34200640 1 -- 172.21.15.16:0/3131452291 <== osd.7 v2:172.21.15.89:6808/1796778630 13 ==== osd_op_reply(88 [pgnls-filter start_epoch 85 out=49b] v81'5 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55958c368000 con 0x55958d17dc00 2024-03-31T15:11:19.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.671+0000 7feb2e800640 1 -- 172.21.15.16:0/3131452291 --> [v2:172.21.15.16:6819/3018526646,v1:172.21.15.16:6822/3018526646] -- osd_op(unknown.0.0:89 13.2e 13:74000000::::head [pgnls-filter start_epoch 85 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e85) -- 0x55958c609000 con 0x55958c5f6400 2024-03-31T15:11:19.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.671+0000 7feb34200640 1 -- 172.21.15.16:0/3131452291 <== osd.2 v2:172.21.15.16:6819/3018526646 16 ==== osd_op_reply(89 [pgnls-filter start_epoch 85 out=49b] v81'10 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55958c368000 con 0x55958c5f6400 2024-03-31T15:11:19.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.671+0000 7feb2de00640 1 -- 172.21.15.16:0/3131452291 --> [v2:172.21.15.89:6800/1856844856,v1:172.21.15.89:6801/1856844856] -- osd_op(unknown.0.0:90 13.1e 13:78000000::::head [pgnls-filter start_epoch 85 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e85) -- 0x55958d17d000 con 0x55958c5f6800 2024-03-31T15:11:19.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.671+0000 7feb34c00640 1 -- 172.21.15.16:0/3131452291 <== osd.4 v2:172.21.15.89:6800/1856844856 13 ==== osd_op_reply(90 [pgnls-filter start_epoch 85 out=49b] v81'5 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55958c368500 con 0x55958c5f6800 2024-03-31T15:11:19.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.671+0000 7feb2e800640 1 -- 172.21.15.16:0/3131452291 --> [v2:172.21.15.16:6808/1398737533,v1:172.21.15.16:6810/1398737533] -- osd_op(unknown.0.0:91 13.3e 13:7c000000::::head [pgnls-filter start_epoch 85 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e85) -- 0x55958d11e800 con 0x55958c5e3c00 2024-03-31T15:11:19.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.671+0000 7feb33800640 1 -- 172.21.15.16:0/3131452291 <== osd.1 v2:172.21.15.16:6808/1398737533 15 ==== osd_op_reply(91 [pgnls-filter start_epoch 85 out=49b] v81'4 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55958c368780 con 0x55958c5e3c00 2024-03-31T15:11:19.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.671+0000 7feb2de00640 1 -- 172.21.15.16:0/3131452291 --> [v2:172.21.15.89:6809/3470471752,v1:172.21.15.89:6812/3470471752] -- osd_op(unknown.0.0:92 13.1 13:80000000::::head [pgnls-filter start_epoch 85 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e85) -- 0x55958c609800 con 0x55958d17d400 2024-03-31T15:11:19.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.675+0000 7feb34c00640 1 -- 172.21.15.16:0/3131452291 <== osd.6 v2:172.21.15.89:6809/3470471752 16 ==== osd_op_reply(92 [pgnls-filter start_epoch 85 out=49b] v81'5 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55958c368500 con 0x55958d17d400 2024-03-31T15:11:19.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.675+0000 7feb2e800640 1 -- 172.21.15.16:0/3131452291 --> [v2:172.21.15.89:6800/1856844856,v1:172.21.15.89:6801/1856844856] -- osd_op(unknown.0.0:93 13.21 13:84000000::::head [pgnls-filter start_epoch 85 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e85) -- 0x55958c613000 con 0x55958c5f6800 2024-03-31T15:11:19.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.675+0000 7feb34c00640 1 -- 172.21.15.16:0/3131452291 <== osd.4 v2:172.21.15.89:6800/1856844856 14 ==== osd_op_reply(93 [pgnls-filter start_epoch 85 out=49b] v81'12 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55958c368500 con 0x55958c5f6800 2024-03-31T15:11:19.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.675+0000 7feb2de00640 1 -- 172.21.15.16:0/3131452291 --> [v2:172.21.15.16:6808/1398737533,v1:172.21.15.16:6810/1398737533] -- osd_op(unknown.0.0:94 13.11 13:88000000::::head [pgnls-filter start_epoch 85 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e85) -- 0x55958c608800 con 0x55958c5e3c00 2024-03-31T15:11:19.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.675+0000 7feb33800640 1 -- 172.21.15.16:0/3131452291 <== osd.1 v2:172.21.15.16:6808/1398737533 16 ==== osd_op_reply(94 [pgnls-filter start_epoch 85 out=49b] v81'6 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55958c368780 con 0x55958c5e3c00 2024-03-31T15:11:19.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.675+0000 7feb2e800640 1 -- 172.21.15.16:0/3131452291 --> [v2:172.21.15.16:6809/3360352974,v1:172.21.15.16:6812/3360352974] -- osd_op(unknown.0.0:95 13.31 13:8c000000::::head [pgnls-filter start_epoch 85 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e85) -- 0x55958c612400 con 0x55958c608000 2024-03-31T15:11:19.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.675+0000 7feb34c00640 1 -- 172.21.15.16:0/3131452291 <== osd.3 v2:172.21.15.16:6809/3360352974 5 ==== osd_op_reply(95 [pgnls-filter start_epoch 85 out=49b] v81'7 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55958c368500 con 0x55958c608000 2024-03-31T15:11:19.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.675+0000 7feb2de00640 1 -- 172.21.15.16:0/3131452291 --> [v2:172.21.15.89:6810/50395527,v1:172.21.15.89:6814/50395527] -- osd_op(unknown.0.0:96 13.9 13:90000000::::head [pgnls-filter start_epoch 85 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e85) -- 0x55958c5e2400 con 0x55958d17cc00 2024-03-31T15:11:19.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.675+0000 7feb34200640 1 -- 172.21.15.16:0/3131452291 <== osd.5 v2:172.21.15.89:6810/50395527 10 ==== osd_op_reply(96 [pgnls-filter start_epoch 85 out=49b] v81'2 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55958c368000 con 0x55958d17cc00 2024-03-31T15:11:19.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.675+0000 7feb2e800640 1 -- 172.21.15.16:0/3131452291 --> [v2:172.21.15.89:6809/3470471752,v1:172.21.15.89:6812/3470471752] -- osd_op(unknown.0.0:97 13.29 13:94000000::::head [pgnls-filter start_epoch 85 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e85) -- 0x55958c613400 con 0x55958d17d400 2024-03-31T15:11:19.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.675+0000 7feb34c00640 1 -- 172.21.15.16:0/3131452291 <== osd.6 v2:172.21.15.89:6809/3470471752 17 ==== osd_op_reply(97 [pgnls-filter start_epoch 85 out=49b] v81'8 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55958c368500 con 0x55958d17d400 2024-03-31T15:11:19.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.675+0000 7feb2de00640 1 -- 172.21.15.16:0/3131452291 --> [v2:172.21.15.16:6809/3360352974,v1:172.21.15.16:6812/3360352974] -- osd_op(unknown.0.0:98 13.19 13:98000000::::head [pgnls-filter start_epoch 85 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e85) -- 0x55958c5e2000 con 0x55958c608000 2024-03-31T15:11:19.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.675+0000 7feb34c00640 1 -- 172.21.15.16:0/3131452291 <== osd.3 v2:172.21.15.16:6809/3360352974 6 ==== osd_op_reply(98 [pgnls-filter start_epoch 85 out=49b] v81'10 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55958c368500 con 0x55958c608000 2024-03-31T15:11:19.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.675+0000 7feb2e800640 1 -- 172.21.15.16:0/3131452291 --> [v2:172.21.15.16:6819/3018526646,v1:172.21.15.16:6822/3018526646] -- osd_op(unknown.0.0:99 13.39 13:9c000000::::head [pgnls-filter start_epoch 85 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e85) -- 0x55958c613800 con 0x55958c5f6400 2024-03-31T15:11:19.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.675+0000 7feb34200640 1 -- 172.21.15.16:0/3131452291 <== osd.2 v2:172.21.15.16:6819/3018526646 17 ==== osd_op_reply(99 [pgnls-filter start_epoch 85 out=49b] v81'5 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55958c368000 con 0x55958c5f6400 2024-03-31T15:11:19.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.675+0000 7feb2de00640 1 -- 172.21.15.16:0/3131452291 --> [v2:172.21.15.89:6800/1856844856,v1:172.21.15.89:6801/1856844856] -- osd_op(unknown.0.0:100 13.5 13:a0000000::::head [pgnls-filter start_epoch 85 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e85) -- 0x55958c5e3800 con 0x55958c5f6800 2024-03-31T15:11:19.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.675+0000 7feb34c00640 1 -- 172.21.15.16:0/3131452291 <== osd.4 v2:172.21.15.89:6800/1856844856 15 ==== osd_op_reply(100 [pgnls-filter start_epoch 85 out=49b] v81'5 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55958c368500 con 0x55958c5f6800 2024-03-31T15:11:19.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.675+0000 7feb2e800640 1 -- 172.21.15.16:0/3131452291 --> [v2:172.21.15.89:6800/1856844856,v1:172.21.15.89:6801/1856844856] -- osd_op(unknown.0.0:101 13.25 13:a4000000::::head [pgnls-filter start_epoch 85 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e85) -- 0x55958c612c00 con 0x55958c5f6800 2024-03-31T15:11:19.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.675+0000 7feb34c00640 1 -- 172.21.15.16:0/3131452291 <== osd.4 v2:172.21.15.89:6800/1856844856 16 ==== osd_op_reply(101 [pgnls-filter start_epoch 85 out=49b] v81'6 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55958c368500 con 0x55958c5f6800 2024-03-31T15:11:19.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.675+0000 7feb2de00640 1 -- 172.21.15.16:0/3131452291 --> [v2:172.21.15.89:6800/1856844856,v1:172.21.15.89:6801/1856844856] -- osd_op(unknown.0.0:102 13.15 13:a8000000::::head [pgnls-filter start_epoch 85 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e85) -- 0x55958c5f7400 con 0x55958c5f6800 2024-03-31T15:11:19.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.675+0000 7feb34c00640 1 -- 172.21.15.16:0/3131452291 <== osd.4 v2:172.21.15.89:6800/1856844856 17 ==== osd_op_reply(102 [pgnls-filter start_epoch 85 out=49b] v81'6 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55958c368500 con 0x55958c5f6800 2024-03-31T15:11:19.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.675+0000 7feb2e800640 1 -- 172.21.15.16:0/3131452291 --> [v2:172.21.15.16:6808/1398737533,v1:172.21.15.16:6810/1398737533] -- osd_op(unknown.0.0:103 13.35 13:ac000000::::head [pgnls-filter start_epoch 85 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e85) -- 0x55958c608400 con 0x55958c5e3c00 2024-03-31T15:11:19.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.675+0000 7feb33800640 1 -- 172.21.15.16:0/3131452291 <== osd.1 v2:172.21.15.16:6808/1398737533 17 ==== osd_op_reply(103 [pgnls-filter start_epoch 85 out=49b] v81'6 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55958c368780 con 0x55958c5e3c00 2024-03-31T15:11:19.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.675+0000 7feb2de00640 1 -- 172.21.15.16:0/3131452291 --> [v2:172.21.15.89:6800/1856844856,v1:172.21.15.89:6801/1856844856] -- osd_op(unknown.0.0:104 13.d 13:b0000000::::head [pgnls-filter start_epoch 85 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e85) -- 0x55958c5f7800 con 0x55958c5f6800 2024-03-31T15:11:19.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.679+0000 7feb34c00640 1 -- 172.21.15.16:0/3131452291 <== osd.4 v2:172.21.15.89:6800/1856844856 18 ==== osd_op_reply(104 [pgnls-filter start_epoch 85 out=49b] v81'7 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55958c368500 con 0x55958c5f6800 2024-03-31T15:11:19.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.679+0000 7feb2e800640 1 -- 172.21.15.16:0/3131452291 --> [v2:172.21.15.16:6800/932712244,v1:172.21.15.16:6801/932712244] -- osd_op(unknown.0.0:105 13.2d 13:b4000000::::head [pgnls-filter start_epoch 85 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e85) -- 0x55958c613000 con 0x55958c5f6c00 2024-03-31T15:11:19.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.679+0000 7feb33800640 1 -- 172.21.15.16:0/3131452291 <== osd.0 v2:172.21.15.16:6800/932712244 7 ==== osd_op_reply(105 [pgnls-filter start_epoch 85 out=49b] v81'7 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55958c368780 con 0x55958c5f6c00 2024-03-31T15:11:19.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.679+0000 7feb2de00640 1 -- 172.21.15.16:0/3131452291 --> [v2:172.21.15.89:6808/1796778630,v1:172.21.15.89:6811/1796778630] -- osd_op(unknown.0.0:106 13.1d 13:b8000000::::head [pgnls-filter start_epoch 85 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e85) -- 0x55958c5f6000 con 0x55958d17dc00 2024-03-31T15:11:19.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.679+0000 7feb34200640 1 -- 172.21.15.16:0/3131452291 <== osd.7 v2:172.21.15.89:6808/1796778630 14 ==== osd_op_reply(106 [pgnls-filter start_epoch 85 out=49b] v81'15 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55958c368000 con 0x55958d17dc00 2024-03-31T15:11:19.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.679+0000 7feb2e800640 1 -- 172.21.15.16:0/3131452291 --> [v2:172.21.15.89:6808/1796778630,v1:172.21.15.89:6811/1796778630] -- osd_op(unknown.0.0:107 13.3d 13:bc000000::::head [pgnls-filter start_epoch 85 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e85) -- 0x55958c612400 con 0x55958d17dc00 2024-03-31T15:11:19.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.679+0000 7feb34200640 1 -- 172.21.15.16:0/3131452291 <== osd.7 v2:172.21.15.89:6808/1796778630 15 ==== osd_op_reply(107 [pgnls-filter start_epoch 85 out=49b] v81'7 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55958c368000 con 0x55958d17dc00 2024-03-31T15:11:19.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.679+0000 7feb2de00640 1 -- 172.21.15.16:0/3131452291 --> [v2:172.21.15.16:6800/932712244,v1:172.21.15.16:6801/932712244] -- osd_op(unknown.0.0:108 13.3 13:c0000000::::head [pgnls-filter start_epoch 85 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e85) -- 0x55958c5f7c00 con 0x55958c5f6c00 2024-03-31T15:11:19.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.679+0000 7feb33800640 1 -- 172.21.15.16:0/3131452291 <== osd.0 v2:172.21.15.16:6800/932712244 8 ==== osd_op_reply(108 [pgnls-filter start_epoch 85 out=49b] v81'7 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55958c368780 con 0x55958c5f6c00 2024-03-31T15:11:19.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.679+0000 7feb2e800640 1 -- 172.21.15.16:0/3131452291 --> [v2:172.21.15.16:6808/1398737533,v1:172.21.15.16:6810/1398737533] -- osd_op(unknown.0.0:109 13.23 13:c4000000::::head [pgnls-filter start_epoch 85 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e85) -- 0x55958c613400 con 0x55958c5e3c00 2024-03-31T15:11:19.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.679+0000 7feb33800640 1 -- 172.21.15.16:0/3131452291 <== osd.1 v2:172.21.15.16:6808/1398737533 18 ==== osd_op_reply(109 [pgnls-filter start_epoch 85 out=49b] v81'7 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55958c368780 con 0x55958c5e3c00 2024-03-31T15:11:19.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.679+0000 7feb2de00640 1 -- 172.21.15.16:0/3131452291 --> [v2:172.21.15.89:6808/1796778630,v1:172.21.15.89:6811/1796778630] -- osd_op(unknown.0.0:110 13.13 13:c8000000::::head [pgnls-filter start_epoch 85 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e85) -- 0x55958c5f7000 con 0x55958d17dc00 2024-03-31T15:11:19.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.679+0000 7feb34200640 1 -- 172.21.15.16:0/3131452291 <== osd.7 v2:172.21.15.89:6808/1796778630 16 ==== osd_op_reply(110 [pgnls-filter start_epoch 85 out=49b] v81'15 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55958c368000 con 0x55958d17dc00 2024-03-31T15:11:19.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.679+0000 7feb2e800640 1 -- 172.21.15.16:0/3131452291 --> [v2:172.21.15.89:6808/1796778630,v1:172.21.15.89:6811/1796778630] -- osd_op(unknown.0.0:111 13.33 13:cc000000::::head [pgnls-filter start_epoch 85 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e85) -- 0x55958c612c00 con 0x55958d17dc00 2024-03-31T15:11:19.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.679+0000 7feb34200640 1 -- 172.21.15.16:0/3131452291 <== osd.7 v2:172.21.15.89:6808/1796778630 17 ==== osd_op_reply(111 [pgnls-filter start_epoch 85 out=49b] v81'10 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55958c368000 con 0x55958d17dc00 2024-03-31T15:11:19.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.679+0000 7feb2de00640 1 -- 172.21.15.16:0/3131452291 --> [v2:172.21.15.89:6810/50395527,v1:172.21.15.89:6814/50395527] -- osd_op(unknown.0.0:112 13.b 13:d0000000::::head [pgnls-filter start_epoch 85 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e85) -- 0x55958d17d800 con 0x55958d17cc00 2024-03-31T15:11:19.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.679+0000 7feb34200640 1 -- 172.21.15.16:0/3131452291 <== osd.5 v2:172.21.15.89:6810/50395527 11 ==== osd_op_reply(112 [pgnls-filter start_epoch 85 out=49b] v81'8 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55958c368000 con 0x55958d17cc00 2024-03-31T15:11:19.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.679+0000 7feb2e800640 1 -- 172.21.15.16:0/3131452291 --> [v2:172.21.15.16:6819/3018526646,v1:172.21.15.16:6822/3018526646] -- osd_op(unknown.0.0:113 13.2b 13:d4000000::::head [pgnls-filter start_epoch 85 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e85) -- 0x55958c5e2000 con 0x55958c5f6400 2024-03-31T15:11:19.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.679+0000 7feb34200640 1 -- 172.21.15.16:0/3131452291 <== osd.2 v2:172.21.15.16:6819/3018526646 18 ==== osd_op_reply(113 [pgnls-filter start_epoch 85 out=49b] v81'11 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55958c368000 con 0x55958c5f6400 2024-03-31T15:11:19.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.679+0000 7feb2de00640 1 -- 172.21.15.16:0/3131452291 --> [v2:172.21.15.89:6800/1856844856,v1:172.21.15.89:6801/1856844856] -- osd_op(unknown.0.0:114 13.1b 13:d8000000::::head [pgnls-filter start_epoch 85 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e85) -- 0x55958c5f6000 con 0x55958c5f6800 2024-03-31T15:11:19.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.679+0000 7feb34c00640 1 -- 172.21.15.16:0/3131452291 <== osd.4 v2:172.21.15.89:6800/1856844856 19 ==== osd_op_reply(114 [pgnls-filter start_epoch 85 out=49b] v81'8 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55958c368500 con 0x55958c5f6800 2024-03-31T15:11:19.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.679+0000 7feb2e800640 1 -- 172.21.15.16:0/3131452291 --> [v2:172.21.15.89:6800/1856844856,v1:172.21.15.89:6801/1856844856] -- osd_op(unknown.0.0:115 13.3b 13:dc000000::::head [pgnls-filter start_epoch 85 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e85) -- 0x55958c5e3800 con 0x55958c5f6800 2024-03-31T15:11:19.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.679+0000 7feb34c00640 1 -- 172.21.15.16:0/3131452291 <== osd.4 v2:172.21.15.89:6800/1856844856 20 ==== osd_op_reply(115 [pgnls-filter start_epoch 85 out=49b] v81'4 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55958c368500 con 0x55958c5f6800 2024-03-31T15:11:19.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.679+0000 7feb2de00640 1 -- 172.21.15.16:0/3131452291 --> [v2:172.21.15.89:6808/1796778630,v1:172.21.15.89:6811/1796778630] -- osd_op(unknown.0.0:116 13.7 13:e0000000::::head [pgnls-filter start_epoch 85 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e85) -- 0x55958c5f7000 con 0x55958d17dc00 2024-03-31T15:11:19.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.683+0000 7feb34200640 1 -- 172.21.15.16:0/3131452291 <== osd.7 v2:172.21.15.89:6808/1796778630 18 ==== osd_op_reply(116 [pgnls-filter start_epoch 85 out=81b] v85'7 uv7 ondisk = 1) ==== 144+0+81 (crc 0 0 0) 0x55958c368000 con 0x55958d17dc00 2024-03-31T15:11:19.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.683+0000 7feb36c488c0 20 datascan.forall_objects: OSD matched oid 10000000001.00000000 2024-03-31T15:11:19.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.683+0000 7feb36c488c0 10 datascan.operator(): handling object 10000000001.0 2024-03-31T15:11:19.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.683+0000 7feb36c488c0 1 -- 172.21.15.16:0/3131452291 --> [v2:172.21.15.89:6808/1796778630,v1:172.21.15.89:6811/1796778630] -- osd_op(unknown.0.0:117 13.7 13:e2e07930:::10000000001.00000000:head [getxattr scan_ceiling in=12b,getxattr scan_max_size in=13b,getxattr scan_max_mtime in=14b,getxattr scan_pool_id in=12b,getxattr parent in=6b,getxattr layout in=6b,getxattr symlink in=7b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e85) -- 0x55958c5e2c00 con 0x55958d17dc00 2024-03-31T15:11:19.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.683+0000 7feb34200640 1 -- 172.21.15.16:0/3131452291 <== osd.7 v2:172.21.15.89:6808/1796778630 19 ==== osd_op_reply(117 10000000001.00000000 [getxattr (22) out=22b,getxattr (8) out=8b,getxattr (8) out=8b,getxattr,getxattr (89) out=89b,getxattr (30) out=30b,getxattr] v0'0 uv7 ondisk = 0) ==== 416+0+157 (crc 0 0 0) 0x55958c368000 con 0x55958d17dc00 2024-03-31T15:11:19.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.683+0000 7feb36c488c0 10 datascan.inject_with_backtrace: inode: 0x0x10000000001 2024-03-31T15:11:19.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.683+0000 7feb36c488c0 10 datascan.inject_with_backtrace: backptr: 0x0x10000000000/file 2024-03-31T15:11:19.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.683+0000 7feb36c488c0 20 datascan.get_frag_of: dirino=0x10000000000 target_dname=file 2024-03-31T15:11:19.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.683+0000 7feb36c488c0 1 -- 172.21.15.16:0/3131452291 --> [v2:172.21.15.16:6800/932712244,v1:172.21.15.16:6801/932712244] -- osd_op(unknown.0.0:118 12.10 12:0cf6ad0f:::10000000000.00000000:head [getxattr parent in=6b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e85) -- 0x55958c5e3000 con 0x55958c5f6c00 2024-03-31T15:11:19.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.683+0000 7feb33800640 1 -- 172.21.15.16:0/3131452291 <== osd.0 v2:172.21.15.16:6800/932712244 9 ==== osd_op_reply(118 10000000000.00000000 [getxattr (59) out=59b] v0'0 uv17 ondisk = 0) ==== 164+0+59 (crc 0 0 0) 0x55958c368780 con 0x55958c5f6c00 2024-03-31T15:11:19.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.683+0000 7feb36c488c0 20 datascan.get_frag_of: got backtrace parent 0x1/dir 2024-03-31T15:11:19.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.683+0000 7feb36c488c0 1 -- 172.21.15.16:0/3131452291 --> [v2:172.21.15.89:6809/3470471752,v1:172.21.15.89:6812/3470471752] -- osd_op(unknown.0.0:119 12.1f 12:ff5b34d6:::1.00000000:head [omap-get-vals-by-keys in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e85) -- 0x55958c5e3400 con 0x55958d17d400 2024-03-31T15:11:19.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.683+0000 7feb34c00640 1 -- 172.21.15.16:0/3131452291 <== osd.6 v2:172.21.15.89:6809/3470471752 18 ==== osd_op_reply(119 1.00000000 [omap-get-vals-by-keys out=520b] v0'0 uv9 ondisk = 0) ==== 154+0+520 (crc 0 0 0) 0x55958c368500 con 0x55958d17d400 2024-03-31T15:11:19.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.683+0000 7feb36c488c0 20 datascan.read_dentry: oid=1.00000000 dname=dir frag=*, r=0 2024-03-31T15:11:19.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.683+0000 7feb36c488c0 20 datascan.get_frag_of: fast path, fragtree is fragtree_t(*) 2024-03-31T15:11:19.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.683+0000 7feb36c488c0 20 datascan.get_frag_of: frag is * 2024-03-31T15:11:19.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.683+0000 7feb36c488c0 1 -- 172.21.15.16:0/3131452291 --> [v2:172.21.15.16:6800/932712244,v1:172.21.15.16:6801/932712244] -- osd_op(unknown.0.0:120 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e85) -- 0x55958c612000 con 0x55958c5f6c00 2024-03-31T15:11:19.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.683+0000 7feb33800640 1 -- 172.21.15.16:0/3131452291 <== osd.0 v2:172.21.15.16:6800/932712244 10 ==== osd_op_reply(120 10000000000.00000000 [omap-get-header out=274b] v0'0 uv17 ondisk = 0) ==== 164+0+274 (crc 0 0 0) 0x55958c368780 con 0x55958c5f6c00 2024-03-31T15:11:19.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.683+0000 7feb36c488c0 10 datascan.find_or_create_dirfrag: read_version = 17 2024-03-31T15:11:19.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.683+0000 7feb36c488c0 20 datascan.find_or_create_dirfrag: Dirfrag already exists: 0x0x10000000000 * 2024-03-31T15:11:19.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.683+0000 7feb36c488c0 1 -- 172.21.15.16:0/3131452291 --> [v2:172.21.15.16:6800/932712244,v1:172.21.15.16:6801/932712244] -- osd_op(unknown.0.0:121 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-get-vals-by-keys in=17b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e85) -- 0x55958c612400 con 0x55958c5f6c00 2024-03-31T15:11:19.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.683+0000 7feb33800640 1 -- 172.21.15.16:0/3131452291 <== osd.0 v2:172.21.15.16:6800/932712244 11 ==== osd_op_reply(121 10000000000.00000000 [omap-get-vals-by-keys out=4b] v0'0 uv17 ondisk = 0) ==== 164+0+4 (crc 0 0 0) 0x55958c368780 con 0x55958c5f6c00 2024-03-31T15:11:19.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.683+0000 7feb36c488c0 20 datascan.read_dentry: oid=10000000000.00000000 dname=file frag=*, r=0 2024-03-31T15:11:19.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.683+0000 7feb36c488c0 20 datascan.read_dentry: file_head not found in result 2024-03-31T15:11:19.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.683+0000 7feb36c488c0 10 datascan.inject_with_backtrace: Linking inode 0x0x10000000001 at 0x0x10000000000/file with size=0 bytes 2024-03-31T15:11:19.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.683+0000 7feb36c488c0 1 -- 172.21.15.16:0/3131452291 --> [v2:172.21.15.16:6800/932712244,v1:172.21.15.16:6801/932712244] -- osd_op(unknown.0.0:122 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-set-vals in=505b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85) -- 0x55958c612800 con 0x55958c5f6c00 2024-03-31T15:11:19.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.683+0000 7feb33800640 1 -- 172.21.15.16:0/3131452291 <== osd.0 v2:172.21.15.16:6800/932712244 12 ==== osd_op_reply(122 10000000000.00000000 [omap-set-vals] v85'18 uv18 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x55958c368780 con 0x55958c5f6c00 2024-03-31T15:11:19.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.683+0000 7feb36c488c0 20 datascan.inject_linkage: Injected dentry 0x0x10000000000/file pointing to 0x0x10000000001 2024-03-31T15:11:19.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.683+0000 7feb36c488c0 1 -- 172.21.15.16:0/3131452291 --> [v2:172.21.15.16:6808/1398737533,v1:172.21.15.16:6810/1398737533] -- osd_op(unknown.0.0:123 13.27 13:e4000000::::head [pgnls-filter start_epoch 85 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e85) -- 0x55958c612c00 con 0x55958c5e3c00 2024-03-31T15:11:19.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.687+0000 7feb33800640 1 -- 172.21.15.16:0/3131452291 <== osd.1 v2:172.21.15.16:6808/1398737533 19 ==== osd_op_reply(123 [pgnls-filter start_epoch 85 out=49b] v81'2 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55958c368780 con 0x55958c5e3c00 2024-03-31T15:11:19.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.687+0000 7feb2de00640 1 -- 172.21.15.16:0/3131452291 --> [v2:172.21.15.89:6810/50395527,v1:172.21.15.89:6814/50395527] -- osd_op(unknown.0.0:124 13.17 13:e8000000::::head [pgnls-filter start_epoch 85 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e85) -- 0x55958c5f7c00 con 0x55958d17cc00 2024-03-31T15:11:19.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.687+0000 7feb34200640 1 -- 172.21.15.16:0/3131452291 <== osd.5 v2:172.21.15.89:6810/50395527 12 ==== osd_op_reply(124 [pgnls-filter start_epoch 85 out=49b] v81'9 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55958c368000 con 0x55958d17cc00 2024-03-31T15:11:19.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.687+0000 7feb2e800640 1 -- 172.21.15.16:0/3131452291 --> [v2:172.21.15.89:6810/50395527,v1:172.21.15.89:6814/50395527] -- osd_op(unknown.0.0:125 13.37 13:ec000000::::head [pgnls-filter start_epoch 85 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e85) -- 0x55958c608c00 con 0x55958d17cc00 2024-03-31T15:11:19.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.687+0000 7feb34200640 1 -- 172.21.15.16:0/3131452291 <== osd.5 v2:172.21.15.89:6810/50395527 13 ==== osd_op_reply(125 [pgnls-filter start_epoch 85 out=49b] v81'11 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55958c368000 con 0x55958d17cc00 2024-03-31T15:11:19.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.687+0000 7feb2de00640 1 -- 172.21.15.16:0/3131452291 --> [v2:172.21.15.89:6809/3470471752,v1:172.21.15.89:6812/3470471752] -- osd_op(unknown.0.0:126 13.f 13:f0000000::::head [pgnls-filter start_epoch 85 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e85) -- 0x55958c609400 con 0x55958d17d400 2024-03-31T15:11:19.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.687+0000 7feb34c00640 1 -- 172.21.15.16:0/3131452291 <== osd.6 v2:172.21.15.89:6809/3470471752 19 ==== osd_op_reply(126 [pgnls-filter start_epoch 85 out=49b] v81'9 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55958c368500 con 0x55958d17d400 2024-03-31T15:11:19.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.687+0000 7feb2e800640 1 -- 172.21.15.16:0/3131452291 --> [v2:172.21.15.89:6800/1856844856,v1:172.21.15.89:6801/1856844856] -- osd_op(unknown.0.0:127 13.2f 13:f4000000::::head [pgnls-filter start_epoch 85 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e85) -- 0x55958c609800 con 0x55958c5f6800 2024-03-31T15:11:19.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.687+0000 7feb34c00640 1 -- 172.21.15.16:0/3131452291 <== osd.4 v2:172.21.15.89:6800/1856844856 21 ==== osd_op_reply(127 [pgnls-filter start_epoch 85 out=49b] v81'8 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55958c368500 con 0x55958c5f6800 2024-03-31T15:11:19.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.687+0000 7feb2de00640 1 -- 172.21.15.16:0/3131452291 --> [v2:172.21.15.89:6809/3470471752,v1:172.21.15.89:6812/3470471752] -- osd_op(unknown.0.0:128 13.1f 13:f8000000::::head [pgnls-filter start_epoch 85 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e85) -- 0x55958c609c00 con 0x55958d17d400 2024-03-31T15:11:19.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.687+0000 7feb34c00640 1 -- 172.21.15.16:0/3131452291 <== osd.6 v2:172.21.15.89:6809/3470471752 20 ==== osd_op_reply(128 [pgnls-filter start_epoch 85 out=49b] v81'6 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55958c368500 con 0x55958d17d400 2024-03-31T15:11:19.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.687+0000 7feb2e800640 1 -- 172.21.15.16:0/3131452291 --> [v2:172.21.15.89:6809/3470471752,v1:172.21.15.89:6812/3470471752] -- osd_op(unknown.0.0:129 13.3f 13:fc000000::::head [pgnls-filter start_epoch 85 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e85) -- 0x55958d17d000 con 0x55958d17d400 2024-03-31T15:11:19.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.687+0000 7feb34c00640 1 -- 172.21.15.16:0/3131452291 <== osd.6 v2:172.21.15.89:6809/3470471752 21 ==== osd_op_reply(129 [pgnls-filter start_epoch 85 out=49b] v81'8 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55958c368500 con 0x55958d17d400 2024-03-31T15:11:19.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.687+0000 7feb36c488c0 1 -- 172.21.15.16:0/3131452291 >> [v2:172.21.15.16:6800/932712244,v1:172.21.15.16:6801/932712244] conn(0x55958c5f6c00 msgr2=0x55958c5f3080 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:11:19.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.687+0000 7feb36c488c0 1 --2- 172.21.15.16:0/3131452291 >> [v2:172.21.15.16:6800/932712244,v1:172.21.15.16:6801/932712244] conn(0x55958c5f6c00 0x55958c5f3080 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:19.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.687+0000 7feb36c488c0 1 -- 172.21.15.16:0/3131452291 >> [v2:172.21.15.16:6808/1398737533,v1:172.21.15.16:6810/1398737533] conn(0x55958c5e3c00 msgr2=0x55958c5f2000 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:11:19.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.687+0000 7feb36c488c0 1 --2- 172.21.15.16:0/3131452291 >> [v2:172.21.15.16:6808/1398737533,v1:172.21.15.16:6810/1398737533] conn(0x55958c5e3c00 0x55958c5f2000 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:19.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.687+0000 7feb36c488c0 1 -- 172.21.15.16:0/3131452291 >> [v2:172.21.15.16:6819/3018526646,v1:172.21.15.16:6822/3018526646] conn(0x55958c5f6400 msgr2=0x55958c5f2b00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:11:19.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.687+0000 7feb36c488c0 1 --2- 172.21.15.16:0/3131452291 >> [v2:172.21.15.16:6819/3018526646,v1:172.21.15.16:6822/3018526646] conn(0x55958c5f6400 0x55958c5f2b00 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:19.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.687+0000 7feb36c488c0 1 -- 172.21.15.16:0/3131452291 >> [v2:172.21.15.16:6809/3360352974,v1:172.21.15.16:6812/3360352974] conn(0x55958c608000 msgr2=0x55958c5f3600 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:11:19.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.687+0000 7feb36c488c0 1 --2- 172.21.15.16:0/3131452291 >> [v2:172.21.15.16:6809/3360352974,v1:172.21.15.16:6812/3360352974] conn(0x55958c608000 0x55958c5f3600 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:19.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.687+0000 7feb36c488c0 1 -- 172.21.15.16:0/3131452291 >> [v2:172.21.15.89:6800/1856844856,v1:172.21.15.89:6801/1856844856] conn(0x55958c5f6800 msgr2=0x55958c5f2580 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:11:19.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.687+0000 7feb36c488c0 1 --2- 172.21.15.16:0/3131452291 >> [v2:172.21.15.89:6800/1856844856,v1:172.21.15.89:6801/1856844856] conn(0x55958c5f6800 0x55958c5f2580 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:19.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.687+0000 7feb36c488c0 1 -- 172.21.15.16:0/3131452291 >> [v2:172.21.15.89:6810/50395527,v1:172.21.15.89:6814/50395527] conn(0x55958d17cc00 msgr2=0x55958cfc6680 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:11:19.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.687+0000 7feb36c488c0 1 --2- 172.21.15.16:0/3131452291 >> [v2:172.21.15.89:6810/50395527,v1:172.21.15.89:6814/50395527] conn(0x55958d17cc00 0x55958cfc6680 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:19.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.687+0000 7feb36c488c0 1 -- 172.21.15.16:0/3131452291 >> [v2:172.21.15.89:6809/3470471752,v1:172.21.15.89:6812/3470471752] conn(0x55958d17d400 msgr2=0x55958cfc6c00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:11:19.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.687+0000 7feb36c488c0 1 --2- 172.21.15.16:0/3131452291 >> [v2:172.21.15.89:6809/3470471752,v1:172.21.15.89:6812/3470471752] conn(0x55958d17d400 0x55958cfc6c00 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:19.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.687+0000 7feb36c488c0 1 -- 172.21.15.16:0/3131452291 >> [v2:172.21.15.89:6808/1796778630,v1:172.21.15.89:6811/1796778630] conn(0x55958d17dc00 msgr2=0x55958cfc7180 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:11:19.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.687+0000 7feb36c488c0 1 --2- 172.21.15.16:0/3131452291 >> [v2:172.21.15.89:6808/1796778630,v1:172.21.15.89:6811/1796778630] conn(0x55958d17dc00 0x55958cfc7180 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:19.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.687+0000 7feb33800640 1 -- 172.21.15.16:0/3131452291 reap_dead start 2024-03-31T15:11:19.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.687+0000 7feb33800640 1 -- 172.21.15.16:0/3131452291 reap_dead start 2024-03-31T15:11:19.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.687+0000 7feb36c488c0 1 -- 172.21.15.16:0/3131452291 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x55958d17c800 msgr2=0x55958cfc6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:11:19.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.687+0000 7feb36c488c0 1 --2- 172.21.15.16:0/3131452291 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x55958d17c800 0x55958cfc6100 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x55958d126960 tx=0x55958cfd45a0 comp rx=0 tx=0).stop 2024-03-31T15:11:19.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.687+0000 7feb36c488c0 1 -- 172.21.15.16:0/3131452291 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55958d11fc00 msgr2=0x55958cfc5b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:11:19.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.687+0000 7feb36c488c0 1 --2- 172.21.15.16:0/3131452291 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55958d11fc00 0x55958cfc5b80 secure :-1 s=READY pgs=1255 cs=0 l=1 rev1=1 crypto rx=0x55958d059020 tx=0x55958cfd4000 comp rx=0 tx=0).stop 2024-03-31T15:11:19.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.687+0000 7feb36c488c0 1 -- 172.21.15.16:0/3131452291 shutdown_connections 2024-03-31T15:11:19.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.687+0000 7feb36c488c0 1 --2- 172.21.15.16:0/3131452291 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x55958d17c800 0x55958cfc6100 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:19.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.687+0000 7feb36c488c0 1 --2- 172.21.15.16:0/3131452291 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55958d11fc00 0x55958cfc5b80 unknown :-1 s=CLOSED pgs=1255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:19.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.687+0000 7feb36c488c0 1 -- 172.21.15.16:0/3131452291 >> 172.21.15.16:0/3131452291 conn(0x55958d11f000 msgr2=0x55958c222000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:11:19.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.687+0000 7feb36c488c0 1 -- 172.21.15.16:0/3131452291 shutdown_connections 2024-03-31T15:11:19.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.687+0000 7feb36c488c0 1 -- 172.21.15.16:0/3131452291 wait complete. 2024-03-31T15:11:19.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.687+0000 7feb36c488c0 1 -- 172.21.15.16:0/1562624489 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55958c1f5000 msgr2=0x55958cfc4000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:11:19.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.687+0000 7feb36c488c0 1 --2- 172.21.15.16:0/1562624489 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55958c1f5000 0x55958cfc4000 secure :-1 s=READY pgs=1253 cs=0 l=1 rev1=1 crypto rx=0x55958d059a70 tx=0x55958cfd4600 comp rx=0 tx=0).stop 2024-03-31T15:11:19.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.687+0000 7feb36c488c0 1 -- 172.21.15.16:0/1562624489 shutdown_connections 2024-03-31T15:11:19.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.687+0000 7feb36c488c0 1 --2- 172.21.15.16:0/1562624489 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55958c1f5800 0x55958cfc4b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:19.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.687+0000 7feb36c488c0 1 --2- 172.21.15.16:0/1562624489 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55958c1f5400 0x55958cfc4580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:19.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.687+0000 7feb36c488c0 1 --2- 172.21.15.16:0/1562624489 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55958c1f5000 0x55958cfc4000 unknown :-1 s=CLOSED pgs=1253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:19.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.687+0000 7feb36c488c0 1 -- 172.21.15.16:0/1562624489 >> 172.21.15.16:0/1562624489 conn(0x55958c1f4c00 msgr2=0x55958c221800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:11:19.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.687+0000 7feb36c488c0 1 -- 172.21.15.16:0/1562624489 shutdown_connections 2024-03-31T15:11:19.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.687+0000 7feb36c488c0 1 -- 172.21.15.16:0/1562624489 wait complete. 2024-03-31T15:11:19.723 DEBUG:tasks.cephfs.filesystem:Ran ['cephfs-data-scan', '--debug-mds=20', '--debug-ms=1', '--debug-objecter=1', 'scan_inodes', 'cephfs_data'] in time 0:00:00.196990, result: 2024-03-31T15:11:19.724 DEBUG:teuthology.orchestra.run.smithi016:> cephfs-data-scan --debug-mds=20 --debug-ms=1 --debug-objecter=1 scan_links 2024-03-31T15:11:19.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.783+0000 7f65f31ca8c0 1 -- start start 2024-03-31T15:11:19.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.783+0000 7f65f31ca8c0 1 Processor -- start 2024-03-31T15:11:19.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.783+0000 7f65f31ca8c0 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d9b7c27800 0x55d9b89f0000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:19.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.783+0000 7f65f31ca8c0 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55d9b7c27400 0x55d9b89f0580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:19.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.783+0000 7f65f31ca8c0 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d9b7c27000 0x55d9b89f0b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:19.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.783+0000 7f65f31ca8c0 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55d9b7d9a600 con 0x55d9b7c27800 2024-03-31T15:11:19.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.783+0000 7f65f31ca8c0 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55d9b7d9a480 con 0x55d9b7c27000 2024-03-31T15:11:19.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.783+0000 7f65f31ca8c0 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x55d9b7d9a300 con 0x55d9b7c27400 2024-03-31T15:11:19.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.783+0000 7f65f1200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d9b7c27800 0x55d9b89f0000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:19.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.783+0000 7f65f1200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d9b7c27800 0x55d9b89f0000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56212/0 (socket says 172.21.15.16:56212) 2024-03-31T15:11:19.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.783+0000 7f65f1200640 1 -- 172.21.15.16:0/3093794445 learned_addr learned my addr 172.21.15.16:0/3093794445 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:11:19.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.783+0000 7f65f0800640 1 --2- 172.21.15.16:0/3093794445 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55d9b7c27400 0x55d9b89f0580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:19.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.783+0000 7f65f1c00640 1 --2- 172.21.15.16:0/3093794445 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d9b7c27000 0x55d9b89f0b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:19.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.783+0000 7f65f1200640 1 -- 172.21.15.16:0/3093794445 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55d9b7c27400 msgr2=0x55d9b89f0580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:11:19.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.783+0000 7f65f1200640 1 --2- 172.21.15.16:0/3093794445 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55d9b7c27400 0x55d9b89f0580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:19.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.783+0000 7f65f1200640 1 -- 172.21.15.16:0/3093794445 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d9b7c27000 msgr2=0x55d9b89f0b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:11:19.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.783+0000 7f65f1200640 1 --2- 172.21.15.16:0/3093794445 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d9b7c27000 0x55d9b89f0b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:19.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.783+0000 7f65f1200640 1 -- 172.21.15.16:0/3093794445 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d9b7c4c1e0 con 0x55d9b7c27800 2024-03-31T15:11:19.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.783+0000 7f65f1c00640 1 --2- 172.21.15.16:0/3093794445 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d9b7c27000 0x55d9b89f0b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:11:19.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.783+0000 7f65f1200640 1 --2- 172.21.15.16:0/3093794445 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d9b7c27800 0x55d9b89f0000 secure :-1 s=READY pgs=1257 cs=0 l=1 rev1=1 crypto rx=0x55d9b8a7d8c0 tx=0x55d9b89aea80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2cd8a21551908c04 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:11:19.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.783+0000 7f65ee000640 1 -- 172.21.15.16:0/3093794445 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x55d9b7c5c4e0 con 0x55d9b7c27800 2024-03-31T15:11:19.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.783+0000 7f65ee000640 1 -- 172.21.15.16:0/3093794445 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x55d9b7c72c40 con 0x55d9b7c27800 2024-03-31T15:11:19.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.783+0000 7f65ee000640 1 -- 172.21.15.16:0/3093794445 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x55d9b7c5cb60 con 0x55d9b7c27800 2024-03-31T15:11:19.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.783+0000 7f65f31ca8c0 1 -- 172.21.15.16:0/3093794445 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d9b7c4d2c0 con 0x55d9b7c27800 2024-03-31T15:11:19.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.783+0000 7f65f31ca8c0 1 -- 172.21.15.16:0/3093794445 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({fsmap=0}) -- 0x55d9b7c4da40 con 0x55d9b7c27800 2024-03-31T15:11:19.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.783+0000 7f65f31ca8c0 4 waiting for MDS map... 2024-03-31T15:11:19.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.783+0000 7f65ee000640 1 -- 172.21.15.16:0/3093794445 <== mon.0 v2:172.21.15.16:3300/0 4 ==== osd_map(85..85 src has 1..85) ==== 5910+0+0 (secure 0 0 0) 0x55d9b8b4a600 con 0x55d9b7c27800 2024-03-31T15:11:19.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.783+0000 7f65ee000640 1 -- 172.21.15.16:0/3093794445 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 69) ==== 2885+0+0 (secure 0 0 0) 0x55d9b7c7d180 con 0x55d9b7c27800 2024-03-31T15:11:19.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.783+0000 7f65f31ca8c0 4 Got MDS map 69 2024-03-31T15:11:19.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.787+0000 7f65f31ca8c0 4 datascan.main: Using metadata pool output 2024-03-31T15:11:19.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.787+0000 7f65f31ca8c0 4 datascan.main: connecting to RADOS... 2024-03-31T15:11:19.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.787+0000 7f65f31ca8c0 1 Processor -- start 2024-03-31T15:11:19.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.787+0000 7f65f31ca8c0 1 -- start start 2024-03-31T15:11:19.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.787+0000 7f65f31ca8c0 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d9b8b4f000 0x55d9b89f1080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:19.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.787+0000 7f65f31ca8c0 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55d9b8b4f400 0x55d9b89f1600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:19.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.787+0000 7f65f31ca8c0 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d9b8b4f800 0x55d9b89f1b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:19.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.787+0000 7f65f31ca8c0 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55d9b7d9a180 con 0x55d9b8b4f800 2024-03-31T15:11:19.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.787+0000 7f65f31ca8c0 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55d9b7c86480 con 0x55d9b8b4f000 2024-03-31T15:11:19.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.787+0000 7f65f31ca8c0 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x55d9b7c86900 con 0x55d9b8b4f400 2024-03-31T15:11:19.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.787+0000 7f65f0800640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d9b8b4f800 0x55d9b89f1b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:19.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.787+0000 7f65f0800640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d9b8b4f800 0x55d9b89f1b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56214/0 (socket says 172.21.15.16:56214) 2024-03-31T15:11:19.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.787+0000 7f65f0800640 1 -- 172.21.15.16:0/2475217666 learned_addr learned my addr 172.21.15.16:0/2475217666 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:11:19.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.787+0000 7f65f0800640 1 -- 172.21.15.16:0/2475217666 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55d9b8b4f400 msgr2=0x55d9b89f1600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:11:19.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.787+0000 7f65f1200640 1 --2- 172.21.15.16:0/2475217666 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55d9b8b4f400 0x55d9b89f1600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:19.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.787+0000 7f65f1c00640 1 --2- 172.21.15.16:0/2475217666 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d9b8b4f000 0x55d9b89f1080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:19.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.787+0000 7f65f0800640 1 --2- 172.21.15.16:0/2475217666 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55d9b8b4f400 0x55d9b89f1600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:19.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.787+0000 7f65f0800640 1 -- 172.21.15.16:0/2475217666 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d9b8b4f000 msgr2=0x55d9b89f1080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:11:19.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.787+0000 7f65f0800640 1 --2- 172.21.15.16:0/2475217666 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d9b8b4f000 0x55d9b89f1080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:19.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.787+0000 7f65f0800640 1 -- 172.21.15.16:0/2475217666 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d9b7c4dc20 con 0x55d9b8b4f800 2024-03-31T15:11:19.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.787+0000 7f65f1200640 1 --2- 172.21.15.16:0/2475217666 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55d9b8b4f400 0x55d9b89f1600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:11:19.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.787+0000 7f65f1c00640 1 --2- 172.21.15.16:0/2475217666 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d9b8b4f000 0x55d9b89f1080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:11:19.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.787+0000 7f65f0800640 1 --2- 172.21.15.16:0/2475217666 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d9b8b4f800 0x55d9b89f1b80 secure :-1 s=READY pgs=1258 cs=0 l=1 rev1=1 crypto rx=0x55d9b8b60e10 tx=0x55d9b7f1a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2cd1a1e93a73b99f server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:11:19.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.787+0000 7f65eb800640 1 -- 172.21.15.16:0/2475217666 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x55d9b7c5c4e0 con 0x55d9b8b4f800 2024-03-31T15:11:19.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.787+0000 7f65eb800640 1 -- 172.21.15.16:0/2475217666 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x55d9b7c72a80 con 0x55d9b8b4f800 2024-03-31T15:11:19.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.787+0000 7f65eb800640 1 -- 172.21.15.16:0/2475217666 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x55d9b7c5cb60 con 0x55d9b8b4f800 2024-03-31T15:11:19.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.787+0000 7f65f31ca8c0 1 -- 172.21.15.16:0/2475217666 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d9b8b4f800 msgr2=0x55d9b89f1b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:11:19.787 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.787+0000 7f65f31ca8c0 1 --2- 172.21.15.16:0/2475217666 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d9b8b4f800 0x55d9b89f1b80 secure :-1 s=READY pgs=1258 cs=0 l=1 rev1=1 crypto rx=0x55d9b8b60e10 tx=0x55d9b7f1a000 comp rx=0 tx=0).stop 2024-03-31T15:11:19.787 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.787+0000 7f65f31ca8c0 1 -- 172.21.15.16:0/2475217666 shutdown_connections 2024-03-31T15:11:19.787 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.787+0000 7f65f31ca8c0 1 --2- 172.21.15.16:0/2475217666 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d9b8b4f800 0x55d9b89f1b80 unknown :-1 s=CLOSED pgs=1258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:19.787 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.787+0000 7f65f31ca8c0 1 --2- 172.21.15.16:0/2475217666 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55d9b8b4f400 0x55d9b89f1600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:19.787 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.787+0000 7f65f31ca8c0 1 --2- 172.21.15.16:0/2475217666 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d9b8b4f000 0x55d9b89f1080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:19.787 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.787+0000 7f65f31ca8c0 1 -- 172.21.15.16:0/2475217666 >> 172.21.15.16:0/2475217666 conn(0x55d9b7c26c00 msgr2=0x55d9b7c53000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:11:19.787 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.787+0000 7f65f31ca8c0 1 -- 172.21.15.16:0/2475217666 shutdown_connections 2024-03-31T15:11:19.787 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.787+0000 7f65f31ca8c0 1 -- 172.21.15.16:0/2475217666 wait complete. 2024-03-31T15:11:19.787 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.787+0000 7f65f31ca8c0 1 Processor -- start 2024-03-31T15:11:19.787 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.791+0000 7f65f31ca8c0 1 -- start start 2024-03-31T15:11:19.787 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.791+0000 7f65f31ca8c0 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d9b8b4f000 0x55d9b89f1080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:19.787 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.791+0000 7f65f31ca8c0 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d9b8b4f400 0x55d9b89f1600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:19.787 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.791+0000 7f65f31ca8c0 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55d9b8b4f800 0x55d9b89f1b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:19.787 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.791+0000 7f65f31ca8c0 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55d9b7d9b380 con 0x55d9b8b4f000 2024-03-31T15:11:19.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.791+0000 7f65f31ca8c0 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55d9b7d9a180 con 0x55d9b8b4f400 2024-03-31T15:11:19.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.791+0000 7f65f31ca8c0 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x55d9b7d9b500 con 0x55d9b8b4f800 2024-03-31T15:11:19.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.791+0000 7f65f1c00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d9b8b4f000 0x55d9b89f1080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:19.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.791+0000 7f65f1c00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d9b8b4f000 0x55d9b89f1080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56222/0 (socket says 172.21.15.16:56222) 2024-03-31T15:11:19.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.791+0000 7f65f1c00640 1 -- 172.21.15.16:0/4116959849 learned_addr learned my addr 172.21.15.16:0/4116959849 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:11:19.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.791+0000 7f65f0800640 1 --2- 172.21.15.16:0/4116959849 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55d9b8b4f800 0x55d9b89f1b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:19.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.791+0000 7f65f1200640 1 --2- 172.21.15.16:0/4116959849 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d9b8b4f400 0x55d9b89f1600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:19.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.791+0000 7f65f1c00640 1 -- 172.21.15.16:0/4116959849 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55d9b8b4f800 msgr2=0x55d9b89f1b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:11:19.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.791+0000 7f65f1c00640 1 --2- 172.21.15.16:0/4116959849 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55d9b8b4f800 0x55d9b89f1b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:19.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.791+0000 7f65f1c00640 1 -- 172.21.15.16:0/4116959849 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d9b8b4f400 msgr2=0x55d9b89f1600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:11:19.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.791+0000 7f65f1c00640 1 --2- 172.21.15.16:0/4116959849 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d9b8b4f400 0x55d9b89f1600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:19.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.791+0000 7f65f1c00640 1 -- 172.21.15.16:0/4116959849 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d9b7c4c1e0 con 0x55d9b8b4f000 2024-03-31T15:11:19.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.791+0000 7f65f1c00640 1 --2- 172.21.15.16:0/4116959849 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d9b8b4f000 0x55d9b89f1080 secure :-1 s=READY pgs=1259 cs=0 l=1 rev1=1 crypto rx=0x55d9b8a7cff0 tx=0x55d9b8994000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:11:19.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.791+0000 7f65e9a00640 1 -- 172.21.15.16:0/4116959849 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x55d9b7c5c9c0 con 0x55d9b8b4f000 2024-03-31T15:11:19.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.791+0000 7f65e9a00640 1 -- 172.21.15.16:0/4116959849 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x55d9b7c72a80 con 0x55d9b8b4f000 2024-03-31T15:11:19.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.791+0000 7f65f31ca8c0 1 -- 172.21.15.16:0/4116959849 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55d9b7c4d2c0 con 0x55d9b8b4f000 2024-03-31T15:11:19.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.791+0000 7f65e9a00640 1 -- 172.21.15.16:0/4116959849 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x55d9b7c5c4e0 con 0x55d9b8b4f000 2024-03-31T15:11:19.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.791+0000 7f65f31ca8c0 1 -- 172.21.15.16:0/4116959849 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d9b7c4da40 con 0x55d9b8b4f000 2024-03-31T15:11:19.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.791+0000 7f65f31ca8c0 4 datascan.init: resolving metadata pool 12 2024-03-31T15:11:19.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.791+0000 7f65e9a00640 1 -- 172.21.15.16:0/4116959849 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x55d9b7f24c00 con 0x55d9b8b4f000 2024-03-31T15:11:19.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.795+0000 7f65e9a00640 1 --2- 172.21.15.16:0/4116959849 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x55d9b7f18c00 0x55d9b89f2100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:19.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.795+0000 7f65e9a00640 1 -- 172.21.15.16:0/4116959849 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 5910+0+0 (secure 0 0 0) 0x55d9b7f1ca00 con 0x55d9b8b4f000 2024-03-31T15:11:19.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.795+0000 7f65f31ca8c0 4 datascan.init: found metadata pool 'cephfs_metadata' 2024-03-31T15:11:19.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.795+0000 7f65f31ca8c0 4 datascan.main: resolving metadata pool 12 2024-03-31T15:11:19.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.795+0000 7f65f31ca8c0 1 --2- 172.21.15.16:0/4116959849 >> [v2:172.21.15.89:6800/1856844856,v1:172.21.15.89:6801/1856844856] conn(0x55d9b8b4fc00 0x55d9b89f2680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:19.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.795+0000 7f65f31ca8c0 1 -- 172.21.15.16:0/4116959849 --> [v2:172.21.15.89:6800/1856844856,v1:172.21.15.89:6801/1856844856] -- osd_op(unknown.0.0:1 12.0 12:00000000::::head [pgnls start_epoch 85 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e85) -- 0x55d9b7f19000 con 0x55d9b8b4fc00 2024-03-31T15:11:19.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.795+0000 7f65f1200640 1 --2- 172.21.15.16:0/4116959849 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x55d9b7f18c00 0x55d9b89f2100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:19.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.795+0000 7f65f0800640 1 --2- 172.21.15.16:0/4116959849 >> [v2:172.21.15.89:6800/1856844856,v1:172.21.15.89:6801/1856844856] conn(0x55d9b8b4fc00 0x55d9b89f2680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:19.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.795+0000 7f65f1200640 1 --2- 172.21.15.16:0/4116959849 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x55d9b7f18c00 0x55d9b89f2100 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x55d9b8a7db90 tx=0x55d9b7cdef00 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:11:19.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.795+0000 7f65f0800640 1 --2- 172.21.15.16:0/4116959849 >> [v2:172.21.15.89:6800/1856844856,v1:172.21.15.89:6801/1856844856] conn(0x55d9b8b4fc00 0x55d9b89f2680 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:11:19.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.795+0000 7f65f0800640 1 -- 172.21.15.16:0/4116959849 <== osd.4 v2:172.21.15.89:6800/1856844856 1 ==== osd_op_reply(1 [pgnls start_epoch 85 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55d9b7d1e000 con 0x55d9b8b4fc00 2024-03-31T15:11:19.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.795+0000 7f65f0800640 1 --2- 172.21.15.16:0/4116959849 >> [v2:172.21.15.16:6800/932712244,v1:172.21.15.16:6801/932712244] conn(0x55d9b7f18800 0x55d9b8020000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:19.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.795+0000 7f65f0800640 1 -- 172.21.15.16:0/4116959849 --> [v2:172.21.15.16:6800/932712244,v1:172.21.15.16:6801/932712244] -- osd_op(unknown.0.0:2 12.10 12:08000000::::head [pgnls start_epoch 85 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e85) -- 0x55d9b7f18000 con 0x55d9b7f18800 2024-03-31T15:11:19.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.795+0000 7f65f1c00640 1 --2- 172.21.15.16:0/4116959849 >> [v2:172.21.15.16:6800/932712244,v1:172.21.15.16:6801/932712244] conn(0x55d9b7f18800 0x55d9b8020000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:19.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.795+0000 7f65f1c00640 1 --2- 172.21.15.16:0/4116959849 >> [v2:172.21.15.16:6800/932712244,v1:172.21.15.16:6801/932712244] conn(0x55d9b7f18800 0x55d9b8020000 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:11:19.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.795+0000 7f65f1c00640 1 -- 172.21.15.16:0/4116959849 <== osd.0 v2:172.21.15.16:6800/932712244 1 ==== osd_op_reply(2 [pgnls start_epoch 85 out=177b] v85'18 uv18 ondisk = 1) ==== 144+0+177 (crc 0 0 0) 0x55d9b7d1e500 con 0x55d9b7f18800 2024-03-31T15:11:19.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.795+0000 7f65f1c00640 1 --2- 172.21.15.16:0/4116959849 >> [v2:172.21.15.16:6809/3360352974,v1:172.21.15.16:6812/3360352974] conn(0x55d9b7f18000 0x55d9b8020580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:19.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.795+0000 7f65f1c00640 1 -- 172.21.15.16:0/4116959849 --> [v2:172.21.15.16:6809/3360352974,v1:172.21.15.16:6812/3360352974] -- osd_op(unknown.0.0:3 12.8 12:10000000::::head [pgnls start_epoch 85 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e85) -- 0x55d9b7cff400 con 0x55d9b7f18000 2024-03-31T15:11:19.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.795+0000 7f65f1200640 1 --2- 172.21.15.16:0/4116959849 >> [v2:172.21.15.16:6809/3360352974,v1:172.21.15.16:6812/3360352974] conn(0x55d9b7f18000 0x55d9b8020580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:19.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.795+0000 7f65f1200640 1 --2- 172.21.15.16:0/4116959849 >> [v2:172.21.15.16:6809/3360352974,v1:172.21.15.16:6812/3360352974] conn(0x55d9b7f18000 0x55d9b8020580 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:11:19.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.795+0000 7f65f1200640 1 -- 172.21.15.16:0/4116959849 <== osd.3 v2:172.21.15.16:6809/3360352974 1 ==== osd_op_reply(3 [pgnls start_epoch 85 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55d9b7d1e780 con 0x55d9b7f18000 2024-03-31T15:11:19.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.799+0000 7f65f1200640 1 --2- 172.21.15.16:0/4116959849 >> [v2:172.21.15.16:6808/1398737533,v1:172.21.15.16:6810/1398737533] conn(0x55d9b7cff400 0x55d9b8020b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:19.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.799+0000 7f65f1200640 1 -- 172.21.15.16:0/4116959849 --> [v2:172.21.15.16:6808/1398737533,v1:172.21.15.16:6810/1398737533] -- osd_op(unknown.0.0:4 12.18 12:18000000::::head [pgnls start_epoch 85 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e85) -- 0x55d9b801a400 con 0x55d9b7cff400 2024-03-31T15:11:19.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.799+0000 7f65f0800640 1 --2- 172.21.15.16:0/4116959849 >> [v2:172.21.15.16:6808/1398737533,v1:172.21.15.16:6810/1398737533] conn(0x55d9b7cff400 0x55d9b8020b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:19.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.799+0000 7f65f0800640 1 --2- 172.21.15.16:0/4116959849 >> [v2:172.21.15.16:6808/1398737533,v1:172.21.15.16:6810/1398737533] conn(0x55d9b7cff400 0x55d9b8020b00 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:11:19.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.799+0000 7f65f0800640 1 -- 172.21.15.16:0/4116959849 <== osd.1 v2:172.21.15.16:6808/1398737533 1 ==== osd_op_reply(4 [pgnls start_epoch 85 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55d9b7d1e000 con 0x55d9b7cff400 2024-03-31T15:11:19.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.799+0000 7f65f0800640 1 --2- 172.21.15.16:0/4116959849 >> [v2:172.21.15.89:6809/3470471752,v1:172.21.15.89:6812/3470471752] conn(0x55d9b801a400 0x55d9b8021080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:19.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.799+0000 7f65f0800640 1 -- 172.21.15.16:0/4116959849 --> [v2:172.21.15.89:6809/3470471752,v1:172.21.15.89:6812/3470471752] -- osd_op(unknown.0.0:5 12.4 12:20000000::::head [pgnls start_epoch 85 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e85) -- 0x55d9b7f18400 con 0x55d9b801a400 2024-03-31T15:11:19.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.799+0000 7f65f1c00640 1 --2- 172.21.15.16:0/4116959849 >> [v2:172.21.15.89:6809/3470471752,v1:172.21.15.89:6812/3470471752] conn(0x55d9b801a400 0x55d9b8021080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:19.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.799+0000 7f65f1c00640 1 --2- 172.21.15.16:0/4116959849 >> [v2:172.21.15.89:6809/3470471752,v1:172.21.15.89:6812/3470471752] conn(0x55d9b801a400 0x55d9b8021080 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:11:19.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.799+0000 7f65f1c00640 1 -- 172.21.15.16:0/4116959849 <== osd.6 v2:172.21.15.89:6809/3470471752 1 ==== osd_op_reply(5 [pgnls start_epoch 85 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55d9b7d1e500 con 0x55d9b801a400 2024-03-31T15:11:19.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.799+0000 7f65f1c00640 1 --2- 172.21.15.16:0/4116959849 >> [v2:172.21.15.89:6810/50395527,v1:172.21.15.89:6814/50395527] conn(0x55d9b7f18400 0x55d9b8021b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:19.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.799+0000 7f65f1c00640 1 -- 172.21.15.16:0/4116959849 --> [v2:172.21.15.89:6810/50395527,v1:172.21.15.89:6814/50395527] -- osd_op(unknown.0.0:6 12.14 12:28000000::::head [pgnls start_epoch 85 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e85) -- 0x55d9b801b000 con 0x55d9b7f18400 2024-03-31T15:11:19.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.799+0000 7f65f1200640 1 --2- 172.21.15.16:0/4116959849 >> [v2:172.21.15.89:6810/50395527,v1:172.21.15.89:6814/50395527] conn(0x55d9b7f18400 0x55d9b8021b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:19.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.799+0000 7f65f1200640 1 --2- 172.21.15.16:0/4116959849 >> [v2:172.21.15.89:6810/50395527,v1:172.21.15.89:6814/50395527] conn(0x55d9b7f18400 0x55d9b8021b80 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:11:19.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.799+0000 7f65f1200640 1 -- 172.21.15.16:0/4116959849 <== osd.5 v2:172.21.15.89:6810/50395527 1 ==== osd_op_reply(6 [pgnls start_epoch 85 out=149b] v84'27 uv27 ondisk = 1) ==== 144+0+149 (crc 0 0 0) 0x55d9b7d1e780 con 0x55d9b7f18400 2024-03-31T15:11:19.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.799+0000 7f65f1200640 1 -- 172.21.15.16:0/4116959849 --> [v2:172.21.15.16:6800/932712244,v1:172.21.15.16:6801/932712244] -- osd_op(unknown.0.0:7 12.c 12:30000000::::head [pgnls start_epoch 85 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e85) -- 0x55d9b801b000 con 0x55d9b7f18800 2024-03-31T15:11:19.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.799+0000 7f65f1c00640 1 -- 172.21.15.16:0/4116959849 <== osd.0 v2:172.21.15.16:6800/932712244 2 ==== osd_op_reply(7 [pgnls start_epoch 85 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55d9b7d1e500 con 0x55d9b7f18800 2024-03-31T15:11:19.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.799+0000 7f65f1c00640 1 -- 172.21.15.16:0/4116959849 --> [v2:172.21.15.16:6800/932712244,v1:172.21.15.16:6801/932712244] -- osd_op(unknown.0.0:8 12.1c 12:38000000::::head [pgnls start_epoch 85 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e85) -- 0x55d9b801b000 con 0x55d9b7f18800 2024-03-31T15:11:19.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.803+0000 7f65f1c00640 1 -- 172.21.15.16:0/4116959849 <== osd.0 v2:172.21.15.16:6800/932712244 3 ==== osd_op_reply(8 [pgnls start_epoch 85 out=73b] v84'3 uv3 ondisk = 1) ==== 144+0+73 (crc 0 0 0) 0x55d9b7d1e500 con 0x55d9b7f18800 2024-03-31T15:11:19.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.803+0000 7f65f1c00640 1 -- 172.21.15.16:0/4116959849 --> [v2:172.21.15.89:6800/1856844856,v1:172.21.15.89:6801/1856844856] -- osd_op(unknown.0.0:9 12.2 12:40000000::::head [pgnls start_epoch 85 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e85) -- 0x55d9b801b000 con 0x55d9b8b4fc00 2024-03-31T15:11:19.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.803+0000 7f65f0800640 1 -- 172.21.15.16:0/4116959849 <== osd.4 v2:172.21.15.89:6800/1856844856 2 ==== osd_op_reply(9 [pgnls start_epoch 85 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55d9b7d1e000 con 0x55d9b8b4fc00 2024-03-31T15:11:19.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.803+0000 7f65f0800640 1 -- 172.21.15.16:0/4116959849 --> [v2:172.21.15.16:6809/3360352974,v1:172.21.15.16:6812/3360352974] -- osd_op(unknown.0.0:10 12.12 12:48000000::::head [pgnls start_epoch 85 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e85) -- 0x55d9b801b000 con 0x55d9b7f18000 2024-03-31T15:11:19.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.803+0000 7f65f1200640 1 -- 172.21.15.16:0/4116959849 <== osd.3 v2:172.21.15.16:6809/3360352974 2 ==== osd_op_reply(10 [pgnls start_epoch 85 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55d9b7d1e780 con 0x55d9b7f18000 2024-03-31T15:11:19.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.803+0000 7f65f1200640 1 --2- 172.21.15.16:0/4116959849 >> [v2:172.21.15.16:6819/3018526646,v1:172.21.15.16:6822/3018526646] conn(0x55d9b801b000 0x55d9b8022680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:19.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.803+0000 7f65f1200640 1 -- 172.21.15.16:0/4116959849 --> [v2:172.21.15.16:6819/3018526646,v1:172.21.15.16:6822/3018526646] -- osd_op(unknown.0.0:11 12.a 12:50000000::::head [pgnls start_epoch 85 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e85) -- 0x55d9b801a800 con 0x55d9b801b000 2024-03-31T15:11:19.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.803+0000 7f65f0800640 1 --2- 172.21.15.16:0/4116959849 >> [v2:172.21.15.16:6819/3018526646,v1:172.21.15.16:6822/3018526646] conn(0x55d9b801b000 0x55d9b8022680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:19.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.803+0000 7f65f0800640 1 --2- 172.21.15.16:0/4116959849 >> [v2:172.21.15.16:6819/3018526646,v1:172.21.15.16:6822/3018526646] conn(0x55d9b801b000 0x55d9b8022680 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:11:19.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.803+0000 7f65f0800640 1 -- 172.21.15.16:0/4116959849 <== osd.2 v2:172.21.15.16:6819/3018526646 1 ==== osd_op_reply(11 [pgnls start_epoch 85 out=81b] v84'3 uv3 ondisk = 1) ==== 144+0+81 (crc 0 0 0) 0x55d9b7d1e000 con 0x55d9b801b000 2024-03-31T15:11:19.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.803+0000 7f65f0800640 1 -- 172.21.15.16:0/4116959849 --> [v2:172.21.15.89:6800/1856844856,v1:172.21.15.89:6801/1856844856] -- osd_op(unknown.0.0:12 12.1a 12:58000000::::head [pgnls start_epoch 85 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e85) -- 0x55d9b801a800 con 0x55d9b8b4fc00 2024-03-31T15:11:19.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.803+0000 7f65f0800640 1 -- 172.21.15.16:0/4116959849 <== osd.4 v2:172.21.15.89:6800/1856844856 3 ==== osd_op_reply(12 [pgnls start_epoch 85 out=49b] v82'3 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55d9b7d1e000 con 0x55d9b8b4fc00 2024-03-31T15:11:19.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.803+0000 7f65f0800640 1 -- 172.21.15.16:0/4116959849 --> [v2:172.21.15.16:6808/1398737533,v1:172.21.15.16:6810/1398737533] -- osd_op(unknown.0.0:13 12.6 12:60000000::::head [pgnls start_epoch 85 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e85) -- 0x55d9b801a800 con 0x55d9b7cff400 2024-03-31T15:11:19.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.803+0000 7f65f0800640 1 -- 172.21.15.16:0/4116959849 <== osd.1 v2:172.21.15.16:6808/1398737533 2 ==== osd_op_reply(13 [pgnls start_epoch 85 out=77b] v84'8 uv8 ondisk = 1) ==== 144+0+77 (crc 0 0 0) 0x55d9b7d1e000 con 0x55d9b7cff400 2024-03-31T15:11:19.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.803+0000 7f65f0800640 1 -- 172.21.15.16:0/4116959849 --> [v2:172.21.15.89:6809/3470471752,v1:172.21.15.89:6812/3470471752] -- osd_op(unknown.0.0:14 12.16 12:68000000::::head [pgnls start_epoch 85 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e85) -- 0x55d9b801a800 con 0x55d9b801a400 2024-03-31T15:11:19.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.803+0000 7f65f1c00640 1 -- 172.21.15.16:0/4116959849 <== osd.6 v2:172.21.15.89:6809/3470471752 2 ==== osd_op_reply(14 [pgnls start_epoch 85 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55d9b7d1e500 con 0x55d9b801a400 2024-03-31T15:11:19.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.803+0000 7f65f1c00640 1 -- 172.21.15.16:0/4116959849 --> [v2:172.21.15.16:6800/932712244,v1:172.21.15.16:6801/932712244] -- osd_op(unknown.0.0:15 12.e 12:70000000::::head [pgnls start_epoch 85 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e85) -- 0x55d9b801a800 con 0x55d9b7f18800 2024-03-31T15:11:19.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.803+0000 7f65f1c00640 1 -- 172.21.15.16:0/4116959849 <== osd.0 v2:172.21.15.16:6800/932712244 4 ==== osd_op_reply(15 [pgnls start_epoch 85 out=73b] v84'3 uv3 ondisk = 1) ==== 144+0+73 (crc 0 0 0) 0x55d9b7d1e500 con 0x55d9b7f18800 2024-03-31T15:11:19.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.803+0000 7f65f1c00640 1 -- 172.21.15.16:0/4116959849 --> [v2:172.21.15.89:6800/1856844856,v1:172.21.15.89:6801/1856844856] -- osd_op(unknown.0.0:16 12.1e 12:78000000::::head [pgnls start_epoch 85 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e85) -- 0x55d9b801a800 con 0x55d9b8b4fc00 2024-03-31T15:11:19.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.803+0000 7f65f0800640 1 -- 172.21.15.16:0/4116959849 <== osd.4 v2:172.21.15.89:6800/1856844856 4 ==== osd_op_reply(16 [pgnls start_epoch 85 out=49b] v82'3 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55d9b7d1e000 con 0x55d9b8b4fc00 2024-03-31T15:11:19.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.803+0000 7f65f0800640 1 -- 172.21.15.16:0/4116959849 --> [v2:172.21.15.89:6809/3470471752,v1:172.21.15.89:6812/3470471752] -- osd_op(unknown.0.0:17 12.1 12:80000000::::head [pgnls start_epoch 85 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e85) -- 0x55d9b801a800 con 0x55d9b801a400 2024-03-31T15:11:19.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.803+0000 7f65f1c00640 1 -- 172.21.15.16:0/4116959849 <== osd.6 v2:172.21.15.89:6809/3470471752 3 ==== osd_op_reply(17 [pgnls start_epoch 85 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55d9b7d1e500 con 0x55d9b801a400 2024-03-31T15:11:19.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.803+0000 7f65f1c00640 1 -- 172.21.15.16:0/4116959849 --> [v2:172.21.15.89:6810/50395527,v1:172.21.15.89:6814/50395527] -- osd_op(unknown.0.0:18 12.11 12:88000000::::head [pgnls start_epoch 85 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e85) -- 0x55d9b801a800 con 0x55d9b7f18400 2024-03-31T15:11:19.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.807+0000 7f65f1200640 1 -- 172.21.15.16:0/4116959849 <== osd.5 v2:172.21.15.89:6810/50395527 2 ==== osd_op_reply(18 [pgnls start_epoch 85 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55d9b7d1e780 con 0x55d9b7f18400 2024-03-31T15:11:19.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.807+0000 7f65f1200640 1 -- 172.21.15.16:0/4116959849 --> [v2:172.21.15.16:6819/3018526646,v1:172.21.15.16:6822/3018526646] -- osd_op(unknown.0.0:19 12.9 12:90000000::::head [pgnls start_epoch 85 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e85) -- 0x55d9b801a800 con 0x55d9b801b000 2024-03-31T15:11:19.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.807+0000 7f65f0800640 1 -- 172.21.15.16:0/4116959849 <== osd.2 v2:172.21.15.16:6819/3018526646 2 ==== osd_op_reply(19 [pgnls start_epoch 85 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55d9b7d1e000 con 0x55d9b801b000 2024-03-31T15:11:19.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.807+0000 7f65f0800640 1 -- 172.21.15.16:0/4116959849 --> [v2:172.21.15.16:6800/932712244,v1:172.21.15.16:6801/932712244] -- osd_op(unknown.0.0:20 12.19 12:98000000::::head [pgnls start_epoch 85 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e85) -- 0x55d9b801a800 con 0x55d9b7f18800 2024-03-31T15:11:19.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.807+0000 7f65f1c00640 1 -- 172.21.15.16:0/4116959849 <== osd.0 v2:172.21.15.16:6800/932712244 5 ==== osd_op_reply(20 [pgnls start_epoch 85 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55d9b7d1e500 con 0x55d9b7f18800 2024-03-31T15:11:19.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.807+0000 7f65f1c00640 1 -- 172.21.15.16:0/4116959849 --> [v2:172.21.15.89:6809/3470471752,v1:172.21.15.89:6812/3470471752] -- osd_op(unknown.0.0:21 12.5 12:a0000000::::head [pgnls start_epoch 85 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e85) -- 0x55d9b801a800 con 0x55d9b801a400 2024-03-31T15:11:19.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.807+0000 7f65f1c00640 1 -- 172.21.15.16:0/4116959849 <== osd.6 v2:172.21.15.89:6809/3470471752 4 ==== osd_op_reply(21 [pgnls start_epoch 85 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55d9b7d1e500 con 0x55d9b801a400 2024-03-31T15:11:19.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.807+0000 7f65f1c00640 1 -- 172.21.15.16:0/4116959849 --> [v2:172.21.15.89:6809/3470471752,v1:172.21.15.89:6812/3470471752] -- osd_op(unknown.0.0:22 12.15 12:a8000000::::head [pgnls start_epoch 85 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e85) -- 0x55d9b801a800 con 0x55d9b801a400 2024-03-31T15:11:19.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.807+0000 7f65f1c00640 1 -- 172.21.15.16:0/4116959849 <== osd.6 v2:172.21.15.89:6809/3470471752 5 ==== osd_op_reply(22 [pgnls start_epoch 85 out=97b] v84'4 uv4 ondisk = 1) ==== 144+0+97 (crc 0 0 0) 0x55d9b7d1e500 con 0x55d9b801a400 2024-03-31T15:11:19.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.807+0000 7f65f1c00640 1 --2- 172.21.15.16:0/4116959849 >> [v2:172.21.15.89:6808/1796778630,v1:172.21.15.89:6811/1796778630] conn(0x55d9b801a800 0x55d9b8022100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:19.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.807+0000 7f65f1c00640 1 -- 172.21.15.16:0/4116959849 --> [v2:172.21.15.89:6808/1796778630,v1:172.21.15.89:6811/1796778630] -- osd_op(unknown.0.0:23 12.d 12:b0000000::::head [pgnls start_epoch 85 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e85) -- 0x55d9b801b400 con 0x55d9b801a800 2024-03-31T15:11:19.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.807+0000 7f65f1c00640 1 --2- 172.21.15.16:0/4116959849 >> [v2:172.21.15.89:6808/1796778630,v1:172.21.15.89:6811/1796778630] conn(0x55d9b801a800 0x55d9b8022100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:19.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.807+0000 7f65f1c00640 1 --2- 172.21.15.16:0/4116959849 >> [v2:172.21.15.89:6808/1796778630,v1:172.21.15.89:6811/1796778630] conn(0x55d9b801a800 0x55d9b8022100 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:11:19.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.807+0000 7f65f1c00640 1 -- 172.21.15.16:0/4116959849 <== osd.7 v2:172.21.15.89:6808/1796778630 1 ==== osd_op_reply(23 [pgnls start_epoch 85 out=74b] v80'1 uv1 ondisk = 1) ==== 144+0+74 (crc 0 0 0) 0x55d9b7d1e500 con 0x55d9b801a800 2024-03-31T15:11:19.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.807+0000 7f65f1c00640 1 -- 172.21.15.16:0/4116959849 --> [v2:172.21.15.16:6800/932712244,v1:172.21.15.16:6801/932712244] -- osd_op(unknown.0.0:24 12.1d 12:b8000000::::head [pgnls start_epoch 85 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e85) -- 0x55d9b801b400 con 0x55d9b7f18800 2024-03-31T15:11:19.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.807+0000 7f65f1c00640 1 -- 172.21.15.16:0/4116959849 <== osd.0 v2:172.21.15.16:6800/932712244 6 ==== osd_op_reply(24 [pgnls start_epoch 85 out=73b] v84'5 uv5 ondisk = 1) ==== 144+0+73 (crc 0 0 0) 0x55d9b7d1e500 con 0x55d9b7f18800 2024-03-31T15:11:19.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.807+0000 7f65f1c00640 1 -- 172.21.15.16:0/4116959849 --> [v2:172.21.15.16:6800/932712244,v1:172.21.15.16:6801/932712244] -- osd_op(unknown.0.0:25 12.3 12:c0000000::::head [pgnls start_epoch 85 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e85) -- 0x55d9b801b400 con 0x55d9b7f18800 2024-03-31T15:11:19.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.807+0000 7f65f1c00640 1 -- 172.21.15.16:0/4116959849 <== osd.0 v2:172.21.15.16:6800/932712244 7 ==== osd_op_reply(25 [pgnls start_epoch 85 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55d9b7d1e500 con 0x55d9b7f18800 2024-03-31T15:11:19.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.807+0000 7f65f1c00640 1 -- 172.21.15.16:0/4116959849 --> [v2:172.21.15.16:6808/1398737533,v1:172.21.15.16:6810/1398737533] -- osd_op(unknown.0.0:26 12.13 12:c8000000::::head [pgnls start_epoch 85 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e85) -- 0x55d9b801b400 con 0x55d9b7cff400 2024-03-31T15:11:19.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.807+0000 7f65f0800640 1 -- 172.21.15.16:0/4116959849 <== osd.1 v2:172.21.15.16:6808/1398737533 3 ==== osd_op_reply(26 [pgnls start_epoch 85 out=98b] v84'4 uv4 ondisk = 1) ==== 144+0+98 (crc 0 0 0) 0x55d9b7d1e000 con 0x55d9b7cff400 2024-03-31T15:11:19.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.807+0000 7f65f0800640 1 -- 172.21.15.16:0/4116959849 --> [v2:172.21.15.16:6809/3360352974,v1:172.21.15.16:6812/3360352974] -- osd_op(unknown.0.0:27 12.b 12:d0000000::::head [pgnls start_epoch 85 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e85) -- 0x55d9b801b400 con 0x55d9b7f18000 2024-03-31T15:11:19.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.807+0000 7f65f1200640 1 -- 172.21.15.16:0/4116959849 <== osd.3 v2:172.21.15.16:6809/3360352974 3 ==== osd_op_reply(27 [pgnls start_epoch 85 out=76b] v84'6 uv6 ondisk = 1) ==== 144+0+76 (crc 0 0 0) 0x55d9b7d1e780 con 0x55d9b7f18000 2024-03-31T15:11:19.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.807+0000 7f65f1200640 1 -- 172.21.15.16:0/4116959849 --> [v2:172.21.15.89:6800/1856844856,v1:172.21.15.89:6801/1856844856] -- osd_op(unknown.0.0:28 12.1b 12:d8000000::::head [pgnls start_epoch 85 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e85) -- 0x55d9b801b400 con 0x55d9b8b4fc00 2024-03-31T15:11:19.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.811+0000 7f65f0800640 1 -- 172.21.15.16:0/4116959849 <== osd.4 v2:172.21.15.89:6800/1856844856 5 ==== osd_op_reply(28 [pgnls start_epoch 85 out=73b] v84'10 uv10 ondisk = 1) ==== 144+0+73 (crc 0 0 0) 0x55d9b7d1e000 con 0x55d9b8b4fc00 2024-03-31T15:11:19.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.811+0000 7f65f0800640 1 -- 172.21.15.16:0/4116959849 --> [v2:172.21.15.16:6808/1398737533,v1:172.21.15.16:6810/1398737533] -- osd_op(unknown.0.0:29 12.7 12:e0000000::::head [pgnls start_epoch 85 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e85) -- 0x55d9b801b400 con 0x55d9b7cff400 2024-03-31T15:11:19.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.811+0000 7f65f0800640 1 -- 172.21.15.16:0/4116959849 <== osd.1 v2:172.21.15.16:6808/1398737533 4 ==== osd_op_reply(29 [pgnls start_epoch 85 out=73b] v84'3 uv3 ondisk = 1) ==== 144+0+73 (crc 0 0 0) 0x55d9b7d1e000 con 0x55d9b7cff400 2024-03-31T15:11:19.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.811+0000 7f65f0800640 1 -- 172.21.15.16:0/4116959849 --> [v2:172.21.15.89:6809/3470471752,v1:172.21.15.89:6812/3470471752] -- osd_op(unknown.0.0:30 12.17 12:e8000000::::head [pgnls start_epoch 85 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e85) -- 0x55d9b801b400 con 0x55d9b801a400 2024-03-31T15:11:19.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.811+0000 7f65f1c00640 1 -- 172.21.15.16:0/4116959849 <== osd.6 v2:172.21.15.89:6809/3470471752 6 ==== osd_op_reply(30 [pgnls start_epoch 85 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55d9b7d1e500 con 0x55d9b801a400 2024-03-31T15:11:19.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.811+0000 7f65f1c00640 1 -- 172.21.15.16:0/4116959849 --> [v2:172.21.15.89:6809/3470471752,v1:172.21.15.89:6812/3470471752] -- osd_op(unknown.0.0:31 12.f 12:f0000000::::head [pgnls start_epoch 85 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e85) -- 0x55d9b801b400 con 0x55d9b801a400 2024-03-31T15:11:19.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.811+0000 7f65f1c00640 1 -- 172.21.15.16:0/4116959849 <== osd.6 v2:172.21.15.89:6809/3470471752 7 ==== osd_op_reply(31 [pgnls start_epoch 85 out=73b] v80'1 uv1 ondisk = 1) ==== 144+0+73 (crc 0 0 0) 0x55d9b7d1e500 con 0x55d9b801a400 2024-03-31T15:11:19.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.811+0000 7f65f1c00640 1 -- 172.21.15.16:0/4116959849 --> [v2:172.21.15.89:6809/3470471752,v1:172.21.15.89:6812/3470471752] -- osd_op(unknown.0.0:32 12.1f 12:f8000000::::head [pgnls start_epoch 85 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e85) -- 0x55d9b801b400 con 0x55d9b801a400 2024-03-31T15:11:19.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.811+0000 7f65f1c00640 1 -- 172.21.15.16:0/4116959849 <== osd.6 v2:172.21.15.89:6809/3470471752 8 ==== osd_op_reply(32 [pgnls start_epoch 85 out=101b] v84'9 uv9 ondisk = 1) ==== 144+0+101 (crc 0 0 0) 0x55d9b7d1e500 con 0x55d9b801a400 2024-03-31T15:11:19.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.811+0000 7f65f31ca8c0 10 datascan.scan_links: step 1: handling object 601.00000000 2024-03-31T15:11:19.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.811+0000 7f65f31ca8c0 1 -- 172.21.15.16:0/4116959849 --> [v2:172.21.15.16:6800/932712244,v1:172.21.15.16:6801/932712244] -- osd_op(unknown.0.0:33 12.10 12:09401a81:::601.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e85) -- 0x55d9b7f19400 con 0x55d9b7f18800 2024-03-31T15:11:19.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.811+0000 7f65f1c00640 1 -- 172.21.15.16:0/4116959849 <== osd.0 v2:172.21.15.16:6800/932712244 8 ==== osd_op_reply(33 601.00000000 [omap-get-vals out=5b] v0'0 uv13 ondisk = 0) ==== 156+0+5 (crc 0 0 0) 0x55d9b7d1e500 con 0x55d9b7f18800 2024-03-31T15:11:19.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.811+0000 7f65f31ca8c0 10 datascan.scan_links: step 1: handling object 602.00000000 2024-03-31T15:11:19.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.811+0000 7f65f31ca8c0 1 -- 172.21.15.16:0/4116959849 --> [v2:172.21.15.16:6800/932712244,v1:172.21.15.16:6801/932712244] -- osd_op(unknown.0.0:34 12.10 12:0bd6d154:::602.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e85) -- 0x55d9b7f19800 con 0x55d9b7f18800 2024-03-31T15:11:19.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.811+0000 7f65f1c00640 1 -- 172.21.15.16:0/4116959849 <== osd.0 v2:172.21.15.16:6800/932712244 9 ==== osd_op_reply(34 602.00000000 [omap-get-vals out=5b] v0'0 uv14 ondisk = 0) ==== 156+0+5 (crc 0 0 0) 0x55d9b7d1e500 con 0x55d9b7f18800 2024-03-31T15:11:19.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.811+0000 7f65f31ca8c0 10 datascan.scan_links: step 1: handling object 10000000000.00000000 2024-03-31T15:11:19.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.811+0000 7f65f31ca8c0 1 -- 172.21.15.16:0/4116959849 --> [v2:172.21.15.16:6800/932712244,v1:172.21.15.16:6801/932712244] -- osd_op(unknown.0.0:35 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e85) -- 0x55d9b7f19c00 con 0x55d9b7f18800 2024-03-31T15:11:19.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.811+0000 7f65f1c00640 1 -- 172.21.15.16:0/4116959849 <== osd.0 v2:172.21.15.16:6800/932712244 10 ==== osd_op_reply(35 10000000000.00000000 [omap-get-vals out=506b] v0'0 uv18 ondisk = 0) ==== 164+0+506 (crc 0 0 0) 0x55d9b7d1e500 con 0x55d9b7f18800 2024-03-31T15:11:19.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.811+0000 7f65f31ca8c0 20 datascan.scan_links: injected ino detected 2024-03-31T15:11:19.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.811+0000 7f65f31ca8c0 10 datascan.scan_links: step 1: handling object 600.00000000 2024-03-31T15:11:19.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.811+0000 7f65f31ca8c0 1 -- 172.21.15.16:0/4116959849 --> [v2:172.21.15.16:6800/932712244,v1:172.21.15.16:6801/932712244] -- osd_op(unknown.0.0:36 12.10 12:0d82a743:::600.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e85) -- 0x55d9b7f19400 con 0x55d9b7f18800 2024-03-31T15:11:19.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.811+0000 7f65f1c00640 1 -- 172.21.15.16:0/4116959849 <== osd.0 v2:172.21.15.16:6800/932712244 11 ==== osd_op_reply(36 600.00000000 [omap-get-vals out=5b] v0'0 uv12 ondisk = 0) ==== 156+0+5 (crc 0 0 0) 0x55d9b7d1e500 con 0x55d9b7f18800 2024-03-31T15:11:19.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.811+0000 7f65f31ca8c0 10 datascan.scan_links: step 1: handling object 603.00000000 2024-03-31T15:11:19.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.811+0000 7f65f31ca8c0 1 -- 172.21.15.16:0/4116959849 --> [v2:172.21.15.16:6800/932712244,v1:172.21.15.16:6801/932712244] -- osd_op(unknown.0.0:37 12.10 12:0d89b25e:::603.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e85) -- 0x55d9b7f19800 con 0x55d9b7f18800 2024-03-31T15:11:19.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.811+0000 7f65f1c00640 1 -- 172.21.15.16:0/4116959849 <== osd.0 v2:172.21.15.16:6800/932712244 12 ==== osd_op_reply(37 603.00000000 [omap-get-vals out=5b] v0'0 uv15 ondisk = 0) ==== 156+0+5 (crc 0 0 0) 0x55d9b7d1e500 con 0x55d9b7f18800 2024-03-31T15:11:19.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.811+0000 7f65f31ca8c0 10 datascan.scan_links: step 1: handling object 1.00000000.inode 2024-03-31T15:11:19.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.811+0000 7f65f31ca8c0 10 datascan.scan_links: Not a dirfrag: '1.00000000.inode' 2024-03-31T15:11:19.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.811+0000 7f65f31ca8c0 10 datascan.scan_links: step 1: handling object 200.00000000 2024-03-31T15:11:19.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.811+0000 7f65f31ca8c0 10 datascan.scan_links: Not a dirfrag (invalid ino): '200.00000000' 2024-03-31T15:11:19.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.811+0000 7f65f31ca8c0 10 datascan.scan_links: step 1: handling object 200.00000001 2024-03-31T15:11:19.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.811+0000 7f65f31ca8c0 10 datascan.scan_links: Not a dirfrag (invalid ino): '200.00000001' 2024-03-31T15:11:19.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.811+0000 7f65f31ca8c0 10 datascan.scan_links: step 1: handling object 606.00000000 2024-03-31T15:11:19.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.811+0000 7f65f31ca8c0 1 -- 172.21.15.16:0/4116959849 --> [v2:172.21.15.89:6810/50395527,v1:172.21.15.89:6814/50395527] -- osd_op(unknown.0.0:38 12.14 12:2f55791f:::606.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e85) -- 0x55d9b7f19c00 con 0x55d9b7f18400 2024-03-31T15:11:19.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.811+0000 7f65f1200640 1 -- 172.21.15.16:0/4116959849 <== osd.5 v2:172.21.15.89:6810/50395527 3 ==== osd_op_reply(38 606.00000000 [omap-get-vals out=5b] v0'0 uv25 ondisk = 0) ==== 156+0+5 (crc 0 0 0) 0x55d9b7d1e780 con 0x55d9b7f18400 2024-03-31T15:11:19.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.811+0000 7f65f31ca8c0 10 datascan.scan_links: step 1: handling object 607.00000000 2024-03-31T15:11:19.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.811+0000 7f65f31ca8c0 1 -- 172.21.15.16:0/4116959849 --> [v2:172.21.15.16:6800/932712244,v1:172.21.15.16:6801/932712244] -- osd_op(unknown.0.0:39 12.1c 12:3ed09add:::607.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e85) -- 0x55d9b801bc00 con 0x55d9b7f18800 2024-03-31T15:11:19.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.811+0000 7f65f1c00640 1 -- 172.21.15.16:0/4116959849 <== osd.0 v2:172.21.15.16:6800/932712244 13 ==== osd_op_reply(39 607.00000000 [omap-get-vals out=5b] v0'0 uv3 ondisk = 0) ==== 156+0+5 (crc 0 0 0) 0x55d9b7d1e500 con 0x55d9b7f18800 2024-03-31T15:11:19.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.811+0000 7f65f31ca8c0 10 datascan.scan_links: step 1: handling object 100000001f6.00000000 2024-03-31T15:11:19.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.811+0000 7f65f31ca8c0 1 -- 172.21.15.16:0/4116959849 --> [v2:172.21.15.16:6819/3018526646,v1:172.21.15.16:6822/3018526646] -- osd_op(unknown.0.0:40 12.a 12:57fe7535:::100000001f6.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e85) -- 0x55d9b801b800 con 0x55d9b801b000 2024-03-31T15:11:19.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.815+0000 7f65f0800640 1 -- 172.21.15.16:0/4116959849 <== osd.2 v2:172.21.15.16:6819/3018526646 3 ==== osd_op_reply(40 100000001f6.00000000 [omap-get-vals out=54b] v0'0 uv3 ondisk = 0) ==== 164+0+54 (crc 0 0 0) 0x55d9b7d1e000 con 0x55d9b801b000 2024-03-31T15:11:19.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.815+0000 7f65f31ca8c0 10 datascan.scan_links: step 1: handling object mds0_openfiles.0 2024-03-31T15:11:19.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.815+0000 7f65f31ca8c0 10 datascan.scan_links: Not a dirfrag: 'mds0_openfiles.0' 2024-03-31T15:11:19.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.815+0000 7f65f31ca8c0 10 datascan.scan_links: step 1: handling object 608.00000000 2024-03-31T15:11:19.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.815+0000 7f65f31ca8c0 1 -- 172.21.15.16:0/4116959849 --> [v2:172.21.15.16:6800/932712244,v1:172.21.15.16:6801/932712244] -- osd_op(unknown.0.0:41 12.e 12:717a0223:::608.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e85) -- 0x55d9b801b400 con 0x55d9b7f18800 2024-03-31T15:11:19.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.815+0000 7f65f1c00640 1 -- 172.21.15.16:0/4116959849 <== osd.0 v2:172.21.15.16:6800/932712244 14 ==== osd_op_reply(41 608.00000000 [omap-get-vals out=5b] v0'0 uv3 ondisk = 0) ==== 156+0+5 (crc 0 0 0) 0x55d9b7d1e500 con 0x55d9b7f18800 2024-03-31T15:11:19.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.815+0000 7f65f31ca8c0 10 datascan.scan_links: step 1: handling object 604.00000000 2024-03-31T15:11:19.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.815+0000 7f65f31ca8c0 1 -- 172.21.15.16:0/4116959849 --> [v2:172.21.15.89:6809/3470471752,v1:172.21.15.89:6812/3470471752] -- osd_op(unknown.0.0:42 12.15 12:a93a17c2:::604.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e85) -- 0x55d9b8044000 con 0x55d9b801a400 2024-03-31T15:11:19.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.815+0000 7f65f1c00640 1 -- 172.21.15.16:0/4116959849 <== osd.6 v2:172.21.15.89:6809/3470471752 9 ==== osd_op_reply(42 604.00000000 [omap-get-vals out=5b] v0'0 uv4 ondisk = 0) ==== 156+0+5 (crc 0 0 0) 0x55d9b7d1e500 con 0x55d9b801a400 2024-03-31T15:11:19.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.815+0000 7f65f31ca8c0 10 datascan.scan_links: step 1: handling object 500.00000000 2024-03-31T15:11:19.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.815+0000 7f65f31ca8c0 10 datascan.scan_links: Not a dirfrag (invalid ino): '500.00000000' 2024-03-31T15:11:19.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.815+0000 7f65f31ca8c0 10 datascan.scan_links: step 1: handling object mds_snaptable 2024-03-31T15:11:19.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.815+0000 7f65f31ca8c0 10 datascan.scan_links: Not a dirfrag: 'mds_snaptable' 2024-03-31T15:11:19.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.815+0000 7f65f31ca8c0 10 datascan.scan_links: step 1: handling object 605.00000000 2024-03-31T15:11:19.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.815+0000 7f65f31ca8c0 1 -- 172.21.15.16:0/4116959849 --> [v2:172.21.15.16:6800/932712244,v1:172.21.15.16:6801/932712244] -- osd_op(unknown.0.0:43 12.1d 12:b871830b:::605.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e85) -- 0x55d9b8044400 con 0x55d9b7f18800 2024-03-31T15:11:19.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.815+0000 7f65f1c00640 1 -- 172.21.15.16:0/4116959849 <== osd.0 v2:172.21.15.16:6800/932712244 15 ==== osd_op_reply(43 605.00000000 [omap-get-vals out=5b] v0'0 uv5 ondisk = 0) ==== 156+0+5 (crc 0 0 0) 0x55d9b7d1e500 con 0x55d9b7f18800 2024-03-31T15:11:19.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.815+0000 7f65f31ca8c0 10 datascan.scan_links: step 1: handling object mds0_inotable 2024-03-31T15:11:19.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.815+0000 7f65f31ca8c0 10 datascan.scan_links: Not a dirfrag: 'mds0_inotable' 2024-03-31T15:11:19.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.815+0000 7f65f31ca8c0 10 datascan.scan_links: step 1: handling object 100.00000000 2024-03-31T15:11:19.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.815+0000 7f65f31ca8c0 1 -- 172.21.15.16:0/4116959849 --> [v2:172.21.15.16:6808/1398737533,v1:172.21.15.16:6810/1398737533] -- osd_op(unknown.0.0:44 12.13 12:cd5a64a3:::100.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e85) -- 0x55d9b8044800 con 0x55d9b7cff400 2024-03-31T15:11:19.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.815+0000 7f65f0800640 1 -- 172.21.15.16:0/4116959849 <== osd.1 v2:172.21.15.16:6808/1398737533 5 ==== osd_op_reply(44 100.00000000 [omap-get-vals out=5195b] v0'0 uv4 ondisk = 0) ==== 156+0+5195 (crc 0 0 0) 0x55d9b7d1e000 con 0x55d9b7cff400 2024-03-31T15:11:19.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.815+0000 7f65f31ca8c0 10 datascan.scan_links: step 1: handling object mds0_sessionmap 2024-03-31T15:11:19.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.815+0000 7f65f31ca8c0 10 datascan.scan_links: Not a dirfrag: 'mds0_sessionmap' 2024-03-31T15:11:19.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.815+0000 7f65f31ca8c0 10 datascan.scan_links: step 1: handling object 200.00000002 2024-03-31T15:11:19.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.815+0000 7f65f31ca8c0 10 datascan.scan_links: Not a dirfrag (invalid ino): '200.00000002' 2024-03-31T15:11:19.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.815+0000 7f65f31ca8c0 10 datascan.scan_links: step 1: handling object 609.00000000 2024-03-31T15:11:19.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.815+0000 7f65f31ca8c0 1 -- 172.21.15.16:0/4116959849 --> [v2:172.21.15.16:6808/1398737533,v1:172.21.15.16:6810/1398737533] -- osd_op(unknown.0.0:45 12.7 12:e0b41d06:::609.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e85) -- 0x55d9b8044c00 con 0x55d9b7cff400 2024-03-31T15:11:19.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.815+0000 7f65f0800640 1 -- 172.21.15.16:0/4116959849 <== osd.1 v2:172.21.15.16:6808/1398737533 6 ==== osd_op_reply(45 609.00000000 [omap-get-vals out=5b] v0'0 uv3 ondisk = 0) ==== 156+0+5 (crc 0 0 0) 0x55d9b7d1e000 con 0x55d9b7cff400 2024-03-31T15:11:19.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.815+0000 7f65f31ca8c0 10 datascan.scan_links: step 1: handling object 400.00000000 2024-03-31T15:11:19.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.815+0000 7f65f31ca8c0 10 datascan.scan_links: Not a dirfrag (invalid ino): '400.00000000' 2024-03-31T15:11:19.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.815+0000 7f65f31ca8c0 10 datascan.scan_links: step 1: handling object 100.00000000.inode 2024-03-31T15:11:19.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.815+0000 7f65f31ca8c0 10 datascan.scan_links: Not a dirfrag: '100.00000000.inode' 2024-03-31T15:11:19.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.815+0000 7f65f31ca8c0 10 datascan.scan_links: step 1: handling object 1.00000000 2024-03-31T15:11:19.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.815+0000 7f65f31ca8c0 1 -- 172.21.15.16:0/4116959849 --> [v2:172.21.15.89:6809/3470471752,v1:172.21.15.89:6812/3470471752] -- osd_op(unknown.0.0:46 12.1f 12:ff5b34d6:::1.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e85) -- 0x55d9b8045000 con 0x55d9b801a400 2024-03-31T15:11:19.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.815+0000 7f65f1c00640 1 -- 172.21.15.16:0/4116959849 <== osd.6 v2:172.21.15.89:6809/3470471752 10 ==== osd_op_reply(46 1.00000000 [omap-get-vals out=1041b] v0'0 uv9 ondisk = 0) ==== 154+0+1041 (crc 0 0 0) 0x55d9b7d1e500 con 0x55d9b801a400 2024-03-31T15:11:19.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.815+0000 7f65f31ca8c0 1 -- 172.21.15.16:0/4116959849 --> [v2:172.21.15.89:6800/1856844856,v1:172.21.15.89:6801/1856844856] -- osd_op(unknown.0.0:47 12.0 12:00000000::::head [pgnls start_epoch 85 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e85) -- 0x55d9b8045400 con 0x55d9b8b4fc00 2024-03-31T15:11:19.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.815+0000 7f65f0800640 1 -- 172.21.15.16:0/4116959849 <== osd.4 v2:172.21.15.89:6800/1856844856 6 ==== osd_op_reply(47 [pgnls start_epoch 85 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55d9b7d1e000 con 0x55d9b8b4fc00 2024-03-31T15:11:19.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.815+0000 7f65f0800640 1 -- 172.21.15.16:0/4116959849 --> [v2:172.21.15.16:6800/932712244,v1:172.21.15.16:6801/932712244] -- osd_op(unknown.0.0:48 12.10 12:08000000::::head [pgnls start_epoch 85 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e85) -- 0x55d9b8045400 con 0x55d9b7f18800 2024-03-31T15:11:19.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.815+0000 7f65f1c00640 1 -- 172.21.15.16:0/4116959849 <== osd.0 v2:172.21.15.16:6800/932712244 16 ==== osd_op_reply(48 [pgnls start_epoch 85 out=177b] v85'18 uv18 ondisk = 1) ==== 144+0+177 (crc 0 0 0) 0x55d9b7d1e500 con 0x55d9b7f18800 2024-03-31T15:11:19.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.815+0000 7f65f1c00640 1 -- 172.21.15.16:0/4116959849 --> [v2:172.21.15.16:6809/3360352974,v1:172.21.15.16:6812/3360352974] -- osd_op(unknown.0.0:49 12.8 12:10000000::::head [pgnls start_epoch 85 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e85) -- 0x55d9b8045400 con 0x55d9b7f18000 2024-03-31T15:11:19.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.815+0000 7f65f1200640 1 -- 172.21.15.16:0/4116959849 <== osd.3 v2:172.21.15.16:6809/3360352974 4 ==== osd_op_reply(49 [pgnls start_epoch 85 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55d9b7d1e780 con 0x55d9b7f18000 2024-03-31T15:11:19.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.815+0000 7f65f1200640 1 -- 172.21.15.16:0/4116959849 --> [v2:172.21.15.16:6808/1398737533,v1:172.21.15.16:6810/1398737533] -- osd_op(unknown.0.0:50 12.18 12:18000000::::head [pgnls start_epoch 85 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e85) -- 0x55d9b8045400 con 0x55d9b7cff400 2024-03-31T15:11:19.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.815+0000 7f65f0800640 1 -- 172.21.15.16:0/4116959849 <== osd.1 v2:172.21.15.16:6808/1398737533 7 ==== osd_op_reply(50 [pgnls start_epoch 85 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55d9b7d1e000 con 0x55d9b7cff400 2024-03-31T15:11:19.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.815+0000 7f65f0800640 1 -- 172.21.15.16:0/4116959849 --> [v2:172.21.15.89:6809/3470471752,v1:172.21.15.89:6812/3470471752] -- osd_op(unknown.0.0:51 12.4 12:20000000::::head [pgnls start_epoch 85 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e85) -- 0x55d9b8045400 con 0x55d9b801a400 2024-03-31T15:11:19.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.819+0000 7f65f1c00640 1 -- 172.21.15.16:0/4116959849 <== osd.6 v2:172.21.15.89:6809/3470471752 11 ==== osd_op_reply(51 [pgnls start_epoch 85 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55d9b7d1e500 con 0x55d9b801a400 2024-03-31T15:11:19.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.819+0000 7f65f1c00640 1 -- 172.21.15.16:0/4116959849 --> [v2:172.21.15.89:6810/50395527,v1:172.21.15.89:6814/50395527] -- osd_op(unknown.0.0:52 12.14 12:28000000::::head [pgnls start_epoch 85 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e85) -- 0x55d9b8045400 con 0x55d9b7f18400 2024-03-31T15:11:19.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.819+0000 7f65f1200640 1 -- 172.21.15.16:0/4116959849 <== osd.5 v2:172.21.15.89:6810/50395527 4 ==== osd_op_reply(52 [pgnls start_epoch 85 out=149b] v84'27 uv27 ondisk = 1) ==== 144+0+149 (crc 0 0 0) 0x55d9b7d1e780 con 0x55d9b7f18400 2024-03-31T15:11:19.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.819+0000 7f65f1200640 1 -- 172.21.15.16:0/4116959849 --> [v2:172.21.15.16:6800/932712244,v1:172.21.15.16:6801/932712244] -- osd_op(unknown.0.0:53 12.c 12:30000000::::head [pgnls start_epoch 85 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e85) -- 0x55d9b8045400 con 0x55d9b7f18800 2024-03-31T15:11:19.818 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.819+0000 7f65f1c00640 1 -- 172.21.15.16:0/4116959849 <== osd.0 v2:172.21.15.16:6800/932712244 17 ==== osd_op_reply(53 [pgnls start_epoch 85 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55d9b7d1e500 con 0x55d9b7f18800 2024-03-31T15:11:19.818 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.819+0000 7f65f1c00640 1 -- 172.21.15.16:0/4116959849 --> [v2:172.21.15.16:6800/932712244,v1:172.21.15.16:6801/932712244] -- osd_op(unknown.0.0:54 12.1c 12:38000000::::head [pgnls start_epoch 85 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e85) -- 0x55d9b8045400 con 0x55d9b7f18800 2024-03-31T15:11:19.818 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.819+0000 7f65f1c00640 1 -- 172.21.15.16:0/4116959849 <== osd.0 v2:172.21.15.16:6800/932712244 18 ==== osd_op_reply(54 [pgnls start_epoch 85 out=73b] v84'3 uv3 ondisk = 1) ==== 144+0+73 (crc 0 0 0) 0x55d9b7d1e500 con 0x55d9b7f18800 2024-03-31T15:11:19.818 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.819+0000 7f65f1c00640 1 -- 172.21.15.16:0/4116959849 --> [v2:172.21.15.89:6800/1856844856,v1:172.21.15.89:6801/1856844856] -- osd_op(unknown.0.0:55 12.2 12:40000000::::head [pgnls start_epoch 85 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e85) -- 0x55d9b8045400 con 0x55d9b8b4fc00 2024-03-31T15:11:19.818 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.819+0000 7f65f0800640 1 -- 172.21.15.16:0/4116959849 <== osd.4 v2:172.21.15.89:6800/1856844856 7 ==== osd_op_reply(55 [pgnls start_epoch 85 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55d9b7d1e000 con 0x55d9b8b4fc00 2024-03-31T15:11:19.818 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.819+0000 7f65f0800640 1 -- 172.21.15.16:0/4116959849 --> [v2:172.21.15.16:6809/3360352974,v1:172.21.15.16:6812/3360352974] -- osd_op(unknown.0.0:56 12.12 12:48000000::::head [pgnls start_epoch 85 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e85) -- 0x55d9b8045400 con 0x55d9b7f18000 2024-03-31T15:11:19.818 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.819+0000 7f65f1200640 1 -- 172.21.15.16:0/4116959849 <== osd.3 v2:172.21.15.16:6809/3360352974 5 ==== osd_op_reply(56 [pgnls start_epoch 85 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55d9b7d1e780 con 0x55d9b7f18000 2024-03-31T15:11:19.818 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.819+0000 7f65f1200640 1 -- 172.21.15.16:0/4116959849 --> [v2:172.21.15.16:6819/3018526646,v1:172.21.15.16:6822/3018526646] -- osd_op(unknown.0.0:57 12.a 12:50000000::::head [pgnls start_epoch 85 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e85) -- 0x55d9b8045400 con 0x55d9b801b000 2024-03-31T15:11:19.818 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.819+0000 7f65f0800640 1 -- 172.21.15.16:0/4116959849 <== osd.2 v2:172.21.15.16:6819/3018526646 4 ==== osd_op_reply(57 [pgnls start_epoch 85 out=81b] v84'3 uv3 ondisk = 1) ==== 144+0+81 (crc 0 0 0) 0x55d9b7d1e000 con 0x55d9b801b000 2024-03-31T15:11:19.818 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.819+0000 7f65f0800640 1 -- 172.21.15.16:0/4116959849 --> [v2:172.21.15.89:6800/1856844856,v1:172.21.15.89:6801/1856844856] -- osd_op(unknown.0.0:58 12.1a 12:58000000::::head [pgnls start_epoch 85 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e85) -- 0x55d9b8045400 con 0x55d9b8b4fc00 2024-03-31T15:11:19.818 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.819+0000 7f65f0800640 1 -- 172.21.15.16:0/4116959849 <== osd.4 v2:172.21.15.89:6800/1856844856 8 ==== osd_op_reply(58 [pgnls start_epoch 85 out=49b] v82'3 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55d9b7d1e000 con 0x55d9b8b4fc00 2024-03-31T15:11:19.818 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.819+0000 7f65f0800640 1 -- 172.21.15.16:0/4116959849 --> [v2:172.21.15.16:6808/1398737533,v1:172.21.15.16:6810/1398737533] -- osd_op(unknown.0.0:59 12.6 12:60000000::::head [pgnls start_epoch 85 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e85) -- 0x55d9b8045400 con 0x55d9b7cff400 2024-03-31T15:11:19.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.819+0000 7f65f0800640 1 -- 172.21.15.16:0/4116959849 <== osd.1 v2:172.21.15.16:6808/1398737533 8 ==== osd_op_reply(59 [pgnls start_epoch 85 out=77b] v84'8 uv8 ondisk = 1) ==== 144+0+77 (crc 0 0 0) 0x55d9b7d1e000 con 0x55d9b7cff400 2024-03-31T15:11:19.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.819+0000 7f65f0800640 1 -- 172.21.15.16:0/4116959849 --> [v2:172.21.15.89:6809/3470471752,v1:172.21.15.89:6812/3470471752] -- osd_op(unknown.0.0:60 12.16 12:68000000::::head [pgnls start_epoch 85 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e85) -- 0x55d9b8045400 con 0x55d9b801a400 2024-03-31T15:11:19.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.819+0000 7f65f1c00640 1 -- 172.21.15.16:0/4116959849 <== osd.6 v2:172.21.15.89:6809/3470471752 12 ==== osd_op_reply(60 [pgnls start_epoch 85 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55d9b7d1e500 con 0x55d9b801a400 2024-03-31T15:11:19.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.819+0000 7f65f1c00640 1 -- 172.21.15.16:0/4116959849 --> [v2:172.21.15.16:6800/932712244,v1:172.21.15.16:6801/932712244] -- osd_op(unknown.0.0:61 12.e 12:70000000::::head [pgnls start_epoch 85 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e85) -- 0x55d9b8045400 con 0x55d9b7f18800 2024-03-31T15:11:19.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.823+0000 7f65f1c00640 1 -- 172.21.15.16:0/4116959849 <== osd.0 v2:172.21.15.16:6800/932712244 19 ==== osd_op_reply(61 [pgnls start_epoch 85 out=73b] v84'3 uv3 ondisk = 1) ==== 144+0+73 (crc 0 0 0) 0x55d9b7d1e500 con 0x55d9b7f18800 2024-03-31T15:11:19.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.823+0000 7f65f1c00640 1 -- 172.21.15.16:0/4116959849 --> [v2:172.21.15.89:6800/1856844856,v1:172.21.15.89:6801/1856844856] -- osd_op(unknown.0.0:62 12.1e 12:78000000::::head [pgnls start_epoch 85 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e85) -- 0x55d9b8045400 con 0x55d9b8b4fc00 2024-03-31T15:11:19.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.823+0000 7f65f0800640 1 -- 172.21.15.16:0/4116959849 <== osd.4 v2:172.21.15.89:6800/1856844856 9 ==== osd_op_reply(62 [pgnls start_epoch 85 out=49b] v82'3 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55d9b7d1e000 con 0x55d9b8b4fc00 2024-03-31T15:11:19.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.823+0000 7f65f0800640 1 -- 172.21.15.16:0/4116959849 --> [v2:172.21.15.89:6809/3470471752,v1:172.21.15.89:6812/3470471752] -- osd_op(unknown.0.0:63 12.1 12:80000000::::head [pgnls start_epoch 85 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e85) -- 0x55d9b8045400 con 0x55d9b801a400 2024-03-31T15:11:19.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.823+0000 7f65f1c00640 1 -- 172.21.15.16:0/4116959849 <== osd.6 v2:172.21.15.89:6809/3470471752 13 ==== osd_op_reply(63 [pgnls start_epoch 85 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55d9b7d1e500 con 0x55d9b801a400 2024-03-31T15:11:19.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.823+0000 7f65f1c00640 1 -- 172.21.15.16:0/4116959849 --> [v2:172.21.15.89:6810/50395527,v1:172.21.15.89:6814/50395527] -- osd_op(unknown.0.0:64 12.11 12:88000000::::head [pgnls start_epoch 85 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e85) -- 0x55d9b8045400 con 0x55d9b7f18400 2024-03-31T15:11:19.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.823+0000 7f65f1200640 1 -- 172.21.15.16:0/4116959849 <== osd.5 v2:172.21.15.89:6810/50395527 5 ==== osd_op_reply(64 [pgnls start_epoch 85 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55d9b7d1e780 con 0x55d9b7f18400 2024-03-31T15:11:19.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.823+0000 7f65f1200640 1 -- 172.21.15.16:0/4116959849 --> [v2:172.21.15.16:6819/3018526646,v1:172.21.15.16:6822/3018526646] -- osd_op(unknown.0.0:65 12.9 12:90000000::::head [pgnls start_epoch 85 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e85) -- 0x55d9b8045400 con 0x55d9b801b000 2024-03-31T15:11:19.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.823+0000 7f65f0800640 1 -- 172.21.15.16:0/4116959849 <== osd.2 v2:172.21.15.16:6819/3018526646 5 ==== osd_op_reply(65 [pgnls start_epoch 85 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55d9b7d1e000 con 0x55d9b801b000 2024-03-31T15:11:19.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.823+0000 7f65f0800640 1 -- 172.21.15.16:0/4116959849 --> [v2:172.21.15.16:6800/932712244,v1:172.21.15.16:6801/932712244] -- osd_op(unknown.0.0:66 12.19 12:98000000::::head [pgnls start_epoch 85 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e85) -- 0x55d9b8045400 con 0x55d9b7f18800 2024-03-31T15:11:19.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.823+0000 7f65f1c00640 1 -- 172.21.15.16:0/4116959849 <== osd.0 v2:172.21.15.16:6800/932712244 20 ==== osd_op_reply(66 [pgnls start_epoch 85 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55d9b7d1e500 con 0x55d9b7f18800 2024-03-31T15:11:19.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.823+0000 7f65f1c00640 1 -- 172.21.15.16:0/4116959849 --> [v2:172.21.15.89:6809/3470471752,v1:172.21.15.89:6812/3470471752] -- osd_op(unknown.0.0:67 12.5 12:a0000000::::head [pgnls start_epoch 85 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e85) -- 0x55d9b8045400 con 0x55d9b801a400 2024-03-31T15:11:19.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.823+0000 7f65f1c00640 1 -- 172.21.15.16:0/4116959849 <== osd.6 v2:172.21.15.89:6809/3470471752 14 ==== osd_op_reply(67 [pgnls start_epoch 85 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55d9b7d1e500 con 0x55d9b801a400 2024-03-31T15:11:19.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.823+0000 7f65f1c00640 1 -- 172.21.15.16:0/4116959849 --> [v2:172.21.15.89:6809/3470471752,v1:172.21.15.89:6812/3470471752] -- osd_op(unknown.0.0:68 12.15 12:a8000000::::head [pgnls start_epoch 85 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e85) -- 0x55d9b8045400 con 0x55d9b801a400 2024-03-31T15:11:19.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.823+0000 7f65f1c00640 1 -- 172.21.15.16:0/4116959849 <== osd.6 v2:172.21.15.89:6809/3470471752 15 ==== osd_op_reply(68 [pgnls start_epoch 85 out=97b] v84'4 uv4 ondisk = 1) ==== 144+0+97 (crc 0 0 0) 0x55d9b7d1e500 con 0x55d9b801a400 2024-03-31T15:11:19.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.823+0000 7f65f1c00640 1 -- 172.21.15.16:0/4116959849 --> [v2:172.21.15.89:6808/1796778630,v1:172.21.15.89:6811/1796778630] -- osd_op(unknown.0.0:69 12.d 12:b0000000::::head [pgnls start_epoch 85 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e85) -- 0x55d9b8045400 con 0x55d9b801a800 2024-03-31T15:11:19.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.823+0000 7f65f1c00640 1 -- 172.21.15.16:0/4116959849 <== osd.7 v2:172.21.15.89:6808/1796778630 2 ==== osd_op_reply(69 [pgnls start_epoch 85 out=74b] v80'1 uv1 ondisk = 1) ==== 144+0+74 (crc 0 0 0) 0x55d9b7d1e500 con 0x55d9b801a800 2024-03-31T15:11:19.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.823+0000 7f65f1c00640 1 -- 172.21.15.16:0/4116959849 --> [v2:172.21.15.16:6800/932712244,v1:172.21.15.16:6801/932712244] -- osd_op(unknown.0.0:70 12.1d 12:b8000000::::head [pgnls start_epoch 85 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e85) -- 0x55d9b8045400 con 0x55d9b7f18800 2024-03-31T15:11:19.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.823+0000 7f65f1c00640 1 -- 172.21.15.16:0/4116959849 <== osd.0 v2:172.21.15.16:6800/932712244 21 ==== osd_op_reply(70 [pgnls start_epoch 85 out=73b] v84'5 uv5 ondisk = 1) ==== 144+0+73 (crc 0 0 0) 0x55d9b7d1e500 con 0x55d9b7f18800 2024-03-31T15:11:19.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.823+0000 7f65f1c00640 1 -- 172.21.15.16:0/4116959849 --> [v2:172.21.15.16:6800/932712244,v1:172.21.15.16:6801/932712244] -- osd_op(unknown.0.0:71 12.3 12:c0000000::::head [pgnls start_epoch 85 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e85) -- 0x55d9b8045400 con 0x55d9b7f18800 2024-03-31T15:11:19.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.823+0000 7f65f1c00640 1 -- 172.21.15.16:0/4116959849 <== osd.0 v2:172.21.15.16:6800/932712244 22 ==== osd_op_reply(71 [pgnls start_epoch 85 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55d9b7d1e500 con 0x55d9b7f18800 2024-03-31T15:11:19.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.823+0000 7f65f1c00640 1 -- 172.21.15.16:0/4116959849 --> [v2:172.21.15.16:6808/1398737533,v1:172.21.15.16:6810/1398737533] -- osd_op(unknown.0.0:72 12.13 12:c8000000::::head [pgnls start_epoch 85 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e85) -- 0x55d9b8045400 con 0x55d9b7cff400 2024-03-31T15:11:19.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.827+0000 7f65f0800640 1 -- 172.21.15.16:0/4116959849 <== osd.1 v2:172.21.15.16:6808/1398737533 9 ==== osd_op_reply(72 [pgnls start_epoch 85 out=98b] v84'4 uv4 ondisk = 1) ==== 144+0+98 (crc 0 0 0) 0x55d9b7d1e000 con 0x55d9b7cff400 2024-03-31T15:11:19.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.827+0000 7f65f0800640 1 -- 172.21.15.16:0/4116959849 --> [v2:172.21.15.16:6809/3360352974,v1:172.21.15.16:6812/3360352974] -- osd_op(unknown.0.0:73 12.b 12:d0000000::::head [pgnls start_epoch 85 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e85) -- 0x55d9b8045400 con 0x55d9b7f18000 2024-03-31T15:11:19.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.827+0000 7f65f1200640 1 -- 172.21.15.16:0/4116959849 <== osd.3 v2:172.21.15.16:6809/3360352974 6 ==== osd_op_reply(73 [pgnls start_epoch 85 out=76b] v84'6 uv6 ondisk = 1) ==== 144+0+76 (crc 0 0 0) 0x55d9b7d1e780 con 0x55d9b7f18000 2024-03-31T15:11:19.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.827+0000 7f65f1200640 1 -- 172.21.15.16:0/4116959849 --> [v2:172.21.15.89:6800/1856844856,v1:172.21.15.89:6801/1856844856] -- osd_op(unknown.0.0:74 12.1b 12:d8000000::::head [pgnls start_epoch 85 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e85) -- 0x55d9b8045400 con 0x55d9b8b4fc00 2024-03-31T15:11:19.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.827+0000 7f65f0800640 1 -- 172.21.15.16:0/4116959849 <== osd.4 v2:172.21.15.89:6800/1856844856 10 ==== osd_op_reply(74 [pgnls start_epoch 85 out=73b] v84'10 uv10 ondisk = 1) ==== 144+0+73 (crc 0 0 0) 0x55d9b7d1e000 con 0x55d9b8b4fc00 2024-03-31T15:11:19.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.827+0000 7f65f0800640 1 -- 172.21.15.16:0/4116959849 --> [v2:172.21.15.16:6808/1398737533,v1:172.21.15.16:6810/1398737533] -- osd_op(unknown.0.0:75 12.7 12:e0000000::::head [pgnls start_epoch 85 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e85) -- 0x55d9b8045400 con 0x55d9b7cff400 2024-03-31T15:11:19.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.827+0000 7f65f0800640 1 -- 172.21.15.16:0/4116959849 <== osd.1 v2:172.21.15.16:6808/1398737533 10 ==== osd_op_reply(75 [pgnls start_epoch 85 out=73b] v84'3 uv3 ondisk = 1) ==== 144+0+73 (crc 0 0 0) 0x55d9b7d1e000 con 0x55d9b7cff400 2024-03-31T15:11:19.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.827+0000 7f65f0800640 1 -- 172.21.15.16:0/4116959849 --> [v2:172.21.15.89:6809/3470471752,v1:172.21.15.89:6812/3470471752] -- osd_op(unknown.0.0:76 12.17 12:e8000000::::head [pgnls start_epoch 85 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e85) -- 0x55d9b8045400 con 0x55d9b801a400 2024-03-31T15:11:19.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.827+0000 7f65f1c00640 1 -- 172.21.15.16:0/4116959849 <== osd.6 v2:172.21.15.89:6809/3470471752 16 ==== osd_op_reply(76 [pgnls start_epoch 85 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55d9b7d1e500 con 0x55d9b801a400 2024-03-31T15:11:19.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.827+0000 7f65f1c00640 1 -- 172.21.15.16:0/4116959849 --> [v2:172.21.15.89:6809/3470471752,v1:172.21.15.89:6812/3470471752] -- osd_op(unknown.0.0:77 12.f 12:f0000000::::head [pgnls start_epoch 85 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e85) -- 0x55d9b8045400 con 0x55d9b801a400 2024-03-31T15:11:19.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.827+0000 7f65f1c00640 1 -- 172.21.15.16:0/4116959849 <== osd.6 v2:172.21.15.89:6809/3470471752 17 ==== osd_op_reply(77 [pgnls start_epoch 85 out=73b] v80'1 uv1 ondisk = 1) ==== 144+0+73 (crc 0 0 0) 0x55d9b7d1e500 con 0x55d9b801a400 2024-03-31T15:11:19.829 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.827+0000 7f65f1c00640 1 -- 172.21.15.16:0/4116959849 --> [v2:172.21.15.89:6809/3470471752,v1:172.21.15.89:6812/3470471752] -- osd_op(unknown.0.0:78 12.1f 12:f8000000::::head [pgnls start_epoch 85 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e85) -- 0x55d9b8045400 con 0x55d9b801a400 2024-03-31T15:11:19.829 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.827+0000 7f65f1c00640 1 -- 172.21.15.16:0/4116959849 <== osd.6 v2:172.21.15.89:6809/3470471752 18 ==== osd_op_reply(78 [pgnls start_epoch 85 out=101b] v84'9 uv9 ondisk = 1) ==== 144+0+101 (crc 0 0 0) 0x55d9b7d1e500 con 0x55d9b801a400 2024-03-31T15:11:19.829 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.827+0000 7f65f31ca8c0 10 datascan.scan_links: step 2: handling object 601.00000000 2024-03-31T15:11:19.829 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.827+0000 7f65f31ca8c0 1 -- 172.21.15.16:0/4116959849 --> [v2:172.21.15.16:6800/932712244,v1:172.21.15.16:6801/932712244] -- osd_op(unknown.0.0:79 12.10 12:09401a81:::601.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e85) -- 0x55d9b8045800 con 0x55d9b7f18800 2024-03-31T15:11:19.829 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.827+0000 7f65f1c00640 1 -- 172.21.15.16:0/4116959849 <== osd.0 v2:172.21.15.16:6800/932712244 23 ==== osd_op_reply(79 601.00000000 [omap-get-vals out=5b] v0'0 uv13 ondisk = 0) ==== 156+0+5 (crc 0 0 0) 0x55d9b7d1e500 con 0x55d9b7f18800 2024-03-31T15:11:19.829 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.827+0000 7f65f31ca8c0 10 datascan.scan_links: step 2: handling object 602.00000000 2024-03-31T15:11:19.829 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.827+0000 7f65f31ca8c0 1 -- 172.21.15.16:0/4116959849 --> [v2:172.21.15.16:6800/932712244,v1:172.21.15.16:6801/932712244] -- osd_op(unknown.0.0:80 12.10 12:0bd6d154:::602.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e85) -- 0x55d9b8045c00 con 0x55d9b7f18800 2024-03-31T15:11:19.829 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.827+0000 7f65f1c00640 1 -- 172.21.15.16:0/4116959849 <== osd.0 v2:172.21.15.16:6800/932712244 24 ==== osd_op_reply(80 602.00000000 [omap-get-vals out=5b] v0'0 uv14 ondisk = 0) ==== 156+0+5 (crc 0 0 0) 0x55d9b7d1e500 con 0x55d9b7f18800 2024-03-31T15:11:19.829 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.827+0000 7f65f31ca8c0 10 datascan.scan_links: step 2: handling object 10000000000.00000000 2024-03-31T15:11:19.829 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.827+0000 7f65f31ca8c0 1 -- 172.21.15.16:0/4116959849 --> [v2:172.21.15.16:6800/932712244,v1:172.21.15.16:6801/932712244] -- osd_op(unknown.0.0:81 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e85) -- 0x55d9b8044000 con 0x55d9b7f18800 2024-03-31T15:11:19.829 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.827+0000 7f65f1c00640 1 -- 172.21.15.16:0/4116959849 <== osd.0 v2:172.21.15.16:6800/932712244 25 ==== osd_op_reply(81 10000000000.00000000 [omap-get-vals out=506b] v0'0 uv18 ondisk = 0) ==== 164+0+506 (crc 0 0 0) 0x55d9b7d1e500 con 0x55d9b7f18800 2024-03-31T15:11:19.829 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.827+0000 7f65f31ca8c0 20 datascan.scan_links: injected ino detected 2024-03-31T15:11:19.829 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.827+0000 7f65f31ca8c0 -1 datascan.scan_links: Bad nlink on 0x10000000001 expected 2 has 1 2024-03-31T15:11:19.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.827+0000 7f65f31ca8c0 20 datascan.scan_links: adding 0x10000000001 for future processing to fix dnfirst 2024-03-31T15:11:19.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.827+0000 7f65f31ca8c0 10 datascan.scan_links: step 2: handling object 600.00000000 2024-03-31T15:11:19.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.827+0000 7f65f31ca8c0 1 -- 172.21.15.16:0/4116959849 --> [v2:172.21.15.16:6800/932712244,v1:172.21.15.16:6801/932712244] -- osd_op(unknown.0.0:82 12.10 12:0d82a743:::600.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e85) -- 0x55d9b8044400 con 0x55d9b7f18800 2024-03-31T15:11:19.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.827+0000 7f65f1c00640 1 -- 172.21.15.16:0/4116959849 <== osd.0 v2:172.21.15.16:6800/932712244 26 ==== osd_op_reply(82 600.00000000 [omap-get-vals out=5b] v0'0 uv12 ondisk = 0) ==== 156+0+5 (crc 0 0 0) 0x55d9b7d1e500 con 0x55d9b7f18800 2024-03-31T15:11:19.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.827+0000 7f65f31ca8c0 10 datascan.scan_links: step 2: handling object 603.00000000 2024-03-31T15:11:19.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.827+0000 7f65f31ca8c0 1 -- 172.21.15.16:0/4116959849 --> [v2:172.21.15.16:6800/932712244,v1:172.21.15.16:6801/932712244] -- osd_op(unknown.0.0:83 12.10 12:0d89b25e:::603.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e85) -- 0x55d9b8045000 con 0x55d9b7f18800 2024-03-31T15:11:19.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.827+0000 7f65f1c00640 1 -- 172.21.15.16:0/4116959849 <== osd.0 v2:172.21.15.16:6800/932712244 27 ==== osd_op_reply(83 603.00000000 [omap-get-vals out=5b] v0'0 uv15 ondisk = 0) ==== 156+0+5 (crc 0 0 0) 0x55d9b7d1e500 con 0x55d9b7f18800 2024-03-31T15:11:19.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.827+0000 7f65f31ca8c0 10 datascan.scan_links: step 2: handling object 1.00000000.inode 2024-03-31T15:11:19.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.827+0000 7f65f31ca8c0 10 datascan.scan_links: Not a dirfrag: '1.00000000.inode' 2024-03-31T15:11:19.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.827+0000 7f65f31ca8c0 10 datascan.scan_links: step 2: handling object 200.00000000 2024-03-31T15:11:19.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.827+0000 7f65f31ca8c0 10 datascan.scan_links: Not a dirfrag (invalid ino): '200.00000000' 2024-03-31T15:11:19.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.827+0000 7f65f31ca8c0 10 datascan.scan_links: step 2: handling object 200.00000001 2024-03-31T15:11:19.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.827+0000 7f65f31ca8c0 10 datascan.scan_links: Not a dirfrag (invalid ino): '200.00000001' 2024-03-31T15:11:19.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.827+0000 7f65f31ca8c0 10 datascan.scan_links: step 2: handling object 606.00000000 2024-03-31T15:11:19.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.827+0000 7f65f31ca8c0 1 -- 172.21.15.16:0/4116959849 --> [v2:172.21.15.89:6810/50395527,v1:172.21.15.89:6814/50395527] -- osd_op(unknown.0.0:84 12.14 12:2f55791f:::606.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e85) -- 0x55d9b801bc00 con 0x55d9b7f18400 2024-03-31T15:11:19.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.831+0000 7f65f1200640 1 -- 172.21.15.16:0/4116959849 <== osd.5 v2:172.21.15.89:6810/50395527 6 ==== osd_op_reply(84 606.00000000 [omap-get-vals out=5b] v0'0 uv25 ondisk = 0) ==== 156+0+5 (crc 0 0 0) 0x55d9b7d1e780 con 0x55d9b7f18400 2024-03-31T15:11:19.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.831+0000 7f65f31ca8c0 10 datascan.scan_links: step 2: handling object 607.00000000 2024-03-31T15:11:19.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.831+0000 7f65f31ca8c0 1 -- 172.21.15.16:0/4116959849 --> [v2:172.21.15.16:6800/932712244,v1:172.21.15.16:6801/932712244] -- osd_op(unknown.0.0:85 12.1c 12:3ed09add:::607.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e85) -- 0x55d9b801b400 con 0x55d9b7f18800 2024-03-31T15:11:19.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.831+0000 7f65f1c00640 1 -- 172.21.15.16:0/4116959849 <== osd.0 v2:172.21.15.16:6800/932712244 28 ==== osd_op_reply(85 607.00000000 [omap-get-vals out=5b] v0'0 uv3 ondisk = 0) ==== 156+0+5 (crc 0 0 0) 0x55d9b7d1e500 con 0x55d9b7f18800 2024-03-31T15:11:19.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.831+0000 7f65f31ca8c0 10 datascan.scan_links: step 2: handling object 100000001f6.00000000 2024-03-31T15:11:19.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.831+0000 7f65f31ca8c0 1 -- 172.21.15.16:0/4116959849 --> [v2:172.21.15.16:6819/3018526646,v1:172.21.15.16:6822/3018526646] -- osd_op(unknown.0.0:86 12.a 12:57fe7535:::100000001f6.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e85) -- 0x55d9b7f19400 con 0x55d9b801b000 2024-03-31T15:11:19.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.831+0000 7f65f0800640 1 -- 172.21.15.16:0/4116959849 <== osd.2 v2:172.21.15.16:6819/3018526646 6 ==== osd_op_reply(86 100000001f6.00000000 [omap-get-vals out=54b] v0'0 uv3 ondisk = 0) ==== 164+0+54 (crc 0 0 0) 0x55d9b7d1e000 con 0x55d9b801b000 2024-03-31T15:11:19.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.831+0000 7f65f31ca8c0 10 datascan.scan_links: step 2: handling object mds0_openfiles.0 2024-03-31T15:11:19.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.831+0000 7f65f31ca8c0 10 datascan.scan_links: Not a dirfrag: 'mds0_openfiles.0' 2024-03-31T15:11:19.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.831+0000 7f65f31ca8c0 10 datascan.scan_links: step 2: handling object 608.00000000 2024-03-31T15:11:19.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.831+0000 7f65f31ca8c0 1 -- 172.21.15.16:0/4116959849 --> [v2:172.21.15.16:6800/932712244,v1:172.21.15.16:6801/932712244] -- osd_op(unknown.0.0:87 12.e 12:717a0223:::608.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e85) -- 0x55d9b7f19800 con 0x55d9b7f18800 2024-03-31T15:11:19.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.831+0000 7f65f1c00640 1 -- 172.21.15.16:0/4116959849 <== osd.0 v2:172.21.15.16:6800/932712244 29 ==== osd_op_reply(87 608.00000000 [omap-get-vals out=5b] v0'0 uv3 ondisk = 0) ==== 156+0+5 (crc 0 0 0) 0x55d9b7d1e500 con 0x55d9b7f18800 2024-03-31T15:11:19.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.831+0000 7f65f31ca8c0 10 datascan.scan_links: step 2: handling object 604.00000000 2024-03-31T15:11:19.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.831+0000 7f65f31ca8c0 1 -- 172.21.15.16:0/4116959849 --> [v2:172.21.15.89:6809/3470471752,v1:172.21.15.89:6812/3470471752] -- osd_op(unknown.0.0:88 12.15 12:a93a17c2:::604.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e85) -- 0x55d9b7f19800 con 0x55d9b801a400 2024-03-31T15:11:19.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.831+0000 7f65f1c00640 1 -- 172.21.15.16:0/4116959849 <== osd.6 v2:172.21.15.89:6809/3470471752 19 ==== osd_op_reply(88 604.00000000 [omap-get-vals out=5b] v0'0 uv4 ondisk = 0) ==== 156+0+5 (crc 0 0 0) 0x55d9b7d1e500 con 0x55d9b801a400 2024-03-31T15:11:19.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.831+0000 7f65f31ca8c0 10 datascan.scan_links: step 2: handling object 500.00000000 2024-03-31T15:11:19.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.831+0000 7f65f31ca8c0 10 datascan.scan_links: Not a dirfrag (invalid ino): '500.00000000' 2024-03-31T15:11:19.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.831+0000 7f65f31ca8c0 10 datascan.scan_links: step 2: handling object mds_snaptable 2024-03-31T15:11:19.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.831+0000 7f65f31ca8c0 10 datascan.scan_links: Not a dirfrag: 'mds_snaptable' 2024-03-31T15:11:19.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.831+0000 7f65f31ca8c0 10 datascan.scan_links: step 2: handling object 605.00000000 2024-03-31T15:11:19.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.831+0000 7f65f31ca8c0 1 -- 172.21.15.16:0/4116959849 --> [v2:172.21.15.16:6800/932712244,v1:172.21.15.16:6801/932712244] -- osd_op(unknown.0.0:89 12.1d 12:b871830b:::605.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e85) -- 0x55d9b801b400 con 0x55d9b7f18800 2024-03-31T15:11:19.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.831+0000 7f65f1c00640 1 -- 172.21.15.16:0/4116959849 <== osd.0 v2:172.21.15.16:6800/932712244 30 ==== osd_op_reply(89 605.00000000 [omap-get-vals out=5b] v0'0 uv5 ondisk = 0) ==== 156+0+5 (crc 0 0 0) 0x55d9b7d1e500 con 0x55d9b7f18800 2024-03-31T15:11:19.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.831+0000 7f65f31ca8c0 10 datascan.scan_links: step 2: handling object mds0_inotable 2024-03-31T15:11:19.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.831+0000 7f65f31ca8c0 10 datascan.scan_links: Not a dirfrag: 'mds0_inotable' 2024-03-31T15:11:19.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.831+0000 7f65f31ca8c0 10 datascan.scan_links: step 2: handling object 100.00000000 2024-03-31T15:11:19.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.831+0000 7f65f31ca8c0 1 -- 172.21.15.16:0/4116959849 --> [v2:172.21.15.16:6808/1398737533,v1:172.21.15.16:6810/1398737533] -- osd_op(unknown.0.0:90 12.13 12:cd5a64a3:::100.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e85) -- 0x55d9b8045400 con 0x55d9b7cff400 2024-03-31T15:11:19.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.831+0000 7f65f0800640 1 -- 172.21.15.16:0/4116959849 <== osd.1 v2:172.21.15.16:6808/1398737533 11 ==== osd_op_reply(90 100.00000000 [omap-get-vals out=5195b] v0'0 uv4 ondisk = 0) ==== 156+0+5195 (crc 0 0 0) 0x55d9b7d1e000 con 0x55d9b7cff400 2024-03-31T15:11:19.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.831+0000 7f65f31ca8c0 10 datascan.scan_links: step 2: handling object mds0_sessionmap 2024-03-31T15:11:19.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.831+0000 7f65f31ca8c0 10 datascan.scan_links: Not a dirfrag: 'mds0_sessionmap' 2024-03-31T15:11:19.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.831+0000 7f65f31ca8c0 10 datascan.scan_links: step 2: handling object 200.00000002 2024-03-31T15:11:19.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.831+0000 7f65f31ca8c0 10 datascan.scan_links: Not a dirfrag (invalid ino): '200.00000002' 2024-03-31T15:11:19.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.831+0000 7f65f31ca8c0 10 datascan.scan_links: step 2: handling object 609.00000000 2024-03-31T15:11:19.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.831+0000 7f65f31ca8c0 1 -- 172.21.15.16:0/4116959849 --> [v2:172.21.15.16:6808/1398737533,v1:172.21.15.16:6810/1398737533] -- osd_op(unknown.0.0:91 12.7 12:e0b41d06:::609.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e85) -- 0x55d9b8045800 con 0x55d9b7cff400 2024-03-31T15:11:19.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.831+0000 7f65f0800640 1 -- 172.21.15.16:0/4116959849 <== osd.1 v2:172.21.15.16:6808/1398737533 12 ==== osd_op_reply(91 609.00000000 [omap-get-vals out=5b] v0'0 uv3 ondisk = 0) ==== 156+0+5 (crc 0 0 0) 0x55d9b7d1e000 con 0x55d9b7cff400 2024-03-31T15:11:19.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.831+0000 7f65f31ca8c0 10 datascan.scan_links: step 2: handling object 400.00000000 2024-03-31T15:11:19.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.831+0000 7f65f31ca8c0 10 datascan.scan_links: Not a dirfrag (invalid ino): '400.00000000' 2024-03-31T15:11:19.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.831+0000 7f65f31ca8c0 10 datascan.scan_links: step 2: handling object 100.00000000.inode 2024-03-31T15:11:19.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.831+0000 7f65f31ca8c0 10 datascan.scan_links: Not a dirfrag: '100.00000000.inode' 2024-03-31T15:11:19.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.831+0000 7f65f31ca8c0 10 datascan.scan_links: step 2: handling object 1.00000000 2024-03-31T15:11:19.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.831+0000 7f65f31ca8c0 1 -- 172.21.15.16:0/4116959849 --> [v2:172.21.15.89:6809/3470471752,v1:172.21.15.89:6812/3470471752] -- osd_op(unknown.0.0:92 12.1f 12:ff5b34d6:::1.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e85) -- 0x55d9b8045c00 con 0x55d9b801a400 2024-03-31T15:11:19.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.831+0000 7f65f1c00640 1 -- 172.21.15.16:0/4116959849 <== osd.6 v2:172.21.15.89:6809/3470471752 20 ==== osd_op_reply(92 1.00000000 [omap-get-vals out=1041b] v0'0 uv9 ondisk = 0) ==== 154+0+1041 (crc 0 0 0) 0x55d9b7d1e500 con 0x55d9b801a400 2024-03-31T15:11:19.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.831+0000 7f65f31ca8c0 10 datascan.scan_links: processing 0 dup_primaries, 1 remote_links 2024-03-31T15:11:19.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.831+0000 7f65f31ca8c0 10 datascan.scan_links: removing dup dentries from 0 objects 2024-03-31T15:11:19.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.831+0000 7f65f31ca8c0 10 datascan.scan_links: processing 1 bad_nlink_inos 2024-03-31T15:11:19.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.831+0000 7f65f31ca8c0 10 datascan.scan_links: handling bad_nlink_ino 0x10000000001 2024-03-31T15:11:19.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.831+0000 7f65f31ca8c0 1 -- 172.21.15.16:0/4116959849 --> [v2:172.21.15.16:6800/932712244,v1:172.21.15.16:6801/932712244] -- osd_op(unknown.0.0:93 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-get-vals-by-keys in=17b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e85) -- 0x55d9b8044000 con 0x55d9b7f18800 2024-03-31T15:11:19.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.831+0000 7f65f1c00640 1 -- 172.21.15.16:0/4116959849 <== osd.0 v2:172.21.15.16:6800/932712244 31 ==== osd_op_reply(93 10000000000.00000000 [omap-get-vals-by-keys out=505b] v0'0 uv18 ondisk = 0) ==== 164+0+505 (crc 0 0 0) 0x55d9b7d1e500 con 0x55d9b7f18800 2024-03-31T15:11:19.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.831+0000 7f65f31ca8c0 20 datascan.read_dentry: oid=10000000000.00000000 dname=file frag=*, r=0 2024-03-31T15:11:19.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.831+0000 7f65f31ca8c0 1 -- 172.21.15.16:0/4116959849 --> [v2:172.21.15.16:6800/932712244,v1:172.21.15.16:6801/932712244] -- osd_op(unknown.0.0:94 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-set-vals in=505b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85) -- 0x55d9b8044400 con 0x55d9b7f18800 2024-03-31T15:11:19.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.835+0000 7f65f1c00640 1 -- 172.21.15.16:0/4116959849 <== osd.0 v2:172.21.15.16:6800/932712244 32 ==== osd_op_reply(94 10000000000.00000000 [omap-set-vals] v85'19 uv19 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x55d9b7d1e500 con 0x55d9b7f18800 2024-03-31T15:11:19.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.835+0000 7f65f31ca8c0 20 datascan.inject_linkage: Injected dentry 0x0x10000000000/file pointing to 0x0x10000000001 2024-03-31T15:11:19.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.835+0000 7f65f31ca8c0 10 datascan.scan_links: processing 1 injected_inos 2024-03-31T15:11:19.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.835+0000 7f65f31ca8c0 10 datascan.scan_links: handling injected_ino 0x10000000001 2024-03-31T15:11:19.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.835+0000 7f65f31ca8c0 20 datascan.scan_links: fixing linkage (dnfirst) of 0x10000000000:file 2024-03-31T15:11:19.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.835+0000 7f65f31ca8c0 1 -- 172.21.15.16:0/4116959849 --> [v2:172.21.15.16:6800/932712244,v1:172.21.15.16:6801/932712244] -- osd_op(unknown.0.0:95 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-get-vals-by-keys in=17b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e85) -- 0x55d9b8045000 con 0x55d9b7f18800 2024-03-31T15:11:19.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.835+0000 7f65f1c00640 1 -- 172.21.15.16:0/4116959849 <== osd.0 v2:172.21.15.16:6800/932712244 33 ==== osd_op_reply(95 10000000000.00000000 [omap-get-vals-by-keys out=505b] v0'0 uv19 ondisk = 0) ==== 164+0+505 (crc 0 0 0) 0x55d9b7d1e500 con 0x55d9b7f18800 2024-03-31T15:11:19.839 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.835+0000 7f65f31ca8c0 20 datascan.read_dentry: oid=10000000000.00000000 dname=file frag=*, r=0 2024-03-31T15:11:19.839 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.835+0000 7f65f31ca8c0 20 datascan.scan_links: first is now 2 2024-03-31T15:11:19.839 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.835+0000 7f65f31ca8c0 1 -- 172.21.15.16:0/4116959849 --> [v2:172.21.15.16:6800/932712244,v1:172.21.15.16:6801/932712244] -- osd_op(unknown.0.0:96 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-set-vals in=505b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85) -- 0x55d9b8044800 con 0x55d9b7f18800 2024-03-31T15:11:19.839 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.835+0000 7f65f1c00640 1 -- 172.21.15.16:0/4116959849 <== osd.0 v2:172.21.15.16:6800/932712244 34 ==== osd_op_reply(96 10000000000.00000000 [omap-set-vals] v85'20 uv20 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x55d9b7d1e500 con 0x55d9b7f18800 2024-03-31T15:11:19.839 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.835+0000 7f65f31ca8c0 20 datascan.inject_linkage: Injected dentry 0x0x10000000000/file pointing to 0x0x10000000001 2024-03-31T15:11:19.839 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.835+0000 7f65f31ca8c0 10 datascan.scan_links: updating inotable 2024-03-31T15:11:19.839 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.835+0000 7f65f31ca8c0 1 -- 172.21.15.16:0/4116959849 --> [v2:172.21.15.16:6808/1398737533,v1:172.21.15.16:6810/1398737533] -- osd_op(unknown.0.0:97 12.13 12:c91d4a1d:::mds0_inotable:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e85) -- 0x55d9b8044c00 con 0x55d9b7cff400 2024-03-31T15:11:19.839 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.835+0000 7f65f0800640 1 -- 172.21.15.16:0/4116959849 <== osd.1 v2:172.21.15.16:6808/1398737533 13 ==== osd_op_reply(97 mds0_inotable [read 0~50 out=50b] v0'0 uv3 ondisk = 0) ==== 157+0+50 (crc 0 0 0) 0x55d9b7d1e000 con 0x55d9b7cff400 2024-03-31T15:11:19.839 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.835+0000 7f65f31ca8c0 10 mds.0.inotable: skip_inos was [0x10000000002~0x1f4,0x100000001f7~0xfffffffe09] 2024-03-31T15:11:19.839 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.835+0000 7f65f31ca8c0 10 mds.0.inotable: skip_inos now [0x100000001f7~0xfffffffe09] 2024-03-31T15:11:19.839 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.835+0000 7f65f31ca8c0 1 -- 172.21.15.16:0/4116959849 --> [v2:172.21.15.16:6808/1398737533,v1:172.21.15.16:6810/1398737533] -- osd_op(unknown.0.0:98 12.13 12:c91d4a1d:::mds0_inotable:head [writefull 0~34 in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85) -- 0x55d9b8045400 con 0x55d9b7cff400 2024-03-31T15:11:19.839 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.839+0000 7f65f0800640 1 -- 172.21.15.16:0/4116959849 <== osd.1 v2:172.21.15.16:6808/1398737533 14 ==== osd_op_reply(98 mds0_inotable [writefull 0~34] v85'5 uv5 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x55d9b7d1e000 con 0x55d9b7cff400 2024-03-31T15:11:19.839 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.839+0000 7f65f31ca8c0 10 datascan.scan_links: updating snaptable 2024-03-31T15:11:19.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.839+0000 7f65f31ca8c0 1 -- 172.21.15.16:0/4116959849 --> [v2:172.21.15.89:6808/1796778630,v1:172.21.15.89:6811/1796778630] -- osd_op(unknown.0.0:99 12.d 12:b50e409b:::mds_snaptable:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e85) -- 0x55d9b7f19000 con 0x55d9b801a800 2024-03-31T15:11:19.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.839+0000 7f65f1c00640 1 -- 172.21.15.16:0/4116959849 <== osd.7 v2:172.21.15.89:6808/1796778630 3 ==== osd_op_reply(99 mds_snaptable [read 0~70 out=70b] v0'0 uv1 ondisk = 0) ==== 157+0+70 (crc 0 0 0) 0x55d9b7d1e280 con 0x55d9b801a800 2024-03-31T15:11:19.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.839+0000 7f65f31ca8c0 1 -- 172.21.15.16:0/4116959849 >> [v2:172.21.15.16:6800/932712244,v1:172.21.15.16:6801/932712244] conn(0x55d9b7f18800 msgr2=0x55d9b8020000 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:11:19.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.839+0000 7f65f31ca8c0 1 --2- 172.21.15.16:0/4116959849 >> [v2:172.21.15.16:6800/932712244,v1:172.21.15.16:6801/932712244] conn(0x55d9b7f18800 0x55d9b8020000 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:19.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.839+0000 7f65f31ca8c0 1 -- 172.21.15.16:0/4116959849 >> [v2:172.21.15.16:6808/1398737533,v1:172.21.15.16:6810/1398737533] conn(0x55d9b7cff400 msgr2=0x55d9b8020b00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:11:19.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.839+0000 7f65f31ca8c0 1 --2- 172.21.15.16:0/4116959849 >> [v2:172.21.15.16:6808/1398737533,v1:172.21.15.16:6810/1398737533] conn(0x55d9b7cff400 0x55d9b8020b00 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:19.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.839+0000 7f65f31ca8c0 1 -- 172.21.15.16:0/4116959849 >> [v2:172.21.15.16:6819/3018526646,v1:172.21.15.16:6822/3018526646] conn(0x55d9b801b000 msgr2=0x55d9b8022680 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:11:19.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.839+0000 7f65f31ca8c0 1 --2- 172.21.15.16:0/4116959849 >> [v2:172.21.15.16:6819/3018526646,v1:172.21.15.16:6822/3018526646] conn(0x55d9b801b000 0x55d9b8022680 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:19.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.839+0000 7f65f31ca8c0 1 -- 172.21.15.16:0/4116959849 >> [v2:172.21.15.16:6809/3360352974,v1:172.21.15.16:6812/3360352974] conn(0x55d9b7f18000 msgr2=0x55d9b8020580 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:11:19.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.839+0000 7f65f31ca8c0 1 --2- 172.21.15.16:0/4116959849 >> [v2:172.21.15.16:6809/3360352974,v1:172.21.15.16:6812/3360352974] conn(0x55d9b7f18000 0x55d9b8020580 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:19.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.839+0000 7f65f31ca8c0 1 -- 172.21.15.16:0/4116959849 >> [v2:172.21.15.89:6800/1856844856,v1:172.21.15.89:6801/1856844856] conn(0x55d9b8b4fc00 msgr2=0x55d9b89f2680 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:11:19.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.839+0000 7f65f31ca8c0 1 --2- 172.21.15.16:0/4116959849 >> [v2:172.21.15.89:6800/1856844856,v1:172.21.15.89:6801/1856844856] conn(0x55d9b8b4fc00 0x55d9b89f2680 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:19.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.839+0000 7f65f31ca8c0 1 -- 172.21.15.16:0/4116959849 >> [v2:172.21.15.89:6810/50395527,v1:172.21.15.89:6814/50395527] conn(0x55d9b7f18400 msgr2=0x55d9b8021b80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:11:19.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.839+0000 7f65f31ca8c0 1 --2- 172.21.15.16:0/4116959849 >> [v2:172.21.15.89:6810/50395527,v1:172.21.15.89:6814/50395527] conn(0x55d9b7f18400 0x55d9b8021b80 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:19.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.839+0000 7f65f31ca8c0 1 -- 172.21.15.16:0/4116959849 >> [v2:172.21.15.89:6809/3470471752,v1:172.21.15.89:6812/3470471752] conn(0x55d9b801a400 msgr2=0x55d9b8021080 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:11:19.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.839+0000 7f65f31ca8c0 1 --2- 172.21.15.16:0/4116959849 >> [v2:172.21.15.89:6809/3470471752,v1:172.21.15.89:6812/3470471752] conn(0x55d9b801a400 0x55d9b8021080 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:19.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.839+0000 7f65f31ca8c0 1 -- 172.21.15.16:0/4116959849 >> [v2:172.21.15.89:6808/1796778630,v1:172.21.15.89:6811/1796778630] conn(0x55d9b801a800 msgr2=0x55d9b8022100 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:11:19.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.839+0000 7f65f31ca8c0 1 --2- 172.21.15.16:0/4116959849 >> [v2:172.21.15.89:6808/1796778630,v1:172.21.15.89:6811/1796778630] conn(0x55d9b801a800 0x55d9b8022100 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:19.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.839+0000 7f65f0800640 1 -- 172.21.15.16:0/4116959849 reap_dead start 2024-03-31T15:11:19.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.839+0000 7f65f0800640 1 -- 172.21.15.16:0/4116959849 reap_dead start 2024-03-31T15:11:19.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.839+0000 7f65f0800640 1 -- 172.21.15.16:0/4116959849 reap_dead start 2024-03-31T15:11:19.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.839+0000 7f65f0800640 1 -- 172.21.15.16:0/4116959849 reap_dead start 2024-03-31T15:11:19.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.839+0000 7f65f31ca8c0 1 -- 172.21.15.16:0/4116959849 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x55d9b7f18c00 msgr2=0x55d9b89f2100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:11:19.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.839+0000 7f65f31ca8c0 1 --2- 172.21.15.16:0/4116959849 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x55d9b7f18c00 0x55d9b89f2100 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x55d9b8a7db90 tx=0x55d9b7cdef00 comp rx=0 tx=0).stop 2024-03-31T15:11:19.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.839+0000 7f65f31ca8c0 1 -- 172.21.15.16:0/4116959849 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d9b8b4f000 msgr2=0x55d9b89f1080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:11:19.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.839+0000 7f65f31ca8c0 1 --2- 172.21.15.16:0/4116959849 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d9b8b4f000 0x55d9b89f1080 secure :-1 s=READY pgs=1259 cs=0 l=1 rev1=1 crypto rx=0x55d9b8a7cff0 tx=0x55d9b8994000 comp rx=0 tx=0).stop 2024-03-31T15:11:19.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.839+0000 7f65f31ca8c0 1 -- 172.21.15.16:0/4116959849 shutdown_connections 2024-03-31T15:11:19.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.839+0000 7f65f31ca8c0 1 --2- 172.21.15.16:0/4116959849 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x55d9b7f18c00 0x55d9b89f2100 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:19.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.839+0000 7f65f31ca8c0 1 --2- 172.21.15.16:0/4116959849 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d9b8b4f000 0x55d9b89f1080 unknown :-1 s=CLOSED pgs=1259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:19.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.839+0000 7f65f31ca8c0 1 -- 172.21.15.16:0/4116959849 >> 172.21.15.16:0/4116959849 conn(0x55d9b7c26c00 msgr2=0x55d9b7c52800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:11:19.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.839+0000 7f65f31ca8c0 1 -- 172.21.15.16:0/4116959849 shutdown_connections 2024-03-31T15:11:19.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.839+0000 7f65f31ca8c0 1 -- 172.21.15.16:0/4116959849 wait complete. 2024-03-31T15:11:19.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.839+0000 7f65f31ca8c0 1 -- 172.21.15.16:0/3093794445 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d9b7c27800 msgr2=0x55d9b89f0000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:11:19.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.839+0000 7f65f31ca8c0 1 --2- 172.21.15.16:0/3093794445 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d9b7c27800 0x55d9b89f0000 secure :-1 s=READY pgs=1257 cs=0 l=1 rev1=1 crypto rx=0x55d9b8a7d8c0 tx=0x55d9b89aea80 comp rx=0 tx=0).stop 2024-03-31T15:11:19.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.839+0000 7f65f31ca8c0 1 -- 172.21.15.16:0/3093794445 shutdown_connections 2024-03-31T15:11:19.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.839+0000 7f65f31ca8c0 1 --2- 172.21.15.16:0/3093794445 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d9b7c27000 0x55d9b89f0b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:19.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.839+0000 7f65f31ca8c0 1 --2- 172.21.15.16:0/3093794445 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55d9b7c27400 0x55d9b89f0580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:19.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.839+0000 7f65f31ca8c0 1 --2- 172.21.15.16:0/3093794445 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d9b7c27800 0x55d9b89f0000 unknown :-1 s=CLOSED pgs=1257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:19.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.839+0000 7f65f31ca8c0 1 -- 172.21.15.16:0/3093794445 >> 172.21.15.16:0/3093794445 conn(0x55d9b7c27c00 msgr2=0x55d9b7c53800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:11:19.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.839+0000 7f65f31ca8c0 1 -- 172.21.15.16:0/3093794445 shutdown_connections 2024-03-31T15:11:19.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:19.839+0000 7f65f31ca8c0 1 -- 172.21.15.16:0/3093794445 wait complete. 2024-03-31T15:11:19.843 DEBUG:tasks.cephfs.filesystem:Ran ['cephfs-data-scan', '--debug-mds=20', '--debug-ms=1', '--debug-objecter=1', 'scan_links'] in time 0:00:00.119092, result: 2024-03-31T15:11:19.843 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs joinable true 2024-03-31T15:11:20.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:20.031+0000 7f2d8a800640 1 Processor -- start 2024-03-31T15:11:20.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:20.031+0000 7f2d8a800640 1 -- start start 2024-03-31T15:11:20.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:20.035+0000 7f2d8a800640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f2d8406d4d0 0x7f2d8406d930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:20.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:20.035+0000 7f2d8a800640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f2d84071550 0x7f2d84071960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:20.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:20.035+0000 7f2d8a800640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2d8406de70 con 0x7f2d841567e0 2024-03-31T15:11:20.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:20.035+0000 7f2d8a800640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f2d8415c7e0 con 0x7f2d84071550 2024-03-31T15:11:20.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:20.035+0000 7f2d8a800640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f2d840736b0 con 0x7f2d8406d4d0 2024-03-31T15:11:20.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:20.035+0000 7f2d83400640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f2d841567e0 0x7f2d84072c80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:33724/0 (socket says 172.21.15.16:33724) 2024-03-31T15:11:20.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:20.035+0000 7f2d83400640 1 -- 172.21.15.16:0/2288543322 learned_addr learned my addr 172.21.15.16:0/2288543322 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:11:20.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:20.035+0000 7f2d83e00640 1 --2- 172.21.15.16:0/2288543322 >> v2:172.21.15.16:3300/0 conn(0x7f2d84071550 0x7f2d84071960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:20.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:20.035+0000 7f2d7be00640 1 --2- 172.21.15.16:0/2288543322 >> v2:172.21.15.89:3300/0 conn(0x7f2d8406d4d0 0x7f2d8406d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:20.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:20.035+0000 7f2d82a00640 1 -- 172.21.15.16:0/2288543322 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1249023818 0 0) 0x7f2d8406de70 con 0x7f2d841567e0 2024-03-31T15:11:20.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:20.035+0000 7f2d82a00640 1 -- 172.21.15.16:0/2288543322 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2d68003530 con 0x7f2d841567e0 2024-03-31T15:11:20.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:20.035+0000 7f2d83e00640 1 -- 172.21.15.16:0/2288543322 >> v2:172.21.15.89:3300/0 conn(0x7f2d8406d4d0 msgr2=0x7f2d8406d930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:11:20.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:20.035+0000 7f2d83e00640 1 --2- 172.21.15.16:0/2288543322 >> v2:172.21.15.89:3300/0 conn(0x7f2d8406d4d0 0x7f2d8406d930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:20.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:20.035+0000 7f2d83e00640 1 -- 172.21.15.16:0/2288543322 >> v1:172.21.15.16:6789/0 conn(0x7f2d841567e0 legacy=0x7f2d84072c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:11:20.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:20.035+0000 7f2d83e00640 1 -- 172.21.15.16:0/2288543322 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d8415caa0 con 0x7f2d84071550 2024-03-31T15:11:20.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:20.035+0000 7f2d83e00640 1 --2- 172.21.15.16:0/2288543322 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2d84071550 0x7f2d84071960 secure :-1 s=READY pgs=1261 cs=0 l=1 rev1=1 crypto rx=0x7f2d8406c2e0 tx=0x7f2d7c01d390 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cdf5fbd368fb6964 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:11:20.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:20.035+0000 7f2d82a00640 1 -- 172.21.15.16:0/2288543322 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d7c0b2070 con 0x7f2d84071550 2024-03-31T15:11:20.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:20.035+0000 7f2d82a00640 1 -- 172.21.15.16:0/2288543322 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2d7c0a5900 con 0x7f2d84071550 2024-03-31T15:11:20.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:20.035+0000 7f2d82a00640 1 -- 172.21.15.16:0/2288543322 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d7c0a5cd0 con 0x7f2d84071550 2024-03-31T15:11:20.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:20.039+0000 7f2d8a800640 1 -- 172.21.15.16:0/2288543322 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2d84071550 msgr2=0x7f2d84071960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:11:20.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:20.039+0000 7f2d8a800640 1 --2- 172.21.15.16:0/2288543322 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2d84071550 0x7f2d84071960 secure :-1 s=READY pgs=1261 cs=0 l=1 rev1=1 crypto rx=0x7f2d8406c2e0 tx=0x7f2d7c01d390 comp rx=0 tx=0).stop 2024-03-31T15:11:20.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:20.039+0000 7f2d8a800640 1 -- 172.21.15.16:0/2288543322 shutdown_connections 2024-03-31T15:11:20.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:20.039+0000 7f2d8a800640 1 --2- 172.21.15.16:0/2288543322 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2d84071550 0x7f2d84071960 unknown :-1 s=CLOSED pgs=1261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:20.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:20.039+0000 7f2d8a800640 1 --2- 172.21.15.16:0/2288543322 >> v2:172.21.15.89:3300/0 conn(0x7f2d8406d4d0 0x7f2d8406d930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:20.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:20.039+0000 7f2d8a800640 1 -- 172.21.15.16:0/2288543322 >> 172.21.15.16:0/2288543322 conn(0x7f2d8406ee40 msgr2=0x7f2d8406b620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:11:20.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:20.039+0000 7f2d8a800640 1 -- 172.21.15.16:0/2288543322 shutdown_connections 2024-03-31T15:11:20.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:20.039+0000 7f2d8a800640 1 -- 172.21.15.16:0/2288543322 wait complete. 2024-03-31T15:11:20.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:20.039+0000 7f2d8a800640 1 Processor -- start 2024-03-31T15:11:20.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:20.039+0000 7f2d8a800640 1 -- start start 2024-03-31T15:11:20.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:20.039+0000 7f2d8a800640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f2d8406d4d0 0x7f2d841faa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:20.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:20.039+0000 7f2d8a800640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2d84071550 0x7f2d84148100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:20.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:20.039+0000 7f2d8a800640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2d841567e0 0x7f2d8415fbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:20.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:20.039+0000 7f2d8a800640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f2d840afbb0 con 0x7f2d84071550 2024-03-31T15:11:20.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:20.039+0000 7f2d8a800640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f2d840afa30 con 0x7f2d841567e0 2024-03-31T15:11:20.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:20.039+0000 7f2d8a800640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f2d840722b0 con 0x7f2d8406d4d0 2024-03-31T15:11:20.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:20.039+0000 7f2d7be00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2d84071550 0x7f2d84148100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:20.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:20.039+0000 7f2d7be00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2d84071550 0x7f2d84148100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56240/0 (socket says 172.21.15.16:56240) 2024-03-31T15:11:20.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:20.039+0000 7f2d7be00640 1 -- 172.21.15.16:0/1635752737 learned_addr learned my addr 172.21.15.16:0/1635752737 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:11:20.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:20.039+0000 7f2d83400640 1 --2- 172.21.15.16:0/1635752737 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f2d8406d4d0 0x7f2d841faa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:20.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:20.039+0000 7f2d83e00640 1 --2- 172.21.15.16:0/1635752737 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2d841567e0 0x7f2d8415fbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:20.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:20.043+0000 7f2d7be00640 1 -- 172.21.15.16:0/1635752737 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f2d8406d4d0 msgr2=0x7f2d841faa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:11:20.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:20.043+0000 7f2d7be00640 1 --2- 172.21.15.16:0/1635752737 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f2d8406d4d0 0x7f2d841faa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:20.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:20.043+0000 7f2d7be00640 1 -- 172.21.15.16:0/1635752737 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2d841567e0 msgr2=0x7f2d8415fbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:11:20.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:20.043+0000 7f2d7be00640 1 --2- 172.21.15.16:0/1635752737 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2d841567e0 0x7f2d8415fbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:20.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:20.043+0000 7f2d7be00640 1 -- 172.21.15.16:0/1635752737 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d841fafd0 con 0x7f2d84071550 2024-03-31T15:11:20.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:20.043+0000 7f2d7be00640 1 --2- 172.21.15.16:0/1635752737 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2d84071550 0x7f2d84148100 secure :-1 s=READY pgs=1262 cs=0 l=1 rev1=1 crypto rx=0x7f2d6c00d830 tx=0x7f2d6c00dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:11:20.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:20.043+0000 7f2d7b400640 1 -- 172.21.15.16:0/1635752737 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d6c00b6c0 con 0x7f2d84071550 2024-03-31T15:11:20.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:20.043+0000 7f2d7b400640 1 -- 172.21.15.16:0/1635752737 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2d6c019070 con 0x7f2d84071550 2024-03-31T15:11:20.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:20.043+0000 7f2d7b400640 1 -- 172.21.15.16:0/1635752737 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d6c00ba60 con 0x7f2d84071550 2024-03-31T15:11:20.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:20.043+0000 7f2d8a800640 1 -- 172.21.15.16:0/1635752737 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2d841fb270 con 0x7f2d84071550 2024-03-31T15:11:20.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:20.043+0000 7f2d8a800640 1 -- 172.21.15.16:0/1635752737 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2d841fb830 con 0x7f2d84071550 2024-03-31T15:11:20.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:20.043+0000 7f2d8a800640 1 -- 172.21.15.16:0/1635752737 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2d84069950 con 0x7f2d84071550 2024-03-31T15:11:20.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:20.043+0000 7f2d7b400640 1 -- 172.21.15.16:0/1635752737 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f2d6c01e460 con 0x7f2d84071550 2024-03-31T15:11:20.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:20.043+0000 7f2d7b400640 1 --2- 172.21.15.16:0/1635752737 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f2d60068380 0x7f2d6006a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:20.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:20.043+0000 7f2d7b400640 1 -- 172.21.15.16:0/1635752737 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 5910+0+0 (secure 0 0 0) 0x7f2d6c086fe0 con 0x7f2d84071550 2024-03-31T15:11:20.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:20.043+0000 7f2d83400640 1 --2- 172.21.15.16:0/1635752737 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f2d60068380 0x7f2d6006a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:20.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:20.047+0000 7f2d83400640 1 --2- 172.21.15.16:0/1635752737 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f2d60068380 0x7f2d6006a840 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f2d74007fd0 tx=0x7f2d74009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:11:20.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:20.047+0000 7f2d7b400640 1 -- 172.21.15.16:0/1635752737 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f2d6c056eb0 con 0x7f2d84071550 2024-03-31T15:11:20.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:20.247+0000 7f2d8a800640 1 -- 172.21.15.16:0/1635752737 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"} v 0) -- 0x7f2d8406e470 con 0x7f2d84071550 2024-03-31T15:11:20.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:20.403+0000 7f2d7b400640 1 -- 172.21.15.16:0/1635752737 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]=0 cephfs marked joinable; MDS may join as newly active. v70) ==== 162+0+0 (secure 0 0 0) 0x7f2d6c05b1b0 con 0x7f2d84071550 2024-03-31T15:11:20.396 INFO:teuthology.orchestra.run.smithi016.stderr:cephfs marked joinable; MDS may join as newly active. 2024-03-31T15:11:20.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:20.403+0000 7f2d8a800640 1 -- 172.21.15.16:0/1635752737 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f2d60068380 msgr2=0x7f2d6006a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:11:20.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:20.403+0000 7f2d8a800640 1 --2- 172.21.15.16:0/1635752737 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f2d60068380 0x7f2d6006a840 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f2d74007fd0 tx=0x7f2d74009040 comp rx=0 tx=0).stop 2024-03-31T15:11:20.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:20.403+0000 7f2d8a800640 1 -- 172.21.15.16:0/1635752737 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2d84071550 msgr2=0x7f2d84148100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:11:20.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:20.403+0000 7f2d8a800640 1 --2- 172.21.15.16:0/1635752737 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2d84071550 0x7f2d84148100 secure :-1 s=READY pgs=1262 cs=0 l=1 rev1=1 crypto rx=0x7f2d6c00d830 tx=0x7f2d6c00dcf0 comp rx=0 tx=0).stop 2024-03-31T15:11:20.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:20.403+0000 7f2d8a800640 1 -- 172.21.15.16:0/1635752737 shutdown_connections 2024-03-31T15:11:20.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:20.403+0000 7f2d8a800640 1 --2- 172.21.15.16:0/1635752737 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f2d60068380 0x7f2d6006a840 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:20.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:20.403+0000 7f2d8a800640 1 --2- 172.21.15.16:0/1635752737 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2d841567e0 0x7f2d8415fbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:20.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:20.403+0000 7f2d8a800640 1 --2- 172.21.15.16:0/1635752737 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2d84071550 0x7f2d84148100 unknown :-1 s=CLOSED pgs=1262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:20.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:20.403+0000 7f2d8a800640 1 --2- 172.21.15.16:0/1635752737 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f2d8406d4d0 0x7f2d841faa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:20.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:20.403+0000 7f2d8a800640 1 -- 172.21.15.16:0/1635752737 >> 172.21.15.16:0/1635752737 conn(0x7f2d8406ee40 msgr2=0x7f2d840b0ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:11:20.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:20.407+0000 7f2d8a800640 1 -- 172.21.15.16:0/1635752737 shutdown_connections 2024-03-31T15:11:20.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:20.407+0000 7f2d8a800640 1 -- 172.21.15.16:0/1635752737 wait complete. 2024-03-31T15:11:20.422 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-31T15:11:20.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:20.571+0000 7f1f29000640 1 Processor -- start 2024-03-31T15:11:20.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:20.571+0000 7f1f29000640 1 -- start start 2024-03-31T15:11:20.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:20.571+0000 7f1f29000640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f1f24145e40 0x7f1f240cc740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:20.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:20.571+0000 7f1f29000640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f1f240ccc80 0x7f1f2406f920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:20.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:20.571+0000 7f1f29000640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1f240d0920 con 0x7f1f2406fe60 2024-03-31T15:11:20.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:20.571+0000 7f1f29000640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f1f240d06d0 con 0x7f1f240ccc80 2024-03-31T15:11:20.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:20.571+0000 7f1f29000640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f1f240d0c00 con 0x7f1f24145e40 2024-03-31T15:11:20.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:20.571+0000 7f1f21400640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f1f240ccc80 0x7f1f2406f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:20.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:20.571+0000 7f1f21400640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f1f240ccc80 0x7f1f2406f920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56252/0 (socket says 172.21.15.16:56252) 2024-03-31T15:11:20.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:20.571+0000 7f1f21400640 1 -- 172.21.15.16:0/1267044062 learned_addr learned my addr 172.21.15.16:0/1267044062 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:11:20.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:20.571+0000 7f1f21e00640 1 --2- 172.21.15.16:0/1267044062 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f1f24145e40 0x7f1f240cc740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:20.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:20.571+0000 7f1f21400640 1 -- 172.21.15.16:0/1267044062 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f1f24145e40 msgr2=0x7f1f240cc740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:11:20.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:20.571+0000 7f1f21400640 1 --2- 172.21.15.16:0/1267044062 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f1f24145e40 0x7f1f240cc740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:20.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:20.571+0000 7f1f21400640 1 -- 172.21.15.16:0/1267044062 >> v1:172.21.15.89:6789/0 conn(0x7f1f2406fe60 legacy=0x7f1f2415fdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:11:20.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:20.571+0000 7f1f21400640 1 -- 172.21.15.16:0/1267044062 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f240d1810 con 0x7f1f240ccc80 2024-03-31T15:11:20.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:20.575+0000 7f1f21400640 1 --2- 172.21.15.16:0/1267044062 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1f240ccc80 0x7f1f2406f920 secure :-1 s=READY pgs=1263 cs=0 l=1 rev1=1 crypto rx=0x7f1f10009870 tx=0x7f1f1001c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15e337f2194f5a0e server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:11:20.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:20.575+0000 7f1f20a00640 1 -- 172.21.15.16:0/1267044062 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f100b0070 con 0x7f1f240ccc80 2024-03-31T15:11:20.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:20.575+0000 7f1f20a00640 1 -- 172.21.15.16:0/1267044062 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1f100ab600 con 0x7f1f240ccc80 2024-03-31T15:11:20.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:20.575+0000 7f1f20a00640 1 -- 172.21.15.16:0/1267044062 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f100ab9d0 con 0x7f1f240ccc80 2024-03-31T15:11:20.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:20.575+0000 7f1f29000640 1 -- 172.21.15.16:0/1267044062 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1f240ccc80 msgr2=0x7f1f2406f920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:11:20.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:20.575+0000 7f1f29000640 1 --2- 172.21.15.16:0/1267044062 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1f240ccc80 0x7f1f2406f920 secure :-1 s=READY pgs=1263 cs=0 l=1 rev1=1 crypto rx=0x7f1f10009870 tx=0x7f1f1001c520 comp rx=0 tx=0).stop 2024-03-31T15:11:20.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:20.575+0000 7f1f29000640 1 -- 172.21.15.16:0/1267044062 shutdown_connections 2024-03-31T15:11:20.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:20.575+0000 7f1f29000640 1 --2- 172.21.15.16:0/1267044062 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1f240ccc80 0x7f1f2406f920 unknown :-1 s=CLOSED pgs=1263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:20.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:20.575+0000 7f1f29000640 1 --2- 172.21.15.16:0/1267044062 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f1f24145e40 0x7f1f240cc740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:20.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:20.575+0000 7f1f29000640 1 -- 172.21.15.16:0/1267044062 >> 172.21.15.16:0/1267044062 conn(0x7f1f24067330 msgr2=0x7f1f24067740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:11:20.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:20.575+0000 7f1f29000640 1 -- 172.21.15.16:0/1267044062 shutdown_connections 2024-03-31T15:11:20.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:20.575+0000 7f1f29000640 1 -- 172.21.15.16:0/1267044062 wait complete. 2024-03-31T15:11:20.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:20.579+0000 7f1f29000640 1 Processor -- start 2024-03-31T15:11:20.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:20.579+0000 7f1f29000640 1 -- start start 2024-03-31T15:11:20.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:20.579+0000 7f1f29000640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f1f2406fe60 0x7f1f2406dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:20.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:20.579+0000 7f1f29000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1f240ccc80 0x7f1f241413f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:20.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:20.579+0000 7f1f29000640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1f24145e40 0x7f1f240b5f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:20.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:20.579+0000 7f1f29000640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f1f240b5240 con 0x7f1f240ccc80 2024-03-31T15:11:20.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:20.579+0000 7f1f29000640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f1f240b50c0 con 0x7f1f24145e40 2024-03-31T15:11:20.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:20.579+0000 7f1f29000640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f1f24169190 con 0x7f1f2406fe60 2024-03-31T15:11:20.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:20.579+0000 7f1f21400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1f240ccc80 0x7f1f241413f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:20.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:20.579+0000 7f1f21400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1f240ccc80 0x7f1f241413f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56254/0 (socket says 172.21.15.16:56254) 2024-03-31T15:11:20.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:20.579+0000 7f1f21400640 1 -- 172.21.15.16:0/847461127 learned_addr learned my addr 172.21.15.16:0/847461127 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:11:20.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:20.579+0000 7f1f21e00640 1 --2- 172.21.15.16:0/847461127 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f1f2406fe60 0x7f1f2406dbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:20.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:20.579+0000 7f1f22800640 1 --2- 172.21.15.16:0/847461127 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1f24145e40 0x7f1f240b5f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:20.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:20.579+0000 7f1f21400640 1 -- 172.21.15.16:0/847461127 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f1f2406fe60 msgr2=0x7f1f2406dbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:11:20.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:20.579+0000 7f1f21400640 1 --2- 172.21.15.16:0/847461127 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f1f2406fe60 0x7f1f2406dbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:20.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:20.579+0000 7f1f21400640 1 -- 172.21.15.16:0/847461127 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1f24145e40 msgr2=0x7f1f240b5f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:11:20.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:20.579+0000 7f1f21400640 1 --2- 172.21.15.16:0/847461127 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1f24145e40 0x7f1f240b5f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:20.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:20.579+0000 7f1f21400640 1 -- 172.21.15.16:0/847461127 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f240b6810 con 0x7f1f240ccc80 2024-03-31T15:11:20.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:20.579+0000 7f1f21400640 1 --2- 172.21.15.16:0/847461127 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1f240ccc80 0x7f1f241413f0 secure :-1 s=READY pgs=1264 cs=0 l=1 rev1=1 crypto rx=0x7f1f100ae040 tx=0x7f1f100192c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:11:20.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:20.579+0000 7f1f1a000640 1 -- 172.21.15.16:0/847461127 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f100b0040 con 0x7f1f240ccc80 2024-03-31T15:11:20.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:20.579+0000 7f1f1a000640 1 -- 172.21.15.16:0/847461127 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1f100b9070 con 0x7f1f240ccc80 2024-03-31T15:11:20.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:20.579+0000 7f1f1a000640 1 -- 172.21.15.16:0/847461127 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f100b4680 con 0x7f1f240ccc80 2024-03-31T15:11:20.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:20.579+0000 7f1f29000640 1 -- 172.21.15.16:0/847461127 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1f240b6af0 con 0x7f1f240ccc80 2024-03-31T15:11:20.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:20.579+0000 7f1f29000640 1 -- 172.21.15.16:0/847461127 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1f240b70b0 con 0x7f1f240ccc80 2024-03-31T15:11:20.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:20.579+0000 7f1f29000640 1 -- 172.21.15.16:0/847461127 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1f240b73e0 con 0x7f1f240ccc80 2024-03-31T15:11:20.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:20.583+0000 7f1f1a000640 1 -- 172.21.15.16:0/847461127 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f1f100b4820 con 0x7f1f240ccc80 2024-03-31T15:11:20.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:20.583+0000 7f1f1a000640 1 --2- 172.21.15.16:0/847461127 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f1f04068380 0x7f1f0406a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:20.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:20.583+0000 7f1f1a000640 1 -- 172.21.15.16:0/847461127 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 5910+0+0 (secure 0 0 0) 0x7f1f10120700 con 0x7f1f240ccc80 2024-03-31T15:11:20.577 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:20.583+0000 7f1f21e00640 1 --2- 172.21.15.16:0/847461127 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f1f04068380 0x7f1f0406a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:20.577 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:20.583+0000 7f1f21e00640 1 --2- 172.21.15.16:0/847461127 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f1f04068380 0x7f1f0406a840 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f1f241454c0 tx=0x7f1f14008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:11:20.580 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:20.587+0000 7f1f1a000640 1 -- 172.21.15.16:0/847461127 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f1f10016110 con 0x7f1f240ccc80 2024-03-31T15:11:20.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:20.783+0000 7f1f29000640 1 -- 172.21.15.16:0/847461127 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1f24141930 con 0x7f1f240ccc80 2024-03-31T15:11:20.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:20.787+0000 7f1f1a000640 1 -- 172.21.15.16:0/847461127 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 71 v71) ==== 94+0+5177 (secure 0 0 0) 0x7f1f100f0500 con 0x7f1f240ccc80 2024-03-31T15:11:20.779 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:11:20.779 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":71,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6494,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/2445558494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6835","nonce":2445558494},{"type":"v1","addr":"172.21.15.89:6837","nonce":2445558494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":56},{"gid":7270,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/3185913858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6833","nonce":3185913858},{"type":"v1","addr":"172.21.15.16:6835","nonce":3185913858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":56},{"gid":7279,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6836/1599468493","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":1599468493},{"type":"v1","addr":"172.21.15.89:6836","nonce":1599468493}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":56}],"filesystems":[{"mdsmap":{"epoch":71,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-31T15:09:50.926196+0000","modified":"2024-03-31T15:11:20.409418+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":85,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8085},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8085":{"gid":8085,"name":"c","rank":0,"incarnation":71,"state":"up:replay","state_seq":1,"addr":"172.21.15.16:6834/3780866829","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":3780866829},{"type":"v1","addr":"172.21.15.16:6834","nonce":3780866829}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-03-31T15:11:20.779 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 71 2024-03-31T15:11:20.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:20.787+0000 7f1f29000640 1 -- 172.21.15.16:0/847461127 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f1f04068380 msgr2=0x7f1f0406a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:11:20.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:20.787+0000 7f1f29000640 1 --2- 172.21.15.16:0/847461127 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f1f04068380 0x7f1f0406a840 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f1f241454c0 tx=0x7f1f14008040 comp rx=0 tx=0).stop 2024-03-31T15:11:20.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:20.787+0000 7f1f29000640 1 -- 172.21.15.16:0/847461127 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1f240ccc80 msgr2=0x7f1f241413f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:11:20.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:20.787+0000 7f1f29000640 1 --2- 172.21.15.16:0/847461127 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1f240ccc80 0x7f1f241413f0 secure :-1 s=READY pgs=1264 cs=0 l=1 rev1=1 crypto rx=0x7f1f100ae040 tx=0x7f1f100192c0 comp rx=0 tx=0).stop 2024-03-31T15:11:20.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:20.787+0000 7f1f29000640 1 -- 172.21.15.16:0/847461127 shutdown_connections 2024-03-31T15:11:20.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:20.787+0000 7f1f29000640 1 --2- 172.21.15.16:0/847461127 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f1f04068380 0x7f1f0406a840 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:20.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:20.787+0000 7f1f29000640 1 --2- 172.21.15.16:0/847461127 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1f24145e40 0x7f1f240b5f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:20.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:20.787+0000 7f1f29000640 1 --2- 172.21.15.16:0/847461127 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1f240ccc80 0x7f1f241413f0 unknown :-1 s=CLOSED pgs=1264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:20.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:20.787+0000 7f1f29000640 1 --2- 172.21.15.16:0/847461127 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f1f2406fe60 0x7f1f2406dbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:20.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:20.787+0000 7f1f29000640 1 -- 172.21.15.16:0/847461127 >> 172.21.15.16:0/847461127 conn(0x7f1f24067330 msgr2=0x7f1f240744f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:11:20.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:20.787+0000 7f1f29000640 1 -- 172.21.15.16:0/847461127 shutdown_connections 2024-03-31T15:11:20.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:20.787+0000 7f1f29000640 1 -- 172.21.15.16:0/847461127 wait complete. 2024-03-31T15:11:21.203 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 71, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-31T15:09:50.926196+0000', 'modified': '2024-03-31T15:11:20.409418+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 85, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8085}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8085': {'gid': 8085, 'name': 'c', 'rank': 0, 'incarnation': 71, 'state': 'up:replay', 'state_seq': 1, 'addr': '172.21.15.16:6834/3780866829', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6832', 'nonce': 3780866829}, {'type': 'v1', 'addr': '172.21.15.16:6834', 'nonce': 3780866829}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [13], 'metadata_pool': 12, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-03-31T15:11:21.204 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8085:up:replay 2024-03-31T15:11:22.205 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-31T15:11:22.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:22.363+0000 7f9c08600640 1 Processor -- start 2024-03-31T15:11:22.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:22.363+0000 7f9c08600640 1 -- start start 2024-03-31T15:11:22.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:22.367+0000 7f9c08600640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f9c000728b0 0x7f9c000b1cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:22.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:22.367+0000 7f9c08600640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9c00160e70 con 0x7f9c00145d10 2024-03-31T15:11:22.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:22.367+0000 7f9c08600640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9c001602f0 con 0x7f9c000b2200 2024-03-31T15:11:22.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:22.367+0000 7f9c08600640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f9c001605d0 con 0x7f9c000728b0 2024-03-31T15:11:22.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:22.367+0000 7f9c05200640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f9c000728b0 0x7f9c000b1cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:22.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:22.367+0000 7f9c05200640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f9c000728b0 0x7f9c000b1cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56268/0 (socket says 172.21.15.16:56268) 2024-03-31T15:11:22.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:22.367+0000 7f9c05200640 1 -- 172.21.15.16:0/3849923342 learned_addr learned my addr 172.21.15.16:0/3849923342 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:11:22.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:22.367+0000 7f9c05c00640 1 --1- 172.21.15.16:0/3849923342 >> v1:172.21.15.16:6789/0 conn(0x7f9c00145d10 0x7f9c000cf790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:33730/0 (socket says 172.21.15.16:33730) 2024-03-31T15:11:22.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:22.367+0000 7f9c05200640 1 -- 172.21.15.16:0/3849923342 >> v1:172.21.15.89:6789/0 conn(0x7f9c000b2200 legacy=0x7f9c0015cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:11:22.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:22.367+0000 7f9c05200640 1 -- 172.21.15.16:0/3849923342 >> v1:172.21.15.16:6789/0 conn(0x7f9c00145d10 legacy=0x7f9c000cf790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:11:22.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:22.367+0000 7f9c05200640 1 -- 172.21.15.16:0/3849923342 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9c001608b0 con 0x7f9c000728b0 2024-03-31T15:11:22.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:22.367+0000 7f9bffe00640 1 -- 172.21.15.16:0/3849923342 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2298449550 0 0) 0x7f9c00160e70 con 0x7f9c00145d10 2024-03-31T15:11:22.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:22.371+0000 7f9c05200640 1 --2- 172.21.15.16:0/3849923342 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9c000728b0 0x7f9c000b1cc0 secure :-1 s=READY pgs=1266 cs=0 l=1 rev1=1 crypto rx=0x7f9bec009870 tx=0x7f9bec01c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b2fd0879389a5eb server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:11:22.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:22.371+0000 7f9bffe00640 1 -- 172.21.15.16:0/3849923342 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bec0b0070 con 0x7f9c000728b0 2024-03-31T15:11:22.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:22.371+0000 7f9bffe00640 1 -- 172.21.15.16:0/3849923342 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9bec0ab600 con 0x7f9c000728b0 2024-03-31T15:11:22.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:22.371+0000 7f9bffe00640 1 -- 172.21.15.16:0/3849923342 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bec0ab9d0 con 0x7f9c000728b0 2024-03-31T15:11:22.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:22.371+0000 7f9c08600640 1 -- 172.21.15.16:0/3849923342 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9c000728b0 msgr2=0x7f9c000b1cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:11:22.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:22.371+0000 7f9c08600640 1 --2- 172.21.15.16:0/3849923342 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9c000728b0 0x7f9c000b1cc0 secure :-1 s=READY pgs=1266 cs=0 l=1 rev1=1 crypto rx=0x7f9bec009870 tx=0x7f9bec01c520 comp rx=0 tx=0).stop 2024-03-31T15:11:22.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:22.371+0000 7f9c08600640 1 -- 172.21.15.16:0/3849923342 shutdown_connections 2024-03-31T15:11:22.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:22.371+0000 7f9c08600640 1 --2- 172.21.15.16:0/3849923342 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9c000728b0 0x7f9c000b1cc0 unknown :-1 s=CLOSED pgs=1266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:22.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:22.371+0000 7f9c08600640 1 -- 172.21.15.16:0/3849923342 >> 172.21.15.16:0/3849923342 conn(0x7f9c0006b620 msgr2=0x7f9c0006ba30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:11:22.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:22.371+0000 7f9c08600640 1 -- 172.21.15.16:0/3849923342 shutdown_connections 2024-03-31T15:11:22.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:22.371+0000 7f9c08600640 1 -- 172.21.15.16:0/3849923342 wait complete. 2024-03-31T15:11:22.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:22.371+0000 7f9c08600640 1 Processor -- start 2024-03-31T15:11:22.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:22.371+0000 7f9c08600640 1 -- start start 2024-03-31T15:11:22.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:22.371+0000 7f9c08600640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9c000728b0 0x7f9c0013ea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:22.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:22.371+0000 7f9c08600640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9c000b2200 0x7f9c000b3b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:22.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:22.371+0000 7f9c08600640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f9c00145d10 0x7f9c001412e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:22.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:22.371+0000 7f9c08600640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f9c000cf4d0 con 0x7f9c000728b0 2024-03-31T15:11:22.368 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:22.371+0000 7f9c08600640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f9c000cf350 con 0x7f9c000b2200 2024-03-31T15:11:22.368 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:22.371+0000 7f9c08600640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f9c00072d40 con 0x7f9c00145d10 2024-03-31T15:11:22.368 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:22.375+0000 7f9c05c00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9c000728b0 0x7f9c0013ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:22.368 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:22.375+0000 7f9c05c00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9c000728b0 0x7f9c0013ea40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56280/0 (socket says 172.21.15.16:56280) 2024-03-31T15:11:22.368 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:22.375+0000 7f9c05c00640 1 -- 172.21.15.16:0/1611191499 learned_addr learned my addr 172.21.15.16:0/1611191499 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:11:22.368 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:22.375+0000 7f9c05200640 1 --2- 172.21.15.16:0/1611191499 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9c000b2200 0x7f9c000b3b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:22.368 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:22.375+0000 7f9c06600640 1 --2- 172.21.15.16:0/1611191499 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f9c00145d10 0x7f9c001412e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:22.368 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:22.375+0000 7f9c05c00640 1 -- 172.21.15.16:0/1611191499 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f9c00145d10 msgr2=0x7f9c001412e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:11:22.368 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:22.375+0000 7f9c05c00640 1 --2- 172.21.15.16:0/1611191499 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f9c00145d10 0x7f9c001412e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:22.368 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:22.375+0000 7f9c05c00640 1 -- 172.21.15.16:0/1611191499 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9c000b2200 msgr2=0x7f9c000b3b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:11:22.368 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:22.375+0000 7f9c05c00640 1 --2- 172.21.15.16:0/1611191499 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9c000b2200 0x7f9c000b3b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:22.368 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:22.375+0000 7f9c05c00640 1 -- 172.21.15.16:0/1611191499 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9c000b42e0 con 0x7f9c000728b0 2024-03-31T15:11:22.368 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:22.375+0000 7f9c06600640 1 --2- 172.21.15.16:0/1611191499 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f9c00145d10 0x7f9c001412e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:11:22.368 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:22.375+0000 7f9c05c00640 1 --2- 172.21.15.16:0/1611191499 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9c000728b0 0x7f9c0013ea40 secure :-1 s=READY pgs=1267 cs=0 l=1 rev1=1 crypto rx=0x7f9bf400c480 tx=0x7f9bf400c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:11:22.368 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:22.375+0000 7f9bfd600640 1 -- 172.21.15.16:0/1611191499 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bf4018070 con 0x7f9c000728b0 2024-03-31T15:11:22.369 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:22.375+0000 7f9bfd600640 1 -- 172.21.15.16:0/1611191499 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9bf4007500 con 0x7f9c000728b0 2024-03-31T15:11:22.369 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:22.375+0000 7f9bfd600640 1 -- 172.21.15.16:0/1611191499 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bf4004a30 con 0x7f9c000728b0 2024-03-31T15:11:22.369 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:22.375+0000 7f9c08600640 1 -- 172.21.15.16:0/1611191499 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9c000adb20 con 0x7f9c000728b0 2024-03-31T15:11:22.369 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:22.375+0000 7f9c08600640 1 -- 172.21.15.16:0/1611191499 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9c000ae0e0 con 0x7f9c000728b0 2024-03-31T15:11:22.369 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:22.375+0000 7f9c08600640 1 -- 172.21.15.16:0/1611191499 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9c00069950 con 0x7f9c000728b0 2024-03-31T15:11:22.369 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:22.375+0000 7f9bfd600640 1 -- 172.21.15.16:0/1611191499 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f9bf4004bd0 con 0x7f9c000728b0 2024-03-31T15:11:22.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:22.379+0000 7f9bfd600640 1 --2- 172.21.15.16:0/1611191499 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f9be0068380 0x7f9be006a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:22.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:22.379+0000 7f9bfd600640 1 -- 172.21.15.16:0/1611191499 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 5910+0+0 (secure 0 0 0) 0x7f9bf4085cb0 con 0x7f9c000728b0 2024-03-31T15:11:22.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:22.379+0000 7f9c05200640 1 --2- 172.21.15.16:0/1611191499 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f9be0068380 0x7f9be006a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:22.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:22.379+0000 7f9c05200640 1 --2- 172.21.15.16:0/1611191499 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f9be0068380 0x7f9be006a840 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f9bec0ad000 tx=0x7f9bec008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:11:22.375 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:22.383+0000 7f9bfd600640 1 -- 172.21.15.16:0/1611191499 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f9bf4055b40 con 0x7f9c000728b0 2024-03-31T15:11:22.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:22.587+0000 7f9c08600640 1 -- 172.21.15.16:0/1611191499 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9c00073a60 con 0x7f9c000728b0 2024-03-31T15:11:22.580 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:22.587+0000 7f9bfd600640 1 -- 172.21.15.16:0/1611191499 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 73 v73) ==== 94+0+5177 (secure 0 0 0) 0x7f9bf4059e40 con 0x7f9c000728b0 2024-03-31T15:11:22.580 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:11:22.580 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":73,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6494,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/2445558494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6835","nonce":2445558494},{"type":"v1","addr":"172.21.15.89:6837","nonce":2445558494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":56},{"gid":7270,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/3185913858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6833","nonce":3185913858},{"type":"v1","addr":"172.21.15.16:6835","nonce":3185913858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":56},{"gid":7279,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6836/1599468493","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":1599468493},{"type":"v1","addr":"172.21.15.89:6836","nonce":1599468493}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":56}],"filesystems":[{"mdsmap":{"epoch":73,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-31T15:09:50.926196+0000","modified":"2024-03-31T15:11:21.417905+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":85,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8085},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8085":{"gid":8085,"name":"c","rank":0,"incarnation":71,"state":"up:rejoin","state_seq":3,"addr":"172.21.15.16:6834/3780866829","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":3780866829},{"type":"v1","addr":"172.21.15.16:6834","nonce":3780866829}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-03-31T15:11:22.582 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 73 2024-03-31T15:11:22.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:22.587+0000 7f9c08600640 1 -- 172.21.15.16:0/1611191499 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f9be0068380 msgr2=0x7f9be006a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:11:22.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:22.587+0000 7f9c08600640 1 --2- 172.21.15.16:0/1611191499 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f9be0068380 0x7f9be006a840 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f9bec0ad000 tx=0x7f9bec008040 comp rx=0 tx=0).stop 2024-03-31T15:11:22.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:22.587+0000 7f9c08600640 1 -- 172.21.15.16:0/1611191499 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9c000728b0 msgr2=0x7f9c0013ea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:11:22.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:22.587+0000 7f9c08600640 1 --2- 172.21.15.16:0/1611191499 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9c000728b0 0x7f9c0013ea40 secure :-1 s=READY pgs=1267 cs=0 l=1 rev1=1 crypto rx=0x7f9bf400c480 tx=0x7f9bf400c940 comp rx=0 tx=0).stop 2024-03-31T15:11:22.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:22.587+0000 7f9c08600640 1 -- 172.21.15.16:0/1611191499 shutdown_connections 2024-03-31T15:11:22.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:22.587+0000 7f9c08600640 1 --2- 172.21.15.16:0/1611191499 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f9be0068380 0x7f9be006a840 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:22.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:22.587+0000 7f9c08600640 1 --2- 172.21.15.16:0/1611191499 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f9c00145d10 0x7f9c001412e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:22.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:22.587+0000 7f9c08600640 1 --2- 172.21.15.16:0/1611191499 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9c000b2200 0x7f9c000b3b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:22.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:22.587+0000 7f9c08600640 1 --2- 172.21.15.16:0/1611191499 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9c000728b0 0x7f9c0013ea40 unknown :-1 s=CLOSED pgs=1267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:22.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:22.587+0000 7f9c08600640 1 -- 172.21.15.16:0/1611191499 >> 172.21.15.16:0/1611191499 conn(0x7f9c0006b620 msgr2=0x7f9c000b00e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:11:22.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:22.587+0000 7f9c08600640 1 -- 172.21.15.16:0/1611191499 shutdown_connections 2024-03-31T15:11:22.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:22.587+0000 7f9c08600640 1 -- 172.21.15.16:0/1611191499 wait complete. 2024-03-31T15:11:23.003 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 73, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-31T15:09:50.926196+0000', 'modified': '2024-03-31T15:11:21.417905+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 85, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8085}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8085': {'gid': 8085, 'name': 'c', 'rank': 0, 'incarnation': 71, 'state': 'up:rejoin', 'state_seq': 3, 'addr': '172.21.15.16:6834/3780866829', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6832', 'nonce': 3780866829}, {'type': 'v1', 'addr': '172.21.15.16:6834', 'nonce': 3780866829}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [13], 'metadata_pool': 12, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-03-31T15:11:23.004 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8085:up:rejoin 2024-03-31T15:11:24.005 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-31T15:11:24.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:24.151+0000 7fd07a400640 1 Processor -- start 2024-03-31T15:11:24.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:24.151+0000 7fd07a400640 1 -- start start 2024-03-31T15:11:24.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:24.155+0000 7fd07a400640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fd0740cd750 0x7fd0741693a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:24.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:24.155+0000 7fd07a400640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd074162010 con 0x7fd07406d490 2024-03-31T15:11:24.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:24.155+0000 7fd07a400640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd0741622b0 con 0x7fd07406f920 2024-03-31T15:11:24.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:24.155+0000 7fd07a400640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fd074162be0 con 0x7fd0740cd750 2024-03-31T15:11:24.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:24.155+0000 7fd072a00640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fd07406d490 0x7fd07406d8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:33742/0 (socket says 172.21.15.16:33742) 2024-03-31T15:11:24.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:24.155+0000 7fd072a00640 1 -- 172.21.15.16:0/3698036929 learned_addr learned my addr 172.21.15.16:0/3698036929 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:11:24.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:24.155+0000 7fd073e00640 1 --2- 172.21.15.16:0/3698036929 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fd0740cd750 0x7fd0741693a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:24.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:24.155+0000 7fd072000640 1 -- 172.21.15.16:0/3698036929 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2100951934 0 0) 0x7fd074162010 con 0x7fd07406d490 2024-03-31T15:11:24.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:24.155+0000 7fd072000640 1 -- 172.21.15.16:0/3698036929 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd058003530 con 0x7fd07406d490 2024-03-31T15:11:24.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:24.155+0000 7fd072000640 1 -- 172.21.15.16:0/3698036929 <== mon.1 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3564931651 0 0) 0x7fd0741622b0 con 0x7fd07406f920 2024-03-31T15:11:24.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:24.155+0000 7fd072000640 1 -- 172.21.15.16:0/3698036929 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd074162010 con 0x7fd07406f920 2024-03-31T15:11:24.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:24.155+0000 7fd073e00640 1 -- 172.21.15.16:0/3698036929 >> v1:172.21.15.89:6789/0 conn(0x7fd07406f920 legacy=0x7fd07406fd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:11:24.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:24.155+0000 7fd073e00640 1 -- 172.21.15.16:0/3698036929 >> v1:172.21.15.16:6789/0 conn(0x7fd07406d490 legacy=0x7fd07406d8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:11:24.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:24.155+0000 7fd073e00640 1 -- 172.21.15.16:0/3698036929 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd074162590 con 0x7fd0740cd750 2024-03-31T15:11:24.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:24.159+0000 7fd073e00640 1 --2- 172.21.15.16:0/3698036929 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fd0740cd750 0x7fd0741693a0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fd06c009dc0 tx=0x7fd06c01c990 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a7f77651af16a18a server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:11:24.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:24.159+0000 7fd072000640 1 -- 172.21.15.16:0/3698036929 <== mon.2 v2:172.21.15.89:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd06c0b2070 con 0x7fd0740cd750 2024-03-31T15:11:24.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:24.159+0000 7fd072000640 1 -- 172.21.15.16:0/3698036929 <== mon.2 v2:172.21.15.89:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd06c0a5850 con 0x7fd0740cd750 2024-03-31T15:11:24.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:24.159+0000 7fd072000640 1 -- 172.21.15.16:0/3698036929 <== mon.2 v2:172.21.15.89:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd06c0a5c00 con 0x7fd0740cd750 2024-03-31T15:11:24.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:24.159+0000 7fd07a400640 1 -- 172.21.15.16:0/3698036929 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fd0740cd750 msgr2=0x7fd0741693a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:11:24.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:24.159+0000 7fd07a400640 1 --2- 172.21.15.16:0/3698036929 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fd0740cd750 0x7fd0741693a0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fd06c009dc0 tx=0x7fd06c01c990 comp rx=0 tx=0).stop 2024-03-31T15:11:24.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:24.159+0000 7fd07a400640 1 -- 172.21.15.16:0/3698036929 shutdown_connections 2024-03-31T15:11:24.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:24.159+0000 7fd07a400640 1 --2- 172.21.15.16:0/3698036929 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fd0740cd750 0x7fd0741693a0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:24.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:24.159+0000 7fd07a400640 1 -- 172.21.15.16:0/3698036929 >> 172.21.15.16:0/3698036929 conn(0x7fd074067330 msgr2=0x7fd074067740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:11:24.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:24.159+0000 7fd07a400640 1 -- 172.21.15.16:0/3698036929 shutdown_connections 2024-03-31T15:11:24.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:24.159+0000 7fd07a400640 1 -- 172.21.15.16:0/3698036929 wait complete. 2024-03-31T15:11:24.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:24.159+0000 7fd07a400640 1 Processor -- start 2024-03-31T15:11:24.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:24.159+0000 7fd07a400640 1 -- start start 2024-03-31T15:11:24.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:24.159+0000 7fd07a400640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd07406d490 0x7fd07413f5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:24.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:24.159+0000 7fd07a400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd07406f920 0x7fd07415c160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:24.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:24.159+0000 7fd07a400640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fd0740cd750 0x7fd07413dea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:24.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:24.159+0000 7fd07a400640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fd07416c8a0 con 0x7fd07406f920 2024-03-31T15:11:24.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:24.159+0000 7fd07a400640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fd07416c720 con 0x7fd07406d490 2024-03-31T15:11:24.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:24.159+0000 7fd07a400640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fd07416ca20 con 0x7fd0740cd750 2024-03-31T15:11:24.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:24.163+0000 7fd072a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd07406f920 0x7fd07415c160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:24.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:24.163+0000 7fd072a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd07406f920 0x7fd07415c160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56286/0 (socket says 172.21.15.16:56286) 2024-03-31T15:11:24.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:24.163+0000 7fd072a00640 1 -- 172.21.15.16:0/676261387 learned_addr learned my addr 172.21.15.16:0/676261387 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:11:24.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:24.163+0000 7fd073e00640 1 --2- 172.21.15.16:0/676261387 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fd0740cd750 0x7fd07413dea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:24.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:24.163+0000 7fd073400640 1 --2- 172.21.15.16:0/676261387 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd07406d490 0x7fd07413f5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:24.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:24.163+0000 7fd072a00640 1 -- 172.21.15.16:0/676261387 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fd0740cd750 msgr2=0x7fd07413dea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:11:24.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:24.163+0000 7fd072a00640 1 --2- 172.21.15.16:0/676261387 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fd0740cd750 0x7fd07413dea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:24.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:24.163+0000 7fd072a00640 1 -- 172.21.15.16:0/676261387 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd07406d490 msgr2=0x7fd07413f5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:11:24.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:24.163+0000 7fd072a00640 1 --2- 172.21.15.16:0/676261387 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd07406d490 0x7fd07413f5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:24.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:24.163+0000 7fd072a00640 1 -- 172.21.15.16:0/676261387 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0740b6210 con 0x7fd07406f920 2024-03-31T15:11:24.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:24.163+0000 7fd073400640 1 --2- 172.21.15.16:0/676261387 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd07406d490 0x7fd07413f5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:11:24.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:24.163+0000 7fd072a00640 1 --2- 172.21.15.16:0/676261387 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd07406f920 0x7fd07415c160 secure :-1 s=READY pgs=1269 cs=0 l=1 rev1=1 crypto rx=0x7fd06000c430 tx=0x7fd06000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:11:24.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:24.163+0000 7fd06b400640 1 -- 172.21.15.16:0/676261387 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd060017070 con 0x7fd07406f920 2024-03-31T15:11:24.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:24.163+0000 7fd06b400640 1 -- 172.21.15.16:0/676261387 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd060005db0 con 0x7fd07406f920 2024-03-31T15:11:24.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:24.163+0000 7fd07a400640 1 -- 172.21.15.16:0/676261387 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd0740b64b0 con 0x7fd07406f920 2024-03-31T15:11:24.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:24.163+0000 7fd06b400640 1 -- 172.21.15.16:0/676261387 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0600072f0 con 0x7fd07406f920 2024-03-31T15:11:24.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:24.163+0000 7fd07a400640 1 -- 172.21.15.16:0/676261387 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd074146e90 con 0x7fd07406f920 2024-03-31T15:11:24.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:24.163+0000 7fd07a400640 1 -- 172.21.15.16:0/676261387 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd074069950 con 0x7fd07406f920 2024-03-31T15:11:24.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:24.163+0000 7fd06b400640 1 -- 172.21.15.16:0/676261387 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fd06001a070 con 0x7fd07406f920 2024-03-31T15:11:24.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:24.163+0000 7fd06b400640 1 --2- 172.21.15.16:0/676261387 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fd054068380 0x7fd05406a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:24.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:24.167+0000 7fd06b400640 1 -- 172.21.15.16:0/676261387 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 5910+0+0 (secure 0 0 0) 0x7fd060085e20 con 0x7fd07406f920 2024-03-31T15:11:24.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:24.167+0000 7fd073400640 1 --2- 172.21.15.16:0/676261387 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fd054068380 0x7fd05406a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:24.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:24.167+0000 7fd073400640 1 --2- 172.21.15.16:0/676261387 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fd054068380 0x7fd05406a840 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7fd064001020 tx=0x7fd064009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:11:24.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:24.175+0000 7fd06b400640 1 -- 172.21.15.16:0/676261387 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fd060055c40 con 0x7fd07406f920 2024-03-31T15:11:24.368 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:24.375+0000 7fd07a400640 1 -- 172.21.15.16:0/676261387 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd0740cf3f0 con 0x7fd07406f920 2024-03-31T15:11:24.368 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:24.375+0000 7fd06b400640 1 -- 172.21.15.16:0/676261387 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 74 v74) ==== 94+0+5184 (secure 0 0 0) 0x7fd060059f40 con 0x7fd07406f920 2024-03-31T15:11:24.369 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:11:24.369 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":74,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6494,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/2445558494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6835","nonce":2445558494},{"type":"v1","addr":"172.21.15.89:6837","nonce":2445558494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":56},{"gid":7270,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/3185913858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6833","nonce":3185913858},{"type":"v1","addr":"172.21.15.16:6835","nonce":3185913858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":56},{"gid":7279,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6836/1599468493","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":1599468493},{"type":"v1","addr":"172.21.15.89:6836","nonce":1599468493}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":56}],"filesystems":[{"mdsmap":{"epoch":74,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-31T15:09:50.926196+0000","modified":"2024-03-31T15:11:23.417783+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":85,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8085},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8085":{"gid":8085,"name":"c","rank":0,"incarnation":71,"state":"up:active","state_seq":4,"addr":"172.21.15.16:6834/3780866829","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":3780866829},{"type":"v1","addr":"172.21.15.16:6834","nonce":3780866829}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8085,"qdb_cluster":[8085]},"id":6}]} 2024-03-31T15:11:24.369 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 74 2024-03-31T15:11:24.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:24.379+0000 7fd07a400640 1 -- 172.21.15.16:0/676261387 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fd054068380 msgr2=0x7fd05406a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:11:24.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:24.379+0000 7fd07a400640 1 --2- 172.21.15.16:0/676261387 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fd054068380 0x7fd05406a840 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7fd064001020 tx=0x7fd064009040 comp rx=0 tx=0).stop 2024-03-31T15:11:24.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:24.379+0000 7fd07a400640 1 -- 172.21.15.16:0/676261387 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd07406f920 msgr2=0x7fd07415c160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:11:24.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:24.379+0000 7fd07a400640 1 --2- 172.21.15.16:0/676261387 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd07406f920 0x7fd07415c160 secure :-1 s=READY pgs=1269 cs=0 l=1 rev1=1 crypto rx=0x7fd06000c430 tx=0x7fd06000c8f0 comp rx=0 tx=0).stop 2024-03-31T15:11:24.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:24.379+0000 7fd07a400640 1 -- 172.21.15.16:0/676261387 shutdown_connections 2024-03-31T15:11:24.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:24.379+0000 7fd07a400640 1 --2- 172.21.15.16:0/676261387 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fd054068380 0x7fd05406a840 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:24.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:24.379+0000 7fd07a400640 1 --2- 172.21.15.16:0/676261387 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fd0740cd750 0x7fd07413dea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:24.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:24.379+0000 7fd07a400640 1 --2- 172.21.15.16:0/676261387 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd07406f920 0x7fd07415c160 unknown :-1 s=CLOSED pgs=1269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:24.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:24.379+0000 7fd07a400640 1 --2- 172.21.15.16:0/676261387 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd07406d490 0x7fd07413f5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:24.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:24.379+0000 7fd07a400640 1 -- 172.21.15.16:0/676261387 >> 172.21.15.16:0/676261387 conn(0x7fd074067330 msgr2=0x7fd07406fd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:11:24.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:24.379+0000 7fd07a400640 1 -- 172.21.15.16:0/676261387 shutdown_connections 2024-03-31T15:11:24.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:24.379+0000 7fd07a400640 1 -- 172.21.15.16:0/676261387 wait complete. 2024-03-31T15:11:24.792 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 74, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-31T15:09:50.926196+0000', 'modified': '2024-03-31T15:11:23.417783+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 85, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8085}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8085': {'gid': 8085, 'name': 'c', 'rank': 0, 'incarnation': 71, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.16:6834/3780866829', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6832', 'nonce': 3780866829}, {'type': 'v1', 'addr': '172.21.15.16:6834', 'nonce': 3780866829}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [13], 'metadata_pool': 12, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8085, 'qdb_cluster': [8085]} 2024-03-31T15:11:24.792 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-31T15:11:24.793 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-03-31T15:11:25.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.015+0000 7f6dc0800640 1 Processor -- start 2024-03-31T15:11:25.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.015+0000 7f6dc0800640 1 -- start start 2024-03-31T15:11:25.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.019+0000 7f6dc0800640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f6db8147e70 0x7f6db806b620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:25.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.019+0000 7f6dc0800640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f6db80736e0 0x7f6db815eb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:25.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.019+0000 7f6dc0800640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6db81621b0 con 0x7f6db80d0c50 2024-03-31T15:11:25.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.019+0000 7f6dc0800640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f6db8162490 con 0x7f6db80736e0 2024-03-31T15:11:25.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.019+0000 7f6dc0800640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f6db8162770 con 0x7f6db8147e70 2024-03-31T15:11:25.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.019+0000 7f6dbd400640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f6db80d0c50 0x7f6db8072fd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:33756/0 (socket says 172.21.15.16:33756) 2024-03-31T15:11:25.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.019+0000 7f6dbd400640 1 -- 172.21.15.16:0/2789337079 learned_addr learned my addr 172.21.15.16:0/2789337079 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:11:25.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.019+0000 7f6dbe800640 1 --2- 172.21.15.16:0/2789337079 >> v2:172.21.15.16:3300/0 conn(0x7f6db80736e0 0x7f6db815eb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:25.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.019+0000 7f6dbde00640 1 --2- 172.21.15.16:0/2789337079 >> v2:172.21.15.89:3300/0 conn(0x7f6db8147e70 0x7f6db806b620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:25.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.019+0000 7f6dbca00640 1 -- 172.21.15.16:0/2789337079 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2182070862 0 0) 0x7f6db81621b0 con 0x7f6db80d0c50 2024-03-31T15:11:25.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.019+0000 7f6dbca00640 1 -- 172.21.15.16:0/2789337079 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6da0003530 con 0x7f6db80d0c50 2024-03-31T15:11:25.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.019+0000 7f6dbe800640 1 -- 172.21.15.16:0/2789337079 >> v2:172.21.15.89:3300/0 conn(0x7f6db8147e70 msgr2=0x7f6db806b620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:11:25.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.019+0000 7f6dbe800640 1 --2- 172.21.15.16:0/2789337079 >> v2:172.21.15.89:3300/0 conn(0x7f6db8147e70 0x7f6db806b620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:25.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.019+0000 7f6dbe800640 1 -- 172.21.15.16:0/2789337079 >> v1:172.21.15.16:6789/0 conn(0x7f6db80d0c50 legacy=0x7f6db8072fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:11:25.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.019+0000 7f6dbe800640 1 -- 172.21.15.16:0/2789337079 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6db8162a50 con 0x7f6db80736e0 2024-03-31T15:11:25.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.019+0000 7f6dbde00640 1 --2- 172.21.15.16:0/2789337079 >> v2:172.21.15.89:3300/0 conn(0x7f6db8147e70 0x7f6db806b620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:11:25.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.023+0000 7f6dbe800640 1 --2- 172.21.15.16:0/2789337079 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6db80736e0 0x7f6db815eb20 secure :-1 s=READY pgs=1271 cs=0 l=1 rev1=1 crypto rx=0x7f6db8069860 tx=0x7f6db401c930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2619af7ca83a19de server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:11:25.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.023+0000 7f6dbca00640 1 -- 172.21.15.16:0/2789337079 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6db40b2070 con 0x7f6db80736e0 2024-03-31T15:11:25.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.023+0000 7f6dbca00640 1 -- 172.21.15.16:0/2789337079 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6db40a5870 con 0x7f6db80736e0 2024-03-31T15:11:25.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.023+0000 7f6dbca00640 1 -- 172.21.15.16:0/2789337079 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6db40a5c20 con 0x7f6db80736e0 2024-03-31T15:11:25.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.023+0000 7f6dc0800640 1 -- 172.21.15.16:0/2789337079 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6db80736e0 msgr2=0x7f6db815eb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:11:25.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.023+0000 7f6dc0800640 1 --2- 172.21.15.16:0/2789337079 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6db80736e0 0x7f6db815eb20 secure :-1 s=READY pgs=1271 cs=0 l=1 rev1=1 crypto rx=0x7f6db8069860 tx=0x7f6db401c930 comp rx=0 tx=0).stop 2024-03-31T15:11:25.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.023+0000 7f6dc0800640 1 -- 172.21.15.16:0/2789337079 shutdown_connections 2024-03-31T15:11:25.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.023+0000 7f6dc0800640 1 --2- 172.21.15.16:0/2789337079 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6db80736e0 0x7f6db815eb20 unknown :-1 s=CLOSED pgs=1271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:25.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.023+0000 7f6dc0800640 1 --2- 172.21.15.16:0/2789337079 >> v2:172.21.15.89:3300/0 conn(0x7f6db8147e70 0x7f6db806b620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:25.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.023+0000 7f6dc0800640 1 -- 172.21.15.16:0/2789337079 >> 172.21.15.16:0/2789337079 conn(0x7f6db80d1060 msgr2=0x7f6db80d1470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:11:25.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.023+0000 7f6dc0800640 1 -- 172.21.15.16:0/2789337079 shutdown_connections 2024-03-31T15:11:25.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.023+0000 7f6dc0800640 1 -- 172.21.15.16:0/2789337079 wait complete. 2024-03-31T15:11:25.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.027+0000 7f6dc0800640 1 Processor -- start 2024-03-31T15:11:25.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.027+0000 7f6dc0800640 1 -- start start 2024-03-31T15:11:25.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.027+0000 7f6dc0800640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f6db80736e0 0x7f6db813d850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:25.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.027+0000 7f6dc0800640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6db80d0c50 0x7f6db813dd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:25.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.027+0000 7f6dc0800640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6db8147e70 0x7f6db80b0c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:25.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.027+0000 7f6dc0800640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f6db815f870 con 0x7f6db80d0c50 2024-03-31T15:11:25.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.027+0000 7f6dc0800640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f6db815f9f0 con 0x7f6db8147e70 2024-03-31T15:11:25.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.027+0000 7f6dc0800640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f6db816b4b0 con 0x7f6db80736e0 2024-03-31T15:11:25.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.027+0000 7f6dbd400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6db80d0c50 0x7f6db813dd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:25.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.027+0000 7f6dbd400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6db80d0c50 0x7f6db813dd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56308/0 (socket says 172.21.15.16:56308) 2024-03-31T15:11:25.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.027+0000 7f6dbd400640 1 -- 172.21.15.16:0/2097424328 learned_addr learned my addr 172.21.15.16:0/2097424328 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:11:25.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.027+0000 7f6dbe800640 1 --2- 172.21.15.16:0/2097424328 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6db8147e70 0x7f6db80b0c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:25.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.027+0000 7f6dbde00640 1 --2- 172.21.15.16:0/2097424328 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f6db80736e0 0x7f6db813d850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:25.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.027+0000 7f6dbd400640 1 -- 172.21.15.16:0/2097424328 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f6db80736e0 msgr2=0x7f6db813d850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:11:25.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.027+0000 7f6dbd400640 1 --2- 172.21.15.16:0/2097424328 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f6db80736e0 0x7f6db813d850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:25.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.027+0000 7f6dbd400640 1 -- 172.21.15.16:0/2097424328 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6db8147e70 msgr2=0x7f6db80b0c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:11:25.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.027+0000 7f6dbd400640 1 --2- 172.21.15.16:0/2097424328 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6db8147e70 0x7f6db80b0c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:25.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.027+0000 7f6dbd400640 1 -- 172.21.15.16:0/2097424328 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6db815e5b0 con 0x7f6db80d0c50 2024-03-31T15:11:25.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.027+0000 7f6dbde00640 1 --2- 172.21.15.16:0/2097424328 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f6db80736e0 0x7f6db813d850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:11:25.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.027+0000 7f6dbd400640 1 --2- 172.21.15.16:0/2097424328 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6db80d0c50 0x7f6db813dd90 secure :-1 s=READY pgs=1272 cs=0 l=1 rev1=1 crypto rx=0x7f6da800c420 tx=0x7f6da800c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:11:25.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.027+0000 7f6db2000640 1 -- 172.21.15.16:0/2097424328 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6da8017070 con 0x7f6db80d0c50 2024-03-31T15:11:25.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.027+0000 7f6db2000640 1 -- 172.21.15.16:0/2097424328 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6da8005d10 con 0x7f6db80d0c50 2024-03-31T15:11:25.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.027+0000 7f6dc0800640 1 -- 172.21.15.16:0/2097424328 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6db813c450 con 0x7f6db80d0c50 2024-03-31T15:11:25.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.027+0000 7f6db2000640 1 -- 172.21.15.16:0/2097424328 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6da8007310 con 0x7f6db80d0c50 2024-03-31T15:11:25.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.027+0000 7f6dc0800640 1 -- 172.21.15.16:0/2097424328 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6db813ca10 con 0x7f6db80d0c50 2024-03-31T15:11:25.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.031+0000 7f6db2000640 1 -- 172.21.15.16:0/2097424328 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f6da801a070 con 0x7f6db80d0c50 2024-03-31T15:11:25.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.031+0000 7f6db2000640 1 --2- 172.21.15.16:0/2097424328 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f6d9c068380 0x7f6d9c06a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:25.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.031+0000 7f6db2000640 1 -- 172.21.15.16:0/2097424328 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 5910+0+0 (secure 0 0 0) 0x7f6da8085f10 con 0x7f6db80d0c50 2024-03-31T15:11:25.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.031+0000 7f6dbde00640 1 --2- 172.21.15.16:0/2097424328 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f6d9c068380 0x7f6d9c06a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:25.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.031+0000 7f6dc3d2a000 1 Processor -- start 2024-03-31T15:11:25.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.031+0000 7f6dc3d2a000 1 -- start start 2024-03-31T15:11:25.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.031+0000 7f6dbde00640 1 --2- 172.21.15.16:0/2097424328 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f6d9c068380 0x7f6d9c06a840 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f6dac0097a0 tx=0x7f6dac008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:11:25.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.031+0000 7f6dc3d2a000 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x56049b4414a0 0x56049b441880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:25.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.035+0000 7f6dc3d2a000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56049b441dc0 0x56049b3f4b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:25.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.035+0000 7f6dc3d2a000 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56049b3f50b0 0x56049b3f5490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:25.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.035+0000 7f6dc3d2a000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x56049b247950 con 0x56049b441dc0 2024-03-31T15:11:25.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.035+0000 7f6dc3d2a000 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x56049b1e4990 con 0x56049b3f50b0 2024-03-31T15:11:25.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.035+0000 7f6dc3d2a000 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x56049b243900 con 0x56049b4414a0 2024-03-31T15:11:25.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.035+0000 7f6dbd400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56049b441dc0 0x56049b3f4b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:25.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.035+0000 7f6dbd400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56049b441dc0 0x56049b3f4b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56316/0 (socket says 172.21.15.16:56316) 2024-03-31T15:11:25.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.035+0000 7f6dbd400640 1 -- 172.21.15.16:0/3214545343 learned_addr learned my addr 172.21.15.16:0/3214545343 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:11:25.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.035+0000 7f6dbde00640 1 --2- 172.21.15.16:0/3214545343 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x56049b4414a0 0x56049b441880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:25.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.035+0000 7f6dbe800640 1 --2- 172.21.15.16:0/3214545343 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56049b3f50b0 0x56049b3f5490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:25.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.035+0000 7f6dbd400640 1 -- 172.21.15.16:0/3214545343 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x56049b4414a0 msgr2=0x56049b441880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:11:25.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.035+0000 7f6dbd400640 1 --2- 172.21.15.16:0/3214545343 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x56049b4414a0 0x56049b441880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:25.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.035+0000 7f6dbd400640 1 -- 172.21.15.16:0/3214545343 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56049b3f50b0 msgr2=0x56049b3f5490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:11:25.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.035+0000 7f6dbd400640 1 --2- 172.21.15.16:0/3214545343 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56049b3f50b0 0x56049b3f5490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:25.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.035+0000 7f6dbd400640 1 -- 172.21.15.16:0/3214545343 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56049b28c100 con 0x56049b441dc0 2024-03-31T15:11:25.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.035+0000 7f6dbd400640 1 --2- 172.21.15.16:0/3214545343 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56049b441dc0 0x56049b3f4b70 secure :-1 s=READY pgs=1273 cs=0 l=1 rev1=1 crypto rx=0x7f6da80058b0 tx=0x7f6da800cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65d7afc2cf4c906 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:11:25.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.035+0000 7f6da6a00640 1 -- 172.21.15.16:0/3214545343 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6da8081070 con 0x56049b441dc0 2024-03-31T15:11:25.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.035+0000 7f6da6a00640 1 -- 172.21.15.16:0/3214545343 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6da8059070 con 0x56049b441dc0 2024-03-31T15:11:25.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.035+0000 7f6da6a00640 1 -- 172.21.15.16:0/3214545343 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6da8052420 con 0x56049b441dc0 2024-03-31T15:11:25.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.035+0000 7f6dc3d2a000 1 -- 172.21.15.16:0/3214545343 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56049b441dc0 msgr2=0x56049b3f4b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:11:25.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.035+0000 7f6dc3d2a000 1 --2- 172.21.15.16:0/3214545343 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56049b441dc0 0x56049b3f4b70 secure :-1 s=READY pgs=1273 cs=0 l=1 rev1=1 crypto rx=0x7f6da80058b0 tx=0x7f6da800cf90 comp rx=0 tx=0).stop 2024-03-31T15:11:25.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.035+0000 7f6dc3d2a000 1 -- 172.21.15.16:0/3214545343 shutdown_connections 2024-03-31T15:11:25.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.035+0000 7f6dc3d2a000 1 --2- 172.21.15.16:0/3214545343 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56049b3f50b0 0x56049b3f5490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:25.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.035+0000 7f6dc3d2a000 1 --2- 172.21.15.16:0/3214545343 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56049b441dc0 0x56049b3f4b70 unknown :-1 s=CLOSED pgs=1273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:25.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.035+0000 7f6dc3d2a000 1 --2- 172.21.15.16:0/3214545343 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x56049b4414a0 0x56049b441880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:25.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.035+0000 7f6dc3d2a000 1 -- 172.21.15.16:0/3214545343 >> 172.21.15.16:0/3214545343 conn(0x56049b2dfb80 msgr2=0x56049b424460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:11:25.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.035+0000 7f6dc3d2a000 1 -- 172.21.15.16:0/3214545343 shutdown_connections 2024-03-31T15:11:25.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.035+0000 7f6dc3d2a000 1 -- 172.21.15.16:0/3214545343 wait complete. 2024-03-31T15:11:25.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.035+0000 7f6dc3d2a000 1 -- start start 2024-03-31T15:11:25.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.035+0000 7f6dc3d2a000 1 Processor -- start 2024-03-31T15:11:25.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.035+0000 7f6dc3d2a000 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x56049b3f50b0 0x56049b4100f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:25.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.035+0000 7f6dc3d2a000 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56049b4414a0 0x56049b410630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:25.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.035+0000 7f6dc3d2a000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56049b441dc0 0x56049b410b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:25.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.035+0000 7f6dc3d2a000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x56049b246c60 con 0x56049b441dc0 2024-03-31T15:11:25.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.035+0000 7f6dc3d2a000 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x56049b2053e0 con 0x56049b4414a0 2024-03-31T15:11:25.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.035+0000 7f6dc3d2a000 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x56049b254060 con 0x56049b3f50b0 2024-03-31T15:11:25.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.035+0000 7f6dc3d2a000 10 client.0 _ll_register_callbacks cb 0x56049b2a80f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T15:11:25.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.035+0000 7f6dbe800640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56049b441dc0 0x56049b410b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:25.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.039+0000 7f6dbe800640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56049b441dc0 0x56049b410b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56324/0 (socket says 172.21.15.16:56324) 2024-03-31T15:11:25.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.039+0000 7f6dbe800640 1 -- 172.21.15.16:0/63531358 learned_addr learned my addr 172.21.15.16:0/63531358 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:11:25.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.039+0000 7f6dbde00640 1 --2- 172.21.15.16:0/63531358 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x56049b3f50b0 0x56049b4100f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:25.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.039+0000 7f6dbd400640 1 --2- 172.21.15.16:0/63531358 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56049b4414a0 0x56049b410630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:25.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.039+0000 7f6dbe800640 1 -- 172.21.15.16:0/63531358 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x56049b3f50b0 msgr2=0x56049b4100f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:11:25.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.039+0000 7f6dbe800640 1 --2- 172.21.15.16:0/63531358 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x56049b3f50b0 0x56049b4100f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:25.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.039+0000 7f6dbe800640 1 -- 172.21.15.16:0/63531358 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56049b4414a0 msgr2=0x56049b410630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:11:25.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.039+0000 7f6dbe800640 1 --2- 172.21.15.16:0/63531358 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56049b4414a0 0x56049b410630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:25.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.039+0000 7f6dbe800640 1 -- 172.21.15.16:0/63531358 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56049b2118c0 con 0x56049b441dc0 2024-03-31T15:11:25.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.039+0000 7f6dbe800640 1 --2- 172.21.15.16:0/63531358 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56049b441dc0 0x56049b410b70 secure :-1 s=READY pgs=1274 cs=0 l=1 rev1=1 crypto rx=0x7f6db401ce10 tx=0x7f6db401b170 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32c876d942c45ec3 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:11:25.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.039+0000 7f6d9b400640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-31T15:11:25.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.039+0000 7f6d9b400640 1 -- 172.21.15.16:0/63531358 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6db40b2040 con 0x56049b441dc0 2024-03-31T15:11:25.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.039+0000 7f6d9b400640 1 -- 172.21.15.16:0/63531358 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 5910+0+0 (secure 0 0 0) 0x7f6db40a5df0 con 0x56049b441dc0 2024-03-31T15:11:25.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.039+0000 7f6dc3d2a000 1 -- 172.21.15.16:0/63531358 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6da8059070 con 0x56049b441dc0 2024-03-31T15:11:25.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.039+0000 7f6d9b400640 1 -- 172.21.15.16:0/63531358 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=74) ==== 24+0+0 (secure 0 0 0) 0x7f6db40c9050 con 0x56049b441dc0 2024-03-31T15:11:25.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.039+0000 7f6dc3d2a000 10 client.8166 fetch_fsmap learned FSMap version 74 2024-03-31T15:11:25.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.039+0000 7f6dc3d2a000 1 -- 172.21.15.16:0/63531358 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=74,monmap=2+}) -- 0x56049b1cfe80 con 0x56049b441dc0 2024-03-31T15:11:25.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.039+0000 7f6d9b400640 1 -- 172.21.15.16:0/63531358 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6da8059070 con 0x56049b441dc0 2024-03-31T15:11:25.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.039+0000 7f6d9b400640 1 -- 172.21.15.16:0/63531358 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 74) ==== 3494+0+0 (secure 0 0 0) 0x7f6db40cc020 con 0x56049b441dc0 2024-03-31T15:11:25.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.039+0000 7f6dc3d2a000 10 client.8166 fetch_fsmap finished waiting for FSMap version 74 2024-03-31T15:11:25.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.039+0000 7f6dc3d2a000 10 client.8166 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-31T15:11:25.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.039+0000 7f6dc3d2a000 20 client.8166 populate_metadata read hostname 'smithi016' 2024-03-31T15:11:25.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.039+0000 7f6dc3d2a000 1 --2- 172.21.15.16:0/63531358 >> [v2:172.21.15.16:6832/3780866829,v1:172.21.15.16:6834/3780866829] conn(0x56049b411380 0x56049b453a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:25.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.039+0000 7f6dc3d2a000 4 client.8166 mds_command: new command op to 8085 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-31T15:11:25.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.039+0000 7f6dc3d2a000 1 -- 172.21.15.16:0/63531358 --> [v2:172.21.15.16:6832/3780866829,v1:172.21.15.16:6834/3780866829] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56049b1e2bc0 con 0x56049b411380 2024-03-31T15:11:25.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.039+0000 7f6dbde00640 1 --2- 172.21.15.16:0/63531358 >> [v2:172.21.15.16:6832/3780866829,v1:172.21.15.16:6834/3780866829] conn(0x56049b411380 0x56049b453a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:25.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.039+0000 7f6dbde00640 1 --2- 172.21.15.16:0/63531358 >> [v2:172.21.15.16:6832/3780866829,v1:172.21.15.16:6834/3780866829] conn(0x56049b411380 0x56049b453a20 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ef938da5c228fac server_cookie=e29fd341cb515007 in_seq=0 out_seq=0 2024-03-31T15:11:25.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.039+0000 7f6d9b400640 10 client.8166 ms_handle_connect on v2:172.21.15.16:6832/3780866829 2024-03-31T15:11:25.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.039+0000 7f6d9b400640 1 -- 172.21.15.16:0/63531358 <== mds.0 v2:172.21.15.16:6832/3780866829 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56049b1e2bc0 con 0x56049b411380 2024-03-31T15:11:25.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.039+0000 7f6d9b400640 10 client.8166 handle_command_reply: tid=0 2024-03-31T15:11:25.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.039+0000 7f6dc3d2a000 1 client.8166 shutdown 2024-03-31T15:11:25.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.039+0000 7f6dc3d2a000 1 -- 172.21.15.16:0/63531358 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56049b441dc0 msgr2=0x56049b410b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:11:25.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.039+0000 7f6dc3d2a000 1 --2- 172.21.15.16:0/63531358 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56049b441dc0 0x56049b410b70 secure :-1 s=READY pgs=1274 cs=0 l=1 rev1=1 crypto rx=0x7f6db401ce10 tx=0x7f6db401b170 comp rx=0 tx=0).stop 2024-03-31T15:11:25.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.039+0000 7f6dc3d2a000 1 -- 172.21.15.16:0/63531358 shutdown_connections 2024-03-31T15:11:25.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.039+0000 7f6dc3d2a000 1 --2- 172.21.15.16:0/63531358 >> [v2:172.21.15.16:6832/3780866829,v1:172.21.15.16:6834/3780866829] conn(0x56049b411380 0x56049b453a20 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:25.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.039+0000 7f6dc3d2a000 1 --2- 172.21.15.16:0/63531358 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56049b441dc0 0x56049b410b70 unknown :-1 s=CLOSED pgs=1274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:25.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.039+0000 7f6dc3d2a000 1 --2- 172.21.15.16:0/63531358 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56049b4414a0 0x56049b410630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:25.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.039+0000 7f6dc3d2a000 1 --2- 172.21.15.16:0/63531358 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x56049b3f50b0 0x56049b4100f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:25.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.039+0000 7f6d9b400640 0 client.8166 ms_handle_reset on v2:172.21.15.16:6832/3780866829 2024-03-31T15:11:25.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.039+0000 7f6dc3d2a000 1 -- 172.21.15.16:0/63531358 >> 172.21.15.16:0/63531358 conn(0x56049b2dfb80 msgr2=0x56049b3f4270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:11:25.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.039+0000 7f6dc3d2a000 1 -- 172.21.15.16:0/63531358 shutdown_connections 2024-03-31T15:11:25.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.039+0000 7f6dc3d2a000 1 -- 172.21.15.16:0/63531358 wait complete. 2024-03-31T15:11:25.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.043+0000 7f6dc3d2a000 20 client.8166 trim_cache size 0 max 16384 2024-03-31T15:11:25.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.059+0000 7f6dc3d2a000 1 Processor -- start 2024-03-31T15:11:25.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.059+0000 7f6dc3d2a000 1 -- start start 2024-03-31T15:11:25.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.059+0000 7f6dc3d2a000 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56049b3f50b0 0x56049b456e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:25.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.059+0000 7f6dc3d2a000 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x56049b4414a0 0x56049b44e200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:25.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.059+0000 7f6dc3d2a000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56049b441dc0 0x56049b44e740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:25.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.059+0000 7f6dc3d2a000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f6db8203740 con 0x56049b441dc0 2024-03-31T15:11:25.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.059+0000 7f6dc3d2a000 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f6db82027a0 con 0x56049b3f50b0 2024-03-31T15:11:25.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.059+0000 7f6dc3d2a000 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f6db8142df0 con 0x56049b4414a0 2024-03-31T15:11:25.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.059+0000 7f6dbe800640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56049b441dc0 0x56049b44e740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:25.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.059+0000 7f6dbe800640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56049b441dc0 0x56049b44e740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56334/0 (socket says 172.21.15.16:56334) 2024-03-31T15:11:25.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.059+0000 7f6dbe800640 1 -- 172.21.15.16:0/3548938979 learned_addr learned my addr 172.21.15.16:0/3548938979 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:11:25.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.059+0000 7f6dbd400640 1 --2- 172.21.15.16:0/3548938979 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x56049b4414a0 0x56049b44e200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:25.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.059+0000 7f6dbde00640 1 --2- 172.21.15.16:0/3548938979 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56049b3f50b0 0x56049b456e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:25.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.059+0000 7f6dbe800640 1 -- 172.21.15.16:0/3548938979 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x56049b4414a0 msgr2=0x56049b44e200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:11:25.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.059+0000 7f6dbe800640 1 --2- 172.21.15.16:0/3548938979 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x56049b4414a0 0x56049b44e200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:25.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.059+0000 7f6dbe800640 1 -- 172.21.15.16:0/3548938979 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56049b3f50b0 msgr2=0x56049b456e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:11:25.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.059+0000 7f6dbe800640 1 --2- 172.21.15.16:0/3548938979 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56049b3f50b0 0x56049b456e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:25.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.059+0000 7f6dbe800640 1 -- 172.21.15.16:0/3548938979 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56049b202490 con 0x56049b441dc0 2024-03-31T15:11:25.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.059+0000 7f6dbde00640 1 --2- 172.21.15.16:0/3548938979 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56049b3f50b0 0x56049b456e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:11:25.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.059+0000 7f6dbe800640 1 --2- 172.21.15.16:0/3548938979 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56049b441dc0 0x56049b44e740 secure :-1 s=READY pgs=1275 cs=0 l=1 rev1=1 crypto rx=0x7f6db4004a90 tx=0x7f6db40b9000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd189abe604f996f server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:11:25.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.059+0000 7f6da6000640 1 -- 172.21.15.16:0/3548938979 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6db40b2040 con 0x56049b441dc0 2024-03-31T15:11:25.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.059+0000 7f6da6000640 1 -- 172.21.15.16:0/3548938979 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6db40a5a10 con 0x56049b441dc0 2024-03-31T15:11:25.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.059+0000 7f6da6000640 1 -- 172.21.15.16:0/3548938979 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6db40a5e00 con 0x56049b441dc0 2024-03-31T15:11:25.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.063+0000 7f6dc3d2a000 1 -- 172.21.15.16:0/3548938979 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56049b441dc0 msgr2=0x56049b44e740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:11:25.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.063+0000 7f6dc3d2a000 1 --2- 172.21.15.16:0/3548938979 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56049b441dc0 0x56049b44e740 secure :-1 s=READY pgs=1275 cs=0 l=1 rev1=1 crypto rx=0x7f6db4004a90 tx=0x7f6db40b9000 comp rx=0 tx=0).stop 2024-03-31T15:11:25.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.063+0000 7f6dc3d2a000 1 -- 172.21.15.16:0/3548938979 shutdown_connections 2024-03-31T15:11:25.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.063+0000 7f6dc3d2a000 1 --2- 172.21.15.16:0/3548938979 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56049b441dc0 0x56049b44e740 unknown :-1 s=CLOSED pgs=1275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:25.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.063+0000 7f6dc3d2a000 1 --2- 172.21.15.16:0/3548938979 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x56049b4414a0 0x56049b44e200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:25.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.063+0000 7f6dc3d2a000 1 --2- 172.21.15.16:0/3548938979 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56049b3f50b0 0x56049b456e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:25.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.063+0000 7f6dc3d2a000 1 -- 172.21.15.16:0/3548938979 >> 172.21.15.16:0/3548938979 conn(0x56049b2dfb80 msgr2=0x56049b4237f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:11:25.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.063+0000 7f6dc3d2a000 1 -- 172.21.15.16:0/3548938979 shutdown_connections 2024-03-31T15:11:25.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.063+0000 7f6dc3d2a000 1 -- 172.21.15.16:0/3548938979 wait complete. 2024-03-31T15:11:25.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.063+0000 7f6dc3d2a000 1 -- start start 2024-03-31T15:11:25.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.063+0000 7f6dc3d2a000 1 Processor -- start 2024-03-31T15:11:25.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.063+0000 7f6dc3d2a000 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56049b3f50b0 0x56049b456d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:25.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.063+0000 7f6dc3d2a000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56049b4414a0 0x56049b453a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:25.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.063+0000 7f6dc3d2a000 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x56049b441dc0 0x56049b453f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:25.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.063+0000 7f6dc3d2a000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x56049b24bf80 con 0x56049b4414a0 2024-03-31T15:11:25.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.063+0000 7f6dc3d2a000 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f6db8143440 con 0x56049b3f50b0 2024-03-31T15:11:25.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.063+0000 7f6dc3d2a000 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x56049b1ede80 con 0x56049b441dc0 2024-03-31T15:11:25.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.063+0000 7f6dc3d2a000 10 client.0 _ll_register_callbacks cb 0x7f6db40b0070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T15:11:25.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.063+0000 7f6dbd400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56049b4414a0 0x56049b453a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:25.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.063+0000 7f6dbd400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56049b4414a0 0x56049b453a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56348/0 (socket says 172.21.15.16:56348) 2024-03-31T15:11:25.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.063+0000 7f6dbd400640 1 -- 172.21.15.16:0/834447746 learned_addr learned my addr 172.21.15.16:0/834447746 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:11:25.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.063+0000 7f6dbde00640 1 --2- 172.21.15.16:0/834447746 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56049b3f50b0 0x56049b456d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:25.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.063+0000 7f6dbe800640 1 --2- 172.21.15.16:0/834447746 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x56049b441dc0 0x56049b453f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:25.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.063+0000 7f6dbd400640 1 -- 172.21.15.16:0/834447746 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x56049b441dc0 msgr2=0x56049b453f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:11:25.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.063+0000 7f6dbd400640 1 --2- 172.21.15.16:0/834447746 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x56049b441dc0 0x56049b453f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:25.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.063+0000 7f6dbd400640 1 -- 172.21.15.16:0/834447746 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56049b3f50b0 msgr2=0x56049b456d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:11:25.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.063+0000 7f6dbd400640 1 --2- 172.21.15.16:0/834447746 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56049b3f50b0 0x56049b456d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:25.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.063+0000 7f6dbd400640 1 -- 172.21.15.16:0/834447746 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56049b200e80 con 0x56049b4414a0 2024-03-31T15:11:25.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.063+0000 7f6dbd400640 1 --2- 172.21.15.16:0/834447746 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56049b4414a0 0x56049b453a20 secure :-1 s=READY pgs=1276 cs=0 l=1 rev1=1 crypto rx=0x7f6da8055d40 tx=0x7f6da8000bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=712adaa524475f05 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:11:25.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.063+0000 7f6d9b400640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-31T15:11:25.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.063+0000 7f6d9b400640 1 -- 172.21.15.16:0/834447746 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6da8079070 con 0x56049b4414a0 2024-03-31T15:11:25.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.063+0000 7f6d9b400640 1 -- 172.21.15.16:0/834447746 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 5910+0+0 (secure 0 0 0) 0x7f6da80074b0 con 0x56049b4414a0 2024-03-31T15:11:25.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.063+0000 7f6dc3d2a000 1 -- 172.21.15.16:0/834447746 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6db40a5a10 con 0x56049b4414a0 2024-03-31T15:11:25.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.063+0000 7f6d9b400640 1 -- 172.21.15.16:0/834447746 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=74) ==== 24+0+0 (secure 0 0 0) 0x7f6da8081040 con 0x56049b4414a0 2024-03-31T15:11:25.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.063+0000 7f6dc3d2a000 10 client.8172 fetch_fsmap learned FSMap version 74 2024-03-31T15:11:25.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.063+0000 7f6dc3d2a000 1 -- 172.21.15.16:0/834447746 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=74,monmap=2+}) -- 0x56049b1ee270 con 0x56049b4414a0 2024-03-31T15:11:25.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.063+0000 7f6d9b400640 1 -- 172.21.15.16:0/834447746 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6db40a5a10 con 0x56049b4414a0 2024-03-31T15:11:25.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.063+0000 7f6d9b400640 1 -- 172.21.15.16:0/834447746 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 74) ==== 3494+0+0 (secure 0 0 0) 0x7f6da805c040 con 0x56049b4414a0 2024-03-31T15:11:25.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.063+0000 7f6dc3d2a000 10 client.8172 fetch_fsmap finished waiting for FSMap version 74 2024-03-31T15:11:25.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.063+0000 7f6dc3d2a000 10 client.8172 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-31T15:11:25.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.063+0000 7f6dc3d2a000 20 client.8172 populate_metadata read hostname 'smithi016' 2024-03-31T15:11:25.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.067+0000 7f6dc3d2a000 1 --2- 172.21.15.16:0/834447746 >> [v2:172.21.15.16:6832/3780866829,v1:172.21.15.16:6834/3780866829] conn(0x56049b424080 0x56049b4544a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:25.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.067+0000 7f6dc3d2a000 4 client.8172 mds_command: new command op to 8085 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-31T15:11:25.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.067+0000 7f6dc3d2a000 1 -- 172.21.15.16:0/834447746 --> [v2:172.21.15.16:6832/3780866829,v1:172.21.15.16:6834/3780866829] -- command(tid 0: {"prefix": "status"}) -- 0x56049b254e90 con 0x56049b424080 2024-03-31T15:11:25.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.067+0000 7f6dbde00640 1 --2- 172.21.15.16:0/834447746 >> [v2:172.21.15.16:6832/3780866829,v1:172.21.15.16:6834/3780866829] conn(0x56049b424080 0x56049b4544a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:25.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.067+0000 7f6dbde00640 1 --2- 172.21.15.16:0/834447746 >> [v2:172.21.15.16:6832/3780866829,v1:172.21.15.16:6834/3780866829] conn(0x56049b424080 0x56049b4544a0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=79883acad778a496 server_cookie=1a84f961360d578a in_seq=0 out_seq=0 2024-03-31T15:11:25.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.067+0000 7f6d9b400640 10 client.8172 ms_handle_connect on v2:172.21.15.16:6832/3780866829 2024-03-31T15:11:25.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.067+0000 7f6d9b400640 1 -- 172.21.15.16:0/834447746 <== mds.0 v2:172.21.15.16:6832/3780866829 1 ==== command_reply(tid 0: 0 ) ==== 8+0+331 (crc 0 0 0) 0x56049b254e90 con 0x56049b424080 2024-03-31T15:11:25.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.067+0000 7f6d9b400640 10 client.8172 handle_command_reply: tid=0 2024-03-31T15:11:25.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.067+0000 7f6dc3d2a000 1 client.8172 shutdown 2024-03-31T15:11:25.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.067+0000 7f6dc3d2a000 1 -- 172.21.15.16:0/834447746 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56049b4414a0 msgr2=0x56049b453a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:11:25.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.067+0000 7f6dc3d2a000 1 --2- 172.21.15.16:0/834447746 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56049b4414a0 0x56049b453a20 secure :-1 s=READY pgs=1276 cs=0 l=1 rev1=1 crypto rx=0x7f6da8055d40 tx=0x7f6da8000bd0 comp rx=0 tx=0).stop 2024-03-31T15:11:25.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.067+0000 7f6dc3d2a000 1 -- 172.21.15.16:0/834447746 shutdown_connections 2024-03-31T15:11:25.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.067+0000 7f6dc3d2a000 1 --2- 172.21.15.16:0/834447746 >> [v2:172.21.15.16:6832/3780866829,v1:172.21.15.16:6834/3780866829] conn(0x56049b424080 0x56049b4544a0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:25.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.067+0000 7f6dc3d2a000 1 --2- 172.21.15.16:0/834447746 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x56049b441dc0 0x56049b453f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:25.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.067+0000 7f6dc3d2a000 1 --2- 172.21.15.16:0/834447746 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56049b4414a0 0x56049b453a20 unknown :-1 s=CLOSED pgs=1276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:25.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.067+0000 7f6dc3d2a000 1 --2- 172.21.15.16:0/834447746 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56049b3f50b0 0x56049b456d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:25.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.067+0000 7f6dc3d2a000 1 -- 172.21.15.16:0/834447746 >> 172.21.15.16:0/834447746 conn(0x56049b2dfb80 msgr2=0x56049b44e200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:11:25.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.067+0000 7f6d9b400640 0 client.8172 ms_handle_reset on v2:172.21.15.16:6832/3780866829 2024-03-31T15:11:25.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.067+0000 7f6dc3d2a000 1 -- 172.21.15.16:0/834447746 shutdown_connections 2024-03-31T15:11:25.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.067+0000 7f6dc3d2a000 1 -- 172.21.15.16:0/834447746 wait complete. 2024-03-31T15:11:25.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.067+0000 7f6dc3d2a000 20 client.8172 trim_cache size 0 max 16384 2024-03-31T15:11:25.064 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-03-31T15:11:25.064 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_fsid": "286eb5da-fb47-4550-b290-d9519c13b9a5", 2024-03-31T15:11:25.064 INFO:teuthology.orchestra.run.smithi016.stdout: "whoami": 0, 2024-03-31T15:11:25.064 INFO:teuthology.orchestra.run.smithi016.stdout: "id": 8085, 2024-03-31T15:11:25.064 INFO:teuthology.orchestra.run.smithi016.stdout: "want_state": "up:active", 2024-03-31T15:11:25.064 INFO:teuthology.orchestra.run.smithi016.stdout: "state": "up:active", 2024-03-31T15:11:25.064 INFO:teuthology.orchestra.run.smithi016.stdout: "fs_name": "cephfs", 2024-03-31T15:11:25.065 INFO:teuthology.orchestra.run.smithi016.stdout: "rank_uptime": 4.6553872930000004, 2024-03-31T15:11:25.065 INFO:teuthology.orchestra.run.smithi016.stdout: "mdsmap_epoch": 74, 2024-03-31T15:11:25.065 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch": 85, 2024-03-31T15:11:25.065 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch_barrier": 85, 2024-03-31T15:11:25.065 INFO:teuthology.orchestra.run.smithi016.stdout: "uptime": 5.6002423869999998 2024-03-31T15:11:25.065 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-03-31T15:11:25.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.067+0000 7f6dc0800640 1 -- 172.21.15.16:0/2097424328 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f6d9c068380 msgr2=0x7f6d9c06a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:11:25.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.067+0000 7f6dc0800640 1 --2- 172.21.15.16:0/2097424328 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f6d9c068380 0x7f6d9c06a840 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f6dac0097a0 tx=0x7f6dac008040 comp rx=0 tx=0).stop 2024-03-31T15:11:25.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.067+0000 7f6dc0800640 1 -- 172.21.15.16:0/2097424328 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6db80d0c50 msgr2=0x7f6db813dd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:11:25.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.067+0000 7f6dc0800640 1 --2- 172.21.15.16:0/2097424328 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6db80d0c50 0x7f6db813dd90 secure :-1 s=READY pgs=1272 cs=0 l=1 rev1=1 crypto rx=0x7f6da800c420 tx=0x7f6da800c8e0 comp rx=0 tx=0).stop 2024-03-31T15:11:25.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.067+0000 7f6dc0800640 1 -- 172.21.15.16:0/2097424328 shutdown_connections 2024-03-31T15:11:25.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.067+0000 7f6dc0800640 1 --2- 172.21.15.16:0/2097424328 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f6d9c068380 0x7f6d9c06a840 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:25.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.067+0000 7f6dc0800640 1 --2- 172.21.15.16:0/2097424328 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6db8147e70 0x7f6db80b0c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:25.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.067+0000 7f6dc0800640 1 --2- 172.21.15.16:0/2097424328 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6db80d0c50 0x7f6db813dd90 unknown :-1 s=CLOSED pgs=1272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:25.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.067+0000 7f6dc0800640 1 --2- 172.21.15.16:0/2097424328 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f6db80736e0 0x7f6db813d850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:25.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.067+0000 7f6dc0800640 1 -- 172.21.15.16:0/2097424328 >> 172.21.15.16:0/2097424328 conn(0x7f6db80d1060 msgr2=0x7f6db806fb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:11:25.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.067+0000 7f6dc0800640 1 -- 172.21.15.16:0/2097424328 shutdown_connections 2024-03-31T15:11:25.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.067+0000 7f6dc0800640 1 -- 172.21.15.16:0/2097424328 wait complete. 2024-03-31T15:11:25.088 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-31T15:11:25.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.235+0000 7fba6d200640 1 Processor -- start 2024-03-31T15:11:25.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.235+0000 7fba6d200640 1 -- start start 2024-03-31T15:11:25.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.235+0000 7fba6d200640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fba6806f920 0x7fba6806fd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:25.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.235+0000 7fba6d200640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fba680d0920 con 0x7fba68145e40 2024-03-31T15:11:25.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.235+0000 7fba6d200640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fba680d0c00 con 0x7fba680d2010 2024-03-31T15:11:25.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.235+0000 7fba6d200640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7fba680702c0 con 0x7fba6806f920 2024-03-31T15:11:25.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.235+0000 7fba5d400640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fba6806f920 0x7fba6806fd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:25.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.235+0000 7fba5d400640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fba6806f920 0x7fba6806fd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56360/0 (socket says 172.21.15.16:56360) 2024-03-31T15:11:25.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.235+0000 7fba5d400640 1 -- 172.21.15.16:0/1610223197 learned_addr learned my addr 172.21.15.16:0/1610223197 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:11:25.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.235+0000 7fba65e00640 1 --1- 172.21.15.16:0/1610223197 >> v1:172.21.15.16:6789/0 conn(0x7fba68145e40 0x7fba680cc740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:33764/0 (socket says 172.21.15.16:33764) 2024-03-31T15:11:25.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.235+0000 7fba5d400640 1 -- 172.21.15.16:0/1610223197 >> v1:172.21.15.89:6789/0 conn(0x7fba680d2010 legacy=0x7fba680d24a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:11:25.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.235+0000 7fba5d400640 1 -- 172.21.15.16:0/1610223197 >> v1:172.21.15.16:6789/0 conn(0x7fba68145e40 legacy=0x7fba680cc740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:11:25.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.235+0000 7fba5d400640 1 -- 172.21.15.16:0/1610223197 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba680d1810 con 0x7fba6806f920 2024-03-31T15:11:25.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.239+0000 7fba5d400640 1 --2- 172.21.15.16:0/1610223197 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fba6806f920 0x7fba6806fd80 secure :-1 s=READY pgs=1278 cs=0 l=1 rev1=1 crypto rx=0x7fba50009870 tx=0x7fba5001c560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f64526189296f30 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:11:25.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.239+0000 7fba65400640 1 -- 172.21.15.16:0/1610223197 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba500b2070 con 0x7fba6806f920 2024-03-31T15:11:25.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.239+0000 7fba65400640 1 -- 172.21.15.16:0/1610223197 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fba500a9070 con 0x7fba6806f920 2024-03-31T15:11:25.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.239+0000 7fba65400640 1 -- 172.21.15.16:0/1610223197 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba500ad880 con 0x7fba6806f920 2024-03-31T15:11:25.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.239+0000 7fba6d200640 1 -- 172.21.15.16:0/1610223197 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fba6806f920 msgr2=0x7fba6806fd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:11:25.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.239+0000 7fba6d200640 1 --2- 172.21.15.16:0/1610223197 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fba6806f920 0x7fba6806fd80 secure :-1 s=READY pgs=1278 cs=0 l=1 rev1=1 crypto rx=0x7fba50009870 tx=0x7fba5001c560 comp rx=0 tx=0).stop 2024-03-31T15:11:25.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.239+0000 7fba6d200640 1 -- 172.21.15.16:0/1610223197 shutdown_connections 2024-03-31T15:11:25.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.239+0000 7fba6d200640 1 --2- 172.21.15.16:0/1610223197 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fba6806f920 0x7fba6806fd80 unknown :-1 s=CLOSED pgs=1278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:25.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.239+0000 7fba6d200640 1 -- 172.21.15.16:0/1610223197 >> 172.21.15.16:0/1610223197 conn(0x7fba68067330 msgr2=0x7fba68067740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:11:25.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.239+0000 7fba6d200640 1 -- 172.21.15.16:0/1610223197 shutdown_connections 2024-03-31T15:11:25.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.239+0000 7fba6d200640 1 -- 172.21.15.16:0/1610223197 wait complete. 2024-03-31T15:11:25.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.243+0000 7fba6d200640 1 Processor -- start 2024-03-31T15:11:25.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.243+0000 7fba6d200640 1 -- start start 2024-03-31T15:11:25.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.243+0000 7fba6d200640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fba6806f920 0x7fba6815cb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:25.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.243+0000 7fba6d200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fba680d2010 0x7fba6813ecf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:25.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.243+0000 7fba6d200640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fba68145e40 0x7fba68142790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:25.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.243+0000 7fba6d200640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fba680d0ee0 con 0x7fba680d2010 2024-03-31T15:11:25.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.243+0000 7fba6d200640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fba680d06d0 con 0x7fba68145e40 2024-03-31T15:11:25.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.243+0000 7fba6d200640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fba680b50c0 con 0x7fba6806f920 2024-03-31T15:11:25.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.243+0000 7fba5d400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fba680d2010 0x7fba6813ecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:25.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.243+0000 7fba5d400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fba680d2010 0x7fba6813ecf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56364/0 (socket says 172.21.15.16:56364) 2024-03-31T15:11:25.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.243+0000 7fba5d400640 1 -- 172.21.15.16:0/137204492 learned_addr learned my addr 172.21.15.16:0/137204492 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:11:25.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.243+0000 7fba65e00640 1 --2- 172.21.15.16:0/137204492 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fba6806f920 0x7fba6815cb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:25.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.243+0000 7fba66800640 1 --2- 172.21.15.16:0/137204492 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fba68145e40 0x7fba68142790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:25.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.243+0000 7fba5d400640 1 -- 172.21.15.16:0/137204492 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fba6806f920 msgr2=0x7fba6815cb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:11:25.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.243+0000 7fba5d400640 1 --2- 172.21.15.16:0/137204492 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fba6806f920 0x7fba6815cb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:25.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.243+0000 7fba5d400640 1 -- 172.21.15.16:0/137204492 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fba68145e40 msgr2=0x7fba68142790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:11:25.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.243+0000 7fba5d400640 1 --2- 172.21.15.16:0/137204492 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fba68145e40 0x7fba68142790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:25.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.243+0000 7fba5d400640 1 -- 172.21.15.16:0/137204492 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba68167fc0 con 0x7fba680d2010 2024-03-31T15:11:25.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.247+0000 7fba5d400640 1 --2- 172.21.15.16:0/137204492 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fba680d2010 0x7fba6813ecf0 secure :-1 s=READY pgs=1279 cs=0 l=1 rev1=1 crypto rx=0x7fba500b0040 tx=0x7fba5001ada0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:11:25.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.247+0000 7fba5ea00640 1 -- 172.21.15.16:0/137204492 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba500b2040 con 0x7fba680d2010 2024-03-31T15:11:25.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.247+0000 7fba5ea00640 1 -- 172.21.15.16:0/137204492 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fba500be070 con 0x7fba680d2010 2024-03-31T15:11:25.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.247+0000 7fba5ea00640 1 -- 172.21.15.16:0/137204492 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba500b8660 con 0x7fba680d2010 2024-03-31T15:11:25.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.247+0000 7fba6d200640 1 -- 172.21.15.16:0/137204492 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fba6815f4c0 con 0x7fba680d2010 2024-03-31T15:11:25.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.247+0000 7fba6d200640 1 -- 172.21.15.16:0/137204492 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fba6815fa80 con 0x7fba680d2010 2024-03-31T15:11:25.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.247+0000 7fba6d200640 1 -- 172.21.15.16:0/137204492 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fba68069950 con 0x7fba680d2010 2024-03-31T15:11:25.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.247+0000 7fba5ea00640 1 -- 172.21.15.16:0/137204492 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fba500ab070 con 0x7fba680d2010 2024-03-31T15:11:25.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.247+0000 7fba5ea00640 1 --2- 172.21.15.16:0/137204492 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fba44068380 0x7fba4406a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:25.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.251+0000 7fba5ea00640 1 -- 172.21.15.16:0/137204492 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 5910+0+0 (secure 0 0 0) 0x7fba50120a70 con 0x7fba680d2010 2024-03-31T15:11:25.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.251+0000 7fba65e00640 1 --2- 172.21.15.16:0/137204492 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fba44068380 0x7fba4406a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:25.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.251+0000 7fba65e00640 1 --2- 172.21.15.16:0/137204492 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fba44068380 0x7fba4406a840 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7fba58000f70 tx=0x7fba58007e50 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:11:25.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.251+0000 7fba5ea00640 1 -- 172.21.15.16:0/137204492 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fba50016150 con 0x7fba680d2010 2024-03-31T15:11:25.406 INFO:tasks.ceph.mds.c.smithi016.stderr:2024-03-31T15:11:25.411+0000 7f86e2a00640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-31T15:11:25.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.579+0000 7fba6d200640 1 -- 172.21.15.16:0/137204492 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7fba680d0480 con 0x7fba680d2010 2024-03-31T15:11:25.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.579+0000 7fba5ea00640 1 -- 172.21.15.16:0/137204492 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v74) ==== 53+0+73 (secure 0 0 0) 0x7fba500f0870 con 0x7fba680d2010 2024-03-31T15:11:25.574 INFO:teuthology.orchestra.run.smithi016.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-31T15:11:25.577 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.583+0000 7fba6d200640 1 -- 172.21.15.16:0/137204492 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fba44068380 msgr2=0x7fba4406a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:11:25.577 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.583+0000 7fba6d200640 1 --2- 172.21.15.16:0/137204492 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fba44068380 0x7fba4406a840 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7fba58000f70 tx=0x7fba58007e50 comp rx=0 tx=0).stop 2024-03-31T15:11:25.577 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.583+0000 7fba6d200640 1 -- 172.21.15.16:0/137204492 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fba680d2010 msgr2=0x7fba6813ecf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:11:25.577 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.583+0000 7fba6d200640 1 --2- 172.21.15.16:0/137204492 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fba680d2010 0x7fba6813ecf0 secure :-1 s=READY pgs=1279 cs=0 l=1 rev1=1 crypto rx=0x7fba500b0040 tx=0x7fba5001ada0 comp rx=0 tx=0).stop 2024-03-31T15:11:25.577 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.583+0000 7fba6d200640 1 -- 172.21.15.16:0/137204492 shutdown_connections 2024-03-31T15:11:25.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.583+0000 7fba6d200640 1 --2- 172.21.15.16:0/137204492 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fba44068380 0x7fba4406a840 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:25.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.583+0000 7fba6d200640 1 --2- 172.21.15.16:0/137204492 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fba68145e40 0x7fba68142790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:25.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.583+0000 7fba6d200640 1 --2- 172.21.15.16:0/137204492 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fba680d2010 0x7fba6813ecf0 unknown :-1 s=CLOSED pgs=1279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:25.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.583+0000 7fba6d200640 1 --2- 172.21.15.16:0/137204492 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fba6806f920 0x7fba6815cb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:25.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.583+0000 7fba6d200640 1 -- 172.21.15.16:0/137204492 >> 172.21.15.16:0/137204492 conn(0x7fba68067330 msgr2=0x7fba6815aee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:11:25.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.583+0000 7fba6d200640 1 -- 172.21.15.16:0/137204492 shutdown_connections 2024-03-31T15:11:25.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25.583+0000 7fba6d200640 1 -- 172.21.15.16:0/137204492 wait complete. 2024-03-31T15:11:25.595 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-31T15:11:25.595 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-31T15:11:25.595 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi016.front.sepia.ceph.com 2024-03-31T15:11:25.595 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-31T15:11:25.595 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-31T15:11:25.595 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-31T15:11:25.595 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-31T15:11:25.595 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-31T15:11:25.596 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-31T15:11:25.596 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:11:25.596 DEBUG:teuthology.orchestra.run.smithi016:> ip addr 2024-03-31T15:11:25.605 INFO:teuthology.orchestra.run.smithi016.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-31T15:11:25.605 INFO:teuthology.orchestra.run.smithi016.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-31T15:11:25.605 INFO:teuthology.orchestra.run.smithi016.stdout: inet 127.0.0.1/8 scope host lo 2024-03-31T15:11:25.605 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2024-03-31T15:11:25.606 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 ::1/128 scope host 2024-03-31T15:11:25.606 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2024-03-31T15:11:25.606 INFO:teuthology.orchestra.run.smithi016.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-03-31T15:11:25.606 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:6b:fd:e0 brd ff:ff:ff:ff:ff:ff 2024-03-31T15:11:25.606 INFO:teuthology.orchestra.run.smithi016.stdout: altname enp7s0f0 2024-03-31T15:11:25.606 INFO:teuthology.orchestra.run.smithi016.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-03-31T15:11:25.606 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:6b:fd:e1 brd ff:ff:ff:ff:ff:ff 2024-03-31T15:11:25.606 INFO:teuthology.orchestra.run.smithi016.stdout: altname enp7s0f1 2024-03-31T15:11:25.606 INFO:teuthology.orchestra.run.smithi016.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-31T15:11:25.606 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:bd:15:64 brd ff:ff:ff:ff:ff:ff 2024-03-31T15:11:25.606 INFO:teuthology.orchestra.run.smithi016.stdout: altname enp1s0f0 2024-03-31T15:11:25.606 INFO:teuthology.orchestra.run.smithi016.stdout: inet 172.21.15.16/20 brd 172.21.15.255 scope global dynamic ens1f0 2024-03-31T15:11:25.606 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft 41995sec preferred_lft 41995sec 2024-03-31T15:11:25.606 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 fe80::ec4:7aff:febd:1564/64 scope link 2024-03-31T15:11:25.607 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2024-03-31T15:11:25.607 INFO:teuthology.orchestra.run.smithi016.stdout:5: ens1f1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-03-31T15:11:25.607 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:bd:15:65 brd ff:ff:ff:ff:ff:ff 2024-03-31T15:11:25.607 INFO:teuthology.orchestra.run.smithi016.stdout: altname enp1s0f1 2024-03-31T15:11:25.607 INFO:teuthology.orchestra.run.smithi016.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2024-03-31T15:11:25.607 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 02:42:d4:1f:7d:73 brd ff:ff:ff:ff:ff:ff 2024-03-31T15:11:25.607 INFO:teuthology.orchestra.run.smithi016.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2024-03-31T15:11:25.607 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2024-03-31T15:11:25.607 INFO:teuthology.orchestra.run.smithi016.stdout:7: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-31T15:11:25.607 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 5e:1f:84:6d:79:ae brd ff:ff:ff:ff:ff:ff 2024-03-31T15:11:25.607 INFO:teuthology.orchestra.run.smithi016.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-31T15:11:25.607 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2024-03-31T15:11:25.607 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 fe80::5c1f:84ff:fe6d:79ae/64 scope link 2024-03-31T15:11:25.607 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2024-03-31T15:11:25.607 INFO:teuthology.orchestra.run.smithi016.stdout:8: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-31T15:11:25.608 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0e:5c:f6:1f:82:2f brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-31T15:11:25.608 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 fe80::c5c:f6ff:fe1f:822f/64 scope link 2024-03-31T15:11:25.608 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2024-03-31T15:11:25.608 INFO:teuthology.orchestra.run.smithi016.stdout:9: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-31T15:11:25.608 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether c6:a6:3d:69:d0:db brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-31T15:11:25.608 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 fe80::c4a6:3dff:fe69:d0db/64 scope link 2024-03-31T15:11:25.608 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2024-03-31T15:11:25.608 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:11:25.608 DEBUG:teuthology.orchestra.run.smithi016:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-31T15:11:25.663 INFO:teuthology.orchestra.run.smithi016.stdout:1 2024-03-31T15:11:25.665 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:11:25.665 DEBUG:teuthology.orchestra.run.smithi016:> route 2024-03-31T15:11:25.719 INFO:teuthology.orchestra.run.smithi016.stdout:Kernel IP routing table 2024-03-31T15:11:25.720 INFO:teuthology.orchestra.run.smithi016.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-31T15:11:25.720 INFO:teuthology.orchestra.run.smithi016.stdout:default _gateway 0.0.0.0 UG 0 0 0 ens1f0 2024-03-31T15:11:25.720 INFO:teuthology.orchestra.run.smithi016.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2024-03-31T15:11:25.720 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 0 0 0 ens1f0 2024-03-31T15:11:25.720 INFO:teuthology.orchestra.run.smithi016.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-31T15:11:25.721 DEBUG:teuthology.orchestra.run.smithi016:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-31T15:11:25.721 DEBUG:teuthology.orchestra.run.smithi016:> set -e 2024-03-31T15:11:25.721 DEBUG:teuthology.orchestra.run.smithi016:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-03-31T15:11:25.722 DEBUG:teuthology.orchestra.run.smithi016:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-03-31T15:11:25.722 DEBUG:teuthology.orchestra.run.smithi016:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-03-31T15:11:25.722 DEBUG:teuthology.orchestra.run.smithi016:> ') 2024-03-31T15:11:25.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-31T15:11:25.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:25 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-31T15:11:25.856 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:11:25.857 DEBUG:teuthology.orchestra.run.smithi016:> ip netns list 2024-03-31T15:11:25.866 INFO:teuthology.orchestra.run.smithi016.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-31T15:11:25.866 INFO:teuthology.orchestra.run.smithi016.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-31T15:11:25.866 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-31T15:11:25.867 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:11:25.867 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip link set brx.0 up 2024-03-31T15:11:25.929 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-03-31T15:11:25.930 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-31T15:11:25.930 DEBUG:teuthology.orchestra.run.smithi016:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-31T15:11:25.977 INFO:teuthology.orchestra.run.smithi016.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-31T15:11:25.978 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-31T15:11:25.978 DEBUG:teuthology.orchestra.run.smithi016:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-31T15:11:26.029 DEBUG:teuthology.orchestra.run.smithi016:> sudo modprobe fuse 2024-03-31T15:11:26.090 DEBUG:teuthology.orchestra.run.smithi016:> cat /proc/self/mounts | awk '{print $2}' 2024-03-31T15:11:26.143 INFO:teuthology.orchestra.run.smithi016.stdout:/sys 2024-03-31T15:11:26.143 INFO:teuthology.orchestra.run.smithi016.stdout:/proc 2024-03-31T15:11:26.143 INFO:teuthology.orchestra.run.smithi016.stdout:/dev 2024-03-31T15:11:26.143 INFO:teuthology.orchestra.run.smithi016.stdout:/dev/pts 2024-03-31T15:11:26.143 INFO:teuthology.orchestra.run.smithi016.stdout:/run 2024-03-31T15:11:26.143 INFO:teuthology.orchestra.run.smithi016.stdout:/ 2024-03-31T15:11:26.143 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/kernel/security 2024-03-31T15:11:26.143 INFO:teuthology.orchestra.run.smithi016.stdout:/dev/shm 2024-03-31T15:11:26.144 INFO:teuthology.orchestra.run.smithi016.stdout:/run/lock 2024-03-31T15:11:26.144 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/cgroup 2024-03-31T15:11:26.144 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/pstore 2024-03-31T15:11:26.144 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/bpf 2024-03-31T15:11:26.144 INFO:teuthology.orchestra.run.smithi016.stdout:/proc/sys/fs/binfmt_misc 2024-03-31T15:11:26.144 INFO:teuthology.orchestra.run.smithi016.stdout:/dev/hugepages 2024-03-31T15:11:26.144 INFO:teuthology.orchestra.run.smithi016.stdout:/dev/mqueue 2024-03-31T15:11:26.144 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/kernel/debug 2024-03-31T15:11:26.144 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/kernel/tracing 2024-03-31T15:11:26.144 INFO:teuthology.orchestra.run.smithi016.stdout:/proc/fs/nfsd 2024-03-31T15:11:26.144 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/fuse/connections 2024-03-31T15:11:26.145 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/kernel/config 2024-03-31T15:11:26.145 INFO:teuthology.orchestra.run.smithi016.stdout:/run/credentials/systemd-sysusers.service 2024-03-31T15:11:26.145 INFO:teuthology.orchestra.run.smithi016.stdout:/run/rpc_pipefs 2024-03-31T15:11:26.145 INFO:teuthology.orchestra.run.smithi016.stdout:/run/user/1000 2024-03-31T15:11:26.145 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/ceph 2024-03-31T15:11:26.145 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/ceph/osd/ceph-0 2024-03-31T15:11:26.145 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/ceph/osd/ceph-1 2024-03-31T15:11:26.145 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/ceph/osd/ceph-2 2024-03-31T15:11:26.145 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/ceph/osd/ceph-3 2024-03-31T15:11:26.145 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns 2024-03-31T15:11:26.145 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-31T15:11:26.145 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-31T15:11:26.145 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-31T15:11:26.145 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-31T15:11:26.146 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:11:26.146 DEBUG:teuthology.orchestra.run.smithi016:> ls /sys/fs/fuse/connections 2024-03-31T15:11:26.197 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-31T15:11:26.197 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-03-31T15:11:26.241 DEBUG:teuthology.orchestra.run.smithi016:> sudo modprobe fuse 2024-03-31T15:11:26.262 DEBUG:teuthology.orchestra.run.smithi016:> cat /proc/self/mounts | awk '{print $2}' 2024-03-31T15:11:26.317 INFO:teuthology.orchestra.run.smithi016.stdout:/sys 2024-03-31T15:11:26.317 INFO:teuthology.orchestra.run.smithi016.stdout:/proc 2024-03-31T15:11:26.317 INFO:teuthology.orchestra.run.smithi016.stdout:/dev 2024-03-31T15:11:26.317 INFO:teuthology.orchestra.run.smithi016.stdout:/dev/pts 2024-03-31T15:11:26.317 INFO:teuthology.orchestra.run.smithi016.stdout:/run 2024-03-31T15:11:26.317 INFO:teuthology.orchestra.run.smithi016.stdout:/ 2024-03-31T15:11:26.317 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/kernel/security 2024-03-31T15:11:26.317 INFO:teuthology.orchestra.run.smithi016.stdout:/dev/shm 2024-03-31T15:11:26.317 INFO:teuthology.orchestra.run.smithi016.stdout:/run/lock 2024-03-31T15:11:26.318 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/cgroup 2024-03-31T15:11:26.318 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/pstore 2024-03-31T15:11:26.318 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/bpf 2024-03-31T15:11:26.318 INFO:teuthology.orchestra.run.smithi016.stdout:/proc/sys/fs/binfmt_misc 2024-03-31T15:11:26.318 INFO:teuthology.orchestra.run.smithi016.stdout:/dev/hugepages 2024-03-31T15:11:26.318 INFO:teuthology.orchestra.run.smithi016.stdout:/dev/mqueue 2024-03-31T15:11:26.318 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/kernel/debug 2024-03-31T15:11:26.318 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/kernel/tracing 2024-03-31T15:11:26.318 INFO:teuthology.orchestra.run.smithi016.stdout:/proc/fs/nfsd 2024-03-31T15:11:26.318 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/fuse/connections 2024-03-31T15:11:26.318 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/kernel/config 2024-03-31T15:11:26.318 INFO:teuthology.orchestra.run.smithi016.stdout:/run/credentials/systemd-sysusers.service 2024-03-31T15:11:26.318 INFO:teuthology.orchestra.run.smithi016.stdout:/run/rpc_pipefs 2024-03-31T15:11:26.319 INFO:teuthology.orchestra.run.smithi016.stdout:/run/user/1000 2024-03-31T15:11:26.319 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/ceph 2024-03-31T15:11:26.319 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/ceph/osd/ceph-0 2024-03-31T15:11:26.319 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/ceph/osd/ceph-1 2024-03-31T15:11:26.319 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/ceph/osd/ceph-2 2024-03-31T15:11:26.319 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/ceph/osd/ceph-3 2024-03-31T15:11:26.319 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns 2024-03-31T15:11:26.319 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-31T15:11:26.319 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-31T15:11:26.319 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-31T15:11:26.319 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-31T15:11:26.320 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:11:26.320 DEBUG:teuthology.orchestra.run.smithi016:> ls /sys/fs/fuse/connections 2024-03-31T15:11:26.339 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi016.stderr:2024-03-31T15:11:26.347+0000 7f429abe8580 -1 init, newargv = 0x558d288f1260 newargc=15 2024-03-31T15:11:26.339 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi016.stderr:ceph-fuse[46616]: starting ceph client 2024-03-31T15:11:26.347 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi016.stderr:ceph-fuse[46616]: starting fuse 2024-03-31T15:11:26.349 INFO:teuthology.orchestra.run.smithi016.stdout:43 2024-03-31T15:11:26.350 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [43] 2024-03-31T15:11:26.350 DEBUG:teuthology.orchestra.run.smithi016:> sudo stdin-killer -- python3 -c ' 2024-03-31T15:11:26.350 DEBUG:teuthology.orchestra.run.smithi016:> import glob 2024-03-31T15:11:26.350 DEBUG:teuthology.orchestra.run.smithi016:> import re 2024-03-31T15:11:26.350 DEBUG:teuthology.orchestra.run.smithi016:> import os 2024-03-31T15:11:26.350 DEBUG:teuthology.orchestra.run.smithi016:> import subprocess 2024-03-31T15:11:26.350 DEBUG:teuthology.orchestra.run.smithi016:> 2024-03-31T15:11:26.350 DEBUG:teuthology.orchestra.run.smithi016:> def _find_admin_socket(client_name): 2024-03-31T15:11:26.350 DEBUG:teuthology.orchestra.run.smithi016:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-03-31T15:11:26.350 DEBUG:teuthology.orchestra.run.smithi016:> files = glob.glob(asok_path) 2024-03-31T15:11:26.350 DEBUG:teuthology.orchestra.run.smithi016:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-03-31T15:11:26.350 DEBUG:teuthology.orchestra.run.smithi016:> 2024-03-31T15:11:26.351 DEBUG:teuthology.orchestra.run.smithi016:> # Given a non-glob path, it better be there 2024-03-31T15:11:26.351 DEBUG:teuthology.orchestra.run.smithi016:> if "*" not in asok_path: 2024-03-31T15:11:26.351 DEBUG:teuthology.orchestra.run.smithi016:> assert(len(files) == 1) 2024-03-31T15:11:26.351 DEBUG:teuthology.orchestra.run.smithi016:> return files[0] 2024-03-31T15:11:26.351 DEBUG:teuthology.orchestra.run.smithi016:> 2024-03-31T15:11:26.351 DEBUG:teuthology.orchestra.run.smithi016:> for f in files: 2024-03-31T15:11:26.351 DEBUG:teuthology.orchestra.run.smithi016:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-31T15:11:26.351 DEBUG:teuthology.orchestra.run.smithi016:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-31T15:11:26.351 DEBUG:teuthology.orchestra.run.smithi016:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-31T15:11:26.351 DEBUG:teuthology.orchestra.run.smithi016:> contents = proc_f.read() 2024-03-31T15:11:26.351 DEBUG:teuthology.orchestra.run.smithi016:> if mountpoint in contents: 2024-03-31T15:11:26.351 DEBUG:teuthology.orchestra.run.smithi016:> return f 2024-03-31T15:11:26.351 DEBUG:teuthology.orchestra.run.smithi016:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-31T15:11:26.351 DEBUG:teuthology.orchestra.run.smithi016:> 2024-03-31T15:11:26.351 DEBUG:teuthology.orchestra.run.smithi016:> print(_find_admin_socket("client.0")) 2024-03-31T15:11:26.351 DEBUG:teuthology.orchestra.run.smithi016:> ' 2024-03-31T15:11:26.472 INFO:teuthology.orchestra.run.smithi016.stdout:/var/run/ceph/ceph-client.0.46616.asok 2024-03-31T15:11:26.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:26 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-31T15:11:26.483 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.46616.asok 2024-03-31T15:11:26.484 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:11:26.484 DEBUG:teuthology.orchestra.run.smithi016:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.46616.asok status 2024-03-31T15:11:26.704 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-03-31T15:11:26.704 INFO:teuthology.orchestra.run.smithi016.stdout: "metadata": { 2024-03-31T15:11:26.704 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph_sha1": "d2271579d5e735cfd6d436b8349c004f06527cb0", 2024-03-31T15:11:26.704 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph_version": "ceph version 19.0.0-2574-gd2271579 (d2271579d5e735cfd6d436b8349c004f06527cb0) squid (dev)", 2024-03-31T15:11:26.705 INFO:teuthology.orchestra.run.smithi016.stdout: "entity_id": "0", 2024-03-31T15:11:26.705 INFO:teuthology.orchestra.run.smithi016.stdout: "hostname": "smithi016", 2024-03-31T15:11:26.705 INFO:teuthology.orchestra.run.smithi016.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-03-31T15:11:26.705 INFO:teuthology.orchestra.run.smithi016.stdout: "pid": "46616", 2024-03-31T15:11:26.705 INFO:teuthology.orchestra.run.smithi016.stdout: "root": "/" 2024-03-31T15:11:26.705 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-31T15:11:26.705 INFO:teuthology.orchestra.run.smithi016.stdout: "dentry_count": 0, 2024-03-31T15:11:26.705 INFO:teuthology.orchestra.run.smithi016.stdout: "dentry_pinned_count": 0, 2024-03-31T15:11:26.705 INFO:teuthology.orchestra.run.smithi016.stdout: "id": 8187, 2024-03-31T15:11:26.705 INFO:teuthology.orchestra.run.smithi016.stdout: "inst": { 2024-03-31T15:11:26.705 INFO:teuthology.orchestra.run.smithi016.stdout: "name": { 2024-03-31T15:11:26.705 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "client", 2024-03-31T15:11:26.705 INFO:teuthology.orchestra.run.smithi016.stdout: "num": 8187 2024-03-31T15:11:26.706 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-31T15:11:26.706 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": { 2024-03-31T15:11:26.706 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-03-31T15:11:26.706 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "192.168.0.1:0", 2024-03-31T15:11:26.706 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 932202664 2024-03-31T15:11:26.706 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-31T15:11:26.706 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-31T15:11:26.706 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": { 2024-03-31T15:11:26.706 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-03-31T15:11:26.706 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "192.168.0.1:0", 2024-03-31T15:11:26.706 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 932202664 2024-03-31T15:11:26.706 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-31T15:11:26.706 INFO:teuthology.orchestra.run.smithi016.stdout: "inst_str": "client.8187 192.168.0.1:0/932202664", 2024-03-31T15:11:26.706 INFO:teuthology.orchestra.run.smithi016.stdout: "addr_str": "192.168.0.1:0/932202664", 2024-03-31T15:11:26.707 INFO:teuthology.orchestra.run.smithi016.stdout: "inode_count": 1, 2024-03-31T15:11:26.707 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_epoch": 74, 2024-03-31T15:11:26.707 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_epoch": 85, 2024-03-31T15:11:26.707 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_epoch_barrier": 0, 2024-03-31T15:11:26.707 INFO:teuthology.orchestra.run.smithi016.stdout: "blocklisted": false, 2024-03-31T15:11:26.707 INFO:teuthology.orchestra.run.smithi016.stdout: "fs_name": "cephfs" 2024-03-31T15:11:26.707 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-03-31T15:11:26.726 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:11:26.726 DEBUG:teuthology.orchestra.run.smithi016:> stat --file-system '--printf=%T 2024-03-31T15:11:26.726 DEBUG:teuthology.orchestra.run.smithi016:> ' -- /home/ubuntu/cephtest/mnt.0 2024-03-31T15:11:26.735 INFO:teuthology.orchestra.run.smithi016.stdout:fuseblk 2024-03-31T15:11:26.736 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-03-31T15:11:26.736 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:11:26.736 DEBUG:teuthology.orchestra.run.smithi016:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-31T15:11:26.798 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:11:26.798 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.6:0 scrub start /dir recursive,repair 2024-03-31T15:11:26.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:26.991+0000 7f73a4400640 1 Processor -- start 2024-03-31T15:11:26.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:26.991+0000 7f73a4400640 1 -- start start 2024-03-31T15:11:26.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:26.995+0000 7f73a4400640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f739c0cc7b0 0x7f739c0b3dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:26.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:26.995+0000 7f73a4400640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f739c0b4300 0x7f739c0d1a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:26.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:26.995+0000 7f73a4400640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f739c0ce2d0 con 0x7f739c0b7900 2024-03-31T15:11:26.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:26.995+0000 7f73a4400640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f739c0b4770 con 0x7f739c0b4300 2024-03-31T15:11:26.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:26.995+0000 7f73a4400640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f739c0ceb10 con 0x7f739c0cc7b0 2024-03-31T15:11:26.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:26.995+0000 7f73a1a00640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f739c0b7900 0x7f739c160740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:33766/0 (socket says 172.21.15.16:33766) 2024-03-31T15:11:26.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:26.995+0000 7f73a1a00640 1 -- 172.21.15.16:0/1814660067 learned_addr learned my addr 172.21.15.16:0/1814660067 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:11:26.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:26.995+0000 7f73a1000640 1 --2- 172.21.15.16:0/1814660067 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f739c0cc7b0 0x7f739c0b3dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:26.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:26.995+0000 7f73a2400640 1 --2- 172.21.15.16:0/1814660067 >> v2:172.21.15.89:3300/0 conn(0x7f739c0b4300 0x7f739c0d1a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:26.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:26.995+0000 7f739be00640 1 -- 172.21.15.16:0/1814660067 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 860707499 0 0) 0x7f7390001120 con 0x7f739c0b7900 2024-03-31T15:11:26.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:26.995+0000 7f739be00640 1 -- 172.21.15.16:0/1814660067 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7384003530 con 0x7f739c0b7900 2024-03-31T15:11:26.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:26.995+0000 7f739be00640 1 -- 172.21.15.16:0/1814660067 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 98782672 0 0) 0x7f7384003530 con 0x7f739c0b7900 2024-03-31T15:11:26.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:26.995+0000 7f739be00640 1 -- 172.21.15.16:0/1814660067 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f739c0cc7b0 msgr2=0x7f739c0b3dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:11:26.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:26.995+0000 7f739be00640 1 --2- 172.21.15.16:0/1814660067 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f739c0cc7b0 0x7f739c0b3dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:26.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:26.995+0000 7f739be00640 1 -- 172.21.15.16:0/1814660067 >> v2:172.21.15.89:3300/0 conn(0x7f739c0b4300 msgr2=0x7f739c0d1a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:11:26.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:26.995+0000 7f739be00640 1 --2- 172.21.15.16:0/1814660067 >> v2:172.21.15.89:3300/0 conn(0x7f739c0b4300 0x7f739c0d1a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:26.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:26.995+0000 7f739be00640 1 -- 172.21.15.16:0/1814660067 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f739c0ce550 con 0x7f739c0b7900 2024-03-31T15:11:26.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:26.995+0000 7f739be00640 1 -- 172.21.15.16:0/1814660067 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 214807623 0 0) 0x7f7390001850 con 0x7f739c0b7900 2024-03-31T15:11:26.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:26.995+0000 7f739be00640 1 -- 172.21.15.16:0/1814660067 >> v1:172.21.15.16:6789/0 conn(0x7f739c0b7900 legacy=0x7f739c160740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:11:26.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:26.995+0000 7f739be00640 1 --2- 172.21.15.16:0/1814660067 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f739c0cc7b0 0x7f73840039c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:26.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:26.995+0000 7f739be00640 1 --2- 172.21.15.16:0/1814660067 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7384003f40 0x7f7384004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:26.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:26.995+0000 7f739be00640 1 --2- 172.21.15.16:0/1814660067 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7384004d50 0x7f73840051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:26.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:26.995+0000 7f739be00640 1 -- 172.21.15.16:0/1814660067 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f739c0b4770 con 0x7f7384003f40 2024-03-31T15:11:26.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:26.995+0000 7f739be00640 1 -- 172.21.15.16:0/1814660067 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f739c0ceb10 con 0x7f7384004d50 2024-03-31T15:11:26.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:26.995+0000 7f739be00640 1 -- 172.21.15.16:0/1814660067 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f738400ba20 con 0x7f739c0cc7b0 2024-03-31T15:11:26.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:26.995+0000 7f73a1000640 1 --2- 172.21.15.16:0/1814660067 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7384003f40 0x7f7384004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:26.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:26.995+0000 7f73a1a00640 1 --2- 172.21.15.16:0/1814660067 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f739c0cc7b0 0x7f73840039c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:26.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:26.995+0000 7f73a2400640 1 --2- 172.21.15.16:0/1814660067 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7384004d50 0x7f73840051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:26.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:26.995+0000 7f73a1000640 1 -- 172.21.15.16:0/1814660067 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f739c0cc7b0 msgr2=0x7f73840039c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:11:26.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:26.995+0000 7f73a1000640 1 --2- 172.21.15.16:0/1814660067 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f739c0cc7b0 0x7f73840039c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:26.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:26.995+0000 7f73a1000640 1 -- 172.21.15.16:0/1814660067 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7384004d50 msgr2=0x7f73840051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:11:26.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:26.995+0000 7f73a1000640 1 --2- 172.21.15.16:0/1814660067 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7384004d50 0x7f73840051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:26.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:26.995+0000 7f73a1000640 1 -- 172.21.15.16:0/1814660067 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7384003530 con 0x7f7384003f40 2024-03-31T15:11:26.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:26.995+0000 7f73a2400640 1 --2- 172.21.15.16:0/1814660067 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7384004d50 0x7f73840051d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:11:26.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:26.999+0000 7f73a1000640 1 --2- 172.21.15.16:0/1814660067 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7384003f40 0x7f7384004810 secure :-1 s=READY pgs=1283 cs=0 l=1 rev1=1 crypto rx=0x7f7388009b10 tx=0x7f739c0d27d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a87e0745322ec74 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:11:26.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:26.999+0000 7f739be00640 1 -- 172.21.15.16:0/1814660067 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73880b2070 con 0x7f7384003f40 2024-03-31T15:11:26.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:26.999+0000 7f739be00640 1 -- 172.21.15.16:0/1814660067 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f73880a5520 con 0x7f7384003f40 2024-03-31T15:11:26.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:26.999+0000 7f739be00640 1 -- 172.21.15.16:0/1814660067 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73880096b0 con 0x7f7384003f40 2024-03-31T15:11:26.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:26.999+0000 7f73a4400640 1 -- 172.21.15.16:0/1814660067 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7384003f40 msgr2=0x7f7384004810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:11:26.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:26.999+0000 7f73a4400640 1 --2- 172.21.15.16:0/1814660067 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7384003f40 0x7f7384004810 secure :-1 s=READY pgs=1283 cs=0 l=1 rev1=1 crypto rx=0x7f7388009b10 tx=0x7f739c0d27d0 comp rx=0 tx=0).stop 2024-03-31T15:11:26.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:26.999+0000 7f73a2400640 1 -- 172.21.15.16:0/1814660067 reap_dead start 2024-03-31T15:11:26.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:26.999+0000 7f73a4400640 1 -- 172.21.15.16:0/1814660067 shutdown_connections 2024-03-31T15:11:26.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:26.999+0000 7f73a4400640 1 -- 172.21.15.16:0/1814660067 >> 172.21.15.16:0/1814660067 conn(0x7f739c06b7f0 msgr2=0x7f739c067330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:11:26.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:26.999+0000 7f73a4400640 1 -- 172.21.15.16:0/1814660067 shutdown_connections 2024-03-31T15:11:26.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:26.999+0000 7f73a4400640 1 -- 172.21.15.16:0/1814660067 wait complete. 2024-03-31T15:11:26.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:26.999+0000 7f73a4400640 1 Processor -- start 2024-03-31T15:11:26.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:26.999+0000 7f73a4400640 1 -- start start 2024-03-31T15:11:26.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.003+0000 7f73a4400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7384003f40 0x7f739c158ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:26.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.003+0000 7f73a4400640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f739c15aba0 0x7f739c1551f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:26.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.003+0000 7f73a4400640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f739c0b31e0 0x7f739c0b3660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:26.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.003+0000 7f73a4400640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f739c15d540 con 0x7f7384003f40 2024-03-31T15:11:26.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.003+0000 7f73a4400640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f739c0b7ce0 con 0x7f739c0b31e0 2024-03-31T15:11:26.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.003+0000 7f73a4400640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f739c15d6c0 con 0x7f739c15aba0 2024-03-31T15:11:26.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.003+0000 7f73a1a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7384003f40 0x7f739c158ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:26.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.003+0000 7f73a1a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7384003f40 0x7f739c158ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56382/0 (socket says 172.21.15.16:56382) 2024-03-31T15:11:26.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.003+0000 7f73a1a00640 1 -- 172.21.15.16:0/2338147853 learned_addr learned my addr 172.21.15.16:0/2338147853 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:11:26.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.003+0000 7f73a1a00640 1 -- 172.21.15.16:0/2338147853 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f739c15aba0 msgr2=0x7f739c1551f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:11:26.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.003+0000 7f73a2400640 1 --2- 172.21.15.16:0/2338147853 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f739c0b31e0 0x7f739c0b3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:26.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.003+0000 7f73a1000640 1 --2- 172.21.15.16:0/2338147853 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f739c15aba0 0x7f739c1551f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:26.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.003+0000 7f73a1a00640 1 --2- 172.21.15.16:0/2338147853 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f739c15aba0 0x7f739c1551f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:26.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.003+0000 7f73a1a00640 1 -- 172.21.15.16:0/2338147853 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f739c0b31e0 msgr2=0x7f739c0b3660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:11:26.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.003+0000 7f73a1a00640 1 --2- 172.21.15.16:0/2338147853 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f739c0b31e0 0x7f739c0b3660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:26.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.003+0000 7f73a1a00640 1 -- 172.21.15.16:0/2338147853 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f739c1595a0 con 0x7f7384003f40 2024-03-31T15:11:26.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.003+0000 7f73a2400640 1 --2- 172.21.15.16:0/2338147853 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f739c0b31e0 0x7f739c0b3660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:11:26.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.003+0000 7f73a1000640 1 --2- 172.21.15.16:0/2338147853 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f739c15aba0 0x7f739c1551f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:11:26.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.003+0000 7f73a1a00640 1 --2- 172.21.15.16:0/2338147853 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7384003f40 0x7f739c158ea0 secure :-1 s=READY pgs=1284 cs=0 l=1 rev1=1 crypto rx=0x7f73900073a0 tx=0x7f7390007860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:11:26.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.003+0000 7f7399600640 1 -- 172.21.15.16:0/2338147853 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f739001a070 con 0x7f7384003f40 2024-03-31T15:11:26.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.003+0000 7f7399600640 1 -- 172.21.15.16:0/2338147853 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f739000f990 con 0x7f7384003f40 2024-03-31T15:11:26.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.003+0000 7f73a4400640 1 -- 172.21.15.16:0/2338147853 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f739c159880 con 0x7f7384003f40 2024-03-31T15:11:26.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.003+0000 7f7399600640 1 -- 172.21.15.16:0/2338147853 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f739000fd60 con 0x7f7384003f40 2024-03-31T15:11:26.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.003+0000 7f73a4400640 1 -- 172.21.15.16:0/2338147853 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f739c159e40 con 0x7f7384003f40 2024-03-31T15:11:26.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.007+0000 7f7399600640 1 -- 172.21.15.16:0/2338147853 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f739001d070 con 0x7f7384003f40 2024-03-31T15:11:26.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.007+0000 7f73a79a3000 1 Processor -- start 2024-03-31T15:11:27.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.007+0000 7f73a79a3000 1 -- start start 2024-03-31T15:11:27.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.007+0000 7f7399600640 1 --2- 172.21.15.16:0/2338147853 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f737c068380 0x7f737c06a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:27.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.007+0000 7f7399600640 1 -- 172.21.15.16:0/2338147853 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 5910+0+0 (secure 0 0 0) 0x7f7390086500 con 0x7f7384003f40 2024-03-31T15:11:27.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.007+0000 7f73a2400640 1 --2- 172.21.15.16:0/2338147853 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f737c068380 0x7f737c06a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:27.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.007+0000 7f73a79a3000 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55dd3353a840 0x55dd3353ac20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:27.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.007+0000 7f73a79a3000 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55dd3353b160 0x55dd334edc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:27.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.007+0000 7f73a79a3000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dd334ee1d0 0x55dd334ee5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:27.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.007+0000 7f73a79a3000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55dd333d8180 con 0x55dd334ee1d0 2024-03-31T15:11:27.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.007+0000 7f73a79a3000 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55dd3333fc90 con 0x55dd3353b160 2024-03-31T15:11:27.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.007+0000 7f73a79a3000 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x55dd332dca00 con 0x55dd3353a840 2024-03-31T15:11:27.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.007+0000 7f73a2400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dd334ee1d0 0x55dd334ee5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:27.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.007+0000 7f73a1a00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55dd3353a840 0x55dd3353ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:27.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.007+0000 7f73a1000640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55dd3353b160 0x55dd334edc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:27.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.007+0000 7f73a2400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dd334ee1d0 0x55dd334ee5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56398/0 (socket says 172.21.15.16:56398) 2024-03-31T15:11:27.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.007+0000 7f73a2400640 1 -- 172.21.15.16:0/541242014 learned_addr learned my addr 172.21.15.16:0/541242014 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:11:27.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.007+0000 7f73a1000640 1 -- 172.21.15.16:0/541242014 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55dd3353a840 msgr2=0x55dd3353ac20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:11:27.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.007+0000 7f73a1000640 1 --2- 172.21.15.16:0/541242014 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55dd3353a840 0x55dd3353ac20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:27.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.007+0000 7f73a1000640 1 -- 172.21.15.16:0/541242014 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dd334ee1d0 msgr2=0x55dd334ee5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:11:27.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.007+0000 7f73a1000640 1 --2- 172.21.15.16:0/541242014 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dd334ee1d0 0x55dd334ee5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:27.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.007+0000 7f73a1000640 1 -- 172.21.15.16:0/541242014 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dd332c5510 con 0x55dd3353b160 2024-03-31T15:11:27.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.007+0000 7f73a2400640 1 --2- 172.21.15.16:0/541242014 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dd334ee1d0 0x55dd334ee5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:11:27.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.007+0000 7f73a1a00640 1 --2- 172.21.15.16:0/541242014 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55dd3353a840 0x55dd3353ac20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:11:27.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.007+0000 7f73a2400640 1 --2- 172.21.15.16:0/2338147853 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f737c068380 0x7f737c06a840 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f7394009fd0 tx=0x7f73940079e0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:11:27.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.007+0000 7f73a1000640 1 --2- 172.21.15.16:0/541242014 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55dd3353b160 0x55dd334edc90 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f73880ac000 tx=0x7f7388004740 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8758fd8da97b09c7 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:11:27.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.007+0000 7f738e000640 1 -- 172.21.15.16:0/541242014 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73880b2040 con 0x55dd3353b160 2024-03-31T15:11:27.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.011+0000 7f738e000640 1 -- 172.21.15.16:0/541242014 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f73880096b0 con 0x55dd3353b160 2024-03-31T15:11:27.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.011+0000 7f738e000640 1 -- 172.21.15.16:0/541242014 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7388004270 con 0x55dd3353b160 2024-03-31T15:11:27.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.011+0000 7f73a79a3000 1 -- 172.21.15.16:0/541242014 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55dd3353b160 msgr2=0x55dd334edc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:11:27.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.011+0000 7f73a79a3000 1 --2- 172.21.15.16:0/541242014 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55dd3353b160 0x55dd334edc90 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f73880ac000 tx=0x7f7388004740 comp rx=0 tx=0).stop 2024-03-31T15:11:27.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.011+0000 7f73a79a3000 1 -- 172.21.15.16:0/541242014 shutdown_connections 2024-03-31T15:11:27.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.011+0000 7f73a79a3000 1 --2- 172.21.15.16:0/541242014 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dd334ee1d0 0x55dd334ee5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:27.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.011+0000 7f73a79a3000 1 --2- 172.21.15.16:0/541242014 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55dd3353b160 0x55dd334edc90 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:27.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.011+0000 7f73a79a3000 1 --2- 172.21.15.16:0/541242014 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55dd3353a840 0x55dd3353ac20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:27.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.011+0000 7f73a79a3000 1 -- 172.21.15.16:0/541242014 >> 172.21.15.16:0/541242014 conn(0x55dd3351b5e0 msgr2=0x55dd3351d9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:11:27.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.011+0000 7f73a79a3000 1 -- 172.21.15.16:0/541242014 shutdown_connections 2024-03-31T15:11:27.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.011+0000 7f73a79a3000 1 -- 172.21.15.16:0/541242014 wait complete. 2024-03-31T15:11:27.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.011+0000 7f73a79a3000 1 -- start start 2024-03-31T15:11:27.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.011+0000 7f73a79a3000 1 Processor -- start 2024-03-31T15:11:27.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.011+0000 7f73a79a3000 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55dd334ee1d0 0x55dd33539fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:27.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.011+0000 7f73a79a3000 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55dd3353a840 0x55dd334f3a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:27.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.011+0000 7f73a79a3000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dd3353b160 0x55dd334f3f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:27.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.011+0000 7f73a79a3000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55dd332fd200 con 0x55dd3353b160 2024-03-31T15:11:27.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.011+0000 7f73a79a3000 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55dd3333bc40 con 0x55dd3353a840 2024-03-31T15:11:27.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.011+0000 7f73a79a3000 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x55dd3330bb20 con 0x55dd334ee1d0 2024-03-31T15:11:27.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.011+0000 7f73a79a3000 10 client.0 _ll_register_callbacks cb 0x55dd334db930 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T15:11:27.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.011+0000 7f73a2400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dd3353b160 0x55dd334f3f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:27.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.011+0000 7f73a2400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dd3353b160 0x55dd334f3f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56408/0 (socket says 172.21.15.16:56408) 2024-03-31T15:11:27.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.011+0000 7f73a2400640 1 -- 172.21.15.16:0/3330939335 learned_addr learned my addr 172.21.15.16:0/3330939335 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:11:27.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.011+0000 7f73a1a00640 1 --2- 172.21.15.16:0/3330939335 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55dd334ee1d0 0x55dd33539fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:27.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.011+0000 7f73a1000640 1 --2- 172.21.15.16:0/3330939335 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55dd3353a840 0x55dd334f3a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:27.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.011+0000 7f73a2400640 1 -- 172.21.15.16:0/3330939335 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55dd334ee1d0 msgr2=0x55dd33539fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:11:27.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.011+0000 7f73a2400640 1 --2- 172.21.15.16:0/3330939335 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55dd334ee1d0 0x55dd33539fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:27.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.011+0000 7f73a2400640 1 -- 172.21.15.16:0/3330939335 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55dd3353a840 msgr2=0x55dd334f3a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:11:27.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.011+0000 7f73a2400640 1 --2- 172.21.15.16:0/3330939335 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55dd3353a840 0x55dd334f3a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:27.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.011+0000 7f73a2400640 1 -- 172.21.15.16:0/3330939335 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dd3334ace0 con 0x55dd3353b160 2024-03-31T15:11:27.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.011+0000 7f73a2400640 1 --2- 172.21.15.16:0/3330939335 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dd3353b160 0x55dd334f3f90 secure :-1 s=READY pgs=1285 cs=0 l=1 rev1=1 crypto rx=0x7f7394018fe0 tx=0x7f7394004a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f401fc718ce10f1 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:11:27.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.011+0000 7f7382a00640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-31T15:11:27.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.011+0000 7f7382a00640 1 -- 172.21.15.16:0/3330939335 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f739402a060 con 0x55dd3353b160 2024-03-31T15:11:27.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.011+0000 7f7382a00640 1 -- 172.21.15.16:0/3330939335 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 5910+0+0 (secure 0 0 0) 0x7f7394003ba0 con 0x55dd3353b160 2024-03-31T15:11:27.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.011+0000 7f73a79a3000 1 -- 172.21.15.16:0/3330939335 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f73880096b0 con 0x55dd3353b160 2024-03-31T15:11:27.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.015+0000 7f7382a00640 1 -- 172.21.15.16:0/3330939335 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=74) ==== 24+0+0 (secure 0 0 0) 0x7f7394030050 con 0x55dd3353b160 2024-03-31T15:11:27.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.015+0000 7f73a79a3000 10 client.8199 fetch_fsmap learned FSMap version 74 2024-03-31T15:11:27.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.015+0000 7f73a79a3000 1 -- 172.21.15.16:0/3330939335 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=74,monmap=2+}) -- 0x55dd332fb7c0 con 0x55dd3353b160 2024-03-31T15:11:27.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.015+0000 7f7382a00640 1 -- 172.21.15.16:0/3330939335 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f73880096b0 con 0x55dd3353b160 2024-03-31T15:11:27.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.015+0000 7f7382a00640 1 -- 172.21.15.16:0/3330939335 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 74) ==== 3494+0+0 (secure 0 0 0) 0x7f7394033020 con 0x55dd3353b160 2024-03-31T15:11:27.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.015+0000 7f73a79a3000 10 client.8199 fetch_fsmap finished waiting for FSMap version 74 2024-03-31T15:11:27.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.015+0000 7f73a79a3000 10 client.8199 resolve_mds: resolved 6:0 to role '6:0' aka daemon mds.c 2024-03-31T15:11:27.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.015+0000 7f73a79a3000 20 client.8199 populate_metadata read hostname 'smithi016' 2024-03-31T15:11:27.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.015+0000 7f73a79a3000 1 --2- 172.21.15.16:0/3330939335 >> [v2:172.21.15.16:6832/3780866829,v1:172.21.15.16:6834/3780866829] conn(0x55dd33541a70 0x55dd33541e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:27.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.015+0000 7f73a79a3000 4 client.8199 mds_command: new command op to 8085 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-31T15:11:27.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.015+0000 7f73a79a3000 1 -- 172.21.15.16:0/3330939335 --> [v2:172.21.15.16:6832/3780866829,v1:172.21.15.16:6834/3780866829] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55dd332dac20 con 0x55dd33541a70 2024-03-31T15:11:27.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.015+0000 7f73a1a00640 1 --2- 172.21.15.16:0/3330939335 >> [v2:172.21.15.16:6832/3780866829,v1:172.21.15.16:6834/3780866829] conn(0x55dd33541a70 0x55dd33541e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:27.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.015+0000 7f73a1a00640 1 --2- 172.21.15.16:0/3330939335 >> [v2:172.21.15.16:6832/3780866829,v1:172.21.15.16:6834/3780866829] conn(0x55dd33541a70 0x55dd33541e80 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c3e30036c8bd4cac server_cookie=24dff1ab4167058d in_seq=0 out_seq=0 2024-03-31T15:11:27.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.015+0000 7f7382a00640 10 client.8199 ms_handle_connect on v2:172.21.15.16:6832/3780866829 2024-03-31T15:11:27.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.015+0000 7f7382a00640 1 -- 172.21.15.16:0/3330939335 <== mds.0 v2:172.21.15.16:6832/3780866829 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55dd332dac20 con 0x55dd33541a70 2024-03-31T15:11:27.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.015+0000 7f7382a00640 10 client.8199 handle_command_reply: tid=0 2024-03-31T15:11:27.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.015+0000 7f73a79a3000 1 client.8199 shutdown 2024-03-31T15:11:27.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.015+0000 7f73a79a3000 1 -- 172.21.15.16:0/3330939335 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dd3353b160 msgr2=0x55dd334f3f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:11:27.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.015+0000 7f73a79a3000 1 --2- 172.21.15.16:0/3330939335 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dd3353b160 0x55dd334f3f90 secure :-1 s=READY pgs=1285 cs=0 l=1 rev1=1 crypto rx=0x7f7394018fe0 tx=0x7f7394004a50 comp rx=0 tx=0).stop 2024-03-31T15:11:27.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.015+0000 7f73a79a3000 1 -- 172.21.15.16:0/3330939335 shutdown_connections 2024-03-31T15:11:27.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.015+0000 7f73a79a3000 1 --2- 172.21.15.16:0/3330939335 >> [v2:172.21.15.16:6832/3780866829,v1:172.21.15.16:6834/3780866829] conn(0x55dd33541a70 0x55dd33541e80 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:27.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.015+0000 7f73a79a3000 1 --2- 172.21.15.16:0/3330939335 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dd3353b160 0x55dd334f3f90 unknown :-1 s=CLOSED pgs=1285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:27.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.015+0000 7f73a79a3000 1 --2- 172.21.15.16:0/3330939335 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55dd3353a840 0x55dd334f3a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:27.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.015+0000 7f7382a00640 0 client.8199 ms_handle_reset on v2:172.21.15.16:6832/3780866829 2024-03-31T15:11:27.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.015+0000 7f73a79a3000 1 --2- 172.21.15.16:0/3330939335 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55dd334ee1d0 0x55dd33539fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:27.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.015+0000 7f73a79a3000 1 -- 172.21.15.16:0/3330939335 >> 172.21.15.16:0/3330939335 conn(0x55dd3351b5e0 msgr2=0x55dd334ed390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:11:27.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.015+0000 7f73a79a3000 1 -- 172.21.15.16:0/3330939335 shutdown_connections 2024-03-31T15:11:27.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.015+0000 7f73a79a3000 1 -- 172.21.15.16:0/3330939335 wait complete. 2024-03-31T15:11:27.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.015+0000 7f73a79a3000 20 client.8199 trim_cache size 0 max 16384 2024-03-31T15:11:27.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.035+0000 7f73a79a3000 1 Processor -- start 2024-03-31T15:11:27.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.035+0000 7f73a79a3000 1 -- start start 2024-03-31T15:11:27.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.035+0000 7f73a79a3000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dd334ee1d0 0x55dd334f5ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:27.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.035+0000 7f73a79a3000 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55dd3353a840 0x55dd334ec3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:27.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.035+0000 7f73a79a3000 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55dd3353b160 0x55dd334ec930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:27.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.035+0000 7f73a79a3000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f739c204420 con 0x55dd334ee1d0 2024-03-31T15:11:27.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.035+0000 7f73a79a3000 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f739c2034c0 con 0x55dd3353a840 2024-03-31T15:11:27.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.035+0000 7f73a79a3000 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f739c161f40 con 0x55dd3353b160 2024-03-31T15:11:27.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.035+0000 7f73a1a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dd334ee1d0 0x55dd334f5ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:27.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.035+0000 7f73a1a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dd334ee1d0 0x55dd334f5ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56420/0 (socket says 172.21.15.16:56420) 2024-03-31T15:11:27.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.035+0000 7f73a1a00640 1 -- 172.21.15.16:0/312560799 learned_addr learned my addr 172.21.15.16:0/312560799 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:11:27.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.035+0000 7f73a2400640 1 --2- 172.21.15.16:0/312560799 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55dd3353b160 0x55dd334ec930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:27.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.035+0000 7f73a1000640 1 --2- 172.21.15.16:0/312560799 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55dd3353a840 0x55dd334ec3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:27.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.035+0000 7f73a1a00640 1 -- 172.21.15.16:0/312560799 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55dd3353b160 msgr2=0x55dd334ec930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:11:27.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.035+0000 7f73a1a00640 1 --2- 172.21.15.16:0/312560799 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55dd3353b160 0x55dd334ec930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:27.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.035+0000 7f73a1a00640 1 -- 172.21.15.16:0/312560799 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55dd3353a840 msgr2=0x55dd334ec3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:11:27.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.035+0000 7f73a1a00640 1 --2- 172.21.15.16:0/312560799 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55dd3353a840 0x55dd334ec3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:27.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.035+0000 7f73a1a00640 1 -- 172.21.15.16:0/312560799 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dd332fbc70 con 0x55dd334ee1d0 2024-03-31T15:11:27.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.035+0000 7f73a1000640 1 --2- 172.21.15.16:0/312560799 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55dd3353a840 0x55dd334ec3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:11:27.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.035+0000 7f73a2400640 1 --2- 172.21.15.16:0/312560799 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55dd3353b160 0x55dd334ec930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:11:27.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.035+0000 7f73a1a00640 1 --2- 172.21.15.16:0/312560799 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dd334ee1d0 0x55dd334f5ea0 secure :-1 s=READY pgs=1286 cs=0 l=1 rev1=1 crypto rx=0x7f7390056270 tx=0x7f7390084820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d3fefdaac3b568 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:11:27.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.035+0000 7f738d600640 1 -- 172.21.15.16:0/312560799 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f739007bae0 con 0x55dd334ee1d0 2024-03-31T15:11:27.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.035+0000 7f738d600640 1 -- 172.21.15.16:0/312560799 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f73900545e0 con 0x55dd334ee1d0 2024-03-31T15:11:27.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.035+0000 7f738d600640 1 -- 172.21.15.16:0/312560799 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73900860c0 con 0x55dd334ee1d0 2024-03-31T15:11:27.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.035+0000 7f73a79a3000 1 -- 172.21.15.16:0/312560799 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dd334ee1d0 msgr2=0x55dd334f5ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:11:27.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.035+0000 7f73a79a3000 1 --2- 172.21.15.16:0/312560799 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dd334ee1d0 0x55dd334f5ea0 secure :-1 s=READY pgs=1286 cs=0 l=1 rev1=1 crypto rx=0x7f7390056270 tx=0x7f7390084820 comp rx=0 tx=0).stop 2024-03-31T15:11:27.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.035+0000 7f73a79a3000 1 -- 172.21.15.16:0/312560799 shutdown_connections 2024-03-31T15:11:27.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.035+0000 7f73a79a3000 1 --2- 172.21.15.16:0/312560799 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55dd3353b160 0x55dd334ec930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:27.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.035+0000 7f73a79a3000 1 --2- 172.21.15.16:0/312560799 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55dd3353a840 0x55dd334ec3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:27.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.035+0000 7f73a79a3000 1 --2- 172.21.15.16:0/312560799 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dd334ee1d0 0x55dd334f5ea0 unknown :-1 s=CLOSED pgs=1286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:27.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.035+0000 7f73a79a3000 1 -- 172.21.15.16:0/312560799 >> 172.21.15.16:0/312560799 conn(0x55dd3351b5e0 msgr2=0x55dd33550b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:11:27.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.035+0000 7f73a79a3000 1 -- 172.21.15.16:0/312560799 shutdown_connections 2024-03-31T15:11:27.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.035+0000 7f73a79a3000 1 -- 172.21.15.16:0/312560799 wait complete. 2024-03-31T15:11:27.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.035+0000 7f73a79a3000 1 -- start start 2024-03-31T15:11:27.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.039+0000 7f73a79a3000 1 Processor -- start 2024-03-31T15:11:27.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.039+0000 7f73a79a3000 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55dd334ee1d0 0x55dd33540de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:27.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.039+0000 7f73a79a3000 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55dd3353a840 0x55dd335aa440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:27.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.039+0000 7f73a79a3000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dd3353b160 0x55dd335aa980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:27.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.039+0000 7f73a79a3000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55dd3330bb20 con 0x55dd3353b160 2024-03-31T15:11:27.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.039+0000 7f73a79a3000 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f739c202ea0 con 0x55dd3353a840 2024-03-31T15:11:27.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.039+0000 7f73a79a3000 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x55dd333078f0 con 0x55dd334ee1d0 2024-03-31T15:11:27.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.039+0000 7f73a79a3000 10 client.0 _ll_register_callbacks cb 0x7f7394007390 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T15:11:27.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.039+0000 7f73a2400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dd3353b160 0x55dd335aa980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:27.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.039+0000 7f73a2400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dd3353b160 0x55dd335aa980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50882/0 (socket says 172.21.15.16:50882) 2024-03-31T15:11:27.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.039+0000 7f73a1a00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55dd334ee1d0 0x55dd33540de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:27.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.039+0000 7f73a2400640 1 -- 172.21.15.16:0/4224349567 learned_addr learned my addr 172.21.15.16:0/4224349567 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:11:27.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.039+0000 7f73a1000640 1 --2- 172.21.15.16:0/4224349567 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55dd3353a840 0x55dd335aa440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:27.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.039+0000 7f73a2400640 1 -- 172.21.15.16:0/4224349567 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55dd334ee1d0 msgr2=0x55dd33540de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:11:27.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.039+0000 7f73a2400640 1 --2- 172.21.15.16:0/4224349567 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55dd334ee1d0 0x55dd33540de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:27.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.039+0000 7f73a2400640 1 -- 172.21.15.16:0/4224349567 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55dd3353a840 msgr2=0x55dd335aa440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:11:27.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.039+0000 7f73a2400640 1 --2- 172.21.15.16:0/4224349567 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55dd3353a840 0x55dd335aa440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:27.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.039+0000 7f73a2400640 1 -- 172.21.15.16:0/4224349567 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dd332dcc50 con 0x55dd3353b160 2024-03-31T15:11:27.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.039+0000 7f73a2400640 1 --2- 172.21.15.16:0/4224349567 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dd3353b160 0x55dd335aa980 secure :-1 s=READY pgs=1287 cs=0 l=1 rev1=1 crypto rx=0x7f7394003e90 tx=0x7f739400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=64d26c6d5250ce2b server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:11:27.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.039+0000 7f7382a00640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-31T15:11:27.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.039+0000 7f7382a00640 1 -- 172.21.15.16:0/4224349567 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73940343f0 con 0x55dd3353b160 2024-03-31T15:11:27.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.039+0000 7f7382a00640 1 -- 172.21.15.16:0/4224349567 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 5910+0+0 (secure 0 0 0) 0x7f7394034590 con 0x55dd3353b160 2024-03-31T15:11:27.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.039+0000 7f73a79a3000 1 -- 172.21.15.16:0/4224349567 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f73900545e0 con 0x55dd3353b160 2024-03-31T15:11:27.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.039+0000 7f7382a00640 1 -- 172.21.15.16:0/4224349567 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=74) ==== 24+0+0 (secure 0 0 0) 0x7f7394044050 con 0x55dd3353b160 2024-03-31T15:11:27.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.039+0000 7f73a79a3000 10 client.8205 fetch_fsmap learned FSMap version 74 2024-03-31T15:11:27.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.039+0000 7f73a79a3000 1 -- 172.21.15.16:0/4224349567 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=74,monmap=2+}) -- 0x55dd332c8500 con 0x55dd3353b160 2024-03-31T15:11:27.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.039+0000 7f7382a00640 1 -- 172.21.15.16:0/4224349567 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f73900545e0 con 0x55dd3353b160 2024-03-31T15:11:27.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.039+0000 7f7382a00640 1 -- 172.21.15.16:0/4224349567 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 74) ==== 3494+0+0 (secure 0 0 0) 0x7f7394047020 con 0x55dd3353b160 2024-03-31T15:11:27.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.039+0000 7f73a79a3000 10 client.8205 fetch_fsmap finished waiting for FSMap version 74 2024-03-31T15:11:27.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.039+0000 7f73a79a3000 10 client.8205 resolve_mds: resolved 6:0 to role '6:0' aka daemon mds.c 2024-03-31T15:11:27.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.039+0000 7f73a79a3000 20 client.8205 populate_metadata read hostname 'smithi016' 2024-03-31T15:11:27.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.039+0000 7f73a79a3000 1 --2- 172.21.15.16:0/4224349567 >> [v2:172.21.15.16:6832/3780866829,v1:172.21.15.16:6834/3780866829] conn(0x55dd335aaec0 0x55dd335a6ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:27.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.039+0000 7f73a79a3000 4 client.8205 mds_command: new command op to 8085 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/dir", "scrubops": ["recursive", "repair"]}] 2024-03-31T15:11:27.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.039+0000 7f73a79a3000 1 -- 172.21.15.16:0/4224349567 --> [v2:172.21.15.16:6832/3780866829,v1:172.21.15.16:6834/3780866829] -- command(tid 0: {"prefix": "scrub start", "path": "/dir", "scrubops": ["recursive", "repair"]}) -- 0x55dd3334d1d0 con 0x55dd335aaec0 2024-03-31T15:11:27.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.039+0000 7f73a1a00640 1 --2- 172.21.15.16:0/4224349567 >> [v2:172.21.15.16:6832/3780866829,v1:172.21.15.16:6834/3780866829] conn(0x55dd335aaec0 0x55dd335a6ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:27.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.039+0000 7f73a1a00640 1 --2- 172.21.15.16:0/4224349567 >> [v2:172.21.15.16:6832/3780866829,v1:172.21.15.16:6834/3780866829] conn(0x55dd335aaec0 0x55dd335a6ed0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=17fe321d031fada5 server_cookie=17196a592ec4e3b8 in_seq=0 out_seq=0 2024-03-31T15:11:27.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.039+0000 7f7382a00640 10 client.8205 ms_handle_connect on v2:172.21.15.16:6832/3780866829 2024-03-31T15:11:27.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.043+0000 7f7382a00640 1 -- 172.21.15.16:0/4224349567 <== mds.0 v2:172.21.15.16:6832/3780866829 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55dd3334d1d0 con 0x55dd335aaec0 2024-03-31T15:11:27.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.043+0000 7f7382a00640 10 client.8205 handle_command_reply: tid=0 2024-03-31T15:11:27.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.043+0000 7f73a79a3000 1 client.8205 shutdown 2024-03-31T15:11:27.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.043+0000 7f73a79a3000 1 -- 172.21.15.16:0/4224349567 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dd3353b160 msgr2=0x55dd335aa980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:11:27.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.043+0000 7f73a79a3000 1 --2- 172.21.15.16:0/4224349567 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dd3353b160 0x55dd335aa980 secure :-1 s=READY pgs=1287 cs=0 l=1 rev1=1 crypto rx=0x7f7394003e90 tx=0x7f739400b040 comp rx=0 tx=0).stop 2024-03-31T15:11:27.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.043+0000 7f73a79a3000 1 -- 172.21.15.16:0/4224349567 shutdown_connections 2024-03-31T15:11:27.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.043+0000 7f73a79a3000 1 --2- 172.21.15.16:0/4224349567 >> [v2:172.21.15.16:6832/3780866829,v1:172.21.15.16:6834/3780866829] conn(0x55dd335aaec0 0x55dd335a6ed0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:27.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.043+0000 7f7382a00640 0 client.8205 ms_handle_reset on v2:172.21.15.16:6832/3780866829 2024-03-31T15:11:27.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.043+0000 7f73a79a3000 1 --2- 172.21.15.16:0/4224349567 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dd3353b160 0x55dd335aa980 unknown :-1 s=CLOSED pgs=1287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:27.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.043+0000 7f73a79a3000 1 --2- 172.21.15.16:0/4224349567 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55dd3353a840 0x55dd335aa440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:27.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.043+0000 7f73a79a3000 1 --2- 172.21.15.16:0/4224349567 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55dd334ee1d0 0x55dd33540de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:27.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.043+0000 7f73a79a3000 1 -- 172.21.15.16:0/4224349567 >> 172.21.15.16:0/4224349567 conn(0x55dd3351b5e0 msgr2=0x55dd335513b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:11:27.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.043+0000 7f73a79a3000 1 -- 172.21.15.16:0/4224349567 shutdown_connections 2024-03-31T15:11:27.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.043+0000 7f73a79a3000 1 -- 172.21.15.16:0/4224349567 wait complete. 2024-03-31T15:11:27.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.043+0000 7f73a79a3000 20 client.8205 trim_cache size 0 max 16384 2024-03-31T15:11:27.040 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-03-31T15:11:27.040 INFO:teuthology.orchestra.run.smithi016.stdout: "return_code": 0, 2024-03-31T15:11:27.040 INFO:teuthology.orchestra.run.smithi016.stdout: "scrub_tag": "d7e59e70-33c2-4498-98fb-4555efd5e0c1", 2024-03-31T15:11:27.040 INFO:teuthology.orchestra.run.smithi016.stdout: "mode": "asynchronous" 2024-03-31T15:11:27.040 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-03-31T15:11:27.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.043+0000 7f73a4400640 1 -- 172.21.15.16:0/2338147853 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f737c068380 msgr2=0x7f737c06a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:11:27.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.043+0000 7f73a4400640 1 --2- 172.21.15.16:0/2338147853 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f737c068380 0x7f737c06a840 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f7394009fd0 tx=0x7f73940079e0 comp rx=0 tx=0).stop 2024-03-31T15:11:27.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.043+0000 7f73a4400640 1 -- 172.21.15.16:0/2338147853 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7384003f40 msgr2=0x7f739c158ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:11:27.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.043+0000 7f73a4400640 1 --2- 172.21.15.16:0/2338147853 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7384003f40 0x7f739c158ea0 secure :-1 s=READY pgs=1284 cs=0 l=1 rev1=1 crypto rx=0x7f73900073a0 tx=0x7f7390007860 comp rx=0 tx=0).stop 2024-03-31T15:11:27.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.043+0000 7f73a4400640 1 -- 172.21.15.16:0/2338147853 shutdown_connections 2024-03-31T15:11:27.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.043+0000 7f73a4400640 1 --2- 172.21.15.16:0/2338147853 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f737c068380 0x7f737c06a840 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:27.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.043+0000 7f73a4400640 1 --2- 172.21.15.16:0/2338147853 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f739c0b31e0 0x7f739c0b3660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:27.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.043+0000 7f73a4400640 1 --2- 172.21.15.16:0/2338147853 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f739c15aba0 0x7f739c1551f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:27.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.043+0000 7f73a4400640 1 --2- 172.21.15.16:0/2338147853 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7384003f40 0x7f739c158ea0 unknown :-1 s=CLOSED pgs=1284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:27.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.043+0000 7f73a4400640 1 -- 172.21.15.16:0/2338147853 >> 172.21.15.16:0/2338147853 conn(0x7f739c06b7f0 msgr2=0x7f739c0b1da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:11:27.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.043+0000 7f73a4400640 1 -- 172.21.15.16:0/2338147853 shutdown_connections 2024-03-31T15:11:27.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.043+0000 7f73a4400640 1 -- 172.21.15.16:0/2338147853 wait complete. 2024-03-31T15:11:27.042 INFO:tasks.ceph.mds.c.smithi016.stderr:2024-03-31T15:11:27.043+0000 7f86dda00640 -1 mds.0.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000001 [...2,head] /dir/file auth v1 ap=1 DIRTYPARENT s=0 nl=2 n() | dirtyparent=1 authpin=1 scrubqueue=0 0x55cab02e6680]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(13)0x10000000001:[<0x10000000000/file v6>,<0x1/dir v13>]//[]","memoryvalue":"(13)0x10000000001:[<0x10000000000/file v1>,<0x1/dir v20>]//[]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-31T15:11:27.055 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:11:27.055 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.6:0 scrub status 2024-03-31T15:11:27.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.215+0000 7fda3a400640 1 Processor -- start 2024-03-31T15:11:27.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.215+0000 7fda3a400640 1 -- start start 2024-03-31T15:11:27.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.215+0000 7fda3a400640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fda3415c1c0 0x7fda340cd4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:27.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.215+0000 7fda3a400640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fda340c5a80 0x7fda340c5ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:27.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.215+0000 7fda3a400640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fda340cdac0 con 0x7fda3415c5a0 2024-03-31T15:11:27.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.215+0000 7fda3a400640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7fda340cdda0 con 0x7fda3415c1c0 2024-03-31T15:11:27.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.215+0000 7fda3a400640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7fda340cfc30 con 0x7fda340c5a80 2024-03-31T15:11:27.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.215+0000 7fda39000640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fda3415c1c0 0x7fda340cd4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:27.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.215+0000 7fda39000640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fda3415c1c0 0x7fda340cd4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50894/0 (socket says 172.21.15.16:50894) 2024-03-31T15:11:27.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.215+0000 7fda39000640 1 -- 172.21.15.16:0/466939049 learned_addr learned my addr 172.21.15.16:0/466939049 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:11:27.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.215+0000 7fda39a00640 1 --2- 172.21.15.16:0/466939049 >> v2:172.21.15.89:3300/0 conn(0x7fda340c5a80 0x7fda340c5ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:27.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.215+0000 7fda39000640 1 -- 172.21.15.16:0/466939049 >> v2:172.21.15.89:3300/0 conn(0x7fda340c5a80 msgr2=0x7fda340c5ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:11:27.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.215+0000 7fda39000640 1 --2- 172.21.15.16:0/466939049 >> v2:172.21.15.89:3300/0 conn(0x7fda340c5a80 0x7fda340c5ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:27.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.215+0000 7fda39000640 1 -- 172.21.15.16:0/466939049 >> v1:172.21.15.89:6789/0 conn(0x7fda3415c5a0 legacy=0x7fda340d0710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:11:27.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.215+0000 7fda39000640 1 -- 172.21.15.16:0/466939049 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda340cff10 con 0x7fda3415c1c0 2024-03-31T15:11:27.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.219+0000 7fda39000640 1 --2- 172.21.15.16:0/466939049 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fda3415c1c0 0x7fda340cd4b0 secure :-1 s=READY pgs=1288 cs=0 l=1 rev1=1 crypto rx=0x7fda34068ba0 tx=0x7fda2801c420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=549fb68ab0954295 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:11:27.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.219+0000 7fda33400640 1 -- 172.21.15.16:0/466939049 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda280ab620 con 0x7fda3415c1c0 2024-03-31T15:11:27.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.219+0000 7fda33400640 1 -- 172.21.15.16:0/466939049 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fda280b0070 con 0x7fda3415c1c0 2024-03-31T15:11:27.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.219+0000 7fda33400640 1 -- 172.21.15.16:0/466939049 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda280ab9e0 con 0x7fda3415c1c0 2024-03-31T15:11:27.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.219+0000 7fda3a400640 1 -- 172.21.15.16:0/466939049 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fda3415c1c0 msgr2=0x7fda340cd4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:11:27.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.219+0000 7fda3a400640 1 --2- 172.21.15.16:0/466939049 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fda3415c1c0 0x7fda340cd4b0 secure :-1 s=READY pgs=1288 cs=0 l=1 rev1=1 crypto rx=0x7fda34068ba0 tx=0x7fda2801c420 comp rx=0 tx=0).stop 2024-03-31T15:11:27.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.219+0000 7fda3a400640 1 -- 172.21.15.16:0/466939049 shutdown_connections 2024-03-31T15:11:27.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.219+0000 7fda3a400640 1 --2- 172.21.15.16:0/466939049 >> v2:172.21.15.89:3300/0 conn(0x7fda340c5a80 0x7fda340c5ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:27.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.219+0000 7fda3a400640 1 --2- 172.21.15.16:0/466939049 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fda3415c1c0 0x7fda340cd4b0 unknown :-1 s=CLOSED pgs=1288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:27.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.219+0000 7fda3a400640 1 -- 172.21.15.16:0/466939049 >> 172.21.15.16:0/466939049 conn(0x7fda340d1f30 msgr2=0x7fda340d2340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:11:27.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.219+0000 7fda3a400640 1 -- 172.21.15.16:0/466939049 shutdown_connections 2024-03-31T15:11:27.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.219+0000 7fda3a400640 1 -- 172.21.15.16:0/466939049 wait complete. 2024-03-31T15:11:27.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.223+0000 7fda3a400640 1 Processor -- start 2024-03-31T15:11:27.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.223+0000 7fda3a400640 1 -- start start 2024-03-31T15:11:27.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.223+0000 7fda3a400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fda340c5a80 0x7fda34158100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:27.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.223+0000 7fda3a400640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fda3415c1c0 0x7fda34157b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:27.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.223+0000 7fda3a400640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fda3415c5a0 0x7fda340c9750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:27.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.223+0000 7fda3a400640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fda3409b270 con 0x7fda340c5a80 2024-03-31T15:11:27.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.223+0000 7fda3a400640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fda340d2da0 con 0x7fda3415c5a0 2024-03-31T15:11:27.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.223+0000 7fda3a400640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fda3409b3f0 con 0x7fda3415c1c0 2024-03-31T15:11:27.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.223+0000 7fda39000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fda340c5a80 0x7fda34158100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:27.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.223+0000 7fda39000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fda340c5a80 0x7fda34158100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50896/0 (socket says 172.21.15.16:50896) 2024-03-31T15:11:27.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.223+0000 7fda39000640 1 -- 172.21.15.16:0/3837683840 learned_addr learned my addr 172.21.15.16:0/3837683840 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:11:27.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.223+0000 7fda39a00640 1 --2- 172.21.15.16:0/3837683840 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fda3415c5a0 0x7fda340c9750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:27.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.223+0000 7fda33e00640 1 --2- 172.21.15.16:0/3837683840 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fda3415c1c0 0x7fda34157b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:27.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.223+0000 7fda39000640 1 -- 172.21.15.16:0/3837683840 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fda3415c1c0 msgr2=0x7fda34157b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:11:27.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.223+0000 7fda39000640 1 --2- 172.21.15.16:0/3837683840 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fda3415c1c0 0x7fda34157b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:27.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.223+0000 7fda39000640 1 -- 172.21.15.16:0/3837683840 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fda3415c5a0 msgr2=0x7fda340c9750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:11:27.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.223+0000 7fda39000640 1 --2- 172.21.15.16:0/3837683840 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fda3415c5a0 0x7fda340c9750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:27.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.223+0000 7fda39000640 1 -- 172.21.15.16:0/3837683840 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda340ca5a0 con 0x7fda340c5a80 2024-03-31T15:11:27.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.223+0000 7fda39000640 1 --2- 172.21.15.16:0/3837683840 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fda340c5a80 0x7fda34158100 secure :-1 s=READY pgs=1289 cs=0 l=1 rev1=1 crypto rx=0x7fda2801a050 tx=0x7fda2801ac60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:11:27.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.223+0000 7fda30c00640 1 -- 172.21.15.16:0/3837683840 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda280abd90 con 0x7fda340c5a80 2024-03-31T15:11:27.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.223+0000 7fda30c00640 1 -- 172.21.15.16:0/3837683840 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fda280be3f0 con 0x7fda340c5a80 2024-03-31T15:11:27.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.223+0000 7fda3a400640 1 -- 172.21.15.16:0/3837683840 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fda340b66e0 con 0x7fda340c5a80 2024-03-31T15:11:27.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.223+0000 7fda30c00640 1 -- 172.21.15.16:0/3837683840 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda280be7c0 con 0x7fda340c5a80 2024-03-31T15:11:27.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.223+0000 7fda3a400640 1 -- 172.21.15.16:0/3837683840 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fda340b6ca0 con 0x7fda340c5a80 2024-03-31T15:11:27.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.227+0000 7fda30c00640 1 -- 172.21.15.16:0/3837683840 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fda280c7430 con 0x7fda340c5a80 2024-03-31T15:11:27.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.227+0000 7fda30c00640 1 --2- 172.21.15.16:0/3837683840 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fda14068380 0x7fda1406a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:27.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.227+0000 7fda30c00640 1 -- 172.21.15.16:0/3837683840 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 5910+0+0 (secure 0 0 0) 0x7fda281210b0 con 0x7fda340c5a80 2024-03-31T15:11:27.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.227+0000 7fda33e00640 1 --2- 172.21.15.16:0/3837683840 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fda14068380 0x7fda1406a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:27.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.227+0000 7fda3ef02000 1 Processor -- start 2024-03-31T15:11:27.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.227+0000 7fda3ef02000 1 -- start start 2024-03-31T15:11:27.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.227+0000 7fda33e00640 1 --2- 172.21.15.16:0/3837683840 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fda14068380 0x7fda1406a840 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7fda3415baa0 tx=0x7fda20009060 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:11:27.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.227+0000 7fda3ef02000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556e898602d0 0x556e898606b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:27.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.227+0000 7fda3ef02000 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x556e89860bf0 0x556e89813470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:27.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.227+0000 7fda3ef02000 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556e898139b0 0x556e89813d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:27.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.227+0000 7fda3ef02000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x556e896fcaf0 con 0x556e898602d0 2024-03-31T15:11:27.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.227+0000 7fda3ef02000 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x556e89665b00 con 0x556e898139b0 2024-03-31T15:11:27.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.227+0000 7fda3ef02000 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x556e89602940 con 0x556e89860bf0 2024-03-31T15:11:27.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.227+0000 7fda39000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556e898602d0 0x556e898606b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:27.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.227+0000 7fda39000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556e898602d0 0x556e898606b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50904/0 (socket says 172.21.15.16:50904) 2024-03-31T15:11:27.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.227+0000 7fda39000640 1 -- 172.21.15.16:0/3839256754 learned_addr learned my addr 172.21.15.16:0/3839256754 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:11:27.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.227+0000 7fda33e00640 1 --2- 172.21.15.16:0/3839256754 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x556e89860bf0 0x556e89813470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:27.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.227+0000 7fda39a00640 1 --2- 172.21.15.16:0/3839256754 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556e898139b0 0x556e89813d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:27.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.231+0000 7fda39000640 1 -- 172.21.15.16:0/3839256754 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x556e89860bf0 msgr2=0x556e89813470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:11:27.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.231+0000 7fda39000640 1 --2- 172.21.15.16:0/3839256754 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x556e89860bf0 0x556e89813470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:27.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.231+0000 7fda39000640 1 -- 172.21.15.16:0/3839256754 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556e898139b0 msgr2=0x556e89813d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:11:27.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.231+0000 7fda39000640 1 --2- 172.21.15.16:0/3839256754 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556e898139b0 0x556e89813d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:27.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.231+0000 7fda39000640 1 -- 172.21.15.16:0/3839256754 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556e895eb380 con 0x556e898602d0 2024-03-31T15:11:27.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.231+0000 7fda39000640 1 --2- 172.21.15.16:0/3839256754 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556e898602d0 0x556e898606b0 secure :-1 s=READY pgs=1290 cs=0 l=1 rev1=1 crypto rx=0x7fda280f0c40 tx=0x7fda280f08a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=51ca96064f560be0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:11:27.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.231+0000 7fda25600640 1 -- 172.21.15.16:0/3839256754 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda28118e60 con 0x556e898602d0 2024-03-31T15:11:27.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.231+0000 7fda25600640 1 -- 172.21.15.16:0/3839256754 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fda280b0030 con 0x556e898602d0 2024-03-31T15:11:27.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.231+0000 7fda25600640 1 -- 172.21.15.16:0/3839256754 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda280f9a70 con 0x556e898602d0 2024-03-31T15:11:27.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.231+0000 7fda3ef02000 1 -- 172.21.15.16:0/3839256754 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556e898602d0 msgr2=0x556e898606b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:11:27.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.231+0000 7fda3ef02000 1 --2- 172.21.15.16:0/3839256754 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556e898602d0 0x556e898606b0 secure :-1 s=READY pgs=1290 cs=0 l=1 rev1=1 crypto rx=0x7fda280f0c40 tx=0x7fda280f08a0 comp rx=0 tx=0).stop 2024-03-31T15:11:27.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.231+0000 7fda3ef02000 1 -- 172.21.15.16:0/3839256754 shutdown_connections 2024-03-31T15:11:27.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.231+0000 7fda3ef02000 1 --2- 172.21.15.16:0/3839256754 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556e898139b0 0x556e89813d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:27.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.231+0000 7fda3ef02000 1 --2- 172.21.15.16:0/3839256754 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x556e89860bf0 0x556e89813470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:27.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.231+0000 7fda3ef02000 1 --2- 172.21.15.16:0/3839256754 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556e898602d0 0x556e898606b0 unknown :-1 s=CLOSED pgs=1290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:27.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.231+0000 7fda3ef02000 1 -- 172.21.15.16:0/3839256754 >> 172.21.15.16:0/3839256754 conn(0x556e89841070 msgr2=0x556e89843460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:11:27.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.231+0000 7fda3ef02000 1 -- 172.21.15.16:0/3839256754 shutdown_connections 2024-03-31T15:11:27.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.231+0000 7fda3ef02000 1 -- 172.21.15.16:0/3839256754 wait complete. 2024-03-31T15:11:27.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.231+0000 7fda3ef02000 1 -- start start 2024-03-31T15:11:27.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.231+0000 7fda3ef02000 1 Processor -- start 2024-03-31T15:11:27.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.231+0000 7fda3ef02000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556e898139b0 0x556e898187b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:27.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.231+0000 7fda3ef02000 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x556e898602d0 0x556e89818cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:27.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.231+0000 7fda3ef02000 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556e89860bf0 0x556e89819230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:27.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.231+0000 7fda3ef02000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x556e896231e0 con 0x556e898139b0 2024-03-31T15:11:27.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.231+0000 7fda3ef02000 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x556e89661ab0 con 0x556e89860bf0 2024-03-31T15:11:27.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.231+0000 7fda3ef02000 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x556e896fcaf0 con 0x556e898602d0 2024-03-31T15:11:27.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.231+0000 7fda3ef02000 10 client.0 _ll_register_callbacks cb 0x556e89801370 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T15:11:27.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.231+0000 7fda39000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556e898139b0 0x556e898187b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:27.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.231+0000 7fda39000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556e898139b0 0x556e898187b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50908/0 (socket says 172.21.15.16:50908) 2024-03-31T15:11:27.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.231+0000 7fda39000640 1 -- 172.21.15.16:0/2805451524 learned_addr learned my addr 172.21.15.16:0/2805451524 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:11:27.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.231+0000 7fda39a00640 1 --2- 172.21.15.16:0/2805451524 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556e89860bf0 0x556e89819230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:27.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.231+0000 7fda33e00640 1 --2- 172.21.15.16:0/2805451524 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x556e898602d0 0x556e89818cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:27.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.235+0000 7fda39000640 1 -- 172.21.15.16:0/2805451524 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x556e898602d0 msgr2=0x556e89818cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:11:27.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.235+0000 7fda39000640 1 --2- 172.21.15.16:0/2805451524 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x556e898602d0 0x556e89818cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:27.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.235+0000 7fda39000640 1 -- 172.21.15.16:0/2805451524 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556e89860bf0 msgr2=0x556e89819230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:11:27.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.235+0000 7fda39000640 1 --2- 172.21.15.16:0/2805451524 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556e89860bf0 0x556e89819230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:27.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.235+0000 7fda39000640 1 -- 172.21.15.16:0/2805451524 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556e89621b70 con 0x556e898139b0 2024-03-31T15:11:27.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.235+0000 7fda33e00640 1 --2- 172.21.15.16:0/2805451524 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x556e898602d0 0x556e89818cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:11:27.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.235+0000 7fda39000640 1 --2- 172.21.15.16:0/2805451524 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556e898139b0 0x556e898187b0 secure :-1 s=READY pgs=1291 cs=0 l=1 rev1=1 crypto rx=0x7fda280f1100 tx=0x7fda28000bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e68208f72c32869 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:11:27.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.235+0000 7fda1a000640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-31T15:11:27.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.235+0000 7fda1a000640 1 -- 172.21.15.16:0/2805451524 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda2811c050 con 0x556e898139b0 2024-03-31T15:11:27.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.235+0000 7fda1a000640 1 -- 172.21.15.16:0/2805451524 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 5910+0+0 (secure 0 0 0) 0x7fda2810adf0 con 0x556e898139b0 2024-03-31T15:11:27.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.235+0000 7fda3ef02000 1 -- 172.21.15.16:0/2805451524 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fda280b0030 con 0x556e898139b0 2024-03-31T15:11:27.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.235+0000 7fda1a000640 1 -- 172.21.15.16:0/2805451524 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=74) ==== 24+0+0 (secure 0 0 0) 0x7fda28118e60 con 0x556e898139b0 2024-03-31T15:11:27.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.235+0000 7fda3ef02000 10 client.8217 fetch_fsmap learned FSMap version 74 2024-03-31T15:11:27.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.235+0000 7fda3ef02000 1 -- 172.21.15.16:0/2805451524 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=74,monmap=2+}) -- 0x556e89602b90 con 0x556e898139b0 2024-03-31T15:11:27.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.235+0000 7fda1a000640 1 -- 172.21.15.16:0/2805451524 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fda280b0030 con 0x556e898139b0 2024-03-31T15:11:27.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.235+0000 7fda1a000640 1 -- 172.21.15.16:0/2805451524 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 74) ==== 3494+0+0 (secure 0 0 0) 0x7fda280f0390 con 0x556e898139b0 2024-03-31T15:11:27.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.235+0000 7fda3ef02000 10 client.8217 fetch_fsmap finished waiting for FSMap version 74 2024-03-31T15:11:27.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.235+0000 7fda3ef02000 10 client.8217 resolve_mds: resolved 6:0 to role '6:0' aka daemon mds.c 2024-03-31T15:11:27.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.235+0000 7fda3ef02000 20 client.8217 populate_metadata read hostname 'smithi016' 2024-03-31T15:11:27.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.235+0000 7fda3ef02000 1 --2- 172.21.15.16:0/2805451524 >> [v2:172.21.15.16:6832/3780866829,v1:172.21.15.16:6834/3780866829] conn(0x556e8986f360 0x556e89871750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:27.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.235+0000 7fda3ef02000 4 client.8217 mds_command: new command op to 8085 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-31T15:11:27.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.235+0000 7fda3ef02000 1 -- 172.21.15.16:0/2805451524 --> [v2:172.21.15.16:6832/3780866829,v1:172.21.15.16:6834/3780866829] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556e89600dd0 con 0x556e8986f360 2024-03-31T15:11:27.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.235+0000 7fda33e00640 1 --2- 172.21.15.16:0/2805451524 >> [v2:172.21.15.16:6832/3780866829,v1:172.21.15.16:6834/3780866829] conn(0x556e8986f360 0x556e89871750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:27.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.235+0000 7fda33e00640 1 --2- 172.21.15.16:0/2805451524 >> [v2:172.21.15.16:6832/3780866829,v1:172.21.15.16:6834/3780866829] conn(0x556e8986f360 0x556e89871750 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cc759d9049a39dd6 server_cookie=3fc9d393379c67de in_seq=0 out_seq=0 2024-03-31T15:11:27.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.235+0000 7fda1a000640 10 client.8217 ms_handle_connect on v2:172.21.15.16:6832/3780866829 2024-03-31T15:11:27.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.235+0000 7fda1a000640 1 -- 172.21.15.16:0/2805451524 <== mds.0 v2:172.21.15.16:6832/3780866829 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x556e89600dd0 con 0x556e8986f360 2024-03-31T15:11:27.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.235+0000 7fda1a000640 10 client.8217 handle_command_reply: tid=0 2024-03-31T15:11:27.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.235+0000 7fda3ef02000 1 client.8217 shutdown 2024-03-31T15:11:27.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.235+0000 7fda3ef02000 1 -- 172.21.15.16:0/2805451524 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556e898139b0 msgr2=0x556e898187b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:11:27.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.235+0000 7fda3ef02000 1 --2- 172.21.15.16:0/2805451524 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556e898139b0 0x556e898187b0 secure :-1 s=READY pgs=1291 cs=0 l=1 rev1=1 crypto rx=0x7fda280f1100 tx=0x7fda28000bd0 comp rx=0 tx=0).stop 2024-03-31T15:11:27.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.235+0000 7fda3ef02000 1 -- 172.21.15.16:0/2805451524 shutdown_connections 2024-03-31T15:11:27.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.235+0000 7fda3ef02000 1 --2- 172.21.15.16:0/2805451524 >> [v2:172.21.15.16:6832/3780866829,v1:172.21.15.16:6834/3780866829] conn(0x556e8986f360 0x556e89871750 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:27.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.235+0000 7fda3ef02000 1 --2- 172.21.15.16:0/2805451524 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556e89860bf0 0x556e89819230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:27.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.235+0000 7fda3ef02000 1 --2- 172.21.15.16:0/2805451524 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x556e898602d0 0x556e89818cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:27.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.235+0000 7fda3ef02000 1 --2- 172.21.15.16:0/2805451524 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556e898139b0 0x556e898187b0 unknown :-1 s=CLOSED pgs=1291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:27.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.235+0000 7fda3ef02000 1 -- 172.21.15.16:0/2805451524 >> 172.21.15.16:0/2805451524 conn(0x556e89841070 msgr2=0x556e89843460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:11:27.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.235+0000 7fda1a000640 0 client.8217 ms_handle_reset on v2:172.21.15.16:6832/3780866829 2024-03-31T15:11:27.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.235+0000 7fda3ef02000 1 -- 172.21.15.16:0/2805451524 shutdown_connections 2024-03-31T15:11:27.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.235+0000 7fda3ef02000 1 -- 172.21.15.16:0/2805451524 wait complete. 2024-03-31T15:11:27.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.235+0000 7fda3ef02000 20 client.8217 trim_cache size 0 max 16384 2024-03-31T15:11:27.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.255+0000 7fda3ef02000 1 Processor -- start 2024-03-31T15:11:27.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.255+0000 7fda3ef02000 1 -- start start 2024-03-31T15:11:27.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.255+0000 7fda3ef02000 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556e898139b0 0x556e89812b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:27.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.255+0000 7fda3ef02000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556e898602d0 0x556e898130b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:27.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.255+0000 7fda3ef02000 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x556e89860bf0 0x556e8985e380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:27.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.255+0000 7fda3ef02000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fda34203610 con 0x556e898602d0 2024-03-31T15:11:27.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.255+0000 7fda3ef02000 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fda3409e140 con 0x556e898139b0 2024-03-31T15:11:27.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.255+0000 7fda3ef02000 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fda340cbc70 con 0x556e89860bf0 2024-03-31T15:11:27.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.255+0000 7fda33e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556e898602d0 0x556e898130b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:27.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.255+0000 7fda33e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556e898602d0 0x556e898130b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50914/0 (socket says 172.21.15.16:50914) 2024-03-31T15:11:27.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.255+0000 7fda33e00640 1 -- 172.21.15.16:0/1153066107 learned_addr learned my addr 172.21.15.16:0/1153066107 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:11:27.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.255+0000 7fda39a00640 1 --2- 172.21.15.16:0/1153066107 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x556e89860bf0 0x556e8985e380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:27.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.255+0000 7fda39000640 1 --2- 172.21.15.16:0/1153066107 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556e898139b0 0x556e89812b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:27.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.255+0000 7fda33e00640 1 -- 172.21.15.16:0/1153066107 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x556e89860bf0 msgr2=0x556e8985e380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:11:27.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.255+0000 7fda33e00640 1 --2- 172.21.15.16:0/1153066107 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x556e89860bf0 0x556e8985e380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:27.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.255+0000 7fda33e00640 1 -- 172.21.15.16:0/1153066107 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556e898139b0 msgr2=0x556e89812b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:11:27.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.255+0000 7fda33e00640 1 --2- 172.21.15.16:0/1153066107 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556e898139b0 0x556e89812b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:27.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.255+0000 7fda33e00640 1 -- 172.21.15.16:0/1153066107 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556e896e6140 con 0x556e898602d0 2024-03-31T15:11:27.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.255+0000 7fda33e00640 1 --2- 172.21.15.16:0/1153066107 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556e898602d0 0x556e898130b0 secure :-1 s=READY pgs=1292 cs=0 l=1 rev1=1 crypto rx=0x556e8981f150 tx=0x7fda20015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e56abd97eff5053 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:11:27.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.255+0000 7fda24c00640 1 -- 172.21.15.16:0/1153066107 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda2001b040 con 0x556e898602d0 2024-03-31T15:11:27.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.255+0000 7fda24c00640 1 -- 172.21.15.16:0/1153066107 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fda2000edc0 con 0x556e898602d0 2024-03-31T15:11:27.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.255+0000 7fda24c00640 1 -- 172.21.15.16:0/1153066107 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda20004020 con 0x556e898602d0 2024-03-31T15:11:27.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.255+0000 7fda3ef02000 1 -- 172.21.15.16:0/1153066107 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556e898602d0 msgr2=0x556e898130b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:11:27.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.255+0000 7fda3ef02000 1 --2- 172.21.15.16:0/1153066107 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556e898602d0 0x556e898130b0 secure :-1 s=READY pgs=1292 cs=0 l=1 rev1=1 crypto rx=0x556e8981f150 tx=0x7fda20015000 comp rx=0 tx=0).stop 2024-03-31T15:11:27.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.255+0000 7fda3ef02000 1 -- 172.21.15.16:0/1153066107 shutdown_connections 2024-03-31T15:11:27.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.255+0000 7fda3ef02000 1 --2- 172.21.15.16:0/1153066107 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x556e89860bf0 0x556e8985e380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:27.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.255+0000 7fda3ef02000 1 --2- 172.21.15.16:0/1153066107 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556e898602d0 0x556e898130b0 unknown :-1 s=CLOSED pgs=1292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:27.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.255+0000 7fda3ef02000 1 --2- 172.21.15.16:0/1153066107 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556e898139b0 0x556e89812b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:27.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.255+0000 7fda3ef02000 1 -- 172.21.15.16:0/1153066107 >> 172.21.15.16:0/1153066107 conn(0x556e89841070 msgr2=0x556e89812460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:11:27.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.255+0000 7fda3ef02000 1 -- 172.21.15.16:0/1153066107 shutdown_connections 2024-03-31T15:11:27.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.255+0000 7fda3ef02000 1 -- 172.21.15.16:0/1153066107 wait complete. 2024-03-31T15:11:27.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.259+0000 7fda3ef02000 1 -- start start 2024-03-31T15:11:27.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.259+0000 7fda3ef02000 1 Processor -- start 2024-03-31T15:11:27.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.259+0000 7fda3ef02000 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556e898139b0 0x556e898cfa50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:27.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.259+0000 7fda3ef02000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556e898602d0 0x556e898cff90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:27.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.259+0000 7fda3ef02000 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x556e89860bf0 0x556e898723d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:27.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.259+0000 7fda3ef02000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fda3409daf0 con 0x556e898602d0 2024-03-31T15:11:27.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.259+0000 7fda3ef02000 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x556e89665b00 con 0x556e898139b0 2024-03-31T15:11:27.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.259+0000 7fda3ef02000 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x556e896231e0 con 0x556e89860bf0 2024-03-31T15:11:27.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.259+0000 7fda3ef02000 10 client.0 _ll_register_callbacks cb 0x7fda2c009bd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T15:11:27.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.259+0000 7fda33e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556e898602d0 0x556e898cff90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:27.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.259+0000 7fda39000640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556e898139b0 0x556e898cfa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:27.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.259+0000 7fda33e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556e898602d0 0x556e898cff90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50918/0 (socket says 172.21.15.16:50918) 2024-03-31T15:11:27.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.259+0000 7fda39a00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x556e89860bf0 0x556e898723d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:27.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.259+0000 7fda33e00640 1 -- 172.21.15.16:0/1055401800 learned_addr learned my addr 172.21.15.16:0/1055401800 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:11:27.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.259+0000 7fda33e00640 1 -- 172.21.15.16:0/1055401800 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x556e89860bf0 msgr2=0x556e898723d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:11:27.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.259+0000 7fda33e00640 1 --2- 172.21.15.16:0/1055401800 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x556e89860bf0 0x556e898723d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:27.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.259+0000 7fda33e00640 1 -- 172.21.15.16:0/1055401800 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556e898139b0 msgr2=0x556e898cfa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:11:27.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.259+0000 7fda33e00640 1 --2- 172.21.15.16:0/1055401800 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556e898139b0 0x556e898cfa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:27.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.259+0000 7fda33e00640 1 -- 172.21.15.16:0/1055401800 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556e895f41e0 con 0x556e898602d0 2024-03-31T15:11:27.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.259+0000 7fda33e00640 1 --2- 172.21.15.16:0/1055401800 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556e898602d0 0x556e898cff90 secure :-1 s=READY pgs=1293 cs=0 l=1 rev1=1 crypto rx=0x7fda2002c040 tx=0x7fda20004690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c5b56056f6e1a04 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:11:27.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.259+0000 7fda1a000640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-31T15:11:27.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.259+0000 7fda1a000640 1 -- 172.21.15.16:0/1055401800 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda20039060 con 0x556e898602d0 2024-03-31T15:11:27.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.259+0000 7fda1a000640 1 -- 172.21.15.16:0/1055401800 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 5910+0+0 (secure 0 0 0) 0x7fda2001b040 con 0x556e898602d0 2024-03-31T15:11:27.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.259+0000 7fda3ef02000 1 -- 172.21.15.16:0/1055401800 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fda2000edc0 con 0x556e898602d0 2024-03-31T15:11:27.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.259+0000 7fda1a000640 1 -- 172.21.15.16:0/1055401800 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=74) ==== 24+0+0 (secure 0 0 0) 0x7fda2003f050 con 0x556e898602d0 2024-03-31T15:11:27.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.259+0000 7fda3ef02000 10 client.8223 fetch_fsmap learned FSMap version 74 2024-03-31T15:11:27.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.259+0000 7fda3ef02000 1 -- 172.21.15.16:0/1055401800 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=74,monmap=2+}) -- 0x556e895f97b0 con 0x556e898602d0 2024-03-31T15:11:27.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.259+0000 7fda1a000640 1 -- 172.21.15.16:0/1055401800 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fda2000edc0 con 0x556e898602d0 2024-03-31T15:11:27.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.259+0000 7fda1a000640 1 -- 172.21.15.16:0/1055401800 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 74) ==== 3494+0+0 (secure 0 0 0) 0x7fda20042020 con 0x556e898602d0 2024-03-31T15:11:27.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.259+0000 7fda3ef02000 10 client.8223 fetch_fsmap finished waiting for FSMap version 74 2024-03-31T15:11:27.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.259+0000 7fda3ef02000 10 client.8223 resolve_mds: resolved 6:0 to role '6:0' aka daemon mds.c 2024-03-31T15:11:27.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.259+0000 7fda3ef02000 20 client.8223 populate_metadata read hostname 'smithi016' 2024-03-31T15:11:27.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.259+0000 7fda3ef02000 1 --2- 172.21.15.16:0/1055401800 >> [v2:172.21.15.16:6832/3780866829,v1:172.21.15.16:6834/3780866829] conn(0x556e89872d90 0x556e898731a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:27.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.259+0000 7fda3ef02000 4 client.8223 mds_command: new command op to 8085 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-31T15:11:27.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.259+0000 7fda3ef02000 1 -- 172.21.15.16:0/1055401800 --> [v2:172.21.15.16:6832/3780866829,v1:172.21.15.16:6834/3780866829] -- command(tid 0: {"prefix": "scrub status"}) -- 0x556e89673040 con 0x556e89872d90 2024-03-31T15:11:27.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.259+0000 7fda39000640 1 --2- 172.21.15.16:0/1055401800 >> [v2:172.21.15.16:6832/3780866829,v1:172.21.15.16:6834/3780866829] conn(0x556e89872d90 0x556e898731a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:27.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.263+0000 7fda1a000640 10 client.8223 ms_handle_connect on v2:172.21.15.16:6832/3780866829 2024-03-31T15:11:27.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.263+0000 7fda39000640 1 --2- 172.21.15.16:0/1055401800 >> [v2:172.21.15.16:6832/3780866829,v1:172.21.15.16:6834/3780866829] conn(0x556e89872d90 0x556e898731a0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=430f3d072ea38468 server_cookie=3e251804f204437d in_seq=0 out_seq=0 2024-03-31T15:11:27.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.263+0000 7fda1a000640 1 -- 172.21.15.16:0/1055401800 <== mds.0 v2:172.21.15.16:6832/3780866829 1 ==== command_reply(tid 0: 0 ) ==== 8+0+263 (crc 0 0 0) 0x556e89673040 con 0x556e89872d90 2024-03-31T15:11:27.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.263+0000 7fda1a000640 10 client.8223 handle_command_reply: tid=0 2024-03-31T15:11:27.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.263+0000 7fda3ef02000 1 client.8223 shutdown 2024-03-31T15:11:27.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.263+0000 7fda3ef02000 1 -- 172.21.15.16:0/1055401800 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556e898602d0 msgr2=0x556e898cff90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:11:27.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.263+0000 7fda3ef02000 1 --2- 172.21.15.16:0/1055401800 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556e898602d0 0x556e898cff90 secure :-1 s=READY pgs=1293 cs=0 l=1 rev1=1 crypto rx=0x7fda2002c040 tx=0x7fda20004690 comp rx=0 tx=0).stop 2024-03-31T15:11:27.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.263+0000 7fda3ef02000 1 -- 172.21.15.16:0/1055401800 shutdown_connections 2024-03-31T15:11:27.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.263+0000 7fda3ef02000 1 --2- 172.21.15.16:0/1055401800 >> [v2:172.21.15.16:6832/3780866829,v1:172.21.15.16:6834/3780866829] conn(0x556e89872d90 0x556e898731a0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:27.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.263+0000 7fda3ef02000 1 --2- 172.21.15.16:0/1055401800 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x556e89860bf0 0x556e898723d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:27.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.263+0000 7fda3ef02000 1 --2- 172.21.15.16:0/1055401800 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556e898602d0 0x556e898cff90 unknown :-1 s=CLOSED pgs=1293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:27.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.263+0000 7fda1a000640 0 client.8223 ms_handle_reset on v2:172.21.15.16:6832/3780866829 2024-03-31T15:11:27.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.263+0000 7fda3ef02000 1 --2- 172.21.15.16:0/1055401800 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556e898139b0 0x556e898cfa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:27.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.263+0000 7fda3ef02000 1 -- 172.21.15.16:0/1055401800 >> 172.21.15.16:0/1055401800 conn(0x556e89841070 msgr2=0x556e89812b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:11:27.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.263+0000 7fda3ef02000 1 -- 172.21.15.16:0/1055401800 shutdown_connections 2024-03-31T15:11:27.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.263+0000 7fda3ef02000 1 -- 172.21.15.16:0/1055401800 wait complete. 2024-03-31T15:11:27.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.263+0000 7fda3ef02000 20 client.8223 trim_cache size 0 max 16384 2024-03-31T15:11:27.259 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-03-31T15:11:27.260 INFO:teuthology.orchestra.run.smithi016.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-31T15:11:27.260 INFO:teuthology.orchestra.run.smithi016.stdout: "scrubs": { 2024-03-31T15:11:27.260 INFO:teuthology.orchestra.run.smithi016.stdout: "d7e59e70-33c2-4498-98fb-4555efd5e0c1": { 2024-03-31T15:11:27.260 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/dir", 2024-03-31T15:11:27.260 INFO:teuthology.orchestra.run.smithi016.stdout: "tag": "d7e59e70-33c2-4498-98fb-4555efd5e0c1", 2024-03-31T15:11:27.260 INFO:teuthology.orchestra.run.smithi016.stdout: "options": "recursive,repair" 2024-03-31T15:11:27.260 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-31T15:11:27.260 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-31T15:11:27.260 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-03-31T15:11:27.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.263+0000 7fda3a400640 1 -- 172.21.15.16:0/3837683840 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fda14068380 msgr2=0x7fda1406a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:11:27.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.263+0000 7fda3a400640 1 --2- 172.21.15.16:0/3837683840 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fda14068380 0x7fda1406a840 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7fda3415baa0 tx=0x7fda20009060 comp rx=0 tx=0).stop 2024-03-31T15:11:27.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.263+0000 7fda3a400640 1 -- 172.21.15.16:0/3837683840 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fda340c5a80 msgr2=0x7fda34158100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:11:27.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.263+0000 7fda3a400640 1 --2- 172.21.15.16:0/3837683840 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fda340c5a80 0x7fda34158100 secure :-1 s=READY pgs=1289 cs=0 l=1 rev1=1 crypto rx=0x7fda2801a050 tx=0x7fda2801ac60 comp rx=0 tx=0).stop 2024-03-31T15:11:27.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.263+0000 7fda3a400640 1 -- 172.21.15.16:0/3837683840 shutdown_connections 2024-03-31T15:11:27.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.263+0000 7fda3a400640 1 --2- 172.21.15.16:0/3837683840 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fda14068380 0x7fda1406a840 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:27.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.263+0000 7fda3a400640 1 --2- 172.21.15.16:0/3837683840 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fda3415c5a0 0x7fda340c9750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:27.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.263+0000 7fda3a400640 1 --2- 172.21.15.16:0/3837683840 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fda3415c1c0 0x7fda34157b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:27.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.263+0000 7fda3a400640 1 --2- 172.21.15.16:0/3837683840 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fda340c5a80 0x7fda34158100 unknown :-1 s=CLOSED pgs=1289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:27.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.263+0000 7fda3a400640 1 -- 172.21.15.16:0/3837683840 >> 172.21.15.16:0/3837683840 conn(0x7fda340d1f30 msgr2=0x7fda3415b010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:11:27.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.263+0000 7fda3a400640 1 -- 172.21.15.16:0/3837683840 shutdown_connections 2024-03-31T15:11:27.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:27.263+0000 7fda3a400640 1 -- 172.21.15.16:0/3837683840 wait complete. 2024-03-31T15:11:27.275 INFO:tasks.cephfs.filesystem:scrub status for tag:d7e59e70-33c2-4498-98fb-4555efd5e0c1 - {'path': '/dir', 'tag': 'd7e59e70-33c2-4498-98fb-4555efd5e0c1', 'options': 'recursive,repair'} 2024-03-31T15:11:37.514 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-31T15:11:37.517 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-31T15:11:57.276 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:11:57.286 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.6:0 scrub status 2024-03-31T15:11:57.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.435+0000 7f4a51e00640 1 Processor -- start 2024-03-31T15:11:57.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.435+0000 7f4a51e00640 1 -- start start 2024-03-31T15:11:57.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.439+0000 7f4a51e00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f4a4c06f920 0x7f4a4c06fd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:57.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.439+0000 7f4a51e00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f4a4c0cf150 0x7f4a4c1419c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:57.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.439+0000 7f4a51e00640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4a4c070240 con 0x7f4a4c0d0a60 2024-03-31T15:11:57.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.439+0000 7f4a51e00640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f4a4c13f670 con 0x7f4a4c06f920 2024-03-31T15:11:57.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.439+0000 7f4a51e00640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f4a4c15ca70 con 0x7f4a4c0cf150 2024-03-31T15:11:57.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.439+0000 7f4a4aa00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f4a4c06f920 0x7f4a4c06fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:57.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.439+0000 7f4a4aa00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f4a4c06f920 0x7f4a4c06fd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48666/0 (socket says 172.21.15.16:48666) 2024-03-31T15:11:57.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.439+0000 7f4a4aa00640 1 -- 172.21.15.16:0/1950061305 learned_addr learned my addr 172.21.15.16:0/1950061305 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:11:57.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.439+0000 7f4a42000640 1 --1- 172.21.15.16:0/1950061305 >> v1:172.21.15.16:6789/0 conn(0x7f4a4c0d0a60 0x7f4a4c0cea40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:35038/0 (socket says 172.21.15.16:35038) 2024-03-31T15:11:57.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.439+0000 7f4a4aa00640 1 -- 172.21.15.16:0/1950061305 >> v2:172.21.15.89:3300/0 conn(0x7f4a4c0cf150 msgr2=0x7f4a4c1419c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:11:57.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.439+0000 7f4a4b400640 1 --2- 172.21.15.16:0/1950061305 >> v2:172.21.15.89:3300/0 conn(0x7f4a4c0cf150 0x7f4a4c1419c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:57.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.439+0000 7f4a4aa00640 1 --2- 172.21.15.16:0/1950061305 >> v2:172.21.15.89:3300/0 conn(0x7f4a4c0cf150 0x7f4a4c1419c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:57.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.439+0000 7f4a4aa00640 1 -- 172.21.15.16:0/1950061305 >> v1:172.21.15.16:6789/0 conn(0x7f4a4c0d0a60 legacy=0x7f4a4c0cea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:11:57.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.439+0000 7f4a4aa00640 1 -- 172.21.15.16:0/1950061305 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a4c15cd50 con 0x7f4a4c06f920 2024-03-31T15:11:57.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.439+0000 7f4a4a000640 1 -- 172.21.15.16:0/1950061305 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 99686722 0 0) 0x7f4a4c070240 con 0x7f4a4c0d0a60 2024-03-31T15:11:57.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.439+0000 7f4a4b400640 1 --2- 172.21.15.16:0/1950061305 >> v2:172.21.15.89:3300/0 conn(0x7f4a4c0cf150 0x7f4a4c1419c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:11:57.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.443+0000 7f4a4aa00640 1 --2- 172.21.15.16:0/1950061305 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4a4c06f920 0x7f4a4c06fd00 secure :-1 s=READY pgs=1295 cs=0 l=1 rev1=1 crypto rx=0x7f4a4c0d2080 tx=0x7f4a3c01c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=74d3135cc7edeff6 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:11:57.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.443+0000 7f4a4a000640 1 -- 172.21.15.16:0/1950061305 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a3c0b0070 con 0x7f4a4c06f920 2024-03-31T15:11:57.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.443+0000 7f4a4a000640 1 -- 172.21.15.16:0/1950061305 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4a3c0ab600 con 0x7f4a4c06f920 2024-03-31T15:11:57.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.443+0000 7f4a4a000640 1 -- 172.21.15.16:0/1950061305 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a3c0ab9d0 con 0x7f4a4c06f920 2024-03-31T15:11:57.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.443+0000 7f4a51e00640 1 -- 172.21.15.16:0/1950061305 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4a4c06f920 msgr2=0x7f4a4c06fd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:11:57.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.443+0000 7f4a51e00640 1 --2- 172.21.15.16:0/1950061305 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4a4c06f920 0x7f4a4c06fd00 secure :-1 s=READY pgs=1295 cs=0 l=1 rev1=1 crypto rx=0x7f4a4c0d2080 tx=0x7f4a3c01c490 comp rx=0 tx=0).stop 2024-03-31T15:11:57.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.443+0000 7f4a51e00640 1 -- 172.21.15.16:0/1950061305 shutdown_connections 2024-03-31T15:11:57.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.443+0000 7f4a51e00640 1 --2- 172.21.15.16:0/1950061305 >> v2:172.21.15.89:3300/0 conn(0x7f4a4c0cf150 0x7f4a4c1419c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:57.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.443+0000 7f4a51e00640 1 --2- 172.21.15.16:0/1950061305 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4a4c06f920 0x7f4a4c06fd00 unknown :-1 s=CLOSED pgs=1295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:57.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.443+0000 7f4a51e00640 1 -- 172.21.15.16:0/1950061305 >> 172.21.15.16:0/1950061305 conn(0x7f4a4c067930 msgr2=0x7f4a4c0b3ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:11:57.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.443+0000 7f4a51e00640 1 -- 172.21.15.16:0/1950061305 shutdown_connections 2024-03-31T15:11:57.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.443+0000 7f4a51e00640 1 -- 172.21.15.16:0/1950061305 wait complete. 2024-03-31T15:11:57.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.443+0000 7f4a51e00640 1 Processor -- start 2024-03-31T15:11:57.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.443+0000 7f4a51e00640 1 -- start start 2024-03-31T15:11:57.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.443+0000 7f4a51e00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f4a4c06f920 0x7f4a4c0b6500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:57.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.447+0000 7f4a51e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4a4c0cf150 0x7f4a4c0b5300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:57.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.447+0000 7f4a51e00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4a4c0d0a60 0x7f4a4c0b5840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:57.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.447+0000 7f4a51e00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f4a4c169230 con 0x7f4a4c0cf150 2024-03-31T15:11:57.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.447+0000 7f4a51e00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f4a4c1690b0 con 0x7f4a4c0d0a60 2024-03-31T15:11:57.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.447+0000 7f4a51e00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f4a4c1693b0 con 0x7f4a4c06f920 2024-03-31T15:11:57.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.447+0000 7f4a42000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4a4c0cf150 0x7f4a4c0b5300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:57.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.447+0000 7f4a42000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4a4c0cf150 0x7f4a4c0b5300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48682/0 (socket says 172.21.15.16:48682) 2024-03-31T15:11:57.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.447+0000 7f4a42000640 1 -- 172.21.15.16:0/806068095 learned_addr learned my addr 172.21.15.16:0/806068095 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:11:57.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.447+0000 7f4a4b400640 1 --2- 172.21.15.16:0/806068095 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4a4c0d0a60 0x7f4a4c0b5840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:57.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.447+0000 7f4a4aa00640 1 --2- 172.21.15.16:0/806068095 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f4a4c06f920 0x7f4a4c0b6500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:57.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.447+0000 7f4a42000640 1 -- 172.21.15.16:0/806068095 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f4a4c06f920 msgr2=0x7f4a4c0b6500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:11:57.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.447+0000 7f4a42000640 1 --2- 172.21.15.16:0/806068095 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f4a4c06f920 0x7f4a4c0b6500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:57.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.447+0000 7f4a42000640 1 -- 172.21.15.16:0/806068095 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4a4c0d0a60 msgr2=0x7f4a4c0b5840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:11:57.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.447+0000 7f4a42000640 1 --2- 172.21.15.16:0/806068095 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4a4c0d0a60 0x7f4a4c0b5840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:57.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.447+0000 7f4a42000640 1 -- 172.21.15.16:0/806068095 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a4c0b2510 con 0x7f4a4c0cf150 2024-03-31T15:11:57.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.447+0000 7f4a42000640 1 --2- 172.21.15.16:0/806068095 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4a4c0cf150 0x7f4a4c0b5300 secure :-1 s=READY pgs=1296 cs=0 l=1 rev1=1 crypto rx=0x7f4a3400c450 tx=0x7f4a3400c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:11:57.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.447+0000 7f4a43e00640 1 -- 172.21.15.16:0/806068095 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a34018070 con 0x7f4a4c0cf150 2024-03-31T15:11:57.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.447+0000 7f4a43e00640 1 -- 172.21.15.16:0/806068095 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4a340074e0 con 0x7f4a4c0cf150 2024-03-31T15:11:57.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.447+0000 7f4a43e00640 1 -- 172.21.15.16:0/806068095 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a340049f0 con 0x7f4a4c0cf150 2024-03-31T15:11:57.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.447+0000 7f4a51e00640 1 -- 172.21.15.16:0/806068095 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4a4c0b27f0 con 0x7f4a4c0cf150 2024-03-31T15:11:57.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.447+0000 7f4a51e00640 1 -- 172.21.15.16:0/806068095 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4a4c0b2db0 con 0x7f4a4c0cf150 2024-03-31T15:11:57.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.447+0000 7f4a43e00640 1 -- 172.21.15.16:0/806068095 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f4a3401b070 con 0x7f4a4c0cf150 2024-03-31T15:11:57.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.451+0000 7f4a55318000 1 Processor -- start 2024-03-31T15:11:57.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.451+0000 7f4a55318000 1 -- start start 2024-03-31T15:11:57.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.451+0000 7f4a43e00640 1 --2- 172.21.15.16:0/806068095 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f4a28068380 0x7f4a2806a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:57.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.451+0000 7f4a43e00640 1 -- 172.21.15.16:0/806068095 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 5910+0+0 (secure 0 0 0) 0x7f4a34085d60 con 0x7f4a4c0cf150 2024-03-31T15:11:57.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.451+0000 7f4a4b400640 1 --2- 172.21.15.16:0/806068095 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f4a28068380 0x7f4a2806a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:57.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.451+0000 7f4a55318000 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555f586dc2d0 0x555f586dc6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:57.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.451+0000 7f4a55318000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555f586dcbf0 0x555f5868f470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:57.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.451+0000 7f4a55318000 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x555f5868f9b0 0x555f5868fd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:57.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.451+0000 7f4a55318000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x555f58578af0 con 0x555f586dcbf0 2024-03-31T15:11:57.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.451+0000 7f4a55318000 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x555f584e1b00 con 0x555f586dc2d0 2024-03-31T15:11:57.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.451+0000 7f4a55318000 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x555f5847e940 con 0x555f5868f9b0 2024-03-31T15:11:57.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.451+0000 7f4a42000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555f586dcbf0 0x555f5868f470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:57.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.451+0000 7f4a42000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555f586dcbf0 0x555f5868f470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48688/0 (socket says 172.21.15.16:48688) 2024-03-31T15:11:57.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.451+0000 7f4a42000640 1 -- 172.21.15.16:0/4035903528 learned_addr learned my addr 172.21.15.16:0/4035903528 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:11:57.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.451+0000 7f4a4aa00640 1 --2- 172.21.15.16:0/4035903528 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555f586dc2d0 0x555f586dc6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:57.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.451+0000 7f4a4b400640 1 --2- 172.21.15.16:0/4035903528 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x555f5868f9b0 0x555f5868fd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:57.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.451+0000 7f4a42000640 1 -- 172.21.15.16:0/4035903528 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x555f5868f9b0 msgr2=0x555f5868fd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:11:57.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.451+0000 7f4a42000640 1 --2- 172.21.15.16:0/4035903528 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x555f5868f9b0 0x555f5868fd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:57.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.451+0000 7f4a42000640 1 -- 172.21.15.16:0/4035903528 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555f586dc2d0 msgr2=0x555f586dc6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:11:57.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.451+0000 7f4a42000640 1 --2- 172.21.15.16:0/4035903528 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555f586dc2d0 0x555f586dc6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:57.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.451+0000 7f4a42000640 1 -- 172.21.15.16:0/4035903528 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555f58467380 con 0x555f586dcbf0 2024-03-31T15:11:57.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.451+0000 7f4a4b400640 1 --2- 172.21.15.16:0/4035903528 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x555f5868f9b0 0x555f5868fd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:11:57.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.451+0000 7f4a4b400640 1 --2- 172.21.15.16:0/806068095 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f4a28068380 0x7f4a2806a840 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f4a44007fd0 tx=0x7f4a44007780 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:11:57.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.451+0000 7f4a42000640 1 --2- 172.21.15.16:0/4035903528 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555f586dcbf0 0x555f5868f470 secure :-1 s=READY pgs=1297 cs=0 l=1 rev1=1 crypto rx=0x7f4a3405e1f0 tx=0x7f4a3405e720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2bfc4624ced1a477 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:11:57.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.451+0000 7f4a3be00640 1 -- 172.21.15.16:0/4035903528 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a34053ef0 con 0x555f586dcbf0 2024-03-31T15:11:57.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.451+0000 7f4a3be00640 1 -- 172.21.15.16:0/4035903528 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4a34052260 con 0x555f586dcbf0 2024-03-31T15:11:57.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.451+0000 7f4a3be00640 1 -- 172.21.15.16:0/4035903528 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a34004d90 con 0x555f586dcbf0 2024-03-31T15:11:57.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.451+0000 7f4a55318000 1 -- 172.21.15.16:0/4035903528 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555f586dcbf0 msgr2=0x555f5868f470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:11:57.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.451+0000 7f4a55318000 1 --2- 172.21.15.16:0/4035903528 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555f586dcbf0 0x555f5868f470 secure :-1 s=READY pgs=1297 cs=0 l=1 rev1=1 crypto rx=0x7f4a3405e1f0 tx=0x7f4a3405e720 comp rx=0 tx=0).stop 2024-03-31T15:11:57.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.451+0000 7f4a55318000 1 -- 172.21.15.16:0/4035903528 shutdown_connections 2024-03-31T15:11:57.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.451+0000 7f4a55318000 1 --2- 172.21.15.16:0/4035903528 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x555f5868f9b0 0x555f5868fd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:57.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.451+0000 7f4a55318000 1 --2- 172.21.15.16:0/4035903528 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555f586dcbf0 0x555f5868f470 unknown :-1 s=CLOSED pgs=1297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:57.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.451+0000 7f4a55318000 1 --2- 172.21.15.16:0/4035903528 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555f586dc2d0 0x555f586dc6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:57.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.451+0000 7f4a55318000 1 -- 172.21.15.16:0/4035903528 >> 172.21.15.16:0/4035903528 conn(0x555f586bd070 msgr2=0x555f586bf460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:11:57.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.451+0000 7f4a55318000 1 -- 172.21.15.16:0/4035903528 shutdown_connections 2024-03-31T15:11:57.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.451+0000 7f4a55318000 1 -- 172.21.15.16:0/4035903528 wait complete. 2024-03-31T15:11:57.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.455+0000 7f4a55318000 1 -- start start 2024-03-31T15:11:57.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.455+0000 7f4a55318000 1 Processor -- start 2024-03-31T15:11:57.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.455+0000 7f4a55318000 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555f5868f9b0 0x555f58694990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:57.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.455+0000 7f4a55318000 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x555f586dc2d0 0x555f58694ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:57.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.455+0000 7f4a55318000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555f586dcbf0 0x555f58695410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:57.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.455+0000 7f4a55318000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x555f5849f1e0 con 0x555f586dcbf0 2024-03-31T15:11:57.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.455+0000 7f4a55318000 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x555f584ddab0 con 0x555f5868f9b0 2024-03-31T15:11:57.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.455+0000 7f4a55318000 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x555f585b0420 con 0x555f586dc2d0 2024-03-31T15:11:57.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.455+0000 7f4a55318000 10 client.0 _ll_register_callbacks cb 0x555f5867d370 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T15:11:57.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.455+0000 7f4a4b400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555f586dcbf0 0x555f58695410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:57.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.455+0000 7f4a4aa00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555f5868f9b0 0x555f58694990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:57.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.455+0000 7f4a4aa00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555f5868f9b0 0x555f58694990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.16:50844/0 (socket says 172.21.15.16:50844) 2024-03-31T15:11:57.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.455+0000 7f4a42000640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x555f586dc2d0 0x555f58694ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:57.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.455+0000 7f4a4b400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555f586dcbf0 0x555f58695410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48690/0 (socket says 172.21.15.16:48690) 2024-03-31T15:11:57.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.455+0000 7f4a4aa00640 1 -- 172.21.15.16:0/1181785054 learned_addr learned my addr 172.21.15.16:0/1181785054 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:11:57.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.455+0000 7f4a4b400640 1 -- 172.21.15.16:0/1181785054 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x555f586dc2d0 msgr2=0x555f58694ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:11:57.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.455+0000 7f4a4b400640 1 --2- 172.21.15.16:0/1181785054 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x555f586dc2d0 0x555f58694ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:57.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.455+0000 7f4a4b400640 1 -- 172.21.15.16:0/1181785054 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555f5868f9b0 msgr2=0x555f58694990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:11:57.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.455+0000 7f4a4b400640 1 --2- 172.21.15.16:0/1181785054 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555f5868f9b0 0x555f58694990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:57.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.455+0000 7f4a4b400640 1 -- 172.21.15.16:0/1181785054 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555f5849db70 con 0x555f586dcbf0 2024-03-31T15:11:57.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.455+0000 7f4a4aa00640 1 --2- 172.21.15.16:0/1181785054 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555f5868f9b0 0x555f58694990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:11:57.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.455+0000 7f4a4b400640 1 --2- 172.21.15.16:0/1181785054 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555f586dcbf0 0x555f58695410 secure :-1 s=READY pgs=1298 cs=0 l=1 rev1=1 crypto rx=0x7f4a4400dda0 tx=0x7f4a44004a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b9f57c0c5f730d0d server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:11:57.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.455+0000 7f4a38c00640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-31T15:11:57.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.459+0000 7f4a38c00640 1 -- 172.21.15.16:0/1181785054 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a44023410 con 0x555f586dcbf0 2024-03-31T15:11:57.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.459+0000 7f4a38c00640 1 -- 172.21.15.16:0/1181785054 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 5910+0+0 (secure 0 0 0) 0x7f4a44013040 con 0x555f586dcbf0 2024-03-31T15:11:57.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.459+0000 7f4a55318000 1 -- 172.21.15.16:0/1181785054 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4a34052260 con 0x555f586dcbf0 2024-03-31T15:11:57.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.459+0000 7f4a38c00640 1 -- 172.21.15.16:0/1181785054 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=74) ==== 24+0+0 (secure 0 0 0) 0x7f4a440235b0 con 0x555f586dcbf0 2024-03-31T15:11:57.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.459+0000 7f4a55318000 10 client.8238 fetch_fsmap learned FSMap version 74 2024-03-31T15:11:57.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.459+0000 7f4a55318000 1 -- 172.21.15.16:0/1181785054 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=74,monmap=2+}) -- 0x555f5847eb90 con 0x555f586dcbf0 2024-03-31T15:11:57.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.459+0000 7f4a38c00640 1 -- 172.21.15.16:0/1181785054 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4a34052260 con 0x555f586dcbf0 2024-03-31T15:11:57.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.459+0000 7f4a38c00640 1 -- 172.21.15.16:0/1181785054 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 74) ==== 3494+0+0 (secure 0 0 0) 0x7f4a44032020 con 0x555f586dcbf0 2024-03-31T15:11:57.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.459+0000 7f4a55318000 10 client.8238 fetch_fsmap finished waiting for FSMap version 74 2024-03-31T15:11:57.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.459+0000 7f4a55318000 10 client.8238 resolve_mds: resolved 6:0 to role '6:0' aka daemon mds.c 2024-03-31T15:11:57.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.459+0000 7f4a55318000 20 client.8238 populate_metadata read hostname 'smithi016' 2024-03-31T15:11:57.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.459+0000 7f4a55318000 1 --2- 172.21.15.16:0/1181785054 >> [v2:172.21.15.16:6832/3780866829,v1:172.21.15.16:6834/3780866829] conn(0x555f58695ce0 0x555f586eda50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:57.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.459+0000 7f4a55318000 4 client.8238 mds_command: new command op to 8085 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-31T15:11:57.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.459+0000 7f4a55318000 1 -- 172.21.15.16:0/1181785054 --> [v2:172.21.15.16:6832/3780866829,v1:172.21.15.16:6834/3780866829] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555f5847cdd0 con 0x555f58695ce0 2024-03-31T15:11:57.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.459+0000 7f4a4aa00640 1 --2- 172.21.15.16:0/1181785054 >> [v2:172.21.15.16:6832/3780866829,v1:172.21.15.16:6834/3780866829] conn(0x555f58695ce0 0x555f586eda50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:57.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.459+0000 7f4a4aa00640 1 --2- 172.21.15.16:0/1181785054 >> [v2:172.21.15.16:6832/3780866829,v1:172.21.15.16:6834/3780866829] conn(0x555f58695ce0 0x555f586eda50 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=40900a3c5ee0f22a server_cookie=b4d46408f4984c47 in_seq=0 out_seq=0 2024-03-31T15:11:57.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.459+0000 7f4a38c00640 10 client.8238 ms_handle_connect on v2:172.21.15.16:6832/3780866829 2024-03-31T15:11:57.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.459+0000 7f4a38c00640 1 -- 172.21.15.16:0/1181785054 <== mds.0 v2:172.21.15.16:6832/3780866829 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x555f5847cdd0 con 0x555f58695ce0 2024-03-31T15:11:57.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.459+0000 7f4a38c00640 10 client.8238 handle_command_reply: tid=0 2024-03-31T15:11:57.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.459+0000 7f4a55318000 1 client.8238 shutdown 2024-03-31T15:11:57.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.459+0000 7f4a55318000 1 -- 172.21.15.16:0/1181785054 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555f586dcbf0 msgr2=0x555f58695410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:11:57.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.459+0000 7f4a55318000 1 --2- 172.21.15.16:0/1181785054 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555f586dcbf0 0x555f58695410 secure :-1 s=READY pgs=1298 cs=0 l=1 rev1=1 crypto rx=0x7f4a4400dda0 tx=0x7f4a44004a50 comp rx=0 tx=0).stop 2024-03-31T15:11:57.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.459+0000 7f4a55318000 1 -- 172.21.15.16:0/1181785054 shutdown_connections 2024-03-31T15:11:57.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.459+0000 7f4a55318000 1 --2- 172.21.15.16:0/1181785054 >> [v2:172.21.15.16:6832/3780866829,v1:172.21.15.16:6834/3780866829] conn(0x555f58695ce0 0x555f586eda50 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:57.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.459+0000 7f4a55318000 1 --2- 172.21.15.16:0/1181785054 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555f586dcbf0 0x555f58695410 unknown :-1 s=CLOSED pgs=1298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:57.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.459+0000 7f4a55318000 1 --2- 172.21.15.16:0/1181785054 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x555f586dc2d0 0x555f58694ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:57.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.459+0000 7f4a55318000 1 --2- 172.21.15.16:0/1181785054 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555f5868f9b0 0x555f58694990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:57.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.459+0000 7f4a38c00640 0 client.8238 ms_handle_reset on v2:172.21.15.16:6832/3780866829 2024-03-31T15:11:57.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.459+0000 7f4a55318000 1 -- 172.21.15.16:0/1181785054 >> 172.21.15.16:0/1181785054 conn(0x555f586bd070 msgr2=0x555f586bf460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:11:57.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.459+0000 7f4a55318000 1 -- 172.21.15.16:0/1181785054 shutdown_connections 2024-03-31T15:11:57.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.459+0000 7f4a55318000 1 -- 172.21.15.16:0/1181785054 wait complete. 2024-03-31T15:11:57.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.459+0000 7f4a55318000 20 client.8238 trim_cache size 0 max 16384 2024-03-31T15:11:57.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.479+0000 7f4a55318000 1 Processor -- start 2024-03-31T15:11:57.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.479+0000 7f4a55318000 1 -- start start 2024-03-31T15:11:57.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.479+0000 7f4a55318000 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555f5868f9b0 0x555f586db9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:57.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.479+0000 7f4a55318000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555f586dc2d0 0x555f586da380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:57.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.479+0000 7f4a55318000 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x555f586dcbf0 0x555f586da8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:57.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.479+0000 7f4a55318000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f4a4c201950 con 0x555f586dc2d0 2024-03-31T15:11:57.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.479+0000 7f4a55318000 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f4a4c200970 con 0x555f5868f9b0 2024-03-31T15:11:57.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.479+0000 7f4a55318000 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f4a4c1ffcd0 con 0x555f586dcbf0 2024-03-31T15:11:57.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.479+0000 7f4a42000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555f586dc2d0 0x555f586da380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:57.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.479+0000 7f4a42000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555f586dc2d0 0x555f586da380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48706/0 (socket says 172.21.15.16:48706) 2024-03-31T15:11:57.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.479+0000 7f4a42000640 1 -- 172.21.15.16:0/2108257924 learned_addr learned my addr 172.21.15.16:0/2108257924 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:11:57.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.479+0000 7f4a4aa00640 1 --2- 172.21.15.16:0/2108257924 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555f5868f9b0 0x555f586db9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:57.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.479+0000 7f4a4b400640 1 --2- 172.21.15.16:0/2108257924 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x555f586dcbf0 0x555f586da8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:57.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.479+0000 7f4a42000640 1 -- 172.21.15.16:0/2108257924 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x555f586dcbf0 msgr2=0x555f586da8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:11:57.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.479+0000 7f4a42000640 1 --2- 172.21.15.16:0/2108257924 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x555f586dcbf0 0x555f586da8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:57.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.479+0000 7f4a42000640 1 -- 172.21.15.16:0/2108257924 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555f5868f9b0 msgr2=0x555f586db9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:11:57.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.479+0000 7f4a42000640 1 --2- 172.21.15.16:0/2108257924 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555f5868f9b0 0x555f586db9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:57.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.479+0000 7f4a42000640 1 -- 172.21.15.16:0/2108257924 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555f58562140 con 0x555f586dc2d0 2024-03-31T15:11:57.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.479+0000 7f4a4aa00640 1 --2- 172.21.15.16:0/2108257924 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555f5868f9b0 0x555f586db9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:11:57.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.479+0000 7f4a4b400640 1 --2- 172.21.15.16:0/2108257924 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x555f586dcbf0 0x555f586da8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:11:57.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.479+0000 7f4a42000640 1 --2- 172.21.15.16:0/2108257924 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555f586dc2d0 0x555f586da380 secure :-1 s=READY pgs=1299 cs=0 l=1 rev1=1 crypto rx=0x7f4a34000f70 tx=0x7f4a34055860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=785498541268c7f4 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:11:57.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.479+0000 7f4a3b400640 1 -- 172.21.15.16:0/2108257924 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a34053ef0 con 0x555f586dc2d0 2024-03-31T15:11:57.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.479+0000 7f4a3b400640 1 -- 172.21.15.16:0/2108257924 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4a34052260 con 0x555f586dc2d0 2024-03-31T15:11:57.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.479+0000 7f4a3b400640 1 -- 172.21.15.16:0/2108257924 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a34004d90 con 0x555f586dc2d0 2024-03-31T15:11:57.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.479+0000 7f4a55318000 1 -- 172.21.15.16:0/2108257924 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555f586dc2d0 msgr2=0x555f586da380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:11:57.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.479+0000 7f4a55318000 1 --2- 172.21.15.16:0/2108257924 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555f586dc2d0 0x555f586da380 secure :-1 s=READY pgs=1299 cs=0 l=1 rev1=1 crypto rx=0x7f4a34000f70 tx=0x7f4a34055860 comp rx=0 tx=0).stop 2024-03-31T15:11:57.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.479+0000 7f4a55318000 1 -- 172.21.15.16:0/2108257924 shutdown_connections 2024-03-31T15:11:57.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.479+0000 7f4a55318000 1 --2- 172.21.15.16:0/2108257924 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x555f586dcbf0 0x555f586da8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:57.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.479+0000 7f4a55318000 1 --2- 172.21.15.16:0/2108257924 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555f586dc2d0 0x555f586da380 unknown :-1 s=CLOSED pgs=1299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:57.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.479+0000 7f4a55318000 1 --2- 172.21.15.16:0/2108257924 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555f5868f9b0 0x555f586db9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:57.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.479+0000 7f4a55318000 1 -- 172.21.15.16:0/2108257924 >> 172.21.15.16:0/2108257924 conn(0x555f586bd070 msgr2=0x555f586939f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:11:57.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.479+0000 7f4a55318000 1 -- 172.21.15.16:0/2108257924 shutdown_connections 2024-03-31T15:11:57.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.479+0000 7f4a55318000 1 -- 172.21.15.16:0/2108257924 wait complete. 2024-03-31T15:11:57.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.483+0000 7f4a55318000 1 -- start start 2024-03-31T15:11:57.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.483+0000 7f4a55318000 1 Processor -- start 2024-03-31T15:11:57.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.483+0000 7f4a55318000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555f5868f9b0 0x555f58696bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:57.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.483+0000 7f4a55318000 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555f586dc2d0 0x555f586eeaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:57.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.483+0000 7f4a55318000 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x555f586dcbf0 0x555f586eefe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:57.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.483+0000 7f4a55318000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x555f5847c940 con 0x555f5868f9b0 2024-03-31T15:11:57.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.483+0000 7f4a55318000 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f4a4c200320 con 0x555f586dc2d0 2024-03-31T15:11:57.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.483+0000 7f4a55318000 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x555f58498750 con 0x555f586dcbf0 2024-03-31T15:11:57.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.483+0000 7f4a55318000 10 client.0 _ll_register_callbacks cb 0x7f4a4400c9d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T15:11:57.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.483+0000 7f4a4aa00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555f5868f9b0 0x555f58696bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:57.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.483+0000 7f4a4aa00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555f5868f9b0 0x555f58696bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48722/0 (socket says 172.21.15.16:48722) 2024-03-31T15:11:57.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.483+0000 7f4a4aa00640 1 -- 172.21.15.16:0/3007808436 learned_addr learned my addr 172.21.15.16:0/3007808436 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:11:57.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.483+0000 7f4a42000640 1 --2- 172.21.15.16:0/3007808436 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555f586dc2d0 0x555f586eeaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:57.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.483+0000 7f4a4b400640 1 --2- 172.21.15.16:0/3007808436 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x555f586dcbf0 0x555f586eefe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:57.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.483+0000 7f4a4aa00640 1 -- 172.21.15.16:0/3007808436 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x555f586dcbf0 msgr2=0x555f586eefe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:11:57.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.483+0000 7f4a4aa00640 1 --2- 172.21.15.16:0/3007808436 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x555f586dcbf0 0x555f586eefe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:57.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.483+0000 7f4a4aa00640 1 -- 172.21.15.16:0/3007808436 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555f586dc2d0 msgr2=0x555f586eeaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:11:57.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.483+0000 7f4a4aa00640 1 --2- 172.21.15.16:0/3007808436 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555f586dc2d0 0x555f586eeaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:57.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.483+0000 7f4a4aa00640 1 -- 172.21.15.16:0/3007808436 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555f584701e0 con 0x555f5868f9b0 2024-03-31T15:11:57.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.483+0000 7f4a4aa00640 1 --2- 172.21.15.16:0/3007808436 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555f5868f9b0 0x555f58696bc0 secure :-1 s=READY pgs=1300 cs=0 l=1 rev1=1 crypto rx=0x7f4a3c01c970 tx=0x7f4a3c01acd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6cd7c3c4df8e1dd6 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:11:57.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.483+0000 7f4a38c00640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-31T15:11:57.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.483+0000 7f4a38c00640 1 -- 172.21.15.16:0/3007808436 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a3c0b0040 con 0x555f5868f9b0 2024-03-31T15:11:57.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.483+0000 7f4a38c00640 1 -- 172.21.15.16:0/3007808436 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 5910+0+0 (secure 0 0 0) 0x7f4a3c004470 con 0x555f5868f9b0 2024-03-31T15:11:57.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.487+0000 7f4a55318000 1 -- 172.21.15.16:0/3007808436 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4a34052260 con 0x555f5868f9b0 2024-03-31T15:11:57.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.487+0000 7f4a38c00640 1 -- 172.21.15.16:0/3007808436 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=74) ==== 24+0+0 (secure 0 0 0) 0x7f4a3c0ab970 con 0x555f5868f9b0 2024-03-31T15:11:57.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.487+0000 7f4a55318000 10 client.8244 fetch_fsmap learned FSMap version 74 2024-03-31T15:11:57.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.487+0000 7f4a55318000 1 -- 172.21.15.16:0/3007808436 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=74,monmap=2+}) -- 0x555f584757b0 con 0x555f5868f9b0 2024-03-31T15:11:57.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.487+0000 7f4a38c00640 1 -- 172.21.15.16:0/3007808436 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4a34052260 con 0x555f5868f9b0 2024-03-31T15:11:57.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.487+0000 7f4a38c00640 1 -- 172.21.15.16:0/3007808436 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 74) ==== 3494+0+0 (secure 0 0 0) 0x7f4a3c0c9020 con 0x555f5868f9b0 2024-03-31T15:11:57.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.487+0000 7f4a55318000 10 client.8244 fetch_fsmap finished waiting for FSMap version 74 2024-03-31T15:11:57.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.487+0000 7f4a55318000 10 client.8244 resolve_mds: resolved 6:0 to role '6:0' aka daemon mds.c 2024-03-31T15:11:57.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.487+0000 7f4a55318000 20 client.8244 populate_metadata read hostname 'smithi016' 2024-03-31T15:11:57.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.487+0000 7f4a55318000 1 --2- 172.21.15.16:0/3007808436 >> [v2:172.21.15.16:6832/3780866829,v1:172.21.15.16:6834/3780866829] conn(0x555f586ef520 0x555f5874ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:57.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.487+0000 7f4a55318000 4 client.8244 mds_command: new command op to 8085 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-31T15:11:57.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.487+0000 7f4a55318000 1 -- 172.21.15.16:0/3007808436 --> [v2:172.21.15.16:6832/3780866829,v1:172.21.15.16:6834/3780866829] -- command(tid 0: {"prefix": "scrub status"}) -- 0x555f584ef040 con 0x555f586ef520 2024-03-31T15:11:57.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.487+0000 7f4a42000640 1 --2- 172.21.15.16:0/3007808436 >> [v2:172.21.15.16:6832/3780866829,v1:172.21.15.16:6834/3780866829] conn(0x555f586ef520 0x555f5874ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:57.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.487+0000 7f4a42000640 1 --2- 172.21.15.16:0/3007808436 >> [v2:172.21.15.16:6832/3780866829,v1:172.21.15.16:6834/3780866829] conn(0x555f586ef520 0x555f5874ba10 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=13602137cacd708 server_cookie=a92f51e19e96ca6a in_seq=0 out_seq=0 2024-03-31T15:11:57.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.487+0000 7f4a38c00640 10 client.8244 ms_handle_connect on v2:172.21.15.16:6832/3780866829 2024-03-31T15:11:57.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.487+0000 7f4a38c00640 1 -- 172.21.15.16:0/3007808436 <== mds.0 v2:172.21.15.16:6832/3780866829 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x555f584ef040 con 0x555f586ef520 2024-03-31T15:11:57.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.487+0000 7f4a38c00640 10 client.8244 handle_command_reply: tid=0 2024-03-31T15:11:57.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.487+0000 7f4a55318000 1 client.8244 shutdown 2024-03-31T15:11:57.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.487+0000 7f4a55318000 1 -- 172.21.15.16:0/3007808436 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555f5868f9b0 msgr2=0x555f58696bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:11:57.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.487+0000 7f4a55318000 1 --2- 172.21.15.16:0/3007808436 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555f5868f9b0 0x555f58696bc0 secure :-1 s=READY pgs=1300 cs=0 l=1 rev1=1 crypto rx=0x7f4a3c01c970 tx=0x7f4a3c01acd0 comp rx=0 tx=0).stop 2024-03-31T15:11:57.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.487+0000 7f4a55318000 1 -- 172.21.15.16:0/3007808436 shutdown_connections 2024-03-31T15:11:57.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.487+0000 7f4a55318000 1 --2- 172.21.15.16:0/3007808436 >> [v2:172.21.15.16:6832/3780866829,v1:172.21.15.16:6834/3780866829] conn(0x555f586ef520 0x555f5874ba10 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:57.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.487+0000 7f4a55318000 1 --2- 172.21.15.16:0/3007808436 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x555f586dcbf0 0x555f586eefe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:57.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.487+0000 7f4a55318000 1 --2- 172.21.15.16:0/3007808436 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555f586dc2d0 0x555f586eeaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:57.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.487+0000 7f4a38c00640 0 client.8244 ms_handle_reset on v2:172.21.15.16:6832/3780866829 2024-03-31T15:11:57.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.487+0000 7f4a55318000 1 --2- 172.21.15.16:0/3007808436 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555f5868f9b0 0x555f58696bc0 unknown :-1 s=CLOSED pgs=1300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:57.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.487+0000 7f4a55318000 1 -- 172.21.15.16:0/3007808436 >> 172.21.15.16:0/3007808436 conn(0x555f586bd070 msgr2=0x555f586960c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:11:57.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.487+0000 7f4a55318000 1 -- 172.21.15.16:0/3007808436 shutdown_connections 2024-03-31T15:11:57.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.487+0000 7f4a55318000 1 -- 172.21.15.16:0/3007808436 wait complete. 2024-03-31T15:11:57.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.487+0000 7f4a55318000 20 client.8244 trim_cache size 0 max 16384 2024-03-31T15:11:57.483 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-03-31T15:11:57.483 INFO:teuthology.orchestra.run.smithi016.stdout: "status": "no active scrubs running", 2024-03-31T15:11:57.483 INFO:teuthology.orchestra.run.smithi016.stdout: "scrubs": {} 2024-03-31T15:11:57.483 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-03-31T15:11:57.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.487+0000 7f4a51e00640 1 -- 172.21.15.16:0/806068095 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f4a28068380 msgr2=0x7f4a2806a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:11:57.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.487+0000 7f4a51e00640 1 --2- 172.21.15.16:0/806068095 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f4a28068380 0x7f4a2806a840 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f4a44007fd0 tx=0x7f4a44007780 comp rx=0 tx=0).stop 2024-03-31T15:11:57.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.487+0000 7f4a51e00640 1 -- 172.21.15.16:0/806068095 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4a4c0cf150 msgr2=0x7f4a4c0b5300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:11:57.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.487+0000 7f4a51e00640 1 --2- 172.21.15.16:0/806068095 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4a4c0cf150 0x7f4a4c0b5300 secure :-1 s=READY pgs=1296 cs=0 l=1 rev1=1 crypto rx=0x7f4a3400c450 tx=0x7f4a3400c910 comp rx=0 tx=0).stop 2024-03-31T15:11:57.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.487+0000 7f4a51e00640 1 -- 172.21.15.16:0/806068095 shutdown_connections 2024-03-31T15:11:57.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.487+0000 7f4a51e00640 1 --2- 172.21.15.16:0/806068095 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f4a28068380 0x7f4a2806a840 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:57.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.487+0000 7f4a51e00640 1 --2- 172.21.15.16:0/806068095 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4a4c0d0a60 0x7f4a4c0b5840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:57.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.487+0000 7f4a51e00640 1 --2- 172.21.15.16:0/806068095 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4a4c0cf150 0x7f4a4c0b5300 unknown :-1 s=CLOSED pgs=1296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:57.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.487+0000 7f4a51e00640 1 --2- 172.21.15.16:0/806068095 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f4a4c06f920 0x7f4a4c0b6500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:57.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.487+0000 7f4a51e00640 1 -- 172.21.15.16:0/806068095 >> 172.21.15.16:0/806068095 conn(0x7f4a4c067930 msgr2=0x7f4a4c06fd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:11:57.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.487+0000 7f4a51e00640 1 -- 172.21.15.16:0/806068095 shutdown_connections 2024-03-31T15:11:57.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.487+0000 7f4a51e00640 1 -- 172.21.15.16:0/806068095 wait complete. 2024-03-31T15:11:57.500 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-31T15:11:57.501 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-31T15:11:57.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.647+0000 7fb4c3a00640 1 Processor -- start 2024-03-31T15:11:57.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.647+0000 7fb4c3a00640 1 -- start start 2024-03-31T15:11:57.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.651+0000 7fb4c3a00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb4bc0ce050 0x7fb4bc15f380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:57.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.651+0000 7fb4c3a00640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb4bc15b4b0 con 0x7fb4bc15b0d0 2024-03-31T15:11:57.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.651+0000 7fb4c3a00640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb4bc06cc30 con 0x7fb4bc06c730 2024-03-31T15:11:57.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.651+0000 7fb4c3a00640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7fb4bc0b4140 con 0x7fb4bc0ce050 2024-03-31T15:11:57.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.651+0000 7fb4b3e00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb4bc0ce050 0x7fb4bc15f380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:57.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.651+0000 7fb4c1000640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fb4bc15b0d0 0x7fb4bc15ec70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:35048/0 (socket says 172.21.15.16:35048) 2024-03-31T15:11:57.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.651+0000 7fb4c1000640 1 -- 172.21.15.16:0/3568133902 learned_addr learned my addr 172.21.15.16:0/3568133902 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:11:57.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.651+0000 7fb4b3e00640 1 -- 172.21.15.16:0/3568133902 >> v1:172.21.15.89:6789/0 conn(0x7fb4bc06c730 legacy=0x7fb4bc0b45d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:11:57.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.651+0000 7fb4b3e00640 1 -- 172.21.15.16:0/3568133902 >> v1:172.21.15.16:6789/0 conn(0x7fb4bc15b0d0 legacy=0x7fb4bc15ec70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:11:57.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.651+0000 7fb4b3e00640 1 -- 172.21.15.16:0/3568133902 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb4bc0b3810 con 0x7fb4bc0ce050 2024-03-31T15:11:57.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.651+0000 7fb4bbe00640 1 -- 172.21.15.16:0/3568133902 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1893003393 0 0) 0x7fb4bc15b4b0 con 0x7fb4bc15b0d0 2024-03-31T15:11:57.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.655+0000 7fb4b3e00640 1 --2- 172.21.15.16:0/3568133902 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb4bc0ce050 0x7fb4bc15f380 secure :-1 s=READY pgs=1302 cs=0 l=1 rev1=1 crypto rx=0x7fb4a40097c0 tx=0x7fb4a401c470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32a11945928095d2 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:11:57.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.655+0000 7fb4bbe00640 1 -- 172.21.15.16:0/3568133902 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4a40a7070 con 0x7fb4bc0ce050 2024-03-31T15:11:57.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.655+0000 7fb4bbe00640 1 -- 172.21.15.16:0/3568133902 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb4a40b0070 con 0x7fb4bc0ce050 2024-03-31T15:11:57.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.655+0000 7fb4bbe00640 1 -- 172.21.15.16:0/3568133902 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4a40ab890 con 0x7fb4bc0ce050 2024-03-31T15:11:57.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.655+0000 7fb4c3a00640 1 -- 172.21.15.16:0/3568133902 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb4bc0ce050 msgr2=0x7fb4bc15f380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:11:57.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.655+0000 7fb4c3a00640 1 --2- 172.21.15.16:0/3568133902 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb4bc0ce050 0x7fb4bc15f380 secure :-1 s=READY pgs=1302 cs=0 l=1 rev1=1 crypto rx=0x7fb4a40097c0 tx=0x7fb4a401c470 comp rx=0 tx=0).stop 2024-03-31T15:11:57.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.655+0000 7fb4c3a00640 1 -- 172.21.15.16:0/3568133902 shutdown_connections 2024-03-31T15:11:57.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.655+0000 7fb4c3a00640 1 --2- 172.21.15.16:0/3568133902 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb4bc0ce050 0x7fb4bc15f380 unknown :-1 s=CLOSED pgs=1302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:57.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.655+0000 7fb4c3a00640 1 -- 172.21.15.16:0/3568133902 >> 172.21.15.16:0/3568133902 conn(0x7fb4bc0d0760 msgr2=0x7fb4bc0d0b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:11:57.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.655+0000 7fb4c3a00640 1 -- 172.21.15.16:0/3568133902 shutdown_connections 2024-03-31T15:11:57.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.655+0000 7fb4c3a00640 1 -- 172.21.15.16:0/3568133902 wait complete. 2024-03-31T15:11:57.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.655+0000 7fb4c3a00640 1 Processor -- start 2024-03-31T15:11:57.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.655+0000 7fb4c3a00640 1 -- start start 2024-03-31T15:11:57.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.655+0000 7fb4c3a00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb4bc06c730 0x7fb4bc14c960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:57.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.655+0000 7fb4c3a00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb4bc0ce050 0x7fb4bc150cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:57.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.655+0000 7fb4c3a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb4bc15b0d0 0x7fb4bc1577c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:57.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.655+0000 7fb4c3a00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fb4bc15fa40 con 0x7fb4bc15b0d0 2024-03-31T15:11:57.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.655+0000 7fb4c3a00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fb4bc15f8c0 con 0x7fb4bc06c730 2024-03-31T15:11:57.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.655+0000 7fb4c3a00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fb4bc06b620 con 0x7fb4bc0ce050 2024-03-31T15:11:57.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.659+0000 7fb4c1a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb4bc15b0d0 0x7fb4bc1577c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:57.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.659+0000 7fb4c1000640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb4bc06c730 0x7fb4bc14c960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:57.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.659+0000 7fb4b3e00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb4bc0ce050 0x7fb4bc150cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:57.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.659+0000 7fb4c1a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb4bc15b0d0 0x7fb4bc1577c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48750/0 (socket says 172.21.15.16:48750) 2024-03-31T15:11:57.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.659+0000 7fb4c1000640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb4bc06c730 0x7fb4bc14c960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.16:50878/0 (socket says 172.21.15.16:50878) 2024-03-31T15:11:57.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.659+0000 7fb4c1a00640 1 -- 172.21.15.16:0/1215574454 learned_addr learned my addr 172.21.15.16:0/1215574454 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:11:57.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.659+0000 7fb4b3e00640 1 -- 172.21.15.16:0/1215574454 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb4bc06c730 msgr2=0x7fb4bc14c960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:11:57.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.659+0000 7fb4b3e00640 1 --2- 172.21.15.16:0/1215574454 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb4bc06c730 0x7fb4bc14c960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:57.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.659+0000 7fb4b3e00640 1 -- 172.21.15.16:0/1215574454 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb4bc15b0d0 msgr2=0x7fb4bc1577c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:11:57.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.659+0000 7fb4b3e00640 1 --2- 172.21.15.16:0/1215574454 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb4bc15b0d0 0x7fb4bc1577c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:57.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.659+0000 7fb4b3e00640 1 -- 172.21.15.16:0/1215574454 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb4bc1515c0 con 0x7fb4bc0ce050 2024-03-31T15:11:57.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.659+0000 7fb4c1a00640 1 --2- 172.21.15.16:0/1215574454 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb4bc15b0d0 0x7fb4bc1577c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:11:57.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.659+0000 7fb4b3e00640 1 --2- 172.21.15.16:0/1215574454 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb4bc0ce050 0x7fb4bc150cc0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fb4a40ac000 tx=0x7fb4a401acb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:11:57.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.659+0000 7fb4b9600640 1 -- 172.21.15.16:0/1215574454 <== mon.2 v2:172.21.15.89:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4a40bf610 con 0x7fb4bc0ce050 2024-03-31T15:11:57.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.659+0000 7fb4b9600640 1 -- 172.21.15.16:0/1215574454 <== mon.2 v2:172.21.15.89:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb4a40bf7f0 con 0x7fb4bc0ce050 2024-03-31T15:11:57.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.659+0000 7fb4c3a00640 1 -- 172.21.15.16:0/1215574454 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb4bc1518a0 con 0x7fb4bc0ce050 2024-03-31T15:11:57.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.659+0000 7fb4b9600640 1 -- 172.21.15.16:0/1215574454 <== mon.2 v2:172.21.15.89:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4a40bfbe0 con 0x7fb4bc0ce050 2024-03-31T15:11:57.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.659+0000 7fb4c3a00640 1 -- 172.21.15.16:0/1215574454 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fb4bc151e60 con 0x7fb4bc0ce050 2024-03-31T15:11:57.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.659+0000 7fb4c3a00640 1 -- 172.21.15.16:0/1215574454 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb4bc069950 con 0x7fb4bc0ce050 2024-03-31T15:11:57.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.659+0000 7fb4b9600640 1 -- 172.21.15.16:0/1215574454 <== mon.2 v2:172.21.15.89:3301/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fb4a40bd020 con 0x7fb4bc0ce050 2024-03-31T15:11:57.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.663+0000 7fb4b9600640 1 --2- 172.21.15.16:0/1215574454 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fb498068380 0x7fb49806a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:57.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.663+0000 7fb4b9600640 1 -- 172.21.15.16:0/1215574454 <== mon.2 v2:172.21.15.89:3301/0 5 ==== osd_map(85..85 src has 1..85) ==== 5910+0+0 (secure 0 0 0) 0x7fb4a4121590 con 0x7fb4bc0ce050 2024-03-31T15:11:57.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.663+0000 7fb4c1000640 1 --2- 172.21.15.16:0/1215574454 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fb498068380 0x7fb49806a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:57.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.663+0000 7fb4c1000640 1 --2- 172.21.15.16:0/1215574454 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fb498068380 0x7fb49806a840 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7fb4ac000f70 tx=0x7fb4ac007240 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:11:57.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.667+0000 7fb4b9600640 1 -- 172.21.15.16:0/1215574454 <== mon.2 v2:172.21.15.89:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fb4a4016090 con 0x7fb4bc0ce050 2024-03-31T15:11:57.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.887+0000 7fb4c3a00640 1 -- 172.21.15.16:0/1215574454 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb4bc06fb10 con 0x7fb4bc0ce050 2024-03-31T15:11:57.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.887+0000 7fb4b9600640 1 -- 172.21.15.16:0/1215574454 <== mon.2 v2:172.21.15.89:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 74 v74) ==== 94+0+5184 (secure 0 0 0) 0x7fb4a40f14a0 con 0x7fb4bc0ce050 2024-03-31T15:11:57.879 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:11:57.879 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":74,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6494,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/2445558494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6835","nonce":2445558494},{"type":"v1","addr":"172.21.15.89:6837","nonce":2445558494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":56},{"gid":7270,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/3185913858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6833","nonce":3185913858},{"type":"v1","addr":"172.21.15.16:6835","nonce":3185913858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":56},{"gid":7279,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6836/1599468493","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":1599468493},{"type":"v1","addr":"172.21.15.89:6836","nonce":1599468493}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":56}],"filesystems":[{"mdsmap":{"epoch":74,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-31T15:09:50.926196+0000","modified":"2024-03-31T15:11:23.417783+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":85,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8085},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8085":{"gid":8085,"name":"c","rank":0,"incarnation":71,"state":"up:active","state_seq":4,"addr":"172.21.15.16:6834/3780866829","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":3780866829},{"type":"v1","addr":"172.21.15.16:6834","nonce":3780866829}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8085,"qdb_cluster":[8085]},"id":6}]} 2024-03-31T15:11:57.879 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 74 2024-03-31T15:11:57.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.887+0000 7fb4c3a00640 1 -- 172.21.15.16:0/1215574454 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fb498068380 msgr2=0x7fb49806a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:11:57.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.887+0000 7fb4c3a00640 1 --2- 172.21.15.16:0/1215574454 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fb498068380 0x7fb49806a840 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7fb4ac000f70 tx=0x7fb4ac007240 comp rx=0 tx=0).stop 2024-03-31T15:11:57.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.887+0000 7fb4c3a00640 1 -- 172.21.15.16:0/1215574454 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb4bc0ce050 msgr2=0x7fb4bc150cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:11:57.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.887+0000 7fb4c3a00640 1 --2- 172.21.15.16:0/1215574454 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb4bc0ce050 0x7fb4bc150cc0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fb4a40ac000 tx=0x7fb4a401acb0 comp rx=0 tx=0).stop 2024-03-31T15:11:57.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.887+0000 7fb4c3a00640 1 -- 172.21.15.16:0/1215574454 shutdown_connections 2024-03-31T15:11:57.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.887+0000 7fb4c3a00640 1 --2- 172.21.15.16:0/1215574454 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fb498068380 0x7fb49806a840 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:57.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.887+0000 7fb4c3a00640 1 --2- 172.21.15.16:0/1215574454 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb4bc15b0d0 0x7fb4bc1577c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:57.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.887+0000 7fb4c3a00640 1 --2- 172.21.15.16:0/1215574454 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb4bc0ce050 0x7fb4bc150cc0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:57.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.887+0000 7fb4c3a00640 1 --2- 172.21.15.16:0/1215574454 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb4bc06c730 0x7fb4bc14c960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:57.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.887+0000 7fb4c3a00640 1 -- 172.21.15.16:0/1215574454 >> 172.21.15.16:0/1215574454 conn(0x7fb4bc0d0760 msgr2=0x7fb4bc0cf2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:11:57.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.887+0000 7fb4c3a00640 1 -- 172.21.15.16:0/1215574454 shutdown_connections 2024-03-31T15:11:57.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:57.887+0000 7fb4c3a00640 1 -- 172.21.15.16:0/1215574454 wait complete. 2024-03-31T15:11:57.901 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c damage ls --format=json-pretty 2024-03-31T15:11:58.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.051+0000 7f4010400640 1 Processor -- start 2024-03-31T15:11:58.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.051+0000 7f4010400640 1 -- start start 2024-03-31T15:11:58.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.051+0000 7f4010400640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f400815e4f0 0x7f40080d4d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:58.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.051+0000 7f4010400640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f400815e8d0 con 0x7f40080d2ed0 2024-03-31T15:11:58.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.051+0000 7f4010400640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f40080d3c60 con 0x7f40080d1270 2024-03-31T15:11:58.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.051+0000 7f4010400640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f40080b81a0 con 0x7f400815e4f0 2024-03-31T15:11:58.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.051+0000 7f400e400640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f40080d2ed0 0x7f40080b7a90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:35062/0 (socket says 172.21.15.16:35062) 2024-03-31T15:11:58.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.051+0000 7f400e400640 1 -- 172.21.15.16:0/30638009 learned_addr learned my addr 172.21.15.16:0/30638009 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:11:58.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.051+0000 7f400da00640 1 --2- 172.21.15.16:0/30638009 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f400815e4f0 0x7f40080d4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:58.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.051+0000 7f4007e00640 1 -- 172.21.15.16:0/30638009 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2021917817 0 0) 0x7f400815e8d0 con 0x7f40080d2ed0 2024-03-31T15:11:58.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.051+0000 7f4007e00640 1 -- 172.21.15.16:0/30638009 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3ff0003530 con 0x7f40080d2ed0 2024-03-31T15:11:58.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.051+0000 7f4007e00640 1 -- 172.21.15.16:0/30638009 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 249073172 0 0) 0x7f3ff0003530 con 0x7f40080d2ed0 2024-03-31T15:11:58.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.051+0000 7f4007e00640 1 -- 172.21.15.16:0/30638009 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f400815e4f0 msgr2=0x7f40080d4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:11:58.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.051+0000 7f4007e00640 1 --2- 172.21.15.16:0/30638009 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f400815e4f0 0x7f40080d4d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:58.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.051+0000 7f4007e00640 1 -- 172.21.15.16:0/30638009 >> v1:172.21.15.89:6789/0 conn(0x7f40080d1270 legacy=0x7f40080b7380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:11:58.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.051+0000 7f4007e00640 1 -- 172.21.15.16:0/30638009 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40080d3f40 con 0x7f40080d2ed0 2024-03-31T15:11:58.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.051+0000 7f4007e00640 1 -- 172.21.15.16:0/30638009 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 214807623 0 0) 0x7f4000001a00 con 0x7f40080d2ed0 2024-03-31T15:11:58.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.051+0000 7f4007e00640 1 -- 172.21.15.16:0/30638009 >> v1:172.21.15.16:6789/0 conn(0x7f40080d2ed0 legacy=0x7f40080b7a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:11:58.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.051+0000 7f4007e00640 1 --2- 172.21.15.16:0/30638009 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3ff00039f0 0x7f3ff0003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:58.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.051+0000 7f4007e00640 1 --2- 172.21.15.16:0/30638009 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f400815e4f0 0x7f3ff0004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:58.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.051+0000 7f4007e00640 1 --2- 172.21.15.16:0/30638009 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3ff0004d50 0x7f3ff00051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:58.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.055+0000 7f4007e00640 1 -- 172.21.15.16:0/30638009 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f3ff000ba80 con 0x7f3ff00039f0 2024-03-31T15:11:58.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.055+0000 7f4007e00640 1 -- 172.21.15.16:0/30638009 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f3ff000bc00 con 0x7f3ff0004d50 2024-03-31T15:11:58.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.055+0000 7f4007e00640 1 -- 172.21.15.16:0/30638009 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f3ff000be00 con 0x7f400815e4f0 2024-03-31T15:11:58.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.055+0000 7f400da00640 1 --2- 172.21.15.16:0/30638009 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3ff00039f0 0x7f3ff0003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:58.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.055+0000 7f400e400640 1 --2- 172.21.15.16:0/30638009 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3ff0004d50 0x7f3ff00051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:58.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.055+0000 7f400d000640 1 --2- 172.21.15.16:0/30638009 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f400815e4f0 0x7f3ff0004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:58.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.055+0000 7f400da00640 1 -- 172.21.15.16:0/30638009 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f400815e4f0 msgr2=0x7f3ff0004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:11:58.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.055+0000 7f400da00640 1 --2- 172.21.15.16:0/30638009 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f400815e4f0 0x7f3ff0004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:58.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.055+0000 7f400da00640 1 -- 172.21.15.16:0/30638009 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3ff0004d50 msgr2=0x7f3ff00051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:11:58.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.055+0000 7f400da00640 1 --2- 172.21.15.16:0/30638009 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3ff0004d50 0x7f3ff00051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:58.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.055+0000 7f400da00640 1 -- 172.21.15.16:0/30638009 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ff0003530 con 0x7f3ff00039f0 2024-03-31T15:11:58.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.055+0000 7f400e400640 1 --2- 172.21.15.16:0/30638009 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3ff0004d50 0x7f3ff00051d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:11:58.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.055+0000 7f400da00640 1 --2- 172.21.15.16:0/30638009 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3ff00039f0 0x7f3ff0003e10 secure :-1 s=READY pgs=1304 cs=0 l=1 rev1=1 crypto rx=0x7f3ffc009ab0 tx=0x7f3ffc01ca70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5a7fc5278afeedf server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:11:58.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.055+0000 7f4007e00640 1 -- 172.21.15.16:0/30638009 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ffc0b2070 con 0x7f3ff00039f0 2024-03-31T15:11:58.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.055+0000 7f4007e00640 1 -- 172.21.15.16:0/30638009 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3ffc0a5530 con 0x7f3ff00039f0 2024-03-31T15:11:58.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.055+0000 7f4007e00640 1 -- 172.21.15.16:0/30638009 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ffc0096b0 con 0x7f3ff00039f0 2024-03-31T15:11:58.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.059+0000 7f4010400640 1 -- 172.21.15.16:0/30638009 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3ff00039f0 msgr2=0x7f3ff0003e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:11:58.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.059+0000 7f4010400640 1 --2- 172.21.15.16:0/30638009 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3ff00039f0 0x7f3ff0003e10 secure :-1 s=READY pgs=1304 cs=0 l=1 rev1=1 crypto rx=0x7f3ffc009ab0 tx=0x7f3ffc01ca70 comp rx=0 tx=0).stop 2024-03-31T15:11:58.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.059+0000 7f400e400640 1 -- 172.21.15.16:0/30638009 reap_dead start 2024-03-31T15:11:58.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.059+0000 7f4010400640 1 -- 172.21.15.16:0/30638009 shutdown_connections 2024-03-31T15:11:58.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.059+0000 7f4010400640 1 -- 172.21.15.16:0/30638009 >> 172.21.15.16:0/30638009 conn(0x7f4008067730 msgr2=0x7f4008067b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:11:58.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.059+0000 7f4010400640 1 -- 172.21.15.16:0/30638009 shutdown_connections 2024-03-31T15:11:58.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.059+0000 7f4010400640 1 -- 172.21.15.16:0/30638009 wait complete. 2024-03-31T15:11:58.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.059+0000 7f4010400640 1 Processor -- start 2024-03-31T15:11:58.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.059+0000 7f4010400640 1 -- start start 2024-03-31T15:11:58.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.059+0000 7f4010400640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3ff00039f0 0x7f4008152530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:58.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.059+0000 7f4010400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f40080cac00 0x7f40080c7e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:58.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.059+0000 7f4010400640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f4008152d70 0x7f40080c8360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:58.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.059+0000 7f4010400640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f40080d50f0 con 0x7f40080cac00 2024-03-31T15:11:58.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.059+0000 7f4010400640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f40080d32e0 con 0x7f3ff00039f0 2024-03-31T15:11:58.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.059+0000 7f4010400640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f40080d5270 con 0x7f4008152d70 2024-03-31T15:11:58.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.059+0000 7f400d000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f40080cac00 0x7f40080c7e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:58.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.059+0000 7f400d000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f40080cac00 0x7f40080c7e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48778/0 (socket says 172.21.15.16:48778) 2024-03-31T15:11:58.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.059+0000 7f400d000640 1 -- 172.21.15.16:0/275366242 learned_addr learned my addr 172.21.15.16:0/275366242 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:11:58.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.059+0000 7f400e400640 1 --2- 172.21.15.16:0/275366242 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f4008152d70 0x7f40080c8360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:58.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.059+0000 7f400da00640 1 --2- 172.21.15.16:0/275366242 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3ff00039f0 0x7f4008152530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:58.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.063+0000 7f400d000640 1 -- 172.21.15.16:0/275366242 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f4008152d70 msgr2=0x7f40080c8360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:11:58.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.063+0000 7f400d000640 1 --2- 172.21.15.16:0/275366242 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f4008152d70 0x7f40080c8360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:58.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.063+0000 7f400d000640 1 -- 172.21.15.16:0/275366242 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3ff00039f0 msgr2=0x7f4008152530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:11:58.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.063+0000 7f400d000640 1 --2- 172.21.15.16:0/275366242 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3ff00039f0 0x7f4008152530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:58.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.063+0000 7f400d000640 1 -- 172.21.15.16:0/275366242 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40081616d0 con 0x7f40080cac00 2024-03-31T15:11:58.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.063+0000 7f400d000640 1 --2- 172.21.15.16:0/275366242 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f40080cac00 0x7f40080c7e20 secure :-1 s=READY pgs=1305 cs=0 l=1 rev1=1 crypto rx=0x7f3ff400e4e0 tx=0x7f3ff400e9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:11:58.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.063+0000 7f4005600640 1 -- 172.21.15.16:0/275366242 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ff40053a0 con 0x7f40080cac00 2024-03-31T15:11:58.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.063+0000 7f4005600640 1 -- 172.21.15.16:0/275366242 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3ff4018070 con 0x7f40080cac00 2024-03-31T15:11:58.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.063+0000 7f4005600640 1 -- 172.21.15.16:0/275366242 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ff4003840 con 0x7f40080cac00 2024-03-31T15:11:58.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.063+0000 7f4010400640 1 -- 172.21.15.16:0/275366242 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f40080ca120 con 0x7f40080cac00 2024-03-31T15:11:58.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.063+0000 7f4010400640 1 -- 172.21.15.16:0/275366242 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f40080ca6e0 con 0x7f40080cac00 2024-03-31T15:11:58.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.063+0000 7f4005600640 1 -- 172.21.15.16:0/275366242 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f3ff40039e0 con 0x7f40080cac00 2024-03-31T15:11:58.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.063+0000 7f4005600640 1 --2- 172.21.15.16:0/275366242 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f3fe8068380 0x7f3fe806a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:58.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.063+0000 7f4005600640 1 -- 172.21.15.16:0/275366242 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 5910+0+0 (secure 0 0 0) 0x7f3ff4086830 con 0x7f40080cac00 2024-03-31T15:11:58.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.067+0000 7f400da00640 1 --2- 172.21.15.16:0/275366242 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f3fe8068380 0x7f3fe806a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:58.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.067+0000 7f401391a000 1 Processor -- start 2024-03-31T15:11:58.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.067+0000 7f401391a000 1 -- start start 2024-03-31T15:11:58.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.067+0000 7f400da00640 1 --2- 172.21.15.16:0/275366242 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f3fe8068380 0x7f3fe806a840 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f3ffc0af000 tx=0x7f3ffc0046d0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:11:58.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.067+0000 7f401391a000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b53c182980 0x55b53c182d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:58.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.067+0000 7f401391a000 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b53c1832a0 0x55b53c135dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:58.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.067+0000 7f401391a000 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55b53c136310 0x55b53c1366f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:58.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.067+0000 7f401391a000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55b53bf86c50 con 0x55b53c182980 2024-03-31T15:11:58.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.067+0000 7f401391a000 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55b53bf238a0 con 0x55b53c1832a0 2024-03-31T15:11:58.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.067+0000 7f401391a000 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x55b53bf82c00 con 0x55b53c136310 2024-03-31T15:11:58.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.067+0000 7f400da00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b53c182980 0x55b53c182d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:58.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.067+0000 7f400da00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b53c182980 0x55b53c182d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48794/0 (socket says 172.21.15.16:48794) 2024-03-31T15:11:58.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.067+0000 7f400da00640 1 -- 172.21.15.16:0/126419072 learned_addr learned my addr 172.21.15.16:0/126419072 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:11:58.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.067+0000 7f400e400640 1 --2- 172.21.15.16:0/126419072 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55b53c136310 0x55b53c1366f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:58.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.067+0000 7f400d000640 1 --2- 172.21.15.16:0/126419072 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b53c1832a0 0x55b53c135dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:58.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.067+0000 7f400da00640 1 -- 172.21.15.16:0/126419072 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55b53c136310 msgr2=0x55b53c1366f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:11:58.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.067+0000 7f400da00640 1 --2- 172.21.15.16:0/126419072 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55b53c136310 0x55b53c1366f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:58.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.067+0000 7f400da00640 1 -- 172.21.15.16:0/126419072 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b53c1832a0 msgr2=0x55b53c135dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:11:58.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.067+0000 7f400da00640 1 --2- 172.21.15.16:0/126419072 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b53c1832a0 0x55b53c135dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:58.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.067+0000 7f400da00640 1 -- 172.21.15.16:0/126419072 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b53bf60ba0 con 0x55b53c182980 2024-03-31T15:11:58.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.067+0000 7f400e400640 1 --2- 172.21.15.16:0/126419072 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55b53c136310 0x55b53c1366f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:11:58.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.067+0000 7f400d000640 1 --2- 172.21.15.16:0/126419072 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b53c1832a0 0x55b53c135dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:11:58.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.067+0000 7f400da00640 1 --2- 172.21.15.16:0/126419072 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b53c182980 0x55b53c182d60 secure :-1 s=READY pgs=1306 cs=0 l=1 rev1=1 crypto rx=0x7f3ffc098350 tx=0x7f3ffc004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e83dd28ec4a1890c server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:11:58.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.067+0000 7f3ffa000640 1 -- 172.21.15.16:0/126419072 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ffc0b2040 con 0x55b53c182980 2024-03-31T15:11:58.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.067+0000 7f3ffa000640 1 -- 172.21.15.16:0/126419072 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3ffc0a5a80 con 0x55b53c182980 2024-03-31T15:11:58.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.067+0000 7f3ffa000640 1 -- 172.21.15.16:0/126419072 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ffc0a5e30 con 0x55b53c182980 2024-03-31T15:11:58.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.067+0000 7f401391a000 1 -- 172.21.15.16:0/126419072 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b53c182980 msgr2=0x55b53c182d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:11:58.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.067+0000 7f401391a000 1 --2- 172.21.15.16:0/126419072 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b53c182980 0x55b53c182d60 secure :-1 s=READY pgs=1306 cs=0 l=1 rev1=1 crypto rx=0x7f3ffc098350 tx=0x7f3ffc004740 comp rx=0 tx=0).stop 2024-03-31T15:11:58.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.067+0000 7f401391a000 1 -- 172.21.15.16:0/126419072 shutdown_connections 2024-03-31T15:11:58.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.067+0000 7f401391a000 1 --2- 172.21.15.16:0/126419072 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55b53c136310 0x55b53c1366f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:58.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.067+0000 7f401391a000 1 --2- 172.21.15.16:0/126419072 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b53c1832a0 0x55b53c135dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:58.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.067+0000 7f401391a000 1 --2- 172.21.15.16:0/126419072 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b53c182980 0x55b53c182d60 unknown :-1 s=CLOSED pgs=1306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:58.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.067+0000 7f401391a000 1 -- 172.21.15.16:0/126419072 >> 172.21.15.16:0/126419072 conn(0x55b53bf339f0 msgr2=0x55b53c165730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:11:58.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.067+0000 7f401391a000 1 -- 172.21.15.16:0/126419072 shutdown_connections 2024-03-31T15:11:58.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.067+0000 7f401391a000 1 -- 172.21.15.16:0/126419072 wait complete. 2024-03-31T15:11:58.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.071+0000 7f401391a000 1 -- start start 2024-03-31T15:11:58.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.071+0000 7f401391a000 1 Processor -- start 2024-03-31T15:11:58.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.071+0000 7f401391a000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b53c136310 0x55b53c182110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:58.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.071+0000 7f401391a000 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b53c182980 0x55b53c13ba90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:58.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.071+0000 7f401391a000 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55b53c1832a0 0x55b53c13bfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:58.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.071+0000 7f401391a000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55b53bf440a0 con 0x55b53c136310 2024-03-31T15:11:58.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.071+0000 7f401391a000 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55b53bf53730 con 0x55b53c182980 2024-03-31T15:11:58.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.071+0000 7f401391a000 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x55b53c039e90 con 0x55b53c1832a0 2024-03-31T15:11:58.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.071+0000 7f401391a000 10 client.0 _ll_register_callbacks cb 0x55b53c123810 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T15:11:58.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.071+0000 7f400da00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b53c136310 0x55b53c182110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:58.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.071+0000 7f400da00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b53c136310 0x55b53c182110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48806/0 (socket says 172.21.15.16:48806) 2024-03-31T15:11:58.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.071+0000 7f400da00640 1 -- 172.21.15.16:0/2354769887 learned_addr learned my addr 172.21.15.16:0/2354769887 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:11:58.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.071+0000 7f400e400640 1 --2- 172.21.15.16:0/2354769887 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55b53c1832a0 0x55b53c13bfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:58.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.071+0000 7f400d000640 1 --2- 172.21.15.16:0/2354769887 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b53c182980 0x55b53c13ba90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:58.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.071+0000 7f400da00640 1 -- 172.21.15.16:0/2354769887 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55b53c1832a0 msgr2=0x55b53c13bfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:11:58.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.071+0000 7f400da00640 1 --2- 172.21.15.16:0/2354769887 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55b53c1832a0 0x55b53c13bfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:58.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.071+0000 7f400da00640 1 -- 172.21.15.16:0/2354769887 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b53c182980 msgr2=0x55b53c13ba90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:11:58.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.071+0000 7f400da00640 1 --2- 172.21.15.16:0/2354769887 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b53c182980 0x55b53c13ba90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:58.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.071+0000 7f400da00640 1 -- 172.21.15.16:0/2354769887 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b53bfcc160 con 0x55b53c136310 2024-03-31T15:11:58.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.071+0000 7f400e400640 1 --2- 172.21.15.16:0/2354769887 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55b53c1832a0 0x55b53c13bfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:11:58.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.071+0000 7f400d000640 1 --2- 172.21.15.16:0/2354769887 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b53c182980 0x55b53c13ba90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:11:58.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.071+0000 7f400da00640 1 --2- 172.21.15.16:0/2354769887 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b53c136310 0x55b53c182110 secure :-1 s=READY pgs=1307 cs=0 l=1 rev1=1 crypto rx=0x7f3ffc098350 tx=0x7f3ffc019810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8dcb3d0bfcff2675 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:11:58.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.071+0000 7f3feea00640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-31T15:11:58.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.071+0000 7f3feea00640 1 -- 172.21.15.16:0/2354769887 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ffc0b2030 con 0x55b53c136310 2024-03-31T15:11:58.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.071+0000 7f401391a000 1 -- 172.21.15.16:0/2354769887 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3ffc0a5a80 con 0x55b53c136310 2024-03-31T15:11:58.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.071+0000 7f3feea00640 1 -- 172.21.15.16:0/2354769887 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 5910+0+0 (secure 0 0 0) 0x7f3ffc003620 con 0x55b53c136310 2024-03-31T15:11:58.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.071+0000 7f3feea00640 1 -- 172.21.15.16:0/2354769887 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=74) ==== 24+0+0 (secure 0 0 0) 0x7f3ffc0d5050 con 0x55b53c136310 2024-03-31T15:11:58.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.071+0000 7f401391a000 10 client.8265 fetch_fsmap learned FSMap version 74 2024-03-31T15:11:58.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.071+0000 7f401391a000 1 -- 172.21.15.16:0/2354769887 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=74,monmap=2+}) -- 0x55b53bf219c0 con 0x55b53c136310 2024-03-31T15:11:58.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.071+0000 7f3feea00640 1 -- 172.21.15.16:0/2354769887 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3ffc0a5a80 con 0x55b53c136310 2024-03-31T15:11:58.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.071+0000 7f3feea00640 1 -- 172.21.15.16:0/2354769887 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 74) ==== 3494+0+0 (secure 0 0 0) 0x7f3ffc003820 con 0x55b53c136310 2024-03-31T15:11:58.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.071+0000 7f401391a000 10 client.8265 fetch_fsmap finished waiting for FSMap version 74 2024-03-31T15:11:58.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.071+0000 7f401391a000 10 client.8265 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-31T15:11:58.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.071+0000 7f401391a000 20 client.8265 populate_metadata read hostname 'smithi016' 2024-03-31T15:11:58.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.071+0000 7f401391a000 1 --2- 172.21.15.16:0/2354769887 >> [v2:172.21.15.16:6832/3780866829,v1:172.21.15.16:6834/3780866829] conn(0x55b53c189dc0 0x55b53c18a1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:58.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.071+0000 7f401391a000 4 client.8265 mds_command: new command op to 8085 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-31T15:11:58.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.071+0000 7f401391a000 1 -- 172.21.15.16:0/2354769887 --> [v2:172.21.15.16:6832/3780866829,v1:172.21.15.16:6834/3780866829] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b53bf21e40 con 0x55b53c189dc0 2024-03-31T15:11:58.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.075+0000 7f400d000640 1 --2- 172.21.15.16:0/2354769887 >> [v2:172.21.15.16:6832/3780866829,v1:172.21.15.16:6834/3780866829] conn(0x55b53c189dc0 0x55b53c18a1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:58.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.075+0000 7f400d000640 1 --2- 172.21.15.16:0/2354769887 >> [v2:172.21.15.16:6832/3780866829,v1:172.21.15.16:6834/3780866829] conn(0x55b53c189dc0 0x55b53c18a1a0 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8a0a92cdd02ff10a server_cookie=8f07610f9ecf5b3c in_seq=0 out_seq=0 2024-03-31T15:11:58.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.075+0000 7f3feea00640 10 client.8265 ms_handle_connect on v2:172.21.15.16:6832/3780866829 2024-03-31T15:11:58.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.075+0000 7f3feea00640 1 -- 172.21.15.16:0/2354769887 <== mds.0 v2:172.21.15.16:6832/3780866829 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55b53bf21e40 con 0x55b53c189dc0 2024-03-31T15:11:58.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.075+0000 7f3feea00640 10 client.8265 handle_command_reply: tid=0 2024-03-31T15:11:58.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.075+0000 7f401391a000 1 client.8265 shutdown 2024-03-31T15:11:58.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.075+0000 7f401391a000 1 -- 172.21.15.16:0/2354769887 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b53c136310 msgr2=0x55b53c182110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:11:58.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.075+0000 7f401391a000 1 --2- 172.21.15.16:0/2354769887 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b53c136310 0x55b53c182110 secure :-1 s=READY pgs=1307 cs=0 l=1 rev1=1 crypto rx=0x7f3ffc098350 tx=0x7f3ffc019810 comp rx=0 tx=0).stop 2024-03-31T15:11:58.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.075+0000 7f401391a000 1 -- 172.21.15.16:0/2354769887 shutdown_connections 2024-03-31T15:11:58.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.075+0000 7f401391a000 1 --2- 172.21.15.16:0/2354769887 >> [v2:172.21.15.16:6832/3780866829,v1:172.21.15.16:6834/3780866829] conn(0x55b53c189dc0 0x55b53c18a1a0 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:58.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.075+0000 7f401391a000 1 --2- 172.21.15.16:0/2354769887 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55b53c1832a0 0x55b53c13bfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:58.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.075+0000 7f401391a000 1 --2- 172.21.15.16:0/2354769887 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b53c182980 0x55b53c13ba90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:58.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.075+0000 7f401391a000 1 --2- 172.21.15.16:0/2354769887 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b53c136310 0x55b53c182110 unknown :-1 s=CLOSED pgs=1307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:58.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.075+0000 7f3feea00640 0 client.8265 ms_handle_reset on v2:172.21.15.16:6832/3780866829 2024-03-31T15:11:58.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.075+0000 7f401391a000 1 -- 172.21.15.16:0/2354769887 >> 172.21.15.16:0/2354769887 conn(0x55b53bf339f0 msgr2=0x55b53c165520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:11:58.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.075+0000 7f401391a000 1 -- 172.21.15.16:0/2354769887 shutdown_connections 2024-03-31T15:11:58.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.075+0000 7f401391a000 1 -- 172.21.15.16:0/2354769887 wait complete. 2024-03-31T15:11:58.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.075+0000 7f401391a000 20 client.8265 trim_cache size 0 max 16384 2024-03-31T15:11:58.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.087+0000 7f401391a000 1 Processor -- start 2024-03-31T15:11:58.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.087+0000 7f401391a000 1 -- start start 2024-03-31T15:11:58.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.091+0000 7f401391a000 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55b53c136310 0x55b53c19a2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:58.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.091+0000 7f401391a000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b53c182980 0x55b53c181140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:58.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.091+0000 7f401391a000 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b53c1832a0 0x55b53c181680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:58.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.091+0000 7f401391a000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f4008203600 con 0x55b53c182980 2024-03-31T15:11:58.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.091+0000 7f401391a000 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f4008202960 con 0x55b53c1832a0 2024-03-31T15:11:58.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.091+0000 7f401391a000 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f4008202fb0 con 0x55b53c136310 2024-03-31T15:11:58.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.091+0000 7f400d000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b53c182980 0x55b53c181140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:58.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.091+0000 7f400d000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b53c182980 0x55b53c181140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48812/0 (socket says 172.21.15.16:48812) 2024-03-31T15:11:58.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.091+0000 7f400d000640 1 -- 172.21.15.16:0/3924992698 learned_addr learned my addr 172.21.15.16:0/3924992698 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:11:58.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.091+0000 7f400da00640 1 --2- 172.21.15.16:0/3924992698 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55b53c136310 0x55b53c19a2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:58.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.091+0000 7f400e400640 1 --2- 172.21.15.16:0/3924992698 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b53c1832a0 0x55b53c181680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:58.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.091+0000 7f400d000640 1 -- 172.21.15.16:0/3924992698 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55b53c136310 msgr2=0x55b53c19a2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:11:58.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.091+0000 7f400d000640 1 --2- 172.21.15.16:0/3924992698 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55b53c136310 0x55b53c19a2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:58.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.091+0000 7f400d000640 1 -- 172.21.15.16:0/3924992698 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b53c1832a0 msgr2=0x55b53c181680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:11:58.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.091+0000 7f400d000640 1 --2- 172.21.15.16:0/3924992698 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b53c1832a0 0x55b53c181680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:58.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.091+0000 7f400d000640 1 -- 172.21.15.16:0/3924992698 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b53bf23af0 con 0x55b53c182980 2024-03-31T15:11:58.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.091+0000 7f400d000640 1 --2- 172.21.15.16:0/3924992698 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b53c182980 0x55b53c181140 secure :-1 s=READY pgs=1308 cs=0 l=1 rev1=1 crypto rx=0x7f3ff40529e0 tx=0x7f3ff40011a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be5220ae8a79d000 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:11:58.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.091+0000 7f3ff9600640 1 -- 172.21.15.16:0/3924992698 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ff4084d00 con 0x55b53c182980 2024-03-31T15:11:58.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.091+0000 7f3ff9600640 1 -- 172.21.15.16:0/3924992698 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3ff4054880 con 0x55b53c182980 2024-03-31T15:11:58.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.091+0000 7f3ff9600640 1 -- 172.21.15.16:0/3924992698 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ff4003cd0 con 0x55b53c182980 2024-03-31T15:11:58.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.091+0000 7f401391a000 1 -- 172.21.15.16:0/3924992698 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b53c182980 msgr2=0x55b53c181140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:11:58.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.091+0000 7f401391a000 1 --2- 172.21.15.16:0/3924992698 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b53c182980 0x55b53c181140 secure :-1 s=READY pgs=1308 cs=0 l=1 rev1=1 crypto rx=0x7f3ff40529e0 tx=0x7f3ff40011a0 comp rx=0 tx=0).stop 2024-03-31T15:11:58.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.091+0000 7f401391a000 1 -- 172.21.15.16:0/3924992698 shutdown_connections 2024-03-31T15:11:58.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.091+0000 7f401391a000 1 --2- 172.21.15.16:0/3924992698 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b53c1832a0 0x55b53c181680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:58.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.091+0000 7f401391a000 1 --2- 172.21.15.16:0/3924992698 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b53c182980 0x55b53c181140 unknown :-1 s=CLOSED pgs=1308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:58.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.091+0000 7f401391a000 1 --2- 172.21.15.16:0/3924992698 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55b53c136310 0x55b53c19a2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:58.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.091+0000 7f401391a000 1 -- 172.21.15.16:0/3924992698 >> 172.21.15.16:0/3924992698 conn(0x55b53bf339f0 msgr2=0x55b53c180a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:11:58.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.091+0000 7f401391a000 1 -- 172.21.15.16:0/3924992698 shutdown_connections 2024-03-31T15:11:58.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.091+0000 7f401391a000 1 -- 172.21.15.16:0/3924992698 wait complete. 2024-03-31T15:11:58.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.091+0000 7f401391a000 1 -- start start 2024-03-31T15:11:58.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.091+0000 7f401391a000 1 Processor -- start 2024-03-31T15:11:58.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.095+0000 7f401391a000 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b53c136310 0x55b53c1f2f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:58.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.095+0000 7f401391a000 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55b53c182980 0x55b53c1f3480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:58.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.095+0000 7f401391a000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b53c1832a0 0x55b53c1866e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:58.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.095+0000 7f401391a000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55b53bf440a0 con 0x55b53c1832a0 2024-03-31T15:11:58.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.095+0000 7f401391a000 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55b53bf53730 con 0x55b53c136310 2024-03-31T15:11:58.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.095+0000 7f401391a000 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x55b53bf927f0 con 0x55b53c182980 2024-03-31T15:11:58.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.095+0000 7f401391a000 10 client.0 _ll_register_callbacks cb 0x7f3ff0003700 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T15:11:58.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.095+0000 7f400e400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b53c1832a0 0x55b53c1866e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:58.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.095+0000 7f400d000640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55b53c182980 0x55b53c1f3480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:58.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.095+0000 7f400da00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b53c136310 0x55b53c1f2f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:58.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.095+0000 7f400e400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b53c1832a0 0x55b53c1866e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48826/0 (socket says 172.21.15.16:48826) 2024-03-31T15:11:58.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.095+0000 7f400e400640 1 -- 172.21.15.16:0/3103817516 learned_addr learned my addr 172.21.15.16:0/3103817516 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:11:58.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.095+0000 7f400da00640 1 -- 172.21.15.16:0/3103817516 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55b53c182980 msgr2=0x55b53c1f3480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:11:58.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.095+0000 7f400da00640 1 --2- 172.21.15.16:0/3103817516 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55b53c182980 0x55b53c1f3480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:58.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.095+0000 7f400da00640 1 -- 172.21.15.16:0/3103817516 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b53c1832a0 msgr2=0x55b53c1866e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:11:58.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.095+0000 7f400da00640 1 --2- 172.21.15.16:0/3103817516 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b53c1832a0 0x55b53c1866e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:58.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.095+0000 7f400da00640 1 -- 172.21.15.16:0/3103817516 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b53bfdf320 con 0x55b53c136310 2024-03-31T15:11:58.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.095+0000 7f400d000640 1 --2- 172.21.15.16:0/3103817516 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55b53c182980 0x55b53c1f3480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:11:58.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.095+0000 7f400e400640 1 --2- 172.21.15.16:0/3103817516 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b53c1832a0 0x55b53c1866e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:11:58.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.095+0000 7f400da00640 1 --2- 172.21.15.16:0/3103817516 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b53c136310 0x55b53c1f2f40 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f3ffc01cfe0 tx=0x7f3ffc0a7070 comp rx=0 tx=0).ready entity=mon.1 client_cookie=982ff45587e6c8dd server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:11:58.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.095+0000 7f3feea00640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-03-31T15:11:58.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.095+0000 7f3feea00640 1 -- 172.21.15.16:0/3103817516 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ffc0b2040 con 0x55b53c136310 2024-03-31T15:11:58.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.095+0000 7f3feea00640 1 -- 172.21.15.16:0/3103817516 <== mon.1 v2:172.21.15.89:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 5910+0+0 (secure 0 0 0) 0x7f3ffc0a59c0 con 0x55b53c136310 2024-03-31T15:11:58.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.095+0000 7f401391a000 1 -- 172.21.15.16:0/3103817516 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3ff4054880 con 0x55b53c136310 2024-03-31T15:11:58.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.095+0000 7f3feea00640 1 -- 172.21.15.16:0/3103817516 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=74) ==== 24+0+0 (secure 0 0 0) 0x7f3ffc003dc0 con 0x55b53c136310 2024-03-31T15:11:58.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.095+0000 7f401391a000 10 client.7906 fetch_fsmap learned FSMap version 74 2024-03-31T15:11:58.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.095+0000 7f401391a000 1 -- 172.21.15.16:0/3103817516 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=74,monmap=2+}) -- 0x55b53bf1aa80 con 0x55b53c136310 2024-03-31T15:11:58.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.095+0000 7f3feea00640 1 -- 172.21.15.16:0/3103817516 <== mon.1 v2:172.21.15.89:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3ff4054880 con 0x55b53c136310 2024-03-31T15:11:58.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.095+0000 7f3feea00640 1 -- 172.21.15.16:0/3103817516 <== mon.1 v2:172.21.15.89:3300/0 5 ==== fsmap(e 74) ==== 3494+0+0 (secure 0 0 0) 0x7f3ffc003620 con 0x55b53c136310 2024-03-31T15:11:58.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.095+0000 7f401391a000 10 client.7906 fetch_fsmap finished waiting for FSMap version 74 2024-03-31T15:11:58.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.095+0000 7f401391a000 10 client.7906 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-31T15:11:58.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.095+0000 7f401391a000 20 client.7906 populate_metadata read hostname 'smithi016' 2024-03-31T15:11:58.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.095+0000 7f401391a000 1 --2- 172.21.15.16:0/3103817516 >> [v2:172.21.15.16:6832/3780866829,v1:172.21.15.16:6834/3780866829] conn(0x55b53c1870a0 0x55b53c187480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:58.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.095+0000 7f401391a000 4 client.7906 mds_command: new command op to 8085 tid=0 multi_id=0 [{"prefix": "damage ls", "format": "json-pretty"}] 2024-03-31T15:11:58.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.095+0000 7f401391a000 1 -- 172.21.15.16:0/3103817516 --> [v2:172.21.15.16:6832/3780866829,v1:172.21.15.16:6834/3780866829] -- command(tid 0: {"prefix": "damage ls", "format": "json-pretty"}) -- 0x55b53bf94190 con 0x55b53c1870a0 2024-03-31T15:11:58.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.095+0000 7f400d000640 1 --2- 172.21.15.16:0/3103817516 >> [v2:172.21.15.16:6832/3780866829,v1:172.21.15.16:6834/3780866829] conn(0x55b53c1870a0 0x55b53c187480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:58.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.099+0000 7f3feea00640 10 client.7906 ms_handle_connect on v2:172.21.15.16:6832/3780866829 2024-03-31T15:11:58.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.099+0000 7f400d000640 1 --2- 172.21.15.16:0/3103817516 >> [v2:172.21.15.16:6832/3780866829,v1:172.21.15.16:6834/3780866829] conn(0x55b53c1870a0 0x55b53c187480 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=61e02222fc4373c2 server_cookie=14fa5a18ac792d66 in_seq=0 out_seq=0 2024-03-31T15:11:58.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.099+0000 7f3feea00640 1 -- 172.21.15.16:0/3103817516 <== mds.0 v2:172.21.15.16:6832/3780866829 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55b53bf94190 con 0x55b53c1870a0 2024-03-31T15:11:58.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.099+0000 7f3feea00640 10 client.7906 handle_command_reply: tid=0 2024-03-31T15:11:58.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.099+0000 7f401391a000 1 client.7906 shutdown 2024-03-31T15:11:58.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.099+0000 7f401391a000 1 -- 172.21.15.16:0/3103817516 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b53c136310 msgr2=0x55b53c1f2f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:11:58.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.099+0000 7f401391a000 1 --2- 172.21.15.16:0/3103817516 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b53c136310 0x55b53c1f2f40 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f3ffc01cfe0 tx=0x7f3ffc0a7070 comp rx=0 tx=0).stop 2024-03-31T15:11:58.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.099+0000 7f401391a000 1 -- 172.21.15.16:0/3103817516 shutdown_connections 2024-03-31T15:11:58.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.099+0000 7f401391a000 1 --2- 172.21.15.16:0/3103817516 >> [v2:172.21.15.16:6832/3780866829,v1:172.21.15.16:6834/3780866829] conn(0x55b53c1870a0 0x55b53c187480 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:58.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.099+0000 7f401391a000 1 --2- 172.21.15.16:0/3103817516 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b53c1832a0 0x55b53c1866e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:58.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.099+0000 7f401391a000 1 --2- 172.21.15.16:0/3103817516 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55b53c182980 0x55b53c1f3480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:58.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.099+0000 7f3feea00640 0 client.7906 ms_handle_reset on v2:172.21.15.16:6832/3780866829 2024-03-31T15:11:58.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.099+0000 7f401391a000 1 --2- 172.21.15.16:0/3103817516 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b53c136310 0x55b53c1f2f40 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:58.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.099+0000 7f401391a000 1 -- 172.21.15.16:0/3103817516 >> 172.21.15.16:0/3103817516 conn(0x55b53bf339f0 msgr2=0x55b53c1812c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:11:58.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.099+0000 7f401391a000 1 -- 172.21.15.16:0/3103817516 shutdown_connections 2024-03-31T15:11:58.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.099+0000 7f401391a000 1 -- 172.21.15.16:0/3103817516 wait complete. 2024-03-31T15:11:58.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.099+0000 7f401391a000 20 client.7906 trim_cache size 0 max 16384 2024-03-31T15:11:58.096 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:11:58.096 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-03-31T15:11:58.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.099+0000 7f4010400640 1 -- 172.21.15.16:0/275366242 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f3fe8068380 msgr2=0x7f3fe806a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:11:58.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.099+0000 7f4010400640 1 --2- 172.21.15.16:0/275366242 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f3fe8068380 0x7f3fe806a840 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f3ffc0af000 tx=0x7f3ffc0046d0 comp rx=0 tx=0).stop 2024-03-31T15:11:58.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.099+0000 7f4010400640 1 -- 172.21.15.16:0/275366242 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f40080cac00 msgr2=0x7f40080c7e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:11:58.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.099+0000 7f4010400640 1 --2- 172.21.15.16:0/275366242 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f40080cac00 0x7f40080c7e20 secure :-1 s=READY pgs=1305 cs=0 l=1 rev1=1 crypto rx=0x7f3ff400e4e0 tx=0x7f3ff400e9a0 comp rx=0 tx=0).stop 2024-03-31T15:11:58.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.099+0000 7f4010400640 1 -- 172.21.15.16:0/275366242 shutdown_connections 2024-03-31T15:11:58.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.099+0000 7f4010400640 1 --2- 172.21.15.16:0/275366242 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f3fe8068380 0x7f3fe806a840 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:58.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.099+0000 7f4010400640 1 --2- 172.21.15.16:0/275366242 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f4008152d70 0x7f40080c8360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:58.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.099+0000 7f4010400640 1 --2- 172.21.15.16:0/275366242 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f40080cac00 0x7f40080c7e20 unknown :-1 s=CLOSED pgs=1305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:58.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.099+0000 7f4010400640 1 --2- 172.21.15.16:0/275366242 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3ff00039f0 0x7f4008152530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:58.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.099+0000 7f4010400640 1 -- 172.21.15.16:0/275366242 >> 172.21.15.16:0/275366242 conn(0x7f4008067730 msgr2=0x7f4008161e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:11:58.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.099+0000 7f4010400640 1 -- 172.21.15.16:0/275366242 shutdown_connections 2024-03-31T15:11:58.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.099+0000 7f4010400640 1 -- 172.21.15.16:0/275366242 wait complete. 2024-03-31T15:11:58.110 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-03-31T15:11:58.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.255+0000 7fd1c4c00640 1 Processor -- start 2024-03-31T15:11:58.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.255+0000 7fd1c4c00640 1 -- start start 2024-03-31T15:11:58.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.259+0000 7fd1c4c00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fd1c0147d60 0x7fd1c00b3130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:58.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.259+0000 7fd1c4c00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fd1c0073310 0x7fd1c006d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:58.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.259+0000 7fd1c4c00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fd1c006da10 0x7fd1c0161f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:58.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.259+0000 7fd1c4c00640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7fd1c0148140 con 0x7fd1c0073310 2024-03-31T15:11:58.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.259+0000 7fd1c4c00640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7fd1c006de60 con 0x7fd1c0147d60 2024-03-31T15:11:58.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.259+0000 7fd1c4c00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fd1c00737c0 con 0x7fd1c006da10 2024-03-31T15:11:58.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.259+0000 7fd1b5400640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fd1c0073310 0x7fd1c006d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:58.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.259+0000 7fd1b5400640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fd1c0073310 0x7fd1c006d4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48842/0 (socket says 172.21.15.16:48842) 2024-03-31T15:11:58.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.259+0000 7fd1b5400640 1 -- 172.21.15.16:0/3208460449 learned_addr learned my addr 172.21.15.16:0/3208460449 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:11:58.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.259+0000 7fd1be800640 1 --2- 172.21.15.16:0/3208460449 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fd1c006da10 0x7fd1c0161f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:58.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.259+0000 7fd1bde00640 1 --2- 172.21.15.16:0/3208460449 >> v2:172.21.15.89:3300/0 conn(0x7fd1c0147d60 0x7fd1c00b3130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:58.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.259+0000 7fd1b5400640 1 -- 172.21.15.16:0/3208460449 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fd1c006da10 msgr2=0x7fd1c0161f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:11:58.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.259+0000 7fd1b5400640 1 --2- 172.21.15.16:0/3208460449 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fd1c006da10 0x7fd1c0161f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:58.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.259+0000 7fd1b5400640 1 -- 172.21.15.16:0/3208460449 >> v2:172.21.15.89:3300/0 conn(0x7fd1c0147d60 msgr2=0x7fd1c00b3130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:11:58.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.259+0000 7fd1b5400640 1 --2- 172.21.15.16:0/3208460449 >> v2:172.21.15.89:3300/0 conn(0x7fd1c0147d60 0x7fd1c00b3130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:58.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.259+0000 7fd1b5400640 1 -- 172.21.15.16:0/3208460449 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1c00b37c0 con 0x7fd1c0073310 2024-03-31T15:11:58.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.263+0000 7fd1b5400640 1 --2- 172.21.15.16:0/3208460449 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd1c0073310 0x7fd1c006d4d0 secure :-1 s=READY pgs=1309 cs=0 l=1 rev1=1 crypto rx=0x7fd1c00b5590 tx=0x7fd1a801c700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d81a5093eefac4b server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:11:58.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.263+0000 7fd1bd400640 1 -- 172.21.15.16:0/3208460449 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1a80a5580 con 0x7fd1c0073310 2024-03-31T15:11:58.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.263+0000 7fd1bd400640 1 -- 172.21.15.16:0/3208460449 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd1a80a5720 con 0x7fd1c0073310 2024-03-31T15:11:58.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.263+0000 7fd1bd400640 1 -- 172.21.15.16:0/3208460449 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1a80a5af0 con 0x7fd1c0073310 2024-03-31T15:11:58.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.263+0000 7fd1c4c00640 1 -- 172.21.15.16:0/3208460449 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd1c0073310 msgr2=0x7fd1c006d4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:11:58.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.263+0000 7fd1c4c00640 1 --2- 172.21.15.16:0/3208460449 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd1c0073310 0x7fd1c006d4d0 secure :-1 s=READY pgs=1309 cs=0 l=1 rev1=1 crypto rx=0x7fd1c00b5590 tx=0x7fd1a801c700 comp rx=0 tx=0).stop 2024-03-31T15:11:58.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.263+0000 7fd1c4c00640 1 -- 172.21.15.16:0/3208460449 shutdown_connections 2024-03-31T15:11:58.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.263+0000 7fd1c4c00640 1 --2- 172.21.15.16:0/3208460449 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fd1c006da10 0x7fd1c0161f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:58.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.263+0000 7fd1c4c00640 1 --2- 172.21.15.16:0/3208460449 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd1c0073310 0x7fd1c006d4d0 unknown :-1 s=CLOSED pgs=1309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:58.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.263+0000 7fd1c4c00640 1 --2- 172.21.15.16:0/3208460449 >> v2:172.21.15.89:3300/0 conn(0x7fd1c0147d60 0x7fd1c00b3130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:58.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.263+0000 7fd1c4c00640 1 -- 172.21.15.16:0/3208460449 >> 172.21.15.16:0/3208460449 conn(0x7fd1c006ee40 msgr2=0x7fd1c006b620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:11:58.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.263+0000 7fd1c4c00640 1 -- 172.21.15.16:0/3208460449 shutdown_connections 2024-03-31T15:11:58.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.263+0000 7fd1c4c00640 1 -- 172.21.15.16:0/3208460449 wait complete. 2024-03-31T15:11:58.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.263+0000 7fd1c4c00640 1 Processor -- start 2024-03-31T15:11:58.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.263+0000 7fd1c4c00640 1 -- start start 2024-03-31T15:11:58.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.263+0000 7fd1c4c00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd1c006da10 0x7fd1c0160930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:58.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.263+0000 7fd1c4c00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fd1c0073310 0x7fd1c015e1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:58.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.263+0000 7fd1c4c00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd1c0147d60 0x7fd1c0149600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:58.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.267+0000 7fd1c4c00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fd1c00ada40 con 0x7fd1c006da10 2024-03-31T15:11:58.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.267+0000 7fd1c4c00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fd1c00b4340 con 0x7fd1c0147d60 2024-03-31T15:11:58.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.267+0000 7fd1c4c00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fd1c00adbc0 con 0x7fd1c0073310 2024-03-31T15:11:58.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.267+0000 7fd1bde00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd1c006da10 0x7fd1c0160930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:58.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.267+0000 7fd1bde00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd1c006da10 0x7fd1c0160930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48846/0 (socket says 172.21.15.16:48846) 2024-03-31T15:11:58.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.267+0000 7fd1bde00640 1 -- 172.21.15.16:0/2683070609 learned_addr learned my addr 172.21.15.16:0/2683070609 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:11:58.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.267+0000 7fd1b5400640 1 --2- 172.21.15.16:0/2683070609 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fd1c0073310 0x7fd1c015e1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:58.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.267+0000 7fd1be800640 1 --2- 172.21.15.16:0/2683070609 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd1c0147d60 0x7fd1c0149600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:58.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.267+0000 7fd1bde00640 1 -- 172.21.15.16:0/2683070609 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fd1c0073310 msgr2=0x7fd1c015e1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:11:58.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.267+0000 7fd1bde00640 1 --2- 172.21.15.16:0/2683070609 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fd1c0073310 0x7fd1c015e1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:58.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.267+0000 7fd1bde00640 1 -- 172.21.15.16:0/2683070609 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd1c0147d60 msgr2=0x7fd1c0149600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:11:58.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.267+0000 7fd1bde00640 1 --2- 172.21.15.16:0/2683070609 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd1c0147d60 0x7fd1c0149600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:58.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.267+0000 7fd1bde00640 1 -- 172.21.15.16:0/2683070609 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1c0148f40 con 0x7fd1c006da10 2024-03-31T15:11:58.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.267+0000 7fd1bde00640 1 --2- 172.21.15.16:0/2683070609 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd1c006da10 0x7fd1c0160930 secure :-1 s=READY pgs=1310 cs=0 l=1 rev1=1 crypto rx=0x7fd1b000d620 tx=0x7fd1b000dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:11:58.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.267+0000 7fd1b6a00640 1 -- 172.21.15.16:0/2683070609 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1b00125f0 con 0x7fd1c006da10 2024-03-31T15:11:58.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.267+0000 7fd1b6a00640 1 -- 172.21.15.16:0/2683070609 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd1b0019070 con 0x7fd1c006da10 2024-03-31T15:11:58.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.267+0000 7fd1c4c00640 1 -- 172.21.15.16:0/2683070609 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd1c0149220 con 0x7fd1c006da10 2024-03-31T15:11:58.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.267+0000 7fd1b6a00640 1 -- 172.21.15.16:0/2683070609 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1b00129b0 con 0x7fd1c006da10 2024-03-31T15:11:58.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.267+0000 7fd1c4c00640 1 -- 172.21.15.16:0/2683070609 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd1c015d9f0 con 0x7fd1c006da10 2024-03-31T15:11:58.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.267+0000 7fd1c4c00640 1 -- 172.21.15.16:0/2683070609 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd1c0069950 con 0x7fd1c006da10 2024-03-31T15:11:58.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.267+0000 7fd1b6a00640 1 -- 172.21.15.16:0/2683070609 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fd1b0012b50 con 0x7fd1c006da10 2024-03-31T15:11:58.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.271+0000 7fd1b6a00640 1 --2- 172.21.15.16:0/2683070609 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fd19c068380 0x7fd19c06a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:58.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.271+0000 7fd1b6a00640 1 -- 172.21.15.16:0/2683070609 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 5910+0+0 (secure 0 0 0) 0x7fd1b0086cd0 con 0x7fd1c006da10 2024-03-31T15:11:58.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.271+0000 7fd1b5400640 1 --2- 172.21.15.16:0/2683070609 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fd19c068380 0x7fd19c06a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:58.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.271+0000 7fd1b5400640 1 --2- 172.21.15.16:0/2683070609 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fd19c068380 0x7fd19c06a840 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7fd1a80ae000 tx=0x7fd1a80a7070 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:11:58.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.275+0000 7fd1b6a00640 1 -- 172.21.15.16:0/2683070609 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fd1b0056b70 con 0x7fd1c006da10 2024-03-31T15:11:58.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.479+0000 7fd1c4c00640 1 -- 172.21.15.16:0/2683070609 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7fd1c006e470 con 0x7fd1c006da10 2024-03-31T15:11:58.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.479+0000 7fd1b6a00640 1 -- 172.21.15.16:0/2683070609 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+46 (secure 0 0 0) 0x7fd1b005ae70 con 0x7fd1c006da10 2024-03-31T15:11:58.472 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:11:58.472 INFO:teuthology.orchestra.run.smithi016.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-03-31T15:11:58.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.479+0000 7fd1c4c00640 1 -- 172.21.15.16:0/2683070609 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fd19c068380 msgr2=0x7fd19c06a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:11:58.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.479+0000 7fd1c4c00640 1 --2- 172.21.15.16:0/2683070609 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fd19c068380 0x7fd19c06a840 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7fd1a80ae000 tx=0x7fd1a80a7070 comp rx=0 tx=0).stop 2024-03-31T15:11:58.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.479+0000 7fd1c4c00640 1 -- 172.21.15.16:0/2683070609 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd1c006da10 msgr2=0x7fd1c0160930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:11:58.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.479+0000 7fd1c4c00640 1 --2- 172.21.15.16:0/2683070609 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd1c006da10 0x7fd1c0160930 secure :-1 s=READY pgs=1310 cs=0 l=1 rev1=1 crypto rx=0x7fd1b000d620 tx=0x7fd1b000dae0 comp rx=0 tx=0).stop 2024-03-31T15:11:58.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.479+0000 7fd1c4c00640 1 -- 172.21.15.16:0/2683070609 shutdown_connections 2024-03-31T15:11:58.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.479+0000 7fd1c4c00640 1 --2- 172.21.15.16:0/2683070609 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fd19c068380 0x7fd19c06a840 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:58.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.479+0000 7fd1c4c00640 1 --2- 172.21.15.16:0/2683070609 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd1c0147d60 0x7fd1c0149600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:58.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.479+0000 7fd1c4c00640 1 --2- 172.21.15.16:0/2683070609 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fd1c0073310 0x7fd1c015e1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:58.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.479+0000 7fd1c4c00640 1 --2- 172.21.15.16:0/2683070609 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd1c006da10 0x7fd1c0160930 unknown :-1 s=CLOSED pgs=1310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:58.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.479+0000 7fd1c4c00640 1 -- 172.21.15.16:0/2683070609 >> 172.21.15.16:0/2683070609 conn(0x7fd1c006ee40 msgr2=0x7fd1c00afe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:11:58.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.479+0000 7fd1c4c00640 1 -- 172.21.15.16:0/2683070609 shutdown_connections 2024-03-31T15:11:58.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.479+0000 7fd1c4c00640 1 -- 172.21.15.16:0/2683070609 wait complete. 2024-03-31T15:11:58.494 DEBUG:tasks.ceph_test_case:wait_until_true: success in 0s and 0 retries 2024-03-31T15:11:58.494 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-31T15:11:58.494 DEBUG:teuthology.orchestra.run.smithi016:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-31T15:11:58.576 DEBUG:teuthology.orchestra.run.smithi089:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-31T15:11:58.670 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-31T15:11:58.670 INFO:tasks.cephfs.mount:Terminating background process 2024-03-31T15:11:58.670 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-03-31T15:11:58.670 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-31T15:11:58.679 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-31T15:11:58.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.867+0000 7fe8e5800640 1 Processor -- start 2024-03-31T15:11:58.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.867+0000 7fe8e5800640 1 -- start start 2024-03-31T15:11:58.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.867+0000 7fe8e5800640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fe8e006f920 0x7fe8e006fd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:58.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.867+0000 7fe8e5800640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fe8e006d490 0x7fe8e006d8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:58.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.867+0000 7fe8e5800640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe8e0162010 con 0x7fe8e00cb440 2024-03-31T15:11:58.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.867+0000 7fe8e5800640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7fe8e0162be0 con 0x7fe8e006f920 2024-03-31T15:11:58.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.867+0000 7fe8e5800640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fe8e0070240 con 0x7fe8e006d490 2024-03-31T15:11:58.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.871+0000 7fe8dea00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fe8e006f920 0x7fe8e006fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:58.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.871+0000 7fe8dea00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fe8e006f920 0x7fe8e006fd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48858/0 (socket says 172.21.15.16:48858) 2024-03-31T15:11:58.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.871+0000 7fe8dea00640 1 -- 172.21.15.16:0/1682597707 learned_addr learned my addr 172.21.15.16:0/1682597707 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:11:58.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.871+0000 7fe8de000640 1 --2- 172.21.15.16:0/1682597707 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fe8e006d490 0x7fe8e006d8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:58.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.871+0000 7fe8dea00640 1 -- 172.21.15.16:0/1682597707 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fe8e006d490 msgr2=0x7fe8e006d8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:11:58.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.871+0000 7fe8dea00640 1 --2- 172.21.15.16:0/1682597707 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fe8e006d490 0x7fe8e006d8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:58.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.871+0000 7fe8dea00640 1 -- 172.21.15.16:0/1682597707 >> v1:172.21.15.89:6789/0 conn(0x7fe8e00cb440 legacy=0x7fe8e00cb8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:11:58.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.871+0000 7fe8dea00640 1 -- 172.21.15.16:0/1682597707 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8e01622b0 con 0x7fe8e006f920 2024-03-31T15:11:58.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.871+0000 7fe8de000640 1 --2- 172.21.15.16:0/1682597707 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fe8e006d490 0x7fe8e006d8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:11:58.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.871+0000 7fe8dea00640 1 --2- 172.21.15.16:0/1682597707 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe8e006f920 0x7fe8e006fd00 secure :-1 s=READY pgs=1311 cs=0 l=1 rev1=1 crypto rx=0x7fe8e0162d60 tx=0x7fe8d001c510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6475faec6982fb82 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:11:58.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.871+0000 7fe8dd600640 1 -- 172.21.15.16:0/1682597707 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8d00aa670 con 0x7fe8e006f920 2024-03-31T15:11:58.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.871+0000 7fe8dd600640 1 -- 172.21.15.16:0/1682597707 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe8d00af070 con 0x7fe8e006f920 2024-03-31T15:11:58.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.871+0000 7fe8dd600640 1 -- 172.21.15.16:0/1682597707 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8d00aaa30 con 0x7fe8e006f920 2024-03-31T15:11:58.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.875+0000 7fe8e5800640 1 -- 172.21.15.16:0/1682597707 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe8e006f920 msgr2=0x7fe8e006fd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:11:58.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.875+0000 7fe8e5800640 1 --2- 172.21.15.16:0/1682597707 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe8e006f920 0x7fe8e006fd00 secure :-1 s=READY pgs=1311 cs=0 l=1 rev1=1 crypto rx=0x7fe8e0162d60 tx=0x7fe8d001c510 comp rx=0 tx=0).stop 2024-03-31T15:11:58.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.875+0000 7fe8e5800640 1 -- 172.21.15.16:0/1682597707 shutdown_connections 2024-03-31T15:11:58.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.875+0000 7fe8e5800640 1 --2- 172.21.15.16:0/1682597707 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fe8e006d490 0x7fe8e006d8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:58.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.875+0000 7fe8e5800640 1 --2- 172.21.15.16:0/1682597707 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe8e006f920 0x7fe8e006fd00 unknown :-1 s=CLOSED pgs=1311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:58.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.875+0000 7fe8e5800640 1 -- 172.21.15.16:0/1682597707 >> 172.21.15.16:0/1682597707 conn(0x7fe8e0067330 msgr2=0x7fe8e0067740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:11:58.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.875+0000 7fe8e5800640 1 -- 172.21.15.16:0/1682597707 shutdown_connections 2024-03-31T15:11:58.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.875+0000 7fe8e5800640 1 -- 172.21.15.16:0/1682597707 wait complete. 2024-03-31T15:11:58.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.875+0000 7fe8e5800640 1 Processor -- start 2024-03-31T15:11:58.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.875+0000 7fe8e5800640 1 -- start start 2024-03-31T15:11:58.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.875+0000 7fe8e5800640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe8e006d490 0x7fe8e00cbb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:58.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.875+0000 7fe8e5800640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe8e006f920 0x7fe8e0138ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:58.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.875+0000 7fe8e5800640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fe8e00cb440 0x7fe8e0139000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:58.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.875+0000 7fe8e5800640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fe8e006dfb0 con 0x7fe8e006f920 2024-03-31T15:11:58.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.875+0000 7fe8e5800640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fe8e006de30 con 0x7fe8e006d490 2024-03-31T15:11:58.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.875+0000 7fe8e5800640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fe8e00cd750 con 0x7fe8e00cb440 2024-03-31T15:11:58.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.875+0000 7fe8de000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe8e006f920 0x7fe8e0138ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:58.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.875+0000 7fe8de000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe8e006f920 0x7fe8e0138ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48862/0 (socket says 172.21.15.16:48862) 2024-03-31T15:11:58.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.875+0000 7fe8de000640 1 -- 172.21.15.16:0/92195742 learned_addr learned my addr 172.21.15.16:0/92195742 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:11:58.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.875+0000 7fe8dea00640 1 --2- 172.21.15.16:0/92195742 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe8e006d490 0x7fe8e00cbb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:58.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.875+0000 7fe8df400640 1 --2- 172.21.15.16:0/92195742 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fe8e00cb440 0x7fe8e0139000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:58.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.875+0000 7fe8de000640 1 -- 172.21.15.16:0/92195742 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fe8e00cb440 msgr2=0x7fe8e0139000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:11:58.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.875+0000 7fe8de000640 1 --2- 172.21.15.16:0/92195742 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fe8e00cb440 0x7fe8e0139000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:58.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.875+0000 7fe8de000640 1 -- 172.21.15.16:0/92195742 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe8e006d490 msgr2=0x7fe8e00cbb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:11:58.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.875+0000 7fe8de000640 1 --2- 172.21.15.16:0/92195742 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe8e006d490 0x7fe8e00cbb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:58.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.875+0000 7fe8de000640 1 -- 172.21.15.16:0/92195742 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8e0139680 con 0x7fe8e006f920 2024-03-31T15:11:58.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.875+0000 7fe8de000640 1 --2- 172.21.15.16:0/92195742 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe8e006f920 0x7fe8e0138ac0 secure :-1 s=READY pgs=1312 cs=0 l=1 rev1=1 crypto rx=0x7fe8cc00e840 tx=0x7fe8cc00ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:11:58.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.879+0000 7fe8d6a00640 1 -- 172.21.15.16:0/92195742 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8cc017070 con 0x7fe8e006f920 2024-03-31T15:11:58.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.879+0000 7fe8d6a00640 1 -- 172.21.15.16:0/92195742 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe8cc00c8d0 con 0x7fe8e006f920 2024-03-31T15:11:58.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.879+0000 7fe8d6a00640 1 -- 172.21.15.16:0/92195742 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8cc00f040 con 0x7fe8e006f920 2024-03-31T15:11:58.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.879+0000 7fe8e5800640 1 -- 172.21.15.16:0/92195742 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe8e0139960 con 0x7fe8e006f920 2024-03-31T15:11:58.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.879+0000 7fe8e5800640 1 -- 172.21.15.16:0/92195742 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe8e00b5a90 con 0x7fe8e006f920 2024-03-31T15:11:58.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.879+0000 7fe8e5800640 1 -- 172.21.15.16:0/92195742 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe8e0069950 con 0x7fe8e006f920 2024-03-31T15:11:58.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.879+0000 7fe8d6a00640 1 -- 172.21.15.16:0/92195742 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fe8cc01a070 con 0x7fe8e006f920 2024-03-31T15:11:58.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.879+0000 7fe8d6a00640 1 --2- 172.21.15.16:0/92195742 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fe8c0068380 0x7fe8c006a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:58.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.879+0000 7fe8d6a00640 1 -- 172.21.15.16:0/92195742 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 5910+0+0 (secure 0 0 0) 0x7fe8cc086630 con 0x7fe8e006f920 2024-03-31T15:11:58.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.879+0000 7fe8dea00640 1 --2- 172.21.15.16:0/92195742 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fe8c0068380 0x7fe8c006a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:58.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.883+0000 7fe8dea00640 1 --2- 172.21.15.16:0/92195742 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fe8c0068380 0x7fe8c006a840 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7fe8d00ae000 tx=0x7fe8d00a7000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:11:58.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:58.883+0000 7fe8d6a00640 1 -- 172.21.15.16:0/92195742 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fe8cc0564c0 con 0x7fe8e006f920 2024-03-31T15:11:59.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:59.051+0000 7fe8e5800640 1 -- 172.21.15.16:0/92195742 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fe8e00cf3f0 con 0x7fe8e006f920 2024-03-31T15:11:59.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:59.051+0000 7fe8d6a00640 1 -- 172.21.15.16:0/92195742 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v85) ==== 74+0+16894 (secure 0 0 0) 0x7fe8cc05a7c0 con 0x7fe8e006f920 2024-03-31T15:11:59.045 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:11:59.045 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":85,"fsid":"286eb5da-fb47-4550-b290-d9519c13b9a5","created":"2024-03-31T15:03:11.438952+0000","modified":"2024-03-31T15:11:19.402716+0000","last_up_change":"2024-03-31T15:03:22.885535+0000","last_in_change":"2024-03-31T15:03:13.941667+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-31T15:03:22.550787+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-03-31T15:09:49.322620+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-03-31T15:09:49.877778+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"92a66450-ef19-40d8-ac3d-d760b23b0bda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6801","nonce":932712244}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6803","nonce":932712244}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6807","nonce":932712244}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6805","nonce":932712244}]},"public_addr":"172.21.15.16:6801/932712244","cluster_addr":"172.21.15.16:6803/932712244","heartbeat_back_addr":"172.21.15.16:6807/932712244","heartbeat_front_addr":"172.21.15.16:6805/932712244","state":["exists","up"]},{"osd":1,"uuid":"da9d1f2b-1968-4959-b843-faa231988e92","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6810","nonce":1398737533}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6811","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6813","nonce":1398737533}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6821","nonce":1398737533}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6815","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6816","nonce":1398737533}]},"public_addr":"172.21.15.16:6810/1398737533","cluster_addr":"172.21.15.16:6813/1398737533","heartbeat_back_addr":"172.21.15.16:6821/1398737533","heartbeat_front_addr":"172.21.15.16:6816/1398737533","state":["exists","up"]},{"osd":2,"uuid":"01a73662-9a21-4b1f-bcbb-42b9636a4638","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6819","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6822","nonce":3018526646}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6825","nonce":3018526646}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6831","nonce":3018526646}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6827","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6829","nonce":3018526646}]},"public_addr":"172.21.15.16:6822/3018526646","cluster_addr":"172.21.15.16:6825/3018526646","heartbeat_back_addr":"172.21.15.16:6831/3018526646","heartbeat_front_addr":"172.21.15.16:6829/3018526646","state":["exists","up"]},{"osd":3,"uuid":"c7771fd3-e18e-4500-ac38-23cf10ebcd75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6809","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6812","nonce":3360352974}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6817","nonce":3360352974}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6828","nonce":3360352974}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6823","nonce":3360352974}]},"public_addr":"172.21.15.16:6812/3360352974","cluster_addr":"172.21.15.16:6817/3360352974","heartbeat_back_addr":"172.21.15.16:6828/3360352974","heartbeat_front_addr":"172.21.15.16:6823/3360352974","state":["exists","up"]},{"osd":4,"uuid":"88260730-1482-4fbd-a40f-cb1124e9df4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6801","nonce":1856844856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6803","nonce":1856844856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6807","nonce":1856844856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6805","nonce":1856844856}]},"public_addr":"172.21.15.89:6801/1856844856","cluster_addr":"172.21.15.89:6803/1856844856","heartbeat_back_addr":"172.21.15.89:6807/1856844856","heartbeat_front_addr":"172.21.15.89:6805/1856844856","state":["exists","up"]},{"osd":5,"uuid":"4cb2c00c-fae0-4dd8-9d27-ec4f0093ccf7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6814","nonce":50395527}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6817","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6820","nonce":50395527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6831","nonce":50395527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6823","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6826","nonce":50395527}]},"public_addr":"172.21.15.89:6814/50395527","cluster_addr":"172.21.15.89:6820/50395527","heartbeat_back_addr":"172.21.15.89:6831/50395527","heartbeat_front_addr":"172.21.15.89:6826/50395527","state":["exists","up"]},{"osd":6,"uuid":"81ce0c19-2554-4cae-acbe-66a05e9e9b9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6809","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6812","nonce":3470471752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6815","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6818","nonce":3470471752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6827","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6829","nonce":3470471752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6821","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6824","nonce":3470471752}]},"public_addr":"172.21.15.89:6812/3470471752","cluster_addr":"172.21.15.89:6818/3470471752","heartbeat_back_addr":"172.21.15.89:6829/3470471752","heartbeat_front_addr":"172.21.15.89:6824/3470471752","state":["exists","up"]},{"osd":7,"uuid":"85e4313b-d37d-442e-8042-153bf20a847d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6811","nonce":1796778630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6813","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6816","nonce":1796778630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6825","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6828","nonce":1796778630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6819","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6822","nonce":1796778630}]},"public_addr":"172.21.15.89:6811/1796778630","cluster_addr":"172.21.15.89:6816/1796778630","heartbeat_back_addr":"172.21.15.89:6828/1796778630","heartbeat_front_addr":"172.21.15.89:6822/1796778630","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.439088+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.529970+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.680512+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:21.044161+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.582111+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.536558+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.409522+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.590229+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"13.d","mappings":[{"from":1,"to":3}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:6832/818074038":"2024-04-01T15:11:19.402687+0000","172.21.15.16:6834/818074038":"2024-04-01T15:11:19.402687+0000","172.21.15.16:6834/2478530200":"2024-04-01T15:10:04.360772+0000","172.21.15.16:6832/2478530200":"2024-04-01T15:10:04.360772+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-31T15:11:59.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:59.055+0000 7fe8e5800640 1 -- 172.21.15.16:0/92195742 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fe8c0068380 msgr2=0x7fe8c006a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:11:59.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:59.055+0000 7fe8e5800640 1 --2- 172.21.15.16:0/92195742 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fe8c0068380 0x7fe8c006a840 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7fe8d00ae000 tx=0x7fe8d00a7000 comp rx=0 tx=0).stop 2024-03-31T15:11:59.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:59.055+0000 7fe8e5800640 1 -- 172.21.15.16:0/92195742 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe8e006f920 msgr2=0x7fe8e0138ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:11:59.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:59.055+0000 7fe8e5800640 1 --2- 172.21.15.16:0/92195742 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe8e006f920 0x7fe8e0138ac0 secure :-1 s=READY pgs=1312 cs=0 l=1 rev1=1 crypto rx=0x7fe8cc00e840 tx=0x7fe8cc00ed00 comp rx=0 tx=0).stop 2024-03-31T15:11:59.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:59.055+0000 7fe8e5800640 1 -- 172.21.15.16:0/92195742 shutdown_connections 2024-03-31T15:11:59.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:59.055+0000 7fe8e5800640 1 --2- 172.21.15.16:0/92195742 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fe8c0068380 0x7fe8c006a840 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:59.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:59.055+0000 7fe8e5800640 1 --2- 172.21.15.16:0/92195742 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fe8e00cb440 0x7fe8e0139000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:59.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:59.055+0000 7fe8e5800640 1 --2- 172.21.15.16:0/92195742 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe8e006f920 0x7fe8e0138ac0 unknown :-1 s=CLOSED pgs=1312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:59.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:59.055+0000 7fe8e5800640 1 --2- 172.21.15.16:0/92195742 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe8e006d490 0x7fe8e00cbb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:59.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:59.055+0000 7fe8e5800640 1 -- 172.21.15.16:0/92195742 >> 172.21.15.16:0/92195742 conn(0x7fe8e0067330 msgr2=0x7fe8e00d2930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:11:59.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:59.055+0000 7fe8e5800640 1 -- 172.21.15.16:0/92195742 shutdown_connections 2024-03-31T15:11:59.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:59.055+0000 7fe8e5800640 1 -- 172.21.15.16:0/92195742 wait complete. 2024-03-31T15:11:59.066 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-31T15:03:22.550787+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 12, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-31T15:09:49.322620+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '80', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 13, 'pool_name': 'cephfs_data', 'create_time': '2024-03-31T15:09:49.877778+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '81', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-31T15:11:59.067 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-31T15:11:59.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:59.219+0000 7f2b4c000640 1 Processor -- start 2024-03-31T15:11:59.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:59.219+0000 7f2b4c000640 1 -- start start 2024-03-31T15:11:59.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:59.219+0000 7f2b4c000640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f2b440d0a60 0x7f2b440cea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:59.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:59.219+0000 7f2b4c000640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f2b440cef80 0x7f2b441419c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:59.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:59.219+0000 7f2b4c000640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2b4415ca70 con 0x7f2b4406f920 2024-03-31T15:11:59.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:59.219+0000 7f2b4c000640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f2b4413f670 con 0x7f2b440cef80 2024-03-31T15:11:59.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:59.219+0000 7f2b4c000640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f2b4415cd50 con 0x7f2b440d0a60 2024-03-31T15:11:59.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:59.219+0000 7f2b4a000640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f2b440cef80 0x7f2b441419c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:59.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:59.219+0000 7f2b4a000640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f2b440cef80 0x7f2b441419c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48866/0 (socket says 172.21.15.16:48866) 2024-03-31T15:11:59.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:59.219+0000 7f2b4a000640 1 -- 172.21.15.16:0/719244869 learned_addr learned my addr 172.21.15.16:0/719244869 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:11:59.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:59.219+0000 7f2b48c00640 1 --2- 172.21.15.16:0/719244869 >> v2:172.21.15.89:3300/0 conn(0x7f2b440d0a60 0x7f2b440cea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:59.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:59.219+0000 7f2b4a000640 1 -- 172.21.15.16:0/719244869 >> v2:172.21.15.89:3300/0 conn(0x7f2b440d0a60 msgr2=0x7f2b440cea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:11:59.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:59.219+0000 7f2b4a000640 1 --2- 172.21.15.16:0/719244869 >> v2:172.21.15.89:3300/0 conn(0x7f2b440d0a60 0x7f2b440cea40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:59.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:59.219+0000 7f2b4a000640 1 -- 172.21.15.16:0/719244869 >> v1:172.21.15.89:6789/0 conn(0x7f2b4406f920 legacy=0x7f2b4406fd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:11:59.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:59.219+0000 7f2b4a000640 1 -- 172.21.15.16:0/719244869 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b4415d030 con 0x7f2b440cef80 2024-03-31T15:11:59.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:59.223+0000 7f2b4a000640 1 --2- 172.21.15.16:0/719244869 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2b440cef80 0x7f2b441419c0 secure :-1 s=READY pgs=1313 cs=0 l=1 rev1=1 crypto rx=0x7f2b440b5070 tx=0x7f2b3c01c9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9e7ab7c29bac150 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:11:59.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:59.223+0000 7f2b43e00640 1 -- 172.21.15.16:0/719244869 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b3c0a5670 con 0x7f2b440cef80 2024-03-31T15:11:59.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:59.223+0000 7f2b43e00640 1 -- 172.21.15.16:0/719244869 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2b3c0a5810 con 0x7f2b440cef80 2024-03-31T15:11:59.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:59.223+0000 7f2b43e00640 1 -- 172.21.15.16:0/719244869 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b3c0a5be0 con 0x7f2b440cef80 2024-03-31T15:11:59.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:59.223+0000 7f2b4c000640 1 -- 172.21.15.16:0/719244869 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2b440cef80 msgr2=0x7f2b441419c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:11:59.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:59.223+0000 7f2b4c000640 1 --2- 172.21.15.16:0/719244869 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2b440cef80 0x7f2b441419c0 secure :-1 s=READY pgs=1313 cs=0 l=1 rev1=1 crypto rx=0x7f2b440b5070 tx=0x7f2b3c01c9b0 comp rx=0 tx=0).stop 2024-03-31T15:11:59.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:59.223+0000 7f2b4c000640 1 -- 172.21.15.16:0/719244869 shutdown_connections 2024-03-31T15:11:59.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:59.223+0000 7f2b4c000640 1 --2- 172.21.15.16:0/719244869 >> v2:172.21.15.89:3300/0 conn(0x7f2b440d0a60 0x7f2b440cea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:59.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:59.223+0000 7f2b4c000640 1 --2- 172.21.15.16:0/719244869 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2b440cef80 0x7f2b441419c0 unknown :-1 s=CLOSED pgs=1313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:59.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:59.223+0000 7f2b4c000640 1 -- 172.21.15.16:0/719244869 >> 172.21.15.16:0/719244869 conn(0x7f2b44067930 msgr2=0x7f2b440b3ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:11:59.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:59.223+0000 7f2b4c000640 1 -- 172.21.15.16:0/719244869 shutdown_connections 2024-03-31T15:11:59.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:59.223+0000 7f2b4c000640 1 -- 172.21.15.16:0/719244869 wait complete. 2024-03-31T15:11:59.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:59.223+0000 7f2b4c000640 1 Processor -- start 2024-03-31T15:11:59.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:59.223+0000 7f2b4c000640 1 -- start start 2024-03-31T15:11:59.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:59.227+0000 7f2b4c000640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2b4406f920 0x7f2b44156450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:59.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:59.227+0000 7f2b4c000640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f2b440cef80 0x7f2b440b1960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:59.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:59.227+0000 7f2b4c000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2b440d0a60 0x7f2b44070e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:59.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:59.227+0000 7f2b4c000640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f2b44169230 con 0x7f2b440d0a60 2024-03-31T15:11:59.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:59.227+0000 7f2b4c000640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f2b441690b0 con 0x7f2b4406f920 2024-03-31T15:11:59.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:59.227+0000 7f2b4c000640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f2b441693b0 con 0x7f2b440cef80 2024-03-31T15:11:59.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:59.227+0000 7f2b4a000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2b440d0a60 0x7f2b44070e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:59.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:59.227+0000 7f2b4a000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2b440d0a60 0x7f2b44070e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48874/0 (socket says 172.21.15.16:48874) 2024-03-31T15:11:59.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:59.227+0000 7f2b4a000640 1 -- 172.21.15.16:0/1478826416 learned_addr learned my addr 172.21.15.16:0/1478826416 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:11:59.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:59.227+0000 7f2b49600640 1 --2- 172.21.15.16:0/1478826416 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2b4406f920 0x7f2b44156450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:59.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:59.227+0000 7f2b48c00640 1 --2- 172.21.15.16:0/1478826416 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f2b440cef80 0x7f2b440b1960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:59.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:59.227+0000 7f2b4a000640 1 -- 172.21.15.16:0/1478826416 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f2b440cef80 msgr2=0x7f2b440b1960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:11:59.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:59.227+0000 7f2b4a000640 1 --2- 172.21.15.16:0/1478826416 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f2b440cef80 0x7f2b440b1960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:59.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:59.227+0000 7f2b4a000640 1 -- 172.21.15.16:0/1478826416 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2b4406f920 msgr2=0x7f2b44156450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:11:59.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:59.227+0000 7f2b4a000640 1 --2- 172.21.15.16:0/1478826416 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2b4406f920 0x7f2b44156450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:59.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:59.227+0000 7f2b4a000640 1 -- 172.21.15.16:0/1478826416 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b44156990 con 0x7f2b440d0a60 2024-03-31T15:11:59.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:59.227+0000 7f2b49600640 1 --2- 172.21.15.16:0/1478826416 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2b4406f920 0x7f2b44156450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:11:59.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:59.227+0000 7f2b48c00640 1 --2- 172.21.15.16:0/1478826416 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f2b440cef80 0x7f2b440b1960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:11:59.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:59.227+0000 7f2b4a000640 1 --2- 172.21.15.16:0/1478826416 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2b440d0a60 0x7f2b44070e60 secure :-1 s=READY pgs=1314 cs=0 l=1 rev1=1 crypto rx=0x7f2b3c0af040 tx=0x7f2b3c019750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:11:59.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:59.227+0000 7f2b41600640 1 -- 172.21.15.16:0/1478826416 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b3c0bc020 con 0x7f2b440d0a60 2024-03-31T15:11:59.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:59.227+0000 7f2b41600640 1 -- 172.21.15.16:0/1478826416 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2b3c0b8670 con 0x7f2b440d0a60 2024-03-31T15:11:59.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:59.227+0000 7f2b4c000640 1 -- 172.21.15.16:0/1478826416 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2b44157790 con 0x7f2b440d0a60 2024-03-31T15:11:59.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:59.227+0000 7f2b41600640 1 -- 172.21.15.16:0/1478826416 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b3c009510 con 0x7f2b440d0a60 2024-03-31T15:11:59.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:59.227+0000 7f2b4c000640 1 -- 172.21.15.16:0/1478826416 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2b44157d50 con 0x7f2b440d0a60 2024-03-31T15:11:59.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:59.227+0000 7f2b4c000640 1 -- 172.21.15.16:0/1478826416 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2b44069950 con 0x7f2b440d0a60 2024-03-31T15:11:59.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:59.231+0000 7f2b41600640 1 -- 172.21.15.16:0/1478826416 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f2b3c0bf020 con 0x7f2b440d0a60 2024-03-31T15:11:59.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:59.231+0000 7f2b41600640 1 --2- 172.21.15.16:0/1478826416 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f2b24068380 0x7f2b2406a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:59.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:59.231+0000 7f2b41600640 1 -- 172.21.15.16:0/1478826416 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 5910+0+0 (secure 0 0 0) 0x7f2b3c1219f0 con 0x7f2b440d0a60 2024-03-31T15:11:59.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:59.231+0000 7f2b49600640 1 --2- 172.21.15.16:0/1478826416 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f2b24068380 0x7f2b2406a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:59.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:59.231+0000 7f2b49600640 1 --2- 172.21.15.16:0/1478826416 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f2b24068380 0x7f2b2406a840 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f2b440d1dc0 tx=0x7f2b38009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:11:59.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:59.235+0000 7f2b41600640 1 -- 172.21.15.16:0/1478826416 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f2b3c0c6030 con 0x7f2b440d0a60 2024-03-31T15:11:59.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:59.399+0000 7f2b4c000640 1 -- 172.21.15.16:0/1478826416 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f2b440b50e0 con 0x7f2b440d0a60 2024-03-31T15:11:59.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:59.399+0000 7f2b41600640 1 -- 172.21.15.16:0/1478826416 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v85) ==== 93+0+10 (secure 0 0 0) 0x7f2b44156990 con 0x7f2b440d0a60 2024-03-31T15:11:59.394 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2024-03-31T15:11:59.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:59.403+0000 7f2b4c000640 1 -- 172.21.15.16:0/1478826416 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f2b24068380 msgr2=0x7f2b2406a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:11:59.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:59.403+0000 7f2b4c000640 1 --2- 172.21.15.16:0/1478826416 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f2b24068380 0x7f2b2406a840 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f2b440d1dc0 tx=0x7f2b38009040 comp rx=0 tx=0).stop 2024-03-31T15:11:59.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:59.403+0000 7f2b4c000640 1 -- 172.21.15.16:0/1478826416 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2b440d0a60 msgr2=0x7f2b44070e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:11:59.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:59.403+0000 7f2b4c000640 1 --2- 172.21.15.16:0/1478826416 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2b440d0a60 0x7f2b44070e60 secure :-1 s=READY pgs=1314 cs=0 l=1 rev1=1 crypto rx=0x7f2b3c0af040 tx=0x7f2b3c019750 comp rx=0 tx=0).stop 2024-03-31T15:11:59.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:59.403+0000 7f2b4c000640 1 -- 172.21.15.16:0/1478826416 shutdown_connections 2024-03-31T15:11:59.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:59.403+0000 7f2b4c000640 1 --2- 172.21.15.16:0/1478826416 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f2b24068380 0x7f2b2406a840 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:59.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:59.403+0000 7f2b4c000640 1 --2- 172.21.15.16:0/1478826416 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2b440d0a60 0x7f2b44070e60 unknown :-1 s=CLOSED pgs=1314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:59.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:59.403+0000 7f2b4c000640 1 --2- 172.21.15.16:0/1478826416 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f2b440cef80 0x7f2b440b1960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:59.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:59.403+0000 7f2b4c000640 1 --2- 172.21.15.16:0/1478826416 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2b4406f920 0x7f2b44156450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:59.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:59.403+0000 7f2b4c000640 1 -- 172.21.15.16:0/1478826416 >> 172.21.15.16:0/1478826416 conn(0x7f2b44067930 msgr2=0x7f2b4415a320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:11:59.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:59.403+0000 7f2b4c000640 1 -- 172.21.15.16:0/1478826416 shutdown_connections 2024-03-31T15:11:59.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:59.403+0000 7f2b4c000640 1 -- 172.21.15.16:0/1478826416 wait complete. 2024-03-31T15:11:59.415 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-31T15:11:59.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:59.579+0000 7f40a3800640 1 Processor -- start 2024-03-31T15:11:59.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:59.579+0000 7f40a3800640 1 -- start start 2024-03-31T15:11:59.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:59.583+0000 7f40a3800640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f409c15e6e0 0x7f409c0b4f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:59.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:59.583+0000 7f40a3800640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f409c15eac0 0x7f409c1513c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:59.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:59.583+0000 7f40a3800640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f409c0b34d0 con 0x7f409c151900 2024-03-31T15:11:59.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:59.583+0000 7f40a3800640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f409c15f300 con 0x7f409c15eac0 2024-03-31T15:11:59.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:59.583+0000 7f40a3800640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f409c0b40e0 con 0x7f409c15e6e0 2024-03-31T15:11:59.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:59.583+0000 7f409be00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f409c15eac0 0x7f409c1513c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:59.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:59.583+0000 7f40a0e00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f409c15e6e0 0x7f409c0b4f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:59.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:59.583+0000 7f409be00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f409c15eac0 0x7f409c1513c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48878/0 (socket says 172.21.15.16:48878) 2024-03-31T15:11:59.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:59.583+0000 7f409be00640 1 -- 172.21.15.16:0/3511602112 learned_addr learned my addr 172.21.15.16:0/3511602112 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:11:59.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:59.583+0000 7f409be00640 1 -- 172.21.15.16:0/3511602112 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f409c15e6e0 msgr2=0x7f409c0b4f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:11:59.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:59.583+0000 7f409be00640 1 --2- 172.21.15.16:0/3511602112 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f409c15e6e0 0x7f409c0b4f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:59.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:59.583+0000 7f409be00640 1 -- 172.21.15.16:0/3511602112 >> v1:172.21.15.89:6789/0 conn(0x7f409c151900 legacy=0x7f409c0cf2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:11:59.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:59.583+0000 7f409be00640 1 -- 172.21.15.16:0/3511602112 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f409c0b37b0 con 0x7f409c15eac0 2024-03-31T15:11:59.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:59.587+0000 7f409be00640 1 --2- 172.21.15.16:0/3511602112 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f409c15eac0 0x7f409c1513c0 secure :-1 s=READY pgs=1315 cs=0 l=1 rev1=1 crypto rx=0x7f4088009870 tx=0x7f408801c560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a99de375d30b9ba3 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:11:59.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:59.587+0000 7f409b400640 1 -- 172.21.15.16:0/3511602112 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40880b1070 con 0x7f409c15eac0 2024-03-31T15:11:59.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:59.587+0000 7f409b400640 1 -- 172.21.15.16:0/3511602112 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f40880ac630 con 0x7f409c15eac0 2024-03-31T15:11:59.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:59.587+0000 7f409b400640 1 -- 172.21.15.16:0/3511602112 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40880a8050 con 0x7f409c15eac0 2024-03-31T15:11:59.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:59.587+0000 7f40a3800640 1 -- 172.21.15.16:0/3511602112 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f409c15eac0 msgr2=0x7f409c1513c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:11:59.580 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:59.587+0000 7f40a3800640 1 --2- 172.21.15.16:0/3511602112 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f409c15eac0 0x7f409c1513c0 secure :-1 s=READY pgs=1315 cs=0 l=1 rev1=1 crypto rx=0x7f4088009870 tx=0x7f408801c560 comp rx=0 tx=0).stop 2024-03-31T15:11:59.580 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:59.587+0000 7f40a3800640 1 -- 172.21.15.16:0/3511602112 shutdown_connections 2024-03-31T15:11:59.580 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:59.587+0000 7f40a3800640 1 --2- 172.21.15.16:0/3511602112 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f409c15eac0 0x7f409c1513c0 unknown :-1 s=CLOSED pgs=1315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:59.580 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:59.587+0000 7f40a3800640 1 --2- 172.21.15.16:0/3511602112 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f409c15e6e0 0x7f409c0b4f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:59.580 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:59.587+0000 7f40a3800640 1 -- 172.21.15.16:0/3511602112 >> 172.21.15.16:0/3511602112 conn(0x7f409c06ee40 msgr2=0x7f409c06b620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:11:59.580 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:59.587+0000 7f40a3800640 1 -- 172.21.15.16:0/3511602112 shutdown_connections 2024-03-31T15:11:59.580 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:59.587+0000 7f40a3800640 1 -- 172.21.15.16:0/3511602112 wait complete. 2024-03-31T15:11:59.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:59.587+0000 7f40a3800640 1 Processor -- start 2024-03-31T15:11:59.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:59.587+0000 7f40a3800640 1 -- start start 2024-03-31T15:11:59.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:59.591+0000 7f40a3800640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f409c151900 0x7f409c154c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:59.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:59.591+0000 7f40a3800640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f409c15e6e0 0x7f409c13b810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:59.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:59.591+0000 7f40a3800640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f409c15eac0 0x7f409c13bd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:59.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:59.591+0000 7f40a3800640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f409c0b32b0 con 0x7f409c15e6e0 2024-03-31T15:11:59.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:59.591+0000 7f40a3800640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f409c0b3130 con 0x7f409c151900 2024-03-31T15:11:59.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:59.591+0000 7f40a3800640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f409c0b5450 con 0x7f409c15eac0 2024-03-31T15:11:59.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:59.591+0000 7f409be00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f409c15e6e0 0x7f409c13b810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:59.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:59.591+0000 7f409be00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f409c15e6e0 0x7f409c13b810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48890/0 (socket says 172.21.15.16:48890) 2024-03-31T15:11:59.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:59.591+0000 7f409be00640 1 -- 172.21.15.16:0/2129666393 learned_addr learned my addr 172.21.15.16:0/2129666393 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:11:59.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:59.591+0000 7f40a1800640 1 --2- 172.21.15.16:0/2129666393 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f409c15eac0 0x7f409c13bd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:59.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:59.591+0000 7f40a0e00640 1 --2- 172.21.15.16:0/2129666393 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f409c151900 0x7f409c154c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:59.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:59.591+0000 7f409be00640 1 -- 172.21.15.16:0/2129666393 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f409c15eac0 msgr2=0x7f409c13bd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:11:59.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:59.591+0000 7f409be00640 1 --2- 172.21.15.16:0/2129666393 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f409c15eac0 0x7f409c13bd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:59.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:59.591+0000 7f409be00640 1 -- 172.21.15.16:0/2129666393 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f409c151900 msgr2=0x7f409c154c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:11:59.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:59.591+0000 7f409be00640 1 --2- 172.21.15.16:0/2129666393 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f409c151900 0x7f409c154c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:59.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:59.591+0000 7f409be00640 1 -- 172.21.15.16:0/2129666393 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f409c14b8a0 con 0x7f409c15e6e0 2024-03-31T15:11:59.585 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:59.591+0000 7f409be00640 1 --2- 172.21.15.16:0/2129666393 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f409c15e6e0 0x7f409c13b810 secure :-1 s=READY pgs=1316 cs=0 l=1 rev1=1 crypto rx=0x7f40880af040 tx=0x7f4088019300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:11:59.585 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:59.591+0000 7f4098c00640 1 -- 172.21.15.16:0/2129666393 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40880bc070 con 0x7f409c15e6e0 2024-03-31T15:11:59.585 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:59.591+0000 7f4098c00640 1 -- 172.21.15.16:0/2129666393 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f40880ab5e0 con 0x7f409c15e6e0 2024-03-31T15:11:59.585 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:59.591+0000 7f4098c00640 1 -- 172.21.15.16:0/2129666393 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40880b1070 con 0x7f409c15e6e0 2024-03-31T15:11:59.585 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:59.591+0000 7f40a3800640 1 -- 172.21.15.16:0/2129666393 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f409c14bb80 con 0x7f409c15e6e0 2024-03-31T15:11:59.585 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:59.591+0000 7f40a3800640 1 -- 172.21.15.16:0/2129666393 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f409c149b40 con 0x7f409c15e6e0 2024-03-31T15:11:59.585 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:59.591+0000 7f40a3800640 1 -- 172.21.15.16:0/2129666393 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f409c139bf0 con 0x7f409c15e6e0 2024-03-31T15:11:59.585 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:59.591+0000 7f4098c00640 1 -- 172.21.15.16:0/2129666393 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f40880bf070 con 0x7f409c15e6e0 2024-03-31T15:11:59.586 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:59.595+0000 7f4098c00640 1 --2- 172.21.15.16:0/2129666393 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f407c0683b0 0x7f407c06a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:59.586 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:59.595+0000 7f4098c00640 1 -- 172.21.15.16:0/2129666393 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 5910+0+0 (secure 0 0 0) 0x7f4088120360 con 0x7f409c15e6e0 2024-03-31T15:11:59.586 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:59.595+0000 7f40a0e00640 1 --2- 172.21.15.16:0/2129666393 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f407c0683b0 0x7f407c06a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:59.587 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:59.595+0000 7f40a0e00640 1 --2- 172.21.15.16:0/2129666393 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f407c0683b0 0x7f407c06a870 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f4090009b60 tx=0x7f4090009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:11:59.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:59.603+0000 7f4098c00640 1 -- 172.21.15.16:0/2129666393 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f4088016150 con 0x7f409c15e6e0 2024-03-31T15:11:59.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:59.767+0000 7f40a3800640 1 -- 172.21.15.16:0/2129666393 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f409c069950 con 0x7f409c15e6e0 2024-03-31T15:11:59.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:59.767+0000 7f4098c00640 1 -- 172.21.15.16:0/2129666393 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v85) ==== 104+0+11 (secure 0 0 0) 0x7f40880f01f0 con 0x7f409c15e6e0 2024-03-31T15:11:59.762 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2024-03-31T15:11:59.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:59.771+0000 7f40a3800640 1 -- 172.21.15.16:0/2129666393 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f407c0683b0 msgr2=0x7f407c06a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:11:59.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:59.771+0000 7f40a3800640 1 --2- 172.21.15.16:0/2129666393 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f407c0683b0 0x7f407c06a870 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f4090009b60 tx=0x7f4090009190 comp rx=0 tx=0).stop 2024-03-31T15:11:59.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:59.771+0000 7f40a3800640 1 -- 172.21.15.16:0/2129666393 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f409c15e6e0 msgr2=0x7f409c13b810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:11:59.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:59.771+0000 7f40a3800640 1 --2- 172.21.15.16:0/2129666393 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f409c15e6e0 0x7f409c13b810 secure :-1 s=READY pgs=1316 cs=0 l=1 rev1=1 crypto rx=0x7f40880af040 tx=0x7f4088019300 comp rx=0 tx=0).stop 2024-03-31T15:11:59.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:59.771+0000 7f40a3800640 1 -- 172.21.15.16:0/2129666393 shutdown_connections 2024-03-31T15:11:59.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:59.771+0000 7f40a3800640 1 --2- 172.21.15.16:0/2129666393 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f407c0683b0 0x7f407c06a870 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:59.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:59.771+0000 7f40a3800640 1 --2- 172.21.15.16:0/2129666393 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f409c15eac0 0x7f409c13bd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:59.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:59.771+0000 7f40a3800640 1 --2- 172.21.15.16:0/2129666393 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f409c15e6e0 0x7f409c13b810 unknown :-1 s=CLOSED pgs=1316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:59.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:59.771+0000 7f40a3800640 1 --2- 172.21.15.16:0/2129666393 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f409c151900 0x7f409c154c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:59.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:59.771+0000 7f40a3800640 1 -- 172.21.15.16:0/2129666393 >> 172.21.15.16:0/2129666393 conn(0x7f409c06ee40 msgr2=0x7f409c15bfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:11:59.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:59.771+0000 7f40a3800640 1 -- 172.21.15.16:0/2129666393 shutdown_connections 2024-03-31T15:11:59.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:59.771+0000 7f40a3800640 1 -- 172.21.15.16:0/2129666393 wait complete. 2024-03-31T15:11:59.784 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-31T15:11:59.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:59.935+0000 7f7ad3400640 1 Processor -- start 2024-03-31T15:11:59.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:59.935+0000 7f7ad3400640 1 -- start start 2024-03-31T15:11:59.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:59.935+0000 7f7ad3400640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f7acc06f920 0x7f7acc06fd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:59.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:59.935+0000 7f7ad3400640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7acc070240 con 0x7f7acc0d0a60 2024-03-31T15:11:59.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:59.935+0000 7f7ad3400640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7acc15ca70 con 0x7f7acc0cf150 2024-03-31T15:11:59.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:59.935+0000 7f7ad3400640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f7acc13f670 con 0x7f7acc06f920 2024-03-31T15:11:59.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:59.935+0000 7f7ad0a00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f7acc06f920 0x7f7acc06fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:59.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:59.935+0000 7f7ad0a00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f7acc06f920 0x7f7acc06fd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48900/0 (socket says 172.21.15.16:48900) 2024-03-31T15:11:59.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:59.935+0000 7f7ad0a00640 1 -- 172.21.15.16:0/4011266549 learned_addr learned my addr 172.21.15.16:0/4011266549 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:11:59.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:59.935+0000 7f7ad0a00640 1 -- 172.21.15.16:0/4011266549 >> v1:172.21.15.89:6789/0 conn(0x7f7acc0cf150 legacy=0x7f7acc1419c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:11:59.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:59.935+0000 7f7ad0a00640 1 -- 172.21.15.16:0/4011266549 >> v1:172.21.15.16:6789/0 conn(0x7f7acc0d0a60 legacy=0x7f7acc0cea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:11:59.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:59.935+0000 7f7ad0a00640 1 -- 172.21.15.16:0/4011266549 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7acc15cd50 con 0x7f7acc06f920 2024-03-31T15:11:59.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:59.939+0000 7f7ad0a00640 1 --2- 172.21.15.16:0/4011266549 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7acc06f920 0x7f7acc06fd00 secure :-1 s=READY pgs=1318 cs=0 l=1 rev1=1 crypto rx=0x7f7abc009870 tx=0x7f7abc01c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb917b20aa124821 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:11:59.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:59.939+0000 7f7acbe00640 1 -- 172.21.15.16:0/4011266549 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7abc0b0070 con 0x7f7acc06f920 2024-03-31T15:11:59.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:59.939+0000 7f7acbe00640 1 -- 172.21.15.16:0/4011266549 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7abc0ab600 con 0x7f7acc06f920 2024-03-31T15:11:59.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:59.939+0000 7f7acbe00640 1 -- 172.21.15.16:0/4011266549 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7abc0ab9d0 con 0x7f7acc06f920 2024-03-31T15:11:59.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:59.939+0000 7f7ad3400640 1 -- 172.21.15.16:0/4011266549 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7acc06f920 msgr2=0x7f7acc06fd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:11:59.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:59.939+0000 7f7ad3400640 1 --2- 172.21.15.16:0/4011266549 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7acc06f920 0x7f7acc06fd00 secure :-1 s=READY pgs=1318 cs=0 l=1 rev1=1 crypto rx=0x7f7abc009870 tx=0x7f7abc01c490 comp rx=0 tx=0).stop 2024-03-31T15:11:59.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:59.939+0000 7f7ad3400640 1 -- 172.21.15.16:0/4011266549 shutdown_connections 2024-03-31T15:11:59.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:59.939+0000 7f7ad3400640 1 --2- 172.21.15.16:0/4011266549 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7acc06f920 0x7f7acc06fd00 unknown :-1 s=CLOSED pgs=1318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:59.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:59.939+0000 7f7ad3400640 1 -- 172.21.15.16:0/4011266549 >> 172.21.15.16:0/4011266549 conn(0x7f7acc067930 msgr2=0x7f7acc0b3ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:11:59.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:59.939+0000 7f7ad3400640 1 -- 172.21.15.16:0/4011266549 shutdown_connections 2024-03-31T15:11:59.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:59.939+0000 7f7ad3400640 1 -- 172.21.15.16:0/4011266549 wait complete. 2024-03-31T15:11:59.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:59.943+0000 7f7ad3400640 1 Processor -- start 2024-03-31T15:11:59.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:59.943+0000 7f7ad3400640 1 -- start start 2024-03-31T15:11:59.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:59.943+0000 7f7ad3400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7acc06f920 0x7f7acc06cae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:59.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:59.943+0000 7f7ad3400640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f7acc0cf150 0x7f7acc0b5300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:59.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:59.943+0000 7f7ad3400640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7acc0d0a60 0x7f7acc0b6b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:59.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:59.943+0000 7f7ad3400640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f7acc16a240 con 0x7f7acc06f920 2024-03-31T15:11:59.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:59.943+0000 7f7ad3400640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f7acc16a0c0 con 0x7f7acc0d0a60 2024-03-31T15:11:59.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:59.943+0000 7f7ad3400640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f7acc16a3c0 con 0x7f7acc0cf150 2024-03-31T15:11:59.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:59.943+0000 7f7ad0a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7acc06f920 0x7f7acc06cae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:59.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:59.943+0000 7f7ad0a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7acc06f920 0x7f7acc06cae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48910/0 (socket says 172.21.15.16:48910) 2024-03-31T15:11:59.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:59.943+0000 7f7ad0a00640 1 -- 172.21.15.16:0/2813847668 learned_addr learned my addr 172.21.15.16:0/2813847668 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:11:59.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:59.943+0000 7f7ad1400640 1 --2- 172.21.15.16:0/2813847668 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7acc0d0a60 0x7f7acc0b6b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:59.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:59.943+0000 7f7ac3e00640 1 --2- 172.21.15.16:0/2813847668 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f7acc0cf150 0x7f7acc0b5300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:59.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:59.943+0000 7f7ad0a00640 1 -- 172.21.15.16:0/2813847668 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f7acc0cf150 msgr2=0x7f7acc0b5300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:11:59.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:59.943+0000 7f7ad0a00640 1 --2- 172.21.15.16:0/2813847668 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f7acc0cf150 0x7f7acc0b5300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:59.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:59.943+0000 7f7ad0a00640 1 -- 172.21.15.16:0/2813847668 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7acc0d0a60 msgr2=0x7f7acc0b6b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:11:59.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:59.943+0000 7f7ad0a00640 1 --2- 172.21.15.16:0/2813847668 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7acc0d0a60 0x7f7acc0b6b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:11:59.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:59.943+0000 7f7ad0a00640 1 -- 172.21.15.16:0/2813847668 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7acc0b5a70 con 0x7f7acc06f920 2024-03-31T15:11:59.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:59.943+0000 7f7ad1400640 1 --2- 172.21.15.16:0/2813847668 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7acc0d0a60 0x7f7acc0b6b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:11:59.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:59.943+0000 7f7ad0a00640 1 --2- 172.21.15.16:0/2813847668 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7acc06f920 0x7f7acc06cae0 secure :-1 s=READY pgs=1319 cs=0 l=1 rev1=1 crypto rx=0x7f7abc0ae040 tx=0x7f7abc01acd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:11:59.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:59.943+0000 7f7ac9600640 1 -- 172.21.15.16:0/2813847668 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7abc0ba020 con 0x7f7acc06f920 2024-03-31T15:11:59.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:59.943+0000 7f7ac9600640 1 -- 172.21.15.16:0/2813847668 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7abc0b6640 con 0x7f7acc06f920 2024-03-31T15:11:59.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:59.943+0000 7f7ac9600640 1 -- 172.21.15.16:0/2813847668 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7abc0b0040 con 0x7f7acc06f920 2024-03-31T15:11:59.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:59.943+0000 7f7ad3400640 1 -- 172.21.15.16:0/2813847668 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7acc0b5d50 con 0x7f7acc06f920 2024-03-31T15:11:59.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:59.943+0000 7f7ad3400640 1 -- 172.21.15.16:0/2813847668 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7acc0b6310 con 0x7f7acc06f920 2024-03-31T15:11:59.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:59.947+0000 7f7ad3400640 1 -- 172.21.15.16:0/2813847668 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7acc069950 con 0x7f7acc06f920 2024-03-31T15:11:59.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:59.947+0000 7f7ac9600640 1 -- 172.21.15.16:0/2813847668 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f7abc0bd020 con 0x7f7acc06f920 2024-03-31T15:11:59.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:59.947+0000 7f7ac9600640 1 --2- 172.21.15.16:0/2813847668 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f7aa80683b0 0x7f7aa806a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:11:59.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:59.947+0000 7f7ac9600640 1 -- 172.21.15.16:0/2813847668 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 5910+0+0 (secure 0 0 0) 0x7f7abc120ca0 con 0x7f7acc06f920 2024-03-31T15:11:59.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:59.947+0000 7f7ac3e00640 1 --2- 172.21.15.16:0/2813847668 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f7aa80683b0 0x7f7aa806a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:11:59.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:59.947+0000 7f7ac3e00640 1 --2- 172.21.15.16:0/2813847668 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f7aa80683b0 0x7f7aa806a870 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f7acc1fad70 tx=0x7f7ab4009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:11:59.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:11:59.951+0000 7f7ac9600640 1 -- 172.21.15.16:0/2813847668 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f7abc0c4030 con 0x7f7acc06f920 2024-03-31T15:12:00.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:00.135+0000 7f7ad3400640 1 -- 172.21.15.16:0/2813847668 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f7acc0b50e0 con 0x7f7acc06f920 2024-03-31T15:12:00.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:00.139+0000 7f7ac9600640 1 -- 172.21.15.16:0/2813847668 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v85) ==== 100+0+11 (secure 0 0 0) 0x7f7abc016110 con 0x7f7acc06f920 2024-03-31T15:12:00.131 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 64 2024-03-31T15:12:00.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:00.139+0000 7f7ad3400640 1 -- 172.21.15.16:0/2813847668 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f7aa80683b0 msgr2=0x7f7aa806a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:00.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:00.139+0000 7f7ad3400640 1 --2- 172.21.15.16:0/2813847668 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f7aa80683b0 0x7f7aa806a870 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f7acc1fad70 tx=0x7f7ab4009040 comp rx=0 tx=0).stop 2024-03-31T15:12:00.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:00.139+0000 7f7ad3400640 1 -- 172.21.15.16:0/2813847668 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7acc06f920 msgr2=0x7f7acc06cae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:00.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:00.139+0000 7f7ad3400640 1 --2- 172.21.15.16:0/2813847668 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7acc06f920 0x7f7acc06cae0 secure :-1 s=READY pgs=1319 cs=0 l=1 rev1=1 crypto rx=0x7f7abc0ae040 tx=0x7f7abc01acd0 comp rx=0 tx=0).stop 2024-03-31T15:12:00.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:00.139+0000 7f7ad3400640 1 -- 172.21.15.16:0/2813847668 shutdown_connections 2024-03-31T15:12:00.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:00.139+0000 7f7ad3400640 1 --2- 172.21.15.16:0/2813847668 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f7aa80683b0 0x7f7aa806a870 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:00.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:00.139+0000 7f7ad3400640 1 --2- 172.21.15.16:0/2813847668 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7acc0d0a60 0x7f7acc0b6b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:00.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:00.139+0000 7f7ad3400640 1 --2- 172.21.15.16:0/2813847668 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f7acc0cf150 0x7f7acc0b5300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:00.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:00.139+0000 7f7ad3400640 1 --2- 172.21.15.16:0/2813847668 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7acc06f920 0x7f7acc06cae0 unknown :-1 s=CLOSED pgs=1319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:00.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:00.139+0000 7f7ad3400640 1 -- 172.21.15.16:0/2813847668 >> 172.21.15.16:0/2813847668 conn(0x7f7acc067930 msgr2=0x7f7acc155b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:12:00.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:00.139+0000 7f7ad3400640 1 -- 172.21.15.16:0/2813847668 shutdown_connections 2024-03-31T15:12:00.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:00.139+0000 7f7ad3400640 1 -- 172.21.15.16:0/2813847668 wait complete. 2024-03-31T15:12:00.152 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-31T15:12:00.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:00.303+0000 7fed45800640 1 Processor -- start 2024-03-31T15:12:00.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:00.303+0000 7fed45800640 1 -- start start 2024-03-31T15:12:00.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:00.303+0000 7fed45800640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fed4009b170 0x7fed400ccb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:00.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:00.303+0000 7fed45800640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fed40070d50 0x7fed400cd0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:00.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:00.303+0000 7fed45800640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fed400d0800 con 0x7fed4015e3b0 2024-03-31T15:12:00.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:00.303+0000 7fed45800640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7fed4015e790 con 0x7fed40070d50 2024-03-31T15:12:00.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:00.303+0000 7fed45800640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7fed4006f5d0 con 0x7fed4009b170 2024-03-31T15:12:00.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:00.303+0000 7fed44e00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fed40070d50 0x7fed400cd0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:00.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:00.303+0000 7fed3fe00640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fed4015e3b0 0x7fed4009aa60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:35080/0 (socket says 172.21.15.16:35080) 2024-03-31T15:12:00.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:00.303+0000 7fed3fe00640 1 -- 172.21.15.16:0/2752262796 learned_addr learned my addr 172.21.15.16:0/2752262796 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:12:00.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:00.303+0000 7fed3f400640 1 --2- 172.21.15.16:0/2752262796 >> v2:172.21.15.89:3300/0 conn(0x7fed4009b170 0x7fed400ccb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:00.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:00.303+0000 7fed44e00640 1 -- 172.21.15.16:0/2752262796 >> v2:172.21.15.89:3300/0 conn(0x7fed4009b170 msgr2=0x7fed400ccb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:12:00.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:00.303+0000 7fed44e00640 1 --2- 172.21.15.16:0/2752262796 >> v2:172.21.15.89:3300/0 conn(0x7fed4009b170 0x7fed400ccb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:00.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:00.303+0000 7fed44e00640 1 -- 172.21.15.16:0/2752262796 >> v1:172.21.15.16:6789/0 conn(0x7fed4015e3b0 legacy=0x7fed4009aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:00.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:00.303+0000 7fed44e00640 1 -- 172.21.15.16:0/2752262796 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed400d0ae0 con 0x7fed40070d50 2024-03-31T15:12:00.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:00.303+0000 7fed3ea00640 1 -- 172.21.15.16:0/2752262796 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1165908013 0 0) 0x7fed400d0800 con 0x7fed4015e3b0 2024-03-31T15:12:00.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:00.307+0000 7fed44e00640 1 --2- 172.21.15.16:0/2752262796 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fed40070d50 0x7fed400cd0d0 secure :-1 s=READY pgs=1321 cs=0 l=1 rev1=1 crypto rx=0x7fed40072ae0 tx=0x7fed3801c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f34c1bb94633bbd2 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:12:00.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:00.307+0000 7fed3ea00640 1 -- 172.21.15.16:0/2752262796 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed380b2070 con 0x7fed40070d50 2024-03-31T15:12:00.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:00.307+0000 7fed3ea00640 1 -- 172.21.15.16:0/2752262796 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fed380a59c0 con 0x7fed40070d50 2024-03-31T15:12:00.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:00.307+0000 7fed3ea00640 1 -- 172.21.15.16:0/2752262796 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed380a5d90 con 0x7fed40070d50 2024-03-31T15:12:00.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:00.307+0000 7fed45800640 1 -- 172.21.15.16:0/2752262796 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fed40070d50 msgr2=0x7fed400cd0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:00.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:00.307+0000 7fed45800640 1 --2- 172.21.15.16:0/2752262796 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fed40070d50 0x7fed400cd0d0 secure :-1 s=READY pgs=1321 cs=0 l=1 rev1=1 crypto rx=0x7fed40072ae0 tx=0x7fed3801c900 comp rx=0 tx=0).stop 2024-03-31T15:12:00.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:00.307+0000 7fed45800640 1 -- 172.21.15.16:0/2752262796 shutdown_connections 2024-03-31T15:12:00.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:00.307+0000 7fed45800640 1 --2- 172.21.15.16:0/2752262796 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fed40070d50 0x7fed400cd0d0 unknown :-1 s=CLOSED pgs=1321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:00.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:00.307+0000 7fed45800640 1 --2- 172.21.15.16:0/2752262796 >> v2:172.21.15.89:3300/0 conn(0x7fed4009b170 0x7fed400ccb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:00.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:00.307+0000 7fed45800640 1 -- 172.21.15.16:0/2752262796 >> 172.21.15.16:0/2752262796 conn(0x7fed400b39c0 msgr2=0x7fed400b3dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:12:00.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:00.307+0000 7fed45800640 1 -- 172.21.15.16:0/2752262796 shutdown_connections 2024-03-31T15:12:00.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:00.307+0000 7fed45800640 1 -- 172.21.15.16:0/2752262796 wait complete. 2024-03-31T15:12:00.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:00.311+0000 7fed45800640 1 Processor -- start 2024-03-31T15:12:00.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:00.311+0000 7fed45800640 1 -- start start 2024-03-31T15:12:00.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:00.311+0000 7fed45800640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fed40070d50 0x7fed40152920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:00.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:00.311+0000 7fed45800640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fed4009b170 0x7fed4006d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:00.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:00.311+0000 7fed45800640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fed4015e3b0 0x7fed4006dc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:00.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:00.311+0000 7fed45800640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fed4016b010 con 0x7fed4015e3b0 2024-03-31T15:12:00.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:00.311+0000 7fed45800640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fed4016ae90 con 0x7fed40070d50 2024-03-31T15:12:00.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:00.311+0000 7fed45800640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fed4016b190 con 0x7fed4009b170 2024-03-31T15:12:00.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:00.311+0000 7fed44e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fed4015e3b0 0x7fed4006dc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:00.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:00.311+0000 7fed44e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fed4015e3b0 0x7fed4006dc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48934/0 (socket says 172.21.15.16:48934) 2024-03-31T15:12:00.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:00.311+0000 7fed44e00640 1 -- 172.21.15.16:0/865227988 learned_addr learned my addr 172.21.15.16:0/865227988 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:12:00.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:00.311+0000 7fed3f400640 1 --2- 172.21.15.16:0/865227988 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fed4009b170 0x7fed4006d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:00.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:00.311+0000 7fed3fe00640 1 --2- 172.21.15.16:0/865227988 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fed40070d50 0x7fed40152920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:00.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:00.311+0000 7fed44e00640 1 -- 172.21.15.16:0/865227988 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fed4009b170 msgr2=0x7fed4006d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:00.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:00.311+0000 7fed44e00640 1 --2- 172.21.15.16:0/865227988 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fed4009b170 0x7fed4006d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:00.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:00.311+0000 7fed44e00640 1 -- 172.21.15.16:0/865227988 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fed40070d50 msgr2=0x7fed40152920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:00.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:00.311+0000 7fed44e00640 1 --2- 172.21.15.16:0/865227988 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fed40070d50 0x7fed40152920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:00.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:00.311+0000 7fed44e00640 1 -- 172.21.15.16:0/865227988 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed40152e60 con 0x7fed4015e3b0 2024-03-31T15:12:00.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:00.311+0000 7fed44e00640 1 --2- 172.21.15.16:0/865227988 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fed4015e3b0 0x7fed4006dc30 secure :-1 s=READY pgs=1322 cs=0 l=1 rev1=1 crypto rx=0x7fed38019510 tx=0x7fed380196a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:12:00.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:00.311+0000 7fed37e00640 1 -- 172.21.15.16:0/865227988 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed380b2050 con 0x7fed4015e3b0 2024-03-31T15:12:00.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:00.311+0000 7fed37e00640 1 -- 172.21.15.16:0/865227988 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fed380be070 con 0x7fed4015e3b0 2024-03-31T15:12:00.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:00.311+0000 7fed37e00640 1 -- 172.21.15.16:0/865227988 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed380ada70 con 0x7fed4015e3b0 2024-03-31T15:12:00.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:00.311+0000 7fed45800640 1 -- 172.21.15.16:0/865227988 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fed40153d60 con 0x7fed4015e3b0 2024-03-31T15:12:00.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:00.311+0000 7fed45800640 1 -- 172.21.15.16:0/865227988 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fed40154320 con 0x7fed4015e3b0 2024-03-31T15:12:00.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:00.315+0000 7fed45800640 1 -- 172.21.15.16:0/865227988 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fed4004f8c0 con 0x7fed4015e3b0 2024-03-31T15:12:00.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:00.315+0000 7fed37e00640 1 -- 172.21.15.16:0/865227988 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fed380adc10 con 0x7fed4015e3b0 2024-03-31T15:12:00.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:00.315+0000 7fed37e00640 1 --2- 172.21.15.16:0/865227988 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fed20068380 0x7fed2006a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:00.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:00.315+0000 7fed37e00640 1 -- 172.21.15.16:0/865227988 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 5910+0+0 (secure 0 0 0) 0x7fed38122730 con 0x7fed4015e3b0 2024-03-31T15:12:00.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:00.315+0000 7fed3fe00640 1 --2- 172.21.15.16:0/865227988 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fed20068380 0x7fed2006a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:00.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:00.319+0000 7fed3fe00640 1 --2- 172.21.15.16:0/865227988 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fed20068380 0x7fed2006a840 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7fed30007fd0 tx=0x7fed30007d60 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:12:00.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:00.319+0000 7fed37e00640 1 -- 172.21.15.16:0/865227988 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fed38016520 con 0x7fed4015e3b0 2024-03-31T15:12:00.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:00.495+0000 7fed45800640 1 -- 172.21.15.16:0/865227988 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fed40068ae0 con 0x7fed4015e3b0 2024-03-31T15:12:00.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:00.495+0000 7fed37e00640 1 -- 172.21.15.16:0/865227988 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 4 entries v85) ==== 80+0+242 (secure 0 0 0) 0x7fed380f25b0 con 0x7fed4015e3b0 2024-03-31T15:12:00.490 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:6832/818074038 2024-04-01T15:11:19.402687+0000 2024-03-31T15:12:00.490 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:6834/818074038 2024-04-01T15:11:19.402687+0000 2024-03-31T15:12:00.490 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:6834/2478530200 2024-04-01T15:10:04.360772+0000 2024-03-31T15:12:00.490 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:6832/2478530200 2024-04-01T15:10:04.360772+0000 2024-03-31T15:12:00.491 INFO:teuthology.orchestra.run.smithi016.stderr:listed 4 entries 2024-03-31T15:12:00.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:00.499+0000 7fed45800640 1 -- 172.21.15.16:0/865227988 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fed20068380 msgr2=0x7fed2006a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:00.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:00.499+0000 7fed45800640 1 --2- 172.21.15.16:0/865227988 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fed20068380 0x7fed2006a840 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7fed30007fd0 tx=0x7fed30007d60 comp rx=0 tx=0).stop 2024-03-31T15:12:00.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:00.499+0000 7fed45800640 1 -- 172.21.15.16:0/865227988 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fed4015e3b0 msgr2=0x7fed4006dc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:00.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:00.499+0000 7fed45800640 1 --2- 172.21.15.16:0/865227988 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fed4015e3b0 0x7fed4006dc30 secure :-1 s=READY pgs=1322 cs=0 l=1 rev1=1 crypto rx=0x7fed38019510 tx=0x7fed380196a0 comp rx=0 tx=0).stop 2024-03-31T15:12:00.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:00.499+0000 7fed45800640 1 -- 172.21.15.16:0/865227988 shutdown_connections 2024-03-31T15:12:00.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:00.499+0000 7fed45800640 1 --2- 172.21.15.16:0/865227988 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fed20068380 0x7fed2006a840 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:00.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:00.499+0000 7fed45800640 1 --2- 172.21.15.16:0/865227988 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fed4015e3b0 0x7fed4006dc30 unknown :-1 s=CLOSED pgs=1322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:00.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:00.499+0000 7fed45800640 1 --2- 172.21.15.16:0/865227988 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fed4009b170 0x7fed4006d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:00.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:00.499+0000 7fed45800640 1 --2- 172.21.15.16:0/865227988 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fed40070d50 0x7fed40152920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:00.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:00.499+0000 7fed45800640 1 -- 172.21.15.16:0/865227988 >> 172.21.15.16:0/865227988 conn(0x7fed400b39c0 msgr2=0x7fed4015a280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:12:00.494 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:00.499+0000 7fed45800640 1 -- 172.21.15.16:0/865227988 shutdown_connections 2024-03-31T15:12:00.494 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:00.499+0000 7fed45800640 1 -- 172.21.15.16:0/865227988 wait complete. 2024-03-31T15:12:00.511 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi016.front.sepia.ceph.com... 2024-03-31T15:12:00.511 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:12:00.511 DEBUG:teuthology.orchestra.run.smithi016:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-03-31T15:12:00.530 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi016.stderr:ceph-fuse[46616]: fuse finished with error 0 and tester_r 0 2024-03-31T15:12:00.533 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi016.front.sepia.ceph.com 2024-03-31T15:12:00.533 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:12:00.534 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-31T15:12:00.609 INFO:tasks.cephfs.mount:Terminating background process 2024-03-31T15:12:00.610 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-03-31T15:12:00.610 DEBUG:teuthology.orchestra.run.smithi089:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-31T15:12:00.619 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16354452k,nr_inodes=4088613,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275344k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=901 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275340k,nr_inodes=818835,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275344k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-31T15:12:00.619 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi089.front.sepia.ceph.com 2024-03-31T15:12:00.619 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:12:00.619 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-03-31T15:12:00.670 INFO:teuthology.orchestra.run.smithi089.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.1': No such file or directory 2024-03-31T15:12:00.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-31T15:12:00.672 INFO:tasks.cephfs.mount:Terminating background process 2024-03-31T15:12:00.672 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-03-31T15:12:00.672 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-31T15:12:00.683 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16357708k,nr_inodes=4089427,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275996k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=6539 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275992k,nr_inodes=818998,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275996k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-31T15:12:00.683 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi016.front.sepia.ceph.com 2024-03-31T15:12:00.683 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:12:00.683 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-03-31T15:12:00.733 INFO:teuthology.orchestra.run.smithi016.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2024-03-31T15:12:00.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-31T15:12:00.735 INFO:tasks.cephfs.mount:Terminating background process 2024-03-31T15:12:00.735 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-03-31T15:12:00.735 DEBUG:teuthology.orchestra.run.smithi089:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-31T15:12:00.744 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16354452k,nr_inodes=4088613,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275344k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=901 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275340k,nr_inodes=818835,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275344k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-31T15:12:00.744 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi089.front.sepia.ceph.com 2024-03-31T15:12:00.744 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:12:00.744 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-03-31T15:12:00.795 INFO:teuthology.orchestra.run.smithi089.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2024-03-31T15:12:00.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-31T15:12:00.796 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-31T15:12:00.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:00.943+0000 7f2de1400640 1 Processor -- start 2024-03-31T15:12:00.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:00.943+0000 7f2de1400640 1 -- start start 2024-03-31T15:12:00.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:00.943+0000 7f2de1400640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f2ddc144fa0 0x7f2ddc0b3cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:00.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:00.943+0000 7f2de1400640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f2ddc0b3030 0x7f2ddc06e6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:00.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:00.943+0000 7f2de1400640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2ddc0b4390 con 0x7f2ddc06ec00 2024-03-31T15:12:00.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:00.943+0000 7f2de1400640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f2ddc06f070 con 0x7f2ddc0b3030 2024-03-31T15:12:00.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:00.943+0000 7f2de1400640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f2ddc0ce240 con 0x7f2ddc144fa0 2024-03-31T15:12:00.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:00.943+0000 7f2dd3e00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f2ddc0b3030 0x7f2ddc06e6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:00.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:00.943+0000 7f2de0a00640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f2ddc06ec00 0x7f2ddc15fe20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:35088/0 (socket says 172.21.15.16:35088) 2024-03-31T15:12:00.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:00.943+0000 7f2dd3e00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f2ddc0b3030 0x7f2ddc06e6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48938/0 (socket says 172.21.15.16:48938) 2024-03-31T15:12:00.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:00.943+0000 7f2de0a00640 1 -- 172.21.15.16:0/957982582 learned_addr learned my addr 172.21.15.16:0/957982582 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:12:00.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:00.943+0000 7f2ddbe00640 1 --2- 172.21.15.16:0/957982582 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f2ddc144fa0 0x7f2ddc0b3cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:00.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:00.943+0000 7f2dd3e00640 1 -- 172.21.15.16:0/957982582 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f2ddc144fa0 msgr2=0x7f2ddc0b3cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:12:00.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:00.943+0000 7f2dd3e00640 1 --2- 172.21.15.16:0/957982582 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f2ddc144fa0 0x7f2ddc0b3cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:00.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:00.943+0000 7f2dd3e00640 1 -- 172.21.15.16:0/957982582 >> v1:172.21.15.16:6789/0 conn(0x7f2ddc06ec00 legacy=0x7f2ddc15fe20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:00.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:00.943+0000 7f2ddb400640 1 -- 172.21.15.16:0/957982582 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3826215077 0 0) 0x7f2ddc0b4390 con 0x7f2ddc06ec00 2024-03-31T15:12:00.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:00.943+0000 7f2dd3e00640 1 -- 172.21.15.16:0/957982582 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ddc0b4670 con 0x7f2ddc0b3030 2024-03-31T15:12:00.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:00.947+0000 7f2dd3e00640 1 --2- 172.21.15.16:0/957982582 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2ddc0b3030 0x7f2ddc06e6c0 secure :-1 s=READY pgs=1324 cs=0 l=1 rev1=1 crypto rx=0x7f2dc40097c0 tx=0x7f2dc401c410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f2d95849eb052ea1 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:12:00.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:00.947+0000 7f2ddb400640 1 -- 172.21.15.16:0/957982582 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2dc40b0070 con 0x7f2ddc0b3030 2024-03-31T15:12:00.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:00.947+0000 7f2ddb400640 1 -- 172.21.15.16:0/957982582 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2dc40ab600 con 0x7f2ddc0b3030 2024-03-31T15:12:00.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:00.947+0000 7f2ddb400640 1 -- 172.21.15.16:0/957982582 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2dc40ab9d0 con 0x7f2ddc0b3030 2024-03-31T15:12:00.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:00.947+0000 7f2de1400640 1 -- 172.21.15.16:0/957982582 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2ddc0b3030 msgr2=0x7f2ddc06e6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:00.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:00.947+0000 7f2de1400640 1 --2- 172.21.15.16:0/957982582 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2ddc0b3030 0x7f2ddc06e6c0 secure :-1 s=READY pgs=1324 cs=0 l=1 rev1=1 crypto rx=0x7f2dc40097c0 tx=0x7f2dc401c410 comp rx=0 tx=0).stop 2024-03-31T15:12:00.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:00.947+0000 7f2de1400640 1 -- 172.21.15.16:0/957982582 shutdown_connections 2024-03-31T15:12:00.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:00.947+0000 7f2de1400640 1 --2- 172.21.15.16:0/957982582 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2ddc0b3030 0x7f2ddc06e6c0 unknown :-1 s=CLOSED pgs=1324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:00.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:00.947+0000 7f2de1400640 1 --2- 172.21.15.16:0/957982582 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f2ddc144fa0 0x7f2ddc0b3cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:00.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:00.947+0000 7f2de1400640 1 -- 172.21.15.16:0/957982582 >> 172.21.15.16:0/957982582 conn(0x7f2ddc0cc9a0 msgr2=0x7f2ddc0ccdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:12:00.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:00.947+0000 7f2de1400640 1 -- 172.21.15.16:0/957982582 shutdown_connections 2024-03-31T15:12:00.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:00.947+0000 7f2de1400640 1 -- 172.21.15.16:0/957982582 wait complete. 2024-03-31T15:12:00.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:00.951+0000 7f2de1400640 1 Processor -- start 2024-03-31T15:12:00.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:00.951+0000 7f2de1400640 1 -- start start 2024-03-31T15:12:00.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:00.951+0000 7f2de1400640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f2ddc06ec00 0x7f2ddc155b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:00.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:00.951+0000 7f2de1400640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2ddc0b3030 0x7f2ddc15d490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:00.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:00.951+0000 7f2de1400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2ddc144fa0 0x7f2ddc140e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:00.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:00.951+0000 7f2de1400640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f2ddc169110 con 0x7f2ddc144fa0 2024-03-31T15:12:00.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:00.951+0000 7f2de1400640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f2ddc168f90 con 0x7f2ddc0b3030 2024-03-31T15:12:00.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:00.951+0000 7f2de1400640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f2ddc169290 con 0x7f2ddc06ec00 2024-03-31T15:12:00.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:00.951+0000 7f2de0a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2ddc144fa0 0x7f2ddc140e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:00.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:00.951+0000 7f2de0a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2ddc144fa0 0x7f2ddc140e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48946/0 (socket says 172.21.15.16:48946) 2024-03-31T15:12:00.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:00.951+0000 7f2de0a00640 1 -- 172.21.15.16:0/3496285473 learned_addr learned my addr 172.21.15.16:0/3496285473 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:12:00.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:00.951+0000 7f2ddbe00640 1 --2- 172.21.15.16:0/3496285473 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f2ddc06ec00 0x7f2ddc155b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:00.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:00.951+0000 7f2dd3e00640 1 --2- 172.21.15.16:0/3496285473 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2ddc0b3030 0x7f2ddc15d490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:00.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:00.951+0000 7f2de0a00640 1 -- 172.21.15.16:0/3496285473 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f2ddc06ec00 msgr2=0x7f2ddc155b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:00.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:00.951+0000 7f2de0a00640 1 --2- 172.21.15.16:0/3496285473 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f2ddc06ec00 0x7f2ddc155b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:00.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:00.951+0000 7f2de0a00640 1 -- 172.21.15.16:0/3496285473 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2ddc0b3030 msgr2=0x7f2ddc15d490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:00.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:00.951+0000 7f2de0a00640 1 --2- 172.21.15.16:0/3496285473 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2ddc0b3030 0x7f2ddc15d490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:00.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:00.951+0000 7f2de0a00640 1 -- 172.21.15.16:0/3496285473 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ddc0b6200 con 0x7f2ddc144fa0 2024-03-31T15:12:00.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:00.951+0000 7f2de0a00640 1 --2- 172.21.15.16:0/3496285473 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2ddc144fa0 0x7f2ddc140e50 secure :-1 s=READY pgs=1325 cs=0 l=1 rev1=1 crypto rx=0x7f2dd400c730 tx=0x7f2dd400cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:12:00.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:00.951+0000 7f2dd8c00640 1 -- 172.21.15.16:0/3496285473 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2dd4018070 con 0x7f2ddc144fa0 2024-03-31T15:12:00.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:00.951+0000 7f2dd8c00640 1 -- 172.21.15.16:0/3496285473 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2dd400ce40 con 0x7f2ddc144fa0 2024-03-31T15:12:00.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:00.951+0000 7f2de1400640 1 -- 172.21.15.16:0/3496285473 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2ddc0b19e0 con 0x7f2ddc144fa0 2024-03-31T15:12:00.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:00.951+0000 7f2dd8c00640 1 -- 172.21.15.16:0/3496285473 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2dd400a860 con 0x7f2ddc144fa0 2024-03-31T15:12:00.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:00.951+0000 7f2de1400640 1 -- 172.21.15.16:0/3496285473 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2ddc15e0e0 con 0x7f2ddc144fa0 2024-03-31T15:12:00.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:00.955+0000 7f2de1400640 1 -- 172.21.15.16:0/3496285473 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2ddc04f8c0 con 0x7f2ddc144fa0 2024-03-31T15:12:00.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:00.955+0000 7f2dd8c00640 1 -- 172.21.15.16:0/3496285473 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f2dd401b070 con 0x7f2ddc144fa0 2024-03-31T15:12:00.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:00.955+0000 7f2dd8c00640 1 --2- 172.21.15.16:0/3496285473 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f2db8068380 0x7f2db806a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:00.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:00.955+0000 7f2dd8c00640 1 -- 172.21.15.16:0/3496285473 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 5910+0+0 (secure 0 0 0) 0x7f2dd4085fb0 con 0x7f2ddc144fa0 2024-03-31T15:12:00.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:00.955+0000 7f2ddbe00640 1 --2- 172.21.15.16:0/3496285473 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f2db8068380 0x7f2db806a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:00.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:00.955+0000 7f2ddbe00640 1 --2- 172.21.15.16:0/3496285473 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f2db8068380 0x7f2db806a840 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f2dcc007a60 tx=0x7f2dcc008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:12:00.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:00.959+0000 7f2dd8c00640 1 -- 172.21.15.16:0/3496285473 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f2dd4055db0 con 0x7f2ddc144fa0 2024-03-31T15:12:01.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:01.155+0000 7f2de1400640 1 -- 172.21.15.16:0/3496285473 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2ddc0cc5e0 con 0x7f2ddc144fa0 2024-03-31T15:12:01.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:01.155+0000 7f2dd8c00640 1 -- 172.21.15.16:0/3496285473 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 74 v74) ==== 94+0+5184 (secure 0 0 0) 0x7f2dd405a0b0 con 0x7f2ddc144fa0 2024-03-31T15:12:01.149 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:12:01.150 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":74,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6494,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/2445558494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6835","nonce":2445558494},{"type":"v1","addr":"172.21.15.89:6837","nonce":2445558494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":56},{"gid":7270,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/3185913858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6833","nonce":3185913858},{"type":"v1","addr":"172.21.15.16:6835","nonce":3185913858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":56},{"gid":7279,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6836/1599468493","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":1599468493},{"type":"v1","addr":"172.21.15.89:6836","nonce":1599468493}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":56}],"filesystems":[{"mdsmap":{"epoch":74,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-31T15:09:50.926196+0000","modified":"2024-03-31T15:11:23.417783+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":85,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8085},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8085":{"gid":8085,"name":"c","rank":0,"incarnation":71,"state":"up:active","state_seq":4,"addr":"172.21.15.16:6834/3780866829","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":3780866829},{"type":"v1","addr":"172.21.15.16:6834","nonce":3780866829}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8085,"qdb_cluster":[8085]},"id":6}]} 2024-03-31T15:12:01.150 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 74 2024-03-31T15:12:01.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:01.159+0000 7f2de1400640 1 -- 172.21.15.16:0/3496285473 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f2db8068380 msgr2=0x7f2db806a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:01.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:01.159+0000 7f2de1400640 1 --2- 172.21.15.16:0/3496285473 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f2db8068380 0x7f2db806a840 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f2dcc007a60 tx=0x7f2dcc008040 comp rx=0 tx=0).stop 2024-03-31T15:12:01.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:01.159+0000 7f2de1400640 1 -- 172.21.15.16:0/3496285473 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2ddc144fa0 msgr2=0x7f2ddc140e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:01.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:01.159+0000 7f2de1400640 1 --2- 172.21.15.16:0/3496285473 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2ddc144fa0 0x7f2ddc140e50 secure :-1 s=READY pgs=1325 cs=0 l=1 rev1=1 crypto rx=0x7f2dd400c730 tx=0x7f2dd400cbf0 comp rx=0 tx=0).stop 2024-03-31T15:12:01.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:01.159+0000 7f2de1400640 1 -- 172.21.15.16:0/3496285473 shutdown_connections 2024-03-31T15:12:01.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:01.159+0000 7f2de1400640 1 --2- 172.21.15.16:0/3496285473 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f2db8068380 0x7f2db806a840 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:01.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:01.159+0000 7f2de1400640 1 --2- 172.21.15.16:0/3496285473 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2ddc144fa0 0x7f2ddc140e50 unknown :-1 s=CLOSED pgs=1325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:01.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:01.159+0000 7f2de1400640 1 --2- 172.21.15.16:0/3496285473 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2ddc0b3030 0x7f2ddc15d490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:01.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:01.159+0000 7f2de1400640 1 --2- 172.21.15.16:0/3496285473 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f2ddc06ec00 0x7f2ddc155b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:01.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:01.159+0000 7f2de1400640 1 -- 172.21.15.16:0/3496285473 >> 172.21.15.16:0/3496285473 conn(0x7f2ddc0cc9a0 msgr2=0x7f2ddc0b36f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:12:01.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:01.159+0000 7f2de1400640 1 -- 172.21.15.16:0/3496285473 shutdown_connections 2024-03-31T15:12:01.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:01.159+0000 7f2de1400640 1 -- 172.21.15.16:0/3496285473 wait complete. 2024-03-31T15:12:01.175 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-31T15:12:01.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:01.323+0000 7f4d95200640 1 Processor -- start 2024-03-31T15:12:01.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:01.323+0000 7f4d95200640 1 -- start start 2024-03-31T15:12:01.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:01.323+0000 7f4d95200640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f4d9006f920 0x7f4d9006fd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:01.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:01.323+0000 7f4d95200640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4d900d0920 con 0x7f4d900d2010 2024-03-31T15:12:01.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:01.323+0000 7f4d95200640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4d900d0c00 con 0x7f4d90145e40 2024-03-31T15:12:01.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:01.323+0000 7f4d95200640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f4d900702c0 con 0x7f4d9006f920 2024-03-31T15:12:01.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:01.327+0000 7f4d8e800640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f4d900d2010 0x7f4d900d24a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:35090/0 (socket says 172.21.15.16:35090) 2024-03-31T15:12:01.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:01.327+0000 7f4d8e800640 1 -- 172.21.15.16:0/1510249691 learned_addr learned my addr 172.21.15.16:0/1510249691 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:12:01.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:01.327+0000 7f4d85400640 1 --2- 172.21.15.16:0/1510249691 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f4d9006f920 0x7f4d9006fd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:01.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:01.327+0000 7f4d8d400640 1 -- 172.21.15.16:0/1510249691 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3598323600 0 0) 0x7f4d900d0920 con 0x7f4d900d2010 2024-03-31T15:12:01.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:01.327+0000 7f4d8d400640 1 -- 172.21.15.16:0/1510249691 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4d74003530 con 0x7f4d900d2010 2024-03-31T15:12:01.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:01.327+0000 7f4d8d400640 1 -- 172.21.15.16:0/1510249691 <== mon.1 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1658815519 0 0) 0x7f4d900d0c00 con 0x7f4d90145e40 2024-03-31T15:12:01.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:01.327+0000 7f4d85400640 1 -- 172.21.15.16:0/1510249691 >> v1:172.21.15.89:6789/0 conn(0x7f4d90145e40 legacy=0x7f4d900cc740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:01.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:01.327+0000 7f4d85400640 1 -- 172.21.15.16:0/1510249691 >> v1:172.21.15.16:6789/0 conn(0x7f4d900d2010 legacy=0x7f4d900d24a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:01.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:01.327+0000 7f4d85400640 1 -- 172.21.15.16:0/1510249691 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d900d1810 con 0x7f4d9006f920 2024-03-31T15:12:01.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:01.327+0000 7f4d85400640 1 --2- 172.21.15.16:0/1510249691 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f4d9006f920 0x7f4d9006fd80 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f4d780097f0 tx=0x7f4d7801c4e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=17cb2ccfd8a9be2f server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:12:01.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:01.331+0000 7f4d8d400640 1 -- 172.21.15.16:0/1510249691 <== mon.2 v2:172.21.15.89:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d780b0070 con 0x7f4d9006f920 2024-03-31T15:12:01.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:01.331+0000 7f4d8d400640 1 -- 172.21.15.16:0/1510249691 <== mon.2 v2:172.21.15.89:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4d780ab600 con 0x7f4d9006f920 2024-03-31T15:12:01.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:01.331+0000 7f4d8d400640 1 -- 172.21.15.16:0/1510249691 <== mon.2 v2:172.21.15.89:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d780ab9b0 con 0x7f4d9006f920 2024-03-31T15:12:01.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:01.331+0000 7f4d95200640 1 -- 172.21.15.16:0/1510249691 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f4d9006f920 msgr2=0x7f4d9006fd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:01.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:01.331+0000 7f4d95200640 1 --2- 172.21.15.16:0/1510249691 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f4d9006f920 0x7f4d9006fd80 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f4d780097f0 tx=0x7f4d7801c4e0 comp rx=0 tx=0).stop 2024-03-31T15:12:01.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:01.331+0000 7f4d95200640 1 -- 172.21.15.16:0/1510249691 shutdown_connections 2024-03-31T15:12:01.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:01.331+0000 7f4d95200640 1 --2- 172.21.15.16:0/1510249691 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f4d9006f920 0x7f4d9006fd80 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:01.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:01.331+0000 7f4d95200640 1 -- 172.21.15.16:0/1510249691 >> 172.21.15.16:0/1510249691 conn(0x7f4d90067330 msgr2=0x7f4d90067740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:12:01.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:01.331+0000 7f4d95200640 1 -- 172.21.15.16:0/1510249691 shutdown_connections 2024-03-31T15:12:01.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:01.331+0000 7f4d95200640 1 -- 172.21.15.16:0/1510249691 wait complete. 2024-03-31T15:12:01.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:01.331+0000 7f4d95200640 1 Processor -- start 2024-03-31T15:12:01.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:01.331+0000 7f4d95200640 1 -- start start 2024-03-31T15:12:01.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:01.331+0000 7f4d95200640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f4d9006f920 0x7f4d9015e9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:01.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:01.331+0000 7f4d95200640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4d900d2010 0x7f4d90142850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:01.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:01.331+0000 7f4d95200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4d90145e40 0x7f4d900c7030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:01.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:01.331+0000 7f4d95200640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f4d900d0ee0 con 0x7f4d90145e40 2024-03-31T15:12:01.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:01.331+0000 7f4d95200640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f4d900d06d0 con 0x7f4d900d2010 2024-03-31T15:12:01.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:01.331+0000 7f4d95200640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f4d900b50c0 con 0x7f4d9006f920 2024-03-31T15:12:01.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:01.331+0000 7f4d8e800640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4d90145e40 0x7f4d900c7030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:01.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:01.331+0000 7f4d8e800640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4d90145e40 0x7f4d900c7030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48948/0 (socket says 172.21.15.16:48948) 2024-03-31T15:12:01.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:01.331+0000 7f4d8e800640 1 -- 172.21.15.16:0/1781063754 learned_addr learned my addr 172.21.15.16:0/1781063754 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:12:01.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:01.335+0000 7f4d8de00640 1 --2- 172.21.15.16:0/1781063754 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f4d9006f920 0x7f4d9015e9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:01.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:01.335+0000 7f4d85400640 1 --2- 172.21.15.16:0/1781063754 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4d900d2010 0x7f4d90142850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:01.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:01.335+0000 7f4d8e800640 1 -- 172.21.15.16:0/1781063754 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f4d9006f920 msgr2=0x7f4d9015e9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:01.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:01.335+0000 7f4d8e800640 1 --2- 172.21.15.16:0/1781063754 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f4d9006f920 0x7f4d9015e9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:01.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:01.335+0000 7f4d8e800640 1 -- 172.21.15.16:0/1781063754 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4d900d2010 msgr2=0x7f4d90142850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:01.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:01.335+0000 7f4d8e800640 1 --2- 172.21.15.16:0/1781063754 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4d900d2010 0x7f4d90142850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:01.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:01.335+0000 7f4d8e800640 1 -- 172.21.15.16:0/1781063754 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d901f8460 con 0x7f4d90145e40 2024-03-31T15:12:01.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:01.335+0000 7f4d8e800640 1 --2- 172.21.15.16:0/1781063754 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4d90145e40 0x7f4d900c7030 secure :-1 s=READY pgs=1327 cs=0 l=1 rev1=1 crypto rx=0x7f4d8800cb40 tx=0x7f4d8800a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:12:01.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:01.335+0000 7f4d86a00640 1 -- 172.21.15.16:0/1781063754 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d88017070 con 0x7f4d90145e40 2024-03-31T15:12:01.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:01.335+0000 7f4d86a00640 1 -- 172.21.15.16:0/1781063754 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4d8800aca0 con 0x7f4d90145e40 2024-03-31T15:12:01.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:01.335+0000 7f4d95200640 1 -- 172.21.15.16:0/1781063754 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4d901f8740 con 0x7f4d90145e40 2024-03-31T15:12:01.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:01.335+0000 7f4d86a00640 1 -- 172.21.15.16:0/1781063754 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d88012620 con 0x7f4d90145e40 2024-03-31T15:12:01.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:01.335+0000 7f4d95200640 1 -- 172.21.15.16:0/1781063754 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4d900c7570 con 0x7f4d90145e40 2024-03-31T15:12:01.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:01.335+0000 7f4d95200640 1 -- 172.21.15.16:0/1781063754 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4d90069950 con 0x7f4d90145e40 2024-03-31T15:12:01.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:01.339+0000 7f4d86a00640 1 -- 172.21.15.16:0/1781063754 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f4d8801a070 con 0x7f4d90145e40 2024-03-31T15:12:01.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:01.339+0000 7f4d86a00640 1 --2- 172.21.15.16:0/1781063754 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f4d6c068380 0x7f4d6c06a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:01.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:01.339+0000 7f4d86a00640 1 -- 172.21.15.16:0/1781063754 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 5910+0+0 (secure 0 0 0) 0x7f4d880867f0 con 0x7f4d90145e40 2024-03-31T15:12:01.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:01.339+0000 7f4d8de00640 1 --2- 172.21.15.16:0/1781063754 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f4d6c068380 0x7f4d6c06a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:01.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:01.339+0000 7f4d8de00640 1 --2- 172.21.15.16:0/1781063754 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f4d6c068380 0x7f4d6c06a840 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f4d80000f70 tx=0x7f4d80009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:12:01.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:01.343+0000 7f4d86a00640 1 -- 172.21.15.16:0/1781063754 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f4d88056d70 con 0x7f4d90145e40 2024-03-31T15:12:01.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:01.511+0000 7f4d95200640 1 -- 172.21.15.16:0/1781063754 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f4d9015aa30 con 0x7f4d90145e40 2024-03-31T15:12:01.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:01.515+0000 7f4d86a00640 1 -- 172.21.15.16:0/1781063754 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v85) ==== 74+0+16894 (secure 0 0 0) 0x7f4d88056670 con 0x7f4d90145e40 2024-03-31T15:12:01.507 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:12:01.507 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":85,"fsid":"286eb5da-fb47-4550-b290-d9519c13b9a5","created":"2024-03-31T15:03:11.438952+0000","modified":"2024-03-31T15:11:19.402716+0000","last_up_change":"2024-03-31T15:03:22.885535+0000","last_in_change":"2024-03-31T15:03:13.941667+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-31T15:03:22.550787+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-03-31T15:09:49.322620+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-03-31T15:09:49.877778+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"92a66450-ef19-40d8-ac3d-d760b23b0bda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6801","nonce":932712244}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6803","nonce":932712244}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6807","nonce":932712244}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6805","nonce":932712244}]},"public_addr":"172.21.15.16:6801/932712244","cluster_addr":"172.21.15.16:6803/932712244","heartbeat_back_addr":"172.21.15.16:6807/932712244","heartbeat_front_addr":"172.21.15.16:6805/932712244","state":["exists","up"]},{"osd":1,"uuid":"da9d1f2b-1968-4959-b843-faa231988e92","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6810","nonce":1398737533}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6811","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6813","nonce":1398737533}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6821","nonce":1398737533}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6815","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6816","nonce":1398737533}]},"public_addr":"172.21.15.16:6810/1398737533","cluster_addr":"172.21.15.16:6813/1398737533","heartbeat_back_addr":"172.21.15.16:6821/1398737533","heartbeat_front_addr":"172.21.15.16:6816/1398737533","state":["exists","up"]},{"osd":2,"uuid":"01a73662-9a21-4b1f-bcbb-42b9636a4638","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6819","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6822","nonce":3018526646}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6825","nonce":3018526646}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6831","nonce":3018526646}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6827","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6829","nonce":3018526646}]},"public_addr":"172.21.15.16:6822/3018526646","cluster_addr":"172.21.15.16:6825/3018526646","heartbeat_back_addr":"172.21.15.16:6831/3018526646","heartbeat_front_addr":"172.21.15.16:6829/3018526646","state":["exists","up"]},{"osd":3,"uuid":"c7771fd3-e18e-4500-ac38-23cf10ebcd75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6809","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6812","nonce":3360352974}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6817","nonce":3360352974}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6828","nonce":3360352974}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6823","nonce":3360352974}]},"public_addr":"172.21.15.16:6812/3360352974","cluster_addr":"172.21.15.16:6817/3360352974","heartbeat_back_addr":"172.21.15.16:6828/3360352974","heartbeat_front_addr":"172.21.15.16:6823/3360352974","state":["exists","up"]},{"osd":4,"uuid":"88260730-1482-4fbd-a40f-cb1124e9df4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6801","nonce":1856844856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6803","nonce":1856844856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6807","nonce":1856844856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6805","nonce":1856844856}]},"public_addr":"172.21.15.89:6801/1856844856","cluster_addr":"172.21.15.89:6803/1856844856","heartbeat_back_addr":"172.21.15.89:6807/1856844856","heartbeat_front_addr":"172.21.15.89:6805/1856844856","state":["exists","up"]},{"osd":5,"uuid":"4cb2c00c-fae0-4dd8-9d27-ec4f0093ccf7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6814","nonce":50395527}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6817","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6820","nonce":50395527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6831","nonce":50395527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6823","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6826","nonce":50395527}]},"public_addr":"172.21.15.89:6814/50395527","cluster_addr":"172.21.15.89:6820/50395527","heartbeat_back_addr":"172.21.15.89:6831/50395527","heartbeat_front_addr":"172.21.15.89:6826/50395527","state":["exists","up"]},{"osd":6,"uuid":"81ce0c19-2554-4cae-acbe-66a05e9e9b9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6809","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6812","nonce":3470471752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6815","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6818","nonce":3470471752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6827","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6829","nonce":3470471752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6821","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6824","nonce":3470471752}]},"public_addr":"172.21.15.89:6812/3470471752","cluster_addr":"172.21.15.89:6818/3470471752","heartbeat_back_addr":"172.21.15.89:6829/3470471752","heartbeat_front_addr":"172.21.15.89:6824/3470471752","state":["exists","up"]},{"osd":7,"uuid":"85e4313b-d37d-442e-8042-153bf20a847d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6811","nonce":1796778630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6813","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6816","nonce":1796778630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6825","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6828","nonce":1796778630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6819","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6822","nonce":1796778630}]},"public_addr":"172.21.15.89:6811/1796778630","cluster_addr":"172.21.15.89:6816/1796778630","heartbeat_back_addr":"172.21.15.89:6828/1796778630","heartbeat_front_addr":"172.21.15.89:6822/1796778630","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.439088+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.529970+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.680512+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:21.044161+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.582111+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.536558+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.409522+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.590229+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"13.d","mappings":[{"from":1,"to":3}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:6832/818074038":"2024-04-01T15:11:19.402687+0000","172.21.15.16:6834/818074038":"2024-04-01T15:11:19.402687+0000","172.21.15.16:6834/2478530200":"2024-04-01T15:10:04.360772+0000","172.21.15.16:6832/2478530200":"2024-04-01T15:10:04.360772+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-31T15:12:01.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:01.515+0000 7f4d95200640 1 -- 172.21.15.16:0/1781063754 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f4d6c068380 msgr2=0x7f4d6c06a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:01.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:01.515+0000 7f4d95200640 1 --2- 172.21.15.16:0/1781063754 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f4d6c068380 0x7f4d6c06a840 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f4d80000f70 tx=0x7f4d80009040 comp rx=0 tx=0).stop 2024-03-31T15:12:01.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:01.515+0000 7f4d95200640 1 -- 172.21.15.16:0/1781063754 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4d90145e40 msgr2=0x7f4d900c7030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:01.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:01.515+0000 7f4d95200640 1 --2- 172.21.15.16:0/1781063754 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4d90145e40 0x7f4d900c7030 secure :-1 s=READY pgs=1327 cs=0 l=1 rev1=1 crypto rx=0x7f4d8800cb40 tx=0x7f4d8800a060 comp rx=0 tx=0).stop 2024-03-31T15:12:01.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:01.515+0000 7f4d95200640 1 -- 172.21.15.16:0/1781063754 shutdown_connections 2024-03-31T15:12:01.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:01.515+0000 7f4d95200640 1 --2- 172.21.15.16:0/1781063754 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f4d6c068380 0x7f4d6c06a840 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:01.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:01.515+0000 7f4d95200640 1 --2- 172.21.15.16:0/1781063754 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4d90145e40 0x7f4d900c7030 unknown :-1 s=CLOSED pgs=1327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:01.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:01.515+0000 7f4d95200640 1 --2- 172.21.15.16:0/1781063754 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4d900d2010 0x7f4d90142850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:01.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:01.515+0000 7f4d95200640 1 --2- 172.21.15.16:0/1781063754 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f4d9006f920 0x7f4d9015e9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:01.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:01.515+0000 7f4d95200640 1 -- 172.21.15.16:0/1781063754 >> 172.21.15.16:0/1781063754 conn(0x7f4d90067330 msgr2=0x7f4d90144e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:12:01.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:01.515+0000 7f4d95200640 1 -- 172.21.15.16:0/1781063754 shutdown_connections 2024-03-31T15:12:01.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:01.515+0000 7f4d95200640 1 -- 172.21.15.16:0/1781063754 wait complete. 2024-03-31T15:12:01.529 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-31T15:03:22.550787+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 12, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-31T15:09:49.322620+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '80', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 13, 'pool_name': 'cephfs_data', 'create_time': '2024-03-31T15:09:49.877778+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '81', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-31T15:12:01.529 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-31T15:12:01.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:01.679+0000 7f1853400640 1 Processor -- start 2024-03-31T15:12:01.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:01.679+0000 7f1853400640 1 -- start start 2024-03-31T15:12:01.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:01.679+0000 7f1853400640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f184c06b620 0x7f184c06ba00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:01.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:01.679+0000 7f1853400640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f184c0b60a0 0x7f184c06fc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:01.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:01.679+0000 7f1853400640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f184c06bf40 con 0x7f184c0d0c00 2024-03-31T15:12:01.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:01.679+0000 7f1853400640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f184c06f820 con 0x7f184c0b60a0 2024-03-31T15:12:01.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:01.679+0000 7f1853400640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f184c0b4a40 con 0x7f184c06b620 2024-03-31T15:12:01.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:01.679+0000 7f1843e00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f184c0b60a0 0x7f184c06fc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:01.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:01.683+0000 7f1843e00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f184c0b60a0 0x7f184c06fc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48950/0 (socket says 172.21.15.16:48950) 2024-03-31T15:12:01.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:01.683+0000 7f1843e00640 1 -- 172.21.15.16:0/4261766806 learned_addr learned my addr 172.21.15.16:0/4261766806 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:12:01.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:01.683+0000 7f1850a00640 1 --2- 172.21.15.16:0/4261766806 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f184c06b620 0x7f184c06ba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:01.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:01.683+0000 7f1843e00640 1 -- 172.21.15.16:0/4261766806 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f184c06b620 msgr2=0x7f184c06ba00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:12:01.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:01.683+0000 7f1843e00640 1 --2- 172.21.15.16:0/4261766806 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f184c06b620 0x7f184c06ba00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:01.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:01.683+0000 7f1843e00640 1 -- 172.21.15.16:0/4261766806 >> v1:172.21.15.89:6789/0 conn(0x7f184c0d0c00 legacy=0x7f184c070180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:01.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:01.683+0000 7f1843e00640 1 -- 172.21.15.16:0/4261766806 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f184c06f540 con 0x7f184c0b60a0 2024-03-31T15:12:01.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:01.683+0000 7f1843e00640 1 --2- 172.21.15.16:0/4261766806 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f184c0b60a0 0x7f184c06fc40 secure :-1 s=READY pgs=1328 cs=0 l=1 rev1=1 crypto rx=0x7f1834009870 tx=0x7f183401c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70759ffcb2e03016 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:12:01.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:01.683+0000 7f184be00640 1 -- 172.21.15.16:0/4261766806 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18340b1070 con 0x7f184c0b60a0 2024-03-31T15:12:01.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:01.683+0000 7f184be00640 1 -- 172.21.15.16:0/4261766806 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f18340ac410 con 0x7f184c0b60a0 2024-03-31T15:12:01.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:01.683+0000 7f184be00640 1 -- 172.21.15.16:0/4261766806 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18340ac7e0 con 0x7f184c0b60a0 2024-03-31T15:12:01.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:01.687+0000 7f1853400640 1 -- 172.21.15.16:0/4261766806 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f184c0b60a0 msgr2=0x7f184c06fc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:01.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:01.687+0000 7f1853400640 1 --2- 172.21.15.16:0/4261766806 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f184c0b60a0 0x7f184c06fc40 secure :-1 s=READY pgs=1328 cs=0 l=1 rev1=1 crypto rx=0x7f1834009870 tx=0x7f183401c4a0 comp rx=0 tx=0).stop 2024-03-31T15:12:01.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:01.687+0000 7f1853400640 1 -- 172.21.15.16:0/4261766806 shutdown_connections 2024-03-31T15:12:01.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:01.687+0000 7f1853400640 1 --2- 172.21.15.16:0/4261766806 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f184c0b60a0 0x7f184c06fc40 unknown :-1 s=CLOSED pgs=1328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:01.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:01.687+0000 7f1853400640 1 --2- 172.21.15.16:0/4261766806 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f184c06b620 0x7f184c06ba00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:01.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:01.687+0000 7f1853400640 1 -- 172.21.15.16:0/4261766806 >> 172.21.15.16:0/4261766806 conn(0x7f184c06e670 msgr2=0x7f184c06ea50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:12:01.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:01.687+0000 7f1853400640 1 -- 172.21.15.16:0/4261766806 shutdown_connections 2024-03-31T15:12:01.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:01.687+0000 7f1853400640 1 -- 172.21.15.16:0/4261766806 wait complete. 2024-03-31T15:12:01.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:01.687+0000 7f1853400640 1 Processor -- start 2024-03-31T15:12:01.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:01.687+0000 7f1853400640 1 -- start start 2024-03-31T15:12:01.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:01.691+0000 7f1853400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f184c06b620 0x7f184c141a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:01.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:01.691+0000 7f1853400640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f184c0b60a0 0x7f184c1615a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:01.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:01.691+0000 7f1853400640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f184c0d0c00 0x7f184c155060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:01.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:01.691+0000 7f1853400640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f184c15e8d0 con 0x7f184c06b620 2024-03-31T15:12:01.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:01.691+0000 7f1853400640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f184c15e750 con 0x7f184c0d0c00 2024-03-31T15:12:01.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:01.691+0000 7f1853400640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f184c06f160 con 0x7f184c0b60a0 2024-03-31T15:12:01.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:01.691+0000 7f1850a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f184c06b620 0x7f184c141a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:01.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:01.691+0000 7f1850a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f184c06b620 0x7f184c141a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48964/0 (socket says 172.21.15.16:48964) 2024-03-31T15:12:01.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:01.691+0000 7f1850a00640 1 -- 172.21.15.16:0/2697387909 learned_addr learned my addr 172.21.15.16:0/2697387909 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:12:01.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:01.691+0000 7f1843e00640 1 --2- 172.21.15.16:0/2697387909 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f184c0b60a0 0x7f184c1615a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:01.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:01.691+0000 7f1851400640 1 --2- 172.21.15.16:0/2697387909 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f184c0d0c00 0x7f184c155060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:01.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:01.691+0000 7f1850a00640 1 -- 172.21.15.16:0/2697387909 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f184c0b60a0 msgr2=0x7f184c1615a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:01.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:01.691+0000 7f1850a00640 1 --2- 172.21.15.16:0/2697387909 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f184c0b60a0 0x7f184c1615a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:01.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:01.691+0000 7f1850a00640 1 -- 172.21.15.16:0/2697387909 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f184c0d0c00 msgr2=0x7f184c155060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:01.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:01.691+0000 7f1850a00640 1 --2- 172.21.15.16:0/2697387909 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f184c0d0c00 0x7f184c155060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:01.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:01.691+0000 7f1850a00640 1 -- 172.21.15.16:0/2697387909 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f184c0680a0 con 0x7f184c06b620 2024-03-31T15:12:01.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:01.691+0000 7f1851400640 1 --2- 172.21.15.16:0/2697387909 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f184c0d0c00 0x7f184c155060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:12:01.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:01.691+0000 7f1850a00640 1 --2- 172.21.15.16:0/2697387909 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f184c06b620 0x7f184c141a60 secure :-1 s=READY pgs=1329 cs=0 l=1 rev1=1 crypto rx=0x7f183c00e870 tx=0x7f183c00ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:12:01.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:01.691+0000 7f1849600640 1 -- 172.21.15.16:0/2697387909 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f183c017070 con 0x7f184c06b620 2024-03-31T15:12:01.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:01.691+0000 7f1849600640 1 -- 172.21.15.16:0/2697387909 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f183c00c940 con 0x7f184c06b620 2024-03-31T15:12:01.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:01.691+0000 7f1853400640 1 -- 172.21.15.16:0/2697387909 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f184c141fa0 con 0x7f184c06b620 2024-03-31T15:12:01.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:01.691+0000 7f1849600640 1 -- 172.21.15.16:0/2697387909 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f183c00ccf0 con 0x7f184c06b620 2024-03-31T15:12:01.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:01.691+0000 7f1853400640 1 -- 172.21.15.16:0/2697387909 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f184c14ff80 con 0x7f184c06b620 2024-03-31T15:12:01.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:01.691+0000 7f1853400640 1 -- 172.21.15.16:0/2697387909 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f184c069950 con 0x7f184c06b620 2024-03-31T15:12:01.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:01.691+0000 7f1849600640 1 -- 172.21.15.16:0/2697387909 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f183c01a070 con 0x7f184c06b620 2024-03-31T15:12:01.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:01.695+0000 7f1849600640 1 --2- 172.21.15.16:0/2697387909 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f18280683b0 0x7f182806a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:01.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:01.695+0000 7f1849600640 1 -- 172.21.15.16:0/2697387909 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 5910+0+0 (secure 0 0 0) 0x7f183c0869b0 con 0x7f184c06b620 2024-03-31T15:12:01.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:01.695+0000 7f1843e00640 1 --2- 172.21.15.16:0/2697387909 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f18280683b0 0x7f182806a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:01.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:01.695+0000 7f1843e00640 1 --2- 172.21.15.16:0/2697387909 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f18280683b0 0x7f182806a870 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f18340ae000 tx=0x7f1834008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:12:01.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:01.699+0000 7f1849600640 1 -- 172.21.15.16:0/2697387909 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f183c056e00 con 0x7f184c06b620 2024-03-31T15:12:01.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:01.863+0000 7f1853400640 1 -- 172.21.15.16:0/2697387909 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f184c070f00 con 0x7f184c06b620 2024-03-31T15:12:01.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:01.863+0000 7f1849600640 1 -- 172.21.15.16:0/2697387909 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v85) ==== 93+0+10 (secure 0 0 0) 0x7f183c056700 con 0x7f184c06b620 2024-03-31T15:12:01.855 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2024-03-31T15:12:01.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:01.863+0000 7f1853400640 1 -- 172.21.15.16:0/2697387909 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f18280683b0 msgr2=0x7f182806a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:01.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:01.863+0000 7f1853400640 1 --2- 172.21.15.16:0/2697387909 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f18280683b0 0x7f182806a870 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f18340ae000 tx=0x7f1834008040 comp rx=0 tx=0).stop 2024-03-31T15:12:01.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:01.863+0000 7f1853400640 1 -- 172.21.15.16:0/2697387909 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f184c06b620 msgr2=0x7f184c141a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:01.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:01.863+0000 7f1853400640 1 --2- 172.21.15.16:0/2697387909 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f184c06b620 0x7f184c141a60 secure :-1 s=READY pgs=1329 cs=0 l=1 rev1=1 crypto rx=0x7f183c00e870 tx=0x7f183c00ec70 comp rx=0 tx=0).stop 2024-03-31T15:12:01.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:01.863+0000 7f1853400640 1 -- 172.21.15.16:0/2697387909 shutdown_connections 2024-03-31T15:12:01.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:01.863+0000 7f1853400640 1 --2- 172.21.15.16:0/2697387909 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f18280683b0 0x7f182806a870 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:01.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:01.863+0000 7f1853400640 1 --2- 172.21.15.16:0/2697387909 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f184c0d0c00 0x7f184c155060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:01.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:01.863+0000 7f1853400640 1 --2- 172.21.15.16:0/2697387909 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f184c0b60a0 0x7f184c1615a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:01.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:01.863+0000 7f1853400640 1 --2- 172.21.15.16:0/2697387909 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f184c06b620 0x7f184c141a60 unknown :-1 s=CLOSED pgs=1329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:01.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:01.863+0000 7f1853400640 1 -- 172.21.15.16:0/2697387909 >> 172.21.15.16:0/2697387909 conn(0x7f184c06e670 msgr2=0x7f184c15d3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:12:01.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:01.863+0000 7f1853400640 1 -- 172.21.15.16:0/2697387909 shutdown_connections 2024-03-31T15:12:01.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:01.863+0000 7f1853400640 1 -- 172.21.15.16:0/2697387909 wait complete. 2024-03-31T15:12:01.877 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-31T15:12:02.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:02.027+0000 7f40d7200640 1 Processor -- start 2024-03-31T15:12:02.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:02.027+0000 7f40d7200640 1 -- start start 2024-03-31T15:12:02.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:02.027+0000 7f40d7200640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f40d006f920 0x7f40d006fd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:02.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:02.027+0000 7f40d7200640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f40d00d0a60 0x7f40d00cea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:02.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:02.027+0000 7f40d7200640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f40d0070240 con 0x7f40d00cef80 2024-03-31T15:12:02.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:02.027+0000 7f40d7200640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f40d013f670 con 0x7f40d006f920 2024-03-31T15:12:02.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:02.027+0000 7f40d7200640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f40d015ca70 con 0x7f40d00d0a60 2024-03-31T15:12:02.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:02.027+0000 7f40cfe00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f40d006f920 0x7f40d006fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:02.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:02.027+0000 7f40cfe00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f40d006f920 0x7f40d006fd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48976/0 (socket says 172.21.15.16:48976) 2024-03-31T15:12:02.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:02.027+0000 7f40cfe00640 1 -- 172.21.15.16:0/4208185810 learned_addr learned my addr 172.21.15.16:0/4208185810 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:12:02.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:02.031+0000 7f40c7e00640 1 --2- 172.21.15.16:0/4208185810 >> v2:172.21.15.89:3300/0 conn(0x7f40d00d0a60 0x7f40d00cea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:02.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:02.031+0000 7f40cfe00640 1 -- 172.21.15.16:0/4208185810 >> v2:172.21.15.89:3300/0 conn(0x7f40d00d0a60 msgr2=0x7f40d00cea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:12:02.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:02.031+0000 7f40cfe00640 1 --2- 172.21.15.16:0/4208185810 >> v2:172.21.15.89:3300/0 conn(0x7f40d00d0a60 0x7f40d00cea40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:02.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:02.031+0000 7f40cfe00640 1 -- 172.21.15.16:0/4208185810 >> v1:172.21.15.89:6789/0 conn(0x7f40d00cef80 legacy=0x7f40d0070e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:02.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:02.031+0000 7f40cfe00640 1 -- 172.21.15.16:0/4208185810 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40d015cd50 con 0x7f40d006f920 2024-03-31T15:12:02.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:02.031+0000 7f40cfe00640 1 --2- 172.21.15.16:0/4208185810 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f40d006f920 0x7f40d006fd00 secure :-1 s=READY pgs=1330 cs=0 l=1 rev1=1 crypto rx=0x7f40d00736b0 tx=0x7f40c001c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b74d15856e57cf5 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:12:02.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:02.031+0000 7f40cf400640 1 -- 172.21.15.16:0/4208185810 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40c00b0070 con 0x7f40d006f920 2024-03-31T15:12:02.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:02.031+0000 7f40cf400640 1 -- 172.21.15.16:0/4208185810 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f40c00ab600 con 0x7f40d006f920 2024-03-31T15:12:02.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:02.031+0000 7f40cf400640 1 -- 172.21.15.16:0/4208185810 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40c00ab9d0 con 0x7f40d006f920 2024-03-31T15:12:02.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:02.035+0000 7f40d7200640 1 -- 172.21.15.16:0/4208185810 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f40d006f920 msgr2=0x7f40d006fd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:02.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:02.035+0000 7f40d7200640 1 --2- 172.21.15.16:0/4208185810 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f40d006f920 0x7f40d006fd00 secure :-1 s=READY pgs=1330 cs=0 l=1 rev1=1 crypto rx=0x7f40d00736b0 tx=0x7f40c001c490 comp rx=0 tx=0).stop 2024-03-31T15:12:02.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:02.035+0000 7f40d7200640 1 -- 172.21.15.16:0/4208185810 shutdown_connections 2024-03-31T15:12:02.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:02.035+0000 7f40d7200640 1 --2- 172.21.15.16:0/4208185810 >> v2:172.21.15.89:3300/0 conn(0x7f40d00d0a60 0x7f40d00cea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:02.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:02.035+0000 7f40d7200640 1 --2- 172.21.15.16:0/4208185810 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f40d006f920 0x7f40d006fd00 unknown :-1 s=CLOSED pgs=1330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:02.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:02.035+0000 7f40d7200640 1 -- 172.21.15.16:0/4208185810 >> 172.21.15.16:0/4208185810 conn(0x7f40d0067930 msgr2=0x7f40d00b3ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:12:02.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:02.035+0000 7f40d7200640 1 -- 172.21.15.16:0/4208185810 shutdown_connections 2024-03-31T15:12:02.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:02.035+0000 7f40d7200640 1 -- 172.21.15.16:0/4208185810 wait complete. 2024-03-31T15:12:02.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:02.035+0000 7f40d7200640 1 Processor -- start 2024-03-31T15:12:02.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:02.035+0000 7f40d7200640 1 -- start start 2024-03-31T15:12:02.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:02.035+0000 7f40d7200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f40d006f920 0x7f40d0153750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:02.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:02.035+0000 7f40d7200640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f40d00cef80 0x7f40d015d850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:02.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:02.035+0000 7f40d7200640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f40d00d0a60 0x7f40d00b65a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:02.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:02.035+0000 7f40d7200640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f40d01690b0 con 0x7f40d006f920 2024-03-31T15:12:02.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:02.035+0000 7f40d7200640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f40d00cf390 con 0x7f40d00cef80 2024-03-31T15:12:02.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:02.035+0000 7f40d7200640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f40d0169230 con 0x7f40d00d0a60 2024-03-31T15:12:02.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:02.035+0000 7f40cfe00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f40d006f920 0x7f40d0153750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:02.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:02.035+0000 7f40cfe00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f40d006f920 0x7f40d0153750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48984/0 (socket says 172.21.15.16:48984) 2024-03-31T15:12:02.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:02.035+0000 7f40cfe00640 1 -- 172.21.15.16:0/3719295071 learned_addr learned my addr 172.21.15.16:0/3719295071 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:12:02.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:02.035+0000 7f40c7e00640 1 --2- 172.21.15.16:0/3719295071 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f40d00cef80 0x7f40d015d850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:02.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:02.035+0000 7f40d5200640 1 --2- 172.21.15.16:0/3719295071 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f40d00d0a60 0x7f40d00b65a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:02.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:02.035+0000 7f40cfe00640 1 -- 172.21.15.16:0/3719295071 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f40d00d0a60 msgr2=0x7f40d00b65a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:02.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:02.035+0000 7f40cfe00640 1 --2- 172.21.15.16:0/3719295071 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f40d00d0a60 0x7f40d00b65a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:02.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:02.035+0000 7f40cfe00640 1 -- 172.21.15.16:0/3719295071 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f40d00cef80 msgr2=0x7f40d015d850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:02.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:02.035+0000 7f40cfe00640 1 --2- 172.21.15.16:0/3719295071 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f40d00cef80 0x7f40d015d850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:02.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:02.035+0000 7f40cfe00640 1 -- 172.21.15.16:0/3719295071 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40d015af40 con 0x7f40d006f920 2024-03-31T15:12:02.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:02.035+0000 7f40cfe00640 1 --2- 172.21.15.16:0/3719295071 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f40d006f920 0x7f40d0153750 secure :-1 s=READY pgs=1331 cs=0 l=1 rev1=1 crypto rx=0x7f40c001a0c0 tx=0x7f40c0019230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:12:02.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:02.039+0000 7f40ccc00640 1 -- 172.21.15.16:0/3719295071 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40c00b0040 con 0x7f40d006f920 2024-03-31T15:12:02.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:02.039+0000 7f40ccc00640 1 -- 172.21.15.16:0/3719295071 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f40c00b9070 con 0x7f40d006f920 2024-03-31T15:12:02.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:02.039+0000 7f40d7200640 1 -- 172.21.15.16:0/3719295071 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f40d0153c90 con 0x7f40d006f920 2024-03-31T15:12:02.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:02.039+0000 7f40ccc00640 1 -- 172.21.15.16:0/3719295071 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40c00b4870 con 0x7f40d006f920 2024-03-31T15:12:02.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:02.039+0000 7f40d7200640 1 -- 172.21.15.16:0/3719295071 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f40d015e480 con 0x7f40d006f920 2024-03-31T15:12:02.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:02.039+0000 7f40d7200640 1 -- 172.21.15.16:0/3719295071 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f40d0069950 con 0x7f40d006f920 2024-03-31T15:12:02.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:02.039+0000 7f40ccc00640 1 -- 172.21.15.16:0/3719295071 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f40c00ab480 con 0x7f40d006f920 2024-03-31T15:12:02.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:02.039+0000 7f40ccc00640 1 --2- 172.21.15.16:0/3719295071 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f40ac068380 0x7f40ac06a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:02.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:02.039+0000 7f40ccc00640 1 -- 172.21.15.16:0/3719295071 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 5910+0+0 (secure 0 0 0) 0x7f40c0120840 con 0x7f40d006f920 2024-03-31T15:12:02.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:02.039+0000 7f40c7e00640 1 --2- 172.21.15.16:0/3719295071 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f40ac068380 0x7f40ac06a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:02.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:02.043+0000 7f40c7e00640 1 --2- 172.21.15.16:0/3719295071 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f40ac068380 0x7f40ac06a840 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f40d006ca10 tx=0x7f40b8008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:12:02.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:02.043+0000 7f40ccc00640 1 -- 172.21.15.16:0/3719295071 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f40c0016110 con 0x7f40d006f920 2024-03-31T15:12:02.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:02.219+0000 7f40d7200640 1 -- 172.21.15.16:0/3719295071 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f40d00b50e0 con 0x7f40d006f920 2024-03-31T15:12:02.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:02.219+0000 7f40ccc00640 1 -- 172.21.15.16:0/3719295071 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v85) ==== 104+0+11 (secure 0 0 0) 0x7f40c00f0c90 con 0x7f40d006f920 2024-03-31T15:12:02.213 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2024-03-31T15:12:02.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:02.223+0000 7f40d7200640 1 -- 172.21.15.16:0/3719295071 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f40ac068380 msgr2=0x7f40ac06a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:02.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:02.223+0000 7f40d7200640 1 --2- 172.21.15.16:0/3719295071 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f40ac068380 0x7f40ac06a840 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f40d006ca10 tx=0x7f40b8008040 comp rx=0 tx=0).stop 2024-03-31T15:12:02.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:02.223+0000 7f40d7200640 1 -- 172.21.15.16:0/3719295071 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f40d006f920 msgr2=0x7f40d0153750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:02.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:02.223+0000 7f40d7200640 1 --2- 172.21.15.16:0/3719295071 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f40d006f920 0x7f40d0153750 secure :-1 s=READY pgs=1331 cs=0 l=1 rev1=1 crypto rx=0x7f40c001a0c0 tx=0x7f40c0019230 comp rx=0 tx=0).stop 2024-03-31T15:12:02.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:02.223+0000 7f40d7200640 1 -- 172.21.15.16:0/3719295071 shutdown_connections 2024-03-31T15:12:02.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:02.223+0000 7f40d7200640 1 --2- 172.21.15.16:0/3719295071 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f40ac068380 0x7f40ac06a840 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:02.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:02.223+0000 7f40d7200640 1 --2- 172.21.15.16:0/3719295071 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f40d00d0a60 0x7f40d00b65a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:02.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:02.223+0000 7f40d7200640 1 --2- 172.21.15.16:0/3719295071 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f40d00cef80 0x7f40d015d850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:02.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:02.223+0000 7f40d7200640 1 --2- 172.21.15.16:0/3719295071 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f40d006f920 0x7f40d0153750 unknown :-1 s=CLOSED pgs=1331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:02.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:02.223+0000 7f40d7200640 1 -- 172.21.15.16:0/3719295071 >> 172.21.15.16:0/3719295071 conn(0x7f40d0067930 msgr2=0x7f40d00b3970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:12:02.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:02.223+0000 7f40d7200640 1 -- 172.21.15.16:0/3719295071 shutdown_connections 2024-03-31T15:12:02.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:02.223+0000 7f40d7200640 1 -- 172.21.15.16:0/3719295071 wait complete. 2024-03-31T15:12:02.234 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-31T15:12:02.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:02.383+0000 7f2e90a00640 1 Processor -- start 2024-03-31T15:12:02.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:02.383+0000 7f2e90a00640 1 -- start start 2024-03-31T15:12:02.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:02.383+0000 7f2e90a00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f2e8c070030 0x7f2e8c143c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:02.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:02.383+0000 7f2e90a00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f2e8c141590 0x7f2e8c144170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:02.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:02.383+0000 7f2e90a00640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2e8c1446b0 con 0x7f2e8c1411b0 2024-03-31T15:12:02.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:02.383+0000 7f2e90a00640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f2e8c15f910 con 0x7f2e8c070030 2024-03-31T15:12:02.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:02.383+0000 7f2e90a00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f2e8c162910 con 0x7f2e8c141590 2024-03-31T15:12:02.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:02.387+0000 7f2e89e00640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f2e8c1411b0 0x7f2e8c06f920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:35100/0 (socket says 172.21.15.16:35100) 2024-03-31T15:12:02.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:02.387+0000 7f2e89e00640 1 -- 172.21.15.16:0/2242875864 learned_addr learned my addr 172.21.15.16:0/2242875864 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:12:02.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:02.387+0000 7f2e83e00640 1 --2- 172.21.15.16:0/2242875864 >> v2:172.21.15.89:3300/0 conn(0x7f2e8c070030 0x7f2e8c143c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:02.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:02.387+0000 7f2e8a800640 1 --2- 172.21.15.16:0/2242875864 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f2e8c141590 0x7f2e8c144170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:02.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:02.387+0000 7f2e89400640 1 -- 172.21.15.16:0/2242875864 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3675213304 0 0) 0x7f2e8c1446b0 con 0x7f2e8c1411b0 2024-03-31T15:12:02.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:02.387+0000 7f2e89400640 1 -- 172.21.15.16:0/2242875864 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2e70003530 con 0x7f2e8c1411b0 2024-03-31T15:12:02.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:02.387+0000 7f2e83e00640 1 -- 172.21.15.16:0/2242875864 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f2e8c141590 msgr2=0x7f2e8c144170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:12:02.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:02.387+0000 7f2e83e00640 1 --2- 172.21.15.16:0/2242875864 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f2e8c141590 0x7f2e8c144170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:02.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:02.387+0000 7f2e83e00640 1 -- 172.21.15.16:0/2242875864 >> v1:172.21.15.16:6789/0 conn(0x7f2e8c1411b0 legacy=0x7f2e8c06f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:02.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:02.387+0000 7f2e83e00640 1 -- 172.21.15.16:0/2242875864 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e8c144980 con 0x7f2e8c070030 2024-03-31T15:12:02.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:02.387+0000 7f2e83e00640 1 --2- 172.21.15.16:0/2242875864 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2e8c070030 0x7f2e8c143c30 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f2e740097c0 tx=0x7f2e7401c470 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8c56db4203508293 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:12:02.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:02.391+0000 7f2e89400640 1 -- 172.21.15.16:0/2242875864 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e740a7070 con 0x7f2e8c070030 2024-03-31T15:12:02.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:02.391+0000 7f2e89400640 1 -- 172.21.15.16:0/2242875864 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2e740b0070 con 0x7f2e8c070030 2024-03-31T15:12:02.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:02.391+0000 7f2e89400640 1 -- 172.21.15.16:0/2242875864 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e740ab870 con 0x7f2e8c070030 2024-03-31T15:12:02.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:02.391+0000 7f2e90a00640 1 -- 172.21.15.16:0/2242875864 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2e8c070030 msgr2=0x7f2e8c143c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:02.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:02.391+0000 7f2e90a00640 1 --2- 172.21.15.16:0/2242875864 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2e8c070030 0x7f2e8c143c30 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f2e740097c0 tx=0x7f2e7401c470 comp rx=0 tx=0).stop 2024-03-31T15:12:02.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:02.391+0000 7f2e90a00640 1 -- 172.21.15.16:0/2242875864 shutdown_connections 2024-03-31T15:12:02.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:02.391+0000 7f2e90a00640 1 --2- 172.21.15.16:0/2242875864 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f2e8c141590 0x7f2e8c144170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:02.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:02.391+0000 7f2e90a00640 1 --2- 172.21.15.16:0/2242875864 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2e8c070030 0x7f2e8c143c30 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:02.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:02.391+0000 7f2e90a00640 1 -- 172.21.15.16:0/2242875864 >> 172.21.15.16:0/2242875864 conn(0x7f2e8c067930 msgr2=0x7f2e8c0b3ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:12:02.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:02.391+0000 7f2e90a00640 1 -- 172.21.15.16:0/2242875864 shutdown_connections 2024-03-31T15:12:02.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:02.391+0000 7f2e90a00640 1 -- 172.21.15.16:0/2242875864 wait complete. 2024-03-31T15:12:02.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:02.391+0000 7f2e90a00640 1 Processor -- start 2024-03-31T15:12:02.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:02.391+0000 7f2e90a00640 1 -- start start 2024-03-31T15:12:02.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:02.391+0000 7f2e90a00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2e8c070030 0x7f2e8c13f320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:02.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:02.391+0000 7f2e90a00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f2e8c1411b0 0x7f2e8c142cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:02.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:02.391+0000 7f2e90a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2e8c141590 0x7f2e8c15faa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:02.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:02.391+0000 7f2e90a00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f2e8c15f110 con 0x7f2e8c141590 2024-03-31T15:12:02.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:02.391+0000 7f2e90a00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f2e8c162d80 con 0x7f2e8c070030 2024-03-31T15:12:02.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:02.391+0000 7f2e90a00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f2e8c15f290 con 0x7f2e8c1411b0 2024-03-31T15:12:02.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:02.391+0000 7f2e8a800640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2e8c141590 0x7f2e8c15faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:02.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:02.391+0000 7f2e8a800640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2e8c141590 0x7f2e8c15faa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48990/0 (socket says 172.21.15.16:48990) 2024-03-31T15:12:02.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:02.391+0000 7f2e8a800640 1 -- 172.21.15.16:0/703411696 learned_addr learned my addr 172.21.15.16:0/703411696 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:12:02.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:02.391+0000 7f2e83e00640 1 --2- 172.21.15.16:0/703411696 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f2e8c1411b0 0x7f2e8c142cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:02.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:02.391+0000 7f2e89e00640 1 --2- 172.21.15.16:0/703411696 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2e8c070030 0x7f2e8c13f320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:02.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:02.395+0000 7f2e83e00640 1 -- 172.21.15.16:0/703411696 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2e8c070030 msgr2=0x7f2e8c13f320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:02.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:02.395+0000 7f2e83e00640 1 --2- 172.21.15.16:0/703411696 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2e8c070030 0x7f2e8c13f320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:02.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:02.395+0000 7f2e83e00640 1 -- 172.21.15.16:0/703411696 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2e8c141590 msgr2=0x7f2e8c15faa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:02.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:02.395+0000 7f2e83e00640 1 --2- 172.21.15.16:0/703411696 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2e8c141590 0x7f2e8c15faa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:02.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:02.395+0000 7f2e83e00640 1 -- 172.21.15.16:0/703411696 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e8c13fe50 con 0x7f2e8c1411b0 2024-03-31T15:12:02.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:02.395+0000 7f2e8a800640 1 --2- 172.21.15.16:0/703411696 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2e8c141590 0x7f2e8c15faa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:12:02.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:02.395+0000 7f2e83e00640 1 --2- 172.21.15.16:0/703411696 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f2e8c1411b0 0x7f2e8c142cc0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f2e740ac000 tx=0x7f2e7401acb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:12:02.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:02.395+0000 7f2e82000640 1 -- 172.21.15.16:0/703411696 <== mon.2 v2:172.21.15.89:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e74006e20 con 0x7f2e8c1411b0 2024-03-31T15:12:02.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:02.395+0000 7f2e82000640 1 -- 172.21.15.16:0/703411696 <== mon.2 v2:172.21.15.89:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2e740bf610 con 0x7f2e8c1411b0 2024-03-31T15:12:02.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:02.395+0000 7f2e82000640 1 -- 172.21.15.16:0/703411696 <== mon.2 v2:172.21.15.89:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e740bf9e0 con 0x7f2e8c1411b0 2024-03-31T15:12:02.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:02.395+0000 7f2e90a00640 1 -- 172.21.15.16:0/703411696 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2e8c141b40 con 0x7f2e8c1411b0 2024-03-31T15:12:02.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:02.395+0000 7f2e90a00640 1 -- 172.21.15.16:0/703411696 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f2e8c143200 con 0x7f2e8c1411b0 2024-03-31T15:12:02.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:02.395+0000 7f2e90a00640 1 -- 172.21.15.16:0/703411696 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2e8c069950 con 0x7f2e8c1411b0 2024-03-31T15:12:02.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:02.395+0000 7f2e82000640 1 -- 172.21.15.16:0/703411696 <== mon.2 v2:172.21.15.89:3301/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f2e740bd020 con 0x7f2e8c1411b0 2024-03-31T15:12:02.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:02.395+0000 7f2e82000640 1 --2- 172.21.15.16:0/703411696 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f2e68068380 0x7f2e6806a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:02.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:02.399+0000 7f2e82000640 1 -- 172.21.15.16:0/703411696 <== mon.2 v2:172.21.15.89:3301/0 5 ==== osd_map(85..85 src has 1..85) ==== 5910+0+0 (secure 0 0 0) 0x7f2e741211a0 con 0x7f2e8c1411b0 2024-03-31T15:12:02.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:02.399+0000 7f2e89e00640 1 --2- 172.21.15.16:0/703411696 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f2e68068380 0x7f2e6806a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:02.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:02.399+0000 7f2e89e00640 1 --2- 172.21.15.16:0/703411696 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f2e68068380 0x7f2e6806a840 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f2e7c007fd0 tx=0x7f2e7c009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:12:02.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:02.399+0000 7f2e82000640 1 -- 172.21.15.16:0/703411696 <== mon.2 v2:172.21.15.89:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f2e74016090 con 0x7f2e8c1411b0 2024-03-31T15:12:02.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:02.567+0000 7f2e90a00640 1 -- 172.21.15.16:0/703411696 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f2e8c15f6f0 con 0x7f2e8c1411b0 2024-03-31T15:12:02.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:02.567+0000 7f2e82000640 1 -- 172.21.15.16:0/703411696 <== mon.2 v2:172.21.15.89:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v85) ==== 100+0+11 (secure 0 0 0) 0x7f2e740f1020 con 0x7f2e8c1411b0 2024-03-31T15:12:02.559 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 64 2024-03-31T15:12:02.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:02.567+0000 7f2e90a00640 1 -- 172.21.15.16:0/703411696 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f2e68068380 msgr2=0x7f2e6806a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:02.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:02.567+0000 7f2e90a00640 1 --2- 172.21.15.16:0/703411696 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f2e68068380 0x7f2e6806a840 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f2e7c007fd0 tx=0x7f2e7c009040 comp rx=0 tx=0).stop 2024-03-31T15:12:02.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:02.567+0000 7f2e90a00640 1 -- 172.21.15.16:0/703411696 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f2e8c1411b0 msgr2=0x7f2e8c142cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:02.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:02.567+0000 7f2e90a00640 1 --2- 172.21.15.16:0/703411696 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f2e8c1411b0 0x7f2e8c142cc0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f2e740ac000 tx=0x7f2e7401acb0 comp rx=0 tx=0).stop 2024-03-31T15:12:02.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:02.567+0000 7f2e90a00640 1 -- 172.21.15.16:0/703411696 shutdown_connections 2024-03-31T15:12:02.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:02.567+0000 7f2e90a00640 1 --2- 172.21.15.16:0/703411696 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f2e68068380 0x7f2e6806a840 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:02.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:02.567+0000 7f2e90a00640 1 --2- 172.21.15.16:0/703411696 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2e8c141590 0x7f2e8c15faa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:02.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:02.567+0000 7f2e90a00640 1 --2- 172.21.15.16:0/703411696 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f2e8c1411b0 0x7f2e8c142cc0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:02.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:02.567+0000 7f2e90a00640 1 --2- 172.21.15.16:0/703411696 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2e8c070030 0x7f2e8c13f320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:02.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:02.567+0000 7f2e90a00640 1 -- 172.21.15.16:0/703411696 >> 172.21.15.16:0/703411696 conn(0x7f2e8c067930 msgr2=0x7f2e8c140280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:12:02.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:02.567+0000 7f2e90a00640 1 -- 172.21.15.16:0/703411696 shutdown_connections 2024-03-31T15:12:02.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:02.567+0000 7f2e90a00640 1 -- 172.21.15.16:0/703411696 wait complete. 2024-03-31T15:12:02.581 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-31T15:12:02.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:02.731+0000 7fad81800640 1 Processor -- start 2024-03-31T15:12:02.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:02.731+0000 7fad81800640 1 -- start start 2024-03-31T15:12:02.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:02.731+0000 7fad81800640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fad7c072610 0x7fad7c15ca50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:02.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:02.731+0000 7fad81800640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fad7c06bd30 0x7fad7c15cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:02.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:02.731+0000 7fad81800640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fad7c073190 con 0x7fad7c1450e0 2024-03-31T15:12:02.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:02.731+0000 7fad81800640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7fad7c073470 con 0x7fad7c06bd30 2024-03-31T15:12:02.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:02.731+0000 7fad81800640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fad7c0704c0 con 0x7fad7c072610 2024-03-31T15:12:02.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:02.731+0000 7fad7b400640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fad7c06bd30 0x7fad7c15cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:02.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:02.731+0000 7fad73e00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fad7c072610 0x7fad7c15ca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:02.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:02.731+0000 7fad7aa00640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fad7c1450e0 0x7fad7c06b620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.16:47024/0 (socket says 172.21.15.16:47024) 2024-03-31T15:12:02.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:02.731+0000 7fad7aa00640 1 -- 172.21.15.16:0/1407650070 learned_addr learned my addr 172.21.15.16:0/1407650070 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:12:02.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:02.731+0000 7fad73e00640 1 -- 172.21.15.16:0/1407650070 >> v2:172.21.15.89:3300/0 conn(0x7fad7c06bd30 msgr2=0x7fad7c15cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:12:02.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:02.731+0000 7fad73e00640 1 --2- 172.21.15.16:0/1407650070 >> v2:172.21.15.89:3300/0 conn(0x7fad7c06bd30 0x7fad7c15cf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:02.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:02.731+0000 7fad73e00640 1 -- 172.21.15.16:0/1407650070 >> v1:172.21.15.89:6789/0 conn(0x7fad7c1450e0 legacy=0x7fad7c06b620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:02.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:02.731+0000 7fad73e00640 1 -- 172.21.15.16:0/1407650070 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad7c0707a0 con 0x7fad7c072610 2024-03-31T15:12:02.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:02.731+0000 7fad7a000640 1 -- 172.21.15.16:0/1407650070 <== mon.1 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2509075255 0 0) 0x7fad7c073190 con 0x7fad7c1450e0 2024-03-31T15:12:02.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:02.731+0000 7fad7b400640 1 --2- 172.21.15.16:0/1407650070 >> v2:172.21.15.89:3300/0 conn(0x7fad7c06bd30 0x7fad7c15cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:12:02.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:02.735+0000 7fad73e00640 1 --2- 172.21.15.16:0/1407650070 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fad7c072610 0x7fad7c15ca50 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fad640097c0 tx=0x7fad6401c410 comp rx=0 tx=0).ready entity=mon.2 client_cookie=76f3d785a9c7a0ff server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:12:02.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:02.735+0000 7fad7a000640 1 -- 172.21.15.16:0/1407650070 <== mon.2 v2:172.21.15.89:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad640b0070 con 0x7fad7c072610 2024-03-31T15:12:02.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:02.735+0000 7fad7a000640 1 -- 172.21.15.16:0/1407650070 <== mon.2 v2:172.21.15.89:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fad640ab600 con 0x7fad7c072610 2024-03-31T15:12:02.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:02.735+0000 7fad7a000640 1 -- 172.21.15.16:0/1407650070 <== mon.2 v2:172.21.15.89:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad640ab9d0 con 0x7fad7c072610 2024-03-31T15:12:02.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:02.735+0000 7fad81800640 1 -- 172.21.15.16:0/1407650070 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fad7c072610 msgr2=0x7fad7c15ca50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:02.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:02.735+0000 7fad81800640 1 --2- 172.21.15.16:0/1407650070 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fad7c072610 0x7fad7c15ca50 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fad640097c0 tx=0x7fad6401c410 comp rx=0 tx=0).stop 2024-03-31T15:12:02.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:02.735+0000 7fad81800640 1 -- 172.21.15.16:0/1407650070 shutdown_connections 2024-03-31T15:12:02.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:02.735+0000 7fad81800640 1 --2- 172.21.15.16:0/1407650070 >> v2:172.21.15.89:3300/0 conn(0x7fad7c06bd30 0x7fad7c15cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:02.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:02.735+0000 7fad81800640 1 --2- 172.21.15.16:0/1407650070 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fad7c072610 0x7fad7c15ca50 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:02.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:02.735+0000 7fad81800640 1 -- 172.21.15.16:0/1407650070 >> 172.21.15.16:0/1407650070 conn(0x7fad7c0d1060 msgr2=0x7fad7c0d1470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:12:02.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:02.735+0000 7fad81800640 1 -- 172.21.15.16:0/1407650070 shutdown_connections 2024-03-31T15:12:02.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:02.735+0000 7fad81800640 1 -- 172.21.15.16:0/1407650070 wait complete. 2024-03-31T15:12:02.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:02.739+0000 7fad81800640 1 Processor -- start 2024-03-31T15:12:02.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:02.739+0000 7fad81800640 1 -- start start 2024-03-31T15:12:02.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:02.739+0000 7fad81800640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fad7c06bd30 0x7fad7c0b5ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:02.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:02.739+0000 7fad81800640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fad7c072610 0x7fad7c0b23e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:02.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:02.739+0000 7fad81800640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fad7c1450e0 0x7fad7c0b3ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:02.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:02.739+0000 7fad81800640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fad7c15d860 con 0x7fad7c1450e0 2024-03-31T15:12:02.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:02.739+0000 7fad81800640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fad7c15d6e0 con 0x7fad7c072610 2024-03-31T15:12:02.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:02.739+0000 7fad81800640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fad7c1697e0 con 0x7fad7c06bd30 2024-03-31T15:12:02.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:02.739+0000 7fad7b400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fad7c1450e0 0x7fad7c0b3ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:02.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:02.739+0000 7fad7b400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fad7c1450e0 0x7fad7c0b3ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48992/0 (socket says 172.21.15.16:48992) 2024-03-31T15:12:02.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:02.739+0000 7fad7b400640 1 -- 172.21.15.16:0/2414110618 learned_addr learned my addr 172.21.15.16:0/2414110618 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:12:02.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:02.739+0000 7fad7aa00640 1 --2- 172.21.15.16:0/2414110618 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fad7c06bd30 0x7fad7c0b5ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:02.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:02.739+0000 7fad73e00640 1 --2- 172.21.15.16:0/2414110618 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fad7c072610 0x7fad7c0b23e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:02.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:02.739+0000 7fad7b400640 1 -- 172.21.15.16:0/2414110618 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fad7c06bd30 msgr2=0x7fad7c0b5ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:02.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:02.739+0000 7fad7b400640 1 --2- 172.21.15.16:0/2414110618 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fad7c06bd30 0x7fad7c0b5ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:02.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:02.739+0000 7fad7b400640 1 -- 172.21.15.16:0/2414110618 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fad7c072610 msgr2=0x7fad7c0b23e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:02.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:02.739+0000 7fad7b400640 1 --2- 172.21.15.16:0/2414110618 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fad7c072610 0x7fad7c0b23e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:02.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:02.739+0000 7fad7b400640 1 -- 172.21.15.16:0/2414110618 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad7c0b4a40 con 0x7fad7c1450e0 2024-03-31T15:12:02.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:02.739+0000 7fad7aa00640 1 --2- 172.21.15.16:0/2414110618 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fad7c06bd30 0x7fad7c0b5ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:12:02.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:02.739+0000 7fad73e00640 1 --2- 172.21.15.16:0/2414110618 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fad7c072610 0x7fad7c0b23e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:12:02.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:02.743+0000 7fad7b400640 1 --2- 172.21.15.16:0/2414110618 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fad7c1450e0 0x7fad7c0b3ff0 secure :-1 s=READY pgs=1333 cs=0 l=1 rev1=1 crypto rx=0x7fad74009e60 tx=0x7fad7400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:12:02.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:02.743+0000 7fad72a00640 1 -- 172.21.15.16:0/2414110618 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad74018030 con 0x7fad7c1450e0 2024-03-31T15:12:02.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:02.743+0000 7fad72a00640 1 -- 172.21.15.16:0/2414110618 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fad7400cdf0 con 0x7fad7c1450e0 2024-03-31T15:12:02.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:02.743+0000 7fad81800640 1 -- 172.21.15.16:0/2414110618 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fad7c0b4d20 con 0x7fad7c1450e0 2024-03-31T15:12:02.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:02.743+0000 7fad72a00640 1 -- 172.21.15.16:0/2414110618 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad74003bc0 con 0x7fad7c1450e0 2024-03-31T15:12:02.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:02.743+0000 7fad81800640 1 -- 172.21.15.16:0/2414110618 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fad7c0b55c0 con 0x7fad7c1450e0 2024-03-31T15:12:02.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:02.743+0000 7fad81800640 1 -- 172.21.15.16:0/2414110618 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fad7c0b4ef0 con 0x7fad7c1450e0 2024-03-31T15:12:02.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:02.743+0000 7fad72a00640 1 -- 172.21.15.16:0/2414110618 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fad7401b070 con 0x7fad7c1450e0 2024-03-31T15:12:02.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:02.743+0000 7fad72a00640 1 --2- 172.21.15.16:0/2414110618 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fad58068380 0x7fad5806a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:02.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:02.747+0000 7fad72a00640 1 -- 172.21.15.16:0/2414110618 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 5910+0+0 (secure 0 0 0) 0x7fad740874b0 con 0x7fad7c1450e0 2024-03-31T15:12:02.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:02.747+0000 7fad7aa00640 1 --2- 172.21.15.16:0/2414110618 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fad58068380 0x7fad5806a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:02.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:02.747+0000 7fad7aa00640 1 --2- 172.21.15.16:0/2414110618 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fad58068380 0x7fad5806a840 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7fad6c007fd0 tx=0x7fad6c009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:12:02.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:02.747+0000 7fad72a00640 1 -- 172.21.15.16:0/2414110618 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fad74010040 con 0x7fad7c1450e0 2024-03-31T15:12:02.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:02.947+0000 7fad81800640 1 -- 172.21.15.16:0/2414110618 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fad7c069950 con 0x7fad7c1450e0 2024-03-31T15:12:02.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:02.947+0000 7fad72a00640 1 -- 172.21.15.16:0/2414110618 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 74 v74) ==== 94+0+5184 (secure 0 0 0) 0x7fad74057900 con 0x7fad7c1450e0 2024-03-31T15:12:02.939 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:12:02.940 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":74,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6494,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/2445558494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6835","nonce":2445558494},{"type":"v1","addr":"172.21.15.89:6837","nonce":2445558494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":56},{"gid":7270,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/3185913858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6833","nonce":3185913858},{"type":"v1","addr":"172.21.15.16:6835","nonce":3185913858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":56},{"gid":7279,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6836/1599468493","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":1599468493},{"type":"v1","addr":"172.21.15.89:6836","nonce":1599468493}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":56}],"filesystems":[{"mdsmap":{"epoch":74,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-31T15:09:50.926196+0000","modified":"2024-03-31T15:11:23.417783+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":85,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8085},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8085":{"gid":8085,"name":"c","rank":0,"incarnation":71,"state":"up:active","state_seq":4,"addr":"172.21.15.16:6834/3780866829","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":3780866829},{"type":"v1","addr":"172.21.15.16:6834","nonce":3780866829}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8085,"qdb_cluster":[8085]},"id":6}]} 2024-03-31T15:12:02.940 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 74 2024-03-31T15:12:02.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:02.947+0000 7fad81800640 1 -- 172.21.15.16:0/2414110618 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fad58068380 msgr2=0x7fad5806a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:02.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:02.947+0000 7fad81800640 1 --2- 172.21.15.16:0/2414110618 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fad58068380 0x7fad5806a840 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7fad6c007fd0 tx=0x7fad6c009040 comp rx=0 tx=0).stop 2024-03-31T15:12:02.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:02.947+0000 7fad81800640 1 -- 172.21.15.16:0/2414110618 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fad7c1450e0 msgr2=0x7fad7c0b3ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:02.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:02.947+0000 7fad81800640 1 --2- 172.21.15.16:0/2414110618 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fad7c1450e0 0x7fad7c0b3ff0 secure :-1 s=READY pgs=1333 cs=0 l=1 rev1=1 crypto rx=0x7fad74009e60 tx=0x7fad7400c490 comp rx=0 tx=0).stop 2024-03-31T15:12:02.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:02.947+0000 7fad81800640 1 -- 172.21.15.16:0/2414110618 shutdown_connections 2024-03-31T15:12:02.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:02.947+0000 7fad81800640 1 --2- 172.21.15.16:0/2414110618 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fad58068380 0x7fad5806a840 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:02.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:02.947+0000 7fad81800640 1 --2- 172.21.15.16:0/2414110618 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fad7c1450e0 0x7fad7c0b3ff0 unknown :-1 s=CLOSED pgs=1333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:02.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:02.947+0000 7fad81800640 1 --2- 172.21.15.16:0/2414110618 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fad7c072610 0x7fad7c0b23e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:02.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:02.947+0000 7fad81800640 1 --2- 172.21.15.16:0/2414110618 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fad7c06bd30 0x7fad7c0b5ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:02.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:02.947+0000 7fad81800640 1 -- 172.21.15.16:0/2414110618 >> 172.21.15.16:0/2414110618 conn(0x7fad7c0d1060 msgr2=0x7fad7c1436f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:12:02.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:02.947+0000 7fad81800640 1 -- 172.21.15.16:0/2414110618 shutdown_connections 2024-03-31T15:12:02.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:02.947+0000 7fad81800640 1 -- 172.21.15.16:0/2414110618 wait complete. 2024-03-31T15:12:02.961 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-31T15:12:03.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:03.107+0000 7f0d5be00640 1 Processor -- start 2024-03-31T15:12:03.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:03.107+0000 7f0d5be00640 1 -- start start 2024-03-31T15:12:03.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:03.111+0000 7f0d5be00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f0d5415f650 0x7f0d5406f920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:03.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:03.111+0000 7f0d5be00640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0d540cbe90 con 0x7f0d5406dbc0 2024-03-31T15:12:03.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:03.111+0000 7f0d5be00640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0d540cc170 con 0x7f0d5406fe60 2024-03-31T15:12:03.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:03.111+0000 7f0d5be00640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f0d540702d0 con 0x7f0d5415f650 2024-03-31T15:12:03.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:03.111+0000 7f0d59400640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f0d5406dbc0 0x7f0d540cb440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:35104/0 (socket says 172.21.15.16:35104) 2024-03-31T15:12:03.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:03.111+0000 7f0d59400640 1 -- 172.21.15.16:0/2420837466 learned_addr learned my addr 172.21.15.16:0/2420837466 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:12:03.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:03.111+0000 7f0d58a00640 1 --2- 172.21.15.16:0/2420837466 >> v2:172.21.15.16:3300/0 conn(0x7f0d5415f650 0x7f0d5406f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:03.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:03.111+0000 7f0d53e00640 1 -- 172.21.15.16:0/2420837466 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 900213623 0 0) 0x7f0d540cbe90 con 0x7f0d5406dbc0 2024-03-31T15:12:03.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:03.111+0000 7f0d53e00640 1 -- 172.21.15.16:0/2420837466 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0d3c003530 con 0x7f0d5406dbc0 2024-03-31T15:12:03.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:03.111+0000 7f0d58a00640 1 -- 172.21.15.16:0/2420837466 >> v1:172.21.15.89:6789/0 conn(0x7f0d5406fe60 legacy=0x7f0d54162180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:03.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:03.111+0000 7f0d58a00640 1 -- 172.21.15.16:0/2420837466 >> v1:172.21.15.16:6789/0 conn(0x7f0d5406dbc0 legacy=0x7f0d540cb440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:03.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:03.111+0000 7f0d58a00640 1 -- 172.21.15.16:0/2420837466 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d5415ee50 con 0x7f0d5415f650 2024-03-31T15:12:03.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:03.111+0000 7f0d58a00640 1 --2- 172.21.15.16:0/2420837466 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0d5415f650 0x7f0d5406f920 secure :-1 s=READY pgs=1335 cs=0 l=1 rev1=1 crypto rx=0x7f0d400097c0 tx=0x7f0d4001c410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e59f9b3a0c954a70 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:12:03.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:03.115+0000 7f0d53e00640 1 -- 172.21.15.16:0/2420837466 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d400b0070 con 0x7f0d5415f650 2024-03-31T15:12:03.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:03.115+0000 7f0d53e00640 1 -- 172.21.15.16:0/2420837466 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0d400ab600 con 0x7f0d5415f650 2024-03-31T15:12:03.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:03.115+0000 7f0d53e00640 1 -- 172.21.15.16:0/2420837466 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d400ab9b0 con 0x7f0d5415f650 2024-03-31T15:12:03.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:03.115+0000 7f0d5be00640 1 -- 172.21.15.16:0/2420837466 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0d5415f650 msgr2=0x7f0d5406f920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:03.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:03.115+0000 7f0d5be00640 1 --2- 172.21.15.16:0/2420837466 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0d5415f650 0x7f0d5406f920 secure :-1 s=READY pgs=1335 cs=0 l=1 rev1=1 crypto rx=0x7f0d400097c0 tx=0x7f0d4001c410 comp rx=0 tx=0).stop 2024-03-31T15:12:03.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:03.115+0000 7f0d5be00640 1 -- 172.21.15.16:0/2420837466 shutdown_connections 2024-03-31T15:12:03.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:03.115+0000 7f0d5be00640 1 --2- 172.21.15.16:0/2420837466 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0d5415f650 0x7f0d5406f920 unknown :-1 s=CLOSED pgs=1335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:03.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:03.115+0000 7f0d5be00640 1 -- 172.21.15.16:0/2420837466 >> 172.21.15.16:0/2420837466 conn(0x7f0d54067330 msgr2=0x7f0d54067740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:12:03.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:03.115+0000 7f0d5be00640 1 -- 172.21.15.16:0/2420837466 shutdown_connections 2024-03-31T15:12:03.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:03.115+0000 7f0d5be00640 1 -- 172.21.15.16:0/2420837466 wait complete. 2024-03-31T15:12:03.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:03.115+0000 7f0d5be00640 1 Processor -- start 2024-03-31T15:12:03.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:03.115+0000 7f0d5be00640 1 -- start start 2024-03-31T15:12:03.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:03.115+0000 7f0d5be00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0d5406dbc0 0x7f0d5413f460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:03.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:03.115+0000 7f0d5be00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0d5406fe60 0x7f0d5415abb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:03.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:03.115+0000 7f0d5be00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f0d5415f650 0x7f0d540b4e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:03.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:03.115+0000 7f0d5be00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f0d54162ca0 con 0x7f0d5406fe60 2024-03-31T15:12:03.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:03.115+0000 7f0d5be00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f0d5406dfa0 con 0x7f0d5406dbc0 2024-03-31T15:12:03.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:03.115+0000 7f0d5be00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f0d5406c950 con 0x7f0d5415f650 2024-03-31T15:12:03.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:03.115+0000 7f0d58a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0d5406fe60 0x7f0d5415abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:03.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:03.115+0000 7f0d58a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0d5406fe60 0x7f0d5415abb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49006/0 (socket says 172.21.15.16:49006) 2024-03-31T15:12:03.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:03.115+0000 7f0d58a00640 1 -- 172.21.15.16:0/406949321 learned_addr learned my addr 172.21.15.16:0/406949321 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:12:03.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:03.115+0000 7f0d59e00640 1 --2- 172.21.15.16:0/406949321 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f0d5415f650 0x7f0d540b4e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:03.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:03.115+0000 7f0d59400640 1 --2- 172.21.15.16:0/406949321 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0d5406dbc0 0x7f0d5413f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:03.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:03.119+0000 7f0d58a00640 1 -- 172.21.15.16:0/406949321 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f0d5415f650 msgr2=0x7f0d540b4e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:03.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:03.119+0000 7f0d58a00640 1 --2- 172.21.15.16:0/406949321 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f0d5415f650 0x7f0d540b4e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:03.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:03.119+0000 7f0d58a00640 1 -- 172.21.15.16:0/406949321 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0d5406dbc0 msgr2=0x7f0d5413f460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:03.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:03.119+0000 7f0d58a00640 1 --2- 172.21.15.16:0/406949321 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0d5406dbc0 0x7f0d5413f460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:03.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:03.119+0000 7f0d58a00640 1 -- 172.21.15.16:0/406949321 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d5413f9a0 con 0x7f0d5406fe60 2024-03-31T15:12:03.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:03.119+0000 7f0d59400640 1 --2- 172.21.15.16:0/406949321 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0d5406dbc0 0x7f0d5413f460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:12:03.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:03.119+0000 7f0d58a00640 1 --2- 172.21.15.16:0/406949321 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0d5406fe60 0x7f0d5415abb0 secure :-1 s=READY pgs=1336 cs=0 l=1 rev1=1 crypto rx=0x7f0d400ae040 tx=0x7f0d400191b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:12:03.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:03.119+0000 7f0d51600640 1 -- 172.21.15.16:0/406949321 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d400b0040 con 0x7f0d5406fe60 2024-03-31T15:12:03.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:03.119+0000 7f0d51600640 1 -- 172.21.15.16:0/406949321 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0d40004030 con 0x7f0d5406fe60 2024-03-31T15:12:03.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:03.119+0000 7f0d5be00640 1 -- 172.21.15.16:0/406949321 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0d5415b490 con 0x7f0d5406fe60 2024-03-31T15:12:03.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:03.119+0000 7f0d51600640 1 -- 172.21.15.16:0/406949321 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d40006e20 con 0x7f0d5406fe60 2024-03-31T15:12:03.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:03.119+0000 7f0d5be00640 1 -- 172.21.15.16:0/406949321 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0d5415ba50 con 0x7f0d5406fe60 2024-03-31T15:12:03.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:03.119+0000 7f0d5be00640 1 -- 172.21.15.16:0/406949321 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0d54155350 con 0x7f0d5406fe60 2024-03-31T15:12:03.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:03.119+0000 7f0d51600640 1 -- 172.21.15.16:0/406949321 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f0d400b3070 con 0x7f0d5406fe60 2024-03-31T15:12:03.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:03.119+0000 7f0d51600640 1 --2- 172.21.15.16:0/406949321 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f0d34068380 0x7f0d3406a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:03.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:03.119+0000 7f0d51600640 1 -- 172.21.15.16:0/406949321 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 5910+0+0 (secure 0 0 0) 0x7f0d40120b40 con 0x7f0d5406fe60 2024-03-31T15:12:03.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:03.123+0000 7f0d59400640 1 --2- 172.21.15.16:0/406949321 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f0d34068380 0x7f0d3406a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:03.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:03.123+0000 7f0d59400640 1 --2- 172.21.15.16:0/406949321 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f0d34068380 0x7f0d3406a840 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f0d480011f0 tx=0x7f0d48009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:12:03.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:03.131+0000 7f0d51600640 1 -- 172.21.15.16:0/406949321 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f0d400ba020 con 0x7f0d5406fe60 2024-03-31T15:12:03.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:03.299+0000 7f0d5be00640 1 -- 172.21.15.16:0/406949321 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f0d54069950 con 0x7f0d5406fe60 2024-03-31T15:12:03.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:03.303+0000 7f0d51600640 1 -- 172.21.15.16:0/406949321 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v85) ==== 74+0+16894 (secure 0 0 0) 0x7f0d40016090 con 0x7f0d5406fe60 2024-03-31T15:12:03.295 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:12:03.295 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":85,"fsid":"286eb5da-fb47-4550-b290-d9519c13b9a5","created":"2024-03-31T15:03:11.438952+0000","modified":"2024-03-31T15:11:19.402716+0000","last_up_change":"2024-03-31T15:03:22.885535+0000","last_in_change":"2024-03-31T15:03:13.941667+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-31T15:03:22.550787+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-03-31T15:09:49.322620+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-03-31T15:09:49.877778+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"92a66450-ef19-40d8-ac3d-d760b23b0bda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6801","nonce":932712244}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6803","nonce":932712244}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6807","nonce":932712244}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6805","nonce":932712244}]},"public_addr":"172.21.15.16:6801/932712244","cluster_addr":"172.21.15.16:6803/932712244","heartbeat_back_addr":"172.21.15.16:6807/932712244","heartbeat_front_addr":"172.21.15.16:6805/932712244","state":["exists","up"]},{"osd":1,"uuid":"da9d1f2b-1968-4959-b843-faa231988e92","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6810","nonce":1398737533}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6811","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6813","nonce":1398737533}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6821","nonce":1398737533}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6815","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6816","nonce":1398737533}]},"public_addr":"172.21.15.16:6810/1398737533","cluster_addr":"172.21.15.16:6813/1398737533","heartbeat_back_addr":"172.21.15.16:6821/1398737533","heartbeat_front_addr":"172.21.15.16:6816/1398737533","state":["exists","up"]},{"osd":2,"uuid":"01a73662-9a21-4b1f-bcbb-42b9636a4638","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6819","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6822","nonce":3018526646}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6825","nonce":3018526646}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6831","nonce":3018526646}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6827","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6829","nonce":3018526646}]},"public_addr":"172.21.15.16:6822/3018526646","cluster_addr":"172.21.15.16:6825/3018526646","heartbeat_back_addr":"172.21.15.16:6831/3018526646","heartbeat_front_addr":"172.21.15.16:6829/3018526646","state":["exists","up"]},{"osd":3,"uuid":"c7771fd3-e18e-4500-ac38-23cf10ebcd75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6809","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6812","nonce":3360352974}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6817","nonce":3360352974}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6828","nonce":3360352974}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6823","nonce":3360352974}]},"public_addr":"172.21.15.16:6812/3360352974","cluster_addr":"172.21.15.16:6817/3360352974","heartbeat_back_addr":"172.21.15.16:6828/3360352974","heartbeat_front_addr":"172.21.15.16:6823/3360352974","state":["exists","up"]},{"osd":4,"uuid":"88260730-1482-4fbd-a40f-cb1124e9df4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6801","nonce":1856844856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6803","nonce":1856844856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6807","nonce":1856844856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6805","nonce":1856844856}]},"public_addr":"172.21.15.89:6801/1856844856","cluster_addr":"172.21.15.89:6803/1856844856","heartbeat_back_addr":"172.21.15.89:6807/1856844856","heartbeat_front_addr":"172.21.15.89:6805/1856844856","state":["exists","up"]},{"osd":5,"uuid":"4cb2c00c-fae0-4dd8-9d27-ec4f0093ccf7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6814","nonce":50395527}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6817","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6820","nonce":50395527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6831","nonce":50395527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6823","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6826","nonce":50395527}]},"public_addr":"172.21.15.89:6814/50395527","cluster_addr":"172.21.15.89:6820/50395527","heartbeat_back_addr":"172.21.15.89:6831/50395527","heartbeat_front_addr":"172.21.15.89:6826/50395527","state":["exists","up"]},{"osd":6,"uuid":"81ce0c19-2554-4cae-acbe-66a05e9e9b9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6809","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6812","nonce":3470471752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6815","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6818","nonce":3470471752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6827","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6829","nonce":3470471752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6821","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6824","nonce":3470471752}]},"public_addr":"172.21.15.89:6812/3470471752","cluster_addr":"172.21.15.89:6818/3470471752","heartbeat_back_addr":"172.21.15.89:6829/3470471752","heartbeat_front_addr":"172.21.15.89:6824/3470471752","state":["exists","up"]},{"osd":7,"uuid":"85e4313b-d37d-442e-8042-153bf20a847d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6811","nonce":1796778630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6813","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6816","nonce":1796778630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6825","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6828","nonce":1796778630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6819","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6822","nonce":1796778630}]},"public_addr":"172.21.15.89:6811/1796778630","cluster_addr":"172.21.15.89:6816/1796778630","heartbeat_back_addr":"172.21.15.89:6828/1796778630","heartbeat_front_addr":"172.21.15.89:6822/1796778630","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.439088+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.529970+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.680512+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:21.044161+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.582111+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.536558+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.409522+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.590229+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"13.d","mappings":[{"from":1,"to":3}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:6832/818074038":"2024-04-01T15:11:19.402687+0000","172.21.15.16:6834/818074038":"2024-04-01T15:11:19.402687+0000","172.21.15.16:6834/2478530200":"2024-04-01T15:10:04.360772+0000","172.21.15.16:6832/2478530200":"2024-04-01T15:10:04.360772+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-31T15:12:03.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:03.303+0000 7f0d5be00640 1 -- 172.21.15.16:0/406949321 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f0d34068380 msgr2=0x7f0d3406a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:03.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:03.303+0000 7f0d5be00640 1 --2- 172.21.15.16:0/406949321 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f0d34068380 0x7f0d3406a840 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f0d480011f0 tx=0x7f0d48009040 comp rx=0 tx=0).stop 2024-03-31T15:12:03.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:03.303+0000 7f0d5be00640 1 -- 172.21.15.16:0/406949321 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0d5406fe60 msgr2=0x7f0d5415abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:03.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:03.303+0000 7f0d5be00640 1 --2- 172.21.15.16:0/406949321 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0d5406fe60 0x7f0d5415abb0 secure :-1 s=READY pgs=1336 cs=0 l=1 rev1=1 crypto rx=0x7f0d400ae040 tx=0x7f0d400191b0 comp rx=0 tx=0).stop 2024-03-31T15:12:03.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:03.303+0000 7f0d5be00640 1 -- 172.21.15.16:0/406949321 shutdown_connections 2024-03-31T15:12:03.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:03.303+0000 7f0d5be00640 1 --2- 172.21.15.16:0/406949321 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f0d34068380 0x7f0d3406a840 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:03.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:03.303+0000 7f0d5be00640 1 --2- 172.21.15.16:0/406949321 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f0d5415f650 0x7f0d540b4e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:03.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:03.303+0000 7f0d5be00640 1 --2- 172.21.15.16:0/406949321 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0d5406fe60 0x7f0d5415abb0 unknown :-1 s=CLOSED pgs=1336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:03.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:03.303+0000 7f0d5be00640 1 --2- 172.21.15.16:0/406949321 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0d5406dbc0 0x7f0d5413f460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:03.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:03.303+0000 7f0d5be00640 1 -- 172.21.15.16:0/406949321 >> 172.21.15.16:0/406949321 conn(0x7f0d54067330 msgr2=0x7f0d5415e140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:12:03.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:03.303+0000 7f0d5be00640 1 -- 172.21.15.16:0/406949321 shutdown_connections 2024-03-31T15:12:03.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:03.303+0000 7f0d5be00640 1 -- 172.21.15.16:0/406949321 wait complete. 2024-03-31T15:12:03.317 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-31T15:12:03.318 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-31T15:12:03.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:03.467+0000 7f295be00640 1 Processor -- start 2024-03-31T15:12:03.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:03.467+0000 7f295be00640 1 -- start start 2024-03-31T15:12:03.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:03.467+0000 7f295be00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f295406f920 0x7f295406fd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:03.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:03.467+0000 7f295be00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f29540d2010 0x7f29540d24a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:03.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:03.467+0000 7f295be00640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f29540d0920 con 0x7f2954145e40 2024-03-31T15:12:03.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:03.467+0000 7f295be00640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f29540702c0 con 0x7f295406f920 2024-03-31T15:12:03.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:03.467+0000 7f295be00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f29540d06d0 con 0x7f29540d2010 2024-03-31T15:12:03.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:03.467+0000 7f2958a00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f295406f920 0x7f295406fd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:03.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:03.467+0000 7f2959400640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f2954145e40 0x7f29540cc740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:35108/0 (socket says 172.21.15.16:35108) 2024-03-31T15:12:03.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:03.467+0000 7f2959400640 1 -- 172.21.15.16:0/4033314034 learned_addr learned my addr 172.21.15.16:0/4033314034 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:12:03.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:03.467+0000 7f2959e00640 1 --2- 172.21.15.16:0/4033314034 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f29540d2010 0x7f29540d24a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:03.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:03.467+0000 7f2953e00640 1 -- 172.21.15.16:0/4033314034 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 892746205 0 0) 0x7f29540d0920 con 0x7f2954145e40 2024-03-31T15:12:03.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:03.467+0000 7f2953e00640 1 -- 172.21.15.16:0/4033314034 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f293c003530 con 0x7f2954145e40 2024-03-31T15:12:03.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:03.467+0000 7f2958a00640 1 -- 172.21.15.16:0/4033314034 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f29540d2010 msgr2=0x7f29540d24a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:12:03.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:03.467+0000 7f2958a00640 1 --2- 172.21.15.16:0/4033314034 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f29540d2010 0x7f29540d24a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:03.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:03.467+0000 7f2958a00640 1 -- 172.21.15.16:0/4033314034 >> v1:172.21.15.16:6789/0 conn(0x7f2954145e40 legacy=0x7f29540cc740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:03.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:03.467+0000 7f2958a00640 1 -- 172.21.15.16:0/4033314034 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29540d0c00 con 0x7f295406f920 2024-03-31T15:12:03.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:03.467+0000 7f2959e00640 1 --2- 172.21.15.16:0/4033314034 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f29540d2010 0x7f29540d24a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:12:03.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:03.471+0000 7f2958a00640 1 --2- 172.21.15.16:0/4033314034 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f295406f920 0x7f295406fd80 secure :-1 s=READY pgs=1338 cs=0 l=1 rev1=1 crypto rx=0x7f29400097c0 tx=0x7f294001c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=813acf48731412e9 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:12:03.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:03.471+0000 7f2953e00640 1 -- 172.21.15.16:0/4033314034 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29400b2070 con 0x7f295406f920 2024-03-31T15:12:03.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:03.471+0000 7f2953e00640 1 -- 172.21.15.16:0/4033314034 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f29400ad640 con 0x7f295406f920 2024-03-31T15:12:03.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:03.471+0000 7f2953e00640 1 -- 172.21.15.16:0/4033314034 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29400ada10 con 0x7f295406f920 2024-03-31T15:12:03.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:03.471+0000 7f295be00640 1 -- 172.21.15.16:0/4033314034 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f295406f920 msgr2=0x7f295406fd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:03.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:03.471+0000 7f295be00640 1 --2- 172.21.15.16:0/4033314034 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f295406f920 0x7f295406fd80 secure :-1 s=READY pgs=1338 cs=0 l=1 rev1=1 crypto rx=0x7f29400097c0 tx=0x7f294001c4a0 comp rx=0 tx=0).stop 2024-03-31T15:12:03.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:03.471+0000 7f295be00640 1 -- 172.21.15.16:0/4033314034 shutdown_connections 2024-03-31T15:12:03.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:03.471+0000 7f295be00640 1 --2- 172.21.15.16:0/4033314034 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f29540d2010 0x7f29540d24a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:03.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:03.471+0000 7f295be00640 1 --2- 172.21.15.16:0/4033314034 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f295406f920 0x7f295406fd80 unknown :-1 s=CLOSED pgs=1338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:03.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:03.471+0000 7f295be00640 1 -- 172.21.15.16:0/4033314034 >> 172.21.15.16:0/4033314034 conn(0x7f2954067330 msgr2=0x7f2954067740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:12:03.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:03.471+0000 7f295be00640 1 -- 172.21.15.16:0/4033314034 shutdown_connections 2024-03-31T15:12:03.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:03.471+0000 7f295be00640 1 -- 172.21.15.16:0/4033314034 wait complete. 2024-03-31T15:12:03.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:03.471+0000 7f295be00640 1 Processor -- start 2024-03-31T15:12:03.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:03.475+0000 7f295be00640 1 -- start start 2024-03-31T15:12:03.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:03.475+0000 7f295be00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f295406f920 0x7f2954157d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:03.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:03.475+0000 7f295be00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f29540d2010 0x7f295413d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:03.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:03.475+0000 7f295be00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2954145e40 0x7f295413db00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:03.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:03.475+0000 7f295be00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f29540d0ee0 con 0x7f29540d2010 2024-03-31T15:12:03.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:03.475+0000 7f295be00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f29540ce9e0 con 0x7f2954145e40 2024-03-31T15:12:03.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:03.475+0000 7f295be00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f29540d2df0 con 0x7f295406f920 2024-03-31T15:12:03.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:03.475+0000 7f2958a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f29540d2010 0x7f295413d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:03.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:03.475+0000 7f2958a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f29540d2010 0x7f295413d5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49014/0 (socket says 172.21.15.16:49014) 2024-03-31T15:12:03.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:03.475+0000 7f2958a00640 1 -- 172.21.15.16:0/1948848357 learned_addr learned my addr 172.21.15.16:0/1948848357 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:12:03.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:03.475+0000 7f2959400640 1 --2- 172.21.15.16:0/1948848357 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f295406f920 0x7f2954157d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:03.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:03.475+0000 7f2959e00640 1 --2- 172.21.15.16:0/1948848357 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2954145e40 0x7f295413db00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:03.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:03.475+0000 7f2958a00640 1 -- 172.21.15.16:0/1948848357 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f295406f920 msgr2=0x7f2954157d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:03.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:03.475+0000 7f2958a00640 1 --2- 172.21.15.16:0/1948848357 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f295406f920 0x7f2954157d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:03.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:03.475+0000 7f2958a00640 1 -- 172.21.15.16:0/1948848357 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2954145e40 msgr2=0x7f295413db00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:03.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:03.475+0000 7f2958a00640 1 --2- 172.21.15.16:0/1948848357 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2954145e40 0x7f295413db00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:03.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:03.475+0000 7f2958a00640 1 -- 172.21.15.16:0/1948848357 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f295404f9f0 con 0x7f29540d2010 2024-03-31T15:12:03.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:03.475+0000 7f2958a00640 1 --2- 172.21.15.16:0/1948848357 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f29540d2010 0x7f295413d5c0 secure :-1 s=READY pgs=1339 cs=0 l=1 rev1=1 crypto rx=0x7f294001a0d0 tx=0x7f2940019240 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:12:03.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:03.475+0000 7f2951600640 1 -- 172.21.15.16:0/1948848357 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29400ba020 con 0x7f29540d2010 2024-03-31T15:12:03.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:03.475+0000 7f2951600640 1 -- 172.21.15.16:0/1948848357 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f29400b6410 con 0x7f29540d2010 2024-03-31T15:12:03.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:03.475+0000 7f2951600640 1 -- 172.21.15.16:0/1948848357 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29400b2040 con 0x7f29540d2010 2024-03-31T15:12:03.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:03.475+0000 7f295be00640 1 -- 172.21.15.16:0/1948848357 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2954141740 con 0x7f29540d2010 2024-03-31T15:12:03.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:03.475+0000 7f295be00640 1 -- 172.21.15.16:0/1948848357 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2954141d00 con 0x7f29540d2010 2024-03-31T15:12:03.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:03.475+0000 7f295be00640 1 -- 172.21.15.16:0/1948848357 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2954069950 con 0x7f29540d2010 2024-03-31T15:12:03.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:03.479+0000 7f2951600640 1 -- 172.21.15.16:0/1948848357 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f29400bd020 con 0x7f29540d2010 2024-03-31T15:12:03.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:03.479+0000 7f2951600640 1 --2- 172.21.15.16:0/1948848357 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f2934068380 0x7f293406a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:03.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:03.479+0000 7f2951600640 1 -- 172.21.15.16:0/1948848357 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 5910+0+0 (secure 0 0 0) 0x7f2940120800 con 0x7f29540d2010 2024-03-31T15:12:03.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:03.479+0000 7f2959400640 1 --2- 172.21.15.16:0/1948848357 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f2934068380 0x7f293406a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:03.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:03.479+0000 7f2959400640 1 --2- 172.21.15.16:0/1948848357 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f2934068380 0x7f293406a840 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f2948007fd0 tx=0x7f2948009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:12:03.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:03.487+0000 7f2951600640 1 -- 172.21.15.16:0/1948848357 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f29400c4030 con 0x7f29540d2010 2024-03-31T15:12:03.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:03.687+0000 7f295be00640 1 -- 172.21.15.16:0/1948848357 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f29540cfa10 con 0x7f29540d2010 2024-03-31T15:12:03.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:03.687+0000 7f2951600640 1 -- 172.21.15.16:0/1948848357 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 74 v74) ==== 94+0+5184 (secure 0 0 0) 0x7f2940016090 con 0x7f29540d2010 2024-03-31T15:12:03.680 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:12:03.680 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":74,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6494,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/2445558494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6835","nonce":2445558494},{"type":"v1","addr":"172.21.15.89:6837","nonce":2445558494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":56},{"gid":7270,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/3185913858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6833","nonce":3185913858},{"type":"v1","addr":"172.21.15.16:6835","nonce":3185913858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":56},{"gid":7279,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6836/1599468493","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":1599468493},{"type":"v1","addr":"172.21.15.89:6836","nonce":1599468493}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":56}],"filesystems":[{"mdsmap":{"epoch":74,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-31T15:09:50.926196+0000","modified":"2024-03-31T15:11:23.417783+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":85,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8085},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8085":{"gid":8085,"name":"c","rank":0,"incarnation":71,"state":"up:active","state_seq":4,"addr":"172.21.15.16:6834/3780866829","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":3780866829},{"type":"v1","addr":"172.21.15.16:6834","nonce":3780866829}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8085,"qdb_cluster":[8085]},"id":6}]} 2024-03-31T15:12:03.681 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 74 2024-03-31T15:12:03.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:03.691+0000 7f295be00640 1 -- 172.21.15.16:0/1948848357 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f2934068380 msgr2=0x7f293406a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:03.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:03.691+0000 7f295be00640 1 --2- 172.21.15.16:0/1948848357 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f2934068380 0x7f293406a840 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f2948007fd0 tx=0x7f2948009040 comp rx=0 tx=0).stop 2024-03-31T15:12:03.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:03.691+0000 7f295be00640 1 -- 172.21.15.16:0/1948848357 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f29540d2010 msgr2=0x7f295413d5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:03.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:03.691+0000 7f295be00640 1 --2- 172.21.15.16:0/1948848357 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f29540d2010 0x7f295413d5c0 secure :-1 s=READY pgs=1339 cs=0 l=1 rev1=1 crypto rx=0x7f294001a0d0 tx=0x7f2940019240 comp rx=0 tx=0).stop 2024-03-31T15:12:03.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:03.691+0000 7f295be00640 1 -- 172.21.15.16:0/1948848357 shutdown_connections 2024-03-31T15:12:03.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:03.691+0000 7f295be00640 1 --2- 172.21.15.16:0/1948848357 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f2934068380 0x7f293406a840 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:03.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:03.691+0000 7f295be00640 1 --2- 172.21.15.16:0/1948848357 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2954145e40 0x7f295413db00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:03.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:03.691+0000 7f295be00640 1 --2- 172.21.15.16:0/1948848357 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f29540d2010 0x7f295413d5c0 unknown :-1 s=CLOSED pgs=1339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:03.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:03.691+0000 7f295be00640 1 --2- 172.21.15.16:0/1948848357 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f295406f920 0x7f2954157d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:03.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:03.691+0000 7f295be00640 1 -- 172.21.15.16:0/1948848357 >> 172.21.15.16:0/1948848357 conn(0x7f2954067330 msgr2=0x7f2954067710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:12:03.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:03.691+0000 7f295be00640 1 -- 172.21.15.16:0/1948848357 shutdown_connections 2024-03-31T15:12:03.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:03.691+0000 7f295be00640 1 -- 172.21.15.16:0/1948848357 wait complete. 2024-03-31T15:12:03.703 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-31T15:12:03.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:03.851+0000 7fd5bcc00640 1 Processor -- start 2024-03-31T15:12:03.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:03.851+0000 7fd5bcc00640 1 -- start start 2024-03-31T15:12:03.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:03.851+0000 7fd5bcc00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fd5b80b9c60 0x7fd5b80b7470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:03.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:03.851+0000 7fd5bcc00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fd5b80b79b0 0x7fd5b80b7e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:03.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:03.851+0000 7fd5bcc00640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd5b80d3990 con 0x7fd5b80d1280 2024-03-31T15:12:03.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:03.851+0000 7fd5bcc00640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7fd5b80d26a0 con 0x7fd5b80b79b0 2024-03-31T15:12:03.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:03.851+0000 7fd5bcc00640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7fd5b80d28f0 con 0x7fd5b80b9c60 2024-03-31T15:12:03.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:03.851+0000 7fd5b6800640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fd5b80b79b0 0x7fd5b80b7e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:03.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:03.851+0000 7fd5b6800640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fd5b80b79b0 0x7fd5b80b7e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49030/0 (socket says 172.21.15.16:49030) 2024-03-31T15:12:03.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:03.851+0000 7fd5b6800640 1 -- 172.21.15.16:0/396922060 learned_addr learned my addr 172.21.15.16:0/396922060 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:12:03.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:03.851+0000 7fd5afe00640 1 --2- 172.21.15.16:0/396922060 >> v2:172.21.15.89:3300/0 conn(0x7fd5b80b9c60 0x7fd5b80b7470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:03.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:03.851+0000 7fd5b6800640 1 -- 172.21.15.16:0/396922060 >> v2:172.21.15.89:3300/0 conn(0x7fd5b80b9c60 msgr2=0x7fd5b80b7470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:12:03.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:03.855+0000 7fd5b6800640 1 --2- 172.21.15.16:0/396922060 >> v2:172.21.15.89:3300/0 conn(0x7fd5b80b9c60 0x7fd5b80b7470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:03.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:03.855+0000 7fd5b6800640 1 -- 172.21.15.16:0/396922060 >> v1:172.21.15.89:6789/0 conn(0x7fd5b80d1280 legacy=0x7fd5b80d1660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:12:03.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:03.855+0000 7fd5b6800640 1 -- 172.21.15.16:0/396922060 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5b80d2bd0 con 0x7fd5b80b79b0 2024-03-31T15:12:03.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:03.855+0000 7fd5afe00640 1 --2- 172.21.15.16:0/396922060 >> v2:172.21.15.89:3300/0 conn(0x7fd5b80b9c60 0x7fd5b80b7470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:12:03.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:03.855+0000 7fd5b6800640 1 --2- 172.21.15.16:0/396922060 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd5b80b79b0 0x7fd5b80b7e60 secure :-1 s=READY pgs=1340 cs=0 l=1 rev1=1 crypto rx=0x7fd5b80b6df0 tx=0x7fd5b001c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a3d29d39a158945a server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:12:03.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:03.855+0000 7fd5b5400640 1 -- 172.21.15.16:0/396922060 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5b00a56d0 con 0x7fd5b80b79b0 2024-03-31T15:12:03.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:03.855+0000 7fd5b5400640 1 -- 172.21.15.16:0/396922060 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd5b00a5870 con 0x7fd5b80b79b0 2024-03-31T15:12:03.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:03.855+0000 7fd5b5400640 1 -- 172.21.15.16:0/396922060 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5b00a5c40 con 0x7fd5b80b79b0 2024-03-31T15:12:03.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:03.855+0000 7fd5bcc00640 1 -- 172.21.15.16:0/396922060 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd5b80b79b0 msgr2=0x7fd5b80b7e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:03.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:03.855+0000 7fd5bcc00640 1 --2- 172.21.15.16:0/396922060 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd5b80b79b0 0x7fd5b80b7e60 secure :-1 s=READY pgs=1340 cs=0 l=1 rev1=1 crypto rx=0x7fd5b80b6df0 tx=0x7fd5b001c920 comp rx=0 tx=0).stop 2024-03-31T15:12:03.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:03.855+0000 7fd5bcc00640 1 -- 172.21.15.16:0/396922060 shutdown_connections 2024-03-31T15:12:03.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:03.855+0000 7fd5bcc00640 1 --2- 172.21.15.16:0/396922060 >> v2:172.21.15.89:3300/0 conn(0x7fd5b80b9c60 0x7fd5b80b7470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:03.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:03.855+0000 7fd5bcc00640 1 --2- 172.21.15.16:0/396922060 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd5b80b79b0 0x7fd5b80b7e60 unknown :-1 s=CLOSED pgs=1340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:03.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:03.855+0000 7fd5bcc00640 1 -- 172.21.15.16:0/396922060 >> 172.21.15.16:0/396922060 conn(0x7fd5b8067730 msgr2=0x7fd5b8067b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:12:03.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:03.855+0000 7fd5bcc00640 1 -- 172.21.15.16:0/396922060 shutdown_connections 2024-03-31T15:12:03.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:03.855+0000 7fd5bcc00640 1 -- 172.21.15.16:0/396922060 wait complete. 2024-03-31T15:12:03.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:03.859+0000 7fd5bcc00640 1 Processor -- start 2024-03-31T15:12:03.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:03.859+0000 7fd5bcc00640 1 -- start start 2024-03-31T15:12:03.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:03.859+0000 7fd5bcc00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd5b80b79b0 0x7fd5b8156430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:03.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:03.859+0000 7fd5bcc00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd5b80b9c60 0x7fd5b80c8230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:03.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:03.859+0000 7fd5bcc00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fd5b80d1280 0x7fd5b80c8770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:03.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:03.859+0000 7fd5bcc00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fd5b80d4010 con 0x7fd5b80b79b0 2024-03-31T15:12:03.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:03.859+0000 7fd5bcc00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fd5b80d3e90 con 0x7fd5b80b9c60 2024-03-31T15:12:03.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:03.859+0000 7fd5bcc00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fd5b816b8a0 con 0x7fd5b80d1280 2024-03-31T15:12:03.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:03.859+0000 7fd5b5e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd5b80b79b0 0x7fd5b8156430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:03.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:03.859+0000 7fd5b5e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd5b80b79b0 0x7fd5b8156430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49040/0 (socket says 172.21.15.16:49040) 2024-03-31T15:12:03.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:03.859+0000 7fd5b5e00640 1 -- 172.21.15.16:0/3267193794 learned_addr learned my addr 172.21.15.16:0/3267193794 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:12:03.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:03.859+0000 7fd5afe00640 1 --2- 172.21.15.16:0/3267193794 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd5b80b9c60 0x7fd5b80c8230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:03.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:03.859+0000 7fd5b6800640 1 --2- 172.21.15.16:0/3267193794 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fd5b80d1280 0x7fd5b80c8770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:03.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:03.859+0000 7fd5b5e00640 1 -- 172.21.15.16:0/3267193794 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fd5b80d1280 msgr2=0x7fd5b80c8770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:03.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:03.859+0000 7fd5b5e00640 1 --2- 172.21.15.16:0/3267193794 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fd5b80d1280 0x7fd5b80c8770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:03.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:03.859+0000 7fd5b5e00640 1 -- 172.21.15.16:0/3267193794 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd5b80b9c60 msgr2=0x7fd5b80c8230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:03.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:03.859+0000 7fd5b5e00640 1 --2- 172.21.15.16:0/3267193794 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd5b80b9c60 0x7fd5b80c8230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:03.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:03.859+0000 7fd5b5e00640 1 -- 172.21.15.16:0/3267193794 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5b80cac40 con 0x7fd5b80b79b0 2024-03-31T15:12:03.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:03.859+0000 7fd5afe00640 1 --2- 172.21.15.16:0/3267193794 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd5b80b9c60 0x7fd5b80c8230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:12:03.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:03.859+0000 7fd5b5e00640 1 --2- 172.21.15.16:0/3267193794 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd5b80b79b0 0x7fd5b8156430 secure :-1 s=READY pgs=1341 cs=0 l=1 rev1=1 crypto rx=0x7fd5a800b440 tx=0x7fd5a800b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:12:03.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:03.859+0000 7fd5ae000640 1 -- 172.21.15.16:0/3267193794 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5a8015020 con 0x7fd5b80b79b0 2024-03-31T15:12:03.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:03.859+0000 7fd5ae000640 1 -- 172.21.15.16:0/3267193794 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd5a8011890 con 0x7fd5b80b79b0 2024-03-31T15:12:03.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:03.859+0000 7fd5bcc00640 1 -- 172.21.15.16:0/3267193794 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd5b80ca3a0 con 0x7fd5b80b79b0 2024-03-31T15:12:03.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:03.859+0000 7fd5ae000640 1 -- 172.21.15.16:0/3267193794 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5a8011c80 con 0x7fd5b80b79b0 2024-03-31T15:12:03.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:03.863+0000 7fd5bcc00640 1 -- 172.21.15.16:0/3267193794 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd5b80ca960 con 0x7fd5b80b79b0 2024-03-31T15:12:03.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:03.863+0000 7fd5bcc00640 1 -- 172.21.15.16:0/3267193794 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd5b8069950 con 0x7fd5b80b79b0 2024-03-31T15:12:03.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:03.863+0000 7fd5ae000640 1 -- 172.21.15.16:0/3267193794 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fd5a8018020 con 0x7fd5b80b79b0 2024-03-31T15:12:03.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:03.863+0000 7fd5ae000640 1 --2- 172.21.15.16:0/3267193794 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fd594068380 0x7fd59406a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:03.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:03.863+0000 7fd5ae000640 1 -- 172.21.15.16:0/3267193794 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 5910+0+0 (secure 0 0 0) 0x7fd5a8086920 con 0x7fd5b80b79b0 2024-03-31T15:12:03.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:03.863+0000 7fd5afe00640 1 --2- 172.21.15.16:0/3267193794 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fd594068380 0x7fd59406a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:03.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:03.863+0000 7fd5afe00640 1 --2- 172.21.15.16:0/3267193794 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fd594068380 0x7fd59406a840 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7fd5b815a5d0 tx=0x7fd5a0008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:12:03.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:03.871+0000 7fd5ae000640 1 -- 172.21.15.16:0/3267193794 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fd5a801f030 con 0x7fd5b80b79b0 2024-03-31T15:12:04.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:04.071+0000 7fd5bcc00640 1 -- 172.21.15.16:0/3267193794 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd5b80b6c70 con 0x7fd5b80b79b0 2024-03-31T15:12:04.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:04.075+0000 7fd5ae000640 1 -- 172.21.15.16:0/3267193794 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 74 v74) ==== 94+0+5184 (secure 0 0 0) 0x7fd5a800d040 con 0x7fd5b80b79b0 2024-03-31T15:12:04.067 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:12:04.067 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":74,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6494,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/2445558494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6835","nonce":2445558494},{"type":"v1","addr":"172.21.15.89:6837","nonce":2445558494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":56},{"gid":7270,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/3185913858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6833","nonce":3185913858},{"type":"v1","addr":"172.21.15.16:6835","nonce":3185913858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":56},{"gid":7279,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6836/1599468493","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":1599468493},{"type":"v1","addr":"172.21.15.89:6836","nonce":1599468493}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":56}],"filesystems":[{"mdsmap":{"epoch":74,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-31T15:09:50.926196+0000","modified":"2024-03-31T15:11:23.417783+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":85,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8085},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8085":{"gid":8085,"name":"c","rank":0,"incarnation":71,"state":"up:active","state_seq":4,"addr":"172.21.15.16:6834/3780866829","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":3780866829},{"type":"v1","addr":"172.21.15.16:6834","nonce":3780866829}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8085,"qdb_cluster":[8085]},"id":6}]} 2024-03-31T15:12:04.067 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 74 2024-03-31T15:12:04.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:04.075+0000 7fd5bcc00640 1 -- 172.21.15.16:0/3267193794 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fd594068380 msgr2=0x7fd59406a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:04.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:04.075+0000 7fd5bcc00640 1 --2- 172.21.15.16:0/3267193794 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fd594068380 0x7fd59406a840 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7fd5b815a5d0 tx=0x7fd5a0008040 comp rx=0 tx=0).stop 2024-03-31T15:12:04.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:04.075+0000 7fd5bcc00640 1 -- 172.21.15.16:0/3267193794 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd5b80b79b0 msgr2=0x7fd5b8156430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:04.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:04.075+0000 7fd5bcc00640 1 --2- 172.21.15.16:0/3267193794 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd5b80b79b0 0x7fd5b8156430 secure :-1 s=READY pgs=1341 cs=0 l=1 rev1=1 crypto rx=0x7fd5a800b440 tx=0x7fd5a800b900 comp rx=0 tx=0).stop 2024-03-31T15:12:04.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:04.075+0000 7fd5bcc00640 1 -- 172.21.15.16:0/3267193794 shutdown_connections 2024-03-31T15:12:04.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:04.075+0000 7fd5bcc00640 1 --2- 172.21.15.16:0/3267193794 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fd594068380 0x7fd59406a840 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:04.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:04.075+0000 7fd5bcc00640 1 --2- 172.21.15.16:0/3267193794 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fd5b80d1280 0x7fd5b80c8770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:04.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:04.075+0000 7fd5bcc00640 1 --2- 172.21.15.16:0/3267193794 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd5b80b9c60 0x7fd5b80c8230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:04.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:04.075+0000 7fd5bcc00640 1 --2- 172.21.15.16:0/3267193794 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd5b80b79b0 0x7fd5b8156430 unknown :-1 s=CLOSED pgs=1341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:04.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:04.075+0000 7fd5bcc00640 1 -- 172.21.15.16:0/3267193794 >> 172.21.15.16:0/3267193794 conn(0x7fd5b8067730 msgr2=0x7fd5b80d41c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:12:04.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:04.075+0000 7fd5bcc00640 1 -- 172.21.15.16:0/3267193794 shutdown_connections 2024-03-31T15:12:04.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:04.075+0000 7fd5bcc00640 1 -- 172.21.15.16:0/3267193794 wait complete. 2024-03-31T15:12:04.088 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-31T15:12:04.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:04.243+0000 7f8f31800640 1 Processor -- start 2024-03-31T15:12:04.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:04.243+0000 7f8f31800640 1 -- start start 2024-03-31T15:12:04.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:04.247+0000 7f8f31800640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f8f2c15f650 0x7f8f2c06f920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:04.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:04.247+0000 7f8f31800640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f8f2c06fe60 0x7f8f2c162180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:04.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:04.247+0000 7f8f31800640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8f2c0cbe90 con 0x7f8f2c06dbc0 2024-03-31T15:12:04.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:04.247+0000 7f8f31800640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f8f2c0702d0 con 0x7f8f2c06fe60 2024-03-31T15:12:04.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:04.247+0000 7f8f31800640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f8f2c15f1c0 con 0x7f8f2c15f650 2024-03-31T15:12:04.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:04.247+0000 7f8f2aa00640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f8f2c06dbc0 0x7f8f2c0cb440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:35112/0 (socket says 172.21.15.16:35112) 2024-03-31T15:12:04.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:04.247+0000 7f8f2aa00640 1 -- 172.21.15.16:0/3570411049 learned_addr learned my addr 172.21.15.16:0/3570411049 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:12:04.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:04.247+0000 7f8f2b400640 1 --2- 172.21.15.16:0/3570411049 >> v2:172.21.15.16:3300/0 conn(0x7f8f2c06fe60 0x7f8f2c162180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:04.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:04.247+0000 7f8f2a000640 1 --2- 172.21.15.16:0/3570411049 >> v2:172.21.15.89:3300/0 conn(0x7f8f2c15f650 0x7f8f2c06f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:04.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:04.247+0000 7f8f29600640 1 -- 172.21.15.16:0/3570411049 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2159294809 0 0) 0x7f8f2c0cbe90 con 0x7f8f2c06dbc0 2024-03-31T15:12:04.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:04.247+0000 7f8f29600640 1 -- 172.21.15.16:0/3570411049 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8f10003530 con 0x7f8f2c06dbc0 2024-03-31T15:12:04.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:04.247+0000 7f8f2b400640 1 -- 172.21.15.16:0/3570411049 >> v2:172.21.15.89:3300/0 conn(0x7f8f2c15f650 msgr2=0x7f8f2c06f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:12:04.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:04.247+0000 7f8f2b400640 1 --2- 172.21.15.16:0/3570411049 >> v2:172.21.15.89:3300/0 conn(0x7f8f2c15f650 0x7f8f2c06f920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:04.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:04.247+0000 7f8f2b400640 1 -- 172.21.15.16:0/3570411049 >> v1:172.21.15.16:6789/0 conn(0x7f8f2c06dbc0 legacy=0x7f8f2c0cb440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:04.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:04.247+0000 7f8f2b400640 1 -- 172.21.15.16:0/3570411049 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f2c0cc170 con 0x7f8f2c06fe60 2024-03-31T15:12:04.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:04.251+0000 7f8f2b400640 1 --2- 172.21.15.16:0/3570411049 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8f2c06fe60 0x7f8f2c162180 secure :-1 s=READY pgs=1343 cs=0 l=1 rev1=1 crypto rx=0x7f8f2c0cf0b0 tx=0x7f8f2401c870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d655a845c9e579b server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:12:04.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:04.251+0000 7f8f29600640 1 -- 172.21.15.16:0/3570411049 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f240b2070 con 0x7f8f2c06fe60 2024-03-31T15:12:04.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:04.251+0000 7f8f29600640 1 -- 172.21.15.16:0/3570411049 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8f240a58f0 con 0x7f8f2c06fe60 2024-03-31T15:12:04.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:04.251+0000 7f8f29600640 1 -- 172.21.15.16:0/3570411049 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f240a5ca0 con 0x7f8f2c06fe60 2024-03-31T15:12:04.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:04.251+0000 7f8f31800640 1 -- 172.21.15.16:0/3570411049 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8f2c06fe60 msgr2=0x7f8f2c162180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:04.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:04.251+0000 7f8f31800640 1 --2- 172.21.15.16:0/3570411049 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8f2c06fe60 0x7f8f2c162180 secure :-1 s=READY pgs=1343 cs=0 l=1 rev1=1 crypto rx=0x7f8f2c0cf0b0 tx=0x7f8f2401c870 comp rx=0 tx=0).stop 2024-03-31T15:12:04.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:04.251+0000 7f8f31800640 1 -- 172.21.15.16:0/3570411049 shutdown_connections 2024-03-31T15:12:04.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:04.251+0000 7f8f31800640 1 --2- 172.21.15.16:0/3570411049 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8f2c06fe60 0x7f8f2c162180 unknown :-1 s=CLOSED pgs=1343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:04.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:04.251+0000 7f8f31800640 1 --2- 172.21.15.16:0/3570411049 >> v2:172.21.15.89:3300/0 conn(0x7f8f2c15f650 0x7f8f2c06f920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:04.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:04.251+0000 7f8f31800640 1 -- 172.21.15.16:0/3570411049 >> 172.21.15.16:0/3570411049 conn(0x7f8f2c067330 msgr2=0x7f8f2c067740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:12:04.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:04.251+0000 7f8f31800640 1 -- 172.21.15.16:0/3570411049 shutdown_connections 2024-03-31T15:12:04.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:04.251+0000 7f8f31800640 1 -- 172.21.15.16:0/3570411049 wait complete. 2024-03-31T15:12:04.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:04.251+0000 7f8f31800640 1 Processor -- start 2024-03-31T15:12:04.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:04.251+0000 7f8f31800640 1 -- start start 2024-03-31T15:12:04.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:04.255+0000 7f8f31800640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8f2c06dbc0 0x7f8f2c1581d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:04.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:04.255+0000 7f8f31800640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8f2c06fe60 0x7f8f2c15a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:04.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:04.255+0000 7f8f31800640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f8f2c15f650 0x7f8f2c1603b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:04.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:04.255+0000 7f8f31800640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f8f2c15fa30 con 0x7f8f2c06fe60 2024-03-31T15:12:04.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:04.255+0000 7f8f31800640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f8f2c06dfa0 con 0x7f8f2c06dbc0 2024-03-31T15:12:04.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:04.255+0000 7f8f31800640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f8f2c15fbb0 con 0x7f8f2c15f650 2024-03-31T15:12:04.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:04.255+0000 7f8f2a000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8f2c06fe60 0x7f8f2c15a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:04.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:04.255+0000 7f8f2a000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8f2c06fe60 0x7f8f2c15a8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49044/0 (socket says 172.21.15.16:49044) 2024-03-31T15:12:04.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:04.255+0000 7f8f2a000640 1 -- 172.21.15.16:0/1582036234 learned_addr learned my addr 172.21.15.16:0/1582036234 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:12:04.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:04.255+0000 7f8f2aa00640 1 --2- 172.21.15.16:0/1582036234 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8f2c06dbc0 0x7f8f2c1581d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:04.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:04.255+0000 7f8f2b400640 1 --2- 172.21.15.16:0/1582036234 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f8f2c15f650 0x7f8f2c1603b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:04.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:04.255+0000 7f8f2a000640 1 -- 172.21.15.16:0/1582036234 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f8f2c15f650 msgr2=0x7f8f2c1603b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:04.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:04.255+0000 7f8f2a000640 1 --2- 172.21.15.16:0/1582036234 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f8f2c15f650 0x7f8f2c1603b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:04.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:04.255+0000 7f8f2a000640 1 -- 172.21.15.16:0/1582036234 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8f2c06dbc0 msgr2=0x7f8f2c1581d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:04.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:04.255+0000 7f8f2a000640 1 --2- 172.21.15.16:0/1582036234 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8f2c06dbc0 0x7f8f2c1581d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:04.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:04.255+0000 7f8f2a000640 1 -- 172.21.15.16:0/1582036234 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f2c151b50 con 0x7f8f2c06fe60 2024-03-31T15:12:04.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:04.255+0000 7f8f2aa00640 1 --2- 172.21.15.16:0/1582036234 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8f2c06dbc0 0x7f8f2c1581d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:12:04.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:04.255+0000 7f8f2a000640 1 --2- 172.21.15.16:0/1582036234 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8f2c06fe60 0x7f8f2c15a8a0 secure :-1 s=READY pgs=1344 cs=0 l=1 rev1=1 crypto rx=0x7f8f1800e960 tx=0x7f8f1800ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:12:04.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:04.255+0000 7f8f22a00640 1 -- 172.21.15.16:0/1582036234 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f18017070 con 0x7f8f2c06fe60 2024-03-31T15:12:04.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:04.255+0000 7f8f22a00640 1 -- 172.21.15.16:0/1582036234 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8f1800ca00 con 0x7f8f2c06fe60 2024-03-31T15:12:04.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:04.255+0000 7f8f31800640 1 -- 172.21.15.16:0/1582036234 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8f2c151e30 con 0x7f8f2c06fe60 2024-03-31T15:12:04.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:04.255+0000 7f8f22a00640 1 -- 172.21.15.16:0/1582036234 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f1800cdb0 con 0x7f8f2c06fe60 2024-03-31T15:12:04.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:04.255+0000 7f8f31800640 1 -- 172.21.15.16:0/1582036234 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8f2c1523f0 con 0x7f8f2c06fe60 2024-03-31T15:12:04.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:04.255+0000 7f8f31800640 1 -- 172.21.15.16:0/1582036234 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8f2c069950 con 0x7f8f2c06fe60 2024-03-31T15:12:04.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:04.259+0000 7f8f22a00640 1 -- 172.21.15.16:0/1582036234 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f8f1801a070 con 0x7f8f2c06fe60 2024-03-31T15:12:04.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:04.259+0000 7f8f22a00640 1 --2- 172.21.15.16:0/1582036234 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f8f0c068380 0x7f8f0c06a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:04.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:04.259+0000 7f8f22a00640 1 -- 172.21.15.16:0/1582036234 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 5910+0+0 (secure 0 0 0) 0x7f8f18086e70 con 0x7f8f2c06fe60 2024-03-31T15:12:04.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:04.259+0000 7f8f2aa00640 1 --2- 172.21.15.16:0/1582036234 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f8f0c068380 0x7f8f0c06a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:04.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:04.259+0000 7f8f2aa00640 1 --2- 172.21.15.16:0/1582036234 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f8f0c068380 0x7f8f0c06a840 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f8f1c005890 tx=0x7f8f1c009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:12:04.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:04.263+0000 7f8f22a00640 1 -- 172.21.15.16:0/1582036234 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f8f18056c70 con 0x7f8f2c06fe60 2024-03-31T15:12:04.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:04.427+0000 7f8f31800640 1 -- 172.21.15.16:0/1582036234 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f8f2c0cf380 con 0x7f8f2c06fe60 2024-03-31T15:12:04.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:04.427+0000 7f8f22a00640 1 -- 172.21.15.16:0/1582036234 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v85) ==== 74+0+16894 (secure 0 0 0) 0x7f8f1805af70 con 0x7f8f2c06fe60 2024-03-31T15:12:04.422 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:12:04.422 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":85,"fsid":"286eb5da-fb47-4550-b290-d9519c13b9a5","created":"2024-03-31T15:03:11.438952+0000","modified":"2024-03-31T15:11:19.402716+0000","last_up_change":"2024-03-31T15:03:22.885535+0000","last_in_change":"2024-03-31T15:03:13.941667+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-31T15:03:22.550787+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-03-31T15:09:49.322620+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-03-31T15:09:49.877778+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"92a66450-ef19-40d8-ac3d-d760b23b0bda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6801","nonce":932712244}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6803","nonce":932712244}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6807","nonce":932712244}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6805","nonce":932712244}]},"public_addr":"172.21.15.16:6801/932712244","cluster_addr":"172.21.15.16:6803/932712244","heartbeat_back_addr":"172.21.15.16:6807/932712244","heartbeat_front_addr":"172.21.15.16:6805/932712244","state":["exists","up"]},{"osd":1,"uuid":"da9d1f2b-1968-4959-b843-faa231988e92","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6810","nonce":1398737533}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6811","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6813","nonce":1398737533}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6821","nonce":1398737533}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6815","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6816","nonce":1398737533}]},"public_addr":"172.21.15.16:6810/1398737533","cluster_addr":"172.21.15.16:6813/1398737533","heartbeat_back_addr":"172.21.15.16:6821/1398737533","heartbeat_front_addr":"172.21.15.16:6816/1398737533","state":["exists","up"]},{"osd":2,"uuid":"01a73662-9a21-4b1f-bcbb-42b9636a4638","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6819","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6822","nonce":3018526646}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6825","nonce":3018526646}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6831","nonce":3018526646}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6827","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6829","nonce":3018526646}]},"public_addr":"172.21.15.16:6822/3018526646","cluster_addr":"172.21.15.16:6825/3018526646","heartbeat_back_addr":"172.21.15.16:6831/3018526646","heartbeat_front_addr":"172.21.15.16:6829/3018526646","state":["exists","up"]},{"osd":3,"uuid":"c7771fd3-e18e-4500-ac38-23cf10ebcd75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6809","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6812","nonce":3360352974}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6817","nonce":3360352974}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6828","nonce":3360352974}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6823","nonce":3360352974}]},"public_addr":"172.21.15.16:6812/3360352974","cluster_addr":"172.21.15.16:6817/3360352974","heartbeat_back_addr":"172.21.15.16:6828/3360352974","heartbeat_front_addr":"172.21.15.16:6823/3360352974","state":["exists","up"]},{"osd":4,"uuid":"88260730-1482-4fbd-a40f-cb1124e9df4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6801","nonce":1856844856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6803","nonce":1856844856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6807","nonce":1856844856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6805","nonce":1856844856}]},"public_addr":"172.21.15.89:6801/1856844856","cluster_addr":"172.21.15.89:6803/1856844856","heartbeat_back_addr":"172.21.15.89:6807/1856844856","heartbeat_front_addr":"172.21.15.89:6805/1856844856","state":["exists","up"]},{"osd":5,"uuid":"4cb2c00c-fae0-4dd8-9d27-ec4f0093ccf7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6814","nonce":50395527}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6817","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6820","nonce":50395527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6831","nonce":50395527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6823","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6826","nonce":50395527}]},"public_addr":"172.21.15.89:6814/50395527","cluster_addr":"172.21.15.89:6820/50395527","heartbeat_back_addr":"172.21.15.89:6831/50395527","heartbeat_front_addr":"172.21.15.89:6826/50395527","state":["exists","up"]},{"osd":6,"uuid":"81ce0c19-2554-4cae-acbe-66a05e9e9b9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6809","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6812","nonce":3470471752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6815","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6818","nonce":3470471752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6827","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6829","nonce":3470471752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6821","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6824","nonce":3470471752}]},"public_addr":"172.21.15.89:6812/3470471752","cluster_addr":"172.21.15.89:6818/3470471752","heartbeat_back_addr":"172.21.15.89:6829/3470471752","heartbeat_front_addr":"172.21.15.89:6824/3470471752","state":["exists","up"]},{"osd":7,"uuid":"85e4313b-d37d-442e-8042-153bf20a847d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6811","nonce":1796778630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6813","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6816","nonce":1796778630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6825","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6828","nonce":1796778630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6819","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6822","nonce":1796778630}]},"public_addr":"172.21.15.89:6811/1796778630","cluster_addr":"172.21.15.89:6816/1796778630","heartbeat_back_addr":"172.21.15.89:6828/1796778630","heartbeat_front_addr":"172.21.15.89:6822/1796778630","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.439088+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.529970+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.680512+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:21.044161+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.582111+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.536558+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.409522+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.590229+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"13.d","mappings":[{"from":1,"to":3}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:6832/818074038":"2024-04-01T15:11:19.402687+0000","172.21.15.16:6834/818074038":"2024-04-01T15:11:19.402687+0000","172.21.15.16:6834/2478530200":"2024-04-01T15:10:04.360772+0000","172.21.15.16:6832/2478530200":"2024-04-01T15:10:04.360772+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-31T15:12:04.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:04.431+0000 7f8f31800640 1 -- 172.21.15.16:0/1582036234 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f8f0c068380 msgr2=0x7f8f0c06a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:04.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:04.431+0000 7f8f31800640 1 --2- 172.21.15.16:0/1582036234 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f8f0c068380 0x7f8f0c06a840 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f8f1c005890 tx=0x7f8f1c009040 comp rx=0 tx=0).stop 2024-03-31T15:12:04.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:04.431+0000 7f8f31800640 1 -- 172.21.15.16:0/1582036234 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8f2c06fe60 msgr2=0x7f8f2c15a8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:04.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:04.431+0000 7f8f31800640 1 --2- 172.21.15.16:0/1582036234 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8f2c06fe60 0x7f8f2c15a8a0 secure :-1 s=READY pgs=1344 cs=0 l=1 rev1=1 crypto rx=0x7f8f1800e960 tx=0x7f8f1800ee20 comp rx=0 tx=0).stop 2024-03-31T15:12:04.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:04.431+0000 7f8f31800640 1 -- 172.21.15.16:0/1582036234 shutdown_connections 2024-03-31T15:12:04.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:04.431+0000 7f8f31800640 1 --2- 172.21.15.16:0/1582036234 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f8f0c068380 0x7f8f0c06a840 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:04.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:04.431+0000 7f8f31800640 1 --2- 172.21.15.16:0/1582036234 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f8f2c15f650 0x7f8f2c1603b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:04.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:04.431+0000 7f8f31800640 1 --2- 172.21.15.16:0/1582036234 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8f2c06fe60 0x7f8f2c15a8a0 unknown :-1 s=CLOSED pgs=1344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:04.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:04.431+0000 7f8f31800640 1 --2- 172.21.15.16:0/1582036234 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8f2c06dbc0 0x7f8f2c1581d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:04.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:04.431+0000 7f8f31800640 1 -- 172.21.15.16:0/1582036234 >> 172.21.15.16:0/1582036234 conn(0x7f8f2c067330 msgr2=0x7f8f2c161590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:12:04.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:04.431+0000 7f8f31800640 1 -- 172.21.15.16:0/1582036234 shutdown_connections 2024-03-31T15:12:04.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:04.431+0000 7f8f31800640 1 -- 172.21.15.16:0/1582036234 wait complete. 2024-03-31T15:12:04.444 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-31T15:12:04.586 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:04.591+0000 7fcd89a00640 1 Processor -- start 2024-03-31T15:12:04.586 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:04.591+0000 7fcd89a00640 1 -- start start 2024-03-31T15:12:04.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:04.595+0000 7fcd89a00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fcd8406f920 0x7fcd8406fd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:04.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:04.595+0000 7fcd89a00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fcd840d2010 0x7fcd840d24a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:04.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:04.595+0000 7fcd89a00640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcd840d0920 con 0x7fcd84145e40 2024-03-31T15:12:04.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:04.595+0000 7fcd89a00640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7fcd840702c0 con 0x7fcd840d2010 2024-03-31T15:12:04.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:04.595+0000 7fcd89a00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fcd840d06d0 con 0x7fcd8406f920 2024-03-31T15:12:04.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:04.595+0000 7fcd82a00640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fcd84145e40 0x7fcd840cc740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:35126/0 (socket says 172.21.15.16:35126) 2024-03-31T15:12:04.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:04.595+0000 7fcd82a00640 1 -- 172.21.15.16:0/2431948664 learned_addr learned my addr 172.21.15.16:0/2431948664 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:12:04.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:04.595+0000 7fcd83400640 1 --2- 172.21.15.16:0/2431948664 >> v2:172.21.15.16:3300/0 conn(0x7fcd840d2010 0x7fcd840d24a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:04.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:04.595+0000 7fcd7be00640 1 --2- 172.21.15.16:0/2431948664 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fcd8406f920 0x7fcd8406fd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:04.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:04.595+0000 7fcd83400640 1 -- 172.21.15.16:0/2431948664 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fcd8406f920 msgr2=0x7fcd8406fd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:12:04.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:04.595+0000 7fcd83400640 1 --2- 172.21.15.16:0/2431948664 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fcd8406f920 0x7fcd8406fd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:04.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:04.595+0000 7fcd83400640 1 -- 172.21.15.16:0/2431948664 >> v1:172.21.15.16:6789/0 conn(0x7fcd84145e40 legacy=0x7fcd840cc740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:04.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:04.595+0000 7fcd83400640 1 -- 172.21.15.16:0/2431948664 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcd840d0c00 con 0x7fcd840d2010 2024-03-31T15:12:04.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:04.595+0000 7fcd82000640 1 -- 172.21.15.16:0/2431948664 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2626085273 0 0) 0x7fcd840d0920 con 0x7fcd84145e40 2024-03-31T15:12:04.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:04.599+0000 7fcd83400640 1 --2- 172.21.15.16:0/2431948664 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcd840d2010 0x7fcd840d24a0 secure :-1 s=READY pgs=1346 cs=0 l=1 rev1=1 crypto rx=0x7fcd7c009d90 tx=0x7fcd7c01c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ad47e409e0935f8 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:12:04.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:04.599+0000 7fcd82000640 1 -- 172.21.15.16:0/2431948664 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd7c0b2070 con 0x7fcd840d2010 2024-03-31T15:12:04.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:04.599+0000 7fcd82000640 1 -- 172.21.15.16:0/2431948664 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcd7c0a5920 con 0x7fcd840d2010 2024-03-31T15:12:04.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:04.599+0000 7fcd82000640 1 -- 172.21.15.16:0/2431948664 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd7c0a5cf0 con 0x7fcd840d2010 2024-03-31T15:12:04.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:04.599+0000 7fcd89a00640 1 -- 172.21.15.16:0/2431948664 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcd840d2010 msgr2=0x7fcd840d24a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:04.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:04.599+0000 7fcd89a00640 1 --2- 172.21.15.16:0/2431948664 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcd840d2010 0x7fcd840d24a0 secure :-1 s=READY pgs=1346 cs=0 l=1 rev1=1 crypto rx=0x7fcd7c009d90 tx=0x7fcd7c01c970 comp rx=0 tx=0).stop 2024-03-31T15:12:04.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:04.599+0000 7fcd89a00640 1 -- 172.21.15.16:0/2431948664 shutdown_connections 2024-03-31T15:12:04.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:04.599+0000 7fcd89a00640 1 --2- 172.21.15.16:0/2431948664 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcd840d2010 0x7fcd840d24a0 unknown :-1 s=CLOSED pgs=1346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:04.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:04.599+0000 7fcd89a00640 1 --2- 172.21.15.16:0/2431948664 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fcd8406f920 0x7fcd8406fd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:04.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:04.599+0000 7fcd89a00640 1 -- 172.21.15.16:0/2431948664 >> 172.21.15.16:0/2431948664 conn(0x7fcd84067330 msgr2=0x7fcd84067740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:12:04.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:04.599+0000 7fcd89a00640 1 -- 172.21.15.16:0/2431948664 shutdown_connections 2024-03-31T15:12:04.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:04.599+0000 7fcd89a00640 1 -- 172.21.15.16:0/2431948664 wait complete. 2024-03-31T15:12:04.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:04.599+0000 7fcd89a00640 1 Processor -- start 2024-03-31T15:12:04.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:04.599+0000 7fcd89a00640 1 -- start start 2024-03-31T15:12:04.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:04.603+0000 7fcd89a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcd8406f920 0x7fcd84157cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:04.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:04.603+0000 7fcd89a00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcd840d2010 0x7fcd840c7030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:04.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:04.603+0000 7fcd89a00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fcd84145e40 0x7fcd840c7570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:04.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:04.603+0000 7fcd89a00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fcd840d0ee0 con 0x7fcd8406f920 2024-03-31T15:12:04.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:04.603+0000 7fcd89a00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fcd840ce9e0 con 0x7fcd840d2010 2024-03-31T15:12:04.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:04.603+0000 7fcd89a00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fcd840d2df0 con 0x7fcd84145e40 2024-03-31T15:12:04.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:04.603+0000 7fcd82a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcd8406f920 0x7fcd84157cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:04.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:04.603+0000 7fcd82a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcd8406f920 0x7fcd84157cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49062/0 (socket says 172.21.15.16:49062) 2024-03-31T15:12:04.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:04.603+0000 7fcd82a00640 1 -- 172.21.15.16:0/3129147913 learned_addr learned my addr 172.21.15.16:0/3129147913 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:12:04.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:04.603+0000 7fcd7be00640 1 --2- 172.21.15.16:0/3129147913 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcd840d2010 0x7fcd840c7030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:04.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:04.603+0000 7fcd83400640 1 --2- 172.21.15.16:0/3129147913 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fcd84145e40 0x7fcd840c7570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:04.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:04.603+0000 7fcd82a00640 1 -- 172.21.15.16:0/3129147913 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fcd84145e40 msgr2=0x7fcd840c7570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:04.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:04.603+0000 7fcd82a00640 1 --2- 172.21.15.16:0/3129147913 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fcd84145e40 0x7fcd840c7570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:04.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:04.603+0000 7fcd82a00640 1 -- 172.21.15.16:0/3129147913 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcd840d2010 msgr2=0x7fcd840c7030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:04.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:04.603+0000 7fcd82a00640 1 --2- 172.21.15.16:0/3129147913 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcd840d2010 0x7fcd840c7030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:04.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:04.603+0000 7fcd82a00640 1 -- 172.21.15.16:0/3129147913 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcd8415f0a0 con 0x7fcd8406f920 2024-03-31T15:12:04.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:04.603+0000 7fcd83400640 1 --2- 172.21.15.16:0/3129147913 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fcd84145e40 0x7fcd840c7570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:12:04.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:04.603+0000 7fcd82a00640 1 --2- 172.21.15.16:0/3129147913 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcd8406f920 0x7fcd84157cd0 secure :-1 s=READY pgs=1347 cs=0 l=1 rev1=1 crypto rx=0x7fcd740011f0 tx=0x7fcd7400c9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:12:04.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:04.603+0000 7fcd7aa00640 1 -- 172.21.15.16:0/3129147913 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd74018070 con 0x7fcd8406f920 2024-03-31T15:12:04.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:04.603+0000 7fcd7aa00640 1 -- 172.21.15.16:0/3129147913 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcd740074e0 con 0x7fcd8406f920 2024-03-31T15:12:04.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:04.603+0000 7fcd7aa00640 1 -- 172.21.15.16:0/3129147913 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd74004a10 con 0x7fcd8406f920 2024-03-31T15:12:04.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:04.603+0000 7fcd89a00640 1 -- 172.21.15.16:0/3129147913 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcd8415f380 con 0x7fcd8406f920 2024-03-31T15:12:04.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:04.603+0000 7fcd89a00640 1 -- 172.21.15.16:0/3129147913 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcd8415a620 con 0x7fcd8406f920 2024-03-31T15:12:04.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:04.603+0000 7fcd89a00640 1 -- 172.21.15.16:0/3129147913 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcd84069950 con 0x7fcd8406f920 2024-03-31T15:12:04.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:04.607+0000 7fcd7aa00640 1 -- 172.21.15.16:0/3129147913 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fcd74004bb0 con 0x7fcd8406f920 2024-03-31T15:12:04.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:04.607+0000 7fcd7aa00640 1 --2- 172.21.15.16:0/3129147913 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fcd60068380 0x7fcd6006a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:04.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:04.607+0000 7fcd7aa00640 1 -- 172.21.15.16:0/3129147913 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 5910+0+0 (secure 0 0 0) 0x7fcd74085ec0 con 0x7fcd8406f920 2024-03-31T15:12:04.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:04.607+0000 7fcd7be00640 1 --2- 172.21.15.16:0/3129147913 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fcd60068380 0x7fcd6006a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:04.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:04.607+0000 7fcd7be00640 1 --2- 172.21.15.16:0/3129147913 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fcd60068380 0x7fcd6006a840 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7fcd8415fcf0 tx=0x7fcd6c008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:12:04.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:04.611+0000 7fcd7aa00640 1 -- 172.21.15.16:0/3129147913 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fcd74056310 con 0x7fcd8406f920 2024-03-31T15:12:04.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:04.811+0000 7fcd89a00640 1 -- 172.21.15.16:0/3129147913 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7fcd8406ff90 con 0x7fcd8406f920 2024-03-31T15:12:04.805 INFO:tasks.ceph.mon.a.smithi016.stderr:2024-03-31T15:12:04.811+0000 7f7dcbc00640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-31T15:12:04.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:04.815+0000 7fcd7aa00640 1 -- 172.21.15.16:0/3129147913 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v75) ==== 161+0+0 (secure 0 0 0) 0x7fcd74055c10 con 0x7fcd8406f920 2024-03-31T15:12:04.808 INFO:teuthology.orchestra.run.smithi016.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-31T15:12:04.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:04.819+0000 7fcd89a00640 1 -- 172.21.15.16:0/3129147913 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fcd60068380 msgr2=0x7fcd6006a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:04.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:04.819+0000 7fcd89a00640 1 --2- 172.21.15.16:0/3129147913 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fcd60068380 0x7fcd6006a840 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7fcd8415fcf0 tx=0x7fcd6c008040 comp rx=0 tx=0).stop 2024-03-31T15:12:04.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:04.819+0000 7fcd89a00640 1 -- 172.21.15.16:0/3129147913 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcd8406f920 msgr2=0x7fcd84157cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:04.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:04.819+0000 7fcd89a00640 1 --2- 172.21.15.16:0/3129147913 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcd8406f920 0x7fcd84157cd0 secure :-1 s=READY pgs=1347 cs=0 l=1 rev1=1 crypto rx=0x7fcd740011f0 tx=0x7fcd7400c9b0 comp rx=0 tx=0).stop 2024-03-31T15:12:04.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:04.819+0000 7fcd89a00640 1 -- 172.21.15.16:0/3129147913 shutdown_connections 2024-03-31T15:12:04.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:04.819+0000 7fcd89a00640 1 --2- 172.21.15.16:0/3129147913 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fcd60068380 0x7fcd6006a840 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:04.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:04.819+0000 7fcd89a00640 1 --2- 172.21.15.16:0/3129147913 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fcd84145e40 0x7fcd840c7570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:04.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:04.819+0000 7fcd89a00640 1 --2- 172.21.15.16:0/3129147913 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcd840d2010 0x7fcd840c7030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:04.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:04.819+0000 7fcd89a00640 1 --2- 172.21.15.16:0/3129147913 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcd8406f920 0x7fcd84157cd0 unknown :-1 s=CLOSED pgs=1347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:04.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:04.819+0000 7fcd89a00640 1 -- 172.21.15.16:0/3129147913 >> 172.21.15.16:0/3129147913 conn(0x7fcd84067330 msgr2=0x7fcd840cff60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:12:04.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:04.819+0000 7fcd89a00640 1 -- 172.21.15.16:0/3129147913 shutdown_connections 2024-03-31T15:12:04.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:04.819+0000 7fcd89a00640 1 -- 172.21.15.16:0/3129147913 wait complete. 2024-03-31T15:12:04.821 INFO:tasks.ceph.mds.c.smithi016.stderr: -2346> 2024-03-31T15:11:25.411+0000 7f86e2a00640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-31T15:12:04.823 INFO:tasks.ceph.mds.c.smithi016.stderr: -1964> 2024-03-31T15:11:27.043+0000 7f86dda00640 -1 mds.0.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000001 [...2,head] /dir/file auth v1 ap=1 DIRTYPARENT s=0 nl=2 n() | dirtyparent=1 authpin=1 scrubqueue=0 0x55cab02e6680]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(13)0x10000000001:[<0x10000000000/file v6>,<0x1/dir v13>]//[]","memoryvalue":"(13)0x10000000001:[<0x10000000000/file v1>,<0x1/dir v20>]//[]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-31T15:12:04.838 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-31T15:12:04.872 INFO:tasks.ceph.mds.c.smithi016.stdout:starting mds.c at 2024-03-31T15:12:04.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:04.991+0000 7fc038c00640 1 Processor -- start 2024-03-31T15:12:04.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:04.991+0000 7fc038c00640 1 -- start start 2024-03-31T15:12:04.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:04.991+0000 7fc038c00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fc0340d0a60 0x7fc0340cea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:04.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:04.991+0000 7fc038c00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fc0340cef80 0x7fc0341419c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:04.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:04.991+0000 7fc038c00640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc03415ca70 con 0x7fc03406f920 2024-03-31T15:12:04.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:04.991+0000 7fc038c00640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7fc03413f670 con 0x7fc0340d0a60 2024-03-31T15:12:04.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:04.991+0000 7fc038c00640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7fc03415cd50 con 0x7fc0340cef80 2024-03-31T15:12:04.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:04.991+0000 7fc031400640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fc0340d0a60 0x7fc0340cea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:04.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:04.991+0000 7fc031400640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fc0340d0a60 0x7fc0340cea40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49080/0 (socket says 172.21.15.16:49080) 2024-03-31T15:12:04.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:04.991+0000 7fc031400640 1 -- 172.21.15.16:0/4019841658 learned_addr learned my addr 172.21.15.16:0/4019841658 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:12:04.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:04.991+0000 7fc032800640 1 --2- 172.21.15.16:0/4019841658 >> v2:172.21.15.89:3300/0 conn(0x7fc0340cef80 0x7fc0341419c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:04.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:04.991+0000 7fc031400640 1 -- 172.21.15.16:0/4019841658 >> v2:172.21.15.89:3300/0 conn(0x7fc0340cef80 msgr2=0x7fc0341419c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:12:04.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:04.991+0000 7fc031400640 1 --2- 172.21.15.16:0/4019841658 >> v2:172.21.15.89:3300/0 conn(0x7fc0340cef80 0x7fc0341419c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:04.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:04.991+0000 7fc031400640 1 -- 172.21.15.16:0/4019841658 >> v1:172.21.15.89:6789/0 conn(0x7fc03406f920 legacy=0x7fc03406fd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:12:04.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:04.991+0000 7fc031400640 1 -- 172.21.15.16:0/4019841658 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc03415d030 con 0x7fc0340d0a60 2024-03-31T15:12:04.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:04.995+0000 7fc031400640 1 --2- 172.21.15.16:0/4019841658 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc0340d0a60 0x7fc0340cea40 secure :-1 s=READY pgs=1351 cs=0 l=1 rev1=1 crypto rx=0x7fc0340b4530 tx=0x7fc02001c4e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb25a6c83d3f997 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:12:04.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:04.995+0000 7fc030a00640 1 -- 172.21.15.16:0/4019841658 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0200b0070 con 0x7fc0340d0a60 2024-03-31T15:12:04.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:04.995+0000 7fc030a00640 1 -- 172.21.15.16:0/4019841658 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc0200ab600 con 0x7fc0340d0a60 2024-03-31T15:12:04.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:04.995+0000 7fc030a00640 1 -- 172.21.15.16:0/4019841658 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0200ab9d0 con 0x7fc0340d0a60 2024-03-31T15:12:04.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:04.995+0000 7fc038c00640 1 -- 172.21.15.16:0/4019841658 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc0340d0a60 msgr2=0x7fc0340cea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:04.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:04.995+0000 7fc038c00640 1 --2- 172.21.15.16:0/4019841658 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc0340d0a60 0x7fc0340cea40 secure :-1 s=READY pgs=1351 cs=0 l=1 rev1=1 crypto rx=0x7fc0340b4530 tx=0x7fc02001c4e0 comp rx=0 tx=0).stop 2024-03-31T15:12:04.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:04.995+0000 7fc038c00640 1 -- 172.21.15.16:0/4019841658 shutdown_connections 2024-03-31T15:12:04.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:04.995+0000 7fc038c00640 1 --2- 172.21.15.16:0/4019841658 >> v2:172.21.15.89:3300/0 conn(0x7fc0340cef80 0x7fc0341419c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:04.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:04.995+0000 7fc038c00640 1 --2- 172.21.15.16:0/4019841658 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc0340d0a60 0x7fc0340cea40 unknown :-1 s=CLOSED pgs=1351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:04.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:04.995+0000 7fc038c00640 1 -- 172.21.15.16:0/4019841658 >> 172.21.15.16:0/4019841658 conn(0x7fc034067930 msgr2=0x7fc0340b3ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:12:04.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:04.995+0000 7fc038c00640 1 -- 172.21.15.16:0/4019841658 shutdown_connections 2024-03-31T15:12:04.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:04.995+0000 7fc038c00640 1 -- 172.21.15.16:0/4019841658 wait complete. 2024-03-31T15:12:04.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:04.995+0000 7fc038c00640 1 Processor -- start 2024-03-31T15:12:04.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:04.995+0000 7fc038c00640 1 -- start start 2024-03-31T15:12:04.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:04.999+0000 7fc038c00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc03406f920 0x7fc0340b6c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:04.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:04.999+0000 7fc038c00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fc0340cef80 0x7fc0340d1ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:04.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:04.999+0000 7fc038c00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc0340d0a60 0x7fc0341573a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:04.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:04.999+0000 7fc038c00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fc034169230 con 0x7fc0340d0a60 2024-03-31T15:12:04.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:04.999+0000 7fc038c00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fc0341690b0 con 0x7fc03406f920 2024-03-31T15:12:04.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:04.999+0000 7fc038c00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fc0341693b0 con 0x7fc0340cef80 2024-03-31T15:12:04.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:04.999+0000 7fc032800640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc0340d0a60 0x7fc0341573a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:04.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:04.999+0000 7fc032800640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc0340d0a60 0x7fc0341573a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49082/0 (socket says 172.21.15.16:49082) 2024-03-31T15:12:04.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:04.999+0000 7fc032800640 1 -- 172.21.15.16:0/308561799 learned_addr learned my addr 172.21.15.16:0/308561799 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:12:04.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:04.999+0000 7fc031e00640 1 --2- 172.21.15.16:0/308561799 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc03406f920 0x7fc0340b6c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:04.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:04.999+0000 7fc031400640 1 --2- 172.21.15.16:0/308561799 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fc0340cef80 0x7fc0340d1ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:04.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:04.999+0000 7fc032800640 1 -- 172.21.15.16:0/308561799 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fc0340cef80 msgr2=0x7fc0340d1ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:04.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:04.999+0000 7fc032800640 1 --2- 172.21.15.16:0/308561799 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fc0340cef80 0x7fc0340d1ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:04.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:04.999+0000 7fc032800640 1 -- 172.21.15.16:0/308561799 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc03406f920 msgr2=0x7fc0340b6c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:04.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:04.999+0000 7fc032800640 1 --2- 172.21.15.16:0/308561799 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc03406f920 0x7fc0340b6c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:04.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:04.999+0000 7fc032800640 1 -- 172.21.15.16:0/308561799 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0340d2610 con 0x7fc0340d0a60 2024-03-31T15:12:04.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:04.999+0000 7fc032800640 1 --2- 172.21.15.16:0/308561799 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc0340d0a60 0x7fc0341573a0 secure :-1 s=READY pgs=1352 cs=0 l=1 rev1=1 crypto rx=0x7fc02c007e40 tx=0x7fc02c00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:12:04.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:04.999+0000 7fc02a000640 1 -- 172.21.15.16:0/308561799 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc02c014620 con 0x7fc0340d0a60 2024-03-31T15:12:04.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:04.999+0000 7fc02a000640 1 -- 172.21.15.16:0/308561799 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc02c019070 con 0x7fc0340d0a60 2024-03-31T15:12:04.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:04.999+0000 7fc038c00640 1 -- 172.21.15.16:0/308561799 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc0340d28f0 con 0x7fc0340d0a60 2024-03-31T15:12:04.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:04.999+0000 7fc02a000640 1 -- 172.21.15.16:0/308561799 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc02c014a20 con 0x7fc0340d0a60 2024-03-31T15:12:04.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:04.999+0000 7fc038c00640 1 -- 172.21.15.16:0/308561799 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc0341578e0 con 0x7fc0340d0a60 2024-03-31T15:12:04.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:04.999+0000 7fc038c00640 1 -- 172.21.15.16:0/308561799 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc034069950 con 0x7fc0340d0a60 2024-03-31T15:12:04.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:04.999+0000 7fc02a000640 1 -- 172.21.15.16:0/308561799 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fc02c014bc0 con 0x7fc0340d0a60 2024-03-31T15:12:04.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:05.003+0000 7fc02a000640 1 --2- 172.21.15.16:0/308561799 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fc014068380 0x7fc01406a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:04.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:05.003+0000 7fc02a000640 1 -- 172.21.15.16:0/308561799 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 5996+0+0 (secure 0 0 0) 0x7fc02c087330 con 0x7fc0340d0a60 2024-03-31T15:12:04.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:05.003+0000 7fc031e00640 1 --2- 172.21.15.16:0/308561799 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fc014068380 0x7fc01406a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:04.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:05.003+0000 7fc031e00640 1 --2- 172.21.15.16:0/308561799 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fc014068380 0x7fc01406a840 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7fc0340d2f30 tx=0x7fc024007e40 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:12:04.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:05.007+0000 7fc02a000640 1 -- 172.21.15.16:0/308561799 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fc02c0570e0 con 0x7fc0340d0a60 2024-03-31T15:12:05.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:05.211+0000 7fc038c00640 1 -- 172.21.15.16:0/308561799 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7fc0340b50e0 con 0x7fc0340d0a60 2024-03-31T15:12:05.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:05.815+0000 7fc02a000640 1 -- 172.21.15.16:0/308561799 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v76) ==== 104+0+0 (secure 0 0 0) 0x7fc02c05b3e0 con 0x7fc0340d0a60 2024-03-31T15:12:05.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:05.819+0000 7fc038c00640 1 -- 172.21.15.16:0/308561799 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fc014068380 msgr2=0x7fc01406a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:05.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:05.819+0000 7fc038c00640 1 --2- 172.21.15.16:0/308561799 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fc014068380 0x7fc01406a840 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7fc0340d2f30 tx=0x7fc024007e40 comp rx=0 tx=0).stop 2024-03-31T15:12:05.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:05.819+0000 7fc038c00640 1 -- 172.21.15.16:0/308561799 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc0340d0a60 msgr2=0x7fc0341573a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:05.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:05.819+0000 7fc038c00640 1 --2- 172.21.15.16:0/308561799 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc0340d0a60 0x7fc0341573a0 secure :-1 s=READY pgs=1352 cs=0 l=1 rev1=1 crypto rx=0x7fc02c007e40 tx=0x7fc02c00e4b0 comp rx=0 tx=0).stop 2024-03-31T15:12:05.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:05.819+0000 7fc038c00640 1 -- 172.21.15.16:0/308561799 shutdown_connections 2024-03-31T15:12:05.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:05.819+0000 7fc038c00640 1 --2- 172.21.15.16:0/308561799 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fc014068380 0x7fc01406a840 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:05.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:05.819+0000 7fc038c00640 1 --2- 172.21.15.16:0/308561799 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc0340d0a60 0x7fc0341573a0 unknown :-1 s=CLOSED pgs=1352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:05.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:05.819+0000 7fc038c00640 1 --2- 172.21.15.16:0/308561799 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fc0340cef80 0x7fc0340d1ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:05.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:05.819+0000 7fc038c00640 1 --2- 172.21.15.16:0/308561799 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc03406f920 0x7fc0340b6c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:05.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:05.819+0000 7fc038c00640 1 -- 172.21.15.16:0/308561799 >> 172.21.15.16:0/308561799 conn(0x7fc034067930 msgr2=0x7fc03415b5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:12:05.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:05.819+0000 7fc038c00640 1 -- 172.21.15.16:0/308561799 shutdown_connections 2024-03-31T15:12:05.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:05.819+0000 7fc038c00640 1 -- 172.21.15.16:0/308561799 wait complete. 2024-03-31T15:12:05.833 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-31T15:12:06.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:06.031+0000 7f242f200640 1 Processor -- start 2024-03-31T15:12:06.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:06.031+0000 7f242f200640 1 -- start start 2024-03-31T15:12:06.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:06.031+0000 7f242f200640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f24280b22f0 0x7f2428071c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:06.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:06.031+0000 7f242f200640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f24280b39f0 con 0x7f242815e400 2024-03-31T15:12:06.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:06.031+0000 7f242f200640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f24280d0b90 con 0x7f2428073090 2024-03-31T15:12:06.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:06.031+0000 7f242f200640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f24280b4250 con 0x7f24280b22f0 2024-03-31T15:12:06.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:06.031+0000 7f2427e00640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f242815e400 0x7f24280b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:35140/0 (socket says 172.21.15.16:35140) 2024-03-31T15:12:06.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:06.031+0000 7f2427e00640 1 -- 172.21.15.16:0/921847657 learned_addr learned my addr 172.21.15.16:0/921847657 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:12:06.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:06.031+0000 7f2427400640 1 -- 172.21.15.16:0/921847657 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 645842515 0 0) 0x7f24280b39f0 con 0x7f242815e400 2024-03-31T15:12:06.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:06.031+0000 7f241f400640 1 --2- 172.21.15.16:0/921847657 >> v2:172.21.15.89:3300/0 conn(0x7f24280b22f0 0x7f2428071c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:06.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:06.031+0000 7f2427400640 1 -- 172.21.15.16:0/921847657 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f240c003530 con 0x7f242815e400 2024-03-31T15:12:06.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:06.031+0000 7f2427400640 1 -- 172.21.15.16:0/921847657 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 492604258 0 0) 0x7f240c003530 con 0x7f242815e400 2024-03-31T15:12:06.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:06.031+0000 7f2427400640 1 -- 172.21.15.16:0/921847657 >> v2:172.21.15.89:3300/0 conn(0x7f24280b22f0 msgr2=0x7f2428071c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:12:06.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:06.031+0000 7f2427400640 1 --2- 172.21.15.16:0/921847657 >> v2:172.21.15.89:3300/0 conn(0x7f24280b22f0 0x7f2428071c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:06.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:06.031+0000 7f2427400640 1 -- 172.21.15.16:0/921847657 >> v1:172.21.15.89:6789/0 conn(0x7f2428073090 legacy=0x7f24280721d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:06.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:06.031+0000 7f2427400640 1 -- 172.21.15.16:0/921847657 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24280b3c90 con 0x7f242815e400 2024-03-31T15:12:06.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:06.031+0000 7f2427400640 1 -- 172.21.15.16:0/921847657 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 214807623 0 0) 0x7f2418001a20 con 0x7f242815e400 2024-03-31T15:12:06.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:06.035+0000 7f2427400640 1 -- 172.21.15.16:0/921847657 >> v1:172.21.15.16:6789/0 conn(0x7f242815e400 legacy=0x7f24280b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:06.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:06.035+0000 7f2427400640 1 --2- 172.21.15.16:0/921847657 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f240c0039c0 0x7f240c003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:06.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:06.035+0000 7f2427400640 1 --2- 172.21.15.16:0/921847657 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f240c004810 0x7f240c004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:06.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:06.035+0000 7f2427400640 1 --2- 172.21.15.16:0/921847657 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f240c0051d0 0x7f240c00b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:06.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:06.035+0000 7f2427400640 1 -- 172.21.15.16:0/921847657 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f240c00c140 con 0x7f240c0039c0 2024-03-31T15:12:06.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:06.035+0000 7f2427400640 1 -- 172.21.15.16:0/921847657 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f240c00c2f0 con 0x7f240c0051d0 2024-03-31T15:12:06.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:06.035+0000 7f2427400640 1 -- 172.21.15.16:0/921847657 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f240c00c4f0 con 0x7f240c004810 2024-03-31T15:12:06.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:06.035+0000 7f2427e00640 1 --2- 172.21.15.16:0/921847657 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f240c0039c0 0x7f240c003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:06.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:06.035+0000 7f241f400640 1 --2- 172.21.15.16:0/921847657 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f240c004810 0x7f240c004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:06.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:06.035+0000 7f242d200640 1 --2- 172.21.15.16:0/921847657 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f240c0051d0 0x7f240c00b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:06.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:06.035+0000 7f2427e00640 1 -- 172.21.15.16:0/921847657 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f240c004810 msgr2=0x7f240c004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:12:06.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:06.035+0000 7f2427e00640 1 --2- 172.21.15.16:0/921847657 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f240c004810 0x7f240c004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:06.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:06.035+0000 7f2427e00640 1 -- 172.21.15.16:0/921847657 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f240c0051d0 msgr2=0x7f240c00b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:12:06.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:06.035+0000 7f2427e00640 1 --2- 172.21.15.16:0/921847657 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f240c0051d0 0x7f240c00b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:06.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:06.035+0000 7f2427e00640 1 -- 172.21.15.16:0/921847657 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f240c003530 con 0x7f240c0039c0 2024-03-31T15:12:06.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:06.035+0000 7f241f400640 1 --2- 172.21.15.16:0/921847657 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f240c004810 0x7f240c004c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:12:06.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:06.035+0000 7f242d200640 1 -- 172.21.15.16:0/921847657 reap_dead start 2024-03-31T15:12:06.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:06.035+0000 7f2427e00640 1 --2- 172.21.15.16:0/921847657 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f240c0039c0 0x7f240c003de0 secure :-1 s=READY pgs=1354 cs=0 l=1 rev1=1 crypto rx=0x7f241800a850 tx=0x7f241801d330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=561c7d597864b63 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:12:06.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:06.035+0000 7f2427400640 1 -- 172.21.15.16:0/921847657 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f241800a4a0 con 0x7f240c0039c0 2024-03-31T15:12:06.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:06.035+0000 7f2427400640 1 -- 172.21.15.16:0/921847657 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f24180ad650 con 0x7f240c0039c0 2024-03-31T15:12:06.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:06.035+0000 7f2427400640 1 -- 172.21.15.16:0/921847657 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24180ada40 con 0x7f240c0039c0 2024-03-31T15:12:06.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:06.039+0000 7f242f200640 1 -- 172.21.15.16:0/921847657 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f240c0039c0 msgr2=0x7f240c003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:06.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:06.039+0000 7f242f200640 1 --2- 172.21.15.16:0/921847657 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f240c0039c0 0x7f240c003de0 secure :-1 s=READY pgs=1354 cs=0 l=1 rev1=1 crypto rx=0x7f241800a850 tx=0x7f241801d330 comp rx=0 tx=0).stop 2024-03-31T15:12:06.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:06.039+0000 7f242f200640 1 -- 172.21.15.16:0/921847657 shutdown_connections 2024-03-31T15:12:06.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:06.039+0000 7f242f200640 1 --2- 172.21.15.16:0/921847657 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f240c0039c0 0x7f240c003de0 unknown :-1 s=CLOSED pgs=1354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:06.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:06.039+0000 7f242f200640 1 -- 172.21.15.16:0/921847657 >> 172.21.15.16:0/921847657 conn(0x7f242806b620 msgr2=0x7f242806ba30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:12:06.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:06.039+0000 7f242f200640 1 -- 172.21.15.16:0/921847657 shutdown_connections 2024-03-31T15:12:06.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:06.039+0000 7f242f200640 1 -- 172.21.15.16:0/921847657 wait complete. 2024-03-31T15:12:06.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:06.039+0000 7f242f200640 1 Processor -- start 2024-03-31T15:12:06.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:06.039+0000 7f242f200640 1 -- start start 2024-03-31T15:12:06.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:06.039+0000 7f242f200640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f240c0039c0 0x7f24280ce6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:06.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:06.039+0000 7f242f200640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2428159600 0x7f2428158f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:06.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:06.039+0000 7f242f200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2428159b50 0x7f24281f72b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:06.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:06.039+0000 7f242f200640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f24280ae0f0 con 0x7f2428159b50 2024-03-31T15:12:06.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:06.039+0000 7f242f200640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f24280adf70 con 0x7f2428159600 2024-03-31T15:12:06.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:06.039+0000 7f242f200640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f24280b3050 con 0x7f240c0039c0 2024-03-31T15:12:06.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:06.039+0000 7f242d200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2428159b50 0x7f24281f72b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:06.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:06.039+0000 7f242d200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2428159b50 0x7f24281f72b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49110/0 (socket says 172.21.15.16:49110) 2024-03-31T15:12:06.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:06.039+0000 7f242d200640 1 -- 172.21.15.16:0/2371240426 learned_addr learned my addr 172.21.15.16:0/2371240426 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:12:06.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:06.039+0000 7f2427e00640 1 --2- 172.21.15.16:0/2371240426 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f240c0039c0 0x7f24280ce6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:06.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:06.039+0000 7f241f400640 1 --2- 172.21.15.16:0/2371240426 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2428159600 0x7f2428158f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:06.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:06.039+0000 7f242d200640 1 -- 172.21.15.16:0/2371240426 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f240c0039c0 msgr2=0x7f24280ce6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:06.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:06.039+0000 7f242d200640 1 --2- 172.21.15.16:0/2371240426 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f240c0039c0 0x7f24280ce6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:06.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:06.039+0000 7f242d200640 1 -- 172.21.15.16:0/2371240426 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2428159600 msgr2=0x7f2428158f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:06.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:06.039+0000 7f242d200640 1 --2- 172.21.15.16:0/2371240426 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2428159600 0x7f2428158f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:06.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:06.039+0000 7f242d200640 1 -- 172.21.15.16:0/2371240426 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24280cd620 con 0x7f2428159b50 2024-03-31T15:12:06.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:06.039+0000 7f241f400640 1 --2- 172.21.15.16:0/2371240426 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2428159600 0x7f2428158f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:12:06.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:06.043+0000 7f242d200640 1 --2- 172.21.15.16:0/2371240426 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2428159b50 0x7f24281f72b0 secure :-1 s=READY pgs=1355 cs=0 l=1 rev1=1 crypto rx=0x7f2428073520 tx=0x7f242000e670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:12:06.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:06.043+0000 7f2424c00640 1 -- 172.21.15.16:0/2371240426 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2420019070 con 0x7f2428159b50 2024-03-31T15:12:06.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:06.043+0000 7f2424c00640 1 -- 172.21.15.16:0/2371240426 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f24200052e0 con 0x7f2428159b50 2024-03-31T15:12:06.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:06.043+0000 7f2424c00640 1 -- 172.21.15.16:0/2371240426 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2420003840 con 0x7f2428159b50 2024-03-31T15:12:06.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:06.043+0000 7f242f200640 1 -- 172.21.15.16:0/2371240426 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f24280cd900 con 0x7f2428159b50 2024-03-31T15:12:06.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:06.043+0000 7f242f200640 1 -- 172.21.15.16:0/2371240426 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f24280cdec0 con 0x7f2428159b50 2024-03-31T15:12:06.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:06.043+0000 7f242f200640 1 -- 172.21.15.16:0/2371240426 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2428069950 con 0x7f2428159b50 2024-03-31T15:12:06.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:06.043+0000 7f2424c00640 1 -- 172.21.15.16:0/2371240426 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f242001c070 con 0x7f2428159b50 2024-03-31T15:12:06.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:06.043+0000 7f2424c00640 1 --2- 172.21.15.16:0/2371240426 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f2404068380 0x7f240406a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:06.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:06.043+0000 7f2424c00640 1 -- 172.21.15.16:0/2371240426 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 5996+0+0 (secure 0 0 0) 0x7f2420085d50 con 0x7f2428159b50 2024-03-31T15:12:06.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:06.047+0000 7f2427e00640 1 --2- 172.21.15.16:0/2371240426 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f2404068380 0x7f240406a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:06.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:06.047+0000 7f2427e00640 1 --2- 172.21.15.16:0/2371240426 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f2404068380 0x7f240406a840 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f24180ad000 tx=0x7f24180a7000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:12:06.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:06.047+0000 7f2424c00640 1 -- 172.21.15.16:0/2371240426 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f2420055b90 con 0x7f2428159b50 2024-03-31T15:12:06.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:06.219+0000 7f242f200640 1 -- 172.21.15.16:0/2371240426 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7f24280af090 con 0x7f2428159b50 2024-03-31T15:12:06.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:06.819+0000 7f2424c00640 1 -- 172.21.15.16:0/2371240426 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v87) ==== 181+0+0 (secure 0 0 0) 0x7f2420059e90 con 0x7f2428159b50 2024-03-31T15:12:06.812 INFO:teuthology.orchestra.run.smithi016.stderr:pool 'cephfs_metadata' removed 2024-03-31T15:12:06.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:06.823+0000 7f242f200640 1 -- 172.21.15.16:0/2371240426 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f2404068380 msgr2=0x7f240406a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:06.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:06.823+0000 7f242f200640 1 --2- 172.21.15.16:0/2371240426 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f2404068380 0x7f240406a840 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f24180ad000 tx=0x7f24180a7000 comp rx=0 tx=0).stop 2024-03-31T15:12:06.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:06.823+0000 7f242f200640 1 -- 172.21.15.16:0/2371240426 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2428159b50 msgr2=0x7f24281f72b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:06.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:06.823+0000 7f242f200640 1 --2- 172.21.15.16:0/2371240426 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2428159b50 0x7f24281f72b0 secure :-1 s=READY pgs=1355 cs=0 l=1 rev1=1 crypto rx=0x7f2428073520 tx=0x7f242000e670 comp rx=0 tx=0).stop 2024-03-31T15:12:06.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:06.823+0000 7f242f200640 1 -- 172.21.15.16:0/2371240426 shutdown_connections 2024-03-31T15:12:06.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:06.823+0000 7f242f200640 1 --2- 172.21.15.16:0/2371240426 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f2404068380 0x7f240406a840 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:06.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:06.823+0000 7f242f200640 1 --2- 172.21.15.16:0/2371240426 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2428159b50 0x7f24281f72b0 unknown :-1 s=CLOSED pgs=1355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:06.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:06.823+0000 7f242f200640 1 --2- 172.21.15.16:0/2371240426 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2428159600 0x7f2428158f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:06.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:06.823+0000 7f242f200640 1 --2- 172.21.15.16:0/2371240426 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f240c0039c0 0x7f24280ce6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:06.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:06.823+0000 7f242f200640 1 -- 172.21.15.16:0/2371240426 >> 172.21.15.16:0/2371240426 conn(0x7f242806b620 msgr2=0x7f24280ae270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:12:06.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:06.823+0000 7f242f200640 1 -- 172.21.15.16:0/2371240426 shutdown_connections 2024-03-31T15:12:06.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:06.823+0000 7f242f200640 1 -- 172.21.15.16:0/2371240426 wait complete. 2024-03-31T15:12:06.839 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-31T15:12:06.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:06.995+0000 7f7021000640 1 Processor -- start 2024-03-31T15:12:06.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:06.995+0000 7f7021000640 1 -- start start 2024-03-31T15:12:06.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:06.999+0000 7f7021000640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f701c13efd0 0x7f701c0b1be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:06.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:06.999+0000 7f7021000640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f701c1604f0 0x7f701c140550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:06.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:06.999+0000 7f7021000640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f701c13f7b0 con 0x7f701c0b2120 2024-03-31T15:12:06.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:06.999+0000 7f7021000640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f701c160940 con 0x7f701c1604f0 2024-03-31T15:12:06.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:06.999+0000 7f7021000640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f701c0b2560 con 0x7f701c13efd0 2024-03-31T15:12:06.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:06.999+0000 7f701a800640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f701c1604f0 0x7f701c140550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:06.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:06.999+0000 7f7013e00640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f701c0b2120 0x7f701c15fde0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.16:47072/0 (socket says 172.21.15.16:47072) 2024-03-31T15:12:06.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:06.999+0000 7f7013e00640 1 -- 172.21.15.16:0/2503084077 learned_addr learned my addr 172.21.15.16:0/2503084077 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:12:06.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:06.999+0000 7f7019e00640 1 --2- 172.21.15.16:0/2503084077 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f701c13efd0 0x7f701c0b1be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:06.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:06.999+0000 7f7019400640 1 -- 172.21.15.16:0/2503084077 <== mon.1 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2210033123 0 0) 0x7f701c13f7b0 con 0x7f701c0b2120 2024-03-31T15:12:06.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:06.999+0000 7f7019400640 1 -- 172.21.15.16:0/2503084077 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7000003530 con 0x7f701c0b2120 2024-03-31T15:12:06.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:06.999+0000 7f701a800640 1 -- 172.21.15.16:0/2503084077 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f701c13efd0 msgr2=0x7f701c0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:12:06.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:06.999+0000 7f701a800640 1 --2- 172.21.15.16:0/2503084077 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f701c13efd0 0x7f701c0b1be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:06.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:06.999+0000 7f701a800640 1 -- 172.21.15.16:0/2503084077 >> v1:172.21.15.89:6789/0 conn(0x7f701c0b2120 legacy=0x7f701c15fde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:06.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:06.999+0000 7f701a800640 1 -- 172.21.15.16:0/2503084077 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f701c160b80 con 0x7f701c1604f0 2024-03-31T15:12:06.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:06.999+0000 7f7019e00640 1 --2- 172.21.15.16:0/2503084077 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f701c13efd0 0x7f701c0b1be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:12:06.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:07.003+0000 7f701a800640 1 --2- 172.21.15.16:0/2503084077 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f701c1604f0 0x7f701c140550 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f7014009d90 tx=0x7f701401c940 comp rx=0 tx=0).ready entity=mon.1 client_cookie=783af8096dfa52d1 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:12:06.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:07.003+0000 7f7019400640 1 -- 172.21.15.16:0/2503084077 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70140b2070 con 0x7f701c1604f0 2024-03-31T15:12:06.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:07.003+0000 7f7019400640 1 -- 172.21.15.16:0/2503084077 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f70140a5a00 con 0x7f701c1604f0 2024-03-31T15:12:06.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:07.003+0000 7f7019400640 1 -- 172.21.15.16:0/2503084077 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70140a5dd0 con 0x7f701c1604f0 2024-03-31T15:12:06.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:07.003+0000 7f7021000640 1 -- 172.21.15.16:0/2503084077 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f701c1604f0 msgr2=0x7f701c140550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:06.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:07.003+0000 7f7021000640 1 --2- 172.21.15.16:0/2503084077 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f701c1604f0 0x7f701c140550 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f7014009d90 tx=0x7f701401c940 comp rx=0 tx=0).stop 2024-03-31T15:12:06.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:07.003+0000 7f7021000640 1 -- 172.21.15.16:0/2503084077 shutdown_connections 2024-03-31T15:12:06.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:07.003+0000 7f7021000640 1 --2- 172.21.15.16:0/2503084077 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f701c1604f0 0x7f701c140550 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:06.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:07.003+0000 7f7021000640 1 --2- 172.21.15.16:0/2503084077 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f701c13efd0 0x7f701c0b1be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:06.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:07.003+0000 7f7021000640 1 -- 172.21.15.16:0/2503084077 >> 172.21.15.16:0/2503084077 conn(0x7f701c06b620 msgr2=0x7f701c06ba30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:12:06.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:07.003+0000 7f7021000640 1 -- 172.21.15.16:0/2503084077 shutdown_connections 2024-03-31T15:12:06.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:07.003+0000 7f7021000640 1 -- 172.21.15.16:0/2503084077 wait complete. 2024-03-31T15:12:06.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:07.003+0000 7f7021000640 1 Processor -- start 2024-03-31T15:12:06.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:07.003+0000 7f7021000640 1 -- start start 2024-03-31T15:12:06.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:07.007+0000 7f7021000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f701c0b2120 0x7f701c13cd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:06.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:07.007+0000 7f7021000640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f701c13efd0 0x7f701c13a550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:06.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:07.007+0000 7f7021000640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f701c1604f0 0x7f701c13aa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:06.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:07.007+0000 7f7021000640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f701c13f3b0 con 0x7f701c0b2120 2024-03-31T15:12:06.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:07.007+0000 7f7021000640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f701c15db10 con 0x7f701c13efd0 2024-03-31T15:12:07.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:07.007+0000 7f7021000640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f701c13f530 con 0x7f701c1604f0 2024-03-31T15:12:07.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:07.007+0000 7f7019e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f701c0b2120 0x7f701c13cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:07.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:07.007+0000 7f7019e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f701c0b2120 0x7f701c13cd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49124/0 (socket says 172.21.15.16:49124) 2024-03-31T15:12:07.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:07.007+0000 7f7019e00640 1 -- 172.21.15.16:0/4234046881 learned_addr learned my addr 172.21.15.16:0/4234046881 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:12:07.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:07.007+0000 7f701a800640 1 --2- 172.21.15.16:0/4234046881 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f701c1604f0 0x7f701c13aa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:07.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:07.007+0000 7f7013e00640 1 --2- 172.21.15.16:0/4234046881 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f701c13efd0 0x7f701c13a550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:07.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:07.007+0000 7f7019e00640 1 -- 172.21.15.16:0/4234046881 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f701c1604f0 msgr2=0x7f701c13aa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:07.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:07.007+0000 7f7019e00640 1 --2- 172.21.15.16:0/4234046881 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f701c1604f0 0x7f701c13aa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:07.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:07.007+0000 7f7019e00640 1 -- 172.21.15.16:0/4234046881 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f701c13efd0 msgr2=0x7f701c13a550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:07.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:07.007+0000 7f7019e00640 1 --2- 172.21.15.16:0/4234046881 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f701c13efd0 0x7f701c13a550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:07.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:07.007+0000 7f7019e00640 1 -- 172.21.15.16:0/4234046881 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f701c13d5c0 con 0x7f701c0b2120 2024-03-31T15:12:07.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:07.007+0000 7f7019e00640 1 --2- 172.21.15.16:0/4234046881 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f701c0b2120 0x7f701c13cd70 secure :-1 s=READY pgs=1356 cs=0 l=1 rev1=1 crypto rx=0x7f700c00ec10 tx=0x7f700c00c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:12:07.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:07.007+0000 7f7012000640 1 -- 172.21.15.16:0/4234046881 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f700c017070 con 0x7f701c0b2120 2024-03-31T15:12:07.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:07.007+0000 7f7012000640 1 -- 172.21.15.16:0/4234046881 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f700c00cc30 con 0x7f701c0b2120 2024-03-31T15:12:07.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:07.007+0000 7f7021000640 1 -- 172.21.15.16:0/4234046881 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f701c13d8a0 con 0x7f701c0b2120 2024-03-31T15:12:07.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:07.007+0000 7f7012000640 1 -- 172.21.15.16:0/4234046881 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f700c013470 con 0x7f701c0b2120 2024-03-31T15:12:07.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:07.007+0000 7f7021000640 1 -- 172.21.15.16:0/4234046881 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f701c13de60 con 0x7f701c0b2120 2024-03-31T15:12:07.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:07.007+0000 7f7021000640 1 -- 172.21.15.16:0/4234046881 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f701c069950 con 0x7f701c0b2120 2024-03-31T15:12:07.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:07.011+0000 7f7012000640 1 -- 172.21.15.16:0/4234046881 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f700c01a070 con 0x7f701c0b2120 2024-03-31T15:12:07.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:07.011+0000 7f7012000640 1 --2- 172.21.15.16:0/4234046881 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f6ff8068380 0x7f6ff806a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:07.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:07.011+0000 7f7012000640 1 -- 172.21.15.16:0/4234046881 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 5558+0+0 (secure 0 0 0) 0x7f700c086c00 con 0x7f701c0b2120 2024-03-31T15:12:07.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:07.011+0000 7f7013e00640 1 --2- 172.21.15.16:0/4234046881 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f6ff8068380 0x7f6ff806a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:07.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:07.011+0000 7f7013e00640 1 --2- 172.21.15.16:0/4234046881 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f6ff8068380 0x7f6ff806a840 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f7004007fd0 tx=0x7f7004009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:12:07.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:07.015+0000 7f7012000640 1 -- 172.21.15.16:0/4234046881 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f700c056b60 con 0x7f701c0b2120 2024-03-31T15:12:07.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:07.191+0000 7f7021000640 1 -- 172.21.15.16:0/4234046881 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7f701c0b29e0 con 0x7f701c0b2120 2024-03-31T15:12:07.595 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-31T15:12:07.598 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-31T15:12:07.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:07.819+0000 7f7012000640 1 -- 172.21.15.16:0/4234046881 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v88) ==== 169+0+0 (secure 0 0 0) 0x7f700c05ae60 con 0x7f701c0b2120 2024-03-31T15:12:07.814 INFO:teuthology.orchestra.run.smithi016.stderr:pool 'cephfs_data' removed 2024-03-31T15:12:07.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:07.823+0000 7f7021000640 1 -- 172.21.15.16:0/4234046881 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f6ff8068380 msgr2=0x7f6ff806a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:07.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:07.823+0000 7f7021000640 1 --2- 172.21.15.16:0/4234046881 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f6ff8068380 0x7f6ff806a840 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f7004007fd0 tx=0x7f7004009040 comp rx=0 tx=0).stop 2024-03-31T15:12:07.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:07.823+0000 7f7021000640 1 -- 172.21.15.16:0/4234046881 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f701c0b2120 msgr2=0x7f701c13cd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:07.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:07.823+0000 7f7021000640 1 --2- 172.21.15.16:0/4234046881 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f701c0b2120 0x7f701c13cd70 secure :-1 s=READY pgs=1356 cs=0 l=1 rev1=1 crypto rx=0x7f700c00ec10 tx=0x7f700c00c520 comp rx=0 tx=0).stop 2024-03-31T15:12:07.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:07.823+0000 7f7021000640 1 -- 172.21.15.16:0/4234046881 shutdown_connections 2024-03-31T15:12:07.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:07.823+0000 7f7021000640 1 --2- 172.21.15.16:0/4234046881 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f6ff8068380 0x7f6ff806a840 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:07.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:07.823+0000 7f7021000640 1 --2- 172.21.15.16:0/4234046881 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f701c1604f0 0x7f701c13aa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:07.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:07.823+0000 7f7021000640 1 --2- 172.21.15.16:0/4234046881 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f701c13efd0 0x7f701c13a550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:07.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:07.823+0000 7f7021000640 1 --2- 172.21.15.16:0/4234046881 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f701c0b2120 0x7f701c13cd70 unknown :-1 s=CLOSED pgs=1356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:07.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:07.823+0000 7f7021000640 1 -- 172.21.15.16:0/4234046881 >> 172.21.15.16:0/4234046881 conn(0x7f701c06b620 msgr2=0x7f701c15fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:12:07.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:07.823+0000 7f7021000640 1 -- 172.21.15.16:0/4234046881 shutdown_connections 2024-03-31T15:12:07.818 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:07.823+0000 7f7021000640 1 -- 172.21.15.16:0/4234046881 wait complete. 2024-03-31T15:12:07.840 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-03-31T15:12:07.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:07.991+0000 7f01c5000640 1 Processor -- start 2024-03-31T15:12:07.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:07.995+0000 7f01c5000640 1 -- start start 2024-03-31T15:12:07.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:07.995+0000 7f01c5000640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f01c01411b0 0x7f01c006f920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:07.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:07.995+0000 7f01c5000640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f01c0141590 0x7f01c006fe60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:07.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:07.995+0000 7f01c5000640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f01c0144980 con 0x7f01c0143c30 2024-03-31T15:12:07.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:07.995+0000 7f01c5000640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f01c015f910 con 0x7f01c01411b0 2024-03-31T15:12:07.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:07.995+0000 7f01c5000640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f01c0162910 con 0x7f01c0141590 2024-03-31T15:12:07.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:07.995+0000 7f01bde00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f01c01411b0 0x7f01c006f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:07.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:07.995+0000 7f01be800640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f01c0143c30 0x7f01c01440a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:41258/0 (socket says 172.21.15.16:41258) 2024-03-31T15:12:07.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:07.995+0000 7f01be800640 1 -- 172.21.15.16:0/2887517547 learned_addr learned my addr 172.21.15.16:0/2887517547 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:12:07.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:07.995+0000 7f01bd400640 1 --2- 172.21.15.16:0/2887517547 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f01c0141590 0x7f01c006fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:07.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:07.995+0000 7f01bde00640 1 -- 172.21.15.16:0/2887517547 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f01c0141590 msgr2=0x7f01c006fe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:12:07.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:07.995+0000 7f01bde00640 1 --2- 172.21.15.16:0/2887517547 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f01c0141590 0x7f01c006fe60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:07.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:07.995+0000 7f01bde00640 1 -- 172.21.15.16:0/2887517547 >> v1:172.21.15.16:6789/0 conn(0x7f01c0143c30 legacy=0x7f01c01440a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:07.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:07.995+0000 7f01bde00640 1 -- 172.21.15.16:0/2887517547 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01c0161fe0 con 0x7f01c01411b0 2024-03-31T15:12:07.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:07.995+0000 7f01bca00640 1 -- 172.21.15.16:0/2887517547 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2127533375 0 0) 0x7f01c0144980 con 0x7f01c0143c30 2024-03-31T15:12:07.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:07.999+0000 7f01bde00640 1 --2- 172.21.15.16:0/2887517547 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f01c01411b0 0x7f01c006f920 secure :-1 s=READY pgs=1358 cs=0 l=1 rev1=1 crypto rx=0x7f01b00097c0 tx=0x7f01b001c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=13b910ece7288164 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:12:07.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:07.999+0000 7f01bca00640 1 -- 172.21.15.16:0/2887517547 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01b00b2070 con 0x7f01c01411b0 2024-03-31T15:12:07.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:07.999+0000 7f01bca00640 1 -- 172.21.15.16:0/2887517547 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f01b00ad640 con 0x7f01c01411b0 2024-03-31T15:12:07.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:07.999+0000 7f01bca00640 1 -- 172.21.15.16:0/2887517547 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01b00ada10 con 0x7f01c01411b0 2024-03-31T15:12:07.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:07.999+0000 7f01c5000640 1 -- 172.21.15.16:0/2887517547 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f01c01411b0 msgr2=0x7f01c006f920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:07.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:07.999+0000 7f01c5000640 1 --2- 172.21.15.16:0/2887517547 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f01c01411b0 0x7f01c006f920 secure :-1 s=READY pgs=1358 cs=0 l=1 rev1=1 crypto rx=0x7f01b00097c0 tx=0x7f01b001c4a0 comp rx=0 tx=0).stop 2024-03-31T15:12:07.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:07.999+0000 7f01c5000640 1 -- 172.21.15.16:0/2887517547 shutdown_connections 2024-03-31T15:12:07.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:07.999+0000 7f01c5000640 1 --2- 172.21.15.16:0/2887517547 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f01c0141590 0x7f01c006fe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:07.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:07.999+0000 7f01c5000640 1 --2- 172.21.15.16:0/2887517547 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f01c01411b0 0x7f01c006f920 unknown :-1 s=CLOSED pgs=1358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:07.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:07.999+0000 7f01c5000640 1 -- 172.21.15.16:0/2887517547 >> 172.21.15.16:0/2887517547 conn(0x7f01c0067930 msgr2=0x7f01c00b3ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:12:07.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:07.999+0000 7f01c5000640 1 -- 172.21.15.16:0/2887517547 shutdown_connections 2024-03-31T15:12:07.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:07.999+0000 7f01c5000640 1 -- 172.21.15.16:0/2887517547 wait complete. 2024-03-31T15:12:07.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:07.999+0000 7f01c5000640 1 Processor -- start 2024-03-31T15:12:07.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:07.999+0000 7f01c5000640 1 -- start start 2024-03-31T15:12:07.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:08.003+0000 7f01c5000640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f01c01411b0 0x7f01c0157120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:07.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:08.003+0000 7f01c5000640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f01c0141590 0x7f01c00b1e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:07.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:08.003+0000 7f01c5000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f01c0143c30 0x7f01c01537e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:07.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:08.003+0000 7f01c5000640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f01c015f290 con 0x7f01c0143c30 2024-03-31T15:12:07.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:08.003+0000 7f01c5000640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f01c015f110 con 0x7f01c01411b0 2024-03-31T15:12:07.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:08.003+0000 7f01c5000640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f01c015f410 con 0x7f01c0141590 2024-03-31T15:12:07.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:08.003+0000 7f01be800640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f01c0143c30 0x7f01c01537e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:07.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:08.003+0000 7f01be800640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f01c0143c30 0x7f01c01537e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37658/0 (socket says 172.21.15.16:37658) 2024-03-31T15:12:07.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:08.003+0000 7f01be800640 1 -- 172.21.15.16:0/2778239608 learned_addr learned my addr 172.21.15.16:0/2778239608 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:12:07.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:08.003+0000 7f01be800640 1 -- 172.21.15.16:0/2778239608 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f01c0141590 msgr2=0x7f01c00b1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:07.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:08.003+0000 7f01be800640 1 --2- 172.21.15.16:0/2778239608 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f01c0141590 0x7f01c00b1e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:07.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:08.003+0000 7f01be800640 1 -- 172.21.15.16:0/2778239608 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f01c01411b0 msgr2=0x7f01c0157120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:07.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:08.003+0000 7f01bde00640 1 --2- 172.21.15.16:0/2778239608 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f01c01411b0 0x7f01c0157120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:07.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:08.003+0000 7f01be800640 1 --2- 172.21.15.16:0/2778239608 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f01c01411b0 0x7f01c0157120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:07.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:08.003+0000 7f01be800640 1 -- 172.21.15.16:0/2778239608 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01c01577b0 con 0x7f01c0143c30 2024-03-31T15:12:07.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:08.003+0000 7f01bde00640 1 --2- 172.21.15.16:0/2778239608 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f01c01411b0 0x7f01c0157120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:12:07.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:08.003+0000 7f01be800640 1 --2- 172.21.15.16:0/2778239608 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f01c0143c30 0x7f01c01537e0 secure :-1 s=READY pgs=1359 cs=0 l=1 rev1=1 crypto rx=0x7f01b800c700 tx=0x7f01b800cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:12:07.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:08.003+0000 7f01b6000640 1 -- 172.21.15.16:0/2778239608 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01b8018070 con 0x7f01c0143c30 2024-03-31T15:12:07.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:08.003+0000 7f01b6000640 1 -- 172.21.15.16:0/2778239608 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f01b800ce10 con 0x7f01c0143c30 2024-03-31T15:12:07.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:08.003+0000 7f01b6000640 1 -- 172.21.15.16:0/2778239608 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01b800a880 con 0x7f01c0143c30 2024-03-31T15:12:07.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:08.003+0000 7f01c5000640 1 -- 172.21.15.16:0/2778239608 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f01c0157a90 con 0x7f01c0143c30 2024-03-31T15:12:07.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:08.003+0000 7f01c5000640 1 -- 172.21.15.16:0/2778239608 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f01c0152fe0 con 0x7f01c0143c30 2024-03-31T15:12:07.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:08.003+0000 7f01c5000640 1 -- 172.21.15.16:0/2778239608 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f01c0069950 con 0x7f01c0143c30 2024-03-31T15:12:07.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:08.003+0000 7f01b6000640 1 -- 172.21.15.16:0/2778239608 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f01b801b070 con 0x7f01c0143c30 2024-03-31T15:12:07.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:08.007+0000 7f01b6000640 1 --2- 172.21.15.16:0/2778239608 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f01a0068380 0x7f01a006a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:07.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:08.007+0000 7f01b6000640 1 -- 172.21.15.16:0/2778239608 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 5115+0+0 (secure 0 0 0) 0x7f01b8085d00 con 0x7f01c0143c30 2024-03-31T15:12:07.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:08.007+0000 7f01bde00640 1 --2- 172.21.15.16:0/2778239608 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f01a0068380 0x7f01a006a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:07.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:08.007+0000 7f01bde00640 1 --2- 172.21.15.16:0/2778239608 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f01a0068380 0x7f01a006a840 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f01b00af000 tx=0x7f01b0008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:12:08.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:08.011+0000 7f01b6000640 1 -- 172.21.15.16:0/2778239608 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f01b8056500 con 0x7f01c0143c30 2024-03-31T15:12:08.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:08.183+0000 7f01c5000640 1 -- 172.21.15.16:0/2778239608 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) -- 0x7f01c00b4a20 con 0x7f01c0143c30 2024-03-31T15:12:08.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:08.187+0000 7f01b6000640 1 -- 172.21.15.16:0/2778239608 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v19) ==== 104+0+0 (secure 0 0 0) 0x7f01b8055e00 con 0x7f01c0143c30 2024-03-31T15:12:08.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:08.187+0000 7f01c5000640 1 -- 172.21.15.16:0/2778239608 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f01a0068380 msgr2=0x7f01a006a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:08.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:08.187+0000 7f01c5000640 1 --2- 172.21.15.16:0/2778239608 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f01a0068380 0x7f01a006a840 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f01b00af000 tx=0x7f01b0008040 comp rx=0 tx=0).stop 2024-03-31T15:12:08.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:08.187+0000 7f01c5000640 1 -- 172.21.15.16:0/2778239608 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f01c0143c30 msgr2=0x7f01c01537e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:08.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:08.187+0000 7f01c5000640 1 --2- 172.21.15.16:0/2778239608 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f01c0143c30 0x7f01c01537e0 secure :-1 s=READY pgs=1359 cs=0 l=1 rev1=1 crypto rx=0x7f01b800c700 tx=0x7f01b800cbc0 comp rx=0 tx=0).stop 2024-03-31T15:12:08.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:08.187+0000 7f01c5000640 1 -- 172.21.15.16:0/2778239608 shutdown_connections 2024-03-31T15:12:08.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:08.187+0000 7f01c5000640 1 --2- 172.21.15.16:0/2778239608 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f01a0068380 0x7f01a006a840 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:08.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:08.187+0000 7f01c5000640 1 --2- 172.21.15.16:0/2778239608 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f01c0143c30 0x7f01c01537e0 unknown :-1 s=CLOSED pgs=1359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:08.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:08.187+0000 7f01c5000640 1 --2- 172.21.15.16:0/2778239608 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f01c0141590 0x7f01c00b1e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:08.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:08.187+0000 7f01c5000640 1 --2- 172.21.15.16:0/2778239608 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f01c01411b0 0x7f01c0157120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:08.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:08.187+0000 7f01c5000640 1 -- 172.21.15.16:0/2778239608 >> 172.21.15.16:0/2778239608 conn(0x7f01c0067930 msgr2=0x7f01c01612a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:12:08.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:08.187+0000 7f01c5000640 1 -- 172.21.15.16:0/2778239608 shutdown_connections 2024-03-31T15:12:08.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:08.187+0000 7f01c5000640 1 -- 172.21.15.16:0/2778239608 wait complete. 2024-03-31T15:12:08.200 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_health_status_after_dirfrag_repair' 2024-03-31T15:12:08.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:08.347+0000 7fd287200640 1 Processor -- start 2024-03-31T15:12:08.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:08.347+0000 7fd287200640 1 -- start start 2024-03-31T15:12:08.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:08.347+0000 7fd287200640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fd28006c1c0 0x7fd28015c8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:08.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:08.347+0000 7fd287200640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd2800d09f0 con 0x7fd28006b620 2024-03-31T15:12:08.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:08.347+0000 7fd287200640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd2800d0cd0 con 0x7fd280145aa0 2024-03-31T15:12:08.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:08.347+0000 7fd287200640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fd28006fd50 con 0x7fd28006c1c0 2024-03-31T15:12:08.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:08.347+0000 7fd285200640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fd28006b620 0x7fd2800cdab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:41266/0 (socket says 172.21.15.16:41266) 2024-03-31T15:12:08.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:08.347+0000 7fd285200640 1 -- 172.21.15.16:0/3757742472 learned_addr learned my addr 172.21.15.16:0/3757742472 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:12:08.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:08.347+0000 7fd277400640 1 --2- 172.21.15.16:0/3757742472 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fd28006c1c0 0x7fd28015c8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:08.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:08.347+0000 7fd27f400640 1 -- 172.21.15.16:0/3757742472 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 800886365 0 0) 0x7fd2800d09f0 con 0x7fd28006b620 2024-03-31T15:12:08.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:08.347+0000 7fd27f400640 1 -- 172.21.15.16:0/3757742472 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd264003530 con 0x7fd28006b620 2024-03-31T15:12:08.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:08.347+0000 7fd27f400640 1 -- 172.21.15.16:0/3757742472 <== mon.1 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 49998871 0 0) 0x7fd2800d0cd0 con 0x7fd280145aa0 2024-03-31T15:12:08.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:08.347+0000 7fd27f400640 1 -- 172.21.15.16:0/3757742472 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd2800d09f0 con 0x7fd280145aa0 2024-03-31T15:12:08.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:08.347+0000 7fd27f400640 1 -- 172.21.15.16:0/3757742472 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3514075725 0 0) 0x7fd264003530 con 0x7fd28006b620 2024-03-31T15:12:08.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:08.347+0000 7fd27f400640 1 -- 172.21.15.16:0/3757742472 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fd28006c1c0 msgr2=0x7fd28015c8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:12:08.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:08.347+0000 7fd27f400640 1 --2- 172.21.15.16:0/3757742472 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fd28006c1c0 0x7fd28015c8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:08.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:08.347+0000 7fd27f400640 1 -- 172.21.15.16:0/3757742472 >> v1:172.21.15.89:6789/0 conn(0x7fd280145aa0 legacy=0x7fd280072e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:08.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:08.347+0000 7fd27f400640 1 -- 172.21.15.16:0/3757742472 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2800d06e0 con 0x7fd28006b620 2024-03-31T15:12:08.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:08.347+0000 7fd27f400640 1 -- 172.21.15.16:0/3757742472 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 214807623 0 0) 0x7fd278001680 con 0x7fd28006b620 2024-03-31T15:12:08.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:08.351+0000 7fd27f400640 1 -- 172.21.15.16:0/3757742472 >> v1:172.21.15.16:6789/0 conn(0x7fd28006b620 legacy=0x7fd2800cdab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:08.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:08.351+0000 7fd27f400640 1 --2- 172.21.15.16:0/3757742472 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd264003b80 0x7fd264003f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:08.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:08.351+0000 7fd27f400640 1 --2- 172.21.15.16:0/3757742472 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fd28006c1c0 0x7fd264004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:08.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:08.351+0000 7fd27f400640 1 --2- 172.21.15.16:0/3757742472 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd264004d50 0x7fd2640051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:08.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:08.351+0000 7fd27f400640 1 -- 172.21.15.16:0/3757742472 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fd28006fd50 con 0x7fd264003b80 2024-03-31T15:12:08.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:08.351+0000 7fd27f400640 1 -- 172.21.15.16:0/3757742472 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fd26400ba80 con 0x7fd264004d50 2024-03-31T15:12:08.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:08.351+0000 7fd27f400640 1 -- 172.21.15.16:0/3757742472 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fd26400bc30 con 0x7fd28006c1c0 2024-03-31T15:12:08.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:08.351+0000 7fd27fe00640 1 --2- 172.21.15.16:0/3757742472 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd264003b80 0x7fd264003f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:08.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:08.351+0000 7fd285200640 1 --2- 172.21.15.16:0/3757742472 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd264004d50 0x7fd2640051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:08.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:08.351+0000 7fd277400640 1 --2- 172.21.15.16:0/3757742472 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fd28006c1c0 0x7fd264004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:08.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:08.351+0000 7fd27fe00640 1 -- 172.21.15.16:0/3757742472 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fd28006c1c0 msgr2=0x7fd264004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:12:08.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:08.351+0000 7fd27fe00640 1 --2- 172.21.15.16:0/3757742472 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fd28006c1c0 0x7fd264004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:08.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:08.351+0000 7fd27fe00640 1 -- 172.21.15.16:0/3757742472 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd264004d50 msgr2=0x7fd2640051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:12:08.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:08.351+0000 7fd27fe00640 1 --2- 172.21.15.16:0/3757742472 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd264004d50 0x7fd2640051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:08.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:08.351+0000 7fd27fe00640 1 -- 172.21.15.16:0/3757742472 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd264003530 con 0x7fd264003b80 2024-03-31T15:12:08.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:08.351+0000 7fd277400640 1 --2- 172.21.15.16:0/3757742472 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fd28006c1c0 0x7fd264004810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:12:08.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:08.351+0000 7fd27fe00640 1 --2- 172.21.15.16:0/3757742472 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd264003b80 0x7fd264003f80 secure :-1 s=READY pgs=1361 cs=0 l=1 rev1=1 crypto rx=0x7fd270001fd0 tx=0x7fd27001c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc92c9691f82039e server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:12:08.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:08.351+0000 7fd27f400640 1 -- 172.21.15.16:0/3757742472 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd270001c20 con 0x7fd264003b80 2024-03-31T15:12:08.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:08.351+0000 7fd27f400640 1 -- 172.21.15.16:0/3757742472 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd270003ba0 con 0x7fd264003b80 2024-03-31T15:12:08.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:08.351+0000 7fd27f400640 1 -- 172.21.15.16:0/3757742472 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2700050d0 con 0x7fd264003b80 2024-03-31T15:12:08.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:08.355+0000 7fd287200640 1 -- 172.21.15.16:0/3757742472 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd264003b80 msgr2=0x7fd264003f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:08.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:08.355+0000 7fd287200640 1 --2- 172.21.15.16:0/3757742472 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd264003b80 0x7fd264003f80 secure :-1 s=READY pgs=1361 cs=0 l=1 rev1=1 crypto rx=0x7fd270001fd0 tx=0x7fd27001c900 comp rx=0 tx=0).stop 2024-03-31T15:12:08.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:08.355+0000 7fd285200640 1 -- 172.21.15.16:0/3757742472 reap_dead start 2024-03-31T15:12:08.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:08.355+0000 7fd287200640 1 -- 172.21.15.16:0/3757742472 shutdown_connections 2024-03-31T15:12:08.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:08.355+0000 7fd287200640 1 -- 172.21.15.16:0/3757742472 >> 172.21.15.16:0/3757742472 conn(0x7fd2800d1060 msgr2=0x7fd2800d1470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:12:08.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:08.355+0000 7fd287200640 1 -- 172.21.15.16:0/3757742472 shutdown_connections 2024-03-31T15:12:08.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:08.355+0000 7fd287200640 1 -- 172.21.15.16:0/3757742472 wait complete. 2024-03-31T15:12:08.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:08.355+0000 7fd287200640 1 Processor -- start 2024-03-31T15:12:08.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:08.355+0000 7fd287200640 1 -- start start 2024-03-31T15:12:08.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:08.355+0000 7fd287200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd264003b80 0x7fd280143c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:08.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:08.355+0000 7fd287200640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd2801629e0 0x7fd2800cb060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:08.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:08.355+0000 7fd287200640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fd2800cb5a0 0x7fd2801f7a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:08.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:08.355+0000 7fd287200640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fd280168280 con 0x7fd264003b80 2024-03-31T15:12:08.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:08.355+0000 7fd287200640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fd280168100 con 0x7fd2801629e0 2024-03-31T15:12:08.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:08.355+0000 7fd287200640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fd280168400 con 0x7fd2800cb5a0 2024-03-31T15:12:08.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:08.355+0000 7fd27fe00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd264003b80 0x7fd280143c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:08.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:08.355+0000 7fd27fe00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd264003b80 0x7fd280143c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37662/0 (socket says 172.21.15.16:37662) 2024-03-31T15:12:08.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:08.355+0000 7fd27fe00640 1 -- 172.21.15.16:0/2082762973 learned_addr learned my addr 172.21.15.16:0/2082762973 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:12:08.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:08.355+0000 7fd285200640 1 --2- 172.21.15.16:0/2082762973 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fd2800cb5a0 0x7fd2801f7a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:08.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:08.355+0000 7fd277400640 1 --2- 172.21.15.16:0/2082762973 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd2801629e0 0x7fd2800cb060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:08.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:08.355+0000 7fd27fe00640 1 -- 172.21.15.16:0/2082762973 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fd2800cb5a0 msgr2=0x7fd2801f7a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:08.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:08.355+0000 7fd27fe00640 1 --2- 172.21.15.16:0/2082762973 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fd2800cb5a0 0x7fd2801f7a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:08.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:08.355+0000 7fd27fe00640 1 -- 172.21.15.16:0/2082762973 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd2801629e0 msgr2=0x7fd2800cb060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:08.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:08.355+0000 7fd27fe00640 1 --2- 172.21.15.16:0/2082762973 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd2801629e0 0x7fd2800cb060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:08.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:08.355+0000 7fd27fe00640 1 -- 172.21.15.16:0/2082762973 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2800cc8e0 con 0x7fd264003b80 2024-03-31T15:12:08.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:08.355+0000 7fd277400640 1 --2- 172.21.15.16:0/2082762973 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd2801629e0 0x7fd2800cb060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:12:08.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:08.359+0000 7fd27fe00640 1 --2- 172.21.15.16:0/2082762973 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd264003b80 0x7fd280143c80 secure :-1 s=READY pgs=1362 cs=0 l=1 rev1=1 crypto rx=0x7fd27001cde0 tx=0x7fd264005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:12:08.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:08.359+0000 7fd27cc00640 1 -- 172.21.15.16:0/2082762973 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2700bf070 con 0x7fd264003b80 2024-03-31T15:12:08.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:08.359+0000 7fd27cc00640 1 -- 172.21.15.16:0/2082762973 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd270001c00 con 0x7fd264003b80 2024-03-31T15:12:08.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:08.359+0000 7fd27cc00640 1 -- 172.21.15.16:0/2082762973 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd270006b60 con 0x7fd264003b80 2024-03-31T15:12:08.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:08.359+0000 7fd287200640 1 -- 172.21.15.16:0/2082762973 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd2800b5a30 con 0x7fd264003b80 2024-03-31T15:12:08.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:08.359+0000 7fd287200640 1 -- 172.21.15.16:0/2082762973 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd2800cc0e0 con 0x7fd264003b80 2024-03-31T15:12:08.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:08.359+0000 7fd287200640 1 -- 172.21.15.16:0/2082762973 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd280069950 con 0x7fd264003b80 2024-03-31T15:12:08.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:08.359+0000 7fd27cc00640 1 -- 172.21.15.16:0/2082762973 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fd2700c2070 con 0x7fd264003b80 2024-03-31T15:12:08.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:08.359+0000 7fd27cc00640 1 --2- 172.21.15.16:0/2082762973 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fd25c068380 0x7fd25c06a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:08.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:08.359+0000 7fd27cc00640 1 -- 172.21.15.16:0/2082762973 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 5115+0+0 (secure 0 0 0) 0x7fd270120a70 con 0x7fd264003b80 2024-03-31T15:12:08.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:08.359+0000 7fd277400640 1 --2- 172.21.15.16:0/2082762973 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fd25c068380 0x7fd25c06a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:08.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:08.363+0000 7fd277400640 1 --2- 172.21.15.16:0/2082762973 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fd25c068380 0x7fd25c06a840 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7fd2800cd080 tx=0x7fd268008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:12:08.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:08.363+0000 7fd27cc00640 1 -- 172.21.15.16:0/2082762973 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fd2800cc8e0 con 0x7fd264003b80 2024-03-31T15:12:08.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:08.587+0000 7fd287200640 1 -- 172.21.15.16:0/2082762973 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_health_status_after_dirfrag_repair"]} v 0) -- 0x7fd28006f4b0 con 0x7fd264003b80 2024-03-31T15:12:08.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:08.831+0000 7fd27cc00640 1 -- 172.21.15.16:0/2082762973 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_health_status_after_dirfrag_repair"]}]=0 v402) ==== 167+0+0 (secure 0 0 0) 0x7fd2700b7070 con 0x7fd264003b80 2024-03-31T15:12:08.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:08.835+0000 7fd287200640 1 -- 172.21.15.16:0/2082762973 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fd25c068380 msgr2=0x7fd25c06a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:08.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:08.835+0000 7fd287200640 1 --2- 172.21.15.16:0/2082762973 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fd25c068380 0x7fd25c06a840 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7fd2800cd080 tx=0x7fd268008040 comp rx=0 tx=0).stop 2024-03-31T15:12:08.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:08.835+0000 7fd287200640 1 -- 172.21.15.16:0/2082762973 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd264003b80 msgr2=0x7fd280143c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:08.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:08.835+0000 7fd287200640 1 --2- 172.21.15.16:0/2082762973 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd264003b80 0x7fd280143c80 secure :-1 s=READY pgs=1362 cs=0 l=1 rev1=1 crypto rx=0x7fd27001cde0 tx=0x7fd264005840 comp rx=0 tx=0).stop 2024-03-31T15:12:08.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:08.835+0000 7fd287200640 1 -- 172.21.15.16:0/2082762973 shutdown_connections 2024-03-31T15:12:08.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:08.835+0000 7fd287200640 1 --2- 172.21.15.16:0/2082762973 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fd25c068380 0x7fd25c06a840 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:08.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:08.835+0000 7fd287200640 1 --2- 172.21.15.16:0/2082762973 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fd2800cb5a0 0x7fd2801f7a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:08.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:08.835+0000 7fd287200640 1 --2- 172.21.15.16:0/2082762973 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd2801629e0 0x7fd2800cb060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:08.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:08.835+0000 7fd287200640 1 --2- 172.21.15.16:0/2082762973 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd264003b80 0x7fd280143c80 unknown :-1 s=CLOSED pgs=1362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:08.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:08.835+0000 7fd287200640 1 -- 172.21.15.16:0/2082762973 >> 172.21.15.16:0/2082762973 conn(0x7fd2800d1060 msgr2=0x7fd28015ce00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:12:08.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:08.835+0000 7fd287200640 1 -- 172.21.15.16:0/2082762973 shutdown_connections 2024-03-31T15:12:08.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:08.835+0000 7fd287200640 1 -- 172.21.15.16:0/2082762973 wait complete. 2024-03-31T15:12:08.846 INFO:tasks.cephfs_test_runner:Test that the damage health status is cleared ... ok 2024-03-31T15:12:08.847 INFO:tasks.cephfs_test_runner:Starting test: test_inotable_sync (tasks.cephfs.test_forward_scrub.TestForwardScrub) 2024-03-31T15:12:08.847 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-31T15:12:08.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:08.995+0000 7f2a7ea00640 1 Processor -- start 2024-03-31T15:12:08.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:08.995+0000 7f2a7ea00640 1 -- start start 2024-03-31T15:12:08.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:08.999+0000 7f2a7ea00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f2a7809b170 0x7f2a780ccb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:08.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:08.999+0000 7f2a7ea00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f2a78070d50 0x7f2a780cd0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:08.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:08.999+0000 7f2a7ea00640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2a780d0800 con 0x7f2a7815e3b0 2024-03-31T15:12:08.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:08.999+0000 7f2a7ea00640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f2a7815e790 con 0x7f2a78070d50 2024-03-31T15:12:08.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:08.999+0000 7f2a7ea00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f2a7806f5d0 con 0x7f2a7809b170 2024-03-31T15:12:08.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:08.999+0000 7f2a7e000640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f2a78070d50 0x7f2a780cd0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:08.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:08.999+0000 7f2a7e000640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f2a78070d50 0x7f2a780cd0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37676/0 (socket says 172.21.15.16:37676) 2024-03-31T15:12:08.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:08.999+0000 7f2a7e000640 1 -- 172.21.15.16:0/1540691527 learned_addr learned my addr 172.21.15.16:0/1540691527 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:12:08.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:08.999+0000 7f2a7cc00640 1 --2- 172.21.15.16:0/1540691527 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f2a7809b170 0x7f2a780ccb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:08.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:08.999+0000 7f2a7e000640 1 -- 172.21.15.16:0/1540691527 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f2a7809b170 msgr2=0x7f2a780ccb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:12:08.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:08.999+0000 7f2a7e000640 1 --2- 172.21.15.16:0/1540691527 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f2a7809b170 0x7f2a780ccb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:08.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:08.999+0000 7f2a7e000640 1 -- 172.21.15.16:0/1540691527 >> v1:172.21.15.89:6789/0 conn(0x7f2a7815e3b0 legacy=0x7f2a7809aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:08.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:08.999+0000 7f2a7e000640 1 -- 172.21.15.16:0/1540691527 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a780d0ae0 con 0x7f2a78070d50 2024-03-31T15:12:08.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:09.003+0000 7f2a7e000640 1 --2- 172.21.15.16:0/1540691527 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2a78070d50 0x7f2a780cd0d0 secure :-1 s=READY pgs=1363 cs=0 l=1 rev1=1 crypto rx=0x7f2a70009e50 tx=0x7f2a7001ca00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=33950f5daff401bc server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:12:08.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:09.003+0000 7f2a77e00640 1 -- 172.21.15.16:0/1540691527 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a700a58d0 con 0x7f2a78070d50 2024-03-31T15:12:08.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:09.003+0000 7f2a77e00640 1 -- 172.21.15.16:0/1540691527 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2a700a5a70 con 0x7f2a78070d50 2024-03-31T15:12:08.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:09.003+0000 7f2a77e00640 1 -- 172.21.15.16:0/1540691527 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a700a5e40 con 0x7f2a78070d50 2024-03-31T15:12:08.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:09.003+0000 7f2a7ea00640 1 -- 172.21.15.16:0/1540691527 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2a78070d50 msgr2=0x7f2a780cd0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:08.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:09.003+0000 7f2a7ea00640 1 --2- 172.21.15.16:0/1540691527 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2a78070d50 0x7f2a780cd0d0 secure :-1 s=READY pgs=1363 cs=0 l=1 rev1=1 crypto rx=0x7f2a70009e50 tx=0x7f2a7001ca00 comp rx=0 tx=0).stop 2024-03-31T15:12:08.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:09.003+0000 7f2a7ea00640 1 -- 172.21.15.16:0/1540691527 shutdown_connections 2024-03-31T15:12:08.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:09.003+0000 7f2a7ea00640 1 --2- 172.21.15.16:0/1540691527 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2a78070d50 0x7f2a780cd0d0 unknown :-1 s=CLOSED pgs=1363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:08.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:09.003+0000 7f2a7ea00640 1 --2- 172.21.15.16:0/1540691527 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f2a7809b170 0x7f2a780ccb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:08.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:09.003+0000 7f2a7ea00640 1 -- 172.21.15.16:0/1540691527 >> 172.21.15.16:0/1540691527 conn(0x7f2a780b39c0 msgr2=0x7f2a780b3dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:12:08.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:09.003+0000 7f2a7ea00640 1 -- 172.21.15.16:0/1540691527 shutdown_connections 2024-03-31T15:12:08.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:09.003+0000 7f2a7ea00640 1 -- 172.21.15.16:0/1540691527 wait complete. 2024-03-31T15:12:08.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:09.003+0000 7f2a7ea00640 1 Processor -- start 2024-03-31T15:12:08.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:09.003+0000 7f2a7ea00640 1 -- start start 2024-03-31T15:12:08.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:09.003+0000 7f2a7ea00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2a78070d50 0x7f2a78152bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:08.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:09.003+0000 7f2a7ea00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f2a7809b170 0x7f2a780c60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:08.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:09.003+0000 7f2a7ea00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2a7815e3b0 0x7f2a7815c600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:08.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:09.003+0000 7f2a7ea00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f2a7816b010 con 0x7f2a78070d50 2024-03-31T15:12:08.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:09.003+0000 7f2a7ea00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f2a7816ae90 con 0x7f2a7815e3b0 2024-03-31T15:12:08.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:09.003+0000 7f2a7ea00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f2a7816b190 con 0x7f2a7809b170 2024-03-31T15:12:08.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:09.003+0000 7f2a7d600640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2a78070d50 0x7f2a78152bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:08.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:09.007+0000 7f2a7d600640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2a78070d50 0x7f2a78152bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37678/0 (socket says 172.21.15.16:37678) 2024-03-31T15:12:08.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:09.007+0000 7f2a7d600640 1 -- 172.21.15.16:0/2947969954 learned_addr learned my addr 172.21.15.16:0/2947969954 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:12:09.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:09.007+0000 7f2a7e000640 1 --2- 172.21.15.16:0/2947969954 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2a7815e3b0 0x7f2a7815c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:09.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:09.007+0000 7f2a7cc00640 1 --2- 172.21.15.16:0/2947969954 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f2a7809b170 0x7f2a780c60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:09.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:09.007+0000 7f2a7d600640 1 -- 172.21.15.16:0/2947969954 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f2a7809b170 msgr2=0x7f2a780c60f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:09.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:09.007+0000 7f2a7d600640 1 --2- 172.21.15.16:0/2947969954 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f2a7809b170 0x7f2a780c60f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:09.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:09.007+0000 7f2a7d600640 1 -- 172.21.15.16:0/2947969954 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2a7815e3b0 msgr2=0x7f2a7815c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:09.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:09.007+0000 7f2a7d600640 1 --2- 172.21.15.16:0/2947969954 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2a7815e3b0 0x7f2a7815c600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:09.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:09.007+0000 7f2a7d600640 1 -- 172.21.15.16:0/2947969954 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a78154ac0 con 0x7f2a78070d50 2024-03-31T15:12:09.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:09.007+0000 7f2a7d600640 1 --2- 172.21.15.16:0/2947969954 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2a78070d50 0x7f2a78152bc0 secure :-1 s=READY pgs=1364 cs=0 l=1 rev1=1 crypto rx=0x7f2a6c00c3a0 tx=0x7f2a6c00c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:12:09.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:09.007+0000 7f2a75600640 1 -- 172.21.15.16:0/2947969954 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a6c0052d0 con 0x7f2a78070d50 2024-03-31T15:12:09.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:09.007+0000 7f2a75600640 1 -- 172.21.15.16:0/2947969954 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2a6c005470 con 0x7f2a78070d50 2024-03-31T15:12:09.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:09.007+0000 7f2a75600640 1 -- 172.21.15.16:0/2947969954 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a6c007260 con 0x7f2a78070d50 2024-03-31T15:12:09.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:09.007+0000 7f2a7ea00640 1 -- 172.21.15.16:0/2947969954 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2a78153e60 con 0x7f2a78070d50 2024-03-31T15:12:09.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:09.007+0000 7f2a7ea00640 1 -- 172.21.15.16:0/2947969954 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2a78154420 con 0x7f2a78070d50 2024-03-31T15:12:09.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:09.007+0000 7f2a7ea00640 1 -- 172.21.15.16:0/2947969954 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2a7804f8c0 con 0x7f2a78070d50 2024-03-31T15:12:09.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:09.007+0000 7f2a75600640 1 -- 172.21.15.16:0/2947969954 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f2a6c007400 con 0x7f2a78070d50 2024-03-31T15:12:09.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:09.011+0000 7f2a75600640 1 --2- 172.21.15.16:0/2947969954 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f2a58068380 0x7f2a5806a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:09.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:09.011+0000 7f2a75600640 1 -- 172.21.15.16:0/2947969954 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 5115+0+0 (secure 0 0 0) 0x7f2a6c085b40 con 0x7f2a78070d50 2024-03-31T15:12:09.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:09.011+0000 7f2a7cc00640 1 --2- 172.21.15.16:0/2947969954 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f2a58068380 0x7f2a5806a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:09.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:09.011+0000 7f2a7cc00640 1 --2- 172.21.15.16:0/2947969954 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f2a58068380 0x7f2a5806a840 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f2a780c5c00 tx=0x7f2a64008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:12:09.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:09.015+0000 7f2a75600640 1 -- 172.21.15.16:0/2947969954 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f2a6c055ce0 con 0x7f2a78070d50 2024-03-31T15:12:09.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:09.183+0000 7f2a7ea00640 1 -- 172.21.15.16:0/2947969954 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f2a78068ae0 con 0x7f2a78070d50 2024-03-31T15:12:09.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:09.183+0000 7f2a75600640 1 -- 172.21.15.16:0/2947969954 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) ==== 74+0+13207 (secure 0 0 0) 0x7f2a6c059fe0 con 0x7f2a78070d50 2024-03-31T15:12:09.175 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:12:09.175 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":88,"fsid":"286eb5da-fb47-4550-b290-d9519c13b9a5","created":"2024-03-31T15:03:11.438952+0000","modified":"2024-03-31T15:12:07.823304+0000","last_up_change":"2024-03-31T15:03:22.885535+0000","last_in_change":"2024-03-31T15:03:13.941667+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-31T15:03:22.550787+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"92a66450-ef19-40d8-ac3d-d760b23b0bda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6801","nonce":932712244}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6803","nonce":932712244}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6807","nonce":932712244}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6805","nonce":932712244}]},"public_addr":"172.21.15.16:6801/932712244","cluster_addr":"172.21.15.16:6803/932712244","heartbeat_back_addr":"172.21.15.16:6807/932712244","heartbeat_front_addr":"172.21.15.16:6805/932712244","state":["exists","up"]},{"osd":1,"uuid":"da9d1f2b-1968-4959-b843-faa231988e92","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6810","nonce":1398737533}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6811","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6813","nonce":1398737533}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6821","nonce":1398737533}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6815","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6816","nonce":1398737533}]},"public_addr":"172.21.15.16:6810/1398737533","cluster_addr":"172.21.15.16:6813/1398737533","heartbeat_back_addr":"172.21.15.16:6821/1398737533","heartbeat_front_addr":"172.21.15.16:6816/1398737533","state":["exists","up"]},{"osd":2,"uuid":"01a73662-9a21-4b1f-bcbb-42b9636a4638","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6819","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6822","nonce":3018526646}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6825","nonce":3018526646}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6831","nonce":3018526646}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6827","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6829","nonce":3018526646}]},"public_addr":"172.21.15.16:6822/3018526646","cluster_addr":"172.21.15.16:6825/3018526646","heartbeat_back_addr":"172.21.15.16:6831/3018526646","heartbeat_front_addr":"172.21.15.16:6829/3018526646","state":["exists","up"]},{"osd":3,"uuid":"c7771fd3-e18e-4500-ac38-23cf10ebcd75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6809","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6812","nonce":3360352974}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6817","nonce":3360352974}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6828","nonce":3360352974}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6823","nonce":3360352974}]},"public_addr":"172.21.15.16:6812/3360352974","cluster_addr":"172.21.15.16:6817/3360352974","heartbeat_back_addr":"172.21.15.16:6828/3360352974","heartbeat_front_addr":"172.21.15.16:6823/3360352974","state":["exists","up"]},{"osd":4,"uuid":"88260730-1482-4fbd-a40f-cb1124e9df4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6801","nonce":1856844856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6803","nonce":1856844856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6807","nonce":1856844856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6805","nonce":1856844856}]},"public_addr":"172.21.15.89:6801/1856844856","cluster_addr":"172.21.15.89:6803/1856844856","heartbeat_back_addr":"172.21.15.89:6807/1856844856","heartbeat_front_addr":"172.21.15.89:6805/1856844856","state":["exists","up"]},{"osd":5,"uuid":"4cb2c00c-fae0-4dd8-9d27-ec4f0093ccf7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6814","nonce":50395527}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6817","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6820","nonce":50395527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6831","nonce":50395527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6823","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6826","nonce":50395527}]},"public_addr":"172.21.15.89:6814/50395527","cluster_addr":"172.21.15.89:6820/50395527","heartbeat_back_addr":"172.21.15.89:6831/50395527","heartbeat_front_addr":"172.21.15.89:6826/50395527","state":["exists","up"]},{"osd":6,"uuid":"81ce0c19-2554-4cae-acbe-66a05e9e9b9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6809","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6812","nonce":3470471752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6815","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6818","nonce":3470471752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6827","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6829","nonce":3470471752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6821","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6824","nonce":3470471752}]},"public_addr":"172.21.15.89:6812/3470471752","cluster_addr":"172.21.15.89:6818/3470471752","heartbeat_back_addr":"172.21.15.89:6829/3470471752","heartbeat_front_addr":"172.21.15.89:6824/3470471752","state":["exists","up"]},{"osd":7,"uuid":"85e4313b-d37d-442e-8042-153bf20a847d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6811","nonce":1796778630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6813","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6816","nonce":1796778630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6825","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6828","nonce":1796778630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6819","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6822","nonce":1796778630}]},"public_addr":"172.21.15.89:6811/1796778630","cluster_addr":"172.21.15.89:6816/1796778630","heartbeat_back_addr":"172.21.15.89:6828/1796778630","heartbeat_front_addr":"172.21.15.89:6822/1796778630","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.439088+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.529970+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.680512+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:21.044161+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.582111+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.536558+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.409522+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.590229+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:6834/3780866829":"2024-04-01T15:12:04.817187+0000","172.21.15.16:6832/3780866829":"2024-04-01T15:12:04.817187+0000","172.21.15.16:6832/818074038":"2024-04-01T15:11:19.402687+0000","172.21.15.16:6834/818074038":"2024-04-01T15:11:19.402687+0000","172.21.15.16:6834/2478530200":"2024-04-01T15:10:04.360772+0000","172.21.15.16:6832/2478530200":"2024-04-01T15:10:04.360772+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-31T15:12:09.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:09.183+0000 7f2a7ea00640 1 -- 172.21.15.16:0/2947969954 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f2a58068380 msgr2=0x7f2a5806a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:09.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:09.183+0000 7f2a7ea00640 1 --2- 172.21.15.16:0/2947969954 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f2a58068380 0x7f2a5806a840 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f2a780c5c00 tx=0x7f2a64008040 comp rx=0 tx=0).stop 2024-03-31T15:12:09.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:09.183+0000 7f2a7ea00640 1 -- 172.21.15.16:0/2947969954 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2a78070d50 msgr2=0x7f2a78152bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:09.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:09.183+0000 7f2a7ea00640 1 --2- 172.21.15.16:0/2947969954 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2a78070d50 0x7f2a78152bc0 secure :-1 s=READY pgs=1364 cs=0 l=1 rev1=1 crypto rx=0x7f2a6c00c3a0 tx=0x7f2a6c00c860 comp rx=0 tx=0).stop 2024-03-31T15:12:09.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:09.183+0000 7f2a7ea00640 1 -- 172.21.15.16:0/2947969954 shutdown_connections 2024-03-31T15:12:09.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:09.183+0000 7f2a7ea00640 1 --2- 172.21.15.16:0/2947969954 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f2a58068380 0x7f2a5806a840 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:09.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:09.183+0000 7f2a7ea00640 1 --2- 172.21.15.16:0/2947969954 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2a7815e3b0 0x7f2a7815c600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:09.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:09.183+0000 7f2a7ea00640 1 --2- 172.21.15.16:0/2947969954 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f2a7809b170 0x7f2a780c60f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:09.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:09.183+0000 7f2a7ea00640 1 --2- 172.21.15.16:0/2947969954 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2a78070d50 0x7f2a78152bc0 unknown :-1 s=CLOSED pgs=1364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:09.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:09.183+0000 7f2a7ea00640 1 -- 172.21.15.16:0/2947969954 >> 172.21.15.16:0/2947969954 conn(0x7f2a780b39c0 msgr2=0x7f2a7809aa60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:12:09.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:09.183+0000 7f2a7ea00640 1 -- 172.21.15.16:0/2947969954 shutdown_connections 2024-03-31T15:12:09.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:09.183+0000 7f2a7ea00640 1 -- 172.21.15.16:0/2947969954 wait complete. 2024-03-31T15:12:09.198 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-31T15:03:22.550787+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-31T15:12:09.199 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-31T15:12:09.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:09.351+0000 7f9770200640 1 Processor -- start 2024-03-31T15:12:09.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:09.351+0000 7f9770200640 1 -- start start 2024-03-31T15:12:09.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:09.351+0000 7f9770200640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f976813dff0 0x7f976806d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:09.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:09.351+0000 7f9770200640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f97680cd0e0 0x7f976806da10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:09.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:09.351+0000 7f9770200640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f976815fbc0 con 0x7f976813e3d0 2024-03-31T15:12:09.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:09.351+0000 7f9770200640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f976815f730 con 0x7f976813dff0 2024-03-31T15:12:09.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:09.351+0000 7f9770200640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f976815ee00 con 0x7f97680cd0e0 2024-03-31T15:12:09.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:09.351+0000 7f976d800640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f976813dff0 0x7f976806d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:09.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:09.351+0000 7f976d800640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f976813dff0 0x7f976806d4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37692/0 (socket says 172.21.15.16:37692) 2024-03-31T15:12:09.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:09.351+0000 7f976d800640 1 -- 172.21.15.16:0/1809291188 learned_addr learned my addr 172.21.15.16:0/1809291188 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:12:09.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:09.351+0000 7f976e200640 1 --2- 172.21.15.16:0/1809291188 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f97680cd0e0 0x7f976806da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:09.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:09.351+0000 7f976d800640 1 -- 172.21.15.16:0/1809291188 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f97680cd0e0 msgr2=0x7f976806da10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:12:09.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:09.351+0000 7f976d800640 1 --2- 172.21.15.16:0/1809291188 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f97680cd0e0 0x7f976806da10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:09.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:09.351+0000 7f976d800640 1 -- 172.21.15.16:0/1809291188 >> v1:172.21.15.89:6789/0 conn(0x7f976813e3d0 legacy=0x7f97680cc9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:09.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:09.351+0000 7f976d800640 1 -- 172.21.15.16:0/1809291188 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f976815f0e0 con 0x7f976813dff0 2024-03-31T15:12:09.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:09.355+0000 7f976d800640 1 --2- 172.21.15.16:0/1809291188 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f976813dff0 0x7f976806d4d0 secure :-1 s=READY pgs=1365 cs=0 l=1 rev1=1 crypto rx=0x7f975c009870 tx=0x7f975c01c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c266f932e36d4cb server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:12:09.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:09.355+0000 7f9767e00640 1 -- 172.21.15.16:0/1809291188 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f975c0b1070 con 0x7f976813dff0 2024-03-31T15:12:09.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:09.355+0000 7f9767e00640 1 -- 172.21.15.16:0/1809291188 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f975c0a8050 con 0x7f976813dff0 2024-03-31T15:12:09.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:09.355+0000 7f9767e00640 1 -- 172.21.15.16:0/1809291188 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f975c0ac860 con 0x7f976813dff0 2024-03-31T15:12:09.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:09.355+0000 7f9770200640 1 -- 172.21.15.16:0/1809291188 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f976813dff0 msgr2=0x7f976806d4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:09.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:09.355+0000 7f9770200640 1 --2- 172.21.15.16:0/1809291188 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f976813dff0 0x7f976806d4d0 secure :-1 s=READY pgs=1365 cs=0 l=1 rev1=1 crypto rx=0x7f975c009870 tx=0x7f975c01c4d0 comp rx=0 tx=0).stop 2024-03-31T15:12:09.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:09.355+0000 7f9770200640 1 -- 172.21.15.16:0/1809291188 shutdown_connections 2024-03-31T15:12:09.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:09.355+0000 7f9770200640 1 --2- 172.21.15.16:0/1809291188 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f97680cd0e0 0x7f976806da10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:09.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:09.355+0000 7f9770200640 1 --2- 172.21.15.16:0/1809291188 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f976813dff0 0x7f976806d4d0 unknown :-1 s=CLOSED pgs=1365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:09.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:09.355+0000 7f9770200640 1 -- 172.21.15.16:0/1809291188 >> 172.21.15.16:0/1809291188 conn(0x7f976806ee40 msgr2=0x7f976806b620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:12:09.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:09.355+0000 7f9770200640 1 -- 172.21.15.16:0/1809291188 shutdown_connections 2024-03-31T15:12:09.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:09.355+0000 7f9770200640 1 -- 172.21.15.16:0/1809291188 wait complete. 2024-03-31T15:12:09.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:09.355+0000 7f9770200640 1 Processor -- start 2024-03-31T15:12:09.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:09.355+0000 7f9770200640 1 -- start start 2024-03-31T15:12:09.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:09.359+0000 7f9770200640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f97680cd0e0 0x7f97680b4250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:09.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:09.359+0000 7f9770200640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f976813dff0 0x7f9768165720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:09.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:09.359+0000 7f9770200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f976813e3d0 0x7f97681f8210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:09.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:09.359+0000 7f9770200640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f97680cf710 con 0x7f976813e3d0 2024-03-31T15:12:09.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:09.359+0000 7f9770200640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f97680cf590 con 0x7f97680cd0e0 2024-03-31T15:12:09.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:09.359+0000 7f9770200640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f97680cf890 con 0x7f976813dff0 2024-03-31T15:12:09.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:09.359+0000 7f976e200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f976813e3d0 0x7f97681f8210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:09.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:09.359+0000 7f976d800640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f97680cd0e0 0x7f97680b4250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:09.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:09.359+0000 7f976e200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f976813e3d0 0x7f97681f8210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37694/0 (socket says 172.21.15.16:37694) 2024-03-31T15:12:09.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:09.359+0000 7f976e200640 1 -- 172.21.15.16:0/696833930 learned_addr learned my addr 172.21.15.16:0/696833930 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:12:09.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:09.359+0000 7f976ce00640 1 --2- 172.21.15.16:0/696833930 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f976813dff0 0x7f9768165720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:09.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:09.359+0000 7f976e200640 1 -- 172.21.15.16:0/696833930 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f976813dff0 msgr2=0x7f9768165720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:09.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:09.359+0000 7f976e200640 1 --2- 172.21.15.16:0/696833930 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f976813dff0 0x7f9768165720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:09.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:09.359+0000 7f976e200640 1 -- 172.21.15.16:0/696833930 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f97680cd0e0 msgr2=0x7f97680b4250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:09.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:09.359+0000 7f976e200640 1 --2- 172.21.15.16:0/696833930 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f97680cd0e0 0x7f97680b4250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:09.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:09.359+0000 7f976e200640 1 -- 172.21.15.16:0/696833930 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97681f8750 con 0x7f976813e3d0 2024-03-31T15:12:09.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:09.359+0000 7f976d800640 1 --2- 172.21.15.16:0/696833930 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f97680cd0e0 0x7f97680b4250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:12:09.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:09.359+0000 7f976e200640 1 --2- 172.21.15.16:0/696833930 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f976813e3d0 0x7f97681f8210 secure :-1 s=READY pgs=1366 cs=0 l=1 rev1=1 crypto rx=0x7f9760007e70 tx=0x7f976000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:12:09.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:09.359+0000 7f9765600640 1 -- 172.21.15.16:0/696833930 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9760014620 con 0x7f976813e3d0 2024-03-31T15:12:09.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:09.359+0000 7f9765600640 1 -- 172.21.15.16:0/696833930 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9760019070 con 0x7f976813e3d0 2024-03-31T15:12:09.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:09.359+0000 7f9770200640 1 -- 172.21.15.16:0/696833930 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f97680b48e0 con 0x7f976813e3d0 2024-03-31T15:12:09.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:09.359+0000 7f9765600640 1 -- 172.21.15.16:0/696833930 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9760014a20 con 0x7f976813e3d0 2024-03-31T15:12:09.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:09.359+0000 7f9770200640 1 -- 172.21.15.16:0/696833930 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9768153740 con 0x7f976813e3d0 2024-03-31T15:12:09.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:09.359+0000 7f9770200640 1 -- 172.21.15.16:0/696833930 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9768069950 con 0x7f976813e3d0 2024-03-31T15:12:09.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:09.363+0000 7f9765600640 1 -- 172.21.15.16:0/696833930 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f9760014bc0 con 0x7f976813e3d0 2024-03-31T15:12:09.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:09.363+0000 7f9765600640 1 --2- 172.21.15.16:0/696833930 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f9748068380 0x7f974806a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:09.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:09.363+0000 7f9765600640 1 -- 172.21.15.16:0/696833930 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 5115+0+0 (secure 0 0 0) 0x7f9760086ed0 con 0x7f976813e3d0 2024-03-31T15:12:09.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:09.363+0000 7f976d800640 1 --2- 172.21.15.16:0/696833930 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f9748068380 0x7f974806a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:09.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:09.363+0000 7f976d800640 1 --2- 172.21.15.16:0/696833930 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f9748068380 0x7f974806a840 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f975c0ae000 tx=0x7f975c0a7000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:12:09.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:09.367+0000 7f9765600640 1 -- 172.21.15.16:0/696833930 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f9760057100 con 0x7f976813e3d0 2024-03-31T15:12:09.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:09.531+0000 7f9770200640 1 -- 172.21.15.16:0/696833930 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f976806e4e0 con 0x7f976813e3d0 2024-03-31T15:12:09.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:09.531+0000 7f9765600640 1 -- 172.21.15.16:0/696833930 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v88) ==== 93+0+10 (secure 0 0 0) 0x7f976005b400 con 0x7f976813e3d0 2024-03-31T15:12:09.523 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2024-03-31T15:12:09.525 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:09.531+0000 7f9770200640 1 -- 172.21.15.16:0/696833930 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f9748068380 msgr2=0x7f974806a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:09.525 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:09.531+0000 7f9770200640 1 --2- 172.21.15.16:0/696833930 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f9748068380 0x7f974806a840 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f975c0ae000 tx=0x7f975c0a7000 comp rx=0 tx=0).stop 2024-03-31T15:12:09.525 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:09.531+0000 7f9770200640 1 -- 172.21.15.16:0/696833930 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f976813e3d0 msgr2=0x7f97681f8210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:09.525 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:09.531+0000 7f9770200640 1 --2- 172.21.15.16:0/696833930 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f976813e3d0 0x7f97681f8210 secure :-1 s=READY pgs=1366 cs=0 l=1 rev1=1 crypto rx=0x7f9760007e70 tx=0x7f976000e4b0 comp rx=0 tx=0).stop 2024-03-31T15:12:09.525 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:09.531+0000 7f9770200640 1 -- 172.21.15.16:0/696833930 shutdown_connections 2024-03-31T15:12:09.525 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:09.531+0000 7f9770200640 1 --2- 172.21.15.16:0/696833930 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f9748068380 0x7f974806a840 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:09.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:09.531+0000 7f9770200640 1 --2- 172.21.15.16:0/696833930 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f976813e3d0 0x7f97681f8210 unknown :-1 s=CLOSED pgs=1366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:09.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:09.531+0000 7f9770200640 1 --2- 172.21.15.16:0/696833930 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f976813dff0 0x7f9768165720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:09.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:09.531+0000 7f9770200640 1 --2- 172.21.15.16:0/696833930 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f97680cd0e0 0x7f97680b4250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:09.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:09.531+0000 7f9770200640 1 -- 172.21.15.16:0/696833930 >> 172.21.15.16:0/696833930 conn(0x7f976806ee40 msgr2=0x7f9768155e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:12:09.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:09.531+0000 7f9770200640 1 -- 172.21.15.16:0/696833930 shutdown_connections 2024-03-31T15:12:09.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:09.531+0000 7f9770200640 1 -- 172.21.15.16:0/696833930 wait complete. 2024-03-31T15:12:09.545 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_inotable_sync' 2024-03-31T15:12:09.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:09.707+0000 7f41d0e00640 1 Processor -- start 2024-03-31T15:12:09.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:09.707+0000 7f41d0e00640 1 -- start start 2024-03-31T15:12:09.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:09.711+0000 7f41d0e00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f41cc0b23d0 0x7f41cc15cd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:09.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:09.711+0000 7f41d0e00640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f41cc160e70 con 0x7f41cc145d10 2024-03-31T15:12:09.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:09.711+0000 7f41d0e00640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f41cc1602f0 con 0x7f41cc0728b0 2024-03-31T15:12:09.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:09.711+0000 7f41d0e00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f41cc1605d0 con 0x7f41cc0b23d0 2024-03-31T15:12:09.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:09.711+0000 7f41c9e00640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f41cc145d10 0x7f41cc0cf790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:41270/0 (socket says 172.21.15.16:41270) 2024-03-31T15:12:09.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:09.711+0000 7f41c9e00640 1 -- 172.21.15.16:0/35128244 learned_addr learned my addr 172.21.15.16:0/35128244 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:12:09.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:09.711+0000 7f41ca800640 1 --2- 172.21.15.16:0/35128244 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f41cc0b23d0 0x7f41cc15cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:09.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:09.711+0000 7f41c9400640 1 -- 172.21.15.16:0/35128244 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3820951408 0 0) 0x7f41cc160e70 con 0x7f41cc145d10 2024-03-31T15:12:09.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:09.711+0000 7f41c9400640 1 -- 172.21.15.16:0/35128244 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f41b0003530 con 0x7f41cc145d10 2024-03-31T15:12:09.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:09.711+0000 7f41c9400640 1 -- 172.21.15.16:0/35128244 <== mon.1 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 254130823 0 0) 0x7f41cc1602f0 con 0x7f41cc0728b0 2024-03-31T15:12:09.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:09.711+0000 7f41c9400640 1 -- 172.21.15.16:0/35128244 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f41cc160e70 con 0x7f41cc0728b0 2024-03-31T15:12:09.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:09.711+0000 7f41c9400640 1 -- 172.21.15.16:0/35128244 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2838494451 0 0) 0x7f41b0003530 con 0x7f41cc145d10 2024-03-31T15:12:09.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:09.711+0000 7f41ca800640 1 -- 172.21.15.16:0/35128244 >> v1:172.21.15.89:6789/0 conn(0x7f41cc0728b0 legacy=0x7f41cc0b1cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:09.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:09.711+0000 7f41ca800640 1 -- 172.21.15.16:0/35128244 >> v1:172.21.15.16:6789/0 conn(0x7f41cc145d10 legacy=0x7f41cc0cf790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:09.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:09.711+0000 7f41ca800640 1 -- 172.21.15.16:0/35128244 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41cc1608b0 con 0x7f41cc0b23d0 2024-03-31T15:12:09.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:09.715+0000 7f41ca800640 1 --2- 172.21.15.16:0/35128244 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f41cc0b23d0 0x7f41cc15cd50 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f41c4009dc0 tx=0x7f41c401c8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3a2d3c5bf6f74591 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:12:09.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:09.715+0000 7f41c9400640 1 -- 172.21.15.16:0/35128244 <== mon.2 v2:172.21.15.89:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41bc001680 con 0x7f41cc0b23d0 2024-03-31T15:12:09.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:09.715+0000 7f41c9400640 1 -- 172.21.15.16:0/35128244 <== mon.2 v2:172.21.15.89:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f41c40a6500 con 0x7f41cc0b23d0 2024-03-31T15:12:09.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:09.715+0000 7f41c9400640 1 -- 172.21.15.16:0/35128244 <== mon.2 v2:172.21.15.89:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41c40a68d0 con 0x7f41cc0b23d0 2024-03-31T15:12:09.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:09.715+0000 7f41d0e00640 1 -- 172.21.15.16:0/35128244 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f41cc0b23d0 msgr2=0x7f41cc15cd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:09.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:09.715+0000 7f41d0e00640 1 --2- 172.21.15.16:0/35128244 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f41cc0b23d0 0x7f41cc15cd50 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f41c4009dc0 tx=0x7f41c401c8b0 comp rx=0 tx=0).stop 2024-03-31T15:12:09.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:09.715+0000 7f41d0e00640 1 -- 172.21.15.16:0/35128244 shutdown_connections 2024-03-31T15:12:09.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:09.715+0000 7f41d0e00640 1 --2- 172.21.15.16:0/35128244 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f41cc0b23d0 0x7f41cc15cd50 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:09.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:09.715+0000 7f41d0e00640 1 -- 172.21.15.16:0/35128244 >> 172.21.15.16:0/35128244 conn(0x7f41cc06b620 msgr2=0x7f41cc06ba30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:12:09.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:09.715+0000 7f41d0e00640 1 -- 172.21.15.16:0/35128244 shutdown_connections 2024-03-31T15:12:09.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:09.715+0000 7f41d0e00640 1 -- 172.21.15.16:0/35128244 wait complete. 2024-03-31T15:12:09.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:09.715+0000 7f41d0e00640 1 Processor -- start 2024-03-31T15:12:09.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:09.715+0000 7f41d0e00640 1 -- start start 2024-03-31T15:12:09.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:09.715+0000 7f41d0e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f41cc0728b0 0x7f41cc14dac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:09.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:09.719+0000 7f41d0e00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f41cc0b23d0 0x7f41cc141320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:09.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:09.719+0000 7f41d0e00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f41cc145d10 0x7f41cc167210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:09.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:09.719+0000 7f41d0e00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f41cc072e40 con 0x7f41cc0728b0 2024-03-31T15:12:09.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:09.719+0000 7f41d0e00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f41cc072cc0 con 0x7f41cc0b23d0 2024-03-31T15:12:09.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:09.719+0000 7f41d0e00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f41cc16a100 con 0x7f41cc145d10 2024-03-31T15:12:09.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:09.719+0000 7f41c9e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f41cc0728b0 0x7f41cc14dac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:09.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:09.719+0000 7f41c9e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f41cc0728b0 0x7f41cc14dac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37708/0 (socket says 172.21.15.16:37708) 2024-03-31T15:12:09.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:09.719+0000 7f41c9e00640 1 -- 172.21.15.16:0/1223448331 learned_addr learned my addr 172.21.15.16:0/1223448331 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:12:09.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:09.719+0000 7f41c1400640 1 --2- 172.21.15.16:0/1223448331 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f41cc0b23d0 0x7f41cc141320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:09.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:09.719+0000 7f41ca800640 1 --2- 172.21.15.16:0/1223448331 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f41cc145d10 0x7f41cc167210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:09.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:09.719+0000 7f41c9e00640 1 -- 172.21.15.16:0/1223448331 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f41cc145d10 msgr2=0x7f41cc167210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:09.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:09.719+0000 7f41c9e00640 1 --2- 172.21.15.16:0/1223448331 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f41cc145d10 0x7f41cc167210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:09.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:09.719+0000 7f41c9e00640 1 -- 172.21.15.16:0/1223448331 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f41cc0b23d0 msgr2=0x7f41cc141320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:09.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:09.719+0000 7f41c9e00640 1 --2- 172.21.15.16:0/1223448331 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f41cc0b23d0 0x7f41cc141320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:09.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:09.719+0000 7f41c9e00640 1 -- 172.21.15.16:0/1223448331 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41cc142c10 con 0x7f41cc0728b0 2024-03-31T15:12:09.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:09.719+0000 7f41c9e00640 1 --2- 172.21.15.16:0/1223448331 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f41cc0728b0 0x7f41cc14dac0 secure :-1 s=READY pgs=1368 cs=0 l=1 rev1=1 crypto rx=0x7f41bc00c750 tx=0x7f41bc00cc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:12:09.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:09.719+0000 7f41c2a00640 1 -- 172.21.15.16:0/1223448331 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41bc005ae0 con 0x7f41cc0728b0 2024-03-31T15:12:09.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:09.719+0000 7f41c2a00640 1 -- 172.21.15.16:0/1223448331 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f41bc005c80 con 0x7f41cc0728b0 2024-03-31T15:12:09.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:09.719+0000 7f41c2a00640 1 -- 172.21.15.16:0/1223448331 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41bc0072b0 con 0x7f41cc0728b0 2024-03-31T15:12:09.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:09.719+0000 7f41d0e00640 1 -- 172.21.15.16:0/1223448331 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f41cc141b70 con 0x7f41cc0728b0 2024-03-31T15:12:09.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:09.719+0000 7f41d0e00640 1 -- 172.21.15.16:0/1223448331 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f41cc142130 con 0x7f41cc0728b0 2024-03-31T15:12:09.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:09.719+0000 7f41d0e00640 1 -- 172.21.15.16:0/1223448331 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f41cc069950 con 0x7f41cc0728b0 2024-03-31T15:12:09.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:09.719+0000 7f41c2a00640 1 -- 172.21.15.16:0/1223448331 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f41bc007450 con 0x7f41cc0728b0 2024-03-31T15:12:09.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:09.723+0000 7f41c2a00640 1 --2- 172.21.15.16:0/1223448331 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f41a8068380 0x7f41a806a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:09.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:09.723+0000 7f41c2a00640 1 -- 172.21.15.16:0/1223448331 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 5115+0+0 (secure 0 0 0) 0x7f41bc086510 con 0x7f41cc0728b0 2024-03-31T15:12:09.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:09.723+0000 7f41c1400640 1 --2- 172.21.15.16:0/1223448331 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f41a8068380 0x7f41a806a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:09.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:09.723+0000 7f41c1400640 1 --2- 172.21.15.16:0/1223448331 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f41a8068380 0x7f41a806a840 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f41cc167f40 tx=0x7f41b4009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:12:09.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:09.727+0000 7f41c2a00640 1 -- 172.21.15.16:0/1223448331 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f41bc056d40 con 0x7f41cc0728b0 2024-03-31T15:12:09.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:09.947+0000 7f41d0e00640 1 -- 172.21.15.16:0/1223448331 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_inotable_sync"]} v 0) -- 0x7f41cc0d0cd0 con 0x7f41cc0728b0 2024-03-31T15:12:10.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:10.843+0000 7f41c2a00640 1 -- 172.21.15.16:0/1223448331 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_inotable_sync"]}]=0 v404) ==== 149+0+0 (secure 0 0 0) 0x7f41bc056640 con 0x7f41cc0728b0 2024-03-31T15:12:10.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:10.843+0000 7f41d0e00640 1 -- 172.21.15.16:0/1223448331 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f41a8068380 msgr2=0x7f41a806a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:10.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:10.843+0000 7f41d0e00640 1 --2- 172.21.15.16:0/1223448331 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f41a8068380 0x7f41a806a840 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f41cc167f40 tx=0x7f41b4009040 comp rx=0 tx=0).stop 2024-03-31T15:12:10.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:10.843+0000 7f41d0e00640 1 -- 172.21.15.16:0/1223448331 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f41cc0728b0 msgr2=0x7f41cc14dac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:10.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:10.843+0000 7f41d0e00640 1 --2- 172.21.15.16:0/1223448331 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f41cc0728b0 0x7f41cc14dac0 secure :-1 s=READY pgs=1368 cs=0 l=1 rev1=1 crypto rx=0x7f41bc00c750 tx=0x7f41bc00cc10 comp rx=0 tx=0).stop 2024-03-31T15:12:10.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:10.843+0000 7f41d0e00640 1 -- 172.21.15.16:0/1223448331 shutdown_connections 2024-03-31T15:12:10.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:10.843+0000 7f41d0e00640 1 --2- 172.21.15.16:0/1223448331 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f41a8068380 0x7f41a806a840 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:10.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:10.843+0000 7f41d0e00640 1 --2- 172.21.15.16:0/1223448331 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f41cc145d10 0x7f41cc167210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:10.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:10.843+0000 7f41d0e00640 1 --2- 172.21.15.16:0/1223448331 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f41cc0b23d0 0x7f41cc141320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:10.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:10.843+0000 7f41d0e00640 1 --2- 172.21.15.16:0/1223448331 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f41cc0728b0 0x7f41cc14dac0 unknown :-1 s=CLOSED pgs=1368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:10.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:10.843+0000 7f41d0e00640 1 -- 172.21.15.16:0/1223448331 >> 172.21.15.16:0/1223448331 conn(0x7f41cc06b620 msgr2=0x7f41cc0cd470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:12:10.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:10.843+0000 7f41d0e00640 1 -- 172.21.15.16:0/1223448331 shutdown_connections 2024-03-31T15:12:10.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:10.843+0000 7f41d0e00640 1 -- 172.21.15.16:0/1223448331 wait complete. 2024-03-31T15:12:10.860 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-03-31T15:12:11.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:11.015+0000 7f132ea00640 1 Processor -- start 2024-03-31T15:12:11.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:11.015+0000 7f132ea00640 1 -- start start 2024-03-31T15:12:11.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:11.015+0000 7f132ea00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f1328070030 0x7f1328143c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:11.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:11.015+0000 7f132ea00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f1328141590 0x7f1328144170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:11.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:11.015+0000 7f132ea00640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f13281446b0 con 0x7f13281411b0 2024-03-31T15:12:11.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:11.015+0000 7f132ea00640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f132815f910 con 0x7f1328141590 2024-03-31T15:12:11.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:11.015+0000 7f132ea00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f1328162910 con 0x7f1328070030 2024-03-31T15:12:11.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:11.015+0000 7f1327400640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f1328070030 0x7f1328143c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:11.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:11.015+0000 7f132ca00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f1328141590 0x7f1328144170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:11.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:11.015+0000 7f1327400640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f1328070030 0x7f1328143c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3301/0 says I am v2:172.21.15.16:47134/0 (socket says 172.21.15.16:47134) 2024-03-31T15:12:11.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:11.015+0000 7f132ca00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f1328141590 0x7f1328144170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.16:55178/0 (socket says 172.21.15.16:55178) 2024-03-31T15:12:11.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:11.015+0000 7f1327400640 1 -- 172.21.15.16:0/1355628888 learned_addr learned my addr 172.21.15.16:0/1355628888 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:12:11.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:11.019+0000 7f1327400640 1 -- 172.21.15.16:0/1355628888 >> v2:172.21.15.89:3300/0 conn(0x7f1328141590 msgr2=0x7f1328144170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:12:11.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:11.019+0000 7f1327400640 1 --2- 172.21.15.16:0/1355628888 >> v2:172.21.15.89:3300/0 conn(0x7f1328141590 0x7f1328144170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:11.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:11.019+0000 7f1327400640 1 -- 172.21.15.16:0/1355628888 >> v1:172.21.15.89:6789/0 conn(0x7f13281411b0 legacy=0x7f132806f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:11.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:11.019+0000 7f1327400640 1 -- 172.21.15.16:0/1355628888 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1328144980 con 0x7f1328070030 2024-03-31T15:12:11.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:11.019+0000 7f1327400640 1 --2- 172.21.15.16:0/1355628888 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f1328070030 0x7f1328143c30 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f1314009870 tx=0x7f131401c530 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e0d81ff916d961bf server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:12:11.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:11.019+0000 7f1326a00640 1 -- 172.21.15.16:0/1355628888 <== mon.2 v2:172.21.15.89:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13140a95f0 con 0x7f1328070030 2024-03-31T15:12:11.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:11.019+0000 7f1326a00640 1 -- 172.21.15.16:0/1355628888 <== mon.2 v2:172.21.15.89:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f13140a9790 con 0x7f1328070030 2024-03-31T15:12:11.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:11.019+0000 7f1326a00640 1 -- 172.21.15.16:0/1355628888 <== mon.2 v2:172.21.15.89:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13140a9b60 con 0x7f1328070030 2024-03-31T15:12:11.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:11.019+0000 7f132ea00640 1 -- 172.21.15.16:0/1355628888 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f1328070030 msgr2=0x7f1328143c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:11.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:11.019+0000 7f132ea00640 1 --2- 172.21.15.16:0/1355628888 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f1328070030 0x7f1328143c30 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f1314009870 tx=0x7f131401c530 comp rx=0 tx=0).stop 2024-03-31T15:12:11.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:11.023+0000 7f132ea00640 1 -- 172.21.15.16:0/1355628888 shutdown_connections 2024-03-31T15:12:11.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:11.023+0000 7f132ea00640 1 --2- 172.21.15.16:0/1355628888 >> v2:172.21.15.89:3300/0 conn(0x7f1328141590 0x7f1328144170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:11.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:11.023+0000 7f132ea00640 1 --2- 172.21.15.16:0/1355628888 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f1328070030 0x7f1328143c30 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:11.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:11.023+0000 7f132ea00640 1 -- 172.21.15.16:0/1355628888 >> 172.21.15.16:0/1355628888 conn(0x7f1328067930 msgr2=0x7f13280b3ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:12:11.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:11.023+0000 7f132ea00640 1 -- 172.21.15.16:0/1355628888 shutdown_connections 2024-03-31T15:12:11.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:11.023+0000 7f132ea00640 1 -- 172.21.15.16:0/1355628888 wait complete. 2024-03-31T15:12:11.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:11.023+0000 7f132ea00640 1 Processor -- start 2024-03-31T15:12:11.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:11.023+0000 7f132ea00640 1 -- start start 2024-03-31T15:12:11.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:11.023+0000 7f132ea00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1328070030 0x7f1328162f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:11.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:11.023+0000 7f132ea00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f13281411b0 0x7f132815ac90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:11.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:11.023+0000 7f132ea00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f1328141590 0x7f132813e1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:11.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:11.023+0000 7f132ea00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f132815f110 con 0x7f13281411b0 2024-03-31T15:12:11.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:11.023+0000 7f132ea00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f1328162d80 con 0x7f1328070030 2024-03-31T15:12:11.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:11.023+0000 7f132ea00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f132815f290 con 0x7f1328141590 2024-03-31T15:12:11.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:11.023+0000 7f1327400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f13281411b0 0x7f132815ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:11.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:11.023+0000 7f1327400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f13281411b0 0x7f132815ac90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37718/0 (socket says 172.21.15.16:37718) 2024-03-31T15:12:11.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:11.023+0000 7f1327400640 1 -- 172.21.15.16:0/1967326959 learned_addr learned my addr 172.21.15.16:0/1967326959 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:12:11.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:11.023+0000 7f132ca00640 1 --2- 172.21.15.16:0/1967326959 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f1328141590 0x7f132813e1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:11.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:11.023+0000 7f1327e00640 1 --2- 172.21.15.16:0/1967326959 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1328070030 0x7f1328162f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:11.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:11.023+0000 7f1327400640 1 -- 172.21.15.16:0/1967326959 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f1328141590 msgr2=0x7f132813e1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:11.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:11.023+0000 7f1327400640 1 --2- 172.21.15.16:0/1967326959 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f1328141590 0x7f132813e1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:11.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:11.023+0000 7f1327400640 1 -- 172.21.15.16:0/1967326959 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1328070030 msgr2=0x7f1328162f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:11.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:11.023+0000 7f1327400640 1 --2- 172.21.15.16:0/1967326959 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1328070030 0x7f1328162f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:11.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:11.023+0000 7f1327400640 1 -- 172.21.15.16:0/1967326959 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1328157d50 con 0x7f13281411b0 2024-03-31T15:12:11.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:11.023+0000 7f1327400640 1 --2- 172.21.15.16:0/1967326959 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f13281411b0 0x7f132815ac90 secure :-1 s=READY pgs=1369 cs=0 l=1 rev1=1 crypto rx=0x7f13140ab040 tx=0x7f131401ad70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:12:11.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:11.027+0000 7f131fe00640 1 -- 172.21.15.16:0/1967326959 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13140b1600 con 0x7f13281411b0 2024-03-31T15:12:11.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:11.027+0000 7f131fe00640 1 -- 172.21.15.16:0/1967326959 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f13140b17a0 con 0x7f13281411b0 2024-03-31T15:12:11.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:11.027+0000 7f131fe00640 1 -- 172.21.15.16:0/1967326959 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13140b1b90 con 0x7f13281411b0 2024-03-31T15:12:11.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:11.027+0000 7f132ea00640 1 -- 172.21.15.16:0/1967326959 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1328158030 con 0x7f13281411b0 2024-03-31T15:12:11.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:11.027+0000 7f132ea00640 1 -- 172.21.15.16:0/1967326959 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f13281585f0 con 0x7f13281411b0 2024-03-31T15:12:11.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:11.027+0000 7f132ea00640 1 -- 172.21.15.16:0/1967326959 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1328069950 con 0x7f13281411b0 2024-03-31T15:12:11.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:11.027+0000 7f131fe00640 1 -- 172.21.15.16:0/1967326959 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f13140bc450 con 0x7f13281411b0 2024-03-31T15:12:11.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:11.027+0000 7f131fe00640 1 --2- 172.21.15.16:0/1967326959 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f1308068380 0x7f130806a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:11.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:11.027+0000 7f131fe00640 1 -- 172.21.15.16:0/1967326959 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 5115+0+0 (secure 0 0 0) 0x7f1314120ef0 con 0x7f13281411b0 2024-03-31T15:12:11.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:11.027+0000 7f1327e00640 1 --2- 172.21.15.16:0/1967326959 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f1308068380 0x7f130806a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:11.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:11.031+0000 7f1327e00640 1 --2- 172.21.15.16:0/1967326959 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f1308068380 0x7f130806a840 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f1318001810 tx=0x7f1318009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:12:11.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:11.031+0000 7f131fe00640 1 -- 172.21.15.16:0/1967326959 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f13140ad020 con 0x7f13281411b0 2024-03-31T15:12:11.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:11.195+0000 7f132ea00640 1 -- 172.21.15.16:0/1967326959 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7f13280b4a20 con 0x7f13281411b0 2024-03-31T15:12:11.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:11.199+0000 7f131fe00640 1 -- 172.21.15.16:0/1967326959 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v20)=0 v20) ==== 122+0+0 (secure 0 0 0) 0x7f1314016150 con 0x7f13281411b0 2024-03-31T15:12:11.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:11.203+0000 7f132ea00640 1 -- 172.21.15.16:0/1967326959 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f1308068380 msgr2=0x7f130806a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:11.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:11.203+0000 7f132ea00640 1 --2- 172.21.15.16:0/1967326959 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f1308068380 0x7f130806a840 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f1318001810 tx=0x7f1318009040 comp rx=0 tx=0).stop 2024-03-31T15:12:11.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:11.203+0000 7f132ea00640 1 -- 172.21.15.16:0/1967326959 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f13281411b0 msgr2=0x7f132815ac90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:11.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:11.203+0000 7f132ea00640 1 --2- 172.21.15.16:0/1967326959 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f13281411b0 0x7f132815ac90 secure :-1 s=READY pgs=1369 cs=0 l=1 rev1=1 crypto rx=0x7f13140ab040 tx=0x7f131401ad70 comp rx=0 tx=0).stop 2024-03-31T15:12:11.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:11.203+0000 7f132ea00640 1 -- 172.21.15.16:0/1967326959 shutdown_connections 2024-03-31T15:12:11.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:11.203+0000 7f132ea00640 1 --2- 172.21.15.16:0/1967326959 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f1308068380 0x7f130806a840 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:11.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:11.203+0000 7f132ea00640 1 --2- 172.21.15.16:0/1967326959 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f1328141590 0x7f132813e1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:11.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:11.203+0000 7f132ea00640 1 --2- 172.21.15.16:0/1967326959 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f13281411b0 0x7f132815ac90 unknown :-1 s=CLOSED pgs=1369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:11.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:11.203+0000 7f132ea00640 1 --2- 172.21.15.16:0/1967326959 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1328070030 0x7f1328162f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:11.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:11.203+0000 7f132ea00640 1 -- 172.21.15.16:0/1967326959 >> 172.21.15.16:0/1967326959 conn(0x7f1328067930 msgr2=0x7f13280b22f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:12:11.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:11.203+0000 7f132ea00640 1 -- 172.21.15.16:0/1967326959 shutdown_connections 2024-03-31T15:12:11.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:11.203+0000 7f132ea00640 1 -- 172.21.15.16:0/1967326959 wait complete. 2024-03-31T15:12:11.214 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-31T15:12:11.215 DEBUG:teuthology.orchestra.run.smithi016:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-31T15:12:11.261 DEBUG:teuthology.orchestra.run.smithi089:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-31T15:12:11.337 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-31T15:12:11.338 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-03-31T15:12:11.338 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-31T15:12:11.347 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16357708k,nr_inodes=4089427,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275996k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=6539 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275992k,nr_inodes=818998,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275996k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-31T15:12:11.347 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-03-31T15:12:11.347 DEBUG:teuthology.orchestra.run.smithi089:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-31T15:12:11.356 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16354452k,nr_inodes=4088613,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275344k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=901 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275340k,nr_inodes=818835,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275344k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-31T15:12:11.356 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-03-31T15:12:11.356 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-31T15:12:11.397 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16357708k,nr_inodes=4089427,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275996k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=6539 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275992k,nr_inodes=818998,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275996k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-31T15:12:11.397 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-03-31T15:12:11.397 DEBUG:teuthology.orchestra.run.smithi089:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-31T15:12:11.406 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16354452k,nr_inodes=4088613,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275344k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=901 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275340k,nr_inodes=818835,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275344k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-31T15:12:11.406 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-03-31T15:12:11.407 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-31T15:12:11.577 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:11.583+0000 7fd69ea00640 1 Processor -- start 2024-03-31T15:12:11.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:11.583+0000 7fd69ea00640 1 -- start start 2024-03-31T15:12:11.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:11.587+0000 7fd69ea00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fd69815eee0 0x7fd698072420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:11.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:11.587+0000 7fd69ea00640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd69815f3c0 con 0x7fd6980b23d0 2024-03-31T15:12:11.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:11.587+0000 7fd69ea00640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd69815f6a0 con 0x7fd69815b3d0 2024-03-31T15:12:11.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:11.587+0000 7fd69ea00640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7fd69815b7b0 con 0x7fd69815eee0 2024-03-31T15:12:11.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:11.587+0000 7fd68f400640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fd6980b23d0 0x7fd698071d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:41280/0 (socket says 172.21.15.16:41280) 2024-03-31T15:12:11.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:11.587+0000 7fd68f400640 1 -- 172.21.15.16:0/1087199551 learned_addr learned my addr 172.21.15.16:0/1087199551 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:12:11.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:11.587+0000 7fd69ca00640 1 --2- 172.21.15.16:0/1087199551 >> v2:172.21.15.89:3300/0 conn(0x7fd69815eee0 0x7fd698072420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:11.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:11.587+0000 7fd697400640 1 -- 172.21.15.16:0/1087199551 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 749723934 0 0) 0x7fd69815f3c0 con 0x7fd6980b23d0 2024-03-31T15:12:11.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:11.587+0000 7fd697400640 1 -- 172.21.15.16:0/1087199551 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd67c003530 con 0x7fd6980b23d0 2024-03-31T15:12:11.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:11.587+0000 7fd697400640 1 -- 172.21.15.16:0/1087199551 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2437334328 0 0) 0x7fd67c003530 con 0x7fd6980b23d0 2024-03-31T15:12:11.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:11.587+0000 7fd697400640 1 -- 172.21.15.16:0/1087199551 >> v2:172.21.15.89:3300/0 conn(0x7fd69815eee0 msgr2=0x7fd698072420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:12:11.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:11.587+0000 7fd697400640 1 --2- 172.21.15.16:0/1087199551 >> v2:172.21.15.89:3300/0 conn(0x7fd69815eee0 0x7fd698072420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:11.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:11.587+0000 7fd697400640 1 -- 172.21.15.16:0/1087199551 >> v1:172.21.15.89:6789/0 conn(0x7fd69815b3d0 legacy=0x7fd6980b1cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:11.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:11.587+0000 7fd697400640 1 -- 172.21.15.16:0/1087199551 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd69815f980 con 0x7fd6980b23d0 2024-03-31T15:12:11.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:11.587+0000 7fd697400640 1 -- 172.21.15.16:0/1087199551 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 214807623 0 0) 0x7fd680001680 con 0x7fd6980b23d0 2024-03-31T15:12:11.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:11.587+0000 7fd697400640 1 -- 172.21.15.16:0/1087199551 >> v1:172.21.15.16:6789/0 conn(0x7fd6980b23d0 legacy=0x7fd698071d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:11.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:11.587+0000 7fd697400640 1 --2- 172.21.15.16:0/1087199551 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd67c003a10 0x7fd67c003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:11.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:11.587+0000 7fd697400640 1 --2- 172.21.15.16:0/1087199551 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fd67c004810 0x7fd67c004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:11.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:11.587+0000 7fd697400640 1 --2- 172.21.15.16:0/1087199551 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd67c0051d0 0x7fd67c00b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:11.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:11.587+0000 7fd697400640 1 -- 172.21.15.16:0/1087199551 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fd69815b7b0 con 0x7fd67c0051d0 2024-03-31T15:12:11.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:11.587+0000 7fd697400640 1 -- 172.21.15.16:0/1087199551 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fd67c00c0d0 con 0x7fd67c003a10 2024-03-31T15:12:11.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:11.587+0000 7fd697400640 1 -- 172.21.15.16:0/1087199551 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fd67c00c2d0 con 0x7fd67c004810 2024-03-31T15:12:11.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:11.587+0000 7fd69ca00640 1 --2- 172.21.15.16:0/1087199551 >> v2:172.21.15.89:3300/0 conn(0x7fd69815eee0 0x7fd698072420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:12:11.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:11.587+0000 7fd69ca00640 1 --2- 172.21.15.16:0/1087199551 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd67c0051d0 0x7fd67c00b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:11.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:11.587+0000 7fd697e00640 1 --2- 172.21.15.16:0/1087199551 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd67c003a10 0x7fd67c003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:11.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:11.587+0000 7fd68f400640 1 --2- 172.21.15.16:0/1087199551 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fd67c004810 0x7fd67c004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:11.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:11.587+0000 7fd69ca00640 1 -- 172.21.15.16:0/1087199551 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fd67c004810 msgr2=0x7fd67c004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:12:11.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:11.587+0000 7fd69ca00640 1 --2- 172.21.15.16:0/1087199551 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fd67c004810 0x7fd67c004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:11.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:11.587+0000 7fd69ca00640 1 -- 172.21.15.16:0/1087199551 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd67c003a10 msgr2=0x7fd67c003e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:12:11.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:11.587+0000 7fd69ca00640 1 --2- 172.21.15.16:0/1087199551 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd67c003a10 0x7fd67c003e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:11.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:11.587+0000 7fd69ca00640 1 -- 172.21.15.16:0/1087199551 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd67c003530 con 0x7fd67c0051d0 2024-03-31T15:12:11.585 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:11.591+0000 7fd69ca00640 1 -- 172.21.15.16:0/1087199551 reap_dead start 2024-03-31T15:12:11.585 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:11.591+0000 7fd69ca00640 1 --2- 172.21.15.16:0/1087199551 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd67c0051d0 0x7fd67c00b9d0 secure :-1 s=READY pgs=1371 cs=0 l=1 rev1=1 crypto rx=0x7fd69000e320 tx=0x7fd69001ce20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=33b4dd0f7f678d51 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:12:11.585 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:11.591+0000 7fd697400640 1 -- 172.21.15.16:0/1087199551 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd690009dc0 con 0x7fd67c0051d0 2024-03-31T15:12:11.585 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:11.591+0000 7fd697400640 1 -- 172.21.15.16:0/1087199551 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd6900a5e30 con 0x7fd67c0051d0 2024-03-31T15:12:11.585 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:11.591+0000 7fd697400640 1 -- 172.21.15.16:0/1087199551 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd690003ba0 con 0x7fd67c0051d0 2024-03-31T15:12:11.585 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:11.591+0000 7fd69ea00640 1 -- 172.21.15.16:0/1087199551 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd67c0051d0 msgr2=0x7fd67c00b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:11.585 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:11.591+0000 7fd69ea00640 1 --2- 172.21.15.16:0/1087199551 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd67c0051d0 0x7fd67c00b9d0 secure :-1 s=READY pgs=1371 cs=0 l=1 rev1=1 crypto rx=0x7fd69000e320 tx=0x7fd69001ce20 comp rx=0 tx=0).stop 2024-03-31T15:12:11.585 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:11.591+0000 7fd69ea00640 1 -- 172.21.15.16:0/1087199551 shutdown_connections 2024-03-31T15:12:11.585 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:11.591+0000 7fd69ea00640 1 --2- 172.21.15.16:0/1087199551 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd67c0051d0 0x7fd67c00b9d0 unknown :-1 s=CLOSED pgs=1371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:11.586 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:11.591+0000 7fd69ea00640 1 -- 172.21.15.16:0/1087199551 >> 172.21.15.16:0/1087199551 conn(0x7fd69806b620 msgr2=0x7fd69806ba30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:12:11.586 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:11.591+0000 7fd69ea00640 1 -- 172.21.15.16:0/1087199551 shutdown_connections 2024-03-31T15:12:11.586 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:11.591+0000 7fd69ea00640 1 -- 172.21.15.16:0/1087199551 wait complete. 2024-03-31T15:12:11.586 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:11.595+0000 7fd69ea00640 1 Processor -- start 2024-03-31T15:12:11.586 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:11.595+0000 7fd69ea00640 1 -- start start 2024-03-31T15:12:11.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:11.595+0000 7fd69ea00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fd67c0051d0 0x7fd69814b310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:11.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:11.595+0000 7fd69ea00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd6981f3b00 0x7fd69815c4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:11.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:11.595+0000 7fd69ea00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd698154cb0 0x7fd698155130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:11.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:11.595+0000 7fd69ea00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fd6980ce2f0 con 0x7fd698154cb0 2024-03-31T15:12:11.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:11.595+0000 7fd69ea00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fd6980717c0 con 0x7fd6981f3b00 2024-03-31T15:12:11.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:11.595+0000 7fd69ea00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fd6980ce470 con 0x7fd67c0051d0 2024-03-31T15:12:11.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:11.595+0000 7fd69ca00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd698154cb0 0x7fd698155130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:11.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:11.595+0000 7fd69ca00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd698154cb0 0x7fd698155130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37730/0 (socket says 172.21.15.16:37730) 2024-03-31T15:12:11.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:11.595+0000 7fd69ca00640 1 -- 172.21.15.16:0/2723106197 learned_addr learned my addr 172.21.15.16:0/2723106197 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:12:11.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:11.595+0000 7fd68f400640 1 --2- 172.21.15.16:0/2723106197 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd6981f3b00 0x7fd69815c4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:11.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:11.595+0000 7fd697e00640 1 --2- 172.21.15.16:0/2723106197 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fd67c0051d0 0x7fd69814b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:11.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:11.595+0000 7fd69ca00640 1 -- 172.21.15.16:0/2723106197 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fd67c0051d0 msgr2=0x7fd69814b310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:11.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:11.595+0000 7fd69ca00640 1 --2- 172.21.15.16:0/2723106197 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fd67c0051d0 0x7fd69814b310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:11.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:11.595+0000 7fd69ca00640 1 -- 172.21.15.16:0/2723106197 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd6981f3b00 msgr2=0x7fd69815c4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:11.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:11.595+0000 7fd69ca00640 1 --2- 172.21.15.16:0/2723106197 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd6981f3b00 0x7fd69815c4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:11.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:11.595+0000 7fd69ca00640 1 -- 172.21.15.16:0/2723106197 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd69814b920 con 0x7fd698154cb0 2024-03-31T15:12:11.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:11.595+0000 7fd697e00640 1 --2- 172.21.15.16:0/2723106197 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fd67c0051d0 0x7fd69814b310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:12:11.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:11.595+0000 7fd69ca00640 1 --2- 172.21.15.16:0/2723106197 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd698154cb0 0x7fd698155130 secure :-1 s=READY pgs=1372 cs=0 l=1 rev1=1 crypto rx=0x7fd69001d6c0 tx=0x7fd690009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:12:11.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:11.595+0000 7fd694c00640 1 -- 172.21.15.16:0/2723106197 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd690009dc0 con 0x7fd698154cb0 2024-03-31T15:12:11.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:11.595+0000 7fd694c00640 1 -- 172.21.15.16:0/2723106197 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd6900a5c30 con 0x7fd698154cb0 2024-03-31T15:12:11.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:11.595+0000 7fd69ea00640 1 -- 172.21.15.16:0/2723106197 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd69814bc00 con 0x7fd698154cb0 2024-03-31T15:12:11.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:11.595+0000 7fd694c00640 1 -- 172.21.15.16:0/2723106197 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6900b6630 con 0x7fd698154cb0 2024-03-31T15:12:11.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:11.595+0000 7fd69ea00640 1 -- 172.21.15.16:0/2723106197 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd69814c1c0 con 0x7fd698154cb0 2024-03-31T15:12:11.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:11.595+0000 7fd69ea00640 1 -- 172.21.15.16:0/2723106197 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd698069950 con 0x7fd698154cb0 2024-03-31T15:12:11.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:11.599+0000 7fd694c00640 1 -- 172.21.15.16:0/2723106197 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fd6900bf020 con 0x7fd698154cb0 2024-03-31T15:12:11.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:11.599+0000 7fd694c00640 1 --2- 172.21.15.16:0/2723106197 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fd6740683b0 0x7fd67406a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:11.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:11.599+0000 7fd694c00640 1 -- 172.21.15.16:0/2723106197 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 5115+0+0 (secure 0 0 0) 0x7fd690121470 con 0x7fd698154cb0 2024-03-31T15:12:11.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:11.599+0000 7fd697e00640 1 --2- 172.21.15.16:0/2723106197 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fd6740683b0 0x7fd67406a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:11.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:11.599+0000 7fd697e00640 1 --2- 172.21.15.16:0/2723106197 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fd6740683b0 0x7fd67406a870 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7fd6980b1e00 tx=0x7fd688009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:12:11.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:11.603+0000 7fd694c00640 1 -- 172.21.15.16:0/2723106197 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fd690016b30 con 0x7fd698154cb0 2024-03-31T15:12:11.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:11.815+0000 7fd69ea00640 1 -- 172.21.15.16:0/2723106197 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd6980cdea0 con 0x7fd698154cb0 2024-03-31T15:12:11.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:11.815+0000 7fd694c00640 1 -- 172.21.15.16:0/2723106197 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 76 v76) ==== 94+0+3869 (secure 0 0 0) 0x7fd6900f1cf0 con 0x7fd698154cb0 2024-03-31T15:12:11.807 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:12:11.807 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":76,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6494,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/2445558494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6835","nonce":2445558494},{"type":"v1","addr":"172.21.15.89:6837","nonce":2445558494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":56},{"gid":7270,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/3185913858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6833","nonce":3185913858},{"type":"v1","addr":"172.21.15.16:6835","nonce":3185913858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":56},{"gid":7279,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6836/1599468493","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":1599468493},{"type":"v1","addr":"172.21.15.89:6836","nonce":1599468493}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":56},{"gid":8400,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6834/2863294463","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":2863294463},{"type":"v1","addr":"172.21.15.16:6834","nonce":2863294463}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":76}],"filesystems":[]} 2024-03-31T15:12:11.807 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 76 2024-03-31T15:12:11.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:11.815+0000 7fd69ea00640 1 -- 172.21.15.16:0/2723106197 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fd6740683b0 msgr2=0x7fd67406a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:11.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:11.815+0000 7fd69ea00640 1 --2- 172.21.15.16:0/2723106197 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fd6740683b0 0x7fd67406a870 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7fd6980b1e00 tx=0x7fd688009040 comp rx=0 tx=0).stop 2024-03-31T15:12:11.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:11.815+0000 7fd69ea00640 1 -- 172.21.15.16:0/2723106197 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd698154cb0 msgr2=0x7fd698155130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:11.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:11.815+0000 7fd69ea00640 1 --2- 172.21.15.16:0/2723106197 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd698154cb0 0x7fd698155130 secure :-1 s=READY pgs=1372 cs=0 l=1 rev1=1 crypto rx=0x7fd69001d6c0 tx=0x7fd690009750 comp rx=0 tx=0).stop 2024-03-31T15:12:11.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:11.815+0000 7fd69ea00640 1 -- 172.21.15.16:0/2723106197 shutdown_connections 2024-03-31T15:12:11.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:11.815+0000 7fd69ea00640 1 --2- 172.21.15.16:0/2723106197 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fd6740683b0 0x7fd67406a870 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:11.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:11.815+0000 7fd69ea00640 1 --2- 172.21.15.16:0/2723106197 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd698154cb0 0x7fd698155130 unknown :-1 s=CLOSED pgs=1372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:11.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:11.815+0000 7fd69ea00640 1 --2- 172.21.15.16:0/2723106197 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd6981f3b00 0x7fd69815c4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:11.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:11.815+0000 7fd69ea00640 1 --2- 172.21.15.16:0/2723106197 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fd67c0051d0 0x7fd69814b310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:11.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:11.815+0000 7fd69ea00640 1 -- 172.21.15.16:0/2723106197 >> 172.21.15.16:0/2723106197 conn(0x7fd69806b620 msgr2=0x7fd698071c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:12:11.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:11.815+0000 7fd69ea00640 1 -- 172.21.15.16:0/2723106197 shutdown_connections 2024-03-31T15:12:11.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:11.815+0000 7fd69ea00640 1 -- 172.21.15.16:0/2723106197 wait complete. 2024-03-31T15:12:11.828 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-31T15:12:11.829 INFO:tasks.ceph.mds.a:Stopping old one... 2024-03-31T15:12:11.829 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-31T15:12:11.829 INFO:teuthology.orchestra.run:waiting for 300 2024-03-31T15:12:11.829 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-31T15:12:11.829 INFO:tasks.ceph.mds.c:Stopping old one... 2024-03-31T15:12:11.829 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-31T15:12:11.830 INFO:teuthology.orchestra.run:waiting for 300 2024-03-31T15:12:11.830 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-31T15:12:11.830 INFO:tasks.ceph.mds.b:Stopping old one... 2024-03-31T15:12:11.830 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-31T15:12:11.830 INFO:teuthology.orchestra.run:waiting for 300 2024-03-31T15:12:11.830 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-31T15:12:11.830 INFO:tasks.ceph.mds.d:Stopping old one... 2024-03-31T15:12:11.831 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-31T15:12:11.831 INFO:teuthology.orchestra.run:waiting for 300 2024-03-31T15:12:11.844 INFO:tasks.ceph.mds.a:Stopped 2024-03-31T15:12:11.844 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-31T15:12:11.845 INFO:tasks.ceph.mds.c:Stopped 2024-03-31T15:12:11.845 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-31T15:12:11.846 INFO:tasks.ceph.mds.b:Stopped 2024-03-31T15:12:11.846 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-31T15:12:11.847 INFO:tasks.ceph.mds.d:Stopped 2024-03-31T15:12:11.847 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-31T15:12:11.889 INFO:tasks.ceph.mds.a:Started 2024-03-31T15:12:11.889 DEBUG:teuthology.parallel:result is None 2024-03-31T15:12:11.890 INFO:tasks.ceph.mds.c:Started 2024-03-31T15:12:11.890 DEBUG:teuthology.parallel:result is None 2024-03-31T15:12:11.893 INFO:tasks.ceph.mds.b:Started 2024-03-31T15:12:11.893 DEBUG:teuthology.parallel:result is None 2024-03-31T15:12:11.895 INFO:tasks.ceph.mds.d:Started 2024-03-31T15:12:11.895 DEBUG:teuthology.parallel:result is None 2024-03-31T15:12:11.896 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-03-31T15:12:11.976 INFO:tasks.ceph.mds.c.smithi016.stdout:starting mds.c at 2024-03-31T15:12:11.976 INFO:tasks.ceph.mds.a.smithi016.stdout:starting mds.a at 2024-03-31T15:12:11.982 INFO:tasks.ceph.mds.b.smithi089.stdout:starting mds.b at 2024-03-31T15:12:11.983 INFO:tasks.ceph.mds.d.smithi089.stdout:starting mds.d at 2024-03-31T15:12:12.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:12.051+0000 7f6aca000640 1 Processor -- start 2024-03-31T15:12:12.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:12.051+0000 7f6aca000640 1 -- start start 2024-03-31T15:12:12.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:12.051+0000 7f6aca000640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f6ac406f920 0x7f6ac406fd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:12.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:12.051+0000 7f6aca000640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f6ac40d2010 0x7f6ac40d24a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:12.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:12.051+0000 7f6aca000640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6ac40d0920 con 0x7f6ac4145e40 2024-03-31T15:12:12.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:12.051+0000 7f6aca000640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f6ac40702c0 con 0x7f6ac406f920 2024-03-31T15:12:12.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:12.051+0000 7f6aca000640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f6ac40d06d0 con 0x7f6ac40d2010 2024-03-31T15:12:12.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:12.051+0000 7f6ac3400640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f6ac4145e40 0x7f6ac40cc740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:41296/0 (socket says 172.21.15.16:41296) 2024-03-31T15:12:12.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:12.051+0000 7f6abbe00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f6ac406f920 0x7f6ac406fd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:12.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:12.051+0000 7f6ac3400640 1 -- 172.21.15.16:0/2394105748 learned_addr learned my addr 172.21.15.16:0/2394105748 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:12:12.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:12.051+0000 7f6ac3e00640 1 --2- 172.21.15.16:0/2394105748 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f6ac40d2010 0x7f6ac40d24a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:12.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:12.051+0000 7f6ac2a00640 1 -- 172.21.15.16:0/2394105748 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3768288878 0 0) 0x7f6ac40d0920 con 0x7f6ac4145e40 2024-03-31T15:12:12.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:12.051+0000 7f6ac2a00640 1 -- 172.21.15.16:0/2394105748 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6aa8003530 con 0x7f6ac4145e40 2024-03-31T15:12:12.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:12.051+0000 7f6abbe00640 1 -- 172.21.15.16:0/2394105748 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f6ac40d2010 msgr2=0x7f6ac40d24a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:12:12.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:12.051+0000 7f6abbe00640 1 --2- 172.21.15.16:0/2394105748 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f6ac40d2010 0x7f6ac40d24a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:12.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:12.051+0000 7f6abbe00640 1 -- 172.21.15.16:0/2394105748 >> v1:172.21.15.16:6789/0 conn(0x7f6ac4145e40 legacy=0x7f6ac40cc740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:12.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:12.051+0000 7f6abbe00640 1 -- 172.21.15.16:0/2394105748 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ac40d0c00 con 0x7f6ac406f920 2024-03-31T15:12:12.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:12.055+0000 7f6abbe00640 1 --2- 172.21.15.16:0/2394105748 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6ac406f920 0x7f6ac406fd80 secure :-1 s=READY pgs=1379 cs=0 l=1 rev1=1 crypto rx=0x7f6aac0097c0 tx=0x7f6aac01c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd2fed07d628ff46 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:12:12.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:12.055+0000 7f6ac2a00640 1 -- 172.21.15.16:0/2394105748 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6aac0b2070 con 0x7f6ac406f920 2024-03-31T15:12:12.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:12.055+0000 7f6ac2a00640 1 -- 172.21.15.16:0/2394105748 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6aac0ad640 con 0x7f6ac406f920 2024-03-31T15:12:12.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:12.055+0000 7f6ac2a00640 1 -- 172.21.15.16:0/2394105748 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6aac0ad9f0 con 0x7f6ac406f920 2024-03-31T15:12:12.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:12.055+0000 7f6aca000640 1 -- 172.21.15.16:0/2394105748 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6ac406f920 msgr2=0x7f6ac406fd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:12.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:12.055+0000 7f6aca000640 1 --2- 172.21.15.16:0/2394105748 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6ac406f920 0x7f6ac406fd80 secure :-1 s=READY pgs=1379 cs=0 l=1 rev1=1 crypto rx=0x7f6aac0097c0 tx=0x7f6aac01c4a0 comp rx=0 tx=0).stop 2024-03-31T15:12:12.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:12.055+0000 7f6aca000640 1 -- 172.21.15.16:0/2394105748 shutdown_connections 2024-03-31T15:12:12.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:12.055+0000 7f6aca000640 1 --2- 172.21.15.16:0/2394105748 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f6ac40d2010 0x7f6ac40d24a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:12.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:12.055+0000 7f6aca000640 1 --2- 172.21.15.16:0/2394105748 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6ac406f920 0x7f6ac406fd80 unknown :-1 s=CLOSED pgs=1379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:12.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:12.055+0000 7f6aca000640 1 -- 172.21.15.16:0/2394105748 >> 172.21.15.16:0/2394105748 conn(0x7f6ac4067330 msgr2=0x7f6ac4067740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:12:12.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:12.055+0000 7f6aca000640 1 -- 172.21.15.16:0/2394105748 shutdown_connections 2024-03-31T15:12:12.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:12.055+0000 7f6aca000640 1 -- 172.21.15.16:0/2394105748 wait complete. 2024-03-31T15:12:12.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:12.059+0000 7f6aca000640 1 Processor -- start 2024-03-31T15:12:12.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:12.059+0000 7f6aca000640 1 -- start start 2024-03-31T15:12:12.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:12.059+0000 7f6aca000640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6ac406f920 0x7f6ac41436a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:12.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:12.059+0000 7f6aca000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6ac40d2010 0x7f6ac4141230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:12.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:12.059+0000 7f6aca000640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f6ac4145e40 0x7f6ac4140420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:12.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:12.059+0000 7f6aca000640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f6ac40d0ee0 con 0x7f6ac40d2010 2024-03-31T15:12:12.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:12.059+0000 7f6aca000640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f6ac40ce9e0 con 0x7f6ac406f920 2024-03-31T15:12:12.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:12.059+0000 7f6aca000640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f6ac40d2df0 con 0x7f6ac4145e40 2024-03-31T15:12:12.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:12.059+0000 7f6abbe00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6ac40d2010 0x7f6ac4141230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:12.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:12.059+0000 7f6abbe00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6ac40d2010 0x7f6ac4141230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37758/0 (socket says 172.21.15.16:37758) 2024-03-31T15:12:12.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:12.059+0000 7f6abbe00640 1 -- 172.21.15.16:0/4288172386 learned_addr learned my addr 172.21.15.16:0/4288172386 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:12:12.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:12.059+0000 7f6ac3400640 1 --2- 172.21.15.16:0/4288172386 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6ac406f920 0x7f6ac41436a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:12.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:12.059+0000 7f6ac3e00640 1 --2- 172.21.15.16:0/4288172386 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f6ac4145e40 0x7f6ac4140420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:12.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:12.059+0000 7f6abbe00640 1 -- 172.21.15.16:0/4288172386 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f6ac4145e40 msgr2=0x7f6ac4140420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:12.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:12.059+0000 7f6abbe00640 1 --2- 172.21.15.16:0/4288172386 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f6ac4145e40 0x7f6ac4140420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:12.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:12.059+0000 7f6abbe00640 1 -- 172.21.15.16:0/4288172386 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6ac406f920 msgr2=0x7f6ac41436a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:12.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:12.059+0000 7f6abbe00640 1 --2- 172.21.15.16:0/4288172386 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6ac406f920 0x7f6ac41436a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:12.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:12.059+0000 7f6abbe00640 1 -- 172.21.15.16:0/4288172386 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ac4143be0 con 0x7f6ac40d2010 2024-03-31T15:12:12.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:12.059+0000 7f6ac3400640 1 --2- 172.21.15.16:0/4288172386 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6ac406f920 0x7f6ac41436a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:12:12.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:12.059+0000 7f6ac3e00640 1 --2- 172.21.15.16:0/4288172386 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f6ac4145e40 0x7f6ac4140420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:12:12.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:12.059+0000 7f6abbe00640 1 --2- 172.21.15.16:0/4288172386 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6ac40d2010 0x7f6ac4141230 secure :-1 s=READY pgs=1380 cs=0 l=1 rev1=1 crypto rx=0x7f6aac0097c0 tx=0x7f6aac019240 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:12:12.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:12.059+0000 7f6abb400640 1 -- 172.21.15.16:0/4288172386 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6aac0ba020 con 0x7f6ac40d2010 2024-03-31T15:12:12.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:12.059+0000 7f6abb400640 1 -- 172.21.15.16:0/4288172386 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6aac0b6410 con 0x7f6ac40d2010 2024-03-31T15:12:12.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:12.059+0000 7f6aca000640 1 -- 172.21.15.16:0/4288172386 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6ac40b55a0 con 0x7f6ac40d2010 2024-03-31T15:12:12.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:12.059+0000 7f6abb400640 1 -- 172.21.15.16:0/4288172386 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6aac0b2040 con 0x7f6ac40d2010 2024-03-31T15:12:12.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:12.059+0000 7f6aca000640 1 -- 172.21.15.16:0/4288172386 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6ac40b5cc0 con 0x7f6ac40d2010 2024-03-31T15:12:12.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:12.063+0000 7f6aca000640 1 -- 172.21.15.16:0/4288172386 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6ac4069950 con 0x7f6ac40d2010 2024-03-31T15:12:12.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:12.063+0000 7f6abb400640 1 -- 172.21.15.16:0/4288172386 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f6aac0bd020 con 0x7f6ac40d2010 2024-03-31T15:12:12.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:12.063+0000 7f6abb400640 1 --2- 172.21.15.16:0/4288172386 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f6aa0068380 0x7f6aa006a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:12.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:12.063+0000 7f6abb400640 1 -- 172.21.15.16:0/4288172386 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 5115+0+0 (secure 0 0 0) 0x7f6aac1204d0 con 0x7f6ac40d2010 2024-03-31T15:12:12.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:12.063+0000 7f6ac3400640 1 --2- 172.21.15.16:0/4288172386 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f6aa0068380 0x7f6aa006a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:12.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:12.063+0000 7f6ac3400640 1 --2- 172.21.15.16:0/4288172386 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f6aa0068380 0x7f6aa006a840 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f6ab4005890 tx=0x7f6ab4009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:12:12.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:12.067+0000 7f6abb400640 1 -- 172.21.15.16:0/4288172386 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f6aac0c4030 con 0x7f6ac40d2010 2024-03-31T15:12:12.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:12.243+0000 7f6aca000640 1 -- 172.21.15.16:0/4288172386 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) -- 0x7f6ac40cf3c0 con 0x7f6ac40d2010 2024-03-31T15:12:12.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:12.855+0000 7f6abb400640 1 -- 172.21.15.16:0/4288172386 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v90) ==== 97+0+0 (secure 0 0 0) 0x7f6aac016090 con 0x7f6ac40d2010 2024-03-31T15:12:12.849 INFO:teuthology.orchestra.run.smithi016.stderr: removed all blocklist entries 2024-03-31T15:12:12.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:12.859+0000 7f6aca000640 1 -- 172.21.15.16:0/4288172386 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f6aa0068380 msgr2=0x7f6aa006a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:12.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:12.859+0000 7f6aca000640 1 --2- 172.21.15.16:0/4288172386 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f6aa0068380 0x7f6aa006a840 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f6ab4005890 tx=0x7f6ab4009040 comp rx=0 tx=0).stop 2024-03-31T15:12:12.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:12.859+0000 7f6aca000640 1 -- 172.21.15.16:0/4288172386 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6ac40d2010 msgr2=0x7f6ac4141230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:12.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:12.859+0000 7f6aca000640 1 --2- 172.21.15.16:0/4288172386 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6ac40d2010 0x7f6ac4141230 secure :-1 s=READY pgs=1380 cs=0 l=1 rev1=1 crypto rx=0x7f6aac0097c0 tx=0x7f6aac019240 comp rx=0 tx=0).stop 2024-03-31T15:12:12.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:12.859+0000 7f6aca000640 1 -- 172.21.15.16:0/4288172386 shutdown_connections 2024-03-31T15:12:12.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:12.859+0000 7f6aca000640 1 --2- 172.21.15.16:0/4288172386 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f6aa0068380 0x7f6aa006a840 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:12.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:12.859+0000 7f6aca000640 1 --2- 172.21.15.16:0/4288172386 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f6ac4145e40 0x7f6ac4140420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:12.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:12.859+0000 7f6aca000640 1 --2- 172.21.15.16:0/4288172386 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6ac40d2010 0x7f6ac4141230 unknown :-1 s=CLOSED pgs=1380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:12.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:12.859+0000 7f6aca000640 1 --2- 172.21.15.16:0/4288172386 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6ac406f920 0x7f6ac41436a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:12.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:12.859+0000 7f6aca000640 1 -- 172.21.15.16:0/4288172386 >> 172.21.15.16:0/4288172386 conn(0x7f6ac4067330 msgr2=0x7f6ac40b24c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:12:12.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:12.859+0000 7f6aca000640 1 -- 172.21.15.16:0/4288172386 shutdown_connections 2024-03-31T15:12:12.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:12.859+0000 7f6aca000640 1 -- 172.21.15.16:0/4288172386 wait complete. 2024-03-31T15:12:12.878 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-03-31T15:12:13.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:13.027+0000 7f04e9800640 1 Processor -- start 2024-03-31T15:12:13.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:13.027+0000 7f04e9800640 1 -- start start 2024-03-31T15:12:13.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:13.031+0000 7f04e9800640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f04e40b23d0 0x7f04e415cd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:13.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:13.031+0000 7f04e9800640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f04e4160e70 con 0x7f04e4145d10 2024-03-31T15:12:13.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:13.031+0000 7f04e9800640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f04e41602f0 con 0x7f04e40728b0 2024-03-31T15:12:13.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:13.031+0000 7f04e9800640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f04e41605d0 con 0x7f04e40b23d0 2024-03-31T15:12:13.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:13.031+0000 7f04e2a00640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f04e4145d10 0x7f04e40cf790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:41306/0 (socket says 172.21.15.16:41306) 2024-03-31T15:12:13.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:13.031+0000 7f04e2a00640 1 -- 172.21.15.16:0/1440559116 learned_addr learned my addr 172.21.15.16:0/1440559116 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:12:13.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:13.031+0000 7f04e3400640 1 --2- 172.21.15.16:0/1440559116 >> v2:172.21.15.89:3300/0 conn(0x7f04e40b23d0 0x7f04e415cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:13.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:13.031+0000 7f04e2000640 1 -- 172.21.15.16:0/1440559116 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 848059911 0 0) 0x7f04e4160e70 con 0x7f04e4145d10 2024-03-31T15:12:13.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:13.031+0000 7f04e2000640 1 -- 172.21.15.16:0/1440559116 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f04c8003530 con 0x7f04e4145d10 2024-03-31T15:12:13.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:13.031+0000 7f04e2000640 1 -- 172.21.15.16:0/1440559116 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1016039820 0 0) 0x7f04c8003530 con 0x7f04e4145d10 2024-03-31T15:12:13.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:13.031+0000 7f04e2000640 1 -- 172.21.15.16:0/1440559116 >> v2:172.21.15.89:3300/0 conn(0x7f04e40b23d0 msgr2=0x7f04e415cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:12:13.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:13.031+0000 7f04e2000640 1 --2- 172.21.15.16:0/1440559116 >> v2:172.21.15.89:3300/0 conn(0x7f04e40b23d0 0x7f04e415cd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:13.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:13.031+0000 7f04e2000640 1 -- 172.21.15.16:0/1440559116 >> v1:172.21.15.89:6789/0 conn(0x7f04e40728b0 legacy=0x7f04e40b1cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:13.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:13.031+0000 7f04e2000640 1 -- 172.21.15.16:0/1440559116 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04e41608b0 con 0x7f04e4145d10 2024-03-31T15:12:13.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:13.031+0000 7f04e2000640 1 -- 172.21.15.16:0/1440559116 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 214807623 0 0) 0x7f04d4001680 con 0x7f04e4145d10 2024-03-31T15:12:13.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:13.031+0000 7f04e2000640 1 -- 172.21.15.16:0/1440559116 >> v1:172.21.15.16:6789/0 conn(0x7f04e4145d10 legacy=0x7f04e40cf790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:13.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:13.031+0000 7f04e2000640 1 --2- 172.21.15.16:0/1440559116 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f04c8003a10 0x7f04c8003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:13.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:13.031+0000 7f04e2000640 1 --2- 172.21.15.16:0/1440559116 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f04c8004810 0x7f04c8004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:13.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:13.031+0000 7f04e2000640 1 --2- 172.21.15.16:0/1440559116 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f04c80051d0 0x7f04c800b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:13.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:13.031+0000 7f04e2000640 1 -- 172.21.15.16:0/1440559116 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f04e41605d0 con 0x7f04c8003a10 2024-03-31T15:12:13.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:13.031+0000 7f04e2000640 1 -- 172.21.15.16:0/1440559116 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f04c800c0d0 con 0x7f04c80051d0 2024-03-31T15:12:13.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:13.031+0000 7f04e2000640 1 -- 172.21.15.16:0/1440559116 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f04c800c2d0 con 0x7f04c8004810 2024-03-31T15:12:13.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:13.031+0000 7f04e2a00640 1 --2- 172.21.15.16:0/1440559116 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f04c8003a10 0x7f04c8003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:13.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:13.031+0000 7f04da000640 1 --2- 172.21.15.16:0/1440559116 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f04c8004810 0x7f04c8004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:13.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:13.031+0000 7f04e3400640 1 --2- 172.21.15.16:0/1440559116 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f04c80051d0 0x7f04c800b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:13.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:13.031+0000 7f04e2a00640 1 -- 172.21.15.16:0/1440559116 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f04c8004810 msgr2=0x7f04c8004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:12:13.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:13.031+0000 7f04e2a00640 1 --2- 172.21.15.16:0/1440559116 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f04c8004810 0x7f04c8004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:13.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:13.031+0000 7f04e2a00640 1 -- 172.21.15.16:0/1440559116 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f04c80051d0 msgr2=0x7f04c800b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:12:13.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:13.031+0000 7f04e2a00640 1 --2- 172.21.15.16:0/1440559116 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f04c80051d0 0x7f04c800b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:13.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:13.031+0000 7f04e2a00640 1 -- 172.21.15.16:0/1440559116 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04c8003530 con 0x7f04c8003a10 2024-03-31T15:12:13.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:13.031+0000 7f04e3400640 1 -- 172.21.15.16:0/1440559116 reap_dead start 2024-03-31T15:12:13.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:13.035+0000 7f04e2a00640 1 --2- 172.21.15.16:0/1440559116 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f04c8003a10 0x7f04c8003e30 secure :-1 s=READY pgs=1382 cs=0 l=1 rev1=1 crypto rx=0x7f04d400c110 tx=0x7f04d401cb40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c374825d1e7a1c3 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:12:13.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:13.035+0000 7f04e2000640 1 -- 172.21.15.16:0/1440559116 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04d4007c50 con 0x7f04c8003a10 2024-03-31T15:12:13.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:13.035+0000 7f04e2000640 1 -- 172.21.15.16:0/1440559116 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f04d40a9070 con 0x7f04c8003a10 2024-03-31T15:12:13.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:13.035+0000 7f04e2000640 1 -- 172.21.15.16:0/1440559116 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04d4005d80 con 0x7f04c8003a10 2024-03-31T15:12:13.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:13.035+0000 7f04e9800640 1 -- 172.21.15.16:0/1440559116 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f04c8003a10 msgr2=0x7f04c8003e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:13.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:13.035+0000 7f04e9800640 1 --2- 172.21.15.16:0/1440559116 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f04c8003a10 0x7f04c8003e30 secure :-1 s=READY pgs=1382 cs=0 l=1 rev1=1 crypto rx=0x7f04d400c110 tx=0x7f04d401cb40 comp rx=0 tx=0).stop 2024-03-31T15:12:13.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:13.035+0000 7f04e9800640 1 -- 172.21.15.16:0/1440559116 shutdown_connections 2024-03-31T15:12:13.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:13.035+0000 7f04e9800640 1 --2- 172.21.15.16:0/1440559116 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f04c8003a10 0x7f04c8003e30 unknown :-1 s=CLOSED pgs=1382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:13.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:13.035+0000 7f04e9800640 1 -- 172.21.15.16:0/1440559116 >> 172.21.15.16:0/1440559116 conn(0x7f04e406b620 msgr2=0x7f04e406ba30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:12:13.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:13.035+0000 7f04e9800640 1 -- 172.21.15.16:0/1440559116 shutdown_connections 2024-03-31T15:12:13.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:13.035+0000 7f04e9800640 1 -- 172.21.15.16:0/1440559116 wait complete. 2024-03-31T15:12:13.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:13.039+0000 7f04e9800640 1 Processor -- start 2024-03-31T15:12:13.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:13.039+0000 7f04e9800640 1 -- start start 2024-03-31T15:12:13.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:13.039+0000 7f04e9800640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f04c8003a10 0x7f04e414e690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:13.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:13.039+0000 7f04e9800640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f04e4144a80 0x7f04e41e7dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:13.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:13.039+0000 7f04e9800640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f04e415e5b0 0x7f04e415ea30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:13.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:13.039+0000 7f04e9800640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f04e40cfd20 con 0x7f04e415e5b0 2024-03-31T15:12:13.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:13.039+0000 7f04e9800640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f04e40cfba0 con 0x7f04e4144a80 2024-03-31T15:12:13.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:13.039+0000 7f04e9800640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f04e40cf350 con 0x7f04c8003a10 2024-03-31T15:12:13.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:13.039+0000 7f04e3400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f04e415e5b0 0x7f04e415ea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:13.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:13.039+0000 7f04e3400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f04e415e5b0 0x7f04e415ea30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37768/0 (socket says 172.21.15.16:37768) 2024-03-31T15:12:13.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:13.039+0000 7f04e3400640 1 -- 172.21.15.16:0/3368817855 learned_addr learned my addr 172.21.15.16:0/3368817855 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:12:13.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:13.039+0000 7f04e2a00640 1 --2- 172.21.15.16:0/3368817855 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f04c8003a10 0x7f04e414e690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:13.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:13.039+0000 7f04da000640 1 --2- 172.21.15.16:0/3368817855 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f04e4144a80 0x7f04e41e7dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:13.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:13.039+0000 7f04e3400640 1 -- 172.21.15.16:0/3368817855 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f04c8003a10 msgr2=0x7f04e414e690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:13.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:13.039+0000 7f04e3400640 1 --2- 172.21.15.16:0/3368817855 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f04c8003a10 0x7f04e414e690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:13.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:13.039+0000 7f04e3400640 1 -- 172.21.15.16:0/3368817855 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f04e4144a80 msgr2=0x7f04e41e7dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:13.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:13.039+0000 7f04e3400640 1 --2- 172.21.15.16:0/3368817855 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f04e4144a80 0x7f04e41e7dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:13.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:13.039+0000 7f04e3400640 1 -- 172.21.15.16:0/3368817855 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04e4143700 con 0x7f04e415e5b0 2024-03-31T15:12:13.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:13.039+0000 7f04e3400640 1 --2- 172.21.15.16:0/3368817855 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f04e415e5b0 0x7f04e415ea30 secure :-1 s=READY pgs=1383 cs=0 l=1 rev1=1 crypto rx=0x7f04dc011bf0 tx=0x7f04dc00f580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:12:13.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:13.039+0000 7f04db400640 1 -- 172.21.15.16:0/3368817855 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04dc00fdb0 con 0x7f04e415e5b0 2024-03-31T15:12:13.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:13.039+0000 7f04db400640 1 -- 172.21.15.16:0/3368817855 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f04dc01d070 con 0x7f04e415e5b0 2024-03-31T15:12:13.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:13.039+0000 7f04db400640 1 -- 172.21.15.16:0/3368817855 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04dc015670 con 0x7f04e415e5b0 2024-03-31T15:12:13.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:13.039+0000 7f04e9800640 1 -- 172.21.15.16:0/3368817855 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f04e41439e0 con 0x7f04e415e5b0 2024-03-31T15:12:13.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:13.039+0000 7f04e9800640 1 -- 172.21.15.16:0/3368817855 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f04e4143fa0 con 0x7f04e415e5b0 2024-03-31T15:12:13.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:13.039+0000 7f04e9800640 1 -- 172.21.15.16:0/3368817855 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f04e41445b0 con 0x7f04e415e5b0 2024-03-31T15:12:13.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:13.043+0000 7f04db400640 1 -- 172.21.15.16:0/3368817855 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f04dc015810 con 0x7f04e415e5b0 2024-03-31T15:12:13.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:13.043+0000 7f04db400640 1 --2- 172.21.15.16:0/3368817855 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f04c0068380 0x7f04c006a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:13.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:13.043+0000 7f04db400640 1 -- 172.21.15.16:0/3368817855 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 4857+0+0 (secure 0 0 0) 0x7f04dc087090 con 0x7f04e415e5b0 2024-03-31T15:12:13.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:13.043+0000 7f04e2a00640 1 --2- 172.21.15.16:0/3368817855 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f04c0068380 0x7f04c006a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:13.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:13.043+0000 7f04e2a00640 1 --2- 172.21.15.16:0/3368817855 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f04c0068380 0x7f04c006a840 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f04d40ab000 tx=0x7f04d40a7070 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:12:13.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:13.047+0000 7f04db400640 1 -- 172.21.15.16:0/3368817855 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f04dc027020 con 0x7f04e415e5b0 2024-03-31T15:12:13.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:13.267+0000 7f04e9800640 1 -- 172.21.15.16:0/3368817855 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) -- 0x7f04e415ef70 con 0x7f04e415e5b0 2024-03-31T15:12:13.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:13.267+0000 7f04db400640 1 -- 172.21.15.16:0/3368817855 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v18) ==== 80+0+5304 (secure 0 0 0) 0x7f04dc010040 con 0x7f04e415e5b0 2024-03-31T15:12:13.261 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:12:13.262 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-03-31T15:12:13.262 INFO:teuthology.orchestra.run.smithi016.stdout: "auth_dump": [ 2024-03-31T15:12:13.262 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-31T15:12:13.262 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "mds.a", 2024-03-31T15:12:13.262 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAOewlmhp1ANRAAYS9zo+p4tLU1YK0ySOZE/Q==", 2024-03-31T15:12:13.262 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2024-03-31T15:12:13.262 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2024-03-31T15:12:13.262 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow *", 2024-03-31T15:12:13.262 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow *", 2024-03-31T15:12:13.262 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2024-03-31T15:12:13.262 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-31T15:12:13.262 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-31T15:12:13.262 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-31T15:12:13.263 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "mds.b", 2024-03-31T15:12:13.263 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAPewlmXxupBhAAtpkt93aJOFqhc7GXLIlkBg==", 2024-03-31T15:12:13.263 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2024-03-31T15:12:13.263 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2024-03-31T15:12:13.263 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow *", 2024-03-31T15:12:13.263 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow *", 2024-03-31T15:12:13.263 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2024-03-31T15:12:13.263 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-31T15:12:13.263 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-31T15:12:13.263 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-31T15:12:13.263 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "mds.c", 2024-03-31T15:12:13.263 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAPewlmjaieARAA32teg02vE/Q8RtqLtnArMw==", 2024-03-31T15:12:13.263 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2024-03-31T15:12:13.264 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2024-03-31T15:12:13.264 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow *", 2024-03-31T15:12:13.264 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow *", 2024-03-31T15:12:13.264 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2024-03-31T15:12:13.264 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-31T15:12:13.264 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-31T15:12:13.264 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-31T15:12:13.264 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "mds.d", 2024-03-31T15:12:13.264 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAPewlmo+4PDhAAmJD0siZB3LjWkvdMq1d8aw==", 2024-03-31T15:12:13.264 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2024-03-31T15:12:13.264 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2024-03-31T15:12:13.264 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow *", 2024-03-31T15:12:13.264 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow *", 2024-03-31T15:12:13.264 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2024-03-31T15:12:13.264 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-31T15:12:13.264 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-31T15:12:13.265 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-31T15:12:13.265 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.0", 2024-03-31T15:12:13.265 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAQewlmVP6kLhAAwFr4jwQgz1N3oIUZuJvmzw==", 2024-03-31T15:12:13.265 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2024-03-31T15:12:13.265 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2024-03-31T15:12:13.265 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2024-03-31T15:12:13.265 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2024-03-31T15:12:13.265 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-31T15:12:13.265 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-31T15:12:13.265 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-31T15:12:13.265 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.1", 2024-03-31T15:12:13.265 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQATewlmBh0gNhAAR6LiuFypwA7d3SwKlWuI/Q==", 2024-03-31T15:12:13.265 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2024-03-31T15:12:13.265 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2024-03-31T15:12:13.265 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2024-03-31T15:12:13.266 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2024-03-31T15:12:13.266 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-31T15:12:13.266 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-31T15:12:13.266 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-31T15:12:13.266 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.2", 2024-03-31T15:12:13.266 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAXewlmtQwhARAACbzc7Woo9BoVFPfK3Pns+g==", 2024-03-31T15:12:13.266 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2024-03-31T15:12:13.266 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2024-03-31T15:12:13.266 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2024-03-31T15:12:13.266 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2024-03-31T15:12:13.266 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-31T15:12:13.266 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-31T15:12:13.266 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-31T15:12:13.267 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.3", 2024-03-31T15:12:13.267 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAaewlmmjWABhAA/hwvARNb51nd5vsnYVD/sg==", 2024-03-31T15:12:13.267 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2024-03-31T15:12:13.267 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2024-03-31T15:12:13.267 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2024-03-31T15:12:13.267 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2024-03-31T15:12:13.267 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-31T15:12:13.267 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-31T15:12:13.267 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-31T15:12:13.267 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.4", 2024-03-31T15:12:13.267 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAgewlmby07CxAAChxE/qPuPboIlg3fDxSOeA==", 2024-03-31T15:12:13.267 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2024-03-31T15:12:13.267 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2024-03-31T15:12:13.267 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2024-03-31T15:12:13.268 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2024-03-31T15:12:13.268 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-31T15:12:13.268 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-31T15:12:13.268 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-31T15:12:13.268 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.5", 2024-03-31T15:12:13.268 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAjewlmlD8yExAA9pgP0heBywrgqc39iU8rpQ==", 2024-03-31T15:12:13.268 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2024-03-31T15:12:13.268 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2024-03-31T15:12:13.268 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2024-03-31T15:12:13.268 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2024-03-31T15:12:13.268 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-31T15:12:13.268 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-31T15:12:13.268 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-31T15:12:13.268 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.6", 2024-03-31T15:12:13.269 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAmewlmzAsTGBAALMjVcQ08sIiTDFaBcq8L6A==", 2024-03-31T15:12:13.269 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2024-03-31T15:12:13.269 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2024-03-31T15:12:13.269 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2024-03-31T15:12:13.269 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2024-03-31T15:12:13.269 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-31T15:12:13.269 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-31T15:12:13.269 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-31T15:12:13.269 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.7", 2024-03-31T15:12:13.269 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQApewlmb+CEHRAAXE+iBumDsb2c4zH3C8f6YA==", 2024-03-31T15:12:13.269 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2024-03-31T15:12:13.269 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2024-03-31T15:12:13.269 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2024-03-31T15:12:13.269 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2024-03-31T15:12:13.270 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-31T15:12:13.270 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-31T15:12:13.270 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-31T15:12:13.270 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "client.0", 2024-03-31T15:12:13.270 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAPewlmlMIDEhAAF0Ll1HcrWHmH6xDKq2Qq9A==", 2024-03-31T15:12:13.270 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2024-03-31T15:12:13.270 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2024-03-31T15:12:13.270 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow r", 2024-03-31T15:12:13.270 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-03-31T15:12:13.270 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-31T15:12:13.270 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-31T15:12:13.270 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-31T15:12:13.270 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "client.1", 2024-03-31T15:12:13.271 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAPewlmdsvqGxAAaVdwWwri0IE/2sg7psilWw==", 2024-03-31T15:12:13.271 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2024-03-31T15:12:13.271 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2024-03-31T15:12:13.271 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow r", 2024-03-31T15:12:13.271 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow rw", 2024-03-31T15:12:13.271 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow rwx" 2024-03-31T15:12:13.271 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-31T15:12:13.271 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-31T15:12:13.271 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-31T15:12:13.271 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "client.2", 2024-03-31T15:12:13.271 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAPewlmQapPGBAAbwXSlCi+zyQ9118WyAJmrQ==", 2024-03-31T15:12:13.271 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2024-03-31T15:12:13.271 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2024-03-31T15:12:13.271 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow r", 2024-03-31T15:12:13.272 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow rw", 2024-03-31T15:12:13.272 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow rwx" 2024-03-31T15:12:13.272 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-31T15:12:13.272 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-31T15:12:13.272 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-31T15:12:13.272 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "client.3", 2024-03-31T15:12:13.272 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAPewlmCw7qIRAAEeIjF85JbyfJ6d3FtMOVhQ==", 2024-03-31T15:12:13.272 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2024-03-31T15:12:13.272 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2024-03-31T15:12:13.272 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow r", 2024-03-31T15:12:13.272 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow rw", 2024-03-31T15:12:13.272 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow rwx" 2024-03-31T15:12:13.272 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-31T15:12:13.273 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-31T15:12:13.273 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-31T15:12:13.273 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "client.admin", 2024-03-31T15:12:13.273 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAOewlmXyPZGRAAtZOheiYhzGxdeJGqqyGxTw==", 2024-03-31T15:12:13.273 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2024-03-31T15:12:13.273 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow *", 2024-03-31T15:12:13.273 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow *", 2024-03-31T15:12:13.273 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow *", 2024-03-31T15:12:13.273 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2024-03-31T15:12:13.273 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-31T15:12:13.273 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-31T15:12:13.273 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-31T15:12:13.273 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "mgr.x", 2024-03-31T15:12:13.273 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAOewlm7UkKMRAA8hqgeV86mTCkRcxdXNdWmw==", 2024-03-31T15:12:13.273 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2024-03-31T15:12:13.274 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow *", 2024-03-31T15:12:13.274 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile mgr", 2024-03-31T15:12:13.274 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2024-03-31T15:12:13.274 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-31T15:12:13.274 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-31T15:12:13.274 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-31T15:12:13.274 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "mgr.y", 2024-03-31T15:12:13.274 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAOewlmq0cpLRAAKTgh04tRaHrpMhxdwgvXYA==", 2024-03-31T15:12:13.274 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2024-03-31T15:12:13.274 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow *", 2024-03-31T15:12:13.274 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile mgr", 2024-03-31T15:12:13.274 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2024-03-31T15:12:13.274 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-31T15:12:13.274 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-31T15:12:13.274 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-03-31T15:12:13.275 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-03-31T15:12:13.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:13.271+0000 7f04e9800640 1 -- 172.21.15.16:0/3368817855 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f04c0068380 msgr2=0x7f04c006a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:13.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:13.271+0000 7f04e9800640 1 --2- 172.21.15.16:0/3368817855 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f04c0068380 0x7f04c006a840 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f04d40ab000 tx=0x7f04d40a7070 comp rx=0 tx=0).stop 2024-03-31T15:12:13.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:13.271+0000 7f04e9800640 1 -- 172.21.15.16:0/3368817855 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f04e415e5b0 msgr2=0x7f04e415ea30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:13.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:13.271+0000 7f04e9800640 1 --2- 172.21.15.16:0/3368817855 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f04e415e5b0 0x7f04e415ea30 secure :-1 s=READY pgs=1383 cs=0 l=1 rev1=1 crypto rx=0x7f04dc011bf0 tx=0x7f04dc00f580 comp rx=0 tx=0).stop 2024-03-31T15:12:13.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:13.271+0000 7f04e9800640 1 -- 172.21.15.16:0/3368817855 shutdown_connections 2024-03-31T15:12:13.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:13.271+0000 7f04e9800640 1 --2- 172.21.15.16:0/3368817855 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f04c0068380 0x7f04c006a840 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:13.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:13.271+0000 7f04e9800640 1 --2- 172.21.15.16:0/3368817855 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f04e415e5b0 0x7f04e415ea30 unknown :-1 s=CLOSED pgs=1383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:13.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:13.271+0000 7f04e9800640 1 --2- 172.21.15.16:0/3368817855 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f04e4144a80 0x7f04e41e7dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:13.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:13.271+0000 7f04e9800640 1 --2- 172.21.15.16:0/3368817855 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f04c8003a10 0x7f04e414e690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:13.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:13.271+0000 7f04e9800640 1 -- 172.21.15.16:0/3368817855 >> 172.21.15.16:0/3368817855 conn(0x7f04e406b620 msgr2=0x7f04e40b4bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:12:13.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:13.271+0000 7f04e9800640 1 -- 172.21.15.16:0/3368817855 shutdown_connections 2024-03-31T15:12:13.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:13.271+0000 7f04e9800640 1 -- 172.21.15.16:0/3368817855 wait complete. 2024-03-31T15:12:13.283 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-31T15:12:13.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:13.431+0000 7f6dccc00640 1 Processor -- start 2024-03-31T15:12:13.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:13.431+0000 7f6dccc00640 1 -- start start 2024-03-31T15:12:13.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:13.431+0000 7f6dccc00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f6dc806f920 0x7f6dc806fd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:13.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:13.431+0000 7f6dccc00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f6dc80cc740 0x7f6dc80ccbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:13.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:13.431+0000 7f6dccc00640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6dc8162010 con 0x7f6dc806d490 2024-03-31T15:12:13.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:13.431+0000 7f6dccc00640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f6dc8162be0 con 0x7f6dc806f920 2024-03-31T15:12:13.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:13.431+0000 7f6dccc00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f6dc8070240 con 0x7f6dc80cc740 2024-03-31T15:12:13.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:13.431+0000 7f6dc5e00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f6dc806f920 0x7f6dc806fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:13.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:13.431+0000 7f6dc6800640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f6dc80cc740 0x7f6dc80ccbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:13.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:13.431+0000 7f6dbfe00640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f6dc806d490 0x7f6dc806d8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.16:43334/0 (socket says 172.21.15.16:43334) 2024-03-31T15:12:13.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:13.431+0000 7f6dbfe00640 1 -- 172.21.15.16:0/56269480 learned_addr learned my addr 172.21.15.16:0/56269480 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:12:13.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:13.431+0000 7f6dc5e00640 1 -- 172.21.15.16:0/56269480 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f6dc80cc740 msgr2=0x7f6dc80ccbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:12:13.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:13.431+0000 7f6dc5e00640 1 --2- 172.21.15.16:0/56269480 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f6dc80cc740 0x7f6dc80ccbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:13.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:13.431+0000 7f6dc5e00640 1 -- 172.21.15.16:0/56269480 >> v1:172.21.15.89:6789/0 conn(0x7f6dc806d490 legacy=0x7f6dc806d8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:13.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:13.431+0000 7f6dc5e00640 1 -- 172.21.15.16:0/56269480 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6dc81622b0 con 0x7f6dc806f920 2024-03-31T15:12:13.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:13.435+0000 7f6dc5e00640 1 --2- 172.21.15.16:0/56269480 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6dc806f920 0x7f6dc806fd00 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f6dc8162d60 tx=0x7f6db801c4a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=964a68b08516fa39 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:12:13.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:13.435+0000 7f6dc5400640 1 -- 172.21.15.16:0/56269480 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6db80b2070 con 0x7f6dc806f920 2024-03-31T15:12:13.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:13.435+0000 7f6dc5400640 1 -- 172.21.15.16:0/56269480 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6db80ad640 con 0x7f6dc806f920 2024-03-31T15:12:13.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:13.435+0000 7f6dc5400640 1 -- 172.21.15.16:0/56269480 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6db80ada10 con 0x7f6dc806f920 2024-03-31T15:12:13.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:13.435+0000 7f6dccc00640 1 -- 172.21.15.16:0/56269480 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6dc806f920 msgr2=0x7f6dc806fd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:13.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:13.435+0000 7f6dccc00640 1 --2- 172.21.15.16:0/56269480 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6dc806f920 0x7f6dc806fd00 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f6dc8162d60 tx=0x7f6db801c4a0 comp rx=0 tx=0).stop 2024-03-31T15:12:13.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:13.435+0000 7f6dccc00640 1 -- 172.21.15.16:0/56269480 shutdown_connections 2024-03-31T15:12:13.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:13.435+0000 7f6dccc00640 1 --2- 172.21.15.16:0/56269480 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f6dc80cc740 0x7f6dc80ccbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:13.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:13.435+0000 7f6dccc00640 1 --2- 172.21.15.16:0/56269480 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6dc806f920 0x7f6dc806fd00 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:13.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:13.435+0000 7f6dccc00640 1 -- 172.21.15.16:0/56269480 >> 172.21.15.16:0/56269480 conn(0x7f6dc8067330 msgr2=0x7f6dc8067740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:12:13.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:13.435+0000 7f6dccc00640 1 -- 172.21.15.16:0/56269480 shutdown_connections 2024-03-31T15:12:13.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:13.435+0000 7f6dccc00640 1 -- 172.21.15.16:0/56269480 wait complete. 2024-03-31T15:12:13.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:13.439+0000 7f6dccc00640 1 Processor -- start 2024-03-31T15:12:13.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:13.439+0000 7f6dccc00640 1 -- start start 2024-03-31T15:12:13.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:13.439+0000 7f6dccc00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6dc806d490 0x7f6dc81613d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:13.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:13.439+0000 7f6dccc00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f6dc806f920 0x7f6dc80b5980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:13.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:13.439+0000 7f6dccc00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6dc80cc740 0x7f6dc80b5ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:13.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:13.439+0000 7f6dccc00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f6dc816b530 con 0x7f6dc806d490 2024-03-31T15:12:13.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:13.439+0000 7f6dccc00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f6dc816b3b0 con 0x7f6dc80cc740 2024-03-31T15:12:13.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:13.439+0000 7f6dccc00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f6dc816b6b0 con 0x7f6dc806f920 2024-03-31T15:12:13.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:13.439+0000 7f6dc5e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6dc806d490 0x7f6dc81613d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:13.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:13.439+0000 7f6dc5e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6dc806d490 0x7f6dc81613d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37772/0 (socket says 172.21.15.16:37772) 2024-03-31T15:12:13.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:13.439+0000 7f6dc5e00640 1 -- 172.21.15.16:0/2167012343 learned_addr learned my addr 172.21.15.16:0/2167012343 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:12:13.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:13.439+0000 7f6dbfe00640 1 --2- 172.21.15.16:0/2167012343 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f6dc806f920 0x7f6dc80b5980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:13.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:13.439+0000 7f6dc6800640 1 --2- 172.21.15.16:0/2167012343 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6dc80cc740 0x7f6dc80b5ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:13.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:13.439+0000 7f6dc5e00640 1 -- 172.21.15.16:0/2167012343 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f6dc806f920 msgr2=0x7f6dc80b5980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:13.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:13.439+0000 7f6dc5e00640 1 --2- 172.21.15.16:0/2167012343 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f6dc806f920 0x7f6dc80b5980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:13.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:13.439+0000 7f6dc5e00640 1 -- 172.21.15.16:0/2167012343 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6dc80cc740 msgr2=0x7f6dc80b5ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:13.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:13.439+0000 7f6dc5e00640 1 --2- 172.21.15.16:0/2167012343 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6dc80cc740 0x7f6dc80b5ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:13.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:13.439+0000 7f6dc5e00640 1 -- 172.21.15.16:0/2167012343 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6dc813f180 con 0x7f6dc806d490 2024-03-31T15:12:13.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:13.439+0000 7f6dc5e00640 1 --2- 172.21.15.16:0/2167012343 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6dc806d490 0x7f6dc81613d0 secure :-1 s=READY pgs=1384 cs=0 l=1 rev1=1 crypto rx=0x7f6db80b0040 tx=0x7f6db8019240 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:12:13.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:13.439+0000 7f6dbe000640 1 -- 172.21.15.16:0/2167012343 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6db80b2040 con 0x7f6dc806d490 2024-03-31T15:12:13.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:13.439+0000 7f6dbe000640 1 -- 172.21.15.16:0/2167012343 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6db80bd070 con 0x7f6dc806d490 2024-03-31T15:12:13.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:13.439+0000 7f6dccc00640 1 -- 172.21.15.16:0/2167012343 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6dc813f460 con 0x7f6dc806d490 2024-03-31T15:12:13.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:13.439+0000 7f6dbe000640 1 -- 172.21.15.16:0/2167012343 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6db80b6630 con 0x7f6dc806d490 2024-03-31T15:12:13.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:13.439+0000 7f6dccc00640 1 -- 172.21.15.16:0/2167012343 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6dc813fa20 con 0x7f6dc806d490 2024-03-31T15:12:13.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:13.443+0000 7f6dccc00640 1 -- 172.21.15.16:0/2167012343 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6dc8069950 con 0x7f6dc806d490 2024-03-31T15:12:13.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:13.443+0000 7f6dbe000640 1 -- 172.21.15.16:0/2167012343 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f6db80b67d0 con 0x7f6dc806d490 2024-03-31T15:12:13.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:13.443+0000 7f6dbe000640 1 --2- 172.21.15.16:0/2167012343 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f6da4068380 0x7f6da406a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:13.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:13.443+0000 7f6dbe000640 1 -- 172.21.15.16:0/2167012343 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 4857+0+0 (secure 0 0 0) 0x7f6db81204c0 con 0x7f6dc806d490 2024-03-31T15:12:13.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:13.443+0000 7f6dbfe00640 1 --2- 172.21.15.16:0/2167012343 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f6da4068380 0x7f6da406a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:13.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:13.443+0000 7f6dbfe00640 1 --2- 172.21.15.16:0/2167012343 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f6da4068380 0x7f6da406a840 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f6dc813e2f0 tx=0x7f6db0007e90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:12:13.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:13.447+0000 7f6dbe000640 1 -- 172.21.15.16:0/2167012343 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f6db8016090 con 0x7f6dc806d490 2024-03-31T15:12:13.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:13.615+0000 7f6dccc00640 1 -- 172.21.15.16:0/2167012343 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f6dc80cf3f0 con 0x7f6dc806d490 2024-03-31T15:12:13.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:13.615+0000 7f6dbe000640 1 -- 172.21.15.16:0/2167012343 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v92) ==== 74+0+12820 (secure 0 0 0) 0x7f6db80f0800 con 0x7f6dc806d490 2024-03-31T15:12:13.608 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:12:13.608 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":92,"fsid":"286eb5da-fb47-4550-b290-d9519c13b9a5","created":"2024-03-31T15:03:11.438952+0000","modified":"2024-03-31T15:12:12.869708+0000","last_up_change":"2024-03-31T15:03:22.885535+0000","last_in_change":"2024-03-31T15:03:13.941667+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-31T15:03:22.550787+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"92a66450-ef19-40d8-ac3d-d760b23b0bda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6801","nonce":932712244}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6803","nonce":932712244}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6807","nonce":932712244}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6805","nonce":932712244}]},"public_addr":"172.21.15.16:6801/932712244","cluster_addr":"172.21.15.16:6803/932712244","heartbeat_back_addr":"172.21.15.16:6807/932712244","heartbeat_front_addr":"172.21.15.16:6805/932712244","state":["exists","up"]},{"osd":1,"uuid":"da9d1f2b-1968-4959-b843-faa231988e92","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6810","nonce":1398737533}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6811","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6813","nonce":1398737533}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6821","nonce":1398737533}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6815","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6816","nonce":1398737533}]},"public_addr":"172.21.15.16:6810/1398737533","cluster_addr":"172.21.15.16:6813/1398737533","heartbeat_back_addr":"172.21.15.16:6821/1398737533","heartbeat_front_addr":"172.21.15.16:6816/1398737533","state":["exists","up"]},{"osd":2,"uuid":"01a73662-9a21-4b1f-bcbb-42b9636a4638","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6819","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6822","nonce":3018526646}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6825","nonce":3018526646}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6831","nonce":3018526646}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6827","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6829","nonce":3018526646}]},"public_addr":"172.21.15.16:6822/3018526646","cluster_addr":"172.21.15.16:6825/3018526646","heartbeat_back_addr":"172.21.15.16:6831/3018526646","heartbeat_front_addr":"172.21.15.16:6829/3018526646","state":["exists","up"]},{"osd":3,"uuid":"c7771fd3-e18e-4500-ac38-23cf10ebcd75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6809","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6812","nonce":3360352974}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6817","nonce":3360352974}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6828","nonce":3360352974}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6823","nonce":3360352974}]},"public_addr":"172.21.15.16:6812/3360352974","cluster_addr":"172.21.15.16:6817/3360352974","heartbeat_back_addr":"172.21.15.16:6828/3360352974","heartbeat_front_addr":"172.21.15.16:6823/3360352974","state":["exists","up"]},{"osd":4,"uuid":"88260730-1482-4fbd-a40f-cb1124e9df4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6801","nonce":1856844856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6803","nonce":1856844856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6807","nonce":1856844856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6805","nonce":1856844856}]},"public_addr":"172.21.15.89:6801/1856844856","cluster_addr":"172.21.15.89:6803/1856844856","heartbeat_back_addr":"172.21.15.89:6807/1856844856","heartbeat_front_addr":"172.21.15.89:6805/1856844856","state":["exists","up"]},{"osd":5,"uuid":"4cb2c00c-fae0-4dd8-9d27-ec4f0093ccf7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6814","nonce":50395527}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6817","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6820","nonce":50395527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6831","nonce":50395527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6823","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6826","nonce":50395527}]},"public_addr":"172.21.15.89:6814/50395527","cluster_addr":"172.21.15.89:6820/50395527","heartbeat_back_addr":"172.21.15.89:6831/50395527","heartbeat_front_addr":"172.21.15.89:6826/50395527","state":["exists","up"]},{"osd":6,"uuid":"81ce0c19-2554-4cae-acbe-66a05e9e9b9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6809","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6812","nonce":3470471752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6815","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6818","nonce":3470471752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6827","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6829","nonce":3470471752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6821","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6824","nonce":3470471752}]},"public_addr":"172.21.15.89:6812/3470471752","cluster_addr":"172.21.15.89:6818/3470471752","heartbeat_back_addr":"172.21.15.89:6829/3470471752","heartbeat_front_addr":"172.21.15.89:6824/3470471752","state":["exists","up"]},{"osd":7,"uuid":"85e4313b-d37d-442e-8042-153bf20a847d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6811","nonce":1796778630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6813","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6816","nonce":1796778630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6825","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6828","nonce":1796778630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6819","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6822","nonce":1796778630}]},"public_addr":"172.21.15.89:6811/1796778630","cluster_addr":"172.21.15.89:6816/1796778630","heartbeat_back_addr":"172.21.15.89:6828/1796778630","heartbeat_front_addr":"172.21.15.89:6822/1796778630","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.439088+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.529970+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.680512+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:21.044161+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.582111+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.536558+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.409522+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.590229+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-31T15:12:13.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:13.615+0000 7f6dccc00640 1 -- 172.21.15.16:0/2167012343 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f6da4068380 msgr2=0x7f6da406a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:13.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:13.615+0000 7f6dccc00640 1 --2- 172.21.15.16:0/2167012343 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f6da4068380 0x7f6da406a840 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f6dc813e2f0 tx=0x7f6db0007e90 comp rx=0 tx=0).stop 2024-03-31T15:12:13.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:13.615+0000 7f6dccc00640 1 -- 172.21.15.16:0/2167012343 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6dc806d490 msgr2=0x7f6dc81613d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:13.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:13.615+0000 7f6dccc00640 1 --2- 172.21.15.16:0/2167012343 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6dc806d490 0x7f6dc81613d0 secure :-1 s=READY pgs=1384 cs=0 l=1 rev1=1 crypto rx=0x7f6db80b0040 tx=0x7f6db8019240 comp rx=0 tx=0).stop 2024-03-31T15:12:13.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:13.619+0000 7f6dccc00640 1 -- 172.21.15.16:0/2167012343 shutdown_connections 2024-03-31T15:12:13.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:13.619+0000 7f6dccc00640 1 --2- 172.21.15.16:0/2167012343 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f6da4068380 0x7f6da406a840 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:13.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:13.619+0000 7f6dccc00640 1 --2- 172.21.15.16:0/2167012343 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6dc80cc740 0x7f6dc80b5ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:13.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:13.619+0000 7f6dccc00640 1 --2- 172.21.15.16:0/2167012343 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f6dc806f920 0x7f6dc80b5980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:13.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:13.619+0000 7f6dccc00640 1 --2- 172.21.15.16:0/2167012343 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6dc806d490 0x7f6dc81613d0 unknown :-1 s=CLOSED pgs=1384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:13.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:13.619+0000 7f6dccc00640 1 -- 172.21.15.16:0/2167012343 >> 172.21.15.16:0/2167012343 conn(0x7f6dc8067330 msgr2=0x7f6dc80d26f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:12:13.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:13.619+0000 7f6dccc00640 1 -- 172.21.15.16:0/2167012343 shutdown_connections 2024-03-31T15:12:13.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:13.619+0000 7f6dccc00640 1 -- 172.21.15.16:0/2167012343 wait complete. 2024-03-31T15:12:13.629 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-31T15:03:22.550787+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-31T15:12:13.630 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-31T15:12:13.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:13.779+0000 7fe56ba00640 1 Processor -- start 2024-03-31T15:12:13.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:13.779+0000 7fe56ba00640 1 -- start start 2024-03-31T15:12:13.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:13.779+0000 7fe56ba00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fe56406d4d0 0x7fe56406d930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:13.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:13.779+0000 7fe56ba00640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe56406de70 con 0x7fe5641567e0 2024-03-31T15:12:13.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:13.779+0000 7fe56ba00640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe56415c7e0 con 0x7fe564071550 2024-03-31T15:12:13.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:13.779+0000 7fe56ba00640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7fe5640736b0 con 0x7fe56406d4d0 2024-03-31T15:12:13.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:13.783+0000 7fe569000640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fe5641567e0 0x7fe564072c80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:41318/0 (socket says 172.21.15.16:41318) 2024-03-31T15:12:13.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:13.783+0000 7fe569000640 1 -- 172.21.15.16:0/1655741930 learned_addr learned my addr 172.21.15.16:0/1655741930 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:12:13.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:13.783+0000 7fe55be00640 1 --2- 172.21.15.16:0/1655741930 >> v2:172.21.15.89:3300/0 conn(0x7fe56406d4d0 0x7fe56406d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:13.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:13.783+0000 7fe563e00640 1 -- 172.21.15.16:0/1655741930 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1289127063 0 0) 0x7fe56406de70 con 0x7fe5641567e0 2024-03-31T15:12:13.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:13.783+0000 7fe563e00640 1 -- 172.21.15.16:0/1655741930 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe548003530 con 0x7fe5641567e0 2024-03-31T15:12:13.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:13.783+0000 7fe563e00640 1 -- 172.21.15.16:0/1655741930 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2635860923 0 0) 0x7fe548003530 con 0x7fe5641567e0 2024-03-31T15:12:13.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:13.783+0000 7fe563e00640 1 -- 172.21.15.16:0/1655741930 >> v2:172.21.15.89:3300/0 conn(0x7fe56406d4d0 msgr2=0x7fe56406d930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:12:13.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:13.783+0000 7fe563e00640 1 --2- 172.21.15.16:0/1655741930 >> v2:172.21.15.89:3300/0 conn(0x7fe56406d4d0 0x7fe56406d930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:13.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:13.783+0000 7fe563e00640 1 -- 172.21.15.16:0/1655741930 >> v1:172.21.15.89:6789/0 conn(0x7fe564071550 legacy=0x7fe564071960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:13.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:13.783+0000 7fe563e00640 1 -- 172.21.15.16:0/1655741930 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe56415caa0 con 0x7fe5641567e0 2024-03-31T15:12:13.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:13.783+0000 7fe563e00640 1 -- 172.21.15.16:0/1655741930 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 214807623 0 0) 0x7fe554001680 con 0x7fe5641567e0 2024-03-31T15:12:13.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:13.783+0000 7fe563e00640 1 -- 172.21.15.16:0/1655741930 >> v1:172.21.15.16:6789/0 conn(0x7fe5641567e0 legacy=0x7fe564072c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:13.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:13.783+0000 7fe563e00640 1 --2- 172.21.15.16:0/1655741930 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe548003a10 0x7fe548003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:13.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:13.783+0000 7fe563e00640 1 --2- 172.21.15.16:0/1655741930 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fe548004810 0x7fe548004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:13.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:13.783+0000 7fe563e00640 1 --2- 172.21.15.16:0/1655741930 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe5480051d0 0x7fe54800b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:13.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:13.783+0000 7fe563e00640 1 -- 172.21.15.16:0/1655741930 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fe5640736b0 con 0x7fe5480051d0 2024-03-31T15:12:13.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:13.783+0000 7fe563e00640 1 -- 172.21.15.16:0/1655741930 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fe54800c0d0 con 0x7fe548003a10 2024-03-31T15:12:13.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:13.783+0000 7fe563e00640 1 -- 172.21.15.16:0/1655741930 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fe54800c2d0 con 0x7fe548004810 2024-03-31T15:12:13.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:13.783+0000 7fe569000640 1 --2- 172.21.15.16:0/1655741930 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe548003a10 0x7fe548003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:13.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:13.783+0000 7fe55be00640 1 --2- 172.21.15.16:0/1655741930 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fe548004810 0x7fe548004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:13.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:13.783+0000 7fe569a00640 1 --2- 172.21.15.16:0/1655741930 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe5480051d0 0x7fe54800b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:13.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:13.783+0000 7fe569000640 1 -- 172.21.15.16:0/1655741930 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fe548004810 msgr2=0x7fe548004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:12:13.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:13.783+0000 7fe569000640 1 --2- 172.21.15.16:0/1655741930 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fe548004810 0x7fe548004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:13.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:13.783+0000 7fe569000640 1 -- 172.21.15.16:0/1655741930 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe5480051d0 msgr2=0x7fe54800b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:12:13.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:13.783+0000 7fe569000640 1 --2- 172.21.15.16:0/1655741930 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe5480051d0 0x7fe54800b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:13.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:13.783+0000 7fe569000640 1 -- 172.21.15.16:0/1655741930 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe548003530 con 0x7fe548003a10 2024-03-31T15:12:13.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:13.783+0000 7fe569a00640 1 --2- 172.21.15.16:0/1655741930 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe5480051d0 0x7fe54800b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:12:13.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:13.783+0000 7fe569a00640 1 -- 172.21.15.16:0/1655741930 reap_dead start 2024-03-31T15:12:13.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:13.787+0000 7fe569000640 1 --2- 172.21.15.16:0/1655741930 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe548003a10 0x7fe548003e30 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7fe55400c110 tx=0x7fe55401cbc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=54475607dda6535c server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:12:13.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:13.787+0000 7fe563e00640 1 -- 172.21.15.16:0/1655741930 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe554007c50 con 0x7fe548003a10 2024-03-31T15:12:13.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:13.787+0000 7fe563e00640 1 -- 172.21.15.16:0/1655741930 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe5540a9070 con 0x7fe548003a10 2024-03-31T15:12:13.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:13.787+0000 7fe563e00640 1 -- 172.21.15.16:0/1655741930 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe554005d80 con 0x7fe548003a10 2024-03-31T15:12:13.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:13.787+0000 7fe56ba00640 1 -- 172.21.15.16:0/1655741930 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe548003a10 msgr2=0x7fe548003e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:13.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:13.787+0000 7fe56ba00640 1 --2- 172.21.15.16:0/1655741930 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe548003a10 0x7fe548003e30 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7fe55400c110 tx=0x7fe55401cbc0 comp rx=0 tx=0).stop 2024-03-31T15:12:13.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:13.787+0000 7fe56ba00640 1 -- 172.21.15.16:0/1655741930 shutdown_connections 2024-03-31T15:12:13.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:13.787+0000 7fe56ba00640 1 --2- 172.21.15.16:0/1655741930 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe548003a10 0x7fe548003e30 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:13.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:13.787+0000 7fe56ba00640 1 -- 172.21.15.16:0/1655741930 >> 172.21.15.16:0/1655741930 conn(0x7fe56406ee40 msgr2=0x7fe56406b620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:12:13.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:13.787+0000 7fe56ba00640 1 -- 172.21.15.16:0/1655741930 shutdown_connections 2024-03-31T15:12:13.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:13.787+0000 7fe56ba00640 1 -- 172.21.15.16:0/1655741930 wait complete. 2024-03-31T15:12:13.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:13.787+0000 7fe56ba00640 1 Processor -- start 2024-03-31T15:12:13.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:13.787+0000 7fe56ba00640 1 -- start start 2024-03-31T15:12:13.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:13.787+0000 7fe56ba00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe548003a10 0x7fe564140960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:13.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:13.791+0000 7fe56ba00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fe564151190 0x7fe56414c590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:13.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:13.791+0000 7fe56ba00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe56414cad0 0x7fe5641f2310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:13.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:13.791+0000 7fe56ba00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fe5640afa30 con 0x7fe56414cad0 2024-03-31T15:12:13.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:13.791+0000 7fe56ba00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fe564156bc0 con 0x7fe548003a10 2024-03-31T15:12:13.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:13.791+0000 7fe56ba00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fe5640afbb0 con 0x7fe564151190 2024-03-31T15:12:13.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:13.791+0000 7fe569a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe56414cad0 0x7fe5641f2310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:13.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:13.791+0000 7fe569a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe56414cad0 0x7fe5641f2310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37788/0 (socket says 172.21.15.16:37788) 2024-03-31T15:12:13.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:13.791+0000 7fe569a00640 1 -- 172.21.15.16:0/2402364489 learned_addr learned my addr 172.21.15.16:0/2402364489 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:12:13.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:13.791+0000 7fe55be00640 1 --2- 172.21.15.16:0/2402364489 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fe564151190 0x7fe56414c590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:13.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:13.791+0000 7fe569000640 1 --2- 172.21.15.16:0/2402364489 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe548003a10 0x7fe564140960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:13.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:13.791+0000 7fe569a00640 1 -- 172.21.15.16:0/2402364489 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fe564151190 msgr2=0x7fe56414c590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:13.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:13.791+0000 7fe569a00640 1 --2- 172.21.15.16:0/2402364489 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fe564151190 0x7fe56414c590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:13.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:13.791+0000 7fe569a00640 1 -- 172.21.15.16:0/2402364489 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe548003a10 msgr2=0x7fe564140960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:13.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:13.791+0000 7fe569a00640 1 --2- 172.21.15.16:0/2402364489 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe548003a10 0x7fe564140960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:13.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:13.791+0000 7fe569a00640 1 -- 172.21.15.16:0/2402364489 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe564140ea0 con 0x7fe56414cad0 2024-03-31T15:12:13.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:13.791+0000 7fe55be00640 1 --2- 172.21.15.16:0/2402364489 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fe564151190 0x7fe56414c590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:12:13.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:13.791+0000 7fe569000640 1 --2- 172.21.15.16:0/2402364489 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe548003a10 0x7fe564140960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:12:13.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:13.791+0000 7fe569a00640 1 --2- 172.21.15.16:0/2402364489 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe56414cad0 0x7fe5641f2310 secure :-1 s=READY pgs=1386 cs=0 l=1 rev1=1 crypto rx=0x7fe56406e7b0 tx=0x7fe55c001a10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:12:13.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:13.791+0000 7fe561600640 1 -- 172.21.15.16:0/2402364489 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe55c01a070 con 0x7fe56414cad0 2024-03-31T15:12:13.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:13.791+0000 7fe561600640 1 -- 172.21.15.16:0/2402364489 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe55c006d70 con 0x7fe56414cad0 2024-03-31T15:12:13.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:13.791+0000 7fe561600640 1 -- 172.21.15.16:0/2402364489 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe55c001a90 con 0x7fe56414cad0 2024-03-31T15:12:13.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:13.791+0000 7fe56ba00640 1 -- 172.21.15.16:0/2402364489 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe56414e0c0 con 0x7fe56414cad0 2024-03-31T15:12:13.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:13.791+0000 7fe56ba00640 1 -- 172.21.15.16:0/2402364489 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe56414e680 con 0x7fe56414cad0 2024-03-31T15:12:13.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:13.791+0000 7fe56ba00640 1 -- 172.21.15.16:0/2402364489 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe56414be90 con 0x7fe56414cad0 2024-03-31T15:12:13.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:13.791+0000 7fe561600640 1 -- 172.21.15.16:0/2402364489 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fe55c003bc0 con 0x7fe56414cad0 2024-03-31T15:12:13.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:13.795+0000 7fe561600640 1 --2- 172.21.15.16:0/2402364489 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fe5400683b0 0x7fe54006a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:13.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:13.795+0000 7fe561600640 1 -- 172.21.15.16:0/2402364489 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 4857+0+0 (secure 0 0 0) 0x7fe55c085e40 con 0x7fe56414cad0 2024-03-31T15:12:13.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:13.795+0000 7fe569000640 1 --2- 172.21.15.16:0/2402364489 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fe5400683b0 0x7fe54006a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:13.787 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:13.795+0000 7fe569000640 1 --2- 172.21.15.16:0/2402364489 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fe5400683b0 0x7fe54006a870 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7fe5540ab000 tx=0x7fe5540b0040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:12:13.790 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:13.799+0000 7fe561600640 1 -- 172.21.15.16:0/2402364489 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fe56414be90 con 0x7fe56414cad0 2024-03-31T15:12:13.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:13.963+0000 7fe56ba00640 1 -- 172.21.15.16:0/2402364489 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fe564069950 con 0x7fe56414cad0 2024-03-31T15:12:13.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:13.963+0000 7fe561600640 1 -- 172.21.15.16:0/2402364489 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v92) ==== 93+0+10 (secure 0 0 0) 0x7fe55c0566b0 con 0x7fe56414cad0 2024-03-31T15:12:13.957 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2024-03-31T15:12:13.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:13.967+0000 7fe56ba00640 1 -- 172.21.15.16:0/2402364489 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fe5400683b0 msgr2=0x7fe54006a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:13.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:13.967+0000 7fe56ba00640 1 --2- 172.21.15.16:0/2402364489 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fe5400683b0 0x7fe54006a870 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7fe5540ab000 tx=0x7fe5540b0040 comp rx=0 tx=0).stop 2024-03-31T15:12:13.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:13.967+0000 7fe56ba00640 1 -- 172.21.15.16:0/2402364489 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe56414cad0 msgr2=0x7fe5641f2310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:13.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:13.967+0000 7fe56ba00640 1 --2- 172.21.15.16:0/2402364489 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe56414cad0 0x7fe5641f2310 secure :-1 s=READY pgs=1386 cs=0 l=1 rev1=1 crypto rx=0x7fe56406e7b0 tx=0x7fe55c001a10 comp rx=0 tx=0).stop 2024-03-31T15:12:13.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:13.967+0000 7fe56ba00640 1 -- 172.21.15.16:0/2402364489 shutdown_connections 2024-03-31T15:12:13.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:13.967+0000 7fe56ba00640 1 --2- 172.21.15.16:0/2402364489 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fe5400683b0 0x7fe54006a870 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:13.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:13.967+0000 7fe56ba00640 1 --2- 172.21.15.16:0/2402364489 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe56414cad0 0x7fe5641f2310 unknown :-1 s=CLOSED pgs=1386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:13.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:13.967+0000 7fe56ba00640 1 --2- 172.21.15.16:0/2402364489 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fe564151190 0x7fe56414c590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:13.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:13.967+0000 7fe56ba00640 1 --2- 172.21.15.16:0/2402364489 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe548003a10 0x7fe564140960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:13.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:13.967+0000 7fe56ba00640 1 -- 172.21.15.16:0/2402364489 >> 172.21.15.16:0/2402364489 conn(0x7fe56406ee40 msgr2=0x7fe564154b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:12:13.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:13.967+0000 7fe56ba00640 1 -- 172.21.15.16:0/2402364489 shutdown_connections 2024-03-31T15:12:13.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:13.967+0000 7fe56ba00640 1 -- 172.21.15.16:0/2402364489 wait complete. 2024-03-31T15:12:13.978 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-31T15:12:14.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:14.127+0000 7ff021200640 1 Processor -- start 2024-03-31T15:12:14.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:14.127+0000 7ff021200640 1 -- start start 2024-03-31T15:12:14.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:14.127+0000 7ff021200640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff01c0b23d0 0x7ff01c071d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:14.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:14.127+0000 7ff021200640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff01c15eee0 0x7ff01c072250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:14.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:14.127+0000 7ff021200640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff01c15f3c0 con 0x7ff01c15b3d0 2024-03-31T15:12:14.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:14.127+0000 7ff021200640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7ff01c15b7b0 con 0x7ff01c0b23d0 2024-03-31T15:12:14.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:14.127+0000 7ff021200640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7ff01c15f6a0 con 0x7ff01c15eee0 2024-03-31T15:12:14.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:14.127+0000 7ff013e00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff01c0b23d0 0x7ff01c071d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:14.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:14.127+0000 7ff013e00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff01c0b23d0 0x7ff01c071d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37792/0 (socket says 172.21.15.16:37792) 2024-03-31T15:12:14.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:14.127+0000 7ff013e00640 1 -- 172.21.15.16:0/1435031567 learned_addr learned my addr 172.21.15.16:0/1435031567 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:12:14.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:14.127+0000 7ff01a800640 1 --2- 172.21.15.16:0/1435031567 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff01c15eee0 0x7ff01c072250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:14.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:14.127+0000 7ff013e00640 1 -- 172.21.15.16:0/1435031567 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff01c15eee0 msgr2=0x7ff01c072250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:12:14.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:14.127+0000 7ff013e00640 1 --2- 172.21.15.16:0/1435031567 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff01c15eee0 0x7ff01c072250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:14.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:14.127+0000 7ff013e00640 1 -- 172.21.15.16:0/1435031567 >> v1:172.21.15.89:6789/0 conn(0x7ff01c15b3d0 legacy=0x7ff01c0b1cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:14.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:14.127+0000 7ff013e00640 1 -- 172.21.15.16:0/1435031567 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff01c15f980 con 0x7ff01c0b23d0 2024-03-31T15:12:14.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:14.131+0000 7ff013e00640 1 --2- 172.21.15.16:0/1435031567 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff01c0b23d0 0x7ff01c071d10 secure :-1 s=READY pgs=1387 cs=0 l=1 rev1=1 crypto rx=0x7ff004009870 tx=0x7ff00401c530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70989486a021154 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:12:14.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:14.131+0000 7ff019400640 1 -- 172.21.15.16:0/1435031567 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0040aa650 con 0x7ff01c0b23d0 2024-03-31T15:12:14.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:14.131+0000 7ff019400640 1 -- 172.21.15.16:0/1435031567 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff0040af070 con 0x7ff01c0b23d0 2024-03-31T15:12:14.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:14.131+0000 7ff019400640 1 -- 172.21.15.16:0/1435031567 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0040aaa10 con 0x7ff01c0b23d0 2024-03-31T15:12:14.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:14.131+0000 7ff021200640 1 -- 172.21.15.16:0/1435031567 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff01c0b23d0 msgr2=0x7ff01c071d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:14.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:14.131+0000 7ff021200640 1 --2- 172.21.15.16:0/1435031567 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff01c0b23d0 0x7ff01c071d10 secure :-1 s=READY pgs=1387 cs=0 l=1 rev1=1 crypto rx=0x7ff004009870 tx=0x7ff00401c530 comp rx=0 tx=0).stop 2024-03-31T15:12:14.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:14.131+0000 7ff021200640 1 -- 172.21.15.16:0/1435031567 shutdown_connections 2024-03-31T15:12:14.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:14.131+0000 7ff021200640 1 --2- 172.21.15.16:0/1435031567 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff01c15eee0 0x7ff01c072250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:14.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:14.131+0000 7ff021200640 1 --2- 172.21.15.16:0/1435031567 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff01c0b23d0 0x7ff01c071d10 unknown :-1 s=CLOSED pgs=1387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:14.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:14.131+0000 7ff021200640 1 -- 172.21.15.16:0/1435031567 >> 172.21.15.16:0/1435031567 conn(0x7ff01c06b620 msgr2=0x7ff01c06ba30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:12:14.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:14.131+0000 7ff021200640 1 -- 172.21.15.16:0/1435031567 shutdown_connections 2024-03-31T15:12:14.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:14.131+0000 7ff021200640 1 -- 172.21.15.16:0/1435031567 wait complete. 2024-03-31T15:12:14.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:14.135+0000 7ff021200640 1 Processor -- start 2024-03-31T15:12:14.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:14.135+0000 7ff021200640 1 -- start start 2024-03-31T15:12:14.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:14.135+0000 7ff021200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff01c0b23d0 0x7ff01c15c8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:14.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:14.135+0000 7ff021200640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff01c15b3d0 0x7ff01c1521c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:14.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:14.135+0000 7ff021200640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff01c15eee0 0x7ff01c1419d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:14.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:14.135+0000 7ff021200640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7ff01c072790 con 0x7ff01c0b23d0 2024-03-31T15:12:14.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:14.135+0000 7ff021200640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7ff01c0737d0 con 0x7ff01c15b3d0 2024-03-31T15:12:14.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:14.135+0000 7ff021200640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7ff01c072910 con 0x7ff01c15eee0 2024-03-31T15:12:14.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:14.135+0000 7ff019e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff01c0b23d0 0x7ff01c15c8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:14.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:14.135+0000 7ff013e00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff01c15b3d0 0x7ff01c1521c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:14.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:14.135+0000 7ff019e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff01c0b23d0 0x7ff01c15c8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37798/0 (socket says 172.21.15.16:37798) 2024-03-31T15:12:14.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:14.135+0000 7ff019e00640 1 -- 172.21.15.16:0/195646292 learned_addr learned my addr 172.21.15.16:0/195646292 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:12:14.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:14.135+0000 7ff01a800640 1 --2- 172.21.15.16:0/195646292 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff01c15eee0 0x7ff01c1419d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:14.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:14.135+0000 7ff019e00640 1 -- 172.21.15.16:0/195646292 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff01c15eee0 msgr2=0x7ff01c1419d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:14.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:14.135+0000 7ff019e00640 1 --2- 172.21.15.16:0/195646292 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff01c15eee0 0x7ff01c1419d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:14.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:14.135+0000 7ff019e00640 1 -- 172.21.15.16:0/195646292 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff01c15b3d0 msgr2=0x7ff01c1521c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:14.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:14.135+0000 7ff019e00640 1 --2- 172.21.15.16:0/195646292 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff01c15b3d0 0x7ff01c1521c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:14.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:14.135+0000 7ff019e00640 1 -- 172.21.15.16:0/195646292 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff01c141f10 con 0x7ff01c0b23d0 2024-03-31T15:12:14.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:14.135+0000 7ff019e00640 1 --2- 172.21.15.16:0/195646292 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff01c0b23d0 0x7ff01c15c8a0 secure :-1 s=READY pgs=1388 cs=0 l=1 rev1=1 crypto rx=0x7ff00c00c500 tx=0x7ff00c00c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:12:14.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:14.135+0000 7ff012000640 1 -- 172.21.15.16:0/195646292 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff00c016020 con 0x7ff01c0b23d0 2024-03-31T15:12:14.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:14.135+0000 7ff012000640 1 -- 172.21.15.16:0/195646292 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff00c005220 con 0x7ff01c0b23d0 2024-03-31T15:12:14.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:14.135+0000 7ff012000640 1 -- 172.21.15.16:0/195646292 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff00c007030 con 0x7ff01c0b23d0 2024-03-31T15:12:14.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:14.135+0000 7ff021200640 1 -- 172.21.15.16:0/195646292 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff01c15bae0 con 0x7ff01c0b23d0 2024-03-31T15:12:14.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:14.135+0000 7ff021200640 1 -- 172.21.15.16:0/195646292 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff01c15c0a0 con 0x7ff01c0b23d0 2024-03-31T15:12:14.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:14.135+0000 7ff021200640 1 -- 172.21.15.16:0/195646292 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff01c069950 con 0x7ff01c0b23d0 2024-03-31T15:12:14.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:14.139+0000 7ff012000640 1 -- 172.21.15.16:0/195646292 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7ff00c0071f0 con 0x7ff01c0b23d0 2024-03-31T15:12:14.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:14.139+0000 7ff012000640 1 --2- 172.21.15.16:0/195646292 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7feff8068380 0x7feff806a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:14.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:14.139+0000 7ff012000640 1 -- 172.21.15.16:0/195646292 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 4857+0+0 (secure 0 0 0) 0x7ff00c085f80 con 0x7ff01c0b23d0 2024-03-31T15:12:14.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:14.139+0000 7ff013e00640 1 --2- 172.21.15.16:0/195646292 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7feff8068380 0x7feff806a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:14.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:14.139+0000 7ff013e00640 1 --2- 172.21.15.16:0/195646292 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7feff8068380 0x7feff806a840 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7ff0040ae000 tx=0x7ff0040a7000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:12:14.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:14.143+0000 7ff012000640 1 -- 172.21.15.16:0/195646292 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7ff00c020020 con 0x7ff01c0b23d0 2024-03-31T15:12:14.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:14.307+0000 7ff021200640 1 -- 172.21.15.16:0/195646292 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7ff01c0cdea0 con 0x7ff01c0b23d0 2024-03-31T15:12:14.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:14.307+0000 7ff012000640 1 -- 172.21.15.16:0/195646292 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v92) ==== 84+0+65 (secure 0 0 0) 0x7ff00c0561b0 con 0x7ff01c0b23d0 2024-03-31T15:12:14.301 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:12:14.301 INFO:teuthology.orchestra.run.smithi016.stdout:[ 2024-03-31T15:12:14.301 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-31T15:12:14.301 INFO:teuthology.orchestra.run.smithi016.stdout: "poolnum": 1, 2024-03-31T15:12:14.301 INFO:teuthology.orchestra.run.smithi016.stdout: "poolname": ".mgr" 2024-03-31T15:12:14.301 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-31T15:12:14.302 INFO:teuthology.orchestra.run.smithi016.stdout:] 2024-03-31T15:12:14.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:14.311+0000 7ff021200640 1 -- 172.21.15.16:0/195646292 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7feff8068380 msgr2=0x7feff806a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:14.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:14.311+0000 7ff021200640 1 --2- 172.21.15.16:0/195646292 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7feff8068380 0x7feff806a840 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7ff0040ae000 tx=0x7ff0040a7000 comp rx=0 tx=0).stop 2024-03-31T15:12:14.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:14.311+0000 7ff021200640 1 -- 172.21.15.16:0/195646292 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff01c0b23d0 msgr2=0x7ff01c15c8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:14.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:14.311+0000 7ff021200640 1 --2- 172.21.15.16:0/195646292 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff01c0b23d0 0x7ff01c15c8a0 secure :-1 s=READY pgs=1388 cs=0 l=1 rev1=1 crypto rx=0x7ff00c00c500 tx=0x7ff00c00c9c0 comp rx=0 tx=0).stop 2024-03-31T15:12:14.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:14.311+0000 7ff021200640 1 -- 172.21.15.16:0/195646292 shutdown_connections 2024-03-31T15:12:14.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:14.311+0000 7ff021200640 1 --2- 172.21.15.16:0/195646292 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7feff8068380 0x7feff806a840 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:14.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:14.311+0000 7ff021200640 1 --2- 172.21.15.16:0/195646292 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff01c15eee0 0x7ff01c1419d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:14.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:14.311+0000 7ff021200640 1 --2- 172.21.15.16:0/195646292 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff01c15b3d0 0x7ff01c1521c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:14.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:14.311+0000 7ff021200640 1 --2- 172.21.15.16:0/195646292 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff01c0b23d0 0x7ff01c15c8a0 unknown :-1 s=CLOSED pgs=1388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:14.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:14.311+0000 7ff021200640 1 -- 172.21.15.16:0/195646292 >> 172.21.15.16:0/195646292 conn(0x7ff01c06b620 msgr2=0x7ff01c0ce7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:12:14.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:14.311+0000 7ff021200640 1 -- 172.21.15.16:0/195646292 shutdown_connections 2024-03-31T15:12:14.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:14.311+0000 7ff021200640 1 -- 172.21.15.16:0/195646292 wait complete. 2024-03-31T15:12:14.322 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-31T15:12:14.322 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-31T15:12:14.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:14.471+0000 7fe54fa00640 1 Processor -- start 2024-03-31T15:12:14.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:14.475+0000 7fe54fa00640 1 -- start start 2024-03-31T15:12:14.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:14.475+0000 7fe54fa00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fe548143f40 0x7fe5480730c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:14.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:14.475+0000 7fe54fa00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fe5480b22f0 0x7fe5480b2f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:14.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:14.475+0000 7fe54fa00640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe548144320 con 0x7fe5480b45d0 2024-03-31T15:12:14.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:14.475+0000 7fe54fa00640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7fe5480b4a30 con 0x7fe5480b22f0 2024-03-31T15:12:14.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:14.475+0000 7fe54fa00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fe54815ef80 con 0x7fe548143f40 2024-03-31T15:12:14.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:14.475+0000 7fe54da00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fe5480b22f0 0x7fe5480b2f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:14.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:14.475+0000 7fe53fe00640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fe5480b45d0 0x7fe5480b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:41326/0 (socket says 172.21.15.16:41326) 2024-03-31T15:12:14.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:14.475+0000 7fe53fe00640 1 -- 172.21.15.16:0/328284241 learned_addr learned my addr 172.21.15.16:0/328284241 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:12:14.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:14.475+0000 7fe54d000640 1 --2- 172.21.15.16:0/328284241 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fe548143f40 0x7fe5480730c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:14.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:14.475+0000 7fe54da00640 1 -- 172.21.15.16:0/328284241 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fe548143f40 msgr2=0x7fe5480730c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:12:14.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:14.475+0000 7fe54da00640 1 --2- 172.21.15.16:0/328284241 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fe548143f40 0x7fe5480730c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:14.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:14.475+0000 7fe54da00640 1 -- 172.21.15.16:0/328284241 >> v1:172.21.15.16:6789/0 conn(0x7fe5480b45d0 legacy=0x7fe5480b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:14.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:14.475+0000 7fe54da00640 1 -- 172.21.15.16:0/328284241 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe54815f260 con 0x7fe5480b22f0 2024-03-31T15:12:14.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:14.475+0000 7fe547e00640 1 -- 172.21.15.16:0/328284241 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4090195460 0 0) 0x7fe548144320 con 0x7fe5480b45d0 2024-03-31T15:12:14.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:14.479+0000 7fe54da00640 1 --2- 172.21.15.16:0/328284241 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe5480b22f0 0x7fe5480b2f50 secure :-1 s=READY pgs=1390 cs=0 l=1 rev1=1 crypto rx=0x7fe540009d90 tx=0x7fe54001c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c2800fd5c12c2c8 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:12:14.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:14.479+0000 7fe547e00640 1 -- 172.21.15.16:0/328284241 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5400b2070 con 0x7fe5480b22f0 2024-03-31T15:12:14.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:14.479+0000 7fe547e00640 1 -- 172.21.15.16:0/328284241 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe5400a5830 con 0x7fe5480b22f0 2024-03-31T15:12:14.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:14.479+0000 7fe547e00640 1 -- 172.21.15.16:0/328284241 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5400a5c00 con 0x7fe5480b22f0 2024-03-31T15:12:14.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:14.479+0000 7fe54fa00640 1 -- 172.21.15.16:0/328284241 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe5480b22f0 msgr2=0x7fe5480b2f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:14.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:14.479+0000 7fe54fa00640 1 --2- 172.21.15.16:0/328284241 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe5480b22f0 0x7fe5480b2f50 secure :-1 s=READY pgs=1390 cs=0 l=1 rev1=1 crypto rx=0x7fe540009d90 tx=0x7fe54001c970 comp rx=0 tx=0).stop 2024-03-31T15:12:14.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:14.479+0000 7fe54fa00640 1 -- 172.21.15.16:0/328284241 shutdown_connections 2024-03-31T15:12:14.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:14.479+0000 7fe54fa00640 1 --2- 172.21.15.16:0/328284241 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe5480b22f0 0x7fe5480b2f50 unknown :-1 s=CLOSED pgs=1390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:14.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:14.479+0000 7fe54fa00640 1 --2- 172.21.15.16:0/328284241 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fe548143f40 0x7fe5480730c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:14.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:14.479+0000 7fe54fa00640 1 -- 172.21.15.16:0/328284241 >> 172.21.15.16:0/328284241 conn(0x7fe54806b620 msgr2=0x7fe54806ba30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:12:14.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:14.479+0000 7fe54fa00640 1 -- 172.21.15.16:0/328284241 shutdown_connections 2024-03-31T15:12:14.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:14.479+0000 7fe54fa00640 1 -- 172.21.15.16:0/328284241 wait complete. 2024-03-31T15:12:14.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:14.479+0000 7fe54fa00640 1 Processor -- start 2024-03-31T15:12:14.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:14.479+0000 7fe54fa00640 1 -- start start 2024-03-31T15:12:14.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:14.483+0000 7fe54fa00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe5480b22f0 0x7fe54813f730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:14.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:14.483+0000 7fe54fa00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe5480b45d0 0x7fe54813d8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:14.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:14.483+0000 7fe54fa00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fe548143f40 0x7fe5480b0ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:14.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:14.483+0000 7fe54fa00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fe5481680b0 con 0x7fe5480b22f0 2024-03-31T15:12:14.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:14.483+0000 7fe54fa00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fe5480cd9a0 con 0x7fe5480b45d0 2024-03-31T15:12:14.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:14.483+0000 7fe54fa00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fe548168230 con 0x7fe548143f40 2024-03-31T15:12:14.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:14.483+0000 7fe54d000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe5480b22f0 0x7fe54813f730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:14.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:14.483+0000 7fe54d000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe5480b22f0 0x7fe54813f730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37822/0 (socket says 172.21.15.16:37822) 2024-03-31T15:12:14.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:14.483+0000 7fe54d000640 1 -- 172.21.15.16:0/4014694384 learned_addr learned my addr 172.21.15.16:0/4014694384 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:12:14.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:14.483+0000 7fe53fe00640 1 --2- 172.21.15.16:0/4014694384 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe5480b45d0 0x7fe54813d8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:14.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:14.483+0000 7fe54da00640 1 --2- 172.21.15.16:0/4014694384 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fe548143f40 0x7fe5480b0ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:14.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:14.483+0000 7fe54d000640 1 -- 172.21.15.16:0/4014694384 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fe548143f40 msgr2=0x7fe5480b0ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:14.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:14.483+0000 7fe54d000640 1 --2- 172.21.15.16:0/4014694384 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fe548143f40 0x7fe5480b0ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:14.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:14.483+0000 7fe54d000640 1 -- 172.21.15.16:0/4014694384 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe5480b45d0 msgr2=0x7fe54813d8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:14.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:14.483+0000 7fe54d000640 1 --2- 172.21.15.16:0/4014694384 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe5480b45d0 0x7fe54813d8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:14.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:14.483+0000 7fe54d000640 1 -- 172.21.15.16:0/4014694384 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe5480afcf0 con 0x7fe5480b22f0 2024-03-31T15:12:14.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:14.483+0000 7fe53fe00640 1 --2- 172.21.15.16:0/4014694384 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe5480b45d0 0x7fe54813d8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:12:14.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:14.483+0000 7fe54d000640 1 --2- 172.21.15.16:0/4014694384 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe5480b22f0 0x7fe54813f730 secure :-1 s=READY pgs=1391 cs=0 l=1 rev1=1 crypto rx=0x7fe53800d800 tx=0x7fe53800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:12:14.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:14.483+0000 7fe545600640 1 -- 172.21.15.16:0/4014694384 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe53800b6c0 con 0x7fe5480b22f0 2024-03-31T15:12:14.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:14.483+0000 7fe545600640 1 -- 172.21.15.16:0/4014694384 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe538019070 con 0x7fe5480b22f0 2024-03-31T15:12:14.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:14.483+0000 7fe545600640 1 -- 172.21.15.16:0/4014694384 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe53800ba60 con 0x7fe5480b22f0 2024-03-31T15:12:14.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:14.483+0000 7fe54fa00640 1 -- 172.21.15.16:0/4014694384 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe5480affd0 con 0x7fe5480b22f0 2024-03-31T15:12:14.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:14.483+0000 7fe54fa00640 1 -- 172.21.15.16:0/4014694384 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe5480b0590 con 0x7fe5480b22f0 2024-03-31T15:12:14.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:14.483+0000 7fe54fa00640 1 -- 172.21.15.16:0/4014694384 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe548069950 con 0x7fe5480b22f0 2024-03-31T15:12:14.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:14.487+0000 7fe545600640 1 -- 172.21.15.16:0/4014694384 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fe53800bc00 con 0x7fe5480b22f0 2024-03-31T15:12:14.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:14.487+0000 7fe545600640 1 --2- 172.21.15.16:0/4014694384 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fe5240683b0 0x7fe52406a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:14.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:14.487+0000 7fe545600640 1 -- 172.21.15.16:0/4014694384 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 4857+0+0 (secure 0 0 0) 0x7fe538086b70 con 0x7fe5480b22f0 2024-03-31T15:12:14.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:14.487+0000 7fe53fe00640 1 --2- 172.21.15.16:0/4014694384 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fe5240683b0 0x7fe52406a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:14.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:14.487+0000 7fe53fe00640 1 --2- 172.21.15.16:0/4014694384 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fe5240683b0 0x7fe52406a870 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7fe5480b1360 tx=0x7fe530007cb0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:12:14.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:14.491+0000 7fe545600640 1 -- 172.21.15.16:0/4014694384 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fe538057470 con 0x7fe5480b22f0 2024-03-31T15:12:14.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:14.659+0000 7fe54fa00640 1 -- 172.21.15.16:0/4014694384 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7fe5480728f0 con 0x7fe5480b22f0 2024-03-31T15:12:14.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:14.871+0000 7fe545600640 1 -- 172.21.15.16:0/4014694384 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v93) ==== 138+0+0 (secure 0 0 0) 0x7fe538056d70 con 0x7fe5480b22f0 2024-03-31T15:12:14.863 INFO:teuthology.orchestra.run.smithi016.stderr:pool 'cephfs_metadata' created 2024-03-31T15:12:14.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:14.871+0000 7fe54fa00640 1 -- 172.21.15.16:0/4014694384 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fe5240683b0 msgr2=0x7fe52406a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:14.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:14.871+0000 7fe54fa00640 1 --2- 172.21.15.16:0/4014694384 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fe5240683b0 0x7fe52406a870 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7fe5480b1360 tx=0x7fe530007cb0 comp rx=0 tx=0).stop 2024-03-31T15:12:14.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:14.871+0000 7fe54fa00640 1 -- 172.21.15.16:0/4014694384 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe5480b22f0 msgr2=0x7fe54813f730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:14.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:14.871+0000 7fe54fa00640 1 --2- 172.21.15.16:0/4014694384 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe5480b22f0 0x7fe54813f730 secure :-1 s=READY pgs=1391 cs=0 l=1 rev1=1 crypto rx=0x7fe53800d800 tx=0x7fe53800dcc0 comp rx=0 tx=0).stop 2024-03-31T15:12:14.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:14.871+0000 7fe54fa00640 1 -- 172.21.15.16:0/4014694384 shutdown_connections 2024-03-31T15:12:14.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:14.871+0000 7fe54fa00640 1 --2- 172.21.15.16:0/4014694384 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fe5240683b0 0x7fe52406a870 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:14.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:14.871+0000 7fe54fa00640 1 --2- 172.21.15.16:0/4014694384 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fe548143f40 0x7fe5480b0ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:14.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:14.871+0000 7fe54fa00640 1 --2- 172.21.15.16:0/4014694384 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe5480b45d0 0x7fe54813d8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:14.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:14.871+0000 7fe54fa00640 1 --2- 172.21.15.16:0/4014694384 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe5480b22f0 0x7fe54813f730 unknown :-1 s=CLOSED pgs=1391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:14.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:14.871+0000 7fe54fa00640 1 -- 172.21.15.16:0/4014694384 >> 172.21.15.16:0/4014694384 conn(0x7fe54806b620 msgr2=0x7fe5480cc140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:12:14.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:14.871+0000 7fe54fa00640 1 -- 172.21.15.16:0/4014694384 shutdown_connections 2024-03-31T15:12:14.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:14.871+0000 7fe54fa00640 1 -- 172.21.15.16:0/4014694384 wait complete. 2024-03-31T15:12:14.891 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-31T15:12:15.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:15.047+0000 7f73c3200640 1 Processor -- start 2024-03-31T15:12:15.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:15.047+0000 7f73c3200640 1 -- start start 2024-03-31T15:12:15.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:15.047+0000 7f73c3200640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f73bc13dff0 0x7f73bc06d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:15.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:15.047+0000 7f73c3200640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f73bc15fbc0 con 0x7f73bc13e3d0 2024-03-31T15:12:15.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:15.047+0000 7f73c3200640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f73bc15ee00 con 0x7f73bc0cd0e0 2024-03-31T15:12:15.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:15.047+0000 7f73c3200640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f73bc15f730 con 0x7f73bc13dff0 2024-03-31T15:12:15.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:15.047+0000 7f73bbe00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f73bc13dff0 0x7f73bc06d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:15.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:15.047+0000 7f73b3e00640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f73bc13e3d0 0x7f73bc0cc9d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:41334/0 (socket says 172.21.15.16:41334) 2024-03-31T15:12:15.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:15.047+0000 7f73b3e00640 1 -- 172.21.15.16:0/1566649467 learned_addr learned my addr 172.21.15.16:0/1566649467 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:12:15.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:15.047+0000 7f73bbe00640 1 -- 172.21.15.16:0/1566649467 >> v1:172.21.15.89:6789/0 conn(0x7f73bc0cd0e0 legacy=0x7f73bc0ceb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:15.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:15.047+0000 7f73bbe00640 1 -- 172.21.15.16:0/1566649467 >> v1:172.21.15.16:6789/0 conn(0x7f73bc13e3d0 legacy=0x7f73bc0cc9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:15.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:15.047+0000 7f73bbe00640 1 -- 172.21.15.16:0/1566649467 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73bc15f0e0 con 0x7f73bc13dff0 2024-03-31T15:12:15.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:15.047+0000 7f73bb400640 1 -- 172.21.15.16:0/1566649467 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2555417067 0 0) 0x7f73bc15fbc0 con 0x7f73bc13e3d0 2024-03-31T15:12:15.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:15.051+0000 7f73bbe00640 1 --2- 172.21.15.16:0/1566649467 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f73bc13dff0 0x7f73bc06d4d0 secure :-1 s=READY pgs=1393 cs=0 l=1 rev1=1 crypto rx=0x7f73ac0097c0 tx=0x7f73ac01c410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c39a510a823d5b5 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:12:15.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:15.051+0000 7f73bb400640 1 -- 172.21.15.16:0/1566649467 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73ac0b0070 con 0x7f73bc13dff0 2024-03-31T15:12:15.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:15.051+0000 7f73bb400640 1 -- 172.21.15.16:0/1566649467 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f73ac0ab600 con 0x7f73bc13dff0 2024-03-31T15:12:15.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:15.051+0000 7f73bb400640 1 -- 172.21.15.16:0/1566649467 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73ac0ab9d0 con 0x7f73bc13dff0 2024-03-31T15:12:15.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:15.051+0000 7f73c3200640 1 -- 172.21.15.16:0/1566649467 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f73bc13dff0 msgr2=0x7f73bc06d4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:15.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:15.051+0000 7f73c3200640 1 --2- 172.21.15.16:0/1566649467 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f73bc13dff0 0x7f73bc06d4d0 secure :-1 s=READY pgs=1393 cs=0 l=1 rev1=1 crypto rx=0x7f73ac0097c0 tx=0x7f73ac01c410 comp rx=0 tx=0).stop 2024-03-31T15:12:15.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:15.051+0000 7f73c3200640 1 -- 172.21.15.16:0/1566649467 shutdown_connections 2024-03-31T15:12:15.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:15.051+0000 7f73c3200640 1 --2- 172.21.15.16:0/1566649467 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f73bc13dff0 0x7f73bc06d4d0 unknown :-1 s=CLOSED pgs=1393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:15.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:15.051+0000 7f73c3200640 1 -- 172.21.15.16:0/1566649467 >> 172.21.15.16:0/1566649467 conn(0x7f73bc06ee40 msgr2=0x7f73bc06b620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:12:15.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:15.051+0000 7f73c3200640 1 -- 172.21.15.16:0/1566649467 shutdown_connections 2024-03-31T15:12:15.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:15.051+0000 7f73c3200640 1 -- 172.21.15.16:0/1566649467 wait complete. 2024-03-31T15:12:15.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:15.055+0000 7f73c3200640 1 Processor -- start 2024-03-31T15:12:15.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:15.055+0000 7f73c3200640 1 -- start start 2024-03-31T15:12:15.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:15.055+0000 7f73c3200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f73bc0cd0e0 0x7f73bc13c170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:15.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:15.055+0000 7f73c3200640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f73bc13dff0 0x7f73bc15d9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:15.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:15.055+0000 7f73c3200640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f73bc13e3d0 0x7f73bc15a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:15.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:15.055+0000 7f73c3200640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f73bc169080 con 0x7f73bc0cd0e0 2024-03-31T15:12:15.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:15.055+0000 7f73c3200640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f73bc0cfca0 con 0x7f73bc13e3d0 2024-03-31T15:12:15.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:15.055+0000 7f73c3200640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f73bc169200 con 0x7f73bc13dff0 2024-03-31T15:12:15.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:15.055+0000 7f73bbe00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f73bc0cd0e0 0x7f73bc13c170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:15.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:15.055+0000 7f73bbe00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f73bc0cd0e0 0x7f73bc13c170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37838/0 (socket says 172.21.15.16:37838) 2024-03-31T15:12:15.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:15.055+0000 7f73bbe00640 1 -- 172.21.15.16:0/3796754165 learned_addr learned my addr 172.21.15.16:0/3796754165 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:12:15.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:15.055+0000 7f73b3e00640 1 --2- 172.21.15.16:0/3796754165 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f73bc13dff0 0x7f73bc15d9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:15.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:15.055+0000 7f73c1200640 1 --2- 172.21.15.16:0/3796754165 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f73bc13e3d0 0x7f73bc15a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:15.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:15.055+0000 7f73bbe00640 1 -- 172.21.15.16:0/3796754165 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f73bc13dff0 msgr2=0x7f73bc15d9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:15.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:15.055+0000 7f73bbe00640 1 --2- 172.21.15.16:0/3796754165 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f73bc13dff0 0x7f73bc15d9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:15.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:15.055+0000 7f73bbe00640 1 -- 172.21.15.16:0/3796754165 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f73bc13e3d0 msgr2=0x7f73bc15a480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:15.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:15.055+0000 7f73bbe00640 1 --2- 172.21.15.16:0/3796754165 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f73bc13e3d0 0x7f73bc15a480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:15.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:15.055+0000 7f73bbe00640 1 -- 172.21.15.16:0/3796754165 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73bc154d30 con 0x7f73bc0cd0e0 2024-03-31T15:12:15.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:15.055+0000 7f73bbe00640 1 --2- 172.21.15.16:0/3796754165 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f73bc0cd0e0 0x7f73bc13c170 secure :-1 s=READY pgs=1394 cs=0 l=1 rev1=1 crypto rx=0x7f73ac0ae040 tx=0x7f73ac01ac50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:12:15.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:15.055+0000 7f73b8c00640 1 -- 172.21.15.16:0/3796754165 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73ac0b0050 con 0x7f73bc0cd0e0 2024-03-31T15:12:15.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:15.055+0000 7f73b8c00640 1 -- 172.21.15.16:0/3796754165 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f73ac0bc070 con 0x7f73bc0cd0e0 2024-03-31T15:12:15.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:15.055+0000 7f73c3200640 1 -- 172.21.15.16:0/3796754165 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f73bc155010 con 0x7f73bc0cd0e0 2024-03-31T15:12:15.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:15.055+0000 7f73b8c00640 1 -- 172.21.15.16:0/3796754165 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73ac0aac40 con 0x7f73bc0cd0e0 2024-03-31T15:12:15.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:15.055+0000 7f73c3200640 1 -- 172.21.15.16:0/3796754165 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f73bc1555d0 con 0x7f73bc0cd0e0 2024-03-31T15:12:15.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:15.055+0000 7f73c3200640 1 -- 172.21.15.16:0/3796754165 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f73bc069950 con 0x7f73bc0cd0e0 2024-03-31T15:12:15.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:15.059+0000 7f73b8c00640 1 -- 172.21.15.16:0/3796754165 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f73ac0c1460 con 0x7f73bc0cd0e0 2024-03-31T15:12:15.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:15.059+0000 7f73b8c00640 1 --2- 172.21.15.16:0/3796754165 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f7398068380 0x7f739806a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:15.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:15.059+0000 7f73b8c00640 1 -- 172.21.15.16:0/3796754165 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 5227+0+0 (secure 0 0 0) 0x7f73ac1211f0 con 0x7f73bc0cd0e0 2024-03-31T15:12:15.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:15.059+0000 7f73b3e00640 1 --2- 172.21.15.16:0/3796754165 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f7398068380 0x7f739806a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:15.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:15.059+0000 7f73b3e00640 1 --2- 172.21.15.16:0/3796754165 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f7398068380 0x7f739806a840 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f73bc0cd020 tx=0x7f73a4007240 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:12:15.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:15.063+0000 7f73b8c00640 1 -- 172.21.15.16:0/3796754165 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f73ac016090 con 0x7f73bc0cd0e0 2024-03-31T15:12:15.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:15.227+0000 7f73c3200640 1 -- 172.21.15.16:0/3796754165 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7f73bc06e4e0 con 0x7f73bc0cd0e0 2024-03-31T15:12:15.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:15.879+0000 7f73b8c00640 1 -- 172.21.15.16:0/3796754165 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v94) ==== 170+0+0 (secure 0 0 0) 0x7f73ac0f1340 con 0x7f73bc0cd0e0 2024-03-31T15:12:15.874 INFO:teuthology.orchestra.run.smithi016.stderr:pool 'cephfs_data' created 2024-03-31T15:12:15.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:15.883+0000 7f73c3200640 1 -- 172.21.15.16:0/3796754165 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f7398068380 msgr2=0x7f739806a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:15.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:15.883+0000 7f73c3200640 1 --2- 172.21.15.16:0/3796754165 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f7398068380 0x7f739806a840 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f73bc0cd020 tx=0x7f73a4007240 comp rx=0 tx=0).stop 2024-03-31T15:12:15.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:15.883+0000 7f73c3200640 1 -- 172.21.15.16:0/3796754165 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f73bc0cd0e0 msgr2=0x7f73bc13c170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:15.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:15.883+0000 7f73c3200640 1 --2- 172.21.15.16:0/3796754165 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f73bc0cd0e0 0x7f73bc13c170 secure :-1 s=READY pgs=1394 cs=0 l=1 rev1=1 crypto rx=0x7f73ac0ae040 tx=0x7f73ac01ac50 comp rx=0 tx=0).stop 2024-03-31T15:12:15.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:15.887+0000 7f73c3200640 1 -- 172.21.15.16:0/3796754165 shutdown_connections 2024-03-31T15:12:15.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:15.887+0000 7f73c3200640 1 --2- 172.21.15.16:0/3796754165 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f7398068380 0x7f739806a840 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:15.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:15.887+0000 7f73c3200640 1 --2- 172.21.15.16:0/3796754165 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f73bc13e3d0 0x7f73bc15a480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:15.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:15.887+0000 7f73c3200640 1 --2- 172.21.15.16:0/3796754165 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f73bc13dff0 0x7f73bc15d9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:15.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:15.887+0000 7f73c3200640 1 --2- 172.21.15.16:0/3796754165 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f73bc0cd0e0 0x7f73bc13c170 unknown :-1 s=CLOSED pgs=1394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:15.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:15.887+0000 7f73c3200640 1 -- 172.21.15.16:0/3796754165 >> 172.21.15.16:0/3796754165 conn(0x7f73bc06ee40 msgr2=0x7f73bc15be60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:12:15.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:15.887+0000 7f73c3200640 1 -- 172.21.15.16:0/3796754165 shutdown_connections 2024-03-31T15:12:15.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:15.891+0000 7f73c3200640 1 -- 172.21.15.16:0/3796754165 wait complete. 2024-03-31T15:12:15.919 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-31T15:12:16.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:16.075+0000 7f9fa6600640 1 Processor -- start 2024-03-31T15:12:16.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:16.075+0000 7f9fa6600640 1 -- start start 2024-03-31T15:12:16.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:16.075+0000 7f9fa6600640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f9fa013e3d0 0x7f9fa00cc9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:16.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:16.075+0000 7f9fa6600640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f9fa006dbe0 0x7f9fa00ccf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:16.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:16.075+0000 7f9fa6600640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9fa015fbc0 con 0x7f9fa013dff0 2024-03-31T15:12:16.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:16.075+0000 7f9fa6600640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f9fa00cd450 con 0x7f9fa013e3d0 2024-03-31T15:12:16.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:16.075+0000 7f9fa6600640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f9fa015f730 con 0x7f9fa006dbe0 2024-03-31T15:12:16.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:16.075+0000 7f9f9ea00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f9fa013e3d0 0x7f9fa00cc9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:16.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:16.075+0000 7f9f9ea00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f9fa013e3d0 0x7f9fa00cc9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37850/0 (socket says 172.21.15.16:37850) 2024-03-31T15:12:16.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:16.075+0000 7f9f9ea00640 1 -- 172.21.15.16:0/3519161524 learned_addr learned my addr 172.21.15.16:0/3519161524 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:12:16.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:16.075+0000 7f9f9fe00640 1 --2- 172.21.15.16:0/3519161524 >> v2:172.21.15.89:3300/0 conn(0x7f9fa006dbe0 0x7f9fa00ccf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:16.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:16.075+0000 7f9f9ea00640 1 -- 172.21.15.16:0/3519161524 >> v2:172.21.15.89:3300/0 conn(0x7f9fa006dbe0 msgr2=0x7f9fa00ccf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:12:16.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:16.075+0000 7f9f9ea00640 1 --2- 172.21.15.16:0/3519161524 >> v2:172.21.15.89:3300/0 conn(0x7f9fa006dbe0 0x7f9fa00ccf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:16.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:16.075+0000 7f9f9ea00640 1 -- 172.21.15.16:0/3519161524 >> v1:172.21.15.89:6789/0 conn(0x7f9fa013dff0 legacy=0x7f9fa006d4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:16.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:16.075+0000 7f9f9ea00640 1 -- 172.21.15.16:0/3519161524 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9fa015ee00 con 0x7f9fa013e3d0 2024-03-31T15:12:16.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:16.079+0000 7f9f9ea00640 1 --2- 172.21.15.16:0/3519161524 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9fa013e3d0 0x7f9fa00cc9d0 secure :-1 s=READY pgs=1395 cs=0 l=1 rev1=1 crypto rx=0x7f9fa015b6e0 tx=0x7f9f8c01c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84a6d029b4bfbc1b server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:12:16.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:16.079+0000 7f9f9e000640 1 -- 172.21.15.16:0/3519161524 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f8c0b2070 con 0x7f9fa013e3d0 2024-03-31T15:12:16.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:16.079+0000 7f9f9e000640 1 -- 172.21.15.16:0/3519161524 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9f8c0ad680 con 0x7f9fa013e3d0 2024-03-31T15:12:16.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:16.079+0000 7f9f9e000640 1 -- 172.21.15.16:0/3519161524 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f8c0ada50 con 0x7f9fa013e3d0 2024-03-31T15:12:16.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:16.079+0000 7f9fa6600640 1 -- 172.21.15.16:0/3519161524 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9fa013e3d0 msgr2=0x7f9fa00cc9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:16.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:16.079+0000 7f9fa6600640 1 --2- 172.21.15.16:0/3519161524 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9fa013e3d0 0x7f9fa00cc9d0 secure :-1 s=READY pgs=1395 cs=0 l=1 rev1=1 crypto rx=0x7f9fa015b6e0 tx=0x7f9f8c01c490 comp rx=0 tx=0).stop 2024-03-31T15:12:16.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:16.079+0000 7f9fa6600640 1 -- 172.21.15.16:0/3519161524 shutdown_connections 2024-03-31T15:12:16.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:16.079+0000 7f9fa6600640 1 --2- 172.21.15.16:0/3519161524 >> v2:172.21.15.89:3300/0 conn(0x7f9fa006dbe0 0x7f9fa00ccf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:16.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:16.079+0000 7f9fa6600640 1 --2- 172.21.15.16:0/3519161524 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9fa013e3d0 0x7f9fa00cc9d0 unknown :-1 s=CLOSED pgs=1395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:16.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:16.079+0000 7f9fa6600640 1 -- 172.21.15.16:0/3519161524 >> 172.21.15.16:0/3519161524 conn(0x7f9fa006ee40 msgr2=0x7f9fa006b620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:12:16.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:16.079+0000 7f9fa6600640 1 -- 172.21.15.16:0/3519161524 shutdown_connections 2024-03-31T15:12:16.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:16.079+0000 7f9fa6600640 1 -- 172.21.15.16:0/3519161524 wait complete. 2024-03-31T15:12:16.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:16.079+0000 7f9fa6600640 1 Processor -- start 2024-03-31T15:12:16.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:16.079+0000 7f9fa6600640 1 -- start start 2024-03-31T15:12:16.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:16.083+0000 7f9fa6600640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9fa006dbe0 0x7f9fa013ca90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:16.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:16.083+0000 7f9fa6600640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9fa013dff0 0x7f9fa013ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:16.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:16.083+0000 7f9fa6600640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f9fa013e3d0 0x7f9fa015a820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:16.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:16.083+0000 7f9fa6600640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f9fa00cf710 con 0x7f9fa013dff0 2024-03-31T15:12:16.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:16.083+0000 7f9fa6600640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f9fa00cf590 con 0x7f9fa006dbe0 2024-03-31T15:12:16.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:16.083+0000 7f9fa6600640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f9fa00cf890 con 0x7f9fa013e3d0 2024-03-31T15:12:16.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:16.083+0000 7f9f9ea00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9fa013dff0 0x7f9fa013ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:16.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:16.083+0000 7f9f9fe00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f9fa013e3d0 0x7f9fa015a820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:16.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:16.083+0000 7f9f9f400640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9fa006dbe0 0x7f9fa013ca90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:16.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:16.083+0000 7f9f9fe00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f9fa013e3d0 0x7f9fa015a820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3301/0 says I am v2:172.21.15.16:47324/0 (socket says 172.21.15.16:47324) 2024-03-31T15:12:16.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:16.083+0000 7f9f9ea00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9fa013dff0 0x7f9fa013ea50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37858/0 (socket says 172.21.15.16:37858) 2024-03-31T15:12:16.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:16.083+0000 7f9f9fe00640 1 -- 172.21.15.16:0/4148011118 learned_addr learned my addr 172.21.15.16:0/4148011118 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:12:16.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:16.083+0000 7f9f9f400640 1 -- 172.21.15.16:0/4148011118 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f9fa013e3d0 msgr2=0x7f9fa015a820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:16.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:16.083+0000 7f9f9f400640 1 --2- 172.21.15.16:0/4148011118 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f9fa013e3d0 0x7f9fa015a820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:16.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:16.083+0000 7f9f9f400640 1 -- 172.21.15.16:0/4148011118 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9fa013dff0 msgr2=0x7f9fa013ea50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:16.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:16.083+0000 7f9f9f400640 1 --2- 172.21.15.16:0/4148011118 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9fa013dff0 0x7f9fa013ea50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:16.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:16.083+0000 7f9f9f400640 1 -- 172.21.15.16:0/4148011118 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9fa00cd1e0 con 0x7f9fa006dbe0 2024-03-31T15:12:16.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:16.083+0000 7f9f9ea00640 1 --2- 172.21.15.16:0/4148011118 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9fa013dff0 0x7f9fa013ea50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:12:16.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:16.083+0000 7f9f9f400640 1 --2- 172.21.15.16:0/4148011118 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9fa006dbe0 0x7f9fa013ca90 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f9f9000c3c0 tx=0x7f9f9000c880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:12:16.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:16.083+0000 7f9f97400640 1 -- 172.21.15.16:0/4148011118 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f900052d0 con 0x7f9fa006dbe0 2024-03-31T15:12:16.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:16.083+0000 7f9f97400640 1 -- 172.21.15.16:0/4148011118 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9f90005470 con 0x7f9fa006dbe0 2024-03-31T15:12:16.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:16.083+0000 7f9f97400640 1 -- 172.21.15.16:0/4148011118 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f90007260 con 0x7f9fa006dbe0 2024-03-31T15:12:16.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:16.083+0000 7f9fa6600640 1 -- 172.21.15.16:0/4148011118 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9fa013f430 con 0x7f9fa006dbe0 2024-03-31T15:12:16.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:16.083+0000 7f9fa6600640 1 -- 172.21.15.16:0/4148011118 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9fa013f9f0 con 0x7f9fa006dbe0 2024-03-31T15:12:16.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:16.083+0000 7f9fa6600640 1 -- 172.21.15.16:0/4148011118 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9fa0069950 con 0x7f9fa006dbe0 2024-03-31T15:12:16.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:16.087+0000 7f9f97400640 1 -- 172.21.15.16:0/4148011118 <== mon.1 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f9f90007400 con 0x7f9fa006dbe0 2024-03-31T15:12:16.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:16.087+0000 7f9f97400640 1 --2- 172.21.15.16:0/4148011118 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f9f80068380 0x7f9f8006a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:16.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:16.087+0000 7f9f97400640 1 -- 172.21.15.16:0/4148011118 <== mon.1 v2:172.21.15.89:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 5609+0+0 (secure 0 0 0) 0x7f9f90085cd0 con 0x7f9fa006dbe0 2024-03-31T15:12:16.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:16.087+0000 7f9f9ea00640 1 --2- 172.21.15.16:0/4148011118 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f9f80068380 0x7f9f8006a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:16.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:16.087+0000 7f9f9ea00640 1 --2- 172.21.15.16:0/4148011118 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f9f80068380 0x7f9f8006a840 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f9f8c0af000 tx=0x7f9f8c0a7000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:12:16.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:16.091+0000 7f9f97400640 1 -- 172.21.15.16:0/4148011118 <== mon.1 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f9f90055ca0 con 0x7f9fa006dbe0 2024-03-31T15:12:16.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:16.287+0000 7f9fa6600640 1 -- 172.21.15.16:0/4148011118 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7f9fa01565f0 con 0x7f9fa006dbe0 2024-03-31T15:12:16.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:16.911+0000 7f9f97400640 1 -- 172.21.15.16:0/4148011118 <== mon.1 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 14 and data pool 15 v79) ==== 174+0+0 (secure 0 0 0) 0x7f9f90059fa0 con 0x7f9fa006dbe0 2024-03-31T15:12:16.906 INFO:teuthology.orchestra.run.smithi016.stderr:new fs with metadata pool 14 and data pool 15 2024-03-31T15:12:16.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:16.915+0000 7f9fa6600640 1 -- 172.21.15.16:0/4148011118 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f9f80068380 msgr2=0x7f9f8006a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:16.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:16.915+0000 7f9fa6600640 1 --2- 172.21.15.16:0/4148011118 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f9f80068380 0x7f9f8006a840 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f9f8c0af000 tx=0x7f9f8c0a7000 comp rx=0 tx=0).stop 2024-03-31T15:12:16.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:16.915+0000 7f9fa6600640 1 -- 172.21.15.16:0/4148011118 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9fa006dbe0 msgr2=0x7f9fa013ca90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:16.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:16.915+0000 7f9fa6600640 1 --2- 172.21.15.16:0/4148011118 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9fa006dbe0 0x7f9fa013ca90 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f9f9000c3c0 tx=0x7f9f9000c880 comp rx=0 tx=0).stop 2024-03-31T15:12:16.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:16.915+0000 7f9fa6600640 1 -- 172.21.15.16:0/4148011118 shutdown_connections 2024-03-31T15:12:16.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:16.915+0000 7f9fa6600640 1 --2- 172.21.15.16:0/4148011118 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f9f80068380 0x7f9f8006a840 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:16.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:16.915+0000 7f9fa6600640 1 --2- 172.21.15.16:0/4148011118 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f9fa013e3d0 0x7f9fa015a820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:16.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:16.915+0000 7f9fa6600640 1 --2- 172.21.15.16:0/4148011118 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9fa013dff0 0x7f9fa013ea50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:16.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:16.915+0000 7f9fa6600640 1 --2- 172.21.15.16:0/4148011118 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9fa006dbe0 0x7f9fa013ca90 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:16.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:16.915+0000 7f9fa6600640 1 -- 172.21.15.16:0/4148011118 >> 172.21.15.16:0/4148011118 conn(0x7f9fa006ee40 msgr2=0x7f9fa015baf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:12:16.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:16.915+0000 7f9fa6600640 1 -- 172.21.15.16:0/4148011118 shutdown_connections 2024-03-31T15:12:16.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:16.915+0000 7f9fa6600640 1 -- 172.21.15.16:0/4148011118 wait complete. 2024-03-31T15:12:16.935 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-31T15:12:17.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:17.083+0000 7f00a0c00640 1 Processor -- start 2024-03-31T15:12:17.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:17.083+0000 7f00a0c00640 1 -- start start 2024-03-31T15:12:17.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:17.087+0000 7f00a0c00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f009c06dbc0 0x7f009c15f650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:17.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:17.087+0000 7f00a0c00640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f009c0cbe90 con 0x7f009c070030 2024-03-31T15:12:17.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:17.087+0000 7f00a0c00640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f009c0cc170 con 0x7f009c0cb440 2024-03-31T15:12:17.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:17.087+0000 7f00a0c00640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f009c15fb90 con 0x7f009c06dbc0 2024-03-31T15:12:17.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:17.087+0000 7f009a800640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f009c070030 0x7f009c162180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:54924/0 (socket says 172.21.15.16:54924) 2024-03-31T15:12:17.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:17.087+0000 7f009a800640 1 -- 172.21.15.16:0/2699951100 learned_addr learned my addr 172.21.15.16:0/2699951100 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:12:17.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:17.087+0000 7f0099e00640 1 --2- 172.21.15.16:0/2699951100 >> v2:172.21.15.89:3300/0 conn(0x7f009c06dbc0 0x7f009c15f650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:17.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:17.087+0000 7f0099400640 1 -- 172.21.15.16:0/2699951100 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 477799644 0 0) 0x7f009c0cbe90 con 0x7f009c070030 2024-03-31T15:12:17.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:17.087+0000 7f0099400640 1 -- 172.21.15.16:0/2699951100 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0080003530 con 0x7f009c070030 2024-03-31T15:12:17.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:17.087+0000 7f0099e00640 1 -- 172.21.15.16:0/2699951100 >> v1:172.21.15.89:6789/0 conn(0x7f009c0cb440 legacy=0x7f009c06f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:17.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:17.087+0000 7f0099e00640 1 -- 172.21.15.16:0/2699951100 >> v1:172.21.15.16:6789/0 conn(0x7f009c070030 legacy=0x7f009c162180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:17.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:17.087+0000 7f0099e00640 1 -- 172.21.15.16:0/2699951100 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f009c15ee50 con 0x7f009c06dbc0 2024-03-31T15:12:17.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:17.091+0000 7f0099e00640 1 --2- 172.21.15.16:0/2699951100 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f009c06dbc0 0x7f009c15f650 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f008c0097c0 tx=0x7f008c01c410 comp rx=0 tx=0).ready entity=mon.1 client_cookie=549881597e478dbe server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:12:17.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:17.091+0000 7f0099400640 1 -- 172.21.15.16:0/2699951100 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f008c0b0070 con 0x7f009c06dbc0 2024-03-31T15:12:17.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:17.091+0000 7f0099400640 1 -- 172.21.15.16:0/2699951100 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f008c0ab600 con 0x7f009c06dbc0 2024-03-31T15:12:17.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:17.091+0000 7f0099400640 1 -- 172.21.15.16:0/2699951100 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f008c0ab9d0 con 0x7f009c06dbc0 2024-03-31T15:12:17.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:17.091+0000 7f00a0c00640 1 -- 172.21.15.16:0/2699951100 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f009c06dbc0 msgr2=0x7f009c15f650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:17.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:17.091+0000 7f00a0c00640 1 --2- 172.21.15.16:0/2699951100 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f009c06dbc0 0x7f009c15f650 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f008c0097c0 tx=0x7f008c01c410 comp rx=0 tx=0).stop 2024-03-31T15:12:17.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:17.091+0000 7f00a0c00640 1 -- 172.21.15.16:0/2699951100 shutdown_connections 2024-03-31T15:12:17.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:17.091+0000 7f00a0c00640 1 --2- 172.21.15.16:0/2699951100 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f009c06dbc0 0x7f009c15f650 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:17.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:17.091+0000 7f00a0c00640 1 -- 172.21.15.16:0/2699951100 >> 172.21.15.16:0/2699951100 conn(0x7f009c067330 msgr2=0x7f009c067740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:12:17.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:17.091+0000 7f00a0c00640 1 -- 172.21.15.16:0/2699951100 shutdown_connections 2024-03-31T15:12:17.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:17.091+0000 7f00a0c00640 1 -- 172.21.15.16:0/2699951100 wait complete. 2024-03-31T15:12:17.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:17.091+0000 7f00a0c00640 1 Processor -- start 2024-03-31T15:12:17.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:17.091+0000 7f00a0c00640 1 -- start start 2024-03-31T15:12:17.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:17.091+0000 7f00a0c00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f009c06dbc0 0x7f009c0b4820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:17.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:17.091+0000 7f00a0c00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f009c070030 0x7f009c143f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:17.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:17.091+0000 7f00a0c00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f009c0cb440 0x7f009c15a4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:17.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:17.091+0000 7f00a0c00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f009c06cad0 con 0x7f009c06dbc0 2024-03-31T15:12:17.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:17.091+0000 7f00a0c00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f009c06c950 con 0x7f009c0cb440 2024-03-31T15:12:17.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:17.091+0000 7f00a0c00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f009c16cd40 con 0x7f009c070030 2024-03-31T15:12:17.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:17.095+0000 7f0099e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f009c06dbc0 0x7f009c0b4820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:17.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:17.095+0000 7f0099e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f009c06dbc0 0x7f009c0b4820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40176/0 (socket says 172.21.15.16:40176) 2024-03-31T15:12:17.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:17.095+0000 7f0099e00640 1 -- 172.21.15.16:0/2727908017 learned_addr learned my addr 172.21.15.16:0/2727908017 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:12:17.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:17.095+0000 7f009a800640 1 --2- 172.21.15.16:0/2727908017 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f009c0cb440 0x7f009c15a4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:17.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:17.095+0000 7f0093e00640 1 --2- 172.21.15.16:0/2727908017 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f009c070030 0x7f009c143f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:17.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:17.095+0000 7f0099e00640 1 -- 172.21.15.16:0/2727908017 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f009c070030 msgr2=0x7f009c143f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:17.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:17.095+0000 7f0099e00640 1 --2- 172.21.15.16:0/2727908017 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f009c070030 0x7f009c143f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:17.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:17.095+0000 7f0099e00640 1 -- 172.21.15.16:0/2727908017 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f009c0cb440 msgr2=0x7f009c15a4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:17.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:17.095+0000 7f0099e00640 1 --2- 172.21.15.16:0/2727908017 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f009c0cb440 0x7f009c15a4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:17.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:17.095+0000 7f0099e00640 1 -- 172.21.15.16:0/2727908017 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f009c0b3780 con 0x7f009c06dbc0 2024-03-31T15:12:17.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:17.095+0000 7f0099e00640 1 --2- 172.21.15.16:0/2727908017 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f009c06dbc0 0x7f009c0b4820 secure :-1 s=READY pgs=1397 cs=0 l=1 rev1=1 crypto rx=0x7f008c019020 tx=0x7f008c0191b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:12:17.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:17.095+0000 7f0092000640 1 -- 172.21.15.16:0/2727908017 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f008c0b8020 con 0x7f009c06dbc0 2024-03-31T15:12:17.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:17.095+0000 7f0092000640 1 -- 172.21.15.16:0/2727908017 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f008c0b4410 con 0x7f009c06dbc0 2024-03-31T15:12:17.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:17.095+0000 7f0092000640 1 -- 172.21.15.16:0/2727908017 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f008c0b0040 con 0x7f009c06dbc0 2024-03-31T15:12:17.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:17.095+0000 7f00a0c00640 1 -- 172.21.15.16:0/2727908017 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f009c0b3a60 con 0x7f009c06dbc0 2024-03-31T15:12:17.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:17.095+0000 7f00a0c00640 1 -- 172.21.15.16:0/2727908017 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f009c0b4020 con 0x7f009c06dbc0 2024-03-31T15:12:17.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:17.095+0000 7f00a0c00640 1 -- 172.21.15.16:0/2727908017 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f009c069950 con 0x7f009c06dbc0 2024-03-31T15:12:17.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:17.095+0000 7f0092000640 1 -- 172.21.15.16:0/2727908017 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f008c0bb020 con 0x7f009c06dbc0 2024-03-31T15:12:17.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:17.095+0000 7f0092000640 1 --2- 172.21.15.16:0/2727908017 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f0078068380 0x7f007806a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:17.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:17.099+0000 7f0092000640 1 -- 172.21.15.16:0/2727908017 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 5709+0+0 (secure 0 0 0) 0x7f008c120580 con 0x7f009c06dbc0 2024-03-31T15:12:17.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:17.099+0000 7f0093e00640 1 --2- 172.21.15.16:0/2727908017 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f0078068380 0x7f007806a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:17.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:17.099+0000 7f0093e00640 1 --2- 172.21.15.16:0/2727908017 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f0078068380 0x7f007806a840 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f009c15e6c0 tx=0x7f0084009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:12:17.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:17.103+0000 7f0092000640 1 -- 172.21.15.16:0/2727908017 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f008c0c2030 con 0x7f009c06dbc0 2024-03-31T15:12:17.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:17.279+0000 7f00a0c00640 1 -- 172.21.15.16:0/2727908017 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f009c0d1590 con 0x7f009c06dbc0 2024-03-31T15:12:17.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:17.279+0000 7f0092000640 1 -- 172.21.15.16:0/2727908017 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v95) ==== 74+0+16549 (secure 0 0 0) 0x7f008c016090 con 0x7f009c06dbc0 2024-03-31T15:12:17.272 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:12:17.272 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":95,"fsid":"286eb5da-fb47-4550-b290-d9519c13b9a5","created":"2024-03-31T15:03:11.438952+0000","modified":"2024-03-31T15:12:16.294251+0000","last_up_change":"2024-03-31T15:03:22.885535+0000","last_in_change":"2024-03-31T15:03:13.941667+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-31T15:03:22.550787+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2024-03-31T15:12:14.671225+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"95","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2024-03-31T15:12:15.242196+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"95","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"92a66450-ef19-40d8-ac3d-d760b23b0bda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6801","nonce":932712244}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6803","nonce":932712244}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6807","nonce":932712244}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6805","nonce":932712244}]},"public_addr":"172.21.15.16:6801/932712244","cluster_addr":"172.21.15.16:6803/932712244","heartbeat_back_addr":"172.21.15.16:6807/932712244","heartbeat_front_addr":"172.21.15.16:6805/932712244","state":["exists","up"]},{"osd":1,"uuid":"da9d1f2b-1968-4959-b843-faa231988e92","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6810","nonce":1398737533}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6811","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6813","nonce":1398737533}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6821","nonce":1398737533}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6815","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6816","nonce":1398737533}]},"public_addr":"172.21.15.16:6810/1398737533","cluster_addr":"172.21.15.16:6813/1398737533","heartbeat_back_addr":"172.21.15.16:6821/1398737533","heartbeat_front_addr":"172.21.15.16:6816/1398737533","state":["exists","up"]},{"osd":2,"uuid":"01a73662-9a21-4b1f-bcbb-42b9636a4638","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6819","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6822","nonce":3018526646}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6825","nonce":3018526646}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6831","nonce":3018526646}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6827","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6829","nonce":3018526646}]},"public_addr":"172.21.15.16:6822/3018526646","cluster_addr":"172.21.15.16:6825/3018526646","heartbeat_back_addr":"172.21.15.16:6831/3018526646","heartbeat_front_addr":"172.21.15.16:6829/3018526646","state":["exists","up"]},{"osd":3,"uuid":"c7771fd3-e18e-4500-ac38-23cf10ebcd75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6809","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6812","nonce":3360352974}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6817","nonce":3360352974}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6828","nonce":3360352974}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6823","nonce":3360352974}]},"public_addr":"172.21.15.16:6812/3360352974","cluster_addr":"172.21.15.16:6817/3360352974","heartbeat_back_addr":"172.21.15.16:6828/3360352974","heartbeat_front_addr":"172.21.15.16:6823/3360352974","state":["exists","up"]},{"osd":4,"uuid":"88260730-1482-4fbd-a40f-cb1124e9df4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6801","nonce":1856844856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6803","nonce":1856844856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6807","nonce":1856844856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6805","nonce":1856844856}]},"public_addr":"172.21.15.89:6801/1856844856","cluster_addr":"172.21.15.89:6803/1856844856","heartbeat_back_addr":"172.21.15.89:6807/1856844856","heartbeat_front_addr":"172.21.15.89:6805/1856844856","state":["exists","up"]},{"osd":5,"uuid":"4cb2c00c-fae0-4dd8-9d27-ec4f0093ccf7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6814","nonce":50395527}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6817","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6820","nonce":50395527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6831","nonce":50395527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6823","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6826","nonce":50395527}]},"public_addr":"172.21.15.89:6814/50395527","cluster_addr":"172.21.15.89:6820/50395527","heartbeat_back_addr":"172.21.15.89:6831/50395527","heartbeat_front_addr":"172.21.15.89:6826/50395527","state":["exists","up"]},{"osd":6,"uuid":"81ce0c19-2554-4cae-acbe-66a05e9e9b9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6809","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6812","nonce":3470471752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6815","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6818","nonce":3470471752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6827","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6829","nonce":3470471752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6821","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6824","nonce":3470471752}]},"public_addr":"172.21.15.89:6812/3470471752","cluster_addr":"172.21.15.89:6818/3470471752","heartbeat_back_addr":"172.21.15.89:6829/3470471752","heartbeat_front_addr":"172.21.15.89:6824/3470471752","state":["exists","up"]},{"osd":7,"uuid":"85e4313b-d37d-442e-8042-153bf20a847d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6811","nonce":1796778630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6813","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6816","nonce":1796778630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6825","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6828","nonce":1796778630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6819","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6822","nonce":1796778630}]},"public_addr":"172.21.15.89:6811/1796778630","cluster_addr":"172.21.15.89:6816/1796778630","heartbeat_back_addr":"172.21.15.89:6828/1796778630","heartbeat_front_addr":"172.21.15.89:6822/1796778630","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.439088+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.529970+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.680512+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:21.044161+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.582111+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.536558+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.409522+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.590229+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-31T15:12:17.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:17.279+0000 7f00a0c00640 1 -- 172.21.15.16:0/2727908017 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f0078068380 msgr2=0x7f007806a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:17.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:17.279+0000 7f00a0c00640 1 --2- 172.21.15.16:0/2727908017 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f0078068380 0x7f007806a840 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f009c15e6c0 tx=0x7f0084009040 comp rx=0 tx=0).stop 2024-03-31T15:12:17.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:17.279+0000 7f00a0c00640 1 -- 172.21.15.16:0/2727908017 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f009c06dbc0 msgr2=0x7f009c0b4820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:17.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:17.279+0000 7f00a0c00640 1 --2- 172.21.15.16:0/2727908017 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f009c06dbc0 0x7f009c0b4820 secure :-1 s=READY pgs=1397 cs=0 l=1 rev1=1 crypto rx=0x7f008c019020 tx=0x7f008c0191b0 comp rx=0 tx=0).stop 2024-03-31T15:12:17.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:17.279+0000 7f00a0c00640 1 -- 172.21.15.16:0/2727908017 shutdown_connections 2024-03-31T15:12:17.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:17.279+0000 7f00a0c00640 1 --2- 172.21.15.16:0/2727908017 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f0078068380 0x7f007806a840 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:17.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:17.279+0000 7f00a0c00640 1 --2- 172.21.15.16:0/2727908017 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f009c0cb440 0x7f009c15a4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:17.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:17.279+0000 7f00a0c00640 1 --2- 172.21.15.16:0/2727908017 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f009c070030 0x7f009c143f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:17.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:17.279+0000 7f00a0c00640 1 --2- 172.21.15.16:0/2727908017 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f009c06dbc0 0x7f009c0b4820 unknown :-1 s=CLOSED pgs=1397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:17.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:17.279+0000 7f00a0c00640 1 -- 172.21.15.16:0/2727908017 >> 172.21.15.16:0/2727908017 conn(0x7f009c067330 msgr2=0x7f009c15d1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:12:17.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:17.279+0000 7f00a0c00640 1 -- 172.21.15.16:0/2727908017 shutdown_connections 2024-03-31T15:12:17.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:17.279+0000 7f00a0c00640 1 -- 172.21.15.16:0/2727908017 wait complete. 2024-03-31T15:12:17.294 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-31T15:12:17.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:17.439+0000 7fbc5ee00640 1 Processor -- start 2024-03-31T15:12:17.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:17.439+0000 7fbc5ee00640 1 -- start start 2024-03-31T15:12:17.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:17.443+0000 7fbc5ee00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fbc580728b0 0x7fbc580b1cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:17.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:17.443+0000 7fbc5ee00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fbc580b2200 0x7fbc5815cd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:17.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:17.443+0000 7fbc5ee00640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbc58160e70 con 0x7fbc58145d10 2024-03-31T15:12:17.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:17.443+0000 7fbc5ee00640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7fbc581602f0 con 0x7fbc580b2200 2024-03-31T15:12:17.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:17.443+0000 7fbc5ee00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fbc581605d0 con 0x7fbc580728b0 2024-03-31T15:12:17.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:17.443+0000 7fbc5ce00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fbc580b2200 0x7fbc5815cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:17.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:17.443+0000 7fbc57e00640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fbc58145d10 0x7fbc580cf790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:54930/0 (socket says 172.21.15.16:54930) 2024-03-31T15:12:17.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:17.443+0000 7fbc57e00640 1 -- 172.21.15.16:0/683568202 learned_addr learned my addr 172.21.15.16:0/683568202 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:12:17.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:17.443+0000 7fbc4f400640 1 --2- 172.21.15.16:0/683568202 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fbc580728b0 0x7fbc580b1cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:17.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:17.443+0000 7fbc5ce00640 1 -- 172.21.15.16:0/683568202 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fbc580728b0 msgr2=0x7fbc580b1cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:12:17.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:17.443+0000 7fbc5ce00640 1 --2- 172.21.15.16:0/683568202 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fbc580728b0 0x7fbc580b1cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:17.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:17.443+0000 7fbc5ce00640 1 -- 172.21.15.16:0/683568202 >> v1:172.21.15.16:6789/0 conn(0x7fbc58145d10 legacy=0x7fbc580cf790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:17.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:17.443+0000 7fbc5ce00640 1 -- 172.21.15.16:0/683568202 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc581608b0 con 0x7fbc580b2200 2024-03-31T15:12:17.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:17.447+0000 7fbc5ce00640 1 --2- 172.21.15.16:0/683568202 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbc580b2200 0x7fbc5815cd50 secure :-1 s=READY pgs=1399 cs=0 l=1 rev1=1 crypto rx=0x7fbc50009d90 tx=0x7fbc5001c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af59237dccf07d7a server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:12:17.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:17.447+0000 7fbc57400640 1 -- 172.21.15.16:0/683568202 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc500b2070 con 0x7fbc580b2200 2024-03-31T15:12:17.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:17.447+0000 7fbc57400640 1 -- 172.21.15.16:0/683568202 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbc500a5830 con 0x7fbc580b2200 2024-03-31T15:12:17.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:17.447+0000 7fbc57400640 1 -- 172.21.15.16:0/683568202 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc500a5c00 con 0x7fbc580b2200 2024-03-31T15:12:17.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:17.447+0000 7fbc5ee00640 1 -- 172.21.15.16:0/683568202 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbc580b2200 msgr2=0x7fbc5815cd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:17.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:17.447+0000 7fbc5ee00640 1 --2- 172.21.15.16:0/683568202 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbc580b2200 0x7fbc5815cd50 secure :-1 s=READY pgs=1399 cs=0 l=1 rev1=1 crypto rx=0x7fbc50009d90 tx=0x7fbc5001c970 comp rx=0 tx=0).stop 2024-03-31T15:12:17.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:17.447+0000 7fbc5ee00640 1 -- 172.21.15.16:0/683568202 shutdown_connections 2024-03-31T15:12:17.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:17.447+0000 7fbc5ee00640 1 --2- 172.21.15.16:0/683568202 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbc580b2200 0x7fbc5815cd50 unknown :-1 s=CLOSED pgs=1399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:17.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:17.447+0000 7fbc5ee00640 1 --2- 172.21.15.16:0/683568202 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fbc580728b0 0x7fbc580b1cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:17.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:17.447+0000 7fbc5ee00640 1 -- 172.21.15.16:0/683568202 >> 172.21.15.16:0/683568202 conn(0x7fbc5806b620 msgr2=0x7fbc5806ba30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:12:17.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:17.447+0000 7fbc5ee00640 1 -- 172.21.15.16:0/683568202 shutdown_connections 2024-03-31T15:12:17.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:17.447+0000 7fbc5ee00640 1 -- 172.21.15.16:0/683568202 wait complete. 2024-03-31T15:12:17.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:17.447+0000 7fbc5ee00640 1 Processor -- start 2024-03-31T15:12:17.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:17.447+0000 7fbc5ee00640 1 -- start start 2024-03-31T15:12:17.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:17.447+0000 7fbc5ee00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbc580728b0 0x7fbc581fa5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:17.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:17.447+0000 7fbc5ee00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fbc580b2200 0x7fbc5813db30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:17.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:17.447+0000 7fbc5ee00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbc58145d10 0x7fbc5813e070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:17.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:17.447+0000 7fbc5ee00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fbc58072e10 con 0x7fbc580728b0 2024-03-31T15:12:17.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:17.447+0000 7fbc5ee00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fbc58072c90 con 0x7fbc58145d10 2024-03-31T15:12:17.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:17.447+0000 7fbc5ee00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fbc581690f0 con 0x7fbc580b2200 2024-03-31T15:12:17.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:17.447+0000 7fbc57e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbc580728b0 0x7fbc581fa5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:17.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:17.451+0000 7fbc57e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbc580728b0 0x7fbc581fa5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40198/0 (socket says 172.21.15.16:40198) 2024-03-31T15:12:17.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:17.451+0000 7fbc57e00640 1 -- 172.21.15.16:0/2583059933 learned_addr learned my addr 172.21.15.16:0/2583059933 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:12:17.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:17.451+0000 7fbc5ce00640 1 --2- 172.21.15.16:0/2583059933 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbc58145d10 0x7fbc5813e070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:17.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:17.451+0000 7fbc4f400640 1 --2- 172.21.15.16:0/2583059933 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fbc580b2200 0x7fbc5813db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:17.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:17.451+0000 7fbc57e00640 1 -- 172.21.15.16:0/2583059933 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fbc580b2200 msgr2=0x7fbc5813db30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:17.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:17.451+0000 7fbc57e00640 1 --2- 172.21.15.16:0/2583059933 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fbc580b2200 0x7fbc5813db30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:17.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:17.451+0000 7fbc57e00640 1 -- 172.21.15.16:0/2583059933 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbc58145d10 msgr2=0x7fbc5813e070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:17.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:17.451+0000 7fbc57e00640 1 --2- 172.21.15.16:0/2583059933 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbc58145d10 0x7fbc5813e070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:17.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:17.451+0000 7fbc57e00640 1 -- 172.21.15.16:0/2583059933 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc58141ed0 con 0x7fbc580728b0 2024-03-31T15:12:17.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:17.451+0000 7fbc57e00640 1 --2- 172.21.15.16:0/2583059933 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbc580728b0 0x7fbc581fa5b0 secure :-1 s=READY pgs=1400 cs=0 l=1 rev1=1 crypto rx=0x7fbc4800c3a0 tx=0x7fbc4800c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:12:17.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:17.451+0000 7fbc54c00640 1 -- 172.21.15.16:0/2583059933 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc48017070 con 0x7fbc580728b0 2024-03-31T15:12:17.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:17.451+0000 7fbc54c00640 1 -- 172.21.15.16:0/2583059933 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbc48005cf0 con 0x7fbc580728b0 2024-03-31T15:12:17.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:17.451+0000 7fbc54c00640 1 -- 172.21.15.16:0/2583059933 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc480072f0 con 0x7fbc580728b0 2024-03-31T15:12:17.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:17.451+0000 7fbc5ee00640 1 -- 172.21.15.16:0/2583059933 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbc581421b0 con 0x7fbc580728b0 2024-03-31T15:12:17.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:17.451+0000 7fbc5ee00640 1 -- 172.21.15.16:0/2583059933 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbc58142770 con 0x7fbc580728b0 2024-03-31T15:12:17.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:17.451+0000 7fbc5ee00640 1 -- 172.21.15.16:0/2583059933 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbc58069950 con 0x7fbc580728b0 2024-03-31T15:12:17.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:17.451+0000 7fbc54c00640 1 -- 172.21.15.16:0/2583059933 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fbc4801a070 con 0x7fbc580728b0 2024-03-31T15:12:17.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:17.451+0000 7fbc54c00640 1 --2- 172.21.15.16:0/2583059933 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fbc34068380 0x7fbc3406a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:17.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:17.455+0000 7fbc54c00640 1 -- 172.21.15.16:0/2583059933 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 5709+0+0 (secure 0 0 0) 0x7fbc48085d90 con 0x7fbc580728b0 2024-03-31T15:12:17.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:17.455+0000 7fbc4f400640 1 --2- 172.21.15.16:0/2583059933 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fbc34068380 0x7fbc3406a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:17.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:17.455+0000 7fbc4f400640 1 --2- 172.21.15.16:0/2583059933 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fbc34068380 0x7fbc3406a840 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7fbc58141280 tx=0x7fbc40008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:12:17.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:17.459+0000 7fbc54c00640 1 -- 172.21.15.16:0/2583059933 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fbc480562b0 con 0x7fbc580728b0 2024-03-31T15:12:17.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:17.623+0000 7fbc5ee00640 1 -- 172.21.15.16:0/2583059933 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fbc580b29b0 con 0x7fbc580728b0 2024-03-31T15:12:17.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:17.623+0000 7fbc54c00640 1 -- 172.21.15.16:0/2583059933 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v95) ==== 74+0+16549 (secure 0 0 0) 0x7fbc48055bb0 con 0x7fbc580728b0 2024-03-31T15:12:17.616 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:12:17.617 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":95,"fsid":"286eb5da-fb47-4550-b290-d9519c13b9a5","created":"2024-03-31T15:03:11.438952+0000","modified":"2024-03-31T15:12:16.294251+0000","last_up_change":"2024-03-31T15:03:22.885535+0000","last_in_change":"2024-03-31T15:03:13.941667+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-31T15:03:22.550787+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2024-03-31T15:12:14.671225+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"95","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2024-03-31T15:12:15.242196+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"95","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"92a66450-ef19-40d8-ac3d-d760b23b0bda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6801","nonce":932712244}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6803","nonce":932712244}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6807","nonce":932712244}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6805","nonce":932712244}]},"public_addr":"172.21.15.16:6801/932712244","cluster_addr":"172.21.15.16:6803/932712244","heartbeat_back_addr":"172.21.15.16:6807/932712244","heartbeat_front_addr":"172.21.15.16:6805/932712244","state":["exists","up"]},{"osd":1,"uuid":"da9d1f2b-1968-4959-b843-faa231988e92","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6810","nonce":1398737533}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6811","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6813","nonce":1398737533}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6821","nonce":1398737533}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6815","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6816","nonce":1398737533}]},"public_addr":"172.21.15.16:6810/1398737533","cluster_addr":"172.21.15.16:6813/1398737533","heartbeat_back_addr":"172.21.15.16:6821/1398737533","heartbeat_front_addr":"172.21.15.16:6816/1398737533","state":["exists","up"]},{"osd":2,"uuid":"01a73662-9a21-4b1f-bcbb-42b9636a4638","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6819","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6822","nonce":3018526646}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6825","nonce":3018526646}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6831","nonce":3018526646}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6827","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6829","nonce":3018526646}]},"public_addr":"172.21.15.16:6822/3018526646","cluster_addr":"172.21.15.16:6825/3018526646","heartbeat_back_addr":"172.21.15.16:6831/3018526646","heartbeat_front_addr":"172.21.15.16:6829/3018526646","state":["exists","up"]},{"osd":3,"uuid":"c7771fd3-e18e-4500-ac38-23cf10ebcd75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6809","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6812","nonce":3360352974}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6817","nonce":3360352974}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6828","nonce":3360352974}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6823","nonce":3360352974}]},"public_addr":"172.21.15.16:6812/3360352974","cluster_addr":"172.21.15.16:6817/3360352974","heartbeat_back_addr":"172.21.15.16:6828/3360352974","heartbeat_front_addr":"172.21.15.16:6823/3360352974","state":["exists","up"]},{"osd":4,"uuid":"88260730-1482-4fbd-a40f-cb1124e9df4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6801","nonce":1856844856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6803","nonce":1856844856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6807","nonce":1856844856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6805","nonce":1856844856}]},"public_addr":"172.21.15.89:6801/1856844856","cluster_addr":"172.21.15.89:6803/1856844856","heartbeat_back_addr":"172.21.15.89:6807/1856844856","heartbeat_front_addr":"172.21.15.89:6805/1856844856","state":["exists","up"]},{"osd":5,"uuid":"4cb2c00c-fae0-4dd8-9d27-ec4f0093ccf7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6814","nonce":50395527}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6817","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6820","nonce":50395527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6831","nonce":50395527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6823","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6826","nonce":50395527}]},"public_addr":"172.21.15.89:6814/50395527","cluster_addr":"172.21.15.89:6820/50395527","heartbeat_back_addr":"172.21.15.89:6831/50395527","heartbeat_front_addr":"172.21.15.89:6826/50395527","state":["exists","up"]},{"osd":6,"uuid":"81ce0c19-2554-4cae-acbe-66a05e9e9b9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6809","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6812","nonce":3470471752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6815","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6818","nonce":3470471752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6827","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6829","nonce":3470471752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6821","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6824","nonce":3470471752}]},"public_addr":"172.21.15.89:6812/3470471752","cluster_addr":"172.21.15.89:6818/3470471752","heartbeat_back_addr":"172.21.15.89:6829/3470471752","heartbeat_front_addr":"172.21.15.89:6824/3470471752","state":["exists","up"]},{"osd":7,"uuid":"85e4313b-d37d-442e-8042-153bf20a847d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6811","nonce":1796778630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6813","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6816","nonce":1796778630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6825","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6828","nonce":1796778630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6819","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6822","nonce":1796778630}]},"public_addr":"172.21.15.89:6811/1796778630","cluster_addr":"172.21.15.89:6816/1796778630","heartbeat_back_addr":"172.21.15.89:6828/1796778630","heartbeat_front_addr":"172.21.15.89:6822/1796778630","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.439088+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.529970+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.680512+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:21.044161+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.582111+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.536558+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.409522+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.590229+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-31T15:12:17.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:17.627+0000 7fbc5ee00640 1 -- 172.21.15.16:0/2583059933 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fbc34068380 msgr2=0x7fbc3406a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:17.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:17.627+0000 7fbc5ee00640 1 --2- 172.21.15.16:0/2583059933 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fbc34068380 0x7fbc3406a840 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7fbc58141280 tx=0x7fbc40008040 comp rx=0 tx=0).stop 2024-03-31T15:12:17.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:17.627+0000 7fbc5ee00640 1 -- 172.21.15.16:0/2583059933 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbc580728b0 msgr2=0x7fbc581fa5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:17.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:17.627+0000 7fbc5ee00640 1 --2- 172.21.15.16:0/2583059933 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbc580728b0 0x7fbc581fa5b0 secure :-1 s=READY pgs=1400 cs=0 l=1 rev1=1 crypto rx=0x7fbc4800c3a0 tx=0x7fbc4800c860 comp rx=0 tx=0).stop 2024-03-31T15:12:17.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:17.627+0000 7fbc5ee00640 1 -- 172.21.15.16:0/2583059933 shutdown_connections 2024-03-31T15:12:17.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:17.627+0000 7fbc5ee00640 1 --2- 172.21.15.16:0/2583059933 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fbc34068380 0x7fbc3406a840 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:17.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:17.627+0000 7fbc5ee00640 1 --2- 172.21.15.16:0/2583059933 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbc58145d10 0x7fbc5813e070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:17.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:17.627+0000 7fbc5ee00640 1 --2- 172.21.15.16:0/2583059933 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fbc580b2200 0x7fbc5813db30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:17.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:17.627+0000 7fbc5ee00640 1 --2- 172.21.15.16:0/2583059933 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbc580728b0 0x7fbc581fa5b0 unknown :-1 s=CLOSED pgs=1400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:17.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:17.627+0000 7fbc5ee00640 1 -- 172.21.15.16:0/2583059933 >> 172.21.15.16:0/2583059933 conn(0x7fbc5806b620 msgr2=0x7fbc580cff50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:12:17.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:17.627+0000 7fbc5ee00640 1 -- 172.21.15.16:0/2583059933 shutdown_connections 2024-03-31T15:12:17.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:17.627+0000 7fbc5ee00640 1 -- 172.21.15.16:0/2583059933 wait complete. 2024-03-31T15:12:17.638 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-31T15:12:17.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:17.787+0000 7f2b23c00640 1 Processor -- start 2024-03-31T15:12:17.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:17.791+0000 7f2b23c00640 1 -- start start 2024-03-31T15:12:17.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:17.791+0000 7f2b23c00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f2b1c145190 0x7f2b1c06f160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:17.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:17.791+0000 7f2b23c00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f2b1c06b620 0x7f2b1c06ba80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:17.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:17.791+0000 7f2b23c00640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2b1c160230 con 0x7f2b1c06fc40 2024-03-31T15:12:17.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:17.791+0000 7f2b23c00640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f2b1c071060 con 0x7f2b1c06b620 2024-03-31T15:12:17.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:17.791+0000 7f2b23c00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f2b1c06bfc0 con 0x7f2b1c145190 2024-03-31T15:12:17.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:17.791+0000 7f2b1be00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f2b1c06b620 0x7f2b1c06ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:17.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:17.791+0000 7f2b1be00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f2b1c06b620 0x7f2b1c06ba80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40214/0 (socket says 172.21.15.16:40214) 2024-03-31T15:12:17.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:17.791+0000 7f2b1be00640 1 -- 172.21.15.16:0/3808852853 learned_addr learned my addr 172.21.15.16:0/3808852853 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:12:17.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:17.791+0000 7f2b21200640 1 --2- 172.21.15.16:0/3808852853 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f2b1c145190 0x7f2b1c06f160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:17.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:17.791+0000 7f2b1be00640 1 -- 172.21.15.16:0/3808852853 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f2b1c145190 msgr2=0x7f2b1c06f160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:12:17.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:17.791+0000 7f2b1be00640 1 --2- 172.21.15.16:0/3808852853 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f2b1c145190 0x7f2b1c06f160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:17.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:17.791+0000 7f2b1be00640 1 -- 172.21.15.16:0/3808852853 >> v1:172.21.15.89:6789/0 conn(0x7f2b1c06fc40 legacy=0x7f2b1c0700d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:17.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:17.791+0000 7f2b1be00640 1 -- 172.21.15.16:0/3808852853 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b1c160510 con 0x7f2b1c06b620 2024-03-31T15:12:17.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:17.795+0000 7f2b1be00640 1 --2- 172.21.15.16:0/3808852853 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2b1c06b620 0x7f2b1c06ba80 secure :-1 s=READY pgs=1401 cs=0 l=1 rev1=1 crypto rx=0x7f2b08009870 tx=0x7f2b0801c440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7937b47e8ba33e5 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:12:17.787 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:17.795+0000 7f2b1b400640 1 -- 172.21.15.16:0/3808852853 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b080b0070 con 0x7f2b1c06b620 2024-03-31T15:12:17.787 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:17.795+0000 7f2b1b400640 1 -- 172.21.15.16:0/3808852853 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2b080ab600 con 0x7f2b1c06b620 2024-03-31T15:12:17.787 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:17.795+0000 7f2b1b400640 1 -- 172.21.15.16:0/3808852853 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b080ab9d0 con 0x7f2b1c06b620 2024-03-31T15:12:17.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:17.795+0000 7f2b23c00640 1 -- 172.21.15.16:0/3808852853 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2b1c06b620 msgr2=0x7f2b1c06ba80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:17.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:17.795+0000 7f2b23c00640 1 --2- 172.21.15.16:0/3808852853 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2b1c06b620 0x7f2b1c06ba80 secure :-1 s=READY pgs=1401 cs=0 l=1 rev1=1 crypto rx=0x7f2b08009870 tx=0x7f2b0801c440 comp rx=0 tx=0).stop 2024-03-31T15:12:17.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:17.795+0000 7f2b23c00640 1 -- 172.21.15.16:0/3808852853 shutdown_connections 2024-03-31T15:12:17.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:17.795+0000 7f2b23c00640 1 --2- 172.21.15.16:0/3808852853 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2b1c06b620 0x7f2b1c06ba80 unknown :-1 s=CLOSED pgs=1401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:17.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:17.795+0000 7f2b23c00640 1 --2- 172.21.15.16:0/3808852853 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f2b1c145190 0x7f2b1c06f160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:17.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:17.795+0000 7f2b23c00640 1 -- 172.21.15.16:0/3808852853 >> 172.21.15.16:0/3808852853 conn(0x7f2b1c06e670 msgr2=0x7f2b1c06ea50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:12:17.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:17.795+0000 7f2b23c00640 1 -- 172.21.15.16:0/3808852853 shutdown_connections 2024-03-31T15:12:17.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:17.795+0000 7f2b23c00640 1 -- 172.21.15.16:0/3808852853 wait complete. 2024-03-31T15:12:17.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:17.795+0000 7f2b23c00640 1 Processor -- start 2024-03-31T15:12:17.790 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:17.799+0000 7f2b23c00640 1 -- start start 2024-03-31T15:12:17.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:17.799+0000 7f2b23c00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2b1c06b620 0x7f2b1c141b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:17.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:17.799+0000 7f2b23c00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2b1c06fc40 0x7f2b1c13f210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:17.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:17.799+0000 7f2b23c00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f2b1c145190 0x7f2b1c156740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:17.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:17.799+0000 7f2b23c00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f2b1c06f820 con 0x7f2b1c06fc40 2024-03-31T15:12:17.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:17.799+0000 7f2b23c00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f2b1c06f6a0 con 0x7f2b1c06b620 2024-03-31T15:12:17.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:17.799+0000 7f2b23c00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f2b1c169a10 con 0x7f2b1c145190 2024-03-31T15:12:17.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:17.799+0000 7f2b1be00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2b1c06fc40 0x7f2b1c13f210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:17.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:17.799+0000 7f2b1be00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2b1c06fc40 0x7f2b1c13f210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40220/0 (socket says 172.21.15.16:40220) 2024-03-31T15:12:17.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:17.799+0000 7f2b1be00640 1 -- 172.21.15.16:0/2447474131 learned_addr learned my addr 172.21.15.16:0/2447474131 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:12:17.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:17.799+0000 7f2b21200640 1 --2- 172.21.15.16:0/2447474131 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2b1c06b620 0x7f2b1c141b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:17.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:17.799+0000 7f2b21c00640 1 --2- 172.21.15.16:0/2447474131 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f2b1c145190 0x7f2b1c156740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:17.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:17.799+0000 7f2b1be00640 1 -- 172.21.15.16:0/2447474131 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f2b1c145190 msgr2=0x7f2b1c156740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:17.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:17.799+0000 7f2b1be00640 1 --2- 172.21.15.16:0/2447474131 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f2b1c145190 0x7f2b1c156740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:17.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:17.799+0000 7f2b1be00640 1 -- 172.21.15.16:0/2447474131 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2b1c06b620 msgr2=0x7f2b1c141b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:17.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:17.799+0000 7f2b1be00640 1 --2- 172.21.15.16:0/2447474131 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2b1c06b620 0x7f2b1c141b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:17.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:17.799+0000 7f2b1be00640 1 -- 172.21.15.16:0/2447474131 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b1c13fc60 con 0x7f2b1c06fc40 2024-03-31T15:12:17.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:17.799+0000 7f2b1be00640 1 --2- 172.21.15.16:0/2447474131 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2b1c06fc40 0x7f2b1c13f210 secure :-1 s=READY pgs=1402 cs=0 l=1 rev1=1 crypto rx=0x7f2b080ae040 tx=0x7f2b080191e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:12:17.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:17.799+0000 7f2b18c00640 1 -- 172.21.15.16:0/2447474131 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b080b0040 con 0x7f2b1c06fc40 2024-03-31T15:12:17.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:17.799+0000 7f2b18c00640 1 -- 172.21.15.16:0/2447474131 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2b080ab400 con 0x7f2b1c06fc40 2024-03-31T15:12:17.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:17.799+0000 7f2b18c00640 1 -- 172.21.15.16:0/2447474131 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b080b4680 con 0x7f2b1c06fc40 2024-03-31T15:12:17.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:17.799+0000 7f2b23c00640 1 -- 172.21.15.16:0/2447474131 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2b1c13ff40 con 0x7f2b1c06fc40 2024-03-31T15:12:17.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:17.799+0000 7f2b23c00640 1 -- 172.21.15.16:0/2447474131 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2b1c140500 con 0x7f2b1c06fc40 2024-03-31T15:12:17.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:17.799+0000 7f2b23c00640 1 -- 172.21.15.16:0/2447474131 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2b1c069950 con 0x7f2b1c06fc40 2024-03-31T15:12:17.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:17.803+0000 7f2b18c00640 1 -- 172.21.15.16:0/2447474131 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f2b080bb020 con 0x7f2b1c06fc40 2024-03-31T15:12:17.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:17.803+0000 7f2b18c00640 1 --2- 172.21.15.16:0/2447474131 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f2afc0683b0 0x7f2afc06a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:17.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:17.803+0000 7f2b18c00640 1 -- 172.21.15.16:0/2447474131 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 5709+0+0 (secure 0 0 0) 0x7f2b0811fb50 con 0x7f2b1c06fc40 2024-03-31T15:12:17.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:17.803+0000 7f2b21200640 1 --2- 172.21.15.16:0/2447474131 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f2afc0683b0 0x7f2afc06a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:17.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:17.803+0000 7f2b21200640 1 --2- 172.21.15.16:0/2447474131 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f2afc0683b0 0x7f2afc06a870 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f2b1c15f6e0 tx=0x7f2b10009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:12:17.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:17.807+0000 7f2b18c00640 1 -- 172.21.15.16:0/2447474131 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f2b080c2020 con 0x7f2b1c06fc40 2024-03-31T15:12:18.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:18.019+0000 7f2b23c00640 1 -- 172.21.15.16:0/2447474131 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f2b1c0737b0 con 0x7f2b1c06fc40 2024-03-31T15:12:18.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:18.911+0000 7f2b18c00640 1 -- 172.21.15.16:0/2447474131 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v82) ==== 118+0+0 (secure 0 0 0) 0x7f2b08016150 con 0x7f2b1c06fc40 2024-03-31T15:12:18.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:18.911+0000 7f2b23c00640 1 -- 172.21.15.16:0/2447474131 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f2afc0683b0 msgr2=0x7f2afc06a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:18.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:18.911+0000 7f2b23c00640 1 --2- 172.21.15.16:0/2447474131 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f2afc0683b0 0x7f2afc06a870 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f2b1c15f6e0 tx=0x7f2b10009190 comp rx=0 tx=0).stop 2024-03-31T15:12:18.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:18.911+0000 7f2b23c00640 1 -- 172.21.15.16:0/2447474131 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2b1c06fc40 msgr2=0x7f2b1c13f210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:18.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:18.911+0000 7f2b23c00640 1 --2- 172.21.15.16:0/2447474131 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2b1c06fc40 0x7f2b1c13f210 secure :-1 s=READY pgs=1402 cs=0 l=1 rev1=1 crypto rx=0x7f2b080ae040 tx=0x7f2b080191e0 comp rx=0 tx=0).stop 2024-03-31T15:12:18.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:18.911+0000 7f2b23c00640 1 -- 172.21.15.16:0/2447474131 shutdown_connections 2024-03-31T15:12:18.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:18.911+0000 7f2b23c00640 1 --2- 172.21.15.16:0/2447474131 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f2afc0683b0 0x7f2afc06a870 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:18.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:18.911+0000 7f2b23c00640 1 --2- 172.21.15.16:0/2447474131 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f2b1c145190 0x7f2b1c156740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:18.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:18.911+0000 7f2b23c00640 1 --2- 172.21.15.16:0/2447474131 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2b1c06fc40 0x7f2b1c13f210 unknown :-1 s=CLOSED pgs=1402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:18.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:18.911+0000 7f2b23c00640 1 --2- 172.21.15.16:0/2447474131 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2b1c06b620 0x7f2b1c141b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:18.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:18.911+0000 7f2b23c00640 1 -- 172.21.15.16:0/2447474131 >> 172.21.15.16:0/2447474131 conn(0x7f2b1c06e670 msgr2=0x7f2b1c165460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:12:18.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:18.911+0000 7f2b23c00640 1 -- 172.21.15.16:0/2447474131 shutdown_connections 2024-03-31T15:12:18.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:18.911+0000 7f2b23c00640 1 -- 172.21.15.16:0/2447474131 wait complete. 2024-03-31T15:12:18.924 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-03-31T15:12:18.925 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-31T15:12:19.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:19.075+0000 7f849d400640 1 Processor -- start 2024-03-31T15:12:19.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:19.075+0000 7f849d400640 1 -- start start 2024-03-31T15:12:19.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:19.079+0000 7f849d400640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f8498070030 0x7f8498143c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:19.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:19.079+0000 7f849d400640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f8498141590 0x7f8498144170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:19.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:19.079+0000 7f849d400640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f84981446b0 con 0x7f84981411b0 2024-03-31T15:12:19.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:19.079+0000 7f849d400640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f849815f910 con 0x7f8498070030 2024-03-31T15:12:19.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:19.079+0000 7f849d400640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f8498162910 con 0x7f8498141590 2024-03-31T15:12:19.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:19.079+0000 7f8497200640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f8498141590 0x7f8498144170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:19.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:19.079+0000 7f8495e00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f8498070030 0x7f8498143c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:19.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:19.079+0000 7f8497200640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f8498141590 0x7f8498144170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3301/0 says I am v2:172.21.15.16:57456/0 (socket says 172.21.15.16:57456) 2024-03-31T15:12:19.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:19.079+0000 7f8497200640 1 -- 172.21.15.16:0/1356441843 learned_addr learned my addr 172.21.15.16:0/1356441843 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:12:19.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:19.079+0000 7f8497200640 1 -- 172.21.15.16:0/1356441843 >> v2:172.21.15.89:3300/0 conn(0x7f8498070030 msgr2=0x7f8498143c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:12:19.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:19.079+0000 7f8497200640 1 --2- 172.21.15.16:0/1356441843 >> v2:172.21.15.89:3300/0 conn(0x7f8498070030 0x7f8498143c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:19.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:19.079+0000 7f8497200640 1 -- 172.21.15.16:0/1356441843 >> v1:172.21.15.89:6789/0 conn(0x7f84981411b0 legacy=0x7f849806f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:19.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:19.079+0000 7f8497200640 1 -- 172.21.15.16:0/1356441843 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8498144980 con 0x7f8498141590 2024-03-31T15:12:19.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:19.079+0000 7f8495e00640 1 --2- 172.21.15.16:0/1356441843 >> v2:172.21.15.89:3300/0 conn(0x7f8498070030 0x7f8498143c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:12:19.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:19.083+0000 7f8497200640 1 --2- 172.21.15.16:0/1356441843 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f8498141590 0x7f8498144170 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f8490009e50 tx=0x7f849001ca00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2cf271170df4270d server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:12:19.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:19.083+0000 7f8495400640 1 -- 172.21.15.16:0/1356441843 <== mon.2 v2:172.21.15.89:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84900a58d0 con 0x7f8498141590 2024-03-31T15:12:19.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:19.083+0000 7f8495400640 1 -- 172.21.15.16:0/1356441843 <== mon.2 v2:172.21.15.89:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f84900a5a70 con 0x7f8498141590 2024-03-31T15:12:19.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:19.083+0000 7f8495400640 1 -- 172.21.15.16:0/1356441843 <== mon.2 v2:172.21.15.89:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84900a5e40 con 0x7f8498141590 2024-03-31T15:12:19.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:19.083+0000 7f849d400640 1 -- 172.21.15.16:0/1356441843 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f8498141590 msgr2=0x7f8498144170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:19.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:19.083+0000 7f849d400640 1 --2- 172.21.15.16:0/1356441843 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f8498141590 0x7f8498144170 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f8490009e50 tx=0x7f849001ca00 comp rx=0 tx=0).stop 2024-03-31T15:12:19.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:19.083+0000 7f849d400640 1 -- 172.21.15.16:0/1356441843 shutdown_connections 2024-03-31T15:12:19.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:19.083+0000 7f849d400640 1 --2- 172.21.15.16:0/1356441843 >> v2:172.21.15.89:3300/0 conn(0x7f8498070030 0x7f8498143c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:19.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:19.083+0000 7f849d400640 1 --2- 172.21.15.16:0/1356441843 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f8498141590 0x7f8498144170 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:19.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:19.083+0000 7f849d400640 1 -- 172.21.15.16:0/1356441843 >> 172.21.15.16:0/1356441843 conn(0x7f8498067930 msgr2=0x7f84980b3ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:12:19.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:19.083+0000 7f849d400640 1 -- 172.21.15.16:0/1356441843 shutdown_connections 2024-03-31T15:12:19.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:19.083+0000 7f849d400640 1 -- 172.21.15.16:0/1356441843 wait complete. 2024-03-31T15:12:19.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:19.083+0000 7f849d400640 1 Processor -- start 2024-03-31T15:12:19.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:19.083+0000 7f849d400640 1 -- start start 2024-03-31T15:12:19.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:19.087+0000 7f849d400640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8498070030 0x7f84981419a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:19.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:19.087+0000 7f849d400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f84981411b0 0x7f8498153b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:19.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:19.087+0000 7f849d400640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f8498141590 0x7f849813f990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:19.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:19.087+0000 7f849d400640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f849815f110 con 0x7f84981411b0 2024-03-31T15:12:19.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:19.087+0000 7f849d400640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f8498162d80 con 0x7f8498070030 2024-03-31T15:12:19.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:19.087+0000 7f849d400640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f849815f290 con 0x7f8498141590 2024-03-31T15:12:19.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:19.087+0000 7f8495e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f84981411b0 0x7f8498153b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:19.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:19.087+0000 7f8495e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f84981411b0 0x7f8498153b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40222/0 (socket says 172.21.15.16:40222) 2024-03-31T15:12:19.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:19.087+0000 7f8495e00640 1 -- 172.21.15.16:0/4286313997 learned_addr learned my addr 172.21.15.16:0/4286313997 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:12:19.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:19.087+0000 7f8496800640 1 --2- 172.21.15.16:0/4286313997 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8498070030 0x7f84981419a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:19.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:19.087+0000 7f8497200640 1 --2- 172.21.15.16:0/4286313997 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f8498141590 0x7f849813f990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:19.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:19.087+0000 7f8495e00640 1 -- 172.21.15.16:0/4286313997 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f8498141590 msgr2=0x7f849813f990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:19.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:19.087+0000 7f8495e00640 1 --2- 172.21.15.16:0/4286313997 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f8498141590 0x7f849813f990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:19.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:19.087+0000 7f8495e00640 1 -- 172.21.15.16:0/4286313997 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8498070030 msgr2=0x7f84981419a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:19.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:19.087+0000 7f8495e00640 1 --2- 172.21.15.16:0/4286313997 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8498070030 0x7f84981419a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:19.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:19.087+0000 7f8495e00640 1 -- 172.21.15.16:0/4286313997 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8498154060 con 0x7f84981411b0 2024-03-31T15:12:19.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:19.087+0000 7f8495e00640 1 --2- 172.21.15.16:0/4286313997 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f84981411b0 0x7f8498153b20 secure :-1 s=READY pgs=1403 cs=0 l=1 rev1=1 crypto rx=0x7f848400ebe0 tx=0x7f848400c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:12:19.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:19.087+0000 7f848ea00640 1 -- 172.21.15.16:0/4286313997 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8484017070 con 0x7f84981411b0 2024-03-31T15:12:19.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:19.087+0000 7f848ea00640 1 -- 172.21.15.16:0/4286313997 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f848400cc30 con 0x7f84981411b0 2024-03-31T15:12:19.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:19.087+0000 7f848ea00640 1 -- 172.21.15.16:0/4286313997 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8484013470 con 0x7f84981411b0 2024-03-31T15:12:19.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:19.087+0000 7f849d400640 1 -- 172.21.15.16:0/4286313997 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8498141ff0 con 0x7f84981411b0 2024-03-31T15:12:19.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:19.087+0000 7f849d400640 1 -- 172.21.15.16:0/4286313997 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f84981425b0 con 0x7f84981411b0 2024-03-31T15:12:19.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:19.087+0000 7f849d400640 1 -- 172.21.15.16:0/4286313997 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8498069950 con 0x7f84981411b0 2024-03-31T15:12:19.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:19.087+0000 7f848ea00640 1 -- 172.21.15.16:0/4286313997 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f848401a070 con 0x7f84981411b0 2024-03-31T15:12:19.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:19.091+0000 7f848ea00640 1 --2- 172.21.15.16:0/4286313997 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f84780683b0 0x7f847806a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:19.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:19.091+0000 7f848ea00640 1 -- 172.21.15.16:0/4286313997 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 5709+0+0 (secure 0 0 0) 0x7f8484086870 con 0x7f84981411b0 2024-03-31T15:12:19.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:19.091+0000 7f8496800640 1 --2- 172.21.15.16:0/4286313997 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f84780683b0 0x7f847806a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:19.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:19.091+0000 7f8496800640 1 --2- 172.21.15.16:0/4286313997 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f84780683b0 0x7f847806a870 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f8498154690 tx=0x7f8488009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:12:19.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:19.095+0000 7f848ea00640 1 -- 172.21.15.16:0/4286313997 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f84840567e0 con 0x7f84981411b0 2024-03-31T15:12:19.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:19.291+0000 7f849d400640 1 -- 172.21.15.16:0/4286313997 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7f84980d2f00 con 0x7f84981411b0 2024-03-31T15:12:19.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:19.915+0000 7f848ea00640 1 -- 172.21.15.16:0/4286313997 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v83) ==== 122+0+0 (secure 0 0 0) 0x7f848405aae0 con 0x7f84981411b0 2024-03-31T15:12:19.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:19.919+0000 7f849d400640 1 -- 172.21.15.16:0/4286313997 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f84780683b0 msgr2=0x7f847806a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:19.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:19.919+0000 7f849d400640 1 --2- 172.21.15.16:0/4286313997 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f84780683b0 0x7f847806a870 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f8498154690 tx=0x7f8488009040 comp rx=0 tx=0).stop 2024-03-31T15:12:19.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:19.919+0000 7f849d400640 1 -- 172.21.15.16:0/4286313997 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f84981411b0 msgr2=0x7f8498153b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:19.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:19.919+0000 7f849d400640 1 --2- 172.21.15.16:0/4286313997 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f84981411b0 0x7f8498153b20 secure :-1 s=READY pgs=1403 cs=0 l=1 rev1=1 crypto rx=0x7f848400ebe0 tx=0x7f848400c520 comp rx=0 tx=0).stop 2024-03-31T15:12:19.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:19.919+0000 7f849d400640 1 -- 172.21.15.16:0/4286313997 shutdown_connections 2024-03-31T15:12:19.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:19.919+0000 7f849d400640 1 --2- 172.21.15.16:0/4286313997 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f84780683b0 0x7f847806a870 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:19.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:19.919+0000 7f849d400640 1 --2- 172.21.15.16:0/4286313997 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f8498141590 0x7f849813f990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:19.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:19.919+0000 7f849d400640 1 --2- 172.21.15.16:0/4286313997 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f84981411b0 0x7f8498153b20 unknown :-1 s=CLOSED pgs=1403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:19.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:19.919+0000 7f849d400640 1 --2- 172.21.15.16:0/4286313997 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8498070030 0x7f84981419a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:19.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:19.919+0000 7f849d400640 1 -- 172.21.15.16:0/4286313997 >> 172.21.15.16:0/4286313997 conn(0x7f8498067930 msgr2=0x7f8498151ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:12:19.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:19.919+0000 7f849d400640 1 -- 172.21.15.16:0/4286313997 shutdown_connections 2024-03-31T15:12:19.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:19.919+0000 7f849d400640 1 -- 172.21.15.16:0/4286313997 wait complete. 2024-03-31T15:12:19.930 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-31T15:12:19.931 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-31T15:12:20.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:20.131+0000 7f4f34000640 1 Processor -- start 2024-03-31T15:12:20.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:20.131+0000 7f4f34000640 1 -- start start 2024-03-31T15:12:20.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:20.135+0000 7f4f34000640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f4f2c0b23d0 0x7f4f2c071d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:20.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:20.135+0000 7f4f34000640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4f2c15f3c0 con 0x7f4f2c15b3d0 2024-03-31T15:12:20.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:20.135+0000 7f4f34000640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4f2c15f6a0 con 0x7f4f2c15eee0 2024-03-31T15:12:20.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:20.135+0000 7f4f34000640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f4f2c15b7b0 con 0x7f4f2c0b23d0 2024-03-31T15:12:20.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:20.135+0000 7f4f31600640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f4f2c15b3d0 0x7f4f2c0b1cc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:54932/0 (socket says 172.21.15.16:54932) 2024-03-31T15:12:20.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:20.135+0000 7f4f31600640 1 -- 172.21.15.16:0/2013060739 learned_addr learned my addr 172.21.15.16:0/2013060739 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:12:20.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:20.135+0000 7f4f30c00640 1 --2- 172.21.15.16:0/2013060739 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f4f2c0b23d0 0x7f4f2c071d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:20.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:20.135+0000 7f4f2be00640 1 -- 172.21.15.16:0/2013060739 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2761775037 0 0) 0x7f4f2c15f3c0 con 0x7f4f2c15b3d0 2024-03-31T15:12:20.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:20.135+0000 7f4f2be00640 1 -- 172.21.15.16:0/2013060739 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4f14003530 con 0x7f4f2c15b3d0 2024-03-31T15:12:20.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:20.135+0000 7f4f2be00640 1 -- 172.21.15.16:0/2013060739 <== mon.1 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1161016342 0 0) 0x7f4f2c15f6a0 con 0x7f4f2c15eee0 2024-03-31T15:12:20.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:20.135+0000 7f4f2be00640 1 -- 172.21.15.16:0/2013060739 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4f2c15f3c0 con 0x7f4f2c15eee0 2024-03-31T15:12:20.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:20.135+0000 7f4f30c00640 1 -- 172.21.15.16:0/2013060739 >> v1:172.21.15.89:6789/0 conn(0x7f4f2c15eee0 legacy=0x7f4f2c072250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:20.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:20.135+0000 7f4f30c00640 1 -- 172.21.15.16:0/2013060739 >> v1:172.21.15.16:6789/0 conn(0x7f4f2c15b3d0 legacy=0x7f4f2c0b1cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:20.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:20.135+0000 7f4f30c00640 1 -- 172.21.15.16:0/2013060739 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f2c15f980 con 0x7f4f2c0b23d0 2024-03-31T15:12:20.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:20.139+0000 7f4f30c00640 1 --2- 172.21.15.16:0/2013060739 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f4f2c0b23d0 0x7f4f2c071d10 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f4f180097f0 tx=0x7f4f1801c4b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bd8d5b8ca0b86d62 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:12:20.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:20.139+0000 7f4f2be00640 1 -- 172.21.15.16:0/2013060739 <== mon.2 v2:172.21.15.89:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f180a7070 con 0x7f4f2c0b23d0 2024-03-31T15:12:20.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:20.139+0000 7f4f2be00640 1 -- 172.21.15.16:0/2013060739 <== mon.2 v2:172.21.15.89:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4f180b0070 con 0x7f4f2c0b23d0 2024-03-31T15:12:20.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:20.139+0000 7f4f2be00640 1 -- 172.21.15.16:0/2013060739 <== mon.2 v2:172.21.15.89:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f180ab870 con 0x7f4f2c0b23d0 2024-03-31T15:12:20.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:20.139+0000 7f4f34000640 1 -- 172.21.15.16:0/2013060739 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f4f2c0b23d0 msgr2=0x7f4f2c071d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:20.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:20.139+0000 7f4f34000640 1 --2- 172.21.15.16:0/2013060739 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f4f2c0b23d0 0x7f4f2c071d10 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f4f180097f0 tx=0x7f4f1801c4b0 comp rx=0 tx=0).stop 2024-03-31T15:12:20.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:20.139+0000 7f4f34000640 1 -- 172.21.15.16:0/2013060739 shutdown_connections 2024-03-31T15:12:20.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:20.139+0000 7f4f34000640 1 --2- 172.21.15.16:0/2013060739 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f4f2c0b23d0 0x7f4f2c071d10 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:20.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:20.139+0000 7f4f34000640 1 -- 172.21.15.16:0/2013060739 >> 172.21.15.16:0/2013060739 conn(0x7f4f2c06b620 msgr2=0x7f4f2c06ba30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:12:20.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:20.139+0000 7f4f34000640 1 -- 172.21.15.16:0/2013060739 shutdown_connections 2024-03-31T15:12:20.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:20.139+0000 7f4f34000640 1 -- 172.21.15.16:0/2013060739 wait complete. 2024-03-31T15:12:20.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:20.139+0000 7f4f34000640 1 Processor -- start 2024-03-31T15:12:20.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:20.139+0000 7f4f34000640 1 -- start start 2024-03-31T15:12:20.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:20.143+0000 7f4f34000640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f4f2c0b23d0 0x7f4f2c0aff90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:20.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:20.143+0000 7f4f34000640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4f2c15b3d0 0x7f4f2c148140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:20.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:20.143+0000 7f4f34000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4f2c15eee0 0x7f4f2c148680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:20.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:20.143+0000 7f4f34000640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f4f2c073580 con 0x7f4f2c15eee0 2024-03-31T15:12:20.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:20.143+0000 7f4f34000640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f4f2c0b45a0 con 0x7f4f2c15b3d0 2024-03-31T15:12:20.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:20.143+0000 7f4f34000640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f4f2c073700 con 0x7f4f2c0b23d0 2024-03-31T15:12:20.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:20.143+0000 7f4f32000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4f2c15eee0 0x7f4f2c148680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:20.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:20.143+0000 7f4f32000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4f2c15eee0 0x7f4f2c148680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40232/0 (socket says 172.21.15.16:40232) 2024-03-31T15:12:20.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:20.143+0000 7f4f32000640 1 -- 172.21.15.16:0/4200879290 learned_addr learned my addr 172.21.15.16:0/4200879290 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:12:20.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:20.143+0000 7f4f31600640 1 --2- 172.21.15.16:0/4200879290 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f4f2c0b23d0 0x7f4f2c0aff90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:20.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:20.143+0000 7f4f30c00640 1 --2- 172.21.15.16:0/4200879290 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4f2c15b3d0 0x7f4f2c148140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:20.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:20.143+0000 7f4f32000640 1 -- 172.21.15.16:0/4200879290 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f4f2c0b23d0 msgr2=0x7f4f2c0aff90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:20.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:20.143+0000 7f4f32000640 1 --2- 172.21.15.16:0/4200879290 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f4f2c0b23d0 0x7f4f2c0aff90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:20.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:20.143+0000 7f4f32000640 1 -- 172.21.15.16:0/4200879290 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4f2c15b3d0 msgr2=0x7f4f2c148140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:20.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:20.143+0000 7f4f32000640 1 --2- 172.21.15.16:0/4200879290 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4f2c15b3d0 0x7f4f2c148140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:20.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:20.143+0000 7f4f32000640 1 -- 172.21.15.16:0/4200879290 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f2c148f00 con 0x7f4f2c15eee0 2024-03-31T15:12:20.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:20.143+0000 7f4f30c00640 1 --2- 172.21.15.16:0/4200879290 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4f2c15b3d0 0x7f4f2c148140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:12:20.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:20.143+0000 7f4f31600640 1 --2- 172.21.15.16:0/4200879290 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f4f2c0b23d0 0x7f4f2c0aff90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:12:20.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:20.143+0000 7f4f32000640 1 --2- 172.21.15.16:0/4200879290 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4f2c15eee0 0x7f4f2c148680 secure :-1 s=READY pgs=1405 cs=0 l=1 rev1=1 crypto rx=0x7f4f2400c790 tx=0x7f4f2400cc50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:12:20.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:20.143+0000 7f4f29600640 1 -- 172.21.15.16:0/4200879290 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f24003b80 con 0x7f4f2c15eee0 2024-03-31T15:12:20.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:20.143+0000 7f4f29600640 1 -- 172.21.15.16:0/4200879290 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4f24003d20 con 0x7f4f2c15eee0 2024-03-31T15:12:20.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:20.143+0000 7f4f34000640 1 -- 172.21.15.16:0/4200879290 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4f2c1491e0 con 0x7f4f2c15eee0 2024-03-31T15:12:20.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:20.143+0000 7f4f29600640 1 -- 172.21.15.16:0/4200879290 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f2400aa50 con 0x7f4f2c15eee0 2024-03-31T15:12:20.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:20.143+0000 7f4f34000640 1 -- 172.21.15.16:0/4200879290 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4f2c148bc0 con 0x7f4f2c15eee0 2024-03-31T15:12:20.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:20.143+0000 7f4f34000640 1 -- 172.21.15.16:0/4200879290 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4f2c069950 con 0x7f4f2c15eee0 2024-03-31T15:12:20.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:20.143+0000 7f4f29600640 1 -- 172.21.15.16:0/4200879290 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f4f2400abf0 con 0x7f4f2c15eee0 2024-03-31T15:12:20.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:20.147+0000 7f4f29600640 1 --2- 172.21.15.16:0/4200879290 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f4f0c068380 0x7f4f0c06a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:20.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:20.147+0000 7f4f29600640 1 -- 172.21.15.16:0/4200879290 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 5709+0+0 (secure 0 0 0) 0x7f4f240869a0 con 0x7f4f2c15eee0 2024-03-31T15:12:20.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:20.147+0000 7f4f31600640 1 --2- 172.21.15.16:0/4200879290 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f4f0c068380 0x7f4f0c06a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:20.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:20.147+0000 7f4f31600640 1 --2- 172.21.15.16:0/4200879290 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f4f0c068380 0x7f4f0c06a840 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f4f200011f0 tx=0x7f4f20009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:12:20.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:20.151+0000 7f4f29600640 1 -- 172.21.15.16:0/4200879290 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f4f24056f50 con 0x7f4f2c15eee0 2024-03-31T15:12:20.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:20.347+0000 7f4f34000640 1 -- 172.21.15.16:0/4200879290 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4f2c0cdea0 con 0x7f4f2c15eee0 2024-03-31T15:12:20.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:20.351+0000 7f4f29600640 1 -- 172.21.15.16:0/4200879290 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 83 v83) ==== 94+0+5180 (secure 0 0 0) 0x7f4f24056850 con 0x7f4f2c15eee0 2024-03-31T15:12:20.342 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:12:20.343 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":83,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7139,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6836/3798928623","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":3798928623},{"type":"v1","addr":"172.21.15.89:6836","nonce":3798928623}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":78},{"gid":7142,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/3074283817","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6835","nonce":3074283817},{"type":"v1","addr":"172.21.15.89:6837","nonce":3074283817}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":78},{"gid":8071,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6834/2110189998","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":2110189998},{"type":"v1","addr":"172.21.15.16:6834","nonce":2110189998}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":78}],"filesystems":[{"mdsmap":{"epoch":83,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-31T15:12:16.294600+0000","modified":"2024-03-31T15:12:19.297339+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8074},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8074":{"gid":8074,"name":"a","rank":0,"incarnation":79,"state":"up:active","state_seq":3,"addr":"172.21.15.16:6835/490717480","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6833","nonce":490717480},{"type":"v1","addr":"172.21.15.16:6835","nonce":490717480}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8074,"qdb_cluster":[8074]},"id":7}]} 2024-03-31T15:12:20.343 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 83 2024-03-31T15:12:20.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:20.351+0000 7f4f34000640 1 -- 172.21.15.16:0/4200879290 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f4f0c068380 msgr2=0x7f4f0c06a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:20.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:20.351+0000 7f4f34000640 1 --2- 172.21.15.16:0/4200879290 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f4f0c068380 0x7f4f0c06a840 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f4f200011f0 tx=0x7f4f20009040 comp rx=0 tx=0).stop 2024-03-31T15:12:20.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:20.351+0000 7f4f34000640 1 -- 172.21.15.16:0/4200879290 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4f2c15eee0 msgr2=0x7f4f2c148680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:20.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:20.351+0000 7f4f34000640 1 --2- 172.21.15.16:0/4200879290 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4f2c15eee0 0x7f4f2c148680 secure :-1 s=READY pgs=1405 cs=0 l=1 rev1=1 crypto rx=0x7f4f2400c790 tx=0x7f4f2400cc50 comp rx=0 tx=0).stop 2024-03-31T15:12:20.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:20.351+0000 7f4f34000640 1 -- 172.21.15.16:0/4200879290 shutdown_connections 2024-03-31T15:12:20.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:20.351+0000 7f4f34000640 1 --2- 172.21.15.16:0/4200879290 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f4f0c068380 0x7f4f0c06a840 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:20.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:20.351+0000 7f4f34000640 1 --2- 172.21.15.16:0/4200879290 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4f2c15eee0 0x7f4f2c148680 unknown :-1 s=CLOSED pgs=1405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:20.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:20.351+0000 7f4f34000640 1 --2- 172.21.15.16:0/4200879290 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4f2c15b3d0 0x7f4f2c148140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:20.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:20.351+0000 7f4f34000640 1 --2- 172.21.15.16:0/4200879290 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f4f2c0b23d0 0x7f4f2c0aff90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:20.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:20.351+0000 7f4f34000640 1 -- 172.21.15.16:0/4200879290 >> 172.21.15.16:0/4200879290 conn(0x7f4f2c06b620 msgr2=0x7f4f2c153340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:12:20.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:20.351+0000 7f4f34000640 1 -- 172.21.15.16:0/4200879290 shutdown_connections 2024-03-31T15:12:20.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:20.351+0000 7f4f34000640 1 -- 172.21.15.16:0/4200879290 wait complete. 2024-03-31T15:12:20.364 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-31T15:12:20.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:20.511+0000 7fdb10000640 1 Processor -- start 2024-03-31T15:12:20.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:20.511+0000 7fdb10000640 1 -- start start 2024-03-31T15:12:20.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:20.511+0000 7fdb10000640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fdb0806dbc0 0x7fdb0815f650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:20.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:20.511+0000 7fdb10000640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fdb08070030 0x7fdb08162180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:20.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:20.511+0000 7fdb10000640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdb080cbe90 con 0x7fdb080cb440 2024-03-31T15:12:20.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:20.511+0000 7fdb10000640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7fdb0815fb90 con 0x7fdb0806dbc0 2024-03-31T15:12:20.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:20.511+0000 7fdb10000640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fdb0815f1c0 con 0x7fdb08070030 2024-03-31T15:12:20.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:20.511+0000 7fdb0e000640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fdb08070030 0x7fdb08162180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:20.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:20.511+0000 7fdb0e000640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fdb08070030 0x7fdb08162180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3301/0 says I am v2:172.21.15.16:57510/0 (socket says 172.21.15.16:57510) 2024-03-31T15:12:20.507 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:20.511+0000 7fdb0e000640 1 -- 172.21.15.16:0/3980887826 learned_addr learned my addr 172.21.15.16:0/3980887826 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:12:20.507 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:20.511+0000 7fdb0d600640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fdb0806dbc0 0x7fdb0815f650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:20.507 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:20.515+0000 7fdb0e000640 1 -- 172.21.15.16:0/3980887826 >> v2:172.21.15.89:3300/0 conn(0x7fdb0806dbc0 msgr2=0x7fdb0815f650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:12:20.507 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:20.515+0000 7fdb0e000640 1 --2- 172.21.15.16:0/3980887826 >> v2:172.21.15.89:3300/0 conn(0x7fdb0806dbc0 0x7fdb0815f650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:20.507 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:20.515+0000 7fdb0e000640 1 -- 172.21.15.16:0/3980887826 >> v1:172.21.15.89:6789/0 conn(0x7fdb080cb440 legacy=0x7fdb0806f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:20.507 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:20.515+0000 7fdb0e000640 1 -- 172.21.15.16:0/3980887826 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb080cc170 con 0x7fdb08070030 2024-03-31T15:12:20.508 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:20.515+0000 7fdb0e000640 1 --2- 172.21.15.16:0/3980887826 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fdb08070030 0x7fdb08162180 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fdb00009e50 tx=0x7fdb0001c970 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f3c45e5bf31026c6 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:12:20.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:20.515+0000 7fdb07e00640 1 -- 172.21.15.16:0/3980887826 <== mon.2 v2:172.21.15.89:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb000a5740 con 0x7fdb08070030 2024-03-31T15:12:20.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:20.515+0000 7fdb07e00640 1 -- 172.21.15.16:0/3980887826 <== mon.2 v2:172.21.15.89:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdb000a58e0 con 0x7fdb08070030 2024-03-31T15:12:20.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:20.515+0000 7fdb07e00640 1 -- 172.21.15.16:0/3980887826 <== mon.2 v2:172.21.15.89:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb000a5cb0 con 0x7fdb08070030 2024-03-31T15:12:20.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:20.515+0000 7fdb10000640 1 -- 172.21.15.16:0/3980887826 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fdb08070030 msgr2=0x7fdb08162180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:20.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:20.515+0000 7fdb10000640 1 --2- 172.21.15.16:0/3980887826 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fdb08070030 0x7fdb08162180 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fdb00009e50 tx=0x7fdb0001c970 comp rx=0 tx=0).stop 2024-03-31T15:12:20.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:20.519+0000 7fdb10000640 1 -- 172.21.15.16:0/3980887826 shutdown_connections 2024-03-31T15:12:20.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:20.519+0000 7fdb10000640 1 --2- 172.21.15.16:0/3980887826 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fdb08070030 0x7fdb08162180 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:20.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:20.519+0000 7fdb10000640 1 --2- 172.21.15.16:0/3980887826 >> v2:172.21.15.89:3300/0 conn(0x7fdb0806dbc0 0x7fdb0815f650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:20.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:20.519+0000 7fdb10000640 1 -- 172.21.15.16:0/3980887826 >> 172.21.15.16:0/3980887826 conn(0x7fdb08067330 msgr2=0x7fdb08067740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:12:20.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:20.519+0000 7fdb10000640 1 -- 172.21.15.16:0/3980887826 shutdown_connections 2024-03-31T15:12:20.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:20.519+0000 7fdb10000640 1 -- 172.21.15.16:0/3980887826 wait complete. 2024-03-31T15:12:20.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:20.519+0000 7fdb10000640 1 Processor -- start 2024-03-31T15:12:20.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:20.519+0000 7fdb10000640 1 -- start start 2024-03-31T15:12:20.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:20.519+0000 7fdb10000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdb0806dbc0 0x7fdb08157c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:20.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:20.519+0000 7fdb10000640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdb08070030 0x7fdb08163c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:20.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:20.519+0000 7fdb10000640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fdb080cb440 0x7fdb08166b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:20.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:20.519+0000 7fdb10000640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fdb0806cad0 con 0x7fdb0806dbc0 2024-03-31T15:12:20.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:20.519+0000 7fdb10000640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fdb0806c950 con 0x7fdb08070030 2024-03-31T15:12:20.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:20.519+0000 7fdb10000640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fdb0816b520 con 0x7fdb080cb440 2024-03-31T15:12:20.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:20.519+0000 7fdb0d600640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdb0806dbc0 0x7fdb08157c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:20.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:20.519+0000 7fdb0d600640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdb0806dbc0 0x7fdb08157c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40240/0 (socket says 172.21.15.16:40240) 2024-03-31T15:12:20.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:20.519+0000 7fdb0d600640 1 -- 172.21.15.16:0/2261161866 learned_addr learned my addr 172.21.15.16:0/2261161866 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:12:20.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:20.519+0000 7fdb0cc00640 1 --2- 172.21.15.16:0/2261161866 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdb08070030 0x7fdb08163c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:20.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:20.519+0000 7fdb0e000640 1 --2- 172.21.15.16:0/2261161866 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fdb080cb440 0x7fdb08166b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:20.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:20.519+0000 7fdb0d600640 1 -- 172.21.15.16:0/2261161866 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fdb080cb440 msgr2=0x7fdb08166b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:20.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:20.519+0000 7fdb0d600640 1 --2- 172.21.15.16:0/2261161866 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fdb080cb440 0x7fdb08166b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:20.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:20.519+0000 7fdb0d600640 1 -- 172.21.15.16:0/2261161866 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdb08070030 msgr2=0x7fdb08163c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:20.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:20.519+0000 7fdb0d600640 1 --2- 172.21.15.16:0/2261161866 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdb08070030 0x7fdb08163c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:20.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:20.519+0000 7fdb0d600640 1 -- 172.21.15.16:0/2261161866 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb08164660 con 0x7fdb0806dbc0 2024-03-31T15:12:20.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:20.519+0000 7fdb0d600640 1 --2- 172.21.15.16:0/2261161866 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdb0806dbc0 0x7fdb08157c40 secure :-1 s=READY pgs=1406 cs=0 l=1 rev1=1 crypto rx=0x7fdafc00e8a0 tx=0x7fdafc00eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:12:20.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:20.519+0000 7fdb05600640 1 -- 172.21.15.16:0/2261161866 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdafc017070 con 0x7fdb0806dbc0 2024-03-31T15:12:20.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:20.523+0000 7fdb05600640 1 -- 172.21.15.16:0/2261161866 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdafc00c940 con 0x7fdb0806dbc0 2024-03-31T15:12:20.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:20.523+0000 7fdb10000640 1 -- 172.21.15.16:0/2261161866 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdb08164940 con 0x7fdb0806dbc0 2024-03-31T15:12:20.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:20.523+0000 7fdb05600640 1 -- 172.21.15.16:0/2261161866 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdafc00ccf0 con 0x7fdb0806dbc0 2024-03-31T15:12:20.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:20.523+0000 7fdb10000640 1 -- 172.21.15.16:0/2261161866 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdb08164f00 con 0x7fdb0806dbc0 2024-03-31T15:12:20.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:20.523+0000 7fdb10000640 1 -- 172.21.15.16:0/2261161866 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdb08069950 con 0x7fdb0806dbc0 2024-03-31T15:12:20.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:20.523+0000 7fdb05600640 1 -- 172.21.15.16:0/2261161866 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fdafc01a070 con 0x7fdb0806dbc0 2024-03-31T15:12:20.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:20.527+0000 7fdb05600640 1 --2- 172.21.15.16:0/2261161866 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fdae80683b0 0x7fdae806a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:20.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:20.527+0000 7fdb05600640 1 -- 172.21.15.16:0/2261161866 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 5709+0+0 (secure 0 0 0) 0x7fdafc086be0 con 0x7fdb0806dbc0 2024-03-31T15:12:20.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:20.527+0000 7fdb0cc00640 1 --2- 172.21.15.16:0/2261161866 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fdae80683b0 0x7fdae806a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:20.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:20.527+0000 7fdb0cc00640 1 --2- 172.21.15.16:0/2261161866 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fdae80683b0 0x7fdae806a870 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7fdb080cdd60 tx=0x7fdaf4009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:12:20.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:20.531+0000 7fdb05600640 1 -- 172.21.15.16:0/2261161866 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fdafc056ac0 con 0x7fdb0806dbc0 2024-03-31T15:12:20.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:20.695+0000 7fdb10000640 1 -- 172.21.15.16:0/2261161866 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fdb080cf380 con 0x7fdb0806dbc0 2024-03-31T15:12:20.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:20.699+0000 7fdb05600640 1 -- 172.21.15.16:0/2261161866 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v96) ==== 74+0+16536 (secure 0 0 0) 0x7fdafc05adc0 con 0x7fdb0806dbc0 2024-03-31T15:12:20.690 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:12:20.690 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":96,"fsid":"286eb5da-fb47-4550-b290-d9519c13b9a5","created":"2024-03-31T15:03:11.438952+0000","modified":"2024-03-31T15:12:17.885852+0000","last_up_change":"2024-03-31T15:03:22.885535+0000","last_in_change":"2024-03-31T15:03:13.941667+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-31T15:03:22.550787+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2024-03-31T15:12:14.671225+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"95","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2024-03-31T15:12:15.242196+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"96","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"92a66450-ef19-40d8-ac3d-d760b23b0bda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6801","nonce":932712244}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6803","nonce":932712244}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6807","nonce":932712244}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6805","nonce":932712244}]},"public_addr":"172.21.15.16:6801/932712244","cluster_addr":"172.21.15.16:6803/932712244","heartbeat_back_addr":"172.21.15.16:6807/932712244","heartbeat_front_addr":"172.21.15.16:6805/932712244","state":["exists","up"]},{"osd":1,"uuid":"da9d1f2b-1968-4959-b843-faa231988e92","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6810","nonce":1398737533}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6811","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6813","nonce":1398737533}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6821","nonce":1398737533}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6815","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6816","nonce":1398737533}]},"public_addr":"172.21.15.16:6810/1398737533","cluster_addr":"172.21.15.16:6813/1398737533","heartbeat_back_addr":"172.21.15.16:6821/1398737533","heartbeat_front_addr":"172.21.15.16:6816/1398737533","state":["exists","up"]},{"osd":2,"uuid":"01a73662-9a21-4b1f-bcbb-42b9636a4638","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6819","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6822","nonce":3018526646}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6825","nonce":3018526646}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6831","nonce":3018526646}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6827","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6829","nonce":3018526646}]},"public_addr":"172.21.15.16:6822/3018526646","cluster_addr":"172.21.15.16:6825/3018526646","heartbeat_back_addr":"172.21.15.16:6831/3018526646","heartbeat_front_addr":"172.21.15.16:6829/3018526646","state":["exists","up"]},{"osd":3,"uuid":"c7771fd3-e18e-4500-ac38-23cf10ebcd75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6809","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6812","nonce":3360352974}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6817","nonce":3360352974}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6828","nonce":3360352974}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6823","nonce":3360352974}]},"public_addr":"172.21.15.16:6812/3360352974","cluster_addr":"172.21.15.16:6817/3360352974","heartbeat_back_addr":"172.21.15.16:6828/3360352974","heartbeat_front_addr":"172.21.15.16:6823/3360352974","state":["exists","up"]},{"osd":4,"uuid":"88260730-1482-4fbd-a40f-cb1124e9df4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6801","nonce":1856844856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6803","nonce":1856844856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6807","nonce":1856844856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6805","nonce":1856844856}]},"public_addr":"172.21.15.89:6801/1856844856","cluster_addr":"172.21.15.89:6803/1856844856","heartbeat_back_addr":"172.21.15.89:6807/1856844856","heartbeat_front_addr":"172.21.15.89:6805/1856844856","state":["exists","up"]},{"osd":5,"uuid":"4cb2c00c-fae0-4dd8-9d27-ec4f0093ccf7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6814","nonce":50395527}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6817","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6820","nonce":50395527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6831","nonce":50395527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6823","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6826","nonce":50395527}]},"public_addr":"172.21.15.89:6814/50395527","cluster_addr":"172.21.15.89:6820/50395527","heartbeat_back_addr":"172.21.15.89:6831/50395527","heartbeat_front_addr":"172.21.15.89:6826/50395527","state":["exists","up"]},{"osd":6,"uuid":"81ce0c19-2554-4cae-acbe-66a05e9e9b9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6809","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6812","nonce":3470471752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6815","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6818","nonce":3470471752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6827","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6829","nonce":3470471752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6821","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6824","nonce":3470471752}]},"public_addr":"172.21.15.89:6812/3470471752","cluster_addr":"172.21.15.89:6818/3470471752","heartbeat_back_addr":"172.21.15.89:6829/3470471752","heartbeat_front_addr":"172.21.15.89:6824/3470471752","state":["exists","up"]},{"osd":7,"uuid":"85e4313b-d37d-442e-8042-153bf20a847d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6811","nonce":1796778630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6813","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6816","nonce":1796778630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6825","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6828","nonce":1796778630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6819","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6822","nonce":1796778630}]},"public_addr":"172.21.15.89:6811/1796778630","cluster_addr":"172.21.15.89:6816/1796778630","heartbeat_back_addr":"172.21.15.89:6828/1796778630","heartbeat_front_addr":"172.21.15.89:6822/1796778630","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.439088+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.529970+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.680512+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:21.044161+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.582111+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.536558+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.409522+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.590229+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-31T15:12:20.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:20.699+0000 7fdb10000640 1 -- 172.21.15.16:0/2261161866 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fdae80683b0 msgr2=0x7fdae806a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:20.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:20.699+0000 7fdb10000640 1 --2- 172.21.15.16:0/2261161866 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fdae80683b0 0x7fdae806a870 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7fdb080cdd60 tx=0x7fdaf4009040 comp rx=0 tx=0).stop 2024-03-31T15:12:20.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:20.699+0000 7fdb10000640 1 -- 172.21.15.16:0/2261161866 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdb0806dbc0 msgr2=0x7fdb08157c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:20.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:20.699+0000 7fdb10000640 1 --2- 172.21.15.16:0/2261161866 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdb0806dbc0 0x7fdb08157c40 secure :-1 s=READY pgs=1406 cs=0 l=1 rev1=1 crypto rx=0x7fdafc00e8a0 tx=0x7fdafc00eca0 comp rx=0 tx=0).stop 2024-03-31T15:12:20.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:20.699+0000 7fdb10000640 1 -- 172.21.15.16:0/2261161866 shutdown_connections 2024-03-31T15:12:20.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:20.699+0000 7fdb10000640 1 --2- 172.21.15.16:0/2261161866 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fdae80683b0 0x7fdae806a870 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:20.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:20.699+0000 7fdb10000640 1 --2- 172.21.15.16:0/2261161866 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fdb080cb440 0x7fdb08166b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:20.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:20.699+0000 7fdb10000640 1 --2- 172.21.15.16:0/2261161866 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdb08070030 0x7fdb08163c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:20.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:20.699+0000 7fdb10000640 1 --2- 172.21.15.16:0/2261161866 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdb0806dbc0 0x7fdb08157c40 unknown :-1 s=CLOSED pgs=1406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:20.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:20.699+0000 7fdb10000640 1 -- 172.21.15.16:0/2261161866 >> 172.21.15.16:0/2261161866 conn(0x7fdb08067330 msgr2=0x7fdb08160db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:12:20.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:20.699+0000 7fdb10000640 1 -- 172.21.15.16:0/2261161866 shutdown_connections 2024-03-31T15:12:20.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:20.699+0000 7fdb10000640 1 -- 172.21.15.16:0/2261161866 wait complete. 2024-03-31T15:12:20.711 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-31T15:12:20.711 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-31T15:12:20.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:20.859+0000 7fb528a00640 1 Processor -- start 2024-03-31T15:12:20.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:20.859+0000 7fb528a00640 1 -- start start 2024-03-31T15:12:20.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:20.859+0000 7fb528a00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fb52406fd80 0x7fb5240d2720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:20.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:20.859+0000 7fb528a00640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb5240d0920 con 0x7fb52406f920 2024-03-31T15:12:20.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:20.859+0000 7fb528a00640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb5240d0c00 con 0x7fb524145e40 2024-03-31T15:12:20.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:20.859+0000 7fb528a00640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7fb5240d06d0 con 0x7fb52406fd80 2024-03-31T15:12:20.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:20.859+0000 7fb51be00640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fb52406f920 0x7fb5240d2010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:54944/0 (socket says 172.21.15.16:54944) 2024-03-31T15:12:20.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:20.859+0000 7fb51be00640 1 -- 172.21.15.16:0/2557409208 learned_addr learned my addr 172.21.15.16:0/2557409208 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:12:20.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:20.859+0000 7fb522800640 1 --2- 172.21.15.16:0/2557409208 >> v2:172.21.15.89:3300/0 conn(0x7fb52406fd80 0x7fb5240d2720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:20.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:20.859+0000 7fb521400640 1 -- 172.21.15.16:0/2557409208 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4276373795 0 0) 0x7fb5240d0920 con 0x7fb52406f920 2024-03-31T15:12:20.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:20.859+0000 7fb521400640 1 -- 172.21.15.16:0/2557409208 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb508003530 con 0x7fb52406f920 2024-03-31T15:12:20.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:20.863+0000 7fb521400640 1 -- 172.21.15.16:0/2557409208 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4076969304 0 0) 0x7fb508003530 con 0x7fb52406f920 2024-03-31T15:12:20.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:20.863+0000 7fb521400640 1 -- 172.21.15.16:0/2557409208 >> v2:172.21.15.89:3300/0 conn(0x7fb52406fd80 msgr2=0x7fb5240d2720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:12:20.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:20.863+0000 7fb521400640 1 --2- 172.21.15.16:0/2557409208 >> v2:172.21.15.89:3300/0 conn(0x7fb52406fd80 0x7fb5240d2720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:20.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:20.863+0000 7fb521400640 1 -- 172.21.15.16:0/2557409208 >> v1:172.21.15.89:6789/0 conn(0x7fb524145e40 legacy=0x7fb5240cc740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:20.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:20.863+0000 7fb521400640 1 -- 172.21.15.16:0/2557409208 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5240d1810 con 0x7fb52406f920 2024-03-31T15:12:20.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:20.863+0000 7fb521400640 1 -- 172.21.15.16:0/2557409208 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 214807623 0 0) 0x7fb50c001680 con 0x7fb52406f920 2024-03-31T15:12:20.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:20.863+0000 7fb521400640 1 -- 172.21.15.16:0/2557409208 >> v1:172.21.15.16:6789/0 conn(0x7fb52406f920 legacy=0x7fb5240d2010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:20.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:20.863+0000 7fb521400640 1 --2- 172.21.15.16:0/2557409208 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb5080039c0 0x7fb508003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:20.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:20.863+0000 7fb521400640 1 --2- 172.21.15.16:0/2557409208 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb508004810 0x7fb508004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:20.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:20.863+0000 7fb521400640 1 --2- 172.21.15.16:0/2557409208 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb5080051d0 0x7fb50800b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:20.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:20.863+0000 7fb521400640 1 -- 172.21.15.16:0/2557409208 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fb5240d06d0 con 0x7fb5080039c0 2024-03-31T15:12:20.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:20.863+0000 7fb521400640 1 -- 172.21.15.16:0/2557409208 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fb50800c070 con 0x7fb508004810 2024-03-31T15:12:20.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:20.863+0000 7fb521400640 1 -- 172.21.15.16:0/2557409208 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fb50800c270 con 0x7fb5080051d0 2024-03-31T15:12:20.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:20.863+0000 7fb521e00640 1 --2- 172.21.15.16:0/2557409208 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb5080039c0 0x7fb508003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:20.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:20.863+0000 7fb51be00640 1 --2- 172.21.15.16:0/2557409208 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb508004810 0x7fb508004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:20.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:20.863+0000 7fb522800640 1 --2- 172.21.15.16:0/2557409208 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb5080051d0 0x7fb50800b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:20.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:20.863+0000 7fb521e00640 1 -- 172.21.15.16:0/2557409208 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb5080051d0 msgr2=0x7fb50800b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:12:20.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:20.863+0000 7fb521e00640 1 --2- 172.21.15.16:0/2557409208 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb5080051d0 0x7fb50800b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:20.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:20.863+0000 7fb521e00640 1 -- 172.21.15.16:0/2557409208 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb508004810 msgr2=0x7fb508004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:12:20.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:20.863+0000 7fb521e00640 1 --2- 172.21.15.16:0/2557409208 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb508004810 0x7fb508004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:20.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:20.863+0000 7fb521e00640 1 -- 172.21.15.16:0/2557409208 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb508003530 con 0x7fb5080039c0 2024-03-31T15:12:20.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:20.863+0000 7fb522800640 1 -- 172.21.15.16:0/2557409208 reap_dead start 2024-03-31T15:12:20.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:20.863+0000 7fb521e00640 1 --2- 172.21.15.16:0/2557409208 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb5080039c0 0x7fb508003de0 secure :-1 s=READY pgs=1408 cs=0 l=1 rev1=1 crypto rx=0x7fb514001780 tx=0x7fb51401c420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b8149065bbe2fb7 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:12:20.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:20.867+0000 7fb521400640 1 -- 172.21.15.16:0/2557409208 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5140013a0 con 0x7fb5080039c0 2024-03-31T15:12:20.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:20.867+0000 7fb521400640 1 -- 172.21.15.16:0/2557409208 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb5140052f0 con 0x7fb5080039c0 2024-03-31T15:12:20.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:20.867+0000 7fb521400640 1 -- 172.21.15.16:0/2557409208 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb514003840 con 0x7fb5080039c0 2024-03-31T15:12:20.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:20.867+0000 7fb528a00640 1 -- 172.21.15.16:0/2557409208 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb5080039c0 msgr2=0x7fb508003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:20.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:20.867+0000 7fb528a00640 1 --2- 172.21.15.16:0/2557409208 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb5080039c0 0x7fb508003de0 secure :-1 s=READY pgs=1408 cs=0 l=1 rev1=1 crypto rx=0x7fb514001780 tx=0x7fb51401c420 comp rx=0 tx=0).stop 2024-03-31T15:12:20.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:20.867+0000 7fb528a00640 1 -- 172.21.15.16:0/2557409208 shutdown_connections 2024-03-31T15:12:20.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:20.867+0000 7fb528a00640 1 --2- 172.21.15.16:0/2557409208 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb5080039c0 0x7fb508003de0 unknown :-1 s=CLOSED pgs=1408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:20.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:20.867+0000 7fb528a00640 1 -- 172.21.15.16:0/2557409208 >> 172.21.15.16:0/2557409208 conn(0x7fb524067330 msgr2=0x7fb524067740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:12:20.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:20.867+0000 7fb528a00640 1 -- 172.21.15.16:0/2557409208 shutdown_connections 2024-03-31T15:12:20.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:20.867+0000 7fb528a00640 1 -- 172.21.15.16:0/2557409208 wait complete. 2024-03-31T15:12:20.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:20.867+0000 7fb528a00640 1 Processor -- start 2024-03-31T15:12:20.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:20.867+0000 7fb528a00640 1 -- start start 2024-03-31T15:12:20.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:20.867+0000 7fb528a00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb5080039c0 0x7fb5241fb190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:20.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:20.867+0000 7fb528a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb5240a9bb0 0x7fb524143be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:20.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:20.867+0000 7fb528a00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb52413e850 0x7fb52413ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:20.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:20.867+0000 7fb528a00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fb5240b50c0 con 0x7fb5240a9bb0 2024-03-31T15:12:20.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:20.867+0000 7fb528a00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fb5240d0ee0 con 0x7fb52413e850 2024-03-31T15:12:20.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:20.867+0000 7fb528a00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fb5240b5240 con 0x7fb5080039c0 2024-03-31T15:12:20.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:20.867+0000 7fb51be00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb5240a9bb0 0x7fb524143be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:20.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:20.867+0000 7fb51be00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb5240a9bb0 0x7fb524143be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40260/0 (socket says 172.21.15.16:40260) 2024-03-31T15:12:20.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:20.867+0000 7fb521e00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb5080039c0 0x7fb5241fb190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:20.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:20.867+0000 7fb51be00640 1 -- 172.21.15.16:0/75437444 learned_addr learned my addr 172.21.15.16:0/75437444 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:12:20.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:20.867+0000 7fb522800640 1 --2- 172.21.15.16:0/75437444 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb52413e850 0x7fb52413ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:20.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:20.871+0000 7fb51be00640 1 -- 172.21.15.16:0/75437444 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb5080039c0 msgr2=0x7fb5241fb190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:20.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:20.871+0000 7fb51be00640 1 --2- 172.21.15.16:0/75437444 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb5080039c0 0x7fb5241fb190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:20.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:20.871+0000 7fb51be00640 1 -- 172.21.15.16:0/75437444 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb52413e850 msgr2=0x7fb52413ecd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:20.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:20.871+0000 7fb51be00640 1 --2- 172.21.15.16:0/75437444 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb52413e850 0x7fb52413ecd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:20.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:20.871+0000 7fb51be00640 1 -- 172.21.15.16:0/75437444 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5241587d0 con 0x7fb5240a9bb0 2024-03-31T15:12:20.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:20.871+0000 7fb51be00640 1 --2- 172.21.15.16:0/75437444 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb5240a9bb0 0x7fb524143be0 secure :-1 s=READY pgs=1409 cs=0 l=1 rev1=1 crypto rx=0x7fb50c0072f0 tx=0x7fb50c0077b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:12:20.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:20.871+0000 7fb51a000640 1 -- 172.21.15.16:0/75437444 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb50c00f8c0 con 0x7fb5240a9bb0 2024-03-31T15:12:20.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:20.871+0000 7fb51a000640 1 -- 172.21.15.16:0/75437444 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb50c018070 con 0x7fb5240a9bb0 2024-03-31T15:12:20.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:20.871+0000 7fb528a00640 1 -- 172.21.15.16:0/75437444 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb524158ab0 con 0x7fb5240a9bb0 2024-03-31T15:12:20.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:20.871+0000 7fb51a000640 1 -- 172.21.15.16:0/75437444 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb50c00fcc0 con 0x7fb5240a9bb0 2024-03-31T15:12:20.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:20.871+0000 7fb528a00640 1 -- 172.21.15.16:0/75437444 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb524159070 con 0x7fb5240a9bb0 2024-03-31T15:12:20.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:20.871+0000 7fb528a00640 1 -- 172.21.15.16:0/75437444 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb524069950 con 0x7fb5240a9bb0 2024-03-31T15:12:20.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:20.871+0000 7fb51a000640 1 -- 172.21.15.16:0/75437444 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fb50c01d460 con 0x7fb5240a9bb0 2024-03-31T15:12:20.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:20.875+0000 7fb51a000640 1 --2- 172.21.15.16:0/75437444 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fb500068380 0x7fb50006a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:20.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:20.875+0000 7fb51a000640 1 -- 172.21.15.16:0/75437444 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 5709+0+0 (secure 0 0 0) 0x7fb50c086900 con 0x7fb5240a9bb0 2024-03-31T15:12:20.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:20.875+0000 7fb521e00640 1 --2- 172.21.15.16:0/75437444 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fb500068380 0x7fb50006a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:20.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:20.875+0000 7fb521e00640 1 --2- 172.21.15.16:0/75437444 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fb500068380 0x7fb50006a840 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7fb514001750 tx=0x7fb5140011a0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:12:20.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:20.875+0000 7fb51a000640 1 -- 172.21.15.16:0/75437444 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fb50c00fe60 con 0x7fb5240a9bb0 2024-03-31T15:12:21.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:21.043+0000 7fb528a00640 1 -- 172.21.15.16:0/75437444 --> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fb524158580 con 0x7fb500068380 2024-03-31T15:12:21.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:21.047+0000 7fb51a000640 1 -- 172.21.15.16:0/75437444 <== mgr.4100 v2:172.21.15.89:6832/1549407273 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252040 (secure 0 0 0) 0x7fb524158580 con 0x7fb500068380 2024-03-31T15:12:21.044 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:12:21.046 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-03-31T15:12:21.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:21.051+0000 7fb528a00640 1 -- 172.21.15.16:0/75437444 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fb500068380 msgr2=0x7fb50006a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:21.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:21.051+0000 7fb528a00640 1 --2- 172.21.15.16:0/75437444 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fb500068380 0x7fb50006a840 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7fb514001750 tx=0x7fb5140011a0 comp rx=0 tx=0).stop 2024-03-31T15:12:21.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:21.051+0000 7fb528a00640 1 -- 172.21.15.16:0/75437444 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb5240a9bb0 msgr2=0x7fb524143be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:21.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:21.051+0000 7fb528a00640 1 --2- 172.21.15.16:0/75437444 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb5240a9bb0 0x7fb524143be0 secure :-1 s=READY pgs=1409 cs=0 l=1 rev1=1 crypto rx=0x7fb50c0072f0 tx=0x7fb50c0077b0 comp rx=0 tx=0).stop 2024-03-31T15:12:21.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:21.051+0000 7fb528a00640 1 -- 172.21.15.16:0/75437444 shutdown_connections 2024-03-31T15:12:21.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:21.051+0000 7fb528a00640 1 --2- 172.21.15.16:0/75437444 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fb500068380 0x7fb50006a840 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:21.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:21.051+0000 7fb528a00640 1 --2- 172.21.15.16:0/75437444 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb52413e850 0x7fb52413ecd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:21.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:21.051+0000 7fb528a00640 1 --2- 172.21.15.16:0/75437444 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb5240a9bb0 0x7fb524143be0 unknown :-1 s=CLOSED pgs=1409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:21.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:21.051+0000 7fb528a00640 1 --2- 172.21.15.16:0/75437444 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb5080039c0 0x7fb5241fb190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:21.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:21.051+0000 7fb528a00640 1 -- 172.21.15.16:0/75437444 >> 172.21.15.16:0/75437444 conn(0x7fb524067330 msgr2=0x7fb5240cbd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:12:21.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:21.051+0000 7fb528a00640 1 -- 172.21.15.16:0/75437444 shutdown_connections 2024-03-31T15:12:21.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:21.051+0000 7fb528a00640 1 -- 172.21.15.16:0/75437444 wait complete. 2024-03-31T15:12:21.064 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":358,"stamp":"2024-03-31T15:12:20.662404+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":151,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":259832,"kb_used_data":10356,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754714888,"statfs":{"total":773094113280,"available":772828045312,"internally_reserved":0,"allocated":10604544,"data_stored":5211896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":255315504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.782412"},"pg_stats":[{"pgid":"15.3f","version":"0'0","reported_seq":9,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T15:12:16.921615+0000","last_change":"2024-03-31T15:12:16.914727+0000","last_active":"2024-03-31T15:12:16.921615+0000","last_peered":"2024-03-31T15:12:16.921615+0000","last_clean":"2024-03-31T15:12:16.921615+0000","last_became_active":"2024-03-31T15:12:16.914453+0000","last_became_peered":"2024-03-31T15:12:16.914453+0000","last_unstale":"2024-03-31T15:12:16.921615+0000","last_undegraded":"2024-03-31T15:12:16.921615+0000","last_fullsized":"2024-03-31T15:12:16.921615+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_clean_scrub_stamp":"2024-03-31T15:12:15.875521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:07:27.355660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.3e","version":"0'0","reported_seq":9,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T15:12:16.904780+0000","last_change":"2024-03-31T15:12:16.898785+0000","last_active":"2024-03-31T15:12:16.904780+0000","last_peered":"2024-03-31T15:12:16.904780+0000","last_clean":"2024-03-31T15:12:16.904780+0000","last_became_active":"2024-03-31T15:12:16.898507+0000","last_became_peered":"2024-03-31T15:12:16.898507+0000","last_unstale":"2024-03-31T15:12:16.904780+0000","last_undegraded":"2024-03-31T15:12:16.904780+0000","last_fullsized":"2024-03-31T15:12:16.904780+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_clean_scrub_stamp":"2024-03-31T15:12:15.875521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T15:30:58.977979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.3d","version":"0'0","reported_seq":9,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T15:12:16.921727+0000","last_change":"2024-03-31T15:12:16.912813+0000","last_active":"2024-03-31T15:12:16.921727+0000","last_peered":"2024-03-31T15:12:16.921727+0000","last_clean":"2024-03-31T15:12:16.921727+0000","last_became_active":"2024-03-31T15:12:16.912410+0000","last_became_peered":"2024-03-31T15:12:16.912410+0000","last_unstale":"2024-03-31T15:12:16.921727+0000","last_undegraded":"2024-03-31T15:12:16.921727+0000","last_fullsized":"2024-03-31T15:12:16.921727+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_clean_scrub_stamp":"2024-03-31T15:12:15.875521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:45:27.907983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.3c","version":"0'0","reported_seq":9,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T15:12:16.919587+0000","last_change":"2024-03-31T15:12:16.910460+0000","last_active":"2024-03-31T15:12:16.919587+0000","last_peered":"2024-03-31T15:12:16.919587+0000","last_clean":"2024-03-31T15:12:16.919587+0000","last_became_active":"2024-03-31T15:12:16.909960+0000","last_became_peered":"2024-03-31T15:12:16.909960+0000","last_unstale":"2024-03-31T15:12:16.919587+0000","last_undegraded":"2024-03-31T15:12:16.919587+0000","last_fullsized":"2024-03-31T15:12:16.919587+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_clean_scrub_stamp":"2024-03-31T15:12:15.875521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:18:26.495815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.3b","version":"0'0","reported_seq":9,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T15:12:16.909003+0000","last_change":"2024-03-31T15:12:16.903384+0000","last_active":"2024-03-31T15:12:16.909003+0000","last_peered":"2024-03-31T15:12:16.909003+0000","last_clean":"2024-03-31T15:12:16.909003+0000","last_became_active":"2024-03-31T15:12:16.903110+0000","last_became_peered":"2024-03-31T15:12:16.903110+0000","last_unstale":"2024-03-31T15:12:16.909003+0000","last_undegraded":"2024-03-31T15:12:16.909003+0000","last_fullsized":"2024-03-31T15:12:16.909003+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_clean_scrub_stamp":"2024-03-31T15:12:15.875521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:16:08.022126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.3a","version":"0'0","reported_seq":9,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T15:12:16.917157+0000","last_change":"2024-03-31T15:12:16.915254+0000","last_active":"2024-03-31T15:12:16.917157+0000","last_peered":"2024-03-31T15:12:16.917157+0000","last_clean":"2024-03-31T15:12:16.917157+0000","last_became_active":"2024-03-31T15:12:16.912283+0000","last_became_peered":"2024-03-31T15:12:16.912283+0000","last_unstale":"2024-03-31T15:12:16.917157+0000","last_undegraded":"2024-03-31T15:12:16.917157+0000","last_fullsized":"2024-03-31T15:12:16.917157+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_clean_scrub_stamp":"2024-03-31T15:12:15.875521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:57:55.489821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.39","version":"0'0","reported_seq":9,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T15:12:16.915004+0000","last_change":"2024-03-31T15:12:16.909380+0000","last_active":"2024-03-31T15:12:16.915004+0000","last_peered":"2024-03-31T15:12:16.915004+0000","last_clean":"2024-03-31T15:12:16.915004+0000","last_became_active":"2024-03-31T15:12:16.907492+0000","last_became_peered":"2024-03-31T15:12:16.907492+0000","last_unstale":"2024-03-31T15:12:16.915004+0000","last_undegraded":"2024-03-31T15:12:16.915004+0000","last_fullsized":"2024-03-31T15:12:16.915004+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_clean_scrub_stamp":"2024-03-31T15:12:15.875521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:43:16.026939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.38","version":"0'0","reported_seq":9,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T15:12:16.913649+0000","last_change":"2024-03-31T15:12:16.907968+0000","last_active":"2024-03-31T15:12:16.913649+0000","last_peered":"2024-03-31T15:12:16.913649+0000","last_clean":"2024-03-31T15:12:16.913649+0000","last_became_active":"2024-03-31T15:12:16.907679+0000","last_became_peered":"2024-03-31T15:12:16.907679+0000","last_unstale":"2024-03-31T15:12:16.913649+0000","last_undegraded":"2024-03-31T15:12:16.913649+0000","last_fullsized":"2024-03-31T15:12:16.913649+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_clean_scrub_stamp":"2024-03-31T15:12:15.875521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:07:57.538752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.37","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-31T15:12:17.894969+0000","last_change":"2024-03-31T15:12:16.913705+0000","last_active":"2024-03-31T15:12:17.894969+0000","last_peered":"2024-03-31T15:12:17.894969+0000","last_clean":"2024-03-31T15:12:17.894969+0000","last_became_active":"2024-03-31T15:12:16.911556+0000","last_became_peered":"2024-03-31T15:12:16.911556+0000","last_unstale":"2024-03-31T15:12:17.894969+0000","last_undegraded":"2024-03-31T15:12:17.894969+0000","last_fullsized":"2024-03-31T15:12:17.894969+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_clean_scrub_stamp":"2024-03-31T15:12:15.875521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:19:07.639622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.36","version":"0'0","reported_seq":9,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T15:12:16.916938+0000","last_change":"2024-03-31T15:12:16.915079+0000","last_active":"2024-03-31T15:12:16.916938+0000","last_peered":"2024-03-31T15:12:16.916938+0000","last_clean":"2024-03-31T15:12:16.916938+0000","last_became_active":"2024-03-31T15:12:16.912820+0000","last_became_peered":"2024-03-31T15:12:16.912820+0000","last_unstale":"2024-03-31T15:12:16.916938+0000","last_undegraded":"2024-03-31T15:12:16.916938+0000","last_fullsized":"2024-03-31T15:12:16.916938+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_clean_scrub_stamp":"2024-03-31T15:12:15.875521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:05:22.446118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.35","version":"0'0","reported_seq":9,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T15:12:16.915049+0000","last_change":"2024-03-31T15:12:16.909343+0000","last_active":"2024-03-31T15:12:16.915049+0000","last_peered":"2024-03-31T15:12:16.915049+0000","last_clean":"2024-03-31T15:12:16.915049+0000","last_became_active":"2024-03-31T15:12:16.908516+0000","last_became_peered":"2024-03-31T15:12:16.908516+0000","last_unstale":"2024-03-31T15:12:16.915049+0000","last_undegraded":"2024-03-31T15:12:16.915049+0000","last_fullsized":"2024-03-31T15:12:16.915049+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_clean_scrub_stamp":"2024-03-31T15:12:15.875521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:41:17.901423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.34","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-31T15:12:17.894874+0000","last_change":"2024-03-31T15:12:16.914958+0000","last_active":"2024-03-31T15:12:17.894874+0000","last_peered":"2024-03-31T15:12:17.894874+0000","last_clean":"2024-03-31T15:12:17.894874+0000","last_became_active":"2024-03-31T15:12:16.914668+0000","last_became_peered":"2024-03-31T15:12:16.914668+0000","last_unstale":"2024-03-31T15:12:17.894874+0000","last_undegraded":"2024-03-31T15:12:17.894874+0000","last_fullsized":"2024-03-31T15:12:17.894874+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_clean_scrub_stamp":"2024-03-31T15:12:15.875521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:41:17.901423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.33","version":"0'0","reported_seq":9,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T15:12:16.910842+0000","last_change":"2024-03-31T15:12:16.906096+0000","last_active":"2024-03-31T15:12:16.910842+0000","last_peered":"2024-03-31T15:12:16.910842+0000","last_clean":"2024-03-31T15:12:16.910842+0000","last_became_active":"2024-03-31T15:12:16.904668+0000","last_became_peered":"2024-03-31T15:12:16.904668+0000","last_unstale":"2024-03-31T15:12:16.910842+0000","last_undegraded":"2024-03-31T15:12:16.910842+0000","last_fullsized":"2024-03-31T15:12:16.910842+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_clean_scrub_stamp":"2024-03-31T15:12:15.875521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:23:34.095265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.32","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-31T15:12:17.889926+0000","last_change":"2024-03-31T15:12:16.903426+0000","last_active":"2024-03-31T15:12:17.889926+0000","last_peered":"2024-03-31T15:12:17.889926+0000","last_clean":"2024-03-31T15:12:17.889926+0000","last_became_active":"2024-03-31T15:12:16.903150+0000","last_became_peered":"2024-03-31T15:12:16.903150+0000","last_unstale":"2024-03-31T15:12:17.889926+0000","last_undegraded":"2024-03-31T15:12:17.889926+0000","last_fullsized":"2024-03-31T15:12:17.889926+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_clean_scrub_stamp":"2024-03-31T15:12:15.875521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:23:48.485186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.31","version":"0'0","reported_seq":9,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T15:12:16.919038+0000","last_change":"2024-03-31T15:12:16.915738+0000","last_active":"2024-03-31T15:12:16.919038+0000","last_peered":"2024-03-31T15:12:16.919038+0000","last_clean":"2024-03-31T15:12:16.919038+0000","last_became_active":"2024-03-31T15:12:16.914246+0000","last_became_peered":"2024-03-31T15:12:16.914246+0000","last_unstale":"2024-03-31T15:12:16.919038+0000","last_undegraded":"2024-03-31T15:12:16.919038+0000","last_fullsized":"2024-03-31T15:12:16.919038+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_clean_scrub_stamp":"2024-03-31T15:12:15.875521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:50:07.394964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.30","version":"0'0","reported_seq":9,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T15:12:16.916968+0000","last_change":"2024-03-31T15:12:16.913825+0000","last_active":"2024-03-31T15:12:16.916968+0000","last_peered":"2024-03-31T15:12:16.916968+0000","last_clean":"2024-03-31T15:12:16.916968+0000","last_became_active":"2024-03-31T15:12:16.913562+0000","last_became_peered":"2024-03-31T15:12:16.913562+0000","last_unstale":"2024-03-31T15:12:16.916968+0000","last_undegraded":"2024-03-31T15:12:16.916968+0000","last_fullsized":"2024-03-31T15:12:16.916968+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_clean_scrub_stamp":"2024-03-31T15:12:15.875521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:52:43.591702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.2f","version":"0'0","reported_seq":9,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T15:12:16.912965+0000","last_change":"2024-03-31T15:12:16.906873+0000","last_active":"2024-03-31T15:12:16.912965+0000","last_peered":"2024-03-31T15:12:16.912965+0000","last_clean":"2024-03-31T15:12:16.912965+0000","last_became_active":"2024-03-31T15:12:16.906618+0000","last_became_peered":"2024-03-31T15:12:16.906618+0000","last_unstale":"2024-03-31T15:12:16.912965+0000","last_undegraded":"2024-03-31T15:12:16.912965+0000","last_fullsized":"2024-03-31T15:12:16.912965+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_clean_scrub_stamp":"2024-03-31T15:12:15.875521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:01:25.766417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.2e","version":"0'0","reported_seq":9,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T15:12:16.918958+0000","last_change":"2024-03-31T15:12:16.910444+0000","last_active":"2024-03-31T15:12:16.918958+0000","last_peered":"2024-03-31T15:12:16.918958+0000","last_clean":"2024-03-31T15:12:16.918958+0000","last_became_active":"2024-03-31T15:12:16.910163+0000","last_became_peered":"2024-03-31T15:12:16.910163+0000","last_unstale":"2024-03-31T15:12:16.918958+0000","last_undegraded":"2024-03-31T15:12:16.918958+0000","last_fullsized":"2024-03-31T15:12:16.918958+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_clean_scrub_stamp":"2024-03-31T15:12:15.875521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:07:57.538752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.2d","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-31T15:12:17.890226+0000","last_change":"2024-03-31T15:12:16.920057+0000","last_active":"2024-03-31T15:12:17.890226+0000","last_peered":"2024-03-31T15:12:17.890226+0000","last_clean":"2024-03-31T15:12:17.890226+0000","last_became_active":"2024-03-31T15:12:16.919496+0000","last_became_peered":"2024-03-31T15:12:16.919496+0000","last_unstale":"2024-03-31T15:12:17.890226+0000","last_undegraded":"2024-03-31T15:12:17.890226+0000","last_fullsized":"2024-03-31T15:12:17.890226+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_clean_scrub_stamp":"2024-03-31T15:12:15.875521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:07:55.682491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.2c","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-31T15:12:17.894800+0000","last_change":"2024-03-31T15:12:16.913343+0000","last_active":"2024-03-31T15:12:17.894800+0000","last_peered":"2024-03-31T15:12:17.894800+0000","last_clean":"2024-03-31T15:12:17.894800+0000","last_became_active":"2024-03-31T15:12:16.911677+0000","last_became_peered":"2024-03-31T15:12:16.911677+0000","last_unstale":"2024-03-31T15:12:17.894800+0000","last_undegraded":"2024-03-31T15:12:17.894800+0000","last_fullsized":"2024-03-31T15:12:17.894800+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_clean_scrub_stamp":"2024-03-31T15:12:15.875521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:28:01.920854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.2b","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-31T15:12:17.890038+0000","last_change":"2024-03-31T15:12:16.898095+0000","last_active":"2024-03-31T15:12:17.890038+0000","last_peered":"2024-03-31T15:12:17.890038+0000","last_clean":"2024-03-31T15:12:17.890038+0000","last_became_active":"2024-03-31T15:12:16.897819+0000","last_became_peered":"2024-03-31T15:12:16.897819+0000","last_unstale":"2024-03-31T15:12:17.890038+0000","last_undegraded":"2024-03-31T15:12:17.890038+0000","last_fullsized":"2024-03-31T15:12:17.890038+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_clean_scrub_stamp":"2024-03-31T15:12:15.875521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:46:47.981809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.2a","version":"0'0","reported_seq":9,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T15:12:16.904854+0000","last_change":"2024-03-31T15:12:16.899377+0000","last_active":"2024-03-31T15:12:16.904854+0000","last_peered":"2024-03-31T15:12:16.904854+0000","last_clean":"2024-03-31T15:12:16.904854+0000","last_became_active":"2024-03-31T15:12:16.898453+0000","last_became_peered":"2024-03-31T15:12:16.898453+0000","last_unstale":"2024-03-31T15:12:16.904854+0000","last_undegraded":"2024-03-31T15:12:16.904854+0000","last_fullsized":"2024-03-31T15:12:16.904854+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_clean_scrub_stamp":"2024-03-31T15:12:15.875521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:23:34.095265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.29","version":"0'0","reported_seq":9,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T15:12:16.912599+0000","last_change":"2024-03-31T15:12:16.903860+0000","last_active":"2024-03-31T15:12:16.912599+0000","last_peered":"2024-03-31T15:12:16.912599+0000","last_clean":"2024-03-31T15:12:16.912599+0000","last_became_active":"2024-03-31T15:12:16.903601+0000","last_became_peered":"2024-03-31T15:12:16.903601+0000","last_unstale":"2024-03-31T15:12:16.912599+0000","last_undegraded":"2024-03-31T15:12:16.912599+0000","last_fullsized":"2024-03-31T15:12:16.912599+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_clean_scrub_stamp":"2024-03-31T15:12:15.875521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:57:29.441394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.28","version":"0'0","reported_seq":9,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T15:12:16.918364+0000","last_change":"2024-03-31T15:12:16.917075+0000","last_active":"2024-03-31T15:12:16.918364+0000","last_peered":"2024-03-31T15:12:16.918364+0000","last_clean":"2024-03-31T15:12:16.918364+0000","last_became_active":"2024-03-31T15:12:16.916550+0000","last_became_peered":"2024-03-31T15:12:16.916550+0000","last_unstale":"2024-03-31T15:12:16.918364+0000","last_undegraded":"2024-03-31T15:12:16.918364+0000","last_fullsized":"2024-03-31T15:12:16.918364+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_clean_scrub_stamp":"2024-03-31T15:12:15.875521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T17:27:02.919268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.27","version":"0'0","reported_seq":9,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T15:12:16.903723+0000","last_change":"2024-03-31T15:12:16.900790+0000","last_active":"2024-03-31T15:12:16.903723+0000","last_peered":"2024-03-31T15:12:16.903723+0000","last_clean":"2024-03-31T15:12:16.903723+0000","last_became_active":"2024-03-31T15:12:16.900476+0000","last_became_peered":"2024-03-31T15:12:16.900476+0000","last_unstale":"2024-03-31T15:12:16.903723+0000","last_undegraded":"2024-03-31T15:12:16.903723+0000","last_fullsized":"2024-03-31T15:12:16.903723+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_clean_scrub_stamp":"2024-03-31T15:12:15.875521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:07:23.831027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.26","version":"0'0","reported_seq":9,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T15:12:16.915186+0000","last_change":"2024-03-31T15:12:16.912156+0000","last_active":"2024-03-31T15:12:16.915186+0000","last_peered":"2024-03-31T15:12:16.915186+0000","last_clean":"2024-03-31T15:12:16.915186+0000","last_became_active":"2024-03-31T15:12:16.911885+0000","last_became_peered":"2024-03-31T15:12:16.911885+0000","last_unstale":"2024-03-31T15:12:16.915186+0000","last_undegraded":"2024-03-31T15:12:16.915186+0000","last_fullsized":"2024-03-31T15:12:16.915186+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_clean_scrub_stamp":"2024-03-31T15:12:15.875521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:18:26.495815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.25","version":"0'0","reported_seq":9,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T15:12:16.915101+0000","last_change":"2024-03-31T15:12:16.910892+0000","last_active":"2024-03-31T15:12:16.915101+0000","last_peered":"2024-03-31T15:12:16.915101+0000","last_clean":"2024-03-31T15:12:16.915101+0000","last_became_active":"2024-03-31T15:12:16.910350+0000","last_became_peered":"2024-03-31T15:12:16.910350+0000","last_unstale":"2024-03-31T15:12:16.915101+0000","last_undegraded":"2024-03-31T15:12:16.915101+0000","last_fullsized":"2024-03-31T15:12:16.915101+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_clean_scrub_stamp":"2024-03-31T15:12:15.875521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:53:55.735548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.24","version":"0'0","reported_seq":9,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T15:12:16.923062+0000","last_change":"2024-03-31T15:12:16.922419+0000","last_active":"2024-03-31T15:12:16.923062+0000","last_peered":"2024-03-31T15:12:16.923062+0000","last_clean":"2024-03-31T15:12:16.923062+0000","last_became_active":"2024-03-31T15:12:16.922130+0000","last_became_peered":"2024-03-31T15:12:16.922130+0000","last_unstale":"2024-03-31T15:12:16.923062+0000","last_undegraded":"2024-03-31T15:12:16.923062+0000","last_fullsized":"2024-03-31T15:12:16.923062+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_clean_scrub_stamp":"2024-03-31T15:12:15.875521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:23:34.095265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.23","version":"0'0","reported_seq":9,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T15:12:16.917448+0000","last_change":"2024-03-31T15:12:16.913049+0000","last_active":"2024-03-31T15:12:16.917448+0000","last_peered":"2024-03-31T15:12:16.917448+0000","last_clean":"2024-03-31T15:12:16.917448+0000","last_became_active":"2024-03-31T15:12:16.912271+0000","last_became_peered":"2024-03-31T15:12:16.912271+0000","last_unstale":"2024-03-31T15:12:16.917448+0000","last_undegraded":"2024-03-31T15:12:16.917448+0000","last_fullsized":"2024-03-31T15:12:16.917448+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_clean_scrub_stamp":"2024-03-31T15:12:15.875521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T17:51:59.038412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.22","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-31T15:12:17.889891+0000","last_change":"2024-03-31T15:12:16.917356+0000","last_active":"2024-03-31T15:12:17.889891+0000","last_peered":"2024-03-31T15:12:17.889891+0000","last_clean":"2024-03-31T15:12:17.889891+0000","last_became_active":"2024-03-31T15:12:16.917092+0000","last_became_peered":"2024-03-31T15:12:16.917092+0000","last_unstale":"2024-03-31T15:12:17.889891+0000","last_undegraded":"2024-03-31T15:12:17.889891+0000","last_fullsized":"2024-03-31T15:12:17.889891+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_clean_scrub_stamp":"2024-03-31T15:12:15.875521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:02:36.135837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.21","version":"0'0","reported_seq":9,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T15:12:16.907738+0000","last_change":"2024-03-31T15:12:16.902593+0000","last_active":"2024-03-31T15:12:16.907738+0000","last_peered":"2024-03-31T15:12:16.907738+0000","last_clean":"2024-03-31T15:12:16.907738+0000","last_became_active":"2024-03-31T15:12:16.902320+0000","last_became_peered":"2024-03-31T15:12:16.902320+0000","last_unstale":"2024-03-31T15:12:16.907738+0000","last_undegraded":"2024-03-31T15:12:16.907738+0000","last_fullsized":"2024-03-31T15:12:16.907738+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_clean_scrub_stamp":"2024-03-31T15:12:15.875521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T15:30:58.977979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.20","version":"0'0","reported_seq":9,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T15:12:16.920194+0000","last_change":"2024-03-31T15:12:16.912669+0000","last_active":"2024-03-31T15:12:16.920194+0000","last_peered":"2024-03-31T15:12:16.920194+0000","last_clean":"2024-03-31T15:12:16.920194+0000","last_became_active":"2024-03-31T15:12:16.908838+0000","last_became_peered":"2024-03-31T15:12:16.908838+0000","last_unstale":"2024-03-31T15:12:16.920194+0000","last_undegraded":"2024-03-31T15:12:16.920194+0000","last_fullsized":"2024-03-31T15:12:16.920194+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_clean_scrub_stamp":"2024-03-31T15:12:15.875521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:19:29.666114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.1e","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-31T15:12:17.890422+0000","last_change":"2024-03-31T15:12:16.905100+0000","last_active":"2024-03-31T15:12:17.890422+0000","last_peered":"2024-03-31T15:12:17.890422+0000","last_clean":"2024-03-31T15:12:17.890422+0000","last_became_active":"2024-03-31T15:12:16.904715+0000","last_became_peered":"2024-03-31T15:12:16.904715+0000","last_unstale":"2024-03-31T15:12:17.890422+0000","last_undegraded":"2024-03-31T15:12:17.890422+0000","last_fullsized":"2024-03-31T15:12:17.890422+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_clean_scrub_stamp":"2024-03-31T15:12:15.875521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:39:17.509700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.1f","version":"95'4","reported_seq":16,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T15:12:16.927963+0000","last_change":"2024-03-31T15:12:15.895041+0000","last_active":"2024-03-31T15:12:16.927963+0000","last_peered":"2024-03-31T15:12:16.927963+0000","last_clean":"2024-03-31T15:12:16.927963+0000","last_became_active":"2024-03-31T15:12:15.894764+0000","last_became_peered":"2024-03-31T15:12:15.894764+0000","last_unstale":"2024-03-31T15:12:16.927963+0000","last_undegraded":"2024-03-31T15:12:16.927963+0000","last_fullsized":"2024-03-31T15:12:16.927963+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:12:14.865356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:12:14.865356+0000","last_clean_scrub_stamp":"2024-03-31T15:12:14.865356+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:07:51.141948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1f","version":"0'0","reported_seq":9,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T15:12:16.918759+0000","last_change":"2024-03-31T15:12:16.908697+0000","last_active":"2024-03-31T15:12:16.918759+0000","last_peered":"2024-03-31T15:12:16.918759+0000","last_clean":"2024-03-31T15:12:16.918759+0000","last_became_active":"2024-03-31T15:12:16.907981+0000","last_became_peered":"2024-03-31T15:12:16.907981+0000","last_unstale":"2024-03-31T15:12:16.918759+0000","last_undegraded":"2024-03-31T15:12:16.918759+0000","last_fullsized":"2024-03-31T15:12:16.918759+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_clean_scrub_stamp":"2024-03-31T15:12:15.875521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:01:25.766417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.1e","version":"95'1","reported_seq":13,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T15:12:16.929532+0000","last_change":"2024-03-31T15:12:15.913830+0000","last_active":"2024-03-31T15:12:16.929532+0000","last_peered":"2024-03-31T15:12:16.929532+0000","last_clean":"2024-03-31T15:12:16.929532+0000","last_became_active":"2024-03-31T15:12:15.913523+0000","last_became_peered":"2024-03-31T15:12:15.913523+0000","last_unstale":"2024-03-31T15:12:16.929532+0000","last_undegraded":"2024-03-31T15:12:16.929532+0000","last_fullsized":"2024-03-31T15:12:16.929532+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:12:14.865356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:12:14.865356+0000","last_clean_scrub_stamp":"2024-03-31T15:12:14.865356+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:45:37.084308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.1c","version":"0'0","reported_seq":9,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T15:12:16.920101+0000","last_change":"2024-03-31T15:12:16.909505+0000","last_active":"2024-03-31T15:12:16.920101+0000","last_peered":"2024-03-31T15:12:16.920101+0000","last_clean":"2024-03-31T15:12:16.920101+0000","last_became_active":"2024-03-31T15:12:16.909207+0000","last_became_peered":"2024-03-31T15:12:16.909207+0000","last_unstale":"2024-03-31T15:12:16.920101+0000","last_undegraded":"2024-03-31T15:12:16.920101+0000","last_fullsized":"2024-03-31T15:12:16.920101+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_clean_scrub_stamp":"2024-03-31T15:12:15.875521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:17:19.301953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.1d","version":"95'2","reported_seq":14,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T15:12:16.928839+0000","last_change":"2024-03-31T15:12:15.903944+0000","last_active":"2024-03-31T15:12:16.928839+0000","last_peered":"2024-03-31T15:12:16.928839+0000","last_clean":"2024-03-31T15:12:16.928839+0000","last_became_active":"2024-03-31T15:12:15.901604+0000","last_became_peered":"2024-03-31T15:12:15.901604+0000","last_unstale":"2024-03-31T15:12:16.928839+0000","last_undegraded":"2024-03-31T15:12:16.928839+0000","last_fullsized":"2024-03-31T15:12:16.928839+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:12:14.865356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:12:14.865356+0000","last_clean_scrub_stamp":"2024-03-31T15:12:14.865356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T15:16:54.328335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1d","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-31T15:12:17.895025+0000","last_change":"2024-03-31T15:12:16.906139+0000","last_active":"2024-03-31T15:12:17.895025+0000","last_peered":"2024-03-31T15:12:17.895025+0000","last_clean":"2024-03-31T15:12:17.895025+0000","last_became_active":"2024-03-31T15:12:16.905695+0000","last_became_peered":"2024-03-31T15:12:16.905695+0000","last_unstale":"2024-03-31T15:12:17.895025+0000","last_undegraded":"2024-03-31T15:12:17.895025+0000","last_fullsized":"2024-03-31T15:12:17.895025+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_clean_scrub_stamp":"2024-03-31T15:12:15.875521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:52:43.591702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.1c","version":"95'2","reported_seq":14,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T15:12:16.929952+0000","last_change":"2024-03-31T15:12:15.902931+0000","last_active":"2024-03-31T15:12:16.929952+0000","last_peered":"2024-03-31T15:12:16.929952+0000","last_clean":"2024-03-31T15:12:16.929952+0000","last_became_active":"2024-03-31T15:12:15.902660+0000","last_became_peered":"2024-03-31T15:12:15.902660+0000","last_unstale":"2024-03-31T15:12:16.929952+0000","last_undegraded":"2024-03-31T15:12:16.929952+0000","last_fullsized":"2024-03-31T15:12:16.929952+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:12:14.865356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:12:14.865356+0000","last_clean_scrub_stamp":"2024-03-31T15:12:14.865356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:07:22.820863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1a","version":"0'0","reported_seq":9,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T15:12:16.914541+0000","last_change":"2024-03-31T15:12:16.910215+0000","last_active":"2024-03-31T15:12:16.914541+0000","last_peered":"2024-03-31T15:12:16.914541+0000","last_clean":"2024-03-31T15:12:16.914541+0000","last_became_active":"2024-03-31T15:12:16.909223+0000","last_became_peered":"2024-03-31T15:12:16.909223+0000","last_unstale":"2024-03-31T15:12:16.914541+0000","last_undegraded":"2024-03-31T15:12:16.914541+0000","last_fullsized":"2024-03-31T15:12:16.914541+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_clean_scrub_stamp":"2024-03-31T15:12:15.875521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:33:59.722746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.1b","version":"95'1","reported_seq":13,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T15:12:16.927579+0000","last_change":"2024-03-31T15:12:15.902342+0000","last_active":"2024-03-31T15:12:16.927579+0000","last_peered":"2024-03-31T15:12:16.927579+0000","last_clean":"2024-03-31T15:12:16.927579+0000","last_became_active":"2024-03-31T15:12:15.901995+0000","last_became_peered":"2024-03-31T15:12:15.901995+0000","last_unstale":"2024-03-31T15:12:16.927579+0000","last_undegraded":"2024-03-31T15:12:16.927579+0000","last_fullsized":"2024-03-31T15:12:16.927579+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:12:14.865356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:12:14.865356+0000","last_clean_scrub_stamp":"2024-03-31T15:12:14.865356+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:52:20.718978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.1b","version":"0'0","reported_seq":9,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T15:12:16.917998+0000","last_change":"2024-03-31T15:12:16.916208+0000","last_active":"2024-03-31T15:12:16.917998+0000","last_peered":"2024-03-31T15:12:16.917998+0000","last_clean":"2024-03-31T15:12:16.917998+0000","last_became_active":"2024-03-31T15:12:16.915935+0000","last_became_peered":"2024-03-31T15:12:16.915935+0000","last_unstale":"2024-03-31T15:12:16.917998+0000","last_undegraded":"2024-03-31T15:12:16.917998+0000","last_fullsized":"2024-03-31T15:12:16.917998+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_clean_scrub_stamp":"2024-03-31T15:12:15.875521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:28:48.945128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.1a","version":"95'1","reported_seq":13,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T15:12:16.928483+0000","last_change":"2024-03-31T15:12:15.907871+0000","last_active":"2024-03-31T15:12:16.928483+0000","last_peered":"2024-03-31T15:12:16.928483+0000","last_clean":"2024-03-31T15:12:16.928483+0000","last_became_active":"2024-03-31T15:12:15.907391+0000","last_became_peered":"2024-03-31T15:12:15.907391+0000","last_unstale":"2024-03-31T15:12:16.928483+0000","last_undegraded":"2024-03-31T15:12:16.928483+0000","last_fullsized":"2024-03-31T15:12:16.928483+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:12:14.865356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:12:14.865356+0000","last_clean_scrub_stamp":"2024-03-31T15:12:14.865356+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:08:53.221389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.18","version":"0'0","reported_seq":9,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T15:12:16.919262+0000","last_change":"2024-03-31T15:12:16.915607+0000","last_active":"2024-03-31T15:12:16.919262+0000","last_peered":"2024-03-31T15:12:16.919262+0000","last_clean":"2024-03-31T15:12:16.919262+0000","last_became_active":"2024-03-31T15:12:16.914893+0000","last_became_peered":"2024-03-31T15:12:16.914893+0000","last_unstale":"2024-03-31T15:12:16.919262+0000","last_undegraded":"2024-03-31T15:12:16.919262+0000","last_fullsized":"2024-03-31T15:12:16.919262+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_clean_scrub_stamp":"2024-03-31T15:12:15.875521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T17:51:59.038412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.19","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T15:12:16.893425+0000","last_change":"2024-03-31T15:12:15.903998+0000","last_active":"2024-03-31T15:12:16.893425+0000","last_peered":"2024-03-31T15:12:16.893425+0000","last_clean":"2024-03-31T15:12:16.893425+0000","last_became_active":"2024-03-31T15:12:15.903460+0000","last_became_peered":"2024-03-31T15:12:15.903460+0000","last_unstale":"2024-03-31T15:12:16.893425+0000","last_undegraded":"2024-03-31T15:12:16.893425+0000","last_fullsized":"2024-03-31T15:12:16.893425+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:12:14.865356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:12:14.865356+0000","last_clean_scrub_stamp":"2024-03-31T15:12:14.865356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:01:28.193287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.19","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-31T15:12:17.894403+0000","last_change":"2024-03-31T15:12:16.907080+0000","last_active":"2024-03-31T15:12:17.894403+0000","last_peered":"2024-03-31T15:12:17.894403+0000","last_clean":"2024-03-31T15:12:17.894403+0000","last_became_active":"2024-03-31T15:12:16.905716+0000","last_became_peered":"2024-03-31T15:12:16.905716+0000","last_unstale":"2024-03-31T15:12:17.894403+0000","last_undegraded":"2024-03-31T15:12:17.894403+0000","last_fullsized":"2024-03-31T15:12:17.894403+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_clean_scrub_stamp":"2024-03-31T15:12:15.875521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:16:14.392760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.18","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T15:12:16.901868+0000","last_change":"2024-03-31T15:12:15.912850+0000","last_active":"2024-03-31T15:12:16.901868+0000","last_peered":"2024-03-31T15:12:16.901868+0000","last_clean":"2024-03-31T15:12:16.901868+0000","last_became_active":"2024-03-31T15:12:15.910394+0000","last_became_peered":"2024-03-31T15:12:15.910394+0000","last_unstale":"2024-03-31T15:12:16.901868+0000","last_undegraded":"2024-03-31T15:12:16.901868+0000","last_fullsized":"2024-03-31T15:12:16.901868+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:12:14.865356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:12:14.865356+0000","last_clean_scrub_stamp":"2024-03-31T15:12:14.865356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:23:00.399133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.16","version":"0'0","reported_seq":9,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T15:12:16.906418+0000","last_change":"2024-03-31T15:12:16.901505+0000","last_active":"2024-03-31T15:12:16.906418+0000","last_peered":"2024-03-31T15:12:16.906418+0000","last_clean":"2024-03-31T15:12:16.906418+0000","last_became_active":"2024-03-31T15:12:16.901246+0000","last_became_peered":"2024-03-31T15:12:16.901246+0000","last_unstale":"2024-03-31T15:12:16.906418+0000","last_undegraded":"2024-03-31T15:12:16.906418+0000","last_fullsized":"2024-03-31T15:12:16.906418+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_clean_scrub_stamp":"2024-03-31T15:12:15.875521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:09:39.960947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.17","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T15:12:16.903995+0000","last_change":"2024-03-31T15:12:15.912638+0000","last_active":"2024-03-31T15:12:16.903995+0000","last_peered":"2024-03-31T15:12:16.903995+0000","last_clean":"2024-03-31T15:12:16.903995+0000","last_became_active":"2024-03-31T15:12:15.912167+0000","last_became_peered":"2024-03-31T15:12:15.912167+0000","last_unstale":"2024-03-31T15:12:16.903995+0000","last_undegraded":"2024-03-31T15:12:16.903995+0000","last_fullsized":"2024-03-31T15:12:16.903995+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:12:14.865356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:12:14.865356+0000","last_clean_scrub_stamp":"2024-03-31T15:12:14.865356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:19:39.893438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.17","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-31T15:12:17.895005+0000","last_change":"2024-03-31T15:12:16.906841+0000","last_active":"2024-03-31T15:12:17.895005+0000","last_peered":"2024-03-31T15:12:17.895005+0000","last_clean":"2024-03-31T15:12:17.895005+0000","last_became_active":"2024-03-31T15:12:16.906537+0000","last_became_peered":"2024-03-31T15:12:16.906537+0000","last_unstale":"2024-03-31T15:12:17.895005+0000","last_undegraded":"2024-03-31T15:12:17.895005+0000","last_fullsized":"2024-03-31T15:12:17.895005+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_clean_scrub_stamp":"2024-03-31T15:12:15.875521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:43:58.063821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.16","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T15:12:16.906234+0000","last_change":"2024-03-31T15:12:15.911664+0000","last_active":"2024-03-31T15:12:16.906234+0000","last_peered":"2024-03-31T15:12:16.906234+0000","last_clean":"2024-03-31T15:12:16.906234+0000","last_became_active":"2024-03-31T15:12:15.911345+0000","last_became_peered":"2024-03-31T15:12:15.911345+0000","last_unstale":"2024-03-31T15:12:16.906234+0000","last_undegraded":"2024-03-31T15:12:16.906234+0000","last_fullsized":"2024-03-31T15:12:16.906234+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:12:14.865356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:12:14.865356+0000","last_clean_scrub_stamp":"2024-03-31T15:12:14.865356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:09:38.950783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.14","version":"0'0","reported_seq":9,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T15:12:16.902544+0000","last_change":"2024-03-31T15:12:16.900255+0000","last_active":"2024-03-31T15:12:16.902544+0000","last_peered":"2024-03-31T15:12:16.902544+0000","last_clean":"2024-03-31T15:12:16.902544+0000","last_became_active":"2024-03-31T15:12:16.899918+0000","last_became_peered":"2024-03-31T15:12:16.899918+0000","last_unstale":"2024-03-31T15:12:16.902544+0000","last_undegraded":"2024-03-31T15:12:16.902544+0000","last_fullsized":"2024-03-31T15:12:16.902544+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_clean_scrub_stamp":"2024-03-31T15:12:15.875521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:34:40.119670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.15","version":"95'3","reported_seq":15,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T15:12:16.931836+0000","last_change":"2024-03-31T15:12:15.909068+0000","last_active":"2024-03-31T15:12:16.931836+0000","last_peered":"2024-03-31T15:12:16.931836+0000","last_clean":"2024-03-31T15:12:16.931836+0000","last_became_active":"2024-03-31T15:12:15.908677+0000","last_became_peered":"2024-03-31T15:12:15.908677+0000","last_unstale":"2024-03-31T15:12:16.931836+0000","last_undegraded":"2024-03-31T15:12:16.931836+0000","last_fullsized":"2024-03-31T15:12:16.931836+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:12:14.865356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:12:14.865356+0000","last_clean_scrub_stamp":"2024-03-31T15:12:14.865356+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:28:00.910690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.15","version":"0'0","reported_seq":9,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T15:12:16.915068+0000","last_change":"2024-03-31T15:12:16.910015+0000","last_active":"2024-03-31T15:12:16.915068+0000","last_peered":"2024-03-31T15:12:16.915068+0000","last_clean":"2024-03-31T15:12:16.915068+0000","last_became_active":"2024-03-31T15:12:16.909651+0000","last_became_peered":"2024-03-31T15:12:16.909651+0000","last_unstale":"2024-03-31T15:12:16.915068+0000","last_undegraded":"2024-03-31T15:12:16.915068+0000","last_fullsized":"2024-03-31T15:12:16.915068+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_clean_scrub_stamp":"2024-03-31T15:12:15.875521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:19:07.639622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.14","version":"95'6","reported_seq":20,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-31T15:12:17.895037+0000","last_change":"2024-03-31T15:12:15.905613+0000","last_active":"2024-03-31T15:12:17.895037+0000","last_peered":"2024-03-31T15:12:17.895037+0000","last_clean":"2024-03-31T15:12:17.895037+0000","last_became_active":"2024-03-31T15:12:15.905340+0000","last_became_peered":"2024-03-31T15:12:15.905340+0000","last_unstale":"2024-03-31T15:12:17.895037+0000","last_undegraded":"2024-03-31T15:12:17.895037+0000","last_fullsized":"2024-03-31T15:12:17.895037+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:12:14.865356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:12:14.865356+0000","last_clean_scrub_stamp":"2024-03-31T15:12:14.865356+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:33:58.712582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.12","version":"0'0","reported_seq":9,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T15:12:16.918724+0000","last_change":"2024-03-31T15:12:16.908659+0000","last_active":"2024-03-31T15:12:16.918724+0000","last_peered":"2024-03-31T15:12:16.918724+0000","last_clean":"2024-03-31T15:12:16.918724+0000","last_became_active":"2024-03-31T15:12:16.907211+0000","last_became_peered":"2024-03-31T15:12:16.907211+0000","last_unstale":"2024-03-31T15:12:16.918724+0000","last_undegraded":"2024-03-31T15:12:16.918724+0000","last_fullsized":"2024-03-31T15:12:16.918724+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_clean_scrub_stamp":"2024-03-31T15:12:15.875521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:57:29.441394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.13","version":"95'2","reported_seq":16,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-31T15:12:17.890033+0000","last_change":"2024-03-31T15:12:15.902014+0000","last_active":"2024-03-31T15:12:17.890033+0000","last_peered":"2024-03-31T15:12:17.890033+0000","last_clean":"2024-03-31T15:12:17.890033+0000","last_became_active":"2024-03-31T15:12:15.900192+0000","last_became_peered":"2024-03-31T15:12:15.900192+0000","last_unstale":"2024-03-31T15:12:17.890033+0000","last_undegraded":"2024-03-31T15:12:17.890033+0000","last_fullsized":"2024-03-31T15:12:17.890033+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:12:14.865356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:12:14.865356+0000","last_clean_scrub_stamp":"2024-03-31T15:12:14.865356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:12:58.240842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.13","version":"0'0","reported_seq":9,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T15:12:16.921178+0000","last_change":"2024-03-31T15:12:16.913045+0000","last_active":"2024-03-31T15:12:16.921178+0000","last_peered":"2024-03-31T15:12:16.921178+0000","last_clean":"2024-03-31T15:12:16.921178+0000","last_became_active":"2024-03-31T15:12:16.911052+0000","last_became_peered":"2024-03-31T15:12:16.911052+0000","last_unstale":"2024-03-31T15:12:16.921178+0000","last_undegraded":"2024-03-31T15:12:16.921178+0000","last_fullsized":"2024-03-31T15:12:16.921178+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_clean_scrub_stamp":"2024-03-31T15:12:15.875521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:21:47.383013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.12","version":"0'0","reported_seq":14,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-31T15:12:17.890228+0000","last_change":"2024-03-31T15:12:15.910589+0000","last_active":"2024-03-31T15:12:17.890228+0000","last_peered":"2024-03-31T15:12:17.890228+0000","last_clean":"2024-03-31T15:12:17.890228+0000","last_became_active":"2024-03-31T15:12:15.910296+0000","last_became_peered":"2024-03-31T15:12:15.910296+0000","last_unstale":"2024-03-31T15:12:17.890228+0000","last_undegraded":"2024-03-31T15:12:17.890228+0000","last_fullsized":"2024-03-31T15:12:17.890228+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:12:14.865356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:12:14.865356+0000","last_clean_scrub_stamp":"2024-03-31T15:12:14.865356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:23:32.128958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.10","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-31T15:12:17.895013+0000","last_change":"2024-03-31T15:12:16.903374+0000","last_active":"2024-03-31T15:12:17.895013+0000","last_peered":"2024-03-31T15:12:17.895013+0000","last_clean":"2024-03-31T15:12:17.895013+0000","last_became_active":"2024-03-31T15:12:16.902503+0000","last_became_peered":"2024-03-31T15:12:16.902503+0000","last_unstale":"2024-03-31T15:12:17.895013+0000","last_undegraded":"2024-03-31T15:12:17.895013+0000","last_fullsized":"2024-03-31T15:12:17.895013+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_clean_scrub_stamp":"2024-03-31T15:12:15.875521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:53:55.735548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.11","version":"0'0","reported_seq":14,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-31T15:12:17.894419+0000","last_change":"2024-03-31T15:12:15.898748+0000","last_active":"2024-03-31T15:12:17.894419+0000","last_peered":"2024-03-31T15:12:17.894419+0000","last_clean":"2024-03-31T15:12:17.894419+0000","last_became_active":"2024-03-31T15:12:15.898311+0000","last_became_peered":"2024-03-31T15:12:15.898311+0000","last_unstale":"2024-03-31T15:12:17.894419+0000","last_undegraded":"2024-03-31T15:12:17.894419+0000","last_fullsized":"2024-03-31T15:12:17.894419+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:12:14.865356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:12:14.865356+0000","last_clean_scrub_stamp":"2024-03-31T15:12:14.865356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T17:17:59.212188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.11","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-31T15:12:17.889725+0000","last_change":"2024-03-31T15:12:16.907101+0000","last_active":"2024-03-31T15:12:17.889725+0000","last_peered":"2024-03-31T15:12:17.889725+0000","last_clean":"2024-03-31T15:12:17.889725+0000","last_became_active":"2024-03-31T15:12:16.906777+0000","last_became_peered":"2024-03-31T15:12:16.906777+0000","last_unstale":"2024-03-31T15:12:17.889725+0000","last_undegraded":"2024-03-31T15:12:17.889725+0000","last_fullsized":"2024-03-31T15:12:17.889725+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_clean_scrub_stamp":"2024-03-31T15:12:15.875521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T15:43:27.285287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.10","version":"95'8","reported_seq":20,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T15:12:16.930393+0000","last_change":"2024-03-31T15:12:15.903073+0000","last_active":"2024-03-31T15:12:16.930393+0000","last_peered":"2024-03-31T15:12:16.930393+0000","last_clean":"2024-03-31T15:12:16.930393+0000","last_became_active":"2024-03-31T15:12:15.902722+0000","last_became_peered":"2024-03-31T15:12:15.902722+0000","last_unstale":"2024-03-31T15:12:16.930393+0000","last_undegraded":"2024-03-31T15:12:16.930393+0000","last_fullsized":"2024-03-31T15:12:16.930393+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:12:14.865356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:12:14.865356+0000","last_clean_scrub_stamp":"2024-03-31T15:12:14.865356+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:37:51.668010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.f","version":"0'0","reported_seq":9,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T15:12:16.915299+0000","last_change":"2024-03-31T15:12:16.911972+0000","last_active":"2024-03-31T15:12:16.915299+0000","last_peered":"2024-03-31T15:12:16.915299+0000","last_clean":"2024-03-31T15:12:16.915299+0000","last_became_active":"2024-03-31T15:12:16.910557+0000","last_became_peered":"2024-03-31T15:12:16.910557+0000","last_unstale":"2024-03-31T15:12:16.915299+0000","last_undegraded":"2024-03-31T15:12:16.915299+0000","last_fullsized":"2024-03-31T15:12:16.915299+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_clean_scrub_stamp":"2024-03-31T15:12:15.875521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:11:19.502245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.e","version":"95'2","reported_seq":16,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-31T15:12:17.894878+0000","last_change":"2024-03-31T15:12:15.902495+0000","last_active":"2024-03-31T15:12:17.894878+0000","last_peered":"2024-03-31T15:12:17.894878+0000","last_clean":"2024-03-31T15:12:17.894878+0000","last_became_active":"2024-03-31T15:12:15.902222+0000","last_became_peered":"2024-03-31T15:12:15.902222+0000","last_unstale":"2024-03-31T15:12:17.894878+0000","last_undegraded":"2024-03-31T15:12:17.894878+0000","last_fullsized":"2024-03-31T15:12:17.894878+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:12:14.865356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:12:14.865356+0000","last_clean_scrub_stamp":"2024-03-31T15:12:14.865356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:47:02.138749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.c","version":"0'0","reported_seq":9,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T15:12:16.905793+0000","last_change":"2024-03-31T15:12:16.901466+0000","last_active":"2024-03-31T15:12:16.905793+0000","last_peered":"2024-03-31T15:12:16.905793+0000","last_clean":"2024-03-31T15:12:16.905793+0000","last_became_active":"2024-03-31T15:12:16.901176+0000","last_became_peered":"2024-03-31T15:12:16.901176+0000","last_unstale":"2024-03-31T15:12:16.905793+0000","last_undegraded":"2024-03-31T15:12:16.905793+0000","last_fullsized":"2024-03-31T15:12:16.905793+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_clean_scrub_stamp":"2024-03-31T15:12:15.875521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T17:18:00.222352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.d","version":"95'1","reported_seq":13,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T15:12:16.927871+0000","last_change":"2024-03-31T15:12:15.900951+0000","last_active":"2024-03-31T15:12:16.927871+0000","last_peered":"2024-03-31T15:12:16.927871+0000","last_clean":"2024-03-31T15:12:16.927871+0000","last_became_active":"2024-03-31T15:12:15.900573+0000","last_became_peered":"2024-03-31T15:12:15.900573+0000","last_unstale":"2024-03-31T15:12:16.927871+0000","last_undegraded":"2024-03-31T15:12:16.927871+0000","last_fullsized":"2024-03-31T15:12:16.927871+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:12:14.865356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:12:14.865356+0000","last_clean_scrub_stamp":"2024-03-31T15:12:14.865356+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:08:53.221389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.d","version":"0'0","reported_seq":9,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T15:12:16.915084+0000","last_change":"2024-03-31T15:12:16.910465+0000","last_active":"2024-03-31T15:12:16.915084+0000","last_peered":"2024-03-31T15:12:16.915084+0000","last_clean":"2024-03-31T15:12:16.915084+0000","last_became_active":"2024-03-31T15:12:16.910098+0000","last_became_peered":"2024-03-31T15:12:16.910098+0000","last_unstale":"2024-03-31T15:12:16.915084+0000","last_undegraded":"2024-03-31T15:12:16.915084+0000","last_fullsized":"2024-03-31T15:12:16.915084+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_clean_scrub_stamp":"2024-03-31T15:12:15.875521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:43:58.063821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.c","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T15:12:16.889117+0000","last_change":"2024-03-31T15:12:15.901363+0000","last_active":"2024-03-31T15:12:16.889117+0000","last_peered":"2024-03-31T15:12:16.889117+0000","last_clean":"2024-03-31T15:12:16.889117+0000","last_became_active":"2024-03-31T15:12:15.900798+0000","last_became_peered":"2024-03-31T15:12:15.900798+0000","last_unstale":"2024-03-31T15:12:16.889117+0000","last_undegraded":"2024-03-31T15:12:16.889117+0000","last_fullsized":"2024-03-31T15:12:16.889117+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:12:14.865356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:12:14.865356+0000","last_clean_scrub_stamp":"2024-03-31T15:12:14.865356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:16:07.011962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.a","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-31T15:12:17.890044+0000","last_change":"2024-03-31T15:12:16.905567+0000","last_active":"2024-03-31T15:12:17.890044+0000","last_peered":"2024-03-31T15:12:17.890044+0000","last_clean":"2024-03-31T15:12:17.890044+0000","last_became_active":"2024-03-31T15:12:16.905274+0000","last_became_peered":"2024-03-31T15:12:16.905274+0000","last_unstale":"2024-03-31T15:12:17.890044+0000","last_undegraded":"2024-03-31T15:12:17.890044+0000","last_fullsized":"2024-03-31T15:12:17.890044+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_clean_scrub_stamp":"2024-03-31T15:12:15.875521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:43:51.074045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.b","version":"95'2","reported_seq":14,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T15:12:16.928838+0000","last_change":"2024-03-31T15:12:15.898375+0000","last_active":"2024-03-31T15:12:16.928838+0000","last_peered":"2024-03-31T15:12:16.928838+0000","last_clean":"2024-03-31T15:12:16.928838+0000","last_became_active":"2024-03-31T15:12:15.898079+0000","last_became_peered":"2024-03-31T15:12:15.898079+0000","last_unstale":"2024-03-31T15:12:16.928838+0000","last_undegraded":"2024-03-31T15:12:16.928838+0000","last_fullsized":"2024-03-31T15:12:16.928838+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:12:14.865356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:12:14.865356+0000","last_clean_scrub_stamp":"2024-03-31T15:12:14.865356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:03:02.364251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.b","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-31T15:12:17.889857+0000","last_change":"2024-03-31T15:12:16.903195+0000","last_active":"2024-03-31T15:12:17.889857+0000","last_peered":"2024-03-31T15:12:17.889857+0000","last_clean":"2024-03-31T15:12:17.889857+0000","last_became_active":"2024-03-31T15:12:16.902876+0000","last_became_peered":"2024-03-31T15:12:16.902876+0000","last_unstale":"2024-03-31T15:12:17.889857+0000","last_undegraded":"2024-03-31T15:12:17.889857+0000","last_fullsized":"2024-03-31T15:12:17.889857+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_clean_scrub_stamp":"2024-03-31T15:12:15.875521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T16:34:10.231547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.a","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T15:12:16.896627+0000","last_change":"2024-03-31T15:12:15.911483+0000","last_active":"2024-03-31T15:12:16.896627+0000","last_peered":"2024-03-31T15:12:16.896627+0000","last_clean":"2024-03-31T15:12:16.896627+0000","last_became_active":"2024-03-31T15:12:15.911137+0000","last_became_peered":"2024-03-31T15:12:15.911137+0000","last_unstale":"2024-03-31T15:12:16.896627+0000","last_undegraded":"2024-03-31T15:12:16.896627+0000","last_fullsized":"2024-03-31T15:12:16.896627+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:12:14.865356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:12:14.865356+0000","last_clean_scrub_stamp":"2024-03-31T15:12:14.865356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:42:35.796151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.8","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-31T15:12:17.889737+0000","last_change":"2024-03-31T15:12:16.901928+0000","last_active":"2024-03-31T15:12:17.889737+0000","last_peered":"2024-03-31T15:12:17.889737+0000","last_clean":"2024-03-31T15:12:17.889737+0000","last_became_active":"2024-03-31T15:12:16.901542+0000","last_became_peered":"2024-03-31T15:12:16.901542+0000","last_unstale":"2024-03-31T15:12:17.889737+0000","last_undegraded":"2024-03-31T15:12:17.889737+0000","last_fullsized":"2024-03-31T15:12:17.889737+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_clean_scrub_stamp":"2024-03-31T15:12:15.875521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:32:35.086735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.9","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T15:12:16.897788+0000","last_change":"2024-03-31T15:12:15.910432+0000","last_active":"2024-03-31T15:12:16.897788+0000","last_peered":"2024-03-31T15:12:16.897788+0000","last_clean":"2024-03-31T15:12:16.897788+0000","last_became_active":"2024-03-31T15:12:15.910086+0000","last_became_peered":"2024-03-31T15:12:15.910086+0000","last_unstale":"2024-03-31T15:12:16.897788+0000","last_undegraded":"2024-03-31T15:12:16.897788+0000","last_fullsized":"2024-03-31T15:12:16.897788+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:12:14.865356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:12:14.865356+0000","last_clean_scrub_stamp":"2024-03-31T15:12:14.865356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:47:02.138749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.9","version":"0'0","reported_seq":9,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T15:12:16.917493+0000","last_change":"2024-03-31T15:12:16.916415+0000","last_active":"2024-03-31T15:12:16.917493+0000","last_peered":"2024-03-31T15:12:16.917493+0000","last_clean":"2024-03-31T15:12:16.917493+0000","last_became_active":"2024-03-31T15:12:16.916134+0000","last_became_peered":"2024-03-31T15:12:16.916134+0000","last_unstale":"2024-03-31T15:12:16.917493+0000","last_undegraded":"2024-03-31T15:12:16.917493+0000","last_fullsized":"2024-03-31T15:12:16.917493+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_clean_scrub_stamp":"2024-03-31T15:12:15.875521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T15:30:58.977979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.8","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T15:12:16.888139+0000","last_change":"2024-03-31T15:12:15.905836+0000","last_active":"2024-03-31T15:12:16.888139+0000","last_peered":"2024-03-31T15:12:16.888139+0000","last_clean":"2024-03-31T15:12:16.888139+0000","last_became_active":"2024-03-31T15:12:15.905095+0000","last_became_peered":"2024-03-31T15:12:15.905095+0000","last_unstale":"2024-03-31T15:12:16.888139+0000","last_undegraded":"2024-03-31T15:12:16.888139+0000","last_fullsized":"2024-03-31T15:12:16.888139+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:12:14.865356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:12:14.865356+0000","last_clean_scrub_stamp":"2024-03-31T15:12:14.865356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:01:28.193287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.6","version":"0'0","reported_seq":9,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T15:12:16.897194+0000","last_change":"2024-03-31T15:12:16.896142+0000","last_active":"2024-03-31T15:12:16.897194+0000","last_peered":"2024-03-31T15:12:16.897194+0000","last_clean":"2024-03-31T15:12:16.897194+0000","last_became_active":"2024-03-31T15:12:16.895668+0000","last_became_peered":"2024-03-31T15:12:16.895668+0000","last_unstale":"2024-03-31T15:12:16.897194+0000","last_undegraded":"2024-03-31T15:12:16.897194+0000","last_fullsized":"2024-03-31T15:12:16.897194+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_clean_scrub_stamp":"2024-03-31T15:12:15.875521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:05:22.446118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.7","version":"95'2","reported_seq":14,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T15:12:16.928235+0000","last_change":"2024-03-31T15:12:15.906062+0000","last_active":"2024-03-31T15:12:16.928235+0000","last_peered":"2024-03-31T15:12:16.928235+0000","last_clean":"2024-03-31T15:12:16.928235+0000","last_became_active":"2024-03-31T15:12:15.903879+0000","last_became_peered":"2024-03-31T15:12:15.903879+0000","last_unstale":"2024-03-31T15:12:16.928235+0000","last_undegraded":"2024-03-31T15:12:16.928235+0000","last_fullsized":"2024-03-31T15:12:16.928235+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:12:14.865356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:12:14.865356+0000","last_clean_scrub_stamp":"2024-03-31T15:12:14.865356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T15:19:58.769761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.7","version":"0'0","reported_seq":9,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T15:12:16.922343+0000","last_change":"2024-03-31T15:12:16.919349+0000","last_active":"2024-03-31T15:12:16.922343+0000","last_peered":"2024-03-31T15:12:16.922343+0000","last_clean":"2024-03-31T15:12:16.922343+0000","last_became_active":"2024-03-31T15:12:16.919063+0000","last_became_peered":"2024-03-31T15:12:16.919063+0000","last_unstale":"2024-03-31T15:12:16.922343+0000","last_undegraded":"2024-03-31T15:12:16.922343+0000","last_fullsized":"2024-03-31T15:12:16.922343+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_clean_scrub_stamp":"2024-03-31T15:12:15.875521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T19:09:46.169527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.6","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T15:12:16.890703+0000","last_change":"2024-03-31T15:12:15.902962+0000","last_active":"2024-03-31T15:12:16.890703+0000","last_peered":"2024-03-31T15:12:16.890703+0000","last_clean":"2024-03-31T15:12:16.890703+0000","last_became_active":"2024-03-31T15:12:15.902021+0000","last_became_peered":"2024-03-31T15:12:15.902021+0000","last_unstale":"2024-03-31T15:12:16.890703+0000","last_undegraded":"2024-03-31T15:12:16.890703+0000","last_fullsized":"2024-03-31T15:12:16.890703+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:12:14.865356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:12:14.865356+0000","last_clean_scrub_stamp":"2024-03-31T15:12:14.865356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:11:18.492080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.4","version":"0'0","reported_seq":9,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T15:12:16.920077+0000","last_change":"2024-03-31T15:12:16.909064+0000","last_active":"2024-03-31T15:12:16.920077+0000","last_peered":"2024-03-31T15:12:16.920077+0000","last_clean":"2024-03-31T15:12:16.920077+0000","last_became_active":"2024-03-31T15:12:16.907598+0000","last_became_peered":"2024-03-31T15:12:16.907598+0000","last_unstale":"2024-03-31T15:12:16.920077+0000","last_undegraded":"2024-03-31T15:12:16.920077+0000","last_fullsized":"2024-03-31T15:12:16.920077+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_clean_scrub_stamp":"2024-03-31T15:12:15.875521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T15:43:52.620756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.5","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T15:12:16.888694+0000","last_change":"2024-03-31T15:12:15.904927+0000","last_active":"2024-03-31T15:12:16.888694+0000","last_peered":"2024-03-31T15:12:16.888694+0000","last_clean":"2024-03-31T15:12:16.888694+0000","last_became_active":"2024-03-31T15:12:15.904665+0000","last_became_peered":"2024-03-31T15:12:15.904665+0000","last_unstale":"2024-03-31T15:12:16.888694+0000","last_undegraded":"2024-03-31T15:12:16.888694+0000","last_fullsized":"2024-03-31T15:12:16.888694+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:12:14.865356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:12:14.865356+0000","last_clean_scrub_stamp":"2024-03-31T15:12:14.865356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:34:39.109506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.5","version":"0'0","reported_seq":9,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T15:12:16.909399+0000","last_change":"2024-03-31T15:12:16.905106+0000","last_active":"2024-03-31T15:12:16.909399+0000","last_peered":"2024-03-31T15:12:16.909399+0000","last_clean":"2024-03-31T15:12:16.909399+0000","last_became_active":"2024-03-31T15:12:16.904806+0000","last_became_peered":"2024-03-31T15:12:16.904806+0000","last_unstale":"2024-03-31T15:12:16.909399+0000","last_undegraded":"2024-03-31T15:12:16.909399+0000","last_fullsized":"2024-03-31T15:12:16.909399+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_clean_scrub_stamp":"2024-03-31T15:12:15.875521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T15:16:55.338499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.4","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T15:12:16.893922+0000","last_change":"2024-03-31T15:12:15.905754+0000","last_active":"2024-03-31T15:12:16.893922+0000","last_peered":"2024-03-31T15:12:16.893922+0000","last_clean":"2024-03-31T15:12:16.893922+0000","last_became_active":"2024-03-31T15:12:15.904700+0000","last_became_peered":"2024-03-31T15:12:15.904700+0000","last_unstale":"2024-03-31T15:12:16.893922+0000","last_undegraded":"2024-03-31T15:12:16.893922+0000","last_fullsized":"2024-03-31T15:12:16.893922+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:12:14.865356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:12:14.865356+0000","last_clean_scrub_stamp":"2024-03-31T15:12:14.865356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:02:19.394431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.2","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-31T15:12:17.890065+0000","last_change":"2024-03-31T15:12:16.904662+0000","last_active":"2024-03-31T15:12:17.890065+0000","last_peered":"2024-03-31T15:12:17.890065+0000","last_clean":"2024-03-31T15:12:17.890065+0000","last_became_active":"2024-03-31T15:12:16.904363+0000","last_became_peered":"2024-03-31T15:12:16.904363+0000","last_unstale":"2024-03-31T15:12:17.890065+0000","last_undegraded":"2024-03-31T15:12:17.890065+0000","last_fullsized":"2024-03-31T15:12:17.890065+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_clean_scrub_stamp":"2024-03-31T15:12:15.875521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:11:08.200799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.3","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T15:12:16.888561+0000","last_change":"2024-03-31T15:12:15.903529+0000","last_active":"2024-03-31T15:12:16.888561+0000","last_peered":"2024-03-31T15:12:16.888561+0000","last_clean":"2024-03-31T15:12:16.888561+0000","last_became_active":"2024-03-31T15:12:15.903212+0000","last_became_peered":"2024-03-31T15:12:15.903212+0000","last_unstale":"2024-03-31T15:12:16.888561+0000","last_undegraded":"2024-03-31T15:12:16.888561+0000","last_fullsized":"2024-03-31T15:12:16.888561+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:12:14.865356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:12:14.865356+0000","last_clean_scrub_stamp":"2024-03-31T15:12:14.865356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:02:19.394431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.3","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-31T15:12:17.894895+0000","last_change":"2024-03-31T15:12:16.907561+0000","last_active":"2024-03-31T15:12:17.894895+0000","last_peered":"2024-03-31T15:12:17.894895+0000","last_clean":"2024-03-31T15:12:17.894895+0000","last_became_active":"2024-03-31T15:12:16.907304+0000","last_became_peered":"2024-03-31T15:12:16.907304+0000","last_unstale":"2024-03-31T15:12:17.894895+0000","last_undegraded":"2024-03-31T15:12:17.894895+0000","last_fullsized":"2024-03-31T15:12:17.894895+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_clean_scrub_stamp":"2024-03-31T15:12:15.875521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:43:16.026939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.2","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T15:12:16.893700+0000","last_change":"2024-03-31T15:12:15.909995+0000","last_active":"2024-03-31T15:12:16.893700+0000","last_peered":"2024-03-31T15:12:16.893700+0000","last_clean":"2024-03-31T15:12:16.893700+0000","last_became_active":"2024-03-31T15:12:15.909465+0000","last_became_peered":"2024-03-31T15:12:15.909465+0000","last_unstale":"2024-03-31T15:12:16.893700+0000","last_undegraded":"2024-03-31T15:12:16.893700+0000","last_fullsized":"2024-03-31T15:12:16.893700+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:12:14.865356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:12:14.865356+0000","last_clean_scrub_stamp":"2024-03-31T15:12:14.865356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:03:02.364251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.0","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-31T15:12:17.889856+0000","last_change":"2024-03-31T15:12:16.903467+0000","last_active":"2024-03-31T15:12:17.889856+0000","last_peered":"2024-03-31T15:12:17.889856+0000","last_clean":"2024-03-31T15:12:17.889856+0000","last_became_active":"2024-03-31T15:12:16.903217+0000","last_became_peered":"2024-03-31T15:12:16.903217+0000","last_unstale":"2024-03-31T15:12:17.889856+0000","last_undegraded":"2024-03-31T15:12:17.889856+0000","last_fullsized":"2024-03-31T15:12:17.889856+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_clean_scrub_stamp":"2024-03-31T15:12:15.875521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:28:33.431240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.1","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T15:12:16.898121+0000","last_change":"2024-03-31T15:12:15.913568+0000","last_active":"2024-03-31T15:12:16.898121+0000","last_peered":"2024-03-31T15:12:16.898121+0000","last_clean":"2024-03-31T15:12:16.898121+0000","last_became_active":"2024-03-31T15:12:15.912885+0000","last_became_peered":"2024-03-31T15:12:15.912885+0000","last_unstale":"2024-03-31T15:12:16.898121+0000","last_undegraded":"2024-03-31T15:12:16.898121+0000","last_fullsized":"2024-03-31T15:12:16.898121+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:12:14.865356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:12:14.865356+0000","last_clean_scrub_stamp":"2024-03-31T15:12:14.865356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:05:21.435954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.1","version":"0'0","reported_seq":9,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T15:12:16.908248+0000","last_change":"2024-03-31T15:12:16.903949+0000","last_active":"2024-03-31T15:12:16.908248+0000","last_peered":"2024-03-31T15:12:16.908248+0000","last_clean":"2024-03-31T15:12:16.908248+0000","last_became_active":"2024-03-31T15:12:16.903684+0000","last_became_peered":"2024-03-31T15:12:16.903684+0000","last_unstale":"2024-03-31T15:12:16.908248+0000","last_undegraded":"2024-03-31T15:12:16.908248+0000","last_fullsized":"2024-03-31T15:12:16.908248+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_clean_scrub_stamp":"2024-03-31T15:12:15.875521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:07:52.152112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.0","version":"0'0","reported_seq":14,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-31T15:12:17.889867+0000","last_change":"2024-03-31T15:12:15.903319+0000","last_active":"2024-03-31T15:12:17.889867+0000","last_peered":"2024-03-31T15:12:17.889867+0000","last_clean":"2024-03-31T15:12:17.889867+0000","last_became_active":"2024-03-31T15:12:15.902939+0000","last_became_peered":"2024-03-31T15:12:15.902939+0000","last_unstale":"2024-03-31T15:12:17.889867+0000","last_undegraded":"2024-03-31T15:12:17.889867+0000","last_fullsized":"2024-03-31T15:12:17.889867+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:12:14.865356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:12:14.865356+0000","last_clean_scrub_stamp":"2024-03-31T15:12:14.865356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:36:28.366348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.e","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-31T15:12:17.890421+0000","last_change":"2024-03-31T15:12:16.905769+0000","last_active":"2024-03-31T15:12:17.890421+0000","last_peered":"2024-03-31T15:12:17.890421+0000","last_clean":"2024-03-31T15:12:17.890421+0000","last_became_active":"2024-03-31T15:12:16.905466+0000","last_became_peered":"2024-03-31T15:12:16.905466+0000","last_unstale":"2024-03-31T15:12:17.890421+0000","last_undegraded":"2024-03-31T15:12:17.890421+0000","last_fullsized":"2024-03-31T15:12:17.890421+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_clean_scrub_stamp":"2024-03-31T15:12:15.875521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:03:21.594309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.f","version":"95'1","reported_seq":13,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T15:12:16.933297+0000","last_change":"2024-03-31T15:12:15.911260+0000","last_active":"2024-03-31T15:12:16.933297+0000","last_peered":"2024-03-31T15:12:16.933297+0000","last_clean":"2024-03-31T15:12:16.933297+0000","last_became_active":"2024-03-31T15:12:15.910954+0000","last_became_peered":"2024-03-31T15:12:15.910954+0000","last_unstale":"2024-03-31T15:12:16.933297+0000","last_undegraded":"2024-03-31T15:12:16.933297+0000","last_fullsized":"2024-03-31T15:12:16.933297+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:12:14.865356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:12:14.865356+0000","last_clean_scrub_stamp":"2024-03-31T15:12:14.865356+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:16:13.382595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":308,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T15:12:16.901433+0000","last_change":"2024-03-31T15:03:24.329890+0000","last_active":"2024-03-31T15:12:16.901433+0000","last_peered":"2024-03-31T15:12:16.901433+0000","last_clean":"2024-03-31T15:12:16.901433+0000","last_became_active":"2024-03-31T15:03:24.328882+0000","last_became_peered":"2024-03-31T15:03:24.328882+0000","last_unstale":"2024-03-31T15:12:16.901433+0000","last_undegraded":"2024-03-31T15:12:16.901433+0000","last_fullsized":"2024-03-31T15:12:16.901433+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:22.885535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:22.885535+0000","last_clean_scrub_stamp":"2024-03-31T15:03:22.885535+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:18:29.004296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":15,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":14,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509552,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32912,"kb_used_data":1728,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338928,"statfs":{"total":96636764160,"available":96603062272,"internally_reserved":0,"allocated":1769472,"data_stored":1092985,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509551,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32336,"kb_used_data":1144,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339504,"statfs":{"total":96636764160,"available":96603652096,"internally_reserved":0,"allocated":1171456,"data_stored":502575,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":15,"seq":64424509550,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32352,"kb_used_data":1172,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339488,"statfs":{"total":96636764160,"available":96603635712,"internally_reserved":0,"allocated":1200128,"data_stored":510541,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509549,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32324,"kb_used_data":1144,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339516,"statfs":{"total":96636764160,"available":96603664384,"internally_reserved":0,"allocated":1171456,"data_stored":502539,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542255,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32328,"kb_used_data":1148,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339512,"statfs":{"total":96636764160,"available":96603660288,"internally_reserved":0,"allocated":1175552,"data_stored":503157,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542255,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32340,"kb_used_data":1156,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339500,"statfs":{"total":96636764160,"available":96603648000,"internally_reserved":0,"allocated":1183744,"data_stored":504139,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542255,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32324,"kb_used_data":1140,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339516,"statfs":{"total":96636764160,"available":96603664384,"internally_reserved":0,"allocated":1167360,"data_stored":502505,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542255,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32916,"kb_used_data":1724,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338924,"statfs":{"total":96636764160,"available":96603058176,"internally_reserved":0,"allocated":1765376,"data_stored":1093455,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-31T15:12:21.068 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-31T15:12:21.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:21.251+0000 7ff7f6000640 1 Processor -- start 2024-03-31T15:12:21.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:21.251+0000 7ff7f6000640 1 -- start start 2024-03-31T15:12:21.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:21.251+0000 7ff7f6000640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff7f00d1280 0x7ff7f00d1660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:21.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:21.251+0000 7ff7f6000640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff7f00b9c60 0x7ff7f00b5d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:21.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:21.251+0000 7ff7f6000640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff7f00d1ba0 con 0x7ff7f00b6290 2024-03-31T15:12:21.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:21.251+0000 7ff7f6000640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7ff7f00d26a0 con 0x7ff7f00b9c60 2024-03-31T15:12:21.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:21.251+0000 7ff7f6000640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7ff7f00d3990 con 0x7ff7f00d1280 2024-03-31T15:12:21.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:21.255+0000 7ff7e7e00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff7f00b9c60 0x7ff7f00b5d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:21.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:21.255+0000 7ff7efe00640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7ff7f00b6290 0x7ff7f00b7470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:54956/0 (socket says 172.21.15.16:54956) 2024-03-31T15:12:21.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:21.255+0000 7ff7efe00640 1 -- 172.21.15.16:0/1947860063 learned_addr learned my addr 172.21.15.16:0/1947860063 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:12:21.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:21.255+0000 7ff7ef400640 1 --2- 172.21.15.16:0/1947860063 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff7f00d1280 0x7ff7f00d1660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:21.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:21.255+0000 7ff7eea00640 1 -- 172.21.15.16:0/1947860063 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3034186203 0 0) 0x7ff7f00d1ba0 con 0x7ff7f00b6290 2024-03-31T15:12:21.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:21.255+0000 7ff7e7e00640 1 -- 172.21.15.16:0/1947860063 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff7f00d1280 msgr2=0x7ff7f00d1660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:12:21.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:21.255+0000 7ff7e7e00640 1 --2- 172.21.15.16:0/1947860063 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff7f00d1280 0x7ff7f00d1660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:21.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:21.255+0000 7ff7e7e00640 1 -- 172.21.15.16:0/1947860063 >> v1:172.21.15.16:6789/0 conn(0x7ff7f00b6290 legacy=0x7ff7f00b7470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:21.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:21.255+0000 7ff7e7e00640 1 -- 172.21.15.16:0/1947860063 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff7f00d28f0 con 0x7ff7f00b9c60 2024-03-31T15:12:21.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:21.255+0000 7ff7e7e00640 1 --2- 172.21.15.16:0/1947860063 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff7f00b9c60 0x7ff7f00b5d50 secure :-1 s=READY pgs=1411 cs=0 l=1 rev1=1 crypto rx=0x7ff7d80097c0 tx=0x7ff7d801c410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d83e22b1cdba8bc1 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:12:21.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:21.255+0000 7ff7eea00640 1 -- 172.21.15.16:0/1947860063 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7d80b0070 con 0x7ff7f00b9c60 2024-03-31T15:12:21.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:21.255+0000 7ff7eea00640 1 -- 172.21.15.16:0/1947860063 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff7d80ab600 con 0x7ff7f00b9c60 2024-03-31T15:12:21.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:21.255+0000 7ff7eea00640 1 -- 172.21.15.16:0/1947860063 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7d80ab9d0 con 0x7ff7f00b9c60 2024-03-31T15:12:21.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:21.259+0000 7ff7f6000640 1 -- 172.21.15.16:0/1947860063 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff7f00b9c60 msgr2=0x7ff7f00b5d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:21.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:21.259+0000 7ff7f6000640 1 --2- 172.21.15.16:0/1947860063 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff7f00b9c60 0x7ff7f00b5d50 secure :-1 s=READY pgs=1411 cs=0 l=1 rev1=1 crypto rx=0x7ff7d80097c0 tx=0x7ff7d801c410 comp rx=0 tx=0).stop 2024-03-31T15:12:21.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:21.259+0000 7ff7f6000640 1 -- 172.21.15.16:0/1947860063 shutdown_connections 2024-03-31T15:12:21.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:21.259+0000 7ff7f6000640 1 --2- 172.21.15.16:0/1947860063 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff7f00b9c60 0x7ff7f00b5d50 unknown :-1 s=CLOSED pgs=1411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:21.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:21.259+0000 7ff7f6000640 1 --2- 172.21.15.16:0/1947860063 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff7f00d1280 0x7ff7f00d1660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:21.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:21.259+0000 7ff7f6000640 1 -- 172.21.15.16:0/1947860063 >> 172.21.15.16:0/1947860063 conn(0x7ff7f0067730 msgr2=0x7ff7f0067b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:12:21.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:21.259+0000 7ff7f6000640 1 -- 172.21.15.16:0/1947860063 shutdown_connections 2024-03-31T15:12:21.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:21.259+0000 7ff7f6000640 1 -- 172.21.15.16:0/1947860063 wait complete. 2024-03-31T15:12:21.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:21.259+0000 7ff7f6000640 1 Processor -- start 2024-03-31T15:12:21.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:21.259+0000 7ff7f6000640 1 -- start start 2024-03-31T15:12:21.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:21.259+0000 7ff7f6000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff7f00b6290 0x7ff7f015a1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:21.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:21.259+0000 7ff7f6000640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff7f00b9c60 0x7ff7f01583e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:21.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:21.259+0000 7ff7f6000640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff7f00d1280 0x7ff7f00cd310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:21.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:21.259+0000 7ff7f6000640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7ff7f00b6ac0 con 0x7ff7f00b6290 2024-03-31T15:12:21.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:21.259+0000 7ff7f6000640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7ff7f0161d80 con 0x7ff7f00b9c60 2024-03-31T15:12:21.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:21.259+0000 7ff7f6000640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7ff7f00d3e90 con 0x7ff7f00d1280 2024-03-31T15:12:21.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:21.259+0000 7ff7ef400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff7f00b6290 0x7ff7f015a1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:21.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:21.259+0000 7ff7ef400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff7f00b6290 0x7ff7f015a1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40274/0 (socket says 172.21.15.16:40274) 2024-03-31T15:12:21.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:21.259+0000 7ff7ef400640 1 -- 172.21.15.16:0/1489290271 learned_addr learned my addr 172.21.15.16:0/1489290271 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:12:21.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:21.259+0000 7ff7e7e00640 1 --2- 172.21.15.16:0/1489290271 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff7f00b9c60 0x7ff7f01583e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:21.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:21.259+0000 7ff7efe00640 1 --2- 172.21.15.16:0/1489290271 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff7f00d1280 0x7ff7f00cd310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:21.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:21.259+0000 7ff7ef400640 1 -- 172.21.15.16:0/1489290271 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff7f00d1280 msgr2=0x7ff7f00cd310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:21.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:21.259+0000 7ff7ef400640 1 --2- 172.21.15.16:0/1489290271 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff7f00d1280 0x7ff7f00cd310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:21.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:21.259+0000 7ff7ef400640 1 -- 172.21.15.16:0/1489290271 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff7f00b9c60 msgr2=0x7ff7f01583e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:21.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:21.259+0000 7ff7ef400640 1 --2- 172.21.15.16:0/1489290271 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff7f00b9c60 0x7ff7f01583e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:21.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:21.259+0000 7ff7ef400640 1 -- 172.21.15.16:0/1489290271 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff7f00cdee0 con 0x7ff7f00b6290 2024-03-31T15:12:21.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:21.259+0000 7ff7efe00640 1 --2- 172.21.15.16:0/1489290271 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff7f00d1280 0x7ff7f00cd310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:12:21.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:21.263+0000 7ff7ef400640 1 --2- 172.21.15.16:0/1489290271 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff7f00b6290 0x7ff7f015a1c0 secure :-1 s=READY pgs=1412 cs=0 l=1 rev1=1 crypto rx=0x7ff7e000d5f0 tx=0x7ff7e000dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:12:21.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:21.263+0000 7ff7e7400640 1 -- 172.21.15.16:0/1489290271 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7e00125f0 con 0x7ff7f00b6290 2024-03-31T15:12:21.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:21.263+0000 7ff7e7400640 1 -- 172.21.15.16:0/1489290271 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff7e0019070 con 0x7ff7f00b6290 2024-03-31T15:12:21.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:21.263+0000 7ff7f6000640 1 -- 172.21.15.16:0/1489290271 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff7f00ce1c0 con 0x7ff7f00b6290 2024-03-31T15:12:21.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:21.263+0000 7ff7e7400640 1 -- 172.21.15.16:0/1489290271 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7e00129b0 con 0x7ff7f00b6290 2024-03-31T15:12:21.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:21.263+0000 7ff7f6000640 1 -- 172.21.15.16:0/1489290271 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff7f00cc090 con 0x7ff7f00b6290 2024-03-31T15:12:21.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:21.263+0000 7ff7f6000640 1 -- 172.21.15.16:0/1489290271 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff7f0069950 con 0x7ff7f00b6290 2024-03-31T15:12:21.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:21.263+0000 7ff7e7400640 1 -- 172.21.15.16:0/1489290271 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7ff7e0012b50 con 0x7ff7f00b6290 2024-03-31T15:12:21.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:21.263+0000 7ff7e7400640 1 --2- 172.21.15.16:0/1489290271 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7ff7cc0683b0 0x7ff7cc06a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:21.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:21.263+0000 7ff7e7400640 1 -- 172.21.15.16:0/1489290271 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 5709+0+0 (secure 0 0 0) 0x7ff7e0086ee0 con 0x7ff7f00b6290 2024-03-31T15:12:21.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:21.263+0000 7ff7e7e00640 1 --2- 172.21.15.16:0/1489290271 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7ff7cc0683b0 0x7ff7cc06a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:21.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:21.267+0000 7ff7e7e00640 1 --2- 172.21.15.16:0/1489290271 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7ff7cc0683b0 0x7ff7cc06a870 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7ff7d80ad000 tx=0x7ff7d8008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:12:21.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:21.267+0000 7ff7e7400640 1 -- 172.21.15.16:0/1489290271 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7ff7e0057380 con 0x7ff7f00b6290 2024-03-31T15:12:21.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:21.431+0000 7ff7f6000640 1 -- 172.21.15.16:0/1489290271 --> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7ff7f00d4e90 con 0x7ff7cc0683b0 2024-03-31T15:12:21.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:21.435+0000 7ff7e7400640 1 -- 172.21.15.16:0/1489290271 <== mgr.4100 v2:172.21.15.89:6832/1549407273 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252040 (secure 0 0 0) 0x7ff7f00d4e90 con 0x7ff7cc0683b0 2024-03-31T15:12:21.432 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:12:21.433 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-03-31T15:12:21.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:21.439+0000 7ff7f6000640 1 -- 172.21.15.16:0/1489290271 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7ff7cc0683b0 msgr2=0x7ff7cc06a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:21.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:21.439+0000 7ff7f6000640 1 --2- 172.21.15.16:0/1489290271 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7ff7cc0683b0 0x7ff7cc06a870 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7ff7d80ad000 tx=0x7ff7d8008040 comp rx=0 tx=0).stop 2024-03-31T15:12:21.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:21.439+0000 7ff7f6000640 1 -- 172.21.15.16:0/1489290271 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff7f00b6290 msgr2=0x7ff7f015a1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:21.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:21.439+0000 7ff7f6000640 1 --2- 172.21.15.16:0/1489290271 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff7f00b6290 0x7ff7f015a1c0 secure :-1 s=READY pgs=1412 cs=0 l=1 rev1=1 crypto rx=0x7ff7e000d5f0 tx=0x7ff7e000dab0 comp rx=0 tx=0).stop 2024-03-31T15:12:21.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:21.439+0000 7ff7f6000640 1 -- 172.21.15.16:0/1489290271 shutdown_connections 2024-03-31T15:12:21.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:21.439+0000 7ff7f6000640 1 --2- 172.21.15.16:0/1489290271 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7ff7cc0683b0 0x7ff7cc06a870 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:21.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:21.439+0000 7ff7f6000640 1 --2- 172.21.15.16:0/1489290271 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff7f00d1280 0x7ff7f00cd310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:21.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:21.439+0000 7ff7f6000640 1 --2- 172.21.15.16:0/1489290271 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff7f00b9c60 0x7ff7f01583e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:21.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:21.439+0000 7ff7f6000640 1 --2- 172.21.15.16:0/1489290271 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff7f00b6290 0x7ff7f015a1c0 unknown :-1 s=CLOSED pgs=1412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:21.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:21.439+0000 7ff7f6000640 1 -- 172.21.15.16:0/1489290271 >> 172.21.15.16:0/1489290271 conn(0x7ff7f0067730 msgr2=0x7ff7f00b5a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:12:21.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:21.439+0000 7ff7f6000640 1 -- 172.21.15.16:0/1489290271 shutdown_connections 2024-03-31T15:12:21.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:21.439+0000 7ff7f6000640 1 -- 172.21.15.16:0/1489290271 wait complete. 2024-03-31T15:12:21.453 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":358,"stamp":"2024-03-31T15:12:20.662404+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":151,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":259832,"kb_used_data":10356,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754714888,"statfs":{"total":773094113280,"available":772828045312,"internally_reserved":0,"allocated":10604544,"data_stored":5211896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":255315504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.782412"},"pg_stats":[{"pgid":"15.3f","version":"0'0","reported_seq":9,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T15:12:16.921615+0000","last_change":"2024-03-31T15:12:16.914727+0000","last_active":"2024-03-31T15:12:16.921615+0000","last_peered":"2024-03-31T15:12:16.921615+0000","last_clean":"2024-03-31T15:12:16.921615+0000","last_became_active":"2024-03-31T15:12:16.914453+0000","last_became_peered":"2024-03-31T15:12:16.914453+0000","last_unstale":"2024-03-31T15:12:16.921615+0000","last_undegraded":"2024-03-31T15:12:16.921615+0000","last_fullsized":"2024-03-31T15:12:16.921615+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_clean_scrub_stamp":"2024-03-31T15:12:15.875521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:07:27.355660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.3e","version":"0'0","reported_seq":9,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T15:12:16.904780+0000","last_change":"2024-03-31T15:12:16.898785+0000","last_active":"2024-03-31T15:12:16.904780+0000","last_peered":"2024-03-31T15:12:16.904780+0000","last_clean":"2024-03-31T15:12:16.904780+0000","last_became_active":"2024-03-31T15:12:16.898507+0000","last_became_peered":"2024-03-31T15:12:16.898507+0000","last_unstale":"2024-03-31T15:12:16.904780+0000","last_undegraded":"2024-03-31T15:12:16.904780+0000","last_fullsized":"2024-03-31T15:12:16.904780+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_clean_scrub_stamp":"2024-03-31T15:12:15.875521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T15:30:58.977979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.3d","version":"0'0","reported_seq":9,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T15:12:16.921727+0000","last_change":"2024-03-31T15:12:16.912813+0000","last_active":"2024-03-31T15:12:16.921727+0000","last_peered":"2024-03-31T15:12:16.921727+0000","last_clean":"2024-03-31T15:12:16.921727+0000","last_became_active":"2024-03-31T15:12:16.912410+0000","last_became_peered":"2024-03-31T15:12:16.912410+0000","last_unstale":"2024-03-31T15:12:16.921727+0000","last_undegraded":"2024-03-31T15:12:16.921727+0000","last_fullsized":"2024-03-31T15:12:16.921727+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_clean_scrub_stamp":"2024-03-31T15:12:15.875521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:45:27.907983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.3c","version":"0'0","reported_seq":9,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T15:12:16.919587+0000","last_change":"2024-03-31T15:12:16.910460+0000","last_active":"2024-03-31T15:12:16.919587+0000","last_peered":"2024-03-31T15:12:16.919587+0000","last_clean":"2024-03-31T15:12:16.919587+0000","last_became_active":"2024-03-31T15:12:16.909960+0000","last_became_peered":"2024-03-31T15:12:16.909960+0000","last_unstale":"2024-03-31T15:12:16.919587+0000","last_undegraded":"2024-03-31T15:12:16.919587+0000","last_fullsized":"2024-03-31T15:12:16.919587+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_clean_scrub_stamp":"2024-03-31T15:12:15.875521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:18:26.495815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.3b","version":"0'0","reported_seq":9,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T15:12:16.909003+0000","last_change":"2024-03-31T15:12:16.903384+0000","last_active":"2024-03-31T15:12:16.909003+0000","last_peered":"2024-03-31T15:12:16.909003+0000","last_clean":"2024-03-31T15:12:16.909003+0000","last_became_active":"2024-03-31T15:12:16.903110+0000","last_became_peered":"2024-03-31T15:12:16.903110+0000","last_unstale":"2024-03-31T15:12:16.909003+0000","last_undegraded":"2024-03-31T15:12:16.909003+0000","last_fullsized":"2024-03-31T15:12:16.909003+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_clean_scrub_stamp":"2024-03-31T15:12:15.875521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:16:08.022126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.3a","version":"0'0","reported_seq":9,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T15:12:16.917157+0000","last_change":"2024-03-31T15:12:16.915254+0000","last_active":"2024-03-31T15:12:16.917157+0000","last_peered":"2024-03-31T15:12:16.917157+0000","last_clean":"2024-03-31T15:12:16.917157+0000","last_became_active":"2024-03-31T15:12:16.912283+0000","last_became_peered":"2024-03-31T15:12:16.912283+0000","last_unstale":"2024-03-31T15:12:16.917157+0000","last_undegraded":"2024-03-31T15:12:16.917157+0000","last_fullsized":"2024-03-31T15:12:16.917157+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_clean_scrub_stamp":"2024-03-31T15:12:15.875521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:57:55.489821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.39","version":"0'0","reported_seq":9,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T15:12:16.915004+0000","last_change":"2024-03-31T15:12:16.909380+0000","last_active":"2024-03-31T15:12:16.915004+0000","last_peered":"2024-03-31T15:12:16.915004+0000","last_clean":"2024-03-31T15:12:16.915004+0000","last_became_active":"2024-03-31T15:12:16.907492+0000","last_became_peered":"2024-03-31T15:12:16.907492+0000","last_unstale":"2024-03-31T15:12:16.915004+0000","last_undegraded":"2024-03-31T15:12:16.915004+0000","last_fullsized":"2024-03-31T15:12:16.915004+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_clean_scrub_stamp":"2024-03-31T15:12:15.875521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:43:16.026939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.38","version":"0'0","reported_seq":9,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T15:12:16.913649+0000","last_change":"2024-03-31T15:12:16.907968+0000","last_active":"2024-03-31T15:12:16.913649+0000","last_peered":"2024-03-31T15:12:16.913649+0000","last_clean":"2024-03-31T15:12:16.913649+0000","last_became_active":"2024-03-31T15:12:16.907679+0000","last_became_peered":"2024-03-31T15:12:16.907679+0000","last_unstale":"2024-03-31T15:12:16.913649+0000","last_undegraded":"2024-03-31T15:12:16.913649+0000","last_fullsized":"2024-03-31T15:12:16.913649+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_clean_scrub_stamp":"2024-03-31T15:12:15.875521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:07:57.538752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.37","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-31T15:12:17.894969+0000","last_change":"2024-03-31T15:12:16.913705+0000","last_active":"2024-03-31T15:12:17.894969+0000","last_peered":"2024-03-31T15:12:17.894969+0000","last_clean":"2024-03-31T15:12:17.894969+0000","last_became_active":"2024-03-31T15:12:16.911556+0000","last_became_peered":"2024-03-31T15:12:16.911556+0000","last_unstale":"2024-03-31T15:12:17.894969+0000","last_undegraded":"2024-03-31T15:12:17.894969+0000","last_fullsized":"2024-03-31T15:12:17.894969+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_clean_scrub_stamp":"2024-03-31T15:12:15.875521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:19:07.639622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.36","version":"0'0","reported_seq":9,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T15:12:16.916938+0000","last_change":"2024-03-31T15:12:16.915079+0000","last_active":"2024-03-31T15:12:16.916938+0000","last_peered":"2024-03-31T15:12:16.916938+0000","last_clean":"2024-03-31T15:12:16.916938+0000","last_became_active":"2024-03-31T15:12:16.912820+0000","last_became_peered":"2024-03-31T15:12:16.912820+0000","last_unstale":"2024-03-31T15:12:16.916938+0000","last_undegraded":"2024-03-31T15:12:16.916938+0000","last_fullsized":"2024-03-31T15:12:16.916938+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_clean_scrub_stamp":"2024-03-31T15:12:15.875521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:05:22.446118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.35","version":"0'0","reported_seq":9,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T15:12:16.915049+0000","last_change":"2024-03-31T15:12:16.909343+0000","last_active":"2024-03-31T15:12:16.915049+0000","last_peered":"2024-03-31T15:12:16.915049+0000","last_clean":"2024-03-31T15:12:16.915049+0000","last_became_active":"2024-03-31T15:12:16.908516+0000","last_became_peered":"2024-03-31T15:12:16.908516+0000","last_unstale":"2024-03-31T15:12:16.915049+0000","last_undegraded":"2024-03-31T15:12:16.915049+0000","last_fullsized":"2024-03-31T15:12:16.915049+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_clean_scrub_stamp":"2024-03-31T15:12:15.875521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:41:17.901423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.34","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-31T15:12:17.894874+0000","last_change":"2024-03-31T15:12:16.914958+0000","last_active":"2024-03-31T15:12:17.894874+0000","last_peered":"2024-03-31T15:12:17.894874+0000","last_clean":"2024-03-31T15:12:17.894874+0000","last_became_active":"2024-03-31T15:12:16.914668+0000","last_became_peered":"2024-03-31T15:12:16.914668+0000","last_unstale":"2024-03-31T15:12:17.894874+0000","last_undegraded":"2024-03-31T15:12:17.894874+0000","last_fullsized":"2024-03-31T15:12:17.894874+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_clean_scrub_stamp":"2024-03-31T15:12:15.875521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:41:17.901423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.33","version":"0'0","reported_seq":9,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T15:12:16.910842+0000","last_change":"2024-03-31T15:12:16.906096+0000","last_active":"2024-03-31T15:12:16.910842+0000","last_peered":"2024-03-31T15:12:16.910842+0000","last_clean":"2024-03-31T15:12:16.910842+0000","last_became_active":"2024-03-31T15:12:16.904668+0000","last_became_peered":"2024-03-31T15:12:16.904668+0000","last_unstale":"2024-03-31T15:12:16.910842+0000","last_undegraded":"2024-03-31T15:12:16.910842+0000","last_fullsized":"2024-03-31T15:12:16.910842+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_clean_scrub_stamp":"2024-03-31T15:12:15.875521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:23:34.095265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.32","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-31T15:12:17.889926+0000","last_change":"2024-03-31T15:12:16.903426+0000","last_active":"2024-03-31T15:12:17.889926+0000","last_peered":"2024-03-31T15:12:17.889926+0000","last_clean":"2024-03-31T15:12:17.889926+0000","last_became_active":"2024-03-31T15:12:16.903150+0000","last_became_peered":"2024-03-31T15:12:16.903150+0000","last_unstale":"2024-03-31T15:12:17.889926+0000","last_undegraded":"2024-03-31T15:12:17.889926+0000","last_fullsized":"2024-03-31T15:12:17.889926+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_clean_scrub_stamp":"2024-03-31T15:12:15.875521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:23:48.485186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.31","version":"0'0","reported_seq":9,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T15:12:16.919038+0000","last_change":"2024-03-31T15:12:16.915738+0000","last_active":"2024-03-31T15:12:16.919038+0000","last_peered":"2024-03-31T15:12:16.919038+0000","last_clean":"2024-03-31T15:12:16.919038+0000","last_became_active":"2024-03-31T15:12:16.914246+0000","last_became_peered":"2024-03-31T15:12:16.914246+0000","last_unstale":"2024-03-31T15:12:16.919038+0000","last_undegraded":"2024-03-31T15:12:16.919038+0000","last_fullsized":"2024-03-31T15:12:16.919038+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_clean_scrub_stamp":"2024-03-31T15:12:15.875521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:50:07.394964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.30","version":"0'0","reported_seq":9,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T15:12:16.916968+0000","last_change":"2024-03-31T15:12:16.913825+0000","last_active":"2024-03-31T15:12:16.916968+0000","last_peered":"2024-03-31T15:12:16.916968+0000","last_clean":"2024-03-31T15:12:16.916968+0000","last_became_active":"2024-03-31T15:12:16.913562+0000","last_became_peered":"2024-03-31T15:12:16.913562+0000","last_unstale":"2024-03-31T15:12:16.916968+0000","last_undegraded":"2024-03-31T15:12:16.916968+0000","last_fullsized":"2024-03-31T15:12:16.916968+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_clean_scrub_stamp":"2024-03-31T15:12:15.875521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:52:43.591702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.2f","version":"0'0","reported_seq":9,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T15:12:16.912965+0000","last_change":"2024-03-31T15:12:16.906873+0000","last_active":"2024-03-31T15:12:16.912965+0000","last_peered":"2024-03-31T15:12:16.912965+0000","last_clean":"2024-03-31T15:12:16.912965+0000","last_became_active":"2024-03-31T15:12:16.906618+0000","last_became_peered":"2024-03-31T15:12:16.906618+0000","last_unstale":"2024-03-31T15:12:16.912965+0000","last_undegraded":"2024-03-31T15:12:16.912965+0000","last_fullsized":"2024-03-31T15:12:16.912965+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_clean_scrub_stamp":"2024-03-31T15:12:15.875521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:01:25.766417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.2e","version":"0'0","reported_seq":9,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T15:12:16.918958+0000","last_change":"2024-03-31T15:12:16.910444+0000","last_active":"2024-03-31T15:12:16.918958+0000","last_peered":"2024-03-31T15:12:16.918958+0000","last_clean":"2024-03-31T15:12:16.918958+0000","last_became_active":"2024-03-31T15:12:16.910163+0000","last_became_peered":"2024-03-31T15:12:16.910163+0000","last_unstale":"2024-03-31T15:12:16.918958+0000","last_undegraded":"2024-03-31T15:12:16.918958+0000","last_fullsized":"2024-03-31T15:12:16.918958+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_clean_scrub_stamp":"2024-03-31T15:12:15.875521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:07:57.538752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.2d","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-31T15:12:17.890226+0000","last_change":"2024-03-31T15:12:16.920057+0000","last_active":"2024-03-31T15:12:17.890226+0000","last_peered":"2024-03-31T15:12:17.890226+0000","last_clean":"2024-03-31T15:12:17.890226+0000","last_became_active":"2024-03-31T15:12:16.919496+0000","last_became_peered":"2024-03-31T15:12:16.919496+0000","last_unstale":"2024-03-31T15:12:17.890226+0000","last_undegraded":"2024-03-31T15:12:17.890226+0000","last_fullsized":"2024-03-31T15:12:17.890226+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_clean_scrub_stamp":"2024-03-31T15:12:15.875521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:07:55.682491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.2c","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-31T15:12:17.894800+0000","last_change":"2024-03-31T15:12:16.913343+0000","last_active":"2024-03-31T15:12:17.894800+0000","last_peered":"2024-03-31T15:12:17.894800+0000","last_clean":"2024-03-31T15:12:17.894800+0000","last_became_active":"2024-03-31T15:12:16.911677+0000","last_became_peered":"2024-03-31T15:12:16.911677+0000","last_unstale":"2024-03-31T15:12:17.894800+0000","last_undegraded":"2024-03-31T15:12:17.894800+0000","last_fullsized":"2024-03-31T15:12:17.894800+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_clean_scrub_stamp":"2024-03-31T15:12:15.875521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:28:01.920854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.2b","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-31T15:12:17.890038+0000","last_change":"2024-03-31T15:12:16.898095+0000","last_active":"2024-03-31T15:12:17.890038+0000","last_peered":"2024-03-31T15:12:17.890038+0000","last_clean":"2024-03-31T15:12:17.890038+0000","last_became_active":"2024-03-31T15:12:16.897819+0000","last_became_peered":"2024-03-31T15:12:16.897819+0000","last_unstale":"2024-03-31T15:12:17.890038+0000","last_undegraded":"2024-03-31T15:12:17.890038+0000","last_fullsized":"2024-03-31T15:12:17.890038+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_clean_scrub_stamp":"2024-03-31T15:12:15.875521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:46:47.981809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.2a","version":"0'0","reported_seq":9,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T15:12:16.904854+0000","last_change":"2024-03-31T15:12:16.899377+0000","last_active":"2024-03-31T15:12:16.904854+0000","last_peered":"2024-03-31T15:12:16.904854+0000","last_clean":"2024-03-31T15:12:16.904854+0000","last_became_active":"2024-03-31T15:12:16.898453+0000","last_became_peered":"2024-03-31T15:12:16.898453+0000","last_unstale":"2024-03-31T15:12:16.904854+0000","last_undegraded":"2024-03-31T15:12:16.904854+0000","last_fullsized":"2024-03-31T15:12:16.904854+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_clean_scrub_stamp":"2024-03-31T15:12:15.875521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:23:34.095265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.29","version":"0'0","reported_seq":9,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T15:12:16.912599+0000","last_change":"2024-03-31T15:12:16.903860+0000","last_active":"2024-03-31T15:12:16.912599+0000","last_peered":"2024-03-31T15:12:16.912599+0000","last_clean":"2024-03-31T15:12:16.912599+0000","last_became_active":"2024-03-31T15:12:16.903601+0000","last_became_peered":"2024-03-31T15:12:16.903601+0000","last_unstale":"2024-03-31T15:12:16.912599+0000","last_undegraded":"2024-03-31T15:12:16.912599+0000","last_fullsized":"2024-03-31T15:12:16.912599+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_clean_scrub_stamp":"2024-03-31T15:12:15.875521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:57:29.441394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.28","version":"0'0","reported_seq":9,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T15:12:16.918364+0000","last_change":"2024-03-31T15:12:16.917075+0000","last_active":"2024-03-31T15:12:16.918364+0000","last_peered":"2024-03-31T15:12:16.918364+0000","last_clean":"2024-03-31T15:12:16.918364+0000","last_became_active":"2024-03-31T15:12:16.916550+0000","last_became_peered":"2024-03-31T15:12:16.916550+0000","last_unstale":"2024-03-31T15:12:16.918364+0000","last_undegraded":"2024-03-31T15:12:16.918364+0000","last_fullsized":"2024-03-31T15:12:16.918364+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_clean_scrub_stamp":"2024-03-31T15:12:15.875521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T17:27:02.919268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.27","version":"0'0","reported_seq":9,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T15:12:16.903723+0000","last_change":"2024-03-31T15:12:16.900790+0000","last_active":"2024-03-31T15:12:16.903723+0000","last_peered":"2024-03-31T15:12:16.903723+0000","last_clean":"2024-03-31T15:12:16.903723+0000","last_became_active":"2024-03-31T15:12:16.900476+0000","last_became_peered":"2024-03-31T15:12:16.900476+0000","last_unstale":"2024-03-31T15:12:16.903723+0000","last_undegraded":"2024-03-31T15:12:16.903723+0000","last_fullsized":"2024-03-31T15:12:16.903723+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_clean_scrub_stamp":"2024-03-31T15:12:15.875521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:07:23.831027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.26","version":"0'0","reported_seq":9,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T15:12:16.915186+0000","last_change":"2024-03-31T15:12:16.912156+0000","last_active":"2024-03-31T15:12:16.915186+0000","last_peered":"2024-03-31T15:12:16.915186+0000","last_clean":"2024-03-31T15:12:16.915186+0000","last_became_active":"2024-03-31T15:12:16.911885+0000","last_became_peered":"2024-03-31T15:12:16.911885+0000","last_unstale":"2024-03-31T15:12:16.915186+0000","last_undegraded":"2024-03-31T15:12:16.915186+0000","last_fullsized":"2024-03-31T15:12:16.915186+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_clean_scrub_stamp":"2024-03-31T15:12:15.875521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:18:26.495815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.25","version":"0'0","reported_seq":9,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T15:12:16.915101+0000","last_change":"2024-03-31T15:12:16.910892+0000","last_active":"2024-03-31T15:12:16.915101+0000","last_peered":"2024-03-31T15:12:16.915101+0000","last_clean":"2024-03-31T15:12:16.915101+0000","last_became_active":"2024-03-31T15:12:16.910350+0000","last_became_peered":"2024-03-31T15:12:16.910350+0000","last_unstale":"2024-03-31T15:12:16.915101+0000","last_undegraded":"2024-03-31T15:12:16.915101+0000","last_fullsized":"2024-03-31T15:12:16.915101+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_clean_scrub_stamp":"2024-03-31T15:12:15.875521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:53:55.735548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.24","version":"0'0","reported_seq":9,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T15:12:16.923062+0000","last_change":"2024-03-31T15:12:16.922419+0000","last_active":"2024-03-31T15:12:16.923062+0000","last_peered":"2024-03-31T15:12:16.923062+0000","last_clean":"2024-03-31T15:12:16.923062+0000","last_became_active":"2024-03-31T15:12:16.922130+0000","last_became_peered":"2024-03-31T15:12:16.922130+0000","last_unstale":"2024-03-31T15:12:16.923062+0000","last_undegraded":"2024-03-31T15:12:16.923062+0000","last_fullsized":"2024-03-31T15:12:16.923062+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_clean_scrub_stamp":"2024-03-31T15:12:15.875521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:23:34.095265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.23","version":"0'0","reported_seq":9,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T15:12:16.917448+0000","last_change":"2024-03-31T15:12:16.913049+0000","last_active":"2024-03-31T15:12:16.917448+0000","last_peered":"2024-03-31T15:12:16.917448+0000","last_clean":"2024-03-31T15:12:16.917448+0000","last_became_active":"2024-03-31T15:12:16.912271+0000","last_became_peered":"2024-03-31T15:12:16.912271+0000","last_unstale":"2024-03-31T15:12:16.917448+0000","last_undegraded":"2024-03-31T15:12:16.917448+0000","last_fullsized":"2024-03-31T15:12:16.917448+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_clean_scrub_stamp":"2024-03-31T15:12:15.875521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T17:51:59.038412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.22","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-31T15:12:17.889891+0000","last_change":"2024-03-31T15:12:16.917356+0000","last_active":"2024-03-31T15:12:17.889891+0000","last_peered":"2024-03-31T15:12:17.889891+0000","last_clean":"2024-03-31T15:12:17.889891+0000","last_became_active":"2024-03-31T15:12:16.917092+0000","last_became_peered":"2024-03-31T15:12:16.917092+0000","last_unstale":"2024-03-31T15:12:17.889891+0000","last_undegraded":"2024-03-31T15:12:17.889891+0000","last_fullsized":"2024-03-31T15:12:17.889891+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_clean_scrub_stamp":"2024-03-31T15:12:15.875521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:02:36.135837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.21","version":"0'0","reported_seq":9,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T15:12:16.907738+0000","last_change":"2024-03-31T15:12:16.902593+0000","last_active":"2024-03-31T15:12:16.907738+0000","last_peered":"2024-03-31T15:12:16.907738+0000","last_clean":"2024-03-31T15:12:16.907738+0000","last_became_active":"2024-03-31T15:12:16.902320+0000","last_became_peered":"2024-03-31T15:12:16.902320+0000","last_unstale":"2024-03-31T15:12:16.907738+0000","last_undegraded":"2024-03-31T15:12:16.907738+0000","last_fullsized":"2024-03-31T15:12:16.907738+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_clean_scrub_stamp":"2024-03-31T15:12:15.875521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T15:30:58.977979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.20","version":"0'0","reported_seq":9,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T15:12:16.920194+0000","last_change":"2024-03-31T15:12:16.912669+0000","last_active":"2024-03-31T15:12:16.920194+0000","last_peered":"2024-03-31T15:12:16.920194+0000","last_clean":"2024-03-31T15:12:16.920194+0000","last_became_active":"2024-03-31T15:12:16.908838+0000","last_became_peered":"2024-03-31T15:12:16.908838+0000","last_unstale":"2024-03-31T15:12:16.920194+0000","last_undegraded":"2024-03-31T15:12:16.920194+0000","last_fullsized":"2024-03-31T15:12:16.920194+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_clean_scrub_stamp":"2024-03-31T15:12:15.875521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:19:29.666114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.1e","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-31T15:12:17.890422+0000","last_change":"2024-03-31T15:12:16.905100+0000","last_active":"2024-03-31T15:12:17.890422+0000","last_peered":"2024-03-31T15:12:17.890422+0000","last_clean":"2024-03-31T15:12:17.890422+0000","last_became_active":"2024-03-31T15:12:16.904715+0000","last_became_peered":"2024-03-31T15:12:16.904715+0000","last_unstale":"2024-03-31T15:12:17.890422+0000","last_undegraded":"2024-03-31T15:12:17.890422+0000","last_fullsized":"2024-03-31T15:12:17.890422+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_clean_scrub_stamp":"2024-03-31T15:12:15.875521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:39:17.509700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.1f","version":"95'4","reported_seq":16,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T15:12:16.927963+0000","last_change":"2024-03-31T15:12:15.895041+0000","last_active":"2024-03-31T15:12:16.927963+0000","last_peered":"2024-03-31T15:12:16.927963+0000","last_clean":"2024-03-31T15:12:16.927963+0000","last_became_active":"2024-03-31T15:12:15.894764+0000","last_became_peered":"2024-03-31T15:12:15.894764+0000","last_unstale":"2024-03-31T15:12:16.927963+0000","last_undegraded":"2024-03-31T15:12:16.927963+0000","last_fullsized":"2024-03-31T15:12:16.927963+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:12:14.865356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:12:14.865356+0000","last_clean_scrub_stamp":"2024-03-31T15:12:14.865356+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:07:51.141948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1f","version":"0'0","reported_seq":9,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T15:12:16.918759+0000","last_change":"2024-03-31T15:12:16.908697+0000","last_active":"2024-03-31T15:12:16.918759+0000","last_peered":"2024-03-31T15:12:16.918759+0000","last_clean":"2024-03-31T15:12:16.918759+0000","last_became_active":"2024-03-31T15:12:16.907981+0000","last_became_peered":"2024-03-31T15:12:16.907981+0000","last_unstale":"2024-03-31T15:12:16.918759+0000","last_undegraded":"2024-03-31T15:12:16.918759+0000","last_fullsized":"2024-03-31T15:12:16.918759+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_clean_scrub_stamp":"2024-03-31T15:12:15.875521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:01:25.766417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.1e","version":"95'1","reported_seq":13,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T15:12:16.929532+0000","last_change":"2024-03-31T15:12:15.913830+0000","last_active":"2024-03-31T15:12:16.929532+0000","last_peered":"2024-03-31T15:12:16.929532+0000","last_clean":"2024-03-31T15:12:16.929532+0000","last_became_active":"2024-03-31T15:12:15.913523+0000","last_became_peered":"2024-03-31T15:12:15.913523+0000","last_unstale":"2024-03-31T15:12:16.929532+0000","last_undegraded":"2024-03-31T15:12:16.929532+0000","last_fullsized":"2024-03-31T15:12:16.929532+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:12:14.865356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:12:14.865356+0000","last_clean_scrub_stamp":"2024-03-31T15:12:14.865356+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:45:37.084308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.1c","version":"0'0","reported_seq":9,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T15:12:16.920101+0000","last_change":"2024-03-31T15:12:16.909505+0000","last_active":"2024-03-31T15:12:16.920101+0000","last_peered":"2024-03-31T15:12:16.920101+0000","last_clean":"2024-03-31T15:12:16.920101+0000","last_became_active":"2024-03-31T15:12:16.909207+0000","last_became_peered":"2024-03-31T15:12:16.909207+0000","last_unstale":"2024-03-31T15:12:16.920101+0000","last_undegraded":"2024-03-31T15:12:16.920101+0000","last_fullsized":"2024-03-31T15:12:16.920101+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_clean_scrub_stamp":"2024-03-31T15:12:15.875521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:17:19.301953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.1d","version":"95'2","reported_seq":14,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T15:12:16.928839+0000","last_change":"2024-03-31T15:12:15.903944+0000","last_active":"2024-03-31T15:12:16.928839+0000","last_peered":"2024-03-31T15:12:16.928839+0000","last_clean":"2024-03-31T15:12:16.928839+0000","last_became_active":"2024-03-31T15:12:15.901604+0000","last_became_peered":"2024-03-31T15:12:15.901604+0000","last_unstale":"2024-03-31T15:12:16.928839+0000","last_undegraded":"2024-03-31T15:12:16.928839+0000","last_fullsized":"2024-03-31T15:12:16.928839+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:12:14.865356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:12:14.865356+0000","last_clean_scrub_stamp":"2024-03-31T15:12:14.865356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T15:16:54.328335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1d","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-31T15:12:17.895025+0000","last_change":"2024-03-31T15:12:16.906139+0000","last_active":"2024-03-31T15:12:17.895025+0000","last_peered":"2024-03-31T15:12:17.895025+0000","last_clean":"2024-03-31T15:12:17.895025+0000","last_became_active":"2024-03-31T15:12:16.905695+0000","last_became_peered":"2024-03-31T15:12:16.905695+0000","last_unstale":"2024-03-31T15:12:17.895025+0000","last_undegraded":"2024-03-31T15:12:17.895025+0000","last_fullsized":"2024-03-31T15:12:17.895025+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_clean_scrub_stamp":"2024-03-31T15:12:15.875521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:52:43.591702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.1c","version":"95'2","reported_seq":14,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T15:12:16.929952+0000","last_change":"2024-03-31T15:12:15.902931+0000","last_active":"2024-03-31T15:12:16.929952+0000","last_peered":"2024-03-31T15:12:16.929952+0000","last_clean":"2024-03-31T15:12:16.929952+0000","last_became_active":"2024-03-31T15:12:15.902660+0000","last_became_peered":"2024-03-31T15:12:15.902660+0000","last_unstale":"2024-03-31T15:12:16.929952+0000","last_undegraded":"2024-03-31T15:12:16.929952+0000","last_fullsized":"2024-03-31T15:12:16.929952+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:12:14.865356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:12:14.865356+0000","last_clean_scrub_stamp":"2024-03-31T15:12:14.865356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:07:22.820863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1a","version":"0'0","reported_seq":9,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T15:12:16.914541+0000","last_change":"2024-03-31T15:12:16.910215+0000","last_active":"2024-03-31T15:12:16.914541+0000","last_peered":"2024-03-31T15:12:16.914541+0000","last_clean":"2024-03-31T15:12:16.914541+0000","last_became_active":"2024-03-31T15:12:16.909223+0000","last_became_peered":"2024-03-31T15:12:16.909223+0000","last_unstale":"2024-03-31T15:12:16.914541+0000","last_undegraded":"2024-03-31T15:12:16.914541+0000","last_fullsized":"2024-03-31T15:12:16.914541+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_clean_scrub_stamp":"2024-03-31T15:12:15.875521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:33:59.722746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.1b","version":"95'1","reported_seq":13,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T15:12:16.927579+0000","last_change":"2024-03-31T15:12:15.902342+0000","last_active":"2024-03-31T15:12:16.927579+0000","last_peered":"2024-03-31T15:12:16.927579+0000","last_clean":"2024-03-31T15:12:16.927579+0000","last_became_active":"2024-03-31T15:12:15.901995+0000","last_became_peered":"2024-03-31T15:12:15.901995+0000","last_unstale":"2024-03-31T15:12:16.927579+0000","last_undegraded":"2024-03-31T15:12:16.927579+0000","last_fullsized":"2024-03-31T15:12:16.927579+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:12:14.865356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:12:14.865356+0000","last_clean_scrub_stamp":"2024-03-31T15:12:14.865356+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:52:20.718978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.1b","version":"0'0","reported_seq":9,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T15:12:16.917998+0000","last_change":"2024-03-31T15:12:16.916208+0000","last_active":"2024-03-31T15:12:16.917998+0000","last_peered":"2024-03-31T15:12:16.917998+0000","last_clean":"2024-03-31T15:12:16.917998+0000","last_became_active":"2024-03-31T15:12:16.915935+0000","last_became_peered":"2024-03-31T15:12:16.915935+0000","last_unstale":"2024-03-31T15:12:16.917998+0000","last_undegraded":"2024-03-31T15:12:16.917998+0000","last_fullsized":"2024-03-31T15:12:16.917998+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_clean_scrub_stamp":"2024-03-31T15:12:15.875521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:28:48.945128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.1a","version":"95'1","reported_seq":13,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T15:12:16.928483+0000","last_change":"2024-03-31T15:12:15.907871+0000","last_active":"2024-03-31T15:12:16.928483+0000","last_peered":"2024-03-31T15:12:16.928483+0000","last_clean":"2024-03-31T15:12:16.928483+0000","last_became_active":"2024-03-31T15:12:15.907391+0000","last_became_peered":"2024-03-31T15:12:15.907391+0000","last_unstale":"2024-03-31T15:12:16.928483+0000","last_undegraded":"2024-03-31T15:12:16.928483+0000","last_fullsized":"2024-03-31T15:12:16.928483+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:12:14.865356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:12:14.865356+0000","last_clean_scrub_stamp":"2024-03-31T15:12:14.865356+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:08:53.221389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.18","version":"0'0","reported_seq":9,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T15:12:16.919262+0000","last_change":"2024-03-31T15:12:16.915607+0000","last_active":"2024-03-31T15:12:16.919262+0000","last_peered":"2024-03-31T15:12:16.919262+0000","last_clean":"2024-03-31T15:12:16.919262+0000","last_became_active":"2024-03-31T15:12:16.914893+0000","last_became_peered":"2024-03-31T15:12:16.914893+0000","last_unstale":"2024-03-31T15:12:16.919262+0000","last_undegraded":"2024-03-31T15:12:16.919262+0000","last_fullsized":"2024-03-31T15:12:16.919262+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_clean_scrub_stamp":"2024-03-31T15:12:15.875521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T17:51:59.038412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.19","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T15:12:16.893425+0000","last_change":"2024-03-31T15:12:15.903998+0000","last_active":"2024-03-31T15:12:16.893425+0000","last_peered":"2024-03-31T15:12:16.893425+0000","last_clean":"2024-03-31T15:12:16.893425+0000","last_became_active":"2024-03-31T15:12:15.903460+0000","last_became_peered":"2024-03-31T15:12:15.903460+0000","last_unstale":"2024-03-31T15:12:16.893425+0000","last_undegraded":"2024-03-31T15:12:16.893425+0000","last_fullsized":"2024-03-31T15:12:16.893425+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:12:14.865356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:12:14.865356+0000","last_clean_scrub_stamp":"2024-03-31T15:12:14.865356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:01:28.193287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.19","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-31T15:12:17.894403+0000","last_change":"2024-03-31T15:12:16.907080+0000","last_active":"2024-03-31T15:12:17.894403+0000","last_peered":"2024-03-31T15:12:17.894403+0000","last_clean":"2024-03-31T15:12:17.894403+0000","last_became_active":"2024-03-31T15:12:16.905716+0000","last_became_peered":"2024-03-31T15:12:16.905716+0000","last_unstale":"2024-03-31T15:12:17.894403+0000","last_undegraded":"2024-03-31T15:12:17.894403+0000","last_fullsized":"2024-03-31T15:12:17.894403+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_clean_scrub_stamp":"2024-03-31T15:12:15.875521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:16:14.392760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.18","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T15:12:16.901868+0000","last_change":"2024-03-31T15:12:15.912850+0000","last_active":"2024-03-31T15:12:16.901868+0000","last_peered":"2024-03-31T15:12:16.901868+0000","last_clean":"2024-03-31T15:12:16.901868+0000","last_became_active":"2024-03-31T15:12:15.910394+0000","last_became_peered":"2024-03-31T15:12:15.910394+0000","last_unstale":"2024-03-31T15:12:16.901868+0000","last_undegraded":"2024-03-31T15:12:16.901868+0000","last_fullsized":"2024-03-31T15:12:16.901868+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:12:14.865356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:12:14.865356+0000","last_clean_scrub_stamp":"2024-03-31T15:12:14.865356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:23:00.399133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.16","version":"0'0","reported_seq":9,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T15:12:16.906418+0000","last_change":"2024-03-31T15:12:16.901505+0000","last_active":"2024-03-31T15:12:16.906418+0000","last_peered":"2024-03-31T15:12:16.906418+0000","last_clean":"2024-03-31T15:12:16.906418+0000","last_became_active":"2024-03-31T15:12:16.901246+0000","last_became_peered":"2024-03-31T15:12:16.901246+0000","last_unstale":"2024-03-31T15:12:16.906418+0000","last_undegraded":"2024-03-31T15:12:16.906418+0000","last_fullsized":"2024-03-31T15:12:16.906418+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_clean_scrub_stamp":"2024-03-31T15:12:15.875521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:09:39.960947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.17","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T15:12:16.903995+0000","last_change":"2024-03-31T15:12:15.912638+0000","last_active":"2024-03-31T15:12:16.903995+0000","last_peered":"2024-03-31T15:12:16.903995+0000","last_clean":"2024-03-31T15:12:16.903995+0000","last_became_active":"2024-03-31T15:12:15.912167+0000","last_became_peered":"2024-03-31T15:12:15.912167+0000","last_unstale":"2024-03-31T15:12:16.903995+0000","last_undegraded":"2024-03-31T15:12:16.903995+0000","last_fullsized":"2024-03-31T15:12:16.903995+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:12:14.865356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:12:14.865356+0000","last_clean_scrub_stamp":"2024-03-31T15:12:14.865356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:19:39.893438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.17","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-31T15:12:17.895005+0000","last_change":"2024-03-31T15:12:16.906841+0000","last_active":"2024-03-31T15:12:17.895005+0000","last_peered":"2024-03-31T15:12:17.895005+0000","last_clean":"2024-03-31T15:12:17.895005+0000","last_became_active":"2024-03-31T15:12:16.906537+0000","last_became_peered":"2024-03-31T15:12:16.906537+0000","last_unstale":"2024-03-31T15:12:17.895005+0000","last_undegraded":"2024-03-31T15:12:17.895005+0000","last_fullsized":"2024-03-31T15:12:17.895005+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_clean_scrub_stamp":"2024-03-31T15:12:15.875521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:43:58.063821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.16","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T15:12:16.906234+0000","last_change":"2024-03-31T15:12:15.911664+0000","last_active":"2024-03-31T15:12:16.906234+0000","last_peered":"2024-03-31T15:12:16.906234+0000","last_clean":"2024-03-31T15:12:16.906234+0000","last_became_active":"2024-03-31T15:12:15.911345+0000","last_became_peered":"2024-03-31T15:12:15.911345+0000","last_unstale":"2024-03-31T15:12:16.906234+0000","last_undegraded":"2024-03-31T15:12:16.906234+0000","last_fullsized":"2024-03-31T15:12:16.906234+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:12:14.865356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:12:14.865356+0000","last_clean_scrub_stamp":"2024-03-31T15:12:14.865356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:09:38.950783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.14","version":"0'0","reported_seq":9,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T15:12:16.902544+0000","last_change":"2024-03-31T15:12:16.900255+0000","last_active":"2024-03-31T15:12:16.902544+0000","last_peered":"2024-03-31T15:12:16.902544+0000","last_clean":"2024-03-31T15:12:16.902544+0000","last_became_active":"2024-03-31T15:12:16.899918+0000","last_became_peered":"2024-03-31T15:12:16.899918+0000","last_unstale":"2024-03-31T15:12:16.902544+0000","last_undegraded":"2024-03-31T15:12:16.902544+0000","last_fullsized":"2024-03-31T15:12:16.902544+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_clean_scrub_stamp":"2024-03-31T15:12:15.875521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:34:40.119670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.15","version":"95'3","reported_seq":15,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T15:12:16.931836+0000","last_change":"2024-03-31T15:12:15.909068+0000","last_active":"2024-03-31T15:12:16.931836+0000","last_peered":"2024-03-31T15:12:16.931836+0000","last_clean":"2024-03-31T15:12:16.931836+0000","last_became_active":"2024-03-31T15:12:15.908677+0000","last_became_peered":"2024-03-31T15:12:15.908677+0000","last_unstale":"2024-03-31T15:12:16.931836+0000","last_undegraded":"2024-03-31T15:12:16.931836+0000","last_fullsized":"2024-03-31T15:12:16.931836+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:12:14.865356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:12:14.865356+0000","last_clean_scrub_stamp":"2024-03-31T15:12:14.865356+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:28:00.910690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.15","version":"0'0","reported_seq":9,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T15:12:16.915068+0000","last_change":"2024-03-31T15:12:16.910015+0000","last_active":"2024-03-31T15:12:16.915068+0000","last_peered":"2024-03-31T15:12:16.915068+0000","last_clean":"2024-03-31T15:12:16.915068+0000","last_became_active":"2024-03-31T15:12:16.909651+0000","last_became_peered":"2024-03-31T15:12:16.909651+0000","last_unstale":"2024-03-31T15:12:16.915068+0000","last_undegraded":"2024-03-31T15:12:16.915068+0000","last_fullsized":"2024-03-31T15:12:16.915068+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_clean_scrub_stamp":"2024-03-31T15:12:15.875521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:19:07.639622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.14","version":"95'6","reported_seq":20,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-31T15:12:17.895037+0000","last_change":"2024-03-31T15:12:15.905613+0000","last_active":"2024-03-31T15:12:17.895037+0000","last_peered":"2024-03-31T15:12:17.895037+0000","last_clean":"2024-03-31T15:12:17.895037+0000","last_became_active":"2024-03-31T15:12:15.905340+0000","last_became_peered":"2024-03-31T15:12:15.905340+0000","last_unstale":"2024-03-31T15:12:17.895037+0000","last_undegraded":"2024-03-31T15:12:17.895037+0000","last_fullsized":"2024-03-31T15:12:17.895037+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:12:14.865356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:12:14.865356+0000","last_clean_scrub_stamp":"2024-03-31T15:12:14.865356+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:33:58.712582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.12","version":"0'0","reported_seq":9,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T15:12:16.918724+0000","last_change":"2024-03-31T15:12:16.908659+0000","last_active":"2024-03-31T15:12:16.918724+0000","last_peered":"2024-03-31T15:12:16.918724+0000","last_clean":"2024-03-31T15:12:16.918724+0000","last_became_active":"2024-03-31T15:12:16.907211+0000","last_became_peered":"2024-03-31T15:12:16.907211+0000","last_unstale":"2024-03-31T15:12:16.918724+0000","last_undegraded":"2024-03-31T15:12:16.918724+0000","last_fullsized":"2024-03-31T15:12:16.918724+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_clean_scrub_stamp":"2024-03-31T15:12:15.875521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:57:29.441394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.13","version":"95'2","reported_seq":16,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-31T15:12:17.890033+0000","last_change":"2024-03-31T15:12:15.902014+0000","last_active":"2024-03-31T15:12:17.890033+0000","last_peered":"2024-03-31T15:12:17.890033+0000","last_clean":"2024-03-31T15:12:17.890033+0000","last_became_active":"2024-03-31T15:12:15.900192+0000","last_became_peered":"2024-03-31T15:12:15.900192+0000","last_unstale":"2024-03-31T15:12:17.890033+0000","last_undegraded":"2024-03-31T15:12:17.890033+0000","last_fullsized":"2024-03-31T15:12:17.890033+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:12:14.865356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:12:14.865356+0000","last_clean_scrub_stamp":"2024-03-31T15:12:14.865356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:12:58.240842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.13","version":"0'0","reported_seq":9,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T15:12:16.921178+0000","last_change":"2024-03-31T15:12:16.913045+0000","last_active":"2024-03-31T15:12:16.921178+0000","last_peered":"2024-03-31T15:12:16.921178+0000","last_clean":"2024-03-31T15:12:16.921178+0000","last_became_active":"2024-03-31T15:12:16.911052+0000","last_became_peered":"2024-03-31T15:12:16.911052+0000","last_unstale":"2024-03-31T15:12:16.921178+0000","last_undegraded":"2024-03-31T15:12:16.921178+0000","last_fullsized":"2024-03-31T15:12:16.921178+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_clean_scrub_stamp":"2024-03-31T15:12:15.875521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:21:47.383013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.12","version":"0'0","reported_seq":14,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-31T15:12:17.890228+0000","last_change":"2024-03-31T15:12:15.910589+0000","last_active":"2024-03-31T15:12:17.890228+0000","last_peered":"2024-03-31T15:12:17.890228+0000","last_clean":"2024-03-31T15:12:17.890228+0000","last_became_active":"2024-03-31T15:12:15.910296+0000","last_became_peered":"2024-03-31T15:12:15.910296+0000","last_unstale":"2024-03-31T15:12:17.890228+0000","last_undegraded":"2024-03-31T15:12:17.890228+0000","last_fullsized":"2024-03-31T15:12:17.890228+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:12:14.865356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:12:14.865356+0000","last_clean_scrub_stamp":"2024-03-31T15:12:14.865356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:23:32.128958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.10","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-31T15:12:17.895013+0000","last_change":"2024-03-31T15:12:16.903374+0000","last_active":"2024-03-31T15:12:17.895013+0000","last_peered":"2024-03-31T15:12:17.895013+0000","last_clean":"2024-03-31T15:12:17.895013+0000","last_became_active":"2024-03-31T15:12:16.902503+0000","last_became_peered":"2024-03-31T15:12:16.902503+0000","last_unstale":"2024-03-31T15:12:17.895013+0000","last_undegraded":"2024-03-31T15:12:17.895013+0000","last_fullsized":"2024-03-31T15:12:17.895013+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_clean_scrub_stamp":"2024-03-31T15:12:15.875521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:53:55.735548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.11","version":"0'0","reported_seq":14,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-31T15:12:17.894419+0000","last_change":"2024-03-31T15:12:15.898748+0000","last_active":"2024-03-31T15:12:17.894419+0000","last_peered":"2024-03-31T15:12:17.894419+0000","last_clean":"2024-03-31T15:12:17.894419+0000","last_became_active":"2024-03-31T15:12:15.898311+0000","last_became_peered":"2024-03-31T15:12:15.898311+0000","last_unstale":"2024-03-31T15:12:17.894419+0000","last_undegraded":"2024-03-31T15:12:17.894419+0000","last_fullsized":"2024-03-31T15:12:17.894419+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:12:14.865356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:12:14.865356+0000","last_clean_scrub_stamp":"2024-03-31T15:12:14.865356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T17:17:59.212188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.11","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-31T15:12:17.889725+0000","last_change":"2024-03-31T15:12:16.907101+0000","last_active":"2024-03-31T15:12:17.889725+0000","last_peered":"2024-03-31T15:12:17.889725+0000","last_clean":"2024-03-31T15:12:17.889725+0000","last_became_active":"2024-03-31T15:12:16.906777+0000","last_became_peered":"2024-03-31T15:12:16.906777+0000","last_unstale":"2024-03-31T15:12:17.889725+0000","last_undegraded":"2024-03-31T15:12:17.889725+0000","last_fullsized":"2024-03-31T15:12:17.889725+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_clean_scrub_stamp":"2024-03-31T15:12:15.875521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T15:43:27.285287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.10","version":"95'8","reported_seq":20,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T15:12:16.930393+0000","last_change":"2024-03-31T15:12:15.903073+0000","last_active":"2024-03-31T15:12:16.930393+0000","last_peered":"2024-03-31T15:12:16.930393+0000","last_clean":"2024-03-31T15:12:16.930393+0000","last_became_active":"2024-03-31T15:12:15.902722+0000","last_became_peered":"2024-03-31T15:12:15.902722+0000","last_unstale":"2024-03-31T15:12:16.930393+0000","last_undegraded":"2024-03-31T15:12:16.930393+0000","last_fullsized":"2024-03-31T15:12:16.930393+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:12:14.865356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:12:14.865356+0000","last_clean_scrub_stamp":"2024-03-31T15:12:14.865356+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:37:51.668010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.f","version":"0'0","reported_seq":9,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T15:12:16.915299+0000","last_change":"2024-03-31T15:12:16.911972+0000","last_active":"2024-03-31T15:12:16.915299+0000","last_peered":"2024-03-31T15:12:16.915299+0000","last_clean":"2024-03-31T15:12:16.915299+0000","last_became_active":"2024-03-31T15:12:16.910557+0000","last_became_peered":"2024-03-31T15:12:16.910557+0000","last_unstale":"2024-03-31T15:12:16.915299+0000","last_undegraded":"2024-03-31T15:12:16.915299+0000","last_fullsized":"2024-03-31T15:12:16.915299+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_clean_scrub_stamp":"2024-03-31T15:12:15.875521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:11:19.502245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.e","version":"95'2","reported_seq":16,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-31T15:12:17.894878+0000","last_change":"2024-03-31T15:12:15.902495+0000","last_active":"2024-03-31T15:12:17.894878+0000","last_peered":"2024-03-31T15:12:17.894878+0000","last_clean":"2024-03-31T15:12:17.894878+0000","last_became_active":"2024-03-31T15:12:15.902222+0000","last_became_peered":"2024-03-31T15:12:15.902222+0000","last_unstale":"2024-03-31T15:12:17.894878+0000","last_undegraded":"2024-03-31T15:12:17.894878+0000","last_fullsized":"2024-03-31T15:12:17.894878+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:12:14.865356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:12:14.865356+0000","last_clean_scrub_stamp":"2024-03-31T15:12:14.865356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:47:02.138749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.c","version":"0'0","reported_seq":9,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T15:12:16.905793+0000","last_change":"2024-03-31T15:12:16.901466+0000","last_active":"2024-03-31T15:12:16.905793+0000","last_peered":"2024-03-31T15:12:16.905793+0000","last_clean":"2024-03-31T15:12:16.905793+0000","last_became_active":"2024-03-31T15:12:16.901176+0000","last_became_peered":"2024-03-31T15:12:16.901176+0000","last_unstale":"2024-03-31T15:12:16.905793+0000","last_undegraded":"2024-03-31T15:12:16.905793+0000","last_fullsized":"2024-03-31T15:12:16.905793+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_clean_scrub_stamp":"2024-03-31T15:12:15.875521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T17:18:00.222352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.d","version":"95'1","reported_seq":13,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T15:12:16.927871+0000","last_change":"2024-03-31T15:12:15.900951+0000","last_active":"2024-03-31T15:12:16.927871+0000","last_peered":"2024-03-31T15:12:16.927871+0000","last_clean":"2024-03-31T15:12:16.927871+0000","last_became_active":"2024-03-31T15:12:15.900573+0000","last_became_peered":"2024-03-31T15:12:15.900573+0000","last_unstale":"2024-03-31T15:12:16.927871+0000","last_undegraded":"2024-03-31T15:12:16.927871+0000","last_fullsized":"2024-03-31T15:12:16.927871+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:12:14.865356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:12:14.865356+0000","last_clean_scrub_stamp":"2024-03-31T15:12:14.865356+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:08:53.221389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.d","version":"0'0","reported_seq":9,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T15:12:16.915084+0000","last_change":"2024-03-31T15:12:16.910465+0000","last_active":"2024-03-31T15:12:16.915084+0000","last_peered":"2024-03-31T15:12:16.915084+0000","last_clean":"2024-03-31T15:12:16.915084+0000","last_became_active":"2024-03-31T15:12:16.910098+0000","last_became_peered":"2024-03-31T15:12:16.910098+0000","last_unstale":"2024-03-31T15:12:16.915084+0000","last_undegraded":"2024-03-31T15:12:16.915084+0000","last_fullsized":"2024-03-31T15:12:16.915084+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_clean_scrub_stamp":"2024-03-31T15:12:15.875521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:43:58.063821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.c","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T15:12:16.889117+0000","last_change":"2024-03-31T15:12:15.901363+0000","last_active":"2024-03-31T15:12:16.889117+0000","last_peered":"2024-03-31T15:12:16.889117+0000","last_clean":"2024-03-31T15:12:16.889117+0000","last_became_active":"2024-03-31T15:12:15.900798+0000","last_became_peered":"2024-03-31T15:12:15.900798+0000","last_unstale":"2024-03-31T15:12:16.889117+0000","last_undegraded":"2024-03-31T15:12:16.889117+0000","last_fullsized":"2024-03-31T15:12:16.889117+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:12:14.865356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:12:14.865356+0000","last_clean_scrub_stamp":"2024-03-31T15:12:14.865356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:16:07.011962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.a","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-31T15:12:17.890044+0000","last_change":"2024-03-31T15:12:16.905567+0000","last_active":"2024-03-31T15:12:17.890044+0000","last_peered":"2024-03-31T15:12:17.890044+0000","last_clean":"2024-03-31T15:12:17.890044+0000","last_became_active":"2024-03-31T15:12:16.905274+0000","last_became_peered":"2024-03-31T15:12:16.905274+0000","last_unstale":"2024-03-31T15:12:17.890044+0000","last_undegraded":"2024-03-31T15:12:17.890044+0000","last_fullsized":"2024-03-31T15:12:17.890044+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_clean_scrub_stamp":"2024-03-31T15:12:15.875521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:43:51.074045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.b","version":"95'2","reported_seq":14,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T15:12:16.928838+0000","last_change":"2024-03-31T15:12:15.898375+0000","last_active":"2024-03-31T15:12:16.928838+0000","last_peered":"2024-03-31T15:12:16.928838+0000","last_clean":"2024-03-31T15:12:16.928838+0000","last_became_active":"2024-03-31T15:12:15.898079+0000","last_became_peered":"2024-03-31T15:12:15.898079+0000","last_unstale":"2024-03-31T15:12:16.928838+0000","last_undegraded":"2024-03-31T15:12:16.928838+0000","last_fullsized":"2024-03-31T15:12:16.928838+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:12:14.865356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:12:14.865356+0000","last_clean_scrub_stamp":"2024-03-31T15:12:14.865356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:03:02.364251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.b","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-31T15:12:17.889857+0000","last_change":"2024-03-31T15:12:16.903195+0000","last_active":"2024-03-31T15:12:17.889857+0000","last_peered":"2024-03-31T15:12:17.889857+0000","last_clean":"2024-03-31T15:12:17.889857+0000","last_became_active":"2024-03-31T15:12:16.902876+0000","last_became_peered":"2024-03-31T15:12:16.902876+0000","last_unstale":"2024-03-31T15:12:17.889857+0000","last_undegraded":"2024-03-31T15:12:17.889857+0000","last_fullsized":"2024-03-31T15:12:17.889857+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_clean_scrub_stamp":"2024-03-31T15:12:15.875521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T16:34:10.231547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.a","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T15:12:16.896627+0000","last_change":"2024-03-31T15:12:15.911483+0000","last_active":"2024-03-31T15:12:16.896627+0000","last_peered":"2024-03-31T15:12:16.896627+0000","last_clean":"2024-03-31T15:12:16.896627+0000","last_became_active":"2024-03-31T15:12:15.911137+0000","last_became_peered":"2024-03-31T15:12:15.911137+0000","last_unstale":"2024-03-31T15:12:16.896627+0000","last_undegraded":"2024-03-31T15:12:16.896627+0000","last_fullsized":"2024-03-31T15:12:16.896627+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:12:14.865356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:12:14.865356+0000","last_clean_scrub_stamp":"2024-03-31T15:12:14.865356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:42:35.796151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.8","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-31T15:12:17.889737+0000","last_change":"2024-03-31T15:12:16.901928+0000","last_active":"2024-03-31T15:12:17.889737+0000","last_peered":"2024-03-31T15:12:17.889737+0000","last_clean":"2024-03-31T15:12:17.889737+0000","last_became_active":"2024-03-31T15:12:16.901542+0000","last_became_peered":"2024-03-31T15:12:16.901542+0000","last_unstale":"2024-03-31T15:12:17.889737+0000","last_undegraded":"2024-03-31T15:12:17.889737+0000","last_fullsized":"2024-03-31T15:12:17.889737+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_clean_scrub_stamp":"2024-03-31T15:12:15.875521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:32:35.086735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.9","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T15:12:16.897788+0000","last_change":"2024-03-31T15:12:15.910432+0000","last_active":"2024-03-31T15:12:16.897788+0000","last_peered":"2024-03-31T15:12:16.897788+0000","last_clean":"2024-03-31T15:12:16.897788+0000","last_became_active":"2024-03-31T15:12:15.910086+0000","last_became_peered":"2024-03-31T15:12:15.910086+0000","last_unstale":"2024-03-31T15:12:16.897788+0000","last_undegraded":"2024-03-31T15:12:16.897788+0000","last_fullsized":"2024-03-31T15:12:16.897788+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:12:14.865356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:12:14.865356+0000","last_clean_scrub_stamp":"2024-03-31T15:12:14.865356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:47:02.138749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.9","version":"0'0","reported_seq":9,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T15:12:16.917493+0000","last_change":"2024-03-31T15:12:16.916415+0000","last_active":"2024-03-31T15:12:16.917493+0000","last_peered":"2024-03-31T15:12:16.917493+0000","last_clean":"2024-03-31T15:12:16.917493+0000","last_became_active":"2024-03-31T15:12:16.916134+0000","last_became_peered":"2024-03-31T15:12:16.916134+0000","last_unstale":"2024-03-31T15:12:16.917493+0000","last_undegraded":"2024-03-31T15:12:16.917493+0000","last_fullsized":"2024-03-31T15:12:16.917493+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_clean_scrub_stamp":"2024-03-31T15:12:15.875521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T15:30:58.977979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.8","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T15:12:16.888139+0000","last_change":"2024-03-31T15:12:15.905836+0000","last_active":"2024-03-31T15:12:16.888139+0000","last_peered":"2024-03-31T15:12:16.888139+0000","last_clean":"2024-03-31T15:12:16.888139+0000","last_became_active":"2024-03-31T15:12:15.905095+0000","last_became_peered":"2024-03-31T15:12:15.905095+0000","last_unstale":"2024-03-31T15:12:16.888139+0000","last_undegraded":"2024-03-31T15:12:16.888139+0000","last_fullsized":"2024-03-31T15:12:16.888139+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:12:14.865356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:12:14.865356+0000","last_clean_scrub_stamp":"2024-03-31T15:12:14.865356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:01:28.193287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.6","version":"0'0","reported_seq":9,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T15:12:16.897194+0000","last_change":"2024-03-31T15:12:16.896142+0000","last_active":"2024-03-31T15:12:16.897194+0000","last_peered":"2024-03-31T15:12:16.897194+0000","last_clean":"2024-03-31T15:12:16.897194+0000","last_became_active":"2024-03-31T15:12:16.895668+0000","last_became_peered":"2024-03-31T15:12:16.895668+0000","last_unstale":"2024-03-31T15:12:16.897194+0000","last_undegraded":"2024-03-31T15:12:16.897194+0000","last_fullsized":"2024-03-31T15:12:16.897194+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_clean_scrub_stamp":"2024-03-31T15:12:15.875521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:05:22.446118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.7","version":"95'2","reported_seq":14,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T15:12:16.928235+0000","last_change":"2024-03-31T15:12:15.906062+0000","last_active":"2024-03-31T15:12:16.928235+0000","last_peered":"2024-03-31T15:12:16.928235+0000","last_clean":"2024-03-31T15:12:16.928235+0000","last_became_active":"2024-03-31T15:12:15.903879+0000","last_became_peered":"2024-03-31T15:12:15.903879+0000","last_unstale":"2024-03-31T15:12:16.928235+0000","last_undegraded":"2024-03-31T15:12:16.928235+0000","last_fullsized":"2024-03-31T15:12:16.928235+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:12:14.865356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:12:14.865356+0000","last_clean_scrub_stamp":"2024-03-31T15:12:14.865356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T15:19:58.769761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.7","version":"0'0","reported_seq":9,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T15:12:16.922343+0000","last_change":"2024-03-31T15:12:16.919349+0000","last_active":"2024-03-31T15:12:16.922343+0000","last_peered":"2024-03-31T15:12:16.922343+0000","last_clean":"2024-03-31T15:12:16.922343+0000","last_became_active":"2024-03-31T15:12:16.919063+0000","last_became_peered":"2024-03-31T15:12:16.919063+0000","last_unstale":"2024-03-31T15:12:16.922343+0000","last_undegraded":"2024-03-31T15:12:16.922343+0000","last_fullsized":"2024-03-31T15:12:16.922343+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_clean_scrub_stamp":"2024-03-31T15:12:15.875521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T19:09:46.169527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.6","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T15:12:16.890703+0000","last_change":"2024-03-31T15:12:15.902962+0000","last_active":"2024-03-31T15:12:16.890703+0000","last_peered":"2024-03-31T15:12:16.890703+0000","last_clean":"2024-03-31T15:12:16.890703+0000","last_became_active":"2024-03-31T15:12:15.902021+0000","last_became_peered":"2024-03-31T15:12:15.902021+0000","last_unstale":"2024-03-31T15:12:16.890703+0000","last_undegraded":"2024-03-31T15:12:16.890703+0000","last_fullsized":"2024-03-31T15:12:16.890703+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:12:14.865356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:12:14.865356+0000","last_clean_scrub_stamp":"2024-03-31T15:12:14.865356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:11:18.492080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.4","version":"0'0","reported_seq":9,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T15:12:16.920077+0000","last_change":"2024-03-31T15:12:16.909064+0000","last_active":"2024-03-31T15:12:16.920077+0000","last_peered":"2024-03-31T15:12:16.920077+0000","last_clean":"2024-03-31T15:12:16.920077+0000","last_became_active":"2024-03-31T15:12:16.907598+0000","last_became_peered":"2024-03-31T15:12:16.907598+0000","last_unstale":"2024-03-31T15:12:16.920077+0000","last_undegraded":"2024-03-31T15:12:16.920077+0000","last_fullsized":"2024-03-31T15:12:16.920077+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_clean_scrub_stamp":"2024-03-31T15:12:15.875521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T15:43:52.620756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.5","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T15:12:16.888694+0000","last_change":"2024-03-31T15:12:15.904927+0000","last_active":"2024-03-31T15:12:16.888694+0000","last_peered":"2024-03-31T15:12:16.888694+0000","last_clean":"2024-03-31T15:12:16.888694+0000","last_became_active":"2024-03-31T15:12:15.904665+0000","last_became_peered":"2024-03-31T15:12:15.904665+0000","last_unstale":"2024-03-31T15:12:16.888694+0000","last_undegraded":"2024-03-31T15:12:16.888694+0000","last_fullsized":"2024-03-31T15:12:16.888694+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:12:14.865356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:12:14.865356+0000","last_clean_scrub_stamp":"2024-03-31T15:12:14.865356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:34:39.109506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.5","version":"0'0","reported_seq":9,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T15:12:16.909399+0000","last_change":"2024-03-31T15:12:16.905106+0000","last_active":"2024-03-31T15:12:16.909399+0000","last_peered":"2024-03-31T15:12:16.909399+0000","last_clean":"2024-03-31T15:12:16.909399+0000","last_became_active":"2024-03-31T15:12:16.904806+0000","last_became_peered":"2024-03-31T15:12:16.904806+0000","last_unstale":"2024-03-31T15:12:16.909399+0000","last_undegraded":"2024-03-31T15:12:16.909399+0000","last_fullsized":"2024-03-31T15:12:16.909399+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_clean_scrub_stamp":"2024-03-31T15:12:15.875521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T15:16:55.338499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.4","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T15:12:16.893922+0000","last_change":"2024-03-31T15:12:15.905754+0000","last_active":"2024-03-31T15:12:16.893922+0000","last_peered":"2024-03-31T15:12:16.893922+0000","last_clean":"2024-03-31T15:12:16.893922+0000","last_became_active":"2024-03-31T15:12:15.904700+0000","last_became_peered":"2024-03-31T15:12:15.904700+0000","last_unstale":"2024-03-31T15:12:16.893922+0000","last_undegraded":"2024-03-31T15:12:16.893922+0000","last_fullsized":"2024-03-31T15:12:16.893922+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:12:14.865356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:12:14.865356+0000","last_clean_scrub_stamp":"2024-03-31T15:12:14.865356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:02:19.394431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.2","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-31T15:12:17.890065+0000","last_change":"2024-03-31T15:12:16.904662+0000","last_active":"2024-03-31T15:12:17.890065+0000","last_peered":"2024-03-31T15:12:17.890065+0000","last_clean":"2024-03-31T15:12:17.890065+0000","last_became_active":"2024-03-31T15:12:16.904363+0000","last_became_peered":"2024-03-31T15:12:16.904363+0000","last_unstale":"2024-03-31T15:12:17.890065+0000","last_undegraded":"2024-03-31T15:12:17.890065+0000","last_fullsized":"2024-03-31T15:12:17.890065+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_clean_scrub_stamp":"2024-03-31T15:12:15.875521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:11:08.200799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.3","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T15:12:16.888561+0000","last_change":"2024-03-31T15:12:15.903529+0000","last_active":"2024-03-31T15:12:16.888561+0000","last_peered":"2024-03-31T15:12:16.888561+0000","last_clean":"2024-03-31T15:12:16.888561+0000","last_became_active":"2024-03-31T15:12:15.903212+0000","last_became_peered":"2024-03-31T15:12:15.903212+0000","last_unstale":"2024-03-31T15:12:16.888561+0000","last_undegraded":"2024-03-31T15:12:16.888561+0000","last_fullsized":"2024-03-31T15:12:16.888561+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:12:14.865356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:12:14.865356+0000","last_clean_scrub_stamp":"2024-03-31T15:12:14.865356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:02:19.394431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.3","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-31T15:12:17.894895+0000","last_change":"2024-03-31T15:12:16.907561+0000","last_active":"2024-03-31T15:12:17.894895+0000","last_peered":"2024-03-31T15:12:17.894895+0000","last_clean":"2024-03-31T15:12:17.894895+0000","last_became_active":"2024-03-31T15:12:16.907304+0000","last_became_peered":"2024-03-31T15:12:16.907304+0000","last_unstale":"2024-03-31T15:12:17.894895+0000","last_undegraded":"2024-03-31T15:12:17.894895+0000","last_fullsized":"2024-03-31T15:12:17.894895+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_clean_scrub_stamp":"2024-03-31T15:12:15.875521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:43:16.026939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.2","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T15:12:16.893700+0000","last_change":"2024-03-31T15:12:15.909995+0000","last_active":"2024-03-31T15:12:16.893700+0000","last_peered":"2024-03-31T15:12:16.893700+0000","last_clean":"2024-03-31T15:12:16.893700+0000","last_became_active":"2024-03-31T15:12:15.909465+0000","last_became_peered":"2024-03-31T15:12:15.909465+0000","last_unstale":"2024-03-31T15:12:16.893700+0000","last_undegraded":"2024-03-31T15:12:16.893700+0000","last_fullsized":"2024-03-31T15:12:16.893700+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:12:14.865356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:12:14.865356+0000","last_clean_scrub_stamp":"2024-03-31T15:12:14.865356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:03:02.364251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.0","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-31T15:12:17.889856+0000","last_change":"2024-03-31T15:12:16.903467+0000","last_active":"2024-03-31T15:12:17.889856+0000","last_peered":"2024-03-31T15:12:17.889856+0000","last_clean":"2024-03-31T15:12:17.889856+0000","last_became_active":"2024-03-31T15:12:16.903217+0000","last_became_peered":"2024-03-31T15:12:16.903217+0000","last_unstale":"2024-03-31T15:12:17.889856+0000","last_undegraded":"2024-03-31T15:12:17.889856+0000","last_fullsized":"2024-03-31T15:12:17.889856+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_clean_scrub_stamp":"2024-03-31T15:12:15.875521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:28:33.431240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.1","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T15:12:16.898121+0000","last_change":"2024-03-31T15:12:15.913568+0000","last_active":"2024-03-31T15:12:16.898121+0000","last_peered":"2024-03-31T15:12:16.898121+0000","last_clean":"2024-03-31T15:12:16.898121+0000","last_became_active":"2024-03-31T15:12:15.912885+0000","last_became_peered":"2024-03-31T15:12:15.912885+0000","last_unstale":"2024-03-31T15:12:16.898121+0000","last_undegraded":"2024-03-31T15:12:16.898121+0000","last_fullsized":"2024-03-31T15:12:16.898121+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:12:14.865356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:12:14.865356+0000","last_clean_scrub_stamp":"2024-03-31T15:12:14.865356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:05:21.435954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.1","version":"0'0","reported_seq":9,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T15:12:16.908248+0000","last_change":"2024-03-31T15:12:16.903949+0000","last_active":"2024-03-31T15:12:16.908248+0000","last_peered":"2024-03-31T15:12:16.908248+0000","last_clean":"2024-03-31T15:12:16.908248+0000","last_became_active":"2024-03-31T15:12:16.903684+0000","last_became_peered":"2024-03-31T15:12:16.903684+0000","last_unstale":"2024-03-31T15:12:16.908248+0000","last_undegraded":"2024-03-31T15:12:16.908248+0000","last_fullsized":"2024-03-31T15:12:16.908248+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_clean_scrub_stamp":"2024-03-31T15:12:15.875521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:07:52.152112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.0","version":"0'0","reported_seq":14,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-31T15:12:17.889867+0000","last_change":"2024-03-31T15:12:15.903319+0000","last_active":"2024-03-31T15:12:17.889867+0000","last_peered":"2024-03-31T15:12:17.889867+0000","last_clean":"2024-03-31T15:12:17.889867+0000","last_became_active":"2024-03-31T15:12:15.902939+0000","last_became_peered":"2024-03-31T15:12:15.902939+0000","last_unstale":"2024-03-31T15:12:17.889867+0000","last_undegraded":"2024-03-31T15:12:17.889867+0000","last_fullsized":"2024-03-31T15:12:17.889867+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:12:14.865356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:12:14.865356+0000","last_clean_scrub_stamp":"2024-03-31T15:12:14.865356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:36:28.366348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.e","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-31T15:12:17.890421+0000","last_change":"2024-03-31T15:12:16.905769+0000","last_active":"2024-03-31T15:12:17.890421+0000","last_peered":"2024-03-31T15:12:17.890421+0000","last_clean":"2024-03-31T15:12:17.890421+0000","last_became_active":"2024-03-31T15:12:16.905466+0000","last_became_peered":"2024-03-31T15:12:16.905466+0000","last_unstale":"2024-03-31T15:12:17.890421+0000","last_undegraded":"2024-03-31T15:12:17.890421+0000","last_fullsized":"2024-03-31T15:12:17.890421+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:12:15.875521+0000","last_clean_scrub_stamp":"2024-03-31T15:12:15.875521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:03:21.594309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.f","version":"95'1","reported_seq":13,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T15:12:16.933297+0000","last_change":"2024-03-31T15:12:15.911260+0000","last_active":"2024-03-31T15:12:16.933297+0000","last_peered":"2024-03-31T15:12:16.933297+0000","last_clean":"2024-03-31T15:12:16.933297+0000","last_became_active":"2024-03-31T15:12:15.910954+0000","last_became_peered":"2024-03-31T15:12:15.910954+0000","last_unstale":"2024-03-31T15:12:16.933297+0000","last_undegraded":"2024-03-31T15:12:16.933297+0000","last_fullsized":"2024-03-31T15:12:16.933297+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:12:14.865356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:12:14.865356+0000","last_clean_scrub_stamp":"2024-03-31T15:12:14.865356+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:16:13.382595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":308,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T15:12:16.901433+0000","last_change":"2024-03-31T15:03:24.329890+0000","last_active":"2024-03-31T15:12:16.901433+0000","last_peered":"2024-03-31T15:12:16.901433+0000","last_clean":"2024-03-31T15:12:16.901433+0000","last_became_active":"2024-03-31T15:03:24.328882+0000","last_became_peered":"2024-03-31T15:03:24.328882+0000","last_unstale":"2024-03-31T15:12:16.901433+0000","last_undegraded":"2024-03-31T15:12:16.901433+0000","last_fullsized":"2024-03-31T15:12:16.901433+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:22.885535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:22.885535+0000","last_clean_scrub_stamp":"2024-03-31T15:03:22.885535+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:18:29.004296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":15,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":14,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509552,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32912,"kb_used_data":1728,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338928,"statfs":{"total":96636764160,"available":96603062272,"internally_reserved":0,"allocated":1769472,"data_stored":1092985,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509551,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32336,"kb_used_data":1144,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339504,"statfs":{"total":96636764160,"available":96603652096,"internally_reserved":0,"allocated":1171456,"data_stored":502575,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":15,"seq":64424509550,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32352,"kb_used_data":1172,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339488,"statfs":{"total":96636764160,"available":96603635712,"internally_reserved":0,"allocated":1200128,"data_stored":510541,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509549,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32324,"kb_used_data":1144,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339516,"statfs":{"total":96636764160,"available":96603664384,"internally_reserved":0,"allocated":1171456,"data_stored":502539,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542255,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32328,"kb_used_data":1148,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339512,"statfs":{"total":96636764160,"available":96603660288,"internally_reserved":0,"allocated":1175552,"data_stored":503157,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542255,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32340,"kb_used_data":1156,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339500,"statfs":{"total":96636764160,"available":96603648000,"internally_reserved":0,"allocated":1183744,"data_stored":504139,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542255,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32324,"kb_used_data":1140,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339516,"statfs":{"total":96636764160,"available":96603664384,"internally_reserved":0,"allocated":1167360,"data_stored":502505,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542255,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32916,"kb_used_data":1724,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338924,"statfs":{"total":96636764160,"available":96603058176,"internally_reserved":0,"allocated":1765376,"data_stored":1093455,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-31T15:12:21.457 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-31T15:12:21.457 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-03-31T15:12:21.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:21.679+0000 7f409ee00640 1 Processor -- start 2024-03-31T15:12:21.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:21.679+0000 7f409ee00640 1 -- start start 2024-03-31T15:12:21.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:21.683+0000 7f409ee00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f40981567e0 0x7f4098072c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:21.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:21.683+0000 7f409ee00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f409806dbe0 0x7f4098071550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:21.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:21.683+0000 7f409ee00640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f409815c7e0 con 0x7f40980731c0 2024-03-31T15:12:21.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:21.683+0000 7f409ee00640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f40980736b0 con 0x7f40981567e0 2024-03-31T15:12:21.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:21.683+0000 7f409ee00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f409815caa0 con 0x7f409806dbe0 2024-03-31T15:12:21.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:21.683+0000 7f4097e00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f40981567e0 0x7f4098072c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:21.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:21.683+0000 7f4097e00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f40981567e0 0x7f4098072c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40286/0 (socket says 172.21.15.16:40286) 2024-03-31T15:12:21.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:21.683+0000 7f4097e00640 1 -- 172.21.15.16:0/540268931 learned_addr learned my addr 172.21.15.16:0/540268931 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:12:21.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:21.683+0000 7f409ce00640 1 --2- 172.21.15.16:0/540268931 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f409806dbe0 0x7f4098071550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:21.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:21.683+0000 7f4097e00640 1 -- 172.21.15.16:0/540268931 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f409806dbe0 msgr2=0x7f4098071550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:12:21.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:21.683+0000 7f4097e00640 1 --2- 172.21.15.16:0/540268931 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f409806dbe0 0x7f4098071550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:21.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:21.683+0000 7f4097e00640 1 -- 172.21.15.16:0/540268931 >> v1:172.21.15.89:6789/0 conn(0x7f40980731c0 legacy=0x7f409806d4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:21.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:21.683+0000 7f4097e00640 1 -- 172.21.15.16:0/540268931 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f409815cd80 con 0x7f40981567e0 2024-03-31T15:12:21.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:21.683+0000 7f4097400640 1 -- 172.21.15.16:0/540268931 <== mon.1 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2474014523 0 0) 0x7f409815c7e0 con 0x7f40980731c0 2024-03-31T15:12:21.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:21.683+0000 7f4097e00640 1 --2- 172.21.15.16:0/540268931 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f40981567e0 0x7f4098072c80 secure :-1 s=READY pgs=1413 cs=0 l=1 rev1=1 crypto rx=0x7f4088009e50 tx=0x7f408801ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9196dc4fc85b0002 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:12:21.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:21.687+0000 7f4097400640 1 -- 172.21.15.16:0/540268931 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40880a9440 con 0x7f40981567e0 2024-03-31T15:12:21.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:21.687+0000 7f4097400640 1 -- 172.21.15.16:0/540268931 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f40880ae070 con 0x7f40981567e0 2024-03-31T15:12:21.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:21.687+0000 7f4097400640 1 -- 172.21.15.16:0/540268931 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40880a9800 con 0x7f40981567e0 2024-03-31T15:12:21.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:21.687+0000 7f409ee00640 1 -- 172.21.15.16:0/540268931 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f40981567e0 msgr2=0x7f4098072c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:21.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:21.687+0000 7f409ee00640 1 --2- 172.21.15.16:0/540268931 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f40981567e0 0x7f4098072c80 secure :-1 s=READY pgs=1413 cs=0 l=1 rev1=1 crypto rx=0x7f4088009e50 tx=0x7f408801ca30 comp rx=0 tx=0).stop 2024-03-31T15:12:21.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:21.687+0000 7f409ee00640 1 -- 172.21.15.16:0/540268931 shutdown_connections 2024-03-31T15:12:21.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:21.687+0000 7f409ee00640 1 --2- 172.21.15.16:0/540268931 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f409806dbe0 0x7f4098071550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:21.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:21.687+0000 7f409ee00640 1 --2- 172.21.15.16:0/540268931 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f40981567e0 0x7f4098072c80 unknown :-1 s=CLOSED pgs=1413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:21.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:21.687+0000 7f409ee00640 1 -- 172.21.15.16:0/540268931 >> 172.21.15.16:0/540268931 conn(0x7f409806ee40 msgr2=0x7f409806b620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:12:21.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:21.687+0000 7f409ee00640 1 -- 172.21.15.16:0/540268931 shutdown_connections 2024-03-31T15:12:21.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:21.687+0000 7f409ee00640 1 -- 172.21.15.16:0/540268931 wait complete. 2024-03-31T15:12:21.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:21.687+0000 7f409ee00640 1 Processor -- start 2024-03-31T15:12:21.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:21.687+0000 7f409ee00640 1 -- start start 2024-03-31T15:12:21.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:21.691+0000 7f409ee00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f409806dbe0 0x7f40981471c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:21.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:21.691+0000 7f409ee00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f40980731c0 0x7f4098151270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:21.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:21.691+0000 7f409ee00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f40981567e0 0x7f40980b45b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:21.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:21.691+0000 7f409ee00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f40981692f0 con 0x7f409806dbe0 2024-03-31T15:12:21.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:21.691+0000 7f409ee00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f4098169170 con 0x7f40981567e0 2024-03-31T15:12:21.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:21.691+0000 7f409ee00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f4098169470 con 0x7f40980731c0 2024-03-31T15:12:21.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:21.691+0000 7f4097e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f409806dbe0 0x7f40981471c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:21.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:21.691+0000 7f4097e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f409806dbe0 0x7f40981471c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40288/0 (socket says 172.21.15.16:40288) 2024-03-31T15:12:21.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:21.691+0000 7f4097e00640 1 -- 172.21.15.16:0/314798006 learned_addr learned my addr 172.21.15.16:0/314798006 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:12:21.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:21.691+0000 7f408f400640 1 --2- 172.21.15.16:0/314798006 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f40980731c0 0x7f4098151270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:21.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:21.691+0000 7f409ce00640 1 --2- 172.21.15.16:0/314798006 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f40981567e0 0x7f40980b45b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:21.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:21.691+0000 7f4097e00640 1 -- 172.21.15.16:0/314798006 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f40980731c0 msgr2=0x7f4098151270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:21.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:21.691+0000 7f4097e00640 1 --2- 172.21.15.16:0/314798006 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f40980731c0 0x7f4098151270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:21.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:21.691+0000 7f4097e00640 1 -- 172.21.15.16:0/314798006 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f40981567e0 msgr2=0x7f40980b45b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:21.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:21.691+0000 7f4097e00640 1 --2- 172.21.15.16:0/314798006 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f40981567e0 0x7f40980b45b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:21.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:21.691+0000 7f4097e00640 1 -- 172.21.15.16:0/314798006 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4098151cc0 con 0x7f409806dbe0 2024-03-31T15:12:21.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:21.691+0000 7f409ce00640 1 --2- 172.21.15.16:0/314798006 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f40981567e0 0x7f40980b45b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:12:21.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:21.691+0000 7f4097e00640 1 --2- 172.21.15.16:0/314798006 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f409806dbe0 0x7f40981471c0 secure :-1 s=READY pgs=1414 cs=0 l=1 rev1=1 crypto rx=0x7f40880ab040 tx=0x7f40880197d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:12:21.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:21.691+0000 7f4094c00640 1 -- 172.21.15.16:0/314798006 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40880b8070 con 0x7f409806dbe0 2024-03-31T15:12:21.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:21.691+0000 7f4094c00640 1 -- 172.21.15.16:0/314798006 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f40880a9850 con 0x7f409806dbe0 2024-03-31T15:12:21.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:21.691+0000 7f409ee00640 1 -- 172.21.15.16:0/314798006 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4098153aa0 con 0x7f409806dbe0 2024-03-31T15:12:21.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:21.691+0000 7f4094c00640 1 -- 172.21.15.16:0/314798006 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40880a9c00 con 0x7f409806dbe0 2024-03-31T15:12:21.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:21.691+0000 7f409ee00640 1 -- 172.21.15.16:0/314798006 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4098154060 con 0x7f409806dbe0 2024-03-31T15:12:21.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:21.691+0000 7f409ee00640 1 -- 172.21.15.16:0/314798006 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4098069950 con 0x7f409806dbe0 2024-03-31T15:12:21.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:21.695+0000 7f4094c00640 1 -- 172.21.15.16:0/314798006 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f40880c3060 con 0x7f409806dbe0 2024-03-31T15:12:21.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:21.695+0000 7f4094c00640 1 --2- 172.21.15.16:0/314798006 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f4074068380 0x7f407406a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:21.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:21.695+0000 7f4094c00640 1 -- 172.21.15.16:0/314798006 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 5709+0+0 (secure 0 0 0) 0x7f4088120800 con 0x7f409806dbe0 2024-03-31T15:12:21.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:21.695+0000 7f408f400640 1 --2- 172.21.15.16:0/314798006 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f4074068380 0x7f407406a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:21.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:21.695+0000 7f408f400640 1 --2- 172.21.15.16:0/314798006 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f4074068380 0x7f407406a840 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f4098154c00 tx=0x7f4080009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:12:21.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:21.703+0000 7f4094c00640 1 -- 172.21.15.16:0/314798006 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f4088016620 con 0x7f409806dbe0 2024-03-31T15:12:21.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:21.923+0000 7f409ee00640 1 -- 172.21.15.16:0/314798006 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) -- 0x7f40980af520 con 0x7f409806dbe0 2024-03-31T15:12:21.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:21.927+0000 7f4094c00640 1 -- 172.21.15.16:0/314798006 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v19) ==== 190+0+147 (secure 0 0 0) 0x7f40880f0d20 con 0x7f409806dbe0 2024-03-31T15:12:21.921 INFO:teuthology.orchestra.run.smithi016.stdout:[client.0] 2024-03-31T15:12:21.921 INFO:teuthology.orchestra.run.smithi016.stdout: key = AQAPewlmlMIDEhAAF0Ll1HcrWHmH6xDKq2Qq9A== 2024-03-31T15:12:21.921 INFO:teuthology.orchestra.run.smithi016.stdout: caps mds = "allow" 2024-03-31T15:12:21.921 INFO:teuthology.orchestra.run.smithi016.stdout: caps mon = "allow r" 2024-03-31T15:12:21.921 INFO:teuthology.orchestra.run.smithi016.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-03-31T15:12:21.922 INFO:teuthology.orchestra.run.smithi016.stderr:updated caps for client.0 2024-03-31T15:12:21.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:21.931+0000 7f409ee00640 1 -- 172.21.15.16:0/314798006 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f4074068380 msgr2=0x7f407406a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:21.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:21.931+0000 7f409ee00640 1 --2- 172.21.15.16:0/314798006 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f4074068380 0x7f407406a840 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f4098154c00 tx=0x7f4080009040 comp rx=0 tx=0).stop 2024-03-31T15:12:21.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:21.931+0000 7f409ee00640 1 -- 172.21.15.16:0/314798006 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f409806dbe0 msgr2=0x7f40981471c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:21.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:21.931+0000 7f409ee00640 1 --2- 172.21.15.16:0/314798006 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f409806dbe0 0x7f40981471c0 secure :-1 s=READY pgs=1414 cs=0 l=1 rev1=1 crypto rx=0x7f40880ab040 tx=0x7f40880197d0 comp rx=0 tx=0).stop 2024-03-31T15:12:21.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:21.931+0000 7f409ee00640 1 -- 172.21.15.16:0/314798006 shutdown_connections 2024-03-31T15:12:21.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:21.931+0000 7f409ee00640 1 --2- 172.21.15.16:0/314798006 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f4074068380 0x7f407406a840 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:21.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:21.931+0000 7f409ee00640 1 --2- 172.21.15.16:0/314798006 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f40981567e0 0x7f40980b45b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:21.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:21.931+0000 7f409ee00640 1 --2- 172.21.15.16:0/314798006 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f40980731c0 0x7f4098151270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:21.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:21.931+0000 7f409ee00640 1 --2- 172.21.15.16:0/314798006 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f409806dbe0 0x7f40981471c0 unknown :-1 s=CLOSED pgs=1414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:21.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:21.931+0000 7f409ee00640 1 -- 172.21.15.16:0/314798006 >> 172.21.15.16:0/314798006 conn(0x7f409806ee40 msgr2=0x7f409806d4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:12:21.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:21.931+0000 7f409ee00640 1 -- 172.21.15.16:0/314798006 shutdown_connections 2024-03-31T15:12:21.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:21.931+0000 7f409ee00640 1 -- 172.21.15.16:0/314798006 wait complete. 2024-03-31T15:12:21.942 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-31T15:12:22.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.091+0000 7f5437200640 1 Processor -- start 2024-03-31T15:12:22.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.091+0000 7f5437200640 1 -- start start 2024-03-31T15:12:22.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.091+0000 7f5437200640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f543815e350 0x7f543806fac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:22.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.091+0000 7f5437200640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f543806bd90 con 0x7f543806b620 2024-03-31T15:12:22.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.091+0000 7f5437200640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f543806c070 con 0x7f54380cdee0 2024-03-31T15:12:22.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.091+0000 7f5437200640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f543815e730 con 0x7f543815e350 2024-03-31T15:12:22.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.091+0000 7f542fe00640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f543806b620 0x7f54380cd7d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:54962/0 (socket says 172.21.15.16:54962) 2024-03-31T15:12:22.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.091+0000 7f542fe00640 1 -- 172.21.15.16:0/1311122528 learned_addr learned my addr 172.21.15.16:0/1311122528 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:12:22.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.091+0000 7f5435e00640 1 --2- 172.21.15.16:0/1311122528 >> v2:172.21.15.89:3300/0 conn(0x7f543815e350 0x7f543806fac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:22.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.091+0000 7f5435400640 1 -- 172.21.15.16:0/1311122528 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1286586063 0 0) 0x7f543806bd90 con 0x7f543806b620 2024-03-31T15:12:22.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.091+0000 7f5435400640 1 -- 172.21.15.16:0/1311122528 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f541c003530 con 0x7f543806b620 2024-03-31T15:12:22.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.091+0000 7f5435e00640 1 -- 172.21.15.16:0/1311122528 >> v1:172.21.15.89:6789/0 conn(0x7f54380cdee0 legacy=0x7f54380ceea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:22.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.091+0000 7f5435e00640 1 -- 172.21.15.16:0/1311122528 >> v1:172.21.15.16:6789/0 conn(0x7f543806b620 legacy=0x7f54380cd7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:22.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.091+0000 7f5435400640 1 -- 172.21.15.16:0/1311122528 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3507141546 0 0) 0x7f541c003530 con 0x7f543806b620 2024-03-31T15:12:22.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.091+0000 7f5435e00640 1 -- 172.21.15.16:0/1311122528 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f543806c350 con 0x7f543815e350 2024-03-31T15:12:22.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.095+0000 7f5435e00640 1 --2- 172.21.15.16:0/1311122528 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f543815e350 0x7f543806fac0 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f54280097c0 tx=0x7f542801c390 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9f2f68ca465658dd server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:12:22.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.095+0000 7f5435400640 1 -- 172.21.15.16:0/1311122528 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54280b1070 con 0x7f543815e350 2024-03-31T15:12:22.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.095+0000 7f5435400640 1 -- 172.21.15.16:0/1311122528 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f54280ac440 con 0x7f543815e350 2024-03-31T15:12:22.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.095+0000 7f5435400640 1 -- 172.21.15.16:0/1311122528 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5420001680 con 0x7f543815e350 2024-03-31T15:12:22.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.095+0000 7f5437200640 1 -- 172.21.15.16:0/1311122528 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f543815e350 msgr2=0x7f543806fac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:22.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.095+0000 7f5437200640 1 --2- 172.21.15.16:0/1311122528 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f543815e350 0x7f543806fac0 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f54280097c0 tx=0x7f542801c390 comp rx=0 tx=0).stop 2024-03-31T15:12:22.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.095+0000 7f5437200640 1 -- 172.21.15.16:0/1311122528 shutdown_connections 2024-03-31T15:12:22.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.095+0000 7f5437200640 1 --2- 172.21.15.16:0/1311122528 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f543815e350 0x7f543806fac0 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:22.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.095+0000 7f5437200640 1 -- 172.21.15.16:0/1311122528 >> 172.21.15.16:0/1311122528 conn(0x7f54380b2ac0 msgr2=0x7f54380b2ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:12:22.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.095+0000 7f5437200640 1 -- 172.21.15.16:0/1311122528 shutdown_connections 2024-03-31T15:12:22.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.095+0000 7f5437200640 1 -- 172.21.15.16:0/1311122528 wait complete. 2024-03-31T15:12:22.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.099+0000 7f5437200640 1 Processor -- start 2024-03-31T15:12:22.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.099+0000 7f5437200640 1 -- start start 2024-03-31T15:12:22.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.099+0000 7f5437200640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f543806b620 0x7f54380b5a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:22.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.099+0000 7f5437200640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f54380cdee0 0x7f5438144e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:22.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.099+0000 7f5437200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f543815e350 0x7f54380b4170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:22.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.099+0000 7f5437200640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f54380cff40 con 0x7f543815e350 2024-03-31T15:12:22.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.099+0000 7f5437200640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f54380cfdc0 con 0x7f54380cdee0 2024-03-31T15:12:22.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.099+0000 7f5437200640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f54380d00c0 con 0x7f543806b620 2024-03-31T15:12:22.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.099+0000 7f5436800640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f543815e350 0x7f54380b4170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:22.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.099+0000 7f5436800640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f543815e350 0x7f54380b4170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40304/0 (socket says 172.21.15.16:40304) 2024-03-31T15:12:22.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.099+0000 7f5436800640 1 -- 172.21.15.16:0/3019702550 learned_addr learned my addr 172.21.15.16:0/3019702550 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:12:22.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.099+0000 7f542fe00640 1 --2- 172.21.15.16:0/3019702550 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f54380cdee0 0x7f5438144e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:22.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.099+0000 7f5435e00640 1 --2- 172.21.15.16:0/3019702550 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f543806b620 0x7f54380b5a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:22.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.099+0000 7f5436800640 1 -- 172.21.15.16:0/3019702550 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f543806b620 msgr2=0x7f54380b5a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:22.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.099+0000 7f5436800640 1 --2- 172.21.15.16:0/3019702550 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f543806b620 0x7f54380b5a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:22.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.099+0000 7f5436800640 1 -- 172.21.15.16:0/3019702550 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f54380cdee0 msgr2=0x7f5438144e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:22.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.099+0000 7f5436800640 1 --2- 172.21.15.16:0/3019702550 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f54380cdee0 0x7f5438144e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:22.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.099+0000 7f5436800640 1 -- 172.21.15.16:0/3019702550 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f54380b49c0 con 0x7f543815e350 2024-03-31T15:12:22.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.099+0000 7f5436800640 1 --2- 172.21.15.16:0/3019702550 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f543815e350 0x7f54380b4170 secure :-1 s=READY pgs=1416 cs=0 l=1 rev1=1 crypto rx=0x7f543000c7a0 tx=0x7f543000cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:12:22.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.099+0000 7f542e000640 1 -- 172.21.15.16:0/3019702550 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5430003b80 con 0x7f543815e350 2024-03-31T15:12:22.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.099+0000 7f542e000640 1 -- 172.21.15.16:0/3019702550 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5430003d20 con 0x7f543815e350 2024-03-31T15:12:22.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.099+0000 7f542e000640 1 -- 172.21.15.16:0/3019702550 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f543000aa30 con 0x7f543815e350 2024-03-31T15:12:22.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.099+0000 7f5437200640 1 -- 172.21.15.16:0/3019702550 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f54380b4ca0 con 0x7f543815e350 2024-03-31T15:12:22.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.099+0000 7f5437200640 1 -- 172.21.15.16:0/3019702550 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f54380b5260 con 0x7f543815e350 2024-03-31T15:12:22.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.099+0000 7f5437200640 1 -- 172.21.15.16:0/3019702550 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f543804f8c0 con 0x7f543815e350 2024-03-31T15:12:22.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.103+0000 7f542e000640 1 -- 172.21.15.16:0/3019702550 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f543000abd0 con 0x7f543815e350 2024-03-31T15:12:22.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.103+0000 7f542e000640 1 --2- 172.21.15.16:0/3019702550 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f5414068380 0x7f541406a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:22.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.103+0000 7f542e000640 1 -- 172.21.15.16:0/3019702550 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 5709+0+0 (secure 0 0 0) 0x7f54300867d0 con 0x7f543815e350 2024-03-31T15:12:22.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.103+0000 7f5435e00640 1 --2- 172.21.15.16:0/3019702550 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f5414068380 0x7f541406a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:22.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.103+0000 7f5435e00640 1 --2- 172.21.15.16:0/3019702550 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f5414068380 0x7f541406a840 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f54280ae000 tx=0x7f54280a6070 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:12:22.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.107+0000 7f542e000640 1 -- 172.21.15.16:0/3019702550 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f54300566a0 con 0x7f543815e350 2024-03-31T15:12:22.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.303+0000 7f5437200640 1 -- 172.21.15.16:0/3019702550 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f54380b1190 con 0x7f543815e350 2024-03-31T15:12:22.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.303+0000 7f542e000640 1 -- 172.21.15.16:0/3019702550 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 83 v83) ==== 94+0+5180 (secure 0 0 0) 0x7f543005a9a0 con 0x7f543815e350 2024-03-31T15:12:22.298 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:12:22.298 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":83,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7139,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6836/3798928623","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":3798928623},{"type":"v1","addr":"172.21.15.89:6836","nonce":3798928623}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":78},{"gid":7142,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/3074283817","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6835","nonce":3074283817},{"type":"v1","addr":"172.21.15.89:6837","nonce":3074283817}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":78},{"gid":8071,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6834/2110189998","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":2110189998},{"type":"v1","addr":"172.21.15.16:6834","nonce":2110189998}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":78}],"filesystems":[{"mdsmap":{"epoch":83,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-31T15:12:16.294600+0000","modified":"2024-03-31T15:12:19.297339+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8074},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8074":{"gid":8074,"name":"a","rank":0,"incarnation":79,"state":"up:active","state_seq":3,"addr":"172.21.15.16:6835/490717480","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6833","nonce":490717480},{"type":"v1","addr":"172.21.15.16:6835","nonce":490717480}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8074,"qdb_cluster":[8074]},"id":7}]} 2024-03-31T15:12:22.298 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 83 2024-03-31T15:12:22.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.307+0000 7f5437200640 1 -- 172.21.15.16:0/3019702550 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f5414068380 msgr2=0x7f541406a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:22.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.307+0000 7f5437200640 1 --2- 172.21.15.16:0/3019702550 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f5414068380 0x7f541406a840 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f54280ae000 tx=0x7f54280a6070 comp rx=0 tx=0).stop 2024-03-31T15:12:22.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.307+0000 7f5437200640 1 -- 172.21.15.16:0/3019702550 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f543815e350 msgr2=0x7f54380b4170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:22.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.307+0000 7f5437200640 1 --2- 172.21.15.16:0/3019702550 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f543815e350 0x7f54380b4170 secure :-1 s=READY pgs=1416 cs=0 l=1 rev1=1 crypto rx=0x7f543000c7a0 tx=0x7f543000cc60 comp rx=0 tx=0).stop 2024-03-31T15:12:22.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.307+0000 7f5437200640 1 -- 172.21.15.16:0/3019702550 shutdown_connections 2024-03-31T15:12:22.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.307+0000 7f5437200640 1 --2- 172.21.15.16:0/3019702550 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f5414068380 0x7f541406a840 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:22.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.307+0000 7f5437200640 1 --2- 172.21.15.16:0/3019702550 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f543815e350 0x7f54380b4170 unknown :-1 s=CLOSED pgs=1416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:22.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.307+0000 7f5437200640 1 --2- 172.21.15.16:0/3019702550 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f54380cdee0 0x7f5438144e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:22.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.307+0000 7f5437200640 1 --2- 172.21.15.16:0/3019702550 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f543806b620 0x7f54380b5a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:22.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.307+0000 7f5437200640 1 -- 172.21.15.16:0/3019702550 >> 172.21.15.16:0/3019702550 conn(0x7f54380b2ac0 msgr2=0x7f54380b2ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:12:22.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.307+0000 7f5437200640 1 -- 172.21.15.16:0/3019702550 shutdown_connections 2024-03-31T15:12:22.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.307+0000 7f5437200640 1 -- 172.21.15.16:0/3019702550 wait complete. 2024-03-31T15:12:22.722 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 83, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-31T15:12:16.294600+0000', 'modified': '2024-03-31T15:12:19.297339+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8074}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8074': {'gid': 8074, 'name': 'a', 'rank': 0, 'incarnation': 79, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.16:6835/490717480', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6833', 'nonce': 490717480}, {'type': 'v1', 'addr': '172.21.15.16:6835', 'nonce': 490717480}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [15], 'metadata_pool': 14, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8074, 'qdb_cluster': [8074]} 2024-03-31T15:12:22.722 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-31T15:12:22.722 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-03-31T15:12:22.853 INFO:tasks.ceph.mds.a.smithi016.stderr:2024-03-31T15:12:22.859+0000 7fde63e00640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-31T15:12:22.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.871+0000 7fd30c200640 1 Processor -- start 2024-03-31T15:12:22.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.871+0000 7fd30c200640 1 -- start start 2024-03-31T15:12:22.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.871+0000 7fd30c200640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fd30406d490 0x7fd30406d8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:22.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.871+0000 7fd30c200640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fd3040cc740 0x7fd3040ccbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:22.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.871+0000 7fd30c200640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd304162010 con 0x7fd30406f920 2024-03-31T15:12:22.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.871+0000 7fd30c200640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7fd304162be0 con 0x7fd30406d490 2024-03-31T15:12:22.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.871+0000 7fd30c200640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fd3041622b0 con 0x7fd3040cc740 2024-03-31T15:12:22.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.871+0000 7fd308e00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fd30406d490 0x7fd30406d8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:22.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.875+0000 7fd308e00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fd30406d490 0x7fd30406d8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40310/0 (socket says 172.21.15.16:40310) 2024-03-31T15:12:22.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.875+0000 7fd308e00640 1 -- 172.21.15.16:0/1346327971 learned_addr learned my addr 172.21.15.16:0/1346327971 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:12:22.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.875+0000 7fd30a200640 1 --2- 172.21.15.16:0/1346327971 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fd3040cc740 0x7fd3040ccbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:22.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.875+0000 7fd308e00640 1 -- 172.21.15.16:0/1346327971 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fd3040cc740 msgr2=0x7fd3040ccbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:12:22.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.875+0000 7fd308e00640 1 --2- 172.21.15.16:0/1346327971 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fd3040cc740 0x7fd3040ccbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:22.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.875+0000 7fd308e00640 1 -- 172.21.15.16:0/1346327971 >> v1:172.21.15.89:6789/0 conn(0x7fd30406f920 legacy=0x7fd30406fd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:22.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.875+0000 7fd308e00640 1 -- 172.21.15.16:0/1346327971 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd304162590 con 0x7fd30406d490 2024-03-31T15:12:22.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.875+0000 7fd308e00640 1 --2- 172.21.15.16:0/1346327971 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd30406d490 0x7fd30406d8f0 secure :-1 s=READY pgs=1417 cs=0 l=1 rev1=1 crypto rx=0x7fd304162d60 tx=0x7fd2f001c510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c16b5db7da0a4e2 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:12:22.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.875+0000 7fd303e00640 1 -- 172.21.15.16:0/1346327971 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2f00ab620 con 0x7fd30406d490 2024-03-31T15:12:22.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.875+0000 7fd303e00640 1 -- 172.21.15.16:0/1346327971 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd2f00b0070 con 0x7fd30406d490 2024-03-31T15:12:22.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.875+0000 7fd303e00640 1 -- 172.21.15.16:0/1346327971 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2f00ab9e0 con 0x7fd30406d490 2024-03-31T15:12:22.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.879+0000 7fd30c200640 1 -- 172.21.15.16:0/1346327971 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd30406d490 msgr2=0x7fd30406d8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:22.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.879+0000 7fd30c200640 1 --2- 172.21.15.16:0/1346327971 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd30406d490 0x7fd30406d8f0 secure :-1 s=READY pgs=1417 cs=0 l=1 rev1=1 crypto rx=0x7fd304162d60 tx=0x7fd2f001c510 comp rx=0 tx=0).stop 2024-03-31T15:12:22.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.879+0000 7fd30c200640 1 -- 172.21.15.16:0/1346327971 shutdown_connections 2024-03-31T15:12:22.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.879+0000 7fd30c200640 1 --2- 172.21.15.16:0/1346327971 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fd3040cc740 0x7fd3040ccbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:22.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.879+0000 7fd30c200640 1 --2- 172.21.15.16:0/1346327971 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd30406d490 0x7fd30406d8f0 unknown :-1 s=CLOSED pgs=1417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:22.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.879+0000 7fd30c200640 1 -- 172.21.15.16:0/1346327971 >> 172.21.15.16:0/1346327971 conn(0x7fd304067330 msgr2=0x7fd304067740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:12:22.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.879+0000 7fd30c200640 1 -- 172.21.15.16:0/1346327971 shutdown_connections 2024-03-31T15:12:22.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.879+0000 7fd30c200640 1 -- 172.21.15.16:0/1346327971 wait complete. 2024-03-31T15:12:22.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.879+0000 7fd30c200640 1 Processor -- start 2024-03-31T15:12:22.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.879+0000 7fd30c200640 1 -- start start 2024-03-31T15:12:22.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.883+0000 7fd30c200640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd30406d490 0x7fd3041611b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:22.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.883+0000 7fd30c200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd30406f920 0x7fd3040b7ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:22.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.883+0000 7fd30c200640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fd3040cc740 0x7fd3040b55c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:22.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.883+0000 7fd30c200640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fd30416b530 con 0x7fd30406f920 2024-03-31T15:12:22.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.883+0000 7fd30c200640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fd30416b3b0 con 0x7fd30406d490 2024-03-31T15:12:22.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.883+0000 7fd30c200640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fd30416b6b0 con 0x7fd3040cc740 2024-03-31T15:12:22.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.883+0000 7fd308e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd30406f920 0x7fd3040b7ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:22.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.883+0000 7fd308e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd30406f920 0x7fd3040b7ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40318/0 (socket says 172.21.15.16:40318) 2024-03-31T15:12:22.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.883+0000 7fd308e00640 1 -- 172.21.15.16:0/574342244 learned_addr learned my addr 172.21.15.16:0/574342244 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:12:22.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.883+0000 7fd30a200640 1 --2- 172.21.15.16:0/574342244 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fd3040cc740 0x7fd3040b55c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:22.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.883+0000 7fd309800640 1 --2- 172.21.15.16:0/574342244 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd30406d490 0x7fd3041611b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:22.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.883+0000 7fd308e00640 1 -- 172.21.15.16:0/574342244 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fd3040cc740 msgr2=0x7fd3040b55c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:22.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.883+0000 7fd308e00640 1 --2- 172.21.15.16:0/574342244 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fd3040cc740 0x7fd3040b55c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:22.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.883+0000 7fd308e00640 1 -- 172.21.15.16:0/574342244 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd30406d490 msgr2=0x7fd3041611b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:22.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.883+0000 7fd308e00640 1 --2- 172.21.15.16:0/574342244 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd30406d490 0x7fd3041611b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:22.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.883+0000 7fd308e00640 1 -- 172.21.15.16:0/574342244 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd304141800 con 0x7fd30406f920 2024-03-31T15:12:22.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.883+0000 7fd308e00640 1 --2- 172.21.15.16:0/574342244 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd30406f920 0x7fd3040b7ad0 secure :-1 s=READY pgs=1418 cs=0 l=1 rev1=1 crypto rx=0x7fd2f001abc0 tx=0x7fd2f001ad50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:12:22.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.883+0000 7fd301600640 1 -- 172.21.15.16:0/574342244 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2f00abce0 con 0x7fd30406f920 2024-03-31T15:12:22.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.883+0000 7fd301600640 1 -- 172.21.15.16:0/574342244 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd2f00be3f0 con 0x7fd30406f920 2024-03-31T15:12:22.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.883+0000 7fd30c200640 1 -- 172.21.15.16:0/574342244 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd304141ae0 con 0x7fd30406f920 2024-03-31T15:12:22.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.883+0000 7fd301600640 1 -- 172.21.15.16:0/574342244 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2f00be7c0 con 0x7fd30406f920 2024-03-31T15:12:22.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.883+0000 7fd30c200640 1 -- 172.21.15.16:0/574342244 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd3041420a0 con 0x7fd30406f920 2024-03-31T15:12:22.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.887+0000 7fd301600640 1 -- 172.21.15.16:0/574342244 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fd2f00b43f0 con 0x7fd30406f920 2024-03-31T15:12:22.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.887+0000 7fd301600640 1 --2- 172.21.15.16:0/574342244 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fd2e4068380 0x7fd2e406a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:22.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.887+0000 7fd301600640 1 -- 172.21.15.16:0/574342244 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 5709+0+0 (secure 0 0 0) 0x7fd2f01208b0 con 0x7fd30406f920 2024-03-31T15:12:22.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.887+0000 7fd309800640 1 --2- 172.21.15.16:0/574342244 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fd2e4068380 0x7fd2e406a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:22.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.887+0000 7fd309800640 1 --2- 172.21.15.16:0/574342244 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fd2e4068380 0x7fd2e406a840 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7fd2f8001810 tx=0x7fd2f8009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:12:22.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.887+0000 7fd30f6c7000 1 Processor -- start 2024-03-31T15:12:22.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.887+0000 7fd30f6c7000 1 -- start start 2024-03-31T15:12:22.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.891+0000 7fd30f6c7000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5574db89a4a0 0x5574db89a880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:22.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.891+0000 7fd30f6c7000 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5574db89adc0 0x5574db84db70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:22.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.891+0000 7fd30f6c7000 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5574db84e0b0 0x5574db84e490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:22.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.891+0000 7fd30f6c7000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5574db6a0950 con 0x5574db89a4a0 2024-03-31T15:12:22.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.891+0000 7fd30f6c7000 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x5574db63d990 con 0x5574db89adc0 2024-03-31T15:12:22.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.891+0000 7fd30f6c7000 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x5574db69c900 con 0x5574db84e0b0 2024-03-31T15:12:22.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.891+0000 7fd309800640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5574db89a4a0 0x5574db89a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:22.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.891+0000 7fd309800640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5574db89a4a0 0x5574db89a880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40320/0 (socket says 172.21.15.16:40320) 2024-03-31T15:12:22.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.891+0000 7fd309800640 1 -- 172.21.15.16:0/703618109 learned_addr learned my addr 172.21.15.16:0/703618109 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:12:22.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.891+0000 7fd309800640 1 -- 172.21.15.16:0/703618109 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5574db84e0b0 msgr2=0x5574db84e490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:12:22.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.891+0000 7fd309800640 1 --2- 172.21.15.16:0/703618109 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5574db84e0b0 0x5574db84e490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:22.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.891+0000 7fd309800640 1 -- 172.21.15.16:0/703618109 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5574db89adc0 msgr2=0x5574db84db70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:12:22.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.891+0000 7fd309800640 1 --2- 172.21.15.16:0/703618109 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5574db89adc0 0x5574db84db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:22.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.891+0000 7fd309800640 1 -- 172.21.15.16:0/703618109 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5574db6e5100 con 0x5574db89a4a0 2024-03-31T15:12:22.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.891+0000 7fd309800640 1 --2- 172.21.15.16:0/703618109 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5574db89a4a0 0x5574db89a880 secure :-1 s=READY pgs=1419 cs=0 l=1 rev1=1 crypto rx=0x7fd2f800faf0 tx=0x7fd2f8000ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd0585cbb713842c server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:12:22.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.891+0000 7fd2f6000640 1 -- 172.21.15.16:0/703618109 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2f8025070 con 0x5574db89a4a0 2024-03-31T15:12:22.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.891+0000 7fd2f6000640 1 -- 172.21.15.16:0/703618109 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd2f8015c30 con 0x5574db89a4a0 2024-03-31T15:12:22.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.891+0000 7fd2f6000640 1 -- 172.21.15.16:0/703618109 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2f8020400 con 0x5574db89a4a0 2024-03-31T15:12:22.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.891+0000 7fd30f6c7000 1 -- 172.21.15.16:0/703618109 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5574db89a4a0 msgr2=0x5574db89a880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:22.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.891+0000 7fd30f6c7000 1 --2- 172.21.15.16:0/703618109 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5574db89a4a0 0x5574db89a880 secure :-1 s=READY pgs=1419 cs=0 l=1 rev1=1 crypto rx=0x7fd2f800faf0 tx=0x7fd2f8000ee0 comp rx=0 tx=0).stop 2024-03-31T15:12:22.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.891+0000 7fd30f6c7000 1 -- 172.21.15.16:0/703618109 shutdown_connections 2024-03-31T15:12:22.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.891+0000 7fd30f6c7000 1 --2- 172.21.15.16:0/703618109 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5574db84e0b0 0x5574db84e490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:22.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.891+0000 7fd30f6c7000 1 --2- 172.21.15.16:0/703618109 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5574db89adc0 0x5574db84db70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:22.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.891+0000 7fd30f6c7000 1 --2- 172.21.15.16:0/703618109 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5574db89a4a0 0x5574db89a880 unknown :-1 s=CLOSED pgs=1419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:22.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.891+0000 7fd30f6c7000 1 -- 172.21.15.16:0/703618109 >> 172.21.15.16:0/703618109 conn(0x5574db738b80 msgr2=0x5574db87d460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:12:22.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.891+0000 7fd30f6c7000 1 -- 172.21.15.16:0/703618109 shutdown_connections 2024-03-31T15:12:22.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.891+0000 7fd30f6c7000 1 -- 172.21.15.16:0/703618109 wait complete. 2024-03-31T15:12:22.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.891+0000 7fd30f6c7000 1 -- start start 2024-03-31T15:12:22.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.891+0000 7fd30f6c7000 1 Processor -- start 2024-03-31T15:12:22.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.895+0000 7fd30f6c7000 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5574db84e0b0 0x5574db8a2600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:22.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.895+0000 7fd30f6c7000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5574db89a4a0 0x5574db8a2b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:22.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.895+0000 7fd30f6c7000 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5574db89adc0 0x5574db868d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:22.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.895+0000 7fd30f6c7000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5574db69fc60 con 0x5574db89a4a0 2024-03-31T15:12:22.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.895+0000 7fd30f6c7000 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x5574db65e3e0 con 0x5574db84e0b0 2024-03-31T15:12:22.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.895+0000 7fd30f6c7000 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x5574db6ad060 con 0x5574db89adc0 2024-03-31T15:12:22.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.895+0000 7fd30f6c7000 10 client.0 _ll_register_callbacks cb 0x5574db7010f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T15:12:22.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.895+0000 7fd308e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5574db89a4a0 0x5574db8a2b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:22.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.895+0000 7fd308e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5574db89a4a0 0x5574db8a2b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40336/0 (socket says 172.21.15.16:40336) 2024-03-31T15:12:22.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.895+0000 7fd308e00640 1 -- 172.21.15.16:0/852876974 learned_addr learned my addr 172.21.15.16:0/852876974 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:12:22.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.895+0000 7fd309800640 1 --2- 172.21.15.16:0/852876974 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5574db84e0b0 0x5574db8a2600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:22.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.895+0000 7fd30a200640 1 --2- 172.21.15.16:0/852876974 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5574db89adc0 0x5574db868d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:22.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.895+0000 7fd308e00640 1 -- 172.21.15.16:0/852876974 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5574db89adc0 msgr2=0x5574db868d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:12:22.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.895+0000 7fd308e00640 1 --2- 172.21.15.16:0/852876974 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5574db89adc0 0x5574db868d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:22.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.895+0000 7fd308e00640 1 -- 172.21.15.16:0/852876974 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5574db84e0b0 msgr2=0x5574db8a2600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:12:22.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.895+0000 7fd308e00640 1 --2- 172.21.15.16:0/852876974 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5574db84e0b0 0x5574db8a2600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:22.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.895+0000 7fd308e00640 1 -- 172.21.15.16:0/852876974 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5574db66a8c0 con 0x5574db89a4a0 2024-03-31T15:12:22.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.895+0000 7fd308e00640 1 --2- 172.21.15.16:0/852876974 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5574db89a4a0 0x5574db8a2b40 secure :-1 s=READY pgs=1420 cs=0 l=1 rev1=1 crypto rx=0x7fd2f00f8d80 tx=0x7fd2f00f02c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4173ffb0c7608441 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:12:22.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.895+0000 7fd2eaa00640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-31T15:12:22.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.895+0000 7fd2eaa00640 1 -- 172.21.15.16:0/852876974 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2f00f4a80 con 0x5574db89a4a0 2024-03-31T15:12:22.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.895+0000 7fd30f6c7000 1 -- 172.21.15.16:0/852876974 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd2f8015c30 con 0x5574db89a4a0 2024-03-31T15:12:22.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.895+0000 7fd2eaa00640 1 -- 172.21.15.16:0/852876974 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 5709+0+0 (secure 0 0 0) 0x7fd2f00efbd0 con 0x5574db89a4a0 2024-03-31T15:12:22.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.895+0000 7fd2eaa00640 1 -- 172.21.15.16:0/852876974 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=83) ==== 24+0+0 (secure 0 0 0) 0x7fd2f010f050 con 0x5574db89a4a0 2024-03-31T15:12:22.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.895+0000 7fd30f6c7000 10 client.8607 fetch_fsmap learned FSMap version 83 2024-03-31T15:12:22.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.895+0000 7fd30f6c7000 1 -- 172.21.15.16:0/852876974 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=83,monmap=2+}) -- 0x5574db628e80 con 0x5574db89a4a0 2024-03-31T15:12:22.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.895+0000 7fd2eaa00640 1 -- 172.21.15.16:0/852876974 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd2f8015c30 con 0x5574db89a4a0 2024-03-31T15:12:22.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.895+0000 7fd2eaa00640 1 -- 172.21.15.16:0/852876974 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 83) ==== 3494+0+0 (secure 0 0 0) 0x7fd2f00ec4d0 con 0x5574db89a4a0 2024-03-31T15:12:22.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.895+0000 7fd30f6c7000 10 client.8607 fetch_fsmap finished waiting for FSMap version 83 2024-03-31T15:12:22.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.895+0000 7fd30f6c7000 10 client.8607 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-31T15:12:22.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.895+0000 7fd30f6c7000 20 client.8607 populate_metadata read hostname 'smithi016' 2024-03-31T15:12:22.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.895+0000 7fd30f6c7000 1 --2- 172.21.15.16:0/852876974 >> [v2:172.21.15.16:6833/490717480,v1:172.21.15.16:6835/490717480] conn(0x5574db869a70 0x5574db869e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:22.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.895+0000 7fd30f6c7000 4 client.8607 mds_command: new command op to 8074 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-31T15:12:22.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.895+0000 7fd30f6c7000 1 -- 172.21.15.16:0/852876974 --> [v2:172.21.15.16:6833/490717480,v1:172.21.15.16:6835/490717480] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5574db63bbc0 con 0x5574db869a70 2024-03-31T15:12:22.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.895+0000 7fd309800640 1 --2- 172.21.15.16:0/852876974 >> [v2:172.21.15.16:6833/490717480,v1:172.21.15.16:6835/490717480] conn(0x5574db869a70 0x5574db869e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:22.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.895+0000 7fd309800640 1 --2- 172.21.15.16:0/852876974 >> [v2:172.21.15.16:6833/490717480,v1:172.21.15.16:6835/490717480] conn(0x5574db869a70 0x5574db869e50 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f8f9096767d209e server_cookie=d491847ae8dea716 in_seq=0 out_seq=0 2024-03-31T15:12:22.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.895+0000 7fd2eaa00640 10 client.8607 ms_handle_connect on v2:172.21.15.16:6833/490717480 2024-03-31T15:12:22.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.895+0000 7fd2eaa00640 1 -- 172.21.15.16:0/852876974 <== mds.0 v2:172.21.15.16:6833/490717480 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5574db63bbc0 con 0x5574db869a70 2024-03-31T15:12:22.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.895+0000 7fd2eaa00640 10 client.8607 handle_command_reply: tid=0 2024-03-31T15:12:22.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.895+0000 7fd30f6c7000 1 client.8607 shutdown 2024-03-31T15:12:22.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.895+0000 7fd30f6c7000 1 -- 172.21.15.16:0/852876974 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5574db89a4a0 msgr2=0x5574db8a2b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:22.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.899+0000 7fd30f6c7000 1 --2- 172.21.15.16:0/852876974 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5574db89a4a0 0x5574db8a2b40 secure :-1 s=READY pgs=1420 cs=0 l=1 rev1=1 crypto rx=0x7fd2f00f8d80 tx=0x7fd2f00f02c0 comp rx=0 tx=0).stop 2024-03-31T15:12:22.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.899+0000 7fd30f6c7000 1 -- 172.21.15.16:0/852876974 shutdown_connections 2024-03-31T15:12:22.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.899+0000 7fd30f6c7000 1 --2- 172.21.15.16:0/852876974 >> [v2:172.21.15.16:6833/490717480,v1:172.21.15.16:6835/490717480] conn(0x5574db869a70 0x5574db869e50 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:22.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.899+0000 7fd30f6c7000 1 --2- 172.21.15.16:0/852876974 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5574db89adc0 0x5574db868d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:22.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.899+0000 7fd30f6c7000 1 --2- 172.21.15.16:0/852876974 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5574db89a4a0 0x5574db8a2b40 unknown :-1 s=CLOSED pgs=1420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:22.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.899+0000 7fd2eaa00640 0 client.8607 ms_handle_reset on v2:172.21.15.16:6833/490717480 2024-03-31T15:12:22.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.899+0000 7fd30f6c7000 1 --2- 172.21.15.16:0/852876974 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5574db84e0b0 0x5574db8a2600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:22.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.899+0000 7fd30f6c7000 1 -- 172.21.15.16:0/852876974 >> 172.21.15.16:0/852876974 conn(0x5574db738b80 msgr2=0x5574db84d270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:12:22.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.899+0000 7fd30f6c7000 1 -- 172.21.15.16:0/852876974 shutdown_connections 2024-03-31T15:12:22.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.899+0000 7fd30f6c7000 1 -- 172.21.15.16:0/852876974 wait complete. 2024-03-31T15:12:22.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.899+0000 7fd30f6c7000 20 client.8607 trim_cache size 0 max 16384 2024-03-31T15:12:22.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.915+0000 7fd30f6c7000 1 Processor -- start 2024-03-31T15:12:22.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.915+0000 7fd30f6c7000 1 -- start start 2024-03-31T15:12:22.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.915+0000 7fd30f6c7000 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5574db84e0b0 0x5574db8553c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:22.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.915+0000 7fd30f6c7000 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5574db89a4a0 0x5574db855900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:22.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.915+0000 7fd30f6c7000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5574db89adc0 0x5574db899770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:22.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.915+0000 7fd30f6c7000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fd304203920 con 0x5574db89adc0 2024-03-31T15:12:22.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.915+0000 7fd30f6c7000 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fd30413f1a0 con 0x5574db89a4a0 2024-03-31T15:12:22.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.915+0000 7fd30f6c7000 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fd30413e500 con 0x5574db84e0b0 2024-03-31T15:12:22.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.915+0000 7fd30a200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5574db89adc0 0x5574db899770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:22.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.915+0000 7fd30a200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5574db89adc0 0x5574db899770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40352/0 (socket says 172.21.15.16:40352) 2024-03-31T15:12:22.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.915+0000 7fd30a200640 1 -- 172.21.15.16:0/3098298620 learned_addr learned my addr 172.21.15.16:0/3098298620 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:12:22.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.915+0000 7fd30a200640 1 -- 172.21.15.16:0/3098298620 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5574db84e0b0 msgr2=0x5574db8553c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:12:22.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.915+0000 7fd308e00640 1 --2- 172.21.15.16:0/3098298620 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5574db89a4a0 0x5574db855900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:22.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.915+0000 7fd30a200640 1 --2- 172.21.15.16:0/3098298620 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5574db84e0b0 0x5574db8553c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:22.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.915+0000 7fd30a200640 1 -- 172.21.15.16:0/3098298620 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5574db89a4a0 msgr2=0x5574db855900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:12:22.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.919+0000 7fd30a200640 1 --2- 172.21.15.16:0/3098298620 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5574db89a4a0 0x5574db855900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:22.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.919+0000 7fd30a200640 1 -- 172.21.15.16:0/3098298620 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5574db65b490 con 0x5574db89adc0 2024-03-31T15:12:22.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.919+0000 7fd308e00640 1 --2- 172.21.15.16:0/3098298620 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5574db89a4a0 0x5574db855900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:12:22.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.919+0000 7fd30a200640 1 --2- 172.21.15.16:0/3098298620 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5574db89adc0 0x5574db899770 secure :-1 s=READY pgs=1421 cs=0 l=1 rev1=1 crypto rx=0x7fd2fc014670 tx=0x7fd2fc014b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca354afeb8cb48f8 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:12:22.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.919+0000 7fd2f5600640 1 -- 172.21.15.16:0/3098298620 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2fc003ba0 con 0x5574db89adc0 2024-03-31T15:12:22.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.919+0000 7fd2f5600640 1 -- 172.21.15.16:0/3098298620 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd2fc003d60 con 0x5574db89adc0 2024-03-31T15:12:22.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.919+0000 7fd2f5600640 1 -- 172.21.15.16:0/3098298620 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2fc0129c0 con 0x5574db89adc0 2024-03-31T15:12:22.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.919+0000 7fd30f6c7000 1 -- 172.21.15.16:0/3098298620 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5574db89adc0 msgr2=0x5574db899770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:22.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.919+0000 7fd30f6c7000 1 --2- 172.21.15.16:0/3098298620 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5574db89adc0 0x5574db899770 secure :-1 s=READY pgs=1421 cs=0 l=1 rev1=1 crypto rx=0x7fd2fc014670 tx=0x7fd2fc014b30 comp rx=0 tx=0).stop 2024-03-31T15:12:22.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.919+0000 7fd30f6c7000 1 -- 172.21.15.16:0/3098298620 shutdown_connections 2024-03-31T15:12:22.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.919+0000 7fd30f6c7000 1 --2- 172.21.15.16:0/3098298620 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5574db89adc0 0x5574db899770 unknown :-1 s=CLOSED pgs=1421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:22.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.919+0000 7fd30f6c7000 1 --2- 172.21.15.16:0/3098298620 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5574db89a4a0 0x5574db855900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:22.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.919+0000 7fd30f6c7000 1 --2- 172.21.15.16:0/3098298620 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5574db84e0b0 0x5574db8553c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:22.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.919+0000 7fd30f6c7000 1 -- 172.21.15.16:0/3098298620 >> 172.21.15.16:0/3098298620 conn(0x5574db738b80 msgr2=0x5574db8a1a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:12:22.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.919+0000 7fd30f6c7000 1 -- 172.21.15.16:0/3098298620 shutdown_connections 2024-03-31T15:12:22.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.919+0000 7fd30f6c7000 1 -- 172.21.15.16:0/3098298620 wait complete. 2024-03-31T15:12:22.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.919+0000 7fd30f6c7000 1 -- start start 2024-03-31T15:12:22.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.919+0000 7fd30f6c7000 1 Processor -- start 2024-03-31T15:12:22.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.919+0000 7fd30f6c7000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5574db84e0b0 0x5574db906c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:22.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.919+0000 7fd30f6c7000 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5574db89a4a0 0x5574db8997b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:22.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.919+0000 7fd30f6c7000 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5574db89adc0 0x5574db899cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:22.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.919+0000 7fd30f6c7000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5574db6a4f80 con 0x5574db84e0b0 2024-03-31T15:12:22.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.919+0000 7fd30f6c7000 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fd30413eb50 con 0x5574db89a4a0 2024-03-31T15:12:22.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.919+0000 7fd30f6c7000 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x5574db69fc60 con 0x5574db89adc0 2024-03-31T15:12:22.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.919+0000 7fd30f6c7000 10 client.0 _ll_register_callbacks cb 0x7fd2fc0099d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T15:12:22.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.919+0000 7fd309800640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5574db84e0b0 0x5574db906c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:22.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.919+0000 7fd309800640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5574db84e0b0 0x5574db906c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40364/0 (socket says 172.21.15.16:40364) 2024-03-31T15:12:22.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.919+0000 7fd309800640 1 -- 172.21.15.16:0/1572579865 learned_addr learned my addr 172.21.15.16:0/1572579865 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:12:22.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.919+0000 7fd30a200640 1 --2- 172.21.15.16:0/1572579865 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5574db89adc0 0x5574db899cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:22.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.919+0000 7fd308e00640 1 --2- 172.21.15.16:0/1572579865 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5574db89a4a0 0x5574db8997b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:22.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.919+0000 7fd309800640 1 -- 172.21.15.16:0/1572579865 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5574db89adc0 msgr2=0x5574db899cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:12:22.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.919+0000 7fd309800640 1 --2- 172.21.15.16:0/1572579865 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5574db89adc0 0x5574db899cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:22.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.919+0000 7fd309800640 1 -- 172.21.15.16:0/1572579865 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5574db89a4a0 msgr2=0x5574db8997b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:12:22.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.919+0000 7fd309800640 1 --2- 172.21.15.16:0/1572579865 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5574db89a4a0 0x5574db8997b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:22.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.919+0000 7fd309800640 1 -- 172.21.15.16:0/1572579865 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5574db659e80 con 0x5574db84e0b0 2024-03-31T15:12:22.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.919+0000 7fd309800640 1 --2- 172.21.15.16:0/1572579865 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5574db84e0b0 0x5574db906c90 secure :-1 s=READY pgs=1422 cs=0 l=1 rev1=1 crypto rx=0x7fd2f800fa90 tx=0x7fd2f8007e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=982a12028362e3d9 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:12:22.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.919+0000 7fd2eaa00640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-31T15:12:22.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.923+0000 7fd2eaa00640 1 -- 172.21.15.16:0/1572579865 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2f8038060 con 0x5574db84e0b0 2024-03-31T15:12:22.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.923+0000 7fd2eaa00640 1 -- 172.21.15.16:0/1572579865 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 5709+0+0 (secure 0 0 0) 0x7fd2f8020920 con 0x5574db84e0b0 2024-03-31T15:12:22.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.923+0000 7fd30f6c7000 1 -- 172.21.15.16:0/1572579865 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd2fc003d60 con 0x5574db84e0b0 2024-03-31T15:12:22.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.923+0000 7fd2eaa00640 1 -- 172.21.15.16:0/1572579865 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=83) ==== 24+0+0 (secure 0 0 0) 0x7fd2f8025040 con 0x5574db84e0b0 2024-03-31T15:12:22.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.923+0000 7fd30f6c7000 10 client.8613 fetch_fsmap learned FSMap version 83 2024-03-31T15:12:22.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.923+0000 7fd30f6c7000 1 -- 172.21.15.16:0/1572579865 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=83,monmap=2+}) -- 0x5574db647270 con 0x5574db84e0b0 2024-03-31T15:12:22.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.923+0000 7fd2eaa00640 1 -- 172.21.15.16:0/1572579865 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd2fc003d60 con 0x5574db84e0b0 2024-03-31T15:12:22.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.923+0000 7fd2eaa00640 1 -- 172.21.15.16:0/1572579865 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 83) ==== 3494+0+0 (secure 0 0 0) 0x7fd2f8020b20 con 0x5574db84e0b0 2024-03-31T15:12:22.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.923+0000 7fd30f6c7000 10 client.8613 fetch_fsmap finished waiting for FSMap version 83 2024-03-31T15:12:22.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.923+0000 7fd30f6c7000 10 client.8613 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-31T15:12:22.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.923+0000 7fd30f6c7000 20 client.8613 populate_metadata read hostname 'smithi016' 2024-03-31T15:12:22.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.923+0000 7fd30f6c7000 1 --2- 172.21.15.16:0/1572579865 >> [v2:172.21.15.16:6833/490717480,v1:172.21.15.16:6835/490717480] conn(0x5574db84c2d0 0x5574db84c6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:22.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.923+0000 7fd30f6c7000 4 client.8613 mds_command: new command op to 8074 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-31T15:12:22.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.923+0000 7fd30f6c7000 1 -- 172.21.15.16:0/1572579865 --> [v2:172.21.15.16:6833/490717480,v1:172.21.15.16:6835/490717480] -- command(tid 0: {"prefix": "status"}) -- 0x5574db6ade90 con 0x5574db84c2d0 2024-03-31T15:12:22.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.923+0000 7fd308e00640 1 --2- 172.21.15.16:0/1572579865 >> [v2:172.21.15.16:6833/490717480,v1:172.21.15.16:6835/490717480] conn(0x5574db84c2d0 0x5574db84c6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:22.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.923+0000 7fd308e00640 1 --2- 172.21.15.16:0/1572579865 >> [v2:172.21.15.16:6833/490717480,v1:172.21.15.16:6835/490717480] conn(0x5574db84c2d0 0x5574db84c6b0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fce1a5ab11047b9b server_cookie=55ca7d8f6acd6e1 in_seq=0 out_seq=0 2024-03-31T15:12:22.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.923+0000 7fd2eaa00640 10 client.8613 ms_handle_connect on v2:172.21.15.16:6833/490717480 2024-03-31T15:12:22.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.923+0000 7fd2eaa00640 1 -- 172.21.15.16:0/1572579865 <== mds.0 v2:172.21.15.16:6833/490717480 1 ==== command_reply(tid 0: 0 ) ==== 8+0+331 (crc 0 0 0) 0x5574db6ade90 con 0x5574db84c2d0 2024-03-31T15:12:22.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.923+0000 7fd2eaa00640 10 client.8613 handle_command_reply: tid=0 2024-03-31T15:12:22.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.923+0000 7fd30f6c7000 1 client.8613 shutdown 2024-03-31T15:12:22.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.923+0000 7fd30f6c7000 1 -- 172.21.15.16:0/1572579865 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5574db84e0b0 msgr2=0x5574db906c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:22.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.923+0000 7fd30f6c7000 1 --2- 172.21.15.16:0/1572579865 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5574db84e0b0 0x5574db906c90 secure :-1 s=READY pgs=1422 cs=0 l=1 rev1=1 crypto rx=0x7fd2f800fa90 tx=0x7fd2f8007e60 comp rx=0 tx=0).stop 2024-03-31T15:12:22.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.923+0000 7fd30f6c7000 1 -- 172.21.15.16:0/1572579865 shutdown_connections 2024-03-31T15:12:22.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.923+0000 7fd30f6c7000 1 --2- 172.21.15.16:0/1572579865 >> [v2:172.21.15.16:6833/490717480,v1:172.21.15.16:6835/490717480] conn(0x5574db84c2d0 0x5574db84c6b0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:22.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.923+0000 7fd30f6c7000 1 --2- 172.21.15.16:0/1572579865 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5574db89adc0 0x5574db899cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:22.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.923+0000 7fd30f6c7000 1 --2- 172.21.15.16:0/1572579865 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5574db89a4a0 0x5574db8997b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:22.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.923+0000 7fd30f6c7000 1 --2- 172.21.15.16:0/1572579865 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5574db84e0b0 0x5574db906c90 unknown :-1 s=CLOSED pgs=1422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:22.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.923+0000 7fd2eaa00640 0 client.8613 ms_handle_reset on v2:172.21.15.16:6833/490717480 2024-03-31T15:12:22.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.923+0000 7fd30f6c7000 1 -- 172.21.15.16:0/1572579865 >> 172.21.15.16:0/1572579865 conn(0x5574db738b80 msgr2=0x5574db8553c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:12:22.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.923+0000 7fd30f6c7000 1 -- 172.21.15.16:0/1572579865 shutdown_connections 2024-03-31T15:12:22.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.923+0000 7fd30f6c7000 1 -- 172.21.15.16:0/1572579865 wait complete. 2024-03-31T15:12:22.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.923+0000 7fd30f6c7000 20 client.8613 trim_cache size 0 max 16384 2024-03-31T15:12:22.921 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-03-31T15:12:22.921 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_fsid": "286eb5da-fb47-4550-b290-d9519c13b9a5", 2024-03-31T15:12:22.921 INFO:teuthology.orchestra.run.smithi016.stdout: "whoami": 0, 2024-03-31T15:12:22.921 INFO:teuthology.orchestra.run.smithi016.stdout: "id": 8074, 2024-03-31T15:12:22.921 INFO:teuthology.orchestra.run.smithi016.stdout: "want_state": "up:active", 2024-03-31T15:12:22.921 INFO:teuthology.orchestra.run.smithi016.stdout: "state": "up:active", 2024-03-31T15:12:22.921 INFO:teuthology.orchestra.run.smithi016.stdout: "fs_name": "cephfs", 2024-03-31T15:12:22.922 INFO:teuthology.orchestra.run.smithi016.stdout: "rank_uptime": 6.0190988909999996, 2024-03-31T15:12:22.922 INFO:teuthology.orchestra.run.smithi016.stdout: "mdsmap_epoch": 83, 2024-03-31T15:12:22.922 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch": 96, 2024-03-31T15:12:22.922 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch_barrier": 96, 2024-03-31T15:12:22.922 INFO:teuthology.orchestra.run.smithi016.stdout: "uptime": 10.935363947999999 2024-03-31T15:12:22.922 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-03-31T15:12:22.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.923+0000 7fd30c200640 1 -- 172.21.15.16:0/574342244 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fd2e4068380 msgr2=0x7fd2e406a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:22.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.923+0000 7fd30c200640 1 --2- 172.21.15.16:0/574342244 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fd2e4068380 0x7fd2e406a840 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7fd2f8001810 tx=0x7fd2f8009040 comp rx=0 tx=0).stop 2024-03-31T15:12:22.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.923+0000 7fd30c200640 1 -- 172.21.15.16:0/574342244 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd30406f920 msgr2=0x7fd3040b7ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:22.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.923+0000 7fd30c200640 1 --2- 172.21.15.16:0/574342244 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd30406f920 0x7fd3040b7ad0 secure :-1 s=READY pgs=1418 cs=0 l=1 rev1=1 crypto rx=0x7fd2f001abc0 tx=0x7fd2f001ad50 comp rx=0 tx=0).stop 2024-03-31T15:12:22.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.923+0000 7fd30c200640 1 -- 172.21.15.16:0/574342244 shutdown_connections 2024-03-31T15:12:22.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.923+0000 7fd30c200640 1 --2- 172.21.15.16:0/574342244 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fd2e4068380 0x7fd2e406a840 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:22.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.923+0000 7fd30c200640 1 --2- 172.21.15.16:0/574342244 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fd3040cc740 0x7fd3040b55c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:22.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.923+0000 7fd30c200640 1 --2- 172.21.15.16:0/574342244 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd30406f920 0x7fd3040b7ad0 unknown :-1 s=CLOSED pgs=1418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:22.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.923+0000 7fd30c200640 1 --2- 172.21.15.16:0/574342244 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd30406d490 0x7fd3041611b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:22.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.923+0000 7fd30c200640 1 -- 172.21.15.16:0/574342244 >> 172.21.15.16:0/574342244 conn(0x7fd304067330 msgr2=0x7fd3040d2630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:12:22.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.923+0000 7fd30c200640 1 -- 172.21.15.16:0/574342244 shutdown_connections 2024-03-31T15:12:22.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:22.923+0000 7fd30c200640 1 -- 172.21.15.16:0/574342244 wait complete. 2024-03-31T15:12:22.937 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-31T15:12:23.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:23.083+0000 7f2c73c00640 1 Processor -- start 2024-03-31T15:12:23.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:23.083+0000 7f2c73c00640 1 -- start start 2024-03-31T15:12:23.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:23.087+0000 7f2c73c00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f2c6c0cef40 0x7f2c6c0b5b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:23.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:23.087+0000 7f2c73c00640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2c6c0ce8a0 con 0x7f2c6c15b4d0 2024-03-31T15:12:23.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:23.087+0000 7f2c73c00640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2c6c0709a0 con 0x7f2c6c0b61e0 2024-03-31T15:12:23.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:23.087+0000 7f2c73c00640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f2c6c06fbe0 con 0x7f2c6c0cef40 2024-03-31T15:12:23.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:23.087+0000 7f2c6be00640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f2c6c15b4d0 0x7f2c6c0cf320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:54970/0 (socket says 172.21.15.16:54970) 2024-03-31T15:12:23.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:23.087+0000 7f2c6be00640 1 -- 172.21.15.16:0/2984923816 learned_addr learned my addr 172.21.15.16:0/2984923816 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:12:23.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:23.087+0000 7f2c71200640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f2c6c0cef40 0x7f2c6c0b5b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:23.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:23.087+0000 7f2c71200640 1 -- 172.21.15.16:0/2984923816 >> v1:172.21.15.89:6789/0 conn(0x7f2c6c0b61e0 legacy=0x7f2c6c0b42e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:23.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:23.087+0000 7f2c71200640 1 -- 172.21.15.16:0/2984923816 >> v1:172.21.15.16:6789/0 conn(0x7f2c6c15b4d0 legacy=0x7f2c6c0cf320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:23.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:23.087+0000 7f2c71200640 1 -- 172.21.15.16:0/2984923816 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c6c06fec0 con 0x7f2c6c0cef40 2024-03-31T15:12:23.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:23.091+0000 7f2c71200640 1 --2- 172.21.15.16:0/2984923816 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2c6c0cef40 0x7f2c6c0b5b50 secure :-1 s=READY pgs=1424 cs=0 l=1 rev1=1 crypto rx=0x7f2c600097c0 tx=0x7f2c6001c470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bebf8b05a7851296 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:12:23.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:23.091+0000 7f2c6b400640 1 -- 172.21.15.16:0/2984923816 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c600a7070 con 0x7f2c6c0cef40 2024-03-31T15:12:23.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:23.091+0000 7f2c6b400640 1 -- 172.21.15.16:0/2984923816 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2c600b0070 con 0x7f2c6c0cef40 2024-03-31T15:12:23.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:23.091+0000 7f2c6b400640 1 -- 172.21.15.16:0/2984923816 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c600ab890 con 0x7f2c6c0cef40 2024-03-31T15:12:23.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:23.091+0000 7f2c73c00640 1 -- 172.21.15.16:0/2984923816 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2c6c0cef40 msgr2=0x7f2c6c0b5b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:23.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:23.091+0000 7f2c73c00640 1 --2- 172.21.15.16:0/2984923816 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2c6c0cef40 0x7f2c6c0b5b50 secure :-1 s=READY pgs=1424 cs=0 l=1 rev1=1 crypto rx=0x7f2c600097c0 tx=0x7f2c6001c470 comp rx=0 tx=0).stop 2024-03-31T15:12:23.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:23.091+0000 7f2c73c00640 1 -- 172.21.15.16:0/2984923816 shutdown_connections 2024-03-31T15:12:23.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:23.091+0000 7f2c73c00640 1 --2- 172.21.15.16:0/2984923816 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2c6c0cef40 0x7f2c6c0b5b50 unknown :-1 s=CLOSED pgs=1424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:23.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:23.091+0000 7f2c73c00640 1 -- 172.21.15.16:0/2984923816 >> 172.21.15.16:0/2984923816 conn(0x7f2c6c0d1060 msgr2=0x7f2c6c0d1470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:12:23.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:23.091+0000 7f2c73c00640 1 -- 172.21.15.16:0/2984923816 shutdown_connections 2024-03-31T15:12:23.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:23.091+0000 7f2c73c00640 1 -- 172.21.15.16:0/2984923816 wait complete. 2024-03-31T15:12:23.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:23.091+0000 7f2c73c00640 1 Processor -- start 2024-03-31T15:12:23.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:23.091+0000 7f2c73c00640 1 -- start start 2024-03-31T15:12:23.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:23.095+0000 7f2c73c00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2c6c0b61e0 0x7f2c6c15a090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:23.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:23.095+0000 7f2c73c00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2c6c0cef40 0x7f2c6c1fb550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:23.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:23.095+0000 7f2c73c00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f2c6c15b4d0 0x7f2c6c0cba00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:23.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:23.095+0000 7f2c73c00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f2c6c0b5380 con 0x7f2c6c0b61e0 2024-03-31T15:12:23.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:23.095+0000 7f2c73c00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f2c6c0b5200 con 0x7f2c6c0cef40 2024-03-31T15:12:23.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:23.095+0000 7f2c73c00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f2c6c0b5500 con 0x7f2c6c15b4d0 2024-03-31T15:12:23.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:23.095+0000 7f2c71200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2c6c0b61e0 0x7f2c6c15a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:23.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:23.095+0000 7f2c71200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2c6c0b61e0 0x7f2c6c15a090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40378/0 (socket says 172.21.15.16:40378) 2024-03-31T15:12:23.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:23.095+0000 7f2c71200640 1 -- 172.21.15.16:0/2968312553 learned_addr learned my addr 172.21.15.16:0/2968312553 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:12:23.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:23.095+0000 7f2c71c00640 1 --2- 172.21.15.16:0/2968312553 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f2c6c15b4d0 0x7f2c6c0cba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:23.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:23.095+0000 7f2c6be00640 1 --2- 172.21.15.16:0/2968312553 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2c6c0cef40 0x7f2c6c1fb550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:23.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:23.095+0000 7f2c71200640 1 -- 172.21.15.16:0/2968312553 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f2c6c15b4d0 msgr2=0x7f2c6c0cba00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:23.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:23.095+0000 7f2c71200640 1 --2- 172.21.15.16:0/2968312553 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f2c6c15b4d0 0x7f2c6c0cba00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:23.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:23.095+0000 7f2c71200640 1 -- 172.21.15.16:0/2968312553 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2c6c0cef40 msgr2=0x7f2c6c1fb550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:23.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:23.095+0000 7f2c71200640 1 --2- 172.21.15.16:0/2968312553 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2c6c0cef40 0x7f2c6c1fb550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:23.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:23.095+0000 7f2c71200640 1 -- 172.21.15.16:0/2968312553 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c6c1fba90 con 0x7f2c6c0b61e0 2024-03-31T15:12:23.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:23.095+0000 7f2c6be00640 1 --2- 172.21.15.16:0/2968312553 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2c6c0cef40 0x7f2c6c1fb550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:12:23.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:23.095+0000 7f2c71200640 1 --2- 172.21.15.16:0/2968312553 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2c6c0b61e0 0x7f2c6c15a090 secure :-1 s=READY pgs=1425 cs=0 l=1 rev1=1 crypto rx=0x7f2c600ad040 tx=0x7f2c6001acb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:12:23.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:23.095+0000 7f2c68c00640 1 -- 172.21.15.16:0/2968312553 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c600bf610 con 0x7f2c6c0b61e0 2024-03-31T15:12:23.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:23.095+0000 7f2c68c00640 1 -- 172.21.15.16:0/2968312553 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2c600bf7b0 con 0x7f2c6c0b61e0 2024-03-31T15:12:23.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:23.095+0000 7f2c68c00640 1 -- 172.21.15.16:0/2968312553 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c600bfba0 con 0x7f2c6c0b61e0 2024-03-31T15:12:23.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:23.095+0000 7f2c73c00640 1 -- 172.21.15.16:0/2968312553 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2c6c155050 con 0x7f2c6c0b61e0 2024-03-31T15:12:23.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:23.095+0000 7f2c73c00640 1 -- 172.21.15.16:0/2968312553 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2c6c155610 con 0x7f2c6c0b61e0 2024-03-31T15:12:23.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:23.095+0000 7f2c73c00640 1 -- 172.21.15.16:0/2968312553 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2c6c069950 con 0x7f2c6c0b61e0 2024-03-31T15:12:23.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:23.099+0000 7f2c68c00640 1 -- 172.21.15.16:0/2968312553 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f2c600bd020 con 0x7f2c6c0b61e0 2024-03-31T15:12:23.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:23.099+0000 7f2c68c00640 1 --2- 172.21.15.16:0/2968312553 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f2c4c068380 0x7f2c4c06a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:23.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:23.099+0000 7f2c68c00640 1 -- 172.21.15.16:0/2968312553 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 5709+0+0 (secure 0 0 0) 0x7f2c60120ff0 con 0x7f2c6c0b61e0 2024-03-31T15:12:23.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:23.099+0000 7f2c6be00640 1 --2- 172.21.15.16:0/2968312553 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f2c4c068380 0x7f2c4c06a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:23.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:23.099+0000 7f2c6be00640 1 --2- 172.21.15.16:0/2968312553 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f2c4c068380 0x7f2c4c06a840 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f2c6c15fc00 tx=0x7f2c58009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:12:23.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:23.107+0000 7f2c68c00640 1 -- 172.21.15.16:0/2968312553 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f2c600c6030 con 0x7f2c6c0b61e0 2024-03-31T15:12:23.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:23.311+0000 7f2c73c00640 1 -- 172.21.15.16:0/2968312553 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f2c6c0cec20 con 0x7f2c6c0b61e0 2024-03-31T15:12:23.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:23.311+0000 7f2c68c00640 1 -- 172.21.15.16:0/2968312553 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v83) ==== 53+0+73 (secure 0 0 0) 0x7f2c60016090 con 0x7f2c6c0b61e0 2024-03-31T15:12:23.305 INFO:teuthology.orchestra.run.smithi016.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-31T15:12:23.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:23.315+0000 7f2c73c00640 1 -- 172.21.15.16:0/2968312553 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f2c4c068380 msgr2=0x7f2c4c06a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:23.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:23.315+0000 7f2c73c00640 1 --2- 172.21.15.16:0/2968312553 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f2c4c068380 0x7f2c4c06a840 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f2c6c15fc00 tx=0x7f2c58009040 comp rx=0 tx=0).stop 2024-03-31T15:12:23.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:23.315+0000 7f2c73c00640 1 -- 172.21.15.16:0/2968312553 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2c6c0b61e0 msgr2=0x7f2c6c15a090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:23.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:23.315+0000 7f2c73c00640 1 --2- 172.21.15.16:0/2968312553 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2c6c0b61e0 0x7f2c6c15a090 secure :-1 s=READY pgs=1425 cs=0 l=1 rev1=1 crypto rx=0x7f2c600ad040 tx=0x7f2c6001acb0 comp rx=0 tx=0).stop 2024-03-31T15:12:23.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:23.315+0000 7f2c73c00640 1 -- 172.21.15.16:0/2968312553 shutdown_connections 2024-03-31T15:12:23.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:23.315+0000 7f2c73c00640 1 --2- 172.21.15.16:0/2968312553 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f2c4c068380 0x7f2c4c06a840 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:23.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:23.315+0000 7f2c73c00640 1 --2- 172.21.15.16:0/2968312553 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f2c6c15b4d0 0x7f2c6c0cba00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:23.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:23.315+0000 7f2c73c00640 1 --2- 172.21.15.16:0/2968312553 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2c6c0cef40 0x7f2c6c1fb550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:23.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:23.315+0000 7f2c73c00640 1 --2- 172.21.15.16:0/2968312553 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2c6c0b61e0 0x7f2c6c15a090 unknown :-1 s=CLOSED pgs=1425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:23.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:23.315+0000 7f2c73c00640 1 -- 172.21.15.16:0/2968312553 >> 172.21.15.16:0/2968312553 conn(0x7f2c6c0d1060 msgr2=0x7f2c6c06b620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:12:23.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:23.315+0000 7f2c73c00640 1 -- 172.21.15.16:0/2968312553 shutdown_connections 2024-03-31T15:12:23.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:23.315+0000 7f2c73c00640 1 -- 172.21.15.16:0/2968312553 wait complete. 2024-03-31T15:12:23.327 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-31T15:12:23.327 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-31T15:12:23.327 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi016.front.sepia.ceph.com 2024-03-31T15:12:23.327 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-31T15:12:23.327 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-31T15:12:23.327 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-31T15:12:23.327 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-31T15:12:23.327 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-31T15:12:23.327 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-31T15:12:23.328 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:12:23.328 DEBUG:teuthology.orchestra.run.smithi016:> ip addr 2024-03-31T15:12:23.337 INFO:teuthology.orchestra.run.smithi016.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-31T15:12:23.337 INFO:teuthology.orchestra.run.smithi016.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-31T15:12:23.337 INFO:teuthology.orchestra.run.smithi016.stdout: inet 127.0.0.1/8 scope host lo 2024-03-31T15:12:23.337 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2024-03-31T15:12:23.338 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 ::1/128 scope host 2024-03-31T15:12:23.338 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2024-03-31T15:12:23.338 INFO:teuthology.orchestra.run.smithi016.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-03-31T15:12:23.338 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:6b:fd:e0 brd ff:ff:ff:ff:ff:ff 2024-03-31T15:12:23.338 INFO:teuthology.orchestra.run.smithi016.stdout: altname enp7s0f0 2024-03-31T15:12:23.338 INFO:teuthology.orchestra.run.smithi016.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-03-31T15:12:23.338 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:6b:fd:e1 brd ff:ff:ff:ff:ff:ff 2024-03-31T15:12:23.338 INFO:teuthology.orchestra.run.smithi016.stdout: altname enp7s0f1 2024-03-31T15:12:23.338 INFO:teuthology.orchestra.run.smithi016.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-31T15:12:23.338 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:bd:15:64 brd ff:ff:ff:ff:ff:ff 2024-03-31T15:12:23.338 INFO:teuthology.orchestra.run.smithi016.stdout: altname enp1s0f0 2024-03-31T15:12:23.338 INFO:teuthology.orchestra.run.smithi016.stdout: inet 172.21.15.16/20 brd 172.21.15.255 scope global dynamic ens1f0 2024-03-31T15:12:23.339 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft 41937sec preferred_lft 41937sec 2024-03-31T15:12:23.339 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 fe80::ec4:7aff:febd:1564/64 scope link 2024-03-31T15:12:23.339 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2024-03-31T15:12:23.339 INFO:teuthology.orchestra.run.smithi016.stdout:5: ens1f1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-03-31T15:12:23.339 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:bd:15:65 brd ff:ff:ff:ff:ff:ff 2024-03-31T15:12:23.339 INFO:teuthology.orchestra.run.smithi016.stdout: altname enp1s0f1 2024-03-31T15:12:23.339 INFO:teuthology.orchestra.run.smithi016.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2024-03-31T15:12:23.339 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 02:42:d4:1f:7d:73 brd ff:ff:ff:ff:ff:ff 2024-03-31T15:12:23.339 INFO:teuthology.orchestra.run.smithi016.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2024-03-31T15:12:23.339 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2024-03-31T15:12:23.339 INFO:teuthology.orchestra.run.smithi016.stdout:7: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-31T15:12:23.339 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 5e:1f:84:6d:79:ae brd ff:ff:ff:ff:ff:ff 2024-03-31T15:12:23.339 INFO:teuthology.orchestra.run.smithi016.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-31T15:12:23.339 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2024-03-31T15:12:23.340 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 fe80::5c1f:84ff:fe6d:79ae/64 scope link 2024-03-31T15:12:23.340 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2024-03-31T15:12:23.340 INFO:teuthology.orchestra.run.smithi016.stdout:8: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-31T15:12:23.340 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0e:5c:f6:1f:82:2f brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-31T15:12:23.340 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 fe80::c5c:f6ff:fe1f:822f/64 scope link 2024-03-31T15:12:23.340 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2024-03-31T15:12:23.340 INFO:teuthology.orchestra.run.smithi016.stdout:9: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-31T15:12:23.340 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether c6:a6:3d:69:d0:db brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-31T15:12:23.340 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 fe80::c4a6:3dff:fe69:d0db/64 scope link 2024-03-31T15:12:23.340 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2024-03-31T15:12:23.340 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:12:23.340 DEBUG:teuthology.orchestra.run.smithi016:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-31T15:12:23.398 INFO:teuthology.orchestra.run.smithi016.stdout:1 2024-03-31T15:12:23.400 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:12:23.401 DEBUG:teuthology.orchestra.run.smithi016:> route 2024-03-31T15:12:23.450 INFO:teuthology.orchestra.run.smithi016.stdout:Kernel IP routing table 2024-03-31T15:12:23.450 INFO:teuthology.orchestra.run.smithi016.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-31T15:12:23.450 INFO:teuthology.orchestra.run.smithi016.stdout:default _gateway 0.0.0.0 UG 0 0 0 ens1f0 2024-03-31T15:12:23.450 INFO:teuthology.orchestra.run.smithi016.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2024-03-31T15:12:23.450 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 0 0 0 ens1f0 2024-03-31T15:12:23.450 INFO:teuthology.orchestra.run.smithi016.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-31T15:12:23.451 DEBUG:teuthology.orchestra.run.smithi016:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-31T15:12:23.451 DEBUG:teuthology.orchestra.run.smithi016:> set -e 2024-03-31T15:12:23.451 DEBUG:teuthology.orchestra.run.smithi016:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-03-31T15:12:23.451 DEBUG:teuthology.orchestra.run.smithi016:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-03-31T15:12:23.451 DEBUG:teuthology.orchestra.run.smithi016:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-03-31T15:12:23.451 DEBUG:teuthology.orchestra.run.smithi016:> ') 2024-03-31T15:12:23.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:23 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-31T15:12:23.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:23 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-31T15:12:23.589 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:12:23.589 DEBUG:teuthology.orchestra.run.smithi016:> ip netns list 2024-03-31T15:12:23.599 INFO:teuthology.orchestra.run.smithi016.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-31T15:12:23.599 INFO:teuthology.orchestra.run.smithi016.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-31T15:12:23.600 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-31T15:12:23.600 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:12:23.600 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip link set brx.0 up 2024-03-31T15:12:23.662 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-03-31T15:12:23.662 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-31T15:12:23.662 DEBUG:teuthology.orchestra.run.smithi016:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-31T15:12:23.714 INFO:teuthology.orchestra.run.smithi016.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-31T15:12:23.715 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-31T15:12:23.715 DEBUG:teuthology.orchestra.run.smithi016:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-31T15:12:23.770 DEBUG:teuthology.orchestra.run.smithi016:> sudo modprobe fuse 2024-03-31T15:12:23.834 DEBUG:teuthology.orchestra.run.smithi016:> cat /proc/self/mounts | awk '{print $2}' 2024-03-31T15:12:23.887 INFO:teuthology.orchestra.run.smithi016.stdout:/sys 2024-03-31T15:12:23.887 INFO:teuthology.orchestra.run.smithi016.stdout:/proc 2024-03-31T15:12:23.887 INFO:teuthology.orchestra.run.smithi016.stdout:/dev 2024-03-31T15:12:23.887 INFO:teuthology.orchestra.run.smithi016.stdout:/dev/pts 2024-03-31T15:12:23.887 INFO:teuthology.orchestra.run.smithi016.stdout:/run 2024-03-31T15:12:23.887 INFO:teuthology.orchestra.run.smithi016.stdout:/ 2024-03-31T15:12:23.887 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/kernel/security 2024-03-31T15:12:23.887 INFO:teuthology.orchestra.run.smithi016.stdout:/dev/shm 2024-03-31T15:12:23.887 INFO:teuthology.orchestra.run.smithi016.stdout:/run/lock 2024-03-31T15:12:23.888 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/cgroup 2024-03-31T15:12:23.888 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/pstore 2024-03-31T15:12:23.888 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/bpf 2024-03-31T15:12:23.888 INFO:teuthology.orchestra.run.smithi016.stdout:/proc/sys/fs/binfmt_misc 2024-03-31T15:12:23.888 INFO:teuthology.orchestra.run.smithi016.stdout:/dev/hugepages 2024-03-31T15:12:23.888 INFO:teuthology.orchestra.run.smithi016.stdout:/dev/mqueue 2024-03-31T15:12:23.888 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/kernel/debug 2024-03-31T15:12:23.888 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/kernel/tracing 2024-03-31T15:12:23.888 INFO:teuthology.orchestra.run.smithi016.stdout:/proc/fs/nfsd 2024-03-31T15:12:23.888 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/fuse/connections 2024-03-31T15:12:23.888 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/kernel/config 2024-03-31T15:12:23.888 INFO:teuthology.orchestra.run.smithi016.stdout:/run/credentials/systemd-sysusers.service 2024-03-31T15:12:23.888 INFO:teuthology.orchestra.run.smithi016.stdout:/run/rpc_pipefs 2024-03-31T15:12:23.888 INFO:teuthology.orchestra.run.smithi016.stdout:/run/user/1000 2024-03-31T15:12:23.888 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/ceph 2024-03-31T15:12:23.889 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/ceph/osd/ceph-0 2024-03-31T15:12:23.889 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/ceph/osd/ceph-1 2024-03-31T15:12:23.889 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/ceph/osd/ceph-2 2024-03-31T15:12:23.889 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/ceph/osd/ceph-3 2024-03-31T15:12:23.889 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns 2024-03-31T15:12:23.889 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-31T15:12:23.889 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-31T15:12:23.889 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-31T15:12:23.889 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-31T15:12:23.890 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:12:23.890 DEBUG:teuthology.orchestra.run.smithi016:> ls /sys/fs/fuse/connections 2024-03-31T15:12:23.941 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-31T15:12:23.941 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-03-31T15:12:23.988 DEBUG:teuthology.orchestra.run.smithi016:> sudo modprobe fuse 2024-03-31T15:12:24.009 DEBUG:teuthology.orchestra.run.smithi016:> cat /proc/self/mounts | awk '{print $2}' 2024-03-31T15:12:24.063 INFO:teuthology.orchestra.run.smithi016.stdout:/sys 2024-03-31T15:12:24.063 INFO:teuthology.orchestra.run.smithi016.stdout:/proc 2024-03-31T15:12:24.063 INFO:teuthology.orchestra.run.smithi016.stdout:/dev 2024-03-31T15:12:24.063 INFO:teuthology.orchestra.run.smithi016.stdout:/dev/pts 2024-03-31T15:12:24.063 INFO:teuthology.orchestra.run.smithi016.stdout:/run 2024-03-31T15:12:24.063 INFO:teuthology.orchestra.run.smithi016.stdout:/ 2024-03-31T15:12:24.063 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/kernel/security 2024-03-31T15:12:24.063 INFO:teuthology.orchestra.run.smithi016.stdout:/dev/shm 2024-03-31T15:12:24.063 INFO:teuthology.orchestra.run.smithi016.stdout:/run/lock 2024-03-31T15:12:24.063 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/cgroup 2024-03-31T15:12:24.063 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/pstore 2024-03-31T15:12:24.064 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/bpf 2024-03-31T15:12:24.064 INFO:teuthology.orchestra.run.smithi016.stdout:/proc/sys/fs/binfmt_misc 2024-03-31T15:12:24.064 INFO:teuthology.orchestra.run.smithi016.stdout:/dev/hugepages 2024-03-31T15:12:24.064 INFO:teuthology.orchestra.run.smithi016.stdout:/dev/mqueue 2024-03-31T15:12:24.064 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/kernel/debug 2024-03-31T15:12:24.064 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/kernel/tracing 2024-03-31T15:12:24.064 INFO:teuthology.orchestra.run.smithi016.stdout:/proc/fs/nfsd 2024-03-31T15:12:24.064 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/fuse/connections 2024-03-31T15:12:24.064 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/kernel/config 2024-03-31T15:12:24.064 INFO:teuthology.orchestra.run.smithi016.stdout:/run/credentials/systemd-sysusers.service 2024-03-31T15:12:24.064 INFO:teuthology.orchestra.run.smithi016.stdout:/run/rpc_pipefs 2024-03-31T15:12:24.064 INFO:teuthology.orchestra.run.smithi016.stdout:/run/user/1000 2024-03-31T15:12:24.064 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/ceph 2024-03-31T15:12:24.064 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/ceph/osd/ceph-0 2024-03-31T15:12:24.065 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/ceph/osd/ceph-1 2024-03-31T15:12:24.065 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/ceph/osd/ceph-2 2024-03-31T15:12:24.065 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/ceph/osd/ceph-3 2024-03-31T15:12:24.065 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns 2024-03-31T15:12:24.065 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-31T15:12:24.065 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-31T15:12:24.065 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-31T15:12:24.065 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-31T15:12:24.066 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:12:24.066 DEBUG:teuthology.orchestra.run.smithi016:> ls /sys/fs/fuse/connections 2024-03-31T15:12:24.084 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi016.stderr:2024-03-31T15:12:24.091+0000 7fbd56123580 -1 init, newargv = 0x55946e8de240 newargc=15 2024-03-31T15:12:24.084 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi016.stderr:ceph-fuse[48278]: starting ceph client 2024-03-31T15:12:24.091 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi016.stderr:ceph-fuse[48278]: starting fuse 2024-03-31T15:12:24.095 INFO:teuthology.orchestra.run.smithi016.stdout:43 2024-03-31T15:12:24.095 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [43] 2024-03-31T15:12:24.096 DEBUG:teuthology.orchestra.run.smithi016:> sudo stdin-killer -- python3 -c ' 2024-03-31T15:12:24.096 DEBUG:teuthology.orchestra.run.smithi016:> import glob 2024-03-31T15:12:24.096 DEBUG:teuthology.orchestra.run.smithi016:> import re 2024-03-31T15:12:24.096 DEBUG:teuthology.orchestra.run.smithi016:> import os 2024-03-31T15:12:24.096 DEBUG:teuthology.orchestra.run.smithi016:> import subprocess 2024-03-31T15:12:24.096 DEBUG:teuthology.orchestra.run.smithi016:> 2024-03-31T15:12:24.096 DEBUG:teuthology.orchestra.run.smithi016:> def _find_admin_socket(client_name): 2024-03-31T15:12:24.096 DEBUG:teuthology.orchestra.run.smithi016:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-03-31T15:12:24.096 DEBUG:teuthology.orchestra.run.smithi016:> files = glob.glob(asok_path) 2024-03-31T15:12:24.096 DEBUG:teuthology.orchestra.run.smithi016:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-03-31T15:12:24.096 DEBUG:teuthology.orchestra.run.smithi016:> 2024-03-31T15:12:24.096 DEBUG:teuthology.orchestra.run.smithi016:> # Given a non-glob path, it better be there 2024-03-31T15:12:24.096 DEBUG:teuthology.orchestra.run.smithi016:> if "*" not in asok_path: 2024-03-31T15:12:24.097 DEBUG:teuthology.orchestra.run.smithi016:> assert(len(files) == 1) 2024-03-31T15:12:24.097 DEBUG:teuthology.orchestra.run.smithi016:> return files[0] 2024-03-31T15:12:24.097 DEBUG:teuthology.orchestra.run.smithi016:> 2024-03-31T15:12:24.097 DEBUG:teuthology.orchestra.run.smithi016:> for f in files: 2024-03-31T15:12:24.097 DEBUG:teuthology.orchestra.run.smithi016:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-31T15:12:24.097 DEBUG:teuthology.orchestra.run.smithi016:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-31T15:12:24.097 DEBUG:teuthology.orchestra.run.smithi016:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-31T15:12:24.097 DEBUG:teuthology.orchestra.run.smithi016:> contents = proc_f.read() 2024-03-31T15:12:24.097 DEBUG:teuthology.orchestra.run.smithi016:> if mountpoint in contents: 2024-03-31T15:12:24.097 DEBUG:teuthology.orchestra.run.smithi016:> return f 2024-03-31T15:12:24.097 DEBUG:teuthology.orchestra.run.smithi016:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-31T15:12:24.097 DEBUG:teuthology.orchestra.run.smithi016:> 2024-03-31T15:12:24.097 DEBUG:teuthology.orchestra.run.smithi016:> print(_find_admin_socket("client.0")) 2024-03-31T15:12:24.097 DEBUG:teuthology.orchestra.run.smithi016:> ' 2024-03-31T15:12:24.223 INFO:teuthology.orchestra.run.smithi016.stdout:/var/run/ceph/ceph-client.0.48278.asok 2024-03-31T15:12:24.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:24 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-31T15:12:24.234 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.48278.asok 2024-03-31T15:12:24.234 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:12:24.234 DEBUG:teuthology.orchestra.run.smithi016:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.48278.asok status 2024-03-31T15:12:24.408 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-03-31T15:12:24.409 INFO:teuthology.orchestra.run.smithi016.stdout: "metadata": { 2024-03-31T15:12:24.409 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph_sha1": "d2271579d5e735cfd6d436b8349c004f06527cb0", 2024-03-31T15:12:24.409 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph_version": "ceph version 19.0.0-2574-gd2271579 (d2271579d5e735cfd6d436b8349c004f06527cb0) squid (dev)", 2024-03-31T15:12:24.409 INFO:teuthology.orchestra.run.smithi016.stdout: "entity_id": "0", 2024-03-31T15:12:24.409 INFO:teuthology.orchestra.run.smithi016.stdout: "hostname": "smithi016", 2024-03-31T15:12:24.409 INFO:teuthology.orchestra.run.smithi016.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-03-31T15:12:24.409 INFO:teuthology.orchestra.run.smithi016.stdout: "pid": "48278", 2024-03-31T15:12:24.409 INFO:teuthology.orchestra.run.smithi016.stdout: "root": "/" 2024-03-31T15:12:24.409 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-31T15:12:24.409 INFO:teuthology.orchestra.run.smithi016.stdout: "dentry_count": 0, 2024-03-31T15:12:24.409 INFO:teuthology.orchestra.run.smithi016.stdout: "dentry_pinned_count": 0, 2024-03-31T15:12:24.410 INFO:teuthology.orchestra.run.smithi016.stdout: "id": 8628, 2024-03-31T15:12:24.410 INFO:teuthology.orchestra.run.smithi016.stdout: "inst": { 2024-03-31T15:12:24.410 INFO:teuthology.orchestra.run.smithi016.stdout: "name": { 2024-03-31T15:12:24.410 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "client", 2024-03-31T15:12:24.410 INFO:teuthology.orchestra.run.smithi016.stdout: "num": 8628 2024-03-31T15:12:24.410 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-31T15:12:24.410 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": { 2024-03-31T15:12:24.410 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-03-31T15:12:24.410 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "192.168.0.1:0", 2024-03-31T15:12:24.410 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 3052768950 2024-03-31T15:12:24.410 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-31T15:12:24.410 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-31T15:12:24.410 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": { 2024-03-31T15:12:24.410 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-03-31T15:12:24.411 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "192.168.0.1:0", 2024-03-31T15:12:24.411 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 3052768950 2024-03-31T15:12:24.411 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-31T15:12:24.411 INFO:teuthology.orchestra.run.smithi016.stdout: "inst_str": "client.8628 192.168.0.1:0/3052768950", 2024-03-31T15:12:24.411 INFO:teuthology.orchestra.run.smithi016.stdout: "addr_str": "192.168.0.1:0/3052768950", 2024-03-31T15:12:24.411 INFO:teuthology.orchestra.run.smithi016.stdout: "inode_count": 1, 2024-03-31T15:12:24.411 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_epoch": 83, 2024-03-31T15:12:24.411 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_epoch": 96, 2024-03-31T15:12:24.411 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_epoch_barrier": 0, 2024-03-31T15:12:24.411 INFO:teuthology.orchestra.run.smithi016.stdout: "blocklisted": false, 2024-03-31T15:12:24.411 INFO:teuthology.orchestra.run.smithi016.stdout: "fs_name": "cephfs" 2024-03-31T15:12:24.411 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-03-31T15:12:24.424 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:12:24.424 DEBUG:teuthology.orchestra.run.smithi016:> stat --file-system '--printf=%T 2024-03-31T15:12:24.424 DEBUG:teuthology.orchestra.run.smithi016:> ' -- /home/ubuntu/cephtest/mnt.0 2024-03-31T15:12:24.435 INFO:teuthology.orchestra.run.smithi016.stdout:fuseblk 2024-03-31T15:12:24.435 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-03-31T15:12:24.435 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:12:24.435 DEBUG:teuthology.orchestra.run.smithi016:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-31T15:12:24.498 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-03-31T15:12:24.498 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-31T15:12:24.550 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:12:24.551 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=file1_sixmegs bs=1M conv=fdatasync count=6 seek=0) 2024-03-31T15:12:24.686 INFO:teuthology.orchestra.run.smithi016.stderr:6+0 records in 2024-03-31T15:12:24.687 INFO:teuthology.orchestra.run.smithi016.stderr:6+0 records out 2024-03-31T15:12:24.687 INFO:teuthology.orchestra.run.smithi016.stderr:6291456 bytes (6.3 MB, 6.0 MiB) copied, 0.0775896 s, 81.1 MB/s 2024-03-31T15:12:24.688 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-03-31T15:12:24.688 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-31T15:12:24.698 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-31T15:12:24.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:24.887+0000 7fce57200640 1 Processor -- start 2024-03-31T15:12:24.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:24.891+0000 7fce57200640 1 -- start start 2024-03-31T15:12:24.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:24.891+0000 7fce57200640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fce500cef40 0x7fce500b5b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:24.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:24.891+0000 7fce57200640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fce5015b4d0 0x7fce500cf320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:24.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:24.891+0000 7fce57200640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fce500ce8a0 con 0x7fce500b61e0 2024-03-31T15:12:24.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:24.891+0000 7fce57200640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7fce500cf860 con 0x7fce5015b4d0 2024-03-31T15:12:24.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:24.891+0000 7fce57200640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fce500709a0 con 0x7fce500cef40 2024-03-31T15:12:24.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:24.891+0000 7fce4fe00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fce500cef40 0x7fce500b5b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:24.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:24.891+0000 7fce4fe00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fce500cef40 0x7fce500b5b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3301/0 says I am v2:172.21.15.16:57658/0 (socket says 172.21.15.16:57658) 2024-03-31T15:12:24.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:24.891+0000 7fce4fe00640 1 -- 172.21.15.16:0/354181528 learned_addr learned my addr 172.21.15.16:0/354181528 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:12:24.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:24.891+0000 7fce47e00640 1 --2- 172.21.15.16:0/354181528 >> v2:172.21.15.89:3300/0 conn(0x7fce5015b4d0 0x7fce500cf320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:24.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:24.891+0000 7fce4fe00640 1 -- 172.21.15.16:0/354181528 >> v2:172.21.15.89:3300/0 conn(0x7fce5015b4d0 msgr2=0x7fce500cf320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:12:24.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:24.891+0000 7fce4fe00640 1 --2- 172.21.15.16:0/354181528 >> v2:172.21.15.89:3300/0 conn(0x7fce5015b4d0 0x7fce500cf320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:24.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:24.891+0000 7fce4fe00640 1 -- 172.21.15.16:0/354181528 >> v1:172.21.15.89:6789/0 conn(0x7fce500b61e0 legacy=0x7fce5015ee10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:24.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:24.891+0000 7fce4fe00640 1 -- 172.21.15.16:0/354181528 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce5006fbe0 con 0x7fce500cef40 2024-03-31T15:12:24.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:24.895+0000 7fce4fe00640 1 --2- 172.21.15.16:0/354181528 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fce500cef40 0x7fce500b5b50 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fce40009870 tx=0x7fce4001e740 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ef6aa01f44c62f53 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:12:24.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:24.895+0000 7fce4f400640 1 -- 172.21.15.16:0/354181528 <== mon.2 v2:172.21.15.89:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce400a8710 con 0x7fce500cef40 2024-03-31T15:12:24.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:24.895+0000 7fce4f400640 1 -- 172.21.15.16:0/354181528 <== mon.2 v2:172.21.15.89:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fce400a88b0 con 0x7fce500cef40 2024-03-31T15:12:24.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:24.895+0000 7fce4f400640 1 -- 172.21.15.16:0/354181528 <== mon.2 v2:172.21.15.89:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce400a8c80 con 0x7fce500cef40 2024-03-31T15:12:24.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:24.895+0000 7fce57200640 1 -- 172.21.15.16:0/354181528 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fce500cef40 msgr2=0x7fce500b5b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:24.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:24.895+0000 7fce57200640 1 --2- 172.21.15.16:0/354181528 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fce500cef40 0x7fce500b5b50 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fce40009870 tx=0x7fce4001e740 comp rx=0 tx=0).stop 2024-03-31T15:12:24.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:24.895+0000 7fce57200640 1 -- 172.21.15.16:0/354181528 shutdown_connections 2024-03-31T15:12:24.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:24.895+0000 7fce57200640 1 --2- 172.21.15.16:0/354181528 >> v2:172.21.15.89:3300/0 conn(0x7fce5015b4d0 0x7fce500cf320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:24.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:24.895+0000 7fce57200640 1 --2- 172.21.15.16:0/354181528 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fce500cef40 0x7fce500b5b50 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:24.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:24.895+0000 7fce57200640 1 -- 172.21.15.16:0/354181528 >> 172.21.15.16:0/354181528 conn(0x7fce500d1060 msgr2=0x7fce500d1470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:12:24.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:24.895+0000 7fce57200640 1 -- 172.21.15.16:0/354181528 shutdown_connections 2024-03-31T15:12:24.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:24.895+0000 7fce57200640 1 -- 172.21.15.16:0/354181528 wait complete. 2024-03-31T15:12:24.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:24.899+0000 7fce57200640 1 Processor -- start 2024-03-31T15:12:24.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:24.899+0000 7fce57200640 1 -- start start 2024-03-31T15:12:24.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:24.899+0000 7fce57200640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fce500b61e0 0x7fce50159a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:24.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:24.899+0000 7fce57200640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fce500cef40 0x7fce5014ff30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:24.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:24.899+0000 7fce57200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fce5015b4d0 0x7fce5014c150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:24.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:24.899+0000 7fce57200640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fce500ce640 con 0x7fce5015b4d0 2024-03-31T15:12:24.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:24.899+0000 7fce57200640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fce500ce4c0 con 0x7fce500b61e0 2024-03-31T15:12:24.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:24.899+0000 7fce57200640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fce500b52f0 con 0x7fce500cef40 2024-03-31T15:12:24.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:24.899+0000 7fce55200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fce5015b4d0 0x7fce5014c150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:24.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:24.899+0000 7fce55200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fce5015b4d0 0x7fce5014c150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40392/0 (socket says 172.21.15.16:40392) 2024-03-31T15:12:24.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:24.899+0000 7fce55200640 1 -- 172.21.15.16:0/3821492858 learned_addr learned my addr 172.21.15.16:0/3821492858 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:12:24.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:24.899+0000 7fce47e00640 1 --2- 172.21.15.16:0/3821492858 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fce500cef40 0x7fce5014ff30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:24.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:24.899+0000 7fce4fe00640 1 --2- 172.21.15.16:0/3821492858 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fce500b61e0 0x7fce50159a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:24.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:24.899+0000 7fce55200640 1 -- 172.21.15.16:0/3821492858 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fce500cef40 msgr2=0x7fce5014ff30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:24.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:24.899+0000 7fce55200640 1 --2- 172.21.15.16:0/3821492858 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fce500cef40 0x7fce5014ff30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:24.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:24.899+0000 7fce55200640 1 -- 172.21.15.16:0/3821492858 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fce500b61e0 msgr2=0x7fce50159a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:24.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:24.899+0000 7fce55200640 1 --2- 172.21.15.16:0/3821492858 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fce500b61e0 0x7fce50159a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:24.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:24.899+0000 7fce55200640 1 -- 172.21.15.16:0/3821492858 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce5015a0f0 con 0x7fce5015b4d0 2024-03-31T15:12:24.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:24.899+0000 7fce4fe00640 1 --2- 172.21.15.16:0/3821492858 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fce500b61e0 0x7fce50159a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:12:24.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:24.899+0000 7fce55200640 1 --2- 172.21.15.16:0/3821492858 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fce5015b4d0 0x7fce5014c150 secure :-1 s=READY pgs=1428 cs=0 l=1 rev1=1 crypto rx=0x7fce4800c730 tx=0x7fce4800cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:12:24.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:24.899+0000 7fce4cc00640 1 -- 172.21.15.16:0/3821492858 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce48003b80 con 0x7fce5015b4d0 2024-03-31T15:12:24.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:24.899+0000 7fce4cc00640 1 -- 172.21.15.16:0/3821492858 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fce48003d20 con 0x7fce5015b4d0 2024-03-31T15:12:24.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:24.899+0000 7fce57200640 1 -- 172.21.15.16:0/3821492858 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fce5015a3d0 con 0x7fce5015b4d0 2024-03-31T15:12:24.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:24.899+0000 7fce4cc00640 1 -- 172.21.15.16:0/3821492858 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce48005310 con 0x7fce5015b4d0 2024-03-31T15:12:24.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:24.899+0000 7fce57200640 1 -- 172.21.15.16:0/3821492858 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fce5014c690 con 0x7fce5015b4d0 2024-03-31T15:12:24.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:24.903+0000 7fce57200640 1 -- 172.21.15.16:0/3821492858 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fce50069950 con 0x7fce5015b4d0 2024-03-31T15:12:24.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:24.903+0000 7fce4cc00640 1 -- 172.21.15.16:0/3821492858 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fce480056c0 con 0x7fce5015b4d0 2024-03-31T15:12:24.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:24.903+0000 7fce4cc00640 1 --2- 172.21.15.16:0/3821492858 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fce2c0683d0 0x7fce2c06a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:24.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:24.903+0000 7fce4cc00640 1 -- 172.21.15.16:0/3821492858 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 5709+0+0 (secure 0 0 0) 0x7fce48086990 con 0x7fce5015b4d0 2024-03-31T15:12:24.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:24.903+0000 7fce4fe00640 1 --2- 172.21.15.16:0/3821492858 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fce2c0683d0 0x7fce2c06a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:24.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:24.907+0000 7fce4fe00640 1 --2- 172.21.15.16:0/3821492858 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fce2c0683d0 0x7fce2c06a890 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7fce400b1000 tx=0x7fce400a9000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:12:24.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:24.911+0000 7fce4cc00640 1 -- 172.21.15.16:0/3821492858 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fce48056850 con 0x7fce5015b4d0 2024-03-31T15:12:25.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:25.075+0000 7fce57200640 1 -- 172.21.15.16:0/3821492858 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fce50157930 con 0x7fce5015b4d0 2024-03-31T15:12:25.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:25.079+0000 7fce4cc00640 1 -- 172.21.15.16:0/3821492858 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v96) ==== 74+0+16536 (secure 0 0 0) 0x7fce4805ab50 con 0x7fce5015b4d0 2024-03-31T15:12:25.070 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:12:25.070 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":96,"fsid":"286eb5da-fb47-4550-b290-d9519c13b9a5","created":"2024-03-31T15:03:11.438952+0000","modified":"2024-03-31T15:12:17.885852+0000","last_up_change":"2024-03-31T15:03:22.885535+0000","last_in_change":"2024-03-31T15:03:13.941667+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-31T15:03:22.550787+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2024-03-31T15:12:14.671225+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"95","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2024-03-31T15:12:15.242196+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"96","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"92a66450-ef19-40d8-ac3d-d760b23b0bda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6801","nonce":932712244}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6803","nonce":932712244}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6807","nonce":932712244}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6805","nonce":932712244}]},"public_addr":"172.21.15.16:6801/932712244","cluster_addr":"172.21.15.16:6803/932712244","heartbeat_back_addr":"172.21.15.16:6807/932712244","heartbeat_front_addr":"172.21.15.16:6805/932712244","state":["exists","up"]},{"osd":1,"uuid":"da9d1f2b-1968-4959-b843-faa231988e92","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6810","nonce":1398737533}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6811","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6813","nonce":1398737533}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6821","nonce":1398737533}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6815","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6816","nonce":1398737533}]},"public_addr":"172.21.15.16:6810/1398737533","cluster_addr":"172.21.15.16:6813/1398737533","heartbeat_back_addr":"172.21.15.16:6821/1398737533","heartbeat_front_addr":"172.21.15.16:6816/1398737533","state":["exists","up"]},{"osd":2,"uuid":"01a73662-9a21-4b1f-bcbb-42b9636a4638","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6819","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6822","nonce":3018526646}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6825","nonce":3018526646}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6831","nonce":3018526646}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6827","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6829","nonce":3018526646}]},"public_addr":"172.21.15.16:6822/3018526646","cluster_addr":"172.21.15.16:6825/3018526646","heartbeat_back_addr":"172.21.15.16:6831/3018526646","heartbeat_front_addr":"172.21.15.16:6829/3018526646","state":["exists","up"]},{"osd":3,"uuid":"c7771fd3-e18e-4500-ac38-23cf10ebcd75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6809","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6812","nonce":3360352974}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6817","nonce":3360352974}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6828","nonce":3360352974}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6823","nonce":3360352974}]},"public_addr":"172.21.15.16:6812/3360352974","cluster_addr":"172.21.15.16:6817/3360352974","heartbeat_back_addr":"172.21.15.16:6828/3360352974","heartbeat_front_addr":"172.21.15.16:6823/3360352974","state":["exists","up"]},{"osd":4,"uuid":"88260730-1482-4fbd-a40f-cb1124e9df4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6801","nonce":1856844856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6803","nonce":1856844856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6807","nonce":1856844856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6805","nonce":1856844856}]},"public_addr":"172.21.15.89:6801/1856844856","cluster_addr":"172.21.15.89:6803/1856844856","heartbeat_back_addr":"172.21.15.89:6807/1856844856","heartbeat_front_addr":"172.21.15.89:6805/1856844856","state":["exists","up"]},{"osd":5,"uuid":"4cb2c00c-fae0-4dd8-9d27-ec4f0093ccf7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6814","nonce":50395527}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6817","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6820","nonce":50395527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6831","nonce":50395527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6823","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6826","nonce":50395527}]},"public_addr":"172.21.15.89:6814/50395527","cluster_addr":"172.21.15.89:6820/50395527","heartbeat_back_addr":"172.21.15.89:6831/50395527","heartbeat_front_addr":"172.21.15.89:6826/50395527","state":["exists","up"]},{"osd":6,"uuid":"81ce0c19-2554-4cae-acbe-66a05e9e9b9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6809","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6812","nonce":3470471752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6815","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6818","nonce":3470471752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6827","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6829","nonce":3470471752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6821","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6824","nonce":3470471752}]},"public_addr":"172.21.15.89:6812/3470471752","cluster_addr":"172.21.15.89:6818/3470471752","heartbeat_back_addr":"172.21.15.89:6829/3470471752","heartbeat_front_addr":"172.21.15.89:6824/3470471752","state":["exists","up"]},{"osd":7,"uuid":"85e4313b-d37d-442e-8042-153bf20a847d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6811","nonce":1796778630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6813","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6816","nonce":1796778630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6825","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6828","nonce":1796778630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6819","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6822","nonce":1796778630}]},"public_addr":"172.21.15.89:6811/1796778630","cluster_addr":"172.21.15.89:6816/1796778630","heartbeat_back_addr":"172.21.15.89:6828/1796778630","heartbeat_front_addr":"172.21.15.89:6822/1796778630","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.439088+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.529970+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.680512+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:21.044161+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.582111+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.536558+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.409522+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.590229+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-31T15:12:25.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:25.079+0000 7fce57200640 1 -- 172.21.15.16:0/3821492858 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fce2c0683d0 msgr2=0x7fce2c06a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:25.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:25.079+0000 7fce57200640 1 --2- 172.21.15.16:0/3821492858 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fce2c0683d0 0x7fce2c06a890 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7fce400b1000 tx=0x7fce400a9000 comp rx=0 tx=0).stop 2024-03-31T15:12:25.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:25.079+0000 7fce57200640 1 -- 172.21.15.16:0/3821492858 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fce5015b4d0 msgr2=0x7fce5014c150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:25.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:25.079+0000 7fce57200640 1 --2- 172.21.15.16:0/3821492858 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fce5015b4d0 0x7fce5014c150 secure :-1 s=READY pgs=1428 cs=0 l=1 rev1=1 crypto rx=0x7fce4800c730 tx=0x7fce4800cbf0 comp rx=0 tx=0).stop 2024-03-31T15:12:25.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:25.079+0000 7fce57200640 1 -- 172.21.15.16:0/3821492858 shutdown_connections 2024-03-31T15:12:25.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:25.079+0000 7fce57200640 1 --2- 172.21.15.16:0/3821492858 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fce2c0683d0 0x7fce2c06a890 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:25.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:25.079+0000 7fce57200640 1 --2- 172.21.15.16:0/3821492858 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fce5015b4d0 0x7fce5014c150 unknown :-1 s=CLOSED pgs=1428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:25.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:25.079+0000 7fce57200640 1 --2- 172.21.15.16:0/3821492858 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fce500cef40 0x7fce5014ff30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:25.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:25.079+0000 7fce57200640 1 --2- 172.21.15.16:0/3821492858 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fce500b61e0 0x7fce50159a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:25.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:25.079+0000 7fce57200640 1 -- 172.21.15.16:0/3821492858 >> 172.21.15.16:0/3821492858 conn(0x7fce500d1060 msgr2=0x7fce50072850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:12:25.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:25.079+0000 7fce57200640 1 -- 172.21.15.16:0/3821492858 shutdown_connections 2024-03-31T15:12:25.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:25.079+0000 7fce57200640 1 -- 172.21.15.16:0/3821492858 wait complete. 2024-03-31T15:12:25.093 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-31T15:03:22.550787+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 14, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-31T15:12:14.671225+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '95', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.5, 'score_stable': 2.5, 'optimal_score': 1, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 15, 'pool_name': 'cephfs_data', 'create_time': '2024-03-31T15:12:15.242196+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '96', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-31T15:12:25.094 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-31T15:12:25.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:25.243+0000 7fcc46800640 1 Processor -- start 2024-03-31T15:12:25.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:25.243+0000 7fcc46800640 1 -- start start 2024-03-31T15:12:25.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:25.243+0000 7fcc46800640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fcc40070030 0x7fcc40143c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:25.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:25.243+0000 7fcc46800640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcc40144980 con 0x7fcc40141590 2024-03-31T15:12:25.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:25.243+0000 7fcc46800640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcc40161fe0 con 0x7fcc401411b0 2024-03-31T15:12:25.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:25.243+0000 7fcc46800640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7fcc4015f910 con 0x7fcc40070030 2024-03-31T15:12:25.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:25.247+0000 7fcc3fe00640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fcc40141590 0x7fcc40144170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:54974/0 (socket says 172.21.15.16:54974) 2024-03-31T15:12:25.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:25.247+0000 7fcc3fe00640 1 -- 172.21.15.16:0/2359642724 learned_addr learned my addr 172.21.15.16:0/2359642724 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:12:25.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:25.247+0000 7fcc37e00640 1 --2- 172.21.15.16:0/2359642724 >> v2:172.21.15.89:3300/0 conn(0x7fcc40070030 0x7fcc40143c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:25.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:25.247+0000 7fcc3ea00640 1 -- 172.21.15.16:0/2359642724 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3767627706 0 0) 0x7fcc40144980 con 0x7fcc40141590 2024-03-31T15:12:25.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:25.247+0000 7fcc3ea00640 1 -- 172.21.15.16:0/2359642724 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcc24003530 con 0x7fcc40141590 2024-03-31T15:12:25.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:25.247+0000 7fcc3ea00640 1 -- 172.21.15.16:0/2359642724 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 93438291 0 0) 0x7fcc24003530 con 0x7fcc40141590 2024-03-31T15:12:25.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:25.247+0000 7fcc3ea00640 1 -- 172.21.15.16:0/2359642724 >> v2:172.21.15.89:3300/0 conn(0x7fcc40070030 msgr2=0x7fcc40143c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:12:25.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:25.247+0000 7fcc3ea00640 1 --2- 172.21.15.16:0/2359642724 >> v2:172.21.15.89:3300/0 conn(0x7fcc40070030 0x7fcc40143c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:25.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:25.247+0000 7fcc3ea00640 1 -- 172.21.15.16:0/2359642724 >> v1:172.21.15.89:6789/0 conn(0x7fcc401411b0 legacy=0x7fcc4006f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:25.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:25.247+0000 7fcc3ea00640 1 -- 172.21.15.16:0/2359642724 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc401622c0 con 0x7fcc40141590 2024-03-31T15:12:25.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:25.247+0000 7fcc3ea00640 1 -- 172.21.15.16:0/2359642724 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 214807623 0 0) 0x7fcc38001a00 con 0x7fcc40141590 2024-03-31T15:12:25.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:25.247+0000 7fcc3ea00640 1 -- 172.21.15.16:0/2359642724 >> v1:172.21.15.16:6789/0 conn(0x7fcc40141590 legacy=0x7fcc40144170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:25.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:25.247+0000 7fcc3ea00640 1 --2- 172.21.15.16:0/2359642724 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcc24003a10 0x7fcc24003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:25.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:25.247+0000 7fcc3ea00640 1 --2- 172.21.15.16:0/2359642724 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fcc24004810 0x7fcc24004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:25.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:25.247+0000 7fcc3ea00640 1 --2- 172.21.15.16:0/2359642724 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcc240051d0 0x7fcc2400b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:25.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:25.247+0000 7fcc3ea00640 1 -- 172.21.15.16:0/2359642724 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fcc2400c0a0 con 0x7fcc24003a10 2024-03-31T15:12:25.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:25.247+0000 7fcc3ea00640 1 -- 172.21.15.16:0/2359642724 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fcc2400c250 con 0x7fcc240051d0 2024-03-31T15:12:25.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:25.247+0000 7fcc3ea00640 1 -- 172.21.15.16:0/2359642724 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fcc2400c450 con 0x7fcc24004810 2024-03-31T15:12:25.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:25.247+0000 7fcc3f400640 1 --2- 172.21.15.16:0/2359642724 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcc24003a10 0x7fcc24003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:25.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:25.247+0000 7fcc3fe00640 1 --2- 172.21.15.16:0/2359642724 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcc240051d0 0x7fcc2400b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:25.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:25.247+0000 7fcc37e00640 1 --2- 172.21.15.16:0/2359642724 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fcc24004810 0x7fcc24004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:25.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:25.247+0000 7fcc3f400640 1 -- 172.21.15.16:0/2359642724 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fcc24004810 msgr2=0x7fcc24004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:12:25.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:25.247+0000 7fcc3f400640 1 --2- 172.21.15.16:0/2359642724 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fcc24004810 0x7fcc24004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:25.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:25.247+0000 7fcc3f400640 1 -- 172.21.15.16:0/2359642724 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcc240051d0 msgr2=0x7fcc2400b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:12:25.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:25.247+0000 7fcc3f400640 1 --2- 172.21.15.16:0/2359642724 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcc240051d0 0x7fcc2400b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:25.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:25.247+0000 7fcc3f400640 1 -- 172.21.15.16:0/2359642724 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc24003530 con 0x7fcc24003a10 2024-03-31T15:12:25.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:25.247+0000 7fcc3fe00640 1 -- 172.21.15.16:0/2359642724 reap_dead start 2024-03-31T15:12:25.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:25.251+0000 7fcc3f400640 1 --2- 172.21.15.16:0/2359642724 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcc24003a10 0x7fcc24003e30 secure :-1 s=READY pgs=1430 cs=0 l=1 rev1=1 crypto rx=0x7fcc30001780 tx=0x7fcc3001c400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d203dffcb0c05c66 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:12:25.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:25.251+0000 7fcc3ea00640 1 -- 172.21.15.16:0/2359642724 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc300013a0 con 0x7fcc24003a10 2024-03-31T15:12:25.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:25.251+0000 7fcc3ea00640 1 -- 172.21.15.16:0/2359642724 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcc300052f0 con 0x7fcc24003a10 2024-03-31T15:12:25.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:25.251+0000 7fcc3ea00640 1 -- 172.21.15.16:0/2359642724 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc30003840 con 0x7fcc24003a10 2024-03-31T15:12:25.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:25.251+0000 7fcc46800640 1 -- 172.21.15.16:0/2359642724 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcc24003a10 msgr2=0x7fcc24003e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:25.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:25.251+0000 7fcc46800640 1 --2- 172.21.15.16:0/2359642724 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcc24003a10 0x7fcc24003e30 secure :-1 s=READY pgs=1430 cs=0 l=1 rev1=1 crypto rx=0x7fcc30001780 tx=0x7fcc3001c400 comp rx=0 tx=0).stop 2024-03-31T15:12:25.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:25.251+0000 7fcc46800640 1 -- 172.21.15.16:0/2359642724 shutdown_connections 2024-03-31T15:12:25.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:25.251+0000 7fcc46800640 1 --2- 172.21.15.16:0/2359642724 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcc24003a10 0x7fcc24003e30 unknown :-1 s=CLOSED pgs=1430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:25.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:25.251+0000 7fcc46800640 1 -- 172.21.15.16:0/2359642724 >> 172.21.15.16:0/2359642724 conn(0x7fcc40067930 msgr2=0x7fcc400b3ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:12:25.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:25.251+0000 7fcc46800640 1 -- 172.21.15.16:0/2359642724 shutdown_connections 2024-03-31T15:12:25.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:25.251+0000 7fcc46800640 1 -- 172.21.15.16:0/2359642724 wait complete. 2024-03-31T15:12:25.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:25.251+0000 7fcc46800640 1 Processor -- start 2024-03-31T15:12:25.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:25.251+0000 7fcc46800640 1 -- start start 2024-03-31T15:12:25.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:25.255+0000 7fcc46800640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fcc24003a10 0x7fcc40156ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:25.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:25.255+0000 7fcc46800640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcc40161850 0x7fcc40159010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:25.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:25.255+0000 7fcc46800640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcc40159550 0x7fcc4015f8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:25.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:25.255+0000 7fcc46800640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fcc401445c0 con 0x7fcc40159550 2024-03-31T15:12:25.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:25.255+0000 7fcc46800640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fcc40144440 con 0x7fcc40161850 2024-03-31T15:12:25.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:25.255+0000 7fcc46800640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fcc40070e60 con 0x7fcc24003a10 2024-03-31T15:12:25.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:25.255+0000 7fcc3fe00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcc40159550 0x7fcc4015f8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:25.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:25.255+0000 7fcc3f400640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fcc24003a10 0x7fcc40156ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:25.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:25.255+0000 7fcc3fe00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcc40159550 0x7fcc4015f8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40422/0 (socket says 172.21.15.16:40422) 2024-03-31T15:12:25.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:25.255+0000 7fcc3fe00640 1 -- 172.21.15.16:0/332240101 learned_addr learned my addr 172.21.15.16:0/332240101 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:12:25.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:25.255+0000 7fcc37e00640 1 --2- 172.21.15.16:0/332240101 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcc40161850 0x7fcc40159010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:25.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:25.255+0000 7fcc3fe00640 1 -- 172.21.15.16:0/332240101 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fcc24003a10 msgr2=0x7fcc40156ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:25.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:25.255+0000 7fcc3fe00640 1 --2- 172.21.15.16:0/332240101 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fcc24003a10 0x7fcc40156ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:25.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:25.255+0000 7fcc3fe00640 1 -- 172.21.15.16:0/332240101 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcc40161850 msgr2=0x7fcc40159010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:25.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:25.255+0000 7fcc3fe00640 1 --2- 172.21.15.16:0/332240101 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcc40161850 0x7fcc40159010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:25.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:25.255+0000 7fcc3fe00640 1 -- 172.21.15.16:0/332240101 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc401573e0 con 0x7fcc40159550 2024-03-31T15:12:25.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:25.255+0000 7fcc3fe00640 1 --2- 172.21.15.16:0/332240101 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcc40159550 0x7fcc4015f8f0 secure :-1 s=READY pgs=1431 cs=0 l=1 rev1=1 crypto rx=0x7fcc40158fb0 tx=0x7fcc38007b10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:12:25.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:25.255+0000 7fcc37400640 1 -- 172.21.15.16:0/332240101 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc3800a9f0 con 0x7fcc40159550 2024-03-31T15:12:25.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:25.255+0000 7fcc37400640 1 -- 172.21.15.16:0/332240101 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcc38018070 con 0x7fcc40159550 2024-03-31T15:12:25.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:25.255+0000 7fcc46800640 1 -- 172.21.15.16:0/332240101 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcc4015eb30 con 0x7fcc40159550 2024-03-31T15:12:25.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:25.255+0000 7fcc37400640 1 -- 172.21.15.16:0/332240101 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc3800add0 con 0x7fcc40159550 2024-03-31T15:12:25.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:25.255+0000 7fcc46800640 1 -- 172.21.15.16:0/332240101 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcc4015f0f0 con 0x7fcc40159550 2024-03-31T15:12:25.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:25.255+0000 7fcc46800640 1 -- 172.21.15.16:0/332240101 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcc40069950 con 0x7fcc40159550 2024-03-31T15:12:25.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:25.255+0000 7fcc37400640 1 -- 172.21.15.16:0/332240101 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fcc3801d460 con 0x7fcc40159550 2024-03-31T15:12:25.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:25.259+0000 7fcc37400640 1 --2- 172.21.15.16:0/332240101 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fcc1c068380 0x7fcc1c06a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:25.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:25.259+0000 7fcc37400640 1 -- 172.21.15.16:0/332240101 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 5709+0+0 (secure 0 0 0) 0x7fcc38086bf0 con 0x7fcc40159550 2024-03-31T15:12:25.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:25.259+0000 7fcc3f400640 1 --2- 172.21.15.16:0/332240101 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fcc1c068380 0x7fcc1c06a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:25.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:25.259+0000 7fcc3f400640 1 --2- 172.21.15.16:0/332240101 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fcc1c068380 0x7fcc1c06a840 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7fcc30001750 tx=0x7fcc300011a0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:12:25.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:25.263+0000 7fcc37400640 1 -- 172.21.15.16:0/332240101 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fcc38057240 con 0x7fcc40159550 2024-03-31T15:12:25.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:25.435+0000 7fcc46800640 1 -- 172.21.15.16:0/332240101 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fcc400b4a20 con 0x7fcc40159550 2024-03-31T15:12:25.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:25.435+0000 7fcc37400640 1 -- 172.21.15.16:0/332240101 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v96) ==== 93+0+10 (secure 0 0 0) 0x7fcc38056b40 con 0x7fcc40159550 2024-03-31T15:12:25.429 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2024-03-31T15:12:25.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:25.439+0000 7fcc46800640 1 -- 172.21.15.16:0/332240101 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fcc1c068380 msgr2=0x7fcc1c06a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:25.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:25.439+0000 7fcc46800640 1 --2- 172.21.15.16:0/332240101 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fcc1c068380 0x7fcc1c06a840 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7fcc30001750 tx=0x7fcc300011a0 comp rx=0 tx=0).stop 2024-03-31T15:12:25.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:25.439+0000 7fcc46800640 1 -- 172.21.15.16:0/332240101 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcc40159550 msgr2=0x7fcc4015f8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:25.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:25.439+0000 7fcc46800640 1 --2- 172.21.15.16:0/332240101 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcc40159550 0x7fcc4015f8f0 secure :-1 s=READY pgs=1431 cs=0 l=1 rev1=1 crypto rx=0x7fcc40158fb0 tx=0x7fcc38007b10 comp rx=0 tx=0).stop 2024-03-31T15:12:25.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:25.439+0000 7fcc46800640 1 -- 172.21.15.16:0/332240101 shutdown_connections 2024-03-31T15:12:25.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:25.439+0000 7fcc46800640 1 --2- 172.21.15.16:0/332240101 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fcc1c068380 0x7fcc1c06a840 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:25.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:25.439+0000 7fcc46800640 1 --2- 172.21.15.16:0/332240101 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcc40159550 0x7fcc4015f8f0 unknown :-1 s=CLOSED pgs=1431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:25.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:25.439+0000 7fcc46800640 1 --2- 172.21.15.16:0/332240101 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcc40161850 0x7fcc40159010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:25.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:25.439+0000 7fcc46800640 1 --2- 172.21.15.16:0/332240101 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fcc24003a10 0x7fcc40156ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:25.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:25.439+0000 7fcc46800640 1 -- 172.21.15.16:0/332240101 >> 172.21.15.16:0/332240101 conn(0x7fcc40067930 msgr2=0x7fcc400b40b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:12:25.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:25.439+0000 7fcc46800640 1 -- 172.21.15.16:0/332240101 shutdown_connections 2024-03-31T15:12:25.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:25.439+0000 7fcc46800640 1 -- 172.21.15.16:0/332240101 wait complete. 2024-03-31T15:12:25.450 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-31T15:12:25.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:25.599+0000 7ffb6b600640 1 Processor -- start 2024-03-31T15:12:25.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:25.599+0000 7ffb6b600640 1 -- start start 2024-03-31T15:12:25.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:25.603+0000 7ffb6b600640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ffb641411b0 0x7ffb6406f920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:25.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:25.603+0000 7ffb6b600640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7ffb64141590 0x7ffb6406fe60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:25.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:25.603+0000 7ffb6b600640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffb64144980 con 0x7ffb64143c30 2024-03-31T15:12:25.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:25.603+0000 7ffb6b600640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7ffb6415f910 con 0x7ffb64141590 2024-03-31T15:12:25.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:25.603+0000 7ffb6b600640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7ffb64162910 con 0x7ffb641411b0 2024-03-31T15:12:25.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:25.603+0000 7ffb69600640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7ffb64143c30 0x7ffb641440a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:54984/0 (socket says 172.21.15.16:54984) 2024-03-31T15:12:25.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:25.603+0000 7ffb69600640 1 -- 172.21.15.16:0/3248555550 learned_addr learned my addr 172.21.15.16:0/3248555550 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:12:25.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:25.603+0000 7ffb68c00640 1 --2- 172.21.15.16:0/3248555550 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ffb641411b0 0x7ffb6406f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:25.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:25.603+0000 7ffb5be00640 1 --2- 172.21.15.16:0/3248555550 >> v2:172.21.15.89:3300/0 conn(0x7ffb64141590 0x7ffb6406fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:25.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:25.603+0000 7ffb63e00640 1 -- 172.21.15.16:0/3248555550 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2474675553 0 0) 0x7ffb64144980 con 0x7ffb64143c30 2024-03-31T15:12:25.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:25.603+0000 7ffb63e00640 1 -- 172.21.15.16:0/3248555550 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ffb48003530 con 0x7ffb64143c30 2024-03-31T15:12:25.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:25.603+0000 7ffb5be00640 1 -- 172.21.15.16:0/3248555550 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ffb641411b0 msgr2=0x7ffb6406f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:12:25.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:25.603+0000 7ffb5be00640 1 --2- 172.21.15.16:0/3248555550 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ffb641411b0 0x7ffb6406f920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:25.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:25.603+0000 7ffb5be00640 1 -- 172.21.15.16:0/3248555550 >> v1:172.21.15.16:6789/0 conn(0x7ffb64143c30 legacy=0x7ffb641440a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:25.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:25.603+0000 7ffb5be00640 1 -- 172.21.15.16:0/3248555550 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffb64161fe0 con 0x7ffb64141590 2024-03-31T15:12:25.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:25.603+0000 7ffb68c00640 1 --2- 172.21.15.16:0/3248555550 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ffb641411b0 0x7ffb6406f920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:12:25.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:25.607+0000 7ffb5be00640 1 --2- 172.21.15.16:0/3248555550 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffb64141590 0x7ffb6406fe60 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7ffb4c0097c0 tx=0x7ffb4c01c4a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f29fb8aeddb0ba98 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:12:25.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:25.607+0000 7ffb63e00640 1 -- 172.21.15.16:0/3248555550 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb4c0b2070 con 0x7ffb64141590 2024-03-31T15:12:25.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:25.607+0000 7ffb63e00640 1 -- 172.21.15.16:0/3248555550 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffb4c0ad640 con 0x7ffb64141590 2024-03-31T15:12:25.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:25.607+0000 7ffb63e00640 1 -- 172.21.15.16:0/3248555550 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb4c0ad9f0 con 0x7ffb64141590 2024-03-31T15:12:25.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:25.607+0000 7ffb6b600640 1 -- 172.21.15.16:0/3248555550 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffb64141590 msgr2=0x7ffb6406fe60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:25.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:25.607+0000 7ffb6b600640 1 --2- 172.21.15.16:0/3248555550 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffb64141590 0x7ffb6406fe60 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7ffb4c0097c0 tx=0x7ffb4c01c4a0 comp rx=0 tx=0).stop 2024-03-31T15:12:25.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:25.607+0000 7ffb6b600640 1 -- 172.21.15.16:0/3248555550 shutdown_connections 2024-03-31T15:12:25.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:25.607+0000 7ffb6b600640 1 --2- 172.21.15.16:0/3248555550 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffb64141590 0x7ffb6406fe60 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:25.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:25.607+0000 7ffb6b600640 1 --2- 172.21.15.16:0/3248555550 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ffb641411b0 0x7ffb6406f920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:25.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:25.607+0000 7ffb6b600640 1 -- 172.21.15.16:0/3248555550 >> 172.21.15.16:0/3248555550 conn(0x7ffb64067930 msgr2=0x7ffb640b3ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:12:25.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:25.607+0000 7ffb6b600640 1 -- 172.21.15.16:0/3248555550 shutdown_connections 2024-03-31T15:12:25.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:25.607+0000 7ffb6b600640 1 -- 172.21.15.16:0/3248555550 wait complete. 2024-03-31T15:12:25.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:25.607+0000 7ffb6b600640 1 Processor -- start 2024-03-31T15:12:25.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:25.607+0000 7ffb6b600640 1 -- start start 2024-03-31T15:12:25.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:25.611+0000 7ffb6b600640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffb641411b0 0x7ffb64140480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:25.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:25.611+0000 7ffb6b600640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffb64141590 0x7ffb6415c0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:25.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:25.611+0000 7ffb6b600640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ffb64143c30 0x7ffb6413ed50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:25.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:25.611+0000 7ffb6b600640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7ffb6415f290 con 0x7ffb64141590 2024-03-31T15:12:25.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:25.611+0000 7ffb6b600640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7ffb6415f110 con 0x7ffb641411b0 2024-03-31T15:12:25.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:25.611+0000 7ffb6b600640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7ffb6415f410 con 0x7ffb64143c30 2024-03-31T15:12:25.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:25.611+0000 7ffb5be00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffb64141590 0x7ffb6415c0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:25.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:25.611+0000 7ffb5be00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffb64141590 0x7ffb6415c0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40428/0 (socket says 172.21.15.16:40428) 2024-03-31T15:12:25.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:25.611+0000 7ffb5be00640 1 -- 172.21.15.16:0/1534494819 learned_addr learned my addr 172.21.15.16:0/1534494819 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:12:25.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:25.611+0000 7ffb68c00640 1 --2- 172.21.15.16:0/1534494819 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffb641411b0 0x7ffb64140480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:25.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:25.611+0000 7ffb69600640 1 --2- 172.21.15.16:0/1534494819 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ffb64143c30 0x7ffb6413ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:25.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:25.611+0000 7ffb5be00640 1 -- 172.21.15.16:0/1534494819 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ffb64143c30 msgr2=0x7ffb6413ed50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:25.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:25.611+0000 7ffb5be00640 1 --2- 172.21.15.16:0/1534494819 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ffb64143c30 0x7ffb6413ed50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:25.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:25.611+0000 7ffb5be00640 1 -- 172.21.15.16:0/1534494819 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffb641411b0 msgr2=0x7ffb64140480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:25.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:25.611+0000 7ffb5be00640 1 --2- 172.21.15.16:0/1534494819 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffb641411b0 0x7ffb64140480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:25.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:25.611+0000 7ffb5be00640 1 -- 172.21.15.16:0/1534494819 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffb64159500 con 0x7ffb64141590 2024-03-31T15:12:25.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:25.611+0000 7ffb68c00640 1 --2- 172.21.15.16:0/1534494819 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffb641411b0 0x7ffb64140480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:12:25.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:25.611+0000 7ffb69600640 1 --2- 172.21.15.16:0/1534494819 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ffb64143c30 0x7ffb6413ed50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:12:25.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:25.611+0000 7ffb5be00640 1 --2- 172.21.15.16:0/1534494819 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffb64141590 0x7ffb6415c0e0 secure :-1 s=READY pgs=1433 cs=0 l=1 rev1=1 crypto rx=0x7ffb4c000c10 tx=0x7ffb4c019240 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:12:25.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:25.611+0000 7ffb61600640 1 -- 172.21.15.16:0/1534494819 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb4c0bc070 con 0x7ffb64141590 2024-03-31T15:12:25.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:25.611+0000 7ffb61600640 1 -- 172.21.15.16:0/1534494819 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffb4c0ad9f0 con 0x7ffb64141590 2024-03-31T15:12:25.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:25.611+0000 7ffb61600640 1 -- 172.21.15.16:0/1534494819 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb4c0b2040 con 0x7ffb64141590 2024-03-31T15:12:25.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:25.611+0000 7ffb6b600640 1 -- 172.21.15.16:0/1534494819 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffb6415c930 con 0x7ffb64141590 2024-03-31T15:12:25.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:25.611+0000 7ffb6b600640 1 -- 172.21.15.16:0/1534494819 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffb64159a80 con 0x7ffb64141590 2024-03-31T15:12:25.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:25.611+0000 7ffb6b600640 1 -- 172.21.15.16:0/1534494819 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffb64159e60 con 0x7ffb64141590 2024-03-31T15:12:25.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:25.615+0000 7ffb61600640 1 -- 172.21.15.16:0/1534494819 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7ffb4c0bf070 con 0x7ffb64141590 2024-03-31T15:12:25.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:25.615+0000 7ffb61600640 1 --2- 172.21.15.16:0/1534494819 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7ffb40068380 0x7ffb4006a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:25.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:25.615+0000 7ffb61600640 1 -- 172.21.15.16:0/1534494819 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 5709+0+0 (secure 0 0 0) 0x7ffb4c11fee0 con 0x7ffb64141590 2024-03-31T15:12:25.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:25.615+0000 7ffb68c00640 1 --2- 172.21.15.16:0/1534494819 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7ffb40068380 0x7ffb4006a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:25.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:25.615+0000 7ffb68c00640 1 --2- 172.21.15.16:0/1534494819 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7ffb40068380 0x7ffb4006a840 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7ffb54009ae0 tx=0x7ffb54008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:12:25.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:25.623+0000 7ffb61600640 1 -- 172.21.15.16:0/1534494819 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7ffb64159e60 con 0x7ffb64141590 2024-03-31T15:12:25.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:25.791+0000 7ffb6b600640 1 -- 172.21.15.16:0/1534494819 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7ffb64069950 con 0x7ffb64141590 2024-03-31T15:12:25.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:25.791+0000 7ffb61600640 1 -- 172.21.15.16:0/1534494819 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v96) ==== 104+0+11 (secure 0 0 0) 0x7ffb4c016090 con 0x7ffb64141590 2024-03-31T15:12:25.783 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2024-03-31T15:12:25.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:25.791+0000 7ffb6b600640 1 -- 172.21.15.16:0/1534494819 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7ffb40068380 msgr2=0x7ffb4006a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:25.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:25.791+0000 7ffb6b600640 1 --2- 172.21.15.16:0/1534494819 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7ffb40068380 0x7ffb4006a840 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7ffb54009ae0 tx=0x7ffb54008040 comp rx=0 tx=0).stop 2024-03-31T15:12:25.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:25.791+0000 7ffb6b600640 1 -- 172.21.15.16:0/1534494819 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffb64141590 msgr2=0x7ffb6415c0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:25.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:25.791+0000 7ffb6b600640 1 --2- 172.21.15.16:0/1534494819 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffb64141590 0x7ffb6415c0e0 secure :-1 s=READY pgs=1433 cs=0 l=1 rev1=1 crypto rx=0x7ffb4c000c10 tx=0x7ffb4c019240 comp rx=0 tx=0).stop 2024-03-31T15:12:25.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:25.791+0000 7ffb6b600640 1 -- 172.21.15.16:0/1534494819 shutdown_connections 2024-03-31T15:12:25.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:25.791+0000 7ffb6b600640 1 --2- 172.21.15.16:0/1534494819 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7ffb40068380 0x7ffb4006a840 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:25.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:25.791+0000 7ffb6b600640 1 --2- 172.21.15.16:0/1534494819 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ffb64143c30 0x7ffb6413ed50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:25.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:25.791+0000 7ffb6b600640 1 --2- 172.21.15.16:0/1534494819 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffb64141590 0x7ffb6415c0e0 unknown :-1 s=CLOSED pgs=1433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:25.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:25.791+0000 7ffb6b600640 1 --2- 172.21.15.16:0/1534494819 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffb641411b0 0x7ffb64140480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:25.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:25.791+0000 7ffb6b600640 1 -- 172.21.15.16:0/1534494819 >> 172.21.15.16:0/1534494819 conn(0x7ffb64067930 msgr2=0x7ffb641432a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:12:25.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:25.791+0000 7ffb6b600640 1 -- 172.21.15.16:0/1534494819 shutdown_connections 2024-03-31T15:12:25.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:25.791+0000 7ffb6b600640 1 -- 172.21.15.16:0/1534494819 wait complete. 2024-03-31T15:12:25.813 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-31T15:12:25.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:25.963+0000 7fa797c00640 1 Processor -- start 2024-03-31T15:12:25.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:25.963+0000 7fa797c00640 1 -- start start 2024-03-31T15:12:25.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:25.963+0000 7fa797c00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fa79006f920 0x7fa79006fd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:25.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:25.963+0000 7fa797c00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fa7900d0a60 0x7fa7900cea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:25.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:25.963+0000 7fa797c00640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa790070240 con 0x7fa7900cef80 2024-03-31T15:12:25.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:25.963+0000 7fa797c00640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7fa79013f670 con 0x7fa7900d0a60 2024-03-31T15:12:25.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:25.963+0000 7fa797c00640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7fa79015ca70 con 0x7fa79006f920 2024-03-31T15:12:25.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:25.967+0000 7fa787e00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fa7900d0a60 0x7fa7900cea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:25.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:25.967+0000 7fa787e00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fa7900d0a60 0x7fa7900cea40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40438/0 (socket says 172.21.15.16:40438) 2024-03-31T15:12:25.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:25.967+0000 7fa787e00640 1 -- 172.21.15.16:0/757279460 learned_addr learned my addr 172.21.15.16:0/757279460 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:12:25.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:25.967+0000 7fa795c00640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fa7900cef80 0x7fa790070e60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:54996/0 (socket says 172.21.15.16:54996) 2024-03-31T15:12:25.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:25.967+0000 7fa795200640 1 --2- 172.21.15.16:0/757279460 >> v2:172.21.15.89:3300/0 conn(0x7fa79006f920 0x7fa79006fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:25.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:25.967+0000 7fa787e00640 1 -- 172.21.15.16:0/757279460 >> v2:172.21.15.89:3300/0 conn(0x7fa79006f920 msgr2=0x7fa79006fd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:12:25.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:25.967+0000 7fa787e00640 1 --2- 172.21.15.16:0/757279460 >> v2:172.21.15.89:3300/0 conn(0x7fa79006f920 0x7fa79006fd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:25.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:25.967+0000 7fa787e00640 1 -- 172.21.15.16:0/757279460 >> v1:172.21.15.16:6789/0 conn(0x7fa7900cef80 legacy=0x7fa790070e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:25.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:25.967+0000 7fa787e00640 1 -- 172.21.15.16:0/757279460 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa79015cd50 con 0x7fa7900d0a60 2024-03-31T15:12:25.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:25.967+0000 7fa787e00640 1 --2- 172.21.15.16:0/757279460 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa7900d0a60 0x7fa7900cea40 secure :-1 s=READY pgs=1435 cs=0 l=1 rev1=1 crypto rx=0x7fa7900708e0 tx=0x7fa77801c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=57a59b73146c28be server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:12:25.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:25.967+0000 7fa78fe00640 1 -- 172.21.15.16:0/757279460 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7780b0070 con 0x7fa7900d0a60 2024-03-31T15:12:25.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:25.967+0000 7fa78fe00640 1 -- 172.21.15.16:0/757279460 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa7780ab600 con 0x7fa7900d0a60 2024-03-31T15:12:25.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:25.967+0000 7fa78fe00640 1 -- 172.21.15.16:0/757279460 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7780ab9d0 con 0x7fa7900d0a60 2024-03-31T15:12:25.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:25.971+0000 7fa797c00640 1 -- 172.21.15.16:0/757279460 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa7900d0a60 msgr2=0x7fa7900cea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:25.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:25.971+0000 7fa797c00640 1 --2- 172.21.15.16:0/757279460 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa7900d0a60 0x7fa7900cea40 secure :-1 s=READY pgs=1435 cs=0 l=1 rev1=1 crypto rx=0x7fa7900708e0 tx=0x7fa77801c490 comp rx=0 tx=0).stop 2024-03-31T15:12:25.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:25.971+0000 7fa797c00640 1 -- 172.21.15.16:0/757279460 shutdown_connections 2024-03-31T15:12:25.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:25.971+0000 7fa797c00640 1 --2- 172.21.15.16:0/757279460 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa7900d0a60 0x7fa7900cea40 unknown :-1 s=CLOSED pgs=1435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:25.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:25.971+0000 7fa797c00640 1 --2- 172.21.15.16:0/757279460 >> v2:172.21.15.89:3300/0 conn(0x7fa79006f920 0x7fa79006fd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:25.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:25.971+0000 7fa797c00640 1 -- 172.21.15.16:0/757279460 >> 172.21.15.16:0/757279460 conn(0x7fa790067930 msgr2=0x7fa7900b3ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:12:25.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:25.971+0000 7fa797c00640 1 -- 172.21.15.16:0/757279460 shutdown_connections 2024-03-31T15:12:25.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:25.971+0000 7fa797c00640 1 -- 172.21.15.16:0/757279460 wait complete. 2024-03-31T15:12:25.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:25.971+0000 7fa797c00640 1 Processor -- start 2024-03-31T15:12:25.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:25.971+0000 7fa797c00640 1 -- start start 2024-03-31T15:12:25.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:25.975+0000 7fa797c00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fa79006f920 0x7fa790152270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:25.968 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:25.975+0000 7fa797c00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa7900cef80 0x7fa79013cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:25.968 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:25.975+0000 7fa797c00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa7900d0a60 0x7fa7900b6350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:25.968 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:25.975+0000 7fa797c00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fa7901690b0 con 0x7fa7900cef80 2024-03-31T15:12:25.968 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:25.975+0000 7fa797c00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fa7900cf390 con 0x7fa7900d0a60 2024-03-31T15:12:25.968 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:25.975+0000 7fa797c00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fa790169230 con 0x7fa79006f920 2024-03-31T15:12:25.968 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:25.975+0000 7fa787e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa7900cef80 0x7fa79013cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:25.968 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:25.975+0000 7fa787e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa7900cef80 0x7fa79013cc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40450/0 (socket says 172.21.15.16:40450) 2024-03-31T15:12:25.968 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:25.975+0000 7fa787e00640 1 -- 172.21.15.16:0/114508624 learned_addr learned my addr 172.21.15.16:0/114508624 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:12:25.968 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:25.975+0000 7fa795c00640 1 --2- 172.21.15.16:0/114508624 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa7900d0a60 0x7fa7900b6350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:25.968 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:25.975+0000 7fa795200640 1 --2- 172.21.15.16:0/114508624 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fa79006f920 0x7fa790152270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:25.968 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:25.975+0000 7fa787e00640 1 -- 172.21.15.16:0/114508624 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fa79006f920 msgr2=0x7fa790152270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:25.968 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:25.975+0000 7fa787e00640 1 --2- 172.21.15.16:0/114508624 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fa79006f920 0x7fa790152270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:25.968 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:25.975+0000 7fa787e00640 1 -- 172.21.15.16:0/114508624 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa7900d0a60 msgr2=0x7fa7900b6350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:25.969 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:25.975+0000 7fa787e00640 1 --2- 172.21.15.16:0/114508624 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa7900d0a60 0x7fa7900b6350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:25.969 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:25.975+0000 7fa787e00640 1 -- 172.21.15.16:0/114508624 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa79013d550 con 0x7fa7900cef80 2024-03-31T15:12:25.969 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:25.975+0000 7fa795c00640 1 --2- 172.21.15.16:0/114508624 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa7900d0a60 0x7fa7900b6350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:12:25.969 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:25.975+0000 7fa787e00640 1 --2- 172.21.15.16:0/114508624 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa7900cef80 0x7fa79013cc90 secure :-1 s=READY pgs=1436 cs=0 l=1 rev1=1 crypto rx=0x7fa7780190a0 tx=0x7fa778019230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:12:25.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:25.975+0000 7fa78d600640 1 -- 172.21.15.16:0/114508624 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7780abe60 con 0x7fa7900cef80 2024-03-31T15:12:25.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:25.975+0000 7fa78d600640 1 -- 172.21.15.16:0/114508624 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa7780b4620 con 0x7fa7900cef80 2024-03-31T15:12:25.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:25.975+0000 7fa797c00640 1 -- 172.21.15.16:0/114508624 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa79013d830 con 0x7fa7900cef80 2024-03-31T15:12:25.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:25.975+0000 7fa78d600640 1 -- 172.21.15.16:0/114508624 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7780b0040 con 0x7fa7900cef80 2024-03-31T15:12:25.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:25.975+0000 7fa797c00640 1 -- 172.21.15.16:0/114508624 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa79013ddf0 con 0x7fa7900cef80 2024-03-31T15:12:25.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:25.975+0000 7fa797c00640 1 -- 172.21.15.16:0/114508624 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa790069950 con 0x7fa7900cef80 2024-03-31T15:12:25.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:25.979+0000 7fa78d600640 1 -- 172.21.15.16:0/114508624 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fa7780ab4c0 con 0x7fa7900cef80 2024-03-31T15:12:25.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:25.979+0000 7fa78d600640 1 --2- 172.21.15.16:0/114508624 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fa76c0683b0 0x7fa76c06a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:25.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:25.979+0000 7fa78d600640 1 -- 172.21.15.16:0/114508624 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 5709+0+0 (secure 0 0 0) 0x7fa7781204c0 con 0x7fa7900cef80 2024-03-31T15:12:25.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:25.979+0000 7fa795200640 1 --2- 172.21.15.16:0/114508624 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fa76c0683b0 0x7fa76c06a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:25.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:25.979+0000 7fa795200640 1 --2- 172.21.15.16:0/114508624 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fa76c0683b0 0x7fa76c06a870 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7fa780009af0 tx=0x7fa780008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:12:25.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:25.987+0000 7fa78d600640 1 -- 172.21.15.16:0/114508624 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fa778016110 con 0x7fa7900cef80 2024-03-31T15:12:26.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:26.151+0000 7fa797c00640 1 -- 172.21.15.16:0/114508624 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fa7900b50e0 con 0x7fa7900cef80 2024-03-31T15:12:26.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:26.151+0000 7fa78d600640 1 -- 172.21.15.16:0/114508624 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v96) ==== 100+0+11 (secure 0 0 0) 0x7fa7780f09e0 con 0x7fa7900cef80 2024-03-31T15:12:26.145 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 64 2024-03-31T15:12:26.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:26.155+0000 7fa797c00640 1 -- 172.21.15.16:0/114508624 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fa76c0683b0 msgr2=0x7fa76c06a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:26.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:26.155+0000 7fa797c00640 1 --2- 172.21.15.16:0/114508624 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fa76c0683b0 0x7fa76c06a870 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7fa780009af0 tx=0x7fa780008040 comp rx=0 tx=0).stop 2024-03-31T15:12:26.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:26.155+0000 7fa797c00640 1 -- 172.21.15.16:0/114508624 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa7900cef80 msgr2=0x7fa79013cc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:26.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:26.155+0000 7fa797c00640 1 --2- 172.21.15.16:0/114508624 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa7900cef80 0x7fa79013cc90 secure :-1 s=READY pgs=1436 cs=0 l=1 rev1=1 crypto rx=0x7fa7780190a0 tx=0x7fa778019230 comp rx=0 tx=0).stop 2024-03-31T15:12:26.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:26.155+0000 7fa797c00640 1 -- 172.21.15.16:0/114508624 shutdown_connections 2024-03-31T15:12:26.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:26.155+0000 7fa797c00640 1 --2- 172.21.15.16:0/114508624 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fa76c0683b0 0x7fa76c06a870 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:26.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:26.155+0000 7fa797c00640 1 --2- 172.21.15.16:0/114508624 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa7900d0a60 0x7fa7900b6350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:26.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:26.155+0000 7fa797c00640 1 --2- 172.21.15.16:0/114508624 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa7900cef80 0x7fa79013cc90 unknown :-1 s=CLOSED pgs=1436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:26.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:26.155+0000 7fa797c00640 1 --2- 172.21.15.16:0/114508624 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fa79006f920 0x7fa790152270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:26.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:26.155+0000 7fa797c00640 1 -- 172.21.15.16:0/114508624 >> 172.21.15.16:0/114508624 conn(0x7fa790067930 msgr2=0x7fa79015aee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:12:26.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:26.155+0000 7fa797c00640 1 -- 172.21.15.16:0/114508624 shutdown_connections 2024-03-31T15:12:26.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:26.155+0000 7fa797c00640 1 -- 172.21.15.16:0/114508624 wait complete. 2024-03-31T15:12:26.166 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-31T15:12:26.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:26.315+0000 7fb85ea00640 1 Processor -- start 2024-03-31T15:12:26.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:26.315+0000 7fb85ea00640 1 -- start start 2024-03-31T15:12:26.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:26.315+0000 7fb85ea00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fb8580b23d0 0x7fb85815cd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:26.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:26.315+0000 7fb85ea00640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb858160e70 con 0x7fb8580728b0 2024-03-31T15:12:26.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:26.315+0000 7fb85ea00640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb8581602f0 con 0x7fb858145d10 2024-03-31T15:12:26.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:26.315+0000 7fb85ea00640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7fb8581605d0 con 0x7fb8580b23d0 2024-03-31T15:12:26.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:26.315+0000 7fb84fe00640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fb8580728b0 0x7fb8580b1cc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:55004/0 (socket says 172.21.15.16:55004) 2024-03-31T15:12:26.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:26.315+0000 7fb84fe00640 1 -- 172.21.15.16:0/1115055194 learned_addr learned my addr 172.21.15.16:0/1115055194 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:12:26.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:26.315+0000 7fb85ca00640 1 --2- 172.21.15.16:0/1115055194 >> v2:172.21.15.89:3300/0 conn(0x7fb8580b23d0 0x7fb85815cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:26.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:26.315+0000 7fb857400640 1 -- 172.21.15.16:0/1115055194 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3562323869 0 0) 0x7fb858160e70 con 0x7fb8580728b0 2024-03-31T15:12:26.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:26.315+0000 7fb857400640 1 -- 172.21.15.16:0/1115055194 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb83c003530 con 0x7fb8580728b0 2024-03-31T15:12:26.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:26.319+0000 7fb85ca00640 1 -- 172.21.15.16:0/1115055194 >> v1:172.21.15.89:6789/0 conn(0x7fb858145d10 legacy=0x7fb8580cf790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:26.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:26.319+0000 7fb85ca00640 1 -- 172.21.15.16:0/1115055194 >> v1:172.21.15.16:6789/0 conn(0x7fb8580728b0 legacy=0x7fb8580b1cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:26.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:26.319+0000 7fb85ca00640 1 -- 172.21.15.16:0/1115055194 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb8581608b0 con 0x7fb8580b23d0 2024-03-31T15:12:26.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:26.319+0000 7fb85ca00640 1 --2- 172.21.15.16:0/1115055194 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb8580b23d0 0x7fb85815cd50 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7fb850009d90 tx=0x7fb85001c970 comp rx=0 tx=0).ready entity=mon.1 client_cookie=76bb14048d53d8d5 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:12:26.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:26.319+0000 7fb857400640 1 -- 172.21.15.16:0/1115055194 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8500b2070 con 0x7fb8580b23d0 2024-03-31T15:12:26.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:26.319+0000 7fb857400640 1 -- 172.21.15.16:0/1115055194 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb8500a5830 con 0x7fb8580b23d0 2024-03-31T15:12:26.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:26.319+0000 7fb857400640 1 -- 172.21.15.16:0/1115055194 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8500a5c00 con 0x7fb8580b23d0 2024-03-31T15:12:26.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:26.319+0000 7fb85ea00640 1 -- 172.21.15.16:0/1115055194 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb8580b23d0 msgr2=0x7fb85815cd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:26.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:26.319+0000 7fb85ea00640 1 --2- 172.21.15.16:0/1115055194 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb8580b23d0 0x7fb85815cd50 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7fb850009d90 tx=0x7fb85001c970 comp rx=0 tx=0).stop 2024-03-31T15:12:26.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:26.319+0000 7fb85ea00640 1 -- 172.21.15.16:0/1115055194 shutdown_connections 2024-03-31T15:12:26.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:26.319+0000 7fb85ea00640 1 --2- 172.21.15.16:0/1115055194 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb8580b23d0 0x7fb85815cd50 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:26.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:26.319+0000 7fb85ea00640 1 -- 172.21.15.16:0/1115055194 >> 172.21.15.16:0/1115055194 conn(0x7fb85806b620 msgr2=0x7fb85806ba30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:12:26.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:26.319+0000 7fb85ea00640 1 -- 172.21.15.16:0/1115055194 shutdown_connections 2024-03-31T15:12:26.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:26.319+0000 7fb85ea00640 1 -- 172.21.15.16:0/1115055194 wait complete. 2024-03-31T15:12:26.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:26.323+0000 7fb85ea00640 1 Processor -- start 2024-03-31T15:12:26.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:26.323+0000 7fb85ea00640 1 -- start start 2024-03-31T15:12:26.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:26.323+0000 7fb85ea00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb8580728b0 0x7fb85815a410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:26.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:26.323+0000 7fb85ea00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb8580b23d0 0x7fb8580b4c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:26.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:26.323+0000 7fb85ea00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb858145d10 0x7fb85813c810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:26.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:26.323+0000 7fb85ea00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fb858072e40 con 0x7fb8580b23d0 2024-03-31T15:12:26.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:26.323+0000 7fb85ea00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fb858072cc0 con 0x7fb8580728b0 2024-03-31T15:12:26.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:26.323+0000 7fb85ea00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fb85816a100 con 0x7fb858145d10 2024-03-31T15:12:26.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:26.323+0000 7fb84fe00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb8580b23d0 0x7fb8580b4c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:26.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:26.323+0000 7fb84fe00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb8580b23d0 0x7fb8580b4c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40454/0 (socket says 172.21.15.16:40454) 2024-03-31T15:12:26.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:26.323+0000 7fb84fe00640 1 -- 172.21.15.16:0/1179270833 learned_addr learned my addr 172.21.15.16:0/1179270833 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:12:26.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:26.323+0000 7fb857e00640 1 --2- 172.21.15.16:0/1179270833 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb8580728b0 0x7fb85815a410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:26.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:26.323+0000 7fb85ca00640 1 --2- 172.21.15.16:0/1179270833 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb858145d10 0x7fb85813c810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:26.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:26.323+0000 7fb84fe00640 1 -- 172.21.15.16:0/1179270833 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb858145d10 msgr2=0x7fb85813c810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:26.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:26.323+0000 7fb84fe00640 1 --2- 172.21.15.16:0/1179270833 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb858145d10 0x7fb85813c810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:26.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:26.323+0000 7fb84fe00640 1 -- 172.21.15.16:0/1179270833 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb8580728b0 msgr2=0x7fb85815a410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:26.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:26.323+0000 7fb84fe00640 1 --2- 172.21.15.16:0/1179270833 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb8580728b0 0x7fb85815a410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:26.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:26.323+0000 7fb84fe00640 1 -- 172.21.15.16:0/1179270833 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb85815a950 con 0x7fb8580b23d0 2024-03-31T15:12:26.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:26.323+0000 7fb84fe00640 1 --2- 172.21.15.16:0/1179270833 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb8580b23d0 0x7fb8580b4c10 secure :-1 s=READY pgs=1438 cs=0 l=1 rev1=1 crypto rx=0x7fb84000c4d0 tx=0x7fb84000c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:12:26.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:26.323+0000 7fb854c00640 1 -- 172.21.15.16:0/1179270833 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb840017070 con 0x7fb8580b23d0 2024-03-31T15:12:26.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:26.323+0000 7fb854c00640 1 -- 172.21.15.16:0/1179270833 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb840005cf0 con 0x7fb8580b23d0 2024-03-31T15:12:26.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:26.323+0000 7fb85ea00640 1 -- 172.21.15.16:0/1179270833 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb8580b5150 con 0x7fb8580b23d0 2024-03-31T15:12:26.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:26.323+0000 7fb854c00640 1 -- 172.21.15.16:0/1179270833 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb840007270 con 0x7fb8580b23d0 2024-03-31T15:12:26.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:26.327+0000 7fb85ea00640 1 -- 172.21.15.16:0/1179270833 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb858143440 con 0x7fb8580b23d0 2024-03-31T15:12:26.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:26.327+0000 7fb85ea00640 1 -- 172.21.15.16:0/1179270833 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb8580b21a0 con 0x7fb8580b23d0 2024-03-31T15:12:26.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:26.327+0000 7fb854c00640 1 -- 172.21.15.16:0/1179270833 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fb84001a070 con 0x7fb8580b23d0 2024-03-31T15:12:26.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:26.327+0000 7fb854c00640 1 --2- 172.21.15.16:0/1179270833 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fb834068380 0x7fb83406a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:26.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:26.327+0000 7fb854c00640 1 -- 172.21.15.16:0/1179270833 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 5709+0+0 (secure 0 0 0) 0x7fb840085c10 con 0x7fb8580b23d0 2024-03-31T15:12:26.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:26.327+0000 7fb857e00640 1 --2- 172.21.15.16:0/1179270833 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fb834068380 0x7fb83406a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:26.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:26.331+0000 7fb857e00640 1 --2- 172.21.15.16:0/1179270833 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fb834068380 0x7fb83406a840 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7fb8580cd500 tx=0x7fb848009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:12:26.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:26.331+0000 7fb854c00640 1 -- 172.21.15.16:0/1179270833 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fb840055c10 con 0x7fb8580b23d0 2024-03-31T15:12:26.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:26.507+0000 7fb85ea00640 1 -- 172.21.15.16:0/1179270833 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fb858069950 con 0x7fb8580b23d0 2024-03-31T15:12:26.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:26.507+0000 7fb854c00640 1 -- 172.21.15.16:0/1179270833 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v96) ==== 80+0+0 (secure 0 0 0) 0x7fb840059f10 con 0x7fb8580b23d0 2024-03-31T15:12:26.500 INFO:teuthology.orchestra.run.smithi016.stderr:listed 0 entries 2024-03-31T15:12:26.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:26.511+0000 7fb85ea00640 1 -- 172.21.15.16:0/1179270833 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fb834068380 msgr2=0x7fb83406a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:26.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:26.511+0000 7fb85ea00640 1 --2- 172.21.15.16:0/1179270833 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fb834068380 0x7fb83406a840 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7fb8580cd500 tx=0x7fb848009040 comp rx=0 tx=0).stop 2024-03-31T15:12:26.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:26.511+0000 7fb85ea00640 1 -- 172.21.15.16:0/1179270833 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb8580b23d0 msgr2=0x7fb8580b4c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:26.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:26.511+0000 7fb85ea00640 1 --2- 172.21.15.16:0/1179270833 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb8580b23d0 0x7fb8580b4c10 secure :-1 s=READY pgs=1438 cs=0 l=1 rev1=1 crypto rx=0x7fb84000c4d0 tx=0x7fb84000c990 comp rx=0 tx=0).stop 2024-03-31T15:12:26.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:26.511+0000 7fb85ea00640 1 -- 172.21.15.16:0/1179270833 shutdown_connections 2024-03-31T15:12:26.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:26.511+0000 7fb85ea00640 1 --2- 172.21.15.16:0/1179270833 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fb834068380 0x7fb83406a840 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:26.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:26.511+0000 7fb85ea00640 1 --2- 172.21.15.16:0/1179270833 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb858145d10 0x7fb85813c810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:26.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:26.511+0000 7fb85ea00640 1 --2- 172.21.15.16:0/1179270833 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb8580b23d0 0x7fb8580b4c10 unknown :-1 s=CLOSED pgs=1438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:26.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:26.511+0000 7fb85ea00640 1 --2- 172.21.15.16:0/1179270833 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb8580728b0 0x7fb85815a410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:26.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:26.511+0000 7fb85ea00640 1 -- 172.21.15.16:0/1179270833 >> 172.21.15.16:0/1179270833 conn(0x7fb85806b620 msgr2=0x7fb8580cff50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:12:26.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:26.511+0000 7fb85ea00640 1 -- 172.21.15.16:0/1179270833 shutdown_connections 2024-03-31T15:12:26.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:26.511+0000 7fb85ea00640 1 -- 172.21.15.16:0/1179270833 wait complete. 2024-03-31T15:12:26.521 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-03-31T15:12:26.521 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-31T15:12:26.530 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-31T15:12:26.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:26.719+0000 7f2f94e00640 1 Processor -- start 2024-03-31T15:12:26.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:26.719+0000 7f2f94e00640 1 -- start start 2024-03-31T15:12:26.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:26.723+0000 7f2f94e00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f2f9006dbc0 0x7f2f9015f650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:26.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:26.723+0000 7f2f94e00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f2f90070030 0x7f2f90162180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:26.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:26.723+0000 7f2f94e00640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2f900cbe90 con 0x7f2f900cb440 2024-03-31T15:12:26.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:26.723+0000 7f2f94e00640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f2f9015fb90 con 0x7f2f90070030 2024-03-31T15:12:26.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:26.723+0000 7f2f94e00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f2f9015f1c0 con 0x7f2f9006dbc0 2024-03-31T15:12:26.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:26.723+0000 7f2f8e800640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f2f90070030 0x7f2f90162180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:26.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:26.723+0000 7f2f8e800640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f2f90070030 0x7f2f90162180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40458/0 (socket says 172.21.15.16:40458) 2024-03-31T15:12:26.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:26.723+0000 7f2f8e800640 1 -- 172.21.15.16:0/2170743485 learned_addr learned my addr 172.21.15.16:0/2170743485 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:12:26.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:26.723+0000 7f2f8de00640 1 --2- 172.21.15.16:0/2170743485 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f2f9006dbc0 0x7f2f9015f650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:26.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:26.723+0000 7f2f8e800640 1 -- 172.21.15.16:0/2170743485 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f2f9006dbc0 msgr2=0x7f2f9015f650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:12:26.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:26.723+0000 7f2f8e800640 1 --2- 172.21.15.16:0/2170743485 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f2f9006dbc0 0x7f2f9015f650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:26.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:26.723+0000 7f2f8e800640 1 -- 172.21.15.16:0/2170743485 >> v1:172.21.15.89:6789/0 conn(0x7f2f900cb440 legacy=0x7f2f9006f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:26.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:26.723+0000 7f2f8e800640 1 -- 172.21.15.16:0/2170743485 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f900cc170 con 0x7f2f90070030 2024-03-31T15:12:26.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:26.727+0000 7f2f8e800640 1 --2- 172.21.15.16:0/2170743485 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2f90070030 0x7f2f90162180 secure :-1 s=READY pgs=1439 cs=0 l=1 rev1=1 crypto rx=0x7f2f88009e50 tx=0x7f2f8801c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7584a9c4db3aa2bc server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:12:26.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:26.727+0000 7f2f8ca00640 1 -- 172.21.15.16:0/2170743485 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f880a5740 con 0x7f2f90070030 2024-03-31T15:12:26.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:26.727+0000 7f2f8ca00640 1 -- 172.21.15.16:0/2170743485 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2f880a58e0 con 0x7f2f90070030 2024-03-31T15:12:26.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:26.727+0000 7f2f8ca00640 1 -- 172.21.15.16:0/2170743485 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f880a5cb0 con 0x7f2f90070030 2024-03-31T15:12:26.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:26.727+0000 7f2f94e00640 1 -- 172.21.15.16:0/2170743485 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2f90070030 msgr2=0x7f2f90162180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:26.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:26.727+0000 7f2f94e00640 1 --2- 172.21.15.16:0/2170743485 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2f90070030 0x7f2f90162180 secure :-1 s=READY pgs=1439 cs=0 l=1 rev1=1 crypto rx=0x7f2f88009e50 tx=0x7f2f8801c970 comp rx=0 tx=0).stop 2024-03-31T15:12:26.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:26.727+0000 7f2f94e00640 1 -- 172.21.15.16:0/2170743485 shutdown_connections 2024-03-31T15:12:26.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:26.727+0000 7f2f94e00640 1 --2- 172.21.15.16:0/2170743485 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2f90070030 0x7f2f90162180 unknown :-1 s=CLOSED pgs=1439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:26.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:26.727+0000 7f2f94e00640 1 --2- 172.21.15.16:0/2170743485 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f2f9006dbc0 0x7f2f9015f650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:26.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:26.727+0000 7f2f94e00640 1 -- 172.21.15.16:0/2170743485 >> 172.21.15.16:0/2170743485 conn(0x7f2f90067330 msgr2=0x7f2f90067740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:12:26.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:26.727+0000 7f2f94e00640 1 -- 172.21.15.16:0/2170743485 shutdown_connections 2024-03-31T15:12:26.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:26.727+0000 7f2f94e00640 1 -- 172.21.15.16:0/2170743485 wait complete. 2024-03-31T15:12:26.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:26.727+0000 7f2f94e00640 1 Processor -- start 2024-03-31T15:12:26.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:26.727+0000 7f2f94e00640 1 -- start start 2024-03-31T15:12:26.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:26.727+0000 7f2f94e00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f2f9006dbc0 0x7f2f9014e550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:26.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:26.727+0000 7f2f94e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2f90070030 0x7f2f901574b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:26.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:26.727+0000 7f2f94e00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2f900cb440 0x7f2f90151e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:26.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:26.727+0000 7f2f94e00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f2f9006cad0 con 0x7f2f90070030 2024-03-31T15:12:26.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:26.727+0000 7f2f94e00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f2f9006c950 con 0x7f2f900cb440 2024-03-31T15:12:26.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:26.727+0000 7f2f94e00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f2f9016b520 con 0x7f2f9006dbc0 2024-03-31T15:12:26.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:26.731+0000 7f2f8d400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2f90070030 0x7f2f901574b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:26.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:26.731+0000 7f2f8de00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f2f9006dbc0 0x7f2f9014e550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:26.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:26.731+0000 7f2f8d400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2f90070030 0x7f2f901574b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40460/0 (socket says 172.21.15.16:40460) 2024-03-31T15:12:26.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:26.731+0000 7f2f8d400640 1 -- 172.21.15.16:0/2355697272 learned_addr learned my addr 172.21.15.16:0/2355697272 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:12:26.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:26.731+0000 7f2f8e800640 1 --2- 172.21.15.16:0/2355697272 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2f900cb440 0x7f2f90151e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:26.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:26.731+0000 7f2f8d400640 1 -- 172.21.15.16:0/2355697272 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f2f9006dbc0 msgr2=0x7f2f9014e550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:26.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:26.731+0000 7f2f8d400640 1 --2- 172.21.15.16:0/2355697272 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f2f9006dbc0 0x7f2f9014e550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:26.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:26.731+0000 7f2f8d400640 1 -- 172.21.15.16:0/2355697272 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2f900cb440 msgr2=0x7f2f90151e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:26.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:26.731+0000 7f2f8d400640 1 --2- 172.21.15.16:0/2355697272 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2f900cb440 0x7f2f90151e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:26.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:26.731+0000 7f2f8d400640 1 -- 172.21.15.16:0/2355697272 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f9014ea90 con 0x7f2f90070030 2024-03-31T15:12:26.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:26.731+0000 7f2f8d400640 1 --2- 172.21.15.16:0/2355697272 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2f90070030 0x7f2f901574b0 secure :-1 s=READY pgs=1440 cs=0 l=1 rev1=1 crypto rx=0x7f2f7c00c420 tx=0x7f2f7c00c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:12:26.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:26.731+0000 7f2f86000640 1 -- 172.21.15.16:0/2355697272 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f7c0052d0 con 0x7f2f90070030 2024-03-31T15:12:26.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:26.731+0000 7f2f86000640 1 -- 172.21.15.16:0/2355697272 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2f7c005470 con 0x7f2f90070030 2024-03-31T15:12:26.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:26.731+0000 7f2f86000640 1 -- 172.21.15.16:0/2355697272 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f7c007260 con 0x7f2f90070030 2024-03-31T15:12:26.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:26.731+0000 7f2f94e00640 1 -- 172.21.15.16:0/2355697272 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2f90157db0 con 0x7f2f90070030 2024-03-31T15:12:26.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:26.731+0000 7f2f94e00640 1 -- 172.21.15.16:0/2355697272 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2f90154210 con 0x7f2f90070030 2024-03-31T15:12:26.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:26.731+0000 7f2f94e00640 1 -- 172.21.15.16:0/2355697272 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2f90069950 con 0x7f2f90070030 2024-03-31T15:12:26.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:26.731+0000 7f2f86000640 1 -- 172.21.15.16:0/2355697272 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f2f7c007400 con 0x7f2f90070030 2024-03-31T15:12:26.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:26.735+0000 7f2f86000640 1 --2- 172.21.15.16:0/2355697272 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f2f70068380 0x7f2f7006a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:26.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:26.735+0000 7f2f86000640 1 -- 172.21.15.16:0/2355697272 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 5709+0+0 (secure 0 0 0) 0x7f2f7c085e20 con 0x7f2f90070030 2024-03-31T15:12:26.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:26.735+0000 7f2f8de00640 1 --2- 172.21.15.16:0/2355697272 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f2f70068380 0x7f2f7006a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:26.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:26.735+0000 7f2f8de00640 1 --2- 172.21.15.16:0/2355697272 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f2f70068380 0x7f2f7006a840 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f2f800079c0 tx=0x7f2f80008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:12:26.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:26.739+0000 7f2f86000640 1 -- 172.21.15.16:0/2355697272 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f2f9014ea90 con 0x7f2f90070030 2024-03-31T15:12:26.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:26.903+0000 7f2f94e00640 1 -- 172.21.15.16:0/2355697272 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f2f900cf380 con 0x7f2f90070030 2024-03-31T15:12:26.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:26.907+0000 7f2f86000640 1 -- 172.21.15.16:0/2355697272 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v96) ==== 74+0+16536 (secure 0 0 0) 0x7f2f7c055d80 con 0x7f2f90070030 2024-03-31T15:12:26.899 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:12:26.899 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":96,"fsid":"286eb5da-fb47-4550-b290-d9519c13b9a5","created":"2024-03-31T15:03:11.438952+0000","modified":"2024-03-31T15:12:17.885852+0000","last_up_change":"2024-03-31T15:03:22.885535+0000","last_in_change":"2024-03-31T15:03:13.941667+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-31T15:03:22.550787+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2024-03-31T15:12:14.671225+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"95","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2024-03-31T15:12:15.242196+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"96","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"92a66450-ef19-40d8-ac3d-d760b23b0bda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6801","nonce":932712244}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6803","nonce":932712244}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6807","nonce":932712244}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6805","nonce":932712244}]},"public_addr":"172.21.15.16:6801/932712244","cluster_addr":"172.21.15.16:6803/932712244","heartbeat_back_addr":"172.21.15.16:6807/932712244","heartbeat_front_addr":"172.21.15.16:6805/932712244","state":["exists","up"]},{"osd":1,"uuid":"da9d1f2b-1968-4959-b843-faa231988e92","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6810","nonce":1398737533}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6811","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6813","nonce":1398737533}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6821","nonce":1398737533}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6815","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6816","nonce":1398737533}]},"public_addr":"172.21.15.16:6810/1398737533","cluster_addr":"172.21.15.16:6813/1398737533","heartbeat_back_addr":"172.21.15.16:6821/1398737533","heartbeat_front_addr":"172.21.15.16:6816/1398737533","state":["exists","up"]},{"osd":2,"uuid":"01a73662-9a21-4b1f-bcbb-42b9636a4638","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6819","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6822","nonce":3018526646}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6825","nonce":3018526646}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6831","nonce":3018526646}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6827","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6829","nonce":3018526646}]},"public_addr":"172.21.15.16:6822/3018526646","cluster_addr":"172.21.15.16:6825/3018526646","heartbeat_back_addr":"172.21.15.16:6831/3018526646","heartbeat_front_addr":"172.21.15.16:6829/3018526646","state":["exists","up"]},{"osd":3,"uuid":"c7771fd3-e18e-4500-ac38-23cf10ebcd75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6809","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6812","nonce":3360352974}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6817","nonce":3360352974}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6828","nonce":3360352974}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6823","nonce":3360352974}]},"public_addr":"172.21.15.16:6812/3360352974","cluster_addr":"172.21.15.16:6817/3360352974","heartbeat_back_addr":"172.21.15.16:6828/3360352974","heartbeat_front_addr":"172.21.15.16:6823/3360352974","state":["exists","up"]},{"osd":4,"uuid":"88260730-1482-4fbd-a40f-cb1124e9df4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6801","nonce":1856844856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6803","nonce":1856844856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6807","nonce":1856844856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6805","nonce":1856844856}]},"public_addr":"172.21.15.89:6801/1856844856","cluster_addr":"172.21.15.89:6803/1856844856","heartbeat_back_addr":"172.21.15.89:6807/1856844856","heartbeat_front_addr":"172.21.15.89:6805/1856844856","state":["exists","up"]},{"osd":5,"uuid":"4cb2c00c-fae0-4dd8-9d27-ec4f0093ccf7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6814","nonce":50395527}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6817","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6820","nonce":50395527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6831","nonce":50395527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6823","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6826","nonce":50395527}]},"public_addr":"172.21.15.89:6814/50395527","cluster_addr":"172.21.15.89:6820/50395527","heartbeat_back_addr":"172.21.15.89:6831/50395527","heartbeat_front_addr":"172.21.15.89:6826/50395527","state":["exists","up"]},{"osd":6,"uuid":"81ce0c19-2554-4cae-acbe-66a05e9e9b9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6809","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6812","nonce":3470471752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6815","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6818","nonce":3470471752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6827","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6829","nonce":3470471752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6821","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6824","nonce":3470471752}]},"public_addr":"172.21.15.89:6812/3470471752","cluster_addr":"172.21.15.89:6818/3470471752","heartbeat_back_addr":"172.21.15.89:6829/3470471752","heartbeat_front_addr":"172.21.15.89:6824/3470471752","state":["exists","up"]},{"osd":7,"uuid":"85e4313b-d37d-442e-8042-153bf20a847d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6811","nonce":1796778630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6813","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6816","nonce":1796778630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6825","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6828","nonce":1796778630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6819","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6822","nonce":1796778630}]},"public_addr":"172.21.15.89:6811/1796778630","cluster_addr":"172.21.15.89:6816/1796778630","heartbeat_back_addr":"172.21.15.89:6828/1796778630","heartbeat_front_addr":"172.21.15.89:6822/1796778630","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.439088+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.529970+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.680512+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:21.044161+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.582111+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.536558+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.409522+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.590229+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-31T15:12:26.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:26.907+0000 7f2f94e00640 1 -- 172.21.15.16:0/2355697272 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f2f70068380 msgr2=0x7f2f7006a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:26.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:26.907+0000 7f2f94e00640 1 --2- 172.21.15.16:0/2355697272 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f2f70068380 0x7f2f7006a840 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f2f800079c0 tx=0x7f2f80008040 comp rx=0 tx=0).stop 2024-03-31T15:12:26.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:26.907+0000 7f2f94e00640 1 -- 172.21.15.16:0/2355697272 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2f90070030 msgr2=0x7f2f901574b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:26.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:26.907+0000 7f2f94e00640 1 --2- 172.21.15.16:0/2355697272 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2f90070030 0x7f2f901574b0 secure :-1 s=READY pgs=1440 cs=0 l=1 rev1=1 crypto rx=0x7f2f7c00c420 tx=0x7f2f7c00c8e0 comp rx=0 tx=0).stop 2024-03-31T15:12:26.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:26.907+0000 7f2f94e00640 1 -- 172.21.15.16:0/2355697272 shutdown_connections 2024-03-31T15:12:26.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:26.907+0000 7f2f94e00640 1 --2- 172.21.15.16:0/2355697272 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f2f70068380 0x7f2f7006a840 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:26.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:26.907+0000 7f2f94e00640 1 --2- 172.21.15.16:0/2355697272 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2f900cb440 0x7f2f90151e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:26.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:26.907+0000 7f2f94e00640 1 --2- 172.21.15.16:0/2355697272 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2f90070030 0x7f2f901574b0 unknown :-1 s=CLOSED pgs=1440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:26.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:26.907+0000 7f2f94e00640 1 --2- 172.21.15.16:0/2355697272 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f2f9006dbc0 0x7f2f9014e550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:26.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:26.907+0000 7f2f94e00640 1 -- 172.21.15.16:0/2355697272 >> 172.21.15.16:0/2355697272 conn(0x7f2f90067330 msgr2=0x7f2f90161520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:12:26.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:26.911+0000 7f2f94e00640 1 -- 172.21.15.16:0/2355697272 shutdown_connections 2024-03-31T15:12:26.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:26.911+0000 7f2f94e00640 1 -- 172.21.15.16:0/2355697272 wait complete. 2024-03-31T15:12:26.922 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-31T15:03:22.550787+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 14, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-31T15:12:14.671225+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '95', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.5, 'score_stable': 2.5, 'optimal_score': 1, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 15, 'pool_name': 'cephfs_data', 'create_time': '2024-03-31T15:12:15.242196+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '96', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-31T15:12:26.922 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-31T15:12:27.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:27.075+0000 7f47a5200640 1 Processor -- start 2024-03-31T15:12:27.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:27.075+0000 7f47a5200640 1 -- start start 2024-03-31T15:12:27.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:27.079+0000 7f47a5200640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f47a0145d50 0x7f47a0160860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:27.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:27.079+0000 7f47a5200640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f47a015c890 0x7f47a006fc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:27.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:27.079+0000 7f47a5200640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f47a0070a10 con 0x7f47a006f160 2024-03-31T15:12:27.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:27.079+0000 7f47a5200640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f47a00ce5c0 con 0x7f47a015c890 2024-03-31T15:12:27.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:27.079+0000 7f47a5200640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f47a00cd9b0 con 0x7f47a0145d50 2024-03-31T15:12:27.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:27.079+0000 7f4795400640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f47a015c890 0x7f47a006fc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:27.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:27.079+0000 7f479e800640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f47a006f160 0x7f47a006b620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:42714/0 (socket says 172.21.15.16:42714) 2024-03-31T15:12:27.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:27.079+0000 7f4795400640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f47a015c890 0x7f47a006fc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41074/0 (socket says 172.21.15.16:41074) 2024-03-31T15:12:27.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:27.079+0000 7f479e800640 1 -- 172.21.15.16:0/270852651 learned_addr learned my addr 172.21.15.16:0/270852651 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:12:27.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:27.079+0000 7f479de00640 1 --2- 172.21.15.16:0/270852651 >> v2:172.21.15.89:3300/0 conn(0x7f47a0145d50 0x7f47a0160860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:27.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:27.079+0000 7f4795400640 1 -- 172.21.15.16:0/270852651 >> v2:172.21.15.89:3300/0 conn(0x7f47a0145d50 msgr2=0x7f47a0160860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:12:27.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:27.079+0000 7f4795400640 1 --2- 172.21.15.16:0/270852651 >> v2:172.21.15.89:3300/0 conn(0x7f47a0145d50 0x7f47a0160860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:27.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:27.079+0000 7f4795400640 1 -- 172.21.15.16:0/270852651 >> v1:172.21.15.16:6789/0 conn(0x7f47a006f160 legacy=0x7f47a006b620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:27.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:27.079+0000 7f4795400640 1 -- 172.21.15.16:0/270852651 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47a00cdc90 con 0x7f47a015c890 2024-03-31T15:12:27.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:27.079+0000 7f479d400640 1 -- 172.21.15.16:0/270852651 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 777203565 0 0) 0x7f47a0070a10 con 0x7f47a006f160 2024-03-31T15:12:27.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:27.079+0000 7f4795400640 1 --2- 172.21.15.16:0/270852651 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f47a015c890 0x7f47a006fc40 secure :-1 s=READY pgs=1442 cs=0 l=1 rev1=1 crypto rx=0x7f47a0068410 tx=0x7f478801c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8002f7b0059cf7ba server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:12:27.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:27.083+0000 7f479d400640 1 -- 172.21.15.16:0/270852651 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47880b2070 con 0x7f47a015c890 2024-03-31T15:12:27.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:27.083+0000 7f479d400640 1 -- 172.21.15.16:0/270852651 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f47880a5760 con 0x7f47a015c890 2024-03-31T15:12:27.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:27.083+0000 7f479d400640 1 -- 172.21.15.16:0/270852651 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47880a5b30 con 0x7f47a015c890 2024-03-31T15:12:27.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:27.083+0000 7f47a5200640 1 -- 172.21.15.16:0/270852651 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f47a015c890 msgr2=0x7f47a006fc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:27.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:27.083+0000 7f47a5200640 1 --2- 172.21.15.16:0/270852651 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f47a015c890 0x7f47a006fc40 secure :-1 s=READY pgs=1442 cs=0 l=1 rev1=1 crypto rx=0x7f47a0068410 tx=0x7f478801c8a0 comp rx=0 tx=0).stop 2024-03-31T15:12:27.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:27.083+0000 7f47a5200640 1 -- 172.21.15.16:0/270852651 shutdown_connections 2024-03-31T15:12:27.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:27.083+0000 7f47a5200640 1 --2- 172.21.15.16:0/270852651 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f47a015c890 0x7f47a006fc40 unknown :-1 s=CLOSED pgs=1442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:27.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:27.083+0000 7f47a5200640 1 --2- 172.21.15.16:0/270852651 >> v2:172.21.15.89:3300/0 conn(0x7f47a0145d50 0x7f47a0160860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:27.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:27.083+0000 7f47a5200640 1 -- 172.21.15.16:0/270852651 >> 172.21.15.16:0/270852651 conn(0x7f47a006e670 msgr2=0x7f47a006ea50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:12:27.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:27.083+0000 7f47a5200640 1 -- 172.21.15.16:0/270852651 shutdown_connections 2024-03-31T15:12:27.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:27.083+0000 7f47a5200640 1 -- 172.21.15.16:0/270852651 wait complete. 2024-03-31T15:12:27.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:27.083+0000 7f47a5200640 1 Processor -- start 2024-03-31T15:12:27.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:27.083+0000 7f47a5200640 1 -- start start 2024-03-31T15:12:27.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:27.083+0000 7f47a5200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f47a006f160 0x7f47a013ec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:27.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:27.083+0000 7f47a5200640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f47a0145d50 0x7f47a01f67e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:27.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:27.083+0000 7f47a5200640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f47a015c890 0x7f47a0166290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:27.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:27.083+0000 7f47a5200640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f47a0169520 con 0x7f47a006f160 2024-03-31T15:12:27.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:27.083+0000 7f47a5200640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f47a01693a0 con 0x7f47a0145d50 2024-03-31T15:12:27.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:27.083+0000 7f47a5200640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f47a01696a0 con 0x7f47a015c890 2024-03-31T15:12:27.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:27.083+0000 7f479de00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f47a006f160 0x7f47a013ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:27.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:27.083+0000 7f479de00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f47a006f160 0x7f47a013ec00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41086/0 (socket says 172.21.15.16:41086) 2024-03-31T15:12:27.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:27.083+0000 7f479de00640 1 -- 172.21.15.16:0/2346344839 learned_addr learned my addr 172.21.15.16:0/2346344839 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:12:27.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:27.083+0000 7f4795400640 1 --2- 172.21.15.16:0/2346344839 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f47a0145d50 0x7f47a01f67e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:27.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:27.087+0000 7f479e800640 1 --2- 172.21.15.16:0/2346344839 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f47a015c890 0x7f47a0166290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:27.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:27.087+0000 7f479de00640 1 -- 172.21.15.16:0/2346344839 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f47a015c890 msgr2=0x7f47a0166290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:27.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:27.087+0000 7f479de00640 1 --2- 172.21.15.16:0/2346344839 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f47a015c890 0x7f47a0166290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:27.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:27.087+0000 7f479de00640 1 -- 172.21.15.16:0/2346344839 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f47a0145d50 msgr2=0x7f47a01f67e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:27.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:27.087+0000 7f479de00640 1 --2- 172.21.15.16:0/2346344839 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f47a0145d50 0x7f47a01f67e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:27.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:27.087+0000 7f479de00640 1 -- 172.21.15.16:0/2346344839 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47a013ffe0 con 0x7f47a006f160 2024-03-31T15:12:27.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:27.087+0000 7f479de00640 1 --2- 172.21.15.16:0/2346344839 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f47a006f160 0x7f47a013ec00 secure :-1 s=READY pgs=1443 cs=0 l=1 rev1=1 crypto rx=0x7f479000d830 tx=0x7f479000dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:12:27.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:27.087+0000 7f4796a00640 1 -- 172.21.15.16:0/2346344839 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f479000b6c0 con 0x7f47a006f160 2024-03-31T15:12:27.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:27.087+0000 7f4796a00640 1 -- 172.21.15.16:0/2346344839 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4790019070 con 0x7f47a006f160 2024-03-31T15:12:27.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:27.087+0000 7f4796a00640 1 -- 172.21.15.16:0/2346344839 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f479000ba60 con 0x7f47a006f160 2024-03-31T15:12:27.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:27.087+0000 7f47a5200640 1 -- 172.21.15.16:0/2346344839 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f47a01438b0 con 0x7f47a006f160 2024-03-31T15:12:27.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:27.087+0000 7f47a5200640 1 -- 172.21.15.16:0/2346344839 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f47a0143e70 con 0x7f47a006f160 2024-03-31T15:12:27.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:27.087+0000 7f47a5200640 1 -- 172.21.15.16:0/2346344839 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f47a0069950 con 0x7f47a006f160 2024-03-31T15:12:27.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:27.087+0000 7f4796a00640 1 -- 172.21.15.16:0/2346344839 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f479001e460 con 0x7f47a006f160 2024-03-31T15:12:27.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:27.087+0000 7f4796a00640 1 --2- 172.21.15.16:0/2346344839 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f477c068380 0x7f477c06a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:27.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:27.087+0000 7f4796a00640 1 -- 172.21.15.16:0/2346344839 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 5709+0+0 (secure 0 0 0) 0x7f4790087000 con 0x7f47a006f160 2024-03-31T15:12:27.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:27.091+0000 7f4795400640 1 --2- 172.21.15.16:0/2346344839 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f477c068380 0x7f477c06a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:27.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:27.091+0000 7f4795400640 1 --2- 172.21.15.16:0/2346344839 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f477c068380 0x7f477c06a840 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f47880af000 tx=0x7f47880a7070 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:12:27.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:27.091+0000 7f4796a00640 1 -- 172.21.15.16:0/2346344839 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f47900575b0 con 0x7f47a006f160 2024-03-31T15:12:27.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:27.255+0000 7f47a5200640 1 -- 172.21.15.16:0/2346344839 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f47a013f140 con 0x7f47a006f160 2024-03-31T15:12:27.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:27.259+0000 7f4796a00640 1 -- 172.21.15.16:0/2346344839 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v96) ==== 93+0+10 (secure 0 0 0) 0x7f4790056eb0 con 0x7f47a006f160 2024-03-31T15:12:27.250 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2024-03-31T15:12:27.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:27.259+0000 7f47a5200640 1 -- 172.21.15.16:0/2346344839 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f477c068380 msgr2=0x7f477c06a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:27.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:27.259+0000 7f47a5200640 1 --2- 172.21.15.16:0/2346344839 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f477c068380 0x7f477c06a840 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f47880af000 tx=0x7f47880a7070 comp rx=0 tx=0).stop 2024-03-31T15:12:27.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:27.259+0000 7f47a5200640 1 -- 172.21.15.16:0/2346344839 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f47a006f160 msgr2=0x7f47a013ec00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:27.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:27.259+0000 7f47a5200640 1 --2- 172.21.15.16:0/2346344839 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f47a006f160 0x7f47a013ec00 secure :-1 s=READY pgs=1443 cs=0 l=1 rev1=1 crypto rx=0x7f479000d830 tx=0x7f479000dcf0 comp rx=0 tx=0).stop 2024-03-31T15:12:27.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:27.259+0000 7f47a5200640 1 -- 172.21.15.16:0/2346344839 shutdown_connections 2024-03-31T15:12:27.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:27.259+0000 7f47a5200640 1 --2- 172.21.15.16:0/2346344839 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f477c068380 0x7f477c06a840 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:27.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:27.259+0000 7f47a5200640 1 --2- 172.21.15.16:0/2346344839 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f47a015c890 0x7f47a0166290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:27.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:27.259+0000 7f47a5200640 1 --2- 172.21.15.16:0/2346344839 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f47a0145d50 0x7f47a01f67e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:27.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:27.259+0000 7f47a5200640 1 --2- 172.21.15.16:0/2346344839 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f47a006f160 0x7f47a013ec00 unknown :-1 s=CLOSED pgs=1443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:27.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:27.259+0000 7f47a5200640 1 -- 172.21.15.16:0/2346344839 >> 172.21.15.16:0/2346344839 conn(0x7f47a006e670 msgr2=0x7f47a015ecc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:12:27.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:27.259+0000 7f47a5200640 1 -- 172.21.15.16:0/2346344839 shutdown_connections 2024-03-31T15:12:27.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:27.259+0000 7f47a5200640 1 -- 172.21.15.16:0/2346344839 wait complete. 2024-03-31T15:12:27.272 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-31T15:12:27.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:27.423+0000 7f4165a00640 1 Processor -- start 2024-03-31T15:12:27.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:27.423+0000 7f4165a00640 1 -- start start 2024-03-31T15:12:27.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:27.423+0000 7f4165a00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f416006f920 0x7f416006fd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:27.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:27.423+0000 7f4165a00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f41600d0a60 0x7f41600cea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:27.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:27.423+0000 7f4165a00640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4160070240 con 0x7f41600cef80 2024-03-31T15:12:27.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:27.423+0000 7f4165a00640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f416013f670 con 0x7f41600d0a60 2024-03-31T15:12:27.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:27.423+0000 7f4165a00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f416015ca70 con 0x7f416006f920 2024-03-31T15:12:27.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:27.423+0000 7f415e000640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f41600d0a60 0x7f41600cea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:27.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:27.423+0000 7f415ea00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f416006f920 0x7f416006fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:27.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:27.423+0000 7f415e000640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f41600d0a60 0x7f41600cea40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.16:44758/0 (socket says 172.21.15.16:44758) 2024-03-31T15:12:27.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:27.423+0000 7f415f400640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f41600cef80 0x7f4160070e60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.16:52350/0 (socket says 172.21.15.16:52350) 2024-03-31T15:12:27.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:27.423+0000 7f415e000640 1 -- 172.21.15.16:0/248541544 learned_addr learned my addr 172.21.15.16:0/248541544 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:12:27.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:27.423+0000 7f415ea00640 1 -- 172.21.15.16:0/248541544 >> v2:172.21.15.89:3300/0 conn(0x7f41600d0a60 msgr2=0x7f41600cea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:12:27.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:27.423+0000 7f415ea00640 1 --2- 172.21.15.16:0/248541544 >> v2:172.21.15.89:3300/0 conn(0x7f41600d0a60 0x7f41600cea40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:27.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:27.423+0000 7f415ea00640 1 -- 172.21.15.16:0/248541544 >> v1:172.21.15.89:6789/0 conn(0x7f41600cef80 legacy=0x7f4160070e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:27.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:27.423+0000 7f415ea00640 1 -- 172.21.15.16:0/248541544 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f416015cd50 con 0x7f416006f920 2024-03-31T15:12:27.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:27.423+0000 7f415e000640 1 --2- 172.21.15.16:0/248541544 >> v2:172.21.15.89:3300/0 conn(0x7f41600d0a60 0x7f41600cea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:12:27.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:27.427+0000 7f415ea00640 1 --2- 172.21.15.16:0/248541544 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f416006f920 0x7f416006fd00 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f41500097c0 tx=0x7f415001c4a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c0f0b3c0f192cd8a server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:12:27.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:27.427+0000 7f415d600640 1 -- 172.21.15.16:0/248541544 <== mon.2 v2:172.21.15.89:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41500b0070 con 0x7f416006f920 2024-03-31T15:12:27.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:27.427+0000 7f415d600640 1 -- 172.21.15.16:0/248541544 <== mon.2 v2:172.21.15.89:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f41500a4e50 con 0x7f416006f920 2024-03-31T15:12:27.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:27.427+0000 7f415d600640 1 -- 172.21.15.16:0/248541544 <== mon.2 v2:172.21.15.89:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41500ab820 con 0x7f416006f920 2024-03-31T15:12:27.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:27.427+0000 7f4165a00640 1 -- 172.21.15.16:0/248541544 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f416006f920 msgr2=0x7f416006fd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:27.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:27.427+0000 7f4165a00640 1 --2- 172.21.15.16:0/248541544 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f416006f920 0x7f416006fd00 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f41500097c0 tx=0x7f415001c4a0 comp rx=0 tx=0).stop 2024-03-31T15:12:27.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:27.427+0000 7f4165a00640 1 -- 172.21.15.16:0/248541544 shutdown_connections 2024-03-31T15:12:27.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:27.427+0000 7f4165a00640 1 --2- 172.21.15.16:0/248541544 >> v2:172.21.15.89:3300/0 conn(0x7f41600d0a60 0x7f41600cea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:27.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:27.427+0000 7f4165a00640 1 --2- 172.21.15.16:0/248541544 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f416006f920 0x7f416006fd00 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:27.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:27.427+0000 7f4165a00640 1 -- 172.21.15.16:0/248541544 >> 172.21.15.16:0/248541544 conn(0x7f4160067930 msgr2=0x7f41600b3ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:12:27.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:27.427+0000 7f4165a00640 1 -- 172.21.15.16:0/248541544 shutdown_connections 2024-03-31T15:12:27.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:27.427+0000 7f4165a00640 1 -- 172.21.15.16:0/248541544 wait complete. 2024-03-31T15:12:27.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:27.431+0000 7f4165a00640 1 Processor -- start 2024-03-31T15:12:27.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:27.431+0000 7f4165a00640 1 -- start start 2024-03-31T15:12:27.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:27.431+0000 7f4165a00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f416006f920 0x7f41601552a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:27.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:27.431+0000 7f4165a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f41600cef80 0x7f4160158ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:27.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:27.431+0000 7f4165a00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f41600d0a60 0x7f4160151df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:27.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:27.431+0000 7f4165a00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f41601690b0 con 0x7f41600cef80 2024-03-31T15:12:27.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:27.431+0000 7f4165a00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f41600cf390 con 0x7f41600d0a60 2024-03-31T15:12:27.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:27.431+0000 7f4165a00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f4160169230 con 0x7f416006f920 2024-03-31T15:12:27.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:27.431+0000 7f415e000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f41600cef80 0x7f4160158ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:27.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:27.431+0000 7f415e000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f41600cef80 0x7f4160158ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41090/0 (socket says 172.21.15.16:41090) 2024-03-31T15:12:27.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:27.431+0000 7f415e000640 1 -- 172.21.15.16:0/1358333087 learned_addr learned my addr 172.21.15.16:0/1358333087 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:12:27.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:27.431+0000 7f415ea00640 1 --2- 172.21.15.16:0/1358333087 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f416006f920 0x7f41601552a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:27.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:27.431+0000 7f415f400640 1 --2- 172.21.15.16:0/1358333087 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f41600d0a60 0x7f4160151df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:27.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:27.431+0000 7f415e000640 1 -- 172.21.15.16:0/1358333087 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f416006f920 msgr2=0x7f41601552a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:27.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:27.431+0000 7f415e000640 1 --2- 172.21.15.16:0/1358333087 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f416006f920 0x7f41601552a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:27.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:27.431+0000 7f415e000640 1 -- 172.21.15.16:0/1358333087 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f41600d0a60 msgr2=0x7f4160151df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:27.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:27.431+0000 7f415e000640 1 --2- 172.21.15.16:0/1358333087 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f41600d0a60 0x7f4160151df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:27.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:27.431+0000 7f415e000640 1 -- 172.21.15.16:0/1358333087 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4160152b30 con 0x7f41600cef80 2024-03-31T15:12:27.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:27.431+0000 7f415f400640 1 --2- 172.21.15.16:0/1358333087 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f41600d0a60 0x7f4160151df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:12:27.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:27.431+0000 7f415e000640 1 --2- 172.21.15.16:0/1358333087 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f41600cef80 0x7f4160158ca0 secure :-1 s=READY pgs=1444 cs=0 l=1 rev1=1 crypto rx=0x7f414c00e9d0 tx=0x7f414c00ee90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:12:27.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:27.431+0000 7f4156a00640 1 -- 172.21.15.16:0/1358333087 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f414c017070 con 0x7f41600cef80 2024-03-31T15:12:27.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:27.431+0000 7f4156a00640 1 -- 172.21.15.16:0/1358333087 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f414c00ca60 con 0x7f41600cef80 2024-03-31T15:12:27.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:27.431+0000 7f4156a00640 1 -- 172.21.15.16:0/1358333087 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f414c00ce10 con 0x7f41600cef80 2024-03-31T15:12:27.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:27.431+0000 7f4165a00640 1 -- 172.21.15.16:0/1358333087 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4160158000 con 0x7f41600cef80 2024-03-31T15:12:27.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:27.431+0000 7f4165a00640 1 -- 172.21.15.16:0/1358333087 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4160152df0 con 0x7f41600cef80 2024-03-31T15:12:27.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:27.431+0000 7f4165a00640 1 -- 172.21.15.16:0/1358333087 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4160069950 con 0x7f41600cef80 2024-03-31T15:12:27.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:27.435+0000 7f4156a00640 1 -- 172.21.15.16:0/1358333087 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f414c01a070 con 0x7f41600cef80 2024-03-31T15:12:27.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:27.435+0000 7f4156a00640 1 --2- 172.21.15.16:0/1358333087 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f4140068380 0x7f414006a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:27.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:27.435+0000 7f4156a00640 1 -- 172.21.15.16:0/1358333087 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 5709+0+0 (secure 0 0 0) 0x7f414c086920 con 0x7f41600cef80 2024-03-31T15:12:27.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:27.435+0000 7f415ea00640 1 --2- 172.21.15.16:0/1358333087 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f4140068380 0x7f414006a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:27.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:27.435+0000 7f415ea00640 1 --2- 172.21.15.16:0/1358333087 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f4140068380 0x7f414006a840 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f41500ad000 tx=0x7f41500a6000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:12:27.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:27.439+0000 7f4156a00640 1 -- 172.21.15.16:0/1358333087 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f414c0567f0 con 0x7f41600cef80 2024-03-31T15:12:27.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:27.603+0000 7f4165a00640 1 -- 172.21.15.16:0/1358333087 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f41600b4a20 con 0x7f41600cef80 2024-03-31T15:12:27.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:27.603+0000 7f4156a00640 1 -- 172.21.15.16:0/1358333087 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v96) ==== 104+0+11 (secure 0 0 0) 0x7f414c05aaf0 con 0x7f41600cef80 2024-03-31T15:12:27.596 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2024-03-31T15:12:27.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:27.603+0000 7f4165a00640 1 -- 172.21.15.16:0/1358333087 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f4140068380 msgr2=0x7f414006a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:27.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:27.603+0000 7f4165a00640 1 --2- 172.21.15.16:0/1358333087 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f4140068380 0x7f414006a840 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f41500ad000 tx=0x7f41500a6000 comp rx=0 tx=0).stop 2024-03-31T15:12:27.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:27.603+0000 7f4165a00640 1 -- 172.21.15.16:0/1358333087 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f41600cef80 msgr2=0x7f4160158ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:27.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:27.603+0000 7f4165a00640 1 --2- 172.21.15.16:0/1358333087 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f41600cef80 0x7f4160158ca0 secure :-1 s=READY pgs=1444 cs=0 l=1 rev1=1 crypto rx=0x7f414c00e9d0 tx=0x7f414c00ee90 comp rx=0 tx=0).stop 2024-03-31T15:12:27.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:27.603+0000 7f4165a00640 1 -- 172.21.15.16:0/1358333087 shutdown_connections 2024-03-31T15:12:27.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:27.603+0000 7f4165a00640 1 --2- 172.21.15.16:0/1358333087 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f4140068380 0x7f414006a840 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:27.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:27.603+0000 7f4165a00640 1 --2- 172.21.15.16:0/1358333087 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f41600d0a60 0x7f4160151df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:27.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:27.603+0000 7f4165a00640 1 --2- 172.21.15.16:0/1358333087 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f41600cef80 0x7f4160158ca0 unknown :-1 s=CLOSED pgs=1444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:27.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:27.603+0000 7f4165a00640 1 --2- 172.21.15.16:0/1358333087 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f416006f920 0x7f41601552a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:27.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:27.603+0000 7f4165a00640 1 -- 172.21.15.16:0/1358333087 >> 172.21.15.16:0/1358333087 conn(0x7f4160067930 msgr2=0x7f416015a2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:12:27.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:27.603+0000 7f4165a00640 1 -- 172.21.15.16:0/1358333087 shutdown_connections 2024-03-31T15:12:27.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:27.603+0000 7f4165a00640 1 -- 172.21.15.16:0/1358333087 wait complete. 2024-03-31T15:12:27.626 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-31T15:12:27.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:27.775+0000 7f87ed600640 1 Processor -- start 2024-03-31T15:12:27.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:27.775+0000 7f87ed600640 1 -- start start 2024-03-31T15:12:27.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:27.779+0000 7f87ed600640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f87e80cf150 0x7f87e81670a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:27.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:27.779+0000 7f87ed600640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f87e815ca70 con 0x7f87e80d0a60 2024-03-31T15:12:27.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:27.779+0000 7f87ed600640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f87e815cd50 con 0x7f87e806f920 2024-03-31T15:12:27.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:27.779+0000 7f87ed600640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f87e813f670 con 0x7f87e80cf150 2024-03-31T15:12:27.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:27.779+0000 7f87dfe00640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f87e80d0a60 0x7f87e80cea40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:42716/0 (socket says 172.21.15.16:42716) 2024-03-31T15:12:27.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:27.779+0000 7f87dfe00640 1 -- 172.21.15.16:0/3871614182 learned_addr learned my addr 172.21.15.16:0/3871614182 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:12:27.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:27.779+0000 7f87e7400640 1 --2- 172.21.15.16:0/3871614182 >> v2:172.21.15.16:3300/0 conn(0x7f87e80cf150 0x7f87e81670a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:27.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:27.779+0000 7f87e7400640 1 -- 172.21.15.16:0/3871614182 >> v1:172.21.15.89:6789/0 conn(0x7f87e806f920 legacy=0x7f87e806fd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:27.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:27.779+0000 7f87e6000640 1 -- 172.21.15.16:0/3871614182 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3088014925 0 0) 0x7f87e815ca70 con 0x7f87e80d0a60 2024-03-31T15:12:27.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:27.779+0000 7f87e7400640 1 -- 172.21.15.16:0/3871614182 >> v1:172.21.15.16:6789/0 conn(0x7f87e80d0a60 legacy=0x7f87e80cea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:27.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:27.779+0000 7f87e7400640 1 -- 172.21.15.16:0/3871614182 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87e815d030 con 0x7f87e80cf150 2024-03-31T15:12:27.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:27.783+0000 7f87e7400640 1 --2- 172.21.15.16:0/3871614182 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f87e80cf150 0x7f87e81670a0 secure :-1 s=READY pgs=1446 cs=0 l=1 rev1=1 crypto rx=0x7f87e0009d90 tx=0x7f87e001c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1efe384d8c4c91fe server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:12:27.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:27.783+0000 7f87e6000640 1 -- 172.21.15.16:0/3871614182 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87e00b2070 con 0x7f87e80cf150 2024-03-31T15:12:27.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:27.783+0000 7f87e6000640 1 -- 172.21.15.16:0/3871614182 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f87e00a5830 con 0x7f87e80cf150 2024-03-31T15:12:27.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:27.783+0000 7f87e6000640 1 -- 172.21.15.16:0/3871614182 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87e00a5c00 con 0x7f87e80cf150 2024-03-31T15:12:27.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:27.783+0000 7f87ed600640 1 -- 172.21.15.16:0/3871614182 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f87e80cf150 msgr2=0x7f87e81670a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:27.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:27.783+0000 7f87ed600640 1 --2- 172.21.15.16:0/3871614182 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f87e80cf150 0x7f87e81670a0 secure :-1 s=READY pgs=1446 cs=0 l=1 rev1=1 crypto rx=0x7f87e0009d90 tx=0x7f87e001c970 comp rx=0 tx=0).stop 2024-03-31T15:12:27.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:27.783+0000 7f87ed600640 1 -- 172.21.15.16:0/3871614182 shutdown_connections 2024-03-31T15:12:27.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:27.783+0000 7f87ed600640 1 --2- 172.21.15.16:0/3871614182 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f87e80cf150 0x7f87e81670a0 unknown :-1 s=CLOSED pgs=1446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:27.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:27.783+0000 7f87ed600640 1 -- 172.21.15.16:0/3871614182 >> 172.21.15.16:0/3871614182 conn(0x7f87e8067930 msgr2=0x7f87e80b3ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:12:27.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:27.783+0000 7f87ed600640 1 -- 172.21.15.16:0/3871614182 shutdown_connections 2024-03-31T15:12:27.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:27.783+0000 7f87ed600640 1 -- 172.21.15.16:0/3871614182 wait complete. 2024-03-31T15:12:27.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:27.787+0000 7f87ed600640 1 Processor -- start 2024-03-31T15:12:27.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:27.787+0000 7f87ed600640 1 -- start start 2024-03-31T15:12:27.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:27.787+0000 7f87ed600640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f87e806f920 0x7f87e8153e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:27.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:27.787+0000 7f87ed600640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f87e80cf150 0x7f87e814f1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:27.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:27.787+0000 7f87ed600640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f87e80d0a60 0x7f87e80b5b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:27.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:27.787+0000 7f87ed600640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f87e816a190 con 0x7f87e80cf150 2024-03-31T15:12:27.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:27.787+0000 7f87ed600640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f87e816a010 con 0x7f87e806f920 2024-03-31T15:12:27.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:27.787+0000 7f87ed600640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f87e816a310 con 0x7f87e80d0a60 2024-03-31T15:12:27.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:27.787+0000 7f87dfe00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f87e80cf150 0x7f87e814f1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:27.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:27.787+0000 7f87dfe00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f87e80cf150 0x7f87e814f1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41106/0 (socket says 172.21.15.16:41106) 2024-03-31T15:12:27.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:27.787+0000 7f87dfe00640 1 -- 172.21.15.16:0/1628257001 learned_addr learned my addr 172.21.15.16:0/1628257001 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:12:27.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:27.787+0000 7f87e6a00640 1 --2- 172.21.15.16:0/1628257001 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f87e806f920 0x7f87e8153e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:27.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:27.787+0000 7f87dfe00640 1 -- 172.21.15.16:0/1628257001 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f87e80d0a60 msgr2=0x7f87e80b5b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:27.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:27.787+0000 7f87e7400640 1 --2- 172.21.15.16:0/1628257001 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f87e80d0a60 0x7f87e80b5b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:27.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:27.787+0000 7f87dfe00640 1 --2- 172.21.15.16:0/1628257001 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f87e80d0a60 0x7f87e80b5b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:27.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:27.787+0000 7f87dfe00640 1 -- 172.21.15.16:0/1628257001 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f87e806f920 msgr2=0x7f87e8153e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:27.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:27.787+0000 7f87dfe00640 1 --2- 172.21.15.16:0/1628257001 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f87e806f920 0x7f87e8153e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:27.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:27.787+0000 7f87dfe00640 1 -- 172.21.15.16:0/1628257001 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87e8152b20 con 0x7f87e80cf150 2024-03-31T15:12:27.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:27.787+0000 7f87e6a00640 1 --2- 172.21.15.16:0/1628257001 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f87e806f920 0x7f87e8153e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:12:27.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:27.787+0000 7f87e7400640 1 --2- 172.21.15.16:0/1628257001 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f87e80d0a60 0x7f87e80b5b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:12:27.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:27.787+0000 7f87dfe00640 1 --2- 172.21.15.16:0/1628257001 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f87e80cf150 0x7f87e814f1c0 secure :-1 s=READY pgs=1447 cs=0 l=1 rev1=1 crypto rx=0x7f87d000c440 tx=0x7f87d000c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:12:27.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:27.787+0000 7f87dea00640 1 -- 172.21.15.16:0/1628257001 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87d0018070 con 0x7f87e80cf150 2024-03-31T15:12:27.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:27.787+0000 7f87dea00640 1 -- 172.21.15.16:0/1628257001 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f87d00074e0 con 0x7f87e80cf150 2024-03-31T15:12:27.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:27.787+0000 7f87ed600640 1 -- 172.21.15.16:0/1628257001 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f87e8152e00 con 0x7f87e80cf150 2024-03-31T15:12:27.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:27.787+0000 7f87dea00640 1 -- 172.21.15.16:0/1628257001 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87d0004a30 con 0x7f87e80cf150 2024-03-31T15:12:27.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:27.787+0000 7f87ed600640 1 -- 172.21.15.16:0/1628257001 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f87e81533c0 con 0x7f87e80cf150 2024-03-31T15:12:27.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:27.787+0000 7f87ed600640 1 -- 172.21.15.16:0/1628257001 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f87e8069950 con 0x7f87e80cf150 2024-03-31T15:12:27.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:27.791+0000 7f87dea00640 1 -- 172.21.15.16:0/1628257001 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f87d001b070 con 0x7f87e80cf150 2024-03-31T15:12:27.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:27.791+0000 7f87dea00640 1 --2- 172.21.15.16:0/1628257001 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f87c4068380 0x7f87c406a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:27.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:27.791+0000 7f87dea00640 1 -- 172.21.15.16:0/1628257001 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 5709+0+0 (secure 0 0 0) 0x7f87d0085c10 con 0x7f87e80cf150 2024-03-31T15:12:27.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:27.791+0000 7f87e6a00640 1 --2- 172.21.15.16:0/1628257001 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f87c4068380 0x7f87c406a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:27.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:27.791+0000 7f87e6a00640 1 --2- 172.21.15.16:0/1628257001 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f87c4068380 0x7f87c406a840 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f87d8000f70 tx=0x7f87d8009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:12:27.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:27.803+0000 7f87dea00640 1 -- 172.21.15.16:0/1628257001 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f87d00561c0 con 0x7f87e80cf150 2024-03-31T15:12:27.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:27.967+0000 7f87ed600640 1 -- 172.21.15.16:0/1628257001 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f87e80b4a20 con 0x7f87e80cf150 2024-03-31T15:12:27.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:27.967+0000 7f87dea00640 1 -- 172.21.15.16:0/1628257001 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v96) ==== 100+0+11 (secure 0 0 0) 0x7f87d0055ac0 con 0x7f87e80cf150 2024-03-31T15:12:27.959 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 64 2024-03-31T15:12:27.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:27.967+0000 7f87ed600640 1 -- 172.21.15.16:0/1628257001 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f87c4068380 msgr2=0x7f87c406a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:27.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:27.967+0000 7f87ed600640 1 --2- 172.21.15.16:0/1628257001 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f87c4068380 0x7f87c406a840 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f87d8000f70 tx=0x7f87d8009040 comp rx=0 tx=0).stop 2024-03-31T15:12:27.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:27.967+0000 7f87ed600640 1 -- 172.21.15.16:0/1628257001 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f87e80cf150 msgr2=0x7f87e814f1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:27.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:27.967+0000 7f87ed600640 1 --2- 172.21.15.16:0/1628257001 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f87e80cf150 0x7f87e814f1c0 secure :-1 s=READY pgs=1447 cs=0 l=1 rev1=1 crypto rx=0x7f87d000c440 tx=0x7f87d000c900 comp rx=0 tx=0).stop 2024-03-31T15:12:27.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:27.967+0000 7f87ed600640 1 -- 172.21.15.16:0/1628257001 shutdown_connections 2024-03-31T15:12:27.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:27.967+0000 7f87ed600640 1 --2- 172.21.15.16:0/1628257001 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f87c4068380 0x7f87c406a840 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:27.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:27.967+0000 7f87ed600640 1 --2- 172.21.15.16:0/1628257001 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f87e80d0a60 0x7f87e80b5b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:27.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:27.967+0000 7f87ed600640 1 --2- 172.21.15.16:0/1628257001 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f87e80cf150 0x7f87e814f1c0 unknown :-1 s=CLOSED pgs=1447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:27.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:27.967+0000 7f87ed600640 1 --2- 172.21.15.16:0/1628257001 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f87e806f920 0x7f87e8153e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:27.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:27.967+0000 7f87ed600640 1 -- 172.21.15.16:0/1628257001 >> 172.21.15.16:0/1628257001 conn(0x7f87e8067930 msgr2=0x7f87e8158f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:12:27.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:27.967+0000 7f87ed600640 1 -- 172.21.15.16:0/1628257001 shutdown_connections 2024-03-31T15:12:27.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:27.967+0000 7f87ed600640 1 -- 172.21.15.16:0/1628257001 wait complete. 2024-03-31T15:12:27.981 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-31T15:12:28.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:28.131+0000 7f370e600640 1 Processor -- start 2024-03-31T15:12:28.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:28.131+0000 7f370e600640 1 -- start start 2024-03-31T15:12:28.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:28.131+0000 7f370e600640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f370806bd30 0x7f37080b0480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:28.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:28.131+0000 7f370e600640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f370815c4f0 con 0x7f3708145ab0 2024-03-31T15:12:28.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:28.131+0000 7f370e600640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f37080ae2a0 con 0x7f37080ade10 2024-03-31T15:12:28.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:28.131+0000 7f370e600640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f370815c7c0 con 0x7f370806bd30 2024-03-31T15:12:28.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:28.131+0000 7f370d200640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f3708145ab0 0x7f370806b620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:42720/0 (socket says 172.21.15.16:42720) 2024-03-31T15:12:28.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:28.131+0000 7f370d200640 1 -- 172.21.15.16:0/617408317 learned_addr learned my addr 172.21.15.16:0/617408317 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:12:28.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:28.135+0000 7f370dc00640 1 --2- 172.21.15.16:0/617408317 >> v2:172.21.15.89:3300/0 conn(0x7f370806bd30 0x7f37080b0480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:28.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:28.135+0000 7f3707400640 1 -- 172.21.15.16:0/617408317 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3594439363 0 0) 0x7f370815c4f0 con 0x7f3708145ab0 2024-03-31T15:12:28.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:28.135+0000 7f3707400640 1 -- 172.21.15.16:0/617408317 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f36f0003530 con 0x7f3708145ab0 2024-03-31T15:12:28.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:28.135+0000 7f3707400640 1 -- 172.21.15.16:0/617408317 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4127051237 0 0) 0x7f36f0003530 con 0x7f3708145ab0 2024-03-31T15:12:28.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:28.135+0000 7f3707400640 1 -- 172.21.15.16:0/617408317 >> v2:172.21.15.89:3300/0 conn(0x7f370806bd30 msgr2=0x7f37080b0480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:12:28.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:28.135+0000 7f3707400640 1 --2- 172.21.15.16:0/617408317 >> v2:172.21.15.89:3300/0 conn(0x7f370806bd30 0x7f37080b0480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:28.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:28.135+0000 7f3707400640 1 -- 172.21.15.16:0/617408317 >> v1:172.21.15.89:6789/0 conn(0x7f37080ade10 legacy=0x7f370806e6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:28.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:28.135+0000 7f3707400640 1 -- 172.21.15.16:0/617408317 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f370815caa0 con 0x7f3708145ab0 2024-03-31T15:12:28.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:28.135+0000 7f3707400640 1 -- 172.21.15.16:0/617408317 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 214807623 0 0) 0x7f36fc001680 con 0x7f3708145ab0 2024-03-31T15:12:28.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:28.135+0000 7f3707400640 1 -- 172.21.15.16:0/617408317 >> v1:172.21.15.16:6789/0 conn(0x7f3708145ab0 legacy=0x7f370806b620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:28.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:28.135+0000 7f3707400640 1 --2- 172.21.15.16:0/617408317 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f36f00039c0 0x7f36f0003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:28.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:28.135+0000 7f3707400640 1 --2- 172.21.15.16:0/617408317 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f36f0004810 0x7f36f0004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:28.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:28.135+0000 7f3707400640 1 --2- 172.21.15.16:0/617408317 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f36f00051d0 0x7f36f000b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:28.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:28.135+0000 7f3707400640 1 -- 172.21.15.16:0/617408317 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f370815c7c0 con 0x7f36f00039c0 2024-03-31T15:12:28.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:28.135+0000 7f3707400640 1 -- 172.21.15.16:0/617408317 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f36f000c070 con 0x7f36f00051d0 2024-03-31T15:12:28.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:28.135+0000 7f3707400640 1 -- 172.21.15.16:0/617408317 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f36f000c270 con 0x7f36f0004810 2024-03-31T15:12:28.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:28.135+0000 7f370dc00640 1 --2- 172.21.15.16:0/617408317 >> v2:172.21.15.89:3300/0 conn(0x7f370806bd30 0x7f37080b0480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:12:28.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:28.135+0000 7f370d200640 1 --2- 172.21.15.16:0/617408317 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f36f00039c0 0x7f36f0003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:28.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:28.135+0000 7f3707e00640 1 --2- 172.21.15.16:0/617408317 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f36f0004810 0x7f36f0004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:28.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:28.135+0000 7f370dc00640 1 --2- 172.21.15.16:0/617408317 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f36f00051d0 0x7f36f000b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:28.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:28.135+0000 7f370d200640 1 -- 172.21.15.16:0/617408317 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f36f0004810 msgr2=0x7f36f0004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:12:28.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:28.135+0000 7f370d200640 1 --2- 172.21.15.16:0/617408317 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f36f0004810 0x7f36f0004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:28.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:28.135+0000 7f370d200640 1 -- 172.21.15.16:0/617408317 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f36f00051d0 msgr2=0x7f36f000b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:12:28.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:28.135+0000 7f370d200640 1 --2- 172.21.15.16:0/617408317 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f36f00051d0 0x7f36f000b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:28.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:28.135+0000 7f370d200640 1 -- 172.21.15.16:0/617408317 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f36f0003530 con 0x7f36f00039c0 2024-03-31T15:12:28.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:28.135+0000 7f370dc00640 1 -- 172.21.15.16:0/617408317 reap_dead start 2024-03-31T15:12:28.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:28.139+0000 7f370d200640 1 --2- 172.21.15.16:0/617408317 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f36f00039c0 0x7f36f0003de0 secure :-1 s=READY pgs=1449 cs=0 l=1 rev1=1 crypto rx=0x7f36fc00c1c0 tx=0x7f36fc01cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c07160d9a558a142 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:12:28.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:28.139+0000 7f3707400640 1 -- 172.21.15.16:0/617408317 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36fc007d00 con 0x7f36f00039c0 2024-03-31T15:12:28.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:28.139+0000 7f3707400640 1 -- 172.21.15.16:0/617408317 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f36fc0a9020 con 0x7f36f00039c0 2024-03-31T15:12:28.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:28.139+0000 7f3707400640 1 -- 172.21.15.16:0/617408317 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36fc005ae0 con 0x7f36f00039c0 2024-03-31T15:12:28.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:28.139+0000 7f370e600640 1 -- 172.21.15.16:0/617408317 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f36f00039c0 msgr2=0x7f36f0003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:28.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:28.139+0000 7f370e600640 1 --2- 172.21.15.16:0/617408317 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f36f00039c0 0x7f36f0003de0 secure :-1 s=READY pgs=1449 cs=0 l=1 rev1=1 crypto rx=0x7f36fc00c1c0 tx=0x7f36fc01cc20 comp rx=0 tx=0).stop 2024-03-31T15:12:28.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:28.139+0000 7f370e600640 1 -- 172.21.15.16:0/617408317 shutdown_connections 2024-03-31T15:12:28.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:28.139+0000 7f370e600640 1 --2- 172.21.15.16:0/617408317 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f36f00039c0 0x7f36f0003de0 unknown :-1 s=CLOSED pgs=1449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:28.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:28.139+0000 7f370e600640 1 -- 172.21.15.16:0/617408317 >> 172.21.15.16:0/617408317 conn(0x7f37080b2b30 msgr2=0x7f37080b2f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:12:28.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:28.139+0000 7f370e600640 1 -- 172.21.15.16:0/617408317 shutdown_connections 2024-03-31T15:12:28.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:28.139+0000 7f370e600640 1 -- 172.21.15.16:0/617408317 wait complete. 2024-03-31T15:12:28.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:28.139+0000 7f370e600640 1 Processor -- start 2024-03-31T15:12:28.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:28.143+0000 7f370e600640 1 -- start start 2024-03-31T15:12:28.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:28.143+0000 7f370e600640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f36f00039c0 0x7f370809bbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:28.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:28.143+0000 7f370e600640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f370813ca80 0x7f37081f4f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:28.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:28.143+0000 7f370e600640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f370813d970 0x7f370813ed80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:28.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:28.143+0000 7f370e600640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f370806b9b0 con 0x7f370813d970 2024-03-31T15:12:28.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:28.143+0000 7f370e600640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f370806b830 con 0x7f370813ca80 2024-03-31T15:12:28.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:28.143+0000 7f370e600640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f370806bb30 con 0x7f36f00039c0 2024-03-31T15:12:28.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:28.143+0000 7f370dc00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f370813d970 0x7f370813ed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:28.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:28.143+0000 7f370dc00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f370813d970 0x7f370813ed80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41118/0 (socket says 172.21.15.16:41118) 2024-03-31T15:12:28.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:28.143+0000 7f370dc00640 1 -- 172.21.15.16:0/3204540193 learned_addr learned my addr 172.21.15.16:0/3204540193 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:12:28.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:28.143+0000 7f3707e00640 1 --2- 172.21.15.16:0/3204540193 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f370813ca80 0x7f37081f4f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:28.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:28.143+0000 7f370d200640 1 --2- 172.21.15.16:0/3204540193 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f36f00039c0 0x7f370809bbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:28.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:28.143+0000 7f370dc00640 1 -- 172.21.15.16:0/3204540193 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f36f00039c0 msgr2=0x7f370809bbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:28.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:28.143+0000 7f370dc00640 1 --2- 172.21.15.16:0/3204540193 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f36f00039c0 0x7f370809bbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:28.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:28.143+0000 7f370dc00640 1 -- 172.21.15.16:0/3204540193 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f370813ca80 msgr2=0x7f37081f4f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:28.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:28.143+0000 7f370dc00640 1 --2- 172.21.15.16:0/3204540193 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f370813ca80 0x7f37081f4f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:28.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:28.143+0000 7f370dc00640 1 -- 172.21.15.16:0/3204540193 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37081f54b0 con 0x7f370813d970 2024-03-31T15:12:28.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:28.143+0000 7f370dc00640 1 --2- 172.21.15.16:0/3204540193 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f370813d970 0x7f370813ed80 secure :-1 s=READY pgs=1450 cs=0 l=1 rev1=1 crypto rx=0x7f37080709a0 tx=0x7f3700009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:12:28.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:28.143+0000 7f3704c00640 1 -- 172.21.15.16:0/3204540193 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3700011dd0 con 0x7f370813d970 2024-03-31T15:12:28.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:28.143+0000 7f3704c00640 1 -- 172.21.15.16:0/3204540193 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f370001b070 con 0x7f370813d970 2024-03-31T15:12:28.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:28.143+0000 7f3704c00640 1 -- 172.21.15.16:0/3204540193 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37000039a0 con 0x7f370813d970 2024-03-31T15:12:28.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:28.143+0000 7f370e600640 1 -- 172.21.15.16:0/3204540193 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f370813e270 con 0x7f370813d970 2024-03-31T15:12:28.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:28.143+0000 7f370e600640 1 -- 172.21.15.16:0/3204540193 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3708140340 con 0x7f370813d970 2024-03-31T15:12:28.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:28.147+0000 7f370e600640 1 -- 172.21.15.16:0/3204540193 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f370804f8c0 con 0x7f370813d970 2024-03-31T15:12:28.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:28.147+0000 7f3704c00640 1 -- 172.21.15.16:0/3204540193 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f3700003b40 con 0x7f370813d970 2024-03-31T15:12:28.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:28.147+0000 7f3704c00640 1 --2- 172.21.15.16:0/3204540193 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f36e80683b0 0x7f36e806a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:28.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:28.147+0000 7f3704c00640 1 -- 172.21.15.16:0/3204540193 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 5709+0+0 (secure 0 0 0) 0x7f3700086fe0 con 0x7f370813d970 2024-03-31T15:12:28.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:28.147+0000 7f370d200640 1 --2- 172.21.15.16:0/3204540193 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f36e80683b0 0x7f36e806a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:28.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:28.151+0000 7f370d200640 1 --2- 172.21.15.16:0/3204540193 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f36e80683b0 0x7f36e806a870 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f36fc0aa000 tx=0x7f37080ae470 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:12:28.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:28.151+0000 7f3704c00640 1 -- 172.21.15.16:0/3204540193 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f37081f54b0 con 0x7f370813d970 2024-03-31T15:12:28.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:28.331+0000 7f370e600640 1 -- 172.21.15.16:0/3204540193 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f37080b1420 con 0x7f370813d970 2024-03-31T15:12:28.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:28.331+0000 7f3704c00640 1 -- 172.21.15.16:0/3204540193 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v96) ==== 80+0+0 (secure 0 0 0) 0x7f3700056ed0 con 0x7f370813d970 2024-03-31T15:12:28.323 INFO:teuthology.orchestra.run.smithi016.stderr:listed 0 entries 2024-03-31T15:12:28.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:28.331+0000 7f370e600640 1 -- 172.21.15.16:0/3204540193 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f36e80683b0 msgr2=0x7f36e806a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:28.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:28.331+0000 7f370e600640 1 --2- 172.21.15.16:0/3204540193 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f36e80683b0 0x7f36e806a870 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f36fc0aa000 tx=0x7f37080ae470 comp rx=0 tx=0).stop 2024-03-31T15:12:28.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:28.331+0000 7f370e600640 1 -- 172.21.15.16:0/3204540193 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f370813d970 msgr2=0x7f370813ed80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:28.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:28.331+0000 7f370e600640 1 --2- 172.21.15.16:0/3204540193 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f370813d970 0x7f370813ed80 secure :-1 s=READY pgs=1450 cs=0 l=1 rev1=1 crypto rx=0x7f37080709a0 tx=0x7f3700009f90 comp rx=0 tx=0).stop 2024-03-31T15:12:28.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:28.331+0000 7f370e600640 1 -- 172.21.15.16:0/3204540193 shutdown_connections 2024-03-31T15:12:28.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:28.331+0000 7f370e600640 1 --2- 172.21.15.16:0/3204540193 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f36e80683b0 0x7f36e806a870 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:28.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:28.331+0000 7f370e600640 1 --2- 172.21.15.16:0/3204540193 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f370813d970 0x7f370813ed80 unknown :-1 s=CLOSED pgs=1450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:28.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:28.331+0000 7f370e600640 1 --2- 172.21.15.16:0/3204540193 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f370813ca80 0x7f37081f4f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:28.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:28.331+0000 7f370e600640 1 --2- 172.21.15.16:0/3204540193 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f36f00039c0 0x7f370809bbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:28.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:28.331+0000 7f370e600640 1 -- 172.21.15.16:0/3204540193 >> 172.21.15.16:0/3204540193 conn(0x7f37080b2b30 msgr2=0x7f370815dfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:12:28.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:28.331+0000 7f370e600640 1 -- 172.21.15.16:0/3204540193 shutdown_connections 2024-03-31T15:12:28.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:28.331+0000 7f370e600640 1 -- 172.21.15.16:0/3204540193 wait complete. 2024-03-31T15:12:28.354 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi016.front.sepia.ceph.com... 2024-03-31T15:12:28.354 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:12:28.354 DEBUG:teuthology.orchestra.run.smithi016:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-03-31T15:12:28.372 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi016.stderr:ceph-fuse[48278]: fuse finished with error 0 and tester_r 0 2024-03-31T15:12:28.377 INFO:teuthology.orchestra.run:waiting for 300 2024-03-31T15:12:28.470 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi016.front.sepia.ceph.com 2024-03-31T15:12:28.470 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:12:28.471 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-31T15:12:28.482 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-31T15:12:28.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:28.671+0000 7fed42200640 1 Processor -- start 2024-03-31T15:12:28.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:28.671+0000 7fed42200640 1 -- start start 2024-03-31T15:12:28.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:28.675+0000 7fed42200640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fed3c070030 0x7fed3c162180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:28.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:28.675+0000 7fed42200640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fed3c0cbe90 con 0x7fed3c15f650 2024-03-31T15:12:28.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:28.675+0000 7fed42200640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fed3c0cc170 con 0x7fed3c06dbc0 2024-03-31T15:12:28.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:28.675+0000 7fed42200640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7fed3c15f1c0 con 0x7fed3c070030 2024-03-31T15:12:28.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:28.675+0000 7fed32a00640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fed3c15f650 0x7fed3c06f920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:42734/0 (socket says 172.21.15.16:42734) 2024-03-31T15:12:28.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:28.675+0000 7fed32a00640 1 -- 172.21.15.16:0/2125092551 learned_addr learned my addr 172.21.15.16:0/2125092551 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:12:28.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:28.675+0000 7fed3be00640 1 --2- 172.21.15.16:0/2125092551 >> v2:172.21.15.89:3300/0 conn(0x7fed3c070030 0x7fed3c162180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:28.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:28.675+0000 7fed3aa00640 1 -- 172.21.15.16:0/2125092551 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2950564396 0 0) 0x7fed3c0cbe90 con 0x7fed3c15f650 2024-03-31T15:12:28.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:28.675+0000 7fed3aa00640 1 -- 172.21.15.16:0/2125092551 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fed20003530 con 0x7fed3c15f650 2024-03-31T15:12:28.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:28.675+0000 7fed3be00640 1 -- 172.21.15.16:0/2125092551 >> v1:172.21.15.89:6789/0 conn(0x7fed3c06dbc0 legacy=0x7fed3c0cb440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:28.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:28.675+0000 7fed3be00640 1 -- 172.21.15.16:0/2125092551 >> v1:172.21.15.16:6789/0 conn(0x7fed3c15f650 legacy=0x7fed3c06f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:28.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:28.675+0000 7fed3be00640 1 -- 172.21.15.16:0/2125092551 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed3c15ee50 con 0x7fed3c070030 2024-03-31T15:12:28.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:28.679+0000 7fed3be00640 1 --2- 172.21.15.16:0/2125092551 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fed3c070030 0x7fed3c162180 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7fed340097c0 tx=0x7fed3401c410 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ad0c1d70b893d2b6 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:12:28.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:28.679+0000 7fed3aa00640 1 -- 172.21.15.16:0/2125092551 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed340b1070 con 0x7fed3c070030 2024-03-31T15:12:28.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:28.679+0000 7fed3aa00640 1 -- 172.21.15.16:0/2125092551 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fed340ac630 con 0x7fed3c070030 2024-03-31T15:12:28.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:28.679+0000 7fed3aa00640 1 -- 172.21.15.16:0/2125092551 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed340a8050 con 0x7fed3c070030 2024-03-31T15:12:28.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:28.679+0000 7fed42200640 1 -- 172.21.15.16:0/2125092551 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fed3c070030 msgr2=0x7fed3c162180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:28.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:28.679+0000 7fed42200640 1 --2- 172.21.15.16:0/2125092551 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fed3c070030 0x7fed3c162180 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7fed340097c0 tx=0x7fed3401c410 comp rx=0 tx=0).stop 2024-03-31T15:12:28.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:28.679+0000 7fed42200640 1 -- 172.21.15.16:0/2125092551 shutdown_connections 2024-03-31T15:12:28.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:28.679+0000 7fed42200640 1 --2- 172.21.15.16:0/2125092551 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fed3c070030 0x7fed3c162180 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:28.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:28.679+0000 7fed42200640 1 -- 172.21.15.16:0/2125092551 >> 172.21.15.16:0/2125092551 conn(0x7fed3c067330 msgr2=0x7fed3c067740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:12:28.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:28.679+0000 7fed42200640 1 -- 172.21.15.16:0/2125092551 shutdown_connections 2024-03-31T15:12:28.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:28.679+0000 7fed42200640 1 -- 172.21.15.16:0/2125092551 wait complete. 2024-03-31T15:12:28.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:28.679+0000 7fed42200640 1 Processor -- start 2024-03-31T15:12:28.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:28.679+0000 7fed42200640 1 -- start start 2024-03-31T15:12:28.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:28.683+0000 7fed42200640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fed3c06dbc0 0x7fed3c0b2d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:28.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:28.683+0000 7fed42200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fed3c070030 0x7fed3c1f7310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:28.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:28.683+0000 7fed42200640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fed3c15f650 0x7fed3c0b23c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:28.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:28.683+0000 7fed42200640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fed3c06c950 con 0x7fed3c070030 2024-03-31T15:12:28.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:28.683+0000 7fed42200640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fed3c06dfa0 con 0x7fed3c06dbc0 2024-03-31T15:12:28.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:28.683+0000 7fed42200640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fed3c06cad0 con 0x7fed3c15f650 2024-03-31T15:12:28.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:28.683+0000 7fed32a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fed3c070030 0x7fed3c1f7310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:28.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:28.683+0000 7fed32a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fed3c070030 0x7fed3c1f7310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41126/0 (socket says 172.21.15.16:41126) 2024-03-31T15:12:28.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:28.683+0000 7fed32a00640 1 -- 172.21.15.16:0/3061891021 learned_addr learned my addr 172.21.15.16:0/3061891021 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:12:28.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:28.683+0000 7fed3b400640 1 --2- 172.21.15.16:0/3061891021 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fed3c06dbc0 0x7fed3c0b2d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:28.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:28.683+0000 7fed3be00640 1 --2- 172.21.15.16:0/3061891021 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fed3c15f650 0x7fed3c0b23c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:28.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:28.683+0000 7fed32a00640 1 -- 172.21.15.16:0/3061891021 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fed3c15f650 msgr2=0x7fed3c0b23c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:28.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:28.683+0000 7fed32a00640 1 --2- 172.21.15.16:0/3061891021 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fed3c15f650 0x7fed3c0b23c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:28.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:28.683+0000 7fed32a00640 1 -- 172.21.15.16:0/3061891021 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fed3c06dbc0 msgr2=0x7fed3c0b2d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:28.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:28.683+0000 7fed32a00640 1 --2- 172.21.15.16:0/3061891021 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fed3c06dbc0 0x7fed3c0b2d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:28.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:28.683+0000 7fed32a00640 1 -- 172.21.15.16:0/3061891021 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed3c0b5e30 con 0x7fed3c070030 2024-03-31T15:12:28.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:28.683+0000 7fed3b400640 1 --2- 172.21.15.16:0/3061891021 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fed3c06dbc0 0x7fed3c0b2d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:12:28.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:28.683+0000 7fed3be00640 1 --2- 172.21.15.16:0/3061891021 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fed3c15f650 0x7fed3c0b23c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:12:28.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:28.683+0000 7fed32a00640 1 --2- 172.21.15.16:0/3061891021 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fed3c070030 0x7fed3c1f7310 secure :-1 s=READY pgs=1452 cs=0 l=1 rev1=1 crypto rx=0x7fed2400c4d0 tx=0x7fed2400c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:12:28.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:28.683+0000 7fed33e00640 1 -- 172.21.15.16:0/3061891021 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed24017070 con 0x7fed3c070030 2024-03-31T15:12:28.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:28.683+0000 7fed33e00640 1 -- 172.21.15.16:0/3061891021 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fed24005cf0 con 0x7fed3c070030 2024-03-31T15:12:28.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:28.683+0000 7fed42200640 1 -- 172.21.15.16:0/3061891021 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fed3c0b63c0 con 0x7fed3c070030 2024-03-31T15:12:28.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:28.683+0000 7fed33e00640 1 -- 172.21.15.16:0/3061891021 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed24007270 con 0x7fed3c070030 2024-03-31T15:12:28.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:28.683+0000 7fed42200640 1 -- 172.21.15.16:0/3061891021 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fed3c0b6980 con 0x7fed3c070030 2024-03-31T15:12:28.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:28.683+0000 7fed42200640 1 -- 172.21.15.16:0/3061891021 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fed3c069950 con 0x7fed3c070030 2024-03-31T15:12:28.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:28.683+0000 7fed33e00640 1 -- 172.21.15.16:0/3061891021 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fed2401a070 con 0x7fed3c070030 2024-03-31T15:12:28.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:28.687+0000 7fed33e00640 1 --2- 172.21.15.16:0/3061891021 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fed18068380 0x7fed1806a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:28.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:28.687+0000 7fed33e00640 1 -- 172.21.15.16:0/3061891021 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 5709+0+0 (secure 0 0 0) 0x7fed24085cb0 con 0x7fed3c070030 2024-03-31T15:12:28.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:28.687+0000 7fed3b400640 1 --2- 172.21.15.16:0/3061891021 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fed18068380 0x7fed1806a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:28.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:28.687+0000 7fed3b400640 1 --2- 172.21.15.16:0/3061891021 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fed18068380 0x7fed1806a840 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7fed2c001d70 tx=0x7fed2c009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:12:28.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:28.691+0000 7fed33e00640 1 -- 172.21.15.16:0/3061891021 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fed24055c10 con 0x7fed3c070030 2024-03-31T15:12:28.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:28.891+0000 7fed42200640 1 -- 172.21.15.16:0/3061891021 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fed3c06fb30 con 0x7fed3c070030 2024-03-31T15:12:28.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:28.891+0000 7fed33e00640 1 -- 172.21.15.16:0/3061891021 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 83 v83) ==== 94+0+5180 (secure 0 0 0) 0x7fed24059f10 con 0x7fed3c070030 2024-03-31T15:12:28.885 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:12:28.885 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":83,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7139,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6836/3798928623","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":3798928623},{"type":"v1","addr":"172.21.15.89:6836","nonce":3798928623}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":78},{"gid":7142,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/3074283817","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6835","nonce":3074283817},{"type":"v1","addr":"172.21.15.89:6837","nonce":3074283817}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":78},{"gid":8071,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6834/2110189998","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":2110189998},{"type":"v1","addr":"172.21.15.16:6834","nonce":2110189998}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":78}],"filesystems":[{"mdsmap":{"epoch":83,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-31T15:12:16.294600+0000","modified":"2024-03-31T15:12:19.297339+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8074},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8074":{"gid":8074,"name":"a","rank":0,"incarnation":79,"state":"up:active","state_seq":3,"addr":"172.21.15.16:6835/490717480","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6833","nonce":490717480},{"type":"v1","addr":"172.21.15.16:6835","nonce":490717480}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8074,"qdb_cluster":[8074]},"id":7}]} 2024-03-31T15:12:28.885 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 83 2024-03-31T15:12:28.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:28.895+0000 7fed42200640 1 -- 172.21.15.16:0/3061891021 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fed18068380 msgr2=0x7fed1806a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:28.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:28.895+0000 7fed42200640 1 --2- 172.21.15.16:0/3061891021 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fed18068380 0x7fed1806a840 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7fed2c001d70 tx=0x7fed2c009040 comp rx=0 tx=0).stop 2024-03-31T15:12:28.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:28.895+0000 7fed42200640 1 -- 172.21.15.16:0/3061891021 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fed3c070030 msgr2=0x7fed3c1f7310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:28.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:28.895+0000 7fed42200640 1 --2- 172.21.15.16:0/3061891021 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fed3c070030 0x7fed3c1f7310 secure :-1 s=READY pgs=1452 cs=0 l=1 rev1=1 crypto rx=0x7fed2400c4d0 tx=0x7fed2400c990 comp rx=0 tx=0).stop 2024-03-31T15:12:28.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:28.895+0000 7fed42200640 1 -- 172.21.15.16:0/3061891021 shutdown_connections 2024-03-31T15:12:28.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:28.895+0000 7fed42200640 1 --2- 172.21.15.16:0/3061891021 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fed18068380 0x7fed1806a840 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:28.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:28.895+0000 7fed42200640 1 --2- 172.21.15.16:0/3061891021 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fed3c15f650 0x7fed3c0b23c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:28.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:28.895+0000 7fed42200640 1 --2- 172.21.15.16:0/3061891021 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fed3c070030 0x7fed3c1f7310 unknown :-1 s=CLOSED pgs=1452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:28.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:28.895+0000 7fed42200640 1 --2- 172.21.15.16:0/3061891021 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fed3c06dbc0 0x7fed3c0b2d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:28.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:28.895+0000 7fed42200640 1 -- 172.21.15.16:0/3061891021 >> 172.21.15.16:0/3061891021 conn(0x7fed3c067330 msgr2=0x7fed3c15c820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:12:28.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:28.895+0000 7fed42200640 1 -- 172.21.15.16:0/3061891021 shutdown_connections 2024-03-31T15:12:28.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:28.895+0000 7fed42200640 1 -- 172.21.15.16:0/3061891021 wait complete. 2024-03-31T15:12:28.908 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json flush journal 2024-03-31T15:12:29.077 INFO:teuthology.orchestra.run.smithi016.stdout:{"message":"","return_code":0} 2024-03-31T15:12:29.078 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2024-03-31T15:12:29.078 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-31T15:12:29.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:29.227+0000 7f4987000640 1 Processor -- start 2024-03-31T15:12:29.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:29.227+0000 7f4987000640 1 -- start start 2024-03-31T15:12:29.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:29.227+0000 7f4987000640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f498006f920 0x7f498006fd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:29.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:29.227+0000 7f4987000640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f49800d2010 0x7f49800d24a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:29.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:29.227+0000 7f4987000640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f49800d0920 con 0x7f4980145e40 2024-03-31T15:12:29.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:29.227+0000 7f4987000640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f49800702c0 con 0x7f49800d2010 2024-03-31T15:12:29.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:29.227+0000 7f4987000640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f49800d06d0 con 0x7f498006f920 2024-03-31T15:12:29.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:29.227+0000 7f4985000640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f49800d2010 0x7f49800d24a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:29.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:29.227+0000 7f4985000640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f49800d2010 0x7f49800d24a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41128/0 (socket says 172.21.15.16:41128) 2024-03-31T15:12:29.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:29.227+0000 7f4985000640 1 -- 172.21.15.16:0/978364699 learned_addr learned my addr 172.21.15.16:0/978364699 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:12:29.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:29.227+0000 7f4977400640 1 --2- 172.21.15.16:0/978364699 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f498006f920 0x7f498006fd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:29.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:29.227+0000 7f4985000640 1 -- 172.21.15.16:0/978364699 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f498006f920 msgr2=0x7f498006fd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:12:29.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:29.227+0000 7f4985000640 1 --2- 172.21.15.16:0/978364699 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f498006f920 0x7f498006fd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:29.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:29.227+0000 7f4985000640 1 -- 172.21.15.16:0/978364699 >> v1:172.21.15.89:6789/0 conn(0x7f4980145e40 legacy=0x7f49800cc740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:29.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:29.227+0000 7f4985000640 1 -- 172.21.15.16:0/978364699 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49800d0c00 con 0x7f49800d2010 2024-03-31T15:12:29.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:29.231+0000 7f4985000640 1 --2- 172.21.15.16:0/978364699 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f49800d2010 0x7f49800d24a0 secure :-1 s=READY pgs=1453 cs=0 l=1 rev1=1 crypto rx=0x7f4978009e50 tx=0x7f497801ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=24e83380fd7a8f72 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:12:29.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:29.231+0000 7f497f400640 1 -- 172.21.15.16:0/978364699 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49780a57f0 con 0x7f49800d2010 2024-03-31T15:12:29.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:29.231+0000 7f497f400640 1 -- 172.21.15.16:0/978364699 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f49780a5990 con 0x7f49800d2010 2024-03-31T15:12:29.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:29.231+0000 7f497f400640 1 -- 172.21.15.16:0/978364699 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49780a5d60 con 0x7f49800d2010 2024-03-31T15:12:29.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:29.231+0000 7f4987000640 1 -- 172.21.15.16:0/978364699 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f49800d2010 msgr2=0x7f49800d24a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:29.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:29.231+0000 7f4987000640 1 --2- 172.21.15.16:0/978364699 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f49800d2010 0x7f49800d24a0 secure :-1 s=READY pgs=1453 cs=0 l=1 rev1=1 crypto rx=0x7f4978009e50 tx=0x7f497801ca30 comp rx=0 tx=0).stop 2024-03-31T15:12:29.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:29.231+0000 7f4987000640 1 -- 172.21.15.16:0/978364699 shutdown_connections 2024-03-31T15:12:29.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:29.231+0000 7f4987000640 1 --2- 172.21.15.16:0/978364699 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f49800d2010 0x7f49800d24a0 unknown :-1 s=CLOSED pgs=1453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:29.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:29.231+0000 7f4987000640 1 --2- 172.21.15.16:0/978364699 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f498006f920 0x7f498006fd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:29.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:29.231+0000 7f4987000640 1 -- 172.21.15.16:0/978364699 >> 172.21.15.16:0/978364699 conn(0x7f4980067330 msgr2=0x7f4980067740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:12:29.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:29.231+0000 7f4987000640 1 -- 172.21.15.16:0/978364699 shutdown_connections 2024-03-31T15:12:29.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:29.231+0000 7f4987000640 1 -- 172.21.15.16:0/978364699 wait complete. 2024-03-31T15:12:29.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:29.235+0000 7f4987000640 1 Processor -- start 2024-03-31T15:12:29.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:29.235+0000 7f4987000640 1 -- start start 2024-03-31T15:12:29.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:29.235+0000 7f4987000640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f498006f920 0x7f498015edd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:29.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:29.235+0000 7f4987000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f49800d2010 0x7f4980142b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:29.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:29.235+0000 7f4987000640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f4980145e40 0x7f4980143e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:29.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:29.235+0000 7f4987000640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f49800d0ee0 con 0x7f49800d2010 2024-03-31T15:12:29.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:29.235+0000 7f4987000640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f49800ce9e0 con 0x7f498006f920 2024-03-31T15:12:29.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:29.235+0000 7f4987000640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f49800d2df0 con 0x7f4980145e40 2024-03-31T15:12:29.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:29.235+0000 7f4977400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f49800d2010 0x7f4980142b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:29.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:29.235+0000 7f4977400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f49800d2010 0x7f4980142b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41140/0 (socket says 172.21.15.16:41140) 2024-03-31T15:12:29.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:29.235+0000 7f4977400640 1 -- 172.21.15.16:0/3938732168 learned_addr learned my addr 172.21.15.16:0/3938732168 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:12:29.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:29.235+0000 7f497fe00640 1 --2- 172.21.15.16:0/3938732168 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f498006f920 0x7f498015edd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:29.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:29.235+0000 7f4985000640 1 --2- 172.21.15.16:0/3938732168 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f4980145e40 0x7f4980143e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:29.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:29.235+0000 7f4977400640 1 -- 172.21.15.16:0/3938732168 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f4980145e40 msgr2=0x7f4980143e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:29.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:29.235+0000 7f4977400640 1 --2- 172.21.15.16:0/3938732168 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f4980145e40 0x7f4980143e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:29.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:29.235+0000 7f4977400640 1 -- 172.21.15.16:0/3938732168 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f498006f920 msgr2=0x7f498015edd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:29.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:29.235+0000 7f4977400640 1 --2- 172.21.15.16:0/3938732168 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f498006f920 0x7f498015edd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:29.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:29.235+0000 7f4977400640 1 -- 172.21.15.16:0/3938732168 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49800d0500 con 0x7f49800d2010 2024-03-31T15:12:29.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:29.235+0000 7f4985000640 1 --2- 172.21.15.16:0/3938732168 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f4980145e40 0x7f4980143e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:12:29.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:29.235+0000 7f4977400640 1 --2- 172.21.15.16:0/3938732168 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f49800d2010 0x7f4980142b80 secure :-1 s=READY pgs=1454 cs=0 l=1 rev1=1 crypto rx=0x7f496800e870 tx=0x7f496800ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:12:29.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:29.235+0000 7f497cc00640 1 -- 172.21.15.16:0/3938732168 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4968017070 con 0x7f49800d2010 2024-03-31T15:12:29.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:29.235+0000 7f497cc00640 1 -- 172.21.15.16:0/3938732168 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f496800c940 con 0x7f49800d2010 2024-03-31T15:12:29.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:29.235+0000 7f497cc00640 1 -- 172.21.15.16:0/3938732168 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f496800ccf0 con 0x7f49800d2010 2024-03-31T15:12:29.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:29.235+0000 7f4987000640 1 -- 172.21.15.16:0/3938732168 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4980143760 con 0x7f49800d2010 2024-03-31T15:12:29.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:29.235+0000 7f4987000640 1 -- 172.21.15.16:0/3938732168 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f498015e010 con 0x7f49800d2010 2024-03-31T15:12:29.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:29.235+0000 7f4987000640 1 -- 172.21.15.16:0/3938732168 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4980069950 con 0x7f49800d2010 2024-03-31T15:12:29.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:29.239+0000 7f497cc00640 1 -- 172.21.15.16:0/3938732168 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f496801a070 con 0x7f49800d2010 2024-03-31T15:12:29.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:29.239+0000 7f497cc00640 1 --2- 172.21.15.16:0/3938732168 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f495c068380 0x7f495c06a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:29.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:29.239+0000 7f497cc00640 1 -- 172.21.15.16:0/3938732168 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 5709+0+0 (secure 0 0 0) 0x7f49680869c0 con 0x7f49800d2010 2024-03-31T15:12:29.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:29.239+0000 7f497fe00640 1 --2- 172.21.15.16:0/3938732168 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f495c068380 0x7f495c06a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:29.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:29.239+0000 7f497fe00640 1 --2- 172.21.15.16:0/3938732168 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f495c068380 0x7f495c06a840 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f4970000f70 tx=0x7f4970009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:12:29.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:29.243+0000 7f497cc00640 1 -- 172.21.15.16:0/3938732168 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f4968056e50 con 0x7f49800d2010 2024-03-31T15:12:29.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:29.439+0000 7f4987000640 1 -- 172.21.15.16:0/3938732168 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f49801430c0 con 0x7f49800d2010 2024-03-31T15:12:29.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:29.443+0000 7f497cc00640 1 -- 172.21.15.16:0/3938732168 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 83 v83) ==== 94+0+5180 (secure 0 0 0) 0x7f4968056750 con 0x7f49800d2010 2024-03-31T15:12:29.434 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:12:29.434 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":83,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7139,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6836/3798928623","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":3798928623},{"type":"v1","addr":"172.21.15.89:6836","nonce":3798928623}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":78},{"gid":7142,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/3074283817","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6835","nonce":3074283817},{"type":"v1","addr":"172.21.15.89:6837","nonce":3074283817}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":78},{"gid":8071,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6834/2110189998","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":2110189998},{"type":"v1","addr":"172.21.15.16:6834","nonce":2110189998}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":78}],"filesystems":[{"mdsmap":{"epoch":83,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-31T15:12:16.294600+0000","modified":"2024-03-31T15:12:19.297339+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8074},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8074":{"gid":8074,"name":"a","rank":0,"incarnation":79,"state":"up:active","state_seq":3,"addr":"172.21.15.16:6835/490717480","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6833","nonce":490717480},{"type":"v1","addr":"172.21.15.16:6835","nonce":490717480}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8074,"qdb_cluster":[8074]},"id":7}]} 2024-03-31T15:12:29.435 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 83 2024-03-31T15:12:29.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:29.443+0000 7f4987000640 1 -- 172.21.15.16:0/3938732168 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f495c068380 msgr2=0x7f495c06a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:29.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:29.443+0000 7f4987000640 1 --2- 172.21.15.16:0/3938732168 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f495c068380 0x7f495c06a840 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f4970000f70 tx=0x7f4970009040 comp rx=0 tx=0).stop 2024-03-31T15:12:29.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:29.443+0000 7f4987000640 1 -- 172.21.15.16:0/3938732168 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f49800d2010 msgr2=0x7f4980142b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:29.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:29.443+0000 7f4987000640 1 --2- 172.21.15.16:0/3938732168 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f49800d2010 0x7f4980142b80 secure :-1 s=READY pgs=1454 cs=0 l=1 rev1=1 crypto rx=0x7f496800e870 tx=0x7f496800ec70 comp rx=0 tx=0).stop 2024-03-31T15:12:29.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:29.443+0000 7f4987000640 1 -- 172.21.15.16:0/3938732168 shutdown_connections 2024-03-31T15:12:29.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:29.443+0000 7f4987000640 1 --2- 172.21.15.16:0/3938732168 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f495c068380 0x7f495c06a840 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:29.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:29.443+0000 7f4987000640 1 --2- 172.21.15.16:0/3938732168 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f4980145e40 0x7f4980143e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:29.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:29.443+0000 7f4987000640 1 --2- 172.21.15.16:0/3938732168 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f49800d2010 0x7f4980142b80 unknown :-1 s=CLOSED pgs=1454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:29.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:29.443+0000 7f4987000640 1 --2- 172.21.15.16:0/3938732168 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f498006f920 0x7f498015edd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:29.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:29.443+0000 7f4987000640 1 -- 172.21.15.16:0/3938732168 >> 172.21.15.16:0/3938732168 conn(0x7f4980067330 msgr2=0x7f4980067710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:12:29.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:29.443+0000 7f4987000640 1 -- 172.21.15.16:0/3938732168 shutdown_connections 2024-03-31T15:12:29.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:29.443+0000 7f4987000640 1 -- 172.21.15.16:0/3938732168 wait complete. 2024-03-31T15:12:29.456 DEBUG:teuthology.orchestra.run.smithi016:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata get mds0_inotable - 2024-03-31T15:12:29.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:29.511+0000 7fa196808080 1 Processor -- start 2024-03-31T15:12:29.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:29.511+0000 7fa196808080 1 -- start start 2024-03-31T15:12:29.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:29.511+0000 7fa196808080 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c9aaf6b040 0x55c9ab03e480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:29.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:29.511+0000 7fa196808080 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55c9aafb12e0 0x55c9ab034850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:29.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:29.511+0000 7fa196808080 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c9ab03b770 0x55c9ab034d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:29.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:29.511+0000 7fa196808080 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55c9aaf0c4b0 con 0x55c9ab03b770 2024-03-31T15:12:29.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:29.511+0000 7fa196808080 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55c9aaf10500 con 0x55c9aaf6b040 2024-03-31T15:12:29.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:29.511+0000 7fa196808080 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x55c9aaf68740 con 0x55c9aafb12e0 2024-03-31T15:12:29.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:29.511+0000 7fa194400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c9ab03b770 0x55c9ab034d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:29.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:29.511+0000 7fa194400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c9ab03b770 0x55c9ab034d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41148/0 (socket says 172.21.15.16:41148) 2024-03-31T15:12:29.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:29.511+0000 7fa194400640 1 -- 172.21.15.16:0/1645908497 learned_addr learned my addr 172.21.15.16:0/1645908497 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:12:29.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:29.511+0000 7fa193a00640 1 --2- 172.21.15.16:0/1645908497 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c9aaf6b040 0x55c9ab03e480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:29.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:29.511+0000 7fa193000640 1 --2- 172.21.15.16:0/1645908497 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55c9aafb12e0 0x55c9ab034850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:29.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:29.511+0000 7fa194400640 1 -- 172.21.15.16:0/1645908497 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55c9aafb12e0 msgr2=0x55c9ab034850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:12:29.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:29.511+0000 7fa194400640 1 --2- 172.21.15.16:0/1645908497 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55c9aafb12e0 0x55c9ab034850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:29.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:29.511+0000 7fa194400640 1 -- 172.21.15.16:0/1645908497 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c9aaf6b040 msgr2=0x55c9ab03e480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:12:29.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:29.511+0000 7fa194400640 1 --2- 172.21.15.16:0/1645908497 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c9aaf6b040 0x55c9ab03e480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:29.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:29.511+0000 7fa194400640 1 -- 172.21.15.16:0/1645908497 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c9aaf67340 con 0x55c9ab03b770 2024-03-31T15:12:29.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:29.511+0000 7fa193a00640 1 --2- 172.21.15.16:0/1645908497 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c9aaf6b040 0x55c9ab03e480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:12:29.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:29.511+0000 7fa194400640 1 --2- 172.21.15.16:0/1645908497 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c9ab03b770 0x55c9ab034d90 secure :-1 s=READY pgs=1457 cs=0 l=1 rev1=1 crypto rx=0x7fa18c001720 tx=0x7fa18c001be0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=93f70aad2fe20053 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:12:29.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:29.511+0000 7fa191c00640 1 -- 172.21.15.16:0/1645908497 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa18c003660 con 0x55c9ab03b770 2024-03-31T15:12:29.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:29.511+0000 7fa191c00640 1 -- 172.21.15.16:0/1645908497 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa18c0062e0 con 0x55c9ab03b770 2024-03-31T15:12:29.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:29.511+0000 7fa191c00640 1 -- 172.21.15.16:0/1645908497 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa18c003a40 con 0x55c9ab03b770 2024-03-31T15:12:29.508 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:29.515+0000 7fa196808080 1 -- 172.21.15.16:0/1645908497 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c9ab03b770 msgr2=0x55c9ab034d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:29.508 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:29.515+0000 7fa196808080 1 --2- 172.21.15.16:0/1645908497 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c9ab03b770 0x55c9ab034d90 secure :-1 s=READY pgs=1457 cs=0 l=1 rev1=1 crypto rx=0x7fa18c001720 tx=0x7fa18c001be0 comp rx=0 tx=0).stop 2024-03-31T15:12:29.508 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:29.515+0000 7fa196808080 1 -- 172.21.15.16:0/1645908497 shutdown_connections 2024-03-31T15:12:29.508 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:29.515+0000 7fa196808080 1 --2- 172.21.15.16:0/1645908497 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c9ab03b770 0x55c9ab034d90 unknown :-1 s=CLOSED pgs=1457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:29.508 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:29.515+0000 7fa196808080 1 --2- 172.21.15.16:0/1645908497 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55c9aafb12e0 0x55c9ab034850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:29.508 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:29.515+0000 7fa196808080 1 --2- 172.21.15.16:0/1645908497 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c9aaf6b040 0x55c9ab03e480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:29.508 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:29.515+0000 7fa196808080 1 -- 172.21.15.16:0/1645908497 >> 172.21.15.16:0/1645908497 conn(0x55c9aafaf0b0 msgr2=0x55c9aaf95ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:12:29.508 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:29.515+0000 7fa196808080 1 -- 172.21.15.16:0/1645908497 shutdown_connections 2024-03-31T15:12:29.508 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:29.515+0000 7fa196808080 1 -- 172.21.15.16:0/1645908497 wait complete. 2024-03-31T15:12:29.508 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:29.515+0000 7fa196808080 1 Processor -- start 2024-03-31T15:12:29.508 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:29.515+0000 7fa196808080 1 -- start start 2024-03-31T15:12:29.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:29.515+0000 7fa196808080 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c9aaf6b040 0x55c9ab0d8fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:29.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:29.515+0000 7fa196808080 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55c9aafb12e0 0x55c9ab024b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:29.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:29.515+0000 7fa196808080 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c9ab03b770 0x55c9ab035d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:29.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:29.515+0000 7fa196808080 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55c9aaf675e0 con 0x55c9aaf6b040 2024-03-31T15:12:29.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:29.515+0000 7fa196808080 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55c9aafad540 con 0x55c9ab03b770 2024-03-31T15:12:29.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:29.515+0000 7fa196808080 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x55c9aaf67880 con 0x55c9aafb12e0 2024-03-31T15:12:29.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:29.515+0000 7fa193a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c9aaf6b040 0x55c9ab0d8fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:29.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:29.515+0000 7fa193a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c9aaf6b040 0x55c9ab0d8fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41156/0 (socket says 172.21.15.16:41156) 2024-03-31T15:12:29.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:29.515+0000 7fa193a00640 1 -- 172.21.15.16:0/2960907472 learned_addr learned my addr 172.21.15.16:0/2960907472 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:12:29.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:29.515+0000 7fa193000640 1 --2- 172.21.15.16:0/2960907472 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55c9aafb12e0 0x55c9ab024b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:29.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:29.515+0000 7fa194400640 1 --2- 172.21.15.16:0/2960907472 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c9ab03b770 0x55c9ab035d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:29.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:29.515+0000 7fa193a00640 1 -- 172.21.15.16:0/2960907472 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55c9aafb12e0 msgr2=0x55c9ab024b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:29.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:29.515+0000 7fa193a00640 1 --2- 172.21.15.16:0/2960907472 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55c9aafb12e0 0x55c9ab024b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:29.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:29.515+0000 7fa193a00640 1 -- 172.21.15.16:0/2960907472 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c9ab03b770 msgr2=0x55c9ab035d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:29.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:29.515+0000 7fa193a00640 1 --2- 172.21.15.16:0/2960907472 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c9ab03b770 0x55c9ab035d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:29.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:29.515+0000 7fa193a00640 1 -- 172.21.15.16:0/2960907472 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c9ab034e30 con 0x55c9aaf6b040 2024-03-31T15:12:29.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:29.515+0000 7fa193a00640 1 --2- 172.21.15.16:0/2960907472 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c9aaf6b040 0x55c9ab0d8fe0 secure :-1 s=READY pgs=1458 cs=0 l=1 rev1=1 crypto rx=0x7fa1840ae040 tx=0x7fa184018150 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:12:29.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:29.515+0000 7fa18be00640 1 -- 172.21.15.16:0/2960907472 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1840bc070 con 0x55c9aaf6b040 2024-03-31T15:12:29.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:29.515+0000 7fa18be00640 1 -- 172.21.15.16:0/2960907472 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa1840b6650 con 0x55c9aaf6b040 2024-03-31T15:12:29.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:29.515+0000 7fa196808080 1 -- 172.21.15.16:0/2960907472 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55c9ab0350d0 con 0x55c9aaf6b040 2024-03-31T15:12:29.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:29.519+0000 7fa18be00640 1 -- 172.21.15.16:0/2960907472 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1840b0040 con 0x55c9aaf6b040 2024-03-31T15:12:29.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:29.519+0000 7fa196808080 1 -- 172.21.15.16:0/2960907472 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c9ab035610 con 0x55c9aaf6b040 2024-03-31T15:12:29.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:29.519+0000 7fa18be00640 1 -- 172.21.15.16:0/2960907472 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fa1840bf070 con 0x55c9aaf6b040 2024-03-31T15:12:29.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:29.519+0000 7fa18be00640 1 --2- 172.21.15.16:0/2960907472 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fa174066d90 0x7fa174069250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:29.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:29.519+0000 7fa18be00640 1 -- 172.21.15.16:0/2960907472 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 5709+0+0 (secure 0 0 0) 0x7fa1841203d0 con 0x55c9aaf6b040 2024-03-31T15:12:29.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:29.519+0000 7fa193000640 1 --2- 172.21.15.16:0/2960907472 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fa174066d90 0x7fa174069250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:29.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:29.519+0000 7fa196808080 1 --2- 172.21.15.16:0/2960907472 >> [v2:172.21.15.16:6819/3018526646,v1:172.21.15.16:6822/3018526646] conn(0x55c9ab034990 0x55c9ab036c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:29.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:29.519+0000 7fa196808080 1 -- 172.21.15.16:0/2960907472 --> [v2:172.21.15.16:6819/3018526646,v1:172.21.15.16:6822/3018526646] -- osd_op(unknown.0.0:1 14.13 14:c91d4a1d:::mds0_inotable:head [read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e96) -- 0x55c9ab0250d0 con 0x55c9ab034990 2024-03-31T15:12:29.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:29.519+0000 7fa194400640 1 --2- 172.21.15.16:0/2960907472 >> [v2:172.21.15.16:6819/3018526646,v1:172.21.15.16:6822/3018526646] conn(0x55c9ab034990 0x55c9ab036c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:29.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:29.523+0000 7fa193000640 1 --2- 172.21.15.16:0/2960907472 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fa174066d90 0x7fa174069250 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7fa180003f20 tx=0x7fa1800038a0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:12:29.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:29.523+0000 7fa194400640 1 --2- 172.21.15.16:0/2960907472 >> [v2:172.21.15.16:6819/3018526646,v1:172.21.15.16:6822/3018526646] conn(0x55c9ab034990 0x55c9ab036c50 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:12:29.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:29.523+0000 7fa194400640 1 -- 172.21.15.16:0/2960907472 <== osd.2 v2:172.21.15.16:6819/3018526646 1 ==== osd_op_reply(1 mds0_inotable [read 0~34 out=34b] v0'0 uv3 ondisk = 0) ==== 157+0+34 (crc 0 0 0) 0x7fa18c01b040 con 0x55c9ab034990 2024-03-31T15:12:29.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:29.523+0000 7fa196808080 1 -- 172.21.15.16:0/2960907472 >> [v2:172.21.15.16:6819/3018526646,v1:172.21.15.16:6822/3018526646] conn(0x55c9ab034990 msgr2=0x55c9ab036c50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:29.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:29.523+0000 7fa196808080 1 --2- 172.21.15.16:0/2960907472 >> [v2:172.21.15.16:6819/3018526646,v1:172.21.15.16:6822/3018526646] conn(0x55c9ab034990 0x55c9ab036c50 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:29.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:29.523+0000 7fa196808080 1 -- 172.21.15.16:0/2960907472 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fa174066d90 msgr2=0x7fa174069250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:29.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:29.523+0000 7fa196808080 1 --2- 172.21.15.16:0/2960907472 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fa174066d90 0x7fa174069250 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7fa180003f20 tx=0x7fa1800038a0 comp rx=0 tx=0).stop 2024-03-31T15:12:29.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:29.523+0000 7fa196808080 1 -- 172.21.15.16:0/2960907472 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c9aaf6b040 msgr2=0x55c9ab0d8fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:29.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:29.523+0000 7fa196808080 1 --2- 172.21.15.16:0/2960907472 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c9aaf6b040 0x55c9ab0d8fe0 secure :-1 s=READY pgs=1458 cs=0 l=1 rev1=1 crypto rx=0x7fa1840ae040 tx=0x7fa184018150 comp rx=0 tx=0).stop 2024-03-31T15:12:29.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:29.523+0000 7fa194400640 1 -- 172.21.15.16:0/2960907472 reap_dead start 2024-03-31T15:12:29.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:29.523+0000 7fa196808080 1 -- 172.21.15.16:0/2960907472 shutdown_connections 2024-03-31T15:12:29.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:29.523+0000 7fa196808080 1 -- 172.21.15.16:0/2960907472 >> 172.21.15.16:0/2960907472 conn(0x55c9aafaf0b0 msgr2=0x55c9aaf661b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:12:29.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:29.523+0000 7fa196808080 1 -- 172.21.15.16:0/2960907472 shutdown_connections 2024-03-31T15:12:29.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:29.523+0000 7fa196808080 1 -- 172.21.15.16:0/2960907472 wait complete. 2024-03-31T15:12:29.519 INFO:teuthology.orchestra.run.smithi016.stdout:����� 2024-03-31T15:12:29.519 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-31T15:12:29.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:29.671+0000 7f35f4e00640 1 Processor -- start 2024-03-31T15:12:29.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:29.671+0000 7f35f4e00640 1 -- start start 2024-03-31T15:12:29.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:29.671+0000 7f35f4e00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f35f015e420 0x7f35f00726d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:29.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:29.671+0000 7f35f4e00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f35f00b36a0 0x7f35f006dc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:29.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:29.671+0000 7f35f4e00640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f35f015e800 con 0x7f35f006e150 2024-03-31T15:12:29.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:29.671+0000 7f35f4e00640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f35f00b3ba0 con 0x7f35f00b36a0 2024-03-31T15:12:29.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:29.671+0000 7f35f4e00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f35f00b3e80 con 0x7f35f015e420 2024-03-31T15:12:29.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:29.671+0000 7f35eea00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f35f00b36a0 0x7f35f006dc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:29.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:29.671+0000 7f35eea00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f35f00b36a0 0x7f35f006dc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41164/0 (socket says 172.21.15.16:41164) 2024-03-31T15:12:29.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:29.671+0000 7f35eea00640 1 -- 172.21.15.16:0/351266343 learned_addr learned my addr 172.21.15.16:0/351266343 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:12:29.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:29.671+0000 7f35ef400640 1 --2- 172.21.15.16:0/351266343 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f35f015e420 0x7f35f00726d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:29.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:29.671+0000 7f35eea00640 1 -- 172.21.15.16:0/351266343 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f35f015e420 msgr2=0x7f35f00726d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:12:29.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:29.671+0000 7f35eea00640 1 --2- 172.21.15.16:0/351266343 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f35f015e420 0x7f35f00726d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:29.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:29.671+0000 7f35eea00640 1 -- 172.21.15.16:0/351266343 >> v1:172.21.15.89:6789/0 conn(0x7f35f006e150 legacy=0x7f35f00b4c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:29.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:29.671+0000 7f35eea00640 1 -- 172.21.15.16:0/351266343 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35f00b4160 con 0x7f35f00b36a0 2024-03-31T15:12:29.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:29.675+0000 7f35eea00640 1 --2- 172.21.15.16:0/351266343 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f35f00b36a0 0x7f35f006dc10 secure :-1 s=READY pgs=1459 cs=0 l=1 rev1=1 crypto rx=0x7f35dc009870 tx=0x7f35dc01c530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e251ee20d88d312d server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:12:29.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:29.675+0000 7f35ee000640 1 -- 172.21.15.16:0/351266343 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35dc0a95f0 con 0x7f35f00b36a0 2024-03-31T15:12:29.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:29.675+0000 7f35ee000640 1 -- 172.21.15.16:0/351266343 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f35dc0a9790 con 0x7f35f00b36a0 2024-03-31T15:12:29.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:29.675+0000 7f35ee000640 1 -- 172.21.15.16:0/351266343 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35dc0a9b60 con 0x7f35f00b36a0 2024-03-31T15:12:29.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:29.675+0000 7f35f4e00640 1 -- 172.21.15.16:0/351266343 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f35f00b36a0 msgr2=0x7f35f006dc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:29.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:29.675+0000 7f35f4e00640 1 --2- 172.21.15.16:0/351266343 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f35f00b36a0 0x7f35f006dc10 secure :-1 s=READY pgs=1459 cs=0 l=1 rev1=1 crypto rx=0x7f35dc009870 tx=0x7f35dc01c530 comp rx=0 tx=0).stop 2024-03-31T15:12:29.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:29.675+0000 7f35f4e00640 1 -- 172.21.15.16:0/351266343 shutdown_connections 2024-03-31T15:12:29.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:29.675+0000 7f35f4e00640 1 --2- 172.21.15.16:0/351266343 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f35f00b36a0 0x7f35f006dc10 unknown :-1 s=CLOSED pgs=1459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:29.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:29.675+0000 7f35f4e00640 1 --2- 172.21.15.16:0/351266343 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f35f015e420 0x7f35f00726d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:29.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:29.675+0000 7f35f4e00640 1 -- 172.21.15.16:0/351266343 >> 172.21.15.16:0/351266343 conn(0x7f35f0071bb0 msgr2=0x7f35f0071fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:12:29.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:29.675+0000 7f35f4e00640 1 -- 172.21.15.16:0/351266343 shutdown_connections 2024-03-31T15:12:29.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:29.675+0000 7f35f4e00640 1 -- 172.21.15.16:0/351266343 wait complete. 2024-03-31T15:12:29.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:29.675+0000 7f35f4e00640 1 Processor -- start 2024-03-31T15:12:29.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:29.675+0000 7f35f4e00640 1 -- start start 2024-03-31T15:12:29.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:29.679+0000 7f35f4e00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f35f006e150 0x7f35f0148150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:29.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:29.679+0000 7f35f4e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f35f00b36a0 0x7f35f0160c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:29.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:29.679+0000 7f35f4e00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f35f015e420 0x7f35f0161140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:29.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:29.679+0000 7f35f4e00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f35f016b480 con 0x7f35f00b36a0 2024-03-31T15:12:29.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:29.679+0000 7f35f4e00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f35f016b300 con 0x7f35f006e150 2024-03-31T15:12:29.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:29.679+0000 7f35f4e00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f35f016b600 con 0x7f35f015e420 2024-03-31T15:12:29.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:29.679+0000 7f35eea00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f35f00b36a0 0x7f35f0160c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:29.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:29.679+0000 7f35eea00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f35f00b36a0 0x7f35f0160c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41176/0 (socket says 172.21.15.16:41176) 2024-03-31T15:12:29.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:29.679+0000 7f35eea00640 1 -- 172.21.15.16:0/3699540602 learned_addr learned my addr 172.21.15.16:0/3699540602 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:12:29.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:29.679+0000 7f35ef400640 1 --2- 172.21.15.16:0/3699540602 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f35f006e150 0x7f35f0148150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:29.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:29.679+0000 7f35efe00640 1 --2- 172.21.15.16:0/3699540602 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f35f015e420 0x7f35f0161140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:29.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:29.679+0000 7f35eea00640 1 -- 172.21.15.16:0/3699540602 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f35f015e420 msgr2=0x7f35f0161140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:29.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:29.679+0000 7f35eea00640 1 --2- 172.21.15.16:0/3699540602 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f35f015e420 0x7f35f0161140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:29.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:29.679+0000 7f35eea00640 1 -- 172.21.15.16:0/3699540602 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f35f006e150 msgr2=0x7f35f0148150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:29.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:29.679+0000 7f35eea00640 1 --2- 172.21.15.16:0/3699540602 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f35f006e150 0x7f35f0148150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:29.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:29.679+0000 7f35eea00640 1 -- 172.21.15.16:0/3699540602 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35f01517b0 con 0x7f35f00b36a0 2024-03-31T15:12:29.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:29.679+0000 7f35ef400640 1 --2- 172.21.15.16:0/3699540602 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f35f006e150 0x7f35f0148150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:12:29.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:29.679+0000 7f35eea00640 1 --2- 172.21.15.16:0/3699540602 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f35f00b36a0 0x7f35f0160c00 secure :-1 s=READY pgs=1460 cs=0 l=1 rev1=1 crypto rx=0x7f35dc0ab040 tx=0x7f35dc01ad70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:12:29.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:29.679+0000 7f35e7400640 1 -- 172.21.15.16:0/3699540602 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35dc0b1600 con 0x7f35f00b36a0 2024-03-31T15:12:29.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:29.679+0000 7f35e7400640 1 -- 172.21.15.16:0/3699540602 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f35dc0b17a0 con 0x7f35f00b36a0 2024-03-31T15:12:29.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:29.679+0000 7f35e7400640 1 -- 172.21.15.16:0/3699540602 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35dc0b1b90 con 0x7f35f00b36a0 2024-03-31T15:12:29.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:29.679+0000 7f35f4e00640 1 -- 172.21.15.16:0/3699540602 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f35f01511f0 con 0x7f35f00b36a0 2024-03-31T15:12:29.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:29.679+0000 7f35f4e00640 1 -- 172.21.15.16:0/3699540602 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f35f01502c0 con 0x7f35f00b36a0 2024-03-31T15:12:29.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:29.679+0000 7f35f4e00640 1 -- 172.21.15.16:0/3699540602 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f35f004f8c0 con 0x7f35f00b36a0 2024-03-31T15:12:29.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:29.683+0000 7f35e7400640 1 -- 172.21.15.16:0/3699540602 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f35dc0bc450 con 0x7f35f00b36a0 2024-03-31T15:12:29.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:29.683+0000 7f35e7400640 1 --2- 172.21.15.16:0/3699540602 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f35d00683b0 0x7f35d006a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:29.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:29.683+0000 7f35e7400640 1 -- 172.21.15.16:0/3699540602 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 5709+0+0 (secure 0 0 0) 0x7f35dc121030 con 0x7f35f00b36a0 2024-03-31T15:12:29.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:29.683+0000 7f35ef400640 1 --2- 172.21.15.16:0/3699540602 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f35d00683b0 0x7f35d006a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:29.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:29.683+0000 7f35ef400640 1 --2- 172.21.15.16:0/3699540602 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f35d00683b0 0x7f35d006a870 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f35e0009b10 tx=0x7f35e0008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:12:29.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:29.687+0000 7f35e7400640 1 -- 172.21.15.16:0/3699540602 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f35dc0c3030 con 0x7f35f00b36a0 2024-03-31T15:12:29.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:29.883+0000 7f35f4e00640 1 -- 172.21.15.16:0/3699540602 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f35f00729f0 con 0x7f35f00b36a0 2024-03-31T15:12:29.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:29.883+0000 7f35e7400640 1 -- 172.21.15.16:0/3699540602 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v83) ==== 53+0+73 (secure 0 0 0) 0x7f35dc0ad020 con 0x7f35f00b36a0 2024-03-31T15:12:29.878 INFO:teuthology.orchestra.run.smithi016.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-31T15:12:29.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:29.887+0000 7f35f4e00640 1 -- 172.21.15.16:0/3699540602 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f35d00683b0 msgr2=0x7f35d006a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:29.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:29.887+0000 7f35f4e00640 1 --2- 172.21.15.16:0/3699540602 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f35d00683b0 0x7f35d006a870 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f35e0009b10 tx=0x7f35e0008040 comp rx=0 tx=0).stop 2024-03-31T15:12:29.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:29.887+0000 7f35f4e00640 1 -- 172.21.15.16:0/3699540602 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f35f00b36a0 msgr2=0x7f35f0160c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:29.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:29.887+0000 7f35f4e00640 1 --2- 172.21.15.16:0/3699540602 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f35f00b36a0 0x7f35f0160c00 secure :-1 s=READY pgs=1460 cs=0 l=1 rev1=1 crypto rx=0x7f35dc0ab040 tx=0x7f35dc01ad70 comp rx=0 tx=0).stop 2024-03-31T15:12:29.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:29.887+0000 7f35f4e00640 1 -- 172.21.15.16:0/3699540602 shutdown_connections 2024-03-31T15:12:29.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:29.887+0000 7f35f4e00640 1 --2- 172.21.15.16:0/3699540602 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f35d00683b0 0x7f35d006a870 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:29.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:29.887+0000 7f35f4e00640 1 --2- 172.21.15.16:0/3699540602 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f35f015e420 0x7f35f0161140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:29.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:29.887+0000 7f35f4e00640 1 --2- 172.21.15.16:0/3699540602 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f35f00b36a0 0x7f35f0160c00 unknown :-1 s=CLOSED pgs=1460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:29.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:29.887+0000 7f35f4e00640 1 --2- 172.21.15.16:0/3699540602 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f35f006e150 0x7f35f0148150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:29.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:29.887+0000 7f35f4e00640 1 -- 172.21.15.16:0/3699540602 >> 172.21.15.16:0/3699540602 conn(0x7f35f0071bb0 msgr2=0x7f35f00b1bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:12:29.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:29.887+0000 7f35f4e00640 1 -- 172.21.15.16:0/3699540602 shutdown_connections 2024-03-31T15:12:29.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:29.887+0000 7f35f4e00640 1 -- 172.21.15.16:0/3699540602 wait complete. 2024-03-31T15:12:29.898 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-31T15:12:29.898 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-31T15:12:29.898 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi016.front.sepia.ceph.com 2024-03-31T15:12:29.898 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-31T15:12:29.899 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-31T15:12:29.899 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-31T15:12:29.899 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-31T15:12:29.899 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-31T15:12:29.899 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-31T15:12:29.899 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:12:29.899 DEBUG:teuthology.orchestra.run.smithi016:> ip addr 2024-03-31T15:12:29.908 INFO:teuthology.orchestra.run.smithi016.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-31T15:12:29.909 INFO:teuthology.orchestra.run.smithi016.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-31T15:12:29.909 INFO:teuthology.orchestra.run.smithi016.stdout: inet 127.0.0.1/8 scope host lo 2024-03-31T15:12:29.909 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2024-03-31T15:12:29.909 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 ::1/128 scope host 2024-03-31T15:12:29.909 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2024-03-31T15:12:29.909 INFO:teuthology.orchestra.run.smithi016.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-03-31T15:12:29.909 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:6b:fd:e0 brd ff:ff:ff:ff:ff:ff 2024-03-31T15:12:29.909 INFO:teuthology.orchestra.run.smithi016.stdout: altname enp7s0f0 2024-03-31T15:12:29.909 INFO:teuthology.orchestra.run.smithi016.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-03-31T15:12:29.909 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:6b:fd:e1 brd ff:ff:ff:ff:ff:ff 2024-03-31T15:12:29.909 INFO:teuthology.orchestra.run.smithi016.stdout: altname enp7s0f1 2024-03-31T15:12:29.909 INFO:teuthology.orchestra.run.smithi016.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-31T15:12:29.909 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:bd:15:64 brd ff:ff:ff:ff:ff:ff 2024-03-31T15:12:29.909 INFO:teuthology.orchestra.run.smithi016.stdout: altname enp1s0f0 2024-03-31T15:12:29.910 INFO:teuthology.orchestra.run.smithi016.stdout: inet 172.21.15.16/20 brd 172.21.15.255 scope global dynamic ens1f0 2024-03-31T15:12:29.910 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft 41931sec preferred_lft 41931sec 2024-03-31T15:12:29.910 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 fe80::ec4:7aff:febd:1564/64 scope link 2024-03-31T15:12:29.910 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2024-03-31T15:12:29.910 INFO:teuthology.orchestra.run.smithi016.stdout:5: ens1f1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-03-31T15:12:29.910 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:bd:15:65 brd ff:ff:ff:ff:ff:ff 2024-03-31T15:12:29.910 INFO:teuthology.orchestra.run.smithi016.stdout: altname enp1s0f1 2024-03-31T15:12:29.910 INFO:teuthology.orchestra.run.smithi016.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2024-03-31T15:12:29.910 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 02:42:d4:1f:7d:73 brd ff:ff:ff:ff:ff:ff 2024-03-31T15:12:29.910 INFO:teuthology.orchestra.run.smithi016.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2024-03-31T15:12:29.910 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2024-03-31T15:12:29.910 INFO:teuthology.orchestra.run.smithi016.stdout:7: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-31T15:12:29.910 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 5e:1f:84:6d:79:ae brd ff:ff:ff:ff:ff:ff 2024-03-31T15:12:29.911 INFO:teuthology.orchestra.run.smithi016.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-31T15:12:29.911 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2024-03-31T15:12:29.911 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 fe80::5c1f:84ff:fe6d:79ae/64 scope link 2024-03-31T15:12:29.911 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2024-03-31T15:12:29.911 INFO:teuthology.orchestra.run.smithi016.stdout:8: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-31T15:12:29.911 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0e:5c:f6:1f:82:2f brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-31T15:12:29.911 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 fe80::c5c:f6ff:fe1f:822f/64 scope link 2024-03-31T15:12:29.911 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2024-03-31T15:12:29.911 INFO:teuthology.orchestra.run.smithi016.stdout:9: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-31T15:12:29.911 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether c6:a6:3d:69:d0:db brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-31T15:12:29.911 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 fe80::c4a6:3dff:fe69:d0db/64 scope link 2024-03-31T15:12:29.911 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2024-03-31T15:12:29.912 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:12:29.912 DEBUG:teuthology.orchestra.run.smithi016:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-31T15:12:29.971 INFO:teuthology.orchestra.run.smithi016.stdout:1 2024-03-31T15:12:29.973 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:12:29.973 DEBUG:teuthology.orchestra.run.smithi016:> route 2024-03-31T15:12:30.026 INFO:teuthology.orchestra.run.smithi016.stdout:Kernel IP routing table 2024-03-31T15:12:30.026 INFO:teuthology.orchestra.run.smithi016.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-31T15:12:30.026 INFO:teuthology.orchestra.run.smithi016.stdout:default _gateway 0.0.0.0 UG 0 0 0 ens1f0 2024-03-31T15:12:30.027 INFO:teuthology.orchestra.run.smithi016.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2024-03-31T15:12:30.027 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 0 0 0 ens1f0 2024-03-31T15:12:30.027 INFO:teuthology.orchestra.run.smithi016.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-31T15:12:30.027 DEBUG:teuthology.orchestra.run.smithi016:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-31T15:12:30.027 DEBUG:teuthology.orchestra.run.smithi016:> set -e 2024-03-31T15:12:30.027 DEBUG:teuthology.orchestra.run.smithi016:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-03-31T15:12:30.027 DEBUG:teuthology.orchestra.run.smithi016:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-03-31T15:12:30.027 DEBUG:teuthology.orchestra.run.smithi016:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-03-31T15:12:30.027 DEBUG:teuthology.orchestra.run.smithi016:> ') 2024-03-31T15:12:30.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:30 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-31T15:12:30.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:30 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-31T15:12:30.164 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:12:30.165 DEBUG:teuthology.orchestra.run.smithi016:> ip netns list 2024-03-31T15:12:30.174 INFO:teuthology.orchestra.run.smithi016.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-31T15:12:30.174 INFO:teuthology.orchestra.run.smithi016.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-31T15:12:30.174 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-31T15:12:30.174 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:12:30.174 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip link set brx.0 up 2024-03-31T15:12:30.237 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-03-31T15:12:30.237 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-31T15:12:30.237 DEBUG:teuthology.orchestra.run.smithi016:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-31T15:12:30.285 INFO:teuthology.orchestra.run.smithi016.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-31T15:12:30.285 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-31T15:12:30.285 DEBUG:teuthology.orchestra.run.smithi016:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-31T15:12:30.336 DEBUG:teuthology.orchestra.run.smithi016:> sudo modprobe fuse 2024-03-31T15:12:30.397 DEBUG:teuthology.orchestra.run.smithi016:> cat /proc/self/mounts | awk '{print $2}' 2024-03-31T15:12:30.446 INFO:teuthology.orchestra.run.smithi016.stdout:/sys 2024-03-31T15:12:30.446 INFO:teuthology.orchestra.run.smithi016.stdout:/proc 2024-03-31T15:12:30.447 INFO:teuthology.orchestra.run.smithi016.stdout:/dev 2024-03-31T15:12:30.447 INFO:teuthology.orchestra.run.smithi016.stdout:/dev/pts 2024-03-31T15:12:30.447 INFO:teuthology.orchestra.run.smithi016.stdout:/run 2024-03-31T15:12:30.447 INFO:teuthology.orchestra.run.smithi016.stdout:/ 2024-03-31T15:12:30.447 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/kernel/security 2024-03-31T15:12:30.447 INFO:teuthology.orchestra.run.smithi016.stdout:/dev/shm 2024-03-31T15:12:30.447 INFO:teuthology.orchestra.run.smithi016.stdout:/run/lock 2024-03-31T15:12:30.447 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/cgroup 2024-03-31T15:12:30.447 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/pstore 2024-03-31T15:12:30.447 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/bpf 2024-03-31T15:12:30.447 INFO:teuthology.orchestra.run.smithi016.stdout:/proc/sys/fs/binfmt_misc 2024-03-31T15:12:30.447 INFO:teuthology.orchestra.run.smithi016.stdout:/dev/hugepages 2024-03-31T15:12:30.448 INFO:teuthology.orchestra.run.smithi016.stdout:/dev/mqueue 2024-03-31T15:12:30.448 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/kernel/debug 2024-03-31T15:12:30.448 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/kernel/tracing 2024-03-31T15:12:30.448 INFO:teuthology.orchestra.run.smithi016.stdout:/proc/fs/nfsd 2024-03-31T15:12:30.448 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/fuse/connections 2024-03-31T15:12:30.448 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/kernel/config 2024-03-31T15:12:30.448 INFO:teuthology.orchestra.run.smithi016.stdout:/run/credentials/systemd-sysusers.service 2024-03-31T15:12:30.448 INFO:teuthology.orchestra.run.smithi016.stdout:/run/rpc_pipefs 2024-03-31T15:12:30.448 INFO:teuthology.orchestra.run.smithi016.stdout:/run/user/1000 2024-03-31T15:12:30.448 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/ceph 2024-03-31T15:12:30.448 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/ceph/osd/ceph-0 2024-03-31T15:12:30.448 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/ceph/osd/ceph-1 2024-03-31T15:12:30.448 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/ceph/osd/ceph-2 2024-03-31T15:12:30.448 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/ceph/osd/ceph-3 2024-03-31T15:12:30.448 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns 2024-03-31T15:12:30.449 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-31T15:12:30.449 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-31T15:12:30.449 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-31T15:12:30.449 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-31T15:12:30.450 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:12:30.450 DEBUG:teuthology.orchestra.run.smithi016:> ls /sys/fs/fuse/connections 2024-03-31T15:12:30.501 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-31T15:12:30.501 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-03-31T15:12:30.544 DEBUG:teuthology.orchestra.run.smithi016:> sudo modprobe fuse 2024-03-31T15:12:30.564 DEBUG:teuthology.orchestra.run.smithi016:> cat /proc/self/mounts | awk '{print $2}' 2024-03-31T15:12:30.619 INFO:teuthology.orchestra.run.smithi016.stdout:/sys 2024-03-31T15:12:30.619 INFO:teuthology.orchestra.run.smithi016.stdout:/proc 2024-03-31T15:12:30.619 INFO:teuthology.orchestra.run.smithi016.stdout:/dev 2024-03-31T15:12:30.619 INFO:teuthology.orchestra.run.smithi016.stdout:/dev/pts 2024-03-31T15:12:30.619 INFO:teuthology.orchestra.run.smithi016.stdout:/run 2024-03-31T15:12:30.619 INFO:teuthology.orchestra.run.smithi016.stdout:/ 2024-03-31T15:12:30.619 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/kernel/security 2024-03-31T15:12:30.619 INFO:teuthology.orchestra.run.smithi016.stdout:/dev/shm 2024-03-31T15:12:30.619 INFO:teuthology.orchestra.run.smithi016.stdout:/run/lock 2024-03-31T15:12:30.619 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/cgroup 2024-03-31T15:12:30.620 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/pstore 2024-03-31T15:12:30.620 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/bpf 2024-03-31T15:12:30.620 INFO:teuthology.orchestra.run.smithi016.stdout:/proc/sys/fs/binfmt_misc 2024-03-31T15:12:30.620 INFO:teuthology.orchestra.run.smithi016.stdout:/dev/hugepages 2024-03-31T15:12:30.620 INFO:teuthology.orchestra.run.smithi016.stdout:/dev/mqueue 2024-03-31T15:12:30.620 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/kernel/debug 2024-03-31T15:12:30.620 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/kernel/tracing 2024-03-31T15:12:30.620 INFO:teuthology.orchestra.run.smithi016.stdout:/proc/fs/nfsd 2024-03-31T15:12:30.620 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/fuse/connections 2024-03-31T15:12:30.620 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/kernel/config 2024-03-31T15:12:30.620 INFO:teuthology.orchestra.run.smithi016.stdout:/run/credentials/systemd-sysusers.service 2024-03-31T15:12:30.620 INFO:teuthology.orchestra.run.smithi016.stdout:/run/rpc_pipefs 2024-03-31T15:12:30.620 INFO:teuthology.orchestra.run.smithi016.stdout:/run/user/1000 2024-03-31T15:12:30.620 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/ceph 2024-03-31T15:12:30.620 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/ceph/osd/ceph-0 2024-03-31T15:12:30.621 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/ceph/osd/ceph-1 2024-03-31T15:12:30.621 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/ceph/osd/ceph-2 2024-03-31T15:12:30.621 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/ceph/osd/ceph-3 2024-03-31T15:12:30.621 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns 2024-03-31T15:12:30.621 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-31T15:12:30.621 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-31T15:12:30.621 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-31T15:12:30.621 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-31T15:12:30.622 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:12:30.622 DEBUG:teuthology.orchestra.run.smithi016:> ls /sys/fs/fuse/connections 2024-03-31T15:12:30.641 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi016.stderr:2024-03-31T15:12:30.647+0000 7f16ea3a9580 -1 init, newargv = 0x55f0de879e60 newargc=15 2024-03-31T15:12:30.641 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi016.stderr:ceph-fuse[48731]: starting ceph client 2024-03-31T15:12:30.649 INFO:teuthology.orchestra.run.smithi016.stdout:43 2024-03-31T15:12:30.649 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi016.stderr:ceph-fuse[48731]: starting fuse 2024-03-31T15:12:30.650 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [43] 2024-03-31T15:12:30.650 DEBUG:teuthology.orchestra.run.smithi016:> sudo stdin-killer -- python3 -c ' 2024-03-31T15:12:30.650 DEBUG:teuthology.orchestra.run.smithi016:> import glob 2024-03-31T15:12:30.650 DEBUG:teuthology.orchestra.run.smithi016:> import re 2024-03-31T15:12:30.650 DEBUG:teuthology.orchestra.run.smithi016:> import os 2024-03-31T15:12:30.650 DEBUG:teuthology.orchestra.run.smithi016:> import subprocess 2024-03-31T15:12:30.650 DEBUG:teuthology.orchestra.run.smithi016:> 2024-03-31T15:12:30.650 DEBUG:teuthology.orchestra.run.smithi016:> def _find_admin_socket(client_name): 2024-03-31T15:12:30.650 DEBUG:teuthology.orchestra.run.smithi016:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-03-31T15:12:30.651 DEBUG:teuthology.orchestra.run.smithi016:> files = glob.glob(asok_path) 2024-03-31T15:12:30.651 DEBUG:teuthology.orchestra.run.smithi016:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-03-31T15:12:30.651 DEBUG:teuthology.orchestra.run.smithi016:> 2024-03-31T15:12:30.651 DEBUG:teuthology.orchestra.run.smithi016:> # Given a non-glob path, it better be there 2024-03-31T15:12:30.651 DEBUG:teuthology.orchestra.run.smithi016:> if "*" not in asok_path: 2024-03-31T15:12:30.651 DEBUG:teuthology.orchestra.run.smithi016:> assert(len(files) == 1) 2024-03-31T15:12:30.651 DEBUG:teuthology.orchestra.run.smithi016:> return files[0] 2024-03-31T15:12:30.651 DEBUG:teuthology.orchestra.run.smithi016:> 2024-03-31T15:12:30.651 DEBUG:teuthology.orchestra.run.smithi016:> for f in files: 2024-03-31T15:12:30.651 DEBUG:teuthology.orchestra.run.smithi016:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-31T15:12:30.651 DEBUG:teuthology.orchestra.run.smithi016:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-31T15:12:30.651 DEBUG:teuthology.orchestra.run.smithi016:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-31T15:12:30.651 DEBUG:teuthology.orchestra.run.smithi016:> contents = proc_f.read() 2024-03-31T15:12:30.652 DEBUG:teuthology.orchestra.run.smithi016:> if mountpoint in contents: 2024-03-31T15:12:30.652 DEBUG:teuthology.orchestra.run.smithi016:> return f 2024-03-31T15:12:30.652 DEBUG:teuthology.orchestra.run.smithi016:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-31T15:12:30.652 DEBUG:teuthology.orchestra.run.smithi016:> 2024-03-31T15:12:30.652 DEBUG:teuthology.orchestra.run.smithi016:> print(_find_admin_socket("client.0")) 2024-03-31T15:12:30.652 DEBUG:teuthology.orchestra.run.smithi016:> ' 2024-03-31T15:12:30.775 INFO:teuthology.orchestra.run.smithi016.stdout:/var/run/ceph/ceph-client.0.48731.asok 2024-03-31T15:12:30.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:30 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-31T15:12:30.787 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.48731.asok 2024-03-31T15:12:30.787 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:12:30.787 DEBUG:teuthology.orchestra.run.smithi016:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.48731.asok status 2024-03-31T15:12:30.960 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-03-31T15:12:30.960 INFO:teuthology.orchestra.run.smithi016.stdout: "metadata": { 2024-03-31T15:12:30.961 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph_sha1": "d2271579d5e735cfd6d436b8349c004f06527cb0", 2024-03-31T15:12:30.961 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph_version": "ceph version 19.0.0-2574-gd2271579 (d2271579d5e735cfd6d436b8349c004f06527cb0) squid (dev)", 2024-03-31T15:12:30.961 INFO:teuthology.orchestra.run.smithi016.stdout: "entity_id": "0", 2024-03-31T15:12:30.961 INFO:teuthology.orchestra.run.smithi016.stdout: "hostname": "smithi016", 2024-03-31T15:12:30.961 INFO:teuthology.orchestra.run.smithi016.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-03-31T15:12:30.961 INFO:teuthology.orchestra.run.smithi016.stdout: "pid": "48731", 2024-03-31T15:12:30.961 INFO:teuthology.orchestra.run.smithi016.stdout: "root": "/" 2024-03-31T15:12:30.961 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-31T15:12:30.961 INFO:teuthology.orchestra.run.smithi016.stdout: "dentry_count": 0, 2024-03-31T15:12:30.961 INFO:teuthology.orchestra.run.smithi016.stdout: "dentry_pinned_count": 0, 2024-03-31T15:12:30.961 INFO:teuthology.orchestra.run.smithi016.stdout: "id": 8727, 2024-03-31T15:12:30.962 INFO:teuthology.orchestra.run.smithi016.stdout: "inst": { 2024-03-31T15:12:30.962 INFO:teuthology.orchestra.run.smithi016.stdout: "name": { 2024-03-31T15:12:30.962 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "client", 2024-03-31T15:12:30.962 INFO:teuthology.orchestra.run.smithi016.stdout: "num": 8727 2024-03-31T15:12:30.962 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-31T15:12:30.962 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": { 2024-03-31T15:12:30.962 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-03-31T15:12:30.962 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "192.168.0.1:0", 2024-03-31T15:12:30.962 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2606242380 2024-03-31T15:12:30.962 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-31T15:12:30.962 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-31T15:12:30.962 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": { 2024-03-31T15:12:30.963 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-03-31T15:12:30.963 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "192.168.0.1:0", 2024-03-31T15:12:30.963 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2606242380 2024-03-31T15:12:30.963 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-31T15:12:30.963 INFO:teuthology.orchestra.run.smithi016.stdout: "inst_str": "client.8727 192.168.0.1:0/2606242380", 2024-03-31T15:12:30.963 INFO:teuthology.orchestra.run.smithi016.stdout: "addr_str": "192.168.0.1:0/2606242380", 2024-03-31T15:12:30.963 INFO:teuthology.orchestra.run.smithi016.stdout: "inode_count": 1, 2024-03-31T15:12:30.963 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_epoch": 83, 2024-03-31T15:12:30.963 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_epoch": 96, 2024-03-31T15:12:30.963 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_epoch_barrier": 0, 2024-03-31T15:12:30.963 INFO:teuthology.orchestra.run.smithi016.stdout: "blocklisted": false, 2024-03-31T15:12:30.964 INFO:teuthology.orchestra.run.smithi016.stdout: "fs_name": "cephfs" 2024-03-31T15:12:30.964 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-03-31T15:12:30.976 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:12:30.976 DEBUG:teuthology.orchestra.run.smithi016:> stat --file-system '--printf=%T 2024-03-31T15:12:30.976 DEBUG:teuthology.orchestra.run.smithi016:> ' -- /home/ubuntu/cephtest/mnt.0 2024-03-31T15:12:30.985 INFO:teuthology.orchestra.run.smithi016.stdout:fuseblk 2024-03-31T15:12:30.985 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-03-31T15:12:30.986 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:12:30.986 DEBUG:teuthology.orchestra.run.smithi016:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-31T15:12:31.048 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-03-31T15:12:31.049 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-31T15:12:31.097 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:12:31.098 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=file2_sixmegs bs=1M conv=fdatasync count=6 seek=0) 2024-03-31T15:12:31.239 INFO:teuthology.orchestra.run.smithi016.stderr:6+0 records in 2024-03-31T15:12:31.239 INFO:teuthology.orchestra.run.smithi016.stderr:6+0 records out 2024-03-31T15:12:31.240 INFO:teuthology.orchestra.run.smithi016.stderr:6291456 bytes (6.3 MB, 6.0 MiB) copied, 0.084953 s, 74.1 MB/s 2024-03-31T15:12:31.241 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-03-31T15:12:31.241 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-31T15:12:31.249 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:12:31.249 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=file3_sixmegs bs=1M conv=fdatasync count=6 seek=0) 2024-03-31T15:12:31.377 INFO:teuthology.orchestra.run.smithi016.stderr:6+0 records in 2024-03-31T15:12:31.377 INFO:teuthology.orchestra.run.smithi016.stderr:6+0 records out 2024-03-31T15:12:31.377 INFO:teuthology.orchestra.run.smithi016.stderr:6291456 bytes (6.3 MB, 6.0 MiB) copied, 0.0729856 s, 86.2 MB/s 2024-03-31T15:12:31.378 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:12:31.379 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec find ./) 2024-03-31T15:12:31.391 INFO:teuthology.orchestra.run.smithi016.stdout:./ 2024-03-31T15:12:31.392 INFO:teuthology.orchestra.run.smithi016.stdout:./file2_sixmegs 2024-03-31T15:12:31.392 INFO:teuthology.orchestra.run.smithi016.stdout:./file3_sixmegs 2024-03-31T15:12:31.392 INFO:teuthology.orchestra.run.smithi016.stdout:./file1_sixmegs 2024-03-31T15:12:31.393 DEBUG:teuthology.orchestra.run.smithi016:> stdin-killer -- python3 -c ' 2024-03-31T15:12:31.393 DEBUG:teuthology.orchestra.run.smithi016:> import os 2024-03-31T15:12:31.393 DEBUG:teuthology.orchestra.run.smithi016:> import stat 2024-03-31T15:12:31.393 DEBUG:teuthology.orchestra.run.smithi016:> 2024-03-31T15:12:31.393 DEBUG:teuthology.orchestra.run.smithi016:> print(os.stat("/home/ubuntu/cephtest/mnt.0/./").st_ino) 2024-03-31T15:12:31.393 DEBUG:teuthology.orchestra.run.smithi016:> ' 2024-03-31T15:12:31.499 INFO:teuthology.orchestra.run.smithi016.stdout:1 2024-03-31T15:12:31.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:31 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-31T15:12:31.508 DEBUG:teuthology.orchestra.run.smithi016:> stdin-killer -- python3 -c ' 2024-03-31T15:12:31.508 DEBUG:teuthology.orchestra.run.smithi016:> import os 2024-03-31T15:12:31.508 DEBUG:teuthology.orchestra.run.smithi016:> import stat 2024-03-31T15:12:31.508 DEBUG:teuthology.orchestra.run.smithi016:> 2024-03-31T15:12:31.508 DEBUG:teuthology.orchestra.run.smithi016:> print(os.stat("/home/ubuntu/cephtest/mnt.0/./file2_sixmegs").st_ino) 2024-03-31T15:12:31.508 DEBUG:teuthology.orchestra.run.smithi016:> ' 2024-03-31T15:12:31.615 INFO:teuthology.orchestra.run.smithi016.stdout:1099511627777 2024-03-31T15:12:31.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:31 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-31T15:12:31.624 DEBUG:teuthology.orchestra.run.smithi016:> stdin-killer -- python3 -c ' 2024-03-31T15:12:31.624 DEBUG:teuthology.orchestra.run.smithi016:> import os 2024-03-31T15:12:31.624 DEBUG:teuthology.orchestra.run.smithi016:> import stat 2024-03-31T15:12:31.624 DEBUG:teuthology.orchestra.run.smithi016:> 2024-03-31T15:12:31.625 DEBUG:teuthology.orchestra.run.smithi016:> print(os.stat("/home/ubuntu/cephtest/mnt.0/./file3_sixmegs").st_ino) 2024-03-31T15:12:31.625 DEBUG:teuthology.orchestra.run.smithi016:> ' 2024-03-31T15:12:31.731 INFO:teuthology.orchestra.run.smithi016.stdout:1099511627778 2024-03-31T15:12:31.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:31 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-31T15:12:31.740 DEBUG:teuthology.orchestra.run.smithi016:> stdin-killer -- python3 -c ' 2024-03-31T15:12:31.740 DEBUG:teuthology.orchestra.run.smithi016:> import os 2024-03-31T15:12:31.741 DEBUG:teuthology.orchestra.run.smithi016:> import stat 2024-03-31T15:12:31.741 DEBUG:teuthology.orchestra.run.smithi016:> 2024-03-31T15:12:31.741 DEBUG:teuthology.orchestra.run.smithi016:> print(os.stat("/home/ubuntu/cephtest/mnt.0/./file1_sixmegs").st_ino) 2024-03-31T15:12:31.741 DEBUG:teuthology.orchestra.run.smithi016:> ' 2024-03-31T15:12:31.847 INFO:teuthology.orchestra.run.smithi016.stdout:1099511627776 2024-03-31T15:12:31.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:31 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-31T15:12:31.857 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-03-31T15:12:31.857 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-31T15:12:31.911 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-31T15:12:32.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:32.103+0000 7fa2e7a00640 1 Processor -- start 2024-03-31T15:12:32.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:32.103+0000 7fa2e7a00640 1 -- start start 2024-03-31T15:12:32.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:32.103+0000 7fa2e7a00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fa2e0145d10 0x7fa2e00cf790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:32.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:32.103+0000 7fa2e7a00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fa2e00728b0 0x7fa2e00b1cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:32.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:32.103+0000 7fa2e7a00640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa2e0160e70 con 0x7fa2e00b2200 2024-03-31T15:12:32.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:32.103+0000 7fa2e7a00640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7fa2e01602f0 con 0x7fa2e00728b0 2024-03-31T15:12:32.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:32.103+0000 7fa2e7a00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fa2e01605d0 con 0x7fa2e0145d10 2024-03-31T15:12:32.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:32.103+0000 7fa2e5a00640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fa2e00b2200 0x7fa2e015cd50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:42754/0 (socket says 172.21.15.16:42754) 2024-03-31T15:12:32.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:32.103+0000 7fa2e5a00640 1 -- 172.21.15.16:0/306084349 learned_addr learned my addr 172.21.15.16:0/306084349 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:12:32.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:32.103+0000 7fa2e5000640 1 --2- 172.21.15.16:0/306084349 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fa2e0145d10 0x7fa2e00cf790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:32.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:32.103+0000 7fa2dfe00640 1 --2- 172.21.15.16:0/306084349 >> v2:172.21.15.89:3300/0 conn(0x7fa2e00728b0 0x7fa2e00b1cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:32.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:32.103+0000 7fa2df400640 1 -- 172.21.15.16:0/306084349 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3774649218 0 0) 0x7fa2e0160e70 con 0x7fa2e00b2200 2024-03-31T15:12:32.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:32.103+0000 7fa2df400640 1 -- 172.21.15.16:0/306084349 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa2c8003530 con 0x7fa2e00b2200 2024-03-31T15:12:32.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:32.103+0000 7fa2df400640 1 -- 172.21.15.16:0/306084349 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2100845235 0 0) 0x7fa2c8003530 con 0x7fa2e00b2200 2024-03-31T15:12:32.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:32.103+0000 7fa2df400640 1 -- 172.21.15.16:0/306084349 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fa2e0145d10 msgr2=0x7fa2e00cf790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:12:32.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:32.103+0000 7fa2df400640 1 --2- 172.21.15.16:0/306084349 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fa2e0145d10 0x7fa2e00cf790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:32.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:32.103+0000 7fa2df400640 1 -- 172.21.15.16:0/306084349 >> v2:172.21.15.89:3300/0 conn(0x7fa2e00728b0 msgr2=0x7fa2e00b1cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:12:32.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:32.103+0000 7fa2df400640 1 --2- 172.21.15.16:0/306084349 >> v2:172.21.15.89:3300/0 conn(0x7fa2e00728b0 0x7fa2e00b1cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:32.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:32.103+0000 7fa2df400640 1 -- 172.21.15.16:0/306084349 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa2e00cfcd0 con 0x7fa2e00b2200 2024-03-31T15:12:32.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:32.103+0000 7fa2df400640 1 -- 172.21.15.16:0/306084349 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 214807623 0 0) 0x7fa2d8001a00 con 0x7fa2e00b2200 2024-03-31T15:12:32.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:32.103+0000 7fa2df400640 1 -- 172.21.15.16:0/306084349 >> v1:172.21.15.16:6789/0 conn(0x7fa2e00b2200 legacy=0x7fa2e015cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:32.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:32.103+0000 7fa2df400640 1 --2- 172.21.15.16:0/306084349 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa2c80039c0 0x7fa2c8003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:32.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:32.103+0000 7fa2df400640 1 --2- 172.21.15.16:0/306084349 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa2c8004810 0x7fa2c8004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:32.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:32.103+0000 7fa2df400640 1 --2- 172.21.15.16:0/306084349 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fa2e0145d10 0x7fa2c80051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:32.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:32.103+0000 7fa2df400640 1 -- 172.21.15.16:0/306084349 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fa2e01602f0 con 0x7fa2c8004810 2024-03-31T15:12:32.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:32.103+0000 7fa2df400640 1 -- 172.21.15.16:0/306084349 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fa2e01605d0 con 0x7fa2c80039c0 2024-03-31T15:12:32.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:32.103+0000 7fa2df400640 1 -- 172.21.15.16:0/306084349 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fa2c800fde0 con 0x7fa2e0145d10 2024-03-31T15:12:32.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:32.107+0000 7fa2dfe00640 1 --2- 172.21.15.16:0/306084349 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa2c8004810 0x7fa2c8004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:32.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:32.107+0000 7fa2e5000640 1 --2- 172.21.15.16:0/306084349 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa2c80039c0 0x7fa2c8003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:32.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:32.107+0000 7fa2e5a00640 1 --2- 172.21.15.16:0/306084349 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fa2e0145d10 0x7fa2c80051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:32.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:32.107+0000 7fa2dfe00640 1 -- 172.21.15.16:0/306084349 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fa2e0145d10 msgr2=0x7fa2c80051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:12:32.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:32.107+0000 7fa2dfe00640 1 --2- 172.21.15.16:0/306084349 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fa2e0145d10 0x7fa2c80051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:32.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:32.107+0000 7fa2dfe00640 1 -- 172.21.15.16:0/306084349 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa2c80039c0 msgr2=0x7fa2c8003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:12:32.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:32.107+0000 7fa2dfe00640 1 --2- 172.21.15.16:0/306084349 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa2c80039c0 0x7fa2c8003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:32.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:32.107+0000 7fa2dfe00640 1 -- 172.21.15.16:0/306084349 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa2c8003530 con 0x7fa2c8004810 2024-03-31T15:12:32.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:32.107+0000 7fa2e5a00640 1 --2- 172.21.15.16:0/306084349 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fa2e0145d10 0x7fa2c80051d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:12:32.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:32.107+0000 7fa2dfe00640 1 --2- 172.21.15.16:0/306084349 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa2c8004810 0x7fa2c8004c90 secure :-1 s=READY pgs=1465 cs=0 l=1 rev1=1 crypto rx=0x7fa2cc009ae0 tx=0x7fa2cc01ca50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7eb229444a9fc581 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:12:32.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:32.107+0000 7fa2df400640 1 -- 172.21.15.16:0/306084349 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2cc0b2070 con 0x7fa2c8004810 2024-03-31T15:12:32.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:32.107+0000 7fa2df400640 1 -- 172.21.15.16:0/306084349 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa2cc0a55c0 con 0x7fa2c8004810 2024-03-31T15:12:32.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:32.107+0000 7fa2df400640 1 -- 172.21.15.16:0/306084349 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2cc0096b0 con 0x7fa2c8004810 2024-03-31T15:12:32.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:32.111+0000 7fa2e7a00640 1 -- 172.21.15.16:0/306084349 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa2c8004810 msgr2=0x7fa2c8004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:32.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:32.111+0000 7fa2e7a00640 1 --2- 172.21.15.16:0/306084349 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa2c8004810 0x7fa2c8004c90 secure :-1 s=READY pgs=1465 cs=0 l=1 rev1=1 crypto rx=0x7fa2cc009ae0 tx=0x7fa2cc01ca50 comp rx=0 tx=0).stop 2024-03-31T15:12:32.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:32.111+0000 7fa2e5a00640 1 -- 172.21.15.16:0/306084349 reap_dead start 2024-03-31T15:12:32.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:32.111+0000 7fa2e7a00640 1 -- 172.21.15.16:0/306084349 shutdown_connections 2024-03-31T15:12:32.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:32.111+0000 7fa2e7a00640 1 -- 172.21.15.16:0/306084349 >> 172.21.15.16:0/306084349 conn(0x7fa2e006b620 msgr2=0x7fa2e006ba30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:12:32.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:32.111+0000 7fa2e7a00640 1 -- 172.21.15.16:0/306084349 shutdown_connections 2024-03-31T15:12:32.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:32.111+0000 7fa2e7a00640 1 -- 172.21.15.16:0/306084349 wait complete. 2024-03-31T15:12:32.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:32.111+0000 7fa2e7a00640 1 Processor -- start 2024-03-31T15:12:32.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:32.111+0000 7fa2e7a00640 1 -- start start 2024-03-31T15:12:32.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:32.115+0000 7fa2e7a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa2c8004810 0x7fa2e0141980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:32.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:32.115+0000 7fa2e7a00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fa2e0159140 0x7fa2e013cb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:32.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:32.115+0000 7fa2e7a00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa2e013d070 0x7fa2e015db90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:32.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:32.115+0000 7fa2e7a00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fa2e015ff80 con 0x7fa2c8004810 2024-03-31T15:12:32.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:32.115+0000 7fa2e7a00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fa2e0160b90 con 0x7fa2e013d070 2024-03-31T15:12:32.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:32.115+0000 7fa2e7a00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fa2e00cf350 con 0x7fa2e0159140 2024-03-31T15:12:32.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:32.115+0000 7fa2e5000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa2c8004810 0x7fa2e0141980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:32.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:32.115+0000 7fa2e5000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa2c8004810 0x7fa2e0141980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41204/0 (socket says 172.21.15.16:41204) 2024-03-31T15:12:32.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:32.115+0000 7fa2e5000640 1 -- 172.21.15.16:0/2839258588 learned_addr learned my addr 172.21.15.16:0/2839258588 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:12:32.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:32.115+0000 7fa2dfe00640 1 --2- 172.21.15.16:0/2839258588 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fa2e0159140 0x7fa2e013cb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:32.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:32.115+0000 7fa2e5a00640 1 --2- 172.21.15.16:0/2839258588 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa2e013d070 0x7fa2e015db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:32.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:32.115+0000 7fa2e5000640 1 -- 172.21.15.16:0/2839258588 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fa2e0159140 msgr2=0x7fa2e013cb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:32.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:32.115+0000 7fa2e5000640 1 --2- 172.21.15.16:0/2839258588 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fa2e0159140 0x7fa2e013cb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:32.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:32.115+0000 7fa2e5000640 1 -- 172.21.15.16:0/2839258588 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa2e013d070 msgr2=0x7fa2e015db90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:32.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:32.115+0000 7fa2e5000640 1 --2- 172.21.15.16:0/2839258588 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa2e013d070 0x7fa2e015db90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:32.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:32.115+0000 7fa2e5000640 1 -- 172.21.15.16:0/2839258588 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa2e015cdd0 con 0x7fa2c8004810 2024-03-31T15:12:32.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:32.115+0000 7fa2e5000640 1 --2- 172.21.15.16:0/2839258588 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa2c8004810 0x7fa2e0141980 secure :-1 s=READY pgs=1466 cs=0 l=1 rev1=1 crypto rx=0x7fa2d4011a00 tx=0x7fa2d4011ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:12:32.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:32.115+0000 7fa2dcc00640 1 -- 172.21.15.16:0/2839258588 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2d400fb50 con 0x7fa2c8004810 2024-03-31T15:12:32.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:32.115+0000 7fa2dcc00640 1 -- 172.21.15.16:0/2839258588 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa2d400fcf0 con 0x7fa2c8004810 2024-03-31T15:12:32.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:32.115+0000 7fa2dcc00640 1 -- 172.21.15.16:0/2839258588 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2d400fea0 con 0x7fa2c8004810 2024-03-31T15:12:32.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:32.115+0000 7fa2e7a00640 1 -- 172.21.15.16:0/2839258588 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa2e015d0b0 con 0x7fa2c8004810 2024-03-31T15:12:32.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:32.115+0000 7fa2e7a00640 1 -- 172.21.15.16:0/2839258588 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa2e0159b90 con 0x7fa2c8004810 2024-03-31T15:12:32.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:32.115+0000 7fa2e7a00640 1 -- 172.21.15.16:0/2839258588 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa2e0069950 con 0x7fa2c8004810 2024-03-31T15:12:32.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:32.119+0000 7fa2dcc00640 1 -- 172.21.15.16:0/2839258588 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fa2d4012040 con 0x7fa2c8004810 2024-03-31T15:12:32.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:32.119+0000 7fa2dcc00640 1 --2- 172.21.15.16:0/2839258588 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fa2c00683b0 0x7fa2c006a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:32.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:32.119+0000 7fa2dcc00640 1 -- 172.21.15.16:0/2839258588 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 5709+0+0 (secure 0 0 0) 0x7fa2d4086ff0 con 0x7fa2c8004810 2024-03-31T15:12:32.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:32.119+0000 7fa2dfe00640 1 --2- 172.21.15.16:0/2839258588 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fa2c00683b0 0x7fa2c006a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:32.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:32.119+0000 7fa2dfe00640 1 --2- 172.21.15.16:0/2839258588 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fa2c00683b0 0x7fa2c006a870 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7fa2cc0af000 tx=0x7fa2cc0046d0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:12:32.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:32.123+0000 7fa2dcc00640 1 -- 172.21.15.16:0/2839258588 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fa2d4019030 con 0x7fa2c8004810 2024-03-31T15:12:32.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:32.299+0000 7fa2e7a00640 1 -- 172.21.15.16:0/2839258588 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fa2e013d450 con 0x7fa2c8004810 2024-03-31T15:12:32.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:32.299+0000 7fa2dcc00640 1 -- 172.21.15.16:0/2839258588 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v96) ==== 74+0+16536 (secure 0 0 0) 0x7fa2d4057540 con 0x7fa2c8004810 2024-03-31T15:12:32.293 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:12:32.293 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":96,"fsid":"286eb5da-fb47-4550-b290-d9519c13b9a5","created":"2024-03-31T15:03:11.438952+0000","modified":"2024-03-31T15:12:17.885852+0000","last_up_change":"2024-03-31T15:03:22.885535+0000","last_in_change":"2024-03-31T15:03:13.941667+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-31T15:03:22.550787+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2024-03-31T15:12:14.671225+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"95","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2024-03-31T15:12:15.242196+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"96","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"92a66450-ef19-40d8-ac3d-d760b23b0bda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6801","nonce":932712244}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6803","nonce":932712244}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6807","nonce":932712244}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6805","nonce":932712244}]},"public_addr":"172.21.15.16:6801/932712244","cluster_addr":"172.21.15.16:6803/932712244","heartbeat_back_addr":"172.21.15.16:6807/932712244","heartbeat_front_addr":"172.21.15.16:6805/932712244","state":["exists","up"]},{"osd":1,"uuid":"da9d1f2b-1968-4959-b843-faa231988e92","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6810","nonce":1398737533}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6811","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6813","nonce":1398737533}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6821","nonce":1398737533}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6815","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6816","nonce":1398737533}]},"public_addr":"172.21.15.16:6810/1398737533","cluster_addr":"172.21.15.16:6813/1398737533","heartbeat_back_addr":"172.21.15.16:6821/1398737533","heartbeat_front_addr":"172.21.15.16:6816/1398737533","state":["exists","up"]},{"osd":2,"uuid":"01a73662-9a21-4b1f-bcbb-42b9636a4638","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6819","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6822","nonce":3018526646}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6825","nonce":3018526646}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6831","nonce":3018526646}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6827","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6829","nonce":3018526646}]},"public_addr":"172.21.15.16:6822/3018526646","cluster_addr":"172.21.15.16:6825/3018526646","heartbeat_back_addr":"172.21.15.16:6831/3018526646","heartbeat_front_addr":"172.21.15.16:6829/3018526646","state":["exists","up"]},{"osd":3,"uuid":"c7771fd3-e18e-4500-ac38-23cf10ebcd75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6809","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6812","nonce":3360352974}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6817","nonce":3360352974}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6828","nonce":3360352974}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6823","nonce":3360352974}]},"public_addr":"172.21.15.16:6812/3360352974","cluster_addr":"172.21.15.16:6817/3360352974","heartbeat_back_addr":"172.21.15.16:6828/3360352974","heartbeat_front_addr":"172.21.15.16:6823/3360352974","state":["exists","up"]},{"osd":4,"uuid":"88260730-1482-4fbd-a40f-cb1124e9df4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6801","nonce":1856844856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6803","nonce":1856844856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6807","nonce":1856844856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6805","nonce":1856844856}]},"public_addr":"172.21.15.89:6801/1856844856","cluster_addr":"172.21.15.89:6803/1856844856","heartbeat_back_addr":"172.21.15.89:6807/1856844856","heartbeat_front_addr":"172.21.15.89:6805/1856844856","state":["exists","up"]},{"osd":5,"uuid":"4cb2c00c-fae0-4dd8-9d27-ec4f0093ccf7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6814","nonce":50395527}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6817","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6820","nonce":50395527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6831","nonce":50395527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6823","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6826","nonce":50395527}]},"public_addr":"172.21.15.89:6814/50395527","cluster_addr":"172.21.15.89:6820/50395527","heartbeat_back_addr":"172.21.15.89:6831/50395527","heartbeat_front_addr":"172.21.15.89:6826/50395527","state":["exists","up"]},{"osd":6,"uuid":"81ce0c19-2554-4cae-acbe-66a05e9e9b9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6809","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6812","nonce":3470471752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6815","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6818","nonce":3470471752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6827","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6829","nonce":3470471752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6821","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6824","nonce":3470471752}]},"public_addr":"172.21.15.89:6812/3470471752","cluster_addr":"172.21.15.89:6818/3470471752","heartbeat_back_addr":"172.21.15.89:6829/3470471752","heartbeat_front_addr":"172.21.15.89:6824/3470471752","state":["exists","up"]},{"osd":7,"uuid":"85e4313b-d37d-442e-8042-153bf20a847d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6811","nonce":1796778630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6813","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6816","nonce":1796778630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6825","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6828","nonce":1796778630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6819","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6822","nonce":1796778630}]},"public_addr":"172.21.15.89:6811/1796778630","cluster_addr":"172.21.15.89:6816/1796778630","heartbeat_back_addr":"172.21.15.89:6828/1796778630","heartbeat_front_addr":"172.21.15.89:6822/1796778630","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.439088+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.529970+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.680512+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:21.044161+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.582111+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.536558+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.409522+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.590229+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-31T15:12:32.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:32.303+0000 7fa2e7a00640 1 -- 172.21.15.16:0/2839258588 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fa2c00683b0 msgr2=0x7fa2c006a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:32.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:32.303+0000 7fa2e7a00640 1 --2- 172.21.15.16:0/2839258588 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fa2c00683b0 0x7fa2c006a870 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7fa2cc0af000 tx=0x7fa2cc0046d0 comp rx=0 tx=0).stop 2024-03-31T15:12:32.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:32.303+0000 7fa2e7a00640 1 -- 172.21.15.16:0/2839258588 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa2c8004810 msgr2=0x7fa2e0141980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:32.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:32.303+0000 7fa2e7a00640 1 --2- 172.21.15.16:0/2839258588 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa2c8004810 0x7fa2e0141980 secure :-1 s=READY pgs=1466 cs=0 l=1 rev1=1 crypto rx=0x7fa2d4011a00 tx=0x7fa2d4011ec0 comp rx=0 tx=0).stop 2024-03-31T15:12:32.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:32.303+0000 7fa2e7a00640 1 -- 172.21.15.16:0/2839258588 shutdown_connections 2024-03-31T15:12:32.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:32.303+0000 7fa2e7a00640 1 --2- 172.21.15.16:0/2839258588 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fa2c00683b0 0x7fa2c006a870 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:32.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:32.303+0000 7fa2e7a00640 1 --2- 172.21.15.16:0/2839258588 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa2e013d070 0x7fa2e015db90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:32.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:32.303+0000 7fa2e7a00640 1 --2- 172.21.15.16:0/2839258588 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fa2e0159140 0x7fa2e013cb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:32.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:32.303+0000 7fa2e7a00640 1 --2- 172.21.15.16:0/2839258588 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa2c8004810 0x7fa2e0141980 unknown :-1 s=CLOSED pgs=1466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:32.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:32.303+0000 7fa2e7a00640 1 -- 172.21.15.16:0/2839258588 >> 172.21.15.16:0/2839258588 conn(0x7fa2e006b620 msgr2=0x7fa2e00afba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:12:32.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:32.303+0000 7fa2e7a00640 1 -- 172.21.15.16:0/2839258588 shutdown_connections 2024-03-31T15:12:32.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:32.303+0000 7fa2e7a00640 1 -- 172.21.15.16:0/2839258588 wait complete. 2024-03-31T15:12:32.315 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-31T15:03:22.550787+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 14, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-31T15:12:14.671225+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '95', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.5, 'score_stable': 2.5, 'optimal_score': 1, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 15, 'pool_name': 'cephfs_data', 'create_time': '2024-03-31T15:12:15.242196+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '96', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-31T15:12:32.315 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-31T15:12:32.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:32.467+0000 7fa7e8400640 1 Processor -- start 2024-03-31T15:12:32.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:32.467+0000 7fa7e8400640 1 -- start start 2024-03-31T15:12:32.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:32.467+0000 7fa7e8400640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fa7e01411b0 0x7fa7e006f920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:32.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:32.467+0000 7fa7e8400640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fa7e0141590 0x7fa7e006fe60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:32.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:32.467+0000 7fa7e8400640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fa7e0143c30 0x7fa7e01440a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:32.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:32.467+0000 7fa7e8400640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7fa7e0162910 con 0x7fa7e01411b0 2024-03-31T15:12:32.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:32.467+0000 7fa7e8400640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7fa7e0144980 con 0x7fa7e0143c30 2024-03-31T15:12:32.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:32.467+0000 7fa7e8400640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fa7e0161fe0 con 0x7fa7e0141590 2024-03-31T15:12:32.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:32.467+0000 7fa7e5a00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fa7e01411b0 0x7fa7e006f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:32.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:32.467+0000 7fa7e5a00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fa7e01411b0 0x7fa7e006f920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41212/0 (socket says 172.21.15.16:41212) 2024-03-31T15:12:32.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:32.467+0000 7fa7e5a00640 1 -- 172.21.15.16:0/2772485401 learned_addr learned my addr 172.21.15.16:0/2772485401 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:12:32.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:32.467+0000 7fa7e6400640 1 --2- 172.21.15.16:0/2772485401 >> v2:172.21.15.89:3300/0 conn(0x7fa7e0143c30 0x7fa7e01440a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:32.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:32.467+0000 7fa7e5000640 1 --2- 172.21.15.16:0/2772485401 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fa7e0141590 0x7fa7e006fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:32.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:32.467+0000 7fa7e5a00640 1 -- 172.21.15.16:0/2772485401 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fa7e0141590 msgr2=0x7fa7e006fe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:12:32.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:32.467+0000 7fa7e5a00640 1 --2- 172.21.15.16:0/2772485401 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fa7e0141590 0x7fa7e006fe60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:32.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:32.467+0000 7fa7e5a00640 1 -- 172.21.15.16:0/2772485401 >> v2:172.21.15.89:3300/0 conn(0x7fa7e0143c30 msgr2=0x7fa7e01440a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:12:32.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:32.467+0000 7fa7e5a00640 1 --2- 172.21.15.16:0/2772485401 >> v2:172.21.15.89:3300/0 conn(0x7fa7e0143c30 0x7fa7e01440a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:32.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:32.467+0000 7fa7e5a00640 1 -- 172.21.15.16:0/2772485401 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7e01622c0 con 0x7fa7e01411b0 2024-03-31T15:12:32.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:32.471+0000 7fa7e5a00640 1 --2- 172.21.15.16:0/2772485401 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa7e01411b0 0x7fa7e006f920 secure :-1 s=READY pgs=1467 cs=0 l=1 rev1=1 crypto rx=0x7fa7e00b1c00 tx=0x7fa7d401c4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5345b430bd5c852e server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:12:32.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:32.471+0000 7fa7dfe00640 1 -- 172.21.15.16:0/2772485401 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7d40ab640 con 0x7fa7e01411b0 2024-03-31T15:12:32.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:32.471+0000 7fa7dfe00640 1 -- 172.21.15.16:0/2772485401 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa7d40b0070 con 0x7fa7e01411b0 2024-03-31T15:12:32.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:32.471+0000 7fa7dfe00640 1 -- 172.21.15.16:0/2772485401 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7d40aba00 con 0x7fa7e01411b0 2024-03-31T15:12:32.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:32.471+0000 7fa7e8400640 1 -- 172.21.15.16:0/2772485401 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa7e01411b0 msgr2=0x7fa7e006f920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:32.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:32.471+0000 7fa7e8400640 1 --2- 172.21.15.16:0/2772485401 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa7e01411b0 0x7fa7e006f920 secure :-1 s=READY pgs=1467 cs=0 l=1 rev1=1 crypto rx=0x7fa7e00b1c00 tx=0x7fa7d401c4f0 comp rx=0 tx=0).stop 2024-03-31T15:12:32.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:32.471+0000 7fa7e8400640 1 -- 172.21.15.16:0/2772485401 shutdown_connections 2024-03-31T15:12:32.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:32.471+0000 7fa7e8400640 1 --2- 172.21.15.16:0/2772485401 >> v2:172.21.15.89:3300/0 conn(0x7fa7e0143c30 0x7fa7e01440a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:32.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:32.471+0000 7fa7e8400640 1 --2- 172.21.15.16:0/2772485401 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fa7e0141590 0x7fa7e006fe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:32.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:32.471+0000 7fa7e8400640 1 --2- 172.21.15.16:0/2772485401 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa7e01411b0 0x7fa7e006f920 unknown :-1 s=CLOSED pgs=1467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:32.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:32.471+0000 7fa7e8400640 1 -- 172.21.15.16:0/2772485401 >> 172.21.15.16:0/2772485401 conn(0x7fa7e0067930 msgr2=0x7fa7e00b3ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:12:32.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:32.471+0000 7fa7e8400640 1 -- 172.21.15.16:0/2772485401 shutdown_connections 2024-03-31T15:12:32.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:32.471+0000 7fa7e8400640 1 -- 172.21.15.16:0/2772485401 wait complete. 2024-03-31T15:12:32.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:32.471+0000 7fa7e8400640 1 Processor -- start 2024-03-31T15:12:32.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:32.475+0000 7fa7e8400640 1 -- start start 2024-03-31T15:12:32.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:32.475+0000 7fa7e8400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa7e01411b0 0x7fa7e0156710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:32.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:32.475+0000 7fa7e8400640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa7e0141590 0x7fa7e0161400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:32.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:32.475+0000 7fa7e8400640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fa7e0143c30 0x7fa7e006c730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:32.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:32.475+0000 7fa7e8400640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fa7e00717f0 con 0x7fa7e01411b0 2024-03-31T15:12:32.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:32.475+0000 7fa7e8400640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fa7e0071670 con 0x7fa7e0141590 2024-03-31T15:12:32.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:32.475+0000 7fa7e8400640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fa7e0071970 con 0x7fa7e0143c30 2024-03-31T15:12:32.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:32.475+0000 7fa7e5a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa7e01411b0 0x7fa7e0156710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:32.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:32.475+0000 7fa7e5a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa7e01411b0 0x7fa7e0156710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41220/0 (socket says 172.21.15.16:41220) 2024-03-31T15:12:32.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:32.475+0000 7fa7e5a00640 1 -- 172.21.15.16:0/938052921 learned_addr learned my addr 172.21.15.16:0/938052921 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:12:32.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:32.475+0000 7fa7e6400640 1 --2- 172.21.15.16:0/938052921 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fa7e0143c30 0x7fa7e006c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:32.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:32.475+0000 7fa7e5000640 1 --2- 172.21.15.16:0/938052921 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa7e0141590 0x7fa7e0161400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:32.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:32.475+0000 7fa7e5a00640 1 -- 172.21.15.16:0/938052921 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fa7e0143c30 msgr2=0x7fa7e006c730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:32.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:32.475+0000 7fa7e5a00640 1 --2- 172.21.15.16:0/938052921 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fa7e0143c30 0x7fa7e006c730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:32.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:32.475+0000 7fa7e5a00640 1 -- 172.21.15.16:0/938052921 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa7e0141590 msgr2=0x7fa7e0161400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:32.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:32.475+0000 7fa7e5a00640 1 --2- 172.21.15.16:0/938052921 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa7e0141590 0x7fa7e0161400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:32.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:32.475+0000 7fa7e5a00640 1 -- 172.21.15.16:0/938052921 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7e015eaa0 con 0x7fa7e01411b0 2024-03-31T15:12:32.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:32.475+0000 7fa7e6400640 1 --2- 172.21.15.16:0/938052921 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fa7e0143c30 0x7fa7e006c730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:12:32.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:32.475+0000 7fa7e5000640 1 --2- 172.21.15.16:0/938052921 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa7e0141590 0x7fa7e0161400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:12:32.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:32.475+0000 7fa7e5a00640 1 --2- 172.21.15.16:0/938052921 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa7e01411b0 0x7fa7e0156710 secure :-1 s=READY pgs=1468 cs=0 l=1 rev1=1 crypto rx=0x7fa7d40ad040 tx=0x7fa7d401ad30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:12:32.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:32.475+0000 7fa7dd600640 1 -- 172.21.15.16:0/938052921 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7d40be3f0 con 0x7fa7e01411b0 2024-03-31T15:12:32.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:32.475+0000 7fa7dd600640 1 -- 172.21.15.16:0/938052921 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa7d40be590 con 0x7fa7e01411b0 2024-03-31T15:12:32.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:32.475+0000 7fa7dd600640 1 -- 172.21.15.16:0/938052921 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7d40be960 con 0x7fa7e01411b0 2024-03-31T15:12:32.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:32.475+0000 7fa7e8400640 1 -- 172.21.15.16:0/938052921 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa7e0161b70 con 0x7fa7e01411b0 2024-03-31T15:12:32.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:32.475+0000 7fa7e8400640 1 -- 172.21.15.16:0/938052921 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa7e013cae0 con 0x7fa7e01411b0 2024-03-31T15:12:32.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:32.475+0000 7fa7e8400640 1 -- 172.21.15.16:0/938052921 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa7e0069950 con 0x7fa7e01411b0 2024-03-31T15:12:32.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:32.479+0000 7fa7dd600640 1 -- 172.21.15.16:0/938052921 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fa7d40b43f0 con 0x7fa7e01411b0 2024-03-31T15:12:32.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:32.479+0000 7fa7dd600640 1 --2- 172.21.15.16:0/938052921 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fa7c0068380 0x7fa7c006a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:32.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:32.479+0000 7fa7dd600640 1 -- 172.21.15.16:0/938052921 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 5709+0+0 (secure 0 0 0) 0x7fa7d4121090 con 0x7fa7e01411b0 2024-03-31T15:12:32.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:32.479+0000 7fa7e5000640 1 --2- 172.21.15.16:0/938052921 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fa7c0068380 0x7fa7c006a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:32.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:32.479+0000 7fa7e5000640 1 --2- 172.21.15.16:0/938052921 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fa7c0068380 0x7fa7c006a840 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7fa7e006d650 tx=0x7fa7cc008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:12:32.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:32.483+0000 7fa7dd600640 1 -- 172.21.15.16:0/938052921 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fa7d40c4060 con 0x7fa7e01411b0 2024-03-31T15:12:32.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:32.783+0000 7fa7e8400640 1 -- 172.21.15.16:0/938052921 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fa7e00b4a20 con 0x7fa7e01411b0 2024-03-31T15:12:32.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:32.783+0000 7fa7dd600640 1 -- 172.21.15.16:0/938052921 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v96) ==== 93+0+10 (secure 0 0 0) 0x7fa7d4016110 con 0x7fa7e01411b0 2024-03-31T15:12:32.776 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2024-03-31T15:12:32.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:32.783+0000 7fa7e8400640 1 -- 172.21.15.16:0/938052921 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fa7c0068380 msgr2=0x7fa7c006a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:32.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:32.783+0000 7fa7e8400640 1 --2- 172.21.15.16:0/938052921 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fa7c0068380 0x7fa7c006a840 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7fa7e006d650 tx=0x7fa7cc008040 comp rx=0 tx=0).stop 2024-03-31T15:12:32.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:32.783+0000 7fa7e8400640 1 -- 172.21.15.16:0/938052921 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa7e01411b0 msgr2=0x7fa7e0156710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:32.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:32.783+0000 7fa7e8400640 1 --2- 172.21.15.16:0/938052921 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa7e01411b0 0x7fa7e0156710 secure :-1 s=READY pgs=1468 cs=0 l=1 rev1=1 crypto rx=0x7fa7d40ad040 tx=0x7fa7d401ad30 comp rx=0 tx=0).stop 2024-03-31T15:12:32.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:32.783+0000 7fa7e8400640 1 -- 172.21.15.16:0/938052921 shutdown_connections 2024-03-31T15:12:32.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:32.783+0000 7fa7e8400640 1 --2- 172.21.15.16:0/938052921 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fa7c0068380 0x7fa7c006a840 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:32.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:32.783+0000 7fa7e8400640 1 --2- 172.21.15.16:0/938052921 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fa7e0143c30 0x7fa7e006c730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:32.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:32.783+0000 7fa7e8400640 1 --2- 172.21.15.16:0/938052921 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa7e0141590 0x7fa7e0161400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:32.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:32.783+0000 7fa7e8400640 1 --2- 172.21.15.16:0/938052921 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa7e01411b0 0x7fa7e0156710 unknown :-1 s=CLOSED pgs=1468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:32.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:32.783+0000 7fa7e8400640 1 -- 172.21.15.16:0/938052921 >> 172.21.15.16:0/938052921 conn(0x7fa7e0067930 msgr2=0x7fa7e015da70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:12:32.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:32.787+0000 7fa7e8400640 1 -- 172.21.15.16:0/938052921 shutdown_connections 2024-03-31T15:12:32.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:32.787+0000 7fa7e8400640 1 -- 172.21.15.16:0/938052921 wait complete. 2024-03-31T15:12:32.807 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-31T15:12:32.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:32.963+0000 7f9718400640 1 Processor -- start 2024-03-31T15:12:32.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:32.963+0000 7f9718400640 1 -- start start 2024-03-31T15:12:32.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:32.963+0000 7f9718400640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f971006f920 0x7f971006fd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:32.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:32.963+0000 7f9718400640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9710070240 con 0x7f97100cf150 2024-03-31T15:12:32.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:32.963+0000 7f9718400640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f971015ca70 con 0x7f97100d0a60 2024-03-31T15:12:32.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:32.963+0000 7f9718400640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f971013f670 con 0x7f971006f920 2024-03-31T15:12:32.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:32.963+0000 7f9715a00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f971006f920 0x7f971006fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:32.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:32.963+0000 7f9716400640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f97100cf150 0x7f97101419c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:42768/0 (socket says 172.21.15.16:42768) 2024-03-31T15:12:32.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:32.963+0000 7f9716400640 1 -- 172.21.15.16:0/3990039679 learned_addr learned my addr 172.21.15.16:0/3990039679 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:12:32.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:32.963+0000 7f9715a00640 1 -- 172.21.15.16:0/3990039679 >> v1:172.21.15.89:6789/0 conn(0x7f97100d0a60 legacy=0x7f97100cea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:32.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:32.963+0000 7f9715a00640 1 -- 172.21.15.16:0/3990039679 >> v1:172.21.15.16:6789/0 conn(0x7f97100cf150 legacy=0x7f97101419c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:32.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:32.963+0000 7f9715a00640 1 -- 172.21.15.16:0/3990039679 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f971015cd50 con 0x7f971006f920 2024-03-31T15:12:32.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:32.967+0000 7f9715a00640 1 --2- 172.21.15.16:0/3990039679 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f971006f920 0x7f971006fd00 secure :-1 s=READY pgs=1470 cs=0 l=1 rev1=1 crypto rx=0x7f97040097c0 tx=0x7f970401c410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c4c933c3d7a95b5a server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:12:32.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:32.967+0000 7f970fe00640 1 -- 172.21.15.16:0/3990039679 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97040b0070 con 0x7f971006f920 2024-03-31T15:12:32.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:32.967+0000 7f970fe00640 1 -- 172.21.15.16:0/3990039679 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f97040ab600 con 0x7f971006f920 2024-03-31T15:12:32.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:32.967+0000 7f970fe00640 1 -- 172.21.15.16:0/3990039679 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97040ab9d0 con 0x7f971006f920 2024-03-31T15:12:32.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:32.967+0000 7f9718400640 1 -- 172.21.15.16:0/3990039679 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f971006f920 msgr2=0x7f971006fd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:32.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:32.967+0000 7f9718400640 1 --2- 172.21.15.16:0/3990039679 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f971006f920 0x7f971006fd00 secure :-1 s=READY pgs=1470 cs=0 l=1 rev1=1 crypto rx=0x7f97040097c0 tx=0x7f970401c410 comp rx=0 tx=0).stop 2024-03-31T15:12:32.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:32.967+0000 7f9718400640 1 -- 172.21.15.16:0/3990039679 shutdown_connections 2024-03-31T15:12:32.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:32.967+0000 7f9718400640 1 --2- 172.21.15.16:0/3990039679 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f971006f920 0x7f971006fd00 unknown :-1 s=CLOSED pgs=1470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:32.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:32.967+0000 7f9718400640 1 -- 172.21.15.16:0/3990039679 >> 172.21.15.16:0/3990039679 conn(0x7f9710067930 msgr2=0x7f97100b3ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:12:32.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:32.967+0000 7f9718400640 1 -- 172.21.15.16:0/3990039679 shutdown_connections 2024-03-31T15:12:32.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:32.967+0000 7f9718400640 1 -- 172.21.15.16:0/3990039679 wait complete. 2024-03-31T15:12:32.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:32.971+0000 7f9718400640 1 Processor -- start 2024-03-31T15:12:32.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:32.971+0000 7f9718400640 1 -- start start 2024-03-31T15:12:32.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:32.971+0000 7f9718400640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f971006f920 0x7f971013d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:32.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:32.971+0000 7f9718400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f97100cf150 0x7f97101556d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:32.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:32.971+0000 7f9718400640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f97100d0a60 0x7f97100b5590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:32.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:32.971+0000 7f9718400640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f971016a240 con 0x7f97100cf150 2024-03-31T15:12:32.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:32.971+0000 7f9718400640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f971016a0c0 con 0x7f97100d0a60 2024-03-31T15:12:32.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:32.971+0000 7f9718400640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f971016a3c0 con 0x7f971006f920 2024-03-31T15:12:32.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:32.971+0000 7f9715000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f97100cf150 0x7f97101556d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:32.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:32.971+0000 7f9715000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f97100cf150 0x7f97101556d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41230/0 (socket says 172.21.15.16:41230) 2024-03-31T15:12:32.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:32.971+0000 7f9715000640 1 -- 172.21.15.16:0/2431116418 learned_addr learned my addr 172.21.15.16:0/2431116418 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:12:32.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:32.971+0000 7f9715a00640 1 --2- 172.21.15.16:0/2431116418 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f971006f920 0x7f971013d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:32.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:32.971+0000 7f9716400640 1 --2- 172.21.15.16:0/2431116418 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f97100d0a60 0x7f97100b5590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:32.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:32.971+0000 7f9715000640 1 -- 172.21.15.16:0/2431116418 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f971006f920 msgr2=0x7f971013d870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:32.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:32.971+0000 7f9715000640 1 --2- 172.21.15.16:0/2431116418 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f971006f920 0x7f971013d870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:32.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:32.971+0000 7f9715000640 1 -- 172.21.15.16:0/2431116418 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f97100d0a60 msgr2=0x7f97100b5590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:32.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:32.971+0000 7f9715000640 1 --2- 172.21.15.16:0/2431116418 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f97100d0a60 0x7f97100b5590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:32.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:32.971+0000 7f9715000640 1 -- 172.21.15.16:0/2431116418 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9710154970 con 0x7f97100cf150 2024-03-31T15:12:32.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:32.971+0000 7f9715000640 1 --2- 172.21.15.16:0/2431116418 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f97100cf150 0x7f97101556d0 secure :-1 s=READY pgs=1471 cs=0 l=1 rev1=1 crypto rx=0x7f96fc00c370 tx=0x7f96fc00c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:12:32.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:32.971+0000 7f970d600640 1 -- 172.21.15.16:0/2431116418 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96fc0052d0 con 0x7f97100cf150 2024-03-31T15:12:32.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:32.971+0000 7f970d600640 1 -- 172.21.15.16:0/2431116418 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f96fc005470 con 0x7f97100cf150 2024-03-31T15:12:32.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:32.971+0000 7f9718400640 1 -- 172.21.15.16:0/2431116418 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9710156af0 con 0x7f97100cf150 2024-03-31T15:12:32.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:32.971+0000 7f970d600640 1 -- 172.21.15.16:0/2431116418 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96fc007260 con 0x7f97100cf150 2024-03-31T15:12:32.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:32.971+0000 7f9718400640 1 -- 172.21.15.16:0/2431116418 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9710154ed0 con 0x7f97100cf150 2024-03-31T15:12:32.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:32.975+0000 7f9718400640 1 -- 172.21.15.16:0/2431116418 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9710069950 con 0x7f97100cf150 2024-03-31T15:12:32.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:32.975+0000 7f970d600640 1 -- 172.21.15.16:0/2431116418 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f96fc007400 con 0x7f97100cf150 2024-03-31T15:12:32.968 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:32.975+0000 7f970d600640 1 --2- 172.21.15.16:0/2431116418 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f96f00683b0 0x7f96f006a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:32.968 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:32.975+0000 7f970d600640 1 -- 172.21.15.16:0/2431116418 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 5709+0+0 (secure 0 0 0) 0x7f96fc085de0 con 0x7f97100cf150 2024-03-31T15:12:32.968 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:32.975+0000 7f9715a00640 1 --2- 172.21.15.16:0/2431116418 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f96f00683b0 0x7f96f006a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:32.969 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:32.975+0000 7f9715a00640 1 --2- 172.21.15.16:0/2431116418 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f96f00683b0 0x7f96f006a870 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f97040ad000 tx=0x7f9704008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:12:32.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:32.983+0000 7f970d600640 1 -- 172.21.15.16:0/2431116418 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f9710154970 con 0x7f97100cf150 2024-03-31T15:12:33.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:33.147+0000 7f9718400640 1 -- 172.21.15.16:0/2431116418 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f97100b50e0 con 0x7f97100cf150 2024-03-31T15:12:33.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:33.147+0000 7f970d600640 1 -- 172.21.15.16:0/2431116418 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v96) ==== 104+0+11 (secure 0 0 0) 0x7f96fc055cb0 con 0x7f97100cf150 2024-03-31T15:12:33.142 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2024-03-31T15:12:33.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:33.151+0000 7f9718400640 1 -- 172.21.15.16:0/2431116418 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f96f00683b0 msgr2=0x7f96f006a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:33.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:33.151+0000 7f9718400640 1 --2- 172.21.15.16:0/2431116418 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f96f00683b0 0x7f96f006a870 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f97040ad000 tx=0x7f9704008040 comp rx=0 tx=0).stop 2024-03-31T15:12:33.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:33.151+0000 7f9718400640 1 -- 172.21.15.16:0/2431116418 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f97100cf150 msgr2=0x7f97101556d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:33.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:33.151+0000 7f9718400640 1 --2- 172.21.15.16:0/2431116418 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f97100cf150 0x7f97101556d0 secure :-1 s=READY pgs=1471 cs=0 l=1 rev1=1 crypto rx=0x7f96fc00c370 tx=0x7f96fc00c830 comp rx=0 tx=0).stop 2024-03-31T15:12:33.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:33.151+0000 7f9718400640 1 -- 172.21.15.16:0/2431116418 shutdown_connections 2024-03-31T15:12:33.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:33.151+0000 7f9718400640 1 --2- 172.21.15.16:0/2431116418 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f96f00683b0 0x7f96f006a870 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:33.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:33.151+0000 7f9718400640 1 --2- 172.21.15.16:0/2431116418 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f97100d0a60 0x7f97100b5590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:33.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:33.151+0000 7f9718400640 1 --2- 172.21.15.16:0/2431116418 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f97100cf150 0x7f97101556d0 unknown :-1 s=CLOSED pgs=1471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:33.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:33.151+0000 7f9718400640 1 --2- 172.21.15.16:0/2431116418 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f971006f920 0x7f971013d870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:33.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:33.151+0000 7f9718400640 1 -- 172.21.15.16:0/2431116418 >> 172.21.15.16:0/2431116418 conn(0x7f9710067930 msgr2=0x7f97101585a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:12:33.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:33.151+0000 7f9718400640 1 -- 172.21.15.16:0/2431116418 shutdown_connections 2024-03-31T15:12:33.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:33.151+0000 7f9718400640 1 -- 172.21.15.16:0/2431116418 wait complete. 2024-03-31T15:12:33.172 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-31T15:12:33.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:33.323+0000 7efe28c00640 1 Processor -- start 2024-03-31T15:12:33.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:33.323+0000 7efe28c00640 1 -- start start 2024-03-31T15:12:33.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:33.323+0000 7efe28c00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7efe2406f920 0x7efe2406fd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:33.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:33.323+0000 7efe28c00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7efe240d0a60 0x7efe240cea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:33.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:33.323+0000 7efe28c00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7efe240cef80 0x7efe24070e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:33.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:33.323+0000 7efe28c00640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7efe2413f670 con 0x7efe2406f920 2024-03-31T15:12:33.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:33.323+0000 7efe28c00640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7efe24070240 con 0x7efe240d0a60 2024-03-31T15:12:33.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:33.323+0000 7efe28c00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7efe2415ca70 con 0x7efe240cef80 2024-03-31T15:12:33.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:33.323+0000 7efe21e00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7efe2406f920 0x7efe2406fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:33.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:33.323+0000 7efe21e00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7efe2406f920 0x7efe2406fd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41238/0 (socket says 172.21.15.16:41238) 2024-03-31T15:12:33.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:33.323+0000 7efe21e00640 1 -- 172.21.15.16:0/396024041 learned_addr learned my addr 172.21.15.16:0/396024041 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:12:33.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:33.323+0000 7efe21400640 1 --2- 172.21.15.16:0/396024041 >> v2:172.21.15.89:3300/0 conn(0x7efe240d0a60 0x7efe240cea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:33.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:33.323+0000 7efe22800640 1 --2- 172.21.15.16:0/396024041 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7efe240cef80 0x7efe24070e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:33.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:33.323+0000 7efe21e00640 1 -- 172.21.15.16:0/396024041 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7efe240cef80 msgr2=0x7efe24070e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:12:33.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:33.323+0000 7efe21e00640 1 --2- 172.21.15.16:0/396024041 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7efe240cef80 0x7efe24070e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:33.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:33.323+0000 7efe21e00640 1 -- 172.21.15.16:0/396024041 >> v2:172.21.15.89:3300/0 conn(0x7efe240d0a60 msgr2=0x7efe240cea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:12:33.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:33.323+0000 7efe21e00640 1 --2- 172.21.15.16:0/396024041 >> v2:172.21.15.89:3300/0 conn(0x7efe240d0a60 0x7efe240cea40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:33.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:33.323+0000 7efe21e00640 1 -- 172.21.15.16:0/396024041 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efe2415cd50 con 0x7efe2406f920 2024-03-31T15:12:33.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:33.323+0000 7efe22800640 1 --2- 172.21.15.16:0/396024041 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7efe240cef80 0x7efe24070e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:12:33.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:33.327+0000 7efe21e00640 1 --2- 172.21.15.16:0/396024041 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efe2406f920 0x7efe2406fd00 secure :-1 s=READY pgs=1472 cs=0 l=1 rev1=1 crypto rx=0x7efe240736b0 tx=0x7efe1401c260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9caa068eca837e94 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:12:33.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:33.327+0000 7efe20a00640 1 -- 172.21.15.16:0/396024041 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe140b0070 con 0x7efe2406f920 2024-03-31T15:12:33.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:33.327+0000 7efe20a00640 1 -- 172.21.15.16:0/396024041 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efe140ab600 con 0x7efe2406f920 2024-03-31T15:12:33.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:33.327+0000 7efe20a00640 1 -- 172.21.15.16:0/396024041 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe140ab9d0 con 0x7efe2406f920 2024-03-31T15:12:33.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:33.327+0000 7efe28c00640 1 -- 172.21.15.16:0/396024041 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efe2406f920 msgr2=0x7efe2406fd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:33.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:33.327+0000 7efe28c00640 1 --2- 172.21.15.16:0/396024041 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efe2406f920 0x7efe2406fd00 secure :-1 s=READY pgs=1472 cs=0 l=1 rev1=1 crypto rx=0x7efe240736b0 tx=0x7efe1401c260 comp rx=0 tx=0).stop 2024-03-31T15:12:33.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:33.327+0000 7efe28c00640 1 -- 172.21.15.16:0/396024041 shutdown_connections 2024-03-31T15:12:33.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:33.327+0000 7efe28c00640 1 --2- 172.21.15.16:0/396024041 >> v2:172.21.15.89:3300/0 conn(0x7efe240d0a60 0x7efe240cea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:33.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:33.327+0000 7efe28c00640 1 --2- 172.21.15.16:0/396024041 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7efe240cef80 0x7efe24070e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:33.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:33.327+0000 7efe28c00640 1 --2- 172.21.15.16:0/396024041 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efe2406f920 0x7efe2406fd00 unknown :-1 s=CLOSED pgs=1472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:33.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:33.327+0000 7efe28c00640 1 -- 172.21.15.16:0/396024041 >> 172.21.15.16:0/396024041 conn(0x7efe24067930 msgr2=0x7efe240b3ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:12:33.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:33.327+0000 7efe28c00640 1 -- 172.21.15.16:0/396024041 shutdown_connections 2024-03-31T15:12:33.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:33.327+0000 7efe28c00640 1 -- 172.21.15.16:0/396024041 wait complete. 2024-03-31T15:12:33.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:33.331+0000 7efe28c00640 1 Processor -- start 2024-03-31T15:12:33.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:33.331+0000 7efe28c00640 1 -- start start 2024-03-31T15:12:33.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:33.335+0000 7efe28c00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efe2406f920 0x7efe2413e660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:33.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:33.335+0000 7efe28c00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efe240cef80 0x7efe2415f830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:33.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:33.335+0000 7efe28c00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7efe240d0a60 0x7efe240b6510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:33.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:33.335+0000 7efe28c00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7efe241419c0 con 0x7efe2406f920 2024-03-31T15:12:33.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:33.335+0000 7efe28c00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7efe240cf390 con 0x7efe240cef80 2024-03-31T15:12:33.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:33.335+0000 7efe28c00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7efe24141b40 con 0x7efe240d0a60 2024-03-31T15:12:33.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:33.335+0000 7efe21e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efe2406f920 0x7efe2413e660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:33.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:33.335+0000 7efe21e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efe2406f920 0x7efe2413e660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41250/0 (socket says 172.21.15.16:41250) 2024-03-31T15:12:33.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:33.335+0000 7efe21e00640 1 -- 172.21.15.16:0/3929826546 learned_addr learned my addr 172.21.15.16:0/3929826546 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:12:33.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:33.335+0000 7efe21400640 1 --2- 172.21.15.16:0/3929826546 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efe240cef80 0x7efe2415f830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:33.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:33.335+0000 7efe22800640 1 --2- 172.21.15.16:0/3929826546 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7efe240d0a60 0x7efe240b6510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:33.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:33.335+0000 7efe21e00640 1 -- 172.21.15.16:0/3929826546 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7efe240d0a60 msgr2=0x7efe240b6510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:33.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:33.335+0000 7efe21e00640 1 --2- 172.21.15.16:0/3929826546 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7efe240d0a60 0x7efe240b6510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:33.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:33.335+0000 7efe21e00640 1 -- 172.21.15.16:0/3929826546 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efe240cef80 msgr2=0x7efe2415f830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:33.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:33.335+0000 7efe21e00640 1 --2- 172.21.15.16:0/3929826546 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efe240cef80 0x7efe2415f830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:33.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:33.335+0000 7efe21e00640 1 -- 172.21.15.16:0/3929826546 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efe240b2a50 con 0x7efe2406f920 2024-03-31T15:12:33.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:33.335+0000 7efe21400640 1 --2- 172.21.15.16:0/3929826546 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efe240cef80 0x7efe2415f830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:12:33.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:33.335+0000 7efe21e00640 1 --2- 172.21.15.16:0/3929826546 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efe2406f920 0x7efe2413e660 secure :-1 s=READY pgs=1473 cs=0 l=1 rev1=1 crypto rx=0x7efe140ae040 tx=0x7efe14019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:12:33.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:33.335+0000 7efe1a000640 1 -- 172.21.15.16:0/3929826546 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe140b0040 con 0x7efe2406f920 2024-03-31T15:12:33.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:33.335+0000 7efe1a000640 1 -- 172.21.15.16:0/3929826546 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efe140b9070 con 0x7efe2406f920 2024-03-31T15:12:33.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:33.335+0000 7efe28c00640 1 -- 172.21.15.16:0/3929826546 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efe240b37d0 con 0x7efe2406f920 2024-03-31T15:12:33.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:33.335+0000 7efe1a000640 1 -- 172.21.15.16:0/3929826546 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe140b4680 con 0x7efe2406f920 2024-03-31T15:12:33.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:33.335+0000 7efe28c00640 1 -- 172.21.15.16:0/3929826546 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efe240b2fd0 con 0x7efe2406f920 2024-03-31T15:12:33.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:33.335+0000 7efe28c00640 1 -- 172.21.15.16:0/3929826546 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efe24069950 con 0x7efe2406f920 2024-03-31T15:12:33.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:33.335+0000 7efe1a000640 1 -- 172.21.15.16:0/3929826546 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7efe140b4820 con 0x7efe2406f920 2024-03-31T15:12:33.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:33.339+0000 7efe1a000640 1 --2- 172.21.15.16:0/3929826546 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7efe04068380 0x7efe0406a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:33.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:33.339+0000 7efe1a000640 1 -- 172.21.15.16:0/3929826546 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 5709+0+0 (secure 0 0 0) 0x7efe14120800 con 0x7efe2406f920 2024-03-31T15:12:33.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:33.339+0000 7efe21400640 1 --2- 172.21.15.16:0/3929826546 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7efe04068380 0x7efe0406a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:33.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:33.339+0000 7efe21400640 1 --2- 172.21.15.16:0/3929826546 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7efe04068380 0x7efe0406a840 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7efe24070140 tx=0x7efe10008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:12:33.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:33.343+0000 7efe1a000640 1 -- 172.21.15.16:0/3929826546 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7efe14016110 con 0x7efe2406f920 2024-03-31T15:12:33.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:33.507+0000 7efe28c00640 1 -- 172.21.15.16:0/3929826546 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7efe240b4a20 con 0x7efe2406f920 2024-03-31T15:12:33.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:33.507+0000 7efe1a000640 1 -- 172.21.15.16:0/3929826546 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v96) ==== 100+0+11 (secure 0 0 0) 0x7efe140f0c90 con 0x7efe2406f920 2024-03-31T15:12:33.499 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 64 2024-03-31T15:12:33.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:33.507+0000 7efe28c00640 1 -- 172.21.15.16:0/3929826546 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7efe04068380 msgr2=0x7efe0406a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:33.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:33.507+0000 7efe28c00640 1 --2- 172.21.15.16:0/3929826546 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7efe04068380 0x7efe0406a840 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7efe24070140 tx=0x7efe10008040 comp rx=0 tx=0).stop 2024-03-31T15:12:33.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:33.507+0000 7efe28c00640 1 -- 172.21.15.16:0/3929826546 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efe2406f920 msgr2=0x7efe2413e660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:33.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:33.507+0000 7efe28c00640 1 --2- 172.21.15.16:0/3929826546 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efe2406f920 0x7efe2413e660 secure :-1 s=READY pgs=1473 cs=0 l=1 rev1=1 crypto rx=0x7efe140ae040 tx=0x7efe14019000 comp rx=0 tx=0).stop 2024-03-31T15:12:33.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:33.507+0000 7efe28c00640 1 -- 172.21.15.16:0/3929826546 shutdown_connections 2024-03-31T15:12:33.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:33.507+0000 7efe28c00640 1 --2- 172.21.15.16:0/3929826546 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7efe04068380 0x7efe0406a840 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:33.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:33.507+0000 7efe28c00640 1 --2- 172.21.15.16:0/3929826546 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7efe240d0a60 0x7efe240b6510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:33.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:33.507+0000 7efe28c00640 1 --2- 172.21.15.16:0/3929826546 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efe240cef80 0x7efe2415f830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:33.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:33.507+0000 7efe28c00640 1 --2- 172.21.15.16:0/3929826546 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efe2406f920 0x7efe2413e660 unknown :-1 s=CLOSED pgs=1473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:33.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:33.507+0000 7efe28c00640 1 -- 172.21.15.16:0/3929826546 >> 172.21.15.16:0/3929826546 conn(0x7efe24067930 msgr2=0x7efe2415aa80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:12:33.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:33.507+0000 7efe28c00640 1 -- 172.21.15.16:0/3929826546 shutdown_connections 2024-03-31T15:12:33.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:33.507+0000 7efe28c00640 1 -- 172.21.15.16:0/3929826546 wait complete. 2024-03-31T15:12:33.529 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-31T15:12:33.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:33.679+0000 7f2a53800640 1 Processor -- start 2024-03-31T15:12:33.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:33.679+0000 7f2a53800640 1 -- start start 2024-03-31T15:12:33.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:33.679+0000 7f2a53800640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f2a4c145e40 0x7f2a4c0cc740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:33.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:33.679+0000 7f2a53800640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2a4c0d0920 con 0x7f2a4c070030 2024-03-31T15:12:33.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:33.679+0000 7f2a53800640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2a4c0d0c00 con 0x7f2a4c0ccc80 2024-03-31T15:12:33.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:33.679+0000 7f2a53800640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f2a4c0d06d0 con 0x7f2a4c145e40 2024-03-31T15:12:33.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:33.679+0000 7f2a51800640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f2a4c070030 0x7f2a4c0d2010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:42772/0 (socket says 172.21.15.16:42772) 2024-03-31T15:12:33.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:33.679+0000 7f2a51800640 1 -- 172.21.15.16:0/303682023 learned_addr learned my addr 172.21.15.16:0/303682023 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:12:33.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:33.679+0000 7f2a50e00640 1 --2- 172.21.15.16:0/303682023 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f2a4c145e40 0x7f2a4c0cc740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:33.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:33.679+0000 7f2a4b400640 1 -- 172.21.15.16:0/303682023 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 586735171 0 0) 0x7f2a4c0d0920 con 0x7f2a4c070030 2024-03-31T15:12:33.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:33.679+0000 7f2a4b400640 1 -- 172.21.15.16:0/303682023 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2a34003530 con 0x7f2a4c070030 2024-03-31T15:12:33.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:33.679+0000 7f2a4b400640 1 -- 172.21.15.16:0/303682023 <== mon.1 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3323789382 0 0) 0x7f2a4c0d0c00 con 0x7f2a4c0ccc80 2024-03-31T15:12:33.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:33.679+0000 7f2a4b400640 1 -- 172.21.15.16:0/303682023 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2a4c0d0920 con 0x7f2a4c0ccc80 2024-03-31T15:12:33.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:33.679+0000 7f2a4b400640 1 -- 172.21.15.16:0/303682023 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3785475392 0 0) 0x7f2a34003530 con 0x7f2a4c070030 2024-03-31T15:12:33.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:33.679+0000 7f2a50e00640 1 -- 172.21.15.16:0/303682023 >> v1:172.21.15.89:6789/0 conn(0x7f2a4c0ccc80 legacy=0x7f2a4c06f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:33.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:33.679+0000 7f2a50e00640 1 -- 172.21.15.16:0/303682023 >> v1:172.21.15.16:6789/0 conn(0x7f2a4c070030 legacy=0x7f2a4c0d2010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:33.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:33.679+0000 7f2a50e00640 1 -- 172.21.15.16:0/303682023 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a4c0d1810 con 0x7f2a4c145e40 2024-03-31T15:12:33.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:33.687+0000 7f2a50e00640 1 --2- 172.21.15.16:0/303682023 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f2a4c145e40 0x7f2a4c0cc740 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f2a400097f0 tx=0x7f2a4001c400 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c1ffe526863e481d server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:12:33.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:33.687+0000 7f2a4b400640 1 -- 172.21.15.16:0/303682023 <== mon.2 v2:172.21.15.89:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a400b0070 con 0x7f2a4c145e40 2024-03-31T15:12:33.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:33.687+0000 7f2a4b400640 1 -- 172.21.15.16:0/303682023 <== mon.2 v2:172.21.15.89:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2a400ab400 con 0x7f2a4c145e40 2024-03-31T15:12:33.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:33.687+0000 7f2a4b400640 1 -- 172.21.15.16:0/303682023 <== mon.2 v2:172.21.15.89:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a44001a00 con 0x7f2a4c145e40 2024-03-31T15:12:33.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:33.687+0000 7f2a53800640 1 -- 172.21.15.16:0/303682023 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f2a4c145e40 msgr2=0x7f2a4c0cc740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:33.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:33.687+0000 7f2a53800640 1 --2- 172.21.15.16:0/303682023 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f2a4c145e40 0x7f2a4c0cc740 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f2a400097f0 tx=0x7f2a4001c400 comp rx=0 tx=0).stop 2024-03-31T15:12:33.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:33.687+0000 7f2a53800640 1 -- 172.21.15.16:0/303682023 shutdown_connections 2024-03-31T15:12:33.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:33.687+0000 7f2a53800640 1 --2- 172.21.15.16:0/303682023 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f2a4c145e40 0x7f2a4c0cc740 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:33.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:33.687+0000 7f2a53800640 1 -- 172.21.15.16:0/303682023 >> 172.21.15.16:0/303682023 conn(0x7f2a4c067330 msgr2=0x7f2a4c067740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:12:33.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:33.687+0000 7f2a53800640 1 -- 172.21.15.16:0/303682023 shutdown_connections 2024-03-31T15:12:33.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:33.687+0000 7f2a53800640 1 -- 172.21.15.16:0/303682023 wait complete. 2024-03-31T15:12:33.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:33.687+0000 7f2a53800640 1 Processor -- start 2024-03-31T15:12:33.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:33.687+0000 7f2a53800640 1 -- start start 2024-03-31T15:12:33.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:33.691+0000 7f2a53800640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f2a4c070030 0x7f2a4c13ab90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:33.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:33.691+0000 7f2a53800640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2a4c0ccc80 0x7f2a4c13ced0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:33.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:33.691+0000 7f2a53800640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2a4c145e40 0x7f2a4c0b6110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:33.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:33.691+0000 7f2a53800640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f2a4c0b5240 con 0x7f2a4c145e40 2024-03-31T15:12:33.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:33.691+0000 7f2a53800640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f2a4c0b50c0 con 0x7f2a4c0ccc80 2024-03-31T15:12:33.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:33.691+0000 7f2a53800640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f2a4c16a1a0 con 0x7f2a4c070030 2024-03-31T15:12:33.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:33.691+0000 7f2a51800640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2a4c145e40 0x7f2a4c0b6110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:33.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:33.691+0000 7f2a51800640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2a4c145e40 0x7f2a4c0b6110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41262/0 (socket says 172.21.15.16:41262) 2024-03-31T15:12:33.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:33.691+0000 7f2a51800640 1 -- 172.21.15.16:0/167824076 learned_addr learned my addr 172.21.15.16:0/167824076 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:12:33.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:33.691+0000 7f2a4be00640 1 --2- 172.21.15.16:0/167824076 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2a4c0ccc80 0x7f2a4c13ced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:33.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:33.691+0000 7f2a50e00640 1 --2- 172.21.15.16:0/167824076 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f2a4c070030 0x7f2a4c13ab90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:33.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:33.691+0000 7f2a51800640 1 -- 172.21.15.16:0/167824076 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f2a4c070030 msgr2=0x7f2a4c13ab90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:33.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:33.691+0000 7f2a51800640 1 --2- 172.21.15.16:0/167824076 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f2a4c070030 0x7f2a4c13ab90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:33.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:33.691+0000 7f2a51800640 1 -- 172.21.15.16:0/167824076 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2a4c0ccc80 msgr2=0x7f2a4c13ced0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:33.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:33.691+0000 7f2a51800640 1 --2- 172.21.15.16:0/167824076 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2a4c0ccc80 0x7f2a4c13ced0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:33.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:33.691+0000 7f2a51800640 1 -- 172.21.15.16:0/167824076 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a4c06dc40 con 0x7f2a4c145e40 2024-03-31T15:12:33.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:33.691+0000 7f2a51800640 1 --2- 172.21.15.16:0/167824076 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2a4c145e40 0x7f2a4c0b6110 secure :-1 s=READY pgs=1475 cs=0 l=1 rev1=1 crypto rx=0x7f2a44001fa0 tx=0x7f2a4400cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:12:33.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:33.691+0000 7f2a48c00640 1 -- 172.21.15.16:0/167824076 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a44012400 con 0x7f2a4c145e40 2024-03-31T15:12:33.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:33.691+0000 7f2a48c00640 1 -- 172.21.15.16:0/167824076 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2a440125a0 con 0x7f2a4c145e40 2024-03-31T15:12:33.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:33.691+0000 7f2a48c00640 1 -- 172.21.15.16:0/167824076 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a440129d0 con 0x7f2a4c145e40 2024-03-31T15:12:33.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:33.691+0000 7f2a53800640 1 -- 172.21.15.16:0/167824076 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2a4c13b410 con 0x7f2a4c145e40 2024-03-31T15:12:33.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:33.691+0000 7f2a53800640 1 -- 172.21.15.16:0/167824076 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2a4c13b6b0 con 0x7f2a4c145e40 2024-03-31T15:12:33.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:33.691+0000 7f2a53800640 1 -- 172.21.15.16:0/167824076 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2a4c069950 con 0x7f2a4c145e40 2024-03-31T15:12:33.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:33.695+0000 7f2a48c00640 1 -- 172.21.15.16:0/167824076 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f2a44012b70 con 0x7f2a4c145e40 2024-03-31T15:12:33.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:33.695+0000 7f2a48c00640 1 --2- 172.21.15.16:0/167824076 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f2a2c068380 0x7f2a2c06a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:33.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:33.695+0000 7f2a48c00640 1 -- 172.21.15.16:0/167824076 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 5709+0+0 (secure 0 0 0) 0x7f2a440870e0 con 0x7f2a4c145e40 2024-03-31T15:12:33.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:33.695+0000 7f2a50e00640 1 --2- 172.21.15.16:0/167824076 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f2a2c068380 0x7f2a2c06a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:33.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:33.695+0000 7f2a50e00640 1 --2- 172.21.15.16:0/167824076 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f2a2c068380 0x7f2a2c06a840 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f2a400ad000 tx=0x7f2a400a6000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:12:33.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:33.699+0000 7f2a48c00640 1 -- 172.21.15.16:0/167824076 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f2a440571a0 con 0x7f2a4c145e40 2024-03-31T15:12:33.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:33.875+0000 7f2a53800640 1 -- 172.21.15.16:0/167824076 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f2a4c0cf3a0 con 0x7f2a4c145e40 2024-03-31T15:12:33.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:33.875+0000 7f2a48c00640 1 -- 172.21.15.16:0/167824076 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v96) ==== 80+0+0 (secure 0 0 0) 0x7f2a4405b4a0 con 0x7f2a4c145e40 2024-03-31T15:12:33.867 INFO:teuthology.orchestra.run.smithi016.stderr:listed 0 entries 2024-03-31T15:12:33.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:33.879+0000 7f2a53800640 1 -- 172.21.15.16:0/167824076 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f2a2c068380 msgr2=0x7f2a2c06a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:33.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:33.879+0000 7f2a53800640 1 --2- 172.21.15.16:0/167824076 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f2a2c068380 0x7f2a2c06a840 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f2a400ad000 tx=0x7f2a400a6000 comp rx=0 tx=0).stop 2024-03-31T15:12:33.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:33.879+0000 7f2a53800640 1 -- 172.21.15.16:0/167824076 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2a4c145e40 msgr2=0x7f2a4c0b6110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:33.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:33.879+0000 7f2a53800640 1 --2- 172.21.15.16:0/167824076 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2a4c145e40 0x7f2a4c0b6110 secure :-1 s=READY pgs=1475 cs=0 l=1 rev1=1 crypto rx=0x7f2a44001fa0 tx=0x7f2a4400cf90 comp rx=0 tx=0).stop 2024-03-31T15:12:33.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:33.879+0000 7f2a53800640 1 -- 172.21.15.16:0/167824076 shutdown_connections 2024-03-31T15:12:33.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:33.879+0000 7f2a53800640 1 --2- 172.21.15.16:0/167824076 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f2a2c068380 0x7f2a2c06a840 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:33.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:33.879+0000 7f2a53800640 1 --2- 172.21.15.16:0/167824076 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2a4c145e40 0x7f2a4c0b6110 unknown :-1 s=CLOSED pgs=1475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:33.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:33.879+0000 7f2a53800640 1 --2- 172.21.15.16:0/167824076 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2a4c0ccc80 0x7f2a4c13ced0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:33.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:33.879+0000 7f2a53800640 1 --2- 172.21.15.16:0/167824076 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f2a4c070030 0x7f2a4c13ab90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:33.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:33.879+0000 7f2a53800640 1 -- 172.21.15.16:0/167824076 >> 172.21.15.16:0/167824076 conn(0x7f2a4c067330 msgr2=0x7f2a4c0cff60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:12:33.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:33.879+0000 7f2a53800640 1 -- 172.21.15.16:0/167824076 shutdown_connections 2024-03-31T15:12:33.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:33.879+0000 7f2a53800640 1 -- 172.21.15.16:0/167824076 wait complete. 2024-03-31T15:12:33.899 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-03-31T15:12:33.899 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-31T15:12:33.908 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-31T15:12:34.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:34.135+0000 7f577fe00640 1 Processor -- start 2024-03-31T15:12:34.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:34.135+0000 7f577fe00640 1 -- start start 2024-03-31T15:12:34.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:34.139+0000 7f577fe00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f57780728b0 0x7f57780b1cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:34.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:34.139+0000 7f577fe00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f57780b2200 0x7f577815cd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:34.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:34.139+0000 7f577fe00640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5778160e70 con 0x7f5778145d10 2024-03-31T15:12:34.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:34.139+0000 7f577fe00640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f57781602f0 con 0x7f57780728b0 2024-03-31T15:12:34.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:34.139+0000 7f577fe00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f57781605d0 con 0x7f57780b2200 2024-03-31T15:12:34.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:34.139+0000 7f577ca00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f57780728b0 0x7f57780b1cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:34.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:34.139+0000 7f577de00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f57780b2200 0x7f577815cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:34.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:34.139+0000 7f577ca00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f57780728b0 0x7f57780b1cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.16:44928/0 (socket says 172.21.15.16:44928) 2024-03-31T15:12:34.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:34.139+0000 7f577ca00640 1 -- 172.21.15.16:0/2998007246 learned_addr learned my addr 172.21.15.16:0/2998007246 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:12:34.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:34.139+0000 7f577ca00640 1 -- 172.21.15.16:0/2998007246 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f57780b2200 msgr2=0x7f577815cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:12:34.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:34.139+0000 7f577ca00640 1 --2- 172.21.15.16:0/2998007246 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f57780b2200 0x7f577815cd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:34.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:34.139+0000 7f577ca00640 1 -- 172.21.15.16:0/2998007246 >> v1:172.21.15.89:6789/0 conn(0x7f5778145d10 legacy=0x7f57780cf790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:34.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:34.139+0000 7f577ca00640 1 -- 172.21.15.16:0/2998007246 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57781608b0 con 0x7f57780728b0 2024-03-31T15:12:34.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:34.143+0000 7f577ca00640 1 --2- 172.21.15.16:0/2998007246 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f57780728b0 0x7f57780b1cc0 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f5764009870 tx=0x7f576401c560 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e9a355f91e3906c4 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:12:34.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:34.143+0000 7f5777e00640 1 -- 172.21.15.16:0/2998007246 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57640b2070 con 0x7f57780728b0 2024-03-31T15:12:34.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:34.143+0000 7f5777e00640 1 -- 172.21.15.16:0/2998007246 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f57640a9070 con 0x7f57780728b0 2024-03-31T15:12:34.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:34.143+0000 7f5777e00640 1 -- 172.21.15.16:0/2998007246 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57640ad880 con 0x7f57780728b0 2024-03-31T15:12:34.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:34.143+0000 7f577fe00640 1 -- 172.21.15.16:0/2998007246 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f57780728b0 msgr2=0x7f57780b1cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:34.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:34.143+0000 7f577fe00640 1 --2- 172.21.15.16:0/2998007246 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f57780728b0 0x7f57780b1cc0 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f5764009870 tx=0x7f576401c560 comp rx=0 tx=0).stop 2024-03-31T15:12:34.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:34.143+0000 7f577fe00640 1 -- 172.21.15.16:0/2998007246 shutdown_connections 2024-03-31T15:12:34.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:34.143+0000 7f577fe00640 1 --2- 172.21.15.16:0/2998007246 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f57780728b0 0x7f57780b1cc0 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:34.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:34.143+0000 7f577fe00640 1 --2- 172.21.15.16:0/2998007246 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f57780b2200 0x7f577815cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:34.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:34.143+0000 7f577fe00640 1 -- 172.21.15.16:0/2998007246 >> 172.21.15.16:0/2998007246 conn(0x7f577806b620 msgr2=0x7f577806ba30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:12:34.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:34.143+0000 7f577fe00640 1 -- 172.21.15.16:0/2998007246 shutdown_connections 2024-03-31T15:12:34.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:34.143+0000 7f577fe00640 1 -- 172.21.15.16:0/2998007246 wait complete. 2024-03-31T15:12:34.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:34.143+0000 7f577fe00640 1 Processor -- start 2024-03-31T15:12:34.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:34.143+0000 7f577fe00640 1 -- start start 2024-03-31T15:12:34.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:34.143+0000 7f577fe00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f57780728b0 0x7f5778157310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:34.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:34.143+0000 7f577fe00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f57780b2200 0x7f577815b730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:34.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:34.143+0000 7f577fe00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5778145d10 0x7f577815bc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:34.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:34.143+0000 7f577fe00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f5778072d40 con 0x7f57780728b0 2024-03-31T15:12:34.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:34.143+0000 7f577fe00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f57780b2610 con 0x7f5778145d10 2024-03-31T15:12:34.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:34.143+0000 7f577fe00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f5778072ec0 con 0x7f57780b2200 2024-03-31T15:12:34.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:34.143+0000 7f577d400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f57780728b0 0x7f5778157310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:34.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:34.147+0000 7f577d400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f57780728b0 0x7f5778157310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41266/0 (socket says 172.21.15.16:41266) 2024-03-31T15:12:34.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:34.147+0000 7f577d400640 1 -- 172.21.15.16:0/3319287308 learned_addr learned my addr 172.21.15.16:0/3319287308 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:12:34.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:34.147+0000 7f577de00640 1 --2- 172.21.15.16:0/3319287308 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5778145d10 0x7f577815bc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:34.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:34.147+0000 7f577ca00640 1 --2- 172.21.15.16:0/3319287308 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f57780b2200 0x7f577815b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:34.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:34.147+0000 7f577d400640 1 -- 172.21.15.16:0/3319287308 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f57780b2200 msgr2=0x7f577815b730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:34.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:34.147+0000 7f577d400640 1 --2- 172.21.15.16:0/3319287308 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f57780b2200 0x7f577815b730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:34.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:34.147+0000 7f577d400640 1 -- 172.21.15.16:0/3319287308 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5778145d10 msgr2=0x7f577815bc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:34.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:34.147+0000 7f577d400640 1 --2- 172.21.15.16:0/3319287308 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5778145d10 0x7f577815bc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:34.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:34.147+0000 7f577d400640 1 -- 172.21.15.16:0/3319287308 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5778142b50 con 0x7f57780728b0 2024-03-31T15:12:34.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:34.147+0000 7f577de00640 1 --2- 172.21.15.16:0/3319287308 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5778145d10 0x7f577815bc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:12:34.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:34.147+0000 7f577ca00640 1 --2- 172.21.15.16:0/3319287308 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f57780b2200 0x7f577815b730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:12:34.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:34.147+0000 7f577d400640 1 --2- 172.21.15.16:0/3319287308 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f57780728b0 0x7f5778157310 secure :-1 s=READY pgs=1476 cs=0 l=1 rev1=1 crypto rx=0x7f576c00c370 tx=0x7f576c00c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:12:34.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:34.147+0000 7f5775600640 1 -- 172.21.15.16:0/3319287308 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f576c0052d0 con 0x7f57780728b0 2024-03-31T15:12:34.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:34.147+0000 7f5775600640 1 -- 172.21.15.16:0/3319287308 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f576c005470 con 0x7f57780728b0 2024-03-31T15:12:34.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:34.147+0000 7f5775600640 1 -- 172.21.15.16:0/3319287308 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f576c007260 con 0x7f57780728b0 2024-03-31T15:12:34.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:34.147+0000 7f577fe00640 1 -- 172.21.15.16:0/3319287308 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f57780b3690 con 0x7f57780728b0 2024-03-31T15:12:34.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:34.147+0000 7f577fe00640 1 -- 172.21.15.16:0/3319287308 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f57780b3c50 con 0x7f57780728b0 2024-03-31T15:12:34.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:34.147+0000 7f577fe00640 1 -- 172.21.15.16:0/3319287308 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f57780b34a0 con 0x7f57780728b0 2024-03-31T15:12:34.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:34.147+0000 7f5775600640 1 -- 172.21.15.16:0/3319287308 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f576c007400 con 0x7f57780728b0 2024-03-31T15:12:34.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:34.147+0000 7f5775600640 1 --2- 172.21.15.16:0/3319287308 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f5758068380 0x7f575806a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:34.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:34.147+0000 7f5775600640 1 -- 172.21.15.16:0/3319287308 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 5709+0+0 (secure 0 0 0) 0x7f576c085e70 con 0x7f57780728b0 2024-03-31T15:12:34.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:34.151+0000 7f577ca00640 1 --2- 172.21.15.16:0/3319287308 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f5758068380 0x7f575806a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:34.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:34.151+0000 7f577ca00640 1 --2- 172.21.15.16:0/3319287308 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f5758068380 0x7f575806a840 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f57640af000 tx=0x7f5764008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:12:34.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:34.155+0000 7f5775600640 1 -- 172.21.15.16:0/3319287308 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f576c056420 con 0x7f57780728b0 2024-03-31T15:12:34.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:34.495+0000 7f577fe00640 1 -- 172.21.15.16:0/3319287308 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f5778069950 con 0x7f57780728b0 2024-03-31T15:12:34.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:34.499+0000 7f5775600640 1 -- 172.21.15.16:0/3319287308 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v96) ==== 74+0+16536 (secure 0 0 0) 0x7f576c055d20 con 0x7f57780728b0 2024-03-31T15:12:34.491 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:12:34.491 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":96,"fsid":"286eb5da-fb47-4550-b290-d9519c13b9a5","created":"2024-03-31T15:03:11.438952+0000","modified":"2024-03-31T15:12:17.885852+0000","last_up_change":"2024-03-31T15:03:22.885535+0000","last_in_change":"2024-03-31T15:03:13.941667+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-31T15:03:22.550787+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2024-03-31T15:12:14.671225+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"95","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2024-03-31T15:12:15.242196+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"96","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"92a66450-ef19-40d8-ac3d-d760b23b0bda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6801","nonce":932712244}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6803","nonce":932712244}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6807","nonce":932712244}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6805","nonce":932712244}]},"public_addr":"172.21.15.16:6801/932712244","cluster_addr":"172.21.15.16:6803/932712244","heartbeat_back_addr":"172.21.15.16:6807/932712244","heartbeat_front_addr":"172.21.15.16:6805/932712244","state":["exists","up"]},{"osd":1,"uuid":"da9d1f2b-1968-4959-b843-faa231988e92","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6810","nonce":1398737533}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6811","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6813","nonce":1398737533}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6821","nonce":1398737533}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6815","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6816","nonce":1398737533}]},"public_addr":"172.21.15.16:6810/1398737533","cluster_addr":"172.21.15.16:6813/1398737533","heartbeat_back_addr":"172.21.15.16:6821/1398737533","heartbeat_front_addr":"172.21.15.16:6816/1398737533","state":["exists","up"]},{"osd":2,"uuid":"01a73662-9a21-4b1f-bcbb-42b9636a4638","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6819","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6822","nonce":3018526646}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6825","nonce":3018526646}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6831","nonce":3018526646}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6827","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6829","nonce":3018526646}]},"public_addr":"172.21.15.16:6822/3018526646","cluster_addr":"172.21.15.16:6825/3018526646","heartbeat_back_addr":"172.21.15.16:6831/3018526646","heartbeat_front_addr":"172.21.15.16:6829/3018526646","state":["exists","up"]},{"osd":3,"uuid":"c7771fd3-e18e-4500-ac38-23cf10ebcd75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6809","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6812","nonce":3360352974}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6817","nonce":3360352974}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6828","nonce":3360352974}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6823","nonce":3360352974}]},"public_addr":"172.21.15.16:6812/3360352974","cluster_addr":"172.21.15.16:6817/3360352974","heartbeat_back_addr":"172.21.15.16:6828/3360352974","heartbeat_front_addr":"172.21.15.16:6823/3360352974","state":["exists","up"]},{"osd":4,"uuid":"88260730-1482-4fbd-a40f-cb1124e9df4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6801","nonce":1856844856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6803","nonce":1856844856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6807","nonce":1856844856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6805","nonce":1856844856}]},"public_addr":"172.21.15.89:6801/1856844856","cluster_addr":"172.21.15.89:6803/1856844856","heartbeat_back_addr":"172.21.15.89:6807/1856844856","heartbeat_front_addr":"172.21.15.89:6805/1856844856","state":["exists","up"]},{"osd":5,"uuid":"4cb2c00c-fae0-4dd8-9d27-ec4f0093ccf7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6814","nonce":50395527}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6817","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6820","nonce":50395527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6831","nonce":50395527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6823","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6826","nonce":50395527}]},"public_addr":"172.21.15.89:6814/50395527","cluster_addr":"172.21.15.89:6820/50395527","heartbeat_back_addr":"172.21.15.89:6831/50395527","heartbeat_front_addr":"172.21.15.89:6826/50395527","state":["exists","up"]},{"osd":6,"uuid":"81ce0c19-2554-4cae-acbe-66a05e9e9b9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6809","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6812","nonce":3470471752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6815","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6818","nonce":3470471752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6827","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6829","nonce":3470471752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6821","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6824","nonce":3470471752}]},"public_addr":"172.21.15.89:6812/3470471752","cluster_addr":"172.21.15.89:6818/3470471752","heartbeat_back_addr":"172.21.15.89:6829/3470471752","heartbeat_front_addr":"172.21.15.89:6824/3470471752","state":["exists","up"]},{"osd":7,"uuid":"85e4313b-d37d-442e-8042-153bf20a847d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6811","nonce":1796778630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6813","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6816","nonce":1796778630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6825","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6828","nonce":1796778630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6819","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6822","nonce":1796778630}]},"public_addr":"172.21.15.89:6811/1796778630","cluster_addr":"172.21.15.89:6816/1796778630","heartbeat_back_addr":"172.21.15.89:6828/1796778630","heartbeat_front_addr":"172.21.15.89:6822/1796778630","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.439088+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.529970+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.680512+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:21.044161+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.582111+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.536558+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.409522+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.590229+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-31T15:12:34.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:34.499+0000 7f577fe00640 1 -- 172.21.15.16:0/3319287308 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f5758068380 msgr2=0x7f575806a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:34.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:34.499+0000 7f577fe00640 1 --2- 172.21.15.16:0/3319287308 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f5758068380 0x7f575806a840 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f57640af000 tx=0x7f5764008040 comp rx=0 tx=0).stop 2024-03-31T15:12:34.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:34.499+0000 7f577fe00640 1 -- 172.21.15.16:0/3319287308 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f57780728b0 msgr2=0x7f5778157310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:34.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:34.499+0000 7f577fe00640 1 --2- 172.21.15.16:0/3319287308 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f57780728b0 0x7f5778157310 secure :-1 s=READY pgs=1476 cs=0 l=1 rev1=1 crypto rx=0x7f576c00c370 tx=0x7f576c00c830 comp rx=0 tx=0).stop 2024-03-31T15:12:34.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:34.499+0000 7f577fe00640 1 -- 172.21.15.16:0/3319287308 shutdown_connections 2024-03-31T15:12:34.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:34.499+0000 7f577fe00640 1 --2- 172.21.15.16:0/3319287308 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f5758068380 0x7f575806a840 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:34.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:34.499+0000 7f577fe00640 1 --2- 172.21.15.16:0/3319287308 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5778145d10 0x7f577815bc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:34.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:34.499+0000 7f577fe00640 1 --2- 172.21.15.16:0/3319287308 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f57780b2200 0x7f577815b730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:34.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:34.499+0000 7f577fe00640 1 --2- 172.21.15.16:0/3319287308 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f57780728b0 0x7f5778157310 unknown :-1 s=CLOSED pgs=1476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:34.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:34.499+0000 7f577fe00640 1 -- 172.21.15.16:0/3319287308 >> 172.21.15.16:0/3319287308 conn(0x7f577806b620 msgr2=0x7f57780cf630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:12:34.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:34.499+0000 7f577fe00640 1 -- 172.21.15.16:0/3319287308 shutdown_connections 2024-03-31T15:12:34.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:34.499+0000 7f577fe00640 1 -- 172.21.15.16:0/3319287308 wait complete. 2024-03-31T15:12:34.512 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-31T15:03:22.550787+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 14, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-31T15:12:14.671225+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '95', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.5, 'score_stable': 2.5, 'optimal_score': 1, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 15, 'pool_name': 'cephfs_data', 'create_time': '2024-03-31T15:12:15.242196+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '96', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-31T15:12:34.513 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-31T15:12:34.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:34.663+0000 7f27f4e00640 1 Processor -- start 2024-03-31T15:12:34.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:34.663+0000 7f27f4e00640 1 -- start start 2024-03-31T15:12:34.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:34.663+0000 7f27f4e00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f27f015c3d0 0x7f27f006fc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:34.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:34.663+0000 7f27f4e00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f27f00d0880 0x7f27f00b4250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:34.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:34.663+0000 7f27f4e00640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f27f0070520 con 0x7f27f00cdc30 2024-03-31T15:12:34.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:34.663+0000 7f27f4e00640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f27f00cfda0 con 0x7f27f015c3d0 2024-03-31T15:12:34.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:34.663+0000 7f27f4e00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f27f00d0080 con 0x7f27f00d0880 2024-03-31T15:12:34.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:34.663+0000 7f27e7e00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f27f015c3d0 0x7f27f006fc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:34.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:34.663+0000 7f27ede00640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f27f00cdc30 0x7f27f00ce010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:42776/0 (socket says 172.21.15.16:42776) 2024-03-31T15:12:34.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:34.663+0000 7f27ede00640 1 -- 172.21.15.16:0/1036585741 learned_addr learned my addr 172.21.15.16:0/1036585741 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:12:34.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:34.667+0000 7f27ee800640 1 --2- 172.21.15.16:0/1036585741 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f27f00d0880 0x7f27f00b4250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:34.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:34.667+0000 7f27e7e00640 1 -- 172.21.15.16:0/1036585741 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f27f00d0880 msgr2=0x7f27f00b4250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:12:34.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:34.667+0000 7f27e7e00640 1 --2- 172.21.15.16:0/1036585741 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f27f00d0880 0x7f27f00b4250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:34.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:34.667+0000 7f27e7e00640 1 -- 172.21.15.16:0/1036585741 >> v1:172.21.15.16:6789/0 conn(0x7f27f00cdc30 legacy=0x7f27f00ce010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:34.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:34.667+0000 7f27e7e00640 1 -- 172.21.15.16:0/1036585741 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27f00d0360 con 0x7f27f015c3d0 2024-03-31T15:12:34.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:34.667+0000 7f27ed400640 1 -- 172.21.15.16:0/1036585741 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2106856586 0 0) 0x7f27f0070520 con 0x7f27f00cdc30 2024-03-31T15:12:34.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:34.667+0000 7f27e7e00640 1 --2- 172.21.15.16:0/1036585741 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f27f015c3d0 0x7f27f006fc40 secure :-1 s=READY pgs=1478 cs=0 l=1 rev1=1 crypto rx=0x7f27d80097c0 tx=0x7f27d801c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec64a93fd7b8711e server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:12:34.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:34.667+0000 7f27ed400640 1 -- 172.21.15.16:0/1036585741 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27d80b1070 con 0x7f27f015c3d0 2024-03-31T15:12:34.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:34.667+0000 7f27ed400640 1 -- 172.21.15.16:0/1036585741 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f27d80ac630 con 0x7f27f015c3d0 2024-03-31T15:12:34.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:34.667+0000 7f27ed400640 1 -- 172.21.15.16:0/1036585741 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27d80a8050 con 0x7f27f015c3d0 2024-03-31T15:12:34.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:34.671+0000 7f27f4e00640 1 -- 172.21.15.16:0/1036585741 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f27f015c3d0 msgr2=0x7f27f006fc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:34.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:34.671+0000 7f27f4e00640 1 --2- 172.21.15.16:0/1036585741 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f27f015c3d0 0x7f27f006fc40 secure :-1 s=READY pgs=1478 cs=0 l=1 rev1=1 crypto rx=0x7f27d80097c0 tx=0x7f27d801c4a0 comp rx=0 tx=0).stop 2024-03-31T15:12:34.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:34.671+0000 7f27f4e00640 1 -- 172.21.15.16:0/1036585741 shutdown_connections 2024-03-31T15:12:34.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:34.671+0000 7f27f4e00640 1 --2- 172.21.15.16:0/1036585741 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f27f00d0880 0x7f27f00b4250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:34.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:34.671+0000 7f27f4e00640 1 --2- 172.21.15.16:0/1036585741 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f27f015c3d0 0x7f27f006fc40 unknown :-1 s=CLOSED pgs=1478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:34.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:34.671+0000 7f27f4e00640 1 -- 172.21.15.16:0/1036585741 >> 172.21.15.16:0/1036585741 conn(0x7f27f006e670 msgr2=0x7f27f006ea50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:12:34.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:34.671+0000 7f27f4e00640 1 -- 172.21.15.16:0/1036585741 shutdown_connections 2024-03-31T15:12:34.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:34.671+0000 7f27f4e00640 1 -- 172.21.15.16:0/1036585741 wait complete. 2024-03-31T15:12:34.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:34.671+0000 7f27f4e00640 1 Processor -- start 2024-03-31T15:12:34.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:34.671+0000 7f27f4e00640 1 -- start start 2024-03-31T15:12:34.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:34.671+0000 7f27f4e00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f27f00cdc30 0x7f27f0164690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:34.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:34.671+0000 7f27f4e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f27f00d0880 0x7f27f00728c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:34.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:34.671+0000 7f27f4e00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f27f015c3d0 0x7f27f014d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:34.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:34.671+0000 7f27f4e00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f27f0159100 con 0x7f27f00d0880 2024-03-31T15:12:34.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:34.671+0000 7f27f4e00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f27f0158f80 con 0x7f27f00cdc30 2024-03-31T15:12:34.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:34.671+0000 7f27f4e00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f27f0168670 con 0x7f27f015c3d0 2024-03-31T15:12:34.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:34.671+0000 7f27e7e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f27f00d0880 0x7f27f00728c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:34.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:34.671+0000 7f27e7e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f27f00d0880 0x7f27f00728c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41280/0 (socket says 172.21.15.16:41280) 2024-03-31T15:12:34.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:34.671+0000 7f27e7e00640 1 -- 172.21.15.16:0/2647005008 learned_addr learned my addr 172.21.15.16:0/2647005008 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:12:34.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:34.671+0000 7f27ede00640 1 --2- 172.21.15.16:0/2647005008 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f27f00cdc30 0x7f27f0164690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:34.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:34.671+0000 7f27ee800640 1 --2- 172.21.15.16:0/2647005008 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f27f015c3d0 0x7f27f014d6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:34.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:34.671+0000 7f27e7e00640 1 -- 172.21.15.16:0/2647005008 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f27f015c3d0 msgr2=0x7f27f014d6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:34.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:34.671+0000 7f27e7e00640 1 --2- 172.21.15.16:0/2647005008 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f27f015c3d0 0x7f27f014d6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:34.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:34.671+0000 7f27e7e00640 1 -- 172.21.15.16:0/2647005008 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f27f00cdc30 msgr2=0x7f27f0164690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:34.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:34.671+0000 7f27e7e00640 1 --2- 172.21.15.16:0/2647005008 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f27f00cdc30 0x7f27f0164690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:34.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:34.671+0000 7f27e7e00640 1 -- 172.21.15.16:0/2647005008 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27f00b12a0 con 0x7f27f00d0880 2024-03-31T15:12:34.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:34.671+0000 7f27ede00640 1 --2- 172.21.15.16:0/2647005008 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f27f00cdc30 0x7f27f0164690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:12:34.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:34.671+0000 7f27ee800640 1 --2- 172.21.15.16:0/2647005008 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f27f015c3d0 0x7f27f014d6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:12:34.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:34.675+0000 7f27e7e00640 1 --2- 172.21.15.16:0/2647005008 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f27f00d0880 0x7f27f00728c0 secure :-1 s=READY pgs=1479 cs=0 l=1 rev1=1 crypto rx=0x7f27d801a0d0 tx=0x7f27d8019240 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:12:34.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:34.675+0000 7f27e6000640 1 -- 172.21.15.16:0/2647005008 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27d80b1040 con 0x7f27f00d0880 2024-03-31T15:12:34.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:34.675+0000 7f27e6000640 1 -- 172.21.15.16:0/2647005008 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f27d80be070 con 0x7f27f00d0880 2024-03-31T15:12:34.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:34.675+0000 7f27e6000640 1 -- 172.21.15.16:0/2647005008 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27d80a8040 con 0x7f27f00d0880 2024-03-31T15:12:34.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:34.675+0000 7f27f4e00640 1 -- 172.21.15.16:0/2647005008 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f27f00b1580 con 0x7f27f00d0880 2024-03-31T15:12:34.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:34.675+0000 7f27f4e00640 1 -- 172.21.15.16:0/2647005008 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f27f00b1b40 con 0x7f27f00d0880 2024-03-31T15:12:34.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:34.675+0000 7f27f4e00640 1 -- 172.21.15.16:0/2647005008 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f27f0069950 con 0x7f27f00d0880 2024-03-31T15:12:34.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:34.675+0000 7f27e6000640 1 -- 172.21.15.16:0/2647005008 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f27d80b4070 con 0x7f27f00d0880 2024-03-31T15:12:34.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:34.679+0000 7f27e6000640 1 --2- 172.21.15.16:0/2647005008 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f27cc068380 0x7f27cc06a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:34.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:34.679+0000 7f27e6000640 1 -- 172.21.15.16:0/2647005008 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 5709+0+0 (secure 0 0 0) 0x7f27d8120a00 con 0x7f27f00d0880 2024-03-31T15:12:34.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:34.679+0000 7f27ede00640 1 --2- 172.21.15.16:0/2647005008 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f27cc068380 0x7f27cc06a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:34.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:34.679+0000 7f27ede00640 1 --2- 172.21.15.16:0/2647005008 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f27cc068380 0x7f27cc06a840 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f27e0000f70 tx=0x7f27e0009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:12:34.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:34.683+0000 7f27e6000640 1 -- 172.21.15.16:0/2647005008 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f27d8016090 con 0x7f27f00d0880 2024-03-31T15:12:34.839 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:34.847+0000 7f27f4e00640 1 -- 172.21.15.16:0/2647005008 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f27f006d740 con 0x7f27f00d0880 2024-03-31T15:12:34.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:34.847+0000 7f27e6000640 1 -- 172.21.15.16:0/2647005008 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v96) ==== 93+0+10 (secure 0 0 0) 0x7f27d80b4360 con 0x7f27f00d0880 2024-03-31T15:12:34.840 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2024-03-31T15:12:34.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:34.851+0000 7f27f4e00640 1 -- 172.21.15.16:0/2647005008 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f27cc068380 msgr2=0x7f27cc06a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:34.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:34.851+0000 7f27f4e00640 1 --2- 172.21.15.16:0/2647005008 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f27cc068380 0x7f27cc06a840 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f27e0000f70 tx=0x7f27e0009040 comp rx=0 tx=0).stop 2024-03-31T15:12:34.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:34.851+0000 7f27f4e00640 1 -- 172.21.15.16:0/2647005008 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f27f00d0880 msgr2=0x7f27f00728c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:34.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:34.851+0000 7f27f4e00640 1 --2- 172.21.15.16:0/2647005008 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f27f00d0880 0x7f27f00728c0 secure :-1 s=READY pgs=1479 cs=0 l=1 rev1=1 crypto rx=0x7f27d801a0d0 tx=0x7f27d8019240 comp rx=0 tx=0).stop 2024-03-31T15:12:34.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:34.851+0000 7f27f4e00640 1 -- 172.21.15.16:0/2647005008 shutdown_connections 2024-03-31T15:12:34.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:34.851+0000 7f27f4e00640 1 --2- 172.21.15.16:0/2647005008 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f27cc068380 0x7f27cc06a840 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:34.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:34.851+0000 7f27f4e00640 1 --2- 172.21.15.16:0/2647005008 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f27f015c3d0 0x7f27f014d6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:34.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:34.851+0000 7f27f4e00640 1 --2- 172.21.15.16:0/2647005008 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f27f00d0880 0x7f27f00728c0 unknown :-1 s=CLOSED pgs=1479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:34.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:34.851+0000 7f27f4e00640 1 --2- 172.21.15.16:0/2647005008 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f27f00cdc30 0x7f27f0164690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:34.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:34.851+0000 7f27f4e00640 1 -- 172.21.15.16:0/2647005008 >> 172.21.15.16:0/2647005008 conn(0x7f27f006e670 msgr2=0x7f27f00b5a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:12:34.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:34.851+0000 7f27f4e00640 1 -- 172.21.15.16:0/2647005008 shutdown_connections 2024-03-31T15:12:34.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:34.851+0000 7f27f4e00640 1 -- 172.21.15.16:0/2647005008 wait complete. 2024-03-31T15:12:34.860 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-31T15:12:35.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:35.011+0000 7fc01ec00640 1 Processor -- start 2024-03-31T15:12:35.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:35.011+0000 7fc01ec00640 1 -- start start 2024-03-31T15:12:35.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:35.015+0000 7fc01ec00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fc01813dff0 0x7fc01806d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:35.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:35.015+0000 7fc01ec00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fc01813e3d0 0x7fc01806da10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:35.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:35.015+0000 7fc01ec00640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc01815fbc0 con 0x7fc0180cc9d0 2024-03-31T15:12:35.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:35.015+0000 7fc01ec00640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7fc01815f730 con 0x7fc01813dff0 2024-03-31T15:12:35.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:35.015+0000 7fc01ec00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fc01815ee00 con 0x7fc01813e3d0 2024-03-31T15:12:35.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:35.015+0000 7fc01cc00640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fc0180cc9d0 0x7fc0180cce30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:42778/0 (socket says 172.21.15.16:42778) 2024-03-31T15:12:35.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:35.015+0000 7fc01cc00640 1 -- 172.21.15.16:0/1834239346 learned_addr learned my addr 172.21.15.16:0/1834239346 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:12:35.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:35.015+0000 7fc017e00640 1 --2- 172.21.15.16:0/1834239346 >> v2:172.21.15.89:3300/0 conn(0x7fc01813dff0 0x7fc01806d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:35.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:35.015+0000 7fc00fe00640 1 --2- 172.21.15.16:0/1834239346 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fc01813e3d0 0x7fc01806da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:35.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:35.015+0000 7fc017400640 1 -- 172.21.15.16:0/1834239346 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 150647359 0 0) 0x7fc01815fbc0 con 0x7fc0180cc9d0 2024-03-31T15:12:35.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:35.015+0000 7fc017400640 1 -- 172.21.15.16:0/1834239346 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbffc003530 con 0x7fc0180cc9d0 2024-03-31T15:12:35.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:35.015+0000 7fc017e00640 1 -- 172.21.15.16:0/1834239346 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fc01813e3d0 msgr2=0x7fc01806da10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:12:35.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:35.015+0000 7fc017e00640 1 --2- 172.21.15.16:0/1834239346 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fc01813e3d0 0x7fc01806da10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:35.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:35.015+0000 7fc017e00640 1 -- 172.21.15.16:0/1834239346 >> v1:172.21.15.16:6789/0 conn(0x7fc0180cc9d0 legacy=0x7fc0180cce30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:35.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:35.015+0000 7fc017e00640 1 -- 172.21.15.16:0/1834239346 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc01815f0e0 con 0x7fc01813dff0 2024-03-31T15:12:35.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:35.019+0000 7fc017e00640 1 --2- 172.21.15.16:0/1834239346 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc01813dff0 0x7fc01806d4d0 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7fc0080097c0 tx=0x7fc00801c410 comp rx=0 tx=0).ready entity=mon.1 client_cookie=97719b90e97491cd server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:12:35.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:35.019+0000 7fc017400640 1 -- 172.21.15.16:0/1834239346 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0080b0070 con 0x7fc01813dff0 2024-03-31T15:12:35.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:35.019+0000 7fc017400640 1 -- 172.21.15.16:0/1834239346 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc0080ab600 con 0x7fc01813dff0 2024-03-31T15:12:35.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:35.019+0000 7fc017400640 1 -- 172.21.15.16:0/1834239346 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0080ab9d0 con 0x7fc01813dff0 2024-03-31T15:12:35.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:35.019+0000 7fc01ec00640 1 -- 172.21.15.16:0/1834239346 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc01813dff0 msgr2=0x7fc01806d4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:35.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:35.019+0000 7fc01ec00640 1 --2- 172.21.15.16:0/1834239346 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc01813dff0 0x7fc01806d4d0 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7fc0080097c0 tx=0x7fc00801c410 comp rx=0 tx=0).stop 2024-03-31T15:12:35.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:35.019+0000 7fc01ec00640 1 -- 172.21.15.16:0/1834239346 shutdown_connections 2024-03-31T15:12:35.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:35.019+0000 7fc01ec00640 1 --2- 172.21.15.16:0/1834239346 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fc01813e3d0 0x7fc01806da10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:35.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:35.019+0000 7fc01ec00640 1 --2- 172.21.15.16:0/1834239346 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc01813dff0 0x7fc01806d4d0 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:35.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:35.019+0000 7fc01ec00640 1 -- 172.21.15.16:0/1834239346 >> 172.21.15.16:0/1834239346 conn(0x7fc01806ee40 msgr2=0x7fc01806b620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:12:35.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:35.019+0000 7fc01ec00640 1 -- 172.21.15.16:0/1834239346 shutdown_connections 2024-03-31T15:12:35.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:35.019+0000 7fc01ec00640 1 -- 172.21.15.16:0/1834239346 wait complete. 2024-03-31T15:12:35.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:35.019+0000 7fc01ec00640 1 Processor -- start 2024-03-31T15:12:35.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:35.019+0000 7fc01ec00640 1 -- start start 2024-03-31T15:12:35.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:35.019+0000 7fc01ec00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc0180cc9d0 0x7fc018154c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:35.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:35.019+0000 7fc01ec00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc01813dff0 0x7fc01813ced0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:35.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:35.019+0000 7fc01ec00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fc01813e3d0 0x7fc01813b450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:35.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:35.019+0000 7fc01ec00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fc0180cf710 con 0x7fc01813dff0 2024-03-31T15:12:35.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:35.019+0000 7fc01ec00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fc0180cf590 con 0x7fc0180cc9d0 2024-03-31T15:12:35.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:35.019+0000 7fc01ec00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fc0180cf890 con 0x7fc01813e3d0 2024-03-31T15:12:35.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:35.023+0000 7fc00fe00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc01813dff0 0x7fc01813ced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:35.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:35.023+0000 7fc00fe00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc01813dff0 0x7fc01813ced0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41290/0 (socket says 172.21.15.16:41290) 2024-03-31T15:12:35.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:35.023+0000 7fc00fe00640 1 -- 172.21.15.16:0/3614337826 learned_addr learned my addr 172.21.15.16:0/3614337826 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:12:35.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:35.023+0000 7fc017e00640 1 --2- 172.21.15.16:0/3614337826 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc0180cc9d0 0x7fc018154c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:35.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:35.023+0000 7fc01cc00640 1 --2- 172.21.15.16:0/3614337826 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fc01813e3d0 0x7fc01813b450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:35.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:35.023+0000 7fc00fe00640 1 -- 172.21.15.16:0/3614337826 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fc01813e3d0 msgr2=0x7fc01813b450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:35.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:35.023+0000 7fc00fe00640 1 --2- 172.21.15.16:0/3614337826 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fc01813e3d0 0x7fc01813b450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:35.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:35.023+0000 7fc00fe00640 1 -- 172.21.15.16:0/3614337826 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc0180cc9d0 msgr2=0x7fc018154c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:35.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:35.023+0000 7fc00fe00640 1 --2- 172.21.15.16:0/3614337826 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc0180cc9d0 0x7fc018154c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:35.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:35.023+0000 7fc00fe00640 1 -- 172.21.15.16:0/3614337826 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc01813be30 con 0x7fc01813dff0 2024-03-31T15:12:35.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:35.023+0000 7fc00fe00640 1 --2- 172.21.15.16:0/3614337826 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc01813dff0 0x7fc01813ced0 secure :-1 s=READY pgs=1481 cs=0 l=1 rev1=1 crypto rx=0x7fc00000e8a0 tx=0x7fc00000eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:12:35.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:35.023+0000 7fc014c00640 1 -- 172.21.15.16:0/3614337826 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc000017070 con 0x7fc01813dff0 2024-03-31T15:12:35.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:35.023+0000 7fc014c00640 1 -- 172.21.15.16:0/3614337826 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc00000c940 con 0x7fc01813dff0 2024-03-31T15:12:35.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:35.023+0000 7fc014c00640 1 -- 172.21.15.16:0/3614337826 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc00000ccf0 con 0x7fc01813dff0 2024-03-31T15:12:35.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:35.023+0000 7fc01ec00640 1 -- 172.21.15.16:0/3614337826 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc01813c110 con 0x7fc01813dff0 2024-03-31T15:12:35.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:35.023+0000 7fc01ec00640 1 -- 172.21.15.16:0/3614337826 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc01813c6d0 con 0x7fc01813dff0 2024-03-31T15:12:35.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:35.023+0000 7fc01ec00640 1 -- 172.21.15.16:0/3614337826 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc018069950 con 0x7fc01813dff0 2024-03-31T15:12:35.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:35.023+0000 7fc014c00640 1 -- 172.21.15.16:0/3614337826 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fc00001a070 con 0x7fc01813dff0 2024-03-31T15:12:35.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:35.023+0000 7fc014c00640 1 --2- 172.21.15.16:0/3614337826 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fbff4068380 0x7fbff406a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:35.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:35.027+0000 7fc014c00640 1 -- 172.21.15.16:0/3614337826 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 5709+0+0 (secure 0 0 0) 0x7fc000086c10 con 0x7fc01813dff0 2024-03-31T15:12:35.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:35.027+0000 7fc017e00640 1 --2- 172.21.15.16:0/3614337826 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fbff4068380 0x7fbff406a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:35.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:35.027+0000 7fc017e00640 1 --2- 172.21.15.16:0/3614337826 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fbff4068380 0x7fbff406a840 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7fc0080ad000 tx=0x7fc0080a6000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:12:35.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:35.027+0000 7fc014c00640 1 -- 172.21.15.16:0/3614337826 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fc000056ae0 con 0x7fc01813dff0 2024-03-31T15:12:35.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:35.275+0000 7fc01ec00640 1 -- 172.21.15.16:0/3614337826 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fc0180b1ba0 con 0x7fc01813dff0 2024-03-31T15:12:35.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:35.279+0000 7fc014c00640 1 -- 172.21.15.16:0/3614337826 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v96) ==== 104+0+11 (secure 0 0 0) 0x7fc00005ade0 con 0x7fc01813dff0 2024-03-31T15:12:35.270 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2024-03-31T15:12:35.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:35.279+0000 7fc01ec00640 1 -- 172.21.15.16:0/3614337826 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fbff4068380 msgr2=0x7fbff406a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:35.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:35.279+0000 7fc01ec00640 1 --2- 172.21.15.16:0/3614337826 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fbff4068380 0x7fbff406a840 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7fc0080ad000 tx=0x7fc0080a6000 comp rx=0 tx=0).stop 2024-03-31T15:12:35.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:35.279+0000 7fc01ec00640 1 -- 172.21.15.16:0/3614337826 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc01813dff0 msgr2=0x7fc01813ced0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:35.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:35.279+0000 7fc01ec00640 1 --2- 172.21.15.16:0/3614337826 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc01813dff0 0x7fc01813ced0 secure :-1 s=READY pgs=1481 cs=0 l=1 rev1=1 crypto rx=0x7fc00000e8a0 tx=0x7fc00000eca0 comp rx=0 tx=0).stop 2024-03-31T15:12:35.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:35.279+0000 7fc01ec00640 1 -- 172.21.15.16:0/3614337826 shutdown_connections 2024-03-31T15:12:35.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:35.279+0000 7fc01ec00640 1 --2- 172.21.15.16:0/3614337826 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fbff4068380 0x7fbff406a840 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:35.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:35.279+0000 7fc01ec00640 1 --2- 172.21.15.16:0/3614337826 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fc01813e3d0 0x7fc01813b450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:35.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:35.279+0000 7fc01ec00640 1 --2- 172.21.15.16:0/3614337826 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc01813dff0 0x7fc01813ced0 unknown :-1 s=CLOSED pgs=1481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:35.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:35.279+0000 7fc01ec00640 1 --2- 172.21.15.16:0/3614337826 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc0180cc9d0 0x7fc018154c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:35.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:35.279+0000 7fc01ec00640 1 -- 172.21.15.16:0/3614337826 >> 172.21.15.16:0/3614337826 conn(0x7fc01806ee40 msgr2=0x7fc0180b3560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:12:35.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:35.279+0000 7fc01ec00640 1 -- 172.21.15.16:0/3614337826 shutdown_connections 2024-03-31T15:12:35.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:35.279+0000 7fc01ec00640 1 -- 172.21.15.16:0/3614337826 wait complete. 2024-03-31T15:12:35.291 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-31T15:12:35.508 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:35.515+0000 7fa84e400640 1 Processor -- start 2024-03-31T15:12:35.508 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:35.515+0000 7fa84e400640 1 -- start start 2024-03-31T15:12:35.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:35.515+0000 7fa84e400640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fa84806f920 0x7fa84806fd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:35.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:35.515+0000 7fa84e400640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa848070240 con 0x7fa8480cf150 2024-03-31T15:12:35.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:35.515+0000 7fa84e400640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa84815ca70 con 0x7fa8480d0a60 2024-03-31T15:12:35.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:35.515+0000 7fa84e400640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7fa84813f670 con 0x7fa84806f920 2024-03-31T15:12:35.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:35.515+0000 7fa847400640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fa84806f920 0x7fa84806fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:35.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:35.515+0000 7fa847e00640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fa8480cf150 0x7fa8481419c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:42790/0 (socket says 172.21.15.16:42790) 2024-03-31T15:12:35.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:35.515+0000 7fa847e00640 1 -- 172.21.15.16:0/1606365310 learned_addr learned my addr 172.21.15.16:0/1606365310 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:12:35.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:35.519+0000 7fa846a00640 1 -- 172.21.15.16:0/1606365310 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1393559759 0 0) 0x7fa848070240 con 0x7fa8480cf150 2024-03-31T15:12:35.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:35.519+0000 7fa846a00640 1 -- 172.21.15.16:0/1606365310 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa82c003530 con 0x7fa8480cf150 2024-03-31T15:12:35.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:35.519+0000 7fa847400640 1 -- 172.21.15.16:0/1606365310 >> v1:172.21.15.89:6789/0 conn(0x7fa8480d0a60 legacy=0x7fa8480cea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:35.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:35.519+0000 7fa847400640 1 -- 172.21.15.16:0/1606365310 >> v1:172.21.15.16:6789/0 conn(0x7fa8480cf150 legacy=0x7fa8481419c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:35.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:35.519+0000 7fa847400640 1 -- 172.21.15.16:0/1606365310 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa84815cd50 con 0x7fa84806f920 2024-03-31T15:12:35.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:35.519+0000 7fa847400640 1 --2- 172.21.15.16:0/1606365310 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa84806f920 0x7fa84806fd00 secure :-1 s=READY pgs=1483 cs=0 l=1 rev1=1 crypto rx=0x7fa8380097c0 tx=0x7fa83801c410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=64f4c52aefa7af3b server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:12:35.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:35.519+0000 7fa846a00640 1 -- 172.21.15.16:0/1606365310 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8380b0070 con 0x7fa84806f920 2024-03-31T15:12:35.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:35.519+0000 7fa846a00640 1 -- 172.21.15.16:0/1606365310 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa8380ab600 con 0x7fa84806f920 2024-03-31T15:12:35.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:35.519+0000 7fa846a00640 1 -- 172.21.15.16:0/1606365310 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8380ab9b0 con 0x7fa84806f920 2024-03-31T15:12:35.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:35.523+0000 7fa84e400640 1 -- 172.21.15.16:0/1606365310 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa84806f920 msgr2=0x7fa84806fd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:35.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:35.523+0000 7fa84e400640 1 --2- 172.21.15.16:0/1606365310 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa84806f920 0x7fa84806fd00 secure :-1 s=READY pgs=1483 cs=0 l=1 rev1=1 crypto rx=0x7fa8380097c0 tx=0x7fa83801c410 comp rx=0 tx=0).stop 2024-03-31T15:12:35.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:35.523+0000 7fa84e400640 1 -- 172.21.15.16:0/1606365310 shutdown_connections 2024-03-31T15:12:35.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:35.523+0000 7fa84e400640 1 --2- 172.21.15.16:0/1606365310 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa84806f920 0x7fa84806fd00 unknown :-1 s=CLOSED pgs=1483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:35.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:35.523+0000 7fa84e400640 1 -- 172.21.15.16:0/1606365310 >> 172.21.15.16:0/1606365310 conn(0x7fa848067930 msgr2=0x7fa8480b3ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:12:35.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:35.523+0000 7fa84e400640 1 -- 172.21.15.16:0/1606365310 shutdown_connections 2024-03-31T15:12:35.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:35.523+0000 7fa84e400640 1 -- 172.21.15.16:0/1606365310 wait complete. 2024-03-31T15:12:35.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:35.523+0000 7fa84e400640 1 Processor -- start 2024-03-31T15:12:35.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:35.523+0000 7fa84e400640 1 -- start start 2024-03-31T15:12:35.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:35.523+0000 7fa84e400640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa84806f920 0x7fa848140990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:35.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:35.523+0000 7fa84e400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa8480cf150 0x7fa84813eb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:35.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:35.523+0000 7fa84e400640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fa8480d0a60 0x7fa84806e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:35.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:35.523+0000 7fa84e400640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fa84816a240 con 0x7fa8480cf150 2024-03-31T15:12:35.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:35.523+0000 7fa84e400640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fa84816a0c0 con 0x7fa84806f920 2024-03-31T15:12:35.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:35.523+0000 7fa84e400640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fa84816a3c0 con 0x7fa8480d0a60 2024-03-31T15:12:35.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:35.523+0000 7fa83fe00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa8480cf150 0x7fa84813eb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:35.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:35.523+0000 7fa83fe00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa8480cf150 0x7fa84813eb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41306/0 (socket says 172.21.15.16:41306) 2024-03-31T15:12:35.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:35.523+0000 7fa83fe00640 1 -- 172.21.15.16:0/3967363996 learned_addr learned my addr 172.21.15.16:0/3967363996 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:12:35.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:35.523+0000 7fa847400640 1 --2- 172.21.15.16:0/3967363996 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa84806f920 0x7fa848140990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:35.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:35.523+0000 7fa847e00640 1 --2- 172.21.15.16:0/3967363996 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fa8480d0a60 0x7fa84806e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:35.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:35.523+0000 7fa83fe00640 1 -- 172.21.15.16:0/3967363996 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fa8480d0a60 msgr2=0x7fa84806e020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:35.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:35.523+0000 7fa83fe00640 1 --2- 172.21.15.16:0/3967363996 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fa8480d0a60 0x7fa84806e020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:35.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:35.523+0000 7fa83fe00640 1 -- 172.21.15.16:0/3967363996 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa84806f920 msgr2=0x7fa848140990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:35.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:35.523+0000 7fa83fe00640 1 --2- 172.21.15.16:0/3967363996 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa84806f920 0x7fa848140990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:35.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:35.523+0000 7fa83fe00640 1 -- 172.21.15.16:0/3967363996 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa84814f350 con 0x7fa8480cf150 2024-03-31T15:12:35.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:35.527+0000 7fa83fe00640 1 --2- 172.21.15.16:0/3967363996 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa8480cf150 0x7fa84813eb90 secure :-1 s=READY pgs=1484 cs=0 l=1 rev1=1 crypto rx=0x7fa83000c3c0 tx=0x7fa83000c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:12:35.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:35.527+0000 7fa83f400640 1 -- 172.21.15.16:0/3967363996 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa830017070 con 0x7fa8480cf150 2024-03-31T15:12:35.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:35.527+0000 7fa83f400640 1 -- 172.21.15.16:0/3967363996 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa8300052d0 con 0x7fa8480cf150 2024-03-31T15:12:35.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:35.527+0000 7fa83f400640 1 -- 172.21.15.16:0/3967363996 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa830003820 con 0x7fa8480cf150 2024-03-31T15:12:35.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:35.527+0000 7fa84e400640 1 -- 172.21.15.16:0/3967363996 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa8480b5f60 con 0x7fa8480cf150 2024-03-31T15:12:35.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:35.527+0000 7fa84e400640 1 -- 172.21.15.16:0/3967363996 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa8481411a0 con 0x7fa8480cf150 2024-03-31T15:12:35.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:35.527+0000 7fa84e400640 1 -- 172.21.15.16:0/3967363996 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa848069950 con 0x7fa8480cf150 2024-03-31T15:12:35.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:35.527+0000 7fa83f400640 1 -- 172.21.15.16:0/3967363996 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fa83001a070 con 0x7fa8480cf150 2024-03-31T15:12:35.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:35.527+0000 7fa83f400640 1 --2- 172.21.15.16:0/3967363996 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fa824068380 0x7fa82406a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:35.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:35.531+0000 7fa83f400640 1 -- 172.21.15.16:0/3967363996 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 5709+0+0 (secure 0 0 0) 0x7fa830085bb0 con 0x7fa8480cf150 2024-03-31T15:12:35.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:35.531+0000 7fa847400640 1 --2- 172.21.15.16:0/3967363996 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fa824068380 0x7fa82406a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:35.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:35.531+0000 7fa847400640 1 --2- 172.21.15.16:0/3967363996 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fa824068380 0x7fa82406a840 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7fa8380ad000 tx=0x7fa8380a6000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:12:35.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:35.535+0000 7fa83f400640 1 -- 172.21.15.16:0/3967363996 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fa830005480 con 0x7fa8480cf150 2024-03-31T15:12:35.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:35.699+0000 7fa84e400640 1 -- 172.21.15.16:0/3967363996 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fa8480b50e0 con 0x7fa8480cf150 2024-03-31T15:12:35.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:35.699+0000 7fa83f400640 1 -- 172.21.15.16:0/3967363996 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v96) ==== 100+0+11 (secure 0 0 0) 0x7fa830055b10 con 0x7fa8480cf150 2024-03-31T15:12:35.692 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 64 2024-03-31T15:12:35.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:35.699+0000 7fa84e400640 1 -- 172.21.15.16:0/3967363996 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fa824068380 msgr2=0x7fa82406a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:35.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:35.699+0000 7fa84e400640 1 --2- 172.21.15.16:0/3967363996 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fa824068380 0x7fa82406a840 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7fa8380ad000 tx=0x7fa8380a6000 comp rx=0 tx=0).stop 2024-03-31T15:12:35.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:35.699+0000 7fa84e400640 1 -- 172.21.15.16:0/3967363996 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa8480cf150 msgr2=0x7fa84813eb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:35.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:35.699+0000 7fa84e400640 1 --2- 172.21.15.16:0/3967363996 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa8480cf150 0x7fa84813eb90 secure :-1 s=READY pgs=1484 cs=0 l=1 rev1=1 crypto rx=0x7fa83000c3c0 tx=0x7fa83000c880 comp rx=0 tx=0).stop 2024-03-31T15:12:35.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:35.699+0000 7fa84e400640 1 -- 172.21.15.16:0/3967363996 shutdown_connections 2024-03-31T15:12:35.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:35.699+0000 7fa84e400640 1 --2- 172.21.15.16:0/3967363996 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fa824068380 0x7fa82406a840 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:35.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:35.699+0000 7fa84e400640 1 --2- 172.21.15.16:0/3967363996 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fa8480d0a60 0x7fa84806e020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:35.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:35.699+0000 7fa84e400640 1 --2- 172.21.15.16:0/3967363996 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa8480cf150 0x7fa84813eb90 unknown :-1 s=CLOSED pgs=1484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:35.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:35.699+0000 7fa84e400640 1 --2- 172.21.15.16:0/3967363996 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa84806f920 0x7fa848140990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:35.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:35.699+0000 7fa84e400640 1 -- 172.21.15.16:0/3967363996 >> 172.21.15.16:0/3967363996 conn(0x7fa848067930 msgr2=0x7fa8481520b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:12:35.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:35.703+0000 7fa84e400640 1 -- 172.21.15.16:0/3967363996 shutdown_connections 2024-03-31T15:12:35.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:35.703+0000 7fa84e400640 1 -- 172.21.15.16:0/3967363996 wait complete. 2024-03-31T15:12:35.722 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-31T15:12:35.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:35.871+0000 7fa3b0a00640 1 Processor -- start 2024-03-31T15:12:35.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:35.871+0000 7fa3b0a00640 1 -- start start 2024-03-31T15:12:35.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:35.871+0000 7fa3b0a00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fa3ac06f920 0x7fa3ac06fd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:35.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:35.871+0000 7fa3b0a00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fa3ac0d2010 0x7fa3ac0d24a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:35.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:35.871+0000 7fa3b0a00640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa3ac0d0920 con 0x7fa3ac145e40 2024-03-31T15:12:35.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:35.871+0000 7fa3b0a00640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7fa3ac0702c0 con 0x7fa3ac0d2010 2024-03-31T15:12:35.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:35.871+0000 7fa3b0a00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fa3ac0d06d0 con 0x7fa3ac06f920 2024-03-31T15:12:35.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:35.871+0000 7fa3aa800640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fa3ac0d2010 0x7fa3ac0d24a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:35.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:35.871+0000 7fa3aa800640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fa3ac0d2010 0x7fa3ac0d24a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41316/0 (socket says 172.21.15.16:41316) 2024-03-31T15:12:35.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:35.871+0000 7fa3aa800640 1 -- 172.21.15.16:0/302389235 learned_addr learned my addr 172.21.15.16:0/302389235 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:12:35.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:35.871+0000 7fa3a3e00640 1 --2- 172.21.15.16:0/302389235 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fa3ac06f920 0x7fa3ac06fd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:35.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:35.871+0000 7fa3aa800640 1 -- 172.21.15.16:0/302389235 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fa3ac06f920 msgr2=0x7fa3ac06fd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:12:35.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:35.871+0000 7fa3aa800640 1 --2- 172.21.15.16:0/302389235 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fa3ac06f920 0x7fa3ac06fd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:35.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:35.871+0000 7fa3aa800640 1 -- 172.21.15.16:0/302389235 >> v1:172.21.15.89:6789/0 conn(0x7fa3ac145e40 legacy=0x7fa3ac0cc740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:35.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:35.871+0000 7fa3aa800640 1 -- 172.21.15.16:0/302389235 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa3ac0d0c00 con 0x7fa3ac0d2010 2024-03-31T15:12:35.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:35.875+0000 7fa3aa800640 1 --2- 172.21.15.16:0/302389235 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa3ac0d2010 0x7fa3ac0d24a0 secure :-1 s=READY pgs=1485 cs=0 l=1 rev1=1 crypto rx=0x7fa3a4009e50 tx=0x7fa3a401ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd56d9b22c851b19 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:12:35.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:35.875+0000 7fa3a9400640 1 -- 172.21.15.16:0/302389235 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3a40a57f0 con 0x7fa3ac0d2010 2024-03-31T15:12:35.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:35.875+0000 7fa3a9400640 1 -- 172.21.15.16:0/302389235 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa3a40a5990 con 0x7fa3ac0d2010 2024-03-31T15:12:35.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:35.875+0000 7fa3a9400640 1 -- 172.21.15.16:0/302389235 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3a40a5d60 con 0x7fa3ac0d2010 2024-03-31T15:12:35.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:35.875+0000 7fa3b0a00640 1 -- 172.21.15.16:0/302389235 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa3ac0d2010 msgr2=0x7fa3ac0d24a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:35.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:35.875+0000 7fa3b0a00640 1 --2- 172.21.15.16:0/302389235 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa3ac0d2010 0x7fa3ac0d24a0 secure :-1 s=READY pgs=1485 cs=0 l=1 rev1=1 crypto rx=0x7fa3a4009e50 tx=0x7fa3a401ca30 comp rx=0 tx=0).stop 2024-03-31T15:12:35.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:35.875+0000 7fa3b0a00640 1 -- 172.21.15.16:0/302389235 shutdown_connections 2024-03-31T15:12:35.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:35.875+0000 7fa3b0a00640 1 --2- 172.21.15.16:0/302389235 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa3ac0d2010 0x7fa3ac0d24a0 unknown :-1 s=CLOSED pgs=1485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:35.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:35.875+0000 7fa3b0a00640 1 --2- 172.21.15.16:0/302389235 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fa3ac06f920 0x7fa3ac06fd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:35.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:35.875+0000 7fa3b0a00640 1 -- 172.21.15.16:0/302389235 >> 172.21.15.16:0/302389235 conn(0x7fa3ac067330 msgr2=0x7fa3ac067740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:12:35.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:35.875+0000 7fa3b0a00640 1 -- 172.21.15.16:0/302389235 shutdown_connections 2024-03-31T15:12:35.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:35.875+0000 7fa3b0a00640 1 -- 172.21.15.16:0/302389235 wait complete. 2024-03-31T15:12:35.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:35.879+0000 7fa3b0a00640 1 Processor -- start 2024-03-31T15:12:35.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:35.879+0000 7fa3b0a00640 1 -- start start 2024-03-31T15:12:35.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:35.879+0000 7fa3b0a00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fa3ac06f920 0x7fa3ac0b7130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:35.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:35.879+0000 7fa3b0a00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa3ac0d2010 0x7fa3ac0b1e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:35.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:35.879+0000 7fa3b0a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa3ac145e40 0x7fa3ac141750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:35.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:35.879+0000 7fa3b0a00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fa3ac0d0ee0 con 0x7fa3ac145e40 2024-03-31T15:12:35.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:35.879+0000 7fa3b0a00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fa3ac0ce9e0 con 0x7fa3ac0d2010 2024-03-31T15:12:35.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:35.879+0000 7fa3b0a00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fa3ac0d2df0 con 0x7fa3ac06f920 2024-03-31T15:12:35.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:35.879+0000 7fa3aa800640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa3ac145e40 0x7fa3ac141750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:35.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:35.879+0000 7fa3aa800640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa3ac145e40 0x7fa3ac141750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41326/0 (socket says 172.21.15.16:41326) 2024-03-31T15:12:35.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:35.879+0000 7fa3aa800640 1 -- 172.21.15.16:0/1111570681 learned_addr learned my addr 172.21.15.16:0/1111570681 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:12:35.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:35.879+0000 7fa3a3e00640 1 --2- 172.21.15.16:0/1111570681 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa3ac0d2010 0x7fa3ac0b1e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:35.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:35.879+0000 7fa3a9e00640 1 --2- 172.21.15.16:0/1111570681 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fa3ac06f920 0x7fa3ac0b7130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:35.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:35.879+0000 7fa3aa800640 1 -- 172.21.15.16:0/1111570681 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fa3ac06f920 msgr2=0x7fa3ac0b7130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:35.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:35.879+0000 7fa3aa800640 1 --2- 172.21.15.16:0/1111570681 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fa3ac06f920 0x7fa3ac0b7130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:35.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:35.879+0000 7fa3aa800640 1 -- 172.21.15.16:0/1111570681 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa3ac0d2010 msgr2=0x7fa3ac0b1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:35.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:35.879+0000 7fa3aa800640 1 --2- 172.21.15.16:0/1111570681 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa3ac0d2010 0x7fa3ac0b1e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:35.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:35.879+0000 7fa3aa800640 1 -- 172.21.15.16:0/1111570681 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa3ac0b5b70 con 0x7fa3ac145e40 2024-03-31T15:12:35.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:35.879+0000 7fa3a9e00640 1 --2- 172.21.15.16:0/1111570681 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fa3ac06f920 0x7fa3ac0b7130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:12:35.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:35.879+0000 7fa3aa800640 1 --2- 172.21.15.16:0/1111570681 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa3ac145e40 0x7fa3ac141750 secure :-1 s=READY pgs=1486 cs=0 l=1 rev1=1 crypto rx=0x7fa3a4009e50 tx=0x7fa3a40197d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:12:35.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:35.879+0000 7fa3a2000640 1 -- 172.21.15.16:0/1111570681 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3a40b8670 con 0x7fa3ac145e40 2024-03-31T15:12:35.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:35.879+0000 7fa3a2000640 1 -- 172.21.15.16:0/1111570681 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa3a40be070 con 0x7fa3ac145e40 2024-03-31T15:12:35.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:35.879+0000 7fa3a2000640 1 -- 172.21.15.16:0/1111570681 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3a40b8a70 con 0x7fa3ac145e40 2024-03-31T15:12:35.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:35.879+0000 7fa3b0a00640 1 -- 172.21.15.16:0/1111570681 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa3ac0b5e50 con 0x7fa3ac145e40 2024-03-31T15:12:35.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:35.879+0000 7fa3b0a00640 1 -- 172.21.15.16:0/1111570681 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa3ac0b6410 con 0x7fa3ac145e40 2024-03-31T15:12:35.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:35.883+0000 7fa3b0a00640 1 -- 172.21.15.16:0/1111570681 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa3ac069950 con 0x7fa3ac145e40 2024-03-31T15:12:35.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:35.883+0000 7fa3a2000640 1 -- 172.21.15.16:0/1111570681 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fa3a40b8c10 con 0x7fa3ac145e40 2024-03-31T15:12:35.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:35.883+0000 7fa3a2000640 1 --2- 172.21.15.16:0/1111570681 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fa388068380 0x7fa38806a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:35.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:35.887+0000 7fa3a2000640 1 -- 172.21.15.16:0/1111570681 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 5709+0+0 (secure 0 0 0) 0x7fa3a4121c00 con 0x7fa3ac145e40 2024-03-31T15:12:35.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:35.887+0000 7fa3a9e00640 1 --2- 172.21.15.16:0/1111570681 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fa388068380 0x7fa38806a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:35.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:35.887+0000 7fa3a9e00640 1 --2- 172.21.15.16:0/1111570681 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fa388068380 0x7fa38806a840 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7fa39c001650 tx=0x7fa39c009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:12:35.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:35.891+0000 7fa3a2000640 1 -- 172.21.15.16:0/1111570681 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fa3a4128080 con 0x7fa3ac145e40 2024-03-31T15:12:36.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:36.067+0000 7fa3b0a00640 1 -- 172.21.15.16:0/1111570681 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fa3ac0cf3c0 con 0x7fa3ac145e40 2024-03-31T15:12:36.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:36.067+0000 7fa3a2000640 1 -- 172.21.15.16:0/1111570681 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v96) ==== 80+0+0 (secure 0 0 0) 0x7fa3a4016620 con 0x7fa3ac145e40 2024-03-31T15:12:36.061 INFO:teuthology.orchestra.run.smithi016.stderr:listed 0 entries 2024-03-31T15:12:36.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:36.071+0000 7fa3b0a00640 1 -- 172.21.15.16:0/1111570681 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fa388068380 msgr2=0x7fa38806a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:36.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:36.071+0000 7fa3b0a00640 1 --2- 172.21.15.16:0/1111570681 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fa388068380 0x7fa38806a840 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7fa39c001650 tx=0x7fa39c009040 comp rx=0 tx=0).stop 2024-03-31T15:12:36.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:36.071+0000 7fa3b0a00640 1 -- 172.21.15.16:0/1111570681 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa3ac145e40 msgr2=0x7fa3ac141750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:36.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:36.071+0000 7fa3b0a00640 1 --2- 172.21.15.16:0/1111570681 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa3ac145e40 0x7fa3ac141750 secure :-1 s=READY pgs=1486 cs=0 l=1 rev1=1 crypto rx=0x7fa3a4009e50 tx=0x7fa3a40197d0 comp rx=0 tx=0).stop 2024-03-31T15:12:36.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:36.071+0000 7fa3b0a00640 1 -- 172.21.15.16:0/1111570681 shutdown_connections 2024-03-31T15:12:36.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:36.071+0000 7fa3b0a00640 1 --2- 172.21.15.16:0/1111570681 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fa388068380 0x7fa38806a840 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:36.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:36.071+0000 7fa3b0a00640 1 --2- 172.21.15.16:0/1111570681 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa3ac145e40 0x7fa3ac141750 unknown :-1 s=CLOSED pgs=1486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:36.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:36.071+0000 7fa3b0a00640 1 --2- 172.21.15.16:0/1111570681 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa3ac0d2010 0x7fa3ac0b1e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:36.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:36.071+0000 7fa3b0a00640 1 --2- 172.21.15.16:0/1111570681 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fa3ac06f920 0x7fa3ac0b7130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:36.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:36.071+0000 7fa3b0a00640 1 -- 172.21.15.16:0/1111570681 >> 172.21.15.16:0/1111570681 conn(0x7fa3ac067330 msgr2=0x7fa3ac15b370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:12:36.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:36.071+0000 7fa3b0a00640 1 -- 172.21.15.16:0/1111570681 shutdown_connections 2024-03-31T15:12:36.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:36.071+0000 7fa3b0a00640 1 -- 172.21.15.16:0/1111570681 wait complete. 2024-03-31T15:12:36.084 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi016.front.sepia.ceph.com... 2024-03-31T15:12:36.085 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:12:36.085 DEBUG:teuthology.orchestra.run.smithi016:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-03-31T15:12:36.102 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi016.stderr:ceph-fuse[48731]: fuse finished with error 0 and tester_r 0 2024-03-31T15:12:36.108 INFO:teuthology.orchestra.run:waiting for 300 2024-03-31T15:12:37.667 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-31T15:12:37.670 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-31T15:12:42.210 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi016.front.sepia.ceph.com 2024-03-31T15:12:42.210 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:12:42.210 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-31T15:12:42.219 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-31T15:12:42.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:42.407+0000 7f3076000640 1 Processor -- start 2024-03-31T15:12:42.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:42.407+0000 7f3076000640 1 -- start start 2024-03-31T15:12:42.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:42.411+0000 7f3076000640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f3070147dd0 0x7f307006c730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:42.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:42.411+0000 7f3076000640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3070147660 con 0x7f3070147280 2024-03-31T15:12:42.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:42.411+0000 7f3076000640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3070162340 con 0x7f3070070030 2024-03-31T15:12:42.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:42.411+0000 7f3076000640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f30701620f0 con 0x7f3070147dd0 2024-03-31T15:12:42.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:42.411+0000 7f306f400640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f3070147280 0x7f307006f920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:36410/0 (socket says 172.21.15.16:36410) 2024-03-31T15:12:42.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:42.411+0000 7f306f400640 1 -- 172.21.15.16:0/4106521549 learned_addr learned my addr 172.21.15.16:0/4106521549 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:12:42.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:42.411+0000 7f306ea00640 1 -- 172.21.15.16:0/4106521549 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2253374886 0 0) 0x7f3070147660 con 0x7f3070147280 2024-03-31T15:12:42.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:42.411+0000 7f306ea00640 1 -- 172.21.15.16:0/4106521549 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3054003530 con 0x7f3070147280 2024-03-31T15:12:42.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:42.411+0000 7f306fe00640 1 --2- 172.21.15.16:0/4106521549 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f3070147dd0 0x7f307006c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:42.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:42.411+0000 7f306ea00640 1 -- 172.21.15.16:0/4106521549 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4011618161 0 0) 0x7f3054003530 con 0x7f3070147280 2024-03-31T15:12:42.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:42.411+0000 7f306ea00640 1 -- 172.21.15.16:0/4106521549 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f3070147dd0 msgr2=0x7f307006c730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:12:42.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:42.411+0000 7f306ea00640 1 --2- 172.21.15.16:0/4106521549 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f3070147dd0 0x7f307006c730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:42.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:42.411+0000 7f306ea00640 1 -- 172.21.15.16:0/4106521549 >> v1:172.21.15.89:6789/0 conn(0x7f3070070030 legacy=0x7f30700cc8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:42.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:42.411+0000 7f306ea00640 1 -- 172.21.15.16:0/4106521549 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3070162620 con 0x7f3070147280 2024-03-31T15:12:42.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:42.411+0000 7f306ea00640 1 -- 172.21.15.16:0/4106521549 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 214807623 0 0) 0x7f3060001680 con 0x7f3070147280 2024-03-31T15:12:42.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:42.411+0000 7f306ea00640 1 -- 172.21.15.16:0/4106521549 >> v1:172.21.15.16:6789/0 conn(0x7f3070147280 legacy=0x7f307006f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:42.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:42.411+0000 7f306ea00640 1 --2- 172.21.15.16:0/4106521549 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f30540039f0 0x7f3054003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:42.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:42.411+0000 7f306ea00640 1 --2- 172.21.15.16:0/4106521549 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3054004810 0x7f3054004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:42.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:42.411+0000 7f306ea00640 1 --2- 172.21.15.16:0/4106521549 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f3070147dd0 0x7f30540051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:42.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:42.411+0000 7f306ea00640 1 -- 172.21.15.16:0/4106521549 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f30701620f0 con 0x7f3054004810 2024-03-31T15:12:42.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:42.411+0000 7f306ea00640 1 -- 172.21.15.16:0/4106521549 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f305400ba80 con 0x7f30540039f0 2024-03-31T15:12:42.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:42.411+0000 7f306ea00640 1 -- 172.21.15.16:0/4106521549 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f305400bc30 con 0x7f3070147dd0 2024-03-31T15:12:42.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:42.411+0000 7f3067e00640 1 --2- 172.21.15.16:0/4106521549 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3054004810 0x7f3054004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:42.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:42.411+0000 7f306f400640 1 --2- 172.21.15.16:0/4106521549 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f30540039f0 0x7f3054003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:42.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:42.411+0000 7f306fe00640 1 --2- 172.21.15.16:0/4106521549 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f3070147dd0 0x7f30540051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:42.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:42.411+0000 7f3067e00640 1 -- 172.21.15.16:0/4106521549 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f3070147dd0 msgr2=0x7f30540051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:12:42.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:42.411+0000 7f3067e00640 1 --2- 172.21.15.16:0/4106521549 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f3070147dd0 0x7f30540051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:42.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:42.411+0000 7f3067e00640 1 -- 172.21.15.16:0/4106521549 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f30540039f0 msgr2=0x7f3054003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:12:42.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:42.411+0000 7f3067e00640 1 --2- 172.21.15.16:0/4106521549 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f30540039f0 0x7f3054003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:42.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:42.411+0000 7f3067e00640 1 -- 172.21.15.16:0/4106521549 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3054003530 con 0x7f3054004810 2024-03-31T15:12:42.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:42.415+0000 7f3067e00640 1 --2- 172.21.15.16:0/4106521549 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3054004810 0x7f3054004c90 secure :-1 s=READY pgs=1488 cs=0 l=1 rev1=1 crypto rx=0x7f3058007fd0 tx=0x7f305801ebb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a33c8dc32c258c6 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:12:42.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:42.415+0000 7f306ea00640 1 -- 172.21.15.16:0/4106521549 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30580b5070 con 0x7f3054004810 2024-03-31T15:12:42.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:42.415+0000 7f306ea00640 1 -- 172.21.15.16:0/4106521549 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f30580a8830 con 0x7f3054004810 2024-03-31T15:12:42.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:42.415+0000 7f306ea00640 1 -- 172.21.15.16:0/4106521549 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3058001460 con 0x7f3054004810 2024-03-31T15:12:42.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:42.415+0000 7f3076000640 1 -- 172.21.15.16:0/4106521549 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3054004810 msgr2=0x7f3054004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:42.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:42.415+0000 7f3076000640 1 --2- 172.21.15.16:0/4106521549 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3054004810 0x7f3054004c90 secure :-1 s=READY pgs=1488 cs=0 l=1 rev1=1 crypto rx=0x7f3058007fd0 tx=0x7f305801ebb0 comp rx=0 tx=0).stop 2024-03-31T15:12:42.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:42.415+0000 7f306fe00640 1 -- 172.21.15.16:0/4106521549 reap_dead start 2024-03-31T15:12:42.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:42.415+0000 7f3076000640 1 -- 172.21.15.16:0/4106521549 shutdown_connections 2024-03-31T15:12:42.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:42.415+0000 7f3076000640 1 -- 172.21.15.16:0/4106521549 >> 172.21.15.16:0/4106521549 conn(0x7f3070067330 msgr2=0x7f3070067740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:12:42.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:42.415+0000 7f3076000640 1 -- 172.21.15.16:0/4106521549 shutdown_connections 2024-03-31T15:12:42.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:42.415+0000 7f3076000640 1 -- 172.21.15.16:0/4106521549 wait complete. 2024-03-31T15:12:42.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:42.415+0000 7f3076000640 1 Processor -- start 2024-03-31T15:12:42.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:42.415+0000 7f3076000640 1 -- start start 2024-03-31T15:12:42.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:42.419+0000 7f3076000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3054004810 0x7f307015deb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:42.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:42.419+0000 7f3076000640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f307013c4f0 0x7f307013c970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:42.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:42.419+0000 7f3076000640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f307013f950 0x7f307013fdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:42.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:42.419+0000 7f3076000640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f307006c730 con 0x7f3054004810 2024-03-31T15:12:42.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:42.419+0000 7f3076000640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f3070162be0 con 0x7f307013f950 2024-03-31T15:12:42.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:42.419+0000 7f3076000640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f307006c8b0 con 0x7f307013c4f0 2024-03-31T15:12:42.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:42.419+0000 7f306f400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3054004810 0x7f307015deb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:42.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:42.419+0000 7f306f400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3054004810 0x7f307015deb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43978/0 (socket says 172.21.15.16:43978) 2024-03-31T15:12:42.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:42.419+0000 7f306f400640 1 -- 172.21.15.16:0/3911283673 learned_addr learned my addr 172.21.15.16:0/3911283673 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:12:42.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:42.419+0000 7f3067e00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f307013c4f0 0x7f307013c970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:42.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:42.419+0000 7f306fe00640 1 --2- 172.21.15.16:0/3911283673 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f307013f950 0x7f307013fdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:42.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:42.419+0000 7f3067e00640 1 -- 172.21.15.16:0/3911283673 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f307013f950 msgr2=0x7f307013fdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:42.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:42.419+0000 7f3067e00640 1 --2- 172.21.15.16:0/3911283673 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f307013f950 0x7f307013fdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:42.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:42.419+0000 7f3067e00640 1 -- 172.21.15.16:0/3911283673 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3054004810 msgr2=0x7f307015deb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:42.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:42.419+0000 7f3067e00640 1 --2- 172.21.15.16:0/3911283673 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3054004810 0x7f307015deb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:42.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:42.419+0000 7f3067e00640 1 -- 172.21.15.16:0/3911283673 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f307013ceb0 con 0x7f307013c4f0 2024-03-31T15:12:42.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:42.419+0000 7f3067e00640 1 --2- 172.21.15.16:0/3911283673 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f307013c4f0 0x7f307013c970 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f3054004730 tx=0x7f3054005840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:12:42.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:42.419+0000 7f3067400640 1 -- 172.21.15.16:0/3911283673 <== mon.2 v2:172.21.15.89:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30580b5040 con 0x7f307013c4f0 2024-03-31T15:12:42.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:42.419+0000 7f3067400640 1 -- 172.21.15.16:0/3911283673 <== mon.2 v2:172.21.15.89:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f30580be070 con 0x7f307013c4f0 2024-03-31T15:12:42.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:42.419+0000 7f3076000640 1 -- 172.21.15.16:0/3911283673 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f307013e8d0 con 0x7f307013c4f0 2024-03-31T15:12:42.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:42.419+0000 7f3067400640 1 -- 172.21.15.16:0/3911283673 <== mon.2 v2:172.21.15.89:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30580a8de0 con 0x7f307013c4f0 2024-03-31T15:12:42.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:42.419+0000 7f3076000640 1 -- 172.21.15.16:0/3911283673 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f307013f150 con 0x7f307013c4f0 2024-03-31T15:12:42.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:42.419+0000 7f3076000640 1 -- 172.21.15.16:0/3911283673 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3070069950 con 0x7f307013c4f0 2024-03-31T15:12:42.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:42.423+0000 7f3067400640 1 -- 172.21.15.16:0/3911283673 <== mon.2 v2:172.21.15.89:3301/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f30580a86f0 con 0x7f307013c4f0 2024-03-31T15:12:42.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:42.423+0000 7f3067400640 1 --2- 172.21.15.16:0/3911283673 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f304c068380 0x7f304c06a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:42.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:42.423+0000 7f3067400640 1 -- 172.21.15.16:0/3911283673 <== mon.2 v2:172.21.15.89:3301/0 5 ==== osd_map(96..96 src has 1..96) ==== 5709+0+0 (secure 0 0 0) 0x7f3058123d20 con 0x7f307013c4f0 2024-03-31T15:12:42.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:42.423+0000 7f306f400640 1 --2- 172.21.15.16:0/3911283673 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f304c068380 0x7f304c06a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:42.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:42.423+0000 7f306f400640 1 --2- 172.21.15.16:0/3911283673 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f304c068380 0x7f304c06a840 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f3070146350 tx=0x7f30600015a0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:12:42.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:42.427+0000 7f3067400640 1 -- 172.21.15.16:0/3911283673 <== mon.2 v2:172.21.15.89:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f3058018920 con 0x7f307013c4f0 2024-03-31T15:12:42.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:42.627+0000 7f3076000640 1 -- 172.21.15.16:0/3911283673 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f30700cf3e0 con 0x7f307013c4f0 2024-03-31T15:12:42.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:42.627+0000 7f3067400640 1 -- 172.21.15.16:0/3911283673 <== mon.2 v2:172.21.15.89:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 83 v83) ==== 94+0+5180 (secure 0 0 0) 0x7f30580f3c70 con 0x7f307013c4f0 2024-03-31T15:12:42.621 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:12:42.621 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":83,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7139,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6836/3798928623","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":3798928623},{"type":"v1","addr":"172.21.15.89:6836","nonce":3798928623}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":78},{"gid":7142,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/3074283817","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6835","nonce":3074283817},{"type":"v1","addr":"172.21.15.89:6837","nonce":3074283817}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":78},{"gid":8071,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6834/2110189998","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":2110189998},{"type":"v1","addr":"172.21.15.16:6834","nonce":2110189998}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":78}],"filesystems":[{"mdsmap":{"epoch":83,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-31T15:12:16.294600+0000","modified":"2024-03-31T15:12:19.297339+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8074},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8074":{"gid":8074,"name":"a","rank":0,"incarnation":79,"state":"up:active","state_seq":3,"addr":"172.21.15.16:6835/490717480","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6833","nonce":490717480},{"type":"v1","addr":"172.21.15.16:6835","nonce":490717480}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8074,"qdb_cluster":[8074]},"id":7}]} 2024-03-31T15:12:42.621 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 83 2024-03-31T15:12:42.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:42.631+0000 7f3076000640 1 -- 172.21.15.16:0/3911283673 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f304c068380 msgr2=0x7f304c06a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:42.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:42.631+0000 7f3076000640 1 --2- 172.21.15.16:0/3911283673 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f304c068380 0x7f304c06a840 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f3070146350 tx=0x7f30600015a0 comp rx=0 tx=0).stop 2024-03-31T15:12:42.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:42.631+0000 7f3076000640 1 -- 172.21.15.16:0/3911283673 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f307013c4f0 msgr2=0x7f307013c970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:42.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:42.631+0000 7f3076000640 1 --2- 172.21.15.16:0/3911283673 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f307013c4f0 0x7f307013c970 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f3054004730 tx=0x7f3054005840 comp rx=0 tx=0).stop 2024-03-31T15:12:42.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:42.631+0000 7f3076000640 1 -- 172.21.15.16:0/3911283673 shutdown_connections 2024-03-31T15:12:42.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:42.631+0000 7f3076000640 1 --2- 172.21.15.16:0/3911283673 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f304c068380 0x7f304c06a840 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:42.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:42.631+0000 7f3076000640 1 --2- 172.21.15.16:0/3911283673 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f307013f950 0x7f307013fdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:42.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:42.631+0000 7f3076000640 1 --2- 172.21.15.16:0/3911283673 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f307013c4f0 0x7f307013c970 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:42.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:42.631+0000 7f3076000640 1 --2- 172.21.15.16:0/3911283673 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3054004810 0x7f307015deb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:42.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:42.631+0000 7f3076000640 1 -- 172.21.15.16:0/3911283673 >> 172.21.15.16:0/3911283673 conn(0x7f3070067330 msgr2=0x7f3070145500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:12:42.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:42.631+0000 7f3076000640 1 -- 172.21.15.16:0/3911283673 shutdown_connections 2024-03-31T15:12:42.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:42.631+0000 7f3076000640 1 -- 172.21.15.16:0/3911283673 wait complete. 2024-03-31T15:12:42.643 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json flush journal 2024-03-31T15:12:42.815 INFO:teuthology.orchestra.run.smithi016.stdout:{"message":"","return_code":0} 2024-03-31T15:12:42.816 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2024-03-31T15:12:42.816 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-03-31T15:12:42.816 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-31T15:12:42.824 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16357708k,nr_inodes=4089427,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275996k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=6539 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275992k,nr_inodes=818998,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275996k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-31T15:12:42.824 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse client.0 is not mounted at ubuntu@smithi016.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.0 2024-03-31T15:12:42.824 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi016.front.sepia.ceph.com 2024-03-31T15:12:42.825 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:12:42.825 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-31T15:12:42.872 INFO:teuthology.orchestra.run.smithi016.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-03-31T15:12:42.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-31T15:12:42.874 DEBUG:teuthology.orchestra.run.smithi016:> sudo daemon-helper kill ceph --cluster ceph -w 2024-03-31T15:12:42.925 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:12:42.926 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.7:0 scrub start / repair,recursive 2024-03-31T15:12:43.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.079+0000 7f0e5d800640 1 Processor -- start 2024-03-31T15:12:43.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.083+0000 7f0e5d800640 1 -- start start 2024-03-31T15:12:43.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.083+0000 7f0e5d800640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f0e5806c960 0x7f0e580d00a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:43.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.083+0000 7f0e5d800640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f0e5806ecf0 0x7f0e5806f100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:43.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.083+0000 7f0e5d800640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0e58162410 con 0x7f0e58161e70 2024-03-31T15:12:43.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.083+0000 7f0e5d800640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f0e581626f0 con 0x7f0e5806c960 2024-03-31T15:12:43.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.083+0000 7f0e5d800640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f0e581629d0 con 0x7f0e5806ecf0 2024-03-31T15:12:43.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.083+0000 7f0e57e00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f0e5806c960 0x7f0e580d00a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:43.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.083+0000 7f0e57e00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f0e5806c960 0x7f0e580d00a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43994/0 (socket says 172.21.15.16:43994) 2024-03-31T15:12:43.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.083+0000 7f0e57e00640 1 -- 172.21.15.16:0/2229963732 learned_addr learned my addr 172.21.15.16:0/2229963732 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:12:43.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.083+0000 7f0e5ce00640 1 --2- 172.21.15.16:0/2229963732 >> v2:172.21.15.89:3300/0 conn(0x7f0e5806ecf0 0x7f0e5806f100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:43.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.083+0000 7f0e57e00640 1 -- 172.21.15.16:0/2229963732 >> v2:172.21.15.89:3300/0 conn(0x7f0e5806ecf0 msgr2=0x7f0e5806f100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:12:43.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.083+0000 7f0e57e00640 1 --2- 172.21.15.16:0/2229963732 >> v2:172.21.15.89:3300/0 conn(0x7f0e5806ecf0 0x7f0e5806f100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:43.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.083+0000 7f0e57e00640 1 -- 172.21.15.16:0/2229963732 >> v1:172.21.15.89:6789/0 conn(0x7f0e58161e70 legacy=0x7f0e5806cd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:43.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.083+0000 7f0e57e00640 1 -- 172.21.15.16:0/2229963732 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e58162cb0 con 0x7f0e5806c960 2024-03-31T15:12:43.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.087+0000 7f0e57e00640 1 --2- 172.21.15.16:0/2229963732 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0e5806c960 0x7f0e580d00a0 secure :-1 s=READY pgs=1489 cs=0 l=1 rev1=1 crypto rx=0x7f0e580b0310 tx=0x7f0e4801c460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab32755acea5f874 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:12:43.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.087+0000 7f0e57400640 1 -- 172.21.15.16:0/2229963732 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e480ab5f0 con 0x7f0e5806c960 2024-03-31T15:12:43.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.087+0000 7f0e57400640 1 -- 172.21.15.16:0/2229963732 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0e480b0070 con 0x7f0e5806c960 2024-03-31T15:12:43.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.087+0000 7f0e57400640 1 -- 172.21.15.16:0/2229963732 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e480ab9b0 con 0x7f0e5806c960 2024-03-31T15:12:43.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.087+0000 7f0e5d800640 1 -- 172.21.15.16:0/2229963732 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0e5806c960 msgr2=0x7f0e580d00a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:43.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.087+0000 7f0e5d800640 1 --2- 172.21.15.16:0/2229963732 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0e5806c960 0x7f0e580d00a0 secure :-1 s=READY pgs=1489 cs=0 l=1 rev1=1 crypto rx=0x7f0e580b0310 tx=0x7f0e4801c460 comp rx=0 tx=0).stop 2024-03-31T15:12:43.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.087+0000 7f0e5d800640 1 -- 172.21.15.16:0/2229963732 shutdown_connections 2024-03-31T15:12:43.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.087+0000 7f0e5d800640 1 --2- 172.21.15.16:0/2229963732 >> v2:172.21.15.89:3300/0 conn(0x7f0e5806ecf0 0x7f0e5806f100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:43.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.087+0000 7f0e5d800640 1 --2- 172.21.15.16:0/2229963732 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0e5806c960 0x7f0e580d00a0 unknown :-1 s=CLOSED pgs=1489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:43.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.087+0000 7f0e5d800640 1 -- 172.21.15.16:0/2229963732 >> 172.21.15.16:0/2229963732 conn(0x7f0e580b2c90 msgr2=0x7f0e580b30a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:12:43.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.087+0000 7f0e5d800640 1 -- 172.21.15.16:0/2229963732 shutdown_connections 2024-03-31T15:12:43.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.087+0000 7f0e5d800640 1 -- 172.21.15.16:0/2229963732 wait complete. 2024-03-31T15:12:43.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.091+0000 7f0e5d800640 1 Processor -- start 2024-03-31T15:12:43.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.091+0000 7f0e5d800640 1 -- start start 2024-03-31T15:12:43.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.091+0000 7f0e5d800640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0e5806c960 0x7f0e5815f1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:43.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.091+0000 7f0e5d800640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f0e5806ecf0 0x7f0e580b3e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:43.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.091+0000 7f0e5d800640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0e58161e70 0x7f0e580b4340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:43.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.091+0000 7f0e5d800640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f0e580732a0 con 0x7f0e58161e70 2024-03-31T15:12:43.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.091+0000 7f0e5d800640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f0e58073120 con 0x7f0e5806c960 2024-03-31T15:12:43.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.091+0000 7f0e5d800640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f0e581617b0 con 0x7f0e5806ecf0 2024-03-31T15:12:43.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.091+0000 7f0e5ce00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0e58161e70 0x7f0e580b4340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:43.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.091+0000 7f0e57e00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0e5806c960 0x7f0e5815f1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:43.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.091+0000 7f0e5ce00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0e58161e70 0x7f0e580b4340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:44006/0 (socket says 172.21.15.16:44006) 2024-03-31T15:12:43.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.091+0000 7f0e5ce00640 1 -- 172.21.15.16:0/1861908899 learned_addr learned my addr 172.21.15.16:0/1861908899 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:12:43.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.091+0000 7f0e4f400640 1 --2- 172.21.15.16:0/1861908899 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f0e5806ecf0 0x7f0e580b3e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:43.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.091+0000 7f0e57e00640 1 -- 172.21.15.16:0/1861908899 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f0e5806ecf0 msgr2=0x7f0e580b3e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:43.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.091+0000 7f0e57e00640 1 --2- 172.21.15.16:0/1861908899 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f0e5806ecf0 0x7f0e580b3e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:43.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.091+0000 7f0e57e00640 1 -- 172.21.15.16:0/1861908899 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0e58161e70 msgr2=0x7f0e580b4340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:43.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.091+0000 7f0e57e00640 1 --2- 172.21.15.16:0/1861908899 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0e58161e70 0x7f0e580b4340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:43.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.091+0000 7f0e57e00640 1 -- 172.21.15.16:0/1861908899 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e58156bc0 con 0x7f0e5806c960 2024-03-31T15:12:43.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.091+0000 7f0e5ce00640 1 --2- 172.21.15.16:0/1861908899 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0e58161e70 0x7f0e580b4340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:12:43.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.091+0000 7f0e57e00640 1 --2- 172.21.15.16:0/1861908899 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0e5806c960 0x7f0e5815f1c0 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f0e480ac000 tx=0x7f0e4801aca0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:12:43.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.091+0000 7f0e54c00640 1 -- 172.21.15.16:0/1861908899 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e480be5f0 con 0x7f0e5806c960 2024-03-31T15:12:43.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.091+0000 7f0e54c00640 1 -- 172.21.15.16:0/1861908899 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0e480be7b0 con 0x7f0e5806c960 2024-03-31T15:12:43.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.091+0000 7f0e5d800640 1 -- 172.21.15.16:0/1861908899 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0e581611f0 con 0x7f0e5806c960 2024-03-31T15:12:43.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.091+0000 7f0e54c00640 1 -- 172.21.15.16:0/1861908899 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e480beba0 con 0x7f0e5806c960 2024-03-31T15:12:43.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.091+0000 7f0e5d800640 1 -- 172.21.15.16:0/1861908899 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0e580b4f40 con 0x7f0e5806c960 2024-03-31T15:12:43.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.095+0000 7f0e54c00640 1 -- 172.21.15.16:0/1861908899 <== mon.1 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f0e480b43f0 con 0x7f0e5806c960 2024-03-31T15:12:43.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.095+0000 7f0e62231000 1 Processor -- start 2024-03-31T15:12:43.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.095+0000 7f0e62231000 1 -- start start 2024-03-31T15:12:43.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.099+0000 7f0e54c00640 1 --2- 172.21.15.16:0/1861908899 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f0e340683b0 0x7f0e3406a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:43.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.099+0000 7f0e54c00640 1 -- 172.21.15.16:0/1861908899 <== mon.1 v2:172.21.15.89:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 5709+0+0 (secure 0 0 0) 0x7f0e48120730 con 0x7f0e5806c960 2024-03-31T15:12:43.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.099+0000 7f0e62231000 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c1d759b840 0x55c1d759bc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:43.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.099+0000 7f0e62231000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c1d759c160 0x55c1d754ec90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:43.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.099+0000 7f0e62231000 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55c1d754f1d0 0x55c1d754f5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:43.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.099+0000 7f0e62231000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55c1d7439180 con 0x55c1d759c160 2024-03-31T15:12:43.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.099+0000 7f0e62231000 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55c1d73a0c90 con 0x55c1d759b840 2024-03-31T15:12:43.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.099+0000 7f0e62231000 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x55c1d733da00 con 0x55c1d754f1d0 2024-03-31T15:12:43.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.099+0000 7f0e5ce00640 1 --2- 172.21.15.16:0/1861908899 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f0e340683b0 0x7f0e3406a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:43.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.099+0000 7f0e4f400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c1d759c160 0x55c1d754ec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:43.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.099+0000 7f0e57e00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c1d759b840 0x55c1d759bc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:43.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.099+0000 7f0e57e00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c1d759b840 0x55c1d759bc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.16:35846/0 (socket says 172.21.15.16:35846) 2024-03-31T15:12:43.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.099+0000 7f0e57e00640 1 -- 172.21.15.16:0/3739661725 learned_addr learned my addr 172.21.15.16:0/3739661725 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:12:43.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.099+0000 7f0e5ce00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55c1d754f1d0 0x55c1d754f5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:43.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.099+0000 7f0e57e00640 1 -- 172.21.15.16:0/3739661725 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55c1d754f1d0 msgr2=0x55c1d754f5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:12:43.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.099+0000 7f0e57e00640 1 --2- 172.21.15.16:0/3739661725 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55c1d754f1d0 0x55c1d754f5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:43.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.099+0000 7f0e57e00640 1 -- 172.21.15.16:0/3739661725 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c1d759c160 msgr2=0x55c1d754ec90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:12:43.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.099+0000 7f0e57e00640 1 --2- 172.21.15.16:0/3739661725 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c1d759c160 0x55c1d754ec90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:43.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.099+0000 7f0e57e00640 1 -- 172.21.15.16:0/3739661725 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c1d7326510 con 0x55c1d759b840 2024-03-31T15:12:43.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.099+0000 7f0e5ce00640 1 --2- 172.21.15.16:0/1861908899 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f0e340683b0 0x7f0e3406a870 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f0e50009fd0 tx=0x7f0e50007860 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:12:43.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.099+0000 7f0e57e00640 1 --2- 172.21.15.16:0/3739661725 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c1d759b840 0x55c1d759bc20 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f0e4811ec80 tx=0x7f0e480f0300 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4ffedbcd8a270a8e server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:12:43.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.099+0000 7f0e4cc00640 1 -- 172.21.15.16:0/3739661725 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e480f4ac0 con 0x55c1d759b840 2024-03-31T15:12:43.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.099+0000 7f0e4cc00640 1 -- 172.21.15.16:0/3739661725 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0e48107070 con 0x55c1d759b840 2024-03-31T15:12:43.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.099+0000 7f0e4cc00640 1 -- 172.21.15.16:0/3739661725 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e480f92f0 con 0x55c1d759b840 2024-03-31T15:12:43.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.099+0000 7f0e62231000 1 -- 172.21.15.16:0/3739661725 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c1d759b840 msgr2=0x55c1d759bc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:43.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.099+0000 7f0e62231000 1 --2- 172.21.15.16:0/3739661725 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c1d759b840 0x55c1d759bc20 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f0e4811ec80 tx=0x7f0e480f0300 comp rx=0 tx=0).stop 2024-03-31T15:12:43.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.099+0000 7f0e62231000 1 -- 172.21.15.16:0/3739661725 shutdown_connections 2024-03-31T15:12:43.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.099+0000 7f0e62231000 1 --2- 172.21.15.16:0/3739661725 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55c1d754f1d0 0x55c1d754f5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:43.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.099+0000 7f0e62231000 1 --2- 172.21.15.16:0/3739661725 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c1d759c160 0x55c1d754ec90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:43.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.099+0000 7f0e62231000 1 --2- 172.21.15.16:0/3739661725 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c1d759b840 0x55c1d759bc20 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:43.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.099+0000 7f0e62231000 1 -- 172.21.15.16:0/3739661725 >> 172.21.15.16:0/3739661725 conn(0x55c1d757c5e0 msgr2=0x55c1d757e9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:12:43.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.099+0000 7f0e62231000 1 -- 172.21.15.16:0/3739661725 shutdown_connections 2024-03-31T15:12:43.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.099+0000 7f0e62231000 1 -- 172.21.15.16:0/3739661725 wait complete. 2024-03-31T15:12:43.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.099+0000 7f0e62231000 1 -- start start 2024-03-31T15:12:43.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.099+0000 7f0e62231000 1 Processor -- start 2024-03-31T15:12:43.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.103+0000 7f0e62231000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c1d754f1d0 0x55c1d759afd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:43.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.103+0000 7f0e62231000 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55c1d759b840 0x55c1d7554aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:43.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.103+0000 7f0e62231000 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c1d759c160 0x55c1d7554fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:43.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.103+0000 7f0e62231000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55c1d735e200 con 0x55c1d754f1d0 2024-03-31T15:12:43.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.103+0000 7f0e62231000 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55c1d739cc40 con 0x55c1d759c160 2024-03-31T15:12:43.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.103+0000 7f0e62231000 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x55c1d736cb20 con 0x55c1d759b840 2024-03-31T15:12:43.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.103+0000 7f0e62231000 10 client.0 _ll_register_callbacks cb 0x55c1d753c930 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T15:12:43.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.103+0000 7f0e4f400640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55c1d759b840 0x55c1d7554aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:43.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.103+0000 7f0e57e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c1d754f1d0 0x55c1d759afd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:43.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.103+0000 7f0e57e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c1d754f1d0 0x55c1d759afd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:44020/0 (socket says 172.21.15.16:44020) 2024-03-31T15:12:43.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.103+0000 7f0e57e00640 1 -- 172.21.15.16:0/708286266 learned_addr learned my addr 172.21.15.16:0/708286266 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:12:43.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.103+0000 7f0e5ce00640 1 --2- 172.21.15.16:0/708286266 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c1d759c160 0x55c1d7554fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:43.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.103+0000 7f0e57e00640 1 -- 172.21.15.16:0/708286266 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55c1d759b840 msgr2=0x55c1d7554aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:12:43.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.103+0000 7f0e57e00640 1 --2- 172.21.15.16:0/708286266 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55c1d759b840 0x55c1d7554aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:43.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.103+0000 7f0e57e00640 1 -- 172.21.15.16:0/708286266 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c1d759c160 msgr2=0x55c1d7554fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:12:43.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.103+0000 7f0e57e00640 1 --2- 172.21.15.16:0/708286266 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c1d759c160 0x55c1d7554fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:43.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.103+0000 7f0e57e00640 1 -- 172.21.15.16:0/708286266 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c1d73abce0 con 0x55c1d754f1d0 2024-03-31T15:12:43.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.103+0000 7f0e57e00640 1 --2- 172.21.15.16:0/708286266 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c1d754f1d0 0x55c1d759afd0 secure :-1 s=READY pgs=1490 cs=0 l=1 rev1=1 crypto rx=0x7f0e48000c10 tx=0x7f0e4801c460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f0ab75d74ee11fcc server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:12:43.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.103+0000 7f0e45600640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-31T15:12:43.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.103+0000 7f0e45600640 1 -- 172.21.15.16:0/708286266 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e480f92f0 con 0x55c1d754f1d0 2024-03-31T15:12:43.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.103+0000 7f0e62231000 1 -- 172.21.15.16:0/708286266 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0e48107070 con 0x55c1d754f1d0 2024-03-31T15:12:43.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.103+0000 7f0e45600640 1 -- 172.21.15.16:0/708286266 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 5709+0+0 (secure 0 0 0) 0x7f0e480efc10 con 0x55c1d754f1d0 2024-03-31T15:12:43.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.103+0000 7f0e45600640 1 -- 172.21.15.16:0/708286266 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=83) ==== 24+0+0 (secure 0 0 0) 0x7f0e480f4ac0 con 0x55c1d754f1d0 2024-03-31T15:12:43.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.103+0000 7f0e62231000 10 client.8811 fetch_fsmap learned FSMap version 83 2024-03-31T15:12:43.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.103+0000 7f0e62231000 1 -- 172.21.15.16:0/708286266 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=83,monmap=2+}) -- 0x55c1d735c7c0 con 0x55c1d754f1d0 2024-03-31T15:12:43.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.103+0000 7f0e45600640 1 -- 172.21.15.16:0/708286266 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0e48107070 con 0x55c1d754f1d0 2024-03-31T15:12:43.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.103+0000 7f0e45600640 1 -- 172.21.15.16:0/708286266 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 83) ==== 3494+0+0 (secure 0 0 0) 0x7f0e480ec510 con 0x55c1d754f1d0 2024-03-31T15:12:43.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.103+0000 7f0e62231000 10 client.8811 fetch_fsmap finished waiting for FSMap version 83 2024-03-31T15:12:43.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.103+0000 7f0e62231000 10 client.8811 resolve_mds: resolved 7:0 to role '7:0' aka daemon mds.a 2024-03-31T15:12:43.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.103+0000 7f0e62231000 20 client.8811 populate_metadata read hostname 'smithi016' 2024-03-31T15:12:43.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.103+0000 7f0e62231000 1 --2- 172.21.15.16:0/708286266 >> [v2:172.21.15.16:6833/490717480,v1:172.21.15.16:6835/490717480] conn(0x55c1d75a2c80 0x55c1d75a3060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:43.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.103+0000 7f0e62231000 4 client.8811 mds_command: new command op to 8074 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-31T15:12:43.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.103+0000 7f0e62231000 1 -- 172.21.15.16:0/708286266 --> [v2:172.21.15.16:6833/490717480,v1:172.21.15.16:6835/490717480] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c1d733bc20 con 0x55c1d75a2c80 2024-03-31T15:12:43.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.103+0000 7f0e4f400640 1 --2- 172.21.15.16:0/708286266 >> [v2:172.21.15.16:6833/490717480,v1:172.21.15.16:6835/490717480] conn(0x55c1d75a2c80 0x55c1d75a3060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:43.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.103+0000 7f0e4f400640 1 --2- 172.21.15.16:0/708286266 >> [v2:172.21.15.16:6833/490717480,v1:172.21.15.16:6835/490717480] conn(0x55c1d75a2c80 0x55c1d75a3060 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3ba64badd6ccaac8 server_cookie=5229cd11a406966f in_seq=0 out_seq=0 2024-03-31T15:12:43.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.103+0000 7f0e45600640 10 client.8811 ms_handle_connect on v2:172.21.15.16:6833/490717480 2024-03-31T15:12:43.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.107+0000 7f0e45600640 1 -- 172.21.15.16:0/708286266 <== mds.0 v2:172.21.15.16:6833/490717480 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55c1d733bc20 con 0x55c1d75a2c80 2024-03-31T15:12:43.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.107+0000 7f0e45600640 10 client.8811 handle_command_reply: tid=0 2024-03-31T15:12:43.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.107+0000 7f0e62231000 1 client.8811 shutdown 2024-03-31T15:12:43.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.107+0000 7f0e62231000 1 -- 172.21.15.16:0/708286266 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c1d754f1d0 msgr2=0x55c1d759afd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:43.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.107+0000 7f0e62231000 1 --2- 172.21.15.16:0/708286266 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c1d754f1d0 0x55c1d759afd0 secure :-1 s=READY pgs=1490 cs=0 l=1 rev1=1 crypto rx=0x7f0e48000c10 tx=0x7f0e4801c460 comp rx=0 tx=0).stop 2024-03-31T15:12:43.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.107+0000 7f0e62231000 1 -- 172.21.15.16:0/708286266 shutdown_connections 2024-03-31T15:12:43.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.107+0000 7f0e62231000 1 --2- 172.21.15.16:0/708286266 >> [v2:172.21.15.16:6833/490717480,v1:172.21.15.16:6835/490717480] conn(0x55c1d75a2c80 0x55c1d75a3060 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:43.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.107+0000 7f0e62231000 1 --2- 172.21.15.16:0/708286266 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c1d759c160 0x55c1d7554fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:43.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.107+0000 7f0e62231000 1 --2- 172.21.15.16:0/708286266 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55c1d759b840 0x55c1d7554aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:43.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.107+0000 7f0e62231000 1 --2- 172.21.15.16:0/708286266 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c1d754f1d0 0x55c1d759afd0 unknown :-1 s=CLOSED pgs=1490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:43.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.107+0000 7f0e45600640 0 client.8811 ms_handle_reset on v2:172.21.15.16:6833/490717480 2024-03-31T15:12:43.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.107+0000 7f0e62231000 1 -- 172.21.15.16:0/708286266 >> 172.21.15.16:0/708286266 conn(0x55c1d757c5e0 msgr2=0x55c1d754e390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:12:43.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.107+0000 7f0e62231000 1 -- 172.21.15.16:0/708286266 shutdown_connections 2024-03-31T15:12:43.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.107+0000 7f0e62231000 1 -- 172.21.15.16:0/708286266 wait complete. 2024-03-31T15:12:43.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.107+0000 7f0e62231000 20 client.8811 trim_cache size 0 max 16384 2024-03-31T15:12:43.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.107+0000 7fb267000640 1 Processor -- start 2024-03-31T15:12:43.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.107+0000 7fb267000640 1 -- start start 2024-03-31T15:12:43.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.107+0000 7fb267000640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb2600d2290 0x7fb2601622e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:43.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.107+0000 7fb267000640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb260162820 0x7fb2601684b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:43.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.107+0000 7fb267000640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb260163030 con 0x7fb260155810 2024-03-31T15:12:43.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.107+0000 7fb267000640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7fb260161f10 con 0x7fb2600d2290 2024-03-31T15:12:43.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.107+0000 7fb267000640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fb26015ec40 con 0x7fb260162820 2024-03-31T15:12:43.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.111+0000 7fb25fe00640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fb260155810 0x7fb2600d1b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:36420/0 (socket says 172.21.15.16:36420) 2024-03-31T15:12:43.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.111+0000 7fb25fe00640 1 -- 172.21.15.16:0/3210573575 learned_addr learned my addr 172.21.15.16:0/3210573575 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:12:43.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.111+0000 7fb25f400640 1 --2- 172.21.15.16:0/3210573575 >> v2:172.21.15.16:3300/0 conn(0x7fb2600d2290 0x7fb2601622e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:43.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.111+0000 7fb265000640 1 --2- 172.21.15.16:0/3210573575 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb260162820 0x7fb2601684b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:43.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.111+0000 7fb25ea00640 1 -- 172.21.15.16:0/3210573575 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1708732119 0 0) 0x7fb260163030 con 0x7fb260155810 2024-03-31T15:12:43.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.111+0000 7fb25f400640 1 -- 172.21.15.16:0/3210573575 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb260162820 msgr2=0x7fb2601684b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:12:43.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.111+0000 7fb25f400640 1 --2- 172.21.15.16:0/3210573575 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb260162820 0x7fb2601684b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:43.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.111+0000 7fb25f400640 1 -- 172.21.15.16:0/3210573575 >> v1:172.21.15.16:6789/0 conn(0x7fb260155810 legacy=0x7fb2600d1b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:43.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.111+0000 7fb25f400640 1 -- 172.21.15.16:0/3210573575 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb26015ef20 con 0x7fb2600d2290 2024-03-31T15:12:43.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.111+0000 7fb25f400640 1 --2- 172.21.15.16:0/3210573575 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb2600d2290 0x7fb2601622e0 secure :-1 s=READY pgs=1492 cs=0 l=1 rev1=1 crypto rx=0x7fb24c0097c0 tx=0x7fb24c01c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40239a2f4231e6f9 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:12:43.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.111+0000 7fb25ea00640 1 -- 172.21.15.16:0/3210573575 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb24c0b1070 con 0x7fb2600d2290 2024-03-31T15:12:43.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.111+0000 7fb25ea00640 1 -- 172.21.15.16:0/3210573575 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb24c0a8050 con 0x7fb2600d2290 2024-03-31T15:12:43.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.111+0000 7fb25ea00640 1 -- 172.21.15.16:0/3210573575 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb24c0ac860 con 0x7fb2600d2290 2024-03-31T15:12:43.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.115+0000 7fb267000640 1 -- 172.21.15.16:0/3210573575 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb2600d2290 msgr2=0x7fb2601622e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:43.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.115+0000 7fb267000640 1 --2- 172.21.15.16:0/3210573575 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb2600d2290 0x7fb2601622e0 secure :-1 s=READY pgs=1492 cs=0 l=1 rev1=1 crypto rx=0x7fb24c0097c0 tx=0x7fb24c01c3e0 comp rx=0 tx=0).stop 2024-03-31T15:12:43.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.115+0000 7fb267000640 1 -- 172.21.15.16:0/3210573575 shutdown_connections 2024-03-31T15:12:43.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.115+0000 7fb267000640 1 --2- 172.21.15.16:0/3210573575 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb260162820 0x7fb2601684b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:43.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.115+0000 7fb267000640 1 --2- 172.21.15.16:0/3210573575 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb2600d2290 0x7fb2601622e0 unknown :-1 s=CLOSED pgs=1492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:43.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.115+0000 7fb267000640 1 -- 172.21.15.16:0/3210573575 >> 172.21.15.16:0/3210573575 conn(0x7fb260067330 msgr2=0x7fb260067740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:12:43.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.115+0000 7fb267000640 1 -- 172.21.15.16:0/3210573575 shutdown_connections 2024-03-31T15:12:43.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.115+0000 7fb267000640 1 -- 172.21.15.16:0/3210573575 wait complete. 2024-03-31T15:12:43.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.115+0000 7fb267000640 1 Processor -- start 2024-03-31T15:12:43.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.115+0000 7fb267000640 1 -- start start 2024-03-31T15:12:43.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.115+0000 7fb267000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb2600d2290 0x7fb26014da30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:43.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.115+0000 7fb267000640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb260155810 0x7fb260150d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:43.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.115+0000 7fb267000640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb260162820 0x7fb2600c92a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:43.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.115+0000 7fb267000640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fb260162c30 con 0x7fb2600d2290 2024-03-31T15:12:43.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.115+0000 7fb267000640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fb26015fa00 con 0x7fb260162820 2024-03-31T15:12:43.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.115+0000 7fb267000640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fb260162db0 con 0x7fb260155810 2024-03-31T15:12:43.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.115+0000 7fb25fe00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb2600d2290 0x7fb26014da30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:43.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.115+0000 7fb25fe00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb2600d2290 0x7fb26014da30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:44030/0 (socket says 172.21.15.16:44030) 2024-03-31T15:12:43.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.115+0000 7fb25fe00640 1 -- 172.21.15.16:0/3141683051 learned_addr learned my addr 172.21.15.16:0/3141683051 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:12:43.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.115+0000 7fb265000640 1 --2- 172.21.15.16:0/3141683051 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb260162820 0x7fb2600c92a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:43.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.115+0000 7fb25f400640 1 --2- 172.21.15.16:0/3141683051 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb260155810 0x7fb260150d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:43.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.115+0000 7fb25fe00640 1 -- 172.21.15.16:0/3141683051 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb260155810 msgr2=0x7fb260150d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:43.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.115+0000 7fb25fe00640 1 --2- 172.21.15.16:0/3141683051 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb260155810 0x7fb260150d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:43.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.115+0000 7fb25fe00640 1 -- 172.21.15.16:0/3141683051 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb260162820 msgr2=0x7fb2600c92a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:43.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.115+0000 7fb25fe00640 1 --2- 172.21.15.16:0/3141683051 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb260162820 0x7fb2600c92a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:43.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.115+0000 7fb25fe00640 1 -- 172.21.15.16:0/3141683051 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2600ca270 con 0x7fb2600d2290 2024-03-31T15:12:43.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.115+0000 7fb265000640 1 --2- 172.21.15.16:0/3141683051 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb260162820 0x7fb2600c92a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:12:43.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.119+0000 7fb25fe00640 1 --2- 172.21.15.16:0/3141683051 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb2600d2290 0x7fb26014da30 secure :-1 s=READY pgs=1493 cs=0 l=1 rev1=1 crypto rx=0x7fb25000c4b0 tx=0x7fb25000c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:12:43.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.119+0000 7fb257e00640 1 -- 172.21.15.16:0/3141683051 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb250018070 con 0x7fb2600d2290 2024-03-31T15:12:43.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.119+0000 7fb257e00640 1 -- 172.21.15.16:0/3141683051 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb250007500 con 0x7fb2600d2290 2024-03-31T15:12:43.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.119+0000 7fb267000640 1 -- 172.21.15.16:0/3141683051 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb2600ca550 con 0x7fb2600d2290 2024-03-31T15:12:43.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.119+0000 7fb257e00640 1 -- 172.21.15.16:0/3141683051 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb250004a30 con 0x7fb2600d2290 2024-03-31T15:12:43.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.119+0000 7fb267000640 1 -- 172.21.15.16:0/3141683051 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb2600cab10 con 0x7fb2600d2290 2024-03-31T15:12:43.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.119+0000 7fb267000640 1 -- 172.21.15.16:0/3141683051 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "status"} v 0) -- 0x7fb260069950 con 0x7fb2600d2290 2024-03-31T15:12:43.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.119+0000 7fb257e00640 1 -- 172.21.15.16:0/3141683051 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fb25001b070 con 0x7fb2600d2290 2024-03-31T15:12:43.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.119+0000 7fb257e00640 1 --2- 172.21.15.16:0/3141683051 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fb2400683d0 0x7fb24006a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:43.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.119+0000 7fb257e00640 1 -- 172.21.15.16:0/3141683051 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 5709+0+0 (secure 0 0 0) 0x7fb250085cf0 con 0x7fb2600d2290 2024-03-31T15:12:43.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.123+0000 7fb257e00640 1 -- 172.21.15.16:0/3141683051 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "status"}]=0 v0) ==== 54+0+511 (secure 0 0 0) 0x7fb2500883c0 con 0x7fb2600d2290 2024-03-31T15:12:43.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.123+0000 7fb25f400640 1 --2- 172.21.15.16:0/3141683051 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fb2400683d0 0x7fb24006a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:43.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.123+0000 7fb267000640 1 -- 172.21.15.16:0/3141683051 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({log-info=0+}) -- 0x7fb2600703e0 con 0x7fb2600d2290 2024-03-31T15:12:43.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.123+0000 7fb25f400640 1 --2- 172.21.15.16:0/3141683051 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fb2400683d0 0x7fb24006a890 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7fb24c0ae000 tx=0x7fb24c008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:12:43.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.127+0000 7f0e62231000 1 Processor -- start 2024-03-31T15:12:43.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.127+0000 7f0e62231000 1 -- start start 2024-03-31T15:12:43.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.127+0000 7f0e62231000 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55c1d754f1d0 0x55c1d7606950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:43.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.127+0000 7f0e62231000 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c1d759b840 0x55c1d7606e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:43.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.127+0000 7f0e62231000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c1d759c160 0x55c1d76073d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:43.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.127+0000 7f0e62231000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f0e582033a0 con 0x55c1d759c160 2024-03-31T15:12:43.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.127+0000 7f0e62231000 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f0e582023c0 con 0x55c1d759b840 2024-03-31T15:12:43.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.127+0000 7f0e62231000 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f0e58201720 con 0x55c1d754f1d0 2024-03-31T15:12:43.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.127+0000 7f0e5ce00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c1d759c160 0x55c1d76073d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:43.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.127+0000 7f0e5ce00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c1d759c160 0x55c1d76073d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:44040/0 (socket says 172.21.15.16:44040) 2024-03-31T15:12:43.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.127+0000 7f0e5ce00640 1 -- 172.21.15.16:0/1966211265 learned_addr learned my addr 172.21.15.16:0/1966211265 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:12:43.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.127+0000 7f0e4f400640 1 --2- 172.21.15.16:0/1966211265 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c1d759b840 0x55c1d7606e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:43.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.127+0000 7f0e5ce00640 1 -- 172.21.15.16:0/1966211265 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55c1d754f1d0 msgr2=0x55c1d7606950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:12:43.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.127+0000 7f0e5ce00640 1 --2- 172.21.15.16:0/1966211265 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55c1d754f1d0 0x55c1d7606950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:43.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.127+0000 7f0e5ce00640 1 -- 172.21.15.16:0/1966211265 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c1d759b840 msgr2=0x55c1d7606e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:12:43.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.127+0000 7f0e5ce00640 1 --2- 172.21.15.16:0/1966211265 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c1d759b840 0x55c1d7606e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:43.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.127+0000 7f0e5ce00640 1 -- 172.21.15.16:0/1966211265 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c1d735cc70 con 0x55c1d759c160 2024-03-31T15:12:43.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.127+0000 7f0e4f400640 1 --2- 172.21.15.16:0/1966211265 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c1d759b840 0x55c1d7606e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:12:43.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.127+0000 7f0e5ce00640 1 --2- 172.21.15.16:0/1966211265 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c1d759c160 0x55c1d76073d0 secure :-1 s=READY pgs=1494 cs=0 l=1 rev1=1 crypto rx=0x7f0e50007c00 tx=0x7f0e5001f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fae2c2a60f2b0977 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:12:43.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.127+0000 7f0e47e00640 1 -- 172.21.15.16:0/1966211265 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e50007c30 con 0x55c1d759c160 2024-03-31T15:12:43.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.127+0000 7f0e47e00640 1 -- 172.21.15.16:0/1966211265 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0e5001b450 con 0x55c1d759c160 2024-03-31T15:12:43.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.127+0000 7f0e47e00640 1 -- 172.21.15.16:0/1966211265 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e5001b800 con 0x55c1d759c160 2024-03-31T15:12:43.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.131+0000 7f0e62231000 1 -- 172.21.15.16:0/1966211265 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c1d759c160 msgr2=0x55c1d76073d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:43.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.131+0000 7f0e62231000 1 --2- 172.21.15.16:0/1966211265 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c1d759c160 0x55c1d76073d0 secure :-1 s=READY pgs=1494 cs=0 l=1 rev1=1 crypto rx=0x7f0e50007c00 tx=0x7f0e5001f000 comp rx=0 tx=0).stop 2024-03-31T15:12:43.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.131+0000 7f0e62231000 1 -- 172.21.15.16:0/1966211265 shutdown_connections 2024-03-31T15:12:43.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.131+0000 7f0e62231000 1 --2- 172.21.15.16:0/1966211265 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c1d759c160 0x55c1d76073d0 unknown :-1 s=CLOSED pgs=1494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:43.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.131+0000 7f0e62231000 1 --2- 172.21.15.16:0/1966211265 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c1d759b840 0x55c1d7606e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:43.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.131+0000 7f0e62231000 1 --2- 172.21.15.16:0/1966211265 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55c1d754f1d0 0x55c1d7606950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:43.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.131+0000 7f0e62231000 1 -- 172.21.15.16:0/1966211265 >> 172.21.15.16:0/1966211265 conn(0x55c1d757c5e0 msgr2=0x55c1d75b1d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:12:43.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.131+0000 7f0e62231000 1 -- 172.21.15.16:0/1966211265 shutdown_connections 2024-03-31T15:12:43.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.131+0000 7f0e62231000 1 -- 172.21.15.16:0/1966211265 wait complete. 2024-03-31T15:12:43.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.131+0000 7f0e62231000 1 -- start start 2024-03-31T15:12:43.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.131+0000 7f0e62231000 1 Processor -- start 2024-03-31T15:12:43.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.131+0000 7f0e62231000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c1d754f1d0 0x55c1d760b970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:43.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.131+0000 7f0e62231000 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55c1d759b840 0x55c1d760beb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:43.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.131+0000 7f0e62231000 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c1d759c160 0x55c1d75ef9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:43.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.131+0000 7f0e62231000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55c1d745c080 con 0x55c1d754f1d0 2024-03-31T15:12:43.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.131+0000 7f0e62231000 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f0e58201d70 con 0x55c1d759c160 2024-03-31T15:12:43.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.131+0000 7f0e62231000 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x55c1d73a0c90 con 0x55c1d759b840 2024-03-31T15:12:43.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.131+0000 7f0e62231000 10 client.0 _ll_register_callbacks cb 0x7f0e400011b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T15:12:43.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.131+0000 7f0e57e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c1d754f1d0 0x55c1d760b970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:43.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.131+0000 7f0e57e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c1d754f1d0 0x55c1d760b970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:44042/0 (socket says 172.21.15.16:44042) 2024-03-31T15:12:43.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.131+0000 7f0e57e00640 1 -- 172.21.15.16:0/2323943059 learned_addr learned my addr 172.21.15.16:0/2323943059 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:12:43.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.131+0000 7f0e5ce00640 1 --2- 172.21.15.16:0/2323943059 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c1d759c160 0x55c1d75ef9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:43.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.131+0000 7f0e4f400640 1 --2- 172.21.15.16:0/2323943059 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55c1d759b840 0x55c1d760beb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:43.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.131+0000 7f0e57e00640 1 -- 172.21.15.16:0/2323943059 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55c1d759b840 msgr2=0x55c1d760beb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:12:43.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.131+0000 7f0e57e00640 1 --2- 172.21.15.16:0/2323943059 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55c1d759b840 0x55c1d760beb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:43.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.131+0000 7f0e57e00640 1 -- 172.21.15.16:0/2323943059 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c1d759c160 msgr2=0x55c1d75ef9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:12:43.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.131+0000 7f0e57e00640 1 --2- 172.21.15.16:0/2323943059 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c1d759c160 0x55c1d75ef9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:43.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.131+0000 7f0e57e00640 1 -- 172.21.15.16:0/2323943059 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c1d733dc50 con 0x55c1d754f1d0 2024-03-31T15:12:43.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.131+0000 7f0e4f400640 1 --2- 172.21.15.16:0/2323943059 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55c1d759b840 0x55c1d760beb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:12:43.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.131+0000 7f0e57e00640 1 --2- 172.21.15.16:0/2323943059 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c1d754f1d0 0x55c1d760b970 secure :-1 s=READY pgs=1495 cs=0 l=1 rev1=1 crypto rx=0x7f0e48097ec0 tx=0x7f0e4811eb30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c32abd534945b34 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:12:43.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.131+0000 7f0e45600640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-31T15:12:43.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.131+0000 7f0e45600640 1 -- 172.21.15.16:0/2323943059 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e480f0120 con 0x55c1d754f1d0 2024-03-31T15:12:43.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.131+0000 7f0e45600640 1 -- 172.21.15.16:0/2323943059 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 5709+0+0 (secure 0 0 0) 0x7f0e480ef890 con 0x55c1d754f1d0 2024-03-31T15:12:43.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.131+0000 7f0e62231000 1 -- 172.21.15.16:0/2323943059 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0e5001b450 con 0x55c1d754f1d0 2024-03-31T15:12:43.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.131+0000 7f0e45600640 1 -- 172.21.15.16:0/2323943059 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=83) ==== 24+0+0 (secure 0 0 0) 0x7f0e480f92f0 con 0x55c1d754f1d0 2024-03-31T15:12:43.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.131+0000 7f0e62231000 10 client.8826 fetch_fsmap learned FSMap version 83 2024-03-31T15:12:43.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.131+0000 7f0e62231000 1 -- 172.21.15.16:0/2323943059 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=83,monmap=2+}) -- 0x55c1d7329500 con 0x55c1d754f1d0 2024-03-31T15:12:43.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.135+0000 7f0e45600640 1 -- 172.21.15.16:0/2323943059 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0e5001b450 con 0x55c1d754f1d0 2024-03-31T15:12:43.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.135+0000 7f0e45600640 1 -- 172.21.15.16:0/2323943059 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 83) ==== 3494+0+0 (secure 0 0 0) 0x7f0e480f1a40 con 0x55c1d754f1d0 2024-03-31T15:12:43.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.135+0000 7f0e62231000 10 client.8826 fetch_fsmap finished waiting for FSMap version 83 2024-03-31T15:12:43.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.135+0000 7f0e62231000 10 client.8826 resolve_mds: resolved 7:0 to role '7:0' aka daemon mds.a 2024-03-31T15:12:43.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.135+0000 7f0e62231000 20 client.8826 populate_metadata read hostname 'smithi016' 2024-03-31T15:12:43.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.135+0000 7f0e62231000 1 --2- 172.21.15.16:0/2323943059 >> [v2:172.21.15.16:6833/490717480,v1:172.21.15.16:6835/490717480] conn(0x55c1d75eff30 0x55c1d75f0340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:43.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.135+0000 7f0e62231000 4 client.8826 mds_command: new command op to 8074 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["repair", "recursive"]}] 2024-03-31T15:12:43.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.135+0000 7f0e62231000 1 -- 172.21.15.16:0/2323943059 --> [v2:172.21.15.16:6833/490717480,v1:172.21.15.16:6835/490717480] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["repair", "recursive"]}) -- 0x55c1d73ae1d0 con 0x55c1d75eff30 2024-03-31T15:12:43.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.135+0000 7f0e4f400640 1 --2- 172.21.15.16:0/2323943059 >> [v2:172.21.15.16:6833/490717480,v1:172.21.15.16:6835/490717480] conn(0x55c1d75eff30 0x55c1d75f0340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:43.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.135+0000 7f0e45600640 10 client.8826 ms_handle_connect on v2:172.21.15.16:6833/490717480 2024-03-31T15:12:43.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.135+0000 7f0e4f400640 1 --2- 172.21.15.16:0/2323943059 >> [v2:172.21.15.16:6833/490717480,v1:172.21.15.16:6835/490717480] conn(0x55c1d75eff30 0x55c1d75f0340 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=71fb0c63e9ea7e11 server_cookie=223db00c76ebade3 in_seq=0 out_seq=0 2024-03-31T15:12:43.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.135+0000 7f0e45600640 1 -- 172.21.15.16:0/2323943059 <== mds.0 v2:172.21.15.16:6833/490717480 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55c1d73ae1d0 con 0x55c1d75eff30 2024-03-31T15:12:43.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.135+0000 7f0e45600640 10 client.8826 handle_command_reply: tid=0 2024-03-31T15:12:43.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.135+0000 7f0e62231000 1 client.8826 shutdown 2024-03-31T15:12:43.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.135+0000 7f0e62231000 1 -- 172.21.15.16:0/2323943059 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c1d754f1d0 msgr2=0x55c1d760b970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:43.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.135+0000 7f0e62231000 1 --2- 172.21.15.16:0/2323943059 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c1d754f1d0 0x55c1d760b970 secure :-1 s=READY pgs=1495 cs=0 l=1 rev1=1 crypto rx=0x7f0e48097ec0 tx=0x7f0e4811eb30 comp rx=0 tx=0).stop 2024-03-31T15:12:43.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.135+0000 7f0e62231000 1 -- 172.21.15.16:0/2323943059 shutdown_connections 2024-03-31T15:12:43.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.135+0000 7f0e62231000 1 --2- 172.21.15.16:0/2323943059 >> [v2:172.21.15.16:6833/490717480,v1:172.21.15.16:6835/490717480] conn(0x55c1d75eff30 0x55c1d75f0340 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:43.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.135+0000 7f0e62231000 1 --2- 172.21.15.16:0/2323943059 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c1d759c160 0x55c1d75ef9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:43.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.135+0000 7f0e62231000 1 --2- 172.21.15.16:0/2323943059 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55c1d759b840 0x55c1d760beb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:43.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.135+0000 7f0e62231000 1 --2- 172.21.15.16:0/2323943059 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c1d754f1d0 0x55c1d760b970 unknown :-1 s=CLOSED pgs=1495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:43.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.135+0000 7f0e62231000 1 -- 172.21.15.16:0/2323943059 >> 172.21.15.16:0/2323943059 conn(0x55c1d757c5e0 msgr2=0x55c1d75b25c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:12:43.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.135+0000 7f0e45600640 0 client.8826 ms_handle_reset on v2:172.21.15.16:6833/490717480 2024-03-31T15:12:43.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.135+0000 7f0e62231000 1 -- 172.21.15.16:0/2323943059 shutdown_connections 2024-03-31T15:12:43.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.135+0000 7f0e62231000 1 -- 172.21.15.16:0/2323943059 wait complete. 2024-03-31T15:12:43.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.135+0000 7f0e62231000 20 client.8826 trim_cache size 0 max 16384 2024-03-31T15:12:43.132 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-03-31T15:12:43.132 INFO:teuthology.orchestra.run.smithi016.stdout: "return_code": 0, 2024-03-31T15:12:43.132 INFO:teuthology.orchestra.run.smithi016.stdout: "scrub_tag": "12e621e9-1357-4985-8a91-c47687ca7ce4", 2024-03-31T15:12:43.132 INFO:teuthology.orchestra.run.smithi016.stdout: "mode": "asynchronous" 2024-03-31T15:12:43.132 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-03-31T15:12:43.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.135+0000 7f0e5d800640 1 -- 172.21.15.16:0/1861908899 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f0e340683b0 msgr2=0x7f0e3406a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:43.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.135+0000 7f0e5d800640 1 --2- 172.21.15.16:0/1861908899 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f0e340683b0 0x7f0e3406a870 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f0e50009fd0 tx=0x7f0e50007860 comp rx=0 tx=0).stop 2024-03-31T15:12:43.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.135+0000 7f0e5d800640 1 -- 172.21.15.16:0/1861908899 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0e5806c960 msgr2=0x7f0e5815f1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:43.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.135+0000 7f0e5d800640 1 --2- 172.21.15.16:0/1861908899 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0e5806c960 0x7f0e5815f1c0 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f0e480ac000 tx=0x7f0e4801aca0 comp rx=0 tx=0).stop 2024-03-31T15:12:43.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.135+0000 7f0e5d800640 1 -- 172.21.15.16:0/1861908899 shutdown_connections 2024-03-31T15:12:43.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.135+0000 7f0e5d800640 1 --2- 172.21.15.16:0/1861908899 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f0e340683b0 0x7f0e3406a870 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:43.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.135+0000 7f0e5d800640 1 --2- 172.21.15.16:0/1861908899 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0e58161e70 0x7f0e580b4340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:43.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.135+0000 7f0e5d800640 1 --2- 172.21.15.16:0/1861908899 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f0e5806ecf0 0x7f0e580b3e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:43.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.135+0000 7f0e5d800640 1 --2- 172.21.15.16:0/1861908899 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0e5806c960 0x7f0e5815f1c0 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:43.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.135+0000 7f0e5d800640 1 -- 172.21.15.16:0/1861908899 >> 172.21.15.16:0/1861908899 conn(0x7f0e580b2c90 msgr2=0x7f0e5806f890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:12:43.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.135+0000 7f0e5d800640 1 -- 172.21.15.16:0/1861908899 shutdown_connections 2024-03-31T15:12:43.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.135+0000 7f0e5d800640 1 -- 172.21.15.16:0/1861908899 wait complete. 2024-03-31T15:12:43.147 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:12:43.147 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.7:0 scrub status 2024-03-31T15:12:43.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.299+0000 7fae2c400640 1 Processor -- start 2024-03-31T15:12:43.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.299+0000 7fae2c400640 1 -- start start 2024-03-31T15:12:43.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.299+0000 7fae2c400640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fae240d0a60 0x7fae240cea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:43.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.299+0000 7fae2c400640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fae240cef80 0x7fae241419c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:43.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.299+0000 7fae2c400640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fae240cf430 con 0x7fae2406f920 2024-03-31T15:12:43.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.299+0000 7fae2c400640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7fae2413f670 con 0x7fae240cef80 2024-03-31T15:12:43.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.299+0000 7fae2c400640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fae2415ca70 con 0x7fae240d0a60 2024-03-31T15:12:43.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.299+0000 7fae29a00640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fae2406f920 0x7fae2406fd00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:36426/0 (socket says 172.21.15.16:36426) 2024-03-31T15:12:43.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.299+0000 7fae29a00640 1 -- 172.21.15.16:0/3225319278 learned_addr learned my addr 172.21.15.16:0/3225319278 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:12:43.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.299+0000 7fae29000640 1 --2- 172.21.15.16:0/3225319278 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fae240d0a60 0x7fae240cea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:43.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.299+0000 7fae2a400640 1 --2- 172.21.15.16:0/3225319278 >> v2:172.21.15.89:3300/0 conn(0x7fae240cef80 0x7fae241419c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:43.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.299+0000 7fae23e00640 1 -- 172.21.15.16:0/3225319278 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1786948728 0 0) 0x7fae240cf430 con 0x7fae2406f920 2024-03-31T15:12:43.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.299+0000 7fae23e00640 1 -- 172.21.15.16:0/3225319278 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fae0c003530 con 0x7fae2406f920 2024-03-31T15:12:43.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.299+0000 7fae2a400640 1 -- 172.21.15.16:0/3225319278 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fae240d0a60 msgr2=0x7fae240cea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:12:43.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.299+0000 7fae2a400640 1 --2- 172.21.15.16:0/3225319278 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fae240d0a60 0x7fae240cea40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:43.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.299+0000 7fae2a400640 1 -- 172.21.15.16:0/3225319278 >> v1:172.21.15.16:6789/0 conn(0x7fae2406f920 legacy=0x7fae2406fd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:43.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.299+0000 7fae2a400640 1 -- 172.21.15.16:0/3225319278 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae2415cd50 con 0x7fae240cef80 2024-03-31T15:12:43.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.303+0000 7fae2a400640 1 --2- 172.21.15.16:0/3225319278 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fae240cef80 0x7fae241419c0 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7fae1c009d90 tx=0x7fae1c01c970 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c90351a7cdc1af98 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:12:43.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.303+0000 7fae23e00640 1 -- 172.21.15.16:0/3225319278 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae1c0b2070 con 0x7fae240cef80 2024-03-31T15:12:43.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.303+0000 7fae23e00640 1 -- 172.21.15.16:0/3225319278 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fae1c0a5830 con 0x7fae240cef80 2024-03-31T15:12:43.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.303+0000 7fae23e00640 1 -- 172.21.15.16:0/3225319278 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae1c0a5c00 con 0x7fae240cef80 2024-03-31T15:12:43.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.303+0000 7fae2c400640 1 -- 172.21.15.16:0/3225319278 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fae240cef80 msgr2=0x7fae241419c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:43.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.303+0000 7fae2c400640 1 --2- 172.21.15.16:0/3225319278 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fae240cef80 0x7fae241419c0 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7fae1c009d90 tx=0x7fae1c01c970 comp rx=0 tx=0).stop 2024-03-31T15:12:43.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.303+0000 7fae2c400640 1 -- 172.21.15.16:0/3225319278 shutdown_connections 2024-03-31T15:12:43.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.303+0000 7fae2c400640 1 --2- 172.21.15.16:0/3225319278 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fae240cef80 0x7fae241419c0 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:43.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.303+0000 7fae2c400640 1 --2- 172.21.15.16:0/3225319278 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fae240d0a60 0x7fae240cea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:43.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.303+0000 7fae2c400640 1 -- 172.21.15.16:0/3225319278 >> 172.21.15.16:0/3225319278 conn(0x7fae24067930 msgr2=0x7fae240b3ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:12:43.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.303+0000 7fae2c400640 1 -- 172.21.15.16:0/3225319278 shutdown_connections 2024-03-31T15:12:43.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.303+0000 7fae2c400640 1 -- 172.21.15.16:0/3225319278 wait complete. 2024-03-31T15:12:43.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.307+0000 7fae2c400640 1 Processor -- start 2024-03-31T15:12:43.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.307+0000 7fae2c400640 1 -- start start 2024-03-31T15:12:43.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.307+0000 7fae2c400640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fae2406f920 0x7fae240b2f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:43.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.307+0000 7fae2c400640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fae240cef80 0x7fae241652e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:43.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.307+0000 7fae2c400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fae240d0a60 0x7fae24156290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:43.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.307+0000 7fae2c400640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fae24169230 con 0x7fae240d0a60 2024-03-31T15:12:43.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.307+0000 7fae2c400640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fae241690b0 con 0x7fae240cef80 2024-03-31T15:12:43.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.307+0000 7fae2c400640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fae241693b0 con 0x7fae2406f920 2024-03-31T15:12:43.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.307+0000 7fae29a00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fae2406f920 0x7fae240b2f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:43.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.307+0000 7fae2a400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fae240d0a60 0x7fae24156290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:43.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.307+0000 7fae29000640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fae240cef80 0x7fae241652e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:43.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.307+0000 7fae29a00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fae2406f920 0x7fae240b2f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3301/0 says I am v2:172.21.15.16:44168/0 (socket says 172.21.15.16:44168) 2024-03-31T15:12:43.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.307+0000 7fae29a00640 1 -- 172.21.15.16:0/1458117191 learned_addr learned my addr 172.21.15.16:0/1458117191 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:12:43.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.311+0000 7fae29a00640 1 -- 172.21.15.16:0/1458117191 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fae240cef80 msgr2=0x7fae241652e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:43.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.311+0000 7fae29a00640 1 --2- 172.21.15.16:0/1458117191 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fae240cef80 0x7fae241652e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:43.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.311+0000 7fae29a00640 1 -- 172.21.15.16:0/1458117191 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fae240d0a60 msgr2=0x7fae24156290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:43.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.311+0000 7fae29a00640 1 --2- 172.21.15.16:0/1458117191 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fae240d0a60 0x7fae24156290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:43.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.311+0000 7fae29a00640 1 -- 172.21.15.16:0/1458117191 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae241567d0 con 0x7fae2406f920 2024-03-31T15:12:43.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.311+0000 7fae2a400640 1 --2- 172.21.15.16:0/1458117191 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fae240d0a60 0x7fae24156290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:12:43.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.311+0000 7fae29000640 1 --2- 172.21.15.16:0/1458117191 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fae240cef80 0x7fae241652e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:12:43.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.311+0000 7fae29a00640 1 --2- 172.21.15.16:0/1458117191 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fae2406f920 0x7fae240b2f50 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fae1800c450 tx=0x7fae1800c910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:12:43.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.311+0000 7fae21600640 1 -- 172.21.15.16:0/1458117191 <== mon.2 v2:172.21.15.89:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae18017070 con 0x7fae2406f920 2024-03-31T15:12:43.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.311+0000 7fae21600640 1 -- 172.21.15.16:0/1458117191 <== mon.2 v2:172.21.15.89:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fae18005cf0 con 0x7fae2406f920 2024-03-31T15:12:43.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.311+0000 7fae2c400640 1 -- 172.21.15.16:0/1458117191 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fae240b35e0 con 0x7fae2406f920 2024-03-31T15:12:43.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.311+0000 7fae21600640 1 -- 172.21.15.16:0/1458117191 <== mon.2 v2:172.21.15.89:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae18007290 con 0x7fae2406f920 2024-03-31T15:12:43.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.311+0000 7fae2c400640 1 -- 172.21.15.16:0/1458117191 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fae24165e50 con 0x7fae2406f920 2024-03-31T15:12:43.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.311+0000 7fae21600640 1 -- 172.21.15.16:0/1458117191 <== mon.2 v2:172.21.15.89:3301/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fae1801a070 con 0x7fae2406f920 2024-03-31T15:12:43.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.315+0000 7fae21600640 1 --2- 172.21.15.16:0/1458117191 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fae04068380 0x7fae0406a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:43.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.315+0000 7fae21600640 1 -- 172.21.15.16:0/1458117191 <== mon.2 v2:172.21.15.89:3301/0 5 ==== osd_map(96..96 src has 1..96) ==== 5709+0+0 (secure 0 0 0) 0x7fae18085d40 con 0x7fae2406f920 2024-03-31T15:12:43.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.315+0000 7fae2f9b2000 1 Processor -- start 2024-03-31T15:12:43.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.315+0000 7fae29000640 1 --2- 172.21.15.16:0/1458117191 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fae04068380 0x7fae0406a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:43.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.315+0000 7fae2f9b2000 1 -- start start 2024-03-31T15:12:43.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.315+0000 7fae29000640 1 --2- 172.21.15.16:0/1458117191 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fae04068380 0x7fae0406a840 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7fae240d0f10 tx=0x7fae10009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:12:43.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.315+0000 7fae2f9b2000 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561f3883f2d0 0x561f3883f6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:43.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.315+0000 7fae2f9b2000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561f3883fbf0 0x561f387f2470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:43.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.315+0000 7fae2f9b2000 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x561f387f29b0 0x561f387f2d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:43.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.315+0000 7fae2f9b2000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x561f386dbaf0 con 0x561f3883fbf0 2024-03-31T15:12:43.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.315+0000 7fae2f9b2000 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x561f38644b00 con 0x561f3883f2d0 2024-03-31T15:12:43.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.315+0000 7fae2f9b2000 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x561f385e1940 con 0x561f387f29b0 2024-03-31T15:12:43.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.315+0000 7fae29000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561f3883fbf0 0x561f387f2470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:43.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.315+0000 7fae29000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561f3883fbf0 0x561f387f2470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:44056/0 (socket says 172.21.15.16:44056) 2024-03-31T15:12:43.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.315+0000 7fae29000640 1 -- 172.21.15.16:0/3004009889 learned_addr learned my addr 172.21.15.16:0/3004009889 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:12:43.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.315+0000 7fae2a400640 1 --2- 172.21.15.16:0/3004009889 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x561f387f29b0 0x561f387f2d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:43.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.315+0000 7fae29a00640 1 --2- 172.21.15.16:0/3004009889 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561f3883f2d0 0x561f3883f6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:43.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.315+0000 7fae29000640 1 -- 172.21.15.16:0/3004009889 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x561f387f29b0 msgr2=0x561f387f2d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:12:43.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.315+0000 7fae29000640 1 --2- 172.21.15.16:0/3004009889 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x561f387f29b0 0x561f387f2d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:43.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.315+0000 7fae29000640 1 -- 172.21.15.16:0/3004009889 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561f3883f2d0 msgr2=0x561f3883f6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:12:43.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.315+0000 7fae29000640 1 --2- 172.21.15.16:0/3004009889 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561f3883f2d0 0x561f3883f6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:43.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.315+0000 7fae29000640 1 -- 172.21.15.16:0/3004009889 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561f385ca380 con 0x561f3883fbf0 2024-03-31T15:12:43.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.315+0000 7fae29000640 1 --2- 172.21.15.16:0/3004009889 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561f3883fbf0 0x561f387f2470 secure :-1 s=READY pgs=1497 cs=0 l=1 rev1=1 crypto rx=0x7fae100041a0 tx=0x7fae10014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7957914b62ab6709 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:12:43.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.315+0000 7fae16000640 1 -- 172.21.15.16:0/3004009889 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae1001d050 con 0x561f3883fbf0 2024-03-31T15:12:43.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.315+0000 7fae16000640 1 -- 172.21.15.16:0/3004009889 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fae10027070 con 0x561f3883fbf0 2024-03-31T15:12:43.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.315+0000 7fae16000640 1 -- 172.21.15.16:0/3004009889 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae10003c70 con 0x561f3883fbf0 2024-03-31T15:12:43.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.315+0000 7fae2f9b2000 1 -- 172.21.15.16:0/3004009889 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561f3883fbf0 msgr2=0x561f387f2470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:43.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.315+0000 7fae2f9b2000 1 --2- 172.21.15.16:0/3004009889 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561f3883fbf0 0x561f387f2470 secure :-1 s=READY pgs=1497 cs=0 l=1 rev1=1 crypto rx=0x7fae100041a0 tx=0x7fae10014000 comp rx=0 tx=0).stop 2024-03-31T15:12:43.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.319+0000 7fae2f9b2000 1 -- 172.21.15.16:0/3004009889 shutdown_connections 2024-03-31T15:12:43.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.319+0000 7fae2f9b2000 1 --2- 172.21.15.16:0/3004009889 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x561f387f29b0 0x561f387f2d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:43.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.319+0000 7fae2f9b2000 1 --2- 172.21.15.16:0/3004009889 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561f3883fbf0 0x561f387f2470 unknown :-1 s=CLOSED pgs=1497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:43.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.319+0000 7fae2f9b2000 1 --2- 172.21.15.16:0/3004009889 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561f3883f2d0 0x561f3883f6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:43.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.319+0000 7fae2f9b2000 1 -- 172.21.15.16:0/3004009889 >> 172.21.15.16:0/3004009889 conn(0x561f38820070 msgr2=0x561f38822460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:12:43.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.319+0000 7fae2f9b2000 1 -- 172.21.15.16:0/3004009889 shutdown_connections 2024-03-31T15:12:43.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.319+0000 7fae2f9b2000 1 -- 172.21.15.16:0/3004009889 wait complete. 2024-03-31T15:12:43.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.319+0000 7fae2f9b2000 1 -- start start 2024-03-31T15:12:43.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.319+0000 7fae2f9b2000 1 Processor -- start 2024-03-31T15:12:43.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.319+0000 7fae2f9b2000 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561f387f29b0 0x561f387f7640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:43.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.319+0000 7fae2f9b2000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561f3883f2d0 0x561f387f7b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:43.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.319+0000 7fae2f9b2000 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x561f3883fbf0 0x561f387f80c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:43.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.319+0000 7fae2f9b2000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x561f386021e0 con 0x561f3883f2d0 2024-03-31T15:12:43.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.319+0000 7fae2f9b2000 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x561f38640ab0 con 0x561f387f29b0 2024-03-31T15:12:43.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.319+0000 7fae2f9b2000 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x561f387fe000 con 0x561f3883fbf0 2024-03-31T15:12:43.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.319+0000 7fae2f9b2000 10 client.0 _ll_register_callbacks cb 0x561f387e0370 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T15:12:43.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.319+0000 7fae29000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561f3883f2d0 0x561f387f7b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:43.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.319+0000 7fae29a00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561f387f29b0 0x561f387f7640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:43.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.319+0000 7fae29000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561f3883f2d0 0x561f387f7b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:44062/0 (socket says 172.21.15.16:44062) 2024-03-31T15:12:43.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.319+0000 7fae2a400640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x561f3883fbf0 0x561f387f80c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:43.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.319+0000 7fae29000640 1 -- 172.21.15.16:0/3218772572 learned_addr learned my addr 172.21.15.16:0/3218772572 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:12:43.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.319+0000 7fae29000640 1 -- 172.21.15.16:0/3218772572 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x561f3883fbf0 msgr2=0x561f387f80c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:12:43.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.319+0000 7fae29000640 1 --2- 172.21.15.16:0/3218772572 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x561f3883fbf0 0x561f387f80c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:43.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.319+0000 7fae29000640 1 -- 172.21.15.16:0/3218772572 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561f387f29b0 msgr2=0x561f387f7640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:12:43.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.319+0000 7fae29000640 1 --2- 172.21.15.16:0/3218772572 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561f387f29b0 0x561f387f7640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:43.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.319+0000 7fae29000640 1 -- 172.21.15.16:0/3218772572 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561f38600b70 con 0x561f3883f2d0 2024-03-31T15:12:43.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.319+0000 7fae29a00640 1 --2- 172.21.15.16:0/3218772572 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561f387f29b0 0x561f387f7640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:12:43.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.319+0000 7fae29000640 1 --2- 172.21.15.16:0/3218772572 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561f3883f2d0 0x561f387f7b80 secure :-1 s=READY pgs=1498 cs=0 l=1 rev1=1 crypto rx=0x561f3864aae0 tx=0x7fae10004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba23df22298ff399 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:12:43.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.319+0000 7fae0aa00640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-31T15:12:43.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.319+0000 7fae0aa00640 1 -- 172.21.15.16:0/3218772572 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae10022800 con 0x561f3883f2d0 2024-03-31T15:12:43.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.319+0000 7fae0aa00640 1 -- 172.21.15.16:0/3218772572 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 5709+0+0 (secure 0 0 0) 0x7fae10037ae0 con 0x561f3883f2d0 2024-03-31T15:12:43.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.319+0000 7fae2f9b2000 1 -- 172.21.15.16:0/3218772572 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fae10027070 con 0x561f3883f2d0 2024-03-31T15:12:43.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.319+0000 7fae0aa00640 1 -- 172.21.15.16:0/3218772572 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=83) ==== 24+0+0 (secure 0 0 0) 0x7fae1001d020 con 0x561f3883f2d0 2024-03-31T15:12:43.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.319+0000 7fae2f9b2000 10 client.8838 fetch_fsmap learned FSMap version 83 2024-03-31T15:12:43.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.319+0000 7fae2f9b2000 1 -- 172.21.15.16:0/3218772572 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=83,monmap=2+}) -- 0x561f385e1b90 con 0x561f3883f2d0 2024-03-31T15:12:43.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.319+0000 7fae0aa00640 1 -- 172.21.15.16:0/3218772572 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fae10027070 con 0x561f3883f2d0 2024-03-31T15:12:43.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.323+0000 7fae0aa00640 1 -- 172.21.15.16:0/3218772572 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 83) ==== 3494+0+0 (secure 0 0 0) 0x7fae10037ce0 con 0x561f3883f2d0 2024-03-31T15:12:43.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.323+0000 7fae2f9b2000 10 client.8838 fetch_fsmap finished waiting for FSMap version 83 2024-03-31T15:12:43.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.323+0000 7fae2f9b2000 10 client.8838 resolve_mds: resolved 7:0 to role '7:0' aka daemon mds.a 2024-03-31T15:12:43.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.323+0000 7fae2f9b2000 20 client.8838 populate_metadata read hostname 'smithi016' 2024-03-31T15:12:43.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.323+0000 7fae2f9b2000 1 --2- 172.21.15.16:0/3218772572 >> [v2:172.21.15.16:6833/490717480,v1:172.21.15.16:6835/490717480] conn(0x561f387f8960 0x561f38850a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:43.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.323+0000 7fae2f9b2000 4 client.8838 mds_command: new command op to 8074 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-31T15:12:43.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.323+0000 7fae2f9b2000 1 -- 172.21.15.16:0/3218772572 --> [v2:172.21.15.16:6833/490717480,v1:172.21.15.16:6835/490717480] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561f385dfdd0 con 0x561f387f8960 2024-03-31T15:12:43.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.323+0000 7fae29a00640 1 --2- 172.21.15.16:0/3218772572 >> [v2:172.21.15.16:6833/490717480,v1:172.21.15.16:6835/490717480] conn(0x561f387f8960 0x561f38850a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:43.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.323+0000 7fae0aa00640 10 client.8838 ms_handle_connect on v2:172.21.15.16:6833/490717480 2024-03-31T15:12:43.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.323+0000 7fae29a00640 1 --2- 172.21.15.16:0/3218772572 >> [v2:172.21.15.16:6833/490717480,v1:172.21.15.16:6835/490717480] conn(0x561f387f8960 0x561f38850a50 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=965aadeedd39f9db server_cookie=9b9121d2bc452b1c in_seq=0 out_seq=0 2024-03-31T15:12:43.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.323+0000 7fae0aa00640 1 -- 172.21.15.16:0/3218772572 <== mds.0 v2:172.21.15.16:6833/490717480 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x561f385dfdd0 con 0x561f387f8960 2024-03-31T15:12:43.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.323+0000 7fae0aa00640 10 client.8838 handle_command_reply: tid=0 2024-03-31T15:12:43.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.323+0000 7fae2f9b2000 1 client.8838 shutdown 2024-03-31T15:12:43.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.323+0000 7fae2f9b2000 1 -- 172.21.15.16:0/3218772572 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561f3883f2d0 msgr2=0x561f387f7b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:43.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.323+0000 7fae2f9b2000 1 --2- 172.21.15.16:0/3218772572 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561f3883f2d0 0x561f387f7b80 secure :-1 s=READY pgs=1498 cs=0 l=1 rev1=1 crypto rx=0x561f3864aae0 tx=0x7fae10004740 comp rx=0 tx=0).stop 2024-03-31T15:12:43.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.323+0000 7fae2f9b2000 1 -- 172.21.15.16:0/3218772572 shutdown_connections 2024-03-31T15:12:43.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.323+0000 7fae2f9b2000 1 --2- 172.21.15.16:0/3218772572 >> [v2:172.21.15.16:6833/490717480,v1:172.21.15.16:6835/490717480] conn(0x561f387f8960 0x561f38850a50 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:43.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.323+0000 7fae2f9b2000 1 --2- 172.21.15.16:0/3218772572 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x561f3883fbf0 0x561f387f80c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:43.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.323+0000 7fae2f9b2000 1 --2- 172.21.15.16:0/3218772572 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561f3883f2d0 0x561f387f7b80 unknown :-1 s=CLOSED pgs=1498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:43.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.323+0000 7fae2f9b2000 1 --2- 172.21.15.16:0/3218772572 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561f387f29b0 0x561f387f7640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:43.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.323+0000 7fae2f9b2000 1 -- 172.21.15.16:0/3218772572 >> 172.21.15.16:0/3218772572 conn(0x561f38820070 msgr2=0x561f38822460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:12:43.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.323+0000 7fae0aa00640 0 client.8838 ms_handle_reset on v2:172.21.15.16:6833/490717480 2024-03-31T15:12:43.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.323+0000 7fae2f9b2000 1 -- 172.21.15.16:0/3218772572 shutdown_connections 2024-03-31T15:12:43.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.323+0000 7fae2f9b2000 1 -- 172.21.15.16:0/3218772572 wait complete. 2024-03-31T15:12:43.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.323+0000 7fae2f9b2000 20 client.8838 trim_cache size 0 max 16384 2024-03-31T15:12:43.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.359+0000 7fae2f9b2000 1 Processor -- start 2024-03-31T15:12:43.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.359+0000 7fae2f9b2000 1 -- start start 2024-03-31T15:12:43.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.359+0000 7fae2f9b2000 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x561f387f29b0 0x561f3883da90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:43.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.359+0000 7fae2f9b2000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561f3883f2d0 0x561f3883dfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:43.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.359+0000 7fae2f9b2000 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561f3883fbf0 0x561f38850a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:43.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.359+0000 7fae2f9b2000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fae2415f610 con 0x561f3883f2d0 2024-03-31T15:12:43.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.359+0000 7fae2f9b2000 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fae2415e970 con 0x561f3883fbf0 2024-03-31T15:12:43.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.359+0000 7fae2f9b2000 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fae2415efc0 con 0x561f387f29b0 2024-03-31T15:12:43.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.359+0000 7fae29000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561f3883f2d0 0x561f3883dfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:43.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.359+0000 7fae29000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561f3883f2d0 0x561f3883dfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:44064/0 (socket says 172.21.15.16:44064) 2024-03-31T15:12:43.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.359+0000 7fae29000640 1 -- 172.21.15.16:0/509737841 learned_addr learned my addr 172.21.15.16:0/509737841 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:12:43.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.359+0000 7fae29a00640 1 --2- 172.21.15.16:0/509737841 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x561f387f29b0 0x561f3883da90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:43.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.359+0000 7fae2a400640 1 --2- 172.21.15.16:0/509737841 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561f3883fbf0 0x561f38850a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:43.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.363+0000 7fae29000640 1 -- 172.21.15.16:0/509737841 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x561f387f29b0 msgr2=0x561f3883da90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:12:43.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.363+0000 7fae29000640 1 --2- 172.21.15.16:0/509737841 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x561f387f29b0 0x561f3883da90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:43.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.363+0000 7fae29000640 1 -- 172.21.15.16:0/509737841 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561f3883fbf0 msgr2=0x561f38850a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:12:43.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.363+0000 7fae29000640 1 --2- 172.21.15.16:0/509737841 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561f3883fbf0 0x561f38850a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:43.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.363+0000 7fae29000640 1 -- 172.21.15.16:0/509737841 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561f3883fa20 con 0x561f3883f2d0 2024-03-31T15:12:43.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.363+0000 7fae29000640 1 --2- 172.21.15.16:0/509737841 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561f3883f2d0 0x561f3883dfd0 secure :-1 s=READY pgs=1499 cs=0 l=1 rev1=1 crypto rx=0x7fae10012010 tx=0x7fae1002f070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=146c9641f8478bae server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:12:43.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.363+0000 7fae15600640 1 -- 172.21.15.16:0/509737841 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae10047070 con 0x561f3883f2d0 2024-03-31T15:12:43.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.363+0000 7fae15600640 1 -- 172.21.15.16:0/509737841 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fae10027070 con 0x561f3883f2d0 2024-03-31T15:12:43.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.363+0000 7fae15600640 1 -- 172.21.15.16:0/509737841 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae10042690 con 0x561f3883f2d0 2024-03-31T15:12:43.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.363+0000 7fae2f9b2000 1 -- 172.21.15.16:0/509737841 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561f3883f2d0 msgr2=0x561f3883dfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:43.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.363+0000 7fae2f9b2000 1 --2- 172.21.15.16:0/509737841 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561f3883f2d0 0x561f3883dfd0 secure :-1 s=READY pgs=1499 cs=0 l=1 rev1=1 crypto rx=0x7fae10012010 tx=0x7fae1002f070 comp rx=0 tx=0).stop 2024-03-31T15:12:43.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.363+0000 7fae2f9b2000 1 -- 172.21.15.16:0/509737841 shutdown_connections 2024-03-31T15:12:43.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.363+0000 7fae2f9b2000 1 --2- 172.21.15.16:0/509737841 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561f3883fbf0 0x561f38850a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:43.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.363+0000 7fae2f9b2000 1 --2- 172.21.15.16:0/509737841 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561f3883f2d0 0x561f3883dfd0 unknown :-1 s=CLOSED pgs=1499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:43.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.363+0000 7fae2f9b2000 1 --2- 172.21.15.16:0/509737841 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x561f387f29b0 0x561f3883da90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:43.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.363+0000 7fae2f9b2000 1 -- 172.21.15.16:0/509737841 >> 172.21.15.16:0/509737841 conn(0x561f38820070 msgr2=0x561f3883d380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:12:43.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.363+0000 7fae2f9b2000 1 -- 172.21.15.16:0/509737841 shutdown_connections 2024-03-31T15:12:43.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.363+0000 7fae2f9b2000 1 -- 172.21.15.16:0/509737841 wait complete. 2024-03-31T15:12:43.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.363+0000 7fae2f9b2000 1 -- start start 2024-03-31T15:12:43.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.363+0000 7fae2f9b2000 1 Processor -- start 2024-03-31T15:12:43.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.363+0000 7fae2f9b2000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561f387f29b0 0x561f38853700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:43.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.363+0000 7fae2f9b2000 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561f3883f2d0 0x561f388ae480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:43.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.363+0000 7fae2f9b2000 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x561f3883fbf0 0x561f388ae9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:43.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.363+0000 7fae2f9b2000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x561f385df940 con 0x561f387f29b0 2024-03-31T15:12:43.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.363+0000 7fae2f9b2000 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x561f38610150 con 0x561f3883f2d0 2024-03-31T15:12:43.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.363+0000 7fae2f9b2000 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x561f385fb750 con 0x561f3883fbf0 2024-03-31T15:12:43.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.363+0000 7fae2f9b2000 10 client.0 _ll_register_callbacks cb 0x7fae100093a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T15:12:43.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.363+0000 7fae29a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561f387f29b0 0x561f38853700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:43.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.363+0000 7fae29a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561f387f29b0 0x561f38853700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:44072/0 (socket says 172.21.15.16:44072) 2024-03-31T15:12:43.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.363+0000 7fae29a00640 1 -- 172.21.15.16:0/1778546594 learned_addr learned my addr 172.21.15.16:0/1778546594 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:12:43.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.363+0000 7fae29000640 1 --2- 172.21.15.16:0/1778546594 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561f3883f2d0 0x561f388ae480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:43.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.363+0000 7fae2a400640 1 --2- 172.21.15.16:0/1778546594 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x561f3883fbf0 0x561f388ae9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:43.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.363+0000 7fae29a00640 1 -- 172.21.15.16:0/1778546594 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x561f3883fbf0 msgr2=0x561f388ae9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:12:43.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.363+0000 7fae29a00640 1 --2- 172.21.15.16:0/1778546594 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x561f3883fbf0 0x561f388ae9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:43.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.363+0000 7fae29a00640 1 -- 172.21.15.16:0/1778546594 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561f3883f2d0 msgr2=0x561f388ae480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:12:43.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.363+0000 7fae29a00640 1 --2- 172.21.15.16:0/1778546594 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561f3883f2d0 0x561f388ae480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:43.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.363+0000 7fae29a00640 1 -- 172.21.15.16:0/1778546594 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561f386c5140 con 0x561f387f29b0 2024-03-31T15:12:43.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.363+0000 7fae2a400640 1 --2- 172.21.15.16:0/1778546594 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x561f3883fbf0 0x561f388ae9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:12:43.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.367+0000 7fae29a00640 1 --2- 172.21.15.16:0/1778546594 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561f387f29b0 0x561f38853700 secure :-1 s=READY pgs=1500 cs=0 l=1 rev1=1 crypto rx=0x7fae180525a0 tx=0x7fae1801e040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=151493f21fa7f13a server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:12:43.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.367+0000 7fae0aa00640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-31T15:12:43.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.367+0000 7fae0aa00640 1 -- 172.21.15.16:0/1778546594 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae1805e4c0 con 0x561f387f29b0 2024-03-31T15:12:43.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.367+0000 7fae0aa00640 1 -- 172.21.15.16:0/1778546594 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 5709+0+0 (secure 0 0 0) 0x7fae18055070 con 0x561f387f29b0 2024-03-31T15:12:43.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.367+0000 7fae2f9b2000 1 -- 172.21.15.16:0/1778546594 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fae10027070 con 0x561f387f29b0 2024-03-31T15:12:43.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.367+0000 7fae0aa00640 1 -- 172.21.15.16:0/1778546594 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=83) ==== 24+0+0 (secure 0 0 0) 0x7fae18073020 con 0x561f387f29b0 2024-03-31T15:12:43.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.367+0000 7fae2f9b2000 10 client.8844 fetch_fsmap learned FSMap version 83 2024-03-31T15:12:43.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.367+0000 7fae2f9b2000 1 -- 172.21.15.16:0/1778546594 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=83,monmap=2+}) -- 0x561f385d31e0 con 0x561f387f29b0 2024-03-31T15:12:43.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.367+0000 7fae0aa00640 1 -- 172.21.15.16:0/1778546594 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fae10027070 con 0x561f387f29b0 2024-03-31T15:12:43.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.367+0000 7fae0aa00640 1 -- 172.21.15.16:0/1778546594 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 83) ==== 3494+0+0 (secure 0 0 0) 0x7fae1807dbc0 con 0x561f387f29b0 2024-03-31T15:12:43.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.367+0000 7fae2f9b2000 10 client.8844 fetch_fsmap finished waiting for FSMap version 83 2024-03-31T15:12:43.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.367+0000 7fae2f9b2000 10 client.8844 resolve_mds: resolved 7:0 to role '7:0' aka daemon mds.a 2024-03-31T15:12:43.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.367+0000 7fae2f9b2000 20 client.8844 populate_metadata read hostname 'smithi016' 2024-03-31T15:12:43.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.367+0000 7fae2f9b2000 1 --2- 172.21.15.16:0/1778546594 >> [v2:172.21.15.16:6833/490717480,v1:172.21.15.16:6835/490717480] conn(0x561f3886a5e0 0x561f3886a9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:12:43.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.367+0000 7fae2f9b2000 4 client.8844 mds_command: new command op to 8074 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-31T15:12:43.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.367+0000 7fae2f9b2000 1 -- 172.21.15.16:0/1778546594 --> [v2:172.21.15.16:6833/490717480,v1:172.21.15.16:6835/490717480] -- command(tid 0: {"prefix": "scrub status"}) -- 0x561f38652040 con 0x561f3886a5e0 2024-03-31T15:12:43.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.367+0000 7fae29000640 1 --2- 172.21.15.16:0/1778546594 >> [v2:172.21.15.16:6833/490717480,v1:172.21.15.16:6835/490717480] conn(0x561f3886a5e0 0x561f3886a9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:12:43.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.367+0000 7fae29000640 1 --2- 172.21.15.16:0/1778546594 >> [v2:172.21.15.16:6833/490717480,v1:172.21.15.16:6835/490717480] conn(0x561f3886a5e0 0x561f3886a9f0 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=56335b7fb2ac5205 server_cookie=b6a361bfe6424a08 in_seq=0 out_seq=0 2024-03-31T15:12:43.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.367+0000 7fae0aa00640 10 client.8844 ms_handle_connect on v2:172.21.15.16:6833/490717480 2024-03-31T15:12:43.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.367+0000 7fae0aa00640 1 -- 172.21.15.16:0/1778546594 <== mds.0 v2:172.21.15.16:6833/490717480 1 ==== command_reply(tid 0: 0 ) ==== 8+0+260 (crc 0 0 0) 0x561f38652040 con 0x561f3886a5e0 2024-03-31T15:12:43.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.367+0000 7fae0aa00640 10 client.8844 handle_command_reply: tid=0 2024-03-31T15:12:43.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.367+0000 7fae2f9b2000 1 client.8844 shutdown 2024-03-31T15:12:43.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.367+0000 7fae2f9b2000 1 -- 172.21.15.16:0/1778546594 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561f387f29b0 msgr2=0x561f38853700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:43.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.367+0000 7fae2f9b2000 1 --2- 172.21.15.16:0/1778546594 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561f387f29b0 0x561f38853700 secure :-1 s=READY pgs=1500 cs=0 l=1 rev1=1 crypto rx=0x7fae180525a0 tx=0x7fae1801e040 comp rx=0 tx=0).stop 2024-03-31T15:12:43.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.367+0000 7fae2f9b2000 1 -- 172.21.15.16:0/1778546594 shutdown_connections 2024-03-31T15:12:43.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.367+0000 7fae2f9b2000 1 --2- 172.21.15.16:0/1778546594 >> [v2:172.21.15.16:6833/490717480,v1:172.21.15.16:6835/490717480] conn(0x561f3886a5e0 0x561f3886a9f0 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:43.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.367+0000 7fae2f9b2000 1 --2- 172.21.15.16:0/1778546594 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x561f3883fbf0 0x561f388ae9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:43.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.367+0000 7fae2f9b2000 1 --2- 172.21.15.16:0/1778546594 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561f3883f2d0 0x561f388ae480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:43.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.367+0000 7fae2f9b2000 1 --2- 172.21.15.16:0/1778546594 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561f387f29b0 0x561f38853700 unknown :-1 s=CLOSED pgs=1500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:43.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.367+0000 7fae0aa00640 0 client.8844 ms_handle_reset on v2:172.21.15.16:6833/490717480 2024-03-31T15:12:43.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.367+0000 7fae2f9b2000 1 -- 172.21.15.16:0/1778546594 >> 172.21.15.16:0/1778546594 conn(0x561f38820070 msgr2=0x561f3883dc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:12:43.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.367+0000 7fae2f9b2000 1 -- 172.21.15.16:0/1778546594 shutdown_connections 2024-03-31T15:12:43.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.367+0000 7fae2f9b2000 1 -- 172.21.15.16:0/1778546594 wait complete. 2024-03-31T15:12:43.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.367+0000 7fae2f9b2000 20 client.8844 trim_cache size 0 max 16384 2024-03-31T15:12:43.364 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-03-31T15:12:43.365 INFO:teuthology.orchestra.run.smithi016.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-31T15:12:43.365 INFO:teuthology.orchestra.run.smithi016.stdout: "scrubs": { 2024-03-31T15:12:43.365 INFO:teuthology.orchestra.run.smithi016.stdout: "12e621e9-1357-4985-8a91-c47687ca7ce4": { 2024-03-31T15:12:43.365 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-03-31T15:12:43.365 INFO:teuthology.orchestra.run.smithi016.stdout: "tag": "12e621e9-1357-4985-8a91-c47687ca7ce4", 2024-03-31T15:12:43.365 INFO:teuthology.orchestra.run.smithi016.stdout: "options": "recursive,repair" 2024-03-31T15:12:43.365 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-31T15:12:43.365 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-31T15:12:43.365 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-03-31T15:12:43.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.367+0000 7fae2c400640 1 -- 172.21.15.16:0/1458117191 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fae04068380 msgr2=0x7fae0406a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:43.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.367+0000 7fae2c400640 1 --2- 172.21.15.16:0/1458117191 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fae04068380 0x7fae0406a840 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7fae240d0f10 tx=0x7fae10009190 comp rx=0 tx=0).stop 2024-03-31T15:12:43.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.367+0000 7fae2c400640 1 -- 172.21.15.16:0/1458117191 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fae2406f920 msgr2=0x7fae240b2f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:12:43.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.367+0000 7fae2c400640 1 --2- 172.21.15.16:0/1458117191 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fae2406f920 0x7fae240b2f50 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fae1800c450 tx=0x7fae1800c910 comp rx=0 tx=0).stop 2024-03-31T15:12:43.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.367+0000 7fae2c400640 1 -- 172.21.15.16:0/1458117191 shutdown_connections 2024-03-31T15:12:43.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.367+0000 7fae2c400640 1 --2- 172.21.15.16:0/1458117191 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fae04068380 0x7fae0406a840 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:43.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.367+0000 7fae2c400640 1 --2- 172.21.15.16:0/1458117191 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fae240d0a60 0x7fae24156290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:43.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.367+0000 7fae2c400640 1 --2- 172.21.15.16:0/1458117191 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fae240cef80 0x7fae241652e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:43.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.367+0000 7fae2c400640 1 --2- 172.21.15.16:0/1458117191 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fae2406f920 0x7fae240b2f50 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:12:43.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.367+0000 7fae2c400640 1 -- 172.21.15.16:0/1458117191 >> 172.21.15.16:0/1458117191 conn(0x7fae24067930 msgr2=0x7fae240b67b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:12:43.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.367+0000 7fae2c400640 1 -- 172.21.15.16:0/1458117191 shutdown_connections 2024-03-31T15:12:43.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:43.367+0000 7fae2c400640 1 -- 172.21.15.16:0/1458117191 wait complete. 2024-03-31T15:12:43.387 INFO:tasks.cephfs.filesystem:scrub status for tag:12e621e9-1357-4985-8a91-c47687ca7ce4 - {'path': '/', 'tag': '12e621e9-1357-4985-8a91-c47687ca7ce4', 'options': 'recursive,repair'} 2024-03-31T15:12:44.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:44.995+0000 7fb257e00640 1 -- 172.21.15.16:0/3141683051 <== mon.0 v2:172.21.15.16:3300/0 7 ==== log(4 entries from seq 1 at 2024-03-31T15:12:43.138952+0000) ==== 695+0+0 (secure 0 0 0) 0x7fb25005c030 con 0x7fb2600d2290 2024-03-31T15:12:44.986 INFO:teuthology.orchestra.run.smithi016.stdout: cluster: 2024-03-31T15:12:44.987 INFO:teuthology.orchestra.run.smithi016.stdout: id: 286eb5da-fb47-4550-b290-d9519c13b9a5 2024-03-31T15:12:44.987 INFO:teuthology.orchestra.run.smithi016.stdout: health: HEALTH_OK 2024-03-31T15:12:44.987 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:12:44.987 INFO:teuthology.orchestra.run.smithi016.stdout: services: 2024-03-31T15:12:44.987 INFO:teuthology.orchestra.run.smithi016.stdout: mon: 3 daemons, quorum a,b,c (age 9m) 2024-03-31T15:12:44.987 INFO:teuthology.orchestra.run.smithi016.stdout: mgr: x(active, since 9m), standbys: y 2024-03-31T15:12:44.987 INFO:teuthology.orchestra.run.smithi016.stdout: mds: 1/1 daemons up, 3 standby 2024-03-31T15:12:44.987 INFO:teuthology.orchestra.run.smithi016.stdout: osd: 8 osds: 8 up (since 9m), 8 in (since 9m) 2024-03-31T15:12:44.987 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:12:44.987 INFO:teuthology.orchestra.run.smithi016.stdout: data: 2024-03-31T15:12:44.987 INFO:teuthology.orchestra.run.smithi016.stdout: volumes: 1/1 healthy 2024-03-31T15:12:44.987 INFO:teuthology.orchestra.run.smithi016.stdout: pools: 3 pools, 97 pgs 2024-03-31T15:12:44.987 INFO:teuthology.orchestra.run.smithi016.stdout: objects: 30 objects, 19 MiB 2024-03-31T15:12:44.988 INFO:teuthology.orchestra.run.smithi016.stdout: usage: 294 MiB used, 720 GiB / 720 GiB avail 2024-03-31T15:12:44.988 INFO:teuthology.orchestra.run.smithi016.stdout: pgs: 97 active+clean 2024-03-31T15:12:44.988 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:12:44.988 INFO:teuthology.orchestra.run.smithi016.stdout: io: 2024-03-31T15:12:44.988 INFO:teuthology.orchestra.run.smithi016.stdout: client: 85 B/s rd, 1.0 MiB/s wr, 0 op/s rd, 2 op/s wr 2024-03-31T15:12:44.988 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:12:44.988 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:12:44.988 INFO:teuthology.orchestra.run.smithi016.stdout:2024-03-31T15:12:43.138952+0000 mds.a [INF] scrub queued for path: / 2024-03-31T15:12:44.988 INFO:teuthology.orchestra.run.smithi016.stdout:2024-03-31T15:12:43.138954+0000 mds.a [INF] scrub summary: idle+waiting paths [/] 2024-03-31T15:12:44.988 INFO:teuthology.orchestra.run.smithi016.stdout:2024-03-31T15:12:43.138958+0000 mds.a [INF] scrub summary: active paths [/] 2024-03-31T15:12:44.988 INFO:teuthology.orchestra.run.smithi016.stdout:2024-03-31T15:12:43.140426+0000 mds.a [INF] scrub summary: idle+waiting paths [/] 2024-03-31T15:12:47.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:12:47.007+0000 7fb257e00640 1 -- 172.21.15.16:0/3141683051 <== mon.0 v2:172.21.15.16:3300/0 8 ==== log(1 entries from seq 5 at 2024-03-31T15:12:46.995599+0000) ==== 189+0+0 (secure 0 0 0) 0x7fb250053d30 con 0x7fb2600d2290 2024-03-31T15:12:47.001 INFO:teuthology.orchestra.run.smithi016.stdout:2024-03-31T15:12:46.995599+0000 mds.a [INF] scrub summary: idle 2024-03-31T15:13:07.723 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-31T15:13:07.727 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-31T15:13:13.388 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:13:13.388 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.7:0 scrub status 2024-03-31T15:13:13.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.538+0000 7f85c5a00640 1 Processor -- start 2024-03-31T15:13:13.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.538+0000 7f85c5a00640 1 -- start start 2024-03-31T15:13:13.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.538+0000 7f85c5a00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f85c015c3d0 0x7f85c006fc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:13.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.538+0000 7f85c5a00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f85c00d0880 0x7f85c00b4250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:13.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.542+0000 7f85c5a00640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f85c0070520 con 0x7f85c00cdc30 2024-03-31T15:13:13.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.542+0000 7f85c5a00640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f85c00cfda0 con 0x7f85c00d0880 2024-03-31T15:13:13.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.542+0000 7f85c5a00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f85c00d0080 con 0x7f85c015c3d0 2024-03-31T15:13:13.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.542+0000 7f85bf400640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f85c00d0880 0x7f85c00b4250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:13.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.542+0000 7f85bea00640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f85c00cdc30 0x7f85c00ce010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:34348/0 (socket says 172.21.15.16:34348) 2024-03-31T15:13:13.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.542+0000 7f85bea00640 1 -- 172.21.15.16:0/4202451402 learned_addr learned my addr 172.21.15.16:0/4202451402 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:13:13.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.542+0000 7f85bf400640 1 -- 172.21.15.16:0/4202451402 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f85c015c3d0 msgr2=0x7f85c006fc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:13:13.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.542+0000 7f85b7e00640 1 --2- 172.21.15.16:0/4202451402 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f85c015c3d0 0x7f85c006fc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:13.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.542+0000 7f85bf400640 1 --2- 172.21.15.16:0/4202451402 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f85c015c3d0 0x7f85c006fc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:13.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.542+0000 7f85be000640 1 -- 172.21.15.16:0/4202451402 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1536318854 0 0) 0x7f85c0070520 con 0x7f85c00cdc30 2024-03-31T15:13:13.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.542+0000 7f85bf400640 1 -- 172.21.15.16:0/4202451402 >> v1:172.21.15.16:6789/0 conn(0x7f85c00cdc30 legacy=0x7f85c00ce010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:13:13.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.542+0000 7f85bf400640 1 -- 172.21.15.16:0/4202451402 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85c00d0360 con 0x7f85c00d0880 2024-03-31T15:13:13.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.546+0000 7f85bf400640 1 --2- 172.21.15.16:0/4202451402 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f85c00d0880 0x7f85c00b4250 secure :-1 s=READY pgs=1502 cs=0 l=1 rev1=1 crypto rx=0x7f85b8009d90 tx=0x7f85b801c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf1f502bc51681f4 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:13:13.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.546+0000 7f85be000640 1 -- 172.21.15.16:0/4202451402 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85b80b2070 con 0x7f85c00d0880 2024-03-31T15:13:13.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.546+0000 7f85be000640 1 -- 172.21.15.16:0/4202451402 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f85b80a5960 con 0x7f85c00d0880 2024-03-31T15:13:13.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.546+0000 7f85be000640 1 -- 172.21.15.16:0/4202451402 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85b80a5d30 con 0x7f85c00d0880 2024-03-31T15:13:13.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.546+0000 7f85c5a00640 1 -- 172.21.15.16:0/4202451402 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f85c00d0880 msgr2=0x7f85c00b4250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:13:13.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.546+0000 7f85c5a00640 1 --2- 172.21.15.16:0/4202451402 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f85c00d0880 0x7f85c00b4250 secure :-1 s=READY pgs=1502 cs=0 l=1 rev1=1 crypto rx=0x7f85b8009d90 tx=0x7f85b801c970 comp rx=0 tx=0).stop 2024-03-31T15:13:13.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.546+0000 7f85c5a00640 1 -- 172.21.15.16:0/4202451402 shutdown_connections 2024-03-31T15:13:13.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.546+0000 7f85c5a00640 1 --2- 172.21.15.16:0/4202451402 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f85c00d0880 0x7f85c00b4250 unknown :-1 s=CLOSED pgs=1502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:13.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.546+0000 7f85c5a00640 1 --2- 172.21.15.16:0/4202451402 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f85c015c3d0 0x7f85c006fc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:13.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.546+0000 7f85c5a00640 1 -- 172.21.15.16:0/4202451402 >> 172.21.15.16:0/4202451402 conn(0x7f85c006e670 msgr2=0x7f85c006ea50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:13:13.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.546+0000 7f85c5a00640 1 -- 172.21.15.16:0/4202451402 shutdown_connections 2024-03-31T15:13:13.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.546+0000 7f85c5a00640 1 -- 172.21.15.16:0/4202451402 wait complete. 2024-03-31T15:13:13.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.550+0000 7f85c5a00640 1 Processor -- start 2024-03-31T15:13:13.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.550+0000 7f85c5a00640 1 -- start start 2024-03-31T15:13:13.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.550+0000 7f85c5a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f85c00cdc30 0x7f85c016a2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:13.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.550+0000 7f85c5a00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f85c00d0880 0x7f85c0164490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:13.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.550+0000 7f85c5a00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f85c015c3d0 0x7f85c0168670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:13.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.550+0000 7f85c5a00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f85c0159100 con 0x7f85c00cdc30 2024-03-31T15:13:13.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.550+0000 7f85c5a00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f85c0158f80 con 0x7f85c00d0880 2024-03-31T15:13:13.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.550+0000 7f85c5a00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f85c006b620 con 0x7f85c015c3d0 2024-03-31T15:13:13.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.550+0000 7f85bea00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f85c00cdc30 0x7f85c016a2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:13.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.550+0000 7f85bea00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f85c00cdc30 0x7f85c016a2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45094/0 (socket says 172.21.15.16:45094) 2024-03-31T15:13:13.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.550+0000 7f85bea00640 1 -- 172.21.15.16:0/1160284342 learned_addr learned my addr 172.21.15.16:0/1160284342 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:13:13.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.550+0000 7f85bf400640 1 --2- 172.21.15.16:0/1160284342 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f85c015c3d0 0x7f85c0168670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:13.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.550+0000 7f85b7e00640 1 --2- 172.21.15.16:0/1160284342 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f85c00d0880 0x7f85c0164490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:13.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.550+0000 7f85bea00640 1 -- 172.21.15.16:0/1160284342 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f85c015c3d0 msgr2=0x7f85c0168670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:13:13.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.550+0000 7f85bea00640 1 --2- 172.21.15.16:0/1160284342 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f85c015c3d0 0x7f85c0168670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:13.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.550+0000 7f85bea00640 1 -- 172.21.15.16:0/1160284342 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f85c00d0880 msgr2=0x7f85c0164490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:13:13.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.550+0000 7f85bea00640 1 --2- 172.21.15.16:0/1160284342 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f85c00d0880 0x7f85c0164490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:13.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.550+0000 7f85bea00640 1 -- 172.21.15.16:0/1160284342 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85c00b27a0 con 0x7f85c00cdc30 2024-03-31T15:13:13.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.550+0000 7f85bea00640 1 --2- 172.21.15.16:0/1160284342 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f85c00cdc30 0x7f85c016a2f0 secure :-1 s=READY pgs=1503 cs=0 l=1 rev1=1 crypto rx=0x7f85b000c520 tx=0x7f85b000c9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:13:13.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.550+0000 7f85b6a00640 1 -- 172.21.15.16:0/1160284342 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85b0018070 con 0x7f85c00cdc30 2024-03-31T15:13:13.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.550+0000 7f85b6a00640 1 -- 172.21.15.16:0/1160284342 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f85b00074e0 con 0x7f85c00cdc30 2024-03-31T15:13:13.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.550+0000 7f85b6a00640 1 -- 172.21.15.16:0/1160284342 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85b00049f0 con 0x7f85c00cdc30 2024-03-31T15:13:13.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.550+0000 7f85c5a00640 1 -- 172.21.15.16:0/1160284342 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f85c0169250 con 0x7f85c00cdc30 2024-03-31T15:13:13.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.550+0000 7f85c5a00640 1 -- 172.21.15.16:0/1160284342 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f85c0169810 con 0x7f85c00cdc30 2024-03-31T15:13:13.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.554+0000 7f85b6a00640 1 -- 172.21.15.16:0/1160284342 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f85b0004b90 con 0x7f85c00cdc30 2024-03-31T15:13:13.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.554+0000 7f85b6a00640 1 --2- 172.21.15.16:0/1160284342 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f859c068380 0x7f859c06a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:13.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.554+0000 7f85b6a00640 1 -- 172.21.15.16:0/1160284342 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 5709+0+0 (secure 0 0 0) 0x7f85b0085d50 con 0x7f85c00cdc30 2024-03-31T15:13:13.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.554+0000 7f85b7e00640 1 --2- 172.21.15.16:0/1160284342 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f859c068380 0x7f859c06a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:13.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.554+0000 7f85c8f2f000 1 Processor -- start 2024-03-31T15:13:13.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.554+0000 7f85c8f2f000 1 -- start start 2024-03-31T15:13:13.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.554+0000 7f85b7e00640 1 --2- 172.21.15.16:0/1160284342 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f859c068380 0x7f859c06a840 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f85c00b0c80 tx=0x7f85a8008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:13:13.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.554+0000 7f85c8f2f000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f5781e82d0 0x55f5781e86b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:13.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.554+0000 7f85c8f2f000 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55f5781e8bf0 0x55f57819b470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:13.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.554+0000 7f85c8f2f000 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f57819b9b0 0x55f57819bd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:13.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.554+0000 7f85c8f2f000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55f578084af0 con 0x55f5781e82d0 2024-03-31T15:13:13.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.554+0000 7f85c8f2f000 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55f577fedb00 con 0x55f57819b9b0 2024-03-31T15:13:13.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.554+0000 7f85c8f2f000 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x55f577f8a940 con 0x55f5781e8bf0 2024-03-31T15:13:13.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.554+0000 7f85bea00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f5781e82d0 0x55f5781e86b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:13.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.554+0000 7f85bea00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f5781e82d0 0x55f5781e86b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45104/0 (socket says 172.21.15.16:45104) 2024-03-31T15:13:13.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.554+0000 7f85bea00640 1 -- 172.21.15.16:0/2648793121 learned_addr learned my addr 172.21.15.16:0/2648793121 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:13:13.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.554+0000 7f85b7e00640 1 --2- 172.21.15.16:0/2648793121 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55f5781e8bf0 0x55f57819b470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:13.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.554+0000 7f85bf400640 1 --2- 172.21.15.16:0/2648793121 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f57819b9b0 0x55f57819bd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:13.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.558+0000 7f85bea00640 1 -- 172.21.15.16:0/2648793121 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55f5781e8bf0 msgr2=0x55f57819b470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:13:13.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.558+0000 7f85bea00640 1 --2- 172.21.15.16:0/2648793121 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55f5781e8bf0 0x55f57819b470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:13.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.558+0000 7f85bea00640 1 -- 172.21.15.16:0/2648793121 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f57819b9b0 msgr2=0x55f57819bd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:13:13.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.558+0000 7f85bea00640 1 --2- 172.21.15.16:0/2648793121 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f57819b9b0 0x55f57819bd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:13.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.558+0000 7f85bea00640 1 -- 172.21.15.16:0/2648793121 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f577f73380 con 0x55f5781e82d0 2024-03-31T15:13:13.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.558+0000 7f85bea00640 1 --2- 172.21.15.16:0/2648793121 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f5781e82d0 0x55f5781e86b0 secure :-1 s=READY pgs=1504 cs=0 l=1 rev1=1 crypto rx=0x7f85b005e2b0 tx=0x7f85b0055b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17e897a2eed6e73 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:13:13.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.558+0000 7f85af400640 1 -- 172.21.15.16:0/2648793121 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85b005e7e0 con 0x55f5781e82d0 2024-03-31T15:13:13.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.558+0000 7f85af400640 1 -- 172.21.15.16:0/2648793121 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f85b0053f00 con 0x55f5781e82d0 2024-03-31T15:13:13.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.558+0000 7f85af400640 1 -- 172.21.15.16:0/2648793121 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85b0052270 con 0x55f5781e82d0 2024-03-31T15:13:13.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.558+0000 7f85c8f2f000 1 -- 172.21.15.16:0/2648793121 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f5781e82d0 msgr2=0x55f5781e86b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:13:13.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.558+0000 7f85c8f2f000 1 --2- 172.21.15.16:0/2648793121 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f5781e82d0 0x55f5781e86b0 secure :-1 s=READY pgs=1504 cs=0 l=1 rev1=1 crypto rx=0x7f85b005e2b0 tx=0x7f85b0055b00 comp rx=0 tx=0).stop 2024-03-31T15:13:13.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.558+0000 7f85c8f2f000 1 -- 172.21.15.16:0/2648793121 shutdown_connections 2024-03-31T15:13:13.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.558+0000 7f85c8f2f000 1 --2- 172.21.15.16:0/2648793121 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f57819b9b0 0x55f57819bd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:13.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.558+0000 7f85c8f2f000 1 --2- 172.21.15.16:0/2648793121 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55f5781e8bf0 0x55f57819b470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:13.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.558+0000 7f85c8f2f000 1 --2- 172.21.15.16:0/2648793121 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f5781e82d0 0x55f5781e86b0 unknown :-1 s=CLOSED pgs=1504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:13.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.558+0000 7f85c8f2f000 1 -- 172.21.15.16:0/2648793121 >> 172.21.15.16:0/2648793121 conn(0x55f5781c9070 msgr2=0x55f5781cb460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:13:13.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.558+0000 7f85c8f2f000 1 -- 172.21.15.16:0/2648793121 shutdown_connections 2024-03-31T15:13:13.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.558+0000 7f85c8f2f000 1 -- 172.21.15.16:0/2648793121 wait complete. 2024-03-31T15:13:13.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.558+0000 7f85c8f2f000 1 -- start start 2024-03-31T15:13:13.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.558+0000 7f85c8f2f000 1 Processor -- start 2024-03-31T15:13:13.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.558+0000 7f85c8f2f000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f57819b9b0 0x55f5781ef710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:13.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.558+0000 7f85c8f2f000 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55f5781e82d0 0x55f5781efc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:13.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.558+0000 7f85c8f2f000 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f5781e8bf0 0x55f5781a0590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:13.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.558+0000 7f85c8f2f000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55f577fab1e0 con 0x55f57819b9b0 2024-03-31T15:13:13.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.558+0000 7f85c8f2f000 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55f577fe9ab0 con 0x55f5781e8bf0 2024-03-31T15:13:13.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.558+0000 7f85c8f2f000 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x55f577fb9150 con 0x55f5781e82d0 2024-03-31T15:13:13.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.558+0000 7f85c8f2f000 10 client.0 _ll_register_callbacks cb 0x55f578189370 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T15:13:13.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.558+0000 7f85bea00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f57819b9b0 0x55f5781ef710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:13.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.558+0000 7f85bea00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f57819b9b0 0x55f5781ef710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45112/0 (socket says 172.21.15.16:45112) 2024-03-31T15:13:13.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.558+0000 7f85bea00640 1 -- 172.21.15.16:0/236184522 learned_addr learned my addr 172.21.15.16:0/236184522 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:13:13.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.558+0000 7f85b7e00640 1 --2- 172.21.15.16:0/236184522 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55f5781e82d0 0x55f5781efc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:13.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.558+0000 7f85bf400640 1 --2- 172.21.15.16:0/236184522 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f5781e8bf0 0x55f5781a0590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:13.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.558+0000 7f85bea00640 1 -- 172.21.15.16:0/236184522 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55f5781e82d0 msgr2=0x55f5781efc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:13:13.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.558+0000 7f85bea00640 1 --2- 172.21.15.16:0/236184522 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55f5781e82d0 0x55f5781efc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:13.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.558+0000 7f85bea00640 1 -- 172.21.15.16:0/236184522 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f5781e8bf0 msgr2=0x55f5781a0590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:13:13.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.558+0000 7f85bea00640 1 --2- 172.21.15.16:0/236184522 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f5781e8bf0 0x55f5781a0590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:13.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.558+0000 7f85bea00640 1 -- 172.21.15.16:0/236184522 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f577fa9b70 con 0x55f57819b9b0 2024-03-31T15:13:13.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.558+0000 7f85b7e00640 1 --2- 172.21.15.16:0/236184522 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55f5781e82d0 0x55f5781efc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:13:13.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.562+0000 7f85bea00640 1 --2- 172.21.15.16:0/236184522 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f57819b9b0 0x55f5781ef710 secure :-1 s=READY pgs=1505 cs=0 l=1 rev1=1 crypto rx=0x7f85b0000f70 tx=0x7f85b0000ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b207b405676f5ce server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:13:13.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.562+0000 7f85a3e00640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-31T15:13:13.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.562+0000 7f85a3e00640 1 -- 172.21.15.16:0/236184522 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85b007ddd0 con 0x55f57819b9b0 2024-03-31T15:13:13.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.562+0000 7f85c8f2f000 1 -- 172.21.15.16:0/236184522 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f85b0053f00 con 0x55f57819b9b0 2024-03-31T15:13:13.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.562+0000 7f85a3e00640 1 -- 172.21.15.16:0/236184522 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 5709+0+0 (secure 0 0 0) 0x7f85b0055050 con 0x55f57819b9b0 2024-03-31T15:13:13.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.562+0000 7f85a3e00640 1 -- 172.21.15.16:0/236184522 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=83) ==== 24+0+0 (secure 0 0 0) 0x7f85b005e7e0 con 0x55f57819b9b0 2024-03-31T15:13:13.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.562+0000 7f85c8f2f000 10 client.8859 fetch_fsmap learned FSMap version 83 2024-03-31T15:13:13.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.562+0000 7f85c8f2f000 1 -- 172.21.15.16:0/236184522 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=83,monmap=2+}) -- 0x55f577f8ab90 con 0x55f57819b9b0 2024-03-31T15:13:13.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.562+0000 7f85a3e00640 1 -- 172.21.15.16:0/236184522 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f85b0053f00 con 0x55f57819b9b0 2024-03-31T15:13:13.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.562+0000 7f85a3e00640 1 -- 172.21.15.16:0/236184522 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 83) ==== 3494+0+0 (secure 0 0 0) 0x7f85b005c1c0 con 0x55f57819b9b0 2024-03-31T15:13:13.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.562+0000 7f85c8f2f000 10 client.8859 fetch_fsmap finished waiting for FSMap version 83 2024-03-31T15:13:13.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.562+0000 7f85c8f2f000 10 client.8859 resolve_mds: resolved 7:0 to role '7:0' aka daemon mds.a 2024-03-31T15:13:13.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.562+0000 7f85c8f2f000 20 client.8859 populate_metadata read hostname 'smithi016' 2024-03-31T15:13:13.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.562+0000 7f85c8f2f000 1 --2- 172.21.15.16:0/236184522 >> [v2:172.21.15.16:6833/490717480,v1:172.21.15.16:6835/490717480] conn(0x55f5781a1300 0x55f5781a16e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:13.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.562+0000 7f85c8f2f000 4 client.8859 mds_command: new command op to 8074 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-31T15:13:13.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.562+0000 7f85c8f2f000 1 -- 172.21.15.16:0/236184522 --> [v2:172.21.15.16:6833/490717480,v1:172.21.15.16:6835/490717480] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f577f88dd0 con 0x55f5781a1300 2024-03-31T15:13:13.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.562+0000 7f85b7e00640 1 --2- 172.21.15.16:0/236184522 >> [v2:172.21.15.16:6833/490717480,v1:172.21.15.16:6835/490717480] conn(0x55f5781a1300 0x55f5781a16e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:13.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.562+0000 7f85b7e00640 1 --2- 172.21.15.16:0/236184522 >> [v2:172.21.15.16:6833/490717480,v1:172.21.15.16:6835/490717480] conn(0x55f5781a1300 0x55f5781a16e0 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1f359ff2f659f6e8 server_cookie=34a2caee84e7af7 in_seq=0 out_seq=0 2024-03-31T15:13:13.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.562+0000 7f85a3e00640 10 client.8859 ms_handle_connect on v2:172.21.15.16:6833/490717480 2024-03-31T15:13:13.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.562+0000 7f85a3e00640 1 -- 172.21.15.16:0/236184522 <== mds.0 v2:172.21.15.16:6833/490717480 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55f577f88dd0 con 0x55f5781a1300 2024-03-31T15:13:13.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.562+0000 7f85a3e00640 10 client.8859 handle_command_reply: tid=0 2024-03-31T15:13:13.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.562+0000 7f85c8f2f000 1 client.8859 shutdown 2024-03-31T15:13:13.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.562+0000 7f85c8f2f000 1 -- 172.21.15.16:0/236184522 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f57819b9b0 msgr2=0x55f5781ef710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:13:13.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.562+0000 7f85c8f2f000 1 --2- 172.21.15.16:0/236184522 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f57819b9b0 0x55f5781ef710 secure :-1 s=READY pgs=1505 cs=0 l=1 rev1=1 crypto rx=0x7f85b0000f70 tx=0x7f85b0000ee0 comp rx=0 tx=0).stop 2024-03-31T15:13:13.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.562+0000 7f85c8f2f000 1 -- 172.21.15.16:0/236184522 shutdown_connections 2024-03-31T15:13:13.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.562+0000 7f85c8f2f000 1 --2- 172.21.15.16:0/236184522 >> [v2:172.21.15.16:6833/490717480,v1:172.21.15.16:6835/490717480] conn(0x55f5781a1300 0x55f5781a16e0 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:13.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.562+0000 7f85c8f2f000 1 --2- 172.21.15.16:0/236184522 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f5781e8bf0 0x55f5781a0590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:13.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.562+0000 7f85c8f2f000 1 --2- 172.21.15.16:0/236184522 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55f5781e82d0 0x55f5781efc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:13.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.562+0000 7f85c8f2f000 1 --2- 172.21.15.16:0/236184522 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f57819b9b0 0x55f5781ef710 unknown :-1 s=CLOSED pgs=1505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:13.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.562+0000 7f85a3e00640 0 client.8859 ms_handle_reset on v2:172.21.15.16:6833/490717480 2024-03-31T15:13:13.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.562+0000 7f85c8f2f000 1 -- 172.21.15.16:0/236184522 >> 172.21.15.16:0/236184522 conn(0x55f5781c9070 msgr2=0x55f5781cb460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:13:13.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.562+0000 7f85c8f2f000 1 -- 172.21.15.16:0/236184522 shutdown_connections 2024-03-31T15:13:13.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.562+0000 7f85c8f2f000 1 -- 172.21.15.16:0/236184522 wait complete. 2024-03-31T15:13:13.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.562+0000 7f85c8f2f000 20 client.8859 trim_cache size 0 max 16384 2024-03-31T15:13:13.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.586+0000 7f85c8f2f000 1 Processor -- start 2024-03-31T15:13:13.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.586+0000 7f85c8f2f000 1 -- start start 2024-03-31T15:13:13.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.586+0000 7f85c8f2f000 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55f57819b9b0 0x55f5781ffe90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:13.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.586+0000 7f85c8f2f000 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f5781e82d0 0x55f5781fe910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:13.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.586+0000 7f85c8f2f000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f5781e8bf0 0x55f5781fee50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:13.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.586+0000 7f85c8f2f000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f85c014fa20 con 0x55f5781e8bf0 2024-03-31T15:13:13.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.586+0000 7f85c8f2f000 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f85c0150070 con 0x55f5781e82d0 2024-03-31T15:13:13.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.586+0000 7f85c8f2f000 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x55f5780bc420 con 0x55f57819b9b0 2024-03-31T15:13:13.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.586+0000 7f85bf400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f5781e8bf0 0x55f5781fee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:13.580 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.586+0000 7f85bf400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f5781e8bf0 0x55f5781fee50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45124/0 (socket says 172.21.15.16:45124) 2024-03-31T15:13:13.580 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.586+0000 7f85bf400640 1 -- 172.21.15.16:0/188301228 learned_addr learned my addr 172.21.15.16:0/188301228 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:13:13.580 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.586+0000 7f85bea00640 1 --2- 172.21.15.16:0/188301228 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55f57819b9b0 0x55f5781ffe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:13.580 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.586+0000 7f85b7e00640 1 --2- 172.21.15.16:0/188301228 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f5781e82d0 0x55f5781fe910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:13.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.586+0000 7f85bf400640 1 -- 172.21.15.16:0/188301228 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55f57819b9b0 msgr2=0x55f5781ffe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:13:13.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.586+0000 7f85bf400640 1 --2- 172.21.15.16:0/188301228 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55f57819b9b0 0x55f5781ffe90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:13.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.586+0000 7f85bf400640 1 -- 172.21.15.16:0/188301228 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f5781e82d0 msgr2=0x55f5781fe910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:13:13.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.586+0000 7f85bf400640 1 --2- 172.21.15.16:0/188301228 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f5781e82d0 0x55f5781fe910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:13.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.586+0000 7f85bf400640 1 -- 172.21.15.16:0/188301228 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f5781e8a20 con 0x55f5781e8bf0 2024-03-31T15:13:13.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.586+0000 7f85b7e00640 1 --2- 172.21.15.16:0/188301228 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f5781e82d0 0x55f5781fe910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:13:13.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.586+0000 7f85bf400640 1 --2- 172.21.15.16:0/188301228 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f5781e8bf0 0x55f5781fee50 secure :-1 s=READY pgs=1506 cs=0 l=1 rev1=1 crypto rx=0x7f85b80b0040 tx=0x7f85b8004ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c5b89b281ec5acdd server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:13:13.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.586+0000 7f85aea00640 1 -- 172.21.15.16:0/188301228 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85b80b2040 con 0x55f5781e8bf0 2024-03-31T15:13:13.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.586+0000 7f85aea00640 1 -- 172.21.15.16:0/188301228 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f85b80a58c0 con 0x55f5781e8bf0 2024-03-31T15:13:13.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.586+0000 7f85aea00640 1 -- 172.21.15.16:0/188301228 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85b80a5cd0 con 0x55f5781e8bf0 2024-03-31T15:13:13.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.586+0000 7f85c8f2f000 1 -- 172.21.15.16:0/188301228 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f5781e8bf0 msgr2=0x55f5781fee50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:13:13.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.586+0000 7f85c8f2f000 1 --2- 172.21.15.16:0/188301228 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f5781e8bf0 0x55f5781fee50 secure :-1 s=READY pgs=1506 cs=0 l=1 rev1=1 crypto rx=0x7f85b80b0040 tx=0x7f85b8004ac0 comp rx=0 tx=0).stop 2024-03-31T15:13:13.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.586+0000 7f85c8f2f000 1 -- 172.21.15.16:0/188301228 shutdown_connections 2024-03-31T15:13:13.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.586+0000 7f85c8f2f000 1 --2- 172.21.15.16:0/188301228 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f5781e8bf0 0x55f5781fee50 unknown :-1 s=CLOSED pgs=1506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:13.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.586+0000 7f85c8f2f000 1 --2- 172.21.15.16:0/188301228 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f5781e82d0 0x55f5781fe910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:13.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.586+0000 7f85c8f2f000 1 --2- 172.21.15.16:0/188301228 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55f57819b9b0 0x55f5781ffe90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:13.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.586+0000 7f85c8f2f000 1 -- 172.21.15.16:0/188301228 >> 172.21.15.16:0/188301228 conn(0x55f5781c9070 msgr2=0x55f5781e6380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:13:13.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.586+0000 7f85c8f2f000 1 -- 172.21.15.16:0/188301228 shutdown_connections 2024-03-31T15:13:13.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.586+0000 7f85c8f2f000 1 -- 172.21.15.16:0/188301228 wait complete. 2024-03-31T15:13:13.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.590+0000 7f85c8f2f000 1 -- start start 2024-03-31T15:13:13.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.590+0000 7f85c8f2f000 1 Processor -- start 2024-03-31T15:13:13.587 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.590+0000 7f85c8f2f000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f57819b9b0 0x55f578254690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:13.587 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.590+0000 7f85c8f2f000 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55f5781e82d0 0x55f5781ec030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:13.587 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.590+0000 7f85c8f2f000 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f5781e8bf0 0x55f5781ec570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:13.587 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.590+0000 7f85c8f2f000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55f577f88940 con 0x55f57819b9b0 2024-03-31T15:13:13.587 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.590+0000 7f85c8f2f000 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55f5781a7000 con 0x55f5781e8bf0 2024-03-31T15:13:13.587 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.590+0000 7f85c8f2f000 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x55f577fa4750 con 0x55f5781e82d0 2024-03-31T15:13:13.587 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.590+0000 7f85c8f2f000 10 client.0 _ll_register_callbacks cb 0x7f85b006d070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T15:13:13.587 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.590+0000 7f85bea00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f57819b9b0 0x55f578254690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:13.587 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.590+0000 7f85bea00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f57819b9b0 0x55f578254690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45130/0 (socket says 172.21.15.16:45130) 2024-03-31T15:13:13.587 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.590+0000 7f85bea00640 1 -- 172.21.15.16:0/1162754536 learned_addr learned my addr 172.21.15.16:0/1162754536 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:13:13.587 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.590+0000 7f85b7e00640 1 --2- 172.21.15.16:0/1162754536 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55f5781e82d0 0x55f5781ec030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:13.587 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.590+0000 7f85bf400640 1 --2- 172.21.15.16:0/1162754536 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f5781e8bf0 0x55f5781ec570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:13.587 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.590+0000 7f85bea00640 1 -- 172.21.15.16:0/1162754536 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55f5781e82d0 msgr2=0x55f5781ec030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:13:13.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.590+0000 7f85bea00640 1 --2- 172.21.15.16:0/1162754536 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55f5781e82d0 0x55f5781ec030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:13.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.590+0000 7f85bea00640 1 -- 172.21.15.16:0/1162754536 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f5781e8bf0 msgr2=0x55f5781ec570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:13:13.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.590+0000 7f85bea00640 1 --2- 172.21.15.16:0/1162754536 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f5781e8bf0 0x55f5781ec570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:13.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.590+0000 7f85bea00640 1 -- 172.21.15.16:0/1162754536 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f57806e140 con 0x55f57819b9b0 2024-03-31T15:13:13.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.590+0000 7f85bf400640 1 --2- 172.21.15.16:0/1162754536 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f5781e8bf0 0x55f5781ec570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:13:13.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.590+0000 7f85bea00640 1 --2- 172.21.15.16:0/1162754536 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f57819b9b0 0x55f578254690 secure :-1 s=READY pgs=1507 cs=0 l=1 rev1=1 crypto rx=0x7f85b0055900 tx=0x7f85b0014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=137f5dcfcfd2e3c7 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:13:13.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.590+0000 7f85a3e00640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-31T15:13:13.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.590+0000 7f85a3e00640 1 -- 172.21.15.16:0/1162754536 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85b007ddd0 con 0x55f57819b9b0 2024-03-31T15:13:13.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.590+0000 7f85a3e00640 1 -- 172.21.15.16:0/1162754536 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 5709+0+0 (secure 0 0 0) 0x7f85b006bcc0 con 0x55f57819b9b0 2024-03-31T15:13:13.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.590+0000 7f85c8f2f000 1 -- 172.21.15.16:0/1162754536 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f85b80a58c0 con 0x55f57819b9b0 2024-03-31T15:13:13.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.590+0000 7f85a3e00640 1 -- 172.21.15.16:0/1162754536 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=83) ==== 24+0+0 (secure 0 0 0) 0x7f85b0055050 con 0x55f57819b9b0 2024-03-31T15:13:13.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.590+0000 7f85c8f2f000 10 client.8865 fetch_fsmap learned FSMap version 83 2024-03-31T15:13:13.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.590+0000 7f85c8f2f000 1 -- 172.21.15.16:0/1162754536 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=83,monmap=2+}) -- 0x55f577f7c1e0 con 0x55f57819b9b0 2024-03-31T15:13:13.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.590+0000 7f85a3e00640 1 -- 172.21.15.16:0/1162754536 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f85b80a58c0 con 0x55f57819b9b0 2024-03-31T15:13:13.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.590+0000 7f85a3e00640 1 -- 172.21.15.16:0/1162754536 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 83) ==== 3494+0+0 (secure 0 0 0) 0x7f85b00551f0 con 0x55f57819b9b0 2024-03-31T15:13:13.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.590+0000 7f85c8f2f000 10 client.8865 fetch_fsmap finished waiting for FSMap version 83 2024-03-31T15:13:13.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.590+0000 7f85c8f2f000 10 client.8865 resolve_mds: resolved 7:0 to role '7:0' aka daemon mds.a 2024-03-31T15:13:13.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.590+0000 7f85c8f2f000 20 client.8865 populate_metadata read hostname 'smithi016' 2024-03-31T15:13:13.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.590+0000 7f85c8f2f000 1 --2- 172.21.15.16:0/1162754536 >> [v2:172.21.15.16:6833/490717480,v1:172.21.15.16:6835/490717480] conn(0x55f5781ecab0 0x55f5781ecec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:13.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.590+0000 7f85c8f2f000 4 client.8865 mds_command: new command op to 8074 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-31T15:13:13.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.590+0000 7f85c8f2f000 1 -- 172.21.15.16:0/1162754536 --> [v2:172.21.15.16:6833/490717480,v1:172.21.15.16:6835/490717480] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55f577ffb040 con 0x55f5781ecab0 2024-03-31T15:13:13.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.590+0000 7f85b7e00640 1 --2- 172.21.15.16:0/1162754536 >> [v2:172.21.15.16:6833/490717480,v1:172.21.15.16:6835/490717480] conn(0x55f5781ecab0 0x55f5781ecec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:13.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.590+0000 7f85b7e00640 1 --2- 172.21.15.16:0/1162754536 >> [v2:172.21.15.16:6833/490717480,v1:172.21.15.16:6835/490717480] conn(0x55f5781ecab0 0x55f5781ecec0 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d0d5a2ea30223c57 server_cookie=f37ace4fb9165f03 in_seq=0 out_seq=0 2024-03-31T15:13:13.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.590+0000 7f85a3e00640 10 client.8865 ms_handle_connect on v2:172.21.15.16:6833/490717480 2024-03-31T15:13:13.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.594+0000 7f85a3e00640 1 -- 172.21.15.16:0/1162754536 <== mds.0 v2:172.21.15.16:6833/490717480 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55f577ffb040 con 0x55f5781ecab0 2024-03-31T15:13:13.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.594+0000 7f85a3e00640 10 client.8865 handle_command_reply: tid=0 2024-03-31T15:13:13.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.594+0000 7f85c8f2f000 1 client.8865 shutdown 2024-03-31T15:13:13.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.594+0000 7f85c8f2f000 1 -- 172.21.15.16:0/1162754536 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f57819b9b0 msgr2=0x55f578254690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:13:13.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.594+0000 7f85c8f2f000 1 --2- 172.21.15.16:0/1162754536 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f57819b9b0 0x55f578254690 secure :-1 s=READY pgs=1507 cs=0 l=1 rev1=1 crypto rx=0x7f85b0055900 tx=0x7f85b0014000 comp rx=0 tx=0).stop 2024-03-31T15:13:13.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.594+0000 7f85c8f2f000 1 -- 172.21.15.16:0/1162754536 shutdown_connections 2024-03-31T15:13:13.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.594+0000 7f85c8f2f000 1 --2- 172.21.15.16:0/1162754536 >> [v2:172.21.15.16:6833/490717480,v1:172.21.15.16:6835/490717480] conn(0x55f5781ecab0 0x55f5781ecec0 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:13.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.594+0000 7f85c8f2f000 1 --2- 172.21.15.16:0/1162754536 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f5781e8bf0 0x55f5781ec570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:13.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.594+0000 7f85c8f2f000 1 --2- 172.21.15.16:0/1162754536 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55f5781e82d0 0x55f5781ec030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:13.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.594+0000 7f85c8f2f000 1 --2- 172.21.15.16:0/1162754536 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f57819b9b0 0x55f578254690 unknown :-1 s=CLOSED pgs=1507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:13.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.594+0000 7f85c8f2f000 1 -- 172.21.15.16:0/1162754536 >> 172.21.15.16:0/1162754536 conn(0x55f5781c9070 msgr2=0x55f5781e6c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:13:13.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.594+0000 7f85a3e00640 0 client.8865 ms_handle_reset on v2:172.21.15.16:6833/490717480 2024-03-31T15:13:13.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.594+0000 7f85c8f2f000 1 -- 172.21.15.16:0/1162754536 shutdown_connections 2024-03-31T15:13:13.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.594+0000 7f85c8f2f000 1 -- 172.21.15.16:0/1162754536 wait complete. 2024-03-31T15:13:13.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.594+0000 7f85c8f2f000 20 client.8865 trim_cache size 0 max 16384 2024-03-31T15:13:13.591 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-03-31T15:13:13.591 INFO:teuthology.orchestra.run.smithi016.stdout: "status": "no active scrubs running", 2024-03-31T15:13:13.591 INFO:teuthology.orchestra.run.smithi016.stdout: "scrubs": {} 2024-03-31T15:13:13.591 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-03-31T15:13:13.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.594+0000 7f85c5a00640 1 -- 172.21.15.16:0/1160284342 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f859c068380 msgr2=0x7f859c06a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:13:13.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.594+0000 7f85c5a00640 1 --2- 172.21.15.16:0/1160284342 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f859c068380 0x7f859c06a840 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f85c00b0c80 tx=0x7f85a8008040 comp rx=0 tx=0).stop 2024-03-31T15:13:13.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.594+0000 7f85c5a00640 1 -- 172.21.15.16:0/1160284342 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f85c00cdc30 msgr2=0x7f85c016a2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:13:13.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.594+0000 7f85c5a00640 1 --2- 172.21.15.16:0/1160284342 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f85c00cdc30 0x7f85c016a2f0 secure :-1 s=READY pgs=1503 cs=0 l=1 rev1=1 crypto rx=0x7f85b000c520 tx=0x7f85b000c9e0 comp rx=0 tx=0).stop 2024-03-31T15:13:13.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.594+0000 7f85c5a00640 1 -- 172.21.15.16:0/1160284342 shutdown_connections 2024-03-31T15:13:13.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.594+0000 7f85c5a00640 1 --2- 172.21.15.16:0/1160284342 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f859c068380 0x7f859c06a840 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:13.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.594+0000 7f85c5a00640 1 --2- 172.21.15.16:0/1160284342 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f85c015c3d0 0x7f85c0168670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:13.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.594+0000 7f85c5a00640 1 --2- 172.21.15.16:0/1160284342 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f85c00d0880 0x7f85c0164490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:13.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.594+0000 7f85c5a00640 1 --2- 172.21.15.16:0/1160284342 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f85c00cdc30 0x7f85c016a2f0 unknown :-1 s=CLOSED pgs=1503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:13.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.594+0000 7f85c5a00640 1 -- 172.21.15.16:0/1160284342 >> 172.21.15.16:0/1160284342 conn(0x7f85c006e670 msgr2=0x7f85c0165450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:13:13.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.594+0000 7f85c5a00640 1 -- 172.21.15.16:0/1160284342 shutdown_connections 2024-03-31T15:13:13.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.594+0000 7f85c5a00640 1 -- 172.21.15.16:0/1160284342 wait complete. 2024-03-31T15:13:13.607 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-31T15:13:13.708 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-31T15:13:13.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.858+0000 7f156b000640 1 Processor -- start 2024-03-31T15:13:13.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.858+0000 7f156b000640 1 -- start start 2024-03-31T15:13:13.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.858+0000 7f156b000640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f156406dbc0 0x7f156415f650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:13.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.858+0000 7f156b000640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f15640cb440 0x7f156406f920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:13.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.858+0000 7f156b000640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f15640cbe90 con 0x7f15640cb8c0 2024-03-31T15:13:13.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.858+0000 7f156b000640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f156415fb90 con 0x7f156406dbc0 2024-03-31T15:13:13.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.858+0000 7f156b000640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f156415f1c0 con 0x7f15640cb440 2024-03-31T15:13:13.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.858+0000 7f155be00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f15640cb440 0x7f156406f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:13.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.858+0000 7f1563e00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f156406dbc0 0x7f156415f650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:13.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.858+0000 7f155be00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f15640cb440 0x7f156406f920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3301/0 says I am v2:172.21.15.16:35466/0 (socket says 172.21.15.16:35466) 2024-03-31T15:13:13.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.858+0000 7f155be00640 1 -- 172.21.15.16:0/4144276589 learned_addr learned my addr 172.21.15.16:0/4144276589 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:13:13.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.858+0000 7f155be00640 1 -- 172.21.15.16:0/4144276589 >> v2:172.21.15.89:3300/0 conn(0x7f156406dbc0 msgr2=0x7f156415f650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:13:13.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.858+0000 7f155be00640 1 --2- 172.21.15.16:0/4144276589 >> v2:172.21.15.89:3300/0 conn(0x7f156406dbc0 0x7f156415f650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:13.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.858+0000 7f155be00640 1 -- 172.21.15.16:0/4144276589 >> v1:172.21.15.89:6789/0 conn(0x7f15640cb8c0 legacy=0x7f1564162180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:13:13.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.858+0000 7f155be00640 1 -- 172.21.15.16:0/4144276589 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15640cc170 con 0x7f15640cb440 2024-03-31T15:13:13.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.862+0000 7f155be00640 1 --2- 172.21.15.16:0/4144276589 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f15640cb440 0x7f156406f920 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f154c009870 tx=0x7f154c01c4a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=75f3aa534899b7df server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:13:13.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.862+0000 7f1563400640 1 -- 172.21.15.16:0/4144276589 <== mon.2 v2:172.21.15.89:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f154c0aa620 con 0x7f15640cb440 2024-03-31T15:13:13.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.862+0000 7f1563400640 1 -- 172.21.15.16:0/4144276589 <== mon.2 v2:172.21.15.89:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f154c0af070 con 0x7f15640cb440 2024-03-31T15:13:13.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.862+0000 7f1563400640 1 -- 172.21.15.16:0/4144276589 <== mon.2 v2:172.21.15.89:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f154c0aa9e0 con 0x7f15640cb440 2024-03-31T15:13:13.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.862+0000 7f156b000640 1 -- 172.21.15.16:0/4144276589 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f15640cb440 msgr2=0x7f156406f920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:13:13.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.862+0000 7f156b000640 1 --2- 172.21.15.16:0/4144276589 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f15640cb440 0x7f156406f920 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f154c009870 tx=0x7f154c01c4a0 comp rx=0 tx=0).stop 2024-03-31T15:13:13.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.862+0000 7f156b000640 1 -- 172.21.15.16:0/4144276589 shutdown_connections 2024-03-31T15:13:13.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.862+0000 7f156b000640 1 --2- 172.21.15.16:0/4144276589 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f15640cb440 0x7f156406f920 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:13.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.862+0000 7f156b000640 1 --2- 172.21.15.16:0/4144276589 >> v2:172.21.15.89:3300/0 conn(0x7f156406dbc0 0x7f156415f650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:13.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.862+0000 7f156b000640 1 -- 172.21.15.16:0/4144276589 >> 172.21.15.16:0/4144276589 conn(0x7f1564067330 msgr2=0x7f1564067740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:13:13.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.862+0000 7f156b000640 1 -- 172.21.15.16:0/4144276589 shutdown_connections 2024-03-31T15:13:13.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.862+0000 7f156b000640 1 -- 172.21.15.16:0/4144276589 wait complete. 2024-03-31T15:13:13.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.866+0000 7f156b000640 1 Processor -- start 2024-03-31T15:13:13.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.866+0000 7f156b000640 1 -- start start 2024-03-31T15:13:13.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.866+0000 7f156b000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f156406dbc0 0x7f1564144630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:13.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.866+0000 7f156b000640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f15640cb440 0x7f156415a570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:13.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.866+0000 7f156b000640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f15640cb8c0 0x7f15640b6a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:13.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.866+0000 7f156b000640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f1564162ca0 con 0x7f156406dbc0 2024-03-31T15:13:13.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.866+0000 7f156b000640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f1564070270 con 0x7f15640cb8c0 2024-03-31T15:13:13.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.866+0000 7f156b000640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f156406c950 con 0x7f15640cb440 2024-03-31T15:13:13.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.866+0000 7f1563e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f156406dbc0 0x7f1564144630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:13.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.866+0000 7f155be00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f15640cb440 0x7f156415a570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:13.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.866+0000 7f1563e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f156406dbc0 0x7f1564144630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45140/0 (socket says 172.21.15.16:45140) 2024-03-31T15:13:13.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.866+0000 7f1563e00640 1 -- 172.21.15.16:0/492663749 learned_addr learned my addr 172.21.15.16:0/492663749 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:13:13.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.866+0000 7f1569000640 1 --2- 172.21.15.16:0/492663749 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f15640cb8c0 0x7f15640b6a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:13.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.866+0000 7f1563e00640 1 -- 172.21.15.16:0/492663749 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f15640cb440 msgr2=0x7f156415a570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:13:13.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.866+0000 7f1563e00640 1 --2- 172.21.15.16:0/492663749 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f15640cb440 0x7f156415a570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:13.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.866+0000 7f1563e00640 1 -- 172.21.15.16:0/492663749 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f15640cb8c0 msgr2=0x7f15640b6a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:13:13.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.866+0000 7f1563e00640 1 --2- 172.21.15.16:0/492663749 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f15640cb8c0 0x7f15640b6a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:13.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.866+0000 7f1563e00640 1 -- 172.21.15.16:0/492663749 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f156415afa0 con 0x7f156406dbc0 2024-03-31T15:13:13.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.866+0000 7f1563e00640 1 --2- 172.21.15.16:0/492663749 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f156406dbc0 0x7f1564144630 secure :-1 s=READY pgs=1508 cs=0 l=1 rev1=1 crypto rx=0x7f155400e8a0 tx=0x7f155400eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:13:13.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.866+0000 7f1560c00640 1 -- 172.21.15.16:0/492663749 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1554017070 con 0x7f156406dbc0 2024-03-31T15:13:13.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.866+0000 7f1560c00640 1 -- 172.21.15.16:0/492663749 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f155400c940 con 0x7f156406dbc0 2024-03-31T15:13:13.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.866+0000 7f1560c00640 1 -- 172.21.15.16:0/492663749 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f155400ccf0 con 0x7f156406dbc0 2024-03-31T15:13:13.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.866+0000 7f156b000640 1 -- 172.21.15.16:0/492663749 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f156415b280 con 0x7f156406dbc0 2024-03-31T15:13:13.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.866+0000 7f156b000640 1 -- 172.21.15.16:0/492663749 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f156415b840 con 0x7f156406dbc0 2024-03-31T15:13:13.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.866+0000 7f156b000640 1 -- 172.21.15.16:0/492663749 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1564069950 con 0x7f156406dbc0 2024-03-31T15:13:13.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.870+0000 7f1560c00640 1 -- 172.21.15.16:0/492663749 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f155401a070 con 0x7f156406dbc0 2024-03-31T15:13:13.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.870+0000 7f1560c00640 1 --2- 172.21.15.16:0/492663749 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f1540068380 0x7f154006a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:13.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.870+0000 7f1560c00640 1 -- 172.21.15.16:0/492663749 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 5709+0+0 (secure 0 0 0) 0x7f1554086ff0 con 0x7f156406dbc0 2024-03-31T15:13:13.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.870+0000 7f155be00640 1 --2- 172.21.15.16:0/492663749 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f1540068380 0x7f154006a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:13.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.870+0000 7f155be00640 1 --2- 172.21.15.16:0/492663749 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f1540068380 0x7f154006a840 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f154c0ae000 tx=0x7f154c0a7000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:13:13.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:13.874+0000 7f1560c00640 1 -- 172.21.15.16:0/492663749 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f1554056f80 con 0x7f156406dbc0 2024-03-31T15:13:14.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.074+0000 7f156b000640 1 -- 172.21.15.16:0/492663749 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f15640cf380 con 0x7f156406dbc0 2024-03-31T15:13:14.068 INFO:tasks.ceph.mon.a.smithi016.stderr:2024-03-31T15:13:14.074+0000 7f7dcbc00640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-31T15:13:14.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.078+0000 7f1560c00640 1 -- 172.21.15.16:0/492663749 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v84) ==== 161+0+0 (secure 0 0 0) 0x7f155405b280 con 0x7f156406dbc0 2024-03-31T15:13:14.070 INFO:teuthology.orchestra.run.smithi016.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-31T15:13:14.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.078+0000 7f156b000640 1 -- 172.21.15.16:0/492663749 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f1540068380 msgr2=0x7f154006a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:13:14.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.078+0000 7f156b000640 1 --2- 172.21.15.16:0/492663749 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f1540068380 0x7f154006a840 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f154c0ae000 tx=0x7f154c0a7000 comp rx=0 tx=0).stop 2024-03-31T15:13:14.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.078+0000 7f156b000640 1 -- 172.21.15.16:0/492663749 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f156406dbc0 msgr2=0x7f1564144630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:13:14.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.078+0000 7f156b000640 1 --2- 172.21.15.16:0/492663749 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f156406dbc0 0x7f1564144630 secure :-1 s=READY pgs=1508 cs=0 l=1 rev1=1 crypto rx=0x7f155400e8a0 tx=0x7f155400eca0 comp rx=0 tx=0).stop 2024-03-31T15:13:14.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.078+0000 7f156b000640 1 -- 172.21.15.16:0/492663749 shutdown_connections 2024-03-31T15:13:14.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.078+0000 7f156b000640 1 --2- 172.21.15.16:0/492663749 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f1540068380 0x7f154006a840 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:14.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.078+0000 7f156b000640 1 --2- 172.21.15.16:0/492663749 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f15640cb8c0 0x7f15640b6a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:14.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.078+0000 7f156b000640 1 --2- 172.21.15.16:0/492663749 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f15640cb440 0x7f156415a570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:14.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.078+0000 7f156b000640 1 --2- 172.21.15.16:0/492663749 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f156406dbc0 0x7f1564144630 unknown :-1 s=CLOSED pgs=1508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:14.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.078+0000 7f156b000640 1 -- 172.21.15.16:0/492663749 >> 172.21.15.16:0/492663749 conn(0x7f1564067330 msgr2=0x7f15640d07c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:13:14.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.078+0000 7f156b000640 1 -- 172.21.15.16:0/492663749 shutdown_connections 2024-03-31T15:13:14.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.082+0000 7f156b000640 1 -- 172.21.15.16:0/492663749 wait complete. 2024-03-31T15:13:14.085 INFO:tasks.ceph.mds.a.smithi016.stderr: -6145> 2024-03-31T15:12:22.859+0000 7fde63e00640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-31T15:13:14.098 DEBUG:teuthology.orchestra.run.smithi016:> cephfs-journal-tool --debug-mds=20 --debug-ms=1 --debug-objecter=1 --rank cephfs:0 event splice --inode=1099511627777 summary 2024-03-31T15:13:14.134 INFO:tasks.ceph.mds.a.smithi016.stdout:starting mds.a at 2024-03-31T15:13:14.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.150+0000 7f973be4a400 1 -- start start 2024-03-31T15:13:14.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.150+0000 7f973be4a400 1 Processor -- start 2024-03-31T15:13:14.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.150+0000 7f973be4a400 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bf3516f800 0x55bf35f30000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:14.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.150+0000 7f973be4a400 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55bf3516f400 0x55bf35f30580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:14.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.150+0000 7f973be4a400 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55bf3516f000 0x55bf35f30b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:14.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.150+0000 7f973be4a400 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55bf35f42600 con 0x55bf3516f800 2024-03-31T15:13:14.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.150+0000 7f973be4a400 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55bf35f42480 con 0x55bf3516f000 2024-03-31T15:13:14.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.150+0000 7f973be4a400 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x55bf35f42300 con 0x55bf3516f400 2024-03-31T15:13:14.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.150+0000 7f9739400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bf3516f800 0x55bf35f30000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:14.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.154+0000 7f9738a00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55bf3516f400 0x55bf35f30580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:14.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.154+0000 7f9738a00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55bf3516f400 0x55bf35f30580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3301/0 says I am v2:172.21.15.16:35506/0 (socket says 172.21.15.16:35506) 2024-03-31T15:13:14.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.154+0000 7f9738a00640 1 -- 172.21.15.16:0/703517262 learned_addr learned my addr 172.21.15.16:0/703517262 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:13:14.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.154+0000 7f9739e00640 1 --2- 172.21.15.16:0/703517262 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55bf3516f000 0x55bf35f30b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:14.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.154+0000 7f9738a00640 1 -- 172.21.15.16:0/703517262 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55bf3516f000 msgr2=0x55bf35f30b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:13:14.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.154+0000 7f9738a00640 1 --2- 172.21.15.16:0/703517262 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55bf3516f000 0x55bf35f30b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:14.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.154+0000 7f9738a00640 1 -- 172.21.15.16:0/703517262 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bf3516f800 msgr2=0x55bf35f30000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:13:14.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.154+0000 7f9738a00640 1 --2- 172.21.15.16:0/703517262 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bf3516f800 0x55bf35f30000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:14.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.154+0000 7f9738a00640 1 -- 172.21.15.16:0/703517262 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bf35f2a960 con 0x55bf3516f400 2024-03-31T15:13:14.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.154+0000 7f9739400640 1 --2- 172.21.15.16:0/703517262 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bf3516f800 0x55bf35f30000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:13:14.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.154+0000 7f9738a00640 1 --2- 172.21.15.16:0/703517262 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55bf3516f400 0x55bf35f30580 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x55bf35fd6990 tx=0x55bf35f40000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bb1ca4b53e934a16 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:13:14.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.154+0000 7f9736200640 1 -- 172.21.15.16:0/703517262 <== mon.2 v2:172.21.15.89:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x55bf351a49c0 con 0x55bf3516f400 2024-03-31T15:13:14.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.154+0000 7f9736200640 1 -- 172.21.15.16:0/703517262 <== mon.2 v2:172.21.15.89:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x55bf351bac40 con 0x55bf3516f400 2024-03-31T15:13:14.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.154+0000 7f9736200640 1 -- 172.21.15.16:0/703517262 <== mon.2 v2:172.21.15.89:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x55bf351a44e0 con 0x55bf3516f400 2024-03-31T15:13:14.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.154+0000 7f973be4a400 1 -- 172.21.15.16:0/703517262 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({osdmap=0}) -- 0x55bf35f2ad20 con 0x55bf3516f400 2024-03-31T15:13:14.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.154+0000 7f973be4a400 1 -- 172.21.15.16:0/703517262 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({fsmap=0}) -- 0x55bf35f2af00 con 0x55bf3516f400 2024-03-31T15:13:14.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.154+0000 7f973be4a400 4 waiting for MDS map... 2024-03-31T15:13:14.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.154+0000 7f9736200640 1 -- 172.21.15.16:0/703517262 <== mon.2 v2:172.21.15.89:3301/0 4 ==== osd_map(97..97 src has 1..97) ==== 5795+0+0 (secure 0 0 0) 0x55bf36096200 con 0x55bf3516f400 2024-03-31T15:13:14.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.154+0000 7f9736200640 1 -- 172.21.15.16:0/703517262 <== mon.2 v2:172.21.15.89:3301/0 5 ==== fsmap(e 84) ==== 2885+0+0 (secure 0 0 0) 0x55bf351c5180 con 0x55bf3516f400 2024-03-31T15:13:14.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.154+0000 7f973be4a400 4 Got MDS map 84 2024-03-31T15:13:14.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.154+0000 7f973be4a400 10 main: JournalTool::main 2024-03-31T15:13:14.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.154+0000 7f973be4a400 4 main: JournalTool: connecting to RADOS... 2024-03-31T15:13:14.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.154+0000 7f973be4a400 1 Processor -- start 2024-03-31T15:13:14.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.154+0000 7f973be4a400 1 -- start start 2024-03-31T15:13:14.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.154+0000 7f973be4a400 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55bf360e8000 0x55bf35f31080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:14.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.154+0000 7f973be4a400 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bf360e8400 0x55bf35f31600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:14.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.154+0000 7f973be4a400 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55bf360e8800 0x55bf35f31b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:14.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.154+0000 7f973be4a400 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55bf35f42180 con 0x55bf360e8400 2024-03-31T15:13:14.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.154+0000 7f973be4a400 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55bf351ce480 con 0x55bf360e8000 2024-03-31T15:13:14.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.154+0000 7f973be4a400 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x55bf351ce900 con 0x55bf360e8800 2024-03-31T15:13:14.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.154+0000 7f9739400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bf360e8400 0x55bf35f31600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:14.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.154+0000 7f9739400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bf360e8400 0x55bf35f31600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45188/0 (socket says 172.21.15.16:45188) 2024-03-31T15:13:14.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.154+0000 7f9739400640 1 -- 172.21.15.16:0/3961684486 learned_addr learned my addr 172.21.15.16:0/3961684486 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:13:14.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.154+0000 7f9739e00640 1 --2- 172.21.15.16:0/3961684486 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55bf360e8000 0x55bf35f31080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:14.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.154+0000 7f9738a00640 1 --2- 172.21.15.16:0/3961684486 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55bf360e8800 0x55bf35f31b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:14.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.158+0000 7f9739400640 1 -- 172.21.15.16:0/3961684486 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55bf360e8800 msgr2=0x55bf35f31b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:13:14.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.158+0000 7f9739400640 1 --2- 172.21.15.16:0/3961684486 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55bf360e8800 0x55bf35f31b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:14.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.158+0000 7f9739400640 1 -- 172.21.15.16:0/3961684486 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55bf360e8000 msgr2=0x55bf35f31080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:13:14.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.158+0000 7f9739400640 1 --2- 172.21.15.16:0/3961684486 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55bf360e8000 0x55bf35f31080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:14.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.158+0000 7f9739400640 1 -- 172.21.15.16:0/3961684486 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bf35f2a1e0 con 0x55bf360e8400 2024-03-31T15:13:14.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.158+0000 7f9738a00640 1 --2- 172.21.15.16:0/3961684486 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55bf360e8800 0x55bf35f31b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:13:14.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.158+0000 7f9739400640 1 --2- 172.21.15.16:0/3961684486 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bf360e8400 0x55bf35f31600 secure :-1 s=READY pgs=1513 cs=0 l=1 rev1=1 crypto rx=0x55bf360f4510 tx=0x55bf35f41ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78d46e1da1c2f2dc server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:13:14.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.158+0000 7f9733a00640 1 -- 172.21.15.16:0/3961684486 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x55bf351a49c0 con 0x55bf360e8400 2024-03-31T15:13:14.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.158+0000 7f9733a00640 1 -- 172.21.15.16:0/3961684486 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x55bf351bb340 con 0x55bf360e8400 2024-03-31T15:13:14.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.158+0000 7f9733a00640 1 -- 172.21.15.16:0/3961684486 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x55bf351a44e0 con 0x55bf360e8400 2024-03-31T15:13:14.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.158+0000 7f973be4a400 1 -- 172.21.15.16:0/3961684486 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bf360e8400 msgr2=0x55bf35f31600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:13:14.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.158+0000 7f973be4a400 1 --2- 172.21.15.16:0/3961684486 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bf360e8400 0x55bf35f31600 secure :-1 s=READY pgs=1513 cs=0 l=1 rev1=1 crypto rx=0x55bf360f4510 tx=0x55bf35f41ec0 comp rx=0 tx=0).stop 2024-03-31T15:13:14.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.158+0000 7f973be4a400 1 -- 172.21.15.16:0/3961684486 shutdown_connections 2024-03-31T15:13:14.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.158+0000 7f973be4a400 1 --2- 172.21.15.16:0/3961684486 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55bf360e8800 0x55bf35f31b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:14.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.158+0000 7f973be4a400 1 --2- 172.21.15.16:0/3961684486 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bf360e8400 0x55bf35f31600 unknown :-1 s=CLOSED pgs=1513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:14.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.158+0000 7f973be4a400 1 --2- 172.21.15.16:0/3961684486 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55bf360e8000 0x55bf35f31080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:14.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.158+0000 7f973be4a400 1 -- 172.21.15.16:0/3961684486 >> 172.21.15.16:0/3961684486 conn(0x55bf3516ec00 msgr2=0x55bf3519b000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:13:14.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.158+0000 7f973be4a400 1 -- 172.21.15.16:0/3961684486 shutdown_connections 2024-03-31T15:13:14.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.158+0000 7f973be4a400 1 -- 172.21.15.16:0/3961684486 wait complete. 2024-03-31T15:13:14.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.158+0000 7f973be4a400 1 Processor -- start 2024-03-31T15:13:14.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.158+0000 7f973be4a400 1 -- start start 2024-03-31T15:13:14.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.158+0000 7f973be4a400 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55bf360e8000 0x55bf35f31080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:14.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.158+0000 7f973be4a400 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55bf360e8400 0x55bf35f31600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:14.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.158+0000 7f973be4a400 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bf360e8800 0x55bf35f31b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:14.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.158+0000 7f973be4a400 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55bf35f43380 con 0x55bf360e8800 2024-03-31T15:13:14.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.158+0000 7f973be4a400 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55bf35f42180 con 0x55bf360e8000 2024-03-31T15:13:14.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.158+0000 7f973be4a400 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x55bf35f43500 con 0x55bf360e8400 2024-03-31T15:13:14.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.158+0000 7f9738a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bf360e8800 0x55bf35f31b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:14.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.158+0000 7f9739400640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55bf360e8400 0x55bf35f31600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:14.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.158+0000 7f9738a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bf360e8800 0x55bf35f31b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45202/0 (socket says 172.21.15.16:45202) 2024-03-31T15:13:14.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.158+0000 7f9738a00640 1 -- 172.21.15.16:0/2555199209 learned_addr learned my addr 172.21.15.16:0/2555199209 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:13:14.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.158+0000 7f9739e00640 1 --2- 172.21.15.16:0/2555199209 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55bf360e8000 0x55bf35f31080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:14.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.158+0000 7f9738a00640 1 -- 172.21.15.16:0/2555199209 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55bf360e8400 msgr2=0x55bf35f31600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:13:14.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.158+0000 7f9738a00640 1 --2- 172.21.15.16:0/2555199209 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55bf360e8400 0x55bf35f31600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:14.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.158+0000 7f9738a00640 1 -- 172.21.15.16:0/2555199209 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55bf360e8000 msgr2=0x55bf35f31080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:13:14.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.158+0000 7f9738a00640 1 --2- 172.21.15.16:0/2555199209 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55bf360e8000 0x55bf35f31080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:14.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.158+0000 7f9738a00640 1 -- 172.21.15.16:0/2555199209 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bf35f2ba40 con 0x55bf360e8800 2024-03-31T15:13:14.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.158+0000 7f9738a00640 1 --2- 172.21.15.16:0/2555199209 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bf360e8800 0x55bf35f31b80 secure :-1 s=READY pgs=1514 cs=0 l=1 rev1=1 crypto rx=0x55bf360f4120 tx=0x55bf35f40300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:13:14.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.158+0000 7f9731c00640 1 -- 172.21.15.16:0/2555199209 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x55bf351a4b60 con 0x55bf360e8800 2024-03-31T15:13:14.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.158+0000 7f9731c00640 1 -- 172.21.15.16:0/2555199209 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x55bf351baa80 con 0x55bf360e8800 2024-03-31T15:13:14.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.158+0000 7f973be4a400 1 -- 172.21.15.16:0/2555199209 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55bf35f2bc20 con 0x55bf360e8800 2024-03-31T15:13:14.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.158+0000 7f9731c00640 1 -- 172.21.15.16:0/2555199209 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x55bf351a4340 con 0x55bf360e8800 2024-03-31T15:13:14.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.158+0000 7f973be4a400 1 -- 172.21.15.16:0/2555199209 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bf35f2af00 con 0x55bf360e8800 2024-03-31T15:13:14.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.158+0000 7f973be4a400 4 main: JournalTool: resolving pool 14 2024-03-31T15:13:14.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.162+0000 7f9731c00640 1 -- 172.21.15.16:0/2555199209 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x55bf360e2c00 con 0x55bf360e8800 2024-03-31T15:13:14.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.162+0000 7f9731c00640 1 --2- 172.21.15.16:0/2555199209 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x55bf35467000 0x55bf35f32100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:14.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.162+0000 7f9731c00640 1 -- 172.21.15.16:0/2555199209 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 5795+0+0 (secure 0 0 0) 0x55bf35452200 con 0x55bf360e8800 2024-03-31T15:13:14.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.162+0000 7f973be4a400 4 main: JournalTool: creating IoCtx.. 2024-03-31T15:13:14.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.162+0000 7f973be4a400 4 main: Executing for rank 0 2024-03-31T15:13:14.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.162+0000 7f973be4a400 4 Filtering by inode '1099511627777' 2024-03-31T15:13:14.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.162+0000 7f973be4a400 1 --2- 172.21.15.16:0/2555199209 >> [v2:172.21.15.89:6808/1796778630,v1:172.21.15.89:6811/1796778630] conn(0x55bf360e8c00 0x55bf35f32680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:14.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.162+0000 7f973be4a400 1 -- 172.21.15.16:0/2555199209 --> [v2:172.21.15.89:6808/1796778630,v1:172.21.15.89:6811/1796778630] -- osd_op(unknown.0.0:1 14.f 14:f1f69726:::400.00000000:head [read 0~2147483647] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e97) -- 0x55bf360e9000 con 0x55bf360e8c00 2024-03-31T15:13:14.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.162+0000 7f9739e00640 1 --2- 172.21.15.16:0/2555199209 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x55bf35467000 0x55bf35f32100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:14.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.162+0000 7f9739400640 1 --2- 172.21.15.16:0/2555199209 >> [v2:172.21.15.89:6808/1796778630,v1:172.21.15.89:6811/1796778630] conn(0x55bf360e8c00 0x55bf35f32680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:14.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.162+0000 7f9739e00640 1 --2- 172.21.15.16:0/2555199209 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x55bf35467000 0x55bf35f32100 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x55bf35f207e0 tx=0x55bf35454360 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:13:14.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.162+0000 7f9739400640 1 --2- 172.21.15.16:0/2555199209 >> [v2:172.21.15.89:6808/1796778630,v1:172.21.15.89:6811/1796778630] conn(0x55bf360e8c00 0x55bf35f32680 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:13:14.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.166+0000 7f9739400640 1 -- 172.21.15.16:0/2555199209 <== osd.7 v2:172.21.15.89:6808/1796778630 1 ==== osd_op_reply(1 400.00000000 [read 0~22 out=22b] v0'0 uv1 ondisk = 0) ==== 156+0+22 (crc 0 0 0) 0x55bf35eea000 con 0x55bf360e8c00 2024-03-31T15:13:14.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.166+0000 7f973be4a400 4 Pointer 400.00000000 is readable 2024-03-31T15:13:14.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.166+0000 7f973be4a400 4 JournalScanner::scan: reading header object '200.00000000' 2024-03-31T15:13:14.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.166+0000 7f973be4a400 1 --2- 172.21.15.16:0/2555199209 >> [v2:172.21.15.89:6800/1856844856,v1:172.21.15.89:6801/1856844856] conn(0x55bf360e9400 0x55bf35f32c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:14.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.166+0000 7f973be4a400 1 -- 172.21.15.16:0/2555199209 --> [v2:172.21.15.89:6800/1856844856,v1:172.21.15.89:6801/1856844856] -- osd_op(unknown.0.0:2 14.14 14:292cf221:::200.00000000:head [read 0~2147483647] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e97) -- 0x55bf35558000 con 0x55bf360e9400 2024-03-31T15:13:14.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.166+0000 7f9739e00640 1 --2- 172.21.15.16:0/2555199209 >> [v2:172.21.15.89:6800/1856844856,v1:172.21.15.89:6801/1856844856] conn(0x55bf360e9400 0x55bf35f32c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:14.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.166+0000 7f9739e00640 1 --2- 172.21.15.16:0/2555199209 >> [v2:172.21.15.89:6800/1856844856,v1:172.21.15.89:6801/1856844856] conn(0x55bf360e9400 0x55bf35f32c00 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:13:14.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.166+0000 7f9739e00640 1 -- 172.21.15.16:0/2555199209 <== osd.4 v2:172.21.15.89:6800/1856844856 1 ==== osd_op_reply(2 200.00000000 [read 0~90 out=90b] v0'0 uv30 ondisk = 0) ==== 156+0+90 (crc 0 0 0) 0x55bf35eea500 con 0x55bf360e9400 2024-03-31T15:13:14.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.166+0000 7f973be4a400 10 Header 0x400000 0x405f0e 0x40628a 2024-03-31T15:13:14.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.166+0000 7f973be4a400 10 Starting journal scan from offset 0x405f0e 2024-03-31T15:13:14.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.166+0000 7f973be4a400 1 -- 172.21.15.16:0/2555199209 --> [v2:172.21.15.89:6800/1856844856,v1:172.21.15.89:6801/1856844856] -- osd_op(unknown.0.0:3 14.14 14:2e2fa760:::200.00000001:head [read 24334~2147483647] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e97) -- 0x55bf35558400 con 0x55bf360e9400 2024-03-31T15:13:14.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.170+0000 7f9739e00640 1 -- 172.21.15.16:0/2555199209 <== osd.4 v2:172.21.15.89:6800/1856844856 2 ==== osd_op_reply(3 200.00000001 [read 24334~892 out=892b] v0'0 uv27 ondisk = 0) ==== 156+0+892 (crc 0 0 0) 0x55bf35eea500 con 0x55bf360e9400 2024-03-31T15:13:14.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.170+0000 7f973be4a400 4 Read 0x37c bytes from 200.00000001 gap=0 2024-03-31T15:13:14.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.170+0000 7f973be4a400 10 Parsing data, 0x37c bytes available 2024-03-31T15:13:14.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.170+0000 7f973be4a400 10 Attempting decode at 0x405f0e 2024-03-31T15:13:14.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.170+0000 7f973be4a400 10 Consumed 0x37c bytes 2024-03-31T15:13:14.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.170+0000 7f973be4a400 10 Valid entry at 0x405f0e 2024-03-31T15:13:14.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.170+0000 7f973be4a400 1 --2- 172.21.15.16:0/2555199209 >> [v2:172.21.15.16:6808/1398737533,v1:172.21.15.16:6810/1398737533] conn(0x55bf35558800 0x55bf35f33180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:14.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.170+0000 7f973be4a400 1 -- 172.21.15.16:0/2555199209 --> [v2:172.21.15.16:6808/1398737533,v1:172.21.15.16:6810/1398737533] -- osd_op(unknown.0.0:4 14.1b 14:ddb4e4d7:::200.00000002:head [read 0~2147483647] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e97) -- 0x55bf35558c00 con 0x55bf35558800 2024-03-31T15:13:14.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.170+0000 7f9739400640 1 --2- 172.21.15.16:0/2555199209 >> [v2:172.21.15.16:6808/1398737533,v1:172.21.15.16:6810/1398737533] conn(0x55bf35558800 0x55bf35f33180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:14.164 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.170+0000 7f9739400640 1 --2- 172.21.15.16:0/2555199209 >> [v2:172.21.15.16:6808/1398737533,v1:172.21.15.16:6810/1398737533] conn(0x55bf35558800 0x55bf35f33180 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:13:14.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.170+0000 7f9739400640 1 -- 172.21.15.16:0/2555199209 <== osd.1 v2:172.21.15.16:6808/1398737533 1 ==== osd_op_reply(4 200.00000002 [read 0~2147483647] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 156+0+0 (crc 0 0 0) 0x55bf35eea000 con 0x55bf35558800 2024-03-31T15:13:14.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.170+0000 7f973be4a400 4 Reached end of journal objects 2024-03-31T15:13:14.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.170+0000 7f973be4a400 4 Scanned objects, 0 missing, 1 valid 2024-03-31T15:13:14.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.170+0000 7f973be4a400 4 Events scanned, 0 gaps 2024-03-31T15:13:14.165 INFO:teuthology.orchestra.run.smithi016.stdout:Events by type: 2024-03-31T15:13:14.165 INFO:teuthology.orchestra.run.smithi016.stdout:Errors: 0 2024-03-31T15:13:14.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.170+0000 7f973be4a400 4 Found 1 valid events 2024-03-31T15:13:14.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.170+0000 7f973be4a400 4 Selected 0 events events for processing 2024-03-31T15:13:14.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.170+0000 7f973be4a400 4 0 events 2024-03-31T15:13:14.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.170+0000 7f973be4a400 1 -- 172.21.15.16:0/2555199209 >> [v2:172.21.15.16:6808/1398737533,v1:172.21.15.16:6810/1398737533] conn(0x55bf35558800 msgr2=0x55bf35f33180 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:13:14.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.170+0000 7f973be4a400 1 --2- 172.21.15.16:0/2555199209 >> [v2:172.21.15.16:6808/1398737533,v1:172.21.15.16:6810/1398737533] conn(0x55bf35558800 0x55bf35f33180 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:14.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.170+0000 7f973be4a400 1 -- 172.21.15.16:0/2555199209 >> [v2:172.21.15.89:6800/1856844856,v1:172.21.15.89:6801/1856844856] conn(0x55bf360e9400 msgr2=0x55bf35f32c00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:13:14.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.170+0000 7f973be4a400 1 --2- 172.21.15.16:0/2555199209 >> [v2:172.21.15.89:6800/1856844856,v1:172.21.15.89:6801/1856844856] conn(0x55bf360e9400 0x55bf35f32c00 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:14.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.170+0000 7f973be4a400 1 -- 172.21.15.16:0/2555199209 >> [v2:172.21.15.89:6808/1796778630,v1:172.21.15.89:6811/1796778630] conn(0x55bf360e8c00 msgr2=0x55bf35f32680 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:13:14.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.170+0000 7f973be4a400 1 --2- 172.21.15.16:0/2555199209 >> [v2:172.21.15.89:6808/1796778630,v1:172.21.15.89:6811/1796778630] conn(0x55bf360e8c00 0x55bf35f32680 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:14.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.170+0000 7f9739400640 1 -- 172.21.15.16:0/2555199209 reap_dead start 2024-03-31T15:13:14.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.170+0000 7f973be4a400 1 -- 172.21.15.16:0/2555199209 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x55bf35467000 msgr2=0x55bf35f32100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:13:14.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.170+0000 7f973be4a400 1 --2- 172.21.15.16:0/2555199209 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x55bf35467000 0x55bf35f32100 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x55bf35f207e0 tx=0x55bf35454360 comp rx=0 tx=0).stop 2024-03-31T15:13:14.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.170+0000 7f973be4a400 1 -- 172.21.15.16:0/2555199209 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bf360e8800 msgr2=0x55bf35f31b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:13:14.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.170+0000 7f973be4a400 1 --2- 172.21.15.16:0/2555199209 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bf360e8800 0x55bf35f31b80 secure :-1 s=READY pgs=1514 cs=0 l=1 rev1=1 crypto rx=0x55bf360f4120 tx=0x55bf35f40300 comp rx=0 tx=0).stop 2024-03-31T15:13:14.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.170+0000 7f973be4a400 1 -- 172.21.15.16:0/2555199209 shutdown_connections 2024-03-31T15:13:14.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.170+0000 7f973be4a400 1 --2- 172.21.15.16:0/2555199209 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x55bf35467000 0x55bf35f32100 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:14.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.170+0000 7f973be4a400 1 --2- 172.21.15.16:0/2555199209 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bf360e8800 0x55bf35f31b80 unknown :-1 s=CLOSED pgs=1514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:14.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.170+0000 7f973be4a400 1 -- 172.21.15.16:0/2555199209 >> 172.21.15.16:0/2555199209 conn(0x55bf3516ec00 msgr2=0x55bf3519b000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:13:14.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.170+0000 7f973be4a400 1 -- 172.21.15.16:0/2555199209 shutdown_connections 2024-03-31T15:13:14.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.170+0000 7f973be4a400 1 -- 172.21.15.16:0/2555199209 wait complete. 2024-03-31T15:13:14.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.174+0000 7f973be4a400 1 -- 172.21.15.16:0/703517262 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55bf3516f400 msgr2=0x55bf35f30580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:13:14.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.174+0000 7f973be4a400 1 --2- 172.21.15.16:0/703517262 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55bf3516f400 0x55bf35f30580 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x55bf35fd6990 tx=0x55bf35f40000 comp rx=0 tx=0).stop 2024-03-31T15:13:14.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.174+0000 7f973be4a400 1 -- 172.21.15.16:0/703517262 shutdown_connections 2024-03-31T15:13:14.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.174+0000 7f973be4a400 1 --2- 172.21.15.16:0/703517262 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55bf3516f000 0x55bf35f30b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:14.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.174+0000 7f973be4a400 1 --2- 172.21.15.16:0/703517262 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55bf3516f400 0x55bf35f30580 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:14.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.174+0000 7f973be4a400 1 --2- 172.21.15.16:0/703517262 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bf3516f800 0x55bf35f30000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:14.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.174+0000 7f973be4a400 1 -- 172.21.15.16:0/703517262 >> 172.21.15.16:0/703517262 conn(0x55bf3516fc00 msgr2=0x55bf3519b800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:13:14.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.174+0000 7f973be4a400 1 -- 172.21.15.16:0/703517262 shutdown_connections 2024-03-31T15:13:14.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.174+0000 7f973be4a400 1 -- 172.21.15.16:0/703517262 wait complete. 2024-03-31T15:13:14.171 DEBUG:tasks.cephfs.filesystem:Ran ['cephfs-journal-tool', '--debug-mds=20', '--debug-ms=1', '--debug-objecter=1', '--rank', 'cephfs:0', 'event', 'splice', '--inode=1099511627777', 'summary'] in time 0:00:00.073281, result: Events by type: Errors: 0 2024-03-31T15:13:14.171 DEBUG:teuthology.orchestra.run.smithi016:> cephfs-journal-tool --debug-mds=20 --debug-ms=1 --debug-objecter=1 --rank cephfs:0 event splice --inode=1099511627778 summary 2024-03-31T15:13:14.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.266+0000 7f88eeb70400 1 -- start start 2024-03-31T15:13:14.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.266+0000 7f88eeb70400 1 Processor -- start 2024-03-31T15:13:14.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.266+0000 7f88eeb70400 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5620fcb59800 0x5620fd916000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:14.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.266+0000 7f88eeb70400 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5620fcb59400 0x5620fd916580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:14.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.266+0000 7f88eeb70400 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5620fcb59000 0x5620fd916b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:14.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.266+0000 7f88eeb70400 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5620fd93c180 con 0x5620fcb59800 2024-03-31T15:13:14.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.266+0000 7f88eeb70400 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x5620fcbb8900 con 0x5620fcb59400 2024-03-31T15:13:14.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.266+0000 7f88eeb70400 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x5620fcbb8480 con 0x5620fcb59000 2024-03-31T15:13:14.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.266+0000 7f88ecc00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5620fcb59800 0x5620fd916000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:14.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.266+0000 7f88ecc00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5620fcb59800 0x5620fd916000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45222/0 (socket says 172.21.15.16:45222) 2024-03-31T15:13:14.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.266+0000 7f88ecc00640 1 -- 172.21.15.16:0/1540991513 learned_addr learned my addr 172.21.15.16:0/1540991513 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:13:14.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.266+0000 7f88ec200640 1 --2- 172.21.15.16:0/1540991513 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5620fcb59400 0x5620fd916580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:14.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.266+0000 7f88ed600640 1 --2- 172.21.15.16:0/1540991513 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5620fcb59000 0x5620fd916b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:14.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.270+0000 7f88ecc00640 1 -- 172.21.15.16:0/1540991513 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5620fcb59000 msgr2=0x5620fd916b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:13:14.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.270+0000 7f88ecc00640 1 --2- 172.21.15.16:0/1540991513 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5620fcb59000 0x5620fd916b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:14.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.270+0000 7f88ecc00640 1 -- 172.21.15.16:0/1540991513 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5620fcb59400 msgr2=0x5620fd916580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:13:14.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.270+0000 7f88ecc00640 1 --2- 172.21.15.16:0/1540991513 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5620fcb59400 0x5620fd916580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:14.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.270+0000 7f88ecc00640 1 -- 172.21.15.16:0/1540991513 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5620fd914b40 con 0x5620fcb59800 2024-03-31T15:13:14.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.270+0000 7f88ec200640 1 --2- 172.21.15.16:0/1540991513 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5620fcb59400 0x5620fd916580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:13:14.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.270+0000 7f88ecc00640 1 --2- 172.21.15.16:0/1540991513 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5620fcb59800 0x5620fd916000 secure :-1 s=READY pgs=1516 cs=0 l=1 rev1=1 crypto rx=0x5620fd9c4a20 tx=0x5620fd922000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=efda033ce4b0eecc server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:13:14.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.270+0000 7f88e9a00640 1 -- 172.21.15.16:0/1540991513 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x5620fcb8e9c0 con 0x5620fcb59800 2024-03-31T15:13:14.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.270+0000 7f88e9a00640 1 -- 172.21.15.16:0/1540991513 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x5620fcba41c0 con 0x5620fcb59800 2024-03-31T15:13:14.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.270+0000 7f88e9a00640 1 -- 172.21.15.16:0/1540991513 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x5620fcb8e4e0 con 0x5620fcb59800 2024-03-31T15:13:14.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.270+0000 7f88eeb70400 1 -- 172.21.15.16:0/1540991513 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5620fd914d20 con 0x5620fcb59800 2024-03-31T15:13:14.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.270+0000 7f88eeb70400 1 -- 172.21.15.16:0/1540991513 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({fsmap=0}) -- 0x5620fd914f00 con 0x5620fcb59800 2024-03-31T15:13:14.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.270+0000 7f88eeb70400 4 waiting for MDS map... 2024-03-31T15:13:14.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.270+0000 7f88e9a00640 1 -- 172.21.15.16:0/1540991513 <== mon.0 v2:172.21.15.16:3300/0 4 ==== osd_map(97..97 src has 1..97) ==== 5795+0+0 (secure 0 0 0) 0x5620fda7c200 con 0x5620fcb59800 2024-03-31T15:13:14.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.270+0000 7f88e9a00640 1 -- 172.21.15.16:0/1540991513 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 84) ==== 2885+0+0 (secure 0 0 0) 0x5620fcbaf180 con 0x5620fcb59800 2024-03-31T15:13:14.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.270+0000 7f88eeb70400 4 Got MDS map 84 2024-03-31T15:13:14.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.270+0000 7f88eeb70400 10 main: JournalTool::main 2024-03-31T15:13:14.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.270+0000 7f88eeb70400 4 main: JournalTool: connecting to RADOS... 2024-03-31T15:13:14.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.270+0000 7f88eeb70400 1 Processor -- start 2024-03-31T15:13:14.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.270+0000 7f88eeb70400 1 -- start start 2024-03-31T15:13:14.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.270+0000 7f88eeb70400 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5620fd93fc00 0x5620fd917080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:14.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.270+0000 7f88eeb70400 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5620fdada000 0x5620fd917600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:14.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.270+0000 7f88eeb70400 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5620fdada400 0x5620fd917b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:14.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.270+0000 7f88eeb70400 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5620fcbb8180 con 0x5620fdada000 2024-03-31T15:13:14.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.270+0000 7f88eeb70400 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x5620fd93cd80 con 0x5620fd93fc00 2024-03-31T15:13:14.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.270+0000 7f88eeb70400 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x5620fd93cf00 con 0x5620fdada400 2024-03-31T15:13:14.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.270+0000 7f88ecc00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5620fdada000 0x5620fd917600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:14.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.270+0000 7f88ecc00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5620fdada000 0x5620fd917600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45236/0 (socket says 172.21.15.16:45236) 2024-03-31T15:13:14.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.270+0000 7f88ecc00640 1 -- 172.21.15.16:0/362851920 learned_addr learned my addr 172.21.15.16:0/362851920 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:13:14.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.270+0000 7f88ed600640 1 --2- 172.21.15.16:0/362851920 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5620fd93fc00 0x5620fd917080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:14.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.270+0000 7f88ec200640 1 --2- 172.21.15.16:0/362851920 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5620fdada400 0x5620fd917b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:14.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.270+0000 7f88ecc00640 1 -- 172.21.15.16:0/362851920 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5620fdada400 msgr2=0x5620fd917b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:13:14.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.270+0000 7f88ecc00640 1 --2- 172.21.15.16:0/362851920 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5620fdada400 0x5620fd917b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:14.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.270+0000 7f88ecc00640 1 -- 172.21.15.16:0/362851920 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5620fd93fc00 msgr2=0x5620fd917080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:13:14.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.270+0000 7f88ecc00640 1 --2- 172.21.15.16:0/362851920 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5620fd93fc00 0x5620fd917080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:14.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.270+0000 7f88ecc00640 1 -- 172.21.15.16:0/362851920 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5620fd9141e0 con 0x5620fdada000 2024-03-31T15:13:14.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.270+0000 7f88ec200640 1 --2- 172.21.15.16:0/362851920 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5620fdada400 0x5620fd917b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:13:14.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.270+0000 7f88ecc00640 1 --2- 172.21.15.16:0/362851920 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5620fdada000 0x5620fd917600 secure :-1 s=READY pgs=1517 cs=0 l=1 rev1=1 crypto rx=0x5620fd9c5a70 tx=0x5620fd9224e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d72cb5d293fab38d server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:13:14.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.270+0000 7f88e7200640 1 -- 172.21.15.16:0/362851920 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x5620fcb8eb60 con 0x5620fdada000 2024-03-31T15:13:14.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.270+0000 7f88e7200640 1 -- 172.21.15.16:0/362851920 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x5620fcba4e00 con 0x5620fdada000 2024-03-31T15:13:14.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.270+0000 7f88e7200640 1 -- 172.21.15.16:0/362851920 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x5620fcb8e340 con 0x5620fdada000 2024-03-31T15:13:14.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.274+0000 7f88eeb70400 1 -- 172.21.15.16:0/362851920 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5620fdada000 msgr2=0x5620fd917600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:13:14.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.274+0000 7f88eeb70400 1 --2- 172.21.15.16:0/362851920 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5620fdada000 0x5620fd917600 secure :-1 s=READY pgs=1517 cs=0 l=1 rev1=1 crypto rx=0x5620fd9c5a70 tx=0x5620fd9224e0 comp rx=0 tx=0).stop 2024-03-31T15:13:14.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.274+0000 7f88eeb70400 1 -- 172.21.15.16:0/362851920 shutdown_connections 2024-03-31T15:13:14.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.274+0000 7f88eeb70400 1 --2- 172.21.15.16:0/362851920 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5620fdada400 0x5620fd917b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:14.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.274+0000 7f88eeb70400 1 --2- 172.21.15.16:0/362851920 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5620fdada000 0x5620fd917600 unknown :-1 s=CLOSED pgs=1517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:14.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.274+0000 7f88eeb70400 1 --2- 172.21.15.16:0/362851920 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5620fd93fc00 0x5620fd917080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:14.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.274+0000 7f88eeb70400 1 -- 172.21.15.16:0/362851920 >> 172.21.15.16:0/362851920 conn(0x5620fcb58c00 msgr2=0x5620fcb87800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:13:14.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.274+0000 7f88eeb70400 1 -- 172.21.15.16:0/362851920 shutdown_connections 2024-03-31T15:13:14.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.274+0000 7f88eeb70400 1 -- 172.21.15.16:0/362851920 wait complete. 2024-03-31T15:13:14.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.274+0000 7f88eeb70400 1 Processor -- start 2024-03-31T15:13:14.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.274+0000 7f88eeb70400 1 -- start start 2024-03-31T15:13:14.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.274+0000 7f88eeb70400 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5620fd93fc00 0x5620fd917080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:14.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.274+0000 7f88eeb70400 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5620fdadb800 0x5620fd917600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:14.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.274+0000 7f88eeb70400 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5620fdadb400 0x5620fd917b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:14.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.274+0000 7f88eeb70400 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5620fd93d200 con 0x5620fdadb800 2024-03-31T15:13:14.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.274+0000 7f88eeb70400 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x5620fd93d080 con 0x5620fd93fc00 2024-03-31T15:13:14.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.274+0000 7f88eeb70400 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x5620fd93d380 con 0x5620fdadb400 2024-03-31T15:13:14.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.274+0000 7f88ecc00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5620fdadb800 0x5620fd917600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:14.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.274+0000 7f88ecc00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5620fdadb800 0x5620fd917600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45246/0 (socket says 172.21.15.16:45246) 2024-03-31T15:13:14.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.274+0000 7f88ecc00640 1 -- 172.21.15.16:0/2978676845 learned_addr learned my addr 172.21.15.16:0/2978676845 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:13:14.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.274+0000 7f88ec200640 1 --2- 172.21.15.16:0/2978676845 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5620fdadb400 0x5620fd917b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:14.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.274+0000 7f88ed600640 1 --2- 172.21.15.16:0/2978676845 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5620fd93fc00 0x5620fd917080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:14.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.274+0000 7f88ecc00640 1 -- 172.21.15.16:0/2978676845 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5620fdadb400 msgr2=0x5620fd917b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:13:14.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.274+0000 7f88ecc00640 1 --2- 172.21.15.16:0/2978676845 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5620fdadb400 0x5620fd917b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:14.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.274+0000 7f88ecc00640 1 -- 172.21.15.16:0/2978676845 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5620fd93fc00 msgr2=0x5620fd917080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:13:14.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.274+0000 7f88ecc00640 1 --2- 172.21.15.16:0/2978676845 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5620fd93fc00 0x5620fd917080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:14.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.274+0000 7f88ecc00640 1 -- 172.21.15.16:0/2978676845 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5620fd915a40 con 0x5620fdadb800 2024-03-31T15:13:14.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.274+0000 7f88ecc00640 1 --2- 172.21.15.16:0/2978676845 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5620fdadb800 0x5620fd917600 secure :-1 s=READY pgs=1518 cs=0 l=1 rev1=1 crypto rx=0x5620fd935260 tx=0x5620fd922480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:13:14.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.274+0000 7f88e5400640 1 -- 172.21.15.16:0/2978676845 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x5620fcb8eb60 con 0x5620fdadb800 2024-03-31T15:13:14.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.274+0000 7f88e5400640 1 -- 172.21.15.16:0/2978676845 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x5620fcba4540 con 0x5620fdadb800 2024-03-31T15:13:14.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.274+0000 7f88eeb70400 1 -- 172.21.15.16:0/2978676845 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x5620fd915c20 con 0x5620fdadb800 2024-03-31T15:13:14.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.274+0000 7f88e5400640 1 -- 172.21.15.16:0/2978676845 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x5620fcb8e340 con 0x5620fdadb800 2024-03-31T15:13:14.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.274+0000 7f88eeb70400 1 -- 172.21.15.16:0/2978676845 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5620fd914f00 con 0x5620fdadb800 2024-03-31T15:13:14.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.274+0000 7f88eeb70400 4 main: JournalTool: resolving pool 14 2024-03-31T15:13:14.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.278+0000 7f88e5400640 1 -- 172.21.15.16:0/2978676845 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x5620fdad8c00 con 0x5620fdadb800 2024-03-31T15:13:14.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.278+0000 7f88e5400640 1 --2- 172.21.15.16:0/2978676845 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x5620fce4d400 0x5620fd918100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:14.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.278+0000 7f88e5400640 1 -- 172.21.15.16:0/2978676845 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 5795+0+0 (secure 0 0 0) 0x5620fce56600 con 0x5620fdadb800 2024-03-31T15:13:14.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.278+0000 7f88eeb70400 4 main: JournalTool: creating IoCtx.. 2024-03-31T15:13:14.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.278+0000 7f88eeb70400 4 main: Executing for rank 0 2024-03-31T15:13:14.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.278+0000 7f88eeb70400 4 Filtering by inode '1099511627778' 2024-03-31T15:13:14.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.278+0000 7f88eeb70400 1 --2- 172.21.15.16:0/2978676845 >> [v2:172.21.15.89:6808/1796778630,v1:172.21.15.89:6811/1796778630] conn(0x5620fdadb000 0x5620fd918680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:14.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.278+0000 7f88eeb70400 1 -- 172.21.15.16:0/2978676845 --> [v2:172.21.15.89:6808/1796778630,v1:172.21.15.89:6811/1796778630] -- osd_op(unknown.0.0:1 14.f 14:f1f69726:::400.00000000:head [read 0~2147483647] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e97) -- 0x5620fdadac00 con 0x5620fdadb000 2024-03-31T15:13:14.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.278+0000 7f88ed600640 1 --2- 172.21.15.16:0/2978676845 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x5620fce4d400 0x5620fd918100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:14.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.278+0000 7f88ec200640 1 --2- 172.21.15.16:0/2978676845 >> [v2:172.21.15.89:6808/1796778630,v1:172.21.15.89:6811/1796778630] conn(0x5620fdadb000 0x5620fd918680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:14.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.278+0000 7f88ed600640 1 --2- 172.21.15.16:0/2978676845 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x5620fce4d400 0x5620fd918100 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x5620fd90a7e0 tx=0x5620fcea0180 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:13:14.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.278+0000 7f88ec200640 1 --2- 172.21.15.16:0/2978676845 >> [v2:172.21.15.89:6808/1796778630,v1:172.21.15.89:6811/1796778630] conn(0x5620fdadb000 0x5620fd918680 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:13:14.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.278+0000 7f88ec200640 1 -- 172.21.15.16:0/2978676845 <== osd.7 v2:172.21.15.89:6808/1796778630 1 ==== osd_op_reply(1 400.00000000 [read 0~22 out=22b] v0'0 uv1 ondisk = 0) ==== 156+0+22 (crc 0 0 0) 0x5620fd8d4000 con 0x5620fdadb000 2024-03-31T15:13:14.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.278+0000 7f88eeb70400 4 Pointer 400.00000000 is readable 2024-03-31T15:13:14.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.278+0000 7f88eeb70400 4 JournalScanner::scan: reading header object '200.00000000' 2024-03-31T15:13:14.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.278+0000 7f88eeb70400 1 --2- 172.21.15.16:0/2978676845 >> [v2:172.21.15.89:6800/1856844856,v1:172.21.15.89:6801/1856844856] conn(0x5620fdada800 0x5620fd918c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:14.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.278+0000 7f88eeb70400 1 -- 172.21.15.16:0/2978676845 --> [v2:172.21.15.89:6800/1856844856,v1:172.21.15.89:6801/1856844856] -- osd_op(unknown.0.0:2 14.14 14:292cf221:::200.00000000:head [read 0~2147483647] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e97) -- 0x5620fdada400 con 0x5620fdada800 2024-03-31T15:13:14.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.278+0000 7f88ed600640 1 --2- 172.21.15.16:0/2978676845 >> [v2:172.21.15.89:6800/1856844856,v1:172.21.15.89:6801/1856844856] conn(0x5620fdada800 0x5620fd918c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:14.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.278+0000 7f88ed600640 1 --2- 172.21.15.16:0/2978676845 >> [v2:172.21.15.89:6800/1856844856,v1:172.21.15.89:6801/1856844856] conn(0x5620fdada800 0x5620fd918c00 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:13:14.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.282+0000 7f88ed600640 1 -- 172.21.15.16:0/2978676845 <== osd.4 v2:172.21.15.89:6800/1856844856 1 ==== osd_op_reply(2 200.00000000 [read 0~90 out=90b] v0'0 uv30 ondisk = 0) ==== 156+0+90 (crc 0 0 0) 0x5620fd8d4500 con 0x5620fdada800 2024-03-31T15:13:14.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.282+0000 7f88eeb70400 10 Header 0x400000 0x405f0e 0x40628a 2024-03-31T15:13:14.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.282+0000 7f88eeb70400 10 Starting journal scan from offset 0x405f0e 2024-03-31T15:13:14.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.282+0000 7f88eeb70400 1 -- 172.21.15.16:0/2978676845 --> [v2:172.21.15.89:6800/1856844856,v1:172.21.15.89:6801/1856844856] -- osd_op(unknown.0.0:3 14.14 14:2e2fa760:::200.00000001:head [read 24334~2147483647] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e97) -- 0x5620fdada000 con 0x5620fdada800 2024-03-31T15:13:14.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.282+0000 7f88ed600640 1 -- 172.21.15.16:0/2978676845 <== osd.4 v2:172.21.15.89:6800/1856844856 2 ==== osd_op_reply(3 200.00000001 [read 24334~892 out=892b] v0'0 uv27 ondisk = 0) ==== 156+0+892 (crc 0 0 0) 0x5620fd8d4500 con 0x5620fdada800 2024-03-31T15:13:14.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.282+0000 7f88eeb70400 4 Read 0x37c bytes from 200.00000001 gap=0 2024-03-31T15:13:14.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.282+0000 7f88eeb70400 10 Parsing data, 0x37c bytes available 2024-03-31T15:13:14.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.282+0000 7f88eeb70400 10 Attempting decode at 0x405f0e 2024-03-31T15:13:14.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.282+0000 7f88eeb70400 10 Consumed 0x37c bytes 2024-03-31T15:13:14.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.282+0000 7f88eeb70400 10 Valid entry at 0x405f0e 2024-03-31T15:13:14.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.282+0000 7f88eeb70400 1 --2- 172.21.15.16:0/2978676845 >> [v2:172.21.15.16:6808/1398737533,v1:172.21.15.16:6810/1398737533] conn(0x5620fcf46000 0x5620fd919180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:14.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.282+0000 7f88eeb70400 1 -- 172.21.15.16:0/2978676845 --> [v2:172.21.15.16:6808/1398737533,v1:172.21.15.16:6810/1398737533] -- osd_op(unknown.0.0:4 14.1b 14:ddb4e4d7:::200.00000002:head [read 0~2147483647] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e97) -- 0x5620fcf46400 con 0x5620fcf46000 2024-03-31T15:13:14.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.282+0000 7f88ecc00640 1 --2- 172.21.15.16:0/2978676845 >> [v2:172.21.15.16:6808/1398737533,v1:172.21.15.16:6810/1398737533] conn(0x5620fcf46000 0x5620fd919180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:14.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.282+0000 7f88ecc00640 1 --2- 172.21.15.16:0/2978676845 >> [v2:172.21.15.16:6808/1398737533,v1:172.21.15.16:6810/1398737533] conn(0x5620fcf46000 0x5620fd919180 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:13:14.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.282+0000 7f88ecc00640 1 -- 172.21.15.16:0/2978676845 <== osd.1 v2:172.21.15.16:6808/1398737533 1 ==== osd_op_reply(4 200.00000002 [read 0~2147483647] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 156+0+0 (crc 0 0 0) 0x5620fd8d4780 con 0x5620fcf46000 2024-03-31T15:13:14.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.282+0000 7f88eeb70400 4 Reached end of journal objects 2024-03-31T15:13:14.279 INFO:teuthology.orchestra.run.smithi016.stdout:Events by type: 2024-03-31T15:13:14.279 INFO:teuthology.orchestra.run.smithi016.stdout:Errors: 0 2024-03-31T15:13:14.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.282+0000 7f88eeb70400 4 Scanned objects, 0 missing, 1 valid 2024-03-31T15:13:14.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.282+0000 7f88eeb70400 4 Events scanned, 0 gaps 2024-03-31T15:13:14.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.282+0000 7f88eeb70400 4 Found 1 valid events 2024-03-31T15:13:14.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.282+0000 7f88eeb70400 4 Selected 0 events events for processing 2024-03-31T15:13:14.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.282+0000 7f88eeb70400 4 0 events 2024-03-31T15:13:14.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.282+0000 7f88eeb70400 1 -- 172.21.15.16:0/2978676845 >> [v2:172.21.15.16:6808/1398737533,v1:172.21.15.16:6810/1398737533] conn(0x5620fcf46000 msgr2=0x5620fd919180 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:13:14.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.282+0000 7f88eeb70400 1 --2- 172.21.15.16:0/2978676845 >> [v2:172.21.15.16:6808/1398737533,v1:172.21.15.16:6810/1398737533] conn(0x5620fcf46000 0x5620fd919180 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:14.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.282+0000 7f88eeb70400 1 -- 172.21.15.16:0/2978676845 >> [v2:172.21.15.89:6800/1856844856,v1:172.21.15.89:6801/1856844856] conn(0x5620fdada800 msgr2=0x5620fd918c00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:13:14.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.282+0000 7f88eeb70400 1 --2- 172.21.15.16:0/2978676845 >> [v2:172.21.15.89:6800/1856844856,v1:172.21.15.89:6801/1856844856] conn(0x5620fdada800 0x5620fd918c00 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:14.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.282+0000 7f88eeb70400 1 -- 172.21.15.16:0/2978676845 >> [v2:172.21.15.89:6808/1796778630,v1:172.21.15.89:6811/1796778630] conn(0x5620fdadb000 msgr2=0x5620fd918680 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:13:14.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.282+0000 7f88eeb70400 1 --2- 172.21.15.16:0/2978676845 >> [v2:172.21.15.89:6808/1796778630,v1:172.21.15.89:6811/1796778630] conn(0x5620fdadb000 0x5620fd918680 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:14.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.282+0000 7f88ec200640 1 -- 172.21.15.16:0/2978676845 reap_dead start 2024-03-31T15:13:14.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.282+0000 7f88eeb70400 1 -- 172.21.15.16:0/2978676845 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x5620fce4d400 msgr2=0x5620fd918100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:13:14.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.282+0000 7f88eeb70400 1 --2- 172.21.15.16:0/2978676845 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x5620fce4d400 0x5620fd918100 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x5620fd90a7e0 tx=0x5620fcea0180 comp rx=0 tx=0).stop 2024-03-31T15:13:14.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.282+0000 7f88eeb70400 1 -- 172.21.15.16:0/2978676845 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5620fdadb800 msgr2=0x5620fd917600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:13:14.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.282+0000 7f88eeb70400 1 --2- 172.21.15.16:0/2978676845 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5620fdadb800 0x5620fd917600 secure :-1 s=READY pgs=1518 cs=0 l=1 rev1=1 crypto rx=0x5620fd935260 tx=0x5620fd922480 comp rx=0 tx=0).stop 2024-03-31T15:13:14.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.282+0000 7f88eeb70400 1 -- 172.21.15.16:0/2978676845 shutdown_connections 2024-03-31T15:13:14.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.282+0000 7f88eeb70400 1 --2- 172.21.15.16:0/2978676845 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x5620fce4d400 0x5620fd918100 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:14.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.282+0000 7f88eeb70400 1 --2- 172.21.15.16:0/2978676845 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5620fdadb800 0x5620fd917600 unknown :-1 s=CLOSED pgs=1518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:14.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.282+0000 7f88eeb70400 1 -- 172.21.15.16:0/2978676845 >> 172.21.15.16:0/2978676845 conn(0x5620fcb58c00 msgr2=0x5620fcb87800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:13:14.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.282+0000 7f88eeb70400 1 -- 172.21.15.16:0/2978676845 shutdown_connections 2024-03-31T15:13:14.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.282+0000 7f88eeb70400 1 -- 172.21.15.16:0/2978676845 wait complete. 2024-03-31T15:13:14.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.282+0000 7f88eeb70400 1 -- 172.21.15.16:0/1540991513 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5620fcb59800 msgr2=0x5620fd916000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:13:14.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.282+0000 7f88eeb70400 1 --2- 172.21.15.16:0/1540991513 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5620fcb59800 0x5620fd916000 secure :-1 s=READY pgs=1516 cs=0 l=1 rev1=1 crypto rx=0x5620fd9c4a20 tx=0x5620fd922000 comp rx=0 tx=0).stop 2024-03-31T15:13:14.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.282+0000 7f88eeb70400 1 -- 172.21.15.16:0/1540991513 shutdown_connections 2024-03-31T15:13:14.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.282+0000 7f88eeb70400 1 --2- 172.21.15.16:0/1540991513 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5620fcb59000 0x5620fd916b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:14.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.282+0000 7f88eeb70400 1 --2- 172.21.15.16:0/1540991513 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5620fcb59400 0x5620fd916580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:14.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.282+0000 7f88eeb70400 1 --2- 172.21.15.16:0/1540991513 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5620fcb59800 0x5620fd916000 unknown :-1 s=CLOSED pgs=1516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:14.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.282+0000 7f88eeb70400 1 -- 172.21.15.16:0/1540991513 >> 172.21.15.16:0/1540991513 conn(0x5620fcb59c00 msgr2=0x5620fcb87000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:13:14.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.282+0000 7f88eeb70400 1 -- 172.21.15.16:0/1540991513 shutdown_connections 2024-03-31T15:13:14.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.282+0000 7f88eeb70400 1 -- 172.21.15.16:0/1540991513 wait complete. 2024-03-31T15:13:14.284 DEBUG:tasks.cephfs.filesystem:Ran ['cephfs-journal-tool', '--debug-mds=20', '--debug-ms=1', '--debug-objecter=1', '--rank', 'cephfs:0', 'event', 'splice', '--inode=1099511627778', 'summary'] in time 0:00:00.112650, result: Events by type: Errors: 0 2024-03-31T15:13:14.284 DEBUG:teuthology.orchestra.run.smithi016:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata put mds0_inotable - 2024-03-31T15:13:14.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.338+0000 7f32f4bc4080 1 Processor -- start 2024-03-31T15:13:14.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.338+0000 7f32f4bc4080 1 -- start start 2024-03-31T15:13:14.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.338+0000 7f32f4bc4080 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55df49881260 0x55df4990fd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:14.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.338+0000 7f32f4bc4080 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55df498869f0 0x55df499a4f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:14.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.338+0000 7f32f4bc4080 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55df49912ab0 0x55df49911400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:14.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.338+0000 7f32f4bc4080 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55df497e14b0 con 0x55df49881260 2024-03-31T15:13:14.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.338+0000 7f32f4bc4080 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55df497e5500 con 0x55df498869f0 2024-03-31T15:13:14.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.338+0000 7f32f4bc4080 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x55df4990d400 con 0x55df49912ab0 2024-03-31T15:13:14.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.338+0000 7f32f1e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55df49881260 0x55df4990fd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:14.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.338+0000 7f32f1e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55df49881260 0x55df4990fd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45254/0 (socket says 172.21.15.16:45254) 2024-03-31T15:13:14.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.338+0000 7f32f1e00640 1 -- 172.21.15.16:0/608983295 learned_addr learned my addr 172.21.15.16:0/608983295 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:13:14.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.338+0000 7f32f2800640 1 --2- 172.21.15.16:0/608983295 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55df49912ab0 0x55df49911400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:14.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.338+0000 7f32f1400640 1 --2- 172.21.15.16:0/608983295 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55df498869f0 0x55df499a4f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:14.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.338+0000 7f32f1e00640 1 -- 172.21.15.16:0/608983295 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55df49912ab0 msgr2=0x55df49911400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:13:14.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.338+0000 7f32f1e00640 1 --2- 172.21.15.16:0/608983295 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55df49912ab0 0x55df49911400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:14.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.338+0000 7f32f1e00640 1 -- 172.21.15.16:0/608983295 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55df498869f0 msgr2=0x55df499a4f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:13:14.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.338+0000 7f32f1e00640 1 --2- 172.21.15.16:0/608983295 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55df498869f0 0x55df499a4f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:14.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.338+0000 7f32f1e00640 1 -- 172.21.15.16:0/608983295 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55df498870d0 con 0x55df49881260 2024-03-31T15:13:14.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.338+0000 7f32f1400640 1 --2- 172.21.15.16:0/608983295 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55df498869f0 0x55df499a4f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:13:14.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.338+0000 7f32f1e00640 1 --2- 172.21.15.16:0/608983295 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55df49881260 0x55df4990fd70 secure :-1 s=READY pgs=1520 cs=0 l=1 rev1=1 crypto rx=0x7f32e400b220 tx=0x7f32e400b6e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=61bfb313da0e6aa5 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:13:14.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.338+0000 7f32ebe00640 1 -- 172.21.15.16:0/608983295 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32e400a040 con 0x55df49881260 2024-03-31T15:13:14.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.338+0000 7f32ebe00640 1 -- 172.21.15.16:0/608983295 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f32e400f620 con 0x55df49881260 2024-03-31T15:13:14.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.338+0000 7f32ebe00640 1 -- 172.21.15.16:0/608983295 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32e400f9f0 con 0x55df49881260 2024-03-31T15:13:14.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.338+0000 7f32f4bc4080 1 -- 172.21.15.16:0/608983295 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55df49881260 msgr2=0x55df4990fd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:13:14.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.338+0000 7f32f4bc4080 1 --2- 172.21.15.16:0/608983295 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55df49881260 0x55df4990fd70 secure :-1 s=READY pgs=1520 cs=0 l=1 rev1=1 crypto rx=0x7f32e400b220 tx=0x7f32e400b6e0 comp rx=0 tx=0).stop 2024-03-31T15:13:14.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.338+0000 7f32f4bc4080 1 -- 172.21.15.16:0/608983295 shutdown_connections 2024-03-31T15:13:14.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.338+0000 7f32f4bc4080 1 --2- 172.21.15.16:0/608983295 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55df49912ab0 0x55df49911400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:14.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.338+0000 7f32f4bc4080 1 --2- 172.21.15.16:0/608983295 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55df498869f0 0x55df499a4f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:14.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.338+0000 7f32f4bc4080 1 --2- 172.21.15.16:0/608983295 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55df49881260 0x55df4990fd70 unknown :-1 s=CLOSED pgs=1520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:14.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.338+0000 7f32f4bc4080 1 -- 172.21.15.16:0/608983295 >> 172.21.15.16:0/608983295 conn(0x55df498840b0 msgr2=0x55df49884490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:13:14.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.338+0000 7f32f4bc4080 1 -- 172.21.15.16:0/608983295 shutdown_connections 2024-03-31T15:13:14.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.338+0000 7f32f4bc4080 1 -- 172.21.15.16:0/608983295 wait complete. 2024-03-31T15:13:14.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.342+0000 7f32f4bc4080 1 Processor -- start 2024-03-31T15:13:14.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.342+0000 7f32f4bc4080 1 -- start start 2024-03-31T15:13:14.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.342+0000 7f32f4bc4080 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55df49881260 0x55df4990e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:14.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.342+0000 7f32f4bc4080 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55df498869f0 0x55df4990bd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:14.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.342+0000 7f32f4bc4080 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55df49912ab0 0x55df4990c250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:14.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.342+0000 7f32f4bc4080 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55df49910780 con 0x55df49881260 2024-03-31T15:13:14.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.342+0000 7f32f4bc4080 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55df499104e0 con 0x55df498869f0 2024-03-31T15:13:14.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.342+0000 7f32f4bc4080 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x55df49910a20 con 0x55df49912ab0 2024-03-31T15:13:14.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.342+0000 7f32f1e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55df49881260 0x55df4990e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:14.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.342+0000 7f32f1e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55df49881260 0x55df4990e7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45264/0 (socket says 172.21.15.16:45264) 2024-03-31T15:13:14.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.342+0000 7f32f1e00640 1 -- 172.21.15.16:0/1064696921 learned_addr learned my addr 172.21.15.16:0/1064696921 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:13:14.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.342+0000 7f32f1400640 1 --2- 172.21.15.16:0/1064696921 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55df498869f0 0x55df4990bd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:14.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.342+0000 7f32f2800640 1 --2- 172.21.15.16:0/1064696921 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55df49912ab0 0x55df4990c250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:14.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.342+0000 7f32f1e00640 1 -- 172.21.15.16:0/1064696921 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55df49912ab0 msgr2=0x55df4990c250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:13:14.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.342+0000 7f32f1e00640 1 --2- 172.21.15.16:0/1064696921 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55df49912ab0 0x55df4990c250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:14.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.342+0000 7f32f1e00640 1 -- 172.21.15.16:0/1064696921 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55df498869f0 msgr2=0x55df4990bd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:13:14.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.342+0000 7f32f1e00640 1 --2- 172.21.15.16:0/1064696921 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55df498869f0 0x55df4990bd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:14.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.342+0000 7f32f1e00640 1 -- 172.21.15.16:0/1064696921 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55df4983c080 con 0x55df49881260 2024-03-31T15:13:14.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.342+0000 7f32f1400640 1 --2- 172.21.15.16:0/1064696921 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55df498869f0 0x55df4990bd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:13:14.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.342+0000 7f32f1e00640 1 --2- 172.21.15.16:0/1064696921 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55df49881260 0x55df4990e7c0 secure :-1 s=READY pgs=1521 cs=0 l=1 rev1=1 crypto rx=0x7f32e400ff70 tx=0x7f32e4000bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:13:14.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.342+0000 7f32eaa00640 1 -- 172.21.15.16:0/1064696921 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32e401c650 con 0x55df49881260 2024-03-31T15:13:14.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.342+0000 7f32eaa00640 1 -- 172.21.15.16:0/1064696921 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f32e4021020 con 0x55df49881260 2024-03-31T15:13:14.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.342+0000 7f32eaa00640 1 -- 172.21.15.16:0/1064696921 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32e401c810 con 0x55df49881260 2024-03-31T15:13:14.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.342+0000 7f32f4bc4080 1 -- 172.21.15.16:0/1064696921 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55df4983c320 con 0x55df49881260 2024-03-31T15:13:14.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.342+0000 7f32f4bc4080 1 -- 172.21.15.16:0/1064696921 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55df4983c860 con 0x55df49881260 2024-03-31T15:13:14.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.346+0000 7f32eaa00640 1 -- 172.21.15.16:0/1064696921 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f32e4017070 con 0x55df49881260 2024-03-31T15:13:14.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.346+0000 7f32eaa00640 1 --2- 172.21.15.16:0/1064696921 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f32d4066d60 0x7f32d4069220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:14.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.346+0000 7f32eaa00640 1 -- 172.21.15.16:0/1064696921 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 5795+0+0 (secure 0 0 0) 0x7f32e4086ff0 con 0x55df49881260 2024-03-31T15:13:14.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.346+0000 7f32f4bc4080 1 --2- 172.21.15.16:0/1064696921 >> [v2:172.21.15.16:6819/3018526646,v1:172.21.15.16:6822/3018526646] conn(0x55df4990ed00 0x55df4983d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:14.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.346+0000 7f32f4bc4080 1 -- 172.21.15.16:0/1064696921 --> [v2:172.21.15.16:6819/3018526646,v1:172.21.15.16:6822/3018526646] -- osd_op(unknown.0.0:1 14.13 14:c91d4a1d:::mds0_inotable:head [writefull 0~34 in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x55df4983d920 con 0x55df4990ed00 2024-03-31T15:13:14.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.346+0000 7f32f1400640 1 --2- 172.21.15.16:0/1064696921 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f32d4066d60 0x7f32d4069220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:14.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.346+0000 7f32f2800640 1 --2- 172.21.15.16:0/1064696921 >> [v2:172.21.15.16:6819/3018526646,v1:172.21.15.16:6822/3018526646] conn(0x55df4990ed00 0x55df4983d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:14.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.346+0000 7f32f2800640 1 --2- 172.21.15.16:0/1064696921 >> [v2:172.21.15.16:6819/3018526646,v1:172.21.15.16:6822/3018526646] conn(0x55df4990ed00 0x55df4983d3e0 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:13:14.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.346+0000 7f32f1400640 1 --2- 172.21.15.16:0/1064696921 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f32d4066d60 0x7f32d4069220 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f32e0000c10 tx=0x7f32e0001790 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:13:14.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.350+0000 7f32f2800640 1 -- 172.21.15.16:0/1064696921 <== osd.2 v2:172.21.15.16:6819/3018526646 1 ==== osd_op_reply(1 mds0_inotable [writefull 0~34] v97'5 uv5 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f32ec0a8070 con 0x55df4990ed00 2024-03-31T15:13:14.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.350+0000 7f32f4bc4080 1 -- 172.21.15.16:0/1064696921 >> [v2:172.21.15.16:6819/3018526646,v1:172.21.15.16:6822/3018526646] conn(0x55df4990ed00 msgr2=0x55df4983d3e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:13:14.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.350+0000 7f32f4bc4080 1 --2- 172.21.15.16:0/1064696921 >> [v2:172.21.15.16:6819/3018526646,v1:172.21.15.16:6822/3018526646] conn(0x55df4990ed00 0x55df4983d3e0 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:14.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.350+0000 7f32f4bc4080 1 -- 172.21.15.16:0/1064696921 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f32d4066d60 msgr2=0x7f32d4069220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:13:14.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.350+0000 7f32f4bc4080 1 --2- 172.21.15.16:0/1064696921 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f32d4066d60 0x7f32d4069220 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f32e0000c10 tx=0x7f32e0001790 comp rx=0 tx=0).stop 2024-03-31T15:13:14.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.350+0000 7f32f4bc4080 1 -- 172.21.15.16:0/1064696921 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55df49881260 msgr2=0x55df4990e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:13:14.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.350+0000 7f32f4bc4080 1 --2- 172.21.15.16:0/1064696921 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55df49881260 0x55df4990e7c0 secure :-1 s=READY pgs=1521 cs=0 l=1 rev1=1 crypto rx=0x7f32e400ff70 tx=0x7f32e4000bd0 comp rx=0 tx=0).stop 2024-03-31T15:13:14.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.350+0000 7f32f2800640 1 -- 172.21.15.16:0/1064696921 reap_dead start 2024-03-31T15:13:14.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.350+0000 7f32f4bc4080 1 -- 172.21.15.16:0/1064696921 shutdown_connections 2024-03-31T15:13:14.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.350+0000 7f32f4bc4080 1 -- 172.21.15.16:0/1064696921 >> 172.21.15.16:0/1064696921 conn(0x55df498840b0 msgr2=0x55df49883150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:13:14.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.350+0000 7f32f4bc4080 1 -- 172.21.15.16:0/1064696921 shutdown_connections 2024-03-31T15:13:14.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.350+0000 7f32f4bc4080 1 -- 172.21.15.16:0/1064696921 wait complete. 2024-03-31T15:13:14.347 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs joinable true 2024-03-31T15:13:14.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.538+0000 7f5d79c00640 1 Processor -- start 2024-03-31T15:13:14.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.538+0000 7f5d79c00640 1 -- start start 2024-03-31T15:13:14.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.542+0000 7f5d79c00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f5d741411b0 0x7f5d7406f920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:14.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.542+0000 7f5d79c00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f5d74144340 0x7f5d7406fe60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:14.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.542+0000 7f5d79c00640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5d74144980 con 0x7f5d74141590 2024-03-31T15:13:14.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.542+0000 7f5d79c00640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f5d7415f910 con 0x7f5d741411b0 2024-03-31T15:13:14.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.542+0000 7f5d79c00640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f5d74162910 con 0x7f5d74144340 2024-03-31T15:13:14.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.542+0000 7f5d72a00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f5d741411b0 0x7f5d7406f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:14.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.542+0000 7f5d72a00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f5d741411b0 0x7f5d7406f920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45272/0 (socket says 172.21.15.16:45272) 2024-03-31T15:13:14.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.542+0000 7f5d72a00640 1 -- 172.21.15.16:0/1525732973 learned_addr learned my addr 172.21.15.16:0/1525732973 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:13:14.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.542+0000 7f5d73400640 1 --2- 172.21.15.16:0/1525732973 >> v2:172.21.15.89:3300/0 conn(0x7f5d74144340 0x7f5d7406fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:14.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.542+0000 7f5d72a00640 1 -- 172.21.15.16:0/1525732973 >> v2:172.21.15.89:3300/0 conn(0x7f5d74144340 msgr2=0x7f5d7406fe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:13:14.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.542+0000 7f5d72a00640 1 --2- 172.21.15.16:0/1525732973 >> v2:172.21.15.89:3300/0 conn(0x7f5d74144340 0x7f5d7406fe60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:14.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.542+0000 7f5d72a00640 1 -- 172.21.15.16:0/1525732973 >> v1:172.21.15.89:6789/0 conn(0x7f5d74141590 legacy=0x7f5d74143c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:13:14.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.542+0000 7f5d72a00640 1 -- 172.21.15.16:0/1525732973 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d74161fe0 con 0x7f5d741411b0 2024-03-31T15:13:14.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.546+0000 7f5d72a00640 1 --2- 172.21.15.16:0/1525732973 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5d741411b0 0x7f5d7406f920 secure :-1 s=READY pgs=1522 cs=0 l=1 rev1=1 crypto rx=0x7f5d740b6c30 tx=0x7f5d6401c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a84c80f14c1b527c server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:13:14.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.546+0000 7f5d72000640 1 -- 172.21.15.16:0/1525732973 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d640ab610 con 0x7f5d741411b0 2024-03-31T15:13:14.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.546+0000 7f5d72000640 1 -- 172.21.15.16:0/1525732973 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5d640b0070 con 0x7f5d741411b0 2024-03-31T15:13:14.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.546+0000 7f5d72000640 1 -- 172.21.15.16:0/1525732973 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d640ab9d0 con 0x7f5d741411b0 2024-03-31T15:13:14.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.546+0000 7f5d79c00640 1 -- 172.21.15.16:0/1525732973 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5d741411b0 msgr2=0x7f5d7406f920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:13:14.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.546+0000 7f5d79c00640 1 --2- 172.21.15.16:0/1525732973 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5d741411b0 0x7f5d7406f920 secure :-1 s=READY pgs=1522 cs=0 l=1 rev1=1 crypto rx=0x7f5d740b6c30 tx=0x7f5d6401c520 comp rx=0 tx=0).stop 2024-03-31T15:13:14.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.546+0000 7f5d79c00640 1 -- 172.21.15.16:0/1525732973 shutdown_connections 2024-03-31T15:13:14.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.546+0000 7f5d79c00640 1 --2- 172.21.15.16:0/1525732973 >> v2:172.21.15.89:3300/0 conn(0x7f5d74144340 0x7f5d7406fe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:14.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.546+0000 7f5d79c00640 1 --2- 172.21.15.16:0/1525732973 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5d741411b0 0x7f5d7406f920 unknown :-1 s=CLOSED pgs=1522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:14.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.546+0000 7f5d79c00640 1 -- 172.21.15.16:0/1525732973 >> 172.21.15.16:0/1525732973 conn(0x7f5d74067930 msgr2=0x7f5d740b3ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:13:14.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.546+0000 7f5d79c00640 1 -- 172.21.15.16:0/1525732973 shutdown_connections 2024-03-31T15:13:14.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.546+0000 7f5d79c00640 1 -- 172.21.15.16:0/1525732973 wait complete. 2024-03-31T15:13:14.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.546+0000 7f5d79c00640 1 Processor -- start 2024-03-31T15:13:14.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.546+0000 7f5d79c00640 1 -- start start 2024-03-31T15:13:14.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.550+0000 7f5d79c00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f5d741411b0 0x7f5d7413c5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:14.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.550+0000 7f5d79c00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5d74141590 0x7f5d7406c730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:14.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.550+0000 7f5d79c00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5d74144340 0x7f5d7413d6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:14.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.550+0000 7f5d79c00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f5d7415f290 con 0x7f5d74141590 2024-03-31T15:13:14.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.550+0000 7f5d79c00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f5d7415f110 con 0x7f5d74144340 2024-03-31T15:13:14.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.550+0000 7f5d79c00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f5d7415f410 con 0x7f5d741411b0 2024-03-31T15:13:14.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.550+0000 7f5d6a000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5d74141590 0x7f5d7406c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:14.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.550+0000 7f5d6a000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5d74141590 0x7f5d7406c730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45276/0 (socket says 172.21.15.16:45276) 2024-03-31T15:13:14.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.550+0000 7f5d6a000640 1 -- 172.21.15.16:0/2916312168 learned_addr learned my addr 172.21.15.16:0/2916312168 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:13:14.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.550+0000 7f5d72a00640 1 --2- 172.21.15.16:0/2916312168 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f5d741411b0 0x7f5d7413c5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:14.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.550+0000 7f5d73400640 1 --2- 172.21.15.16:0/2916312168 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5d74144340 0x7f5d7413d6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:14.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.550+0000 7f5d6a000640 1 -- 172.21.15.16:0/2916312168 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f5d741411b0 msgr2=0x7f5d7413c5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:13:14.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.550+0000 7f5d6a000640 1 --2- 172.21.15.16:0/2916312168 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f5d741411b0 0x7f5d7413c5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:14.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.550+0000 7f5d6a000640 1 -- 172.21.15.16:0/2916312168 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5d74144340 msgr2=0x7f5d7413d6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:13:14.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.550+0000 7f5d6a000640 1 --2- 172.21.15.16:0/2916312168 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5d74144340 0x7f5d7413d6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:14.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.550+0000 7f5d6a000640 1 -- 172.21.15.16:0/2916312168 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d740b1d80 con 0x7f5d74141590 2024-03-31T15:13:14.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.550+0000 7f5d6a000640 1 --2- 172.21.15.16:0/2916312168 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5d74141590 0x7f5d7406c730 secure :-1 s=READY pgs=1523 cs=0 l=1 rev1=1 crypto rx=0x7f5d5c00c7a0 tx=0x7f5d5c00cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:13:14.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.550+0000 7f5d6b400640 1 -- 172.21.15.16:0/2916312168 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d5c016020 con 0x7f5d74141590 2024-03-31T15:13:14.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.550+0000 7f5d6b400640 1 -- 172.21.15.16:0/2916312168 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5d5c006d80 con 0x7f5d74141590 2024-03-31T15:13:14.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.550+0000 7f5d6b400640 1 -- 172.21.15.16:0/2916312168 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d5c003bc0 con 0x7f5d74141590 2024-03-31T15:13:14.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.550+0000 7f5d79c00640 1 -- 172.21.15.16:0/2916312168 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5d740b2060 con 0x7f5d74141590 2024-03-31T15:13:14.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.550+0000 7f5d79c00640 1 -- 172.21.15.16:0/2916312168 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5d740b2620 con 0x7f5d74141590 2024-03-31T15:13:14.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.550+0000 7f5d79c00640 1 -- 172.21.15.16:0/2916312168 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5d74069950 con 0x7f5d74141590 2024-03-31T15:13:14.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.554+0000 7f5d6b400640 1 -- 172.21.15.16:0/2916312168 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f5d5c00a850 con 0x7f5d74141590 2024-03-31T15:13:14.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.554+0000 7f5d6b400640 1 --2- 172.21.15.16:0/2916312168 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f5d50068380 0x7f5d5006a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:14.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.554+0000 7f5d6b400640 1 -- 172.21.15.16:0/2916312168 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 5795+0+0 (secure 0 0 0) 0x7f5d5c0866f0 con 0x7f5d74141590 2024-03-31T15:13:14.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.554+0000 7f5d72a00640 1 --2- 172.21.15.16:0/2916312168 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f5d50068380 0x7f5d5006a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:14.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.554+0000 7f5d72a00640 1 --2- 172.21.15.16:0/2916312168 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f5d50068380 0x7f5d5006a840 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f5d640af000 tx=0x7f5d640a7000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:13:14.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.562+0000 7f5d6b400640 1 -- 172.21.15.16:0/2916312168 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f5d5c020020 con 0x7f5d74141590 2024-03-31T15:13:14.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:14.762+0000 7f5d79c00640 1 -- 172.21.15.16:0/2916312168 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"} v 0) -- 0x7f5d740b3ea0 con 0x7f5d74141590 2024-03-31T15:13:15.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:15.078+0000 7f5d6b400640 1 -- 172.21.15.16:0/2916312168 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]=0 cephfs marked joinable; MDS may join as newly active. v85) ==== 162+0+0 (secure 0 0 0) 0x7f5d5c056c40 con 0x7f5d74141590 2024-03-31T15:13:15.072 INFO:teuthology.orchestra.run.smithi016.stderr:cephfs marked joinable; MDS may join as newly active. 2024-03-31T15:13:15.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:15.082+0000 7f5d79c00640 1 -- 172.21.15.16:0/2916312168 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f5d50068380 msgr2=0x7f5d5006a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:13:15.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:15.082+0000 7f5d79c00640 1 --2- 172.21.15.16:0/2916312168 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f5d50068380 0x7f5d5006a840 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f5d640af000 tx=0x7f5d640a7000 comp rx=0 tx=0).stop 2024-03-31T15:13:15.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:15.082+0000 7f5d79c00640 1 -- 172.21.15.16:0/2916312168 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5d74141590 msgr2=0x7f5d7406c730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:13:15.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:15.082+0000 7f5d79c00640 1 --2- 172.21.15.16:0/2916312168 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5d74141590 0x7f5d7406c730 secure :-1 s=READY pgs=1523 cs=0 l=1 rev1=1 crypto rx=0x7f5d5c00c7a0 tx=0x7f5d5c00cc60 comp rx=0 tx=0).stop 2024-03-31T15:13:15.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:15.082+0000 7f5d79c00640 1 -- 172.21.15.16:0/2916312168 shutdown_connections 2024-03-31T15:13:15.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:15.082+0000 7f5d79c00640 1 --2- 172.21.15.16:0/2916312168 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f5d50068380 0x7f5d5006a840 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:15.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:15.082+0000 7f5d79c00640 1 --2- 172.21.15.16:0/2916312168 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5d74144340 0x7f5d7413d6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:15.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:15.082+0000 7f5d79c00640 1 --2- 172.21.15.16:0/2916312168 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5d74141590 0x7f5d7406c730 unknown :-1 s=CLOSED pgs=1523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:15.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:15.082+0000 7f5d79c00640 1 --2- 172.21.15.16:0/2916312168 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f5d741411b0 0x7f5d7413c5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:15.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:15.082+0000 7f5d79c00640 1 -- 172.21.15.16:0/2916312168 >> 172.21.15.16:0/2916312168 conn(0x7f5d74067930 msgr2=0x7f5d74151fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:13:15.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:15.082+0000 7f5d79c00640 1 -- 172.21.15.16:0/2916312168 shutdown_connections 2024-03-31T15:13:15.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:15.082+0000 7f5d79c00640 1 -- 172.21.15.16:0/2916312168 wait complete. 2024-03-31T15:13:15.098 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-31T15:13:15.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:15.246+0000 7fb202c00640 1 Processor -- start 2024-03-31T15:13:15.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:15.246+0000 7fb202c00640 1 -- start start 2024-03-31T15:13:15.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:15.246+0000 7fb202c00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fb1fc147280 0x7fb1fc06c730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:15.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:15.246+0000 7fb202c00640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb1fc147660 con 0x7fb1fc147dd0 2024-03-31T15:13:15.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:15.246+0000 7fb202c00640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb1fc162340 con 0x7fb1fc070030 2024-03-31T15:13:15.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:15.246+0000 7fb202c00640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7fb1fc1620f0 con 0x7fb1fc147280 2024-03-31T15:13:15.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:15.246+0000 7fb1f3e00640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fb1fc147dd0 0x7fb1fc06f920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:34376/0 (socket says 172.21.15.16:34376) 2024-03-31T15:13:15.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:15.246+0000 7fb1f3e00640 1 -- 172.21.15.16:0/3101726983 learned_addr learned my addr 172.21.15.16:0/3101726983 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:13:15.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:15.246+0000 7fb1fbe00640 1 --2- 172.21.15.16:0/3101726983 >> v2:172.21.15.89:3300/0 conn(0x7fb1fc147280 0x7fb1fc06c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:15.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:15.246+0000 7fb1fb400640 1 -- 172.21.15.16:0/3101726983 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 832143631 0 0) 0x7fb1fc147660 con 0x7fb1fc147dd0 2024-03-31T15:13:15.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:15.246+0000 7fb1fb400640 1 -- 172.21.15.16:0/3101726983 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb1e0003530 con 0x7fb1fc147dd0 2024-03-31T15:13:15.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:15.246+0000 7fb1fbe00640 1 -- 172.21.15.16:0/3101726983 >> v1:172.21.15.89:6789/0 conn(0x7fb1fc070030 legacy=0x7fb1fc0cc8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:13:15.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:15.246+0000 7fb1fb400640 1 -- 172.21.15.16:0/3101726983 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1344910156 0 0) 0x7fb1e0003530 con 0x7fb1fc147dd0 2024-03-31T15:13:15.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:15.246+0000 7fb1fbe00640 1 -- 172.21.15.16:0/3101726983 >> v1:172.21.15.16:6789/0 conn(0x7fb1fc147dd0 legacy=0x7fb1fc06f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:13:15.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:15.246+0000 7fb1fbe00640 1 -- 172.21.15.16:0/3101726983 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1fc162620 con 0x7fb1fc147280 2024-03-31T15:13:15.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:15.250+0000 7fb1fbe00640 1 --2- 172.21.15.16:0/3101726983 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb1fc147280 0x7fb1fc06c730 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7fb1fc162270 tx=0x7fb1ec01c3c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5dfc508448247ace server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:13:15.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:15.250+0000 7fb1fb400640 1 -- 172.21.15.16:0/3101726983 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1ec0b0070 con 0x7fb1fc147280 2024-03-31T15:13:15.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:15.250+0000 7fb1fb400640 1 -- 172.21.15.16:0/3101726983 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb1ec0ab400 con 0x7fb1fc147280 2024-03-31T15:13:15.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:15.250+0000 7fb1fb400640 1 -- 172.21.15.16:0/3101726983 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1e4001680 con 0x7fb1fc147280 2024-03-31T15:13:15.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:15.250+0000 7fb202c00640 1 -- 172.21.15.16:0/3101726983 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb1fc147280 msgr2=0x7fb1fc06c730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:13:15.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:15.250+0000 7fb202c00640 1 --2- 172.21.15.16:0/3101726983 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb1fc147280 0x7fb1fc06c730 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7fb1fc162270 tx=0x7fb1ec01c3c0 comp rx=0 tx=0).stop 2024-03-31T15:13:15.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:15.250+0000 7fb202c00640 1 -- 172.21.15.16:0/3101726983 shutdown_connections 2024-03-31T15:13:15.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:15.250+0000 7fb202c00640 1 --2- 172.21.15.16:0/3101726983 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb1fc147280 0x7fb1fc06c730 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:15.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:15.250+0000 7fb202c00640 1 -- 172.21.15.16:0/3101726983 >> 172.21.15.16:0/3101726983 conn(0x7fb1fc067330 msgr2=0x7fb1fc067740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:13:15.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:15.250+0000 7fb202c00640 1 -- 172.21.15.16:0/3101726983 shutdown_connections 2024-03-31T15:13:15.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:15.250+0000 7fb202c00640 1 -- 172.21.15.16:0/3101726983 wait complete. 2024-03-31T15:13:15.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:15.254+0000 7fb202c00640 1 Processor -- start 2024-03-31T15:13:15.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:15.254+0000 7fb202c00640 1 -- start start 2024-03-31T15:13:15.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:15.254+0000 7fb202c00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb1fc070030 0x7fb1fc0b5cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:15.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:15.254+0000 7fb202c00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb1fc147280 0x7fb1fc1454d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:15.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:15.254+0000 7fb202c00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb1fc147dd0 0x7fb1fc142bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:15.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:15.254+0000 7fb202c00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fb1fc0cdbe0 con 0x7fb1fc147dd0 2024-03-31T15:13:15.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:15.254+0000 7fb202c00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fb1fc0cda60 con 0x7fb1fc070030 2024-03-31T15:13:15.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:15.254+0000 7fb202c00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fb1fc16c4a0 con 0x7fb1fc147280 2024-03-31T15:13:15.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:15.254+0000 7fb200c00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb1fc147dd0 0x7fb1fc142bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:15.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:15.254+0000 7fb200c00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb1fc147dd0 0x7fb1fc142bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45282/0 (socket says 172.21.15.16:45282) 2024-03-31T15:13:15.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:15.254+0000 7fb200c00640 1 -- 172.21.15.16:0/2756251216 learned_addr learned my addr 172.21.15.16:0/2756251216 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:13:15.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:15.254+0000 7fb1f3e00640 1 --2- 172.21.15.16:0/2756251216 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb1fc147280 0x7fb1fc1454d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:15.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:15.254+0000 7fb1fbe00640 1 --2- 172.21.15.16:0/2756251216 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb1fc070030 0x7fb1fc0b5cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:15.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:15.254+0000 7fb200c00640 1 -- 172.21.15.16:0/2756251216 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb1fc147280 msgr2=0x7fb1fc1454d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:13:15.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:15.254+0000 7fb200c00640 1 --2- 172.21.15.16:0/2756251216 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb1fc147280 0x7fb1fc1454d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:15.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:15.254+0000 7fb200c00640 1 -- 172.21.15.16:0/2756251216 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb1fc070030 msgr2=0x7fb1fc0b5cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:13:15.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:15.254+0000 7fb200c00640 1 --2- 172.21.15.16:0/2756251216 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb1fc070030 0x7fb1fc0b5cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:15.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:15.254+0000 7fb200c00640 1 -- 172.21.15.16:0/2756251216 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1fc145a10 con 0x7fb1fc147dd0 2024-03-31T15:13:15.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:15.254+0000 7fb1f3e00640 1 --2- 172.21.15.16:0/2756251216 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb1fc147280 0x7fb1fc1454d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:13:15.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:15.254+0000 7fb200c00640 1 --2- 172.21.15.16:0/2756251216 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb1fc147dd0 0x7fb1fc142bf0 secure :-1 s=READY pgs=1525 cs=0 l=1 rev1=1 crypto rx=0x7fb1f400c730 tx=0x7fb1f400cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:13:15.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:15.254+0000 7fb1f8c00640 1 -- 172.21.15.16:0/2756251216 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1f4003b80 con 0x7fb1fc147dd0 2024-03-31T15:13:15.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:15.254+0000 7fb1f8c00640 1 -- 172.21.15.16:0/2756251216 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb1f4003d20 con 0x7fb1fc147dd0 2024-03-31T15:13:15.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:15.254+0000 7fb1f8c00640 1 -- 172.21.15.16:0/2756251216 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1f4005310 con 0x7fb1fc147dd0 2024-03-31T15:13:15.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:15.254+0000 7fb202c00640 1 -- 172.21.15.16:0/2756251216 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb1fc145cb0 con 0x7fb1fc147dd0 2024-03-31T15:13:15.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:15.254+0000 7fb202c00640 1 -- 172.21.15.16:0/2756251216 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb1fc146270 con 0x7fb1fc147dd0 2024-03-31T15:13:15.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:15.254+0000 7fb202c00640 1 -- 172.21.15.16:0/2756251216 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb1fc069950 con 0x7fb1fc147dd0 2024-03-31T15:13:15.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:15.258+0000 7fb1f8c00640 1 -- 172.21.15.16:0/2756251216 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fb1f40056c0 con 0x7fb1fc147dd0 2024-03-31T15:13:15.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:15.258+0000 7fb1f8c00640 1 --2- 172.21.15.16:0/2756251216 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fb1d8068380 0x7fb1d806a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:15.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:15.258+0000 7fb1f8c00640 1 -- 172.21.15.16:0/2756251216 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 5795+0+0 (secure 0 0 0) 0x7fb1f4086a90 con 0x7fb1fc147dd0 2024-03-31T15:13:15.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:15.258+0000 7fb1fbe00640 1 --2- 172.21.15.16:0/2756251216 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fb1d8068380 0x7fb1d806a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:15.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:15.258+0000 7fb1fbe00640 1 --2- 172.21.15.16:0/2756251216 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fb1d8068380 0x7fb1d806a840 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7fb1ec0ad000 tx=0x7fb1ec0a6000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:13:15.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:15.262+0000 7fb1f8c00640 1 -- 172.21.15.16:0/2756251216 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fb1f4056f60 con 0x7fb1fc147dd0 2024-03-31T15:13:15.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:15.458+0000 7fb202c00640 1 -- 172.21.15.16:0/2756251216 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb1fc0679d0 con 0x7fb1fc147dd0 2024-03-31T15:13:15.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:15.462+0000 7fb1f8c00640 1 -- 172.21.15.16:0/2756251216 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 86 v86) ==== 94+0+5174 (secure 0 0 0) 0x7fb1f4056860 con 0x7fb1fc147dd0 2024-03-31T15:13:15.454 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:13:15.454 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":86,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7139,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6836/3798928623","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":3798928623},{"type":"v1","addr":"172.21.15.89:6836","nonce":3798928623}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":78},{"gid":7142,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/3074283817","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6835","nonce":3074283817},{"type":"v1","addr":"172.21.15.89:6837","nonce":3074283817}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":78},{"gid":8071,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6834/2110189998","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":2110189998},{"type":"v1","addr":"172.21.15.16:6834","nonce":2110189998}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":78}],"filesystems":[{"mdsmap":{"epoch":86,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-31T15:12:16.294600+0000","modified":"2024-03-31T15:13:15.085748+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":97,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8874},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8874":{"gid":8874,"name":"a","rank":0,"incarnation":86,"state":"up:replay","state_seq":1,"addr":"172.21.15.16:6835/567808781","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6833","nonce":567808781},{"type":"v1","addr":"172.21.15.16:6835","nonce":567808781}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":7}]} 2024-03-31T15:13:15.454 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 86 2024-03-31T15:13:15.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:15.462+0000 7fb202c00640 1 -- 172.21.15.16:0/2756251216 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fb1d8068380 msgr2=0x7fb1d806a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:13:15.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:15.462+0000 7fb202c00640 1 --2- 172.21.15.16:0/2756251216 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fb1d8068380 0x7fb1d806a840 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7fb1ec0ad000 tx=0x7fb1ec0a6000 comp rx=0 tx=0).stop 2024-03-31T15:13:15.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:15.462+0000 7fb202c00640 1 -- 172.21.15.16:0/2756251216 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb1fc147dd0 msgr2=0x7fb1fc142bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:13:15.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:15.462+0000 7fb202c00640 1 --2- 172.21.15.16:0/2756251216 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb1fc147dd0 0x7fb1fc142bf0 secure :-1 s=READY pgs=1525 cs=0 l=1 rev1=1 crypto rx=0x7fb1f400c730 tx=0x7fb1f400cbf0 comp rx=0 tx=0).stop 2024-03-31T15:13:15.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:15.462+0000 7fb202c00640 1 -- 172.21.15.16:0/2756251216 shutdown_connections 2024-03-31T15:13:15.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:15.462+0000 7fb202c00640 1 --2- 172.21.15.16:0/2756251216 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fb1d8068380 0x7fb1d806a840 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:15.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:15.462+0000 7fb202c00640 1 --2- 172.21.15.16:0/2756251216 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb1fc147dd0 0x7fb1fc142bf0 unknown :-1 s=CLOSED pgs=1525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:15.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:15.462+0000 7fb202c00640 1 --2- 172.21.15.16:0/2756251216 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb1fc147280 0x7fb1fc1454d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:15.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:15.462+0000 7fb202c00640 1 --2- 172.21.15.16:0/2756251216 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb1fc070030 0x7fb1fc0b5cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:15.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:15.462+0000 7fb202c00640 1 -- 172.21.15.16:0/2756251216 >> 172.21.15.16:0/2756251216 conn(0x7fb1fc067330 msgr2=0x7fb1fc0d1050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:13:15.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:15.462+0000 7fb202c00640 1 -- 172.21.15.16:0/2756251216 shutdown_connections 2024-03-31T15:13:15.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:15.462+0000 7fb202c00640 1 -- 172.21.15.16:0/2756251216 wait complete. 2024-03-31T15:13:15.878 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 86, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-31T15:12:16.294600+0000', 'modified': '2024-03-31T15:13:15.085748+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 97, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8874}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8874': {'gid': 8874, 'name': 'a', 'rank': 0, 'incarnation': 86, 'state': 'up:replay', 'state_seq': 1, 'addr': '172.21.15.16:6835/567808781', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6833', 'nonce': 567808781}, {'type': 'v1', 'addr': '172.21.15.16:6835', 'nonce': 567808781}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [15], 'metadata_pool': 14, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-03-31T15:13:15.878 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8874:up:replay 2024-03-31T15:13:16.879 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-31T15:13:17.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:17.038+0000 7f4956200640 1 Processor -- start 2024-03-31T15:13:17.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:17.038+0000 7f4956200640 1 -- start start 2024-03-31T15:13:17.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:17.038+0000 7f4956200640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f4950145d10 0x7f49500cf790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:17.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:17.038+0000 7f4956200640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f49500728b0 0x7f49500b1cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:17.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:17.038+0000 7f4956200640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f49500b2200 0x7f495015cd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:17.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:17.038+0000 7f4956200640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f4950160e70 con 0x7f49500b2200 2024-03-31T15:13:17.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:17.038+0000 7f4956200640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f49500cfcd0 con 0x7f49500728b0 2024-03-31T15:13:17.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:17.038+0000 7f4956200640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f49501602f0 con 0x7f4950145d10 2024-03-31T15:13:17.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:17.038+0000 7f494fe00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f49500b2200 0x7f495015cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:17.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:17.038+0000 7f494fe00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f49500b2200 0x7f495015cd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35368/0 (socket says 172.21.15.16:35368) 2024-03-31T15:13:17.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:17.038+0000 7f494fe00640 1 -- 172.21.15.16:0/914491381 learned_addr learned my addr 172.21.15.16:0/914491381 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:13:17.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:17.038+0000 7f494ea00640 1 --2- 172.21.15.16:0/914491381 >> v2:172.21.15.89:3300/0 conn(0x7f49500728b0 0x7f49500b1cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:17.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:17.038+0000 7f494f400640 1 --2- 172.21.15.16:0/914491381 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f4950145d10 0x7f49500cf790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:17.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:17.038+0000 7f494fe00640 1 -- 172.21.15.16:0/914491381 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f4950145d10 msgr2=0x7f49500cf790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:13:17.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:17.038+0000 7f494fe00640 1 --2- 172.21.15.16:0/914491381 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f4950145d10 0x7f49500cf790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:17.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:17.038+0000 7f494fe00640 1 -- 172.21.15.16:0/914491381 >> v2:172.21.15.89:3300/0 conn(0x7f49500728b0 msgr2=0x7f49500b1cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:13:17.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:17.038+0000 7f494fe00640 1 --2- 172.21.15.16:0/914491381 >> v2:172.21.15.89:3300/0 conn(0x7f49500728b0 0x7f49500b1cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:17.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:17.038+0000 7f494fe00640 1 -- 172.21.15.16:0/914491381 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49501605d0 con 0x7f49500b2200 2024-03-31T15:13:17.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:17.038+0000 7f494f400640 1 --2- 172.21.15.16:0/914491381 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f4950145d10 0x7f49500cf790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:13:17.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:17.038+0000 7f494ea00640 1 --2- 172.21.15.16:0/914491381 >> v2:172.21.15.89:3300/0 conn(0x7f49500728b0 0x7f49500b1cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:13:17.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:17.042+0000 7f494fe00640 1 --2- 172.21.15.16:0/914491381 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f49500b2200 0x7f495015cd50 secure :-1 s=READY pgs=1526 cs=0 l=1 rev1=1 crypto rx=0x7f4950069160 tx=0x7f494801c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5c9b3e943624c40 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:13:17.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:17.042+0000 7f494e000640 1 -- 172.21.15.16:0/914491381 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49480a5670 con 0x7f49500b2200 2024-03-31T15:13:17.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:17.042+0000 7f494e000640 1 -- 172.21.15.16:0/914491381 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f49480a5810 con 0x7f49500b2200 2024-03-31T15:13:17.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:17.042+0000 7f494e000640 1 -- 172.21.15.16:0/914491381 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49480a5be0 con 0x7f49500b2200 2024-03-31T15:13:17.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:17.042+0000 7f4956200640 1 -- 172.21.15.16:0/914491381 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f49500b2200 msgr2=0x7f495015cd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:13:17.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:17.042+0000 7f4956200640 1 --2- 172.21.15.16:0/914491381 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f49500b2200 0x7f495015cd50 secure :-1 s=READY pgs=1526 cs=0 l=1 rev1=1 crypto rx=0x7f4950069160 tx=0x7f494801c9f0 comp rx=0 tx=0).stop 2024-03-31T15:13:17.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:17.042+0000 7f4956200640 1 -- 172.21.15.16:0/914491381 shutdown_connections 2024-03-31T15:13:17.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:17.042+0000 7f4956200640 1 --2- 172.21.15.16:0/914491381 >> v2:172.21.15.89:3300/0 conn(0x7f49500728b0 0x7f49500b1cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:17.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:17.042+0000 7f4956200640 1 --2- 172.21.15.16:0/914491381 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f49500b2200 0x7f495015cd50 unknown :-1 s=CLOSED pgs=1526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:17.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:17.042+0000 7f4956200640 1 --2- 172.21.15.16:0/914491381 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f4950145d10 0x7f49500cf790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:17.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:17.042+0000 7f4956200640 1 -- 172.21.15.16:0/914491381 >> 172.21.15.16:0/914491381 conn(0x7f495006b620 msgr2=0x7f495006ba30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:13:17.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:17.042+0000 7f4956200640 1 -- 172.21.15.16:0/914491381 shutdown_connections 2024-03-31T15:13:17.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:17.042+0000 7f4956200640 1 -- 172.21.15.16:0/914491381 wait complete. 2024-03-31T15:13:17.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:17.046+0000 7f4956200640 1 Processor -- start 2024-03-31T15:13:17.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:17.046+0000 7f4956200640 1 -- start start 2024-03-31T15:13:17.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:17.046+0000 7f4956200640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f49500728b0 0x7f49500b1c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:17.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:17.046+0000 7f4956200640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f49500b2200 0x7f49501f73d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:17.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:17.046+0000 7f4956200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4950145d10 0x7f495013fd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:17.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:17.046+0000 7f4956200640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f49500cf4d0 con 0x7f4950145d10 2024-03-31T15:13:17.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:17.046+0000 7f4956200640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f49500cf350 con 0x7f49500b2200 2024-03-31T15:13:17.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:17.046+0000 7f4956200640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f4950072c90 con 0x7f49500728b0 2024-03-31T15:13:17.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:17.046+0000 7f494fe00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4950145d10 0x7f495013fd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:17.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:17.046+0000 7f494fe00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4950145d10 0x7f495013fd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35374/0 (socket says 172.21.15.16:35374) 2024-03-31T15:13:17.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:17.046+0000 7f494fe00640 1 -- 172.21.15.16:0/587180750 learned_addr learned my addr 172.21.15.16:0/587180750 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:13:17.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:17.046+0000 7f494ea00640 1 --2- 172.21.15.16:0/587180750 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f49500b2200 0x7f49501f73d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:17.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:17.046+0000 7f494f400640 1 --2- 172.21.15.16:0/587180750 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f49500728b0 0x7f49500b1c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:17.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:17.046+0000 7f494fe00640 1 -- 172.21.15.16:0/587180750 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f49500728b0 msgr2=0x7f49500b1c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:13:17.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:17.046+0000 7f494fe00640 1 --2- 172.21.15.16:0/587180750 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f49500728b0 0x7f49500b1c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:17.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:17.046+0000 7f494fe00640 1 -- 172.21.15.16:0/587180750 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f49500b2200 msgr2=0x7f49501f73d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:13:17.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:17.046+0000 7f494fe00640 1 --2- 172.21.15.16:0/587180750 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f49500b2200 0x7f49501f73d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:17.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:17.046+0000 7f494fe00640 1 -- 172.21.15.16:0/587180750 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49501402d0 con 0x7f4950145d10 2024-03-31T15:13:17.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:17.046+0000 7f494f400640 1 --2- 172.21.15.16:0/587180750 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f49500728b0 0x7f49500b1c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:13:17.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:17.046+0000 7f494fe00640 1 --2- 172.21.15.16:0/587180750 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4950145d10 0x7f495013fd90 secure :-1 s=READY pgs=1527 cs=0 l=1 rev1=1 crypto rx=0x7f49480af040 tx=0x7f4948019790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:13:17.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:17.046+0000 7f4947400640 1 -- 172.21.15.16:0/587180750 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4948009510 con 0x7f4950145d10 2024-03-31T15:13:17.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:17.046+0000 7f4947400640 1 -- 172.21.15.16:0/587180750 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f49480be070 con 0x7f4950145d10 2024-03-31T15:13:17.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:17.046+0000 7f4956200640 1 -- 172.21.15.16:0/587180750 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4950140570 con 0x7f4950145d10 2024-03-31T15:13:17.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:17.046+0000 7f4947400640 1 -- 172.21.15.16:0/587180750 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49480b88b0 con 0x7f4950145d10 2024-03-31T15:13:17.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:17.046+0000 7f4956200640 1 -- 172.21.15.16:0/587180750 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f49500afcb0 con 0x7f4950145d10 2024-03-31T15:13:17.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:17.050+0000 7f4956200640 1 -- 172.21.15.16:0/587180750 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4950069950 con 0x7f4950145d10 2024-03-31T15:13:17.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:17.050+0000 7f4947400640 1 -- 172.21.15.16:0/587180750 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f49480b8a50 con 0x7f4950145d10 2024-03-31T15:13:17.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:17.050+0000 7f4947400640 1 --2- 172.21.15.16:0/587180750 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f4930068380 0x7f493006a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:17.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:17.050+0000 7f4947400640 1 -- 172.21.15.16:0/587180750 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 5795+0+0 (secure 0 0 0) 0x7f4948121b80 con 0x7f4950145d10 2024-03-31T15:13:17.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:17.050+0000 7f494f400640 1 --2- 172.21.15.16:0/587180750 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f4930068380 0x7f493006a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:17.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:17.050+0000 7f494f400640 1 --2- 172.21.15.16:0/587180750 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f4930068380 0x7f493006a840 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f49400097a0 tx=0x7f4940008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:13:17.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:17.054+0000 7f4947400640 1 -- 172.21.15.16:0/587180750 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f49480165e0 con 0x7f4950145d10 2024-03-31T15:13:17.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:17.254+0000 7f4956200640 1 -- 172.21.15.16:0/587180750 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f49500b2710 con 0x7f4950145d10 2024-03-31T15:13:17.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:17.254+0000 7f4947400640 1 -- 172.21.15.16:0/587180750 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 88 v88) ==== 94+0+5174 (secure 0 0 0) 0x7f49480f2080 con 0x7f4950145d10 2024-03-31T15:13:17.249 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:13:17.249 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":88,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7139,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6836/3798928623","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":3798928623},{"type":"v1","addr":"172.21.15.89:6836","nonce":3798928623}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":78},{"gid":7142,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/3074283817","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6835","nonce":3074283817},{"type":"v1","addr":"172.21.15.89:6837","nonce":3074283817}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":78},{"gid":8071,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6834/2110189998","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":2110189998},{"type":"v1","addr":"172.21.15.16:6834","nonce":2110189998}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":78}],"filesystems":[{"mdsmap":{"epoch":88,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-31T15:12:16.294600+0000","modified":"2024-03-31T15:13:16.094376+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":97,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8874},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8874":{"gid":8874,"name":"a","rank":0,"incarnation":86,"state":"up:rejoin","state_seq":3,"addr":"172.21.15.16:6835/567808781","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6833","nonce":567808781},{"type":"v1","addr":"172.21.15.16:6835","nonce":567808781}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":7}]} 2024-03-31T15:13:17.250 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 88 2024-03-31T15:13:17.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:17.258+0000 7f4956200640 1 -- 172.21.15.16:0/587180750 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f4930068380 msgr2=0x7f493006a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:13:17.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:17.258+0000 7f4956200640 1 --2- 172.21.15.16:0/587180750 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f4930068380 0x7f493006a840 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f49400097a0 tx=0x7f4940008040 comp rx=0 tx=0).stop 2024-03-31T15:13:17.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:17.258+0000 7f4956200640 1 -- 172.21.15.16:0/587180750 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4950145d10 msgr2=0x7f495013fd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:13:17.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:17.258+0000 7f4956200640 1 --2- 172.21.15.16:0/587180750 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4950145d10 0x7f495013fd90 secure :-1 s=READY pgs=1527 cs=0 l=1 rev1=1 crypto rx=0x7f49480af040 tx=0x7f4948019790 comp rx=0 tx=0).stop 2024-03-31T15:13:17.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:17.258+0000 7f4956200640 1 -- 172.21.15.16:0/587180750 shutdown_connections 2024-03-31T15:13:17.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:17.258+0000 7f4956200640 1 --2- 172.21.15.16:0/587180750 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f4930068380 0x7f493006a840 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:17.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:17.258+0000 7f4956200640 1 --2- 172.21.15.16:0/587180750 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4950145d10 0x7f495013fd90 unknown :-1 s=CLOSED pgs=1527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:17.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:17.258+0000 7f4956200640 1 --2- 172.21.15.16:0/587180750 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f49500b2200 0x7f49501f73d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:17.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:17.258+0000 7f4956200640 1 --2- 172.21.15.16:0/587180750 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f49500728b0 0x7f49500b1c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:17.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:17.258+0000 7f4956200640 1 -- 172.21.15.16:0/587180750 >> 172.21.15.16:0/587180750 conn(0x7f495006b620 msgr2=0x7f49500b4700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:13:17.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:17.258+0000 7f4956200640 1 -- 172.21.15.16:0/587180750 shutdown_connections 2024-03-31T15:13:17.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:17.258+0000 7f4956200640 1 -- 172.21.15.16:0/587180750 wait complete. 2024-03-31T15:13:17.674 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 88, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-31T15:12:16.294600+0000', 'modified': '2024-03-31T15:13:16.094376+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 97, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8874}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8874': {'gid': 8874, 'name': 'a', 'rank': 0, 'incarnation': 86, 'state': 'up:rejoin', 'state_seq': 3, 'addr': '172.21.15.16:6835/567808781', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6833', 'nonce': 567808781}, {'type': 'v1', 'addr': '172.21.15.16:6835', 'nonce': 567808781}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [15], 'metadata_pool': 14, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-03-31T15:13:17.674 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8874:up:rejoin 2024-03-31T15:13:18.675 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-31T15:13:18.818 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:18.826+0000 7f3078e00640 1 Processor -- start 2024-03-31T15:13:18.818 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:18.826+0000 7f3078e00640 1 -- start start 2024-03-31T15:13:18.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:18.826+0000 7f3078e00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f307406fd80 0x7f30740d2720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:18.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:18.826+0000 7f3078e00640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f30740d0920 con 0x7f3074145e40 2024-03-31T15:13:18.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:18.826+0000 7f3078e00640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f30740d0c00 con 0x7f307406f920 2024-03-31T15:13:18.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:18.826+0000 7f3078e00640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f30740d06d0 con 0x7f307406fd80 2024-03-31T15:13:18.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:18.826+0000 7f3071e00640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f3074145e40 0x7f30740cc740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:33930/0 (socket says 172.21.15.16:33930) 2024-03-31T15:13:18.820 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:18.826+0000 7f3071e00640 1 -- 172.21.15.16:0/2833799733 learned_addr learned my addr 172.21.15.16:0/2833799733 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:13:18.820 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:18.826+0000 7f3072800640 1 --2- 172.21.15.16:0/2833799733 >> v2:172.21.15.89:3300/0 conn(0x7f307406fd80 0x7f30740d2720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:18.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:18.826+0000 7f3070a00640 1 -- 172.21.15.16:0/2833799733 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4195046236 0 0) 0x7f30740d0920 con 0x7f3074145e40 2024-03-31T15:13:18.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:18.826+0000 7f3070a00640 1 -- 172.21.15.16:0/2833799733 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3058003530 con 0x7f3074145e40 2024-03-31T15:13:18.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:18.826+0000 7f3070a00640 1 -- 172.21.15.16:0/2833799733 <== mon.1 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4054764417 0 0) 0x7f30740d0c00 con 0x7f307406f920 2024-03-31T15:13:18.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:18.826+0000 7f3070a00640 1 -- 172.21.15.16:0/2833799733 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f30740d0920 con 0x7f307406f920 2024-03-31T15:13:18.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:18.826+0000 7f3070a00640 1 -- 172.21.15.16:0/2833799733 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1080684542 0 0) 0x7f3058003530 con 0x7f3074145e40 2024-03-31T15:13:18.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:18.826+0000 7f3072800640 1 -- 172.21.15.16:0/2833799733 >> v1:172.21.15.89:6789/0 conn(0x7f307406f920 legacy=0x7f30740d2010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:13:18.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:18.826+0000 7f3072800640 1 -- 172.21.15.16:0/2833799733 >> v1:172.21.15.16:6789/0 conn(0x7f3074145e40 legacy=0x7f30740cc740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:13:18.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:18.826+0000 7f3072800640 1 -- 172.21.15.16:0/2833799733 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f30740d1810 con 0x7f307406fd80 2024-03-31T15:13:18.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:18.830+0000 7f3072800640 1 --2- 172.21.15.16:0/2833799733 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f307406fd80 0x7f30740d2720 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f306c009d90 tx=0x7f306c01c890 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c3e58087111f9574 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:13:18.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:18.830+0000 7f3070a00640 1 -- 172.21.15.16:0/2833799733 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f306c0b2070 con 0x7f307406fd80 2024-03-31T15:13:18.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:18.830+0000 7f3070a00640 1 -- 172.21.15.16:0/2833799733 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f306c0a55c0 con 0x7f307406fd80 2024-03-31T15:13:18.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:18.830+0000 7f3070a00640 1 -- 172.21.15.16:0/2833799733 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3064001680 con 0x7f307406fd80 2024-03-31T15:13:18.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:18.830+0000 7f3078e00640 1 -- 172.21.15.16:0/2833799733 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f307406fd80 msgr2=0x7f30740d2720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:13:18.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:18.830+0000 7f3078e00640 1 --2- 172.21.15.16:0/2833799733 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f307406fd80 0x7f30740d2720 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f306c009d90 tx=0x7f306c01c890 comp rx=0 tx=0).stop 2024-03-31T15:13:18.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:18.830+0000 7f3078e00640 1 -- 172.21.15.16:0/2833799733 shutdown_connections 2024-03-31T15:13:18.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:18.830+0000 7f3078e00640 1 --2- 172.21.15.16:0/2833799733 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f307406fd80 0x7f30740d2720 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:18.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:18.830+0000 7f3078e00640 1 -- 172.21.15.16:0/2833799733 >> 172.21.15.16:0/2833799733 conn(0x7f3074067330 msgr2=0x7f3074067740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:13:18.825 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:18.830+0000 7f3078e00640 1 -- 172.21.15.16:0/2833799733 shutdown_connections 2024-03-31T15:13:18.825 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:18.830+0000 7f3078e00640 1 -- 172.21.15.16:0/2833799733 wait complete. 2024-03-31T15:13:18.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:18.834+0000 7f3078e00640 1 Processor -- start 2024-03-31T15:13:18.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:18.834+0000 7f3078e00640 1 -- start start 2024-03-31T15:13:18.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:18.834+0000 7f3078e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f307406f920 0x7f3074144f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:18.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:18.834+0000 7f3078e00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f307406fd80 0x7f3074159bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:18.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:18.834+0000 7f3078e00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f3074145e40 0x7f307415a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:18.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:18.834+0000 7f3078e00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f30740d0ee0 con 0x7f307406f920 2024-03-31T15:13:18.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:18.834+0000 7f3078e00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f30740ce9e0 con 0x7f307406fd80 2024-03-31T15:13:18.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:18.834+0000 7f3078e00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f30740b50c0 con 0x7f3074145e40 2024-03-31T15:13:18.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:18.834+0000 7f3071e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f307406f920 0x7f3074144f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:18.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:18.834+0000 7f3071e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f307406f920 0x7f3074144f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35382/0 (socket says 172.21.15.16:35382) 2024-03-31T15:13:18.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:18.834+0000 7f3071e00640 1 -- 172.21.15.16:0/2052101668 learned_addr learned my addr 172.21.15.16:0/2052101668 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:13:18.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:18.834+0000 7f3072800640 1 --2- 172.21.15.16:0/2052101668 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f3074145e40 0x7f307415a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:18.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:18.834+0000 7f3071400640 1 --2- 172.21.15.16:0/2052101668 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f307406fd80 0x7f3074159bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:18.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:18.838+0000 7f3071e00640 1 -- 172.21.15.16:0/2052101668 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f3074145e40 msgr2=0x7f307415a100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:13:18.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:18.838+0000 7f3071e00640 1 --2- 172.21.15.16:0/2052101668 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f3074145e40 0x7f307415a100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:18.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:18.838+0000 7f3071e00640 1 -- 172.21.15.16:0/2052101668 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f307406fd80 msgr2=0x7f3074159bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:13:18.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:18.838+0000 7f3071e00640 1 --2- 172.21.15.16:0/2052101668 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f307406fd80 0x7f3074159bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:18.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:18.838+0000 7f3071e00640 1 -- 172.21.15.16:0/2052101668 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f307414eaf0 con 0x7f307406f920 2024-03-31T15:13:18.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:18.838+0000 7f3071400640 1 --2- 172.21.15.16:0/2052101668 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f307406fd80 0x7f3074159bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:13:18.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:18.838+0000 7f3071e00640 1 --2- 172.21.15.16:0/2052101668 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f307406f920 0x7f3074144f20 secure :-1 s=READY pgs=1529 cs=0 l=1 rev1=1 crypto rx=0x7f306400c7b0 tx=0x7f306400cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:13:18.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:18.838+0000 7f306a000640 1 -- 172.21.15.16:0/2052101668 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3064017070 con 0x7f307406f920 2024-03-31T15:13:18.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:18.838+0000 7f306a000640 1 -- 172.21.15.16:0/2052101668 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3064005b00 con 0x7f307406f920 2024-03-31T15:13:18.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:18.838+0000 7f3078e00640 1 -- 172.21.15.16:0/2052101668 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f307415e680 con 0x7f307406f920 2024-03-31T15:13:18.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:18.838+0000 7f306a000640 1 -- 172.21.15.16:0/2052101668 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3064007070 con 0x7f307406f920 2024-03-31T15:13:18.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:18.838+0000 7f3078e00640 1 -- 172.21.15.16:0/2052101668 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f307415ec40 con 0x7f307406f920 2024-03-31T15:13:18.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:18.838+0000 7f3078e00640 1 -- 172.21.15.16:0/2052101668 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3074069950 con 0x7f307406f920 2024-03-31T15:13:18.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:18.838+0000 7f306a000640 1 -- 172.21.15.16:0/2052101668 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f306401a070 con 0x7f307406f920 2024-03-31T15:13:18.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:18.838+0000 7f306a000640 1 --2- 172.21.15.16:0/2052101668 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f3054068380 0x7f305406a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:18.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:18.838+0000 7f306a000640 1 -- 172.21.15.16:0/2052101668 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 5795+0+0 (secure 0 0 0) 0x7f3064085f00 con 0x7f307406f920 2024-03-31T15:13:18.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:18.842+0000 7f3071400640 1 --2- 172.21.15.16:0/2052101668 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f3054068380 0x7f305406a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:18.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:18.842+0000 7f3071400640 1 --2- 172.21.15.16:0/2052101668 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f3054068380 0x7f305406a840 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f30740b3930 tx=0x7f3060009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:13:18.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:18.842+0000 7f306a000640 1 -- 172.21.15.16:0/2052101668 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f3064055e00 con 0x7f307406f920 2024-03-31T15:13:19.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.046+0000 7f3078e00640 1 -- 172.21.15.16:0/2052101668 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f30740cf3c0 con 0x7f307406f920 2024-03-31T15:13:19.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.046+0000 7f306a000640 1 -- 172.21.15.16:0/2052101668 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 89 v89) ==== 94+0+5181 (secure 0 0 0) 0x7f306405a100 con 0x7f307406f920 2024-03-31T15:13:19.040 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:13:19.040 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":89,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7139,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6836/3798928623","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":3798928623},{"type":"v1","addr":"172.21.15.89:6836","nonce":3798928623}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":78},{"gid":7142,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/3074283817","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6835","nonce":3074283817},{"type":"v1","addr":"172.21.15.89:6837","nonce":3074283817}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":78},{"gid":8071,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6834/2110189998","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":2110189998},{"type":"v1","addr":"172.21.15.16:6834","nonce":2110189998}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":78}],"filesystems":[{"mdsmap":{"epoch":89,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-31T15:12:16.294600+0000","modified":"2024-03-31T15:13:18.099535+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":97,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8874},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8874":{"gid":8874,"name":"a","rank":0,"incarnation":86,"state":"up:active","state_seq":4,"addr":"172.21.15.16:6835/567808781","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6833","nonce":567808781},{"type":"v1","addr":"172.21.15.16:6835","nonce":567808781}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8874,"qdb_cluster":[8874]},"id":7}]} 2024-03-31T15:13:19.040 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 89 2024-03-31T15:13:19.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.050+0000 7f3078e00640 1 -- 172.21.15.16:0/2052101668 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f3054068380 msgr2=0x7f305406a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:13:19.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.050+0000 7f3078e00640 1 --2- 172.21.15.16:0/2052101668 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f3054068380 0x7f305406a840 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f30740b3930 tx=0x7f3060009040 comp rx=0 tx=0).stop 2024-03-31T15:13:19.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.050+0000 7f3078e00640 1 -- 172.21.15.16:0/2052101668 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f307406f920 msgr2=0x7f3074144f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:13:19.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.050+0000 7f3078e00640 1 --2- 172.21.15.16:0/2052101668 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f307406f920 0x7f3074144f20 secure :-1 s=READY pgs=1529 cs=0 l=1 rev1=1 crypto rx=0x7f306400c7b0 tx=0x7f306400cc70 comp rx=0 tx=0).stop 2024-03-31T15:13:19.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.050+0000 7f3078e00640 1 -- 172.21.15.16:0/2052101668 shutdown_connections 2024-03-31T15:13:19.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.050+0000 7f3078e00640 1 --2- 172.21.15.16:0/2052101668 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f3054068380 0x7f305406a840 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:19.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.050+0000 7f3078e00640 1 --2- 172.21.15.16:0/2052101668 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f3074145e40 0x7f307415a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:19.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.050+0000 7f3078e00640 1 --2- 172.21.15.16:0/2052101668 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f307406fd80 0x7f3074159bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:19.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.050+0000 7f3078e00640 1 --2- 172.21.15.16:0/2052101668 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f307406f920 0x7f3074144f20 unknown :-1 s=CLOSED pgs=1529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:19.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.050+0000 7f3078e00640 1 -- 172.21.15.16:0/2052101668 >> 172.21.15.16:0/2052101668 conn(0x7f3074067330 msgr2=0x7f30740d2010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:13:19.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.050+0000 7f3078e00640 1 -- 172.21.15.16:0/2052101668 shutdown_connections 2024-03-31T15:13:19.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.050+0000 7f3078e00640 1 -- 172.21.15.16:0/2052101668 wait complete. 2024-03-31T15:13:19.466 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 89, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-31T15:12:16.294600+0000', 'modified': '2024-03-31T15:13:18.099535+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 97, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8874}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8874': {'gid': 8874, 'name': 'a', 'rank': 0, 'incarnation': 86, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.16:6835/567808781', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6833', 'nonce': 567808781}, {'type': 'v1', 'addr': '172.21.15.16:6835', 'nonce': 567808781}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [15], 'metadata_pool': 14, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8874, 'qdb_cluster': [8874]} 2024-03-31T15:13:19.466 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-31T15:13:19.466 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-03-31T15:13:19.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.614+0000 7f2eb8c00640 1 Processor -- start 2024-03-31T15:13:19.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.614+0000 7f2eb8c00640 1 -- start start 2024-03-31T15:13:19.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.614+0000 7f2eb8c00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f2eb406f920 0x7f2eb406fd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:19.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.614+0000 7f2eb8c00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f2eb40cc740 0x7f2eb40ccbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:19.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.614+0000 7f2eb8c00640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2eb4162010 con 0x7f2eb406d490 2024-03-31T15:13:19.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.614+0000 7f2eb8c00640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f2eb4162be0 con 0x7f2eb40cc740 2024-03-31T15:13:19.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.614+0000 7f2eb8c00640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f2eb4070240 con 0x7f2eb406f920 2024-03-31T15:13:19.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.614+0000 7f2eb2800640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f2eb40cc740 0x7f2eb40ccbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:19.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.614+0000 7f2eb1400640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f2eb406d490 0x7f2eb406d8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:33940/0 (socket says 172.21.15.16:33940) 2024-03-31T15:13:19.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.614+0000 7f2eb1400640 1 -- 172.21.15.16:0/806405444 learned_addr learned my addr 172.21.15.16:0/806405444 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:13:19.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.614+0000 7f2eb1e00640 1 --2- 172.21.15.16:0/806405444 >> v2:172.21.15.89:3300/0 conn(0x7f2eb406f920 0x7f2eb406fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:19.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.618+0000 7f2eb2800640 1 -- 172.21.15.16:0/806405444 >> v2:172.21.15.89:3300/0 conn(0x7f2eb406f920 msgr2=0x7f2eb406fd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:13:19.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.618+0000 7f2eb2800640 1 --2- 172.21.15.16:0/806405444 >> v2:172.21.15.89:3300/0 conn(0x7f2eb406f920 0x7f2eb406fd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:19.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.618+0000 7f2eb2800640 1 -- 172.21.15.16:0/806405444 >> v1:172.21.15.16:6789/0 conn(0x7f2eb406d490 legacy=0x7f2eb406d8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:13:19.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.618+0000 7f2eb2800640 1 -- 172.21.15.16:0/806405444 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2eb41622b0 con 0x7f2eb40cc740 2024-03-31T15:13:19.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.618+0000 7f2eb0a00640 1 -- 172.21.15.16:0/806405444 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 423176010 0 0) 0x7f2eb4162010 con 0x7f2eb406d490 2024-03-31T15:13:19.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.622+0000 7f2eb2800640 1 --2- 172.21.15.16:0/806405444 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2eb40cc740 0x7f2eb40ccbd0 secure :-1 s=READY pgs=1531 cs=0 l=1 rev1=1 crypto rx=0x7f2eb40ced00 tx=0x7f2eac01c930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe84a96af50e9123 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:13:19.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.622+0000 7f2eb0a00640 1 -- 172.21.15.16:0/806405444 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2eac0b2070 con 0x7f2eb40cc740 2024-03-31T15:13:19.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.622+0000 7f2eb0a00640 1 -- 172.21.15.16:0/806405444 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2eac0a57f0 con 0x7f2eb40cc740 2024-03-31T15:13:19.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.622+0000 7f2eb0a00640 1 -- 172.21.15.16:0/806405444 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2eac0a5bc0 con 0x7f2eb40cc740 2024-03-31T15:13:19.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.622+0000 7f2eb8c00640 1 -- 172.21.15.16:0/806405444 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2eb40cc740 msgr2=0x7f2eb40ccbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:13:19.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.622+0000 7f2eb8c00640 1 --2- 172.21.15.16:0/806405444 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2eb40cc740 0x7f2eb40ccbd0 secure :-1 s=READY pgs=1531 cs=0 l=1 rev1=1 crypto rx=0x7f2eb40ced00 tx=0x7f2eac01c930 comp rx=0 tx=0).stop 2024-03-31T15:13:19.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.622+0000 7f2eb8c00640 1 -- 172.21.15.16:0/806405444 shutdown_connections 2024-03-31T15:13:19.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.622+0000 7f2eb8c00640 1 --2- 172.21.15.16:0/806405444 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2eb40cc740 0x7f2eb40ccbd0 unknown :-1 s=CLOSED pgs=1531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:19.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.622+0000 7f2eb8c00640 1 --2- 172.21.15.16:0/806405444 >> v2:172.21.15.89:3300/0 conn(0x7f2eb406f920 0x7f2eb406fd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:19.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.622+0000 7f2eb8c00640 1 -- 172.21.15.16:0/806405444 >> 172.21.15.16:0/806405444 conn(0x7f2eb4067330 msgr2=0x7f2eb4067740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:13:19.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.622+0000 7f2eb8c00640 1 -- 172.21.15.16:0/806405444 shutdown_connections 2024-03-31T15:13:19.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.622+0000 7f2eb8c00640 1 -- 172.21.15.16:0/806405444 wait complete. 2024-03-31T15:13:19.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.622+0000 7f2eb8c00640 1 Processor -- start 2024-03-31T15:13:19.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.622+0000 7f2eb8c00640 1 -- start start 2024-03-31T15:13:19.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.626+0000 7f2eb8c00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f2eb406d490 0x7f2eb413a4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:19.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.626+0000 7f2eb8c00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2eb406f920 0x7f2eb415f3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:19.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.626+0000 7f2eb8c00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2eb40cc740 0x7f2eb41fb200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:19.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.626+0000 7f2eb8c00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f2eb416b530 con 0x7f2eb406f920 2024-03-31T15:13:19.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.626+0000 7f2eb8c00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f2eb416b3b0 con 0x7f2eb40cc740 2024-03-31T15:13:19.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.626+0000 7f2eb8c00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f2eb416b6b0 con 0x7f2eb406d490 2024-03-31T15:13:19.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.626+0000 7f2eb1400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2eb406f920 0x7f2eb415f3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:19.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.626+0000 7f2eb1400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2eb406f920 0x7f2eb415f3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35396/0 (socket says 172.21.15.16:35396) 2024-03-31T15:13:19.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.626+0000 7f2eb1400640 1 -- 172.21.15.16:0/3667987222 learned_addr learned my addr 172.21.15.16:0/3667987222 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:13:19.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.626+0000 7f2eb2800640 1 --2- 172.21.15.16:0/3667987222 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2eb40cc740 0x7f2eb41fb200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:19.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.626+0000 7f2eb1e00640 1 --2- 172.21.15.16:0/3667987222 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f2eb406d490 0x7f2eb413a4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:19.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.626+0000 7f2eb1400640 1 -- 172.21.15.16:0/3667987222 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f2eb406d490 msgr2=0x7f2eb413a4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:13:19.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.626+0000 7f2eb1400640 1 --2- 172.21.15.16:0/3667987222 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f2eb406d490 0x7f2eb413a4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:19.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.626+0000 7f2eb1400640 1 -- 172.21.15.16:0/3667987222 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2eb40cc740 msgr2=0x7f2eb41fb200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:13:19.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.626+0000 7f2eb1400640 1 --2- 172.21.15.16:0/3667987222 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2eb40cc740 0x7f2eb41fb200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:19.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.626+0000 7f2eb1400640 1 -- 172.21.15.16:0/3667987222 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2eb413b0c0 con 0x7f2eb406f920 2024-03-31T15:13:19.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.626+0000 7f2eb1400640 1 --2- 172.21.15.16:0/3667987222 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2eb406f920 0x7f2eb415f3b0 secure :-1 s=READY pgs=1532 cs=0 l=1 rev1=1 crypto rx=0x7f2ea000c480 tx=0x7f2ea000c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:13:19.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.626+0000 7f2eaa000640 1 -- 172.21.15.16:0/3667987222 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ea0018070 con 0x7f2eb406f920 2024-03-31T15:13:19.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.626+0000 7f2eaa000640 1 -- 172.21.15.16:0/3667987222 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2ea0007500 con 0x7f2eb406f920 2024-03-31T15:13:19.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.626+0000 7f2eaa000640 1 -- 172.21.15.16:0/3667987222 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ea0004a30 con 0x7f2eb406f920 2024-03-31T15:13:19.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.626+0000 7f2eb8c00640 1 -- 172.21.15.16:0/3667987222 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2eb413b3a0 con 0x7f2eb406f920 2024-03-31T15:13:19.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.626+0000 7f2eb8c00640 1 -- 172.21.15.16:0/3667987222 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2eb41face0 con 0x7f2eb406f920 2024-03-31T15:13:19.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.630+0000 7f2eaa000640 1 -- 172.21.15.16:0/3667987222 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f2ea0004bd0 con 0x7f2eb406f920 2024-03-31T15:13:19.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.630+0000 7f2eaa000640 1 --2- 172.21.15.16:0/3667987222 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f2e94068380 0x7f2e9406a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:19.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.630+0000 7f2eaa000640 1 -- 172.21.15.16:0/3667987222 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 5795+0+0 (secure 0 0 0) 0x7f2ea0085c40 con 0x7f2eb406f920 2024-03-31T15:13:19.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.630+0000 7f2eb1e00640 1 --2- 172.21.15.16:0/3667987222 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f2e94068380 0x7f2e9406a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:19.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.630+0000 7f2ebc053000 1 Processor -- start 2024-03-31T15:13:19.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.630+0000 7f2ebc053000 1 -- start start 2024-03-31T15:13:19.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.630+0000 7f2eb1e00640 1 --2- 172.21.15.16:0/3667987222 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f2e94068380 0x7f2e9406a840 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f2eb4140ad0 tx=0x7f2ea4009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:13:19.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.630+0000 7f2ebc053000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a09455e4a0 0x55a09455e880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:19.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.630+0000 7f2ebc053000 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55a09455edc0 0x55a094511b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:19.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.630+0000 7f2ebc053000 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a0945120b0 0x55a094512490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:19.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.634+0000 7f2ebc053000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55a094364950 con 0x55a09455e4a0 2024-03-31T15:13:19.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.634+0000 7f2ebc053000 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55a094301990 con 0x55a0945120b0 2024-03-31T15:13:19.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.634+0000 7f2ebc053000 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x55a094360900 con 0x55a09455edc0 2024-03-31T15:13:19.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.634+0000 7f2eb1e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a09455e4a0 0x55a09455e880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:19.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.634+0000 7f2eb1e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a09455e4a0 0x55a09455e880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35400/0 (socket says 172.21.15.16:35400) 2024-03-31T15:13:19.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.634+0000 7f2eb1e00640 1 -- 172.21.15.16:0/1022163168 learned_addr learned my addr 172.21.15.16:0/1022163168 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:13:19.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.634+0000 7f2eb2800640 1 --2- 172.21.15.16:0/1022163168 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a0945120b0 0x55a094512490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:19.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.634+0000 7f2eb1400640 1 --2- 172.21.15.16:0/1022163168 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55a09455edc0 0x55a094511b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:19.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.634+0000 7f2eb1e00640 1 -- 172.21.15.16:0/1022163168 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55a09455edc0 msgr2=0x55a094511b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:13:19.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.634+0000 7f2eb1e00640 1 --2- 172.21.15.16:0/1022163168 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55a09455edc0 0x55a094511b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:19.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.634+0000 7f2eb1e00640 1 -- 172.21.15.16:0/1022163168 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a0945120b0 msgr2=0x55a094512490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:13:19.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.634+0000 7f2eb1e00640 1 --2- 172.21.15.16:0/1022163168 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a0945120b0 0x55a094512490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:19.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.634+0000 7f2eb1e00640 1 -- 172.21.15.16:0/1022163168 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a0943a9100 con 0x55a09455e4a0 2024-03-31T15:13:19.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.634+0000 7f2eb1e00640 1 --2- 172.21.15.16:0/1022163168 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a09455e4a0 0x55a09455e880 secure :-1 s=READY pgs=1533 cs=0 l=1 rev1=1 crypto rx=0x7f2ea4011e90 tx=0x7f2ea4014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fac620935ff4b881 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:13:19.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.634+0000 7f2e9ea00640 1 -- 172.21.15.16:0/1022163168 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ea4014bd0 con 0x55a09455e4a0 2024-03-31T15:13:19.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.634+0000 7f2e9ea00640 1 -- 172.21.15.16:0/1022163168 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2ea401b070 con 0x55a09455e4a0 2024-03-31T15:13:19.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.634+0000 7f2e9ea00640 1 -- 172.21.15.16:0/1022163168 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ea4004030 con 0x55a09455e4a0 2024-03-31T15:13:19.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.634+0000 7f2ebc053000 1 -- 172.21.15.16:0/1022163168 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a09455e4a0 msgr2=0x55a09455e880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:13:19.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.634+0000 7f2ebc053000 1 --2- 172.21.15.16:0/1022163168 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a09455e4a0 0x55a09455e880 secure :-1 s=READY pgs=1533 cs=0 l=1 rev1=1 crypto rx=0x7f2ea4011e90 tx=0x7f2ea4014000 comp rx=0 tx=0).stop 2024-03-31T15:13:19.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.634+0000 7f2ebc053000 1 -- 172.21.15.16:0/1022163168 shutdown_connections 2024-03-31T15:13:19.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.634+0000 7f2ebc053000 1 --2- 172.21.15.16:0/1022163168 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a0945120b0 0x55a094512490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:19.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.634+0000 7f2ebc053000 1 --2- 172.21.15.16:0/1022163168 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55a09455edc0 0x55a094511b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:19.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.634+0000 7f2ebc053000 1 --2- 172.21.15.16:0/1022163168 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a09455e4a0 0x55a09455e880 unknown :-1 s=CLOSED pgs=1533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:19.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.634+0000 7f2ebc053000 1 -- 172.21.15.16:0/1022163168 >> 172.21.15.16:0/1022163168 conn(0x55a0943fcb80 msgr2=0x55a094541460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:13:19.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.634+0000 7f2ebc053000 1 -- 172.21.15.16:0/1022163168 shutdown_connections 2024-03-31T15:13:19.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.634+0000 7f2ebc053000 1 -- 172.21.15.16:0/1022163168 wait complete. 2024-03-31T15:13:19.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.634+0000 7f2ebc053000 1 -- start start 2024-03-31T15:13:19.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.634+0000 7f2ebc053000 1 Processor -- start 2024-03-31T15:13:19.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.638+0000 7f2ebc053000 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55a0945120b0 0x55a09451ee00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:19.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.638+0000 7f2ebc053000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a09455e4a0 0x55a09451f340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:19.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.638+0000 7f2ebc053000 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a09455edc0 0x55a09451f880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:19.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.638+0000 7f2ebc053000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55a094363c60 con 0x55a09455e4a0 2024-03-31T15:13:19.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.638+0000 7f2ebc053000 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55a0943223e0 con 0x55a09455edc0 2024-03-31T15:13:19.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.638+0000 7f2ebc053000 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x55a094371060 con 0x55a0945120b0 2024-03-31T15:13:19.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.638+0000 7f2ebc053000 10 client.0 _ll_register_callbacks cb 0x55a0943c50f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T15:13:19.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.638+0000 7f2eb1400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a09455e4a0 0x55a09451f340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:19.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.638+0000 7f2eb1400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a09455e4a0 0x55a09451f340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35402/0 (socket says 172.21.15.16:35402) 2024-03-31T15:13:19.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.638+0000 7f2eb1400640 1 -- 172.21.15.16:0/3738961637 learned_addr learned my addr 172.21.15.16:0/3738961637 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:13:19.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.638+0000 7f2eb2800640 1 --2- 172.21.15.16:0/3738961637 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a09455edc0 0x55a09451f880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:19.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.638+0000 7f2eb1e00640 1 --2- 172.21.15.16:0/3738961637 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55a0945120b0 0x55a09451ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:19.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.638+0000 7f2eb1400640 1 -- 172.21.15.16:0/3738961637 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55a0945120b0 msgr2=0x55a09451ee00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:13:19.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.638+0000 7f2eb1400640 1 --2- 172.21.15.16:0/3738961637 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55a0945120b0 0x55a09451ee00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:19.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.638+0000 7f2eb1400640 1 -- 172.21.15.16:0/3738961637 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a09455edc0 msgr2=0x55a09451f880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:13:19.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.638+0000 7f2eb1400640 1 --2- 172.21.15.16:0/3738961637 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a09455edc0 0x55a09451f880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:19.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.638+0000 7f2eb1400640 1 -- 172.21.15.16:0/3738961637 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a09432e8c0 con 0x55a09455e4a0 2024-03-31T15:13:19.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.638+0000 7f2eb1e00640 1 --2- 172.21.15.16:0/3738961637 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55a0945120b0 0x55a09451ee00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:13:19.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.638+0000 7f2eb1400640 1 --2- 172.21.15.16:0/3738961637 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a09455e4a0 0x55a09451f340 secure :-1 s=READY pgs=1534 cs=0 l=1 rev1=1 crypto rx=0x7f2ea000c450 tx=0x7f2ea0059f00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9060368777b9794e server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:13:19.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.638+0000 7f2e93400640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-31T15:13:19.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.638+0000 7f2e93400640 1 -- 172.21.15.16:0/3738961637 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ea005c2c0 con 0x55a09455e4a0 2024-03-31T15:13:19.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.638+0000 7f2e93400640 1 -- 172.21.15.16:0/3738961637 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 5795+0+0 (secure 0 0 0) 0x7f2ea0054f90 con 0x55a09455e4a0 2024-03-31T15:13:19.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.638+0000 7f2ebc053000 1 -- 172.21.15.16:0/3738961637 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2ea401b070 con 0x55a09455e4a0 2024-03-31T15:13:19.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.638+0000 7f2e93400640 1 -- 172.21.15.16:0/3738961637 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=89) ==== 24+0+0 (secure 0 0 0) 0x7f2ea0055190 con 0x55a09455e4a0 2024-03-31T15:13:19.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.638+0000 7f2ebc053000 10 client.8949 fetch_fsmap learned FSMap version 89 2024-03-31T15:13:19.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.638+0000 7f2ebc053000 1 -- 172.21.15.16:0/3738961637 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=89,monmap=2+}) -- 0x55a0942ece80 con 0x55a09455e4a0 2024-03-31T15:13:19.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.638+0000 7f2e93400640 1 -- 172.21.15.16:0/3738961637 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2ea401b070 con 0x55a09455e4a0 2024-03-31T15:13:19.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.638+0000 7f2e93400640 1 -- 172.21.15.16:0/3738961637 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 89) ==== 3494+0+0 (secure 0 0 0) 0x7f2ea0070030 con 0x55a09455e4a0 2024-03-31T15:13:19.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.638+0000 7f2ebc053000 10 client.8949 fetch_fsmap finished waiting for FSMap version 89 2024-03-31T15:13:19.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.638+0000 7f2ebc053000 10 client.8949 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-31T15:13:19.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.638+0000 7f2ebc053000 20 client.8949 populate_metadata read hostname 'smithi016' 2024-03-31T15:13:19.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.638+0000 7f2ebc053000 1 --2- 172.21.15.16:0/3738961637 >> [v2:172.21.15.16:6833/567808781,v1:172.21.15.16:6835/567808781] conn(0x55a094520120 0x55a094570a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:19.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.638+0000 7f2ebc053000 4 client.8949 mds_command: new command op to 8874 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-31T15:13:19.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.638+0000 7f2ebc053000 1 -- 172.21.15.16:0/3738961637 --> [v2:172.21.15.16:6833/567808781,v1:172.21.15.16:6835/567808781] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a0942ffbc0 con 0x55a094520120 2024-03-31T15:13:19.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.638+0000 7f2eb1e00640 1 --2- 172.21.15.16:0/3738961637 >> [v2:172.21.15.16:6833/567808781,v1:172.21.15.16:6835/567808781] conn(0x55a094520120 0x55a094570a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:19.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.642+0000 7f2e93400640 10 client.8949 ms_handle_connect on v2:172.21.15.16:6833/567808781 2024-03-31T15:13:19.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.642+0000 7f2eb1e00640 1 --2- 172.21.15.16:0/3738961637 >> [v2:172.21.15.16:6833/567808781,v1:172.21.15.16:6835/567808781] conn(0x55a094520120 0x55a094570a20 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dab3e00443a47fa0 server_cookie=c9eb2ad934eacd55 in_seq=0 out_seq=0 2024-03-31T15:13:19.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.642+0000 7f2e93400640 1 -- 172.21.15.16:0/3738961637 <== mds.0 v2:172.21.15.16:6833/567808781 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55a0942ffbc0 con 0x55a094520120 2024-03-31T15:13:19.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.642+0000 7f2e93400640 10 client.8949 handle_command_reply: tid=0 2024-03-31T15:13:19.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.642+0000 7f2ebc053000 1 client.8949 shutdown 2024-03-31T15:13:19.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.642+0000 7f2ebc053000 1 -- 172.21.15.16:0/3738961637 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a09455e4a0 msgr2=0x55a09451f340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:13:19.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.642+0000 7f2ebc053000 1 --2- 172.21.15.16:0/3738961637 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a09455e4a0 0x55a09451f340 secure :-1 s=READY pgs=1534 cs=0 l=1 rev1=1 crypto rx=0x7f2ea000c450 tx=0x7f2ea0059f00 comp rx=0 tx=0).stop 2024-03-31T15:13:19.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.642+0000 7f2ebc053000 1 -- 172.21.15.16:0/3738961637 shutdown_connections 2024-03-31T15:13:19.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.642+0000 7f2ebc053000 1 --2- 172.21.15.16:0/3738961637 >> [v2:172.21.15.16:6833/567808781,v1:172.21.15.16:6835/567808781] conn(0x55a094520120 0x55a094570a20 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:19.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.642+0000 7f2ebc053000 1 --2- 172.21.15.16:0/3738961637 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a09455edc0 0x55a09451f880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:19.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.642+0000 7f2ebc053000 1 --2- 172.21.15.16:0/3738961637 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a09455e4a0 0x55a09451f340 unknown :-1 s=CLOSED pgs=1534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:19.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.642+0000 7f2ebc053000 1 --2- 172.21.15.16:0/3738961637 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55a0945120b0 0x55a09451ee00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:19.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.642+0000 7f2e93400640 0 client.8949 ms_handle_reset on v2:172.21.15.16:6833/567808781 2024-03-31T15:13:19.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.642+0000 7f2ebc053000 1 -- 172.21.15.16:0/3738961637 >> 172.21.15.16:0/3738961637 conn(0x55a0943fcb80 msgr2=0x55a094511270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:13:19.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.642+0000 7f2ebc053000 1 -- 172.21.15.16:0/3738961637 shutdown_connections 2024-03-31T15:13:19.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.642+0000 7f2ebc053000 1 -- 172.21.15.16:0/3738961637 wait complete. 2024-03-31T15:13:19.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.642+0000 7f2ebc053000 20 client.8949 trim_cache size 0 max 16384 2024-03-31T15:13:19.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.674+0000 7f2ebc053000 1 Processor -- start 2024-03-31T15:13:19.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.674+0000 7f2ebc053000 1 -- start start 2024-03-31T15:13:19.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.674+0000 7f2ebc053000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a0945120b0 0x55a09455dda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:19.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.674+0000 7f2ebc053000 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a09455e4a0 0x55a094540600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:19.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.674+0000 7f2ebc053000 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55a09455edc0 0x55a094540b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:19.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.674+0000 7f2ebc053000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f2eb4203900 con 0x55a0945120b0 2024-03-31T15:13:19.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.674+0000 7f2ebc053000 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f2eb40b5950 con 0x55a09455e4a0 2024-03-31T15:13:19.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.674+0000 7f2ebc053000 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f2eb41605a0 con 0x55a09455edc0 2024-03-31T15:13:19.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.674+0000 7f2eb1e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a0945120b0 0x55a09455dda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:19.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.674+0000 7f2eb1e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a0945120b0 0x55a09455dda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35404/0 (socket says 172.21.15.16:35404) 2024-03-31T15:13:19.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.674+0000 7f2eb1e00640 1 -- 172.21.15.16:0/2434039365 learned_addr learned my addr 172.21.15.16:0/2434039365 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:13:19.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.674+0000 7f2eb1400640 1 --2- 172.21.15.16:0/2434039365 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a09455e4a0 0x55a094540600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:19.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.674+0000 7f2eb2800640 1 --2- 172.21.15.16:0/2434039365 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55a09455edc0 0x55a094540b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:19.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.674+0000 7f2eb1e00640 1 -- 172.21.15.16:0/2434039365 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55a09455edc0 msgr2=0x55a094540b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:13:19.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.674+0000 7f2eb1e00640 1 --2- 172.21.15.16:0/2434039365 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55a09455edc0 0x55a094540b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:19.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.674+0000 7f2eb1e00640 1 -- 172.21.15.16:0/2434039365 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a09455e4a0 msgr2=0x55a094540600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:13:19.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.674+0000 7f2eb1e00640 1 --2- 172.21.15.16:0/2434039365 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a09455e4a0 0x55a094540600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:19.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.674+0000 7f2eb1e00640 1 -- 172.21.15.16:0/2434039365 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a09431f490 con 0x55a0945120b0 2024-03-31T15:13:19.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.674+0000 7f2eb2800640 1 --2- 172.21.15.16:0/2434039365 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55a09455edc0 0x55a094540b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:13:19.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.674+0000 7f2eb1e00640 1 --2- 172.21.15.16:0/2434039365 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a0945120b0 0x55a09455dda0 secure :-1 s=READY pgs=1535 cs=0 l=1 rev1=1 crypto rx=0x7f2ea4011e30 tx=0x7f2ea4016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=36b2dacd55abe85 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:13:19.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.674+0000 7f2e9e000640 1 -- 172.21.15.16:0/2434039365 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ea4004230 con 0x55a0945120b0 2024-03-31T15:13:19.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.674+0000 7f2e9e000640 1 -- 172.21.15.16:0/2434039365 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2ea401b040 con 0x55a0945120b0 2024-03-31T15:13:19.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.674+0000 7f2e9e000640 1 -- 172.21.15.16:0/2434039365 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ea4003620 con 0x55a0945120b0 2024-03-31T15:13:19.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.674+0000 7f2ebc053000 1 -- 172.21.15.16:0/2434039365 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a0945120b0 msgr2=0x55a09455dda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:13:19.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.674+0000 7f2ebc053000 1 --2- 172.21.15.16:0/2434039365 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a0945120b0 0x55a09455dda0 secure :-1 s=READY pgs=1535 cs=0 l=1 rev1=1 crypto rx=0x7f2ea4011e30 tx=0x7f2ea4016000 comp rx=0 tx=0).stop 2024-03-31T15:13:19.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.674+0000 7f2ebc053000 1 -- 172.21.15.16:0/2434039365 shutdown_connections 2024-03-31T15:13:19.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.674+0000 7f2ebc053000 1 --2- 172.21.15.16:0/2434039365 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55a09455edc0 0x55a094540b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:19.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.674+0000 7f2ebc053000 1 --2- 172.21.15.16:0/2434039365 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a09455e4a0 0x55a094540600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:19.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.674+0000 7f2ebc053000 1 --2- 172.21.15.16:0/2434039365 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a0945120b0 0x55a09455dda0 unknown :-1 s=CLOSED pgs=1535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:19.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.674+0000 7f2ebc053000 1 -- 172.21.15.16:0/2434039365 >> 172.21.15.16:0/2434039365 conn(0x55a0943fcb80 msgr2=0x55a09456ba90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:13:19.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.674+0000 7f2ebc053000 1 -- 172.21.15.16:0/2434039365 shutdown_connections 2024-03-31T15:13:19.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.674+0000 7f2ebc053000 1 -- 172.21.15.16:0/2434039365 wait complete. 2024-03-31T15:13:19.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.674+0000 7f2ebc053000 1 -- start start 2024-03-31T15:13:19.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.674+0000 7f2ebc053000 1 Processor -- start 2024-03-31T15:13:19.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.678+0000 7f2ebc053000 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55a0945120b0 0x55a094511710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:19.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.678+0000 7f2ebc053000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a09455e4a0 0x55a0945b1c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:19.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.678+0000 7f2ebc053000 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a09455edc0 0x55a0945b2190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:19.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.678+0000 7f2ebc053000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55a094368f80 con 0x55a09455e4a0 2024-03-31T15:13:19.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.678+0000 7f2ebc053000 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f2eb40b5300 con 0x55a09455edc0 2024-03-31T15:13:19.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.678+0000 7f2ebc053000 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x55a094326080 con 0x55a0945120b0 2024-03-31T15:13:19.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.678+0000 7f2ebc053000 10 client.0 _ll_register_callbacks cb 0x7f2ea0055730 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T15:13:19.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.678+0000 7f2eb1400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a09455e4a0 0x55a0945b1c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:19.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.678+0000 7f2eb1400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a09455e4a0 0x55a0945b1c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35408/0 (socket says 172.21.15.16:35408) 2024-03-31T15:13:19.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.678+0000 7f2eb1400640 1 -- 172.21.15.16:0/198829394 learned_addr learned my addr 172.21.15.16:0/198829394 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:13:19.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.678+0000 7f2eb1e00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55a0945120b0 0x55a094511710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:19.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.678+0000 7f2eb2800640 1 --2- 172.21.15.16:0/198829394 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a09455edc0 0x55a0945b2190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:19.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.678+0000 7f2eb1400640 1 -- 172.21.15.16:0/198829394 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55a0945120b0 msgr2=0x55a094511710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:13:19.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.678+0000 7f2eb1400640 1 --2- 172.21.15.16:0/198829394 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55a0945120b0 0x55a094511710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:19.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.678+0000 7f2eb1400640 1 -- 172.21.15.16:0/198829394 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a09455edc0 msgr2=0x55a0945b2190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:13:19.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.678+0000 7f2eb1400640 1 --2- 172.21.15.16:0/198829394 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a09455edc0 0x55a0945b2190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:19.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.678+0000 7f2eb1400640 1 -- 172.21.15.16:0/198829394 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a09431de80 con 0x55a09455e4a0 2024-03-31T15:13:19.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.678+0000 7f2eb1400640 1 --2- 172.21.15.16:0/198829394 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a09455e4a0 0x55a0945b1c50 secure :-1 s=READY pgs=1536 cs=0 l=1 rev1=1 crypto rx=0x7f2ea0055ad0 tx=0x7f2ea0083fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=147bc1cdbff2c636 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:13:19.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.678+0000 7f2e93400640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-31T15:13:19.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.678+0000 7f2e93400640 1 -- 172.21.15.16:0/198829394 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ea0054c10 con 0x55a09455e4a0 2024-03-31T15:13:19.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.678+0000 7f2e93400640 1 -- 172.21.15.16:0/198829394 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 5795+0+0 (secure 0 0 0) 0x7f2ea00740f0 con 0x55a09455e4a0 2024-03-31T15:13:19.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.678+0000 7f2ebc053000 1 -- 172.21.15.16:0/198829394 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2ea401b040 con 0x55a09455e4a0 2024-03-31T15:13:19.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.678+0000 7f2e93400640 1 -- 172.21.15.16:0/198829394 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=89) ==== 24+0+0 (secure 0 0 0) 0x7f2ea0054db0 con 0x55a09455e4a0 2024-03-31T15:13:19.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.678+0000 7f2ebc053000 10 client.8955 fetch_fsmap learned FSMap version 89 2024-03-31T15:13:19.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.678+0000 7f2ebc053000 1 -- 172.21.15.16:0/198829394 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=89,monmap=2+}) -- 0x55a09430b270 con 0x55a09455e4a0 2024-03-31T15:13:19.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.678+0000 7f2e93400640 1 -- 172.21.15.16:0/198829394 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2ea401b040 con 0x55a09455e4a0 2024-03-31T15:13:19.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.678+0000 7f2e93400640 1 -- 172.21.15.16:0/198829394 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 89) ==== 3494+0+0 (secure 0 0 0) 0x7f2ea0070040 con 0x55a09455e4a0 2024-03-31T15:13:19.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.678+0000 7f2ebc053000 10 client.8955 fetch_fsmap finished waiting for FSMap version 89 2024-03-31T15:13:19.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.678+0000 7f2ebc053000 10 client.8955 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-31T15:13:19.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.678+0000 7f2ebc053000 20 client.8955 populate_metadata read hostname 'smithi016' 2024-03-31T15:13:19.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.678+0000 7f2ebc053000 1 --2- 172.21.15.16:0/198829394 >> [v2:172.21.15.16:6833/567808781,v1:172.21.15.16:6835/567808781] conn(0x55a094511c50 0x55a0945b2a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:19.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.678+0000 7f2ebc053000 4 client.8955 mds_command: new command op to 8874 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-31T15:13:19.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.678+0000 7f2ebc053000 1 -- 172.21.15.16:0/198829394 --> [v2:172.21.15.16:6833/567808781,v1:172.21.15.16:6835/567808781] -- command(tid 0: {"prefix": "status"}) -- 0x55a094371e90 con 0x55a094511c50 2024-03-31T15:13:19.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.678+0000 7f2eb1e00640 1 --2- 172.21.15.16:0/198829394 >> [v2:172.21.15.16:6833/567808781,v1:172.21.15.16:6835/567808781] conn(0x55a094511c50 0x55a0945b2a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:19.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.678+0000 7f2e93400640 10 client.8955 ms_handle_connect on v2:172.21.15.16:6833/567808781 2024-03-31T15:13:19.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.678+0000 7f2eb1e00640 1 --2- 172.21.15.16:0/198829394 >> [v2:172.21.15.16:6833/567808781,v1:172.21.15.16:6835/567808781] conn(0x55a094511c50 0x55a0945b2a30 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2505875907dc5400 server_cookie=e6689d3022b0be5a in_seq=0 out_seq=0 2024-03-31T15:13:19.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.678+0000 7f2e93400640 1 -- 172.21.15.16:0/198829394 <== mds.0 v2:172.21.15.16:6833/567808781 1 ==== command_reply(tid 0: 0 ) ==== 8+0+331 (crc 0 0 0) 0x55a094371e90 con 0x55a094511c50 2024-03-31T15:13:19.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.678+0000 7f2e93400640 10 client.8955 handle_command_reply: tid=0 2024-03-31T15:13:19.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.678+0000 7f2ebc053000 1 client.8955 shutdown 2024-03-31T15:13:19.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.678+0000 7f2ebc053000 1 -- 172.21.15.16:0/198829394 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a09455e4a0 msgr2=0x55a0945b1c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:13:19.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.678+0000 7f2ebc053000 1 --2- 172.21.15.16:0/198829394 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a09455e4a0 0x55a0945b1c50 secure :-1 s=READY pgs=1536 cs=0 l=1 rev1=1 crypto rx=0x7f2ea0055ad0 tx=0x7f2ea0083fd0 comp rx=0 tx=0).stop 2024-03-31T15:13:19.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.678+0000 7f2ebc053000 1 -- 172.21.15.16:0/198829394 shutdown_connections 2024-03-31T15:13:19.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.678+0000 7f2ebc053000 1 --2- 172.21.15.16:0/198829394 >> [v2:172.21.15.16:6833/567808781,v1:172.21.15.16:6835/567808781] conn(0x55a094511c50 0x55a0945b2a30 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:19.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.678+0000 7f2ebc053000 1 --2- 172.21.15.16:0/198829394 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a09455edc0 0x55a0945b2190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:19.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.678+0000 7f2ebc053000 1 --2- 172.21.15.16:0/198829394 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a09455e4a0 0x55a0945b1c50 unknown :-1 s=CLOSED pgs=1536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:19.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.678+0000 7f2ebc053000 1 --2- 172.21.15.16:0/198829394 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55a0945120b0 0x55a094511710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:19.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.682+0000 7f2e93400640 0 client.8955 ms_handle_reset on v2:172.21.15.16:6833/567808781 2024-03-31T15:13:19.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.682+0000 7f2ebc053000 1 -- 172.21.15.16:0/198829394 >> 172.21.15.16:0/198829394 conn(0x55a0943fcb80 msgr2=0x55a094540d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:13:19.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.682+0000 7f2ebc053000 1 -- 172.21.15.16:0/198829394 shutdown_connections 2024-03-31T15:13:19.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.682+0000 7f2ebc053000 1 -- 172.21.15.16:0/198829394 wait complete. 2024-03-31T15:13:19.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.682+0000 7f2ebc053000 20 client.8955 trim_cache size 0 max 16384 2024-03-31T15:13:19.677 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-03-31T15:13:19.677 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_fsid": "286eb5da-fb47-4550-b290-d9519c13b9a5", 2024-03-31T15:13:19.677 INFO:teuthology.orchestra.run.smithi016.stdout: "whoami": 0, 2024-03-31T15:13:19.677 INFO:teuthology.orchestra.run.smithi016.stdout: "id": 8874, 2024-03-31T15:13:19.678 INFO:teuthology.orchestra.run.smithi016.stdout: "want_state": "up:active", 2024-03-31T15:13:19.678 INFO:teuthology.orchestra.run.smithi016.stdout: "state": "up:active", 2024-03-31T15:13:19.678 INFO:teuthology.orchestra.run.smithi016.stdout: "fs_name": "cephfs", 2024-03-31T15:13:19.678 INFO:teuthology.orchestra.run.smithi016.stdout: "rank_uptime": 4.5932672090000004, 2024-03-31T15:13:19.678 INFO:teuthology.orchestra.run.smithi016.stdout: "mdsmap_epoch": 89, 2024-03-31T15:13:19.678 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch": 97, 2024-03-31T15:13:19.678 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch_barrier": 97, 2024-03-31T15:13:19.678 INFO:teuthology.orchestra.run.smithi016.stdout: "uptime": 5.5350600600000002 2024-03-31T15:13:19.678 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-03-31T15:13:19.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.682+0000 7f2eb8c00640 1 -- 172.21.15.16:0/3667987222 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f2e94068380 msgr2=0x7f2e9406a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:13:19.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.682+0000 7f2eb8c00640 1 --2- 172.21.15.16:0/3667987222 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f2e94068380 0x7f2e9406a840 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f2eb4140ad0 tx=0x7f2ea4009190 comp rx=0 tx=0).stop 2024-03-31T15:13:19.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.682+0000 7f2eb8c00640 1 -- 172.21.15.16:0/3667987222 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2eb406f920 msgr2=0x7f2eb415f3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:13:19.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.682+0000 7f2eb8c00640 1 --2- 172.21.15.16:0/3667987222 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2eb406f920 0x7f2eb415f3b0 secure :-1 s=READY pgs=1532 cs=0 l=1 rev1=1 crypto rx=0x7f2ea000c480 tx=0x7f2ea000c940 comp rx=0 tx=0).stop 2024-03-31T15:13:19.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.682+0000 7f2eb8c00640 1 -- 172.21.15.16:0/3667987222 shutdown_connections 2024-03-31T15:13:19.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.682+0000 7f2eb8c00640 1 --2- 172.21.15.16:0/3667987222 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f2e94068380 0x7f2e9406a840 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:19.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.682+0000 7f2eb8c00640 1 --2- 172.21.15.16:0/3667987222 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2eb40cc740 0x7f2eb41fb200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:19.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.682+0000 7f2eb8c00640 1 --2- 172.21.15.16:0/3667987222 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2eb406f920 0x7f2eb415f3b0 unknown :-1 s=CLOSED pgs=1532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:19.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.682+0000 7f2eb8c00640 1 --2- 172.21.15.16:0/3667987222 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f2eb406d490 0x7f2eb413a4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:19.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.682+0000 7f2eb8c00640 1 -- 172.21.15.16:0/3667987222 >> 172.21.15.16:0/3667987222 conn(0x7f2eb4067330 msgr2=0x7f2eb40d0a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:13:19.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.682+0000 7f2eb8c00640 1 -- 172.21.15.16:0/3667987222 shutdown_connections 2024-03-31T15:13:19.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.682+0000 7f2eb8c00640 1 -- 172.21.15.16:0/3667987222 wait complete. 2024-03-31T15:13:19.694 DEBUG:teuthology.orchestra.run.smithi016:> sudo daemon-helper kill ceph --cluster ceph -w 2024-03-31T15:13:19.697 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:13:19.697 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.7:0 scrub start / repair,recursive 2024-03-31T15:13:19.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.850+0000 7ff775c00640 1 Processor -- start 2024-03-31T15:13:19.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.850+0000 7ff775c00640 1 -- start start 2024-03-31T15:13:19.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.850+0000 7ff775c00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff770147230 0x7ff7700b3dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:19.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.850+0000 7ff775c00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff7700b4300 0x7ff7700d0930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:19.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.850+0000 7ff775c00640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff7700d2020 con 0x7ff7700d0e70 2024-03-31T15:13:19.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.850+0000 7ff775c00640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7ff7700b4760 con 0x7ff770147230 2024-03-31T15:13:19.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.850+0000 7ff775c00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7ff7700d2c30 con 0x7ff7700b4300 2024-03-31T15:13:19.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.850+0000 7ff76ea00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff770147230 0x7ff7700b3dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:19.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.850+0000 7ff76ea00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff770147230 0x7ff7700b3dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35414/0 (socket says 172.21.15.16:35414) 2024-03-31T15:13:19.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.850+0000 7ff76ea00640 1 -- 172.21.15.16:0/3186118474 learned_addr learned my addr 172.21.15.16:0/3186118474 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:13:19.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.850+0000 7ff766000640 1 --2- 172.21.15.16:0/3186118474 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff7700b4300 0x7ff7700d0930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:19.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.854+0000 7ff76ea00640 1 -- 172.21.15.16:0/3186118474 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff7700b4300 msgr2=0x7ff7700d0930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:13:19.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.854+0000 7ff76ea00640 1 --2- 172.21.15.16:0/3186118474 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff7700b4300 0x7ff7700d0930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:19.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.854+0000 7ff76ea00640 1 -- 172.21.15.16:0/3186118474 >> v1:172.21.15.89:6789/0 conn(0x7ff7700d0e70 legacy=0x7ff770162110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:13:19.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.854+0000 7ff76ea00640 1 -- 172.21.15.16:0/3186118474 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff7700d2300 con 0x7ff770147230 2024-03-31T15:13:19.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.854+0000 7ff76ea00640 1 --2- 172.21.15.16:0/3186118474 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff770147230 0x7ff7700b3dc0 secure :-1 s=READY pgs=1537 cs=0 l=1 rev1=1 crypto rx=0x7ff760009e50 tx=0x7ff76001ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d181d2fdfe825ed0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:13:19.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.854+0000 7ff76e000640 1 -- 172.21.15.16:0/3186118474 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7600a57f0 con 0x7ff770147230 2024-03-31T15:13:19.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.854+0000 7ff76e000640 1 -- 172.21.15.16:0/3186118474 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff7600a5990 con 0x7ff770147230 2024-03-31T15:13:19.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.854+0000 7ff76e000640 1 -- 172.21.15.16:0/3186118474 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7600a5d60 con 0x7ff770147230 2024-03-31T15:13:19.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.854+0000 7ff775c00640 1 -- 172.21.15.16:0/3186118474 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff770147230 msgr2=0x7ff7700b3dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:13:19.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.854+0000 7ff775c00640 1 --2- 172.21.15.16:0/3186118474 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff770147230 0x7ff7700b3dc0 secure :-1 s=READY pgs=1537 cs=0 l=1 rev1=1 crypto rx=0x7ff760009e50 tx=0x7ff76001ca30 comp rx=0 tx=0).stop 2024-03-31T15:13:19.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.858+0000 7ff775c00640 1 -- 172.21.15.16:0/3186118474 shutdown_connections 2024-03-31T15:13:19.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.858+0000 7ff775c00640 1 --2- 172.21.15.16:0/3186118474 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff7700b4300 0x7ff7700d0930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:19.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.858+0000 7ff775c00640 1 --2- 172.21.15.16:0/3186118474 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff770147230 0x7ff7700b3dc0 unknown :-1 s=CLOSED pgs=1537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:19.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.858+0000 7ff775c00640 1 -- 172.21.15.16:0/3186118474 >> 172.21.15.16:0/3186118474 conn(0x7ff77006b7f0 msgr2=0x7ff770067330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:13:19.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.858+0000 7ff775c00640 1 -- 172.21.15.16:0/3186118474 shutdown_connections 2024-03-31T15:13:19.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.858+0000 7ff775c00640 1 -- 172.21.15.16:0/3186118474 wait complete. 2024-03-31T15:13:19.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.858+0000 7ff775c00640 1 Processor -- start 2024-03-31T15:13:19.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.858+0000 7ff775c00640 1 -- start start 2024-03-31T15:13:19.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.862+0000 7ff775c00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff7700b4300 0x7ff77015bf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:19.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.862+0000 7ff775c00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff7700d0e70 0x7ff7700b5bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:19.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.862+0000 7ff775c00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff770147230 0x7ff77015eb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:19.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.862+0000 7ff775c00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7ff77016b280 con 0x7ff7700d0e70 2024-03-31T15:13:19.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.862+0000 7ff775c00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7ff7700cc4c0 con 0x7ff770147230 2024-03-31T15:13:19.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.862+0000 7ff775c00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7ff77016b400 con 0x7ff7700b4300 2024-03-31T15:13:19.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.862+0000 7ff766000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff7700d0e70 0x7ff7700b5bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:19.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.862+0000 7ff76ea00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff7700b4300 0x7ff77015bf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:19.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.862+0000 7ff766000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff7700d0e70 0x7ff7700b5bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35416/0 (socket says 172.21.15.16:35416) 2024-03-31T15:13:19.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.862+0000 7ff766000640 1 -- 172.21.15.16:0/3836821108 learned_addr learned my addr 172.21.15.16:0/3836821108 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:13:19.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.862+0000 7ff76f400640 1 --2- 172.21.15.16:0/3836821108 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff770147230 0x7ff77015eb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:19.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.862+0000 7ff766000640 1 -- 172.21.15.16:0/3836821108 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff7700b4300 msgr2=0x7ff77015bf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:13:19.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.862+0000 7ff766000640 1 --2- 172.21.15.16:0/3836821108 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff7700b4300 0x7ff77015bf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:19.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.862+0000 7ff766000640 1 -- 172.21.15.16:0/3836821108 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff770147230 msgr2=0x7ff77015eb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:13:19.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.862+0000 7ff766000640 1 --2- 172.21.15.16:0/3836821108 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff770147230 0x7ff77015eb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:19.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.862+0000 7ff766000640 1 -- 172.21.15.16:0/3836821108 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff77015f570 con 0x7ff7700d0e70 2024-03-31T15:13:19.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.862+0000 7ff766000640 1 --2- 172.21.15.16:0/3836821108 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff7700d0e70 0x7ff7700b5bf0 secure :-1 s=READY pgs=1538 cs=0 l=1 rev1=1 crypto rx=0x7ff75800e930 tx=0x7ff75800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:13:19.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.862+0000 7ff767400640 1 -- 172.21.15.16:0/3836821108 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff758017070 con 0x7ff7700d0e70 2024-03-31T15:13:19.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.862+0000 7ff767400640 1 -- 172.21.15.16:0/3836821108 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff75800ca00 con 0x7ff7700d0e70 2024-03-31T15:13:19.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.862+0000 7ff775c00640 1 -- 172.21.15.16:0/3836821108 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff77015f850 con 0x7ff7700d0e70 2024-03-31T15:13:19.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.862+0000 7ff767400640 1 -- 172.21.15.16:0/3836821108 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff75800cdb0 con 0x7ff7700d0e70 2024-03-31T15:13:19.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.862+0000 7ff775c00640 1 -- 172.21.15.16:0/3836821108 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff77015c7c0 con 0x7ff7700d0e70 2024-03-31T15:13:19.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.866+0000 7ff767400640 1 -- 172.21.15.16:0/3836821108 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7ff75801a070 con 0x7ff7700d0e70 2024-03-31T15:13:19.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.866+0000 7ff779073000 1 Processor -- start 2024-03-31T15:13:19.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.866+0000 7ff767400640 1 --2- 172.21.15.16:0/3836821108 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7ff74c068380 0x7ff74c06a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:19.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.866+0000 7ff779073000 1 -- start start 2024-03-31T15:13:19.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.866+0000 7ff767400640 1 -- 172.21.15.16:0/3836821108 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 5795+0+0 (secure 0 0 0) 0x7ff758086e00 con 0x7ff7700d0e70 2024-03-31T15:13:19.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.866+0000 7ff76f400640 1 --2- 172.21.15.16:0/3836821108 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7ff74c068380 0x7ff74c06a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:19.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.866+0000 7ff779073000 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5647cdea7840 0x5647cdea7c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:19.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.866+0000 7ff779073000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5647cdea8160 0x5647cde5ac90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:19.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.866+0000 7ff779073000 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5647cde5b1d0 0x5647cde5b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:19.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.866+0000 7ff779073000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5647cdd45180 con 0x5647cdea8160 2024-03-31T15:13:19.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.866+0000 7ff779073000 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x5647cdcacc90 con 0x5647cde5b1d0 2024-03-31T15:13:19.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.866+0000 7ff779073000 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x5647cdc49a00 con 0x5647cdea7840 2024-03-31T15:13:19.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.866+0000 7ff766000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5647cdea8160 0x5647cde5ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:19.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.866+0000 7ff766000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5647cdea8160 0x5647cde5ac90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35428/0 (socket says 172.21.15.16:35428) 2024-03-31T15:13:19.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.866+0000 7ff766000640 1 -- 172.21.15.16:0/2523905899 learned_addr learned my addr 172.21.15.16:0/2523905899 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:13:19.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.866+0000 7ff76ea00640 1 --2- 172.21.15.16:0/2523905899 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5647cdea7840 0x5647cdea7c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:19.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.866+0000 7ff76f400640 1 --2- 172.21.15.16:0/2523905899 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5647cde5b1d0 0x5647cde5b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:19.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.866+0000 7ff76f400640 1 --2- 172.21.15.16:0/3836821108 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7ff74c068380 0x7ff74c06a840 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7ff768007fd0 tx=0x7ff768009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:13:19.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.866+0000 7ff766000640 1 -- 172.21.15.16:0/2523905899 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5647cdea7840 msgr2=0x5647cdea7c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:13:19.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.866+0000 7ff766000640 1 --2- 172.21.15.16:0/2523905899 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5647cdea7840 0x5647cdea7c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:19.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.866+0000 7ff766000640 1 -- 172.21.15.16:0/2523905899 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5647cde5b1d0 msgr2=0x5647cde5b5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:13:19.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.866+0000 7ff766000640 1 --2- 172.21.15.16:0/2523905899 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5647cde5b1d0 0x5647cde5b5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:19.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.866+0000 7ff766000640 1 -- 172.21.15.16:0/2523905899 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5647cdc32510 con 0x5647cdea8160 2024-03-31T15:13:19.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.866+0000 7ff766000640 1 --2- 172.21.15.16:0/2523905899 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5647cdea8160 0x5647cde5ac90 secure :-1 s=READY pgs=1539 cs=0 l=1 rev1=1 crypto rx=0x7ff75805f330 tx=0x7ff758056e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b99e0a3006e84d76 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:13:19.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.870+0000 7ff75f400640 1 -- 172.21.15.16:0/2523905899 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff758021040 con 0x5647cdea8160 2024-03-31T15:13:19.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.870+0000 7ff75f400640 1 -- 172.21.15.16:0/2523905899 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff758054dd0 con 0x5647cdea8160 2024-03-31T15:13:19.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.870+0000 7ff75f400640 1 -- 172.21.15.16:0/2523905899 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7580533a0 con 0x5647cdea8160 2024-03-31T15:13:19.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.870+0000 7ff779073000 1 -- 172.21.15.16:0/2523905899 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5647cdea8160 msgr2=0x5647cde5ac90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:13:19.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.870+0000 7ff779073000 1 --2- 172.21.15.16:0/2523905899 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5647cdea8160 0x5647cde5ac90 secure :-1 s=READY pgs=1539 cs=0 l=1 rev1=1 crypto rx=0x7ff75805f330 tx=0x7ff758056e70 comp rx=0 tx=0).stop 2024-03-31T15:13:19.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.870+0000 7ff779073000 1 -- 172.21.15.16:0/2523905899 shutdown_connections 2024-03-31T15:13:19.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.870+0000 7ff779073000 1 --2- 172.21.15.16:0/2523905899 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5647cde5b1d0 0x5647cde5b5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:19.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.870+0000 7ff779073000 1 --2- 172.21.15.16:0/2523905899 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5647cdea8160 0x5647cde5ac90 unknown :-1 s=CLOSED pgs=1539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:19.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.870+0000 7ff779073000 1 --2- 172.21.15.16:0/2523905899 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5647cdea7840 0x5647cdea7c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:19.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.870+0000 7ff779073000 1 -- 172.21.15.16:0/2523905899 >> 172.21.15.16:0/2523905899 conn(0x5647cde885e0 msgr2=0x5647cde8a9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:13:19.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.870+0000 7ff779073000 1 -- 172.21.15.16:0/2523905899 shutdown_connections 2024-03-31T15:13:19.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.870+0000 7ff779073000 1 -- 172.21.15.16:0/2523905899 wait complete. 2024-03-31T15:13:19.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.870+0000 7ff779073000 1 -- start start 2024-03-31T15:13:19.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.870+0000 7ff779073000 1 Processor -- start 2024-03-31T15:13:19.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.870+0000 7ff779073000 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5647cde5b1d0 0x5647cdea6fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:19.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.870+0000 7ff779073000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5647cdea7840 0x5647cde877e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:19.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.870+0000 7ff779073000 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5647cdea8160 0x5647cde87d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:19.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.870+0000 7ff779073000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5647cdc6a200 con 0x5647cdea7840 2024-03-31T15:13:19.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.870+0000 7ff779073000 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x5647cdca8c40 con 0x5647cdea8160 2024-03-31T15:13:19.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.870+0000 7ff779073000 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x5647cdc78b20 con 0x5647cde5b1d0 2024-03-31T15:13:19.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.870+0000 7ff779073000 10 client.0 _ll_register_callbacks cb 0x5647cde48930 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T15:13:19.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.870+0000 7ff766000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5647cdea7840 0x5647cde877e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:19.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.870+0000 7ff766000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5647cdea7840 0x5647cde877e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35442/0 (socket says 172.21.15.16:35442) 2024-03-31T15:13:19.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.870+0000 7ff766000640 1 -- 172.21.15.16:0/1486448346 learned_addr learned my addr 172.21.15.16:0/1486448346 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:13:19.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.870+0000 7ff76ea00640 1 --2- 172.21.15.16:0/1486448346 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5647cde5b1d0 0x5647cdea6fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:19.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.870+0000 7ff76f400640 1 --2- 172.21.15.16:0/1486448346 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5647cdea8160 0x5647cde87d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:19.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.870+0000 7ff766000640 1 -- 172.21.15.16:0/1486448346 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5647cde5b1d0 msgr2=0x5647cdea6fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:13:19.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.870+0000 7ff766000640 1 --2- 172.21.15.16:0/1486448346 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5647cde5b1d0 0x5647cdea6fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:19.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.870+0000 7ff766000640 1 -- 172.21.15.16:0/1486448346 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5647cdea8160 msgr2=0x5647cde87d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:13:19.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.870+0000 7ff766000640 1 --2- 172.21.15.16:0/1486448346 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5647cdea8160 0x5647cde87d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:19.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.870+0000 7ff766000640 1 -- 172.21.15.16:0/1486448346 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5647cdcb7ce0 con 0x5647cdea7840 2024-03-31T15:13:19.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.870+0000 7ff766000640 1 --2- 172.21.15.16:0/1486448346 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5647cdea7840 0x5647cde877e0 secure :-1 s=READY pgs=1540 cs=0 l=1 rev1=1 crypto rx=0x7ff758053190 tx=0x7ff758000bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae04c3823d6ec944 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:13:19.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.870+0000 7ff753e00640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-31T15:13:19.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.874+0000 7ff753e00640 1 -- 172.21.15.16:0/1486448346 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff758021070 con 0x5647cdea7840 2024-03-31T15:13:19.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.874+0000 7ff779073000 1 -- 172.21.15.16:0/1486448346 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff758054dd0 con 0x5647cdea7840 2024-03-31T15:13:19.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.874+0000 7ff753e00640 1 -- 172.21.15.16:0/1486448346 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 5795+0+0 (secure 0 0 0) 0x7ff75807eda0 con 0x5647cdea7840 2024-03-31T15:13:19.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.874+0000 7ff753e00640 1 -- 172.21.15.16:0/1486448346 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=89) ==== 24+0+0 (secure 0 0 0) 0x7ff758056180 con 0x5647cdea7840 2024-03-31T15:13:19.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.874+0000 7ff779073000 10 client.8967 fetch_fsmap learned FSMap version 89 2024-03-31T15:13:19.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.874+0000 7ff779073000 1 -- 172.21.15.16:0/1486448346 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=89,monmap=2+}) -- 0x5647cdc687c0 con 0x5647cdea7840 2024-03-31T15:13:19.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.874+0000 7ff753e00640 1 -- 172.21.15.16:0/1486448346 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff758054dd0 con 0x5647cdea7840 2024-03-31T15:13:19.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.874+0000 7ff753e00640 1 -- 172.21.15.16:0/1486448346 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 89) ==== 3494+0+0 (secure 0 0 0) 0x7ff758056320 con 0x5647cdea7840 2024-03-31T15:13:19.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.874+0000 7ff779073000 10 client.8967 fetch_fsmap finished waiting for FSMap version 89 2024-03-31T15:13:19.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.874+0000 7ff779073000 10 client.8967 resolve_mds: resolved 7:0 to role '7:0' aka daemon mds.a 2024-03-31T15:13:19.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.874+0000 7ff779073000 20 client.8967 populate_metadata read hostname 'smithi016' 2024-03-31T15:13:19.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.874+0000 7ff779073000 1 --2- 172.21.15.16:0/1486448346 >> [v2:172.21.15.16:6833/567808781,v1:172.21.15.16:6835/567808781] conn(0x5647cde61300 0x5647cdeb9b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:19.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.874+0000 7ff779073000 4 client.8967 mds_command: new command op to 8874 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-31T15:13:19.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.874+0000 7ff779073000 1 -- 172.21.15.16:0/1486448346 --> [v2:172.21.15.16:6833/567808781,v1:172.21.15.16:6835/567808781] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5647cdc47c20 con 0x5647cde61300 2024-03-31T15:13:19.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.874+0000 7ff76ea00640 1 --2- 172.21.15.16:0/1486448346 >> [v2:172.21.15.16:6833/567808781,v1:172.21.15.16:6835/567808781] conn(0x5647cde61300 0x5647cdeb9b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:19.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.874+0000 7ff76ea00640 1 --2- 172.21.15.16:0/1486448346 >> [v2:172.21.15.16:6833/567808781,v1:172.21.15.16:6835/567808781] conn(0x5647cde61300 0x5647cdeb9b00 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1ca215ad37f26bad server_cookie=917fa58b609e9d9d in_seq=0 out_seq=0 2024-03-31T15:13:19.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.874+0000 7ff753e00640 10 client.8967 ms_handle_connect on v2:172.21.15.16:6833/567808781 2024-03-31T15:13:19.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.874+0000 7ff753e00640 1 -- 172.21.15.16:0/1486448346 <== mds.0 v2:172.21.15.16:6833/567808781 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5647cdc47c20 con 0x5647cde61300 2024-03-31T15:13:19.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.874+0000 7ff753e00640 10 client.8967 handle_command_reply: tid=0 2024-03-31T15:13:19.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.874+0000 7ff779073000 1 client.8967 shutdown 2024-03-31T15:13:19.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.874+0000 7ff779073000 1 -- 172.21.15.16:0/1486448346 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5647cdea7840 msgr2=0x5647cde877e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:13:19.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.874+0000 7ff779073000 1 --2- 172.21.15.16:0/1486448346 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5647cdea7840 0x5647cde877e0 secure :-1 s=READY pgs=1540 cs=0 l=1 rev1=1 crypto rx=0x7ff758053190 tx=0x7ff758000bd0 comp rx=0 tx=0).stop 2024-03-31T15:13:19.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.874+0000 7ff779073000 1 -- 172.21.15.16:0/1486448346 shutdown_connections 2024-03-31T15:13:19.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.874+0000 7ff779073000 1 --2- 172.21.15.16:0/1486448346 >> [v2:172.21.15.16:6833/567808781,v1:172.21.15.16:6835/567808781] conn(0x5647cde61300 0x5647cdeb9b00 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:19.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.874+0000 7ff779073000 1 --2- 172.21.15.16:0/1486448346 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5647cdea8160 0x5647cde87d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:19.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.874+0000 7ff779073000 1 --2- 172.21.15.16:0/1486448346 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5647cdea7840 0x5647cde877e0 unknown :-1 s=CLOSED pgs=1540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:19.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.874+0000 7ff753e00640 0 client.8967 ms_handle_reset on v2:172.21.15.16:6833/567808781 2024-03-31T15:13:19.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.874+0000 7ff779073000 1 --2- 172.21.15.16:0/1486448346 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5647cde5b1d0 0x5647cdea6fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:19.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.874+0000 7ff779073000 1 -- 172.21.15.16:0/1486448346 >> 172.21.15.16:0/1486448346 conn(0x5647cde885e0 msgr2=0x5647cde5a390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:13:19.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.874+0000 7ff779073000 1 -- 172.21.15.16:0/1486448346 shutdown_connections 2024-03-31T15:13:19.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.874+0000 7ff779073000 1 -- 172.21.15.16:0/1486448346 wait complete. 2024-03-31T15:13:19.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.874+0000 7ff779073000 20 client.8967 trim_cache size 0 max 16384 2024-03-31T15:13:19.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.878+0000 7fb788600640 1 Processor -- start 2024-03-31T15:13:19.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.878+0000 7fb788600640 1 -- start start 2024-03-31T15:13:19.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.882+0000 7fb788600640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb7800ce470 0x7fb780160830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:19.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.882+0000 7fb788600640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb7800cca70 con 0x7fb78015c440 2024-03-31T15:13:19.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.882+0000 7fb788600640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb7800ccd50 con 0x7fb7800cfda0 2024-03-31T15:13:19.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.882+0000 7fb788600640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fb7800cc820 con 0x7fb7800ce470 2024-03-31T15:13:19.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.882+0000 7fb785c00640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fb78015c440 0x7fb78015d220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:33944/0 (socket says 172.21.15.16:33944) 2024-03-31T15:13:19.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.882+0000 7fb785c00640 1 -- 172.21.15.16:0/66727744 learned_addr learned my addr 172.21.15.16:0/66727744 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:13:19.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.882+0000 7fb785200640 1 --2- 172.21.15.16:0/66727744 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb7800ce470 0x7fb780160830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:19.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.882+0000 7fb785200640 1 -- 172.21.15.16:0/66727744 >> v1:172.21.15.89:6789/0 conn(0x7fb7800cfda0 legacy=0x7fb7800d1410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:13:19.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.882+0000 7fb785200640 1 -- 172.21.15.16:0/66727744 >> v1:172.21.15.16:6789/0 conn(0x7fb78015c440 legacy=0x7fb78015d220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:13:19.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.882+0000 7fb785200640 1 -- 172.21.15.16:0/66727744 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb78015ca20 con 0x7fb7800ce470 2024-03-31T15:13:19.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.882+0000 7fb77fe00640 1 -- 172.21.15.16:0/66727744 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 314544369 0 0) 0x7fb7800cca70 con 0x7fb78015c440 2024-03-31T15:13:19.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.886+0000 7fb785200640 1 --2- 172.21.15.16:0/66727744 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb7800ce470 0x7fb780160830 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fb76c0097f0 tx=0x7fb76c01c4e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4deff23f5053db66 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:13:19.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.886+0000 7fb77fe00640 1 -- 172.21.15.16:0/66727744 <== mon.2 v2:172.21.15.89:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb76c0b0070 con 0x7fb7800ce470 2024-03-31T15:13:19.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.886+0000 7fb77fe00640 1 -- 172.21.15.16:0/66727744 <== mon.2 v2:172.21.15.89:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb76c0ab600 con 0x7fb7800ce470 2024-03-31T15:13:19.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.886+0000 7fb77fe00640 1 -- 172.21.15.16:0/66727744 <== mon.2 v2:172.21.15.89:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb76c0ab9d0 con 0x7fb7800ce470 2024-03-31T15:13:19.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.886+0000 7fb788600640 1 -- 172.21.15.16:0/66727744 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb7800ce470 msgr2=0x7fb780160830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:13:19.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.886+0000 7fb788600640 1 --2- 172.21.15.16:0/66727744 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb7800ce470 0x7fb780160830 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fb76c0097f0 tx=0x7fb76c01c4e0 comp rx=0 tx=0).stop 2024-03-31T15:13:19.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.886+0000 7fb788600640 1 -- 172.21.15.16:0/66727744 shutdown_connections 2024-03-31T15:13:19.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.886+0000 7fb788600640 1 --2- 172.21.15.16:0/66727744 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb7800ce470 0x7fb780160830 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:19.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.886+0000 7fb788600640 1 -- 172.21.15.16:0/66727744 >> 172.21.15.16:0/66727744 conn(0x7fb780067330 msgr2=0x7fb780067740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:13:19.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.886+0000 7fb788600640 1 -- 172.21.15.16:0/66727744 shutdown_connections 2024-03-31T15:13:19.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.886+0000 7fb788600640 1 -- 172.21.15.16:0/66727744 wait complete. 2024-03-31T15:13:19.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.890+0000 7fb788600640 1 Processor -- start 2024-03-31T15:13:19.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.890+0000 7fb788600640 1 -- start start 2024-03-31T15:13:19.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.890+0000 7fb788600640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb7800ce470 0x7fb780154530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:19.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.890+0000 7fb788600640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb7800cfda0 0x7fb7800cb910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:19.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.890+0000 7fb788600640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb7800cbe50 0x7fb7800c86e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:19.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.890+0000 7fb788600640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fb7800d05d0 con 0x7fb7800cbe50 2024-03-31T15:13:19.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.890+0000 7fb788600640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fb7800cd030 con 0x7fb7800cfda0 2024-03-31T15:13:19.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.890+0000 7fb788600640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fb7800d0750 con 0x7fb7800ce470 2024-03-31T15:13:19.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.890+0000 7fb786600640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb7800cbe50 0x7fb7800c86e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:19.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.890+0000 7fb786600640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb7800cbe50 0x7fb7800c86e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35450/0 (socket says 172.21.15.16:35450) 2024-03-31T15:13:19.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.890+0000 7fb786600640 1 -- 172.21.15.16:0/3659354657 learned_addr learned my addr 172.21.15.16:0/3659354657 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:13:19.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.890+0000 7fb785200640 1 --2- 172.21.15.16:0/3659354657 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb7800cfda0 0x7fb7800cb910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:19.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.890+0000 7fb785c00640 1 --2- 172.21.15.16:0/3659354657 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb7800ce470 0x7fb780154530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:19.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.890+0000 7fb786600640 1 -- 172.21.15.16:0/3659354657 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb7800ce470 msgr2=0x7fb780154530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:13:19.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.890+0000 7fb786600640 1 --2- 172.21.15.16:0/3659354657 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb7800ce470 0x7fb780154530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:19.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.890+0000 7fb786600640 1 -- 172.21.15.16:0/3659354657 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb7800cfda0 msgr2=0x7fb7800cb910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:13:19.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.890+0000 7fb786600640 1 --2- 172.21.15.16:0/3659354657 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb7800cfda0 0x7fb7800cb910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:19.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.890+0000 7fb786600640 1 -- 172.21.15.16:0/3659354657 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb780154a70 con 0x7fb7800cbe50 2024-03-31T15:13:19.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.890+0000 7fb785c00640 1 --2- 172.21.15.16:0/3659354657 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb7800ce470 0x7fb780154530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:13:19.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.890+0000 7fb786600640 1 --2- 172.21.15.16:0/3659354657 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb7800cbe50 0x7fb7800c86e0 secure :-1 s=READY pgs=1542 cs=0 l=1 rev1=1 crypto rx=0x7fb77800c7a0 tx=0x7fb77800cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:13:19.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.890+0000 7fb77d600640 1 -- 172.21.15.16:0/3659354657 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb778003b80 con 0x7fb7800cbe50 2024-03-31T15:13:19.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.890+0000 7fb77d600640 1 -- 172.21.15.16:0/3659354657 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb778003d20 con 0x7fb7800cbe50 2024-03-31T15:13:19.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.890+0000 7fb788600640 1 -- 172.21.15.16:0/3659354657 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb78015b6f0 con 0x7fb7800cbe50 2024-03-31T15:13:19.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.890+0000 7fb77d600640 1 -- 172.21.15.16:0/3659354657 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb77800aa30 con 0x7fb7800cbe50 2024-03-31T15:13:19.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.890+0000 7fb788600640 1 -- 172.21.15.16:0/3659354657 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb78015b130 con 0x7fb7800cbe50 2024-03-31T15:13:19.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.894+0000 7fb788600640 1 -- 172.21.15.16:0/3659354657 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "status"} v 0) -- 0x7fb780069950 con 0x7fb7800cbe50 2024-03-31T15:13:19.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.894+0000 7fb77d600640 1 -- 172.21.15.16:0/3659354657 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fb77800abd0 con 0x7fb7800cbe50 2024-03-31T15:13:19.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.894+0000 7fb77d600640 1 --2- 172.21.15.16:0/3659354657 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fb760068380 0x7fb76006a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:19.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.894+0000 7fb77d600640 1 -- 172.21.15.16:0/3659354657 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 5795+0+0 (secure 0 0 0) 0x7fb7780872e0 con 0x7fb7800cbe50 2024-03-31T15:13:19.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.894+0000 7fb785c00640 1 --2- 172.21.15.16:0/3659354657 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fb760068380 0x7fb76006a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:19.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.894+0000 7fb77d600640 1 -- 172.21.15.16:0/3659354657 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "status"}]=0 v0) ==== 54+0+515 (secure 0 0 0) 0x7fb780154a70 con 0x7fb7800cbe50 2024-03-31T15:13:19.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.894+0000 7fb788600640 1 -- 172.21.15.16:0/3659354657 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({log-info=0+}) -- 0x7fb78015f6f0 con 0x7fb7800cbe50 2024-03-31T15:13:19.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.898+0000 7fb77d600640 1 -- 172.21.15.16:0/3659354657 <== mon.0 v2:172.21.15.16:3300/0 7 ==== log(2 entries from seq 1044 at 2024-03-31T15:13:18.099623+0000) ==== 389+0+0 (secure 0 0 0) 0x7fb7780551d0 con 0x7fb7800cbe50 2024-03-31T15:13:19.890 INFO:teuthology.orchestra.run.smithi016.stdout: cluster: 2024-03-31T15:13:19.890 INFO:teuthology.orchestra.run.smithi016.stdout: id: 286eb5da-fb47-4550-b290-d9519c13b9a5 2024-03-31T15:13:19.890 INFO:teuthology.orchestra.run.smithi016.stdout: health: HEALTH_OK 2024-03-31T15:13:19.890 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:13:19.891 INFO:teuthology.orchestra.run.smithi016.stdout: services: 2024-03-31T15:13:19.891 INFO:teuthology.orchestra.run.smithi016.stdout: mon: 3 daemons, quorum a,b,c (age 10m) 2024-03-31T15:13:19.891 INFO:teuthology.orchestra.run.smithi016.stdout: mgr: x(active, since 10m), standbys: y 2024-03-31T15:13:19.891 INFO:teuthology.orchestra.run.smithi016.stdout: mds: 1/1 daemons up, 3 standby 2024-03-31T15:13:19.891 INFO:teuthology.orchestra.run.smithi016.stdout: osd: 8 osds: 8 up (since 9m), 8 in (since 10m) 2024-03-31T15:13:19.891 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:13:19.891 INFO:teuthology.orchestra.run.smithi016.stdout: data: 2024-03-31T15:13:19.891 INFO:teuthology.orchestra.run.smithi016.stdout: volumes: 1/1 healthy 2024-03-31T15:13:19.891 INFO:teuthology.orchestra.run.smithi016.stdout: pools: 3 pools, 97 pgs 2024-03-31T15:13:19.891 INFO:teuthology.orchestra.run.smithi016.stdout: objects: 30 objects, 19 MiB 2024-03-31T15:13:19.891 INFO:teuthology.orchestra.run.smithi016.stdout: usage: 294 MiB used, 720 GiB / 720 GiB avail 2024-03-31T15:13:19.891 INFO:teuthology.orchestra.run.smithi016.stdout: pgs: 97 active+clean 2024-03-31T15:13:19.892 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:13:19.892 INFO:teuthology.orchestra.run.smithi016.stdout: io: 2024-03-31T15:13:19.892 INFO:teuthology.orchestra.run.smithi016.stdout: client: 4.9 KiB/s rd, 102 B/s wr, 4 op/s rd, 0 op/s wr 2024-03-31T15:13:19.892 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:13:19.892 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:13:19.892 INFO:teuthology.orchestra.run.smithi016.stdout:2024-03-31T15:13:18.099623+0000 mon.a [INF] Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-31T15:13:19.892 INFO:teuthology.orchestra.run.smithi016.stdout:2024-03-31T15:13:18.099647+0000 mon.a [INF] Cluster is now healthy 2024-03-31T15:13:19.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.898+0000 7fb785c00640 1 --2- 172.21.15.16:0/3659354657 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fb760068380 0x7fb76006a840 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7fb774000f70 tx=0x7fb774007dd0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:13:19.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.898+0000 7ff779073000 1 Processor -- start 2024-03-31T15:13:19.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.898+0000 7ff779073000 1 -- start start 2024-03-31T15:13:19.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.898+0000 7ff779073000 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5647cde5b1d0 0x5647cde62d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:19.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.898+0000 7ff779073000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5647cdea7840 0x5647cde593f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:19.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.898+0000 7ff779073000 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5647cdea8160 0x5647cde59930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:19.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.898+0000 7ff779073000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7ff770203cd0 con 0x5647cdea7840 2024-03-31T15:13:19.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.898+0000 7ff779073000 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7ff770202d50 con 0x5647cde5b1d0 2024-03-31T15:13:19.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.898+0000 7ff779073000 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7ff7700b3b60 con 0x5647cdea8160 2024-03-31T15:13:19.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.898+0000 7ff766000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5647cdea7840 0x5647cde593f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:19.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.898+0000 7ff766000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5647cdea7840 0x5647cde593f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35456/0 (socket says 172.21.15.16:35456) 2024-03-31T15:13:19.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.898+0000 7ff766000640 1 -- 172.21.15.16:0/1299563333 learned_addr learned my addr 172.21.15.16:0/1299563333 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:13:19.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.898+0000 7ff76ea00640 1 --2- 172.21.15.16:0/1299563333 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5647cde5b1d0 0x5647cde62d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:19.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.898+0000 7ff76f400640 1 --2- 172.21.15.16:0/1299563333 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5647cdea8160 0x5647cde59930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:19.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.898+0000 7ff766000640 1 -- 172.21.15.16:0/1299563333 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5647cdea8160 msgr2=0x5647cde59930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:13:19.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.898+0000 7ff766000640 1 --2- 172.21.15.16:0/1299563333 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5647cdea8160 0x5647cde59930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:19.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.898+0000 7ff766000640 1 -- 172.21.15.16:0/1299563333 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5647cde5b1d0 msgr2=0x5647cde62d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:13:19.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.898+0000 7ff766000640 1 --2- 172.21.15.16:0/1299563333 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5647cde5b1d0 0x5647cde62d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:19.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.898+0000 7ff766000640 1 -- 172.21.15.16:0/1299563333 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5647cdc68c70 con 0x5647cdea7840 2024-03-31T15:13:19.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.898+0000 7ff766000640 1 --2- 172.21.15.16:0/1299563333 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5647cdea7840 0x5647cde593f0 secure :-1 s=READY pgs=1543 cs=0 l=1 rev1=1 crypto rx=0x7ff758056c70 tx=0x7ff758021000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5773917d6b144cd server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:13:19.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.898+0000 7ff75ea00640 1 -- 172.21.15.16:0/1299563333 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff758052a80 con 0x5647cdea7840 2024-03-31T15:13:19.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.902+0000 7ff75ea00640 1 -- 172.21.15.16:0/1299563333 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff758054dd0 con 0x5647cdea7840 2024-03-31T15:13:19.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.902+0000 7ff75ea00640 1 -- 172.21.15.16:0/1299563333 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff758052cf0 con 0x5647cdea7840 2024-03-31T15:13:19.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.902+0000 7ff779073000 1 -- 172.21.15.16:0/1299563333 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5647cdea7840 msgr2=0x5647cde593f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:13:19.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.902+0000 7ff779073000 1 --2- 172.21.15.16:0/1299563333 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5647cdea7840 0x5647cde593f0 secure :-1 s=READY pgs=1543 cs=0 l=1 rev1=1 crypto rx=0x7ff758056c70 tx=0x7ff758021000 comp rx=0 tx=0).stop 2024-03-31T15:13:19.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.902+0000 7ff779073000 1 -- 172.21.15.16:0/1299563333 shutdown_connections 2024-03-31T15:13:19.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.902+0000 7ff779073000 1 --2- 172.21.15.16:0/1299563333 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5647cdea8160 0x5647cde59930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:19.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.902+0000 7ff779073000 1 --2- 172.21.15.16:0/1299563333 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5647cdea7840 0x5647cde593f0 unknown :-1 s=CLOSED pgs=1543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:19.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.902+0000 7ff779073000 1 --2- 172.21.15.16:0/1299563333 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5647cde5b1d0 0x5647cde62d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:19.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.902+0000 7ff779073000 1 -- 172.21.15.16:0/1299563333 >> 172.21.15.16:0/1299563333 conn(0x5647cde885e0 msgr2=0x5647cdea6180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:13:19.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.902+0000 7ff779073000 1 -- 172.21.15.16:0/1299563333 shutdown_connections 2024-03-31T15:13:19.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.902+0000 7ff779073000 1 -- 172.21.15.16:0/1299563333 wait complete. 2024-03-31T15:13:19.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.902+0000 7ff779073000 1 -- start start 2024-03-31T15:13:19.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.902+0000 7ff779073000 1 Processor -- start 2024-03-31T15:13:19.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.902+0000 7ff779073000 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5647cde5b1d0 0x5647cde5fd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:19.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.902+0000 7ff779073000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5647cdea7840 0x5647cde60290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:19.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.902+0000 7ff779073000 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5647cdea8160 0x5647cdebab50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:19.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.902+0000 7ff779073000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7ff77015d7d0 con 0x5647cdea7840 2024-03-31T15:13:19.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.902+0000 7ff779073000 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x5647cdcabfa0 con 0x5647cde5b1d0 2024-03-31T15:13:19.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.902+0000 7ff779073000 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x5647cde5f480 con 0x5647cdea8160 2024-03-31T15:13:19.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.902+0000 7ff779073000 10 client.0 _ll_register_callbacks cb 0x7ff768017040 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T15:13:19.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.902+0000 7ff766000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5647cdea7840 0x5647cde60290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:19.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.902+0000 7ff76f400640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5647cdea8160 0x5647cdebab50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:19.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.902+0000 7ff766000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5647cdea7840 0x5647cde60290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35470/0 (socket says 172.21.15.16:35470) 2024-03-31T15:13:19.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.902+0000 7ff766000640 1 -- 172.21.15.16:0/833788527 learned_addr learned my addr 172.21.15.16:0/833788527 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:13:19.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.902+0000 7ff76ea00640 1 --2- 172.21.15.16:0/833788527 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5647cde5b1d0 0x5647cde5fd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:19.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.902+0000 7ff766000640 1 -- 172.21.15.16:0/833788527 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5647cdea8160 msgr2=0x5647cdebab50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:13:19.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.902+0000 7ff766000640 1 --2- 172.21.15.16:0/833788527 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5647cdea8160 0x5647cdebab50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:19.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.902+0000 7ff766000640 1 -- 172.21.15.16:0/833788527 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5647cde5b1d0 msgr2=0x5647cde5fd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:13:19.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.902+0000 7ff766000640 1 --2- 172.21.15.16:0/833788527 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5647cde5b1d0 0x5647cde5fd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:19.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.902+0000 7ff766000640 1 -- 172.21.15.16:0/833788527 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5647cdc49c50 con 0x5647cdea7840 2024-03-31T15:13:19.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.906+0000 7ff766000640 1 --2- 172.21.15.16:0/833788527 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5647cdea7840 0x5647cde60290 secure :-1 s=READY pgs=1544 cs=0 l=1 rev1=1 crypto rx=0x7ff758009fd0 tx=0x7ff758052fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e046359cc1d1ca4 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:13:19.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.906+0000 7ff753e00640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-31T15:13:19.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.906+0000 7ff753e00640 1 -- 172.21.15.16:0/833788527 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff758021070 con 0x5647cdea7840 2024-03-31T15:13:19.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.906+0000 7ff753e00640 1 -- 172.21.15.16:0/833788527 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 5795+0+0 (secure 0 0 0) 0x7ff758055df0 con 0x5647cdea7840 2024-03-31T15:13:19.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.906+0000 7ff779073000 1 -- 172.21.15.16:0/833788527 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff758054dd0 con 0x5647cdea7840 2024-03-31T15:13:19.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.906+0000 7ff753e00640 1 -- 172.21.15.16:0/833788527 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=89) ==== 24+0+0 (secure 0 0 0) 0x7ff758052a80 con 0x5647cdea7840 2024-03-31T15:13:19.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.906+0000 7ff779073000 10 client.8979 fetch_fsmap learned FSMap version 89 2024-03-31T15:13:19.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.906+0000 7ff779073000 1 -- 172.21.15.16:0/833788527 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=89,monmap=2+}) -- 0x5647cdc35500 con 0x5647cdea7840 2024-03-31T15:13:19.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.906+0000 7ff753e00640 1 -- 172.21.15.16:0/833788527 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff758054dd0 con 0x5647cdea7840 2024-03-31T15:13:19.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.906+0000 7ff753e00640 1 -- 172.21.15.16:0/833788527 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 89) ==== 3494+0+0 (secure 0 0 0) 0x7ff758075a50 con 0x5647cdea7840 2024-03-31T15:13:19.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.906+0000 7ff779073000 10 client.8979 fetch_fsmap finished waiting for FSMap version 89 2024-03-31T15:13:19.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.906+0000 7ff779073000 10 client.8979 resolve_mds: resolved 7:0 to role '7:0' aka daemon mds.a 2024-03-31T15:13:19.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.906+0000 7ff779073000 20 client.8979 populate_metadata read hostname 'smithi016' 2024-03-31T15:13:19.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.906+0000 7ff779073000 1 --2- 172.21.15.16:0/833788527 >> [v2:172.21.15.16:6833/567808781,v1:172.21.15.16:6835/567808781] conn(0x5647cdebb090 0x5647cdebb470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:19.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.906+0000 7ff779073000 4 client.8979 mds_command: new command op to 8874 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["repair", "recursive"]}] 2024-03-31T15:13:19.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.906+0000 7ff779073000 1 -- 172.21.15.16:0/833788527 --> [v2:172.21.15.16:6833/567808781,v1:172.21.15.16:6835/567808781] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["repair", "recursive"]}) -- 0x5647cdcba1d0 con 0x5647cdebb090 2024-03-31T15:13:19.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.906+0000 7ff76ea00640 1 --2- 172.21.15.16:0/833788527 >> [v2:172.21.15.16:6833/567808781,v1:172.21.15.16:6835/567808781] conn(0x5647cdebb090 0x5647cdebb470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:19.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.906+0000 7ff76ea00640 1 --2- 172.21.15.16:0/833788527 >> [v2:172.21.15.16:6833/567808781,v1:172.21.15.16:6835/567808781] conn(0x5647cdebb090 0x5647cdebb470 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1c03eb619a958eb8 server_cookie=7f98ef5824b14436 in_seq=0 out_seq=0 2024-03-31T15:13:19.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.906+0000 7ff753e00640 10 client.8979 ms_handle_connect on v2:172.21.15.16:6833/567808781 2024-03-31T15:13:19.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.906+0000 7ff753e00640 1 -- 172.21.15.16:0/833788527 <== mds.0 v2:172.21.15.16:6833/567808781 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x5647cdcba1d0 con 0x5647cdebb090 2024-03-31T15:13:19.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.906+0000 7ff753e00640 10 client.8979 handle_command_reply: tid=0 2024-03-31T15:13:19.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.906+0000 7ff779073000 1 client.8979 shutdown 2024-03-31T15:13:19.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.906+0000 7ff779073000 1 -- 172.21.15.16:0/833788527 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5647cdea7840 msgr2=0x5647cde60290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:13:19.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.906+0000 7ff779073000 1 --2- 172.21.15.16:0/833788527 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5647cdea7840 0x5647cde60290 secure :-1 s=READY pgs=1544 cs=0 l=1 rev1=1 crypto rx=0x7ff758009fd0 tx=0x7ff758052fd0 comp rx=0 tx=0).stop 2024-03-31T15:13:19.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.906+0000 7ff779073000 1 -- 172.21.15.16:0/833788527 shutdown_connections 2024-03-31T15:13:19.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.906+0000 7ff779073000 1 --2- 172.21.15.16:0/833788527 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5647cdea8160 0x5647cdebab50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:19.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.906+0000 7ff779073000 1 --2- 172.21.15.16:0/833788527 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5647cdea7840 0x5647cde60290 unknown :-1 s=CLOSED pgs=1544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:19.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.906+0000 7ff779073000 1 --2- 172.21.15.16:0/833788527 >> [v2:172.21.15.16:6833/567808781,v1:172.21.15.16:6835/567808781] conn(0x5647cdebb090 0x5647cdebb470 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:19.905 INFO:tasks.ceph.mds.a.smithi016.stderr:2024-03-31T15:13:19.906+0000 7f7630600640 -1 log_channel(cluster) log [ERR] : scrub: inode wrongly marked free: 0x10000000001 2024-03-31T15:13:19.905 INFO:tasks.ceph.mds.a.smithi016.stderr:2024-03-31T15:13:19.906+0000 7f7630600640 -1 log_channel(cluster) log [ERR] : inode table repaired for inode: 0x10000000001 2024-03-31T15:13:19.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.906+0000 7ff779073000 1 --2- 172.21.15.16:0/833788527 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5647cde5b1d0 0x5647cde5fd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:19.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.906+0000 7ff779073000 1 -- 172.21.15.16:0/833788527 >> 172.21.15.16:0/833788527 conn(0x5647cde885e0 msgr2=0x5647cde5f640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:13:19.906 INFO:tasks.ceph.mds.a.smithi016.stderr:2024-03-31T15:13:19.906+0000 7f7630600640 -1 log_channel(cluster) log [ERR] : scrub: inode wrongly marked free: 0x10000000002 2024-03-31T15:13:19.906 INFO:tasks.ceph.mds.a.smithi016.stderr:2024-03-31T15:13:19.906+0000 7f7630600640 -1 log_channel(cluster) log [ERR] : inode table repaired for inode: 0x10000000002 2024-03-31T15:13:19.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.906+0000 7ff753e00640 0 client.8979 ms_handle_reset on v2:172.21.15.16:6833/567808781 2024-03-31T15:13:19.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.906+0000 7ff779073000 1 -- 172.21.15.16:0/833788527 shutdown_connections 2024-03-31T15:13:19.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.910+0000 7ff779073000 1 -- 172.21.15.16:0/833788527 wait complete. 2024-03-31T15:13:19.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.910+0000 7ff779073000 20 client.8979 trim_cache size 0 max 16384 2024-03-31T15:13:19.907 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-03-31T15:13:19.907 INFO:teuthology.orchestra.run.smithi016.stdout: "return_code": 0, 2024-03-31T15:13:19.907 INFO:teuthology.orchestra.run.smithi016.stdout: "scrub_tag": "27d925c0-a099-4c41-a15a-d2a10311437d", 2024-03-31T15:13:19.908 INFO:teuthology.orchestra.run.smithi016.stdout: "mode": "asynchronous" 2024-03-31T15:13:19.908 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-03-31T15:13:19.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.910+0000 7ff775c00640 1 -- 172.21.15.16:0/3836821108 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7ff74c068380 msgr2=0x7ff74c06a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:13:19.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.910+0000 7ff775c00640 1 --2- 172.21.15.16:0/3836821108 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7ff74c068380 0x7ff74c06a840 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7ff768007fd0 tx=0x7ff768009040 comp rx=0 tx=0).stop 2024-03-31T15:13:19.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.910+0000 7ff775c00640 1 -- 172.21.15.16:0/3836821108 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff7700d0e70 msgr2=0x7ff7700b5bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:13:19.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.910+0000 7ff775c00640 1 --2- 172.21.15.16:0/3836821108 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff7700d0e70 0x7ff7700b5bf0 secure :-1 s=READY pgs=1538 cs=0 l=1 rev1=1 crypto rx=0x7ff75800e930 tx=0x7ff75800edf0 comp rx=0 tx=0).stop 2024-03-31T15:13:19.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.910+0000 7ff775c00640 1 -- 172.21.15.16:0/3836821108 shutdown_connections 2024-03-31T15:13:19.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.910+0000 7ff775c00640 1 --2- 172.21.15.16:0/3836821108 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7ff74c068380 0x7ff74c06a840 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:19.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.910+0000 7ff775c00640 1 --2- 172.21.15.16:0/3836821108 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff770147230 0x7ff77015eb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:19.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.910+0000 7ff775c00640 1 --2- 172.21.15.16:0/3836821108 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff7700d0e70 0x7ff7700b5bf0 unknown :-1 s=CLOSED pgs=1538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:19.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.910+0000 7ff775c00640 1 --2- 172.21.15.16:0/3836821108 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff7700b4300 0x7ff77015bf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:19.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.910+0000 7ff775c00640 1 -- 172.21.15.16:0/3836821108 >> 172.21.15.16:0/3836821108 conn(0x7ff77006b7f0 msgr2=0x7ff77006d6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:13:19.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.910+0000 7ff775c00640 1 -- 172.21.15.16:0/3836821108 shutdown_connections 2024-03-31T15:13:19.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:19.910+0000 7ff775c00640 1 -- 172.21.15.16:0/3836821108 wait complete. 2024-03-31T15:13:19.922 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:13:19.922 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.7:0 scrub status 2024-03-31T15:13:20.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:20.070+0000 7fa385e00640 1 Processor -- start 2024-03-31T15:13:20.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:20.070+0000 7fa385e00640 1 -- start start 2024-03-31T15:13:20.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:20.074+0000 7fa385e00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fa38006f920 0x7fa38006fd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:20.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:20.074+0000 7fa385e00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fa3800cf150 0x7fa3801419c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:20.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:20.074+0000 7fa385e00640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa380070240 con 0x7fa3800d0a60 2024-03-31T15:13:20.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:20.074+0000 7fa385e00640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7fa38013f670 con 0x7fa3800cf150 2024-03-31T15:13:20.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:20.074+0000 7fa385e00640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7fa38015ca70 con 0x7fa38006f920 2024-03-31T15:13:20.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:20.074+0000 7fa37f400640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fa3800cf150 0x7fa3801419c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:20.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:20.074+0000 7fa37f400640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fa3800cf150 0x7fa3801419c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35476/0 (socket says 172.21.15.16:35476) 2024-03-31T15:13:20.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:20.074+0000 7fa37f400640 1 -- 172.21.15.16:0/2809887535 learned_addr learned my addr 172.21.15.16:0/2809887535 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:13:20.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:20.074+0000 7fa37ea00640 1 --2- 172.21.15.16:0/2809887535 >> v2:172.21.15.89:3300/0 conn(0x7fa38006f920 0x7fa38006fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:20.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:20.074+0000 7fa37f400640 1 -- 172.21.15.16:0/2809887535 >> v2:172.21.15.89:3300/0 conn(0x7fa38006f920 msgr2=0x7fa38006fd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:13:20.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:20.074+0000 7fa37f400640 1 --2- 172.21.15.16:0/2809887535 >> v2:172.21.15.89:3300/0 conn(0x7fa38006f920 0x7fa38006fd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:20.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:20.074+0000 7fa37f400640 1 -- 172.21.15.16:0/2809887535 >> v1:172.21.15.89:6789/0 conn(0x7fa3800d0a60 legacy=0x7fa3800cea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:13:20.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:20.074+0000 7fa37f400640 1 -- 172.21.15.16:0/2809887535 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa38015cd50 con 0x7fa3800cf150 2024-03-31T15:13:20.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:20.078+0000 7fa37f400640 1 --2- 172.21.15.16:0/2809887535 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa3800cf150 0x7fa3801419c0 secure :-1 s=READY pgs=1545 cs=0 l=1 rev1=1 crypto rx=0x7fa3800708e0 tx=0x7fa37801c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10b8236b0f5c9fa server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:13:20.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:20.078+0000 7fa37e000640 1 -- 172.21.15.16:0/2809887535 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3780a5620 con 0x7fa3800cf150 2024-03-31T15:13:20.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:20.078+0000 7fa37e000640 1 -- 172.21.15.16:0/2809887535 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa3780a57c0 con 0x7fa3800cf150 2024-03-31T15:13:20.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:20.078+0000 7fa37e000640 1 -- 172.21.15.16:0/2809887535 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3780a5b90 con 0x7fa3800cf150 2024-03-31T15:13:20.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:20.078+0000 7fa385e00640 1 -- 172.21.15.16:0/2809887535 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa3800cf150 msgr2=0x7fa3801419c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:13:20.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:20.078+0000 7fa385e00640 1 --2- 172.21.15.16:0/2809887535 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa3800cf150 0x7fa3801419c0 secure :-1 s=READY pgs=1545 cs=0 l=1 rev1=1 crypto rx=0x7fa3800708e0 tx=0x7fa37801c960 comp rx=0 tx=0).stop 2024-03-31T15:13:20.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:20.078+0000 7fa385e00640 1 -- 172.21.15.16:0/2809887535 shutdown_connections 2024-03-31T15:13:20.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:20.078+0000 7fa385e00640 1 --2- 172.21.15.16:0/2809887535 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa3800cf150 0x7fa3801419c0 unknown :-1 s=CLOSED pgs=1545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:20.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:20.078+0000 7fa385e00640 1 --2- 172.21.15.16:0/2809887535 >> v2:172.21.15.89:3300/0 conn(0x7fa38006f920 0x7fa38006fd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:20.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:20.078+0000 7fa385e00640 1 -- 172.21.15.16:0/2809887535 >> 172.21.15.16:0/2809887535 conn(0x7fa380067930 msgr2=0x7fa3800b3ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:13:20.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:20.078+0000 7fa385e00640 1 -- 172.21.15.16:0/2809887535 shutdown_connections 2024-03-31T15:13:20.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:20.078+0000 7fa385e00640 1 -- 172.21.15.16:0/2809887535 wait complete. 2024-03-31T15:13:20.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:20.078+0000 7fa385e00640 1 Processor -- start 2024-03-31T15:13:20.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:20.078+0000 7fa385e00640 1 -- start start 2024-03-31T15:13:20.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:20.082+0000 7fa385e00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa38006f920 0x7fa380157040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:20.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:20.082+0000 7fa385e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa3800cf150 0x7fa380158980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:20.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:20.082+0000 7fa385e00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fa3800d0a60 0x7fa380158ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:20.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:20.082+0000 7fa385e00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fa380169230 con 0x7fa3800cf150 2024-03-31T15:13:20.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:20.082+0000 7fa385e00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fa3801690b0 con 0x7fa38006f920 2024-03-31T15:13:20.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:20.082+0000 7fa385e00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fa3801693b0 con 0x7fa3800d0a60 2024-03-31T15:13:20.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:20.082+0000 7fa377e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa3800cf150 0x7fa380158980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:20.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:20.082+0000 7fa37ea00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa38006f920 0x7fa380157040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:20.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:20.082+0000 7fa377e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa3800cf150 0x7fa380158980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35478/0 (socket says 172.21.15.16:35478) 2024-03-31T15:13:20.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:20.082+0000 7fa377e00640 1 -- 172.21.15.16:0/270148185 learned_addr learned my addr 172.21.15.16:0/270148185 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:13:20.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:20.082+0000 7fa37f400640 1 --2- 172.21.15.16:0/270148185 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fa3800d0a60 0x7fa380158ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:20.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:20.082+0000 7fa377e00640 1 -- 172.21.15.16:0/270148185 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fa3800d0a60 msgr2=0x7fa380158ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:13:20.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:20.082+0000 7fa377e00640 1 --2- 172.21.15.16:0/270148185 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fa3800d0a60 0x7fa380158ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:20.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:20.082+0000 7fa377e00640 1 -- 172.21.15.16:0/270148185 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa38006f920 msgr2=0x7fa380157040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:13:20.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:20.082+0000 7fa377e00640 1 --2- 172.21.15.16:0/270148185 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa38006f920 0x7fa380157040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:20.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:20.082+0000 7fa377e00640 1 -- 172.21.15.16:0/270148185 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa3800b3060 con 0x7fa3800cf150 2024-03-31T15:13:20.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:20.082+0000 7fa377e00640 1 --2- 172.21.15.16:0/270148185 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa3800cf150 0x7fa380158980 secure :-1 s=READY pgs=1546 cs=0 l=1 rev1=1 crypto rx=0x7fa36800c340 tx=0x7fa36800c800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:13:20.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:20.082+0000 7fa377400640 1 -- 172.21.15.16:0/270148185 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3680052d0 con 0x7fa3800cf150 2024-03-31T15:13:20.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:20.082+0000 7fa377400640 1 -- 172.21.15.16:0/270148185 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa368005470 con 0x7fa3800cf150 2024-03-31T15:13:20.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:20.082+0000 7fa377400640 1 -- 172.21.15.16:0/270148185 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa368007260 con 0x7fa3800cf150 2024-03-31T15:13:20.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:20.082+0000 7fa385e00640 1 -- 172.21.15.16:0/270148185 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa3800b2aa0 con 0x7fa3800cf150 2024-03-31T15:13:20.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:20.082+0000 7fa385e00640 1 -- 172.21.15.16:0/270148185 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa38015d980 con 0x7fa3800cf150 2024-03-31T15:13:20.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:20.086+0000 7fa377400640 1 -- 172.21.15.16:0/270148185 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fa368007400 con 0x7fa3800cf150 2024-03-31T15:13:20.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:20.086+0000 7fa389356000 1 Processor -- start 2024-03-31T15:13:20.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:20.086+0000 7fa389356000 1 -- start start 2024-03-31T15:13:20.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:20.086+0000 7fa377400640 1 --2- 172.21.15.16:0/270148185 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fa35c0683d0 0x7fa35c06a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:20.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:20.086+0000 7fa377400640 1 -- 172.21.15.16:0/270148185 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 5795+0+0 (secure 0 0 0) 0x7fa368085d30 con 0x7fa3800cf150 2024-03-31T15:13:20.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:20.086+0000 7fa389356000 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5558c4bfc2d0 0x5558c4bfc6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:20.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:20.086+0000 7fa389356000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5558c4bfcbf0 0x5558c4baf470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:20.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:20.086+0000 7fa389356000 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5558c4baf9b0 0x5558c4bafd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:20.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:20.086+0000 7fa389356000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5558c4a98af0 con 0x5558c4bfcbf0 2024-03-31T15:13:20.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:20.086+0000 7fa389356000 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x5558c4a01b00 con 0x5558c4baf9b0 2024-03-31T15:13:20.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:20.086+0000 7fa389356000 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x5558c499e940 con 0x5558c4bfc2d0 2024-03-31T15:13:20.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:20.086+0000 7fa37f400640 1 --2- 172.21.15.16:0/270148185 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fa35c0683d0 0x7fa35c06a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:20.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:20.086+0000 7fa377e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5558c4bfcbf0 0x5558c4baf470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:20.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:20.086+0000 7fa377e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5558c4bfcbf0 0x5558c4baf470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35482/0 (socket says 172.21.15.16:35482) 2024-03-31T15:13:20.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:20.086+0000 7fa377e00640 1 -- 172.21.15.16:0/1809359905 learned_addr learned my addr 172.21.15.16:0/1809359905 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:13:20.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:20.086+0000 7fa37ea00640 1 --2- 172.21.15.16:0/1809359905 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5558c4bfc2d0 0x5558c4bfc6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:20.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:20.086+0000 7fa37f400640 1 --2- 172.21.15.16:0/1809359905 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5558c4baf9b0 0x5558c4bafd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:20.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:20.086+0000 7fa377e00640 1 -- 172.21.15.16:0/1809359905 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5558c4bfc2d0 msgr2=0x5558c4bfc6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:13:20.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:20.086+0000 7fa377e00640 1 --2- 172.21.15.16:0/1809359905 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5558c4bfc2d0 0x5558c4bfc6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:20.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:20.086+0000 7fa377e00640 1 -- 172.21.15.16:0/1809359905 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5558c4baf9b0 msgr2=0x5558c4bafd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:13:20.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:20.086+0000 7fa377e00640 1 --2- 172.21.15.16:0/1809359905 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5558c4baf9b0 0x5558c4bafd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:20.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:20.086+0000 7fa377e00640 1 -- 172.21.15.16:0/1809359905 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5558c4987380 con 0x5558c4bfcbf0 2024-03-31T15:13:20.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:20.086+0000 7fa377e00640 1 --2- 172.21.15.16:0/1809359905 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5558c4bfcbf0 0x5558c4baf470 secure :-1 s=READY pgs=1547 cs=0 l=1 rev1=1 crypto rx=0x7fa368055c40 tx=0x7fa368055cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4612ce53e125bc server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:13:20.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:20.086+0000 7fa37f400640 1 --2- 172.21.15.16:0/270148185 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fa35c0683d0 0x7fa35c06a890 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7fa3780ae000 tx=0x7fa378004ac0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:13:20.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:20.090+0000 7fa36fe00640 1 -- 172.21.15.16:0/1809359905 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa36805e7e0 con 0x5558c4bfcbf0 2024-03-31T15:13:20.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:20.090+0000 7fa36fe00640 1 -- 172.21.15.16:0/1809359905 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa368053d50 con 0x5558c4bfcbf0 2024-03-31T15:13:20.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:20.090+0000 7fa36fe00640 1 -- 172.21.15.16:0/1809359905 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa368052320 con 0x5558c4bfcbf0 2024-03-31T15:13:20.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:20.090+0000 7fa389356000 1 -- 172.21.15.16:0/1809359905 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5558c4bfcbf0 msgr2=0x5558c4baf470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:13:20.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:20.090+0000 7fa389356000 1 --2- 172.21.15.16:0/1809359905 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5558c4bfcbf0 0x5558c4baf470 secure :-1 s=READY pgs=1547 cs=0 l=1 rev1=1 crypto rx=0x7fa368055c40 tx=0x7fa368055cb0 comp rx=0 tx=0).stop 2024-03-31T15:13:20.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:20.090+0000 7fa389356000 1 -- 172.21.15.16:0/1809359905 shutdown_connections 2024-03-31T15:13:20.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:20.090+0000 7fa389356000 1 --2- 172.21.15.16:0/1809359905 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5558c4baf9b0 0x5558c4bafd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:20.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:20.090+0000 7fa389356000 1 --2- 172.21.15.16:0/1809359905 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5558c4bfcbf0 0x5558c4baf470 unknown :-1 s=CLOSED pgs=1547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:20.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:20.090+0000 7fa389356000 1 --2- 172.21.15.16:0/1809359905 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5558c4bfc2d0 0x5558c4bfc6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:20.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:20.090+0000 7fa389356000 1 -- 172.21.15.16:0/1809359905 >> 172.21.15.16:0/1809359905 conn(0x5558c4bdd070 msgr2=0x5558c4bdf460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:13:20.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:20.090+0000 7fa389356000 1 -- 172.21.15.16:0/1809359905 shutdown_connections 2024-03-31T15:13:20.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:20.090+0000 7fa389356000 1 -- 172.21.15.16:0/1809359905 wait complete. 2024-03-31T15:13:20.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:20.090+0000 7fa389356000 1 -- start start 2024-03-31T15:13:20.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:20.090+0000 7fa389356000 1 Processor -- start 2024-03-31T15:13:20.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:20.090+0000 7fa389356000 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5558c4baf9b0 0x5558c4c03420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:20.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:20.090+0000 7fa389356000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5558c4bfc2d0 0x5558c4c03960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:20.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:20.090+0000 7fa389356000 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5558c4bfcbf0 0x5558c4bb45a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:20.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:20.090+0000 7fa389356000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5558c49bf1e0 con 0x5558c4bfc2d0 2024-03-31T15:13:20.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:20.090+0000 7fa389356000 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x5558c49fdab0 con 0x5558c4baf9b0 2024-03-31T15:13:20.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:20.090+0000 7fa389356000 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x5558c4abc250 con 0x5558c4bfcbf0 2024-03-31T15:13:20.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:20.090+0000 7fa389356000 10 client.0 _ll_register_callbacks cb 0x5558c4b9d370 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T15:13:20.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:20.090+0000 7fa377e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5558c4bfc2d0 0x5558c4c03960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:20.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:20.090+0000 7fa377e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5558c4bfc2d0 0x5558c4c03960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35484/0 (socket says 172.21.15.16:35484) 2024-03-31T15:13:20.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:20.090+0000 7fa377e00640 1 -- 172.21.15.16:0/1015951040 learned_addr learned my addr 172.21.15.16:0/1015951040 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:13:20.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:20.090+0000 7fa37f400640 1 --2- 172.21.15.16:0/1015951040 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5558c4bfcbf0 0x5558c4bb45a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:20.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:20.090+0000 7fa37ea00640 1 --2- 172.21.15.16:0/1015951040 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5558c4baf9b0 0x5558c4c03420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:20.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:20.090+0000 7fa377e00640 1 -- 172.21.15.16:0/1015951040 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5558c4bfcbf0 msgr2=0x5558c4bb45a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:13:20.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:20.090+0000 7fa377e00640 1 --2- 172.21.15.16:0/1015951040 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5558c4bfcbf0 0x5558c4bb45a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:20.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:20.090+0000 7fa377e00640 1 -- 172.21.15.16:0/1015951040 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5558c4baf9b0 msgr2=0x5558c4c03420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:13:20.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:20.090+0000 7fa377e00640 1 --2- 172.21.15.16:0/1015951040 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5558c4baf9b0 0x5558c4c03420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:20.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:20.090+0000 7fa377e00640 1 -- 172.21.15.16:0/1015951040 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5558c49bdb70 con 0x5558c4bfc2d0 2024-03-31T15:13:20.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:20.090+0000 7fa377e00640 1 --2- 172.21.15.16:0/1015951040 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5558c4bfc2d0 0x5558c4c03960 secure :-1 s=READY pgs=1548 cs=0 l=1 rev1=1 crypto rx=0x7fa368055c40 tx=0x7fa368000ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3bd96b480ebf9b80 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:13:20.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:20.090+0000 7fa36cc00640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-31T15:13:20.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:20.090+0000 7fa36cc00640 1 -- 172.21.15.16:0/1015951040 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa36807dbd0 con 0x5558c4bfc2d0 2024-03-31T15:13:20.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:20.090+0000 7fa36cc00640 1 -- 172.21.15.16:0/1015951040 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 5795+0+0 (secure 0 0 0) 0x7fa36807dd70 con 0x5558c4bfc2d0 2024-03-31T15:13:20.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:20.090+0000 7fa389356000 1 -- 172.21.15.16:0/1015951040 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa368053d50 con 0x5558c4bfc2d0 2024-03-31T15:13:20.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:20.094+0000 7fa36cc00640 1 -- 172.21.15.16:0/1015951040 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=89) ==== 24+0+0 (secure 0 0 0) 0x7fa368055100 con 0x5558c4bfc2d0 2024-03-31T15:13:20.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:20.094+0000 7fa389356000 10 client.8991 fetch_fsmap learned FSMap version 89 2024-03-31T15:13:20.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:20.094+0000 7fa389356000 1 -- 172.21.15.16:0/1015951040 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=89,monmap=2+}) -- 0x5558c499eb90 con 0x5558c4bfc2d0 2024-03-31T15:13:20.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:20.094+0000 7fa36cc00640 1 -- 172.21.15.16:0/1015951040 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa368053d50 con 0x5558c4bfc2d0 2024-03-31T15:13:20.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:20.094+0000 7fa36cc00640 1 -- 172.21.15.16:0/1015951040 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 89) ==== 3494+0+0 (secure 0 0 0) 0x7fa3680552a0 con 0x5558c4bfc2d0 2024-03-31T15:13:20.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:20.094+0000 7fa389356000 10 client.8991 fetch_fsmap finished waiting for FSMap version 89 2024-03-31T15:13:20.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:20.094+0000 7fa389356000 10 client.8991 resolve_mds: resolved 7:0 to role '7:0' aka daemon mds.a 2024-03-31T15:13:20.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:20.094+0000 7fa389356000 20 client.8991 populate_metadata read hostname 'smithi016' 2024-03-31T15:13:20.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:20.094+0000 7fa389356000 1 --2- 172.21.15.16:0/1015951040 >> [v2:172.21.15.16:6833/567808781,v1:172.21.15.16:6835/567808781] conn(0x5558c4bb5310 0x5558c4bb56f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:20.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:20.094+0000 7fa389356000 4 client.8991 mds_command: new command op to 8874 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-31T15:13:20.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:20.094+0000 7fa389356000 1 -- 172.21.15.16:0/1015951040 --> [v2:172.21.15.16:6833/567808781,v1:172.21.15.16:6835/567808781] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5558c499cdd0 con 0x5558c4bb5310 2024-03-31T15:13:20.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:20.094+0000 7fa37ea00640 1 --2- 172.21.15.16:0/1015951040 >> [v2:172.21.15.16:6833/567808781,v1:172.21.15.16:6835/567808781] conn(0x5558c4bb5310 0x5558c4bb56f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:20.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:20.094+0000 7fa37ea00640 1 --2- 172.21.15.16:0/1015951040 >> [v2:172.21.15.16:6833/567808781,v1:172.21.15.16:6835/567808781] conn(0x5558c4bb5310 0x5558c4bb56f0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c1f1ea56dadda6d server_cookie=83ead497ccad4d26 in_seq=0 out_seq=0 2024-03-31T15:13:20.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:20.094+0000 7fa36cc00640 10 client.8991 ms_handle_connect on v2:172.21.15.16:6833/567808781 2024-03-31T15:13:20.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:20.094+0000 7fa36cc00640 1 -- 172.21.15.16:0/1015951040 <== mds.0 v2:172.21.15.16:6833/567808781 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5558c499cdd0 con 0x5558c4bb5310 2024-03-31T15:13:20.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:20.094+0000 7fa36cc00640 10 client.8991 handle_command_reply: tid=0 2024-03-31T15:13:20.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:20.094+0000 7fa389356000 1 client.8991 shutdown 2024-03-31T15:13:20.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:20.094+0000 7fa389356000 1 -- 172.21.15.16:0/1015951040 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5558c4bfc2d0 msgr2=0x5558c4c03960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:13:20.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:20.094+0000 7fa389356000 1 --2- 172.21.15.16:0/1015951040 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5558c4bfc2d0 0x5558c4c03960 secure :-1 s=READY pgs=1548 cs=0 l=1 rev1=1 crypto rx=0x7fa368055c40 tx=0x7fa368000ee0 comp rx=0 tx=0).stop 2024-03-31T15:13:20.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:20.094+0000 7fa389356000 1 -- 172.21.15.16:0/1015951040 shutdown_connections 2024-03-31T15:13:20.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:20.094+0000 7fa389356000 1 --2- 172.21.15.16:0/1015951040 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5558c4bfcbf0 0x5558c4bb45a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:20.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:20.094+0000 7fa389356000 1 --2- 172.21.15.16:0/1015951040 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5558c4bfc2d0 0x5558c4c03960 unknown :-1 s=CLOSED pgs=1548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:20.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:20.094+0000 7fa389356000 1 --2- 172.21.15.16:0/1015951040 >> [v2:172.21.15.16:6833/567808781,v1:172.21.15.16:6835/567808781] conn(0x5558c4bb5310 0x5558c4bb56f0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:20.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:20.094+0000 7fa389356000 1 --2- 172.21.15.16:0/1015951040 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5558c4baf9b0 0x5558c4c03420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:20.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:20.094+0000 7fa389356000 1 -- 172.21.15.16:0/1015951040 >> 172.21.15.16:0/1015951040 conn(0x5558c4bdd070 msgr2=0x5558c4bdf460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:13:20.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:20.094+0000 7fa36cc00640 0 client.8991 ms_handle_reset on v2:172.21.15.16:6833/567808781 2024-03-31T15:13:20.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:20.094+0000 7fa389356000 1 -- 172.21.15.16:0/1015951040 shutdown_connections 2024-03-31T15:13:20.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:20.094+0000 7fa389356000 1 -- 172.21.15.16:0/1015951040 wait complete. 2024-03-31T15:13:20.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:20.094+0000 7fa389356000 20 client.8991 trim_cache size 0 max 16384 2024-03-31T15:13:20.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:20.114+0000 7fa389356000 1 Processor -- start 2024-03-31T15:13:20.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:20.114+0000 7fa389356000 1 -- start start 2024-03-31T15:13:20.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:20.114+0000 7fa389356000 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5558c4baf9b0 0x5558c4c125b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:20.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:20.114+0000 7fa389356000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5558c4bfc2d0 0x5558c4c12af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:20.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:20.114+0000 7fa389356000 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5558c4bfcbf0 0x5558c4c132f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:20.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:20.114+0000 7fa389356000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fa3800b27e0 con 0x5558c4bfc2d0 2024-03-31T15:13:20.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:20.114+0000 7fa389356000 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fa38015eea0 con 0x5558c4baf9b0 2024-03-31T15:13:20.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:20.114+0000 7fa389356000 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x5558c49cd150 con 0x5558c4bfcbf0 2024-03-31T15:13:20.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:20.114+0000 7fa377e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5558c4bfc2d0 0x5558c4c12af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:20.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:20.114+0000 7fa377e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5558c4bfc2d0 0x5558c4c12af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35486/0 (socket says 172.21.15.16:35486) 2024-03-31T15:13:20.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:20.114+0000 7fa377e00640 1 -- 172.21.15.16:0/2712237590 learned_addr learned my addr 172.21.15.16:0/2712237590 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:13:20.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:20.114+0000 7fa37ea00640 1 --2- 172.21.15.16:0/2712237590 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5558c4baf9b0 0x5558c4c125b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:20.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:20.114+0000 7fa37f400640 1 --2- 172.21.15.16:0/2712237590 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5558c4bfcbf0 0x5558c4c132f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:20.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:20.114+0000 7fa377e00640 1 -- 172.21.15.16:0/2712237590 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5558c4bfcbf0 msgr2=0x5558c4c132f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:13:20.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:20.114+0000 7fa377e00640 1 --2- 172.21.15.16:0/2712237590 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5558c4bfcbf0 0x5558c4c132f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:20.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:20.114+0000 7fa377e00640 1 -- 172.21.15.16:0/2712237590 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5558c4baf9b0 msgr2=0x5558c4c125b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:13:20.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:20.114+0000 7fa377e00640 1 --2- 172.21.15.16:0/2712237590 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5558c4baf9b0 0x5558c4c125b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:20.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:20.114+0000 7fa377e00640 1 -- 172.21.15.16:0/2712237590 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5558c4a82140 con 0x5558c4bfc2d0 2024-03-31T15:13:20.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:20.114+0000 7fa377e00640 1 --2- 172.21.15.16:0/2712237590 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5558c4bfc2d0 0x5558c4c12af0 secure :-1 s=READY pgs=1549 cs=0 l=1 rev1=1 crypto rx=0x7fa368000bf0 tx=0x7fa36807f040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b408bef784429fe7 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:13:20.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:20.114+0000 7fa36f400640 1 -- 172.21.15.16:0/2712237590 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa36805c360 con 0x5558c4bfc2d0 2024-03-31T15:13:20.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:20.114+0000 7fa36f400640 1 -- 172.21.15.16:0/2712237590 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa3680552a0 con 0x5558c4bfc2d0 2024-03-31T15:13:20.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:20.114+0000 7fa36f400640 1 -- 172.21.15.16:0/2712237590 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa36807dbd0 con 0x5558c4bfc2d0 2024-03-31T15:13:20.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:20.114+0000 7fa389356000 1 -- 172.21.15.16:0/2712237590 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5558c4bfc2d0 msgr2=0x5558c4c12af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:13:20.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:20.114+0000 7fa389356000 1 --2- 172.21.15.16:0/2712237590 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5558c4bfc2d0 0x5558c4c12af0 secure :-1 s=READY pgs=1549 cs=0 l=1 rev1=1 crypto rx=0x7fa368000bf0 tx=0x7fa36807f040 comp rx=0 tx=0).stop 2024-03-31T15:13:20.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:20.114+0000 7fa389356000 1 -- 172.21.15.16:0/2712237590 shutdown_connections 2024-03-31T15:13:20.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:20.114+0000 7fa389356000 1 --2- 172.21.15.16:0/2712237590 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5558c4bfcbf0 0x5558c4c132f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:20.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:20.114+0000 7fa389356000 1 --2- 172.21.15.16:0/2712237590 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5558c4bfc2d0 0x5558c4c12af0 unknown :-1 s=CLOSED pgs=1549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:20.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:20.114+0000 7fa389356000 1 --2- 172.21.15.16:0/2712237590 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5558c4baf9b0 0x5558c4c125b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:20.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:20.114+0000 7fa389356000 1 -- 172.21.15.16:0/2712237590 >> 172.21.15.16:0/2712237590 conn(0x5558c4bdd070 msgr2=0x5558c4badbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:13:20.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:20.114+0000 7fa389356000 1 -- 172.21.15.16:0/2712237590 shutdown_connections 2024-03-31T15:13:20.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:20.114+0000 7fa389356000 1 -- 172.21.15.16:0/2712237590 wait complete. 2024-03-31T15:13:20.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:20.118+0000 7fa389356000 1 -- start start 2024-03-31T15:13:20.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:20.118+0000 7fa389356000 1 Processor -- start 2024-03-31T15:13:20.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:20.118+0000 7fa389356000 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5558c4baf9b0 0x5558c4bfb290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:20.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:20.118+0000 7fa389356000 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5558c4bfc2d0 0x5558c4bfb7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:20.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:20.118+0000 7fa389356000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5558c4bfcbf0 0x5558c4c00030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:20.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:20.118+0000 7fa389356000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5558c4ad0420 con 0x5558c4bfcbf0 2024-03-31T15:13:20.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:20.118+0000 7fa389356000 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x5558c499c940 con 0x5558c4bfc2d0 2024-03-31T15:13:20.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:20.118+0000 7fa389356000 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x5558c49b8750 con 0x5558c4baf9b0 2024-03-31T15:13:20.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:20.118+0000 7fa389356000 10 client.0 _ll_register_callbacks cb 0x7fa36806d070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T15:13:20.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:20.118+0000 7fa37f400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5558c4bfcbf0 0x5558c4c00030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:20.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:20.118+0000 7fa37f400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5558c4bfcbf0 0x5558c4c00030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35496/0 (socket says 172.21.15.16:35496) 2024-03-31T15:13:20.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:20.118+0000 7fa37f400640 1 -- 172.21.15.16:0/1561973528 learned_addr learned my addr 172.21.15.16:0/1561973528 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:13:20.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:20.118+0000 7fa377e00640 1 --2- 172.21.15.16:0/1561973528 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5558c4bfc2d0 0x5558c4bfb7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:20.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:20.118+0000 7fa37ea00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5558c4baf9b0 0x5558c4bfb290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:20.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:20.118+0000 7fa377e00640 1 -- 172.21.15.16:0/1561973528 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5558c4baf9b0 msgr2=0x5558c4bfb290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:13:20.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:20.118+0000 7fa377e00640 1 --2- 172.21.15.16:0/1561973528 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5558c4baf9b0 0x5558c4bfb290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:20.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:20.118+0000 7fa377e00640 1 -- 172.21.15.16:0/1561973528 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5558c4bfcbf0 msgr2=0x5558c4c00030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:13:20.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:20.118+0000 7fa377e00640 1 --2- 172.21.15.16:0/1561973528 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5558c4bfcbf0 0x5558c4c00030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:20.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:20.118+0000 7fa377e00640 1 -- 172.21.15.16:0/1561973528 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5558c49901e0 con 0x5558c4bfc2d0 2024-03-31T15:13:20.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:20.118+0000 7fa37f400640 1 --2- 172.21.15.16:0/1561973528 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5558c4bfcbf0 0x5558c4c00030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:13:20.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:20.118+0000 7fa377e00640 1 --2- 172.21.15.16:0/1561973528 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5558c4bfc2d0 0x5558c4bfb7d0 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7fa36800cd00 tx=0x7fa368001130 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fcc7ed5ce045094f server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:13:20.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:20.118+0000 7fa36cc00640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-03-31T15:13:20.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:20.118+0000 7fa36cc00640 1 -- 172.21.15.16:0/1561973528 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa36807ddd0 con 0x5558c4bfc2d0 2024-03-31T15:13:20.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:20.118+0000 7fa36cc00640 1 -- 172.21.15.16:0/1561973528 <== mon.1 v2:172.21.15.89:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 5795+0+0 (secure 0 0 0) 0x7fa36807d870 con 0x5558c4bfc2d0 2024-03-31T15:13:20.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:20.118+0000 7fa389356000 1 -- 172.21.15.16:0/1561973528 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa3680552a0 con 0x5558c4bfc2d0 2024-03-31T15:13:20.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:20.118+0000 7fa36cc00640 1 -- 172.21.15.16:0/1561973528 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=89) ==== 24+0+0 (secure 0 0 0) 0x7fa368051a00 con 0x5558c4bfc2d0 2024-03-31T15:13:20.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:20.118+0000 7fa389356000 10 client.8560 fetch_fsmap learned FSMap version 89 2024-03-31T15:13:20.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:20.118+0000 7fa389356000 1 -- 172.21.15.16:0/1561973528 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=89,monmap=2+}) -- 0x5558c49957b0 con 0x5558c4bfc2d0 2024-03-31T15:13:20.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:20.118+0000 7fa36cc00640 1 -- 172.21.15.16:0/1561973528 <== mon.1 v2:172.21.15.89:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa3680552a0 con 0x5558c4bfc2d0 2024-03-31T15:13:20.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:20.118+0000 7fa36cc00640 1 -- 172.21.15.16:0/1561973528 <== mon.1 v2:172.21.15.89:3300/0 5 ==== fsmap(e 89) ==== 3494+0+0 (secure 0 0 0) 0x7fa368051ba0 con 0x5558c4bfc2d0 2024-03-31T15:13:20.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:20.118+0000 7fa389356000 10 client.8560 fetch_fsmap finished waiting for FSMap version 89 2024-03-31T15:13:20.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:20.118+0000 7fa389356000 10 client.8560 resolve_mds: resolved 7:0 to role '7:0' aka daemon mds.a 2024-03-31T15:13:20.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:20.118+0000 7fa389356000 20 client.8560 populate_metadata read hostname 'smithi016' 2024-03-31T15:13:20.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:20.118+0000 7fa389356000 1 --2- 172.21.15.16:0/1561973528 >> [v2:172.21.15.16:6833/567808781,v1:172.21.15.16:6835/567808781] conn(0x5558c4c00570 0x5558c4c00980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:20.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:20.118+0000 7fa389356000 4 client.8560 mds_command: new command op to 8874 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-31T15:13:20.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:20.118+0000 7fa389356000 1 -- 172.21.15.16:0/1561973528 --> [v2:172.21.15.16:6833/567808781,v1:172.21.15.16:6835/567808781] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5558c4a0f040 con 0x5558c4c00570 2024-03-31T15:13:20.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:20.118+0000 7fa37ea00640 1 --2- 172.21.15.16:0/1561973528 >> [v2:172.21.15.16:6833/567808781,v1:172.21.15.16:6835/567808781] conn(0x5558c4c00570 0x5558c4c00980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:20.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:20.122+0000 7fa37ea00640 1 --2- 172.21.15.16:0/1561973528 >> [v2:172.21.15.16:6833/567808781,v1:172.21.15.16:6835/567808781] conn(0x5558c4c00570 0x5558c4c00980 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6133cbe89a5296a8 server_cookie=9760c6b247a92deb in_seq=0 out_seq=0 2024-03-31T15:13:20.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:20.122+0000 7fa36cc00640 10 client.8560 ms_handle_connect on v2:172.21.15.16:6833/567808781 2024-03-31T15:13:20.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:20.122+0000 7fa36cc00640 1 -- 172.21.15.16:0/1561973528 <== mds.0 v2:172.21.15.16:6833/567808781 1 ==== command_reply(tid 0: 0 ) ==== 8+0+260 (crc 0 0 0) 0x5558c4a0f040 con 0x5558c4c00570 2024-03-31T15:13:20.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:20.122+0000 7fa36cc00640 10 client.8560 handle_command_reply: tid=0 2024-03-31T15:13:20.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:20.122+0000 7fa389356000 1 client.8560 shutdown 2024-03-31T15:13:20.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:20.122+0000 7fa389356000 1 -- 172.21.15.16:0/1561973528 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5558c4bfc2d0 msgr2=0x5558c4bfb7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:13:20.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:20.122+0000 7fa389356000 1 --2- 172.21.15.16:0/1561973528 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5558c4bfc2d0 0x5558c4bfb7d0 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7fa36800cd00 tx=0x7fa368001130 comp rx=0 tx=0).stop 2024-03-31T15:13:20.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:20.122+0000 7fa389356000 1 -- 172.21.15.16:0/1561973528 shutdown_connections 2024-03-31T15:13:20.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:20.122+0000 7fa389356000 1 --2- 172.21.15.16:0/1561973528 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5558c4bfcbf0 0x5558c4c00030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:20.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:20.122+0000 7fa389356000 1 --2- 172.21.15.16:0/1561973528 >> [v2:172.21.15.16:6833/567808781,v1:172.21.15.16:6835/567808781] conn(0x5558c4c00570 0x5558c4c00980 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:20.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:20.122+0000 7fa389356000 1 --2- 172.21.15.16:0/1561973528 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5558c4bfc2d0 0x5558c4bfb7d0 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:20.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:20.122+0000 7fa389356000 1 --2- 172.21.15.16:0/1561973528 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5558c4baf9b0 0x5558c4bfb290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:20.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:20.122+0000 7fa36cc00640 0 client.8560 ms_handle_reset on v2:172.21.15.16:6833/567808781 2024-03-31T15:13:20.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:20.122+0000 7fa389356000 1 -- 172.21.15.16:0/1561973528 >> 172.21.15.16:0/1561973528 conn(0x5558c4bdd070 msgr2=0x5558c4bb4120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:13:20.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:20.122+0000 7fa389356000 1 -- 172.21.15.16:0/1561973528 shutdown_connections 2024-03-31T15:13:20.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:20.122+0000 7fa389356000 1 -- 172.21.15.16:0/1561973528 wait complete. 2024-03-31T15:13:20.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:20.122+0000 7fa389356000 20 client.8560 trim_cache size 0 max 16384 2024-03-31T15:13:20.118 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-03-31T15:13:20.118 INFO:teuthology.orchestra.run.smithi016.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-31T15:13:20.118 INFO:teuthology.orchestra.run.smithi016.stdout: "scrubs": { 2024-03-31T15:13:20.118 INFO:teuthology.orchestra.run.smithi016.stdout: "27d925c0-a099-4c41-a15a-d2a10311437d": { 2024-03-31T15:13:20.118 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-03-31T15:13:20.118 INFO:teuthology.orchestra.run.smithi016.stdout: "tag": "27d925c0-a099-4c41-a15a-d2a10311437d", 2024-03-31T15:13:20.119 INFO:teuthology.orchestra.run.smithi016.stdout: "options": "recursive,repair" 2024-03-31T15:13:20.119 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-31T15:13:20.119 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-31T15:13:20.119 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-03-31T15:13:20.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:20.122+0000 7fa385e00640 1 -- 172.21.15.16:0/270148185 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fa35c0683d0 msgr2=0x7fa35c06a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:13:20.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:20.122+0000 7fa385e00640 1 --2- 172.21.15.16:0/270148185 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fa35c0683d0 0x7fa35c06a890 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7fa3780ae000 tx=0x7fa378004ac0 comp rx=0 tx=0).stop 2024-03-31T15:13:20.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:20.122+0000 7fa385e00640 1 -- 172.21.15.16:0/270148185 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa3800cf150 msgr2=0x7fa380158980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:13:20.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:20.122+0000 7fa385e00640 1 --2- 172.21.15.16:0/270148185 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa3800cf150 0x7fa380158980 secure :-1 s=READY pgs=1546 cs=0 l=1 rev1=1 crypto rx=0x7fa36800c340 tx=0x7fa36800c800 comp rx=0 tx=0).stop 2024-03-31T15:13:20.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:20.122+0000 7fa385e00640 1 -- 172.21.15.16:0/270148185 shutdown_connections 2024-03-31T15:13:20.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:20.122+0000 7fa385e00640 1 --2- 172.21.15.16:0/270148185 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fa35c0683d0 0x7fa35c06a890 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:20.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:20.122+0000 7fa385e00640 1 --2- 172.21.15.16:0/270148185 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fa3800d0a60 0x7fa380158ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:20.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:20.122+0000 7fa385e00640 1 --2- 172.21.15.16:0/270148185 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa3800cf150 0x7fa380158980 unknown :-1 s=CLOSED pgs=1546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:20.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:20.122+0000 7fa385e00640 1 --2- 172.21.15.16:0/270148185 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa38006f920 0x7fa380157040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:20.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:20.122+0000 7fa385e00640 1 -- 172.21.15.16:0/270148185 >> 172.21.15.16:0/270148185 conn(0x7fa380067930 msgr2=0x7fa3800d18f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:13:20.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:20.122+0000 7fa385e00640 1 -- 172.21.15.16:0/270148185 shutdown_connections 2024-03-31T15:13:20.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:20.122+0000 7fa385e00640 1 -- 172.21.15.16:0/270148185 wait complete. 2024-03-31T15:13:20.135 INFO:tasks.cephfs.filesystem:scrub status for tag:27d925c0-a099-4c41-a15a-d2a10311437d - {'path': '/', 'tag': '27d925c0-a099-4c41-a15a-d2a10311437d', 'options': 'recursive,repair'} 2024-03-31T15:13:21.079 INFO:tasks.ceph.mds.a.smithi016.stderr:2024-03-31T15:13:21.086+0000 7f7635600640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-31T15:13:21.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:21.110+0000 7fb77d600640 1 -- 172.21.15.16:0/3659354657 <== mon.0 v2:172.21.15.16:3300/0 8 ==== log(8 entries from seq 1 at 2024-03-31T15:13:19.912050+0000) ==== 1407+0+0 (secure 0 0 0) 0x7fb778055430 con 0x7fb7800cbe50 2024-03-31T15:13:21.106 INFO:teuthology.orchestra.run.smithi016.stdout:2024-03-31T15:13:19.912050+0000 mds.a [INF] scrub queued for path: / 2024-03-31T15:13:21.106 INFO:teuthology.orchestra.run.smithi016.stdout:2024-03-31T15:13:19.912056+0000 mds.a [INF] scrub summary: idle+waiting paths [/] 2024-03-31T15:13:21.106 INFO:teuthology.orchestra.run.smithi016.stdout:2024-03-31T15:13:19.912063+0000 mds.a [INF] scrub summary: active paths [/] 2024-03-31T15:13:21.106 INFO:teuthology.orchestra.run.smithi016.stdout:2024-03-31T15:13:19.913278+0000 mds.a [ERR] scrub: inode wrongly marked free: 0x10000000001 2024-03-31T15:13:21.106 INFO:teuthology.orchestra.run.smithi016.stdout:2024-03-31T15:13:19.913286+0000 mds.a [ERR] inode table repaired for inode: 0x10000000001 2024-03-31T15:13:21.106 INFO:teuthology.orchestra.run.smithi016.stdout:2024-03-31T15:13:19.913365+0000 mds.a [ERR] scrub: inode wrongly marked free: 0x10000000002 2024-03-31T15:13:21.106 INFO:teuthology.orchestra.run.smithi016.stdout:2024-03-31T15:13:19.913371+0000 mds.a [ERR] inode table repaired for inode: 0x10000000002 2024-03-31T15:13:21.107 INFO:teuthology.orchestra.run.smithi016.stdout:2024-03-31T15:13:19.913952+0000 mds.a [INF] scrub summary: idle+waiting paths [/] 2024-03-31T15:13:26.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:26.118+0000 7fb77d600640 1 -- 172.21.15.16:0/3659354657 <== mon.0 v2:172.21.15.16:3300/0 9 ==== log(1 entries from seq 9 at 2024-03-31T15:13:24.153043+0000) ==== 189+0+0 (secure 0 0 0) 0x7fb778085820 con 0x7fb7800cbe50 2024-03-31T15:13:26.113 INFO:teuthology.orchestra.run.smithi016.stdout:2024-03-31T15:13:24.153043+0000 mds.a [INF] scrub summary: idle 2024-03-31T15:13:37.778 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-31T15:13:37.781 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-31T15:13:50.137 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:13:50.137 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.7:0 scrub status 2024-03-31T15:13:50.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.286+0000 7fb23c000640 1 Processor -- start 2024-03-31T15:13:50.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.286+0000 7fb23c000640 1 -- start start 2024-03-31T15:13:50.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.290+0000 7fb23c000640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fb234145d50 0x7fb234160860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:50.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.290+0000 7fb23c000640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb234070350 0x7fb2340cea50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:50.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.290+0000 7fb23c000640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb234070a10 con 0x7fb23415c890 2024-03-31T15:13:50.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.290+0000 7fb23c000640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7fb2340ce5c0 con 0x7fb234145d50 2024-03-31T15:13:50.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.290+0000 7fb23c000640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fb2340cd9b0 con 0x7fb234070350 2024-03-31T15:13:50.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.290+0000 7fb238c00640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fb23415c890 0x7fb23406fc40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:44730/0 (socket says 172.21.15.16:44730) 2024-03-31T15:13:50.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.290+0000 7fb238c00640 1 -- 172.21.15.16:0/3878724079 learned_addr learned my addr 172.21.15.16:0/3878724079 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:13:50.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.290+0000 7fb239600640 1 --2- 172.21.15.16:0/3878724079 >> v2:172.21.15.89:3300/0 conn(0x7fb234145d50 0x7fb234160860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:50.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.290+0000 7fb23a000640 1 --2- 172.21.15.16:0/3878724079 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb234070350 0x7fb2340cea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:50.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.290+0000 7fb233e00640 1 -- 172.21.15.16:0/3878724079 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 713652875 0 0) 0x7fb234070a10 con 0x7fb23415c890 2024-03-31T15:13:50.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.290+0000 7fb233e00640 1 -- 172.21.15.16:0/3878724079 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb21c003530 con 0x7fb23415c890 2024-03-31T15:13:50.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.290+0000 7fb239600640 1 -- 172.21.15.16:0/3878724079 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb234070350 msgr2=0x7fb2340cea50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:13:50.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.290+0000 7fb239600640 1 --2- 172.21.15.16:0/3878724079 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb234070350 0x7fb2340cea50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:50.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.290+0000 7fb239600640 1 -- 172.21.15.16:0/3878724079 >> v1:172.21.15.16:6789/0 conn(0x7fb23415c890 legacy=0x7fb23406fc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:13:50.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.290+0000 7fb239600640 1 -- 172.21.15.16:0/3878724079 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2340cdc90 con 0x7fb234145d50 2024-03-31T15:13:50.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.294+0000 7fb239600640 1 --2- 172.21.15.16:0/3878724079 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb234145d50 0x7fb234160860 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7fb2280097c0 tx=0x7fb22801c470 comp rx=0 tx=0).ready entity=mon.1 client_cookie=eb0ba162e4f612da server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:13:50.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.294+0000 7fb233e00640 1 -- 172.21.15.16:0/3878724079 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2280a7070 con 0x7fb234145d50 2024-03-31T15:13:50.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.294+0000 7fb233e00640 1 -- 172.21.15.16:0/3878724079 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb2280b0070 con 0x7fb234145d50 2024-03-31T15:13:50.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.294+0000 7fb233e00640 1 -- 172.21.15.16:0/3878724079 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2280ab890 con 0x7fb234145d50 2024-03-31T15:13:50.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.294+0000 7fb23c000640 1 -- 172.21.15.16:0/3878724079 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb234145d50 msgr2=0x7fb234160860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:13:50.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.294+0000 7fb23c000640 1 --2- 172.21.15.16:0/3878724079 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb234145d50 0x7fb234160860 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7fb2280097c0 tx=0x7fb22801c470 comp rx=0 tx=0).stop 2024-03-31T15:13:50.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.294+0000 7fb23c000640 1 -- 172.21.15.16:0/3878724079 shutdown_connections 2024-03-31T15:13:50.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.294+0000 7fb23c000640 1 --2- 172.21.15.16:0/3878724079 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb234070350 0x7fb2340cea50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:50.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.294+0000 7fb23c000640 1 --2- 172.21.15.16:0/3878724079 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb234145d50 0x7fb234160860 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:50.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.294+0000 7fb23c000640 1 -- 172.21.15.16:0/3878724079 >> 172.21.15.16:0/3878724079 conn(0x7fb23406e670 msgr2=0x7fb23406ea50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:13:50.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.294+0000 7fb23c000640 1 -- 172.21.15.16:0/3878724079 shutdown_connections 2024-03-31T15:13:50.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.294+0000 7fb23c000640 1 -- 172.21.15.16:0/3878724079 wait complete. 2024-03-31T15:13:50.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.294+0000 7fb23c000640 1 Processor -- start 2024-03-31T15:13:50.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.294+0000 7fb23c000640 1 -- start start 2024-03-31T15:13:50.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.294+0000 7fb23c000640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb234070350 0x7fb2340ccab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:50.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.294+0000 7fb23c000640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb234145d50 0x7fb23414a640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:50.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.294+0000 7fb23c000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb23415c890 0x7fb23414ab80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:50.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.294+0000 7fb23c000640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fb234169520 con 0x7fb23415c890 2024-03-31T15:13:50.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.294+0000 7fb23c000640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fb2341693a0 con 0x7fb234145d50 2024-03-31T15:13:50.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.294+0000 7fb23c000640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fb2341696a0 con 0x7fb234070350 2024-03-31T15:13:50.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.294+0000 7fb23a000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb23415c890 0x7fb23414ab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:50.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.294+0000 7fb23a000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb23415c890 0x7fb23414ab80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50150/0 (socket says 172.21.15.16:50150) 2024-03-31T15:13:50.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.298+0000 7fb23a000640 1 -- 172.21.15.16:0/754120345 learned_addr learned my addr 172.21.15.16:0/754120345 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:13:50.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.298+0000 7fb239600640 1 --2- 172.21.15.16:0/754120345 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb234070350 0x7fb2340ccab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:50.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.298+0000 7fb23a000640 1 -- 172.21.15.16:0/754120345 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb234070350 msgr2=0x7fb2340ccab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:13:50.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.298+0000 7fb238c00640 1 --2- 172.21.15.16:0/754120345 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb234145d50 0x7fb23414a640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:50.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.298+0000 7fb23a000640 1 --2- 172.21.15.16:0/754120345 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb234070350 0x7fb2340ccab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:50.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.298+0000 7fb23a000640 1 -- 172.21.15.16:0/754120345 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb234145d50 msgr2=0x7fb23414a640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:13:50.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.298+0000 7fb23a000640 1 --2- 172.21.15.16:0/754120345 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb234145d50 0x7fb23414a640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:50.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.298+0000 7fb23a000640 1 -- 172.21.15.16:0/754120345 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb234144df0 con 0x7fb23415c890 2024-03-31T15:13:50.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.298+0000 7fb239600640 1 --2- 172.21.15.16:0/754120345 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb234070350 0x7fb2340ccab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:13:50.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.298+0000 7fb238c00640 1 --2- 172.21.15.16:0/754120345 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb234145d50 0x7fb23414a640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:13:50.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.298+0000 7fb23a000640 1 --2- 172.21.15.16:0/754120345 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb23415c890 0x7fb23414ab80 secure :-1 s=READY pgs=1551 cs=0 l=1 rev1=1 crypto rx=0x7fb22c00ef90 tx=0x7fb22c00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:13:50.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.298+0000 7fb231600640 1 -- 172.21.15.16:0/754120345 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb22c018020 con 0x7fb23415c890 2024-03-31T15:13:50.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.298+0000 7fb231600640 1 -- 172.21.15.16:0/754120345 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb22c014480 con 0x7fb23415c890 2024-03-31T15:13:50.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.298+0000 7fb231600640 1 -- 172.21.15.16:0/754120345 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb22c014870 con 0x7fb23415c890 2024-03-31T15:13:50.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.298+0000 7fb23c000640 1 -- 172.21.15.16:0/754120345 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb234148750 con 0x7fb23415c890 2024-03-31T15:13:50.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.298+0000 7fb23c000640 1 -- 172.21.15.16:0/754120345 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb234148d10 con 0x7fb23415c890 2024-03-31T15:13:50.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.298+0000 7fb231600640 1 -- 172.21.15.16:0/754120345 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fb22c01b020 con 0x7fb23415c890 2024-03-31T15:13:50.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.302+0000 7fb231600640 1 --2- 172.21.15.16:0/754120345 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fb214068380 0x7fb21406a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:50.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.302+0000 7fb231600640 1 -- 172.21.15.16:0/754120345 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 5795+0+0 (secure 0 0 0) 0x7fb22c087960 con 0x7fb23415c890 2024-03-31T15:13:50.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.302+0000 7fb239600640 1 --2- 172.21.15.16:0/754120345 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fb214068380 0x7fb21406a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:50.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.302+0000 7fb23f3de000 1 Processor -- start 2024-03-31T15:13:50.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.302+0000 7fb23f3de000 1 -- start start 2024-03-31T15:13:50.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.302+0000 7fb239600640 1 --2- 172.21.15.16:0/754120345 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fb214068380 0x7fb21406a840 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7fb2280af000 tx=0x7fb228008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:13:50.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.302+0000 7fb23f3de000 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5653a5f192d0 0x5653a5f196b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:50.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.302+0000 7fb23f3de000 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5653a5f19bf0 0x5653a5ecc470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:50.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.302+0000 7fb23f3de000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5653a5ecc9b0 0x5653a5eccd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:50.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.302+0000 7fb23f3de000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5653a5db5af0 con 0x5653a5ecc9b0 2024-03-31T15:13:50.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.302+0000 7fb23f3de000 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x5653a5d1eb00 con 0x5653a5f19bf0 2024-03-31T15:13:50.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.302+0000 7fb23f3de000 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x5653a5cbb940 con 0x5653a5f192d0 2024-03-31T15:13:50.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.302+0000 7fb23a000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5653a5ecc9b0 0x5653a5eccd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:50.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.302+0000 7fb23a000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5653a5ecc9b0 0x5653a5eccd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50152/0 (socket says 172.21.15.16:50152) 2024-03-31T15:13:50.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.302+0000 7fb23a000640 1 -- 172.21.15.16:0/1321025688 learned_addr learned my addr 172.21.15.16:0/1321025688 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:13:50.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.302+0000 7fb239600640 1 --2- 172.21.15.16:0/1321025688 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5653a5f192d0 0x5653a5f196b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:50.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.302+0000 7fb238c00640 1 --2- 172.21.15.16:0/1321025688 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5653a5f19bf0 0x5653a5ecc470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:50.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.302+0000 7fb23a000640 1 -- 172.21.15.16:0/1321025688 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5653a5f192d0 msgr2=0x5653a5f196b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:13:50.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.302+0000 7fb23a000640 1 --2- 172.21.15.16:0/1321025688 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5653a5f192d0 0x5653a5f196b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:50.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.302+0000 7fb23a000640 1 -- 172.21.15.16:0/1321025688 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5653a5f19bf0 msgr2=0x5653a5ecc470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:13:50.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.302+0000 7fb23a000640 1 --2- 172.21.15.16:0/1321025688 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5653a5f19bf0 0x5653a5ecc470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:50.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.302+0000 7fb23a000640 1 -- 172.21.15.16:0/1321025688 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5653a5ca4380 con 0x5653a5ecc9b0 2024-03-31T15:13:50.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.302+0000 7fb238c00640 1 --2- 172.21.15.16:0/1321025688 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5653a5f19bf0 0x5653a5ecc470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:13:50.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.302+0000 7fb23a000640 1 --2- 172.21.15.16:0/1321025688 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5653a5ecc9b0 0x5653a5eccd90 secure :-1 s=READY pgs=1552 cs=0 l=1 rev1=1 crypto rx=0x7fb22c00c990 tx=0x7fb22c057ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55c175764908f651 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:13:50.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.302+0000 7fb226000640 1 -- 172.21.15.16:0/1321025688 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb22c084070 con 0x5653a5ecc9b0 2024-03-31T15:13:50.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.302+0000 7fb226000640 1 -- 172.21.15.16:0/1321025688 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb22c060300 con 0x5653a5ecc9b0 2024-03-31T15:13:50.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.302+0000 7fb226000640 1 -- 172.21.15.16:0/1321025688 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb22c055a20 con 0x5653a5ecc9b0 2024-03-31T15:13:50.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.302+0000 7fb23f3de000 1 -- 172.21.15.16:0/1321025688 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5653a5ecc9b0 msgr2=0x5653a5eccd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:13:50.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.302+0000 7fb23f3de000 1 --2- 172.21.15.16:0/1321025688 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5653a5ecc9b0 0x5653a5eccd90 secure :-1 s=READY pgs=1552 cs=0 l=1 rev1=1 crypto rx=0x7fb22c00c990 tx=0x7fb22c057ee0 comp rx=0 tx=0).stop 2024-03-31T15:13:50.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.302+0000 7fb23f3de000 1 -- 172.21.15.16:0/1321025688 shutdown_connections 2024-03-31T15:13:50.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.302+0000 7fb23f3de000 1 --2- 172.21.15.16:0/1321025688 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5653a5ecc9b0 0x5653a5eccd90 unknown :-1 s=CLOSED pgs=1552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:50.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.302+0000 7fb23f3de000 1 --2- 172.21.15.16:0/1321025688 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5653a5f19bf0 0x5653a5ecc470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:50.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.302+0000 7fb23f3de000 1 --2- 172.21.15.16:0/1321025688 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5653a5f192d0 0x5653a5f196b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:50.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.302+0000 7fb23f3de000 1 -- 172.21.15.16:0/1321025688 >> 172.21.15.16:0/1321025688 conn(0x5653a5efa070 msgr2=0x5653a5efc460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:13:50.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.302+0000 7fb23f3de000 1 -- 172.21.15.16:0/1321025688 shutdown_connections 2024-03-31T15:13:50.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.302+0000 7fb23f3de000 1 -- 172.21.15.16:0/1321025688 wait complete. 2024-03-31T15:13:50.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.306+0000 7fb23f3de000 1 -- start start 2024-03-31T15:13:50.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.306+0000 7fb23f3de000 1 Processor -- start 2024-03-31T15:13:50.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.306+0000 7fb23f3de000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5653a5ecc9b0 0x5653a5f20630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:50.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.306+0000 7fb23f3de000 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5653a5f192d0 0x5653a5f20b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:50.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.306+0000 7fb23f3de000 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5653a5f19bf0 0x5653a5ed1260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:50.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.306+0000 7fb23f3de000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5653a5cdc1e0 con 0x5653a5ecc9b0 2024-03-31T15:13:50.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.306+0000 7fb23f3de000 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x5653a5d1aab0 con 0x5653a5f192d0 2024-03-31T15:13:50.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.306+0000 7fb23f3de000 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x5653a5cea150 con 0x5653a5f19bf0 2024-03-31T15:13:50.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.306+0000 7fb23f3de000 10 client.0 _ll_register_callbacks cb 0x5653a5eba370 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T15:13:50.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.306+0000 7fb239600640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5653a5ecc9b0 0x5653a5f20630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:50.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.306+0000 7fb239600640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5653a5ecc9b0 0x5653a5f20630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50166/0 (socket says 172.21.15.16:50166) 2024-03-31T15:13:50.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.306+0000 7fb239600640 1 -- 172.21.15.16:0/2213370449 learned_addr learned my addr 172.21.15.16:0/2213370449 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:13:50.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.306+0000 7fb23a000640 1 --2- 172.21.15.16:0/2213370449 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5653a5f19bf0 0x5653a5ed1260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:50.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.306+0000 7fb238c00640 1 --2- 172.21.15.16:0/2213370449 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5653a5f192d0 0x5653a5f20b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:50.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.306+0000 7fb239600640 1 -- 172.21.15.16:0/2213370449 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5653a5f19bf0 msgr2=0x5653a5ed1260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:13:50.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.306+0000 7fb239600640 1 --2- 172.21.15.16:0/2213370449 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5653a5f19bf0 0x5653a5ed1260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:50.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.306+0000 7fb239600640 1 -- 172.21.15.16:0/2213370449 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5653a5f192d0 msgr2=0x5653a5f20b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:13:50.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.306+0000 7fb239600640 1 --2- 172.21.15.16:0/2213370449 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5653a5f192d0 0x5653a5f20b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:50.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.306+0000 7fb239600640 1 -- 172.21.15.16:0/2213370449 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5653a5cdab70 con 0x5653a5ecc9b0 2024-03-31T15:13:50.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.306+0000 7fb239600640 1 --2- 172.21.15.16:0/2213370449 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5653a5ecc9b0 0x5653a5f20630 secure :-1 s=READY pgs=1553 cs=0 l=1 rev1=1 crypto rx=0x7fb22801ab20 tx=0x7fb22801acb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=86fca84fcf2bab7c server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:13:50.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.306+0000 7fb21aa00640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-31T15:13:50.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.306+0000 7fb21aa00640 1 -- 172.21.15.16:0/2213370449 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2280b5b50 con 0x5653a5ecc9b0 2024-03-31T15:13:50.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.306+0000 7fb21aa00640 1 -- 172.21.15.16:0/2213370449 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 5795+0+0 (secure 0 0 0) 0x7fb2280b5cf0 con 0x5653a5ecc9b0 2024-03-31T15:13:50.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.306+0000 7fb23f3de000 1 -- 172.21.15.16:0/2213370449 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb22c060300 con 0x5653a5ecc9b0 2024-03-31T15:13:50.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.306+0000 7fb21aa00640 1 -- 172.21.15.16:0/2213370449 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=89) ==== 24+0+0 (secure 0 0 0) 0x7fb2280cf050 con 0x5653a5ecc9b0 2024-03-31T15:13:50.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.306+0000 7fb23f3de000 10 client.9009 fetch_fsmap learned FSMap version 89 2024-03-31T15:13:50.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.306+0000 7fb23f3de000 1 -- 172.21.15.16:0/2213370449 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=89,monmap=2+}) -- 0x5653a5cbbb90 con 0x5653a5ecc9b0 2024-03-31T15:13:50.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.306+0000 7fb21aa00640 1 -- 172.21.15.16:0/2213370449 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb22c060300 con 0x5653a5ecc9b0 2024-03-31T15:13:50.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.306+0000 7fb21aa00640 1 -- 172.21.15.16:0/2213370449 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 89) ==== 3494+0+0 (secure 0 0 0) 0x7fb2280d2020 con 0x5653a5ecc9b0 2024-03-31T15:13:50.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.306+0000 7fb23f3de000 10 client.9009 fetch_fsmap finished waiting for FSMap version 89 2024-03-31T15:13:50.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.306+0000 7fb23f3de000 10 client.9009 resolve_mds: resolved 7:0 to role '7:0' aka daemon mds.a 2024-03-31T15:13:50.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.306+0000 7fb23f3de000 20 client.9009 populate_metadata read hostname 'smithi016' 2024-03-31T15:13:50.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.306+0000 7fb23f3de000 1 --2- 172.21.15.16:0/2213370449 >> [v2:172.21.15.16:6833/567808781,v1:172.21.15.16:6835/567808781] conn(0x5653a5ed1fd0 0x5653a5ed23b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:50.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.306+0000 7fb23f3de000 4 client.9009 mds_command: new command op to 8874 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-31T15:13:50.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.306+0000 7fb23f3de000 1 -- 172.21.15.16:0/2213370449 --> [v2:172.21.15.16:6833/567808781,v1:172.21.15.16:6835/567808781] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5653a5cb9dd0 con 0x5653a5ed1fd0 2024-03-31T15:13:50.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.306+0000 7fb238c00640 1 --2- 172.21.15.16:0/2213370449 >> [v2:172.21.15.16:6833/567808781,v1:172.21.15.16:6835/567808781] conn(0x5653a5ed1fd0 0x5653a5ed23b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:50.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.310+0000 7fb238c00640 1 --2- 172.21.15.16:0/2213370449 >> [v2:172.21.15.16:6833/567808781,v1:172.21.15.16:6835/567808781] conn(0x5653a5ed1fd0 0x5653a5ed23b0 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=aa2a2d14c544ff9 server_cookie=bead3809c1c221ab in_seq=0 out_seq=0 2024-03-31T15:13:50.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.310+0000 7fb21aa00640 10 client.9009 ms_handle_connect on v2:172.21.15.16:6833/567808781 2024-03-31T15:13:50.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.310+0000 7fb21aa00640 1 -- 172.21.15.16:0/2213370449 <== mds.0 v2:172.21.15.16:6833/567808781 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5653a5cb9dd0 con 0x5653a5ed1fd0 2024-03-31T15:13:50.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.310+0000 7fb21aa00640 10 client.9009 handle_command_reply: tid=0 2024-03-31T15:13:50.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.310+0000 7fb23f3de000 1 client.9009 shutdown 2024-03-31T15:13:50.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.310+0000 7fb23f3de000 1 -- 172.21.15.16:0/2213370449 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5653a5ecc9b0 msgr2=0x5653a5f20630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:13:50.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.310+0000 7fb23f3de000 1 --2- 172.21.15.16:0/2213370449 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5653a5ecc9b0 0x5653a5f20630 secure :-1 s=READY pgs=1553 cs=0 l=1 rev1=1 crypto rx=0x7fb22801ab20 tx=0x7fb22801acb0 comp rx=0 tx=0).stop 2024-03-31T15:13:50.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.310+0000 7fb23f3de000 1 -- 172.21.15.16:0/2213370449 shutdown_connections 2024-03-31T15:13:50.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.310+0000 7fb23f3de000 1 --2- 172.21.15.16:0/2213370449 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5653a5f19bf0 0x5653a5ed1260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:50.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.310+0000 7fb23f3de000 1 --2- 172.21.15.16:0/2213370449 >> [v2:172.21.15.16:6833/567808781,v1:172.21.15.16:6835/567808781] conn(0x5653a5ed1fd0 0x5653a5ed23b0 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:50.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.310+0000 7fb23f3de000 1 --2- 172.21.15.16:0/2213370449 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5653a5f192d0 0x5653a5f20b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:50.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.310+0000 7fb23f3de000 1 --2- 172.21.15.16:0/2213370449 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5653a5ecc9b0 0x5653a5f20630 unknown :-1 s=CLOSED pgs=1553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:50.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.310+0000 7fb21aa00640 0 client.9009 ms_handle_reset on v2:172.21.15.16:6833/567808781 2024-03-31T15:13:50.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.310+0000 7fb23f3de000 1 -- 172.21.15.16:0/2213370449 >> 172.21.15.16:0/2213370449 conn(0x5653a5efa070 msgr2=0x5653a5efc460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:13:50.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.310+0000 7fb23f3de000 1 -- 172.21.15.16:0/2213370449 shutdown_connections 2024-03-31T15:13:50.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.310+0000 7fb23f3de000 1 -- 172.21.15.16:0/2213370449 wait complete. 2024-03-31T15:13:50.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.310+0000 7fb23f3de000 20 client.9009 trim_cache size 0 max 16384 2024-03-31T15:13:50.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.326+0000 7fb23f3de000 1 Processor -- start 2024-03-31T15:13:50.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.326+0000 7fb23f3de000 1 -- start start 2024-03-31T15:13:50.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.330+0000 7fb23f3de000 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5653a5efa070 0x5653a5f18390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:50.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.330+0000 7fb23f3de000 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5653a5ecc9b0 0x5653a5f188d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:50.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.330+0000 7fb23f3de000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5653a5f192d0 0x5653a5ecabd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:50.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.330+0000 7fb23f3de000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fb234143fc0 con 0x5653a5f192d0 2024-03-31T15:13:50.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.330+0000 7fb23f3de000 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fb234144610 con 0x5653a5efa070 2024-03-31T15:13:50.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.330+0000 7fb23f3de000 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x5653a5cb9940 con 0x5653a5ecc9b0 2024-03-31T15:13:50.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.330+0000 7fb23a000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5653a5f192d0 0x5653a5ecabd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:50.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.330+0000 7fb23a000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5653a5f192d0 0x5653a5ecabd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50182/0 (socket says 172.21.15.16:50182) 2024-03-31T15:13:50.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.330+0000 7fb23a000640 1 -- 172.21.15.16:0/2482040585 learned_addr learned my addr 172.21.15.16:0/2482040585 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:13:50.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.330+0000 7fb239600640 1 --2- 172.21.15.16:0/2482040585 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5653a5efa070 0x5653a5f18390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:50.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.330+0000 7fb238c00640 1 --2- 172.21.15.16:0/2482040585 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5653a5ecc9b0 0x5653a5f188d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:50.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.330+0000 7fb23a000640 1 -- 172.21.15.16:0/2482040585 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5653a5ecc9b0 msgr2=0x5653a5f188d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:13:50.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.330+0000 7fb23a000640 1 --2- 172.21.15.16:0/2482040585 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5653a5ecc9b0 0x5653a5f188d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:50.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.330+0000 7fb23a000640 1 -- 172.21.15.16:0/2482040585 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5653a5efa070 msgr2=0x5653a5f18390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:13:50.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.330+0000 7fb23a000640 1 --2- 172.21.15.16:0/2482040585 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5653a5efa070 0x5653a5f18390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:50.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.330+0000 7fb23a000640 1 -- 172.21.15.16:0/2482040585 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5653a5f19a20 con 0x5653a5f192d0 2024-03-31T15:13:50.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.330+0000 7fb23a000640 1 --2- 172.21.15.16:0/2482040585 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5653a5f192d0 0x5653a5ecabd0 secure :-1 s=READY pgs=1554 cs=0 l=1 rev1=1 crypto rx=0x7fb22c057fd0 tx=0x7fb22c053af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ecd545de986ad2e2 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:13:50.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.330+0000 7fb225600640 1 -- 172.21.15.16:0/2482040585 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb22c084070 con 0x5653a5f192d0 2024-03-31T15:13:50.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.330+0000 7fb225600640 1 -- 172.21.15.16:0/2482040585 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb22c073070 con 0x5653a5f192d0 2024-03-31T15:13:50.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.330+0000 7fb225600640 1 -- 172.21.15.16:0/2482040585 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb22c055a20 con 0x5653a5f192d0 2024-03-31T15:13:50.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.330+0000 7fb23f3de000 1 -- 172.21.15.16:0/2482040585 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5653a5f192d0 msgr2=0x5653a5ecabd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:13:50.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.330+0000 7fb23f3de000 1 --2- 172.21.15.16:0/2482040585 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5653a5f192d0 0x5653a5ecabd0 secure :-1 s=READY pgs=1554 cs=0 l=1 rev1=1 crypto rx=0x7fb22c057fd0 tx=0x7fb22c053af0 comp rx=0 tx=0).stop 2024-03-31T15:13:50.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.330+0000 7fb23f3de000 1 -- 172.21.15.16:0/2482040585 shutdown_connections 2024-03-31T15:13:50.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.330+0000 7fb23f3de000 1 --2- 172.21.15.16:0/2482040585 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5653a5f192d0 0x5653a5ecabd0 unknown :-1 s=CLOSED pgs=1554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:50.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.330+0000 7fb23f3de000 1 --2- 172.21.15.16:0/2482040585 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5653a5ecc9b0 0x5653a5f188d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:50.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.330+0000 7fb23f3de000 1 --2- 172.21.15.16:0/2482040585 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5653a5efa070 0x5653a5f18390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:50.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.330+0000 7fb23f3de000 1 -- 172.21.15.16:0/2482040585 >> 172.21.15.16:0/2482040585 conn(0x5653a5f270b0 msgr2=0x5653a5ed0d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:13:50.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.330+0000 7fb23f3de000 1 -- 172.21.15.16:0/2482040585 shutdown_connections 2024-03-31T15:13:50.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.330+0000 7fb23f3de000 1 -- 172.21.15.16:0/2482040585 wait complete. 2024-03-31T15:13:50.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.330+0000 7fb23f3de000 1 -- start start 2024-03-31T15:13:50.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.330+0000 7fb23f3de000 1 Processor -- start 2024-03-31T15:13:50.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.330+0000 7fb23f3de000 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5653a5ecc9b0 0x5653a5ef8840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:50.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.330+0000 7fb23f3de000 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5653a5efa070 0x5653a5ef8d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:50.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.330+0000 7fb23f3de000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5653a5f192d0 0x5653a5f86000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:50.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.330+0000 7fb23f3de000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5653a5dd9250 con 0x5653a5f192d0 2024-03-31T15:13:50.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.330+0000 7fb23f3de000 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x5653a5cd5750 con 0x5653a5ecc9b0 2024-03-31T15:13:50.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.330+0000 7fb23f3de000 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x5653a5ded420 con 0x5653a5efa070 2024-03-31T15:13:50.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.330+0000 7fb23f3de000 10 client.0 _ll_register_callbacks cb 0x7fb22c0095f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T15:13:50.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.334+0000 7fb23a000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5653a5f192d0 0x5653a5f86000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:50.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.334+0000 7fb238c00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5653a5efa070 0x5653a5ef8d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:50.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.334+0000 7fb23a000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5653a5f192d0 0x5653a5f86000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50186/0 (socket says 172.21.15.16:50186) 2024-03-31T15:13:50.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.334+0000 7fb23a000640 1 -- 172.21.15.16:0/2998835984 learned_addr learned my addr 172.21.15.16:0/2998835984 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:13:50.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.334+0000 7fb239600640 1 --2- 172.21.15.16:0/2998835984 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5653a5ecc9b0 0x5653a5ef8840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:50.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.334+0000 7fb23a000640 1 -- 172.21.15.16:0/2998835984 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5653a5efa070 msgr2=0x5653a5ef8d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:13:50.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.334+0000 7fb23a000640 1 --2- 172.21.15.16:0/2998835984 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5653a5efa070 0x5653a5ef8d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:50.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.334+0000 7fb23a000640 1 -- 172.21.15.16:0/2998835984 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5653a5ecc9b0 msgr2=0x5653a5ef8840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:13:50.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.334+0000 7fb23a000640 1 --2- 172.21.15.16:0/2998835984 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5653a5ecc9b0 0x5653a5ef8840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:50.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.334+0000 7fb23a000640 1 -- 172.21.15.16:0/2998835984 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5653a5d9f140 con 0x5653a5f192d0 2024-03-31T15:13:50.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.334+0000 7fb23a000640 1 --2- 172.21.15.16:0/2998835984 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5653a5f192d0 0x5653a5f86000 secure :-1 s=READY pgs=1555 cs=0 l=1 rev1=1 crypto rx=0x7fb22c05fdd0 tx=0x7fb22c059000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb56182f95470fc3 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:13:50.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.334+0000 7fb21aa00640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-31T15:13:50.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.334+0000 7fb21aa00640 1 -- 172.21.15.16:0/2998835984 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb22c084050 con 0x5653a5f192d0 2024-03-31T15:13:50.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.334+0000 7fb21aa00640 1 -- 172.21.15.16:0/2998835984 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 5795+0+0 (secure 0 0 0) 0x7fb22c056b70 con 0x5653a5f192d0 2024-03-31T15:13:50.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.334+0000 7fb23f3de000 1 -- 172.21.15.16:0/2998835984 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb22c073070 con 0x5653a5f192d0 2024-03-31T15:13:50.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.334+0000 7fb21aa00640 1 -- 172.21.15.16:0/2998835984 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=89) ==== 24+0+0 (secure 0 0 0) 0x7fb22c060300 con 0x5653a5f192d0 2024-03-31T15:13:50.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.334+0000 7fb23f3de000 10 client.9015 fetch_fsmap learned FSMap version 89 2024-03-31T15:13:50.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.334+0000 7fb23f3de000 1 -- 172.21.15.16:0/2998835984 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=89,monmap=2+}) -- 0x5653a5cad1e0 con 0x5653a5f192d0 2024-03-31T15:13:50.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.334+0000 7fb21aa00640 1 -- 172.21.15.16:0/2998835984 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb22c073070 con 0x5653a5f192d0 2024-03-31T15:13:50.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.334+0000 7fb21aa00640 1 -- 172.21.15.16:0/2998835984 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 89) ==== 3494+0+0 (secure 0 0 0) 0x7fb22c07f260 con 0x5653a5f192d0 2024-03-31T15:13:50.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.334+0000 7fb23f3de000 10 client.9015 fetch_fsmap finished waiting for FSMap version 89 2024-03-31T15:13:50.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.334+0000 7fb23f3de000 10 client.9015 resolve_mds: resolved 7:0 to role '7:0' aka daemon mds.a 2024-03-31T15:13:50.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.334+0000 7fb23f3de000 20 client.9015 populate_metadata read hostname 'smithi016' 2024-03-31T15:13:50.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.334+0000 7fb23f3de000 1 --2- 172.21.15.16:0/2998835984 >> [v2:172.21.15.16:6833/567808781,v1:172.21.15.16:6835/567808781] conn(0x5653a5f19bf0 0x5653a5f86540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:50.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.334+0000 7fb23f3de000 4 client.9015 mds_command: new command op to 8874 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-31T15:13:50.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.334+0000 7fb23f3de000 1 -- 172.21.15.16:0/2998835984 --> [v2:172.21.15.16:6833/567808781,v1:172.21.15.16:6835/567808781] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5653a5d2c040 con 0x5653a5f19bf0 2024-03-31T15:13:50.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.334+0000 7fb239600640 1 --2- 172.21.15.16:0/2998835984 >> [v2:172.21.15.16:6833/567808781,v1:172.21.15.16:6835/567808781] conn(0x5653a5f19bf0 0x5653a5f86540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:50.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.334+0000 7fb239600640 1 --2- 172.21.15.16:0/2998835984 >> [v2:172.21.15.16:6833/567808781,v1:172.21.15.16:6835/567808781] conn(0x5653a5f19bf0 0x5653a5f86540 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b4fae299327cd3b4 server_cookie=64937308867e495e in_seq=0 out_seq=0 2024-03-31T15:13:50.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.334+0000 7fb21aa00640 10 client.9015 ms_handle_connect on v2:172.21.15.16:6833/567808781 2024-03-31T15:13:50.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.334+0000 7fb21aa00640 1 -- 172.21.15.16:0/2998835984 <== mds.0 v2:172.21.15.16:6833/567808781 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x5653a5d2c040 con 0x5653a5f19bf0 2024-03-31T15:13:50.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.334+0000 7fb21aa00640 10 client.9015 handle_command_reply: tid=0 2024-03-31T15:13:50.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.334+0000 7fb23f3de000 1 client.9015 shutdown 2024-03-31T15:13:50.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.334+0000 7fb23f3de000 1 -- 172.21.15.16:0/2998835984 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5653a5f192d0 msgr2=0x5653a5f86000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:13:50.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.334+0000 7fb23f3de000 1 --2- 172.21.15.16:0/2998835984 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5653a5f192d0 0x5653a5f86000 secure :-1 s=READY pgs=1555 cs=0 l=1 rev1=1 crypto rx=0x7fb22c05fdd0 tx=0x7fb22c059000 comp rx=0 tx=0).stop 2024-03-31T15:13:50.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.334+0000 7fb23f3de000 1 -- 172.21.15.16:0/2998835984 shutdown_connections 2024-03-31T15:13:50.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.334+0000 7fb23f3de000 1 --2- 172.21.15.16:0/2998835984 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5653a5f192d0 0x5653a5f86000 unknown :-1 s=CLOSED pgs=1555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:50.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.334+0000 7fb23f3de000 1 --2- 172.21.15.16:0/2998835984 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5653a5efa070 0x5653a5ef8d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:50.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.334+0000 7fb23f3de000 1 --2- 172.21.15.16:0/2998835984 >> [v2:172.21.15.16:6833/567808781,v1:172.21.15.16:6835/567808781] conn(0x5653a5f19bf0 0x5653a5f86540 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:50.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.334+0000 7fb23f3de000 1 --2- 172.21.15.16:0/2998835984 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5653a5ecc9b0 0x5653a5ef8840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:50.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.334+0000 7fb21aa00640 0 client.9015 ms_handle_reset on v2:172.21.15.16:6833/567808781 2024-03-31T15:13:50.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.334+0000 7fb23f3de000 1 -- 172.21.15.16:0/2998835984 >> 172.21.15.16:0/2998835984 conn(0x5653a5f270b0 msgr2=0x5653a5ecabd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:13:50.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.334+0000 7fb23f3de000 1 -- 172.21.15.16:0/2998835984 shutdown_connections 2024-03-31T15:13:50.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.334+0000 7fb23f3de000 1 -- 172.21.15.16:0/2998835984 wait complete. 2024-03-31T15:13:50.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.334+0000 7fb23f3de000 20 client.9015 trim_cache size 0 max 16384 2024-03-31T15:13:50.333 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-03-31T15:13:50.333 INFO:teuthology.orchestra.run.smithi016.stdout: "status": "no active scrubs running", 2024-03-31T15:13:50.333 INFO:teuthology.orchestra.run.smithi016.stdout: "scrubs": {} 2024-03-31T15:13:50.333 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-03-31T15:13:50.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.334+0000 7fb23c000640 1 -- 172.21.15.16:0/754120345 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fb214068380 msgr2=0x7fb21406a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:13:50.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.334+0000 7fb23c000640 1 --2- 172.21.15.16:0/754120345 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fb214068380 0x7fb21406a840 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7fb2280af000 tx=0x7fb228008040 comp rx=0 tx=0).stop 2024-03-31T15:13:50.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.334+0000 7fb23c000640 1 -- 172.21.15.16:0/754120345 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb23415c890 msgr2=0x7fb23414ab80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:13:50.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.334+0000 7fb23c000640 1 --2- 172.21.15.16:0/754120345 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb23415c890 0x7fb23414ab80 secure :-1 s=READY pgs=1551 cs=0 l=1 rev1=1 crypto rx=0x7fb22c00ef90 tx=0x7fb22c00c490 comp rx=0 tx=0).stop 2024-03-31T15:13:50.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.338+0000 7fb23c000640 1 -- 172.21.15.16:0/754120345 shutdown_connections 2024-03-31T15:13:50.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.338+0000 7fb23c000640 1 --2- 172.21.15.16:0/754120345 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fb214068380 0x7fb21406a840 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:50.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.338+0000 7fb23c000640 1 --2- 172.21.15.16:0/754120345 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb23415c890 0x7fb23414ab80 unknown :-1 s=CLOSED pgs=1551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:50.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.338+0000 7fb23c000640 1 --2- 172.21.15.16:0/754120345 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb234145d50 0x7fb23414a640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:50.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.338+0000 7fb23c000640 1 --2- 172.21.15.16:0/754120345 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb234070350 0x7fb2340ccab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:50.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.338+0000 7fb23c000640 1 -- 172.21.15.16:0/754120345 >> 172.21.15.16:0/754120345 conn(0x7fb23406e670 msgr2=0x7fb234166970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:13:50.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.338+0000 7fb23c000640 1 -- 172.21.15.16:0/754120345 shutdown_connections 2024-03-31T15:13:50.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.338+0000 7fb23c000640 1 -- 172.21.15.16:0/754120345 wait complete. 2024-03-31T15:13:50.348 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-31T15:13:50.449 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-31T15:13:50.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.598+0000 7f4947e00640 1 Processor -- start 2024-03-31T15:13:50.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.598+0000 7f4947e00640 1 -- start start 2024-03-31T15:13:50.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.598+0000 7f4947e00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f4940145e40 0x7f49400cc740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:50.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.598+0000 7f4947e00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f49400ccc80 0x7f494006f920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:50.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.598+0000 7f4947e00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f494006fe60 0x7f494015fdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:50.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.598+0000 7f4947e00640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f49400d06d0 con 0x7f4940145e40 2024-03-31T15:13:50.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.598+0000 7f4947e00640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f49400d0920 con 0x7f49400ccc80 2024-03-31T15:13:50.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.598+0000 7f4947e00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f49400d0c00 con 0x7f494006fe60 2024-03-31T15:13:50.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.598+0000 7f4945400640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f4940145e40 0x7f49400cc740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:50.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.598+0000 7f4945400640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f4940145e40 0x7f49400cc740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50190/0 (socket says 172.21.15.16:50190) 2024-03-31T15:13:50.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.598+0000 7f4945400640 1 -- 172.21.15.16:0/3119290958 learned_addr learned my addr 172.21.15.16:0/3119290958 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:13:50.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.598+0000 7f4944a00640 1 --2- 172.21.15.16:0/3119290958 >> v2:172.21.15.89:3300/0 conn(0x7f49400ccc80 0x7f494006f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:50.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.598+0000 7f4945e00640 1 --2- 172.21.15.16:0/3119290958 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f494006fe60 0x7f494015fdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:50.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.602+0000 7f4945400640 1 -- 172.21.15.16:0/3119290958 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f494006fe60 msgr2=0x7f494015fdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:13:50.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.602+0000 7f4945400640 1 --2- 172.21.15.16:0/3119290958 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f494006fe60 0x7f494015fdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:50.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.602+0000 7f4945400640 1 -- 172.21.15.16:0/3119290958 >> v2:172.21.15.89:3300/0 conn(0x7f49400ccc80 msgr2=0x7f494006f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:13:50.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.602+0000 7f4945400640 1 --2- 172.21.15.16:0/3119290958 >> v2:172.21.15.89:3300/0 conn(0x7f49400ccc80 0x7f494006f920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:50.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.602+0000 7f4945400640 1 -- 172.21.15.16:0/3119290958 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49400d1810 con 0x7f4940145e40 2024-03-31T15:13:50.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.602+0000 7f4945e00640 1 --2- 172.21.15.16:0/3119290958 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f494006fe60 0x7f494015fdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:13:50.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.602+0000 7f4944a00640 1 --2- 172.21.15.16:0/3119290958 >> v2:172.21.15.89:3300/0 conn(0x7f49400ccc80 0x7f494006f920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:13:50.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.602+0000 7f4945400640 1 --2- 172.21.15.16:0/3119290958 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4940145e40 0x7f49400cc740 secure :-1 s=READY pgs=1556 cs=0 l=1 rev1=1 crypto rx=0x7f4940073cf0 tx=0x7f493401c260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a90541438bed2293 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:13:50.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.602+0000 7f493fe00640 1 -- 172.21.15.16:0/3119290958 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49340b0070 con 0x7f4940145e40 2024-03-31T15:13:50.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.602+0000 7f493fe00640 1 -- 172.21.15.16:0/3119290958 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f49340ab600 con 0x7f4940145e40 2024-03-31T15:13:50.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.602+0000 7f493fe00640 1 -- 172.21.15.16:0/3119290958 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49340ab9d0 con 0x7f4940145e40 2024-03-31T15:13:50.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.606+0000 7f4947e00640 1 -- 172.21.15.16:0/3119290958 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4940145e40 msgr2=0x7f49400cc740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:13:50.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.606+0000 7f4947e00640 1 --2- 172.21.15.16:0/3119290958 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4940145e40 0x7f49400cc740 secure :-1 s=READY pgs=1556 cs=0 l=1 rev1=1 crypto rx=0x7f4940073cf0 tx=0x7f493401c260 comp rx=0 tx=0).stop 2024-03-31T15:13:50.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.606+0000 7f4947e00640 1 -- 172.21.15.16:0/3119290958 shutdown_connections 2024-03-31T15:13:50.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.606+0000 7f4947e00640 1 --2- 172.21.15.16:0/3119290958 >> v2:172.21.15.89:3300/0 conn(0x7f49400ccc80 0x7f494006f920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:50.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.606+0000 7f4947e00640 1 --2- 172.21.15.16:0/3119290958 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f494006fe60 0x7f494015fdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:50.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.606+0000 7f4947e00640 1 --2- 172.21.15.16:0/3119290958 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4940145e40 0x7f49400cc740 unknown :-1 s=CLOSED pgs=1556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:50.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.606+0000 7f4947e00640 1 -- 172.21.15.16:0/3119290958 >> 172.21.15.16:0/3119290958 conn(0x7f4940067330 msgr2=0x7f4940067740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:13:50.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.606+0000 7f4947e00640 1 -- 172.21.15.16:0/3119290958 shutdown_connections 2024-03-31T15:13:50.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.606+0000 7f4947e00640 1 -- 172.21.15.16:0/3119290958 wait complete. 2024-03-31T15:13:50.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.606+0000 7f4947e00640 1 Processor -- start 2024-03-31T15:13:50.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.606+0000 7f4947e00640 1 -- start start 2024-03-31T15:13:50.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.606+0000 7f4947e00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f494006fe60 0x7f49401448d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:50.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.606+0000 7f4947e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f49400ccc80 0x7f49401543e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:50.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.606+0000 7f4947e00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4940145e40 0x7f4940159df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:50.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.606+0000 7f4947e00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f49400b5240 con 0x7f49400ccc80 2024-03-31T15:13:50.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.606+0000 7f4947e00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f49400b50c0 con 0x7f4940145e40 2024-03-31T15:13:50.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.606+0000 7f4947e00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f49400d2010 con 0x7f494006fe60 2024-03-31T15:13:50.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.606+0000 7f4944a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f49400ccc80 0x7f49401543e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:50.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.606+0000 7f4944a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f49400ccc80 0x7f49401543e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50196/0 (socket says 172.21.15.16:50196) 2024-03-31T15:13:50.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.606+0000 7f4944a00640 1 -- 172.21.15.16:0/3541303851 learned_addr learned my addr 172.21.15.16:0/3541303851 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:13:50.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.606+0000 7f4945400640 1 --2- 172.21.15.16:0/3541303851 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f494006fe60 0x7f49401448d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:50.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.606+0000 7f4945e00640 1 --2- 172.21.15.16:0/3541303851 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4940145e40 0x7f4940159df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:50.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.606+0000 7f4944a00640 1 -- 172.21.15.16:0/3541303851 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f494006fe60 msgr2=0x7f49401448d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:13:50.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.606+0000 7f4944a00640 1 --2- 172.21.15.16:0/3541303851 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f494006fe60 0x7f49401448d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:50.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.606+0000 7f4944a00640 1 -- 172.21.15.16:0/3541303851 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4940145e40 msgr2=0x7f4940159df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:13:50.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.606+0000 7f4944a00640 1 --2- 172.21.15.16:0/3541303851 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4940145e40 0x7f4940159df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:50.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.606+0000 7f4944a00640 1 -- 172.21.15.16:0/3541303851 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4940154e60 con 0x7f49400ccc80 2024-03-31T15:13:50.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.610+0000 7f4944a00640 1 --2- 172.21.15.16:0/3541303851 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f49400ccc80 0x7f49401543e0 secure :-1 s=READY pgs=1557 cs=0 l=1 rev1=1 crypto rx=0x7f492c00e810 tx=0x7f492c00ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:13:50.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.610+0000 7f493d600640 1 -- 172.21.15.16:0/3541303851 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f492c017070 con 0x7f49400ccc80 2024-03-31T15:13:50.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.610+0000 7f493d600640 1 -- 172.21.15.16:0/3541303851 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f492c00c8d0 con 0x7f49400ccc80 2024-03-31T15:13:50.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.610+0000 7f493d600640 1 -- 172.21.15.16:0/3541303851 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f492c00f040 con 0x7f49400ccc80 2024-03-31T15:13:50.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.610+0000 7f4947e00640 1 -- 172.21.15.16:0/3541303851 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f49400b5460 con 0x7f49400ccc80 2024-03-31T15:13:50.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.610+0000 7f4947e00640 1 -- 172.21.15.16:0/3541303851 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4940155140 con 0x7f49400ccc80 2024-03-31T15:13:50.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.610+0000 7f4947e00640 1 -- 172.21.15.16:0/3541303851 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4940069950 con 0x7f49400ccc80 2024-03-31T15:13:50.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.610+0000 7f493d600640 1 -- 172.21.15.16:0/3541303851 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f492c01a070 con 0x7f49400ccc80 2024-03-31T15:13:50.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.610+0000 7f493d600640 1 --2- 172.21.15.16:0/3541303851 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f4920068380 0x7f492006a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:50.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.610+0000 7f493d600640 1 -- 172.21.15.16:0/3541303851 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 5795+0+0 (secure 0 0 0) 0x7f492c086620 con 0x7f49400ccc80 2024-03-31T15:13:50.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.610+0000 7f4945400640 1 --2- 172.21.15.16:0/3541303851 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f4920068380 0x7f492006a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:50.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.614+0000 7f4945400640 1 --2- 172.21.15.16:0/3541303851 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f4920068380 0x7f492006a840 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f49340ad000 tx=0x7f49340a6000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:13:50.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.614+0000 7f493d600640 1 -- 172.21.15.16:0/3541303851 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f492c056490 con 0x7f49400ccc80 2024-03-31T15:13:50.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.818+0000 7f4947e00640 1 -- 172.21.15.16:0/3541303851 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f49400cf3c0 con 0x7f49400ccc80 2024-03-31T15:13:50.811 INFO:tasks.ceph.mon.a.smithi016.stderr:2024-03-31T15:13:50.818+0000 7f7dcbc00640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-31T15:13:50.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.822+0000 7f493d600640 1 -- 172.21.15.16:0/3541303851 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v90) ==== 161+0+0 (secure 0 0 0) 0x7f492c05a790 con 0x7f49400ccc80 2024-03-31T15:13:50.813 INFO:teuthology.orchestra.run.smithi016.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-31T15:13:50.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.822+0000 7f4947e00640 1 -- 172.21.15.16:0/3541303851 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f4920068380 msgr2=0x7f492006a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:13:50.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.822+0000 7f4947e00640 1 --2- 172.21.15.16:0/3541303851 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f4920068380 0x7f492006a840 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f49340ad000 tx=0x7f49340a6000 comp rx=0 tx=0).stop 2024-03-31T15:13:50.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.822+0000 7f4947e00640 1 -- 172.21.15.16:0/3541303851 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f49400ccc80 msgr2=0x7f49401543e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:13:50.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.822+0000 7f4947e00640 1 --2- 172.21.15.16:0/3541303851 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f49400ccc80 0x7f49401543e0 secure :-1 s=READY pgs=1557 cs=0 l=1 rev1=1 crypto rx=0x7f492c00e810 tx=0x7f492c00ecd0 comp rx=0 tx=0).stop 2024-03-31T15:13:50.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.822+0000 7f4947e00640 1 -- 172.21.15.16:0/3541303851 shutdown_connections 2024-03-31T15:13:50.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.822+0000 7f4947e00640 1 --2- 172.21.15.16:0/3541303851 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f4920068380 0x7f492006a840 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:50.818 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.822+0000 7f4947e00640 1 --2- 172.21.15.16:0/3541303851 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4940145e40 0x7f4940159df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:50.818 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.822+0000 7f4947e00640 1 --2- 172.21.15.16:0/3541303851 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f49400ccc80 0x7f49401543e0 secure :-1 s=CLOSED pgs=1557 cs=0 l=1 rev1=1 crypto rx=0x7f492c00e810 tx=0x7f492c00ecd0 comp rx=0 tx=0).stop 2024-03-31T15:13:50.818 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.822+0000 7f4947e00640 1 --2- 172.21.15.16:0/3541303851 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f494006fe60 0x7f49401448d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:50.818 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.822+0000 7f4947e00640 1 -- 172.21.15.16:0/3541303851 >> 172.21.15.16:0/3541303851 conn(0x7f4940067330 msgr2=0x7f49400b1c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:13:50.818 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.826+0000 7f4947e00640 1 -- 172.21.15.16:0/3541303851 shutdown_connections 2024-03-31T15:13:50.818 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.826+0000 7f4947e00640 1 -- 172.21.15.16:0/3541303851 wait complete. 2024-03-31T15:13:50.824 INFO:tasks.ceph.mds.a.smithi016.stderr: -1147> 2024-03-31T15:13:19.906+0000 7f7630600640 -1 log_channel(cluster) log [ERR] : scrub: inode wrongly marked free: 0x10000000001 2024-03-31T15:13:50.824 INFO:tasks.ceph.mds.a.smithi016.stderr: -1144> 2024-03-31T15:13:19.906+0000 7f7630600640 -1 log_channel(cluster) log [ERR] : inode table repaired for inode: 0x10000000001 2024-03-31T15:13:50.824 INFO:tasks.ceph.mds.a.smithi016.stderr: -1128> 2024-03-31T15:13:19.906+0000 7f7630600640 -1 log_channel(cluster) log [ERR] : scrub: inode wrongly marked free: 0x10000000002 2024-03-31T15:13:50.824 INFO:tasks.ceph.mds.a.smithi016.stderr: -1125> 2024-03-31T15:13:19.906+0000 7f7630600640 -1 log_channel(cluster) log [ERR] : inode table repaired for inode: 0x10000000002 2024-03-31T15:13:50.824 INFO:tasks.ceph.mds.a.smithi016.stderr: -960> 2024-03-31T15:13:21.086+0000 7f7635600640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-31T15:13:50.841 DEBUG:teuthology.orchestra.run.smithi016:> cephfs-table-tool --debug-mds=20 --debug-ms=1 --debug-objecter=1 0 show inode 2024-03-31T15:13:50.869 INFO:tasks.ceph.mds.a.smithi016.stdout:starting mds.a at 2024-03-31T15:13:50.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.894+0000 7f2784f31400 1 -- start start 2024-03-31T15:13:50.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.894+0000 7f2784f31400 1 Processor -- start 2024-03-31T15:13:50.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.894+0000 7f2784f31400 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x561015005800 0x561015dc0000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:50.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.894+0000 7f2784f31400 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561015005400 0x561015dc0580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:50.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.894+0000 7f2784f31400 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561015005000 0x561015dc0b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:50.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.894+0000 7f2784f31400 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x561015ddc180 con 0x561015005400 2024-03-31T15:13:50.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.894+0000 7f2784f31400 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x561015064900 con 0x561015005000 2024-03-31T15:13:50.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.894+0000 7f2784f31400 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x561015064480 con 0x561015005800 2024-03-31T15:13:50.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.894+0000 7f2782600640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561015005400 0x561015dc0580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:50.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.894+0000 7f2782600640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561015005400 0x561015dc0580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50226/0 (socket says 172.21.15.16:50226) 2024-03-31T15:13:50.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.894+0000 7f2782600640 1 -- 172.21.15.16:0/668127171 learned_addr learned my addr 172.21.15.16:0/668127171 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:13:50.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.894+0000 7f2783a00640 1 --2- 172.21.15.16:0/668127171 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561015005000 0x561015dc0b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:50.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.894+0000 7f2783000640 1 --2- 172.21.15.16:0/668127171 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x561015005800 0x561015dc0000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:50.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.894+0000 7f2782600640 1 -- 172.21.15.16:0/668127171 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x561015005800 msgr2=0x561015dc0000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:13:50.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.894+0000 7f2782600640 1 --2- 172.21.15.16:0/668127171 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x561015005800 0x561015dc0000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:50.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.894+0000 7f2782600640 1 -- 172.21.15.16:0/668127171 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561015005000 msgr2=0x561015dc0b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:13:50.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.894+0000 7f2782600640 1 --2- 172.21.15.16:0/668127171 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561015005000 0x561015dc0b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:50.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.894+0000 7f2782600640 1 -- 172.21.15.16:0/668127171 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561015dbeb40 con 0x561015005400 2024-03-31T15:13:50.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.894+0000 7f2783000640 1 --2- 172.21.15.16:0/668127171 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x561015005800 0x561015dc0000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:13:50.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.894+0000 7f2782600640 1 --2- 172.21.15.16:0/668127171 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561015005400 0x561015dc0580 secure :-1 s=READY pgs=1562 cs=0 l=1 rev1=1 crypto rx=0x561015e7cd20 tx=0x561015d84c60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c2816765f1542bd7 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:13:50.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.894+0000 7f277fe00640 1 -- 172.21.15.16:0/668127171 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x56101503a4e0 con 0x561015005400 2024-03-31T15:13:50.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.894+0000 7f277fe00640 1 -- 172.21.15.16:0/668127171 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x5610150501c0 con 0x561015005400 2024-03-31T15:13:50.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.894+0000 7f277fe00640 1 -- 172.21.15.16:0/668127171 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x56101503ab60 con 0x561015005400 2024-03-31T15:13:50.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.894+0000 7f2784f31400 1 -- 172.21.15.16:0/668127171 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561015dbed20 con 0x561015005400 2024-03-31T15:13:50.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.894+0000 7f2784f31400 1 -- 172.21.15.16:0/668127171 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({fsmap=0}) -- 0x561015dbef00 con 0x561015005400 2024-03-31T15:13:50.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.894+0000 7f2784f31400 4 waiting for MDS map... 2024-03-31T15:13:50.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.894+0000 7f277fe00640 1 -- 172.21.15.16:0/668127171 <== mon.0 v2:172.21.15.16:3300/0 4 ==== osd_map(98..98 src has 1..98) ==== 5881+0+0 (secure 0 0 0) 0x561015f1a600 con 0x561015005400 2024-03-31T15:13:50.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.894+0000 7f277fe00640 1 -- 172.21.15.16:0/668127171 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 90) ==== 2885+0+0 (secure 0 0 0) 0x56101505b180 con 0x561015005400 2024-03-31T15:13:50.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.898+0000 7f2784f31400 4 Got MDS map 90 2024-03-31T15:13:50.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.898+0000 7f2784f31400 10 main: main 2024-03-31T15:13:50.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.898+0000 7f2784f31400 4 main: connecting to RADOS... 2024-03-31T15:13:50.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.898+0000 7f2784f31400 1 Processor -- start 2024-03-31T15:13:50.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.898+0000 7f2784f31400 1 -- start start 2024-03-31T15:13:50.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.898+0000 7f2784f31400 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561015f17000 0x561015dc1080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:50.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.898+0000 7f2784f31400 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561015f17400 0x561015dc1600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:50.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.898+0000 7f2784f31400 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x561015f17800 0x561015dc1b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:50.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.898+0000 7f2784f31400 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x561015064180 con 0x561015f17000 2024-03-31T15:13:50.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.898+0000 7f2784f31400 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x561015ddcd80 con 0x561015f17400 2024-03-31T15:13:50.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.898+0000 7f2784f31400 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x561015ddcf00 con 0x561015f17800 2024-03-31T15:13:50.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.898+0000 7f2783a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561015f17000 0x561015dc1080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:50.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.898+0000 7f2783a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561015f17000 0x561015dc1080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50234/0 (socket says 172.21.15.16:50234) 2024-03-31T15:13:50.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.898+0000 7f2783a00640 1 -- 172.21.15.16:0/2241747645 learned_addr learned my addr 172.21.15.16:0/2241747645 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:13:50.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.898+0000 7f2783000640 1 --2- 172.21.15.16:0/2241747645 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561015f17400 0x561015dc1600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:50.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.898+0000 7f2782600640 1 --2- 172.21.15.16:0/2241747645 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x561015f17800 0x561015dc1b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:50.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.898+0000 7f2783a00640 1 -- 172.21.15.16:0/2241747645 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x561015f17800 msgr2=0x561015dc1b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:13:50.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.898+0000 7f2783a00640 1 --2- 172.21.15.16:0/2241747645 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x561015f17800 0x561015dc1b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:50.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.898+0000 7f2783a00640 1 -- 172.21.15.16:0/2241747645 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561015f17400 msgr2=0x561015dc1600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:13:50.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.898+0000 7f2783a00640 1 --2- 172.21.15.16:0/2241747645 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561015f17400 0x561015dc1600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:50.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.898+0000 7f2783a00640 1 -- 172.21.15.16:0/2241747645 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561015dbed20 con 0x561015f17000 2024-03-31T15:13:50.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.898+0000 7f2782600640 1 --2- 172.21.15.16:0/2241747645 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x561015f17800 0x561015dc1b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:13:50.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.898+0000 7f2783a00640 1 --2- 172.21.15.16:0/2241747645 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561015f17000 0x561015dc1080 secure :-1 s=READY pgs=1563 cs=0 l=1 rev1=1 crypto rx=0x561015f8a060 tx=0x561015dd2000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6027847584361557 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:13:50.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.898+0000 7f277d600640 1 -- 172.21.15.16:0/2241747645 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x56101503a9c0 con 0x561015f17000 2024-03-31T15:13:50.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.898+0000 7f277d600640 1 -- 172.21.15.16:0/2241747645 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x561015050e00 con 0x561015f17000 2024-03-31T15:13:50.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.898+0000 7f277d600640 1 -- 172.21.15.16:0/2241747645 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x56101503a4e0 con 0x561015f17000 2024-03-31T15:13:50.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.898+0000 7f2784f31400 1 -- 172.21.15.16:0/2241747645 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561015f17000 msgr2=0x561015dc1080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:13:50.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.898+0000 7f2784f31400 1 --2- 172.21.15.16:0/2241747645 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561015f17000 0x561015dc1080 secure :-1 s=READY pgs=1563 cs=0 l=1 rev1=1 crypto rx=0x561015f8a060 tx=0x561015dd2000 comp rx=0 tx=0).stop 2024-03-31T15:13:50.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.898+0000 7f2784f31400 1 -- 172.21.15.16:0/2241747645 shutdown_connections 2024-03-31T15:13:50.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.898+0000 7f2784f31400 1 --2- 172.21.15.16:0/2241747645 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x561015f17800 0x561015dc1b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:50.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.898+0000 7f2784f31400 1 --2- 172.21.15.16:0/2241747645 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561015f17400 0x561015dc1600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:50.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.898+0000 7f2784f31400 1 --2- 172.21.15.16:0/2241747645 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561015f17000 0x561015dc1080 unknown :-1 s=CLOSED pgs=1563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:50.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.898+0000 7f2784f31400 1 -- 172.21.15.16:0/2241747645 >> 172.21.15.16:0/2241747645 conn(0x561015004c00 msgr2=0x561015033800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:13:50.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.898+0000 7f2784f31400 1 -- 172.21.15.16:0/2241747645 shutdown_connections 2024-03-31T15:13:50.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.898+0000 7f2784f31400 1 -- 172.21.15.16:0/2241747645 wait complete. 2024-03-31T15:13:50.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.898+0000 7f2784f31400 1 Processor -- start 2024-03-31T15:13:50.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.902+0000 7f2784f31400 1 -- start start 2024-03-31T15:13:50.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.902+0000 7f2784f31400 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561015f17000 0x561015dc1080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:50.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.902+0000 7f2784f31400 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561015f17400 0x561015dc1600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:50.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.902+0000 7f2784f31400 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x561015f17800 0x561015dc1b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:50.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.902+0000 7f2784f31400 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x561015ddd200 con 0x561015f17400 2024-03-31T15:13:50.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.902+0000 7f2784f31400 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x561015ddd080 con 0x561015f17000 2024-03-31T15:13:50.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.902+0000 7f2784f31400 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x561015ddd380 con 0x561015f17800 2024-03-31T15:13:50.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.902+0000 7f2783000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561015f17400 0x561015dc1600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:50.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.902+0000 7f2783000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561015f17400 0x561015dc1600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50240/0 (socket says 172.21.15.16:50240) 2024-03-31T15:13:50.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.902+0000 7f2783000640 1 -- 172.21.15.16:0/2295598442 learned_addr learned my addr 172.21.15.16:0/2295598442 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:13:50.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.902+0000 7f2782600640 1 --2- 172.21.15.16:0/2295598442 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x561015f17800 0x561015dc1b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:50.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.902+0000 7f2783a00640 1 --2- 172.21.15.16:0/2295598442 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561015f17000 0x561015dc1080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:50.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.902+0000 7f2783000640 1 -- 172.21.15.16:0/2295598442 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x561015f17800 msgr2=0x561015dc1b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:13:50.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.902+0000 7f2783000640 1 --2- 172.21.15.16:0/2295598442 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x561015f17800 0x561015dc1b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:50.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.902+0000 7f2783000640 1 -- 172.21.15.16:0/2295598442 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561015f17000 msgr2=0x561015dc1080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:13:50.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.902+0000 7f2783000640 1 --2- 172.21.15.16:0/2295598442 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561015f17000 0x561015dc1080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:50.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.902+0000 7f2783000640 1 -- 172.21.15.16:0/2295598442 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561015dbfc20 con 0x561015f17400 2024-03-31T15:13:50.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.902+0000 7f2783000640 1 --2- 172.21.15.16:0/2295598442 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561015f17400 0x561015dc1600 secure :-1 s=READY pgs=1564 cs=0 l=1 rev1=1 crypto rx=0x561015f8b440 tx=0x5610152fc000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:13:50.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.902+0000 7f277b800640 1 -- 172.21.15.16:0/2295598442 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x56101503a9c0 con 0x561015f17400 2024-03-31T15:13:50.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.902+0000 7f277b800640 1 -- 172.21.15.16:0/2295598442 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x561015050540 con 0x561015f17400 2024-03-31T15:13:50.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.902+0000 7f277b800640 1 -- 172.21.15.16:0/2295598442 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x56101503a4e0 con 0x561015f17400 2024-03-31T15:13:50.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.902+0000 7f2784f31400 1 -- 172.21.15.16:0/2295598442 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x561015dbfe00 con 0x561015f17400 2024-03-31T15:13:50.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.902+0000 7f2784f31400 1 -- 172.21.15.16:0/2295598442 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561015dbfe00 con 0x561015f17400 2024-03-31T15:13:50.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.902+0000 7f2784f31400 4 main: resolving pool 14 2024-03-31T15:13:50.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.902+0000 7f277b800640 1 -- 172.21.15.16:0/2295598442 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x561015f8cc00 con 0x561015f17400 2024-03-31T15:13:50.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.906+0000 7f277b800640 1 --2- 172.21.15.16:0/2295598442 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x5610152f8c00 0x561015dc2100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:50.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.906+0000 7f277b800640 1 -- 172.21.15.16:0/2295598442 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 5881+0+0 (secure 0 0 0) 0x5610152fa800 con 0x561015f17400 2024-03-31T15:13:50.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.906+0000 7f2784f31400 4 main: creating IoCtx.. 2024-03-31T15:13:50.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.906+0000 7f2784f31400 1 --2- 172.21.15.16:0/2295598442 >> [v2:172.21.15.16:6819/3018526646,v1:172.21.15.16:6822/3018526646] conn(0x561015f17c00 0x561015dc2680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:50.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.906+0000 7f2784f31400 1 -- 172.21.15.16:0/2295598442 --> [v2:172.21.15.16:6819/3018526646,v1:172.21.15.16:6822/3018526646] -- osd_op(unknown.0.0:1 14.13 14:c91d4a1d:::mds0_inotable:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e98) -- 0x5610152f9000 con 0x561015f17c00 2024-03-31T15:13:50.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.906+0000 7f2783a00640 1 --2- 172.21.15.16:0/2295598442 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x5610152f8c00 0x561015dc2100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:50.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.906+0000 7f2782600640 1 --2- 172.21.15.16:0/2295598442 >> [v2:172.21.15.16:6819/3018526646,v1:172.21.15.16:6822/3018526646] conn(0x561015f17c00 0x561015dc2680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:50.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.906+0000 7f2783a00640 1 --2- 172.21.15.16:0/2295598442 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x5610152f8c00 0x561015dc2100 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x561015f8a060 tx=0x561015dd23c0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:13:50.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.906+0000 7f2782600640 1 --2- 172.21.15.16:0/2295598442 >> [v2:172.21.15.16:6819/3018526646,v1:172.21.15.16:6822/3018526646] conn(0x561015f17c00 0x561015dc2680 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:13:50.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.910+0000 7f2782600640 1 -- 172.21.15.16:0/2295598442 <== osd.2 v2:172.21.15.16:6819/3018526646 1 ==== osd_op_reply(1 mds0_inotable [read 0~34 out=34b] v0'0 uv7 ondisk = 0) ==== 157+0+34 (crc 0 0 0) 0x561015d80000 con 0x561015f17c00 2024-03-31T15:13:50.902 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-03-31T15:13:50.902 INFO:teuthology.orchestra.run.smithi016.stdout: "0": { 2024-03-31T15:13:50.902 INFO:teuthology.orchestra.run.smithi016.stdout: "data": { 2024-03-31T15:13:50.902 INFO:teuthology.orchestra.run.smithi016.stdout: "version": 5, 2024-03-31T15:13:50.902 INFO:teuthology.orchestra.run.smithi016.stdout: "inotable": { 2024-03-31T15:13:50.902 INFO:teuthology.orchestra.run.smithi016.stdout: "projected_free": [ 2024-03-31T15:13:50.902 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-31T15:13:50.902 INFO:teuthology.orchestra.run.smithi016.stdout: "start": 1099511627779, 2024-03-31T15:13:50.902 INFO:teuthology.orchestra.run.smithi016.stdout: "len": 1099511627773 2024-03-31T15:13:50.902 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-31T15:13:50.902 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-03-31T15:13:50.903 INFO:teuthology.orchestra.run.smithi016.stdout: "free": [ 2024-03-31T15:13:50.903 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-31T15:13:50.903 INFO:teuthology.orchestra.run.smithi016.stdout: "start": 1099511627779, 2024-03-31T15:13:50.903 INFO:teuthology.orchestra.run.smithi016.stdout: "len": 1099511627773 2024-03-31T15:13:50.903 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-31T15:13:50.903 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-03-31T15:13:50.903 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-31T15:13:50.903 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-31T15:13:50.903 INFO:teuthology.orchestra.run.smithi016.stdout: "result": 0 2024-03-31T15:13:50.903 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-31T15:13:50.903 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-03-31T15:13:50.903 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:13:50.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.910+0000 7f2784f31400 1 -- 172.21.15.16:0/2295598442 >> [v2:172.21.15.16:6819/3018526646,v1:172.21.15.16:6822/3018526646] conn(0x561015f17c00 msgr2=0x561015dc2680 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:13:50.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.910+0000 7f2784f31400 1 --2- 172.21.15.16:0/2295598442 >> [v2:172.21.15.16:6819/3018526646,v1:172.21.15.16:6822/3018526646] conn(0x561015f17c00 0x561015dc2680 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:50.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.910+0000 7f2784f31400 1 -- 172.21.15.16:0/2295598442 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x5610152f8c00 msgr2=0x561015dc2100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:13:50.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.910+0000 7f2784f31400 1 --2- 172.21.15.16:0/2295598442 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x5610152f8c00 0x561015dc2100 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x561015f8a060 tx=0x561015dd23c0 comp rx=0 tx=0).stop 2024-03-31T15:13:50.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.910+0000 7f2784f31400 1 -- 172.21.15.16:0/2295598442 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561015f17400 msgr2=0x561015dc1600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:13:50.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.910+0000 7f2784f31400 1 --2- 172.21.15.16:0/2295598442 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561015f17400 0x561015dc1600 secure :-1 s=READY pgs=1564 cs=0 l=1 rev1=1 crypto rx=0x561015f8b440 tx=0x5610152fc000 comp rx=0 tx=0).stop 2024-03-31T15:13:50.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.910+0000 7f2783000640 1 -- 172.21.15.16:0/2295598442 reap_dead start 2024-03-31T15:13:50.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.910+0000 7f2784f31400 1 -- 172.21.15.16:0/2295598442 shutdown_connections 2024-03-31T15:13:50.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.910+0000 7f2784f31400 1 -- 172.21.15.16:0/2295598442 >> 172.21.15.16:0/2295598442 conn(0x561015004c00 msgr2=0x561015033800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:13:50.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.910+0000 7f2784f31400 1 -- 172.21.15.16:0/2295598442 shutdown_connections 2024-03-31T15:13:50.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.910+0000 7f2784f31400 1 -- 172.21.15.16:0/2295598442 wait complete. 2024-03-31T15:13:50.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.910+0000 7f2784f31400 1 -- 172.21.15.16:0/668127171 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561015005400 msgr2=0x561015dc0580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:13:50.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.910+0000 7f2784f31400 1 --2- 172.21.15.16:0/668127171 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561015005400 0x561015dc0580 secure :-1 s=READY pgs=1562 cs=0 l=1 rev1=1 crypto rx=0x561015e7cd20 tx=0x561015d84c60 comp rx=0 tx=0).stop 2024-03-31T15:13:50.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.910+0000 7f2784f31400 1 -- 172.21.15.16:0/668127171 shutdown_connections 2024-03-31T15:13:50.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.910+0000 7f2784f31400 1 --2- 172.21.15.16:0/668127171 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561015005000 0x561015dc0b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:50.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.910+0000 7f2784f31400 1 --2- 172.21.15.16:0/668127171 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561015005400 0x561015dc0580 unknown :-1 s=CLOSED pgs=1562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:50.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.910+0000 7f2784f31400 1 --2- 172.21.15.16:0/668127171 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x561015005800 0x561015dc0000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:50.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.910+0000 7f2784f31400 1 -- 172.21.15.16:0/668127171 >> 172.21.15.16:0/668127171 conn(0x561015005c00 msgr2=0x561015033000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:13:50.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.910+0000 7f2784f31400 1 -- 172.21.15.16:0/668127171 shutdown_connections 2024-03-31T15:13:50.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:50.910+0000 7f2784f31400 1 -- 172.21.15.16:0/668127171 wait complete. 2024-03-31T15:13:50.907 DEBUG:tasks.cephfs.filesystem:Ran ['cephfs-table-tool', '--debug-mds=20', '--debug-ms=1', '--debug-objecter=1', '0', 'show', 'inode'] in time 0:00:00.065784, result: { "0": { "data": { "version": 5, "inotable": { "projected_free": [ { "start": 1099511627779, "len": 1099511627773 } ], "free": [ { "start": 1099511627779, "len": 1099511627773 } ] } }, "result": 0 } } 2024-03-31T15:13:50.907 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-31T15:13:50.908 DEBUG:teuthology.orchestra.run.smithi016:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-31T15:13:51.019 DEBUG:teuthology.orchestra.run.smithi089:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-31T15:13:51.112 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-31T15:13:51.112 INFO:tasks.cephfs.mount:Terminating background process 2024-03-31T15:13:51.112 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-03-31T15:13:51.112 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-31T15:13:51.121 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16357708k,nr_inodes=4089427,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275996k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=6539 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275992k,nr_inodes=818998,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275996k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-31T15:13:51.121 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi016.front.sepia.ceph.com 2024-03-31T15:13:51.121 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:13:51.121 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-31T15:13:51.172 INFO:teuthology.orchestra.run.smithi016.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-03-31T15:13:51.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-31T15:13:51.173 INFO:tasks.cephfs.mount:Terminating background process 2024-03-31T15:13:51.173 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-03-31T15:13:51.173 DEBUG:teuthology.orchestra.run.smithi089:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-31T15:13:51.183 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16354452k,nr_inodes=4088613,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275344k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=901 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275340k,nr_inodes=818835,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275344k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-31T15:13:51.183 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi089.front.sepia.ceph.com 2024-03-31T15:13:51.183 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:13:51.183 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-03-31T15:13:51.237 INFO:teuthology.orchestra.run.smithi089.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.1': No such file or directory 2024-03-31T15:13:51.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-31T15:13:51.239 INFO:tasks.cephfs.mount:Terminating background process 2024-03-31T15:13:51.239 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-03-31T15:13:51.239 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-31T15:13:51.247 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16357708k,nr_inodes=4089427,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275996k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=6539 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275992k,nr_inodes=818998,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275996k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-31T15:13:51.247 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi016.front.sepia.ceph.com 2024-03-31T15:13:51.248 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:13:51.248 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-03-31T15:13:51.296 INFO:teuthology.orchestra.run.smithi016.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2024-03-31T15:13:51.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-31T15:13:51.297 INFO:tasks.cephfs.mount:Terminating background process 2024-03-31T15:13:51.297 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-03-31T15:13:51.297 DEBUG:teuthology.orchestra.run.smithi089:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-31T15:13:51.307 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16354452k,nr_inodes=4088613,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275344k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=901 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275340k,nr_inodes=818835,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275344k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-31T15:13:51.307 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi089.front.sepia.ceph.com 2024-03-31T15:13:51.307 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:13:51.307 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-03-31T15:13:51.357 INFO:teuthology.orchestra.run.smithi089.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2024-03-31T15:13:51.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-31T15:13:51.358 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-31T15:13:51.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:51.506+0000 7f1fdc800640 1 Processor -- start 2024-03-31T15:13:51.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:51.506+0000 7f1fdc800640 1 -- start start 2024-03-31T15:13:51.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:51.506+0000 7f1fdc800640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f1fd40cef40 0x7f1fd40b5b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:51.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:51.506+0000 7f1fdc800640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1fd40ce8a0 con 0x7f1fd415b4d0 2024-03-31T15:13:51.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:51.506+0000 7f1fdc800640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1fd40709a0 con 0x7f1fd40b61e0 2024-03-31T15:13:51.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:51.506+0000 7f1fdc800640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f1fd406fbe0 con 0x7f1fd40cef40 2024-03-31T15:13:51.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:51.506+0000 7f1fd9400640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f1fd415b4d0 0x7f1fd40cf320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:44746/0 (socket says 172.21.15.16:44746) 2024-03-31T15:13:51.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:51.506+0000 7f1fd9400640 1 -- 172.21.15.16:0/792621888 learned_addr learned my addr 172.21.15.16:0/792621888 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:13:51.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:51.506+0000 7f1fd9e00640 1 --2- 172.21.15.16:0/792621888 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f1fd40cef40 0x7f1fd40b5b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:51.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:51.506+0000 7f1fd8a00640 1 -- 172.21.15.16:0/792621888 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 741688498 0 0) 0x7f1fd40ce8a0 con 0x7f1fd415b4d0 2024-03-31T15:13:51.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:51.506+0000 7f1fd8a00640 1 -- 172.21.15.16:0/792621888 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1fbc003530 con 0x7f1fd415b4d0 2024-03-31T15:13:51.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:51.506+0000 7f1fd8a00640 1 -- 172.21.15.16:0/792621888 <== mon.1 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 689723651 0 0) 0x7f1fd40709a0 con 0x7f1fd40b61e0 2024-03-31T15:13:51.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:51.506+0000 7f1fd8a00640 1 -- 172.21.15.16:0/792621888 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1fd40ce8a0 con 0x7f1fd40b61e0 2024-03-31T15:13:51.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:51.506+0000 7f1fd8a00640 1 -- 172.21.15.16:0/792621888 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1340040381 0 0) 0x7f1fbc003530 con 0x7f1fd415b4d0 2024-03-31T15:13:51.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:51.506+0000 7f1fd8a00640 1 -- 172.21.15.16:0/792621888 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f1fd40cef40 msgr2=0x7f1fd40b5b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:13:51.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:51.506+0000 7f1fd8a00640 1 --2- 172.21.15.16:0/792621888 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f1fd40cef40 0x7f1fd40b5b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:51.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:51.506+0000 7f1fd8a00640 1 -- 172.21.15.16:0/792621888 >> v1:172.21.15.89:6789/0 conn(0x7f1fd40b61e0 legacy=0x7f1fd40b42e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:13:51.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:51.506+0000 7f1fd8a00640 1 -- 172.21.15.16:0/792621888 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1fd406fec0 con 0x7f1fd415b4d0 2024-03-31T15:13:51.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:51.506+0000 7f1fd8a00640 1 -- 172.21.15.16:0/792621888 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 214807623 0 0) 0x7f1fc4001680 con 0x7f1fd415b4d0 2024-03-31T15:13:51.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:51.510+0000 7f1fd8a00640 1 -- 172.21.15.16:0/792621888 >> v1:172.21.15.16:6789/0 conn(0x7f1fd415b4d0 legacy=0x7f1fd40cf320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:13:51.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:51.510+0000 7f1fd8a00640 1 --2- 172.21.15.16:0/792621888 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1fbc003b80 0x7f1fbc003f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:51.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:51.510+0000 7f1fd8a00640 1 --2- 172.21.15.16:0/792621888 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f1fd40cef40 0x7f1fbc004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:51.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:51.510+0000 7f1fd8a00640 1 --2- 172.21.15.16:0/792621888 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1fbc004d50 0x7f1fbc0051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:51.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:51.510+0000 7f1fd8a00640 1 -- 172.21.15.16:0/792621888 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f1fd406fbe0 con 0x7f1fbc003b80 2024-03-31T15:13:51.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:51.510+0000 7f1fd8a00640 1 -- 172.21.15.16:0/792621888 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f1fbc00ba80 con 0x7f1fbc004d50 2024-03-31T15:13:51.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:51.510+0000 7f1fd8a00640 1 -- 172.21.15.16:0/792621888 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f1fbc00bc80 con 0x7f1fd40cef40 2024-03-31T15:13:51.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:51.510+0000 7f1fd9e00640 1 --2- 172.21.15.16:0/792621888 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1fbc003b80 0x7f1fbc003f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:51.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:51.510+0000 7f1fd9400640 1 --2- 172.21.15.16:0/792621888 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f1fd40cef40 0x7f1fbc004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:51.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:51.510+0000 7f1fda800640 1 --2- 172.21.15.16:0/792621888 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1fbc004d50 0x7f1fbc0051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:51.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:51.510+0000 7f1fd9e00640 1 -- 172.21.15.16:0/792621888 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f1fd40cef40 msgr2=0x7f1fbc004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:13:51.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:51.510+0000 7f1fd9e00640 1 --2- 172.21.15.16:0/792621888 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f1fd40cef40 0x7f1fbc004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:51.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:51.510+0000 7f1fd9e00640 1 -- 172.21.15.16:0/792621888 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1fbc004d50 msgr2=0x7f1fbc0051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:13:51.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:51.510+0000 7f1fd9e00640 1 --2- 172.21.15.16:0/792621888 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1fbc004d50 0x7f1fbc0051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:51.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:51.510+0000 7f1fd9e00640 1 -- 172.21.15.16:0/792621888 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1fbc003530 con 0x7f1fbc003b80 2024-03-31T15:13:51.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:51.510+0000 7f1fd9e00640 1 --2- 172.21.15.16:0/792621888 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1fbc003b80 0x7f1fbc003f80 secure :-1 s=READY pgs=1566 cs=0 l=1 rev1=1 crypto rx=0x7f1fc8009ae0 tx=0x7f1fc801ca40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55245367332f705a server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:13:51.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:51.510+0000 7f1fd8a00640 1 -- 172.21.15.16:0/792621888 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fc80b2070 con 0x7f1fbc003b80 2024-03-31T15:13:51.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:51.510+0000 7f1fd8a00640 1 -- 172.21.15.16:0/792621888 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1fc80a5520 con 0x7f1fbc003b80 2024-03-31T15:13:51.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:51.510+0000 7f1fd8a00640 1 -- 172.21.15.16:0/792621888 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fc80096b0 con 0x7f1fbc003b80 2024-03-31T15:13:51.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:51.514+0000 7f1fdc800640 1 -- 172.21.15.16:0/792621888 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1fbc003b80 msgr2=0x7f1fbc003f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:13:51.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:51.514+0000 7f1fdc800640 1 --2- 172.21.15.16:0/792621888 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1fbc003b80 0x7f1fbc003f80 secure :-1 s=READY pgs=1566 cs=0 l=1 rev1=1 crypto rx=0x7f1fc8009ae0 tx=0x7f1fc801ca40 comp rx=0 tx=0).stop 2024-03-31T15:13:51.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:51.514+0000 7f1fda800640 1 -- 172.21.15.16:0/792621888 reap_dead start 2024-03-31T15:13:51.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:51.514+0000 7f1fdc800640 1 -- 172.21.15.16:0/792621888 shutdown_connections 2024-03-31T15:13:51.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:51.514+0000 7f1fdc800640 1 -- 172.21.15.16:0/792621888 >> 172.21.15.16:0/792621888 conn(0x7f1fd40d1060 msgr2=0x7f1fd40d1470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:13:51.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:51.514+0000 7f1fdc800640 1 -- 172.21.15.16:0/792621888 shutdown_connections 2024-03-31T15:13:51.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:51.514+0000 7f1fdc800640 1 -- 172.21.15.16:0/792621888 wait complete. 2024-03-31T15:13:51.507 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:51.514+0000 7f1fdc800640 1 Processor -- start 2024-03-31T15:13:51.507 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:51.514+0000 7f1fdc800640 1 -- start start 2024-03-31T15:13:51.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:51.514+0000 7f1fdc800640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1fbc003b80 0x7f1fd4158140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:51.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:51.514+0000 7f1fdc800640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f1fd4157860 0x7f1fd415e5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:51.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:51.514+0000 7f1fdc800640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1fd41476c0 0x7f1fd4147b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:51.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:51.514+0000 7f1fdc800640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f1fd40b5380 con 0x7f1fd41476c0 2024-03-31T15:13:51.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:51.514+0000 7f1fdc800640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f1fd40b5200 con 0x7f1fbc003b80 2024-03-31T15:13:51.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:51.514+0000 7f1fdc800640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f1fd40b5500 con 0x7f1fd4157860 2024-03-31T15:13:51.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:51.514+0000 7f1fda800640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1fd41476c0 0x7f1fd4147b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:51.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:51.514+0000 7f1fda800640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1fd41476c0 0x7f1fd4147b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50264/0 (socket says 172.21.15.16:50264) 2024-03-31T15:13:51.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:51.514+0000 7f1fda800640 1 -- 172.21.15.16:0/3804993758 learned_addr learned my addr 172.21.15.16:0/3804993758 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:13:51.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:51.514+0000 7f1fd9400640 1 --2- 172.21.15.16:0/3804993758 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f1fd4157860 0x7f1fd415e5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:51.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:51.514+0000 7f1fd9e00640 1 --2- 172.21.15.16:0/3804993758 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1fbc003b80 0x7f1fd4158140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:51.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:51.514+0000 7f1fda800640 1 -- 172.21.15.16:0/3804993758 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f1fd4157860 msgr2=0x7f1fd415e5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:13:51.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:51.514+0000 7f1fda800640 1 --2- 172.21.15.16:0/3804993758 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f1fd4157860 0x7f1fd415e5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:51.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:51.514+0000 7f1fda800640 1 -- 172.21.15.16:0/3804993758 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1fbc003b80 msgr2=0x7f1fd4158140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:13:51.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:51.514+0000 7f1fda800640 1 --2- 172.21.15.16:0/3804993758 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1fbc003b80 0x7f1fd4158140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:51.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:51.514+0000 7f1fda800640 1 -- 172.21.15.16:0/3804993758 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1fd4158720 con 0x7f1fd41476c0 2024-03-31T15:13:51.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:51.514+0000 7f1fda800640 1 --2- 172.21.15.16:0/3804993758 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1fd41476c0 0x7f1fd4147b20 secure :-1 s=READY pgs=1567 cs=0 l=1 rev1=1 crypto rx=0x7f1fd000e690 tx=0x7f1fd000eb50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:13:51.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:51.518+0000 7f1fce000640 1 -- 172.21.15.16:0/3804993758 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fd0003980 con 0x7f1fd41476c0 2024-03-31T15:13:51.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:51.518+0000 7f1fce000640 1 -- 172.21.15.16:0/3804993758 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1fd0018070 con 0x7f1fd41476c0 2024-03-31T15:13:51.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:51.518+0000 7f1fce000640 1 -- 172.21.15.16:0/3804993758 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fd0003d60 con 0x7f1fd41476c0 2024-03-31T15:13:51.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:51.518+0000 7f1fdc800640 1 -- 172.21.15.16:0/3804993758 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1fd4158a00 con 0x7f1fd41476c0 2024-03-31T15:13:51.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:51.518+0000 7f1fdc800640 1 -- 172.21.15.16:0/3804993758 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1fd4153950 con 0x7f1fd41476c0 2024-03-31T15:13:51.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:51.518+0000 7f1fdc800640 1 -- 172.21.15.16:0/3804993758 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1fd4069950 con 0x7f1fd41476c0 2024-03-31T15:13:51.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:51.518+0000 7f1fce000640 1 -- 172.21.15.16:0/3804993758 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f1fd00050d0 con 0x7f1fd41476c0 2024-03-31T15:13:51.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:51.518+0000 7f1fce000640 1 --2- 172.21.15.16:0/3804993758 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f1fb80683b0 0x7f1fb806a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:51.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:51.518+0000 7f1fce000640 1 -- 172.21.15.16:0/3804993758 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 5881+0+0 (secure 0 0 0) 0x7f1fd0086620 con 0x7f1fd41476c0 2024-03-31T15:13:51.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:51.518+0000 7f1fd9e00640 1 --2- 172.21.15.16:0/3804993758 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f1fb80683b0 0x7f1fb806a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:51.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:51.522+0000 7f1fd9e00640 1 --2- 172.21.15.16:0/3804993758 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f1fb80683b0 0x7f1fb806a870 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f1fc80af000 tx=0x7f1fc80046d0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:13:51.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:51.522+0000 7f1fce000640 1 -- 172.21.15.16:0/3804993758 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f1fd00564d0 con 0x7f1fd41476c0 2024-03-31T15:13:51.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:51.722+0000 7f1fdc800640 1 -- 172.21.15.16:0/3804993758 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1fd4070c00 con 0x7f1fd41476c0 2024-03-31T15:13:51.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:51.722+0000 7f1fce000640 1 -- 172.21.15.16:0/3804993758 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 90 v90) ==== 94+0+4344 (secure 0 0 0) 0x7f1fd0084e20 con 0x7f1fd41476c0 2024-03-31T15:13:51.717 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:13:51.717 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":90,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7139,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6836/3798928623","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":3798928623},{"type":"v1","addr":"172.21.15.89:6836","nonce":3798928623}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":78},{"gid":7142,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/3074283817","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6835","nonce":3074283817},{"type":"v1","addr":"172.21.15.89:6837","nonce":3074283817}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":78},{"gid":8071,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6834/2110189998","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":2110189998},{"type":"v1","addr":"172.21.15.16:6834","nonce":2110189998}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":78}],"filesystems":[{"mdsmap":{"epoch":90,"flags":19,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-31T15:12:16.294600+0000","modified":"2024-03-31T15:13:50.823935+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":98,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":7}]} 2024-03-31T15:13:51.717 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 90 2024-03-31T15:13:51.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:51.726+0000 7f1fdc800640 1 -- 172.21.15.16:0/3804993758 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f1fb80683b0 msgr2=0x7f1fb806a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:13:51.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:51.726+0000 7f1fdc800640 1 --2- 172.21.15.16:0/3804993758 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f1fb80683b0 0x7f1fb806a870 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f1fc80af000 tx=0x7f1fc80046d0 comp rx=0 tx=0).stop 2024-03-31T15:13:51.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:51.726+0000 7f1fdc800640 1 -- 172.21.15.16:0/3804993758 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1fd41476c0 msgr2=0x7f1fd4147b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:13:51.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:51.726+0000 7f1fdc800640 1 --2- 172.21.15.16:0/3804993758 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1fd41476c0 0x7f1fd4147b20 secure :-1 s=READY pgs=1567 cs=0 l=1 rev1=1 crypto rx=0x7f1fd000e690 tx=0x7f1fd000eb50 comp rx=0 tx=0).stop 2024-03-31T15:13:51.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:51.726+0000 7f1fdc800640 1 -- 172.21.15.16:0/3804993758 shutdown_connections 2024-03-31T15:13:51.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:51.726+0000 7f1fdc800640 1 --2- 172.21.15.16:0/3804993758 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f1fb80683b0 0x7f1fb806a870 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:51.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:51.726+0000 7f1fdc800640 1 --2- 172.21.15.16:0/3804993758 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1fd41476c0 0x7f1fd4147b20 unknown :-1 s=CLOSED pgs=1567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:51.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:51.726+0000 7f1fdc800640 1 --2- 172.21.15.16:0/3804993758 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f1fd4157860 0x7f1fd415e5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:51.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:51.726+0000 7f1fdc800640 1 --2- 172.21.15.16:0/3804993758 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1fbc003b80 0x7f1fd4158140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:51.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:51.726+0000 7f1fdc800640 1 -- 172.21.15.16:0/3804993758 >> 172.21.15.16:0/3804993758 conn(0x7f1fd40d1060 msgr2=0x7f1fd4072610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:13:51.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:51.726+0000 7f1fdc800640 1 -- 172.21.15.16:0/3804993758 shutdown_connections 2024-03-31T15:13:51.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:51.726+0000 7f1fdc800640 1 -- 172.21.15.16:0/3804993758 wait complete. 2024-03-31T15:13:51.739 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-31T15:13:51.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:51.886+0000 7f92a9e00640 1 Processor -- start 2024-03-31T15:13:51.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:51.886+0000 7f92a9e00640 1 -- start start 2024-03-31T15:13:51.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:51.886+0000 7f92a9e00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f92a40cef40 0x7f92a40b5b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:51.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:51.886+0000 7f92a9e00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f92a415b4d0 0x7f92a40cf320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:51.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:51.886+0000 7f92a9e00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f92a40b61e0 0x7f92a415ee10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:51.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:51.886+0000 7f92a9e00640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f92a40ce8a0 con 0x7f92a40cef40 2024-03-31T15:13:51.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:51.886+0000 7f92a9e00640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f92a40cf860 con 0x7f92a40b61e0 2024-03-31T15:13:51.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:51.886+0000 7f92a9e00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f92a40709a0 con 0x7f92a415b4d0 2024-03-31T15:13:51.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:51.886+0000 7f92a2a00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f92a40cef40 0x7f92a40b5b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:51.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:51.886+0000 7f92a2a00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f92a40cef40 0x7f92a40b5b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50266/0 (socket says 172.21.15.16:50266) 2024-03-31T15:13:51.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:51.886+0000 7f92a2a00640 1 -- 172.21.15.16:0/2887598554 learned_addr learned my addr 172.21.15.16:0/2887598554 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:13:51.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:51.886+0000 7f92a3400640 1 --2- 172.21.15.16:0/2887598554 >> v2:172.21.15.89:3300/0 conn(0x7f92a40b61e0 0x7f92a415ee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:51.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:51.886+0000 7f929be00640 1 --2- 172.21.15.16:0/2887598554 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f92a415b4d0 0x7f92a40cf320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:51.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:51.890+0000 7f92a2a00640 1 -- 172.21.15.16:0/2887598554 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f92a415b4d0 msgr2=0x7f92a40cf320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:13:51.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:51.890+0000 7f92a2a00640 1 --2- 172.21.15.16:0/2887598554 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f92a415b4d0 0x7f92a40cf320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:51.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:51.890+0000 7f92a2a00640 1 -- 172.21.15.16:0/2887598554 >> v2:172.21.15.89:3300/0 conn(0x7f92a40b61e0 msgr2=0x7f92a415ee10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:13:51.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:51.890+0000 7f92a2a00640 1 --2- 172.21.15.16:0/2887598554 >> v2:172.21.15.89:3300/0 conn(0x7f92a40b61e0 0x7f92a415ee10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:51.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:51.890+0000 7f92a2a00640 1 -- 172.21.15.16:0/2887598554 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92a406fbe0 con 0x7f92a40cef40 2024-03-31T15:13:51.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:51.890+0000 7f92a3400640 1 --2- 172.21.15.16:0/2887598554 >> v2:172.21.15.89:3300/0 conn(0x7f92a40b61e0 0x7f92a415ee10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:13:51.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:51.890+0000 7f929be00640 1 --2- 172.21.15.16:0/2887598554 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f92a415b4d0 0x7f92a40cf320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:13:51.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:51.890+0000 7f92a2a00640 1 --2- 172.21.15.16:0/2887598554 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f92a40cef40 0x7f92a40b5b50 secure :-1 s=READY pgs=1568 cs=0 l=1 rev1=1 crypto rx=0x7f92a40d1c10 tx=0x7f929401c230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d50bd1092165729f server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:13:51.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:51.890+0000 7f92a2000640 1 -- 172.21.15.16:0/2887598554 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92940b0070 con 0x7f92a40cef40 2024-03-31T15:13:51.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:51.890+0000 7f92a2000640 1 -- 172.21.15.16:0/2887598554 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f92940ab600 con 0x7f92a40cef40 2024-03-31T15:13:51.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:51.890+0000 7f92a2000640 1 -- 172.21.15.16:0/2887598554 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92940ab9d0 con 0x7f92a40cef40 2024-03-31T15:13:51.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:51.890+0000 7f92a9e00640 1 -- 172.21.15.16:0/2887598554 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f92a40cef40 msgr2=0x7f92a40b5b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:13:51.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:51.890+0000 7f92a9e00640 1 --2- 172.21.15.16:0/2887598554 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f92a40cef40 0x7f92a40b5b50 secure :-1 s=READY pgs=1568 cs=0 l=1 rev1=1 crypto rx=0x7f92a40d1c10 tx=0x7f929401c230 comp rx=0 tx=0).stop 2024-03-31T15:13:51.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:51.890+0000 7f92a9e00640 1 -- 172.21.15.16:0/2887598554 shutdown_connections 2024-03-31T15:13:51.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:51.890+0000 7f92a9e00640 1 --2- 172.21.15.16:0/2887598554 >> v2:172.21.15.89:3300/0 conn(0x7f92a40b61e0 0x7f92a415ee10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:51.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:51.890+0000 7f92a9e00640 1 --2- 172.21.15.16:0/2887598554 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f92a415b4d0 0x7f92a40cf320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:51.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:51.890+0000 7f92a9e00640 1 --2- 172.21.15.16:0/2887598554 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f92a40cef40 0x7f92a40b5b50 unknown :-1 s=CLOSED pgs=1568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:51.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:51.890+0000 7f92a9e00640 1 -- 172.21.15.16:0/2887598554 >> 172.21.15.16:0/2887598554 conn(0x7f92a40d1060 msgr2=0x7f92a40d1470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:13:51.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:51.890+0000 7f92a9e00640 1 -- 172.21.15.16:0/2887598554 shutdown_connections 2024-03-31T15:13:51.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:51.894+0000 7f92a9e00640 1 -- 172.21.15.16:0/2887598554 wait complete. 2024-03-31T15:13:51.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:51.894+0000 7f92a9e00640 1 Processor -- start 2024-03-31T15:13:51.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:51.894+0000 7f92a9e00640 1 -- start start 2024-03-31T15:13:51.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:51.894+0000 7f92a9e00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f92a40b61e0 0x7f92a414c9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:51.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:51.894+0000 7f92a9e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f92a40cef40 0x7f92a415b8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:51.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:51.894+0000 7f92a9e00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f92a415b4d0 0x7f92a4158da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:51.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:51.894+0000 7f92a9e00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f92a40ce640 con 0x7f92a40cef40 2024-03-31T15:13:51.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:51.894+0000 7f92a9e00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f92a40ce4c0 con 0x7f92a40b61e0 2024-03-31T15:13:51.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:51.894+0000 7f92a9e00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f92a406b620 con 0x7f92a415b4d0 2024-03-31T15:13:51.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:51.894+0000 7f929be00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f92a40cef40 0x7f92a415b8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:51.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:51.894+0000 7f929be00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f92a40cef40 0x7f92a415b8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50276/0 (socket says 172.21.15.16:50276) 2024-03-31T15:13:51.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:51.894+0000 7f929be00640 1 -- 172.21.15.16:0/3597887451 learned_addr learned my addr 172.21.15.16:0/3597887451 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:13:51.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:51.894+0000 7f92a2a00640 1 --2- 172.21.15.16:0/3597887451 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f92a40b61e0 0x7f92a414c9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:51.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:51.894+0000 7f92a3400640 1 --2- 172.21.15.16:0/3597887451 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f92a415b4d0 0x7f92a4158da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:51.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:51.894+0000 7f929be00640 1 -- 172.21.15.16:0/3597887451 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f92a415b4d0 msgr2=0x7f92a4158da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:13:51.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:51.894+0000 7f929be00640 1 --2- 172.21.15.16:0/3597887451 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f92a415b4d0 0x7f92a4158da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:51.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:51.894+0000 7f929be00640 1 -- 172.21.15.16:0/3597887451 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f92a40b61e0 msgr2=0x7f92a414c9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:13:51.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:51.894+0000 7f929be00640 1 --2- 172.21.15.16:0/3597887451 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f92a40b61e0 0x7f92a414c9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:51.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:51.894+0000 7f929be00640 1 -- 172.21.15.16:0/3597887451 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92a4166970 con 0x7f92a40cef40 2024-03-31T15:13:51.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:51.894+0000 7f92a2a00640 1 --2- 172.21.15.16:0/3597887451 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f92a40b61e0 0x7f92a414c9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:13:51.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:51.894+0000 7f929be00640 1 --2- 172.21.15.16:0/3597887451 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f92a40cef40 0x7f92a415b8b0 secure :-1 s=READY pgs=1569 cs=0 l=1 rev1=1 crypto rx=0x7f928c00c7f0 tx=0x7f928c00ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:13:51.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:51.894+0000 7f929b400640 1 -- 172.21.15.16:0/3597887451 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f928c015070 con 0x7f92a40cef40 2024-03-31T15:13:51.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:51.894+0000 7f929b400640 1 -- 172.21.15.16:0/3597887451 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f928c007820 con 0x7f92a40cef40 2024-03-31T15:13:51.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:51.894+0000 7f92a9e00640 1 -- 172.21.15.16:0/3597887451 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f92a4166c50 con 0x7f92a40cef40 2024-03-31T15:13:51.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:51.894+0000 7f929b400640 1 -- 172.21.15.16:0/3597887451 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f928c007c10 con 0x7f92a40cef40 2024-03-31T15:13:51.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:51.898+0000 7f92a9e00640 1 -- 172.21.15.16:0/3597887451 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f92a4167210 con 0x7f92a40cef40 2024-03-31T15:13:51.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:51.898+0000 7f92a9e00640 1 -- 172.21.15.16:0/3597887451 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f92a4069950 con 0x7f92a40cef40 2024-03-31T15:13:51.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:51.898+0000 7f929b400640 1 -- 172.21.15.16:0/3597887451 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f928c018070 con 0x7f92a40cef40 2024-03-31T15:13:51.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:51.898+0000 7f929b400640 1 --2- 172.21.15.16:0/3597887451 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f9280068380 0x7f928006a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:51.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:51.898+0000 7f929b400640 1 -- 172.21.15.16:0/3597887451 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 5881+0+0 (secure 0 0 0) 0x7f928c086c00 con 0x7f92a40cef40 2024-03-31T15:13:51.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:51.898+0000 7f92a2a00640 1 --2- 172.21.15.16:0/3597887451 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f9280068380 0x7f928006a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:51.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:51.902+0000 7f92a2a00640 1 --2- 172.21.15.16:0/3597887451 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f9280068380 0x7f928006a840 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f92940ad000 tx=0x7f92940a6000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:13:51.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:51.906+0000 7f929b400640 1 -- 172.21.15.16:0/3597887451 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f928c057070 con 0x7f92a40cef40 2024-03-31T15:13:52.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:52.070+0000 7f92a9e00640 1 -- 172.21.15.16:0/3597887451 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f92a41673e0 con 0x7f92a40cef40 2024-03-31T15:13:52.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:52.070+0000 7f929b400640 1 -- 172.21.15.16:0/3597887451 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v98) ==== 74+0+16791 (secure 0 0 0) 0x7f928c056970 con 0x7f92a40cef40 2024-03-31T15:13:52.063 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:13:52.064 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":98,"fsid":"286eb5da-fb47-4550-b290-d9519c13b9a5","created":"2024-03-31T15:03:11.438952+0000","modified":"2024-03-31T15:13:50.823773+0000","last_up_change":"2024-03-31T15:03:22.885535+0000","last_in_change":"2024-03-31T15:03:13.941667+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-31T15:03:22.550787+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2024-03-31T15:12:14.671225+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"95","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2024-03-31T15:12:15.242196+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"96","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"92a66450-ef19-40d8-ac3d-d760b23b0bda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6801","nonce":932712244}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6803","nonce":932712244}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6807","nonce":932712244}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6805","nonce":932712244}]},"public_addr":"172.21.15.16:6801/932712244","cluster_addr":"172.21.15.16:6803/932712244","heartbeat_back_addr":"172.21.15.16:6807/932712244","heartbeat_front_addr":"172.21.15.16:6805/932712244","state":["exists","up"]},{"osd":1,"uuid":"da9d1f2b-1968-4959-b843-faa231988e92","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6810","nonce":1398737533}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6811","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6813","nonce":1398737533}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6821","nonce":1398737533}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6815","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6816","nonce":1398737533}]},"public_addr":"172.21.15.16:6810/1398737533","cluster_addr":"172.21.15.16:6813/1398737533","heartbeat_back_addr":"172.21.15.16:6821/1398737533","heartbeat_front_addr":"172.21.15.16:6816/1398737533","state":["exists","up"]},{"osd":2,"uuid":"01a73662-9a21-4b1f-bcbb-42b9636a4638","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6819","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6822","nonce":3018526646}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6825","nonce":3018526646}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6831","nonce":3018526646}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6827","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6829","nonce":3018526646}]},"public_addr":"172.21.15.16:6822/3018526646","cluster_addr":"172.21.15.16:6825/3018526646","heartbeat_back_addr":"172.21.15.16:6831/3018526646","heartbeat_front_addr":"172.21.15.16:6829/3018526646","state":["exists","up"]},{"osd":3,"uuid":"c7771fd3-e18e-4500-ac38-23cf10ebcd75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6809","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6812","nonce":3360352974}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6817","nonce":3360352974}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6828","nonce":3360352974}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6823","nonce":3360352974}]},"public_addr":"172.21.15.16:6812/3360352974","cluster_addr":"172.21.15.16:6817/3360352974","heartbeat_back_addr":"172.21.15.16:6828/3360352974","heartbeat_front_addr":"172.21.15.16:6823/3360352974","state":["exists","up"]},{"osd":4,"uuid":"88260730-1482-4fbd-a40f-cb1124e9df4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6801","nonce":1856844856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6803","nonce":1856844856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6807","nonce":1856844856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6805","nonce":1856844856}]},"public_addr":"172.21.15.89:6801/1856844856","cluster_addr":"172.21.15.89:6803/1856844856","heartbeat_back_addr":"172.21.15.89:6807/1856844856","heartbeat_front_addr":"172.21.15.89:6805/1856844856","state":["exists","up"]},{"osd":5,"uuid":"4cb2c00c-fae0-4dd8-9d27-ec4f0093ccf7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6814","nonce":50395527}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6817","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6820","nonce":50395527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6831","nonce":50395527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6823","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6826","nonce":50395527}]},"public_addr":"172.21.15.89:6814/50395527","cluster_addr":"172.21.15.89:6820/50395527","heartbeat_back_addr":"172.21.15.89:6831/50395527","heartbeat_front_addr":"172.21.15.89:6826/50395527","state":["exists","up"]},{"osd":6,"uuid":"81ce0c19-2554-4cae-acbe-66a05e9e9b9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6809","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6812","nonce":3470471752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6815","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6818","nonce":3470471752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6827","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6829","nonce":3470471752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6821","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6824","nonce":3470471752}]},"public_addr":"172.21.15.89:6812/3470471752","cluster_addr":"172.21.15.89:6818/3470471752","heartbeat_back_addr":"172.21.15.89:6829/3470471752","heartbeat_front_addr":"172.21.15.89:6824/3470471752","state":["exists","up"]},{"osd":7,"uuid":"85e4313b-d37d-442e-8042-153bf20a847d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6811","nonce":1796778630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6813","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6816","nonce":1796778630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6825","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6828","nonce":1796778630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6819","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6822","nonce":1796778630}]},"public_addr":"172.21.15.89:6811/1796778630","cluster_addr":"172.21.15.89:6816/1796778630","heartbeat_back_addr":"172.21.15.89:6828/1796778630","heartbeat_front_addr":"172.21.15.89:6822/1796778630","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.439088+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.529970+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.680512+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:21.044161+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.582111+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.536558+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.409522+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.590229+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:6835/567808781":"2024-04-01T15:13:50.823751+0000","172.21.15.16:6833/567808781":"2024-04-01T15:13:50.823751+0000","172.21.15.16:6835/490717480":"2024-04-01T15:13:14.080258+0000","172.21.15.16:6833/490717480":"2024-04-01T15:13:14.080258+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-31T15:13:52.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:52.070+0000 7f92a9e00640 1 -- 172.21.15.16:0/3597887451 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f9280068380 msgr2=0x7f928006a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:13:52.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:52.070+0000 7f92a9e00640 1 --2- 172.21.15.16:0/3597887451 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f9280068380 0x7f928006a840 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f92940ad000 tx=0x7f92940a6000 comp rx=0 tx=0).stop 2024-03-31T15:13:52.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:52.070+0000 7f92a9e00640 1 -- 172.21.15.16:0/3597887451 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f92a40cef40 msgr2=0x7f92a415b8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:13:52.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:52.070+0000 7f92a9e00640 1 --2- 172.21.15.16:0/3597887451 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f92a40cef40 0x7f92a415b8b0 secure :-1 s=READY pgs=1569 cs=0 l=1 rev1=1 crypto rx=0x7f928c00c7f0 tx=0x7f928c00ccb0 comp rx=0 tx=0).stop 2024-03-31T15:13:52.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:52.074+0000 7f92a9e00640 1 -- 172.21.15.16:0/3597887451 shutdown_connections 2024-03-31T15:13:52.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:52.074+0000 7f92a9e00640 1 --2- 172.21.15.16:0/3597887451 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f9280068380 0x7f928006a840 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:52.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:52.074+0000 7f92a9e00640 1 --2- 172.21.15.16:0/3597887451 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f92a415b4d0 0x7f92a4158da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:52.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:52.074+0000 7f92a9e00640 1 --2- 172.21.15.16:0/3597887451 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f92a40cef40 0x7f92a415b8b0 unknown :-1 s=CLOSED pgs=1569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:52.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:52.074+0000 7f92a9e00640 1 --2- 172.21.15.16:0/3597887451 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f92a40b61e0 0x7f92a414c9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:52.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:52.074+0000 7f92a9e00640 1 -- 172.21.15.16:0/3597887451 >> 172.21.15.16:0/3597887451 conn(0x7f92a40d1060 msgr2=0x7f92a4157df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:13:52.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:52.074+0000 7f92a9e00640 1 -- 172.21.15.16:0/3597887451 shutdown_connections 2024-03-31T15:13:52.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:52.074+0000 7f92a9e00640 1 -- 172.21.15.16:0/3597887451 wait complete. 2024-03-31T15:13:52.085 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-31T15:03:22.550787+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 14, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-31T15:12:14.671225+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '95', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.5, 'score_stable': 2.5, 'optimal_score': 1, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 15, 'pool_name': 'cephfs_data', 'create_time': '2024-03-31T15:12:15.242196+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '96', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-31T15:13:52.085 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-31T15:13:52.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:52.234+0000 7fbf00800640 1 Processor -- start 2024-03-31T15:13:52.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:52.234+0000 7fbf00800640 1 -- start start 2024-03-31T15:13:52.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:52.238+0000 7fbf00800640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fbef81411b0 0x7fbef806f920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:52.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:52.238+0000 7fbf00800640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fbef8141590 0x7fbef806fe60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:52.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:52.238+0000 7fbf00800640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbef8144980 con 0x7fbef8143c30 2024-03-31T15:13:52.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:52.238+0000 7fbf00800640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7fbef815f910 con 0x7fbef8141590 2024-03-31T15:13:52.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:52.238+0000 7fbf00800640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fbef8162910 con 0x7fbef81411b0 2024-03-31T15:13:52.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:52.238+0000 7fbefd400640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fbef8141590 0x7fbef806fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:52.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:52.238+0000 7fbefe800640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fbef8143c30 0x7fbef81440a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:44758/0 (socket says 172.21.15.16:44758) 2024-03-31T15:13:52.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:52.238+0000 7fbefe800640 1 -- 172.21.15.16:0/3394201464 learned_addr learned my addr 172.21.15.16:0/3394201464 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:13:52.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:52.238+0000 7fbefde00640 1 --2- 172.21.15.16:0/3394201464 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fbef81411b0 0x7fbef806f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:52.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:52.238+0000 7fbefd400640 1 -- 172.21.15.16:0/3394201464 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fbef81411b0 msgr2=0x7fbef806f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:13:52.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:52.238+0000 7fbefd400640 1 --2- 172.21.15.16:0/3394201464 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fbef81411b0 0x7fbef806f920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:52.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:52.238+0000 7fbefd400640 1 -- 172.21.15.16:0/3394201464 >> v1:172.21.15.16:6789/0 conn(0x7fbef8143c30 legacy=0x7fbef81440a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:13:52.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:52.238+0000 7fbefd400640 1 -- 172.21.15.16:0/3394201464 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbef8161fe0 con 0x7fbef8141590 2024-03-31T15:13:52.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:52.238+0000 7fbefde00640 1 --2- 172.21.15.16:0/3394201464 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fbef81411b0 0x7fbef806f920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:13:52.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:52.238+0000 7fbefca00640 1 -- 172.21.15.16:0/3394201464 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2716019396 0 0) 0x7fbef8144980 con 0x7fbef8143c30 2024-03-31T15:13:52.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:52.242+0000 7fbefd400640 1 --2- 172.21.15.16:0/3394201464 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbef8141590 0x7fbef806fe60 secure :-1 s=READY pgs=1571 cs=0 l=1 rev1=1 crypto rx=0x7fbee80097c0 tx=0x7fbee801c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8ccf2cfab8118d1 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:13:52.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:52.242+0000 7fbefca00640 1 -- 172.21.15.16:0/3394201464 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbee80b2070 con 0x7fbef8141590 2024-03-31T15:13:52.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:52.242+0000 7fbefca00640 1 -- 172.21.15.16:0/3394201464 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbee80ad640 con 0x7fbef8141590 2024-03-31T15:13:52.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:52.242+0000 7fbefca00640 1 -- 172.21.15.16:0/3394201464 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbee80ada10 con 0x7fbef8141590 2024-03-31T15:13:52.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:52.242+0000 7fbf00800640 1 -- 172.21.15.16:0/3394201464 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbef8141590 msgr2=0x7fbef806fe60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:13:52.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:52.242+0000 7fbf00800640 1 --2- 172.21.15.16:0/3394201464 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbef8141590 0x7fbef806fe60 secure :-1 s=READY pgs=1571 cs=0 l=1 rev1=1 crypto rx=0x7fbee80097c0 tx=0x7fbee801c4a0 comp rx=0 tx=0).stop 2024-03-31T15:13:52.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:52.242+0000 7fbf00800640 1 -- 172.21.15.16:0/3394201464 shutdown_connections 2024-03-31T15:13:52.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:52.242+0000 7fbf00800640 1 --2- 172.21.15.16:0/3394201464 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbef8141590 0x7fbef806fe60 unknown :-1 s=CLOSED pgs=1571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:52.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:52.242+0000 7fbf00800640 1 --2- 172.21.15.16:0/3394201464 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fbef81411b0 0x7fbef806f920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:52.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:52.242+0000 7fbf00800640 1 -- 172.21.15.16:0/3394201464 >> 172.21.15.16:0/3394201464 conn(0x7fbef8067930 msgr2=0x7fbef80b3ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:13:52.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:52.242+0000 7fbf00800640 1 -- 172.21.15.16:0/3394201464 shutdown_connections 2024-03-31T15:13:52.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:52.242+0000 7fbf00800640 1 -- 172.21.15.16:0/3394201464 wait complete. 2024-03-31T15:13:52.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:52.242+0000 7fbf00800640 1 Processor -- start 2024-03-31T15:13:52.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:52.242+0000 7fbf00800640 1 -- start start 2024-03-31T15:13:52.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:52.242+0000 7fbf00800640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fbef81411b0 0x7fbef813afc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:52.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:52.242+0000 7fbf00800640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbef8141590 0x7fbef8151e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:52.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:52.242+0000 7fbf00800640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbef8143c30 0x7fbef81527c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:52.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:52.246+0000 7fbf00800640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fbef815f290 con 0x7fbef8141590 2024-03-31T15:13:52.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:52.246+0000 7fbf00800640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fbef815f110 con 0x7fbef8143c30 2024-03-31T15:13:52.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:52.246+0000 7fbf00800640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fbef815f410 con 0x7fbef81411b0 2024-03-31T15:13:52.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:52.246+0000 7fbefd400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbef8141590 0x7fbef8151e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:52.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:52.246+0000 7fbefd400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbef8141590 0x7fbef8151e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50302/0 (socket says 172.21.15.16:50302) 2024-03-31T15:13:52.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:52.246+0000 7fbefd400640 1 -- 172.21.15.16:0/476953891 learned_addr learned my addr 172.21.15.16:0/476953891 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:13:52.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:52.246+0000 7fbefde00640 1 --2- 172.21.15.16:0/476953891 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fbef81411b0 0x7fbef813afc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:52.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:52.246+0000 7fbefe800640 1 --2- 172.21.15.16:0/476953891 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbef8143c30 0x7fbef81527c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:52.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:52.246+0000 7fbefd400640 1 -- 172.21.15.16:0/476953891 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fbef81411b0 msgr2=0x7fbef813afc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:13:52.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:52.246+0000 7fbefd400640 1 --2- 172.21.15.16:0/476953891 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fbef81411b0 0x7fbef813afc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:52.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:52.246+0000 7fbefd400640 1 -- 172.21.15.16:0/476953891 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbef8143c30 msgr2=0x7fbef81527c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:13:52.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:52.246+0000 7fbefd400640 1 --2- 172.21.15.16:0/476953891 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbef8143c30 0x7fbef81527c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:52.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:52.246+0000 7fbefd400640 1 -- 172.21.15.16:0/476953891 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbef8153390 con 0x7fbef8141590 2024-03-31T15:13:52.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:52.246+0000 7fbefd400640 1 --2- 172.21.15.16:0/476953891 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbef8141590 0x7fbef8151e30 secure :-1 s=READY pgs=1572 cs=0 l=1 rev1=1 crypto rx=0x7fbee801a0d0 tx=0x7fbee8019240 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:13:52.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:52.246+0000 7fbef2000640 1 -- 172.21.15.16:0/476953891 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbee80ba020 con 0x7fbef8141590 2024-03-31T15:13:52.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:52.246+0000 7fbef2000640 1 -- 172.21.15.16:0/476953891 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbee80b6630 con 0x7fbef8141590 2024-03-31T15:13:52.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:52.246+0000 7fbf00800640 1 -- 172.21.15.16:0/476953891 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbef8153630 con 0x7fbef8141590 2024-03-31T15:13:52.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:52.246+0000 7fbef2000640 1 -- 172.21.15.16:0/476953891 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbee80b2040 con 0x7fbef8141590 2024-03-31T15:13:52.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:52.246+0000 7fbf00800640 1 -- 172.21.15.16:0/476953891 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbef813ecf0 con 0x7fbef8141590 2024-03-31T15:13:52.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:52.246+0000 7fbf00800640 1 -- 172.21.15.16:0/476953891 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbef8069950 con 0x7fbef8141590 2024-03-31T15:13:52.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:52.246+0000 7fbef2000640 1 -- 172.21.15.16:0/476953891 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fbee80bd020 con 0x7fbef8141590 2024-03-31T15:13:52.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:52.250+0000 7fbef2000640 1 --2- 172.21.15.16:0/476953891 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fbedc068380 0x7fbedc06a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:52.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:52.250+0000 7fbef2000640 1 -- 172.21.15.16:0/476953891 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 5881+0+0 (secure 0 0 0) 0x7fbee8120900 con 0x7fbef8141590 2024-03-31T15:13:52.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:52.250+0000 7fbefde00640 1 --2- 172.21.15.16:0/476953891 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fbedc068380 0x7fbedc06a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:52.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:52.250+0000 7fbefde00640 1 --2- 172.21.15.16:0/476953891 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fbedc068380 0x7fbedc06a840 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7fbeec00ab60 tx=0x7fbeec008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:13:52.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:52.254+0000 7fbef2000640 1 -- 172.21.15.16:0/476953891 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fbee80c4030 con 0x7fbef8141590 2024-03-31T15:13:52.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:52.418+0000 7fbf00800640 1 -- 172.21.15.16:0/476953891 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fbef8161810 con 0x7fbef8141590 2024-03-31T15:13:52.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:52.418+0000 7fbef2000640 1 -- 172.21.15.16:0/476953891 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v98) ==== 93+0+10 (secure 0 0 0) 0x7fbee8016090 con 0x7fbef8141590 2024-03-31T15:13:52.410 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2024-03-31T15:13:52.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:52.418+0000 7fbf00800640 1 -- 172.21.15.16:0/476953891 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fbedc068380 msgr2=0x7fbedc06a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:13:52.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:52.418+0000 7fbf00800640 1 --2- 172.21.15.16:0/476953891 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fbedc068380 0x7fbedc06a840 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7fbeec00ab60 tx=0x7fbeec008040 comp rx=0 tx=0).stop 2024-03-31T15:13:52.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:52.418+0000 7fbf00800640 1 -- 172.21.15.16:0/476953891 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbef8141590 msgr2=0x7fbef8151e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:13:52.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:52.418+0000 7fbf00800640 1 --2- 172.21.15.16:0/476953891 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbef8141590 0x7fbef8151e30 secure :-1 s=READY pgs=1572 cs=0 l=1 rev1=1 crypto rx=0x7fbee801a0d0 tx=0x7fbee8019240 comp rx=0 tx=0).stop 2024-03-31T15:13:52.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:52.418+0000 7fbf00800640 1 -- 172.21.15.16:0/476953891 shutdown_connections 2024-03-31T15:13:52.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:52.418+0000 7fbf00800640 1 --2- 172.21.15.16:0/476953891 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fbedc068380 0x7fbedc06a840 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:52.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:52.418+0000 7fbf00800640 1 --2- 172.21.15.16:0/476953891 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbef8143c30 0x7fbef81527c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:52.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:52.418+0000 7fbf00800640 1 --2- 172.21.15.16:0/476953891 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbef8141590 0x7fbef8151e30 unknown :-1 s=CLOSED pgs=1572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:52.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:52.418+0000 7fbf00800640 1 --2- 172.21.15.16:0/476953891 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fbef81411b0 0x7fbef813afc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:52.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:52.418+0000 7fbf00800640 1 -- 172.21.15.16:0/476953891 >> 172.21.15.16:0/476953891 conn(0x7fbef8067930 msgr2=0x7fbef815faa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:13:52.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:52.418+0000 7fbf00800640 1 -- 172.21.15.16:0/476953891 shutdown_connections 2024-03-31T15:13:52.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:52.418+0000 7fbf00800640 1 -- 172.21.15.16:0/476953891 wait complete. 2024-03-31T15:13:52.433 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-31T15:13:52.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:52.586+0000 7f080d600640 1 Processor -- start 2024-03-31T15:13:52.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:52.586+0000 7f080d600640 1 -- start start 2024-03-31T15:13:52.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:52.586+0000 7f080d600640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f080815e4f0 0x7f08080d4d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:52.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:52.586+0000 7f080d600640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f08080d16e0 0x7f08080d2ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:52.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:52.586+0000 7f080d600640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f080815e8d0 con 0x7f08080d1270 2024-03-31T15:13:52.580 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:52.586+0000 7f080d600640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f08080d3410 con 0x7f080815e4f0 2024-03-31T15:13:52.580 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:52.586+0000 7f080d600640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f08080d4870 con 0x7f08080d16e0 2024-03-31T15:13:52.580 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:52.586+0000 7f0806a00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f080815e4f0 0x7f08080d4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:52.580 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:52.586+0000 7f0806a00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f080815e4f0 0x7f08080d4d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50314/0 (socket says 172.21.15.16:50314) 2024-03-31T15:13:52.580 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:52.586+0000 7f0806a00640 1 -- 172.21.15.16:0/2304391285 learned_addr learned my addr 172.21.15.16:0/2304391285 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:13:52.580 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:52.586+0000 7f0807400640 1 --2- 172.21.15.16:0/2304391285 >> v2:172.21.15.89:3300/0 conn(0x7f08080d16e0 0x7f08080d2ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:52.580 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:52.586+0000 7f0806a00640 1 -- 172.21.15.16:0/2304391285 >> v2:172.21.15.89:3300/0 conn(0x7f08080d16e0 msgr2=0x7f08080d2ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:13:52.580 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:52.586+0000 7f0806a00640 1 --2- 172.21.15.16:0/2304391285 >> v2:172.21.15.89:3300/0 conn(0x7f08080d16e0 0x7f08080d2ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:52.580 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:52.586+0000 7f0806a00640 1 -- 172.21.15.16:0/2304391285 >> v1:172.21.15.89:6789/0 conn(0x7f08080d1270 legacy=0x7f08080b7380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:13:52.580 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:52.586+0000 7f0806a00640 1 -- 172.21.15.16:0/2304391285 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08080d3c60 con 0x7f080815e4f0 2024-03-31T15:13:52.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:52.590+0000 7f0806a00640 1 --2- 172.21.15.16:0/2304391285 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f080815e4f0 0x7f08080d4d00 secure :-1 s=READY pgs=1573 cs=0 l=1 rev1=1 crypto rx=0x7f08080d0af0 tx=0x7f07f801c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=41eb3cedee2e6f5d server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:13:52.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:52.590+0000 7f0806000640 1 -- 172.21.15.16:0/2304391285 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07f80b2070 con 0x7f080815e4f0 2024-03-31T15:13:52.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:52.590+0000 7f0806000640 1 -- 172.21.15.16:0/2304391285 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f07f80ad680 con 0x7f080815e4f0 2024-03-31T15:13:52.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:52.590+0000 7f0806000640 1 -- 172.21.15.16:0/2304391285 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07f80ada50 con 0x7f080815e4f0 2024-03-31T15:13:52.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:52.590+0000 7f080d600640 1 -- 172.21.15.16:0/2304391285 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f080815e4f0 msgr2=0x7f08080d4d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:13:52.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:52.590+0000 7f080d600640 1 --2- 172.21.15.16:0/2304391285 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f080815e4f0 0x7f08080d4d00 secure :-1 s=READY pgs=1573 cs=0 l=1 rev1=1 crypto rx=0x7f08080d0af0 tx=0x7f07f801c490 comp rx=0 tx=0).stop 2024-03-31T15:13:52.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:52.590+0000 7f080d600640 1 -- 172.21.15.16:0/2304391285 shutdown_connections 2024-03-31T15:13:52.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:52.590+0000 7f080d600640 1 --2- 172.21.15.16:0/2304391285 >> v2:172.21.15.89:3300/0 conn(0x7f08080d16e0 0x7f08080d2ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:52.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:52.590+0000 7f080d600640 1 --2- 172.21.15.16:0/2304391285 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f080815e4f0 0x7f08080d4d00 unknown :-1 s=CLOSED pgs=1573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:52.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:52.590+0000 7f080d600640 1 -- 172.21.15.16:0/2304391285 >> 172.21.15.16:0/2304391285 conn(0x7f0808067730 msgr2=0x7f0808067b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:13:52.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:52.590+0000 7f080d600640 1 -- 172.21.15.16:0/2304391285 shutdown_connections 2024-03-31T15:13:52.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:52.590+0000 7f080d600640 1 -- 172.21.15.16:0/2304391285 wait complete. 2024-03-31T15:13:52.585 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:52.590+0000 7f080d600640 1 Processor -- start 2024-03-31T15:13:52.585 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:52.590+0000 7f080d600640 1 -- start start 2024-03-31T15:13:52.587 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:52.594+0000 7f080d600640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f08080d1270 0x7f08081531c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:52.587 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:52.594+0000 7f080d600640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f08080d16e0 0x7f08081505d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:52.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:52.594+0000 7f080d600640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f080815e4f0 0x7f08080ca2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:52.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:52.594+0000 7f080d600640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f08080d1af0 con 0x7f08080d1270 2024-03-31T15:13:52.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:52.594+0000 7f080d600640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f08080b3770 con 0x7f080815e4f0 2024-03-31T15:13:52.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:52.594+0000 7f080d600640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f08080d3810 con 0x7f08080d16e0 2024-03-31T15:13:52.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:52.594+0000 7f0806a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f08080d1270 0x7f08081531c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:52.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:52.594+0000 7f0806a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f08080d1270 0x7f08081531c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50320/0 (socket says 172.21.15.16:50320) 2024-03-31T15:13:52.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:52.594+0000 7f0806a00640 1 -- 172.21.15.16:0/2215713095 learned_addr learned my addr 172.21.15.16:0/2215713095 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:13:52.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:52.594+0000 7f0807400640 1 --2- 172.21.15.16:0/2215713095 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f080815e4f0 0x7f08080ca2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:52.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:52.594+0000 7f07ffe00640 1 --2- 172.21.15.16:0/2215713095 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f08080d16e0 0x7f08081505d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:52.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:52.594+0000 7f0806a00640 1 -- 172.21.15.16:0/2215713095 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f08080d16e0 msgr2=0x7f08081505d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:13:52.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:52.594+0000 7f0806a00640 1 --2- 172.21.15.16:0/2215713095 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f08080d16e0 0x7f08081505d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:52.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:52.594+0000 7f0806a00640 1 -- 172.21.15.16:0/2215713095 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f080815e4f0 msgr2=0x7f08080ca2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:13:52.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:52.594+0000 7f0806a00640 1 --2- 172.21.15.16:0/2215713095 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f080815e4f0 0x7f08080ca2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:52.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:52.594+0000 7f0806a00640 1 -- 172.21.15.16:0/2215713095 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0808153700 con 0x7f08080d1270 2024-03-31T15:13:52.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:52.594+0000 7f0806a00640 1 --2- 172.21.15.16:0/2215713095 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f08080d1270 0x7f08081531c0 secure :-1 s=READY pgs=1574 cs=0 l=1 rev1=1 crypto rx=0x7f07f801a0c0 tx=0x7f07f801acd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:13:52.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:52.594+0000 7f07fea00640 1 -- 172.21.15.16:0/2215713095 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07f80b2040 con 0x7f08080d1270 2024-03-31T15:13:52.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:52.594+0000 7f07fea00640 1 -- 172.21.15.16:0/2215713095 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f07f80bd070 con 0x7f08080d1270 2024-03-31T15:13:52.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:52.594+0000 7f07fea00640 1 -- 172.21.15.16:0/2215713095 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07f80ade30 con 0x7f08080d1270 2024-03-31T15:13:52.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:52.594+0000 7f080d600640 1 -- 172.21.15.16:0/2215713095 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f08081539c0 con 0x7f08080d1270 2024-03-31T15:13:52.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:52.594+0000 7f080d600640 1 -- 172.21.15.16:0/2215713095 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0808153f80 con 0x7f08080d1270 2024-03-31T15:13:52.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:52.594+0000 7f080d600640 1 -- 172.21.15.16:0/2215713095 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0808069950 con 0x7f08080d1270 2024-03-31T15:13:52.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:52.598+0000 7f07fea00640 1 -- 172.21.15.16:0/2215713095 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f07f80c2460 con 0x7f08080d1270 2024-03-31T15:13:52.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:52.598+0000 7f07fea00640 1 --2- 172.21.15.16:0/2215713095 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f07e4068380 0x7f07e406a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:52.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:52.598+0000 7f07fea00640 1 -- 172.21.15.16:0/2215713095 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 5881+0+0 (secure 0 0 0) 0x7f07f8120c10 con 0x7f08080d1270 2024-03-31T15:13:52.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:52.598+0000 7f07ffe00640 1 --2- 172.21.15.16:0/2215713095 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f07e4068380 0x7f07e406a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:52.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:52.598+0000 7f07ffe00640 1 --2- 172.21.15.16:0/2215713095 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f07e4068380 0x7f07e406a840 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f08081fa570 tx=0x7f07f0009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:13:52.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:52.602+0000 7f07fea00640 1 -- 172.21.15.16:0/2215713095 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f07f8016110 con 0x7f08080d1270 2024-03-31T15:13:52.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:52.774+0000 7f080d600640 1 -- 172.21.15.16:0/2215713095 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f0808150b10 con 0x7f08080d1270 2024-03-31T15:13:52.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:52.774+0000 7f07fea00640 1 -- 172.21.15.16:0/2215713095 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v98) ==== 104+0+11 (secure 0 0 0) 0x7f07f80f0a30 con 0x7f08080d1270 2024-03-31T15:13:52.766 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2024-03-31T15:13:52.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:52.774+0000 7f080d600640 1 -- 172.21.15.16:0/2215713095 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f07e4068380 msgr2=0x7f07e406a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:13:52.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:52.774+0000 7f080d600640 1 --2- 172.21.15.16:0/2215713095 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f07e4068380 0x7f07e406a840 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f08081fa570 tx=0x7f07f0009040 comp rx=0 tx=0).stop 2024-03-31T15:13:52.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:52.774+0000 7f080d600640 1 -- 172.21.15.16:0/2215713095 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f08080d1270 msgr2=0x7f08081531c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:13:52.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:52.774+0000 7f080d600640 1 --2- 172.21.15.16:0/2215713095 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f08080d1270 0x7f08081531c0 secure :-1 s=READY pgs=1574 cs=0 l=1 rev1=1 crypto rx=0x7f07f801a0c0 tx=0x7f07f801acd0 comp rx=0 tx=0).stop 2024-03-31T15:13:52.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:52.774+0000 7f080d600640 1 -- 172.21.15.16:0/2215713095 shutdown_connections 2024-03-31T15:13:52.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:52.774+0000 7f080d600640 1 --2- 172.21.15.16:0/2215713095 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f07e4068380 0x7f07e406a840 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:52.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:52.774+0000 7f080d600640 1 --2- 172.21.15.16:0/2215713095 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f080815e4f0 0x7f08080ca2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:52.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:52.774+0000 7f080d600640 1 --2- 172.21.15.16:0/2215713095 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f08080d16e0 0x7f08081505d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:52.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:52.774+0000 7f080d600640 1 --2- 172.21.15.16:0/2215713095 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f08080d1270 0x7f08081531c0 unknown :-1 s=CLOSED pgs=1574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:52.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:52.774+0000 7f080d600640 1 -- 172.21.15.16:0/2215713095 >> 172.21.15.16:0/2215713095 conn(0x7f0808067730 msgr2=0x7f080815eac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:13:52.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:52.774+0000 7f080d600640 1 -- 172.21.15.16:0/2215713095 shutdown_connections 2024-03-31T15:13:52.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:52.774+0000 7f080d600640 1 -- 172.21.15.16:0/2215713095 wait complete. 2024-03-31T15:13:52.787 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-31T15:13:52.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:52.938+0000 7f321ce00640 1 Processor -- start 2024-03-31T15:13:52.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:52.942+0000 7f321ce00640 1 -- start start 2024-03-31T15:13:52.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:52.942+0000 7f321ce00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f32181567e0 0x7f3218072c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:52.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:52.942+0000 7f321ce00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f321806dbe0 0x7f3218071550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:52.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:52.942+0000 7f321ce00640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f321815c7e0 con 0x7f32180731c0 2024-03-31T15:13:52.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:52.942+0000 7f321ce00640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f32180736b0 con 0x7f32181567e0 2024-03-31T15:13:52.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:52.942+0000 7f321ce00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f321815caa0 con 0x7f321806dbe0 2024-03-31T15:13:52.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:52.942+0000 7f320d400640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f32180731c0 0x7f321806d4d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:44768/0 (socket says 172.21.15.16:44768) 2024-03-31T15:13:52.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:52.942+0000 7f320d400640 1 -- 172.21.15.16:0/3935220959 learned_addr learned my addr 172.21.15.16:0/3935220959 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:13:52.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:52.942+0000 7f3215e00640 1 --2- 172.21.15.16:0/3935220959 >> v2:172.21.15.89:3300/0 conn(0x7f32181567e0 0x7f3218072c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:52.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:52.942+0000 7f3216800640 1 --2- 172.21.15.16:0/3935220959 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f321806dbe0 0x7f3218071550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:52.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:52.942+0000 7f3215400640 1 -- 172.21.15.16:0/3935220959 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4188447485 0 0) 0x7f321815c7e0 con 0x7f32180731c0 2024-03-31T15:13:52.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:52.942+0000 7f3215400640 1 -- 172.21.15.16:0/3935220959 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f31fc003530 con 0x7f32180731c0 2024-03-31T15:13:52.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:52.942+0000 7f3215e00640 1 -- 172.21.15.16:0/3935220959 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f321806dbe0 msgr2=0x7f3218071550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:13:52.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:52.942+0000 7f3215e00640 1 --2- 172.21.15.16:0/3935220959 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f321806dbe0 0x7f3218071550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:52.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:52.942+0000 7f3215e00640 1 -- 172.21.15.16:0/3935220959 >> v1:172.21.15.16:6789/0 conn(0x7f32180731c0 legacy=0x7f321806d4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:13:52.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:52.942+0000 7f3215e00640 1 -- 172.21.15.16:0/3935220959 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f321815cd80 con 0x7f32181567e0 2024-03-31T15:13:52.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:52.946+0000 7f3215e00640 1 --2- 172.21.15.16:0/3935220959 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f32181567e0 0x7f3218072c80 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f32080097c0 tx=0x7f320801c4a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cf0739050a476f4b server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:13:52.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:52.946+0000 7f3215400640 1 -- 172.21.15.16:0/3935220959 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32080b1070 con 0x7f32181567e0 2024-03-31T15:13:52.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:52.946+0000 7f3215400640 1 -- 172.21.15.16:0/3935220959 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f32080ac440 con 0x7f32181567e0 2024-03-31T15:13:52.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:52.946+0000 7f3215400640 1 -- 172.21.15.16:0/3935220959 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32080ac810 con 0x7f32181567e0 2024-03-31T15:13:52.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:52.946+0000 7f321ce00640 1 -- 172.21.15.16:0/3935220959 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f32181567e0 msgr2=0x7f3218072c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:13:52.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:52.946+0000 7f321ce00640 1 --2- 172.21.15.16:0/3935220959 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f32181567e0 0x7f3218072c80 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f32080097c0 tx=0x7f320801c4a0 comp rx=0 tx=0).stop 2024-03-31T15:13:52.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:52.946+0000 7f321ce00640 1 -- 172.21.15.16:0/3935220959 shutdown_connections 2024-03-31T15:13:52.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:52.946+0000 7f321ce00640 1 --2- 172.21.15.16:0/3935220959 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f321806dbe0 0x7f3218071550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:52.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:52.946+0000 7f321ce00640 1 --2- 172.21.15.16:0/3935220959 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f32181567e0 0x7f3218072c80 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:52.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:52.946+0000 7f321ce00640 1 -- 172.21.15.16:0/3935220959 >> 172.21.15.16:0/3935220959 conn(0x7f321806ee40 msgr2=0x7f321806b620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:13:52.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:52.946+0000 7f321ce00640 1 -- 172.21.15.16:0/3935220959 shutdown_connections 2024-03-31T15:13:52.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:52.946+0000 7f321ce00640 1 -- 172.21.15.16:0/3935220959 wait complete. 2024-03-31T15:13:52.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:52.950+0000 7f321ce00640 1 Processor -- start 2024-03-31T15:13:52.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:52.950+0000 7f321ce00640 1 -- start start 2024-03-31T15:13:52.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:52.950+0000 7f321ce00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f321806dbe0 0x7f321814ec90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:52.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:52.950+0000 7f321ce00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f32180731c0 0x7f32181fab10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:52.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:52.950+0000 7f321ce00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f32181567e0 0x7f32181457b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:52.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:52.950+0000 7f321ce00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f32181692f0 con 0x7f32180731c0 2024-03-31T15:13:52.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:52.950+0000 7f321ce00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f3218169170 con 0x7f32181567e0 2024-03-31T15:13:52.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:52.950+0000 7f321ce00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f3218169470 con 0x7f321806dbe0 2024-03-31T15:13:52.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:52.950+0000 7f320d400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f32180731c0 0x7f32181fab10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:52.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:52.950+0000 7f320d400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f32180731c0 0x7f32181fab10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50328/0 (socket says 172.21.15.16:50328) 2024-03-31T15:13:52.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:52.950+0000 7f320d400640 1 -- 172.21.15.16:0/2584553656 learned_addr learned my addr 172.21.15.16:0/2584553656 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:13:52.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:52.950+0000 7f3215e00640 1 --2- 172.21.15.16:0/2584553656 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f321806dbe0 0x7f321814ec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:52.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:52.950+0000 7f3216800640 1 --2- 172.21.15.16:0/2584553656 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f32181567e0 0x7f32181457b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:52.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:52.950+0000 7f320d400640 1 -- 172.21.15.16:0/2584553656 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f321806dbe0 msgr2=0x7f321814ec90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:13:52.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:52.950+0000 7f320d400640 1 --2- 172.21.15.16:0/2584553656 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f321806dbe0 0x7f321814ec90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:52.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:52.950+0000 7f320d400640 1 -- 172.21.15.16:0/2584553656 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f32181567e0 msgr2=0x7f32181457b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:13:52.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:52.950+0000 7f320d400640 1 --2- 172.21.15.16:0/2584553656 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f32181567e0 0x7f32181457b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:52.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:52.950+0000 7f320d400640 1 -- 172.21.15.16:0/2584553656 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f321814f270 con 0x7f32180731c0 2024-03-31T15:13:52.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:52.950+0000 7f320d400640 1 --2- 172.21.15.16:0/2584553656 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f32180731c0 0x7f32181fab10 secure :-1 s=READY pgs=1576 cs=0 l=1 rev1=1 crypto rx=0x7f320000c450 tx=0x7f320000c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:13:52.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:52.950+0000 7f320ea00640 1 -- 172.21.15.16:0/2584553656 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3200017070 con 0x7f32180731c0 2024-03-31T15:13:52.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:52.950+0000 7f320ea00640 1 -- 172.21.15.16:0/2584553656 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3200005d10 con 0x7f32180731c0 2024-03-31T15:13:52.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:52.950+0000 7f320ea00640 1 -- 172.21.15.16:0/2584553656 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32000072b0 con 0x7f32180731c0 2024-03-31T15:13:52.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:52.950+0000 7f321ce00640 1 -- 172.21.15.16:0/2584553656 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f321814f550 con 0x7f32180731c0 2024-03-31T15:13:52.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:52.950+0000 7f321ce00640 1 -- 172.21.15.16:0/2584553656 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f321814fb10 con 0x7f32180731c0 2024-03-31T15:13:52.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:52.954+0000 7f321ce00640 1 -- 172.21.15.16:0/2584553656 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3218069950 con 0x7f32180731c0 2024-03-31T15:13:52.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:52.954+0000 7f320ea00640 1 -- 172.21.15.16:0/2584553656 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f320001a070 con 0x7f32180731c0 2024-03-31T15:13:52.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:52.954+0000 7f320ea00640 1 --2- 172.21.15.16:0/2584553656 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f31f4068380 0x7f31f406a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:52.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:52.954+0000 7f320ea00640 1 -- 172.21.15.16:0/2584553656 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 5881+0+0 (secure 0 0 0) 0x7f3200085d00 con 0x7f32180731c0 2024-03-31T15:13:52.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:52.954+0000 7f3215e00640 1 --2- 172.21.15.16:0/2584553656 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f31f4068380 0x7f31f406a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:52.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:52.954+0000 7f3215e00640 1 --2- 172.21.15.16:0/2584553656 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f31f4068380 0x7f31f406a840 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f32080ae000 tx=0x7f32080a6070 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:13:52.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:52.958+0000 7f320ea00640 1 -- 172.21.15.16:0/2584553656 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f3200055c70 con 0x7f32180731c0 2024-03-31T15:13:53.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:53.122+0000 7f321ce00640 1 -- 172.21.15.16:0/2584553656 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f32180af520 con 0x7f32180731c0 2024-03-31T15:13:53.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:53.122+0000 7f320ea00640 1 -- 172.21.15.16:0/2584553656 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v98) ==== 100+0+11 (secure 0 0 0) 0x7f3200059f70 con 0x7f32180731c0 2024-03-31T15:13:53.116 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 64 2024-03-31T15:13:53.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:53.126+0000 7f321ce00640 1 -- 172.21.15.16:0/2584553656 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f31f4068380 msgr2=0x7f31f406a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:13:53.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:53.126+0000 7f321ce00640 1 --2- 172.21.15.16:0/2584553656 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f31f4068380 0x7f31f406a840 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f32080ae000 tx=0x7f32080a6070 comp rx=0 tx=0).stop 2024-03-31T15:13:53.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:53.126+0000 7f321ce00640 1 -- 172.21.15.16:0/2584553656 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f32180731c0 msgr2=0x7f32181fab10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:13:53.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:53.126+0000 7f321ce00640 1 --2- 172.21.15.16:0/2584553656 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f32180731c0 0x7f32181fab10 secure :-1 s=READY pgs=1576 cs=0 l=1 rev1=1 crypto rx=0x7f320000c450 tx=0x7f320000c910 comp rx=0 tx=0).stop 2024-03-31T15:13:53.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:53.126+0000 7f321ce00640 1 -- 172.21.15.16:0/2584553656 shutdown_connections 2024-03-31T15:13:53.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:53.126+0000 7f321ce00640 1 --2- 172.21.15.16:0/2584553656 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f31f4068380 0x7f31f406a840 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:53.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:53.126+0000 7f321ce00640 1 --2- 172.21.15.16:0/2584553656 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f32181567e0 0x7f32181457b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:53.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:53.126+0000 7f321ce00640 1 --2- 172.21.15.16:0/2584553656 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f32180731c0 0x7f32181fab10 unknown :-1 s=CLOSED pgs=1576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:53.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:53.126+0000 7f321ce00640 1 --2- 172.21.15.16:0/2584553656 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f321806dbe0 0x7f321814ec90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:53.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:53.126+0000 7f321ce00640 1 -- 172.21.15.16:0/2584553656 >> 172.21.15.16:0/2584553656 conn(0x7f321806ee40 msgr2=0x7f321806d4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:13:53.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:53.126+0000 7f321ce00640 1 -- 172.21.15.16:0/2584553656 shutdown_connections 2024-03-31T15:13:53.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:53.126+0000 7f321ce00640 1 -- 172.21.15.16:0/2584553656 wait complete. 2024-03-31T15:13:53.137 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-31T15:13:53.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:53.286+0000 7f201ea00640 1 Processor -- start 2024-03-31T15:13:53.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:53.286+0000 7f201ea00640 1 -- start start 2024-03-31T15:13:53.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:53.286+0000 7f201ea00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f2018145d10 0x7f20180cf790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:53.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:53.286+0000 7f201ea00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f20180b23d0 0x7f201815cd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:53.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:53.286+0000 7f201ea00640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2018160e70 con 0x7f20180728b0 2024-03-31T15:13:53.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:53.286+0000 7f201ea00640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f20181602f0 con 0x7f20180b23d0 2024-03-31T15:13:53.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:53.286+0000 7f201ea00640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f20181605d0 con 0x7f2018145d10 2024-03-31T15:13:53.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:53.286+0000 7f201ca00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f20180b23d0 0x7f201815cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:53.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:53.286+0000 7f201ca00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f20180b23d0 0x7f201815cd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50330/0 (socket says 172.21.15.16:50330) 2024-03-31T15:13:53.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:53.286+0000 7f201ca00640 1 -- 172.21.15.16:0/3405056448 learned_addr learned my addr 172.21.15.16:0/3405056448 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:13:53.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:53.286+0000 7f2017e00640 1 --2- 172.21.15.16:0/3405056448 >> v2:172.21.15.89:3300/0 conn(0x7f2018145d10 0x7f20180cf790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:53.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:53.286+0000 7f201ca00640 1 -- 172.21.15.16:0/3405056448 >> v2:172.21.15.89:3300/0 conn(0x7f2018145d10 msgr2=0x7f20180cf790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:13:53.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:53.286+0000 7f201ca00640 1 --2- 172.21.15.16:0/3405056448 >> v2:172.21.15.89:3300/0 conn(0x7f2018145d10 0x7f20180cf790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:53.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:53.286+0000 7f201ca00640 1 -- 172.21.15.16:0/3405056448 >> v1:172.21.15.89:6789/0 conn(0x7f20180728b0 legacy=0x7f20180b1cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:13:53.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:53.286+0000 7f201ca00640 1 -- 172.21.15.16:0/3405056448 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20180cfcd0 con 0x7f20180b23d0 2024-03-31T15:13:53.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:53.290+0000 7f201ca00640 1 --2- 172.21.15.16:0/3405056448 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f20180b23d0 0x7f201815cd50 secure :-1 s=READY pgs=1577 cs=0 l=1 rev1=1 crypto rx=0x7f20180d1060 tx=0x7f201001c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=596ef90ba38683ce server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:13:53.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:53.290+0000 7f2017400640 1 -- 172.21.15.16:0/3405056448 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20100a56c0 con 0x7f20180b23d0 2024-03-31T15:13:53.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:53.290+0000 7f2017400640 1 -- 172.21.15.16:0/3405056448 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f20100a5860 con 0x7f20180b23d0 2024-03-31T15:13:53.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:53.290+0000 7f2017400640 1 -- 172.21.15.16:0/3405056448 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20100a5c30 con 0x7f20180b23d0 2024-03-31T15:13:53.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:53.290+0000 7f201ea00640 1 -- 172.21.15.16:0/3405056448 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f20180b23d0 msgr2=0x7f201815cd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:13:53.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:53.290+0000 7f201ea00640 1 --2- 172.21.15.16:0/3405056448 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f20180b23d0 0x7f201815cd50 secure :-1 s=READY pgs=1577 cs=0 l=1 rev1=1 crypto rx=0x7f20180d1060 tx=0x7f201001c9f0 comp rx=0 tx=0).stop 2024-03-31T15:13:53.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:53.290+0000 7f201ea00640 1 -- 172.21.15.16:0/3405056448 shutdown_connections 2024-03-31T15:13:53.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:53.290+0000 7f201ea00640 1 --2- 172.21.15.16:0/3405056448 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f20180b23d0 0x7f201815cd50 unknown :-1 s=CLOSED pgs=1577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:53.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:53.290+0000 7f201ea00640 1 --2- 172.21.15.16:0/3405056448 >> v2:172.21.15.89:3300/0 conn(0x7f2018145d10 0x7f20180cf790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:53.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:53.290+0000 7f201ea00640 1 -- 172.21.15.16:0/3405056448 >> 172.21.15.16:0/3405056448 conn(0x7f201806b620 msgr2=0x7f201806ba30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:13:53.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:53.290+0000 7f201ea00640 1 -- 172.21.15.16:0/3405056448 shutdown_connections 2024-03-31T15:13:53.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:53.290+0000 7f201ea00640 1 -- 172.21.15.16:0/3405056448 wait complete. 2024-03-31T15:13:53.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:53.294+0000 7f201ea00640 1 Processor -- start 2024-03-31T15:13:53.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:53.294+0000 7f201ea00640 1 -- start start 2024-03-31T15:13:53.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:53.294+0000 7f201ea00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f20180728b0 0x7f201815aef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:53.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:53.294+0000 7f201ea00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f20180b23d0 0x7f201815ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:53.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:53.294+0000 7f201ea00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2018145d10 0x7f201815b830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:53.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:53.294+0000 7f201ea00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f20180cf4d0 con 0x7f20180728b0 2024-03-31T15:13:53.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:53.294+0000 7f201ea00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f20180cf350 con 0x7f2018145d10 2024-03-31T15:13:53.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:53.294+0000 7f201ea00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f2018072cc0 con 0x7f20180b23d0 2024-03-31T15:13:53.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:53.294+0000 7f2017e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f20180728b0 0x7f201815aef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:53.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:53.294+0000 7f2017e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f20180728b0 0x7f201815aef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50334/0 (socket says 172.21.15.16:50334) 2024-03-31T15:13:53.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:53.294+0000 7f2017e00640 1 -- 172.21.15.16:0/3089096882 learned_addr learned my addr 172.21.15.16:0/3089096882 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:13:53.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:53.294+0000 7f201ca00640 1 --2- 172.21.15.16:0/3089096882 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2018145d10 0x7f201815b830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:53.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:53.294+0000 7f200fe00640 1 --2- 172.21.15.16:0/3089096882 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f20180b23d0 0x7f201815ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:53.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:53.294+0000 7f2017e00640 1 -- 172.21.15.16:0/3089096882 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f20180b23d0 msgr2=0x7f201815ecd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:13:53.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:53.294+0000 7f2017e00640 1 --2- 172.21.15.16:0/3089096882 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f20180b23d0 0x7f201815ecd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:53.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:53.294+0000 7f2017e00640 1 -- 172.21.15.16:0/3089096882 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2018145d10 msgr2=0x7f201815b830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:13:53.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:53.294+0000 7f2017e00640 1 --2- 172.21.15.16:0/3089096882 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2018145d10 0x7f201815b830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:53.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:53.294+0000 7f2017e00640 1 -- 172.21.15.16:0/3089096882 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2018166070 con 0x7f20180728b0 2024-03-31T15:13:53.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:53.294+0000 7f200fe00640 1 --2- 172.21.15.16:0/3089096882 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f20180b23d0 0x7f201815ecd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:13:53.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:53.294+0000 7f2017e00640 1 --2- 172.21.15.16:0/3089096882 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f20180728b0 0x7f201815aef0 secure :-1 s=READY pgs=1578 cs=0 l=1 rev1=1 crypto rx=0x7f200800e8a0 tx=0x7f200800eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:13:53.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:53.294+0000 7f2014c00640 1 -- 172.21.15.16:0/3089096882 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2008017070 con 0x7f20180728b0 2024-03-31T15:13:53.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:53.294+0000 7f2014c00640 1 -- 172.21.15.16:0/3089096882 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f200800c940 con 0x7f20180728b0 2024-03-31T15:13:53.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:53.294+0000 7f2014c00640 1 -- 172.21.15.16:0/3089096882 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f200800ccf0 con 0x7f20180728b0 2024-03-31T15:13:53.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:53.294+0000 7f201ea00640 1 -- 172.21.15.16:0/3089096882 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f201813c810 con 0x7f20180728b0 2024-03-31T15:13:53.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:53.294+0000 7f201ea00640 1 -- 172.21.15.16:0/3089096882 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f201813cdd0 con 0x7f20180728b0 2024-03-31T15:13:53.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:53.294+0000 7f201ea00640 1 -- 172.21.15.16:0/3089096882 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2018069950 con 0x7f20180728b0 2024-03-31T15:13:53.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:53.298+0000 7f2014c00640 1 -- 172.21.15.16:0/3089096882 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f200801a070 con 0x7f20180728b0 2024-03-31T15:13:53.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:53.298+0000 7f2014c00640 1 --2- 172.21.15.16:0/3089096882 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f1ff4068380 0x7f1ff406a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:53.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:53.298+0000 7f2014c00640 1 -- 172.21.15.16:0/3089096882 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 5881+0+0 (secure 0 0 0) 0x7f2008086940 con 0x7f20180728b0 2024-03-31T15:13:53.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:53.298+0000 7f200fe00640 1 --2- 172.21.15.16:0/3089096882 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f1ff4068380 0x7f1ff406a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:53.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:53.298+0000 7f200fe00640 1 --2- 172.21.15.16:0/3089096882 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f1ff4068380 0x7f1ff406a840 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f2018165b00 tx=0x7f2000009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:13:53.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:53.302+0000 7f2014c00640 1 -- 172.21.15.16:0/3089096882 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f2008056760 con 0x7f20180728b0 2024-03-31T15:13:53.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:53.502+0000 7f201ea00640 1 -- 172.21.15.16:0/3089096882 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f20180b1eb0 con 0x7f20180728b0 2024-03-31T15:13:53.494 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:53.502+0000 7f2014c00640 1 -- 172.21.15.16:0/3089096882 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 91 v91) ==== 94+0+5169 (secure 0 0 0) 0x7f200805aa60 con 0x7f20180728b0 2024-03-31T15:13:53.494 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:13:53.494 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":91,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7139,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6836/3798928623","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":3798928623},{"type":"v1","addr":"172.21.15.89:6836","nonce":3798928623}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":78},{"gid":7142,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/3074283817","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6835","nonce":3074283817},{"type":"v1","addr":"172.21.15.89:6837","nonce":3074283817}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":78},{"gid":8071,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6834/2110189998","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":2110189998},{"type":"v1","addr":"172.21.15.16:6834","nonce":2110189998}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":78},{"gid":9027,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/2938692325","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6833","nonce":2938692325},{"type":"v1","addr":"172.21.15.16:6835","nonce":2938692325}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":91}],"filesystems":[{"mdsmap":{"epoch":90,"flags":19,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-31T15:12:16.294600+0000","modified":"2024-03-31T15:13:50.823935+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":98,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":7}]} 2024-03-31T15:13:53.495 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 91 2024-03-31T15:13:53.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:53.502+0000 7f201ea00640 1 -- 172.21.15.16:0/3089096882 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f1ff4068380 msgr2=0x7f1ff406a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:13:53.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:53.502+0000 7f201ea00640 1 --2- 172.21.15.16:0/3089096882 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f1ff4068380 0x7f1ff406a840 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f2018165b00 tx=0x7f2000009040 comp rx=0 tx=0).stop 2024-03-31T15:13:53.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:53.502+0000 7f201ea00640 1 -- 172.21.15.16:0/3089096882 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f20180728b0 msgr2=0x7f201815aef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:13:53.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:53.502+0000 7f201ea00640 1 --2- 172.21.15.16:0/3089096882 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f20180728b0 0x7f201815aef0 secure :-1 s=READY pgs=1578 cs=0 l=1 rev1=1 crypto rx=0x7f200800e8a0 tx=0x7f200800eca0 comp rx=0 tx=0).stop 2024-03-31T15:13:53.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:53.502+0000 7f201ea00640 1 -- 172.21.15.16:0/3089096882 shutdown_connections 2024-03-31T15:13:53.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:53.502+0000 7f201ea00640 1 --2- 172.21.15.16:0/3089096882 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f1ff4068380 0x7f1ff406a840 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:53.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:53.502+0000 7f201ea00640 1 --2- 172.21.15.16:0/3089096882 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2018145d10 0x7f201815b830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:53.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:53.502+0000 7f201ea00640 1 --2- 172.21.15.16:0/3089096882 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f20180b23d0 0x7f201815ecd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:53.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:53.502+0000 7f201ea00640 1 --2- 172.21.15.16:0/3089096882 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f20180728b0 0x7f201815aef0 unknown :-1 s=CLOSED pgs=1578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:53.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:53.502+0000 7f201ea00640 1 -- 172.21.15.16:0/3089096882 >> 172.21.15.16:0/3089096882 conn(0x7f201806b620 msgr2=0x7f20180cff50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:13:53.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:53.502+0000 7f201ea00640 1 -- 172.21.15.16:0/3089096882 shutdown_connections 2024-03-31T15:13:53.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:53.502+0000 7f201ea00640 1 -- 172.21.15.16:0/3089096882 wait complete. 2024-03-31T15:13:53.515 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-31T15:13:53.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:53.662+0000 7f79ade00640 1 Processor -- start 2024-03-31T15:13:53.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:53.662+0000 7f79ade00640 1 -- start start 2024-03-31T15:13:53.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:53.662+0000 7f79ade00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f79a80728b0 0x7f79a80b1cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:53.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:53.662+0000 7f79ade00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f79a80b2200 0x7f79a815cd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:53.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:53.662+0000 7f79ade00640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f79a8160e70 con 0x7f79a8145d10 2024-03-31T15:13:53.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:53.662+0000 7f79ade00640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f79a81602f0 con 0x7f79a80b2200 2024-03-31T15:13:53.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:53.662+0000 7f79ade00640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f79a81605d0 con 0x7f79a80728b0 2024-03-31T15:13:53.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:53.662+0000 7f79a6a00640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f79a8145d10 0x7f79a80cf790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:44772/0 (socket says 172.21.15.16:44772) 2024-03-31T15:13:53.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:53.662+0000 7f79a7400640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f79a80b2200 0x7f79a815cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:53.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:53.662+0000 7f79a6a00640 1 -- 172.21.15.16:0/605074588 learned_addr learned my addr 172.21.15.16:0/605074588 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:13:53.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:53.662+0000 7f799e000640 1 --2- 172.21.15.16:0/605074588 >> v2:172.21.15.89:3300/0 conn(0x7f79a80728b0 0x7f79a80b1cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:53.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:53.662+0000 7f79a7400640 1 -- 172.21.15.16:0/605074588 >> v2:172.21.15.89:3300/0 conn(0x7f79a80728b0 msgr2=0x7f79a80b1cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:13:53.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:53.662+0000 7f79a7400640 1 --2- 172.21.15.16:0/605074588 >> v2:172.21.15.89:3300/0 conn(0x7f79a80728b0 0x7f79a80b1cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:53.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:53.662+0000 7f79a7400640 1 -- 172.21.15.16:0/605074588 >> v1:172.21.15.16:6789/0 conn(0x7f79a8145d10 legacy=0x7f79a80cf790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:13:53.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:53.662+0000 7f79a7400640 1 -- 172.21.15.16:0/605074588 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79a81608b0 con 0x7f79a80b2200 2024-03-31T15:13:53.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:53.662+0000 7f79a6000640 1 -- 172.21.15.16:0/605074588 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 657270391 0 0) 0x7f79a8160e70 con 0x7f79a8145d10 2024-03-31T15:13:53.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:53.662+0000 7f799e000640 1 --2- 172.21.15.16:0/605074588 >> v2:172.21.15.89:3300/0 conn(0x7f79a80728b0 0x7f79a80b1cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:13:53.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:53.666+0000 7f79a7400640 1 --2- 172.21.15.16:0/605074588 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f79a80b2200 0x7f79a815cd50 secure :-1 s=READY pgs=1580 cs=0 l=1 rev1=1 crypto rx=0x7f79a80b2980 tx=0x7f79a001c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14f664d8a4d4473c server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:13:53.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:53.666+0000 7f79a6000640 1 -- 172.21.15.16:0/605074588 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79a00b0070 con 0x7f79a80b2200 2024-03-31T15:13:53.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:53.666+0000 7f79a6000640 1 -- 172.21.15.16:0/605074588 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f79a00ab600 con 0x7f79a80b2200 2024-03-31T15:13:53.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:53.666+0000 7f79a6000640 1 -- 172.21.15.16:0/605074588 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79a00ab9d0 con 0x7f79a80b2200 2024-03-31T15:13:53.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:53.666+0000 7f79ade00640 1 -- 172.21.15.16:0/605074588 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f79a80b2200 msgr2=0x7f79a815cd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:13:53.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:53.666+0000 7f79ade00640 1 --2- 172.21.15.16:0/605074588 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f79a80b2200 0x7f79a815cd50 secure :-1 s=READY pgs=1580 cs=0 l=1 rev1=1 crypto rx=0x7f79a80b2980 tx=0x7f79a001c480 comp rx=0 tx=0).stop 2024-03-31T15:13:53.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:53.666+0000 7f79ade00640 1 -- 172.21.15.16:0/605074588 shutdown_connections 2024-03-31T15:13:53.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:53.666+0000 7f79ade00640 1 --2- 172.21.15.16:0/605074588 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f79a80b2200 0x7f79a815cd50 unknown :-1 s=CLOSED pgs=1580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:53.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:53.666+0000 7f79ade00640 1 --2- 172.21.15.16:0/605074588 >> v2:172.21.15.89:3300/0 conn(0x7f79a80728b0 0x7f79a80b1cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:53.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:53.666+0000 7f79ade00640 1 -- 172.21.15.16:0/605074588 >> 172.21.15.16:0/605074588 conn(0x7f79a806b620 msgr2=0x7f79a806ba30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:13:53.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:53.666+0000 7f79ade00640 1 -- 172.21.15.16:0/605074588 shutdown_connections 2024-03-31T15:13:53.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:53.666+0000 7f79ade00640 1 -- 172.21.15.16:0/605074588 wait complete. 2024-03-31T15:13:53.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:53.670+0000 7f79ade00640 1 Processor -- start 2024-03-31T15:13:53.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:53.670+0000 7f79ade00640 1 -- start start 2024-03-31T15:13:53.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:53.670+0000 7f79ade00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f79a80728b0 0x7f79a8138bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:53.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:53.670+0000 7f79ade00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f79a80b2200 0x7f79a8157d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:53.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:53.670+0000 7f79ade00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f79a8145d10 0x7f79a815b2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:53.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:53.670+0000 7f79ade00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f79a8072e40 con 0x7f79a80728b0 2024-03-31T15:13:53.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:53.670+0000 7f79ade00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f79a8072cc0 con 0x7f79a80b2200 2024-03-31T15:13:53.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:53.670+0000 7f79ade00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f79a81690f0 con 0x7f79a8145d10 2024-03-31T15:13:53.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:53.670+0000 7f79a6a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f79a80728b0 0x7f79a8138bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:53.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:53.670+0000 7f79a6a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f79a80728b0 0x7f79a8138bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50344/0 (socket says 172.21.15.16:50344) 2024-03-31T15:13:53.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:53.670+0000 7f79a6a00640 1 -- 172.21.15.16:0/991522767 learned_addr learned my addr 172.21.15.16:0/991522767 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:13:53.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:53.670+0000 7f799e000640 1 --2- 172.21.15.16:0/991522767 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f79a80b2200 0x7f79a8157d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:53.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:53.670+0000 7f79a7400640 1 --2- 172.21.15.16:0/991522767 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f79a8145d10 0x7f79a815b2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:53.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:53.670+0000 7f79a6a00640 1 -- 172.21.15.16:0/991522767 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f79a8145d10 msgr2=0x7f79a815b2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:13:53.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:53.670+0000 7f79a6a00640 1 --2- 172.21.15.16:0/991522767 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f79a8145d10 0x7f79a815b2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:53.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:53.670+0000 7f79a6a00640 1 -- 172.21.15.16:0/991522767 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f79a80b2200 msgr2=0x7f79a8157d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:13:53.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:53.670+0000 7f79a6a00640 1 --2- 172.21.15.16:0/991522767 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f79a80b2200 0x7f79a8157d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:53.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:53.670+0000 7f79a6a00640 1 -- 172.21.15.16:0/991522767 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79a81582a0 con 0x7f79a80728b0 2024-03-31T15:13:53.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:53.670+0000 7f79a6a00640 1 --2- 172.21.15.16:0/991522767 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f79a80728b0 0x7f79a8138bd0 secure :-1 s=READY pgs=1581 cs=0 l=1 rev1=1 crypto rx=0x7f799800c480 tx=0x7f799800c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:13:53.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:53.670+0000 7f799fe00640 1 -- 172.21.15.16:0/991522767 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7998018070 con 0x7f79a80728b0 2024-03-31T15:13:53.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:53.670+0000 7f799fe00640 1 -- 172.21.15.16:0/991522767 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7998007500 con 0x7f79a80728b0 2024-03-31T15:13:53.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:53.670+0000 7f79ade00640 1 -- 172.21.15.16:0/991522767 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f79a8159b60 con 0x7f79a80728b0 2024-03-31T15:13:53.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:53.670+0000 7f799fe00640 1 -- 172.21.15.16:0/991522767 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7998004a30 con 0x7f79a80728b0 2024-03-31T15:13:53.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:53.670+0000 7f79ade00640 1 -- 172.21.15.16:0/991522767 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f79a815f4b0 con 0x7f79a80728b0 2024-03-31T15:13:53.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:53.674+0000 7f79ade00640 1 -- 172.21.15.16:0/991522767 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f79a8069950 con 0x7f79a80728b0 2024-03-31T15:13:53.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:53.674+0000 7f799fe00640 1 -- 172.21.15.16:0/991522767 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f799801b070 con 0x7f79a80728b0 2024-03-31T15:13:53.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:53.674+0000 7f799fe00640 1 --2- 172.21.15.16:0/991522767 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f7984068380 0x7f798406a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:53.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:53.674+0000 7f799fe00640 1 -- 172.21.15.16:0/991522767 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 5881+0+0 (secure 0 0 0) 0x7f7998085d60 con 0x7f79a80728b0 2024-03-31T15:13:53.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:53.674+0000 7f799e000640 1 --2- 172.21.15.16:0/991522767 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f7984068380 0x7f798406a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:53.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:53.674+0000 7f799e000640 1 --2- 172.21.15.16:0/991522767 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f7984068380 0x7f798406a840 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f79a81577f0 tx=0x7f7990003a20 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:13:53.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:53.678+0000 7f799fe00640 1 -- 172.21.15.16:0/991522767 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f7998056260 con 0x7f79a80728b0 2024-03-31T15:13:53.835 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:53.842+0000 7f79ade00640 1 -- 172.21.15.16:0/991522767 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f79a80ce040 con 0x7f79a80728b0 2024-03-31T15:13:53.836 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:53.842+0000 7f799fe00640 1 -- 172.21.15.16:0/991522767 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v98) ==== 74+0+16791 (secure 0 0 0) 0x7f7998055b60 con 0x7f79a80728b0 2024-03-31T15:13:53.836 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:13:53.836 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":98,"fsid":"286eb5da-fb47-4550-b290-d9519c13b9a5","created":"2024-03-31T15:03:11.438952+0000","modified":"2024-03-31T15:13:50.823773+0000","last_up_change":"2024-03-31T15:03:22.885535+0000","last_in_change":"2024-03-31T15:03:13.941667+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-31T15:03:22.550787+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2024-03-31T15:12:14.671225+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"95","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2024-03-31T15:12:15.242196+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"96","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"92a66450-ef19-40d8-ac3d-d760b23b0bda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6801","nonce":932712244}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6803","nonce":932712244}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6807","nonce":932712244}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6805","nonce":932712244}]},"public_addr":"172.21.15.16:6801/932712244","cluster_addr":"172.21.15.16:6803/932712244","heartbeat_back_addr":"172.21.15.16:6807/932712244","heartbeat_front_addr":"172.21.15.16:6805/932712244","state":["exists","up"]},{"osd":1,"uuid":"da9d1f2b-1968-4959-b843-faa231988e92","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6810","nonce":1398737533}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6811","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6813","nonce":1398737533}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6821","nonce":1398737533}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6815","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6816","nonce":1398737533}]},"public_addr":"172.21.15.16:6810/1398737533","cluster_addr":"172.21.15.16:6813/1398737533","heartbeat_back_addr":"172.21.15.16:6821/1398737533","heartbeat_front_addr":"172.21.15.16:6816/1398737533","state":["exists","up"]},{"osd":2,"uuid":"01a73662-9a21-4b1f-bcbb-42b9636a4638","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6819","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6822","nonce":3018526646}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6825","nonce":3018526646}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6831","nonce":3018526646}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6827","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6829","nonce":3018526646}]},"public_addr":"172.21.15.16:6822/3018526646","cluster_addr":"172.21.15.16:6825/3018526646","heartbeat_back_addr":"172.21.15.16:6831/3018526646","heartbeat_front_addr":"172.21.15.16:6829/3018526646","state":["exists","up"]},{"osd":3,"uuid":"c7771fd3-e18e-4500-ac38-23cf10ebcd75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6809","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6812","nonce":3360352974}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6817","nonce":3360352974}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6828","nonce":3360352974}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6823","nonce":3360352974}]},"public_addr":"172.21.15.16:6812/3360352974","cluster_addr":"172.21.15.16:6817/3360352974","heartbeat_back_addr":"172.21.15.16:6828/3360352974","heartbeat_front_addr":"172.21.15.16:6823/3360352974","state":["exists","up"]},{"osd":4,"uuid":"88260730-1482-4fbd-a40f-cb1124e9df4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6801","nonce":1856844856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6803","nonce":1856844856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6807","nonce":1856844856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6805","nonce":1856844856}]},"public_addr":"172.21.15.89:6801/1856844856","cluster_addr":"172.21.15.89:6803/1856844856","heartbeat_back_addr":"172.21.15.89:6807/1856844856","heartbeat_front_addr":"172.21.15.89:6805/1856844856","state":["exists","up"]},{"osd":5,"uuid":"4cb2c00c-fae0-4dd8-9d27-ec4f0093ccf7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6814","nonce":50395527}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6817","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6820","nonce":50395527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6831","nonce":50395527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6823","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6826","nonce":50395527}]},"public_addr":"172.21.15.89:6814/50395527","cluster_addr":"172.21.15.89:6820/50395527","heartbeat_back_addr":"172.21.15.89:6831/50395527","heartbeat_front_addr":"172.21.15.89:6826/50395527","state":["exists","up"]},{"osd":6,"uuid":"81ce0c19-2554-4cae-acbe-66a05e9e9b9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6809","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6812","nonce":3470471752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6815","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6818","nonce":3470471752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6827","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6829","nonce":3470471752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6821","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6824","nonce":3470471752}]},"public_addr":"172.21.15.89:6812/3470471752","cluster_addr":"172.21.15.89:6818/3470471752","heartbeat_back_addr":"172.21.15.89:6829/3470471752","heartbeat_front_addr":"172.21.15.89:6824/3470471752","state":["exists","up"]},{"osd":7,"uuid":"85e4313b-d37d-442e-8042-153bf20a847d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6811","nonce":1796778630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6813","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6816","nonce":1796778630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6825","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6828","nonce":1796778630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6819","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6822","nonce":1796778630}]},"public_addr":"172.21.15.89:6811/1796778630","cluster_addr":"172.21.15.89:6816/1796778630","heartbeat_back_addr":"172.21.15.89:6828/1796778630","heartbeat_front_addr":"172.21.15.89:6822/1796778630","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.439088+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.529970+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.680512+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:21.044161+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.582111+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.536558+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.409522+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.590229+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:6835/567808781":"2024-04-01T15:13:50.823751+0000","172.21.15.16:6833/567808781":"2024-04-01T15:13:50.823751+0000","172.21.15.16:6835/490717480":"2024-04-01T15:13:14.080258+0000","172.21.15.16:6833/490717480":"2024-04-01T15:13:14.080258+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-31T15:13:53.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:53.846+0000 7f79ade00640 1 -- 172.21.15.16:0/991522767 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f7984068380 msgr2=0x7f798406a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:13:53.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:53.846+0000 7f79ade00640 1 --2- 172.21.15.16:0/991522767 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f7984068380 0x7f798406a840 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f79a81577f0 tx=0x7f7990003a20 comp rx=0 tx=0).stop 2024-03-31T15:13:53.839 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:53.846+0000 7f79ade00640 1 -- 172.21.15.16:0/991522767 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f79a80728b0 msgr2=0x7f79a8138bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:13:53.839 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:53.846+0000 7f79ade00640 1 --2- 172.21.15.16:0/991522767 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f79a80728b0 0x7f79a8138bd0 secure :-1 s=READY pgs=1581 cs=0 l=1 rev1=1 crypto rx=0x7f799800c480 tx=0x7f799800c940 comp rx=0 tx=0).stop 2024-03-31T15:13:53.839 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:53.846+0000 7f79ade00640 1 -- 172.21.15.16:0/991522767 shutdown_connections 2024-03-31T15:13:53.839 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:53.846+0000 7f79ade00640 1 --2- 172.21.15.16:0/991522767 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f7984068380 0x7f798406a840 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:53.839 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:53.846+0000 7f79ade00640 1 --2- 172.21.15.16:0/991522767 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f79a8145d10 0x7f79a815b2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:53.839 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:53.846+0000 7f79ade00640 1 --2- 172.21.15.16:0/991522767 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f79a80b2200 0x7f79a8157d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:53.839 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:53.846+0000 7f79ade00640 1 --2- 172.21.15.16:0/991522767 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f79a80728b0 0x7f79a8138bd0 unknown :-1 s=CLOSED pgs=1581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:53.839 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:53.846+0000 7f79ade00640 1 -- 172.21.15.16:0/991522767 >> 172.21.15.16:0/991522767 conn(0x7f79a806b620 msgr2=0x7f79a80b0660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:13:53.839 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:53.846+0000 7f79ade00640 1 -- 172.21.15.16:0/991522767 shutdown_connections 2024-03-31T15:13:53.839 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:53.846+0000 7f79ade00640 1 -- 172.21.15.16:0/991522767 wait complete. 2024-03-31T15:13:53.858 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-31T15:13:53.858 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-31T15:13:53.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:54.006+0000 7f5315000640 1 Processor -- start 2024-03-31T15:13:53.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:54.006+0000 7f5315000640 1 -- start start 2024-03-31T15:13:54.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:54.006+0000 7f5315000640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f5310147e70 0x7f531006b620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:54.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:54.006+0000 7f5315000640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f53101621b0 con 0x7f53100d0c50 2024-03-31T15:13:54.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:54.006+0000 7f5315000640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5310162490 con 0x7f53100736e0 2024-03-31T15:13:54.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:54.006+0000 7f5315000640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f5310162770 con 0x7f5310147e70 2024-03-31T15:13:54.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:54.006+0000 7f5305400640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f53100d0c50 0x7f5310072fd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:44778/0 (socket says 172.21.15.16:44778) 2024-03-31T15:13:54.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:54.006+0000 7f5305400640 1 -- 172.21.15.16:0/3562179918 learned_addr learned my addr 172.21.15.16:0/3562179918 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:13:54.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:54.006+0000 7f530de00640 1 --2- 172.21.15.16:0/3562179918 >> v2:172.21.15.89:3300/0 conn(0x7f5310147e70 0x7f531006b620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:54.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:54.006+0000 7f530d400640 1 -- 172.21.15.16:0/3562179918 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3652564449 0 0) 0x7f53101621b0 con 0x7f53100d0c50 2024-03-31T15:13:54.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:54.006+0000 7f530d400640 1 -- 172.21.15.16:0/3562179918 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f52f4003530 con 0x7f53100d0c50 2024-03-31T15:13:54.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:54.006+0000 7f530d400640 1 -- 172.21.15.16:0/3562179918 <== mon.1 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2432324841 0 0) 0x7f5310162490 con 0x7f53100736e0 2024-03-31T15:13:54.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:54.006+0000 7f530d400640 1 -- 172.21.15.16:0/3562179918 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f53101621b0 con 0x7f53100736e0 2024-03-31T15:13:54.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:54.006+0000 7f530de00640 1 -- 172.21.15.16:0/3562179918 >> v1:172.21.15.89:6789/0 conn(0x7f53100736e0 legacy=0x7f531015eb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:13:54.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:54.006+0000 7f530de00640 1 -- 172.21.15.16:0/3562179918 >> v1:172.21.15.16:6789/0 conn(0x7f53100d0c50 legacy=0x7f5310072fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:13:54.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:54.006+0000 7f530de00640 1 -- 172.21.15.16:0/3562179918 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5310162a50 con 0x7f5310147e70 2024-03-31T15:13:54.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:54.010+0000 7f530de00640 1 --2- 172.21.15.16:0/3562179918 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5310147e70 0x7f531006b620 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f5300009d90 tx=0x7f530001c970 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f7b96e4fff10a35d server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:13:54.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:54.010+0000 7f530d400640 1 -- 172.21.15.16:0/3562179918 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53000b2070 con 0x7f5310147e70 2024-03-31T15:13:54.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:54.010+0000 7f530d400640 1 -- 172.21.15.16:0/3562179918 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f53000a58b0 con 0x7f5310147e70 2024-03-31T15:13:54.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:54.010+0000 7f530d400640 1 -- 172.21.15.16:0/3562179918 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53000a5c60 con 0x7f5310147e70 2024-03-31T15:13:54.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:54.010+0000 7f5315000640 1 -- 172.21.15.16:0/3562179918 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5310147e70 msgr2=0x7f531006b620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:13:54.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:54.010+0000 7f5315000640 1 --2- 172.21.15.16:0/3562179918 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5310147e70 0x7f531006b620 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f5300009d90 tx=0x7f530001c970 comp rx=0 tx=0).stop 2024-03-31T15:13:54.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:54.010+0000 7f5315000640 1 -- 172.21.15.16:0/3562179918 shutdown_connections 2024-03-31T15:13:54.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:54.010+0000 7f5315000640 1 --2- 172.21.15.16:0/3562179918 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5310147e70 0x7f531006b620 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:54.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:54.010+0000 7f5315000640 1 -- 172.21.15.16:0/3562179918 >> 172.21.15.16:0/3562179918 conn(0x7f53100d1060 msgr2=0x7f53100d1470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:13:54.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:54.010+0000 7f5315000640 1 -- 172.21.15.16:0/3562179918 shutdown_connections 2024-03-31T15:13:54.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:54.014+0000 7f5315000640 1 -- 172.21.15.16:0/3562179918 wait complete. 2024-03-31T15:13:54.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:54.014+0000 7f5315000640 1 Processor -- start 2024-03-31T15:13:54.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:54.014+0000 7f5315000640 1 -- start start 2024-03-31T15:13:54.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:54.014+0000 7f5315000640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f53100736e0 0x7f531015d970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:54.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:54.014+0000 7f5315000640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f53100d0c50 0x7f5310138420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:54.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:54.014+0000 7f5315000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5310147e70 0x7f5310138960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:54.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:54.014+0000 7f5315000640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f531016c640 con 0x7f5310147e70 2024-03-31T15:13:54.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:54.014+0000 7f5315000640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f531016c4c0 con 0x7f53100d0c50 2024-03-31T15:13:54.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:54.014+0000 7f5315000640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f531016c7c0 con 0x7f53100736e0 2024-03-31T15:13:54.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:54.014+0000 7f530e800640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5310147e70 0x7f5310138960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:54.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:54.014+0000 7f5305400640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f53100d0c50 0x7f5310138420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:54.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:54.014+0000 7f530e800640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5310147e70 0x7f5310138960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50358/0 (socket says 172.21.15.16:50358) 2024-03-31T15:13:54.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:54.014+0000 7f530de00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f53100736e0 0x7f531015d970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:54.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:54.014+0000 7f530e800640 1 -- 172.21.15.16:0/1314742033 learned_addr learned my addr 172.21.15.16:0/1314742033 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:13:54.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:54.014+0000 7f530e800640 1 -- 172.21.15.16:0/1314742033 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f53100736e0 msgr2=0x7f531015d970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:13:54.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:54.014+0000 7f530e800640 1 --2- 172.21.15.16:0/1314742033 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f53100736e0 0x7f531015d970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:54.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:54.014+0000 7f530e800640 1 -- 172.21.15.16:0/1314742033 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f53100d0c50 msgr2=0x7f5310138420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:13:54.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:54.014+0000 7f530e800640 1 --2- 172.21.15.16:0/1314742033 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f53100d0c50 0x7f5310138420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:54.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:54.014+0000 7f530e800640 1 -- 172.21.15.16:0/1314742033 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5310138fc0 con 0x7f5310147e70 2024-03-31T15:13:54.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:54.014+0000 7f530e800640 1 --2- 172.21.15.16:0/1314742033 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5310147e70 0x7f5310138960 secure :-1 s=READY pgs=1583 cs=0 l=1 rev1=1 crypto rx=0x7f530800c460 tx=0x7f530800c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:13:54.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:54.014+0000 7f5306a00640 1 -- 172.21.15.16:0/1314742033 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53080052d0 con 0x7f5310147e70 2024-03-31T15:13:54.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:54.018+0000 7f5306a00640 1 -- 172.21.15.16:0/1314742033 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5308005470 con 0x7f5310147e70 2024-03-31T15:13:54.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:54.018+0000 7f5315000640 1 -- 172.21.15.16:0/1314742033 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f53101392a0 con 0x7f5310147e70 2024-03-31T15:13:54.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:54.018+0000 7f5306a00640 1 -- 172.21.15.16:0/1314742033 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5308012870 con 0x7f5310147e70 2024-03-31T15:13:54.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:54.018+0000 7f5315000640 1 -- 172.21.15.16:0/1314742033 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f53100ae0d0 con 0x7f5310147e70 2024-03-31T15:13:54.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:54.018+0000 7f5315000640 1 -- 172.21.15.16:0/1314742033 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f53100ae2d0 con 0x7f5310147e70 2024-03-31T15:13:54.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:54.018+0000 7f5306a00640 1 -- 172.21.15.16:0/1314742033 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f5308012a10 con 0x7f5310147e70 2024-03-31T15:13:54.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:54.018+0000 7f5306a00640 1 --2- 172.21.15.16:0/1314742033 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f52ec068380 0x7f52ec06a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:54.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:54.018+0000 7f5306a00640 1 -- 172.21.15.16:0/1314742033 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 5881+0+0 (secure 0 0 0) 0x7f5308086520 con 0x7f5310147e70 2024-03-31T15:13:54.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:54.018+0000 7f530de00640 1 --2- 172.21.15.16:0/1314742033 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f52ec068380 0x7f52ec06a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:54.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:54.018+0000 7f530de00640 1 --2- 172.21.15.16:0/1314742033 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f52ec068380 0x7f52ec06a840 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f53000af000 tx=0x7f53000a7070 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:13:54.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:54.022+0000 7f5306a00640 1 -- 172.21.15.16:0/1314742033 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f53100ae2d0 con 0x7f5310147e70 2024-03-31T15:13:54.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:54.226+0000 7f5315000640 1 -- 172.21.15.16:0/1314742033 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5310069950 con 0x7f5310147e70 2024-03-31T15:13:54.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:54.226+0000 7f5306a00640 1 -- 172.21.15.16:0/1314742033 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 91 v91) ==== 94+0+5169 (secure 0 0 0) 0x7f5308056350 con 0x7f5310147e70 2024-03-31T15:13:54.218 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:13:54.218 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":91,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7139,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6836/3798928623","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":3798928623},{"type":"v1","addr":"172.21.15.89:6836","nonce":3798928623}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":78},{"gid":7142,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/3074283817","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6835","nonce":3074283817},{"type":"v1","addr":"172.21.15.89:6837","nonce":3074283817}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":78},{"gid":8071,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6834/2110189998","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":2110189998},{"type":"v1","addr":"172.21.15.16:6834","nonce":2110189998}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":78},{"gid":9027,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/2938692325","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6833","nonce":2938692325},{"type":"v1","addr":"172.21.15.16:6835","nonce":2938692325}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":91}],"filesystems":[{"mdsmap":{"epoch":90,"flags":19,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-31T15:12:16.294600+0000","modified":"2024-03-31T15:13:50.823935+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":98,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":7}]} 2024-03-31T15:13:54.218 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 91 2024-03-31T15:13:54.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:54.226+0000 7f5315000640 1 -- 172.21.15.16:0/1314742033 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f52ec068380 msgr2=0x7f52ec06a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:13:54.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:54.226+0000 7f5315000640 1 --2- 172.21.15.16:0/1314742033 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f52ec068380 0x7f52ec06a840 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f53000af000 tx=0x7f53000a7070 comp rx=0 tx=0).stop 2024-03-31T15:13:54.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:54.226+0000 7f5315000640 1 -- 172.21.15.16:0/1314742033 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5310147e70 msgr2=0x7f5310138960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:13:54.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:54.226+0000 7f5315000640 1 --2- 172.21.15.16:0/1314742033 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5310147e70 0x7f5310138960 secure :-1 s=READY pgs=1583 cs=0 l=1 rev1=1 crypto rx=0x7f530800c460 tx=0x7f530800c920 comp rx=0 tx=0).stop 2024-03-31T15:13:54.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:54.226+0000 7f5315000640 1 -- 172.21.15.16:0/1314742033 shutdown_connections 2024-03-31T15:13:54.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:54.226+0000 7f5315000640 1 --2- 172.21.15.16:0/1314742033 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f52ec068380 0x7f52ec06a840 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:54.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:54.226+0000 7f5315000640 1 --2- 172.21.15.16:0/1314742033 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5310147e70 0x7f5310138960 unknown :-1 s=CLOSED pgs=1583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:54.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:54.226+0000 7f5315000640 1 --2- 172.21.15.16:0/1314742033 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f53100d0c50 0x7f5310138420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:54.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:54.226+0000 7f5315000640 1 --2- 172.21.15.16:0/1314742033 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f53100736e0 0x7f531015d970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:54.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:54.226+0000 7f5315000640 1 -- 172.21.15.16:0/1314742033 >> 172.21.15.16:0/1314742033 conn(0x7f53100d1060 msgr2=0x7f531013fd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:13:54.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:54.226+0000 7f5315000640 1 -- 172.21.15.16:0/1314742033 shutdown_connections 2024-03-31T15:13:54.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:54.226+0000 7f5315000640 1 -- 172.21.15.16:0/1314742033 wait complete. 2024-03-31T15:13:54.239 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-31T15:13:54.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:54.386+0000 7ffa73200640 1 Processor -- start 2024-03-31T15:13:54.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:54.386+0000 7ffa73200640 1 -- start start 2024-03-31T15:13:54.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:54.386+0000 7ffa73200640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ffa6c06fd80 0x7ffa6c0d2720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:54.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:54.386+0000 7ffa73200640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffa6c0d0920 con 0x7ffa6c145e40 2024-03-31T15:13:54.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:54.386+0000 7ffa73200640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffa6c0d0c00 con 0x7ffa6c06f920 2024-03-31T15:13:54.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:54.386+0000 7ffa73200640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7ffa6c0d06d0 con 0x7ffa6c06fd80 2024-03-31T15:13:54.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:54.386+0000 7ffa6be00640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7ffa6c145e40 0x7ffa6c0cc740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:44786/0 (socket says 172.21.15.16:44786) 2024-03-31T15:13:54.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:54.390+0000 7ffa6be00640 1 -- 172.21.15.16:0/952561807 learned_addr learned my addr 172.21.15.16:0/952561807 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:13:54.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:54.390+0000 7ffa71200640 1 --2- 172.21.15.16:0/952561807 >> v2:172.21.15.16:3300/0 conn(0x7ffa6c06fd80 0x7ffa6c0d2720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:54.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:54.390+0000 7ffa6b400640 1 -- 172.21.15.16:0/952561807 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 665369414 0 0) 0x7ffa6c0d0920 con 0x7ffa6c145e40 2024-03-31T15:13:54.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:54.390+0000 7ffa71200640 1 -- 172.21.15.16:0/952561807 >> v1:172.21.15.89:6789/0 conn(0x7ffa6c06f920 legacy=0x7ffa6c0d2010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:13:54.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:54.390+0000 7ffa71200640 1 -- 172.21.15.16:0/952561807 >> v1:172.21.15.16:6789/0 conn(0x7ffa6c145e40 legacy=0x7ffa6c0cc740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:13:54.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:54.390+0000 7ffa71200640 1 -- 172.21.15.16:0/952561807 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa6c0d1810 con 0x7ffa6c06fd80 2024-03-31T15:13:54.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:54.390+0000 7ffa71200640 1 --2- 172.21.15.16:0/952561807 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffa6c06fd80 0x7ffa6c0d2720 secure :-1 s=READY pgs=1585 cs=0 l=1 rev1=1 crypto rx=0x7ffa64009d90 tx=0x7ffa6401c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b59111b4b80184c1 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:13:54.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:54.390+0000 7ffa6b400640 1 -- 172.21.15.16:0/952561807 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa640b2070 con 0x7ffa6c06fd80 2024-03-31T15:13:54.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:54.390+0000 7ffa6b400640 1 -- 172.21.15.16:0/952561807 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffa640a5920 con 0x7ffa6c06fd80 2024-03-31T15:13:54.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:54.390+0000 7ffa6b400640 1 -- 172.21.15.16:0/952561807 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa640a5cf0 con 0x7ffa6c06fd80 2024-03-31T15:13:54.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:54.394+0000 7ffa73200640 1 -- 172.21.15.16:0/952561807 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffa6c06fd80 msgr2=0x7ffa6c0d2720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:13:54.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:54.394+0000 7ffa73200640 1 --2- 172.21.15.16:0/952561807 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffa6c06fd80 0x7ffa6c0d2720 secure :-1 s=READY pgs=1585 cs=0 l=1 rev1=1 crypto rx=0x7ffa64009d90 tx=0x7ffa6401c970 comp rx=0 tx=0).stop 2024-03-31T15:13:54.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:54.394+0000 7ffa73200640 1 -- 172.21.15.16:0/952561807 shutdown_connections 2024-03-31T15:13:54.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:54.394+0000 7ffa73200640 1 --2- 172.21.15.16:0/952561807 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffa6c06fd80 0x7ffa6c0d2720 unknown :-1 s=CLOSED pgs=1585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:54.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:54.394+0000 7ffa73200640 1 -- 172.21.15.16:0/952561807 >> 172.21.15.16:0/952561807 conn(0x7ffa6c067330 msgr2=0x7ffa6c067740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:13:54.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:54.394+0000 7ffa73200640 1 -- 172.21.15.16:0/952561807 shutdown_connections 2024-03-31T15:13:54.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:54.394+0000 7ffa73200640 1 -- 172.21.15.16:0/952561807 wait complete. 2024-03-31T15:13:54.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:54.394+0000 7ffa73200640 1 Processor -- start 2024-03-31T15:13:54.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:54.394+0000 7ffa73200640 1 -- start start 2024-03-31T15:13:54.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:54.394+0000 7ffa73200640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffa6c06f920 0x7ffa6c06d590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:54.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:54.394+0000 7ffa73200640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ffa6c06fd80 0x7ffa6c15efe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:54.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:54.394+0000 7ffa73200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffa6c145e40 0x7ffa6c15f520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:54.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:54.394+0000 7ffa73200640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7ffa6c0d0ee0 con 0x7ffa6c145e40 2024-03-31T15:13:54.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:54.394+0000 7ffa73200640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7ffa6c0ce9e0 con 0x7ffa6c06f920 2024-03-31T15:13:54.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:54.394+0000 7ffa73200640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7ffa6c0b50c0 con 0x7ffa6c06fd80 2024-03-31T15:13:54.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:54.394+0000 7ffa71200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffa6c145e40 0x7ffa6c15f520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:54.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:54.394+0000 7ffa71200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffa6c145e40 0x7ffa6c15f520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50368/0 (socket says 172.21.15.16:50368) 2024-03-31T15:13:54.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:54.394+0000 7ffa71200640 1 -- 172.21.15.16:0/2981191255 learned_addr learned my addr 172.21.15.16:0/2981191255 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:13:54.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:54.394+0000 7ffa6be00640 1 --2- 172.21.15.16:0/2981191255 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffa6c06f920 0x7ffa6c06d590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:54.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:54.394+0000 7ffa63e00640 1 --2- 172.21.15.16:0/2981191255 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ffa6c06fd80 0x7ffa6c15efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:54.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:54.394+0000 7ffa71200640 1 -- 172.21.15.16:0/2981191255 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ffa6c06fd80 msgr2=0x7ffa6c15efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:13:54.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:54.394+0000 7ffa71200640 1 --2- 172.21.15.16:0/2981191255 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ffa6c06fd80 0x7ffa6c15efe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:54.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:54.394+0000 7ffa71200640 1 -- 172.21.15.16:0/2981191255 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffa6c06f920 msgr2=0x7ffa6c06d590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:13:54.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:54.394+0000 7ffa71200640 1 --2- 172.21.15.16:0/2981191255 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffa6c06f920 0x7ffa6c06d590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:54.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:54.394+0000 7ffa71200640 1 -- 172.21.15.16:0/2981191255 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa6c06e750 con 0x7ffa6c145e40 2024-03-31T15:13:54.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:54.394+0000 7ffa71200640 1 --2- 172.21.15.16:0/2981191255 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffa6c145e40 0x7ffa6c15f520 secure :-1 s=READY pgs=1586 cs=0 l=1 rev1=1 crypto rx=0x7ffa64009d90 tx=0x7ffa64019710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:13:54.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:54.398+0000 7ffa68c00640 1 -- 172.21.15.16:0/2981191255 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa640bd020 con 0x7ffa6c145e40 2024-03-31T15:13:54.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:54.398+0000 7ffa68c00640 1 -- 172.21.15.16:0/2981191255 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffa640ad6c0 con 0x7ffa6c145e40 2024-03-31T15:13:54.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:54.398+0000 7ffa68c00640 1 -- 172.21.15.16:0/2981191255 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa640b2040 con 0x7ffa6c145e40 2024-03-31T15:13:54.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:54.398+0000 7ffa73200640 1 -- 172.21.15.16:0/2981191255 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffa6c15aca0 con 0x7ffa6c145e40 2024-03-31T15:13:54.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:54.398+0000 7ffa73200640 1 -- 172.21.15.16:0/2981191255 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffa6c15b260 con 0x7ffa6c145e40 2024-03-31T15:13:54.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:54.398+0000 7ffa73200640 1 -- 172.21.15.16:0/2981191255 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffa6c069950 con 0x7ffa6c145e40 2024-03-31T15:13:54.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:54.398+0000 7ffa68c00640 1 -- 172.21.15.16:0/2981191255 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7ffa640c0020 con 0x7ffa6c145e40 2024-03-31T15:13:54.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:54.398+0000 7ffa68c00640 1 --2- 172.21.15.16:0/2981191255 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7ffa48068380 0x7ffa4806a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:54.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:54.398+0000 7ffa68c00640 1 -- 172.21.15.16:0/2981191255 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 5881+0+0 (secure 0 0 0) 0x7ffa64121b10 con 0x7ffa6c145e40 2024-03-31T15:13:54.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:54.398+0000 7ffa6be00640 1 --2- 172.21.15.16:0/2981191255 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7ffa48068380 0x7ffa4806a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:54.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:54.402+0000 7ffa6be00640 1 --2- 172.21.15.16:0/2981191255 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7ffa48068380 0x7ffa4806a840 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7ffa5c007a90 tx=0x7ffa5c0079e0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:13:54.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:54.402+0000 7ffa68c00640 1 -- 172.21.15.16:0/2981191255 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7ffa640c7030 con 0x7ffa6c145e40 2024-03-31T15:13:54.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:54.602+0000 7ffa73200640 1 -- 172.21.15.16:0/2981191255 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ffa6c0cf3c0 con 0x7ffa6c145e40 2024-03-31T15:13:54.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:54.602+0000 7ffa68c00640 1 -- 172.21.15.16:0/2981191255 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 91 v91) ==== 94+0+5169 (secure 0 0 0) 0x7ffa64016560 con 0x7ffa6c145e40 2024-03-31T15:13:54.596 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:13:54.596 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":91,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7139,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6836/3798928623","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":3798928623},{"type":"v1","addr":"172.21.15.89:6836","nonce":3798928623}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":78},{"gid":7142,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/3074283817","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6835","nonce":3074283817},{"type":"v1","addr":"172.21.15.89:6837","nonce":3074283817}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":78},{"gid":8071,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6834/2110189998","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":2110189998},{"type":"v1","addr":"172.21.15.16:6834","nonce":2110189998}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":78},{"gid":9027,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/2938692325","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6833","nonce":2938692325},{"type":"v1","addr":"172.21.15.16:6835","nonce":2938692325}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":91}],"filesystems":[{"mdsmap":{"epoch":90,"flags":19,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-31T15:12:16.294600+0000","modified":"2024-03-31T15:13:50.823935+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":98,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":7}]} 2024-03-31T15:13:54.596 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 91 2024-03-31T15:13:54.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:54.606+0000 7ffa73200640 1 -- 172.21.15.16:0/2981191255 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7ffa48068380 msgr2=0x7ffa4806a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:13:54.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:54.606+0000 7ffa73200640 1 --2- 172.21.15.16:0/2981191255 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7ffa48068380 0x7ffa4806a840 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7ffa5c007a90 tx=0x7ffa5c0079e0 comp rx=0 tx=0).stop 2024-03-31T15:13:54.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:54.606+0000 7ffa73200640 1 -- 172.21.15.16:0/2981191255 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffa6c145e40 msgr2=0x7ffa6c15f520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:13:54.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:54.606+0000 7ffa73200640 1 --2- 172.21.15.16:0/2981191255 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffa6c145e40 0x7ffa6c15f520 secure :-1 s=READY pgs=1586 cs=0 l=1 rev1=1 crypto rx=0x7ffa64009d90 tx=0x7ffa64019710 comp rx=0 tx=0).stop 2024-03-31T15:13:54.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:54.606+0000 7ffa73200640 1 -- 172.21.15.16:0/2981191255 shutdown_connections 2024-03-31T15:13:54.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:54.606+0000 7ffa73200640 1 --2- 172.21.15.16:0/2981191255 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7ffa48068380 0x7ffa4806a840 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:54.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:54.606+0000 7ffa73200640 1 --2- 172.21.15.16:0/2981191255 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffa6c145e40 0x7ffa6c15f520 unknown :-1 s=CLOSED pgs=1586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:54.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:54.606+0000 7ffa73200640 1 --2- 172.21.15.16:0/2981191255 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ffa6c06fd80 0x7ffa6c15efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:54.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:54.606+0000 7ffa73200640 1 --2- 172.21.15.16:0/2981191255 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffa6c06f920 0x7ffa6c06d590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:54.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:54.606+0000 7ffa73200640 1 -- 172.21.15.16:0/2981191255 >> 172.21.15.16:0/2981191255 conn(0x7ffa6c067330 msgr2=0x7ffa6c0cb860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:13:54.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:54.606+0000 7ffa73200640 1 -- 172.21.15.16:0/2981191255 shutdown_connections 2024-03-31T15:13:54.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:54.606+0000 7ffa73200640 1 -- 172.21.15.16:0/2981191255 wait complete. 2024-03-31T15:13:54.618 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-31T15:13:54.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:54.766+0000 7fef2d400640 1 Processor -- start 2024-03-31T15:13:54.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:54.766+0000 7fef2d400640 1 -- start start 2024-03-31T15:13:54.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:54.766+0000 7fef2d400640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fef2806f920 0x7fef2806fd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:54.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:54.766+0000 7fef2d400640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fef280d2010 0x7fef280d24a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:54.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:54.766+0000 7fef2d400640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fef280d0920 con 0x7fef28145e40 2024-03-31T15:13:54.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:54.766+0000 7fef2d400640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7fef280702c0 con 0x7fef2806f920 2024-03-31T15:13:54.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:54.766+0000 7fef2d400640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7fef280d06d0 con 0x7fef280d2010 2024-03-31T15:13:54.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:54.766+0000 7fef1fe00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fef2806f920 0x7fef2806fd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:54.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:54.766+0000 7fef1fe00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fef2806f920 0x7fef2806fd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50374/0 (socket says 172.21.15.16:50374) 2024-03-31T15:13:54.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:54.766+0000 7fef1fe00640 1 -- 172.21.15.16:0/3918219497 learned_addr learned my addr 172.21.15.16:0/3918219497 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:13:54.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:54.766+0000 7fef27200640 1 --2- 172.21.15.16:0/3918219497 >> v2:172.21.15.89:3300/0 conn(0x7fef280d2010 0x7fef280d24a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:54.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:54.766+0000 7fef1fe00640 1 -- 172.21.15.16:0/3918219497 >> v2:172.21.15.89:3300/0 conn(0x7fef280d2010 msgr2=0x7fef280d24a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:13:54.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:54.766+0000 7fef1fe00640 1 --2- 172.21.15.16:0/3918219497 >> v2:172.21.15.89:3300/0 conn(0x7fef280d2010 0x7fef280d24a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:54.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:54.766+0000 7fef1fe00640 1 -- 172.21.15.16:0/3918219497 >> v1:172.21.15.89:6789/0 conn(0x7fef28145e40 legacy=0x7fef280cc740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:13:54.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:54.766+0000 7fef1fe00640 1 -- 172.21.15.16:0/3918219497 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef280d0c00 con 0x7fef2806f920 2024-03-31T15:13:54.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:54.766+0000 7fef27200640 1 --2- 172.21.15.16:0/3918219497 >> v2:172.21.15.89:3300/0 conn(0x7fef280d2010 0x7fef280d24a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:13:54.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:54.770+0000 7fef1fe00640 1 --2- 172.21.15.16:0/3918219497 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fef2806f920 0x7fef2806fd80 secure :-1 s=READY pgs=1587 cs=0 l=1 rev1=1 crypto rx=0x7fef28160e90 tx=0x7fef1001c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c98eff587005c73 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:13:54.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:54.770+0000 7fef25e00640 1 -- 172.21.15.16:0/3918219497 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef100ab610 con 0x7fef2806f920 2024-03-31T15:13:54.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:54.770+0000 7fef25e00640 1 -- 172.21.15.16:0/3918219497 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fef100b0070 con 0x7fef2806f920 2024-03-31T15:13:54.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:54.770+0000 7fef25e00640 1 -- 172.21.15.16:0/3918219497 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef100ab9d0 con 0x7fef2806f920 2024-03-31T15:13:54.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:54.770+0000 7fef2d400640 1 -- 172.21.15.16:0/3918219497 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fef2806f920 msgr2=0x7fef2806fd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:13:54.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:54.770+0000 7fef2d400640 1 --2- 172.21.15.16:0/3918219497 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fef2806f920 0x7fef2806fd80 secure :-1 s=READY pgs=1587 cs=0 l=1 rev1=1 crypto rx=0x7fef28160e90 tx=0x7fef1001c520 comp rx=0 tx=0).stop 2024-03-31T15:13:54.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:54.770+0000 7fef2d400640 1 -- 172.21.15.16:0/3918219497 shutdown_connections 2024-03-31T15:13:54.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:54.770+0000 7fef2d400640 1 --2- 172.21.15.16:0/3918219497 >> v2:172.21.15.89:3300/0 conn(0x7fef280d2010 0x7fef280d24a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:54.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:54.770+0000 7fef2d400640 1 --2- 172.21.15.16:0/3918219497 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fef2806f920 0x7fef2806fd80 unknown :-1 s=CLOSED pgs=1587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:54.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:54.770+0000 7fef2d400640 1 -- 172.21.15.16:0/3918219497 >> 172.21.15.16:0/3918219497 conn(0x7fef28067330 msgr2=0x7fef28067740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:13:54.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:54.770+0000 7fef2d400640 1 -- 172.21.15.16:0/3918219497 shutdown_connections 2024-03-31T15:13:54.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:54.770+0000 7fef2d400640 1 -- 172.21.15.16:0/3918219497 wait complete. 2024-03-31T15:13:54.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:54.774+0000 7fef2d400640 1 Processor -- start 2024-03-31T15:13:54.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:54.774+0000 7fef2d400640 1 -- start start 2024-03-31T15:13:54.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:54.774+0000 7fef2d400640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fef2806f920 0x7fef28144100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:54.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:54.774+0000 7fef2d400640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fef280d2010 0x7fef2815d840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:54.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:54.774+0000 7fef2d400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fef28145e40 0x7fef281574c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:54.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:54.774+0000 7fef2d400640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fef280d0ee0 con 0x7fef28145e40 2024-03-31T15:13:54.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:54.774+0000 7fef2d400640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fef280ce9e0 con 0x7fef280d2010 2024-03-31T15:13:54.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:54.774+0000 7fef2d400640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fef280d2df0 con 0x7fef2806f920 2024-03-31T15:13:54.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:54.774+0000 7fef27200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fef28145e40 0x7fef281574c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:54.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:54.774+0000 7fef27200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fef28145e40 0x7fef281574c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50390/0 (socket says 172.21.15.16:50390) 2024-03-31T15:13:54.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:54.774+0000 7fef27200640 1 -- 172.21.15.16:0/418179627 learned_addr learned my addr 172.21.15.16:0/418179627 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:13:54.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:54.774+0000 7fef26800640 1 --2- 172.21.15.16:0/418179627 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fef2806f920 0x7fef28144100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:54.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:54.774+0000 7fef1fe00640 1 --2- 172.21.15.16:0/418179627 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fef280d2010 0x7fef2815d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:54.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:54.774+0000 7fef27200640 1 -- 172.21.15.16:0/418179627 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fef2806f920 msgr2=0x7fef28144100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:13:54.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:54.774+0000 7fef27200640 1 --2- 172.21.15.16:0/418179627 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fef2806f920 0x7fef28144100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:54.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:54.774+0000 7fef27200640 1 -- 172.21.15.16:0/418179627 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fef280d2010 msgr2=0x7fef2815d840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:13:54.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:54.774+0000 7fef27200640 1 --2- 172.21.15.16:0/418179627 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fef280d2010 0x7fef2815d840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:54.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:54.774+0000 7fef27200640 1 -- 172.21.15.16:0/418179627 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef2806d390 con 0x7fef28145e40 2024-03-31T15:13:54.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:54.774+0000 7fef27200640 1 --2- 172.21.15.16:0/418179627 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fef28145e40 0x7fef281574c0 secure :-1 s=READY pgs=1588 cs=0 l=1 rev1=1 crypto rx=0x7fef2000cf20 tx=0x7fef2000a7d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:13:54.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:54.774+0000 7fef1ea00640 1 -- 172.21.15.16:0/418179627 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef2001a070 con 0x7fef28145e40 2024-03-31T15:13:54.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:54.774+0000 7fef1ea00640 1 -- 172.21.15.16:0/418179627 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fef2000d840 con 0x7fef28145e40 2024-03-31T15:13:54.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:54.774+0000 7fef1ea00640 1 -- 172.21.15.16:0/418179627 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef2000dc30 con 0x7fef28145e40 2024-03-31T15:13:54.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:54.774+0000 7fef2d400640 1 -- 172.21.15.16:0/418179627 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fef2806d670 con 0x7fef28145e40 2024-03-31T15:13:54.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:54.774+0000 7fef2d400640 1 -- 172.21.15.16:0/418179627 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fef2806dc30 con 0x7fef28145e40 2024-03-31T15:13:54.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:54.778+0000 7fef2d400640 1 -- 172.21.15.16:0/418179627 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fef28069950 con 0x7fef28145e40 2024-03-31T15:13:54.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:54.778+0000 7fef1ea00640 1 -- 172.21.15.16:0/418179627 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fef2001d070 con 0x7fef28145e40 2024-03-31T15:13:54.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:54.778+0000 7fef1ea00640 1 --2- 172.21.15.16:0/418179627 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fef04068380 0x7fef0406a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:54.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:54.778+0000 7fef1ea00640 1 -- 172.21.15.16:0/418179627 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 5881+0+0 (secure 0 0 0) 0x7fef20087300 con 0x7fef28145e40 2024-03-31T15:13:54.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:54.778+0000 7fef26800640 1 --2- 172.21.15.16:0/418179627 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fef04068380 0x7fef0406a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:54.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:54.778+0000 7fef26800640 1 --2- 172.21.15.16:0/418179627 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fef04068380 0x7fef0406a840 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7fef18001770 tx=0x7fef18009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:13:54.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:54.782+0000 7fef1ea00640 1 -- 172.21.15.16:0/418179627 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fef20057130 con 0x7fef28145e40 2024-03-31T15:13:54.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:54.946+0000 7fef2d400640 1 -- 172.21.15.16:0/418179627 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fef280cf3c0 con 0x7fef28145e40 2024-03-31T15:13:54.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:54.950+0000 7fef1ea00640 1 -- 172.21.15.16:0/418179627 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v98) ==== 74+0+16791 (secure 0 0 0) 0x7fef2005b430 con 0x7fef28145e40 2024-03-31T15:13:54.941 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:13:54.942 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":98,"fsid":"286eb5da-fb47-4550-b290-d9519c13b9a5","created":"2024-03-31T15:03:11.438952+0000","modified":"2024-03-31T15:13:50.823773+0000","last_up_change":"2024-03-31T15:03:22.885535+0000","last_in_change":"2024-03-31T15:03:13.941667+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-31T15:03:22.550787+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2024-03-31T15:12:14.671225+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"95","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2024-03-31T15:12:15.242196+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"96","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"92a66450-ef19-40d8-ac3d-d760b23b0bda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6801","nonce":932712244}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6803","nonce":932712244}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6807","nonce":932712244}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6805","nonce":932712244}]},"public_addr":"172.21.15.16:6801/932712244","cluster_addr":"172.21.15.16:6803/932712244","heartbeat_back_addr":"172.21.15.16:6807/932712244","heartbeat_front_addr":"172.21.15.16:6805/932712244","state":["exists","up"]},{"osd":1,"uuid":"da9d1f2b-1968-4959-b843-faa231988e92","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6810","nonce":1398737533}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6811","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6813","nonce":1398737533}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6821","nonce":1398737533}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6815","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6816","nonce":1398737533}]},"public_addr":"172.21.15.16:6810/1398737533","cluster_addr":"172.21.15.16:6813/1398737533","heartbeat_back_addr":"172.21.15.16:6821/1398737533","heartbeat_front_addr":"172.21.15.16:6816/1398737533","state":["exists","up"]},{"osd":2,"uuid":"01a73662-9a21-4b1f-bcbb-42b9636a4638","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6819","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6822","nonce":3018526646}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6825","nonce":3018526646}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6831","nonce":3018526646}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6827","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6829","nonce":3018526646}]},"public_addr":"172.21.15.16:6822/3018526646","cluster_addr":"172.21.15.16:6825/3018526646","heartbeat_back_addr":"172.21.15.16:6831/3018526646","heartbeat_front_addr":"172.21.15.16:6829/3018526646","state":["exists","up"]},{"osd":3,"uuid":"c7771fd3-e18e-4500-ac38-23cf10ebcd75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6809","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6812","nonce":3360352974}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6817","nonce":3360352974}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6828","nonce":3360352974}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6823","nonce":3360352974}]},"public_addr":"172.21.15.16:6812/3360352974","cluster_addr":"172.21.15.16:6817/3360352974","heartbeat_back_addr":"172.21.15.16:6828/3360352974","heartbeat_front_addr":"172.21.15.16:6823/3360352974","state":["exists","up"]},{"osd":4,"uuid":"88260730-1482-4fbd-a40f-cb1124e9df4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6801","nonce":1856844856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6803","nonce":1856844856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6807","nonce":1856844856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6805","nonce":1856844856}]},"public_addr":"172.21.15.89:6801/1856844856","cluster_addr":"172.21.15.89:6803/1856844856","heartbeat_back_addr":"172.21.15.89:6807/1856844856","heartbeat_front_addr":"172.21.15.89:6805/1856844856","state":["exists","up"]},{"osd":5,"uuid":"4cb2c00c-fae0-4dd8-9d27-ec4f0093ccf7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6814","nonce":50395527}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6817","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6820","nonce":50395527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6831","nonce":50395527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6823","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6826","nonce":50395527}]},"public_addr":"172.21.15.89:6814/50395527","cluster_addr":"172.21.15.89:6820/50395527","heartbeat_back_addr":"172.21.15.89:6831/50395527","heartbeat_front_addr":"172.21.15.89:6826/50395527","state":["exists","up"]},{"osd":6,"uuid":"81ce0c19-2554-4cae-acbe-66a05e9e9b9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6809","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6812","nonce":3470471752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6815","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6818","nonce":3470471752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6827","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6829","nonce":3470471752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6821","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6824","nonce":3470471752}]},"public_addr":"172.21.15.89:6812/3470471752","cluster_addr":"172.21.15.89:6818/3470471752","heartbeat_back_addr":"172.21.15.89:6829/3470471752","heartbeat_front_addr":"172.21.15.89:6824/3470471752","state":["exists","up"]},{"osd":7,"uuid":"85e4313b-d37d-442e-8042-153bf20a847d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6811","nonce":1796778630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6813","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6816","nonce":1796778630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6825","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6828","nonce":1796778630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6819","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6822","nonce":1796778630}]},"public_addr":"172.21.15.89:6811/1796778630","cluster_addr":"172.21.15.89:6816/1796778630","heartbeat_back_addr":"172.21.15.89:6828/1796778630","heartbeat_front_addr":"172.21.15.89:6822/1796778630","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.439088+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.529970+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.680512+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:21.044161+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.582111+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.536558+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.409522+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.590229+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:6835/567808781":"2024-04-01T15:13:50.823751+0000","172.21.15.16:6833/567808781":"2024-04-01T15:13:50.823751+0000","172.21.15.16:6835/490717480":"2024-04-01T15:13:14.080258+0000","172.21.15.16:6833/490717480":"2024-04-01T15:13:14.080258+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-31T15:13:54.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:54.950+0000 7fef2d400640 1 -- 172.21.15.16:0/418179627 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fef04068380 msgr2=0x7fef0406a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:13:54.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:54.950+0000 7fef2d400640 1 --2- 172.21.15.16:0/418179627 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fef04068380 0x7fef0406a840 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7fef18001770 tx=0x7fef18009040 comp rx=0 tx=0).stop 2024-03-31T15:13:54.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:54.950+0000 7fef2d400640 1 -- 172.21.15.16:0/418179627 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fef28145e40 msgr2=0x7fef281574c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:13:54.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:54.950+0000 7fef2d400640 1 --2- 172.21.15.16:0/418179627 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fef28145e40 0x7fef281574c0 secure :-1 s=READY pgs=1588 cs=0 l=1 rev1=1 crypto rx=0x7fef2000cf20 tx=0x7fef2000a7d0 comp rx=0 tx=0).stop 2024-03-31T15:13:54.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:54.950+0000 7fef2d400640 1 -- 172.21.15.16:0/418179627 shutdown_connections 2024-03-31T15:13:54.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:54.950+0000 7fef2d400640 1 --2- 172.21.15.16:0/418179627 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fef04068380 0x7fef0406a840 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:54.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:54.950+0000 7fef2d400640 1 --2- 172.21.15.16:0/418179627 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fef28145e40 0x7fef281574c0 unknown :-1 s=CLOSED pgs=1588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:54.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:54.950+0000 7fef2d400640 1 --2- 172.21.15.16:0/418179627 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fef280d2010 0x7fef2815d840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:54.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:54.950+0000 7fef2d400640 1 --2- 172.21.15.16:0/418179627 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fef2806f920 0x7fef28144100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:54.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:54.950+0000 7fef2d400640 1 -- 172.21.15.16:0/418179627 >> 172.21.15.16:0/418179627 conn(0x7fef28067330 msgr2=0x7fef2815f3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:13:54.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:54.950+0000 7fef2d400640 1 -- 172.21.15.16:0/418179627 shutdown_connections 2024-03-31T15:13:54.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:54.950+0000 7fef2d400640 1 -- 172.21.15.16:0/418179627 wait complete. 2024-03-31T15:13:54.962 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-31T15:13:55.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:55.110+0000 7fdf74c00640 1 Processor -- start 2024-03-31T15:13:55.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:55.110+0000 7fdf74c00640 1 -- start start 2024-03-31T15:13:55.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:55.114+0000 7fdf74c00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fdf70145d10 0x7fdf700cf790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:55.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:55.114+0000 7fdf74c00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fdf700728b0 0x7fdf700b1cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:55.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:55.114+0000 7fdf74c00640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdf70160e70 con 0x7fdf700b2200 2024-03-31T15:13:55.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:55.114+0000 7fdf74c00640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7fdf701602f0 con 0x7fdf70145d10 2024-03-31T15:13:55.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:55.114+0000 7fdf74c00640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7fdf701605d0 con 0x7fdf700728b0 2024-03-31T15:13:55.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:55.114+0000 7fdf6de00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fdf70145d10 0x7fdf700cf790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:55.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:55.114+0000 7fdf6de00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fdf70145d10 0x7fdf700cf790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50396/0 (socket says 172.21.15.16:50396) 2024-03-31T15:13:55.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:55.114+0000 7fdf6de00640 1 -- 172.21.15.16:0/2695568890 learned_addr learned my addr 172.21.15.16:0/2695568890 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:13:55.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:55.114+0000 7fdf65400640 1 --2- 172.21.15.16:0/2695568890 >> v2:172.21.15.89:3300/0 conn(0x7fdf700728b0 0x7fdf700b1cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:55.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:55.114+0000 7fdf6de00640 1 -- 172.21.15.16:0/2695568890 >> v2:172.21.15.89:3300/0 conn(0x7fdf700728b0 msgr2=0x7fdf700b1cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:13:55.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:55.114+0000 7fdf6de00640 1 --2- 172.21.15.16:0/2695568890 >> v2:172.21.15.89:3300/0 conn(0x7fdf700728b0 0x7fdf700b1cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:55.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:55.114+0000 7fdf6de00640 1 -- 172.21.15.16:0/2695568890 >> v1:172.21.15.89:6789/0 conn(0x7fdf700b2200 legacy=0x7fdf7015cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:13:55.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:55.114+0000 7fdf6de00640 1 -- 172.21.15.16:0/2695568890 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf700cfcd0 con 0x7fdf70145d10 2024-03-31T15:13:55.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:55.118+0000 7fdf6de00640 1 --2- 172.21.15.16:0/2695568890 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdf70145d10 0x7fdf700cf790 secure :-1 s=READY pgs=1589 cs=0 l=1 rev1=1 crypto rx=0x7fdf700b2980 tx=0x7fdf6001c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a7871afe8ffa255 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:13:55.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:55.118+0000 7fdf6d400640 1 -- 172.21.15.16:0/2695568890 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf600a56c0 con 0x7fdf70145d10 2024-03-31T15:13:55.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:55.118+0000 7fdf6d400640 1 -- 172.21.15.16:0/2695568890 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdf600a5860 con 0x7fdf70145d10 2024-03-31T15:13:55.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:55.118+0000 7fdf6d400640 1 -- 172.21.15.16:0/2695568890 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf600a5c30 con 0x7fdf70145d10 2024-03-31T15:13:55.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:55.118+0000 7fdf74c00640 1 -- 172.21.15.16:0/2695568890 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdf70145d10 msgr2=0x7fdf700cf790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:13:55.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:55.118+0000 7fdf74c00640 1 --2- 172.21.15.16:0/2695568890 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdf70145d10 0x7fdf700cf790 secure :-1 s=READY pgs=1589 cs=0 l=1 rev1=1 crypto rx=0x7fdf700b2980 tx=0x7fdf6001c9f0 comp rx=0 tx=0).stop 2024-03-31T15:13:55.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:55.118+0000 7fdf74c00640 1 -- 172.21.15.16:0/2695568890 shutdown_connections 2024-03-31T15:13:55.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:55.118+0000 7fdf74c00640 1 --2- 172.21.15.16:0/2695568890 >> v2:172.21.15.89:3300/0 conn(0x7fdf700728b0 0x7fdf700b1cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:55.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:55.118+0000 7fdf74c00640 1 --2- 172.21.15.16:0/2695568890 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdf70145d10 0x7fdf700cf790 unknown :-1 s=CLOSED pgs=1589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:55.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:55.118+0000 7fdf74c00640 1 -- 172.21.15.16:0/2695568890 >> 172.21.15.16:0/2695568890 conn(0x7fdf7006b620 msgr2=0x7fdf7006ba30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:13:55.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:55.118+0000 7fdf74c00640 1 -- 172.21.15.16:0/2695568890 shutdown_connections 2024-03-31T15:13:55.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:55.118+0000 7fdf74c00640 1 -- 172.21.15.16:0/2695568890 wait complete. 2024-03-31T15:13:55.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:55.118+0000 7fdf74c00640 1 Processor -- start 2024-03-31T15:13:55.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:55.118+0000 7fdf74c00640 1 -- start start 2024-03-31T15:13:55.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:55.122+0000 7fdf74c00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fdf700728b0 0x7fdf70159800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:55.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:55.122+0000 7fdf74c00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdf700b2200 0x7fdf7015b7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:55.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:55.122+0000 7fdf74c00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdf70145d10 0x7fdf70139b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:55.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:55.122+0000 7fdf74c00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fdf7015d870 con 0x7fdf70145d10 2024-03-31T15:13:55.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:55.122+0000 7fdf74c00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fdf700b2610 con 0x7fdf700b2200 2024-03-31T15:13:55.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:55.122+0000 7fdf74c00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fdf700cf350 con 0x7fdf700728b0 2024-03-31T15:13:55.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:55.122+0000 7fdf6e800640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdf70145d10 0x7fdf70139b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:55.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:55.122+0000 7fdf6de00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fdf700728b0 0x7fdf70159800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:55.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:55.122+0000 7fdf6e800640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdf70145d10 0x7fdf70139b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50404/0 (socket says 172.21.15.16:50404) 2024-03-31T15:13:55.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:55.122+0000 7fdf6e800640 1 -- 172.21.15.16:0/677959604 learned_addr learned my addr 172.21.15.16:0/677959604 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:13:55.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:55.122+0000 7fdf65400640 1 --2- 172.21.15.16:0/677959604 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdf700b2200 0x7fdf7015b7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:55.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:55.122+0000 7fdf6e800640 1 -- 172.21.15.16:0/677959604 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fdf700728b0 msgr2=0x7fdf70159800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:13:55.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:55.122+0000 7fdf6e800640 1 --2- 172.21.15.16:0/677959604 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fdf700728b0 0x7fdf70159800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:55.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:55.122+0000 7fdf6e800640 1 -- 172.21.15.16:0/677959604 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdf700b2200 msgr2=0x7fdf7015b7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:13:55.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:55.122+0000 7fdf6e800640 1 --2- 172.21.15.16:0/677959604 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdf700b2200 0x7fdf7015b7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:55.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:55.122+0000 7fdf6e800640 1 -- 172.21.15.16:0/677959604 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf7013ab30 con 0x7fdf70145d10 2024-03-31T15:13:55.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:55.122+0000 7fdf6e800640 1 --2- 172.21.15.16:0/677959604 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdf70145d10 0x7fdf70139b10 secure :-1 s=READY pgs=1590 cs=0 l=1 rev1=1 crypto rx=0x7fdf6800c430 tx=0x7fdf6800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:13:55.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:55.122+0000 7fdf66a00640 1 -- 172.21.15.16:0/677959604 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf68005220 con 0x7fdf70145d10 2024-03-31T15:13:55.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:55.122+0000 7fdf66a00640 1 -- 172.21.15.16:0/677959604 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdf680053c0 con 0x7fdf70145d10 2024-03-31T15:13:55.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:55.122+0000 7fdf66a00640 1 -- 172.21.15.16:0/677959604 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf68007270 con 0x7fdf70145d10 2024-03-31T15:13:55.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:55.122+0000 7fdf74c00640 1 -- 172.21.15.16:0/677959604 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdf7013ae10 con 0x7fdf70145d10 2024-03-31T15:13:55.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:55.122+0000 7fdf74c00640 1 -- 172.21.15.16:0/677959604 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdf7013b3d0 con 0x7fdf70145d10 2024-03-31T15:13:55.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:55.122+0000 7fdf74c00640 1 -- 172.21.15.16:0/677959604 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdf70069950 con 0x7fdf70145d10 2024-03-31T15:13:55.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:55.122+0000 7fdf66a00640 1 -- 172.21.15.16:0/677959604 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fdf68019020 con 0x7fdf70145d10 2024-03-31T15:13:55.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:55.126+0000 7fdf66a00640 1 --2- 172.21.15.16:0/677959604 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fdf4c068380 0x7fdf4c06a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:55.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:55.126+0000 7fdf66a00640 1 -- 172.21.15.16:0/677959604 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 5881+0+0 (secure 0 0 0) 0x7fdf68085d50 con 0x7fdf70145d10 2024-03-31T15:13:55.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:55.126+0000 7fdf6de00640 1 --2- 172.21.15.16:0/677959604 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fdf4c068380 0x7fdf4c06a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:55.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:55.126+0000 7fdf6de00640 1 --2- 172.21.15.16:0/677959604 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fdf4c068380 0x7fdf4c06a840 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7fdf600ae000 tx=0x7fdf600a7070 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:13:55.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:55.130+0000 7fdf66a00640 1 -- 172.21.15.16:0/677959604 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fdf68055b70 con 0x7fdf70145d10 2024-03-31T15:13:55.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:55.326+0000 7fdf74c00640 1 -- 172.21.15.16:0/677959604 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7fdf7015e4f0 con 0x7fdf70145d10 2024-03-31T15:13:56.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:56.186+0000 7fdf66a00640 1 -- 172.21.15.16:0/677959604 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v92) ==== 161+0+0 (secure 0 0 0) 0x7fdf68059e70 con 0x7fdf70145d10 2024-03-31T15:13:56.180 INFO:teuthology.orchestra.run.smithi016.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-31T15:13:56.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:56.190+0000 7fdf74c00640 1 -- 172.21.15.16:0/677959604 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fdf4c068380 msgr2=0x7fdf4c06a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:13:56.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:56.190+0000 7fdf74c00640 1 --2- 172.21.15.16:0/677959604 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fdf4c068380 0x7fdf4c06a840 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7fdf600ae000 tx=0x7fdf600a7070 comp rx=0 tx=0).stop 2024-03-31T15:13:56.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:56.190+0000 7fdf74c00640 1 -- 172.21.15.16:0/677959604 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdf70145d10 msgr2=0x7fdf70139b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:13:56.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:56.190+0000 7fdf74c00640 1 --2- 172.21.15.16:0/677959604 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdf70145d10 0x7fdf70139b10 secure :-1 s=READY pgs=1590 cs=0 l=1 rev1=1 crypto rx=0x7fdf6800c430 tx=0x7fdf6800c8f0 comp rx=0 tx=0).stop 2024-03-31T15:13:56.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:56.190+0000 7fdf74c00640 1 -- 172.21.15.16:0/677959604 shutdown_connections 2024-03-31T15:13:56.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:56.190+0000 7fdf74c00640 1 --2- 172.21.15.16:0/677959604 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fdf4c068380 0x7fdf4c06a840 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:56.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:56.190+0000 7fdf74c00640 1 --2- 172.21.15.16:0/677959604 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdf70145d10 0x7fdf70139b10 unknown :-1 s=CLOSED pgs=1590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:56.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:56.190+0000 7fdf74c00640 1 --2- 172.21.15.16:0/677959604 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdf700b2200 0x7fdf7015b7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:56.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:56.190+0000 7fdf74c00640 1 --2- 172.21.15.16:0/677959604 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fdf700728b0 0x7fdf70159800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:56.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:56.190+0000 7fdf74c00640 1 -- 172.21.15.16:0/677959604 >> 172.21.15.16:0/677959604 conn(0x7fdf7006b620 msgr2=0x7fdf701666c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:13:56.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:56.190+0000 7fdf74c00640 1 -- 172.21.15.16:0/677959604 shutdown_connections 2024-03-31T15:13:56.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:56.190+0000 7fdf74c00640 1 -- 172.21.15.16:0/677959604 wait complete. 2024-03-31T15:13:56.206 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-31T15:13:56.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:56.358+0000 7f23d0600640 1 Processor -- start 2024-03-31T15:13:56.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:56.358+0000 7f23d0600640 1 -- start start 2024-03-31T15:13:56.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:56.362+0000 7f23d0600640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f23c813e3d0 0x7f23c80cc9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:56.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:56.362+0000 7f23d0600640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f23c806dbe0 0x7f23c80ccf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:56.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:56.362+0000 7f23d0600640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f23c815fbc0 con 0x7f23c813dff0 2024-03-31T15:13:56.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:56.362+0000 7f23d0600640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f23c80cd450 con 0x7f23c806dbe0 2024-03-31T15:13:56.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:56.362+0000 7f23d0600640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f23c815f730 con 0x7f23c813e3d0 2024-03-31T15:13:56.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:56.362+0000 7f23cd200640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f23c813e3d0 0x7f23c80cc9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:56.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:56.362+0000 7f23ce600640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f23c806dbe0 0x7f23c80ccf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:56.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:56.362+0000 7f23cdc00640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f23c813dff0 0x7f23c806d4d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.16:43944/0 (socket says 172.21.15.16:43944) 2024-03-31T15:13:56.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:56.362+0000 7f23cdc00640 1 -- 172.21.15.16:0/1907767646 learned_addr learned my addr 172.21.15.16:0/1907767646 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:13:56.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:56.362+0000 7f23ce600640 1 -- 172.21.15.16:0/1907767646 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f23c813e3d0 msgr2=0x7f23c80cc9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:13:56.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:56.362+0000 7f23ce600640 1 --2- 172.21.15.16:0/1907767646 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f23c813e3d0 0x7f23c80cc9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:56.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:56.362+0000 7f23ce600640 1 -- 172.21.15.16:0/1907767646 >> v1:172.21.15.89:6789/0 conn(0x7f23c813dff0 legacy=0x7f23c806d4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:13:56.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:56.362+0000 7f23ce600640 1 -- 172.21.15.16:0/1907767646 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f23c815ee00 con 0x7f23c806dbe0 2024-03-31T15:13:56.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:56.366+0000 7f23ce600640 1 --2- 172.21.15.16:0/1907767646 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f23c806dbe0 0x7f23c80ccf10 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f23c0009d90 tx=0x7f23c001c8e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=76da4af50f76cbfa server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:13:56.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:56.366+0000 7f23c7e00640 1 -- 172.21.15.16:0/1907767646 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23c00b2070 con 0x7f23c806dbe0 2024-03-31T15:13:56.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:56.366+0000 7f23c7e00640 1 -- 172.21.15.16:0/1907767646 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f23c00a5790 con 0x7f23c806dbe0 2024-03-31T15:13:56.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:56.366+0000 7f23c7e00640 1 -- 172.21.15.16:0/1907767646 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23c00a5b60 con 0x7f23c806dbe0 2024-03-31T15:13:56.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:56.366+0000 7f23d0600640 1 -- 172.21.15.16:0/1907767646 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f23c806dbe0 msgr2=0x7f23c80ccf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:13:56.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:56.366+0000 7f23d0600640 1 --2- 172.21.15.16:0/1907767646 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f23c806dbe0 0x7f23c80ccf10 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f23c0009d90 tx=0x7f23c001c8e0 comp rx=0 tx=0).stop 2024-03-31T15:13:56.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:56.366+0000 7f23d0600640 1 -- 172.21.15.16:0/1907767646 shutdown_connections 2024-03-31T15:13:56.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:56.366+0000 7f23d0600640 1 --2- 172.21.15.16:0/1907767646 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f23c806dbe0 0x7f23c80ccf10 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:56.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:56.366+0000 7f23d0600640 1 --2- 172.21.15.16:0/1907767646 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f23c813e3d0 0x7f23c80cc9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:56.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:56.366+0000 7f23d0600640 1 -- 172.21.15.16:0/1907767646 >> 172.21.15.16:0/1907767646 conn(0x7f23c806ee40 msgr2=0x7f23c806b620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:13:56.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:56.366+0000 7f23d0600640 1 -- 172.21.15.16:0/1907767646 shutdown_connections 2024-03-31T15:13:56.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:56.366+0000 7f23d0600640 1 -- 172.21.15.16:0/1907767646 wait complete. 2024-03-31T15:13:56.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:56.366+0000 7f23d0600640 1 Processor -- start 2024-03-31T15:13:56.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:56.366+0000 7f23d0600640 1 -- start start 2024-03-31T15:13:56.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:56.366+0000 7f23d0600640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f23c806dbe0 0x7f23c8138380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:56.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:56.366+0000 7f23d0600640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f23c813dff0 0x7f23c815c160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:56.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:56.366+0000 7f23d0600640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f23c813e3d0 0x7f23c81598c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:56.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:56.366+0000 7f23d0600640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f23c80720f0 con 0x7f23c806dbe0 2024-03-31T15:13:56.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:56.366+0000 7f23d0600640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f23c8071f70 con 0x7f23c813e3d0 2024-03-31T15:13:56.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:56.366+0000 7f23d0600640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f23c8072270 con 0x7f23c813dff0 2024-03-31T15:13:56.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:56.370+0000 7f23cdc00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f23c806dbe0 0x7f23c8138380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:56.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:56.370+0000 7f23cdc00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f23c806dbe0 0x7f23c8138380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50414/0 (socket says 172.21.15.16:50414) 2024-03-31T15:13:56.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:56.370+0000 7f23cdc00640 1 -- 172.21.15.16:0/2781975219 learned_addr learned my addr 172.21.15.16:0/2781975219 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:13:56.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:56.370+0000 7f23cd200640 1 --2- 172.21.15.16:0/2781975219 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f23c813dff0 0x7f23c815c160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:56.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:56.370+0000 7f23ce600640 1 --2- 172.21.15.16:0/2781975219 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f23c813e3d0 0x7f23c81598c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:56.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:56.370+0000 7f23cdc00640 1 -- 172.21.15.16:0/2781975219 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f23c813dff0 msgr2=0x7f23c815c160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:13:56.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:56.370+0000 7f23cdc00640 1 --2- 172.21.15.16:0/2781975219 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f23c813dff0 0x7f23c815c160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:56.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:56.370+0000 7f23cdc00640 1 -- 172.21.15.16:0/2781975219 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f23c813e3d0 msgr2=0x7f23c81598c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:13:56.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:56.370+0000 7f23cdc00640 1 --2- 172.21.15.16:0/2781975219 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f23c813e3d0 0x7f23c81598c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:56.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:56.370+0000 7f23cdc00640 1 -- 172.21.15.16:0/2781975219 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f23c815c9b0 con 0x7f23c806dbe0 2024-03-31T15:13:56.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:56.370+0000 7f23cdc00640 1 --2- 172.21.15.16:0/2781975219 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f23c806dbe0 0x7f23c8138380 secure :-1 s=READY pgs=1591 cs=0 l=1 rev1=1 crypto rx=0x7f23bc00c420 tx=0x7f23bc00c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:13:56.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:56.370+0000 7f23c5600640 1 -- 172.21.15.16:0/2781975219 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23bc017070 con 0x7f23c806dbe0 2024-03-31T15:13:56.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:56.370+0000 7f23c5600640 1 -- 172.21.15.16:0/2781975219 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f23bc005cf0 con 0x7f23c806dbe0 2024-03-31T15:13:56.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:56.370+0000 7f23d0600640 1 -- 172.21.15.16:0/2781975219 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f23c815cc90 con 0x7f23c806dbe0 2024-03-31T15:13:56.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:56.370+0000 7f23c5600640 1 -- 172.21.15.16:0/2781975219 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23bc0072f0 con 0x7f23c806dbe0 2024-03-31T15:13:56.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:56.370+0000 7f23d0600640 1 -- 172.21.15.16:0/2781975219 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f23c815d250 con 0x7f23c806dbe0 2024-03-31T15:13:56.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:56.370+0000 7f23d0600640 1 -- 172.21.15.16:0/2781975219 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f23c815d490 con 0x7f23c806dbe0 2024-03-31T15:13:56.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:56.370+0000 7f23c5600640 1 -- 172.21.15.16:0/2781975219 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f23bc01a070 con 0x7f23c806dbe0 2024-03-31T15:13:56.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:56.374+0000 7f23c5600640 1 --2- 172.21.15.16:0/2781975219 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f23a8068380 0x7f23a806a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:56.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:56.374+0000 7f23c5600640 1 -- 172.21.15.16:0/2781975219 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 5881+0+0 (secure 0 0 0) 0x7f23bc085de0 con 0x7f23c806dbe0 2024-03-31T15:13:56.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:56.374+0000 7f23cd200640 1 --2- 172.21.15.16:0/2781975219 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f23a8068380 0x7f23a806a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:56.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:56.374+0000 7f23cd200640 1 --2- 172.21.15.16:0/2781975219 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f23a8068380 0x7f23a806a840 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f23c8144e20 tx=0x7f23b4009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:13:56.369 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:56.378+0000 7f23c5600640 1 -- 172.21.15.16:0/2781975219 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f23c815d490 con 0x7f23c806dbe0 2024-03-31T15:13:56.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:56.574+0000 7f23d0600640 1 -- 172.21.15.16:0/2781975219 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7f23c806e4e0 con 0x7f23c806dbe0 2024-03-31T15:13:57.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:57.190+0000 7f23c5600640 1 -- 172.21.15.16:0/2781975219 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v93) ==== 104+0+0 (secure 0 0 0) 0x7f23bc055c90 con 0x7f23c806dbe0 2024-03-31T15:13:57.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:57.194+0000 7f23d0600640 1 -- 172.21.15.16:0/2781975219 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f23a8068380 msgr2=0x7f23a806a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:13:57.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:57.194+0000 7f23d0600640 1 --2- 172.21.15.16:0/2781975219 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f23a8068380 0x7f23a806a840 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f23c8144e20 tx=0x7f23b4009190 comp rx=0 tx=0).stop 2024-03-31T15:13:57.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:57.194+0000 7f23d0600640 1 -- 172.21.15.16:0/2781975219 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f23c806dbe0 msgr2=0x7f23c8138380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:13:57.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:57.194+0000 7f23d0600640 1 --2- 172.21.15.16:0/2781975219 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f23c806dbe0 0x7f23c8138380 secure :-1 s=READY pgs=1591 cs=0 l=1 rev1=1 crypto rx=0x7f23bc00c420 tx=0x7f23bc00c8e0 comp rx=0 tx=0).stop 2024-03-31T15:13:57.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:57.194+0000 7f23d0600640 1 -- 172.21.15.16:0/2781975219 shutdown_connections 2024-03-31T15:13:57.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:57.194+0000 7f23d0600640 1 --2- 172.21.15.16:0/2781975219 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f23a8068380 0x7f23a806a840 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:57.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:57.194+0000 7f23d0600640 1 --2- 172.21.15.16:0/2781975219 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f23c813e3d0 0x7f23c81598c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:57.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:57.194+0000 7f23d0600640 1 --2- 172.21.15.16:0/2781975219 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f23c813dff0 0x7f23c815c160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:57.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:57.194+0000 7f23d0600640 1 --2- 172.21.15.16:0/2781975219 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f23c806dbe0 0x7f23c8138380 unknown :-1 s=CLOSED pgs=1591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:57.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:57.194+0000 7f23d0600640 1 -- 172.21.15.16:0/2781975219 >> 172.21.15.16:0/2781975219 conn(0x7f23c806ee40 msgr2=0x7f23c815a1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:13:57.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:57.194+0000 7f23d0600640 1 -- 172.21.15.16:0/2781975219 shutdown_connections 2024-03-31T15:13:57.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:57.194+0000 7f23d0600640 1 -- 172.21.15.16:0/2781975219 wait complete. 2024-03-31T15:13:57.210 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-31T15:13:57.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:57.366+0000 7fc44e800640 1 Processor -- start 2024-03-31T15:13:57.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:57.366+0000 7fc44e800640 1 -- start start 2024-03-31T15:13:57.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:57.366+0000 7fc44e800640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fc44815a6a0 0x7fc44815aa80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:57.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:57.366+0000 7fc44e800640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fc4480b3e70 0x7fc448071e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:57.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:57.366+0000 7fc44e800640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc4480b4370 con 0x7fc4480b2f50 2024-03-31T15:13:57.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:57.366+0000 7fc44e800640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7fc4480b4ef0 con 0x7fc4480b3e70 2024-03-31T15:13:57.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:57.366+0000 7fc44e800640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7fc4480b4650 con 0x7fc44815a6a0 2024-03-31T15:13:57.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:57.370+0000 7fc43ea00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fc4480b3e70 0x7fc448071e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:57.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:57.370+0000 7fc447e00640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fc4480b2f50 0x7fc44815e4a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:57166/0 (socket says 172.21.15.16:57166) 2024-03-31T15:13:57.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:57.370+0000 7fc447e00640 1 -- 172.21.15.16:0/4216544873 learned_addr learned my addr 172.21.15.16:0/4216544873 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:13:57.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:57.370+0000 7fc447400640 1 --2- 172.21.15.16:0/4216544873 >> v2:172.21.15.89:3300/0 conn(0x7fc44815a6a0 0x7fc44815aa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:57.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:57.370+0000 7fc43ea00640 1 -- 172.21.15.16:0/4216544873 >> v2:172.21.15.89:3300/0 conn(0x7fc44815a6a0 msgr2=0x7fc44815aa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:13:57.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:57.370+0000 7fc43ea00640 1 --2- 172.21.15.16:0/4216544873 >> v2:172.21.15.89:3300/0 conn(0x7fc44815a6a0 0x7fc44815aa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:57.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:57.370+0000 7fc43ea00640 1 -- 172.21.15.16:0/4216544873 >> v1:172.21.15.16:6789/0 conn(0x7fc4480b2f50 legacy=0x7fc44815e4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:13:57.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:57.370+0000 7fc43ea00640 1 -- 172.21.15.16:0/4216544873 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4480b4930 con 0x7fc4480b3e70 2024-03-31T15:13:57.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:57.370+0000 7fc446a00640 1 -- 172.21.15.16:0/4216544873 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2065472365 0 0) 0x7fc4480b4370 con 0x7fc4480b2f50 2024-03-31T15:13:57.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:57.370+0000 7fc43ea00640 1 --2- 172.21.15.16:0/4216544873 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc4480b3e70 0x7fc448071e80 secure :-1 s=READY pgs=1593 cs=0 l=1 rev1=1 crypto rx=0x7fc448070d20 tx=0x7fc43001c450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aeda587c46680a09 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:13:57.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:57.370+0000 7fc446a00640 1 -- 172.21.15.16:0/4216544873 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4300a7070 con 0x7fc4480b3e70 2024-03-31T15:13:57.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:57.370+0000 7fc446a00640 1 -- 172.21.15.16:0/4216544873 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc4300b0070 con 0x7fc4480b3e70 2024-03-31T15:13:57.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:57.370+0000 7fc446a00640 1 -- 172.21.15.16:0/4216544873 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4300ab890 con 0x7fc4480b3e70 2024-03-31T15:13:57.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:57.374+0000 7fc44e800640 1 -- 172.21.15.16:0/4216544873 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc4480b3e70 msgr2=0x7fc448071e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:13:57.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:57.374+0000 7fc44e800640 1 --2- 172.21.15.16:0/4216544873 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc4480b3e70 0x7fc448071e80 secure :-1 s=READY pgs=1593 cs=0 l=1 rev1=1 crypto rx=0x7fc448070d20 tx=0x7fc43001c450 comp rx=0 tx=0).stop 2024-03-31T15:13:57.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:57.374+0000 7fc44e800640 1 -- 172.21.15.16:0/4216544873 shutdown_connections 2024-03-31T15:13:57.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:57.374+0000 7fc44e800640 1 --2- 172.21.15.16:0/4216544873 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc4480b3e70 0x7fc448071e80 unknown :-1 s=CLOSED pgs=1593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:57.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:57.374+0000 7fc44e800640 1 --2- 172.21.15.16:0/4216544873 >> v2:172.21.15.89:3300/0 conn(0x7fc44815a6a0 0x7fc44815aa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:57.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:57.374+0000 7fc44e800640 1 -- 172.21.15.16:0/4216544873 >> 172.21.15.16:0/4216544873 conn(0x7fc44806b620 msgr2=0x7fc44806ba30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:13:57.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:57.374+0000 7fc44e800640 1 -- 172.21.15.16:0/4216544873 shutdown_connections 2024-03-31T15:13:57.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:57.374+0000 7fc44e800640 1 -- 172.21.15.16:0/4216544873 wait complete. 2024-03-31T15:13:57.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:57.374+0000 7fc44e800640 1 Processor -- start 2024-03-31T15:13:57.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:57.374+0000 7fc44e800640 1 -- start start 2024-03-31T15:13:57.369 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:57.374+0000 7fc44e800640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc4480b2f50 0x7fc448142720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:57.369 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:57.374+0000 7fc44e800640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fc4480b3e70 0x7fc448150430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:57.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:57.374+0000 7fc44e800640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc44815a6a0 0x7fc448150970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:57.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:57.374+0000 7fc44e800640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fc44815ef40 con 0x7fc4480b2f50 2024-03-31T15:13:57.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:57.374+0000 7fc44e800640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fc44815edc0 con 0x7fc44815a6a0 2024-03-31T15:13:57.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:57.374+0000 7fc44e800640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fc44815f0c0 con 0x7fc4480b3e70 2024-03-31T15:13:57.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:57.374+0000 7fc447400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc4480b2f50 0x7fc448142720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:57.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:57.374+0000 7fc447400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc4480b2f50 0x7fc448142720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40584/0 (socket says 172.21.15.16:40584) 2024-03-31T15:13:57.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:57.374+0000 7fc447400640 1 -- 172.21.15.16:0/3896295987 learned_addr learned my addr 172.21.15.16:0/3896295987 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:13:57.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:57.374+0000 7fc43ea00640 1 --2- 172.21.15.16:0/3896295987 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fc4480b3e70 0x7fc448150430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:57.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:57.374+0000 7fc447e00640 1 --2- 172.21.15.16:0/3896295987 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc44815a6a0 0x7fc448150970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:57.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:57.374+0000 7fc447400640 1 -- 172.21.15.16:0/3896295987 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fc4480b3e70 msgr2=0x7fc448150430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:13:57.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:57.374+0000 7fc447400640 1 --2- 172.21.15.16:0/3896295987 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fc4480b3e70 0x7fc448150430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:57.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:57.374+0000 7fc447400640 1 -- 172.21.15.16:0/3896295987 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc44815a6a0 msgr2=0x7fc448150970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:13:57.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:57.374+0000 7fc447400640 1 --2- 172.21.15.16:0/3896295987 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc44815a6a0 0x7fc448150970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:57.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:57.374+0000 7fc447400640 1 -- 172.21.15.16:0/3896295987 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4480cffd0 con 0x7fc4480b2f50 2024-03-31T15:13:57.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:57.378+0000 7fc447400640 1 --2- 172.21.15.16:0/3896295987 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc4480b2f50 0x7fc448142720 secure :-1 s=READY pgs=1594 cs=0 l=1 rev1=1 crypto rx=0x7fc43800d6d0 tx=0x7fc43800db90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:13:57.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:57.378+0000 7fc43fe00640 1 -- 172.21.15.16:0/3896295987 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc438015070 con 0x7fc4480b2f50 2024-03-31T15:13:57.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:57.378+0000 7fc43fe00640 1 -- 172.21.15.16:0/3896295987 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc438007900 con 0x7fc4480b2f50 2024-03-31T15:13:57.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:57.378+0000 7fc43fe00640 1 -- 172.21.15.16:0/3896295987 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc43800b040 con 0x7fc4480b2f50 2024-03-31T15:13:57.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:57.378+0000 7fc44e800640 1 -- 172.21.15.16:0/3896295987 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc4480cdab0 con 0x7fc4480b2f50 2024-03-31T15:13:57.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:57.378+0000 7fc44e800640 1 -- 172.21.15.16:0/3896295987 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc4480ce070 con 0x7fc4480b2f50 2024-03-31T15:13:57.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:57.378+0000 7fc44e800640 1 -- 172.21.15.16:0/3896295987 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc448069950 con 0x7fc4480b2f50 2024-03-31T15:13:57.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:57.378+0000 7fc43fe00640 1 -- 172.21.15.16:0/3896295987 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fc438018070 con 0x7fc4480b2f50 2024-03-31T15:13:57.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:57.378+0000 7fc43fe00640 1 --2- 172.21.15.16:0/3896295987 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fc424068380 0x7fc42406a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:57.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:57.378+0000 7fc43fe00640 1 -- 172.21.15.16:0/3896295987 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 5881+0+0 (secure 0 0 0) 0x7fc438086a70 con 0x7fc4480b2f50 2024-03-31T15:13:57.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:57.382+0000 7fc43ea00640 1 --2- 172.21.15.16:0/3896295987 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fc424068380 0x7fc42406a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:57.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:57.382+0000 7fc43ea00640 1 --2- 172.21.15.16:0/3896295987 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fc424068380 0x7fc42406a840 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7fc4300af000 tx=0x7fc4300a7000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:13:57.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:57.386+0000 7fc43fe00640 1 -- 172.21.15.16:0/3896295987 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fc438056890 con 0x7fc4480b2f50 2024-03-31T15:13:57.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:57.546+0000 7fc44e800640 1 -- 172.21.15.16:0/3896295987 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7fc448157d70 con 0x7fc4480b2f50 2024-03-31T15:13:58.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:58.194+0000 7fc43fe00640 1 -- 172.21.15.16:0/3896295987 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v99) ==== 181+0+0 (secure 0 0 0) 0x7fc43805ab90 con 0x7fc4480b2f50 2024-03-31T15:13:58.187 INFO:teuthology.orchestra.run.smithi016.stderr:pool 'cephfs_metadata' removed 2024-03-31T15:13:58.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:58.198+0000 7fc44e800640 1 -- 172.21.15.16:0/3896295987 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fc424068380 msgr2=0x7fc42406a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:13:58.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:58.198+0000 7fc44e800640 1 --2- 172.21.15.16:0/3896295987 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fc424068380 0x7fc42406a840 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7fc4300af000 tx=0x7fc4300a7000 comp rx=0 tx=0).stop 2024-03-31T15:13:58.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:58.198+0000 7fc44e800640 1 -- 172.21.15.16:0/3896295987 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc4480b2f50 msgr2=0x7fc448142720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:13:58.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:58.198+0000 7fc44e800640 1 --2- 172.21.15.16:0/3896295987 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc4480b2f50 0x7fc448142720 secure :-1 s=READY pgs=1594 cs=0 l=1 rev1=1 crypto rx=0x7fc43800d6d0 tx=0x7fc43800db90 comp rx=0 tx=0).stop 2024-03-31T15:13:58.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:58.198+0000 7fc44e800640 1 -- 172.21.15.16:0/3896295987 shutdown_connections 2024-03-31T15:13:58.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:58.198+0000 7fc44e800640 1 --2- 172.21.15.16:0/3896295987 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fc424068380 0x7fc42406a840 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:58.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:58.198+0000 7fc44e800640 1 --2- 172.21.15.16:0/3896295987 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc44815a6a0 0x7fc448150970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:58.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:58.198+0000 7fc44e800640 1 --2- 172.21.15.16:0/3896295987 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fc4480b3e70 0x7fc448150430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:58.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:58.198+0000 7fc44e800640 1 --2- 172.21.15.16:0/3896295987 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc4480b2f50 0x7fc448142720 unknown :-1 s=CLOSED pgs=1594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:58.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:58.198+0000 7fc44e800640 1 -- 172.21.15.16:0/3896295987 >> 172.21.15.16:0/3896295987 conn(0x7fc44806b620 msgr2=0x7fc4481599c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:13:58.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:58.198+0000 7fc44e800640 1 -- 172.21.15.16:0/3896295987 shutdown_connections 2024-03-31T15:13:58.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:58.198+0000 7fc44e800640 1 -- 172.21.15.16:0/3896295987 wait complete. 2024-03-31T15:13:58.215 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-31T15:13:58.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:58.370+0000 7fdb50000640 1 Processor -- start 2024-03-31T15:13:58.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:58.370+0000 7fdb50000640 1 -- start start 2024-03-31T15:13:58.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:58.370+0000 7fdb50000640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fdb48143f40 0x7fdb480730c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:58.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:58.370+0000 7fdb50000640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fdb480b22f0 0x7fdb480b2f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:58.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:58.370+0000 7fdb50000640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdb48144320 con 0x7fdb480b45d0 2024-03-31T15:13:58.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:58.370+0000 7fdb50000640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7fdb480b4a30 con 0x7fdb48143f40 2024-03-31T15:13:58.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:58.370+0000 7fdb50000640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fdb4815ef80 con 0x7fdb480b22f0 2024-03-31T15:13:58.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:58.370+0000 7fdb4d600640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fdb48143f40 0x7fdb480730c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:58.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:58.370+0000 7fdb4d600640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fdb48143f40 0x7fdb480730c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40588/0 (socket says 172.21.15.16:40588) 2024-03-31T15:13:58.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:58.370+0000 7fdb4d600640 1 -- 172.21.15.16:0/2706664686 learned_addr learned my addr 172.21.15.16:0/2706664686 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:13:58.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:58.370+0000 7fdb4e000640 1 --2- 172.21.15.16:0/2706664686 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fdb480b22f0 0x7fdb480b2f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:58.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:58.370+0000 7fdb4d600640 1 -- 172.21.15.16:0/2706664686 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fdb480b22f0 msgr2=0x7fdb480b2f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:13:58.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:58.370+0000 7fdb4d600640 1 --2- 172.21.15.16:0/2706664686 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fdb480b22f0 0x7fdb480b2f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:58.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:58.370+0000 7fdb4d600640 1 -- 172.21.15.16:0/2706664686 >> v1:172.21.15.89:6789/0 conn(0x7fdb480b45d0 legacy=0x7fdb480b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:13:58.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:58.370+0000 7fdb4d600640 1 -- 172.21.15.16:0/2706664686 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb4815f260 con 0x7fdb48143f40 2024-03-31T15:13:58.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:58.370+0000 7fdb4e000640 1 --2- 172.21.15.16:0/2706664686 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fdb480b22f0 0x7fdb480b2f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:13:58.368 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:58.374+0000 7fdb4d600640 1 --2- 172.21.15.16:0/2706664686 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdb48143f40 0x7fdb480730c0 secure :-1 s=READY pgs=1595 cs=0 l=1 rev1=1 crypto rx=0x7fdb3c009870 tx=0x7fdb3c01c560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a628b92adea0537 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:13:58.368 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:58.374+0000 7fdb47e00640 1 -- 172.21.15.16:0/2706664686 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb3c0b2070 con 0x7fdb48143f40 2024-03-31T15:13:58.368 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:58.374+0000 7fdb47e00640 1 -- 172.21.15.16:0/2706664686 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdb3c0a9070 con 0x7fdb48143f40 2024-03-31T15:13:58.368 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:58.374+0000 7fdb47e00640 1 -- 172.21.15.16:0/2706664686 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb3c0ad880 con 0x7fdb48143f40 2024-03-31T15:13:58.369 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:58.374+0000 7fdb50000640 1 -- 172.21.15.16:0/2706664686 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdb48143f40 msgr2=0x7fdb480730c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:13:58.369 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:58.374+0000 7fdb50000640 1 --2- 172.21.15.16:0/2706664686 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdb48143f40 0x7fdb480730c0 secure :-1 s=READY pgs=1595 cs=0 l=1 rev1=1 crypto rx=0x7fdb3c009870 tx=0x7fdb3c01c560 comp rx=0 tx=0).stop 2024-03-31T15:13:58.369 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:58.374+0000 7fdb50000640 1 -- 172.21.15.16:0/2706664686 shutdown_connections 2024-03-31T15:13:58.369 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:58.374+0000 7fdb50000640 1 --2- 172.21.15.16:0/2706664686 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fdb480b22f0 0x7fdb480b2f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:58.369 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:58.374+0000 7fdb50000640 1 --2- 172.21.15.16:0/2706664686 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdb48143f40 0x7fdb480730c0 unknown :-1 s=CLOSED pgs=1595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:58.369 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:58.374+0000 7fdb50000640 1 -- 172.21.15.16:0/2706664686 >> 172.21.15.16:0/2706664686 conn(0x7fdb4806b620 msgr2=0x7fdb4806ba30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:13:58.369 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:58.374+0000 7fdb50000640 1 -- 172.21.15.16:0/2706664686 shutdown_connections 2024-03-31T15:13:58.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:58.374+0000 7fdb50000640 1 -- 172.21.15.16:0/2706664686 wait complete. 2024-03-31T15:13:58.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:58.378+0000 7fdb50000640 1 Processor -- start 2024-03-31T15:13:58.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:58.378+0000 7fdb50000640 1 -- start start 2024-03-31T15:13:58.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:58.378+0000 7fdb50000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdb480b22f0 0x7fdb4815fe60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:58.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:58.378+0000 7fdb50000640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdb480b45d0 0x7fdb4813eba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:58.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:58.378+0000 7fdb50000640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fdb48143f40 0x7fdb480cfa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:58.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:58.378+0000 7fdb50000640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fdb48168230 con 0x7fdb480b22f0 2024-03-31T15:13:58.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:58.378+0000 7fdb50000640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fdb481680b0 con 0x7fdb480b45d0 2024-03-31T15:13:58.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:58.378+0000 7fdb50000640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fdb481683b0 con 0x7fdb48143f40 2024-03-31T15:13:58.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:58.378+0000 7fdb4d600640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdb480b22f0 0x7fdb4815fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:58.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:58.378+0000 7fdb4d600640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdb480b22f0 0x7fdb4815fe60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40602/0 (socket says 172.21.15.16:40602) 2024-03-31T15:13:58.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:58.378+0000 7fdb4d600640 1 -- 172.21.15.16:0/1251757955 learned_addr learned my addr 172.21.15.16:0/1251757955 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:13:58.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:58.378+0000 7fdb4cc00640 1 --2- 172.21.15.16:0/1251757955 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdb480b45d0 0x7fdb4813eba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:58.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:58.378+0000 7fdb4e000640 1 --2- 172.21.15.16:0/1251757955 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fdb48143f40 0x7fdb480cfa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:58.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:58.378+0000 7fdb4d600640 1 -- 172.21.15.16:0/1251757955 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fdb48143f40 msgr2=0x7fdb480cfa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:13:58.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:58.378+0000 7fdb4d600640 1 --2- 172.21.15.16:0/1251757955 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fdb48143f40 0x7fdb480cfa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:58.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:58.378+0000 7fdb4d600640 1 -- 172.21.15.16:0/1251757955 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdb480b45d0 msgr2=0x7fdb4813eba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:13:58.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:58.378+0000 7fdb4d600640 1 --2- 172.21.15.16:0/1251757955 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdb480b45d0 0x7fdb4813eba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:58.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:58.378+0000 7fdb4d600640 1 -- 172.21.15.16:0/1251757955 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb48160440 con 0x7fdb480b22f0 2024-03-31T15:13:58.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:58.378+0000 7fdb4e000640 1 --2- 172.21.15.16:0/1251757955 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fdb48143f40 0x7fdb480cfa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:13:58.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:58.378+0000 7fdb4d600640 1 --2- 172.21.15.16:0/1251757955 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdb480b22f0 0x7fdb4815fe60 secure :-1 s=READY pgs=1596 cs=0 l=1 rev1=1 crypto rx=0x7fdb3c0b0040 tx=0x7fdb3c01ada0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:13:58.375 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:58.378+0000 7fdb45600640 1 -- 172.21.15.16:0/1251757955 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb3c0b2040 con 0x7fdb480b22f0 2024-03-31T15:13:58.375 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:58.378+0000 7fdb45600640 1 -- 172.21.15.16:0/1251757955 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdb3c0adc10 con 0x7fdb480b22f0 2024-03-31T15:13:58.375 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:58.378+0000 7fdb45600640 1 -- 172.21.15.16:0/1251757955 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb3c0b8420 con 0x7fdb480b22f0 2024-03-31T15:13:58.375 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:58.378+0000 7fdb50000640 1 -- 172.21.15.16:0/1251757955 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdb48141b20 con 0x7fdb480b22f0 2024-03-31T15:13:58.375 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:58.378+0000 7fdb50000640 1 -- 172.21.15.16:0/1251757955 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdb481420e0 con 0x7fdb480b22f0 2024-03-31T15:13:58.375 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:58.382+0000 7fdb50000640 1 -- 172.21.15.16:0/1251757955 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdb48069950 con 0x7fdb480b22f0 2024-03-31T15:13:58.375 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:58.382+0000 7fdb45600640 1 -- 172.21.15.16:0/1251757955 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fdb3c0c0030 con 0x7fdb480b22f0 2024-03-31T15:13:58.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:58.382+0000 7fdb45600640 1 --2- 172.21.15.16:0/1251757955 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fdb280683b0 0x7fdb2806a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:58.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:58.382+0000 7fdb45600640 1 -- 172.21.15.16:0/1251757955 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 5443+0+0 (secure 0 0 0) 0x7fdb3c120190 con 0x7fdb480b22f0 2024-03-31T15:13:58.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:58.382+0000 7fdb4cc00640 1 --2- 172.21.15.16:0/1251757955 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fdb280683b0 0x7fdb2806a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:58.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:58.382+0000 7fdb4cc00640 1 --2- 172.21.15.16:0/1251757955 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fdb280683b0 0x7fdb2806a870 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7fdb4815bec0 tx=0x7fdb34009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:13:58.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:58.386+0000 7fdb45600640 1 -- 172.21.15.16:0/1251757955 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fdb3c0c7060 con 0x7fdb480b22f0 2024-03-31T15:13:58.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:58.554+0000 7fdb50000640 1 -- 172.21.15.16:0/1251757955 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7fdb48071d20 con 0x7fdb480b22f0 2024-03-31T15:13:59.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:59.206+0000 7fdb45600640 1 -- 172.21.15.16:0/1251757955 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v100) ==== 169+0+0 (secure 0 0 0) 0x7fdb3c016150 con 0x7fdb480b22f0 2024-03-31T15:13:59.200 INFO:teuthology.orchestra.run.smithi016.stderr:pool 'cephfs_data' removed 2024-03-31T15:13:59.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:59.210+0000 7fdb50000640 1 -- 172.21.15.16:0/1251757955 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fdb280683b0 msgr2=0x7fdb2806a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:13:59.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:59.210+0000 7fdb50000640 1 --2- 172.21.15.16:0/1251757955 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fdb280683b0 0x7fdb2806a870 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7fdb4815bec0 tx=0x7fdb34009040 comp rx=0 tx=0).stop 2024-03-31T15:13:59.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:59.210+0000 7fdb50000640 1 -- 172.21.15.16:0/1251757955 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdb480b22f0 msgr2=0x7fdb4815fe60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:13:59.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:59.210+0000 7fdb50000640 1 --2- 172.21.15.16:0/1251757955 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdb480b22f0 0x7fdb4815fe60 secure :-1 s=READY pgs=1596 cs=0 l=1 rev1=1 crypto rx=0x7fdb3c0b0040 tx=0x7fdb3c01ada0 comp rx=0 tx=0).stop 2024-03-31T15:13:59.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:59.210+0000 7fdb50000640 1 -- 172.21.15.16:0/1251757955 shutdown_connections 2024-03-31T15:13:59.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:59.210+0000 7fdb50000640 1 --2- 172.21.15.16:0/1251757955 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fdb280683b0 0x7fdb2806a870 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:59.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:59.210+0000 7fdb50000640 1 --2- 172.21.15.16:0/1251757955 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fdb48143f40 0x7fdb480cfa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:59.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:59.210+0000 7fdb50000640 1 --2- 172.21.15.16:0/1251757955 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdb480b45d0 0x7fdb4813eba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:59.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:59.210+0000 7fdb50000640 1 --2- 172.21.15.16:0/1251757955 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdb480b22f0 0x7fdb4815fe60 unknown :-1 s=CLOSED pgs=1596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:59.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:59.210+0000 7fdb50000640 1 -- 172.21.15.16:0/1251757955 >> 172.21.15.16:0/1251757955 conn(0x7fdb4806b620 msgr2=0x7fdb480cda20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:13:59.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:59.210+0000 7fdb50000640 1 -- 172.21.15.16:0/1251757955 shutdown_connections 2024-03-31T15:13:59.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:59.210+0000 7fdb50000640 1 -- 172.21.15.16:0/1251757955 wait complete. 2024-03-31T15:13:59.225 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-03-31T15:13:59.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:59.386+0000 7fd9ce600640 1 Processor -- start 2024-03-31T15:13:59.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:59.386+0000 7fd9ce600640 1 -- start start 2024-03-31T15:13:59.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:59.386+0000 7fd9ce600640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fd9c8165150 0x7fd9c8167540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:59.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:59.386+0000 7fd9ce600640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd9c815c7e0 con 0x7fd9c806d4d0 2024-03-31T15:13:59.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:59.386+0000 7fd9ce600640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd9c80736b0 con 0x7fd9c81567e0 2024-03-31T15:13:59.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:59.386+0000 7fd9ce600640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7fd9c815caa0 con 0x7fd9c8165150 2024-03-31T15:13:59.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:59.386+0000 7fd9bea00640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fd9c806d4d0 0x7fd9c806d930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:57180/0 (socket says 172.21.15.16:57180) 2024-03-31T15:13:59.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:59.386+0000 7fd9bea00640 1 -- 172.21.15.16:0/2102709853 learned_addr learned my addr 172.21.15.16:0/2102709853 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:13:59.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:59.386+0000 7fd9c7e00640 1 --2- 172.21.15.16:0/2102709853 >> v2:172.21.15.89:3300/0 conn(0x7fd9c8165150 0x7fd9c8167540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:59.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:59.386+0000 7fd9c6a00640 1 -- 172.21.15.16:0/2102709853 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 918157011 0 0) 0x7fd9c815c7e0 con 0x7fd9c806d4d0 2024-03-31T15:13:59.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:59.386+0000 7fd9c6a00640 1 -- 172.21.15.16:0/2102709853 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd9ac003530 con 0x7fd9c806d4d0 2024-03-31T15:13:59.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:59.386+0000 7fd9c6a00640 1 -- 172.21.15.16:0/2102709853 <== mon.1 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4188589684 0 0) 0x7fd9c80736b0 con 0x7fd9c81567e0 2024-03-31T15:13:59.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:59.386+0000 7fd9c6a00640 1 -- 172.21.15.16:0/2102709853 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd9c815c7e0 con 0x7fd9c81567e0 2024-03-31T15:13:59.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:59.386+0000 7fd9c6a00640 1 -- 172.21.15.16:0/2102709853 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3024115787 0 0) 0x7fd9ac003530 con 0x7fd9c806d4d0 2024-03-31T15:13:59.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:59.386+0000 7fd9c6a00640 1 -- 172.21.15.16:0/2102709853 >> v2:172.21.15.89:3300/0 conn(0x7fd9c8165150 msgr2=0x7fd9c8167540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:13:59.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:59.386+0000 7fd9c6a00640 1 --2- 172.21.15.16:0/2102709853 >> v2:172.21.15.89:3300/0 conn(0x7fd9c8165150 0x7fd9c8167540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:59.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:59.386+0000 7fd9c6a00640 1 -- 172.21.15.16:0/2102709853 >> v1:172.21.15.89:6789/0 conn(0x7fd9c81567e0 legacy=0x7fd9c8072c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:13:59.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:59.386+0000 7fd9c6a00640 1 -- 172.21.15.16:0/2102709853 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9c815cd80 con 0x7fd9c806d4d0 2024-03-31T15:13:59.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:59.386+0000 7fd9c6a00640 1 -- 172.21.15.16:0/2102709853 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 214807623 0 0) 0x7fd9b0001a00 con 0x7fd9c806d4d0 2024-03-31T15:13:59.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:59.386+0000 7fd9c7e00640 1 --2- 172.21.15.16:0/2102709853 >> v2:172.21.15.89:3300/0 conn(0x7fd9c8165150 0x7fd9c8167540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:13:59.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:59.386+0000 7fd9c6a00640 1 -- 172.21.15.16:0/2102709853 >> v1:172.21.15.16:6789/0 conn(0x7fd9c806d4d0 legacy=0x7fd9c806d930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:13:59.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:59.386+0000 7fd9c6a00640 1 --2- 172.21.15.16:0/2102709853 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd9ac003ba0 0x7fd9ac003fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:59.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:59.386+0000 7fd9c6a00640 1 --2- 172.21.15.16:0/2102709853 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd9ac004810 0x7fd9ac004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:59.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:59.386+0000 7fd9c6a00640 1 --2- 172.21.15.16:0/2102709853 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fd9ac0051d0 0x7fd9ac00b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:59.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:59.386+0000 7fd9c6a00640 1 -- 172.21.15.16:0/2102709853 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fd9c815caa0 con 0x7fd9ac004810 2024-03-31T15:13:59.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:59.386+0000 7fd9c6a00640 1 -- 172.21.15.16:0/2102709853 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fd9ac00c0a0 con 0x7fd9ac003ba0 2024-03-31T15:13:59.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:59.386+0000 7fd9c6a00640 1 -- 172.21.15.16:0/2102709853 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fd9ac00c2a0 con 0x7fd9ac0051d0 2024-03-31T15:13:59.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:59.386+0000 7fd9bea00640 1 --2- 172.21.15.16:0/2102709853 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd9ac004810 0x7fd9ac004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:59.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:59.386+0000 7fd9c7400640 1 --2- 172.21.15.16:0/2102709853 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd9ac003ba0 0x7fd9ac003fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:59.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:59.386+0000 7fd9c7e00640 1 --2- 172.21.15.16:0/2102709853 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fd9ac0051d0 0x7fd9ac00b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:59.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:59.390+0000 7fd9bea00640 1 -- 172.21.15.16:0/2102709853 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fd9ac0051d0 msgr2=0x7fd9ac00b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:13:59.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:59.390+0000 7fd9bea00640 1 --2- 172.21.15.16:0/2102709853 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fd9ac0051d0 0x7fd9ac00b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:59.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:59.390+0000 7fd9bea00640 1 -- 172.21.15.16:0/2102709853 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd9ac003ba0 msgr2=0x7fd9ac003fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:13:59.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:59.390+0000 7fd9bea00640 1 --2- 172.21.15.16:0/2102709853 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd9ac003ba0 0x7fd9ac003fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:59.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:59.390+0000 7fd9bea00640 1 -- 172.21.15.16:0/2102709853 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9ac003530 con 0x7fd9ac004810 2024-03-31T15:13:59.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:59.390+0000 7fd9c7e00640 1 -- 172.21.15.16:0/2102709853 reap_dead start 2024-03-31T15:13:59.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:59.390+0000 7fd9bea00640 1 --2- 172.21.15.16:0/2102709853 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd9ac004810 0x7fd9ac004c90 secure :-1 s=READY pgs=1598 cs=0 l=1 rev1=1 crypto rx=0x7fd9b000a750 tx=0x7fd9b001d200 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4bca1d2fd55d5437 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:13:59.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:59.390+0000 7fd9c6a00640 1 -- 172.21.15.16:0/2102709853 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9b000a370 con 0x7fd9ac004810 2024-03-31T15:13:59.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:59.390+0000 7fd9c6a00640 1 -- 172.21.15.16:0/2102709853 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd9b00ad630 con 0x7fd9ac004810 2024-03-31T15:13:59.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:59.390+0000 7fd9c6a00640 1 -- 172.21.15.16:0/2102709853 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9b00ada20 con 0x7fd9ac004810 2024-03-31T15:13:59.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:59.390+0000 7fd9ce600640 1 -- 172.21.15.16:0/2102709853 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd9ac004810 msgr2=0x7fd9ac004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:13:59.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:59.390+0000 7fd9ce600640 1 --2- 172.21.15.16:0/2102709853 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd9ac004810 0x7fd9ac004c90 secure :-1 s=READY pgs=1598 cs=0 l=1 rev1=1 crypto rx=0x7fd9b000a750 tx=0x7fd9b001d200 comp rx=0 tx=0).stop 2024-03-31T15:13:59.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:59.390+0000 7fd9ce600640 1 -- 172.21.15.16:0/2102709853 shutdown_connections 2024-03-31T15:13:59.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:59.390+0000 7fd9ce600640 1 --2- 172.21.15.16:0/2102709853 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd9ac004810 0x7fd9ac004c90 unknown :-1 s=CLOSED pgs=1598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:59.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:59.390+0000 7fd9ce600640 1 -- 172.21.15.16:0/2102709853 >> 172.21.15.16:0/2102709853 conn(0x7fd9c806ee40 msgr2=0x7fd9c806b620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:13:59.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:59.390+0000 7fd9ce600640 1 -- 172.21.15.16:0/2102709853 shutdown_connections 2024-03-31T15:13:59.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:59.390+0000 7fd9ce600640 1 -- 172.21.15.16:0/2102709853 wait complete. 2024-03-31T15:13:59.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:59.394+0000 7fd9ce600640 1 Processor -- start 2024-03-31T15:13:59.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:59.394+0000 7fd9ce600640 1 -- start start 2024-03-31T15:13:59.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:59.394+0000 7fd9ce600640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fd9ac004810 0x7fd9c81481d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:59.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:59.394+0000 7fd9ce600640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd9c8148710 0x7fd9c815af90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:59.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:59.394+0000 7fd9ce600640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd9c815b4d0 0x7fd9c815b980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:59.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:59.394+0000 7fd9ce600640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fd9c80afbb0 con 0x7fd9c815b4d0 2024-03-31T15:13:59.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:59.394+0000 7fd9ce600640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fd9c80afa30 con 0x7fd9c8148710 2024-03-31T15:13:59.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:59.394+0000 7fd9ce600640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fd9c815d580 con 0x7fd9ac004810 2024-03-31T15:13:59.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:59.394+0000 7fd9c7e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd9c815b4d0 0x7fd9c815b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:59.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:59.394+0000 7fd9c7e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd9c815b4d0 0x7fd9c815b980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40634/0 (socket says 172.21.15.16:40634) 2024-03-31T15:13:59.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:59.394+0000 7fd9c7e00640 1 -- 172.21.15.16:0/2717758709 learned_addr learned my addr 172.21.15.16:0/2717758709 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:13:59.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:59.394+0000 7fd9bea00640 1 --2- 172.21.15.16:0/2717758709 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd9c8148710 0x7fd9c815af90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:59.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:59.394+0000 7fd9c7400640 1 --2- 172.21.15.16:0/2717758709 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fd9ac004810 0x7fd9c81481d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:59.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:59.394+0000 7fd9c7e00640 1 -- 172.21.15.16:0/2717758709 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fd9ac004810 msgr2=0x7fd9c81481d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:13:59.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:59.394+0000 7fd9c7e00640 1 --2- 172.21.15.16:0/2717758709 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fd9ac004810 0x7fd9c81481d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:59.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:59.394+0000 7fd9c7e00640 1 -- 172.21.15.16:0/2717758709 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd9c8148710 msgr2=0x7fd9c815af90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:13:59.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:59.394+0000 7fd9c7e00640 1 --2- 172.21.15.16:0/2717758709 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd9c8148710 0x7fd9c815af90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:59.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:59.394+0000 7fd9c7e00640 1 -- 172.21.15.16:0/2717758709 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9c80b10b0 con 0x7fd9c815b4d0 2024-03-31T15:13:59.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:59.394+0000 7fd9bea00640 1 --2- 172.21.15.16:0/2717758709 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd9c8148710 0x7fd9c815af90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:13:59.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:59.394+0000 7fd9c7e00640 1 --2- 172.21.15.16:0/2717758709 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd9c815b4d0 0x7fd9c815b980 secure :-1 s=READY pgs=1599 cs=0 l=1 rev1=1 crypto rx=0x7fd9c80b5610 tx=0x7fd9c0011e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:13:59.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:59.394+0000 7fd9bfe00640 1 -- 172.21.15.16:0/2717758709 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9c0004020 con 0x7fd9c815b4d0 2024-03-31T15:13:59.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:59.394+0000 7fd9bfe00640 1 -- 172.21.15.16:0/2717758709 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd9c00041e0 con 0x7fd9c815b4d0 2024-03-31T15:13:59.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:59.394+0000 7fd9bfe00640 1 -- 172.21.15.16:0/2717758709 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9c00043d0 con 0x7fd9c815b4d0 2024-03-31T15:13:59.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:59.394+0000 7fd9ce600640 1 -- 172.21.15.16:0/2717758709 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd9c80b1390 con 0x7fd9c815b4d0 2024-03-31T15:13:59.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:59.394+0000 7fd9ce600640 1 -- 172.21.15.16:0/2717758709 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd9c80b1950 con 0x7fd9c815b4d0 2024-03-31T15:13:59.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:59.398+0000 7fd9ce600640 1 -- 172.21.15.16:0/2717758709 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd9c8069950 con 0x7fd9c815b4d0 2024-03-31T15:13:59.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:59.398+0000 7fd9bfe00640 1 -- 172.21.15.16:0/2717758709 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fd9c001f020 con 0x7fd9c815b4d0 2024-03-31T15:13:59.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:59.398+0000 7fd9bfe00640 1 --2- 172.21.15.16:0/2717758709 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fd9a40683b0 0x7fd9a406a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:59.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:59.398+0000 7fd9bfe00640 1 -- 172.21.15.16:0/2717758709 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 5029+0+0 (secure 0 0 0) 0x7fd9c0086270 con 0x7fd9c815b4d0 2024-03-31T15:13:59.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:59.398+0000 7fd9c7400640 1 --2- 172.21.15.16:0/2717758709 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fd9a40683b0 0x7fd9a406a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:59.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:59.402+0000 7fd9c7400640 1 --2- 172.21.15.16:0/2717758709 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fd9a40683b0 0x7fd9a406a870 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7fd9b8007860 tx=0x7fd9b80077f0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:13:59.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:59.402+0000 7fd9bfe00640 1 -- 172.21.15.16:0/2717758709 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fd9c0056ad0 con 0x7fd9c815b4d0 2024-03-31T15:13:59.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:59.570+0000 7fd9ce600640 1 -- 172.21.15.16:0/2717758709 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) -- 0x7fd9c806e470 con 0x7fd9c815b4d0 2024-03-31T15:13:59.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:59.574+0000 7fd9bfe00640 1 -- 172.21.15.16:0/2717758709 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v21) ==== 104+0+0 (secure 0 0 0) 0x7fd9c0012e10 con 0x7fd9c815b4d0 2024-03-31T15:13:59.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:59.574+0000 7fd9ce600640 1 -- 172.21.15.16:0/2717758709 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fd9a40683b0 msgr2=0x7fd9a406a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:13:59.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:59.574+0000 7fd9ce600640 1 --2- 172.21.15.16:0/2717758709 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fd9a40683b0 0x7fd9a406a870 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7fd9b8007860 tx=0x7fd9b80077f0 comp rx=0 tx=0).stop 2024-03-31T15:13:59.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:59.574+0000 7fd9ce600640 1 -- 172.21.15.16:0/2717758709 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd9c815b4d0 msgr2=0x7fd9c815b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:13:59.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:59.574+0000 7fd9ce600640 1 --2- 172.21.15.16:0/2717758709 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd9c815b4d0 0x7fd9c815b980 secure :-1 s=READY pgs=1599 cs=0 l=1 rev1=1 crypto rx=0x7fd9c80b5610 tx=0x7fd9c0011e30 comp rx=0 tx=0).stop 2024-03-31T15:13:59.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:59.574+0000 7fd9ce600640 1 -- 172.21.15.16:0/2717758709 shutdown_connections 2024-03-31T15:13:59.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:59.574+0000 7fd9ce600640 1 --2- 172.21.15.16:0/2717758709 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fd9a40683b0 0x7fd9a406a870 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:59.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:59.574+0000 7fd9ce600640 1 --2- 172.21.15.16:0/2717758709 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd9c815b4d0 0x7fd9c815b980 unknown :-1 s=CLOSED pgs=1599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:59.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:59.574+0000 7fd9ce600640 1 --2- 172.21.15.16:0/2717758709 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd9c8148710 0x7fd9c815af90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:59.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:59.574+0000 7fd9ce600640 1 --2- 172.21.15.16:0/2717758709 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fd9ac004810 0x7fd9c81481d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:59.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:59.574+0000 7fd9ce600640 1 -- 172.21.15.16:0/2717758709 >> 172.21.15.16:0/2717758709 conn(0x7fd9c806ee40 msgr2=0x7fd9c815fbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:13:59.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:59.574+0000 7fd9ce600640 1 -- 172.21.15.16:0/2717758709 shutdown_connections 2024-03-31T15:13:59.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:59.574+0000 7fd9ce600640 1 -- 172.21.15.16:0/2717758709 wait complete. 2024-03-31T15:13:59.588 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_inotable_sync' 2024-03-31T15:13:59.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:59.734+0000 7f0543e00640 1 Processor -- start 2024-03-31T15:13:59.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:59.734+0000 7f0543e00640 1 -- start start 2024-03-31T15:13:59.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:59.738+0000 7f0543e00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f053c0728b0 0x7f053c0b1cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:59.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:59.738+0000 7f0543e00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f053c0b2200 0x7f053c15cd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:59.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:59.738+0000 7f0543e00640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f053c160e70 con 0x7f053c145d10 2024-03-31T15:13:59.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:59.738+0000 7f0543e00640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f053c1602f0 con 0x7f053c0728b0 2024-03-31T15:13:59.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:59.738+0000 7f0543e00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f053c1605d0 con 0x7f053c0b2200 2024-03-31T15:13:59.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:59.738+0000 7f0541e00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f053c0b2200 0x7f053c15cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:59.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:59.738+0000 7f0541e00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f053c0b2200 0x7f053c15cd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3301/0 says I am v2:172.21.15.16:44582/0 (socket says 172.21.15.16:44582) 2024-03-31T15:13:59.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:59.738+0000 7f0540a00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f053c0728b0 0x7f053c0b1cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:59.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:59.738+0000 7f0541400640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f053c145d10 0x7f053c0cf790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.16:47580/0 (socket says 172.21.15.16:47580) 2024-03-31T15:13:59.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:59.738+0000 7f0541e00640 1 -- 172.21.15.16:0/1744816199 learned_addr learned my addr 172.21.15.16:0/1744816199 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:13:59.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:59.738+0000 7f0541e00640 1 -- 172.21.15.16:0/1744816199 >> v2:172.21.15.89:3300/0 conn(0x7f053c0728b0 msgr2=0x7f053c0b1cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:13:59.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:59.738+0000 7f0541e00640 1 --2- 172.21.15.16:0/1744816199 >> v2:172.21.15.89:3300/0 conn(0x7f053c0728b0 0x7f053c0b1cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:59.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:59.738+0000 7f0541e00640 1 -- 172.21.15.16:0/1744816199 >> v1:172.21.15.89:6789/0 conn(0x7f053c145d10 legacy=0x7f053c0cf790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:13:59.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:59.738+0000 7f0541e00640 1 -- 172.21.15.16:0/1744816199 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f053c1608b0 con 0x7f053c0b2200 2024-03-31T15:13:59.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:59.738+0000 7f053be00640 1 -- 172.21.15.16:0/1744816199 <== mon.1 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2493785391 0 0) 0x7f053c160e70 con 0x7f053c145d10 2024-03-31T15:13:59.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:59.738+0000 7f0541e00640 1 --2- 172.21.15.16:0/1744816199 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f053c0b2200 0x7f053c15cd50 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f0534009e50 tx=0x7f053401ca30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2971e1834151e6ad server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:13:59.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:59.742+0000 7f053be00640 1 -- 172.21.15.16:0/1744816199 <== mon.2 v2:172.21.15.89:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05340a5700 con 0x7f053c0b2200 2024-03-31T15:13:59.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:59.742+0000 7f053be00640 1 -- 172.21.15.16:0/1744816199 <== mon.2 v2:172.21.15.89:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f05340a58a0 con 0x7f053c0b2200 2024-03-31T15:13:59.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:59.742+0000 7f053be00640 1 -- 172.21.15.16:0/1744816199 <== mon.2 v2:172.21.15.89:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05340a5c70 con 0x7f053c0b2200 2024-03-31T15:13:59.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:59.742+0000 7f0543e00640 1 -- 172.21.15.16:0/1744816199 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f053c0b2200 msgr2=0x7f053c15cd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:13:59.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:59.742+0000 7f0543e00640 1 --2- 172.21.15.16:0/1744816199 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f053c0b2200 0x7f053c15cd50 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f0534009e50 tx=0x7f053401ca30 comp rx=0 tx=0).stop 2024-03-31T15:13:59.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:59.742+0000 7f0543e00640 1 -- 172.21.15.16:0/1744816199 shutdown_connections 2024-03-31T15:13:59.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:59.742+0000 7f0543e00640 1 --2- 172.21.15.16:0/1744816199 >> v2:172.21.15.89:3300/0 conn(0x7f053c0728b0 0x7f053c0b1cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:59.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:59.742+0000 7f0543e00640 1 --2- 172.21.15.16:0/1744816199 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f053c0b2200 0x7f053c15cd50 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:59.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:59.742+0000 7f0543e00640 1 -- 172.21.15.16:0/1744816199 >> 172.21.15.16:0/1744816199 conn(0x7f053c06b620 msgr2=0x7f053c06ba30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:13:59.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:59.742+0000 7f0543e00640 1 -- 172.21.15.16:0/1744816199 shutdown_connections 2024-03-31T15:13:59.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:59.742+0000 7f0543e00640 1 -- 172.21.15.16:0/1744816199 wait complete. 2024-03-31T15:13:59.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:59.742+0000 7f0543e00640 1 Processor -- start 2024-03-31T15:13:59.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:59.742+0000 7f0543e00640 1 -- start start 2024-03-31T15:13:59.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:59.742+0000 7f0543e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f053c0728b0 0x7f053c15b940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:59.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:59.742+0000 7f0543e00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f053c0b2200 0x7f053c13c810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:59.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:59.742+0000 7f0543e00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f053c145d10 0x7f053c15b170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:59.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:59.742+0000 7f0543e00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f053c072e40 con 0x7f053c0728b0 2024-03-31T15:13:59.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:59.742+0000 7f0543e00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f053c072cc0 con 0x7f053c0b2200 2024-03-31T15:13:59.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:59.742+0000 7f0543e00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f053c1690f0 con 0x7f053c145d10 2024-03-31T15:13:59.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:59.742+0000 7f0541400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f053c0728b0 0x7f053c15b940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:59.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:59.742+0000 7f0541400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f053c0728b0 0x7f053c15b940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40642/0 (socket says 172.21.15.16:40642) 2024-03-31T15:13:59.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:59.742+0000 7f0541400640 1 -- 172.21.15.16:0/1418316301 learned_addr learned my addr 172.21.15.16:0/1418316301 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:13:59.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:59.746+0000 7f0540a00640 1 --2- 172.21.15.16:0/1418316301 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f053c0b2200 0x7f053c13c810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:59.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:59.746+0000 7f0541e00640 1 --2- 172.21.15.16:0/1418316301 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f053c145d10 0x7f053c15b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:59.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:59.746+0000 7f0541400640 1 -- 172.21.15.16:0/1418316301 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f053c145d10 msgr2=0x7f053c15b170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:13:59.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:59.746+0000 7f0541400640 1 --2- 172.21.15.16:0/1418316301 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f053c145d10 0x7f053c15b170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:59.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:59.746+0000 7f0541400640 1 -- 172.21.15.16:0/1418316301 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f053c0b2200 msgr2=0x7f053c13c810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:13:59.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:59.746+0000 7f0541400640 1 --2- 172.21.15.16:0/1418316301 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f053c0b2200 0x7f053c13c810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:13:59.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:59.746+0000 7f0541400640 1 -- 172.21.15.16:0/1418316301 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f053c0b44b0 con 0x7f053c0728b0 2024-03-31T15:13:59.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:59.746+0000 7f0541e00640 1 --2- 172.21.15.16:0/1418316301 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f053c145d10 0x7f053c15b170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:13:59.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:59.746+0000 7f0541400640 1 --2- 172.21.15.16:0/1418316301 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f053c0728b0 0x7f053c15b940 secure :-1 s=READY pgs=1600 cs=0 l=1 rev1=1 crypto rx=0x7f053000c410 tx=0x7f053000c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:13:59.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:59.746+0000 7f0539600640 1 -- 172.21.15.16:0/1418316301 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05300074e0 con 0x7f053c0728b0 2024-03-31T15:13:59.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:59.746+0000 7f0539600640 1 -- 172.21.15.16:0/1418316301 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f05300047f0 con 0x7f053c0728b0 2024-03-31T15:13:59.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:59.746+0000 7f0539600640 1 -- 172.21.15.16:0/1418316301 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0530007680 con 0x7f053c0728b0 2024-03-31T15:13:59.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:59.746+0000 7f0543e00640 1 -- 172.21.15.16:0/1418316301 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f053c0b4790 con 0x7f053c0728b0 2024-03-31T15:13:59.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:59.746+0000 7f0543e00640 1 -- 172.21.15.16:0/1418316301 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f053c144f70 con 0x7f053c0728b0 2024-03-31T15:13:59.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:59.746+0000 7f0543e00640 1 -- 172.21.15.16:0/1418316301 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f053c069950 con 0x7f053c0728b0 2024-03-31T15:13:59.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:59.746+0000 7f0539600640 1 -- 172.21.15.16:0/1418316301 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f0530011070 con 0x7f053c0728b0 2024-03-31T15:13:59.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:59.746+0000 7f0539600640 1 --2- 172.21.15.16:0/1418316301 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f051c068380 0x7f051c06a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:13:59.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:59.746+0000 7f0539600640 1 -- 172.21.15.16:0/1418316301 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 5029+0+0 (secure 0 0 0) 0x7f05300861c0 con 0x7f053c0728b0 2024-03-31T15:13:59.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:59.750+0000 7f0540a00640 1 --2- 172.21.15.16:0/1418316301 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f051c068380 0x7f051c06a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:13:59.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:59.750+0000 7f0540a00640 1 --2- 172.21.15.16:0/1418316301 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f051c068380 0x7f051c06a840 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f053c166080 tx=0x7f0528009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:13:59.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:59.754+0000 7f0539600640 1 -- 172.21.15.16:0/1418316301 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f0530056ab0 con 0x7f053c0728b0 2024-03-31T15:13:59.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:13:59.986+0000 7f0543e00640 1 -- 172.21.15.16:0/1418316301 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_inotable_sync"]} v 0) -- 0x7f053c166a90 con 0x7f053c0728b0 2024-03-31T15:14:00.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:00.210+0000 7f0539600640 1 -- 172.21.15.16:0/1418316301 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_inotable_sync"]}]=0 v482) ==== 146+0+0 (secure 0 0 0) 0x7f05300563b0 con 0x7f053c0728b0 2024-03-31T15:14:00.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:00.210+0000 7f0543e00640 1 -- 172.21.15.16:0/1418316301 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f051c068380 msgr2=0x7f051c06a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:00.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:00.210+0000 7f0543e00640 1 --2- 172.21.15.16:0/1418316301 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f051c068380 0x7f051c06a840 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f053c166080 tx=0x7f0528009190 comp rx=0 tx=0).stop 2024-03-31T15:14:00.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:00.210+0000 7f0543e00640 1 -- 172.21.15.16:0/1418316301 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f053c0728b0 msgr2=0x7f053c15b940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:00.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:00.210+0000 7f0543e00640 1 --2- 172.21.15.16:0/1418316301 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f053c0728b0 0x7f053c15b940 secure :-1 s=READY pgs=1600 cs=0 l=1 rev1=1 crypto rx=0x7f053000c410 tx=0x7f053000c8d0 comp rx=0 tx=0).stop 2024-03-31T15:14:00.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:00.210+0000 7f0543e00640 1 -- 172.21.15.16:0/1418316301 shutdown_connections 2024-03-31T15:14:00.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:00.210+0000 7f0543e00640 1 --2- 172.21.15.16:0/1418316301 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f051c068380 0x7f051c06a840 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:00.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:00.210+0000 7f0543e00640 1 --2- 172.21.15.16:0/1418316301 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f053c145d10 0x7f053c15b170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:00.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:00.210+0000 7f0543e00640 1 --2- 172.21.15.16:0/1418316301 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f053c0b2200 0x7f053c13c810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:00.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:00.210+0000 7f0543e00640 1 --2- 172.21.15.16:0/1418316301 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f053c0728b0 0x7f053c15b940 unknown :-1 s=CLOSED pgs=1600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:00.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:00.210+0000 7f0543e00640 1 -- 172.21.15.16:0/1418316301 >> 172.21.15.16:0/1418316301 conn(0x7f053c06b620 msgr2=0x7f053c0afd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:14:00.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:00.210+0000 7f0543e00640 1 -- 172.21.15.16:0/1418316301 shutdown_connections 2024-03-31T15:14:00.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:00.210+0000 7f0543e00640 1 -- 172.21.15.16:0/1418316301 wait complete. 2024-03-31T15:14:00.224 INFO:tasks.cephfs_test_runner:test_inotable_sync (tasks.cephfs.test_forward_scrub.TestForwardScrub) ... ok 2024-03-31T15:14:00.224 INFO:tasks.cephfs_test_runner:Starting test: test_orphan_scan (tasks.cephfs.test_forward_scrub.TestForwardScrub) 2024-03-31T15:14:00.224 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-31T15:14:00.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:00.370+0000 7f8565800640 1 Processor -- start 2024-03-31T15:14:00.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:00.370+0000 7f8565800640 1 -- start start 2024-03-31T15:14:00.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:00.370+0000 7f8565800640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f856015b4d0 0x7f85600b5b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:00.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:00.370+0000 7f8565800640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f85600ce8a0 con 0x7f85600cef40 2024-03-31T15:14:00.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:00.374+0000 7f8565800640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f85600709a0 con 0x7f85600b61e0 2024-03-31T15:14:00.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:00.374+0000 7f8565800640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f856006fbe0 con 0x7f856015b4d0 2024-03-31T15:14:00.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:00.374+0000 7f855ea00640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f85600cef40 0x7f85600cf320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:57184/0 (socket says 172.21.15.16:57184) 2024-03-31T15:14:00.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:00.374+0000 7f855ea00640 1 -- 172.21.15.16:0/512196060 learned_addr learned my addr 172.21.15.16:0/512196060 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:14:00.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:00.374+0000 7f8556000640 1 --2- 172.21.15.16:0/512196060 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f856015b4d0 0x7f85600b5b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:00.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:00.374+0000 7f855e000640 1 -- 172.21.15.16:0/512196060 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1354145511 0 0) 0x7f85600ce8a0 con 0x7f85600cef40 2024-03-31T15:14:00.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:00.374+0000 7f855e000640 1 -- 172.21.15.16:0/512196060 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8544003530 con 0x7f85600cef40 2024-03-31T15:14:00.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:00.374+0000 7f855e000640 1 -- 172.21.15.16:0/512196060 <== mon.1 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3801037210 0 0) 0x7f85600709a0 con 0x7f85600b61e0 2024-03-31T15:14:00.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:00.374+0000 7f8556000640 1 -- 172.21.15.16:0/512196060 >> v1:172.21.15.89:6789/0 conn(0x7f85600b61e0 legacy=0x7f85600b42e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:00.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:00.374+0000 7f8556000640 1 -- 172.21.15.16:0/512196060 >> v1:172.21.15.16:6789/0 conn(0x7f85600cef40 legacy=0x7f85600cf320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:00.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:00.374+0000 7f8556000640 1 -- 172.21.15.16:0/512196060 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f856006fec0 con 0x7f856015b4d0 2024-03-31T15:14:00.369 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:00.374+0000 7f8556000640 1 --2- 172.21.15.16:0/512196060 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f856015b4d0 0x7f85600b5b50 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f85500097f0 tx=0x7f855001c4b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=189840c68528a7c2 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:00.369 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:00.378+0000 7f855e000640 1 -- 172.21.15.16:0/512196060 <== mon.2 v2:172.21.15.89:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85500aa610 con 0x7f856015b4d0 2024-03-31T15:14:00.369 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:00.378+0000 7f855e000640 1 -- 172.21.15.16:0/512196060 <== mon.2 v2:172.21.15.89:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f85500af070 con 0x7f856015b4d0 2024-03-31T15:14:00.369 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:00.378+0000 7f855e000640 1 -- 172.21.15.16:0/512196060 <== mon.2 v2:172.21.15.89:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85500aa9d0 con 0x7f856015b4d0 2024-03-31T15:14:00.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:00.378+0000 7f8565800640 1 -- 172.21.15.16:0/512196060 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f856015b4d0 msgr2=0x7f85600b5b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:00.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:00.378+0000 7f8565800640 1 --2- 172.21.15.16:0/512196060 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f856015b4d0 0x7f85600b5b50 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f85500097f0 tx=0x7f855001c4b0 comp rx=0 tx=0).stop 2024-03-31T15:14:00.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:00.378+0000 7f8565800640 1 -- 172.21.15.16:0/512196060 shutdown_connections 2024-03-31T15:14:00.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:00.378+0000 7f8565800640 1 --2- 172.21.15.16:0/512196060 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f856015b4d0 0x7f85600b5b50 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:00.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:00.378+0000 7f8565800640 1 -- 172.21.15.16:0/512196060 >> 172.21.15.16:0/512196060 conn(0x7f85600d1060 msgr2=0x7f85600d1470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:14:00.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:00.378+0000 7f8565800640 1 -- 172.21.15.16:0/512196060 shutdown_connections 2024-03-31T15:14:00.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:00.378+0000 7f8565800640 1 -- 172.21.15.16:0/512196060 wait complete. 2024-03-31T15:14:00.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:00.378+0000 7f8565800640 1 Processor -- start 2024-03-31T15:14:00.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:00.378+0000 7f8565800640 1 -- start start 2024-03-31T15:14:00.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:00.378+0000 7f8565800640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f85600b61e0 0x7f8560073150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:00.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:00.378+0000 7f8565800640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f85600cef40 0x7f8560142940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:00.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:00.378+0000 7f8565800640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f856015b4d0 0x7f85600c8620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:00.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:00.378+0000 7f8565800640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f85600b5380 con 0x7f856015b4d0 2024-03-31T15:14:00.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:00.378+0000 7f8565800640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f85600b5200 con 0x7f85600b61e0 2024-03-31T15:14:00.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:00.378+0000 7f8565800640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f85600b5500 con 0x7f85600cef40 2024-03-31T15:14:00.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:00.378+0000 7f855f400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f856015b4d0 0x7f85600c8620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:00.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:00.378+0000 7f855f400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f856015b4d0 0x7f85600c8620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40656/0 (socket says 172.21.15.16:40656) 2024-03-31T15:14:00.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:00.378+0000 7f855f400640 1 -- 172.21.15.16:0/831663197 learned_addr learned my addr 172.21.15.16:0/831663197 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:14:00.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:00.378+0000 7f855ea00640 1 --2- 172.21.15.16:0/831663197 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f85600b61e0 0x7f8560073150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:00.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:00.378+0000 7f8556000640 1 --2- 172.21.15.16:0/831663197 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f85600cef40 0x7f8560142940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:00.375 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:00.382+0000 7f855f400640 1 -- 172.21.15.16:0/831663197 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f85600cef40 msgr2=0x7f8560142940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:00.375 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:00.382+0000 7f855f400640 1 --2- 172.21.15.16:0/831663197 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f85600cef40 0x7f8560142940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:00.375 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:00.382+0000 7f855f400640 1 -- 172.21.15.16:0/831663197 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f85600b61e0 msgr2=0x7f8560073150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:00.375 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:00.382+0000 7f855f400640 1 --2- 172.21.15.16:0/831663197 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f85600b61e0 0x7f8560073150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:00.375 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:00.382+0000 7f855f400640 1 -- 172.21.15.16:0/831663197 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f856014fd60 con 0x7f856015b4d0 2024-03-31T15:14:00.375 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:00.382+0000 7f855f400640 1 --2- 172.21.15.16:0/831663197 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f856015b4d0 0x7f85600c8620 secure :-1 s=READY pgs=1602 cs=0 l=1 rev1=1 crypto rx=0x7f855800c750 tx=0x7f855800cc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:00.375 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:00.382+0000 7f8557400640 1 -- 172.21.15.16:0/831663197 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f855800ce60 con 0x7f856015b4d0 2024-03-31T15:14:00.375 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:00.382+0000 7f8557400640 1 -- 172.21.15.16:0/831663197 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8558003db0 con 0x7f856015b4d0 2024-03-31T15:14:00.375 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:00.382+0000 7f8557400640 1 -- 172.21.15.16:0/831663197 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85580052b0 con 0x7f856015b4d0 2024-03-31T15:14:00.375 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:00.382+0000 7f8565800640 1 -- 172.21.15.16:0/831663197 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8560156500 con 0x7f856015b4d0 2024-03-31T15:14:00.375 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:00.382+0000 7f8565800640 1 -- 172.21.15.16:0/831663197 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f85600c61c0 con 0x7f856015b4d0 2024-03-31T15:14:00.375 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:00.382+0000 7f8565800640 1 -- 172.21.15.16:0/831663197 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8560156a20 con 0x7f856015b4d0 2024-03-31T15:14:00.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:00.382+0000 7f8557400640 1 -- 172.21.15.16:0/831663197 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f8558005450 con 0x7f856015b4d0 2024-03-31T15:14:00.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:00.386+0000 7f8557400640 1 --2- 172.21.15.16:0/831663197 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f853c068380 0x7f853c06a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:00.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:00.386+0000 7f8557400640 1 -- 172.21.15.16:0/831663197 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 5029+0+0 (secure 0 0 0) 0x7f85580865e0 con 0x7f856015b4d0 2024-03-31T15:14:00.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:00.386+0000 7f855ea00640 1 --2- 172.21.15.16:0/831663197 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f853c068380 0x7f853c06a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:00.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:00.386+0000 7f855ea00640 1 --2- 172.21.15.16:0/831663197 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f853c068380 0x7f853c06a840 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f8548007fd0 tx=0x7f8548009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:00.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:00.390+0000 7f8557400640 1 -- 172.21.15.16:0/831663197 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f856014fd60 con 0x7f856015b4d0 2024-03-31T15:14:00.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:00.554+0000 7f8565800640 1 -- 172.21.15.16:0/831663197 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f8560156c60 con 0x7f856015b4d0 2024-03-31T15:14:00.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:00.554+0000 7f8557400640 1 -- 172.21.15.16:0/831663197 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v100) ==== 74+0+13076 (secure 0 0 0) 0x7f8558056750 con 0x7f856015b4d0 2024-03-31T15:14:00.547 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:14:00.547 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":100,"fsid":"286eb5da-fb47-4550-b290-d9519c13b9a5","created":"2024-03-31T15:03:11.438952+0000","modified":"2024-03-31T15:13:59.206860+0000","last_up_change":"2024-03-31T15:03:22.885535+0000","last_in_change":"2024-03-31T15:03:13.941667+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-31T15:03:22.550787+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"92a66450-ef19-40d8-ac3d-d760b23b0bda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6801","nonce":932712244}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6803","nonce":932712244}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6807","nonce":932712244}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6805","nonce":932712244}]},"public_addr":"172.21.15.16:6801/932712244","cluster_addr":"172.21.15.16:6803/932712244","heartbeat_back_addr":"172.21.15.16:6807/932712244","heartbeat_front_addr":"172.21.15.16:6805/932712244","state":["exists","up"]},{"osd":1,"uuid":"da9d1f2b-1968-4959-b843-faa231988e92","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6810","nonce":1398737533}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6811","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6813","nonce":1398737533}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6821","nonce":1398737533}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6815","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6816","nonce":1398737533}]},"public_addr":"172.21.15.16:6810/1398737533","cluster_addr":"172.21.15.16:6813/1398737533","heartbeat_back_addr":"172.21.15.16:6821/1398737533","heartbeat_front_addr":"172.21.15.16:6816/1398737533","state":["exists","up"]},{"osd":2,"uuid":"01a73662-9a21-4b1f-bcbb-42b9636a4638","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6819","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6822","nonce":3018526646}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6825","nonce":3018526646}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6831","nonce":3018526646}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6827","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6829","nonce":3018526646}]},"public_addr":"172.21.15.16:6822/3018526646","cluster_addr":"172.21.15.16:6825/3018526646","heartbeat_back_addr":"172.21.15.16:6831/3018526646","heartbeat_front_addr":"172.21.15.16:6829/3018526646","state":["exists","up"]},{"osd":3,"uuid":"c7771fd3-e18e-4500-ac38-23cf10ebcd75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6809","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6812","nonce":3360352974}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6817","nonce":3360352974}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6828","nonce":3360352974}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6823","nonce":3360352974}]},"public_addr":"172.21.15.16:6812/3360352974","cluster_addr":"172.21.15.16:6817/3360352974","heartbeat_back_addr":"172.21.15.16:6828/3360352974","heartbeat_front_addr":"172.21.15.16:6823/3360352974","state":["exists","up"]},{"osd":4,"uuid":"88260730-1482-4fbd-a40f-cb1124e9df4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6801","nonce":1856844856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6803","nonce":1856844856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6807","nonce":1856844856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6805","nonce":1856844856}]},"public_addr":"172.21.15.89:6801/1856844856","cluster_addr":"172.21.15.89:6803/1856844856","heartbeat_back_addr":"172.21.15.89:6807/1856844856","heartbeat_front_addr":"172.21.15.89:6805/1856844856","state":["exists","up"]},{"osd":5,"uuid":"4cb2c00c-fae0-4dd8-9d27-ec4f0093ccf7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6814","nonce":50395527}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6817","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6820","nonce":50395527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6831","nonce":50395527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6823","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6826","nonce":50395527}]},"public_addr":"172.21.15.89:6814/50395527","cluster_addr":"172.21.15.89:6820/50395527","heartbeat_back_addr":"172.21.15.89:6831/50395527","heartbeat_front_addr":"172.21.15.89:6826/50395527","state":["exists","up"]},{"osd":6,"uuid":"81ce0c19-2554-4cae-acbe-66a05e9e9b9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6809","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6812","nonce":3470471752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6815","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6818","nonce":3470471752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6827","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6829","nonce":3470471752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6821","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6824","nonce":3470471752}]},"public_addr":"172.21.15.89:6812/3470471752","cluster_addr":"172.21.15.89:6818/3470471752","heartbeat_back_addr":"172.21.15.89:6829/3470471752","heartbeat_front_addr":"172.21.15.89:6824/3470471752","state":["exists","up"]},{"osd":7,"uuid":"85e4313b-d37d-442e-8042-153bf20a847d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6811","nonce":1796778630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6813","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6816","nonce":1796778630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6825","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6828","nonce":1796778630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6819","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6822","nonce":1796778630}]},"public_addr":"172.21.15.89:6811/1796778630","cluster_addr":"172.21.15.89:6816/1796778630","heartbeat_back_addr":"172.21.15.89:6828/1796778630","heartbeat_front_addr":"172.21.15.89:6822/1796778630","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.439088+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.529970+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.680512+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:21.044161+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.582111+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.536558+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.409522+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.590229+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:6835/567808781":"2024-04-01T15:13:50.823751+0000","172.21.15.16:6833/567808781":"2024-04-01T15:13:50.823751+0000","172.21.15.16:6835/490717480":"2024-04-01T15:13:14.080258+0000","172.21.15.16:6833/490717480":"2024-04-01T15:13:14.080258+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-31T15:14:00.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:00.554+0000 7f8565800640 1 -- 172.21.15.16:0/831663197 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f853c068380 msgr2=0x7f853c06a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:00.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:00.554+0000 7f8565800640 1 --2- 172.21.15.16:0/831663197 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f853c068380 0x7f853c06a840 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f8548007fd0 tx=0x7f8548009040 comp rx=0 tx=0).stop 2024-03-31T15:14:00.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:00.554+0000 7f8565800640 1 -- 172.21.15.16:0/831663197 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f856015b4d0 msgr2=0x7f85600c8620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:00.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:00.554+0000 7f8565800640 1 --2- 172.21.15.16:0/831663197 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f856015b4d0 0x7f85600c8620 secure :-1 s=READY pgs=1602 cs=0 l=1 rev1=1 crypto rx=0x7f855800c750 tx=0x7f855800cc10 comp rx=0 tx=0).stop 2024-03-31T15:14:00.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:00.554+0000 7f8565800640 1 -- 172.21.15.16:0/831663197 shutdown_connections 2024-03-31T15:14:00.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:00.554+0000 7f8565800640 1 --2- 172.21.15.16:0/831663197 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f853c068380 0x7f853c06a840 unknown :-1 s=CLOSED pgs=681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:00.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:00.554+0000 7f8565800640 1 --2- 172.21.15.16:0/831663197 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f856015b4d0 0x7f85600c8620 unknown :-1 s=CLOSED pgs=1602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:00.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:00.554+0000 7f8565800640 1 --2- 172.21.15.16:0/831663197 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f85600cef40 0x7f8560142940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:00.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:00.554+0000 7f8565800640 1 --2- 172.21.15.16:0/831663197 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f85600b61e0 0x7f8560073150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:00.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:00.558+0000 7f8565800640 1 -- 172.21.15.16:0/831663197 >> 172.21.15.16:0/831663197 conn(0x7f85600d1060 msgr2=0x7f85600b5950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:14:00.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:00.558+0000 7f8565800640 1 -- 172.21.15.16:0/831663197 shutdown_connections 2024-03-31T15:14:00.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:00.558+0000 7f8565800640 1 -- 172.21.15.16:0/831663197 wait complete. 2024-03-31T15:14:00.568 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-31T15:03:22.550787+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-31T15:14:00.568 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-31T15:14:00.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:00.718+0000 7f6e18400640 1 Processor -- start 2024-03-31T15:14:00.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:00.718+0000 7f6e18400640 1 -- start start 2024-03-31T15:14:00.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:00.718+0000 7f6e18400640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f6e10070030 0x7f6e10143c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:00.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:00.718+0000 7f6e18400640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f6e10141590 0x7f6e10144170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:00.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:00.718+0000 7f6e18400640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6e101446b0 con 0x7f6e101411b0 2024-03-31T15:14:00.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:00.718+0000 7f6e18400640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f6e1015f910 con 0x7f6e10070030 2024-03-31T15:14:00.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:00.718+0000 7f6e18400640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f6e10162910 con 0x7f6e10141590 2024-03-31T15:14:00.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:00.718+0000 7f6e15000640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f6e10070030 0x7f6e10143c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:00.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:00.718+0000 7f6e15000640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f6e10070030 0x7f6e10143c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40666/0 (socket says 172.21.15.16:40666) 2024-03-31T15:14:00.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:00.718+0000 7f6e15000640 1 -- 172.21.15.16:0/1139642539 learned_addr learned my addr 172.21.15.16:0/1139642539 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:14:00.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:00.718+0000 7f6e15a00640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f6e101411b0 0x7f6e1006f920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:57198/0 (socket says 172.21.15.16:57198) 2024-03-31T15:14:00.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:00.722+0000 7f6e15000640 1 -- 172.21.15.16:0/1139642539 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f6e10141590 msgr2=0x7f6e10144170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:14:00.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:00.722+0000 7f6e16400640 1 --2- 172.21.15.16:0/1139642539 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f6e10141590 0x7f6e10144170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:00.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:00.722+0000 7f6e15000640 1 --2- 172.21.15.16:0/1139642539 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f6e10141590 0x7f6e10144170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:00.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:00.722+0000 7f6e15000640 1 -- 172.21.15.16:0/1139642539 >> v1:172.21.15.16:6789/0 conn(0x7f6e101411b0 legacy=0x7f6e1006f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:00.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:00.722+0000 7f6e15000640 1 -- 172.21.15.16:0/1139642539 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6e10144980 con 0x7f6e10070030 2024-03-31T15:14:00.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:00.722+0000 7f6e16400640 1 --2- 172.21.15.16:0/1139642539 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f6e10141590 0x7f6e10144170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:14:00.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:00.722+0000 7f6e15000640 1 --2- 172.21.15.16:0/1139642539 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6e10070030 0x7f6e10143c30 secure :-1 s=READY pgs=1604 cs=0 l=1 rev1=1 crypto rx=0x7f6dfc009870 tx=0x7f6dfc01c530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ed2330218de7e7f server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:00.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:00.722+0000 7f6e0fe00640 1 -- 172.21.15.16:0/1139642539 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6dfc0a95f0 con 0x7f6e10070030 2024-03-31T15:14:00.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:00.722+0000 7f6e0fe00640 1 -- 172.21.15.16:0/1139642539 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6dfc0a9790 con 0x7f6e10070030 2024-03-31T15:14:00.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:00.722+0000 7f6e0fe00640 1 -- 172.21.15.16:0/1139642539 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6dfc0a9b60 con 0x7f6e10070030 2024-03-31T15:14:00.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:00.722+0000 7f6e18400640 1 -- 172.21.15.16:0/1139642539 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6e10070030 msgr2=0x7f6e10143c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:00.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:00.722+0000 7f6e18400640 1 --2- 172.21.15.16:0/1139642539 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6e10070030 0x7f6e10143c30 secure :-1 s=READY pgs=1604 cs=0 l=1 rev1=1 crypto rx=0x7f6dfc009870 tx=0x7f6dfc01c530 comp rx=0 tx=0).stop 2024-03-31T15:14:00.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:00.726+0000 7f6e18400640 1 -- 172.21.15.16:0/1139642539 shutdown_connections 2024-03-31T15:14:00.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:00.726+0000 7f6e18400640 1 --2- 172.21.15.16:0/1139642539 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f6e10141590 0x7f6e10144170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:00.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:00.726+0000 7f6e18400640 1 --2- 172.21.15.16:0/1139642539 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6e10070030 0x7f6e10143c30 unknown :-1 s=CLOSED pgs=1604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:00.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:00.726+0000 7f6e18400640 1 -- 172.21.15.16:0/1139642539 >> 172.21.15.16:0/1139642539 conn(0x7f6e10067930 msgr2=0x7f6e100b3ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:14:00.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:00.726+0000 7f6e18400640 1 -- 172.21.15.16:0/1139642539 shutdown_connections 2024-03-31T15:14:00.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:00.726+0000 7f6e18400640 1 -- 172.21.15.16:0/1139642539 wait complete. 2024-03-31T15:14:00.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:00.726+0000 7f6e18400640 1 Processor -- start 2024-03-31T15:14:00.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:00.726+0000 7f6e18400640 1 -- start start 2024-03-31T15:14:00.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:00.726+0000 7f6e18400640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f6e10070030 0x7f6e10158fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:00.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:00.726+0000 7f6e18400640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6e101411b0 0x7f6e1013e830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:00.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:00.726+0000 7f6e18400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6e10141590 0x7f6e1013c7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:00.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:00.726+0000 7f6e18400640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f6e1015f110 con 0x7f6e10141590 2024-03-31T15:14:00.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:00.726+0000 7f6e18400640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f6e10162d80 con 0x7f6e101411b0 2024-03-31T15:14:00.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:00.726+0000 7f6e18400640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f6e1015f290 con 0x7f6e10070030 2024-03-31T15:14:00.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:00.726+0000 7f6e16400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6e10141590 0x7f6e1013c7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:00.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:00.726+0000 7f6e16400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6e10141590 0x7f6e1013c7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40674/0 (socket says 172.21.15.16:40674) 2024-03-31T15:14:00.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:00.726+0000 7f6e16400640 1 -- 172.21.15.16:0/1243443424 learned_addr learned my addr 172.21.15.16:0/1243443424 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:14:00.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:00.726+0000 7f6e15a00640 1 --2- 172.21.15.16:0/1243443424 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f6e10070030 0x7f6e10158fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:00.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:00.726+0000 7f6e15000640 1 --2- 172.21.15.16:0/1243443424 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6e101411b0 0x7f6e1013e830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:00.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:00.726+0000 7f6e16400640 1 -- 172.21.15.16:0/1243443424 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f6e10070030 msgr2=0x7f6e10158fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:00.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:00.726+0000 7f6e16400640 1 --2- 172.21.15.16:0/1243443424 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f6e10070030 0x7f6e10158fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:00.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:00.726+0000 7f6e16400640 1 -- 172.21.15.16:0/1243443424 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6e101411b0 msgr2=0x7f6e1013e830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:00.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:00.726+0000 7f6e16400640 1 --2- 172.21.15.16:0/1243443424 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6e101411b0 0x7f6e1013e830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:00.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:00.726+0000 7f6e16400640 1 -- 172.21.15.16:0/1243443424 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6e1013ef40 con 0x7f6e10141590 2024-03-31T15:14:00.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:00.726+0000 7f6e16400640 1 --2- 172.21.15.16:0/1243443424 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6e10141590 0x7f6e1013c7d0 secure :-1 s=READY pgs=1605 cs=0 l=1 rev1=1 crypto rx=0x7f6e0800cd80 tx=0x7f6e0800a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:00.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:00.730+0000 7f6e0d600640 1 -- 172.21.15.16:0/1243443424 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e08003ba0 con 0x7f6e10141590 2024-03-31T15:14:00.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:00.730+0000 7f6e0d600640 1 -- 172.21.15.16:0/1243443424 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6e08003d60 con 0x7f6e10141590 2024-03-31T15:14:00.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:00.730+0000 7f6e0d600640 1 -- 172.21.15.16:0/1243443424 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e08004260 con 0x7f6e10141590 2024-03-31T15:14:00.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:00.730+0000 7f6e18400640 1 -- 172.21.15.16:0/1243443424 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6e1013ba10 con 0x7f6e10141590 2024-03-31T15:14:00.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:00.730+0000 7f6e18400640 1 -- 172.21.15.16:0/1243443424 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6e1013bfd0 con 0x7f6e10141590 2024-03-31T15:14:00.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:00.730+0000 7f6e18400640 1 -- 172.21.15.16:0/1243443424 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6e10069950 con 0x7f6e10141590 2024-03-31T15:14:00.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:00.730+0000 7f6e0d600640 1 -- 172.21.15.16:0/1243443424 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f6e08004400 con 0x7f6e10141590 2024-03-31T15:14:00.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:00.730+0000 7f6e0d600640 1 --2- 172.21.15.16:0/1243443424 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f6df00683b0 0x7f6df006a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:00.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:00.730+0000 7f6e0d600640 1 -- 172.21.15.16:0/1243443424 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 5029+0+0 (secure 0 0 0) 0x7f6e08087110 con 0x7f6e10141590 2024-03-31T15:14:00.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:00.730+0000 7f6e15a00640 1 --2- 172.21.15.16:0/1243443424 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f6df00683b0 0x7f6df006a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:00.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:00.734+0000 7f6e15a00640 1 --2- 172.21.15.16:0/1243443424 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f6df00683b0 0x7f6df006a870 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f6e04000f70 tx=0x7f6e04007e90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:00.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:00.734+0000 7f6e0d600640 1 -- 172.21.15.16:0/1243443424 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f6e08014030 con 0x7f6e10141590 2024-03-31T15:14:00.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:00.898+0000 7f6e18400640 1 -- 172.21.15.16:0/1243443424 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f6e100b4a20 con 0x7f6e10141590 2024-03-31T15:14:00.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:00.902+0000 7f6e0d600640 1 -- 172.21.15.16:0/1243443424 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v100) ==== 93+0+10 (secure 0 0 0) 0x7f6e08057280 con 0x7f6e10141590 2024-03-31T15:14:00.894 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2024-03-31T15:14:00.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:00.902+0000 7f6e18400640 1 -- 172.21.15.16:0/1243443424 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f6df00683b0 msgr2=0x7f6df006a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:00.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:00.902+0000 7f6e18400640 1 --2- 172.21.15.16:0/1243443424 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f6df00683b0 0x7f6df006a870 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f6e04000f70 tx=0x7f6e04007e90 comp rx=0 tx=0).stop 2024-03-31T15:14:00.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:00.902+0000 7f6e18400640 1 -- 172.21.15.16:0/1243443424 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6e10141590 msgr2=0x7f6e1013c7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:00.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:00.902+0000 7f6e18400640 1 --2- 172.21.15.16:0/1243443424 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6e10141590 0x7f6e1013c7d0 secure :-1 s=READY pgs=1605 cs=0 l=1 rev1=1 crypto rx=0x7f6e0800cd80 tx=0x7f6e0800a650 comp rx=0 tx=0).stop 2024-03-31T15:14:00.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:00.902+0000 7f6e18400640 1 -- 172.21.15.16:0/1243443424 shutdown_connections 2024-03-31T15:14:00.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:00.902+0000 7f6e18400640 1 --2- 172.21.15.16:0/1243443424 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f6df00683b0 0x7f6df006a870 unknown :-1 s=CLOSED pgs=682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:00.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:00.902+0000 7f6e18400640 1 --2- 172.21.15.16:0/1243443424 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6e10141590 0x7f6e1013c7d0 unknown :-1 s=CLOSED pgs=1605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:00.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:00.902+0000 7f6e18400640 1 --2- 172.21.15.16:0/1243443424 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6e101411b0 0x7f6e1013e830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:00.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:00.902+0000 7f6e18400640 1 --2- 172.21.15.16:0/1243443424 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f6e10070030 0x7f6e10158fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:00.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:00.902+0000 7f6e18400640 1 -- 172.21.15.16:0/1243443424 >> 172.21.15.16:0/1243443424 conn(0x7f6e10067930 msgr2=0x7f6e1015faa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:14:00.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:00.902+0000 7f6e18400640 1 -- 172.21.15.16:0/1243443424 shutdown_connections 2024-03-31T15:14:00.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:00.902+0000 7f6e18400640 1 -- 172.21.15.16:0/1243443424 wait complete. 2024-03-31T15:14:00.915 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_orphan_scan' 2024-03-31T15:14:01.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:01.062+0000 7f240d800640 1 Processor -- start 2024-03-31T15:14:01.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:01.062+0000 7f240d800640 1 -- start start 2024-03-31T15:14:01.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:01.062+0000 7f240d800640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f2408147e70 0x7f240806b620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:01.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:01.062+0000 7f240d800640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f24080d0c50 0x7f2408072fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:01.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:01.062+0000 7f240d800640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f2408073510 0x7f24080cdab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:01.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:01.062+0000 7f240d800640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f24080739a0 con 0x7f24080d0c50 2024-03-31T15:14:01.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:01.062+0000 7f240d800640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f24081621b0 con 0x7f2408147e70 2024-03-31T15:14:01.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:01.062+0000 7f240d800640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f2408162490 con 0x7f2408073510 2024-03-31T15:14:01.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:01.062+0000 7f23ffe00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f24080d0c50 0x7f2408072fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:01.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:01.062+0000 7f23ffe00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f24080d0c50 0x7f2408072fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40688/0 (socket says 172.21.15.16:40688) 2024-03-31T15:14:01.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:01.062+0000 7f23ffe00640 1 -- 172.21.15.16:0/925551849 learned_addr learned my addr 172.21.15.16:0/925551849 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:14:01.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:01.062+0000 7f2406a00640 1 --2- 172.21.15.16:0/925551849 >> v2:172.21.15.89:3300/0 conn(0x7f2408147e70 0x7f240806b620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:01.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:01.062+0000 7f2407400640 1 --2- 172.21.15.16:0/925551849 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f2408073510 0x7f24080cdab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:01.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:01.062+0000 7f23ffe00640 1 -- 172.21.15.16:0/925551849 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f2408073510 msgr2=0x7f24080cdab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:14:01.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:01.062+0000 7f23ffe00640 1 --2- 172.21.15.16:0/925551849 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f2408073510 0x7f24080cdab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:01.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:01.062+0000 7f23ffe00640 1 -- 172.21.15.16:0/925551849 >> v2:172.21.15.89:3300/0 conn(0x7f2408147e70 msgr2=0x7f240806b620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:14:01.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:01.062+0000 7f23ffe00640 1 --2- 172.21.15.16:0/925551849 >> v2:172.21.15.89:3300/0 conn(0x7f2408147e70 0x7f240806b620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:01.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:01.062+0000 7f23ffe00640 1 -- 172.21.15.16:0/925551849 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2408162770 con 0x7f24080d0c50 2024-03-31T15:14:01.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:01.062+0000 7f2406a00640 1 --2- 172.21.15.16:0/925551849 >> v2:172.21.15.89:3300/0 conn(0x7f2408147e70 0x7f240806b620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:14:01.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:01.066+0000 7f23ffe00640 1 --2- 172.21.15.16:0/925551849 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f24080d0c50 0x7f2408072fd0 secure :-1 s=READY pgs=1606 cs=0 l=1 rev1=1 crypto rx=0x7f2408069860 tx=0x7f23f001c260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=453eeb44b6955ced server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:01.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:01.066+0000 7f2406000640 1 -- 172.21.15.16:0/925551849 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23f00b0070 con 0x7f24080d0c50 2024-03-31T15:14:01.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:01.066+0000 7f2406000640 1 -- 172.21.15.16:0/925551849 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f23f00ab600 con 0x7f24080d0c50 2024-03-31T15:14:01.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:01.066+0000 7f2406000640 1 -- 172.21.15.16:0/925551849 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23f00ab9d0 con 0x7f24080d0c50 2024-03-31T15:14:01.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:01.066+0000 7f240d800640 1 -- 172.21.15.16:0/925551849 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f24080d0c50 msgr2=0x7f2408072fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:01.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:01.066+0000 7f240d800640 1 --2- 172.21.15.16:0/925551849 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f24080d0c50 0x7f2408072fd0 secure :-1 s=READY pgs=1606 cs=0 l=1 rev1=1 crypto rx=0x7f2408069860 tx=0x7f23f001c260 comp rx=0 tx=0).stop 2024-03-31T15:14:01.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:01.066+0000 7f240d800640 1 -- 172.21.15.16:0/925551849 shutdown_connections 2024-03-31T15:14:01.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:01.066+0000 7f240d800640 1 --2- 172.21.15.16:0/925551849 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f2408073510 0x7f24080cdab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:01.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:01.066+0000 7f240d800640 1 --2- 172.21.15.16:0/925551849 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f24080d0c50 0x7f2408072fd0 unknown :-1 s=CLOSED pgs=1606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:01.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:01.066+0000 7f240d800640 1 --2- 172.21.15.16:0/925551849 >> v2:172.21.15.89:3300/0 conn(0x7f2408147e70 0x7f240806b620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:01.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:01.066+0000 7f240d800640 1 -- 172.21.15.16:0/925551849 >> 172.21.15.16:0/925551849 conn(0x7f24080d1060 msgr2=0x7f24080d1470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:14:01.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:01.066+0000 7f240d800640 1 -- 172.21.15.16:0/925551849 shutdown_connections 2024-03-31T15:14:01.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:01.066+0000 7f240d800640 1 -- 172.21.15.16:0/925551849 wait complete. 2024-03-31T15:14:01.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:01.070+0000 7f240d800640 1 Processor -- start 2024-03-31T15:14:01.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:01.070+0000 7f240d800640 1 -- start start 2024-03-31T15:14:01.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:01.070+0000 7f240d800640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2408073510 0x7f240815da20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:01.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:01.070+0000 7f240d800640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f24080d0c50 0x7f24080b5020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:01.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:01.070+0000 7f240d800640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f2408147e70 0x7f24080b5560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:01.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:01.070+0000 7f240d800640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f240806bce0 con 0x7f2408073510 2024-03-31T15:14:01.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:01.070+0000 7f240d800640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f240806bb60 con 0x7f24080d0c50 2024-03-31T15:14:01.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:01.070+0000 7f240d800640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f240815eb20 con 0x7f2408147e70 2024-03-31T15:14:01.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:01.070+0000 7f2406a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2408073510 0x7f240815da20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:01.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:01.070+0000 7f2406a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2408073510 0x7f240815da20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40696/0 (socket says 172.21.15.16:40696) 2024-03-31T15:14:01.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:01.070+0000 7f2406a00640 1 -- 172.21.15.16:0/582839621 learned_addr learned my addr 172.21.15.16:0/582839621 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:14:01.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:01.070+0000 7f23ffe00640 1 --2- 172.21.15.16:0/582839621 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f24080d0c50 0x7f24080b5020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:01.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:01.070+0000 7f2407400640 1 --2- 172.21.15.16:0/582839621 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f2408147e70 0x7f24080b5560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:01.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:01.070+0000 7f2406a00640 1 -- 172.21.15.16:0/582839621 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f2408147e70 msgr2=0x7f24080b5560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:01.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:01.070+0000 7f2406a00640 1 --2- 172.21.15.16:0/582839621 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f2408147e70 0x7f24080b5560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:01.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:01.070+0000 7f2406a00640 1 -- 172.21.15.16:0/582839621 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f24080d0c50 msgr2=0x7f24080b5020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:01.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:01.070+0000 7f2406a00640 1 --2- 172.21.15.16:0/582839621 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f24080d0c50 0x7f24080b5020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:01.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:01.070+0000 7f2406a00640 1 -- 172.21.15.16:0/582839621 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f240813c8a0 con 0x7f2408073510 2024-03-31T15:14:01.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:01.070+0000 7f2406a00640 1 --2- 172.21.15.16:0/582839621 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2408073510 0x7f240815da20 secure :-1 s=READY pgs=1607 cs=0 l=1 rev1=1 crypto rx=0x7f23f800d800 tx=0x7f23f800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:01.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:01.070+0000 7f23fea00640 1 -- 172.21.15.16:0/582839621 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23f800b6c0 con 0x7f2408073510 2024-03-31T15:14:01.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:01.070+0000 7f23fea00640 1 -- 172.21.15.16:0/582839621 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f23f8019070 con 0x7f2408073510 2024-03-31T15:14:01.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:01.070+0000 7f23fea00640 1 -- 172.21.15.16:0/582839621 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23f800baa0 con 0x7f2408073510 2024-03-31T15:14:01.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:01.070+0000 7f240d800640 1 -- 172.21.15.16:0/582839621 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f240813cb40 con 0x7f2408073510 2024-03-31T15:14:01.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:01.070+0000 7f240d800640 1 -- 172.21.15.16:0/582839621 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f240813d100 con 0x7f2408073510 2024-03-31T15:14:01.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:01.074+0000 7f240d800640 1 -- 172.21.15.16:0/582839621 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2408069950 con 0x7f2408073510 2024-03-31T15:14:01.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:01.074+0000 7f23fea00640 1 -- 172.21.15.16:0/582839621 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f23f800bc40 con 0x7f2408073510 2024-03-31T15:14:01.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:01.074+0000 7f23fea00640 1 --2- 172.21.15.16:0/582839621 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f23e40683b0 0x7f23e406a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:01.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:01.074+0000 7f23fea00640 1 -- 172.21.15.16:0/582839621 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 5029+0+0 (secure 0 0 0) 0x7f23f80869f0 con 0x7f2408073510 2024-03-31T15:14:01.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:01.074+0000 7f23ffe00640 1 --2- 172.21.15.16:0/582839621 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f23e40683b0 0x7f23e406a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:01.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:01.074+0000 7f23ffe00640 1 --2- 172.21.15.16:0/582839621 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f23e40683b0 0x7f23e406a870 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f23f00ad000 tx=0x7f23f0008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:01.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:01.078+0000 7f23fea00640 1 -- 172.21.15.16:0/582839621 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f23f80572e0 con 0x7f2408073510 2024-03-31T15:14:01.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:01.302+0000 7f240d800640 1 -- 172.21.15.16:0/582839621 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_orphan_scan"]} v 0) -- 0x7f24080d1bf0 con 0x7f2408073510 2024-03-31T15:14:02.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:02.218+0000 7f23fea00640 1 -- 172.21.15.16:0/582839621 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_orphan_scan"]}]=0 v484) ==== 147+0+0 (secure 0 0 0) 0x7f23f8056be0 con 0x7f2408073510 2024-03-31T15:14:02.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:02.218+0000 7f240d800640 1 -- 172.21.15.16:0/582839621 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f23e40683b0 msgr2=0x7f23e406a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:02.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:02.218+0000 7f240d800640 1 --2- 172.21.15.16:0/582839621 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f23e40683b0 0x7f23e406a870 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f23f00ad000 tx=0x7f23f0008040 comp rx=0 tx=0).stop 2024-03-31T15:14:02.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:02.218+0000 7f240d800640 1 -- 172.21.15.16:0/582839621 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2408073510 msgr2=0x7f240815da20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:02.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:02.218+0000 7f240d800640 1 --2- 172.21.15.16:0/582839621 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2408073510 0x7f240815da20 secure :-1 s=READY pgs=1607 cs=0 l=1 rev1=1 crypto rx=0x7f23f800d800 tx=0x7f23f800dcc0 comp rx=0 tx=0).stop 2024-03-31T15:14:02.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:02.218+0000 7f240d800640 1 -- 172.21.15.16:0/582839621 shutdown_connections 2024-03-31T15:14:02.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:02.218+0000 7f240d800640 1 --2- 172.21.15.16:0/582839621 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f23e40683b0 0x7f23e406a870 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:02.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:02.218+0000 7f240d800640 1 --2- 172.21.15.16:0/582839621 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f2408147e70 0x7f24080b5560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:02.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:02.218+0000 7f240d800640 1 --2- 172.21.15.16:0/582839621 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f24080d0c50 0x7f24080b5020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:02.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:02.218+0000 7f240d800640 1 --2- 172.21.15.16:0/582839621 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2408073510 0x7f240815da20 unknown :-1 s=CLOSED pgs=1607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:02.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:02.218+0000 7f240d800640 1 -- 172.21.15.16:0/582839621 >> 172.21.15.16:0/582839621 conn(0x7f24080d1060 msgr2=0x7f2408072ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:14:02.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:02.218+0000 7f240d800640 1 -- 172.21.15.16:0/582839621 shutdown_connections 2024-03-31T15:14:02.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:02.218+0000 7f240d800640 1 -- 172.21.15.16:0/582839621 wait complete. 2024-03-31T15:14:02.237 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-03-31T15:14:02.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:02.394+0000 7fc4ee200640 1 Processor -- start 2024-03-31T15:14:02.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:02.394+0000 7fc4ee200640 1 -- start start 2024-03-31T15:14:02.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:02.394+0000 7fc4ee200640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fc4e813dff0 0x7fc4e806d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:02.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:02.394+0000 7fc4ee200640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fc4e813e3d0 0x7fc4e806da10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:02.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:02.394+0000 7fc4ee200640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fc4e80cc9d0 0x7fc4e80cce30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:02.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:02.394+0000 7fc4ee200640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7fc4e815f730 con 0x7fc4e813e3d0 2024-03-31T15:14:02.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:02.394+0000 7fc4ee200640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7fc4e815fbc0 con 0x7fc4e80cc9d0 2024-03-31T15:14:02.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:02.394+0000 7fc4ee200640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fc4e815ee00 con 0x7fc4e813dff0 2024-03-31T15:14:02.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:02.394+0000 7fc4e6a00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fc4e813e3d0 0x7fc4e806da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:02.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:02.394+0000 7fc4e6a00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fc4e813e3d0 0x7fc4e806da10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40708/0 (socket says 172.21.15.16:40708) 2024-03-31T15:14:02.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:02.394+0000 7fc4e6a00640 1 -- 172.21.15.16:0/1536140996 learned_addr learned my addr 172.21.15.16:0/1536140996 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:14:02.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:02.394+0000 7fc4e7e00640 1 --2- 172.21.15.16:0/1536140996 >> v2:172.21.15.89:3300/0 conn(0x7fc4e80cc9d0 0x7fc4e80cce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:02.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:02.394+0000 7fc4e7400640 1 --2- 172.21.15.16:0/1536140996 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fc4e813dff0 0x7fc4e806d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:02.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:02.394+0000 7fc4e6a00640 1 -- 172.21.15.16:0/1536140996 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fc4e813dff0 msgr2=0x7fc4e806d4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:14:02.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:02.394+0000 7fc4e6a00640 1 --2- 172.21.15.16:0/1536140996 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fc4e813dff0 0x7fc4e806d4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:02.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:02.394+0000 7fc4e6a00640 1 -- 172.21.15.16:0/1536140996 >> v2:172.21.15.89:3300/0 conn(0x7fc4e80cc9d0 msgr2=0x7fc4e80cce30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:14:02.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:02.394+0000 7fc4e6a00640 1 --2- 172.21.15.16:0/1536140996 >> v2:172.21.15.89:3300/0 conn(0x7fc4e80cc9d0 0x7fc4e80cce30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:02.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:02.394+0000 7fc4e6a00640 1 -- 172.21.15.16:0/1536140996 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4e815f0e0 con 0x7fc4e813e3d0 2024-03-31T15:14:02.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:02.398+0000 7fc4e6a00640 1 --2- 172.21.15.16:0/1536140996 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc4e813e3d0 0x7fc4e806da10 secure :-1 s=READY pgs=1608 cs=0 l=1 rev1=1 crypto rx=0x7fc4e80cc980 tx=0x7fc4d401cef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=466fe02b9c0e5dec server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:02.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:02.398+0000 7fc4e6000640 1 -- 172.21.15.16:0/1536140996 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4d40b2070 con 0x7fc4e813e3d0 2024-03-31T15:14:02.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:02.398+0000 7fc4e6000640 1 -- 172.21.15.16:0/1536140996 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc4d40a9070 con 0x7fc4e813e3d0 2024-03-31T15:14:02.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:02.398+0000 7fc4e6000640 1 -- 172.21.15.16:0/1536140996 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4d40ad880 con 0x7fc4e813e3d0 2024-03-31T15:14:02.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:02.398+0000 7fc4ee200640 1 -- 172.21.15.16:0/1536140996 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc4e813e3d0 msgr2=0x7fc4e806da10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:02.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:02.398+0000 7fc4ee200640 1 --2- 172.21.15.16:0/1536140996 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc4e813e3d0 0x7fc4e806da10 secure :-1 s=READY pgs=1608 cs=0 l=1 rev1=1 crypto rx=0x7fc4e80cc980 tx=0x7fc4d401cef0 comp rx=0 tx=0).stop 2024-03-31T15:14:02.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:02.398+0000 7fc4ee200640 1 -- 172.21.15.16:0/1536140996 shutdown_connections 2024-03-31T15:14:02.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:02.398+0000 7fc4ee200640 1 --2- 172.21.15.16:0/1536140996 >> v2:172.21.15.89:3300/0 conn(0x7fc4e80cc9d0 0x7fc4e80cce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:02.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:02.398+0000 7fc4ee200640 1 --2- 172.21.15.16:0/1536140996 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc4e813e3d0 0x7fc4e806da10 unknown :-1 s=CLOSED pgs=1608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:02.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:02.398+0000 7fc4ee200640 1 --2- 172.21.15.16:0/1536140996 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fc4e813dff0 0x7fc4e806d4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:02.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:02.398+0000 7fc4ee200640 1 -- 172.21.15.16:0/1536140996 >> 172.21.15.16:0/1536140996 conn(0x7fc4e806ee40 msgr2=0x7fc4e806b620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:14:02.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:02.398+0000 7fc4ee200640 1 -- 172.21.15.16:0/1536140996 shutdown_connections 2024-03-31T15:14:02.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:02.398+0000 7fc4ee200640 1 -- 172.21.15.16:0/1536140996 wait complete. 2024-03-31T15:14:02.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:02.402+0000 7fc4ee200640 1 Processor -- start 2024-03-31T15:14:02.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:02.402+0000 7fc4ee200640 1 -- start start 2024-03-31T15:14:02.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:02.402+0000 7fc4ee200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc4e80cc9d0 0x7fc4e80b3560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:02.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:02.402+0000 7fc4ee200640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc4e813dff0 0x7fc4e80cf550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:02.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:02.402+0000 7fc4ee200640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fc4e813e3d0 0x7fc4e80b4c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:02.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:02.402+0000 7fc4ee200640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fc4e8073200 con 0x7fc4e80cc9d0 2024-03-31T15:14:02.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:02.402+0000 7fc4ee200640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fc4e8073080 con 0x7fc4e813dff0 2024-03-31T15:14:02.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:02.402+0000 7fc4ee200640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fc4e8073380 con 0x7fc4e813e3d0 2024-03-31T15:14:02.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:02.402+0000 7fc4e7400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc4e80cc9d0 0x7fc4e80b3560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:02.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:02.402+0000 7fc4e7400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc4e80cc9d0 0x7fc4e80b3560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40726/0 (socket says 172.21.15.16:40726) 2024-03-31T15:14:02.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:02.402+0000 7fc4e7400640 1 -- 172.21.15.16:0/1473060656 learned_addr learned my addr 172.21.15.16:0/1473060656 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:14:02.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:02.402+0000 7fc4e6a00640 1 --2- 172.21.15.16:0/1473060656 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc4e813dff0 0x7fc4e80cf550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:02.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:02.402+0000 7fc4e7e00640 1 --2- 172.21.15.16:0/1473060656 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fc4e813e3d0 0x7fc4e80b4c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:02.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:02.402+0000 7fc4e7400640 1 -- 172.21.15.16:0/1473060656 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fc4e813e3d0 msgr2=0x7fc4e80b4c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:02.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:02.402+0000 7fc4e7400640 1 --2- 172.21.15.16:0/1473060656 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fc4e813e3d0 0x7fc4e80b4c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:02.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:02.402+0000 7fc4e7400640 1 -- 172.21.15.16:0/1473060656 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc4e813dff0 msgr2=0x7fc4e80cf550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:02.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:02.402+0000 7fc4e7400640 1 --2- 172.21.15.16:0/1473060656 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc4e813dff0 0x7fc4e80cf550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:02.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:02.402+0000 7fc4e7400640 1 -- 172.21.15.16:0/1473060656 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4e80b3bf0 con 0x7fc4e80cc9d0 2024-03-31T15:14:02.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:02.402+0000 7fc4e7400640 1 --2- 172.21.15.16:0/1473060656 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc4e80cc9d0 0x7fc4e80b3560 secure :-1 s=READY pgs=1609 cs=0 l=1 rev1=1 crypto rx=0x7fc4d8007ef0 tx=0x7fc4d800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:02.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:02.402+0000 7fc4df400640 1 -- 172.21.15.16:0/1473060656 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4d8014620 con 0x7fc4e80cc9d0 2024-03-31T15:14:02.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:02.402+0000 7fc4df400640 1 -- 172.21.15.16:0/1473060656 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc4d8019070 con 0x7fc4e80cc9d0 2024-03-31T15:14:02.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:02.402+0000 7fc4df400640 1 -- 172.21.15.16:0/1473060656 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4d8014a00 con 0x7fc4e80cc9d0 2024-03-31T15:14:02.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:02.402+0000 7fc4ee200640 1 -- 172.21.15.16:0/1473060656 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc4e80b3ed0 con 0x7fc4e80cc9d0 2024-03-31T15:14:02.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:02.402+0000 7fc4ee200640 1 -- 172.21.15.16:0/1473060656 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc4e80b4490 con 0x7fc4e80cc9d0 2024-03-31T15:14:02.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:02.402+0000 7fc4ee200640 1 -- 172.21.15.16:0/1473060656 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc4e8069950 con 0x7fc4e80cc9d0 2024-03-31T15:14:02.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:02.406+0000 7fc4df400640 1 -- 172.21.15.16:0/1473060656 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fc4d8014ba0 con 0x7fc4e80cc9d0 2024-03-31T15:14:02.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:02.406+0000 7fc4df400640 1 --2- 172.21.15.16:0/1473060656 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fc4c8068380 0x7fc4c806a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:02.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:02.406+0000 7fc4df400640 1 -- 172.21.15.16:0/1473060656 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 5029+0+0 (secure 0 0 0) 0x7fc4d8086fd0 con 0x7fc4e80cc9d0 2024-03-31T15:14:02.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:02.406+0000 7fc4e6a00640 1 --2- 172.21.15.16:0/1473060656 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fc4c8068380 0x7fc4c806a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:02.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:02.406+0000 7fc4e6a00640 1 --2- 172.21.15.16:0/1473060656 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fc4c8068380 0x7fc4c806a840 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7fc4d40af000 tx=0x7fc4d4004740 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:02.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:02.410+0000 7fc4df400640 1 -- 172.21.15.16:0/1473060656 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fc4d8057790 con 0x7fc4e80cc9d0 2024-03-31T15:14:02.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:02.574+0000 7fc4ee200640 1 -- 172.21.15.16:0/1473060656 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7fc4e815e560 con 0x7fc4e80cc9d0 2024-03-31T15:14:02.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:02.578+0000 7fc4df400640 1 -- 172.21.15.16:0/1473060656 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v22)=0 v22) ==== 122+0+0 (secure 0 0 0) 0x7fc4d8057090 con 0x7fc4e80cc9d0 2024-03-31T15:14:02.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:02.582+0000 7fc4ee200640 1 -- 172.21.15.16:0/1473060656 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fc4c8068380 msgr2=0x7fc4c806a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:02.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:02.582+0000 7fc4ee200640 1 --2- 172.21.15.16:0/1473060656 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fc4c8068380 0x7fc4c806a840 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7fc4d40af000 tx=0x7fc4d4004740 comp rx=0 tx=0).stop 2024-03-31T15:14:02.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:02.582+0000 7fc4ee200640 1 -- 172.21.15.16:0/1473060656 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc4e80cc9d0 msgr2=0x7fc4e80b3560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:02.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:02.582+0000 7fc4ee200640 1 --2- 172.21.15.16:0/1473060656 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc4e80cc9d0 0x7fc4e80b3560 secure :-1 s=READY pgs=1609 cs=0 l=1 rev1=1 crypto rx=0x7fc4d8007ef0 tx=0x7fc4d800e4b0 comp rx=0 tx=0).stop 2024-03-31T15:14:02.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:02.582+0000 7fc4ee200640 1 -- 172.21.15.16:0/1473060656 shutdown_connections 2024-03-31T15:14:02.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:02.582+0000 7fc4ee200640 1 --2- 172.21.15.16:0/1473060656 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fc4c8068380 0x7fc4c806a840 unknown :-1 s=CLOSED pgs=684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:02.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:02.582+0000 7fc4ee200640 1 --2- 172.21.15.16:0/1473060656 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fc4e813e3d0 0x7fc4e80b4c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:02.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:02.582+0000 7fc4ee200640 1 --2- 172.21.15.16:0/1473060656 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc4e813dff0 0x7fc4e80cf550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:02.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:02.582+0000 7fc4ee200640 1 --2- 172.21.15.16:0/1473060656 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc4e80cc9d0 0x7fc4e80b3560 unknown :-1 s=CLOSED pgs=1609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:02.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:02.582+0000 7fc4ee200640 1 -- 172.21.15.16:0/1473060656 >> 172.21.15.16:0/1473060656 conn(0x7fc4e806ee40 msgr2=0x7fc4e80b0120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:14:02.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:02.582+0000 7fc4ee200640 1 -- 172.21.15.16:0/1473060656 shutdown_connections 2024-03-31T15:14:02.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:02.582+0000 7fc4ee200640 1 -- 172.21.15.16:0/1473060656 wait complete. 2024-03-31T15:14:02.594 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-31T15:14:02.594 DEBUG:teuthology.orchestra.run.smithi016:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-31T15:14:02.664 DEBUG:teuthology.orchestra.run.smithi089:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-31T15:14:02.780 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-31T15:14:02.780 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-03-31T15:14:02.780 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-31T15:14:02.789 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16357708k,nr_inodes=4089427,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275996k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=6539 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275992k,nr_inodes=818998,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275996k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-31T15:14:02.789 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-03-31T15:14:02.789 DEBUG:teuthology.orchestra.run.smithi089:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-31T15:14:02.799 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16354452k,nr_inodes=4088613,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275344k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=901 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275340k,nr_inodes=818835,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275344k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-31T15:14:02.799 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-03-31T15:14:02.799 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-31T15:14:02.841 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16357708k,nr_inodes=4089427,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275996k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=6539 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275992k,nr_inodes=818998,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275996k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-31T15:14:02.841 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-03-31T15:14:02.841 DEBUG:teuthology.orchestra.run.smithi089:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-31T15:14:02.850 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16354452k,nr_inodes=4088613,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275344k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=901 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275340k,nr_inodes=818835,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275344k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-31T15:14:02.850 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-03-31T15:14:02.850 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-31T15:14:03.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:03.030+0000 7f38f0400640 1 Processor -- start 2024-03-31T15:14:03.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:03.030+0000 7f38f0400640 1 -- start start 2024-03-31T15:14:03.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:03.034+0000 7f38f0400640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f38e8145d10 0x7f38e80cf790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:03.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:03.034+0000 7f38f0400640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f38e80728b0 0x7f38e80b1cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:03.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:03.034+0000 7f38f0400640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f38e8160e70 con 0x7f38e80b2200 2024-03-31T15:14:03.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:03.034+0000 7f38f0400640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f38e81602f0 con 0x7f38e80728b0 2024-03-31T15:14:03.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:03.034+0000 7f38f0400640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f38e81605d0 con 0x7f38e8145d10 2024-03-31T15:14:03.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:03.034+0000 7f38ed000640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f38e80728b0 0x7f38e80b1cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:03.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:03.034+0000 7f38ee400640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f38e80b2200 0x7f38e815cd50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:57210/0 (socket says 172.21.15.16:57210) 2024-03-31T15:14:03.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:03.034+0000 7f38ee400640 1 -- 172.21.15.16:0/515749583 learned_addr learned my addr 172.21.15.16:0/515749583 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:14:03.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:03.034+0000 7f38eda00640 1 --2- 172.21.15.16:0/515749583 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f38e8145d10 0x7f38e80cf790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:03.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:03.034+0000 7f38ed000640 1 -- 172.21.15.16:0/515749583 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f38e8145d10 msgr2=0x7f38e80cf790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:14:03.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:03.034+0000 7f38ed000640 1 --2- 172.21.15.16:0/515749583 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f38e8145d10 0x7f38e80cf790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:03.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:03.034+0000 7f38ed000640 1 -- 172.21.15.16:0/515749583 >> v1:172.21.15.16:6789/0 conn(0x7f38e80b2200 legacy=0x7f38e815cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:03.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:03.034+0000 7f38e7e00640 1 -- 172.21.15.16:0/515749583 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4162659824 0 0) 0x7f38e8160e70 con 0x7f38e80b2200 2024-03-31T15:14:03.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:03.034+0000 7f38ed000640 1 -- 172.21.15.16:0/515749583 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38e80cfcd0 con 0x7f38e80728b0 2024-03-31T15:14:03.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:03.038+0000 7f38ed000640 1 --2- 172.21.15.16:0/515749583 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f38e80728b0 0x7f38e80b1cc0 secure :-1 s=READY pgs=1611 cs=0 l=1 rev1=1 crypto rx=0x7f38d40097c0 tx=0x7f38d401c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b20c1611a6f355a server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:03.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:03.038+0000 7f38e7e00640 1 -- 172.21.15.16:0/515749583 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38d40b0070 con 0x7f38e80728b0 2024-03-31T15:14:03.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:03.038+0000 7f38e7e00640 1 -- 172.21.15.16:0/515749583 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f38d40ab600 con 0x7f38e80728b0 2024-03-31T15:14:03.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:03.038+0000 7f38e7e00640 1 -- 172.21.15.16:0/515749583 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38d40ab9d0 con 0x7f38e80728b0 2024-03-31T15:14:03.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:03.038+0000 7f38f0400640 1 -- 172.21.15.16:0/515749583 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f38e80728b0 msgr2=0x7f38e80b1cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:03.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:03.038+0000 7f38f0400640 1 --2- 172.21.15.16:0/515749583 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f38e80728b0 0x7f38e80b1cc0 secure :-1 s=READY pgs=1611 cs=0 l=1 rev1=1 crypto rx=0x7f38d40097c0 tx=0x7f38d401c4a0 comp rx=0 tx=0).stop 2024-03-31T15:14:03.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:03.038+0000 7f38f0400640 1 -- 172.21.15.16:0/515749583 shutdown_connections 2024-03-31T15:14:03.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:03.038+0000 7f38f0400640 1 --2- 172.21.15.16:0/515749583 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f38e80728b0 0x7f38e80b1cc0 unknown :-1 s=CLOSED pgs=1611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:03.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:03.038+0000 7f38f0400640 1 --2- 172.21.15.16:0/515749583 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f38e8145d10 0x7f38e80cf790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:03.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:03.038+0000 7f38f0400640 1 -- 172.21.15.16:0/515749583 >> 172.21.15.16:0/515749583 conn(0x7f38e806b620 msgr2=0x7f38e806ba30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:14:03.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:03.038+0000 7f38f0400640 1 -- 172.21.15.16:0/515749583 shutdown_connections 2024-03-31T15:14:03.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:03.038+0000 7f38f0400640 1 -- 172.21.15.16:0/515749583 wait complete. 2024-03-31T15:14:03.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:03.038+0000 7f38f0400640 1 Processor -- start 2024-03-31T15:14:03.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:03.038+0000 7f38f0400640 1 -- start start 2024-03-31T15:14:03.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:03.042+0000 7f38f0400640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f38e80728b0 0x7f38e80b4450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:03.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:03.042+0000 7f38f0400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f38e80b2200 0x7f38e81591a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:03.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:03.042+0000 7f38f0400640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f38e8145d10 0x7f38e81596e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:03.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:03.042+0000 7f38f0400640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f38e815d870 con 0x7f38e80b2200 2024-03-31T15:14:03.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:03.042+0000 7f38f0400640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f38e80b2650 con 0x7f38e8145d10 2024-03-31T15:14:03.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:03.042+0000 7f38f0400640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f38e80cf350 con 0x7f38e80728b0 2024-03-31T15:14:03.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:03.042+0000 7f38ed000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f38e80b2200 0x7f38e81591a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:03.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:03.042+0000 7f38ed000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f38e80b2200 0x7f38e81591a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40742/0 (socket says 172.21.15.16:40742) 2024-03-31T15:14:03.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:03.042+0000 7f38ed000640 1 -- 172.21.15.16:0/2745806549 learned_addr learned my addr 172.21.15.16:0/2745806549 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:14:03.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:03.042+0000 7f38eda00640 1 --2- 172.21.15.16:0/2745806549 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f38e80728b0 0x7f38e80b4450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:03.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:03.042+0000 7f38ee400640 1 --2- 172.21.15.16:0/2745806549 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f38e8145d10 0x7f38e81596e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:03.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:03.042+0000 7f38ed000640 1 -- 172.21.15.16:0/2745806549 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f38e80728b0 msgr2=0x7f38e80b4450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:03.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:03.042+0000 7f38ed000640 1 --2- 172.21.15.16:0/2745806549 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f38e80728b0 0x7f38e80b4450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:03.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:03.042+0000 7f38ed000640 1 -- 172.21.15.16:0/2745806549 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f38e8145d10 msgr2=0x7f38e81596e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:03.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:03.042+0000 7f38ed000640 1 --2- 172.21.15.16:0/2745806549 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f38e8145d10 0x7f38e81596e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:03.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:03.042+0000 7f38ed000640 1 -- 172.21.15.16:0/2745806549 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38e80b11d0 con 0x7f38e80b2200 2024-03-31T15:14:03.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:03.042+0000 7f38ee400640 1 --2- 172.21.15.16:0/2745806549 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f38e8145d10 0x7f38e81596e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:14:03.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:03.042+0000 7f38ed000640 1 --2- 172.21.15.16:0/2745806549 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f38e80b2200 0x7f38e81591a0 secure :-1 s=READY pgs=1612 cs=0 l=1 rev1=1 crypto rx=0x7f38d40ae040 tx=0x7f38d4019240 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:03.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:03.042+0000 7f38e5600640 1 -- 172.21.15.16:0/2745806549 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38d40ba070 con 0x7f38e80b2200 2024-03-31T15:14:03.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:03.042+0000 7f38e5600640 1 -- 172.21.15.16:0/2745806549 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f38d40b43f0 con 0x7f38e80b2200 2024-03-31T15:14:03.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:03.042+0000 7f38e5600640 1 -- 172.21.15.16:0/2745806549 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38d40b0040 con 0x7f38e80b2200 2024-03-31T15:14:03.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:03.042+0000 7f38f0400640 1 -- 172.21.15.16:0/2745806549 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f38e80b1490 con 0x7f38e80b2200 2024-03-31T15:14:03.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:03.042+0000 7f38f0400640 1 -- 172.21.15.16:0/2745806549 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f38e80b1a50 con 0x7f38e80b2200 2024-03-31T15:14:03.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:03.042+0000 7f38f0400640 1 -- 172.21.15.16:0/2745806549 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f38e8069950 con 0x7f38e80b2200 2024-03-31T15:14:03.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:03.042+0000 7f38e5600640 1 -- 172.21.15.16:0/2745806549 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f38d40bd070 con 0x7f38e80b2200 2024-03-31T15:14:03.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:03.046+0000 7f38e5600640 1 --2- 172.21.15.16:0/2745806549 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f38c8068380 0x7f38c806a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:03.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:03.046+0000 7f38e5600640 1 -- 172.21.15.16:0/2745806549 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 5029+0+0 (secure 0 0 0) 0x7f38d411f590 con 0x7f38e80b2200 2024-03-31T15:14:03.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:03.046+0000 7f38eda00640 1 --2- 172.21.15.16:0/2745806549 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f38c8068380 0x7f38c806a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:03.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:03.046+0000 7f38eda00640 1 --2- 172.21.15.16:0/2745806549 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f38c8068380 0x7f38c806a840 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f38e81fae80 tx=0x7f38dc009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:03.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:03.050+0000 7f38e5600640 1 -- 172.21.15.16:0/2745806549 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f38d4016090 con 0x7f38e80b2200 2024-03-31T15:14:03.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:03.246+0000 7f38f0400640 1 -- 172.21.15.16:0/2745806549 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f38e813fb50 con 0x7f38e80b2200 2024-03-31T15:14:03.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:03.250+0000 7f38e5600640 1 -- 172.21.15.16:0/2745806549 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 93 v93) ==== 94+0+3869 (secure 0 0 0) 0x7f38d40ef7a0 con 0x7f38e80b2200 2024-03-31T15:14:03.241 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:14:03.242 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":93,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7139,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6836/3798928623","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":3798928623},{"type":"v1","addr":"172.21.15.89:6836","nonce":3798928623}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":78},{"gid":7142,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/3074283817","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6835","nonce":3074283817},{"type":"v1","addr":"172.21.15.89:6837","nonce":3074283817}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":78},{"gid":8071,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6834/2110189998","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":2110189998},{"type":"v1","addr":"172.21.15.16:6834","nonce":2110189998}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":78},{"gid":9027,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/2938692325","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6833","nonce":2938692325},{"type":"v1","addr":"172.21.15.16:6835","nonce":2938692325}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":91}],"filesystems":[]} 2024-03-31T15:14:03.242 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 93 2024-03-31T15:14:03.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:03.250+0000 7f38f0400640 1 -- 172.21.15.16:0/2745806549 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f38c8068380 msgr2=0x7f38c806a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:03.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:03.250+0000 7f38f0400640 1 --2- 172.21.15.16:0/2745806549 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f38c8068380 0x7f38c806a840 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f38e81fae80 tx=0x7f38dc009190 comp rx=0 tx=0).stop 2024-03-31T15:14:03.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:03.250+0000 7f38f0400640 1 -- 172.21.15.16:0/2745806549 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f38e80b2200 msgr2=0x7f38e81591a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:03.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:03.250+0000 7f38f0400640 1 --2- 172.21.15.16:0/2745806549 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f38e80b2200 0x7f38e81591a0 secure :-1 s=READY pgs=1612 cs=0 l=1 rev1=1 crypto rx=0x7f38d40ae040 tx=0x7f38d4019240 comp rx=0 tx=0).stop 2024-03-31T15:14:03.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:03.250+0000 7f38f0400640 1 -- 172.21.15.16:0/2745806549 shutdown_connections 2024-03-31T15:14:03.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:03.250+0000 7f38f0400640 1 --2- 172.21.15.16:0/2745806549 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f38c8068380 0x7f38c806a840 unknown :-1 s=CLOSED pgs=685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:03.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:03.250+0000 7f38f0400640 1 --2- 172.21.15.16:0/2745806549 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f38e8145d10 0x7f38e81596e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:03.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:03.250+0000 7f38f0400640 1 --2- 172.21.15.16:0/2745806549 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f38e80b2200 0x7f38e81591a0 unknown :-1 s=CLOSED pgs=1612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:03.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:03.250+0000 7f38f0400640 1 --2- 172.21.15.16:0/2745806549 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f38e80728b0 0x7f38e80b4450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:03.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:03.250+0000 7f38f0400640 1 -- 172.21.15.16:0/2745806549 >> 172.21.15.16:0/2745806549 conn(0x7f38e806b620 msgr2=0x7f38e8072d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:14:03.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:03.250+0000 7f38f0400640 1 -- 172.21.15.16:0/2745806549 shutdown_connections 2024-03-31T15:14:03.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:03.250+0000 7f38f0400640 1 -- 172.21.15.16:0/2745806549 wait complete. 2024-03-31T15:14:03.263 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-31T15:14:03.263 INFO:tasks.ceph.mds.a:Stopping old one... 2024-03-31T15:14:03.263 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-31T15:14:03.263 INFO:teuthology.orchestra.run:waiting for 300 2024-03-31T15:14:03.264 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-31T15:14:03.264 INFO:tasks.ceph.mds.c:Stopping old one... 2024-03-31T15:14:03.264 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-31T15:14:03.264 INFO:teuthology.orchestra.run:waiting for 300 2024-03-31T15:14:03.264 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-31T15:14:03.264 INFO:tasks.ceph.mds.b:Stopping old one... 2024-03-31T15:14:03.264 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-31T15:14:03.265 INFO:teuthology.orchestra.run:waiting for 300 2024-03-31T15:14:03.265 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-31T15:14:03.265 INFO:tasks.ceph.mds.d:Stopping old one... 2024-03-31T15:14:03.265 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-31T15:14:03.265 INFO:teuthology.orchestra.run:waiting for 300 2024-03-31T15:14:03.278 INFO:tasks.ceph.mds.b:Stopped 2024-03-31T15:14:03.278 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-31T15:14:03.279 INFO:tasks.ceph.mds.a:Stopped 2024-03-31T15:14:03.280 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-31T15:14:03.280 INFO:tasks.ceph.mds.c:Stopped 2024-03-31T15:14:03.280 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-31T15:14:03.282 INFO:tasks.ceph.mds.d:Stopped 2024-03-31T15:14:03.282 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-31T15:14:03.327 INFO:tasks.ceph.mds.b:Started 2024-03-31T15:14:03.327 DEBUG:teuthology.parallel:result is None 2024-03-31T15:14:03.328 INFO:tasks.ceph.mds.a:Started 2024-03-31T15:14:03.328 DEBUG:teuthology.parallel:result is None 2024-03-31T15:14:03.329 INFO:tasks.ceph.mds.c:Started 2024-03-31T15:14:03.329 DEBUG:teuthology.parallel:result is None 2024-03-31T15:14:03.330 INFO:tasks.ceph.mds.d:Started 2024-03-31T15:14:03.330 DEBUG:teuthology.parallel:result is None 2024-03-31T15:14:03.330 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-03-31T15:14:03.414 INFO:tasks.ceph.mds.c.smithi016.stdout:starting mds.c at 2024-03-31T15:14:03.414 INFO:tasks.ceph.mds.a.smithi016.stdout:starting mds.a at 2024-03-31T15:14:03.419 INFO:tasks.ceph.mds.b.smithi089.stdout:starting mds.b at 2024-03-31T15:14:03.419 INFO:tasks.ceph.mds.d.smithi089.stdout:starting mds.d at 2024-03-31T15:14:03.420 INFO:tasks.ceph.mds.b.smithi089.stderr:2024-03-31T15:14:03.431+0000 7f4ed1c00640 -1 PosixStack listen unable to listen on v2:0.0.0.0:6834/0: (98) Address already in use 2024-03-31T15:14:03.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:03.482+0000 7f6309e00640 1 Processor -- start 2024-03-31T15:14:03.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:03.482+0000 7f6309e00640 1 -- start start 2024-03-31T15:14:03.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:03.482+0000 7f6309e00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f6304145d10 0x7f63040cf790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:03.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:03.482+0000 7f6309e00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f63040b23d0 0x7f630415cd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:03.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:03.482+0000 7f6309e00640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6304160e70 con 0x7f63040728b0 2024-03-31T15:14:03.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:03.482+0000 7f6309e00640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f63041602f0 con 0x7f6304145d10 2024-03-31T15:14:03.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:03.482+0000 7f6309e00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f63041605d0 con 0x7f63040b23d0 2024-03-31T15:14:03.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:03.486+0000 7f62fa000640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f63040728b0 0x7f63040b1cc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:57234/0 (socket says 172.21.15.16:57234) 2024-03-31T15:14:03.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:03.486+0000 7f62fa000640 1 -- 172.21.15.16:0/4078452194 learned_addr learned my addr 172.21.15.16:0/4078452194 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:14:03.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:03.486+0000 7f6302a00640 1 --2- 172.21.15.16:0/4078452194 >> v2:172.21.15.89:3300/0 conn(0x7f6304145d10 0x7f63040cf790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:03.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:03.486+0000 7f6303400640 1 --2- 172.21.15.16:0/4078452194 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f63040b23d0 0x7f630415cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:03.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:03.486+0000 7f6302a00640 1 -- 172.21.15.16:0/4078452194 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f63040b23d0 msgr2=0x7f630415cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:14:03.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:03.486+0000 7f6302000640 1 -- 172.21.15.16:0/4078452194 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 512607985 0 0) 0x7f6304160e70 con 0x7f63040728b0 2024-03-31T15:14:03.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:03.486+0000 7f6302a00640 1 --2- 172.21.15.16:0/4078452194 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f63040b23d0 0x7f630415cd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:03.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:03.486+0000 7f6302a00640 1 -- 172.21.15.16:0/4078452194 >> v1:172.21.15.16:6789/0 conn(0x7f63040728b0 legacy=0x7f63040b1cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:03.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:03.486+0000 7f6302a00640 1 -- 172.21.15.16:0/4078452194 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f63040cfcd0 con 0x7f6304145d10 2024-03-31T15:14:03.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:03.486+0000 7f6303400640 1 --2- 172.21.15.16:0/4078452194 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f63040b23d0 0x7f630415cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:14:03.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:03.486+0000 7f6302a00640 1 --2- 172.21.15.16:0/4078452194 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6304145d10 0x7f63040cf790 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f62f4009d90 tx=0x7f62f401c970 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5353319fa87252ac server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:03.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:03.486+0000 7f6302000640 1 -- 172.21.15.16:0/4078452194 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62f40b2070 con 0x7f6304145d10 2024-03-31T15:14:03.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:03.486+0000 7f6302000640 1 -- 172.21.15.16:0/4078452194 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f62f40a5830 con 0x7f6304145d10 2024-03-31T15:14:03.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:03.486+0000 7f6302000640 1 -- 172.21.15.16:0/4078452194 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62f40a5c00 con 0x7f6304145d10 2024-03-31T15:14:03.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:03.490+0000 7f6309e00640 1 -- 172.21.15.16:0/4078452194 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6304145d10 msgr2=0x7f63040cf790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:03.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:03.490+0000 7f6309e00640 1 --2- 172.21.15.16:0/4078452194 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6304145d10 0x7f63040cf790 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f62f4009d90 tx=0x7f62f401c970 comp rx=0 tx=0).stop 2024-03-31T15:14:03.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:03.490+0000 7f6309e00640 1 -- 172.21.15.16:0/4078452194 shutdown_connections 2024-03-31T15:14:03.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:03.490+0000 7f6309e00640 1 --2- 172.21.15.16:0/4078452194 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f63040b23d0 0x7f630415cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:03.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:03.490+0000 7f6309e00640 1 --2- 172.21.15.16:0/4078452194 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6304145d10 0x7f63040cf790 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:03.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:03.490+0000 7f6309e00640 1 -- 172.21.15.16:0/4078452194 >> 172.21.15.16:0/4078452194 conn(0x7f630406b620 msgr2=0x7f630406ba30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:14:03.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:03.490+0000 7f6309e00640 1 -- 172.21.15.16:0/4078452194 shutdown_connections 2024-03-31T15:14:03.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:03.490+0000 7f6309e00640 1 -- 172.21.15.16:0/4078452194 wait complete. 2024-03-31T15:14:03.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:03.490+0000 7f6309e00640 1 Processor -- start 2024-03-31T15:14:03.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:03.490+0000 7f6309e00640 1 -- start start 2024-03-31T15:14:03.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:03.490+0000 7f6309e00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f63040728b0 0x7f6304153570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:03.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:03.490+0000 7f6309e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f63040b23d0 0x7f630413b730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:03.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:03.490+0000 7f6309e00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f6304145d10 0x7f6304157050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:03.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:03.490+0000 7f6309e00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f63040cf4d0 con 0x7f63040b23d0 2024-03-31T15:14:03.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:03.490+0000 7f6309e00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f63040cf350 con 0x7f63040728b0 2024-03-31T15:14:03.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:03.490+0000 7f6309e00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f6304072d00 con 0x7f6304145d10 2024-03-31T15:14:03.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:03.490+0000 7f62fa000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f63040b23d0 0x7f630413b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:03.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:03.490+0000 7f62fa000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f63040b23d0 0x7f630413b730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40772/0 (socket says 172.21.15.16:40772) 2024-03-31T15:14:03.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:03.490+0000 7f62fa000640 1 -- 172.21.15.16:0/79771062 learned_addr learned my addr 172.21.15.16:0/79771062 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:14:03.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:03.490+0000 7f6302a00640 1 --2- 172.21.15.16:0/79771062 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f63040728b0 0x7f6304153570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:03.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:03.490+0000 7f6303400640 1 --2- 172.21.15.16:0/79771062 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f6304145d10 0x7f6304157050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:03.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:03.490+0000 7f62fa000640 1 -- 172.21.15.16:0/79771062 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f6304145d10 msgr2=0x7f6304157050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:03.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:03.490+0000 7f62fa000640 1 --2- 172.21.15.16:0/79771062 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f6304145d10 0x7f6304157050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:03.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:03.490+0000 7f62fa000640 1 -- 172.21.15.16:0/79771062 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f63040728b0 msgr2=0x7f6304153570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:03.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:03.490+0000 7f62fa000640 1 --2- 172.21.15.16:0/79771062 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f63040728b0 0x7f6304153570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:03.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:03.490+0000 7f62fa000640 1 -- 172.21.15.16:0/79771062 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f630413bc70 con 0x7f63040b23d0 2024-03-31T15:14:03.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:03.494+0000 7f62fa000640 1 --2- 172.21.15.16:0/79771062 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f63040b23d0 0x7f630413b730 secure :-1 s=READY pgs=1619 cs=0 l=1 rev1=1 crypto rx=0x7f62ec00c480 tx=0x7f62ec00c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:03.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:03.494+0000 7f62fbe00640 1 -- 172.21.15.16:0/79771062 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62ec0074e0 con 0x7f63040b23d0 2024-03-31T15:14:03.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:03.494+0000 7f62fbe00640 1 -- 172.21.15.16:0/79771062 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f62ec0047f0 con 0x7f63040b23d0 2024-03-31T15:14:03.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:03.494+0000 7f62fbe00640 1 -- 172.21.15.16:0/79771062 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62ec004be0 con 0x7f63040b23d0 2024-03-31T15:14:03.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:03.494+0000 7f6309e00640 1 -- 172.21.15.16:0/79771062 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6304153c50 con 0x7f63040b23d0 2024-03-31T15:14:03.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:03.494+0000 7f6309e00640 1 -- 172.21.15.16:0/79771062 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6304154210 con 0x7f63040b23d0 2024-03-31T15:14:03.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:03.494+0000 7f6309e00640 1 -- 172.21.15.16:0/79771062 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6304069950 con 0x7f63040b23d0 2024-03-31T15:14:03.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:03.494+0000 7f62fbe00640 1 -- 172.21.15.16:0/79771062 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f62ec011070 con 0x7f63040b23d0 2024-03-31T15:14:03.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:03.494+0000 7f62fbe00640 1 --2- 172.21.15.16:0/79771062 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f62e00683b0 0x7f62e006a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:03.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:03.494+0000 7f62fbe00640 1 -- 172.21.15.16:0/79771062 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 5029+0+0 (secure 0 0 0) 0x7f62ec0861f0 con 0x7f63040b23d0 2024-03-31T15:14:03.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:03.494+0000 7f6302a00640 1 --2- 172.21.15.16:0/79771062 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f62e00683b0 0x7f62e006a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:03.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:03.498+0000 7f6302a00640 1 --2- 172.21.15.16:0/79771062 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f62e00683b0 0x7f62e006a870 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f62f40af000 tx=0x7f62f40a7070 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:03.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:03.498+0000 7f62fbe00640 1 -- 172.21.15.16:0/79771062 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f630413bc70 con 0x7f63040b23d0 2024-03-31T15:14:03.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:03.678+0000 7f6309e00640 1 -- 172.21.15.16:0/79771062 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) -- 0x7f63040b1640 con 0x7f63040b23d0 2024-03-31T15:14:04.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:04.602+0000 7f62fbe00640 1 -- 172.21.15.16:0/79771062 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v105) ==== 97+0+0 (secure 0 0 0) 0x7f62ec0569b0 con 0x7f63040b23d0 2024-03-31T15:14:04.597 INFO:teuthology.orchestra.run.smithi016.stderr: removed all blocklist entries 2024-03-31T15:14:04.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:04.606+0000 7f6309e00640 1 -- 172.21.15.16:0/79771062 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f62e00683b0 msgr2=0x7f62e006a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:04.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:04.606+0000 7f6309e00640 1 --2- 172.21.15.16:0/79771062 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f62e00683b0 0x7f62e006a870 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f62f40af000 tx=0x7f62f40a7070 comp rx=0 tx=0).stop 2024-03-31T15:14:04.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:04.606+0000 7f6309e00640 1 -- 172.21.15.16:0/79771062 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f63040b23d0 msgr2=0x7f630413b730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:04.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:04.606+0000 7f6309e00640 1 --2- 172.21.15.16:0/79771062 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f63040b23d0 0x7f630413b730 secure :-1 s=READY pgs=1619 cs=0 l=1 rev1=1 crypto rx=0x7f62ec00c480 tx=0x7f62ec00c880 comp rx=0 tx=0).stop 2024-03-31T15:14:04.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:04.606+0000 7f6309e00640 1 -- 172.21.15.16:0/79771062 shutdown_connections 2024-03-31T15:14:04.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:04.606+0000 7f6309e00640 1 --2- 172.21.15.16:0/79771062 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f62e00683b0 0x7f62e006a870 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:04.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:04.606+0000 7f6309e00640 1 --2- 172.21.15.16:0/79771062 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f6304145d10 0x7f6304157050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:04.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:04.606+0000 7f6309e00640 1 --2- 172.21.15.16:0/79771062 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f63040b23d0 0x7f630413b730 unknown :-1 s=CLOSED pgs=1619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:04.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:04.606+0000 7f6309e00640 1 --2- 172.21.15.16:0/79771062 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f63040728b0 0x7f6304153570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:04.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:04.606+0000 7f6309e00640 1 -- 172.21.15.16:0/79771062 >> 172.21.15.16:0/79771062 conn(0x7f630406b620 msgr2=0x7f63040b1b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:14:04.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:04.606+0000 7f6309e00640 1 -- 172.21.15.16:0/79771062 shutdown_connections 2024-03-31T15:14:04.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:04.606+0000 7f6309e00640 1 -- 172.21.15.16:0/79771062 wait complete. 2024-03-31T15:14:04.624 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-03-31T15:14:04.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:04.774+0000 7f1f81600640 1 Processor -- start 2024-03-31T15:14:04.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:04.774+0000 7f1f81600640 1 -- start start 2024-03-31T15:14:04.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:04.778+0000 7f1f81600640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f1f7c15b3d0 0x7f1f7c0b1cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:04.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:04.778+0000 7f1f81600640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f1f7c0ce2f0 0x7f1f7c0b2200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:04.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:04.778+0000 7f1f81600640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1f7c15f3c0 con 0x7f1f7c15eee0 2024-03-31T15:14:04.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:04.778+0000 7f1f81600640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f1f7c15b7b0 con 0x7f1f7c15b3d0 2024-03-31T15:14:04.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:04.778+0000 7f1f81600640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f1f7c15f6a0 con 0x7f1f7c0ce2f0 2024-03-31T15:14:04.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:04.778+0000 7f1f7aa00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f1f7c15b3d0 0x7f1f7c0b1cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:04.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:04.778+0000 7f1f72000640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f1f7c15eee0 0x7f1f7c0cc9c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:57250/0 (socket says 172.21.15.16:57250) 2024-03-31T15:14:04.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:04.778+0000 7f1f72000640 1 -- 172.21.15.16:0/2306934681 learned_addr learned my addr 172.21.15.16:0/2306934681 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:14:04.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:04.778+0000 7f1f7b400640 1 --2- 172.21.15.16:0/2306934681 >> v2:172.21.15.89:3300/0 conn(0x7f1f7c0ce2f0 0x7f1f7c0b2200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:04.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:04.778+0000 7f1f7aa00640 1 -- 172.21.15.16:0/2306934681 >> v2:172.21.15.89:3300/0 conn(0x7f1f7c0ce2f0 msgr2=0x7f1f7c0b2200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:14:04.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:04.778+0000 7f1f7aa00640 1 --2- 172.21.15.16:0/2306934681 >> v2:172.21.15.89:3300/0 conn(0x7f1f7c0ce2f0 0x7f1f7c0b2200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:04.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:04.778+0000 7f1f7aa00640 1 -- 172.21.15.16:0/2306934681 >> v1:172.21.15.16:6789/0 conn(0x7f1f7c15eee0 legacy=0x7f1f7c0cc9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:04.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:04.778+0000 7f1f7a000640 1 -- 172.21.15.16:0/2306934681 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 846468481 0 0) 0x7f1f7c15f3c0 con 0x7f1f7c15eee0 2024-03-31T15:14:04.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:04.778+0000 7f1f7aa00640 1 -- 172.21.15.16:0/2306934681 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f7c15f980 con 0x7f1f7c15b3d0 2024-03-31T15:14:04.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:04.782+0000 7f1f7aa00640 1 --2- 172.21.15.16:0/2306934681 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1f7c15b3d0 0x7f1f7c0b1cc0 secure :-1 s=READY pgs=1621 cs=0 l=1 rev1=1 crypto rx=0x7f1f7c06f8a0 tx=0x7f1f6c01c450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b2c7e4ed77f079ba server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:04.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:04.782+0000 7f1f7a000640 1 -- 172.21.15.16:0/2306934681 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f6c0ab610 con 0x7f1f7c15b3d0 2024-03-31T15:14:04.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:04.782+0000 7f1f7a000640 1 -- 172.21.15.16:0/2306934681 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1f6c0b0070 con 0x7f1f7c15b3d0 2024-03-31T15:14:04.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:04.782+0000 7f1f7a000640 1 -- 172.21.15.16:0/2306934681 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f6c0ab9d0 con 0x7f1f7c15b3d0 2024-03-31T15:14:04.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:04.782+0000 7f1f81600640 1 -- 172.21.15.16:0/2306934681 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1f7c15b3d0 msgr2=0x7f1f7c0b1cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:04.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:04.782+0000 7f1f81600640 1 --2- 172.21.15.16:0/2306934681 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1f7c15b3d0 0x7f1f7c0b1cc0 secure :-1 s=READY pgs=1621 cs=0 l=1 rev1=1 crypto rx=0x7f1f7c06f8a0 tx=0x7f1f6c01c450 comp rx=0 tx=0).stop 2024-03-31T15:14:04.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:04.782+0000 7f1f81600640 1 -- 172.21.15.16:0/2306934681 shutdown_connections 2024-03-31T15:14:04.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:04.782+0000 7f1f81600640 1 --2- 172.21.15.16:0/2306934681 >> v2:172.21.15.89:3300/0 conn(0x7f1f7c0ce2f0 0x7f1f7c0b2200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:04.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:04.782+0000 7f1f81600640 1 --2- 172.21.15.16:0/2306934681 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1f7c15b3d0 0x7f1f7c0b1cc0 unknown :-1 s=CLOSED pgs=1621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:04.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:04.782+0000 7f1f81600640 1 -- 172.21.15.16:0/2306934681 >> 172.21.15.16:0/2306934681 conn(0x7f1f7c06b620 msgr2=0x7f1f7c06ba30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:14:04.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:04.782+0000 7f1f81600640 1 -- 172.21.15.16:0/2306934681 shutdown_connections 2024-03-31T15:14:04.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:04.782+0000 7f1f81600640 1 -- 172.21.15.16:0/2306934681 wait complete. 2024-03-31T15:14:04.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:04.782+0000 7f1f81600640 1 Processor -- start 2024-03-31T15:14:04.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:04.782+0000 7f1f81600640 1 -- start start 2024-03-31T15:14:04.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:04.782+0000 7f1f81600640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1f7c0ce2f0 0x7f1f7c1438c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:04.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:04.782+0000 7f1f81600640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1f7c15b3d0 0x7f1f7c15de30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:04.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:04.782+0000 7f1f81600640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f1f7c15eee0 0x7f1f7c15e370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:04.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:04.782+0000 7f1f81600640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f1f7c166600 con 0x7f1f7c15b3d0 2024-03-31T15:14:04.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:04.782+0000 7f1f81600640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f1f7c166480 con 0x7f1f7c0ce2f0 2024-03-31T15:14:04.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:04.782+0000 7f1f81600640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f1f7c166780 con 0x7f1f7c15eee0 2024-03-31T15:14:04.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:04.782+0000 7f1f72000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1f7c15b3d0 0x7f1f7c15de30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:04.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:04.782+0000 7f1f72000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1f7c15b3d0 0x7f1f7c15de30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40796/0 (socket says 172.21.15.16:40796) 2024-03-31T15:14:04.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:04.782+0000 7f1f72000640 1 -- 172.21.15.16:0/337849014 learned_addr learned my addr 172.21.15.16:0/337849014 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:14:04.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:04.782+0000 7f1f7aa00640 1 --2- 172.21.15.16:0/337849014 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1f7c0ce2f0 0x7f1f7c1438c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:04.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:04.786+0000 7f1f7b400640 1 --2- 172.21.15.16:0/337849014 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f1f7c15eee0 0x7f1f7c15e370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:04.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:04.786+0000 7f1f72000640 1 -- 172.21.15.16:0/337849014 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f1f7c15eee0 msgr2=0x7f1f7c15e370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:04.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:04.786+0000 7f1f72000640 1 --2- 172.21.15.16:0/337849014 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f1f7c15eee0 0x7f1f7c15e370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:04.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:04.786+0000 7f1f72000640 1 -- 172.21.15.16:0/337849014 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1f7c0ce2f0 msgr2=0x7f1f7c1438c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:04.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:04.786+0000 7f1f72000640 1 --2- 172.21.15.16:0/337849014 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1f7c0ce2f0 0x7f1f7c1438c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:04.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:04.786+0000 7f1f72000640 1 -- 172.21.15.16:0/337849014 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f7c167d60 con 0x7f1f7c15b3d0 2024-03-31T15:14:04.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:04.786+0000 7f1f7b400640 1 --2- 172.21.15.16:0/337849014 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f1f7c15eee0 0x7f1f7c15e370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:14:04.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:04.786+0000 7f1f72000640 1 --2- 172.21.15.16:0/337849014 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1f7c15b3d0 0x7f1f7c15de30 secure :-1 s=READY pgs=1622 cs=0 l=1 rev1=1 crypto rx=0x7f1f6400c480 tx=0x7f1f6400c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:04.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:04.786+0000 7f1f73400640 1 -- 172.21.15.16:0/337849014 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f64018070 con 0x7f1f7c15b3d0 2024-03-31T15:14:04.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:04.786+0000 7f1f73400640 1 -- 172.21.15.16:0/337849014 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1f64007500 con 0x7f1f7c15b3d0 2024-03-31T15:14:04.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:04.786+0000 7f1f73400640 1 -- 172.21.15.16:0/337849014 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f64004a30 con 0x7f1f7c15b3d0 2024-03-31T15:14:04.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:04.786+0000 7f1f81600640 1 -- 172.21.15.16:0/337849014 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1f7c168040 con 0x7f1f7c15b3d0 2024-03-31T15:14:04.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:04.786+0000 7f1f81600640 1 -- 172.21.15.16:0/337849014 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1f7c143e00 con 0x7f1f7c15b3d0 2024-03-31T15:14:04.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:04.786+0000 7f1f81600640 1 -- 172.21.15.16:0/337849014 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1f7c069950 con 0x7f1f7c15b3d0 2024-03-31T15:14:04.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:04.786+0000 7f1f73400640 1 -- 172.21.15.16:0/337849014 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f1f6401b070 con 0x7f1f7c15b3d0 2024-03-31T15:14:04.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:04.790+0000 7f1f73400640 1 --2- 172.21.15.16:0/337849014 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f1f58068380 0x7f1f5806a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:04.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:04.790+0000 7f1f73400640 1 -- 172.21.15.16:0/337849014 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 4857+0+0 (secure 0 0 0) 0x7f1f64085960 con 0x7f1f7c15b3d0 2024-03-31T15:14:04.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:04.790+0000 7f1f7aa00640 1 --2- 172.21.15.16:0/337849014 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f1f58068380 0x7f1f5806a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:04.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:04.790+0000 7f1f7aa00640 1 --2- 172.21.15.16:0/337849014 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f1f58068380 0x7f1f5806a840 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f1f6c0af000 tx=0x7f1f6c0a7000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:04.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:04.794+0000 7f1f73400640 1 -- 172.21.15.16:0/337849014 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f1f64055c10 con 0x7f1f7c15b3d0 2024-03-31T15:14:05.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:05.010+0000 7f1f81600640 1 -- 172.21.15.16:0/337849014 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) -- 0x7f1f7c1684f0 con 0x7f1f7c15b3d0 2024-03-31T15:14:05.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:05.014+0000 7f1f73400640 1 -- 172.21.15.16:0/337849014 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v19) ==== 80+0+5304 (secure 0 0 0) 0x7f1f64059f10 con 0x7f1f7c15b3d0 2024-03-31T15:14:05.005 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:14:05.006 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-03-31T15:14:05.006 INFO:teuthology.orchestra.run.smithi016.stdout: "auth_dump": [ 2024-03-31T15:14:05.006 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-31T15:14:05.006 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "mds.a", 2024-03-31T15:14:05.006 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAOewlmhp1ANRAAYS9zo+p4tLU1YK0ySOZE/Q==", 2024-03-31T15:14:05.006 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2024-03-31T15:14:05.006 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2024-03-31T15:14:05.006 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow *", 2024-03-31T15:14:05.006 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow *", 2024-03-31T15:14:05.006 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2024-03-31T15:14:05.006 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-31T15:14:05.006 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-31T15:14:05.006 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-31T15:14:05.007 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "mds.b", 2024-03-31T15:14:05.007 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAPewlmXxupBhAAtpkt93aJOFqhc7GXLIlkBg==", 2024-03-31T15:14:05.007 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2024-03-31T15:14:05.007 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2024-03-31T15:14:05.007 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow *", 2024-03-31T15:14:05.007 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow *", 2024-03-31T15:14:05.007 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2024-03-31T15:14:05.007 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-31T15:14:05.007 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-31T15:14:05.007 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-31T15:14:05.007 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "mds.c", 2024-03-31T15:14:05.007 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAPewlmjaieARAA32teg02vE/Q8RtqLtnArMw==", 2024-03-31T15:14:05.007 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2024-03-31T15:14:05.007 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2024-03-31T15:14:05.008 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow *", 2024-03-31T15:14:05.008 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow *", 2024-03-31T15:14:05.008 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2024-03-31T15:14:05.008 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-31T15:14:05.008 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-31T15:14:05.008 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-31T15:14:05.008 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "mds.d", 2024-03-31T15:14:05.008 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAPewlmo+4PDhAAmJD0siZB3LjWkvdMq1d8aw==", 2024-03-31T15:14:05.008 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2024-03-31T15:14:05.008 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2024-03-31T15:14:05.008 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow *", 2024-03-31T15:14:05.008 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow *", 2024-03-31T15:14:05.008 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2024-03-31T15:14:05.008 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-31T15:14:05.009 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-31T15:14:05.009 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-31T15:14:05.009 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.0", 2024-03-31T15:14:05.009 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAQewlmVP6kLhAAwFr4jwQgz1N3oIUZuJvmzw==", 2024-03-31T15:14:05.009 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2024-03-31T15:14:05.009 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2024-03-31T15:14:05.009 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2024-03-31T15:14:05.009 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2024-03-31T15:14:05.009 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-31T15:14:05.009 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-31T15:14:05.009 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-31T15:14:05.009 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.1", 2024-03-31T15:14:05.009 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQATewlmBh0gNhAAR6LiuFypwA7d3SwKlWuI/Q==", 2024-03-31T15:14:05.009 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2024-03-31T15:14:05.010 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2024-03-31T15:14:05.010 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2024-03-31T15:14:05.010 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2024-03-31T15:14:05.010 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-31T15:14:05.010 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-31T15:14:05.010 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-31T15:14:05.010 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.2", 2024-03-31T15:14:05.010 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAXewlmtQwhARAACbzc7Woo9BoVFPfK3Pns+g==", 2024-03-31T15:14:05.010 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2024-03-31T15:14:05.010 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2024-03-31T15:14:05.010 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2024-03-31T15:14:05.010 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2024-03-31T15:14:05.010 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-31T15:14:05.010 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-31T15:14:05.011 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-31T15:14:05.011 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.3", 2024-03-31T15:14:05.011 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAaewlmmjWABhAA/hwvARNb51nd5vsnYVD/sg==", 2024-03-31T15:14:05.011 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2024-03-31T15:14:05.011 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2024-03-31T15:14:05.011 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2024-03-31T15:14:05.011 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2024-03-31T15:14:05.011 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-31T15:14:05.011 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-31T15:14:05.011 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-31T15:14:05.011 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.4", 2024-03-31T15:14:05.011 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAgewlmby07CxAAChxE/qPuPboIlg3fDxSOeA==", 2024-03-31T15:14:05.011 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2024-03-31T15:14:05.011 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2024-03-31T15:14:05.012 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2024-03-31T15:14:05.012 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2024-03-31T15:14:05.012 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-31T15:14:05.012 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-31T15:14:05.012 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-31T15:14:05.012 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.5", 2024-03-31T15:14:05.012 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAjewlmlD8yExAA9pgP0heBywrgqc39iU8rpQ==", 2024-03-31T15:14:05.012 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2024-03-31T15:14:05.012 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2024-03-31T15:14:05.012 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2024-03-31T15:14:05.012 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2024-03-31T15:14:05.012 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-31T15:14:05.012 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-31T15:14:05.013 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-31T15:14:05.013 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.6", 2024-03-31T15:14:05.013 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAmewlmzAsTGBAALMjVcQ08sIiTDFaBcq8L6A==", 2024-03-31T15:14:05.013 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2024-03-31T15:14:05.013 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2024-03-31T15:14:05.013 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2024-03-31T15:14:05.013 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2024-03-31T15:14:05.013 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-31T15:14:05.013 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-31T15:14:05.013 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-31T15:14:05.013 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.7", 2024-03-31T15:14:05.013 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQApewlmb+CEHRAAXE+iBumDsb2c4zH3C8f6YA==", 2024-03-31T15:14:05.014 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2024-03-31T15:14:05.014 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2024-03-31T15:14:05.014 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2024-03-31T15:14:05.014 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2024-03-31T15:14:05.014 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-31T15:14:05.014 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-31T15:14:05.014 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-31T15:14:05.014 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "client.0", 2024-03-31T15:14:05.014 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAPewlmlMIDEhAAF0Ll1HcrWHmH6xDKq2Qq9A==", 2024-03-31T15:14:05.015 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2024-03-31T15:14:05.015 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2024-03-31T15:14:05.015 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow r", 2024-03-31T15:14:05.015 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-03-31T15:14:05.015 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-31T15:14:05.015 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-31T15:14:05.015 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-31T15:14:05.015 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "client.1", 2024-03-31T15:14:05.015 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAPewlmdsvqGxAAaVdwWwri0IE/2sg7psilWw==", 2024-03-31T15:14:05.015 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2024-03-31T15:14:05.015 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2024-03-31T15:14:05.015 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow r", 2024-03-31T15:14:05.015 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow rw", 2024-03-31T15:14:05.015 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow rwx" 2024-03-31T15:14:05.016 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-31T15:14:05.016 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-31T15:14:05.016 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-31T15:14:05.016 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "client.2", 2024-03-31T15:14:05.016 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAPewlmQapPGBAAbwXSlCi+zyQ9118WyAJmrQ==", 2024-03-31T15:14:05.016 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2024-03-31T15:14:05.016 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2024-03-31T15:14:05.016 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow r", 2024-03-31T15:14:05.016 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow rw", 2024-03-31T15:14:05.016 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow rwx" 2024-03-31T15:14:05.016 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-31T15:14:05.016 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-31T15:14:05.016 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-31T15:14:05.016 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "client.3", 2024-03-31T15:14:05.016 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAPewlmCw7qIRAAEeIjF85JbyfJ6d3FtMOVhQ==", 2024-03-31T15:14:05.017 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2024-03-31T15:14:05.017 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2024-03-31T15:14:05.017 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow r", 2024-03-31T15:14:05.017 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow rw", 2024-03-31T15:14:05.017 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow rwx" 2024-03-31T15:14:05.017 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-31T15:14:05.017 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-31T15:14:05.017 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-31T15:14:05.017 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "client.admin", 2024-03-31T15:14:05.017 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAOewlmXyPZGRAAtZOheiYhzGxdeJGqqyGxTw==", 2024-03-31T15:14:05.017 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2024-03-31T15:14:05.017 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow *", 2024-03-31T15:14:05.017 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow *", 2024-03-31T15:14:05.017 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow *", 2024-03-31T15:14:05.017 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2024-03-31T15:14:05.018 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-31T15:14:05.018 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-31T15:14:05.018 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-31T15:14:05.018 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "mgr.x", 2024-03-31T15:14:05.018 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAOewlm7UkKMRAA8hqgeV86mTCkRcxdXNdWmw==", 2024-03-31T15:14:05.018 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2024-03-31T15:14:05.018 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow *", 2024-03-31T15:14:05.018 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile mgr", 2024-03-31T15:14:05.018 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2024-03-31T15:14:05.018 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-31T15:14:05.018 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-31T15:14:05.018 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-31T15:14:05.018 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "mgr.y", 2024-03-31T15:14:05.019 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAOewlmq0cpLRAAKTgh04tRaHrpMhxdwgvXYA==", 2024-03-31T15:14:05.019 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2024-03-31T15:14:05.019 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow *", 2024-03-31T15:14:05.019 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile mgr", 2024-03-31T15:14:05.019 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2024-03-31T15:14:05.019 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-31T15:14:05.019 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-31T15:14:05.019 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-03-31T15:14:05.019 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-03-31T15:14:05.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:05.014+0000 7f1f81600640 1 -- 172.21.15.16:0/337849014 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f1f58068380 msgr2=0x7f1f5806a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:05.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:05.014+0000 7f1f81600640 1 --2- 172.21.15.16:0/337849014 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f1f58068380 0x7f1f5806a840 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f1f6c0af000 tx=0x7f1f6c0a7000 comp rx=0 tx=0).stop 2024-03-31T15:14:05.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:05.014+0000 7f1f81600640 1 -- 172.21.15.16:0/337849014 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1f7c15b3d0 msgr2=0x7f1f7c15de30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:05.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:05.014+0000 7f1f81600640 1 --2- 172.21.15.16:0/337849014 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1f7c15b3d0 0x7f1f7c15de30 secure :-1 s=READY pgs=1622 cs=0 l=1 rev1=1 crypto rx=0x7f1f6400c480 tx=0x7f1f6400c940 comp rx=0 tx=0).stop 2024-03-31T15:14:05.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:05.014+0000 7f1f81600640 1 -- 172.21.15.16:0/337849014 shutdown_connections 2024-03-31T15:14:05.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:05.014+0000 7f1f81600640 1 --2- 172.21.15.16:0/337849014 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f1f58068380 0x7f1f5806a840 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:05.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:05.014+0000 7f1f81600640 1 --2- 172.21.15.16:0/337849014 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f1f7c15eee0 0x7f1f7c15e370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:05.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:05.014+0000 7f1f81600640 1 --2- 172.21.15.16:0/337849014 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1f7c15b3d0 0x7f1f7c15de30 unknown :-1 s=CLOSED pgs=1622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:05.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:05.014+0000 7f1f81600640 1 --2- 172.21.15.16:0/337849014 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1f7c0ce2f0 0x7f1f7c1438c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:05.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:05.014+0000 7f1f81600640 1 -- 172.21.15.16:0/337849014 >> 172.21.15.16:0/337849014 conn(0x7f1f7c06b620 msgr2=0x7f1f7c165650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:14:05.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:05.014+0000 7f1f81600640 1 -- 172.21.15.16:0/337849014 shutdown_connections 2024-03-31T15:14:05.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:05.014+0000 7f1f81600640 1 -- 172.21.15.16:0/337849014 wait complete. 2024-03-31T15:14:05.028 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-31T15:14:05.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:05.174+0000 7f4e90600640 1 Processor -- start 2024-03-31T15:14:05.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:05.174+0000 7f4e90600640 1 -- start start 2024-03-31T15:14:05.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:05.174+0000 7f4e90600640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f4e88145d10 0x7f4e880cf790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:05.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:05.174+0000 7f4e90600640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f4e880728b0 0x7f4e880b1cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:05.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:05.178+0000 7f4e90600640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4e88160e70 con 0x7f4e880b2200 2024-03-31T15:14:05.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:05.178+0000 7f4e90600640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f4e881602f0 con 0x7f4e880728b0 2024-03-31T15:14:05.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:05.178+0000 7f4e90600640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f4e881605d0 con 0x7f4e88145d10 2024-03-31T15:14:05.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:05.178+0000 7f4e8d200640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f4e880728b0 0x7f4e880b1cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:05.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:05.178+0000 7f4e8d200640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f4e880728b0 0x7f4e880b1cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40810/0 (socket says 172.21.15.16:40810) 2024-03-31T15:14:05.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:05.178+0000 7f4e8d200640 1 -- 172.21.15.16:0/180769462 learned_addr learned my addr 172.21.15.16:0/180769462 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:14:05.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:05.178+0000 7f4e8dc00640 1 --2- 172.21.15.16:0/180769462 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f4e88145d10 0x7f4e880cf790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:05.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:05.178+0000 7f4e8d200640 1 -- 172.21.15.16:0/180769462 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f4e88145d10 msgr2=0x7f4e880cf790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:14:05.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:05.178+0000 7f4e8d200640 1 --2- 172.21.15.16:0/180769462 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f4e88145d10 0x7f4e880cf790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:05.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:05.178+0000 7f4e8d200640 1 -- 172.21.15.16:0/180769462 >> v1:172.21.15.89:6789/0 conn(0x7f4e880b2200 legacy=0x7f4e8815cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:05.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:05.178+0000 7f4e8d200640 1 -- 172.21.15.16:0/180769462 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e880cfcd0 con 0x7f4e880728b0 2024-03-31T15:14:05.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:05.178+0000 7f4e8dc00640 1 --2- 172.21.15.16:0/180769462 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f4e88145d10 0x7f4e880cf790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:14:05.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:05.178+0000 7f4e8d200640 1 --2- 172.21.15.16:0/180769462 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4e880728b0 0x7f4e880b1cc0 secure :-1 s=READY pgs=1623 cs=0 l=1 rev1=1 crypto rx=0x7f4e74009870 tx=0x7f4e7401c560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e89fd720894d91a5 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:05.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:05.178+0000 7f4e87e00640 1 -- 172.21.15.16:0/180769462 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e740b2070 con 0x7f4e880728b0 2024-03-31T15:14:05.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:05.178+0000 7f4e87e00640 1 -- 172.21.15.16:0/180769462 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4e740a9070 con 0x7f4e880728b0 2024-03-31T15:14:05.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:05.178+0000 7f4e87e00640 1 -- 172.21.15.16:0/180769462 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e740ad880 con 0x7f4e880728b0 2024-03-31T15:14:05.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:05.182+0000 7f4e90600640 1 -- 172.21.15.16:0/180769462 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4e880728b0 msgr2=0x7f4e880b1cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:05.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:05.182+0000 7f4e90600640 1 --2- 172.21.15.16:0/180769462 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4e880728b0 0x7f4e880b1cc0 secure :-1 s=READY pgs=1623 cs=0 l=1 rev1=1 crypto rx=0x7f4e74009870 tx=0x7f4e7401c560 comp rx=0 tx=0).stop 2024-03-31T15:14:05.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:05.182+0000 7f4e90600640 1 -- 172.21.15.16:0/180769462 shutdown_connections 2024-03-31T15:14:05.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:05.182+0000 7f4e90600640 1 --2- 172.21.15.16:0/180769462 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4e880728b0 0x7f4e880b1cc0 unknown :-1 s=CLOSED pgs=1623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:05.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:05.182+0000 7f4e90600640 1 --2- 172.21.15.16:0/180769462 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f4e88145d10 0x7f4e880cf790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:05.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:05.182+0000 7f4e90600640 1 -- 172.21.15.16:0/180769462 >> 172.21.15.16:0/180769462 conn(0x7f4e8806b620 msgr2=0x7f4e8806ba30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:14:05.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:05.182+0000 7f4e90600640 1 -- 172.21.15.16:0/180769462 shutdown_connections 2024-03-31T15:14:05.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:05.182+0000 7f4e90600640 1 -- 172.21.15.16:0/180769462 wait complete. 2024-03-31T15:14:05.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:05.182+0000 7f4e90600640 1 Processor -- start 2024-03-31T15:14:05.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:05.182+0000 7f4e90600640 1 -- start start 2024-03-31T15:14:05.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:05.182+0000 7f4e90600640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4e880728b0 0x7f4e880b18b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:05.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:05.182+0000 7f4e90600640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f4e880b2200 0x7f4e880b4bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:05.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:05.182+0000 7f4e90600640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4e88145d10 0x7f4e8813c810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:05.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:05.182+0000 7f4e90600640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f4e8815d870 con 0x7f4e880728b0 2024-03-31T15:14:05.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:05.182+0000 7f4e90600640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f4e880b2610 con 0x7f4e88145d10 2024-03-31T15:14:05.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:05.182+0000 7f4e90600640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f4e880cf350 con 0x7f4e880b2200 2024-03-31T15:14:05.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:05.182+0000 7f4e8dc00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4e880728b0 0x7f4e880b18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:05.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:05.182+0000 7f4e8dc00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4e880728b0 0x7f4e880b18b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40818/0 (socket says 172.21.15.16:40818) 2024-03-31T15:14:05.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:05.182+0000 7f4e8dc00640 1 -- 172.21.15.16:0/46127085 learned_addr learned my addr 172.21.15.16:0/46127085 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:14:05.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:05.182+0000 7f4e8e600640 1 --2- 172.21.15.16:0/46127085 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4e88145d10 0x7f4e8813c810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:05.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:05.182+0000 7f4e8d200640 1 --2- 172.21.15.16:0/46127085 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f4e880b2200 0x7f4e880b4bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:05.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:05.182+0000 7f4e8dc00640 1 -- 172.21.15.16:0/46127085 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f4e880b2200 msgr2=0x7f4e880b4bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:05.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:05.182+0000 7f4e8dc00640 1 --2- 172.21.15.16:0/46127085 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f4e880b2200 0x7f4e880b4bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:05.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:05.182+0000 7f4e8dc00640 1 -- 172.21.15.16:0/46127085 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4e88145d10 msgr2=0x7f4e8813c810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:05.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:05.182+0000 7f4e8dc00640 1 --2- 172.21.15.16:0/46127085 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4e88145d10 0x7f4e8813c810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:05.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:05.186+0000 7f4e8dc00640 1 -- 172.21.15.16:0/46127085 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e880b5100 con 0x7f4e880728b0 2024-03-31T15:14:05.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:05.186+0000 7f4e8dc00640 1 --2- 172.21.15.16:0/46127085 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4e880728b0 0x7f4e880b18b0 secure :-1 s=READY pgs=1624 cs=0 l=1 rev1=1 crypto rx=0x7f4e7c00e840 tx=0x7f4e7c00ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:05.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:05.186+0000 7f4e85600640 1 -- 172.21.15.16:0/46127085 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e7c017070 con 0x7f4e880728b0 2024-03-31T15:14:05.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:05.186+0000 7f4e85600640 1 -- 172.21.15.16:0/46127085 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4e7c00c8d0 con 0x7f4e880728b0 2024-03-31T15:14:05.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:05.186+0000 7f4e85600640 1 -- 172.21.15.16:0/46127085 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e7c00f040 con 0x7f4e880728b0 2024-03-31T15:14:05.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:05.186+0000 7f4e90600640 1 -- 172.21.15.16:0/46127085 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4e88144d00 con 0x7f4e880728b0 2024-03-31T15:14:05.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:05.186+0000 7f4e90600640 1 -- 172.21.15.16:0/46127085 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4e880afcd0 con 0x7f4e880728b0 2024-03-31T15:14:05.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:05.186+0000 7f4e90600640 1 -- 172.21.15.16:0/46127085 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4e88069950 con 0x7f4e880728b0 2024-03-31T15:14:05.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:05.186+0000 7f4e85600640 1 -- 172.21.15.16:0/46127085 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f4e7c01a070 con 0x7f4e880728b0 2024-03-31T15:14:05.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:05.186+0000 7f4e85600640 1 --2- 172.21.15.16:0/46127085 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f4e68068380 0x7f4e6806a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:05.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:05.186+0000 7f4e85600640 1 -- 172.21.15.16:0/46127085 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 4857+0+0 (secure 0 0 0) 0x7f4e7c086250 con 0x7f4e880728b0 2024-03-31T15:14:05.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:05.186+0000 7f4e8d200640 1 --2- 172.21.15.16:0/46127085 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f4e68068380 0x7f4e6806a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:05.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:05.190+0000 7f4e8d200640 1 --2- 172.21.15.16:0/46127085 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f4e68068380 0x7f4e6806a840 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f4e740af000 tx=0x7f4e74008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:05.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:05.190+0000 7f4e85600640 1 -- 172.21.15.16:0/46127085 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f4e7c056500 con 0x7f4e880728b0 2024-03-31T15:14:05.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:05.358+0000 7f4e90600640 1 -- 172.21.15.16:0/46127085 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f4e88142d20 con 0x7f4e880728b0 2024-03-31T15:14:05.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:05.358+0000 7f4e85600640 1 -- 172.21.15.16:0/46127085 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v105) ==== 74+0+12821 (secure 0 0 0) 0x7f4e7c05a800 con 0x7f4e880728b0 2024-03-31T15:14:05.353 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:14:05.353 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":105,"fsid":"286eb5da-fb47-4550-b290-d9519c13b9a5","created":"2024-03-31T15:03:11.438952+0000","modified":"2024-03-31T15:14:04.607232+0000","last_up_change":"2024-03-31T15:03:22.885535+0000","last_in_change":"2024-03-31T15:03:13.941667+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-31T15:03:22.550787+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"92a66450-ef19-40d8-ac3d-d760b23b0bda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6801","nonce":932712244}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6803","nonce":932712244}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6807","nonce":932712244}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6805","nonce":932712244}]},"public_addr":"172.21.15.16:6801/932712244","cluster_addr":"172.21.15.16:6803/932712244","heartbeat_back_addr":"172.21.15.16:6807/932712244","heartbeat_front_addr":"172.21.15.16:6805/932712244","state":["exists","up"]},{"osd":1,"uuid":"da9d1f2b-1968-4959-b843-faa231988e92","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6810","nonce":1398737533}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6811","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6813","nonce":1398737533}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6821","nonce":1398737533}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6815","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6816","nonce":1398737533}]},"public_addr":"172.21.15.16:6810/1398737533","cluster_addr":"172.21.15.16:6813/1398737533","heartbeat_back_addr":"172.21.15.16:6821/1398737533","heartbeat_front_addr":"172.21.15.16:6816/1398737533","state":["exists","up"]},{"osd":2,"uuid":"01a73662-9a21-4b1f-bcbb-42b9636a4638","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6819","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6822","nonce":3018526646}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6825","nonce":3018526646}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6831","nonce":3018526646}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6827","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6829","nonce":3018526646}]},"public_addr":"172.21.15.16:6822/3018526646","cluster_addr":"172.21.15.16:6825/3018526646","heartbeat_back_addr":"172.21.15.16:6831/3018526646","heartbeat_front_addr":"172.21.15.16:6829/3018526646","state":["exists","up"]},{"osd":3,"uuid":"c7771fd3-e18e-4500-ac38-23cf10ebcd75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6809","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6812","nonce":3360352974}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6817","nonce":3360352974}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6828","nonce":3360352974}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6823","nonce":3360352974}]},"public_addr":"172.21.15.16:6812/3360352974","cluster_addr":"172.21.15.16:6817/3360352974","heartbeat_back_addr":"172.21.15.16:6828/3360352974","heartbeat_front_addr":"172.21.15.16:6823/3360352974","state":["exists","up"]},{"osd":4,"uuid":"88260730-1482-4fbd-a40f-cb1124e9df4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6801","nonce":1856844856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6803","nonce":1856844856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6807","nonce":1856844856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6805","nonce":1856844856}]},"public_addr":"172.21.15.89:6801/1856844856","cluster_addr":"172.21.15.89:6803/1856844856","heartbeat_back_addr":"172.21.15.89:6807/1856844856","heartbeat_front_addr":"172.21.15.89:6805/1856844856","state":["exists","up"]},{"osd":5,"uuid":"4cb2c00c-fae0-4dd8-9d27-ec4f0093ccf7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6814","nonce":50395527}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6817","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6820","nonce":50395527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6831","nonce":50395527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6823","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6826","nonce":50395527}]},"public_addr":"172.21.15.89:6814/50395527","cluster_addr":"172.21.15.89:6820/50395527","heartbeat_back_addr":"172.21.15.89:6831/50395527","heartbeat_front_addr":"172.21.15.89:6826/50395527","state":["exists","up"]},{"osd":6,"uuid":"81ce0c19-2554-4cae-acbe-66a05e9e9b9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6809","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6812","nonce":3470471752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6815","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6818","nonce":3470471752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6827","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6829","nonce":3470471752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6821","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6824","nonce":3470471752}]},"public_addr":"172.21.15.89:6812/3470471752","cluster_addr":"172.21.15.89:6818/3470471752","heartbeat_back_addr":"172.21.15.89:6829/3470471752","heartbeat_front_addr":"172.21.15.89:6824/3470471752","state":["exists","up"]},{"osd":7,"uuid":"85e4313b-d37d-442e-8042-153bf20a847d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6811","nonce":1796778630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6813","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6816","nonce":1796778630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6825","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6828","nonce":1796778630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6819","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6822","nonce":1796778630}]},"public_addr":"172.21.15.89:6811/1796778630","cluster_addr":"172.21.15.89:6816/1796778630","heartbeat_back_addr":"172.21.15.89:6828/1796778630","heartbeat_front_addr":"172.21.15.89:6822/1796778630","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.439088+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.529970+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.680512+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:21.044161+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.582111+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.536558+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.409522+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.590229+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-31T15:14:05.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:05.362+0000 7f4e90600640 1 -- 172.21.15.16:0/46127085 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f4e68068380 msgr2=0x7f4e6806a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:05.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:05.362+0000 7f4e90600640 1 --2- 172.21.15.16:0/46127085 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f4e68068380 0x7f4e6806a840 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f4e740af000 tx=0x7f4e74008040 comp rx=0 tx=0).stop 2024-03-31T15:14:05.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:05.362+0000 7f4e90600640 1 -- 172.21.15.16:0/46127085 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4e880728b0 msgr2=0x7f4e880b18b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:05.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:05.362+0000 7f4e90600640 1 --2- 172.21.15.16:0/46127085 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4e880728b0 0x7f4e880b18b0 secure :-1 s=READY pgs=1624 cs=0 l=1 rev1=1 crypto rx=0x7f4e7c00e840 tx=0x7f4e7c00ed00 comp rx=0 tx=0).stop 2024-03-31T15:14:05.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:05.362+0000 7f4e90600640 1 -- 172.21.15.16:0/46127085 shutdown_connections 2024-03-31T15:14:05.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:05.362+0000 7f4e90600640 1 --2- 172.21.15.16:0/46127085 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f4e68068380 0x7f4e6806a840 unknown :-1 s=CLOSED pgs=692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:05.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:05.362+0000 7f4e90600640 1 --2- 172.21.15.16:0/46127085 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4e88145d10 0x7f4e8813c810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:05.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:05.362+0000 7f4e90600640 1 --2- 172.21.15.16:0/46127085 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f4e880b2200 0x7f4e880b4bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:05.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:05.362+0000 7f4e90600640 1 --2- 172.21.15.16:0/46127085 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4e880728b0 0x7f4e880b18b0 unknown :-1 s=CLOSED pgs=1624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:05.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:05.362+0000 7f4e90600640 1 -- 172.21.15.16:0/46127085 >> 172.21.15.16:0/46127085 conn(0x7f4e8806b620 msgr2=0x7f4e8815bf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:14:05.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:05.362+0000 7f4e90600640 1 -- 172.21.15.16:0/46127085 shutdown_connections 2024-03-31T15:14:05.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:05.362+0000 7f4e90600640 1 -- 172.21.15.16:0/46127085 wait complete. 2024-03-31T15:14:05.374 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-31T15:03:22.550787+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-31T15:14:05.374 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-31T15:14:05.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:05.526+0000 7f6dae400640 1 Processor -- start 2024-03-31T15:14:05.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:05.526+0000 7f6dae400640 1 -- start start 2024-03-31T15:14:05.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:05.526+0000 7f6dae400640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f6da813dff0 0x7f6da806d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:05.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:05.526+0000 7f6dae400640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f6da80cd0e0 0x7f6da806da10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:05.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:05.526+0000 7f6dae400640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6da815fbc0 con 0x7f6da813e3d0 2024-03-31T15:14:05.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:05.526+0000 7f6dae400640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f6da815f730 con 0x7f6da80cd0e0 2024-03-31T15:14:05.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:05.526+0000 7f6dae400640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f6da815ee00 con 0x7f6da813dff0 2024-03-31T15:14:05.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:05.530+0000 7f6da7400640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f6da813dff0 0x7f6da806d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:05.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:05.530+0000 7f6da7400640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f6da813dff0 0x7f6da806d4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3301/0 says I am v2:172.21.15.16:44818/0 (socket says 172.21.15.16:44818) 2024-03-31T15:14:05.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:05.530+0000 7f6da7400640 1 -- 172.21.15.16:0/3659418084 learned_addr learned my addr 172.21.15.16:0/3659418084 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:14:05.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:05.530+0000 7f6da7e00640 1 --2- 172.21.15.16:0/3659418084 >> v2:172.21.15.89:3300/0 conn(0x7f6da80cd0e0 0x7f6da806da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:05.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:05.530+0000 7f6da6a00640 1 --1- 172.21.15.16:0/3659418084 >> v1:172.21.15.89:6789/0 conn(0x7f6da813e3d0 0x7f6da80cc9d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.16:47604/0 (socket says 172.21.15.16:47604) 2024-03-31T15:14:05.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:05.530+0000 7f6da7400640 1 -- 172.21.15.16:0/3659418084 >> v2:172.21.15.89:3300/0 conn(0x7f6da80cd0e0 msgr2=0x7f6da806da10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:14:05.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:05.530+0000 7f6da7400640 1 --2- 172.21.15.16:0/3659418084 >> v2:172.21.15.89:3300/0 conn(0x7f6da80cd0e0 0x7f6da806da10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:05.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:05.530+0000 7f6da7400640 1 -- 172.21.15.16:0/3659418084 >> v1:172.21.15.89:6789/0 conn(0x7f6da813e3d0 legacy=0x7f6da80cc9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:05.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:05.530+0000 7f6da7400640 1 -- 172.21.15.16:0/3659418084 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6da815f0e0 con 0x7f6da813dff0 2024-03-31T15:14:05.525 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:05.530+0000 7f6da7400640 1 --2- 172.21.15.16:0/3659418084 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f6da813dff0 0x7f6da806d4d0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f6d98009870 tx=0x7f6d9801c4d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=96b5a3c1034f7eec server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:05.525 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:05.530+0000 7f6da6000640 1 -- 172.21.15.16:0/3659418084 <== mon.2 v2:172.21.15.89:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d980b0070 con 0x7f6da813dff0 2024-03-31T15:14:05.525 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:05.530+0000 7f6da6000640 1 -- 172.21.15.16:0/3659418084 <== mon.2 v2:172.21.15.89:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6d980ab600 con 0x7f6da813dff0 2024-03-31T15:14:05.525 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:05.530+0000 7f6da6000640 1 -- 172.21.15.16:0/3659418084 <== mon.2 v2:172.21.15.89:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d980ab9d0 con 0x7f6da813dff0 2024-03-31T15:14:05.525 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:05.534+0000 7f6dae400640 1 -- 172.21.15.16:0/3659418084 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f6da813dff0 msgr2=0x7f6da806d4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:05.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:05.534+0000 7f6dae400640 1 --2- 172.21.15.16:0/3659418084 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f6da813dff0 0x7f6da806d4d0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f6d98009870 tx=0x7f6d9801c4d0 comp rx=0 tx=0).stop 2024-03-31T15:14:05.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:05.534+0000 7f6dae400640 1 -- 172.21.15.16:0/3659418084 shutdown_connections 2024-03-31T15:14:05.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:05.534+0000 7f6dae400640 1 --2- 172.21.15.16:0/3659418084 >> v2:172.21.15.89:3300/0 conn(0x7f6da80cd0e0 0x7f6da806da10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:05.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:05.534+0000 7f6dae400640 1 --2- 172.21.15.16:0/3659418084 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f6da813dff0 0x7f6da806d4d0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:05.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:05.534+0000 7f6dae400640 1 -- 172.21.15.16:0/3659418084 >> 172.21.15.16:0/3659418084 conn(0x7f6da806ee40 msgr2=0x7f6da806b620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:14:05.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:05.534+0000 7f6dae400640 1 -- 172.21.15.16:0/3659418084 shutdown_connections 2024-03-31T15:14:05.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:05.534+0000 7f6dae400640 1 -- 172.21.15.16:0/3659418084 wait complete. 2024-03-31T15:14:05.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:05.534+0000 7f6dae400640 1 Processor -- start 2024-03-31T15:14:05.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:05.534+0000 7f6dae400640 1 -- start start 2024-03-31T15:14:05.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:05.534+0000 7f6dae400640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f6da80cd0e0 0x7f6da80b1720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:05.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:05.534+0000 7f6dae400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6da813dff0 0x7f6da813ce00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:05.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:05.534+0000 7f6dae400640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6da813e3d0 0x7f6da8154e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:05.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:05.534+0000 7f6dae400640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f6da80cf710 con 0x7f6da813dff0 2024-03-31T15:14:05.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:05.534+0000 7f6dae400640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f6da80cf590 con 0x7f6da813e3d0 2024-03-31T15:14:05.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:05.534+0000 7f6dae400640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f6da80cf890 con 0x7f6da80cd0e0 2024-03-31T15:14:05.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:05.534+0000 7f6da6a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6da813dff0 0x7f6da813ce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:05.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:05.534+0000 7f6da6a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6da813dff0 0x7f6da813ce00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40826/0 (socket says 172.21.15.16:40826) 2024-03-31T15:14:05.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:05.534+0000 7f6da6a00640 1 -- 172.21.15.16:0/2877386073 learned_addr learned my addr 172.21.15.16:0/2877386073 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:14:05.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:05.534+0000 7f6da7400640 1 --2- 172.21.15.16:0/2877386073 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f6da80cd0e0 0x7f6da80b1720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:05.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:05.534+0000 7f6da7e00640 1 --2- 172.21.15.16:0/2877386073 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6da813e3d0 0x7f6da8154e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:05.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:05.534+0000 7f6da6a00640 1 -- 172.21.15.16:0/2877386073 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f6da80cd0e0 msgr2=0x7f6da80b1720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:05.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:05.534+0000 7f6da6a00640 1 --2- 172.21.15.16:0/2877386073 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f6da80cd0e0 0x7f6da80b1720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:05.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:05.534+0000 7f6da6a00640 1 -- 172.21.15.16:0/2877386073 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6da813e3d0 msgr2=0x7f6da8154e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:05.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:05.534+0000 7f6da6a00640 1 --2- 172.21.15.16:0/2877386073 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6da813e3d0 0x7f6da8154e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:05.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:05.534+0000 7f6da6a00640 1 -- 172.21.15.16:0/2877386073 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6da80b0680 con 0x7f6da813dff0 2024-03-31T15:14:05.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:05.534+0000 7f6da6a00640 1 --2- 172.21.15.16:0/2877386073 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6da813dff0 0x7f6da813ce00 secure :-1 s=READY pgs=1625 cs=0 l=1 rev1=1 crypto rx=0x7f6d9400c3a0 tx=0x7f6d9400c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:05.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:05.538+0000 7f6d9f400640 1 -- 172.21.15.16:0/2877386073 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d940074e0 con 0x7f6da813dff0 2024-03-31T15:14:05.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:05.538+0000 7f6d9f400640 1 -- 172.21.15.16:0/2877386073 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6d94007680 con 0x7f6da813dff0 2024-03-31T15:14:05.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:05.538+0000 7f6dae400640 1 -- 172.21.15.16:0/2877386073 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6da80b0960 con 0x7f6da813dff0 2024-03-31T15:14:05.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:05.538+0000 7f6d9f400640 1 -- 172.21.15.16:0/2877386073 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d94004a30 con 0x7f6da813dff0 2024-03-31T15:14:05.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:05.538+0000 7f6dae400640 1 -- 172.21.15.16:0/2877386073 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6da80b0f20 con 0x7f6da813dff0 2024-03-31T15:14:05.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:05.538+0000 7f6dae400640 1 -- 172.21.15.16:0/2877386073 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6da8069950 con 0x7f6da813dff0 2024-03-31T15:14:05.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:05.538+0000 7f6d9f400640 1 -- 172.21.15.16:0/2877386073 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f6d9401a020 con 0x7f6da813dff0 2024-03-31T15:14:05.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:05.538+0000 7f6d9f400640 1 --2- 172.21.15.16:0/2877386073 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f6d88068380 0x7f6d8806a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:05.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:05.538+0000 7f6d9f400640 1 -- 172.21.15.16:0/2877386073 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 4857+0+0 (secure 0 0 0) 0x7f6d94085cf0 con 0x7f6da813dff0 2024-03-31T15:14:05.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:05.538+0000 7f6da7400640 1 --2- 172.21.15.16:0/2877386073 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f6d88068380 0x7f6d8806a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:05.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:05.542+0000 7f6da7400640 1 --2- 172.21.15.16:0/2877386073 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f6d88068380 0x7f6d8806a840 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f6d980ad000 tx=0x7f6d980a6000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:05.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:05.542+0000 7f6d9f400640 1 -- 172.21.15.16:0/2877386073 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f6d9408a070 con 0x7f6da813dff0 2024-03-31T15:14:05.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:05.706+0000 7f6dae400640 1 -- 172.21.15.16:0/2877386073 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f6da8159bf0 con 0x7f6da813dff0 2024-03-31T15:14:05.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:05.706+0000 7f6d9f400640 1 -- 172.21.15.16:0/2877386073 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v105) ==== 93+0+10 (secure 0 0 0) 0x7f6d94055fb0 con 0x7f6da813dff0 2024-03-31T15:14:05.700 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2024-03-31T15:14:05.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:05.710+0000 7f6dae400640 1 -- 172.21.15.16:0/2877386073 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f6d88068380 msgr2=0x7f6d8806a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:05.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:05.710+0000 7f6dae400640 1 --2- 172.21.15.16:0/2877386073 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f6d88068380 0x7f6d8806a840 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f6d980ad000 tx=0x7f6d980a6000 comp rx=0 tx=0).stop 2024-03-31T15:14:05.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:05.710+0000 7f6dae400640 1 -- 172.21.15.16:0/2877386073 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6da813dff0 msgr2=0x7f6da813ce00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:05.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:05.710+0000 7f6dae400640 1 --2- 172.21.15.16:0/2877386073 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6da813dff0 0x7f6da813ce00 secure :-1 s=READY pgs=1625 cs=0 l=1 rev1=1 crypto rx=0x7f6d9400c3a0 tx=0x7f6d9400c860 comp rx=0 tx=0).stop 2024-03-31T15:14:05.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:05.710+0000 7f6dae400640 1 -- 172.21.15.16:0/2877386073 shutdown_connections 2024-03-31T15:14:05.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:05.710+0000 7f6dae400640 1 --2- 172.21.15.16:0/2877386073 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f6d88068380 0x7f6d8806a840 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:05.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:05.710+0000 7f6dae400640 1 --2- 172.21.15.16:0/2877386073 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6da813e3d0 0x7f6da8154e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:05.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:05.710+0000 7f6dae400640 1 --2- 172.21.15.16:0/2877386073 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6da813dff0 0x7f6da813ce00 unknown :-1 s=CLOSED pgs=1625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:05.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:05.710+0000 7f6dae400640 1 --2- 172.21.15.16:0/2877386073 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f6da80cd0e0 0x7f6da80b1720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:05.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:05.710+0000 7f6dae400640 1 -- 172.21.15.16:0/2877386073 >> 172.21.15.16:0/2877386073 conn(0x7f6da806ee40 msgr2=0x7f6da8169b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:14:05.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:05.710+0000 7f6dae400640 1 -- 172.21.15.16:0/2877386073 shutdown_connections 2024-03-31T15:14:05.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:05.710+0000 7f6dae400640 1 -- 172.21.15.16:0/2877386073 wait complete. 2024-03-31T15:14:05.722 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-31T15:14:05.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:05.870+0000 7f38d0800640 1 Processor -- start 2024-03-31T15:14:05.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:05.870+0000 7f38d0800640 1 -- start start 2024-03-31T15:14:05.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:05.870+0000 7f38d0800640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f38c80728b0 0x7f38c80b1cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:05.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:05.870+0000 7f38d0800640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f38c80b2200 0x7f38c815cd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:05.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:05.870+0000 7f38d0800640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f38c8160e70 con 0x7f38c8145d10 2024-03-31T15:14:05.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:05.870+0000 7f38d0800640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f38c81602f0 con 0x7f38c80b2200 2024-03-31T15:14:05.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:05.870+0000 7f38d0800640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f38c81605d0 con 0x7f38c80728b0 2024-03-31T15:14:05.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:05.870+0000 7f38cde00640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f38c8145d10 0x7f38c80cf790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:57256/0 (socket says 172.21.15.16:57256) 2024-03-31T15:14:05.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:05.870+0000 7f38cde00640 1 -- 172.21.15.16:0/2908006569 learned_addr learned my addr 172.21.15.16:0/2908006569 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:14:05.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:05.870+0000 7f38ce800640 1 --2- 172.21.15.16:0/2908006569 >> v2:172.21.15.16:3300/0 conn(0x7f38c80b2200 0x7f38c815cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:05.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:05.870+0000 7f38cd400640 1 --2- 172.21.15.16:0/2908006569 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f38c80728b0 0x7f38c80b1cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:05.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:05.870+0000 7f38cca00640 1 -- 172.21.15.16:0/2908006569 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 78190891 0 0) 0x7f38c8160e70 con 0x7f38c8145d10 2024-03-31T15:14:05.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:05.870+0000 7f38ce800640 1 -- 172.21.15.16:0/2908006569 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f38c80728b0 msgr2=0x7f38c80b1cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:14:05.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:05.870+0000 7f38ce800640 1 --2- 172.21.15.16:0/2908006569 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f38c80728b0 0x7f38c80b1cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:05.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:05.870+0000 7f38ce800640 1 -- 172.21.15.16:0/2908006569 >> v1:172.21.15.16:6789/0 conn(0x7f38c8145d10 legacy=0x7f38c80cf790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:05.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:05.870+0000 7f38ce800640 1 -- 172.21.15.16:0/2908006569 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38c81608b0 con 0x7f38c80b2200 2024-03-31T15:14:05.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:05.874+0000 7f38ce800640 1 --2- 172.21.15.16:0/2908006569 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f38c80b2200 0x7f38c815cd50 secure :-1 s=READY pgs=1627 cs=0 l=1 rev1=1 crypto rx=0x7f38c4009d90 tx=0x7f38c401c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8d1e649d58840c1 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:05.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:05.874+0000 7f38cca00640 1 -- 172.21.15.16:0/2908006569 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38c40b2070 con 0x7f38c80b2200 2024-03-31T15:14:05.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:05.874+0000 7f38cca00640 1 -- 172.21.15.16:0/2908006569 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f38c40a5830 con 0x7f38c80b2200 2024-03-31T15:14:05.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:05.874+0000 7f38cca00640 1 -- 172.21.15.16:0/2908006569 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38c40a5c00 con 0x7f38c80b2200 2024-03-31T15:14:05.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:05.874+0000 7f38d0800640 1 -- 172.21.15.16:0/2908006569 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f38c80b2200 msgr2=0x7f38c815cd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:05.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:05.874+0000 7f38d0800640 1 --2- 172.21.15.16:0/2908006569 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f38c80b2200 0x7f38c815cd50 secure :-1 s=READY pgs=1627 cs=0 l=1 rev1=1 crypto rx=0x7f38c4009d90 tx=0x7f38c401c970 comp rx=0 tx=0).stop 2024-03-31T15:14:05.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:05.874+0000 7f38d0800640 1 -- 172.21.15.16:0/2908006569 shutdown_connections 2024-03-31T15:14:05.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:05.874+0000 7f38d0800640 1 --2- 172.21.15.16:0/2908006569 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f38c80b2200 0x7f38c815cd50 unknown :-1 s=CLOSED pgs=1627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:05.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:05.874+0000 7f38d0800640 1 --2- 172.21.15.16:0/2908006569 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f38c80728b0 0x7f38c80b1cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:05.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:05.874+0000 7f38d0800640 1 -- 172.21.15.16:0/2908006569 >> 172.21.15.16:0/2908006569 conn(0x7f38c806b620 msgr2=0x7f38c806ba30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:14:05.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:05.874+0000 7f38d0800640 1 -- 172.21.15.16:0/2908006569 shutdown_connections 2024-03-31T15:14:05.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:05.874+0000 7f38d0800640 1 -- 172.21.15.16:0/2908006569 wait complete. 2024-03-31T15:14:05.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:05.874+0000 7f38d0800640 1 Processor -- start 2024-03-31T15:14:05.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:05.874+0000 7f38d0800640 1 -- start start 2024-03-31T15:14:05.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:05.878+0000 7f38d0800640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f38c80728b0 0x7f38c8141310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:05.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:05.878+0000 7f38d0800640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f38c80b2200 0x7f38c8157560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:05.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:05.878+0000 7f38d0800640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f38c8145d10 0x7f38c8141dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:05.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:05.878+0000 7f38d0800640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f38c8072e10 con 0x7f38c80b2200 2024-03-31T15:14:05.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:05.878+0000 7f38d0800640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f38c8072c90 con 0x7f38c80728b0 2024-03-31T15:14:05.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:05.878+0000 7f38d0800640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f38c81690f0 con 0x7f38c8145d10 2024-03-31T15:14:05.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:05.878+0000 7f38cd400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f38c80b2200 0x7f38c8157560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:05.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:05.878+0000 7f38cd400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f38c80b2200 0x7f38c8157560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40854/0 (socket says 172.21.15.16:40854) 2024-03-31T15:14:05.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:05.878+0000 7f38cd400640 1 -- 172.21.15.16:0/2583121285 learned_addr learned my addr 172.21.15.16:0/2583121285 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:14:05.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:05.878+0000 7f38cde00640 1 --2- 172.21.15.16:0/2583121285 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f38c80728b0 0x7f38c8141310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:05.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:05.878+0000 7f38ce800640 1 --2- 172.21.15.16:0/2583121285 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f38c8145d10 0x7f38c8141dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:05.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:05.878+0000 7f38cd400640 1 -- 172.21.15.16:0/2583121285 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f38c8145d10 msgr2=0x7f38c8141dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:05.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:05.878+0000 7f38cd400640 1 --2- 172.21.15.16:0/2583121285 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f38c8145d10 0x7f38c8141dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:05.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:05.878+0000 7f38cd400640 1 -- 172.21.15.16:0/2583121285 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f38c80728b0 msgr2=0x7f38c8141310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:05.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:05.878+0000 7f38cd400640 1 --2- 172.21.15.16:0/2583121285 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f38c80728b0 0x7f38c8141310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:05.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:05.878+0000 7f38cd400640 1 -- 172.21.15.16:0/2583121285 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38c8140870 con 0x7f38c80b2200 2024-03-31T15:14:05.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:05.878+0000 7f38cde00640 1 --2- 172.21.15.16:0/2583121285 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f38c80728b0 0x7f38c8141310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:14:05.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:05.878+0000 7f38ce800640 1 --2- 172.21.15.16:0/2583121285 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f38c8145d10 0x7f38c8141dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:14:05.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:05.878+0000 7f38cd400640 1 --2- 172.21.15.16:0/2583121285 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f38c80b2200 0x7f38c8157560 secure :-1 s=READY pgs=1628 cs=0 l=1 rev1=1 crypto rx=0x7f38b800d800 tx=0x7f38b800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:05.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:05.878+0000 7f38c2000640 1 -- 172.21.15.16:0/2583121285 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38b800b6c0 con 0x7f38c80b2200 2024-03-31T15:14:05.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:05.878+0000 7f38c2000640 1 -- 172.21.15.16:0/2583121285 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f38b8019070 con 0x7f38c80b2200 2024-03-31T15:14:05.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:05.878+0000 7f38d0800640 1 -- 172.21.15.16:0/2583121285 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f38c8139430 con 0x7f38c80b2200 2024-03-31T15:14:05.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:05.878+0000 7f38c2000640 1 -- 172.21.15.16:0/2583121285 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38b800ba60 con 0x7f38c80b2200 2024-03-31T15:14:05.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:05.878+0000 7f38d0800640 1 -- 172.21.15.16:0/2583121285 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f38c8140b10 con 0x7f38c80b2200 2024-03-31T15:14:05.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:05.878+0000 7f38d0800640 1 -- 172.21.15.16:0/2583121285 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f38c8069950 con 0x7f38c80b2200 2024-03-31T15:14:05.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:05.882+0000 7f38c2000640 1 -- 172.21.15.16:0/2583121285 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f38b800bc00 con 0x7f38c80b2200 2024-03-31T15:14:05.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:05.882+0000 7f38c2000640 1 --2- 172.21.15.16:0/2583121285 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f38ac068380 0x7f38ac06a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:05.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:05.882+0000 7f38c2000640 1 -- 172.21.15.16:0/2583121285 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 4857+0+0 (secure 0 0 0) 0x7f38b8086bb0 con 0x7f38c80b2200 2024-03-31T15:14:05.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:05.882+0000 7f38cde00640 1 --2- 172.21.15.16:0/2583121285 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f38ac068380 0x7f38ac06a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:05.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:05.882+0000 7f38cde00640 1 --2- 172.21.15.16:0/2583121285 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f38ac068380 0x7f38ac06a840 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f38bc000f70 tx=0x7f38bc009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:05.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:05.886+0000 7f38c2000640 1 -- 172.21.15.16:0/2583121285 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f38b8057420 con 0x7f38c80b2200 2024-03-31T15:14:06.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:06.050+0000 7f38d0800640 1 -- 172.21.15.16:0/2583121285 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7f38c8165a50 con 0x7f38c80b2200 2024-03-31T15:14:06.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:06.050+0000 7f38c2000640 1 -- 172.21.15.16:0/2583121285 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v105) ==== 84+0+65 (secure 0 0 0) 0x7f38b8056d20 con 0x7f38c80b2200 2024-03-31T15:14:06.044 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:14:06.044 INFO:teuthology.orchestra.run.smithi016.stdout:[ 2024-03-31T15:14:06.045 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-31T15:14:06.045 INFO:teuthology.orchestra.run.smithi016.stdout: "poolnum": 1, 2024-03-31T15:14:06.045 INFO:teuthology.orchestra.run.smithi016.stdout: "poolname": ".mgr" 2024-03-31T15:14:06.045 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-31T15:14:06.045 INFO:teuthology.orchestra.run.smithi016.stdout:] 2024-03-31T15:14:06.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:06.054+0000 7f38d0800640 1 -- 172.21.15.16:0/2583121285 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f38ac068380 msgr2=0x7f38ac06a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:06.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:06.054+0000 7f38d0800640 1 --2- 172.21.15.16:0/2583121285 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f38ac068380 0x7f38ac06a840 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f38bc000f70 tx=0x7f38bc009040 comp rx=0 tx=0).stop 2024-03-31T15:14:06.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:06.054+0000 7f38d0800640 1 -- 172.21.15.16:0/2583121285 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f38c80b2200 msgr2=0x7f38c8157560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:06.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:06.054+0000 7f38d0800640 1 --2- 172.21.15.16:0/2583121285 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f38c80b2200 0x7f38c8157560 secure :-1 s=READY pgs=1628 cs=0 l=1 rev1=1 crypto rx=0x7f38b800d800 tx=0x7f38b800dcc0 comp rx=0 tx=0).stop 2024-03-31T15:14:06.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:06.054+0000 7f38d0800640 1 -- 172.21.15.16:0/2583121285 shutdown_connections 2024-03-31T15:14:06.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:06.054+0000 7f38d0800640 1 --2- 172.21.15.16:0/2583121285 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f38ac068380 0x7f38ac06a840 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:06.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:06.054+0000 7f38d0800640 1 --2- 172.21.15.16:0/2583121285 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f38c8145d10 0x7f38c8141dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:06.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:06.054+0000 7f38d0800640 1 --2- 172.21.15.16:0/2583121285 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f38c80b2200 0x7f38c8157560 unknown :-1 s=CLOSED pgs=1628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:06.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:06.054+0000 7f38d0800640 1 --2- 172.21.15.16:0/2583121285 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f38c80728b0 0x7f38c8141310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:06.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:06.054+0000 7f38d0800640 1 -- 172.21.15.16:0/2583121285 >> 172.21.15.16:0/2583121285 conn(0x7f38c806b620 msgr2=0x7f38c80b4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:14:06.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:06.054+0000 7f38d0800640 1 -- 172.21.15.16:0/2583121285 shutdown_connections 2024-03-31T15:14:06.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:06.054+0000 7f38d0800640 1 -- 172.21.15.16:0/2583121285 wait complete. 2024-03-31T15:14:06.066 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-31T15:14:06.066 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-31T15:14:06.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:06.218+0000 7f41cba00640 1 Processor -- start 2024-03-31T15:14:06.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:06.218+0000 7f41cba00640 1 -- start start 2024-03-31T15:14:06.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:06.218+0000 7f41cba00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f41c40d1040 0x7f41c4169270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:06.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:06.218+0000 7f41cba00640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f41c40d14d0 con 0x7f41c40b44d0 2024-03-31T15:14:06.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:06.218+0000 7f41cba00640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f41c40d2020 con 0x7f41c4147230 2024-03-31T15:14:06.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:06.218+0000 7f41cba00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f41c40d2c30 con 0x7f41c40d1040 2024-03-31T15:14:06.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:06.218+0000 7f41bbe00640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f41c40b44d0 0x7f41c40d0930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:57268/0 (socket says 172.21.15.16:57268) 2024-03-31T15:14:06.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:06.222+0000 7f41bbe00640 1 -- 172.21.15.16:0/1795521570 learned_addr learned my addr 172.21.15.16:0/1795521570 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:14:06.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:06.222+0000 7f41c9a00640 1 --2- 172.21.15.16:0/1795521570 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f41c40d1040 0x7f41c4169270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:06.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:06.222+0000 7f41c3e00640 1 -- 172.21.15.16:0/1795521570 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3831126508 0 0) 0x7f41c40d14d0 con 0x7f41c40b44d0 2024-03-31T15:14:06.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:06.222+0000 7f41c3e00640 1 -- 172.21.15.16:0/1795521570 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f41a8003530 con 0x7f41c40b44d0 2024-03-31T15:14:06.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:06.222+0000 7f41c9a00640 1 -- 172.21.15.16:0/1795521570 >> v1:172.21.15.89:6789/0 conn(0x7f41c4147230 legacy=0x7f41c40b3dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:06.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:06.222+0000 7f41c3e00640 1 -- 172.21.15.16:0/1795521570 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3656636266 0 0) 0x7f41a8003530 con 0x7f41c40b44d0 2024-03-31T15:14:06.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:06.222+0000 7f41c9a00640 1 -- 172.21.15.16:0/1795521570 >> v1:172.21.15.16:6789/0 conn(0x7f41c40b44d0 legacy=0x7f41c40d0930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:06.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:06.222+0000 7f41c9a00640 1 -- 172.21.15.16:0/1795521570 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41c40d2300 con 0x7f41c40d1040 2024-03-31T15:14:06.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:06.222+0000 7f41c9a00640 1 --2- 172.21.15.16:0/1795521570 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f41c40d1040 0x7f41c4169270 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f41bc009dc0 tx=0x7f41bc01c760 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6c14c3035dcce09f server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:06.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:06.222+0000 7f41c3e00640 1 -- 172.21.15.16:0/1795521570 <== mon.2 v2:172.21.15.89:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41bc0ac070 con 0x7f41c40d1040 2024-03-31T15:14:06.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:06.222+0000 7f41c3e00640 1 -- 172.21.15.16:0/1795521570 <== mon.2 v2:172.21.15.89:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f41bc0a64a0 con 0x7f41c40d1040 2024-03-31T15:14:06.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:06.222+0000 7f41c3e00640 1 -- 172.21.15.16:0/1795521570 <== mon.2 v2:172.21.15.89:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41ac001680 con 0x7f41c40d1040 2024-03-31T15:14:06.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:06.226+0000 7f41cba00640 1 -- 172.21.15.16:0/1795521570 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f41c40d1040 msgr2=0x7f41c4169270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:06.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:06.226+0000 7f41cba00640 1 --2- 172.21.15.16:0/1795521570 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f41c40d1040 0x7f41c4169270 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f41bc009dc0 tx=0x7f41bc01c760 comp rx=0 tx=0).stop 2024-03-31T15:14:06.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:06.226+0000 7f41cba00640 1 -- 172.21.15.16:0/1795521570 shutdown_connections 2024-03-31T15:14:06.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:06.226+0000 7f41cba00640 1 --2- 172.21.15.16:0/1795521570 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f41c40d1040 0x7f41c4169270 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:06.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:06.226+0000 7f41cba00640 1 -- 172.21.15.16:0/1795521570 >> 172.21.15.16:0/1795521570 conn(0x7f41c406b7f0 msgr2=0x7f41c4067330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:14:06.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:06.226+0000 7f41cba00640 1 -- 172.21.15.16:0/1795521570 shutdown_connections 2024-03-31T15:14:06.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:06.226+0000 7f41cba00640 1 -- 172.21.15.16:0/1795521570 wait complete. 2024-03-31T15:14:06.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:06.226+0000 7f41cba00640 1 Processor -- start 2024-03-31T15:14:06.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:06.226+0000 7f41cba00640 1 -- start start 2024-03-31T15:14:06.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:06.226+0000 7f41cba00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f41c40b44d0 0x7f41c4155920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:06.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:06.226+0000 7f41cba00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f41c40d1040 0x7f41c40b30b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:06.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:06.226+0000 7f41cba00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f41c4147230 0x7f41c413ac90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:06.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:06.226+0000 7f41cba00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f41c416bfd0 con 0x7f41c4147230 2024-03-31T15:14:06.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:06.226+0000 7f41cba00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f41c40cc4c0 con 0x7f41c40b44d0 2024-03-31T15:14:06.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:06.226+0000 7f41cba00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f41c416c150 con 0x7f41c40d1040 2024-03-31T15:14:06.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:06.226+0000 7f41c9a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f41c4147230 0x7f41c413ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:06.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:06.226+0000 7f41c9a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f41c4147230 0x7f41c413ac90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40870/0 (socket says 172.21.15.16:40870) 2024-03-31T15:14:06.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:06.226+0000 7f41c9a00640 1 -- 172.21.15.16:0/3976986569 learned_addr learned my addr 172.21.15.16:0/3976986569 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:14:06.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:06.226+0000 7f41c9000640 1 --2- 172.21.15.16:0/3976986569 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f41c40b44d0 0x7f41c4155920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:06.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:06.226+0000 7f41bbe00640 1 --2- 172.21.15.16:0/3976986569 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f41c40d1040 0x7f41c40b30b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:06.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:06.226+0000 7f41c9a00640 1 -- 172.21.15.16:0/3976986569 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f41c40d1040 msgr2=0x7f41c40b30b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:06.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:06.226+0000 7f41c9a00640 1 --2- 172.21.15.16:0/3976986569 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f41c40d1040 0x7f41c40b30b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:06.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:06.226+0000 7f41c9a00640 1 -- 172.21.15.16:0/3976986569 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f41c40b44d0 msgr2=0x7f41c4155920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:06.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:06.226+0000 7f41c9a00640 1 --2- 172.21.15.16:0/3976986569 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f41c40b44d0 0x7f41c4155920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:06.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:06.226+0000 7f41c9a00640 1 -- 172.21.15.16:0/3976986569 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41c40b35f0 con 0x7f41c4147230 2024-03-31T15:14:06.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:06.230+0000 7f41c9a00640 1 --2- 172.21.15.16:0/3976986569 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f41c4147230 0x7f41c413ac90 secure :-1 s=READY pgs=1630 cs=0 l=1 rev1=1 crypto rx=0x7f41bc098230 tx=0x7f41bc019500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:06.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:06.230+0000 7f41c1600640 1 -- 172.21.15.16:0/3976986569 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41bc0bf070 con 0x7f41c4147230 2024-03-31T15:14:06.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:06.230+0000 7f41cba00640 1 -- 172.21.15.16:0/3976986569 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f41c4155f90 con 0x7f41c4147230 2024-03-31T15:14:06.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:06.230+0000 7f41cba00640 1 -- 172.21.15.16:0/3976986569 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f41c4156550 con 0x7f41c4147230 2024-03-31T15:14:06.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:06.230+0000 7f41cba00640 1 -- 172.21.15.16:0/3976986569 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f41c4069950 con 0x7f41c4147230 2024-03-31T15:14:06.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:06.230+0000 7f41c1600640 1 -- 172.21.15.16:0/3976986569 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f41bc0a64a0 con 0x7f41c4147230 2024-03-31T15:14:06.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:06.230+0000 7f41c1600640 1 -- 172.21.15.16:0/3976986569 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41bc0ac050 con 0x7f41c4147230 2024-03-31T15:14:06.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:06.234+0000 7f41c1600640 1 -- 172.21.15.16:0/3976986569 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f41bc0c2070 con 0x7f41c4147230 2024-03-31T15:14:06.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:06.234+0000 7f41c1600640 1 --2- 172.21.15.16:0/3976986569 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f41a00683b0 0x7f41a006a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:06.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:06.234+0000 7f41c1600640 1 -- 172.21.15.16:0/3976986569 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 4857+0+0 (secure 0 0 0) 0x7f41bc120f30 con 0x7f41c4147230 2024-03-31T15:14:06.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:06.234+0000 7f41c9000640 1 --2- 172.21.15.16:0/3976986569 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f41a00683b0 0x7f41a006a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:06.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:06.234+0000 7f41c9000640 1 --2- 172.21.15.16:0/3976986569 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f41a00683b0 0x7f41a006a870 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f41b4007fd0 tx=0x7f41b4009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:06.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:06.238+0000 7f41c1600640 1 -- 172.21.15.16:0/3976986569 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f41bc14ddb0 con 0x7f41c4147230 2024-03-31T15:14:06.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:06.402+0000 7f41cba00640 1 -- 172.21.15.16:0/3976986569 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7f41c413c250 con 0x7f41c4147230 2024-03-31T15:14:07.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:07.238+0000 7f41c1600640 1 -- 172.21.15.16:0/3976986569 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v106) ==== 138+0+0 (secure 0 0 0) 0x7f41bc0ca020 con 0x7f41c4147230 2024-03-31T15:14:07.230 INFO:teuthology.orchestra.run.smithi016.stderr:pool 'cephfs_metadata' created 2024-03-31T15:14:07.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:07.242+0000 7f41cba00640 1 -- 172.21.15.16:0/3976986569 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f41a00683b0 msgr2=0x7f41a006a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:07.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:07.242+0000 7f41cba00640 1 --2- 172.21.15.16:0/3976986569 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f41a00683b0 0x7f41a006a870 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f41b4007fd0 tx=0x7f41b4009040 comp rx=0 tx=0).stop 2024-03-31T15:14:07.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:07.242+0000 7f41cba00640 1 -- 172.21.15.16:0/3976986569 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f41c4147230 msgr2=0x7f41c413ac90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:07.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:07.242+0000 7f41cba00640 1 --2- 172.21.15.16:0/3976986569 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f41c4147230 0x7f41c413ac90 secure :-1 s=READY pgs=1630 cs=0 l=1 rev1=1 crypto rx=0x7f41bc098230 tx=0x7f41bc019500 comp rx=0 tx=0).stop 2024-03-31T15:14:07.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:07.242+0000 7f41cba00640 1 -- 172.21.15.16:0/3976986569 shutdown_connections 2024-03-31T15:14:07.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:07.242+0000 7f41cba00640 1 --2- 172.21.15.16:0/3976986569 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f41a00683b0 0x7f41a006a870 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:07.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:07.242+0000 7f41cba00640 1 --2- 172.21.15.16:0/3976986569 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f41c4147230 0x7f41c413ac90 unknown :-1 s=CLOSED pgs=1630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:07.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:07.242+0000 7f41cba00640 1 --2- 172.21.15.16:0/3976986569 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f41c40d1040 0x7f41c40b30b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:07.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:07.242+0000 7f41cba00640 1 --2- 172.21.15.16:0/3976986569 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f41c40b44d0 0x7f41c4155920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:07.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:07.242+0000 7f41cba00640 1 -- 172.21.15.16:0/3976986569 >> 172.21.15.16:0/3976986569 conn(0x7f41c406b7f0 msgr2=0x7f41c40b3c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:14:07.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:07.242+0000 7f41cba00640 1 -- 172.21.15.16:0/3976986569 shutdown_connections 2024-03-31T15:14:07.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:07.242+0000 7f41cba00640 1 -- 172.21.15.16:0/3976986569 wait complete. 2024-03-31T15:14:07.263 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-31T15:14:07.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:07.422+0000 7f1c8c000640 1 Processor -- start 2024-03-31T15:14:07.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:07.422+0000 7f1c8c000640 1 -- start start 2024-03-31T15:14:07.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:07.422+0000 7f1c8c000640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f1c84070030 0x7f1c84143c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:07.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:07.422+0000 7f1c8c000640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f1c84141590 0x7f1c84144170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:07.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:07.422+0000 7f1c8c000640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1c841446b0 con 0x7f1c841411b0 2024-03-31T15:14:07.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:07.422+0000 7f1c8c000640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f1c8415f910 con 0x7f1c84070030 2024-03-31T15:14:07.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:07.422+0000 7f1c8c000640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f1c84162910 con 0x7f1c84141590 2024-03-31T15:14:07.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:07.422+0000 7f1c89600640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f1c841411b0 0x7f1c8406f920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:57568/0 (socket says 172.21.15.16:57568) 2024-03-31T15:14:07.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:07.422+0000 7f1c89600640 1 -- 172.21.15.16:0/3525999992 learned_addr learned my addr 172.21.15.16:0/3525999992 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:14:07.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:07.422+0000 7f1c88c00640 1 --2- 172.21.15.16:0/3525999992 >> v2:172.21.15.16:3300/0 conn(0x7f1c84070030 0x7f1c84143c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:07.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:07.422+0000 7f1c8a000640 1 --2- 172.21.15.16:0/3525999992 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f1c84141590 0x7f1c84144170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:07.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:07.422+0000 7f1c88c00640 1 -- 172.21.15.16:0/3525999992 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f1c84141590 msgr2=0x7f1c84144170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:14:07.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:07.422+0000 7f1c88c00640 1 --2- 172.21.15.16:0/3525999992 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f1c84141590 0x7f1c84144170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:07.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:07.422+0000 7f1c88c00640 1 -- 172.21.15.16:0/3525999992 >> v1:172.21.15.16:6789/0 conn(0x7f1c841411b0 legacy=0x7f1c8406f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:07.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:07.422+0000 7f1c88c00640 1 -- 172.21.15.16:0/3525999992 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c84144980 con 0x7f1c84070030 2024-03-31T15:14:07.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:07.426+0000 7f1c88c00640 1 --2- 172.21.15.16:0/3525999992 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c84070030 0x7f1c84143c30 secure :-1 s=READY pgs=1632 cs=0 l=1 rev1=1 crypto rx=0x7f1c700097c0 tx=0x7f1c7001c470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=57d98eb838ed0b8b server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:07.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:07.426+0000 7f1c83e00640 1 -- 172.21.15.16:0/3525999992 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c700a7070 con 0x7f1c84070030 2024-03-31T15:14:07.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:07.426+0000 7f1c83e00640 1 -- 172.21.15.16:0/3525999992 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1c700b0070 con 0x7f1c84070030 2024-03-31T15:14:07.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:07.426+0000 7f1c83e00640 1 -- 172.21.15.16:0/3525999992 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c700ab890 con 0x7f1c84070030 2024-03-31T15:14:07.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:07.426+0000 7f1c8c000640 1 -- 172.21.15.16:0/3525999992 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c84070030 msgr2=0x7f1c84143c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:07.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:07.426+0000 7f1c8c000640 1 --2- 172.21.15.16:0/3525999992 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c84070030 0x7f1c84143c30 secure :-1 s=READY pgs=1632 cs=0 l=1 rev1=1 crypto rx=0x7f1c700097c0 tx=0x7f1c7001c470 comp rx=0 tx=0).stop 2024-03-31T15:14:07.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:07.426+0000 7f1c8c000640 1 -- 172.21.15.16:0/3525999992 shutdown_connections 2024-03-31T15:14:07.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:07.426+0000 7f1c8c000640 1 --2- 172.21.15.16:0/3525999992 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f1c84141590 0x7f1c84144170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:07.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:07.426+0000 7f1c8c000640 1 --2- 172.21.15.16:0/3525999992 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c84070030 0x7f1c84143c30 unknown :-1 s=CLOSED pgs=1632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:07.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:07.426+0000 7f1c8c000640 1 -- 172.21.15.16:0/3525999992 >> 172.21.15.16:0/3525999992 conn(0x7f1c84067930 msgr2=0x7f1c840b3ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:14:07.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:07.426+0000 7f1c8c000640 1 -- 172.21.15.16:0/3525999992 shutdown_connections 2024-03-31T15:14:07.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:07.426+0000 7f1c8c000640 1 -- 172.21.15.16:0/3525999992 wait complete. 2024-03-31T15:14:07.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:07.430+0000 7f1c8c000640 1 Processor -- start 2024-03-31T15:14:07.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:07.430+0000 7f1c8c000640 1 -- start start 2024-03-31T15:14:07.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:07.430+0000 7f1c8c000640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f1c84070030 0x7f1c8413d610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:07.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:07.430+0000 7f1c8c000640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1c841411b0 0x7f1c84152170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:07.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:07.430+0000 7f1c8c000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c84141590 0x7f1c84142a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:07.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:07.430+0000 7f1c8c000640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f1c8415f110 con 0x7f1c84141590 2024-03-31T15:14:07.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:07.430+0000 7f1c8c000640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f1c84162d80 con 0x7f1c841411b0 2024-03-31T15:14:07.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:07.430+0000 7f1c8c000640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f1c8415f290 con 0x7f1c84070030 2024-03-31T15:14:07.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:07.430+0000 7f1c8a000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c84141590 0x7f1c84142a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:07.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:07.430+0000 7f1c8a000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c84141590 0x7f1c84142a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50026/0 (socket says 172.21.15.16:50026) 2024-03-31T15:14:07.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:07.430+0000 7f1c8a000640 1 -- 172.21.15.16:0/1606379229 learned_addr learned my addr 172.21.15.16:0/1606379229 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:14:07.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:07.430+0000 7f1c88c00640 1 --2- 172.21.15.16:0/1606379229 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1c841411b0 0x7f1c84152170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:07.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:07.430+0000 7f1c89600640 1 --2- 172.21.15.16:0/1606379229 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f1c84070030 0x7f1c8413d610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:07.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:07.430+0000 7f1c8a000640 1 -- 172.21.15.16:0/1606379229 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f1c84070030 msgr2=0x7f1c8413d610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:07.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:07.430+0000 7f1c8a000640 1 --2- 172.21.15.16:0/1606379229 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f1c84070030 0x7f1c8413d610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:07.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:07.430+0000 7f1c8a000640 1 -- 172.21.15.16:0/1606379229 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1c841411b0 msgr2=0x7f1c84152170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:07.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:07.430+0000 7f1c8a000640 1 --2- 172.21.15.16:0/1606379229 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1c841411b0 0x7f1c84152170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:07.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:07.430+0000 7f1c8a000640 1 -- 172.21.15.16:0/1606379229 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c8413e680 con 0x7f1c84141590 2024-03-31T15:14:07.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:07.430+0000 7f1c89600640 1 --2- 172.21.15.16:0/1606379229 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f1c84070030 0x7f1c8413d610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:14:07.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:07.430+0000 7f1c8a000640 1 --2- 172.21.15.16:0/1606379229 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c84141590 0x7f1c84142a10 secure :-1 s=READY pgs=1633 cs=0 l=1 rev1=1 crypto rx=0x7f1c7c009fd0 tx=0x7f1c7c00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:07.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:07.430+0000 7f1c81600640 1 -- 172.21.15.16:0/1606379229 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c7c014420 con 0x7f1c84141590 2024-03-31T15:14:07.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:07.430+0000 7f1c81600640 1 -- 172.21.15.16:0/1606379229 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1c7c019070 con 0x7f1c84141590 2024-03-31T15:14:07.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:07.430+0000 7f1c81600640 1 -- 172.21.15.16:0/1606379229 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c7c014800 con 0x7f1c84141590 2024-03-31T15:14:07.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:07.430+0000 7f1c8c000640 1 -- 172.21.15.16:0/1606379229 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1c8413e0c0 con 0x7f1c84141590 2024-03-31T15:14:07.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:07.430+0000 7f1c8c000640 1 -- 172.21.15.16:0/1606379229 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1c841567a0 con 0x7f1c84141590 2024-03-31T15:14:07.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:07.434+0000 7f1c8c000640 1 -- 172.21.15.16:0/1606379229 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1c84069950 con 0x7f1c84141590 2024-03-31T15:14:07.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:07.434+0000 7f1c81600640 1 -- 172.21.15.16:0/1606379229 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f1c7c0149a0 con 0x7f1c84141590 2024-03-31T15:14:07.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:07.434+0000 7f1c81600640 1 --2- 172.21.15.16:0/1606379229 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f1c64068380 0x7f1c6406a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:07.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:07.434+0000 7f1c81600640 1 -- 172.21.15.16:0/1606379229 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 5227+0+0 (secure 0 0 0) 0x7f1c7c086ff0 con 0x7f1c84141590 2024-03-31T15:14:07.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:07.434+0000 7f1c89600640 1 --2- 172.21.15.16:0/1606379229 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f1c64068380 0x7f1c6406a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:07.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:07.434+0000 7f1c89600640 1 --2- 172.21.15.16:0/1606379229 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f1c64068380 0x7f1c6406a840 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f1c78007fd0 tx=0x7f1c78009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:07.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:07.438+0000 7f1c81600640 1 -- 172.21.15.16:0/1606379229 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f1c7c0577e0 con 0x7f1c84141590 2024-03-31T15:14:07.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:07.606+0000 7f1c8c000640 1 -- 172.21.15.16:0/1606379229 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7f1c8415f6f0 con 0x7f1c84141590 2024-03-31T15:14:07.833 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-31T15:14:07.836 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-31T15:14:08.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:08.242+0000 7f1c81600640 1 -- 172.21.15.16:0/1606379229 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v107) ==== 170+0+0 (secure 0 0 0) 0x7f1c7c0570e0 con 0x7f1c84141590 2024-03-31T15:14:08.234 INFO:teuthology.orchestra.run.smithi016.stderr:pool 'cephfs_data' created 2024-03-31T15:14:08.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:08.242+0000 7f1c8c000640 1 -- 172.21.15.16:0/1606379229 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f1c64068380 msgr2=0x7f1c6406a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:08.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:08.242+0000 7f1c8c000640 1 --2- 172.21.15.16:0/1606379229 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f1c64068380 0x7f1c6406a840 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f1c78007fd0 tx=0x7f1c78009040 comp rx=0 tx=0).stop 2024-03-31T15:14:08.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:08.242+0000 7f1c8c000640 1 -- 172.21.15.16:0/1606379229 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c84141590 msgr2=0x7f1c84142a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:08.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:08.242+0000 7f1c8c000640 1 --2- 172.21.15.16:0/1606379229 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c84141590 0x7f1c84142a10 secure :-1 s=READY pgs=1633 cs=0 l=1 rev1=1 crypto rx=0x7f1c7c009fd0 tx=0x7f1c7c00e4b0 comp rx=0 tx=0).stop 2024-03-31T15:14:08.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:08.242+0000 7f1c8c000640 1 -- 172.21.15.16:0/1606379229 shutdown_connections 2024-03-31T15:14:08.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:08.242+0000 7f1c8c000640 1 --2- 172.21.15.16:0/1606379229 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f1c64068380 0x7f1c6406a840 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:08.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:08.242+0000 7f1c8c000640 1 --2- 172.21.15.16:0/1606379229 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c84141590 0x7f1c84142a10 unknown :-1 s=CLOSED pgs=1633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:08.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:08.242+0000 7f1c8c000640 1 --2- 172.21.15.16:0/1606379229 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1c841411b0 0x7f1c84152170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:08.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:08.242+0000 7f1c8c000640 1 --2- 172.21.15.16:0/1606379229 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f1c84070030 0x7f1c8413d610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:08.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:08.242+0000 7f1c8c000640 1 -- 172.21.15.16:0/1606379229 >> 172.21.15.16:0/1606379229 conn(0x7f1c84067930 msgr2=0x7f1c840b2690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:14:08.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:08.242+0000 7f1c8c000640 1 -- 172.21.15.16:0/1606379229 shutdown_connections 2024-03-31T15:14:08.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:08.242+0000 7f1c8c000640 1 -- 172.21.15.16:0/1606379229 wait complete. 2024-03-31T15:14:08.275 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-31T15:14:08.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:08.430+0000 7f911ca00640 1 Processor -- start 2024-03-31T15:14:08.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:08.430+0000 7f911ca00640 1 -- start start 2024-03-31T15:14:08.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:08.430+0000 7f911ca00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f91181567e0 0x7f9118072c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:08.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:08.430+0000 7f911ca00640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f911815c7e0 con 0x7f91180731c0 2024-03-31T15:14:08.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:08.430+0000 7f911ca00640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f91180736b0 con 0x7f911806dbe0 2024-03-31T15:14:08.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:08.430+0000 7f911ca00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f911815caa0 con 0x7f91181567e0 2024-03-31T15:14:08.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:08.430+0000 7f9115400640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f91180731c0 0x7f911806d4d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:57574/0 (socket says 172.21.15.16:57574) 2024-03-31T15:14:08.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:08.430+0000 7f9115400640 1 -- 172.21.15.16:0/3399370102 learned_addr learned my addr 172.21.15.16:0/3399370102 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:14:08.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:08.430+0000 7f9115e00640 1 --2- 172.21.15.16:0/3399370102 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f91181567e0 0x7f9118072c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:08.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:08.430+0000 7f9114a00640 1 -- 172.21.15.16:0/3399370102 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3158384947 0 0) 0x7f911815c7e0 con 0x7f91180731c0 2024-03-31T15:14:08.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:08.430+0000 7f9114a00640 1 -- 172.21.15.16:0/3399370102 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f90fc003530 con 0x7f91180731c0 2024-03-31T15:14:08.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:08.430+0000 7f9114a00640 1 -- 172.21.15.16:0/3399370102 <== mon.1 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2390914117 0 0) 0x7f91180736b0 con 0x7f911806dbe0 2024-03-31T15:14:08.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:08.430+0000 7f9115e00640 1 -- 172.21.15.16:0/3399370102 >> v1:172.21.15.89:6789/0 conn(0x7f911806dbe0 legacy=0x7f9118071550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:08.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:08.430+0000 7f9115e00640 1 -- 172.21.15.16:0/3399370102 >> v1:172.21.15.16:6789/0 conn(0x7f91180731c0 legacy=0x7f911806d4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:08.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:08.430+0000 7f9115e00640 1 -- 172.21.15.16:0/3399370102 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f911815cd80 con 0x7f91181567e0 2024-03-31T15:14:08.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:08.434+0000 7f9115e00640 1 --2- 172.21.15.16:0/3399370102 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f91181567e0 0x7f9118072c80 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f91080097f0 tx=0x7f910801c4e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c4ffc73ad4e5e0f8 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:08.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:08.434+0000 7f9114a00640 1 -- 172.21.15.16:0/3399370102 <== mon.2 v2:172.21.15.89:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91080b0070 con 0x7f91181567e0 2024-03-31T15:14:08.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:08.434+0000 7f9114a00640 1 -- 172.21.15.16:0/3399370102 <== mon.2 v2:172.21.15.89:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f91080ab400 con 0x7f91181567e0 2024-03-31T15:14:08.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:08.434+0000 7f9114a00640 1 -- 172.21.15.16:0/3399370102 <== mon.2 v2:172.21.15.89:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91080ab7d0 con 0x7f91181567e0 2024-03-31T15:14:08.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:08.434+0000 7f911ca00640 1 -- 172.21.15.16:0/3399370102 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f91181567e0 msgr2=0x7f9118072c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:08.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:08.434+0000 7f911ca00640 1 --2- 172.21.15.16:0/3399370102 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f91181567e0 0x7f9118072c80 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f91080097f0 tx=0x7f910801c4e0 comp rx=0 tx=0).stop 2024-03-31T15:14:08.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:08.434+0000 7f911ca00640 1 -- 172.21.15.16:0/3399370102 shutdown_connections 2024-03-31T15:14:08.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:08.434+0000 7f911ca00640 1 --2- 172.21.15.16:0/3399370102 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f91181567e0 0x7f9118072c80 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:08.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:08.434+0000 7f911ca00640 1 -- 172.21.15.16:0/3399370102 >> 172.21.15.16:0/3399370102 conn(0x7f911806ee40 msgr2=0x7f911806b620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:14:08.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:08.434+0000 7f911ca00640 1 -- 172.21.15.16:0/3399370102 shutdown_connections 2024-03-31T15:14:08.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:08.434+0000 7f911ca00640 1 -- 172.21.15.16:0/3399370102 wait complete. 2024-03-31T15:14:08.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:08.438+0000 7f911ca00640 1 Processor -- start 2024-03-31T15:14:08.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:08.438+0000 7f911ca00640 1 -- start start 2024-03-31T15:14:08.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:08.438+0000 7f911ca00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f911806dbe0 0x7f9118144190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:08.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:08.438+0000 7f911ca00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f91180731c0 0x7f9118145930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:08.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:08.438+0000 7f911ca00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f91181567e0 0x7f9118146120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:08.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:08.438+0000 7f911ca00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f911816a300 con 0x7f91180731c0 2024-03-31T15:14:08.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:08.438+0000 7f911ca00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f911816a180 con 0x7f911806dbe0 2024-03-31T15:14:08.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:08.438+0000 7f911ca00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f911816a480 con 0x7f91181567e0 2024-03-31T15:14:08.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:08.438+0000 7f9115400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f91180731c0 0x7f9118145930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:08.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:08.438+0000 7f9115400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f91180731c0 0x7f9118145930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50030/0 (socket says 172.21.15.16:50030) 2024-03-31T15:14:08.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:08.438+0000 7f9115400640 1 -- 172.21.15.16:0/3968105551 learned_addr learned my addr 172.21.15.16:0/3968105551 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:14:08.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:08.438+0000 7f9115e00640 1 --2- 172.21.15.16:0/3968105551 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f911806dbe0 0x7f9118144190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:08.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:08.438+0000 7f9116800640 1 --2- 172.21.15.16:0/3968105551 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f91181567e0 0x7f9118146120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:08.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:08.438+0000 7f9115400640 1 -- 172.21.15.16:0/3968105551 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f91181567e0 msgr2=0x7f9118146120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:08.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:08.438+0000 7f9115400640 1 --2- 172.21.15.16:0/3968105551 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f91181567e0 0x7f9118146120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:08.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:08.438+0000 7f9115400640 1 -- 172.21.15.16:0/3968105551 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f911806dbe0 msgr2=0x7f9118144190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:08.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:08.438+0000 7f9115400640 1 --2- 172.21.15.16:0/3968105551 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f911806dbe0 0x7f9118144190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:08.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:08.438+0000 7f9115400640 1 -- 172.21.15.16:0/3968105551 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f911815e1a0 con 0x7f91180731c0 2024-03-31T15:14:08.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:08.438+0000 7f9116800640 1 --2- 172.21.15.16:0/3968105551 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f91181567e0 0x7f9118146120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:14:08.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:08.438+0000 7f9115400640 1 --2- 172.21.15.16:0/3968105551 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f91180731c0 0x7f9118145930 secure :-1 s=READY pgs=1635 cs=0 l=1 rev1=1 crypto rx=0x7f910400c450 tx=0x7f910400c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:08.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:08.438+0000 7f910e000640 1 -- 172.21.15.16:0/3968105551 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9104017070 con 0x7f91180731c0 2024-03-31T15:14:08.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:08.438+0000 7f910e000640 1 -- 172.21.15.16:0/3968105551 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9104005d10 con 0x7f91180731c0 2024-03-31T15:14:08.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:08.438+0000 7f910e000640 1 -- 172.21.15.16:0/3968105551 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91040072b0 con 0x7f91180731c0 2024-03-31T15:14:08.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:08.438+0000 7f911ca00640 1 -- 172.21.15.16:0/3968105551 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f911815e480 con 0x7f91180731c0 2024-03-31T15:14:08.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:08.438+0000 7f911ca00640 1 -- 172.21.15.16:0/3968105551 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f911815ea40 con 0x7f91180731c0 2024-03-31T15:14:08.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:08.438+0000 7f911ca00640 1 -- 172.21.15.16:0/3968105551 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9118069950 con 0x7f91180731c0 2024-03-31T15:14:08.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:08.442+0000 7f910e000640 1 -- 172.21.15.16:0/3968105551 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f910401a070 con 0x7f91180731c0 2024-03-31T15:14:08.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:08.442+0000 7f910e000640 1 --2- 172.21.15.16:0/3968105551 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f90f80683b0 0x7f90f806a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:08.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:08.442+0000 7f910e000640 1 -- 172.21.15.16:0/3968105551 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 5609+0+0 (secure 0 0 0) 0x7f9104085c80 con 0x7f91180731c0 2024-03-31T15:14:08.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:08.442+0000 7f9115e00640 1 --2- 172.21.15.16:0/3968105551 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f90f80683b0 0x7f90f806a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:08.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:08.442+0000 7f9115e00640 1 --2- 172.21.15.16:0/3968105551 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f90f80683b0 0x7f90f806a870 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f91080ad000 tx=0x7f91080a6000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:08.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:08.446+0000 7f910e000640 1 -- 172.21.15.16:0/3968105551 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f9104055c70 con 0x7f91180731c0 2024-03-31T15:14:08.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:08.646+0000 7f911ca00640 1 -- 172.21.15.16:0/3968105551 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7f91180cd7b0 con 0x7f91180731c0 2024-03-31T15:14:09.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:09.258+0000 7f910e000640 1 -- 172.21.15.16:0/3968105551 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 16 and data pool 17 v95) ==== 174+0+0 (secure 0 0 0) 0x7f9104059f70 con 0x7f91180731c0 2024-03-31T15:14:09.256 INFO:teuthology.orchestra.run.smithi016.stderr:new fs with metadata pool 16 and data pool 17 2024-03-31T15:14:09.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:09.266+0000 7f911ca00640 1 -- 172.21.15.16:0/3968105551 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f90f80683b0 msgr2=0x7f90f806a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:09.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:09.266+0000 7f911ca00640 1 --2- 172.21.15.16:0/3968105551 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f90f80683b0 0x7f90f806a870 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f91080ad000 tx=0x7f91080a6000 comp rx=0 tx=0).stop 2024-03-31T15:14:09.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:09.266+0000 7f911ca00640 1 -- 172.21.15.16:0/3968105551 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f91180731c0 msgr2=0x7f9118145930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:09.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:09.266+0000 7f911ca00640 1 --2- 172.21.15.16:0/3968105551 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f91180731c0 0x7f9118145930 secure :-1 s=READY pgs=1635 cs=0 l=1 rev1=1 crypto rx=0x7f910400c450 tx=0x7f910400c910 comp rx=0 tx=0).stop 2024-03-31T15:14:09.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:09.270+0000 7f911ca00640 1 -- 172.21.15.16:0/3968105551 shutdown_connections 2024-03-31T15:14:09.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:09.270+0000 7f911ca00640 1 --2- 172.21.15.16:0/3968105551 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f90f80683b0 0x7f90f806a870 secure :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f91080ad000 tx=0x7f91080a6000 comp rx=0 tx=0).stop 2024-03-31T15:14:09.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:09.270+0000 7f911ca00640 1 --2- 172.21.15.16:0/3968105551 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f91181567e0 0x7f9118146120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:09.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:09.270+0000 7f911ca00640 1 --2- 172.21.15.16:0/3968105551 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f91180731c0 0x7f9118145930 unknown :-1 s=CLOSED pgs=1635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:09.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:09.270+0000 7f911ca00640 1 --2- 172.21.15.16:0/3968105551 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f911806dbe0 0x7f9118144190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:09.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:09.270+0000 7f911ca00640 1 -- 172.21.15.16:0/3968105551 >> 172.21.15.16:0/3968105551 conn(0x7f911806ee40 msgr2=0x7f911815a4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:14:09.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:09.274+0000 7f911ca00640 1 -- 172.21.15.16:0/3968105551 shutdown_connections 2024-03-31T15:14:09.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:09.274+0000 7f911ca00640 1 -- 172.21.15.16:0/3968105551 wait complete. 2024-03-31T15:14:09.306 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-31T15:14:09.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:09.454+0000 7f106ae00640 1 Processor -- start 2024-03-31T15:14:09.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:09.454+0000 7f106ae00640 1 -- start start 2024-03-31T15:14:09.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:09.458+0000 7f106ae00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f1064145d10 0x7f10640cf790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:09.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:09.458+0000 7f106ae00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f10640b23d0 0x7f106415cd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:09.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:09.458+0000 7f106ae00640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1064160e70 con 0x7f10640728b0 2024-03-31T15:14:09.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:09.458+0000 7f106ae00640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f10641602f0 con 0x7f1064145d10 2024-03-31T15:14:09.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:09.458+0000 7f106ae00640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f10641605d0 con 0x7f10640b23d0 2024-03-31T15:14:09.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:09.458+0000 7f1063e00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f1064145d10 0x7f10640cf790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:09.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:09.458+0000 7f105b400640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f10640728b0 0x7f10640b1cc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:57590/0 (socket says 172.21.15.16:57590) 2024-03-31T15:14:09.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:09.458+0000 7f105b400640 1 -- 172.21.15.16:0/890641591 learned_addr learned my addr 172.21.15.16:0/890641591 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:14:09.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:09.458+0000 7f1068e00640 1 --2- 172.21.15.16:0/890641591 >> v2:172.21.15.89:3300/0 conn(0x7f10640b23d0 0x7f106415cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:09.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:09.458+0000 7f1063e00640 1 -- 172.21.15.16:0/890641591 >> v2:172.21.15.89:3300/0 conn(0x7f10640b23d0 msgr2=0x7f106415cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:14:09.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:09.458+0000 7f1063400640 1 -- 172.21.15.16:0/890641591 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2293958441 0 0) 0x7f1064160e70 con 0x7f10640728b0 2024-03-31T15:14:09.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:09.458+0000 7f1063e00640 1 --2- 172.21.15.16:0/890641591 >> v2:172.21.15.89:3300/0 conn(0x7f10640b23d0 0x7f106415cd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:09.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:09.458+0000 7f1063e00640 1 -- 172.21.15.16:0/890641591 >> v1:172.21.15.16:6789/0 conn(0x7f10640728b0 legacy=0x7f10640b1cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:09.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:09.458+0000 7f1063e00640 1 -- 172.21.15.16:0/890641591 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10640cfcd0 con 0x7f1064145d10 2024-03-31T15:14:09.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:09.458+0000 7f1063e00640 1 --2- 172.21.15.16:0/890641591 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1064145d10 0x7f10640cf790 secure :-1 s=READY pgs=1637 cs=0 l=1 rev1=1 crypto rx=0x7f1064071590 tx=0x7f105401c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ac8967ae2fccdae server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:09.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:09.462+0000 7f1063400640 1 -- 172.21.15.16:0/890641591 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10540b0070 con 0x7f1064145d10 2024-03-31T15:14:09.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:09.462+0000 7f1063400640 1 -- 172.21.15.16:0/890641591 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f10540ab600 con 0x7f1064145d10 2024-03-31T15:14:09.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:09.462+0000 7f1063400640 1 -- 172.21.15.16:0/890641591 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10540ab9d0 con 0x7f1064145d10 2024-03-31T15:14:09.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:09.462+0000 7f106ae00640 1 -- 172.21.15.16:0/890641591 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1064145d10 msgr2=0x7f10640cf790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:09.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:09.462+0000 7f106ae00640 1 --2- 172.21.15.16:0/890641591 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1064145d10 0x7f10640cf790 secure :-1 s=READY pgs=1637 cs=0 l=1 rev1=1 crypto rx=0x7f1064071590 tx=0x7f105401c480 comp rx=0 tx=0).stop 2024-03-31T15:14:09.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:09.462+0000 7f106ae00640 1 -- 172.21.15.16:0/890641591 shutdown_connections 2024-03-31T15:14:09.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:09.462+0000 7f106ae00640 1 --2- 172.21.15.16:0/890641591 >> v2:172.21.15.89:3300/0 conn(0x7f10640b23d0 0x7f106415cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:09.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:09.462+0000 7f106ae00640 1 --2- 172.21.15.16:0/890641591 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1064145d10 0x7f10640cf790 unknown :-1 s=CLOSED pgs=1637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:09.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:09.462+0000 7f106ae00640 1 -- 172.21.15.16:0/890641591 >> 172.21.15.16:0/890641591 conn(0x7f106406b620 msgr2=0x7f106406ba30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:14:09.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:09.462+0000 7f106ae00640 1 -- 172.21.15.16:0/890641591 shutdown_connections 2024-03-31T15:14:09.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:09.462+0000 7f106ae00640 1 -- 172.21.15.16:0/890641591 wait complete. 2024-03-31T15:14:09.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:09.462+0000 7f106ae00640 1 Processor -- start 2024-03-31T15:14:09.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:09.462+0000 7f106ae00640 1 -- start start 2024-03-31T15:14:09.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:09.462+0000 7f106ae00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f10640728b0 0x7f106415bc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:09.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:09.462+0000 7f106ae00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f10640b23d0 0x7f1064165910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:09.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:09.462+0000 7f106ae00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f1064145d10 0x7f1064162a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:09.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:09.462+0000 7f106ae00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f10640cf4d0 con 0x7f10640728b0 2024-03-31T15:14:09.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:09.462+0000 7f106ae00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f10640cf350 con 0x7f10640b23d0 2024-03-31T15:14:09.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:09.462+0000 7f106ae00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f1064072d00 con 0x7f1064145d10 2024-03-31T15:14:09.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:09.462+0000 7f1063e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f10640728b0 0x7f106415bc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:09.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:09.462+0000 7f1063e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f10640728b0 0x7f106415bc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50056/0 (socket says 172.21.15.16:50056) 2024-03-31T15:14:09.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:09.462+0000 7f1063e00640 1 -- 172.21.15.16:0/1609275925 learned_addr learned my addr 172.21.15.16:0/1609275925 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:14:09.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:09.462+0000 7f1068e00640 1 --2- 172.21.15.16:0/1609275925 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f1064145d10 0x7f1064162a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:09.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:09.462+0000 7f105b400640 1 --2- 172.21.15.16:0/1609275925 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f10640b23d0 0x7f1064165910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:09.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:09.466+0000 7f1063e00640 1 -- 172.21.15.16:0/1609275925 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f1064145d10 msgr2=0x7f1064162a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:09.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:09.466+0000 7f1063e00640 1 --2- 172.21.15.16:0/1609275925 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f1064145d10 0x7f1064162a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:09.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:09.466+0000 7f1063e00640 1 -- 172.21.15.16:0/1609275925 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f10640b23d0 msgr2=0x7f1064165910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:09.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:09.466+0000 7f1063e00640 1 --2- 172.21.15.16:0/1609275925 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f10640b23d0 0x7f1064165910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:09.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:09.466+0000 7f1063e00640 1 -- 172.21.15.16:0/1609275925 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f106415c1c0 con 0x7f10640728b0 2024-03-31T15:14:09.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:09.466+0000 7f1063e00640 1 --2- 172.21.15.16:0/1609275925 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f10640728b0 0x7f106415bc80 secure :-1 s=READY pgs=1638 cs=0 l=1 rev1=1 crypto rx=0x7f105401a0b0 tx=0x7f105401acc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:09.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:09.466+0000 7f1060c00640 1 -- 172.21.15.16:0/1609275925 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10540b0050 con 0x7f10640728b0 2024-03-31T15:14:09.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:09.466+0000 7f1060c00640 1 -- 172.21.15.16:0/1609275925 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f10540b1040 con 0x7f10640728b0 2024-03-31T15:14:09.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:09.466+0000 7f106ae00640 1 -- 172.21.15.16:0/1609275925 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1064144890 con 0x7f10640728b0 2024-03-31T15:14:09.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:09.466+0000 7f1060c00640 1 -- 172.21.15.16:0/1609275925 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1054006e20 con 0x7f10640728b0 2024-03-31T15:14:09.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:09.466+0000 7f106ae00640 1 -- 172.21.15.16:0/1609275925 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1064141d10 con 0x7f10640728b0 2024-03-31T15:14:09.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:09.466+0000 7f106ae00640 1 -- 172.21.15.16:0/1609275925 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1064069950 con 0x7f10640728b0 2024-03-31T15:14:09.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:09.466+0000 7f1060c00640 1 -- 172.21.15.16:0/1609275925 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f10540be060 con 0x7f10640728b0 2024-03-31T15:14:09.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:09.466+0000 7f1060c00640 1 --2- 172.21.15.16:0/1609275925 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f10400683b0 0x7f104006a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:09.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:09.466+0000 7f1060c00640 1 -- 172.21.15.16:0/1609275925 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 5709+0+0 (secure 0 0 0) 0x7f1054120130 con 0x7f10640728b0 2024-03-31T15:14:09.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:09.470+0000 7f105b400640 1 --2- 172.21.15.16:0/1609275925 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f10400683b0 0x7f104006a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:09.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:09.470+0000 7f105b400640 1 --2- 172.21.15.16:0/1609275925 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f10400683b0 0x7f104006a870 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f1064143f40 tx=0x7f104c007d60 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:09.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:09.470+0000 7f1060c00640 1 -- 172.21.15.16:0/1609275925 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f1054016070 con 0x7f10640728b0 2024-03-31T15:14:09.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:09.638+0000 7f106ae00640 1 -- 172.21.15.16:0/1609275925 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f10640b31d0 con 0x7f10640728b0 2024-03-31T15:14:09.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:09.642+0000 7f1060c00640 1 -- 172.21.15.16:0/1609275925 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v108) ==== 74+0+16568 (secure 0 0 0) 0x7f10540f0090 con 0x7f10640728b0 2024-03-31T15:14:09.633 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:14:09.633 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":108,"fsid":"286eb5da-fb47-4550-b290-d9519c13b9a5","created":"2024-03-31T15:03:11.438952+0000","modified":"2024-03-31T15:14:08.651850+0000","last_up_change":"2024-03-31T15:03:22.885535+0000","last_in_change":"2024-03-31T15:03:13.941667+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-31T15:03:22.550787+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-03-31T15:14:06.417304+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"108","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-03-31T15:14:07.622562+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"108","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"92a66450-ef19-40d8-ac3d-d760b23b0bda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6801","nonce":932712244}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6803","nonce":932712244}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6807","nonce":932712244}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6805","nonce":932712244}]},"public_addr":"172.21.15.16:6801/932712244","cluster_addr":"172.21.15.16:6803/932712244","heartbeat_back_addr":"172.21.15.16:6807/932712244","heartbeat_front_addr":"172.21.15.16:6805/932712244","state":["exists","up"]},{"osd":1,"uuid":"da9d1f2b-1968-4959-b843-faa231988e92","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6810","nonce":1398737533}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6811","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6813","nonce":1398737533}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6821","nonce":1398737533}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6815","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6816","nonce":1398737533}]},"public_addr":"172.21.15.16:6810/1398737533","cluster_addr":"172.21.15.16:6813/1398737533","heartbeat_back_addr":"172.21.15.16:6821/1398737533","heartbeat_front_addr":"172.21.15.16:6816/1398737533","state":["exists","up"]},{"osd":2,"uuid":"01a73662-9a21-4b1f-bcbb-42b9636a4638","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6819","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6822","nonce":3018526646}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6825","nonce":3018526646}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6831","nonce":3018526646}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6827","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6829","nonce":3018526646}]},"public_addr":"172.21.15.16:6822/3018526646","cluster_addr":"172.21.15.16:6825/3018526646","heartbeat_back_addr":"172.21.15.16:6831/3018526646","heartbeat_front_addr":"172.21.15.16:6829/3018526646","state":["exists","up"]},{"osd":3,"uuid":"c7771fd3-e18e-4500-ac38-23cf10ebcd75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6809","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6812","nonce":3360352974}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6817","nonce":3360352974}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6828","nonce":3360352974}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6823","nonce":3360352974}]},"public_addr":"172.21.15.16:6812/3360352974","cluster_addr":"172.21.15.16:6817/3360352974","heartbeat_back_addr":"172.21.15.16:6828/3360352974","heartbeat_front_addr":"172.21.15.16:6823/3360352974","state":["exists","up"]},{"osd":4,"uuid":"88260730-1482-4fbd-a40f-cb1124e9df4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6801","nonce":1856844856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6803","nonce":1856844856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6807","nonce":1856844856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6805","nonce":1856844856}]},"public_addr":"172.21.15.89:6801/1856844856","cluster_addr":"172.21.15.89:6803/1856844856","heartbeat_back_addr":"172.21.15.89:6807/1856844856","heartbeat_front_addr":"172.21.15.89:6805/1856844856","state":["exists","up"]},{"osd":5,"uuid":"4cb2c00c-fae0-4dd8-9d27-ec4f0093ccf7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6814","nonce":50395527}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6817","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6820","nonce":50395527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6831","nonce":50395527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6823","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6826","nonce":50395527}]},"public_addr":"172.21.15.89:6814/50395527","cluster_addr":"172.21.15.89:6820/50395527","heartbeat_back_addr":"172.21.15.89:6831/50395527","heartbeat_front_addr":"172.21.15.89:6826/50395527","state":["exists","up"]},{"osd":6,"uuid":"81ce0c19-2554-4cae-acbe-66a05e9e9b9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6809","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6812","nonce":3470471752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6815","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6818","nonce":3470471752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6827","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6829","nonce":3470471752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6821","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6824","nonce":3470471752}]},"public_addr":"172.21.15.89:6812/3470471752","cluster_addr":"172.21.15.89:6818/3470471752","heartbeat_back_addr":"172.21.15.89:6829/3470471752","heartbeat_front_addr":"172.21.15.89:6824/3470471752","state":["exists","up"]},{"osd":7,"uuid":"85e4313b-d37d-442e-8042-153bf20a847d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6811","nonce":1796778630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6813","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6816","nonce":1796778630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6825","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6828","nonce":1796778630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6819","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6822","nonce":1796778630}]},"public_addr":"172.21.15.89:6811/1796778630","cluster_addr":"172.21.15.89:6816/1796778630","heartbeat_back_addr":"172.21.15.89:6828/1796778630","heartbeat_front_addr":"172.21.15.89:6822/1796778630","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.439088+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.529970+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.680512+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:21.044161+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.582111+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.536558+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.409522+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.590229+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-31T15:14:09.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:09.642+0000 7f106ae00640 1 -- 172.21.15.16:0/1609275925 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f10400683b0 msgr2=0x7f104006a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:09.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:09.642+0000 7f106ae00640 1 --2- 172.21.15.16:0/1609275925 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f10400683b0 0x7f104006a870 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f1064143f40 tx=0x7f104c007d60 comp rx=0 tx=0).stop 2024-03-31T15:14:09.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:09.642+0000 7f106ae00640 1 -- 172.21.15.16:0/1609275925 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f10640728b0 msgr2=0x7f106415bc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:09.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:09.642+0000 7f106ae00640 1 --2- 172.21.15.16:0/1609275925 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f10640728b0 0x7f106415bc80 secure :-1 s=READY pgs=1638 cs=0 l=1 rev1=1 crypto rx=0x7f105401a0b0 tx=0x7f105401acc0 comp rx=0 tx=0).stop 2024-03-31T15:14:09.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:09.642+0000 7f106ae00640 1 -- 172.21.15.16:0/1609275925 shutdown_connections 2024-03-31T15:14:09.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:09.642+0000 7f106ae00640 1 --2- 172.21.15.16:0/1609275925 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f10400683b0 0x7f104006a870 unknown :-1 s=CLOSED pgs=698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:09.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:09.642+0000 7f106ae00640 1 --2- 172.21.15.16:0/1609275925 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f1064145d10 0x7f1064162a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:09.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:09.642+0000 7f106ae00640 1 --2- 172.21.15.16:0/1609275925 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f10640b23d0 0x7f1064165910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:09.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:09.642+0000 7f106ae00640 1 --2- 172.21.15.16:0/1609275925 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f10640728b0 0x7f106415bc80 unknown :-1 s=CLOSED pgs=1638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:09.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:09.642+0000 7f106ae00640 1 -- 172.21.15.16:0/1609275925 >> 172.21.15.16:0/1609275925 conn(0x7f106406b620 msgr2=0x7f10640cff50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:14:09.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:09.642+0000 7f106ae00640 1 -- 172.21.15.16:0/1609275925 shutdown_connections 2024-03-31T15:14:09.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:09.642+0000 7f106ae00640 1 -- 172.21.15.16:0/1609275925 wait complete. 2024-03-31T15:14:09.655 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-31T15:14:09.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:09.802+0000 7f9f5ba00640 1 Processor -- start 2024-03-31T15:14:09.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:09.802+0000 7f9f5ba00640 1 -- start start 2024-03-31T15:14:09.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:09.802+0000 7f9f5ba00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f9f54070030 0x7f9f54162180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:09.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:09.802+0000 7f9f5ba00640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9f540cbe90 con 0x7f9f5406dbc0 2024-03-31T15:14:09.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:09.802+0000 7f9f5ba00640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9f540cc170 con 0x7f9f5415f650 2024-03-31T15:14:09.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:09.802+0000 7f9f5ba00640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f9f5415f1c0 con 0x7f9f54070030 2024-03-31T15:14:09.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:09.802+0000 7f9f59a00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f9f54070030 0x7f9f54162180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:09.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:09.802+0000 7f9f59000640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f9f5406dbc0 0x7f9f540cb440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:57600/0 (socket says 172.21.15.16:57600) 2024-03-31T15:14:09.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:09.802+0000 7f9f59000640 1 -- 172.21.15.16:0/3286625847 learned_addr learned my addr 172.21.15.16:0/3286625847 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:14:09.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:09.802+0000 7f9f59a00640 1 -- 172.21.15.16:0/3286625847 >> v1:172.21.15.89:6789/0 conn(0x7f9f5415f650 legacy=0x7f9f5406f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:09.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:09.802+0000 7f9f59a00640 1 -- 172.21.15.16:0/3286625847 >> v1:172.21.15.16:6789/0 conn(0x7f9f5406dbc0 legacy=0x7f9f540cb440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:09.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:09.802+0000 7f9f59a00640 1 -- 172.21.15.16:0/3286625847 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f5415ee50 con 0x7f9f54070030 2024-03-31T15:14:09.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:09.802+0000 7f9f53e00640 1 -- 172.21.15.16:0/3286625847 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2925528879 0 0) 0x7f9f540cbe90 con 0x7f9f5406dbc0 2024-03-31T15:14:09.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:09.806+0000 7f9f59a00640 1 --2- 172.21.15.16:0/3286625847 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9f54070030 0x7f9f54162180 secure :-1 s=READY pgs=1640 cs=0 l=1 rev1=1 crypto rx=0x7f9f4c009d90 tx=0x7f9f4c01c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=20799abf1d571cd2 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:09.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:09.806+0000 7f9f53e00640 1 -- 172.21.15.16:0/3286625847 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f4c0b2070 con 0x7f9f54070030 2024-03-31T15:14:09.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:09.806+0000 7f9f53e00640 1 -- 172.21.15.16:0/3286625847 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9f4c0a5960 con 0x7f9f54070030 2024-03-31T15:14:09.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:09.806+0000 7f9f53e00640 1 -- 172.21.15.16:0/3286625847 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f4c0a5d30 con 0x7f9f54070030 2024-03-31T15:14:09.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:09.806+0000 7f9f5ba00640 1 -- 172.21.15.16:0/3286625847 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9f54070030 msgr2=0x7f9f54162180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:09.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:09.806+0000 7f9f5ba00640 1 --2- 172.21.15.16:0/3286625847 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9f54070030 0x7f9f54162180 secure :-1 s=READY pgs=1640 cs=0 l=1 rev1=1 crypto rx=0x7f9f4c009d90 tx=0x7f9f4c01c8e0 comp rx=0 tx=0).stop 2024-03-31T15:14:09.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:09.806+0000 7f9f5ba00640 1 -- 172.21.15.16:0/3286625847 shutdown_connections 2024-03-31T15:14:09.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:09.806+0000 7f9f5ba00640 1 --2- 172.21.15.16:0/3286625847 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9f54070030 0x7f9f54162180 unknown :-1 s=CLOSED pgs=1640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:09.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:09.806+0000 7f9f5ba00640 1 -- 172.21.15.16:0/3286625847 >> 172.21.15.16:0/3286625847 conn(0x7f9f54067330 msgr2=0x7f9f54067740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:14:09.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:09.806+0000 7f9f5ba00640 1 -- 172.21.15.16:0/3286625847 shutdown_connections 2024-03-31T15:14:09.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:09.806+0000 7f9f5ba00640 1 -- 172.21.15.16:0/3286625847 wait complete. 2024-03-31T15:14:09.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:09.810+0000 7f9f5ba00640 1 Processor -- start 2024-03-31T15:14:09.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:09.810+0000 7f9f5ba00640 1 -- start start 2024-03-31T15:14:09.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:09.810+0000 7f9f5ba00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9f5406dbc0 0x7f9f540b35a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:09.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:09.810+0000 7f9f5ba00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9f54070030 0x7f9f540b4490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:09.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:09.810+0000 7f9f5ba00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f9f5415f650 0x7f9f54140b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:09.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:09.810+0000 7f9f5ba00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f9f5406c950 con 0x7f9f5406dbc0 2024-03-31T15:14:09.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:09.810+0000 7f9f5ba00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f9f5406dfa0 con 0x7f9f54070030 2024-03-31T15:14:09.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:09.810+0000 7f9f5ba00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f9f5406cad0 con 0x7f9f5415f650 2024-03-31T15:14:09.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:09.810+0000 7f9f59000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9f5406dbc0 0x7f9f540b35a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:09.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:09.810+0000 7f9f59000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9f5406dbc0 0x7f9f540b35a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50074/0 (socket says 172.21.15.16:50074) 2024-03-31T15:14:09.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:09.810+0000 7f9f59000640 1 -- 172.21.15.16:0/1258314564 learned_addr learned my addr 172.21.15.16:0/1258314564 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:14:09.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:09.810+0000 7f9f59a00640 1 --2- 172.21.15.16:0/1258314564 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f9f5415f650 0x7f9f54140b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:09.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:09.810+0000 7f9f59000640 1 -- 172.21.15.16:0/1258314564 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f9f5415f650 msgr2=0x7f9f54140b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:09.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:09.810+0000 7f9f4be00640 1 --2- 172.21.15.16:0/1258314564 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9f54070030 0x7f9f540b4490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:09.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:09.810+0000 7f9f59000640 1 --2- 172.21.15.16:0/1258314564 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f9f5415f650 0x7f9f54140b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:09.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:09.810+0000 7f9f59000640 1 -- 172.21.15.16:0/1258314564 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9f54070030 msgr2=0x7f9f540b4490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:09.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:09.810+0000 7f9f59000640 1 --2- 172.21.15.16:0/1258314564 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9f54070030 0x7f9f540b4490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:09.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:09.810+0000 7f9f59000640 1 -- 172.21.15.16:0/1258314564 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f5415d4d0 con 0x7f9f5406dbc0 2024-03-31T15:14:09.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:09.810+0000 7f9f59a00640 1 --2- 172.21.15.16:0/1258314564 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f9f5415f650 0x7f9f54140b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:14:09.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:09.810+0000 7f9f4be00640 1 --2- 172.21.15.16:0/1258314564 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9f54070030 0x7f9f540b4490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:14:09.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:09.810+0000 7f9f59000640 1 --2- 172.21.15.16:0/1258314564 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9f5406dbc0 0x7f9f540b35a0 secure :-1 s=READY pgs=1641 cs=0 l=1 rev1=1 crypto rx=0x7f9f4400c440 tx=0x7f9f4400c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:09.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:09.810+0000 7f9f51600640 1 -- 172.21.15.16:0/1258314564 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f44018070 con 0x7f9f5406dbc0 2024-03-31T15:14:09.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:09.810+0000 7f9f51600640 1 -- 172.21.15.16:0/1258314564 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9f440074e0 con 0x7f9f5406dbc0 2024-03-31T15:14:09.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:09.810+0000 7f9f51600640 1 -- 172.21.15.16:0/1258314564 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f44004a10 con 0x7f9f5406dbc0 2024-03-31T15:14:09.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:09.810+0000 7f9f5ba00640 1 -- 172.21.15.16:0/1258314564 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9f541410a0 con 0x7f9f5406dbc0 2024-03-31T15:14:09.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:09.810+0000 7f9f5ba00640 1 -- 172.21.15.16:0/1258314564 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9f540b2ac0 con 0x7f9f5406dbc0 2024-03-31T15:14:09.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:09.814+0000 7f9f5ba00640 1 -- 172.21.15.16:0/1258314564 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9f54069950 con 0x7f9f5406dbc0 2024-03-31T15:14:09.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:09.814+0000 7f9f51600640 1 -- 172.21.15.16:0/1258314564 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f9f4401b070 con 0x7f9f5406dbc0 2024-03-31T15:14:09.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:09.814+0000 7f9f51600640 1 --2- 172.21.15.16:0/1258314564 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f9f30068380 0x7f9f3006a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:09.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:09.814+0000 7f9f51600640 1 -- 172.21.15.16:0/1258314564 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 5709+0+0 (secure 0 0 0) 0x7f9f44085b30 con 0x7f9f5406dbc0 2024-03-31T15:14:09.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:09.814+0000 7f9f4be00640 1 --2- 172.21.15.16:0/1258314564 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f9f30068380 0x7f9f3006a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:09.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:09.814+0000 7f9f4be00640 1 --2- 172.21.15.16:0/1258314564 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f9f30068380 0x7f9f3006a840 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f9f540cea20 tx=0x7f9f3c009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:09.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:09.818+0000 7f9f51600640 1 -- 172.21.15.16:0/1258314564 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f9f44056180 con 0x7f9f5406dbc0 2024-03-31T15:14:09.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:09.982+0000 7f9f5ba00640 1 -- 172.21.15.16:0/1258314564 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f9f540cf380 con 0x7f9f5406dbc0 2024-03-31T15:14:09.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:09.986+0000 7f9f51600640 1 -- 172.21.15.16:0/1258314564 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v108) ==== 74+0+16568 (secure 0 0 0) 0x7f9f44055a80 con 0x7f9f5406dbc0 2024-03-31T15:14:09.978 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:14:09.978 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":108,"fsid":"286eb5da-fb47-4550-b290-d9519c13b9a5","created":"2024-03-31T15:03:11.438952+0000","modified":"2024-03-31T15:14:08.651850+0000","last_up_change":"2024-03-31T15:03:22.885535+0000","last_in_change":"2024-03-31T15:03:13.941667+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-31T15:03:22.550787+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-03-31T15:14:06.417304+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"108","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-03-31T15:14:07.622562+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"108","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"92a66450-ef19-40d8-ac3d-d760b23b0bda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6801","nonce":932712244}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6803","nonce":932712244}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6807","nonce":932712244}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6805","nonce":932712244}]},"public_addr":"172.21.15.16:6801/932712244","cluster_addr":"172.21.15.16:6803/932712244","heartbeat_back_addr":"172.21.15.16:6807/932712244","heartbeat_front_addr":"172.21.15.16:6805/932712244","state":["exists","up"]},{"osd":1,"uuid":"da9d1f2b-1968-4959-b843-faa231988e92","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6810","nonce":1398737533}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6811","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6813","nonce":1398737533}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6821","nonce":1398737533}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6815","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6816","nonce":1398737533}]},"public_addr":"172.21.15.16:6810/1398737533","cluster_addr":"172.21.15.16:6813/1398737533","heartbeat_back_addr":"172.21.15.16:6821/1398737533","heartbeat_front_addr":"172.21.15.16:6816/1398737533","state":["exists","up"]},{"osd":2,"uuid":"01a73662-9a21-4b1f-bcbb-42b9636a4638","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6819","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6822","nonce":3018526646}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6825","nonce":3018526646}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6831","nonce":3018526646}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6827","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6829","nonce":3018526646}]},"public_addr":"172.21.15.16:6822/3018526646","cluster_addr":"172.21.15.16:6825/3018526646","heartbeat_back_addr":"172.21.15.16:6831/3018526646","heartbeat_front_addr":"172.21.15.16:6829/3018526646","state":["exists","up"]},{"osd":3,"uuid":"c7771fd3-e18e-4500-ac38-23cf10ebcd75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6809","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6812","nonce":3360352974}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6817","nonce":3360352974}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6828","nonce":3360352974}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6823","nonce":3360352974}]},"public_addr":"172.21.15.16:6812/3360352974","cluster_addr":"172.21.15.16:6817/3360352974","heartbeat_back_addr":"172.21.15.16:6828/3360352974","heartbeat_front_addr":"172.21.15.16:6823/3360352974","state":["exists","up"]},{"osd":4,"uuid":"88260730-1482-4fbd-a40f-cb1124e9df4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6801","nonce":1856844856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6803","nonce":1856844856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6807","nonce":1856844856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6805","nonce":1856844856}]},"public_addr":"172.21.15.89:6801/1856844856","cluster_addr":"172.21.15.89:6803/1856844856","heartbeat_back_addr":"172.21.15.89:6807/1856844856","heartbeat_front_addr":"172.21.15.89:6805/1856844856","state":["exists","up"]},{"osd":5,"uuid":"4cb2c00c-fae0-4dd8-9d27-ec4f0093ccf7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6814","nonce":50395527}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6817","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6820","nonce":50395527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6831","nonce":50395527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6823","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6826","nonce":50395527}]},"public_addr":"172.21.15.89:6814/50395527","cluster_addr":"172.21.15.89:6820/50395527","heartbeat_back_addr":"172.21.15.89:6831/50395527","heartbeat_front_addr":"172.21.15.89:6826/50395527","state":["exists","up"]},{"osd":6,"uuid":"81ce0c19-2554-4cae-acbe-66a05e9e9b9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6809","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6812","nonce":3470471752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6815","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6818","nonce":3470471752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6827","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6829","nonce":3470471752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6821","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6824","nonce":3470471752}]},"public_addr":"172.21.15.89:6812/3470471752","cluster_addr":"172.21.15.89:6818/3470471752","heartbeat_back_addr":"172.21.15.89:6829/3470471752","heartbeat_front_addr":"172.21.15.89:6824/3470471752","state":["exists","up"]},{"osd":7,"uuid":"85e4313b-d37d-442e-8042-153bf20a847d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6811","nonce":1796778630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6813","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6816","nonce":1796778630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6825","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6828","nonce":1796778630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6819","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6822","nonce":1796778630}]},"public_addr":"172.21.15.89:6811/1796778630","cluster_addr":"172.21.15.89:6816/1796778630","heartbeat_back_addr":"172.21.15.89:6828/1796778630","heartbeat_front_addr":"172.21.15.89:6822/1796778630","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.439088+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.529970+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.680512+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:21.044161+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.582111+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.536558+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.409522+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.590229+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-31T15:14:09.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:09.986+0000 7f9f5ba00640 1 -- 172.21.15.16:0/1258314564 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f9f30068380 msgr2=0x7f9f3006a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:09.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:09.986+0000 7f9f5ba00640 1 --2- 172.21.15.16:0/1258314564 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f9f30068380 0x7f9f3006a840 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f9f540cea20 tx=0x7f9f3c009040 comp rx=0 tx=0).stop 2024-03-31T15:14:09.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:09.986+0000 7f9f5ba00640 1 -- 172.21.15.16:0/1258314564 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9f5406dbc0 msgr2=0x7f9f540b35a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:09.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:09.986+0000 7f9f5ba00640 1 --2- 172.21.15.16:0/1258314564 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9f5406dbc0 0x7f9f540b35a0 secure :-1 s=READY pgs=1641 cs=0 l=1 rev1=1 crypto rx=0x7f9f4400c440 tx=0x7f9f4400c900 comp rx=0 tx=0).stop 2024-03-31T15:14:09.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:09.986+0000 7f9f5ba00640 1 -- 172.21.15.16:0/1258314564 shutdown_connections 2024-03-31T15:14:09.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:09.986+0000 7f9f5ba00640 1 --2- 172.21.15.16:0/1258314564 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f9f30068380 0x7f9f3006a840 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:09.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:09.986+0000 7f9f5ba00640 1 --2- 172.21.15.16:0/1258314564 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f9f5415f650 0x7f9f54140b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:09.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:09.986+0000 7f9f5ba00640 1 --2- 172.21.15.16:0/1258314564 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9f54070030 0x7f9f540b4490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:09.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:09.986+0000 7f9f5ba00640 1 --2- 172.21.15.16:0/1258314564 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9f5406dbc0 0x7f9f540b35a0 unknown :-1 s=CLOSED pgs=1641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:09.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:09.986+0000 7f9f5ba00640 1 -- 172.21.15.16:0/1258314564 >> 172.21.15.16:0/1258314564 conn(0x7f9f54067330 msgr2=0x7f9f5415e1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:14:09.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:09.986+0000 7f9f5ba00640 1 -- 172.21.15.16:0/1258314564 shutdown_connections 2024-03-31T15:14:09.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:09.986+0000 7f9f5ba00640 1 -- 172.21.15.16:0/1258314564 wait complete. 2024-03-31T15:14:10.002 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-31T15:14:10.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:10.154+0000 7f78bd600640 1 Processor -- start 2024-03-31T15:14:10.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:10.154+0000 7f78bd600640 1 -- start start 2024-03-31T15:14:10.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:10.154+0000 7f78bd600640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f78b813dff0 0x7f78b806d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:10.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:10.154+0000 7f78bd600640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f78b813e3d0 0x7f78b806da10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:10.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:10.154+0000 7f78bd600640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f78b80cc9d0 0x7f78b80cce60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:10.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:10.154+0000 7f78bd600640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f78b815f730 con 0x7f78b813dff0 2024-03-31T15:14:10.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:10.154+0000 7f78bd600640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f78b815fbc0 con 0x7f78b813e3d0 2024-03-31T15:14:10.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:10.154+0000 7f78bd600640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f78b815ee00 con 0x7f78b80cc9d0 2024-03-31T15:14:10.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:10.154+0000 7f78b6a00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f78b813dff0 0x7f78b806d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:10.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:10.154+0000 7f78b6a00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f78b813dff0 0x7f78b806d4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50088/0 (socket says 172.21.15.16:50088) 2024-03-31T15:14:10.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:10.154+0000 7f78b6a00640 1 -- 172.21.15.16:0/2303417735 learned_addr learned my addr 172.21.15.16:0/2303417735 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:14:10.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:10.154+0000 7f78b7400640 1 --2- 172.21.15.16:0/2303417735 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f78b80cc9d0 0x7f78b80cce60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:10.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:10.154+0000 7f78afe00640 1 --2- 172.21.15.16:0/2303417735 >> v2:172.21.15.89:3300/0 conn(0x7f78b813e3d0 0x7f78b806da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:10.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:10.154+0000 7f78b6a00640 1 -- 172.21.15.16:0/2303417735 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f78b80cc9d0 msgr2=0x7f78b80cce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:14:10.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:10.154+0000 7f78b6a00640 1 --2- 172.21.15.16:0/2303417735 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f78b80cc9d0 0x7f78b80cce60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:10.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:10.154+0000 7f78b6a00640 1 -- 172.21.15.16:0/2303417735 >> v2:172.21.15.89:3300/0 conn(0x7f78b813e3d0 msgr2=0x7f78b806da10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:14:10.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:10.154+0000 7f78b6a00640 1 --2- 172.21.15.16:0/2303417735 >> v2:172.21.15.89:3300/0 conn(0x7f78b813e3d0 0x7f78b806da10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:10.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:10.154+0000 7f78b6a00640 1 -- 172.21.15.16:0/2303417735 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78b815f0e0 con 0x7f78b813dff0 2024-03-31T15:14:10.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:10.154+0000 7f78afe00640 1 --2- 172.21.15.16:0/2303417735 >> v2:172.21.15.89:3300/0 conn(0x7f78b813e3d0 0x7f78b806da10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:14:10.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:10.158+0000 7f78b6a00640 1 --2- 172.21.15.16:0/2303417735 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f78b813dff0 0x7f78b806d4d0 secure :-1 s=READY pgs=1642 cs=0 l=1 rev1=1 crypto rx=0x7f78b806f620 tx=0x7f78a801c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=80325bd9d3912869 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:10.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:10.158+0000 7f78b6000640 1 -- 172.21.15.16:0/2303417735 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78a80b0070 con 0x7f78b813dff0 2024-03-31T15:14:10.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:10.158+0000 7f78b6000640 1 -- 172.21.15.16:0/2303417735 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f78a80ab600 con 0x7f78b813dff0 2024-03-31T15:14:10.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:10.158+0000 7f78b6000640 1 -- 172.21.15.16:0/2303417735 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78a80ab9d0 con 0x7f78b813dff0 2024-03-31T15:14:10.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:10.158+0000 7f78bd600640 1 -- 172.21.15.16:0/2303417735 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f78b813dff0 msgr2=0x7f78b806d4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:10.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:10.158+0000 7f78bd600640 1 --2- 172.21.15.16:0/2303417735 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f78b813dff0 0x7f78b806d4d0 secure :-1 s=READY pgs=1642 cs=0 l=1 rev1=1 crypto rx=0x7f78b806f620 tx=0x7f78a801c490 comp rx=0 tx=0).stop 2024-03-31T15:14:10.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:10.158+0000 7f78bd600640 1 -- 172.21.15.16:0/2303417735 shutdown_connections 2024-03-31T15:14:10.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:10.158+0000 7f78bd600640 1 --2- 172.21.15.16:0/2303417735 >> v2:172.21.15.89:3300/0 conn(0x7f78b813e3d0 0x7f78b806da10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:10.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:10.158+0000 7f78bd600640 1 --2- 172.21.15.16:0/2303417735 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f78b80cc9d0 0x7f78b80cce60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:10.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:10.158+0000 7f78bd600640 1 --2- 172.21.15.16:0/2303417735 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f78b813dff0 0x7f78b806d4d0 unknown :-1 s=CLOSED pgs=1642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:10.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:10.158+0000 7f78bd600640 1 -- 172.21.15.16:0/2303417735 >> 172.21.15.16:0/2303417735 conn(0x7f78b806ee40 msgr2=0x7f78b806b620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:14:10.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:10.158+0000 7f78bd600640 1 -- 172.21.15.16:0/2303417735 shutdown_connections 2024-03-31T15:14:10.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:10.158+0000 7f78bd600640 1 -- 172.21.15.16:0/2303417735 wait complete. 2024-03-31T15:14:10.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:10.162+0000 7f78bd600640 1 Processor -- start 2024-03-31T15:14:10.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:10.162+0000 7f78bd600640 1 -- start start 2024-03-31T15:14:10.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:10.162+0000 7f78bd600640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f78b80cc9d0 0x7f78b81550e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:10.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:10.162+0000 7f78bd600640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f78b813dff0 0x7f78b81560b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:10.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:10.162+0000 7f78bd600640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f78b813e3d0 0x7f78b815ce50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:10.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:10.162+0000 7f78bd600640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f78b8073200 con 0x7f78b813e3d0 2024-03-31T15:14:10.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:10.162+0000 7f78bd600640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f78b8073080 con 0x7f78b80cc9d0 2024-03-31T15:14:10.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:10.162+0000 7f78bd600640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f78b8073380 con 0x7f78b813dff0 2024-03-31T15:14:10.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:10.162+0000 7f78b7400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f78b813e3d0 0x7f78b815ce50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:10.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:10.162+0000 7f78b7400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f78b813e3d0 0x7f78b815ce50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50096/0 (socket says 172.21.15.16:50096) 2024-03-31T15:14:10.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:10.162+0000 7f78b7400640 1 -- 172.21.15.16:0/629680908 learned_addr learned my addr 172.21.15.16:0/629680908 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:14:10.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:10.162+0000 7f78b6a00640 1 --2- 172.21.15.16:0/629680908 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f78b80cc9d0 0x7f78b81550e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:10.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:10.162+0000 7f78afe00640 1 --2- 172.21.15.16:0/629680908 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f78b813dff0 0x7f78b81560b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:10.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:10.162+0000 7f78b7400640 1 -- 172.21.15.16:0/629680908 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f78b813dff0 msgr2=0x7f78b81560b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:10.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:10.162+0000 7f78b7400640 1 --2- 172.21.15.16:0/629680908 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f78b813dff0 0x7f78b81560b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:10.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:10.162+0000 7f78b7400640 1 -- 172.21.15.16:0/629680908 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f78b80cc9d0 msgr2=0x7f78b81550e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:10.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:10.162+0000 7f78b7400640 1 --2- 172.21.15.16:0/629680908 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f78b80cc9d0 0x7f78b81550e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:10.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:10.162+0000 7f78b7400640 1 -- 172.21.15.16:0/629680908 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78b81f6f20 con 0x7f78b813e3d0 2024-03-31T15:14:10.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:10.162+0000 7f78b6a00640 1 --2- 172.21.15.16:0/629680908 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f78b80cc9d0 0x7f78b81550e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:14:10.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:10.162+0000 7f78b7400640 1 --2- 172.21.15.16:0/629680908 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f78b813e3d0 0x7f78b815ce50 secure :-1 s=READY pgs=1643 cs=0 l=1 rev1=1 crypto rx=0x7f78b000eed0 tx=0x7f78b000b480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:10.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:10.162+0000 7f78aea00640 1 -- 172.21.15.16:0/629680908 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78b0018020 con 0x7f78b813e3d0 2024-03-31T15:14:10.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:10.162+0000 7f78aea00640 1 -- 172.21.15.16:0/629680908 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f78b000bd20 con 0x7f78b813e3d0 2024-03-31T15:14:10.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:10.162+0000 7f78aea00640 1 -- 172.21.15.16:0/629680908 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78b0014640 con 0x7f78b813e3d0 2024-03-31T15:14:10.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:10.162+0000 7f78bd600640 1 -- 172.21.15.16:0/629680908 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f78b81f7200 con 0x7f78b813e3d0 2024-03-31T15:14:10.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:10.162+0000 7f78bd600640 1 -- 172.21.15.16:0/629680908 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f78b81f77c0 con 0x7f78b813e3d0 2024-03-31T15:14:10.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:10.162+0000 7f78bd600640 1 -- 172.21.15.16:0/629680908 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f78b81f7990 con 0x7f78b813e3d0 2024-03-31T15:14:10.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:10.166+0000 7f78aea00640 1 -- 172.21.15.16:0/629680908 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f78b001b020 con 0x7f78b813e3d0 2024-03-31T15:14:10.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:10.166+0000 7f78aea00640 1 --2- 172.21.15.16:0/629680908 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f7894068380 0x7f789406a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:10.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:10.166+0000 7f78aea00640 1 -- 172.21.15.16:0/629680908 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 5709+0+0 (secure 0 0 0) 0x7f78b0087090 con 0x7f78b813e3d0 2024-03-31T15:14:10.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:10.166+0000 7f78b6a00640 1 --2- 172.21.15.16:0/629680908 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f7894068380 0x7f789406a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:10.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:10.166+0000 7f78b6a00640 1 --2- 172.21.15.16:0/629680908 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f7894068380 0x7f789406a840 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f78a80ad000 tx=0x7f78a80a6000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:10.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:10.170+0000 7f78aea00640 1 -- 172.21.15.16:0/629680908 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f78b0022070 con 0x7f78b813e3d0 2024-03-31T15:14:10.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:10.374+0000 7f78bd600640 1 -- 172.21.15.16:0/629680908 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f78b8069950 con 0x7f78b813e3d0 2024-03-31T15:14:11.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:11.270+0000 7f78aea00640 1 -- 172.21.15.16:0/629680908 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v98) ==== 118+0+0 (secure 0 0 0) 0x7f78b00575b0 con 0x7f78b813e3d0 2024-03-31T15:14:11.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:11.274+0000 7f78bd600640 1 -- 172.21.15.16:0/629680908 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f7894068380 msgr2=0x7f789406a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:11.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:11.274+0000 7f78bd600640 1 --2- 172.21.15.16:0/629680908 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f7894068380 0x7f789406a840 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f78a80ad000 tx=0x7f78a80a6000 comp rx=0 tx=0).stop 2024-03-31T15:14:11.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:11.274+0000 7f78bd600640 1 -- 172.21.15.16:0/629680908 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f78b813e3d0 msgr2=0x7f78b815ce50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:11.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:11.274+0000 7f78bd600640 1 --2- 172.21.15.16:0/629680908 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f78b813e3d0 0x7f78b815ce50 secure :-1 s=READY pgs=1643 cs=0 l=1 rev1=1 crypto rx=0x7f78b000eed0 tx=0x7f78b000b480 comp rx=0 tx=0).stop 2024-03-31T15:14:11.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:11.274+0000 7f78bd600640 1 -- 172.21.15.16:0/629680908 shutdown_connections 2024-03-31T15:14:11.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:11.274+0000 7f78bd600640 1 --2- 172.21.15.16:0/629680908 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f7894068380 0x7f789406a840 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:11.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:11.274+0000 7f78bd600640 1 --2- 172.21.15.16:0/629680908 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f78b813e3d0 0x7f78b815ce50 unknown :-1 s=CLOSED pgs=1643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:11.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:11.274+0000 7f78bd600640 1 --2- 172.21.15.16:0/629680908 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f78b813dff0 0x7f78b81560b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:11.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:11.274+0000 7f78bd600640 1 --2- 172.21.15.16:0/629680908 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f78b80cc9d0 0x7f78b81550e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:11.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:11.274+0000 7f78bd600640 1 -- 172.21.15.16:0/629680908 >> 172.21.15.16:0/629680908 conn(0x7f78b806ee40 msgr2=0x7f78b80aee30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:14:11.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:11.274+0000 7f78bd600640 1 -- 172.21.15.16:0/629680908 shutdown_connections 2024-03-31T15:14:11.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:11.274+0000 7f78bd600640 1 -- 172.21.15.16:0/629680908 wait complete. 2024-03-31T15:14:11.287 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-03-31T15:14:11.287 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-31T15:14:11.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:11.442+0000 7fdf27c00640 1 Processor -- start 2024-03-31T15:14:11.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:11.442+0000 7fdf27c00640 1 -- start start 2024-03-31T15:14:11.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:11.442+0000 7fdf27c00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fdf201411b0 0x7fdf2006f920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:11.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:11.442+0000 7fdf27c00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fdf20141590 0x7fdf2006fe60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:11.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:11.442+0000 7fdf27c00640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdf20144980 con 0x7fdf20143c30 2024-03-31T15:14:11.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:11.442+0000 7fdf27c00640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7fdf2015f910 con 0x7fdf20141590 2024-03-31T15:14:11.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:11.442+0000 7fdf27c00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fdf20162910 con 0x7fdf201411b0 2024-03-31T15:14:11.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:11.442+0000 7fdf25c00640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fdf20143c30 0x7fdf201440a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:57614/0 (socket says 172.21.15.16:57614) 2024-03-31T15:14:11.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:11.442+0000 7fdf25c00640 1 -- 172.21.15.16:0/204201216 learned_addr learned my addr 172.21.15.16:0/204201216 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:14:11.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:11.442+0000 7fdf25200640 1 --2- 172.21.15.16:0/204201216 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fdf201411b0 0x7fdf2006f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:11.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:11.442+0000 7fdf17e00640 1 --2- 172.21.15.16:0/204201216 >> v2:172.21.15.89:3300/0 conn(0x7fdf20141590 0x7fdf2006fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:11.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:11.442+0000 7fdf1fe00640 1 -- 172.21.15.16:0/204201216 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3676672225 0 0) 0x7fdf20144980 con 0x7fdf20143c30 2024-03-31T15:14:11.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:11.442+0000 7fdf1fe00640 1 -- 172.21.15.16:0/204201216 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdf04003530 con 0x7fdf20143c30 2024-03-31T15:14:11.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:11.442+0000 7fdf25200640 1 -- 172.21.15.16:0/204201216 >> v2:172.21.15.89:3300/0 conn(0x7fdf20141590 msgr2=0x7fdf2006fe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:14:11.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:11.442+0000 7fdf25200640 1 --2- 172.21.15.16:0/204201216 >> v2:172.21.15.89:3300/0 conn(0x7fdf20141590 0x7fdf2006fe60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:11.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:11.442+0000 7fdf25200640 1 -- 172.21.15.16:0/204201216 >> v1:172.21.15.16:6789/0 conn(0x7fdf20143c30 legacy=0x7fdf201440a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:11.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:11.442+0000 7fdf25200640 1 -- 172.21.15.16:0/204201216 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf20161fe0 con 0x7fdf201411b0 2024-03-31T15:14:11.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:11.442+0000 7fdf17e00640 1 --2- 172.21.15.16:0/204201216 >> v2:172.21.15.89:3300/0 conn(0x7fdf20141590 0x7fdf2006fe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:14:11.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:11.446+0000 7fdf25200640 1 --2- 172.21.15.16:0/204201216 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fdf201411b0 0x7fdf2006f920 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fdf100097c0 tx=0x7fdf1001c4a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d2080ef85600e543 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:11.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:11.446+0000 7fdf1fe00640 1 -- 172.21.15.16:0/204201216 <== mon.2 v2:172.21.15.89:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf100b2070 con 0x7fdf201411b0 2024-03-31T15:14:11.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:11.446+0000 7fdf1fe00640 1 -- 172.21.15.16:0/204201216 <== mon.2 v2:172.21.15.89:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdf100ad640 con 0x7fdf201411b0 2024-03-31T15:14:11.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:11.446+0000 7fdf1fe00640 1 -- 172.21.15.16:0/204201216 <== mon.2 v2:172.21.15.89:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf100ada10 con 0x7fdf201411b0 2024-03-31T15:14:11.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:11.446+0000 7fdf27c00640 1 -- 172.21.15.16:0/204201216 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fdf201411b0 msgr2=0x7fdf2006f920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:11.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:11.446+0000 7fdf27c00640 1 --2- 172.21.15.16:0/204201216 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fdf201411b0 0x7fdf2006f920 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fdf100097c0 tx=0x7fdf1001c4a0 comp rx=0 tx=0).stop 2024-03-31T15:14:11.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:11.446+0000 7fdf27c00640 1 -- 172.21.15.16:0/204201216 shutdown_connections 2024-03-31T15:14:11.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:11.446+0000 7fdf27c00640 1 --2- 172.21.15.16:0/204201216 >> v2:172.21.15.89:3300/0 conn(0x7fdf20141590 0x7fdf2006fe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:11.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:11.446+0000 7fdf27c00640 1 --2- 172.21.15.16:0/204201216 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fdf201411b0 0x7fdf2006f920 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:11.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:11.446+0000 7fdf27c00640 1 -- 172.21.15.16:0/204201216 >> 172.21.15.16:0/204201216 conn(0x7fdf20067930 msgr2=0x7fdf200b3ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:14:11.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:11.446+0000 7fdf27c00640 1 -- 172.21.15.16:0/204201216 shutdown_connections 2024-03-31T15:14:11.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:11.446+0000 7fdf27c00640 1 -- 172.21.15.16:0/204201216 wait complete. 2024-03-31T15:14:11.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:11.450+0000 7fdf27c00640 1 Processor -- start 2024-03-31T15:14:11.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:11.450+0000 7fdf27c00640 1 -- start start 2024-03-31T15:14:11.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:11.450+0000 7fdf27c00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fdf201411b0 0x7fdf2013e880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:11.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:11.450+0000 7fdf27c00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdf20141590 0x7fdf20168960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:11.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:11.450+0000 7fdf27c00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdf20143c30 0x7fdf2006d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:11.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:11.450+0000 7fdf27c00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fdf2015f290 con 0x7fdf20141590 2024-03-31T15:14:11.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:11.450+0000 7fdf27c00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fdf2015f110 con 0x7fdf20143c30 2024-03-31T15:14:11.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:11.450+0000 7fdf27c00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fdf2015f410 con 0x7fdf201411b0 2024-03-31T15:14:11.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:11.450+0000 7fdf17e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdf20141590 0x7fdf20168960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:11.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:11.454+0000 7fdf17e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdf20141590 0x7fdf20168960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50102/0 (socket says 172.21.15.16:50102) 2024-03-31T15:14:11.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:11.454+0000 7fdf17e00640 1 -- 172.21.15.16:0/2211528203 learned_addr learned my addr 172.21.15.16:0/2211528203 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:14:11.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:11.454+0000 7fdf25200640 1 --2- 172.21.15.16:0/2211528203 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fdf201411b0 0x7fdf2013e880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:11.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:11.454+0000 7fdf25c00640 1 --2- 172.21.15.16:0/2211528203 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdf20143c30 0x7fdf2006d5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:11.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:11.454+0000 7fdf17e00640 1 -- 172.21.15.16:0/2211528203 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fdf201411b0 msgr2=0x7fdf2013e880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:11.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:11.454+0000 7fdf17e00640 1 --2- 172.21.15.16:0/2211528203 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fdf201411b0 0x7fdf2013e880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:11.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:11.454+0000 7fdf17e00640 1 -- 172.21.15.16:0/2211528203 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdf20143c30 msgr2=0x7fdf2006d5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:11.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:11.454+0000 7fdf17e00640 1 --2- 172.21.15.16:0/2211528203 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdf20143c30 0x7fdf2006d5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:11.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:11.454+0000 7fdf17e00640 1 -- 172.21.15.16:0/2211528203 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf2015bd10 con 0x7fdf20141590 2024-03-31T15:14:11.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:11.454+0000 7fdf17e00640 1 --2- 172.21.15.16:0/2211528203 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdf20141590 0x7fdf20168960 secure :-1 s=READY pgs=1645 cs=0 l=1 rev1=1 crypto rx=0x7fdf0800ea00 tx=0x7fdf0800eec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:11.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:11.454+0000 7fdf1d600640 1 -- 172.21.15.16:0/2211528203 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf08017070 con 0x7fdf20141590 2024-03-31T15:14:11.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:11.454+0000 7fdf1d600640 1 -- 172.21.15.16:0/2211528203 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdf0800ca60 con 0x7fdf20141590 2024-03-31T15:14:11.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:11.454+0000 7fdf1d600640 1 -- 172.21.15.16:0/2211528203 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf0800ce50 con 0x7fdf20141590 2024-03-31T15:14:11.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:11.454+0000 7fdf27c00640 1 -- 172.21.15.16:0/2211528203 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdf2013f430 con 0x7fdf20141590 2024-03-31T15:14:11.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:11.454+0000 7fdf27c00640 1 -- 172.21.15.16:0/2211528203 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdf2013f9f0 con 0x7fdf20141590 2024-03-31T15:14:11.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:11.454+0000 7fdf27c00640 1 -- 172.21.15.16:0/2211528203 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdf20069950 con 0x7fdf20141590 2024-03-31T15:14:11.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:11.458+0000 7fdf1d600640 1 -- 172.21.15.16:0/2211528203 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fdf0801a070 con 0x7fdf20141590 2024-03-31T15:14:11.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:11.458+0000 7fdf1d600640 1 --2- 172.21.15.16:0/2211528203 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fdefc068380 0x7fdefc06a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:11.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:11.458+0000 7fdf1d600640 1 -- 172.21.15.16:0/2211528203 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 5709+0+0 (secure 0 0 0) 0x7fdf08086500 con 0x7fdf20141590 2024-03-31T15:14:11.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:11.458+0000 7fdf25200640 1 --2- 172.21.15.16:0/2211528203 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fdefc068380 0x7fdefc06a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:11.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:11.458+0000 7fdf25200640 1 --2- 172.21.15.16:0/2211528203 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fdefc068380 0x7fdefc06a840 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7fdf100af000 tx=0x7fdf10008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:11.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:11.462+0000 7fdf1d600640 1 -- 172.21.15.16:0/2211528203 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fdf2015bd10 con 0x7fdf20141590 2024-03-31T15:14:11.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:11.662+0000 7fdf27c00640 1 -- 172.21.15.16:0/2211528203 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7fdf20070a30 con 0x7fdf20141590 2024-03-31T15:14:12.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:12.274+0000 7fdf1d600640 1 -- 172.21.15.16:0/2211528203 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v99) ==== 122+0+0 (secure 0 0 0) 0x7fdf080563e0 con 0x7fdf20141590 2024-03-31T15:14:12.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:12.274+0000 7fdf27c00640 1 -- 172.21.15.16:0/2211528203 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fdefc068380 msgr2=0x7fdefc06a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:12.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:12.274+0000 7fdf27c00640 1 --2- 172.21.15.16:0/2211528203 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fdefc068380 0x7fdefc06a840 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7fdf100af000 tx=0x7fdf10008040 comp rx=0 tx=0).stop 2024-03-31T15:14:12.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:12.274+0000 7fdf27c00640 1 -- 172.21.15.16:0/2211528203 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdf20141590 msgr2=0x7fdf20168960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:12.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:12.274+0000 7fdf27c00640 1 --2- 172.21.15.16:0/2211528203 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdf20141590 0x7fdf20168960 secure :-1 s=READY pgs=1645 cs=0 l=1 rev1=1 crypto rx=0x7fdf0800ea00 tx=0x7fdf0800eec0 comp rx=0 tx=0).stop 2024-03-31T15:14:12.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:12.274+0000 7fdf27c00640 1 -- 172.21.15.16:0/2211528203 shutdown_connections 2024-03-31T15:14:12.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:12.274+0000 7fdf27c00640 1 --2- 172.21.15.16:0/2211528203 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fdefc068380 0x7fdefc06a840 unknown :-1 s=CLOSED pgs=701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:12.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:12.274+0000 7fdf27c00640 1 --2- 172.21.15.16:0/2211528203 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdf20143c30 0x7fdf2006d5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:12.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:12.274+0000 7fdf27c00640 1 --2- 172.21.15.16:0/2211528203 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdf20141590 0x7fdf20168960 unknown :-1 s=CLOSED pgs=1645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:12.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:12.274+0000 7fdf27c00640 1 --2- 172.21.15.16:0/2211528203 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fdf201411b0 0x7fdf2013e880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:12.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:12.274+0000 7fdf27c00640 1 -- 172.21.15.16:0/2211528203 >> 172.21.15.16:0/2211528203 conn(0x7fdf20067930 msgr2=0x7fdf20143130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:14:12.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:12.274+0000 7fdf27c00640 1 -- 172.21.15.16:0/2211528203 shutdown_connections 2024-03-31T15:14:12.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:12.274+0000 7fdf27c00640 1 -- 172.21.15.16:0/2211528203 wait complete. 2024-03-31T15:14:12.289 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-31T15:14:12.289 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-31T15:14:12.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:12.438+0000 7f7f83e00640 1 Processor -- start 2024-03-31T15:14:12.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:12.438+0000 7f7f83e00640 1 -- start start 2024-03-31T15:14:12.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:12.438+0000 7f7f83e00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f7f7c145d10 0x7f7f7c0cf790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:12.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:12.438+0000 7f7f83e00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f7f7c0728b0 0x7f7f7c0b1cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:12.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:12.438+0000 7f7f83e00640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7f7c160e70 con 0x7f7f7c0b2200 2024-03-31T15:14:12.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:12.438+0000 7f7f83e00640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f7f7c1602f0 con 0x7f7f7c145d10 2024-03-31T15:14:12.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:12.438+0000 7f7f83e00640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f7f7c1605d0 con 0x7f7f7c0728b0 2024-03-31T15:14:12.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:12.438+0000 7f7f81400640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f7f7c145d10 0x7f7f7c0cf790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:12.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:12.438+0000 7f7f81e00640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f7f7c0b2200 0x7f7f7c15cd50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:57622/0 (socket says 172.21.15.16:57622) 2024-03-31T15:14:12.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:12.438+0000 7f7f81e00640 1 -- 172.21.15.16:0/850972464 learned_addr learned my addr 172.21.15.16:0/850972464 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:14:12.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:12.438+0000 7f7f80a00640 1 --2- 172.21.15.16:0/850972464 >> v2:172.21.15.89:3300/0 conn(0x7f7f7c0728b0 0x7f7f7c0b1cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:12.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:12.438+0000 7f7f81400640 1 -- 172.21.15.16:0/850972464 >> v2:172.21.15.89:3300/0 conn(0x7f7f7c0728b0 msgr2=0x7f7f7c0b1cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:14:12.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:12.438+0000 7f7f81400640 1 --2- 172.21.15.16:0/850972464 >> v2:172.21.15.89:3300/0 conn(0x7f7f7c0728b0 0x7f7f7c0b1cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:12.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:12.438+0000 7f7f81400640 1 -- 172.21.15.16:0/850972464 >> v1:172.21.15.16:6789/0 conn(0x7f7f7c0b2200 legacy=0x7f7f7c15cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:12.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:12.438+0000 7f7f81400640 1 -- 172.21.15.16:0/850972464 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f7c0cfcd0 con 0x7f7f7c145d10 2024-03-31T15:14:12.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:12.442+0000 7f7f81400640 1 --2- 172.21.15.16:0/850972464 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7f7c145d10 0x7f7f7c0cf790 secure :-1 s=READY pgs=1647 cs=0 l=1 rev1=1 crypto rx=0x7f7f7c0b2980 tx=0x7f7f7001c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d4497f3cf9814a server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:12.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:12.442+0000 7f7f7be00640 1 -- 172.21.15.16:0/850972464 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f700b0070 con 0x7f7f7c145d10 2024-03-31T15:14:12.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:12.442+0000 7f7f7be00640 1 -- 172.21.15.16:0/850972464 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7f700ab600 con 0x7f7f7c145d10 2024-03-31T15:14:12.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:12.442+0000 7f7f7be00640 1 -- 172.21.15.16:0/850972464 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f700ab9d0 con 0x7f7f7c145d10 2024-03-31T15:14:12.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:12.442+0000 7f7f83e00640 1 -- 172.21.15.16:0/850972464 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7f7c145d10 msgr2=0x7f7f7c0cf790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:12.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:12.442+0000 7f7f83e00640 1 --2- 172.21.15.16:0/850972464 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7f7c145d10 0x7f7f7c0cf790 secure :-1 s=READY pgs=1647 cs=0 l=1 rev1=1 crypto rx=0x7f7f7c0b2980 tx=0x7f7f7001c480 comp rx=0 tx=0).stop 2024-03-31T15:14:12.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:12.442+0000 7f7f83e00640 1 -- 172.21.15.16:0/850972464 shutdown_connections 2024-03-31T15:14:12.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:12.442+0000 7f7f83e00640 1 --2- 172.21.15.16:0/850972464 >> v2:172.21.15.89:3300/0 conn(0x7f7f7c0728b0 0x7f7f7c0b1cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:12.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:12.442+0000 7f7f83e00640 1 --2- 172.21.15.16:0/850972464 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7f7c145d10 0x7f7f7c0cf790 unknown :-1 s=CLOSED pgs=1647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:12.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:12.442+0000 7f7f83e00640 1 -- 172.21.15.16:0/850972464 >> 172.21.15.16:0/850972464 conn(0x7f7f7c06b620 msgr2=0x7f7f7c06ba30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:14:12.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:12.442+0000 7f7f83e00640 1 -- 172.21.15.16:0/850972464 shutdown_connections 2024-03-31T15:14:12.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:12.442+0000 7f7f83e00640 1 -- 172.21.15.16:0/850972464 wait complete. 2024-03-31T15:14:12.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:12.446+0000 7f7f83e00640 1 Processor -- start 2024-03-31T15:14:12.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:12.446+0000 7f7f83e00640 1 -- start start 2024-03-31T15:14:12.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:12.446+0000 7f7f83e00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7f7c0728b0 0x7f7f7c141c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:12.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:12.446+0000 7f7f83e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7f7c0b2200 0x7f7f7c13b970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:12.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:12.446+0000 7f7f83e00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f7f7c145d10 0x7f7f7c143f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:12.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:12.446+0000 7f7f83e00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f7f7c15d870 con 0x7f7f7c0b2200 2024-03-31T15:14:12.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:12.446+0000 7f7f83e00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f7f7c0b2610 con 0x7f7f7c0728b0 2024-03-31T15:14:12.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:12.446+0000 7f7f83e00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f7f7c0cf350 con 0x7f7f7c145d10 2024-03-31T15:14:12.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:12.446+0000 7f7f80a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7f7c0b2200 0x7f7f7c13b970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:12.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:12.446+0000 7f7f80a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7f7c0b2200 0x7f7f7c13b970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50114/0 (socket says 172.21.15.16:50114) 2024-03-31T15:14:12.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:12.446+0000 7f7f80a00640 1 -- 172.21.15.16:0/3233555214 learned_addr learned my addr 172.21.15.16:0/3233555214 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:14:12.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:12.446+0000 7f7f81400640 1 --2- 172.21.15.16:0/3233555214 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7f7c0728b0 0x7f7f7c141c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:12.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:12.446+0000 7f7f81e00640 1 --2- 172.21.15.16:0/3233555214 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f7f7c145d10 0x7f7f7c143f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:12.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:12.446+0000 7f7f80a00640 1 -- 172.21.15.16:0/3233555214 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f7f7c145d10 msgr2=0x7f7f7c143f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:12.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:12.446+0000 7f7f80a00640 1 --2- 172.21.15.16:0/3233555214 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f7f7c145d10 0x7f7f7c143f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:12.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:12.446+0000 7f7f80a00640 1 -- 172.21.15.16:0/3233555214 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7f7c0728b0 msgr2=0x7f7f7c141c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:12.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:12.446+0000 7f7f80a00640 1 --2- 172.21.15.16:0/3233555214 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7f7c0728b0 0x7f7f7c141c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:12.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:12.446+0000 7f7f80a00640 1 -- 172.21.15.16:0/3233555214 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f7c13beb0 con 0x7f7f7c0b2200 2024-03-31T15:14:12.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:12.446+0000 7f7f80a00640 1 --2- 172.21.15.16:0/3233555214 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7f7c0b2200 0x7f7f7c13b970 secure :-1 s=READY pgs=1648 cs=0 l=1 rev1=1 crypto rx=0x7f7f6800d800 tx=0x7f7f6800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:12.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:12.446+0000 7f7f79600640 1 -- 172.21.15.16:0/3233555214 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f6800b6c0 con 0x7f7f7c0b2200 2024-03-31T15:14:12.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:12.446+0000 7f7f79600640 1 -- 172.21.15.16:0/3233555214 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7f68019070 con 0x7f7f7c0b2200 2024-03-31T15:14:12.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:12.446+0000 7f7f79600640 1 -- 172.21.15.16:0/3233555214 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f6800ba60 con 0x7f7f7c0b2200 2024-03-31T15:14:12.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:12.446+0000 7f7f83e00640 1 -- 172.21.15.16:0/3233555214 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7f7c161f00 con 0x7f7f7c0b2200 2024-03-31T15:14:12.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:12.446+0000 7f7f83e00640 1 -- 172.21.15.16:0/3233555214 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7f7c163310 con 0x7f7f7c0b2200 2024-03-31T15:14:12.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:12.446+0000 7f7f83e00640 1 -- 172.21.15.16:0/3233555214 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7f7c069950 con 0x7f7f7c0b2200 2024-03-31T15:14:12.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:12.450+0000 7f7f79600640 1 -- 172.21.15.16:0/3233555214 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f7f6800bc00 con 0x7f7f7c0b2200 2024-03-31T15:14:12.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:12.450+0000 7f7f79600640 1 --2- 172.21.15.16:0/3233555214 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f7f5c068380 0x7f7f5c06a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:12.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:12.450+0000 7f7f79600640 1 -- 172.21.15.16:0/3233555214 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 5709+0+0 (secure 0 0 0) 0x7f7f68086e40 con 0x7f7f7c0b2200 2024-03-31T15:14:12.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:12.450+0000 7f7f81400640 1 --2- 172.21.15.16:0/3233555214 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f7f5c068380 0x7f7f5c06a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:12.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:12.450+0000 7f7f81400640 1 --2- 172.21.15.16:0/3233555214 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f7f5c068380 0x7f7f5c06a840 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f7f700ad000 tx=0x7f7f70008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:12.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:12.454+0000 7f7f79600640 1 -- 172.21.15.16:0/3233555214 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f7f68057360 con 0x7f7f7c0b2200 2024-03-31T15:14:12.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:12.654+0000 7f7f83e00640 1 -- 172.21.15.16:0/3233555214 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7f7c15db20 con 0x7f7f7c0b2200 2024-03-31T15:14:12.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:12.654+0000 7f7f79600640 1 -- 172.21.15.16:0/3233555214 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 99 v99) ==== 94+0+5180 (secure 0 0 0) 0x7f7f68056c60 con 0x7f7f7c0b2200 2024-03-31T15:14:12.648 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:14:12.648 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":99,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7631,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6836/1628897112","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":1628897112},{"type":"v1","addr":"172.21.15.89:6836","nonce":1628897112}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":94},{"gid":8746,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/1036724837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6835","nonce":1036724837},{"type":"v1","addr":"172.21.15.89:6837","nonce":1036724837}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":94},{"gid":9192,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6834/472139147","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":472139147},{"type":"v1","addr":"172.21.15.16:6834","nonce":472139147}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":94}],"filesystems":[{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-31T15:14:08.652180+0000","modified":"2024-03-31T15:14:11.668085+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":9195},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9195":{"gid":9195,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.16:6835/2564952016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6833","nonce":2564952016},{"type":"v1","addr":"172.21.15.16:6835","nonce":2564952016}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9195,"qdb_cluster":[9195]},"id":8}]} 2024-03-31T15:14:12.648 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 99 2024-03-31T15:14:12.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:12.658+0000 7f7f83e00640 1 -- 172.21.15.16:0/3233555214 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f7f5c068380 msgr2=0x7f7f5c06a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:12.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:12.658+0000 7f7f83e00640 1 --2- 172.21.15.16:0/3233555214 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f7f5c068380 0x7f7f5c06a840 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f7f700ad000 tx=0x7f7f70008040 comp rx=0 tx=0).stop 2024-03-31T15:14:12.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:12.658+0000 7f7f83e00640 1 -- 172.21.15.16:0/3233555214 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7f7c0b2200 msgr2=0x7f7f7c13b970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:12.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:12.658+0000 7f7f83e00640 1 --2- 172.21.15.16:0/3233555214 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7f7c0b2200 0x7f7f7c13b970 secure :-1 s=READY pgs=1648 cs=0 l=1 rev1=1 crypto rx=0x7f7f6800d800 tx=0x7f7f6800dcc0 comp rx=0 tx=0).stop 2024-03-31T15:14:12.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:12.658+0000 7f7f83e00640 1 -- 172.21.15.16:0/3233555214 shutdown_connections 2024-03-31T15:14:12.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:12.658+0000 7f7f83e00640 1 --2- 172.21.15.16:0/3233555214 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f7f5c068380 0x7f7f5c06a840 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:12.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:12.658+0000 7f7f83e00640 1 --2- 172.21.15.16:0/3233555214 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f7f7c145d10 0x7f7f7c143f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:12.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:12.658+0000 7f7f83e00640 1 --2- 172.21.15.16:0/3233555214 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7f7c0b2200 0x7f7f7c13b970 unknown :-1 s=CLOSED pgs=1648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:12.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:12.658+0000 7f7f83e00640 1 --2- 172.21.15.16:0/3233555214 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7f7c0728b0 0x7f7f7c141c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:12.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:12.658+0000 7f7f83e00640 1 -- 172.21.15.16:0/3233555214 >> 172.21.15.16:0/3233555214 conn(0x7f7f7c06b620 msgr2=0x7f7f7c15e5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:14:12.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:12.658+0000 7f7f83e00640 1 -- 172.21.15.16:0/3233555214 shutdown_connections 2024-03-31T15:14:12.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:12.658+0000 7f7f83e00640 1 -- 172.21.15.16:0/3233555214 wait complete. 2024-03-31T15:14:12.670 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-31T15:14:12.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:12.818+0000 7fd76f200640 1 Processor -- start 2024-03-31T15:14:12.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:12.818+0000 7fd76f200640 1 -- start start 2024-03-31T15:14:12.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:12.818+0000 7fd76f200640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fd7680b3f20 0x7fd76806b620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:12.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:12.818+0000 7fd76f200640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd76815e9b0 con 0x7fd76806dc20 2024-03-31T15:14:12.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:12.818+0000 7fd76f200640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd76815ec70 con 0x7fd7681629f0 2024-03-31T15:14:12.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:12.818+0000 7fd76f200640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fd76815ef50 con 0x7fd7680b3f20 2024-03-31T15:14:12.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:12.818+0000 7fd76d200640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fd76806dc20 0x7fd76806bb60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:57628/0 (socket says 172.21.15.16:57628) 2024-03-31T15:14:12.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:12.818+0000 7fd76d200640 1 -- 172.21.15.16:0/2794862809 learned_addr learned my addr 172.21.15.16:0/2794862809 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:14:12.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:12.818+0000 7fd75fe00640 1 --2- 172.21.15.16:0/2794862809 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fd7680b3f20 0x7fd76806b620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:12.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:12.818+0000 7fd767400640 1 -- 172.21.15.16:0/2794862809 <== mon.1 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1195289858 0 0) 0x7fd76815ec70 con 0x7fd7681629f0 2024-03-31T15:14:12.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:12.822+0000 7fd767400640 1 -- 172.21.15.16:0/2794862809 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd74c003530 con 0x7fd7681629f0 2024-03-31T15:14:12.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:12.822+0000 7fd767400640 1 -- 172.21.15.16:0/2794862809 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3440331732 0 0) 0x7fd76815e9b0 con 0x7fd76806dc20 2024-03-31T15:14:12.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:12.822+0000 7fd75fe00640 1 -- 172.21.15.16:0/2794862809 >> v1:172.21.15.89:6789/0 conn(0x7fd7681629f0 legacy=0x7fd7680b3810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:12.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:12.822+0000 7fd75fe00640 1 -- 172.21.15.16:0/2794862809 >> v1:172.21.15.16:6789/0 conn(0x7fd76806dc20 legacy=0x7fd76806bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:12.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:12.822+0000 7fd75fe00640 1 -- 172.21.15.16:0/2794862809 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd76815f230 con 0x7fd7680b3f20 2024-03-31T15:14:12.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:12.822+0000 7fd75fe00640 1 --2- 172.21.15.16:0/2794862809 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fd7680b3f20 0x7fd76806b620 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fd7500097f0 tx=0x7fd76806ca80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=74a1ed1613228a3d server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:12.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:12.822+0000 7fd767400640 1 -- 172.21.15.16:0/2794862809 <== mon.2 v2:172.21.15.89:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7500ab5f0 con 0x7fd7680b3f20 2024-03-31T15:14:12.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:12.822+0000 7fd767400640 1 -- 172.21.15.16:0/2794862809 <== mon.2 v2:172.21.15.89:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd7500b0070 con 0x7fd7680b3f20 2024-03-31T15:14:12.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:12.822+0000 7fd767400640 1 -- 172.21.15.16:0/2794862809 <== mon.2 v2:172.21.15.89:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7500ab9b0 con 0x7fd7680b3f20 2024-03-31T15:14:12.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:12.822+0000 7fd76f200640 1 -- 172.21.15.16:0/2794862809 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fd7680b3f20 msgr2=0x7fd76806b620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:12.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:12.822+0000 7fd76f200640 1 --2- 172.21.15.16:0/2794862809 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fd7680b3f20 0x7fd76806b620 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fd7500097f0 tx=0x7fd76806ca80 comp rx=0 tx=0).stop 2024-03-31T15:14:12.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:12.822+0000 7fd76f200640 1 -- 172.21.15.16:0/2794862809 shutdown_connections 2024-03-31T15:14:12.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:12.822+0000 7fd76f200640 1 --2- 172.21.15.16:0/2794862809 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fd7680b3f20 0x7fd76806b620 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:12.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:12.822+0000 7fd76f200640 1 -- 172.21.15.16:0/2794862809 >> 172.21.15.16:0/2794862809 conn(0x7fd7680d0760 msgr2=0x7fd7680d0b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:14:12.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:12.826+0000 7fd76f200640 1 -- 172.21.15.16:0/2794862809 shutdown_connections 2024-03-31T15:14:12.818 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:12.826+0000 7fd76f200640 1 -- 172.21.15.16:0/2794862809 wait complete. 2024-03-31T15:14:12.818 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:12.826+0000 7fd76f200640 1 Processor -- start 2024-03-31T15:14:12.818 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:12.826+0000 7fd76f200640 1 -- start start 2024-03-31T15:14:12.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:12.826+0000 7fd76f200640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd76806dc20 0x7fd76814ba00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:12.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:12.826+0000 7fd76f200640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fd7680b3f20 0x7fd76814c1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:12.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:12.826+0000 7fd76f200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd7681629f0 0x7fd7680aefd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:12.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:12.826+0000 7fd76f200640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fd76816c4d0 con 0x7fd7681629f0 2024-03-31T15:14:12.822 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:12.826+0000 7fd76f200640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fd76816c350 con 0x7fd76806dc20 2024-03-31T15:14:12.822 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:12.826+0000 7fd76f200640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fd76816c650 con 0x7fd7680b3f20 2024-03-31T15:14:12.822 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:12.826+0000 7fd76d200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd7681629f0 0x7fd7680aefd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:12.822 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:12.826+0000 7fd76d200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd7681629f0 0x7fd7680aefd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50120/0 (socket says 172.21.15.16:50120) 2024-03-31T15:14:12.822 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:12.826+0000 7fd76d200640 1 -- 172.21.15.16:0/2139158624 learned_addr learned my addr 172.21.15.16:0/2139158624 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:14:12.822 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:12.826+0000 7fd75fe00640 1 --2- 172.21.15.16:0/2139158624 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fd7680b3f20 0x7fd76814c1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:12.822 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:12.826+0000 7fd767e00640 1 --2- 172.21.15.16:0/2139158624 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd76806dc20 0x7fd76814ba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:12.822 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:12.826+0000 7fd76d200640 1 -- 172.21.15.16:0/2139158624 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fd7680b3f20 msgr2=0x7fd76814c1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:12.822 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:12.826+0000 7fd76d200640 1 --2- 172.21.15.16:0/2139158624 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fd7680b3f20 0x7fd76814c1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:12.822 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:12.826+0000 7fd76d200640 1 -- 172.21.15.16:0/2139158624 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd76806dc20 msgr2=0x7fd76814ba00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:12.822 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:12.826+0000 7fd76d200640 1 --2- 172.21.15.16:0/2139158624 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd76806dc20 0x7fd76814ba00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:12.822 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:12.826+0000 7fd76d200640 1 -- 172.21.15.16:0/2139158624 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd768157070 con 0x7fd7681629f0 2024-03-31T15:14:12.822 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:12.826+0000 7fd76d200640 1 --2- 172.21.15.16:0/2139158624 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd7681629f0 0x7fd7680aefd0 secure :-1 s=READY pgs=1650 cs=0 l=1 rev1=1 crypto rx=0x7fd76000c6d0 tx=0x7fd76000cb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:12.822 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:12.830+0000 7fd764c00640 1 -- 172.21.15.16:0/2139158624 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd760018070 con 0x7fd7681629f0 2024-03-31T15:14:12.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:12.830+0000 7fd76f200640 1 -- 172.21.15.16:0/2139158624 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd76814bf40 con 0x7fd7681629f0 2024-03-31T15:14:12.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:12.830+0000 7fd764c00640 1 -- 172.21.15.16:0/2139158624 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd760004d70 con 0x7fd7681629f0 2024-03-31T15:14:12.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:12.830+0000 7fd764c00640 1 -- 172.21.15.16:0/2139158624 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd76000a880 con 0x7fd7681629f0 2024-03-31T15:14:12.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:12.830+0000 7fd76f200640 1 -- 172.21.15.16:0/2139158624 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd7681572e0 con 0x7fd7681629f0 2024-03-31T15:14:12.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:12.830+0000 7fd76f200640 1 -- 172.21.15.16:0/2139158624 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd768069950 con 0x7fd7681629f0 2024-03-31T15:14:12.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:12.830+0000 7fd764c00640 1 -- 172.21.15.16:0/2139158624 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fd76000ac30 con 0x7fd7681629f0 2024-03-31T15:14:12.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:12.830+0000 7fd764c00640 1 --2- 172.21.15.16:0/2139158624 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fd744068380 0x7fd74406a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:12.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:12.830+0000 7fd764c00640 1 -- 172.21.15.16:0/2139158624 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 5709+0+0 (secure 0 0 0) 0x7fd760086150 con 0x7fd7681629f0 2024-03-31T15:14:12.825 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:12.830+0000 7fd767e00640 1 --2- 172.21.15.16:0/2139158624 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fd744068380 0x7fd74406a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:12.825 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:12.834+0000 7fd767e00640 1 --2- 172.21.15.16:0/2139158624 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fd744068380 0x7fd74406a840 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7fd7580011f0 tx=0x7fd758009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:12.829 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:12.834+0000 7fd764c00640 1 -- 172.21.15.16:0/2139158624 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fd760056030 con 0x7fd7681629f0 2024-03-31T15:14:12.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:13.002+0000 7fd76f200640 1 -- 172.21.15.16:0/2139158624 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fd768157d40 con 0x7fd7681629f0 2024-03-31T15:14:12.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:13.006+0000 7fd764c00640 1 -- 172.21.15.16:0/2139158624 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v109) ==== 74+0+16555 (secure 0 0 0) 0x7fd76005a330 con 0x7fd7681629f0 2024-03-31T15:14:12.998 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:14:12.998 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":109,"fsid":"286eb5da-fb47-4550-b290-d9519c13b9a5","created":"2024-03-31T15:03:11.438952+0000","modified":"2024-03-31T15:14:10.265103+0000","last_up_change":"2024-03-31T15:03:22.885535+0000","last_in_change":"2024-03-31T15:03:13.941667+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-31T15:03:22.550787+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-03-31T15:14:06.417304+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"108","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-03-31T15:14:07.622562+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"109","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"92a66450-ef19-40d8-ac3d-d760b23b0bda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6801","nonce":932712244}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6803","nonce":932712244}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6807","nonce":932712244}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6805","nonce":932712244}]},"public_addr":"172.21.15.16:6801/932712244","cluster_addr":"172.21.15.16:6803/932712244","heartbeat_back_addr":"172.21.15.16:6807/932712244","heartbeat_front_addr":"172.21.15.16:6805/932712244","state":["exists","up"]},{"osd":1,"uuid":"da9d1f2b-1968-4959-b843-faa231988e92","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6810","nonce":1398737533}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6811","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6813","nonce":1398737533}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6821","nonce":1398737533}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6815","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6816","nonce":1398737533}]},"public_addr":"172.21.15.16:6810/1398737533","cluster_addr":"172.21.15.16:6813/1398737533","heartbeat_back_addr":"172.21.15.16:6821/1398737533","heartbeat_front_addr":"172.21.15.16:6816/1398737533","state":["exists","up"]},{"osd":2,"uuid":"01a73662-9a21-4b1f-bcbb-42b9636a4638","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6819","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6822","nonce":3018526646}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6825","nonce":3018526646}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6831","nonce":3018526646}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6827","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6829","nonce":3018526646}]},"public_addr":"172.21.15.16:6822/3018526646","cluster_addr":"172.21.15.16:6825/3018526646","heartbeat_back_addr":"172.21.15.16:6831/3018526646","heartbeat_front_addr":"172.21.15.16:6829/3018526646","state":["exists","up"]},{"osd":3,"uuid":"c7771fd3-e18e-4500-ac38-23cf10ebcd75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6809","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6812","nonce":3360352974}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6817","nonce":3360352974}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6828","nonce":3360352974}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6823","nonce":3360352974}]},"public_addr":"172.21.15.16:6812/3360352974","cluster_addr":"172.21.15.16:6817/3360352974","heartbeat_back_addr":"172.21.15.16:6828/3360352974","heartbeat_front_addr":"172.21.15.16:6823/3360352974","state":["exists","up"]},{"osd":4,"uuid":"88260730-1482-4fbd-a40f-cb1124e9df4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6801","nonce":1856844856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6803","nonce":1856844856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6807","nonce":1856844856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6805","nonce":1856844856}]},"public_addr":"172.21.15.89:6801/1856844856","cluster_addr":"172.21.15.89:6803/1856844856","heartbeat_back_addr":"172.21.15.89:6807/1856844856","heartbeat_front_addr":"172.21.15.89:6805/1856844856","state":["exists","up"]},{"osd":5,"uuid":"4cb2c00c-fae0-4dd8-9d27-ec4f0093ccf7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6814","nonce":50395527}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6817","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6820","nonce":50395527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6831","nonce":50395527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6823","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6826","nonce":50395527}]},"public_addr":"172.21.15.89:6814/50395527","cluster_addr":"172.21.15.89:6820/50395527","heartbeat_back_addr":"172.21.15.89:6831/50395527","heartbeat_front_addr":"172.21.15.89:6826/50395527","state":["exists","up"]},{"osd":6,"uuid":"81ce0c19-2554-4cae-acbe-66a05e9e9b9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6809","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6812","nonce":3470471752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6815","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6818","nonce":3470471752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6827","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6829","nonce":3470471752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6821","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6824","nonce":3470471752}]},"public_addr":"172.21.15.89:6812/3470471752","cluster_addr":"172.21.15.89:6818/3470471752","heartbeat_back_addr":"172.21.15.89:6829/3470471752","heartbeat_front_addr":"172.21.15.89:6824/3470471752","state":["exists","up"]},{"osd":7,"uuid":"85e4313b-d37d-442e-8042-153bf20a847d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6811","nonce":1796778630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6813","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6816","nonce":1796778630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6825","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6828","nonce":1796778630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6819","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6822","nonce":1796778630}]},"public_addr":"172.21.15.89:6811/1796778630","cluster_addr":"172.21.15.89:6816/1796778630","heartbeat_back_addr":"172.21.15.89:6828/1796778630","heartbeat_front_addr":"172.21.15.89:6822/1796778630","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.439088+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.529970+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.680512+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:21.044161+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.582111+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.536558+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.409522+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.590229+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-31T15:14:13.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:13.006+0000 7fd76f200640 1 -- 172.21.15.16:0/2139158624 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fd744068380 msgr2=0x7fd74406a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:13.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:13.006+0000 7fd76f200640 1 --2- 172.21.15.16:0/2139158624 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fd744068380 0x7fd74406a840 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7fd7580011f0 tx=0x7fd758009040 comp rx=0 tx=0).stop 2024-03-31T15:14:13.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:13.006+0000 7fd76f200640 1 -- 172.21.15.16:0/2139158624 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd7681629f0 msgr2=0x7fd7680aefd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:13.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:13.006+0000 7fd76f200640 1 --2- 172.21.15.16:0/2139158624 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd7681629f0 0x7fd7680aefd0 secure :-1 s=READY pgs=1650 cs=0 l=1 rev1=1 crypto rx=0x7fd76000c6d0 tx=0x7fd76000cb90 comp rx=0 tx=0).stop 2024-03-31T15:14:13.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:13.006+0000 7fd76f200640 1 -- 172.21.15.16:0/2139158624 shutdown_connections 2024-03-31T15:14:13.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:13.006+0000 7fd76f200640 1 --2- 172.21.15.16:0/2139158624 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fd744068380 0x7fd74406a840 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:13.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:13.006+0000 7fd76f200640 1 --2- 172.21.15.16:0/2139158624 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd7681629f0 0x7fd7680aefd0 unknown :-1 s=CLOSED pgs=1650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:13.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:13.006+0000 7fd76f200640 1 --2- 172.21.15.16:0/2139158624 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fd7680b3f20 0x7fd76814c1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:13.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:13.006+0000 7fd76f200640 1 --2- 172.21.15.16:0/2139158624 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd76806dc20 0x7fd76814ba00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:13.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:13.006+0000 7fd76f200640 1 -- 172.21.15.16:0/2139158624 >> 172.21.15.16:0/2139158624 conn(0x7fd7680d0760 msgr2=0x7fd768071d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:14:13.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:13.006+0000 7fd76f200640 1 -- 172.21.15.16:0/2139158624 shutdown_connections 2024-03-31T15:14:13.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:13.006+0000 7fd76f200640 1 -- 172.21.15.16:0/2139158624 wait complete. 2024-03-31T15:14:13.019 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-31T15:14:13.020 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-31T15:14:13.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:13.166+0000 7fe1cce00640 1 Processor -- start 2024-03-31T15:14:13.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:13.166+0000 7fe1cce00640 1 -- start start 2024-03-31T15:14:13.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:13.166+0000 7fe1cce00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fe1c806f920 0x7fe1c806fd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:13.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:13.166+0000 7fe1cce00640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe1c80d0920 con 0x7fe1c8145e40 2024-03-31T15:14:13.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:13.166+0000 7fe1cce00640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe1c80d0c00 con 0x7fe1c80d2010 2024-03-31T15:14:13.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:13.166+0000 7fe1cce00640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7fe1c80702c0 con 0x7fe1c806f920 2024-03-31T15:14:13.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:13.166+0000 7fe1c5e00640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fe1c8145e40 0x7fe1c80cc740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:57640/0 (socket says 172.21.15.16:57640) 2024-03-31T15:14:13.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:13.166+0000 7fe1c5e00640 1 -- 172.21.15.16:0/3417692811 learned_addr learned my addr 172.21.15.16:0/3417692811 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:14:13.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:13.166+0000 7fe1bfe00640 1 --2- 172.21.15.16:0/3417692811 >> v2:172.21.15.89:3300/0 conn(0x7fe1c806f920 0x7fe1c806fd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:13.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:13.170+0000 7fe1c5400640 1 -- 172.21.15.16:0/3417692811 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 852586343 0 0) 0x7fe1c80d0920 con 0x7fe1c8145e40 2024-03-31T15:14:13.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:13.170+0000 7fe1c5400640 1 -- 172.21.15.16:0/3417692811 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe1ac003530 con 0x7fe1c8145e40 2024-03-31T15:14:13.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:13.170+0000 7fe1bfe00640 1 -- 172.21.15.16:0/3417692811 >> v1:172.21.15.89:6789/0 conn(0x7fe1c80d2010 legacy=0x7fe1c80d24a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:13.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:13.170+0000 7fe1bfe00640 1 -- 172.21.15.16:0/3417692811 >> v1:172.21.15.16:6789/0 conn(0x7fe1c8145e40 legacy=0x7fe1c80cc740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:13.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:13.170+0000 7fe1c5400640 1 -- 172.21.15.16:0/3417692811 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3524860357 0 0) 0x7fe1ac003530 con 0x7fe1c8145e40 2024-03-31T15:14:13.164 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:13.170+0000 7fe1bfe00640 1 -- 172.21.15.16:0/3417692811 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe1c80d1810 con 0x7fe1c806f920 2024-03-31T15:14:13.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:13.170+0000 7fe1bfe00640 1 --2- 172.21.15.16:0/3417692811 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe1c806f920 0x7fe1c806fd80 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7fe1b00097c0 tx=0x7fe1b001c3c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a019056d150509eb server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:13.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:13.170+0000 7fe1c5400640 1 -- 172.21.15.16:0/3417692811 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1b00b0070 con 0x7fe1c806f920 2024-03-31T15:14:13.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:13.170+0000 7fe1c5400640 1 -- 172.21.15.16:0/3417692811 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe1b00ab400 con 0x7fe1c806f920 2024-03-31T15:14:13.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:13.170+0000 7fe1c5400640 1 -- 172.21.15.16:0/3417692811 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1b8001680 con 0x7fe1c806f920 2024-03-31T15:14:13.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:13.174+0000 7fe1cce00640 1 -- 172.21.15.16:0/3417692811 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe1c806f920 msgr2=0x7fe1c806fd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:13.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:13.174+0000 7fe1cce00640 1 --2- 172.21.15.16:0/3417692811 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe1c806f920 0x7fe1c806fd80 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7fe1b00097c0 tx=0x7fe1b001c3c0 comp rx=0 tx=0).stop 2024-03-31T15:14:13.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:13.174+0000 7fe1cce00640 1 -- 172.21.15.16:0/3417692811 shutdown_connections 2024-03-31T15:14:13.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:13.174+0000 7fe1cce00640 1 --2- 172.21.15.16:0/3417692811 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe1c806f920 0x7fe1c806fd80 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:13.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:13.174+0000 7fe1cce00640 1 -- 172.21.15.16:0/3417692811 >> 172.21.15.16:0/3417692811 conn(0x7fe1c8067330 msgr2=0x7fe1c8067740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:14:13.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:13.174+0000 7fe1cce00640 1 -- 172.21.15.16:0/3417692811 shutdown_connections 2024-03-31T15:14:13.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:13.174+0000 7fe1cce00640 1 -- 172.21.15.16:0/3417692811 wait complete. 2024-03-31T15:14:13.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:13.174+0000 7fe1cce00640 1 Processor -- start 2024-03-31T15:14:13.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:13.174+0000 7fe1cce00640 1 -- start start 2024-03-31T15:14:13.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:13.174+0000 7fe1cce00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe1c806f920 0x7fe1c806c730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:13.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:13.174+0000 7fe1cce00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe1c80d2010 0x7fe1c806cc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:13.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:13.174+0000 7fe1cce00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fe1c8145e40 0x7fe1c81585d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:13.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:13.174+0000 7fe1cce00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fe1c80d0ee0 con 0x7fe1c806f920 2024-03-31T15:14:13.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:13.174+0000 7fe1cce00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fe1c80d06d0 con 0x7fe1c80d2010 2024-03-31T15:14:13.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:13.174+0000 7fe1cce00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fe1c80b50c0 con 0x7fe1c8145e40 2024-03-31T15:14:13.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:13.174+0000 7fe1c5e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe1c806f920 0x7fe1c806c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:13.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:13.174+0000 7fe1c5e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe1c806f920 0x7fe1c806c730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50124/0 (socket says 172.21.15.16:50124) 2024-03-31T15:14:13.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:13.174+0000 7fe1c5e00640 1 -- 172.21.15.16:0/612513817 learned_addr learned my addr 172.21.15.16:0/612513817 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:14:13.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:13.174+0000 7fe1bfe00640 1 --2- 172.21.15.16:0/612513817 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe1c80d2010 0x7fe1c806cc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:13.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:13.174+0000 7fe1c6800640 1 --2- 172.21.15.16:0/612513817 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fe1c8145e40 0x7fe1c81585d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:13.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:13.174+0000 7fe1c5e00640 1 -- 172.21.15.16:0/612513817 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fe1c8145e40 msgr2=0x7fe1c81585d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:13.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:13.174+0000 7fe1c5e00640 1 --2- 172.21.15.16:0/612513817 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fe1c8145e40 0x7fe1c81585d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:13.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:13.174+0000 7fe1c5e00640 1 -- 172.21.15.16:0/612513817 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe1c80d2010 msgr2=0x7fe1c806cc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:13.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:13.174+0000 7fe1c5e00640 1 --2- 172.21.15.16:0/612513817 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe1c80d2010 0x7fe1c806cc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:13.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:13.174+0000 7fe1c5e00640 1 -- 172.21.15.16:0/612513817 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe1c80b7130 con 0x7fe1c806f920 2024-03-31T15:14:13.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:13.174+0000 7fe1bfe00640 1 --2- 172.21.15.16:0/612513817 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe1c80d2010 0x7fe1c806cc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:14:13.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:13.174+0000 7fe1c5e00640 1 --2- 172.21.15.16:0/612513817 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe1c806f920 0x7fe1c806c730 secure :-1 s=READY pgs=1652 cs=0 l=1 rev1=1 crypto rx=0x7fe1b800c730 tx=0x7fe1b800cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:13.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:13.178+0000 7fe1be000640 1 -- 172.21.15.16:0/612513817 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1b8017070 con 0x7fe1c806f920 2024-03-31T15:14:13.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:13.178+0000 7fe1be000640 1 -- 172.21.15.16:0/612513817 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe1b8005b00 con 0x7fe1c806f920 2024-03-31T15:14:13.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:13.178+0000 7fe1be000640 1 -- 172.21.15.16:0/612513817 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1b8007070 con 0x7fe1c806f920 2024-03-31T15:14:13.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:13.178+0000 7fe1cce00640 1 -- 172.21.15.16:0/612513817 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe1c8159320 con 0x7fe1c806f920 2024-03-31T15:14:13.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:13.178+0000 7fe1cce00640 1 -- 172.21.15.16:0/612513817 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe1c80b7410 con 0x7fe1c806f920 2024-03-31T15:14:13.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:13.178+0000 7fe1cce00640 1 -- 172.21.15.16:0/612513817 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe1c81599e0 con 0x7fe1c806f920 2024-03-31T15:14:13.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:13.178+0000 7fe1be000640 1 -- 172.21.15.16:0/612513817 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fe1b801a070 con 0x7fe1c806f920 2024-03-31T15:14:13.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:13.178+0000 7fe1be000640 1 --2- 172.21.15.16:0/612513817 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fe1a40683b0 0x7fe1a406a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:13.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:13.178+0000 7fe1be000640 1 -- 172.21.15.16:0/612513817 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 5709+0+0 (secure 0 0 0) 0x7fe1b8085f30 con 0x7fe1c806f920 2024-03-31T15:14:13.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:13.178+0000 7fe1bfe00640 1 --2- 172.21.15.16:0/612513817 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fe1a40683b0 0x7fe1a406a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:13.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:13.182+0000 7fe1bfe00640 1 --2- 172.21.15.16:0/612513817 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fe1a40683b0 0x7fe1a406a870 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7fe1b00ad000 tx=0x7fe1b00a6000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:13.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:13.182+0000 7fe1be000640 1 -- 172.21.15.16:0/612513817 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fe1c81599e0 con 0x7fe1c806f920 2024-03-31T15:14:13.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:13.346+0000 7fe1cce00640 1 -- 172.21.15.16:0/612513817 --> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fe1c80b6ee0 con 0x7fe1a40683b0 2024-03-31T15:14:13.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:13.350+0000 7fe1be000640 1 -- 172.21.15.16:0/612513817 <== mgr.4100 v2:172.21.15.89:6832/1549407273 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+250056 (secure 0 0 0) 0x7fe1c80b6ee0 con 0x7fe1a40683b0 2024-03-31T15:14:13.347 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:14:13.349 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-03-31T15:14:13.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:13.354+0000 7fe1cce00640 1 -- 172.21.15.16:0/612513817 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fe1a40683b0 msgr2=0x7fe1a406a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:13.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:13.354+0000 7fe1cce00640 1 --2- 172.21.15.16:0/612513817 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fe1a40683b0 0x7fe1a406a870 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7fe1b00ad000 tx=0x7fe1b00a6000 comp rx=0 tx=0).stop 2024-03-31T15:14:13.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:13.354+0000 7fe1cce00640 1 -- 172.21.15.16:0/612513817 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe1c806f920 msgr2=0x7fe1c806c730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:13.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:13.354+0000 7fe1cce00640 1 --2- 172.21.15.16:0/612513817 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe1c806f920 0x7fe1c806c730 secure :-1 s=READY pgs=1652 cs=0 l=1 rev1=1 crypto rx=0x7fe1b800c730 tx=0x7fe1b800cbf0 comp rx=0 tx=0).stop 2024-03-31T15:14:13.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:13.354+0000 7fe1cce00640 1 -- 172.21.15.16:0/612513817 shutdown_connections 2024-03-31T15:14:13.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:13.354+0000 7fe1cce00640 1 --2- 172.21.15.16:0/612513817 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fe1a40683b0 0x7fe1a406a870 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:13.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:13.354+0000 7fe1cce00640 1 --2- 172.21.15.16:0/612513817 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fe1c8145e40 0x7fe1c81585d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:13.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:13.354+0000 7fe1cce00640 1 --2- 172.21.15.16:0/612513817 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe1c80d2010 0x7fe1c806cc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:13.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:13.354+0000 7fe1cce00640 1 --2- 172.21.15.16:0/612513817 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe1c806f920 0x7fe1c806c730 unknown :-1 s=CLOSED pgs=1652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:13.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:13.354+0000 7fe1cce00640 1 -- 172.21.15.16:0/612513817 >> 172.21.15.16:0/612513817 conn(0x7fe1c8067330 msgr2=0x7fe1c8160610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:14:13.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:13.354+0000 7fe1cce00640 1 -- 172.21.15.16:0/612513817 shutdown_connections 2024-03-31T15:14:13.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:13.354+0000 7fe1cce00640 1 -- 172.21.15.16:0/612513817 wait complete. 2024-03-31T15:14:13.368 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":427,"stamp":"2024-03-31T15:14:12.693312+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592196,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":179,"num_write_kb":2429,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":136,"ondisk_log_size":136,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":104,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":265232,"kb_used_data":11616,"kb_used_omap":12,"kb_used_meta":253491,"kb_avail":754709488,"statfs":{"total":773094113280,"available":772822515712,"internally_reserved":0,"allocated":11894784,"data_stored":5792070,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":259575344},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":32,"apply_latency_ms":32,"commit_latency_ns":32000000,"apply_latency_ns":32000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1828,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.446790"},"pg_stats":[{"pgid":"17.3f","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:10.892463+0000","last_change":"2024-03-31T15:14:09.277746+0000","last_active":"2024-03-31T15:14:10.892463+0000","last_peered":"2024-03-31T15:14:10.892463+0000","last_clean":"2024-03-31T15:14:10.892463+0000","last_became_active":"2024-03-31T15:14:09.277207+0000","last_became_peered":"2024-03-31T15:14:09.277207+0000","last_unstale":"2024-03-31T15:14:10.892463+0000","last_undegraded":"2024-03-31T15:14:10.892463+0000","last_fullsized":"2024-03-31T15:14:10.892463+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T17:42:57.076247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3e","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-03-31T15:14:08.253600+0000","last_change":"2024-03-31T15:14:08.253600+0000","last_active":"2024-03-31T15:14:08.242094+0000","last_peered":"2024-03-31T15:14:08.242094+0000","last_clean":"2024-03-31T15:14:08.242094+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:14:08.253600+0000","last_undegraded":"2024-03-31T15:14:08.253600+0000","last_fullsized":"2024-03-31T15:14:08.253600+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.3d","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-03-31T15:14:08.258560+0000","last_change":"2024-03-31T15:14:08.258560+0000","last_active":"2024-03-31T15:14:08.242094+0000","last_peered":"2024-03-31T15:14:08.242094+0000","last_clean":"2024-03-31T15:14:08.242094+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:14:08.258560+0000","last_undegraded":"2024-03-31T15:14:08.258560+0000","last_fullsized":"2024-03-31T15:14:08.258560+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.3c","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:10.892010+0000","last_change":"2024-03-31T15:14:09.276706+0000","last_active":"2024-03-31T15:14:10.892010+0000","last_peered":"2024-03-31T15:14:10.892010+0000","last_clean":"2024-03-31T15:14:10.892010+0000","last_became_active":"2024-03-31T15:14:09.275545+0000","last_became_peered":"2024-03-31T15:14:09.275545+0000","last_unstale":"2024-03-31T15:14:10.892010+0000","last_undegraded":"2024-03-31T15:14:10.892010+0000","last_fullsized":"2024-03-31T15:14:10.892010+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T19:37:12.397888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3b","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-03-31T15:14:08.255591+0000","last_change":"2024-03-31T15:14:08.255591+0000","last_active":"2024-03-31T15:14:08.242094+0000","last_peered":"2024-03-31T15:14:08.242094+0000","last_clean":"2024-03-31T15:14:08.242094+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:14:08.255591+0000","last_undegraded":"2024-03-31T15:14:08.255591+0000","last_fullsized":"2024-03-31T15:14:08.255591+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.3a","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:11.390994+0000","last_change":"2024-03-31T15:14:09.283260+0000","last_active":"2024-03-31T15:14:11.390994+0000","last_peered":"2024-03-31T15:14:11.390994+0000","last_clean":"2024-03-31T15:14:11.390994+0000","last_became_active":"2024-03-31T15:14:09.282995+0000","last_became_peered":"2024-03-31T15:14:09.282995+0000","last_unstale":"2024-03-31T15:14:11.390994+0000","last_undegraded":"2024-03-31T15:14:11.390994+0000","last_fullsized":"2024-03-31T15:14:11.390994+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:27:30.858064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.39","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-03-31T15:14:08.259762+0000","last_change":"2024-03-31T15:14:08.259762+0000","last_active":"2024-03-31T15:14:08.242094+0000","last_peered":"2024-03-31T15:14:08.242094+0000","last_clean":"2024-03-31T15:14:08.242094+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:14:08.259762+0000","last_undegraded":"2024-03-31T15:14:08.259762+0000","last_fullsized":"2024-03-31T15:14:08.259762+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.38","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-03-31T15:14:08.258038+0000","last_change":"2024-03-31T15:14:08.258038+0000","last_active":"2024-03-31T15:14:08.242094+0000","last_peered":"2024-03-31T15:14:08.242094+0000","last_clean":"2024-03-31T15:14:08.242094+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:14:08.258038+0000","last_undegraded":"2024-03-31T15:14:08.258038+0000","last_fullsized":"2024-03-31T15:14:08.258038+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.37","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:10.892467+0000","last_change":"2024-03-31T15:14:09.284680+0000","last_active":"2024-03-31T15:14:10.892467+0000","last_peered":"2024-03-31T15:14:10.892467+0000","last_clean":"2024-03-31T15:14:10.892467+0000","last_became_active":"2024-03-31T15:14:09.282533+0000","last_became_peered":"2024-03-31T15:14:09.282533+0000","last_unstale":"2024-03-31T15:14:10.892467+0000","last_undegraded":"2024-03-31T15:14:10.892467+0000","last_fullsized":"2024-03-31T15:14:10.892467+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:04:35.760087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.36","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:10.892246+0000","last_change":"2024-03-31T15:14:09.290368+0000","last_active":"2024-03-31T15:14:10.892246+0000","last_peered":"2024-03-31T15:14:10.892246+0000","last_clean":"2024-03-31T15:14:10.892246+0000","last_became_active":"2024-03-31T15:14:09.290077+0000","last_became_peered":"2024-03-31T15:14:09.290077+0000","last_unstale":"2024-03-31T15:14:10.892246+0000","last_undegraded":"2024-03-31T15:14:10.892246+0000","last_fullsized":"2024-03-31T15:14:10.892246+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:23:53.326762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.35","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-03-31T15:14:08.253324+0000","last_change":"2024-03-31T15:14:08.253324+0000","last_active":"2024-03-31T15:14:08.242094+0000","last_peered":"2024-03-31T15:14:08.242094+0000","last_clean":"2024-03-31T15:14:08.242094+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:14:08.253324+0000","last_undegraded":"2024-03-31T15:14:08.253324+0000","last_fullsized":"2024-03-31T15:14:08.253324+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.34","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:10.892056+0000","last_change":"2024-03-31T15:14:09.276847+0000","last_active":"2024-03-31T15:14:10.892056+0000","last_peered":"2024-03-31T15:14:10.892056+0000","last_clean":"2024-03-31T15:14:10.892056+0000","last_became_active":"2024-03-31T15:14:09.275249+0000","last_became_peered":"2024-03-31T15:14:09.275249+0000","last_unstale":"2024-03-31T15:14:10.892056+0000","last_undegraded":"2024-03-31T15:14:10.892056+0000","last_fullsized":"2024-03-31T15:14:10.892056+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T16:56:09.377594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.33","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-03-31T15:14:08.258326+0000","last_change":"2024-03-31T15:14:08.258326+0000","last_active":"2024-03-31T15:14:08.242094+0000","last_peered":"2024-03-31T15:14:08.242094+0000","last_clean":"2024-03-31T15:14:08.242094+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:14:08.258326+0000","last_undegraded":"2024-03-31T15:14:08.258326+0000","last_fullsized":"2024-03-31T15:14:08.258326+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.32","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-03-31T15:14:08.257741+0000","last_change":"2024-03-31T15:14:08.257741+0000","last_active":"2024-03-31T15:14:08.242094+0000","last_peered":"2024-03-31T15:14:08.242094+0000","last_clean":"2024-03-31T15:14:08.242094+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:14:08.257741+0000","last_undegraded":"2024-03-31T15:14:08.257741+0000","last_fullsized":"2024-03-31T15:14:08.257741+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.31","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:11.390625+0000","last_change":"2024-03-31T15:14:09.285784+0000","last_active":"2024-03-31T15:14:11.390625+0000","last_peered":"2024-03-31T15:14:11.390625+0000","last_clean":"2024-03-31T15:14:11.390625+0000","last_became_active":"2024-03-31T15:14:09.285374+0000","last_became_peered":"2024-03-31T15:14:09.285374+0000","last_unstale":"2024-03-31T15:14:11.390625+0000","last_undegraded":"2024-03-31T15:14:11.390625+0000","last_fullsized":"2024-03-31T15:14:11.390625+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:34:11.338914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.30","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:10.891716+0000","last_change":"2024-03-31T15:14:09.287510+0000","last_active":"2024-03-31T15:14:10.891716+0000","last_peered":"2024-03-31T15:14:10.891716+0000","last_clean":"2024-03-31T15:14:10.891716+0000","last_became_active":"2024-03-31T15:14:09.287238+0000","last_became_peered":"2024-03-31T15:14:09.287238+0000","last_unstale":"2024-03-31T15:14:10.891716+0000","last_undegraded":"2024-03-31T15:14:10.891716+0000","last_fullsized":"2024-03-31T15:14:10.891716+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:14:09.225080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.2f","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-03-31T15:14:08.254889+0000","last_change":"2024-03-31T15:14:08.254889+0000","last_active":"2024-03-31T15:14:08.242094+0000","last_peered":"2024-03-31T15:14:08.242094+0000","last_clean":"2024-03-31T15:14:08.242094+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:14:08.254889+0000","last_undegraded":"2024-03-31T15:14:08.254889+0000","last_fullsized":"2024-03-31T15:14:08.254889+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2e","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:10.279438+0000","last_change":"2024-03-31T15:14:09.276215+0000","last_active":"2024-03-31T15:14:10.279438+0000","last_peered":"2024-03-31T15:14:10.279438+0000","last_clean":"2024-03-31T15:14:10.279438+0000","last_became_active":"2024-03-31T15:14:09.275956+0000","last_became_peered":"2024-03-31T15:14:09.275956+0000","last_unstale":"2024-03-31T15:14:10.279438+0000","last_undegraded":"2024-03-31T15:14:10.279438+0000","last_fullsized":"2024-03-31T15:14:10.279438+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:54:50.675440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.2d","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-03-31T15:14:08.258151+0000","last_change":"2024-03-31T15:14:08.258151+0000","last_active":"2024-03-31T15:14:08.242094+0000","last_peered":"2024-03-31T15:14:08.242094+0000","last_clean":"2024-03-31T15:14:08.242094+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:14:08.258151+0000","last_undegraded":"2024-03-31T15:14:08.258151+0000","last_fullsized":"2024-03-31T15:14:08.258151+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.2c","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-03-31T15:14:08.253580+0000","last_change":"2024-03-31T15:14:08.253580+0000","last_active":"2024-03-31T15:14:08.242094+0000","last_peered":"2024-03-31T15:14:08.242094+0000","last_clean":"2024-03-31T15:14:08.242094+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:14:08.253580+0000","last_undegraded":"2024-03-31T15:14:08.253580+0000","last_fullsized":"2024-03-31T15:14:08.253580+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2b","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-03-31T15:14:08.253223+0000","last_change":"2024-03-31T15:14:08.253223+0000","last_active":"2024-03-31T15:14:08.242094+0000","last_peered":"2024-03-31T15:14:08.242094+0000","last_clean":"2024-03-31T15:14:08.242094+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:14:08.253223+0000","last_undegraded":"2024-03-31T15:14:08.253223+0000","last_fullsized":"2024-03-31T15:14:08.253223+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2a","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:11.390332+0000","last_change":"2024-03-31T15:14:09.275037+0000","last_active":"2024-03-31T15:14:11.390332+0000","last_peered":"2024-03-31T15:14:11.390332+0000","last_clean":"2024-03-31T15:14:11.390332+0000","last_became_active":"2024-03-31T15:14:09.274177+0000","last_became_peered":"2024-03-31T15:14:09.274177+0000","last_unstale":"2024-03-31T15:14:11.390332+0000","last_undegraded":"2024-03-31T15:14:11.390332+0000","last_fullsized":"2024-03-31T15:14:11.390332+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:09:48.241483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.29","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-03-31T15:14:08.253496+0000","last_change":"2024-03-31T15:14:08.253496+0000","last_active":"2024-03-31T15:14:08.242094+0000","last_peered":"2024-03-31T15:14:08.242094+0000","last_clean":"2024-03-31T15:14:08.242094+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:14:08.253496+0000","last_undegraded":"2024-03-31T15:14:08.253496+0000","last_fullsized":"2024-03-31T15:14:08.253496+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.28","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:11.390297+0000","last_change":"2024-03-31T15:14:09.283380+0000","last_active":"2024-03-31T15:14:11.390297+0000","last_peered":"2024-03-31T15:14:11.390297+0000","last_clean":"2024-03-31T15:14:11.390297+0000","last_became_active":"2024-03-31T15:14:09.283097+0000","last_became_peered":"2024-03-31T15:14:09.283097+0000","last_unstale":"2024-03-31T15:14:11.390297+0000","last_undegraded":"2024-03-31T15:14:11.390297+0000","last_fullsized":"2024-03-31T15:14:11.390297+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:00:11.817601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.27","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-03-31T15:14:08.254451+0000","last_change":"2024-03-31T15:14:08.254451+0000","last_active":"2024-03-31T15:14:08.242094+0000","last_peered":"2024-03-31T15:14:08.242094+0000","last_clean":"2024-03-31T15:14:08.242094+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:14:08.254451+0000","last_undegraded":"2024-03-31T15:14:08.254451+0000","last_fullsized":"2024-03-31T15:14:08.254451+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.26","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:10.892253+0000","last_change":"2024-03-31T15:14:09.276124+0000","last_active":"2024-03-31T15:14:10.892253+0000","last_peered":"2024-03-31T15:14:10.892253+0000","last_clean":"2024-03-31T15:14:10.892253+0000","last_became_active":"2024-03-31T15:14:09.275495+0000","last_became_peered":"2024-03-31T15:14:09.275495+0000","last_unstale":"2024-03-31T15:14:10.892253+0000","last_undegraded":"2024-03-31T15:14:10.892253+0000","last_fullsized":"2024-03-31T15:14:10.892253+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:38:53.950142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.25","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:11.391422+0000","last_change":"2024-03-31T15:14:09.290617+0000","last_active":"2024-03-31T15:14:11.391422+0000","last_peered":"2024-03-31T15:14:11.391422+0000","last_clean":"2024-03-31T15:14:11.391422+0000","last_became_active":"2024-03-31T15:14:09.290308+0000","last_became_peered":"2024-03-31T15:14:09.290308+0000","last_unstale":"2024-03-31T15:14:11.391422+0000","last_undegraded":"2024-03-31T15:14:11.391422+0000","last_fullsized":"2024-03-31T15:14:11.391422+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:43:42.509329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.24","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:11.391301+0000","last_change":"2024-03-31T15:14:09.290865+0000","last_active":"2024-03-31T15:14:11.391301+0000","last_peered":"2024-03-31T15:14:11.391301+0000","last_clean":"2024-03-31T15:14:11.391301+0000","last_became_active":"2024-03-31T15:14:09.290601+0000","last_became_peered":"2024-03-31T15:14:09.290601+0000","last_unstale":"2024-03-31T15:14:11.391301+0000","last_undegraded":"2024-03-31T15:14:11.391301+0000","last_fullsized":"2024-03-31T15:14:11.391301+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T19:21:02.579812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.23","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-03-31T15:14:08.258913+0000","last_change":"2024-03-31T15:14:08.258913+0000","last_active":"2024-03-31T15:14:08.242094+0000","last_peered":"2024-03-31T15:14:08.242094+0000","last_clean":"2024-03-31T15:14:08.242094+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:14:08.258913+0000","last_undegraded":"2024-03-31T15:14:08.258913+0000","last_fullsized":"2024-03-31T15:14:08.258913+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.22","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-03-31T15:14:08.260326+0000","last_change":"2024-03-31T15:14:08.260326+0000","last_active":"2024-03-31T15:14:08.242094+0000","last_peered":"2024-03-31T15:14:08.242094+0000","last_clean":"2024-03-31T15:14:08.242094+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:14:08.260326+0000","last_undegraded":"2024-03-31T15:14:08.260326+0000","last_fullsized":"2024-03-31T15:14:08.260326+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.21","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:10.892072+0000","last_change":"2024-03-31T15:14:09.276137+0000","last_active":"2024-03-31T15:14:10.892072+0000","last_peered":"2024-03-31T15:14:10.892072+0000","last_clean":"2024-03-31T15:14:10.892072+0000","last_became_active":"2024-03-31T15:14:09.275873+0000","last_became_peered":"2024-03-31T15:14:09.275873+0000","last_unstale":"2024-03-31T15:14:10.892072+0000","last_undegraded":"2024-03-31T15:14:10.892072+0000","last_fullsized":"2024-03-31T15:14:10.892072+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T17:03:26.938790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.20","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:11.389806+0000","last_change":"2024-03-31T15:14:09.285377+0000","last_active":"2024-03-31T15:14:11.389806+0000","last_peered":"2024-03-31T15:14:11.389806+0000","last_clean":"2024-03-31T15:14:11.389806+0000","last_became_active":"2024-03-31T15:14:09.285077+0000","last_became_peered":"2024-03-31T15:14:09.285077+0000","last_unstale":"2024-03-31T15:14:11.389806+0000","last_undegraded":"2024-03-31T15:14:11.389806+0000","last_fullsized":"2024-03-31T15:14:11.389806+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T15:39:45.840997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.1e","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:11.391059+0000","last_change":"2024-03-31T15:14:09.286260+0000","last_active":"2024-03-31T15:14:11.391059+0000","last_peered":"2024-03-31T15:14:11.391059+0000","last_clean":"2024-03-31T15:14:11.391059+0000","last_became_active":"2024-03-31T15:14:09.285985+0000","last_became_peered":"2024-03-31T15:14:09.285985+0000","last_unstale":"2024-03-31T15:14:11.391059+0000","last_undegraded":"2024-03-31T15:14:11.391059+0000","last_fullsized":"2024-03-31T15:14:11.391059+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T15:58:28.943455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1f","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-31T15:14:08.272805+0000","last_change":"2024-03-31T15:14:08.268805+0000","last_active":"2024-03-31T15:14:08.272805+0000","last_peered":"2024-03-31T15:14:08.272805+0000","last_clean":"2024-03-31T15:14:08.272805+0000","last_became_active":"2024-03-31T15:14:08.268515+0000","last_became_peered":"2024-03-31T15:14:08.268515+0000","last_unstale":"2024-03-31T15:14:08.272805+0000","last_undegraded":"2024-03-31T15:14:08.272805+0000","last_fullsized":"2024-03-31T15:14:08.272805+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_clean_scrub_stamp":"2024-03-31T15:14:07.235046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:47:23.413340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1f","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:10.278621+0000","last_change":"2024-03-31T15:14:09.281053+0000","last_active":"2024-03-31T15:14:10.278621+0000","last_peered":"2024-03-31T15:14:10.278621+0000","last_clean":"2024-03-31T15:14:10.278621+0000","last_became_active":"2024-03-31T15:14:09.280735+0000","last_became_peered":"2024-03-31T15:14:09.280735+0000","last_unstale":"2024-03-31T15:14:10.278621+0000","last_undegraded":"2024-03-31T15:14:10.278621+0000","last_fullsized":"2024-03-31T15:14:10.278621+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:55:28.086183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.1e","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-31T15:14:08.273342+0000","last_change":"2024-03-31T15:14:08.270517+0000","last_active":"2024-03-31T15:14:08.273342+0000","last_peered":"2024-03-31T15:14:08.273342+0000","last_clean":"2024-03-31T15:14:08.273342+0000","last_became_active":"2024-03-31T15:14:08.270219+0000","last_became_peered":"2024-03-31T15:14:08.270219+0000","last_unstale":"2024-03-31T15:14:08.273342+0000","last_undegraded":"2024-03-31T15:14:08.273342+0000","last_fullsized":"2024-03-31T15:14:08.273342+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_clean_scrub_stamp":"2024-03-31T15:14:07.235046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:40:13.939016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1c","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:11.391070+0000","last_change":"2024-03-31T15:14:09.285776+0000","last_active":"2024-03-31T15:14:11.391070+0000","last_peered":"2024-03-31T15:14:11.391070+0000","last_clean":"2024-03-31T15:14:11.391070+0000","last_became_active":"2024-03-31T15:14:09.285383+0000","last_became_peered":"2024-03-31T15:14:09.285383+0000","last_unstale":"2024-03-31T15:14:11.391070+0000","last_undegraded":"2024-03-31T15:14:11.391070+0000","last_fullsized":"2024-03-31T15:14:11.391070+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:45:29.558658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1d","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-31T15:14:08.274496+0000","last_change":"2024-03-31T15:14:08.268956+0000","last_active":"2024-03-31T15:14:08.274496+0000","last_peered":"2024-03-31T15:14:08.274496+0000","last_clean":"2024-03-31T15:14:08.274496+0000","last_became_active":"2024-03-31T15:14:08.268672+0000","last_became_peered":"2024-03-31T15:14:08.268672+0000","last_unstale":"2024-03-31T15:14:08.274496+0000","last_undegraded":"2024-03-31T15:14:08.274496+0000","last_fullsized":"2024-03-31T15:14:08.274496+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_clean_scrub_stamp":"2024-03-31T15:14:07.235046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:55:47.095073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1d","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-03-31T15:14:08.257689+0000","last_change":"2024-03-31T15:14:08.257689+0000","last_active":"2024-03-31T15:14:08.242094+0000","last_peered":"2024-03-31T15:14:08.242094+0000","last_clean":"2024-03-31T15:14:08.242094+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:14:08.257689+0000","last_undegraded":"2024-03-31T15:14:08.257689+0000","last_fullsized":"2024-03-31T15:14:08.257689+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1c","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-31T15:14:08.274904+0000","last_change":"2024-03-31T15:14:08.269987+0000","last_active":"2024-03-31T15:14:08.274904+0000","last_peered":"2024-03-31T15:14:08.274904+0000","last_clean":"2024-03-31T15:14:08.274904+0000","last_became_active":"2024-03-31T15:14:08.269241+0000","last_became_peered":"2024-03-31T15:14:08.269241+0000","last_unstale":"2024-03-31T15:14:08.274904+0000","last_undegraded":"2024-03-31T15:14:08.274904+0000","last_fullsized":"2024-03-31T15:14:08.274904+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_clean_scrub_stamp":"2024-03-31T15:14:07.235046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:54:34.951227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1a","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:10.891958+0000","last_change":"2024-03-31T15:14:09.282143+0000","last_active":"2024-03-31T15:14:10.891958+0000","last_peered":"2024-03-31T15:14:10.891958+0000","last_clean":"2024-03-31T15:14:10.891958+0000","last_became_active":"2024-03-31T15:14:09.281851+0000","last_became_peered":"2024-03-31T15:14:09.281851+0000","last_unstale":"2024-03-31T15:14:10.891958+0000","last_undegraded":"2024-03-31T15:14:10.891958+0000","last_fullsized":"2024-03-31T15:14:10.891958+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:07:40.319619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.1b","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-31T15:14:08.275156+0000","last_change":"2024-03-31T15:14:08.271392+0000","last_active":"2024-03-31T15:14:08.275156+0000","last_peered":"2024-03-31T15:14:08.275156+0000","last_clean":"2024-03-31T15:14:08.275156+0000","last_became_active":"2024-03-31T15:14:08.271138+0000","last_became_peered":"2024-03-31T15:14:08.271138+0000","last_unstale":"2024-03-31T15:14:08.275156+0000","last_undegraded":"2024-03-31T15:14:08.275156+0000","last_fullsized":"2024-03-31T15:14:08.275156+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_clean_scrub_stamp":"2024-03-31T15:14:07.235046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:45:22.838594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1b","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-03-31T15:14:08.258473+0000","last_change":"2024-03-31T15:14:08.258473+0000","last_active":"2024-03-31T15:14:08.242094+0000","last_peered":"2024-03-31T15:14:08.242094+0000","last_clean":"2024-03-31T15:14:08.242094+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:14:08.258473+0000","last_undegraded":"2024-03-31T15:14:08.258473+0000","last_fullsized":"2024-03-31T15:14:08.258473+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.1a","version":"108'1","reported_seq":15,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:11.390384+0000","last_change":"2024-03-31T15:14:08.273955+0000","last_active":"2024-03-31T15:14:11.390384+0000","last_peered":"2024-03-31T15:14:11.390384+0000","last_clean":"2024-03-31T15:14:11.390384+0000","last_became_active":"2024-03-31T15:14:08.273646+0000","last_became_peered":"2024-03-31T15:14:08.273646+0000","last_unstale":"2024-03-31T15:14:11.390384+0000","last_undegraded":"2024-03-31T15:14:11.390384+0000","last_fullsized":"2024-03-31T15:14:11.390384+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_clean_scrub_stamp":"2024-03-31T15:14:07.235046+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:27:01.349324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.18","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:10.891733+0000","last_change":"2024-03-31T15:14:09.278486+0000","last_active":"2024-03-31T15:14:10.891733+0000","last_peered":"2024-03-31T15:14:10.891733+0000","last_clean":"2024-03-31T15:14:10.891733+0000","last_became_active":"2024-03-31T15:14:09.278193+0000","last_became_peered":"2024-03-31T15:14:09.278193+0000","last_unstale":"2024-03-31T15:14:10.891733+0000","last_undegraded":"2024-03-31T15:14:10.891733+0000","last_fullsized":"2024-03-31T15:14:10.891733+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:42:40.870161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-31T15:14:08.270406+0000","last_change":"2024-03-31T15:14:08.266416+0000","last_active":"2024-03-31T15:14:08.270406+0000","last_peered":"2024-03-31T15:14:08.270406+0000","last_clean":"2024-03-31T15:14:08.270406+0000","last_became_active":"2024-03-31T15:14:08.266116+0000","last_became_peered":"2024-03-31T15:14:08.266116+0000","last_unstale":"2024-03-31T15:14:08.270406+0000","last_undegraded":"2024-03-31T15:14:08.270406+0000","last_fullsized":"2024-03-31T15:14:08.270406+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_clean_scrub_stamp":"2024-03-31T15:14:07.235046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:05:08.886029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.19","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-03-31T15:14:08.260207+0000","last_change":"2024-03-31T15:14:08.260207+0000","last_active":"2024-03-31T15:14:08.242094+0000","last_peered":"2024-03-31T15:14:08.242094+0000","last_clean":"2024-03-31T15:14:08.242094+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:14:08.260207+0000","last_undegraded":"2024-03-31T15:14:08.260207+0000","last_fullsized":"2024-03-31T15:14:08.260207+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":14,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:10.891808+0000","last_change":"2024-03-31T15:14:08.276025+0000","last_active":"2024-03-31T15:14:10.891808+0000","last_peered":"2024-03-31T15:14:10.891808+0000","last_clean":"2024-03-31T15:14:10.891808+0000","last_became_active":"2024-03-31T15:14:08.275541+0000","last_became_peered":"2024-03-31T15:14:08.275541+0000","last_unstale":"2024-03-31T15:14:10.891808+0000","last_undegraded":"2024-03-31T15:14:10.891808+0000","last_fullsized":"2024-03-31T15:14:10.891808+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_clean_scrub_stamp":"2024-03-31T15:14:07.235046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:34:33.623018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.16","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-03-31T15:14:08.259804+0000","last_change":"2024-03-31T15:14:08.259804+0000","last_active":"2024-03-31T15:14:08.242094+0000","last_peered":"2024-03-31T15:14:08.242094+0000","last_clean":"2024-03-31T15:14:08.242094+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:14:08.259804+0000","last_undegraded":"2024-03-31T15:14:08.259804+0000","last_fullsized":"2024-03-31T15:14:08.259804+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.17","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-31T15:14:08.278508+0000","last_change":"2024-03-31T15:14:08.275403+0000","last_active":"2024-03-31T15:14:08.278508+0000","last_peered":"2024-03-31T15:14:08.278508+0000","last_clean":"2024-03-31T15:14:08.278508+0000","last_became_active":"2024-03-31T15:14:08.275122+0000","last_became_peered":"2024-03-31T15:14:08.275122+0000","last_unstale":"2024-03-31T15:14:08.278508+0000","last_undegraded":"2024-03-31T15:14:08.278508+0000","last_fullsized":"2024-03-31T15:14:08.278508+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_clean_scrub_stamp":"2024-03-31T15:14:07.235046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T15:33:09.566750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.17","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:10.278750+0000","last_change":"2024-03-31T15:14:09.298453+0000","last_active":"2024-03-31T15:14:10.278750+0000","last_peered":"2024-03-31T15:14:10.278750+0000","last_clean":"2024-03-31T15:14:10.278750+0000","last_became_active":"2024-03-31T15:14:09.298160+0000","last_became_peered":"2024-03-31T15:14:09.298160+0000","last_unstale":"2024-03-31T15:14:10.278750+0000","last_undegraded":"2024-03-31T15:14:10.278750+0000","last_fullsized":"2024-03-31T15:14:10.278750+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:48:01.826468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":14,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:11.391501+0000","last_change":"2024-03-31T15:14:08.272714+0000","last_active":"2024-03-31T15:14:11.391501+0000","last_peered":"2024-03-31T15:14:11.391501+0000","last_clean":"2024-03-31T15:14:11.391501+0000","last_became_active":"2024-03-31T15:14:08.272416+0000","last_became_peered":"2024-03-31T15:14:08.272416+0000","last_unstale":"2024-03-31T15:14:11.391501+0000","last_undegraded":"2024-03-31T15:14:11.391501+0000","last_fullsized":"2024-03-31T15:14:11.391501+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_clean_scrub_stamp":"2024-03-31T15:14:07.235046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:34:33.623018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.14","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:11.391432+0000","last_change":"2024-03-31T15:14:09.281244+0000","last_active":"2024-03-31T15:14:11.391432+0000","last_peered":"2024-03-31T15:14:11.391432+0000","last_clean":"2024-03-31T15:14:11.391432+0000","last_became_active":"2024-03-31T15:14:09.280267+0000","last_became_peered":"2024-03-31T15:14:09.280267+0000","last_unstale":"2024-03-31T15:14:11.391432+0000","last_undegraded":"2024-03-31T15:14:11.391432+0000","last_fullsized":"2024-03-31T15:14:11.391432+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:23:54.993797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.15","version":"108'3","reported_seq":17,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:10.891863+0000","last_change":"2024-03-31T15:14:08.274322+0000","last_active":"2024-03-31T15:14:10.891863+0000","last_peered":"2024-03-31T15:14:10.891863+0000","last_clean":"2024-03-31T15:14:10.891863+0000","last_became_active":"2024-03-31T15:14:08.274025+0000","last_became_peered":"2024-03-31T15:14:08.274025+0000","last_unstale":"2024-03-31T15:14:10.891863+0000","last_undegraded":"2024-03-31T15:14:10.891863+0000","last_fullsized":"2024-03-31T15:14:10.891863+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_clean_scrub_stamp":"2024-03-31T15:14:07.235046+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:54:49.668391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.15","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-03-31T15:14:08.258737+0000","last_change":"2024-03-31T15:14:08.258737+0000","last_active":"2024-03-31T15:14:08.242094+0000","last_peered":"2024-03-31T15:14:08.242094+0000","last_clean":"2024-03-31T15:14:08.242094+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:14:08.258737+0000","last_undegraded":"2024-03-31T15:14:08.258737+0000","last_fullsized":"2024-03-31T15:14:08.258737+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.14","version":"108'6","reported_seq":20,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:10.892187+0000","last_change":"2024-03-31T15:14:08.277536+0000","last_active":"2024-03-31T15:14:10.892187+0000","last_peered":"2024-03-31T15:14:10.892187+0000","last_clean":"2024-03-31T15:14:10.892187+0000","last_became_active":"2024-03-31T15:14:08.277279+0000","last_became_peered":"2024-03-31T15:14:08.277279+0000","last_unstale":"2024-03-31T15:14:10.892187+0000","last_undegraded":"2024-03-31T15:14:10.892187+0000","last_fullsized":"2024-03-31T15:14:10.892187+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_clean_scrub_stamp":"2024-03-31T15:14:07.235046+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:08:03.916239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.12","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:11.390394+0000","last_change":"2024-03-31T15:14:09.275078+0000","last_active":"2024-03-31T15:14:11.390394+0000","last_peered":"2024-03-31T15:14:11.390394+0000","last_clean":"2024-03-31T15:14:11.390394+0000","last_became_active":"2024-03-31T15:14:09.274698+0000","last_became_peered":"2024-03-31T15:14:09.274698+0000","last_unstale":"2024-03-31T15:14:11.390394+0000","last_undegraded":"2024-03-31T15:14:11.390394+0000","last_fullsized":"2024-03-31T15:14:11.390394+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:30:56.093324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.13","version":"108'2","reported_seq":16,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:11.390958+0000","last_change":"2024-03-31T15:14:08.266097+0000","last_active":"2024-03-31T15:14:11.390958+0000","last_peered":"2024-03-31T15:14:11.390958+0000","last_clean":"2024-03-31T15:14:11.390958+0000","last_became_active":"2024-03-31T15:14:08.265812+0000","last_became_peered":"2024-03-31T15:14:08.265812+0000","last_unstale":"2024-03-31T15:14:11.390958+0000","last_undegraded":"2024-03-31T15:14:11.390958+0000","last_fullsized":"2024-03-31T15:14:11.390958+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_clean_scrub_stamp":"2024-03-31T15:14:07.235046+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:49:52.886409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.13","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:11.391329+0000","last_change":"2024-03-31T15:14:09.282087+0000","last_active":"2024-03-31T15:14:11.391329+0000","last_peered":"2024-03-31T15:14:11.391329+0000","last_clean":"2024-03-31T15:14:11.391329+0000","last_became_active":"2024-03-31T15:14:09.281693+0000","last_became_peered":"2024-03-31T15:14:09.281693+0000","last_unstale":"2024-03-31T15:14:11.391329+0000","last_undegraded":"2024-03-31T15:14:11.391329+0000","last_fullsized":"2024-03-31T15:14:11.391329+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:32:06.156300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":14,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:10.278350+0000","last_change":"2024-03-31T15:14:08.272919+0000","last_active":"2024-03-31T15:14:10.278350+0000","last_peered":"2024-03-31T15:14:10.278350+0000","last_clean":"2024-03-31T15:14:10.278350+0000","last_became_active":"2024-03-31T15:14:08.272309+0000","last_became_peered":"2024-03-31T15:14:08.272309+0000","last_unstale":"2024-03-31T15:14:10.278350+0000","last_undegraded":"2024-03-31T15:14:10.278350+0000","last_fullsized":"2024-03-31T15:14:10.278350+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_clean_scrub_stamp":"2024-03-31T15:14:07.235046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:56:10.303847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.11","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:11.390150+0000","last_change":"2024-03-31T15:14:09.274826+0000","last_active":"2024-03-31T15:14:11.390150+0000","last_peered":"2024-03-31T15:14:11.390150+0000","last_clean":"2024-03-31T15:14:11.390150+0000","last_became_active":"2024-03-31T15:14:09.274550+0000","last_became_peered":"2024-03-31T15:14:09.274550+0000","last_unstale":"2024-03-31T15:14:11.390150+0000","last_undegraded":"2024-03-31T15:14:11.390150+0000","last_fullsized":"2024-03-31T15:14:11.390150+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T17:25:28.357433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.10","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-31T15:14:08.278587+0000","last_change":"2024-03-31T15:14:08.276042+0000","last_active":"2024-03-31T15:14:08.278587+0000","last_peered":"2024-03-31T15:14:08.278587+0000","last_clean":"2024-03-31T15:14:08.278587+0000","last_became_active":"2024-03-31T15:14:08.275766+0000","last_became_peered":"2024-03-31T15:14:08.275766+0000","last_unstale":"2024-03-31T15:14:08.278587+0000","last_undegraded":"2024-03-31T15:14:08.278587+0000","last_fullsized":"2024-03-31T15:14:08.278587+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_clean_scrub_stamp":"2024-03-31T15:14:07.235046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:38:29.678222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.e","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-03-31T15:14:08.258499+0000","last_change":"2024-03-31T15:14:08.258499+0000","last_active":"2024-03-31T15:14:08.242094+0000","last_peered":"2024-03-31T15:14:08.242094+0000","last_clean":"2024-03-31T15:14:08.242094+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:14:08.258499+0000","last_undegraded":"2024-03-31T15:14:08.258499+0000","last_fullsized":"2024-03-31T15:14:08.258499+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.f","version":"108'1","reported_seq":15,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:10.892528+0000","last_change":"2024-03-31T15:14:08.273521+0000","last_active":"2024-03-31T15:14:10.892528+0000","last_peered":"2024-03-31T15:14:10.892528+0000","last_clean":"2024-03-31T15:14:10.892528+0000","last_became_active":"2024-03-31T15:14:08.271683+0000","last_became_peered":"2024-03-31T15:14:08.271683+0000","last_unstale":"2024-03-31T15:14:10.892528+0000","last_undegraded":"2024-03-31T15:14:10.892528+0000","last_fullsized":"2024-03-31T15:14:10.892528+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_clean_scrub_stamp":"2024-03-31T15:14:07.235046+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:08:12.284361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.f","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:11.391301+0000","last_change":"2024-03-31T15:14:09.292401+0000","last_active":"2024-03-31T15:14:11.391301+0000","last_peered":"2024-03-31T15:14:11.391301+0000","last_clean":"2024-03-31T15:14:11.391301+0000","last_became_active":"2024-03-31T15:14:09.292126+0000","last_became_peered":"2024-03-31T15:14:09.292126+0000","last_unstale":"2024-03-31T15:14:11.391301+0000","last_undegraded":"2024-03-31T15:14:11.391301+0000","last_fullsized":"2024-03-31T15:14:11.391301+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:24:54.943510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.e","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-31T15:14:08.276874+0000","last_change":"2024-03-31T15:14:08.272765+0000","last_active":"2024-03-31T15:14:08.276874+0000","last_peered":"2024-03-31T15:14:08.276874+0000","last_clean":"2024-03-31T15:14:08.276874+0000","last_became_active":"2024-03-31T15:14:08.272496+0000","last_became_peered":"2024-03-31T15:14:08.272496+0000","last_unstale":"2024-03-31T15:14:08.276874+0000","last_undegraded":"2024-03-31T15:14:08.276874+0000","last_fullsized":"2024-03-31T15:14:08.276874+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_clean_scrub_stamp":"2024-03-31T15:14:07.235046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:30:40.304653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.c","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-03-31T15:14:08.257389+0000","last_change":"2024-03-31T15:14:08.257389+0000","last_active":"2024-03-31T15:14:08.242094+0000","last_peered":"2024-03-31T15:14:08.242094+0000","last_clean":"2024-03-31T15:14:08.242094+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:14:08.257389+0000","last_undegraded":"2024-03-31T15:14:08.257389+0000","last_fullsized":"2024-03-31T15:14:08.257389+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.d","version":"108'1","reported_seq":15,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:10.278500+0000","last_change":"2024-03-31T15:14:08.278494+0000","last_active":"2024-03-31T15:14:10.278500+0000","last_peered":"2024-03-31T15:14:10.278500+0000","last_clean":"2024-03-31T15:14:10.278500+0000","last_became_active":"2024-03-31T15:14:08.278267+0000","last_became_peered":"2024-03-31T15:14:08.278267+0000","last_unstale":"2024-03-31T15:14:10.278500+0000","last_undegraded":"2024-03-31T15:14:10.278500+0000","last_fullsized":"2024-03-31T15:14:10.278500+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_clean_scrub_stamp":"2024-03-31T15:14:07.235046+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:49:25.058336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.d","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-03-31T15:14:08.257107+0000","last_change":"2024-03-31T15:14:08.257107+0000","last_active":"2024-03-31T15:14:08.242094+0000","last_peered":"2024-03-31T15:14:08.242094+0000","last_clean":"2024-03-31T15:14:08.242094+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:14:08.257107+0000","last_undegraded":"2024-03-31T15:14:08.257107+0000","last_fullsized":"2024-03-31T15:14:08.257107+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-31T15:14:08.275180+0000","last_change":"2024-03-31T15:14:08.271960+0000","last_active":"2024-03-31T15:14:08.275180+0000","last_peered":"2024-03-31T15:14:08.275180+0000","last_clean":"2024-03-31T15:14:08.275180+0000","last_became_active":"2024-03-31T15:14:08.271691+0000","last_became_peered":"2024-03-31T15:14:08.271691+0000","last_unstale":"2024-03-31T15:14:08.275180+0000","last_undegraded":"2024-03-31T15:14:08.275180+0000","last_fullsized":"2024-03-31T15:14:08.275180+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_clean_scrub_stamp":"2024-03-31T15:14:07.235046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T17:04:17.133375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.a","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-03-31T15:14:08.257121+0000","last_change":"2024-03-31T15:14:08.257121+0000","last_active":"2024-03-31T15:14:08.242094+0000","last_peered":"2024-03-31T15:14:08.242094+0000","last_clean":"2024-03-31T15:14:08.242094+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:14:08.257121+0000","last_undegraded":"2024-03-31T15:14:08.257121+0000","last_fullsized":"2024-03-31T15:14:08.257121+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.b","version":"108'2","reported_seq":16,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:11.391062+0000","last_change":"2024-03-31T15:14:08.274852+0000","last_active":"2024-03-31T15:14:11.391062+0000","last_peered":"2024-03-31T15:14:11.391062+0000","last_clean":"2024-03-31T15:14:11.391062+0000","last_became_active":"2024-03-31T15:14:08.274568+0000","last_became_peered":"2024-03-31T15:14:08.274568+0000","last_unstale":"2024-03-31T15:14:11.391062+0000","last_undegraded":"2024-03-31T15:14:11.391062+0000","last_fullsized":"2024-03-31T15:14:11.391062+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_clean_scrub_stamp":"2024-03-31T15:14:07.235046+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T19:34:36.202544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.b","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:10.891810+0000","last_change":"2024-03-31T15:14:09.280942+0000","last_active":"2024-03-31T15:14:10.891810+0000","last_peered":"2024-03-31T15:14:10.891810+0000","last_clean":"2024-03-31T15:14:10.891810+0000","last_became_active":"2024-03-31T15:14:09.280678+0000","last_became_peered":"2024-03-31T15:14:09.280678+0000","last_unstale":"2024-03-31T15:14:10.891810+0000","last_undegraded":"2024-03-31T15:14:10.891810+0000","last_fullsized":"2024-03-31T15:14:10.891810+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:00:22.586826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-31T15:14:08.278516+0000","last_change":"2024-03-31T15:14:08.275261+0000","last_active":"2024-03-31T15:14:08.278516+0000","last_peered":"2024-03-31T15:14:08.278516+0000","last_clean":"2024-03-31T15:14:08.278516+0000","last_became_active":"2024-03-31T15:14:08.274961+0000","last_became_peered":"2024-03-31T15:14:08.274961+0000","last_unstale":"2024-03-31T15:14:08.278516+0000","last_undegraded":"2024-03-31T15:14:08.278516+0000","last_fullsized":"2024-03-31T15:14:08.278516+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_clean_scrub_stamp":"2024-03-31T15:14:07.235046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T17:28:54.278793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.8","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-03-31T15:14:08.257584+0000","last_change":"2024-03-31T15:14:08.257584+0000","last_active":"2024-03-31T15:14:08.242094+0000","last_peered":"2024-03-31T15:14:08.242094+0000","last_clean":"2024-03-31T15:14:08.242094+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:14:08.257584+0000","last_undegraded":"2024-03-31T15:14:08.257584+0000","last_fullsized":"2024-03-31T15:14:08.257584+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":14,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:11.390727+0000","last_change":"2024-03-31T15:14:08.275022+0000","last_active":"2024-03-31T15:14:11.390727+0000","last_peered":"2024-03-31T15:14:11.390727+0000","last_clean":"2024-03-31T15:14:11.390727+0000","last_became_active":"2024-03-31T15:14:08.274740+0000","last_became_peered":"2024-03-31T15:14:08.274740+0000","last_unstale":"2024-03-31T15:14:11.390727+0000","last_undegraded":"2024-03-31T15:14:11.390727+0000","last_fullsized":"2024-03-31T15:14:11.390727+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_clean_scrub_stamp":"2024-03-31T15:14:07.235046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:25:35.260804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.9","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:11.390344+0000","last_change":"2024-03-31T15:14:09.291069+0000","last_active":"2024-03-31T15:14:11.390344+0000","last_peered":"2024-03-31T15:14:11.390344+0000","last_clean":"2024-03-31T15:14:11.390344+0000","last_became_active":"2024-03-31T15:14:09.290802+0000","last_became_peered":"2024-03-31T15:14:09.290802+0000","last_unstale":"2024-03-31T15:14:11.390344+0000","last_undegraded":"2024-03-31T15:14:11.390344+0000","last_fullsized":"2024-03-31T15:14:11.390344+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:36:20.417016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":14,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:10.891861+0000","last_change":"2024-03-31T15:14:08.270333+0000","last_active":"2024-03-31T15:14:10.891861+0000","last_peered":"2024-03-31T15:14:10.891861+0000","last_clean":"2024-03-31T15:14:10.891861+0000","last_became_active":"2024-03-31T15:14:08.269866+0000","last_became_peered":"2024-03-31T15:14:08.269866+0000","last_unstale":"2024-03-31T15:14:10.891861+0000","last_undegraded":"2024-03-31T15:14:10.891861+0000","last_fullsized":"2024-03-31T15:14:10.891861+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_clean_scrub_stamp":"2024-03-31T15:14:07.235046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:55:27.079135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.6","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:10.278592+0000","last_change":"2024-03-31T15:14:09.299295+0000","last_active":"2024-03-31T15:14:10.278592+0000","last_peered":"2024-03-31T15:14:10.278592+0000","last_clean":"2024-03-31T15:14:10.278592+0000","last_became_active":"2024-03-31T15:14:09.299016+0000","last_became_peered":"2024-03-31T15:14:09.299016+0000","last_unstale":"2024-03-31T15:14:10.278592+0000","last_undegraded":"2024-03-31T15:14:10.278592+0000","last_fullsized":"2024-03-31T15:14:10.278592+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:34:34.630067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.7","version":"108'2","reported_seq":16,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:11.391777+0000","last_change":"2024-03-31T15:14:08.276158+0000","last_active":"2024-03-31T15:14:11.391777+0000","last_peered":"2024-03-31T15:14:11.391777+0000","last_clean":"2024-03-31T15:14:11.391777+0000","last_became_active":"2024-03-31T15:14:08.275856+0000","last_became_peered":"2024-03-31T15:14:08.275856+0000","last_unstale":"2024-03-31T15:14:11.391777+0000","last_undegraded":"2024-03-31T15:14:11.391777+0000","last_fullsized":"2024-03-31T15:14:11.391777+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_clean_scrub_stamp":"2024-03-31T15:14:07.235046+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:48:00.819420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.7","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:10.892615+0000","last_change":"2024-03-31T15:14:09.277522+0000","last_active":"2024-03-31T15:14:10.892615+0000","last_peered":"2024-03-31T15:14:10.892615+0000","last_clean":"2024-03-31T15:14:10.892615+0000","last_became_active":"2024-03-31T15:14:09.276889+0000","last_became_peered":"2024-03-31T15:14:09.276889+0000","last_unstale":"2024-03-31T15:14:10.892615+0000","last_undegraded":"2024-03-31T15:14:10.892615+0000","last_fullsized":"2024-03-31T15:14:10.892615+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:45:51.393113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.6","version":"109'1","reported_seq":15,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:10.278704+0000","last_change":"2024-03-31T15:14:08.265332+0000","last_active":"2024-03-31T15:14:10.278704+0000","last_peered":"2024-03-31T15:14:10.278704+0000","last_clean":"2024-03-31T15:14:10.278704+0000","last_became_active":"2024-03-31T15:14:08.264838+0000","last_became_peered":"2024-03-31T15:14:08.264838+0000","last_unstale":"2024-03-31T15:14:10.278704+0000","last_undegraded":"2024-03-31T15:14:10.278704+0000","last_fullsized":"2024-03-31T15:14:10.278704+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_clean_scrub_stamp":"2024-03-31T15:14:07.235046+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:32:53.111430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.4","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:10.892152+0000","last_change":"2024-03-31T15:14:09.279412+0000","last_active":"2024-03-31T15:14:10.892152+0000","last_peered":"2024-03-31T15:14:10.892152+0000","last_clean":"2024-03-31T15:14:10.892152+0000","last_became_active":"2024-03-31T15:14:09.278696+0000","last_became_peered":"2024-03-31T15:14:09.278696+0000","last_unstale":"2024-03-31T15:14:10.892152+0000","last_undegraded":"2024-03-31T15:14:10.892152+0000","last_fullsized":"2024-03-31T15:14:10.892152+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:57:34.106053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":14,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:10.891892+0000","last_change":"2024-03-31T15:14:08.270001+0000","last_active":"2024-03-31T15:14:10.891892+0000","last_peered":"2024-03-31T15:14:10.891892+0000","last_clean":"2024-03-31T15:14:10.891892+0000","last_became_active":"2024-03-31T15:14:08.268897+0000","last_became_peered":"2024-03-31T15:14:08.268897+0000","last_unstale":"2024-03-31T15:14:10.891892+0000","last_undegraded":"2024-03-31T15:14:10.891892+0000","last_fullsized":"2024-03-31T15:14:10.891892+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_clean_scrub_stamp":"2024-03-31T15:14:07.235046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T19:21:04.678038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.5","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:10.278439+0000","last_change":"2024-03-31T15:14:09.292175+0000","last_active":"2024-03-31T15:14:10.278439+0000","last_peered":"2024-03-31T15:14:10.278439+0000","last_clean":"2024-03-31T15:14:10.278439+0000","last_became_active":"2024-03-31T15:14:09.291875+0000","last_became_peered":"2024-03-31T15:14:09.291875+0000","last_unstale":"2024-03-31T15:14:10.278439+0000","last_undegraded":"2024-03-31T15:14:10.278439+0000","last_fullsized":"2024-03-31T15:14:10.278439+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:27:02.356373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":14,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:11.391479+0000","last_change":"2024-03-31T15:14:08.268133+0000","last_active":"2024-03-31T15:14:11.391479+0000","last_peered":"2024-03-31T15:14:11.391479+0000","last_clean":"2024-03-31T15:14:11.391479+0000","last_became_active":"2024-03-31T15:14:08.267768+0000","last_became_peered":"2024-03-31T15:14:08.267768+0000","last_unstale":"2024-03-31T15:14:11.391479+0000","last_undegraded":"2024-03-31T15:14:11.391479+0000","last_fullsized":"2024-03-31T15:14:11.391479+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_clean_scrub_stamp":"2024-03-31T15:14:07.235046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:55:27.079135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.2","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-03-31T15:14:08.254000+0000","last_change":"2024-03-31T15:14:08.254000+0000","last_active":"2024-03-31T15:14:08.242094+0000","last_peered":"2024-03-31T15:14:08.242094+0000","last_clean":"2024-03-31T15:14:08.242094+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:14:08.254000+0000","last_undegraded":"2024-03-31T15:14:08.254000+0000","last_fullsized":"2024-03-31T15:14:08.254000+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-31T15:14:08.272766+0000","last_change":"2024-03-31T15:14:08.268809+0000","last_active":"2024-03-31T15:14:08.272766+0000","last_peered":"2024-03-31T15:14:08.272766+0000","last_clean":"2024-03-31T15:14:08.272766+0000","last_became_active":"2024-03-31T15:14:08.268473+0000","last_became_peered":"2024-03-31T15:14:08.268473+0000","last_unstale":"2024-03-31T15:14:08.272766+0000","last_undegraded":"2024-03-31T15:14:08.272766+0000","last_fullsized":"2024-03-31T15:14:08.272766+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_clean_scrub_stamp":"2024-03-31T15:14:07.235046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:09:33.273967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.3","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-03-31T15:14:08.255261+0000","last_change":"2024-03-31T15:14:08.255261+0000","last_active":"2024-03-31T15:14:08.242094+0000","last_peered":"2024-03-31T15:14:08.242094+0000","last_clean":"2024-03-31T15:14:08.242094+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:14:08.255261+0000","last_undegraded":"2024-03-31T15:14:08.255261+0000","last_fullsized":"2024-03-31T15:14:08.255261+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-31T15:14:08.272985+0000","last_change":"2024-03-31T15:14:08.269651+0000","last_active":"2024-03-31T15:14:08.272985+0000","last_peered":"2024-03-31T15:14:08.272985+0000","last_clean":"2024-03-31T15:14:08.272985+0000","last_became_active":"2024-03-31T15:14:08.269372+0000","last_became_peered":"2024-03-31T15:14:08.269372+0000","last_unstale":"2024-03-31T15:14:08.272985+0000","last_undegraded":"2024-03-31T15:14:08.272985+0000","last_fullsized":"2024-03-31T15:14:08.272985+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_clean_scrub_stamp":"2024-03-31T15:14:07.235046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:15:08.190993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.0","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-03-31T15:14:08.257010+0000","last_change":"2024-03-31T15:14:08.257010+0000","last_active":"2024-03-31T15:14:08.242094+0000","last_peered":"2024-03-31T15:14:08.242094+0000","last_clean":"2024-03-31T15:14:08.242094+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:14:08.257010+0000","last_undegraded":"2024-03-31T15:14:08.257010+0000","last_fullsized":"2024-03-31T15:14:08.257010+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-31T15:14:08.277067+0000","last_change":"2024-03-31T15:14:08.271032+0000","last_active":"2024-03-31T15:14:08.277067+0000","last_peered":"2024-03-31T15:14:08.277067+0000","last_clean":"2024-03-31T15:14:08.277067+0000","last_became_active":"2024-03-31T15:14:08.270781+0000","last_became_peered":"2024-03-31T15:14:08.270781+0000","last_unstale":"2024-03-31T15:14:08.277067+0000","last_undegraded":"2024-03-31T15:14:08.277067+0000","last_fullsized":"2024-03-31T15:14:08.277067+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_clean_scrub_stamp":"2024-03-31T15:14:07.235046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:59:46.849346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:10.892154+0000","last_change":"2024-03-31T15:14:09.288490+0000","last_active":"2024-03-31T15:14:10.892154+0000","last_peered":"2024-03-31T15:14:10.892154+0000","last_clean":"2024-03-31T15:14:10.892154+0000","last_became_active":"2024-03-31T15:14:09.284507+0000","last_became_peered":"2024-03-31T15:14:09.284507+0000","last_unstale":"2024-03-31T15:14:10.892154+0000","last_undegraded":"2024-03-31T15:14:10.892154+0000","last_fullsized":"2024-03-31T15:14:10.892154+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:09:03.491873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":14,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:10.891910+0000","last_change":"2024-03-31T15:14:08.270270+0000","last_active":"2024-03-31T15:14:10.891910+0000","last_peered":"2024-03-31T15:14:10.891910+0000","last_clean":"2024-03-31T15:14:10.891910+0000","last_became_active":"2024-03-31T15:14:08.269482+0000","last_became_peered":"2024-03-31T15:14:08.269482+0000","last_unstale":"2024-03-31T15:14:10.891910+0000","last_undegraded":"2024-03-31T15:14:10.891910+0000","last_fullsized":"2024-03-31T15:14:10.891910+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_clean_scrub_stamp":"2024-03-31T15:14:07.235046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:48:00.819420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.10","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:11.390177+0000","last_change":"2024-03-31T15:14:09.277917+0000","last_active":"2024-03-31T15:14:11.390177+0000","last_peered":"2024-03-31T15:14:11.390177+0000","last_clean":"2024-03-31T15:14:11.390177+0000","last_became_active":"2024-03-31T15:14:09.277651+0000","last_became_peered":"2024-03-31T15:14:09.277651+0000","last_unstale":"2024-03-31T15:14:11.390177+0000","last_undegraded":"2024-03-31T15:14:11.390177+0000","last_fullsized":"2024-03-31T15:14:11.390177+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T16:10:45.541190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-31T15:14:08.275222+0000","last_change":"2024-03-31T15:14:08.271941+0000","last_active":"2024-03-31T15:14:08.275222+0000","last_peered":"2024-03-31T15:14:08.275222+0000","last_clean":"2024-03-31T15:14:08.275222+0000","last_became_active":"2024-03-31T15:14:08.271589+0000","last_became_peered":"2024-03-31T15:14:08.271589+0000","last_unstale":"2024-03-31T15:14:08.275222+0000","last_undegraded":"2024-03-31T15:14:08.275222+0000","last_fullsized":"2024-03-31T15:14:08.275222+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_clean_scrub_stamp":"2024-03-31T15:14:07.235046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:41:56.592702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":331,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-31T15:14:08.262750+0000","last_change":"2024-03-31T15:03:24.329890+0000","last_active":"2024-03-31T15:14:08.262750+0000","last_peered":"2024-03-31T15:14:08.262750+0000","last_clean":"2024-03-31T15:14:08.262750+0000","last_became_active":"2024-03-31T15:03:24.328882+0000","last_became_peered":"2024-03-31T15:03:24.328882+0000","last_unstale":"2024-03-31T15:14:08.262750+0000","last_undegraded":"2024-03-31T15:14:08.262750+0000","last_fullsized":"2024-03-31T15:14:08.262750+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:22.885535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:22.885535+0000","last_clean_scrub_stamp":"2024-03-31T15:03:22.885535+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:18:29.004296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":17,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":16,"num_pg":32,"stat_sum":{"num_bytes":1828,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":19,"ondisk_log_size":19,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509574,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33040,"kb_used_data":1860,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338800,"statfs":{"total":96636764160,"available":96602931200,"internally_reserved":0,"allocated":1904640,"data_stored":1155091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509573,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32476,"kb_used_data":1292,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339364,"statfs":{"total":96636764160,"available":96603508736,"internally_reserved":0,"allocated":1323008,"data_stored":570998,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":15,"seq":64424509572,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32460,"kb_used_data":1280,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339380,"statfs":{"total":96636764160,"available":96603525120,"internally_reserved":0,"allocated":1310720,"data_stored":564723,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509571,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36632,"kb_used_data":1292,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335208,"statfs":{"total":96636764160,"available":96599252992,"internally_reserved":0,"allocated":1323008,"data_stored":570998,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542278,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32508,"kb_used_data":1320,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339332,"statfs":{"total":96636764160,"available":96603475968,"internally_reserved":0,"allocated":1351680,"data_stored":584143,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542278,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32512,"kb_used_data":1332,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339328,"statfs":{"total":96636764160,"available":96603471872,"internally_reserved":0,"allocated":1363968,"data_stored":585755,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542278,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32504,"kb_used_data":1324,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339336,"statfs":{"total":96636764160,"available":96603480064,"internally_reserved":0,"allocated":1355776,"data_stored":584197,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542278,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33100,"kb_used_data":1916,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338740,"statfs":{"total":96636764160,"available":96602869760,"internally_reserved":0,"allocated":1961984,"data_stored":1176165,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-31T15:14:13.372 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-31T15:14:13.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:13.554+0000 7fac84600640 1 Processor -- start 2024-03-31T15:14:13.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:13.554+0000 7fac84600640 1 -- start start 2024-03-31T15:14:13.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:13.558+0000 7fac84600640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fac7c0b9c60 0x7fac7c0b7470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:13.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:13.558+0000 7fac84600640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fac7c0b79b0 0x7fac7c0b7e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:13.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:13.558+0000 7fac84600640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fac7c0d3990 con 0x7fac7c0d1280 2024-03-31T15:14:13.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:13.558+0000 7fac84600640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7fac7c0d26a0 con 0x7fac7c0b9c60 2024-03-31T15:14:13.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:13.558+0000 7fac84600640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7fac7c0d28f0 con 0x7fac7c0b79b0 2024-03-31T15:14:13.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:13.558+0000 7fac81200640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fac7c0b9c60 0x7fac7c0b7470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:13.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:13.558+0000 7fac81c00640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fac7c0d1280 0x7fac7c0d1660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:57648/0 (socket says 172.21.15.16:57648) 2024-03-31T15:14:13.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:13.558+0000 7fac81c00640 1 -- 172.21.15.16:0/3211755570 learned_addr learned my addr 172.21.15.16:0/3211755570 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:14:13.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:13.558+0000 7fac82600640 1 --2- 172.21.15.16:0/3211755570 >> v2:172.21.15.89:3300/0 conn(0x7fac7c0b79b0 0x7fac7c0b7e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:13.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:13.558+0000 7fac81200640 1 -- 172.21.15.16:0/3211755570 >> v2:172.21.15.89:3300/0 conn(0x7fac7c0b79b0 msgr2=0x7fac7c0b7e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:14:13.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:13.558+0000 7fac81200640 1 --2- 172.21.15.16:0/3211755570 >> v2:172.21.15.89:3300/0 conn(0x7fac7c0b79b0 0x7fac7c0b7e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:13.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:13.558+0000 7fac81200640 1 -- 172.21.15.16:0/3211755570 >> v1:172.21.15.16:6789/0 conn(0x7fac7c0d1280 legacy=0x7fac7c0d1660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:13.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:13.558+0000 7fac81200640 1 -- 172.21.15.16:0/3211755570 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac7c0d2bd0 con 0x7fac7c0b9c60 2024-03-31T15:14:13.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:13.558+0000 7fac82600640 1 --2- 172.21.15.16:0/3211755570 >> v2:172.21.15.89:3300/0 conn(0x7fac7c0b79b0 0x7fac7c0b7e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:14:13.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:13.562+0000 7fac81200640 1 --2- 172.21.15.16:0/3211755570 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fac7c0b9c60 0x7fac7c0b7470 secure :-1 s=READY pgs=1654 cs=0 l=1 rev1=1 crypto rx=0x7fac7c0b7240 tx=0x7fac6801c3f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48835350299c69f server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:13.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:13.562+0000 7fac7be00640 1 -- 172.21.15.16:0/3211755570 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac680b0070 con 0x7fac7c0b9c60 2024-03-31T15:14:13.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:13.562+0000 7fac7be00640 1 -- 172.21.15.16:0/3211755570 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fac680ab600 con 0x7fac7c0b9c60 2024-03-31T15:14:13.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:13.562+0000 7fac7be00640 1 -- 172.21.15.16:0/3211755570 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac680ab9d0 con 0x7fac7c0b9c60 2024-03-31T15:14:13.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:13.562+0000 7fac84600640 1 -- 172.21.15.16:0/3211755570 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fac7c0b9c60 msgr2=0x7fac7c0b7470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:13.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:13.562+0000 7fac84600640 1 --2- 172.21.15.16:0/3211755570 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fac7c0b9c60 0x7fac7c0b7470 secure :-1 s=READY pgs=1654 cs=0 l=1 rev1=1 crypto rx=0x7fac7c0b7240 tx=0x7fac6801c3f0 comp rx=0 tx=0).stop 2024-03-31T15:14:13.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:13.562+0000 7fac84600640 1 -- 172.21.15.16:0/3211755570 shutdown_connections 2024-03-31T15:14:13.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:13.562+0000 7fac84600640 1 --2- 172.21.15.16:0/3211755570 >> v2:172.21.15.89:3300/0 conn(0x7fac7c0b79b0 0x7fac7c0b7e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:13.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:13.562+0000 7fac84600640 1 --2- 172.21.15.16:0/3211755570 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fac7c0b9c60 0x7fac7c0b7470 unknown :-1 s=CLOSED pgs=1654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:13.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:13.562+0000 7fac84600640 1 -- 172.21.15.16:0/3211755570 >> 172.21.15.16:0/3211755570 conn(0x7fac7c067730 msgr2=0x7fac7c067b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:14:13.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:13.562+0000 7fac84600640 1 -- 172.21.15.16:0/3211755570 shutdown_connections 2024-03-31T15:14:13.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:13.562+0000 7fac84600640 1 -- 172.21.15.16:0/3211755570 wait complete. 2024-03-31T15:14:13.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:13.566+0000 7fac84600640 1 Processor -- start 2024-03-31T15:14:13.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:13.566+0000 7fac84600640 1 -- start start 2024-03-31T15:14:13.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:13.566+0000 7fac84600640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fac7c0b79b0 0x7fac7c0cda20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:13.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:13.566+0000 7fac84600640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fac7c0b9c60 0x7fac7c0b4360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:13.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:13.566+0000 7fac84600640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fac7c0d1280 0x7fac7c0b48a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:13.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:13.566+0000 7fac84600640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fac7c0d4010 con 0x7fac7c0b79b0 2024-03-31T15:14:13.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:13.566+0000 7fac84600640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fac7c0d3e90 con 0x7fac7c0b9c60 2024-03-31T15:14:13.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:13.566+0000 7fac84600640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fac7c0ba0e0 con 0x7fac7c0d1280 2024-03-31T15:14:13.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:13.566+0000 7fac81c00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fac7c0b79b0 0x7fac7c0cda20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:13.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:13.566+0000 7fac81c00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fac7c0b79b0 0x7fac7c0cda20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50136/0 (socket says 172.21.15.16:50136) 2024-03-31T15:14:13.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:13.566+0000 7fac81c00640 1 -- 172.21.15.16:0/412010722 learned_addr learned my addr 172.21.15.16:0/412010722 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:14:13.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:13.566+0000 7fac81200640 1 --2- 172.21.15.16:0/412010722 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fac7c0b9c60 0x7fac7c0b4360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:13.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:13.566+0000 7fac82600640 1 --2- 172.21.15.16:0/412010722 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fac7c0d1280 0x7fac7c0b48a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:13.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:13.566+0000 7fac81c00640 1 -- 172.21.15.16:0/412010722 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fac7c0d1280 msgr2=0x7fac7c0b48a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:13.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:13.566+0000 7fac81c00640 1 --2- 172.21.15.16:0/412010722 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fac7c0d1280 0x7fac7c0b48a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:13.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:13.566+0000 7fac81c00640 1 -- 172.21.15.16:0/412010722 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fac7c0b9c60 msgr2=0x7fac7c0b4360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:13.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:13.566+0000 7fac81c00640 1 --2- 172.21.15.16:0/412010722 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fac7c0b9c60 0x7fac7c0b4360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:13.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:13.566+0000 7fac81c00640 1 -- 172.21.15.16:0/412010722 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac7c0cbdb0 con 0x7fac7c0b79b0 2024-03-31T15:14:13.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:13.566+0000 7fac82600640 1 --2- 172.21.15.16:0/412010722 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fac7c0d1280 0x7fac7c0b48a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:14:13.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:13.566+0000 7fac81c00640 1 --2- 172.21.15.16:0/412010722 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fac7c0b79b0 0x7fac7c0cda20 secure :-1 s=READY pgs=1655 cs=0 l=1 rev1=1 crypto rx=0x7fac7000c3a0 tx=0x7fac7000c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:13.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:13.566+0000 7fac79600640 1 -- 172.21.15.16:0/412010722 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac70017070 con 0x7fac7c0b79b0 2024-03-31T15:14:13.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:13.566+0000 7fac79600640 1 -- 172.21.15.16:0/412010722 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fac70005cf0 con 0x7fac7c0b79b0 2024-03-31T15:14:13.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:13.566+0000 7fac79600640 1 -- 172.21.15.16:0/412010722 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac700072f0 con 0x7fac7c0b79b0 2024-03-31T15:14:13.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:13.566+0000 7fac84600640 1 -- 172.21.15.16:0/412010722 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fac7c0cc090 con 0x7fac7c0b79b0 2024-03-31T15:14:13.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:13.566+0000 7fac84600640 1 -- 172.21.15.16:0/412010722 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fac7c0cc650 con 0x7fac7c0b79b0 2024-03-31T15:14:13.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:13.566+0000 7fac84600640 1 -- 172.21.15.16:0/412010722 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fac7c069950 con 0x7fac7c0b79b0 2024-03-31T15:14:13.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:13.570+0000 7fac79600640 1 -- 172.21.15.16:0/412010722 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fac7001a070 con 0x7fac7c0b79b0 2024-03-31T15:14:13.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:13.570+0000 7fac79600640 1 --2- 172.21.15.16:0/412010722 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fac5c068380 0x7fac5c06a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:13.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:13.570+0000 7fac79600640 1 -- 172.21.15.16:0/412010722 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 5709+0+0 (secure 0 0 0) 0x7fac70085d00 con 0x7fac7c0b79b0 2024-03-31T15:14:13.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:13.570+0000 7fac81200640 1 --2- 172.21.15.16:0/412010722 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fac5c068380 0x7fac5c06a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:13.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:13.570+0000 7fac81200640 1 --2- 172.21.15.16:0/412010722 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fac5c068380 0x7fac5c06a840 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7fac680ad000 tx=0x7fac68008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:13.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:13.578+0000 7fac79600640 1 -- 172.21.15.16:0/412010722 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fac700562b0 con 0x7fac7c0b79b0 2024-03-31T15:14:13.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:13.742+0000 7fac84600640 1 -- 172.21.15.16:0/412010722 --> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fac7c0b3f80 con 0x7fac5c068380 2024-03-31T15:14:13.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:13.750+0000 7fac79600640 1 -- 172.21.15.16:0/412010722 <== mgr.4100 v2:172.21.15.89:6832/1549407273 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+250056 (secure 0 0 0) 0x7fac7c0b3f80 con 0x7fac5c068380 2024-03-31T15:14:13.744 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:14:13.745 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-03-31T15:14:13.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:13.750+0000 7fac84600640 1 -- 172.21.15.16:0/412010722 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fac5c068380 msgr2=0x7fac5c06a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:13.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:13.750+0000 7fac84600640 1 --2- 172.21.15.16:0/412010722 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fac5c068380 0x7fac5c06a840 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7fac680ad000 tx=0x7fac68008040 comp rx=0 tx=0).stop 2024-03-31T15:14:13.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:13.750+0000 7fac84600640 1 -- 172.21.15.16:0/412010722 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fac7c0b79b0 msgr2=0x7fac7c0cda20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:13.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:13.750+0000 7fac84600640 1 --2- 172.21.15.16:0/412010722 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fac7c0b79b0 0x7fac7c0cda20 secure :-1 s=READY pgs=1655 cs=0 l=1 rev1=1 crypto rx=0x7fac7000c3a0 tx=0x7fac7000c860 comp rx=0 tx=0).stop 2024-03-31T15:14:13.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:13.750+0000 7fac84600640 1 -- 172.21.15.16:0/412010722 shutdown_connections 2024-03-31T15:14:13.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:13.750+0000 7fac84600640 1 --2- 172.21.15.16:0/412010722 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fac5c068380 0x7fac5c06a840 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:13.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:13.750+0000 7fac84600640 1 --2- 172.21.15.16:0/412010722 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fac7c0d1280 0x7fac7c0b48a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:13.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:13.750+0000 7fac84600640 1 --2- 172.21.15.16:0/412010722 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fac7c0b9c60 0x7fac7c0b4360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:13.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:13.750+0000 7fac84600640 1 --2- 172.21.15.16:0/412010722 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fac7c0b79b0 0x7fac7c0cda20 unknown :-1 s=CLOSED pgs=1655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:13.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:13.750+0000 7fac84600640 1 -- 172.21.15.16:0/412010722 >> 172.21.15.16:0/412010722 conn(0x7fac7c067730 msgr2=0x7fac7c160d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:14:13.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:13.750+0000 7fac84600640 1 -- 172.21.15.16:0/412010722 shutdown_connections 2024-03-31T15:14:13.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:13.754+0000 7fac84600640 1 -- 172.21.15.16:0/412010722 wait complete. 2024-03-31T15:14:13.764 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":427,"stamp":"2024-03-31T15:14:12.693312+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592196,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":179,"num_write_kb":2429,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":136,"ondisk_log_size":136,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":104,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":265232,"kb_used_data":11616,"kb_used_omap":12,"kb_used_meta":253491,"kb_avail":754709488,"statfs":{"total":773094113280,"available":772822515712,"internally_reserved":0,"allocated":11894784,"data_stored":5792070,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":259575344},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":32,"apply_latency_ms":32,"commit_latency_ns":32000000,"apply_latency_ns":32000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1828,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.446790"},"pg_stats":[{"pgid":"17.3f","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:10.892463+0000","last_change":"2024-03-31T15:14:09.277746+0000","last_active":"2024-03-31T15:14:10.892463+0000","last_peered":"2024-03-31T15:14:10.892463+0000","last_clean":"2024-03-31T15:14:10.892463+0000","last_became_active":"2024-03-31T15:14:09.277207+0000","last_became_peered":"2024-03-31T15:14:09.277207+0000","last_unstale":"2024-03-31T15:14:10.892463+0000","last_undegraded":"2024-03-31T15:14:10.892463+0000","last_fullsized":"2024-03-31T15:14:10.892463+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T17:42:57.076247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3e","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-03-31T15:14:08.253600+0000","last_change":"2024-03-31T15:14:08.253600+0000","last_active":"2024-03-31T15:14:08.242094+0000","last_peered":"2024-03-31T15:14:08.242094+0000","last_clean":"2024-03-31T15:14:08.242094+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:14:08.253600+0000","last_undegraded":"2024-03-31T15:14:08.253600+0000","last_fullsized":"2024-03-31T15:14:08.253600+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.3d","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-03-31T15:14:08.258560+0000","last_change":"2024-03-31T15:14:08.258560+0000","last_active":"2024-03-31T15:14:08.242094+0000","last_peered":"2024-03-31T15:14:08.242094+0000","last_clean":"2024-03-31T15:14:08.242094+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:14:08.258560+0000","last_undegraded":"2024-03-31T15:14:08.258560+0000","last_fullsized":"2024-03-31T15:14:08.258560+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.3c","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:10.892010+0000","last_change":"2024-03-31T15:14:09.276706+0000","last_active":"2024-03-31T15:14:10.892010+0000","last_peered":"2024-03-31T15:14:10.892010+0000","last_clean":"2024-03-31T15:14:10.892010+0000","last_became_active":"2024-03-31T15:14:09.275545+0000","last_became_peered":"2024-03-31T15:14:09.275545+0000","last_unstale":"2024-03-31T15:14:10.892010+0000","last_undegraded":"2024-03-31T15:14:10.892010+0000","last_fullsized":"2024-03-31T15:14:10.892010+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T19:37:12.397888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3b","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-03-31T15:14:08.255591+0000","last_change":"2024-03-31T15:14:08.255591+0000","last_active":"2024-03-31T15:14:08.242094+0000","last_peered":"2024-03-31T15:14:08.242094+0000","last_clean":"2024-03-31T15:14:08.242094+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:14:08.255591+0000","last_undegraded":"2024-03-31T15:14:08.255591+0000","last_fullsized":"2024-03-31T15:14:08.255591+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.3a","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:11.390994+0000","last_change":"2024-03-31T15:14:09.283260+0000","last_active":"2024-03-31T15:14:11.390994+0000","last_peered":"2024-03-31T15:14:11.390994+0000","last_clean":"2024-03-31T15:14:11.390994+0000","last_became_active":"2024-03-31T15:14:09.282995+0000","last_became_peered":"2024-03-31T15:14:09.282995+0000","last_unstale":"2024-03-31T15:14:11.390994+0000","last_undegraded":"2024-03-31T15:14:11.390994+0000","last_fullsized":"2024-03-31T15:14:11.390994+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:27:30.858064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.39","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-03-31T15:14:08.259762+0000","last_change":"2024-03-31T15:14:08.259762+0000","last_active":"2024-03-31T15:14:08.242094+0000","last_peered":"2024-03-31T15:14:08.242094+0000","last_clean":"2024-03-31T15:14:08.242094+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:14:08.259762+0000","last_undegraded":"2024-03-31T15:14:08.259762+0000","last_fullsized":"2024-03-31T15:14:08.259762+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.38","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-03-31T15:14:08.258038+0000","last_change":"2024-03-31T15:14:08.258038+0000","last_active":"2024-03-31T15:14:08.242094+0000","last_peered":"2024-03-31T15:14:08.242094+0000","last_clean":"2024-03-31T15:14:08.242094+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:14:08.258038+0000","last_undegraded":"2024-03-31T15:14:08.258038+0000","last_fullsized":"2024-03-31T15:14:08.258038+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.37","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:10.892467+0000","last_change":"2024-03-31T15:14:09.284680+0000","last_active":"2024-03-31T15:14:10.892467+0000","last_peered":"2024-03-31T15:14:10.892467+0000","last_clean":"2024-03-31T15:14:10.892467+0000","last_became_active":"2024-03-31T15:14:09.282533+0000","last_became_peered":"2024-03-31T15:14:09.282533+0000","last_unstale":"2024-03-31T15:14:10.892467+0000","last_undegraded":"2024-03-31T15:14:10.892467+0000","last_fullsized":"2024-03-31T15:14:10.892467+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:04:35.760087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.36","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:10.892246+0000","last_change":"2024-03-31T15:14:09.290368+0000","last_active":"2024-03-31T15:14:10.892246+0000","last_peered":"2024-03-31T15:14:10.892246+0000","last_clean":"2024-03-31T15:14:10.892246+0000","last_became_active":"2024-03-31T15:14:09.290077+0000","last_became_peered":"2024-03-31T15:14:09.290077+0000","last_unstale":"2024-03-31T15:14:10.892246+0000","last_undegraded":"2024-03-31T15:14:10.892246+0000","last_fullsized":"2024-03-31T15:14:10.892246+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:23:53.326762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.35","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-03-31T15:14:08.253324+0000","last_change":"2024-03-31T15:14:08.253324+0000","last_active":"2024-03-31T15:14:08.242094+0000","last_peered":"2024-03-31T15:14:08.242094+0000","last_clean":"2024-03-31T15:14:08.242094+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:14:08.253324+0000","last_undegraded":"2024-03-31T15:14:08.253324+0000","last_fullsized":"2024-03-31T15:14:08.253324+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.34","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:10.892056+0000","last_change":"2024-03-31T15:14:09.276847+0000","last_active":"2024-03-31T15:14:10.892056+0000","last_peered":"2024-03-31T15:14:10.892056+0000","last_clean":"2024-03-31T15:14:10.892056+0000","last_became_active":"2024-03-31T15:14:09.275249+0000","last_became_peered":"2024-03-31T15:14:09.275249+0000","last_unstale":"2024-03-31T15:14:10.892056+0000","last_undegraded":"2024-03-31T15:14:10.892056+0000","last_fullsized":"2024-03-31T15:14:10.892056+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T16:56:09.377594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.33","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-03-31T15:14:08.258326+0000","last_change":"2024-03-31T15:14:08.258326+0000","last_active":"2024-03-31T15:14:08.242094+0000","last_peered":"2024-03-31T15:14:08.242094+0000","last_clean":"2024-03-31T15:14:08.242094+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:14:08.258326+0000","last_undegraded":"2024-03-31T15:14:08.258326+0000","last_fullsized":"2024-03-31T15:14:08.258326+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.32","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-03-31T15:14:08.257741+0000","last_change":"2024-03-31T15:14:08.257741+0000","last_active":"2024-03-31T15:14:08.242094+0000","last_peered":"2024-03-31T15:14:08.242094+0000","last_clean":"2024-03-31T15:14:08.242094+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:14:08.257741+0000","last_undegraded":"2024-03-31T15:14:08.257741+0000","last_fullsized":"2024-03-31T15:14:08.257741+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.31","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:11.390625+0000","last_change":"2024-03-31T15:14:09.285784+0000","last_active":"2024-03-31T15:14:11.390625+0000","last_peered":"2024-03-31T15:14:11.390625+0000","last_clean":"2024-03-31T15:14:11.390625+0000","last_became_active":"2024-03-31T15:14:09.285374+0000","last_became_peered":"2024-03-31T15:14:09.285374+0000","last_unstale":"2024-03-31T15:14:11.390625+0000","last_undegraded":"2024-03-31T15:14:11.390625+0000","last_fullsized":"2024-03-31T15:14:11.390625+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:34:11.338914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.30","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:10.891716+0000","last_change":"2024-03-31T15:14:09.287510+0000","last_active":"2024-03-31T15:14:10.891716+0000","last_peered":"2024-03-31T15:14:10.891716+0000","last_clean":"2024-03-31T15:14:10.891716+0000","last_became_active":"2024-03-31T15:14:09.287238+0000","last_became_peered":"2024-03-31T15:14:09.287238+0000","last_unstale":"2024-03-31T15:14:10.891716+0000","last_undegraded":"2024-03-31T15:14:10.891716+0000","last_fullsized":"2024-03-31T15:14:10.891716+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:14:09.225080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.2f","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-03-31T15:14:08.254889+0000","last_change":"2024-03-31T15:14:08.254889+0000","last_active":"2024-03-31T15:14:08.242094+0000","last_peered":"2024-03-31T15:14:08.242094+0000","last_clean":"2024-03-31T15:14:08.242094+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:14:08.254889+0000","last_undegraded":"2024-03-31T15:14:08.254889+0000","last_fullsized":"2024-03-31T15:14:08.254889+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2e","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:10.279438+0000","last_change":"2024-03-31T15:14:09.276215+0000","last_active":"2024-03-31T15:14:10.279438+0000","last_peered":"2024-03-31T15:14:10.279438+0000","last_clean":"2024-03-31T15:14:10.279438+0000","last_became_active":"2024-03-31T15:14:09.275956+0000","last_became_peered":"2024-03-31T15:14:09.275956+0000","last_unstale":"2024-03-31T15:14:10.279438+0000","last_undegraded":"2024-03-31T15:14:10.279438+0000","last_fullsized":"2024-03-31T15:14:10.279438+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:54:50.675440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.2d","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-03-31T15:14:08.258151+0000","last_change":"2024-03-31T15:14:08.258151+0000","last_active":"2024-03-31T15:14:08.242094+0000","last_peered":"2024-03-31T15:14:08.242094+0000","last_clean":"2024-03-31T15:14:08.242094+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:14:08.258151+0000","last_undegraded":"2024-03-31T15:14:08.258151+0000","last_fullsized":"2024-03-31T15:14:08.258151+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.2c","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-03-31T15:14:08.253580+0000","last_change":"2024-03-31T15:14:08.253580+0000","last_active":"2024-03-31T15:14:08.242094+0000","last_peered":"2024-03-31T15:14:08.242094+0000","last_clean":"2024-03-31T15:14:08.242094+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:14:08.253580+0000","last_undegraded":"2024-03-31T15:14:08.253580+0000","last_fullsized":"2024-03-31T15:14:08.253580+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2b","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-03-31T15:14:08.253223+0000","last_change":"2024-03-31T15:14:08.253223+0000","last_active":"2024-03-31T15:14:08.242094+0000","last_peered":"2024-03-31T15:14:08.242094+0000","last_clean":"2024-03-31T15:14:08.242094+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:14:08.253223+0000","last_undegraded":"2024-03-31T15:14:08.253223+0000","last_fullsized":"2024-03-31T15:14:08.253223+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2a","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:11.390332+0000","last_change":"2024-03-31T15:14:09.275037+0000","last_active":"2024-03-31T15:14:11.390332+0000","last_peered":"2024-03-31T15:14:11.390332+0000","last_clean":"2024-03-31T15:14:11.390332+0000","last_became_active":"2024-03-31T15:14:09.274177+0000","last_became_peered":"2024-03-31T15:14:09.274177+0000","last_unstale":"2024-03-31T15:14:11.390332+0000","last_undegraded":"2024-03-31T15:14:11.390332+0000","last_fullsized":"2024-03-31T15:14:11.390332+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:09:48.241483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.29","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-03-31T15:14:08.253496+0000","last_change":"2024-03-31T15:14:08.253496+0000","last_active":"2024-03-31T15:14:08.242094+0000","last_peered":"2024-03-31T15:14:08.242094+0000","last_clean":"2024-03-31T15:14:08.242094+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:14:08.253496+0000","last_undegraded":"2024-03-31T15:14:08.253496+0000","last_fullsized":"2024-03-31T15:14:08.253496+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.28","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:11.390297+0000","last_change":"2024-03-31T15:14:09.283380+0000","last_active":"2024-03-31T15:14:11.390297+0000","last_peered":"2024-03-31T15:14:11.390297+0000","last_clean":"2024-03-31T15:14:11.390297+0000","last_became_active":"2024-03-31T15:14:09.283097+0000","last_became_peered":"2024-03-31T15:14:09.283097+0000","last_unstale":"2024-03-31T15:14:11.390297+0000","last_undegraded":"2024-03-31T15:14:11.390297+0000","last_fullsized":"2024-03-31T15:14:11.390297+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:00:11.817601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.27","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-03-31T15:14:08.254451+0000","last_change":"2024-03-31T15:14:08.254451+0000","last_active":"2024-03-31T15:14:08.242094+0000","last_peered":"2024-03-31T15:14:08.242094+0000","last_clean":"2024-03-31T15:14:08.242094+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:14:08.254451+0000","last_undegraded":"2024-03-31T15:14:08.254451+0000","last_fullsized":"2024-03-31T15:14:08.254451+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.26","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:10.892253+0000","last_change":"2024-03-31T15:14:09.276124+0000","last_active":"2024-03-31T15:14:10.892253+0000","last_peered":"2024-03-31T15:14:10.892253+0000","last_clean":"2024-03-31T15:14:10.892253+0000","last_became_active":"2024-03-31T15:14:09.275495+0000","last_became_peered":"2024-03-31T15:14:09.275495+0000","last_unstale":"2024-03-31T15:14:10.892253+0000","last_undegraded":"2024-03-31T15:14:10.892253+0000","last_fullsized":"2024-03-31T15:14:10.892253+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:38:53.950142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.25","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:11.391422+0000","last_change":"2024-03-31T15:14:09.290617+0000","last_active":"2024-03-31T15:14:11.391422+0000","last_peered":"2024-03-31T15:14:11.391422+0000","last_clean":"2024-03-31T15:14:11.391422+0000","last_became_active":"2024-03-31T15:14:09.290308+0000","last_became_peered":"2024-03-31T15:14:09.290308+0000","last_unstale":"2024-03-31T15:14:11.391422+0000","last_undegraded":"2024-03-31T15:14:11.391422+0000","last_fullsized":"2024-03-31T15:14:11.391422+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:43:42.509329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.24","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:11.391301+0000","last_change":"2024-03-31T15:14:09.290865+0000","last_active":"2024-03-31T15:14:11.391301+0000","last_peered":"2024-03-31T15:14:11.391301+0000","last_clean":"2024-03-31T15:14:11.391301+0000","last_became_active":"2024-03-31T15:14:09.290601+0000","last_became_peered":"2024-03-31T15:14:09.290601+0000","last_unstale":"2024-03-31T15:14:11.391301+0000","last_undegraded":"2024-03-31T15:14:11.391301+0000","last_fullsized":"2024-03-31T15:14:11.391301+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T19:21:02.579812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.23","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-03-31T15:14:08.258913+0000","last_change":"2024-03-31T15:14:08.258913+0000","last_active":"2024-03-31T15:14:08.242094+0000","last_peered":"2024-03-31T15:14:08.242094+0000","last_clean":"2024-03-31T15:14:08.242094+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:14:08.258913+0000","last_undegraded":"2024-03-31T15:14:08.258913+0000","last_fullsized":"2024-03-31T15:14:08.258913+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.22","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-03-31T15:14:08.260326+0000","last_change":"2024-03-31T15:14:08.260326+0000","last_active":"2024-03-31T15:14:08.242094+0000","last_peered":"2024-03-31T15:14:08.242094+0000","last_clean":"2024-03-31T15:14:08.242094+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:14:08.260326+0000","last_undegraded":"2024-03-31T15:14:08.260326+0000","last_fullsized":"2024-03-31T15:14:08.260326+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.21","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:10.892072+0000","last_change":"2024-03-31T15:14:09.276137+0000","last_active":"2024-03-31T15:14:10.892072+0000","last_peered":"2024-03-31T15:14:10.892072+0000","last_clean":"2024-03-31T15:14:10.892072+0000","last_became_active":"2024-03-31T15:14:09.275873+0000","last_became_peered":"2024-03-31T15:14:09.275873+0000","last_unstale":"2024-03-31T15:14:10.892072+0000","last_undegraded":"2024-03-31T15:14:10.892072+0000","last_fullsized":"2024-03-31T15:14:10.892072+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T17:03:26.938790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.20","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:11.389806+0000","last_change":"2024-03-31T15:14:09.285377+0000","last_active":"2024-03-31T15:14:11.389806+0000","last_peered":"2024-03-31T15:14:11.389806+0000","last_clean":"2024-03-31T15:14:11.389806+0000","last_became_active":"2024-03-31T15:14:09.285077+0000","last_became_peered":"2024-03-31T15:14:09.285077+0000","last_unstale":"2024-03-31T15:14:11.389806+0000","last_undegraded":"2024-03-31T15:14:11.389806+0000","last_fullsized":"2024-03-31T15:14:11.389806+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T15:39:45.840997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.1e","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:11.391059+0000","last_change":"2024-03-31T15:14:09.286260+0000","last_active":"2024-03-31T15:14:11.391059+0000","last_peered":"2024-03-31T15:14:11.391059+0000","last_clean":"2024-03-31T15:14:11.391059+0000","last_became_active":"2024-03-31T15:14:09.285985+0000","last_became_peered":"2024-03-31T15:14:09.285985+0000","last_unstale":"2024-03-31T15:14:11.391059+0000","last_undegraded":"2024-03-31T15:14:11.391059+0000","last_fullsized":"2024-03-31T15:14:11.391059+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T15:58:28.943455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1f","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-31T15:14:08.272805+0000","last_change":"2024-03-31T15:14:08.268805+0000","last_active":"2024-03-31T15:14:08.272805+0000","last_peered":"2024-03-31T15:14:08.272805+0000","last_clean":"2024-03-31T15:14:08.272805+0000","last_became_active":"2024-03-31T15:14:08.268515+0000","last_became_peered":"2024-03-31T15:14:08.268515+0000","last_unstale":"2024-03-31T15:14:08.272805+0000","last_undegraded":"2024-03-31T15:14:08.272805+0000","last_fullsized":"2024-03-31T15:14:08.272805+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_clean_scrub_stamp":"2024-03-31T15:14:07.235046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:47:23.413340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1f","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:10.278621+0000","last_change":"2024-03-31T15:14:09.281053+0000","last_active":"2024-03-31T15:14:10.278621+0000","last_peered":"2024-03-31T15:14:10.278621+0000","last_clean":"2024-03-31T15:14:10.278621+0000","last_became_active":"2024-03-31T15:14:09.280735+0000","last_became_peered":"2024-03-31T15:14:09.280735+0000","last_unstale":"2024-03-31T15:14:10.278621+0000","last_undegraded":"2024-03-31T15:14:10.278621+0000","last_fullsized":"2024-03-31T15:14:10.278621+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:55:28.086183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.1e","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-31T15:14:08.273342+0000","last_change":"2024-03-31T15:14:08.270517+0000","last_active":"2024-03-31T15:14:08.273342+0000","last_peered":"2024-03-31T15:14:08.273342+0000","last_clean":"2024-03-31T15:14:08.273342+0000","last_became_active":"2024-03-31T15:14:08.270219+0000","last_became_peered":"2024-03-31T15:14:08.270219+0000","last_unstale":"2024-03-31T15:14:08.273342+0000","last_undegraded":"2024-03-31T15:14:08.273342+0000","last_fullsized":"2024-03-31T15:14:08.273342+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_clean_scrub_stamp":"2024-03-31T15:14:07.235046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:40:13.939016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1c","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:11.391070+0000","last_change":"2024-03-31T15:14:09.285776+0000","last_active":"2024-03-31T15:14:11.391070+0000","last_peered":"2024-03-31T15:14:11.391070+0000","last_clean":"2024-03-31T15:14:11.391070+0000","last_became_active":"2024-03-31T15:14:09.285383+0000","last_became_peered":"2024-03-31T15:14:09.285383+0000","last_unstale":"2024-03-31T15:14:11.391070+0000","last_undegraded":"2024-03-31T15:14:11.391070+0000","last_fullsized":"2024-03-31T15:14:11.391070+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:45:29.558658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1d","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-31T15:14:08.274496+0000","last_change":"2024-03-31T15:14:08.268956+0000","last_active":"2024-03-31T15:14:08.274496+0000","last_peered":"2024-03-31T15:14:08.274496+0000","last_clean":"2024-03-31T15:14:08.274496+0000","last_became_active":"2024-03-31T15:14:08.268672+0000","last_became_peered":"2024-03-31T15:14:08.268672+0000","last_unstale":"2024-03-31T15:14:08.274496+0000","last_undegraded":"2024-03-31T15:14:08.274496+0000","last_fullsized":"2024-03-31T15:14:08.274496+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_clean_scrub_stamp":"2024-03-31T15:14:07.235046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:55:47.095073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1d","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-03-31T15:14:08.257689+0000","last_change":"2024-03-31T15:14:08.257689+0000","last_active":"2024-03-31T15:14:08.242094+0000","last_peered":"2024-03-31T15:14:08.242094+0000","last_clean":"2024-03-31T15:14:08.242094+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:14:08.257689+0000","last_undegraded":"2024-03-31T15:14:08.257689+0000","last_fullsized":"2024-03-31T15:14:08.257689+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1c","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-31T15:14:08.274904+0000","last_change":"2024-03-31T15:14:08.269987+0000","last_active":"2024-03-31T15:14:08.274904+0000","last_peered":"2024-03-31T15:14:08.274904+0000","last_clean":"2024-03-31T15:14:08.274904+0000","last_became_active":"2024-03-31T15:14:08.269241+0000","last_became_peered":"2024-03-31T15:14:08.269241+0000","last_unstale":"2024-03-31T15:14:08.274904+0000","last_undegraded":"2024-03-31T15:14:08.274904+0000","last_fullsized":"2024-03-31T15:14:08.274904+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_clean_scrub_stamp":"2024-03-31T15:14:07.235046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:54:34.951227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1a","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:10.891958+0000","last_change":"2024-03-31T15:14:09.282143+0000","last_active":"2024-03-31T15:14:10.891958+0000","last_peered":"2024-03-31T15:14:10.891958+0000","last_clean":"2024-03-31T15:14:10.891958+0000","last_became_active":"2024-03-31T15:14:09.281851+0000","last_became_peered":"2024-03-31T15:14:09.281851+0000","last_unstale":"2024-03-31T15:14:10.891958+0000","last_undegraded":"2024-03-31T15:14:10.891958+0000","last_fullsized":"2024-03-31T15:14:10.891958+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:07:40.319619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.1b","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-31T15:14:08.275156+0000","last_change":"2024-03-31T15:14:08.271392+0000","last_active":"2024-03-31T15:14:08.275156+0000","last_peered":"2024-03-31T15:14:08.275156+0000","last_clean":"2024-03-31T15:14:08.275156+0000","last_became_active":"2024-03-31T15:14:08.271138+0000","last_became_peered":"2024-03-31T15:14:08.271138+0000","last_unstale":"2024-03-31T15:14:08.275156+0000","last_undegraded":"2024-03-31T15:14:08.275156+0000","last_fullsized":"2024-03-31T15:14:08.275156+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_clean_scrub_stamp":"2024-03-31T15:14:07.235046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:45:22.838594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1b","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-03-31T15:14:08.258473+0000","last_change":"2024-03-31T15:14:08.258473+0000","last_active":"2024-03-31T15:14:08.242094+0000","last_peered":"2024-03-31T15:14:08.242094+0000","last_clean":"2024-03-31T15:14:08.242094+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:14:08.258473+0000","last_undegraded":"2024-03-31T15:14:08.258473+0000","last_fullsized":"2024-03-31T15:14:08.258473+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.1a","version":"108'1","reported_seq":15,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:11.390384+0000","last_change":"2024-03-31T15:14:08.273955+0000","last_active":"2024-03-31T15:14:11.390384+0000","last_peered":"2024-03-31T15:14:11.390384+0000","last_clean":"2024-03-31T15:14:11.390384+0000","last_became_active":"2024-03-31T15:14:08.273646+0000","last_became_peered":"2024-03-31T15:14:08.273646+0000","last_unstale":"2024-03-31T15:14:11.390384+0000","last_undegraded":"2024-03-31T15:14:11.390384+0000","last_fullsized":"2024-03-31T15:14:11.390384+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_clean_scrub_stamp":"2024-03-31T15:14:07.235046+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:27:01.349324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.18","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:10.891733+0000","last_change":"2024-03-31T15:14:09.278486+0000","last_active":"2024-03-31T15:14:10.891733+0000","last_peered":"2024-03-31T15:14:10.891733+0000","last_clean":"2024-03-31T15:14:10.891733+0000","last_became_active":"2024-03-31T15:14:09.278193+0000","last_became_peered":"2024-03-31T15:14:09.278193+0000","last_unstale":"2024-03-31T15:14:10.891733+0000","last_undegraded":"2024-03-31T15:14:10.891733+0000","last_fullsized":"2024-03-31T15:14:10.891733+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:42:40.870161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-31T15:14:08.270406+0000","last_change":"2024-03-31T15:14:08.266416+0000","last_active":"2024-03-31T15:14:08.270406+0000","last_peered":"2024-03-31T15:14:08.270406+0000","last_clean":"2024-03-31T15:14:08.270406+0000","last_became_active":"2024-03-31T15:14:08.266116+0000","last_became_peered":"2024-03-31T15:14:08.266116+0000","last_unstale":"2024-03-31T15:14:08.270406+0000","last_undegraded":"2024-03-31T15:14:08.270406+0000","last_fullsized":"2024-03-31T15:14:08.270406+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_clean_scrub_stamp":"2024-03-31T15:14:07.235046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:05:08.886029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.19","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-03-31T15:14:08.260207+0000","last_change":"2024-03-31T15:14:08.260207+0000","last_active":"2024-03-31T15:14:08.242094+0000","last_peered":"2024-03-31T15:14:08.242094+0000","last_clean":"2024-03-31T15:14:08.242094+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:14:08.260207+0000","last_undegraded":"2024-03-31T15:14:08.260207+0000","last_fullsized":"2024-03-31T15:14:08.260207+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":14,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:10.891808+0000","last_change":"2024-03-31T15:14:08.276025+0000","last_active":"2024-03-31T15:14:10.891808+0000","last_peered":"2024-03-31T15:14:10.891808+0000","last_clean":"2024-03-31T15:14:10.891808+0000","last_became_active":"2024-03-31T15:14:08.275541+0000","last_became_peered":"2024-03-31T15:14:08.275541+0000","last_unstale":"2024-03-31T15:14:10.891808+0000","last_undegraded":"2024-03-31T15:14:10.891808+0000","last_fullsized":"2024-03-31T15:14:10.891808+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_clean_scrub_stamp":"2024-03-31T15:14:07.235046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:34:33.623018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.16","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-03-31T15:14:08.259804+0000","last_change":"2024-03-31T15:14:08.259804+0000","last_active":"2024-03-31T15:14:08.242094+0000","last_peered":"2024-03-31T15:14:08.242094+0000","last_clean":"2024-03-31T15:14:08.242094+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:14:08.259804+0000","last_undegraded":"2024-03-31T15:14:08.259804+0000","last_fullsized":"2024-03-31T15:14:08.259804+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.17","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-31T15:14:08.278508+0000","last_change":"2024-03-31T15:14:08.275403+0000","last_active":"2024-03-31T15:14:08.278508+0000","last_peered":"2024-03-31T15:14:08.278508+0000","last_clean":"2024-03-31T15:14:08.278508+0000","last_became_active":"2024-03-31T15:14:08.275122+0000","last_became_peered":"2024-03-31T15:14:08.275122+0000","last_unstale":"2024-03-31T15:14:08.278508+0000","last_undegraded":"2024-03-31T15:14:08.278508+0000","last_fullsized":"2024-03-31T15:14:08.278508+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_clean_scrub_stamp":"2024-03-31T15:14:07.235046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T15:33:09.566750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.17","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:10.278750+0000","last_change":"2024-03-31T15:14:09.298453+0000","last_active":"2024-03-31T15:14:10.278750+0000","last_peered":"2024-03-31T15:14:10.278750+0000","last_clean":"2024-03-31T15:14:10.278750+0000","last_became_active":"2024-03-31T15:14:09.298160+0000","last_became_peered":"2024-03-31T15:14:09.298160+0000","last_unstale":"2024-03-31T15:14:10.278750+0000","last_undegraded":"2024-03-31T15:14:10.278750+0000","last_fullsized":"2024-03-31T15:14:10.278750+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:48:01.826468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":14,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:11.391501+0000","last_change":"2024-03-31T15:14:08.272714+0000","last_active":"2024-03-31T15:14:11.391501+0000","last_peered":"2024-03-31T15:14:11.391501+0000","last_clean":"2024-03-31T15:14:11.391501+0000","last_became_active":"2024-03-31T15:14:08.272416+0000","last_became_peered":"2024-03-31T15:14:08.272416+0000","last_unstale":"2024-03-31T15:14:11.391501+0000","last_undegraded":"2024-03-31T15:14:11.391501+0000","last_fullsized":"2024-03-31T15:14:11.391501+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_clean_scrub_stamp":"2024-03-31T15:14:07.235046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:34:33.623018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.14","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:11.391432+0000","last_change":"2024-03-31T15:14:09.281244+0000","last_active":"2024-03-31T15:14:11.391432+0000","last_peered":"2024-03-31T15:14:11.391432+0000","last_clean":"2024-03-31T15:14:11.391432+0000","last_became_active":"2024-03-31T15:14:09.280267+0000","last_became_peered":"2024-03-31T15:14:09.280267+0000","last_unstale":"2024-03-31T15:14:11.391432+0000","last_undegraded":"2024-03-31T15:14:11.391432+0000","last_fullsized":"2024-03-31T15:14:11.391432+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:23:54.993797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.15","version":"108'3","reported_seq":17,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:10.891863+0000","last_change":"2024-03-31T15:14:08.274322+0000","last_active":"2024-03-31T15:14:10.891863+0000","last_peered":"2024-03-31T15:14:10.891863+0000","last_clean":"2024-03-31T15:14:10.891863+0000","last_became_active":"2024-03-31T15:14:08.274025+0000","last_became_peered":"2024-03-31T15:14:08.274025+0000","last_unstale":"2024-03-31T15:14:10.891863+0000","last_undegraded":"2024-03-31T15:14:10.891863+0000","last_fullsized":"2024-03-31T15:14:10.891863+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_clean_scrub_stamp":"2024-03-31T15:14:07.235046+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:54:49.668391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.15","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-03-31T15:14:08.258737+0000","last_change":"2024-03-31T15:14:08.258737+0000","last_active":"2024-03-31T15:14:08.242094+0000","last_peered":"2024-03-31T15:14:08.242094+0000","last_clean":"2024-03-31T15:14:08.242094+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:14:08.258737+0000","last_undegraded":"2024-03-31T15:14:08.258737+0000","last_fullsized":"2024-03-31T15:14:08.258737+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.14","version":"108'6","reported_seq":20,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:10.892187+0000","last_change":"2024-03-31T15:14:08.277536+0000","last_active":"2024-03-31T15:14:10.892187+0000","last_peered":"2024-03-31T15:14:10.892187+0000","last_clean":"2024-03-31T15:14:10.892187+0000","last_became_active":"2024-03-31T15:14:08.277279+0000","last_became_peered":"2024-03-31T15:14:08.277279+0000","last_unstale":"2024-03-31T15:14:10.892187+0000","last_undegraded":"2024-03-31T15:14:10.892187+0000","last_fullsized":"2024-03-31T15:14:10.892187+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_clean_scrub_stamp":"2024-03-31T15:14:07.235046+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:08:03.916239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.12","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:11.390394+0000","last_change":"2024-03-31T15:14:09.275078+0000","last_active":"2024-03-31T15:14:11.390394+0000","last_peered":"2024-03-31T15:14:11.390394+0000","last_clean":"2024-03-31T15:14:11.390394+0000","last_became_active":"2024-03-31T15:14:09.274698+0000","last_became_peered":"2024-03-31T15:14:09.274698+0000","last_unstale":"2024-03-31T15:14:11.390394+0000","last_undegraded":"2024-03-31T15:14:11.390394+0000","last_fullsized":"2024-03-31T15:14:11.390394+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:30:56.093324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.13","version":"108'2","reported_seq":16,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:11.390958+0000","last_change":"2024-03-31T15:14:08.266097+0000","last_active":"2024-03-31T15:14:11.390958+0000","last_peered":"2024-03-31T15:14:11.390958+0000","last_clean":"2024-03-31T15:14:11.390958+0000","last_became_active":"2024-03-31T15:14:08.265812+0000","last_became_peered":"2024-03-31T15:14:08.265812+0000","last_unstale":"2024-03-31T15:14:11.390958+0000","last_undegraded":"2024-03-31T15:14:11.390958+0000","last_fullsized":"2024-03-31T15:14:11.390958+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_clean_scrub_stamp":"2024-03-31T15:14:07.235046+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:49:52.886409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.13","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:11.391329+0000","last_change":"2024-03-31T15:14:09.282087+0000","last_active":"2024-03-31T15:14:11.391329+0000","last_peered":"2024-03-31T15:14:11.391329+0000","last_clean":"2024-03-31T15:14:11.391329+0000","last_became_active":"2024-03-31T15:14:09.281693+0000","last_became_peered":"2024-03-31T15:14:09.281693+0000","last_unstale":"2024-03-31T15:14:11.391329+0000","last_undegraded":"2024-03-31T15:14:11.391329+0000","last_fullsized":"2024-03-31T15:14:11.391329+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:32:06.156300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":14,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:10.278350+0000","last_change":"2024-03-31T15:14:08.272919+0000","last_active":"2024-03-31T15:14:10.278350+0000","last_peered":"2024-03-31T15:14:10.278350+0000","last_clean":"2024-03-31T15:14:10.278350+0000","last_became_active":"2024-03-31T15:14:08.272309+0000","last_became_peered":"2024-03-31T15:14:08.272309+0000","last_unstale":"2024-03-31T15:14:10.278350+0000","last_undegraded":"2024-03-31T15:14:10.278350+0000","last_fullsized":"2024-03-31T15:14:10.278350+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_clean_scrub_stamp":"2024-03-31T15:14:07.235046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:56:10.303847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.11","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:11.390150+0000","last_change":"2024-03-31T15:14:09.274826+0000","last_active":"2024-03-31T15:14:11.390150+0000","last_peered":"2024-03-31T15:14:11.390150+0000","last_clean":"2024-03-31T15:14:11.390150+0000","last_became_active":"2024-03-31T15:14:09.274550+0000","last_became_peered":"2024-03-31T15:14:09.274550+0000","last_unstale":"2024-03-31T15:14:11.390150+0000","last_undegraded":"2024-03-31T15:14:11.390150+0000","last_fullsized":"2024-03-31T15:14:11.390150+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T17:25:28.357433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.10","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-31T15:14:08.278587+0000","last_change":"2024-03-31T15:14:08.276042+0000","last_active":"2024-03-31T15:14:08.278587+0000","last_peered":"2024-03-31T15:14:08.278587+0000","last_clean":"2024-03-31T15:14:08.278587+0000","last_became_active":"2024-03-31T15:14:08.275766+0000","last_became_peered":"2024-03-31T15:14:08.275766+0000","last_unstale":"2024-03-31T15:14:08.278587+0000","last_undegraded":"2024-03-31T15:14:08.278587+0000","last_fullsized":"2024-03-31T15:14:08.278587+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_clean_scrub_stamp":"2024-03-31T15:14:07.235046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:38:29.678222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.e","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-03-31T15:14:08.258499+0000","last_change":"2024-03-31T15:14:08.258499+0000","last_active":"2024-03-31T15:14:08.242094+0000","last_peered":"2024-03-31T15:14:08.242094+0000","last_clean":"2024-03-31T15:14:08.242094+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:14:08.258499+0000","last_undegraded":"2024-03-31T15:14:08.258499+0000","last_fullsized":"2024-03-31T15:14:08.258499+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.f","version":"108'1","reported_seq":15,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:10.892528+0000","last_change":"2024-03-31T15:14:08.273521+0000","last_active":"2024-03-31T15:14:10.892528+0000","last_peered":"2024-03-31T15:14:10.892528+0000","last_clean":"2024-03-31T15:14:10.892528+0000","last_became_active":"2024-03-31T15:14:08.271683+0000","last_became_peered":"2024-03-31T15:14:08.271683+0000","last_unstale":"2024-03-31T15:14:10.892528+0000","last_undegraded":"2024-03-31T15:14:10.892528+0000","last_fullsized":"2024-03-31T15:14:10.892528+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_clean_scrub_stamp":"2024-03-31T15:14:07.235046+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:08:12.284361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.f","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:11.391301+0000","last_change":"2024-03-31T15:14:09.292401+0000","last_active":"2024-03-31T15:14:11.391301+0000","last_peered":"2024-03-31T15:14:11.391301+0000","last_clean":"2024-03-31T15:14:11.391301+0000","last_became_active":"2024-03-31T15:14:09.292126+0000","last_became_peered":"2024-03-31T15:14:09.292126+0000","last_unstale":"2024-03-31T15:14:11.391301+0000","last_undegraded":"2024-03-31T15:14:11.391301+0000","last_fullsized":"2024-03-31T15:14:11.391301+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:24:54.943510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.e","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-31T15:14:08.276874+0000","last_change":"2024-03-31T15:14:08.272765+0000","last_active":"2024-03-31T15:14:08.276874+0000","last_peered":"2024-03-31T15:14:08.276874+0000","last_clean":"2024-03-31T15:14:08.276874+0000","last_became_active":"2024-03-31T15:14:08.272496+0000","last_became_peered":"2024-03-31T15:14:08.272496+0000","last_unstale":"2024-03-31T15:14:08.276874+0000","last_undegraded":"2024-03-31T15:14:08.276874+0000","last_fullsized":"2024-03-31T15:14:08.276874+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_clean_scrub_stamp":"2024-03-31T15:14:07.235046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:30:40.304653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.c","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-03-31T15:14:08.257389+0000","last_change":"2024-03-31T15:14:08.257389+0000","last_active":"2024-03-31T15:14:08.242094+0000","last_peered":"2024-03-31T15:14:08.242094+0000","last_clean":"2024-03-31T15:14:08.242094+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:14:08.257389+0000","last_undegraded":"2024-03-31T15:14:08.257389+0000","last_fullsized":"2024-03-31T15:14:08.257389+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.d","version":"108'1","reported_seq":15,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:10.278500+0000","last_change":"2024-03-31T15:14:08.278494+0000","last_active":"2024-03-31T15:14:10.278500+0000","last_peered":"2024-03-31T15:14:10.278500+0000","last_clean":"2024-03-31T15:14:10.278500+0000","last_became_active":"2024-03-31T15:14:08.278267+0000","last_became_peered":"2024-03-31T15:14:08.278267+0000","last_unstale":"2024-03-31T15:14:10.278500+0000","last_undegraded":"2024-03-31T15:14:10.278500+0000","last_fullsized":"2024-03-31T15:14:10.278500+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_clean_scrub_stamp":"2024-03-31T15:14:07.235046+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:49:25.058336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.d","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-03-31T15:14:08.257107+0000","last_change":"2024-03-31T15:14:08.257107+0000","last_active":"2024-03-31T15:14:08.242094+0000","last_peered":"2024-03-31T15:14:08.242094+0000","last_clean":"2024-03-31T15:14:08.242094+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:14:08.257107+0000","last_undegraded":"2024-03-31T15:14:08.257107+0000","last_fullsized":"2024-03-31T15:14:08.257107+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-31T15:14:08.275180+0000","last_change":"2024-03-31T15:14:08.271960+0000","last_active":"2024-03-31T15:14:08.275180+0000","last_peered":"2024-03-31T15:14:08.275180+0000","last_clean":"2024-03-31T15:14:08.275180+0000","last_became_active":"2024-03-31T15:14:08.271691+0000","last_became_peered":"2024-03-31T15:14:08.271691+0000","last_unstale":"2024-03-31T15:14:08.275180+0000","last_undegraded":"2024-03-31T15:14:08.275180+0000","last_fullsized":"2024-03-31T15:14:08.275180+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_clean_scrub_stamp":"2024-03-31T15:14:07.235046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T17:04:17.133375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.a","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-03-31T15:14:08.257121+0000","last_change":"2024-03-31T15:14:08.257121+0000","last_active":"2024-03-31T15:14:08.242094+0000","last_peered":"2024-03-31T15:14:08.242094+0000","last_clean":"2024-03-31T15:14:08.242094+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:14:08.257121+0000","last_undegraded":"2024-03-31T15:14:08.257121+0000","last_fullsized":"2024-03-31T15:14:08.257121+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.b","version":"108'2","reported_seq":16,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:11.391062+0000","last_change":"2024-03-31T15:14:08.274852+0000","last_active":"2024-03-31T15:14:11.391062+0000","last_peered":"2024-03-31T15:14:11.391062+0000","last_clean":"2024-03-31T15:14:11.391062+0000","last_became_active":"2024-03-31T15:14:08.274568+0000","last_became_peered":"2024-03-31T15:14:08.274568+0000","last_unstale":"2024-03-31T15:14:11.391062+0000","last_undegraded":"2024-03-31T15:14:11.391062+0000","last_fullsized":"2024-03-31T15:14:11.391062+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_clean_scrub_stamp":"2024-03-31T15:14:07.235046+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T19:34:36.202544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.b","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:10.891810+0000","last_change":"2024-03-31T15:14:09.280942+0000","last_active":"2024-03-31T15:14:10.891810+0000","last_peered":"2024-03-31T15:14:10.891810+0000","last_clean":"2024-03-31T15:14:10.891810+0000","last_became_active":"2024-03-31T15:14:09.280678+0000","last_became_peered":"2024-03-31T15:14:09.280678+0000","last_unstale":"2024-03-31T15:14:10.891810+0000","last_undegraded":"2024-03-31T15:14:10.891810+0000","last_fullsized":"2024-03-31T15:14:10.891810+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:00:22.586826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-31T15:14:08.278516+0000","last_change":"2024-03-31T15:14:08.275261+0000","last_active":"2024-03-31T15:14:08.278516+0000","last_peered":"2024-03-31T15:14:08.278516+0000","last_clean":"2024-03-31T15:14:08.278516+0000","last_became_active":"2024-03-31T15:14:08.274961+0000","last_became_peered":"2024-03-31T15:14:08.274961+0000","last_unstale":"2024-03-31T15:14:08.278516+0000","last_undegraded":"2024-03-31T15:14:08.278516+0000","last_fullsized":"2024-03-31T15:14:08.278516+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_clean_scrub_stamp":"2024-03-31T15:14:07.235046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T17:28:54.278793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.8","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-03-31T15:14:08.257584+0000","last_change":"2024-03-31T15:14:08.257584+0000","last_active":"2024-03-31T15:14:08.242094+0000","last_peered":"2024-03-31T15:14:08.242094+0000","last_clean":"2024-03-31T15:14:08.242094+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:14:08.257584+0000","last_undegraded":"2024-03-31T15:14:08.257584+0000","last_fullsized":"2024-03-31T15:14:08.257584+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":14,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:11.390727+0000","last_change":"2024-03-31T15:14:08.275022+0000","last_active":"2024-03-31T15:14:11.390727+0000","last_peered":"2024-03-31T15:14:11.390727+0000","last_clean":"2024-03-31T15:14:11.390727+0000","last_became_active":"2024-03-31T15:14:08.274740+0000","last_became_peered":"2024-03-31T15:14:08.274740+0000","last_unstale":"2024-03-31T15:14:11.390727+0000","last_undegraded":"2024-03-31T15:14:11.390727+0000","last_fullsized":"2024-03-31T15:14:11.390727+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_clean_scrub_stamp":"2024-03-31T15:14:07.235046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:25:35.260804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.9","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:11.390344+0000","last_change":"2024-03-31T15:14:09.291069+0000","last_active":"2024-03-31T15:14:11.390344+0000","last_peered":"2024-03-31T15:14:11.390344+0000","last_clean":"2024-03-31T15:14:11.390344+0000","last_became_active":"2024-03-31T15:14:09.290802+0000","last_became_peered":"2024-03-31T15:14:09.290802+0000","last_unstale":"2024-03-31T15:14:11.390344+0000","last_undegraded":"2024-03-31T15:14:11.390344+0000","last_fullsized":"2024-03-31T15:14:11.390344+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:36:20.417016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":14,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:10.891861+0000","last_change":"2024-03-31T15:14:08.270333+0000","last_active":"2024-03-31T15:14:10.891861+0000","last_peered":"2024-03-31T15:14:10.891861+0000","last_clean":"2024-03-31T15:14:10.891861+0000","last_became_active":"2024-03-31T15:14:08.269866+0000","last_became_peered":"2024-03-31T15:14:08.269866+0000","last_unstale":"2024-03-31T15:14:10.891861+0000","last_undegraded":"2024-03-31T15:14:10.891861+0000","last_fullsized":"2024-03-31T15:14:10.891861+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_clean_scrub_stamp":"2024-03-31T15:14:07.235046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:55:27.079135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.6","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:10.278592+0000","last_change":"2024-03-31T15:14:09.299295+0000","last_active":"2024-03-31T15:14:10.278592+0000","last_peered":"2024-03-31T15:14:10.278592+0000","last_clean":"2024-03-31T15:14:10.278592+0000","last_became_active":"2024-03-31T15:14:09.299016+0000","last_became_peered":"2024-03-31T15:14:09.299016+0000","last_unstale":"2024-03-31T15:14:10.278592+0000","last_undegraded":"2024-03-31T15:14:10.278592+0000","last_fullsized":"2024-03-31T15:14:10.278592+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:34:34.630067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.7","version":"108'2","reported_seq":16,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:11.391777+0000","last_change":"2024-03-31T15:14:08.276158+0000","last_active":"2024-03-31T15:14:11.391777+0000","last_peered":"2024-03-31T15:14:11.391777+0000","last_clean":"2024-03-31T15:14:11.391777+0000","last_became_active":"2024-03-31T15:14:08.275856+0000","last_became_peered":"2024-03-31T15:14:08.275856+0000","last_unstale":"2024-03-31T15:14:11.391777+0000","last_undegraded":"2024-03-31T15:14:11.391777+0000","last_fullsized":"2024-03-31T15:14:11.391777+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_clean_scrub_stamp":"2024-03-31T15:14:07.235046+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:48:00.819420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.7","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:10.892615+0000","last_change":"2024-03-31T15:14:09.277522+0000","last_active":"2024-03-31T15:14:10.892615+0000","last_peered":"2024-03-31T15:14:10.892615+0000","last_clean":"2024-03-31T15:14:10.892615+0000","last_became_active":"2024-03-31T15:14:09.276889+0000","last_became_peered":"2024-03-31T15:14:09.276889+0000","last_unstale":"2024-03-31T15:14:10.892615+0000","last_undegraded":"2024-03-31T15:14:10.892615+0000","last_fullsized":"2024-03-31T15:14:10.892615+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:45:51.393113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.6","version":"109'1","reported_seq":15,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:10.278704+0000","last_change":"2024-03-31T15:14:08.265332+0000","last_active":"2024-03-31T15:14:10.278704+0000","last_peered":"2024-03-31T15:14:10.278704+0000","last_clean":"2024-03-31T15:14:10.278704+0000","last_became_active":"2024-03-31T15:14:08.264838+0000","last_became_peered":"2024-03-31T15:14:08.264838+0000","last_unstale":"2024-03-31T15:14:10.278704+0000","last_undegraded":"2024-03-31T15:14:10.278704+0000","last_fullsized":"2024-03-31T15:14:10.278704+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_clean_scrub_stamp":"2024-03-31T15:14:07.235046+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:32:53.111430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.4","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:10.892152+0000","last_change":"2024-03-31T15:14:09.279412+0000","last_active":"2024-03-31T15:14:10.892152+0000","last_peered":"2024-03-31T15:14:10.892152+0000","last_clean":"2024-03-31T15:14:10.892152+0000","last_became_active":"2024-03-31T15:14:09.278696+0000","last_became_peered":"2024-03-31T15:14:09.278696+0000","last_unstale":"2024-03-31T15:14:10.892152+0000","last_undegraded":"2024-03-31T15:14:10.892152+0000","last_fullsized":"2024-03-31T15:14:10.892152+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:57:34.106053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":14,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:10.891892+0000","last_change":"2024-03-31T15:14:08.270001+0000","last_active":"2024-03-31T15:14:10.891892+0000","last_peered":"2024-03-31T15:14:10.891892+0000","last_clean":"2024-03-31T15:14:10.891892+0000","last_became_active":"2024-03-31T15:14:08.268897+0000","last_became_peered":"2024-03-31T15:14:08.268897+0000","last_unstale":"2024-03-31T15:14:10.891892+0000","last_undegraded":"2024-03-31T15:14:10.891892+0000","last_fullsized":"2024-03-31T15:14:10.891892+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_clean_scrub_stamp":"2024-03-31T15:14:07.235046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T19:21:04.678038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.5","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:10.278439+0000","last_change":"2024-03-31T15:14:09.292175+0000","last_active":"2024-03-31T15:14:10.278439+0000","last_peered":"2024-03-31T15:14:10.278439+0000","last_clean":"2024-03-31T15:14:10.278439+0000","last_became_active":"2024-03-31T15:14:09.291875+0000","last_became_peered":"2024-03-31T15:14:09.291875+0000","last_unstale":"2024-03-31T15:14:10.278439+0000","last_undegraded":"2024-03-31T15:14:10.278439+0000","last_fullsized":"2024-03-31T15:14:10.278439+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:27:02.356373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":14,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:11.391479+0000","last_change":"2024-03-31T15:14:08.268133+0000","last_active":"2024-03-31T15:14:11.391479+0000","last_peered":"2024-03-31T15:14:11.391479+0000","last_clean":"2024-03-31T15:14:11.391479+0000","last_became_active":"2024-03-31T15:14:08.267768+0000","last_became_peered":"2024-03-31T15:14:08.267768+0000","last_unstale":"2024-03-31T15:14:11.391479+0000","last_undegraded":"2024-03-31T15:14:11.391479+0000","last_fullsized":"2024-03-31T15:14:11.391479+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_clean_scrub_stamp":"2024-03-31T15:14:07.235046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:55:27.079135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.2","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-03-31T15:14:08.254000+0000","last_change":"2024-03-31T15:14:08.254000+0000","last_active":"2024-03-31T15:14:08.242094+0000","last_peered":"2024-03-31T15:14:08.242094+0000","last_clean":"2024-03-31T15:14:08.242094+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:14:08.254000+0000","last_undegraded":"2024-03-31T15:14:08.254000+0000","last_fullsized":"2024-03-31T15:14:08.254000+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-31T15:14:08.272766+0000","last_change":"2024-03-31T15:14:08.268809+0000","last_active":"2024-03-31T15:14:08.272766+0000","last_peered":"2024-03-31T15:14:08.272766+0000","last_clean":"2024-03-31T15:14:08.272766+0000","last_became_active":"2024-03-31T15:14:08.268473+0000","last_became_peered":"2024-03-31T15:14:08.268473+0000","last_unstale":"2024-03-31T15:14:08.272766+0000","last_undegraded":"2024-03-31T15:14:08.272766+0000","last_fullsized":"2024-03-31T15:14:08.272766+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_clean_scrub_stamp":"2024-03-31T15:14:07.235046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:09:33.273967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.3","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-03-31T15:14:08.255261+0000","last_change":"2024-03-31T15:14:08.255261+0000","last_active":"2024-03-31T15:14:08.242094+0000","last_peered":"2024-03-31T15:14:08.242094+0000","last_clean":"2024-03-31T15:14:08.242094+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:14:08.255261+0000","last_undegraded":"2024-03-31T15:14:08.255261+0000","last_fullsized":"2024-03-31T15:14:08.255261+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-31T15:14:08.272985+0000","last_change":"2024-03-31T15:14:08.269651+0000","last_active":"2024-03-31T15:14:08.272985+0000","last_peered":"2024-03-31T15:14:08.272985+0000","last_clean":"2024-03-31T15:14:08.272985+0000","last_became_active":"2024-03-31T15:14:08.269372+0000","last_became_peered":"2024-03-31T15:14:08.269372+0000","last_unstale":"2024-03-31T15:14:08.272985+0000","last_undegraded":"2024-03-31T15:14:08.272985+0000","last_fullsized":"2024-03-31T15:14:08.272985+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_clean_scrub_stamp":"2024-03-31T15:14:07.235046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:15:08.190993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.0","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-03-31T15:14:08.257010+0000","last_change":"2024-03-31T15:14:08.257010+0000","last_active":"2024-03-31T15:14:08.242094+0000","last_peered":"2024-03-31T15:14:08.242094+0000","last_clean":"2024-03-31T15:14:08.242094+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:14:08.257010+0000","last_undegraded":"2024-03-31T15:14:08.257010+0000","last_fullsized":"2024-03-31T15:14:08.257010+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-31T15:14:08.277067+0000","last_change":"2024-03-31T15:14:08.271032+0000","last_active":"2024-03-31T15:14:08.277067+0000","last_peered":"2024-03-31T15:14:08.277067+0000","last_clean":"2024-03-31T15:14:08.277067+0000","last_became_active":"2024-03-31T15:14:08.270781+0000","last_became_peered":"2024-03-31T15:14:08.270781+0000","last_unstale":"2024-03-31T15:14:08.277067+0000","last_undegraded":"2024-03-31T15:14:08.277067+0000","last_fullsized":"2024-03-31T15:14:08.277067+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_clean_scrub_stamp":"2024-03-31T15:14:07.235046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:59:46.849346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:10.892154+0000","last_change":"2024-03-31T15:14:09.288490+0000","last_active":"2024-03-31T15:14:10.892154+0000","last_peered":"2024-03-31T15:14:10.892154+0000","last_clean":"2024-03-31T15:14:10.892154+0000","last_became_active":"2024-03-31T15:14:09.284507+0000","last_became_peered":"2024-03-31T15:14:09.284507+0000","last_unstale":"2024-03-31T15:14:10.892154+0000","last_undegraded":"2024-03-31T15:14:10.892154+0000","last_fullsized":"2024-03-31T15:14:10.892154+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:09:03.491873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":14,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:10.891910+0000","last_change":"2024-03-31T15:14:08.270270+0000","last_active":"2024-03-31T15:14:10.891910+0000","last_peered":"2024-03-31T15:14:10.891910+0000","last_clean":"2024-03-31T15:14:10.891910+0000","last_became_active":"2024-03-31T15:14:08.269482+0000","last_became_peered":"2024-03-31T15:14:08.269482+0000","last_unstale":"2024-03-31T15:14:10.891910+0000","last_undegraded":"2024-03-31T15:14:10.891910+0000","last_fullsized":"2024-03-31T15:14:10.891910+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_clean_scrub_stamp":"2024-03-31T15:14:07.235046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:48:00.819420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.10","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:11.390177+0000","last_change":"2024-03-31T15:14:09.277917+0000","last_active":"2024-03-31T15:14:11.390177+0000","last_peered":"2024-03-31T15:14:11.390177+0000","last_clean":"2024-03-31T15:14:11.390177+0000","last_became_active":"2024-03-31T15:14:09.277651+0000","last_became_peered":"2024-03-31T15:14:09.277651+0000","last_unstale":"2024-03-31T15:14:11.390177+0000","last_undegraded":"2024-03-31T15:14:11.390177+0000","last_fullsized":"2024-03-31T15:14:11.390177+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T16:10:45.541190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-31T15:14:08.275222+0000","last_change":"2024-03-31T15:14:08.271941+0000","last_active":"2024-03-31T15:14:08.275222+0000","last_peered":"2024-03-31T15:14:08.275222+0000","last_clean":"2024-03-31T15:14:08.275222+0000","last_became_active":"2024-03-31T15:14:08.271589+0000","last_became_peered":"2024-03-31T15:14:08.271589+0000","last_unstale":"2024-03-31T15:14:08.275222+0000","last_undegraded":"2024-03-31T15:14:08.275222+0000","last_fullsized":"2024-03-31T15:14:08.275222+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_clean_scrub_stamp":"2024-03-31T15:14:07.235046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:41:56.592702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":331,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-31T15:14:08.262750+0000","last_change":"2024-03-31T15:03:24.329890+0000","last_active":"2024-03-31T15:14:08.262750+0000","last_peered":"2024-03-31T15:14:08.262750+0000","last_clean":"2024-03-31T15:14:08.262750+0000","last_became_active":"2024-03-31T15:03:24.328882+0000","last_became_peered":"2024-03-31T15:03:24.328882+0000","last_unstale":"2024-03-31T15:14:08.262750+0000","last_undegraded":"2024-03-31T15:14:08.262750+0000","last_fullsized":"2024-03-31T15:14:08.262750+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:22.885535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:22.885535+0000","last_clean_scrub_stamp":"2024-03-31T15:03:22.885535+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:18:29.004296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":17,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":16,"num_pg":32,"stat_sum":{"num_bytes":1828,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":19,"ondisk_log_size":19,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509574,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33040,"kb_used_data":1860,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338800,"statfs":{"total":96636764160,"available":96602931200,"internally_reserved":0,"allocated":1904640,"data_stored":1155091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509573,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32476,"kb_used_data":1292,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339364,"statfs":{"total":96636764160,"available":96603508736,"internally_reserved":0,"allocated":1323008,"data_stored":570998,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":15,"seq":64424509572,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32460,"kb_used_data":1280,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339380,"statfs":{"total":96636764160,"available":96603525120,"internally_reserved":0,"allocated":1310720,"data_stored":564723,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509571,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36632,"kb_used_data":1292,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335208,"statfs":{"total":96636764160,"available":96599252992,"internally_reserved":0,"allocated":1323008,"data_stored":570998,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542278,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32508,"kb_used_data":1320,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339332,"statfs":{"total":96636764160,"available":96603475968,"internally_reserved":0,"allocated":1351680,"data_stored":584143,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542278,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32512,"kb_used_data":1332,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339328,"statfs":{"total":96636764160,"available":96603471872,"internally_reserved":0,"allocated":1363968,"data_stored":585755,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542278,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32504,"kb_used_data":1324,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339336,"statfs":{"total":96636764160,"available":96603480064,"internally_reserved":0,"allocated":1355776,"data_stored":584197,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542278,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33100,"kb_used_data":1916,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338740,"statfs":{"total":96636764160,"available":96602869760,"internally_reserved":0,"allocated":1961984,"data_stored":1176165,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-31T15:14:13.768 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-31T15:14:13.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:13.954+0000 7faabf800640 1 Processor -- start 2024-03-31T15:14:13.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:13.954+0000 7faabf800640 1 -- start start 2024-03-31T15:14:13.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:13.958+0000 7faabf800640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7faab8147e70 0x7faab806b620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:13.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:13.958+0000 7faabf800640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7faab80736e0 0x7faab815eb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:13.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:13.958+0000 7faabf800640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faab81621b0 con 0x7faab80d0c50 2024-03-31T15:14:13.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:13.958+0000 7faabf800640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7faab8162490 con 0x7faab8147e70 2024-03-31T15:14:13.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:13.958+0000 7faabf800640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7faab8162770 con 0x7faab80736e0 2024-03-31T15:14:13.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:13.958+0000 7faabce00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7faab8147e70 0x7faab806b620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:13.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:13.958+0000 7faabce00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7faab8147e70 0x7faab806b620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50142/0 (socket says 172.21.15.16:50142) 2024-03-31T15:14:13.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:13.958+0000 7faabce00640 1 -- 172.21.15.16:0/764438076 learned_addr learned my addr 172.21.15.16:0/764438076 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:14:13.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:13.958+0000 7faabd800640 1 --2- 172.21.15.16:0/764438076 >> v2:172.21.15.89:3300/0 conn(0x7faab80736e0 0x7faab815eb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:13.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:13.958+0000 7faabce00640 1 -- 172.21.15.16:0/764438076 >> v2:172.21.15.89:3300/0 conn(0x7faab80736e0 msgr2=0x7faab815eb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:14:13.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:13.958+0000 7faabce00640 1 --2- 172.21.15.16:0/764438076 >> v2:172.21.15.89:3300/0 conn(0x7faab80736e0 0x7faab815eb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:13.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:13.958+0000 7faabce00640 1 -- 172.21.15.16:0/764438076 >> v1:172.21.15.89:6789/0 conn(0x7faab80d0c50 legacy=0x7faab8072fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:13.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:13.958+0000 7faabce00640 1 -- 172.21.15.16:0/764438076 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faab8162a50 con 0x7faab8147e70 2024-03-31T15:14:13.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:13.962+0000 7faabce00640 1 --2- 172.21.15.16:0/764438076 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faab8147e70 0x7faab806b620 secure :-1 s=READY pgs=1656 cs=0 l=1 rev1=1 crypto rx=0x7faab806e280 tx=0x7faaa801c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=542314abc5b95a9f server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:13.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:13.962+0000 7faab7e00640 1 -- 172.21.15.16:0/764438076 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faaa80b1070 con 0x7faab8147e70 2024-03-31T15:14:13.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:13.962+0000 7faab7e00640 1 -- 172.21.15.16:0/764438076 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faaa80ac600 con 0x7faab8147e70 2024-03-31T15:14:13.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:13.962+0000 7faab7e00640 1 -- 172.21.15.16:0/764438076 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faaa80ac9d0 con 0x7faab8147e70 2024-03-31T15:14:13.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:13.962+0000 7faabf800640 1 -- 172.21.15.16:0/764438076 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faab8147e70 msgr2=0x7faab806b620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:13.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:13.962+0000 7faabf800640 1 --2- 172.21.15.16:0/764438076 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faab8147e70 0x7faab806b620 secure :-1 s=READY pgs=1656 cs=0 l=1 rev1=1 crypto rx=0x7faab806e280 tx=0x7faaa801c520 comp rx=0 tx=0).stop 2024-03-31T15:14:13.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:13.962+0000 7faabf800640 1 -- 172.21.15.16:0/764438076 shutdown_connections 2024-03-31T15:14:13.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:13.962+0000 7faabf800640 1 --2- 172.21.15.16:0/764438076 >> v2:172.21.15.89:3300/0 conn(0x7faab80736e0 0x7faab815eb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:13.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:13.962+0000 7faabf800640 1 --2- 172.21.15.16:0/764438076 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faab8147e70 0x7faab806b620 unknown :-1 s=CLOSED pgs=1656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:13.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:13.962+0000 7faabf800640 1 -- 172.21.15.16:0/764438076 >> 172.21.15.16:0/764438076 conn(0x7faab80d1060 msgr2=0x7faab80d1470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:14:13.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:13.962+0000 7faabf800640 1 -- 172.21.15.16:0/764438076 shutdown_connections 2024-03-31T15:14:13.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:13.962+0000 7faabf800640 1 -- 172.21.15.16:0/764438076 wait complete. 2024-03-31T15:14:13.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:13.966+0000 7faabf800640 1 Processor -- start 2024-03-31T15:14:13.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:13.966+0000 7faabf800640 1 -- start start 2024-03-31T15:14:13.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:13.966+0000 7faabf800640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7faab80736e0 0x7faab815d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:13.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:13.966+0000 7faabf800640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faab80d0c50 0x7faab80aef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:13.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:13.966+0000 7faabf800640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faab8147e70 0x7faab813f9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:13.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:13.966+0000 7faabf800640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7faab815f870 con 0x7faab80d0c50 2024-03-31T15:14:13.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:13.966+0000 7faabf800640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7faab815f9f0 con 0x7faab8147e70 2024-03-31T15:14:13.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:13.966+0000 7faabf800640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7faab816b4b0 con 0x7faab80736e0 2024-03-31T15:14:13.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:13.966+0000 7faaafe00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faab80d0c50 0x7faab80aef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:13.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:13.966+0000 7faaafe00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faab80d0c50 0x7faab80aef60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50146/0 (socket says 172.21.15.16:50146) 2024-03-31T15:14:13.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:13.966+0000 7faaafe00640 1 -- 172.21.15.16:0/1560341080 learned_addr learned my addr 172.21.15.16:0/1560341080 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:14:13.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:13.966+0000 7faabce00640 1 --2- 172.21.15.16:0/1560341080 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7faab80736e0 0x7faab815d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:13.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:13.966+0000 7faabd800640 1 --2- 172.21.15.16:0/1560341080 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faab8147e70 0x7faab813f9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:13.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:13.966+0000 7faaafe00640 1 -- 172.21.15.16:0/1560341080 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7faab80736e0 msgr2=0x7faab815d420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:13.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:13.966+0000 7faaafe00640 1 --2- 172.21.15.16:0/1560341080 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7faab80736e0 0x7faab815d420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:13.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:13.966+0000 7faaafe00640 1 -- 172.21.15.16:0/1560341080 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faab8147e70 msgr2=0x7faab813f9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:13.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:13.966+0000 7faaafe00640 1 --2- 172.21.15.16:0/1560341080 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faab8147e70 0x7faab813f9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:13.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:13.966+0000 7faaafe00640 1 -- 172.21.15.16:0/1560341080 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faab8143b00 con 0x7faab80d0c50 2024-03-31T15:14:13.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:13.966+0000 7faaafe00640 1 --2- 172.21.15.16:0/1560341080 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faab80d0c50 0x7faab80aef60 secure :-1 s=READY pgs=1657 cs=0 l=1 rev1=1 crypto rx=0x7faaa000c4d0 tx=0x7faaa000c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:13.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:13.966+0000 7faab5600640 1 -- 172.21.15.16:0/1560341080 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faaa00052d0 con 0x7faab80d0c50 2024-03-31T15:14:13.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:13.966+0000 7faab5600640 1 -- 172.21.15.16:0/1560341080 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faaa0005470 con 0x7faab80d0c50 2024-03-31T15:14:13.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:13.966+0000 7faab5600640 1 -- 172.21.15.16:0/1560341080 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faaa0012830 con 0x7faab80d0c50 2024-03-31T15:14:13.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:13.966+0000 7faabf800640 1 -- 172.21.15.16:0/1560341080 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faab8143de0 con 0x7faab80d0c50 2024-03-31T15:14:13.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:13.966+0000 7faabf800640 1 -- 172.21.15.16:0/1560341080 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faab815dec0 con 0x7faab80d0c50 2024-03-31T15:14:13.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:13.970+0000 7faabf800640 1 -- 172.21.15.16:0/1560341080 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faab8069950 con 0x7faab80d0c50 2024-03-31T15:14:13.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:13.970+0000 7faab5600640 1 -- 172.21.15.16:0/1560341080 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7faaa00129d0 con 0x7faab80d0c50 2024-03-31T15:14:13.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:13.970+0000 7faab5600640 1 --2- 172.21.15.16:0/1560341080 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7faa94068380 0x7faa9406a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:13.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:13.970+0000 7faab5600640 1 -- 172.21.15.16:0/1560341080 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 5709+0+0 (secure 0 0 0) 0x7faaa0086470 con 0x7faab80d0c50 2024-03-31T15:14:13.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:13.970+0000 7faabce00640 1 --2- 172.21.15.16:0/1560341080 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7faa94068380 0x7faa9406a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:13.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:13.970+0000 7faabce00640 1 --2- 172.21.15.16:0/1560341080 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7faa94068380 0x7faa9406a840 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7faaa80ae000 tx=0x7faaa80a7000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:13.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:13.974+0000 7faab5600640 1 -- 172.21.15.16:0/1560341080 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7faaa00563d0 con 0x7faab80d0c50 2024-03-31T15:14:14.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:14.134+0000 7faabf800640 1 -- 172.21.15.16:0/1560341080 --> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7faab81438b0 con 0x7faa94068380 2024-03-31T15:14:14.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:14.142+0000 7faab5600640 1 -- 172.21.15.16:0/1560341080 <== mgr.4100 v2:172.21.15.89:6832/1549407273 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+250056 (secure 0 0 0) 0x7faab81438b0 con 0x7faa94068380 2024-03-31T15:14:14.138 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:14:14.139 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-03-31T15:14:14.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:14.146+0000 7faabf800640 1 -- 172.21.15.16:0/1560341080 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7faa94068380 msgr2=0x7faa9406a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:14.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:14.146+0000 7faabf800640 1 --2- 172.21.15.16:0/1560341080 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7faa94068380 0x7faa9406a840 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7faaa80ae000 tx=0x7faaa80a7000 comp rx=0 tx=0).stop 2024-03-31T15:14:14.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:14.146+0000 7faabf800640 1 -- 172.21.15.16:0/1560341080 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faab80d0c50 msgr2=0x7faab80aef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:14.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:14.146+0000 7faabf800640 1 --2- 172.21.15.16:0/1560341080 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faab80d0c50 0x7faab80aef60 secure :-1 s=READY pgs=1657 cs=0 l=1 rev1=1 crypto rx=0x7faaa000c4d0 tx=0x7faaa000c990 comp rx=0 tx=0).stop 2024-03-31T15:14:14.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:14.146+0000 7faabf800640 1 -- 172.21.15.16:0/1560341080 shutdown_connections 2024-03-31T15:14:14.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:14.146+0000 7faabf800640 1 --2- 172.21.15.16:0/1560341080 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7faa94068380 0x7faa9406a840 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:14.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:14.146+0000 7faabf800640 1 --2- 172.21.15.16:0/1560341080 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faab8147e70 0x7faab813f9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:14.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:14.146+0000 7faabf800640 1 --2- 172.21.15.16:0/1560341080 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faab80d0c50 0x7faab80aef60 unknown :-1 s=CLOSED pgs=1657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:14.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:14.146+0000 7faabf800640 1 --2- 172.21.15.16:0/1560341080 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7faab80736e0 0x7faab815d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:14.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:14.146+0000 7faabf800640 1 -- 172.21.15.16:0/1560341080 >> 172.21.15.16:0/1560341080 conn(0x7faab80d1060 msgr2=0x7faab8161280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:14:14.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:14.146+0000 7faabf800640 1 -- 172.21.15.16:0/1560341080 shutdown_connections 2024-03-31T15:14:14.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:14.146+0000 7faabf800640 1 -- 172.21.15.16:0/1560341080 wait complete. 2024-03-31T15:14:14.159 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":427,"stamp":"2024-03-31T15:14:12.693312+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592196,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":179,"num_write_kb":2429,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":136,"ondisk_log_size":136,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":104,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":265232,"kb_used_data":11616,"kb_used_omap":12,"kb_used_meta":253491,"kb_avail":754709488,"statfs":{"total":773094113280,"available":772822515712,"internally_reserved":0,"allocated":11894784,"data_stored":5792070,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":259575344},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":32,"apply_latency_ms":32,"commit_latency_ns":32000000,"apply_latency_ns":32000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1828,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.446790"},"pg_stats":[{"pgid":"17.3f","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:10.892463+0000","last_change":"2024-03-31T15:14:09.277746+0000","last_active":"2024-03-31T15:14:10.892463+0000","last_peered":"2024-03-31T15:14:10.892463+0000","last_clean":"2024-03-31T15:14:10.892463+0000","last_became_active":"2024-03-31T15:14:09.277207+0000","last_became_peered":"2024-03-31T15:14:09.277207+0000","last_unstale":"2024-03-31T15:14:10.892463+0000","last_undegraded":"2024-03-31T15:14:10.892463+0000","last_fullsized":"2024-03-31T15:14:10.892463+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T17:42:57.076247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3e","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-03-31T15:14:08.253600+0000","last_change":"2024-03-31T15:14:08.253600+0000","last_active":"2024-03-31T15:14:08.242094+0000","last_peered":"2024-03-31T15:14:08.242094+0000","last_clean":"2024-03-31T15:14:08.242094+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:14:08.253600+0000","last_undegraded":"2024-03-31T15:14:08.253600+0000","last_fullsized":"2024-03-31T15:14:08.253600+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.3d","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-03-31T15:14:08.258560+0000","last_change":"2024-03-31T15:14:08.258560+0000","last_active":"2024-03-31T15:14:08.242094+0000","last_peered":"2024-03-31T15:14:08.242094+0000","last_clean":"2024-03-31T15:14:08.242094+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:14:08.258560+0000","last_undegraded":"2024-03-31T15:14:08.258560+0000","last_fullsized":"2024-03-31T15:14:08.258560+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.3c","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:10.892010+0000","last_change":"2024-03-31T15:14:09.276706+0000","last_active":"2024-03-31T15:14:10.892010+0000","last_peered":"2024-03-31T15:14:10.892010+0000","last_clean":"2024-03-31T15:14:10.892010+0000","last_became_active":"2024-03-31T15:14:09.275545+0000","last_became_peered":"2024-03-31T15:14:09.275545+0000","last_unstale":"2024-03-31T15:14:10.892010+0000","last_undegraded":"2024-03-31T15:14:10.892010+0000","last_fullsized":"2024-03-31T15:14:10.892010+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T19:37:12.397888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3b","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-03-31T15:14:08.255591+0000","last_change":"2024-03-31T15:14:08.255591+0000","last_active":"2024-03-31T15:14:08.242094+0000","last_peered":"2024-03-31T15:14:08.242094+0000","last_clean":"2024-03-31T15:14:08.242094+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:14:08.255591+0000","last_undegraded":"2024-03-31T15:14:08.255591+0000","last_fullsized":"2024-03-31T15:14:08.255591+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.3a","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:11.390994+0000","last_change":"2024-03-31T15:14:09.283260+0000","last_active":"2024-03-31T15:14:11.390994+0000","last_peered":"2024-03-31T15:14:11.390994+0000","last_clean":"2024-03-31T15:14:11.390994+0000","last_became_active":"2024-03-31T15:14:09.282995+0000","last_became_peered":"2024-03-31T15:14:09.282995+0000","last_unstale":"2024-03-31T15:14:11.390994+0000","last_undegraded":"2024-03-31T15:14:11.390994+0000","last_fullsized":"2024-03-31T15:14:11.390994+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:27:30.858064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.39","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-03-31T15:14:08.259762+0000","last_change":"2024-03-31T15:14:08.259762+0000","last_active":"2024-03-31T15:14:08.242094+0000","last_peered":"2024-03-31T15:14:08.242094+0000","last_clean":"2024-03-31T15:14:08.242094+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:14:08.259762+0000","last_undegraded":"2024-03-31T15:14:08.259762+0000","last_fullsized":"2024-03-31T15:14:08.259762+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.38","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-03-31T15:14:08.258038+0000","last_change":"2024-03-31T15:14:08.258038+0000","last_active":"2024-03-31T15:14:08.242094+0000","last_peered":"2024-03-31T15:14:08.242094+0000","last_clean":"2024-03-31T15:14:08.242094+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:14:08.258038+0000","last_undegraded":"2024-03-31T15:14:08.258038+0000","last_fullsized":"2024-03-31T15:14:08.258038+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.37","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:10.892467+0000","last_change":"2024-03-31T15:14:09.284680+0000","last_active":"2024-03-31T15:14:10.892467+0000","last_peered":"2024-03-31T15:14:10.892467+0000","last_clean":"2024-03-31T15:14:10.892467+0000","last_became_active":"2024-03-31T15:14:09.282533+0000","last_became_peered":"2024-03-31T15:14:09.282533+0000","last_unstale":"2024-03-31T15:14:10.892467+0000","last_undegraded":"2024-03-31T15:14:10.892467+0000","last_fullsized":"2024-03-31T15:14:10.892467+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:04:35.760087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.36","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:10.892246+0000","last_change":"2024-03-31T15:14:09.290368+0000","last_active":"2024-03-31T15:14:10.892246+0000","last_peered":"2024-03-31T15:14:10.892246+0000","last_clean":"2024-03-31T15:14:10.892246+0000","last_became_active":"2024-03-31T15:14:09.290077+0000","last_became_peered":"2024-03-31T15:14:09.290077+0000","last_unstale":"2024-03-31T15:14:10.892246+0000","last_undegraded":"2024-03-31T15:14:10.892246+0000","last_fullsized":"2024-03-31T15:14:10.892246+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:23:53.326762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.35","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-03-31T15:14:08.253324+0000","last_change":"2024-03-31T15:14:08.253324+0000","last_active":"2024-03-31T15:14:08.242094+0000","last_peered":"2024-03-31T15:14:08.242094+0000","last_clean":"2024-03-31T15:14:08.242094+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:14:08.253324+0000","last_undegraded":"2024-03-31T15:14:08.253324+0000","last_fullsized":"2024-03-31T15:14:08.253324+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.34","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:10.892056+0000","last_change":"2024-03-31T15:14:09.276847+0000","last_active":"2024-03-31T15:14:10.892056+0000","last_peered":"2024-03-31T15:14:10.892056+0000","last_clean":"2024-03-31T15:14:10.892056+0000","last_became_active":"2024-03-31T15:14:09.275249+0000","last_became_peered":"2024-03-31T15:14:09.275249+0000","last_unstale":"2024-03-31T15:14:10.892056+0000","last_undegraded":"2024-03-31T15:14:10.892056+0000","last_fullsized":"2024-03-31T15:14:10.892056+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T16:56:09.377594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.33","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-03-31T15:14:08.258326+0000","last_change":"2024-03-31T15:14:08.258326+0000","last_active":"2024-03-31T15:14:08.242094+0000","last_peered":"2024-03-31T15:14:08.242094+0000","last_clean":"2024-03-31T15:14:08.242094+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:14:08.258326+0000","last_undegraded":"2024-03-31T15:14:08.258326+0000","last_fullsized":"2024-03-31T15:14:08.258326+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.32","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-03-31T15:14:08.257741+0000","last_change":"2024-03-31T15:14:08.257741+0000","last_active":"2024-03-31T15:14:08.242094+0000","last_peered":"2024-03-31T15:14:08.242094+0000","last_clean":"2024-03-31T15:14:08.242094+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:14:08.257741+0000","last_undegraded":"2024-03-31T15:14:08.257741+0000","last_fullsized":"2024-03-31T15:14:08.257741+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.31","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:11.390625+0000","last_change":"2024-03-31T15:14:09.285784+0000","last_active":"2024-03-31T15:14:11.390625+0000","last_peered":"2024-03-31T15:14:11.390625+0000","last_clean":"2024-03-31T15:14:11.390625+0000","last_became_active":"2024-03-31T15:14:09.285374+0000","last_became_peered":"2024-03-31T15:14:09.285374+0000","last_unstale":"2024-03-31T15:14:11.390625+0000","last_undegraded":"2024-03-31T15:14:11.390625+0000","last_fullsized":"2024-03-31T15:14:11.390625+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:34:11.338914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.30","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:10.891716+0000","last_change":"2024-03-31T15:14:09.287510+0000","last_active":"2024-03-31T15:14:10.891716+0000","last_peered":"2024-03-31T15:14:10.891716+0000","last_clean":"2024-03-31T15:14:10.891716+0000","last_became_active":"2024-03-31T15:14:09.287238+0000","last_became_peered":"2024-03-31T15:14:09.287238+0000","last_unstale":"2024-03-31T15:14:10.891716+0000","last_undegraded":"2024-03-31T15:14:10.891716+0000","last_fullsized":"2024-03-31T15:14:10.891716+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:14:09.225080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.2f","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-03-31T15:14:08.254889+0000","last_change":"2024-03-31T15:14:08.254889+0000","last_active":"2024-03-31T15:14:08.242094+0000","last_peered":"2024-03-31T15:14:08.242094+0000","last_clean":"2024-03-31T15:14:08.242094+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:14:08.254889+0000","last_undegraded":"2024-03-31T15:14:08.254889+0000","last_fullsized":"2024-03-31T15:14:08.254889+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2e","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:10.279438+0000","last_change":"2024-03-31T15:14:09.276215+0000","last_active":"2024-03-31T15:14:10.279438+0000","last_peered":"2024-03-31T15:14:10.279438+0000","last_clean":"2024-03-31T15:14:10.279438+0000","last_became_active":"2024-03-31T15:14:09.275956+0000","last_became_peered":"2024-03-31T15:14:09.275956+0000","last_unstale":"2024-03-31T15:14:10.279438+0000","last_undegraded":"2024-03-31T15:14:10.279438+0000","last_fullsized":"2024-03-31T15:14:10.279438+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:54:50.675440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.2d","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-03-31T15:14:08.258151+0000","last_change":"2024-03-31T15:14:08.258151+0000","last_active":"2024-03-31T15:14:08.242094+0000","last_peered":"2024-03-31T15:14:08.242094+0000","last_clean":"2024-03-31T15:14:08.242094+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:14:08.258151+0000","last_undegraded":"2024-03-31T15:14:08.258151+0000","last_fullsized":"2024-03-31T15:14:08.258151+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.2c","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-03-31T15:14:08.253580+0000","last_change":"2024-03-31T15:14:08.253580+0000","last_active":"2024-03-31T15:14:08.242094+0000","last_peered":"2024-03-31T15:14:08.242094+0000","last_clean":"2024-03-31T15:14:08.242094+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:14:08.253580+0000","last_undegraded":"2024-03-31T15:14:08.253580+0000","last_fullsized":"2024-03-31T15:14:08.253580+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2b","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-03-31T15:14:08.253223+0000","last_change":"2024-03-31T15:14:08.253223+0000","last_active":"2024-03-31T15:14:08.242094+0000","last_peered":"2024-03-31T15:14:08.242094+0000","last_clean":"2024-03-31T15:14:08.242094+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:14:08.253223+0000","last_undegraded":"2024-03-31T15:14:08.253223+0000","last_fullsized":"2024-03-31T15:14:08.253223+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2a","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:11.390332+0000","last_change":"2024-03-31T15:14:09.275037+0000","last_active":"2024-03-31T15:14:11.390332+0000","last_peered":"2024-03-31T15:14:11.390332+0000","last_clean":"2024-03-31T15:14:11.390332+0000","last_became_active":"2024-03-31T15:14:09.274177+0000","last_became_peered":"2024-03-31T15:14:09.274177+0000","last_unstale":"2024-03-31T15:14:11.390332+0000","last_undegraded":"2024-03-31T15:14:11.390332+0000","last_fullsized":"2024-03-31T15:14:11.390332+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:09:48.241483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.29","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-03-31T15:14:08.253496+0000","last_change":"2024-03-31T15:14:08.253496+0000","last_active":"2024-03-31T15:14:08.242094+0000","last_peered":"2024-03-31T15:14:08.242094+0000","last_clean":"2024-03-31T15:14:08.242094+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:14:08.253496+0000","last_undegraded":"2024-03-31T15:14:08.253496+0000","last_fullsized":"2024-03-31T15:14:08.253496+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.28","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:11.390297+0000","last_change":"2024-03-31T15:14:09.283380+0000","last_active":"2024-03-31T15:14:11.390297+0000","last_peered":"2024-03-31T15:14:11.390297+0000","last_clean":"2024-03-31T15:14:11.390297+0000","last_became_active":"2024-03-31T15:14:09.283097+0000","last_became_peered":"2024-03-31T15:14:09.283097+0000","last_unstale":"2024-03-31T15:14:11.390297+0000","last_undegraded":"2024-03-31T15:14:11.390297+0000","last_fullsized":"2024-03-31T15:14:11.390297+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:00:11.817601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.27","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-03-31T15:14:08.254451+0000","last_change":"2024-03-31T15:14:08.254451+0000","last_active":"2024-03-31T15:14:08.242094+0000","last_peered":"2024-03-31T15:14:08.242094+0000","last_clean":"2024-03-31T15:14:08.242094+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:14:08.254451+0000","last_undegraded":"2024-03-31T15:14:08.254451+0000","last_fullsized":"2024-03-31T15:14:08.254451+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.26","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:10.892253+0000","last_change":"2024-03-31T15:14:09.276124+0000","last_active":"2024-03-31T15:14:10.892253+0000","last_peered":"2024-03-31T15:14:10.892253+0000","last_clean":"2024-03-31T15:14:10.892253+0000","last_became_active":"2024-03-31T15:14:09.275495+0000","last_became_peered":"2024-03-31T15:14:09.275495+0000","last_unstale":"2024-03-31T15:14:10.892253+0000","last_undegraded":"2024-03-31T15:14:10.892253+0000","last_fullsized":"2024-03-31T15:14:10.892253+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:38:53.950142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.25","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:11.391422+0000","last_change":"2024-03-31T15:14:09.290617+0000","last_active":"2024-03-31T15:14:11.391422+0000","last_peered":"2024-03-31T15:14:11.391422+0000","last_clean":"2024-03-31T15:14:11.391422+0000","last_became_active":"2024-03-31T15:14:09.290308+0000","last_became_peered":"2024-03-31T15:14:09.290308+0000","last_unstale":"2024-03-31T15:14:11.391422+0000","last_undegraded":"2024-03-31T15:14:11.391422+0000","last_fullsized":"2024-03-31T15:14:11.391422+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:43:42.509329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.24","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:11.391301+0000","last_change":"2024-03-31T15:14:09.290865+0000","last_active":"2024-03-31T15:14:11.391301+0000","last_peered":"2024-03-31T15:14:11.391301+0000","last_clean":"2024-03-31T15:14:11.391301+0000","last_became_active":"2024-03-31T15:14:09.290601+0000","last_became_peered":"2024-03-31T15:14:09.290601+0000","last_unstale":"2024-03-31T15:14:11.391301+0000","last_undegraded":"2024-03-31T15:14:11.391301+0000","last_fullsized":"2024-03-31T15:14:11.391301+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T19:21:02.579812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.23","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-03-31T15:14:08.258913+0000","last_change":"2024-03-31T15:14:08.258913+0000","last_active":"2024-03-31T15:14:08.242094+0000","last_peered":"2024-03-31T15:14:08.242094+0000","last_clean":"2024-03-31T15:14:08.242094+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:14:08.258913+0000","last_undegraded":"2024-03-31T15:14:08.258913+0000","last_fullsized":"2024-03-31T15:14:08.258913+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.22","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-03-31T15:14:08.260326+0000","last_change":"2024-03-31T15:14:08.260326+0000","last_active":"2024-03-31T15:14:08.242094+0000","last_peered":"2024-03-31T15:14:08.242094+0000","last_clean":"2024-03-31T15:14:08.242094+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:14:08.260326+0000","last_undegraded":"2024-03-31T15:14:08.260326+0000","last_fullsized":"2024-03-31T15:14:08.260326+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.21","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:10.892072+0000","last_change":"2024-03-31T15:14:09.276137+0000","last_active":"2024-03-31T15:14:10.892072+0000","last_peered":"2024-03-31T15:14:10.892072+0000","last_clean":"2024-03-31T15:14:10.892072+0000","last_became_active":"2024-03-31T15:14:09.275873+0000","last_became_peered":"2024-03-31T15:14:09.275873+0000","last_unstale":"2024-03-31T15:14:10.892072+0000","last_undegraded":"2024-03-31T15:14:10.892072+0000","last_fullsized":"2024-03-31T15:14:10.892072+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T17:03:26.938790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.20","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:11.389806+0000","last_change":"2024-03-31T15:14:09.285377+0000","last_active":"2024-03-31T15:14:11.389806+0000","last_peered":"2024-03-31T15:14:11.389806+0000","last_clean":"2024-03-31T15:14:11.389806+0000","last_became_active":"2024-03-31T15:14:09.285077+0000","last_became_peered":"2024-03-31T15:14:09.285077+0000","last_unstale":"2024-03-31T15:14:11.389806+0000","last_undegraded":"2024-03-31T15:14:11.389806+0000","last_fullsized":"2024-03-31T15:14:11.389806+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T15:39:45.840997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.1e","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:11.391059+0000","last_change":"2024-03-31T15:14:09.286260+0000","last_active":"2024-03-31T15:14:11.391059+0000","last_peered":"2024-03-31T15:14:11.391059+0000","last_clean":"2024-03-31T15:14:11.391059+0000","last_became_active":"2024-03-31T15:14:09.285985+0000","last_became_peered":"2024-03-31T15:14:09.285985+0000","last_unstale":"2024-03-31T15:14:11.391059+0000","last_undegraded":"2024-03-31T15:14:11.391059+0000","last_fullsized":"2024-03-31T15:14:11.391059+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T15:58:28.943455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1f","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-31T15:14:08.272805+0000","last_change":"2024-03-31T15:14:08.268805+0000","last_active":"2024-03-31T15:14:08.272805+0000","last_peered":"2024-03-31T15:14:08.272805+0000","last_clean":"2024-03-31T15:14:08.272805+0000","last_became_active":"2024-03-31T15:14:08.268515+0000","last_became_peered":"2024-03-31T15:14:08.268515+0000","last_unstale":"2024-03-31T15:14:08.272805+0000","last_undegraded":"2024-03-31T15:14:08.272805+0000","last_fullsized":"2024-03-31T15:14:08.272805+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_clean_scrub_stamp":"2024-03-31T15:14:07.235046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:47:23.413340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1f","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:10.278621+0000","last_change":"2024-03-31T15:14:09.281053+0000","last_active":"2024-03-31T15:14:10.278621+0000","last_peered":"2024-03-31T15:14:10.278621+0000","last_clean":"2024-03-31T15:14:10.278621+0000","last_became_active":"2024-03-31T15:14:09.280735+0000","last_became_peered":"2024-03-31T15:14:09.280735+0000","last_unstale":"2024-03-31T15:14:10.278621+0000","last_undegraded":"2024-03-31T15:14:10.278621+0000","last_fullsized":"2024-03-31T15:14:10.278621+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:55:28.086183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.1e","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-31T15:14:08.273342+0000","last_change":"2024-03-31T15:14:08.270517+0000","last_active":"2024-03-31T15:14:08.273342+0000","last_peered":"2024-03-31T15:14:08.273342+0000","last_clean":"2024-03-31T15:14:08.273342+0000","last_became_active":"2024-03-31T15:14:08.270219+0000","last_became_peered":"2024-03-31T15:14:08.270219+0000","last_unstale":"2024-03-31T15:14:08.273342+0000","last_undegraded":"2024-03-31T15:14:08.273342+0000","last_fullsized":"2024-03-31T15:14:08.273342+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_clean_scrub_stamp":"2024-03-31T15:14:07.235046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:40:13.939016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1c","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:11.391070+0000","last_change":"2024-03-31T15:14:09.285776+0000","last_active":"2024-03-31T15:14:11.391070+0000","last_peered":"2024-03-31T15:14:11.391070+0000","last_clean":"2024-03-31T15:14:11.391070+0000","last_became_active":"2024-03-31T15:14:09.285383+0000","last_became_peered":"2024-03-31T15:14:09.285383+0000","last_unstale":"2024-03-31T15:14:11.391070+0000","last_undegraded":"2024-03-31T15:14:11.391070+0000","last_fullsized":"2024-03-31T15:14:11.391070+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:45:29.558658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1d","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-31T15:14:08.274496+0000","last_change":"2024-03-31T15:14:08.268956+0000","last_active":"2024-03-31T15:14:08.274496+0000","last_peered":"2024-03-31T15:14:08.274496+0000","last_clean":"2024-03-31T15:14:08.274496+0000","last_became_active":"2024-03-31T15:14:08.268672+0000","last_became_peered":"2024-03-31T15:14:08.268672+0000","last_unstale":"2024-03-31T15:14:08.274496+0000","last_undegraded":"2024-03-31T15:14:08.274496+0000","last_fullsized":"2024-03-31T15:14:08.274496+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_clean_scrub_stamp":"2024-03-31T15:14:07.235046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:55:47.095073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1d","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-03-31T15:14:08.257689+0000","last_change":"2024-03-31T15:14:08.257689+0000","last_active":"2024-03-31T15:14:08.242094+0000","last_peered":"2024-03-31T15:14:08.242094+0000","last_clean":"2024-03-31T15:14:08.242094+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:14:08.257689+0000","last_undegraded":"2024-03-31T15:14:08.257689+0000","last_fullsized":"2024-03-31T15:14:08.257689+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1c","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-31T15:14:08.274904+0000","last_change":"2024-03-31T15:14:08.269987+0000","last_active":"2024-03-31T15:14:08.274904+0000","last_peered":"2024-03-31T15:14:08.274904+0000","last_clean":"2024-03-31T15:14:08.274904+0000","last_became_active":"2024-03-31T15:14:08.269241+0000","last_became_peered":"2024-03-31T15:14:08.269241+0000","last_unstale":"2024-03-31T15:14:08.274904+0000","last_undegraded":"2024-03-31T15:14:08.274904+0000","last_fullsized":"2024-03-31T15:14:08.274904+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_clean_scrub_stamp":"2024-03-31T15:14:07.235046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:54:34.951227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1a","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:10.891958+0000","last_change":"2024-03-31T15:14:09.282143+0000","last_active":"2024-03-31T15:14:10.891958+0000","last_peered":"2024-03-31T15:14:10.891958+0000","last_clean":"2024-03-31T15:14:10.891958+0000","last_became_active":"2024-03-31T15:14:09.281851+0000","last_became_peered":"2024-03-31T15:14:09.281851+0000","last_unstale":"2024-03-31T15:14:10.891958+0000","last_undegraded":"2024-03-31T15:14:10.891958+0000","last_fullsized":"2024-03-31T15:14:10.891958+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:07:40.319619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.1b","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-31T15:14:08.275156+0000","last_change":"2024-03-31T15:14:08.271392+0000","last_active":"2024-03-31T15:14:08.275156+0000","last_peered":"2024-03-31T15:14:08.275156+0000","last_clean":"2024-03-31T15:14:08.275156+0000","last_became_active":"2024-03-31T15:14:08.271138+0000","last_became_peered":"2024-03-31T15:14:08.271138+0000","last_unstale":"2024-03-31T15:14:08.275156+0000","last_undegraded":"2024-03-31T15:14:08.275156+0000","last_fullsized":"2024-03-31T15:14:08.275156+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_clean_scrub_stamp":"2024-03-31T15:14:07.235046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:45:22.838594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1b","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-03-31T15:14:08.258473+0000","last_change":"2024-03-31T15:14:08.258473+0000","last_active":"2024-03-31T15:14:08.242094+0000","last_peered":"2024-03-31T15:14:08.242094+0000","last_clean":"2024-03-31T15:14:08.242094+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:14:08.258473+0000","last_undegraded":"2024-03-31T15:14:08.258473+0000","last_fullsized":"2024-03-31T15:14:08.258473+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.1a","version":"108'1","reported_seq":15,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:11.390384+0000","last_change":"2024-03-31T15:14:08.273955+0000","last_active":"2024-03-31T15:14:11.390384+0000","last_peered":"2024-03-31T15:14:11.390384+0000","last_clean":"2024-03-31T15:14:11.390384+0000","last_became_active":"2024-03-31T15:14:08.273646+0000","last_became_peered":"2024-03-31T15:14:08.273646+0000","last_unstale":"2024-03-31T15:14:11.390384+0000","last_undegraded":"2024-03-31T15:14:11.390384+0000","last_fullsized":"2024-03-31T15:14:11.390384+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_clean_scrub_stamp":"2024-03-31T15:14:07.235046+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:27:01.349324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.18","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:10.891733+0000","last_change":"2024-03-31T15:14:09.278486+0000","last_active":"2024-03-31T15:14:10.891733+0000","last_peered":"2024-03-31T15:14:10.891733+0000","last_clean":"2024-03-31T15:14:10.891733+0000","last_became_active":"2024-03-31T15:14:09.278193+0000","last_became_peered":"2024-03-31T15:14:09.278193+0000","last_unstale":"2024-03-31T15:14:10.891733+0000","last_undegraded":"2024-03-31T15:14:10.891733+0000","last_fullsized":"2024-03-31T15:14:10.891733+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:42:40.870161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-31T15:14:08.270406+0000","last_change":"2024-03-31T15:14:08.266416+0000","last_active":"2024-03-31T15:14:08.270406+0000","last_peered":"2024-03-31T15:14:08.270406+0000","last_clean":"2024-03-31T15:14:08.270406+0000","last_became_active":"2024-03-31T15:14:08.266116+0000","last_became_peered":"2024-03-31T15:14:08.266116+0000","last_unstale":"2024-03-31T15:14:08.270406+0000","last_undegraded":"2024-03-31T15:14:08.270406+0000","last_fullsized":"2024-03-31T15:14:08.270406+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_clean_scrub_stamp":"2024-03-31T15:14:07.235046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:05:08.886029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.19","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-03-31T15:14:08.260207+0000","last_change":"2024-03-31T15:14:08.260207+0000","last_active":"2024-03-31T15:14:08.242094+0000","last_peered":"2024-03-31T15:14:08.242094+0000","last_clean":"2024-03-31T15:14:08.242094+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:14:08.260207+0000","last_undegraded":"2024-03-31T15:14:08.260207+0000","last_fullsized":"2024-03-31T15:14:08.260207+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":14,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:10.891808+0000","last_change":"2024-03-31T15:14:08.276025+0000","last_active":"2024-03-31T15:14:10.891808+0000","last_peered":"2024-03-31T15:14:10.891808+0000","last_clean":"2024-03-31T15:14:10.891808+0000","last_became_active":"2024-03-31T15:14:08.275541+0000","last_became_peered":"2024-03-31T15:14:08.275541+0000","last_unstale":"2024-03-31T15:14:10.891808+0000","last_undegraded":"2024-03-31T15:14:10.891808+0000","last_fullsized":"2024-03-31T15:14:10.891808+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_clean_scrub_stamp":"2024-03-31T15:14:07.235046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:34:33.623018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.16","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-03-31T15:14:08.259804+0000","last_change":"2024-03-31T15:14:08.259804+0000","last_active":"2024-03-31T15:14:08.242094+0000","last_peered":"2024-03-31T15:14:08.242094+0000","last_clean":"2024-03-31T15:14:08.242094+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:14:08.259804+0000","last_undegraded":"2024-03-31T15:14:08.259804+0000","last_fullsized":"2024-03-31T15:14:08.259804+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.17","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-31T15:14:08.278508+0000","last_change":"2024-03-31T15:14:08.275403+0000","last_active":"2024-03-31T15:14:08.278508+0000","last_peered":"2024-03-31T15:14:08.278508+0000","last_clean":"2024-03-31T15:14:08.278508+0000","last_became_active":"2024-03-31T15:14:08.275122+0000","last_became_peered":"2024-03-31T15:14:08.275122+0000","last_unstale":"2024-03-31T15:14:08.278508+0000","last_undegraded":"2024-03-31T15:14:08.278508+0000","last_fullsized":"2024-03-31T15:14:08.278508+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_clean_scrub_stamp":"2024-03-31T15:14:07.235046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T15:33:09.566750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.17","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:10.278750+0000","last_change":"2024-03-31T15:14:09.298453+0000","last_active":"2024-03-31T15:14:10.278750+0000","last_peered":"2024-03-31T15:14:10.278750+0000","last_clean":"2024-03-31T15:14:10.278750+0000","last_became_active":"2024-03-31T15:14:09.298160+0000","last_became_peered":"2024-03-31T15:14:09.298160+0000","last_unstale":"2024-03-31T15:14:10.278750+0000","last_undegraded":"2024-03-31T15:14:10.278750+0000","last_fullsized":"2024-03-31T15:14:10.278750+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:48:01.826468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":14,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:11.391501+0000","last_change":"2024-03-31T15:14:08.272714+0000","last_active":"2024-03-31T15:14:11.391501+0000","last_peered":"2024-03-31T15:14:11.391501+0000","last_clean":"2024-03-31T15:14:11.391501+0000","last_became_active":"2024-03-31T15:14:08.272416+0000","last_became_peered":"2024-03-31T15:14:08.272416+0000","last_unstale":"2024-03-31T15:14:11.391501+0000","last_undegraded":"2024-03-31T15:14:11.391501+0000","last_fullsized":"2024-03-31T15:14:11.391501+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_clean_scrub_stamp":"2024-03-31T15:14:07.235046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:34:33.623018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.14","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:11.391432+0000","last_change":"2024-03-31T15:14:09.281244+0000","last_active":"2024-03-31T15:14:11.391432+0000","last_peered":"2024-03-31T15:14:11.391432+0000","last_clean":"2024-03-31T15:14:11.391432+0000","last_became_active":"2024-03-31T15:14:09.280267+0000","last_became_peered":"2024-03-31T15:14:09.280267+0000","last_unstale":"2024-03-31T15:14:11.391432+0000","last_undegraded":"2024-03-31T15:14:11.391432+0000","last_fullsized":"2024-03-31T15:14:11.391432+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:23:54.993797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.15","version":"108'3","reported_seq":17,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:10.891863+0000","last_change":"2024-03-31T15:14:08.274322+0000","last_active":"2024-03-31T15:14:10.891863+0000","last_peered":"2024-03-31T15:14:10.891863+0000","last_clean":"2024-03-31T15:14:10.891863+0000","last_became_active":"2024-03-31T15:14:08.274025+0000","last_became_peered":"2024-03-31T15:14:08.274025+0000","last_unstale":"2024-03-31T15:14:10.891863+0000","last_undegraded":"2024-03-31T15:14:10.891863+0000","last_fullsized":"2024-03-31T15:14:10.891863+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_clean_scrub_stamp":"2024-03-31T15:14:07.235046+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:54:49.668391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.15","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-03-31T15:14:08.258737+0000","last_change":"2024-03-31T15:14:08.258737+0000","last_active":"2024-03-31T15:14:08.242094+0000","last_peered":"2024-03-31T15:14:08.242094+0000","last_clean":"2024-03-31T15:14:08.242094+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:14:08.258737+0000","last_undegraded":"2024-03-31T15:14:08.258737+0000","last_fullsized":"2024-03-31T15:14:08.258737+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.14","version":"108'6","reported_seq":20,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:10.892187+0000","last_change":"2024-03-31T15:14:08.277536+0000","last_active":"2024-03-31T15:14:10.892187+0000","last_peered":"2024-03-31T15:14:10.892187+0000","last_clean":"2024-03-31T15:14:10.892187+0000","last_became_active":"2024-03-31T15:14:08.277279+0000","last_became_peered":"2024-03-31T15:14:08.277279+0000","last_unstale":"2024-03-31T15:14:10.892187+0000","last_undegraded":"2024-03-31T15:14:10.892187+0000","last_fullsized":"2024-03-31T15:14:10.892187+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_clean_scrub_stamp":"2024-03-31T15:14:07.235046+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:08:03.916239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.12","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:11.390394+0000","last_change":"2024-03-31T15:14:09.275078+0000","last_active":"2024-03-31T15:14:11.390394+0000","last_peered":"2024-03-31T15:14:11.390394+0000","last_clean":"2024-03-31T15:14:11.390394+0000","last_became_active":"2024-03-31T15:14:09.274698+0000","last_became_peered":"2024-03-31T15:14:09.274698+0000","last_unstale":"2024-03-31T15:14:11.390394+0000","last_undegraded":"2024-03-31T15:14:11.390394+0000","last_fullsized":"2024-03-31T15:14:11.390394+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:30:56.093324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.13","version":"108'2","reported_seq":16,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:11.390958+0000","last_change":"2024-03-31T15:14:08.266097+0000","last_active":"2024-03-31T15:14:11.390958+0000","last_peered":"2024-03-31T15:14:11.390958+0000","last_clean":"2024-03-31T15:14:11.390958+0000","last_became_active":"2024-03-31T15:14:08.265812+0000","last_became_peered":"2024-03-31T15:14:08.265812+0000","last_unstale":"2024-03-31T15:14:11.390958+0000","last_undegraded":"2024-03-31T15:14:11.390958+0000","last_fullsized":"2024-03-31T15:14:11.390958+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_clean_scrub_stamp":"2024-03-31T15:14:07.235046+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:49:52.886409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.13","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:11.391329+0000","last_change":"2024-03-31T15:14:09.282087+0000","last_active":"2024-03-31T15:14:11.391329+0000","last_peered":"2024-03-31T15:14:11.391329+0000","last_clean":"2024-03-31T15:14:11.391329+0000","last_became_active":"2024-03-31T15:14:09.281693+0000","last_became_peered":"2024-03-31T15:14:09.281693+0000","last_unstale":"2024-03-31T15:14:11.391329+0000","last_undegraded":"2024-03-31T15:14:11.391329+0000","last_fullsized":"2024-03-31T15:14:11.391329+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:32:06.156300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":14,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:10.278350+0000","last_change":"2024-03-31T15:14:08.272919+0000","last_active":"2024-03-31T15:14:10.278350+0000","last_peered":"2024-03-31T15:14:10.278350+0000","last_clean":"2024-03-31T15:14:10.278350+0000","last_became_active":"2024-03-31T15:14:08.272309+0000","last_became_peered":"2024-03-31T15:14:08.272309+0000","last_unstale":"2024-03-31T15:14:10.278350+0000","last_undegraded":"2024-03-31T15:14:10.278350+0000","last_fullsized":"2024-03-31T15:14:10.278350+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_clean_scrub_stamp":"2024-03-31T15:14:07.235046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:56:10.303847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.11","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:11.390150+0000","last_change":"2024-03-31T15:14:09.274826+0000","last_active":"2024-03-31T15:14:11.390150+0000","last_peered":"2024-03-31T15:14:11.390150+0000","last_clean":"2024-03-31T15:14:11.390150+0000","last_became_active":"2024-03-31T15:14:09.274550+0000","last_became_peered":"2024-03-31T15:14:09.274550+0000","last_unstale":"2024-03-31T15:14:11.390150+0000","last_undegraded":"2024-03-31T15:14:11.390150+0000","last_fullsized":"2024-03-31T15:14:11.390150+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T17:25:28.357433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.10","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-31T15:14:08.278587+0000","last_change":"2024-03-31T15:14:08.276042+0000","last_active":"2024-03-31T15:14:08.278587+0000","last_peered":"2024-03-31T15:14:08.278587+0000","last_clean":"2024-03-31T15:14:08.278587+0000","last_became_active":"2024-03-31T15:14:08.275766+0000","last_became_peered":"2024-03-31T15:14:08.275766+0000","last_unstale":"2024-03-31T15:14:08.278587+0000","last_undegraded":"2024-03-31T15:14:08.278587+0000","last_fullsized":"2024-03-31T15:14:08.278587+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_clean_scrub_stamp":"2024-03-31T15:14:07.235046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:38:29.678222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.e","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-03-31T15:14:08.258499+0000","last_change":"2024-03-31T15:14:08.258499+0000","last_active":"2024-03-31T15:14:08.242094+0000","last_peered":"2024-03-31T15:14:08.242094+0000","last_clean":"2024-03-31T15:14:08.242094+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:14:08.258499+0000","last_undegraded":"2024-03-31T15:14:08.258499+0000","last_fullsized":"2024-03-31T15:14:08.258499+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.f","version":"108'1","reported_seq":15,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:10.892528+0000","last_change":"2024-03-31T15:14:08.273521+0000","last_active":"2024-03-31T15:14:10.892528+0000","last_peered":"2024-03-31T15:14:10.892528+0000","last_clean":"2024-03-31T15:14:10.892528+0000","last_became_active":"2024-03-31T15:14:08.271683+0000","last_became_peered":"2024-03-31T15:14:08.271683+0000","last_unstale":"2024-03-31T15:14:10.892528+0000","last_undegraded":"2024-03-31T15:14:10.892528+0000","last_fullsized":"2024-03-31T15:14:10.892528+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_clean_scrub_stamp":"2024-03-31T15:14:07.235046+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:08:12.284361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.f","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:11.391301+0000","last_change":"2024-03-31T15:14:09.292401+0000","last_active":"2024-03-31T15:14:11.391301+0000","last_peered":"2024-03-31T15:14:11.391301+0000","last_clean":"2024-03-31T15:14:11.391301+0000","last_became_active":"2024-03-31T15:14:09.292126+0000","last_became_peered":"2024-03-31T15:14:09.292126+0000","last_unstale":"2024-03-31T15:14:11.391301+0000","last_undegraded":"2024-03-31T15:14:11.391301+0000","last_fullsized":"2024-03-31T15:14:11.391301+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:24:54.943510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.e","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-31T15:14:08.276874+0000","last_change":"2024-03-31T15:14:08.272765+0000","last_active":"2024-03-31T15:14:08.276874+0000","last_peered":"2024-03-31T15:14:08.276874+0000","last_clean":"2024-03-31T15:14:08.276874+0000","last_became_active":"2024-03-31T15:14:08.272496+0000","last_became_peered":"2024-03-31T15:14:08.272496+0000","last_unstale":"2024-03-31T15:14:08.276874+0000","last_undegraded":"2024-03-31T15:14:08.276874+0000","last_fullsized":"2024-03-31T15:14:08.276874+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_clean_scrub_stamp":"2024-03-31T15:14:07.235046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:30:40.304653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.c","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-03-31T15:14:08.257389+0000","last_change":"2024-03-31T15:14:08.257389+0000","last_active":"2024-03-31T15:14:08.242094+0000","last_peered":"2024-03-31T15:14:08.242094+0000","last_clean":"2024-03-31T15:14:08.242094+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:14:08.257389+0000","last_undegraded":"2024-03-31T15:14:08.257389+0000","last_fullsized":"2024-03-31T15:14:08.257389+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.d","version":"108'1","reported_seq":15,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:10.278500+0000","last_change":"2024-03-31T15:14:08.278494+0000","last_active":"2024-03-31T15:14:10.278500+0000","last_peered":"2024-03-31T15:14:10.278500+0000","last_clean":"2024-03-31T15:14:10.278500+0000","last_became_active":"2024-03-31T15:14:08.278267+0000","last_became_peered":"2024-03-31T15:14:08.278267+0000","last_unstale":"2024-03-31T15:14:10.278500+0000","last_undegraded":"2024-03-31T15:14:10.278500+0000","last_fullsized":"2024-03-31T15:14:10.278500+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_clean_scrub_stamp":"2024-03-31T15:14:07.235046+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:49:25.058336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.d","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-03-31T15:14:08.257107+0000","last_change":"2024-03-31T15:14:08.257107+0000","last_active":"2024-03-31T15:14:08.242094+0000","last_peered":"2024-03-31T15:14:08.242094+0000","last_clean":"2024-03-31T15:14:08.242094+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:14:08.257107+0000","last_undegraded":"2024-03-31T15:14:08.257107+0000","last_fullsized":"2024-03-31T15:14:08.257107+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-31T15:14:08.275180+0000","last_change":"2024-03-31T15:14:08.271960+0000","last_active":"2024-03-31T15:14:08.275180+0000","last_peered":"2024-03-31T15:14:08.275180+0000","last_clean":"2024-03-31T15:14:08.275180+0000","last_became_active":"2024-03-31T15:14:08.271691+0000","last_became_peered":"2024-03-31T15:14:08.271691+0000","last_unstale":"2024-03-31T15:14:08.275180+0000","last_undegraded":"2024-03-31T15:14:08.275180+0000","last_fullsized":"2024-03-31T15:14:08.275180+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_clean_scrub_stamp":"2024-03-31T15:14:07.235046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T17:04:17.133375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.a","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-03-31T15:14:08.257121+0000","last_change":"2024-03-31T15:14:08.257121+0000","last_active":"2024-03-31T15:14:08.242094+0000","last_peered":"2024-03-31T15:14:08.242094+0000","last_clean":"2024-03-31T15:14:08.242094+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:14:08.257121+0000","last_undegraded":"2024-03-31T15:14:08.257121+0000","last_fullsized":"2024-03-31T15:14:08.257121+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.b","version":"108'2","reported_seq":16,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:11.391062+0000","last_change":"2024-03-31T15:14:08.274852+0000","last_active":"2024-03-31T15:14:11.391062+0000","last_peered":"2024-03-31T15:14:11.391062+0000","last_clean":"2024-03-31T15:14:11.391062+0000","last_became_active":"2024-03-31T15:14:08.274568+0000","last_became_peered":"2024-03-31T15:14:08.274568+0000","last_unstale":"2024-03-31T15:14:11.391062+0000","last_undegraded":"2024-03-31T15:14:11.391062+0000","last_fullsized":"2024-03-31T15:14:11.391062+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_clean_scrub_stamp":"2024-03-31T15:14:07.235046+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T19:34:36.202544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.b","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:10.891810+0000","last_change":"2024-03-31T15:14:09.280942+0000","last_active":"2024-03-31T15:14:10.891810+0000","last_peered":"2024-03-31T15:14:10.891810+0000","last_clean":"2024-03-31T15:14:10.891810+0000","last_became_active":"2024-03-31T15:14:09.280678+0000","last_became_peered":"2024-03-31T15:14:09.280678+0000","last_unstale":"2024-03-31T15:14:10.891810+0000","last_undegraded":"2024-03-31T15:14:10.891810+0000","last_fullsized":"2024-03-31T15:14:10.891810+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:00:22.586826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-31T15:14:08.278516+0000","last_change":"2024-03-31T15:14:08.275261+0000","last_active":"2024-03-31T15:14:08.278516+0000","last_peered":"2024-03-31T15:14:08.278516+0000","last_clean":"2024-03-31T15:14:08.278516+0000","last_became_active":"2024-03-31T15:14:08.274961+0000","last_became_peered":"2024-03-31T15:14:08.274961+0000","last_unstale":"2024-03-31T15:14:08.278516+0000","last_undegraded":"2024-03-31T15:14:08.278516+0000","last_fullsized":"2024-03-31T15:14:08.278516+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_clean_scrub_stamp":"2024-03-31T15:14:07.235046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T17:28:54.278793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.8","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-03-31T15:14:08.257584+0000","last_change":"2024-03-31T15:14:08.257584+0000","last_active":"2024-03-31T15:14:08.242094+0000","last_peered":"2024-03-31T15:14:08.242094+0000","last_clean":"2024-03-31T15:14:08.242094+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:14:08.257584+0000","last_undegraded":"2024-03-31T15:14:08.257584+0000","last_fullsized":"2024-03-31T15:14:08.257584+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":14,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:11.390727+0000","last_change":"2024-03-31T15:14:08.275022+0000","last_active":"2024-03-31T15:14:11.390727+0000","last_peered":"2024-03-31T15:14:11.390727+0000","last_clean":"2024-03-31T15:14:11.390727+0000","last_became_active":"2024-03-31T15:14:08.274740+0000","last_became_peered":"2024-03-31T15:14:08.274740+0000","last_unstale":"2024-03-31T15:14:11.390727+0000","last_undegraded":"2024-03-31T15:14:11.390727+0000","last_fullsized":"2024-03-31T15:14:11.390727+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_clean_scrub_stamp":"2024-03-31T15:14:07.235046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:25:35.260804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.9","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:11.390344+0000","last_change":"2024-03-31T15:14:09.291069+0000","last_active":"2024-03-31T15:14:11.390344+0000","last_peered":"2024-03-31T15:14:11.390344+0000","last_clean":"2024-03-31T15:14:11.390344+0000","last_became_active":"2024-03-31T15:14:09.290802+0000","last_became_peered":"2024-03-31T15:14:09.290802+0000","last_unstale":"2024-03-31T15:14:11.390344+0000","last_undegraded":"2024-03-31T15:14:11.390344+0000","last_fullsized":"2024-03-31T15:14:11.390344+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:36:20.417016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":14,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:10.891861+0000","last_change":"2024-03-31T15:14:08.270333+0000","last_active":"2024-03-31T15:14:10.891861+0000","last_peered":"2024-03-31T15:14:10.891861+0000","last_clean":"2024-03-31T15:14:10.891861+0000","last_became_active":"2024-03-31T15:14:08.269866+0000","last_became_peered":"2024-03-31T15:14:08.269866+0000","last_unstale":"2024-03-31T15:14:10.891861+0000","last_undegraded":"2024-03-31T15:14:10.891861+0000","last_fullsized":"2024-03-31T15:14:10.891861+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_clean_scrub_stamp":"2024-03-31T15:14:07.235046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:55:27.079135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.6","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:10.278592+0000","last_change":"2024-03-31T15:14:09.299295+0000","last_active":"2024-03-31T15:14:10.278592+0000","last_peered":"2024-03-31T15:14:10.278592+0000","last_clean":"2024-03-31T15:14:10.278592+0000","last_became_active":"2024-03-31T15:14:09.299016+0000","last_became_peered":"2024-03-31T15:14:09.299016+0000","last_unstale":"2024-03-31T15:14:10.278592+0000","last_undegraded":"2024-03-31T15:14:10.278592+0000","last_fullsized":"2024-03-31T15:14:10.278592+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:34:34.630067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.7","version":"108'2","reported_seq":16,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:11.391777+0000","last_change":"2024-03-31T15:14:08.276158+0000","last_active":"2024-03-31T15:14:11.391777+0000","last_peered":"2024-03-31T15:14:11.391777+0000","last_clean":"2024-03-31T15:14:11.391777+0000","last_became_active":"2024-03-31T15:14:08.275856+0000","last_became_peered":"2024-03-31T15:14:08.275856+0000","last_unstale":"2024-03-31T15:14:11.391777+0000","last_undegraded":"2024-03-31T15:14:11.391777+0000","last_fullsized":"2024-03-31T15:14:11.391777+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_clean_scrub_stamp":"2024-03-31T15:14:07.235046+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:48:00.819420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.7","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:10.892615+0000","last_change":"2024-03-31T15:14:09.277522+0000","last_active":"2024-03-31T15:14:10.892615+0000","last_peered":"2024-03-31T15:14:10.892615+0000","last_clean":"2024-03-31T15:14:10.892615+0000","last_became_active":"2024-03-31T15:14:09.276889+0000","last_became_peered":"2024-03-31T15:14:09.276889+0000","last_unstale":"2024-03-31T15:14:10.892615+0000","last_undegraded":"2024-03-31T15:14:10.892615+0000","last_fullsized":"2024-03-31T15:14:10.892615+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:45:51.393113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.6","version":"109'1","reported_seq":15,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:10.278704+0000","last_change":"2024-03-31T15:14:08.265332+0000","last_active":"2024-03-31T15:14:10.278704+0000","last_peered":"2024-03-31T15:14:10.278704+0000","last_clean":"2024-03-31T15:14:10.278704+0000","last_became_active":"2024-03-31T15:14:08.264838+0000","last_became_peered":"2024-03-31T15:14:08.264838+0000","last_unstale":"2024-03-31T15:14:10.278704+0000","last_undegraded":"2024-03-31T15:14:10.278704+0000","last_fullsized":"2024-03-31T15:14:10.278704+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_clean_scrub_stamp":"2024-03-31T15:14:07.235046+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:32:53.111430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.4","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:10.892152+0000","last_change":"2024-03-31T15:14:09.279412+0000","last_active":"2024-03-31T15:14:10.892152+0000","last_peered":"2024-03-31T15:14:10.892152+0000","last_clean":"2024-03-31T15:14:10.892152+0000","last_became_active":"2024-03-31T15:14:09.278696+0000","last_became_peered":"2024-03-31T15:14:09.278696+0000","last_unstale":"2024-03-31T15:14:10.892152+0000","last_undegraded":"2024-03-31T15:14:10.892152+0000","last_fullsized":"2024-03-31T15:14:10.892152+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:57:34.106053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":14,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:10.891892+0000","last_change":"2024-03-31T15:14:08.270001+0000","last_active":"2024-03-31T15:14:10.891892+0000","last_peered":"2024-03-31T15:14:10.891892+0000","last_clean":"2024-03-31T15:14:10.891892+0000","last_became_active":"2024-03-31T15:14:08.268897+0000","last_became_peered":"2024-03-31T15:14:08.268897+0000","last_unstale":"2024-03-31T15:14:10.891892+0000","last_undegraded":"2024-03-31T15:14:10.891892+0000","last_fullsized":"2024-03-31T15:14:10.891892+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_clean_scrub_stamp":"2024-03-31T15:14:07.235046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T19:21:04.678038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.5","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:10.278439+0000","last_change":"2024-03-31T15:14:09.292175+0000","last_active":"2024-03-31T15:14:10.278439+0000","last_peered":"2024-03-31T15:14:10.278439+0000","last_clean":"2024-03-31T15:14:10.278439+0000","last_became_active":"2024-03-31T15:14:09.291875+0000","last_became_peered":"2024-03-31T15:14:09.291875+0000","last_unstale":"2024-03-31T15:14:10.278439+0000","last_undegraded":"2024-03-31T15:14:10.278439+0000","last_fullsized":"2024-03-31T15:14:10.278439+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:27:02.356373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":14,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:11.391479+0000","last_change":"2024-03-31T15:14:08.268133+0000","last_active":"2024-03-31T15:14:11.391479+0000","last_peered":"2024-03-31T15:14:11.391479+0000","last_clean":"2024-03-31T15:14:11.391479+0000","last_became_active":"2024-03-31T15:14:08.267768+0000","last_became_peered":"2024-03-31T15:14:08.267768+0000","last_unstale":"2024-03-31T15:14:11.391479+0000","last_undegraded":"2024-03-31T15:14:11.391479+0000","last_fullsized":"2024-03-31T15:14:11.391479+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_clean_scrub_stamp":"2024-03-31T15:14:07.235046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:55:27.079135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.2","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-03-31T15:14:08.254000+0000","last_change":"2024-03-31T15:14:08.254000+0000","last_active":"2024-03-31T15:14:08.242094+0000","last_peered":"2024-03-31T15:14:08.242094+0000","last_clean":"2024-03-31T15:14:08.242094+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:14:08.254000+0000","last_undegraded":"2024-03-31T15:14:08.254000+0000","last_fullsized":"2024-03-31T15:14:08.254000+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-31T15:14:08.272766+0000","last_change":"2024-03-31T15:14:08.268809+0000","last_active":"2024-03-31T15:14:08.272766+0000","last_peered":"2024-03-31T15:14:08.272766+0000","last_clean":"2024-03-31T15:14:08.272766+0000","last_became_active":"2024-03-31T15:14:08.268473+0000","last_became_peered":"2024-03-31T15:14:08.268473+0000","last_unstale":"2024-03-31T15:14:08.272766+0000","last_undegraded":"2024-03-31T15:14:08.272766+0000","last_fullsized":"2024-03-31T15:14:08.272766+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_clean_scrub_stamp":"2024-03-31T15:14:07.235046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:09:33.273967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.3","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-03-31T15:14:08.255261+0000","last_change":"2024-03-31T15:14:08.255261+0000","last_active":"2024-03-31T15:14:08.242094+0000","last_peered":"2024-03-31T15:14:08.242094+0000","last_clean":"2024-03-31T15:14:08.242094+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:14:08.255261+0000","last_undegraded":"2024-03-31T15:14:08.255261+0000","last_fullsized":"2024-03-31T15:14:08.255261+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-31T15:14:08.272985+0000","last_change":"2024-03-31T15:14:08.269651+0000","last_active":"2024-03-31T15:14:08.272985+0000","last_peered":"2024-03-31T15:14:08.272985+0000","last_clean":"2024-03-31T15:14:08.272985+0000","last_became_active":"2024-03-31T15:14:08.269372+0000","last_became_peered":"2024-03-31T15:14:08.269372+0000","last_unstale":"2024-03-31T15:14:08.272985+0000","last_undegraded":"2024-03-31T15:14:08.272985+0000","last_fullsized":"2024-03-31T15:14:08.272985+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_clean_scrub_stamp":"2024-03-31T15:14:07.235046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:15:08.190993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.0","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-03-31T15:14:08.257010+0000","last_change":"2024-03-31T15:14:08.257010+0000","last_active":"2024-03-31T15:14:08.242094+0000","last_peered":"2024-03-31T15:14:08.242094+0000","last_clean":"2024-03-31T15:14:08.242094+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:14:08.257010+0000","last_undegraded":"2024-03-31T15:14:08.257010+0000","last_fullsized":"2024-03-31T15:14:08.257010+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-31T15:14:08.277067+0000","last_change":"2024-03-31T15:14:08.271032+0000","last_active":"2024-03-31T15:14:08.277067+0000","last_peered":"2024-03-31T15:14:08.277067+0000","last_clean":"2024-03-31T15:14:08.277067+0000","last_became_active":"2024-03-31T15:14:08.270781+0000","last_became_peered":"2024-03-31T15:14:08.270781+0000","last_unstale":"2024-03-31T15:14:08.277067+0000","last_undegraded":"2024-03-31T15:14:08.277067+0000","last_fullsized":"2024-03-31T15:14:08.277067+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_clean_scrub_stamp":"2024-03-31T15:14:07.235046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:59:46.849346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:10.892154+0000","last_change":"2024-03-31T15:14:09.288490+0000","last_active":"2024-03-31T15:14:10.892154+0000","last_peered":"2024-03-31T15:14:10.892154+0000","last_clean":"2024-03-31T15:14:10.892154+0000","last_became_active":"2024-03-31T15:14:09.284507+0000","last_became_peered":"2024-03-31T15:14:09.284507+0000","last_unstale":"2024-03-31T15:14:10.892154+0000","last_undegraded":"2024-03-31T15:14:10.892154+0000","last_fullsized":"2024-03-31T15:14:10.892154+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:09:03.491873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":14,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:10.891910+0000","last_change":"2024-03-31T15:14:08.270270+0000","last_active":"2024-03-31T15:14:10.891910+0000","last_peered":"2024-03-31T15:14:10.891910+0000","last_clean":"2024-03-31T15:14:10.891910+0000","last_became_active":"2024-03-31T15:14:08.269482+0000","last_became_peered":"2024-03-31T15:14:08.269482+0000","last_unstale":"2024-03-31T15:14:10.891910+0000","last_undegraded":"2024-03-31T15:14:10.891910+0000","last_fullsized":"2024-03-31T15:14:10.891910+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_clean_scrub_stamp":"2024-03-31T15:14:07.235046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:48:00.819420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.10","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:11.390177+0000","last_change":"2024-03-31T15:14:09.277917+0000","last_active":"2024-03-31T15:14:11.390177+0000","last_peered":"2024-03-31T15:14:11.390177+0000","last_clean":"2024-03-31T15:14:11.390177+0000","last_became_active":"2024-03-31T15:14:09.277651+0000","last_became_peered":"2024-03-31T15:14:09.277651+0000","last_unstale":"2024-03-31T15:14:11.390177+0000","last_undegraded":"2024-03-31T15:14:11.390177+0000","last_fullsized":"2024-03-31T15:14:11.390177+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T16:10:45.541190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-31T15:14:08.275222+0000","last_change":"2024-03-31T15:14:08.271941+0000","last_active":"2024-03-31T15:14:08.275222+0000","last_peered":"2024-03-31T15:14:08.275222+0000","last_clean":"2024-03-31T15:14:08.275222+0000","last_became_active":"2024-03-31T15:14:08.271589+0000","last_became_peered":"2024-03-31T15:14:08.271589+0000","last_unstale":"2024-03-31T15:14:08.275222+0000","last_undegraded":"2024-03-31T15:14:08.275222+0000","last_fullsized":"2024-03-31T15:14:08.275222+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_clean_scrub_stamp":"2024-03-31T15:14:07.235046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:41:56.592702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":331,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-31T15:14:08.262750+0000","last_change":"2024-03-31T15:03:24.329890+0000","last_active":"2024-03-31T15:14:08.262750+0000","last_peered":"2024-03-31T15:14:08.262750+0000","last_clean":"2024-03-31T15:14:08.262750+0000","last_became_active":"2024-03-31T15:03:24.328882+0000","last_became_peered":"2024-03-31T15:03:24.328882+0000","last_unstale":"2024-03-31T15:14:08.262750+0000","last_undegraded":"2024-03-31T15:14:08.262750+0000","last_fullsized":"2024-03-31T15:14:08.262750+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:22.885535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:22.885535+0000","last_clean_scrub_stamp":"2024-03-31T15:03:22.885535+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:18:29.004296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":17,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":16,"num_pg":32,"stat_sum":{"num_bytes":1828,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":19,"ondisk_log_size":19,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509574,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33040,"kb_used_data":1860,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338800,"statfs":{"total":96636764160,"available":96602931200,"internally_reserved":0,"allocated":1904640,"data_stored":1155091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509573,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32476,"kb_used_data":1292,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339364,"statfs":{"total":96636764160,"available":96603508736,"internally_reserved":0,"allocated":1323008,"data_stored":570998,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":15,"seq":64424509572,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32460,"kb_used_data":1280,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339380,"statfs":{"total":96636764160,"available":96603525120,"internally_reserved":0,"allocated":1310720,"data_stored":564723,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509571,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36632,"kb_used_data":1292,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335208,"statfs":{"total":96636764160,"available":96599252992,"internally_reserved":0,"allocated":1323008,"data_stored":570998,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542278,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32508,"kb_used_data":1320,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339332,"statfs":{"total":96636764160,"available":96603475968,"internally_reserved":0,"allocated":1351680,"data_stored":584143,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542278,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32512,"kb_used_data":1332,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339328,"statfs":{"total":96636764160,"available":96603471872,"internally_reserved":0,"allocated":1363968,"data_stored":585755,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542278,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32504,"kb_used_data":1324,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339336,"statfs":{"total":96636764160,"available":96603480064,"internally_reserved":0,"allocated":1355776,"data_stored":584197,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542278,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33100,"kb_used_data":1916,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338740,"statfs":{"total":96636764160,"available":96602869760,"internally_reserved":0,"allocated":1961984,"data_stored":1176165,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-31T15:14:14.163 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.3e is not active+clean 2024-03-31T15:14:14.163 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.3e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 107, 'state': 'creating+peering', 'last_fresh': '2024-03-31T15:14:08.253600+0000', 'last_change': '2024-03-31T15:14:08.253600+0000', 'last_active': '2024-03-31T15:14:08.242094+0000', 'last_peered': '2024-03-31T15:14:08.242094+0000', 'last_clean': '2024-03-31T15:14:08.242094+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-31T15:14:08.253600+0000', 'last_undegraded': '2024-03-31T15:14:08.253600+0000', 'last_fullsized': '2024-03-31T15:14:08.253600+0000', 'mapping_epoch': 107, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 107, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-31T15:14:08.242094+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-31T15:14:08.242094+0000', 'last_clean_scrub_stamp': '2024-03-31T15:14:08.242094+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 1], 'acting': [4, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-31T15:14:14.163 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.3d is not active+clean 2024-03-31T15:14:14.163 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.3d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 107, 'state': 'creating+peering', 'last_fresh': '2024-03-31T15:14:08.258560+0000', 'last_change': '2024-03-31T15:14:08.258560+0000', 'last_active': '2024-03-31T15:14:08.242094+0000', 'last_peered': '2024-03-31T15:14:08.242094+0000', 'last_clean': '2024-03-31T15:14:08.242094+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-31T15:14:08.258560+0000', 'last_undegraded': '2024-03-31T15:14:08.258560+0000', 'last_fullsized': '2024-03-31T15:14:08.258560+0000', 'mapping_epoch': 107, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 107, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-31T15:14:08.242094+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-31T15:14:08.242094+0000', 'last_clean_scrub_stamp': '2024-03-31T15:14:08.242094+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 4], 'acting': [7, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-31T15:14:14.163 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.3b is not active+clean 2024-03-31T15:14:14.164 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.3b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 107, 'state': 'creating+peering', 'last_fresh': '2024-03-31T15:14:08.255591+0000', 'last_change': '2024-03-31T15:14:08.255591+0000', 'last_active': '2024-03-31T15:14:08.242094+0000', 'last_peered': '2024-03-31T15:14:08.242094+0000', 'last_clean': '2024-03-31T15:14:08.242094+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-31T15:14:08.255591+0000', 'last_undegraded': '2024-03-31T15:14:08.255591+0000', 'last_fullsized': '2024-03-31T15:14:08.255591+0000', 'mapping_epoch': 107, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 107, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-31T15:14:08.242094+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-31T15:14:08.242094+0000', 'last_clean_scrub_stamp': '2024-03-31T15:14:08.242094+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-31T15:14:14.164 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.39 is not active+clean 2024-03-31T15:14:14.164 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.39', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 107, 'state': 'creating+peering', 'last_fresh': '2024-03-31T15:14:08.259762+0000', 'last_change': '2024-03-31T15:14:08.259762+0000', 'last_active': '2024-03-31T15:14:08.242094+0000', 'last_peered': '2024-03-31T15:14:08.242094+0000', 'last_clean': '2024-03-31T15:14:08.242094+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-31T15:14:08.259762+0000', 'last_undegraded': '2024-03-31T15:14:08.259762+0000', 'last_fullsized': '2024-03-31T15:14:08.259762+0000', 'mapping_epoch': 107, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 107, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-31T15:14:08.242094+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-31T15:14:08.242094+0000', 'last_clean_scrub_stamp': '2024-03-31T15:14:08.242094+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 6], 'acting': [5, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-31T15:14:14.164 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.38 is not active+clean 2024-03-31T15:14:14.164 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.38', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 107, 'state': 'creating+peering', 'last_fresh': '2024-03-31T15:14:08.258038+0000', 'last_change': '2024-03-31T15:14:08.258038+0000', 'last_active': '2024-03-31T15:14:08.242094+0000', 'last_peered': '2024-03-31T15:14:08.242094+0000', 'last_clean': '2024-03-31T15:14:08.242094+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-31T15:14:08.258038+0000', 'last_undegraded': '2024-03-31T15:14:08.258038+0000', 'last_fullsized': '2024-03-31T15:14:08.258038+0000', 'mapping_epoch': 107, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 107, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-31T15:14:08.242094+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-31T15:14:08.242094+0000', 'last_clean_scrub_stamp': '2024-03-31T15:14:08.242094+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 1], 'acting': [7, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-31T15:14:14.164 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.35 is not active+clean 2024-03-31T15:14:14.164 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.35', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 107, 'state': 'creating+peering', 'last_fresh': '2024-03-31T15:14:08.253324+0000', 'last_change': '2024-03-31T15:14:08.253324+0000', 'last_active': '2024-03-31T15:14:08.242094+0000', 'last_peered': '2024-03-31T15:14:08.242094+0000', 'last_clean': '2024-03-31T15:14:08.242094+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-31T15:14:08.253324+0000', 'last_undegraded': '2024-03-31T15:14:08.253324+0000', 'last_fullsized': '2024-03-31T15:14:08.253324+0000', 'mapping_epoch': 107, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 107, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-31T15:14:08.242094+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-31T15:14:08.242094+0000', 'last_clean_scrub_stamp': '2024-03-31T15:14:08.242094+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 6], 'acting': [4, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-31T15:14:14.164 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.33 is not active+clean 2024-03-31T15:14:14.164 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.33', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 107, 'state': 'creating+peering', 'last_fresh': '2024-03-31T15:14:08.258326+0000', 'last_change': '2024-03-31T15:14:08.258326+0000', 'last_active': '2024-03-31T15:14:08.242094+0000', 'last_peered': '2024-03-31T15:14:08.242094+0000', 'last_clean': '2024-03-31T15:14:08.242094+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-31T15:14:08.258326+0000', 'last_undegraded': '2024-03-31T15:14:08.258326+0000', 'last_fullsized': '2024-03-31T15:14:08.258326+0000', 'mapping_epoch': 107, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 107, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-31T15:14:08.242094+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-31T15:14:08.242094+0000', 'last_clean_scrub_stamp': '2024-03-31T15:14:08.242094+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 2], 'acting': [7, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-31T15:14:14.164 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.32 is not active+clean 2024-03-31T15:14:14.165 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.32', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 107, 'state': 'creating+peering', 'last_fresh': '2024-03-31T15:14:08.257741+0000', 'last_change': '2024-03-31T15:14:08.257741+0000', 'last_active': '2024-03-31T15:14:08.242094+0000', 'last_peered': '2024-03-31T15:14:08.242094+0000', 'last_clean': '2024-03-31T15:14:08.242094+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-31T15:14:08.257741+0000', 'last_undegraded': '2024-03-31T15:14:08.257741+0000', 'last_fullsized': '2024-03-31T15:14:08.257741+0000', 'mapping_epoch': 107, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 107, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-31T15:14:08.242094+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-31T15:14:08.242094+0000', 'last_clean_scrub_stamp': '2024-03-31T15:14:08.242094+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 6], 'acting': [7, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-31T15:14:14.165 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.2f is not active+clean 2024-03-31T15:14:14.165 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.2f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 107, 'state': 'creating+peering', 'last_fresh': '2024-03-31T15:14:08.254889+0000', 'last_change': '2024-03-31T15:14:08.254889+0000', 'last_active': '2024-03-31T15:14:08.242094+0000', 'last_peered': '2024-03-31T15:14:08.242094+0000', 'last_clean': '2024-03-31T15:14:08.242094+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-31T15:14:08.254889+0000', 'last_undegraded': '2024-03-31T15:14:08.254889+0000', 'last_fullsized': '2024-03-31T15:14:08.254889+0000', 'mapping_epoch': 107, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 107, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-31T15:14:08.242094+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-31T15:14:08.242094+0000', 'last_clean_scrub_stamp': '2024-03-31T15:14:08.242094+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 6], 'acting': [4, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-31T15:14:14.165 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.2d is not active+clean 2024-03-31T15:14:14.165 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.2d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 107, 'state': 'creating+peering', 'last_fresh': '2024-03-31T15:14:08.258151+0000', 'last_change': '2024-03-31T15:14:08.258151+0000', 'last_active': '2024-03-31T15:14:08.242094+0000', 'last_peered': '2024-03-31T15:14:08.242094+0000', 'last_clean': '2024-03-31T15:14:08.242094+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-31T15:14:08.258151+0000', 'last_undegraded': '2024-03-31T15:14:08.258151+0000', 'last_fullsized': '2024-03-31T15:14:08.258151+0000', 'mapping_epoch': 107, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 107, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-31T15:14:08.242094+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-31T15:14:08.242094+0000', 'last_clean_scrub_stamp': '2024-03-31T15:14:08.242094+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 2], 'acting': [7, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-31T15:14:14.165 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.2c is not active+clean 2024-03-31T15:14:14.165 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.2c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 107, 'state': 'creating+peering', 'last_fresh': '2024-03-31T15:14:08.253580+0000', 'last_change': '2024-03-31T15:14:08.253580+0000', 'last_active': '2024-03-31T15:14:08.242094+0000', 'last_peered': '2024-03-31T15:14:08.242094+0000', 'last_clean': '2024-03-31T15:14:08.242094+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-31T15:14:08.253580+0000', 'last_undegraded': '2024-03-31T15:14:08.253580+0000', 'last_fullsized': '2024-03-31T15:14:08.253580+0000', 'mapping_epoch': 107, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 107, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-31T15:14:08.242094+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-31T15:14:08.242094+0000', 'last_clean_scrub_stamp': '2024-03-31T15:14:08.242094+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 7], 'acting': [4, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-31T15:14:14.165 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.2b is not active+clean 2024-03-31T15:14:14.165 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.2b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 107, 'state': 'creating+peering', 'last_fresh': '2024-03-31T15:14:08.253223+0000', 'last_change': '2024-03-31T15:14:08.253223+0000', 'last_active': '2024-03-31T15:14:08.242094+0000', 'last_peered': '2024-03-31T15:14:08.242094+0000', 'last_clean': '2024-03-31T15:14:08.242094+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-31T15:14:08.253223+0000', 'last_undegraded': '2024-03-31T15:14:08.253223+0000', 'last_fullsized': '2024-03-31T15:14:08.253223+0000', 'mapping_epoch': 107, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 107, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-31T15:14:08.242094+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-31T15:14:08.242094+0000', 'last_clean_scrub_stamp': '2024-03-31T15:14:08.242094+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 3], 'acting': [4, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-31T15:14:14.165 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.29 is not active+clean 2024-03-31T15:14:14.166 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.29', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 107, 'state': 'creating+peering', 'last_fresh': '2024-03-31T15:14:08.253496+0000', 'last_change': '2024-03-31T15:14:08.253496+0000', 'last_active': '2024-03-31T15:14:08.242094+0000', 'last_peered': '2024-03-31T15:14:08.242094+0000', 'last_clean': '2024-03-31T15:14:08.242094+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-31T15:14:08.253496+0000', 'last_undegraded': '2024-03-31T15:14:08.253496+0000', 'last_fullsized': '2024-03-31T15:14:08.253496+0000', 'mapping_epoch': 107, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 107, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-31T15:14:08.242094+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-31T15:14:08.242094+0000', 'last_clean_scrub_stamp': '2024-03-31T15:14:08.242094+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 7], 'acting': [4, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-31T15:14:14.166 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.27 is not active+clean 2024-03-31T15:14:14.166 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.27', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 107, 'state': 'creating+peering', 'last_fresh': '2024-03-31T15:14:08.254451+0000', 'last_change': '2024-03-31T15:14:08.254451+0000', 'last_active': '2024-03-31T15:14:08.242094+0000', 'last_peered': '2024-03-31T15:14:08.242094+0000', 'last_clean': '2024-03-31T15:14:08.242094+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-31T15:14:08.254451+0000', 'last_undegraded': '2024-03-31T15:14:08.254451+0000', 'last_fullsized': '2024-03-31T15:14:08.254451+0000', 'mapping_epoch': 107, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 107, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-31T15:14:08.242094+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-31T15:14:08.242094+0000', 'last_clean_scrub_stamp': '2024-03-31T15:14:08.242094+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 6], 'acting': [4, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-31T15:14:14.166 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.23 is not active+clean 2024-03-31T15:14:14.166 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.23', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 107, 'state': 'creating+peering', 'last_fresh': '2024-03-31T15:14:08.258913+0000', 'last_change': '2024-03-31T15:14:08.258913+0000', 'last_active': '2024-03-31T15:14:08.242094+0000', 'last_peered': '2024-03-31T15:14:08.242094+0000', 'last_clean': '2024-03-31T15:14:08.242094+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-31T15:14:08.258913+0000', 'last_undegraded': '2024-03-31T15:14:08.258913+0000', 'last_fullsized': '2024-03-31T15:14:08.258913+0000', 'mapping_epoch': 107, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 107, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-31T15:14:08.242094+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-31T15:14:08.242094+0000', 'last_clean_scrub_stamp': '2024-03-31T15:14:08.242094+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 7], 'acting': [4, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-31T15:14:14.166 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.22 is not active+clean 2024-03-31T15:14:14.166 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.22', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 107, 'state': 'creating+peering', 'last_fresh': '2024-03-31T15:14:08.260326+0000', 'last_change': '2024-03-31T15:14:08.260326+0000', 'last_active': '2024-03-31T15:14:08.242094+0000', 'last_peered': '2024-03-31T15:14:08.242094+0000', 'last_clean': '2024-03-31T15:14:08.242094+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-31T15:14:08.260326+0000', 'last_undegraded': '2024-03-31T15:14:08.260326+0000', 'last_fullsized': '2024-03-31T15:14:08.260326+0000', 'mapping_epoch': 107, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 107, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-31T15:14:08.242094+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-31T15:14:08.242094+0000', 'last_clean_scrub_stamp': '2024-03-31T15:14:08.242094+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 3], 'acting': [5, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-31T15:14:14.166 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.1d is not active+clean 2024-03-31T15:14:14.166 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.1d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 107, 'state': 'creating+peering', 'last_fresh': '2024-03-31T15:14:08.257689+0000', 'last_change': '2024-03-31T15:14:08.257689+0000', 'last_active': '2024-03-31T15:14:08.242094+0000', 'last_peered': '2024-03-31T15:14:08.242094+0000', 'last_clean': '2024-03-31T15:14:08.242094+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-31T15:14:08.257689+0000', 'last_undegraded': '2024-03-31T15:14:08.257689+0000', 'last_fullsized': '2024-03-31T15:14:08.257689+0000', 'mapping_epoch': 107, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 107, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-31T15:14:08.242094+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-31T15:14:08.242094+0000', 'last_clean_scrub_stamp': '2024-03-31T15:14:08.242094+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 2], 'acting': [7, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-31T15:14:14.166 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.1b is not active+clean 2024-03-31T15:14:14.166 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.1b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 107, 'state': 'creating+peering', 'last_fresh': '2024-03-31T15:14:08.258473+0000', 'last_change': '2024-03-31T15:14:08.258473+0000', 'last_active': '2024-03-31T15:14:08.242094+0000', 'last_peered': '2024-03-31T15:14:08.242094+0000', 'last_clean': '2024-03-31T15:14:08.242094+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-31T15:14:08.258473+0000', 'last_undegraded': '2024-03-31T15:14:08.258473+0000', 'last_fullsized': '2024-03-31T15:14:08.258473+0000', 'mapping_epoch': 107, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 107, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-31T15:14:08.242094+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-31T15:14:08.242094+0000', 'last_clean_scrub_stamp': '2024-03-31T15:14:08.242094+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 1], 'acting': [4, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-31T15:14:14.167 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.19 is not active+clean 2024-03-31T15:14:14.167 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.19', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 107, 'state': 'creating+peering', 'last_fresh': '2024-03-31T15:14:08.260207+0000', 'last_change': '2024-03-31T15:14:08.260207+0000', 'last_active': '2024-03-31T15:14:08.242094+0000', 'last_peered': '2024-03-31T15:14:08.242094+0000', 'last_clean': '2024-03-31T15:14:08.242094+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-31T15:14:08.260207+0000', 'last_undegraded': '2024-03-31T15:14:08.260207+0000', 'last_fullsized': '2024-03-31T15:14:08.260207+0000', 'mapping_epoch': 107, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 107, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-31T15:14:08.242094+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-31T15:14:08.242094+0000', 'last_clean_scrub_stamp': '2024-03-31T15:14:08.242094+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 6], 'acting': [5, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-31T15:14:14.167 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.16 is not active+clean 2024-03-31T15:14:14.167 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.16', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 107, 'state': 'creating+peering', 'last_fresh': '2024-03-31T15:14:08.259804+0000', 'last_change': '2024-03-31T15:14:08.259804+0000', 'last_active': '2024-03-31T15:14:08.242094+0000', 'last_peered': '2024-03-31T15:14:08.242094+0000', 'last_clean': '2024-03-31T15:14:08.242094+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-31T15:14:08.259804+0000', 'last_undegraded': '2024-03-31T15:14:08.259804+0000', 'last_fullsized': '2024-03-31T15:14:08.259804+0000', 'mapping_epoch': 107, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 107, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-31T15:14:08.242094+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-31T15:14:08.242094+0000', 'last_clean_scrub_stamp': '2024-03-31T15:14:08.242094+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 2], 'acting': [5, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-31T15:14:14.167 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.15 is not active+clean 2024-03-31T15:14:14.167 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.15', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 107, 'state': 'creating+peering', 'last_fresh': '2024-03-31T15:14:08.258737+0000', 'last_change': '2024-03-31T15:14:08.258737+0000', 'last_active': '2024-03-31T15:14:08.242094+0000', 'last_peered': '2024-03-31T15:14:08.242094+0000', 'last_clean': '2024-03-31T15:14:08.242094+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-31T15:14:08.258737+0000', 'last_undegraded': '2024-03-31T15:14:08.258737+0000', 'last_fullsized': '2024-03-31T15:14:08.258737+0000', 'mapping_epoch': 107, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 107, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-31T15:14:08.242094+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-31T15:14:08.242094+0000', 'last_clean_scrub_stamp': '2024-03-31T15:14:08.242094+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 7], 'acting': [5, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-31T15:14:14.167 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.e is not active+clean 2024-03-31T15:14:14.167 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 107, 'state': 'creating+peering', 'last_fresh': '2024-03-31T15:14:08.258499+0000', 'last_change': '2024-03-31T15:14:08.258499+0000', 'last_active': '2024-03-31T15:14:08.242094+0000', 'last_peered': '2024-03-31T15:14:08.242094+0000', 'last_clean': '2024-03-31T15:14:08.242094+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-31T15:14:08.258499+0000', 'last_undegraded': '2024-03-31T15:14:08.258499+0000', 'last_fullsized': '2024-03-31T15:14:08.258499+0000', 'mapping_epoch': 107, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 107, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-31T15:14:08.242094+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-31T15:14:08.242094+0000', 'last_clean_scrub_stamp': '2024-03-31T15:14:08.242094+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0], 'acting': [2, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-31T15:14:14.167 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.c is not active+clean 2024-03-31T15:14:14.167 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 107, 'state': 'creating+peering', 'last_fresh': '2024-03-31T15:14:08.257389+0000', 'last_change': '2024-03-31T15:14:08.257389+0000', 'last_active': '2024-03-31T15:14:08.242094+0000', 'last_peered': '2024-03-31T15:14:08.242094+0000', 'last_clean': '2024-03-31T15:14:08.242094+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-31T15:14:08.257389+0000', 'last_undegraded': '2024-03-31T15:14:08.257389+0000', 'last_fullsized': '2024-03-31T15:14:08.257389+0000', 'mapping_epoch': 107, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 107, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-31T15:14:08.242094+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-31T15:14:08.242094+0000', 'last_clean_scrub_stamp': '2024-03-31T15:14:08.242094+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 0], 'acting': [7, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-31T15:14:14.168 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.d is not active+clean 2024-03-31T15:14:14.168 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 107, 'state': 'creating+peering', 'last_fresh': '2024-03-31T15:14:08.257107+0000', 'last_change': '2024-03-31T15:14:08.257107+0000', 'last_active': '2024-03-31T15:14:08.242094+0000', 'last_peered': '2024-03-31T15:14:08.242094+0000', 'last_clean': '2024-03-31T15:14:08.242094+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-31T15:14:08.257107+0000', 'last_undegraded': '2024-03-31T15:14:08.257107+0000', 'last_fullsized': '2024-03-31T15:14:08.257107+0000', 'mapping_epoch': 107, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 107, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-31T15:14:08.242094+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-31T15:14:08.242094+0000', 'last_clean_scrub_stamp': '2024-03-31T15:14:08.242094+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 5], 'acting': [7, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-31T15:14:14.168 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.a is not active+clean 2024-03-31T15:14:14.168 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 107, 'state': 'creating+peering', 'last_fresh': '2024-03-31T15:14:08.257121+0000', 'last_change': '2024-03-31T15:14:08.257121+0000', 'last_active': '2024-03-31T15:14:08.242094+0000', 'last_peered': '2024-03-31T15:14:08.242094+0000', 'last_clean': '2024-03-31T15:14:08.242094+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-31T15:14:08.257121+0000', 'last_undegraded': '2024-03-31T15:14:08.257121+0000', 'last_fullsized': '2024-03-31T15:14:08.257121+0000', 'mapping_epoch': 107, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 107, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-31T15:14:08.242094+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-31T15:14:08.242094+0000', 'last_clean_scrub_stamp': '2024-03-31T15:14:08.242094+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 4], 'acting': [7, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-31T15:14:14.168 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.8 is not active+clean 2024-03-31T15:14:14.168 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.8', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 107, 'state': 'creating+peering', 'last_fresh': '2024-03-31T15:14:08.257584+0000', 'last_change': '2024-03-31T15:14:08.257584+0000', 'last_active': '2024-03-31T15:14:08.242094+0000', 'last_peered': '2024-03-31T15:14:08.242094+0000', 'last_clean': '2024-03-31T15:14:08.242094+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-31T15:14:08.257584+0000', 'last_undegraded': '2024-03-31T15:14:08.257584+0000', 'last_fullsized': '2024-03-31T15:14:08.257584+0000', 'mapping_epoch': 107, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 107, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-31T15:14:08.242094+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-31T15:14:08.242094+0000', 'last_clean_scrub_stamp': '2024-03-31T15:14:08.242094+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 2], 'acting': [7, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-31T15:14:14.168 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.2 is not active+clean 2024-03-31T15:14:14.168 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.2', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 107, 'state': 'creating+peering', 'last_fresh': '2024-03-31T15:14:08.254000+0000', 'last_change': '2024-03-31T15:14:08.254000+0000', 'last_active': '2024-03-31T15:14:08.242094+0000', 'last_peered': '2024-03-31T15:14:08.242094+0000', 'last_clean': '2024-03-31T15:14:08.242094+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-31T15:14:08.254000+0000', 'last_undegraded': '2024-03-31T15:14:08.254000+0000', 'last_fullsized': '2024-03-31T15:14:08.254000+0000', 'mapping_epoch': 107, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 107, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-31T15:14:08.242094+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-31T15:14:08.242094+0000', 'last_clean_scrub_stamp': '2024-03-31T15:14:08.242094+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 5], 'acting': [4, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-31T15:14:14.168 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.3 is not active+clean 2024-03-31T15:14:14.168 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.3', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 107, 'state': 'creating+peering', 'last_fresh': '2024-03-31T15:14:08.255261+0000', 'last_change': '2024-03-31T15:14:08.255261+0000', 'last_active': '2024-03-31T15:14:08.242094+0000', 'last_peered': '2024-03-31T15:14:08.242094+0000', 'last_clean': '2024-03-31T15:14:08.242094+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-31T15:14:08.255261+0000', 'last_undegraded': '2024-03-31T15:14:08.255261+0000', 'last_fullsized': '2024-03-31T15:14:08.255261+0000', 'mapping_epoch': 107, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 107, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-31T15:14:08.242094+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-31T15:14:08.242094+0000', 'last_clean_scrub_stamp': '2024-03-31T15:14:08.242094+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 2], 'acting': [4, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-31T15:14:14.169 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.0 is not active+clean 2024-03-31T15:14:14.169 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.0', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 107, 'state': 'creating+peering', 'last_fresh': '2024-03-31T15:14:08.257010+0000', 'last_change': '2024-03-31T15:14:08.257010+0000', 'last_active': '2024-03-31T15:14:08.242094+0000', 'last_peered': '2024-03-31T15:14:08.242094+0000', 'last_clean': '2024-03-31T15:14:08.242094+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-31T15:14:08.257010+0000', 'last_undegraded': '2024-03-31T15:14:08.257010+0000', 'last_fullsized': '2024-03-31T15:14:08.257010+0000', 'mapping_epoch': 107, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 107, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-31T15:14:08.242094+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-31T15:14:08.242094+0000', 'last_clean_scrub_stamp': '2024-03-31T15:14:08.242094+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 6], 'acting': [7, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-31T15:14:14.169 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-03-31T15:14:14.225 INFO:tasks.ceph.mds.c.smithi016.stderr:2024-03-31T15:14:14.234+0000 7f62e9600640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-31T15:14:14.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:14.350+0000 7fb42bc00640 1 Processor -- start 2024-03-31T15:14:14.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:14.350+0000 7fb42bc00640 1 -- start start 2024-03-31T15:14:14.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:14.350+0000 7fb42bc00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fb424070350 0x7fb424160860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:14.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:14.350+0000 7fb42bc00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb42415c890 0x7fb4240cea50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:14.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:14.350+0000 7fb42bc00640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb424070a10 con 0x7fb424145d50 2024-03-31T15:14:14.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:14.350+0000 7fb42bc00640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7fb4240ce5c0 con 0x7fb42415c890 2024-03-31T15:14:14.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:14.350+0000 7fb42bc00640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7fb4240cd9b0 con 0x7fb424070350 2024-03-31T15:14:14.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:14.350+0000 7fb429c00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb42415c890 0x7fb4240cea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:14.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:14.350+0000 7fb429c00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb42415c890 0x7fb4240cea50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50158/0 (socket says 172.21.15.16:50158) 2024-03-31T15:14:14.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:14.350+0000 7fb429c00640 1 -- 172.21.15.16:0/720388066 learned_addr learned my addr 172.21.15.16:0/720388066 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:14:14.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:14.350+0000 7fb423e00640 1 --2- 172.21.15.16:0/720388066 >> v2:172.21.15.89:3300/0 conn(0x7fb424070350 0x7fb424160860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:14.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:14.350+0000 7fb429c00640 1 -- 172.21.15.16:0/720388066 >> v2:172.21.15.89:3300/0 conn(0x7fb424070350 msgr2=0x7fb424160860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:14:14.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:14.350+0000 7fb429c00640 1 --2- 172.21.15.16:0/720388066 >> v2:172.21.15.89:3300/0 conn(0x7fb424070350 0x7fb424160860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:14.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:14.350+0000 7fb429c00640 1 -- 172.21.15.16:0/720388066 >> v1:172.21.15.89:6789/0 conn(0x7fb424145d50 legacy=0x7fb42406fc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:14:14.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:14.350+0000 7fb429c00640 1 -- 172.21.15.16:0/720388066 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb4240cdc90 con 0x7fb42415c890 2024-03-31T15:14:14.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:14.350+0000 7fb423e00640 1 --2- 172.21.15.16:0/720388066 >> v2:172.21.15.89:3300/0 conn(0x7fb424070350 0x7fb424160860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:14:14.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:14.354+0000 7fb429c00640 1 --2- 172.21.15.16:0/720388066 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb42415c890 0x7fb4240cea50 secure :-1 s=READY pgs=1658 cs=0 l=1 rev1=1 crypto rx=0x7fb4240d08a0 tx=0x7fb41c01c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c5f7dd440fe31d36 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:14.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:14.354+0000 7fb423400640 1 -- 172.21.15.16:0/720388066 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb41c0a5880 con 0x7fb42415c890 2024-03-31T15:14:14.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:14.354+0000 7fb423400640 1 -- 172.21.15.16:0/720388066 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb41c0a5a20 con 0x7fb42415c890 2024-03-31T15:14:14.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:14.354+0000 7fb423400640 1 -- 172.21.15.16:0/720388066 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb41c0a5df0 con 0x7fb42415c890 2024-03-31T15:14:14.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:14.354+0000 7fb42bc00640 1 -- 172.21.15.16:0/720388066 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb42415c890 msgr2=0x7fb4240cea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:14.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:14.354+0000 7fb42bc00640 1 --2- 172.21.15.16:0/720388066 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb42415c890 0x7fb4240cea50 secure :-1 s=READY pgs=1658 cs=0 l=1 rev1=1 crypto rx=0x7fb4240d08a0 tx=0x7fb41c01c9c0 comp rx=0 tx=0).stop 2024-03-31T15:14:14.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:14.354+0000 7fb42bc00640 1 -- 172.21.15.16:0/720388066 shutdown_connections 2024-03-31T15:14:14.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:14.354+0000 7fb42bc00640 1 --2- 172.21.15.16:0/720388066 >> v2:172.21.15.89:3300/0 conn(0x7fb424070350 0x7fb424160860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:14.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:14.354+0000 7fb42bc00640 1 --2- 172.21.15.16:0/720388066 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb42415c890 0x7fb4240cea50 unknown :-1 s=CLOSED pgs=1658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:14.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:14.354+0000 7fb42bc00640 1 -- 172.21.15.16:0/720388066 >> 172.21.15.16:0/720388066 conn(0x7fb42406e670 msgr2=0x7fb42406ea50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:14:14.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:14.354+0000 7fb42bc00640 1 -- 172.21.15.16:0/720388066 shutdown_connections 2024-03-31T15:14:14.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:14.354+0000 7fb42bc00640 1 -- 172.21.15.16:0/720388066 wait complete. 2024-03-31T15:14:14.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:14.358+0000 7fb42bc00640 1 Processor -- start 2024-03-31T15:14:14.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:14.358+0000 7fb42bc00640 1 -- start start 2024-03-31T15:14:14.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:14.358+0000 7fb42bc00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb424070350 0x7fb42415f340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:14.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:14.358+0000 7fb42bc00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb424145d50 0x7fb424157ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:14.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:14.358+0000 7fb42bc00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb42415c890 0x7fb42415d6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:14.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:14.358+0000 7fb42bc00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fb424169520 con 0x7fb42415c890 2024-03-31T15:14:14.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:14.358+0000 7fb42bc00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fb4241693a0 con 0x7fb424070350 2024-03-31T15:14:14.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:14.358+0000 7fb42bc00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fb4241696a0 con 0x7fb424145d50 2024-03-31T15:14:14.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:14.358+0000 7fb429c00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb42415c890 0x7fb42415d6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:14.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:14.358+0000 7fb429c00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb42415c890 0x7fb42415d6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50164/0 (socket says 172.21.15.16:50164) 2024-03-31T15:14:14.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:14.358+0000 7fb429c00640 1 -- 172.21.15.16:0/294017035 learned_addr learned my addr 172.21.15.16:0/294017035 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:14:14.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:14.358+0000 7fb423e00640 1 --2- 172.21.15.16:0/294017035 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb424145d50 0x7fb424157ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:14.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:14.358+0000 7fb429200640 1 --2- 172.21.15.16:0/294017035 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb424070350 0x7fb42415f340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:14.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:14.358+0000 7fb429c00640 1 -- 172.21.15.16:0/294017035 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb424145d50 msgr2=0x7fb424157ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:14.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:14.358+0000 7fb429c00640 1 --2- 172.21.15.16:0/294017035 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb424145d50 0x7fb424157ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:14.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:14.358+0000 7fb429c00640 1 -- 172.21.15.16:0/294017035 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb424070350 msgr2=0x7fb42415f340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:14.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:14.358+0000 7fb429c00640 1 --2- 172.21.15.16:0/294017035 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb424070350 0x7fb42415f340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:14.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:14.358+0000 7fb429c00640 1 -- 172.21.15.16:0/294017035 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb42413d110 con 0x7fb42415c890 2024-03-31T15:14:14.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:14.358+0000 7fb429c00640 1 --2- 172.21.15.16:0/294017035 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb42415c890 0x7fb42415d6c0 secure :-1 s=READY pgs=1659 cs=0 l=1 rev1=1 crypto rx=0x7fb41c01a5f0 tx=0x7fb41c019760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:14.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:14.358+0000 7fb420c00640 1 -- 172.21.15.16:0/294017035 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb41c0b8670 con 0x7fb42415c890 2024-03-31T15:14:14.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:14.358+0000 7fb420c00640 1 -- 172.21.15.16:0/294017035 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb41c0be070 con 0x7fb42415c890 2024-03-31T15:14:14.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:14.358+0000 7fb420c00640 1 -- 172.21.15.16:0/294017035 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb41c0b8a90 con 0x7fb42415c890 2024-03-31T15:14:14.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:14.358+0000 7fb42bc00640 1 -- 172.21.15.16:0/294017035 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb42415e2a0 con 0x7fb42415c890 2024-03-31T15:14:14.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:14.358+0000 7fb42bc00640 1 -- 172.21.15.16:0/294017035 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb42415e860 con 0x7fb42415c890 2024-03-31T15:14:14.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:14.358+0000 7fb42bc00640 1 -- 172.21.15.16:0/294017035 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb424069950 con 0x7fb42415c890 2024-03-31T15:14:14.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:14.362+0000 7fb420c00640 1 -- 172.21.15.16:0/294017035 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fb41c0b8c30 con 0x7fb42415c890 2024-03-31T15:14:14.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:14.362+0000 7fb420c00640 1 --2- 172.21.15.16:0/294017035 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fb404068380 0x7fb40406a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:14.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:14.362+0000 7fb420c00640 1 -- 172.21.15.16:0/294017035 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 5709+0+0 (secure 0 0 0) 0x7fb41c122550 con 0x7fb42415c890 2024-03-31T15:14:14.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:14.362+0000 7fb429200640 1 --2- 172.21.15.16:0/294017035 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fb404068380 0x7fb40406a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:14.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:14.362+0000 7fb429200640 1 --2- 172.21.15.16:0/294017035 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fb404068380 0x7fb40406a840 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7fb418001810 tx=0x7fb418009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:14.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:14.370+0000 7fb420c00640 1 -- 172.21.15.16:0/294017035 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fb41c0165e0 con 0x7fb42415c890 2024-03-31T15:14:14.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:14.590+0000 7fb42bc00640 1 -- 172.21.15.16:0/294017035 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) -- 0x7fb4240b5980 con 0x7fb42415c890 2024-03-31T15:14:14.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:14.590+0000 7fb420c00640 1 -- 172.21.15.16:0/294017035 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) ==== 72+0+1101 (secure 0 0 0) 0x7fb41c0f25a0 con 0x7fb42415c890 2024-03-31T15:14:14.584 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:14:14.584 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"286eb5da-fb47-4550-b290-d9519c13b9a5","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":663,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":109,"num_osds":8,"num_up_osds":8,"osd_up_since":1711897402,"num_in_osds":8,"osd_in_since":1711897393,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":68},{"state_name":"creating+peering","count":29}],"num_pgs":97,"num_pools":3,"num_objects":15,"data_bytes":592196,"bytes_used":271597568,"bytes_avail":772822515712,"bytes_total":773094113280,"inactive_pgs_ratio":0.29896906018257141,"write_bytes_sec":2444,"read_op_per_sec":0,"write_op_per_sec":3},"fsmap":{"epoch":99,"id":8,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":8,"rank":0,"name":"c","status":"up:active","gid":9195}],"up:standby":3},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":26,"modified":"2024-03-31T15:13:14.677309+0000","services":{}},"progress_events":{}} 2024-03-31T15:14:14.586 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:14.594+0000 7fb42bc00640 1 -- 172.21.15.16:0/294017035 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fb404068380 msgr2=0x7fb40406a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:14.586 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:14.594+0000 7fb42bc00640 1 --2- 172.21.15.16:0/294017035 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fb404068380 0x7fb40406a840 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7fb418001810 tx=0x7fb418009040 comp rx=0 tx=0).stop 2024-03-31T15:14:14.586 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:14.594+0000 7fb42bc00640 1 -- 172.21.15.16:0/294017035 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb42415c890 msgr2=0x7fb42415d6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:14.586 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:14.594+0000 7fb42bc00640 1 --2- 172.21.15.16:0/294017035 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb42415c890 0x7fb42415d6c0 secure :-1 s=READY pgs=1659 cs=0 l=1 rev1=1 crypto rx=0x7fb41c01a5f0 tx=0x7fb41c019760 comp rx=0 tx=0).stop 2024-03-31T15:14:14.586 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:14.594+0000 7fb42bc00640 1 -- 172.21.15.16:0/294017035 shutdown_connections 2024-03-31T15:14:14.586 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:14.594+0000 7fb42bc00640 1 --2- 172.21.15.16:0/294017035 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fb404068380 0x7fb40406a840 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:14.587 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:14.594+0000 7fb42bc00640 1 --2- 172.21.15.16:0/294017035 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb42415c890 0x7fb42415d6c0 unknown :-1 s=CLOSED pgs=1659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:14.587 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:14.594+0000 7fb42bc00640 1 --2- 172.21.15.16:0/294017035 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb424145d50 0x7fb424157ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:14.587 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:14.594+0000 7fb42bc00640 1 --2- 172.21.15.16:0/294017035 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb424070350 0x7fb42415f340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:14.587 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:14.594+0000 7fb42bc00640 1 -- 172.21.15.16:0/294017035 >> 172.21.15.16:0/294017035 conn(0x7fb42406e670 msgr2=0x7fb424144ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:14:14.587 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:14.594+0000 7fb42bc00640 1 -- 172.21.15.16:0/294017035 shutdown_connections 2024-03-31T15:14:14.587 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:14.594+0000 7fb42bc00640 1 -- 172.21.15.16:0/294017035 wait complete. 2024-03-31T15:14:14.615 INFO:tasks.cephfs.filesystem.ceph_manager:no progress seen, keeping timeout for now 2024-03-31T15:14:14.616 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-31T15:14:14.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:14.802+0000 7f02a9800640 1 Processor -- start 2024-03-31T15:14:14.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:14.802+0000 7f02a9800640 1 -- start start 2024-03-31T15:14:14.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:14.802+0000 7f02a9800640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f02a406dbc0 0x7f02a415f650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:14.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:14.802+0000 7f02a9800640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f02a4070030 0x7f02a4162180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:14.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:14.802+0000 7f02a9800640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f02a40cbe90 con 0x7f02a40cb440 2024-03-31T15:14:14.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:14.802+0000 7f02a9800640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f02a415fb90 con 0x7f02a406dbc0 2024-03-31T15:14:14.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:14.802+0000 7f02a9800640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f02a415f1c0 con 0x7f02a4070030 2024-03-31T15:14:14.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:14.802+0000 7f02a2a00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f02a406dbc0 0x7f02a415f650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:14.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:14.802+0000 7f02a2a00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f02a406dbc0 0x7f02a415f650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50172/0 (socket says 172.21.15.16:50172) 2024-03-31T15:14:14.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:14.802+0000 7f02a2a00640 1 -- 172.21.15.16:0/1906260741 learned_addr learned my addr 172.21.15.16:0/1906260741 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:14:14.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:14.806+0000 7f02a3400640 1 --2- 172.21.15.16:0/1906260741 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f02a4070030 0x7f02a4162180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:14.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:14.806+0000 7f02a2a00640 1 -- 172.21.15.16:0/1906260741 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f02a4070030 msgr2=0x7f02a4162180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:14:14.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:14.806+0000 7f02a2a00640 1 --2- 172.21.15.16:0/1906260741 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f02a4070030 0x7f02a4162180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:14.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:14.806+0000 7f02a2a00640 1 -- 172.21.15.16:0/1906260741 >> v1:172.21.15.89:6789/0 conn(0x7f02a40cb440 legacy=0x7f02a406f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:14.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:14.806+0000 7f02a2a00640 1 -- 172.21.15.16:0/1906260741 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02a40cc170 con 0x7f02a406dbc0 2024-03-31T15:14:14.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:14.806+0000 7f02a2a00640 1 --2- 172.21.15.16:0/1906260741 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f02a406dbc0 0x7f02a415f650 secure :-1 s=READY pgs=1660 cs=0 l=1 rev1=1 crypto rx=0x7f0294009e50 tx=0x7f029401c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c206a97f7eb58e9 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:14.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:14.806+0000 7f02a2000640 1 -- 172.21.15.16:0/1906260741 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02940a5740 con 0x7f02a406dbc0 2024-03-31T15:14:14.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:14.806+0000 7f02a2000640 1 -- 172.21.15.16:0/1906260741 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f02940a58e0 con 0x7f02a406dbc0 2024-03-31T15:14:14.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:14.806+0000 7f02a2000640 1 -- 172.21.15.16:0/1906260741 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02940a5cb0 con 0x7f02a406dbc0 2024-03-31T15:14:14.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:14.806+0000 7f02a9800640 1 -- 172.21.15.16:0/1906260741 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f02a406dbc0 msgr2=0x7f02a415f650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:14.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:14.806+0000 7f02a9800640 1 --2- 172.21.15.16:0/1906260741 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f02a406dbc0 0x7f02a415f650 secure :-1 s=READY pgs=1660 cs=0 l=1 rev1=1 crypto rx=0x7f0294009e50 tx=0x7f029401c970 comp rx=0 tx=0).stop 2024-03-31T15:14:14.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:14.810+0000 7f02a9800640 1 -- 172.21.15.16:0/1906260741 shutdown_connections 2024-03-31T15:14:14.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:14.810+0000 7f02a9800640 1 --2- 172.21.15.16:0/1906260741 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f02a4070030 0x7f02a4162180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:14.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:14.810+0000 7f02a9800640 1 --2- 172.21.15.16:0/1906260741 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f02a406dbc0 0x7f02a415f650 unknown :-1 s=CLOSED pgs=1660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:14.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:14.810+0000 7f02a9800640 1 -- 172.21.15.16:0/1906260741 >> 172.21.15.16:0/1906260741 conn(0x7f02a4067330 msgr2=0x7f02a4067740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:14:14.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:14.810+0000 7f02a9800640 1 -- 172.21.15.16:0/1906260741 shutdown_connections 2024-03-31T15:14:14.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:14.810+0000 7f02a9800640 1 -- 172.21.15.16:0/1906260741 wait complete. 2024-03-31T15:14:14.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:14.810+0000 7f02a9800640 1 Processor -- start 2024-03-31T15:14:14.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:14.810+0000 7f02a9800640 1 -- start start 2024-03-31T15:14:14.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:14.810+0000 7f02a9800640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f02a406dbc0 0x7f02a415aa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:14.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:14.810+0000 7f02a9800640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f02a4070030 0x7f02a4150570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:14.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:14.810+0000 7f02a9800640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f02a40cb440 0x7f02a4150cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:14.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:14.810+0000 7f02a9800640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f02a406cad0 con 0x7f02a4070030 2024-03-31T15:14:14.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:14.810+0000 7f02a9800640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f02a406c950 con 0x7f02a40cb440 2024-03-31T15:14:14.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:14.810+0000 7f02a9800640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f02a416b520 con 0x7f02a406dbc0 2024-03-31T15:14:14.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:14.810+0000 7f029a000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f02a4070030 0x7f02a4150570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:14.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:14.810+0000 7f029a000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f02a4070030 0x7f02a4150570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50188/0 (socket says 172.21.15.16:50188) 2024-03-31T15:14:14.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:14.810+0000 7f029a000640 1 -- 172.21.15.16:0/1793137186 learned_addr learned my addr 172.21.15.16:0/1793137186 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:14:14.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:14.810+0000 7f02a3400640 1 --2- 172.21.15.16:0/1793137186 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f02a40cb440 0x7f02a4150cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:14.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:14.810+0000 7f02a2a00640 1 --2- 172.21.15.16:0/1793137186 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f02a406dbc0 0x7f02a415aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:14.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:14.810+0000 7f029a000640 1 -- 172.21.15.16:0/1793137186 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f02a406dbc0 msgr2=0x7f02a415aa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:14.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:14.810+0000 7f029a000640 1 --2- 172.21.15.16:0/1793137186 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f02a406dbc0 0x7f02a415aa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:14.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:14.810+0000 7f029a000640 1 -- 172.21.15.16:0/1793137186 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f02a40cb440 msgr2=0x7f02a4150cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:14.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:14.810+0000 7f029a000640 1 --2- 172.21.15.16:0/1793137186 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f02a40cb440 0x7f02a4150cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:14.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:14.810+0000 7f029a000640 1 -- 172.21.15.16:0/1793137186 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02a4150ab0 con 0x7f02a4070030 2024-03-31T15:14:14.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:14.810+0000 7f029a000640 1 --2- 172.21.15.16:0/1793137186 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f02a4070030 0x7f02a4150570 secure :-1 s=READY pgs=1661 cs=0 l=1 rev1=1 crypto rx=0x7f028c00c3a0 tx=0x7f028c00c7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:14.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:14.810+0000 7f029b400640 1 -- 172.21.15.16:0/1793137186 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f028c0052d0 con 0x7f02a4070030 2024-03-31T15:14:14.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:14.814+0000 7f029b400640 1 -- 172.21.15.16:0/1793137186 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f028c005470 con 0x7f02a4070030 2024-03-31T15:14:14.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:14.814+0000 7f02a9800640 1 -- 172.21.15.16:0/1793137186 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f02a415b0d0 con 0x7f02a4070030 2024-03-31T15:14:14.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:14.814+0000 7f029b400640 1 -- 172.21.15.16:0/1793137186 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f028c007260 con 0x7f02a4070030 2024-03-31T15:14:14.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:14.814+0000 7f02a9800640 1 -- 172.21.15.16:0/1793137186 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f02a415b690 con 0x7f02a4070030 2024-03-31T15:14:14.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:14.814+0000 7f02a9800640 1 -- 172.21.15.16:0/1793137186 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f02a4069950 con 0x7f02a4070030 2024-03-31T15:14:14.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:14.814+0000 7f029b400640 1 -- 172.21.15.16:0/1793137186 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f028c007400 con 0x7f02a4070030 2024-03-31T15:14:14.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:14.814+0000 7f029b400640 1 --2- 172.21.15.16:0/1793137186 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f0280068380 0x7f028006a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:14.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:14.814+0000 7f029b400640 1 -- 172.21.15.16:0/1793137186 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 5709+0+0 (secure 0 0 0) 0x7f028c085d00 con 0x7f02a4070030 2024-03-31T15:14:14.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:14.814+0000 7f02a2a00640 1 --2- 172.21.15.16:0/1793137186 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f0280068380 0x7f028006a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:14.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:14.818+0000 7f02a2a00640 1 --2- 172.21.15.16:0/1793137186 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f0280068380 0x7f028006a840 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f02940ae000 tx=0x7f02940a7070 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:14.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:14.826+0000 7f029b400640 1 -- 172.21.15.16:0/1793137186 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f02a4150ab0 con 0x7f02a4070030 2024-03-31T15:14:14.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:14.986+0000 7f02a9800640 1 -- 172.21.15.16:0/1793137186 --> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f02a40cf380 con 0x7f0280068380 2024-03-31T15:14:14.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:14.994+0000 7f029b400640 1 -- 172.21.15.16:0/1793137186 <== mgr.4100 v2:172.21.15.89:6832/1549407273 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252482 (secure 0 0 0) 0x7f02a40cf380 con 0x7f0280068380 2024-03-31T15:14:14.990 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:14:14.991 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-03-31T15:14:14.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:14.998+0000 7f02a9800640 1 -- 172.21.15.16:0/1793137186 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f0280068380 msgr2=0x7f028006a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:14.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:14.998+0000 7f02a9800640 1 --2- 172.21.15.16:0/1793137186 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f0280068380 0x7f028006a840 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f02940ae000 tx=0x7f02940a7070 comp rx=0 tx=0).stop 2024-03-31T15:14:14.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:14.998+0000 7f02a9800640 1 -- 172.21.15.16:0/1793137186 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f02a4070030 msgr2=0x7f02a4150570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:14.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:14.998+0000 7f02a9800640 1 --2- 172.21.15.16:0/1793137186 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f02a4070030 0x7f02a4150570 secure :-1 s=READY pgs=1661 cs=0 l=1 rev1=1 crypto rx=0x7f028c00c3a0 tx=0x7f028c00c7a0 comp rx=0 tx=0).stop 2024-03-31T15:14:14.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:14.998+0000 7f02a9800640 1 -- 172.21.15.16:0/1793137186 shutdown_connections 2024-03-31T15:14:14.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:14.998+0000 7f02a9800640 1 --2- 172.21.15.16:0/1793137186 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f0280068380 0x7f028006a840 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:14.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:14.998+0000 7f02a9800640 1 --2- 172.21.15.16:0/1793137186 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f02a40cb440 0x7f02a4150cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:14.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:14.998+0000 7f02a9800640 1 --2- 172.21.15.16:0/1793137186 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f02a4070030 0x7f02a4150570 unknown :-1 s=CLOSED pgs=1661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:14.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:14.998+0000 7f02a9800640 1 --2- 172.21.15.16:0/1793137186 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f02a406dbc0 0x7f02a415aa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:14.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:14.998+0000 7f02a9800640 1 -- 172.21.15.16:0/1793137186 >> 172.21.15.16:0/1793137186 conn(0x7f02a4067330 msgr2=0x7f02a4161270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:14:14.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:14.998+0000 7f02a9800640 1 -- 172.21.15.16:0/1793137186 shutdown_connections 2024-03-31T15:14:14.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:14.998+0000 7f02a9800640 1 -- 172.21.15.16:0/1793137186 wait complete. 2024-03-31T15:14:15.011 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":428,"stamp":"2024-03-31T15:14:14.694003+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":172,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":265372,"kb_used_data":11752,"kb_used_omap":12,"kb_used_meta":253491,"kb_avail":754709348,"statfs":{"total":773094113280,"available":772822372352,"internally_reserved":0,"allocated":12034048,"data_stored":5858140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":259575344},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":29,"apply_latency_ms":29,"commit_latency_ns":29000000,"apply_latency_ns":29000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.442217"},"pg_stats":[{"pgid":"17.3f","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:10.892463+0000","last_change":"2024-03-31T15:14:09.277746+0000","last_active":"2024-03-31T15:14:10.892463+0000","last_peered":"2024-03-31T15:14:10.892463+0000","last_clean":"2024-03-31T15:14:10.892463+0000","last_became_active":"2024-03-31T15:14:09.277207+0000","last_became_peered":"2024-03-31T15:14:09.277207+0000","last_unstale":"2024-03-31T15:14:10.892463+0000","last_undegraded":"2024-03-31T15:14:10.892463+0000","last_fullsized":"2024-03-31T15:14:10.892463+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T17:42:57.076247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3e","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:10.279940+0000","last_change":"2024-03-31T15:14:09.287185+0000","last_active":"2024-03-31T15:14:10.279940+0000","last_peered":"2024-03-31T15:14:10.279940+0000","last_clean":"2024-03-31T15:14:10.279940+0000","last_became_active":"2024-03-31T15:14:09.286886+0000","last_became_peered":"2024-03-31T15:14:09.286886+0000","last_unstale":"2024-03-31T15:14:10.279940+0000","last_undegraded":"2024-03-31T15:14:10.279940+0000","last_fullsized":"2024-03-31T15:14:10.279940+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:49:32.932894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.3d","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:10.892344+0000","last_change":"2024-03-31T15:14:09.287821+0000","last_active":"2024-03-31T15:14:10.892344+0000","last_peered":"2024-03-31T15:14:10.892344+0000","last_clean":"2024-03-31T15:14:10.892344+0000","last_became_active":"2024-03-31T15:14:09.287214+0000","last_became_peered":"2024-03-31T15:14:09.287214+0000","last_unstale":"2024-03-31T15:14:10.892344+0000","last_undegraded":"2024-03-31T15:14:10.892344+0000","last_fullsized":"2024-03-31T15:14:10.892344+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T17:45:10.698978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.3c","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:10.892010+0000","last_change":"2024-03-31T15:14:09.276706+0000","last_active":"2024-03-31T15:14:10.892010+0000","last_peered":"2024-03-31T15:14:10.892010+0000","last_clean":"2024-03-31T15:14:10.892010+0000","last_became_active":"2024-03-31T15:14:09.275545+0000","last_became_peered":"2024-03-31T15:14:09.275545+0000","last_unstale":"2024-03-31T15:14:10.892010+0000","last_undegraded":"2024-03-31T15:14:10.892010+0000","last_fullsized":"2024-03-31T15:14:10.892010+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T19:37:12.397888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3b","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:11.390113+0000","last_change":"2024-03-31T15:14:09.276514+0000","last_active":"2024-03-31T15:14:11.390113+0000","last_peered":"2024-03-31T15:14:11.390113+0000","last_clean":"2024-03-31T15:14:11.390113+0000","last_became_active":"2024-03-31T15:14:09.276241+0000","last_became_peered":"2024-03-31T15:14:09.276241+0000","last_unstale":"2024-03-31T15:14:11.390113+0000","last_undegraded":"2024-03-31T15:14:11.390113+0000","last_fullsized":"2024-03-31T15:14:11.390113+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:42:03.309978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.3a","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:11.390994+0000","last_change":"2024-03-31T15:14:09.283260+0000","last_active":"2024-03-31T15:14:11.390994+0000","last_peered":"2024-03-31T15:14:11.390994+0000","last_clean":"2024-03-31T15:14:11.390994+0000","last_became_active":"2024-03-31T15:14:09.282995+0000","last_became_peered":"2024-03-31T15:14:09.282995+0000","last_unstale":"2024-03-31T15:14:11.390994+0000","last_undegraded":"2024-03-31T15:14:11.390994+0000","last_fullsized":"2024-03-31T15:14:11.390994+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:27:30.858064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.39","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:10.276772+0000","last_change":"2024-03-31T15:14:09.275503+0000","last_active":"2024-03-31T15:14:10.276772+0000","last_peered":"2024-03-31T15:14:10.276772+0000","last_clean":"2024-03-31T15:14:10.276772+0000","last_became_active":"2024-03-31T15:14:09.275190+0000","last_became_peered":"2024-03-31T15:14:09.275190+0000","last_unstale":"2024-03-31T15:14:10.276772+0000","last_undegraded":"2024-03-31T15:14:10.276772+0000","last_fullsized":"2024-03-31T15:14:10.276772+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:22:10.709767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.38","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:10.892255+0000","last_change":"2024-03-31T15:14:09.290122+0000","last_active":"2024-03-31T15:14:10.892255+0000","last_peered":"2024-03-31T15:14:10.892255+0000","last_clean":"2024-03-31T15:14:10.892255+0000","last_became_active":"2024-03-31T15:14:09.289694+0000","last_became_peered":"2024-03-31T15:14:09.289694+0000","last_unstale":"2024-03-31T15:14:10.892255+0000","last_undegraded":"2024-03-31T15:14:10.892255+0000","last_fullsized":"2024-03-31T15:14:10.892255+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:54:09.048015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.37","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:10.892467+0000","last_change":"2024-03-31T15:14:09.284680+0000","last_active":"2024-03-31T15:14:10.892467+0000","last_peered":"2024-03-31T15:14:10.892467+0000","last_clean":"2024-03-31T15:14:10.892467+0000","last_became_active":"2024-03-31T15:14:09.282533+0000","last_became_peered":"2024-03-31T15:14:09.282533+0000","last_unstale":"2024-03-31T15:14:10.892467+0000","last_undegraded":"2024-03-31T15:14:10.892467+0000","last_fullsized":"2024-03-31T15:14:10.892467+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:04:35.760087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.36","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:10.892246+0000","last_change":"2024-03-31T15:14:09.290368+0000","last_active":"2024-03-31T15:14:10.892246+0000","last_peered":"2024-03-31T15:14:10.892246+0000","last_clean":"2024-03-31T15:14:10.892246+0000","last_became_active":"2024-03-31T15:14:09.290077+0000","last_became_peered":"2024-03-31T15:14:09.290077+0000","last_unstale":"2024-03-31T15:14:10.892246+0000","last_undegraded":"2024-03-31T15:14:10.892246+0000","last_fullsized":"2024-03-31T15:14:10.892246+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:23:53.326762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.35","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:10.279841+0000","last_change":"2024-03-31T15:14:09.274713+0000","last_active":"2024-03-31T15:14:10.279841+0000","last_peered":"2024-03-31T15:14:10.279841+0000","last_clean":"2024-03-31T15:14:10.279841+0000","last_became_active":"2024-03-31T15:14:09.274389+0000","last_became_peered":"2024-03-31T15:14:09.274389+0000","last_unstale":"2024-03-31T15:14:10.279841+0000","last_undegraded":"2024-03-31T15:14:10.279841+0000","last_fullsized":"2024-03-31T15:14:10.279841+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:48:41.198019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.34","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:10.892056+0000","last_change":"2024-03-31T15:14:09.276847+0000","last_active":"2024-03-31T15:14:10.892056+0000","last_peered":"2024-03-31T15:14:10.892056+0000","last_clean":"2024-03-31T15:14:10.892056+0000","last_became_active":"2024-03-31T15:14:09.275249+0000","last_became_peered":"2024-03-31T15:14:09.275249+0000","last_unstale":"2024-03-31T15:14:10.892056+0000","last_undegraded":"2024-03-31T15:14:10.892056+0000","last_fullsized":"2024-03-31T15:14:10.892056+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T16:56:09.377594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.33","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:10.892611+0000","last_change":"2024-03-31T15:14:09.292405+0000","last_active":"2024-03-31T15:14:10.892611+0000","last_peered":"2024-03-31T15:14:10.892611+0000","last_clean":"2024-03-31T15:14:10.892611+0000","last_became_active":"2024-03-31T15:14:09.292107+0000","last_became_peered":"2024-03-31T15:14:09.292107+0000","last_unstale":"2024-03-31T15:14:10.892611+0000","last_undegraded":"2024-03-31T15:14:10.892611+0000","last_fullsized":"2024-03-31T15:14:10.892611+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:55:14.681666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.32","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:10.892110+0000","last_change":"2024-03-31T15:14:09.286608+0000","last_active":"2024-03-31T15:14:10.892110+0000","last_peered":"2024-03-31T15:14:10.892110+0000","last_clean":"2024-03-31T15:14:10.892110+0000","last_became_active":"2024-03-31T15:14:09.285944+0000","last_became_peered":"2024-03-31T15:14:09.285944+0000","last_unstale":"2024-03-31T15:14:10.892110+0000","last_undegraded":"2024-03-31T15:14:10.892110+0000","last_fullsized":"2024-03-31T15:14:10.892110+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:51:29.224313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.31","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:11.390625+0000","last_change":"2024-03-31T15:14:09.285784+0000","last_active":"2024-03-31T15:14:11.390625+0000","last_peered":"2024-03-31T15:14:11.390625+0000","last_clean":"2024-03-31T15:14:11.390625+0000","last_became_active":"2024-03-31T15:14:09.285374+0000","last_became_peered":"2024-03-31T15:14:09.285374+0000","last_unstale":"2024-03-31T15:14:11.390625+0000","last_undegraded":"2024-03-31T15:14:11.390625+0000","last_fullsized":"2024-03-31T15:14:11.390625+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:34:11.338914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.30","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:10.891716+0000","last_change":"2024-03-31T15:14:09.287510+0000","last_active":"2024-03-31T15:14:10.891716+0000","last_peered":"2024-03-31T15:14:10.891716+0000","last_clean":"2024-03-31T15:14:10.891716+0000","last_became_active":"2024-03-31T15:14:09.287238+0000","last_became_peered":"2024-03-31T15:14:09.287238+0000","last_unstale":"2024-03-31T15:14:10.891716+0000","last_undegraded":"2024-03-31T15:14:10.891716+0000","last_fullsized":"2024-03-31T15:14:10.891716+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:14:09.225080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.2f","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:10.279850+0000","last_change":"2024-03-31T15:14:09.277762+0000","last_active":"2024-03-31T15:14:10.279850+0000","last_peered":"2024-03-31T15:14:10.279850+0000","last_clean":"2024-03-31T15:14:10.279850+0000","last_became_active":"2024-03-31T15:14:09.277487+0000","last_became_peered":"2024-03-31T15:14:09.277487+0000","last_unstale":"2024-03-31T15:14:10.279850+0000","last_undegraded":"2024-03-31T15:14:10.279850+0000","last_fullsized":"2024-03-31T15:14:10.279850+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:15:07.162929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2e","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:10.279438+0000","last_change":"2024-03-31T15:14:09.276215+0000","last_active":"2024-03-31T15:14:10.279438+0000","last_peered":"2024-03-31T15:14:10.279438+0000","last_clean":"2024-03-31T15:14:10.279438+0000","last_became_active":"2024-03-31T15:14:09.275956+0000","last_became_peered":"2024-03-31T15:14:09.275956+0000","last_unstale":"2024-03-31T15:14:10.279438+0000","last_undegraded":"2024-03-31T15:14:10.279438+0000","last_fullsized":"2024-03-31T15:14:10.279438+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:54:50.675440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.2d","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:10.892501+0000","last_change":"2024-03-31T15:14:09.287937+0000","last_active":"2024-03-31T15:14:10.892501+0000","last_peered":"2024-03-31T15:14:10.892501+0000","last_clean":"2024-03-31T15:14:10.892501+0000","last_became_active":"2024-03-31T15:14:09.287391+0000","last_became_peered":"2024-03-31T15:14:09.287391+0000","last_unstale":"2024-03-31T15:14:10.892501+0000","last_undegraded":"2024-03-31T15:14:10.892501+0000","last_fullsized":"2024-03-31T15:14:10.892501+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:36:58.066600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.2c","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:10.279545+0000","last_change":"2024-03-31T15:14:09.283362+0000","last_active":"2024-03-31T15:14:10.279545+0000","last_peered":"2024-03-31T15:14:10.279545+0000","last_clean":"2024-03-31T15:14:10.279545+0000","last_became_active":"2024-03-31T15:14:09.283040+0000","last_became_peered":"2024-03-31T15:14:09.283040+0000","last_unstale":"2024-03-31T15:14:10.279545+0000","last_undegraded":"2024-03-31T15:14:10.279545+0000","last_fullsized":"2024-03-31T15:14:10.279545+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:43:36.989252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2b","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:10.279313+0000","last_change":"2024-03-31T15:14:09.278966+0000","last_active":"2024-03-31T15:14:10.279313+0000","last_peered":"2024-03-31T15:14:10.279313+0000","last_clean":"2024-03-31T15:14:10.279313+0000","last_became_active":"2024-03-31T15:14:09.278453+0000","last_became_peered":"2024-03-31T15:14:09.278453+0000","last_unstale":"2024-03-31T15:14:10.279313+0000","last_undegraded":"2024-03-31T15:14:10.279313+0000","last_fullsized":"2024-03-31T15:14:10.279313+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:16:53.323124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2a","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:11.390332+0000","last_change":"2024-03-31T15:14:09.275037+0000","last_active":"2024-03-31T15:14:11.390332+0000","last_peered":"2024-03-31T15:14:11.390332+0000","last_clean":"2024-03-31T15:14:11.390332+0000","last_became_active":"2024-03-31T15:14:09.274177+0000","last_became_peered":"2024-03-31T15:14:09.274177+0000","last_unstale":"2024-03-31T15:14:11.390332+0000","last_undegraded":"2024-03-31T15:14:11.390332+0000","last_fullsized":"2024-03-31T15:14:11.390332+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:09:48.241483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.29","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:10.277953+0000","last_change":"2024-03-31T15:14:09.279623+0000","last_active":"2024-03-31T15:14:10.277953+0000","last_peered":"2024-03-31T15:14:10.277953+0000","last_clean":"2024-03-31T15:14:10.277953+0000","last_became_active":"2024-03-31T15:14:09.279356+0000","last_became_peered":"2024-03-31T15:14:09.279356+0000","last_unstale":"2024-03-31T15:14:10.277953+0000","last_undegraded":"2024-03-31T15:14:10.277953+0000","last_fullsized":"2024-03-31T15:14:10.277953+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:20:10.912650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.28","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:11.390297+0000","last_change":"2024-03-31T15:14:09.283380+0000","last_active":"2024-03-31T15:14:11.390297+0000","last_peered":"2024-03-31T15:14:11.390297+0000","last_clean":"2024-03-31T15:14:11.390297+0000","last_became_active":"2024-03-31T15:14:09.283097+0000","last_became_peered":"2024-03-31T15:14:09.283097+0000","last_unstale":"2024-03-31T15:14:11.390297+0000","last_undegraded":"2024-03-31T15:14:11.390297+0000","last_fullsized":"2024-03-31T15:14:11.390297+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:00:11.817601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.27","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:10.280178+0000","last_change":"2024-03-31T15:14:09.277583+0000","last_active":"2024-03-31T15:14:10.280178+0000","last_peered":"2024-03-31T15:14:10.280178+0000","last_clean":"2024-03-31T15:14:10.280178+0000","last_became_active":"2024-03-31T15:14:09.277320+0000","last_became_peered":"2024-03-31T15:14:09.277320+0000","last_unstale":"2024-03-31T15:14:10.280178+0000","last_undegraded":"2024-03-31T15:14:10.280178+0000","last_fullsized":"2024-03-31T15:14:10.280178+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:48:47.011312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.26","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:10.892253+0000","last_change":"2024-03-31T15:14:09.276124+0000","last_active":"2024-03-31T15:14:10.892253+0000","last_peered":"2024-03-31T15:14:10.892253+0000","last_clean":"2024-03-31T15:14:10.892253+0000","last_became_active":"2024-03-31T15:14:09.275495+0000","last_became_peered":"2024-03-31T15:14:09.275495+0000","last_unstale":"2024-03-31T15:14:10.892253+0000","last_undegraded":"2024-03-31T15:14:10.892253+0000","last_fullsized":"2024-03-31T15:14:10.892253+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:38:53.950142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.25","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:11.391422+0000","last_change":"2024-03-31T15:14:09.290617+0000","last_active":"2024-03-31T15:14:11.391422+0000","last_peered":"2024-03-31T15:14:11.391422+0000","last_clean":"2024-03-31T15:14:11.391422+0000","last_became_active":"2024-03-31T15:14:09.290308+0000","last_became_peered":"2024-03-31T15:14:09.290308+0000","last_unstale":"2024-03-31T15:14:11.391422+0000","last_undegraded":"2024-03-31T15:14:11.391422+0000","last_fullsized":"2024-03-31T15:14:11.391422+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:43:42.509329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.24","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:11.391301+0000","last_change":"2024-03-31T15:14:09.290865+0000","last_active":"2024-03-31T15:14:11.391301+0000","last_peered":"2024-03-31T15:14:11.391301+0000","last_clean":"2024-03-31T15:14:11.391301+0000","last_became_active":"2024-03-31T15:14:09.290601+0000","last_became_peered":"2024-03-31T15:14:09.290601+0000","last_unstale":"2024-03-31T15:14:11.391301+0000","last_undegraded":"2024-03-31T15:14:11.391301+0000","last_fullsized":"2024-03-31T15:14:11.391301+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T19:21:02.579812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.23","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:10.278903+0000","last_change":"2024-03-31T15:14:09.278728+0000","last_active":"2024-03-31T15:14:10.278903+0000","last_peered":"2024-03-31T15:14:10.278903+0000","last_clean":"2024-03-31T15:14:10.278903+0000","last_became_active":"2024-03-31T15:14:09.278092+0000","last_became_peered":"2024-03-31T15:14:09.278092+0000","last_unstale":"2024-03-31T15:14:10.278903+0000","last_undegraded":"2024-03-31T15:14:10.278903+0000","last_fullsized":"2024-03-31T15:14:10.278903+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:49:03.965468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.22","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:10.277030+0000","last_change":"2024-03-31T15:14:09.279075+0000","last_active":"2024-03-31T15:14:10.277030+0000","last_peered":"2024-03-31T15:14:10.277030+0000","last_clean":"2024-03-31T15:14:10.277030+0000","last_became_active":"2024-03-31T15:14:09.278785+0000","last_became_peered":"2024-03-31T15:14:09.278785+0000","last_unstale":"2024-03-31T15:14:10.277030+0000","last_undegraded":"2024-03-31T15:14:10.277030+0000","last_fullsized":"2024-03-31T15:14:10.277030+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:24:12.224782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.21","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:10.892072+0000","last_change":"2024-03-31T15:14:09.276137+0000","last_active":"2024-03-31T15:14:10.892072+0000","last_peered":"2024-03-31T15:14:10.892072+0000","last_clean":"2024-03-31T15:14:10.892072+0000","last_became_active":"2024-03-31T15:14:09.275873+0000","last_became_peered":"2024-03-31T15:14:09.275873+0000","last_unstale":"2024-03-31T15:14:10.892072+0000","last_undegraded":"2024-03-31T15:14:10.892072+0000","last_fullsized":"2024-03-31T15:14:10.892072+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T17:03:26.938790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.20","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:11.389806+0000","last_change":"2024-03-31T15:14:09.285377+0000","last_active":"2024-03-31T15:14:11.389806+0000","last_peered":"2024-03-31T15:14:11.389806+0000","last_clean":"2024-03-31T15:14:11.389806+0000","last_became_active":"2024-03-31T15:14:09.285077+0000","last_became_peered":"2024-03-31T15:14:09.285077+0000","last_unstale":"2024-03-31T15:14:11.389806+0000","last_undegraded":"2024-03-31T15:14:11.389806+0000","last_fullsized":"2024-03-31T15:14:11.389806+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T15:39:45.840997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.1e","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:11.391059+0000","last_change":"2024-03-31T15:14:09.286260+0000","last_active":"2024-03-31T15:14:11.391059+0000","last_peered":"2024-03-31T15:14:11.391059+0000","last_clean":"2024-03-31T15:14:11.391059+0000","last_became_active":"2024-03-31T15:14:09.285985+0000","last_became_peered":"2024-03-31T15:14:09.285985+0000","last_unstale":"2024-03-31T15:14:11.391059+0000","last_undegraded":"2024-03-31T15:14:11.391059+0000","last_fullsized":"2024-03-31T15:14:11.391059+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T15:58:28.943455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1f","version":"108'4","reported_seq":18,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:10.277222+0000","last_change":"2024-03-31T15:14:08.268805+0000","last_active":"2024-03-31T15:14:10.277222+0000","last_peered":"2024-03-31T15:14:10.277222+0000","last_clean":"2024-03-31T15:14:10.277222+0000","last_became_active":"2024-03-31T15:14:08.268515+0000","last_became_peered":"2024-03-31T15:14:08.268515+0000","last_unstale":"2024-03-31T15:14:10.277222+0000","last_undegraded":"2024-03-31T15:14:10.277222+0000","last_fullsized":"2024-03-31T15:14:10.277222+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_clean_scrub_stamp":"2024-03-31T15:14:07.235046+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:16:55.261359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1f","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:10.278621+0000","last_change":"2024-03-31T15:14:09.281053+0000","last_active":"2024-03-31T15:14:10.278621+0000","last_peered":"2024-03-31T15:14:10.278621+0000","last_clean":"2024-03-31T15:14:10.278621+0000","last_became_active":"2024-03-31T15:14:09.280735+0000","last_became_peered":"2024-03-31T15:14:09.280735+0000","last_unstale":"2024-03-31T15:14:10.278621+0000","last_undegraded":"2024-03-31T15:14:10.278621+0000","last_fullsized":"2024-03-31T15:14:10.278621+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:55:28.086183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.1e","version":"108'1","reported_seq":15,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:10.892384+0000","last_change":"2024-03-31T15:14:08.270517+0000","last_active":"2024-03-31T15:14:10.892384+0000","last_peered":"2024-03-31T15:14:10.892384+0000","last_clean":"2024-03-31T15:14:10.892384+0000","last_became_active":"2024-03-31T15:14:08.270219+0000","last_became_peered":"2024-03-31T15:14:08.270219+0000","last_unstale":"2024-03-31T15:14:10.892384+0000","last_undegraded":"2024-03-31T15:14:10.892384+0000","last_fullsized":"2024-03-31T15:14:10.892384+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_clean_scrub_stamp":"2024-03-31T15:14:07.235046+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:06:47.238926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1c","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:11.391070+0000","last_change":"2024-03-31T15:14:09.285776+0000","last_active":"2024-03-31T15:14:11.391070+0000","last_peered":"2024-03-31T15:14:11.391070+0000","last_clean":"2024-03-31T15:14:11.391070+0000","last_became_active":"2024-03-31T15:14:09.285383+0000","last_became_peered":"2024-03-31T15:14:09.285383+0000","last_unstale":"2024-03-31T15:14:11.391070+0000","last_undegraded":"2024-03-31T15:14:11.391070+0000","last_fullsized":"2024-03-31T15:14:11.391070+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:45:29.558658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1d","version":"108'2","reported_seq":16,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:11.390584+0000","last_change":"2024-03-31T15:14:08.268956+0000","last_active":"2024-03-31T15:14:11.390584+0000","last_peered":"2024-03-31T15:14:11.390584+0000","last_clean":"2024-03-31T15:14:11.390584+0000","last_became_active":"2024-03-31T15:14:08.268672+0000","last_became_peered":"2024-03-31T15:14:08.268672+0000","last_unstale":"2024-03-31T15:14:11.390584+0000","last_undegraded":"2024-03-31T15:14:11.390584+0000","last_fullsized":"2024-03-31T15:14:11.390584+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_clean_scrub_stamp":"2024-03-31T15:14:07.235046+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T15:35:25.437508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1d","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:10.892687+0000","last_change":"2024-03-31T15:14:09.288130+0000","last_active":"2024-03-31T15:14:10.892687+0000","last_peered":"2024-03-31T15:14:10.892687+0000","last_clean":"2024-03-31T15:14:10.892687+0000","last_became_active":"2024-03-31T15:14:09.287597+0000","last_became_peered":"2024-03-31T15:14:09.287597+0000","last_unstale":"2024-03-31T15:14:10.892687+0000","last_undegraded":"2024-03-31T15:14:10.892687+0000","last_fullsized":"2024-03-31T15:14:10.892687+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:02:21.465704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1c","version":"108'2","reported_seq":16,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:11.391531+0000","last_change":"2024-03-31T15:14:08.269987+0000","last_active":"2024-03-31T15:14:11.391531+0000","last_peered":"2024-03-31T15:14:11.391531+0000","last_clean":"2024-03-31T15:14:11.391531+0000","last_became_active":"2024-03-31T15:14:08.269241+0000","last_became_peered":"2024-03-31T15:14:08.269241+0000","last_unstale":"2024-03-31T15:14:11.391531+0000","last_undegraded":"2024-03-31T15:14:11.391531+0000","last_fullsized":"2024-03-31T15:14:11.391531+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_clean_scrub_stamp":"2024-03-31T15:14:07.235046+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:56:08.718098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1a","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:10.891958+0000","last_change":"2024-03-31T15:14:09.282143+0000","last_active":"2024-03-31T15:14:10.891958+0000","last_peered":"2024-03-31T15:14:10.891958+0000","last_clean":"2024-03-31T15:14:10.891958+0000","last_became_active":"2024-03-31T15:14:09.281851+0000","last_became_peered":"2024-03-31T15:14:09.281851+0000","last_unstale":"2024-03-31T15:14:10.891958+0000","last_undegraded":"2024-03-31T15:14:10.891958+0000","last_fullsized":"2024-03-31T15:14:10.891958+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:07:40.319619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.1b","version":"108'1","reported_seq":15,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:10.276959+0000","last_change":"2024-03-31T15:14:08.271392+0000","last_active":"2024-03-31T15:14:10.276959+0000","last_peered":"2024-03-31T15:14:10.276959+0000","last_clean":"2024-03-31T15:14:10.276959+0000","last_became_active":"2024-03-31T15:14:08.271138+0000","last_became_peered":"2024-03-31T15:14:08.271138+0000","last_unstale":"2024-03-31T15:14:10.276959+0000","last_undegraded":"2024-03-31T15:14:10.276959+0000","last_fullsized":"2024-03-31T15:14:10.276959+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_clean_scrub_stamp":"2024-03-31T15:14:07.235046+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T16:18:45.057526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1b","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:10.279089+0000","last_change":"2024-03-31T15:14:09.288830+0000","last_active":"2024-03-31T15:14:10.279089+0000","last_peered":"2024-03-31T15:14:10.279089+0000","last_clean":"2024-03-31T15:14:10.279089+0000","last_became_active":"2024-03-31T15:14:09.288486+0000","last_became_peered":"2024-03-31T15:14:09.288486+0000","last_unstale":"2024-03-31T15:14:10.279089+0000","last_undegraded":"2024-03-31T15:14:10.279089+0000","last_fullsized":"2024-03-31T15:14:10.279089+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:53:27.090923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.1a","version":"108'1","reported_seq":15,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:11.390384+0000","last_change":"2024-03-31T15:14:08.273955+0000","last_active":"2024-03-31T15:14:11.390384+0000","last_peered":"2024-03-31T15:14:11.390384+0000","last_clean":"2024-03-31T15:14:11.390384+0000","last_became_active":"2024-03-31T15:14:08.273646+0000","last_became_peered":"2024-03-31T15:14:08.273646+0000","last_unstale":"2024-03-31T15:14:11.390384+0000","last_undegraded":"2024-03-31T15:14:11.390384+0000","last_fullsized":"2024-03-31T15:14:11.390384+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_clean_scrub_stamp":"2024-03-31T15:14:07.235046+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:27:01.349324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.18","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:10.891733+0000","last_change":"2024-03-31T15:14:09.278486+0000","last_active":"2024-03-31T15:14:10.891733+0000","last_peered":"2024-03-31T15:14:10.891733+0000","last_clean":"2024-03-31T15:14:10.891733+0000","last_became_active":"2024-03-31T15:14:09.278193+0000","last_became_peered":"2024-03-31T15:14:09.278193+0000","last_unstale":"2024-03-31T15:14:10.891733+0000","last_undegraded":"2024-03-31T15:14:10.891733+0000","last_fullsized":"2024-03-31T15:14:10.891733+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:42:40.870161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":14,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:10.278054+0000","last_change":"2024-03-31T15:14:08.266416+0000","last_active":"2024-03-31T15:14:10.278054+0000","last_peered":"2024-03-31T15:14:10.278054+0000","last_clean":"2024-03-31T15:14:10.278054+0000","last_became_active":"2024-03-31T15:14:08.266116+0000","last_became_peered":"2024-03-31T15:14:08.266116+0000","last_unstale":"2024-03-31T15:14:10.278054+0000","last_undegraded":"2024-03-31T15:14:10.278054+0000","last_fullsized":"2024-03-31T15:14:10.278054+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_clean_scrub_stamp":"2024-03-31T15:14:07.235046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:24:11.217734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.19","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:10.276686+0000","last_change":"2024-03-31T15:14:09.271307+0000","last_active":"2024-03-31T15:14:10.276686+0000","last_peered":"2024-03-31T15:14:10.276686+0000","last_clean":"2024-03-31T15:14:10.276686+0000","last_became_active":"2024-03-31T15:14:09.271017+0000","last_became_peered":"2024-03-31T15:14:09.271017+0000","last_unstale":"2024-03-31T15:14:10.276686+0000","last_undegraded":"2024-03-31T15:14:10.276686+0000","last_fullsized":"2024-03-31T15:14:10.276686+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:15:09.198042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":14,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:10.891808+0000","last_change":"2024-03-31T15:14:08.276025+0000","last_active":"2024-03-31T15:14:10.891808+0000","last_peered":"2024-03-31T15:14:10.891808+0000","last_clean":"2024-03-31T15:14:10.891808+0000","last_became_active":"2024-03-31T15:14:08.275541+0000","last_became_peered":"2024-03-31T15:14:08.275541+0000","last_unstale":"2024-03-31T15:14:10.891808+0000","last_undegraded":"2024-03-31T15:14:10.891808+0000","last_fullsized":"2024-03-31T15:14:10.891808+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_clean_scrub_stamp":"2024-03-31T15:14:07.235046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:34:33.623018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.16","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:10.277387+0000","last_change":"2024-03-31T15:14:09.273769+0000","last_active":"2024-03-31T15:14:10.277387+0000","last_peered":"2024-03-31T15:14:10.277387+0000","last_clean":"2024-03-31T15:14:10.277387+0000","last_became_active":"2024-03-31T15:14:09.273470+0000","last_became_peered":"2024-03-31T15:14:09.273470+0000","last_unstale":"2024-03-31T15:14:10.277387+0000","last_undegraded":"2024-03-31T15:14:10.277387+0000","last_fullsized":"2024-03-31T15:14:10.277387+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:27:17.610346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.17","version":"0'0","reported_seq":14,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:11.390501+0000","last_change":"2024-03-31T15:14:08.275403+0000","last_active":"2024-03-31T15:14:11.390501+0000","last_peered":"2024-03-31T15:14:11.390501+0000","last_clean":"2024-03-31T15:14:11.390501+0000","last_became_active":"2024-03-31T15:14:08.275122+0000","last_became_peered":"2024-03-31T15:14:08.275122+0000","last_unstale":"2024-03-31T15:14:11.390501+0000","last_undegraded":"2024-03-31T15:14:11.390501+0000","last_fullsized":"2024-03-31T15:14:11.390501+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_clean_scrub_stamp":"2024-03-31T15:14:07.235046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:03:52.622502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.17","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:10.278750+0000","last_change":"2024-03-31T15:14:09.298453+0000","last_active":"2024-03-31T15:14:10.278750+0000","last_peered":"2024-03-31T15:14:10.278750+0000","last_clean":"2024-03-31T15:14:10.278750+0000","last_became_active":"2024-03-31T15:14:09.298160+0000","last_became_peered":"2024-03-31T15:14:09.298160+0000","last_unstale":"2024-03-31T15:14:10.278750+0000","last_undegraded":"2024-03-31T15:14:10.278750+0000","last_fullsized":"2024-03-31T15:14:10.278750+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:48:01.826468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":14,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:11.391501+0000","last_change":"2024-03-31T15:14:08.272714+0000","last_active":"2024-03-31T15:14:11.391501+0000","last_peered":"2024-03-31T15:14:11.391501+0000","last_clean":"2024-03-31T15:14:11.391501+0000","last_became_active":"2024-03-31T15:14:08.272416+0000","last_became_peered":"2024-03-31T15:14:08.272416+0000","last_unstale":"2024-03-31T15:14:11.391501+0000","last_undegraded":"2024-03-31T15:14:11.391501+0000","last_fullsized":"2024-03-31T15:14:11.391501+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_clean_scrub_stamp":"2024-03-31T15:14:07.235046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:34:33.623018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.14","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:11.391432+0000","last_change":"2024-03-31T15:14:09.281244+0000","last_active":"2024-03-31T15:14:11.391432+0000","last_peered":"2024-03-31T15:14:11.391432+0000","last_clean":"2024-03-31T15:14:11.391432+0000","last_became_active":"2024-03-31T15:14:09.280267+0000","last_became_peered":"2024-03-31T15:14:09.280267+0000","last_unstale":"2024-03-31T15:14:11.391432+0000","last_undegraded":"2024-03-31T15:14:11.391432+0000","last_fullsized":"2024-03-31T15:14:11.391432+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:23:54.993797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.15","version":"108'3","reported_seq":17,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:10.891863+0000","last_change":"2024-03-31T15:14:08.274322+0000","last_active":"2024-03-31T15:14:10.891863+0000","last_peered":"2024-03-31T15:14:10.891863+0000","last_clean":"2024-03-31T15:14:10.891863+0000","last_became_active":"2024-03-31T15:14:08.274025+0000","last_became_peered":"2024-03-31T15:14:08.274025+0000","last_unstale":"2024-03-31T15:14:10.891863+0000","last_undegraded":"2024-03-31T15:14:10.891863+0000","last_fullsized":"2024-03-31T15:14:10.891863+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_clean_scrub_stamp":"2024-03-31T15:14:07.235046+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:54:49.668391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.15","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:10.277296+0000","last_change":"2024-03-31T15:14:09.285492+0000","last_active":"2024-03-31T15:14:10.277296+0000","last_peered":"2024-03-31T15:14:10.277296+0000","last_clean":"2024-03-31T15:14:10.277296+0000","last_became_active":"2024-03-31T15:14:09.285203+0000","last_became_peered":"2024-03-31T15:14:09.285203+0000","last_unstale":"2024-03-31T15:14:10.277296+0000","last_undegraded":"2024-03-31T15:14:10.277296+0000","last_fullsized":"2024-03-31T15:14:10.277296+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:39:31.641218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.14","version":"108'6","reported_seq":20,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:10.892187+0000","last_change":"2024-03-31T15:14:08.277536+0000","last_active":"2024-03-31T15:14:10.892187+0000","last_peered":"2024-03-31T15:14:10.892187+0000","last_clean":"2024-03-31T15:14:10.892187+0000","last_became_active":"2024-03-31T15:14:08.277279+0000","last_became_peered":"2024-03-31T15:14:08.277279+0000","last_unstale":"2024-03-31T15:14:10.892187+0000","last_undegraded":"2024-03-31T15:14:10.892187+0000","last_fullsized":"2024-03-31T15:14:10.892187+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_clean_scrub_stamp":"2024-03-31T15:14:07.235046+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:08:03.916239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.12","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:11.390394+0000","last_change":"2024-03-31T15:14:09.275078+0000","last_active":"2024-03-31T15:14:11.390394+0000","last_peered":"2024-03-31T15:14:11.390394+0000","last_clean":"2024-03-31T15:14:11.390394+0000","last_became_active":"2024-03-31T15:14:09.274698+0000","last_became_peered":"2024-03-31T15:14:09.274698+0000","last_unstale":"2024-03-31T15:14:11.390394+0000","last_undegraded":"2024-03-31T15:14:11.390394+0000","last_fullsized":"2024-03-31T15:14:11.390394+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:30:56.093324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.13","version":"108'2","reported_seq":16,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:11.390958+0000","last_change":"2024-03-31T15:14:08.266097+0000","last_active":"2024-03-31T15:14:11.390958+0000","last_peered":"2024-03-31T15:14:11.390958+0000","last_clean":"2024-03-31T15:14:11.390958+0000","last_became_active":"2024-03-31T15:14:08.265812+0000","last_became_peered":"2024-03-31T15:14:08.265812+0000","last_unstale":"2024-03-31T15:14:11.390958+0000","last_undegraded":"2024-03-31T15:14:11.390958+0000","last_fullsized":"2024-03-31T15:14:11.390958+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_clean_scrub_stamp":"2024-03-31T15:14:07.235046+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:49:52.886409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.13","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:11.391329+0000","last_change":"2024-03-31T15:14:09.282087+0000","last_active":"2024-03-31T15:14:11.391329+0000","last_peered":"2024-03-31T15:14:11.391329+0000","last_clean":"2024-03-31T15:14:11.391329+0000","last_became_active":"2024-03-31T15:14:09.281693+0000","last_became_peered":"2024-03-31T15:14:09.281693+0000","last_unstale":"2024-03-31T15:14:11.391329+0000","last_undegraded":"2024-03-31T15:14:11.391329+0000","last_fullsized":"2024-03-31T15:14:11.391329+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:32:06.156300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":14,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:10.278350+0000","last_change":"2024-03-31T15:14:08.272919+0000","last_active":"2024-03-31T15:14:10.278350+0000","last_peered":"2024-03-31T15:14:10.278350+0000","last_clean":"2024-03-31T15:14:10.278350+0000","last_became_active":"2024-03-31T15:14:08.272309+0000","last_became_peered":"2024-03-31T15:14:08.272309+0000","last_unstale":"2024-03-31T15:14:10.278350+0000","last_undegraded":"2024-03-31T15:14:10.278350+0000","last_fullsized":"2024-03-31T15:14:10.278350+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_clean_scrub_stamp":"2024-03-31T15:14:07.235046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:56:10.303847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.11","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:11.390150+0000","last_change":"2024-03-31T15:14:09.274826+0000","last_active":"2024-03-31T15:14:11.390150+0000","last_peered":"2024-03-31T15:14:11.390150+0000","last_clean":"2024-03-31T15:14:11.390150+0000","last_became_active":"2024-03-31T15:14:09.274550+0000","last_became_peered":"2024-03-31T15:14:09.274550+0000","last_unstale":"2024-03-31T15:14:11.390150+0000","last_undegraded":"2024-03-31T15:14:11.390150+0000","last_fullsized":"2024-03-31T15:14:11.390150+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T17:25:28.357433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.10","version":"108'8","reported_seq":22,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:11.389660+0000","last_change":"2024-03-31T15:14:08.276042+0000","last_active":"2024-03-31T15:14:11.389660+0000","last_peered":"2024-03-31T15:14:11.389660+0000","last_clean":"2024-03-31T15:14:11.389660+0000","last_became_active":"2024-03-31T15:14:08.275766+0000","last_became_peered":"2024-03-31T15:14:08.275766+0000","last_unstale":"2024-03-31T15:14:11.389660+0000","last_undegraded":"2024-03-31T15:14:11.389660+0000","last_fullsized":"2024-03-31T15:14:11.389660+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_clean_scrub_stamp":"2024-03-31T15:14:07.235046+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:19:14.135624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.e","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:11.390586+0000","last_change":"2024-03-31T15:14:09.298576+0000","last_active":"2024-03-31T15:14:11.390586+0000","last_peered":"2024-03-31T15:14:11.390586+0000","last_clean":"2024-03-31T15:14:11.390586+0000","last_became_active":"2024-03-31T15:14:09.298059+0000","last_became_peered":"2024-03-31T15:14:09.298059+0000","last_unstale":"2024-03-31T15:14:11.390586+0000","last_undegraded":"2024-03-31T15:14:11.390586+0000","last_fullsized":"2024-03-31T15:14:11.390586+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T16:46:33.733000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.f","version":"108'1","reported_seq":15,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:10.892528+0000","last_change":"2024-03-31T15:14:08.273521+0000","last_active":"2024-03-31T15:14:10.892528+0000","last_peered":"2024-03-31T15:14:10.892528+0000","last_clean":"2024-03-31T15:14:10.892528+0000","last_became_active":"2024-03-31T15:14:08.271683+0000","last_became_peered":"2024-03-31T15:14:08.271683+0000","last_unstale":"2024-03-31T15:14:10.892528+0000","last_undegraded":"2024-03-31T15:14:10.892528+0000","last_fullsized":"2024-03-31T15:14:10.892528+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_clean_scrub_stamp":"2024-03-31T15:14:07.235046+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:08:12.284361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.f","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:11.391301+0000","last_change":"2024-03-31T15:14:09.292401+0000","last_active":"2024-03-31T15:14:11.391301+0000","last_peered":"2024-03-31T15:14:11.391301+0000","last_clean":"2024-03-31T15:14:11.391301+0000","last_became_active":"2024-03-31T15:14:09.292126+0000","last_became_peered":"2024-03-31T15:14:09.292126+0000","last_unstale":"2024-03-31T15:14:11.391301+0000","last_undegraded":"2024-03-31T15:14:11.391301+0000","last_fullsized":"2024-03-31T15:14:11.391301+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:24:54.943510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.e","version":"108'2","reported_seq":16,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:11.390584+0000","last_change":"2024-03-31T15:14:08.272765+0000","last_active":"2024-03-31T15:14:11.390584+0000","last_peered":"2024-03-31T15:14:11.390584+0000","last_clean":"2024-03-31T15:14:11.390584+0000","last_became_active":"2024-03-31T15:14:08.272496+0000","last_became_peered":"2024-03-31T15:14:08.272496+0000","last_unstale":"2024-03-31T15:14:11.390584+0000","last_undegraded":"2024-03-31T15:14:11.390584+0000","last_fullsized":"2024-03-31T15:14:11.390584+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_clean_scrub_stamp":"2024-03-31T15:14:07.235046+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:18:11.474845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.c","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:10.892335+0000","last_change":"2024-03-31T15:14:09.299017+0000","last_active":"2024-03-31T15:14:10.892335+0000","last_peered":"2024-03-31T15:14:10.892335+0000","last_clean":"2024-03-31T15:14:10.892335+0000","last_became_active":"2024-03-31T15:14:09.298654+0000","last_became_peered":"2024-03-31T15:14:09.298654+0000","last_unstale":"2024-03-31T15:14:10.892335+0000","last_undegraded":"2024-03-31T15:14:10.892335+0000","last_fullsized":"2024-03-31T15:14:10.892335+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:35:57.110653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.d","version":"108'1","reported_seq":15,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:10.278500+0000","last_change":"2024-03-31T15:14:08.278494+0000","last_active":"2024-03-31T15:14:10.278500+0000","last_peered":"2024-03-31T15:14:10.278500+0000","last_clean":"2024-03-31T15:14:10.278500+0000","last_became_active":"2024-03-31T15:14:08.278267+0000","last_became_peered":"2024-03-31T15:14:08.278267+0000","last_unstale":"2024-03-31T15:14:10.278500+0000","last_undegraded":"2024-03-31T15:14:10.278500+0000","last_fullsized":"2024-03-31T15:14:10.278500+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_clean_scrub_stamp":"2024-03-31T15:14:07.235046+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:49:25.058336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.d","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:10.892849+0000","last_change":"2024-03-31T15:14:09.288545+0000","last_active":"2024-03-31T15:14:10.892849+0000","last_peered":"2024-03-31T15:14:10.892849+0000","last_clean":"2024-03-31T15:14:10.892849+0000","last_became_active":"2024-03-31T15:14:09.288260+0000","last_became_peered":"2024-03-31T15:14:09.288260+0000","last_unstale":"2024-03-31T15:14:10.892849+0000","last_undegraded":"2024-03-31T15:14:10.892849+0000","last_fullsized":"2024-03-31T15:14:10.892849+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:16:47.900467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":14,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:10.277423+0000","last_change":"2024-03-31T15:14:08.271960+0000","last_active":"2024-03-31T15:14:10.277423+0000","last_peered":"2024-03-31T15:14:10.277423+0000","last_clean":"2024-03-31T15:14:10.277423+0000","last_became_active":"2024-03-31T15:14:08.271691+0000","last_became_peered":"2024-03-31T15:14:08.271691+0000","last_unstale":"2024-03-31T15:14:10.277423+0000","last_undegraded":"2024-03-31T15:14:10.277423+0000","last_fullsized":"2024-03-31T15:14:10.277423+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_clean_scrub_stamp":"2024-03-31T15:14:07.235046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T16:20:03.178688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.a","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:10.892273+0000","last_change":"2024-03-31T15:14:09.286866+0000","last_active":"2024-03-31T15:14:10.892273+0000","last_peered":"2024-03-31T15:14:10.892273+0000","last_clean":"2024-03-31T15:14:10.892273+0000","last_became_active":"2024-03-31T15:14:09.286282+0000","last_became_peered":"2024-03-31T15:14:09.286282+0000","last_unstale":"2024-03-31T15:14:10.892273+0000","last_undegraded":"2024-03-31T15:14:10.892273+0000","last_fullsized":"2024-03-31T15:14:10.892273+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T17:02:20.640638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.b","version":"108'2","reported_seq":16,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:11.391062+0000","last_change":"2024-03-31T15:14:08.274852+0000","last_active":"2024-03-31T15:14:11.391062+0000","last_peered":"2024-03-31T15:14:11.391062+0000","last_clean":"2024-03-31T15:14:11.391062+0000","last_became_active":"2024-03-31T15:14:08.274568+0000","last_became_peered":"2024-03-31T15:14:08.274568+0000","last_unstale":"2024-03-31T15:14:11.391062+0000","last_undegraded":"2024-03-31T15:14:11.391062+0000","last_fullsized":"2024-03-31T15:14:11.391062+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_clean_scrub_stamp":"2024-03-31T15:14:07.235046+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T19:34:36.202544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.b","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:10.891810+0000","last_change":"2024-03-31T15:14:09.280942+0000","last_active":"2024-03-31T15:14:10.891810+0000","last_peered":"2024-03-31T15:14:10.891810+0000","last_clean":"2024-03-31T15:14:10.891810+0000","last_became_active":"2024-03-31T15:14:09.280678+0000","last_became_peered":"2024-03-31T15:14:09.280678+0000","last_unstale":"2024-03-31T15:14:10.891810+0000","last_undegraded":"2024-03-31T15:14:10.891810+0000","last_fullsized":"2024-03-31T15:14:10.891810+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:00:22.586826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":14,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:11.389585+0000","last_change":"2024-03-31T15:14:08.275261+0000","last_active":"2024-03-31T15:14:11.389585+0000","last_peered":"2024-03-31T15:14:11.389585+0000","last_clean":"2024-03-31T15:14:11.389585+0000","last_became_active":"2024-03-31T15:14:08.274961+0000","last_became_peered":"2024-03-31T15:14:08.274961+0000","last_unstale":"2024-03-31T15:14:11.389585+0000","last_undegraded":"2024-03-31T15:14:11.389585+0000","last_fullsized":"2024-03-31T15:14:11.389585+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_clean_scrub_stamp":"2024-03-31T15:14:07.235046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:05:46.368323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.8","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:10.892352+0000","last_change":"2024-03-31T15:14:09.284118+0000","last_active":"2024-03-31T15:14:10.892352+0000","last_peered":"2024-03-31T15:14:10.892352+0000","last_clean":"2024-03-31T15:14:10.892352+0000","last_became_active":"2024-03-31T15:14:09.283439+0000","last_became_peered":"2024-03-31T15:14:09.283439+0000","last_unstale":"2024-03-31T15:14:10.892352+0000","last_undegraded":"2024-03-31T15:14:10.892352+0000","last_fullsized":"2024-03-31T15:14:10.892352+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T19:55:36.064563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":14,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:11.390727+0000","last_change":"2024-03-31T15:14:08.275022+0000","last_active":"2024-03-31T15:14:11.390727+0000","last_peered":"2024-03-31T15:14:11.390727+0000","last_clean":"2024-03-31T15:14:11.390727+0000","last_became_active":"2024-03-31T15:14:08.274740+0000","last_became_peered":"2024-03-31T15:14:08.274740+0000","last_unstale":"2024-03-31T15:14:11.390727+0000","last_undegraded":"2024-03-31T15:14:11.390727+0000","last_fullsized":"2024-03-31T15:14:11.390727+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_clean_scrub_stamp":"2024-03-31T15:14:07.235046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:25:35.260804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.9","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:11.390344+0000","last_change":"2024-03-31T15:14:09.291069+0000","last_active":"2024-03-31T15:14:11.390344+0000","last_peered":"2024-03-31T15:14:11.390344+0000","last_clean":"2024-03-31T15:14:11.390344+0000","last_became_active":"2024-03-31T15:14:09.290802+0000","last_became_peered":"2024-03-31T15:14:09.290802+0000","last_unstale":"2024-03-31T15:14:11.390344+0000","last_undegraded":"2024-03-31T15:14:11.390344+0000","last_fullsized":"2024-03-31T15:14:11.390344+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:36:20.417016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":14,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:10.891861+0000","last_change":"2024-03-31T15:14:08.270333+0000","last_active":"2024-03-31T15:14:10.891861+0000","last_peered":"2024-03-31T15:14:10.891861+0000","last_clean":"2024-03-31T15:14:10.891861+0000","last_became_active":"2024-03-31T15:14:08.269866+0000","last_became_peered":"2024-03-31T15:14:08.269866+0000","last_unstale":"2024-03-31T15:14:10.891861+0000","last_undegraded":"2024-03-31T15:14:10.891861+0000","last_fullsized":"2024-03-31T15:14:10.891861+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_clean_scrub_stamp":"2024-03-31T15:14:07.235046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:55:27.079135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.6","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:10.278592+0000","last_change":"2024-03-31T15:14:09.299295+0000","last_active":"2024-03-31T15:14:10.278592+0000","last_peered":"2024-03-31T15:14:10.278592+0000","last_clean":"2024-03-31T15:14:10.278592+0000","last_became_active":"2024-03-31T15:14:09.299016+0000","last_became_peered":"2024-03-31T15:14:09.299016+0000","last_unstale":"2024-03-31T15:14:10.278592+0000","last_undegraded":"2024-03-31T15:14:10.278592+0000","last_fullsized":"2024-03-31T15:14:10.278592+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:34:34.630067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.7","version":"108'2","reported_seq":16,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:11.391777+0000","last_change":"2024-03-31T15:14:08.276158+0000","last_active":"2024-03-31T15:14:11.391777+0000","last_peered":"2024-03-31T15:14:11.391777+0000","last_clean":"2024-03-31T15:14:11.391777+0000","last_became_active":"2024-03-31T15:14:08.275856+0000","last_became_peered":"2024-03-31T15:14:08.275856+0000","last_unstale":"2024-03-31T15:14:11.391777+0000","last_undegraded":"2024-03-31T15:14:11.391777+0000","last_fullsized":"2024-03-31T15:14:11.391777+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_clean_scrub_stamp":"2024-03-31T15:14:07.235046+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:48:00.819420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.7","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:10.892615+0000","last_change":"2024-03-31T15:14:09.277522+0000","last_active":"2024-03-31T15:14:10.892615+0000","last_peered":"2024-03-31T15:14:10.892615+0000","last_clean":"2024-03-31T15:14:10.892615+0000","last_became_active":"2024-03-31T15:14:09.276889+0000","last_became_peered":"2024-03-31T15:14:09.276889+0000","last_unstale":"2024-03-31T15:14:10.892615+0000","last_undegraded":"2024-03-31T15:14:10.892615+0000","last_fullsized":"2024-03-31T15:14:10.892615+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:45:51.393113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.6","version":"109'1","reported_seq":15,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:10.278704+0000","last_change":"2024-03-31T15:14:08.265332+0000","last_active":"2024-03-31T15:14:10.278704+0000","last_peered":"2024-03-31T15:14:10.278704+0000","last_clean":"2024-03-31T15:14:10.278704+0000","last_became_active":"2024-03-31T15:14:08.264838+0000","last_became_peered":"2024-03-31T15:14:08.264838+0000","last_unstale":"2024-03-31T15:14:10.278704+0000","last_undegraded":"2024-03-31T15:14:10.278704+0000","last_fullsized":"2024-03-31T15:14:10.278704+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_clean_scrub_stamp":"2024-03-31T15:14:07.235046+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:32:53.111430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.4","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:10.892152+0000","last_change":"2024-03-31T15:14:09.279412+0000","last_active":"2024-03-31T15:14:10.892152+0000","last_peered":"2024-03-31T15:14:10.892152+0000","last_clean":"2024-03-31T15:14:10.892152+0000","last_became_active":"2024-03-31T15:14:09.278696+0000","last_became_peered":"2024-03-31T15:14:09.278696+0000","last_unstale":"2024-03-31T15:14:10.892152+0000","last_undegraded":"2024-03-31T15:14:10.892152+0000","last_fullsized":"2024-03-31T15:14:10.892152+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:57:34.106053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":14,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:10.891892+0000","last_change":"2024-03-31T15:14:08.270001+0000","last_active":"2024-03-31T15:14:10.891892+0000","last_peered":"2024-03-31T15:14:10.891892+0000","last_clean":"2024-03-31T15:14:10.891892+0000","last_became_active":"2024-03-31T15:14:08.268897+0000","last_became_peered":"2024-03-31T15:14:08.268897+0000","last_unstale":"2024-03-31T15:14:10.891892+0000","last_undegraded":"2024-03-31T15:14:10.891892+0000","last_fullsized":"2024-03-31T15:14:10.891892+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_clean_scrub_stamp":"2024-03-31T15:14:07.235046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T19:21:04.678038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.5","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:10.278439+0000","last_change":"2024-03-31T15:14:09.292175+0000","last_active":"2024-03-31T15:14:10.278439+0000","last_peered":"2024-03-31T15:14:10.278439+0000","last_clean":"2024-03-31T15:14:10.278439+0000","last_became_active":"2024-03-31T15:14:09.291875+0000","last_became_peered":"2024-03-31T15:14:09.291875+0000","last_unstale":"2024-03-31T15:14:10.278439+0000","last_undegraded":"2024-03-31T15:14:10.278439+0000","last_fullsized":"2024-03-31T15:14:10.278439+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:27:02.356373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":14,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:11.391479+0000","last_change":"2024-03-31T15:14:08.268133+0000","last_active":"2024-03-31T15:14:11.391479+0000","last_peered":"2024-03-31T15:14:11.391479+0000","last_clean":"2024-03-31T15:14:11.391479+0000","last_became_active":"2024-03-31T15:14:08.267768+0000","last_became_peered":"2024-03-31T15:14:08.267768+0000","last_unstale":"2024-03-31T15:14:11.391479+0000","last_undegraded":"2024-03-31T15:14:11.391479+0000","last_fullsized":"2024-03-31T15:14:11.391479+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_clean_scrub_stamp":"2024-03-31T15:14:07.235046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:55:27.079135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.2","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:10.278552+0000","last_change":"2024-03-31T15:14:09.276195+0000","last_active":"2024-03-31T15:14:10.278552+0000","last_peered":"2024-03-31T15:14:10.278552+0000","last_clean":"2024-03-31T15:14:10.278552+0000","last_became_active":"2024-03-31T15:14:09.275925+0000","last_became_peered":"2024-03-31T15:14:09.275925+0000","last_unstale":"2024-03-31T15:14:10.278552+0000","last_undegraded":"2024-03-31T15:14:10.278552+0000","last_fullsized":"2024-03-31T15:14:10.278552+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T19:39:13.736880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":14,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:10.277017+0000","last_change":"2024-03-31T15:14:08.268809+0000","last_active":"2024-03-31T15:14:10.277017+0000","last_peered":"2024-03-31T15:14:10.277017+0000","last_clean":"2024-03-31T15:14:10.277017+0000","last_became_active":"2024-03-31T15:14:08.268473+0000","last_became_peered":"2024-03-31T15:14:08.268473+0000","last_unstale":"2024-03-31T15:14:10.277017+0000","last_undegraded":"2024-03-31T15:14:10.277017+0000","last_fullsized":"2024-03-31T15:14:10.277017+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_clean_scrub_stamp":"2024-03-31T15:14:07.235046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T16:45:55.109941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.3","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:10.279269+0000","last_change":"2024-03-31T15:14:09.273676+0000","last_active":"2024-03-31T15:14:10.279269+0000","last_peered":"2024-03-31T15:14:10.279269+0000","last_clean":"2024-03-31T15:14:10.279269+0000","last_became_active":"2024-03-31T15:14:09.273367+0000","last_became_peered":"2024-03-31T15:14:09.273367+0000","last_unstale":"2024-03-31T15:14:10.279269+0000","last_undegraded":"2024-03-31T15:14:10.279269+0000","last_fullsized":"2024-03-31T15:14:10.279269+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:10:29.340449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":14,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:10.278567+0000","last_change":"2024-03-31T15:14:08.269651+0000","last_active":"2024-03-31T15:14:10.278567+0000","last_peered":"2024-03-31T15:14:10.278567+0000","last_clean":"2024-03-31T15:14:10.278567+0000","last_became_active":"2024-03-31T15:14:08.269372+0000","last_became_peered":"2024-03-31T15:14:08.269372+0000","last_unstale":"2024-03-31T15:14:10.278567+0000","last_undegraded":"2024-03-31T15:14:10.278567+0000","last_fullsized":"2024-03-31T15:14:10.278567+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_clean_scrub_stamp":"2024-03-31T15:14:07.235046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:22:09.702718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.0","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:10.892507+0000","last_change":"2024-03-31T15:14:09.284384+0000","last_active":"2024-03-31T15:14:10.892507+0000","last_peered":"2024-03-31T15:14:10.892507+0000","last_clean":"2024-03-31T15:14:10.892507+0000","last_became_active":"2024-03-31T15:14:09.283766+0000","last_became_peered":"2024-03-31T15:14:09.283766+0000","last_unstale":"2024-03-31T15:14:10.892507+0000","last_undegraded":"2024-03-31T15:14:10.892507+0000","last_fullsized":"2024-03-31T15:14:10.892507+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:03:38.532236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1","version":"0'0","reported_seq":14,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:11.389640+0000","last_change":"2024-03-31T15:14:08.271032+0000","last_active":"2024-03-31T15:14:11.389640+0000","last_peered":"2024-03-31T15:14:11.389640+0000","last_clean":"2024-03-31T15:14:11.389640+0000","last_became_active":"2024-03-31T15:14:08.270781+0000","last_became_peered":"2024-03-31T15:14:08.270781+0000","last_unstale":"2024-03-31T15:14:11.389640+0000","last_undegraded":"2024-03-31T15:14:11.389640+0000","last_fullsized":"2024-03-31T15:14:11.389640+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_clean_scrub_stamp":"2024-03-31T15:14:07.235046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:59:15.102040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:10.892154+0000","last_change":"2024-03-31T15:14:09.288490+0000","last_active":"2024-03-31T15:14:10.892154+0000","last_peered":"2024-03-31T15:14:10.892154+0000","last_clean":"2024-03-31T15:14:10.892154+0000","last_became_active":"2024-03-31T15:14:09.284507+0000","last_became_peered":"2024-03-31T15:14:09.284507+0000","last_unstale":"2024-03-31T15:14:10.892154+0000","last_undegraded":"2024-03-31T15:14:10.892154+0000","last_fullsized":"2024-03-31T15:14:10.892154+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:09:03.491873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":14,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:10.891910+0000","last_change":"2024-03-31T15:14:08.270270+0000","last_active":"2024-03-31T15:14:10.891910+0000","last_peered":"2024-03-31T15:14:10.891910+0000","last_clean":"2024-03-31T15:14:10.891910+0000","last_became_active":"2024-03-31T15:14:08.269482+0000","last_became_peered":"2024-03-31T15:14:08.269482+0000","last_unstale":"2024-03-31T15:14:10.891910+0000","last_undegraded":"2024-03-31T15:14:10.891910+0000","last_fullsized":"2024-03-31T15:14:10.891910+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_clean_scrub_stamp":"2024-03-31T15:14:07.235046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:48:00.819420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.10","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:11.390177+0000","last_change":"2024-03-31T15:14:09.277917+0000","last_active":"2024-03-31T15:14:11.390177+0000","last_peered":"2024-03-31T15:14:11.390177+0000","last_clean":"2024-03-31T15:14:11.390177+0000","last_became_active":"2024-03-31T15:14:09.277651+0000","last_became_peered":"2024-03-31T15:14:09.277651+0000","last_unstale":"2024-03-31T15:14:11.390177+0000","last_undegraded":"2024-03-31T15:14:11.390177+0000","last_fullsized":"2024-03-31T15:14:11.390177+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T16:10:45.541190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":14,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:10.276816+0000","last_change":"2024-03-31T15:14:08.271941+0000","last_active":"2024-03-31T15:14:10.276816+0000","last_peered":"2024-03-31T15:14:10.276816+0000","last_clean":"2024-03-31T15:14:10.276816+0000","last_became_active":"2024-03-31T15:14:08.271589+0000","last_became_peered":"2024-03-31T15:14:08.271589+0000","last_unstale":"2024-03-31T15:14:10.276816+0000","last_undegraded":"2024-03-31T15:14:10.276816+0000","last_fullsized":"2024-03-31T15:14:10.276816+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_clean_scrub_stamp":"2024-03-31T15:14:07.235046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T19:47:47.083449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":335,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:10.892630+0000","last_change":"2024-03-31T15:03:24.329890+0000","last_active":"2024-03-31T15:14:10.892630+0000","last_peered":"2024-03-31T15:14:10.892630+0000","last_clean":"2024-03-31T15:14:10.892630+0000","last_became_active":"2024-03-31T15:03:24.328882+0000","last_became_peered":"2024-03-31T15:03:24.328882+0000","last_unstale":"2024-03-31T15:14:10.892630+0000","last_undegraded":"2024-03-31T15:14:10.892630+0000","last_fullsized":"2024-03-31T15:14:10.892630+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:22.885535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:22.885535+0000","last_clean_scrub_stamp":"2024-03-31T15:03:22.885535+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:18:29.004296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":17,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":16,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509575,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33084,"kb_used_data":1904,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338756,"statfs":{"total":96636764160,"available":96602886144,"internally_reserved":0,"allocated":1949696,"data_stored":1175057,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509574,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32508,"kb_used_data":1324,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339332,"statfs":{"total":96636764160,"available":96603475968,"internally_reserved":0,"allocated":1355776,"data_stored":584677,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":15,"seq":64424509573,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32500,"kb_used_data":1316,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339340,"statfs":{"total":96636764160,"available":96603484160,"internally_reserved":0,"allocated":1347584,"data_stored":584073,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509572,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36656,"kb_used_data":1316,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335184,"statfs":{"total":96636764160,"available":96599228416,"internally_reserved":0,"allocated":1347584,"data_stored":584073,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542278,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32508,"kb_used_data":1320,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339332,"statfs":{"total":96636764160,"available":96603475968,"internally_reserved":0,"allocated":1351680,"data_stored":584143,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542278,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32512,"kb_used_data":1332,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339328,"statfs":{"total":96636764160,"available":96603471872,"internally_reserved":0,"allocated":1363968,"data_stored":585755,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542278,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32504,"kb_used_data":1324,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339336,"statfs":{"total":96636764160,"available":96603480064,"internally_reserved":0,"allocated":1355776,"data_stored":584197,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542278,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33100,"kb_used_data":1916,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338740,"statfs":{"total":96636764160,"available":96602869760,"internally_reserved":0,"allocated":1961984,"data_stored":1176165,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-31T15:14:18.016 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-31T15:14:18.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:18.162+0000 7f8920600640 1 Processor -- start 2024-03-31T15:14:18.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:18.162+0000 7f8920600640 1 -- start start 2024-03-31T15:14:18.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:18.166+0000 7f8920600640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f8918145e40 0x7f89180cc740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:18.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:18.166+0000 7f8920600640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f89180ccc80 0x7f891806f920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:18.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:18.166+0000 7f8920600640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f89180d0920 con 0x7f891806fe60 2024-03-31T15:14:18.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:18.166+0000 7f8920600640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f89180d06d0 con 0x7f89180ccc80 2024-03-31T15:14:18.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:18.166+0000 7f8920600640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f89180d0c00 con 0x7f8918145e40 2024-03-31T15:14:18.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:18.166+0000 7f891d200640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f89180ccc80 0x7f891806f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:18.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:18.166+0000 7f891d200640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f89180ccc80 0x7f891806f920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42752/0 (socket says 172.21.15.16:42752) 2024-03-31T15:14:18.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:18.166+0000 7f891d200640 1 -- 172.21.15.16:0/432510234 learned_addr learned my addr 172.21.15.16:0/432510234 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:14:18.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:18.166+0000 7f891dc00640 1 --2- 172.21.15.16:0/432510234 >> v2:172.21.15.89:3300/0 conn(0x7f8918145e40 0x7f89180cc740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:18.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:18.166+0000 7f891d200640 1 -- 172.21.15.16:0/432510234 >> v2:172.21.15.89:3300/0 conn(0x7f8918145e40 msgr2=0x7f89180cc740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:14:18.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:18.166+0000 7f891d200640 1 --2- 172.21.15.16:0/432510234 >> v2:172.21.15.89:3300/0 conn(0x7f8918145e40 0x7f89180cc740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:18.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:18.166+0000 7f891d200640 1 -- 172.21.15.16:0/432510234 >> v1:172.21.15.89:6789/0 conn(0x7f891806fe60 legacy=0x7f891815fdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:14:18.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:18.166+0000 7f891d200640 1 -- 172.21.15.16:0/432510234 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89180d1810 con 0x7f89180ccc80 2024-03-31T15:14:18.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:18.166+0000 7f891d200640 1 --2- 172.21.15.16:0/432510234 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f89180ccc80 0x7f891806f920 secure :-1 s=READY pgs=1662 cs=0 l=1 rev1=1 crypto rx=0x7f8918072e60 tx=0x7f890401c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=193e5f9cb6f555ee server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:18.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:18.170+0000 7f8917e00640 1 -- 172.21.15.16:0/432510234 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89040b0070 con 0x7f89180ccc80 2024-03-31T15:14:18.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:18.170+0000 7f8917e00640 1 -- 172.21.15.16:0/432510234 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f89040ab600 con 0x7f89180ccc80 2024-03-31T15:14:18.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:18.170+0000 7f8917e00640 1 -- 172.21.15.16:0/432510234 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89040ab9d0 con 0x7f89180ccc80 2024-03-31T15:14:18.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:18.170+0000 7f8920600640 1 -- 172.21.15.16:0/432510234 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f89180ccc80 msgr2=0x7f891806f920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:18.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:18.170+0000 7f8920600640 1 --2- 172.21.15.16:0/432510234 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f89180ccc80 0x7f891806f920 secure :-1 s=READY pgs=1662 cs=0 l=1 rev1=1 crypto rx=0x7f8918072e60 tx=0x7f890401c520 comp rx=0 tx=0).stop 2024-03-31T15:14:18.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:18.170+0000 7f8920600640 1 -- 172.21.15.16:0/432510234 shutdown_connections 2024-03-31T15:14:18.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:18.170+0000 7f8920600640 1 --2- 172.21.15.16:0/432510234 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f89180ccc80 0x7f891806f920 unknown :-1 s=CLOSED pgs=1662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:18.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:18.170+0000 7f8920600640 1 --2- 172.21.15.16:0/432510234 >> v2:172.21.15.89:3300/0 conn(0x7f8918145e40 0x7f89180cc740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:18.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:18.170+0000 7f8920600640 1 -- 172.21.15.16:0/432510234 >> 172.21.15.16:0/432510234 conn(0x7f8918067330 msgr2=0x7f8918067740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:14:18.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:18.170+0000 7f8920600640 1 -- 172.21.15.16:0/432510234 shutdown_connections 2024-03-31T15:14:18.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:18.170+0000 7f8920600640 1 -- 172.21.15.16:0/432510234 wait complete. 2024-03-31T15:14:18.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:18.170+0000 7f8920600640 1 Processor -- start 2024-03-31T15:14:18.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:18.170+0000 7f8920600640 1 -- start start 2024-03-31T15:14:18.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:18.170+0000 7f8920600640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f891806fe60 0x7f89181426c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:18.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:18.170+0000 7f8920600640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f89180ccc80 0x7f891815e2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:18.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:18.170+0000 7f8920600640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8918145e40 0x7f89180b6690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:18.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:18.170+0000 7f8920600640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f89180b5240 con 0x7f8918145e40 2024-03-31T15:14:18.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:18.170+0000 7f8920600640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f89180b50c0 con 0x7f891806fe60 2024-03-31T15:14:18.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:18.170+0000 7f8920600640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f8918169190 con 0x7f89180ccc80 2024-03-31T15:14:18.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:18.170+0000 7f891e600640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8918145e40 0x7f89180b6690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:18.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:18.170+0000 7f891dc00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f891806fe60 0x7f89181426c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:18.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:18.174+0000 7f891dc00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f891806fe60 0x7f89181426c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.16:45168/0 (socket says 172.21.15.16:45168) 2024-03-31T15:14:18.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:18.174+0000 7f891dc00640 1 -- 172.21.15.16:0/4109792619 learned_addr learned my addr 172.21.15.16:0/4109792619 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:14:18.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:18.174+0000 7f891d200640 1 --2- 172.21.15.16:0/4109792619 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f89180ccc80 0x7f891815e2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:18.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:18.174+0000 7f891e600640 1 -- 172.21.15.16:0/4109792619 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f89180ccc80 msgr2=0x7f891815e2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:18.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:18.174+0000 7f891e600640 1 --2- 172.21.15.16:0/4109792619 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f89180ccc80 0x7f891815e2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:18.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:18.174+0000 7f891e600640 1 -- 172.21.15.16:0/4109792619 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f891806fe60 msgr2=0x7f89181426c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:18.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:18.174+0000 7f891e600640 1 --2- 172.21.15.16:0/4109792619 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f891806fe60 0x7f89181426c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:18.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:18.174+0000 7f891e600640 1 -- 172.21.15.16:0/4109792619 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f891815ef00 con 0x7f8918145e40 2024-03-31T15:14:18.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:18.174+0000 7f891dc00640 1 --2- 172.21.15.16:0/4109792619 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f891806fe60 0x7f89181426c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:14:18.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:18.174+0000 7f891e600640 1 --2- 172.21.15.16:0/4109792619 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8918145e40 0x7f89180b6690 secure :-1 s=READY pgs=1663 cs=0 l=1 rev1=1 crypto rx=0x7f891000c730 tx=0x7f891000cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:18.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:18.174+0000 7f8915600640 1 -- 172.21.15.16:0/4109792619 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8910003b80 con 0x7f8918145e40 2024-03-31T15:14:18.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:18.174+0000 7f8915600640 1 -- 172.21.15.16:0/4109792619 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8910003d20 con 0x7f8918145e40 2024-03-31T15:14:18.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:18.174+0000 7f8920600640 1 -- 172.21.15.16:0/4109792619 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f891815ec20 con 0x7f8918145e40 2024-03-31T15:14:18.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:18.174+0000 7f8915600640 1 -- 172.21.15.16:0/4109792619 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f891000aa30 con 0x7f8918145e40 2024-03-31T15:14:18.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:18.174+0000 7f8920600640 1 -- 172.21.15.16:0/4109792619 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8918143850 con 0x7f8918145e40 2024-03-31T15:14:18.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:18.174+0000 7f8920600640 1 -- 172.21.15.16:0/4109792619 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8918069950 con 0x7f8918145e40 2024-03-31T15:14:18.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:18.174+0000 7f8915600640 1 -- 172.21.15.16:0/4109792619 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f891000abd0 con 0x7f8918145e40 2024-03-31T15:14:18.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:18.174+0000 7f8915600640 1 --2- 172.21.15.16:0/4109792619 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f88f8068380 0x7f88f806a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:18.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:18.178+0000 7f8915600640 1 -- 172.21.15.16:0/4109792619 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 5709+0+0 (secure 0 0 0) 0x7f8910086770 con 0x7f8918145e40 2024-03-31T15:14:18.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:18.178+0000 7f891dc00640 1 --2- 172.21.15.16:0/4109792619 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f88f8068380 0x7f88f806a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:18.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:18.178+0000 7f891dc00640 1 --2- 172.21.15.16:0/4109792619 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f88f8068380 0x7f88f806a840 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f890c007e60 tx=0x7f890c008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:18.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:18.182+0000 7f8915600640 1 -- 172.21.15.16:0/4109792619 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f8910056640 con 0x7f8918145e40 2024-03-31T15:14:18.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:18.346+0000 7f8920600640 1 -- 172.21.15.16:0/4109792619 --> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f89180d2020 con 0x7f88f8068380 2024-03-31T15:14:18.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:18.354+0000 7f8915600640 1 -- 172.21.15.16:0/4109792619 <== mgr.4100 v2:172.21.15.89:6832/1549407273 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252435 (secure 0 0 0) 0x7f89180d2020 con 0x7f88f8068380 2024-03-31T15:14:18.348 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:14:18.350 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-03-31T15:14:18.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:18.358+0000 7f8920600640 1 -- 172.21.15.16:0/4109792619 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f88f8068380 msgr2=0x7f88f806a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:18.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:18.358+0000 7f8920600640 1 --2- 172.21.15.16:0/4109792619 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f88f8068380 0x7f88f806a840 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f890c007e60 tx=0x7f890c008040 comp rx=0 tx=0).stop 2024-03-31T15:14:18.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:18.358+0000 7f8920600640 1 -- 172.21.15.16:0/4109792619 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8918145e40 msgr2=0x7f89180b6690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:18.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:18.358+0000 7f8920600640 1 --2- 172.21.15.16:0/4109792619 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8918145e40 0x7f89180b6690 secure :-1 s=READY pgs=1663 cs=0 l=1 rev1=1 crypto rx=0x7f891000c730 tx=0x7f891000cbf0 comp rx=0 tx=0).stop 2024-03-31T15:14:18.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:18.358+0000 7f8920600640 1 -- 172.21.15.16:0/4109792619 shutdown_connections 2024-03-31T15:14:18.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:18.358+0000 7f8920600640 1 --2- 172.21.15.16:0/4109792619 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f88f8068380 0x7f88f806a840 unknown :-1 s=CLOSED pgs=709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:18.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:18.358+0000 7f8920600640 1 --2- 172.21.15.16:0/4109792619 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8918145e40 0x7f89180b6690 unknown :-1 s=CLOSED pgs=1663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:18.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:18.358+0000 7f8920600640 1 --2- 172.21.15.16:0/4109792619 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f89180ccc80 0x7f891815e2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:18.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:18.358+0000 7f8920600640 1 --2- 172.21.15.16:0/4109792619 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f891806fe60 0x7f89181426c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:18.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:18.358+0000 7f8920600640 1 -- 172.21.15.16:0/4109792619 >> 172.21.15.16:0/4109792619 conn(0x7f8918067330 msgr2=0x7f89180d2540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:14:18.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:18.358+0000 7f8920600640 1 -- 172.21.15.16:0/4109792619 shutdown_connections 2024-03-31T15:14:18.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:18.358+0000 7f8920600640 1 -- 172.21.15.16:0/4109792619 wait complete. 2024-03-31T15:14:18.368 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":430,"stamp":"2024-03-31T15:14:18.316597+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":265372,"kb_used_data":11752,"kb_used_omap":12,"kb_used_meta":253491,"kb_avail":754709348,"statfs":{"total":773094113280,"available":772822372352,"internally_reserved":0,"allocated":12034048,"data_stored":5858140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":259575344},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.054318"},"pg_stats":[{"pgid":"17.3f","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:10.892463+0000","last_change":"2024-03-31T15:14:09.277746+0000","last_active":"2024-03-31T15:14:10.892463+0000","last_peered":"2024-03-31T15:14:10.892463+0000","last_clean":"2024-03-31T15:14:10.892463+0000","last_became_active":"2024-03-31T15:14:09.277207+0000","last_became_peered":"2024-03-31T15:14:09.277207+0000","last_unstale":"2024-03-31T15:14:10.892463+0000","last_undegraded":"2024-03-31T15:14:10.892463+0000","last_fullsized":"2024-03-31T15:14:10.892463+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T17:42:57.076247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3e","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:10.279940+0000","last_change":"2024-03-31T15:14:09.287185+0000","last_active":"2024-03-31T15:14:10.279940+0000","last_peered":"2024-03-31T15:14:10.279940+0000","last_clean":"2024-03-31T15:14:10.279940+0000","last_became_active":"2024-03-31T15:14:09.286886+0000","last_became_peered":"2024-03-31T15:14:09.286886+0000","last_unstale":"2024-03-31T15:14:10.279940+0000","last_undegraded":"2024-03-31T15:14:10.279940+0000","last_fullsized":"2024-03-31T15:14:10.279940+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:49:32.932894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.3d","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:10.892344+0000","last_change":"2024-03-31T15:14:09.287821+0000","last_active":"2024-03-31T15:14:10.892344+0000","last_peered":"2024-03-31T15:14:10.892344+0000","last_clean":"2024-03-31T15:14:10.892344+0000","last_became_active":"2024-03-31T15:14:09.287214+0000","last_became_peered":"2024-03-31T15:14:09.287214+0000","last_unstale":"2024-03-31T15:14:10.892344+0000","last_undegraded":"2024-03-31T15:14:10.892344+0000","last_fullsized":"2024-03-31T15:14:10.892344+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T17:45:10.698978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.3c","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:10.892010+0000","last_change":"2024-03-31T15:14:09.276706+0000","last_active":"2024-03-31T15:14:10.892010+0000","last_peered":"2024-03-31T15:14:10.892010+0000","last_clean":"2024-03-31T15:14:10.892010+0000","last_became_active":"2024-03-31T15:14:09.275545+0000","last_became_peered":"2024-03-31T15:14:09.275545+0000","last_unstale":"2024-03-31T15:14:10.892010+0000","last_undegraded":"2024-03-31T15:14:10.892010+0000","last_fullsized":"2024-03-31T15:14:10.892010+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T19:37:12.397888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3b","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:11.390113+0000","last_change":"2024-03-31T15:14:09.276514+0000","last_active":"2024-03-31T15:14:11.390113+0000","last_peered":"2024-03-31T15:14:11.390113+0000","last_clean":"2024-03-31T15:14:11.390113+0000","last_became_active":"2024-03-31T15:14:09.276241+0000","last_became_peered":"2024-03-31T15:14:09.276241+0000","last_unstale":"2024-03-31T15:14:11.390113+0000","last_undegraded":"2024-03-31T15:14:11.390113+0000","last_fullsized":"2024-03-31T15:14:11.390113+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:42:03.309978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.3a","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:11.390994+0000","last_change":"2024-03-31T15:14:09.283260+0000","last_active":"2024-03-31T15:14:11.390994+0000","last_peered":"2024-03-31T15:14:11.390994+0000","last_clean":"2024-03-31T15:14:11.390994+0000","last_became_active":"2024-03-31T15:14:09.282995+0000","last_became_peered":"2024-03-31T15:14:09.282995+0000","last_unstale":"2024-03-31T15:14:11.390994+0000","last_undegraded":"2024-03-31T15:14:11.390994+0000","last_fullsized":"2024-03-31T15:14:11.390994+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:27:30.858064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.39","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:10.276772+0000","last_change":"2024-03-31T15:14:09.275503+0000","last_active":"2024-03-31T15:14:10.276772+0000","last_peered":"2024-03-31T15:14:10.276772+0000","last_clean":"2024-03-31T15:14:10.276772+0000","last_became_active":"2024-03-31T15:14:09.275190+0000","last_became_peered":"2024-03-31T15:14:09.275190+0000","last_unstale":"2024-03-31T15:14:10.276772+0000","last_undegraded":"2024-03-31T15:14:10.276772+0000","last_fullsized":"2024-03-31T15:14:10.276772+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:22:10.709767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.38","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:10.892255+0000","last_change":"2024-03-31T15:14:09.290122+0000","last_active":"2024-03-31T15:14:10.892255+0000","last_peered":"2024-03-31T15:14:10.892255+0000","last_clean":"2024-03-31T15:14:10.892255+0000","last_became_active":"2024-03-31T15:14:09.289694+0000","last_became_peered":"2024-03-31T15:14:09.289694+0000","last_unstale":"2024-03-31T15:14:10.892255+0000","last_undegraded":"2024-03-31T15:14:10.892255+0000","last_fullsized":"2024-03-31T15:14:10.892255+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:54:09.048015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.37","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:10.892467+0000","last_change":"2024-03-31T15:14:09.284680+0000","last_active":"2024-03-31T15:14:10.892467+0000","last_peered":"2024-03-31T15:14:10.892467+0000","last_clean":"2024-03-31T15:14:10.892467+0000","last_became_active":"2024-03-31T15:14:09.282533+0000","last_became_peered":"2024-03-31T15:14:09.282533+0000","last_unstale":"2024-03-31T15:14:10.892467+0000","last_undegraded":"2024-03-31T15:14:10.892467+0000","last_fullsized":"2024-03-31T15:14:10.892467+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:04:35.760087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.36","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:10.892246+0000","last_change":"2024-03-31T15:14:09.290368+0000","last_active":"2024-03-31T15:14:10.892246+0000","last_peered":"2024-03-31T15:14:10.892246+0000","last_clean":"2024-03-31T15:14:10.892246+0000","last_became_active":"2024-03-31T15:14:09.290077+0000","last_became_peered":"2024-03-31T15:14:09.290077+0000","last_unstale":"2024-03-31T15:14:10.892246+0000","last_undegraded":"2024-03-31T15:14:10.892246+0000","last_fullsized":"2024-03-31T15:14:10.892246+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:23:53.326762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.35","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:10.279841+0000","last_change":"2024-03-31T15:14:09.274713+0000","last_active":"2024-03-31T15:14:10.279841+0000","last_peered":"2024-03-31T15:14:10.279841+0000","last_clean":"2024-03-31T15:14:10.279841+0000","last_became_active":"2024-03-31T15:14:09.274389+0000","last_became_peered":"2024-03-31T15:14:09.274389+0000","last_unstale":"2024-03-31T15:14:10.279841+0000","last_undegraded":"2024-03-31T15:14:10.279841+0000","last_fullsized":"2024-03-31T15:14:10.279841+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:48:41.198019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.34","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:10.892056+0000","last_change":"2024-03-31T15:14:09.276847+0000","last_active":"2024-03-31T15:14:10.892056+0000","last_peered":"2024-03-31T15:14:10.892056+0000","last_clean":"2024-03-31T15:14:10.892056+0000","last_became_active":"2024-03-31T15:14:09.275249+0000","last_became_peered":"2024-03-31T15:14:09.275249+0000","last_unstale":"2024-03-31T15:14:10.892056+0000","last_undegraded":"2024-03-31T15:14:10.892056+0000","last_fullsized":"2024-03-31T15:14:10.892056+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T16:56:09.377594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.33","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:10.892611+0000","last_change":"2024-03-31T15:14:09.292405+0000","last_active":"2024-03-31T15:14:10.892611+0000","last_peered":"2024-03-31T15:14:10.892611+0000","last_clean":"2024-03-31T15:14:10.892611+0000","last_became_active":"2024-03-31T15:14:09.292107+0000","last_became_peered":"2024-03-31T15:14:09.292107+0000","last_unstale":"2024-03-31T15:14:10.892611+0000","last_undegraded":"2024-03-31T15:14:10.892611+0000","last_fullsized":"2024-03-31T15:14:10.892611+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:55:14.681666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.32","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:10.892110+0000","last_change":"2024-03-31T15:14:09.286608+0000","last_active":"2024-03-31T15:14:10.892110+0000","last_peered":"2024-03-31T15:14:10.892110+0000","last_clean":"2024-03-31T15:14:10.892110+0000","last_became_active":"2024-03-31T15:14:09.285944+0000","last_became_peered":"2024-03-31T15:14:09.285944+0000","last_unstale":"2024-03-31T15:14:10.892110+0000","last_undegraded":"2024-03-31T15:14:10.892110+0000","last_fullsized":"2024-03-31T15:14:10.892110+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:51:29.224313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.31","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:11.390625+0000","last_change":"2024-03-31T15:14:09.285784+0000","last_active":"2024-03-31T15:14:11.390625+0000","last_peered":"2024-03-31T15:14:11.390625+0000","last_clean":"2024-03-31T15:14:11.390625+0000","last_became_active":"2024-03-31T15:14:09.285374+0000","last_became_peered":"2024-03-31T15:14:09.285374+0000","last_unstale":"2024-03-31T15:14:11.390625+0000","last_undegraded":"2024-03-31T15:14:11.390625+0000","last_fullsized":"2024-03-31T15:14:11.390625+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:34:11.338914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.30","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:10.891716+0000","last_change":"2024-03-31T15:14:09.287510+0000","last_active":"2024-03-31T15:14:10.891716+0000","last_peered":"2024-03-31T15:14:10.891716+0000","last_clean":"2024-03-31T15:14:10.891716+0000","last_became_active":"2024-03-31T15:14:09.287238+0000","last_became_peered":"2024-03-31T15:14:09.287238+0000","last_unstale":"2024-03-31T15:14:10.891716+0000","last_undegraded":"2024-03-31T15:14:10.891716+0000","last_fullsized":"2024-03-31T15:14:10.891716+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:14:09.225080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.2f","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:10.279850+0000","last_change":"2024-03-31T15:14:09.277762+0000","last_active":"2024-03-31T15:14:10.279850+0000","last_peered":"2024-03-31T15:14:10.279850+0000","last_clean":"2024-03-31T15:14:10.279850+0000","last_became_active":"2024-03-31T15:14:09.277487+0000","last_became_peered":"2024-03-31T15:14:09.277487+0000","last_unstale":"2024-03-31T15:14:10.279850+0000","last_undegraded":"2024-03-31T15:14:10.279850+0000","last_fullsized":"2024-03-31T15:14:10.279850+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:15:07.162929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2e","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:10.279438+0000","last_change":"2024-03-31T15:14:09.276215+0000","last_active":"2024-03-31T15:14:10.279438+0000","last_peered":"2024-03-31T15:14:10.279438+0000","last_clean":"2024-03-31T15:14:10.279438+0000","last_became_active":"2024-03-31T15:14:09.275956+0000","last_became_peered":"2024-03-31T15:14:09.275956+0000","last_unstale":"2024-03-31T15:14:10.279438+0000","last_undegraded":"2024-03-31T15:14:10.279438+0000","last_fullsized":"2024-03-31T15:14:10.279438+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:54:50.675440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.2d","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:10.892501+0000","last_change":"2024-03-31T15:14:09.287937+0000","last_active":"2024-03-31T15:14:10.892501+0000","last_peered":"2024-03-31T15:14:10.892501+0000","last_clean":"2024-03-31T15:14:10.892501+0000","last_became_active":"2024-03-31T15:14:09.287391+0000","last_became_peered":"2024-03-31T15:14:09.287391+0000","last_unstale":"2024-03-31T15:14:10.892501+0000","last_undegraded":"2024-03-31T15:14:10.892501+0000","last_fullsized":"2024-03-31T15:14:10.892501+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:36:58.066600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.2c","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:10.279545+0000","last_change":"2024-03-31T15:14:09.283362+0000","last_active":"2024-03-31T15:14:10.279545+0000","last_peered":"2024-03-31T15:14:10.279545+0000","last_clean":"2024-03-31T15:14:10.279545+0000","last_became_active":"2024-03-31T15:14:09.283040+0000","last_became_peered":"2024-03-31T15:14:09.283040+0000","last_unstale":"2024-03-31T15:14:10.279545+0000","last_undegraded":"2024-03-31T15:14:10.279545+0000","last_fullsized":"2024-03-31T15:14:10.279545+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:43:36.989252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2b","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:10.279313+0000","last_change":"2024-03-31T15:14:09.278966+0000","last_active":"2024-03-31T15:14:10.279313+0000","last_peered":"2024-03-31T15:14:10.279313+0000","last_clean":"2024-03-31T15:14:10.279313+0000","last_became_active":"2024-03-31T15:14:09.278453+0000","last_became_peered":"2024-03-31T15:14:09.278453+0000","last_unstale":"2024-03-31T15:14:10.279313+0000","last_undegraded":"2024-03-31T15:14:10.279313+0000","last_fullsized":"2024-03-31T15:14:10.279313+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:16:53.323124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2a","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:11.390332+0000","last_change":"2024-03-31T15:14:09.275037+0000","last_active":"2024-03-31T15:14:11.390332+0000","last_peered":"2024-03-31T15:14:11.390332+0000","last_clean":"2024-03-31T15:14:11.390332+0000","last_became_active":"2024-03-31T15:14:09.274177+0000","last_became_peered":"2024-03-31T15:14:09.274177+0000","last_unstale":"2024-03-31T15:14:11.390332+0000","last_undegraded":"2024-03-31T15:14:11.390332+0000","last_fullsized":"2024-03-31T15:14:11.390332+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:09:48.241483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.29","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:10.277953+0000","last_change":"2024-03-31T15:14:09.279623+0000","last_active":"2024-03-31T15:14:10.277953+0000","last_peered":"2024-03-31T15:14:10.277953+0000","last_clean":"2024-03-31T15:14:10.277953+0000","last_became_active":"2024-03-31T15:14:09.279356+0000","last_became_peered":"2024-03-31T15:14:09.279356+0000","last_unstale":"2024-03-31T15:14:10.277953+0000","last_undegraded":"2024-03-31T15:14:10.277953+0000","last_fullsized":"2024-03-31T15:14:10.277953+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:20:10.912650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.28","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:11.390297+0000","last_change":"2024-03-31T15:14:09.283380+0000","last_active":"2024-03-31T15:14:11.390297+0000","last_peered":"2024-03-31T15:14:11.390297+0000","last_clean":"2024-03-31T15:14:11.390297+0000","last_became_active":"2024-03-31T15:14:09.283097+0000","last_became_peered":"2024-03-31T15:14:09.283097+0000","last_unstale":"2024-03-31T15:14:11.390297+0000","last_undegraded":"2024-03-31T15:14:11.390297+0000","last_fullsized":"2024-03-31T15:14:11.390297+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:00:11.817601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.27","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:10.280178+0000","last_change":"2024-03-31T15:14:09.277583+0000","last_active":"2024-03-31T15:14:10.280178+0000","last_peered":"2024-03-31T15:14:10.280178+0000","last_clean":"2024-03-31T15:14:10.280178+0000","last_became_active":"2024-03-31T15:14:09.277320+0000","last_became_peered":"2024-03-31T15:14:09.277320+0000","last_unstale":"2024-03-31T15:14:10.280178+0000","last_undegraded":"2024-03-31T15:14:10.280178+0000","last_fullsized":"2024-03-31T15:14:10.280178+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:48:47.011312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.26","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:10.892253+0000","last_change":"2024-03-31T15:14:09.276124+0000","last_active":"2024-03-31T15:14:10.892253+0000","last_peered":"2024-03-31T15:14:10.892253+0000","last_clean":"2024-03-31T15:14:10.892253+0000","last_became_active":"2024-03-31T15:14:09.275495+0000","last_became_peered":"2024-03-31T15:14:09.275495+0000","last_unstale":"2024-03-31T15:14:10.892253+0000","last_undegraded":"2024-03-31T15:14:10.892253+0000","last_fullsized":"2024-03-31T15:14:10.892253+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:38:53.950142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.25","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:11.391422+0000","last_change":"2024-03-31T15:14:09.290617+0000","last_active":"2024-03-31T15:14:11.391422+0000","last_peered":"2024-03-31T15:14:11.391422+0000","last_clean":"2024-03-31T15:14:11.391422+0000","last_became_active":"2024-03-31T15:14:09.290308+0000","last_became_peered":"2024-03-31T15:14:09.290308+0000","last_unstale":"2024-03-31T15:14:11.391422+0000","last_undegraded":"2024-03-31T15:14:11.391422+0000","last_fullsized":"2024-03-31T15:14:11.391422+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:43:42.509329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.24","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:11.391301+0000","last_change":"2024-03-31T15:14:09.290865+0000","last_active":"2024-03-31T15:14:11.391301+0000","last_peered":"2024-03-31T15:14:11.391301+0000","last_clean":"2024-03-31T15:14:11.391301+0000","last_became_active":"2024-03-31T15:14:09.290601+0000","last_became_peered":"2024-03-31T15:14:09.290601+0000","last_unstale":"2024-03-31T15:14:11.391301+0000","last_undegraded":"2024-03-31T15:14:11.391301+0000","last_fullsized":"2024-03-31T15:14:11.391301+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T19:21:02.579812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.23","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:10.278903+0000","last_change":"2024-03-31T15:14:09.278728+0000","last_active":"2024-03-31T15:14:10.278903+0000","last_peered":"2024-03-31T15:14:10.278903+0000","last_clean":"2024-03-31T15:14:10.278903+0000","last_became_active":"2024-03-31T15:14:09.278092+0000","last_became_peered":"2024-03-31T15:14:09.278092+0000","last_unstale":"2024-03-31T15:14:10.278903+0000","last_undegraded":"2024-03-31T15:14:10.278903+0000","last_fullsized":"2024-03-31T15:14:10.278903+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:49:03.965468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.22","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:10.277030+0000","last_change":"2024-03-31T15:14:09.279075+0000","last_active":"2024-03-31T15:14:10.277030+0000","last_peered":"2024-03-31T15:14:10.277030+0000","last_clean":"2024-03-31T15:14:10.277030+0000","last_became_active":"2024-03-31T15:14:09.278785+0000","last_became_peered":"2024-03-31T15:14:09.278785+0000","last_unstale":"2024-03-31T15:14:10.277030+0000","last_undegraded":"2024-03-31T15:14:10.277030+0000","last_fullsized":"2024-03-31T15:14:10.277030+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:24:12.224782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.21","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:10.892072+0000","last_change":"2024-03-31T15:14:09.276137+0000","last_active":"2024-03-31T15:14:10.892072+0000","last_peered":"2024-03-31T15:14:10.892072+0000","last_clean":"2024-03-31T15:14:10.892072+0000","last_became_active":"2024-03-31T15:14:09.275873+0000","last_became_peered":"2024-03-31T15:14:09.275873+0000","last_unstale":"2024-03-31T15:14:10.892072+0000","last_undegraded":"2024-03-31T15:14:10.892072+0000","last_fullsized":"2024-03-31T15:14:10.892072+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T17:03:26.938790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.20","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:11.389806+0000","last_change":"2024-03-31T15:14:09.285377+0000","last_active":"2024-03-31T15:14:11.389806+0000","last_peered":"2024-03-31T15:14:11.389806+0000","last_clean":"2024-03-31T15:14:11.389806+0000","last_became_active":"2024-03-31T15:14:09.285077+0000","last_became_peered":"2024-03-31T15:14:09.285077+0000","last_unstale":"2024-03-31T15:14:11.389806+0000","last_undegraded":"2024-03-31T15:14:11.389806+0000","last_fullsized":"2024-03-31T15:14:11.389806+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T15:39:45.840997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.1e","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:11.391059+0000","last_change":"2024-03-31T15:14:09.286260+0000","last_active":"2024-03-31T15:14:11.391059+0000","last_peered":"2024-03-31T15:14:11.391059+0000","last_clean":"2024-03-31T15:14:11.391059+0000","last_became_active":"2024-03-31T15:14:09.285985+0000","last_became_peered":"2024-03-31T15:14:09.285985+0000","last_unstale":"2024-03-31T15:14:11.391059+0000","last_undegraded":"2024-03-31T15:14:11.391059+0000","last_fullsized":"2024-03-31T15:14:11.391059+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T15:58:28.943455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1f","version":"108'4","reported_seq":18,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:10.277222+0000","last_change":"2024-03-31T15:14:08.268805+0000","last_active":"2024-03-31T15:14:10.277222+0000","last_peered":"2024-03-31T15:14:10.277222+0000","last_clean":"2024-03-31T15:14:10.277222+0000","last_became_active":"2024-03-31T15:14:08.268515+0000","last_became_peered":"2024-03-31T15:14:08.268515+0000","last_unstale":"2024-03-31T15:14:10.277222+0000","last_undegraded":"2024-03-31T15:14:10.277222+0000","last_fullsized":"2024-03-31T15:14:10.277222+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_clean_scrub_stamp":"2024-03-31T15:14:07.235046+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:16:55.261359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1f","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:10.278621+0000","last_change":"2024-03-31T15:14:09.281053+0000","last_active":"2024-03-31T15:14:10.278621+0000","last_peered":"2024-03-31T15:14:10.278621+0000","last_clean":"2024-03-31T15:14:10.278621+0000","last_became_active":"2024-03-31T15:14:09.280735+0000","last_became_peered":"2024-03-31T15:14:09.280735+0000","last_unstale":"2024-03-31T15:14:10.278621+0000","last_undegraded":"2024-03-31T15:14:10.278621+0000","last_fullsized":"2024-03-31T15:14:10.278621+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:55:28.086183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.1e","version":"108'1","reported_seq":15,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:10.892384+0000","last_change":"2024-03-31T15:14:08.270517+0000","last_active":"2024-03-31T15:14:10.892384+0000","last_peered":"2024-03-31T15:14:10.892384+0000","last_clean":"2024-03-31T15:14:10.892384+0000","last_became_active":"2024-03-31T15:14:08.270219+0000","last_became_peered":"2024-03-31T15:14:08.270219+0000","last_unstale":"2024-03-31T15:14:10.892384+0000","last_undegraded":"2024-03-31T15:14:10.892384+0000","last_fullsized":"2024-03-31T15:14:10.892384+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_clean_scrub_stamp":"2024-03-31T15:14:07.235046+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:06:47.238926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1c","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:11.391070+0000","last_change":"2024-03-31T15:14:09.285776+0000","last_active":"2024-03-31T15:14:11.391070+0000","last_peered":"2024-03-31T15:14:11.391070+0000","last_clean":"2024-03-31T15:14:11.391070+0000","last_became_active":"2024-03-31T15:14:09.285383+0000","last_became_peered":"2024-03-31T15:14:09.285383+0000","last_unstale":"2024-03-31T15:14:11.391070+0000","last_undegraded":"2024-03-31T15:14:11.391070+0000","last_fullsized":"2024-03-31T15:14:11.391070+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:45:29.558658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1d","version":"108'2","reported_seq":16,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:11.390584+0000","last_change":"2024-03-31T15:14:08.268956+0000","last_active":"2024-03-31T15:14:11.390584+0000","last_peered":"2024-03-31T15:14:11.390584+0000","last_clean":"2024-03-31T15:14:11.390584+0000","last_became_active":"2024-03-31T15:14:08.268672+0000","last_became_peered":"2024-03-31T15:14:08.268672+0000","last_unstale":"2024-03-31T15:14:11.390584+0000","last_undegraded":"2024-03-31T15:14:11.390584+0000","last_fullsized":"2024-03-31T15:14:11.390584+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_clean_scrub_stamp":"2024-03-31T15:14:07.235046+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T15:35:25.437508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1d","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:10.892687+0000","last_change":"2024-03-31T15:14:09.288130+0000","last_active":"2024-03-31T15:14:10.892687+0000","last_peered":"2024-03-31T15:14:10.892687+0000","last_clean":"2024-03-31T15:14:10.892687+0000","last_became_active":"2024-03-31T15:14:09.287597+0000","last_became_peered":"2024-03-31T15:14:09.287597+0000","last_unstale":"2024-03-31T15:14:10.892687+0000","last_undegraded":"2024-03-31T15:14:10.892687+0000","last_fullsized":"2024-03-31T15:14:10.892687+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:02:21.465704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1c","version":"108'2","reported_seq":16,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:11.391531+0000","last_change":"2024-03-31T15:14:08.269987+0000","last_active":"2024-03-31T15:14:11.391531+0000","last_peered":"2024-03-31T15:14:11.391531+0000","last_clean":"2024-03-31T15:14:11.391531+0000","last_became_active":"2024-03-31T15:14:08.269241+0000","last_became_peered":"2024-03-31T15:14:08.269241+0000","last_unstale":"2024-03-31T15:14:11.391531+0000","last_undegraded":"2024-03-31T15:14:11.391531+0000","last_fullsized":"2024-03-31T15:14:11.391531+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_clean_scrub_stamp":"2024-03-31T15:14:07.235046+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:56:08.718098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1a","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:10.891958+0000","last_change":"2024-03-31T15:14:09.282143+0000","last_active":"2024-03-31T15:14:10.891958+0000","last_peered":"2024-03-31T15:14:10.891958+0000","last_clean":"2024-03-31T15:14:10.891958+0000","last_became_active":"2024-03-31T15:14:09.281851+0000","last_became_peered":"2024-03-31T15:14:09.281851+0000","last_unstale":"2024-03-31T15:14:10.891958+0000","last_undegraded":"2024-03-31T15:14:10.891958+0000","last_fullsized":"2024-03-31T15:14:10.891958+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:07:40.319619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.1b","version":"108'1","reported_seq":15,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:10.276959+0000","last_change":"2024-03-31T15:14:08.271392+0000","last_active":"2024-03-31T15:14:10.276959+0000","last_peered":"2024-03-31T15:14:10.276959+0000","last_clean":"2024-03-31T15:14:10.276959+0000","last_became_active":"2024-03-31T15:14:08.271138+0000","last_became_peered":"2024-03-31T15:14:08.271138+0000","last_unstale":"2024-03-31T15:14:10.276959+0000","last_undegraded":"2024-03-31T15:14:10.276959+0000","last_fullsized":"2024-03-31T15:14:10.276959+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_clean_scrub_stamp":"2024-03-31T15:14:07.235046+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T16:18:45.057526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1b","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:10.279089+0000","last_change":"2024-03-31T15:14:09.288830+0000","last_active":"2024-03-31T15:14:10.279089+0000","last_peered":"2024-03-31T15:14:10.279089+0000","last_clean":"2024-03-31T15:14:10.279089+0000","last_became_active":"2024-03-31T15:14:09.288486+0000","last_became_peered":"2024-03-31T15:14:09.288486+0000","last_unstale":"2024-03-31T15:14:10.279089+0000","last_undegraded":"2024-03-31T15:14:10.279089+0000","last_fullsized":"2024-03-31T15:14:10.279089+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:53:27.090923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.1a","version":"108'1","reported_seq":15,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:11.390384+0000","last_change":"2024-03-31T15:14:08.273955+0000","last_active":"2024-03-31T15:14:11.390384+0000","last_peered":"2024-03-31T15:14:11.390384+0000","last_clean":"2024-03-31T15:14:11.390384+0000","last_became_active":"2024-03-31T15:14:08.273646+0000","last_became_peered":"2024-03-31T15:14:08.273646+0000","last_unstale":"2024-03-31T15:14:11.390384+0000","last_undegraded":"2024-03-31T15:14:11.390384+0000","last_fullsized":"2024-03-31T15:14:11.390384+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_clean_scrub_stamp":"2024-03-31T15:14:07.235046+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:27:01.349324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.18","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:10.891733+0000","last_change":"2024-03-31T15:14:09.278486+0000","last_active":"2024-03-31T15:14:10.891733+0000","last_peered":"2024-03-31T15:14:10.891733+0000","last_clean":"2024-03-31T15:14:10.891733+0000","last_became_active":"2024-03-31T15:14:09.278193+0000","last_became_peered":"2024-03-31T15:14:09.278193+0000","last_unstale":"2024-03-31T15:14:10.891733+0000","last_undegraded":"2024-03-31T15:14:10.891733+0000","last_fullsized":"2024-03-31T15:14:10.891733+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:42:40.870161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":14,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:10.278054+0000","last_change":"2024-03-31T15:14:08.266416+0000","last_active":"2024-03-31T15:14:10.278054+0000","last_peered":"2024-03-31T15:14:10.278054+0000","last_clean":"2024-03-31T15:14:10.278054+0000","last_became_active":"2024-03-31T15:14:08.266116+0000","last_became_peered":"2024-03-31T15:14:08.266116+0000","last_unstale":"2024-03-31T15:14:10.278054+0000","last_undegraded":"2024-03-31T15:14:10.278054+0000","last_fullsized":"2024-03-31T15:14:10.278054+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_clean_scrub_stamp":"2024-03-31T15:14:07.235046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:24:11.217734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.19","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:10.276686+0000","last_change":"2024-03-31T15:14:09.271307+0000","last_active":"2024-03-31T15:14:10.276686+0000","last_peered":"2024-03-31T15:14:10.276686+0000","last_clean":"2024-03-31T15:14:10.276686+0000","last_became_active":"2024-03-31T15:14:09.271017+0000","last_became_peered":"2024-03-31T15:14:09.271017+0000","last_unstale":"2024-03-31T15:14:10.276686+0000","last_undegraded":"2024-03-31T15:14:10.276686+0000","last_fullsized":"2024-03-31T15:14:10.276686+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:15:09.198042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":14,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:10.891808+0000","last_change":"2024-03-31T15:14:08.276025+0000","last_active":"2024-03-31T15:14:10.891808+0000","last_peered":"2024-03-31T15:14:10.891808+0000","last_clean":"2024-03-31T15:14:10.891808+0000","last_became_active":"2024-03-31T15:14:08.275541+0000","last_became_peered":"2024-03-31T15:14:08.275541+0000","last_unstale":"2024-03-31T15:14:10.891808+0000","last_undegraded":"2024-03-31T15:14:10.891808+0000","last_fullsized":"2024-03-31T15:14:10.891808+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_clean_scrub_stamp":"2024-03-31T15:14:07.235046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:34:33.623018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.16","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:10.277387+0000","last_change":"2024-03-31T15:14:09.273769+0000","last_active":"2024-03-31T15:14:10.277387+0000","last_peered":"2024-03-31T15:14:10.277387+0000","last_clean":"2024-03-31T15:14:10.277387+0000","last_became_active":"2024-03-31T15:14:09.273470+0000","last_became_peered":"2024-03-31T15:14:09.273470+0000","last_unstale":"2024-03-31T15:14:10.277387+0000","last_undegraded":"2024-03-31T15:14:10.277387+0000","last_fullsized":"2024-03-31T15:14:10.277387+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:27:17.610346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.17","version":"0'0","reported_seq":14,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:11.390501+0000","last_change":"2024-03-31T15:14:08.275403+0000","last_active":"2024-03-31T15:14:11.390501+0000","last_peered":"2024-03-31T15:14:11.390501+0000","last_clean":"2024-03-31T15:14:11.390501+0000","last_became_active":"2024-03-31T15:14:08.275122+0000","last_became_peered":"2024-03-31T15:14:08.275122+0000","last_unstale":"2024-03-31T15:14:11.390501+0000","last_undegraded":"2024-03-31T15:14:11.390501+0000","last_fullsized":"2024-03-31T15:14:11.390501+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_clean_scrub_stamp":"2024-03-31T15:14:07.235046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:03:52.622502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.17","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:10.278750+0000","last_change":"2024-03-31T15:14:09.298453+0000","last_active":"2024-03-31T15:14:10.278750+0000","last_peered":"2024-03-31T15:14:10.278750+0000","last_clean":"2024-03-31T15:14:10.278750+0000","last_became_active":"2024-03-31T15:14:09.298160+0000","last_became_peered":"2024-03-31T15:14:09.298160+0000","last_unstale":"2024-03-31T15:14:10.278750+0000","last_undegraded":"2024-03-31T15:14:10.278750+0000","last_fullsized":"2024-03-31T15:14:10.278750+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:48:01.826468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":14,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:11.391501+0000","last_change":"2024-03-31T15:14:08.272714+0000","last_active":"2024-03-31T15:14:11.391501+0000","last_peered":"2024-03-31T15:14:11.391501+0000","last_clean":"2024-03-31T15:14:11.391501+0000","last_became_active":"2024-03-31T15:14:08.272416+0000","last_became_peered":"2024-03-31T15:14:08.272416+0000","last_unstale":"2024-03-31T15:14:11.391501+0000","last_undegraded":"2024-03-31T15:14:11.391501+0000","last_fullsized":"2024-03-31T15:14:11.391501+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_clean_scrub_stamp":"2024-03-31T15:14:07.235046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:34:33.623018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.14","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:11.391432+0000","last_change":"2024-03-31T15:14:09.281244+0000","last_active":"2024-03-31T15:14:11.391432+0000","last_peered":"2024-03-31T15:14:11.391432+0000","last_clean":"2024-03-31T15:14:11.391432+0000","last_became_active":"2024-03-31T15:14:09.280267+0000","last_became_peered":"2024-03-31T15:14:09.280267+0000","last_unstale":"2024-03-31T15:14:11.391432+0000","last_undegraded":"2024-03-31T15:14:11.391432+0000","last_fullsized":"2024-03-31T15:14:11.391432+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:23:54.993797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.15","version":"108'3","reported_seq":17,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:10.891863+0000","last_change":"2024-03-31T15:14:08.274322+0000","last_active":"2024-03-31T15:14:10.891863+0000","last_peered":"2024-03-31T15:14:10.891863+0000","last_clean":"2024-03-31T15:14:10.891863+0000","last_became_active":"2024-03-31T15:14:08.274025+0000","last_became_peered":"2024-03-31T15:14:08.274025+0000","last_unstale":"2024-03-31T15:14:10.891863+0000","last_undegraded":"2024-03-31T15:14:10.891863+0000","last_fullsized":"2024-03-31T15:14:10.891863+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_clean_scrub_stamp":"2024-03-31T15:14:07.235046+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:54:49.668391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.15","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:10.277296+0000","last_change":"2024-03-31T15:14:09.285492+0000","last_active":"2024-03-31T15:14:10.277296+0000","last_peered":"2024-03-31T15:14:10.277296+0000","last_clean":"2024-03-31T15:14:10.277296+0000","last_became_active":"2024-03-31T15:14:09.285203+0000","last_became_peered":"2024-03-31T15:14:09.285203+0000","last_unstale":"2024-03-31T15:14:10.277296+0000","last_undegraded":"2024-03-31T15:14:10.277296+0000","last_fullsized":"2024-03-31T15:14:10.277296+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:39:31.641218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.14","version":"108'6","reported_seq":20,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:10.892187+0000","last_change":"2024-03-31T15:14:08.277536+0000","last_active":"2024-03-31T15:14:10.892187+0000","last_peered":"2024-03-31T15:14:10.892187+0000","last_clean":"2024-03-31T15:14:10.892187+0000","last_became_active":"2024-03-31T15:14:08.277279+0000","last_became_peered":"2024-03-31T15:14:08.277279+0000","last_unstale":"2024-03-31T15:14:10.892187+0000","last_undegraded":"2024-03-31T15:14:10.892187+0000","last_fullsized":"2024-03-31T15:14:10.892187+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_clean_scrub_stamp":"2024-03-31T15:14:07.235046+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:08:03.916239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.12","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:11.390394+0000","last_change":"2024-03-31T15:14:09.275078+0000","last_active":"2024-03-31T15:14:11.390394+0000","last_peered":"2024-03-31T15:14:11.390394+0000","last_clean":"2024-03-31T15:14:11.390394+0000","last_became_active":"2024-03-31T15:14:09.274698+0000","last_became_peered":"2024-03-31T15:14:09.274698+0000","last_unstale":"2024-03-31T15:14:11.390394+0000","last_undegraded":"2024-03-31T15:14:11.390394+0000","last_fullsized":"2024-03-31T15:14:11.390394+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:30:56.093324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.13","version":"108'2","reported_seq":16,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:11.390958+0000","last_change":"2024-03-31T15:14:08.266097+0000","last_active":"2024-03-31T15:14:11.390958+0000","last_peered":"2024-03-31T15:14:11.390958+0000","last_clean":"2024-03-31T15:14:11.390958+0000","last_became_active":"2024-03-31T15:14:08.265812+0000","last_became_peered":"2024-03-31T15:14:08.265812+0000","last_unstale":"2024-03-31T15:14:11.390958+0000","last_undegraded":"2024-03-31T15:14:11.390958+0000","last_fullsized":"2024-03-31T15:14:11.390958+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_clean_scrub_stamp":"2024-03-31T15:14:07.235046+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:49:52.886409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.13","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:11.391329+0000","last_change":"2024-03-31T15:14:09.282087+0000","last_active":"2024-03-31T15:14:11.391329+0000","last_peered":"2024-03-31T15:14:11.391329+0000","last_clean":"2024-03-31T15:14:11.391329+0000","last_became_active":"2024-03-31T15:14:09.281693+0000","last_became_peered":"2024-03-31T15:14:09.281693+0000","last_unstale":"2024-03-31T15:14:11.391329+0000","last_undegraded":"2024-03-31T15:14:11.391329+0000","last_fullsized":"2024-03-31T15:14:11.391329+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:32:06.156300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":14,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:10.278350+0000","last_change":"2024-03-31T15:14:08.272919+0000","last_active":"2024-03-31T15:14:10.278350+0000","last_peered":"2024-03-31T15:14:10.278350+0000","last_clean":"2024-03-31T15:14:10.278350+0000","last_became_active":"2024-03-31T15:14:08.272309+0000","last_became_peered":"2024-03-31T15:14:08.272309+0000","last_unstale":"2024-03-31T15:14:10.278350+0000","last_undegraded":"2024-03-31T15:14:10.278350+0000","last_fullsized":"2024-03-31T15:14:10.278350+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_clean_scrub_stamp":"2024-03-31T15:14:07.235046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:56:10.303847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.11","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:11.390150+0000","last_change":"2024-03-31T15:14:09.274826+0000","last_active":"2024-03-31T15:14:11.390150+0000","last_peered":"2024-03-31T15:14:11.390150+0000","last_clean":"2024-03-31T15:14:11.390150+0000","last_became_active":"2024-03-31T15:14:09.274550+0000","last_became_peered":"2024-03-31T15:14:09.274550+0000","last_unstale":"2024-03-31T15:14:11.390150+0000","last_undegraded":"2024-03-31T15:14:11.390150+0000","last_fullsized":"2024-03-31T15:14:11.390150+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T17:25:28.357433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.10","version":"108'8","reported_seq":22,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:11.389660+0000","last_change":"2024-03-31T15:14:08.276042+0000","last_active":"2024-03-31T15:14:11.389660+0000","last_peered":"2024-03-31T15:14:11.389660+0000","last_clean":"2024-03-31T15:14:11.389660+0000","last_became_active":"2024-03-31T15:14:08.275766+0000","last_became_peered":"2024-03-31T15:14:08.275766+0000","last_unstale":"2024-03-31T15:14:11.389660+0000","last_undegraded":"2024-03-31T15:14:11.389660+0000","last_fullsized":"2024-03-31T15:14:11.389660+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_clean_scrub_stamp":"2024-03-31T15:14:07.235046+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:19:14.135624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.e","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:11.390586+0000","last_change":"2024-03-31T15:14:09.298576+0000","last_active":"2024-03-31T15:14:11.390586+0000","last_peered":"2024-03-31T15:14:11.390586+0000","last_clean":"2024-03-31T15:14:11.390586+0000","last_became_active":"2024-03-31T15:14:09.298059+0000","last_became_peered":"2024-03-31T15:14:09.298059+0000","last_unstale":"2024-03-31T15:14:11.390586+0000","last_undegraded":"2024-03-31T15:14:11.390586+0000","last_fullsized":"2024-03-31T15:14:11.390586+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T16:46:33.733000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.f","version":"108'1","reported_seq":15,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:10.892528+0000","last_change":"2024-03-31T15:14:08.273521+0000","last_active":"2024-03-31T15:14:10.892528+0000","last_peered":"2024-03-31T15:14:10.892528+0000","last_clean":"2024-03-31T15:14:10.892528+0000","last_became_active":"2024-03-31T15:14:08.271683+0000","last_became_peered":"2024-03-31T15:14:08.271683+0000","last_unstale":"2024-03-31T15:14:10.892528+0000","last_undegraded":"2024-03-31T15:14:10.892528+0000","last_fullsized":"2024-03-31T15:14:10.892528+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_clean_scrub_stamp":"2024-03-31T15:14:07.235046+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:08:12.284361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.f","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:11.391301+0000","last_change":"2024-03-31T15:14:09.292401+0000","last_active":"2024-03-31T15:14:11.391301+0000","last_peered":"2024-03-31T15:14:11.391301+0000","last_clean":"2024-03-31T15:14:11.391301+0000","last_became_active":"2024-03-31T15:14:09.292126+0000","last_became_peered":"2024-03-31T15:14:09.292126+0000","last_unstale":"2024-03-31T15:14:11.391301+0000","last_undegraded":"2024-03-31T15:14:11.391301+0000","last_fullsized":"2024-03-31T15:14:11.391301+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:24:54.943510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.e","version":"108'2","reported_seq":16,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:11.390584+0000","last_change":"2024-03-31T15:14:08.272765+0000","last_active":"2024-03-31T15:14:11.390584+0000","last_peered":"2024-03-31T15:14:11.390584+0000","last_clean":"2024-03-31T15:14:11.390584+0000","last_became_active":"2024-03-31T15:14:08.272496+0000","last_became_peered":"2024-03-31T15:14:08.272496+0000","last_unstale":"2024-03-31T15:14:11.390584+0000","last_undegraded":"2024-03-31T15:14:11.390584+0000","last_fullsized":"2024-03-31T15:14:11.390584+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_clean_scrub_stamp":"2024-03-31T15:14:07.235046+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:18:11.474845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.c","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:10.892335+0000","last_change":"2024-03-31T15:14:09.299017+0000","last_active":"2024-03-31T15:14:10.892335+0000","last_peered":"2024-03-31T15:14:10.892335+0000","last_clean":"2024-03-31T15:14:10.892335+0000","last_became_active":"2024-03-31T15:14:09.298654+0000","last_became_peered":"2024-03-31T15:14:09.298654+0000","last_unstale":"2024-03-31T15:14:10.892335+0000","last_undegraded":"2024-03-31T15:14:10.892335+0000","last_fullsized":"2024-03-31T15:14:10.892335+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:35:57.110653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.d","version":"108'1","reported_seq":15,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:10.278500+0000","last_change":"2024-03-31T15:14:08.278494+0000","last_active":"2024-03-31T15:14:10.278500+0000","last_peered":"2024-03-31T15:14:10.278500+0000","last_clean":"2024-03-31T15:14:10.278500+0000","last_became_active":"2024-03-31T15:14:08.278267+0000","last_became_peered":"2024-03-31T15:14:08.278267+0000","last_unstale":"2024-03-31T15:14:10.278500+0000","last_undegraded":"2024-03-31T15:14:10.278500+0000","last_fullsized":"2024-03-31T15:14:10.278500+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_clean_scrub_stamp":"2024-03-31T15:14:07.235046+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:49:25.058336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.d","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:10.892849+0000","last_change":"2024-03-31T15:14:09.288545+0000","last_active":"2024-03-31T15:14:10.892849+0000","last_peered":"2024-03-31T15:14:10.892849+0000","last_clean":"2024-03-31T15:14:10.892849+0000","last_became_active":"2024-03-31T15:14:09.288260+0000","last_became_peered":"2024-03-31T15:14:09.288260+0000","last_unstale":"2024-03-31T15:14:10.892849+0000","last_undegraded":"2024-03-31T15:14:10.892849+0000","last_fullsized":"2024-03-31T15:14:10.892849+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:16:47.900467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":14,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:10.277423+0000","last_change":"2024-03-31T15:14:08.271960+0000","last_active":"2024-03-31T15:14:10.277423+0000","last_peered":"2024-03-31T15:14:10.277423+0000","last_clean":"2024-03-31T15:14:10.277423+0000","last_became_active":"2024-03-31T15:14:08.271691+0000","last_became_peered":"2024-03-31T15:14:08.271691+0000","last_unstale":"2024-03-31T15:14:10.277423+0000","last_undegraded":"2024-03-31T15:14:10.277423+0000","last_fullsized":"2024-03-31T15:14:10.277423+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_clean_scrub_stamp":"2024-03-31T15:14:07.235046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T16:20:03.178688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.a","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:10.892273+0000","last_change":"2024-03-31T15:14:09.286866+0000","last_active":"2024-03-31T15:14:10.892273+0000","last_peered":"2024-03-31T15:14:10.892273+0000","last_clean":"2024-03-31T15:14:10.892273+0000","last_became_active":"2024-03-31T15:14:09.286282+0000","last_became_peered":"2024-03-31T15:14:09.286282+0000","last_unstale":"2024-03-31T15:14:10.892273+0000","last_undegraded":"2024-03-31T15:14:10.892273+0000","last_fullsized":"2024-03-31T15:14:10.892273+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T17:02:20.640638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.b","version":"108'2","reported_seq":16,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:11.391062+0000","last_change":"2024-03-31T15:14:08.274852+0000","last_active":"2024-03-31T15:14:11.391062+0000","last_peered":"2024-03-31T15:14:11.391062+0000","last_clean":"2024-03-31T15:14:11.391062+0000","last_became_active":"2024-03-31T15:14:08.274568+0000","last_became_peered":"2024-03-31T15:14:08.274568+0000","last_unstale":"2024-03-31T15:14:11.391062+0000","last_undegraded":"2024-03-31T15:14:11.391062+0000","last_fullsized":"2024-03-31T15:14:11.391062+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_clean_scrub_stamp":"2024-03-31T15:14:07.235046+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T19:34:36.202544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.b","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:10.891810+0000","last_change":"2024-03-31T15:14:09.280942+0000","last_active":"2024-03-31T15:14:10.891810+0000","last_peered":"2024-03-31T15:14:10.891810+0000","last_clean":"2024-03-31T15:14:10.891810+0000","last_became_active":"2024-03-31T15:14:09.280678+0000","last_became_peered":"2024-03-31T15:14:09.280678+0000","last_unstale":"2024-03-31T15:14:10.891810+0000","last_undegraded":"2024-03-31T15:14:10.891810+0000","last_fullsized":"2024-03-31T15:14:10.891810+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:00:22.586826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":14,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:11.389585+0000","last_change":"2024-03-31T15:14:08.275261+0000","last_active":"2024-03-31T15:14:11.389585+0000","last_peered":"2024-03-31T15:14:11.389585+0000","last_clean":"2024-03-31T15:14:11.389585+0000","last_became_active":"2024-03-31T15:14:08.274961+0000","last_became_peered":"2024-03-31T15:14:08.274961+0000","last_unstale":"2024-03-31T15:14:11.389585+0000","last_undegraded":"2024-03-31T15:14:11.389585+0000","last_fullsized":"2024-03-31T15:14:11.389585+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_clean_scrub_stamp":"2024-03-31T15:14:07.235046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:05:46.368323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.8","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:10.892352+0000","last_change":"2024-03-31T15:14:09.284118+0000","last_active":"2024-03-31T15:14:10.892352+0000","last_peered":"2024-03-31T15:14:10.892352+0000","last_clean":"2024-03-31T15:14:10.892352+0000","last_became_active":"2024-03-31T15:14:09.283439+0000","last_became_peered":"2024-03-31T15:14:09.283439+0000","last_unstale":"2024-03-31T15:14:10.892352+0000","last_undegraded":"2024-03-31T15:14:10.892352+0000","last_fullsized":"2024-03-31T15:14:10.892352+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T19:55:36.064563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":14,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:11.390727+0000","last_change":"2024-03-31T15:14:08.275022+0000","last_active":"2024-03-31T15:14:11.390727+0000","last_peered":"2024-03-31T15:14:11.390727+0000","last_clean":"2024-03-31T15:14:11.390727+0000","last_became_active":"2024-03-31T15:14:08.274740+0000","last_became_peered":"2024-03-31T15:14:08.274740+0000","last_unstale":"2024-03-31T15:14:11.390727+0000","last_undegraded":"2024-03-31T15:14:11.390727+0000","last_fullsized":"2024-03-31T15:14:11.390727+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_clean_scrub_stamp":"2024-03-31T15:14:07.235046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:25:35.260804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.9","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:11.390344+0000","last_change":"2024-03-31T15:14:09.291069+0000","last_active":"2024-03-31T15:14:11.390344+0000","last_peered":"2024-03-31T15:14:11.390344+0000","last_clean":"2024-03-31T15:14:11.390344+0000","last_became_active":"2024-03-31T15:14:09.290802+0000","last_became_peered":"2024-03-31T15:14:09.290802+0000","last_unstale":"2024-03-31T15:14:11.390344+0000","last_undegraded":"2024-03-31T15:14:11.390344+0000","last_fullsized":"2024-03-31T15:14:11.390344+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:36:20.417016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":14,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:10.891861+0000","last_change":"2024-03-31T15:14:08.270333+0000","last_active":"2024-03-31T15:14:10.891861+0000","last_peered":"2024-03-31T15:14:10.891861+0000","last_clean":"2024-03-31T15:14:10.891861+0000","last_became_active":"2024-03-31T15:14:08.269866+0000","last_became_peered":"2024-03-31T15:14:08.269866+0000","last_unstale":"2024-03-31T15:14:10.891861+0000","last_undegraded":"2024-03-31T15:14:10.891861+0000","last_fullsized":"2024-03-31T15:14:10.891861+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_clean_scrub_stamp":"2024-03-31T15:14:07.235046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:55:27.079135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.6","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:10.278592+0000","last_change":"2024-03-31T15:14:09.299295+0000","last_active":"2024-03-31T15:14:10.278592+0000","last_peered":"2024-03-31T15:14:10.278592+0000","last_clean":"2024-03-31T15:14:10.278592+0000","last_became_active":"2024-03-31T15:14:09.299016+0000","last_became_peered":"2024-03-31T15:14:09.299016+0000","last_unstale":"2024-03-31T15:14:10.278592+0000","last_undegraded":"2024-03-31T15:14:10.278592+0000","last_fullsized":"2024-03-31T15:14:10.278592+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:34:34.630067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.7","version":"108'2","reported_seq":16,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:11.391777+0000","last_change":"2024-03-31T15:14:08.276158+0000","last_active":"2024-03-31T15:14:11.391777+0000","last_peered":"2024-03-31T15:14:11.391777+0000","last_clean":"2024-03-31T15:14:11.391777+0000","last_became_active":"2024-03-31T15:14:08.275856+0000","last_became_peered":"2024-03-31T15:14:08.275856+0000","last_unstale":"2024-03-31T15:14:11.391777+0000","last_undegraded":"2024-03-31T15:14:11.391777+0000","last_fullsized":"2024-03-31T15:14:11.391777+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_clean_scrub_stamp":"2024-03-31T15:14:07.235046+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:48:00.819420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.7","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:10.892615+0000","last_change":"2024-03-31T15:14:09.277522+0000","last_active":"2024-03-31T15:14:10.892615+0000","last_peered":"2024-03-31T15:14:10.892615+0000","last_clean":"2024-03-31T15:14:10.892615+0000","last_became_active":"2024-03-31T15:14:09.276889+0000","last_became_peered":"2024-03-31T15:14:09.276889+0000","last_unstale":"2024-03-31T15:14:10.892615+0000","last_undegraded":"2024-03-31T15:14:10.892615+0000","last_fullsized":"2024-03-31T15:14:10.892615+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:45:51.393113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.6","version":"109'1","reported_seq":15,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:10.278704+0000","last_change":"2024-03-31T15:14:08.265332+0000","last_active":"2024-03-31T15:14:10.278704+0000","last_peered":"2024-03-31T15:14:10.278704+0000","last_clean":"2024-03-31T15:14:10.278704+0000","last_became_active":"2024-03-31T15:14:08.264838+0000","last_became_peered":"2024-03-31T15:14:08.264838+0000","last_unstale":"2024-03-31T15:14:10.278704+0000","last_undegraded":"2024-03-31T15:14:10.278704+0000","last_fullsized":"2024-03-31T15:14:10.278704+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_clean_scrub_stamp":"2024-03-31T15:14:07.235046+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:32:53.111430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.4","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:10.892152+0000","last_change":"2024-03-31T15:14:09.279412+0000","last_active":"2024-03-31T15:14:10.892152+0000","last_peered":"2024-03-31T15:14:10.892152+0000","last_clean":"2024-03-31T15:14:10.892152+0000","last_became_active":"2024-03-31T15:14:09.278696+0000","last_became_peered":"2024-03-31T15:14:09.278696+0000","last_unstale":"2024-03-31T15:14:10.892152+0000","last_undegraded":"2024-03-31T15:14:10.892152+0000","last_fullsized":"2024-03-31T15:14:10.892152+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:57:34.106053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":14,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:10.891892+0000","last_change":"2024-03-31T15:14:08.270001+0000","last_active":"2024-03-31T15:14:10.891892+0000","last_peered":"2024-03-31T15:14:10.891892+0000","last_clean":"2024-03-31T15:14:10.891892+0000","last_became_active":"2024-03-31T15:14:08.268897+0000","last_became_peered":"2024-03-31T15:14:08.268897+0000","last_unstale":"2024-03-31T15:14:10.891892+0000","last_undegraded":"2024-03-31T15:14:10.891892+0000","last_fullsized":"2024-03-31T15:14:10.891892+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_clean_scrub_stamp":"2024-03-31T15:14:07.235046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T19:21:04.678038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.5","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:10.278439+0000","last_change":"2024-03-31T15:14:09.292175+0000","last_active":"2024-03-31T15:14:10.278439+0000","last_peered":"2024-03-31T15:14:10.278439+0000","last_clean":"2024-03-31T15:14:10.278439+0000","last_became_active":"2024-03-31T15:14:09.291875+0000","last_became_peered":"2024-03-31T15:14:09.291875+0000","last_unstale":"2024-03-31T15:14:10.278439+0000","last_undegraded":"2024-03-31T15:14:10.278439+0000","last_fullsized":"2024-03-31T15:14:10.278439+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:27:02.356373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":14,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:11.391479+0000","last_change":"2024-03-31T15:14:08.268133+0000","last_active":"2024-03-31T15:14:11.391479+0000","last_peered":"2024-03-31T15:14:11.391479+0000","last_clean":"2024-03-31T15:14:11.391479+0000","last_became_active":"2024-03-31T15:14:08.267768+0000","last_became_peered":"2024-03-31T15:14:08.267768+0000","last_unstale":"2024-03-31T15:14:11.391479+0000","last_undegraded":"2024-03-31T15:14:11.391479+0000","last_fullsized":"2024-03-31T15:14:11.391479+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_clean_scrub_stamp":"2024-03-31T15:14:07.235046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:55:27.079135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.2","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:10.278552+0000","last_change":"2024-03-31T15:14:09.276195+0000","last_active":"2024-03-31T15:14:10.278552+0000","last_peered":"2024-03-31T15:14:10.278552+0000","last_clean":"2024-03-31T15:14:10.278552+0000","last_became_active":"2024-03-31T15:14:09.275925+0000","last_became_peered":"2024-03-31T15:14:09.275925+0000","last_unstale":"2024-03-31T15:14:10.278552+0000","last_undegraded":"2024-03-31T15:14:10.278552+0000","last_fullsized":"2024-03-31T15:14:10.278552+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T19:39:13.736880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":14,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:10.277017+0000","last_change":"2024-03-31T15:14:08.268809+0000","last_active":"2024-03-31T15:14:10.277017+0000","last_peered":"2024-03-31T15:14:10.277017+0000","last_clean":"2024-03-31T15:14:10.277017+0000","last_became_active":"2024-03-31T15:14:08.268473+0000","last_became_peered":"2024-03-31T15:14:08.268473+0000","last_unstale":"2024-03-31T15:14:10.277017+0000","last_undegraded":"2024-03-31T15:14:10.277017+0000","last_fullsized":"2024-03-31T15:14:10.277017+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_clean_scrub_stamp":"2024-03-31T15:14:07.235046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T16:45:55.109941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.3","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:10.279269+0000","last_change":"2024-03-31T15:14:09.273676+0000","last_active":"2024-03-31T15:14:10.279269+0000","last_peered":"2024-03-31T15:14:10.279269+0000","last_clean":"2024-03-31T15:14:10.279269+0000","last_became_active":"2024-03-31T15:14:09.273367+0000","last_became_peered":"2024-03-31T15:14:09.273367+0000","last_unstale":"2024-03-31T15:14:10.279269+0000","last_undegraded":"2024-03-31T15:14:10.279269+0000","last_fullsized":"2024-03-31T15:14:10.279269+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:10:29.340449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":14,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:10.278567+0000","last_change":"2024-03-31T15:14:08.269651+0000","last_active":"2024-03-31T15:14:10.278567+0000","last_peered":"2024-03-31T15:14:10.278567+0000","last_clean":"2024-03-31T15:14:10.278567+0000","last_became_active":"2024-03-31T15:14:08.269372+0000","last_became_peered":"2024-03-31T15:14:08.269372+0000","last_unstale":"2024-03-31T15:14:10.278567+0000","last_undegraded":"2024-03-31T15:14:10.278567+0000","last_fullsized":"2024-03-31T15:14:10.278567+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_clean_scrub_stamp":"2024-03-31T15:14:07.235046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:22:09.702718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.0","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:10.892507+0000","last_change":"2024-03-31T15:14:09.284384+0000","last_active":"2024-03-31T15:14:10.892507+0000","last_peered":"2024-03-31T15:14:10.892507+0000","last_clean":"2024-03-31T15:14:10.892507+0000","last_became_active":"2024-03-31T15:14:09.283766+0000","last_became_peered":"2024-03-31T15:14:09.283766+0000","last_unstale":"2024-03-31T15:14:10.892507+0000","last_undegraded":"2024-03-31T15:14:10.892507+0000","last_fullsized":"2024-03-31T15:14:10.892507+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:03:38.532236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1","version":"0'0","reported_seq":14,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:11.389640+0000","last_change":"2024-03-31T15:14:08.271032+0000","last_active":"2024-03-31T15:14:11.389640+0000","last_peered":"2024-03-31T15:14:11.389640+0000","last_clean":"2024-03-31T15:14:11.389640+0000","last_became_active":"2024-03-31T15:14:08.270781+0000","last_became_peered":"2024-03-31T15:14:08.270781+0000","last_unstale":"2024-03-31T15:14:11.389640+0000","last_undegraded":"2024-03-31T15:14:11.389640+0000","last_fullsized":"2024-03-31T15:14:11.389640+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_clean_scrub_stamp":"2024-03-31T15:14:07.235046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:59:15.102040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:10.892154+0000","last_change":"2024-03-31T15:14:09.288490+0000","last_active":"2024-03-31T15:14:10.892154+0000","last_peered":"2024-03-31T15:14:10.892154+0000","last_clean":"2024-03-31T15:14:10.892154+0000","last_became_active":"2024-03-31T15:14:09.284507+0000","last_became_peered":"2024-03-31T15:14:09.284507+0000","last_unstale":"2024-03-31T15:14:10.892154+0000","last_undegraded":"2024-03-31T15:14:10.892154+0000","last_fullsized":"2024-03-31T15:14:10.892154+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:09:03.491873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":14,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:10.891910+0000","last_change":"2024-03-31T15:14:08.270270+0000","last_active":"2024-03-31T15:14:10.891910+0000","last_peered":"2024-03-31T15:14:10.891910+0000","last_clean":"2024-03-31T15:14:10.891910+0000","last_became_active":"2024-03-31T15:14:08.269482+0000","last_became_peered":"2024-03-31T15:14:08.269482+0000","last_unstale":"2024-03-31T15:14:10.891910+0000","last_undegraded":"2024-03-31T15:14:10.891910+0000","last_fullsized":"2024-03-31T15:14:10.891910+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_clean_scrub_stamp":"2024-03-31T15:14:07.235046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:48:00.819420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.10","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:11.390177+0000","last_change":"2024-03-31T15:14:09.277917+0000","last_active":"2024-03-31T15:14:11.390177+0000","last_peered":"2024-03-31T15:14:11.390177+0000","last_clean":"2024-03-31T15:14:11.390177+0000","last_became_active":"2024-03-31T15:14:09.277651+0000","last_became_peered":"2024-03-31T15:14:09.277651+0000","last_unstale":"2024-03-31T15:14:11.390177+0000","last_undegraded":"2024-03-31T15:14:11.390177+0000","last_fullsized":"2024-03-31T15:14:11.390177+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:08.242094+0000","last_clean_scrub_stamp":"2024-03-31T15:14:08.242094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T16:10:45.541190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":14,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:10.276816+0000","last_change":"2024-03-31T15:14:08.271941+0000","last_active":"2024-03-31T15:14:10.276816+0000","last_peered":"2024-03-31T15:14:10.276816+0000","last_clean":"2024-03-31T15:14:10.276816+0000","last_became_active":"2024-03-31T15:14:08.271589+0000","last_became_peered":"2024-03-31T15:14:08.271589+0000","last_unstale":"2024-03-31T15:14:10.276816+0000","last_undegraded":"2024-03-31T15:14:10.276816+0000","last_fullsized":"2024-03-31T15:14:10.276816+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:14:07.235046+0000","last_clean_scrub_stamp":"2024-03-31T15:14:07.235046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T19:47:47.083449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":335,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-31T15:14:10.892630+0000","last_change":"2024-03-31T15:03:24.329890+0000","last_active":"2024-03-31T15:14:10.892630+0000","last_peered":"2024-03-31T15:14:10.892630+0000","last_clean":"2024-03-31T15:14:10.892630+0000","last_became_active":"2024-03-31T15:03:24.328882+0000","last_became_peered":"2024-03-31T15:03:24.328882+0000","last_unstale":"2024-03-31T15:14:10.892630+0000","last_undegraded":"2024-03-31T15:14:10.892630+0000","last_fullsized":"2024-03-31T15:14:10.892630+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:22.885535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:22.885535+0000","last_clean_scrub_stamp":"2024-03-31T15:03:22.885535+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:18:29.004296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":17,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":16,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509575,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33084,"kb_used_data":1904,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338756,"statfs":{"total":96636764160,"available":96602886144,"internally_reserved":0,"allocated":1949696,"data_stored":1175057,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509574,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32508,"kb_used_data":1324,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339332,"statfs":{"total":96636764160,"available":96603475968,"internally_reserved":0,"allocated":1355776,"data_stored":584677,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":15,"seq":64424509573,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32500,"kb_used_data":1316,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339340,"statfs":{"total":96636764160,"available":96603484160,"internally_reserved":0,"allocated":1347584,"data_stored":584073,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509572,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36656,"kb_used_data":1316,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335184,"statfs":{"total":96636764160,"available":96599228416,"internally_reserved":0,"allocated":1347584,"data_stored":584073,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542279,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32508,"kb_used_data":1320,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339332,"statfs":{"total":96636764160,"available":96603475968,"internally_reserved":0,"allocated":1351680,"data_stored":584143,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542279,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32512,"kb_used_data":1332,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339328,"statfs":{"total":96636764160,"available":96603471872,"internally_reserved":0,"allocated":1363968,"data_stored":585755,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542279,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32504,"kb_used_data":1324,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339336,"statfs":{"total":96636764160,"available":96603480064,"internally_reserved":0,"allocated":1355776,"data_stored":584197,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542279,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33100,"kb_used_data":1916,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338740,"statfs":{"total":96636764160,"available":96602869760,"internally_reserved":0,"allocated":1961984,"data_stored":1176165,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-31T15:14:18.372 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-31T15:14:18.372 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-03-31T15:14:18.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:18.570+0000 7f9b70c00640 1 Processor -- start 2024-03-31T15:14:18.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:18.570+0000 7f9b70c00640 1 -- start start 2024-03-31T15:14:18.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:18.574+0000 7f9b70c00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f9b6c0cd640 0x7f9b6c166b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:18.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:18.574+0000 7f9b70c00640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9b6c0cf940 con 0x7f9b6c06cac0 2024-03-31T15:14:18.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:18.574+0000 7f9b70c00640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9b6c0cfc20 con 0x7f9b6c06e460 2024-03-31T15:14:18.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:18.574+0000 7f9b70c00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f9b6c06ff80 con 0x7f9b6c0cd640 2024-03-31T15:14:18.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:18.574+0000 7f9b63e00640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f9b6c06cac0 0x7f9b6c0ccf30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:53430/0 (socket says 172.21.15.16:53430) 2024-03-31T15:14:18.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:18.574+0000 7f9b63e00640 1 -- 172.21.15.16:0/215740037 learned_addr learned my addr 172.21.15.16:0/215740037 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:14:18.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:18.574+0000 7f9b6be00640 1 --2- 172.21.15.16:0/215740037 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f9b6c0cd640 0x7f9b6c166b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:18.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:18.574+0000 7f9b6aa00640 1 -- 172.21.15.16:0/215740037 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 812781866 0 0) 0x7f9b6c0cf940 con 0x7f9b6c06cac0 2024-03-31T15:14:18.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:18.574+0000 7f9b6aa00640 1 -- 172.21.15.16:0/215740037 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9b50003530 con 0x7f9b6c06cac0 2024-03-31T15:14:18.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:18.574+0000 7f9b6aa00640 1 -- 172.21.15.16:0/215740037 <== mon.1 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2711108677 0 0) 0x7f9b6c0cfc20 con 0x7f9b6c06e460 2024-03-31T15:14:18.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:18.574+0000 7f9b6aa00640 1 -- 172.21.15.16:0/215740037 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9b6c0cf940 con 0x7f9b6c06e460 2024-03-31T15:14:18.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:18.574+0000 7f9b6aa00640 1 -- 172.21.15.16:0/215740037 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 70817234 0 0) 0x7f9b50003530 con 0x7f9b6c06cac0 2024-03-31T15:14:18.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:18.574+0000 7f9b6aa00640 1 -- 172.21.15.16:0/215740037 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f9b6c0cd640 msgr2=0x7f9b6c166b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:14:18.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:18.574+0000 7f9b6aa00640 1 --2- 172.21.15.16:0/215740037 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f9b6c0cd640 0x7f9b6c166b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:18.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:18.574+0000 7f9b6aa00640 1 -- 172.21.15.16:0/215740037 >> v1:172.21.15.89:6789/0 conn(0x7f9b6c06e460 legacy=0x7f9b6c06e840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:18.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:18.574+0000 7f9b6aa00640 1 -- 172.21.15.16:0/215740037 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b6c0cff00 con 0x7f9b6c06cac0 2024-03-31T15:14:18.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:18.574+0000 7f9b6aa00640 1 -- 172.21.15.16:0/215740037 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 214807623 0 0) 0x7f9b54001680 con 0x7f9b6c06cac0 2024-03-31T15:14:18.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:18.574+0000 7f9b6aa00640 1 -- 172.21.15.16:0/215740037 >> v1:172.21.15.16:6789/0 conn(0x7f9b6c06cac0 legacy=0x7f9b6c0ccf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:18.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:18.574+0000 7f9b6aa00640 1 --2- 172.21.15.16:0/215740037 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f9b6c0cd640 0x7f9b50003b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:18.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:18.574+0000 7f9b6aa00640 1 --2- 172.21.15.16:0/215740037 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9b500040e0 0x7f9b50004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:18.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:18.574+0000 7f9b6aa00640 1 --2- 172.21.15.16:0/215740037 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9b50004d50 0x7f9b500051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:18.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:18.574+0000 7f9b6aa00640 1 -- 172.21.15.16:0/215740037 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f9b6c06ff80 con 0x7f9b500040e0 2024-03-31T15:14:18.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:18.574+0000 7f9b6aa00640 1 -- 172.21.15.16:0/215740037 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f9b5000bb60 con 0x7f9b50004d50 2024-03-31T15:14:18.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:18.574+0000 7f9b6aa00640 1 -- 172.21.15.16:0/215740037 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f9b5000bd30 con 0x7f9b6c0cd640 2024-03-31T15:14:18.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:18.574+0000 7f9b63e00640 1 --2- 172.21.15.16:0/215740037 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9b500040e0 0x7f9b50004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:18.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:18.574+0000 7f9b6b400640 1 --2- 172.21.15.16:0/215740037 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f9b6c0cd640 0x7f9b50003b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:18.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:18.574+0000 7f9b6be00640 1 --2- 172.21.15.16:0/215740037 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9b50004d50 0x7f9b500051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:18.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:18.574+0000 7f9b63e00640 1 -- 172.21.15.16:0/215740037 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f9b6c0cd640 msgr2=0x7f9b50003b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:14:18.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:18.574+0000 7f9b63e00640 1 --2- 172.21.15.16:0/215740037 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f9b6c0cd640 0x7f9b50003b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:18.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:18.574+0000 7f9b63e00640 1 -- 172.21.15.16:0/215740037 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9b50004d50 msgr2=0x7f9b500051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:14:18.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:18.574+0000 7f9b63e00640 1 --2- 172.21.15.16:0/215740037 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9b50004d50 0x7f9b500051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:18.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:18.574+0000 7f9b63e00640 1 -- 172.21.15.16:0/215740037 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b50003530 con 0x7f9b500040e0 2024-03-31T15:14:18.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:18.574+0000 7f9b6be00640 1 --2- 172.21.15.16:0/215740037 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9b50004d50 0x7f9b500051d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:14:18.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:18.578+0000 7f9b63e00640 1 --2- 172.21.15.16:0/215740037 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9b500040e0 0x7f9b50004810 secure :-1 s=READY pgs=1665 cs=0 l=1 rev1=1 crypto rx=0x7f9b5400c190 tx=0x7f9b5401cbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4756067d243126e7 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:18.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:18.578+0000 7f9b6aa00640 1 -- 172.21.15.16:0/215740037 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b54007cd0 con 0x7f9b500040e0 2024-03-31T15:14:18.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:18.578+0000 7f9b6aa00640 1 -- 172.21.15.16:0/215740037 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9b540a9020 con 0x7f9b500040e0 2024-03-31T15:14:18.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:18.578+0000 7f9b6aa00640 1 -- 172.21.15.16:0/215740037 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b54005b20 con 0x7f9b500040e0 2024-03-31T15:14:18.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:18.578+0000 7f9b70c00640 1 -- 172.21.15.16:0/215740037 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9b500040e0 msgr2=0x7f9b50004810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:18.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:18.578+0000 7f9b70c00640 1 --2- 172.21.15.16:0/215740037 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9b500040e0 0x7f9b50004810 secure :-1 s=READY pgs=1665 cs=0 l=1 rev1=1 crypto rx=0x7f9b5400c190 tx=0x7f9b5401cbe0 comp rx=0 tx=0).stop 2024-03-31T15:14:18.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:18.578+0000 7f9b6be00640 1 -- 172.21.15.16:0/215740037 reap_dead start 2024-03-31T15:14:18.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:18.578+0000 7f9b70c00640 1 -- 172.21.15.16:0/215740037 shutdown_connections 2024-03-31T15:14:18.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:18.578+0000 7f9b70c00640 1 -- 172.21.15.16:0/215740037 >> 172.21.15.16:0/215740037 conn(0x7f9b6c0d0580 msgr2=0x7f9b6c0d0990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:14:18.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:18.578+0000 7f9b70c00640 1 -- 172.21.15.16:0/215740037 shutdown_connections 2024-03-31T15:14:18.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:18.578+0000 7f9b70c00640 1 -- 172.21.15.16:0/215740037 wait complete. 2024-03-31T15:14:18.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:18.578+0000 7f9b70c00640 1 Processor -- start 2024-03-31T15:14:18.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:18.578+0000 7f9b70c00640 1 -- start start 2024-03-31T15:14:18.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:18.582+0000 7f9b70c00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9b500040e0 0x7f9b6c158100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:18.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:18.582+0000 7f9b70c00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9b6c14ac20 0x7f9b6c14b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:18.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:18.582+0000 7f9b70c00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f9b6c15e480 0x7f9b6c15e900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:18.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:18.582+0000 7f9b70c00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f9b6c167700 con 0x7f9b500040e0 2024-03-31T15:14:18.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:18.582+0000 7f9b70c00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f9b6c167580 con 0x7f9b6c14ac20 2024-03-31T15:14:18.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:18.582+0000 7f9b70c00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f9b6c167880 con 0x7f9b6c15e480 2024-03-31T15:14:18.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:18.582+0000 7f9b6b400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9b500040e0 0x7f9b6c158100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:18.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:18.582+0000 7f9b6b400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9b500040e0 0x7f9b6c158100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42784/0 (socket says 172.21.15.16:42784) 2024-03-31T15:14:18.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:18.582+0000 7f9b6b400640 1 -- 172.21.15.16:0/3509249121 learned_addr learned my addr 172.21.15.16:0/3509249121 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:14:18.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:18.582+0000 7f9b63e00640 1 --2- 172.21.15.16:0/3509249121 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9b6c14ac20 0x7f9b6c14b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:18.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:18.582+0000 7f9b6be00640 1 --2- 172.21.15.16:0/3509249121 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f9b6c15e480 0x7f9b6c15e900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:18.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:18.582+0000 7f9b6b400640 1 -- 172.21.15.16:0/3509249121 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f9b6c15e480 msgr2=0x7f9b6c15e900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:18.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:18.582+0000 7f9b6b400640 1 --2- 172.21.15.16:0/3509249121 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f9b6c15e480 0x7f9b6c15e900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:18.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:18.582+0000 7f9b6b400640 1 -- 172.21.15.16:0/3509249121 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9b6c14ac20 msgr2=0x7f9b6c14b0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:18.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:18.582+0000 7f9b6b400640 1 --2- 172.21.15.16:0/3509249121 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9b6c14ac20 0x7f9b6c14b0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:18.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:18.582+0000 7f9b6b400640 1 -- 172.21.15.16:0/3509249121 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b6c0c6560 con 0x7f9b500040e0 2024-03-31T15:14:18.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:18.582+0000 7f9b6be00640 1 --2- 172.21.15.16:0/3509249121 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f9b6c15e480 0x7f9b6c15e900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:14:18.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:18.582+0000 7f9b6b400640 1 --2- 172.21.15.16:0/3509249121 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9b500040e0 0x7f9b6c158100 secure :-1 s=READY pgs=1666 cs=0 l=1 rev1=1 crypto rx=0x7f9b5c00f760 tx=0x7f9b5c00fc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:18.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:18.582+0000 7f9b63400640 1 -- 172.21.15.16:0/3509249121 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b5c00fe70 con 0x7f9b500040e0 2024-03-31T15:14:18.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:18.582+0000 7f9b63400640 1 -- 172.21.15.16:0/3509249121 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9b5c018070 con 0x7f9b500040e0 2024-03-31T15:14:18.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:18.582+0000 7f9b63400640 1 -- 172.21.15.16:0/3509249121 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b5c00d730 con 0x7f9b500040e0 2024-03-31T15:14:18.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:18.582+0000 7f9b70c00640 1 -- 172.21.15.16:0/3509249121 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9b6c0c6840 con 0x7f9b500040e0 2024-03-31T15:14:18.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:18.582+0000 7f9b70c00640 1 -- 172.21.15.16:0/3509249121 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9b6c0c6e00 con 0x7f9b500040e0 2024-03-31T15:14:18.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:18.582+0000 7f9b70c00640 1 -- 172.21.15.16:0/3509249121 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9b6c04f8c0 con 0x7f9b500040e0 2024-03-31T15:14:18.577 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:18.582+0000 7f9b63400640 1 -- 172.21.15.16:0/3509249121 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f9b5c00d8d0 con 0x7f9b500040e0 2024-03-31T15:14:18.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:18.586+0000 7f9b63400640 1 --2- 172.21.15.16:0/3509249121 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f9b480683b0 0x7f9b4806a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:18.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:18.586+0000 7f9b63400640 1 -- 172.21.15.16:0/3509249121 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(110..110 src has 1..110) ==== 5767+0+0 (secure 0 0 0) 0x7f9b5c086940 con 0x7f9b500040e0 2024-03-31T15:14:18.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:18.586+0000 7f9b63e00640 1 --2- 172.21.15.16:0/3509249121 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f9b480683b0 0x7f9b4806a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:18.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:18.586+0000 7f9b63e00640 1 --2- 172.21.15.16:0/3509249121 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f9b480683b0 0x7f9b4806a870 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f9b540aa000 tx=0x7f9b54001a40 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:18.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:18.590+0000 7f9b63400640 1 -- 172.21.15.16:0/3509249121 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f9b5c056e20 con 0x7f9b500040e0 2024-03-31T15:14:18.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:18.810+0000 7f9b70c00640 1 -- 172.21.15.16:0/3509249121 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) -- 0x7f9b6c0cf4f0 con 0x7f9b500040e0 2024-03-31T15:14:18.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:18.814+0000 7f9b63400640 1 -- 172.21.15.16:0/3509249121 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v20) ==== 190+0+147 (secure 0 0 0) 0x7f9b5c0860e0 con 0x7f9b500040e0 2024-03-31T15:14:18.806 INFO:teuthology.orchestra.run.smithi016.stdout:[client.0] 2024-03-31T15:14:18.806 INFO:teuthology.orchestra.run.smithi016.stdout: key = AQAPewlmlMIDEhAAF0Ll1HcrWHmH6xDKq2Qq9A== 2024-03-31T15:14:18.807 INFO:teuthology.orchestra.run.smithi016.stdout: caps mds = "allow" 2024-03-31T15:14:18.807 INFO:teuthology.orchestra.run.smithi016.stdout: caps mon = "allow r" 2024-03-31T15:14:18.807 INFO:teuthology.orchestra.run.smithi016.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-03-31T15:14:18.807 INFO:teuthology.orchestra.run.smithi016.stderr:updated caps for client.0 2024-03-31T15:14:18.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:18.814+0000 7f9b70c00640 1 -- 172.21.15.16:0/3509249121 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f9b480683b0 msgr2=0x7f9b4806a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:18.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:18.814+0000 7f9b70c00640 1 --2- 172.21.15.16:0/3509249121 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f9b480683b0 0x7f9b4806a870 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f9b540aa000 tx=0x7f9b54001a40 comp rx=0 tx=0).stop 2024-03-31T15:14:18.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:18.814+0000 7f9b70c00640 1 -- 172.21.15.16:0/3509249121 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9b500040e0 msgr2=0x7f9b6c158100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:18.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:18.814+0000 7f9b70c00640 1 --2- 172.21.15.16:0/3509249121 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9b500040e0 0x7f9b6c158100 secure :-1 s=READY pgs=1666 cs=0 l=1 rev1=1 crypto rx=0x7f9b5c00f760 tx=0x7f9b5c00fc20 comp rx=0 tx=0).stop 2024-03-31T15:14:18.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:18.814+0000 7f9b70c00640 1 -- 172.21.15.16:0/3509249121 shutdown_connections 2024-03-31T15:14:18.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:18.814+0000 7f9b70c00640 1 --2- 172.21.15.16:0/3509249121 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f9b480683b0 0x7f9b4806a870 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:18.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:18.814+0000 7f9b70c00640 1 --2- 172.21.15.16:0/3509249121 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f9b6c15e480 0x7f9b6c15e900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:18.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:18.814+0000 7f9b70c00640 1 --2- 172.21.15.16:0/3509249121 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9b6c14ac20 0x7f9b6c14b0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:18.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:18.814+0000 7f9b70c00640 1 --2- 172.21.15.16:0/3509249121 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9b500040e0 0x7f9b6c158100 unknown :-1 s=CLOSED pgs=1666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:18.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:18.814+0000 7f9b70c00640 1 -- 172.21.15.16:0/3509249121 >> 172.21.15.16:0/3509249121 conn(0x7f9b6c0d0580 msgr2=0x7f9b6c15acf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:14:18.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:18.814+0000 7f9b70c00640 1 -- 172.21.15.16:0/3509249121 shutdown_connections 2024-03-31T15:14:18.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:18.814+0000 7f9b70c00640 1 -- 172.21.15.16:0/3509249121 wait complete. 2024-03-31T15:14:18.829 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-31T15:14:18.969 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:18.978+0000 7f5ba8e00640 1 Processor -- start 2024-03-31T15:14:18.969 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:18.978+0000 7f5ba8e00640 1 -- start start 2024-03-31T15:14:18.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:18.978+0000 7f5ba8e00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f5ba4145aa0 0x7f5ba415c8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:18.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:18.978+0000 7f5ba8e00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f5ba406c1c0 0x7f5ba4072e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:18.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:18.978+0000 7f5ba8e00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f5ba406b620 0x7f5ba406bab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:18.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:18.978+0000 7f5ba8e00640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f5ba406fd50 con 0x7f5ba4145aa0 2024-03-31T15:14:18.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:18.978+0000 7f5ba8e00640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f5ba40d09f0 con 0x7f5ba406c1c0 2024-03-31T15:14:18.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:18.978+0000 7f5ba8e00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f5ba40d0cd0 con 0x7f5ba406b620 2024-03-31T15:14:18.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:18.978+0000 7f5ba1e00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f5ba4145aa0 0x7f5ba415c8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:18.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:18.978+0000 7f5ba1e00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f5ba4145aa0 0x7f5ba415c8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42794/0 (socket says 172.21.15.16:42794) 2024-03-31T15:14:18.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:18.978+0000 7f5ba1e00640 1 -- 172.21.15.16:0/977932785 learned_addr learned my addr 172.21.15.16:0/977932785 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:14:18.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:18.978+0000 7f5ba2800640 1 --2- 172.21.15.16:0/977932785 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f5ba406b620 0x7f5ba406bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:18.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:18.978+0000 7f5ba1400640 1 --2- 172.21.15.16:0/977932785 >> v2:172.21.15.89:3300/0 conn(0x7f5ba406c1c0 0x7f5ba4072e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:18.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:18.978+0000 7f5ba1e00640 1 -- 172.21.15.16:0/977932785 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f5ba406b620 msgr2=0x7f5ba406bab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:14:18.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:18.978+0000 7f5ba1e00640 1 --2- 172.21.15.16:0/977932785 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f5ba406b620 0x7f5ba406bab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:18.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:18.978+0000 7f5ba1e00640 1 -- 172.21.15.16:0/977932785 >> v2:172.21.15.89:3300/0 conn(0x7f5ba406c1c0 msgr2=0x7f5ba4072e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:14:18.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:18.978+0000 7f5ba1e00640 1 --2- 172.21.15.16:0/977932785 >> v2:172.21.15.89:3300/0 conn(0x7f5ba406c1c0 0x7f5ba4072e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:18.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:18.978+0000 7f5ba1e00640 1 -- 172.21.15.16:0/977932785 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ba40d06e0 con 0x7f5ba4145aa0 2024-03-31T15:14:18.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:18.982+0000 7f5ba1e00640 1 --2- 172.21.15.16:0/977932785 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5ba4145aa0 0x7f5ba415c8e0 secure :-1 s=READY pgs=1667 cs=0 l=1 rev1=1 crypto rx=0x7f5ba40b7310 tx=0x7f5b9401c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0b92b7536c72777 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:18.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:18.982+0000 7f5ba0a00640 1 -- 172.21.15.16:0/977932785 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b940b0070 con 0x7f5ba4145aa0 2024-03-31T15:14:18.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:18.982+0000 7f5ba0a00640 1 -- 172.21.15.16:0/977932785 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5b940ab600 con 0x7f5ba4145aa0 2024-03-31T15:14:18.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:18.982+0000 7f5ba0a00640 1 -- 172.21.15.16:0/977932785 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b940ab9d0 con 0x7f5ba4145aa0 2024-03-31T15:14:18.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:18.982+0000 7f5ba8e00640 1 -- 172.21.15.16:0/977932785 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5ba4145aa0 msgr2=0x7f5ba415c8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:18.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:18.982+0000 7f5ba8e00640 1 --2- 172.21.15.16:0/977932785 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5ba4145aa0 0x7f5ba415c8e0 secure :-1 s=READY pgs=1667 cs=0 l=1 rev1=1 crypto rx=0x7f5ba40b7310 tx=0x7f5b9401c520 comp rx=0 tx=0).stop 2024-03-31T15:14:18.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:18.982+0000 7f5ba8e00640 1 -- 172.21.15.16:0/977932785 shutdown_connections 2024-03-31T15:14:18.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:18.982+0000 7f5ba8e00640 1 --2- 172.21.15.16:0/977932785 >> v2:172.21.15.89:3300/0 conn(0x7f5ba406c1c0 0x7f5ba4072e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:18.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:18.982+0000 7f5ba8e00640 1 --2- 172.21.15.16:0/977932785 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f5ba406b620 0x7f5ba406bab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:18.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:18.982+0000 7f5ba8e00640 1 --2- 172.21.15.16:0/977932785 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5ba4145aa0 0x7f5ba415c8e0 unknown :-1 s=CLOSED pgs=1667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:18.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:18.982+0000 7f5ba8e00640 1 -- 172.21.15.16:0/977932785 >> 172.21.15.16:0/977932785 conn(0x7f5ba40d1060 msgr2=0x7f5ba40d1470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:14:18.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:18.982+0000 7f5ba8e00640 1 -- 172.21.15.16:0/977932785 shutdown_connections 2024-03-31T15:14:18.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:18.982+0000 7f5ba8e00640 1 -- 172.21.15.16:0/977932785 wait complete. 2024-03-31T15:14:18.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:18.982+0000 7f5ba8e00640 1 Processor -- start 2024-03-31T15:14:18.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:18.982+0000 7f5ba8e00640 1 -- start start 2024-03-31T15:14:18.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:18.986+0000 7f5ba8e00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5ba406b620 0x7f5ba415fb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:18.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:18.986+0000 7f5ba8e00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f5ba406c1c0 0x7f5ba415ebf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:18.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:18.986+0000 7f5ba8e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5ba4145aa0 0x7f5ba415f130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:18.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:18.986+0000 7f5ba8e00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f5ba4070dc0 con 0x7f5ba4145aa0 2024-03-31T15:14:18.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:18.986+0000 7f5ba8e00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f5ba4070c40 con 0x7f5ba406b620 2024-03-31T15:14:18.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:18.986+0000 7f5ba8e00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f5ba4070f40 con 0x7f5ba406c1c0 2024-03-31T15:14:18.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:18.986+0000 7f5ba2800640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5ba4145aa0 0x7f5ba415f130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:18.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:18.986+0000 7f5ba1e00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5ba406b620 0x7f5ba415fb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:18.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:18.986+0000 7f5ba2800640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5ba4145aa0 0x7f5ba415f130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42800/0 (socket says 172.21.15.16:42800) 2024-03-31T15:14:18.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:18.986+0000 7f5ba2800640 1 -- 172.21.15.16:0/1517136226 learned_addr learned my addr 172.21.15.16:0/1517136226 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:14:18.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:18.986+0000 7f5ba1400640 1 --2- 172.21.15.16:0/1517136226 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f5ba406c1c0 0x7f5ba415ebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:18.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:18.986+0000 7f5ba2800640 1 -- 172.21.15.16:0/1517136226 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f5ba406c1c0 msgr2=0x7f5ba415ebf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:18.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:18.986+0000 7f5ba2800640 1 --2- 172.21.15.16:0/1517136226 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f5ba406c1c0 0x7f5ba415ebf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:18.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:18.986+0000 7f5ba2800640 1 -- 172.21.15.16:0/1517136226 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5ba406b620 msgr2=0x7f5ba415fb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:18.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:18.986+0000 7f5ba2800640 1 --2- 172.21.15.16:0/1517136226 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5ba406b620 0x7f5ba415fb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:18.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:18.986+0000 7f5ba2800640 1 -- 172.21.15.16:0/1517136226 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ba415d870 con 0x7f5ba4145aa0 2024-03-31T15:14:18.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:18.986+0000 7f5ba2800640 1 --2- 172.21.15.16:0/1517136226 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5ba4145aa0 0x7f5ba415f130 secure :-1 s=READY pgs=1668 cs=0 l=1 rev1=1 crypto rx=0x7f5b9c007e90 tx=0x7f5b9c00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:18.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:18.986+0000 7f5b9a000640 1 -- 172.21.15.16:0/1517136226 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b9c014420 con 0x7f5ba4145aa0 2024-03-31T15:14:18.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:18.986+0000 7f5b9a000640 1 -- 172.21.15.16:0/1517136226 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5b9c019070 con 0x7f5ba4145aa0 2024-03-31T15:14:18.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:18.986+0000 7f5b9a000640 1 -- 172.21.15.16:0/1517136226 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b9c014800 con 0x7f5ba4145aa0 2024-03-31T15:14:18.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:18.986+0000 7f5ba8e00640 1 -- 172.21.15.16:0/1517136226 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5ba415db50 con 0x7f5ba4145aa0 2024-03-31T15:14:18.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:18.986+0000 7f5ba8e00640 1 -- 172.21.15.16:0/1517136226 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5ba415e110 con 0x7f5ba4145aa0 2024-03-31T15:14:18.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:18.986+0000 7f5ba8e00640 1 -- 172.21.15.16:0/1517136226 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5ba4069950 con 0x7f5ba4145aa0 2024-03-31T15:14:18.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:18.990+0000 7f5b9a000640 1 -- 172.21.15.16:0/1517136226 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f5b9c0149a0 con 0x7f5ba4145aa0 2024-03-31T15:14:18.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:18.990+0000 7f5b9a000640 1 --2- 172.21.15.16:0/1517136226 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f5b84068380 0x7f5b8406a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:18.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:18.990+0000 7f5b9a000640 1 -- 172.21.15.16:0/1517136226 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(110..110 src has 1..110) ==== 5767+0+0 (secure 0 0 0) 0x7f5b9c087180 con 0x7f5ba4145aa0 2024-03-31T15:14:18.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:18.990+0000 7f5ba1e00640 1 --2- 172.21.15.16:0/1517136226 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f5b84068380 0x7f5b8406a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:18.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:18.990+0000 7f5ba1e00640 1 --2- 172.21.15.16:0/1517136226 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f5b84068380 0x7f5b8406a840 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f5b940ad000 tx=0x7f5b94008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:18.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:18.994+0000 7f5b9a000640 1 -- 172.21.15.16:0/1517136226 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f5b9c057020 con 0x7f5ba4145aa0 2024-03-31T15:14:19.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.194+0000 7f5ba8e00640 1 -- 172.21.15.16:0/1517136226 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5ba40b2090 con 0x7f5ba4145aa0 2024-03-31T15:14:19.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.194+0000 7f5b9a000640 1 -- 172.21.15.16:0/1517136226 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 99 v99) ==== 94+0+5180 (secure 0 0 0) 0x7f5b9c05b320 con 0x7f5ba4145aa0 2024-03-31T15:14:19.186 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:14:19.186 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":99,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7631,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6836/1628897112","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":1628897112},{"type":"v1","addr":"172.21.15.89:6836","nonce":1628897112}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":94},{"gid":8746,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/1036724837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6835","nonce":1036724837},{"type":"v1","addr":"172.21.15.89:6837","nonce":1036724837}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":94},{"gid":9192,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6834/472139147","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":472139147},{"type":"v1","addr":"172.21.15.16:6834","nonce":472139147}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":94}],"filesystems":[{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-31T15:14:08.652180+0000","modified":"2024-03-31T15:14:11.668085+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":9195},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9195":{"gid":9195,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.16:6835/2564952016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6833","nonce":2564952016},{"type":"v1","addr":"172.21.15.16:6835","nonce":2564952016}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9195,"qdb_cluster":[9195]},"id":8}]} 2024-03-31T15:14:19.186 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 99 2024-03-31T15:14:19.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.194+0000 7f5ba8e00640 1 -- 172.21.15.16:0/1517136226 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f5b84068380 msgr2=0x7f5b8406a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:19.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.194+0000 7f5ba8e00640 1 --2- 172.21.15.16:0/1517136226 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f5b84068380 0x7f5b8406a840 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f5b940ad000 tx=0x7f5b94008040 comp rx=0 tx=0).stop 2024-03-31T15:14:19.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.194+0000 7f5ba8e00640 1 -- 172.21.15.16:0/1517136226 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5ba4145aa0 msgr2=0x7f5ba415f130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:19.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.194+0000 7f5ba8e00640 1 --2- 172.21.15.16:0/1517136226 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5ba4145aa0 0x7f5ba415f130 secure :-1 s=READY pgs=1668 cs=0 l=1 rev1=1 crypto rx=0x7f5b9c007e90 tx=0x7f5b9c00e4b0 comp rx=0 tx=0).stop 2024-03-31T15:14:19.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.194+0000 7f5ba8e00640 1 -- 172.21.15.16:0/1517136226 shutdown_connections 2024-03-31T15:14:19.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.194+0000 7f5ba8e00640 1 --2- 172.21.15.16:0/1517136226 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f5b84068380 0x7f5b8406a840 unknown :-1 s=CLOSED pgs=711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:19.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.194+0000 7f5ba8e00640 1 --2- 172.21.15.16:0/1517136226 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5ba4145aa0 0x7f5ba415f130 unknown :-1 s=CLOSED pgs=1668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:19.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.194+0000 7f5ba8e00640 1 --2- 172.21.15.16:0/1517136226 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f5ba406c1c0 0x7f5ba415ebf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:19.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.194+0000 7f5ba8e00640 1 --2- 172.21.15.16:0/1517136226 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5ba406b620 0x7f5ba415fb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:19.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.194+0000 7f5ba8e00640 1 -- 172.21.15.16:0/1517136226 >> 172.21.15.16:0/1517136226 conn(0x7f5ba40d1060 msgr2=0x7f5ba40b25b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:14:19.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.194+0000 7f5ba8e00640 1 -- 172.21.15.16:0/1517136226 shutdown_connections 2024-03-31T15:14:19.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.194+0000 7f5ba8e00640 1 -- 172.21.15.16:0/1517136226 wait complete. 2024-03-31T15:14:19.611 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 99, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-31T15:14:08.652180+0000', 'modified': '2024-03-31T15:14:11.668085+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 9195}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9195': {'gid': 9195, 'name': 'c', 'rank': 0, 'incarnation': 95, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.16:6835/2564952016', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6833', 'nonce': 2564952016}, {'type': 'v1', 'addr': '172.21.15.16:6835', 'nonce': 2564952016}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [17], 'metadata_pool': 16, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9195, 'qdb_cluster': [9195]} 2024-03-31T15:14:19.611 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-31T15:14:19.611 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-03-31T15:14:19.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.758+0000 7f5483a00640 1 Processor -- start 2024-03-31T15:14:19.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.758+0000 7f5483a00640 1 -- start start 2024-03-31T15:14:19.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.762+0000 7f5483a00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f547c145e40 0x7f547c0cc740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:19.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.762+0000 7f5483a00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f547c0ccc80 0x7f547c06f920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:19.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.762+0000 7f5483a00640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f547c0d0920 con 0x7f547c06fe60 2024-03-31T15:14:19.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.762+0000 7f5483a00640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f547c0d06d0 con 0x7f547c0ccc80 2024-03-31T15:14:19.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.762+0000 7f5483a00640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f547c0d0c00 con 0x7f547c145e40 2024-03-31T15:14:19.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.762+0000 7f5473e00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f547c0ccc80 0x7f547c06f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:19.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.762+0000 7f5473e00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f547c0ccc80 0x7f547c06f920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42814/0 (socket says 172.21.15.16:42814) 2024-03-31T15:14:19.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.762+0000 7f5473e00640 1 -- 172.21.15.16:0/2119178699 learned_addr learned my addr 172.21.15.16:0/2119178699 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:14:19.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.762+0000 7f5481000640 1 --2- 172.21.15.16:0/2119178699 >> v2:172.21.15.89:3300/0 conn(0x7f547c145e40 0x7f547c0cc740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:19.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.762+0000 7f5473e00640 1 -- 172.21.15.16:0/2119178699 >> v2:172.21.15.89:3300/0 conn(0x7f547c145e40 msgr2=0x7f547c0cc740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:14:19.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.762+0000 7f5473e00640 1 --2- 172.21.15.16:0/2119178699 >> v2:172.21.15.89:3300/0 conn(0x7f547c145e40 0x7f547c0cc740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:19.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.762+0000 7f5473e00640 1 -- 172.21.15.16:0/2119178699 >> v1:172.21.15.16:6789/0 conn(0x7f547c06fe60 legacy=0x7f547c15fdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:19.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.762+0000 7f5473e00640 1 -- 172.21.15.16:0/2119178699 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f547c0d1810 con 0x7f547c0ccc80 2024-03-31T15:14:19.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.762+0000 7f547be00640 1 -- 172.21.15.16:0/2119178699 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2474416086 0 0) 0x7f547c0d0920 con 0x7f547c06fe60 2024-03-31T15:14:19.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.766+0000 7f5473e00640 1 --2- 172.21.15.16:0/2119178699 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f547c0ccc80 0x7f547c06f920 secure :-1 s=READY pgs=1670 cs=0 l=1 rev1=1 crypto rx=0x7f547c072e60 tx=0x7f546401c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=511cb4da22d92ed0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:19.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.766+0000 7f547be00640 1 -- 172.21.15.16:0/2119178699 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54640b0070 con 0x7f547c0ccc80 2024-03-31T15:14:19.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.766+0000 7f547be00640 1 -- 172.21.15.16:0/2119178699 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f54640ab600 con 0x7f547c0ccc80 2024-03-31T15:14:19.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.766+0000 7f547be00640 1 -- 172.21.15.16:0/2119178699 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54640ab9d0 con 0x7f547c0ccc80 2024-03-31T15:14:19.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.766+0000 7f5483a00640 1 -- 172.21.15.16:0/2119178699 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f547c0ccc80 msgr2=0x7f547c06f920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:19.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.766+0000 7f5483a00640 1 --2- 172.21.15.16:0/2119178699 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f547c0ccc80 0x7f547c06f920 secure :-1 s=READY pgs=1670 cs=0 l=1 rev1=1 crypto rx=0x7f547c072e60 tx=0x7f546401c520 comp rx=0 tx=0).stop 2024-03-31T15:14:19.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.766+0000 7f5483a00640 1 -- 172.21.15.16:0/2119178699 shutdown_connections 2024-03-31T15:14:19.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.766+0000 7f5483a00640 1 --2- 172.21.15.16:0/2119178699 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f547c0ccc80 0x7f547c06f920 unknown :-1 s=CLOSED pgs=1670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:19.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.766+0000 7f5483a00640 1 --2- 172.21.15.16:0/2119178699 >> v2:172.21.15.89:3300/0 conn(0x7f547c145e40 0x7f547c0cc740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:19.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.766+0000 7f5483a00640 1 -- 172.21.15.16:0/2119178699 >> 172.21.15.16:0/2119178699 conn(0x7f547c067330 msgr2=0x7f547c067740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:14:19.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.766+0000 7f5483a00640 1 -- 172.21.15.16:0/2119178699 shutdown_connections 2024-03-31T15:14:19.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.766+0000 7f5483a00640 1 -- 172.21.15.16:0/2119178699 wait complete. 2024-03-31T15:14:19.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.766+0000 7f5483a00640 1 Processor -- start 2024-03-31T15:14:19.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.766+0000 7f5483a00640 1 -- start start 2024-03-31T15:14:19.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.766+0000 7f5483a00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f547c06fe60 0x7f547c13fd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:19.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.766+0000 7f5483a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f547c0ccc80 0x7f547c0cb440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:19.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.766+0000 7f5483a00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f547c145e40 0x7f547c0cb980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:19.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.766+0000 7f5483a00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f547c0b5240 con 0x7f547c0ccc80 2024-03-31T15:14:19.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.766+0000 7f5483a00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f547c0b50c0 con 0x7f547c145e40 2024-03-31T15:14:19.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.766+0000 7f5483a00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f547c169190 con 0x7f547c06fe60 2024-03-31T15:14:19.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.766+0000 7f5473e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f547c0ccc80 0x7f547c0cb440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:19.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.766+0000 7f5473e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f547c0ccc80 0x7f547c0cb440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42816/0 (socket says 172.21.15.16:42816) 2024-03-31T15:14:19.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.766+0000 7f5473e00640 1 -- 172.21.15.16:0/291060701 learned_addr learned my addr 172.21.15.16:0/291060701 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:14:19.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.766+0000 7f5481a00640 1 --2- 172.21.15.16:0/291060701 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f547c145e40 0x7f547c0cb980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:19.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.770+0000 7f5481000640 1 --2- 172.21.15.16:0/291060701 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f547c06fe60 0x7f547c13fd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:19.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.770+0000 7f5473e00640 1 -- 172.21.15.16:0/291060701 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f547c06fe60 msgr2=0x7f547c13fd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:19.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.770+0000 7f5473e00640 1 --2- 172.21.15.16:0/291060701 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f547c06fe60 0x7f547c13fd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:19.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.770+0000 7f5473e00640 1 -- 172.21.15.16:0/291060701 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f547c145e40 msgr2=0x7f547c0cb980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:19.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.770+0000 7f5473e00640 1 --2- 172.21.15.16:0/291060701 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f547c145e40 0x7f547c0cb980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:19.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.770+0000 7f5473e00640 1 -- 172.21.15.16:0/291060701 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f547c142b80 con 0x7f547c0ccc80 2024-03-31T15:14:19.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.770+0000 7f5473e00640 1 --2- 172.21.15.16:0/291060701 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f547c0ccc80 0x7f547c0cb440 secure :-1 s=READY pgs=1671 cs=0 l=1 rev1=1 crypto rx=0x7f546401a150 tx=0x7f546401ad60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:19.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.770+0000 7f5479600640 1 -- 172.21.15.16:0/291060701 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54640b0040 con 0x7f547c0ccc80 2024-03-31T15:14:19.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.770+0000 7f5479600640 1 -- 172.21.15.16:0/291060701 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5464004030 con 0x7f547c0ccc80 2024-03-31T15:14:19.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.770+0000 7f5479600640 1 -- 172.21.15.16:0/291060701 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5464004400 con 0x7f547c0ccc80 2024-03-31T15:14:19.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.770+0000 7f5483a00640 1 -- 172.21.15.16:0/291060701 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f547c143440 con 0x7f547c0ccc80 2024-03-31T15:14:19.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.770+0000 7f5483a00640 1 -- 172.21.15.16:0/291060701 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f547c143a00 con 0x7f547c0ccc80 2024-03-31T15:14:19.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.770+0000 7f5479600640 1 -- 172.21.15.16:0/291060701 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f5464003620 con 0x7f547c0ccc80 2024-03-31T15:14:19.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.770+0000 7f5479600640 1 --2- 172.21.15.16:0/291060701 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f54580683d0 0x7f545806a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:19.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.770+0000 7f5479600640 1 -- 172.21.15.16:0/291060701 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 5767+0+0 (secure 0 0 0) 0x7f5464120ac0 con 0x7f547c0ccc80 2024-03-31T15:14:19.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.774+0000 7f5481000640 1 --2- 172.21.15.16:0/291060701 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f54580683d0 0x7f545806a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:19.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.774+0000 7f5486bff000 1 Processor -- start 2024-03-31T15:14:19.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.774+0000 7f5486bff000 1 -- start start 2024-03-31T15:14:19.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.774+0000 7f5481000640 1 --2- 172.21.15.16:0/291060701 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f54580683d0 0x7f545806a890 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f546c009af0 tx=0x7f546c008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:19.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.774+0000 7f5486bff000 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5629eb5234a0 0x5629eb523880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:19.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.774+0000 7f5486bff000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5629eb523dc0 0x5629eb4d6b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:19.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.774+0000 7f5486bff000 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5629eb4d70b0 0x5629eb4d7490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:19.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.774+0000 7f5486bff000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5629eb329950 con 0x5629eb523dc0 2024-03-31T15:14:19.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.774+0000 7f5486bff000 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x5629eb2c6990 con 0x5629eb5234a0 2024-03-31T15:14:19.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.774+0000 7f5486bff000 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x5629eb325900 con 0x5629eb4d70b0 2024-03-31T15:14:19.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.774+0000 7f5473e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5629eb523dc0 0x5629eb4d6b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:19.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.774+0000 7f5473e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5629eb523dc0 0x5629eb4d6b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42822/0 (socket says 172.21.15.16:42822) 2024-03-31T15:14:19.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.774+0000 7f5473e00640 1 -- 172.21.15.16:0/1406871706 learned_addr learned my addr 172.21.15.16:0/1406871706 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:14:19.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.774+0000 7f5481000640 1 --2- 172.21.15.16:0/1406871706 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5629eb5234a0 0x5629eb523880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:19.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.774+0000 7f5481a00640 1 --2- 172.21.15.16:0/1406871706 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5629eb4d70b0 0x5629eb4d7490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:19.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.774+0000 7f5473e00640 1 -- 172.21.15.16:0/1406871706 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5629eb4d70b0 msgr2=0x5629eb4d7490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:14:19.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.774+0000 7f5473e00640 1 --2- 172.21.15.16:0/1406871706 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5629eb4d70b0 0x5629eb4d7490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:19.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.774+0000 7f5473e00640 1 -- 172.21.15.16:0/1406871706 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5629eb5234a0 msgr2=0x5629eb523880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:14:19.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.774+0000 7f5473e00640 1 --2- 172.21.15.16:0/1406871706 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5629eb5234a0 0x5629eb523880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:19.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.774+0000 7f5473e00640 1 -- 172.21.15.16:0/1406871706 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5629eb36e100 con 0x5629eb523dc0 2024-03-31T15:14:19.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.774+0000 7f5481a00640 1 --2- 172.21.15.16:0/1406871706 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5629eb4d70b0 0x5629eb4d7490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:14:19.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.774+0000 7f5473e00640 1 --2- 172.21.15.16:0/1406871706 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5629eb523dc0 0x5629eb4d6b70 secure :-1 s=READY pgs=1672 cs=0 l=1 rev1=1 crypto rx=0x7f546411f120 tx=0x7f54640a8000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ba51d85dbd87585 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:19.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.774+0000 7f5471600640 1 -- 172.21.15.16:0/1406871706 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54640f0bd0 con 0x5629eb523dc0 2024-03-31T15:14:19.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.774+0000 7f5471600640 1 -- 172.21.15.16:0/1406871706 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f54640f4f80 con 0x5629eb523dc0 2024-03-31T15:14:19.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.774+0000 7f5471600640 1 -- 172.21.15.16:0/1406871706 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54640f97b0 con 0x5629eb523dc0 2024-03-31T15:14:19.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.774+0000 7f5486bff000 1 -- 172.21.15.16:0/1406871706 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5629eb523dc0 msgr2=0x5629eb4d6b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:19.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.774+0000 7f5486bff000 1 --2- 172.21.15.16:0/1406871706 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5629eb523dc0 0x5629eb4d6b70 secure :-1 s=READY pgs=1672 cs=0 l=1 rev1=1 crypto rx=0x7f546411f120 tx=0x7f54640a8000 comp rx=0 tx=0).stop 2024-03-31T15:14:19.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.774+0000 7f5486bff000 1 -- 172.21.15.16:0/1406871706 shutdown_connections 2024-03-31T15:14:19.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.774+0000 7f5486bff000 1 --2- 172.21.15.16:0/1406871706 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5629eb4d70b0 0x5629eb4d7490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:19.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.774+0000 7f5486bff000 1 --2- 172.21.15.16:0/1406871706 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5629eb523dc0 0x5629eb4d6b70 unknown :-1 s=CLOSED pgs=1672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:19.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.774+0000 7f5486bff000 1 --2- 172.21.15.16:0/1406871706 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5629eb5234a0 0x5629eb523880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:19.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.774+0000 7f5486bff000 1 -- 172.21.15.16:0/1406871706 >> 172.21.15.16:0/1406871706 conn(0x5629eb3c1b80 msgr2=0x5629eb506460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:14:19.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.774+0000 7f5486bff000 1 -- 172.21.15.16:0/1406871706 shutdown_connections 2024-03-31T15:14:19.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.774+0000 7f5486bff000 1 -- 172.21.15.16:0/1406871706 wait complete. 2024-03-31T15:14:19.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.774+0000 7f5486bff000 1 -- start start 2024-03-31T15:14:19.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.778+0000 7f5486bff000 1 Processor -- start 2024-03-31T15:14:19.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.778+0000 7f5486bff000 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5629eb4d70b0 0x5629eb4e3e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:19.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.778+0000 7f5486bff000 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5629eb5234a0 0x5629eb4e4340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:19.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.778+0000 7f5486bff000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5629eb523dc0 0x5629eb4e4880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:19.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.778+0000 7f5486bff000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5629eb328c60 con 0x5629eb523dc0 2024-03-31T15:14:19.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.778+0000 7f5486bff000 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x5629eb2e73e0 con 0x5629eb4d70b0 2024-03-31T15:14:19.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.778+0000 7f5486bff000 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x5629eb329950 con 0x5629eb5234a0 2024-03-31T15:14:19.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.778+0000 7f5486bff000 10 client.0 _ll_register_callbacks cb 0x5629eb38a0f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T15:14:19.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.778+0000 7f5481a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5629eb523dc0 0x5629eb4e4880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:19.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.778+0000 7f5481a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5629eb523dc0 0x5629eb4e4880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42828/0 (socket says 172.21.15.16:42828) 2024-03-31T15:14:19.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.778+0000 7f5481a00640 1 -- 172.21.15.16:0/2454902042 learned_addr learned my addr 172.21.15.16:0/2454902042 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:14:19.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.778+0000 7f5481000640 1 --2- 172.21.15.16:0/2454902042 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5629eb4d70b0 0x5629eb4e3e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:19.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.778+0000 7f5473e00640 1 --2- 172.21.15.16:0/2454902042 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5629eb5234a0 0x5629eb4e4340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:19.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.778+0000 7f5481a00640 1 -- 172.21.15.16:0/2454902042 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5629eb5234a0 msgr2=0x5629eb4e4340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:14:19.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.778+0000 7f5481a00640 1 --2- 172.21.15.16:0/2454902042 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5629eb5234a0 0x5629eb4e4340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:19.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.778+0000 7f5481a00640 1 -- 172.21.15.16:0/2454902042 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5629eb4d70b0 msgr2=0x5629eb4e3e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:14:19.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.778+0000 7f5481a00640 1 --2- 172.21.15.16:0/2454902042 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5629eb4d70b0 0x5629eb4e3e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:19.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.778+0000 7f5481a00640 1 -- 172.21.15.16:0/2454902042 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5629eb2f38c0 con 0x5629eb523dc0 2024-03-31T15:14:19.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.778+0000 7f5481a00640 1 --2- 172.21.15.16:0/2454902042 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5629eb523dc0 0x5629eb4e4880 secure :-1 s=READY pgs=1673 cs=0 l=1 rev1=1 crypto rx=0x7f5474005970 tx=0x7f54740018c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f13eb8475471a4c3 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:19.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.778+0000 7f546a000640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-31T15:14:19.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.778+0000 7f546a000640 1 -- 172.21.15.16:0/2454902042 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5474013570 con 0x5629eb523dc0 2024-03-31T15:14:19.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.778+0000 7f546a000640 1 -- 172.21.15.16:0/2454902042 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(111..111 src has 1..111) ==== 5767+0+0 (secure 0 0 0) 0x7f5474013710 con 0x5629eb523dc0 2024-03-31T15:14:19.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.778+0000 7f5486bff000 1 -- 172.21.15.16:0/2454902042 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f54640f4f80 con 0x5629eb523dc0 2024-03-31T15:14:19.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.778+0000 7f546a000640 1 -- 172.21.15.16:0/2454902042 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=99) ==== 24+0+0 (secure 0 0 0) 0x7f5474021050 con 0x5629eb523dc0 2024-03-31T15:14:19.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.778+0000 7f5486bff000 10 client.9366 fetch_fsmap learned FSMap version 99 2024-03-31T15:14:19.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.778+0000 7f5486bff000 1 -- 172.21.15.16:0/2454902042 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=99,monmap=2+}) -- 0x5629eb2b1e80 con 0x5629eb523dc0 2024-03-31T15:14:19.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.778+0000 7f546a000640 1 -- 172.21.15.16:0/2454902042 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f54640f4f80 con 0x5629eb523dc0 2024-03-31T15:14:19.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.778+0000 7f546a000640 1 -- 172.21.15.16:0/2454902042 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 99) ==== 3494+0+0 (secure 0 0 0) 0x7f5474024020 con 0x5629eb523dc0 2024-03-31T15:14:19.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.778+0000 7f5486bff000 10 client.9366 fetch_fsmap finished waiting for FSMap version 99 2024-03-31T15:14:19.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.778+0000 7f5486bff000 10 client.9366 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-31T15:14:19.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.778+0000 7f5486bff000 20 client.9366 populate_metadata read hostname 'smithi016' 2024-03-31T15:14:19.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.778+0000 7f5486bff000 1 --2- 172.21.15.16:0/2454902042 >> [v2:172.21.15.16:6833/2564952016,v1:172.21.15.16:6835/2564952016] conn(0x5629eb4e5120 0x5629eb535a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:19.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.778+0000 7f5486bff000 4 client.9366 mds_command: new command op to 9195 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-31T15:14:19.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.778+0000 7f5486bff000 1 -- 172.21.15.16:0/2454902042 --> [v2:172.21.15.16:6833/2564952016,v1:172.21.15.16:6835/2564952016] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5629eb2c4bc0 con 0x5629eb4e5120 2024-03-31T15:14:19.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.778+0000 7f5481000640 1 --2- 172.21.15.16:0/2454902042 >> [v2:172.21.15.16:6833/2564952016,v1:172.21.15.16:6835/2564952016] conn(0x5629eb4e5120 0x5629eb535a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:19.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.778+0000 7f5481000640 1 --2- 172.21.15.16:0/2454902042 >> [v2:172.21.15.16:6833/2564952016,v1:172.21.15.16:6835/2564952016] conn(0x5629eb4e5120 0x5629eb535a20 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9eb41f4c0c17a00 server_cookie=b522fd7dc3d7876a in_seq=0 out_seq=0 2024-03-31T15:14:19.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.778+0000 7f546a000640 10 client.9366 ms_handle_connect on v2:172.21.15.16:6833/2564952016 2024-03-31T15:14:19.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.782+0000 7f546a000640 1 -- 172.21.15.16:0/2454902042 <== mds.0 v2:172.21.15.16:6833/2564952016 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5629eb2c4bc0 con 0x5629eb4e5120 2024-03-31T15:14:19.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.782+0000 7f546a000640 10 client.9366 handle_command_reply: tid=0 2024-03-31T15:14:19.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.782+0000 7f5486bff000 1 client.9366 shutdown 2024-03-31T15:14:19.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.782+0000 7f5486bff000 1 -- 172.21.15.16:0/2454902042 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5629eb523dc0 msgr2=0x5629eb4e4880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:19.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.782+0000 7f5486bff000 1 --2- 172.21.15.16:0/2454902042 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5629eb523dc0 0x5629eb4e4880 secure :-1 s=READY pgs=1673 cs=0 l=1 rev1=1 crypto rx=0x7f5474005970 tx=0x7f54740018c0 comp rx=0 tx=0).stop 2024-03-31T15:14:19.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.782+0000 7f5486bff000 1 -- 172.21.15.16:0/2454902042 shutdown_connections 2024-03-31T15:14:19.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.782+0000 7f5486bff000 1 --2- 172.21.15.16:0/2454902042 >> [v2:172.21.15.16:6833/2564952016,v1:172.21.15.16:6835/2564952016] conn(0x5629eb4e5120 0x5629eb535a20 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:19.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.782+0000 7f5486bff000 1 --2- 172.21.15.16:0/2454902042 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5629eb523dc0 0x5629eb4e4880 unknown :-1 s=CLOSED pgs=1673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:19.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.782+0000 7f5486bff000 1 --2- 172.21.15.16:0/2454902042 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5629eb5234a0 0x5629eb4e4340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:19.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.782+0000 7f5486bff000 1 --2- 172.21.15.16:0/2454902042 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5629eb4d70b0 0x5629eb4e3e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:19.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.782+0000 7f546a000640 0 client.9366 ms_handle_reset on v2:172.21.15.16:6833/2564952016 2024-03-31T15:14:19.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.782+0000 7f5486bff000 1 -- 172.21.15.16:0/2454902042 >> 172.21.15.16:0/2454902042 conn(0x5629eb3c1b80 msgr2=0x5629eb4d6270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:14:19.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.782+0000 7f5486bff000 1 -- 172.21.15.16:0/2454902042 shutdown_connections 2024-03-31T15:14:19.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.782+0000 7f5486bff000 1 -- 172.21.15.16:0/2454902042 wait complete. 2024-03-31T15:14:19.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.782+0000 7f5486bff000 20 client.9366 trim_cache size 0 max 16384 2024-03-31T15:14:19.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.798+0000 7f5486bff000 1 Processor -- start 2024-03-31T15:14:19.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.798+0000 7f5486bff000 1 -- start start 2024-03-31T15:14:19.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.802+0000 7f5486bff000 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5629eb4d70b0 0x5629eb4d62e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:19.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.802+0000 7f5486bff000 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5629eb5234a0 0x5629eb4d6820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:19.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.802+0000 7f5486bff000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5629eb523dc0 0x5629eb505bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:19.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.802+0000 7f5486bff000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f547c203820 con 0x5629eb523dc0 2024-03-31T15:14:19.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.802+0000 7f5486bff000 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f547c202880 con 0x5629eb4d70b0 2024-03-31T15:14:19.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.802+0000 7f5486bff000 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f547c201be0 con 0x5629eb5234a0 2024-03-31T15:14:19.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.802+0000 7f5481a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5629eb523dc0 0x5629eb505bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:19.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.802+0000 7f5473e00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5629eb5234a0 0x5629eb4d6820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:19.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.802+0000 7f5481a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5629eb523dc0 0x5629eb505bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42834/0 (socket says 172.21.15.16:42834) 2024-03-31T15:14:19.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.802+0000 7f5481000640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5629eb4d70b0 0x5629eb4d62e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:19.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.802+0000 7f5473e00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5629eb5234a0 0x5629eb4d6820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3301/0 says I am v2:172.21.15.16:38724/0 (socket says 172.21.15.16:38724) 2024-03-31T15:14:19.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.802+0000 7f5481a00640 1 -- 172.21.15.16:0/1769914116 learned_addr learned my addr 172.21.15.16:0/1769914116 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:14:19.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.802+0000 7f5481a00640 1 -- 172.21.15.16:0/1769914116 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5629eb5234a0 msgr2=0x5629eb4d6820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:14:19.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.802+0000 7f5481a00640 1 --2- 172.21.15.16:0/1769914116 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5629eb5234a0 0x5629eb4d6820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:19.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.802+0000 7f5481a00640 1 -- 172.21.15.16:0/1769914116 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5629eb4d70b0 msgr2=0x5629eb4d62e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:14:19.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.802+0000 7f5481a00640 1 --2- 172.21.15.16:0/1769914116 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5629eb4d70b0 0x5629eb4d62e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:19.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.802+0000 7f5481a00640 1 -- 172.21.15.16:0/1769914116 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5629eb2e4490 con 0x5629eb523dc0 2024-03-31T15:14:19.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.802+0000 7f5481000640 1 --2- 172.21.15.16:0/1769914116 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5629eb4d70b0 0x5629eb4d62e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:14:19.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.802+0000 7f5473e00640 1 --2- 172.21.15.16:0/1769914116 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5629eb5234a0 0x5629eb4d6820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:14:19.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.802+0000 7f5481a00640 1 --2- 172.21.15.16:0/1769914116 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5629eb523dc0 0x5629eb505bb0 secure :-1 s=READY pgs=1674 cs=0 l=1 rev1=1 crypto rx=0x7f547401f040 tx=0x7f547400e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=549231bec1b87b8f server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:19.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.802+0000 7f5470c00640 1 -- 172.21.15.16:0/1769914116 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54740053f0 con 0x5629eb523dc0 2024-03-31T15:14:19.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.802+0000 7f5470c00640 1 -- 172.21.15.16:0/1769914116 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5474019070 con 0x5629eb523dc0 2024-03-31T15:14:19.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.802+0000 7f5470c00640 1 -- 172.21.15.16:0/1769914116 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5474013ea0 con 0x5629eb523dc0 2024-03-31T15:14:19.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.802+0000 7f5486bff000 1 -- 172.21.15.16:0/1769914116 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5629eb523dc0 msgr2=0x5629eb505bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:19.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.802+0000 7f5486bff000 1 --2- 172.21.15.16:0/1769914116 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5629eb523dc0 0x5629eb505bb0 secure :-1 s=READY pgs=1674 cs=0 l=1 rev1=1 crypto rx=0x7f547401f040 tx=0x7f547400e000 comp rx=0 tx=0).stop 2024-03-31T15:14:19.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.802+0000 7f5486bff000 1 -- 172.21.15.16:0/1769914116 shutdown_connections 2024-03-31T15:14:19.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.802+0000 7f5486bff000 1 --2- 172.21.15.16:0/1769914116 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5629eb523dc0 0x5629eb505bb0 unknown :-1 s=CLOSED pgs=1674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:19.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.802+0000 7f5486bff000 1 --2- 172.21.15.16:0/1769914116 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5629eb5234a0 0x5629eb4d6820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:19.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.802+0000 7f5486bff000 1 --2- 172.21.15.16:0/1769914116 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5629eb4d70b0 0x5629eb4d62e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:19.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.802+0000 7f5486bff000 1 -- 172.21.15.16:0/1769914116 >> 172.21.15.16:0/1769914116 conn(0x5629eb3c1b80 msgr2=0x5629eb4f3e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:14:19.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.802+0000 7f5486bff000 1 -- 172.21.15.16:0/1769914116 shutdown_connections 2024-03-31T15:14:19.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.802+0000 7f5486bff000 1 -- 172.21.15.16:0/1769914116 wait complete. 2024-03-31T15:14:19.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.802+0000 7f5486bff000 1 -- start start 2024-03-31T15:14:19.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.802+0000 7f5486bff000 1 Processor -- start 2024-03-31T15:14:19.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.806+0000 7f5486bff000 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5629eb4d70b0 0x5629eb58fdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:19.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.806+0000 7f5486bff000 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5629eb5234a0 0x5629eb504ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:19.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.806+0000 7f5486bff000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5629eb523dc0 0x5629eb5050e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:19.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.806+0000 7f5486bff000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5629eb336060 con 0x5629eb523dc0 2024-03-31T15:14:19.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.806+0000 7f5486bff000 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f547c202230 con 0x5629eb4d70b0 2024-03-31T15:14:19.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.806+0000 7f5486bff000 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x5629eb32df80 con 0x5629eb5234a0 2024-03-31T15:14:19.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.806+0000 7f5486bff000 10 client.0 _ll_register_callbacks cb 0x7f5474003b40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T15:14:19.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.806+0000 7f5481a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5629eb523dc0 0x5629eb5050e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:19.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.806+0000 7f5481a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5629eb523dc0 0x5629eb5050e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42848/0 (socket says 172.21.15.16:42848) 2024-03-31T15:14:19.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.806+0000 7f5481a00640 1 -- 172.21.15.16:0/3363085819 learned_addr learned my addr 172.21.15.16:0/3363085819 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:14:19.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.806+0000 7f5481000640 1 --2- 172.21.15.16:0/3363085819 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5629eb4d70b0 0x5629eb58fdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:19.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.806+0000 7f5473e00640 1 --2- 172.21.15.16:0/3363085819 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5629eb5234a0 0x5629eb504ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:19.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.806+0000 7f5481a00640 1 -- 172.21.15.16:0/3363085819 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5629eb5234a0 msgr2=0x5629eb504ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:14:19.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.806+0000 7f5481a00640 1 --2- 172.21.15.16:0/3363085819 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5629eb5234a0 0x5629eb504ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:19.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.806+0000 7f5481a00640 1 -- 172.21.15.16:0/3363085819 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5629eb4d70b0 msgr2=0x5629eb58fdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:14:19.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.806+0000 7f5481a00640 1 --2- 172.21.15.16:0/3363085819 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5629eb4d70b0 0x5629eb58fdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:19.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.806+0000 7f5481a00640 1 -- 172.21.15.16:0/3363085819 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5629eb2e2e80 con 0x5629eb523dc0 2024-03-31T15:14:19.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.806+0000 7f5481a00640 1 --2- 172.21.15.16:0/3363085819 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5629eb523dc0 0x5629eb5050e0 secure :-1 s=READY pgs=1675 cs=0 l=1 rev1=1 crypto rx=0x5629eb2e7fa0 tx=0x7f5474009040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e99fbb0ac4e6324f server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:19.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.806+0000 7f546a000640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-31T15:14:19.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.806+0000 7f546a000640 1 -- 172.21.15.16:0/3363085819 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5474034070 con 0x5629eb523dc0 2024-03-31T15:14:19.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.806+0000 7f546a000640 1 -- 172.21.15.16:0/3363085819 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(111..111 src has 1..111) ==== 5767+0+0 (secure 0 0 0) 0x7f5474013770 con 0x5629eb523dc0 2024-03-31T15:14:19.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.806+0000 7f5486bff000 1 -- 172.21.15.16:0/3363085819 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5474019070 con 0x5629eb523dc0 2024-03-31T15:14:19.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.806+0000 7f546a000640 1 -- 172.21.15.16:0/3363085819 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=99) ==== 24+0+0 (secure 0 0 0) 0x7f547403f050 con 0x5629eb523dc0 2024-03-31T15:14:19.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.806+0000 7f5486bff000 10 client.9372 fetch_fsmap learned FSMap version 99 2024-03-31T15:14:19.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.806+0000 7f5486bff000 1 -- 172.21.15.16:0/3363085819 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=99,monmap=2+}) -- 0x5629eb2d0270 con 0x5629eb523dc0 2024-03-31T15:14:19.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.806+0000 7f546a000640 1 -- 172.21.15.16:0/3363085819 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5474019070 con 0x5629eb523dc0 2024-03-31T15:14:19.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.806+0000 7f546a000640 1 -- 172.21.15.16:0/3363085819 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 99) ==== 3494+0+0 (secure 0 0 0) 0x7f5474042020 con 0x5629eb523dc0 2024-03-31T15:14:19.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.806+0000 7f5486bff000 10 client.9372 fetch_fsmap finished waiting for FSMap version 99 2024-03-31T15:14:19.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.806+0000 7f5486bff000 10 client.9372 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-31T15:14:19.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.806+0000 7f5486bff000 20 client.9372 populate_metadata read hostname 'smithi016' 2024-03-31T15:14:19.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.806+0000 7f5486bff000 1 --2- 172.21.15.16:0/3363085819 >> [v2:172.21.15.16:6833/2564952016,v1:172.21.15.16:6835/2564952016] conn(0x5629eb590330 0x5629eb505620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:19.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.806+0000 7f5486bff000 4 client.9372 mds_command: new command op to 9195 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-31T15:14:19.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.806+0000 7f5486bff000 1 -- 172.21.15.16:0/3363085819 --> [v2:172.21.15.16:6833/2564952016,v1:172.21.15.16:6835/2564952016] -- command(tid 0: {"prefix": "status"}) -- 0x5629eb336e90 con 0x5629eb590330 2024-03-31T15:14:19.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.806+0000 7f5481000640 1 --2- 172.21.15.16:0/3363085819 >> [v2:172.21.15.16:6833/2564952016,v1:172.21.15.16:6835/2564952016] conn(0x5629eb590330 0x5629eb505620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:19.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.806+0000 7f5481000640 1 --2- 172.21.15.16:0/3363085819 >> [v2:172.21.15.16:6833/2564952016,v1:172.21.15.16:6835/2564952016] conn(0x5629eb590330 0x5629eb505620 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f9fa26b0c41f94b7 server_cookie=4dc865f28819e5fb in_seq=0 out_seq=0 2024-03-31T15:14:19.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.806+0000 7f546a000640 10 client.9372 ms_handle_connect on v2:172.21.15.16:6833/2564952016 2024-03-31T15:14:19.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.806+0000 7f546a000640 1 -- 172.21.15.16:0/3363085819 <== mds.0 v2:172.21.15.16:6833/2564952016 1 ==== command_reply(tid 0: 0 ) ==== 8+0+325 (crc 0 0 0) 0x5629eb336e90 con 0x5629eb590330 2024-03-31T15:14:19.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.806+0000 7f546a000640 10 client.9372 handle_command_reply: tid=0 2024-03-31T15:14:19.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.806+0000 7f5486bff000 1 client.9372 shutdown 2024-03-31T15:14:19.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.806+0000 7f5486bff000 1 -- 172.21.15.16:0/3363085819 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5629eb523dc0 msgr2=0x5629eb5050e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:19.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.806+0000 7f5486bff000 1 --2- 172.21.15.16:0/3363085819 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5629eb523dc0 0x5629eb5050e0 secure :-1 s=READY pgs=1675 cs=0 l=1 rev1=1 crypto rx=0x5629eb2e7fa0 tx=0x7f5474009040 comp rx=0 tx=0).stop 2024-03-31T15:14:19.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.806+0000 7f5486bff000 1 -- 172.21.15.16:0/3363085819 shutdown_connections 2024-03-31T15:14:19.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.806+0000 7f5486bff000 1 --2- 172.21.15.16:0/3363085819 >> [v2:172.21.15.16:6833/2564952016,v1:172.21.15.16:6835/2564952016] conn(0x5629eb590330 0x5629eb505620 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:19.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.806+0000 7f5486bff000 1 --2- 172.21.15.16:0/3363085819 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5629eb523dc0 0x5629eb5050e0 unknown :-1 s=CLOSED pgs=1675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:19.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.806+0000 7f5486bff000 1 --2- 172.21.15.16:0/3363085819 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5629eb5234a0 0x5629eb504ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:19.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.806+0000 7f5486bff000 1 --2- 172.21.15.16:0/3363085819 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5629eb4d70b0 0x5629eb58fdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:19.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.806+0000 7f5486bff000 1 -- 172.21.15.16:0/3363085819 >> 172.21.15.16:0/3363085819 conn(0x5629eb3c1b80 msgr2=0x5629eb4de1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:14:19.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.806+0000 7f546a000640 0 client.9372 ms_handle_reset on v2:172.21.15.16:6833/2564952016 2024-03-31T15:14:19.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.806+0000 7f5486bff000 1 -- 172.21.15.16:0/3363085819 shutdown_connections 2024-03-31T15:14:19.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.806+0000 7f5486bff000 1 -- 172.21.15.16:0/3363085819 wait complete. 2024-03-31T15:14:19.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.806+0000 7f5486bff000 20 client.9372 trim_cache size 0 max 16384 2024-03-31T15:14:19.804 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-03-31T15:14:19.804 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_fsid": "286eb5da-fb47-4550-b290-d9519c13b9a5", 2024-03-31T15:14:19.804 INFO:teuthology.orchestra.run.smithi016.stdout: "whoami": 0, 2024-03-31T15:14:19.804 INFO:teuthology.orchestra.run.smithi016.stdout: "id": 9195, 2024-03-31T15:14:19.805 INFO:teuthology.orchestra.run.smithi016.stdout: "want_state": "up:active", 2024-03-31T15:14:19.805 INFO:teuthology.orchestra.run.smithi016.stdout: "state": "up:active", 2024-03-31T15:14:19.805 INFO:teuthology.orchestra.run.smithi016.stdout: "fs_name": "cephfs", 2024-03-31T15:14:19.805 INFO:teuthology.orchestra.run.smithi016.stdout: "rank_uptime": 10.5443658, 2024-03-31T15:14:19.805 INFO:teuthology.orchestra.run.smithi016.stdout: "mdsmap_epoch": 99, 2024-03-31T15:14:19.805 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch": 111, 2024-03-31T15:14:19.805 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch_barrier": 109, 2024-03-31T15:14:19.805 INFO:teuthology.orchestra.run.smithi016.stdout: "uptime": 16.381610987999998 2024-03-31T15:14:19.805 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-03-31T15:14:19.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.810+0000 7f5483a00640 1 -- 172.21.15.16:0/291060701 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f54580683d0 msgr2=0x7f545806a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:19.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.810+0000 7f5483a00640 1 --2- 172.21.15.16:0/291060701 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f54580683d0 0x7f545806a890 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f546c009af0 tx=0x7f546c008040 comp rx=0 tx=0).stop 2024-03-31T15:14:19.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.810+0000 7f5483a00640 1 -- 172.21.15.16:0/291060701 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f547c0ccc80 msgr2=0x7f547c0cb440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:19.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.810+0000 7f5483a00640 1 --2- 172.21.15.16:0/291060701 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f547c0ccc80 0x7f547c0cb440 secure :-1 s=READY pgs=1671 cs=0 l=1 rev1=1 crypto rx=0x7f546401a150 tx=0x7f546401ad60 comp rx=0 tx=0).stop 2024-03-31T15:14:19.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.810+0000 7f5483a00640 1 -- 172.21.15.16:0/291060701 shutdown_connections 2024-03-31T15:14:19.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.810+0000 7f5483a00640 1 --2- 172.21.15.16:0/291060701 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f54580683d0 0x7f545806a890 unknown :-1 s=CLOSED pgs=712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:19.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.810+0000 7f5483a00640 1 --2- 172.21.15.16:0/291060701 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f547c145e40 0x7f547c0cb980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:19.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.810+0000 7f5483a00640 1 --2- 172.21.15.16:0/291060701 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f547c0ccc80 0x7f547c0cb440 unknown :-1 s=CLOSED pgs=1671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:19.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.810+0000 7f5483a00640 1 --2- 172.21.15.16:0/291060701 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f547c06fe60 0x7f547c13fd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:19.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.810+0000 7f5483a00640 1 -- 172.21.15.16:0/291060701 >> 172.21.15.16:0/291060701 conn(0x7f547c067330 msgr2=0x7f547c15c400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:14:19.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.810+0000 7f5483a00640 1 -- 172.21.15.16:0/291060701 shutdown_connections 2024-03-31T15:14:19.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.810+0000 7f5483a00640 1 -- 172.21.15.16:0/291060701 wait complete. 2024-03-31T15:14:19.821 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-31T15:14:19.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.966+0000 7f28d9a00640 1 Processor -- start 2024-03-31T15:14:19.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.966+0000 7f28d9a00640 1 -- start start 2024-03-31T15:14:19.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.970+0000 7f28d9a00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f28d406dbc0 0x7f28d415f650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:19.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.970+0000 7f28d9a00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f28d40cb440 0x7f28d406f920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:19.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.970+0000 7f28d9a00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f28d40cb8c0 0x7f28d406fe60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:19.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.970+0000 7f28d9a00640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f28d415fb90 con 0x7f28d40cb8c0 2024-03-31T15:14:19.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.970+0000 7f28d9a00640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f28d415f1c0 con 0x7f28d40cb440 2024-03-31T15:14:19.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.970+0000 7f28d9a00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f28d40cbe90 con 0x7f28d406dbc0 2024-03-31T15:14:19.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.970+0000 7f28d3400640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f28d40cb8c0 0x7f28d406fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:19.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.970+0000 7f28d3400640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f28d40cb8c0 0x7f28d406fe60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42862/0 (socket says 172.21.15.16:42862) 2024-03-31T15:14:19.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.970+0000 7f28d3400640 1 -- 172.21.15.16:0/2732885598 learned_addr learned my addr 172.21.15.16:0/2732885598 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:14:19.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.970+0000 7f28cbe00640 1 --2- 172.21.15.16:0/2732885598 >> v2:172.21.15.89:3300/0 conn(0x7f28d40cb440 0x7f28d406f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:19.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.970+0000 7f28d2a00640 1 --2- 172.21.15.16:0/2732885598 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f28d406dbc0 0x7f28d415f650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:19.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.970+0000 7f28d3400640 1 -- 172.21.15.16:0/2732885598 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f28d406dbc0 msgr2=0x7f28d415f650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:14:19.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.970+0000 7f28d3400640 1 --2- 172.21.15.16:0/2732885598 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f28d406dbc0 0x7f28d415f650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:19.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.970+0000 7f28d3400640 1 -- 172.21.15.16:0/2732885598 >> v2:172.21.15.89:3300/0 conn(0x7f28d40cb440 msgr2=0x7f28d406f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:14:19.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.970+0000 7f28d3400640 1 --2- 172.21.15.16:0/2732885598 >> v2:172.21.15.89:3300/0 conn(0x7f28d40cb440 0x7f28d406f920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:19.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.970+0000 7f28d3400640 1 -- 172.21.15.16:0/2732885598 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28d40cc170 con 0x7f28d40cb8c0 2024-03-31T15:14:19.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.974+0000 7f28d3400640 1 --2- 172.21.15.16:0/2732885598 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f28d40cb8c0 0x7f28d406fe60 secure :-1 s=READY pgs=1676 cs=0 l=1 rev1=1 crypto rx=0x7f28d40d31b0 tx=0x7f28cc01c930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96156032d26a4d3c server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:19.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.974+0000 7f28d2000640 1 -- 172.21.15.16:0/2732885598 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28cc0a5770 con 0x7f28d40cb8c0 2024-03-31T15:14:19.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.974+0000 7f28d2000640 1 -- 172.21.15.16:0/2732885598 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f28cc0a5910 con 0x7f28d40cb8c0 2024-03-31T15:14:19.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.974+0000 7f28d2000640 1 -- 172.21.15.16:0/2732885598 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28cc0a5ce0 con 0x7f28d40cb8c0 2024-03-31T15:14:19.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.974+0000 7f28d9a00640 1 -- 172.21.15.16:0/2732885598 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f28d40cb8c0 msgr2=0x7f28d406fe60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:19.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.974+0000 7f28d9a00640 1 --2- 172.21.15.16:0/2732885598 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f28d40cb8c0 0x7f28d406fe60 secure :-1 s=READY pgs=1676 cs=0 l=1 rev1=1 crypto rx=0x7f28d40d31b0 tx=0x7f28cc01c930 comp rx=0 tx=0).stop 2024-03-31T15:14:19.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.974+0000 7f28d9a00640 1 -- 172.21.15.16:0/2732885598 shutdown_connections 2024-03-31T15:14:19.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.974+0000 7f28d9a00640 1 --2- 172.21.15.16:0/2732885598 >> v2:172.21.15.89:3300/0 conn(0x7f28d40cb440 0x7f28d406f920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:19.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.974+0000 7f28d9a00640 1 --2- 172.21.15.16:0/2732885598 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f28d40cb8c0 0x7f28d406fe60 unknown :-1 s=CLOSED pgs=1676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:19.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.974+0000 7f28d9a00640 1 --2- 172.21.15.16:0/2732885598 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f28d406dbc0 0x7f28d415f650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:19.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.974+0000 7f28d9a00640 1 -- 172.21.15.16:0/2732885598 >> 172.21.15.16:0/2732885598 conn(0x7f28d4067330 msgr2=0x7f28d4067740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:14:19.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.974+0000 7f28d9a00640 1 -- 172.21.15.16:0/2732885598 shutdown_connections 2024-03-31T15:14:19.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.974+0000 7f28d9a00640 1 -- 172.21.15.16:0/2732885598 wait complete. 2024-03-31T15:14:19.968 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.974+0000 7f28d9a00640 1 Processor -- start 2024-03-31T15:14:19.969 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.974+0000 7f28d9a00640 1 -- start start 2024-03-31T15:14:19.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.978+0000 7f28d9a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f28d406dbc0 0x7f28d41424c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:19.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.978+0000 7f28d9a00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f28d40cb440 0x7f28d4141c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:19.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.978+0000 7f28d9a00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f28d40cb8c0 0x7f28d415b240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:19.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.978+0000 7f28d9a00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f28d406cad0 con 0x7f28d406dbc0 2024-03-31T15:14:19.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.978+0000 7f28d9a00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f28d406c950 con 0x7f28d40cb440 2024-03-31T15:14:19.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.978+0000 7f28d9a00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f28d4162990 con 0x7f28d40cb8c0 2024-03-31T15:14:19.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.978+0000 7f28d2a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f28d406dbc0 0x7f28d41424c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:19.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.978+0000 7f28d2a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f28d406dbc0 0x7f28d41424c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42876/0 (socket says 172.21.15.16:42876) 2024-03-31T15:14:19.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.978+0000 7f28d2a00640 1 -- 172.21.15.16:0/3290245798 learned_addr learned my addr 172.21.15.16:0/3290245798 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:14:19.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.978+0000 7f28cbe00640 1 --2- 172.21.15.16:0/3290245798 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f28d40cb440 0x7f28d4141c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:19.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.978+0000 7f28d3400640 1 --2- 172.21.15.16:0/3290245798 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f28d40cb8c0 0x7f28d415b240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:19.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.978+0000 7f28d2a00640 1 -- 172.21.15.16:0/3290245798 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f28d40cb8c0 msgr2=0x7f28d415b240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:19.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.978+0000 7f28d2a00640 1 --2- 172.21.15.16:0/3290245798 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f28d40cb8c0 0x7f28d415b240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:19.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.978+0000 7f28d2a00640 1 -- 172.21.15.16:0/3290245798 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f28d40cb440 msgr2=0x7f28d4141c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:19.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.978+0000 7f28d2a00640 1 --2- 172.21.15.16:0/3290245798 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f28d40cb440 0x7f28d4141c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:19.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.978+0000 7f28d2a00640 1 -- 172.21.15.16:0/3290245798 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28d4142dc0 con 0x7f28d406dbc0 2024-03-31T15:14:19.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.978+0000 7f28d3400640 1 --2- 172.21.15.16:0/3290245798 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f28d40cb8c0 0x7f28d415b240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:14:19.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.978+0000 7f28d2a00640 1 --2- 172.21.15.16:0/3290245798 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f28d406dbc0 0x7f28d41424c0 secure :-1 s=READY pgs=1677 cs=0 l=1 rev1=1 crypto rx=0x7f28c400e8b0 tx=0x7f28c400ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:19.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.978+0000 7f28caa00640 1 -- 172.21.15.16:0/3290245798 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28c4017070 con 0x7f28d406dbc0 2024-03-31T15:14:19.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.978+0000 7f28caa00640 1 -- 172.21.15.16:0/3290245798 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f28c400c940 con 0x7f28d406dbc0 2024-03-31T15:14:19.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.978+0000 7f28caa00640 1 -- 172.21.15.16:0/3290245798 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28c400cd10 con 0x7f28d406dbc0 2024-03-31T15:14:19.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.978+0000 7f28d9a00640 1 -- 172.21.15.16:0/3290245798 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f28d41430a0 con 0x7f28d406dbc0 2024-03-31T15:14:19.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.978+0000 7f28d9a00640 1 -- 172.21.15.16:0/3290245798 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f28d4143660 con 0x7f28d406dbc0 2024-03-31T15:14:19.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.982+0000 7f28d9a00640 1 -- 172.21.15.16:0/3290245798 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f28d4069950 con 0x7f28d406dbc0 2024-03-31T15:14:19.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.982+0000 7f28caa00640 1 -- 172.21.15.16:0/3290245798 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f28c401a070 con 0x7f28d406dbc0 2024-03-31T15:14:19.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.982+0000 7f28caa00640 1 --2- 172.21.15.16:0/3290245798 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f28b0068380 0x7f28b006a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:19.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.982+0000 7f28caa00640 1 -- 172.21.15.16:0/3290245798 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 5767+0+0 (secure 0 0 0) 0x7f28c4086e40 con 0x7f28d406dbc0 2024-03-31T15:14:19.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.982+0000 7f28cbe00640 1 --2- 172.21.15.16:0/3290245798 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f28b0068380 0x7f28b006a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:19.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.982+0000 7f28cbe00640 1 --2- 172.21.15.16:0/3290245798 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f28b0068380 0x7f28b006a840 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f28d415be80 tx=0x7f28bc008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:19.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:19.986+0000 7f28caa00640 1 -- 172.21.15.16:0/3290245798 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f28c4057320 con 0x7f28d406dbc0 2024-03-31T15:14:20.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:20.186+0000 7f28d9a00640 1 -- 172.21.15.16:0/3290245798 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f28d406e460 con 0x7f28d406dbc0 2024-03-31T15:14:20.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:20.186+0000 7f28caa00640 1 -- 172.21.15.16:0/3290245798 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v99) ==== 53+0+73 (secure 0 0 0) 0x7f28c4056c20 con 0x7f28d406dbc0 2024-03-31T15:14:20.179 INFO:teuthology.orchestra.run.smithi016.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-31T15:14:20.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:20.190+0000 7f28d9a00640 1 -- 172.21.15.16:0/3290245798 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f28b0068380 msgr2=0x7f28b006a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:20.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:20.190+0000 7f28d9a00640 1 --2- 172.21.15.16:0/3290245798 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f28b0068380 0x7f28b006a840 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f28d415be80 tx=0x7f28bc008040 comp rx=0 tx=0).stop 2024-03-31T15:14:20.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:20.190+0000 7f28d9a00640 1 -- 172.21.15.16:0/3290245798 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f28d406dbc0 msgr2=0x7f28d41424c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:20.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:20.190+0000 7f28d9a00640 1 --2- 172.21.15.16:0/3290245798 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f28d406dbc0 0x7f28d41424c0 secure :-1 s=READY pgs=1677 cs=0 l=1 rev1=1 crypto rx=0x7f28c400e8b0 tx=0x7f28c400ed70 comp rx=0 tx=0).stop 2024-03-31T15:14:20.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:20.190+0000 7f28d9a00640 1 -- 172.21.15.16:0/3290245798 shutdown_connections 2024-03-31T15:14:20.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:20.190+0000 7f28d9a00640 1 --2- 172.21.15.16:0/3290245798 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f28b0068380 0x7f28b006a840 unknown :-1 s=CLOSED pgs=713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:20.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:20.190+0000 7f28d9a00640 1 --2- 172.21.15.16:0/3290245798 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f28d40cb8c0 0x7f28d415b240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:20.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:20.190+0000 7f28d9a00640 1 --2- 172.21.15.16:0/3290245798 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f28d40cb440 0x7f28d4141c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:20.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:20.190+0000 7f28d9a00640 1 --2- 172.21.15.16:0/3290245798 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f28d406dbc0 0x7f28d41424c0 unknown :-1 s=CLOSED pgs=1677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:20.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:20.190+0000 7f28d9a00640 1 -- 172.21.15.16:0/3290245798 >> 172.21.15.16:0/3290245798 conn(0x7f28d4067330 msgr2=0x7f28d415d9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:14:20.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:20.190+0000 7f28d9a00640 1 -- 172.21.15.16:0/3290245798 shutdown_connections 2024-03-31T15:14:20.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:20.190+0000 7f28d9a00640 1 -- 172.21.15.16:0/3290245798 wait complete. 2024-03-31T15:14:20.201 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-31T15:14:20.201 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-31T15:14:20.201 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi016.front.sepia.ceph.com 2024-03-31T15:14:20.201 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-31T15:14:20.201 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-31T15:14:20.201 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-31T15:14:20.201 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-31T15:14:20.201 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-31T15:14:20.201 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-31T15:14:20.202 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:14:20.202 DEBUG:teuthology.orchestra.run.smithi016:> ip addr 2024-03-31T15:14:20.211 INFO:teuthology.orchestra.run.smithi016.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-31T15:14:20.211 INFO:teuthology.orchestra.run.smithi016.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-31T15:14:20.211 INFO:teuthology.orchestra.run.smithi016.stdout: inet 127.0.0.1/8 scope host lo 2024-03-31T15:14:20.211 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2024-03-31T15:14:20.212 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 ::1/128 scope host 2024-03-31T15:14:20.212 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2024-03-31T15:14:20.212 INFO:teuthology.orchestra.run.smithi016.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-03-31T15:14:20.212 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:6b:fd:e0 brd ff:ff:ff:ff:ff:ff 2024-03-31T15:14:20.212 INFO:teuthology.orchestra.run.smithi016.stdout: altname enp7s0f0 2024-03-31T15:14:20.212 INFO:teuthology.orchestra.run.smithi016.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-03-31T15:14:20.212 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:6b:fd:e1 brd ff:ff:ff:ff:ff:ff 2024-03-31T15:14:20.212 INFO:teuthology.orchestra.run.smithi016.stdout: altname enp7s0f1 2024-03-31T15:14:20.212 INFO:teuthology.orchestra.run.smithi016.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-31T15:14:20.212 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:bd:15:64 brd ff:ff:ff:ff:ff:ff 2024-03-31T15:14:20.212 INFO:teuthology.orchestra.run.smithi016.stdout: altname enp1s0f0 2024-03-31T15:14:20.212 INFO:teuthology.orchestra.run.smithi016.stdout: inet 172.21.15.16/20 brd 172.21.15.255 scope global dynamic ens1f0 2024-03-31T15:14:20.212 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft 41821sec preferred_lft 41821sec 2024-03-31T15:14:20.212 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 fe80::ec4:7aff:febd:1564/64 scope link 2024-03-31T15:14:20.212 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2024-03-31T15:14:20.212 INFO:teuthology.orchestra.run.smithi016.stdout:5: ens1f1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-03-31T15:14:20.213 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:bd:15:65 brd ff:ff:ff:ff:ff:ff 2024-03-31T15:14:20.213 INFO:teuthology.orchestra.run.smithi016.stdout: altname enp1s0f1 2024-03-31T15:14:20.213 INFO:teuthology.orchestra.run.smithi016.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2024-03-31T15:14:20.213 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 02:42:d4:1f:7d:73 brd ff:ff:ff:ff:ff:ff 2024-03-31T15:14:20.213 INFO:teuthology.orchestra.run.smithi016.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2024-03-31T15:14:20.213 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2024-03-31T15:14:20.213 INFO:teuthology.orchestra.run.smithi016.stdout:7: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-31T15:14:20.213 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 5e:1f:84:6d:79:ae brd ff:ff:ff:ff:ff:ff 2024-03-31T15:14:20.213 INFO:teuthology.orchestra.run.smithi016.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-31T15:14:20.213 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2024-03-31T15:14:20.213 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 fe80::5c1f:84ff:fe6d:79ae/64 scope link 2024-03-31T15:14:20.213 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2024-03-31T15:14:20.213 INFO:teuthology.orchestra.run.smithi016.stdout:8: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-31T15:14:20.213 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0e:5c:f6:1f:82:2f brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-31T15:14:20.213 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 fe80::c5c:f6ff:fe1f:822f/64 scope link 2024-03-31T15:14:20.213 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2024-03-31T15:14:20.214 INFO:teuthology.orchestra.run.smithi016.stdout:9: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-31T15:14:20.214 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether c6:a6:3d:69:d0:db brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-31T15:14:20.214 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 fe80::c4a6:3dff:fe69:d0db/64 scope link 2024-03-31T15:14:20.214 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2024-03-31T15:14:20.214 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:14:20.214 DEBUG:teuthology.orchestra.run.smithi016:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-31T15:14:20.274 INFO:teuthology.orchestra.run.smithi016.stdout:1 2024-03-31T15:14:20.277 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:14:20.277 DEBUG:teuthology.orchestra.run.smithi016:> route 2024-03-31T15:14:20.330 INFO:teuthology.orchestra.run.smithi016.stdout:Kernel IP routing table 2024-03-31T15:14:20.330 INFO:teuthology.orchestra.run.smithi016.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-31T15:14:20.330 INFO:teuthology.orchestra.run.smithi016.stdout:default _gateway 0.0.0.0 UG 0 0 0 ens1f0 2024-03-31T15:14:20.330 INFO:teuthology.orchestra.run.smithi016.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2024-03-31T15:14:20.330 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 0 0 0 ens1f0 2024-03-31T15:14:20.330 INFO:teuthology.orchestra.run.smithi016.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-31T15:14:20.330 DEBUG:teuthology.orchestra.run.smithi016:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-31T15:14:20.330 DEBUG:teuthology.orchestra.run.smithi016:> set -e 2024-03-31T15:14:20.331 DEBUG:teuthology.orchestra.run.smithi016:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-03-31T15:14:20.331 DEBUG:teuthology.orchestra.run.smithi016:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-03-31T15:14:20.331 DEBUG:teuthology.orchestra.run.smithi016:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-03-31T15:14:20.331 DEBUG:teuthology.orchestra.run.smithi016:> ') 2024-03-31T15:14:20.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:20 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-31T15:14:20.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:20 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-31T15:14:20.467 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:14:20.468 DEBUG:teuthology.orchestra.run.smithi016:> ip netns list 2024-03-31T15:14:20.477 INFO:teuthology.orchestra.run.smithi016.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-31T15:14:20.477 INFO:teuthology.orchestra.run.smithi016.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-31T15:14:20.477 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-31T15:14:20.477 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:14:20.477 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip link set brx.0 up 2024-03-31T15:14:20.540 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-03-31T15:14:20.540 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-31T15:14:20.540 DEBUG:teuthology.orchestra.run.smithi016:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-31T15:14:20.588 INFO:teuthology.orchestra.run.smithi016.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-31T15:14:20.589 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-31T15:14:20.589 DEBUG:teuthology.orchestra.run.smithi016:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-31T15:14:20.641 DEBUG:teuthology.orchestra.run.smithi016:> sudo modprobe fuse 2024-03-31T15:14:20.705 DEBUG:teuthology.orchestra.run.smithi016:> cat /proc/self/mounts | awk '{print $2}' 2024-03-31T15:14:20.758 INFO:teuthology.orchestra.run.smithi016.stdout:/sys 2024-03-31T15:14:20.758 INFO:teuthology.orchestra.run.smithi016.stdout:/proc 2024-03-31T15:14:20.758 INFO:teuthology.orchestra.run.smithi016.stdout:/dev 2024-03-31T15:14:20.758 INFO:teuthology.orchestra.run.smithi016.stdout:/dev/pts 2024-03-31T15:14:20.758 INFO:teuthology.orchestra.run.smithi016.stdout:/run 2024-03-31T15:14:20.758 INFO:teuthology.orchestra.run.smithi016.stdout:/ 2024-03-31T15:14:20.758 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/kernel/security 2024-03-31T15:14:20.758 INFO:teuthology.orchestra.run.smithi016.stdout:/dev/shm 2024-03-31T15:14:20.758 INFO:teuthology.orchestra.run.smithi016.stdout:/run/lock 2024-03-31T15:14:20.759 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/cgroup 2024-03-31T15:14:20.759 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/pstore 2024-03-31T15:14:20.759 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/bpf 2024-03-31T15:14:20.759 INFO:teuthology.orchestra.run.smithi016.stdout:/proc/sys/fs/binfmt_misc 2024-03-31T15:14:20.759 INFO:teuthology.orchestra.run.smithi016.stdout:/dev/hugepages 2024-03-31T15:14:20.759 INFO:teuthology.orchestra.run.smithi016.stdout:/dev/mqueue 2024-03-31T15:14:20.759 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/kernel/debug 2024-03-31T15:14:20.759 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/kernel/tracing 2024-03-31T15:14:20.759 INFO:teuthology.orchestra.run.smithi016.stdout:/proc/fs/nfsd 2024-03-31T15:14:20.759 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/fuse/connections 2024-03-31T15:14:20.759 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/kernel/config 2024-03-31T15:14:20.759 INFO:teuthology.orchestra.run.smithi016.stdout:/run/credentials/systemd-sysusers.service 2024-03-31T15:14:20.759 INFO:teuthology.orchestra.run.smithi016.stdout:/run/rpc_pipefs 2024-03-31T15:14:20.759 INFO:teuthology.orchestra.run.smithi016.stdout:/run/user/1000 2024-03-31T15:14:20.760 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/ceph 2024-03-31T15:14:20.760 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/ceph/osd/ceph-0 2024-03-31T15:14:20.760 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/ceph/osd/ceph-1 2024-03-31T15:14:20.760 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/ceph/osd/ceph-2 2024-03-31T15:14:20.760 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/ceph/osd/ceph-3 2024-03-31T15:14:20.760 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns 2024-03-31T15:14:20.760 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-31T15:14:20.760 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-31T15:14:20.760 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-31T15:14:20.760 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-31T15:14:20.761 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:14:20.761 DEBUG:teuthology.orchestra.run.smithi016:> ls /sys/fs/fuse/connections 2024-03-31T15:14:20.812 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-31T15:14:20.812 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-03-31T15:14:20.855 DEBUG:teuthology.orchestra.run.smithi016:> sudo modprobe fuse 2024-03-31T15:14:20.876 DEBUG:teuthology.orchestra.run.smithi016:> cat /proc/self/mounts | awk '{print $2}' 2024-03-31T15:14:20.931 INFO:teuthology.orchestra.run.smithi016.stdout:/sys 2024-03-31T15:14:20.931 INFO:teuthology.orchestra.run.smithi016.stdout:/proc 2024-03-31T15:14:20.932 INFO:teuthology.orchestra.run.smithi016.stdout:/dev 2024-03-31T15:14:20.932 INFO:teuthology.orchestra.run.smithi016.stdout:/dev/pts 2024-03-31T15:14:20.932 INFO:teuthology.orchestra.run.smithi016.stdout:/run 2024-03-31T15:14:20.932 INFO:teuthology.orchestra.run.smithi016.stdout:/ 2024-03-31T15:14:20.932 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/kernel/security 2024-03-31T15:14:20.932 INFO:teuthology.orchestra.run.smithi016.stdout:/dev/shm 2024-03-31T15:14:20.932 INFO:teuthology.orchestra.run.smithi016.stdout:/run/lock 2024-03-31T15:14:20.932 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/cgroup 2024-03-31T15:14:20.932 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/pstore 2024-03-31T15:14:20.932 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/bpf 2024-03-31T15:14:20.932 INFO:teuthology.orchestra.run.smithi016.stdout:/proc/sys/fs/binfmt_misc 2024-03-31T15:14:20.932 INFO:teuthology.orchestra.run.smithi016.stdout:/dev/hugepages 2024-03-31T15:14:20.932 INFO:teuthology.orchestra.run.smithi016.stdout:/dev/mqueue 2024-03-31T15:14:20.932 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/kernel/debug 2024-03-31T15:14:20.932 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/kernel/tracing 2024-03-31T15:14:20.932 INFO:teuthology.orchestra.run.smithi016.stdout:/proc/fs/nfsd 2024-03-31T15:14:20.933 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/fuse/connections 2024-03-31T15:14:20.933 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/kernel/config 2024-03-31T15:14:20.933 INFO:teuthology.orchestra.run.smithi016.stdout:/run/credentials/systemd-sysusers.service 2024-03-31T15:14:20.933 INFO:teuthology.orchestra.run.smithi016.stdout:/run/rpc_pipefs 2024-03-31T15:14:20.933 INFO:teuthology.orchestra.run.smithi016.stdout:/run/user/1000 2024-03-31T15:14:20.933 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/ceph 2024-03-31T15:14:20.933 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/ceph/osd/ceph-0 2024-03-31T15:14:20.933 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/ceph/osd/ceph-1 2024-03-31T15:14:20.933 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/ceph/osd/ceph-2 2024-03-31T15:14:20.933 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/ceph/osd/ceph-3 2024-03-31T15:14:20.933 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns 2024-03-31T15:14:20.933 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-31T15:14:20.933 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-31T15:14:20.933 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-31T15:14:20.934 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-31T15:14:20.934 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:14:20.935 DEBUG:teuthology.orchestra.run.smithi016:> ls /sys/fs/fuse/connections 2024-03-31T15:14:20.952 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi016.stderr:2024-03-31T15:14:20.958+0000 7f93d56b8580 -1 init, newargv = 0x55bf67cd8240 newargc=15 2024-03-31T15:14:20.952 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi016.stderr:ceph-fuse[51184]: starting ceph client 2024-03-31T15:14:20.959 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi016.stderr:ceph-fuse[51184]: starting fuse 2024-03-31T15:14:20.961 INFO:teuthology.orchestra.run.smithi016.stdout:43 2024-03-31T15:14:20.962 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [43] 2024-03-31T15:14:20.962 DEBUG:teuthology.orchestra.run.smithi016:> sudo stdin-killer -- python3 -c ' 2024-03-31T15:14:20.962 DEBUG:teuthology.orchestra.run.smithi016:> import glob 2024-03-31T15:14:20.962 DEBUG:teuthology.orchestra.run.smithi016:> import re 2024-03-31T15:14:20.962 DEBUG:teuthology.orchestra.run.smithi016:> import os 2024-03-31T15:14:20.962 DEBUG:teuthology.orchestra.run.smithi016:> import subprocess 2024-03-31T15:14:20.962 DEBUG:teuthology.orchestra.run.smithi016:> 2024-03-31T15:14:20.962 DEBUG:teuthology.orchestra.run.smithi016:> def _find_admin_socket(client_name): 2024-03-31T15:14:20.962 DEBUG:teuthology.orchestra.run.smithi016:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-03-31T15:14:20.962 DEBUG:teuthology.orchestra.run.smithi016:> files = glob.glob(asok_path) 2024-03-31T15:14:20.962 DEBUG:teuthology.orchestra.run.smithi016:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-03-31T15:14:20.962 DEBUG:teuthology.orchestra.run.smithi016:> 2024-03-31T15:14:20.963 DEBUG:teuthology.orchestra.run.smithi016:> # Given a non-glob path, it better be there 2024-03-31T15:14:20.963 DEBUG:teuthology.orchestra.run.smithi016:> if "*" not in asok_path: 2024-03-31T15:14:20.963 DEBUG:teuthology.orchestra.run.smithi016:> assert(len(files) == 1) 2024-03-31T15:14:20.963 DEBUG:teuthology.orchestra.run.smithi016:> return files[0] 2024-03-31T15:14:20.963 DEBUG:teuthology.orchestra.run.smithi016:> 2024-03-31T15:14:20.963 DEBUG:teuthology.orchestra.run.smithi016:> for f in files: 2024-03-31T15:14:20.963 DEBUG:teuthology.orchestra.run.smithi016:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-31T15:14:20.963 DEBUG:teuthology.orchestra.run.smithi016:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-31T15:14:20.963 DEBUG:teuthology.orchestra.run.smithi016:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-31T15:14:20.963 DEBUG:teuthology.orchestra.run.smithi016:> contents = proc_f.read() 2024-03-31T15:14:20.963 DEBUG:teuthology.orchestra.run.smithi016:> if mountpoint in contents: 2024-03-31T15:14:20.963 DEBUG:teuthology.orchestra.run.smithi016:> return f 2024-03-31T15:14:20.963 DEBUG:teuthology.orchestra.run.smithi016:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-31T15:14:20.963 DEBUG:teuthology.orchestra.run.smithi016:> 2024-03-31T15:14:20.963 DEBUG:teuthology.orchestra.run.smithi016:> print(_find_admin_socket("client.0")) 2024-03-31T15:14:20.964 DEBUG:teuthology.orchestra.run.smithi016:> ' 2024-03-31T15:14:21.087 INFO:teuthology.orchestra.run.smithi016.stdout:/var/run/ceph/ceph-client.0.51184.asok 2024-03-31T15:14:21.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:21 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-31T15:14:21.098 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.51184.asok 2024-03-31T15:14:21.098 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:14:21.099 DEBUG:teuthology.orchestra.run.smithi016:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.51184.asok status 2024-03-31T15:14:21.273 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-03-31T15:14:21.273 INFO:teuthology.orchestra.run.smithi016.stdout: "metadata": { 2024-03-31T15:14:21.273 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph_sha1": "d2271579d5e735cfd6d436b8349c004f06527cb0", 2024-03-31T15:14:21.274 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph_version": "ceph version 19.0.0-2574-gd2271579 (d2271579d5e735cfd6d436b8349c004f06527cb0) squid (dev)", 2024-03-31T15:14:21.274 INFO:teuthology.orchestra.run.smithi016.stdout: "entity_id": "0", 2024-03-31T15:14:21.274 INFO:teuthology.orchestra.run.smithi016.stdout: "hostname": "smithi016", 2024-03-31T15:14:21.274 INFO:teuthology.orchestra.run.smithi016.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-03-31T15:14:21.274 INFO:teuthology.orchestra.run.smithi016.stdout: "pid": "51184", 2024-03-31T15:14:21.274 INFO:teuthology.orchestra.run.smithi016.stdout: "root": "/" 2024-03-31T15:14:21.274 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-31T15:14:21.274 INFO:teuthology.orchestra.run.smithi016.stdout: "dentry_count": 0, 2024-03-31T15:14:21.274 INFO:teuthology.orchestra.run.smithi016.stdout: "dentry_pinned_count": 0, 2024-03-31T15:14:21.274 INFO:teuthology.orchestra.run.smithi016.stdout: "id": 9387, 2024-03-31T15:14:21.274 INFO:teuthology.orchestra.run.smithi016.stdout: "inst": { 2024-03-31T15:14:21.274 INFO:teuthology.orchestra.run.smithi016.stdout: "name": { 2024-03-31T15:14:21.274 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "client", 2024-03-31T15:14:21.275 INFO:teuthology.orchestra.run.smithi016.stdout: "num": 9387 2024-03-31T15:14:21.275 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-31T15:14:21.275 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": { 2024-03-31T15:14:21.275 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-03-31T15:14:21.275 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "192.168.0.1:0", 2024-03-31T15:14:21.275 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 1856401852 2024-03-31T15:14:21.275 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-31T15:14:21.275 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-31T15:14:21.275 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": { 2024-03-31T15:14:21.275 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-03-31T15:14:21.275 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "192.168.0.1:0", 2024-03-31T15:14:21.275 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 1856401852 2024-03-31T15:14:21.275 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-31T15:14:21.275 INFO:teuthology.orchestra.run.smithi016.stdout: "inst_str": "client.9387 192.168.0.1:0/1856401852", 2024-03-31T15:14:21.275 INFO:teuthology.orchestra.run.smithi016.stdout: "addr_str": "192.168.0.1:0/1856401852", 2024-03-31T15:14:21.276 INFO:teuthology.orchestra.run.smithi016.stdout: "inode_count": 1, 2024-03-31T15:14:21.276 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_epoch": 99, 2024-03-31T15:14:21.276 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_epoch": 111, 2024-03-31T15:14:21.276 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_epoch_barrier": 0, 2024-03-31T15:14:21.276 INFO:teuthology.orchestra.run.smithi016.stdout: "blocklisted": false, 2024-03-31T15:14:21.276 INFO:teuthology.orchestra.run.smithi016.stdout: "fs_name": "cephfs" 2024-03-31T15:14:21.276 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-03-31T15:14:21.289 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:14:21.289 DEBUG:teuthology.orchestra.run.smithi016:> stat --file-system '--printf=%T 2024-03-31T15:14:21.289 DEBUG:teuthology.orchestra.run.smithi016:> ' -- /home/ubuntu/cephtest/mnt.0 2024-03-31T15:14:21.299 INFO:teuthology.orchestra.run.smithi016.stdout:fuseblk 2024-03-31T15:14:21.299 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-03-31T15:14:21.299 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:14:21.299 DEBUG:teuthology.orchestra.run.smithi016:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-31T15:14:21.366 DEBUG:teuthology.orchestra.run.smithi016:> stdin-killer -- python3 -c ' 2024-03-31T15:14:21.366 DEBUG:teuthology.orchestra.run.smithi016:> import os 2024-03-31T15:14:21.366 DEBUG:teuthology.orchestra.run.smithi016:> mount_point = "/home/ubuntu/cephtest/mnt.0" 2024-03-31T15:14:21.366 DEBUG:teuthology.orchestra.run.smithi016:> parent = os.path.join(mount_point, "parent") 2024-03-31T15:14:21.366 DEBUG:teuthology.orchestra.run.smithi016:> os.mkdir(parent) 2024-03-31T15:14:21.366 DEBUG:teuthology.orchestra.run.smithi016:> flushed = os.path.join(parent, "flushed") 2024-03-31T15:14:21.366 DEBUG:teuthology.orchestra.run.smithi016:> os.mkdir(flushed) 2024-03-31T15:14:21.367 DEBUG:teuthology.orchestra.run.smithi016:> for f in ["alpha", "bravo", "charlie"]: 2024-03-31T15:14:21.367 DEBUG:teuthology.orchestra.run.smithi016:> open(os.path.join(flushed, f), '"'"'w'"'"').write(f) 2024-03-31T15:14:21.367 DEBUG:teuthology.orchestra.run.smithi016:> ' 2024-03-31T15:14:21.494 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:21 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-31T15:14:21.500 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:14:21.500 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec find ./) 2024-03-31T15:14:21.562 INFO:teuthology.orchestra.run.smithi016.stdout:./ 2024-03-31T15:14:21.562 INFO:teuthology.orchestra.run.smithi016.stdout:./parent 2024-03-31T15:14:21.562 INFO:teuthology.orchestra.run.smithi016.stdout:./parent/flushed 2024-03-31T15:14:21.562 INFO:teuthology.orchestra.run.smithi016.stdout:./parent/flushed/charlie 2024-03-31T15:14:21.562 INFO:teuthology.orchestra.run.smithi016.stdout:./parent/flushed/alpha 2024-03-31T15:14:21.562 INFO:teuthology.orchestra.run.smithi016.stdout:./parent/flushed/bravo 2024-03-31T15:14:21.563 DEBUG:teuthology.orchestra.run.smithi016:> stdin-killer -- python3 -c ' 2024-03-31T15:14:21.563 DEBUG:teuthology.orchestra.run.smithi016:> import os 2024-03-31T15:14:21.563 DEBUG:teuthology.orchestra.run.smithi016:> import stat 2024-03-31T15:14:21.563 DEBUG:teuthology.orchestra.run.smithi016:> 2024-03-31T15:14:21.563 DEBUG:teuthology.orchestra.run.smithi016:> print(os.stat("/home/ubuntu/cephtest/mnt.0/./").st_ino) 2024-03-31T15:14:21.564 DEBUG:teuthology.orchestra.run.smithi016:> ' 2024-03-31T15:14:21.670 INFO:teuthology.orchestra.run.smithi016.stdout:1 2024-03-31T15:14:21.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:21 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-31T15:14:21.679 DEBUG:teuthology.orchestra.run.smithi016:> stdin-killer -- python3 -c ' 2024-03-31T15:14:21.679 DEBUG:teuthology.orchestra.run.smithi016:> import os 2024-03-31T15:14:21.679 DEBUG:teuthology.orchestra.run.smithi016:> import stat 2024-03-31T15:14:21.679 DEBUG:teuthology.orchestra.run.smithi016:> 2024-03-31T15:14:21.679 DEBUG:teuthology.orchestra.run.smithi016:> print(os.stat("/home/ubuntu/cephtest/mnt.0/./parent").st_ino) 2024-03-31T15:14:21.679 DEBUG:teuthology.orchestra.run.smithi016:> ' 2024-03-31T15:14:21.785 INFO:teuthology.orchestra.run.smithi016.stdout:1099511627776 2024-03-31T15:14:21.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:21 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-31T15:14:21.794 DEBUG:teuthology.orchestra.run.smithi016:> stdin-killer -- python3 -c ' 2024-03-31T15:14:21.794 DEBUG:teuthology.orchestra.run.smithi016:> import os 2024-03-31T15:14:21.794 DEBUG:teuthology.orchestra.run.smithi016:> import stat 2024-03-31T15:14:21.794 DEBUG:teuthology.orchestra.run.smithi016:> 2024-03-31T15:14:21.794 DEBUG:teuthology.orchestra.run.smithi016:> print(os.stat("/home/ubuntu/cephtest/mnt.0/./parent/flushed").st_ino) 2024-03-31T15:14:21.794 DEBUG:teuthology.orchestra.run.smithi016:> ' 2024-03-31T15:14:21.903 INFO:teuthology.orchestra.run.smithi016.stdout:1099511627777 2024-03-31T15:14:21.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:21 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-31T15:14:21.912 DEBUG:teuthology.orchestra.run.smithi016:> stdin-killer -- python3 -c ' 2024-03-31T15:14:21.912 DEBUG:teuthology.orchestra.run.smithi016:> import os 2024-03-31T15:14:21.912 DEBUG:teuthology.orchestra.run.smithi016:> import stat 2024-03-31T15:14:21.912 DEBUG:teuthology.orchestra.run.smithi016:> 2024-03-31T15:14:21.912 DEBUG:teuthology.orchestra.run.smithi016:> print(os.stat("/home/ubuntu/cephtest/mnt.0/./parent/flushed/charlie").st_ino) 2024-03-31T15:14:21.912 DEBUG:teuthology.orchestra.run.smithi016:> ' 2024-03-31T15:14:22.019 INFO:teuthology.orchestra.run.smithi016.stdout:1099511628280 2024-03-31T15:14:22.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:22 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-31T15:14:22.027 DEBUG:teuthology.orchestra.run.smithi016:> stdin-killer -- python3 -c ' 2024-03-31T15:14:22.028 DEBUG:teuthology.orchestra.run.smithi016:> import os 2024-03-31T15:14:22.028 DEBUG:teuthology.orchestra.run.smithi016:> import stat 2024-03-31T15:14:22.028 DEBUG:teuthology.orchestra.run.smithi016:> 2024-03-31T15:14:22.028 DEBUG:teuthology.orchestra.run.smithi016:> print(os.stat("/home/ubuntu/cephtest/mnt.0/./parent/flushed/alpha").st_ino) 2024-03-31T15:14:22.028 DEBUG:teuthology.orchestra.run.smithi016:> ' 2024-03-31T15:14:22.134 INFO:teuthology.orchestra.run.smithi016.stdout:1099511627778 2024-03-31T15:14:22.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:22 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-31T15:14:22.144 DEBUG:teuthology.orchestra.run.smithi016:> stdin-killer -- python3 -c ' 2024-03-31T15:14:22.144 DEBUG:teuthology.orchestra.run.smithi016:> import os 2024-03-31T15:14:22.144 DEBUG:teuthology.orchestra.run.smithi016:> import stat 2024-03-31T15:14:22.144 DEBUG:teuthology.orchestra.run.smithi016:> 2024-03-31T15:14:22.144 DEBUG:teuthology.orchestra.run.smithi016:> print(os.stat("/home/ubuntu/cephtest/mnt.0/./parent/flushed/bravo").st_ino) 2024-03-31T15:14:22.144 DEBUG:teuthology.orchestra.run.smithi016:> ' 2024-03-31T15:14:22.260 INFO:teuthology.orchestra.run.smithi016.stdout:1099511628279 2024-03-31T15:14:22.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:22 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-31T15:14:22.272 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-03-31T15:14:22.272 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-31T15:14:22.322 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-31T15:14:22.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:22.514+0000 7f8a5ec00640 1 Processor -- start 2024-03-31T15:14:22.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:22.514+0000 7f8a5ec00640 1 -- start start 2024-03-31T15:14:22.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:22.518+0000 7f8a5ec00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f8a5806dbc0 0x7f8a5815f650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:22.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:22.518+0000 7f8a5ec00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f8a58070030 0x7f8a58162180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:22.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:22.518+0000 7f8a5ec00640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8a580cbe90 con 0x7f8a580cb440 2024-03-31T15:14:22.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:22.518+0000 7f8a5ec00640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f8a5815fb90 con 0x7f8a58070030 2024-03-31T15:14:22.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:22.518+0000 7f8a5ec00640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f8a5815f1c0 con 0x7f8a5806dbc0 2024-03-31T15:14:22.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:22.518+0000 7f8a5cc00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f8a58070030 0x7f8a58162180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:22.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:22.518+0000 7f8a5cc00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f8a58070030 0x7f8a58162180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42880/0 (socket says 172.21.15.16:42880) 2024-03-31T15:14:22.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:22.518+0000 7f8a5cc00640 1 -- 172.21.15.16:0/3926255278 learned_addr learned my addr 172.21.15.16:0/3926255278 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:14:22.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:22.518+0000 7f8a57e00640 1 --2- 172.21.15.16:0/3926255278 >> v2:172.21.15.89:3300/0 conn(0x7f8a5806dbc0 0x7f8a5815f650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:22.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:22.518+0000 7f8a5cc00640 1 -- 172.21.15.16:0/3926255278 >> v2:172.21.15.89:3300/0 conn(0x7f8a5806dbc0 msgr2=0x7f8a5815f650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:14:22.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:22.518+0000 7f8a5cc00640 1 --2- 172.21.15.16:0/3926255278 >> v2:172.21.15.89:3300/0 conn(0x7f8a5806dbc0 0x7f8a5815f650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:22.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:22.518+0000 7f8a5cc00640 1 -- 172.21.15.16:0/3926255278 >> v1:172.21.15.89:6789/0 conn(0x7f8a580cb440 legacy=0x7f8a5806f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:14:22.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:22.518+0000 7f8a5cc00640 1 -- 172.21.15.16:0/3926255278 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a580cc170 con 0x7f8a58070030 2024-03-31T15:14:22.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:22.522+0000 7f8a5cc00640 1 --2- 172.21.15.16:0/3926255278 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8a58070030 0x7f8a58162180 secure :-1 s=READY pgs=1681 cs=0 l=1 rev1=1 crypto rx=0x7f8a580b8380 tx=0x7f8a5001c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3fd50d4b79643f19 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:22.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:22.522+0000 7f8a57400640 1 -- 172.21.15.16:0/3926255278 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a500a57d0 con 0x7f8a58070030 2024-03-31T15:14:22.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:22.522+0000 7f8a57400640 1 -- 172.21.15.16:0/3926255278 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8a500a5970 con 0x7f8a58070030 2024-03-31T15:14:22.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:22.522+0000 7f8a57400640 1 -- 172.21.15.16:0/3926255278 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a500a5d40 con 0x7f8a58070030 2024-03-31T15:14:22.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:22.522+0000 7f8a5ec00640 1 -- 172.21.15.16:0/3926255278 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8a58070030 msgr2=0x7f8a58162180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:22.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:22.522+0000 7f8a5ec00640 1 --2- 172.21.15.16:0/3926255278 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8a58070030 0x7f8a58162180 secure :-1 s=READY pgs=1681 cs=0 l=1 rev1=1 crypto rx=0x7f8a580b8380 tx=0x7f8a5001c8f0 comp rx=0 tx=0).stop 2024-03-31T15:14:22.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:22.522+0000 7f8a5ec00640 1 -- 172.21.15.16:0/3926255278 shutdown_connections 2024-03-31T15:14:22.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:22.522+0000 7f8a5ec00640 1 --2- 172.21.15.16:0/3926255278 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8a58070030 0x7f8a58162180 unknown :-1 s=CLOSED pgs=1681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:22.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:22.522+0000 7f8a5ec00640 1 --2- 172.21.15.16:0/3926255278 >> v2:172.21.15.89:3300/0 conn(0x7f8a5806dbc0 0x7f8a5815f650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:22.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:22.522+0000 7f8a5ec00640 1 -- 172.21.15.16:0/3926255278 >> 172.21.15.16:0/3926255278 conn(0x7f8a58067330 msgr2=0x7f8a58067740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:14:22.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:22.522+0000 7f8a5ec00640 1 -- 172.21.15.16:0/3926255278 shutdown_connections 2024-03-31T15:14:22.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:22.522+0000 7f8a5ec00640 1 -- 172.21.15.16:0/3926255278 wait complete. 2024-03-31T15:14:22.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:22.522+0000 7f8a5ec00640 1 Processor -- start 2024-03-31T15:14:22.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:22.522+0000 7f8a5ec00640 1 -- start start 2024-03-31T15:14:22.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:22.526+0000 7f8a5ec00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f8a5806dbc0 0x7f8a58152200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:22.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:22.526+0000 7f8a5ec00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8a58070030 0x7f8a580b5880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:22.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:22.526+0000 7f8a5ec00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8a580cb440 0x7f8a580d0b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:22.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:22.526+0000 7f8a5ec00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f8a5806cad0 con 0x7f8a58070030 2024-03-31T15:14:22.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:22.526+0000 7f8a5ec00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f8a5806c950 con 0x7f8a580cb440 2024-03-31T15:14:22.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:22.526+0000 7f8a5ec00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f8a5816b520 con 0x7f8a5806dbc0 2024-03-31T15:14:22.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:22.526+0000 7f8a4f400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8a58070030 0x7f8a580b5880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:22.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:22.526+0000 7f8a57e00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f8a5806dbc0 0x7f8a58152200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:22.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:22.526+0000 7f8a57e00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f8a5806dbc0 0x7f8a58152200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3301/0 says I am v2:172.21.15.16:38758/0 (socket says 172.21.15.16:38758) 2024-03-31T15:14:22.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:22.526+0000 7f8a57e00640 1 -- 172.21.15.16:0/185378935 learned_addr learned my addr 172.21.15.16:0/185378935 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:14:22.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:22.526+0000 7f8a4f400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8a58070030 0x7f8a580b5880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42890/0 (socket says 172.21.15.16:42890) 2024-03-31T15:14:22.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:22.526+0000 7f8a5cc00640 1 --2- 172.21.15.16:0/185378935 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8a580cb440 0x7f8a580d0b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:22.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:22.526+0000 7f8a4f400640 1 -- 172.21.15.16:0/185378935 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f8a5806dbc0 msgr2=0x7f8a58152200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:22.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:22.526+0000 7f8a4f400640 1 --2- 172.21.15.16:0/185378935 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f8a5806dbc0 0x7f8a58152200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:22.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:22.526+0000 7f8a4f400640 1 -- 172.21.15.16:0/185378935 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8a580cb440 msgr2=0x7f8a580d0b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:22.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:22.526+0000 7f8a4f400640 1 --2- 172.21.15.16:0/185378935 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8a580cb440 0x7f8a580d0b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:22.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:22.526+0000 7f8a4f400640 1 -- 172.21.15.16:0/185378935 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a5814b940 con 0x7f8a58070030 2024-03-31T15:14:22.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:22.526+0000 7f8a4f400640 1 --2- 172.21.15.16:0/185378935 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8a58070030 0x7f8a580b5880 secure :-1 s=READY pgs=1682 cs=0 l=1 rev1=1 crypto rx=0x7f8a4000c410 tx=0x7f8a4000c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:22.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:22.526+0000 7f8a54c00640 1 -- 172.21.15.16:0/185378935 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a40017070 con 0x7f8a58070030 2024-03-31T15:14:22.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:22.526+0000 7f8a54c00640 1 -- 172.21.15.16:0/185378935 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8a400069e0 con 0x7f8a58070030 2024-03-31T15:14:22.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:22.526+0000 7f8a5ec00640 1 -- 172.21.15.16:0/185378935 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8a5814bc20 con 0x7f8a58070030 2024-03-31T15:14:22.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:22.526+0000 7f8a54c00640 1 -- 172.21.15.16:0/185378935 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a40006e10 con 0x7f8a58070030 2024-03-31T15:14:22.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:22.526+0000 7f8a5ec00640 1 -- 172.21.15.16:0/185378935 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8a5814c1e0 con 0x7f8a58070030 2024-03-31T15:14:22.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:22.526+0000 7f8a5ec00640 1 -- 172.21.15.16:0/185378935 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8a58069950 con 0x7f8a58070030 2024-03-31T15:14:22.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:22.526+0000 7f8a54c00640 1 -- 172.21.15.16:0/185378935 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f8a4001a070 con 0x7f8a58070030 2024-03-31T15:14:22.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:22.530+0000 7f8a54c00640 1 --2- 172.21.15.16:0/185378935 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f8a34068380 0x7f8a3406a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:22.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:22.530+0000 7f8a54c00640 1 -- 172.21.15.16:0/185378935 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 5767+0+0 (secure 0 0 0) 0x7f8a40085d80 con 0x7f8a58070030 2024-03-31T15:14:22.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:22.530+0000 7f8a57e00640 1 --2- 172.21.15.16:0/185378935 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f8a34068380 0x7f8a3406a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:22.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:22.530+0000 7f8a57e00640 1 --2- 172.21.15.16:0/185378935 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f8a34068380 0x7f8a3406a840 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f8a48007a70 tx=0x7f8a48008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:22.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:22.534+0000 7f8a54c00640 1 -- 172.21.15.16:0/185378935 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f8a40055c10 con 0x7f8a58070030 2024-03-31T15:14:22.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:22.818+0000 7f8a5ec00640 1 -- 172.21.15.16:0/185378935 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f8a58158c40 con 0x7f8a58070030 2024-03-31T15:14:22.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:22.818+0000 7f8a54c00640 1 -- 172.21.15.16:0/185378935 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v111) ==== 74+0+16648 (secure 0 0 0) 0x7f8a40059f10 con 0x7f8a58070030 2024-03-31T15:14:22.812 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:14:22.812 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":111,"fsid":"286eb5da-fb47-4550-b290-d9519c13b9a5","created":"2024-03-31T15:03:11.438952+0000","modified":"2024-03-31T15:14:19.306886+0000","last_up_change":"2024-03-31T15:03:22.885535+0000","last_in_change":"2024-03-31T15:03:13.941667+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-31T15:03:22.550787+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-03-31T15:14:06.417304+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"108","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-03-31T15:14:07.622562+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"109","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"92a66450-ef19-40d8-ac3d-d760b23b0bda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6801","nonce":932712244}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6803","nonce":932712244}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6807","nonce":932712244}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6805","nonce":932712244}]},"public_addr":"172.21.15.16:6801/932712244","cluster_addr":"172.21.15.16:6803/932712244","heartbeat_back_addr":"172.21.15.16:6807/932712244","heartbeat_front_addr":"172.21.15.16:6805/932712244","state":["exists","up"]},{"osd":1,"uuid":"da9d1f2b-1968-4959-b843-faa231988e92","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6810","nonce":1398737533}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6811","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6813","nonce":1398737533}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6821","nonce":1398737533}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6815","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6816","nonce":1398737533}]},"public_addr":"172.21.15.16:6810/1398737533","cluster_addr":"172.21.15.16:6813/1398737533","heartbeat_back_addr":"172.21.15.16:6821/1398737533","heartbeat_front_addr":"172.21.15.16:6816/1398737533","state":["exists","up"]},{"osd":2,"uuid":"01a73662-9a21-4b1f-bcbb-42b9636a4638","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6819","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6822","nonce":3018526646}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6825","nonce":3018526646}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6831","nonce":3018526646}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6827","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6829","nonce":3018526646}]},"public_addr":"172.21.15.16:6822/3018526646","cluster_addr":"172.21.15.16:6825/3018526646","heartbeat_back_addr":"172.21.15.16:6831/3018526646","heartbeat_front_addr":"172.21.15.16:6829/3018526646","state":["exists","up"]},{"osd":3,"uuid":"c7771fd3-e18e-4500-ac38-23cf10ebcd75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6809","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6812","nonce":3360352974}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6817","nonce":3360352974}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6828","nonce":3360352974}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6823","nonce":3360352974}]},"public_addr":"172.21.15.16:6812/3360352974","cluster_addr":"172.21.15.16:6817/3360352974","heartbeat_back_addr":"172.21.15.16:6828/3360352974","heartbeat_front_addr":"172.21.15.16:6823/3360352974","state":["exists","up"]},{"osd":4,"uuid":"88260730-1482-4fbd-a40f-cb1124e9df4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6801","nonce":1856844856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6803","nonce":1856844856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6807","nonce":1856844856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6805","nonce":1856844856}]},"public_addr":"172.21.15.89:6801/1856844856","cluster_addr":"172.21.15.89:6803/1856844856","heartbeat_back_addr":"172.21.15.89:6807/1856844856","heartbeat_front_addr":"172.21.15.89:6805/1856844856","state":["exists","up"]},{"osd":5,"uuid":"4cb2c00c-fae0-4dd8-9d27-ec4f0093ccf7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6814","nonce":50395527}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6817","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6820","nonce":50395527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6831","nonce":50395527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6823","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6826","nonce":50395527}]},"public_addr":"172.21.15.89:6814/50395527","cluster_addr":"172.21.15.89:6820/50395527","heartbeat_back_addr":"172.21.15.89:6831/50395527","heartbeat_front_addr":"172.21.15.89:6826/50395527","state":["exists","up"]},{"osd":6,"uuid":"81ce0c19-2554-4cae-acbe-66a05e9e9b9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6809","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6812","nonce":3470471752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6815","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6818","nonce":3470471752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6827","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6829","nonce":3470471752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6821","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6824","nonce":3470471752}]},"public_addr":"172.21.15.89:6812/3470471752","cluster_addr":"172.21.15.89:6818/3470471752","heartbeat_back_addr":"172.21.15.89:6829/3470471752","heartbeat_front_addr":"172.21.15.89:6824/3470471752","state":["exists","up"]},{"osd":7,"uuid":"85e4313b-d37d-442e-8042-153bf20a847d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6811","nonce":1796778630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6813","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6816","nonce":1796778630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6825","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6828","nonce":1796778630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6819","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6822","nonce":1796778630}]},"public_addr":"172.21.15.89:6811/1796778630","cluster_addr":"172.21.15.89:6816/1796778630","heartbeat_back_addr":"172.21.15.89:6828/1796778630","heartbeat_front_addr":"172.21.15.89:6822/1796778630","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.439088+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.529970+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.680512+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:21.044161+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.582111+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.536558+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.409522+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.590229+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"17.1","mappings":[{"from":0,"to":5}]},{"pgid":"17.21","mappings":[{"from":0,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-31T15:14:22.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:22.822+0000 7f8a5ec00640 1 -- 172.21.15.16:0/185378935 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f8a34068380 msgr2=0x7f8a3406a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:22.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:22.822+0000 7f8a5ec00640 1 --2- 172.21.15.16:0/185378935 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f8a34068380 0x7f8a3406a840 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f8a48007a70 tx=0x7f8a48008040 comp rx=0 tx=0).stop 2024-03-31T15:14:22.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:22.822+0000 7f8a5ec00640 1 -- 172.21.15.16:0/185378935 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8a58070030 msgr2=0x7f8a580b5880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:22.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:22.822+0000 7f8a5ec00640 1 --2- 172.21.15.16:0/185378935 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8a58070030 0x7f8a580b5880 secure :-1 s=READY pgs=1682 cs=0 l=1 rev1=1 crypto rx=0x7f8a4000c410 tx=0x7f8a4000c8d0 comp rx=0 tx=0).stop 2024-03-31T15:14:22.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:22.822+0000 7f8a5ec00640 1 -- 172.21.15.16:0/185378935 shutdown_connections 2024-03-31T15:14:22.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:22.822+0000 7f8a5ec00640 1 --2- 172.21.15.16:0/185378935 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f8a34068380 0x7f8a3406a840 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:22.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:22.822+0000 7f8a5ec00640 1 --2- 172.21.15.16:0/185378935 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8a580cb440 0x7f8a580d0b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:22.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:22.822+0000 7f8a5ec00640 1 --2- 172.21.15.16:0/185378935 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8a58070030 0x7f8a580b5880 unknown :-1 s=CLOSED pgs=1682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:22.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:22.822+0000 7f8a5ec00640 1 --2- 172.21.15.16:0/185378935 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f8a5806dbc0 0x7f8a58152200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:22.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:22.822+0000 7f8a5ec00640 1 -- 172.21.15.16:0/185378935 >> 172.21.15.16:0/185378935 conn(0x7f8a58067330 msgr2=0x7f8a58161550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:14:22.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:22.822+0000 7f8a5ec00640 1 -- 172.21.15.16:0/185378935 shutdown_connections 2024-03-31T15:14:22.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:22.822+0000 7f8a5ec00640 1 -- 172.21.15.16:0/185378935 wait complete. 2024-03-31T15:14:22.833 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-31T15:03:22.550787+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 16, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-31T15:14:06.417304+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '108', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 17, 'pool_name': 'cephfs_data', 'create_time': '2024-03-31T15:14:07.622562+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '109', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-31T15:14:22.833 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-31T15:14:22.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:22.986+0000 7ff155800640 1 Processor -- start 2024-03-31T15:14:22.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:22.986+0000 7ff155800640 1 -- start start 2024-03-31T15:14:22.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:22.986+0000 7ff155800640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7ff1501567e0 0x7ff150072c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:22.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:22.986+0000 7ff155800640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff1500731c0 0x7ff15006d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:22.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:22.986+0000 7ff155800640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff15015c7e0 con 0x7ff15006da10 2024-03-31T15:14:22.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:22.986+0000 7ff155800640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7ff15006de80 con 0x7ff1501567e0 2024-03-31T15:14:22.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:22.986+0000 7ff155800640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7ff1500736b0 con 0x7ff1500731c0 2024-03-31T15:14:22.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:22.986+0000 7ff14f400640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7ff15006da10 0x7ff15015fdd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:53452/0 (socket says 172.21.15.16:53452) 2024-03-31T15:14:22.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:22.986+0000 7ff14f400640 1 -- 172.21.15.16:0/2429627643 learned_addr learned my addr 172.21.15.16:0/2429627643 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:14:22.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:22.986+0000 7ff14ea00640 1 --2- 172.21.15.16:0/2429627643 >> v2:172.21.15.89:3300/0 conn(0x7ff1501567e0 0x7ff150072c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:22.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:22.986+0000 7ff146000640 1 --2- 172.21.15.16:0/2429627643 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff1500731c0 0x7ff15006d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:22.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:22.986+0000 7ff14e000640 1 -- 172.21.15.16:0/2429627643 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2711825394 0 0) 0x7ff15015c7e0 con 0x7ff15006da10 2024-03-31T15:14:22.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:22.986+0000 7ff14e000640 1 -- 172.21.15.16:0/2429627643 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff134003530 con 0x7ff15006da10 2024-03-31T15:14:22.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:22.986+0000 7ff14ea00640 1 -- 172.21.15.16:0/2429627643 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff1500731c0 msgr2=0x7ff15006d4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:14:22.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:22.986+0000 7ff14ea00640 1 --2- 172.21.15.16:0/2429627643 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff1500731c0 0x7ff15006d4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:22.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:22.986+0000 7ff14ea00640 1 -- 172.21.15.16:0/2429627643 >> v1:172.21.15.16:6789/0 conn(0x7ff15006da10 legacy=0x7ff15015fdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:22.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:22.986+0000 7ff14ea00640 1 -- 172.21.15.16:0/2429627643 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff15015caa0 con 0x7ff1501567e0 2024-03-31T15:14:22.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:22.990+0000 7ff14ea00640 1 --2- 172.21.15.16:0/2429627643 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff1501567e0 0x7ff150072c80 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7ff1400097c0 tx=0x7ff14001c4a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1b0141e273145053 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:22.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:22.990+0000 7ff14e000640 1 -- 172.21.15.16:0/2429627643 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1400b0070 con 0x7ff1501567e0 2024-03-31T15:14:22.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:22.990+0000 7ff14e000640 1 -- 172.21.15.16:0/2429627643 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff1400a4e50 con 0x7ff1501567e0 2024-03-31T15:14:22.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:22.990+0000 7ff14e000640 1 -- 172.21.15.16:0/2429627643 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1400ab800 con 0x7ff1501567e0 2024-03-31T15:14:22.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:22.990+0000 7ff155800640 1 -- 172.21.15.16:0/2429627643 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff1501567e0 msgr2=0x7ff150072c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:22.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:22.990+0000 7ff155800640 1 --2- 172.21.15.16:0/2429627643 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff1501567e0 0x7ff150072c80 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7ff1400097c0 tx=0x7ff14001c4a0 comp rx=0 tx=0).stop 2024-03-31T15:14:22.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:22.990+0000 7ff155800640 1 -- 172.21.15.16:0/2429627643 shutdown_connections 2024-03-31T15:14:22.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:22.990+0000 7ff155800640 1 --2- 172.21.15.16:0/2429627643 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff1500731c0 0x7ff15006d4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:22.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:22.990+0000 7ff155800640 1 --2- 172.21.15.16:0/2429627643 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff1501567e0 0x7ff150072c80 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:22.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:22.990+0000 7ff155800640 1 -- 172.21.15.16:0/2429627643 >> 172.21.15.16:0/2429627643 conn(0x7ff15006ee40 msgr2=0x7ff15006b620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:14:22.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:22.990+0000 7ff155800640 1 -- 172.21.15.16:0/2429627643 shutdown_connections 2024-03-31T15:14:22.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:22.990+0000 7ff155800640 1 -- 172.21.15.16:0/2429627643 wait complete. 2024-03-31T15:14:22.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:22.994+0000 7ff155800640 1 Processor -- start 2024-03-31T15:14:22.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:22.994+0000 7ff155800640 1 -- start start 2024-03-31T15:14:22.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:22.994+0000 7ff155800640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff15006da10 0x7ff15015ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:22.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:22.994+0000 7ff155800640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff1500731c0 0x7ff150150a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:22.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:22.994+0000 7ff155800640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff1501567e0 0x7ff150150f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:22.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:22.994+0000 7ff155800640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7ff15015d700 con 0x7ff15006da10 2024-03-31T15:14:22.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:22.994+0000 7ff155800640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7ff15015d580 con 0x7ff1500731c0 2024-03-31T15:14:22.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:22.994+0000 7ff155800640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7ff150169170 con 0x7ff1501567e0 2024-03-31T15:14:22.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:22.994+0000 7ff14ea00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff15006da10 0x7ff15015ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:22.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:22.994+0000 7ff14ea00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff15006da10 0x7ff15015ae10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42892/0 (socket says 172.21.15.16:42892) 2024-03-31T15:14:22.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:22.994+0000 7ff14ea00640 1 -- 172.21.15.16:0/3583923304 learned_addr learned my addr 172.21.15.16:0/3583923304 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:14:22.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:22.994+0000 7ff14f400640 1 --2- 172.21.15.16:0/3583923304 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff1501567e0 0x7ff150150f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:22.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:22.994+0000 7ff146000640 1 --2- 172.21.15.16:0/3583923304 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff1500731c0 0x7ff150150a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:22.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:22.994+0000 7ff14ea00640 1 -- 172.21.15.16:0/3583923304 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff1501567e0 msgr2=0x7ff150150f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:22.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:22.994+0000 7ff14ea00640 1 --2- 172.21.15.16:0/3583923304 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff1501567e0 0x7ff150150f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:22.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:22.994+0000 7ff14ea00640 1 -- 172.21.15.16:0/3583923304 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff1500731c0 msgr2=0x7ff150150a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:22.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:22.994+0000 7ff14ea00640 1 --2- 172.21.15.16:0/3583923304 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff1500731c0 0x7ff150150a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:22.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:22.994+0000 7ff14ea00640 1 -- 172.21.15.16:0/3583923304 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff150068a20 con 0x7ff15006da10 2024-03-31T15:14:22.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:22.994+0000 7ff14ea00640 1 --2- 172.21.15.16:0/3583923304 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff15006da10 0x7ff15015ae10 secure :-1 s=READY pgs=1684 cs=0 l=1 rev1=1 crypto rx=0x7ff14001ab50 tx=0x7ff140019240 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:22.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:22.994+0000 7ff147400640 1 -- 172.21.15.16:0/3583923304 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1400b0040 con 0x7ff15006da10 2024-03-31T15:14:22.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:22.994+0000 7ff147400640 1 -- 172.21.15.16:0/3583923304 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff1400bb070 con 0x7ff15006da10 2024-03-31T15:14:22.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:22.994+0000 7ff147400640 1 -- 172.21.15.16:0/3583923304 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff140003aa0 con 0x7ff15006da10 2024-03-31T15:14:22.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:22.994+0000 7ff155800640 1 -- 172.21.15.16:0/3583923304 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff15015b770 con 0x7ff15006da10 2024-03-31T15:14:22.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:22.994+0000 7ff155800640 1 -- 172.21.15.16:0/3583923304 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff15014d5c0 con 0x7ff15006da10 2024-03-31T15:14:22.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:22.998+0000 7ff155800640 1 -- 172.21.15.16:0/3583923304 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff150069950 con 0x7ff15006da10 2024-03-31T15:14:22.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:22.998+0000 7ff147400640 1 -- 172.21.15.16:0/3583923304 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7ff1400b3070 con 0x7ff15006da10 2024-03-31T15:14:22.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:22.998+0000 7ff147400640 1 --2- 172.21.15.16:0/3583923304 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7ff12c068380 0x7ff12c06a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:22.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:22.998+0000 7ff147400640 1 -- 172.21.15.16:0/3583923304 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 5767+0+0 (secure 0 0 0) 0x7ff1401204f0 con 0x7ff15006da10 2024-03-31T15:14:22.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:22.998+0000 7ff146000640 1 --2- 172.21.15.16:0/3583923304 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7ff12c068380 0x7ff12c06a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:22.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:22.998+0000 7ff146000640 1 --2- 172.21.15.16:0/3583923304 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7ff12c068380 0x7ff12c06a840 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7ff150153620 tx=0x7ff138009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:22.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:23.002+0000 7ff147400640 1 -- 172.21.15.16:0/3583923304 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7ff15015b770 con 0x7ff15006da10 2024-03-31T15:14:23.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:23.166+0000 7ff155800640 1 -- 172.21.15.16:0/3583923304 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7ff150152fb0 con 0x7ff15006da10 2024-03-31T15:14:23.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:23.170+0000 7ff147400640 1 -- 172.21.15.16:0/3583923304 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v111) ==== 93+0+10 (secure 0 0 0) 0x7ff140016090 con 0x7ff15006da10 2024-03-31T15:14:23.161 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2024-03-31T15:14:23.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:23.170+0000 7ff155800640 1 -- 172.21.15.16:0/3583923304 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7ff12c068380 msgr2=0x7ff12c06a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:23.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:23.170+0000 7ff155800640 1 --2- 172.21.15.16:0/3583923304 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7ff12c068380 0x7ff12c06a840 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7ff150153620 tx=0x7ff138009190 comp rx=0 tx=0).stop 2024-03-31T15:14:23.164 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:23.170+0000 7ff155800640 1 -- 172.21.15.16:0/3583923304 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff15006da10 msgr2=0x7ff15015ae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:23.164 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:23.170+0000 7ff155800640 1 --2- 172.21.15.16:0/3583923304 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff15006da10 0x7ff15015ae10 secure :-1 s=READY pgs=1684 cs=0 l=1 rev1=1 crypto rx=0x7ff14001ab50 tx=0x7ff140019240 comp rx=0 tx=0).stop 2024-03-31T15:14:23.164 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:23.170+0000 7ff155800640 1 -- 172.21.15.16:0/3583923304 shutdown_connections 2024-03-31T15:14:23.164 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:23.170+0000 7ff155800640 1 --2- 172.21.15.16:0/3583923304 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7ff12c068380 0x7ff12c06a840 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:23.164 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:23.170+0000 7ff155800640 1 --2- 172.21.15.16:0/3583923304 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff1501567e0 0x7ff150150f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:23.164 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:23.170+0000 7ff155800640 1 --2- 172.21.15.16:0/3583923304 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff1500731c0 0x7ff150150a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:23.164 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:23.170+0000 7ff155800640 1 --2- 172.21.15.16:0/3583923304 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff15006da10 0x7ff15015ae10 unknown :-1 s=CLOSED pgs=1684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:23.164 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:23.170+0000 7ff155800640 1 -- 172.21.15.16:0/3583923304 >> 172.21.15.16:0/3583923304 conn(0x7ff15006ee40 msgr2=0x7ff15015f010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:14:23.164 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:23.170+0000 7ff155800640 1 -- 172.21.15.16:0/3583923304 shutdown_connections 2024-03-31T15:14:23.164 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:23.170+0000 7ff155800640 1 -- 172.21.15.16:0/3583923304 wait complete. 2024-03-31T15:14:23.182 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-31T15:14:23.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:23.338+0000 7f6a0be00640 1 Processor -- start 2024-03-31T15:14:23.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:23.338+0000 7f6a0be00640 1 -- start start 2024-03-31T15:14:23.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:23.338+0000 7f6a0be00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f6a0c15af60 0x7f6a0c06e360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:23.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:23.338+0000 7f6a0be00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f6a0c072ae0 0x7f6a0c0afb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:23.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:23.338+0000 7f6a0be00640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6a0c0cd0d0 con 0x7f6a0c0cc6f0 2024-03-31T15:14:23.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:23.338+0000 7f6a0be00640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f6a0c0aee70 con 0x7f6a0c15af60 2024-03-31T15:14:23.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:23.338+0000 7f6a0be00640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f6a0c0cd3b0 con 0x7f6a0c072ae0 2024-03-31T15:14:23.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:23.338+0000 7f6a0aa00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f6a0c15af60 0x7f6a0c06e360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:23.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:23.338+0000 7f6a02000640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f6a0c0cc6f0 0x7f6a0c0723d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:53460/0 (socket says 172.21.15.16:53460) 2024-03-31T15:14:23.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:23.338+0000 7f6a02000640 1 -- 172.21.15.16:0/3012976885 learned_addr learned my addr 172.21.15.16:0/3012976885 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:14:23.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:23.338+0000 7f6a0b400640 1 --2- 172.21.15.16:0/3012976885 >> v2:172.21.15.89:3300/0 conn(0x7f6a0c072ae0 0x7f6a0c0afb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:23.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:23.338+0000 7f6a0aa00640 1 -- 172.21.15.16:0/3012976885 >> v2:172.21.15.89:3300/0 conn(0x7f6a0c072ae0 msgr2=0x7f6a0c0afb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:14:23.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:23.338+0000 7f6a0aa00640 1 --2- 172.21.15.16:0/3012976885 >> v2:172.21.15.89:3300/0 conn(0x7f6a0c072ae0 0x7f6a0c0afb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:23.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:23.338+0000 7f6a0aa00640 1 -- 172.21.15.16:0/3012976885 >> v1:172.21.15.16:6789/0 conn(0x7f6a0c0cc6f0 legacy=0x7f6a0c0723d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:23.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:23.338+0000 7f6a0aa00640 1 -- 172.21.15.16:0/3012976885 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a0c0af0a0 con 0x7f6a0c15af60 2024-03-31T15:14:23.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:23.338+0000 7f6a0a000640 1 -- 172.21.15.16:0/3012976885 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2643112083 0 0) 0x7f6a0c0cd0d0 con 0x7f6a0c0cc6f0 2024-03-31T15:14:23.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:23.342+0000 7f6a0aa00640 1 --2- 172.21.15.16:0/3012976885 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6a0c15af60 0x7f6a0c06e360 secure :-1 s=READY pgs=1686 cs=0 l=1 rev1=1 crypto rx=0x7f6a0c0d0da0 tx=0x7f69fc01c3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88cbe3d9608a813c server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:23.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:23.346+0000 7f6a0a000640 1 -- 172.21.15.16:0/3012976885 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69fc0b0070 con 0x7f6a0c15af60 2024-03-31T15:14:23.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:23.346+0000 7f6a0a000640 1 -- 172.21.15.16:0/3012976885 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f69fc0ab600 con 0x7f6a0c15af60 2024-03-31T15:14:23.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:23.346+0000 7f6a0a000640 1 -- 172.21.15.16:0/3012976885 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69fc0ab9d0 con 0x7f6a0c15af60 2024-03-31T15:14:23.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:23.346+0000 7f6a0be00640 1 -- 172.21.15.16:0/3012976885 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6a0c15af60 msgr2=0x7f6a0c06e360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:23.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:23.346+0000 7f6a0be00640 1 --2- 172.21.15.16:0/3012976885 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6a0c15af60 0x7f6a0c06e360 secure :-1 s=READY pgs=1686 cs=0 l=1 rev1=1 crypto rx=0x7f6a0c0d0da0 tx=0x7f69fc01c3c0 comp rx=0 tx=0).stop 2024-03-31T15:14:23.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:23.346+0000 7f6a0be00640 1 -- 172.21.15.16:0/3012976885 shutdown_connections 2024-03-31T15:14:23.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:23.346+0000 7f6a0be00640 1 --2- 172.21.15.16:0/3012976885 >> v2:172.21.15.89:3300/0 conn(0x7f6a0c072ae0 0x7f6a0c0afb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:23.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:23.346+0000 7f6a0be00640 1 --2- 172.21.15.16:0/3012976885 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6a0c15af60 0x7f6a0c06e360 unknown :-1 s=CLOSED pgs=1686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:23.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:23.346+0000 7f6a0be00640 1 -- 172.21.15.16:0/3012976885 >> 172.21.15.16:0/3012976885 conn(0x7f6a0c0d0df0 msgr2=0x7f6a0c0d1200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:14:23.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:23.346+0000 7f6a0be00640 1 -- 172.21.15.16:0/3012976885 shutdown_connections 2024-03-31T15:14:23.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:23.346+0000 7f6a0be00640 1 -- 172.21.15.16:0/3012976885 wait complete. 2024-03-31T15:14:23.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:23.346+0000 7f6a0be00640 1 Processor -- start 2024-03-31T15:14:23.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:23.346+0000 7f6a0be00640 1 -- start start 2024-03-31T15:14:23.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:23.346+0000 7f6a0be00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6a0c072ae0 0x7f6a0c0c6f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:23.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:23.346+0000 7f6a0be00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6a0c0cc6f0 0x7f6a0c0b3ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:23.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:23.346+0000 7f6a0be00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f6a0c15af60 0x7f6a0c0b52f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:23.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:23.346+0000 7f6a0be00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f6a0c15f970 con 0x7f6a0c072ae0 2024-03-31T15:14:23.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:23.346+0000 7f6a0be00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f6a0c15b340 con 0x7f6a0c0cc6f0 2024-03-31T15:14:23.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:23.346+0000 7f6a0be00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f6a0c15faf0 con 0x7f6a0c15af60 2024-03-31T15:14:23.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:23.346+0000 7f6a0aa00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6a0c072ae0 0x7f6a0c0c6f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:23.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:23.346+0000 7f6a0aa00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6a0c072ae0 0x7f6a0c0c6f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42910/0 (socket says 172.21.15.16:42910) 2024-03-31T15:14:23.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:23.346+0000 7f6a0aa00640 1 -- 172.21.15.16:0/4057345467 learned_addr learned my addr 172.21.15.16:0/4057345467 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:14:23.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:23.346+0000 7f6a02000640 1 --2- 172.21.15.16:0/4057345467 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6a0c0cc6f0 0x7f6a0c0b3ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:23.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:23.346+0000 7f6a0aa00640 1 -- 172.21.15.16:0/4057345467 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f6a0c15af60 msgr2=0x7f6a0c0b52f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:23.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:23.346+0000 7f6a0aa00640 1 --2- 172.21.15.16:0/4057345467 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f6a0c15af60 0x7f6a0c0b52f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:23.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:23.346+0000 7f6a0aa00640 1 -- 172.21.15.16:0/4057345467 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6a0c0cc6f0 msgr2=0x7f6a0c0b3ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:23.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:23.346+0000 7f6a0aa00640 1 --2- 172.21.15.16:0/4057345467 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6a0c0cc6f0 0x7f6a0c0b3ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:23.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:23.346+0000 7f6a0aa00640 1 -- 172.21.15.16:0/4057345467 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a0c0b4250 con 0x7f6a0c072ae0 2024-03-31T15:14:23.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:23.350+0000 7f6a02000640 1 --2- 172.21.15.16:0/4057345467 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6a0c0cc6f0 0x7f6a0c0b3ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:14:23.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:23.350+0000 7f6a0aa00640 1 --2- 172.21.15.16:0/4057345467 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6a0c072ae0 0x7f6a0c0c6f30 secure :-1 s=READY pgs=1687 cs=0 l=1 rev1=1 crypto rx=0x7f69fc0ae040 tx=0x7f69fc019160 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:23.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:23.350+0000 7f6a03e00640 1 -- 172.21.15.16:0/4057345467 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69fc0b8020 con 0x7f6a0c072ae0 2024-03-31T15:14:23.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:23.350+0000 7f6a03e00640 1 -- 172.21.15.16:0/4057345467 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f69fc0b4410 con 0x7f6a0c072ae0 2024-03-31T15:14:23.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:23.350+0000 7f6a03e00640 1 -- 172.21.15.16:0/4057345467 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69fc0b0040 con 0x7f6a0c072ae0 2024-03-31T15:14:23.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:23.350+0000 7f6a0be00640 1 -- 172.21.15.16:0/4057345467 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6a0c0b4530 con 0x7f6a0c072ae0 2024-03-31T15:14:23.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:23.350+0000 7f6a0be00640 1 -- 172.21.15.16:0/4057345467 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6a0c0b4af0 con 0x7f6a0c072ae0 2024-03-31T15:14:23.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:23.350+0000 7f6a0be00640 1 -- 172.21.15.16:0/4057345467 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6a0c04f8c0 con 0x7f6a0c072ae0 2024-03-31T15:14:23.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:23.350+0000 7f6a03e00640 1 -- 172.21.15.16:0/4057345467 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f69fc0bb020 con 0x7f6a0c072ae0 2024-03-31T15:14:23.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:23.350+0000 7f6a03e00640 1 --2- 172.21.15.16:0/4057345467 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f69e8068380 0x7f69e806a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:23.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:23.350+0000 7f6a03e00640 1 -- 172.21.15.16:0/4057345467 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 5767+0+0 (secure 0 0 0) 0x7f69fc120650 con 0x7f6a0c072ae0 2024-03-31T15:14:23.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:23.350+0000 7f6a02000640 1 --2- 172.21.15.16:0/4057345467 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f69e8068380 0x7f69e806a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:23.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:23.354+0000 7f6a02000640 1 --2- 172.21.15.16:0/4057345467 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f69e8068380 0x7f69e806a840 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f6a0c0b69c0 tx=0x7f69f4009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:23.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:23.362+0000 7f6a03e00640 1 -- 172.21.15.16:0/4057345467 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f69fc0c2030 con 0x7f6a0c072ae0 2024-03-31T15:14:23.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:23.526+0000 7f6a0be00640 1 -- 172.21.15.16:0/4057345467 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f6a0c074b60 con 0x7f6a0c072ae0 2024-03-31T15:14:23.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:23.526+0000 7f6a03e00640 1 -- 172.21.15.16:0/4057345467 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v111) ==== 104+0+11 (secure 0 0 0) 0x7f69fc016070 con 0x7f6a0c072ae0 2024-03-31T15:14:23.518 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2024-03-31T15:14:23.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:23.526+0000 7f6a0be00640 1 -- 172.21.15.16:0/4057345467 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f69e8068380 msgr2=0x7f69e806a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:23.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:23.526+0000 7f6a0be00640 1 --2- 172.21.15.16:0/4057345467 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f69e8068380 0x7f69e806a840 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f6a0c0b69c0 tx=0x7f69f4009040 comp rx=0 tx=0).stop 2024-03-31T15:14:23.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:23.526+0000 7f6a0be00640 1 -- 172.21.15.16:0/4057345467 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6a0c072ae0 msgr2=0x7f6a0c0c6f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:23.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:23.526+0000 7f6a0be00640 1 --2- 172.21.15.16:0/4057345467 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6a0c072ae0 0x7f6a0c0c6f30 secure :-1 s=READY pgs=1687 cs=0 l=1 rev1=1 crypto rx=0x7f69fc0ae040 tx=0x7f69fc019160 comp rx=0 tx=0).stop 2024-03-31T15:14:23.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:23.526+0000 7f6a0be00640 1 -- 172.21.15.16:0/4057345467 shutdown_connections 2024-03-31T15:14:23.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:23.526+0000 7f6a0be00640 1 --2- 172.21.15.16:0/4057345467 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f69e8068380 0x7f69e806a840 unknown :-1 s=CLOSED pgs=716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:23.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:23.526+0000 7f6a0be00640 1 --2- 172.21.15.16:0/4057345467 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f6a0c15af60 0x7f6a0c0b52f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:23.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:23.526+0000 7f6a0be00640 1 --2- 172.21.15.16:0/4057345467 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6a0c0cc6f0 0x7f6a0c0b3ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:23.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:23.526+0000 7f6a0be00640 1 --2- 172.21.15.16:0/4057345467 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6a0c072ae0 0x7f6a0c0c6f30 unknown :-1 s=CLOSED pgs=1687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:23.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:23.526+0000 7f6a0be00640 1 -- 172.21.15.16:0/4057345467 >> 172.21.15.16:0/4057345467 conn(0x7f6a0c0d0df0 msgr2=0x7f6a0c0cbcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:14:23.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:23.526+0000 7f6a0be00640 1 -- 172.21.15.16:0/4057345467 shutdown_connections 2024-03-31T15:14:23.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:23.526+0000 7f6a0be00640 1 -- 172.21.15.16:0/4057345467 wait complete. 2024-03-31T15:14:23.543 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-31T15:14:23.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:23.694+0000 7fadaca00640 1 Processor -- start 2024-03-31T15:14:23.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:23.694+0000 7fadaca00640 1 -- start start 2024-03-31T15:14:23.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:23.694+0000 7fadaca00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fada81411b0 0x7fada806f920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:23.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:23.694+0000 7fadaca00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fada8141590 0x7fada806fe60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:23.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:23.694+0000 7fadaca00640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fada8144980 con 0x7fada8143c30 2024-03-31T15:14:23.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:23.694+0000 7fadaca00640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7fada815f910 con 0x7fada8141590 2024-03-31T15:14:23.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:23.694+0000 7fadaca00640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7fada8162910 con 0x7fada81411b0 2024-03-31T15:14:23.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:23.694+0000 7fada6800640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fada8143c30 0x7fada81440a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:53470/0 (socket says 172.21.15.16:53470) 2024-03-31T15:14:23.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:23.694+0000 7fada6800640 1 -- 172.21.15.16:0/2625931010 learned_addr learned my addr 172.21.15.16:0/2625931010 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:14:23.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:23.694+0000 7fad9fe00640 1 --2- 172.21.15.16:0/2625931010 >> v2:172.21.15.16:3300/0 conn(0x7fada8141590 0x7fada806fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:23.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:23.694+0000 7fada5e00640 1 --2- 172.21.15.16:0/2625931010 >> v2:172.21.15.89:3300/0 conn(0x7fada81411b0 0x7fada806f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:23.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:23.694+0000 7fad9fe00640 1 -- 172.21.15.16:0/2625931010 >> v2:172.21.15.89:3300/0 conn(0x7fada81411b0 msgr2=0x7fada806f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:14:23.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:23.694+0000 7fad9fe00640 1 --2- 172.21.15.16:0/2625931010 >> v2:172.21.15.89:3300/0 conn(0x7fada81411b0 0x7fada806f920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:23.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:23.694+0000 7fad9fe00640 1 -- 172.21.15.16:0/2625931010 >> v1:172.21.15.16:6789/0 conn(0x7fada8143c30 legacy=0x7fada81440a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:23.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:23.694+0000 7fad9fe00640 1 -- 172.21.15.16:0/2625931010 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fada8161fe0 con 0x7fada8141590 2024-03-31T15:14:23.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:23.694+0000 7fada5400640 1 -- 172.21.15.16:0/2625931010 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1549370248 0 0) 0x7fada8144980 con 0x7fada8143c30 2024-03-31T15:14:23.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:23.702+0000 7fad9fe00640 1 --2- 172.21.15.16:0/2625931010 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fada8141590 0x7fada806fe60 secure :-1 s=READY pgs=1689 cs=0 l=1 rev1=1 crypto rx=0x7fada80d3120 tx=0x7fad9001c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8d16a4955efc4a3 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:23.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:23.702+0000 7fada5400640 1 -- 172.21.15.16:0/2625931010 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad900b0070 con 0x7fada8141590 2024-03-31T15:14:23.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:23.702+0000 7fada5400640 1 -- 172.21.15.16:0/2625931010 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fad900ab600 con 0x7fada8141590 2024-03-31T15:14:23.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:23.702+0000 7fada5400640 1 -- 172.21.15.16:0/2625931010 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad900ab9d0 con 0x7fada8141590 2024-03-31T15:14:23.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:23.702+0000 7fadaca00640 1 -- 172.21.15.16:0/2625931010 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fada8141590 msgr2=0x7fada806fe60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:23.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:23.702+0000 7fadaca00640 1 --2- 172.21.15.16:0/2625931010 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fada8141590 0x7fada806fe60 secure :-1 s=READY pgs=1689 cs=0 l=1 rev1=1 crypto rx=0x7fada80d3120 tx=0x7fad9001c480 comp rx=0 tx=0).stop 2024-03-31T15:14:23.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:23.702+0000 7fadaca00640 1 -- 172.21.15.16:0/2625931010 shutdown_connections 2024-03-31T15:14:23.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:23.702+0000 7fadaca00640 1 --2- 172.21.15.16:0/2625931010 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fada8141590 0x7fada806fe60 unknown :-1 s=CLOSED pgs=1689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:23.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:23.702+0000 7fadaca00640 1 --2- 172.21.15.16:0/2625931010 >> v2:172.21.15.89:3300/0 conn(0x7fada81411b0 0x7fada806f920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:23.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:23.702+0000 7fadaca00640 1 -- 172.21.15.16:0/2625931010 >> 172.21.15.16:0/2625931010 conn(0x7fada8067930 msgr2=0x7fada80b3ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:14:23.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:23.702+0000 7fadaca00640 1 -- 172.21.15.16:0/2625931010 shutdown_connections 2024-03-31T15:14:23.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:23.702+0000 7fadaca00640 1 -- 172.21.15.16:0/2625931010 wait complete. 2024-03-31T15:14:23.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:23.702+0000 7fadaca00640 1 Processor -- start 2024-03-31T15:14:23.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:23.702+0000 7fadaca00640 1 -- start start 2024-03-31T15:14:23.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:23.702+0000 7fadaca00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fada81411b0 0x7fada8152830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:23.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:23.702+0000 7fadaca00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fada8141590 0x7fada813bb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:23.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:23.702+0000 7fadaca00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fada8143c30 0x7fada813cf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:23.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:23.702+0000 7fadaca00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fada815f290 con 0x7fada81411b0 2024-03-31T15:14:23.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:23.702+0000 7fadaca00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fada815f110 con 0x7fada8143c30 2024-03-31T15:14:23.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:23.702+0000 7fadaca00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fada815f410 con 0x7fada8141590 2024-03-31T15:14:23.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:23.702+0000 7fada5e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fada81411b0 0x7fada8152830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:23.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:23.706+0000 7fada5e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fada81411b0 0x7fada8152830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42940/0 (socket says 172.21.15.16:42940) 2024-03-31T15:14:23.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:23.706+0000 7fada5e00640 1 -- 172.21.15.16:0/1157277192 learned_addr learned my addr 172.21.15.16:0/1157277192 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:14:23.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:23.706+0000 7fad9fe00640 1 --2- 172.21.15.16:0/1157277192 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fada8141590 0x7fada813bb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:23.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:23.706+0000 7fada6800640 1 --2- 172.21.15.16:0/1157277192 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fada8143c30 0x7fada813cf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:23.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:23.706+0000 7fada5e00640 1 -- 172.21.15.16:0/1157277192 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fada8141590 msgr2=0x7fada813bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:23.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:23.706+0000 7fada5e00640 1 --2- 172.21.15.16:0/1157277192 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fada8141590 0x7fada813bb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:23.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:23.706+0000 7fada5e00640 1 -- 172.21.15.16:0/1157277192 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fada8143c30 msgr2=0x7fada813cf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:23.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:23.706+0000 7fada5e00640 1 --2- 172.21.15.16:0/1157277192 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fada8143c30 0x7fada813cf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:23.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:23.706+0000 7fada5e00640 1 -- 172.21.15.16:0/1157277192 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fada80b3770 con 0x7fada81411b0 2024-03-31T15:14:23.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:23.706+0000 7fada6800640 1 --2- 172.21.15.16:0/1157277192 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fada8143c30 0x7fada813cf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:14:23.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:23.706+0000 7fada5e00640 1 --2- 172.21.15.16:0/1157277192 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fada81411b0 0x7fada8152830 secure :-1 s=READY pgs=1690 cs=0 l=1 rev1=1 crypto rx=0x7fad9800e960 tx=0x7fad9800ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:23.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:23.706+0000 7fad9e000640 1 -- 172.21.15.16:0/1157277192 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad98017070 con 0x7fada81411b0 2024-03-31T15:14:23.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:23.706+0000 7fad9e000640 1 -- 172.21.15.16:0/1157277192 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fad9800ca00 con 0x7fada81411b0 2024-03-31T15:14:23.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:23.706+0000 7fad9e000640 1 -- 172.21.15.16:0/1157277192 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad9800cdb0 con 0x7fada81411b0 2024-03-31T15:14:23.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:23.706+0000 7fadaca00640 1 -- 172.21.15.16:0/1157277192 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fada813c470 con 0x7fada81411b0 2024-03-31T15:14:23.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:23.706+0000 7fadaca00640 1 -- 172.21.15.16:0/1157277192 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fada813c0a0 con 0x7fada81411b0 2024-03-31T15:14:23.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:23.706+0000 7fadaca00640 1 -- 172.21.15.16:0/1157277192 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fada8069950 con 0x7fada81411b0 2024-03-31T15:14:23.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:23.706+0000 7fad9e000640 1 -- 172.21.15.16:0/1157277192 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fad9801a070 con 0x7fada81411b0 2024-03-31T15:14:23.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:23.706+0000 7fad9e000640 1 --2- 172.21.15.16:0/1157277192 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fad84068380 0x7fad8406a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:23.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:23.710+0000 7fad9e000640 1 -- 172.21.15.16:0/1157277192 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 5767+0+0 (secure 0 0 0) 0x7fad98086d50 con 0x7fada81411b0 2024-03-31T15:14:23.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:23.710+0000 7fad9fe00640 1 --2- 172.21.15.16:0/1157277192 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fad84068380 0x7fad8406a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:23.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:23.710+0000 7fad9fe00640 1 --2- 172.21.15.16:0/1157277192 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fad84068380 0x7fad8406a840 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7fad900ad000 tx=0x7fad90008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:23.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:23.710+0000 7fad9e000640 1 -- 172.21.15.16:0/1157277192 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fad98056c70 con 0x7fada81411b0 2024-03-31T15:14:24.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:24.014+0000 7fadaca00640 1 -- 172.21.15.16:0/1157277192 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fada80b4a20 con 0x7fada81411b0 2024-03-31T15:14:24.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:24.014+0000 7fad9e000640 1 -- 172.21.15.16:0/1157277192 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v111) ==== 100+0+11 (secure 0 0 0) 0x7fad9805af70 con 0x7fada81411b0 2024-03-31T15:14:24.008 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 64 2024-03-31T15:14:24.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:24.018+0000 7fadaca00640 1 -- 172.21.15.16:0/1157277192 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fad84068380 msgr2=0x7fad8406a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:24.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:24.018+0000 7fadaca00640 1 --2- 172.21.15.16:0/1157277192 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fad84068380 0x7fad8406a840 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7fad900ad000 tx=0x7fad90008040 comp rx=0 tx=0).stop 2024-03-31T15:14:24.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:24.018+0000 7fadaca00640 1 -- 172.21.15.16:0/1157277192 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fada81411b0 msgr2=0x7fada8152830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:24.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:24.018+0000 7fadaca00640 1 --2- 172.21.15.16:0/1157277192 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fada81411b0 0x7fada8152830 secure :-1 s=READY pgs=1690 cs=0 l=1 rev1=1 crypto rx=0x7fad9800e960 tx=0x7fad9800ee20 comp rx=0 tx=0).stop 2024-03-31T15:14:24.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:24.018+0000 7fadaca00640 1 -- 172.21.15.16:0/1157277192 shutdown_connections 2024-03-31T15:14:24.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:24.018+0000 7fadaca00640 1 --2- 172.21.15.16:0/1157277192 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fad84068380 0x7fad8406a840 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:24.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:24.018+0000 7fadaca00640 1 --2- 172.21.15.16:0/1157277192 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fada8143c30 0x7fada813cf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:24.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:24.018+0000 7fadaca00640 1 --2- 172.21.15.16:0/1157277192 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fada8141590 0x7fada813bb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:24.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:24.018+0000 7fadaca00640 1 --2- 172.21.15.16:0/1157277192 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fada81411b0 0x7fada8152830 unknown :-1 s=CLOSED pgs=1690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:24.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:24.018+0000 7fadaca00640 1 -- 172.21.15.16:0/1157277192 >> 172.21.15.16:0/1157277192 conn(0x7fada8067930 msgr2=0x7fada80b5030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:14:24.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:24.018+0000 7fadaca00640 1 -- 172.21.15.16:0/1157277192 shutdown_connections 2024-03-31T15:14:24.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:24.018+0000 7fadaca00640 1 -- 172.21.15.16:0/1157277192 wait complete. 2024-03-31T15:14:24.029 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-31T15:14:24.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:24.178+0000 7f33bce00640 1 Processor -- start 2024-03-31T15:14:24.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:24.178+0000 7f33bce00640 1 -- start start 2024-03-31T15:14:24.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:24.178+0000 7f33bce00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f33b806dbc0 0x7f33b815f650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:24.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:24.178+0000 7f33bce00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f33b8070030 0x7f33b8162180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:24.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:24.178+0000 7f33bce00640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f33b80cbe90 con 0x7f33b80cb440 2024-03-31T15:14:24.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:24.178+0000 7f33bce00640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f33b815fb90 con 0x7f33b806dbc0 2024-03-31T15:14:24.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:24.178+0000 7f33bce00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f33b815f1c0 con 0x7f33b8070030 2024-03-31T15:14:24.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:24.178+0000 7f33afe00640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f33b80cb440 0x7f33b806f920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:53474/0 (socket says 172.21.15.16:53474) 2024-03-31T15:14:24.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:24.178+0000 7f33afe00640 1 -- 172.21.15.16:0/703210247 learned_addr learned my addr 172.21.15.16:0/703210247 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:14:24.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:24.178+0000 7f33b6800640 1 --2- 172.21.15.16:0/703210247 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f33b8070030 0x7f33b8162180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:24.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:24.178+0000 7f33b5e00640 1 --2- 172.21.15.16:0/703210247 >> v2:172.21.15.89:3300/0 conn(0x7f33b806dbc0 0x7f33b815f650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:24.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:24.178+0000 7f33b5400640 1 -- 172.21.15.16:0/703210247 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 675563820 0 0) 0x7f33b80cbe90 con 0x7f33b80cb440 2024-03-31T15:14:24.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:24.178+0000 7f33b5400640 1 -- 172.21.15.16:0/703210247 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f339c003530 con 0x7f33b80cb440 2024-03-31T15:14:24.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:24.178+0000 7f33b5e00640 1 -- 172.21.15.16:0/703210247 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f33b8070030 msgr2=0x7f33b8162180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:14:24.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:24.178+0000 7f33b5e00640 1 --2- 172.21.15.16:0/703210247 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f33b8070030 0x7f33b8162180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:24.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:24.178+0000 7f33b5e00640 1 -- 172.21.15.16:0/703210247 >> v1:172.21.15.16:6789/0 conn(0x7f33b80cb440 legacy=0x7f33b806f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:24.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:24.178+0000 7f33b5e00640 1 -- 172.21.15.16:0/703210247 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f33b80cc170 con 0x7f33b806dbc0 2024-03-31T15:14:24.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:24.178+0000 7f33b6800640 1 --2- 172.21.15.16:0/703210247 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f33b8070030 0x7f33b8162180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:14:24.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:24.182+0000 7f33b5e00640 1 --2- 172.21.15.16:0/703210247 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f33b806dbc0 0x7f33b815f650 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f33a80097c0 tx=0x7f33a801c3e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a9aa478e2dd91445 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:24.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:24.182+0000 7f33b5400640 1 -- 172.21.15.16:0/703210247 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33a80b0070 con 0x7f33b806dbc0 2024-03-31T15:14:24.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:24.182+0000 7f33b5400640 1 -- 172.21.15.16:0/703210247 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f33a80ab600 con 0x7f33b806dbc0 2024-03-31T15:14:24.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:24.182+0000 7f33b5400640 1 -- 172.21.15.16:0/703210247 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33a80ab9b0 con 0x7f33b806dbc0 2024-03-31T15:14:24.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:24.182+0000 7f33bce00640 1 -- 172.21.15.16:0/703210247 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f33b806dbc0 msgr2=0x7f33b815f650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:24.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:24.182+0000 7f33bce00640 1 --2- 172.21.15.16:0/703210247 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f33b806dbc0 0x7f33b815f650 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f33a80097c0 tx=0x7f33a801c3e0 comp rx=0 tx=0).stop 2024-03-31T15:14:24.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:24.182+0000 7f33bce00640 1 -- 172.21.15.16:0/703210247 shutdown_connections 2024-03-31T15:14:24.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:24.182+0000 7f33bce00640 1 --2- 172.21.15.16:0/703210247 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f33b8070030 0x7f33b8162180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:24.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:24.182+0000 7f33bce00640 1 --2- 172.21.15.16:0/703210247 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f33b806dbc0 0x7f33b815f650 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:24.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:24.182+0000 7f33bce00640 1 -- 172.21.15.16:0/703210247 >> 172.21.15.16:0/703210247 conn(0x7f33b8067330 msgr2=0x7f33b8067740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:14:24.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:24.182+0000 7f33bce00640 1 -- 172.21.15.16:0/703210247 shutdown_connections 2024-03-31T15:14:24.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:24.182+0000 7f33bce00640 1 -- 172.21.15.16:0/703210247 wait complete. 2024-03-31T15:14:24.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:24.186+0000 7f33bce00640 1 Processor -- start 2024-03-31T15:14:24.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:24.186+0000 7f33bce00640 1 -- start start 2024-03-31T15:14:24.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:24.186+0000 7f33bce00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f33b806dbc0 0x7f33b8140450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:24.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:24.186+0000 7f33bce00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f33b8070030 0x7f33b81409f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:24.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:24.186+0000 7f33bce00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f33b80cb440 0x7f33b815ca20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:24.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:24.186+0000 7f33bce00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f33b806cad0 con 0x7f33b8070030 2024-03-31T15:14:24.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:24.186+0000 7f33bce00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f33b806c950 con 0x7f33b80cb440 2024-03-31T15:14:24.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:24.186+0000 7f33bce00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f33b816b520 con 0x7f33b806dbc0 2024-03-31T15:14:24.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:24.186+0000 7f33afe00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f33b8070030 0x7f33b81409f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:24.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:24.186+0000 7f33afe00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f33b8070030 0x7f33b81409f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42948/0 (socket says 172.21.15.16:42948) 2024-03-31T15:14:24.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:24.186+0000 7f33afe00640 1 -- 172.21.15.16:0/3697219451 learned_addr learned my addr 172.21.15.16:0/3697219451 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:14:24.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:24.186+0000 7f33b5e00640 1 --2- 172.21.15.16:0/3697219451 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f33b806dbc0 0x7f33b8140450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:24.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:24.186+0000 7f33b6800640 1 --2- 172.21.15.16:0/3697219451 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f33b80cb440 0x7f33b815ca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:24.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:24.186+0000 7f33afe00640 1 -- 172.21.15.16:0/3697219451 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f33b806dbc0 msgr2=0x7f33b8140450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:24.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:24.186+0000 7f33afe00640 1 --2- 172.21.15.16:0/3697219451 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f33b806dbc0 0x7f33b8140450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:24.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:24.186+0000 7f33afe00640 1 -- 172.21.15.16:0/3697219451 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f33b80cb440 msgr2=0x7f33b815ca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:24.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:24.186+0000 7f33afe00640 1 --2- 172.21.15.16:0/3697219451 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f33b80cb440 0x7f33b815ca20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:24.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:24.186+0000 7f33afe00640 1 -- 172.21.15.16:0/3697219451 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f33b815b0f0 con 0x7f33b8070030 2024-03-31T15:14:24.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:24.186+0000 7f33b5e00640 1 --2- 172.21.15.16:0/3697219451 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f33b806dbc0 0x7f33b8140450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:14:24.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:24.186+0000 7f33afe00640 1 --2- 172.21.15.16:0/3697219451 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f33b8070030 0x7f33b81409f0 secure :-1 s=READY pgs=1692 cs=0 l=1 rev1=1 crypto rx=0x7f33a000c3c0 tx=0x7f33a000c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:24.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:24.186+0000 7f33ae000640 1 -- 172.21.15.16:0/3697219451 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33a0017070 con 0x7f33b8070030 2024-03-31T15:14:24.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:24.186+0000 7f33ae000640 1 -- 172.21.15.16:0/3697219451 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f33a0005cf0 con 0x7f33b8070030 2024-03-31T15:14:24.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:24.186+0000 7f33ae000640 1 -- 172.21.15.16:0/3697219451 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33a00072f0 con 0x7f33b8070030 2024-03-31T15:14:24.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:24.186+0000 7f33bce00640 1 -- 172.21.15.16:0/3697219451 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f33b815ab30 con 0x7f33b8070030 2024-03-31T15:14:24.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:24.186+0000 7f33bce00640 1 -- 172.21.15.16:0/3697219451 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f33b8159820 con 0x7f33b8070030 2024-03-31T15:14:24.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:24.190+0000 7f33bce00640 1 -- 172.21.15.16:0/3697219451 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f33b8069950 con 0x7f33b8070030 2024-03-31T15:14:24.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:24.190+0000 7f33ae000640 1 -- 172.21.15.16:0/3697219451 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f33a001a070 con 0x7f33b8070030 2024-03-31T15:14:24.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:24.190+0000 7f33ae000640 1 --2- 172.21.15.16:0/3697219451 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f33940683b0 0x7f339406a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:24.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:24.190+0000 7f33ae000640 1 -- 172.21.15.16:0/3697219451 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 5767+0+0 (secure 0 0 0) 0x7f33a0085d50 con 0x7f33b8070030 2024-03-31T15:14:24.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:24.190+0000 7f33b5e00640 1 --2- 172.21.15.16:0/3697219451 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f33940683b0 0x7f339406a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:24.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:24.190+0000 7f33b5e00640 1 --2- 172.21.15.16:0/3697219451 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f33940683b0 0x7f339406a870 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f33a80ad000 tx=0x7f33a80a6000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:24.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:24.194+0000 7f33ae000640 1 -- 172.21.15.16:0/3697219451 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f33a00562c0 con 0x7f33b8070030 2024-03-31T15:14:24.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:24.462+0000 7f33bce00640 1 -- 172.21.15.16:0/3697219451 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f33b81620b0 con 0x7f33b8070030 2024-03-31T15:14:24.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:24.462+0000 7f33ae000640 1 -- 172.21.15.16:0/3697219451 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v111) ==== 80+0+0 (secure 0 0 0) 0x7f33a0055bc0 con 0x7f33b8070030 2024-03-31T15:14:24.455 INFO:teuthology.orchestra.run.smithi016.stderr:listed 0 entries 2024-03-31T15:14:24.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:24.462+0000 7f33bce00640 1 -- 172.21.15.16:0/3697219451 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f33940683b0 msgr2=0x7f339406a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:24.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:24.466+0000 7f33bce00640 1 --2- 172.21.15.16:0/3697219451 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f33940683b0 0x7f339406a870 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f33a80ad000 tx=0x7f33a80a6000 comp rx=0 tx=0).stop 2024-03-31T15:14:24.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:24.466+0000 7f33bce00640 1 -- 172.21.15.16:0/3697219451 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f33b8070030 msgr2=0x7f33b81409f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:24.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:24.466+0000 7f33bce00640 1 --2- 172.21.15.16:0/3697219451 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f33b8070030 0x7f33b81409f0 secure :-1 s=READY pgs=1692 cs=0 l=1 rev1=1 crypto rx=0x7f33a000c3c0 tx=0x7f33a000c880 comp rx=0 tx=0).stop 2024-03-31T15:14:24.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:24.466+0000 7f33bce00640 1 -- 172.21.15.16:0/3697219451 shutdown_connections 2024-03-31T15:14:24.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:24.466+0000 7f33bce00640 1 --2- 172.21.15.16:0/3697219451 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f33940683b0 0x7f339406a870 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:24.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:24.466+0000 7f33bce00640 1 --2- 172.21.15.16:0/3697219451 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f33b80cb440 0x7f33b815ca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:24.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:24.466+0000 7f33bce00640 1 --2- 172.21.15.16:0/3697219451 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f33b8070030 0x7f33b81409f0 unknown :-1 s=CLOSED pgs=1692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:24.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:24.466+0000 7f33bce00640 1 --2- 172.21.15.16:0/3697219451 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f33b806dbc0 0x7f33b8140450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:24.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:24.466+0000 7f33bce00640 1 -- 172.21.15.16:0/3697219451 >> 172.21.15.16:0/3697219451 conn(0x7f33b8067330 msgr2=0x7f33b81446a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:14:24.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:24.466+0000 7f33bce00640 1 -- 172.21.15.16:0/3697219451 shutdown_connections 2024-03-31T15:14:24.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:24.466+0000 7f33bce00640 1 -- 172.21.15.16:0/3697219451 wait complete. 2024-03-31T15:14:24.476 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-03-31T15:14:24.476 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-31T15:14:24.485 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-31T15:14:24.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:24.674+0000 7facc8e00640 1 Processor -- start 2024-03-31T15:14:24.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:24.674+0000 7facc8e00640 1 -- start start 2024-03-31T15:14:24.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:24.678+0000 7facc8e00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7facc406f920 0x7facc406fd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:24.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:24.678+0000 7facc8e00640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7facc4162010 con 0x7facc40cc740 2024-03-31T15:14:24.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:24.678+0000 7facc8e00640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7facc4070240 con 0x7facc406d490 2024-03-31T15:14:24.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:24.678+0000 7facc8e00640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7facc4162be0 con 0x7facc406f920 2024-03-31T15:14:24.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:24.678+0000 7facc1e00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7facc406f920 0x7facc406fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:24.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:24.678+0000 7facc2800640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7facc40cc740 0x7facc40ccbd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:53482/0 (socket says 172.21.15.16:53482) 2024-03-31T15:14:24.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:24.678+0000 7facc2800640 1 -- 172.21.15.16:0/1130606552 learned_addr learned my addr 172.21.15.16:0/1130606552 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:14:24.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:24.678+0000 7facc1e00640 1 -- 172.21.15.16:0/1130606552 >> v1:172.21.15.89:6789/0 conn(0x7facc406d490 legacy=0x7facc406d8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:24.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:24.678+0000 7facc1e00640 1 -- 172.21.15.16:0/1130606552 >> v1:172.21.15.16:6789/0 conn(0x7facc40cc740 legacy=0x7facc40ccbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:24.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:24.678+0000 7facc1e00640 1 -- 172.21.15.16:0/1130606552 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7facc41622b0 con 0x7facc406f920 2024-03-31T15:14:24.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:24.682+0000 7facc1e00640 1 --2- 172.21.15.16:0/1130606552 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7facc406f920 0x7facc406fd00 secure :-1 s=READY pgs=1694 cs=0 l=1 rev1=1 crypto rx=0x7facb40097c0 tx=0x7facb401c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c03077c25b74b10 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:24.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:24.682+0000 7facc1400640 1 -- 172.21.15.16:0/1130606552 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facb40b1070 con 0x7facc406f920 2024-03-31T15:14:24.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:24.682+0000 7facc1400640 1 -- 172.21.15.16:0/1130606552 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7facb40ac640 con 0x7facc406f920 2024-03-31T15:14:24.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:24.682+0000 7facc1400640 1 -- 172.21.15.16:0/1130606552 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facb40aca10 con 0x7facc406f920 2024-03-31T15:14:24.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:24.682+0000 7facc8e00640 1 -- 172.21.15.16:0/1130606552 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7facc406f920 msgr2=0x7facc406fd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:24.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:24.682+0000 7facc8e00640 1 --2- 172.21.15.16:0/1130606552 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7facc406f920 0x7facc406fd00 secure :-1 s=READY pgs=1694 cs=0 l=1 rev1=1 crypto rx=0x7facb40097c0 tx=0x7facb401c4a0 comp rx=0 tx=0).stop 2024-03-31T15:14:24.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:24.682+0000 7facc8e00640 1 -- 172.21.15.16:0/1130606552 shutdown_connections 2024-03-31T15:14:24.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:24.682+0000 7facc8e00640 1 --2- 172.21.15.16:0/1130606552 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7facc406f920 0x7facc406fd00 unknown :-1 s=CLOSED pgs=1694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:24.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:24.682+0000 7facc8e00640 1 -- 172.21.15.16:0/1130606552 >> 172.21.15.16:0/1130606552 conn(0x7facc4067330 msgr2=0x7facc4067740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:14:24.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:24.682+0000 7facc8e00640 1 -- 172.21.15.16:0/1130606552 shutdown_connections 2024-03-31T15:14:24.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:24.682+0000 7facc8e00640 1 -- 172.21.15.16:0/1130606552 wait complete. 2024-03-31T15:14:24.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:24.682+0000 7facc8e00640 1 Processor -- start 2024-03-31T15:14:24.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:24.682+0000 7facc8e00640 1 -- start start 2024-03-31T15:14:24.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:24.682+0000 7facc8e00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7facc406d490 0x7facc41464b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:24.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:24.682+0000 7facc8e00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7facc406f920 0x7facc4142530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:24.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:24.682+0000 7facc8e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7facc40cc740 0x7facc41f75b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:24.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:24.682+0000 7facc8e00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7facc416c540 con 0x7facc40cc740 2024-03-31T15:14:24.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:24.682+0000 7facc8e00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7facc416c3c0 con 0x7facc406f920 2024-03-31T15:14:24.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:24.682+0000 7facc8e00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7facc416c6c0 con 0x7facc406d490 2024-03-31T15:14:24.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:24.682+0000 7facc2800640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7facc40cc740 0x7facc41f75b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:24.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:24.686+0000 7facc2800640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7facc40cc740 0x7facc41f75b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42972/0 (socket says 172.21.15.16:42972) 2024-03-31T15:14:24.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:24.686+0000 7facc2800640 1 -- 172.21.15.16:0/3052220825 learned_addr learned my addr 172.21.15.16:0/3052220825 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:14:24.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:24.686+0000 7facc1e00640 1 --2- 172.21.15.16:0/3052220825 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7facc406d490 0x7facc41464b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:24.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:24.686+0000 7facb9400640 1 --2- 172.21.15.16:0/3052220825 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7facc406f920 0x7facc4142530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:24.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:24.686+0000 7facc2800640 1 -- 172.21.15.16:0/3052220825 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7facc406d490 msgr2=0x7facc41464b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:24.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:24.686+0000 7facc2800640 1 --2- 172.21.15.16:0/3052220825 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7facc406d490 0x7facc41464b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:24.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:24.686+0000 7facc2800640 1 -- 172.21.15.16:0/3052220825 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7facc406f920 msgr2=0x7facc4142530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:24.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:24.686+0000 7facc2800640 1 --2- 172.21.15.16:0/3052220825 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7facc406f920 0x7facc4142530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:24.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:24.686+0000 7facc2800640 1 -- 172.21.15.16:0/3052220825 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7facc4144610 con 0x7facc40cc740 2024-03-31T15:14:24.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:24.686+0000 7facb9400640 1 --2- 172.21.15.16:0/3052220825 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7facc406f920 0x7facc4142530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:14:24.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:24.686+0000 7facc2800640 1 --2- 172.21.15.16:0/3052220825 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7facc40cc740 0x7facc41f75b0 secure :-1 s=READY pgs=1695 cs=0 l=1 rev1=1 crypto rx=0x7facbc00cbc0 tx=0x7facbc00a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:24.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:24.686+0000 7facbaa00640 1 -- 172.21.15.16:0/3052220825 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facbc017070 con 0x7facc40cc740 2024-03-31T15:14:24.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:24.686+0000 7facbaa00640 1 -- 172.21.15.16:0/3052220825 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7facbc00ace0 con 0x7facc40cc740 2024-03-31T15:14:24.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:24.686+0000 7facbaa00640 1 -- 172.21.15.16:0/3052220825 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facbc012600 con 0x7facc40cc740 2024-03-31T15:14:24.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:24.686+0000 7facc8e00640 1 -- 172.21.15.16:0/3052220825 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7facc41448f0 con 0x7facc40cc740 2024-03-31T15:14:24.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:24.686+0000 7facc8e00640 1 -- 172.21.15.16:0/3052220825 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7facc4144eb0 con 0x7facc40cc740 2024-03-31T15:14:24.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:24.686+0000 7facc8e00640 1 -- 172.21.15.16:0/3052220825 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7facc4069950 con 0x7facc40cc740 2024-03-31T15:14:24.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:24.686+0000 7facbaa00640 1 -- 172.21.15.16:0/3052220825 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7facbc01a070 con 0x7facc40cc740 2024-03-31T15:14:24.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:24.690+0000 7facbaa00640 1 --2- 172.21.15.16:0/3052220825 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7faca0068380 0x7faca006a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:24.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:24.690+0000 7facbaa00640 1 -- 172.21.15.16:0/3052220825 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 5767+0+0 (secure 0 0 0) 0x7facbc086670 con 0x7facc40cc740 2024-03-31T15:14:24.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:24.690+0000 7facc1e00640 1 --2- 172.21.15.16:0/3052220825 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7faca0068380 0x7faca006a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:24.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:24.690+0000 7facc1e00640 1 --2- 172.21.15.16:0/3052220825 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7faca0068380 0x7faca006a840 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7facb40ae000 tx=0x7facb40a6070 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:24.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:24.698+0000 7facbaa00640 1 -- 172.21.15.16:0/3052220825 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7facbc056590 con 0x7facc40cc740 2024-03-31T15:14:25.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:25.022+0000 7facc8e00640 1 -- 172.21.15.16:0/3052220825 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7facc40cdc10 con 0x7facc40cc740 2024-03-31T15:14:25.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:25.026+0000 7facbaa00640 1 -- 172.21.15.16:0/3052220825 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v111) ==== 74+0+16648 (secure 0 0 0) 0x7facbc05a890 con 0x7facc40cc740 2024-03-31T15:14:25.018 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:14:25.018 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":111,"fsid":"286eb5da-fb47-4550-b290-d9519c13b9a5","created":"2024-03-31T15:03:11.438952+0000","modified":"2024-03-31T15:14:19.306886+0000","last_up_change":"2024-03-31T15:03:22.885535+0000","last_in_change":"2024-03-31T15:03:13.941667+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-31T15:03:22.550787+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-03-31T15:14:06.417304+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"108","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-03-31T15:14:07.622562+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"109","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"92a66450-ef19-40d8-ac3d-d760b23b0bda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6801","nonce":932712244}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6803","nonce":932712244}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6807","nonce":932712244}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6805","nonce":932712244}]},"public_addr":"172.21.15.16:6801/932712244","cluster_addr":"172.21.15.16:6803/932712244","heartbeat_back_addr":"172.21.15.16:6807/932712244","heartbeat_front_addr":"172.21.15.16:6805/932712244","state":["exists","up"]},{"osd":1,"uuid":"da9d1f2b-1968-4959-b843-faa231988e92","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6810","nonce":1398737533}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6811","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6813","nonce":1398737533}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6821","nonce":1398737533}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6815","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6816","nonce":1398737533}]},"public_addr":"172.21.15.16:6810/1398737533","cluster_addr":"172.21.15.16:6813/1398737533","heartbeat_back_addr":"172.21.15.16:6821/1398737533","heartbeat_front_addr":"172.21.15.16:6816/1398737533","state":["exists","up"]},{"osd":2,"uuid":"01a73662-9a21-4b1f-bcbb-42b9636a4638","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6819","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6822","nonce":3018526646}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6825","nonce":3018526646}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6831","nonce":3018526646}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6827","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6829","nonce":3018526646}]},"public_addr":"172.21.15.16:6822/3018526646","cluster_addr":"172.21.15.16:6825/3018526646","heartbeat_back_addr":"172.21.15.16:6831/3018526646","heartbeat_front_addr":"172.21.15.16:6829/3018526646","state":["exists","up"]},{"osd":3,"uuid":"c7771fd3-e18e-4500-ac38-23cf10ebcd75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6809","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6812","nonce":3360352974}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6817","nonce":3360352974}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6828","nonce":3360352974}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6823","nonce":3360352974}]},"public_addr":"172.21.15.16:6812/3360352974","cluster_addr":"172.21.15.16:6817/3360352974","heartbeat_back_addr":"172.21.15.16:6828/3360352974","heartbeat_front_addr":"172.21.15.16:6823/3360352974","state":["exists","up"]},{"osd":4,"uuid":"88260730-1482-4fbd-a40f-cb1124e9df4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6801","nonce":1856844856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6803","nonce":1856844856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6807","nonce":1856844856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6805","nonce":1856844856}]},"public_addr":"172.21.15.89:6801/1856844856","cluster_addr":"172.21.15.89:6803/1856844856","heartbeat_back_addr":"172.21.15.89:6807/1856844856","heartbeat_front_addr":"172.21.15.89:6805/1856844856","state":["exists","up"]},{"osd":5,"uuid":"4cb2c00c-fae0-4dd8-9d27-ec4f0093ccf7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6814","nonce":50395527}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6817","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6820","nonce":50395527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6831","nonce":50395527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6823","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6826","nonce":50395527}]},"public_addr":"172.21.15.89:6814/50395527","cluster_addr":"172.21.15.89:6820/50395527","heartbeat_back_addr":"172.21.15.89:6831/50395527","heartbeat_front_addr":"172.21.15.89:6826/50395527","state":["exists","up"]},{"osd":6,"uuid":"81ce0c19-2554-4cae-acbe-66a05e9e9b9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6809","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6812","nonce":3470471752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6815","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6818","nonce":3470471752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6827","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6829","nonce":3470471752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6821","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6824","nonce":3470471752}]},"public_addr":"172.21.15.89:6812/3470471752","cluster_addr":"172.21.15.89:6818/3470471752","heartbeat_back_addr":"172.21.15.89:6829/3470471752","heartbeat_front_addr":"172.21.15.89:6824/3470471752","state":["exists","up"]},{"osd":7,"uuid":"85e4313b-d37d-442e-8042-153bf20a847d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6811","nonce":1796778630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6813","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6816","nonce":1796778630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6825","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6828","nonce":1796778630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6819","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6822","nonce":1796778630}]},"public_addr":"172.21.15.89:6811/1796778630","cluster_addr":"172.21.15.89:6816/1796778630","heartbeat_back_addr":"172.21.15.89:6828/1796778630","heartbeat_front_addr":"172.21.15.89:6822/1796778630","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.439088+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.529970+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.680512+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:21.044161+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.582111+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.536558+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.409522+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.590229+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"17.1","mappings":[{"from":0,"to":5}]},{"pgid":"17.21","mappings":[{"from":0,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-31T15:14:25.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:25.026+0000 7facc8e00640 1 -- 172.21.15.16:0/3052220825 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7faca0068380 msgr2=0x7faca006a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:25.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:25.026+0000 7facc8e00640 1 --2- 172.21.15.16:0/3052220825 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7faca0068380 0x7faca006a840 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7facb40ae000 tx=0x7facb40a6070 comp rx=0 tx=0).stop 2024-03-31T15:14:25.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:25.026+0000 7facc8e00640 1 -- 172.21.15.16:0/3052220825 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7facc40cc740 msgr2=0x7facc41f75b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:25.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:25.026+0000 7facc8e00640 1 --2- 172.21.15.16:0/3052220825 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7facc40cc740 0x7facc41f75b0 secure :-1 s=READY pgs=1695 cs=0 l=1 rev1=1 crypto rx=0x7facbc00cbc0 tx=0x7facbc00a430 comp rx=0 tx=0).stop 2024-03-31T15:14:25.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:25.026+0000 7facc8e00640 1 -- 172.21.15.16:0/3052220825 shutdown_connections 2024-03-31T15:14:25.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:25.026+0000 7facc8e00640 1 --2- 172.21.15.16:0/3052220825 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7faca0068380 0x7faca006a840 unknown :-1 s=CLOSED pgs=719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:25.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:25.026+0000 7facc8e00640 1 --2- 172.21.15.16:0/3052220825 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7facc40cc740 0x7facc41f75b0 unknown :-1 s=CLOSED pgs=1695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:25.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:25.026+0000 7facc8e00640 1 --2- 172.21.15.16:0/3052220825 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7facc406f920 0x7facc4142530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:25.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:25.026+0000 7facc8e00640 1 --2- 172.21.15.16:0/3052220825 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7facc406d490 0x7facc41464b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:25.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:25.026+0000 7facc8e00640 1 -- 172.21.15.16:0/3052220825 >> 172.21.15.16:0/3052220825 conn(0x7facc4067330 msgr2=0x7facc40d1160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:14:25.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:25.026+0000 7facc8e00640 1 -- 172.21.15.16:0/3052220825 shutdown_connections 2024-03-31T15:14:25.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:25.026+0000 7facc8e00640 1 -- 172.21.15.16:0/3052220825 wait complete. 2024-03-31T15:14:25.040 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-31T15:03:22.550787+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 16, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-31T15:14:06.417304+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '108', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 17, 'pool_name': 'cephfs_data', 'create_time': '2024-03-31T15:14:07.622562+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '109', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-31T15:14:25.040 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-31T15:14:25.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:25.190+0000 7fb32cc00640 1 Processor -- start 2024-03-31T15:14:25.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:25.190+0000 7fb32cc00640 1 -- start start 2024-03-31T15:14:25.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:25.190+0000 7fb32cc00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb32806f920 0x7fb32806fd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:25.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:25.190+0000 7fb32cc00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb3280d0a60 0x7fb3280cea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:25.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:25.190+0000 7fb32cc00640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb328070240 con 0x7fb3280cef80 2024-03-31T15:14:25.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:25.190+0000 7fb32cc00640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7fb32813f670 con 0x7fb32806f920 2024-03-31T15:14:25.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:25.190+0000 7fb32cc00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fb32815ca70 con 0x7fb3280d0a60 2024-03-31T15:14:25.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:25.190+0000 7fb325e00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb32806f920 0x7fb32806fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:25.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:25.194+0000 7fb325e00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb32806f920 0x7fb32806fd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42974/0 (socket says 172.21.15.16:42974) 2024-03-31T15:14:25.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:25.194+0000 7fb325e00640 1 -- 172.21.15.16:0/4094010788 learned_addr learned my addr 172.21.15.16:0/4094010788 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:14:25.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:25.194+0000 7fb31d400640 1 --2- 172.21.15.16:0/4094010788 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb3280d0a60 0x7fb3280cea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:25.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:25.194+0000 7fb325e00640 1 -- 172.21.15.16:0/4094010788 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb3280d0a60 msgr2=0x7fb3280cea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:14:25.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:25.194+0000 7fb325e00640 1 --2- 172.21.15.16:0/4094010788 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb3280d0a60 0x7fb3280cea40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:25.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:25.194+0000 7fb325e00640 1 -- 172.21.15.16:0/4094010788 >> v1:172.21.15.89:6789/0 conn(0x7fb3280cef80 legacy=0x7fb328070e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:25.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:25.194+0000 7fb325e00640 1 -- 172.21.15.16:0/4094010788 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb32815cd50 con 0x7fb32806f920 2024-03-31T15:14:25.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:25.194+0000 7fb325e00640 1 --2- 172.21.15.16:0/4094010788 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb32806f920 0x7fb32806fd00 secure :-1 s=READY pgs=1696 cs=0 l=1 rev1=1 crypto rx=0x7fb318009870 tx=0x7fb31801c560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a958172a0753eb09 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:25.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:25.194+0000 7fb325400640 1 -- 172.21.15.16:0/4094010788 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3180b0070 con 0x7fb32806f920 2024-03-31T15:14:25.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:25.194+0000 7fb325400640 1 -- 172.21.15.16:0/4094010788 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb3180ab600 con 0x7fb32806f920 2024-03-31T15:14:25.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:25.194+0000 7fb325400640 1 -- 172.21.15.16:0/4094010788 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3180ab9d0 con 0x7fb32806f920 2024-03-31T15:14:25.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:25.194+0000 7fb32cc00640 1 -- 172.21.15.16:0/4094010788 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb32806f920 msgr2=0x7fb32806fd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:25.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:25.194+0000 7fb32cc00640 1 --2- 172.21.15.16:0/4094010788 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb32806f920 0x7fb32806fd00 secure :-1 s=READY pgs=1696 cs=0 l=1 rev1=1 crypto rx=0x7fb318009870 tx=0x7fb31801c560 comp rx=0 tx=0).stop 2024-03-31T15:14:25.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:25.198+0000 7fb32cc00640 1 -- 172.21.15.16:0/4094010788 shutdown_connections 2024-03-31T15:14:25.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:25.198+0000 7fb32cc00640 1 --2- 172.21.15.16:0/4094010788 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb3280d0a60 0x7fb3280cea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:25.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:25.198+0000 7fb32cc00640 1 --2- 172.21.15.16:0/4094010788 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb32806f920 0x7fb32806fd00 unknown :-1 s=CLOSED pgs=1696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:25.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:25.198+0000 7fb32cc00640 1 -- 172.21.15.16:0/4094010788 >> 172.21.15.16:0/4094010788 conn(0x7fb328067930 msgr2=0x7fb3280b3ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:14:25.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:25.198+0000 7fb32cc00640 1 -- 172.21.15.16:0/4094010788 shutdown_connections 2024-03-31T15:14:25.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:25.198+0000 7fb32cc00640 1 -- 172.21.15.16:0/4094010788 wait complete. 2024-03-31T15:14:25.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:25.198+0000 7fb32cc00640 1 Processor -- start 2024-03-31T15:14:25.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:25.198+0000 7fb32cc00640 1 -- start start 2024-03-31T15:14:25.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:25.198+0000 7fb32cc00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb32806f920 0x7fb32815e260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:25.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:25.198+0000 7fb32cc00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb3280cef80 0x7fb3280ce680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:25.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:25.198+0000 7fb32cc00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb3280d0a60 0x7fb328159080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:25.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:25.198+0000 7fb32cc00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fb3281690b0 con 0x7fb3280cef80 2024-03-31T15:14:25.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:25.198+0000 7fb32cc00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fb3280cf390 con 0x7fb32806f920 2024-03-31T15:14:25.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:25.198+0000 7fb32cc00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fb328169230 con 0x7fb3280d0a60 2024-03-31T15:14:25.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:25.198+0000 7fb31d400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb3280cef80 0x7fb3280ce680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:25.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:25.198+0000 7fb325e00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb32806f920 0x7fb32815e260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:25.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:25.198+0000 7fb325e00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb32806f920 0x7fb32815e260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.16:45418/0 (socket says 172.21.15.16:45418) 2024-03-31T15:14:25.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:25.198+0000 7fb325e00640 1 -- 172.21.15.16:0/1104366200 learned_addr learned my addr 172.21.15.16:0/1104366200 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:14:25.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:25.198+0000 7fb326800640 1 --2- 172.21.15.16:0/1104366200 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb3280d0a60 0x7fb328159080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:25.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:25.198+0000 7fb31d400640 1 -- 172.21.15.16:0/1104366200 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb3280d0a60 msgr2=0x7fb328159080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:25.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:25.198+0000 7fb31d400640 1 --2- 172.21.15.16:0/1104366200 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb3280d0a60 0x7fb328159080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:25.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:25.198+0000 7fb31d400640 1 -- 172.21.15.16:0/1104366200 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb32806f920 msgr2=0x7fb32815e260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:25.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:25.198+0000 7fb31d400640 1 --2- 172.21.15.16:0/1104366200 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb32806f920 0x7fb32815e260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:25.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:25.198+0000 7fb31d400640 1 -- 172.21.15.16:0/1104366200 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb32815e8f0 con 0x7fb3280cef80 2024-03-31T15:14:25.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:25.198+0000 7fb325e00640 1 --2- 172.21.15.16:0/1104366200 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb32806f920 0x7fb32815e260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:14:25.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:25.202+0000 7fb31d400640 1 --2- 172.21.15.16:0/1104366200 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb3280cef80 0x7fb3280ce680 secure :-1 s=READY pgs=1697 cs=0 l=1 rev1=1 crypto rx=0x7fb31000e870 tx=0x7fb31000ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:25.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:25.202+0000 7fb31ea00640 1 -- 172.21.15.16:0/1104366200 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb310017070 con 0x7fb3280cef80 2024-03-31T15:14:25.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:25.202+0000 7fb31ea00640 1 -- 172.21.15.16:0/1104366200 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb31000c940 con 0x7fb3280cef80 2024-03-31T15:14:25.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:25.202+0000 7fb32cc00640 1 -- 172.21.15.16:0/1104366200 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb32815ebd0 con 0x7fb3280cef80 2024-03-31T15:14:25.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:25.202+0000 7fb31ea00640 1 -- 172.21.15.16:0/1104366200 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb31000ccf0 con 0x7fb3280cef80 2024-03-31T15:14:25.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:25.202+0000 7fb32cc00640 1 -- 172.21.15.16:0/1104366200 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb32815f190 con 0x7fb3280cef80 2024-03-31T15:14:25.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:25.202+0000 7fb32cc00640 1 -- 172.21.15.16:0/1104366200 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb328069950 con 0x7fb3280cef80 2024-03-31T15:14:25.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:25.202+0000 7fb31ea00640 1 -- 172.21.15.16:0/1104366200 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fb31001a070 con 0x7fb3280cef80 2024-03-31T15:14:25.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:25.202+0000 7fb31ea00640 1 --2- 172.21.15.16:0/1104366200 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fb304068380 0x7fb30406a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:25.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:25.202+0000 7fb31ea00640 1 -- 172.21.15.16:0/1104366200 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 5767+0+0 (secure 0 0 0) 0x7fb310086c30 con 0x7fb3280cef80 2024-03-31T15:14:25.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:25.202+0000 7fb325e00640 1 --2- 172.21.15.16:0/1104366200 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fb304068380 0x7fb30406a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:25.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:25.206+0000 7fb325e00640 1 --2- 172.21.15.16:0/1104366200 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fb304068380 0x7fb30406a840 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7fb318009870 tx=0x7fb3180a6000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:25.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:25.206+0000 7fb31ea00640 1 -- 172.21.15.16:0/1104366200 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fb31008d080 con 0x7fb3280cef80 2024-03-31T15:14:25.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:25.382+0000 7fb32cc00640 1 -- 172.21.15.16:0/1104366200 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fb3280b4a20 con 0x7fb3280cef80 2024-03-31T15:14:25.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:25.382+0000 7fb31ea00640 1 -- 172.21.15.16:0/1104366200 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v111) ==== 93+0+10 (secure 0 0 0) 0x7fb310056bd0 con 0x7fb3280cef80 2024-03-31T15:14:25.374 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2024-03-31T15:14:25.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:25.382+0000 7fb32cc00640 1 -- 172.21.15.16:0/1104366200 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fb304068380 msgr2=0x7fb30406a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:25.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:25.382+0000 7fb32cc00640 1 --2- 172.21.15.16:0/1104366200 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fb304068380 0x7fb30406a840 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7fb318009870 tx=0x7fb3180a6000 comp rx=0 tx=0).stop 2024-03-31T15:14:25.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:25.382+0000 7fb32cc00640 1 -- 172.21.15.16:0/1104366200 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb3280cef80 msgr2=0x7fb3280ce680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:25.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:25.382+0000 7fb32cc00640 1 --2- 172.21.15.16:0/1104366200 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb3280cef80 0x7fb3280ce680 secure :-1 s=READY pgs=1697 cs=0 l=1 rev1=1 crypto rx=0x7fb31000e870 tx=0x7fb31000ed30 comp rx=0 tx=0).stop 2024-03-31T15:14:25.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:25.382+0000 7fb32cc00640 1 -- 172.21.15.16:0/1104366200 shutdown_connections 2024-03-31T15:14:25.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:25.382+0000 7fb32cc00640 1 --2- 172.21.15.16:0/1104366200 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fb304068380 0x7fb30406a840 unknown :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:25.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:25.382+0000 7fb32cc00640 1 --2- 172.21.15.16:0/1104366200 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb3280d0a60 0x7fb328159080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:25.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:25.382+0000 7fb32cc00640 1 --2- 172.21.15.16:0/1104366200 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb3280cef80 0x7fb3280ce680 unknown :-1 s=CLOSED pgs=1697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:25.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:25.382+0000 7fb32cc00640 1 --2- 172.21.15.16:0/1104366200 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb32806f920 0x7fb32815e260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:25.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:25.382+0000 7fb32cc00640 1 -- 172.21.15.16:0/1104366200 >> 172.21.15.16:0/1104366200 conn(0x7fb328067930 msgr2=0x7fb328070e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:14:25.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:25.382+0000 7fb32cc00640 1 -- 172.21.15.16:0/1104366200 shutdown_connections 2024-03-31T15:14:25.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:25.382+0000 7fb32cc00640 1 -- 172.21.15.16:0/1104366200 wait complete. 2024-03-31T15:14:25.396 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-31T15:14:25.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:25.546+0000 7fca1ae00640 1 Processor -- start 2024-03-31T15:14:25.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:25.546+0000 7fca1ae00640 1 -- start start 2024-03-31T15:14:25.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:25.546+0000 7fca1ae00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fca141411b0 0x7fca1406f920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:25.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:25.546+0000 7fca1ae00640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fca14144980 con 0x7fca14141590 2024-03-31T15:14:25.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:25.546+0000 7fca1ae00640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fca14161fe0 con 0x7fca14144340 2024-03-31T15:14:25.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:25.546+0000 7fca1ae00640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7fca14162910 con 0x7fca141411b0 2024-03-31T15:14:25.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:25.546+0000 7fca0b400640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fca14141590 0x7fca14143c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:53494/0 (socket says 172.21.15.16:53494) 2024-03-31T15:14:25.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:25.546+0000 7fca0b400640 1 -- 172.21.15.16:0/180758335 learned_addr learned my addr 172.21.15.16:0/180758335 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:14:25.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:25.550+0000 7fca13e00640 1 --2- 172.21.15.16:0/180758335 >> v2:172.21.15.89:3300/0 conn(0x7fca141411b0 0x7fca1406f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:25.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:25.550+0000 7fca13400640 1 -- 172.21.15.16:0/180758335 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1611245257 0 0) 0x7fca14144980 con 0x7fca14141590 2024-03-31T15:14:25.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:25.550+0000 7fca13400640 1 -- 172.21.15.16:0/180758335 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc9f8003530 con 0x7fca14141590 2024-03-31T15:14:25.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:25.550+0000 7fca13400640 1 -- 172.21.15.16:0/180758335 <== mon.1 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1399382437 0 0) 0x7fca14161fe0 con 0x7fca14144340 2024-03-31T15:14:25.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:25.550+0000 7fca13400640 1 -- 172.21.15.16:0/180758335 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fca14144980 con 0x7fca14144340 2024-03-31T15:14:25.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:25.550+0000 7fca13e00640 1 -- 172.21.15.16:0/180758335 >> v1:172.21.15.89:6789/0 conn(0x7fca14144340 legacy=0x7fca1415e900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:25.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:25.550+0000 7fca13e00640 1 -- 172.21.15.16:0/180758335 >> v1:172.21.15.16:6789/0 conn(0x7fca14141590 legacy=0x7fca14143c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:25.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:25.550+0000 7fca13e00640 1 -- 172.21.15.16:0/180758335 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca141622c0 con 0x7fca141411b0 2024-03-31T15:14:25.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:25.554+0000 7fca13e00640 1 --2- 172.21.15.16:0/180758335 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fca141411b0 0x7fca1406f920 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7fca040097c0 tx=0x7fca0401c4a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1a7b430e3af22a82 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:25.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:25.554+0000 7fca13400640 1 -- 172.21.15.16:0/180758335 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca040b0070 con 0x7fca141411b0 2024-03-31T15:14:25.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:25.554+0000 7fca13400640 1 -- 172.21.15.16:0/180758335 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fca040ab600 con 0x7fca141411b0 2024-03-31T15:14:25.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:25.554+0000 7fca13400640 1 -- 172.21.15.16:0/180758335 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca040ab9d0 con 0x7fca141411b0 2024-03-31T15:14:25.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:25.554+0000 7fca1ae00640 1 -- 172.21.15.16:0/180758335 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fca141411b0 msgr2=0x7fca1406f920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:25.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:25.554+0000 7fca1ae00640 1 --2- 172.21.15.16:0/180758335 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fca141411b0 0x7fca1406f920 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7fca040097c0 tx=0x7fca0401c4a0 comp rx=0 tx=0).stop 2024-03-31T15:14:25.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:25.554+0000 7fca1ae00640 1 -- 172.21.15.16:0/180758335 shutdown_connections 2024-03-31T15:14:25.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:25.554+0000 7fca1ae00640 1 --2- 172.21.15.16:0/180758335 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fca141411b0 0x7fca1406f920 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:25.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:25.554+0000 7fca1ae00640 1 -- 172.21.15.16:0/180758335 >> 172.21.15.16:0/180758335 conn(0x7fca14067930 msgr2=0x7fca140b3ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:14:25.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:25.554+0000 7fca1ae00640 1 -- 172.21.15.16:0/180758335 shutdown_connections 2024-03-31T15:14:25.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:25.554+0000 7fca1ae00640 1 -- 172.21.15.16:0/180758335 wait complete. 2024-03-31T15:14:25.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:25.558+0000 7fca1ae00640 1 Processor -- start 2024-03-31T15:14:25.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:25.558+0000 7fca1ae00640 1 -- start start 2024-03-31T15:14:25.554 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:25.558+0000 7fca1ae00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fca141411b0 0x7fca141436b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:25.554 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:25.558+0000 7fca1ae00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fca14141590 0x7fca14152df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:25.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:25.558+0000 7fca1ae00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fca14144340 0x7fca14153330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:25.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:25.558+0000 7fca1ae00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fca1416c680 con 0x7fca14141590 2024-03-31T15:14:25.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:25.558+0000 7fca1ae00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fca1416c500 con 0x7fca14144340 2024-03-31T15:14:25.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:25.558+0000 7fca1ae00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fca1416c800 con 0x7fca141411b0 2024-03-31T15:14:25.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:25.558+0000 7fca0b400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fca14141590 0x7fca14152df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:25.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:25.558+0000 7fca0b400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fca14141590 0x7fca14152df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42992/0 (socket says 172.21.15.16:42992) 2024-03-31T15:14:25.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:25.558+0000 7fca0b400640 1 -- 172.21.15.16:0/288285862 learned_addr learned my addr 172.21.15.16:0/288285862 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:14:25.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:25.558+0000 7fca13e00640 1 --2- 172.21.15.16:0/288285862 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fca141411b0 0x7fca141436b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:25.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:25.558+0000 7fca18e00640 1 --2- 172.21.15.16:0/288285862 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fca14144340 0x7fca14153330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:25.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:25.562+0000 7fca0b400640 1 -- 172.21.15.16:0/288285862 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fca141411b0 msgr2=0x7fca141436b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:25.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:25.562+0000 7fca0b400640 1 --2- 172.21.15.16:0/288285862 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fca141411b0 0x7fca141436b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:25.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:25.562+0000 7fca0b400640 1 -- 172.21.15.16:0/288285862 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fca14144340 msgr2=0x7fca14153330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:25.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:25.562+0000 7fca0b400640 1 --2- 172.21.15.16:0/288285862 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fca14144340 0x7fca14153330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:25.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:25.562+0000 7fca0b400640 1 -- 172.21.15.16:0/288285862 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca1413dae0 con 0x7fca14141590 2024-03-31T15:14:25.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:25.562+0000 7fca0b400640 1 --2- 172.21.15.16:0/288285862 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fca14141590 0x7fca14152df0 secure :-1 s=READY pgs=1699 cs=0 l=1 rev1=1 crypto rx=0x7fc9fc00c4d0 tx=0x7fc9fc00c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:25.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:25.562+0000 7fca10c00640 1 -- 172.21.15.16:0/288285862 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9fc017070 con 0x7fca14141590 2024-03-31T15:14:25.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:25.562+0000 7fca10c00640 1 -- 172.21.15.16:0/288285862 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc9fc005cf0 con 0x7fca14141590 2024-03-31T15:14:25.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:25.562+0000 7fca10c00640 1 -- 172.21.15.16:0/288285862 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9fc0072b0 con 0x7fca14141590 2024-03-31T15:14:25.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:25.562+0000 7fca1ae00640 1 -- 172.21.15.16:0/288285862 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fca1413ddc0 con 0x7fca14141590 2024-03-31T15:14:25.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:25.562+0000 7fca1ae00640 1 -- 172.21.15.16:0/288285862 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fca1413e380 con 0x7fca14141590 2024-03-31T15:14:25.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:25.562+0000 7fca1ae00640 1 -- 172.21.15.16:0/288285862 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fca14069950 con 0x7fca14141590 2024-03-31T15:14:25.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:25.562+0000 7fca10c00640 1 -- 172.21.15.16:0/288285862 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fc9fc01a070 con 0x7fca14141590 2024-03-31T15:14:25.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:25.562+0000 7fca10c00640 1 --2- 172.21.15.16:0/288285862 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fc9f0068380 0x7fc9f006a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:25.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:25.562+0000 7fca10c00640 1 -- 172.21.15.16:0/288285862 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 5767+0+0 (secure 0 0 0) 0x7fc9fc085ec0 con 0x7fca14141590 2024-03-31T15:14:25.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:25.566+0000 7fca13e00640 1 --2- 172.21.15.16:0/288285862 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fc9f0068380 0x7fc9f006a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:25.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:25.566+0000 7fca13e00640 1 --2- 172.21.15.16:0/288285862 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fc9f0068380 0x7fc9f006a840 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7fca040ad000 tx=0x7fca040a6000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:25.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:25.566+0000 7fca10c00640 1 -- 172.21.15.16:0/288285862 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fc9fc056320 con 0x7fca14141590 2024-03-31T15:14:25.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:25.730+0000 7fca1ae00640 1 -- 172.21.15.16:0/288285862 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fca140b4a20 con 0x7fca14141590 2024-03-31T15:14:25.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:25.730+0000 7fca10c00640 1 -- 172.21.15.16:0/288285862 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v111) ==== 104+0+11 (secure 0 0 0) 0x7fc9fc055c20 con 0x7fca14141590 2024-03-31T15:14:25.725 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2024-03-31T15:14:25.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:25.734+0000 7fca1ae00640 1 -- 172.21.15.16:0/288285862 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fc9f0068380 msgr2=0x7fc9f006a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:25.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:25.734+0000 7fca1ae00640 1 --2- 172.21.15.16:0/288285862 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fc9f0068380 0x7fc9f006a840 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7fca040ad000 tx=0x7fca040a6000 comp rx=0 tx=0).stop 2024-03-31T15:14:25.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:25.734+0000 7fca1ae00640 1 -- 172.21.15.16:0/288285862 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fca14141590 msgr2=0x7fca14152df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:25.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:25.734+0000 7fca1ae00640 1 --2- 172.21.15.16:0/288285862 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fca14141590 0x7fca14152df0 secure :-1 s=READY pgs=1699 cs=0 l=1 rev1=1 crypto rx=0x7fc9fc00c4d0 tx=0x7fc9fc00c990 comp rx=0 tx=0).stop 2024-03-31T15:14:25.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:25.734+0000 7fca1ae00640 1 -- 172.21.15.16:0/288285862 shutdown_connections 2024-03-31T15:14:25.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:25.734+0000 7fca1ae00640 1 --2- 172.21.15.16:0/288285862 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fc9f0068380 0x7fc9f006a840 unknown :-1 s=CLOSED pgs=721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:25.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:25.734+0000 7fca1ae00640 1 --2- 172.21.15.16:0/288285862 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fca14144340 0x7fca14153330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:25.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:25.734+0000 7fca1ae00640 1 --2- 172.21.15.16:0/288285862 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fca14141590 0x7fca14152df0 unknown :-1 s=CLOSED pgs=1699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:25.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:25.734+0000 7fca1ae00640 1 --2- 172.21.15.16:0/288285862 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fca141411b0 0x7fca141436b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:25.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:25.734+0000 7fca1ae00640 1 -- 172.21.15.16:0/288285862 >> 172.21.15.16:0/288285862 conn(0x7fca14067930 msgr2=0x7fca1406dd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:14:25.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:25.734+0000 7fca1ae00640 1 -- 172.21.15.16:0/288285862 shutdown_connections 2024-03-31T15:14:25.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:25.734+0000 7fca1ae00640 1 -- 172.21.15.16:0/288285862 wait complete. 2024-03-31T15:14:25.746 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-31T15:14:25.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:25.897+0000 7f86a3000640 1 Processor -- start 2024-03-31T15:14:25.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:25.897+0000 7f86a3000640 1 -- start start 2024-03-31T15:14:25.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:25.897+0000 7f86a3000640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f869c1567e0 0x7f869c072c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:25.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:25.897+0000 7f86a3000640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f869c06dbe0 0x7f869c071550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:25.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:25.897+0000 7f86a3000640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f869c15c7e0 con 0x7f869c0731c0 2024-03-31T15:14:25.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:25.897+0000 7f86a3000640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f869c0736b0 con 0x7f869c1567e0 2024-03-31T15:14:25.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:25.897+0000 7f86a3000640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f869c15caa0 con 0x7f869c06dbe0 2024-03-31T15:14:25.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:25.897+0000 7f869be00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f869c1567e0 0x7f869c072c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:25.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:25.897+0000 7f869b400640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f869c0731c0 0x7f869c06d4d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.16:51054/0 (socket says 172.21.15.16:51054) 2024-03-31T15:14:25.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:25.897+0000 7f869b400640 1 -- 172.21.15.16:0/3610129662 learned_addr learned my addr 172.21.15.16:0/3610129662 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:14:25.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:25.897+0000 7f86a1000640 1 --2- 172.21.15.16:0/3610129662 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f869c06dbe0 0x7f869c071550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:25.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:25.897+0000 7f869aa00640 1 -- 172.21.15.16:0/3610129662 <== mon.1 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3243494990 0 0) 0x7f869c15c7e0 con 0x7f869c0731c0 2024-03-31T15:14:25.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:25.897+0000 7f869aa00640 1 -- 172.21.15.16:0/3610129662 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8680003530 con 0x7f869c0731c0 2024-03-31T15:14:25.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:25.897+0000 7f86a1000640 1 -- 172.21.15.16:0/3610129662 >> v2:172.21.15.89:3300/0 conn(0x7f869c1567e0 msgr2=0x7f869c072c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:14:25.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:25.897+0000 7f86a1000640 1 --2- 172.21.15.16:0/3610129662 >> v2:172.21.15.89:3300/0 conn(0x7f869c1567e0 0x7f869c072c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:25.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:25.897+0000 7f86a1000640 1 -- 172.21.15.16:0/3610129662 >> v1:172.21.15.89:6789/0 conn(0x7f869c0731c0 legacy=0x7f869c06d4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:25.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:25.897+0000 7f86a1000640 1 -- 172.21.15.16:0/3610129662 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f869c15cd80 con 0x7f869c06dbe0 2024-03-31T15:14:25.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:25.897+0000 7f869be00640 1 --2- 172.21.15.16:0/3610129662 >> v2:172.21.15.89:3300/0 conn(0x7f869c1567e0 0x7f869c072c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:14:25.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:25.901+0000 7f86a1000640 1 --2- 172.21.15.16:0/3610129662 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f869c06dbe0 0x7f869c071550 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f8694009d90 tx=0x7f869401c970 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9990f8e9934b630 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:25.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:25.905+0000 7f869aa00640 1 -- 172.21.15.16:0/3610129662 <== mon.2 v2:172.21.15.89:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86940aa610 con 0x7f869c06dbe0 2024-03-31T15:14:25.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:25.905+0000 7f869aa00640 1 -- 172.21.15.16:0/3610129662 <== mon.2 v2:172.21.15.89:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f86940af070 con 0x7f869c06dbe0 2024-03-31T15:14:25.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:25.905+0000 7f869aa00640 1 -- 172.21.15.16:0/3610129662 <== mon.2 v2:172.21.15.89:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86940aa9d0 con 0x7f869c06dbe0 2024-03-31T15:14:25.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:25.905+0000 7f86a3000640 1 -- 172.21.15.16:0/3610129662 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f869c06dbe0 msgr2=0x7f869c071550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:25.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:25.905+0000 7f86a3000640 1 --2- 172.21.15.16:0/3610129662 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f869c06dbe0 0x7f869c071550 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f8694009d90 tx=0x7f869401c970 comp rx=0 tx=0).stop 2024-03-31T15:14:25.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:25.905+0000 7f86a3000640 1 -- 172.21.15.16:0/3610129662 shutdown_connections 2024-03-31T15:14:25.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:25.905+0000 7f86a3000640 1 --2- 172.21.15.16:0/3610129662 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f869c06dbe0 0x7f869c071550 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:25.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:25.905+0000 7f86a3000640 1 --2- 172.21.15.16:0/3610129662 >> v2:172.21.15.89:3300/0 conn(0x7f869c1567e0 0x7f869c072c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:25.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:25.905+0000 7f86a3000640 1 -- 172.21.15.16:0/3610129662 >> 172.21.15.16:0/3610129662 conn(0x7f869c06ee40 msgr2=0x7f869c06b620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:14:25.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:25.905+0000 7f86a3000640 1 -- 172.21.15.16:0/3610129662 shutdown_connections 2024-03-31T15:14:25.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:25.905+0000 7f86a3000640 1 -- 172.21.15.16:0/3610129662 wait complete. 2024-03-31T15:14:25.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:25.905+0000 7f86a3000640 1 Processor -- start 2024-03-31T15:14:25.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:25.905+0000 7f86a3000640 1 -- start start 2024-03-31T15:14:25.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:25.905+0000 7f86a3000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f869c06dbe0 0x7f869c150410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:25.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:25.905+0000 7f86a3000640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f869c0731c0 0x7f869c1479d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:25.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:25.905+0000 7f86a3000640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f869c1567e0 0x7f869c15b580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:25.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:25.905+0000 7f86a3000640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f869c169170 con 0x7f869c06dbe0 2024-03-31T15:14:25.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:25.905+0000 7f86a3000640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f869c15d770 con 0x7f869c0731c0 2024-03-31T15:14:25.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:25.905+0000 7f86a3000640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f869c1692f0 con 0x7f869c1567e0 2024-03-31T15:14:25.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:25.905+0000 7f869be00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f869c06dbe0 0x7f869c150410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:25.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:25.909+0000 7f869be00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f869c06dbe0 0x7f869c150410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43004/0 (socket says 172.21.15.16:43004) 2024-03-31T15:14:25.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:25.909+0000 7f869be00640 1 -- 172.21.15.16:0/2956040935 learned_addr learned my addr 172.21.15.16:0/2956040935 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:14:25.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:25.909+0000 7f869b400640 1 --2- 172.21.15.16:0/2956040935 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f869c0731c0 0x7f869c1479d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:25.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:25.909+0000 7f86a1000640 1 --2- 172.21.15.16:0/2956040935 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f869c1567e0 0x7f869c15b580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:25.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:25.909+0000 7f869be00640 1 -- 172.21.15.16:0/2956040935 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f869c1567e0 msgr2=0x7f869c15b580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:25.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:25.909+0000 7f869be00640 1 --2- 172.21.15.16:0/2956040935 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f869c1567e0 0x7f869c15b580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:25.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:25.909+0000 7f869be00640 1 -- 172.21.15.16:0/2956040935 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f869c0731c0 msgr2=0x7f869c1479d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:25.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:25.909+0000 7f869be00640 1 --2- 172.21.15.16:0/2956040935 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f869c0731c0 0x7f869c1479d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:25.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:25.909+0000 7f869be00640 1 -- 172.21.15.16:0/2956040935 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f869c0b4700 con 0x7f869c06dbe0 2024-03-31T15:14:25.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:25.909+0000 7f869b400640 1 --2- 172.21.15.16:0/2956040935 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f869c0731c0 0x7f869c1479d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:14:25.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:25.909+0000 7f869be00640 1 --2- 172.21.15.16:0/2956040935 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f869c06dbe0 0x7f869c150410 secure :-1 s=READY pgs=1700 cs=0 l=1 rev1=1 crypto rx=0x7f868c00e980 tx=0x7f868c00ee40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:25.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:25.909+0000 7f8693e00640 1 -- 172.21.15.16:0/2956040935 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f868c017070 con 0x7f869c06dbe0 2024-03-31T15:14:25.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:25.909+0000 7f8693e00640 1 -- 172.21.15.16:0/2956040935 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f868c00ca00 con 0x7f869c06dbe0 2024-03-31T15:14:25.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:25.909+0000 7f8693e00640 1 -- 172.21.15.16:0/2956040935 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f868c00cdd0 con 0x7f869c06dbe0 2024-03-31T15:14:25.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:25.909+0000 7f86a3000640 1 -- 172.21.15.16:0/2956040935 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f869c150d80 con 0x7f869c06dbe0 2024-03-31T15:14:25.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:25.909+0000 7f86a3000640 1 -- 172.21.15.16:0/2956040935 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f869c0b4c80 con 0x7f869c06dbe0 2024-03-31T15:14:25.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:25.909+0000 7f86a3000640 1 -- 172.21.15.16:0/2956040935 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f869c069950 con 0x7f869c06dbe0 2024-03-31T15:14:25.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:25.909+0000 7f8693e00640 1 -- 172.21.15.16:0/2956040935 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f868c01a070 con 0x7f869c06dbe0 2024-03-31T15:14:25.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:25.913+0000 7f8693e00640 1 --2- 172.21.15.16:0/2956040935 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f867c0683b0 0x7f867c06a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:25.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:25.913+0000 7f8693e00640 1 -- 172.21.15.16:0/2956040935 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 5767+0+0 (secure 0 0 0) 0x7f868c086850 con 0x7f869c06dbe0 2024-03-31T15:14:25.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:25.913+0000 7f869b400640 1 --2- 172.21.15.16:0/2956040935 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f867c0683b0 0x7f867c06a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:25.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:25.913+0000 7f869b400640 1 --2- 172.21.15.16:0/2956040935 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f867c0683b0 0x7f867c06a870 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f869c1520e0 tx=0x7f8688009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:25.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:25.917+0000 7f8693e00640 1 -- 172.21.15.16:0/2956040935 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f868c0566e0 con 0x7f869c06dbe0 2024-03-31T15:14:26.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:26.193+0000 7f86a3000640 1 -- 172.21.15.16:0/2956040935 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f869c0b0670 con 0x7f869c06dbe0 2024-03-31T15:14:26.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:26.193+0000 7f8693e00640 1 -- 172.21.15.16:0/2956040935 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v111) ==== 100+0+11 (secure 0 0 0) 0x7f868c05a9e0 con 0x7f869c06dbe0 2024-03-31T15:14:26.187 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 64 2024-03-31T15:14:26.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:26.193+0000 7f86a3000640 1 -- 172.21.15.16:0/2956040935 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f867c0683b0 msgr2=0x7f867c06a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:26.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:26.193+0000 7f86a3000640 1 --2- 172.21.15.16:0/2956040935 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f867c0683b0 0x7f867c06a870 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f869c1520e0 tx=0x7f8688009040 comp rx=0 tx=0).stop 2024-03-31T15:14:26.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:26.193+0000 7f86a3000640 1 -- 172.21.15.16:0/2956040935 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f869c06dbe0 msgr2=0x7f869c150410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:26.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:26.193+0000 7f86a3000640 1 --2- 172.21.15.16:0/2956040935 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f869c06dbe0 0x7f869c150410 secure :-1 s=READY pgs=1700 cs=0 l=1 rev1=1 crypto rx=0x7f868c00e980 tx=0x7f868c00ee40 comp rx=0 tx=0).stop 2024-03-31T15:14:26.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:26.197+0000 7f86a3000640 1 -- 172.21.15.16:0/2956040935 shutdown_connections 2024-03-31T15:14:26.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:26.197+0000 7f86a3000640 1 --2- 172.21.15.16:0/2956040935 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f867c0683b0 0x7f867c06a870 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:26.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:26.197+0000 7f86a3000640 1 --2- 172.21.15.16:0/2956040935 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f869c1567e0 0x7f869c15b580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:26.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:26.197+0000 7f86a3000640 1 --2- 172.21.15.16:0/2956040935 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f869c0731c0 0x7f869c1479d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:26.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:26.197+0000 7f86a3000640 1 --2- 172.21.15.16:0/2956040935 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f869c06dbe0 0x7f869c150410 unknown :-1 s=CLOSED pgs=1700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:26.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:26.197+0000 7f86a3000640 1 -- 172.21.15.16:0/2956040935 >> 172.21.15.16:0/2956040935 conn(0x7f869c06ee40 msgr2=0x7f869c1666c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:14:26.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:26.197+0000 7f86a3000640 1 -- 172.21.15.16:0/2956040935 shutdown_connections 2024-03-31T15:14:26.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:26.197+0000 7f86a3000640 1 -- 172.21.15.16:0/2956040935 wait complete. 2024-03-31T15:14:26.208 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-31T15:14:26.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:26.357+0000 7efca1800640 1 Processor -- start 2024-03-31T15:14:26.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:26.357+0000 7efca1800640 1 -- start start 2024-03-31T15:14:26.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:26.357+0000 7efca1800640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7efc9c147280 0x7efc9c06c730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:26.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:26.357+0000 7efca1800640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7efc9c070030 0x7efc9c0cc8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:26.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:26.357+0000 7efca1800640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efc9c147660 con 0x7efc9c147dd0 2024-03-31T15:14:26.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:26.357+0000 7efca1800640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7efc9c1620f0 con 0x7efc9c147280 2024-03-31T15:14:26.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:26.357+0000 7efca1800640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7efc9c162340 con 0x7efc9c070030 2024-03-31T15:14:26.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:26.357+0000 7efc9b400640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7efc9c070030 0x7efc9c0cc8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:26.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:26.357+0000 7efc9b400640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7efc9c070030 0x7efc9c0cc8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3301/0 says I am v2:172.21.15.16:38830/0 (socket says 172.21.15.16:38830) 2024-03-31T15:14:26.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:26.357+0000 7efc9b400640 1 -- 172.21.15.16:0/1149100326 learned_addr learned my addr 172.21.15.16:0/1149100326 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:14:26.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:26.357+0000 7efc9a000640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7efc9c147dd0 0x7efc9c06f920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.16:51056/0 (socket says 172.21.15.16:51056) 2024-03-31T15:14:26.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:26.357+0000 7efc9aa00640 1 --2- 172.21.15.16:0/1149100326 >> v2:172.21.15.89:3300/0 conn(0x7efc9c147280 0x7efc9c06c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:26.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:26.357+0000 7efc99600640 1 -- 172.21.15.16:0/1149100326 <== mon.1 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1920246471 0 0) 0x7efc9c147660 con 0x7efc9c147dd0 2024-03-31T15:14:26.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:26.357+0000 7efc9b400640 1 -- 172.21.15.16:0/1149100326 >> v2:172.21.15.89:3300/0 conn(0x7efc9c147280 msgr2=0x7efc9c06c730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:14:26.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:26.357+0000 7efc9b400640 1 --2- 172.21.15.16:0/1149100326 >> v2:172.21.15.89:3300/0 conn(0x7efc9c147280 0x7efc9c06c730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:26.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:26.357+0000 7efc9b400640 1 -- 172.21.15.16:0/1149100326 >> v1:172.21.15.89:6789/0 conn(0x7efc9c147dd0 legacy=0x7efc9c06f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:26.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:26.357+0000 7efc9b400640 1 -- 172.21.15.16:0/1149100326 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc9c162620 con 0x7efc9c070030 2024-03-31T15:14:26.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:26.361+0000 7efc9b400640 1 --2- 172.21.15.16:0/1149100326 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7efc9c070030 0x7efc9c0cc8f0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7efc94009e50 tx=0x7efc9401ca30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7492f9a9683fa751 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:26.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:26.361+0000 7efc99600640 1 -- 172.21.15.16:0/1149100326 <== mon.2 v2:172.21.15.89:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc940a5700 con 0x7efc9c070030 2024-03-31T15:14:26.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:26.361+0000 7efc99600640 1 -- 172.21.15.16:0/1149100326 <== mon.2 v2:172.21.15.89:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efc940a58a0 con 0x7efc9c070030 2024-03-31T15:14:26.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:26.361+0000 7efc99600640 1 -- 172.21.15.16:0/1149100326 <== mon.2 v2:172.21.15.89:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc940a5c70 con 0x7efc9c070030 2024-03-31T15:14:26.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:26.361+0000 7efca1800640 1 -- 172.21.15.16:0/1149100326 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7efc9c070030 msgr2=0x7efc9c0cc8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:26.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:26.361+0000 7efca1800640 1 --2- 172.21.15.16:0/1149100326 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7efc9c070030 0x7efc9c0cc8f0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7efc94009e50 tx=0x7efc9401ca30 comp rx=0 tx=0).stop 2024-03-31T15:14:26.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:26.361+0000 7efca1800640 1 -- 172.21.15.16:0/1149100326 shutdown_connections 2024-03-31T15:14:26.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:26.361+0000 7efca1800640 1 --2- 172.21.15.16:0/1149100326 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7efc9c070030 0x7efc9c0cc8f0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:26.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:26.361+0000 7efca1800640 1 --2- 172.21.15.16:0/1149100326 >> v2:172.21.15.89:3300/0 conn(0x7efc9c147280 0x7efc9c06c730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:26.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:26.361+0000 7efca1800640 1 -- 172.21.15.16:0/1149100326 >> 172.21.15.16:0/1149100326 conn(0x7efc9c067330 msgr2=0x7efc9c067740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:14:26.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:26.361+0000 7efca1800640 1 -- 172.21.15.16:0/1149100326 shutdown_connections 2024-03-31T15:14:26.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:26.361+0000 7efca1800640 1 -- 172.21.15.16:0/1149100326 wait complete. 2024-03-31T15:14:26.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:26.365+0000 7efca1800640 1 Processor -- start 2024-03-31T15:14:26.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:26.365+0000 7efca1800640 1 -- start start 2024-03-31T15:14:26.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:26.365+0000 7efca1800640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efc9c070030 0x7efc9c15e010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:26.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:26.365+0000 7efca1800640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7efc9c147280 0x7efc9c0d2770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:26.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:26.365+0000 7efca1800640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efc9c147dd0 0x7efc9c0d07c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:26.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:26.365+0000 7efca1800640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7efc9c0cda50 con 0x7efc9c147dd0 2024-03-31T15:14:26.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:26.365+0000 7efca1800640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7efc9c0cd8d0 con 0x7efc9c070030 2024-03-31T15:14:26.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:26.365+0000 7efca1800640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7efc9c0cdbd0 con 0x7efc9c147280 2024-03-31T15:14:26.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:26.365+0000 7efc9b400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efc9c147dd0 0x7efc9c0d07c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:26.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:26.365+0000 7efc9b400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efc9c147dd0 0x7efc9c0d07c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43012/0 (socket says 172.21.15.16:43012) 2024-03-31T15:14:26.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:26.365+0000 7efc9b400640 1 -- 172.21.15.16:0/3758073138 learned_addr learned my addr 172.21.15.16:0/3758073138 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:14:26.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:26.365+0000 7efc9aa00640 1 --2- 172.21.15.16:0/3758073138 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efc9c070030 0x7efc9c15e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:26.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:26.365+0000 7efc9a000640 1 --2- 172.21.15.16:0/3758073138 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7efc9c147280 0x7efc9c0d2770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:26.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:26.365+0000 7efc9b400640 1 -- 172.21.15.16:0/3758073138 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7efc9c147280 msgr2=0x7efc9c0d2770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:26.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:26.365+0000 7efc9b400640 1 --2- 172.21.15.16:0/3758073138 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7efc9c147280 0x7efc9c0d2770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:26.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:26.365+0000 7efc9b400640 1 -- 172.21.15.16:0/3758073138 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efc9c070030 msgr2=0x7efc9c15e010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:26.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:26.365+0000 7efc9b400640 1 --2- 172.21.15.16:0/3758073138 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efc9c070030 0x7efc9c15e010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:26.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:26.365+0000 7efc9b400640 1 -- 172.21.15.16:0/3758073138 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc9c13b4f0 con 0x7efc9c147dd0 2024-03-31T15:14:26.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:26.365+0000 7efc9aa00640 1 --2- 172.21.15.16:0/3758073138 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efc9c070030 0x7efc9c15e010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:14:26.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:26.365+0000 7efc9b400640 1 --2- 172.21.15.16:0/3758073138 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efc9c147dd0 0x7efc9c0d07c0 secure :-1 s=READY pgs=1701 cs=0 l=1 rev1=1 crypto rx=0x7efc940af040 tx=0x7efc940197d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:26.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:26.365+0000 7efc92a00640 1 -- 172.21.15.16:0/3758073138 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc94009510 con 0x7efc9c147dd0 2024-03-31T15:14:26.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:26.365+0000 7efc92a00640 1 -- 172.21.15.16:0/3758073138 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efc940be070 con 0x7efc9c147dd0 2024-03-31T15:14:26.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:26.365+0000 7efca1800640 1 -- 172.21.15.16:0/3758073138 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efc9c13b7b0 con 0x7efc9c147dd0 2024-03-31T15:14:26.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:26.365+0000 7efc92a00640 1 -- 172.21.15.16:0/3758073138 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc940b88d0 con 0x7efc9c147dd0 2024-03-31T15:14:26.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:26.365+0000 7efca1800640 1 -- 172.21.15.16:0/3758073138 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efc9c13bd70 con 0x7efc9c147dd0 2024-03-31T15:14:26.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:26.369+0000 7efca1800640 1 -- 172.21.15.16:0/3758073138 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efc9c069950 con 0x7efc9c147dd0 2024-03-31T15:14:26.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:26.369+0000 7efc92a00640 1 -- 172.21.15.16:0/3758073138 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7efc940b8a70 con 0x7efc9c147dd0 2024-03-31T15:14:26.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:26.369+0000 7efc92a00640 1 --2- 172.21.15.16:0/3758073138 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7efc7c068380 0x7efc7c06a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:26.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:26.369+0000 7efc92a00640 1 -- 172.21.15.16:0/3758073138 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 5767+0+0 (secure 0 0 0) 0x7efc94121ac0 con 0x7efc9c147dd0 2024-03-31T15:14:26.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:26.369+0000 7efc9aa00640 1 --2- 172.21.15.16:0/3758073138 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7efc7c068380 0x7efc7c06a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:26.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:26.369+0000 7efc9aa00640 1 --2- 172.21.15.16:0/3758073138 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7efc7c068380 0x7efc7c06a840 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7efc8c009bb0 tx=0x7efc8c008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:26.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:26.377+0000 7efc92a00640 1 -- 172.21.15.16:0/3758073138 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7efc94016620 con 0x7efc9c147dd0 2024-03-31T15:14:26.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:26.553+0000 7efca1800640 1 -- 172.21.15.16:0/3758073138 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7efc9c0cf3e0 con 0x7efc9c147dd0 2024-03-31T15:14:26.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:26.553+0000 7efc92a00640 1 -- 172.21.15.16:0/3758073138 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v111) ==== 80+0+0 (secure 0 0 0) 0x7efc940f20c0 con 0x7efc9c147dd0 2024-03-31T15:14:26.549 INFO:teuthology.orchestra.run.smithi016.stderr:listed 0 entries 2024-03-31T15:14:26.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:26.557+0000 7efca1800640 1 -- 172.21.15.16:0/3758073138 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7efc7c068380 msgr2=0x7efc7c06a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:26.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:26.557+0000 7efca1800640 1 --2- 172.21.15.16:0/3758073138 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7efc7c068380 0x7efc7c06a840 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7efc8c009bb0 tx=0x7efc8c008040 comp rx=0 tx=0).stop 2024-03-31T15:14:26.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:26.557+0000 7efca1800640 1 -- 172.21.15.16:0/3758073138 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efc9c147dd0 msgr2=0x7efc9c0d07c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:26.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:26.557+0000 7efca1800640 1 --2- 172.21.15.16:0/3758073138 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efc9c147dd0 0x7efc9c0d07c0 secure :-1 s=READY pgs=1701 cs=0 l=1 rev1=1 crypto rx=0x7efc940af040 tx=0x7efc940197d0 comp rx=0 tx=0).stop 2024-03-31T15:14:26.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:26.557+0000 7efca1800640 1 -- 172.21.15.16:0/3758073138 shutdown_connections 2024-03-31T15:14:26.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:26.557+0000 7efca1800640 1 --2- 172.21.15.16:0/3758073138 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7efc7c068380 0x7efc7c06a840 unknown :-1 s=CLOSED pgs=723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:26.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:26.557+0000 7efca1800640 1 --2- 172.21.15.16:0/3758073138 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efc9c147dd0 0x7efc9c0d07c0 unknown :-1 s=CLOSED pgs=1701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:26.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:26.557+0000 7efca1800640 1 --2- 172.21.15.16:0/3758073138 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7efc9c147280 0x7efc9c0d2770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:26.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:26.557+0000 7efca1800640 1 --2- 172.21.15.16:0/3758073138 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efc9c070030 0x7efc9c15e010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:26.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:26.557+0000 7efca1800640 1 -- 172.21.15.16:0/3758073138 >> 172.21.15.16:0/3758073138 conn(0x7efc9c067330 msgr2=0x7efc9c15ed10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:14:26.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:26.557+0000 7efca1800640 1 -- 172.21.15.16:0/3758073138 shutdown_connections 2024-03-31T15:14:26.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:26.557+0000 7efca1800640 1 -- 172.21.15.16:0/3758073138 wait complete. 2024-03-31T15:14:26.572 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi016.front.sepia.ceph.com... 2024-03-31T15:14:26.572 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:14:26.572 DEBUG:teuthology.orchestra.run.smithi016:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-03-31T15:14:26.591 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi016.stderr:ceph-fuse[51184]: fuse finished with error 0 and tester_r 0 2024-03-31T15:14:26.594 INFO:teuthology.orchestra.run:waiting for 300 2024-03-31T15:14:32.695 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi016.front.sepia.ceph.com 2024-03-31T15:14:32.695 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:14:32.696 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-31T15:14:32.705 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.8:0 flush journal 2024-03-31T15:14:32.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.897+0000 7f5ad5400640 1 Processor -- start 2024-03-31T15:14:32.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.897+0000 7f5ad5400640 1 -- start start 2024-03-31T15:14:32.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.901+0000 7f5ad5400640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f5ad0070030 0x7f5ad0143c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:32.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.901+0000 7f5ad5400640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5ad0144980 con 0x7f5ad0141590 2024-03-31T15:14:32.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.901+0000 7f5ad5400640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5ad0161fe0 con 0x7f5ad01411b0 2024-03-31T15:14:32.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.901+0000 7f5ad5400640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f5ad015f910 con 0x7f5ad0070030 2024-03-31T15:14:32.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.901+0000 7f5acf200640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f5ad0141590 0x7f5ad0144170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:50220/0 (socket says 172.21.15.16:50220) 2024-03-31T15:14:32.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.901+0000 7f5acf200640 1 -- 172.21.15.16:0/3528967263 learned_addr learned my addr 172.21.15.16:0/3528967263 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:14:32.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.901+0000 7f5ac5e00640 1 --2- 172.21.15.16:0/3528967263 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f5ad0070030 0x7f5ad0143c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:32.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.901+0000 7f5acde00640 1 -- 172.21.15.16:0/3528967263 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2867158788 0 0) 0x7f5ad0144980 con 0x7f5ad0141590 2024-03-31T15:14:32.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.901+0000 7f5acde00640 1 -- 172.21.15.16:0/3528967263 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5ab4003530 con 0x7f5ad0141590 2024-03-31T15:14:32.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.901+0000 7f5acde00640 1 -- 172.21.15.16:0/3528967263 <== mon.1 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3873169983 0 0) 0x7f5ad0161fe0 con 0x7f5ad01411b0 2024-03-31T15:14:32.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.901+0000 7f5acde00640 1 -- 172.21.15.16:0/3528967263 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5ad0144980 con 0x7f5ad01411b0 2024-03-31T15:14:32.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.901+0000 7f5acde00640 1 -- 172.21.15.16:0/3528967263 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3514900799 0 0) 0x7f5ab4003530 con 0x7f5ad0141590 2024-03-31T15:14:32.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.901+0000 7f5acde00640 1 -- 172.21.15.16:0/3528967263 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f5ad0070030 msgr2=0x7f5ad0143c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:14:32.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.901+0000 7f5acde00640 1 --2- 172.21.15.16:0/3528967263 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f5ad0070030 0x7f5ad0143c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:32.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.901+0000 7f5acde00640 1 -- 172.21.15.16:0/3528967263 >> v1:172.21.15.89:6789/0 conn(0x7f5ad01411b0 legacy=0x7f5ad006f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:32.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.901+0000 7f5acde00640 1 -- 172.21.15.16:0/3528967263 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ad01622c0 con 0x7f5ad0141590 2024-03-31T15:14:32.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.901+0000 7f5acde00640 1 -- 172.21.15.16:0/3528967263 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 214807623 0 0) 0x7f5ac8001a00 con 0x7f5ad0141590 2024-03-31T15:14:32.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.901+0000 7f5acde00640 1 -- 172.21.15.16:0/3528967263 >> v1:172.21.15.16:6789/0 conn(0x7f5ad0141590 legacy=0x7f5ad0144170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:32.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.901+0000 7f5acde00640 1 --2- 172.21.15.16:0/3528967263 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5ab4003b80 0x7f5ab4003f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:32.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.901+0000 7f5acde00640 1 --2- 172.21.15.16:0/3528967263 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f5ab4004810 0x7f5ab4004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:32.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.901+0000 7f5acde00640 1 --2- 172.21.15.16:0/3528967263 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5ab40051d0 0x7f5ab400b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:32.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.901+0000 7f5acde00640 1 -- 172.21.15.16:0/3528967263 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f5ab400bfc0 con 0x7f5ab40051d0 2024-03-31T15:14:32.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.901+0000 7f5acde00640 1 -- 172.21.15.16:0/3528967263 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f5ab400c140 con 0x7f5ab4003b80 2024-03-31T15:14:32.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.901+0000 7f5acde00640 1 -- 172.21.15.16:0/3528967263 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f5ab400c340 con 0x7f5ab4004810 2024-03-31T15:14:32.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.901+0000 7f5ac5e00640 1 --2- 172.21.15.16:0/3528967263 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f5ad0070030 0x7f5ad0143c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:14:32.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.901+0000 7f5acf200640 1 --2- 172.21.15.16:0/3528967263 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5ab40051d0 0x7f5ab400b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:32.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.901+0000 7f5ace800640 1 --2- 172.21.15.16:0/3528967263 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5ab4003b80 0x7f5ab4003f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:32.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.901+0000 7f5ac5e00640 1 --2- 172.21.15.16:0/3528967263 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f5ab4004810 0x7f5ab4004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:32.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.901+0000 7f5acf200640 1 -- 172.21.15.16:0/3528967263 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f5ab4004810 msgr2=0x7f5ab4004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:14:32.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.901+0000 7f5acf200640 1 --2- 172.21.15.16:0/3528967263 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f5ab4004810 0x7f5ab4004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:32.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.901+0000 7f5acf200640 1 -- 172.21.15.16:0/3528967263 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5ab4003b80 msgr2=0x7f5ab4003f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:14:32.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.901+0000 7f5acf200640 1 --2- 172.21.15.16:0/3528967263 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5ab4003b80 0x7f5ab4003f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:32.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.901+0000 7f5acf200640 1 -- 172.21.15.16:0/3528967263 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ab4003530 con 0x7f5ab40051d0 2024-03-31T15:14:32.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.905+0000 7f5acf200640 1 --2- 172.21.15.16:0/3528967263 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5ab40051d0 0x7f5ab400b9d0 secure :-1 s=READY pgs=1703 cs=0 l=1 rev1=1 crypto rx=0x7f5ac800a7f0 tx=0x7f5ac801fa60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=22e1c8141ea6b8dc server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:32.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.905+0000 7f5acde00640 1 -- 172.21.15.16:0/3528967263 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ac800a410 con 0x7f5ab40051d0 2024-03-31T15:14:32.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.905+0000 7f5acde00640 1 -- 172.21.15.16:0/3528967263 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5ac80aa690 con 0x7f5ab40051d0 2024-03-31T15:14:32.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.905+0000 7f5acde00640 1 -- 172.21.15.16:0/3528967263 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ac80aaa80 con 0x7f5ab40051d0 2024-03-31T15:14:32.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.905+0000 7f5ad5400640 1 -- 172.21.15.16:0/3528967263 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5ab40051d0 msgr2=0x7f5ab400b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:32.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.905+0000 7f5ad5400640 1 --2- 172.21.15.16:0/3528967263 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5ab40051d0 0x7f5ab400b9d0 secure :-1 s=READY pgs=1703 cs=0 l=1 rev1=1 crypto rx=0x7f5ac800a7f0 tx=0x7f5ac801fa60 comp rx=0 tx=0).stop 2024-03-31T15:14:32.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.905+0000 7f5acf200640 1 -- 172.21.15.16:0/3528967263 reap_dead start 2024-03-31T15:14:32.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.905+0000 7f5ad5400640 1 -- 172.21.15.16:0/3528967263 shutdown_connections 2024-03-31T15:14:32.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.905+0000 7f5ad5400640 1 -- 172.21.15.16:0/3528967263 >> 172.21.15.16:0/3528967263 conn(0x7f5ad0067930 msgr2=0x7f5ad00b3ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:14:32.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.905+0000 7f5ad5400640 1 -- 172.21.15.16:0/3528967263 shutdown_connections 2024-03-31T15:14:32.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.905+0000 7f5ad5400640 1 -- 172.21.15.16:0/3528967263 wait complete. 2024-03-31T15:14:32.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.905+0000 7f5ad5400640 1 Processor -- start 2024-03-31T15:14:32.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.905+0000 7f5ad5400640 1 -- start start 2024-03-31T15:14:32.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.909+0000 7f5ad5400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5ab40051d0 0x7f5ad015ad50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:32.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.909+0000 7f5ad5400640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f5ad01565a0 0x7f5ad01606c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:32.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.909+0000 7f5ad5400640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5ad0156a00 0x7f5ad0160c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:32.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.909+0000 7f5ad5400640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f5ad0162da0 con 0x7f5ab40051d0 2024-03-31T15:14:32.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.909+0000 7f5ad5400640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f5ad0162910 con 0x7f5ad0156a00 2024-03-31T15:14:32.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.909+0000 7f5ad5400640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f5ad0143c30 con 0x7f5ad01565a0 2024-03-31T15:14:32.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.909+0000 7f5ace800640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5ab40051d0 0x7f5ad015ad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:32.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.909+0000 7f5ace800640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5ab40051d0 0x7f5ad015ad50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50720/0 (socket says 172.21.15.16:50720) 2024-03-31T15:14:32.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.909+0000 7f5ace800640 1 -- 172.21.15.16:0/1519507241 learned_addr learned my addr 172.21.15.16:0/1519507241 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:14:32.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.909+0000 7f5acf200640 1 --2- 172.21.15.16:0/1519507241 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5ad0156a00 0x7f5ad0160c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:32.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.909+0000 7f5ac5e00640 1 --2- 172.21.15.16:0/1519507241 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f5ad01565a0 0x7f5ad01606c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:32.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.909+0000 7f5ace800640 1 -- 172.21.15.16:0/1519507241 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f5ad01565a0 msgr2=0x7f5ad01606c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:32.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.909+0000 7f5ace800640 1 --2- 172.21.15.16:0/1519507241 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f5ad01565a0 0x7f5ad01606c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:32.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.909+0000 7f5ace800640 1 -- 172.21.15.16:0/1519507241 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5ad0156a00 msgr2=0x7f5ad0160c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:32.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.909+0000 7f5ace800640 1 --2- 172.21.15.16:0/1519507241 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5ad0156a00 0x7f5ad0160c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:32.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.909+0000 7f5ace800640 1 -- 172.21.15.16:0/1519507241 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ad0159cb0 con 0x7f5ab40051d0 2024-03-31T15:14:32.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.909+0000 7f5ace800640 1 --2- 172.21.15.16:0/1519507241 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5ab40051d0 0x7f5ad015ad50 secure :-1 s=READY pgs=1704 cs=0 l=1 rev1=1 crypto rx=0x7f5ac000f730 tx=0x7f5ac000fbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:32.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.909+0000 7f5ac7400640 1 -- 172.21.15.16:0/1519507241 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ac000fe40 con 0x7f5ab40051d0 2024-03-31T15:14:32.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.909+0000 7f5ac7400640 1 -- 172.21.15.16:0/1519507241 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5ac0018070 con 0x7f5ab40051d0 2024-03-31T15:14:32.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.909+0000 7f5ac7400640 1 -- 172.21.15.16:0/1519507241 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ac000d730 con 0x7f5ab40051d0 2024-03-31T15:14:32.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.909+0000 7f5ad5400640 1 -- 172.21.15.16:0/1519507241 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5ad0159f90 con 0x7f5ab40051d0 2024-03-31T15:14:32.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.909+0000 7f5ad5400640 1 -- 172.21.15.16:0/1519507241 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5ad015a550 con 0x7f5ab40051d0 2024-03-31T15:14:32.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.909+0000 7f5ac7400640 1 -- 172.21.15.16:0/1519507241 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f5ac000d8d0 con 0x7f5ab40051d0 2024-03-31T15:14:32.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.913+0000 7f5ac7400640 1 --2- 172.21.15.16:0/1519507241 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f5aac068380 0x7f5aac06a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:32.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.913+0000 7f5ac7400640 1 -- 172.21.15.16:0/1519507241 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 5767+0+0 (secure 0 0 0) 0x7f5ac0086a90 con 0x7f5ab40051d0 2024-03-31T15:14:32.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.913+0000 7f5ac5e00640 1 --2- 172.21.15.16:0/1519507241 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f5aac068380 0x7f5aac06a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:32.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.913+0000 7f5ad87f5000 1 Processor -- start 2024-03-31T15:14:32.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.913+0000 7f5ad87f5000 1 -- start start 2024-03-31T15:14:32.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.913+0000 7f5ac5e00640 1 --2- 172.21.15.16:0/1519507241 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f5aac068380 0x7f5aac06a840 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f5ad006d8c0 tx=0x7f5ab8008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:32.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.913+0000 7f5ad87f5000 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x560ec06012d0 0x560ec06016b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:32.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.913+0000 7f5ad87f5000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560ec0601bf0 0x560ec05b4470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:32.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.913+0000 7f5ad87f5000 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560ec05b49b0 0x560ec05b4d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:32.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.913+0000 7f5ad87f5000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x560ec049daf0 con 0x560ec0601bf0 2024-03-31T15:14:32.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.913+0000 7f5ad87f5000 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x560ec0406b00 con 0x560ec05b49b0 2024-03-31T15:14:32.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.913+0000 7f5ad87f5000 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x560ec03a3940 con 0x560ec06012d0 2024-03-31T15:14:32.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.913+0000 7f5ac5e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560ec0601bf0 0x560ec05b4470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:32.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.913+0000 7f5ac5e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560ec0601bf0 0x560ec05b4470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50732/0 (socket says 172.21.15.16:50732) 2024-03-31T15:14:32.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.913+0000 7f5ac5e00640 1 -- 172.21.15.16:0/1272814799 learned_addr learned my addr 172.21.15.16:0/1272814799 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:14:32.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.913+0000 7f5acf200640 1 --2- 172.21.15.16:0/1272814799 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560ec05b49b0 0x560ec05b4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:32.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.913+0000 7f5ace800640 1 --2- 172.21.15.16:0/1272814799 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x560ec06012d0 0x560ec06016b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:32.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.913+0000 7f5ac5e00640 1 -- 172.21.15.16:0/1272814799 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x560ec06012d0 msgr2=0x560ec06016b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:14:32.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.913+0000 7f5ac5e00640 1 --2- 172.21.15.16:0/1272814799 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x560ec06012d0 0x560ec06016b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:32.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.913+0000 7f5ac5e00640 1 -- 172.21.15.16:0/1272814799 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560ec05b49b0 msgr2=0x560ec05b4d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:14:32.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.913+0000 7f5ac5e00640 1 --2- 172.21.15.16:0/1272814799 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560ec05b49b0 0x560ec05b4d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:32.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.913+0000 7f5ac5e00640 1 -- 172.21.15.16:0/1272814799 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560ec038c380 con 0x560ec0601bf0 2024-03-31T15:14:32.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.913+0000 7f5acf200640 1 --2- 172.21.15.16:0/1272814799 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560ec05b49b0 0x560ec05b4d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:14:32.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.913+0000 7f5ace800640 1 --2- 172.21.15.16:0/1272814799 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x560ec06012d0 0x560ec06016b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:14:32.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.913+0000 7f5ac5e00640 1 --2- 172.21.15.16:0/1272814799 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560ec0601bf0 0x560ec05b4470 secure :-1 s=READY pgs=1705 cs=0 l=1 rev1=1 crypto rx=0x7f5ab8009ce0 tx=0x7f5ab8017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1adb1c7762e7be3e server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:32.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.913+0000 7f5abf400640 1 -- 172.21.15.16:0/1272814799 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ab8027070 con 0x560ec0601bf0 2024-03-31T15:14:32.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.913+0000 7f5abf400640 1 -- 172.21.15.16:0/1272814799 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5ab8023450 con 0x560ec0601bf0 2024-03-31T15:14:32.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.913+0000 7f5abf400640 1 -- 172.21.15.16:0/1272814799 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ab8023800 con 0x560ec0601bf0 2024-03-31T15:14:32.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.917+0000 7f5ad87f5000 1 -- 172.21.15.16:0/1272814799 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560ec0601bf0 msgr2=0x560ec05b4470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:32.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.917+0000 7f5ad87f5000 1 --2- 172.21.15.16:0/1272814799 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560ec0601bf0 0x560ec05b4470 secure :-1 s=READY pgs=1705 cs=0 l=1 rev1=1 crypto rx=0x7f5ab8009ce0 tx=0x7f5ab8017000 comp rx=0 tx=0).stop 2024-03-31T15:14:32.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.917+0000 7f5ad87f5000 1 -- 172.21.15.16:0/1272814799 shutdown_connections 2024-03-31T15:14:32.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.917+0000 7f5ad87f5000 1 --2- 172.21.15.16:0/1272814799 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560ec05b49b0 0x560ec05b4d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:32.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.917+0000 7f5ad87f5000 1 --2- 172.21.15.16:0/1272814799 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560ec0601bf0 0x560ec05b4470 unknown :-1 s=CLOSED pgs=1705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:32.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.917+0000 7f5ad87f5000 1 --2- 172.21.15.16:0/1272814799 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x560ec06012d0 0x560ec06016b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:32.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.917+0000 7f5ad87f5000 1 -- 172.21.15.16:0/1272814799 >> 172.21.15.16:0/1272814799 conn(0x560ec05e2070 msgr2=0x560ec05e4460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:14:32.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.917+0000 7f5ad87f5000 1 -- 172.21.15.16:0/1272814799 shutdown_connections 2024-03-31T15:14:32.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.917+0000 7f5ad87f5000 1 -- 172.21.15.16:0/1272814799 wait complete. 2024-03-31T15:14:32.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.917+0000 7f5ad87f5000 1 -- start start 2024-03-31T15:14:32.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.917+0000 7f5ad87f5000 1 Processor -- start 2024-03-31T15:14:32.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.917+0000 7f5ad87f5000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560ec05b49b0 0x560ec0608420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:32.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.917+0000 7f5ad87f5000 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x560ec06012d0 0x560ec0608960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:32.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.917+0000 7f5ad87f5000 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560ec0601bf0 0x560ec05b9520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:32.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.917+0000 7f5ad87f5000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x560ec03c41e0 con 0x560ec05b49b0 2024-03-31T15:14:32.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.917+0000 7f5ad87f5000 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x560ec0402ab0 con 0x560ec0601bf0 2024-03-31T15:14:32.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.917+0000 7f5ad87f5000 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x560ec05c0000 con 0x560ec06012d0 2024-03-31T15:14:32.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.917+0000 7f5ad87f5000 10 client.0 _ll_register_callbacks cb 0x560ec05a2370 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T15:14:32.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.917+0000 7f5ace800640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560ec05b49b0 0x560ec0608420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:32.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.917+0000 7f5ace800640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560ec05b49b0 0x560ec0608420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50746/0 (socket says 172.21.15.16:50746) 2024-03-31T15:14:32.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.917+0000 7f5ace800640 1 -- 172.21.15.16:0/2203716567 learned_addr learned my addr 172.21.15.16:0/2203716567 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:14:32.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.917+0000 7f5ac5e00640 1 --2- 172.21.15.16:0/2203716567 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x560ec06012d0 0x560ec0608960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:32.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.917+0000 7f5acf200640 1 --2- 172.21.15.16:0/2203716567 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560ec0601bf0 0x560ec05b9520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:32.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.917+0000 7f5ace800640 1 -- 172.21.15.16:0/2203716567 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x560ec06012d0 msgr2=0x560ec0608960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:14:32.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.917+0000 7f5ace800640 1 --2- 172.21.15.16:0/2203716567 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x560ec06012d0 0x560ec0608960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:32.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.917+0000 7f5ace800640 1 -- 172.21.15.16:0/2203716567 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560ec0601bf0 msgr2=0x560ec05b9520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:14:32.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.917+0000 7f5ace800640 1 --2- 172.21.15.16:0/2203716567 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560ec0601bf0 0x560ec05b9520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:32.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.917+0000 7f5ace800640 1 -- 172.21.15.16:0/2203716567 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560ec03c2b70 con 0x560ec05b49b0 2024-03-31T15:14:32.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.917+0000 7f5ace800640 1 --2- 172.21.15.16:0/2203716567 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560ec05b49b0 0x560ec0608420 secure :-1 s=READY pgs=1706 cs=0 l=1 rev1=1 crypto rx=0x7f5ac005ef20 tx=0x7f5ac0056870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32148368f92d6e05 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:32.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.917+0000 7f5ab3e00640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-31T15:14:32.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.917+0000 7f5ab3e00640 1 -- 172.21.15.16:0/2203716567 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ac005f450 con 0x560ec05b49b0 2024-03-31T15:14:32.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.917+0000 7f5ab3e00640 1 -- 172.21.15.16:0/2203716567 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(111..111 src has 1..111) ==== 5767+0+0 (secure 0 0 0) 0x7f5ac0054b70 con 0x560ec05b49b0 2024-03-31T15:14:32.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.917+0000 7f5ad87f5000 1 -- 172.21.15.16:0/2203716567 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5ab8023450 con 0x560ec05b49b0 2024-03-31T15:14:32.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.917+0000 7f5ab3e00640 1 -- 172.21.15.16:0/2203716567 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=99) ==== 24+0+0 (secure 0 0 0) 0x7f5ac000dbc0 con 0x560ec05b49b0 2024-03-31T15:14:32.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.917+0000 7f5ad87f5000 10 client.9462 fetch_fsmap learned FSMap version 99 2024-03-31T15:14:32.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.917+0000 7f5ad87f5000 1 -- 172.21.15.16:0/2203716567 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=99,monmap=2+}) -- 0x560ec03a3b90 con 0x560ec05b49b0 2024-03-31T15:14:32.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.917+0000 7f5ab3e00640 1 -- 172.21.15.16:0/2203716567 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5ab8023450 con 0x560ec05b49b0 2024-03-31T15:14:32.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.921+0000 7f5ab3e00640 1 -- 172.21.15.16:0/2203716567 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 99) ==== 3494+0+0 (secure 0 0 0) 0x7f5ac0057ba0 con 0x560ec05b49b0 2024-03-31T15:14:32.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.921+0000 7f5ad87f5000 10 client.9462 fetch_fsmap finished waiting for FSMap version 99 2024-03-31T15:14:32.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.921+0000 7f5ad87f5000 10 client.9462 resolve_mds: resolved 8:0 to role '8:0' aka daemon mds.c 2024-03-31T15:14:32.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.921+0000 7f5ad87f5000 20 client.9462 populate_metadata read hostname 'smithi016' 2024-03-31T15:14:32.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.921+0000 7f5ad87f5000 1 --2- 172.21.15.16:0/2203716567 >> [v2:172.21.15.16:6833/2564952016,v1:172.21.15.16:6835/2564952016] conn(0x560ec05ba290 0x560ec05ba670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:32.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.921+0000 7f5ad87f5000 4 client.9462 mds_command: new command op to 9195 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-31T15:14:32.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.921+0000 7f5ad87f5000 1 -- 172.21.15.16:0/2203716567 --> [v2:172.21.15.16:6833/2564952016,v1:172.21.15.16:6835/2564952016] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560ec03a1dd0 con 0x560ec05ba290 2024-03-31T15:14:32.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.921+0000 7f5ac5e00640 1 --2- 172.21.15.16:0/2203716567 >> [v2:172.21.15.16:6833/2564952016,v1:172.21.15.16:6835/2564952016] conn(0x560ec05ba290 0x560ec05ba670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:32.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.921+0000 7f5ab3e00640 10 client.9462 ms_handle_connect on v2:172.21.15.16:6833/2564952016 2024-03-31T15:14:32.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.921+0000 7f5ac5e00640 1 --2- 172.21.15.16:0/2203716567 >> [v2:172.21.15.16:6833/2564952016,v1:172.21.15.16:6835/2564952016] conn(0x560ec05ba290 0x560ec05ba670 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=25e512702ca74e8b server_cookie=9e24dca4139262d0 in_seq=0 out_seq=0 2024-03-31T15:14:32.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.921+0000 7f5ab3e00640 1 -- 172.21.15.16:0/2203716567 <== mds.0 v2:172.21.15.16:6833/2564952016 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x560ec03a1dd0 con 0x560ec05ba290 2024-03-31T15:14:32.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.921+0000 7f5ab3e00640 10 client.9462 handle_command_reply: tid=0 2024-03-31T15:14:32.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.921+0000 7f5ad87f5000 1 client.9462 shutdown 2024-03-31T15:14:32.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.921+0000 7f5ad87f5000 1 -- 172.21.15.16:0/2203716567 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560ec05b49b0 msgr2=0x560ec0608420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:32.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.921+0000 7f5ad87f5000 1 --2- 172.21.15.16:0/2203716567 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560ec05b49b0 0x560ec0608420 secure :-1 s=READY pgs=1706 cs=0 l=1 rev1=1 crypto rx=0x7f5ac005ef20 tx=0x7f5ac0056870 comp rx=0 tx=0).stop 2024-03-31T15:14:32.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.921+0000 7f5ad87f5000 1 -- 172.21.15.16:0/2203716567 shutdown_connections 2024-03-31T15:14:32.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.921+0000 7f5ad87f5000 1 --2- 172.21.15.16:0/2203716567 >> [v2:172.21.15.16:6833/2564952016,v1:172.21.15.16:6835/2564952016] conn(0x560ec05ba290 0x560ec05ba670 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:32.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.921+0000 7f5ad87f5000 1 --2- 172.21.15.16:0/2203716567 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560ec0601bf0 0x560ec05b9520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:32.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.921+0000 7f5ad87f5000 1 --2- 172.21.15.16:0/2203716567 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x560ec06012d0 0x560ec0608960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:32.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.921+0000 7f5ad87f5000 1 --2- 172.21.15.16:0/2203716567 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560ec05b49b0 0x560ec0608420 unknown :-1 s=CLOSED pgs=1706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:32.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.921+0000 7f5ab3e00640 0 client.9462 ms_handle_reset on v2:172.21.15.16:6833/2564952016 2024-03-31T15:14:32.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.921+0000 7f5ad87f5000 1 -- 172.21.15.16:0/2203716567 >> 172.21.15.16:0/2203716567 conn(0x560ec05e2070 msgr2=0x560ec05e4460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:14:32.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.921+0000 7f5ad87f5000 1 -- 172.21.15.16:0/2203716567 shutdown_connections 2024-03-31T15:14:32.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.921+0000 7f5ad87f5000 1 -- 172.21.15.16:0/2203716567 wait complete. 2024-03-31T15:14:32.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.921+0000 7f5ad87f5000 20 client.9462 trim_cache size 0 max 16384 2024-03-31T15:14:32.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.937+0000 7f5ad87f5000 1 Processor -- start 2024-03-31T15:14:32.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.937+0000 7f5ad87f5000 1 -- start start 2024-03-31T15:14:32.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.937+0000 7f5ad87f5000 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560ec05e2070 0x560ec05ff380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:32.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.937+0000 7f5ad87f5000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560ec05b49b0 0x560ec05ff8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:32.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.937+0000 7f5ad87f5000 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x560ec06012d0 0x560ec06175b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:32.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.937+0000 7f5ad87f5000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f5ad015f650 con 0x560ec05b49b0 2024-03-31T15:14:32.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.937+0000 7f5ad87f5000 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f5ad0133bc0 con 0x560ec05e2070 2024-03-31T15:14:32.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.937+0000 7f5ad87f5000 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x560ec03d2150 con 0x560ec06012d0 2024-03-31T15:14:32.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.937+0000 7f5ac5e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560ec05b49b0 0x560ec05ff8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:32.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.937+0000 7f5ac5e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560ec05b49b0 0x560ec05ff8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50760/0 (socket says 172.21.15.16:50760) 2024-03-31T15:14:32.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.937+0000 7f5ac5e00640 1 -- 172.21.15.16:0/168608193 learned_addr learned my addr 172.21.15.16:0/168608193 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:14:32.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.937+0000 7f5ace800640 1 --2- 172.21.15.16:0/168608193 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560ec05e2070 0x560ec05ff380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:32.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.937+0000 7f5acf200640 1 --2- 172.21.15.16:0/168608193 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x560ec06012d0 0x560ec06175b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:32.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.937+0000 7f5ac5e00640 1 -- 172.21.15.16:0/168608193 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x560ec06012d0 msgr2=0x560ec06175b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:14:32.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.937+0000 7f5ac5e00640 1 --2- 172.21.15.16:0/168608193 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x560ec06012d0 0x560ec06175b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:32.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.937+0000 7f5ac5e00640 1 -- 172.21.15.16:0/168608193 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560ec05e2070 msgr2=0x560ec05ff380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:14:32.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.937+0000 7f5ac5e00640 1 --2- 172.21.15.16:0/168608193 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560ec05e2070 0x560ec05ff380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:32.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.937+0000 7f5ac5e00640 1 -- 172.21.15.16:0/168608193 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560ec0487140 con 0x560ec05b49b0 2024-03-31T15:14:32.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.937+0000 7f5ac5e00640 1 --2- 172.21.15.16:0/168608193 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560ec05b49b0 0x560ec05ff8c0 secure :-1 s=READY pgs=1707 cs=0 l=1 rev1=1 crypto rx=0x7f5ab8008010 tx=0x7f5ab8019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf9cc5cd1c8c27a4 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:32.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.937+0000 7f5abea00640 1 -- 172.21.15.16:0/168608193 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ab801a040 con 0x560ec05b49b0 2024-03-31T15:14:32.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.937+0000 7f5abea00640 1 -- 172.21.15.16:0/168608193 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5ab8036bc0 con 0x560ec05b49b0 2024-03-31T15:14:32.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.937+0000 7f5abea00640 1 -- 172.21.15.16:0/168608193 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ab8027040 con 0x560ec05b49b0 2024-03-31T15:14:32.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.937+0000 7f5ad87f5000 1 -- 172.21.15.16:0/168608193 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560ec05b49b0 msgr2=0x560ec05ff8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:32.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.937+0000 7f5ad87f5000 1 --2- 172.21.15.16:0/168608193 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560ec05b49b0 0x560ec05ff8c0 secure :-1 s=READY pgs=1707 cs=0 l=1 rev1=1 crypto rx=0x7f5ab8008010 tx=0x7f5ab8019000 comp rx=0 tx=0).stop 2024-03-31T15:14:32.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.937+0000 7f5ad87f5000 1 -- 172.21.15.16:0/168608193 shutdown_connections 2024-03-31T15:14:32.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.937+0000 7f5ad87f5000 1 --2- 172.21.15.16:0/168608193 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x560ec06012d0 0x560ec06175b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:32.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.937+0000 7f5ad87f5000 1 --2- 172.21.15.16:0/168608193 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560ec05b49b0 0x560ec05ff8c0 unknown :-1 s=CLOSED pgs=1707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:32.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.937+0000 7f5ad87f5000 1 --2- 172.21.15.16:0/168608193 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560ec05e2070 0x560ec05ff380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:32.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.937+0000 7f5ad87f5000 1 -- 172.21.15.16:0/168608193 >> 172.21.15.16:0/168608193 conn(0x560ec060eea0 msgr2=0x560ec05b3df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:14:32.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.937+0000 7f5ad87f5000 1 -- 172.21.15.16:0/168608193 shutdown_connections 2024-03-31T15:14:32.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.937+0000 7f5ad87f5000 1 -- 172.21.15.16:0/168608193 wait complete. 2024-03-31T15:14:32.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.941+0000 7f5ad87f5000 1 -- start start 2024-03-31T15:14:32.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.941+0000 7f5ad87f5000 1 Processor -- start 2024-03-31T15:14:32.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.941+0000 7f5ad87f5000 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x560ec05b49b0 0x560ec0600a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:32.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.941+0000 7f5ad87f5000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560ec05e2070 0x560ec06706a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:32.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.941+0000 7f5ad87f5000 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560ec06012d0 0x560ec0670be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:32.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.941+0000 7f5ad87f5000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x560ec03a1940 con 0x560ec05e2070 2024-03-31T15:14:32.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.941+0000 7f5ad87f5000 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x560ec04d5420 con 0x560ec06012d0 2024-03-31T15:14:32.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.941+0000 7f5ad87f5000 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x560ec03bd750 con 0x560ec05b49b0 2024-03-31T15:14:32.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.941+0000 7f5ad87f5000 10 client.0 _ll_register_callbacks cb 0x7f5ac0052b10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T15:14:32.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.941+0000 7f5ac5e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560ec05e2070 0x560ec06706a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:32.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.941+0000 7f5ac5e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560ec05e2070 0x560ec06706a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50768/0 (socket says 172.21.15.16:50768) 2024-03-31T15:14:32.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.941+0000 7f5ac5e00640 1 -- 172.21.15.16:0/469140862 learned_addr learned my addr 172.21.15.16:0/469140862 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:14:32.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.941+0000 7f5ace800640 1 --2- 172.21.15.16:0/469140862 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x560ec05b49b0 0x560ec0600a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:32.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.941+0000 7f5ac5e00640 1 -- 172.21.15.16:0/469140862 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x560ec05b49b0 msgr2=0x560ec0600a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:14:32.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.941+0000 7f5acf200640 1 --2- 172.21.15.16:0/469140862 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560ec06012d0 0x560ec0670be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:32.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.941+0000 7f5ac5e00640 1 --2- 172.21.15.16:0/469140862 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x560ec05b49b0 0x560ec0600a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:32.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.941+0000 7f5ac5e00640 1 -- 172.21.15.16:0/469140862 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560ec06012d0 msgr2=0x560ec0670be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:14:32.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.941+0000 7f5ac5e00640 1 --2- 172.21.15.16:0/469140862 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560ec06012d0 0x560ec0670be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:32.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.941+0000 7f5ac5e00640 1 -- 172.21.15.16:0/469140862 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560ec03951e0 con 0x560ec05e2070 2024-03-31T15:14:32.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.941+0000 7f5ace800640 1 --2- 172.21.15.16:0/469140862 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x560ec05b49b0 0x560ec0600a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:14:32.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.941+0000 7f5acf200640 1 --2- 172.21.15.16:0/469140862 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560ec06012d0 0x560ec0670be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:14:32.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.941+0000 7f5ac5e00640 1 --2- 172.21.15.16:0/469140862 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560ec05e2070 0x560ec06706a0 secure :-1 s=READY pgs=1708 cs=0 l=1 rev1=1 crypto rx=0x7f5ab800b010 tx=0x7f5ab8004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=650e0d3866d52a16 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:32.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.941+0000 7f5ab3e00640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-31T15:14:32.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.941+0000 7f5ab3e00640 1 -- 172.21.15.16:0/469140862 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ab803d070 con 0x560ec05e2070 2024-03-31T15:14:32.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.941+0000 7f5ab3e00640 1 -- 172.21.15.16:0/469140862 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(111..111 src has 1..111) ==== 5767+0+0 (secure 0 0 0) 0x7f5ab8023b50 con 0x560ec05e2070 2024-03-31T15:14:32.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.941+0000 7f5ad87f5000 1 -- 172.21.15.16:0/469140862 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5ab8036bc0 con 0x560ec05e2070 2024-03-31T15:14:32.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.941+0000 7f5ab3e00640 1 -- 172.21.15.16:0/469140862 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=99) ==== 24+0+0 (secure 0 0 0) 0x7f5ab8049050 con 0x560ec05e2070 2024-03-31T15:14:32.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.941+0000 7f5ad87f5000 10 client.9468 fetch_fsmap learned FSMap version 99 2024-03-31T15:14:32.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.941+0000 7f5ad87f5000 1 -- 172.21.15.16:0/469140862 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=99,monmap=2+}) -- 0x560ec039a7b0 con 0x560ec05e2070 2024-03-31T15:14:32.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.941+0000 7f5ab3e00640 1 -- 172.21.15.16:0/469140862 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5ab8036bc0 con 0x560ec05e2070 2024-03-31T15:14:32.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.941+0000 7f5ab3e00640 1 -- 172.21.15.16:0/469140862 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 99) ==== 3494+0+0 (secure 0 0 0) 0x7f5ab804c020 con 0x560ec05e2070 2024-03-31T15:14:32.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.941+0000 7f5ad87f5000 10 client.9468 fetch_fsmap finished waiting for FSMap version 99 2024-03-31T15:14:32.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.941+0000 7f5ad87f5000 10 client.9468 resolve_mds: resolved 8:0 to role '8:0' aka daemon mds.c 2024-03-31T15:14:32.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.941+0000 7f5ad87f5000 20 client.9468 populate_metadata read hostname 'smithi016' 2024-03-31T15:14:32.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.941+0000 7f5ad87f5000 1 --2- 172.21.15.16:0/469140862 >> [v2:172.21.15.16:6833/2564952016,v1:172.21.15.16:6835/2564952016] conn(0x560ec0601bf0 0x560ec0605030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:32.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.941+0000 7f5ad87f5000 4 client.9468 mds_command: new command op to 9195 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-31T15:14:32.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.941+0000 7f5ad87f5000 1 -- 172.21.15.16:0/469140862 --> [v2:172.21.15.16:6833/2564952016,v1:172.21.15.16:6835/2564952016] -- command(tid 0: {"prefix": "flush journal"}) -- 0x560ec0414040 con 0x560ec0601bf0 2024-03-31T15:14:32.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.941+0000 7f5ace800640 1 --2- 172.21.15.16:0/469140862 >> [v2:172.21.15.16:6833/2564952016,v1:172.21.15.16:6835/2564952016] conn(0x560ec0601bf0 0x560ec0605030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:32.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.945+0000 7f5ab3e00640 10 client.9468 ms_handle_connect on v2:172.21.15.16:6833/2564952016 2024-03-31T15:14:32.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.945+0000 7f5ace800640 1 --2- 172.21.15.16:0/469140862 >> [v2:172.21.15.16:6833/2564952016,v1:172.21.15.16:6835/2564952016] conn(0x560ec0601bf0 0x560ec0605030 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f621a1e25b9381a server_cookie=5167e277b0ef4ca in_seq=0 out_seq=0 2024-03-31T15:14:32.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.953+0000 7f5ab3e00640 1 -- 172.21.15.16:0/469140862 <== mds.0 v2:172.21.15.16:6833/2564952016 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x560ec0414040 con 0x560ec0601bf0 2024-03-31T15:14:32.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.953+0000 7f5ab3e00640 10 client.9468 handle_command_reply: tid=0 2024-03-31T15:14:32.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.953+0000 7f5ad87f5000 1 client.9468 shutdown 2024-03-31T15:14:32.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.953+0000 7f5ad87f5000 1 -- 172.21.15.16:0/469140862 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560ec05e2070 msgr2=0x560ec06706a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:32.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.953+0000 7f5ad87f5000 1 --2- 172.21.15.16:0/469140862 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560ec05e2070 0x560ec06706a0 secure :-1 s=READY pgs=1708 cs=0 l=1 rev1=1 crypto rx=0x7f5ab800b010 tx=0x7f5ab8004740 comp rx=0 tx=0).stop 2024-03-31T15:14:32.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.953+0000 7f5ad87f5000 1 -- 172.21.15.16:0/469140862 shutdown_connections 2024-03-31T15:14:32.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.953+0000 7f5ad87f5000 1 --2- 172.21.15.16:0/469140862 >> [v2:172.21.15.16:6833/2564952016,v1:172.21.15.16:6835/2564952016] conn(0x560ec0601bf0 0x560ec0605030 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:32.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.953+0000 7f5ad87f5000 1 --2- 172.21.15.16:0/469140862 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560ec06012d0 0x560ec0670be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:32.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.953+0000 7f5ad87f5000 1 --2- 172.21.15.16:0/469140862 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560ec05e2070 0x560ec06706a0 unknown :-1 s=CLOSED pgs=1708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:32.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.953+0000 7f5ab3e00640 0 client.9468 ms_handle_reset on v2:172.21.15.16:6833/2564952016 2024-03-31T15:14:32.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.953+0000 7f5ad87f5000 1 --2- 172.21.15.16:0/469140862 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x560ec05b49b0 0x560ec0600a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:32.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.953+0000 7f5ad87f5000 1 -- 172.21.15.16:0/469140862 >> 172.21.15.16:0/469140862 conn(0x560ec060eea0 msgr2=0x560ec0617e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:14:32.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.953+0000 7f5ad87f5000 1 -- 172.21.15.16:0/469140862 shutdown_connections 2024-03-31T15:14:32.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.953+0000 7f5ad87f5000 1 -- 172.21.15.16:0/469140862 wait complete. 2024-03-31T15:14:32.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.953+0000 7f5ad87f5000 20 client.9468 trim_cache size 0 max 16384 2024-03-31T15:14:32.955 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-03-31T15:14:32.955 INFO:teuthology.orchestra.run.smithi016.stdout: "message": "", 2024-03-31T15:14:32.955 INFO:teuthology.orchestra.run.smithi016.stdout: "return_code": 0 2024-03-31T15:14:32.955 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-03-31T15:14:32.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.953+0000 7f5ad5400640 1 -- 172.21.15.16:0/1519507241 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f5aac068380 msgr2=0x7f5aac06a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:32.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.953+0000 7f5ad5400640 1 --2- 172.21.15.16:0/1519507241 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f5aac068380 0x7f5aac06a840 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f5ad006d8c0 tx=0x7f5ab8008040 comp rx=0 tx=0).stop 2024-03-31T15:14:32.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.953+0000 7f5ad5400640 1 -- 172.21.15.16:0/1519507241 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5ab40051d0 msgr2=0x7f5ad015ad50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:32.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.953+0000 7f5ad5400640 1 --2- 172.21.15.16:0/1519507241 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5ab40051d0 0x7f5ad015ad50 secure :-1 s=READY pgs=1704 cs=0 l=1 rev1=1 crypto rx=0x7f5ac000f730 tx=0x7f5ac000fbf0 comp rx=0 tx=0).stop 2024-03-31T15:14:32.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.953+0000 7f5ad5400640 1 -- 172.21.15.16:0/1519507241 shutdown_connections 2024-03-31T15:14:32.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.953+0000 7f5ad5400640 1 --2- 172.21.15.16:0/1519507241 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f5aac068380 0x7f5aac06a840 unknown :-1 s=CLOSED pgs=724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:32.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.953+0000 7f5ad5400640 1 --2- 172.21.15.16:0/1519507241 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5ad0156a00 0x7f5ad0160c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:32.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.953+0000 7f5ad5400640 1 --2- 172.21.15.16:0/1519507241 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f5ad01565a0 0x7f5ad01606c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:32.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.953+0000 7f5ad5400640 1 --2- 172.21.15.16:0/1519507241 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5ab40051d0 0x7f5ad015ad50 unknown :-1 s=CLOSED pgs=1704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:32.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.953+0000 7f5ad5400640 1 -- 172.21.15.16:0/1519507241 >> 172.21.15.16:0/1519507241 conn(0x7f5ad0067930 msgr2=0x7f5ad0142920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:14:32.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.953+0000 7f5ad5400640 1 -- 172.21.15.16:0/1519507241 shutdown_connections 2024-03-31T15:14:32.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:32.953+0000 7f5ad5400640 1 -- 172.21.15.16:0/1519507241 wait complete. 2024-03-31T15:14:32.973 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-31T15:14:33.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:33.133+0000 7f0087000640 1 Processor -- start 2024-03-31T15:14:33.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:33.133+0000 7f0087000640 1 -- start start 2024-03-31T15:14:33.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:33.137+0000 7f0087000640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f008006d490 0x7f008006d8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:33.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:33.137+0000 7f0087000640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f00800cc740 0x7f00800ccbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:33.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:33.137+0000 7f0087000640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0080162010 con 0x7f008006f920 2024-03-31T15:14:33.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:33.137+0000 7f0087000640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f0080162be0 con 0x7f008006d490 2024-03-31T15:14:33.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:33.137+0000 7f0087000640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f00801622b0 con 0x7f00800cc740 2024-03-31T15:14:33.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:33.137+0000 7f0085000640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f00800cc740 0x7f00800ccbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:33.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:33.137+0000 7f007fe00640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f008006f920 0x7f008006fd00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.16:44492/0 (socket says 172.21.15.16:44492) 2024-03-31T15:14:33.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:33.137+0000 7f0085000640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f00800cc740 0x7f00800ccbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3301/0 says I am v2:172.21.15.16:60230/0 (socket says 172.21.15.16:60230) 2024-03-31T15:14:33.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:33.137+0000 7f007fe00640 1 -- 172.21.15.16:0/1390282360 learned_addr learned my addr 172.21.15.16:0/1390282360 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:14:33.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:33.137+0000 7f007f400640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f008006d490 0x7f008006d8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:33.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:33.137+0000 7f007f400640 1 -- 172.21.15.16:0/1390282360 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f00800cc740 msgr2=0x7f00800ccbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:14:33.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:33.137+0000 7f007f400640 1 --2- 172.21.15.16:0/1390282360 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f00800cc740 0x7f00800ccbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:33.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:33.137+0000 7f007f400640 1 -- 172.21.15.16:0/1390282360 >> v1:172.21.15.89:6789/0 conn(0x7f008006f920 legacy=0x7f008006fd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:33.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:33.137+0000 7f007f400640 1 -- 172.21.15.16:0/1390282360 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0080162590 con 0x7f008006d490 2024-03-31T15:14:33.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:33.137+0000 7f007ea00640 1 -- 172.21.15.16:0/1390282360 <== mon.1 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 172737035 0 0) 0x7f0080162010 con 0x7f008006f920 2024-03-31T15:14:33.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:33.141+0000 7f007f400640 1 --2- 172.21.15.16:0/1390282360 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f008006d490 0x7f008006d8f0 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f0080162d60 tx=0x7f006c01c470 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b28a4bc8e0c4876e server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:33.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:33.141+0000 7f007ea00640 1 -- 172.21.15.16:0/1390282360 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f006c0b2070 con 0x7f008006d490 2024-03-31T15:14:33.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:33.141+0000 7f007ea00640 1 -- 172.21.15.16:0/1390282360 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f006c0ad640 con 0x7f008006d490 2024-03-31T15:14:33.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:33.141+0000 7f007ea00640 1 -- 172.21.15.16:0/1390282360 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f006c0ada10 con 0x7f008006d490 2024-03-31T15:14:33.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:33.141+0000 7f0087000640 1 -- 172.21.15.16:0/1390282360 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f008006d490 msgr2=0x7f008006d8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:33.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:33.141+0000 7f0087000640 1 --2- 172.21.15.16:0/1390282360 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f008006d490 0x7f008006d8f0 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f0080162d60 tx=0x7f006c01c470 comp rx=0 tx=0).stop 2024-03-31T15:14:33.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:33.141+0000 7f0087000640 1 -- 172.21.15.16:0/1390282360 shutdown_connections 2024-03-31T15:14:33.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:33.141+0000 7f0087000640 1 --2- 172.21.15.16:0/1390282360 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f008006d490 0x7f008006d8f0 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:33.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:33.141+0000 7f0087000640 1 --2- 172.21.15.16:0/1390282360 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f00800cc740 0x7f00800ccbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:33.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:33.141+0000 7f0087000640 1 -- 172.21.15.16:0/1390282360 >> 172.21.15.16:0/1390282360 conn(0x7f0080067330 msgr2=0x7f0080067740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:14:33.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:33.141+0000 7f0087000640 1 -- 172.21.15.16:0/1390282360 shutdown_connections 2024-03-31T15:14:33.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:33.141+0000 7f0087000640 1 -- 172.21.15.16:0/1390282360 wait complete. 2024-03-31T15:14:33.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:33.141+0000 7f0087000640 1 Processor -- start 2024-03-31T15:14:33.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:33.141+0000 7f0087000640 1 -- start start 2024-03-31T15:14:33.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:33.145+0000 7f0087000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f008006d490 0x7f008015bec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:33.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:33.145+0000 7f0087000640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f008006f920 0x7f00800b7160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:33.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:33.145+0000 7f0087000640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f00800cc740 0x7f00800b2d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:33.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:33.145+0000 7f0087000640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f008016b530 con 0x7f008006d490 2024-03-31T15:14:33.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:33.145+0000 7f0087000640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f008016b3b0 con 0x7f00800cc740 2024-03-31T15:14:33.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:33.145+0000 7f0087000640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f008016b6b0 con 0x7f008006f920 2024-03-31T15:14:33.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:33.145+0000 7f007fe00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f008006d490 0x7f008015bec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:33.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:33.145+0000 7f007fe00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f008006d490 0x7f008015bec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50776/0 (socket says 172.21.15.16:50776) 2024-03-31T15:14:33.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:33.145+0000 7f007fe00640 1 -- 172.21.15.16:0/2653841702 learned_addr learned my addr 172.21.15.16:0/2653841702 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:14:33.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:33.145+0000 7f007f400640 1 --2- 172.21.15.16:0/2653841702 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f008006f920 0x7f00800b7160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:33.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:33.145+0000 7f0085000640 1 --2- 172.21.15.16:0/2653841702 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f00800cc740 0x7f00800b2d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:33.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:33.145+0000 7f007fe00640 1 -- 172.21.15.16:0/2653841702 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f008006f920 msgr2=0x7f00800b7160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:33.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:33.145+0000 7f007fe00640 1 --2- 172.21.15.16:0/2653841702 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f008006f920 0x7f00800b7160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:33.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:33.145+0000 7f007fe00640 1 -- 172.21.15.16:0/2653841702 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f00800cc740 msgr2=0x7f00800b2d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:33.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:33.145+0000 7f007fe00640 1 --2- 172.21.15.16:0/2653841702 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f00800cc740 0x7f00800b2d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:33.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:33.145+0000 7f007fe00640 1 -- 172.21.15.16:0/2653841702 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00800b7b40 con 0x7f008006d490 2024-03-31T15:14:33.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:33.145+0000 7f007fe00640 1 --2- 172.21.15.16:0/2653841702 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f008006d490 0x7f008015bec0 secure :-1 s=READY pgs=1709 cs=0 l=1 rev1=1 crypto rx=0x7f00700011f0 tx=0x7f007000c9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:33.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:33.145+0000 7f0077e00640 1 -- 172.21.15.16:0/2653841702 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0070018070 con 0x7f008006d490 2024-03-31T15:14:33.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:33.145+0000 7f0077e00640 1 -- 172.21.15.16:0/2653841702 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f00700074e0 con 0x7f008006d490 2024-03-31T15:14:33.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:33.145+0000 7f0087000640 1 -- 172.21.15.16:0/2653841702 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f00800b5000 con 0x7f008006d490 2024-03-31T15:14:33.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:33.145+0000 7f0077e00640 1 -- 172.21.15.16:0/2653841702 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0070004a10 con 0x7f008006d490 2024-03-31T15:14:33.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:33.145+0000 7f0087000640 1 -- 172.21.15.16:0/2653841702 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f00800b55c0 con 0x7f008006d490 2024-03-31T15:14:33.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:33.145+0000 7f0087000640 1 -- 172.21.15.16:0/2653841702 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0080069950 con 0x7f008006d490 2024-03-31T15:14:33.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:33.149+0000 7f0077e00640 1 -- 172.21.15.16:0/2653841702 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f007001b070 con 0x7f008006d490 2024-03-31T15:14:33.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:33.149+0000 7f0077e00640 1 --2- 172.21.15.16:0/2653841702 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f0060068380 0x7f006006a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:33.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:33.149+0000 7f0077e00640 1 -- 172.21.15.16:0/2653841702 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 5767+0+0 (secure 0 0 0) 0x7f0070085cb0 con 0x7f008006d490 2024-03-31T15:14:33.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:33.149+0000 7f007f400640 1 --2- 172.21.15.16:0/2653841702 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f0060068380 0x7f006006a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:33.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:33.149+0000 7f007f400640 1 --2- 172.21.15.16:0/2653841702 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f0060068380 0x7f006006a840 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7f006c0af000 tx=0x7f006c008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:33.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:33.153+0000 7f0077e00640 1 -- 172.21.15.16:0/2653841702 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f0070055bd0 con 0x7f008006d490 2024-03-31T15:14:33.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:33.357+0000 7f0087000640 1 -- 172.21.15.16:0/2653841702 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f00800cf3f0 con 0x7f008006d490 2024-03-31T15:14:33.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:33.361+0000 7f0077e00640 1 -- 172.21.15.16:0/2653841702 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v99) ==== 53+0+73 (secure 0 0 0) 0x7f0070059ed0 con 0x7f008006d490 2024-03-31T15:14:33.353 INFO:teuthology.orchestra.run.smithi016.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-31T15:14:33.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:33.361+0000 7f0087000640 1 -- 172.21.15.16:0/2653841702 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f0060068380 msgr2=0x7f006006a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:33.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:33.361+0000 7f0087000640 1 --2- 172.21.15.16:0/2653841702 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f0060068380 0x7f006006a840 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7f006c0af000 tx=0x7f006c008040 comp rx=0 tx=0).stop 2024-03-31T15:14:33.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:33.361+0000 7f0087000640 1 -- 172.21.15.16:0/2653841702 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f008006d490 msgr2=0x7f008015bec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:33.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:33.361+0000 7f0087000640 1 --2- 172.21.15.16:0/2653841702 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f008006d490 0x7f008015bec0 secure :-1 s=READY pgs=1709 cs=0 l=1 rev1=1 crypto rx=0x7f00700011f0 tx=0x7f007000c9b0 comp rx=0 tx=0).stop 2024-03-31T15:14:33.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:33.361+0000 7f0087000640 1 -- 172.21.15.16:0/2653841702 shutdown_connections 2024-03-31T15:14:33.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:33.361+0000 7f0087000640 1 --2- 172.21.15.16:0/2653841702 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f0060068380 0x7f006006a840 unknown :-1 s=CLOSED pgs=725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:33.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:33.361+0000 7f0087000640 1 --2- 172.21.15.16:0/2653841702 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f00800cc740 0x7f00800b2d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:33.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:33.361+0000 7f0087000640 1 --2- 172.21.15.16:0/2653841702 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f008006f920 0x7f00800b7160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:33.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:33.361+0000 7f0087000640 1 --2- 172.21.15.16:0/2653841702 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f008006d490 0x7f008015bec0 unknown :-1 s=CLOSED pgs=1709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:33.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:33.361+0000 7f0087000640 1 -- 172.21.15.16:0/2653841702 >> 172.21.15.16:0/2653841702 conn(0x7f0080067330 msgr2=0x7f00800d1200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:14:33.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:33.361+0000 7f0087000640 1 -- 172.21.15.16:0/2653841702 shutdown_connections 2024-03-31T15:14:33.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:33.361+0000 7f0087000640 1 -- 172.21.15.16:0/2653841702 wait complete. 2024-03-31T15:14:33.375 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-31T15:14:33.375 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-31T15:14:33.375 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi016.front.sepia.ceph.com 2024-03-31T15:14:33.375 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-31T15:14:33.375 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-31T15:14:33.375 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-31T15:14:33.375 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-31T15:14:33.375 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-31T15:14:33.376 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-31T15:14:33.376 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:14:33.376 DEBUG:teuthology.orchestra.run.smithi016:> ip addr 2024-03-31T15:14:33.425 INFO:teuthology.orchestra.run.smithi016.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-31T15:14:33.425 INFO:teuthology.orchestra.run.smithi016.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-31T15:14:33.425 INFO:teuthology.orchestra.run.smithi016.stdout: inet 127.0.0.1/8 scope host lo 2024-03-31T15:14:33.425 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2024-03-31T15:14:33.425 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 ::1/128 scope host 2024-03-31T15:14:33.426 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2024-03-31T15:14:33.426 INFO:teuthology.orchestra.run.smithi016.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-03-31T15:14:33.426 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:6b:fd:e0 brd ff:ff:ff:ff:ff:ff 2024-03-31T15:14:33.426 INFO:teuthology.orchestra.run.smithi016.stdout: altname enp7s0f0 2024-03-31T15:14:33.426 INFO:teuthology.orchestra.run.smithi016.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-03-31T15:14:33.426 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:6b:fd:e1 brd ff:ff:ff:ff:ff:ff 2024-03-31T15:14:33.426 INFO:teuthology.orchestra.run.smithi016.stdout: altname enp7s0f1 2024-03-31T15:14:33.426 INFO:teuthology.orchestra.run.smithi016.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-31T15:14:33.426 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:bd:15:64 brd ff:ff:ff:ff:ff:ff 2024-03-31T15:14:33.426 INFO:teuthology.orchestra.run.smithi016.stdout: altname enp1s0f0 2024-03-31T15:14:33.426 INFO:teuthology.orchestra.run.smithi016.stdout: inet 172.21.15.16/20 brd 172.21.15.255 scope global dynamic ens1f0 2024-03-31T15:14:33.426 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft 41807sec preferred_lft 41807sec 2024-03-31T15:14:33.426 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 fe80::ec4:7aff:febd:1564/64 scope link 2024-03-31T15:14:33.427 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2024-03-31T15:14:33.427 INFO:teuthology.orchestra.run.smithi016.stdout:5: ens1f1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-03-31T15:14:33.427 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:bd:15:65 brd ff:ff:ff:ff:ff:ff 2024-03-31T15:14:33.427 INFO:teuthology.orchestra.run.smithi016.stdout: altname enp1s0f1 2024-03-31T15:14:33.427 INFO:teuthology.orchestra.run.smithi016.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2024-03-31T15:14:33.427 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 02:42:d4:1f:7d:73 brd ff:ff:ff:ff:ff:ff 2024-03-31T15:14:33.427 INFO:teuthology.orchestra.run.smithi016.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2024-03-31T15:14:33.427 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2024-03-31T15:14:33.427 INFO:teuthology.orchestra.run.smithi016.stdout:7: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-31T15:14:33.427 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 5e:1f:84:6d:79:ae brd ff:ff:ff:ff:ff:ff 2024-03-31T15:14:33.427 INFO:teuthology.orchestra.run.smithi016.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-31T15:14:33.427 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2024-03-31T15:14:33.427 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 fe80::5c1f:84ff:fe6d:79ae/64 scope link 2024-03-31T15:14:33.427 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2024-03-31T15:14:33.427 INFO:teuthology.orchestra.run.smithi016.stdout:8: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-31T15:14:33.427 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0e:5c:f6:1f:82:2f brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-31T15:14:33.428 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 fe80::c5c:f6ff:fe1f:822f/64 scope link 2024-03-31T15:14:33.428 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2024-03-31T15:14:33.428 INFO:teuthology.orchestra.run.smithi016.stdout:9: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-31T15:14:33.428 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether c6:a6:3d:69:d0:db brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-31T15:14:33.428 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 fe80::c4a6:3dff:fe69:d0db/64 scope link 2024-03-31T15:14:33.428 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2024-03-31T15:14:33.428 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:14:33.428 DEBUG:teuthology.orchestra.run.smithi016:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-31T15:14:33.486 INFO:teuthology.orchestra.run.smithi016.stdout:1 2024-03-31T15:14:33.488 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:14:33.488 DEBUG:teuthology.orchestra.run.smithi016:> route 2024-03-31T15:14:33.542 INFO:teuthology.orchestra.run.smithi016.stdout:Kernel IP routing table 2024-03-31T15:14:33.542 INFO:teuthology.orchestra.run.smithi016.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-31T15:14:33.542 INFO:teuthology.orchestra.run.smithi016.stdout:default _gateway 0.0.0.0 UG 0 0 0 ens1f0 2024-03-31T15:14:33.543 INFO:teuthology.orchestra.run.smithi016.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2024-03-31T15:14:33.543 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 0 0 0 ens1f0 2024-03-31T15:14:33.543 INFO:teuthology.orchestra.run.smithi016.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-31T15:14:33.543 DEBUG:teuthology.orchestra.run.smithi016:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-31T15:14:33.543 DEBUG:teuthology.orchestra.run.smithi016:> set -e 2024-03-31T15:14:33.543 DEBUG:teuthology.orchestra.run.smithi016:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-03-31T15:14:33.543 DEBUG:teuthology.orchestra.run.smithi016:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-03-31T15:14:33.543 DEBUG:teuthology.orchestra.run.smithi016:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-03-31T15:14:33.543 DEBUG:teuthology.orchestra.run.smithi016:> ') 2024-03-31T15:14:33.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:33 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-31T15:14:33.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:33 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-31T15:14:33.679 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:14:33.679 DEBUG:teuthology.orchestra.run.smithi016:> ip netns list 2024-03-31T15:14:33.688 INFO:teuthology.orchestra.run.smithi016.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-31T15:14:33.688 INFO:teuthology.orchestra.run.smithi016.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-31T15:14:33.689 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-31T15:14:33.689 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:14:33.689 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip link set brx.0 up 2024-03-31T15:14:33.754 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-03-31T15:14:33.754 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-31T15:14:33.754 DEBUG:teuthology.orchestra.run.smithi016:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-31T15:14:33.810 INFO:teuthology.orchestra.run.smithi016.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-31T15:14:33.810 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-31T15:14:33.810 DEBUG:teuthology.orchestra.run.smithi016:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-31T15:14:33.865 DEBUG:teuthology.orchestra.run.smithi016:> sudo modprobe fuse 2024-03-31T15:14:33.929 DEBUG:teuthology.orchestra.run.smithi016:> cat /proc/self/mounts | awk '{print $2}' 2024-03-31T15:14:33.983 INFO:teuthology.orchestra.run.smithi016.stdout:/sys 2024-03-31T15:14:33.983 INFO:teuthology.orchestra.run.smithi016.stdout:/proc 2024-03-31T15:14:33.983 INFO:teuthology.orchestra.run.smithi016.stdout:/dev 2024-03-31T15:14:33.983 INFO:teuthology.orchestra.run.smithi016.stdout:/dev/pts 2024-03-31T15:14:33.983 INFO:teuthology.orchestra.run.smithi016.stdout:/run 2024-03-31T15:14:33.983 INFO:teuthology.orchestra.run.smithi016.stdout:/ 2024-03-31T15:14:33.983 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/kernel/security 2024-03-31T15:14:33.983 INFO:teuthology.orchestra.run.smithi016.stdout:/dev/shm 2024-03-31T15:14:33.983 INFO:teuthology.orchestra.run.smithi016.stdout:/run/lock 2024-03-31T15:14:33.984 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/cgroup 2024-03-31T15:14:33.984 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/pstore 2024-03-31T15:14:33.984 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/bpf 2024-03-31T15:14:33.984 INFO:teuthology.orchestra.run.smithi016.stdout:/proc/sys/fs/binfmt_misc 2024-03-31T15:14:33.984 INFO:teuthology.orchestra.run.smithi016.stdout:/dev/hugepages 2024-03-31T15:14:33.984 INFO:teuthology.orchestra.run.smithi016.stdout:/dev/mqueue 2024-03-31T15:14:33.984 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/kernel/debug 2024-03-31T15:14:33.984 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/kernel/tracing 2024-03-31T15:14:33.984 INFO:teuthology.orchestra.run.smithi016.stdout:/proc/fs/nfsd 2024-03-31T15:14:33.984 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/fuse/connections 2024-03-31T15:14:33.984 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/kernel/config 2024-03-31T15:14:33.984 INFO:teuthology.orchestra.run.smithi016.stdout:/run/credentials/systemd-sysusers.service 2024-03-31T15:14:33.984 INFO:teuthology.orchestra.run.smithi016.stdout:/run/rpc_pipefs 2024-03-31T15:14:33.984 INFO:teuthology.orchestra.run.smithi016.stdout:/run/user/1000 2024-03-31T15:14:33.985 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/ceph 2024-03-31T15:14:33.985 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/ceph/osd/ceph-0 2024-03-31T15:14:33.985 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/ceph/osd/ceph-1 2024-03-31T15:14:33.985 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/ceph/osd/ceph-2 2024-03-31T15:14:33.985 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/ceph/osd/ceph-3 2024-03-31T15:14:33.985 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns 2024-03-31T15:14:33.985 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-31T15:14:33.985 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-31T15:14:33.985 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-31T15:14:33.985 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-31T15:14:33.986 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:14:33.986 DEBUG:teuthology.orchestra.run.smithi016:> ls /sys/fs/fuse/connections 2024-03-31T15:14:34.037 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-31T15:14:34.037 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-03-31T15:14:34.088 DEBUG:teuthology.orchestra.run.smithi016:> sudo modprobe fuse 2024-03-31T15:14:34.108 DEBUG:teuthology.orchestra.run.smithi016:> cat /proc/self/mounts | awk '{print $2}' 2024-03-31T15:14:34.163 INFO:teuthology.orchestra.run.smithi016.stdout:/sys 2024-03-31T15:14:34.164 INFO:teuthology.orchestra.run.smithi016.stdout:/proc 2024-03-31T15:14:34.164 INFO:teuthology.orchestra.run.smithi016.stdout:/dev 2024-03-31T15:14:34.164 INFO:teuthology.orchestra.run.smithi016.stdout:/dev/pts 2024-03-31T15:14:34.164 INFO:teuthology.orchestra.run.smithi016.stdout:/run 2024-03-31T15:14:34.164 INFO:teuthology.orchestra.run.smithi016.stdout:/ 2024-03-31T15:14:34.164 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/kernel/security 2024-03-31T15:14:34.164 INFO:teuthology.orchestra.run.smithi016.stdout:/dev/shm 2024-03-31T15:14:34.164 INFO:teuthology.orchestra.run.smithi016.stdout:/run/lock 2024-03-31T15:14:34.164 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/cgroup 2024-03-31T15:14:34.164 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/pstore 2024-03-31T15:14:34.164 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/bpf 2024-03-31T15:14:34.164 INFO:teuthology.orchestra.run.smithi016.stdout:/proc/sys/fs/binfmt_misc 2024-03-31T15:14:34.165 INFO:teuthology.orchestra.run.smithi016.stdout:/dev/hugepages 2024-03-31T15:14:34.165 INFO:teuthology.orchestra.run.smithi016.stdout:/dev/mqueue 2024-03-31T15:14:34.165 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/kernel/debug 2024-03-31T15:14:34.165 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/kernel/tracing 2024-03-31T15:14:34.165 INFO:teuthology.orchestra.run.smithi016.stdout:/proc/fs/nfsd 2024-03-31T15:14:34.165 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/fuse/connections 2024-03-31T15:14:34.165 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/kernel/config 2024-03-31T15:14:34.165 INFO:teuthology.orchestra.run.smithi016.stdout:/run/credentials/systemd-sysusers.service 2024-03-31T15:14:34.165 INFO:teuthology.orchestra.run.smithi016.stdout:/run/rpc_pipefs 2024-03-31T15:14:34.165 INFO:teuthology.orchestra.run.smithi016.stdout:/run/user/1000 2024-03-31T15:14:34.165 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/ceph 2024-03-31T15:14:34.165 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/ceph/osd/ceph-0 2024-03-31T15:14:34.165 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/ceph/osd/ceph-1 2024-03-31T15:14:34.165 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/ceph/osd/ceph-2 2024-03-31T15:14:34.166 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/ceph/osd/ceph-3 2024-03-31T15:14:34.166 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns 2024-03-31T15:14:34.166 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-31T15:14:34.166 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-31T15:14:34.166 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-31T15:14:34.166 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-31T15:14:34.167 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:14:34.167 DEBUG:teuthology.orchestra.run.smithi016:> ls /sys/fs/fuse/connections 2024-03-31T15:14:34.184 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi016.stderr:2024-03-31T15:14:34.189+0000 7f7d9ee48580 -1 init, newargv = 0x5646aeadc460 newargc=15 2024-03-31T15:14:34.185 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi016.stderr:ceph-fuse[51624]: starting ceph client 2024-03-31T15:14:34.191 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi016.stderr:ceph-fuse[51624]: starting fuse 2024-03-31T15:14:34.195 INFO:teuthology.orchestra.run.smithi016.stdout:43 2024-03-31T15:14:34.196 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [43] 2024-03-31T15:14:34.196 DEBUG:teuthology.orchestra.run.smithi016:> sudo stdin-killer -- python3 -c ' 2024-03-31T15:14:34.196 DEBUG:teuthology.orchestra.run.smithi016:> import glob 2024-03-31T15:14:34.196 DEBUG:teuthology.orchestra.run.smithi016:> import re 2024-03-31T15:14:34.196 DEBUG:teuthology.orchestra.run.smithi016:> import os 2024-03-31T15:14:34.196 DEBUG:teuthology.orchestra.run.smithi016:> import subprocess 2024-03-31T15:14:34.196 DEBUG:teuthology.orchestra.run.smithi016:> 2024-03-31T15:14:34.196 DEBUG:teuthology.orchestra.run.smithi016:> def _find_admin_socket(client_name): 2024-03-31T15:14:34.196 DEBUG:teuthology.orchestra.run.smithi016:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-03-31T15:14:34.197 DEBUG:teuthology.orchestra.run.smithi016:> files = glob.glob(asok_path) 2024-03-31T15:14:34.197 DEBUG:teuthology.orchestra.run.smithi016:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-03-31T15:14:34.197 DEBUG:teuthology.orchestra.run.smithi016:> 2024-03-31T15:14:34.197 DEBUG:teuthology.orchestra.run.smithi016:> # Given a non-glob path, it better be there 2024-03-31T15:14:34.197 DEBUG:teuthology.orchestra.run.smithi016:> if "*" not in asok_path: 2024-03-31T15:14:34.197 DEBUG:teuthology.orchestra.run.smithi016:> assert(len(files) == 1) 2024-03-31T15:14:34.197 DEBUG:teuthology.orchestra.run.smithi016:> return files[0] 2024-03-31T15:14:34.197 DEBUG:teuthology.orchestra.run.smithi016:> 2024-03-31T15:14:34.197 DEBUG:teuthology.orchestra.run.smithi016:> for f in files: 2024-03-31T15:14:34.197 DEBUG:teuthology.orchestra.run.smithi016:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-31T15:14:34.197 DEBUG:teuthology.orchestra.run.smithi016:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-31T15:14:34.197 DEBUG:teuthology.orchestra.run.smithi016:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-31T15:14:34.197 DEBUG:teuthology.orchestra.run.smithi016:> contents = proc_f.read() 2024-03-31T15:14:34.197 DEBUG:teuthology.orchestra.run.smithi016:> if mountpoint in contents: 2024-03-31T15:14:34.197 DEBUG:teuthology.orchestra.run.smithi016:> return f 2024-03-31T15:14:34.198 DEBUG:teuthology.orchestra.run.smithi016:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-31T15:14:34.198 DEBUG:teuthology.orchestra.run.smithi016:> 2024-03-31T15:14:34.198 DEBUG:teuthology.orchestra.run.smithi016:> print(_find_admin_socket("client.0")) 2024-03-31T15:14:34.198 DEBUG:teuthology.orchestra.run.smithi016:> ' 2024-03-31T15:14:34.324 INFO:teuthology.orchestra.run.smithi016.stdout:/var/run/ceph/ceph-client.0.51624.asok 2024-03-31T15:14:34.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:34 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-31T15:14:34.335 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.51624.asok 2024-03-31T15:14:34.335 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:14:34.335 DEBUG:teuthology.orchestra.run.smithi016:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.51624.asok status 2024-03-31T15:14:34.508 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-03-31T15:14:34.508 INFO:teuthology.orchestra.run.smithi016.stdout: "metadata": { 2024-03-31T15:14:34.508 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph_sha1": "d2271579d5e735cfd6d436b8349c004f06527cb0", 2024-03-31T15:14:34.508 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph_version": "ceph version 19.0.0-2574-gd2271579 (d2271579d5e735cfd6d436b8349c004f06527cb0) squid (dev)", 2024-03-31T15:14:34.508 INFO:teuthology.orchestra.run.smithi016.stdout: "entity_id": "0", 2024-03-31T15:14:34.508 INFO:teuthology.orchestra.run.smithi016.stdout: "hostname": "smithi016", 2024-03-31T15:14:34.508 INFO:teuthology.orchestra.run.smithi016.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-03-31T15:14:34.508 INFO:teuthology.orchestra.run.smithi016.stdout: "pid": "51624", 2024-03-31T15:14:34.508 INFO:teuthology.orchestra.run.smithi016.stdout: "root": "/" 2024-03-31T15:14:34.508 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-31T15:14:34.508 INFO:teuthology.orchestra.run.smithi016.stdout: "dentry_count": 0, 2024-03-31T15:14:34.509 INFO:teuthology.orchestra.run.smithi016.stdout: "dentry_pinned_count": 0, 2024-03-31T15:14:34.509 INFO:teuthology.orchestra.run.smithi016.stdout: "id": 9477, 2024-03-31T15:14:34.509 INFO:teuthology.orchestra.run.smithi016.stdout: "inst": { 2024-03-31T15:14:34.509 INFO:teuthology.orchestra.run.smithi016.stdout: "name": { 2024-03-31T15:14:34.509 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "client", 2024-03-31T15:14:34.509 INFO:teuthology.orchestra.run.smithi016.stdout: "num": 9477 2024-03-31T15:14:34.509 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-31T15:14:34.509 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": { 2024-03-31T15:14:34.509 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-03-31T15:14:34.509 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "192.168.0.1:0", 2024-03-31T15:14:34.509 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 4063426641 2024-03-31T15:14:34.509 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-31T15:14:34.509 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-31T15:14:34.509 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": { 2024-03-31T15:14:34.510 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-03-31T15:14:34.510 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "192.168.0.1:0", 2024-03-31T15:14:34.510 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 4063426641 2024-03-31T15:14:34.510 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-31T15:14:34.510 INFO:teuthology.orchestra.run.smithi016.stdout: "inst_str": "client.9477 192.168.0.1:0/4063426641", 2024-03-31T15:14:34.510 INFO:teuthology.orchestra.run.smithi016.stdout: "addr_str": "192.168.0.1:0/4063426641", 2024-03-31T15:14:34.510 INFO:teuthology.orchestra.run.smithi016.stdout: "inode_count": 1, 2024-03-31T15:14:34.510 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_epoch": 99, 2024-03-31T15:14:34.510 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_epoch": 111, 2024-03-31T15:14:34.510 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_epoch_barrier": 0, 2024-03-31T15:14:34.510 INFO:teuthology.orchestra.run.smithi016.stdout: "blocklisted": false, 2024-03-31T15:14:34.510 INFO:teuthology.orchestra.run.smithi016.stdout: "fs_name": "cephfs" 2024-03-31T15:14:34.510 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-03-31T15:14:34.523 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:14:34.523 DEBUG:teuthology.orchestra.run.smithi016:> stat --file-system '--printf=%T 2024-03-31T15:14:34.523 DEBUG:teuthology.orchestra.run.smithi016:> ' -- /home/ubuntu/cephtest/mnt.0 2024-03-31T15:14:34.533 INFO:teuthology.orchestra.run.smithi016.stdout:fuseblk 2024-03-31T15:14:34.533 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-03-31T15:14:34.533 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:14:34.533 DEBUG:teuthology.orchestra.run.smithi016:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-31T15:14:34.596 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:14:34.596 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir parent/unflushed) 2024-03-31T15:14:34.652 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:14:34.652 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=./parent/unflushed/jfile bs=1M count=8) 2024-03-31T15:14:34.787 INFO:teuthology.orchestra.run.smithi016.stderr:8+0 records in 2024-03-31T15:14:34.787 INFO:teuthology.orchestra.run.smithi016.stderr:8+0 records out 2024-03-31T15:14:34.787 INFO:teuthology.orchestra.run.smithi016.stderr:8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.0791659 s, 106 MB/s 2024-03-31T15:14:34.788 DEBUG:teuthology.orchestra.run.smithi016:> stdin-killer -- python3 -c ' 2024-03-31T15:14:34.788 DEBUG:teuthology.orchestra.run.smithi016:> import os 2024-03-31T15:14:34.788 DEBUG:teuthology.orchestra.run.smithi016:> import stat 2024-03-31T15:14:34.788 DEBUG:teuthology.orchestra.run.smithi016:> 2024-03-31T15:14:34.789 DEBUG:teuthology.orchestra.run.smithi016:> print(os.stat("/home/ubuntu/cephtest/mnt.0/./parent/unflushed").st_ino) 2024-03-31T15:14:34.789 DEBUG:teuthology.orchestra.run.smithi016:> ' 2024-03-31T15:14:34.863 INFO:teuthology.orchestra.run.smithi016.stdout:1099511627779 2024-03-31T15:14:34.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:34 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-31T15:14:34.873 DEBUG:teuthology.orchestra.run.smithi016:> stdin-killer -- python3 -c ' 2024-03-31T15:14:34.873 DEBUG:teuthology.orchestra.run.smithi016:> import os 2024-03-31T15:14:34.873 DEBUG:teuthology.orchestra.run.smithi016:> import stat 2024-03-31T15:14:34.873 DEBUG:teuthology.orchestra.run.smithi016:> 2024-03-31T15:14:34.873 DEBUG:teuthology.orchestra.run.smithi016:> print(os.stat("/home/ubuntu/cephtest/mnt.0/./parent/unflushed/jfile").st_ino) 2024-03-31T15:14:34.873 DEBUG:teuthology.orchestra.run.smithi016:> ' 2024-03-31T15:14:34.983 INFO:teuthology.orchestra.run.smithi016.stdout:1099511627780 2024-03-31T15:14:34.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:34 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-31T15:14:34.993 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-03-31T15:14:34.993 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-31T15:14:35.045 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-31T15:14:35.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:35.237+0000 7fde16a00640 1 Processor -- start 2024-03-31T15:14:35.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:35.237+0000 7fde16a00640 1 -- start start 2024-03-31T15:14:35.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:35.237+0000 7fde16a00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fde10147e70 0x7fde1006b620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:35.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:35.237+0000 7fde16a00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fde100d0c50 0x7fde10072fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:35.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:35.237+0000 7fde16a00640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fde101621b0 con 0x7fde10073510 2024-03-31T15:14:35.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:35.237+0000 7fde16a00640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7fde100739a0 con 0x7fde10147e70 2024-03-31T15:14:35.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:35.237+0000 7fde16a00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fde10162490 con 0x7fde100d0c50 2024-03-31T15:14:35.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:35.237+0000 7fde0fe00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fde10147e70 0x7fde1006b620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:35.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:35.237+0000 7fde14a00640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fde10073510 0x7fde100cdab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:50224/0 (socket says 172.21.15.16:50224) 2024-03-31T15:14:35.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:35.237+0000 7fde14a00640 1 -- 172.21.15.16:0/2328431943 learned_addr learned my addr 172.21.15.16:0/2328431943 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:14:35.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:35.237+0000 7fde07e00640 1 --2- 172.21.15.16:0/2328431943 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fde100d0c50 0x7fde10072fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:35.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:35.237+0000 7fde0fe00640 1 -- 172.21.15.16:0/2328431943 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fde100d0c50 msgr2=0x7fde10072fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:14:35.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:35.237+0000 7fde0fe00640 1 --2- 172.21.15.16:0/2328431943 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fde100d0c50 0x7fde10072fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:35.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:35.237+0000 7fde0fe00640 1 -- 172.21.15.16:0/2328431943 >> v1:172.21.15.16:6789/0 conn(0x7fde10073510 legacy=0x7fde100cdab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:35.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:35.237+0000 7fde0fe00640 1 -- 172.21.15.16:0/2328431943 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde10162770 con 0x7fde10147e70 2024-03-31T15:14:35.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:35.237+0000 7fde0f400640 1 -- 172.21.15.16:0/2328431943 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4008259429 0 0) 0x7fde101621b0 con 0x7fde10073510 2024-03-31T15:14:35.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:35.241+0000 7fde0fe00640 1 --2- 172.21.15.16:0/2328431943 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fde10147e70 0x7fde1006b620 secure :-1 s=READY pgs=1713 cs=0 l=1 rev1=1 crypto rx=0x7fde000097c0 tx=0x7fde0001c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15fff47e5619c049 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:35.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:35.241+0000 7fde0f400640 1 -- 172.21.15.16:0/2328431943 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde000b0070 con 0x7fde10147e70 2024-03-31T15:14:35.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:35.241+0000 7fde0f400640 1 -- 172.21.15.16:0/2328431943 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fde000a4e50 con 0x7fde10147e70 2024-03-31T15:14:35.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:35.241+0000 7fde0f400640 1 -- 172.21.15.16:0/2328431943 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde000ab820 con 0x7fde10147e70 2024-03-31T15:14:35.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:35.241+0000 7fde16a00640 1 -- 172.21.15.16:0/2328431943 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fde10147e70 msgr2=0x7fde1006b620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:35.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:35.241+0000 7fde16a00640 1 --2- 172.21.15.16:0/2328431943 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fde10147e70 0x7fde1006b620 secure :-1 s=READY pgs=1713 cs=0 l=1 rev1=1 crypto rx=0x7fde000097c0 tx=0x7fde0001c4a0 comp rx=0 tx=0).stop 2024-03-31T15:14:35.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:35.241+0000 7fde16a00640 1 -- 172.21.15.16:0/2328431943 shutdown_connections 2024-03-31T15:14:35.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:35.241+0000 7fde16a00640 1 --2- 172.21.15.16:0/2328431943 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fde100d0c50 0x7fde10072fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:35.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:35.241+0000 7fde16a00640 1 --2- 172.21.15.16:0/2328431943 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fde10147e70 0x7fde1006b620 unknown :-1 s=CLOSED pgs=1713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:35.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:35.241+0000 7fde16a00640 1 -- 172.21.15.16:0/2328431943 >> 172.21.15.16:0/2328431943 conn(0x7fde100d1060 msgr2=0x7fde100d1470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:14:35.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:35.241+0000 7fde16a00640 1 -- 172.21.15.16:0/2328431943 shutdown_connections 2024-03-31T15:14:35.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:35.241+0000 7fde16a00640 1 -- 172.21.15.16:0/2328431943 wait complete. 2024-03-31T15:14:35.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:35.245+0000 7fde16a00640 1 Processor -- start 2024-03-31T15:14:35.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:35.245+0000 7fde16a00640 1 -- start start 2024-03-31T15:14:35.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:35.245+0000 7fde16a00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fde10073510 0x7fde1015c280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:35.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:35.245+0000 7fde16a00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fde100d0c50 0x7fde10145720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:35.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:35.245+0000 7fde16a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fde10147e70 0x7fde10145c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:35.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:35.245+0000 7fde16a00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fde1006bce0 con 0x7fde10147e70 2024-03-31T15:14:35.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:35.245+0000 7fde16a00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fde1006bb60 con 0x7fde10073510 2024-03-31T15:14:35.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:35.245+0000 7fde16a00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fde1016b4b0 con 0x7fde100d0c50 2024-03-31T15:14:35.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:35.245+0000 7fde14a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fde10147e70 0x7fde10145c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:35.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:35.249+0000 7fde14a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fde10147e70 0x7fde10145c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50804/0 (socket says 172.21.15.16:50804) 2024-03-31T15:14:35.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:35.249+0000 7fde14a00640 1 -- 172.21.15.16:0/4222146802 learned_addr learned my addr 172.21.15.16:0/4222146802 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:14:35.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:35.249+0000 7fde07e00640 1 --2- 172.21.15.16:0/4222146802 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fde100d0c50 0x7fde10145720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:35.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:35.249+0000 7fde0fe00640 1 --2- 172.21.15.16:0/4222146802 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fde10073510 0x7fde1015c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:35.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:35.249+0000 7fde14a00640 1 -- 172.21.15.16:0/4222146802 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fde100d0c50 msgr2=0x7fde10145720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:35.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:35.249+0000 7fde14a00640 1 --2- 172.21.15.16:0/4222146802 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fde100d0c50 0x7fde10145720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:35.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:35.249+0000 7fde14a00640 1 -- 172.21.15.16:0/4222146802 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fde10073510 msgr2=0x7fde1015c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:35.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:35.249+0000 7fde14a00640 1 --2- 172.21.15.16:0/4222146802 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fde10073510 0x7fde1015c280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:35.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:35.249+0000 7fde14a00640 1 -- 172.21.15.16:0/4222146802 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde1015d7d0 con 0x7fde10147e70 2024-03-31T15:14:35.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:35.249+0000 7fde14a00640 1 --2- 172.21.15.16:0/4222146802 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fde10147e70 0x7fde10145c60 secure :-1 s=READY pgs=1714 cs=0 l=1 rev1=1 crypto rx=0x7fde0800c700 tx=0x7fde0800cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:35.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:35.249+0000 7fde0cc00640 1 -- 172.21.15.16:0/4222146802 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde08018070 con 0x7fde10147e70 2024-03-31T15:14:35.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:35.249+0000 7fde0cc00640 1 -- 172.21.15.16:0/4222146802 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fde0800ce10 con 0x7fde10147e70 2024-03-31T15:14:35.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:35.249+0000 7fde16a00640 1 -- 172.21.15.16:0/4222146802 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fde1015dab0 con 0x7fde10147e70 2024-03-31T15:14:35.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:35.249+0000 7fde0cc00640 1 -- 172.21.15.16:0/4222146802 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde0800a880 con 0x7fde10147e70 2024-03-31T15:14:35.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:35.249+0000 7fde16a00640 1 -- 172.21.15.16:0/4222146802 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fde100b4190 con 0x7fde10147e70 2024-03-31T15:14:35.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:35.249+0000 7fde16a00640 1 -- 172.21.15.16:0/4222146802 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fde10069950 con 0x7fde10147e70 2024-03-31T15:14:35.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:35.249+0000 7fde0cc00640 1 -- 172.21.15.16:0/4222146802 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fde0801b070 con 0x7fde10147e70 2024-03-31T15:14:35.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:35.253+0000 7fde0cc00640 1 --2- 172.21.15.16:0/4222146802 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fddec068380 0x7fddec06a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:35.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:35.253+0000 7fde0cc00640 1 -- 172.21.15.16:0/4222146802 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 5767+0+0 (secure 0 0 0) 0x7fde08085f70 con 0x7fde10147e70 2024-03-31T15:14:35.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:35.253+0000 7fde0fe00640 1 --2- 172.21.15.16:0/4222146802 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fddec068380 0x7fddec06a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:35.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:35.253+0000 7fde0fe00640 1 --2- 172.21.15.16:0/4222146802 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fddec068380 0x7fddec06a840 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7fde000ad000 tx=0x7fde000a6000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:35.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:35.257+0000 7fde0cc00640 1 -- 172.21.15.16:0/4222146802 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fde08055e00 con 0x7fde10147e70 2024-03-31T15:14:35.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:35.425+0000 7fde16a00640 1 -- 172.21.15.16:0/4222146802 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fde1015dc80 con 0x7fde10147e70 2024-03-31T15:14:35.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:35.425+0000 7fde0cc00640 1 -- 172.21.15.16:0/4222146802 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v111) ==== 74+0+16648 (secure 0 0 0) 0x7fde0805a100 con 0x7fde10147e70 2024-03-31T15:14:35.421 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:14:35.422 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":111,"fsid":"286eb5da-fb47-4550-b290-d9519c13b9a5","created":"2024-03-31T15:03:11.438952+0000","modified":"2024-03-31T15:14:19.306886+0000","last_up_change":"2024-03-31T15:03:22.885535+0000","last_in_change":"2024-03-31T15:03:13.941667+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-31T15:03:22.550787+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-03-31T15:14:06.417304+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"108","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-03-31T15:14:07.622562+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"109","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"92a66450-ef19-40d8-ac3d-d760b23b0bda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6801","nonce":932712244}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6803","nonce":932712244}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6807","nonce":932712244}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6805","nonce":932712244}]},"public_addr":"172.21.15.16:6801/932712244","cluster_addr":"172.21.15.16:6803/932712244","heartbeat_back_addr":"172.21.15.16:6807/932712244","heartbeat_front_addr":"172.21.15.16:6805/932712244","state":["exists","up"]},{"osd":1,"uuid":"da9d1f2b-1968-4959-b843-faa231988e92","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6810","nonce":1398737533}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6811","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6813","nonce":1398737533}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6821","nonce":1398737533}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6815","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6816","nonce":1398737533}]},"public_addr":"172.21.15.16:6810/1398737533","cluster_addr":"172.21.15.16:6813/1398737533","heartbeat_back_addr":"172.21.15.16:6821/1398737533","heartbeat_front_addr":"172.21.15.16:6816/1398737533","state":["exists","up"]},{"osd":2,"uuid":"01a73662-9a21-4b1f-bcbb-42b9636a4638","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6819","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6822","nonce":3018526646}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6825","nonce":3018526646}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6831","nonce":3018526646}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6827","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6829","nonce":3018526646}]},"public_addr":"172.21.15.16:6822/3018526646","cluster_addr":"172.21.15.16:6825/3018526646","heartbeat_back_addr":"172.21.15.16:6831/3018526646","heartbeat_front_addr":"172.21.15.16:6829/3018526646","state":["exists","up"]},{"osd":3,"uuid":"c7771fd3-e18e-4500-ac38-23cf10ebcd75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6809","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6812","nonce":3360352974}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6817","nonce":3360352974}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6828","nonce":3360352974}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6823","nonce":3360352974}]},"public_addr":"172.21.15.16:6812/3360352974","cluster_addr":"172.21.15.16:6817/3360352974","heartbeat_back_addr":"172.21.15.16:6828/3360352974","heartbeat_front_addr":"172.21.15.16:6823/3360352974","state":["exists","up"]},{"osd":4,"uuid":"88260730-1482-4fbd-a40f-cb1124e9df4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6801","nonce":1856844856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6803","nonce":1856844856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6807","nonce":1856844856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6805","nonce":1856844856}]},"public_addr":"172.21.15.89:6801/1856844856","cluster_addr":"172.21.15.89:6803/1856844856","heartbeat_back_addr":"172.21.15.89:6807/1856844856","heartbeat_front_addr":"172.21.15.89:6805/1856844856","state":["exists","up"]},{"osd":5,"uuid":"4cb2c00c-fae0-4dd8-9d27-ec4f0093ccf7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6814","nonce":50395527}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6817","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6820","nonce":50395527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6831","nonce":50395527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6823","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6826","nonce":50395527}]},"public_addr":"172.21.15.89:6814/50395527","cluster_addr":"172.21.15.89:6820/50395527","heartbeat_back_addr":"172.21.15.89:6831/50395527","heartbeat_front_addr":"172.21.15.89:6826/50395527","state":["exists","up"]},{"osd":6,"uuid":"81ce0c19-2554-4cae-acbe-66a05e9e9b9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6809","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6812","nonce":3470471752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6815","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6818","nonce":3470471752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6827","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6829","nonce":3470471752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6821","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6824","nonce":3470471752}]},"public_addr":"172.21.15.89:6812/3470471752","cluster_addr":"172.21.15.89:6818/3470471752","heartbeat_back_addr":"172.21.15.89:6829/3470471752","heartbeat_front_addr":"172.21.15.89:6824/3470471752","state":["exists","up"]},{"osd":7,"uuid":"85e4313b-d37d-442e-8042-153bf20a847d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6811","nonce":1796778630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6813","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6816","nonce":1796778630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6825","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6828","nonce":1796778630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6819","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6822","nonce":1796778630}]},"public_addr":"172.21.15.89:6811/1796778630","cluster_addr":"172.21.15.89:6816/1796778630","heartbeat_back_addr":"172.21.15.89:6828/1796778630","heartbeat_front_addr":"172.21.15.89:6822/1796778630","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.439088+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.529970+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.680512+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:21.044161+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.582111+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.536558+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.409522+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.590229+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"17.1","mappings":[{"from":0,"to":5}]},{"pgid":"17.21","mappings":[{"from":0,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-31T15:14:35.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:35.429+0000 7fde16a00640 1 -- 172.21.15.16:0/4222146802 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fddec068380 msgr2=0x7fddec06a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:35.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:35.429+0000 7fde16a00640 1 --2- 172.21.15.16:0/4222146802 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fddec068380 0x7fddec06a840 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7fde000ad000 tx=0x7fde000a6000 comp rx=0 tx=0).stop 2024-03-31T15:14:35.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:35.429+0000 7fde16a00640 1 -- 172.21.15.16:0/4222146802 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fde10147e70 msgr2=0x7fde10145c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:35.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:35.429+0000 7fde16a00640 1 --2- 172.21.15.16:0/4222146802 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fde10147e70 0x7fde10145c60 secure :-1 s=READY pgs=1714 cs=0 l=1 rev1=1 crypto rx=0x7fde0800c700 tx=0x7fde0800cbc0 comp rx=0 tx=0).stop 2024-03-31T15:14:35.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:35.429+0000 7fde16a00640 1 -- 172.21.15.16:0/4222146802 shutdown_connections 2024-03-31T15:14:35.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:35.429+0000 7fde16a00640 1 --2- 172.21.15.16:0/4222146802 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fddec068380 0x7fddec06a840 unknown :-1 s=CLOSED pgs=726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:35.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:35.429+0000 7fde16a00640 1 --2- 172.21.15.16:0/4222146802 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fde10147e70 0x7fde10145c60 unknown :-1 s=CLOSED pgs=1714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:35.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:35.429+0000 7fde16a00640 1 --2- 172.21.15.16:0/4222146802 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fde100d0c50 0x7fde10145720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:35.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:35.429+0000 7fde16a00640 1 --2- 172.21.15.16:0/4222146802 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fde10073510 0x7fde1015c280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:35.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:35.429+0000 7fde16a00640 1 -- 172.21.15.16:0/4222146802 >> 172.21.15.16:0/4222146802 conn(0x7fde100d1060 msgr2=0x7fde10072ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:14:35.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:35.429+0000 7fde16a00640 1 -- 172.21.15.16:0/4222146802 shutdown_connections 2024-03-31T15:14:35.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:35.429+0000 7fde16a00640 1 -- 172.21.15.16:0/4222146802 wait complete. 2024-03-31T15:14:35.443 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-31T15:03:22.550787+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 16, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-31T15:14:06.417304+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '108', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 17, 'pool_name': 'cephfs_data', 'create_time': '2024-03-31T15:14:07.622562+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '109', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-31T15:14:35.443 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-31T15:14:35.587 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:35.593+0000 7f9ea5e00640 1 Processor -- start 2024-03-31T15:14:35.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:35.593+0000 7f9ea5e00640 1 -- start start 2024-03-31T15:14:35.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:35.593+0000 7f9ea5e00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f9ea01411b0 0x7f9ea006f920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:35.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:35.593+0000 7f9ea5e00640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9ea0144980 con 0x7f9ea0141590 2024-03-31T15:14:35.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:35.593+0000 7f9ea5e00640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9ea0161fe0 con 0x7f9ea0144340 2024-03-31T15:14:35.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:35.593+0000 7f9ea5e00640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f9ea0162910 con 0x7f9ea01411b0 2024-03-31T15:14:35.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:35.597+0000 7f9e9ea00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f9ea01411b0 0x7f9ea006f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:35.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:35.597+0000 7f9e97e00640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f9ea0141590 0x7f9ea0143c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:50226/0 (socket says 172.21.15.16:50226) 2024-03-31T15:14:35.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:35.597+0000 7f9e97e00640 1 -- 172.21.15.16:0/2467580210 learned_addr learned my addr 172.21.15.16:0/2467580210 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:14:35.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:35.597+0000 7f9e9ea00640 1 -- 172.21.15.16:0/2467580210 >> v1:172.21.15.89:6789/0 conn(0x7f9ea0144340 legacy=0x7f9ea015e900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:35.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:35.597+0000 7f9e9ea00640 1 -- 172.21.15.16:0/2467580210 >> v1:172.21.15.16:6789/0 conn(0x7f9ea0141590 legacy=0x7f9ea0143c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:35.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:35.597+0000 7f9e9ea00640 1 -- 172.21.15.16:0/2467580210 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ea01622c0 con 0x7f9ea01411b0 2024-03-31T15:14:35.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:35.597+0000 7f9e9ea00640 1 --2- 172.21.15.16:0/2467580210 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9ea01411b0 0x7f9ea006f920 secure :-1 s=READY pgs=1716 cs=0 l=1 rev1=1 crypto rx=0x7f9e900097c0 tx=0x7f9e9001c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5090eabd723bd24 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:35.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:35.597+0000 7f9e9e000640 1 -- 172.21.15.16:0/2467580210 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e900b0070 con 0x7f9ea01411b0 2024-03-31T15:14:35.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:35.597+0000 7f9e9e000640 1 -- 172.21.15.16:0/2467580210 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9e900ab600 con 0x7f9ea01411b0 2024-03-31T15:14:35.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:35.597+0000 7f9e9e000640 1 -- 172.21.15.16:0/2467580210 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e900ab9d0 con 0x7f9ea01411b0 2024-03-31T15:14:35.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:35.601+0000 7f9ea5e00640 1 -- 172.21.15.16:0/2467580210 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9ea01411b0 msgr2=0x7f9ea006f920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:35.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:35.601+0000 7f9ea5e00640 1 --2- 172.21.15.16:0/2467580210 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9ea01411b0 0x7f9ea006f920 secure :-1 s=READY pgs=1716 cs=0 l=1 rev1=1 crypto rx=0x7f9e900097c0 tx=0x7f9e9001c4a0 comp rx=0 tx=0).stop 2024-03-31T15:14:35.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:35.601+0000 7f9ea5e00640 1 -- 172.21.15.16:0/2467580210 shutdown_connections 2024-03-31T15:14:35.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:35.601+0000 7f9ea5e00640 1 --2- 172.21.15.16:0/2467580210 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9ea01411b0 0x7f9ea006f920 unknown :-1 s=CLOSED pgs=1716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:35.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:35.601+0000 7f9ea5e00640 1 -- 172.21.15.16:0/2467580210 >> 172.21.15.16:0/2467580210 conn(0x7f9ea0067930 msgr2=0x7f9ea00b3ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:14:35.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:35.601+0000 7f9ea5e00640 1 -- 172.21.15.16:0/2467580210 shutdown_connections 2024-03-31T15:14:35.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:35.601+0000 7f9ea5e00640 1 -- 172.21.15.16:0/2467580210 wait complete. 2024-03-31T15:14:35.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:35.601+0000 7f9ea5e00640 1 Processor -- start 2024-03-31T15:14:35.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:35.601+0000 7f9ea5e00640 1 -- start start 2024-03-31T15:14:35.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:35.601+0000 7f9ea5e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9ea01411b0 0x7f9ea0156be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:35.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:35.601+0000 7f9ea5e00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f9ea0141590 0x7f9ea0155540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:35.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:35.601+0000 7f9ea5e00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9ea0144340 0x7f9ea00b3be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:35.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:35.601+0000 7f9ea5e00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f9ea016c680 con 0x7f9ea01411b0 2024-03-31T15:14:35.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:35.601+0000 7f9ea5e00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f9ea016c500 con 0x7f9ea0144340 2024-03-31T15:14:35.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:35.601+0000 7f9ea5e00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f9ea016c800 con 0x7f9ea0141590 2024-03-31T15:14:35.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:35.601+0000 7f9e9ea00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9ea01411b0 0x7f9ea0156be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:35.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:35.601+0000 7f9e9ea00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9ea01411b0 0x7f9ea0156be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50812/0 (socket says 172.21.15.16:50812) 2024-03-31T15:14:35.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:35.601+0000 7f9e9ea00640 1 -- 172.21.15.16:0/2759640881 learned_addr learned my addr 172.21.15.16:0/2759640881 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:14:35.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:35.601+0000 7f9e9ea00640 1 -- 172.21.15.16:0/2759640881 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f9ea0141590 msgr2=0x7f9ea0155540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:35.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:35.601+0000 7f9e97e00640 1 --2- 172.21.15.16:0/2759640881 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f9ea0141590 0x7f9ea0155540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:35.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:35.601+0000 7f9e9f400640 1 --2- 172.21.15.16:0/2759640881 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9ea0144340 0x7f9ea00b3be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:35.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:35.601+0000 7f9e9ea00640 1 --2- 172.21.15.16:0/2759640881 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f9ea0141590 0x7f9ea0155540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:35.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:35.601+0000 7f9e9ea00640 1 -- 172.21.15.16:0/2759640881 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9ea0144340 msgr2=0x7f9ea00b3be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:35.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:35.601+0000 7f9e9ea00640 1 --2- 172.21.15.16:0/2759640881 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9ea0144340 0x7f9ea00b3be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:35.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:35.601+0000 7f9e9ea00640 1 -- 172.21.15.16:0/2759640881 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ea0155b40 con 0x7f9ea01411b0 2024-03-31T15:14:35.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:35.601+0000 7f9e97e00640 1 --2- 172.21.15.16:0/2759640881 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f9ea0141590 0x7f9ea0155540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:14:35.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:35.601+0000 7f9e9f400640 1 --2- 172.21.15.16:0/2759640881 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9ea0144340 0x7f9ea00b3be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:14:35.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:35.605+0000 7f9e9ea00640 1 --2- 172.21.15.16:0/2759640881 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9ea01411b0 0x7f9ea0156be0 secure :-1 s=READY pgs=1717 cs=0 l=1 rev1=1 crypto rx=0x7f9e900ae040 tx=0x7f9e9001ace0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:35.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:35.605+0000 7f9e97400640 1 -- 172.21.15.16:0/2759640881 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e900b0050 con 0x7f9ea01411b0 2024-03-31T15:14:35.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:35.605+0000 7f9e97400640 1 -- 172.21.15.16:0/2759640881 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9e900bc070 con 0x7f9ea01411b0 2024-03-31T15:14:35.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:35.605+0000 7f9e97400640 1 -- 172.21.15.16:0/2759640881 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e900b5a50 con 0x7f9ea01411b0 2024-03-31T15:14:35.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:35.605+0000 7f9ea5e00640 1 -- 172.21.15.16:0/2759640881 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9ea0155e20 con 0x7f9ea01411b0 2024-03-31T15:14:35.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:35.605+0000 7f9ea5e00640 1 -- 172.21.15.16:0/2759640881 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9ea01563e0 con 0x7f9ea01411b0 2024-03-31T15:14:35.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:35.605+0000 7f9ea5e00640 1 -- 172.21.15.16:0/2759640881 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9ea0069950 con 0x7f9ea01411b0 2024-03-31T15:14:35.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:35.605+0000 7f9e97400640 1 -- 172.21.15.16:0/2759640881 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f9e900ab4a0 con 0x7f9ea01411b0 2024-03-31T15:14:35.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:35.605+0000 7f9e97400640 1 --2- 172.21.15.16:0/2759640881 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f9e7c068380 0x7f9e7c06a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:35.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:35.609+0000 7f9e97400640 1 -- 172.21.15.16:0/2759640881 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 5767+0+0 (secure 0 0 0) 0x7f9e901213a0 con 0x7f9ea01411b0 2024-03-31T15:14:35.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:35.609+0000 7f9e97e00640 1 --2- 172.21.15.16:0/2759640881 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f9e7c068380 0x7f9e7c06a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:35.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:35.609+0000 7f9e97e00640 1 --2- 172.21.15.16:0/2759640881 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f9e7c068380 0x7f9e7c06a840 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f9ea01faed0 tx=0x7f9e88009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:35.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:35.609+0000 7f9e97400640 1 -- 172.21.15.16:0/2759640881 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f9e90016090 con 0x7f9ea01411b0 2024-03-31T15:14:35.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:35.853+0000 7f9ea5e00640 1 -- 172.21.15.16:0/2759640881 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f9ea00b4a20 con 0x7f9ea01411b0 2024-03-31T15:14:35.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:35.853+0000 7f9e97400640 1 -- 172.21.15.16:0/2759640881 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v111) ==== 93+0+10 (secure 0 0 0) 0x7f9e900f1910 con 0x7f9ea01411b0 2024-03-31T15:14:35.847 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2024-03-31T15:14:35.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:35.857+0000 7f9ea5e00640 1 -- 172.21.15.16:0/2759640881 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f9e7c068380 msgr2=0x7f9e7c06a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:35.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:35.857+0000 7f9ea5e00640 1 --2- 172.21.15.16:0/2759640881 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f9e7c068380 0x7f9e7c06a840 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f9ea01faed0 tx=0x7f9e88009040 comp rx=0 tx=0).stop 2024-03-31T15:14:35.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:35.857+0000 7f9ea5e00640 1 -- 172.21.15.16:0/2759640881 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9ea01411b0 msgr2=0x7f9ea0156be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:35.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:35.857+0000 7f9ea5e00640 1 --2- 172.21.15.16:0/2759640881 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9ea01411b0 0x7f9ea0156be0 secure :-1 s=READY pgs=1717 cs=0 l=1 rev1=1 crypto rx=0x7f9e900ae040 tx=0x7f9e9001ace0 comp rx=0 tx=0).stop 2024-03-31T15:14:35.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:35.857+0000 7f9ea5e00640 1 -- 172.21.15.16:0/2759640881 shutdown_connections 2024-03-31T15:14:35.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:35.857+0000 7f9ea5e00640 1 --2- 172.21.15.16:0/2759640881 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f9e7c068380 0x7f9e7c06a840 unknown :-1 s=CLOSED pgs=727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:35.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:35.857+0000 7f9ea5e00640 1 --2- 172.21.15.16:0/2759640881 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9ea0144340 0x7f9ea00b3be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:35.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:35.857+0000 7f9ea5e00640 1 --2- 172.21.15.16:0/2759640881 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f9ea0141590 0x7f9ea0155540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:35.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:35.857+0000 7f9ea5e00640 1 --2- 172.21.15.16:0/2759640881 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9ea01411b0 0x7f9ea0156be0 unknown :-1 s=CLOSED pgs=1717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:35.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:35.857+0000 7f9ea5e00640 1 -- 172.21.15.16:0/2759640881 >> 172.21.15.16:0/2759640881 conn(0x7f9ea0067930 msgr2=0x7f9ea01617b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:14:35.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:35.857+0000 7f9ea5e00640 1 -- 172.21.15.16:0/2759640881 shutdown_connections 2024-03-31T15:14:35.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:35.857+0000 7f9ea5e00640 1 -- 172.21.15.16:0/2759640881 wait complete. 2024-03-31T15:14:35.870 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-31T15:14:36.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:36.021+0000 7f325d400640 1 Processor -- start 2024-03-31T15:14:36.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:36.021+0000 7f325d400640 1 -- start start 2024-03-31T15:14:36.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:36.021+0000 7f325d400640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f3258161a70 0x7f32580c6990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:36.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:36.021+0000 7f325d400640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f32580d0e70 0x7f3258072b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:36.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:36.021+0000 7f325d400640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f3258068650 0x7f3258073080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:36.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:36.021+0000 7f325d400640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f32580cfa10 con 0x7f3258068650 2024-03-31T15:14:36.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:36.021+0000 7f325d400640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f3258073a20 con 0x7f3258161a70 2024-03-31T15:14:36.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:36.021+0000 7f325d400640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f3258099850 con 0x7f32580d0e70 2024-03-31T15:14:36.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:36.021+0000 7f325ca00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f3258068650 0x7f3258073080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:36.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:36.021+0000 7f325ca00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f3258068650 0x7f3258073080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50816/0 (socket says 172.21.15.16:50816) 2024-03-31T15:14:36.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:36.021+0000 7f325ca00640 1 -- 172.21.15.16:0/609932785 learned_addr learned my addr 172.21.15.16:0/609932785 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:14:36.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:36.021+0000 7f3257e00640 1 --2- 172.21.15.16:0/609932785 >> v2:172.21.15.89:3300/0 conn(0x7f3258161a70 0x7f32580c6990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:36.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:36.021+0000 7f324fe00640 1 --2- 172.21.15.16:0/609932785 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f32580d0e70 0x7f3258072b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:36.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:36.021+0000 7f325ca00640 1 -- 172.21.15.16:0/609932785 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f32580d0e70 msgr2=0x7f3258072b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:14:36.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:36.021+0000 7f325ca00640 1 --2- 172.21.15.16:0/609932785 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f32580d0e70 0x7f3258072b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:36.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:36.021+0000 7f325ca00640 1 -- 172.21.15.16:0/609932785 >> v2:172.21.15.89:3300/0 conn(0x7f3258161a70 msgr2=0x7f32580c6990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:14:36.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:36.021+0000 7f325ca00640 1 --2- 172.21.15.16:0/609932785 >> v2:172.21.15.89:3300/0 conn(0x7f3258161a70 0x7f32580c6990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:36.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:36.021+0000 7f325ca00640 1 -- 172.21.15.16:0/609932785 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32580cf6e0 con 0x7f3258068650 2024-03-31T15:14:36.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:36.029+0000 7f325ca00640 1 --2- 172.21.15.16:0/609932785 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3258068650 0x7f3258073080 secure :-1 s=READY pgs=1718 cs=0 l=1 rev1=1 crypto rx=0x7f32580cdc40 tx=0x7f325001f1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=49b9ebe5f37634b5 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:36.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:36.029+0000 7f3257400640 1 -- 172.21.15.16:0/609932785 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32500b4070 con 0x7f3258068650 2024-03-31T15:14:36.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:36.029+0000 7f3257400640 1 -- 172.21.15.16:0/609932785 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f32500af400 con 0x7f3258068650 2024-03-31T15:14:36.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:36.029+0000 7f3257400640 1 -- 172.21.15.16:0/609932785 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32500af7d0 con 0x7f3258068650 2024-03-31T15:14:36.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:36.029+0000 7f325d400640 1 -- 172.21.15.16:0/609932785 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3258068650 msgr2=0x7f3258073080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:36.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:36.029+0000 7f325d400640 1 --2- 172.21.15.16:0/609932785 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3258068650 0x7f3258073080 secure :-1 s=READY pgs=1718 cs=0 l=1 rev1=1 crypto rx=0x7f32580cdc40 tx=0x7f325001f1e0 comp rx=0 tx=0).stop 2024-03-31T15:14:36.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:36.029+0000 7f325d400640 1 -- 172.21.15.16:0/609932785 shutdown_connections 2024-03-31T15:14:36.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:36.029+0000 7f325d400640 1 --2- 172.21.15.16:0/609932785 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3258068650 0x7f3258073080 unknown :-1 s=CLOSED pgs=1718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:36.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:36.029+0000 7f325d400640 1 --2- 172.21.15.16:0/609932785 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f32580d0e70 0x7f3258072b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:36.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:36.029+0000 7f325d400640 1 --2- 172.21.15.16:0/609932785 >> v2:172.21.15.89:3300/0 conn(0x7f3258161a70 0x7f32580c6990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:36.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:36.029+0000 7f325d400640 1 -- 172.21.15.16:0/609932785 >> 172.21.15.16:0/609932785 conn(0x7f32580d0140 msgr2=0x7f32580d0550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:14:36.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:36.029+0000 7f325d400640 1 -- 172.21.15.16:0/609932785 shutdown_connections 2024-03-31T15:14:36.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:36.029+0000 7f325d400640 1 -- 172.21.15.16:0/609932785 wait complete. 2024-03-31T15:14:36.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:36.029+0000 7f325d400640 1 Processor -- start 2024-03-31T15:14:36.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:36.029+0000 7f325d400640 1 -- start start 2024-03-31T15:14:36.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:36.033+0000 7f325d400640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3258068650 0x7f325814b580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:36.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:36.033+0000 7f325d400640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f32580d0e70 0x7f32580b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:36.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:36.033+0000 7f325d400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3258161a70 0x7f32580b5440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:36.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:36.033+0000 7f325d400640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f3258099d20 con 0x7f3258161a70 2024-03-31T15:14:36.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:36.033+0000 7f325d400640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f32580c5430 con 0x7f3258068650 2024-03-31T15:14:36.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:36.033+0000 7f325d400640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f325814bac0 con 0x7f32580d0e70 2024-03-31T15:14:36.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:36.033+0000 7f325ca00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3258161a70 0x7f32580b5440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:36.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:36.033+0000 7f324fe00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f32580d0e70 0x7f32580b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:36.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:36.033+0000 7f3257e00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3258068650 0x7f325814b580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:36.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:36.033+0000 7f324fe00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f32580d0e70 0x7f32580b4f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3301/0 says I am v2:172.21.15.16:60274/0 (socket says 172.21.15.16:60274) 2024-03-31T15:14:36.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:36.033+0000 7f324fe00640 1 -- 172.21.15.16:0/3708255506 learned_addr learned my addr 172.21.15.16:0/3708255506 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:14:36.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:36.033+0000 7f325ca00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3258161a70 0x7f32580b5440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50818/0 (socket says 172.21.15.16:50818) 2024-03-31T15:14:36.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:36.033+0000 7f325ca00640 1 -- 172.21.15.16:0/3708255506 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f32580d0e70 msgr2=0x7f32580b4f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:36.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:36.033+0000 7f325ca00640 1 --2- 172.21.15.16:0/3708255506 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f32580d0e70 0x7f32580b4f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:36.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:36.033+0000 7f325ca00640 1 -- 172.21.15.16:0/3708255506 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3258068650 msgr2=0x7f325814b580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:36.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:36.033+0000 7f325ca00640 1 --2- 172.21.15.16:0/3708255506 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3258068650 0x7f325814b580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:36.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:36.033+0000 7f325ca00640 1 -- 172.21.15.16:0/3708255506 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32580c72b0 con 0x7f3258161a70 2024-03-31T15:14:36.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:36.033+0000 7f325ca00640 1 --2- 172.21.15.16:0/3708255506 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3258161a70 0x7f32580b5440 secure :-1 s=READY pgs=1719 cs=0 l=1 rev1=1 crypto rx=0x7f32500ae000 tx=0x7f325001bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:36.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:36.033+0000 7f3254c00640 1 -- 172.21.15.16:0/3708255506 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32500b4040 con 0x7f3258161a70 2024-03-31T15:14:36.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:36.033+0000 7f3254c00640 1 -- 172.21.15.16:0/3708255506 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f32500090d0 con 0x7f3258161a70 2024-03-31T15:14:36.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:36.033+0000 7f3254c00640 1 -- 172.21.15.16:0/3708255506 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3250009510 con 0x7f3258161a70 2024-03-31T15:14:36.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:36.033+0000 7f325d400640 1 -- 172.21.15.16:0/3708255506 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f32580c7590 con 0x7f3258161a70 2024-03-31T15:14:36.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:36.033+0000 7f325d400640 1 -- 172.21.15.16:0/3708255506 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f32580c7b50 con 0x7f3258161a70 2024-03-31T15:14:36.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:36.033+0000 7f325d400640 1 -- 172.21.15.16:0/3708255506 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f325804f8c0 con 0x7f3258161a70 2024-03-31T15:14:36.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:36.033+0000 7f3254c00640 1 -- 172.21.15.16:0/3708255506 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f32500c3020 con 0x7f3258161a70 2024-03-31T15:14:36.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:36.037+0000 7f3254c00640 1 --2- 172.21.15.16:0/3708255506 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f3234068380 0x7f323406a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:36.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:36.037+0000 7f3254c00640 1 -- 172.21.15.16:0/3708255506 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 5767+0+0 (secure 0 0 0) 0x7f3250124200 con 0x7f3258161a70 2024-03-31T15:14:36.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:36.037+0000 7f3257e00640 1 --2- 172.21.15.16:0/3708255506 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f3234068380 0x7f323406a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:36.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:36.037+0000 7f3257e00640 1 --2- 172.21.15.16:0/3708255506 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f3234068380 0x7f323406a840 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f3248009ac0 tx=0x7f3248009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:36.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:36.041+0000 7f3254c00640 1 -- 172.21.15.16:0/3708255506 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f32500ca020 con 0x7f3258161a70 2024-03-31T15:14:36.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:36.209+0000 7f325d400640 1 -- 172.21.15.16:0/3708255506 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f325806f2f0 con 0x7f3258161a70 2024-03-31T15:14:36.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:36.209+0000 7f3254c00640 1 -- 172.21.15.16:0/3708255506 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v111) ==== 104+0+11 (secure 0 0 0) 0x7f3250018d40 con 0x7f3258161a70 2024-03-31T15:14:36.205 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2024-03-31T15:14:36.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:36.213+0000 7f325d400640 1 -- 172.21.15.16:0/3708255506 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f3234068380 msgr2=0x7f323406a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:36.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:36.213+0000 7f325d400640 1 --2- 172.21.15.16:0/3708255506 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f3234068380 0x7f323406a840 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f3248009ac0 tx=0x7f3248009190 comp rx=0 tx=0).stop 2024-03-31T15:14:36.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:36.213+0000 7f325d400640 1 -- 172.21.15.16:0/3708255506 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3258161a70 msgr2=0x7f32580b5440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:36.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:36.213+0000 7f325d400640 1 --2- 172.21.15.16:0/3708255506 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3258161a70 0x7f32580b5440 secure :-1 s=READY pgs=1719 cs=0 l=1 rev1=1 crypto rx=0x7f32500ae000 tx=0x7f325001bf80 comp rx=0 tx=0).stop 2024-03-31T15:14:36.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:36.213+0000 7f325d400640 1 -- 172.21.15.16:0/3708255506 shutdown_connections 2024-03-31T15:14:36.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:36.213+0000 7f325d400640 1 --2- 172.21.15.16:0/3708255506 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f3234068380 0x7f323406a840 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:36.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:36.213+0000 7f325d400640 1 --2- 172.21.15.16:0/3708255506 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3258161a70 0x7f32580b5440 unknown :-1 s=CLOSED pgs=1719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:36.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:36.213+0000 7f325d400640 1 --2- 172.21.15.16:0/3708255506 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f32580d0e70 0x7f32580b4f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:36.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:36.213+0000 7f325d400640 1 --2- 172.21.15.16:0/3708255506 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3258068650 0x7f325814b580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:36.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:36.213+0000 7f325d400640 1 -- 172.21.15.16:0/3708255506 >> 172.21.15.16:0/3708255506 conn(0x7f32580d0140 msgr2=0x7f32581602b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:14:36.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:36.213+0000 7f325d400640 1 -- 172.21.15.16:0/3708255506 shutdown_connections 2024-03-31T15:14:36.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:36.213+0000 7f325d400640 1 -- 172.21.15.16:0/3708255506 wait complete. 2024-03-31T15:14:36.226 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-31T15:14:36.369 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:36.373+0000 7f992f200640 1 Processor -- start 2024-03-31T15:14:36.369 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:36.373+0000 7f992f200640 1 -- start start 2024-03-31T15:14:36.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:36.377+0000 7f992f200640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f9928145d50 0x7f9928160860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:36.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:36.377+0000 7f992f200640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9928070a10 con 0x7f992815c890 2024-03-31T15:14:36.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:36.377+0000 7f992f200640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f99280cd9b0 con 0x7f9928070350 2024-03-31T15:14:36.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:36.377+0000 7f992f200640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f99280ce5c0 con 0x7f9928145d50 2024-03-31T15:14:36.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:36.377+0000 7f991fe00640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f992815c890 0x7f992806fc40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:50236/0 (socket says 172.21.15.16:50236) 2024-03-31T15:14:36.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:36.377+0000 7f991fe00640 1 -- 172.21.15.16:0/386452454 learned_addr learned my addr 172.21.15.16:0/386452454 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:14:36.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:36.377+0000 7f9927e00640 1 --2- 172.21.15.16:0/386452454 >> v2:172.21.15.89:3300/0 conn(0x7f9928145d50 0x7f9928160860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:36.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:36.377+0000 7f9927400640 1 -- 172.21.15.16:0/386452454 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3122030518 0 0) 0x7f9928070a10 con 0x7f992815c890 2024-03-31T15:14:36.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:36.377+0000 7f9927400640 1 -- 172.21.15.16:0/386452454 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f990c003530 con 0x7f992815c890 2024-03-31T15:14:36.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:36.377+0000 7f9927400640 1 -- 172.21.15.16:0/386452454 <== mon.1 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 388475895 0 0) 0x7f99280cd9b0 con 0x7f9928070350 2024-03-31T15:14:36.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:36.377+0000 7f9927400640 1 -- 172.21.15.16:0/386452454 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9928070a10 con 0x7f9928070350 2024-03-31T15:14:36.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:36.377+0000 7f9927e00640 1 -- 172.21.15.16:0/386452454 >> v1:172.21.15.89:6789/0 conn(0x7f9928070350 legacy=0x7f99280cea50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:36.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:36.377+0000 7f9927400640 1 -- 172.21.15.16:0/386452454 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2916540134 0 0) 0x7f990c003530 con 0x7f992815c890 2024-03-31T15:14:36.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:36.377+0000 7f9927e00640 1 -- 172.21.15.16:0/386452454 >> v1:172.21.15.16:6789/0 conn(0x7f992815c890 legacy=0x7f992806fc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:36.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:36.377+0000 7f9927e00640 1 -- 172.21.15.16:0/386452454 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99280cdc90 con 0x7f9928145d50 2024-03-31T15:14:36.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:36.381+0000 7f9927e00640 1 --2- 172.21.15.16:0/386452454 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9928145d50 0x7f9928160860 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f99180097c0 tx=0x7f991801c470 comp rx=0 tx=0).ready entity=mon.1 client_cookie=13cc8960e36caf00 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:36.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:36.381+0000 7f9927400640 1 -- 172.21.15.16:0/386452454 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99180a7070 con 0x7f9928145d50 2024-03-31T15:14:36.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:36.381+0000 7f9927400640 1 -- 172.21.15.16:0/386452454 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f99180b0070 con 0x7f9928145d50 2024-03-31T15:14:36.375 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:36.381+0000 7f9927400640 1 -- 172.21.15.16:0/386452454 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99180ab890 con 0x7f9928145d50 2024-03-31T15:14:36.375 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:36.381+0000 7f992f200640 1 -- 172.21.15.16:0/386452454 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9928145d50 msgr2=0x7f9928160860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:36.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:36.381+0000 7f992f200640 1 --2- 172.21.15.16:0/386452454 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9928145d50 0x7f9928160860 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f99180097c0 tx=0x7f991801c470 comp rx=0 tx=0).stop 2024-03-31T15:14:36.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:36.381+0000 7f992f200640 1 -- 172.21.15.16:0/386452454 shutdown_connections 2024-03-31T15:14:36.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:36.381+0000 7f992f200640 1 --2- 172.21.15.16:0/386452454 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9928145d50 0x7f9928160860 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:36.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:36.381+0000 7f992f200640 1 -- 172.21.15.16:0/386452454 >> 172.21.15.16:0/386452454 conn(0x7f992806e670 msgr2=0x7f992806ea50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:14:36.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:36.381+0000 7f992f200640 1 -- 172.21.15.16:0/386452454 shutdown_connections 2024-03-31T15:14:36.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:36.381+0000 7f992f200640 1 -- 172.21.15.16:0/386452454 wait complete. 2024-03-31T15:14:36.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:36.381+0000 7f992f200640 1 Processor -- start 2024-03-31T15:14:36.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:36.385+0000 7f992f200640 1 -- start start 2024-03-31T15:14:36.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:36.385+0000 7f992f200640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9928070350 0x7f992813d9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:36.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:36.385+0000 7f992f200640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f9928145d50 0x7f992813cdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:36.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:36.385+0000 7f992f200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f992815c890 0x7f9928143350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:36.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:36.385+0000 7f992f200640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f992816a530 con 0x7f992815c890 2024-03-31T15:14:36.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:36.385+0000 7f992f200640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f992816a3b0 con 0x7f9928070350 2024-03-31T15:14:36.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:36.385+0000 7f992f200640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f992816a6b0 con 0x7f9928145d50 2024-03-31T15:14:36.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:36.385+0000 7f992d200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f992815c890 0x7f9928143350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:36.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:36.385+0000 7f992d200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f992815c890 0x7f9928143350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50822/0 (socket says 172.21.15.16:50822) 2024-03-31T15:14:36.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:36.385+0000 7f992d200640 1 -- 172.21.15.16:0/1039595384 learned_addr learned my addr 172.21.15.16:0/1039595384 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:14:36.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:36.385+0000 7f991fe00640 1 --2- 172.21.15.16:0/1039595384 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f9928145d50 0x7f992813cdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:36.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:36.385+0000 7f9927e00640 1 --2- 172.21.15.16:0/1039595384 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9928070350 0x7f992813d9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:36.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:36.385+0000 7f992d200640 1 -- 172.21.15.16:0/1039595384 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f9928145d50 msgr2=0x7f992813cdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:36.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:36.385+0000 7f992d200640 1 --2- 172.21.15.16:0/1039595384 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f9928145d50 0x7f992813cdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:36.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:36.385+0000 7f992d200640 1 -- 172.21.15.16:0/1039595384 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9928070350 msgr2=0x7f992813d9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:36.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:36.385+0000 7f992d200640 1 --2- 172.21.15.16:0/1039595384 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9928070350 0x7f992813d9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:36.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:36.385+0000 7f992d200640 1 -- 172.21.15.16:0/1039595384 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99280b4f70 con 0x7f992815c890 2024-03-31T15:14:36.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:36.385+0000 7f992d200640 1 --2- 172.21.15.16:0/1039595384 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f992815c890 0x7f9928143350 secure :-1 s=READY pgs=1721 cs=0 l=1 rev1=1 crypto rx=0x7f992000c7a0 tx=0x7f992000cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:36.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:36.385+0000 7f9924c00640 1 -- 172.21.15.16:0/1039595384 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9920003b80 con 0x7f992815c890 2024-03-31T15:14:36.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:36.385+0000 7f9924c00640 1 -- 172.21.15.16:0/1039595384 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9920003d20 con 0x7f992815c890 2024-03-31T15:14:36.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:36.385+0000 7f992f200640 1 -- 172.21.15.16:0/1039595384 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f99280b5250 con 0x7f992815c890 2024-03-31T15:14:36.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:36.385+0000 7f9924c00640 1 -- 172.21.15.16:0/1039595384 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f992000aa90 con 0x7f992815c890 2024-03-31T15:14:36.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:36.385+0000 7f992f200640 1 -- 172.21.15.16:0/1039595384 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f99280b5810 con 0x7f992815c890 2024-03-31T15:14:36.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:36.385+0000 7f992f200640 1 -- 172.21.15.16:0/1039595384 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9928069950 con 0x7f992815c890 2024-03-31T15:14:36.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:36.389+0000 7f9924c00640 1 -- 172.21.15.16:0/1039595384 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f992000ac30 con 0x7f992815c890 2024-03-31T15:14:36.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:36.389+0000 7f9924c00640 1 --2- 172.21.15.16:0/1039595384 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f99040683b0 0x7f990406a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:36.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:36.389+0000 7f9924c00640 1 -- 172.21.15.16:0/1039595384 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 5767+0+0 (secure 0 0 0) 0x7f9920086840 con 0x7f992815c890 2024-03-31T15:14:36.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:36.389+0000 7f9927e00640 1 --2- 172.21.15.16:0/1039595384 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f99040683b0 0x7f990406a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:36.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:36.389+0000 7f9927e00640 1 --2- 172.21.15.16:0/1039595384 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f99040683b0 0x7f990406a870 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f99180af000 tx=0x7f99180a7000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:36.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:36.393+0000 7f9924c00640 1 -- 172.21.15.16:0/1039595384 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f99200566d0 con 0x7f992815c890 2024-03-31T15:14:36.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:36.693+0000 7f992f200640 1 -- 172.21.15.16:0/1039595384 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f99280ceb50 con 0x7f992815c890 2024-03-31T15:14:36.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:36.693+0000 7f9924c00640 1 -- 172.21.15.16:0/1039595384 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v111) ==== 100+0+11 (secure 0 0 0) 0x7f992005a9d0 con 0x7f992815c890 2024-03-31T15:14:36.687 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 64 2024-03-31T15:14:36.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:36.693+0000 7f992f200640 1 -- 172.21.15.16:0/1039595384 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f99040683b0 msgr2=0x7f990406a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:36.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:36.693+0000 7f992f200640 1 --2- 172.21.15.16:0/1039595384 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f99040683b0 0x7f990406a870 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f99180af000 tx=0x7f99180a7000 comp rx=0 tx=0).stop 2024-03-31T15:14:36.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:36.693+0000 7f992f200640 1 -- 172.21.15.16:0/1039595384 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f992815c890 msgr2=0x7f9928143350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:36.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:36.697+0000 7f992f200640 1 --2- 172.21.15.16:0/1039595384 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f992815c890 0x7f9928143350 secure :-1 s=READY pgs=1721 cs=0 l=1 rev1=1 crypto rx=0x7f992000c7a0 tx=0x7f992000cc60 comp rx=0 tx=0).stop 2024-03-31T15:14:36.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:36.697+0000 7f992f200640 1 -- 172.21.15.16:0/1039595384 shutdown_connections 2024-03-31T15:14:36.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:36.697+0000 7f992f200640 1 --2- 172.21.15.16:0/1039595384 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f99040683b0 0x7f990406a870 unknown :-1 s=CLOSED pgs=729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:36.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:36.697+0000 7f992f200640 1 --2- 172.21.15.16:0/1039595384 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f992815c890 0x7f9928143350 unknown :-1 s=CLOSED pgs=1721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:36.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:36.697+0000 7f992f200640 1 --2- 172.21.15.16:0/1039595384 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f9928145d50 0x7f992813cdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:36.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:36.697+0000 7f992f200640 1 --2- 172.21.15.16:0/1039595384 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9928070350 0x7f992813d9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:36.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:36.697+0000 7f992f200640 1 -- 172.21.15.16:0/1039595384 >> 172.21.15.16:0/1039595384 conn(0x7f992806e670 msgr2=0x7f992806beb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:14:36.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:36.697+0000 7f992f200640 1 -- 172.21.15.16:0/1039595384 shutdown_connections 2024-03-31T15:14:36.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:36.697+0000 7f992f200640 1 -- 172.21.15.16:0/1039595384 wait complete. 2024-03-31T15:14:36.717 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-31T15:14:36.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:36.865+0000 7fcf63e00640 1 Processor -- start 2024-03-31T15:14:36.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:36.865+0000 7fcf63e00640 1 -- start start 2024-03-31T15:14:36.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:36.865+0000 7fcf63e00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fcf64070d50 0x7fcf640cd2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:36.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:36.865+0000 7fcf63e00640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcf640d0800 con 0x7fcf6415e3b0 2024-03-31T15:14:36.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:36.865+0000 7fcf63e00640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcf640d0ae0 con 0x7fcf6409b170 2024-03-31T15:14:36.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:36.865+0000 7fcf63e00640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7fcf6415e790 con 0x7fcf64070d50 2024-03-31T15:14:36.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:36.865+0000 7fcf62a00640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fcf6415e3b0 0x7fcf6409aa60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:50250/0 (socket says 172.21.15.16:50250) 2024-03-31T15:14:36.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:36.865+0000 7fcf62a00640 1 -- 172.21.15.16:0/4177709957 learned_addr learned my addr 172.21.15.16:0/4177709957 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:14:36.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:36.865+0000 7fcf63400640 1 --2- 172.21.15.16:0/4177709957 >> v2:172.21.15.89:3300/0 conn(0x7fcf64070d50 0x7fcf640cd2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:36.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:36.865+0000 7fcf62000640 1 -- 172.21.15.16:0/4177709957 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1789593116 0 0) 0x7fcf640d0800 con 0x7fcf6415e3b0 2024-03-31T15:14:36.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:36.865+0000 7fcf62000640 1 -- 172.21.15.16:0/4177709957 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcf48003530 con 0x7fcf6415e3b0 2024-03-31T15:14:36.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:36.865+0000 7fcf62000640 1 -- 172.21.15.16:0/4177709957 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 690852233 0 0) 0x7fcf48003530 con 0x7fcf6415e3b0 2024-03-31T15:14:36.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:36.865+0000 7fcf62000640 1 -- 172.21.15.16:0/4177709957 >> v2:172.21.15.89:3300/0 conn(0x7fcf64070d50 msgr2=0x7fcf640cd2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:14:36.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:36.865+0000 7fcf62000640 1 --2- 172.21.15.16:0/4177709957 >> v2:172.21.15.89:3300/0 conn(0x7fcf64070d50 0x7fcf640cd2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:36.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:36.865+0000 7fcf62000640 1 -- 172.21.15.16:0/4177709957 >> v1:172.21.15.89:6789/0 conn(0x7fcf6409b170 legacy=0x7fcf640ccb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:36.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:36.865+0000 7fcf62000640 1 -- 172.21.15.16:0/4177709957 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcf640d0dc0 con 0x7fcf6415e3b0 2024-03-31T15:14:36.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:36.865+0000 7fcf62000640 1 -- 172.21.15.16:0/4177709957 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 214807623 0 0) 0x7fcf54001680 con 0x7fcf6415e3b0 2024-03-31T15:14:36.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:36.865+0000 7fcf62000640 1 -- 172.21.15.16:0/4177709957 >> v1:172.21.15.16:6789/0 conn(0x7fcf6415e3b0 legacy=0x7fcf6409aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:36.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:36.865+0000 7fcf62000640 1 --2- 172.21.15.16:0/4177709957 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcf480039c0 0x7fcf48003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:36.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:36.865+0000 7fcf62000640 1 --2- 172.21.15.16:0/4177709957 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcf48004810 0x7fcf48004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:36.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:36.865+0000 7fcf62000640 1 --2- 172.21.15.16:0/4177709957 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fcf480051d0 0x7fcf4800b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:36.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:36.865+0000 7fcf62000640 1 -- 172.21.15.16:0/4177709957 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fcf6415e790 con 0x7fcf48004810 2024-03-31T15:14:36.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:36.865+0000 7fcf62000640 1 -- 172.21.15.16:0/4177709957 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fcf4800c070 con 0x7fcf480039c0 2024-03-31T15:14:36.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:36.865+0000 7fcf62000640 1 -- 172.21.15.16:0/4177709957 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fcf4800c270 con 0x7fcf480051d0 2024-03-31T15:14:36.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:36.869+0000 7fcf5be00640 1 --2- 172.21.15.16:0/4177709957 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcf48004810 0x7fcf48004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:36.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:36.869+0000 7fcf62a00640 1 --2- 172.21.15.16:0/4177709957 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcf480039c0 0x7fcf48003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:36.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:36.869+0000 7fcf63400640 1 --2- 172.21.15.16:0/4177709957 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fcf480051d0 0x7fcf4800b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:36.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:36.869+0000 7fcf62a00640 1 -- 172.21.15.16:0/4177709957 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fcf480051d0 msgr2=0x7fcf4800b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:14:36.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:36.869+0000 7fcf62a00640 1 --2- 172.21.15.16:0/4177709957 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fcf480051d0 0x7fcf4800b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:36.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:36.869+0000 7fcf62a00640 1 -- 172.21.15.16:0/4177709957 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcf48004810 msgr2=0x7fcf48004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:14:36.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:36.869+0000 7fcf62a00640 1 --2- 172.21.15.16:0/4177709957 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcf48004810 0x7fcf48004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:36.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:36.869+0000 7fcf62a00640 1 -- 172.21.15.16:0/4177709957 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcf48003530 con 0x7fcf480039c0 2024-03-31T15:14:36.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:36.869+0000 7fcf63400640 1 -- 172.21.15.16:0/4177709957 reap_dead start 2024-03-31T15:14:36.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:36.869+0000 7fcf5be00640 1 --2- 172.21.15.16:0/4177709957 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcf48004810 0x7fcf48004c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:14:36.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:36.869+0000 7fcf62a00640 1 --2- 172.21.15.16:0/4177709957 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcf480039c0 0x7fcf48003de0 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7fcf5400c210 tx=0x7fcf5401f330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4fac2c3c2268d5aa server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:36.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:36.869+0000 7fcf62000640 1 -- 172.21.15.16:0/4177709957 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf54007d20 con 0x7fcf480039c0 2024-03-31T15:14:36.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:36.873+0000 7fcf62000640 1 -- 172.21.15.16:0/4177709957 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcf54005b00 con 0x7fcf480039c0 2024-03-31T15:14:36.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:36.873+0000 7fcf62000640 1 -- 172.21.15.16:0/4177709957 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf54007070 con 0x7fcf480039c0 2024-03-31T15:14:36.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:36.873+0000 7fcf63e00640 1 -- 172.21.15.16:0/4177709957 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcf480039c0 msgr2=0x7fcf48003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:36.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:36.873+0000 7fcf63e00640 1 --2- 172.21.15.16:0/4177709957 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcf480039c0 0x7fcf48003de0 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7fcf5400c210 tx=0x7fcf5401f330 comp rx=0 tx=0).stop 2024-03-31T15:14:36.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:36.873+0000 7fcf63e00640 1 -- 172.21.15.16:0/4177709957 shutdown_connections 2024-03-31T15:14:36.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:36.873+0000 7fcf63e00640 1 --2- 172.21.15.16:0/4177709957 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcf480039c0 0x7fcf48003de0 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:36.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:36.873+0000 7fcf63e00640 1 -- 172.21.15.16:0/4177709957 >> 172.21.15.16:0/4177709957 conn(0x7fcf640b39c0 msgr2=0x7fcf640b3dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:14:36.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:36.873+0000 7fcf63e00640 1 -- 172.21.15.16:0/4177709957 shutdown_connections 2024-03-31T15:14:36.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:36.873+0000 7fcf63e00640 1 -- 172.21.15.16:0/4177709957 wait complete. 2024-03-31T15:14:36.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:36.873+0000 7fcf63e00640 1 Processor -- start 2024-03-31T15:14:36.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:36.873+0000 7fcf63e00640 1 -- start start 2024-03-31T15:14:36.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:36.873+0000 7fcf63e00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcf480039c0 0x7fcf6415e940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:36.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:36.873+0000 7fcf63e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcf6414d630 0x7fcf640b74e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:36.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:36.873+0000 7fcf63e00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fcf640b16f0 0x7fcf640b1ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:36.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:36.873+0000 7fcf63e00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fcf640d04b0 con 0x7fcf6414d630 2024-03-31T15:14:36.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:36.873+0000 7fcf63e00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fcf6406f5d0 con 0x7fcf480039c0 2024-03-31T15:14:36.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:36.873+0000 7fcf63e00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fcf640b1360 con 0x7fcf640b16f0 2024-03-31T15:14:36.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:36.873+0000 7fcf5be00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcf6414d630 0x7fcf640b74e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:36.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:36.873+0000 7fcf5be00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcf6414d630 0x7fcf640b74e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50830/0 (socket says 172.21.15.16:50830) 2024-03-31T15:14:36.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:36.873+0000 7fcf5be00640 1 -- 172.21.15.16:0/139611262 learned_addr learned my addr 172.21.15.16:0/139611262 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:14:36.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:36.873+0000 7fcf62a00640 1 --2- 172.21.15.16:0/139611262 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcf480039c0 0x7fcf6415e940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:36.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:36.873+0000 7fcf63400640 1 --2- 172.21.15.16:0/139611262 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fcf640b16f0 0x7fcf640b1ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:36.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:36.873+0000 7fcf5be00640 1 -- 172.21.15.16:0/139611262 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fcf640b16f0 msgr2=0x7fcf640b1ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:36.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:36.873+0000 7fcf5be00640 1 --2- 172.21.15.16:0/139611262 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fcf640b16f0 0x7fcf640b1ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:36.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:36.873+0000 7fcf5be00640 1 -- 172.21.15.16:0/139611262 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcf480039c0 msgr2=0x7fcf6415e940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:36.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:36.873+0000 7fcf5be00640 1 --2- 172.21.15.16:0/139611262 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcf480039c0 0x7fcf6415e940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:36.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:36.873+0000 7fcf5be00640 1 -- 172.21.15.16:0/139611262 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcf6415f000 con 0x7fcf6414d630 2024-03-31T15:14:36.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:36.873+0000 7fcf62a00640 1 --2- 172.21.15.16:0/139611262 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcf480039c0 0x7fcf6415e940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:14:36.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:36.873+0000 7fcf63400640 1 --2- 172.21.15.16:0/139611262 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fcf640b16f0 0x7fcf640b1ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:14:36.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:36.877+0000 7fcf5be00640 1 --2- 172.21.15.16:0/139611262 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcf6414d630 0x7fcf640b74e0 secure :-1 s=READY pgs=1723 cs=0 l=1 rev1=1 crypto rx=0x7fcf4c00f770 tx=0x7fcf4c00fc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:36.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:36.877+0000 7fcf5aa00640 1 -- 172.21.15.16:0/139611262 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf4c01a070 con 0x7fcf6414d630 2024-03-31T15:14:36.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:36.877+0000 7fcf5aa00640 1 -- 172.21.15.16:0/139611262 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcf4c00fe80 con 0x7fcf6414d630 2024-03-31T15:14:36.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:36.877+0000 7fcf63e00640 1 -- 172.21.15.16:0/139611262 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcf6415f2e0 con 0x7fcf6414d630 2024-03-31T15:14:36.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:36.877+0000 7fcf5aa00640 1 -- 172.21.15.16:0/139611262 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf4c0013a0 con 0x7fcf6414d630 2024-03-31T15:14:36.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:36.877+0000 7fcf63e00640 1 -- 172.21.15.16:0/139611262 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcf640ca430 con 0x7fcf6414d630 2024-03-31T15:14:36.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:36.877+0000 7fcf63e00640 1 -- 172.21.15.16:0/139611262 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcf6404f8c0 con 0x7fcf6414d630 2024-03-31T15:14:36.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:36.877+0000 7fcf5aa00640 1 -- 172.21.15.16:0/139611262 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fcf4c01d070 con 0x7fcf6414d630 2024-03-31T15:14:36.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:36.877+0000 7fcf5aa00640 1 --2- 172.21.15.16:0/139611262 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fcf400683b0 0x7fcf4006a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:36.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:36.877+0000 7fcf5aa00640 1 -- 172.21.15.16:0/139611262 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 5767+0+0 (secure 0 0 0) 0x7fcf4c0861e0 con 0x7fcf6414d630 2024-03-31T15:14:36.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:36.877+0000 7fcf62a00640 1 --2- 172.21.15.16:0/139611262 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fcf400683b0 0x7fcf4006a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:36.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:36.881+0000 7fcf62a00640 1 --2- 172.21.15.16:0/139611262 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fcf400683b0 0x7fcf4006a870 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7fcf540ae000 tx=0x7fcf54007040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:36.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:36.881+0000 7fcf5aa00640 1 -- 172.21.15.16:0/139611262 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fcf4c0566c0 con 0x7fcf6414d630 2024-03-31T15:14:37.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:37.177+0000 7fcf63e00640 1 -- 172.21.15.16:0/139611262 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fcf64068ae0 con 0x7fcf6414d630 2024-03-31T15:14:37.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:37.177+0000 7fcf5aa00640 1 -- 172.21.15.16:0/139611262 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v111) ==== 80+0+0 (secure 0 0 0) 0x7fcf4c055fc0 con 0x7fcf6414d630 2024-03-31T15:14:37.172 INFO:teuthology.orchestra.run.smithi016.stderr:listed 0 entries 2024-03-31T15:14:37.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:37.181+0000 7fcf63e00640 1 -- 172.21.15.16:0/139611262 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fcf400683b0 msgr2=0x7fcf4006a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:37.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:37.181+0000 7fcf63e00640 1 --2- 172.21.15.16:0/139611262 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fcf400683b0 0x7fcf4006a870 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7fcf540ae000 tx=0x7fcf54007040 comp rx=0 tx=0).stop 2024-03-31T15:14:37.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:37.181+0000 7fcf63e00640 1 -- 172.21.15.16:0/139611262 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcf6414d630 msgr2=0x7fcf640b74e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:37.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:37.181+0000 7fcf63e00640 1 --2- 172.21.15.16:0/139611262 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcf6414d630 0x7fcf640b74e0 secure :-1 s=READY pgs=1723 cs=0 l=1 rev1=1 crypto rx=0x7fcf4c00f770 tx=0x7fcf4c00fc30 comp rx=0 tx=0).stop 2024-03-31T15:14:37.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:37.181+0000 7fcf63e00640 1 -- 172.21.15.16:0/139611262 shutdown_connections 2024-03-31T15:14:37.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:37.181+0000 7fcf63e00640 1 --2- 172.21.15.16:0/139611262 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fcf400683b0 0x7fcf4006a870 unknown :-1 s=CLOSED pgs=730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:37.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:37.181+0000 7fcf63e00640 1 --2- 172.21.15.16:0/139611262 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fcf640b16f0 0x7fcf640b1ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:37.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:37.181+0000 7fcf63e00640 1 --2- 172.21.15.16:0/139611262 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcf6414d630 0x7fcf640b74e0 unknown :-1 s=CLOSED pgs=1723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:37.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:37.181+0000 7fcf63e00640 1 --2- 172.21.15.16:0/139611262 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcf480039c0 0x7fcf6415e940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:37.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:37.181+0000 7fcf63e00640 1 -- 172.21.15.16:0/139611262 >> 172.21.15.16:0/139611262 conn(0x7fcf640b39c0 msgr2=0x7fcf64070400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:14:37.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:37.181+0000 7fcf63e00640 1 -- 172.21.15.16:0/139611262 shutdown_connections 2024-03-31T15:14:37.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:37.181+0000 7fcf63e00640 1 -- 172.21.15.16:0/139611262 wait complete. 2024-03-31T15:14:37.194 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-03-31T15:14:37.194 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-31T15:14:37.203 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-31T15:14:37.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:37.393+0000 7f93d6a00640 1 Processor -- start 2024-03-31T15:14:37.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:37.393+0000 7f93d6a00640 1 -- start start 2024-03-31T15:14:37.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:37.393+0000 7f93d6a00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f93d00d0640 0x7f93d00b5650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:37.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:37.393+0000 7f93d6a00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f93d00b5b90 0x7f93d00cd820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:37.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:37.393+0000 7f93d6a00640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f93d0160cf0 con 0x7f93d00cdd60 2024-03-31T15:14:37.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:37.393+0000 7f93d6a00640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f93d015a260 con 0x7f93d00d0640 2024-03-31T15:14:37.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:37.393+0000 7f93d6a00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f93d0160a10 con 0x7f93d00b5b90 2024-03-31T15:14:37.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:37.393+0000 7f93d5600640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f93d00d0640 0x7f93d00b5650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:37.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:37.393+0000 7f93d5600640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f93d00d0640 0x7f93d00b5650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40586/0 (socket says 172.21.15.16:40586) 2024-03-31T15:14:37.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:37.393+0000 7f93d5600640 1 -- 172.21.15.16:0/1731248726 learned_addr learned my addr 172.21.15.16:0/1731248726 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:14:37.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:37.393+0000 7f93d4c00640 1 --2- 172.21.15.16:0/1731248726 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f93d00b5b90 0x7f93d00cd820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:37.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:37.393+0000 7f93d5600640 1 -- 172.21.15.16:0/1731248726 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f93d00b5b90 msgr2=0x7f93d00cd820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:14:37.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:37.393+0000 7f93d5600640 1 --2- 172.21.15.16:0/1731248726 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f93d00b5b90 0x7f93d00cd820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:37.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:37.393+0000 7f93d5600640 1 -- 172.21.15.16:0/1731248726 >> v1:172.21.15.89:6789/0 conn(0x7f93d00cdd60 legacy=0x7f93d00cf4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:37.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:37.393+0000 7f93d5600640 1 -- 172.21.15.16:0/1731248726 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f93d00b4b70 con 0x7f93d00d0640 2024-03-31T15:14:37.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:37.397+0000 7f93d5600640 1 --2- 172.21.15.16:0/1731248726 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f93d00d0640 0x7f93d00b5650 secure :-1 s=READY pgs=1724 cs=0 l=1 rev1=1 crypto rx=0x7f93c4009870 tx=0x7f93c401c560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=79d60bd83ff0fb4d server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:37.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:37.397+0000 7f93cfe00640 1 -- 172.21.15.16:0/1731248726 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93c40b2070 con 0x7f93d00d0640 2024-03-31T15:14:37.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:37.397+0000 7f93cfe00640 1 -- 172.21.15.16:0/1731248726 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f93c40a9070 con 0x7f93d00d0640 2024-03-31T15:14:37.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:37.397+0000 7f93cfe00640 1 -- 172.21.15.16:0/1731248726 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93c40ad880 con 0x7f93d00d0640 2024-03-31T15:14:37.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:37.397+0000 7f93d6a00640 1 -- 172.21.15.16:0/1731248726 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f93d00d0640 msgr2=0x7f93d00b5650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:37.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:37.397+0000 7f93d6a00640 1 --2- 172.21.15.16:0/1731248726 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f93d00d0640 0x7f93d00b5650 secure :-1 s=READY pgs=1724 cs=0 l=1 rev1=1 crypto rx=0x7f93c4009870 tx=0x7f93c401c560 comp rx=0 tx=0).stop 2024-03-31T15:14:37.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:37.397+0000 7f93d6a00640 1 -- 172.21.15.16:0/1731248726 shutdown_connections 2024-03-31T15:14:37.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:37.397+0000 7f93d6a00640 1 --2- 172.21.15.16:0/1731248726 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f93d00b5b90 0x7f93d00cd820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:37.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:37.397+0000 7f93d6a00640 1 --2- 172.21.15.16:0/1731248726 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f93d00d0640 0x7f93d00b5650 unknown :-1 s=CLOSED pgs=1724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:37.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:37.397+0000 7f93d6a00640 1 -- 172.21.15.16:0/1731248726 >> 172.21.15.16:0/1731248726 conn(0x7f93d00b2ac0 msgr2=0x7f93d00b2ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:14:37.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:37.397+0000 7f93d6a00640 1 -- 172.21.15.16:0/1731248726 shutdown_connections 2024-03-31T15:14:37.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:37.397+0000 7f93d6a00640 1 -- 172.21.15.16:0/1731248726 wait complete. 2024-03-31T15:14:37.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:37.397+0000 7f93d6a00640 1 Processor -- start 2024-03-31T15:14:37.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:37.397+0000 7f93d6a00640 1 -- start start 2024-03-31T15:14:37.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:37.401+0000 7f93d6a00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f93d00b5b90 0x7f93d0159820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:37.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:37.401+0000 7f93d6a00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f93d00cdd60 0x7f93d006c4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:37.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:37.401+0000 7f93d6a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f93d00d0640 0x7f93d009bbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:37.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:37.401+0000 7f93d6a00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f93d01692e0 con 0x7f93d00d0640 2024-03-31T15:14:37.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:37.401+0000 7f93d6a00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f93d0169160 con 0x7f93d00cdd60 2024-03-31T15:14:37.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:37.401+0000 7f93d6a00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f93d0169460 con 0x7f93d00b5b90 2024-03-31T15:14:37.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:37.401+0000 7f93d6000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f93d00d0640 0x7f93d009bbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:37.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:37.401+0000 7f93d4c00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f93d00cdd60 0x7f93d006c4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:37.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:37.401+0000 7f93d5600640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f93d00b5b90 0x7f93d0159820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:37.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:37.401+0000 7f93d4c00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f93d00cdd60 0x7f93d006c4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.16:42176/0 (socket says 172.21.15.16:42176) 2024-03-31T15:14:37.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:37.401+0000 7f93d4c00640 1 -- 172.21.15.16:0/2308310780 learned_addr learned my addr 172.21.15.16:0/2308310780 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:14:37.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:37.401+0000 7f93d5600640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f93d00b5b90 0x7f93d0159820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3301/0 says I am v2:172.21.15.16:57606/0 (socket says 172.21.15.16:57606) 2024-03-31T15:14:37.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:37.401+0000 7f93d5600640 1 -- 172.21.15.16:0/2308310780 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f93d00cdd60 msgr2=0x7f93d006c4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:37.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:37.401+0000 7f93d5600640 1 --2- 172.21.15.16:0/2308310780 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f93d00cdd60 0x7f93d006c4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:37.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:37.401+0000 7f93d5600640 1 -- 172.21.15.16:0/2308310780 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f93d00d0640 msgr2=0x7f93d009bbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:37.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:37.401+0000 7f93d5600640 1 --2- 172.21.15.16:0/2308310780 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f93d00d0640 0x7f93d009bbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:37.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:37.401+0000 7f93d5600640 1 -- 172.21.15.16:0/2308310780 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f93d015e2a0 con 0x7f93d00b5b90 2024-03-31T15:14:37.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:37.401+0000 7f93d5600640 1 --2- 172.21.15.16:0/2308310780 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f93d00b5b90 0x7f93d0159820 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f93d0073700 tx=0x7f93c401ada0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:37.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:37.401+0000 7f93cd600640 1 -- 172.21.15.16:0/2308310780 <== mon.2 v2:172.21.15.89:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93c40bc020 con 0x7f93d00b5b90 2024-03-31T15:14:37.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:37.401+0000 7f93cd600640 1 -- 172.21.15.16:0/2308310780 <== mon.2 v2:172.21.15.89:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f93c40b8420 con 0x7f93d00b5b90 2024-03-31T15:14:37.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:37.401+0000 7f93d6a00640 1 -- 172.21.15.16:0/2308310780 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f93d015e580 con 0x7f93d00b5b90 2024-03-31T15:14:37.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:37.401+0000 7f93cd600640 1 -- 172.21.15.16:0/2308310780 <== mon.2 v2:172.21.15.89:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93c40b2040 con 0x7f93d00b5b90 2024-03-31T15:14:37.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:37.401+0000 7f93d6a00640 1 -- 172.21.15.16:0/2308310780 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f93d015eb40 con 0x7f93d00b5b90 2024-03-31T15:14:37.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:37.401+0000 7f93d6a00640 1 -- 172.21.15.16:0/2308310780 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f93d004f8c0 con 0x7f93d00b5b90 2024-03-31T15:14:37.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:37.405+0000 7f93cd600640 1 -- 172.21.15.16:0/2308310780 <== mon.2 v2:172.21.15.89:3301/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f93c40bf020 con 0x7f93d00b5b90 2024-03-31T15:14:37.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:37.405+0000 7f93cd600640 1 --2- 172.21.15.16:0/2308310780 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f93b0068380 0x7f93b006a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:37.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:37.405+0000 7f93cd600640 1 -- 172.21.15.16:0/2308310780 <== mon.2 v2:172.21.15.89:3301/0 5 ==== osd_map(111..111 src has 1..111) ==== 5767+0+0 (secure 0 0 0) 0x7f93c4120a10 con 0x7f93d00b5b90 2024-03-31T15:14:37.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:37.405+0000 7f93d4c00640 1 --2- 172.21.15.16:0/2308310780 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f93b0068380 0x7f93b006a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:37.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:37.405+0000 7f93d4c00640 1 --2- 172.21.15.16:0/2308310780 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f93b0068380 0x7f93b006a840 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f93d00b2450 tx=0x7f93bc009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:37.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:37.409+0000 7f93cd600640 1 -- 172.21.15.16:0/2308310780 <== mon.2 v2:172.21.15.89:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f93c40c6030 con 0x7f93d00b5b90 2024-03-31T15:14:37.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:37.573+0000 7f93d6a00640 1 -- 172.21.15.16:0/2308310780 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f93d00cf280 con 0x7f93d00b5b90 2024-03-31T15:14:37.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:37.577+0000 7f93cd600640 1 -- 172.21.15.16:0/2308310780 <== mon.2 v2:172.21.15.89:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v111) ==== 74+0+16648 (secure 0 0 0) 0x7f93c4016150 con 0x7f93d00b5b90 2024-03-31T15:14:37.570 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:14:37.571 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":111,"fsid":"286eb5da-fb47-4550-b290-d9519c13b9a5","created":"2024-03-31T15:03:11.438952+0000","modified":"2024-03-31T15:14:19.306886+0000","last_up_change":"2024-03-31T15:03:22.885535+0000","last_in_change":"2024-03-31T15:03:13.941667+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-31T15:03:22.550787+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-03-31T15:14:06.417304+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"108","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-03-31T15:14:07.622562+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"109","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"92a66450-ef19-40d8-ac3d-d760b23b0bda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6801","nonce":932712244}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6803","nonce":932712244}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6807","nonce":932712244}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6805","nonce":932712244}]},"public_addr":"172.21.15.16:6801/932712244","cluster_addr":"172.21.15.16:6803/932712244","heartbeat_back_addr":"172.21.15.16:6807/932712244","heartbeat_front_addr":"172.21.15.16:6805/932712244","state":["exists","up"]},{"osd":1,"uuid":"da9d1f2b-1968-4959-b843-faa231988e92","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6810","nonce":1398737533}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6811","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6813","nonce":1398737533}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6821","nonce":1398737533}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6815","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6816","nonce":1398737533}]},"public_addr":"172.21.15.16:6810/1398737533","cluster_addr":"172.21.15.16:6813/1398737533","heartbeat_back_addr":"172.21.15.16:6821/1398737533","heartbeat_front_addr":"172.21.15.16:6816/1398737533","state":["exists","up"]},{"osd":2,"uuid":"01a73662-9a21-4b1f-bcbb-42b9636a4638","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6819","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6822","nonce":3018526646}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6825","nonce":3018526646}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6831","nonce":3018526646}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6827","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6829","nonce":3018526646}]},"public_addr":"172.21.15.16:6822/3018526646","cluster_addr":"172.21.15.16:6825/3018526646","heartbeat_back_addr":"172.21.15.16:6831/3018526646","heartbeat_front_addr":"172.21.15.16:6829/3018526646","state":["exists","up"]},{"osd":3,"uuid":"c7771fd3-e18e-4500-ac38-23cf10ebcd75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6809","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6812","nonce":3360352974}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6817","nonce":3360352974}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6828","nonce":3360352974}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6823","nonce":3360352974}]},"public_addr":"172.21.15.16:6812/3360352974","cluster_addr":"172.21.15.16:6817/3360352974","heartbeat_back_addr":"172.21.15.16:6828/3360352974","heartbeat_front_addr":"172.21.15.16:6823/3360352974","state":["exists","up"]},{"osd":4,"uuid":"88260730-1482-4fbd-a40f-cb1124e9df4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6801","nonce":1856844856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6803","nonce":1856844856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6807","nonce":1856844856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6805","nonce":1856844856}]},"public_addr":"172.21.15.89:6801/1856844856","cluster_addr":"172.21.15.89:6803/1856844856","heartbeat_back_addr":"172.21.15.89:6807/1856844856","heartbeat_front_addr":"172.21.15.89:6805/1856844856","state":["exists","up"]},{"osd":5,"uuid":"4cb2c00c-fae0-4dd8-9d27-ec4f0093ccf7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6814","nonce":50395527}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6817","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6820","nonce":50395527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6831","nonce":50395527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6823","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6826","nonce":50395527}]},"public_addr":"172.21.15.89:6814/50395527","cluster_addr":"172.21.15.89:6820/50395527","heartbeat_back_addr":"172.21.15.89:6831/50395527","heartbeat_front_addr":"172.21.15.89:6826/50395527","state":["exists","up"]},{"osd":6,"uuid":"81ce0c19-2554-4cae-acbe-66a05e9e9b9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6809","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6812","nonce":3470471752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6815","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6818","nonce":3470471752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6827","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6829","nonce":3470471752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6821","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6824","nonce":3470471752}]},"public_addr":"172.21.15.89:6812/3470471752","cluster_addr":"172.21.15.89:6818/3470471752","heartbeat_back_addr":"172.21.15.89:6829/3470471752","heartbeat_front_addr":"172.21.15.89:6824/3470471752","state":["exists","up"]},{"osd":7,"uuid":"85e4313b-d37d-442e-8042-153bf20a847d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6811","nonce":1796778630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6813","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6816","nonce":1796778630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6825","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6828","nonce":1796778630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6819","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6822","nonce":1796778630}]},"public_addr":"172.21.15.89:6811/1796778630","cluster_addr":"172.21.15.89:6816/1796778630","heartbeat_back_addr":"172.21.15.89:6828/1796778630","heartbeat_front_addr":"172.21.15.89:6822/1796778630","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.439088+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.529970+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.680512+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:21.044161+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.582111+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.536558+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.409522+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.590229+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"17.1","mappings":[{"from":0,"to":5}]},{"pgid":"17.21","mappings":[{"from":0,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-31T15:14:37.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:37.577+0000 7f93d6a00640 1 -- 172.21.15.16:0/2308310780 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f93b0068380 msgr2=0x7f93b006a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:37.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:37.577+0000 7f93d6a00640 1 --2- 172.21.15.16:0/2308310780 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f93b0068380 0x7f93b006a840 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f93d00b2450 tx=0x7f93bc009190 comp rx=0 tx=0).stop 2024-03-31T15:14:37.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:37.577+0000 7f93d6a00640 1 -- 172.21.15.16:0/2308310780 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f93d00b5b90 msgr2=0x7f93d0159820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:37.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:37.577+0000 7f93d6a00640 1 --2- 172.21.15.16:0/2308310780 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f93d00b5b90 0x7f93d0159820 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f93d0073700 tx=0x7f93c401ada0 comp rx=0 tx=0).stop 2024-03-31T15:14:37.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:37.577+0000 7f93d6a00640 1 -- 172.21.15.16:0/2308310780 shutdown_connections 2024-03-31T15:14:37.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:37.577+0000 7f93d6a00640 1 --2- 172.21.15.16:0/2308310780 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f93b0068380 0x7f93b006a840 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:37.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:37.577+0000 7f93d6a00640 1 --2- 172.21.15.16:0/2308310780 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f93d00d0640 0x7f93d009bbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:37.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:37.577+0000 7f93d6a00640 1 --2- 172.21.15.16:0/2308310780 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f93d00cdd60 0x7f93d006c4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:37.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:37.577+0000 7f93d6a00640 1 --2- 172.21.15.16:0/2308310780 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f93d00b5b90 0x7f93d0159820 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:37.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:37.577+0000 7f93d6a00640 1 -- 172.21.15.16:0/2308310780 >> 172.21.15.16:0/2308310780 conn(0x7f93d00b2ac0 msgr2=0x7f93d006b620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:14:37.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:37.577+0000 7f93d6a00640 1 -- 172.21.15.16:0/2308310780 shutdown_connections 2024-03-31T15:14:37.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:37.577+0000 7f93d6a00640 1 -- 172.21.15.16:0/2308310780 wait complete. 2024-03-31T15:14:37.593 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-31T15:03:22.550787+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 16, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-31T15:14:06.417304+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '108', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 17, 'pool_name': 'cephfs_data', 'create_time': '2024-03-31T15:14:07.622562+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '109', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-31T15:14:37.593 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-31T15:14:37.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:37.741+0000 7f68dae00640 1 Processor -- start 2024-03-31T15:14:37.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:37.741+0000 7f68dae00640 1 -- start start 2024-03-31T15:14:37.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:37.741+0000 7f68dae00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f68d406f920 0x7f68d406fd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:37.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:37.741+0000 7f68dae00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f68d40cf150 0x7f68d41419c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:37.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:37.741+0000 7f68dae00640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f68d4070240 con 0x7f68d40d0a60 2024-03-31T15:14:37.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:37.741+0000 7f68dae00640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f68d413f670 con 0x7f68d40cf150 2024-03-31T15:14:37.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:37.741+0000 7f68dae00640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f68d415ca70 con 0x7f68d406f920 2024-03-31T15:14:37.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:37.745+0000 7f68d8e00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f68d40cf150 0x7f68d41419c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:37.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:37.745+0000 7f68d8e00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f68d40cf150 0x7f68d41419c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40596/0 (socket says 172.21.15.16:40596) 2024-03-31T15:14:37.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:37.745+0000 7f68d8e00640 1 -- 172.21.15.16:0/2434073767 learned_addr learned my addr 172.21.15.16:0/2434073767 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:14:37.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:37.745+0000 7f68d3e00640 1 --2- 172.21.15.16:0/2434073767 >> v2:172.21.15.89:3300/0 conn(0x7f68d406f920 0x7f68d406fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:37.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:37.745+0000 7f68d8e00640 1 -- 172.21.15.16:0/2434073767 >> v2:172.21.15.89:3300/0 conn(0x7f68d406f920 msgr2=0x7f68d406fd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:14:37.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:37.745+0000 7f68d8e00640 1 --2- 172.21.15.16:0/2434073767 >> v2:172.21.15.89:3300/0 conn(0x7f68d406f920 0x7f68d406fd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:37.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:37.745+0000 7f68d8e00640 1 -- 172.21.15.16:0/2434073767 >> v1:172.21.15.89:6789/0 conn(0x7f68d40d0a60 legacy=0x7f68d40cea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:37.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:37.745+0000 7f68d8e00640 1 -- 172.21.15.16:0/2434073767 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68d415cd50 con 0x7f68d40cf150 2024-03-31T15:14:37.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:37.745+0000 7f68d8e00640 1 --2- 172.21.15.16:0/2434073767 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f68d40cf150 0x7f68d41419c0 secure :-1 s=READY pgs=1725 cs=0 l=1 rev1=1 crypto rx=0x7f68d40708e0 tx=0x7f68cc01c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c017e169bf4f880 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:37.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:37.745+0000 7f68d3400640 1 -- 172.21.15.16:0/2434073767 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68cc0a5620 con 0x7f68d40cf150 2024-03-31T15:14:37.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:37.745+0000 7f68d3400640 1 -- 172.21.15.16:0/2434073767 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f68cc0a57c0 con 0x7f68d40cf150 2024-03-31T15:14:37.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:37.745+0000 7f68d3400640 1 -- 172.21.15.16:0/2434073767 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68cc0a5b90 con 0x7f68d40cf150 2024-03-31T15:14:37.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:37.749+0000 7f68dae00640 1 -- 172.21.15.16:0/2434073767 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f68d40cf150 msgr2=0x7f68d41419c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:37.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:37.749+0000 7f68dae00640 1 --2- 172.21.15.16:0/2434073767 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f68d40cf150 0x7f68d41419c0 secure :-1 s=READY pgs=1725 cs=0 l=1 rev1=1 crypto rx=0x7f68d40708e0 tx=0x7f68cc01c960 comp rx=0 tx=0).stop 2024-03-31T15:14:37.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:37.749+0000 7f68dae00640 1 -- 172.21.15.16:0/2434073767 shutdown_connections 2024-03-31T15:14:37.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:37.749+0000 7f68dae00640 1 --2- 172.21.15.16:0/2434073767 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f68d40cf150 0x7f68d41419c0 unknown :-1 s=CLOSED pgs=1725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:37.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:37.749+0000 7f68dae00640 1 --2- 172.21.15.16:0/2434073767 >> v2:172.21.15.89:3300/0 conn(0x7f68d406f920 0x7f68d406fd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:37.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:37.749+0000 7f68dae00640 1 -- 172.21.15.16:0/2434073767 >> 172.21.15.16:0/2434073767 conn(0x7f68d4067930 msgr2=0x7f68d40b3ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:14:37.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:37.749+0000 7f68dae00640 1 -- 172.21.15.16:0/2434073767 shutdown_connections 2024-03-31T15:14:37.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:37.749+0000 7f68dae00640 1 -- 172.21.15.16:0/2434073767 wait complete. 2024-03-31T15:14:37.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:37.749+0000 7f68dae00640 1 Processor -- start 2024-03-31T15:14:37.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:37.749+0000 7f68dae00640 1 -- start start 2024-03-31T15:14:37.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:37.749+0000 7f68dae00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f68d406f920 0x7f68d40b3960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:37.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:37.749+0000 7f68dae00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f68d40cf150 0x7f68d415eb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:37.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:37.749+0000 7f68dae00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f68d40d0a60 0x7f68d406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:37.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:37.749+0000 7f68dae00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f68d4169230 con 0x7f68d40cf150 2024-03-31T15:14:37.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:37.749+0000 7f68dae00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f68d41690b0 con 0x7f68d40d0a60 2024-03-31T15:14:37.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:37.749+0000 7f68dae00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f68d41693b0 con 0x7f68d406f920 2024-03-31T15:14:37.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:37.749+0000 7f68cbe00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f68d40cf150 0x7f68d415eb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:37.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:37.749+0000 7f68cbe00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f68d40cf150 0x7f68d415eb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40604/0 (socket says 172.21.15.16:40604) 2024-03-31T15:14:37.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:37.749+0000 7f68cbe00640 1 -- 172.21.15.16:0/3589037849 learned_addr learned my addr 172.21.15.16:0/3589037849 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:14:37.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:37.749+0000 7f68d3e00640 1 --2- 172.21.15.16:0/3589037849 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f68d406f920 0x7f68d40b3960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:37.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:37.749+0000 7f68d8e00640 1 --2- 172.21.15.16:0/3589037849 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f68d40d0a60 0x7f68d406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:37.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:37.749+0000 7f68cbe00640 1 -- 172.21.15.16:0/3589037849 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f68d406f920 msgr2=0x7f68d40b3960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:37.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:37.749+0000 7f68cbe00640 1 --2- 172.21.15.16:0/3589037849 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f68d406f920 0x7f68d40b3960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:37.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:37.749+0000 7f68cbe00640 1 -- 172.21.15.16:0/3589037849 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f68d40d0a60 msgr2=0x7f68d406d9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:37.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:37.749+0000 7f68cbe00640 1 --2- 172.21.15.16:0/3589037849 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f68d40d0a60 0x7f68d406d9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:37.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:37.749+0000 7f68cbe00640 1 -- 172.21.15.16:0/3589037849 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68d40b3ea0 con 0x7f68d40cf150 2024-03-31T15:14:37.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:37.753+0000 7f68cbe00640 1 --2- 172.21.15.16:0/3589037849 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f68d40cf150 0x7f68d415eb00 secure :-1 s=READY pgs=1726 cs=0 l=1 rev1=1 crypto rx=0x7f68bc00c340 tx=0x7f68bc00c800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:37.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:37.753+0000 7f68d0c00640 1 -- 172.21.15.16:0/3589037849 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68bc0052d0 con 0x7f68d40cf150 2024-03-31T15:14:37.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:37.753+0000 7f68d0c00640 1 -- 172.21.15.16:0/3589037849 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f68bc005470 con 0x7f68d40cf150 2024-03-31T15:14:37.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:37.753+0000 7f68d0c00640 1 -- 172.21.15.16:0/3589037849 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68bc007260 con 0x7f68d40cf150 2024-03-31T15:14:37.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:37.753+0000 7f68dae00640 1 -- 172.21.15.16:0/3589037849 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f68d406e380 con 0x7f68d40cf150 2024-03-31T15:14:37.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:37.753+0000 7f68dae00640 1 -- 172.21.15.16:0/3589037849 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f68d40b3160 con 0x7f68d40cf150 2024-03-31T15:14:37.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:37.753+0000 7f68dae00640 1 -- 172.21.15.16:0/3589037849 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f68d4069950 con 0x7f68d40cf150 2024-03-31T15:14:37.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:37.753+0000 7f68d0c00640 1 -- 172.21.15.16:0/3589037849 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f68bc007400 con 0x7f68d40cf150 2024-03-31T15:14:37.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:37.757+0000 7f68d0c00640 1 --2- 172.21.15.16:0/3589037849 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f68b0068380 0x7f68b006a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:37.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:37.757+0000 7f68d0c00640 1 -- 172.21.15.16:0/3589037849 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 5767+0+0 (secure 0 0 0) 0x7f68bc085d80 con 0x7f68d40cf150 2024-03-31T15:14:37.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:37.757+0000 7f68d3e00640 1 --2- 172.21.15.16:0/3589037849 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f68b0068380 0x7f68b006a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:37.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:37.757+0000 7f68d3e00640 1 --2- 172.21.15.16:0/3589037849 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f68b0068380 0x7f68b006a840 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f68c4007a60 tx=0x7f68c4008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:37.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:37.761+0000 7f68d0c00640 1 -- 172.21.15.16:0/3589037849 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f68bc055cb0 con 0x7f68d40cf150 2024-03-31T15:14:37.906 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-31T15:14:37.909 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-31T15:14:37.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:37.925+0000 7f68dae00640 1 -- 172.21.15.16:0/3589037849 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f68d40b4a20 con 0x7f68d40cf150 2024-03-31T15:14:37.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:37.925+0000 7f68d0c00640 1 -- 172.21.15.16:0/3589037849 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v111) ==== 93+0+10 (secure 0 0 0) 0x7f68bc059fb0 con 0x7f68d40cf150 2024-03-31T15:14:37.918 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2024-03-31T15:14:37.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:37.925+0000 7f68dae00640 1 -- 172.21.15.16:0/3589037849 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f68b0068380 msgr2=0x7f68b006a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:37.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:37.925+0000 7f68dae00640 1 --2- 172.21.15.16:0/3589037849 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f68b0068380 0x7f68b006a840 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f68c4007a60 tx=0x7f68c4008040 comp rx=0 tx=0).stop 2024-03-31T15:14:37.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:37.925+0000 7f68dae00640 1 -- 172.21.15.16:0/3589037849 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f68d40cf150 msgr2=0x7f68d415eb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:37.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:37.925+0000 7f68dae00640 1 --2- 172.21.15.16:0/3589037849 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f68d40cf150 0x7f68d415eb00 secure :-1 s=READY pgs=1726 cs=0 l=1 rev1=1 crypto rx=0x7f68bc00c340 tx=0x7f68bc00c800 comp rx=0 tx=0).stop 2024-03-31T15:14:37.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:37.925+0000 7f68dae00640 1 -- 172.21.15.16:0/3589037849 shutdown_connections 2024-03-31T15:14:37.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:37.925+0000 7f68dae00640 1 --2- 172.21.15.16:0/3589037849 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f68b0068380 0x7f68b006a840 unknown :-1 s=CLOSED pgs=732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:37.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:37.925+0000 7f68dae00640 1 --2- 172.21.15.16:0/3589037849 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f68d40d0a60 0x7f68d406d9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:37.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:37.925+0000 7f68dae00640 1 --2- 172.21.15.16:0/3589037849 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f68d40cf150 0x7f68d415eb00 unknown :-1 s=CLOSED pgs=1726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:37.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:37.925+0000 7f68dae00640 1 --2- 172.21.15.16:0/3589037849 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f68d406f920 0x7f68d40b3960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:37.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:37.925+0000 7f68dae00640 1 -- 172.21.15.16:0/3589037849 >> 172.21.15.16:0/3589037849 conn(0x7f68d4067930 msgr2=0x7f68d415ab20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:14:37.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:37.925+0000 7f68dae00640 1 -- 172.21.15.16:0/3589037849 shutdown_connections 2024-03-31T15:14:37.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:37.925+0000 7f68dae00640 1 -- 172.21.15.16:0/3589037849 wait complete. 2024-03-31T15:14:37.940 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-31T15:14:38.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:38.153+0000 7f96be200640 1 Processor -- start 2024-03-31T15:14:38.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:38.153+0000 7f96be200640 1 -- start start 2024-03-31T15:14:38.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:38.153+0000 7f96be200640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f96b806d4d0 0x7f96b806d930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:38.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:38.153+0000 7f96be200640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f96b8071550 0x7f96b8071980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:38.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:38.153+0000 7f96be200640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f96b806de70 con 0x7f96b81567e0 2024-03-31T15:14:38.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:38.153+0000 7f96be200640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f96b815c7e0 con 0x7f96b806d4d0 2024-03-31T15:14:38.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:38.153+0000 7f96be200640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f96b80736b0 con 0x7f96b8071550 2024-03-31T15:14:38.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:38.153+0000 7f96b7e00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f96b8071550 0x7f96b8071980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:38.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:38.153+0000 7f96b7e00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f96b8071550 0x7f96b8071980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3301/0 says I am v2:172.21.15.16:57624/0 (socket says 172.21.15.16:57624) 2024-03-31T15:14:38.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:38.153+0000 7f96b7400640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f96b81567e0 0x7f96b8072c80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.16:58470/0 (socket says 172.21.15.16:58470) 2024-03-31T15:14:38.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:38.153+0000 7f96b7e00640 1 -- 172.21.15.16:0/932254646 learned_addr learned my addr 172.21.15.16:0/932254646 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:14:38.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:38.153+0000 7f96aea00640 1 --2- 172.21.15.16:0/932254646 >> v2:172.21.15.89:3300/0 conn(0x7f96b806d4d0 0x7f96b806d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:38.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:38.153+0000 7f96b7e00640 1 -- 172.21.15.16:0/932254646 >> v2:172.21.15.89:3300/0 conn(0x7f96b806d4d0 msgr2=0x7f96b806d930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:14:38.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:38.153+0000 7f96b7e00640 1 --2- 172.21.15.16:0/932254646 >> v2:172.21.15.89:3300/0 conn(0x7f96b806d4d0 0x7f96b806d930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:38.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:38.153+0000 7f96b7e00640 1 -- 172.21.15.16:0/932254646 >> v1:172.21.15.89:6789/0 conn(0x7f96b81567e0 legacy=0x7f96b8072c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:38.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:38.153+0000 7f96b6a00640 1 -- 172.21.15.16:0/932254646 <== mon.1 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 435799510 0 0) 0x7f96b806de70 con 0x7f96b81567e0 2024-03-31T15:14:38.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:38.153+0000 7f96b7e00640 1 -- 172.21.15.16:0/932254646 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96b815caa0 con 0x7f96b8071550 2024-03-31T15:14:38.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:38.153+0000 7f96aea00640 1 --2- 172.21.15.16:0/932254646 >> v2:172.21.15.89:3300/0 conn(0x7f96b806d4d0 0x7f96b806d930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:14:38.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:38.157+0000 7f96b7e00640 1 --2- 172.21.15.16:0/932254646 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f96b8071550 0x7f96b8071980 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f96b0009870 tx=0x7f96b001cfc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=835e2ca6aa146dec server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:38.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:38.157+0000 7f96b6a00640 1 -- 172.21.15.16:0/932254646 <== mon.2 v2:172.21.15.89:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96b00a7070 con 0x7f96b8071550 2024-03-31T15:14:38.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:38.157+0000 7f96b6a00640 1 -- 172.21.15.16:0/932254646 <== mon.2 v2:172.21.15.89:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f96b00b0070 con 0x7f96b8071550 2024-03-31T15:14:38.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:38.157+0000 7f96b6a00640 1 -- 172.21.15.16:0/932254646 <== mon.2 v2:172.21.15.89:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96b00ab890 con 0x7f96b8071550 2024-03-31T15:14:38.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:38.157+0000 7f96be200640 1 -- 172.21.15.16:0/932254646 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f96b8071550 msgr2=0x7f96b8071980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:38.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:38.157+0000 7f96be200640 1 --2- 172.21.15.16:0/932254646 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f96b8071550 0x7f96b8071980 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f96b0009870 tx=0x7f96b001cfc0 comp rx=0 tx=0).stop 2024-03-31T15:14:38.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:38.157+0000 7f96be200640 1 -- 172.21.15.16:0/932254646 shutdown_connections 2024-03-31T15:14:38.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:38.157+0000 7f96be200640 1 --2- 172.21.15.16:0/932254646 >> v2:172.21.15.89:3300/0 conn(0x7f96b806d4d0 0x7f96b806d930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:38.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:38.157+0000 7f96be200640 1 --2- 172.21.15.16:0/932254646 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f96b8071550 0x7f96b8071980 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:38.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:38.157+0000 7f96be200640 1 -- 172.21.15.16:0/932254646 >> 172.21.15.16:0/932254646 conn(0x7f96b806ee40 msgr2=0x7f96b806b620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:14:38.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:38.157+0000 7f96be200640 1 -- 172.21.15.16:0/932254646 shutdown_connections 2024-03-31T15:14:38.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:38.157+0000 7f96be200640 1 -- 172.21.15.16:0/932254646 wait complete. 2024-03-31T15:14:38.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:38.161+0000 7f96be200640 1 Processor -- start 2024-03-31T15:14:38.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:38.161+0000 7f96be200640 1 -- start start 2024-03-31T15:14:38.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:38.165+0000 7f96be200640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f96b806d4d0 0x7f96b80b2eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:38.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:38.165+0000 7f96be200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f96b8071550 0x7f96b815d7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:38.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:38.165+0000 7f96be200640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f96b81567e0 0x7f96b814eea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:38.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:38.165+0000 7f96be200640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f96b80afbb0 con 0x7f96b8071550 2024-03-31T15:14:38.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:38.165+0000 7f96be200640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f96b80afa30 con 0x7f96b81567e0 2024-03-31T15:14:38.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:38.165+0000 7f96be200640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f96b80722d0 con 0x7f96b806d4d0 2024-03-31T15:14:38.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:38.165+0000 7f96aea00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f96b8071550 0x7f96b815d7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:38.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:38.165+0000 7f96aea00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f96b8071550 0x7f96b815d7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40616/0 (socket says 172.21.15.16:40616) 2024-03-31T15:14:38.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:38.165+0000 7f96aea00640 1 -- 172.21.15.16:0/974184842 learned_addr learned my addr 172.21.15.16:0/974184842 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:14:38.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:38.165+0000 7f96b7400640 1 --2- 172.21.15.16:0/974184842 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f96b806d4d0 0x7f96b80b2eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:38.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:38.165+0000 7f96b7e00640 1 --2- 172.21.15.16:0/974184842 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f96b81567e0 0x7f96b814eea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:38.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:38.165+0000 7f96aea00640 1 -- 172.21.15.16:0/974184842 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f96b806d4d0 msgr2=0x7f96b80b2eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:38.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:38.165+0000 7f96aea00640 1 --2- 172.21.15.16:0/974184842 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f96b806d4d0 0x7f96b80b2eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:38.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:38.165+0000 7f96aea00640 1 -- 172.21.15.16:0/974184842 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f96b81567e0 msgr2=0x7f96b814eea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:38.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:38.165+0000 7f96aea00640 1 --2- 172.21.15.16:0/974184842 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f96b81567e0 0x7f96b814eea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:38.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:38.165+0000 7f96aea00640 1 -- 172.21.15.16:0/974184842 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96b8159ad0 con 0x7f96b8071550 2024-03-31T15:14:38.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:38.165+0000 7f96aea00640 1 --2- 172.21.15.16:0/974184842 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f96b8071550 0x7f96b815d7e0 secure :-1 s=READY pgs=1727 cs=0 l=1 rev1=1 crypto rx=0x7f96a000efc0 tx=0x7f96a000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:38.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:38.165+0000 7f96afe00640 1 -- 172.21.15.16:0/974184842 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96a0018020 con 0x7f96b8071550 2024-03-31T15:14:38.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:38.165+0000 7f96afe00640 1 -- 172.21.15.16:0/974184842 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f96a0014680 con 0x7f96b8071550 2024-03-31T15:14:38.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:38.165+0000 7f96afe00640 1 -- 172.21.15.16:0/974184842 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96a0014a50 con 0x7f96b8071550 2024-03-31T15:14:38.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:38.165+0000 7f96be200640 1 -- 172.21.15.16:0/974184842 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f96b8159db0 con 0x7f96b8071550 2024-03-31T15:14:38.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:38.165+0000 7f96be200640 1 -- 172.21.15.16:0/974184842 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f96b814e980 con 0x7f96b8071550 2024-03-31T15:14:38.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:38.165+0000 7f96be200640 1 -- 172.21.15.16:0/974184842 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f96b8069950 con 0x7f96b8071550 2024-03-31T15:14:38.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:38.169+0000 7f96afe00640 1 -- 172.21.15.16:0/974184842 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f96a001b020 con 0x7f96b8071550 2024-03-31T15:14:38.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:38.169+0000 7f96afe00640 1 --2- 172.21.15.16:0/974184842 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f9694068380 0x7f969406a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:38.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:38.169+0000 7f96afe00640 1 -- 172.21.15.16:0/974184842 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 5767+0+0 (secure 0 0 0) 0x7f96a0087690 con 0x7f96b8071550 2024-03-31T15:14:38.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:38.169+0000 7f96b7400640 1 --2- 172.21.15.16:0/974184842 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f9694068380 0x7f969406a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:38.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:38.169+0000 7f96b7400640 1 --2- 172.21.15.16:0/974184842 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f9694068380 0x7f969406a840 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f96a8007fd0 tx=0x7f96a8007d60 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:38.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:38.173+0000 7f96afe00640 1 -- 172.21.15.16:0/974184842 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f96a0022030 con 0x7f96b8071550 2024-03-31T15:14:38.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:38.333+0000 7f96be200640 1 -- 172.21.15.16:0/974184842 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f96b806e470 con 0x7f96b8071550 2024-03-31T15:14:38.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:38.333+0000 7f96afe00640 1 -- 172.21.15.16:0/974184842 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v111) ==== 104+0+11 (secure 0 0 0) 0x7f96a0010040 con 0x7f96b8071550 2024-03-31T15:14:38.329 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2024-03-31T15:14:38.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:38.337+0000 7f96be200640 1 -- 172.21.15.16:0/974184842 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f9694068380 msgr2=0x7f969406a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:38.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:38.337+0000 7f96be200640 1 --2- 172.21.15.16:0/974184842 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f9694068380 0x7f969406a840 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f96a8007fd0 tx=0x7f96a8007d60 comp rx=0 tx=0).stop 2024-03-31T15:14:38.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:38.337+0000 7f96be200640 1 -- 172.21.15.16:0/974184842 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f96b8071550 msgr2=0x7f96b815d7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:38.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:38.337+0000 7f96be200640 1 --2- 172.21.15.16:0/974184842 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f96b8071550 0x7f96b815d7e0 secure :-1 s=READY pgs=1727 cs=0 l=1 rev1=1 crypto rx=0x7f96a000efc0 tx=0x7f96a000c490 comp rx=0 tx=0).stop 2024-03-31T15:14:38.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:38.337+0000 7f96be200640 1 -- 172.21.15.16:0/974184842 shutdown_connections 2024-03-31T15:14:38.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:38.337+0000 7f96be200640 1 --2- 172.21.15.16:0/974184842 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f9694068380 0x7f969406a840 unknown :-1 s=CLOSED pgs=733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:38.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:38.337+0000 7f96be200640 1 --2- 172.21.15.16:0/974184842 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f96b81567e0 0x7f96b814eea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:38.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:38.337+0000 7f96be200640 1 --2- 172.21.15.16:0/974184842 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f96b8071550 0x7f96b815d7e0 unknown :-1 s=CLOSED pgs=1727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:38.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:38.337+0000 7f96be200640 1 --2- 172.21.15.16:0/974184842 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f96b806d4d0 0x7f96b80b2eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:38.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:38.337+0000 7f96be200640 1 -- 172.21.15.16:0/974184842 >> 172.21.15.16:0/974184842 conn(0x7f96b806ee40 msgr2=0x7f96b814ffb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:14:38.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:38.337+0000 7f96be200640 1 -- 172.21.15.16:0/974184842 shutdown_connections 2024-03-31T15:14:38.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:38.337+0000 7f96be200640 1 -- 172.21.15.16:0/974184842 wait complete. 2024-03-31T15:14:38.351 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-31T15:14:38.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:38.501+0000 7f3b45a00640 1 Processor -- start 2024-03-31T15:14:38.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:38.501+0000 7f3b45a00640 1 -- start start 2024-03-31T15:14:38.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:38.501+0000 7f3b45a00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f3b400d0a60 0x7f3b400cea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:38.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:38.501+0000 7f3b45a00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f3b400cef80 0x7f3b401419c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:38.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:38.501+0000 7f3b45a00640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3b4015ca70 con 0x7f3b4006f920 2024-03-31T15:14:38.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:38.501+0000 7f3b45a00640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f3b4013f670 con 0x7f3b400cef80 2024-03-31T15:14:38.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:38.501+0000 7f3b45a00640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f3b4015cd50 con 0x7f3b400d0a60 2024-03-31T15:14:38.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:38.501+0000 7f3b3ea00640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f3b4006f920 0x7f3b4006fd00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:33436/0 (socket says 172.21.15.16:33436) 2024-03-31T15:14:38.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:38.501+0000 7f3b3ea00640 1 -- 172.21.15.16:0/94242312 learned_addr learned my addr 172.21.15.16:0/94242312 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:14:38.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:38.501+0000 7f3b3f400640 1 --2- 172.21.15.16:0/94242312 >> v2:172.21.15.16:3300/0 conn(0x7f3b400cef80 0x7f3b401419c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:38.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:38.501+0000 7f3b37e00640 1 --2- 172.21.15.16:0/94242312 >> v2:172.21.15.89:3300/0 conn(0x7f3b400d0a60 0x7f3b400cea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:38.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:38.501+0000 7f3b3e000640 1 -- 172.21.15.16:0/94242312 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3989591230 0 0) 0x7f3b4015ca70 con 0x7f3b4006f920 2024-03-31T15:14:38.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:38.501+0000 7f3b3e000640 1 -- 172.21.15.16:0/94242312 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3b24003530 con 0x7f3b4006f920 2024-03-31T15:14:38.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:38.501+0000 7f3b3f400640 1 -- 172.21.15.16:0/94242312 >> v2:172.21.15.89:3300/0 conn(0x7f3b400d0a60 msgr2=0x7f3b400cea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:14:38.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:38.501+0000 7f3b3f400640 1 --2- 172.21.15.16:0/94242312 >> v2:172.21.15.89:3300/0 conn(0x7f3b400d0a60 0x7f3b400cea40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:38.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:38.501+0000 7f3b3f400640 1 -- 172.21.15.16:0/94242312 >> v1:172.21.15.16:6789/0 conn(0x7f3b4006f920 legacy=0x7f3b4006fd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:38.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:38.501+0000 7f3b3f400640 1 -- 172.21.15.16:0/94242312 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b4015d030 con 0x7f3b400cef80 2024-03-31T15:14:38.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:38.505+0000 7f3b3f400640 1 --2- 172.21.15.16:0/94242312 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3b400cef80 0x7f3b401419c0 secure :-1 s=READY pgs=1729 cs=0 l=1 rev1=1 crypto rx=0x7f3b400b5070 tx=0x7f3b3801c930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0fa5ac3a4d3ffeb server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:38.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:38.505+0000 7f3b3e000640 1 -- 172.21.15.16:0/94242312 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b380b2070 con 0x7f3b400cef80 2024-03-31T15:14:38.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:38.505+0000 7f3b3e000640 1 -- 172.21.15.16:0/94242312 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3b380a57f0 con 0x7f3b400cef80 2024-03-31T15:14:38.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:38.505+0000 7f3b3e000640 1 -- 172.21.15.16:0/94242312 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b380a5bc0 con 0x7f3b400cef80 2024-03-31T15:14:38.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:38.505+0000 7f3b45a00640 1 -- 172.21.15.16:0/94242312 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3b400cef80 msgr2=0x7f3b401419c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:38.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:38.505+0000 7f3b45a00640 1 --2- 172.21.15.16:0/94242312 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3b400cef80 0x7f3b401419c0 secure :-1 s=READY pgs=1729 cs=0 l=1 rev1=1 crypto rx=0x7f3b400b5070 tx=0x7f3b3801c930 comp rx=0 tx=0).stop 2024-03-31T15:14:38.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:38.505+0000 7f3b45a00640 1 -- 172.21.15.16:0/94242312 shutdown_connections 2024-03-31T15:14:38.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:38.505+0000 7f3b45a00640 1 --2- 172.21.15.16:0/94242312 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3b400cef80 0x7f3b401419c0 unknown :-1 s=CLOSED pgs=1729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:38.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:38.505+0000 7f3b45a00640 1 --2- 172.21.15.16:0/94242312 >> v2:172.21.15.89:3300/0 conn(0x7f3b400d0a60 0x7f3b400cea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:38.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:38.505+0000 7f3b45a00640 1 -- 172.21.15.16:0/94242312 >> 172.21.15.16:0/94242312 conn(0x7f3b40067930 msgr2=0x7f3b400b3ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:14:38.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:38.505+0000 7f3b45a00640 1 -- 172.21.15.16:0/94242312 shutdown_connections 2024-03-31T15:14:38.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:38.505+0000 7f3b45a00640 1 -- 172.21.15.16:0/94242312 wait complete. 2024-03-31T15:14:38.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:38.509+0000 7f3b45a00640 1 Processor -- start 2024-03-31T15:14:38.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:38.509+0000 7f3b45a00640 1 -- start start 2024-03-31T15:14:38.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:38.509+0000 7f3b45a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3b4006f920 0x7f3b400b5a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:38.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:38.509+0000 7f3b45a00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3b400cef80 0x7f3b40140aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:38.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:38.509+0000 7f3b45a00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f3b400d0a60 0x7f3b40155350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:38.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:38.509+0000 7f3b45a00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f3b40169230 con 0x7f3b4006f920 2024-03-31T15:14:38.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:38.509+0000 7f3b45a00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f3b401690b0 con 0x7f3b400cef80 2024-03-31T15:14:38.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:38.509+0000 7f3b45a00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f3b401693b0 con 0x7f3b400d0a60 2024-03-31T15:14:38.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:38.509+0000 7f3b3ea00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3b4006f920 0x7f3b400b5a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:38.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:38.509+0000 7f3b3ea00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3b4006f920 0x7f3b400b5a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40644/0 (socket says 172.21.15.16:40644) 2024-03-31T15:14:38.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:38.509+0000 7f3b3ea00640 1 -- 172.21.15.16:0/3747655828 learned_addr learned my addr 172.21.15.16:0/3747655828 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:14:38.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:38.509+0000 7f3b3f400640 1 --2- 172.21.15.16:0/3747655828 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f3b400d0a60 0x7f3b40155350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:38.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:38.509+0000 7f3b37e00640 1 --2- 172.21.15.16:0/3747655828 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3b400cef80 0x7f3b40140aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:38.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:38.509+0000 7f3b3ea00640 1 -- 172.21.15.16:0/3747655828 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f3b400d0a60 msgr2=0x7f3b40155350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:38.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:38.509+0000 7f3b3ea00640 1 --2- 172.21.15.16:0/3747655828 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f3b400d0a60 0x7f3b40155350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:38.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:38.509+0000 7f3b3ea00640 1 -- 172.21.15.16:0/3747655828 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3b400cef80 msgr2=0x7f3b40140aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:38.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:38.509+0000 7f3b3ea00640 1 --2- 172.21.15.16:0/3747655828 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3b400cef80 0x7f3b40140aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:38.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:38.509+0000 7f3b3ea00640 1 -- 172.21.15.16:0/3747655828 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b400b5fb0 con 0x7f3b4006f920 2024-03-31T15:14:38.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:38.509+0000 7f3b3f400640 1 --2- 172.21.15.16:0/3747655828 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f3b400d0a60 0x7f3b40155350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:14:38.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:38.509+0000 7f3b37e00640 1 --2- 172.21.15.16:0/3747655828 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3b400cef80 0x7f3b40140aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:14:38.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:38.509+0000 7f3b3ea00640 1 --2- 172.21.15.16:0/3747655828 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3b4006f920 0x7f3b400b5a70 secure :-1 s=READY pgs=1730 cs=0 l=1 rev1=1 crypto rx=0x7f3b3000c450 tx=0x7f3b3000c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:38.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:38.509+0000 7f3b36a00640 1 -- 172.21.15.16:0/3747655828 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b30017070 con 0x7f3b4006f920 2024-03-31T15:14:38.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:38.509+0000 7f3b36a00640 1 -- 172.21.15.16:0/3747655828 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3b30005cf0 con 0x7f3b4006f920 2024-03-31T15:14:38.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:38.509+0000 7f3b36a00640 1 -- 172.21.15.16:0/3747655828 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b30007290 con 0x7f3b4006f920 2024-03-31T15:14:38.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:38.509+0000 7f3b45a00640 1 -- 172.21.15.16:0/3747655828 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3b401570d0 con 0x7f3b4006f920 2024-03-31T15:14:38.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:38.509+0000 7f3b45a00640 1 -- 172.21.15.16:0/3747655828 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3b4006dbc0 con 0x7f3b4006f920 2024-03-31T15:14:38.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:38.513+0000 7f3b45a00640 1 -- 172.21.15.16:0/3747655828 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3b40069950 con 0x7f3b4006f920 2024-03-31T15:14:38.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:38.513+0000 7f3b36a00640 1 -- 172.21.15.16:0/3747655828 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f3b3001a070 con 0x7f3b4006f920 2024-03-31T15:14:38.507 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:38.513+0000 7f3b36a00640 1 --2- 172.21.15.16:0/3747655828 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f3b1c0683b0 0x7f3b1c06a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:38.507 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:38.513+0000 7f3b36a00640 1 -- 172.21.15.16:0/3747655828 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 5767+0+0 (secure 0 0 0) 0x7f3b30085dd0 con 0x7f3b4006f920 2024-03-31T15:14:38.508 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:38.513+0000 7f3b37e00640 1 --2- 172.21.15.16:0/3747655828 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f3b1c0683b0 0x7f3b1c06a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:38.508 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:38.513+0000 7f3b37e00640 1 --2- 172.21.15.16:0/3747655828 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f3b1c0683b0 0x7f3b1c06a870 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f3b401561c0 tx=0x7f3b28008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:38.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:38.517+0000 7f3b36a00640 1 -- 172.21.15.16:0/3747655828 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f3b30056350 con 0x7f3b4006f920 2024-03-31T15:14:38.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:38.681+0000 7f3b45a00640 1 -- 172.21.15.16:0/3747655828 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f3b400b50e0 con 0x7f3b4006f920 2024-03-31T15:14:38.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:38.681+0000 7f3b36a00640 1 -- 172.21.15.16:0/3747655828 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v111) ==== 100+0+11 (secure 0 0 0) 0x7f3b30055c50 con 0x7f3b4006f920 2024-03-31T15:14:38.675 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 64 2024-03-31T15:14:38.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:38.685+0000 7f3b45a00640 1 -- 172.21.15.16:0/3747655828 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f3b1c0683b0 msgr2=0x7f3b1c06a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:38.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:38.685+0000 7f3b45a00640 1 --2- 172.21.15.16:0/3747655828 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f3b1c0683b0 0x7f3b1c06a870 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f3b401561c0 tx=0x7f3b28008040 comp rx=0 tx=0).stop 2024-03-31T15:14:38.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:38.685+0000 7f3b45a00640 1 -- 172.21.15.16:0/3747655828 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3b4006f920 msgr2=0x7f3b400b5a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:38.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:38.685+0000 7f3b45a00640 1 --2- 172.21.15.16:0/3747655828 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3b4006f920 0x7f3b400b5a70 secure :-1 s=READY pgs=1730 cs=0 l=1 rev1=1 crypto rx=0x7f3b3000c450 tx=0x7f3b3000c910 comp rx=0 tx=0).stop 2024-03-31T15:14:38.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:38.685+0000 7f3b45a00640 1 -- 172.21.15.16:0/3747655828 shutdown_connections 2024-03-31T15:14:38.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:38.685+0000 7f3b45a00640 1 --2- 172.21.15.16:0/3747655828 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f3b1c0683b0 0x7f3b1c06a870 unknown :-1 s=CLOSED pgs=734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:38.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:38.685+0000 7f3b45a00640 1 --2- 172.21.15.16:0/3747655828 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f3b400d0a60 0x7f3b40155350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:38.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:38.685+0000 7f3b45a00640 1 --2- 172.21.15.16:0/3747655828 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3b400cef80 0x7f3b40140aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:38.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:38.685+0000 7f3b45a00640 1 --2- 172.21.15.16:0/3747655828 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3b4006f920 0x7f3b400b5a70 unknown :-1 s=CLOSED pgs=1730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:38.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:38.685+0000 7f3b45a00640 1 -- 172.21.15.16:0/3747655828 >> 172.21.15.16:0/3747655828 conn(0x7f3b40067930 msgr2=0x7f3b401563c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:14:38.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:38.685+0000 7f3b45a00640 1 -- 172.21.15.16:0/3747655828 shutdown_connections 2024-03-31T15:14:38.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:38.685+0000 7f3b45a00640 1 -- 172.21.15.16:0/3747655828 wait complete. 2024-03-31T15:14:38.698 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-31T15:14:38.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:38.849+0000 7fa448200640 1 Processor -- start 2024-03-31T15:14:38.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:38.849+0000 7fa448200640 1 -- start start 2024-03-31T15:14:38.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:38.849+0000 7fa448200640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fa44006dbc0 0x7fa44015f650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:38.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:38.849+0000 7fa448200640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fa440070030 0x7fa440162180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:38.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:38.849+0000 7fa448200640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa4400cbe90 con 0x7fa4400cb440 2024-03-31T15:14:38.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:38.849+0000 7fa448200640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7fa44015fb90 con 0x7fa44006dbc0 2024-03-31T15:14:38.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:38.849+0000 7fa448200640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fa44015f1c0 con 0x7fa440070030 2024-03-31T15:14:38.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:38.849+0000 7fa444e00640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fa4400cb440 0x7fa44006f920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:33442/0 (socket says 172.21.15.16:33442) 2024-03-31T15:14:38.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:38.849+0000 7fa444e00640 1 -- 172.21.15.16:0/2983340816 learned_addr learned my addr 172.21.15.16:0/2983340816 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:14:38.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:38.849+0000 7fa446200640 1 --2- 172.21.15.16:0/2983340816 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fa440070030 0x7fa440162180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:38.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:38.849+0000 7fa445800640 1 --2- 172.21.15.16:0/2983340816 >> v2:172.21.15.89:3300/0 conn(0x7fa44006dbc0 0x7fa44015f650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:38.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:38.849+0000 7fa43fe00640 1 -- 172.21.15.16:0/2983340816 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2702939289 0 0) 0x7fa4400cbe90 con 0x7fa4400cb440 2024-03-31T15:14:38.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:38.849+0000 7fa43fe00640 1 -- 172.21.15.16:0/2983340816 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa428003530 con 0x7fa4400cb440 2024-03-31T15:14:38.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:38.849+0000 7fa446200640 1 -- 172.21.15.16:0/2983340816 >> v2:172.21.15.89:3300/0 conn(0x7fa44006dbc0 msgr2=0x7fa44015f650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:14:38.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:38.849+0000 7fa446200640 1 --2- 172.21.15.16:0/2983340816 >> v2:172.21.15.89:3300/0 conn(0x7fa44006dbc0 0x7fa44015f650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:38.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:38.849+0000 7fa446200640 1 -- 172.21.15.16:0/2983340816 >> v1:172.21.15.16:6789/0 conn(0x7fa4400cb440 legacy=0x7fa44006f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:38.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:38.849+0000 7fa446200640 1 -- 172.21.15.16:0/2983340816 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4400cc170 con 0x7fa440070030 2024-03-31T15:14:38.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:38.849+0000 7fa445800640 1 --2- 172.21.15.16:0/2983340816 >> v2:172.21.15.89:3300/0 conn(0x7fa44006dbc0 0x7fa44015f650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:14:38.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:38.853+0000 7fa446200640 1 --2- 172.21.15.16:0/2983340816 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fa440070030 0x7fa440162180 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fa438009d90 tx=0x7fa43801c8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3e35b39bf66f8234 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:38.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:38.853+0000 7fa43fe00640 1 -- 172.21.15.16:0/2983340816 <== mon.2 v2:172.21.15.89:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4380b2070 con 0x7fa440070030 2024-03-31T15:14:38.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:38.853+0000 7fa43fe00640 1 -- 172.21.15.16:0/2983340816 <== mon.2 v2:172.21.15.89:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa4380a5880 con 0x7fa440070030 2024-03-31T15:14:38.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:38.853+0000 7fa43fe00640 1 -- 172.21.15.16:0/2983340816 <== mon.2 v2:172.21.15.89:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4380a5c50 con 0x7fa440070030 2024-03-31T15:14:38.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:38.853+0000 7fa448200640 1 -- 172.21.15.16:0/2983340816 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fa440070030 msgr2=0x7fa440162180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:38.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:38.853+0000 7fa448200640 1 --2- 172.21.15.16:0/2983340816 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fa440070030 0x7fa440162180 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fa438009d90 tx=0x7fa43801c8b0 comp rx=0 tx=0).stop 2024-03-31T15:14:38.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:38.853+0000 7fa448200640 1 -- 172.21.15.16:0/2983340816 shutdown_connections 2024-03-31T15:14:38.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:38.853+0000 7fa448200640 1 --2- 172.21.15.16:0/2983340816 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fa440070030 0x7fa440162180 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:38.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:38.853+0000 7fa448200640 1 --2- 172.21.15.16:0/2983340816 >> v2:172.21.15.89:3300/0 conn(0x7fa44006dbc0 0x7fa44015f650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:38.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:38.853+0000 7fa448200640 1 -- 172.21.15.16:0/2983340816 >> 172.21.15.16:0/2983340816 conn(0x7fa440067330 msgr2=0x7fa440067740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:14:38.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:38.853+0000 7fa448200640 1 -- 172.21.15.16:0/2983340816 shutdown_connections 2024-03-31T15:14:38.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:38.853+0000 7fa448200640 1 -- 172.21.15.16:0/2983340816 wait complete. 2024-03-31T15:14:38.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:38.857+0000 7fa448200640 1 Processor -- start 2024-03-31T15:14:38.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:38.857+0000 7fa448200640 1 -- start start 2024-03-31T15:14:38.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:38.857+0000 7fa448200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa44006dbc0 0x7fa4401580b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:38.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:38.857+0000 7fa448200640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fa440070030 0x7fa4400d0ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:38.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:38.857+0000 7fa448200640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa4400cb440 0x7fa4400d1020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:38.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:38.857+0000 7fa448200640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fa44006cad0 con 0x7fa44006dbc0 2024-03-31T15:14:38.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:38.857+0000 7fa448200640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fa44006c950 con 0x7fa4400cb440 2024-03-31T15:14:38.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:38.857+0000 7fa448200640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fa44016b520 con 0x7fa440070030 2024-03-31T15:14:38.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:38.857+0000 7fa445800640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa44006dbc0 0x7fa4401580b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:38.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:38.857+0000 7fa445800640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa44006dbc0 0x7fa4401580b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40654/0 (socket says 172.21.15.16:40654) 2024-03-31T15:14:38.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:38.857+0000 7fa445800640 1 -- 172.21.15.16:0/3009887717 learned_addr learned my addr 172.21.15.16:0/3009887717 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:14:38.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:38.857+0000 7fa445800640 1 -- 172.21.15.16:0/3009887717 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fa440070030 msgr2=0x7fa4400d0ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:38.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:38.857+0000 7fa445800640 1 --2- 172.21.15.16:0/3009887717 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fa440070030 0x7fa4400d0ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:38.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:38.857+0000 7fa445800640 1 -- 172.21.15.16:0/3009887717 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa4400cb440 msgr2=0x7fa4400d1020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:38.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:38.857+0000 7fa445800640 1 --2- 172.21.15.16:0/3009887717 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa4400cb440 0x7fa4400d1020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:38.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:38.857+0000 7fa445800640 1 -- 172.21.15.16:0/3009887717 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa440143fa0 con 0x7fa44006dbc0 2024-03-31T15:14:38.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:38.857+0000 7fa445800640 1 --2- 172.21.15.16:0/3009887717 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa44006dbc0 0x7fa4401580b0 secure :-1 s=READY pgs=1732 cs=0 l=1 rev1=1 crypto rx=0x7fa43400e980 tx=0x7fa43400ee40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:38.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:38.857+0000 7fa43d600640 1 -- 172.21.15.16:0/3009887717 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa434017070 con 0x7fa44006dbc0 2024-03-31T15:14:38.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:38.857+0000 7fa43d600640 1 -- 172.21.15.16:0/3009887717 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa43400ca00 con 0x7fa44006dbc0 2024-03-31T15:14:38.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:38.857+0000 7fa43d600640 1 -- 172.21.15.16:0/3009887717 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa43400cdd0 con 0x7fa44006dbc0 2024-03-31T15:14:38.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:38.857+0000 7fa448200640 1 -- 172.21.15.16:0/3009887717 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa440144280 con 0x7fa44006dbc0 2024-03-31T15:14:38.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:38.857+0000 7fa448200640 1 -- 172.21.15.16:0/3009887717 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa440144840 con 0x7fa44006dbc0 2024-03-31T15:14:38.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:38.857+0000 7fa448200640 1 -- 172.21.15.16:0/3009887717 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa440144c80 con 0x7fa44006dbc0 2024-03-31T15:14:38.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:38.861+0000 7fa43d600640 1 -- 172.21.15.16:0/3009887717 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fa43401a070 con 0x7fa44006dbc0 2024-03-31T15:14:38.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:38.861+0000 7fa43d600640 1 --2- 172.21.15.16:0/3009887717 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fa420068380 0x7fa42006a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:38.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:38.861+0000 7fa43d600640 1 -- 172.21.15.16:0/3009887717 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 5767+0+0 (secure 0 0 0) 0x7fa434086be0 con 0x7fa44006dbc0 2024-03-31T15:14:38.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:38.861+0000 7fa444e00640 1 --2- 172.21.15.16:0/3009887717 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fa420068380 0x7fa42006a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:38.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:38.861+0000 7fa444e00640 1 --2- 172.21.15.16:0/3009887717 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fa420068380 0x7fa42006a840 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7fa42c007fd0 tx=0x7fa42c007900 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:38.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:38.865+0000 7fa43d600640 1 -- 172.21.15.16:0/3009887717 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fa4340571f0 con 0x7fa44006dbc0 2024-03-31T15:14:39.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:39.045+0000 7fa448200640 1 -- 172.21.15.16:0/3009887717 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fa4400cf380 con 0x7fa44006dbc0 2024-03-31T15:14:39.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:39.045+0000 7fa43d600640 1 -- 172.21.15.16:0/3009887717 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v111) ==== 80+0+0 (secure 0 0 0) 0x7fa434056af0 con 0x7fa44006dbc0 2024-03-31T15:14:39.038 INFO:teuthology.orchestra.run.smithi016.stderr:listed 0 entries 2024-03-31T15:14:39.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:39.045+0000 7fa448200640 1 -- 172.21.15.16:0/3009887717 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fa420068380 msgr2=0x7fa42006a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:39.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:39.045+0000 7fa448200640 1 --2- 172.21.15.16:0/3009887717 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fa420068380 0x7fa42006a840 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7fa42c007fd0 tx=0x7fa42c007900 comp rx=0 tx=0).stop 2024-03-31T15:14:39.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:39.045+0000 7fa448200640 1 -- 172.21.15.16:0/3009887717 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa44006dbc0 msgr2=0x7fa4401580b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:39.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:39.045+0000 7fa448200640 1 --2- 172.21.15.16:0/3009887717 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa44006dbc0 0x7fa4401580b0 secure :-1 s=READY pgs=1732 cs=0 l=1 rev1=1 crypto rx=0x7fa43400e980 tx=0x7fa43400ee40 comp rx=0 tx=0).stop 2024-03-31T15:14:39.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:39.045+0000 7fa448200640 1 -- 172.21.15.16:0/3009887717 shutdown_connections 2024-03-31T15:14:39.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:39.045+0000 7fa448200640 1 --2- 172.21.15.16:0/3009887717 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fa420068380 0x7fa42006a840 unknown :-1 s=CLOSED pgs=735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:39.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:39.045+0000 7fa448200640 1 --2- 172.21.15.16:0/3009887717 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa4400cb440 0x7fa4400d1020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:39.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:39.045+0000 7fa448200640 1 --2- 172.21.15.16:0/3009887717 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fa440070030 0x7fa4400d0ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:39.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:39.045+0000 7fa448200640 1 --2- 172.21.15.16:0/3009887717 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa44006dbc0 0x7fa4401580b0 unknown :-1 s=CLOSED pgs=1732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:39.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:39.045+0000 7fa448200640 1 -- 172.21.15.16:0/3009887717 >> 172.21.15.16:0/3009887717 conn(0x7fa440067330 msgr2=0x7fa44015c7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:14:39.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:39.045+0000 7fa448200640 1 -- 172.21.15.16:0/3009887717 shutdown_connections 2024-03-31T15:14:39.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:39.045+0000 7fa448200640 1 -- 172.21.15.16:0/3009887717 wait complete. 2024-03-31T15:14:39.059 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi016.front.sepia.ceph.com... 2024-03-31T15:14:39.060 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:14:39.060 DEBUG:teuthology.orchestra.run.smithi016:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-03-31T15:14:39.086 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi016.stderr:ceph-fuse[51624]: fuse finished with error 0 and tester_r 0 2024-03-31T15:14:39.091 INFO:teuthology.orchestra.run:waiting for 300 2024-03-31T15:14:39.169 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi016.front.sepia.ceph.com 2024-03-31T15:14:39.170 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:14:39.170 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-31T15:14:39.179 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-03-31T15:14:39.179 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-31T15:14:39.232 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16357708k,nr_inodes=4089427,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275996k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=6539 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275992k,nr_inodes=818998,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275996k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-31T15:14:39.232 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse client.0 is not mounted at ubuntu@smithi016.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.0 2024-03-31T15:14:39.232 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi016.front.sepia.ceph.com 2024-03-31T15:14:39.232 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:14:39.233 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-31T15:14:39.280 INFO:teuthology.orchestra.run.smithi016.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-03-31T15:14:39.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-31T15:14:39.281 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-31T15:14:39.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:39.469+0000 7fd4f2800640 1 Processor -- start 2024-03-31T15:14:39.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:39.469+0000 7fd4f2800640 1 -- start start 2024-03-31T15:14:39.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:39.473+0000 7fd4f2800640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fd4ec1629f0 0x7fd4ec0b3810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:39.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:39.473+0000 7fd4f2800640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fd4ec06dc20 0x7fd4ec0b3d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:39.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:39.473+0000 7fd4f2800640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd4ec15e9b0 con 0x7fd4ec0673e0 2024-03-31T15:14:39.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:39.473+0000 7fd4f2800640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7fd4ec15ec70 con 0x7fd4ec06dc20 2024-03-31T15:14:39.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:39.473+0000 7fd4f2800640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fd4ec15ef50 con 0x7fd4ec1629f0 2024-03-31T15:14:39.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:39.473+0000 7fd4ebe00640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fd4ec0673e0 0x7fd4ec06e0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:33454/0 (socket says 172.21.15.16:33454) 2024-03-31T15:14:39.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:39.473+0000 7fd4ebe00640 1 -- 172.21.15.16:0/81399616 learned_addr learned my addr 172.21.15.16:0/81399616 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:14:39.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:39.473+0000 7fd4e2a00640 1 --2- 172.21.15.16:0/81399616 >> v2:172.21.15.89:3300/0 conn(0x7fd4ec06dc20 0x7fd4ec0b3d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:39.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:39.473+0000 7fd4eb400640 1 --2- 172.21.15.16:0/81399616 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fd4ec1629f0 0x7fd4ec0b3810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:39.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:39.473+0000 7fd4eaa00640 1 -- 172.21.15.16:0/81399616 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3393127804 0 0) 0x7fd4ec15e9b0 con 0x7fd4ec0673e0 2024-03-31T15:14:39.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:39.473+0000 7fd4eaa00640 1 -- 172.21.15.16:0/81399616 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd4d0003530 con 0x7fd4ec0673e0 2024-03-31T15:14:39.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:39.473+0000 7fd4e2a00640 1 -- 172.21.15.16:0/81399616 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fd4ec1629f0 msgr2=0x7fd4ec0b3810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:14:39.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:39.473+0000 7fd4eaa00640 1 -- 172.21.15.16:0/81399616 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 434941982 0 0) 0x7fd4d0003530 con 0x7fd4ec0673e0 2024-03-31T15:14:39.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:39.473+0000 7fd4e2a00640 1 --2- 172.21.15.16:0/81399616 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fd4ec1629f0 0x7fd4ec0b3810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:39.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:39.473+0000 7fd4e2a00640 1 -- 172.21.15.16:0/81399616 >> v1:172.21.15.16:6789/0 conn(0x7fd4ec0673e0 legacy=0x7fd4ec06e0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:39.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:39.473+0000 7fd4e2a00640 1 -- 172.21.15.16:0/81399616 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd4ec15f230 con 0x7fd4ec06dc20 2024-03-31T15:14:39.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:39.473+0000 7fd4eb400640 1 --2- 172.21.15.16:0/81399616 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fd4ec1629f0 0x7fd4ec0b3810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:14:39.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:39.477+0000 7fd4e2a00640 1 --2- 172.21.15.16:0/81399616 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd4ec06dc20 0x7fd4ec0b3d50 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7fd4d4009d90 tx=0x7fd4d401c890 comp rx=0 tx=0).ready entity=mon.1 client_cookie=492a72efdf175b06 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:39.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:39.477+0000 7fd4eaa00640 1 -- 172.21.15.16:0/81399616 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4d40b2070 con 0x7fd4ec06dc20 2024-03-31T15:14:39.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:39.477+0000 7fd4eaa00640 1 -- 172.21.15.16:0/81399616 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd4d40a55a0 con 0x7fd4ec06dc20 2024-03-31T15:14:39.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:39.477+0000 7fd4eaa00640 1 -- 172.21.15.16:0/81399616 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4e4001680 con 0x7fd4ec06dc20 2024-03-31T15:14:39.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:39.477+0000 7fd4f2800640 1 -- 172.21.15.16:0/81399616 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd4ec06dc20 msgr2=0x7fd4ec0b3d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:39.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:39.477+0000 7fd4f2800640 1 --2- 172.21.15.16:0/81399616 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd4ec06dc20 0x7fd4ec0b3d50 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7fd4d4009d90 tx=0x7fd4d401c890 comp rx=0 tx=0).stop 2024-03-31T15:14:39.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:39.477+0000 7fd4f2800640 1 -- 172.21.15.16:0/81399616 shutdown_connections 2024-03-31T15:14:39.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:39.477+0000 7fd4f2800640 1 --2- 172.21.15.16:0/81399616 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd4ec06dc20 0x7fd4ec0b3d50 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:39.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:39.477+0000 7fd4f2800640 1 --2- 172.21.15.16:0/81399616 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fd4ec1629f0 0x7fd4ec0b3810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:39.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:39.477+0000 7fd4f2800640 1 -- 172.21.15.16:0/81399616 >> 172.21.15.16:0/81399616 conn(0x7fd4ec0d0760 msgr2=0x7fd4ec0d0b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:14:39.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:39.477+0000 7fd4f2800640 1 -- 172.21.15.16:0/81399616 shutdown_connections 2024-03-31T15:14:39.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:39.477+0000 7fd4f2800640 1 -- 172.21.15.16:0/81399616 wait complete. 2024-03-31T15:14:39.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:39.477+0000 7fd4f2800640 1 Processor -- start 2024-03-31T15:14:39.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:39.477+0000 7fd4f2800640 1 -- start start 2024-03-31T15:14:39.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:39.481+0000 7fd4f2800640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd4ec0673e0 0x7fd4ec1533d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:39.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:39.481+0000 7fd4f2800640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fd4ec06dc20 0x7fd4ec154d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:39.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:39.481+0000 7fd4f2800640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd4ec1629f0 0x7fd4ec15b3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:39.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:39.481+0000 7fd4f2800640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fd4ec16b4c0 con 0x7fd4ec0673e0 2024-03-31T15:14:39.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:39.481+0000 7fd4f2800640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fd4ec16b340 con 0x7fd4ec1629f0 2024-03-31T15:14:39.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:39.481+0000 7fd4f2800640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fd4ec16b640 con 0x7fd4ec06dc20 2024-03-31T15:14:39.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:39.481+0000 7fd4eb400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd4ec0673e0 0x7fd4ec1533d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:39.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:39.481+0000 7fd4eb400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd4ec0673e0 0x7fd4ec1533d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40662/0 (socket says 172.21.15.16:40662) 2024-03-31T15:14:39.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:39.481+0000 7fd4eb400640 1 -- 172.21.15.16:0/587946846 learned_addr learned my addr 172.21.15.16:0/587946846 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:14:39.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:39.481+0000 7fd4eb400640 1 -- 172.21.15.16:0/587946846 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fd4ec06dc20 msgr2=0x7fd4ec154d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:39.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:39.481+0000 7fd4eb400640 1 --2- 172.21.15.16:0/587946846 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fd4ec06dc20 0x7fd4ec154d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:39.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:39.481+0000 7fd4eb400640 1 -- 172.21.15.16:0/587946846 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd4ec1629f0 msgr2=0x7fd4ec15b3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:39.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:39.481+0000 7fd4eb400640 1 --2- 172.21.15.16:0/587946846 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd4ec1629f0 0x7fd4ec15b3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:39.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:39.481+0000 7fd4eb400640 1 -- 172.21.15.16:0/587946846 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd4ec159c40 con 0x7fd4ec0673e0 2024-03-31T15:14:39.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:39.481+0000 7fd4eb400640 1 --2- 172.21.15.16:0/587946846 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd4ec0673e0 0x7fd4ec1533d0 secure :-1 s=READY pgs=1734 cs=0 l=1 rev1=1 crypto rx=0x7fd4dc00ebe0 tx=0x7fd4dc00c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:39.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:39.481+0000 7fd4e3e00640 1 -- 172.21.15.16:0/587946846 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4dc017070 con 0x7fd4ec0673e0 2024-03-31T15:14:39.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:39.481+0000 7fd4e3e00640 1 -- 172.21.15.16:0/587946846 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd4dc00cc30 con 0x7fd4ec0673e0 2024-03-31T15:14:39.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:39.481+0000 7fd4e3e00640 1 -- 172.21.15.16:0/587946846 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4dc013470 con 0x7fd4ec0673e0 2024-03-31T15:14:39.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:39.481+0000 7fd4f2800640 1 -- 172.21.15.16:0/587946846 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd4ec159f20 con 0x7fd4ec0673e0 2024-03-31T15:14:39.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:39.481+0000 7fd4f2800640 1 -- 172.21.15.16:0/587946846 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd4ec15a4e0 con 0x7fd4ec0673e0 2024-03-31T15:14:39.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:39.481+0000 7fd4f2800640 1 -- 172.21.15.16:0/587946846 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd4ec069950 con 0x7fd4ec0673e0 2024-03-31T15:14:39.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:39.481+0000 7fd4e3e00640 1 -- 172.21.15.16:0/587946846 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fd4dc01a070 con 0x7fd4ec0673e0 2024-03-31T15:14:39.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:39.485+0000 7fd4e3e00640 1 --2- 172.21.15.16:0/587946846 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fd4c8068380 0x7fd4c806a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:39.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:39.485+0000 7fd4e3e00640 1 -- 172.21.15.16:0/587946846 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 5767+0+0 (secure 0 0 0) 0x7fd4dc086990 con 0x7fd4ec0673e0 2024-03-31T15:14:39.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:39.485+0000 7fd4e2a00640 1 --2- 172.21.15.16:0/587946846 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fd4c8068380 0x7fd4c806a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:39.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:39.485+0000 7fd4e2a00640 1 --2- 172.21.15.16:0/587946846 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fd4c8068380 0x7fd4c806a840 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7fd4d40af000 tx=0x7fd4d4004ac0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:39.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:39.489+0000 7fd4e3e00640 1 -- 172.21.15.16:0/587946846 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fd4dc056f00 con 0x7fd4ec0673e0 2024-03-31T15:14:39.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:39.709+0000 7fd4f2800640 1 -- 172.21.15.16:0/587946846 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7fd4ec15d3d0 con 0x7fd4ec0673e0 2024-03-31T15:14:39.703 INFO:tasks.ceph.mon.a.smithi016.stderr:2024-03-31T15:14:39.709+0000 7f7dcbc00640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-31T15:14:39.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:39.713+0000 7fd4e3e00640 1 -- 172.21.15.16:0/587946846 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v100) ==== 161+0+0 (secure 0 0 0) 0x7fd4dc05d0d0 con 0x7fd4ec0673e0 2024-03-31T15:14:39.705 INFO:teuthology.orchestra.run.smithi016.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-31T15:14:39.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:39.713+0000 7fd4f2800640 1 -- 172.21.15.16:0/587946846 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fd4c8068380 msgr2=0x7fd4c806a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:39.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:39.713+0000 7fd4f2800640 1 --2- 172.21.15.16:0/587946846 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fd4c8068380 0x7fd4c806a840 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7fd4d40af000 tx=0x7fd4d4004ac0 comp rx=0 tx=0).stop 2024-03-31T15:14:39.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:39.713+0000 7fd4f2800640 1 -- 172.21.15.16:0/587946846 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd4ec0673e0 msgr2=0x7fd4ec1533d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:39.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:39.713+0000 7fd4f2800640 1 --2- 172.21.15.16:0/587946846 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd4ec0673e0 0x7fd4ec1533d0 secure :-1 s=READY pgs=1734 cs=0 l=1 rev1=1 crypto rx=0x7fd4dc00ebe0 tx=0x7fd4dc00c520 comp rx=0 tx=0).stop 2024-03-31T15:14:39.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:39.713+0000 7fd4f2800640 1 -- 172.21.15.16:0/587946846 shutdown_connections 2024-03-31T15:14:39.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:39.713+0000 7fd4f2800640 1 --2- 172.21.15.16:0/587946846 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fd4c8068380 0x7fd4c806a840 unknown :-1 s=CLOSED pgs=736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:39.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:39.713+0000 7fd4f2800640 1 --2- 172.21.15.16:0/587946846 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd4ec1629f0 0x7fd4ec15b3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:39.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:39.713+0000 7fd4f2800640 1 --2- 172.21.15.16:0/587946846 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fd4ec06dc20 0x7fd4ec154d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:39.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:39.713+0000 7fd4f2800640 1 --2- 172.21.15.16:0/587946846 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd4ec0673e0 0x7fd4ec1533d0 unknown :-1 s=CLOSED pgs=1734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:39.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:39.713+0000 7fd4f2800640 1 -- 172.21.15.16:0/587946846 >> 172.21.15.16:0/587946846 conn(0x7fd4ec0d0760 msgr2=0x7fd4ec06e0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:14:39.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:39.713+0000 7fd4f2800640 1 -- 172.21.15.16:0/587946846 shutdown_connections 2024-03-31T15:14:39.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:39.713+0000 7fd4f2800640 1 -- 172.21.15.16:0/587946846 wait complete. 2024-03-31T15:14:39.720 INFO:tasks.ceph.mds.c.smithi016.stderr: -7958> 2024-03-31T15:14:14.234+0000 7f62e9600640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-31T15:14:39.730 DEBUG:teuthology.orchestra.run.smithi016:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-31T15:14:39.733 DEBUG:teuthology.orchestra.run.smithi089:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-31T15:14:39.773 INFO:tasks.ceph.mds.c.smithi016.stdout:starting mds.c at 2024-03-31T15:14:39.775 INFO:teuthology.orchestra.run.smithi016.stdout:[global] 2024-03-31T15:14:39.775 INFO:teuthology.orchestra.run.smithi016.stdout: chdir = "" 2024-03-31T15:14:39.775 INFO:teuthology.orchestra.run.smithi016.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-31T15:14:39.775 INFO:teuthology.orchestra.run.smithi016.stdout: auth supported = cephx 2024-03-31T15:14:39.775 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:14:39.775 INFO:teuthology.orchestra.run.smithi016.stdout: filestore xattr use omap = true 2024-03-31T15:14:39.775 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:14:39.776 INFO:teuthology.orchestra.run.smithi016.stdout: mon clock drift allowed = 1.000 2024-03-31T15:14:39.776 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:14:39.776 INFO:teuthology.orchestra.run.smithi016.stdout: osd crush chooseleaf type = 0 2024-03-31T15:14:39.776 INFO:teuthology.orchestra.run.smithi016.stdout: auth debug = true 2024-03-31T15:14:39.776 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:14:39.776 INFO:teuthology.orchestra.run.smithi016.stdout: ms die on old message = true 2024-03-31T15:14:39.776 INFO:teuthology.orchestra.run.smithi016.stdout: ms die on bug = true 2024-03-31T15:14:39.776 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:14:39.776 INFO:teuthology.orchestra.run.smithi016.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-31T15:14:39.776 INFO:teuthology.orchestra.run.smithi016.stdout: mon pg warn max object skew = 0 2024-03-31T15:14:39.776 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:14:39.776 INFO:teuthology.orchestra.run.smithi016.stdout: # disable pg_autoscaler by default for new pools 2024-03-31T15:14:39.776 INFO:teuthology.orchestra.run.smithi016.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-31T15:14:39.776 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:14:39.776 INFO:teuthology.orchestra.run.smithi016.stdout: osd pool default size = 2 2024-03-31T15:14:39.776 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:14:39.777 INFO:teuthology.orchestra.run.smithi016.stdout: mon osd allow primary affinity = true 2024-03-31T15:14:39.777 INFO:teuthology.orchestra.run.smithi016.stdout: mon osd allow pg remap = true 2024-03-31T15:14:39.777 INFO:teuthology.orchestra.run.smithi016.stdout: mon warn on legacy crush tunables = false 2024-03-31T15:14:39.777 INFO:teuthology.orchestra.run.smithi016.stdout: mon warn on crush straw calc version zero = false 2024-03-31T15:14:39.777 INFO:teuthology.orchestra.run.smithi016.stdout: mon warn on no sortbitwise = false 2024-03-31T15:14:39.777 INFO:teuthology.orchestra.run.smithi016.stdout: mon warn on osd down out interval zero = false 2024-03-31T15:14:39.777 INFO:teuthology.orchestra.run.smithi016.stdout: mon warn on too few osds = false 2024-03-31T15:14:39.777 INFO:teuthology.orchestra.run.smithi016.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-31T15:14:39.777 INFO:teuthology.orchestra.run.smithi016.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-31T15:14:39.777 INFO:teuthology.orchestra.run.smithi016.stdout: mon_allow_pool_size_one = true 2024-03-31T15:14:39.777 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:14:39.777 INFO:teuthology.orchestra.run.smithi016.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-31T15:14:39.777 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:14:39.777 INFO:teuthology.orchestra.run.smithi016.stdout: osd default data pool replay window = 5 2024-03-31T15:14:39.777 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:14:39.778 INFO:teuthology.orchestra.run.smithi016.stdout: mon allow pool delete = true 2024-03-31T15:14:39.778 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:14:39.778 INFO:teuthology.orchestra.run.smithi016.stdout: mon cluster log file level = debug 2024-03-31T15:14:39.778 INFO:teuthology.orchestra.run.smithi016.stdout: debug asserts on shutdown = true 2024-03-31T15:14:39.778 INFO:teuthology.orchestra.run.smithi016.stdout: mon health detail to clog = false 2024-03-31T15:14:39.778 INFO:teuthology.orchestra.run.smithi016.stdout: mon host = "172.21.15.16,172.21.15.89,[v2:172.21.15.89:3301,v1:172.21.15.89:6790]" 2024-03-31T15:14:39.778 INFO:teuthology.orchestra.run.smithi016.stdout: fsid = 286eb5da-fb47-4550-b290-d9519c13b9a5 2024-03-31T15:14:39.778 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:14:39.778 INFO:teuthology.orchestra.run.smithi016.stdout:[osd] 2024-03-31T15:14:39.778 INFO:teuthology.orchestra.run.smithi016.stdout: osd journal size = 100 2024-03-31T15:14:39.778 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:14:39.778 INFO:teuthology.orchestra.run.smithi016.stdout: osd scrub load threshold = 5.0 2024-03-31T15:14:39.778 INFO:teuthology.orchestra.run.smithi016.stdout: osd scrub max interval = 600 2024-03-31T15:14:39.778 INFO:teuthology.orchestra.run.smithi016.stdout: osd mclock profile = high_recovery_ops 2024-03-31T15:14:39.779 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:14:39.779 INFO:teuthology.orchestra.run.smithi016.stdout: osd recover clone overlap = true 2024-03-31T15:14:39.779 INFO:teuthology.orchestra.run.smithi016.stdout: osd recovery max chunk = 1048576 2024-03-31T15:14:39.779 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:14:39.779 INFO:teuthology.orchestra.run.smithi016.stdout: osd debug shutdown = true 2024-03-31T15:14:39.779 INFO:teuthology.orchestra.run.smithi016.stdout: osd debug op order = true 2024-03-31T15:14:39.779 INFO:teuthology.orchestra.run.smithi016.stdout: osd debug verify stray on activate = true 2024-03-31T15:14:39.779 INFO:teuthology.orchestra.run.smithi016.stdout: osd debug trim objects = true 2024-03-31T15:14:39.779 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:14:39.779 INFO:teuthology.orchestra.run.smithi016.stdout: osd open classes on start = true 2024-03-31T15:14:39.779 INFO:teuthology.orchestra.run.smithi016.stdout: osd debug pg log writeout = true 2024-03-31T15:14:39.779 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:14:39.779 INFO:teuthology.orchestra.run.smithi016.stdout: osd deep scrub update digest min age = 30 2024-03-31T15:14:39.779 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:14:39.779 INFO:teuthology.orchestra.run.smithi016.stdout: osd map max advance = 10 2024-03-31T15:14:39.779 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:14:39.780 INFO:teuthology.orchestra.run.smithi016.stdout: journal zero on create = true 2024-03-31T15:14:39.780 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:14:39.780 INFO:teuthology.orchestra.run.smithi016.stdout: filestore ondisk finisher threads = 3 2024-03-31T15:14:39.780 INFO:teuthology.orchestra.run.smithi016.stdout: filestore apply finisher threads = 3 2024-03-31T15:14:39.780 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:14:39.780 INFO:teuthology.orchestra.run.smithi016.stdout: bdev debug aio = true 2024-03-31T15:14:39.780 INFO:teuthology.orchestra.run.smithi016.stdout: osd debug misdirected ops = true 2024-03-31T15:14:39.780 INFO:teuthology.orchestra.run.smithi016.stdout: bdev async discard = True 2024-03-31T15:14:39.780 INFO:teuthology.orchestra.run.smithi016.stdout: bdev enable discard = True 2024-03-31T15:14:39.780 INFO:teuthology.orchestra.run.smithi016.stdout: bluestore allocator = bitmap 2024-03-31T15:14:39.780 INFO:teuthology.orchestra.run.smithi016.stdout: bluestore block size = 96636764160 2024-03-31T15:14:39.780 INFO:teuthology.orchestra.run.smithi016.stdout: bluestore fsck on mount = True 2024-03-31T15:14:39.780 INFO:teuthology.orchestra.run.smithi016.stdout: debug bluefs = 1/20 2024-03-31T15:14:39.780 INFO:teuthology.orchestra.run.smithi016.stdout: debug bluestore = 1/20 2024-03-31T15:14:39.780 INFO:teuthology.orchestra.run.smithi016.stdout: debug ms = 1 2024-03-31T15:14:39.781 INFO:teuthology.orchestra.run.smithi016.stdout: debug osd = 20 2024-03-31T15:14:39.781 INFO:teuthology.orchestra.run.smithi016.stdout: debug rocksdb = 4/10 2024-03-31T15:14:39.781 INFO:teuthology.orchestra.run.smithi016.stdout: mon osd backfillfull_ratio = 0.85 2024-03-31T15:14:39.781 INFO:teuthology.orchestra.run.smithi016.stdout: mon osd full ratio = 0.9 2024-03-31T15:14:39.781 INFO:teuthology.orchestra.run.smithi016.stdout: mon osd nearfull ratio = 0.8 2024-03-31T15:14:39.781 INFO:teuthology.orchestra.run.smithi016.stdout: osd failsafe full ratio = 0.95 2024-03-31T15:14:39.781 INFO:teuthology.orchestra.run.smithi016.stdout: osd objectstore = bluestore 2024-03-31T15:14:39.781 INFO:teuthology.orchestra.run.smithi016.stdout: osd op complaint time = 180 2024-03-31T15:14:39.781 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:14:39.781 INFO:teuthology.orchestra.run.smithi016.stdout:[mgr] 2024-03-31T15:14:39.781 INFO:teuthology.orchestra.run.smithi016.stdout: debug ms = 1 2024-03-31T15:14:39.781 INFO:teuthology.orchestra.run.smithi016.stdout: debug mgr = 20 2024-03-31T15:14:39.782 INFO:teuthology.orchestra.run.smithi016.stdout: debug mon = 20 2024-03-31T15:14:39.782 INFO:teuthology.orchestra.run.smithi016.stdout: debug auth = 20 2024-03-31T15:14:39.782 INFO:teuthology.orchestra.run.smithi016.stdout: mon reweight min pgs per osd = 4 2024-03-31T15:14:39.782 INFO:teuthology.orchestra.run.smithi016.stdout: mon reweight min bytes per osd = 10 2024-03-31T15:14:39.782 INFO:teuthology.orchestra.run.smithi016.stdout: mgr/telemetry/nag = false 2024-03-31T15:14:39.782 INFO:teuthology.orchestra.run.smithi016.stdout: debug client = 20 2024-03-31T15:14:39.782 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:14:39.782 INFO:teuthology.orchestra.run.smithi016.stdout:[mon] 2024-03-31T15:14:39.782 INFO:teuthology.orchestra.run.smithi016.stdout: debug ms = 1 2024-03-31T15:14:39.782 INFO:teuthology.orchestra.run.smithi016.stdout: debug mon = 20 2024-03-31T15:14:39.782 INFO:teuthology.orchestra.run.smithi016.stdout: debug paxos = 20 2024-03-31T15:14:39.783 INFO:teuthology.orchestra.run.smithi016.stdout: debug auth = 20 2024-03-31T15:14:39.783 INFO:teuthology.orchestra.run.smithi016.stdout: mon data avail warn = 5 2024-03-31T15:14:39.783 INFO:teuthology.orchestra.run.smithi016.stdout: mon mgr mkfs grace = 240 2024-03-31T15:14:39.783 INFO:teuthology.orchestra.run.smithi016.stdout: mon reweight min pgs per osd = 4 2024-03-31T15:14:39.783 INFO:teuthology.orchestra.run.smithi016.stdout: mon osd reporter subtree level = osd 2024-03-31T15:14:39.783 INFO:teuthology.orchestra.run.smithi016.stdout: mon osd prime pg temp = true 2024-03-31T15:14:39.783 INFO:teuthology.orchestra.run.smithi016.stdout: mon reweight min bytes per osd = 10 2024-03-31T15:14:39.783 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:14:39.783 INFO:teuthology.orchestra.run.smithi016.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-31T15:14:39.783 INFO:teuthology.orchestra.run.smithi016.stdout: auth mon ticket ttl = 660 # 11m 2024-03-31T15:14:39.783 INFO:teuthology.orchestra.run.smithi016.stdout: auth service ticket ttl = 240 # 4m 2024-03-31T15:14:39.783 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:14:39.783 INFO:teuthology.orchestra.run.smithi016.stdout: # don't complain about insecure global_id in the test suite 2024-03-31T15:14:39.783 INFO:teuthology.orchestra.run.smithi016.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-31T15:14:39.784 INFO:teuthology.orchestra.run.smithi016.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-31T15:14:39.784 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:14:39.784 INFO:teuthology.orchestra.run.smithi016.stdout: # 1m isn't quite enough 2024-03-31T15:14:39.784 INFO:teuthology.orchestra.run.smithi016.stdout: mon_down_mkfs_grace = 2m 2024-03-31T15:14:39.784 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:14:39.784 INFO:teuthology.orchestra.run.smithi016.stdout: mon_warn_on_filestore_osds = false 2024-03-31T15:14:39.784 INFO:teuthology.orchestra.run.smithi016.stdout: mon down mkfs grace = 300 2024-03-31T15:14:39.784 INFO:teuthology.orchestra.run.smithi016.stdout: mon op complaint time = 120 2024-03-31T15:14:39.784 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:14:39.784 INFO:teuthology.orchestra.run.smithi016.stdout:[client] 2024-03-31T15:14:39.784 INFO:teuthology.orchestra.run.smithi016.stdout: rgw cache enabled = true 2024-03-31T15:14:39.784 INFO:teuthology.orchestra.run.smithi016.stdout: rgw enable ops log = true 2024-03-31T15:14:39.784 INFO:teuthology.orchestra.run.smithi016.stdout: rgw enable usage log = true 2024-03-31T15:14:39.784 INFO:teuthology.orchestra.run.smithi016.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-31T15:14:39.784 INFO:teuthology.orchestra.run.smithi016.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-31T15:14:39.784 INFO:teuthology.orchestra.run.smithi016.stdout: client mount timeout = 600 2024-03-31T15:14:39.785 INFO:teuthology.orchestra.run.smithi016.stdout: debug client = 20 2024-03-31T15:14:39.785 INFO:teuthology.orchestra.run.smithi016.stdout: debug ms = 1 2024-03-31T15:14:39.785 INFO:teuthology.orchestra.run.smithi016.stdout: pid file = "" 2024-03-31T15:14:39.785 INFO:teuthology.orchestra.run.smithi016.stdout: rados mon op timeout = 900 2024-03-31T15:14:39.785 INFO:teuthology.orchestra.run.smithi016.stdout: rados osd op timeout = 900 2024-03-31T15:14:39.785 INFO:teuthology.orchestra.run.smithi016.stdout:[mon.a] 2024-03-31T15:14:39.785 INFO:teuthology.orchestra.run.smithi016.stdout:[mon.b] 2024-03-31T15:14:39.785 INFO:teuthology.orchestra.run.smithi016.stdout:[mon.c] 2024-03-31T15:14:39.785 INFO:teuthology.orchestra.run.smithi016.stdout:[mds.a] 2024-03-31T15:14:39.785 INFO:teuthology.orchestra.run.smithi016.stdout:[mds.c] 2024-03-31T15:14:39.785 INFO:teuthology.orchestra.run.smithi016.stdout:[mds.b] 2024-03-31T15:14:39.785 INFO:teuthology.orchestra.run.smithi016.stdout:[mds.d] 2024-03-31T15:14:39.785 INFO:teuthology.orchestra.run.smithi016.stdout:[mds] 2024-03-31T15:14:39.785 INFO:teuthology.orchestra.run.smithi016.stdout: debug mds = 20 2024-03-31T15:14:39.785 INFO:teuthology.orchestra.run.smithi016.stdout: debug mds balancer = 20 2024-03-31T15:14:39.785 INFO:teuthology.orchestra.run.smithi016.stdout: debug ms = 1 2024-03-31T15:14:39.786 INFO:teuthology.orchestra.run.smithi016.stdout: mds debug frag = True 2024-03-31T15:14:39.786 INFO:teuthology.orchestra.run.smithi016.stdout: mds debug scatterstat = True 2024-03-31T15:14:39.786 INFO:teuthology.orchestra.run.smithi016.stdout: mds op complaint time = 180 2024-03-31T15:14:39.786 INFO:teuthology.orchestra.run.smithi016.stdout: mds verify scatter = False 2024-03-31T15:14:39.786 INFO:teuthology.orchestra.run.smithi016.stdout: osd op complaint time = 180 2024-03-31T15:14:39.786 INFO:teuthology.orchestra.run.smithi016.stdout: rados mon op timeout = 900 2024-03-31T15:14:39.786 INFO:teuthology.orchestra.run.smithi016.stdout: rados osd op timeout = 900 2024-03-31T15:14:39.786 INFO:teuthology.orchestra.run.smithi089.stdout:[global] 2024-03-31T15:14:39.786 INFO:teuthology.orchestra.run.smithi089.stdout: chdir = "" 2024-03-31T15:14:39.787 INFO:teuthology.orchestra.run.smithi089.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-31T15:14:39.787 INFO:teuthology.orchestra.run.smithi089.stdout: auth supported = cephx 2024-03-31T15:14:39.787 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-31T15:14:39.787 INFO:teuthology.orchestra.run.smithi089.stdout: filestore xattr use omap = true 2024-03-31T15:14:39.787 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-31T15:14:39.787 INFO:teuthology.orchestra.run.smithi089.stdout: mon clock drift allowed = 1.000 2024-03-31T15:14:39.787 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-31T15:14:39.787 INFO:teuthology.orchestra.run.smithi089.stdout: osd crush chooseleaf type = 0 2024-03-31T15:14:39.787 INFO:teuthology.orchestra.run.smithi089.stdout: auth debug = true 2024-03-31T15:14:39.787 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-31T15:14:39.787 INFO:teuthology.orchestra.run.smithi089.stdout: ms die on old message = true 2024-03-31T15:14:39.787 INFO:teuthology.orchestra.run.smithi089.stdout: ms die on bug = true 2024-03-31T15:14:39.787 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-31T15:14:39.787 INFO:teuthology.orchestra.run.smithi089.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-31T15:14:39.787 INFO:teuthology.orchestra.run.smithi089.stdout: mon pg warn max object skew = 0 2024-03-31T15:14:39.787 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-31T15:14:39.788 INFO:teuthology.orchestra.run.smithi089.stdout: # disable pg_autoscaler by default for new pools 2024-03-31T15:14:39.788 INFO:teuthology.orchestra.run.smithi089.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-31T15:14:39.788 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-31T15:14:39.788 INFO:teuthology.orchestra.run.smithi089.stdout: osd pool default size = 2 2024-03-31T15:14:39.788 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-31T15:14:39.788 INFO:teuthology.orchestra.run.smithi089.stdout: mon osd allow primary affinity = true 2024-03-31T15:14:39.788 INFO:teuthology.orchestra.run.smithi089.stdout: mon osd allow pg remap = true 2024-03-31T15:14:39.788 INFO:teuthology.orchestra.run.smithi089.stdout: mon warn on legacy crush tunables = false 2024-03-31T15:14:39.788 INFO:teuthology.orchestra.run.smithi089.stdout: mon warn on crush straw calc version zero = false 2024-03-31T15:14:39.788 INFO:teuthology.orchestra.run.smithi089.stdout: mon warn on no sortbitwise = false 2024-03-31T15:14:39.788 INFO:teuthology.orchestra.run.smithi089.stdout: mon warn on osd down out interval zero = false 2024-03-31T15:14:39.788 INFO:teuthology.orchestra.run.smithi089.stdout: mon warn on too few osds = false 2024-03-31T15:14:39.788 INFO:teuthology.orchestra.run.smithi089.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-31T15:14:39.788 INFO:teuthology.orchestra.run.smithi089.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-31T15:14:39.788 INFO:teuthology.orchestra.run.smithi089.stdout: mon_allow_pool_size_one = true 2024-03-31T15:14:39.788 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-31T15:14:39.789 INFO:teuthology.orchestra.run.smithi089.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-31T15:14:39.789 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-31T15:14:39.789 INFO:teuthology.orchestra.run.smithi089.stdout: osd default data pool replay window = 5 2024-03-31T15:14:39.789 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-31T15:14:39.789 INFO:teuthology.orchestra.run.smithi089.stdout: mon allow pool delete = true 2024-03-31T15:14:39.789 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-31T15:14:39.789 INFO:teuthology.orchestra.run.smithi089.stdout: mon cluster log file level = debug 2024-03-31T15:14:39.789 INFO:teuthology.orchestra.run.smithi089.stdout: debug asserts on shutdown = true 2024-03-31T15:14:39.789 INFO:teuthology.orchestra.run.smithi089.stdout: mon health detail to clog = false 2024-03-31T15:14:39.789 INFO:teuthology.orchestra.run.smithi089.stdout: mon host = "172.21.15.16,172.21.15.89,[v2:172.21.15.89:3301,v1:172.21.15.89:6790]" 2024-03-31T15:14:39.789 INFO:teuthology.orchestra.run.smithi089.stdout: fsid = 286eb5da-fb47-4550-b290-d9519c13b9a5 2024-03-31T15:14:39.789 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-31T15:14:39.789 INFO:teuthology.orchestra.run.smithi089.stdout:[osd] 2024-03-31T15:14:39.789 INFO:teuthology.orchestra.run.smithi089.stdout: osd journal size = 100 2024-03-31T15:14:39.789 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-31T15:14:39.789 INFO:teuthology.orchestra.run.smithi089.stdout: osd scrub load threshold = 5.0 2024-03-31T15:14:39.790 INFO:teuthology.orchestra.run.smithi089.stdout: osd scrub max interval = 600 2024-03-31T15:14:39.790 INFO:teuthology.orchestra.run.smithi089.stdout: osd mclock profile = high_recovery_ops 2024-03-31T15:14:39.790 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-31T15:14:39.790 INFO:teuthology.orchestra.run.smithi089.stdout: osd recover clone overlap = true 2024-03-31T15:14:39.790 INFO:teuthology.orchestra.run.smithi089.stdout: osd recovery max chunk = 1048576 2024-03-31T15:14:39.790 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-31T15:14:39.790 INFO:teuthology.orchestra.run.smithi089.stdout: osd debug shutdown = true 2024-03-31T15:14:39.790 INFO:teuthology.orchestra.run.smithi089.stdout: osd debug op order = true 2024-03-31T15:14:39.790 INFO:teuthology.orchestra.run.smithi089.stdout: osd debug verify stray on activate = true 2024-03-31T15:14:39.790 INFO:teuthology.orchestra.run.smithi089.stdout: osd debug trim objects = true 2024-03-31T15:14:39.790 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-31T15:14:39.790 INFO:teuthology.orchestra.run.smithi089.stdout: osd open classes on start = true 2024-03-31T15:14:39.790 INFO:teuthology.orchestra.run.smithi089.stdout: osd debug pg log writeout = true 2024-03-31T15:14:39.791 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-31T15:14:39.791 INFO:teuthology.orchestra.run.smithi089.stdout: osd deep scrub update digest min age = 30 2024-03-31T15:14:39.791 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-31T15:14:39.791 INFO:teuthology.orchestra.run.smithi089.stdout: osd map max advance = 10 2024-03-31T15:14:39.791 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-31T15:14:39.791 INFO:teuthology.orchestra.run.smithi089.stdout: journal zero on create = true 2024-03-31T15:14:39.791 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-31T15:14:39.791 INFO:teuthology.orchestra.run.smithi089.stdout: filestore ondisk finisher threads = 3 2024-03-31T15:14:39.791 INFO:teuthology.orchestra.run.smithi089.stdout: filestore apply finisher threads = 3 2024-03-31T15:14:39.791 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-31T15:14:39.791 INFO:teuthology.orchestra.run.smithi089.stdout: bdev debug aio = true 2024-03-31T15:14:39.791 INFO:teuthology.orchestra.run.smithi089.stdout: osd debug misdirected ops = true 2024-03-31T15:14:39.791 INFO:teuthology.orchestra.run.smithi089.stdout: bdev async discard = True 2024-03-31T15:14:39.791 INFO:teuthology.orchestra.run.smithi089.stdout: bdev enable discard = True 2024-03-31T15:14:39.791 INFO:teuthology.orchestra.run.smithi089.stdout: bluestore allocator = bitmap 2024-03-31T15:14:39.791 INFO:teuthology.orchestra.run.smithi089.stdout: bluestore block size = 96636764160 2024-03-31T15:14:39.792 INFO:teuthology.orchestra.run.smithi089.stdout: bluestore fsck on mount = True 2024-03-31T15:14:39.792 INFO:teuthology.orchestra.run.smithi089.stdout: debug bluefs = 1/20 2024-03-31T15:14:39.792 INFO:teuthology.orchestra.run.smithi089.stdout: debug bluestore = 1/20 2024-03-31T15:14:39.792 INFO:teuthology.orchestra.run.smithi089.stdout: debug ms = 1 2024-03-31T15:14:39.792 INFO:teuthology.orchestra.run.smithi089.stdout: debug osd = 20 2024-03-31T15:14:39.792 INFO:teuthology.orchestra.run.smithi089.stdout: debug rocksdb = 4/10 2024-03-31T15:14:39.792 INFO:teuthology.orchestra.run.smithi089.stdout: mon osd backfillfull_ratio = 0.85 2024-03-31T15:14:39.792 INFO:teuthology.orchestra.run.smithi089.stdout: mon osd full ratio = 0.9 2024-03-31T15:14:39.792 INFO:teuthology.orchestra.run.smithi089.stdout: mon osd nearfull ratio = 0.8 2024-03-31T15:14:39.793 INFO:teuthology.orchestra.run.smithi089.stdout: osd failsafe full ratio = 0.95 2024-03-31T15:14:39.793 INFO:teuthology.orchestra.run.smithi089.stdout: osd objectstore = bluestore 2024-03-31T15:14:39.793 INFO:teuthology.orchestra.run.smithi089.stdout: osd op complaint time = 180 2024-03-31T15:14:39.793 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-31T15:14:39.793 INFO:teuthology.orchestra.run.smithi089.stdout:[mgr] 2024-03-31T15:14:39.793 INFO:teuthology.orchestra.run.smithi089.stdout: debug ms = 1 2024-03-31T15:14:39.793 INFO:teuthology.orchestra.run.smithi089.stdout: debug mgr = 20 2024-03-31T15:14:39.793 INFO:teuthology.orchestra.run.smithi089.stdout: debug mon = 20 2024-03-31T15:14:39.793 INFO:teuthology.orchestra.run.smithi089.stdout: debug auth = 20 2024-03-31T15:14:39.793 INFO:teuthology.orchestra.run.smithi089.stdout: mon reweight min pgs per osd = 4 2024-03-31T15:14:39.793 INFO:teuthology.orchestra.run.smithi089.stdout: mon reweight min bytes per osd = 10 2024-03-31T15:14:39.793 INFO:teuthology.orchestra.run.smithi089.stdout: mgr/telemetry/nag = false 2024-03-31T15:14:39.794 INFO:teuthology.orchestra.run.smithi089.stdout: debug client = 20 2024-03-31T15:14:39.794 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-31T15:14:39.794 INFO:teuthology.orchestra.run.smithi089.stdout:[mon] 2024-03-31T15:14:39.794 INFO:teuthology.orchestra.run.smithi089.stdout: debug ms = 1 2024-03-31T15:14:39.794 INFO:teuthology.orchestra.run.smithi089.stdout: debug mon = 20 2024-03-31T15:14:39.794 INFO:teuthology.orchestra.run.smithi089.stdout: debug paxos = 20 2024-03-31T15:14:39.794 INFO:teuthology.orchestra.run.smithi089.stdout: debug auth = 20 2024-03-31T15:14:39.794 INFO:teuthology.orchestra.run.smithi089.stdout: mon data avail warn = 5 2024-03-31T15:14:39.794 INFO:teuthology.orchestra.run.smithi089.stdout: mon mgr mkfs grace = 240 2024-03-31T15:14:39.794 INFO:teuthology.orchestra.run.smithi089.stdout: mon reweight min pgs per osd = 4 2024-03-31T15:14:39.794 INFO:teuthology.orchestra.run.smithi089.stdout: mon osd reporter subtree level = osd 2024-03-31T15:14:39.794 INFO:teuthology.orchestra.run.smithi089.stdout: mon osd prime pg temp = true 2024-03-31T15:14:39.794 INFO:teuthology.orchestra.run.smithi089.stdout: mon reweight min bytes per osd = 10 2024-03-31T15:14:39.795 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-31T15:14:39.795 INFO:teuthology.orchestra.run.smithi089.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-31T15:14:39.795 INFO:teuthology.orchestra.run.smithi089.stdout: auth mon ticket ttl = 660 # 11m 2024-03-31T15:14:39.795 INFO:teuthology.orchestra.run.smithi089.stdout: auth service ticket ttl = 240 # 4m 2024-03-31T15:14:39.795 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-31T15:14:39.795 INFO:teuthology.orchestra.run.smithi089.stdout: # don't complain about insecure global_id in the test suite 2024-03-31T15:14:39.795 INFO:teuthology.orchestra.run.smithi089.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-31T15:14:39.795 INFO:teuthology.orchestra.run.smithi089.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-31T15:14:39.795 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-31T15:14:39.795 INFO:teuthology.orchestra.run.smithi089.stdout: # 1m isn't quite enough 2024-03-31T15:14:39.795 INFO:teuthology.orchestra.run.smithi089.stdout: mon_down_mkfs_grace = 2m 2024-03-31T15:14:39.795 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-31T15:14:39.795 INFO:teuthology.orchestra.run.smithi089.stdout: mon_warn_on_filestore_osds = false 2024-03-31T15:14:39.795 INFO:teuthology.orchestra.run.smithi089.stdout: mon down mkfs grace = 300 2024-03-31T15:14:39.796 INFO:teuthology.orchestra.run.smithi089.stdout: mon op complaint time = 120 2024-03-31T15:14:39.796 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-31T15:14:39.796 INFO:teuthology.orchestra.run.smithi089.stdout:[client] 2024-03-31T15:14:39.796 INFO:teuthology.orchestra.run.smithi089.stdout: rgw cache enabled = true 2024-03-31T15:14:39.796 INFO:teuthology.orchestra.run.smithi089.stdout: rgw enable ops log = true 2024-03-31T15:14:39.796 INFO:teuthology.orchestra.run.smithi089.stdout: rgw enable usage log = true 2024-03-31T15:14:39.796 INFO:teuthology.orchestra.run.smithi089.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-31T15:14:39.796 INFO:teuthology.orchestra.run.smithi089.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-31T15:14:39.796 INFO:teuthology.orchestra.run.smithi089.stdout: client mount timeout = 600 2024-03-31T15:14:39.796 INFO:teuthology.orchestra.run.smithi089.stdout: debug client = 20 2024-03-31T15:14:39.796 INFO:teuthology.orchestra.run.smithi089.stdout: debug ms = 1 2024-03-31T15:14:39.796 INFO:teuthology.orchestra.run.smithi089.stdout: pid file = "" 2024-03-31T15:14:39.796 INFO:teuthology.orchestra.run.smithi089.stdout: rados mon op timeout = 900 2024-03-31T15:14:39.796 INFO:teuthology.orchestra.run.smithi089.stdout: rados osd op timeout = 900 2024-03-31T15:14:39.796 INFO:teuthology.orchestra.run.smithi089.stdout:[mon.a] 2024-03-31T15:14:39.797 INFO:teuthology.orchestra.run.smithi089.stdout:[mon.b] 2024-03-31T15:14:39.797 INFO:teuthology.orchestra.run.smithi089.stdout:[mon.c] 2024-03-31T15:14:39.797 INFO:teuthology.orchestra.run.smithi089.stdout:[mds.a] 2024-03-31T15:14:39.797 INFO:teuthology.orchestra.run.smithi089.stdout:[mds.c] 2024-03-31T15:14:39.797 INFO:teuthology.orchestra.run.smithi089.stdout:[mds.b] 2024-03-31T15:14:39.797 INFO:teuthology.orchestra.run.smithi089.stdout:[mds.d] 2024-03-31T15:14:39.797 INFO:teuthology.orchestra.run.smithi089.stdout:[mds] 2024-03-31T15:14:39.797 INFO:teuthology.orchestra.run.smithi089.stdout: debug mds = 20 2024-03-31T15:14:39.797 INFO:teuthology.orchestra.run.smithi089.stdout: debug mds balancer = 20 2024-03-31T15:14:39.797 INFO:teuthology.orchestra.run.smithi089.stdout: debug ms = 1 2024-03-31T15:14:39.797 INFO:teuthology.orchestra.run.smithi089.stdout: mds debug frag = True 2024-03-31T15:14:39.797 INFO:teuthology.orchestra.run.smithi089.stdout: mds debug scatterstat = True 2024-03-31T15:14:39.797 INFO:teuthology.orchestra.run.smithi089.stdout: mds op complaint time = 180 2024-03-31T15:14:39.797 INFO:teuthology.orchestra.run.smithi089.stdout: mds verify scatter = False 2024-03-31T15:14:39.798 INFO:teuthology.orchestra.run.smithi089.stdout: osd op complaint time = 180 2024-03-31T15:14:39.798 INFO:teuthology.orchestra.run.smithi089.stdout: rados mon op timeout = 900 2024-03-31T15:14:39.798 INFO:teuthology.orchestra.run.smithi089.stdout: rados osd op timeout = 900 2024-03-31T15:14:39.800 DEBUG:teuthology.orchestra.run.smithi016:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-31T15:14:39.843 DEBUG:teuthology.orchestra.run.smithi089:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-31T15:14:39.886 INFO:teuthology.orchestra.run.smithi016.stdout:[global] 2024-03-31T15:14:39.886 INFO:teuthology.orchestra.run.smithi016.stdout: chdir = "" 2024-03-31T15:14:39.886 INFO:teuthology.orchestra.run.smithi016.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-31T15:14:39.886 INFO:teuthology.orchestra.run.smithi016.stdout: auth supported = cephx 2024-03-31T15:14:39.886 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:14:39.886 INFO:teuthology.orchestra.run.smithi016.stdout: filestore xattr use omap = true 2024-03-31T15:14:39.887 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:14:39.887 INFO:teuthology.orchestra.run.smithi016.stdout: mon clock drift allowed = 1.000 2024-03-31T15:14:39.887 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:14:39.887 INFO:teuthology.orchestra.run.smithi016.stdout: osd crush chooseleaf type = 0 2024-03-31T15:14:39.887 INFO:teuthology.orchestra.run.smithi016.stdout: auth debug = true 2024-03-31T15:14:39.887 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:14:39.887 INFO:teuthology.orchestra.run.smithi016.stdout: ms die on old message = true 2024-03-31T15:14:39.887 INFO:teuthology.orchestra.run.smithi016.stdout: ms die on bug = true 2024-03-31T15:14:39.887 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:14:39.887 INFO:teuthology.orchestra.run.smithi016.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-31T15:14:39.887 INFO:teuthology.orchestra.run.smithi016.stdout: mon pg warn max object skew = 0 2024-03-31T15:14:39.887 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:14:39.887 INFO:teuthology.orchestra.run.smithi016.stdout: # disable pg_autoscaler by default for new pools 2024-03-31T15:14:39.887 INFO:teuthology.orchestra.run.smithi016.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-31T15:14:39.887 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:14:39.888 INFO:teuthology.orchestra.run.smithi016.stdout: osd pool default size = 2 2024-03-31T15:14:39.888 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:14:39.888 INFO:teuthology.orchestra.run.smithi016.stdout: mon osd allow primary affinity = true 2024-03-31T15:14:39.888 INFO:teuthology.orchestra.run.smithi016.stdout: mon osd allow pg remap = true 2024-03-31T15:14:39.888 INFO:teuthology.orchestra.run.smithi016.stdout: mon warn on legacy crush tunables = false 2024-03-31T15:14:39.888 INFO:teuthology.orchestra.run.smithi016.stdout: mon warn on crush straw calc version zero = false 2024-03-31T15:14:39.888 INFO:teuthology.orchestra.run.smithi016.stdout: mon warn on no sortbitwise = false 2024-03-31T15:14:39.888 INFO:teuthology.orchestra.run.smithi016.stdout: mon warn on osd down out interval zero = false 2024-03-31T15:14:39.888 INFO:teuthology.orchestra.run.smithi016.stdout: mon warn on too few osds = false 2024-03-31T15:14:39.888 INFO:teuthology.orchestra.run.smithi016.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-31T15:14:39.888 INFO:teuthology.orchestra.run.smithi016.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-31T15:14:39.888 INFO:teuthology.orchestra.run.smithi016.stdout: mon_allow_pool_size_one = true 2024-03-31T15:14:39.888 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:14:39.889 INFO:teuthology.orchestra.run.smithi016.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-31T15:14:39.889 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:14:39.889 INFO:teuthology.orchestra.run.smithi016.stdout: osd default data pool replay window = 5 2024-03-31T15:14:39.889 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:14:39.889 INFO:teuthology.orchestra.run.smithi016.stdout: mon allow pool delete = true 2024-03-31T15:14:39.889 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:14:39.889 INFO:teuthology.orchestra.run.smithi016.stdout: mon cluster log file level = debug 2024-03-31T15:14:39.889 INFO:teuthology.orchestra.run.smithi016.stdout: debug asserts on shutdown = true 2024-03-31T15:14:39.889 INFO:teuthology.orchestra.run.smithi016.stdout: mon health detail to clog = false 2024-03-31T15:14:39.889 INFO:teuthology.orchestra.run.smithi016.stdout: mon host = "172.21.15.16,172.21.15.89,[v2:172.21.15.89:3301,v1:172.21.15.89:6790]" 2024-03-31T15:14:39.889 INFO:teuthology.orchestra.run.smithi016.stdout: fsid = 286eb5da-fb47-4550-b290-d9519c13b9a5 2024-03-31T15:14:39.889 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:14:39.889 INFO:teuthology.orchestra.run.smithi016.stdout:[osd] 2024-03-31T15:14:39.889 INFO:teuthology.orchestra.run.smithi016.stdout: osd journal size = 100 2024-03-31T15:14:39.890 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:14:39.890 INFO:teuthology.orchestra.run.smithi016.stdout: osd scrub load threshold = 5.0 2024-03-31T15:14:39.890 INFO:teuthology.orchestra.run.smithi016.stdout: osd scrub max interval = 600 2024-03-31T15:14:39.890 INFO:teuthology.orchestra.run.smithi016.stdout: osd mclock profile = high_recovery_ops 2024-03-31T15:14:39.890 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:14:39.890 INFO:teuthology.orchestra.run.smithi016.stdout: osd recover clone overlap = true 2024-03-31T15:14:39.890 INFO:teuthology.orchestra.run.smithi016.stdout: osd recovery max chunk = 1048576 2024-03-31T15:14:39.890 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:14:39.890 INFO:teuthology.orchestra.run.smithi016.stdout: osd debug shutdown = true 2024-03-31T15:14:39.890 INFO:teuthology.orchestra.run.smithi016.stdout: osd debug op order = true 2024-03-31T15:14:39.890 INFO:teuthology.orchestra.run.smithi016.stdout: osd debug verify stray on activate = true 2024-03-31T15:14:39.890 INFO:teuthology.orchestra.run.smithi016.stdout: osd debug trim objects = true 2024-03-31T15:14:39.890 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:14:39.890 INFO:teuthology.orchestra.run.smithi016.stdout: osd open classes on start = true 2024-03-31T15:14:39.891 INFO:teuthology.orchestra.run.smithi016.stdout: osd debug pg log writeout = true 2024-03-31T15:14:39.891 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:14:39.891 INFO:teuthology.orchestra.run.smithi016.stdout: osd deep scrub update digest min age = 30 2024-03-31T15:14:39.891 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:14:39.891 INFO:teuthology.orchestra.run.smithi016.stdout: osd map max advance = 10 2024-03-31T15:14:39.891 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:14:39.891 INFO:teuthology.orchestra.run.smithi016.stdout: journal zero on create = true 2024-03-31T15:14:39.891 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:14:39.891 INFO:teuthology.orchestra.run.smithi016.stdout: filestore ondisk finisher threads = 3 2024-03-31T15:14:39.891 INFO:teuthology.orchestra.run.smithi016.stdout: filestore apply finisher threads = 3 2024-03-31T15:14:39.891 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:14:39.891 INFO:teuthology.orchestra.run.smithi016.stdout: bdev debug aio = true 2024-03-31T15:14:39.891 INFO:teuthology.orchestra.run.smithi016.stdout: osd debug misdirected ops = true 2024-03-31T15:14:39.891 INFO:teuthology.orchestra.run.smithi016.stdout: bdev async discard = True 2024-03-31T15:14:39.891 INFO:teuthology.orchestra.run.smithi016.stdout: bdev enable discard = True 2024-03-31T15:14:39.892 INFO:teuthology.orchestra.run.smithi016.stdout: bluestore allocator = bitmap 2024-03-31T15:14:39.892 INFO:teuthology.orchestra.run.smithi016.stdout: bluestore block size = 96636764160 2024-03-31T15:14:39.892 INFO:teuthology.orchestra.run.smithi016.stdout: bluestore fsck on mount = True 2024-03-31T15:14:39.892 INFO:teuthology.orchestra.run.smithi016.stdout: debug bluefs = 1/20 2024-03-31T15:14:39.892 INFO:teuthology.orchestra.run.smithi016.stdout: debug bluestore = 1/20 2024-03-31T15:14:39.892 INFO:teuthology.orchestra.run.smithi016.stdout: debug ms = 1 2024-03-31T15:14:39.892 INFO:teuthology.orchestra.run.smithi016.stdout: debug osd = 20 2024-03-31T15:14:39.892 INFO:teuthology.orchestra.run.smithi016.stdout: debug rocksdb = 4/10 2024-03-31T15:14:39.892 INFO:teuthology.orchestra.run.smithi016.stdout: mon osd backfillfull_ratio = 0.85 2024-03-31T15:14:39.892 INFO:teuthology.orchestra.run.smithi016.stdout: mon osd full ratio = 0.9 2024-03-31T15:14:39.892 INFO:teuthology.orchestra.run.smithi016.stdout: mon osd nearfull ratio = 0.8 2024-03-31T15:14:39.892 INFO:teuthology.orchestra.run.smithi016.stdout: osd failsafe full ratio = 0.95 2024-03-31T15:14:39.892 INFO:teuthology.orchestra.run.smithi016.stdout: osd objectstore = bluestore 2024-03-31T15:14:39.892 INFO:teuthology.orchestra.run.smithi016.stdout: osd op complaint time = 180 2024-03-31T15:14:39.893 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:14:39.893 INFO:teuthology.orchestra.run.smithi016.stdout:[mgr] 2024-03-31T15:14:39.893 INFO:teuthology.orchestra.run.smithi016.stdout: debug ms = 1 2024-03-31T15:14:39.893 INFO:teuthology.orchestra.run.smithi016.stdout: debug mgr = 20 2024-03-31T15:14:39.893 INFO:teuthology.orchestra.run.smithi016.stdout: debug mon = 20 2024-03-31T15:14:39.893 INFO:teuthology.orchestra.run.smithi016.stdout: debug auth = 20 2024-03-31T15:14:39.893 INFO:teuthology.orchestra.run.smithi016.stdout: mon reweight min pgs per osd = 4 2024-03-31T15:14:39.893 INFO:teuthology.orchestra.run.smithi016.stdout: mon reweight min bytes per osd = 10 2024-03-31T15:14:39.893 INFO:teuthology.orchestra.run.smithi016.stdout: mgr/telemetry/nag = false 2024-03-31T15:14:39.893 INFO:teuthology.orchestra.run.smithi016.stdout: debug client = 20 2024-03-31T15:14:39.893 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:14:39.893 INFO:teuthology.orchestra.run.smithi016.stdout:[mon] 2024-03-31T15:14:39.893 INFO:teuthology.orchestra.run.smithi016.stdout: debug ms = 1 2024-03-31T15:14:39.893 INFO:teuthology.orchestra.run.smithi016.stdout: debug mon = 20 2024-03-31T15:14:39.893 INFO:teuthology.orchestra.run.smithi016.stdout: debug paxos = 20 2024-03-31T15:14:39.894 INFO:teuthology.orchestra.run.smithi016.stdout: debug auth = 20 2024-03-31T15:14:39.894 INFO:teuthology.orchestra.run.smithi016.stdout: mon data avail warn = 5 2024-03-31T15:14:39.894 INFO:teuthology.orchestra.run.smithi016.stdout: mon mgr mkfs grace = 240 2024-03-31T15:14:39.894 INFO:teuthology.orchestra.run.smithi016.stdout: mon reweight min pgs per osd = 4 2024-03-31T15:14:39.894 INFO:teuthology.orchestra.run.smithi016.stdout: mon osd reporter subtree level = osd 2024-03-31T15:14:39.894 INFO:teuthology.orchestra.run.smithi016.stdout: mon osd prime pg temp = true 2024-03-31T15:14:39.894 INFO:teuthology.orchestra.run.smithi016.stdout: mon reweight min bytes per osd = 10 2024-03-31T15:14:39.894 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:14:39.894 INFO:teuthology.orchestra.run.smithi016.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-31T15:14:39.894 INFO:teuthology.orchestra.run.smithi016.stdout: auth mon ticket ttl = 660 # 11m 2024-03-31T15:14:39.894 INFO:teuthology.orchestra.run.smithi016.stdout: auth service ticket ttl = 240 # 4m 2024-03-31T15:14:39.894 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:14:39.894 INFO:teuthology.orchestra.run.smithi016.stdout: # don't complain about insecure global_id in the test suite 2024-03-31T15:14:39.894 INFO:teuthology.orchestra.run.smithi016.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-31T15:14:39.895 INFO:teuthology.orchestra.run.smithi016.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-31T15:14:39.895 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:14:39.895 INFO:teuthology.orchestra.run.smithi016.stdout: # 1m isn't quite enough 2024-03-31T15:14:39.895 INFO:teuthology.orchestra.run.smithi016.stdout: mon_down_mkfs_grace = 2m 2024-03-31T15:14:39.895 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:14:39.895 INFO:teuthology.orchestra.run.smithi016.stdout: mon_warn_on_filestore_osds = false 2024-03-31T15:14:39.895 INFO:teuthology.orchestra.run.smithi016.stdout: mon down mkfs grace = 300 2024-03-31T15:14:39.895 INFO:teuthology.orchestra.run.smithi016.stdout: mon op complaint time = 120 2024-03-31T15:14:39.895 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:14:39.895 INFO:teuthology.orchestra.run.smithi016.stdout:[client] 2024-03-31T15:14:39.895 INFO:teuthology.orchestra.run.smithi016.stdout: rgw cache enabled = true 2024-03-31T15:14:39.895 INFO:teuthology.orchestra.run.smithi016.stdout: rgw enable ops log = true 2024-03-31T15:14:39.895 INFO:teuthology.orchestra.run.smithi016.stdout: rgw enable usage log = true 2024-03-31T15:14:39.895 INFO:teuthology.orchestra.run.smithi016.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-31T15:14:39.895 INFO:teuthology.orchestra.run.smithi016.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-31T15:14:39.896 INFO:teuthology.orchestra.run.smithi016.stdout: client mount timeout = 600 2024-03-31T15:14:39.896 INFO:teuthology.orchestra.run.smithi016.stdout: debug client = 20 2024-03-31T15:14:39.896 INFO:teuthology.orchestra.run.smithi016.stdout: debug ms = 1 2024-03-31T15:14:39.896 INFO:teuthology.orchestra.run.smithi016.stdout: pid file = "" 2024-03-31T15:14:39.896 INFO:teuthology.orchestra.run.smithi016.stdout: rados mon op timeout = 900 2024-03-31T15:14:39.896 INFO:teuthology.orchestra.run.smithi016.stdout: rados osd op timeout = 900 2024-03-31T15:14:39.896 INFO:teuthology.orchestra.run.smithi016.stdout:[mon.a] 2024-03-31T15:14:39.896 INFO:teuthology.orchestra.run.smithi016.stdout:[mon.b] 2024-03-31T15:14:39.896 INFO:teuthology.orchestra.run.smithi016.stdout:[mon.c] 2024-03-31T15:14:39.896 INFO:teuthology.orchestra.run.smithi016.stdout:[mds.a] 2024-03-31T15:14:39.896 INFO:teuthology.orchestra.run.smithi016.stdout:[mds.c] 2024-03-31T15:14:39.896 INFO:teuthology.orchestra.run.smithi016.stdout:[mds.b] 2024-03-31T15:14:39.896 INFO:teuthology.orchestra.run.smithi016.stdout:[mds.d] 2024-03-31T15:14:39.896 INFO:teuthology.orchestra.run.smithi016.stdout:[mds] 2024-03-31T15:14:39.897 INFO:teuthology.orchestra.run.smithi016.stdout: debug mds = 20 2024-03-31T15:14:39.897 INFO:teuthology.orchestra.run.smithi016.stdout: debug mds balancer = 20 2024-03-31T15:14:39.897 INFO:teuthology.orchestra.run.smithi016.stdout: debug ms = 1 2024-03-31T15:14:39.897 INFO:teuthology.orchestra.run.smithi016.stdout: mds debug frag = True 2024-03-31T15:14:39.897 INFO:teuthology.orchestra.run.smithi016.stdout: mds debug scatterstat = False 2024-03-31T15:14:39.897 INFO:teuthology.orchestra.run.smithi016.stdout: mds op complaint time = 180 2024-03-31T15:14:39.897 INFO:teuthology.orchestra.run.smithi016.stdout: mds verify scatter = False 2024-03-31T15:14:39.897 INFO:teuthology.orchestra.run.smithi016.stdout: osd op complaint time = 180 2024-03-31T15:14:39.897 INFO:teuthology.orchestra.run.smithi016.stdout: rados mon op timeout = 900 2024-03-31T15:14:39.897 INFO:teuthology.orchestra.run.smithi016.stdout: rados osd op timeout = 900 2024-03-31T15:14:39.898 INFO:teuthology.orchestra.run.smithi089.stdout:[global] 2024-03-31T15:14:39.898 INFO:teuthology.orchestra.run.smithi089.stdout: chdir = "" 2024-03-31T15:14:39.898 INFO:teuthology.orchestra.run.smithi089.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-31T15:14:39.898 INFO:teuthology.orchestra.run.smithi089.stdout: auth supported = cephx 2024-03-31T15:14:39.898 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-31T15:14:39.898 INFO:teuthology.orchestra.run.smithi089.stdout: filestore xattr use omap = true 2024-03-31T15:14:39.898 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-31T15:14:39.898 INFO:teuthology.orchestra.run.smithi089.stdout: mon clock drift allowed = 1.000 2024-03-31T15:14:39.898 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-31T15:14:39.898 INFO:teuthology.orchestra.run.smithi089.stdout: osd crush chooseleaf type = 0 2024-03-31T15:14:39.898 INFO:teuthology.orchestra.run.smithi089.stdout: auth debug = true 2024-03-31T15:14:39.898 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-31T15:14:39.899 INFO:teuthology.orchestra.run.smithi089.stdout: ms die on old message = true 2024-03-31T15:14:39.899 INFO:teuthology.orchestra.run.smithi089.stdout: ms die on bug = true 2024-03-31T15:14:39.899 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-31T15:14:39.899 INFO:teuthology.orchestra.run.smithi089.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-31T15:14:39.899 INFO:teuthology.orchestra.run.smithi089.stdout: mon pg warn max object skew = 0 2024-03-31T15:14:39.899 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-31T15:14:39.899 INFO:teuthology.orchestra.run.smithi089.stdout: # disable pg_autoscaler by default for new pools 2024-03-31T15:14:39.899 INFO:teuthology.orchestra.run.smithi089.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-31T15:14:39.899 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-31T15:14:39.899 INFO:teuthology.orchestra.run.smithi089.stdout: osd pool default size = 2 2024-03-31T15:14:39.899 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-31T15:14:39.899 INFO:teuthology.orchestra.run.smithi089.stdout: mon osd allow primary affinity = true 2024-03-31T15:14:39.899 INFO:teuthology.orchestra.run.smithi089.stdout: mon osd allow pg remap = true 2024-03-31T15:14:39.899 INFO:teuthology.orchestra.run.smithi089.stdout: mon warn on legacy crush tunables = false 2024-03-31T15:14:39.900 INFO:teuthology.orchestra.run.smithi089.stdout: mon warn on crush straw calc version zero = false 2024-03-31T15:14:39.900 INFO:teuthology.orchestra.run.smithi089.stdout: mon warn on no sortbitwise = false 2024-03-31T15:14:39.900 INFO:teuthology.orchestra.run.smithi089.stdout: mon warn on osd down out interval zero = false 2024-03-31T15:14:39.900 INFO:teuthology.orchestra.run.smithi089.stdout: mon warn on too few osds = false 2024-03-31T15:14:39.900 INFO:teuthology.orchestra.run.smithi089.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-31T15:14:39.900 INFO:teuthology.orchestra.run.smithi089.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-31T15:14:39.900 INFO:teuthology.orchestra.run.smithi089.stdout: mon_allow_pool_size_one = true 2024-03-31T15:14:39.900 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-31T15:14:39.900 INFO:teuthology.orchestra.run.smithi089.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-31T15:14:39.900 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-31T15:14:39.900 INFO:teuthology.orchestra.run.smithi089.stdout: osd default data pool replay window = 5 2024-03-31T15:14:39.900 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-31T15:14:39.900 INFO:teuthology.orchestra.run.smithi089.stdout: mon allow pool delete = true 2024-03-31T15:14:39.900 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-31T15:14:39.900 INFO:teuthology.orchestra.run.smithi089.stdout: mon cluster log file level = debug 2024-03-31T15:14:39.901 INFO:teuthology.orchestra.run.smithi089.stdout: debug asserts on shutdown = true 2024-03-31T15:14:39.901 INFO:teuthology.orchestra.run.smithi089.stdout: mon health detail to clog = false 2024-03-31T15:14:39.901 INFO:teuthology.orchestra.run.smithi089.stdout: mon host = "172.21.15.16,172.21.15.89,[v2:172.21.15.89:3301,v1:172.21.15.89:6790]" 2024-03-31T15:14:39.901 INFO:teuthology.orchestra.run.smithi089.stdout: fsid = 286eb5da-fb47-4550-b290-d9519c13b9a5 2024-03-31T15:14:39.901 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-31T15:14:39.901 INFO:teuthology.orchestra.run.smithi089.stdout:[osd] 2024-03-31T15:14:39.901 INFO:teuthology.orchestra.run.smithi089.stdout: osd journal size = 100 2024-03-31T15:14:39.901 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-31T15:14:39.901 INFO:teuthology.orchestra.run.smithi089.stdout: osd scrub load threshold = 5.0 2024-03-31T15:14:39.901 INFO:teuthology.orchestra.run.smithi089.stdout: osd scrub max interval = 600 2024-03-31T15:14:39.901 INFO:teuthology.orchestra.run.smithi089.stdout: osd mclock profile = high_recovery_ops 2024-03-31T15:14:39.901 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-31T15:14:39.901 INFO:teuthology.orchestra.run.smithi089.stdout: osd recover clone overlap = true 2024-03-31T15:14:39.901 INFO:teuthology.orchestra.run.smithi089.stdout: osd recovery max chunk = 1048576 2024-03-31T15:14:39.901 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-31T15:14:39.902 INFO:teuthology.orchestra.run.smithi089.stdout: osd debug shutdown = true 2024-03-31T15:14:39.902 INFO:teuthology.orchestra.run.smithi089.stdout: osd debug op order = true 2024-03-31T15:14:39.902 INFO:teuthology.orchestra.run.smithi089.stdout: osd debug verify stray on activate = true 2024-03-31T15:14:39.902 INFO:teuthology.orchestra.run.smithi089.stdout: osd debug trim objects = true 2024-03-31T15:14:39.902 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-31T15:14:39.902 INFO:teuthology.orchestra.run.smithi089.stdout: osd open classes on start = true 2024-03-31T15:14:39.902 INFO:teuthology.orchestra.run.smithi089.stdout: osd debug pg log writeout = true 2024-03-31T15:14:39.902 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-31T15:14:39.902 INFO:teuthology.orchestra.run.smithi089.stdout: osd deep scrub update digest min age = 30 2024-03-31T15:14:39.902 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-31T15:14:39.902 INFO:teuthology.orchestra.run.smithi089.stdout: osd map max advance = 10 2024-03-31T15:14:39.902 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-31T15:14:39.902 INFO:teuthology.orchestra.run.smithi089.stdout: journal zero on create = true 2024-03-31T15:14:39.902 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-31T15:14:39.902 INFO:teuthology.orchestra.run.smithi089.stdout: filestore ondisk finisher threads = 3 2024-03-31T15:14:39.903 INFO:teuthology.orchestra.run.smithi089.stdout: filestore apply finisher threads = 3 2024-03-31T15:14:39.903 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-31T15:14:39.903 INFO:teuthology.orchestra.run.smithi089.stdout: bdev debug aio = true 2024-03-31T15:14:39.903 INFO:teuthology.orchestra.run.smithi089.stdout: osd debug misdirected ops = true 2024-03-31T15:14:39.903 INFO:teuthology.orchestra.run.smithi089.stdout: bdev async discard = True 2024-03-31T15:14:39.903 INFO:teuthology.orchestra.run.smithi089.stdout: bdev enable discard = True 2024-03-31T15:14:39.903 INFO:teuthology.orchestra.run.smithi089.stdout: bluestore allocator = bitmap 2024-03-31T15:14:39.903 INFO:teuthology.orchestra.run.smithi089.stdout: bluestore block size = 96636764160 2024-03-31T15:14:39.903 INFO:teuthology.orchestra.run.smithi089.stdout: bluestore fsck on mount = True 2024-03-31T15:14:39.903 INFO:teuthology.orchestra.run.smithi089.stdout: debug bluefs = 1/20 2024-03-31T15:14:39.903 INFO:teuthology.orchestra.run.smithi089.stdout: debug bluestore = 1/20 2024-03-31T15:14:39.903 INFO:teuthology.orchestra.run.smithi089.stdout: debug ms = 1 2024-03-31T15:14:39.903 INFO:teuthology.orchestra.run.smithi089.stdout: debug osd = 20 2024-03-31T15:14:39.903 INFO:teuthology.orchestra.run.smithi089.stdout: debug rocksdb = 4/10 2024-03-31T15:14:39.903 INFO:teuthology.orchestra.run.smithi089.stdout: mon osd backfillfull_ratio = 0.85 2024-03-31T15:14:39.903 INFO:teuthology.orchestra.run.smithi089.stdout: mon osd full ratio = 0.9 2024-03-31T15:14:39.904 INFO:teuthology.orchestra.run.smithi089.stdout: mon osd nearfull ratio = 0.8 2024-03-31T15:14:39.904 INFO:teuthology.orchestra.run.smithi089.stdout: osd failsafe full ratio = 0.95 2024-03-31T15:14:39.904 INFO:teuthology.orchestra.run.smithi089.stdout: osd objectstore = bluestore 2024-03-31T15:14:39.904 INFO:teuthology.orchestra.run.smithi089.stdout: osd op complaint time = 180 2024-03-31T15:14:39.904 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-31T15:14:39.904 INFO:teuthology.orchestra.run.smithi089.stdout:[mgr] 2024-03-31T15:14:39.904 INFO:teuthology.orchestra.run.smithi089.stdout: debug ms = 1 2024-03-31T15:14:39.904 INFO:teuthology.orchestra.run.smithi089.stdout: debug mgr = 20 2024-03-31T15:14:39.904 INFO:teuthology.orchestra.run.smithi089.stdout: debug mon = 20 2024-03-31T15:14:39.904 INFO:teuthology.orchestra.run.smithi089.stdout: debug auth = 20 2024-03-31T15:14:39.904 INFO:teuthology.orchestra.run.smithi089.stdout: mon reweight min pgs per osd = 4 2024-03-31T15:14:39.904 INFO:teuthology.orchestra.run.smithi089.stdout: mon reweight min bytes per osd = 10 2024-03-31T15:14:39.904 INFO:teuthology.orchestra.run.smithi089.stdout: mgr/telemetry/nag = false 2024-03-31T15:14:39.904 INFO:teuthology.orchestra.run.smithi089.stdout: debug client = 20 2024-03-31T15:14:39.904 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-31T15:14:39.904 INFO:teuthology.orchestra.run.smithi089.stdout:[mon] 2024-03-31T15:14:39.905 INFO:teuthology.orchestra.run.smithi089.stdout: debug ms = 1 2024-03-31T15:14:39.905 INFO:teuthology.orchestra.run.smithi089.stdout: debug mon = 20 2024-03-31T15:14:39.905 INFO:teuthology.orchestra.run.smithi089.stdout: debug paxos = 20 2024-03-31T15:14:39.905 INFO:teuthology.orchestra.run.smithi089.stdout: debug auth = 20 2024-03-31T15:14:39.905 INFO:teuthology.orchestra.run.smithi089.stdout: mon data avail warn = 5 2024-03-31T15:14:39.905 INFO:teuthology.orchestra.run.smithi089.stdout: mon mgr mkfs grace = 240 2024-03-31T15:14:39.905 INFO:teuthology.orchestra.run.smithi089.stdout: mon reweight min pgs per osd = 4 2024-03-31T15:14:39.905 INFO:teuthology.orchestra.run.smithi089.stdout: mon osd reporter subtree level = osd 2024-03-31T15:14:39.905 INFO:teuthology.orchestra.run.smithi089.stdout: mon osd prime pg temp = true 2024-03-31T15:14:39.905 INFO:teuthology.orchestra.run.smithi089.stdout: mon reweight min bytes per osd = 10 2024-03-31T15:14:39.905 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-31T15:14:39.905 INFO:teuthology.orchestra.run.smithi089.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-31T15:14:39.905 INFO:teuthology.orchestra.run.smithi089.stdout: auth mon ticket ttl = 660 # 11m 2024-03-31T15:14:39.905 INFO:teuthology.orchestra.run.smithi089.stdout: auth service ticket ttl = 240 # 4m 2024-03-31T15:14:39.905 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-31T15:14:39.905 INFO:teuthology.orchestra.run.smithi089.stdout: # don't complain about insecure global_id in the test suite 2024-03-31T15:14:39.906 INFO:teuthology.orchestra.run.smithi089.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-31T15:14:39.906 INFO:teuthology.orchestra.run.smithi089.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-31T15:14:39.906 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-31T15:14:39.906 INFO:teuthology.orchestra.run.smithi089.stdout: # 1m isn't quite enough 2024-03-31T15:14:39.906 INFO:teuthology.orchestra.run.smithi089.stdout: mon_down_mkfs_grace = 2m 2024-03-31T15:14:39.906 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-31T15:14:39.906 INFO:teuthology.orchestra.run.smithi089.stdout: mon_warn_on_filestore_osds = false 2024-03-31T15:14:39.906 INFO:teuthology.orchestra.run.smithi089.stdout: mon down mkfs grace = 300 2024-03-31T15:14:39.906 INFO:teuthology.orchestra.run.smithi089.stdout: mon op complaint time = 120 2024-03-31T15:14:39.906 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-31T15:14:39.906 INFO:teuthology.orchestra.run.smithi089.stdout:[client] 2024-03-31T15:14:39.906 INFO:teuthology.orchestra.run.smithi089.stdout: rgw cache enabled = true 2024-03-31T15:14:39.906 INFO:teuthology.orchestra.run.smithi089.stdout: rgw enable ops log = true 2024-03-31T15:14:39.907 INFO:teuthology.orchestra.run.smithi089.stdout: rgw enable usage log = true 2024-03-31T15:14:39.907 INFO:teuthology.orchestra.run.smithi089.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-31T15:14:39.907 INFO:teuthology.orchestra.run.smithi089.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-31T15:14:39.907 INFO:teuthology.orchestra.run.smithi089.stdout: client mount timeout = 600 2024-03-31T15:14:39.907 INFO:teuthology.orchestra.run.smithi089.stdout: debug client = 20 2024-03-31T15:14:39.907 INFO:teuthology.orchestra.run.smithi089.stdout: debug ms = 1 2024-03-31T15:14:39.907 INFO:teuthology.orchestra.run.smithi089.stdout: pid file = "" 2024-03-31T15:14:39.907 INFO:teuthology.orchestra.run.smithi089.stdout: rados mon op timeout = 900 2024-03-31T15:14:39.907 INFO:teuthology.orchestra.run.smithi089.stdout: rados osd op timeout = 900 2024-03-31T15:14:39.907 INFO:teuthology.orchestra.run.smithi089.stdout:[mon.a] 2024-03-31T15:14:39.907 INFO:teuthology.orchestra.run.smithi089.stdout:[mon.b] 2024-03-31T15:14:39.907 INFO:teuthology.orchestra.run.smithi089.stdout:[mon.c] 2024-03-31T15:14:39.907 INFO:teuthology.orchestra.run.smithi089.stdout:[mds.a] 2024-03-31T15:14:39.907 INFO:teuthology.orchestra.run.smithi089.stdout:[mds.c] 2024-03-31T15:14:39.907 INFO:teuthology.orchestra.run.smithi089.stdout:[mds.b] 2024-03-31T15:14:39.908 INFO:teuthology.orchestra.run.smithi089.stdout:[mds.d] 2024-03-31T15:14:39.908 INFO:teuthology.orchestra.run.smithi089.stdout:[mds] 2024-03-31T15:14:39.908 INFO:teuthology.orchestra.run.smithi089.stdout: debug mds = 20 2024-03-31T15:14:39.908 INFO:teuthology.orchestra.run.smithi089.stdout: debug mds balancer = 20 2024-03-31T15:14:39.908 INFO:teuthology.orchestra.run.smithi089.stdout: debug ms = 1 2024-03-31T15:14:39.908 INFO:teuthology.orchestra.run.smithi089.stdout: mds debug frag = True 2024-03-31T15:14:39.908 INFO:teuthology.orchestra.run.smithi089.stdout: mds debug scatterstat = False 2024-03-31T15:14:39.908 INFO:teuthology.orchestra.run.smithi089.stdout: mds op complaint time = 180 2024-03-31T15:14:39.908 INFO:teuthology.orchestra.run.smithi089.stdout: mds verify scatter = False 2024-03-31T15:14:39.908 INFO:teuthology.orchestra.run.smithi089.stdout: osd op complaint time = 180 2024-03-31T15:14:39.908 INFO:teuthology.orchestra.run.smithi089.stdout: rados mon op timeout = 900 2024-03-31T15:14:39.908 INFO:teuthology.orchestra.run.smithi089.stdout: rados osd op timeout = 900 2024-03-31T15:14:39.910 DEBUG:teuthology.orchestra.run.smithi016:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata rmomapkey 10000000001.00000000 bravo_head 2024-03-31T15:14:39.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:40.005+0000 7fd607a03080 1 Processor -- start 2024-03-31T15:14:39.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:40.005+0000 7fd607a03080 1 -- start start 2024-03-31T15:14:40.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:40.005+0000 7fd607a03080 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557755d7b4d0 0x557755da66b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:40.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:40.005+0000 7fd607a03080 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x557755dc0850 0x557755e4cf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:40.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:40.005+0000 7fd607a03080 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557755e4f950 0x557755e4d440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:40.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:40.005+0000 7fd607a03080 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x557755d1e4b0 con 0x557755d7b4d0 2024-03-31T15:14:40.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:40.005+0000 7fd607a03080 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x557755d22500 con 0x557755e4f950 2024-03-31T15:14:40.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:40.005+0000 7fd607a03080 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x557755dc1c50 con 0x557755dc0850 2024-03-31T15:14:40.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:40.005+0000 7fd604c00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557755d7b4d0 0x557755da66b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:40.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:40.005+0000 7fd604c00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557755d7b4d0 0x557755da66b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40702/0 (socket says 172.21.15.16:40702) 2024-03-31T15:14:40.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:40.005+0000 7fd604c00640 1 -- 172.21.15.16:0/2955573772 learned_addr learned my addr 172.21.15.16:0/2955573772 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:14:40.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:40.005+0000 7fd5ffe00640 1 --2- 172.21.15.16:0/2955573772 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x557755dc0850 0x557755e4cf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:40.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:40.005+0000 7fd605600640 1 --2- 172.21.15.16:0/2955573772 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557755e4f950 0x557755e4d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:40.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:40.005+0000 7fd604c00640 1 -- 172.21.15.16:0/2955573772 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x557755dc0850 msgr2=0x557755e4cf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:14:40.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:40.005+0000 7fd604c00640 1 --2- 172.21.15.16:0/2955573772 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x557755dc0850 0x557755e4cf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:40.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:40.005+0000 7fd604c00640 1 -- 172.21.15.16:0/2955573772 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557755e4f950 msgr2=0x557755e4d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:14:40.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:40.005+0000 7fd604c00640 1 --2- 172.21.15.16:0/2955573772 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557755e4f950 0x557755e4d440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:40.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:40.005+0000 7fd604c00640 1 -- 172.21.15.16:0/2955573772 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557755e52610 con 0x557755d7b4d0 2024-03-31T15:14:40.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:40.005+0000 7fd604c00640 1 --2- 172.21.15.16:0/2955573772 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557755d7b4d0 0x557755da66b0 secure :-1 s=READY pgs=1740 cs=0 l=1 rev1=1 crypto rx=0x7fd5f8001eb0 tx=0x7fd5f800b680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d13d1b14c0933c2 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:40.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:40.009+0000 7fd5fea00640 1 -- 172.21.15.16:0/2955573772 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5f800a040 con 0x557755d7b4d0 2024-03-31T15:14:40.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:40.009+0000 7fd5fea00640 1 -- 172.21.15.16:0/2955573772 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd5f800f620 con 0x557755d7b4d0 2024-03-31T15:14:40.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:40.009+0000 7fd5fea00640 1 -- 172.21.15.16:0/2955573772 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5f800fa10 con 0x557755d7b4d0 2024-03-31T15:14:40.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:40.009+0000 7fd607a03080 1 -- 172.21.15.16:0/2955573772 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557755d7b4d0 msgr2=0x557755da66b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:40.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:40.009+0000 7fd607a03080 1 --2- 172.21.15.16:0/2955573772 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557755d7b4d0 0x557755da66b0 secure :-1 s=READY pgs=1740 cs=0 l=1 rev1=1 crypto rx=0x7fd5f8001eb0 tx=0x7fd5f800b680 comp rx=0 tx=0).stop 2024-03-31T15:14:40.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:40.009+0000 7fd607a03080 1 -- 172.21.15.16:0/2955573772 shutdown_connections 2024-03-31T15:14:40.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:40.009+0000 7fd607a03080 1 --2- 172.21.15.16:0/2955573772 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557755e4f950 0x557755e4d440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:40.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:40.009+0000 7fd607a03080 1 --2- 172.21.15.16:0/2955573772 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x557755dc0850 0x557755e4cf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:40.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:40.009+0000 7fd607a03080 1 --2- 172.21.15.16:0/2955573772 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557755d7b4d0 0x557755da66b0 unknown :-1 s=CLOSED pgs=1740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:40.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:40.009+0000 7fd607a03080 1 -- 172.21.15.16:0/2955573772 >> 172.21.15.16:0/2955573772 conn(0x557755da4050 msgr2=0x557755db9370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:14:40.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:40.009+0000 7fd607a03080 1 -- 172.21.15.16:0/2955573772 shutdown_connections 2024-03-31T15:14:40.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:40.009+0000 7fd607a03080 1 -- 172.21.15.16:0/2955573772 wait complete. 2024-03-31T15:14:40.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:40.009+0000 7fd607a03080 1 Processor -- start 2024-03-31T15:14:40.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:40.009+0000 7fd607a03080 1 -- start start 2024-03-31T15:14:40.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:40.009+0000 7fd607a03080 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557755d7b4d0 0x557755dbc6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:40.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:40.009+0000 7fd607a03080 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557755dc0850 0x557755da8be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:40.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:40.009+0000 7fd607a03080 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x557755e4f950 0x557755eeb190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:40.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:40.009+0000 7fd607a03080 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x557755e52370 con 0x557755d7b4d0 2024-03-31T15:14:40.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:40.009+0000 7fd607a03080 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x557755da6bf0 con 0x557755dc0850 2024-03-31T15:14:40.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:40.009+0000 7fd607a03080 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x557755e50ff0 con 0x557755e4f950 2024-03-31T15:14:40.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:40.009+0000 7fd604c00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557755d7b4d0 0x557755dbc6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:40.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:40.009+0000 7fd604c00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557755d7b4d0 0x557755dbc6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40712/0 (socket says 172.21.15.16:40712) 2024-03-31T15:14:40.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:40.009+0000 7fd604c00640 1 -- 172.21.15.16:0/1518692828 learned_addr learned my addr 172.21.15.16:0/1518692828 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:14:40.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:40.009+0000 7fd5ffe00640 1 --2- 172.21.15.16:0/1518692828 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557755dc0850 0x557755da8be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:40.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:40.009+0000 7fd605600640 1 --2- 172.21.15.16:0/1518692828 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x557755e4f950 0x557755eeb190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:40.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:40.009+0000 7fd604c00640 1 -- 172.21.15.16:0/1518692828 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x557755e4f950 msgr2=0x557755eeb190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:40.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:40.009+0000 7fd604c00640 1 --2- 172.21.15.16:0/1518692828 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x557755e4f950 0x557755eeb190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:40.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:40.009+0000 7fd604c00640 1 -- 172.21.15.16:0/1518692828 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557755dc0850 msgr2=0x557755da8be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:40.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:40.009+0000 7fd604c00640 1 --2- 172.21.15.16:0/1518692828 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557755dc0850 0x557755da8be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:40.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:40.009+0000 7fd604c00640 1 -- 172.21.15.16:0/1518692828 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557755dbcfe0 con 0x557755d7b4d0 2024-03-31T15:14:40.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:40.009+0000 7fd604c00640 1 --2- 172.21.15.16:0/1518692828 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557755d7b4d0 0x557755dbc6d0 secure :-1 s=READY pgs=1741 cs=0 l=1 rev1=1 crypto rx=0x7fd5f800e650 tx=0x7fd5f8000bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:40.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:40.009+0000 7fd5fd600640 1 -- 172.21.15.16:0/1518692828 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5f800fe10 con 0x557755d7b4d0 2024-03-31T15:14:40.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:40.009+0000 7fd5fd600640 1 -- 172.21.15.16:0/1518692828 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd5f8021020 con 0x557755d7b4d0 2024-03-31T15:14:40.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:40.009+0000 7fd5fd600640 1 -- 172.21.15.16:0/1518692828 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5f800ea40 con 0x557755d7b4d0 2024-03-31T15:14:40.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:40.009+0000 7fd607a03080 1 -- 172.21.15.16:0/1518692828 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x557755dbd280 con 0x557755d7b4d0 2024-03-31T15:14:40.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:40.009+0000 7fd607a03080 1 -- 172.21.15.16:0/1518692828 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557755dbd7c0 con 0x557755d7b4d0 2024-03-31T15:14:40.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:40.013+0000 7fd5fd600640 1 -- 172.21.15.16:0/1518692828 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fd5f8017070 con 0x557755d7b4d0 2024-03-31T15:14:40.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:40.013+0000 7fd5fd600640 1 --2- 172.21.15.16:0/1518692828 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fd5e4066d60 0x7fd5e4069220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:40.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:40.013+0000 7fd5fd600640 1 -- 172.21.15.16:0/1518692828 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(112..112 src has 1..112) ==== 5853+0+0 (secure 0 0 0) 0x7fd5f8087060 con 0x557755d7b4d0 2024-03-31T15:14:40.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:40.013+0000 7fd607a03080 1 --2- 172.21.15.16:0/1518692828 >> [v2:172.21.15.16:6809/3360352974,v1:172.21.15.16:6812/3360352974] conn(0x557755da9120 0x557755dbb1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:40.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:40.013+0000 7fd607a03080 1 -- 172.21.15.16:0/1518692828 --> [v2:172.21.15.16:6809/3360352974,v1:172.21.15.16:6812/3360352974] -- osd_op(unknown.0.0:1 16.7 16:e2e07930:::10000000001.00000000:head [omap-rm-keys in=18b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x557755e445d0 con 0x557755da9120 2024-03-31T15:14:40.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:40.013+0000 7fd5ffe00640 1 --2- 172.21.15.16:0/1518692828 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fd5e4066d60 0x7fd5e4069220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:40.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:40.013+0000 7fd605600640 1 --2- 172.21.15.16:0/1518692828 >> [v2:172.21.15.16:6809/3360352974,v1:172.21.15.16:6812/3360352974] conn(0x557755da9120 0x557755dbb1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:40.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:40.013+0000 7fd605600640 1 --2- 172.21.15.16:0/1518692828 >> [v2:172.21.15.16:6809/3360352974,v1:172.21.15.16:6812/3360352974] conn(0x557755da9120 0x557755dbb1c0 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:40.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:40.013+0000 7fd5ffe00640 1 --2- 172.21.15.16:0/1518692828 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fd5e4066d60 0x7fd5e4069220 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7fd5f0000c10 tx=0x7fd5f0010040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:40.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:40.017+0000 7fd605600640 1 -- 172.21.15.16:0/1518692828 <== osd.3 v2:172.21.15.16:6809/3360352974 1 ==== osd_op_reply(1 10000000001.00000000 [omap-rm-keys] v112'5 uv5 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7fd6000b3040 con 0x557755da9120 2024-03-31T15:14:40.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:40.017+0000 7fd607a03080 1 -- 172.21.15.16:0/1518692828 >> [v2:172.21.15.16:6809/3360352974,v1:172.21.15.16:6812/3360352974] conn(0x557755da9120 msgr2=0x557755dbb1c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:40.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:40.017+0000 7fd607a03080 1 --2- 172.21.15.16:0/1518692828 >> [v2:172.21.15.16:6809/3360352974,v1:172.21.15.16:6812/3360352974] conn(0x557755da9120 0x557755dbb1c0 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:40.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:40.017+0000 7fd607a03080 1 -- 172.21.15.16:0/1518692828 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fd5e4066d60 msgr2=0x7fd5e4069220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:40.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:40.017+0000 7fd607a03080 1 --2- 172.21.15.16:0/1518692828 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fd5e4066d60 0x7fd5e4069220 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7fd5f0000c10 tx=0x7fd5f0010040 comp rx=0 tx=0).stop 2024-03-31T15:14:40.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:40.017+0000 7fd607a03080 1 -- 172.21.15.16:0/1518692828 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557755d7b4d0 msgr2=0x557755dbc6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:40.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:40.017+0000 7fd607a03080 1 --2- 172.21.15.16:0/1518692828 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557755d7b4d0 0x557755dbc6d0 secure :-1 s=READY pgs=1741 cs=0 l=1 rev1=1 crypto rx=0x7fd5f800e650 tx=0x7fd5f8000bd0 comp rx=0 tx=0).stop 2024-03-31T15:14:40.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:40.017+0000 7fd605600640 1 -- 172.21.15.16:0/1518692828 reap_dead start 2024-03-31T15:14:40.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:40.017+0000 7fd607a03080 1 -- 172.21.15.16:0/1518692828 shutdown_connections 2024-03-31T15:14:40.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:40.017+0000 7fd607a03080 1 -- 172.21.15.16:0/1518692828 >> 172.21.15.16:0/1518692828 conn(0x557755da4050 msgr2=0x557755db8ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:14:40.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:40.017+0000 7fd607a03080 1 -- 172.21.15.16:0/1518692828 shutdown_connections 2024-03-31T15:14:40.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:40.017+0000 7fd607a03080 1 -- 172.21.15.16:0/1518692828 wait complete. 2024-03-31T15:14:40.015 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs joinable true 2024-03-31T15:14:40.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:40.209+0000 7fb7bf200640 1 Processor -- start 2024-03-31T15:14:40.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:40.209+0000 7fb7bf200640 1 -- start start 2024-03-31T15:14:40.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:40.209+0000 7fb7bf200640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb7b80b14c0 0x7fb7b806ff10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:40.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:40.209+0000 7fb7bf200640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fb7b80d02c0 0x7fb7b8070450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:40.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:40.209+0000 7fb7bf200640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb7b80d0760 con 0x7fb7b806f710 2024-03-31T15:14:40.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:40.209+0000 7fb7bf200640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7fb7b80cf3b0 con 0x7fb7b80d02c0 2024-03-31T15:14:40.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:40.209+0000 7fb7bf200640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fb7b80d1370 con 0x7fb7b80b14c0 2024-03-31T15:14:40.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:40.209+0000 7fb7bde00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb7b80b14c0 0x7fb7b806ff10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:40.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:40.209+0000 7fb7be800640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fb7b806f710 0x7fb7b815dc00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.16:58510/0 (socket says 172.21.15.16:58510) 2024-03-31T15:14:40.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:40.209+0000 7fb7be800640 1 -- 172.21.15.16:0/3530175049 learned_addr learned my addr 172.21.15.16:0/3530175049 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:14:40.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:40.209+0000 7fb7bd400640 1 --2- 172.21.15.16:0/3530175049 >> v2:172.21.15.89:3300/0 conn(0x7fb7b80d02c0 0x7fb7b8070450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:40.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:40.213+0000 7fb7bca00640 1 -- 172.21.15.16:0/3530175049 <== mon.1 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1384826794 0 0) 0x7fb7b80d0760 con 0x7fb7b806f710 2024-03-31T15:14:40.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:40.213+0000 7fb7bd400640 1 -- 172.21.15.16:0/3530175049 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb7b80b14c0 msgr2=0x7fb7b806ff10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:14:40.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:40.213+0000 7fb7bd400640 1 --2- 172.21.15.16:0/3530175049 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb7b80b14c0 0x7fb7b806ff10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:40.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:40.213+0000 7fb7bd400640 1 -- 172.21.15.16:0/3530175049 >> v1:172.21.15.89:6789/0 conn(0x7fb7b806f710 legacy=0x7fb7b815dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:40.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:40.213+0000 7fb7bd400640 1 -- 172.21.15.16:0/3530175049 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7b80d0a40 con 0x7fb7b80d02c0 2024-03-31T15:14:40.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:40.213+0000 7fb7bd400640 1 --2- 172.21.15.16:0/3530175049 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb7b80d02c0 0x7fb7b8070450 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7fb7a80097c0 tx=0x7fb7a801c4a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9951f146c0297c56 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:40.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:40.213+0000 7fb7bca00640 1 -- 172.21.15.16:0/3530175049 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7a80b2070 con 0x7fb7b80d02c0 2024-03-31T15:14:40.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:40.213+0000 7fb7bca00640 1 -- 172.21.15.16:0/3530175049 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb7a80ad640 con 0x7fb7b80d02c0 2024-03-31T15:14:40.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:40.213+0000 7fb7bca00640 1 -- 172.21.15.16:0/3530175049 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7a80ada10 con 0x7fb7b80d02c0 2024-03-31T15:14:40.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:40.217+0000 7fb7bf200640 1 -- 172.21.15.16:0/3530175049 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb7b80d02c0 msgr2=0x7fb7b8070450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:40.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:40.217+0000 7fb7bf200640 1 --2- 172.21.15.16:0/3530175049 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb7b80d02c0 0x7fb7b8070450 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7fb7a80097c0 tx=0x7fb7a801c4a0 comp rx=0 tx=0).stop 2024-03-31T15:14:40.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:40.217+0000 7fb7bf200640 1 -- 172.21.15.16:0/3530175049 shutdown_connections 2024-03-31T15:14:40.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:40.217+0000 7fb7bf200640 1 --2- 172.21.15.16:0/3530175049 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb7b80d02c0 0x7fb7b8070450 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:40.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:40.217+0000 7fb7bf200640 1 --2- 172.21.15.16:0/3530175049 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb7b80b14c0 0x7fb7b806ff10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:40.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:40.217+0000 7fb7bf200640 1 -- 172.21.15.16:0/3530175049 >> 172.21.15.16:0/3530175049 conn(0x7fb7b80d1f90 msgr2=0x7fb7b80d23a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:14:40.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:40.217+0000 7fb7bf200640 1 -- 172.21.15.16:0/3530175049 shutdown_connections 2024-03-31T15:14:40.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:40.217+0000 7fb7bf200640 1 -- 172.21.15.16:0/3530175049 wait complete. 2024-03-31T15:14:40.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:40.217+0000 7fb7bf200640 1 Processor -- start 2024-03-31T15:14:40.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:40.217+0000 7fb7bf200640 1 -- start start 2024-03-31T15:14:40.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:40.217+0000 7fb7bf200640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb7b806f710 0x7fb7b81500c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:40.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:40.217+0000 7fb7bf200640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb7b80b14c0 0x7fb7b815ea70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:40.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:40.217+0000 7fb7bf200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb7b80d02c0 0x7fb7b815dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:40.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:40.217+0000 7fb7bf200640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fb7b815e8a0 con 0x7fb7b80d02c0 2024-03-31T15:14:40.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:40.217+0000 7fb7bf200640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fb7b815e720 con 0x7fb7b80b14c0 2024-03-31T15:14:40.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:40.217+0000 7fb7bf200640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fb7b80ccdf0 con 0x7fb7b806f710 2024-03-31T15:14:40.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:40.217+0000 7fb7be800640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb7b80d02c0 0x7fb7b815dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:40.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:40.217+0000 7fb7be800640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb7b80d02c0 0x7fb7b815dcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40714/0 (socket says 172.21.15.16:40714) 2024-03-31T15:14:40.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:40.217+0000 7fb7be800640 1 -- 172.21.15.16:0/2083987664 learned_addr learned my addr 172.21.15.16:0/2083987664 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:14:40.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:40.217+0000 7fb7bde00640 1 --2- 172.21.15.16:0/2083987664 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb7b806f710 0x7fb7b81500c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:40.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:40.217+0000 7fb7bd400640 1 --2- 172.21.15.16:0/2083987664 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb7b80b14c0 0x7fb7b815ea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:40.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:40.217+0000 7fb7be800640 1 -- 172.21.15.16:0/2083987664 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb7b806f710 msgr2=0x7fb7b81500c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:40.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:40.217+0000 7fb7be800640 1 --2- 172.21.15.16:0/2083987664 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb7b806f710 0x7fb7b81500c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:40.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:40.217+0000 7fb7be800640 1 -- 172.21.15.16:0/2083987664 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb7b80b14c0 msgr2=0x7fb7b815ea70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:40.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:40.217+0000 7fb7be800640 1 --2- 172.21.15.16:0/2083987664 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb7b80b14c0 0x7fb7b815ea70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:40.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:40.217+0000 7fb7be800640 1 -- 172.21.15.16:0/2083987664 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7b8160470 con 0x7fb7b80d02c0 2024-03-31T15:14:40.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:40.217+0000 7fb7be800640 1 --2- 172.21.15.16:0/2083987664 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb7b80d02c0 0x7fb7b815dcd0 secure :-1 s=READY pgs=1742 cs=0 l=1 rev1=1 crypto rx=0x7fb7b400c7a0 tx=0x7fb7b400cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:40.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:40.221+0000 7fb7b2000640 1 -- 172.21.15.16:0/2083987664 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7b4018070 con 0x7fb7b80d02c0 2024-03-31T15:14:40.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:40.221+0000 7fb7b2000640 1 -- 172.21.15.16:0/2083987664 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb7b400a640 con 0x7fb7b80d02c0 2024-03-31T15:14:40.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:40.221+0000 7fb7b2000640 1 -- 172.21.15.16:0/2083987664 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7b400aa70 con 0x7fb7b80d02c0 2024-03-31T15:14:40.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:40.221+0000 7fb7bf200640 1 -- 172.21.15.16:0/2083987664 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb7b8160750 con 0x7fb7b80d02c0 2024-03-31T15:14:40.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:40.221+0000 7fb7bf200640 1 -- 172.21.15.16:0/2083987664 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb7b8160d10 con 0x7fb7b80d02c0 2024-03-31T15:14:40.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:40.221+0000 7fb7bf200640 1 -- 172.21.15.16:0/2083987664 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb7b804f8c0 con 0x7fb7b80d02c0 2024-03-31T15:14:40.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:40.221+0000 7fb7b2000640 1 -- 172.21.15.16:0/2083987664 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fb7b400ac10 con 0x7fb7b80d02c0 2024-03-31T15:14:40.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:40.221+0000 7fb7b2000640 1 --2- 172.21.15.16:0/2083987664 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fb79c068380 0x7fb79c06a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:40.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:40.221+0000 7fb7b2000640 1 -- 172.21.15.16:0/2083987664 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(112..112 src has 1..112) ==== 5853+0+0 (secure 0 0 0) 0x7fb7b4085fc0 con 0x7fb7b80d02c0 2024-03-31T15:14:40.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:40.221+0000 7fb7bde00640 1 --2- 172.21.15.16:0/2083987664 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fb79c068380 0x7fb79c06a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:40.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:40.221+0000 7fb7bde00640 1 --2- 172.21.15.16:0/2083987664 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fb79c068380 0x7fb79c06a840 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7fb7b809ff80 tx=0x7fb7ac009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:40.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:40.225+0000 7fb7b2000640 1 -- 172.21.15.16:0/2083987664 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fb7b4055e90 con 0x7fb7b80d02c0 2024-03-31T15:14:40.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:40.429+0000 7fb7bf200640 1 -- 172.21.15.16:0/2083987664 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"} v 0) -- 0x7fb7b806fcf0 con 0x7fb7b80d02c0 2024-03-31T15:14:40.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:40.713+0000 7fb7b2000640 1 -- 172.21.15.16:0/2083987664 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]=0 cephfs marked joinable; MDS may join as newly active. v101) ==== 162+0+0 (secure 0 0 0) 0x7fb7b405a190 con 0x7fb7b80d02c0 2024-03-31T15:14:40.707 INFO:teuthology.orchestra.run.smithi016.stderr:cephfs marked joinable; MDS may join as newly active. 2024-03-31T15:14:40.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:40.717+0000 7fb7bf200640 1 -- 172.21.15.16:0/2083987664 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fb79c068380 msgr2=0x7fb79c06a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:40.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:40.717+0000 7fb7bf200640 1 --2- 172.21.15.16:0/2083987664 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fb79c068380 0x7fb79c06a840 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7fb7b809ff80 tx=0x7fb7ac009190 comp rx=0 tx=0).stop 2024-03-31T15:14:40.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:40.717+0000 7fb7bf200640 1 -- 172.21.15.16:0/2083987664 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb7b80d02c0 msgr2=0x7fb7b815dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:40.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:40.717+0000 7fb7bf200640 1 --2- 172.21.15.16:0/2083987664 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb7b80d02c0 0x7fb7b815dcd0 secure :-1 s=READY pgs=1742 cs=0 l=1 rev1=1 crypto rx=0x7fb7b400c7a0 tx=0x7fb7b400cc60 comp rx=0 tx=0).stop 2024-03-31T15:14:40.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:40.717+0000 7fb7bf200640 1 -- 172.21.15.16:0/2083987664 shutdown_connections 2024-03-31T15:14:40.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:40.717+0000 7fb7bf200640 1 --2- 172.21.15.16:0/2083987664 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fb79c068380 0x7fb79c06a840 unknown :-1 s=CLOSED pgs=738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:40.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:40.717+0000 7fb7bf200640 1 --2- 172.21.15.16:0/2083987664 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb7b80d02c0 0x7fb7b815dcd0 unknown :-1 s=CLOSED pgs=1742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:40.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:40.717+0000 7fb7bf200640 1 --2- 172.21.15.16:0/2083987664 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb7b80b14c0 0x7fb7b815ea70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:40.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:40.717+0000 7fb7bf200640 1 --2- 172.21.15.16:0/2083987664 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb7b806f710 0x7fb7b81500c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:40.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:40.717+0000 7fb7bf200640 1 -- 172.21.15.16:0/2083987664 >> 172.21.15.16:0/2083987664 conn(0x7fb7b80d1f90 msgr2=0x7fb7b806ff10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:14:40.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:40.717+0000 7fb7bf200640 1 -- 172.21.15.16:0/2083987664 shutdown_connections 2024-03-31T15:14:40.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:40.717+0000 7fb7bf200640 1 -- 172.21.15.16:0/2083987664 wait complete. 2024-03-31T15:14:40.735 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-31T15:14:40.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:40.881+0000 7fbd3ae00640 1 Processor -- start 2024-03-31T15:14:40.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:40.881+0000 7fbd3ae00640 1 -- start start 2024-03-31T15:14:40.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:40.881+0000 7fbd3ae00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fbd340b23d0 0x7fbd34071d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:40.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:40.881+0000 7fbd3ae00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fbd3415eee0 0x7fbd34072250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:40.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:40.881+0000 7fbd3ae00640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbd3415f3c0 con 0x7fbd3415b3d0 2024-03-31T15:14:40.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:40.881+0000 7fbd3ae00640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7fbd3415b7b0 con 0x7fbd3415eee0 2024-03-31T15:14:40.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:40.881+0000 7fbd3ae00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fbd3415f6a0 con 0x7fbd340b23d0 2024-03-31T15:14:40.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:40.881+0000 7fbd38e00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fbd3415eee0 0x7fbd34072250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:40.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:40.885+0000 7fbd38e00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fbd3415eee0 0x7fbd34072250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40728/0 (socket says 172.21.15.16:40728) 2024-03-31T15:14:40.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:40.885+0000 7fbd38e00640 1 -- 172.21.15.16:0/1973441855 learned_addr learned my addr 172.21.15.16:0/1973441855 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:14:40.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:40.885+0000 7fbd33e00640 1 --1- 172.21.15.16:0/1973441855 >> v1:172.21.15.89:6789/0 conn(0x7fbd3415b3d0 0x7fbd340b1cc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.16:58520/0 (socket says 172.21.15.16:58520) 2024-03-31T15:14:40.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:40.885+0000 7fbd2b400640 1 --2- 172.21.15.16:0/1973441855 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fbd340b23d0 0x7fbd34071d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:40.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:40.885+0000 7fbd38e00640 1 -- 172.21.15.16:0/1973441855 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fbd340b23d0 msgr2=0x7fbd34071d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:14:40.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:40.885+0000 7fbd38e00640 1 --2- 172.21.15.16:0/1973441855 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fbd340b23d0 0x7fbd34071d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:40.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:40.885+0000 7fbd38e00640 1 -- 172.21.15.16:0/1973441855 >> v1:172.21.15.89:6789/0 conn(0x7fbd3415b3d0 legacy=0x7fbd340b1cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:40.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:40.885+0000 7fbd38e00640 1 -- 172.21.15.16:0/1973441855 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd3415f980 con 0x7fbd3415eee0 2024-03-31T15:14:40.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:40.885+0000 7fbd2b400640 1 --2- 172.21.15.16:0/1973441855 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fbd340b23d0 0x7fbd34071d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:14:40.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:40.885+0000 7fbd38e00640 1 --2- 172.21.15.16:0/1973441855 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbd3415eee0 0x7fbd34072250 secure :-1 s=READY pgs=1743 cs=0 l=1 rev1=1 crypto rx=0x7fbd2c009e50 tx=0x7fbd2c01ca00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d98c35eba237d11 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:40.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:40.885+0000 7fbd33400640 1 -- 172.21.15.16:0/1973441855 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd2c0a5790 con 0x7fbd3415eee0 2024-03-31T15:14:40.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:40.885+0000 7fbd33400640 1 -- 172.21.15.16:0/1973441855 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbd2c0a5930 con 0x7fbd3415eee0 2024-03-31T15:14:40.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:40.885+0000 7fbd33400640 1 -- 172.21.15.16:0/1973441855 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd2c0a5d00 con 0x7fbd3415eee0 2024-03-31T15:14:40.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:40.885+0000 7fbd3ae00640 1 -- 172.21.15.16:0/1973441855 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbd3415eee0 msgr2=0x7fbd34072250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:40.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:40.889+0000 7fbd3ae00640 1 --2- 172.21.15.16:0/1973441855 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbd3415eee0 0x7fbd34072250 secure :-1 s=READY pgs=1743 cs=0 l=1 rev1=1 crypto rx=0x7fbd2c009e50 tx=0x7fbd2c01ca00 comp rx=0 tx=0).stop 2024-03-31T15:14:40.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:40.889+0000 7fbd3ae00640 1 -- 172.21.15.16:0/1973441855 shutdown_connections 2024-03-31T15:14:40.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:40.889+0000 7fbd3ae00640 1 --2- 172.21.15.16:0/1973441855 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbd3415eee0 0x7fbd34072250 unknown :-1 s=CLOSED pgs=1743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:40.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:40.889+0000 7fbd3ae00640 1 --2- 172.21.15.16:0/1973441855 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fbd340b23d0 0x7fbd34071d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:40.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:40.889+0000 7fbd3ae00640 1 -- 172.21.15.16:0/1973441855 >> 172.21.15.16:0/1973441855 conn(0x7fbd3406b620 msgr2=0x7fbd3406ba30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:14:40.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:40.889+0000 7fbd3ae00640 1 -- 172.21.15.16:0/1973441855 shutdown_connections 2024-03-31T15:14:40.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:40.889+0000 7fbd3ae00640 1 -- 172.21.15.16:0/1973441855 wait complete. 2024-03-31T15:14:40.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:40.889+0000 7fbd3ae00640 1 Processor -- start 2024-03-31T15:14:40.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:40.889+0000 7fbd3ae00640 1 -- start start 2024-03-31T15:14:40.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:40.889+0000 7fbd3ae00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fbd340b23d0 0x7fbd341f8990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:40.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:40.889+0000 7fbd3ae00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbd3415b3d0 0x7fbd341fa710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:40.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:40.889+0000 7fbd3ae00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbd3415eee0 0x7fbd340b0690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:40.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:40.889+0000 7fbd3ae00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fbd34073910 con 0x7fbd3415b3d0 2024-03-31T15:14:40.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:40.889+0000 7fbd3ae00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fbd34073790 con 0x7fbd3415eee0 2024-03-31T15:14:40.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:40.889+0000 7fbd3ae00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fbd34072790 con 0x7fbd340b23d0 2024-03-31T15:14:40.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:40.889+0000 7fbd2b400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbd3415b3d0 0x7fbd341fa710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:40.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:40.889+0000 7fbd2b400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbd3415b3d0 0x7fbd341fa710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40734/0 (socket says 172.21.15.16:40734) 2024-03-31T15:14:40.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:40.889+0000 7fbd2b400640 1 -- 172.21.15.16:0/743800756 learned_addr learned my addr 172.21.15.16:0/743800756 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:14:40.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:40.889+0000 7fbd38e00640 1 --2- 172.21.15.16:0/743800756 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbd3415eee0 0x7fbd340b0690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:40.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:40.889+0000 7fbd33e00640 1 --2- 172.21.15.16:0/743800756 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fbd340b23d0 0x7fbd341f8990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:40.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:40.889+0000 7fbd2b400640 1 -- 172.21.15.16:0/743800756 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fbd340b23d0 msgr2=0x7fbd341f8990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:40.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:40.889+0000 7fbd2b400640 1 --2- 172.21.15.16:0/743800756 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fbd340b23d0 0x7fbd341f8990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:40.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:40.889+0000 7fbd2b400640 1 -- 172.21.15.16:0/743800756 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbd3415eee0 msgr2=0x7fbd340b0690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:40.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:40.889+0000 7fbd2b400640 1 --2- 172.21.15.16:0/743800756 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbd3415eee0 0x7fbd340b0690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:40.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:40.889+0000 7fbd2b400640 1 -- 172.21.15.16:0/743800756 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd3414b2e0 con 0x7fbd3415b3d0 2024-03-31T15:14:40.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:40.889+0000 7fbd33e00640 1 --2- 172.21.15.16:0/743800756 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fbd340b23d0 0x7fbd341f8990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:14:40.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:40.889+0000 7fbd2b400640 1 --2- 172.21.15.16:0/743800756 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbd3415b3d0 0x7fbd341fa710 secure :-1 s=READY pgs=1744 cs=0 l=1 rev1=1 crypto rx=0x7fbd1c00e840 tx=0x7fbd1c00ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:40.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:40.889+0000 7fbd30c00640 1 -- 172.21.15.16:0/743800756 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd1c017070 con 0x7fbd3415b3d0 2024-03-31T15:14:40.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:40.893+0000 7fbd30c00640 1 -- 172.21.15.16:0/743800756 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbd1c00c8d0 con 0x7fbd3415b3d0 2024-03-31T15:14:40.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:40.893+0000 7fbd3ae00640 1 -- 172.21.15.16:0/743800756 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbd3414b550 con 0x7fbd3415b3d0 2024-03-31T15:14:40.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:40.893+0000 7fbd30c00640 1 -- 172.21.15.16:0/743800756 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd1c00f040 con 0x7fbd3415b3d0 2024-03-31T15:14:40.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:40.893+0000 7fbd3ae00640 1 -- 172.21.15.16:0/743800756 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbd34165cb0 con 0x7fbd3415b3d0 2024-03-31T15:14:40.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:40.893+0000 7fbd3ae00640 1 -- 172.21.15.16:0/743800756 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbd34069950 con 0x7fbd3415b3d0 2024-03-31T15:14:40.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:40.893+0000 7fbd30c00640 1 -- 172.21.15.16:0/743800756 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fbd1c01a070 con 0x7fbd3415b3d0 2024-03-31T15:14:40.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:40.893+0000 7fbd30c00640 1 --2- 172.21.15.16:0/743800756 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fbd10068380 0x7fbd1006a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:40.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:40.893+0000 7fbd30c00640 1 -- 172.21.15.16:0/743800756 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(112..112 src has 1..112) ==== 5853+0+0 (secure 0 0 0) 0x7fbd1c086630 con 0x7fbd3415b3d0 2024-03-31T15:14:40.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:40.893+0000 7fbd33e00640 1 --2- 172.21.15.16:0/743800756 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fbd10068380 0x7fbd1006a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:40.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:40.897+0000 7fbd33e00640 1 --2- 172.21.15.16:0/743800756 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fbd10068380 0x7fbd1006a840 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7fbd24000f70 tx=0x7fbd24009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:40.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:40.897+0000 7fbd30c00640 1 -- 172.21.15.16:0/743800756 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fbd1c056500 con 0x7fbd3415b3d0 2024-03-31T15:14:41.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:41.097+0000 7fbd3ae00640 1 -- 172.21.15.16:0/743800756 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbd340cdea0 con 0x7fbd3415b3d0 2024-03-31T15:14:41.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:41.101+0000 7fbd30c00640 1 -- 172.21.15.16:0/743800756 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 102 v102) ==== 95+0+5178 (secure 0 0 0) 0x7fbd1c05a800 con 0x7fbd3415b3d0 2024-03-31T15:14:41.093 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:14:41.093 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":102,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7631,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6836/1628897112","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":1628897112},{"type":"v1","addr":"172.21.15.89:6836","nonce":1628897112}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":94},{"gid":8746,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/1036724837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6835","nonce":1036724837},{"type":"v1","addr":"172.21.15.89:6837","nonce":1036724837}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":94},{"gid":9192,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6834/472139147","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":472139147},{"type":"v1","addr":"172.21.15.16:6834","nonce":472139147}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":94}],"filesystems":[{"mdsmap":{"epoch":102,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-31T15:14:08.652180+0000","modified":"2024-03-31T15:14:40.721492+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":112,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":9558},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9558":{"gid":9558,"name":"c","rank":0,"incarnation":102,"state":"up:replay","state_seq":1,"addr":"172.21.15.16:6835/4142860904","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6833","nonce":4142860904},{"type":"v1","addr":"172.21.15.16:6835","nonce":4142860904}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":8}]} 2024-03-31T15:14:41.094 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 102 2024-03-31T15:14:41.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:41.101+0000 7fbd3ae00640 1 -- 172.21.15.16:0/743800756 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fbd10068380 msgr2=0x7fbd1006a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:41.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:41.101+0000 7fbd3ae00640 1 --2- 172.21.15.16:0/743800756 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fbd10068380 0x7fbd1006a840 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7fbd24000f70 tx=0x7fbd24009040 comp rx=0 tx=0).stop 2024-03-31T15:14:41.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:41.101+0000 7fbd3ae00640 1 -- 172.21.15.16:0/743800756 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbd3415b3d0 msgr2=0x7fbd341fa710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:41.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:41.101+0000 7fbd3ae00640 1 --2- 172.21.15.16:0/743800756 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbd3415b3d0 0x7fbd341fa710 secure :-1 s=READY pgs=1744 cs=0 l=1 rev1=1 crypto rx=0x7fbd1c00e840 tx=0x7fbd1c00ed00 comp rx=0 tx=0).stop 2024-03-31T15:14:41.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:41.101+0000 7fbd3ae00640 1 -- 172.21.15.16:0/743800756 shutdown_connections 2024-03-31T15:14:41.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:41.101+0000 7fbd3ae00640 1 --2- 172.21.15.16:0/743800756 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fbd10068380 0x7fbd1006a840 unknown :-1 s=CLOSED pgs=740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:41.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:41.101+0000 7fbd3ae00640 1 --2- 172.21.15.16:0/743800756 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbd3415eee0 0x7fbd340b0690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:41.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:41.101+0000 7fbd3ae00640 1 --2- 172.21.15.16:0/743800756 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbd3415b3d0 0x7fbd341fa710 unknown :-1 s=CLOSED pgs=1744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:41.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:41.101+0000 7fbd3ae00640 1 --2- 172.21.15.16:0/743800756 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fbd340b23d0 0x7fbd341f8990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:41.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:41.101+0000 7fbd3ae00640 1 -- 172.21.15.16:0/743800756 >> 172.21.15.16:0/743800756 conn(0x7fbd3406b620 msgr2=0x7fbd3415a680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:14:41.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:41.101+0000 7fbd3ae00640 1 -- 172.21.15.16:0/743800756 shutdown_connections 2024-03-31T15:14:41.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:41.101+0000 7fbd3ae00640 1 -- 172.21.15.16:0/743800756 wait complete. 2024-03-31T15:14:41.516 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 102, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-31T15:14:08.652180+0000', 'modified': '2024-03-31T15:14:40.721492+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 112, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 9558}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9558': {'gid': 9558, 'name': 'c', 'rank': 0, 'incarnation': 102, 'state': 'up:replay', 'state_seq': 1, 'addr': '172.21.15.16:6835/4142860904', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6833', 'nonce': 4142860904}, {'type': 'v1', 'addr': '172.21.15.16:6835', 'nonce': 4142860904}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [17], 'metadata_pool': 16, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-03-31T15:14:41.516 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9558:up:replay 2024-03-31T15:14:42.517 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-31T15:14:42.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:42.677+0000 7fb3e1000640 1 Processor -- start 2024-03-31T15:14:42.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:42.677+0000 7fb3e1000640 1 -- start start 2024-03-31T15:14:42.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:42.677+0000 7fb3e1000640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fb3dc0736e0 0x7fb3dc1694a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:42.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:42.677+0000 7fb3e1000640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb3dc06bd30 con 0x7fb3dc147e70 2024-03-31T15:14:42.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:42.677+0000 7fb3e1000640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb3dc1621b0 con 0x7fb3dc0d0c50 2024-03-31T15:14:42.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:42.677+0000 7fb3e1000640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7fb3dc162490 con 0x7fb3dc0736e0 2024-03-31T15:14:42.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:42.677+0000 7fb3d9e00640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fb3dc147e70 0x7fb3dc06b620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:33468/0 (socket says 172.21.15.16:33468) 2024-03-31T15:14:42.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:42.677+0000 7fb3d9e00640 1 -- 172.21.15.16:0/443946515 learned_addr learned my addr 172.21.15.16:0/443946515 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:14:42.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:42.677+0000 7fb3da800640 1 --2- 172.21.15.16:0/443946515 >> v2:172.21.15.89:3300/0 conn(0x7fb3dc0736e0 0x7fb3dc1694a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:42.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:42.677+0000 7fb3d9400640 1 -- 172.21.15.16:0/443946515 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4055139830 0 0) 0x7fb3dc06bd30 con 0x7fb3dc147e70 2024-03-31T15:14:42.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:42.677+0000 7fb3d9400640 1 -- 172.21.15.16:0/443946515 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb3c0003530 con 0x7fb3dc147e70 2024-03-31T15:14:42.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:42.677+0000 7fb3d9400640 1 -- 172.21.15.16:0/443946515 <== mon.1 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3598174187 0 0) 0x7fb3dc1621b0 con 0x7fb3dc0d0c50 2024-03-31T15:14:42.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:42.677+0000 7fb3d9400640 1 -- 172.21.15.16:0/443946515 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb3dc06bd30 con 0x7fb3dc0d0c50 2024-03-31T15:14:42.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:42.677+0000 7fb3d9400640 1 -- 172.21.15.16:0/443946515 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4142411846 0 0) 0x7fb3c0003530 con 0x7fb3dc147e70 2024-03-31T15:14:42.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:42.677+0000 7fb3d9400640 1 -- 172.21.15.16:0/443946515 >> v2:172.21.15.89:3300/0 conn(0x7fb3dc0736e0 msgr2=0x7fb3dc1694a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:14:42.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:42.677+0000 7fb3d9400640 1 --2- 172.21.15.16:0/443946515 >> v2:172.21.15.89:3300/0 conn(0x7fb3dc0736e0 0x7fb3dc1694a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:42.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:42.677+0000 7fb3d9400640 1 -- 172.21.15.16:0/443946515 >> v1:172.21.15.89:6789/0 conn(0x7fb3dc0d0c50 legacy=0x7fb3dc072fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:42.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:42.677+0000 7fb3d9400640 1 -- 172.21.15.16:0/443946515 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3dc162770 con 0x7fb3dc147e70 2024-03-31T15:14:42.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:42.677+0000 7fb3d9400640 1 -- 172.21.15.16:0/443946515 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 214807623 0 0) 0x7fb3cc001680 con 0x7fb3dc147e70 2024-03-31T15:14:42.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:42.677+0000 7fb3d9400640 1 -- 172.21.15.16:0/443946515 >> v1:172.21.15.16:6789/0 conn(0x7fb3dc147e70 legacy=0x7fb3dc06b620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:42.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:42.677+0000 7fb3d9400640 1 --2- 172.21.15.16:0/443946515 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb3c0003b50 0x7fb3c0003f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:42.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:42.677+0000 7fb3d9400640 1 --2- 172.21.15.16:0/443946515 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb3c0004810 0x7fb3c0004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:42.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:42.677+0000 7fb3d9400640 1 --2- 172.21.15.16:0/443946515 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb3c00051d0 0x7fb3c000b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:42.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:42.677+0000 7fb3d9400640 1 -- 172.21.15.16:0/443946515 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fb3dc162490 con 0x7fb3c0004810 2024-03-31T15:14:42.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:42.677+0000 7fb3d9400640 1 -- 172.21.15.16:0/443946515 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fb3c000c050 con 0x7fb3c0003b50 2024-03-31T15:14:42.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:42.677+0000 7fb3d9400640 1 -- 172.21.15.16:0/443946515 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fb3c000c250 con 0x7fb3c00051d0 2024-03-31T15:14:42.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:42.677+0000 7fb3d3e00640 1 --2- 172.21.15.16:0/443946515 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb3c0004810 0x7fb3c0004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:42.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:42.677+0000 7fb3da800640 1 --2- 172.21.15.16:0/443946515 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb3c00051d0 0x7fb3c000b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:42.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:42.677+0000 7fb3d9e00640 1 --2- 172.21.15.16:0/443946515 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb3c0003b50 0x7fb3c0003f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:42.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:42.677+0000 7fb3d3e00640 1 -- 172.21.15.16:0/443946515 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb3c00051d0 msgr2=0x7fb3c000b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:14:42.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:42.677+0000 7fb3d3e00640 1 --2- 172.21.15.16:0/443946515 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb3c00051d0 0x7fb3c000b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:42.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:42.677+0000 7fb3d3e00640 1 -- 172.21.15.16:0/443946515 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb3c0003b50 msgr2=0x7fb3c0003f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:14:42.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:42.677+0000 7fb3d3e00640 1 --2- 172.21.15.16:0/443946515 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb3c0003b50 0x7fb3c0003f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:42.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:42.677+0000 7fb3d3e00640 1 -- 172.21.15.16:0/443946515 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3c0003530 con 0x7fb3c0004810 2024-03-31T15:14:42.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:42.677+0000 7fb3da800640 1 -- 172.21.15.16:0/443946515 reap_dead start 2024-03-31T15:14:42.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:42.681+0000 7fb3d3e00640 1 --2- 172.21.15.16:0/443946515 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb3c0004810 0x7fb3c0004c90 secure :-1 s=READY pgs=1746 cs=0 l=1 rev1=1 crypto rx=0x7fb3dc0d1030 tx=0x7fb3c401ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea4183d43d6959b4 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:42.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:42.681+0000 7fb3d9400640 1 -- 172.21.15.16:0/443946515 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3c4001520 con 0x7fb3c0004810 2024-03-31T15:14:42.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:42.681+0000 7fb3d9400640 1 -- 172.21.15.16:0/443946515 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb3c40052f0 con 0x7fb3c0004810 2024-03-31T15:14:42.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:42.681+0000 7fb3d9400640 1 -- 172.21.15.16:0/443946515 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3c40a88d0 con 0x7fb3c0004810 2024-03-31T15:14:42.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:42.681+0000 7fb3e1000640 1 -- 172.21.15.16:0/443946515 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb3c0004810 msgr2=0x7fb3c0004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:42.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:42.681+0000 7fb3e1000640 1 --2- 172.21.15.16:0/443946515 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb3c0004810 0x7fb3c0004c90 secure :-1 s=READY pgs=1746 cs=0 l=1 rev1=1 crypto rx=0x7fb3dc0d1030 tx=0x7fb3c401ec70 comp rx=0 tx=0).stop 2024-03-31T15:14:42.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:42.681+0000 7fb3e1000640 1 -- 172.21.15.16:0/443946515 shutdown_connections 2024-03-31T15:14:42.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:42.681+0000 7fb3e1000640 1 --2- 172.21.15.16:0/443946515 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb3c0004810 0x7fb3c0004c90 unknown :-1 s=CLOSED pgs=1746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:42.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:42.681+0000 7fb3e1000640 1 -- 172.21.15.16:0/443946515 >> 172.21.15.16:0/443946515 conn(0x7fb3dc0d1060 msgr2=0x7fb3dc0d1470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:14:42.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:42.681+0000 7fb3e1000640 1 -- 172.21.15.16:0/443946515 shutdown_connections 2024-03-31T15:14:42.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:42.681+0000 7fb3e1000640 1 -- 172.21.15.16:0/443946515 wait complete. 2024-03-31T15:14:42.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:42.685+0000 7fb3e1000640 1 Processor -- start 2024-03-31T15:14:42.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:42.685+0000 7fb3e1000640 1 -- start start 2024-03-31T15:14:42.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:42.685+0000 7fb3e1000640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb3c0004810 0x7fb3dc1f76b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:42.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:42.685+0000 7fb3e1000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb3dc139c70 0x7fb3dc15e9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:42.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:42.685+0000 7fb3e1000640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb3dc0aff80 0x7fb3dc1464c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:42.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:42.685+0000 7fb3e1000640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fb3dc162f70 con 0x7fb3dc139c70 2024-03-31T15:14:42.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:42.685+0000 7fb3e1000640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fb3dc06bfa0 con 0x7fb3dc0aff80 2024-03-31T15:14:42.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:42.685+0000 7fb3e1000640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fb3dc072fd0 con 0x7fb3c0004810 2024-03-31T15:14:42.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:42.685+0000 7fb3d3e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb3dc139c70 0x7fb3dc15e9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:42.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:42.685+0000 7fb3d3e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb3dc139c70 0x7fb3dc15e9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40744/0 (socket says 172.21.15.16:40744) 2024-03-31T15:14:42.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:42.685+0000 7fb3d3e00640 1 -- 172.21.15.16:0/1203460599 learned_addr learned my addr 172.21.15.16:0/1203460599 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:14:42.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:42.685+0000 7fb3d9e00640 1 --2- 172.21.15.16:0/1203460599 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb3c0004810 0x7fb3dc1f76b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:42.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:42.685+0000 7fb3da800640 1 --2- 172.21.15.16:0/1203460599 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb3dc0aff80 0x7fb3dc1464c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:42.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:42.685+0000 7fb3d3e00640 1 -- 172.21.15.16:0/1203460599 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb3c0004810 msgr2=0x7fb3dc1f76b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:42.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:42.685+0000 7fb3d3e00640 1 --2- 172.21.15.16:0/1203460599 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb3c0004810 0x7fb3dc1f76b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:42.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:42.685+0000 7fb3d3e00640 1 -- 172.21.15.16:0/1203460599 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb3dc0aff80 msgr2=0x7fb3dc1464c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:42.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:42.685+0000 7fb3d3e00640 1 --2- 172.21.15.16:0/1203460599 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb3dc0aff80 0x7fb3dc1464c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:42.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:42.685+0000 7fb3d3e00640 1 -- 172.21.15.16:0/1203460599 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3dc0aeee0 con 0x7fb3dc139c70 2024-03-31T15:14:42.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:42.685+0000 7fb3d3e00640 1 --2- 172.21.15.16:0/1203460599 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb3dc139c70 0x7fb3dc15e9b0 secure :-1 s=READY pgs=1747 cs=0 l=1 rev1=1 crypto rx=0x7fb3c4000c10 tx=0x7fb3c0005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:42.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:42.685+0000 7fb3d2000640 1 -- 172.21.15.16:0/1203460599 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3c40bd070 con 0x7fb3dc139c70 2024-03-31T15:14:42.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:42.685+0000 7fb3d2000640 1 -- 172.21.15.16:0/1203460599 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb3c40054a0 con 0x7fb3dc139c70 2024-03-31T15:14:42.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:42.685+0000 7fb3d2000640 1 -- 172.21.15.16:0/1203460599 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3c4001500 con 0x7fb3dc139c70 2024-03-31T15:14:42.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:42.685+0000 7fb3e1000640 1 -- 172.21.15.16:0/1203460599 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb3dc0af1c0 con 0x7fb3dc139c70 2024-03-31T15:14:42.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:42.689+0000 7fb3e1000640 1 -- 172.21.15.16:0/1203460599 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb3dc0af780 con 0x7fb3dc139c70 2024-03-31T15:14:42.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:42.689+0000 7fb3e1000640 1 -- 172.21.15.16:0/1203460599 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb3dc069950 con 0x7fb3dc139c70 2024-03-31T15:14:42.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:42.689+0000 7fb3d2000640 1 -- 172.21.15.16:0/1203460599 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fb3c40067e0 con 0x7fb3dc139c70 2024-03-31T15:14:42.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:42.689+0000 7fb3d2000640 1 --2- 172.21.15.16:0/1203460599 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fb3b80683b0 0x7fb3b806a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:42.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:42.689+0000 7fb3d2000640 1 -- 172.21.15.16:0/1203460599 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(112..112 src has 1..112) ==== 5853+0+0 (secure 0 0 0) 0x7fb3c4122f60 con 0x7fb3dc139c70 2024-03-31T15:14:42.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:42.689+0000 7fb3d9e00640 1 --2- 172.21.15.16:0/1203460599 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fb3b80683b0 0x7fb3b806a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:42.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:42.689+0000 7fb3d9e00640 1 --2- 172.21.15.16:0/1203460599 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fb3b80683b0 0x7fb3b806a870 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7fb3cc001060 tx=0x7fb3cc007040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:42.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:42.693+0000 7fb3d2000640 1 -- 172.21.15.16:0/1203460599 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fb3c4018980 con 0x7fb3dc139c70 2024-03-31T15:14:42.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:42.897+0000 7fb3e1000640 1 -- 172.21.15.16:0/1203460599 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb3dc0b2140 con 0x7fb3dc139c70 2024-03-31T15:14:42.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:42.901+0000 7fb3d2000640 1 -- 172.21.15.16:0/1203460599 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 104 v104) ==== 95+0+5178 (secure 0 0 0) 0x7fb3c40f2e30 con 0x7fb3dc139c70 2024-03-31T15:14:42.893 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:14:42.893 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":104,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7631,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6836/1628897112","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":1628897112},{"type":"v1","addr":"172.21.15.89:6836","nonce":1628897112}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":94},{"gid":8746,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/1036724837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6835","nonce":1036724837},{"type":"v1","addr":"172.21.15.89:6837","nonce":1036724837}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":94},{"gid":9192,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6834/472139147","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":472139147},{"type":"v1","addr":"172.21.15.16:6834","nonce":472139147}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":94}],"filesystems":[{"mdsmap":{"epoch":104,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-31T15:14:08.652180+0000","modified":"2024-03-31T15:14:41.727136+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":112,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":9558},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9558":{"gid":9558,"name":"c","rank":0,"incarnation":102,"state":"up:rejoin","state_seq":3,"addr":"172.21.15.16:6835/4142860904","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6833","nonce":4142860904},{"type":"v1","addr":"172.21.15.16:6835","nonce":4142860904}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":8}]} 2024-03-31T15:14:42.893 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 104 2024-03-31T15:14:42.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:42.901+0000 7fb3e1000640 1 -- 172.21.15.16:0/1203460599 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fb3b80683b0 msgr2=0x7fb3b806a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:42.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:42.901+0000 7fb3e1000640 1 --2- 172.21.15.16:0/1203460599 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fb3b80683b0 0x7fb3b806a870 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7fb3cc001060 tx=0x7fb3cc007040 comp rx=0 tx=0).stop 2024-03-31T15:14:42.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:42.901+0000 7fb3e1000640 1 -- 172.21.15.16:0/1203460599 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb3dc139c70 msgr2=0x7fb3dc15e9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:42.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:42.901+0000 7fb3e1000640 1 --2- 172.21.15.16:0/1203460599 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb3dc139c70 0x7fb3dc15e9b0 secure :-1 s=READY pgs=1747 cs=0 l=1 rev1=1 crypto rx=0x7fb3c4000c10 tx=0x7fb3c0005840 comp rx=0 tx=0).stop 2024-03-31T15:14:42.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:42.901+0000 7fb3e1000640 1 -- 172.21.15.16:0/1203460599 shutdown_connections 2024-03-31T15:14:42.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:42.901+0000 7fb3e1000640 1 --2- 172.21.15.16:0/1203460599 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fb3b80683b0 0x7fb3b806a870 unknown :-1 s=CLOSED pgs=741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:42.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:42.901+0000 7fb3e1000640 1 --2- 172.21.15.16:0/1203460599 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb3dc0aff80 0x7fb3dc1464c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:42.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:42.901+0000 7fb3e1000640 1 --2- 172.21.15.16:0/1203460599 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb3dc139c70 0x7fb3dc15e9b0 unknown :-1 s=CLOSED pgs=1747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:42.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:42.901+0000 7fb3e1000640 1 --2- 172.21.15.16:0/1203460599 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb3c0004810 0x7fb3dc1f76b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:42.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:42.901+0000 7fb3e1000640 1 -- 172.21.15.16:0/1203460599 >> 172.21.15.16:0/1203460599 conn(0x7fb3dc0d1060 msgr2=0x7fb3dc06b620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:14:42.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:42.901+0000 7fb3e1000640 1 -- 172.21.15.16:0/1203460599 shutdown_connections 2024-03-31T15:14:42.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:42.901+0000 7fb3e1000640 1 -- 172.21.15.16:0/1203460599 wait complete. 2024-03-31T15:14:43.316 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 104, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-31T15:14:08.652180+0000', 'modified': '2024-03-31T15:14:41.727136+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 112, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 9558}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9558': {'gid': 9558, 'name': 'c', 'rank': 0, 'incarnation': 102, 'state': 'up:rejoin', 'state_seq': 3, 'addr': '172.21.15.16:6835/4142860904', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6833', 'nonce': 4142860904}, {'type': 'v1', 'addr': '172.21.15.16:6835', 'nonce': 4142860904}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [17], 'metadata_pool': 16, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-03-31T15:14:43.316 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9558:up:rejoin 2024-03-31T15:14:44.317 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-31T15:14:44.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:44.465+0000 7f07efa00640 1 Processor -- start 2024-03-31T15:14:44.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:44.465+0000 7f07efa00640 1 -- start start 2024-03-31T15:14:44.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:44.465+0000 7f07efa00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f07e806f920 0x7f07e806fd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:44.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:44.465+0000 7f07efa00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f07e80d2010 0x7f07e80d24a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:44.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:44.465+0000 7f07efa00640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f07e80d0920 con 0x7f07e8145e40 2024-03-31T15:14:44.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:44.465+0000 7f07efa00640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f07e80702c0 con 0x7f07e80d2010 2024-03-31T15:14:44.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:44.465+0000 7f07efa00640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f07e80d06d0 con 0x7f07e806f920 2024-03-31T15:14:44.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:44.465+0000 7f07ed000640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f07e8145e40 0x7f07e80cc740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:33472/0 (socket says 172.21.15.16:33472) 2024-03-31T15:14:44.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:44.465+0000 7f07ed000640 1 -- 172.21.15.16:0/3443239911 learned_addr learned my addr 172.21.15.16:0/3443239911 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:14:44.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:44.465+0000 7f07eda00640 1 --2- 172.21.15.16:0/3443239911 >> v2:172.21.15.16:3300/0 conn(0x7f07e80d2010 0x7f07e80d24a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:44.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:44.465+0000 7f07e7e00640 1 --2- 172.21.15.16:0/3443239911 >> v2:172.21.15.89:3300/0 conn(0x7f07e806f920 0x7f07e806fd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:44.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:44.465+0000 7f07e7400640 1 -- 172.21.15.16:0/3443239911 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2452820453 0 0) 0x7f07e80d0920 con 0x7f07e8145e40 2024-03-31T15:14:44.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:44.465+0000 7f07eda00640 1 -- 172.21.15.16:0/3443239911 >> v2:172.21.15.89:3300/0 conn(0x7f07e806f920 msgr2=0x7f07e806fd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:14:44.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:44.465+0000 7f07eda00640 1 --2- 172.21.15.16:0/3443239911 >> v2:172.21.15.89:3300/0 conn(0x7f07e806f920 0x7f07e806fd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:44.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:44.465+0000 7f07eda00640 1 -- 172.21.15.16:0/3443239911 >> v1:172.21.15.16:6789/0 conn(0x7f07e8145e40 legacy=0x7f07e80cc740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:44.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:44.465+0000 7f07eda00640 1 -- 172.21.15.16:0/3443239911 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07e80d0c00 con 0x7f07e80d2010 2024-03-31T15:14:44.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:44.465+0000 7f07e7e00640 1 --2- 172.21.15.16:0/3443239911 >> v2:172.21.15.89:3300/0 conn(0x7f07e806f920 0x7f07e806fd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:14:44.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:44.469+0000 7f07eda00640 1 --2- 172.21.15.16:0/3443239911 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f07e80d2010 0x7f07e80d24a0 secure :-1 s=READY pgs=1749 cs=0 l=1 rev1=1 crypto rx=0x7f07e8073cf0 tx=0x7f07e001c930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf79a41338228bcc server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:44.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:44.469+0000 7f07e7400640 1 -- 172.21.15.16:0/3443239911 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07e00b2070 con 0x7f07e80d2010 2024-03-31T15:14:44.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:44.469+0000 7f07e7400640 1 -- 172.21.15.16:0/3443239911 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f07e00a58e0 con 0x7f07e80d2010 2024-03-31T15:14:44.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:44.469+0000 7f07e7400640 1 -- 172.21.15.16:0/3443239911 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07e00a5cb0 con 0x7f07e80d2010 2024-03-31T15:14:44.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:44.469+0000 7f07efa00640 1 -- 172.21.15.16:0/3443239911 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f07e80d2010 msgr2=0x7f07e80d24a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:44.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:44.469+0000 7f07efa00640 1 --2- 172.21.15.16:0/3443239911 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f07e80d2010 0x7f07e80d24a0 secure :-1 s=READY pgs=1749 cs=0 l=1 rev1=1 crypto rx=0x7f07e8073cf0 tx=0x7f07e001c930 comp rx=0 tx=0).stop 2024-03-31T15:14:44.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:44.469+0000 7f07efa00640 1 -- 172.21.15.16:0/3443239911 shutdown_connections 2024-03-31T15:14:44.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:44.469+0000 7f07efa00640 1 --2- 172.21.15.16:0/3443239911 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f07e80d2010 0x7f07e80d24a0 unknown :-1 s=CLOSED pgs=1749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:44.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:44.469+0000 7f07efa00640 1 --2- 172.21.15.16:0/3443239911 >> v2:172.21.15.89:3300/0 conn(0x7f07e806f920 0x7f07e806fd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:44.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:44.469+0000 7f07efa00640 1 -- 172.21.15.16:0/3443239911 >> 172.21.15.16:0/3443239911 conn(0x7f07e8067330 msgr2=0x7f07e8067740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:14:44.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:44.469+0000 7f07efa00640 1 -- 172.21.15.16:0/3443239911 shutdown_connections 2024-03-31T15:14:44.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:44.469+0000 7f07efa00640 1 -- 172.21.15.16:0/3443239911 wait complete. 2024-03-31T15:14:44.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:44.469+0000 7f07efa00640 1 Processor -- start 2024-03-31T15:14:44.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:44.473+0000 7f07efa00640 1 -- start start 2024-03-31T15:14:44.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:44.473+0000 7f07efa00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f07e806f920 0x7f07e80c7030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:44.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:44.473+0000 7f07efa00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f07e80d2010 0x7f07e815b000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:44.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:44.473+0000 7f07efa00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f07e8145e40 0x7f07e8157df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:44.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:44.473+0000 7f07efa00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f07e80d0ee0 con 0x7f07e80d2010 2024-03-31T15:14:44.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:44.473+0000 7f07efa00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f07e80ce9e0 con 0x7f07e8145e40 2024-03-31T15:14:44.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:44.473+0000 7f07efa00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f07e80d2df0 con 0x7f07e806f920 2024-03-31T15:14:44.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:44.473+0000 7f07e7e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f07e80d2010 0x7f07e815b000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:44.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:44.473+0000 7f07e7e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f07e80d2010 0x7f07e815b000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40756/0 (socket says 172.21.15.16:40756) 2024-03-31T15:14:44.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:44.473+0000 7f07e7e00640 1 -- 172.21.15.16:0/646628720 learned_addr learned my addr 172.21.15.16:0/646628720 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:14:44.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:44.473+0000 7f07ed000640 1 --2- 172.21.15.16:0/646628720 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f07e806f920 0x7f07e80c7030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:44.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:44.473+0000 7f07eda00640 1 --2- 172.21.15.16:0/646628720 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f07e8145e40 0x7f07e8157df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:44.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:44.473+0000 7f07e7e00640 1 -- 172.21.15.16:0/646628720 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f07e806f920 msgr2=0x7f07e80c7030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:44.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:44.473+0000 7f07e7e00640 1 --2- 172.21.15.16:0/646628720 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f07e806f920 0x7f07e80c7030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:44.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:44.473+0000 7f07e7e00640 1 -- 172.21.15.16:0/646628720 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f07e8145e40 msgr2=0x7f07e8157df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:44.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:44.473+0000 7f07e7e00640 1 --2- 172.21.15.16:0/646628720 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f07e8145e40 0x7f07e8157df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:44.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:44.473+0000 7f07e7e00640 1 -- 172.21.15.16:0/646628720 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07e80c7650 con 0x7f07e80d2010 2024-03-31T15:14:44.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:44.473+0000 7f07e7e00640 1 --2- 172.21.15.16:0/646628720 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f07e80d2010 0x7f07e815b000 secure :-1 s=READY pgs=1750 cs=0 l=1 rev1=1 crypto rx=0x7f07d400e810 tx=0x7f07d400ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:44.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:44.473+0000 7f07e4c00640 1 -- 172.21.15.16:0/646628720 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07d4017070 con 0x7f07e80d2010 2024-03-31T15:14:44.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:44.473+0000 7f07e4c00640 1 -- 172.21.15.16:0/646628720 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f07d400c8d0 con 0x7f07e80d2010 2024-03-31T15:14:44.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:44.473+0000 7f07efa00640 1 -- 172.21.15.16:0/646628720 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f07e80c7930 con 0x7f07e80d2010 2024-03-31T15:14:44.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:44.473+0000 7f07e4c00640 1 -- 172.21.15.16:0/646628720 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07d400f040 con 0x7f07e80d2010 2024-03-31T15:14:44.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:44.473+0000 7f07efa00640 1 -- 172.21.15.16:0/646628720 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f07e80c7ef0 con 0x7f07e80d2010 2024-03-31T15:14:44.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:44.473+0000 7f07efa00640 1 -- 172.21.15.16:0/646628720 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f07e8069950 con 0x7f07e80d2010 2024-03-31T15:14:44.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:44.477+0000 7f07e4c00640 1 -- 172.21.15.16:0/646628720 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f07d401a070 con 0x7f07e80d2010 2024-03-31T15:14:44.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:44.477+0000 7f07e4c00640 1 --2- 172.21.15.16:0/646628720 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f07c8068380 0x7f07c806a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:44.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:44.477+0000 7f07e4c00640 1 -- 172.21.15.16:0/646628720 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(112..112 src has 1..112) ==== 5853+0+0 (secure 0 0 0) 0x7f07d4086600 con 0x7f07e80d2010 2024-03-31T15:14:44.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:44.477+0000 7f07ed000640 1 --2- 172.21.15.16:0/646628720 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f07c8068380 0x7f07c806a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:44.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:44.477+0000 7f07ed000640 1 --2- 172.21.15.16:0/646628720 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f07c8068380 0x7f07c806a840 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f07dc000f70 tx=0x7f07dc007df0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:44.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:44.481+0000 7f07e4c00640 1 -- 172.21.15.16:0/646628720 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f07d40564d0 con 0x7f07e80d2010 2024-03-31T15:14:44.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:44.681+0000 7f07efa00640 1 -- 172.21.15.16:0/646628720 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f07e80b2850 con 0x7f07e80d2010 2024-03-31T15:14:44.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:44.681+0000 7f07e4c00640 1 -- 172.21.15.16:0/646628720 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 105 v105) ==== 95+0+5185 (secure 0 0 0) 0x7f07d4084e20 con 0x7f07e80d2010 2024-03-31T15:14:44.675 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:14:44.675 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":105,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7631,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6836/1628897112","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":1628897112},{"type":"v1","addr":"172.21.15.89:6836","nonce":1628897112}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":94},{"gid":8746,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/1036724837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6835","nonce":1036724837},{"type":"v1","addr":"172.21.15.89:6837","nonce":1036724837}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":94},{"gid":9192,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6834/472139147","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":472139147},{"type":"v1","addr":"172.21.15.16:6834","nonce":472139147}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":94}],"filesystems":[{"mdsmap":{"epoch":105,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-31T15:14:08.652180+0000","modified":"2024-03-31T15:14:43.727451+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":112,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":9558},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9558":{"gid":9558,"name":"c","rank":0,"incarnation":102,"state":"up:active","state_seq":4,"addr":"172.21.15.16:6835/4142860904","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6833","nonce":4142860904},{"type":"v1","addr":"172.21.15.16:6835","nonce":4142860904}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9558,"qdb_cluster":[9558]},"id":8}]} 2024-03-31T15:14:44.675 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 105 2024-03-31T15:14:44.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:44.685+0000 7f07efa00640 1 -- 172.21.15.16:0/646628720 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f07c8068380 msgr2=0x7f07c806a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:44.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:44.685+0000 7f07efa00640 1 --2- 172.21.15.16:0/646628720 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f07c8068380 0x7f07c806a840 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f07dc000f70 tx=0x7f07dc007df0 comp rx=0 tx=0).stop 2024-03-31T15:14:44.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:44.685+0000 7f07efa00640 1 -- 172.21.15.16:0/646628720 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f07e80d2010 msgr2=0x7f07e815b000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:44.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:44.685+0000 7f07efa00640 1 --2- 172.21.15.16:0/646628720 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f07e80d2010 0x7f07e815b000 secure :-1 s=READY pgs=1750 cs=0 l=1 rev1=1 crypto rx=0x7f07d400e810 tx=0x7f07d400ecd0 comp rx=0 tx=0).stop 2024-03-31T15:14:44.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:44.685+0000 7f07efa00640 1 -- 172.21.15.16:0/646628720 shutdown_connections 2024-03-31T15:14:44.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:44.685+0000 7f07efa00640 1 --2- 172.21.15.16:0/646628720 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f07c8068380 0x7f07c806a840 unknown :-1 s=CLOSED pgs=742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:44.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:44.685+0000 7f07efa00640 1 --2- 172.21.15.16:0/646628720 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f07e8145e40 0x7f07e8157df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:44.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:44.685+0000 7f07efa00640 1 --2- 172.21.15.16:0/646628720 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f07e80d2010 0x7f07e815b000 unknown :-1 s=CLOSED pgs=1750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:44.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:44.685+0000 7f07efa00640 1 --2- 172.21.15.16:0/646628720 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f07e806f920 0x7f07e80c7030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:44.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:44.685+0000 7f07efa00640 1 -- 172.21.15.16:0/646628720 >> 172.21.15.16:0/646628720 conn(0x7f07e8067330 msgr2=0x7f07e815f500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:14:44.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:44.685+0000 7f07efa00640 1 -- 172.21.15.16:0/646628720 shutdown_connections 2024-03-31T15:14:44.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:44.685+0000 7f07efa00640 1 -- 172.21.15.16:0/646628720 wait complete. 2024-03-31T15:14:45.108 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 105, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-31T15:14:08.652180+0000', 'modified': '2024-03-31T15:14:43.727451+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 112, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 9558}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9558': {'gid': 9558, 'name': 'c', 'rank': 0, 'incarnation': 102, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.16:6835/4142860904', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6833', 'nonce': 4142860904}, {'type': 'v1', 'addr': '172.21.15.16:6835', 'nonce': 4142860904}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [17], 'metadata_pool': 16, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9558, 'qdb_cluster': [9558]} 2024-03-31T15:14:45.108 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-31T15:14:45.108 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-03-31T15:14:45.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.257+0000 7f4b1da00640 1 Processor -- start 2024-03-31T15:14:45.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.257+0000 7f4b1da00640 1 -- start start 2024-03-31T15:14:45.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.261+0000 7f4b1da00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f4b18158700 0x7f4b180706d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:45.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.261+0000 7f4b1da00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f4b1806e4c0 0x7f4b1815f120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:45.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.261+0000 7f4b1da00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f4b1806b620 0x7f4b1806ba90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:45.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.261+0000 7f4b1da00640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f4b1806bfd0 con 0x7f4b1806e4c0 2024-03-31T15:14:45.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.261+0000 7f4b1da00640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f4b1815ec90 con 0x7f4b1806b620 2024-03-31T15:14:45.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.261+0000 7f4b1da00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f4b1806ec30 con 0x7f4b18158700 2024-03-31T15:14:45.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.261+0000 7f4b0fe00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f4b1806e4c0 0x7f4b1815f120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:45.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.261+0000 7f4b0fe00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f4b1806e4c0 0x7f4b1815f120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40760/0 (socket says 172.21.15.16:40760) 2024-03-31T15:14:45.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.261+0000 7f4b0fe00640 1 -- 172.21.15.16:0/2815178773 learned_addr learned my addr 172.21.15.16:0/2815178773 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:14:45.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.261+0000 7f4b17e00640 1 --2- 172.21.15.16:0/2815178773 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f4b18158700 0x7f4b180706d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:45.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.261+0000 7f4b1d000640 1 --2- 172.21.15.16:0/2815178773 >> v2:172.21.15.89:3300/0 conn(0x7f4b1806b620 0x7f4b1806ba90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:45.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.261+0000 7f4b0fe00640 1 -- 172.21.15.16:0/2815178773 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f4b18158700 msgr2=0x7f4b180706d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:14:45.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.261+0000 7f4b0fe00640 1 --2- 172.21.15.16:0/2815178773 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f4b18158700 0x7f4b180706d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:45.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.261+0000 7f4b0fe00640 1 -- 172.21.15.16:0/2815178773 >> v2:172.21.15.89:3300/0 conn(0x7f4b1806b620 msgr2=0x7f4b1806ba90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:14:45.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.261+0000 7f4b0fe00640 1 --2- 172.21.15.16:0/2815178773 >> v2:172.21.15.89:3300/0 conn(0x7f4b1806b620 0x7f4b1806ba90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:45.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.261+0000 7f4b0fe00640 1 -- 172.21.15.16:0/2815178773 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b1806ef10 con 0x7f4b1806e4c0 2024-03-31T15:14:45.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.265+0000 7f4b0fe00640 1 --2- 172.21.15.16:0/2815178773 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b1806e4c0 0x7f4b1815f120 secure :-1 s=READY pgs=1751 cs=0 l=1 rev1=1 crypto rx=0x7f4b18072220 tx=0x7f4b0001c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78d148540483a874 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:45.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.265+0000 7f4b17400640 1 -- 172.21.15.16:0/2815178773 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b000b0070 con 0x7f4b1806e4c0 2024-03-31T15:14:45.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.265+0000 7f4b17400640 1 -- 172.21.15.16:0/2815178773 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4b000ab600 con 0x7f4b1806e4c0 2024-03-31T15:14:45.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.265+0000 7f4b17400640 1 -- 172.21.15.16:0/2815178773 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b000ab9d0 con 0x7f4b1806e4c0 2024-03-31T15:14:45.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.265+0000 7f4b1da00640 1 -- 172.21.15.16:0/2815178773 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b1806e4c0 msgr2=0x7f4b1815f120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:45.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.265+0000 7f4b1da00640 1 --2- 172.21.15.16:0/2815178773 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b1806e4c0 0x7f4b1815f120 secure :-1 s=READY pgs=1751 cs=0 l=1 rev1=1 crypto rx=0x7f4b18072220 tx=0x7f4b0001c520 comp rx=0 tx=0).stop 2024-03-31T15:14:45.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.265+0000 7f4b1da00640 1 -- 172.21.15.16:0/2815178773 shutdown_connections 2024-03-31T15:14:45.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.265+0000 7f4b1da00640 1 --2- 172.21.15.16:0/2815178773 >> v2:172.21.15.89:3300/0 conn(0x7f4b1806b620 0x7f4b1806ba90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:45.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.265+0000 7f4b1da00640 1 --2- 172.21.15.16:0/2815178773 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b1806e4c0 0x7f4b1815f120 unknown :-1 s=CLOSED pgs=1751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:45.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.265+0000 7f4b1da00640 1 --2- 172.21.15.16:0/2815178773 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f4b18158700 0x7f4b180706d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:45.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.265+0000 7f4b1da00640 1 -- 172.21.15.16:0/2815178773 >> 172.21.15.16:0/2815178773 conn(0x7f4b180b2b30 msgr2=0x7f4b180b2f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:14:45.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.265+0000 7f4b1da00640 1 -- 172.21.15.16:0/2815178773 shutdown_connections 2024-03-31T15:14:45.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.265+0000 7f4b1da00640 1 -- 172.21.15.16:0/2815178773 wait complete. 2024-03-31T15:14:45.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.265+0000 7f4b1da00640 1 Processor -- start 2024-03-31T15:14:45.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.265+0000 7f4b1da00640 1 -- start start 2024-03-31T15:14:45.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.269+0000 7f4b1da00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b1806b620 0x7f4b1814b270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:45.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.269+0000 7f4b1da00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f4b1806e4c0 0x7f4b180b16e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:45.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.269+0000 7f4b1da00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4b18158700 0x7f4b180b1c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:45.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.269+0000 7f4b1da00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f4b180d0980 con 0x7f4b1806b620 2024-03-31T15:14:45.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.269+0000 7f4b1da00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f4b18158ae0 con 0x7f4b18158700 2024-03-31T15:14:45.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.269+0000 7f4b1da00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f4b180d0b00 con 0x7f4b1806e4c0 2024-03-31T15:14:45.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.269+0000 7f4b17e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b1806b620 0x7f4b1814b270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:45.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.269+0000 7f4b17e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b1806b620 0x7f4b1814b270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40768/0 (socket says 172.21.15.16:40768) 2024-03-31T15:14:45.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.269+0000 7f4b17e00640 1 -- 172.21.15.16:0/2573198869 learned_addr learned my addr 172.21.15.16:0/2573198869 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:14:45.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.269+0000 7f4b1d000640 1 --2- 172.21.15.16:0/2573198869 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4b18158700 0x7f4b180b1c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:45.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.269+0000 7f4b0fe00640 1 --2- 172.21.15.16:0/2573198869 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f4b1806e4c0 0x7f4b180b16e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:45.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.269+0000 7f4b17e00640 1 -- 172.21.15.16:0/2573198869 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f4b1806e4c0 msgr2=0x7f4b180b16e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:45.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.269+0000 7f4b17e00640 1 --2- 172.21.15.16:0/2573198869 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f4b1806e4c0 0x7f4b180b16e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:45.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.269+0000 7f4b17e00640 1 -- 172.21.15.16:0/2573198869 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4b18158700 msgr2=0x7f4b180b1c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:45.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.269+0000 7f4b17e00640 1 --2- 172.21.15.16:0/2573198869 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4b18158700 0x7f4b180b1c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:45.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.269+0000 7f4b17e00640 1 -- 172.21.15.16:0/2573198869 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b1815a680 con 0x7f4b1806b620 2024-03-31T15:14:45.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.269+0000 7f4b0fe00640 1 --2- 172.21.15.16:0/2573198869 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f4b1806e4c0 0x7f4b180b16e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:14:45.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.269+0000 7f4b17e00640 1 --2- 172.21.15.16:0/2573198869 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b1806b620 0x7f4b1814b270 secure :-1 s=READY pgs=1752 cs=0 l=1 rev1=1 crypto rx=0x7f4b0800d800 tx=0x7f4b0800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:45.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.269+0000 7f4b14c00640 1 -- 172.21.15.16:0/2573198869 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b0800b6c0 con 0x7f4b1806b620 2024-03-31T15:14:45.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.269+0000 7f4b14c00640 1 -- 172.21.15.16:0/2573198869 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4b08019070 con 0x7f4b1806b620 2024-03-31T15:14:45.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.269+0000 7f4b1da00640 1 -- 172.21.15.16:0/2573198869 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4b181facc0 con 0x7f4b1806b620 2024-03-31T15:14:45.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.269+0000 7f4b14c00640 1 -- 172.21.15.16:0/2573198869 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b0800ba60 con 0x7f4b1806b620 2024-03-31T15:14:45.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.269+0000 7f4b1da00640 1 -- 172.21.15.16:0/2573198869 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4b181fb7a0 con 0x7f4b1806b620 2024-03-31T15:14:45.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.269+0000 7f4b14c00640 1 -- 172.21.15.16:0/2573198869 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f4b0800bc00 con 0x7f4b1806b620 2024-03-31T15:14:45.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.273+0000 7f4b14c00640 1 --2- 172.21.15.16:0/2573198869 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f4af4068380 0x7f4af406a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:45.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.273+0000 7f4b14c00640 1 -- 172.21.15.16:0/2573198869 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(112..112 src has 1..112) ==== 5853+0+0 (secure 0 0 0) 0x7f4b08086eb0 con 0x7f4b1806b620 2024-03-31T15:14:45.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.273+0000 7f4b0fe00640 1 --2- 172.21.15.16:0/2573198869 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f4af4068380 0x7f4af406a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:45.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.273+0000 7f4b22493000 1 Processor -- start 2024-03-31T15:14:45.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.273+0000 7f4b22493000 1 -- start start 2024-03-31T15:14:45.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.273+0000 7f4b0fe00640 1 --2- 172.21.15.16:0/2573198869 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f4af4068380 0x7f4af406a840 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f4b000ad000 tx=0x7f4b00008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:45.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.273+0000 7f4b22493000 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563bb604b4a0 0x563bb604b880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:45.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.273+0000 7f4b22493000 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x563bb604bdc0 0x563bb5ffeb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:45.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.273+0000 7f4b22493000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563bb5fff0b0 0x563bb5fff490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:45.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.273+0000 7f4b22493000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x563bb5e51950 con 0x563bb5fff0b0 2024-03-31T15:14:45.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.273+0000 7f4b22493000 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x563bb5dee990 con 0x563bb604b4a0 2024-03-31T15:14:45.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.273+0000 7f4b22493000 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x563bb5e4d900 con 0x563bb604bdc0 2024-03-31T15:14:45.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.273+0000 7f4b1d000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563bb5fff0b0 0x563bb5fff490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:45.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.273+0000 7f4b17e00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563bb604b4a0 0x563bb604b880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:45.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.273+0000 7f4b1d000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563bb5fff0b0 0x563bb5fff490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40778/0 (socket says 172.21.15.16:40778) 2024-03-31T15:14:45.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.273+0000 7f4b1d000640 1 -- 172.21.15.16:0/3457294431 learned_addr learned my addr 172.21.15.16:0/3457294431 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:14:45.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.273+0000 7f4b0fe00640 1 --2- 172.21.15.16:0/3457294431 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x563bb604bdc0 0x563bb5ffeb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:45.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.273+0000 7f4b1d000640 1 -- 172.21.15.16:0/3457294431 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x563bb604bdc0 msgr2=0x563bb5ffeb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:14:45.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.273+0000 7f4b1d000640 1 --2- 172.21.15.16:0/3457294431 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x563bb604bdc0 0x563bb5ffeb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:45.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.273+0000 7f4b1d000640 1 -- 172.21.15.16:0/3457294431 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563bb604b4a0 msgr2=0x563bb604b880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:14:45.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.273+0000 7f4b1d000640 1 --2- 172.21.15.16:0/3457294431 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563bb604b4a0 0x563bb604b880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:45.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.273+0000 7f4b1d000640 1 -- 172.21.15.16:0/3457294431 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563bb5e96100 con 0x563bb5fff0b0 2024-03-31T15:14:45.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.273+0000 7f4b17e00640 1 --2- 172.21.15.16:0/3457294431 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563bb604b4a0 0x563bb604b880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:14:45.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.273+0000 7f4b1d000640 1 --2- 172.21.15.16:0/3457294431 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563bb5fff0b0 0x563bb5fff490 secure :-1 s=READY pgs=1753 cs=0 l=1 rev1=1 crypto rx=0x7f4b10007410 tx=0x7f4b100078d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12b4a8a0da837661 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:45.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.273+0000 7f4b0cc00640 1 -- 172.21.15.16:0/3457294431 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b100147e0 con 0x563bb5fff0b0 2024-03-31T15:14:45.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.273+0000 7f4b0cc00640 1 -- 172.21.15.16:0/3457294431 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4b1001a070 con 0x563bb5fff0b0 2024-03-31T15:14:45.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.273+0000 7f4b0cc00640 1 -- 172.21.15.16:0/3457294431 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b10014b80 con 0x563bb5fff0b0 2024-03-31T15:14:45.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.277+0000 7f4b22493000 1 -- 172.21.15.16:0/3457294431 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563bb5fff0b0 msgr2=0x563bb5fff490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:45.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.277+0000 7f4b22493000 1 --2- 172.21.15.16:0/3457294431 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563bb5fff0b0 0x563bb5fff490 secure :-1 s=READY pgs=1753 cs=0 l=1 rev1=1 crypto rx=0x7f4b10007410 tx=0x7f4b100078d0 comp rx=0 tx=0).stop 2024-03-31T15:14:45.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.277+0000 7f4b22493000 1 -- 172.21.15.16:0/3457294431 shutdown_connections 2024-03-31T15:14:45.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.277+0000 7f4b22493000 1 --2- 172.21.15.16:0/3457294431 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563bb5fff0b0 0x563bb5fff490 unknown :-1 s=CLOSED pgs=1753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:45.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.277+0000 7f4b22493000 1 --2- 172.21.15.16:0/3457294431 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x563bb604bdc0 0x563bb5ffeb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:45.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.277+0000 7f4b22493000 1 --2- 172.21.15.16:0/3457294431 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563bb604b4a0 0x563bb604b880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:45.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.277+0000 7f4b22493000 1 -- 172.21.15.16:0/3457294431 >> 172.21.15.16:0/3457294431 conn(0x563bb5ee9b80 msgr2=0x563bb602e460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:14:45.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.277+0000 7f4b22493000 1 -- 172.21.15.16:0/3457294431 shutdown_connections 2024-03-31T15:14:45.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.277+0000 7f4b22493000 1 -- 172.21.15.16:0/3457294431 wait complete. 2024-03-31T15:14:45.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.277+0000 7f4b22493000 1 -- start start 2024-03-31T15:14:45.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.277+0000 7f4b22493000 1 Processor -- start 2024-03-31T15:14:45.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.277+0000 7f4b22493000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563bb5fff0b0 0x563bb60536e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:45.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.277+0000 7f4b22493000 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x563bb604b4a0 0x563bb6053c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:45.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.277+0000 7f4b22493000 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563bb604bdc0 0x563bb6019d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:45.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.277+0000 7f4b22493000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x563bb5e50c60 con 0x563bb5fff0b0 2024-03-31T15:14:45.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.277+0000 7f4b22493000 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x563bb5e0f3e0 con 0x563bb604bdc0 2024-03-31T15:14:45.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.277+0000 7f4b22493000 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x563bb5e5e060 con 0x563bb604b4a0 2024-03-31T15:14:45.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.277+0000 7f4b22493000 10 client.0 _ll_register_callbacks cb 0x563bb5eb20f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T15:14:45.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.277+0000 7f4b17e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563bb5fff0b0 0x563bb60536e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:45.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.277+0000 7f4b17e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563bb5fff0b0 0x563bb60536e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40780/0 (socket says 172.21.15.16:40780) 2024-03-31T15:14:45.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.277+0000 7f4b17e00640 1 -- 172.21.15.16:0/3084705189 learned_addr learned my addr 172.21.15.16:0/3084705189 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:14:45.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.277+0000 7f4b0fe00640 1 --2- 172.21.15.16:0/3084705189 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x563bb604b4a0 0x563bb6053c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:45.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.277+0000 7f4b1d000640 1 --2- 172.21.15.16:0/3084705189 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563bb604bdc0 0x563bb6019d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:45.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.277+0000 7f4b17e00640 1 -- 172.21.15.16:0/3084705189 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x563bb604b4a0 msgr2=0x563bb6053c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:14:45.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.277+0000 7f4b17e00640 1 --2- 172.21.15.16:0/3084705189 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x563bb604b4a0 0x563bb6053c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:45.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.277+0000 7f4b17e00640 1 -- 172.21.15.16:0/3084705189 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563bb604bdc0 msgr2=0x563bb6019d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:14:45.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.277+0000 7f4b17e00640 1 --2- 172.21.15.16:0/3084705189 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563bb604bdc0 0x563bb6019d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:45.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.277+0000 7f4b17e00640 1 -- 172.21.15.16:0/3084705189 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563bb5e1b8c0 con 0x563bb5fff0b0 2024-03-31T15:14:45.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.277+0000 7f4b17e00640 1 --2- 172.21.15.16:0/3084705189 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563bb5fff0b0 0x563bb60536e0 secure :-1 s=READY pgs=1754 cs=0 l=1 rev1=1 crypto rx=0x7f4b0800b4b0 tx=0x7f4b08056d90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d45010d857c7125a server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:45.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.277+0000 7f4b05600640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-31T15:14:45.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.277+0000 7f4b05600640 1 -- 172.21.15.16:0/3084705189 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b08053340 con 0x563bb5fff0b0 2024-03-31T15:14:45.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.277+0000 7f4b05600640 1 -- 172.21.15.16:0/3084705189 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(112..112 src has 1..112) ==== 5853+0+0 (secure 0 0 0) 0x7f4b0806f070 con 0x563bb5fff0b0 2024-03-31T15:14:45.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.277+0000 7f4b22493000 1 -- 172.21.15.16:0/3084705189 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4b1001a070 con 0x563bb5fff0b0 2024-03-31T15:14:45.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.277+0000 7f4b05600640 1 -- 172.21.15.16:0/3084705189 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=105) ==== 24+0+0 (secure 0 0 0) 0x7f4b08074050 con 0x563bb5fff0b0 2024-03-31T15:14:45.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.277+0000 7f4b22493000 10 client.9606 fetch_fsmap learned FSMap version 105 2024-03-31T15:14:45.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.277+0000 7f4b22493000 1 -- 172.21.15.16:0/3084705189 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=105,monmap=2+}) -- 0x563bb5dd9e80 con 0x563bb5fff0b0 2024-03-31T15:14:45.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.277+0000 7f4b05600640 1 -- 172.21.15.16:0/3084705189 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4b1001a070 con 0x563bb5fff0b0 2024-03-31T15:14:45.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.277+0000 7f4b05600640 1 -- 172.21.15.16:0/3084705189 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 105) ==== 3494+0+0 (secure 0 0 0) 0x7f4b08013070 con 0x563bb5fff0b0 2024-03-31T15:14:45.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.277+0000 7f4b22493000 10 client.9606 fetch_fsmap finished waiting for FSMap version 105 2024-03-31T15:14:45.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.277+0000 7f4b22493000 10 client.9606 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-31T15:14:45.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.281+0000 7f4b22493000 20 client.9606 populate_metadata read hostname 'smithi016' 2024-03-31T15:14:45.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.281+0000 7f4b22493000 1 --2- 172.21.15.16:0/3084705189 >> [v2:172.21.15.16:6833/4142860904,v1:172.21.15.16:6835/4142860904] conn(0x563bb601aa70 0x563bb601ae50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:45.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.281+0000 7f4b22493000 4 client.9606 mds_command: new command op to 9558 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-31T15:14:45.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.281+0000 7f4b22493000 1 -- 172.21.15.16:0/3084705189 --> [v2:172.21.15.16:6833/4142860904,v1:172.21.15.16:6835/4142860904] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563bb5decbc0 con 0x563bb601aa70 2024-03-31T15:14:45.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.281+0000 7f4b0fe00640 1 --2- 172.21.15.16:0/3084705189 >> [v2:172.21.15.16:6833/4142860904,v1:172.21.15.16:6835/4142860904] conn(0x563bb601aa70 0x563bb601ae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:45.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.281+0000 7f4b0fe00640 1 --2- 172.21.15.16:0/3084705189 >> [v2:172.21.15.16:6833/4142860904,v1:172.21.15.16:6835/4142860904] conn(0x563bb601aa70 0x563bb601ae50 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1d956cb703d371e5 server_cookie=27f2d17c4d63ae1e in_seq=0 out_seq=0 2024-03-31T15:14:45.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.281+0000 7f4b05600640 10 client.9606 ms_handle_connect on v2:172.21.15.16:6833/4142860904 2024-03-31T15:14:45.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.281+0000 7f4b05600640 1 -- 172.21.15.16:0/3084705189 <== mds.0 v2:172.21.15.16:6833/4142860904 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x563bb5decbc0 con 0x563bb601aa70 2024-03-31T15:14:45.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.281+0000 7f4b05600640 10 client.9606 handle_command_reply: tid=0 2024-03-31T15:14:45.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.281+0000 7f4b22493000 1 client.9606 shutdown 2024-03-31T15:14:45.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.281+0000 7f4b22493000 1 -- 172.21.15.16:0/3084705189 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563bb5fff0b0 msgr2=0x563bb60536e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:45.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.281+0000 7f4b22493000 1 --2- 172.21.15.16:0/3084705189 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563bb5fff0b0 0x563bb60536e0 secure :-1 s=READY pgs=1754 cs=0 l=1 rev1=1 crypto rx=0x7f4b0800b4b0 tx=0x7f4b08056d90 comp rx=0 tx=0).stop 2024-03-31T15:14:45.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.281+0000 7f4b22493000 1 -- 172.21.15.16:0/3084705189 shutdown_connections 2024-03-31T15:14:45.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.281+0000 7f4b22493000 1 --2- 172.21.15.16:0/3084705189 >> [v2:172.21.15.16:6833/4142860904,v1:172.21.15.16:6835/4142860904] conn(0x563bb601aa70 0x563bb601ae50 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:45.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.281+0000 7f4b22493000 1 --2- 172.21.15.16:0/3084705189 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563bb604bdc0 0x563bb6019d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:45.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.281+0000 7f4b22493000 1 --2- 172.21.15.16:0/3084705189 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x563bb604b4a0 0x563bb6053c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:45.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.281+0000 7f4b22493000 1 --2- 172.21.15.16:0/3084705189 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563bb5fff0b0 0x563bb60536e0 unknown :-1 s=CLOSED pgs=1754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:45.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.281+0000 7f4b22493000 1 -- 172.21.15.16:0/3084705189 >> 172.21.15.16:0/3084705189 conn(0x563bb5ee9b80 msgr2=0x563bb5ffe270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:14:45.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.281+0000 7f4b05600640 0 client.9606 ms_handle_reset on v2:172.21.15.16:6833/4142860904 2024-03-31T15:14:45.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.281+0000 7f4b22493000 1 -- 172.21.15.16:0/3084705189 shutdown_connections 2024-03-31T15:14:45.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.281+0000 7f4b22493000 1 -- 172.21.15.16:0/3084705189 wait complete. 2024-03-31T15:14:45.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.281+0000 7f4b22493000 20 client.9606 trim_cache size 0 max 16384 2024-03-31T15:14:45.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.301+0000 7f4b22493000 1 Processor -- start 2024-03-31T15:14:45.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.301+0000 7f4b22493000 1 -- start start 2024-03-31T15:14:45.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.301+0000 7f4b22493000 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x563bb5fff0b0 0x563bb602e140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:45.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.301+0000 7f4b22493000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563bb604b4a0 0x563bb6006980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:45.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.301+0000 7f4b22493000 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563bb604bdc0 0x563bb6006ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:45.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.301+0000 7f4b22493000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f4b18203880 con 0x563bb604b4a0 2024-03-31T15:14:45.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.301+0000 7f4b22493000 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f4b1815a210 con 0x563bb604bdc0 2024-03-31T15:14:45.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.301+0000 7f4b22493000 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f4b18159570 con 0x563bb5fff0b0 2024-03-31T15:14:45.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.301+0000 7f4b0fe00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563bb604b4a0 0x563bb6006980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:45.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.301+0000 7f4b0fe00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563bb604b4a0 0x563bb6006980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40792/0 (socket says 172.21.15.16:40792) 2024-03-31T15:14:45.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.301+0000 7f4b0fe00640 1 -- 172.21.15.16:0/3505898820 learned_addr learned my addr 172.21.15.16:0/3505898820 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:14:45.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.301+0000 7f4b1d000640 1 --2- 172.21.15.16:0/3505898820 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563bb604bdc0 0x563bb6006ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:45.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.301+0000 7f4b17e00640 1 --2- 172.21.15.16:0/3505898820 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x563bb5fff0b0 0x563bb602e140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:45.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.301+0000 7f4b0fe00640 1 -- 172.21.15.16:0/3505898820 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x563bb5fff0b0 msgr2=0x563bb602e140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:14:45.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.301+0000 7f4b0fe00640 1 --2- 172.21.15.16:0/3505898820 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x563bb5fff0b0 0x563bb602e140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:45.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.301+0000 7f4b0fe00640 1 -- 172.21.15.16:0/3505898820 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563bb604bdc0 msgr2=0x563bb6006ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:14:45.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.301+0000 7f4b0fe00640 1 --2- 172.21.15.16:0/3505898820 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563bb604bdc0 0x563bb6006ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:45.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.301+0000 7f4b0fe00640 1 -- 172.21.15.16:0/3505898820 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563bb5e0c490 con 0x563bb604b4a0 2024-03-31T15:14:45.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.301+0000 7f4b1d000640 1 --2- 172.21.15.16:0/3505898820 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563bb604bdc0 0x563bb6006ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:14:45.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.301+0000 7f4b17e00640 1 --2- 172.21.15.16:0/3505898820 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x563bb5fff0b0 0x563bb602e140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:14:45.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.301+0000 7f4b0fe00640 1 --2- 172.21.15.16:0/3505898820 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563bb604b4a0 0x563bb6006980 secure :-1 s=READY pgs=1755 cs=0 l=1 rev1=1 crypto rx=0x7f4b000ae040 tx=0x7f4b00004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=144f263f5f73e357 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:45.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.301+0000 7f4b07e00640 1 -- 172.21.15.16:0/3505898820 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b000b0040 con 0x563bb604b4a0 2024-03-31T15:14:45.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.301+0000 7f4b07e00640 1 -- 172.21.15.16:0/3505898820 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4b000cd030 con 0x563bb604b4a0 2024-03-31T15:14:45.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.301+0000 7f4b07e00640 1 -- 172.21.15.16:0/3505898820 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b000b66e0 con 0x563bb604b4a0 2024-03-31T15:14:45.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.301+0000 7f4b22493000 1 -- 172.21.15.16:0/3505898820 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563bb604b4a0 msgr2=0x563bb6006980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:45.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.301+0000 7f4b22493000 1 --2- 172.21.15.16:0/3505898820 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563bb604b4a0 0x563bb6006980 secure :-1 s=READY pgs=1755 cs=0 l=1 rev1=1 crypto rx=0x7f4b000ae040 tx=0x7f4b00004740 comp rx=0 tx=0).stop 2024-03-31T15:14:45.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.301+0000 7f4b22493000 1 -- 172.21.15.16:0/3505898820 shutdown_connections 2024-03-31T15:14:45.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.301+0000 7f4b22493000 1 --2- 172.21.15.16:0/3505898820 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563bb604bdc0 0x563bb6006ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:45.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.301+0000 7f4b22493000 1 --2- 172.21.15.16:0/3505898820 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563bb604b4a0 0x563bb6006980 unknown :-1 s=CLOSED pgs=1755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:45.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.301+0000 7f4b22493000 1 --2- 172.21.15.16:0/3505898820 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x563bb5fff0b0 0x563bb602e140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:45.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.301+0000 7f4b22493000 1 -- 172.21.15.16:0/3505898820 >> 172.21.15.16:0/3505898820 conn(0x563bb5ee9b80 msgr2=0x563bb6006270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:14:45.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.301+0000 7f4b22493000 1 -- 172.21.15.16:0/3505898820 shutdown_connections 2024-03-31T15:14:45.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.301+0000 7f4b22493000 1 -- 172.21.15.16:0/3505898820 wait complete. 2024-03-31T15:14:45.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.301+0000 7f4b22493000 1 -- start start 2024-03-31T15:14:45.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.301+0000 7f4b22493000 1 Processor -- start 2024-03-31T15:14:45.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.305+0000 7f4b22493000 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563bb5fff0b0 0x563bb6006ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:45.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.305+0000 7f4b22493000 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x563bb604b4a0 0x563bb5ffd2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:45.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.305+0000 7f4b22493000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563bb604bdc0 0x563bb5ffd810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:45.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.305+0000 7f4b22493000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x563bb5e55f80 con 0x563bb604bdc0 2024-03-31T15:14:45.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.305+0000 7f4b22493000 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f4b18159bc0 con 0x563bb5fff0b0 2024-03-31T15:14:45.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.305+0000 7f4b22493000 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x563bb5dd5b10 con 0x563bb604b4a0 2024-03-31T15:14:45.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.305+0000 7f4b22493000 10 client.0 _ll_register_callbacks cb 0x7f4b080046d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T15:14:45.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.305+0000 7f4b1d000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563bb604bdc0 0x563bb5ffd810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:45.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.305+0000 7f4b1d000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563bb604bdc0 0x563bb5ffd810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40804/0 (socket says 172.21.15.16:40804) 2024-03-31T15:14:45.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.305+0000 7f4b17e00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563bb5fff0b0 0x563bb6006ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:45.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.305+0000 7f4b1d000640 1 -- 172.21.15.16:0/533892929 learned_addr learned my addr 172.21.15.16:0/533892929 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:14:45.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.305+0000 7f4b0fe00640 1 --2- 172.21.15.16:0/533892929 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x563bb604b4a0 0x563bb5ffd2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:45.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.305+0000 7f4b1d000640 1 -- 172.21.15.16:0/533892929 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x563bb604b4a0 msgr2=0x563bb5ffd2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:14:45.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.305+0000 7f4b1d000640 1 --2- 172.21.15.16:0/533892929 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x563bb604b4a0 0x563bb5ffd2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:45.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.305+0000 7f4b1d000640 1 -- 172.21.15.16:0/533892929 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563bb5fff0b0 msgr2=0x563bb6006ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:14:45.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.305+0000 7f4b1d000640 1 --2- 172.21.15.16:0/533892929 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563bb5fff0b0 0x563bb6006ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:45.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.305+0000 7f4b1d000640 1 -- 172.21.15.16:0/533892929 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563bb5e0ae80 con 0x563bb604bdc0 2024-03-31T15:14:45.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.305+0000 7f4b17e00640 1 --2- 172.21.15.16:0/533892929 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563bb5fff0b0 0x563bb6006ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:14:45.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.305+0000 7f4b0fe00640 1 --2- 172.21.15.16:0/533892929 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x563bb604b4a0 0x563bb5ffd2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:14:45.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.305+0000 7f4b1d000640 1 --2- 172.21.15.16:0/533892929 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563bb604bdc0 0x563bb5ffd810 secure :-1 s=READY pgs=1756 cs=0 l=1 rev1=1 crypto rx=0x563bb5ea8310 tx=0x7f4b10004ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=969eca7b4b86203e server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:45.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.305+0000 7f4b05600640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-31T15:14:45.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.305+0000 7f4b05600640 1 -- 172.21.15.16:0/533892929 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b1002c060 con 0x563bb604bdc0 2024-03-31T15:14:45.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.305+0000 7f4b05600640 1 -- 172.21.15.16:0/533892929 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(112..112 src has 1..112) ==== 5853+0+0 (secure 0 0 0) 0x7f4b10014be0 con 0x563bb604bdc0 2024-03-31T15:14:45.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.305+0000 7f4b22493000 1 -- 172.21.15.16:0/533892929 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4b000cd030 con 0x563bb604bdc0 2024-03-31T15:14:45.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.305+0000 7f4b05600640 1 -- 172.21.15.16:0/533892929 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=105) ==== 24+0+0 (secure 0 0 0) 0x7f4b10032050 con 0x563bb604bdc0 2024-03-31T15:14:45.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.305+0000 7f4b22493000 10 client.9612 fetch_fsmap learned FSMap version 105 2024-03-31T15:14:45.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.305+0000 7f4b22493000 1 -- 172.21.15.16:0/533892929 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=105,monmap=2+}) -- 0x563bb5df8270 con 0x563bb604bdc0 2024-03-31T15:14:45.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.305+0000 7f4b05600640 1 -- 172.21.15.16:0/533892929 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4b000cd030 con 0x563bb604bdc0 2024-03-31T15:14:45.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.305+0000 7f4b05600640 1 -- 172.21.15.16:0/533892929 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 105) ==== 3494+0+0 (secure 0 0 0) 0x7f4b10035020 con 0x563bb604bdc0 2024-03-31T15:14:45.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.305+0000 7f4b22493000 10 client.9612 fetch_fsmap finished waiting for FSMap version 105 2024-03-31T15:14:45.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.305+0000 7f4b22493000 10 client.9612 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-31T15:14:45.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.305+0000 7f4b22493000 20 client.9612 populate_metadata read hostname 'smithi016' 2024-03-31T15:14:45.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.305+0000 7f4b22493000 1 --2- 172.21.15.16:0/533892929 >> [v2:172.21.15.16:6833/4142860904,v1:172.21.15.16:6835/4142860904] conn(0x563bb602e380 0x563bb5ffdd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:45.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.305+0000 7f4b22493000 4 client.9612 mds_command: new command op to 9558 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-31T15:14:45.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.305+0000 7f4b22493000 1 -- 172.21.15.16:0/533892929 --> [v2:172.21.15.16:6833/4142860904,v1:172.21.15.16:6835/4142860904] -- command(tid 0: {"prefix": "status"}) -- 0x563bb5e5ee90 con 0x563bb602e380 2024-03-31T15:14:45.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.305+0000 7f4b17e00640 1 --2- 172.21.15.16:0/533892929 >> [v2:172.21.15.16:6833/4142860904,v1:172.21.15.16:6835/4142860904] conn(0x563bb602e380 0x563bb5ffdd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:45.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.305+0000 7f4b17e00640 1 --2- 172.21.15.16:0/533892929 >> [v2:172.21.15.16:6833/4142860904,v1:172.21.15.16:6835/4142860904] conn(0x563bb602e380 0x563bb5ffdd50 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5a97ae5ece1467fc server_cookie=401dff2a623f8881 in_seq=0 out_seq=0 2024-03-31T15:14:45.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.305+0000 7f4b05600640 10 client.9612 ms_handle_connect on v2:172.21.15.16:6833/4142860904 2024-03-31T15:14:45.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.305+0000 7f4b05600640 1 -- 172.21.15.16:0/533892929 <== mds.0 v2:172.21.15.16:6833/4142860904 1 ==== command_reply(tid 0: 0 ) ==== 8+0+334 (crc 0 0 0) 0x563bb5e5ee90 con 0x563bb602e380 2024-03-31T15:14:45.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.305+0000 7f4b05600640 10 client.9612 handle_command_reply: tid=0 2024-03-31T15:14:45.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.305+0000 7f4b22493000 1 client.9612 shutdown 2024-03-31T15:14:45.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.309+0000 7f4b22493000 1 -- 172.21.15.16:0/533892929 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563bb604bdc0 msgr2=0x563bb5ffd810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:45.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.309+0000 7f4b22493000 1 --2- 172.21.15.16:0/533892929 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563bb604bdc0 0x563bb5ffd810 secure :-1 s=READY pgs=1756 cs=0 l=1 rev1=1 crypto rx=0x563bb5ea8310 tx=0x7f4b10004ac0 comp rx=0 tx=0).stop 2024-03-31T15:14:45.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.309+0000 7f4b22493000 1 -- 172.21.15.16:0/533892929 shutdown_connections 2024-03-31T15:14:45.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.309+0000 7f4b22493000 1 --2- 172.21.15.16:0/533892929 >> [v2:172.21.15.16:6833/4142860904,v1:172.21.15.16:6835/4142860904] conn(0x563bb602e380 0x563bb5ffdd50 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:45.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.309+0000 7f4b22493000 1 --2- 172.21.15.16:0/533892929 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563bb604bdc0 0x563bb5ffd810 unknown :-1 s=CLOSED pgs=1756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:45.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.309+0000 7f4b22493000 1 --2- 172.21.15.16:0/533892929 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x563bb604b4a0 0x563bb5ffd2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:45.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.309+0000 7f4b22493000 1 --2- 172.21.15.16:0/533892929 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563bb5fff0b0 0x563bb6006ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:45.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.309+0000 7f4b05600640 0 client.9612 ms_handle_reset on v2:172.21.15.16:6833/4142860904 2024-03-31T15:14:45.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.309+0000 7f4b22493000 1 -- 172.21.15.16:0/533892929 >> 172.21.15.16:0/533892929 conn(0x563bb5ee9b80 msgr2=0x563bb60528e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:14:45.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.309+0000 7f4b22493000 1 -- 172.21.15.16:0/533892929 shutdown_connections 2024-03-31T15:14:45.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.309+0000 7f4b22493000 1 -- 172.21.15.16:0/533892929 wait complete. 2024-03-31T15:14:45.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.309+0000 7f4b22493000 20 client.9612 trim_cache size 0 max 16384 2024-03-31T15:14:45.306 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-03-31T15:14:45.306 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_fsid": "286eb5da-fb47-4550-b290-d9519c13b9a5", 2024-03-31T15:14:45.306 INFO:teuthology.orchestra.run.smithi016.stdout: "whoami": 0, 2024-03-31T15:14:45.306 INFO:teuthology.orchestra.run.smithi016.stdout: "id": 9558, 2024-03-31T15:14:45.307 INFO:teuthology.orchestra.run.smithi016.stdout: "want_state": "up:active", 2024-03-31T15:14:45.307 INFO:teuthology.orchestra.run.smithi016.stdout: "state": "up:active", 2024-03-31T15:14:45.307 INFO:teuthology.orchestra.run.smithi016.stdout: "fs_name": "cephfs", 2024-03-31T15:14:45.307 INFO:teuthology.orchestra.run.smithi016.stdout: "rank_uptime": 4.5855115169999996, 2024-03-31T15:14:45.307 INFO:teuthology.orchestra.run.smithi016.stdout: "mdsmap_epoch": 105, 2024-03-31T15:14:45.307 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch": 112, 2024-03-31T15:14:45.307 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch_barrier": 112, 2024-03-31T15:14:45.307 INFO:teuthology.orchestra.run.smithi016.stdout: "uptime": 5.5247219970000003 2024-03-31T15:14:45.307 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-03-31T15:14:45.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.309+0000 7f4b1da00640 1 -- 172.21.15.16:0/2573198869 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f4af4068380 msgr2=0x7f4af406a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:45.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.309+0000 7f4b1da00640 1 --2- 172.21.15.16:0/2573198869 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f4af4068380 0x7f4af406a840 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f4b000ad000 tx=0x7f4b00008040 comp rx=0 tx=0).stop 2024-03-31T15:14:45.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.309+0000 7f4b1da00640 1 -- 172.21.15.16:0/2573198869 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b1806b620 msgr2=0x7f4b1814b270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:45.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.309+0000 7f4b1da00640 1 --2- 172.21.15.16:0/2573198869 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b1806b620 0x7f4b1814b270 secure :-1 s=READY pgs=1752 cs=0 l=1 rev1=1 crypto rx=0x7f4b0800d800 tx=0x7f4b0800dcc0 comp rx=0 tx=0).stop 2024-03-31T15:14:45.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.309+0000 7f4b1da00640 1 -- 172.21.15.16:0/2573198869 shutdown_connections 2024-03-31T15:14:45.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.309+0000 7f4b1da00640 1 --2- 172.21.15.16:0/2573198869 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f4af4068380 0x7f4af406a840 unknown :-1 s=CLOSED pgs=743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:45.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.309+0000 7f4b1da00640 1 --2- 172.21.15.16:0/2573198869 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4b18158700 0x7f4b180b1c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:45.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.309+0000 7f4b1da00640 1 --2- 172.21.15.16:0/2573198869 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f4b1806e4c0 0x7f4b180b16e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:45.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.309+0000 7f4b1da00640 1 --2- 172.21.15.16:0/2573198869 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b1806b620 0x7f4b1814b270 unknown :-1 s=CLOSED pgs=1752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:45.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.309+0000 7f4b1da00640 1 -- 172.21.15.16:0/2573198869 >> 172.21.15.16:0/2573198869 conn(0x7f4b180b2b30 msgr2=0x7f4b180b59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:14:45.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.309+0000 7f4b1da00640 1 -- 172.21.15.16:0/2573198869 shutdown_connections 2024-03-31T15:14:45.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.309+0000 7f4b1da00640 1 -- 172.21.15.16:0/2573198869 wait complete. 2024-03-31T15:14:45.323 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-31T15:14:45.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.469+0000 7f9f7e200640 1 Processor -- start 2024-03-31T15:14:45.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.469+0000 7f9f7e200640 1 -- start start 2024-03-31T15:14:45.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.469+0000 7f9f7e200640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f9f78145e40 0x7f9f780cc740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:45.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.469+0000 7f9f7e200640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f9f780ccc80 0x7f9f7806f920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:45.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.469+0000 7f9f7e200640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9f780d0920 con 0x7f9f7806fe60 2024-03-31T15:14:45.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.469+0000 7f9f7e200640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f9f780d06d0 con 0x7f9f78145e40 2024-03-31T15:14:45.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.469+0000 7f9f7e200640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f9f780d0c00 con 0x7f9f780ccc80 2024-03-31T15:14:45.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.469+0000 7f9f77e00640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f9f7806fe60 0x7f9f7815fdf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:33474/0 (socket says 172.21.15.16:33474) 2024-03-31T15:14:45.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.469+0000 7f9f77e00640 1 -- 172.21.15.16:0/2113487447 learned_addr learned my addr 172.21.15.16:0/2113487447 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:14:45.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.469+0000 7f9f77400640 1 --2- 172.21.15.16:0/2113487447 >> v2:172.21.15.89:3300/0 conn(0x7f9f78145e40 0x7f9f780cc740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:45.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.469+0000 7f9f76a00640 1 --2- 172.21.15.16:0/2113487447 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f9f780ccc80 0x7f9f7806f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:45.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.473+0000 7f9f76000640 1 -- 172.21.15.16:0/2113487447 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2081448816 0 0) 0x7f9f780d0920 con 0x7f9f7806fe60 2024-03-31T15:14:45.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.473+0000 7f9f76000640 1 -- 172.21.15.16:0/2113487447 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9f5c003530 con 0x7f9f7806fe60 2024-03-31T15:14:45.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.473+0000 7f9f77400640 1 -- 172.21.15.16:0/2113487447 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f9f780ccc80 msgr2=0x7f9f7806f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:14:45.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.473+0000 7f9f77400640 1 --2- 172.21.15.16:0/2113487447 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f9f780ccc80 0x7f9f7806f920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:45.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.473+0000 7f9f77400640 1 -- 172.21.15.16:0/2113487447 >> v1:172.21.15.16:6789/0 conn(0x7f9f7806fe60 legacy=0x7f9f7815fdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:45.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.473+0000 7f9f77400640 1 -- 172.21.15.16:0/2113487447 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f780d1810 con 0x7f9f78145e40 2024-03-31T15:14:45.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.473+0000 7f9f76a00640 1 --2- 172.21.15.16:0/2113487447 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f9f780ccc80 0x7f9f7806f920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:14:45.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.473+0000 7f9f77400640 1 --2- 172.21.15.16:0/2113487447 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9f78145e40 0x7f9f780cc740 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f9f680097c0 tx=0x7f9f6801c4a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5869851e0da035f5 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:45.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.473+0000 7f9f76000640 1 -- 172.21.15.16:0/2113487447 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f680b0070 con 0x7f9f78145e40 2024-03-31T15:14:45.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.473+0000 7f9f76000640 1 -- 172.21.15.16:0/2113487447 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9f680a4e50 con 0x7f9f78145e40 2024-03-31T15:14:45.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.473+0000 7f9f76000640 1 -- 172.21.15.16:0/2113487447 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f680ab820 con 0x7f9f78145e40 2024-03-31T15:14:45.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.477+0000 7f9f7e200640 1 -- 172.21.15.16:0/2113487447 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9f78145e40 msgr2=0x7f9f780cc740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:45.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.477+0000 7f9f7e200640 1 --2- 172.21.15.16:0/2113487447 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9f78145e40 0x7f9f780cc740 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f9f680097c0 tx=0x7f9f6801c4a0 comp rx=0 tx=0).stop 2024-03-31T15:14:45.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.477+0000 7f9f7e200640 1 -- 172.21.15.16:0/2113487447 shutdown_connections 2024-03-31T15:14:45.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.477+0000 7f9f7e200640 1 --2- 172.21.15.16:0/2113487447 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f9f780ccc80 0x7f9f7806f920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:45.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.477+0000 7f9f7e200640 1 --2- 172.21.15.16:0/2113487447 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9f78145e40 0x7f9f780cc740 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:45.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.477+0000 7f9f7e200640 1 -- 172.21.15.16:0/2113487447 >> 172.21.15.16:0/2113487447 conn(0x7f9f78067330 msgr2=0x7f9f78067740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:14:45.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.477+0000 7f9f7e200640 1 -- 172.21.15.16:0/2113487447 shutdown_connections 2024-03-31T15:14:45.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.477+0000 7f9f7e200640 1 -- 172.21.15.16:0/2113487447 wait complete. 2024-03-31T15:14:45.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.477+0000 7f9f7e200640 1 Processor -- start 2024-03-31T15:14:45.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.477+0000 7f9f7e200640 1 -- start start 2024-03-31T15:14:45.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.481+0000 7f9f7e200640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9f7806fe60 0x7f9f780cbf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:45.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.481+0000 7f9f7e200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9f780ccc80 0x7f9f780c8900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:45.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.481+0000 7f9f7e200640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f9f78145e40 0x7f9f78160ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:45.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.481+0000 7f9f7e200640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f9f780b5240 con 0x7f9f780ccc80 2024-03-31T15:14:45.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.481+0000 7f9f7e200640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f9f780b50c0 con 0x7f9f7806fe60 2024-03-31T15:14:45.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.481+0000 7f9f7e200640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f9f78169190 con 0x7f9f78145e40 2024-03-31T15:14:45.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.481+0000 7f9f76a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9f780ccc80 0x7f9f780c8900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:45.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.481+0000 7f9f76a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9f780ccc80 0x7f9f780c8900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40812/0 (socket says 172.21.15.16:40812) 2024-03-31T15:14:45.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.481+0000 7f9f76a00640 1 -- 172.21.15.16:0/121023495 learned_addr learned my addr 172.21.15.16:0/121023495 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:14:45.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.481+0000 7f9f77400640 1 --2- 172.21.15.16:0/121023495 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9f7806fe60 0x7f9f780cbf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:45.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.481+0000 7f9f77e00640 1 --2- 172.21.15.16:0/121023495 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f9f78145e40 0x7f9f78160ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:45.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.481+0000 7f9f76a00640 1 -- 172.21.15.16:0/121023495 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f9f78145e40 msgr2=0x7f9f78160ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:45.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.481+0000 7f9f76a00640 1 --2- 172.21.15.16:0/121023495 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f9f78145e40 0x7f9f78160ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:45.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.481+0000 7f9f76a00640 1 -- 172.21.15.16:0/121023495 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9f7806fe60 msgr2=0x7f9f780cbf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:45.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.481+0000 7f9f76a00640 1 --2- 172.21.15.16:0/121023495 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9f7806fe60 0x7f9f780cbf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:45.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.481+0000 7f9f76a00640 1 -- 172.21.15.16:0/121023495 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f7815fb50 con 0x7f9f780ccc80 2024-03-31T15:14:45.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.481+0000 7f9f76a00640 1 --2- 172.21.15.16:0/121023495 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9f780ccc80 0x7f9f780c8900 secure :-1 s=READY pgs=1758 cs=0 l=1 rev1=1 crypto rx=0x7f9f6400ebe0 tx=0x7f9f6400c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:45.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.481+0000 7f9f6f400640 1 -- 172.21.15.16:0/121023495 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f64017070 con 0x7f9f780ccc80 2024-03-31T15:14:45.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.481+0000 7f9f6f400640 1 -- 172.21.15.16:0/121023495 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9f6400cbd0 con 0x7f9f780ccc80 2024-03-31T15:14:45.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.481+0000 7f9f7e200640 1 -- 172.21.15.16:0/121023495 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9f780b4a50 con 0x7f9f780ccc80 2024-03-31T15:14:45.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.481+0000 7f9f6f400640 1 -- 172.21.15.16:0/121023495 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f64013470 con 0x7f9f780ccc80 2024-03-31T15:14:45.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.481+0000 7f9f7e200640 1 -- 172.21.15.16:0/121023495 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9f78161420 con 0x7f9f780ccc80 2024-03-31T15:14:45.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.481+0000 7f9f7e200640 1 -- 172.21.15.16:0/121023495 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9f78069950 con 0x7f9f780ccc80 2024-03-31T15:14:45.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.485+0000 7f9f6f400640 1 -- 172.21.15.16:0/121023495 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f9f6401a070 con 0x7f9f780ccc80 2024-03-31T15:14:45.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.485+0000 7f9f6f400640 1 --2- 172.21.15.16:0/121023495 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f9f580683b0 0x7f9f5806a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:45.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.485+0000 7f9f6f400640 1 -- 172.21.15.16:0/121023495 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(112..112 src has 1..112) ==== 5853+0+0 (secure 0 0 0) 0x7f9f640868e0 con 0x7f9f780ccc80 2024-03-31T15:14:45.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.485+0000 7f9f77400640 1 --2- 172.21.15.16:0/121023495 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f9f580683b0 0x7f9f5806a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:45.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.485+0000 7f9f77400640 1 --2- 172.21.15.16:0/121023495 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f9f580683b0 0x7f9f5806a870 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f9f680ad000 tx=0x7f9f680a6000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:45.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.493+0000 7f9f6f400640 1 -- 172.21.15.16:0/121023495 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f9f64056730 con 0x7f9f780ccc80 2024-03-31T15:14:45.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.697+0000 7f9f7e200640 1 -- 172.21.15.16:0/121023495 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f9f780c8fe0 con 0x7f9f780ccc80 2024-03-31T15:14:45.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.697+0000 7f9f6f400640 1 -- 172.21.15.16:0/121023495 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v105) ==== 53+0+73 (secure 0 0 0) 0x7f9f6405aa30 con 0x7f9f780ccc80 2024-03-31T15:14:45.690 INFO:teuthology.orchestra.run.smithi016.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-31T15:14:45.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.697+0000 7f9f7e200640 1 -- 172.21.15.16:0/121023495 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f9f580683b0 msgr2=0x7f9f5806a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:45.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.697+0000 7f9f7e200640 1 --2- 172.21.15.16:0/121023495 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f9f580683b0 0x7f9f5806a870 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f9f680ad000 tx=0x7f9f680a6000 comp rx=0 tx=0).stop 2024-03-31T15:14:45.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.697+0000 7f9f7e200640 1 -- 172.21.15.16:0/121023495 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9f780ccc80 msgr2=0x7f9f780c8900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:45.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.697+0000 7f9f7e200640 1 --2- 172.21.15.16:0/121023495 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9f780ccc80 0x7f9f780c8900 secure :-1 s=READY pgs=1758 cs=0 l=1 rev1=1 crypto rx=0x7f9f6400ebe0 tx=0x7f9f6400c520 comp rx=0 tx=0).stop 2024-03-31T15:14:45.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.697+0000 7f9f7e200640 1 -- 172.21.15.16:0/121023495 shutdown_connections 2024-03-31T15:14:45.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.697+0000 7f9f7e200640 1 --2- 172.21.15.16:0/121023495 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f9f580683b0 0x7f9f5806a870 unknown :-1 s=CLOSED pgs=744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:45.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.697+0000 7f9f7e200640 1 --2- 172.21.15.16:0/121023495 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f9f78145e40 0x7f9f78160ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:45.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.697+0000 7f9f7e200640 1 --2- 172.21.15.16:0/121023495 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9f780ccc80 0x7f9f780c8900 unknown :-1 s=CLOSED pgs=1758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:45.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.697+0000 7f9f7e200640 1 --2- 172.21.15.16:0/121023495 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9f7806fe60 0x7f9f780cbf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:45.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.697+0000 7f9f7e200640 1 -- 172.21.15.16:0/121023495 >> 172.21.15.16:0/121023495 conn(0x7f9f78067330 msgr2=0x7f9f7815f440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:14:45.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.697+0000 7f9f7e200640 1 -- 172.21.15.16:0/121023495 shutdown_connections 2024-03-31T15:14:45.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45.697+0000 7f9f7e200640 1 -- 172.21.15.16:0/121023495 wait complete. 2024-03-31T15:14:45.712 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-31T15:14:45.712 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-31T15:14:45.712 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi016.front.sepia.ceph.com 2024-03-31T15:14:45.712 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-31T15:14:45.712 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-31T15:14:45.712 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-31T15:14:45.712 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-31T15:14:45.713 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-31T15:14:45.713 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-31T15:14:45.713 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:14:45.713 DEBUG:teuthology.orchestra.run.smithi016:> ip addr 2024-03-31T15:14:45.717 INFO:tasks.ceph.mds.c.smithi016.stderr:2024-03-31T15:14:45.725+0000 7ff49ee00640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-31T15:14:45.722 INFO:teuthology.orchestra.run.smithi016.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-31T15:14:45.723 INFO:teuthology.orchestra.run.smithi016.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-31T15:14:45.723 INFO:teuthology.orchestra.run.smithi016.stdout: inet 127.0.0.1/8 scope host lo 2024-03-31T15:14:45.723 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2024-03-31T15:14:45.723 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 ::1/128 scope host 2024-03-31T15:14:45.723 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2024-03-31T15:14:45.723 INFO:teuthology.orchestra.run.smithi016.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-03-31T15:14:45.723 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:6b:fd:e0 brd ff:ff:ff:ff:ff:ff 2024-03-31T15:14:45.723 INFO:teuthology.orchestra.run.smithi016.stdout: altname enp7s0f0 2024-03-31T15:14:45.723 INFO:teuthology.orchestra.run.smithi016.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-03-31T15:14:45.723 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:6b:fd:e1 brd ff:ff:ff:ff:ff:ff 2024-03-31T15:14:45.723 INFO:teuthology.orchestra.run.smithi016.stdout: altname enp7s0f1 2024-03-31T15:14:45.723 INFO:teuthology.orchestra.run.smithi016.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-31T15:14:45.724 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:bd:15:64 brd ff:ff:ff:ff:ff:ff 2024-03-31T15:14:45.724 INFO:teuthology.orchestra.run.smithi016.stdout: altname enp1s0f0 2024-03-31T15:14:45.724 INFO:teuthology.orchestra.run.smithi016.stdout: inet 172.21.15.16/20 brd 172.21.15.255 scope global dynamic ens1f0 2024-03-31T15:14:45.724 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft 41795sec preferred_lft 41795sec 2024-03-31T15:14:45.724 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 fe80::ec4:7aff:febd:1564/64 scope link 2024-03-31T15:14:45.724 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2024-03-31T15:14:45.724 INFO:teuthology.orchestra.run.smithi016.stdout:5: ens1f1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-03-31T15:14:45.724 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:bd:15:65 brd ff:ff:ff:ff:ff:ff 2024-03-31T15:14:45.724 INFO:teuthology.orchestra.run.smithi016.stdout: altname enp1s0f1 2024-03-31T15:14:45.724 INFO:teuthology.orchestra.run.smithi016.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2024-03-31T15:14:45.724 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 02:42:d4:1f:7d:73 brd ff:ff:ff:ff:ff:ff 2024-03-31T15:14:45.724 INFO:teuthology.orchestra.run.smithi016.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2024-03-31T15:14:45.724 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2024-03-31T15:14:45.724 INFO:teuthology.orchestra.run.smithi016.stdout:7: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-31T15:14:45.724 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 5e:1f:84:6d:79:ae brd ff:ff:ff:ff:ff:ff 2024-03-31T15:14:45.725 INFO:teuthology.orchestra.run.smithi016.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-31T15:14:45.725 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2024-03-31T15:14:45.725 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 fe80::5c1f:84ff:fe6d:79ae/64 scope link 2024-03-31T15:14:45.725 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2024-03-31T15:14:45.725 INFO:teuthology.orchestra.run.smithi016.stdout:8: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-31T15:14:45.725 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0e:5c:f6:1f:82:2f brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-31T15:14:45.725 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 fe80::c5c:f6ff:fe1f:822f/64 scope link 2024-03-31T15:14:45.725 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2024-03-31T15:14:45.725 INFO:teuthology.orchestra.run.smithi016.stdout:9: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-31T15:14:45.725 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether c6:a6:3d:69:d0:db brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-31T15:14:45.725 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 fe80::c4a6:3dff:fe69:d0db/64 scope link 2024-03-31T15:14:45.725 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2024-03-31T15:14:45.726 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:14:45.726 DEBUG:teuthology.orchestra.run.smithi016:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-31T15:14:45.782 INFO:teuthology.orchestra.run.smithi016.stdout:1 2024-03-31T15:14:45.784 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:14:45.784 DEBUG:teuthology.orchestra.run.smithi016:> route 2024-03-31T15:14:45.837 INFO:teuthology.orchestra.run.smithi016.stdout:Kernel IP routing table 2024-03-31T15:14:45.837 INFO:teuthology.orchestra.run.smithi016.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-31T15:14:45.837 INFO:teuthology.orchestra.run.smithi016.stdout:default _gateway 0.0.0.0 UG 0 0 0 ens1f0 2024-03-31T15:14:45.837 INFO:teuthology.orchestra.run.smithi016.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2024-03-31T15:14:45.837 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 0 0 0 ens1f0 2024-03-31T15:14:45.837 INFO:teuthology.orchestra.run.smithi016.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-31T15:14:45.838 DEBUG:teuthology.orchestra.run.smithi016:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-31T15:14:45.838 DEBUG:teuthology.orchestra.run.smithi016:> set -e 2024-03-31T15:14:45.838 DEBUG:teuthology.orchestra.run.smithi016:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-03-31T15:14:45.838 DEBUG:teuthology.orchestra.run.smithi016:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-03-31T15:14:45.838 DEBUG:teuthology.orchestra.run.smithi016:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-03-31T15:14:45.838 DEBUG:teuthology.orchestra.run.smithi016:> ') 2024-03-31T15:14:45.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-31T15:14:45.969 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:45 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-31T15:14:45.976 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:14:45.976 DEBUG:teuthology.orchestra.run.smithi016:> ip netns list 2024-03-31T15:14:45.986 INFO:teuthology.orchestra.run.smithi016.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-31T15:14:45.986 INFO:teuthology.orchestra.run.smithi016.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-31T15:14:45.986 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-31T15:14:45.986 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:14:45.986 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip link set brx.0 up 2024-03-31T15:14:46.048 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-03-31T15:14:46.048 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-31T15:14:46.049 DEBUG:teuthology.orchestra.run.smithi016:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-31T15:14:46.100 INFO:teuthology.orchestra.run.smithi016.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-31T15:14:46.100 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-31T15:14:46.101 DEBUG:teuthology.orchestra.run.smithi016:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-31T15:14:46.152 DEBUG:teuthology.orchestra.run.smithi016:> sudo modprobe fuse 2024-03-31T15:14:46.212 DEBUG:teuthology.orchestra.run.smithi016:> cat /proc/self/mounts | awk '{print $2}' 2024-03-31T15:14:46.262 INFO:teuthology.orchestra.run.smithi016.stdout:/sys 2024-03-31T15:14:46.262 INFO:teuthology.orchestra.run.smithi016.stdout:/proc 2024-03-31T15:14:46.262 INFO:teuthology.orchestra.run.smithi016.stdout:/dev 2024-03-31T15:14:46.262 INFO:teuthology.orchestra.run.smithi016.stdout:/dev/pts 2024-03-31T15:14:46.262 INFO:teuthology.orchestra.run.smithi016.stdout:/run 2024-03-31T15:14:46.262 INFO:teuthology.orchestra.run.smithi016.stdout:/ 2024-03-31T15:14:46.262 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/kernel/security 2024-03-31T15:14:46.263 INFO:teuthology.orchestra.run.smithi016.stdout:/dev/shm 2024-03-31T15:14:46.263 INFO:teuthology.orchestra.run.smithi016.stdout:/run/lock 2024-03-31T15:14:46.263 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/cgroup 2024-03-31T15:14:46.263 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/pstore 2024-03-31T15:14:46.263 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/bpf 2024-03-31T15:14:46.263 INFO:teuthology.orchestra.run.smithi016.stdout:/proc/sys/fs/binfmt_misc 2024-03-31T15:14:46.263 INFO:teuthology.orchestra.run.smithi016.stdout:/dev/hugepages 2024-03-31T15:14:46.263 INFO:teuthology.orchestra.run.smithi016.stdout:/dev/mqueue 2024-03-31T15:14:46.263 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/kernel/debug 2024-03-31T15:14:46.263 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/kernel/tracing 2024-03-31T15:14:46.263 INFO:teuthology.orchestra.run.smithi016.stdout:/proc/fs/nfsd 2024-03-31T15:14:46.263 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/fuse/connections 2024-03-31T15:14:46.263 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/kernel/config 2024-03-31T15:14:46.263 INFO:teuthology.orchestra.run.smithi016.stdout:/run/credentials/systemd-sysusers.service 2024-03-31T15:14:46.264 INFO:teuthology.orchestra.run.smithi016.stdout:/run/rpc_pipefs 2024-03-31T15:14:46.264 INFO:teuthology.orchestra.run.smithi016.stdout:/run/user/1000 2024-03-31T15:14:46.264 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/ceph 2024-03-31T15:14:46.264 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/ceph/osd/ceph-0 2024-03-31T15:14:46.264 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/ceph/osd/ceph-1 2024-03-31T15:14:46.264 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/ceph/osd/ceph-2 2024-03-31T15:14:46.264 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/ceph/osd/ceph-3 2024-03-31T15:14:46.264 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns 2024-03-31T15:14:46.264 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-31T15:14:46.264 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-31T15:14:46.264 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-31T15:14:46.264 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-31T15:14:46.265 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:14:46.265 DEBUG:teuthology.orchestra.run.smithi016:> ls /sys/fs/fuse/connections 2024-03-31T15:14:46.316 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-31T15:14:46.316 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-03-31T15:14:46.359 DEBUG:teuthology.orchestra.run.smithi016:> sudo modprobe fuse 2024-03-31T15:14:46.380 DEBUG:teuthology.orchestra.run.smithi016:> cat /proc/self/mounts | awk '{print $2}' 2024-03-31T15:14:46.433 INFO:teuthology.orchestra.run.smithi016.stdout:/sys 2024-03-31T15:14:46.433 INFO:teuthology.orchestra.run.smithi016.stdout:/proc 2024-03-31T15:14:46.433 INFO:teuthology.orchestra.run.smithi016.stdout:/dev 2024-03-31T15:14:46.433 INFO:teuthology.orchestra.run.smithi016.stdout:/dev/pts 2024-03-31T15:14:46.434 INFO:teuthology.orchestra.run.smithi016.stdout:/run 2024-03-31T15:14:46.434 INFO:teuthology.orchestra.run.smithi016.stdout:/ 2024-03-31T15:14:46.434 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/kernel/security 2024-03-31T15:14:46.434 INFO:teuthology.orchestra.run.smithi016.stdout:/dev/shm 2024-03-31T15:14:46.434 INFO:teuthology.orchestra.run.smithi016.stdout:/run/lock 2024-03-31T15:14:46.434 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/cgroup 2024-03-31T15:14:46.434 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/pstore 2024-03-31T15:14:46.434 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/bpf 2024-03-31T15:14:46.434 INFO:teuthology.orchestra.run.smithi016.stdout:/proc/sys/fs/binfmt_misc 2024-03-31T15:14:46.434 INFO:teuthology.orchestra.run.smithi016.stdout:/dev/hugepages 2024-03-31T15:14:46.434 INFO:teuthology.orchestra.run.smithi016.stdout:/dev/mqueue 2024-03-31T15:14:46.434 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/kernel/debug 2024-03-31T15:14:46.434 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/kernel/tracing 2024-03-31T15:14:46.434 INFO:teuthology.orchestra.run.smithi016.stdout:/proc/fs/nfsd 2024-03-31T15:14:46.435 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/fuse/connections 2024-03-31T15:14:46.435 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/kernel/config 2024-03-31T15:14:46.435 INFO:teuthology.orchestra.run.smithi016.stdout:/run/credentials/systemd-sysusers.service 2024-03-31T15:14:46.435 INFO:teuthology.orchestra.run.smithi016.stdout:/run/rpc_pipefs 2024-03-31T15:14:46.435 INFO:teuthology.orchestra.run.smithi016.stdout:/run/user/1000 2024-03-31T15:14:46.435 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/ceph 2024-03-31T15:14:46.435 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/ceph/osd/ceph-0 2024-03-31T15:14:46.435 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/ceph/osd/ceph-1 2024-03-31T15:14:46.435 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/ceph/osd/ceph-2 2024-03-31T15:14:46.435 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/ceph/osd/ceph-3 2024-03-31T15:14:46.435 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns 2024-03-31T15:14:46.435 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-31T15:14:46.435 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-31T15:14:46.435 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-31T15:14:46.436 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-31T15:14:46.436 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:14:46.437 DEBUG:teuthology.orchestra.run.smithi016:> ls /sys/fs/fuse/connections 2024-03-31T15:14:46.454 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi016.stderr:2024-03-31T15:14:46.461+0000 7f843e049580 -1 init, newargv = 0x55e99aa16b30 newargc=15 2024-03-31T15:14:46.454 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi016.stderr:ceph-fuse[52262]: starting ceph client 2024-03-31T15:14:46.461 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi016.stderr:ceph-fuse[52262]: starting fuse 2024-03-31T15:14:46.463 INFO:teuthology.orchestra.run.smithi016.stdout:43 2024-03-31T15:14:46.463 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [43] 2024-03-31T15:14:46.464 DEBUG:teuthology.orchestra.run.smithi016:> sudo stdin-killer -- python3 -c ' 2024-03-31T15:14:46.464 DEBUG:teuthology.orchestra.run.smithi016:> import glob 2024-03-31T15:14:46.464 DEBUG:teuthology.orchestra.run.smithi016:> import re 2024-03-31T15:14:46.464 DEBUG:teuthology.orchestra.run.smithi016:> import os 2024-03-31T15:14:46.464 DEBUG:teuthology.orchestra.run.smithi016:> import subprocess 2024-03-31T15:14:46.464 DEBUG:teuthology.orchestra.run.smithi016:> 2024-03-31T15:14:46.464 DEBUG:teuthology.orchestra.run.smithi016:> def _find_admin_socket(client_name): 2024-03-31T15:14:46.464 DEBUG:teuthology.orchestra.run.smithi016:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-03-31T15:14:46.464 DEBUG:teuthology.orchestra.run.smithi016:> files = glob.glob(asok_path) 2024-03-31T15:14:46.464 DEBUG:teuthology.orchestra.run.smithi016:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-03-31T15:14:46.464 DEBUG:teuthology.orchestra.run.smithi016:> 2024-03-31T15:14:46.464 DEBUG:teuthology.orchestra.run.smithi016:> # Given a non-glob path, it better be there 2024-03-31T15:14:46.464 DEBUG:teuthology.orchestra.run.smithi016:> if "*" not in asok_path: 2024-03-31T15:14:46.464 DEBUG:teuthology.orchestra.run.smithi016:> assert(len(files) == 1) 2024-03-31T15:14:46.465 DEBUG:teuthology.orchestra.run.smithi016:> return files[0] 2024-03-31T15:14:46.465 DEBUG:teuthology.orchestra.run.smithi016:> 2024-03-31T15:14:46.465 DEBUG:teuthology.orchestra.run.smithi016:> for f in files: 2024-03-31T15:14:46.465 DEBUG:teuthology.orchestra.run.smithi016:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-31T15:14:46.465 DEBUG:teuthology.orchestra.run.smithi016:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-31T15:14:46.465 DEBUG:teuthology.orchestra.run.smithi016:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-31T15:14:46.465 DEBUG:teuthology.orchestra.run.smithi016:> contents = proc_f.read() 2024-03-31T15:14:46.465 DEBUG:teuthology.orchestra.run.smithi016:> if mountpoint in contents: 2024-03-31T15:14:46.465 DEBUG:teuthology.orchestra.run.smithi016:> return f 2024-03-31T15:14:46.465 DEBUG:teuthology.orchestra.run.smithi016:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-31T15:14:46.465 DEBUG:teuthology.orchestra.run.smithi016:> 2024-03-31T15:14:46.465 DEBUG:teuthology.orchestra.run.smithi016:> print(_find_admin_socket("client.0")) 2024-03-31T15:14:46.465 DEBUG:teuthology.orchestra.run.smithi016:> ' 2024-03-31T15:14:46.586 INFO:teuthology.orchestra.run.smithi016.stdout:/var/run/ceph/ceph-client.0.52262.asok 2024-03-31T15:14:46.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:46 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-31T15:14:46.597 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.52262.asok 2024-03-31T15:14:46.597 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:14:46.598 DEBUG:teuthology.orchestra.run.smithi016:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.52262.asok status 2024-03-31T15:14:46.773 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-03-31T15:14:46.773 INFO:teuthology.orchestra.run.smithi016.stdout: "metadata": { 2024-03-31T15:14:46.774 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph_sha1": "d2271579d5e735cfd6d436b8349c004f06527cb0", 2024-03-31T15:14:46.774 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph_version": "ceph version 19.0.0-2574-gd2271579 (d2271579d5e735cfd6d436b8349c004f06527cb0) squid (dev)", 2024-03-31T15:14:46.774 INFO:teuthology.orchestra.run.smithi016.stdout: "entity_id": "0", 2024-03-31T15:14:46.774 INFO:teuthology.orchestra.run.smithi016.stdout: "hostname": "smithi016", 2024-03-31T15:14:46.774 INFO:teuthology.orchestra.run.smithi016.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-03-31T15:14:46.774 INFO:teuthology.orchestra.run.smithi016.stdout: "pid": "52262", 2024-03-31T15:14:46.774 INFO:teuthology.orchestra.run.smithi016.stdout: "root": "/" 2024-03-31T15:14:46.774 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-31T15:14:46.774 INFO:teuthology.orchestra.run.smithi016.stdout: "dentry_count": 0, 2024-03-31T15:14:46.774 INFO:teuthology.orchestra.run.smithi016.stdout: "dentry_pinned_count": 0, 2024-03-31T15:14:46.774 INFO:teuthology.orchestra.run.smithi016.stdout: "id": 9624, 2024-03-31T15:14:46.774 INFO:teuthology.orchestra.run.smithi016.stdout: "inst": { 2024-03-31T15:14:46.774 INFO:teuthology.orchestra.run.smithi016.stdout: "name": { 2024-03-31T15:14:46.774 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "client", 2024-03-31T15:14:46.775 INFO:teuthology.orchestra.run.smithi016.stdout: "num": 9624 2024-03-31T15:14:46.775 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-31T15:14:46.775 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": { 2024-03-31T15:14:46.775 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-03-31T15:14:46.775 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "192.168.0.1:0", 2024-03-31T15:14:46.775 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 3090767977 2024-03-31T15:14:46.775 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-31T15:14:46.775 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-31T15:14:46.775 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": { 2024-03-31T15:14:46.775 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-03-31T15:14:46.775 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "192.168.0.1:0", 2024-03-31T15:14:46.775 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 3090767977 2024-03-31T15:14:46.775 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-31T15:14:46.775 INFO:teuthology.orchestra.run.smithi016.stdout: "inst_str": "client.9624 192.168.0.1:0/3090767977", 2024-03-31T15:14:46.775 INFO:teuthology.orchestra.run.smithi016.stdout: "addr_str": "192.168.0.1:0/3090767977", 2024-03-31T15:14:46.775 INFO:teuthology.orchestra.run.smithi016.stdout: "inode_count": 1, 2024-03-31T15:14:46.776 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_epoch": 105, 2024-03-31T15:14:46.776 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_epoch": 112, 2024-03-31T15:14:46.776 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_epoch_barrier": 0, 2024-03-31T15:14:46.776 INFO:teuthology.orchestra.run.smithi016.stdout: "blocklisted": false, 2024-03-31T15:14:46.776 INFO:teuthology.orchestra.run.smithi016.stdout: "fs_name": "cephfs" 2024-03-31T15:14:46.776 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-03-31T15:14:46.793 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:14:46.793 DEBUG:teuthology.orchestra.run.smithi016:> stat --file-system '--printf=%T 2024-03-31T15:14:46.793 DEBUG:teuthology.orchestra.run.smithi016:> ' -- /home/ubuntu/cephtest/mnt.0 2024-03-31T15:14:46.802 INFO:teuthology.orchestra.run.smithi016.stdout:fuseblk 2024-03-31T15:14:46.802 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-03-31T15:14:46.803 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:14:46.803 DEBUG:teuthology.orchestra.run.smithi016:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-31T15:14:46.864 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:14:46.864 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec find ./) 2024-03-31T15:14:46.926 INFO:teuthology.orchestra.run.smithi016.stdout:./ 2024-03-31T15:14:46.926 INFO:teuthology.orchestra.run.smithi016.stdout:./parent 2024-03-31T15:14:46.926 INFO:teuthology.orchestra.run.smithi016.stdout:./parent/flushed 2024-03-31T15:14:46.926 INFO:teuthology.orchestra.run.smithi016.stdout:./parent/flushed/charlie 2024-03-31T15:14:46.927 INFO:teuthology.orchestra.run.smithi016.stdout:./parent/flushed/alpha 2024-03-31T15:14:46.927 INFO:teuthology.orchestra.run.smithi016.stdout:./parent/unflushed 2024-03-31T15:14:46.927 INFO:teuthology.orchestra.run.smithi016.stdout:./parent/unflushed/jfile 2024-03-31T15:14:46.928 DEBUG:teuthology.orchestra.run.smithi016:> stdin-killer -- python3 -c ' 2024-03-31T15:14:46.928 DEBUG:teuthology.orchestra.run.smithi016:> import os 2024-03-31T15:14:46.928 DEBUG:teuthology.orchestra.run.smithi016:> import stat 2024-03-31T15:14:46.928 DEBUG:teuthology.orchestra.run.smithi016:> 2024-03-31T15:14:46.928 DEBUG:teuthology.orchestra.run.smithi016:> print(os.stat("/home/ubuntu/cephtest/mnt.0/./").st_ino) 2024-03-31T15:14:46.928 DEBUG:teuthology.orchestra.run.smithi016:> ' 2024-03-31T15:14:47.035 INFO:teuthology.orchestra.run.smithi016.stdout:1 2024-03-31T15:14:47.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:47 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-31T15:14:47.044 DEBUG:teuthology.orchestra.run.smithi016:> stdin-killer -- python3 -c ' 2024-03-31T15:14:47.045 DEBUG:teuthology.orchestra.run.smithi016:> import os 2024-03-31T15:14:47.045 DEBUG:teuthology.orchestra.run.smithi016:> import stat 2024-03-31T15:14:47.045 DEBUG:teuthology.orchestra.run.smithi016:> 2024-03-31T15:14:47.045 DEBUG:teuthology.orchestra.run.smithi016:> print(os.stat("/home/ubuntu/cephtest/mnt.0/./parent").st_ino) 2024-03-31T15:14:47.045 DEBUG:teuthology.orchestra.run.smithi016:> ' 2024-03-31T15:14:47.154 INFO:teuthology.orchestra.run.smithi016.stdout:1099511627776 2024-03-31T15:14:47.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:47 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-31T15:14:47.164 DEBUG:teuthology.orchestra.run.smithi016:> stdin-killer -- python3 -c ' 2024-03-31T15:14:47.164 DEBUG:teuthology.orchestra.run.smithi016:> import os 2024-03-31T15:14:47.164 DEBUG:teuthology.orchestra.run.smithi016:> import stat 2024-03-31T15:14:47.165 DEBUG:teuthology.orchestra.run.smithi016:> 2024-03-31T15:14:47.165 DEBUG:teuthology.orchestra.run.smithi016:> print(os.stat("/home/ubuntu/cephtest/mnt.0/./parent/flushed").st_ino) 2024-03-31T15:14:47.165 DEBUG:teuthology.orchestra.run.smithi016:> ' 2024-03-31T15:14:47.276 INFO:teuthology.orchestra.run.smithi016.stdout:1099511627777 2024-03-31T15:14:47.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:47 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-31T15:14:47.285 DEBUG:teuthology.orchestra.run.smithi016:> stdin-killer -- python3 -c ' 2024-03-31T15:14:47.285 DEBUG:teuthology.orchestra.run.smithi016:> import os 2024-03-31T15:14:47.285 DEBUG:teuthology.orchestra.run.smithi016:> import stat 2024-03-31T15:14:47.285 DEBUG:teuthology.orchestra.run.smithi016:> 2024-03-31T15:14:47.285 DEBUG:teuthology.orchestra.run.smithi016:> print(os.stat("/home/ubuntu/cephtest/mnt.0/./parent/flushed/charlie").st_ino) 2024-03-31T15:14:47.285 DEBUG:teuthology.orchestra.run.smithi016:> ' 2024-03-31T15:14:47.400 INFO:teuthology.orchestra.run.smithi016.stdout:1099511628280 2024-03-31T15:14:47.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:47 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-31T15:14:47.408 DEBUG:teuthology.orchestra.run.smithi016:> stdin-killer -- python3 -c ' 2024-03-31T15:14:47.409 DEBUG:teuthology.orchestra.run.smithi016:> import os 2024-03-31T15:14:47.409 DEBUG:teuthology.orchestra.run.smithi016:> import stat 2024-03-31T15:14:47.409 DEBUG:teuthology.orchestra.run.smithi016:> 2024-03-31T15:14:47.409 DEBUG:teuthology.orchestra.run.smithi016:> print(os.stat("/home/ubuntu/cephtest/mnt.0/./parent/flushed/alpha").st_ino) 2024-03-31T15:14:47.409 DEBUG:teuthology.orchestra.run.smithi016:> ' 2024-03-31T15:14:47.518 INFO:teuthology.orchestra.run.smithi016.stdout:1099511627778 2024-03-31T15:14:47.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:47 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-31T15:14:47.527 DEBUG:teuthology.orchestra.run.smithi016:> stdin-killer -- python3 -c ' 2024-03-31T15:14:47.527 DEBUG:teuthology.orchestra.run.smithi016:> import os 2024-03-31T15:14:47.527 DEBUG:teuthology.orchestra.run.smithi016:> import stat 2024-03-31T15:14:47.527 DEBUG:teuthology.orchestra.run.smithi016:> 2024-03-31T15:14:47.528 DEBUG:teuthology.orchestra.run.smithi016:> print(os.stat("/home/ubuntu/cephtest/mnt.0/./parent/unflushed").st_ino) 2024-03-31T15:14:47.528 DEBUG:teuthology.orchestra.run.smithi016:> ' 2024-03-31T15:14:47.634 INFO:teuthology.orchestra.run.smithi016.stdout:1099511627779 2024-03-31T15:14:47.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:47 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-31T15:14:47.643 DEBUG:teuthology.orchestra.run.smithi016:> stdin-killer -- python3 -c ' 2024-03-31T15:14:47.643 DEBUG:teuthology.orchestra.run.smithi016:> import os 2024-03-31T15:14:47.643 DEBUG:teuthology.orchestra.run.smithi016:> import stat 2024-03-31T15:14:47.643 DEBUG:teuthology.orchestra.run.smithi016:> 2024-03-31T15:14:47.643 DEBUG:teuthology.orchestra.run.smithi016:> print(os.stat("/home/ubuntu/cephtest/mnt.0/./parent/unflushed/jfile").st_ino) 2024-03-31T15:14:47.643 DEBUG:teuthology.orchestra.run.smithi016:> ' 2024-03-31T15:14:47.751 INFO:teuthology.orchestra.run.smithi016.stdout:1099511627780 2024-03-31T15:14:47.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:47 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-31T15:14:47.760 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-03-31T15:14:47.761 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-31T15:14:47.818 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-31T15:14:48.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:48.005+0000 7f5f45e00640 1 Processor -- start 2024-03-31T15:14:48.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:48.005+0000 7f5f45e00640 1 -- start start 2024-03-31T15:14:48.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:48.009+0000 7f5f45e00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f5f4006f920 0x7f5f4006fd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:48.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:48.009+0000 7f5f45e00640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5f400d0920 con 0x7f5f40145e40 2024-03-31T15:14:48.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:48.009+0000 7f5f45e00640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5f400d0c00 con 0x7f5f400d2010 2024-03-31T15:14:48.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:48.009+0000 7f5f45e00640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f5f400702c0 con 0x7f5f4006f920 2024-03-31T15:14:48.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:48.009+0000 7f5f3ea00640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f5f40145e40 0x7f5f400cc740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:55410/0 (socket says 172.21.15.16:55410) 2024-03-31T15:14:48.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:48.009+0000 7f5f3ea00640 1 -- 172.21.15.16:0/2648753519 learned_addr learned my addr 172.21.15.16:0/2648753519 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:14:48.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:48.009+0000 7f5f36000640 1 --2- 172.21.15.16:0/2648753519 >> v2:172.21.15.89:3300/0 conn(0x7f5f4006f920 0x7f5f4006fd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:48.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:48.009+0000 7f5f3e000640 1 -- 172.21.15.16:0/2648753519 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3447861027 0 0) 0x7f5f400d0920 con 0x7f5f40145e40 2024-03-31T15:14:48.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:48.009+0000 7f5f3e000640 1 -- 172.21.15.16:0/2648753519 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5f24003530 con 0x7f5f40145e40 2024-03-31T15:14:48.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:48.009+0000 7f5f3e000640 1 -- 172.21.15.16:0/2648753519 <== mon.1 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2405697588 0 0) 0x7f5f400d0c00 con 0x7f5f400d2010 2024-03-31T15:14:48.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:48.009+0000 7f5f3e000640 1 -- 172.21.15.16:0/2648753519 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5f400d0920 con 0x7f5f400d2010 2024-03-31T15:14:48.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:48.009+0000 7f5f3e000640 1 -- 172.21.15.16:0/2648753519 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1518284820 0 0) 0x7f5f24003530 con 0x7f5f40145e40 2024-03-31T15:14:48.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:48.009+0000 7f5f36000640 1 -- 172.21.15.16:0/2648753519 >> v1:172.21.15.89:6789/0 conn(0x7f5f400d2010 legacy=0x7f5f400d24a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:48.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:48.009+0000 7f5f36000640 1 -- 172.21.15.16:0/2648753519 >> v1:172.21.15.16:6789/0 conn(0x7f5f40145e40 legacy=0x7f5f400cc740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:48.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:48.009+0000 7f5f36000640 1 -- 172.21.15.16:0/2648753519 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5f400d1810 con 0x7f5f4006f920 2024-03-31T15:14:48.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:48.013+0000 7f5f36000640 1 --2- 172.21.15.16:0/2648753519 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5f4006f920 0x7f5f4006fd80 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f5f28009d90 tx=0x7f5f2801c890 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e5bdf5971d990071 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:48.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:48.013+0000 7f5f3e000640 1 -- 172.21.15.16:0/2648753519 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f30001680 con 0x7f5f4006f920 2024-03-31T15:14:48.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:48.013+0000 7f5f3e000640 1 -- 172.21.15.16:0/2648753519 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5f280a64e0 con 0x7f5f4006f920 2024-03-31T15:14:48.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:48.013+0000 7f5f3e000640 1 -- 172.21.15.16:0/2648753519 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f280a68b0 con 0x7f5f4006f920 2024-03-31T15:14:48.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:48.013+0000 7f5f45e00640 1 -- 172.21.15.16:0/2648753519 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5f4006f920 msgr2=0x7f5f4006fd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:48.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:48.013+0000 7f5f45e00640 1 --2- 172.21.15.16:0/2648753519 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5f4006f920 0x7f5f4006fd80 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f5f28009d90 tx=0x7f5f2801c890 comp rx=0 tx=0).stop 2024-03-31T15:14:48.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:48.013+0000 7f5f45e00640 1 -- 172.21.15.16:0/2648753519 shutdown_connections 2024-03-31T15:14:48.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:48.013+0000 7f5f45e00640 1 --2- 172.21.15.16:0/2648753519 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5f4006f920 0x7f5f4006fd80 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:48.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:48.013+0000 7f5f45e00640 1 -- 172.21.15.16:0/2648753519 >> 172.21.15.16:0/2648753519 conn(0x7f5f40067330 msgr2=0x7f5f40067740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:14:48.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:48.013+0000 7f5f45e00640 1 -- 172.21.15.16:0/2648753519 shutdown_connections 2024-03-31T15:14:48.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:48.013+0000 7f5f45e00640 1 -- 172.21.15.16:0/2648753519 wait complete. 2024-03-31T15:14:48.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:48.013+0000 7f5f45e00640 1 Processor -- start 2024-03-31T15:14:48.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:48.013+0000 7f5f45e00640 1 -- start start 2024-03-31T15:14:48.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:48.017+0000 7f5f45e00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5f4006f920 0x7f5f4015bff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:48.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:48.017+0000 7f5f45e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5f400d2010 0x7f5f4013fbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:48.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:48.017+0000 7f5f45e00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f5f40145e40 0x7f5f4015fd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:48.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:48.017+0000 7f5f45e00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f5f400d0ee0 con 0x7f5f400d2010 2024-03-31T15:14:48.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:48.017+0000 7f5f45e00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f5f400d06d0 con 0x7f5f4006f920 2024-03-31T15:14:48.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:48.017+0000 7f5f45e00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f5f400b50c0 con 0x7f5f40145e40 2024-03-31T15:14:48.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:48.017+0000 7f5f36000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5f400d2010 0x7f5f4013fbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:48.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:48.017+0000 7f5f36000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5f400d2010 0x7f5f4013fbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55268/0 (socket says 172.21.15.16:55268) 2024-03-31T15:14:48.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:48.017+0000 7f5f36000640 1 -- 172.21.15.16:0/1729031934 learned_addr learned my addr 172.21.15.16:0/1729031934 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:14:48.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:48.017+0000 7f5f3ea00640 1 --2- 172.21.15.16:0/1729031934 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5f4006f920 0x7f5f4015bff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:48.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:48.017+0000 7f5f3f400640 1 --2- 172.21.15.16:0/1729031934 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f5f40145e40 0x7f5f4015fd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:48.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:48.017+0000 7f5f36000640 1 -- 172.21.15.16:0/1729031934 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f5f40145e40 msgr2=0x7f5f4015fd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:48.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:48.017+0000 7f5f36000640 1 --2- 172.21.15.16:0/1729031934 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f5f40145e40 0x7f5f4015fd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:48.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:48.017+0000 7f5f36000640 1 -- 172.21.15.16:0/1729031934 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5f4006f920 msgr2=0x7f5f4015bff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:48.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:48.017+0000 7f5f36000640 1 --2- 172.21.15.16:0/1729031934 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5f4006f920 0x7f5f4015bff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:48.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:48.017+0000 7f5f36000640 1 -- 172.21.15.16:0/1729031934 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5f40140120 con 0x7f5f400d2010 2024-03-31T15:14:48.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:48.017+0000 7f5f36000640 1 --2- 172.21.15.16:0/1729031934 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5f400d2010 0x7f5f4013fbe0 secure :-1 s=READY pgs=1762 cs=0 l=1 rev1=1 crypto rx=0x7f5f2801a4c0 tx=0x7f5f28019630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:48.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:48.017+0000 7f5f37e00640 1 -- 172.21.15.16:0/1729031934 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f28009450 con 0x7f5f400d2010 2024-03-31T15:14:48.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:48.017+0000 7f5f37e00640 1 -- 172.21.15.16:0/1729031934 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5f28003bc0 con 0x7f5f400d2010 2024-03-31T15:14:48.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:48.017+0000 7f5f45e00640 1 -- 172.21.15.16:0/1729031934 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5f4015ebc0 con 0x7f5f400d2010 2024-03-31T15:14:48.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:48.017+0000 7f5f37e00640 1 -- 172.21.15.16:0/1729031934 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f280a6e60 con 0x7f5f400d2010 2024-03-31T15:14:48.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:48.017+0000 7f5f45e00640 1 -- 172.21.15.16:0/1729031934 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5f4015f180 con 0x7f5f400d2010 2024-03-31T15:14:48.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:48.017+0000 7f5f45e00640 1 -- 172.21.15.16:0/1729031934 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5f40069950 con 0x7f5f400d2010 2024-03-31T15:14:48.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:48.021+0000 7f5f37e00640 1 -- 172.21.15.16:0/1729031934 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f5f280b8640 con 0x7f5f400d2010 2024-03-31T15:14:48.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:48.021+0000 7f5f37e00640 1 --2- 172.21.15.16:0/1729031934 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f5f1c0683b0 0x7f5f1c06a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:48.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:48.021+0000 7f5f37e00640 1 -- 172.21.15.16:0/1729031934 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(112..112 src has 1..112) ==== 5853+0+0 (secure 0 0 0) 0x7f5f281212c0 con 0x7f5f400d2010 2024-03-31T15:14:48.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:48.021+0000 7f5f3ea00640 1 --2- 172.21.15.16:0/1729031934 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f5f1c0683b0 0x7f5f1c06a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:48.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:48.021+0000 7f5f3ea00640 1 --2- 172.21.15.16:0/1729031934 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f5f1c0683b0 0x7f5f1c06a870 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f5f30000fe0 tx=0x7f5f30001720 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:48.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:48.025+0000 7f5f37e00640 1 -- 172.21.15.16:0/1729031934 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f5f28016480 con 0x7f5f400d2010 2024-03-31T15:14:48.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:48.189+0000 7f5f45e00640 1 -- 172.21.15.16:0/1729031934 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f5f400b79e0 con 0x7f5f400d2010 2024-03-31T15:14:48.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:48.193+0000 7f5f37e00640 1 -- 172.21.15.16:0/1729031934 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v112) ==== 74+0+16777 (secure 0 0 0) 0x7f5f280f1230 con 0x7f5f400d2010 2024-03-31T15:14:48.186 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:14:48.186 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":112,"fsid":"286eb5da-fb47-4550-b290-d9519c13b9a5","created":"2024-03-31T15:03:11.438952+0000","modified":"2024-03-31T15:14:39.715636+0000","last_up_change":"2024-03-31T15:03:22.885535+0000","last_in_change":"2024-03-31T15:03:13.941667+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-31T15:03:22.550787+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-03-31T15:14:06.417304+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"108","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-03-31T15:14:07.622562+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"109","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"92a66450-ef19-40d8-ac3d-d760b23b0bda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6801","nonce":932712244}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6803","nonce":932712244}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6807","nonce":932712244}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6805","nonce":932712244}]},"public_addr":"172.21.15.16:6801/932712244","cluster_addr":"172.21.15.16:6803/932712244","heartbeat_back_addr":"172.21.15.16:6807/932712244","heartbeat_front_addr":"172.21.15.16:6805/932712244","state":["exists","up"]},{"osd":1,"uuid":"da9d1f2b-1968-4959-b843-faa231988e92","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6810","nonce":1398737533}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6811","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6813","nonce":1398737533}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6821","nonce":1398737533}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6815","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6816","nonce":1398737533}]},"public_addr":"172.21.15.16:6810/1398737533","cluster_addr":"172.21.15.16:6813/1398737533","heartbeat_back_addr":"172.21.15.16:6821/1398737533","heartbeat_front_addr":"172.21.15.16:6816/1398737533","state":["exists","up"]},{"osd":2,"uuid":"01a73662-9a21-4b1f-bcbb-42b9636a4638","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6819","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6822","nonce":3018526646}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6825","nonce":3018526646}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6831","nonce":3018526646}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6827","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6829","nonce":3018526646}]},"public_addr":"172.21.15.16:6822/3018526646","cluster_addr":"172.21.15.16:6825/3018526646","heartbeat_back_addr":"172.21.15.16:6831/3018526646","heartbeat_front_addr":"172.21.15.16:6829/3018526646","state":["exists","up"]},{"osd":3,"uuid":"c7771fd3-e18e-4500-ac38-23cf10ebcd75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6809","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6812","nonce":3360352974}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6817","nonce":3360352974}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6828","nonce":3360352974}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6823","nonce":3360352974}]},"public_addr":"172.21.15.16:6812/3360352974","cluster_addr":"172.21.15.16:6817/3360352974","heartbeat_back_addr":"172.21.15.16:6828/3360352974","heartbeat_front_addr":"172.21.15.16:6823/3360352974","state":["exists","up"]},{"osd":4,"uuid":"88260730-1482-4fbd-a40f-cb1124e9df4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6801","nonce":1856844856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6803","nonce":1856844856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6807","nonce":1856844856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6805","nonce":1856844856}]},"public_addr":"172.21.15.89:6801/1856844856","cluster_addr":"172.21.15.89:6803/1856844856","heartbeat_back_addr":"172.21.15.89:6807/1856844856","heartbeat_front_addr":"172.21.15.89:6805/1856844856","state":["exists","up"]},{"osd":5,"uuid":"4cb2c00c-fae0-4dd8-9d27-ec4f0093ccf7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6814","nonce":50395527}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6817","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6820","nonce":50395527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6831","nonce":50395527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6823","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6826","nonce":50395527}]},"public_addr":"172.21.15.89:6814/50395527","cluster_addr":"172.21.15.89:6820/50395527","heartbeat_back_addr":"172.21.15.89:6831/50395527","heartbeat_front_addr":"172.21.15.89:6826/50395527","state":["exists","up"]},{"osd":6,"uuid":"81ce0c19-2554-4cae-acbe-66a05e9e9b9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6809","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6812","nonce":3470471752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6815","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6818","nonce":3470471752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6827","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6829","nonce":3470471752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6821","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6824","nonce":3470471752}]},"public_addr":"172.21.15.89:6812/3470471752","cluster_addr":"172.21.15.89:6818/3470471752","heartbeat_back_addr":"172.21.15.89:6829/3470471752","heartbeat_front_addr":"172.21.15.89:6824/3470471752","state":["exists","up"]},{"osd":7,"uuid":"85e4313b-d37d-442e-8042-153bf20a847d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6811","nonce":1796778630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6813","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6816","nonce":1796778630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6825","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6828","nonce":1796778630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6819","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6822","nonce":1796778630}]},"public_addr":"172.21.15.89:6811/1796778630","cluster_addr":"172.21.15.89:6816/1796778630","heartbeat_back_addr":"172.21.15.89:6828/1796778630","heartbeat_front_addr":"172.21.15.89:6822/1796778630","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.439088+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.529970+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.680512+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:21.044161+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.582111+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.536558+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.409522+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.590229+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"17.1","mappings":[{"from":0,"to":5}]},{"pgid":"17.21","mappings":[{"from":0,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:6835/2564952016":"2024-04-01T15:14:39.715615+0000","172.21.15.16:6833/2564952016":"2024-04-01T15:14:39.715615+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-31T15:14:48.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:48.193+0000 7f5f45e00640 1 -- 172.21.15.16:0/1729031934 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f5f1c0683b0 msgr2=0x7f5f1c06a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:48.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:48.193+0000 7f5f45e00640 1 --2- 172.21.15.16:0/1729031934 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f5f1c0683b0 0x7f5f1c06a870 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f5f30000fe0 tx=0x7f5f30001720 comp rx=0 tx=0).stop 2024-03-31T15:14:48.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:48.193+0000 7f5f45e00640 1 -- 172.21.15.16:0/1729031934 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5f400d2010 msgr2=0x7f5f4013fbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:48.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:48.193+0000 7f5f45e00640 1 --2- 172.21.15.16:0/1729031934 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5f400d2010 0x7f5f4013fbe0 secure :-1 s=READY pgs=1762 cs=0 l=1 rev1=1 crypto rx=0x7f5f2801a4c0 tx=0x7f5f28019630 comp rx=0 tx=0).stop 2024-03-31T15:14:48.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:48.193+0000 7f5f45e00640 1 -- 172.21.15.16:0/1729031934 shutdown_connections 2024-03-31T15:14:48.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:48.193+0000 7f5f45e00640 1 --2- 172.21.15.16:0/1729031934 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f5f1c0683b0 0x7f5f1c06a870 unknown :-1 s=CLOSED pgs=745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:48.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:48.193+0000 7f5f45e00640 1 --2- 172.21.15.16:0/1729031934 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f5f40145e40 0x7f5f4015fd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:48.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:48.193+0000 7f5f45e00640 1 --2- 172.21.15.16:0/1729031934 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5f400d2010 0x7f5f4013fbe0 unknown :-1 s=CLOSED pgs=1762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:48.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:48.193+0000 7f5f45e00640 1 --2- 172.21.15.16:0/1729031934 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5f4006f920 0x7f5f4015bff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:48.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:48.193+0000 7f5f45e00640 1 -- 172.21.15.16:0/1729031934 >> 172.21.15.16:0/1729031934 conn(0x7f5f40067330 msgr2=0x7f5f400d2480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:14:48.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:48.193+0000 7f5f45e00640 1 -- 172.21.15.16:0/1729031934 shutdown_connections 2024-03-31T15:14:48.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:48.193+0000 7f5f45e00640 1 -- 172.21.15.16:0/1729031934 wait complete. 2024-03-31T15:14:48.207 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-31T15:03:22.550787+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 16, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-31T15:14:06.417304+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '108', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 17, 'pool_name': 'cephfs_data', 'create_time': '2024-03-31T15:14:07.622562+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '109', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-31T15:14:48.208 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-31T15:14:48.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:48.361+0000 7f95b9800640 1 Processor -- start 2024-03-31T15:14:48.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:48.361+0000 7f95b9800640 1 -- start start 2024-03-31T15:14:48.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:48.361+0000 7f95b9800640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f95b4165150 0x7f95b4167540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:48.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:48.361+0000 7f95b9800640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f95b415c7e0 con 0x7f95b406d4d0 2024-03-31T15:14:48.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:48.361+0000 7f95b9800640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f95b40736b0 con 0x7f95b41567e0 2024-03-31T15:14:48.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:48.361+0000 7f95b9800640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f95b415caa0 con 0x7f95b4165150 2024-03-31T15:14:48.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:48.361+0000 7f95b2000640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f95b406d4d0 0x7f95b406d930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:55426/0 (socket says 172.21.15.16:55426) 2024-03-31T15:14:48.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:48.361+0000 7f95b2000640 1 -- 172.21.15.16:0/2138945481 learned_addr learned my addr 172.21.15.16:0/2138945481 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:14:48.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:48.361+0000 7f95b3400640 1 --2- 172.21.15.16:0/2138945481 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f95b4165150 0x7f95b4167540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:48.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:48.361+0000 7f95b1600640 1 -- 172.21.15.16:0/2138945481 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2296949165 0 0) 0x7f95b415c7e0 con 0x7f95b406d4d0 2024-03-31T15:14:48.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:48.361+0000 7f95b1600640 1 -- 172.21.15.16:0/2138945481 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9598003530 con 0x7f95b406d4d0 2024-03-31T15:14:48.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:48.361+0000 7f95b1600640 1 -- 172.21.15.16:0/2138945481 <== mon.1 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2724350379 0 0) 0x7f95b40736b0 con 0x7f95b41567e0 2024-03-31T15:14:48.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:48.361+0000 7f95b1600640 1 -- 172.21.15.16:0/2138945481 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f95b415c7e0 con 0x7f95b41567e0 2024-03-31T15:14:48.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:48.361+0000 7f95b3400640 1 -- 172.21.15.16:0/2138945481 >> v1:172.21.15.89:6789/0 conn(0x7f95b41567e0 legacy=0x7f95b4072c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:48.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:48.361+0000 7f95b3400640 1 -- 172.21.15.16:0/2138945481 >> v1:172.21.15.16:6789/0 conn(0x7f95b406d4d0 legacy=0x7f95b406d930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:48.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:48.361+0000 7f95b3400640 1 -- 172.21.15.16:0/2138945481 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95b415cd80 con 0x7f95b4165150 2024-03-31T15:14:48.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:48.365+0000 7f95b3400640 1 --2- 172.21.15.16:0/2138945481 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f95b4165150 0x7f95b4167540 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f95ac009dc0 tx=0x7f95ac01c990 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7a1ec4040b347aaa server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:48.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:48.365+0000 7f95b1600640 1 -- 172.21.15.16:0/2138945481 <== mon.2 v2:172.21.15.89:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95ac0b2070 con 0x7f95b4165150 2024-03-31T15:14:48.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:48.365+0000 7f95b1600640 1 -- 172.21.15.16:0/2138945481 <== mon.2 v2:172.21.15.89:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f95ac0a5850 con 0x7f95b4165150 2024-03-31T15:14:48.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:48.365+0000 7f95b1600640 1 -- 172.21.15.16:0/2138945481 <== mon.2 v2:172.21.15.89:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95ac0a5c20 con 0x7f95b4165150 2024-03-31T15:14:48.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:48.365+0000 7f95b9800640 1 -- 172.21.15.16:0/2138945481 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f95b4165150 msgr2=0x7f95b4167540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:48.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:48.365+0000 7f95b9800640 1 --2- 172.21.15.16:0/2138945481 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f95b4165150 0x7f95b4167540 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f95ac009dc0 tx=0x7f95ac01c990 comp rx=0 tx=0).stop 2024-03-31T15:14:48.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:48.365+0000 7f95b9800640 1 -- 172.21.15.16:0/2138945481 shutdown_connections 2024-03-31T15:14:48.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:48.365+0000 7f95b9800640 1 --2- 172.21.15.16:0/2138945481 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f95b4165150 0x7f95b4167540 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:48.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:48.365+0000 7f95b9800640 1 -- 172.21.15.16:0/2138945481 >> 172.21.15.16:0/2138945481 conn(0x7f95b406ee40 msgr2=0x7f95b406b620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:14:48.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:48.365+0000 7f95b9800640 1 -- 172.21.15.16:0/2138945481 shutdown_connections 2024-03-31T15:14:48.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:48.365+0000 7f95b9800640 1 -- 172.21.15.16:0/2138945481 wait complete. 2024-03-31T15:14:48.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:48.369+0000 7f95b9800640 1 Processor -- start 2024-03-31T15:14:48.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:48.369+0000 7f95b9800640 1 -- start start 2024-03-31T15:14:48.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:48.369+0000 7f95b9800640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f95b406d4d0 0x7f95b415ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:48.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:48.369+0000 7f95b9800640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f95b41567e0 0x7f95b4152b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:48.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:48.369+0000 7f95b9800640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f95b4165150 0x7f95b41661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:48.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:48.369+0000 7f95b9800640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f95b415d6e0 con 0x7f95b41567e0 2024-03-31T15:14:48.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:48.369+0000 7f95b9800640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f95b415d560 con 0x7f95b406d4d0 2024-03-31T15:14:48.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:48.369+0000 7f95b9800640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f95b416a6b0 con 0x7f95b4165150 2024-03-31T15:14:48.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:48.369+0000 7f95b2000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f95b41567e0 0x7f95b4152b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:48.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:48.369+0000 7f95b2000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f95b41567e0 0x7f95b4152b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55270/0 (socket says 172.21.15.16:55270) 2024-03-31T15:14:48.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:48.369+0000 7f95b2000640 1 -- 172.21.15.16:0/4283232358 learned_addr learned my addr 172.21.15.16:0/4283232358 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:14:48.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:48.369+0000 7f95b3400640 1 --2- 172.21.15.16:0/4283232358 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f95b4165150 0x7f95b41661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:48.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:48.369+0000 7f95b2a00640 1 --2- 172.21.15.16:0/4283232358 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f95b406d4d0 0x7f95b415ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:48.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:48.369+0000 7f95b2000640 1 -- 172.21.15.16:0/4283232358 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f95b4165150 msgr2=0x7f95b41661b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:48.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:48.369+0000 7f95b2000640 1 --2- 172.21.15.16:0/4283232358 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f95b4165150 0x7f95b41661b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:48.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:48.369+0000 7f95b2000640 1 -- 172.21.15.16:0/4283232358 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f95b406d4d0 msgr2=0x7f95b415ae10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:48.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:48.369+0000 7f95b2000640 1 --2- 172.21.15.16:0/4283232358 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f95b406d4d0 0x7f95b415ae10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:48.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:48.369+0000 7f95b2000640 1 -- 172.21.15.16:0/4283232358 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95b415b3f0 con 0x7f95b41567e0 2024-03-31T15:14:48.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:48.369+0000 7f95b2a00640 1 --2- 172.21.15.16:0/4283232358 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f95b406d4d0 0x7f95b415ae10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:14:48.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:48.369+0000 7f95b2000640 1 --2- 172.21.15.16:0/4283232358 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f95b41567e0 0x7f95b4152b60 secure :-1 s=READY pgs=1764 cs=0 l=1 rev1=1 crypto rx=0x7f95a000c4d0 tx=0x7f95a000c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:48.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:48.369+0000 7f95aaa00640 1 -- 172.21.15.16:0/4283232358 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95a0017070 con 0x7f95b41567e0 2024-03-31T15:14:48.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:48.369+0000 7f95aaa00640 1 -- 172.21.15.16:0/4283232358 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f95a0005cf0 con 0x7f95b41567e0 2024-03-31T15:14:48.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:48.369+0000 7f95aaa00640 1 -- 172.21.15.16:0/4283232358 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95a00072b0 con 0x7f95b41567e0 2024-03-31T15:14:48.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:48.369+0000 7f95b9800640 1 -- 172.21.15.16:0/4283232358 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f95b415b6d0 con 0x7f95b41567e0 2024-03-31T15:14:48.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:48.369+0000 7f95b9800640 1 -- 172.21.15.16:0/4283232358 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f95b415bc90 con 0x7f95b41567e0 2024-03-31T15:14:48.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:48.373+0000 7f95b9800640 1 -- 172.21.15.16:0/4283232358 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f95b4069950 con 0x7f95b41567e0 2024-03-31T15:14:48.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:48.373+0000 7f95aaa00640 1 -- 172.21.15.16:0/4283232358 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f95a001a070 con 0x7f95b41567e0 2024-03-31T15:14:48.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:48.373+0000 7f95aaa00640 1 --2- 172.21.15.16:0/4283232358 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f9594068380 0x7f959406a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:48.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:48.373+0000 7f95aaa00640 1 -- 172.21.15.16:0/4283232358 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(112..112 src has 1..112) ==== 5853+0+0 (secure 0 0 0) 0x7f95a0085d20 con 0x7f95b41567e0 2024-03-31T15:14:48.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:48.373+0000 7f95b2a00640 1 --2- 172.21.15.16:0/4283232358 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f9594068380 0x7f959406a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:48.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:48.373+0000 7f95b2a00640 1 --2- 172.21.15.16:0/4283232358 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f9594068380 0x7f959406a840 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f95a40011f0 tx=0x7f95a4009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:48.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:48.377+0000 7f95aaa00640 1 -- 172.21.15.16:0/4283232358 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f95a0055c90 con 0x7f95b41567e0 2024-03-31T15:14:48.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:48.541+0000 7f95b9800640 1 -- 172.21.15.16:0/4283232358 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f95b406e470 con 0x7f95b41567e0 2024-03-31T15:14:48.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:48.541+0000 7f95aaa00640 1 -- 172.21.15.16:0/4283232358 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v112) ==== 93+0+10 (secure 0 0 0) 0x7f95a0059f90 con 0x7f95b41567e0 2024-03-31T15:14:48.536 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2024-03-31T15:14:48.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:48.545+0000 7f95b9800640 1 -- 172.21.15.16:0/4283232358 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f9594068380 msgr2=0x7f959406a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:48.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:48.545+0000 7f95b9800640 1 --2- 172.21.15.16:0/4283232358 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f9594068380 0x7f959406a840 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f95a40011f0 tx=0x7f95a4009040 comp rx=0 tx=0).stop 2024-03-31T15:14:48.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:48.545+0000 7f95b9800640 1 -- 172.21.15.16:0/4283232358 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f95b41567e0 msgr2=0x7f95b4152b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:48.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:48.545+0000 7f95b9800640 1 --2- 172.21.15.16:0/4283232358 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f95b41567e0 0x7f95b4152b60 secure :-1 s=READY pgs=1764 cs=0 l=1 rev1=1 crypto rx=0x7f95a000c4d0 tx=0x7f95a000c990 comp rx=0 tx=0).stop 2024-03-31T15:14:48.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:48.545+0000 7f95b9800640 1 -- 172.21.15.16:0/4283232358 shutdown_connections 2024-03-31T15:14:48.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:48.545+0000 7f95b9800640 1 --2- 172.21.15.16:0/4283232358 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f9594068380 0x7f959406a840 unknown :-1 s=CLOSED pgs=746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:48.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:48.545+0000 7f95b9800640 1 --2- 172.21.15.16:0/4283232358 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f95b4165150 0x7f95b41661b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:48.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:48.545+0000 7f95b9800640 1 --2- 172.21.15.16:0/4283232358 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f95b41567e0 0x7f95b4152b60 unknown :-1 s=CLOSED pgs=1764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:48.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:48.545+0000 7f95b9800640 1 --2- 172.21.15.16:0/4283232358 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f95b406d4d0 0x7f95b415ae10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:48.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:48.545+0000 7f95b9800640 1 -- 172.21.15.16:0/4283232358 >> 172.21.15.16:0/4283232358 conn(0x7f95b406ee40 msgr2=0x7f95b40aec60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:14:48.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:48.545+0000 7f95b9800640 1 -- 172.21.15.16:0/4283232358 shutdown_connections 2024-03-31T15:14:48.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:48.545+0000 7f95b9800640 1 -- 172.21.15.16:0/4283232358 wait complete. 2024-03-31T15:14:48.556 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-31T15:14:48.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:48.705+0000 7f0784200640 1 Processor -- start 2024-03-31T15:14:48.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:48.705+0000 7f0784200640 1 -- start start 2024-03-31T15:14:48.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:48.709+0000 7f0784200640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f077c06f920 0x7f077c06fd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:48.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:48.709+0000 7f0784200640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f077c0d0a60 0x7f077c0cea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:48.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:48.709+0000 7f0784200640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f077c070240 con 0x7f077c0cef80 2024-03-31T15:14:48.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:48.709+0000 7f0784200640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f077c13f670 con 0x7f077c0d0a60 2024-03-31T15:14:48.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:48.709+0000 7f0784200640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f077c15ca70 con 0x7f077c06f920 2024-03-31T15:14:48.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:48.709+0000 7f0782200640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f077c0cef80 0x7f077c070e60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:55440/0 (socket says 172.21.15.16:55440) 2024-03-31T15:14:48.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:48.709+0000 7f0782200640 1 -- 172.21.15.16:0/2130617074 learned_addr learned my addr 172.21.15.16:0/2130617074 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:14:48.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:48.709+0000 7f0780e00640 1 --2- 172.21.15.16:0/2130617074 >> v2:172.21.15.89:3300/0 conn(0x7f077c0d0a60 0x7f077c0cea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:48.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:48.709+0000 7f0781800640 1 --2- 172.21.15.16:0/2130617074 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f077c06f920 0x7f077c06fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:48.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:48.709+0000 7f077be00640 1 -- 172.21.15.16:0/2130617074 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 487595023 0 0) 0x7f077c070240 con 0x7f077c0cef80 2024-03-31T15:14:48.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:48.709+0000 7f077be00640 1 -- 172.21.15.16:0/2130617074 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0764003530 con 0x7f077c0cef80 2024-03-31T15:14:48.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:48.709+0000 7f0780e00640 1 -- 172.21.15.16:0/2130617074 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f077c06f920 msgr2=0x7f077c06fd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:14:48.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:48.709+0000 7f0780e00640 1 --2- 172.21.15.16:0/2130617074 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f077c06f920 0x7f077c06fd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:48.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:48.709+0000 7f0780e00640 1 -- 172.21.15.16:0/2130617074 >> v1:172.21.15.16:6789/0 conn(0x7f077c0cef80 legacy=0x7f077c070e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:48.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:48.709+0000 7f0780e00640 1 -- 172.21.15.16:0/2130617074 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f077c15cd50 con 0x7f077c0d0a60 2024-03-31T15:14:48.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:48.713+0000 7f0780e00640 1 --2- 172.21.15.16:0/2130617074 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f077c0d0a60 0x7f077c0cea40 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f07680097c0 tx=0x7f076801c4a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=92cc461bf6eefa37 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:48.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:48.713+0000 7f077be00640 1 -- 172.21.15.16:0/2130617074 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07680b0070 con 0x7f077c0d0a60 2024-03-31T15:14:48.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:48.713+0000 7f077be00640 1 -- 172.21.15.16:0/2130617074 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f07680a4e50 con 0x7f077c0d0a60 2024-03-31T15:14:48.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:48.713+0000 7f077be00640 1 -- 172.21.15.16:0/2130617074 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07680ab820 con 0x7f077c0d0a60 2024-03-31T15:14:48.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:48.713+0000 7f0784200640 1 -- 172.21.15.16:0/2130617074 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f077c0d0a60 msgr2=0x7f077c0cea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:48.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:48.713+0000 7f0784200640 1 --2- 172.21.15.16:0/2130617074 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f077c0d0a60 0x7f077c0cea40 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f07680097c0 tx=0x7f076801c4a0 comp rx=0 tx=0).stop 2024-03-31T15:14:48.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:48.713+0000 7f0784200640 1 -- 172.21.15.16:0/2130617074 shutdown_connections 2024-03-31T15:14:48.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:48.713+0000 7f0784200640 1 --2- 172.21.15.16:0/2130617074 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f077c0d0a60 0x7f077c0cea40 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:48.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:48.713+0000 7f0784200640 1 --2- 172.21.15.16:0/2130617074 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f077c06f920 0x7f077c06fd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:48.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:48.713+0000 7f0784200640 1 -- 172.21.15.16:0/2130617074 >> 172.21.15.16:0/2130617074 conn(0x7f077c067930 msgr2=0x7f077c0b3ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:14:48.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:48.713+0000 7f0784200640 1 -- 172.21.15.16:0/2130617074 shutdown_connections 2024-03-31T15:14:48.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:48.713+0000 7f0784200640 1 -- 172.21.15.16:0/2130617074 wait complete. 2024-03-31T15:14:48.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:48.713+0000 7f0784200640 1 Processor -- start 2024-03-31T15:14:48.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:48.713+0000 7f0784200640 1 -- start start 2024-03-31T15:14:48.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:48.717+0000 7f0784200640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f077c06f920 0x7f077c1579d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:48.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:48.717+0000 7f0784200640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f077c0cef80 0x7f077c13fa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:48.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:48.717+0000 7f0784200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f077c0d0a60 0x7f077c15e800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:48.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:48.717+0000 7f0784200640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f077c1690b0 con 0x7f077c0d0a60 2024-03-31T15:14:48.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:48.717+0000 7f0784200640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f077c0cf390 con 0x7f077c06f920 2024-03-31T15:14:48.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:48.717+0000 7f0784200640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f077c169230 con 0x7f077c0cef80 2024-03-31T15:14:48.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:48.717+0000 7f0782200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f077c0d0a60 0x7f077c15e800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:48.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:48.717+0000 7f0781800640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f077c06f920 0x7f077c1579d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:48.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:48.717+0000 7f0782200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f077c0d0a60 0x7f077c15e800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55272/0 (socket says 172.21.15.16:55272) 2024-03-31T15:14:48.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:48.717+0000 7f0782200640 1 -- 172.21.15.16:0/253308010 learned_addr learned my addr 172.21.15.16:0/253308010 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:14:48.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:48.717+0000 7f0780e00640 1 --2- 172.21.15.16:0/253308010 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f077c0cef80 0x7f077c13fa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:48.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:48.717+0000 7f0782200640 1 -- 172.21.15.16:0/253308010 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f077c0cef80 msgr2=0x7f077c13fa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:48.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:48.717+0000 7f0782200640 1 --2- 172.21.15.16:0/253308010 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f077c0cef80 0x7f077c13fa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:48.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:48.717+0000 7f0782200640 1 -- 172.21.15.16:0/253308010 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f077c06f920 msgr2=0x7f077c1579d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:48.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:48.717+0000 7f0782200640 1 --2- 172.21.15.16:0/253308010 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f077c06f920 0x7f077c1579d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:48.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:48.717+0000 7f0782200640 1 -- 172.21.15.16:0/253308010 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f077c13d040 con 0x7f077c0d0a60 2024-03-31T15:14:48.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:48.717+0000 7f0782200640 1 --2- 172.21.15.16:0/253308010 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f077c0d0a60 0x7f077c15e800 secure :-1 s=READY pgs=1766 cs=0 l=1 rev1=1 crypto rx=0x7f077400cb40 tx=0x7f077400a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:48.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:48.717+0000 7f0779600640 1 -- 172.21.15.16:0/253308010 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0774017070 con 0x7f077c0d0a60 2024-03-31T15:14:48.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:48.717+0000 7f0779600640 1 -- 172.21.15.16:0/253308010 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f077400aca0 con 0x7f077c0d0a60 2024-03-31T15:14:48.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:48.717+0000 7f0784200640 1 -- 172.21.15.16:0/253308010 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f077c15daa0 con 0x7f077c0d0a60 2024-03-31T15:14:48.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:48.717+0000 7f0779600640 1 -- 172.21.15.16:0/253308010 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0774012620 con 0x7f077c0d0a60 2024-03-31T15:14:48.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:48.717+0000 7f0784200640 1 -- 172.21.15.16:0/253308010 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f077c15e300 con 0x7f077c0d0a60 2024-03-31T15:14:48.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:48.717+0000 7f0784200640 1 -- 172.21.15.16:0/253308010 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f077c069950 con 0x7f077c0d0a60 2024-03-31T15:14:48.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:48.717+0000 7f0779600640 1 -- 172.21.15.16:0/253308010 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f077401a070 con 0x7f077c0d0a60 2024-03-31T15:14:48.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:48.721+0000 7f0779600640 1 --2- 172.21.15.16:0/253308010 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f075c068380 0x7f075c06a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:48.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:48.721+0000 7f0779600640 1 -- 172.21.15.16:0/253308010 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(112..112 src has 1..112) ==== 5853+0+0 (secure 0 0 0) 0x7f07740867e0 con 0x7f077c0d0a60 2024-03-31T15:14:48.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:48.721+0000 7f0781800640 1 --2- 172.21.15.16:0/253308010 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f075c068380 0x7f075c06a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:48.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:48.721+0000 7f0781800640 1 --2- 172.21.15.16:0/253308010 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f075c068380 0x7f075c06a840 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f0770009ac0 tx=0x7f0770009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:48.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:48.725+0000 7f0779600640 1 -- 172.21.15.16:0/253308010 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f07740566b0 con 0x7f077c0d0a60 2024-03-31T15:14:48.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:48.889+0000 7f0784200640 1 -- 172.21.15.16:0/253308010 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f077c0b50e0 con 0x7f077c0d0a60 2024-03-31T15:14:48.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:48.889+0000 7f0779600640 1 -- 172.21.15.16:0/253308010 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v112) ==== 104+0+11 (secure 0 0 0) 0x7f077405a9b0 con 0x7f077c0d0a60 2024-03-31T15:14:48.884 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2024-03-31T15:14:48.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:48.893+0000 7f0784200640 1 -- 172.21.15.16:0/253308010 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f075c068380 msgr2=0x7f075c06a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:48.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:48.893+0000 7f0784200640 1 --2- 172.21.15.16:0/253308010 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f075c068380 0x7f075c06a840 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f0770009ac0 tx=0x7f0770009190 comp rx=0 tx=0).stop 2024-03-31T15:14:48.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:48.893+0000 7f0784200640 1 -- 172.21.15.16:0/253308010 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f077c0d0a60 msgr2=0x7f077c15e800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:48.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:48.893+0000 7f0784200640 1 --2- 172.21.15.16:0/253308010 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f077c0d0a60 0x7f077c15e800 secure :-1 s=READY pgs=1766 cs=0 l=1 rev1=1 crypto rx=0x7f077400cb40 tx=0x7f077400a060 comp rx=0 tx=0).stop 2024-03-31T15:14:48.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:48.893+0000 7f0784200640 1 -- 172.21.15.16:0/253308010 shutdown_connections 2024-03-31T15:14:48.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:48.893+0000 7f0784200640 1 --2- 172.21.15.16:0/253308010 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f075c068380 0x7f075c06a840 unknown :-1 s=CLOSED pgs=747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:48.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:48.893+0000 7f0784200640 1 --2- 172.21.15.16:0/253308010 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f077c0d0a60 0x7f077c15e800 unknown :-1 s=CLOSED pgs=1766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:48.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:48.893+0000 7f0784200640 1 --2- 172.21.15.16:0/253308010 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f077c0cef80 0x7f077c13fa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:48.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:48.893+0000 7f0784200640 1 --2- 172.21.15.16:0/253308010 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f077c06f920 0x7f077c1579d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:48.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:48.893+0000 7f0784200640 1 -- 172.21.15.16:0/253308010 >> 172.21.15.16:0/253308010 conn(0x7f077c067930 msgr2=0x7f077c13e360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:14:48.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:48.893+0000 7f0784200640 1 -- 172.21.15.16:0/253308010 shutdown_connections 2024-03-31T15:14:48.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:48.893+0000 7f0784200640 1 -- 172.21.15.16:0/253308010 wait complete. 2024-03-31T15:14:48.907 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-31T15:14:49.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:49.057+0000 7f550fa00640 1 Processor -- start 2024-03-31T15:14:49.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:49.057+0000 7f550fa00640 1 -- start start 2024-03-31T15:14:49.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:49.057+0000 7f550fa00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f55080d0a60 0x7f55080cea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:49.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:49.057+0000 7f550fa00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f55080cef80 0x7f55081419c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:49.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:49.057+0000 7f550fa00640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f55080cf430 con 0x7f550806f920 2024-03-31T15:14:49.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:49.057+0000 7f550fa00640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f550813f670 con 0x7f55080cef80 2024-03-31T15:14:49.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:49.057+0000 7f550fa00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f550815ca70 con 0x7f55080d0a60 2024-03-31T15:14:49.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:49.057+0000 7f550d000640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f550806f920 0x7f550806fd00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:55442/0 (socket says 172.21.15.16:55442) 2024-03-31T15:14:49.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:49.057+0000 7f550d000640 1 -- 172.21.15.16:0/2926761911 learned_addr learned my addr 172.21.15.16:0/2926761911 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:14:49.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:49.057+0000 7f550da00640 1 --2- 172.21.15.16:0/2926761911 >> v2:172.21.15.89:3300/0 conn(0x7f55080cef80 0x7f55081419c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:49.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:49.057+0000 7f5507e00640 1 --2- 172.21.15.16:0/2926761911 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f55080d0a60 0x7f55080cea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:49.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:49.057+0000 7f5507400640 1 -- 172.21.15.16:0/2926761911 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3161041708 0 0) 0x7f55080cf430 con 0x7f550806f920 2024-03-31T15:14:49.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:49.057+0000 7f5507400640 1 -- 172.21.15.16:0/2926761911 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f54f0003530 con 0x7f550806f920 2024-03-31T15:14:49.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:49.057+0000 7f5507400640 1 -- 172.21.15.16:0/2926761911 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1128467860 0 0) 0x7f54f0003530 con 0x7f550806f920 2024-03-31T15:14:49.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:49.057+0000 7f550da00640 1 -- 172.21.15.16:0/2926761911 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f55080d0a60 msgr2=0x7f55080cea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:14:49.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:49.057+0000 7f550da00640 1 --2- 172.21.15.16:0/2926761911 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f55080d0a60 0x7f55080cea40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:49.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:49.057+0000 7f550da00640 1 -- 172.21.15.16:0/2926761911 >> v1:172.21.15.16:6789/0 conn(0x7f550806f920 legacy=0x7f550806fd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:49.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:49.057+0000 7f550da00640 1 -- 172.21.15.16:0/2926761911 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f550815cd50 con 0x7f55080cef80 2024-03-31T15:14:49.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:49.061+0000 7f550da00640 1 --2- 172.21.15.16:0/2926761911 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f55080cef80 0x7f55081419c0 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f5500009d90 tx=0x7f550001c890 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a9e9ac3e30b23f66 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:49.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:49.061+0000 7f5507400640 1 -- 172.21.15.16:0/2926761911 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55000ac070 con 0x7f55080cef80 2024-03-31T15:14:49.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:49.061+0000 7f5507400640 1 -- 172.21.15.16:0/2926761911 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f55000a64a0 con 0x7f55080cef80 2024-03-31T15:14:49.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:49.061+0000 7f5507400640 1 -- 172.21.15.16:0/2926761911 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54fc001680 con 0x7f55080cef80 2024-03-31T15:14:49.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:49.061+0000 7f550fa00640 1 -- 172.21.15.16:0/2926761911 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f55080cef80 msgr2=0x7f55081419c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:49.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:49.061+0000 7f550fa00640 1 --2- 172.21.15.16:0/2926761911 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f55080cef80 0x7f55081419c0 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f5500009d90 tx=0x7f550001c890 comp rx=0 tx=0).stop 2024-03-31T15:14:49.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:49.061+0000 7f550fa00640 1 -- 172.21.15.16:0/2926761911 shutdown_connections 2024-03-31T15:14:49.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:49.061+0000 7f550fa00640 1 --2- 172.21.15.16:0/2926761911 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f55080cef80 0x7f55081419c0 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:49.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:49.061+0000 7f550fa00640 1 --2- 172.21.15.16:0/2926761911 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f55080d0a60 0x7f55080cea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:49.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:49.061+0000 7f550fa00640 1 -- 172.21.15.16:0/2926761911 >> 172.21.15.16:0/2926761911 conn(0x7f5508067930 msgr2=0x7f55080b3ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:14:49.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:49.061+0000 7f550fa00640 1 -- 172.21.15.16:0/2926761911 shutdown_connections 2024-03-31T15:14:49.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:49.061+0000 7f550fa00640 1 -- 172.21.15.16:0/2926761911 wait complete. 2024-03-31T15:14:49.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:49.065+0000 7f550fa00640 1 Processor -- start 2024-03-31T15:14:49.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:49.065+0000 7f550fa00640 1 -- start start 2024-03-31T15:14:49.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:49.065+0000 7f550fa00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f550806f920 0x7f55080b3e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:49.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:49.065+0000 7f550fa00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f55080cef80 0x7f550806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:49.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:49.065+0000 7f550fa00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f55080d0a60 0x7f550813e1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:49.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:49.065+0000 7f550fa00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f5508169230 con 0x7f550806f920 2024-03-31T15:14:49.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:49.065+0000 7f550fa00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f55081690b0 con 0x7f55080d0a60 2024-03-31T15:14:49.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:49.065+0000 7f550fa00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f55081693b0 con 0x7f55080cef80 2024-03-31T15:14:49.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:49.065+0000 7f550d000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f550806f920 0x7f55080b3e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:49.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:49.065+0000 7f550d000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f550806f920 0x7f55080b3e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55282/0 (socket says 172.21.15.16:55282) 2024-03-31T15:14:49.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:49.065+0000 7f550d000640 1 -- 172.21.15.16:0/3708457034 learned_addr learned my addr 172.21.15.16:0/3708457034 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:14:49.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:49.065+0000 7f5507e00640 1 --2- 172.21.15.16:0/3708457034 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f55080cef80 0x7f550806d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:49.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:49.065+0000 7f550da00640 1 --2- 172.21.15.16:0/3708457034 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f55080d0a60 0x7f550813e1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:49.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:49.065+0000 7f550d000640 1 -- 172.21.15.16:0/3708457034 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f55080cef80 msgr2=0x7f550806d390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:49.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:49.065+0000 7f550d000640 1 --2- 172.21.15.16:0/3708457034 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f55080cef80 0x7f550806d390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:49.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:49.065+0000 7f550d000640 1 -- 172.21.15.16:0/3708457034 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f55080d0a60 msgr2=0x7f550813e1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:49.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:49.065+0000 7f550d000640 1 --2- 172.21.15.16:0/3708457034 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f55080d0a60 0x7f550813e1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:49.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:49.065+0000 7f550d000640 1 -- 172.21.15.16:0/3708457034 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55080b43c0 con 0x7f550806f920 2024-03-31T15:14:49.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:49.065+0000 7f550d000640 1 --2- 172.21.15.16:0/3708457034 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f550806f920 0x7f55080b3e80 secure :-1 s=READY pgs=1768 cs=0 l=1 rev1=1 crypto rx=0x7f54fc00c760 tx=0x7f54fc00cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:49.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:49.065+0000 7f5504c00640 1 -- 172.21.15.16:0/3708457034 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54fc017070 con 0x7f550806f920 2024-03-31T15:14:49.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:49.065+0000 7f5504c00640 1 -- 172.21.15.16:0/3708457034 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f54fc005ae0 con 0x7f550806f920 2024-03-31T15:14:49.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:49.065+0000 7f5504c00640 1 -- 172.21.15.16:0/3708457034 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54fc007070 con 0x7f550806f920 2024-03-31T15:14:49.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:49.065+0000 7f550fa00640 1 -- 172.21.15.16:0/3708457034 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5508158120 con 0x7f550806f920 2024-03-31T15:14:49.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:49.065+0000 7f550fa00640 1 -- 172.21.15.16:0/3708457034 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f550806cb90 con 0x7f550806f920 2024-03-31T15:14:49.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:49.065+0000 7f550fa00640 1 -- 172.21.15.16:0/3708457034 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5508069950 con 0x7f550806f920 2024-03-31T15:14:49.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:49.069+0000 7f5504c00640 1 -- 172.21.15.16:0/3708457034 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f54fc01a070 con 0x7f550806f920 2024-03-31T15:14:49.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:49.069+0000 7f5504c00640 1 --2- 172.21.15.16:0/3708457034 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f54e8068380 0x7f54e806a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:49.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:49.069+0000 7f5504c00640 1 -- 172.21.15.16:0/3708457034 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(112..112 src has 1..112) ==== 5853+0+0 (secure 0 0 0) 0x7f54fc085fe0 con 0x7f550806f920 2024-03-31T15:14:49.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:49.069+0000 7f5507e00640 1 --2- 172.21.15.16:0/3708457034 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f54e8068380 0x7f54e806a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:49.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:49.069+0000 7f5507e00640 1 --2- 172.21.15.16:0/3708457034 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f54e8068380 0x7f54e806a840 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f5508156c10 tx=0x7f54f4008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:49.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:49.073+0000 7f5504c00640 1 -- 172.21.15.16:0/3708457034 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f54fc056490 con 0x7f550806f920 2024-03-31T15:14:49.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:49.237+0000 7f550fa00640 1 -- 172.21.15.16:0/3708457034 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f55080b50e0 con 0x7f550806f920 2024-03-31T15:14:49.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:49.237+0000 7f5504c00640 1 -- 172.21.15.16:0/3708457034 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v112) ==== 100+0+11 (secure 0 0 0) 0x7f54fc055d90 con 0x7f550806f920 2024-03-31T15:14:49.230 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 64 2024-03-31T15:14:49.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:49.237+0000 7f550fa00640 1 -- 172.21.15.16:0/3708457034 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f54e8068380 msgr2=0x7f54e806a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:49.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:49.237+0000 7f550fa00640 1 --2- 172.21.15.16:0/3708457034 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f54e8068380 0x7f54e806a840 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f5508156c10 tx=0x7f54f4008040 comp rx=0 tx=0).stop 2024-03-31T15:14:49.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:49.237+0000 7f550fa00640 1 -- 172.21.15.16:0/3708457034 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f550806f920 msgr2=0x7f55080b3e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:49.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:49.237+0000 7f550fa00640 1 --2- 172.21.15.16:0/3708457034 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f550806f920 0x7f55080b3e80 secure :-1 s=READY pgs=1768 cs=0 l=1 rev1=1 crypto rx=0x7f54fc00c760 tx=0x7f54fc00cc20 comp rx=0 tx=0).stop 2024-03-31T15:14:49.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:49.237+0000 7f550fa00640 1 -- 172.21.15.16:0/3708457034 shutdown_connections 2024-03-31T15:14:49.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:49.237+0000 7f550fa00640 1 --2- 172.21.15.16:0/3708457034 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f54e8068380 0x7f54e806a840 unknown :-1 s=CLOSED pgs=748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:49.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:49.237+0000 7f550fa00640 1 --2- 172.21.15.16:0/3708457034 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f55080d0a60 0x7f550813e1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:49.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:49.237+0000 7f550fa00640 1 --2- 172.21.15.16:0/3708457034 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f55080cef80 0x7f550806d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:49.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:49.237+0000 7f550fa00640 1 --2- 172.21.15.16:0/3708457034 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f550806f920 0x7f55080b3e80 unknown :-1 s=CLOSED pgs=1768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:49.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:49.237+0000 7f550fa00640 1 -- 172.21.15.16:0/3708457034 >> 172.21.15.16:0/3708457034 conn(0x7f5508067930 msgr2=0x7f550815acc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:14:49.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:49.237+0000 7f550fa00640 1 -- 172.21.15.16:0/3708457034 shutdown_connections 2024-03-31T15:14:49.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:49.237+0000 7f550fa00640 1 -- 172.21.15.16:0/3708457034 wait complete. 2024-03-31T15:14:49.252 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-31T15:14:49.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:49.401+0000 7f099ea00640 1 Processor -- start 2024-03-31T15:14:49.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:49.401+0000 7f099ea00640 1 -- start start 2024-03-31T15:14:49.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:49.405+0000 7f099ea00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f0998145e40 0x7f09980cc740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:49.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:49.405+0000 7f099ea00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f09980ccc80 0x7f099806f920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:49.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:49.405+0000 7f099ea00640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f09980d0920 con 0x7f099806fe60 2024-03-31T15:14:49.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:49.405+0000 7f099ea00640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f09980d06d0 con 0x7f09980ccc80 2024-03-31T15:14:49.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:49.405+0000 7f099ea00640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f09980d0c00 con 0x7f0998145e40 2024-03-31T15:14:49.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:49.405+0000 7f098fe00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f09980ccc80 0x7f099806f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:49.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:49.405+0000 7f098fe00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f09980ccc80 0x7f099806f920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55290/0 (socket says 172.21.15.16:55290) 2024-03-31T15:14:49.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:49.405+0000 7f098fe00640 1 -- 172.21.15.16:0/1895705493 learned_addr learned my addr 172.21.15.16:0/1895705493 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:14:49.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:49.405+0000 7f0997e00640 1 --2- 172.21.15.16:0/1895705493 >> v2:172.21.15.89:3300/0 conn(0x7f0998145e40 0x7f09980cc740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:49.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:49.405+0000 7f098fe00640 1 -- 172.21.15.16:0/1895705493 >> v2:172.21.15.89:3300/0 conn(0x7f0998145e40 msgr2=0x7f09980cc740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:14:49.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:49.405+0000 7f098fe00640 1 --2- 172.21.15.16:0/1895705493 >> v2:172.21.15.89:3300/0 conn(0x7f0998145e40 0x7f09980cc740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:49.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:49.405+0000 7f098fe00640 1 -- 172.21.15.16:0/1895705493 >> v1:172.21.15.89:6789/0 conn(0x7f099806fe60 legacy=0x7f099815fdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:49.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:49.405+0000 7f098fe00640 1 -- 172.21.15.16:0/1895705493 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09980d1810 con 0x7f09980ccc80 2024-03-31T15:14:49.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:49.409+0000 7f098fe00640 1 --2- 172.21.15.16:0/1895705493 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f09980ccc80 0x7f099806f920 secure :-1 s=READY pgs=1769 cs=0 l=1 rev1=1 crypto rx=0x7f0998072e60 tx=0x7f098001c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=979ae63796975998 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:49.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:49.409+0000 7f0997400640 1 -- 172.21.15.16:0/1895705493 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09800b0070 con 0x7f09980ccc80 2024-03-31T15:14:49.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:49.409+0000 7f0997400640 1 -- 172.21.15.16:0/1895705493 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f09800ab600 con 0x7f09980ccc80 2024-03-31T15:14:49.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:49.409+0000 7f0997400640 1 -- 172.21.15.16:0/1895705493 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09800ab9d0 con 0x7f09980ccc80 2024-03-31T15:14:49.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:49.409+0000 7f099ea00640 1 -- 172.21.15.16:0/1895705493 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f09980ccc80 msgr2=0x7f099806f920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:49.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:49.409+0000 7f099ea00640 1 --2- 172.21.15.16:0/1895705493 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f09980ccc80 0x7f099806f920 secure :-1 s=READY pgs=1769 cs=0 l=1 rev1=1 crypto rx=0x7f0998072e60 tx=0x7f098001c520 comp rx=0 tx=0).stop 2024-03-31T15:14:49.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:49.409+0000 7f099ea00640 1 -- 172.21.15.16:0/1895705493 shutdown_connections 2024-03-31T15:14:49.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:49.409+0000 7f099ea00640 1 --2- 172.21.15.16:0/1895705493 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f09980ccc80 0x7f099806f920 unknown :-1 s=CLOSED pgs=1769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:49.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:49.409+0000 7f099ea00640 1 --2- 172.21.15.16:0/1895705493 >> v2:172.21.15.89:3300/0 conn(0x7f0998145e40 0x7f09980cc740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:49.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:49.409+0000 7f099ea00640 1 -- 172.21.15.16:0/1895705493 >> 172.21.15.16:0/1895705493 conn(0x7f0998067330 msgr2=0x7f0998067740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:14:49.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:49.409+0000 7f099ea00640 1 -- 172.21.15.16:0/1895705493 shutdown_connections 2024-03-31T15:14:49.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:49.409+0000 7f099ea00640 1 -- 172.21.15.16:0/1895705493 wait complete. 2024-03-31T15:14:49.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:49.409+0000 7f099ea00640 1 Processor -- start 2024-03-31T15:14:49.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:49.409+0000 7f099ea00640 1 -- start start 2024-03-31T15:14:49.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:49.409+0000 7f099ea00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f099806fe60 0x7f0998143b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:49.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:49.409+0000 7f099ea00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f09980ccc80 0x7f0998142800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:49.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:49.413+0000 7f099ea00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f0998145e40 0x7f0998141220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:49.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:49.413+0000 7f099ea00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f09980b5240 con 0x7f09980ccc80 2024-03-31T15:14:49.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:49.413+0000 7f099ea00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f09980b50c0 con 0x7f099806fe60 2024-03-31T15:14:49.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:49.413+0000 7f099ea00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f0998169190 con 0x7f0998145e40 2024-03-31T15:14:49.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:49.413+0000 7f098fe00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f09980ccc80 0x7f0998142800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:49.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:49.413+0000 7f098fe00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f09980ccc80 0x7f0998142800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55302/0 (socket says 172.21.15.16:55302) 2024-03-31T15:14:49.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:49.413+0000 7f098fe00640 1 -- 172.21.15.16:0/2677235222 learned_addr learned my addr 172.21.15.16:0/2677235222 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:14:49.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:49.413+0000 7f0997e00640 1 --2- 172.21.15.16:0/2677235222 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f099806fe60 0x7f0998143b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:49.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:49.413+0000 7f099ca00640 1 --2- 172.21.15.16:0/2677235222 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f0998145e40 0x7f0998141220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:49.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:49.413+0000 7f098fe00640 1 -- 172.21.15.16:0/2677235222 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f0998145e40 msgr2=0x7f0998141220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:49.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:49.413+0000 7f098fe00640 1 --2- 172.21.15.16:0/2677235222 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f0998145e40 0x7f0998141220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:49.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:49.413+0000 7f098fe00640 1 -- 172.21.15.16:0/2677235222 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f099806fe60 msgr2=0x7f0998143b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:49.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:49.413+0000 7f098fe00640 1 --2- 172.21.15.16:0/2677235222 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f099806fe60 0x7f0998143b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:49.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:49.413+0000 7f098fe00640 1 -- 172.21.15.16:0/2677235222 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0998142d40 con 0x7f09980ccc80 2024-03-31T15:14:49.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:49.413+0000 7f099ca00640 1 --2- 172.21.15.16:0/2677235222 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f0998145e40 0x7f0998141220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:14:49.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:49.413+0000 7f098fe00640 1 --2- 172.21.15.16:0/2677235222 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f09980ccc80 0x7f0998142800 secure :-1 s=READY pgs=1770 cs=0 l=1 rev1=1 crypto rx=0x7f0980019130 tx=0x7f09800192c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:49.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:49.413+0000 7f0994c00640 1 -- 172.21.15.16:0/2677235222 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09800b9070 con 0x7f09980ccc80 2024-03-31T15:14:49.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:49.413+0000 7f0994c00640 1 -- 172.21.15.16:0/2677235222 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f09800b4400 con 0x7f09980ccc80 2024-03-31T15:14:49.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:49.413+0000 7f099ea00640 1 -- 172.21.15.16:0/2677235222 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0998144210 con 0x7f09980ccc80 2024-03-31T15:14:49.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:49.413+0000 7f0994c00640 1 -- 172.21.15.16:0/2677235222 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09800b0040 con 0x7f09980ccc80 2024-03-31T15:14:49.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:49.413+0000 7f099ea00640 1 -- 172.21.15.16:0/2677235222 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f09981447d0 con 0x7f09980ccc80 2024-03-31T15:14:49.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:49.413+0000 7f099ea00640 1 -- 172.21.15.16:0/2677235222 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0998069950 con 0x7f09980ccc80 2024-03-31T15:14:49.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:49.413+0000 7f0994c00640 1 -- 172.21.15.16:0/2677235222 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f09800bc070 con 0x7f09980ccc80 2024-03-31T15:14:49.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:49.417+0000 7f0994c00640 1 --2- 172.21.15.16:0/2677235222 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f0974068380 0x7f097406a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:49.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:49.417+0000 7f0994c00640 1 -- 172.21.15.16:0/2677235222 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(112..112 src has 1..112) ==== 5853+0+0 (secure 0 0 0) 0x7f098011fac0 con 0x7f09980ccc80 2024-03-31T15:14:49.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:49.417+0000 7f0997e00640 1 --2- 172.21.15.16:0/2677235222 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f0974068380 0x7f097406a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:49.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:49.417+0000 7f0997e00640 1 --2- 172.21.15.16:0/2677235222 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f0974068380 0x7f097406a840 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f099806fdd0 tx=0x7f0988008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:49.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:49.421+0000 7f0994c00640 1 -- 172.21.15.16:0/2677235222 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f0980016110 con 0x7f09980ccc80 2024-03-31T15:14:49.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:49.597+0000 7f099ea00640 1 -- 172.21.15.16:0/2677235222 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f09980d2400 con 0x7f09980ccc80 2024-03-31T15:14:49.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:49.597+0000 7f0994c00640 1 -- 172.21.15.16:0/2677235222 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 2 entries v112) ==== 80+0+122 (secure 0 0 0) 0x7f09800eff50 con 0x7f09980ccc80 2024-03-31T15:14:49.593 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:6835/2564952016 2024-04-01T15:14:39.715615+0000 2024-03-31T15:14:49.593 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:6833/2564952016 2024-04-01T15:14:39.715615+0000 2024-03-31T15:14:49.593 INFO:teuthology.orchestra.run.smithi016.stderr:listed 2 entries 2024-03-31T15:14:49.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:49.601+0000 7f099ea00640 1 -- 172.21.15.16:0/2677235222 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f0974068380 msgr2=0x7f097406a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:49.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:49.601+0000 7f099ea00640 1 --2- 172.21.15.16:0/2677235222 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f0974068380 0x7f097406a840 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f099806fdd0 tx=0x7f0988008040 comp rx=0 tx=0).stop 2024-03-31T15:14:49.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:49.601+0000 7f099ea00640 1 -- 172.21.15.16:0/2677235222 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f09980ccc80 msgr2=0x7f0998142800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:49.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:49.601+0000 7f099ea00640 1 --2- 172.21.15.16:0/2677235222 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f09980ccc80 0x7f0998142800 secure :-1 s=READY pgs=1770 cs=0 l=1 rev1=1 crypto rx=0x7f0980019130 tx=0x7f09800192c0 comp rx=0 tx=0).stop 2024-03-31T15:14:49.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:49.601+0000 7f099ea00640 1 -- 172.21.15.16:0/2677235222 shutdown_connections 2024-03-31T15:14:49.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:49.601+0000 7f099ea00640 1 --2- 172.21.15.16:0/2677235222 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f0974068380 0x7f097406a840 unknown :-1 s=CLOSED pgs=749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:49.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:49.601+0000 7f099ea00640 1 --2- 172.21.15.16:0/2677235222 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f0998145e40 0x7f0998141220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:49.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:49.601+0000 7f099ea00640 1 --2- 172.21.15.16:0/2677235222 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f09980ccc80 0x7f0998142800 unknown :-1 s=CLOSED pgs=1770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:49.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:49.601+0000 7f099ea00640 1 --2- 172.21.15.16:0/2677235222 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f099806fe60 0x7f0998143b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:49.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:49.601+0000 7f099ea00640 1 -- 172.21.15.16:0/2677235222 >> 172.21.15.16:0/2677235222 conn(0x7f0998067330 msgr2=0x7f09980b2b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:14:49.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:49.601+0000 7f099ea00640 1 -- 172.21.15.16:0/2677235222 shutdown_connections 2024-03-31T15:14:49.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:49.601+0000 7f099ea00640 1 -- 172.21.15.16:0/2677235222 wait complete. 2024-03-31T15:14:49.615 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-03-31T15:14:49.616 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-31T15:14:49.626 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-31T15:14:49.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:49.825+0000 7f5bf2400640 1 Processor -- start 2024-03-31T15:14:49.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:49.829+0000 7f5bf2400640 1 -- start start 2024-03-31T15:14:49.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:49.829+0000 7f5bf2400640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f5bec15b3d0 0x7f5bec0b1cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:49.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:49.829+0000 7f5bf2400640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f5bec0ce2f0 0x7f5bec0b2200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:49.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:49.829+0000 7f5bf2400640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5bec15f3c0 con 0x7f5bec15eee0 2024-03-31T15:14:49.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:49.829+0000 7f5bf2400640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f5bec15b7b0 con 0x7f5bec15b3d0 2024-03-31T15:14:49.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:49.829+0000 7f5bf2400640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f5bec15f6a0 con 0x7f5bec0ce2f0 2024-03-31T15:14:49.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:49.829+0000 7f5beb400640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f5bec15b3d0 0x7f5bec0b1cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:49.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:49.829+0000 7f5beb400640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f5bec15b3d0 0x7f5bec0b1cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55306/0 (socket says 172.21.15.16:55306) 2024-03-31T15:14:49.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:49.829+0000 7f5beb400640 1 -- 172.21.15.16:0/2467707566 learned_addr learned my addr 172.21.15.16:0/2467707566 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:14:49.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:49.829+0000 7f5bebe00640 1 --2- 172.21.15.16:0/2467707566 >> v2:172.21.15.89:3300/0 conn(0x7f5bec0ce2f0 0x7f5bec0b2200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:49.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:49.829+0000 7f5beb400640 1 -- 172.21.15.16:0/2467707566 >> v2:172.21.15.89:3300/0 conn(0x7f5bec0ce2f0 msgr2=0x7f5bec0b2200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:14:49.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:49.829+0000 7f5beb400640 1 --2- 172.21.15.16:0/2467707566 >> v2:172.21.15.89:3300/0 conn(0x7f5bec0ce2f0 0x7f5bec0b2200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:49.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:49.829+0000 7f5beb400640 1 -- 172.21.15.16:0/2467707566 >> v1:172.21.15.89:6789/0 conn(0x7f5bec15eee0 legacy=0x7f5bec0cc9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:49.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:49.829+0000 7f5beb400640 1 -- 172.21.15.16:0/2467707566 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5bec15f980 con 0x7f5bec15b3d0 2024-03-31T15:14:49.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:49.833+0000 7f5beb400640 1 --2- 172.21.15.16:0/2467707566 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5bec15b3d0 0x7f5bec0b1cc0 secure :-1 s=READY pgs=1771 cs=0 l=1 rev1=1 crypto rx=0x7f5bec06f8a0 tx=0x7f5bdc01ebc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=44c7e0ee29e21bf2 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:49.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:49.833+0000 7f5beaa00640 1 -- 172.21.15.16:0/2467707566 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bdc0b4070 con 0x7f5bec15b3d0 2024-03-31T15:14:49.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:49.833+0000 7f5beaa00640 1 -- 172.21.15.16:0/2467707566 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5bdc0a7c50 con 0x7f5bec15b3d0 2024-03-31T15:14:49.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:49.833+0000 7f5beaa00640 1 -- 172.21.15.16:0/2467707566 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bdc0af440 con 0x7f5bec15b3d0 2024-03-31T15:14:49.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:49.833+0000 7f5bf2400640 1 -- 172.21.15.16:0/2467707566 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5bec15b3d0 msgr2=0x7f5bec0b1cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:49.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:49.833+0000 7f5bf2400640 1 --2- 172.21.15.16:0/2467707566 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5bec15b3d0 0x7f5bec0b1cc0 secure :-1 s=READY pgs=1771 cs=0 l=1 rev1=1 crypto rx=0x7f5bec06f8a0 tx=0x7f5bdc01ebc0 comp rx=0 tx=0).stop 2024-03-31T15:14:49.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:49.833+0000 7f5bf2400640 1 -- 172.21.15.16:0/2467707566 shutdown_connections 2024-03-31T15:14:49.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:49.833+0000 7f5bf2400640 1 --2- 172.21.15.16:0/2467707566 >> v2:172.21.15.89:3300/0 conn(0x7f5bec0ce2f0 0x7f5bec0b2200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:49.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:49.833+0000 7f5bf2400640 1 --2- 172.21.15.16:0/2467707566 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5bec15b3d0 0x7f5bec0b1cc0 unknown :-1 s=CLOSED pgs=1771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:49.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:49.833+0000 7f5bf2400640 1 -- 172.21.15.16:0/2467707566 >> 172.21.15.16:0/2467707566 conn(0x7f5bec06b620 msgr2=0x7f5bec06ba30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:14:49.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:49.833+0000 7f5bf2400640 1 -- 172.21.15.16:0/2467707566 shutdown_connections 2024-03-31T15:14:49.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:49.833+0000 7f5bf2400640 1 -- 172.21.15.16:0/2467707566 wait complete. 2024-03-31T15:14:49.829 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:49.837+0000 7f5bf2400640 1 Processor -- start 2024-03-31T15:14:49.829 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:49.837+0000 7f5bf2400640 1 -- start start 2024-03-31T15:14:49.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:49.837+0000 7f5bf2400640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5bec0ce2f0 0x7f5bec1367f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:49.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:49.837+0000 7f5bf2400640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f5bec15b3d0 0x7f5bec15c810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:49.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:49.837+0000 7f5bf2400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5bec15eee0 0x7f5bec15cd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:49.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:49.837+0000 7f5bf2400640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f5bec0d0350 con 0x7f5bec15eee0 2024-03-31T15:14:49.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:49.837+0000 7f5bf2400640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f5bec0d01d0 con 0x7f5bec0ce2f0 2024-03-31T15:14:49.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:49.837+0000 7f5bf2400640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f5bec0d04d0 con 0x7f5bec15b3d0 2024-03-31T15:14:49.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:49.837+0000 7f5bebe00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5bec15eee0 0x7f5bec15cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:49.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:49.837+0000 7f5bebe00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5bec15eee0 0x7f5bec15cd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55322/0 (socket says 172.21.15.16:55322) 2024-03-31T15:14:49.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:49.837+0000 7f5bebe00640 1 -- 172.21.15.16:0/1958669833 learned_addr learned my addr 172.21.15.16:0/1958669833 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:14:49.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:49.837+0000 7f5be3e00640 1 --2- 172.21.15.16:0/1958669833 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f5bec15b3d0 0x7f5bec15c810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:49.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:49.837+0000 7f5beb400640 1 --2- 172.21.15.16:0/1958669833 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5bec0ce2f0 0x7f5bec1367f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:49.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:49.837+0000 7f5bebe00640 1 -- 172.21.15.16:0/1958669833 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f5bec15b3d0 msgr2=0x7f5bec15c810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:49.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:49.837+0000 7f5bebe00640 1 --2- 172.21.15.16:0/1958669833 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f5bec15b3d0 0x7f5bec15c810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:49.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:49.837+0000 7f5bebe00640 1 -- 172.21.15.16:0/1958669833 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5bec0ce2f0 msgr2=0x7f5bec1367f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:49.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:49.837+0000 7f5bebe00640 1 --2- 172.21.15.16:0/1958669833 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5bec0ce2f0 0x7f5bec1367f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:49.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:49.837+0000 7f5bebe00640 1 -- 172.21.15.16:0/1958669833 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5bec0b0330 con 0x7f5bec15eee0 2024-03-31T15:14:49.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:49.837+0000 7f5bebe00640 1 --2- 172.21.15.16:0/1958669833 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5bec15eee0 0x7f5bec15cd50 secure :-1 s=READY pgs=1772 cs=0 l=1 rev1=1 crypto rx=0x7f5be4007e70 tx=0x7f5be400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:49.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:49.837+0000 7f5be3400640 1 -- 172.21.15.16:0/1958669833 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5be4014620 con 0x7f5bec15eee0 2024-03-31T15:14:49.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:49.837+0000 7f5be3400640 1 -- 172.21.15.16:0/1958669833 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5be4019070 con 0x7f5bec15eee0 2024-03-31T15:14:49.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:49.837+0000 7f5be3400640 1 -- 172.21.15.16:0/1958669833 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5be4014a20 con 0x7f5bec15eee0 2024-03-31T15:14:49.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:49.837+0000 7f5bf2400640 1 -- 172.21.15.16:0/1958669833 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5bec15b930 con 0x7f5bec15eee0 2024-03-31T15:14:49.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:49.837+0000 7f5bf2400640 1 -- 172.21.15.16:0/1958669833 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5bec0b05d0 con 0x7f5bec15eee0 2024-03-31T15:14:49.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:49.837+0000 7f5bf2400640 1 -- 172.21.15.16:0/1958669833 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5bec069950 con 0x7f5bec15eee0 2024-03-31T15:14:49.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:49.841+0000 7f5be3400640 1 -- 172.21.15.16:0/1958669833 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f5be4014bc0 con 0x7f5bec15eee0 2024-03-31T15:14:49.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:49.841+0000 7f5be3400640 1 --2- 172.21.15.16:0/1958669833 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f5bc8068380 0x7f5bc806a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:49.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:49.841+0000 7f5be3400640 1 -- 172.21.15.16:0/1958669833 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(112..112 src has 1..112) ==== 5853+0+0 (secure 0 0 0) 0x7f5be40872b0 con 0x7f5bec15eee0 2024-03-31T15:14:49.835 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:49.841+0000 7f5beb400640 1 --2- 172.21.15.16:0/1958669833 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f5bc8068380 0x7f5bc806a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:49.836 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:49.841+0000 7f5beb400640 1 --2- 172.21.15.16:0/1958669833 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f5bc8068380 0x7f5bc806a840 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f5bdc0b1000 tx=0x7f5bdc0a9070 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:49.839 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:49.845+0000 7f5be3400640 1 -- 172.21.15.16:0/1958669833 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f5be40570f0 con 0x7f5bec15eee0 2024-03-31T15:14:50.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:50.013+0000 7f5bf2400640 1 -- 172.21.15.16:0/1958669833 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f5bec0cdea0 con 0x7f5bec15eee0 2024-03-31T15:14:50.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:50.013+0000 7f5be3400640 1 -- 172.21.15.16:0/1958669833 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v112) ==== 74+0+16777 (secure 0 0 0) 0x7f5be405b3f0 con 0x7f5bec15eee0 2024-03-31T15:14:50.008 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:14:50.008 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":112,"fsid":"286eb5da-fb47-4550-b290-d9519c13b9a5","created":"2024-03-31T15:03:11.438952+0000","modified":"2024-03-31T15:14:39.715636+0000","last_up_change":"2024-03-31T15:03:22.885535+0000","last_in_change":"2024-03-31T15:03:13.941667+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-31T15:03:22.550787+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-03-31T15:14:06.417304+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"108","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-03-31T15:14:07.622562+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"109","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"92a66450-ef19-40d8-ac3d-d760b23b0bda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6801","nonce":932712244}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6803","nonce":932712244}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6807","nonce":932712244}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6805","nonce":932712244}]},"public_addr":"172.21.15.16:6801/932712244","cluster_addr":"172.21.15.16:6803/932712244","heartbeat_back_addr":"172.21.15.16:6807/932712244","heartbeat_front_addr":"172.21.15.16:6805/932712244","state":["exists","up"]},{"osd":1,"uuid":"da9d1f2b-1968-4959-b843-faa231988e92","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6810","nonce":1398737533}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6811","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6813","nonce":1398737533}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6821","nonce":1398737533}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6815","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6816","nonce":1398737533}]},"public_addr":"172.21.15.16:6810/1398737533","cluster_addr":"172.21.15.16:6813/1398737533","heartbeat_back_addr":"172.21.15.16:6821/1398737533","heartbeat_front_addr":"172.21.15.16:6816/1398737533","state":["exists","up"]},{"osd":2,"uuid":"01a73662-9a21-4b1f-bcbb-42b9636a4638","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6819","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6822","nonce":3018526646}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6825","nonce":3018526646}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6831","nonce":3018526646}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6827","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6829","nonce":3018526646}]},"public_addr":"172.21.15.16:6822/3018526646","cluster_addr":"172.21.15.16:6825/3018526646","heartbeat_back_addr":"172.21.15.16:6831/3018526646","heartbeat_front_addr":"172.21.15.16:6829/3018526646","state":["exists","up"]},{"osd":3,"uuid":"c7771fd3-e18e-4500-ac38-23cf10ebcd75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6809","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6812","nonce":3360352974}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6817","nonce":3360352974}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6828","nonce":3360352974}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6823","nonce":3360352974}]},"public_addr":"172.21.15.16:6812/3360352974","cluster_addr":"172.21.15.16:6817/3360352974","heartbeat_back_addr":"172.21.15.16:6828/3360352974","heartbeat_front_addr":"172.21.15.16:6823/3360352974","state":["exists","up"]},{"osd":4,"uuid":"88260730-1482-4fbd-a40f-cb1124e9df4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6801","nonce":1856844856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6803","nonce":1856844856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6807","nonce":1856844856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6805","nonce":1856844856}]},"public_addr":"172.21.15.89:6801/1856844856","cluster_addr":"172.21.15.89:6803/1856844856","heartbeat_back_addr":"172.21.15.89:6807/1856844856","heartbeat_front_addr":"172.21.15.89:6805/1856844856","state":["exists","up"]},{"osd":5,"uuid":"4cb2c00c-fae0-4dd8-9d27-ec4f0093ccf7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6814","nonce":50395527}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6817","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6820","nonce":50395527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6831","nonce":50395527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6823","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6826","nonce":50395527}]},"public_addr":"172.21.15.89:6814/50395527","cluster_addr":"172.21.15.89:6820/50395527","heartbeat_back_addr":"172.21.15.89:6831/50395527","heartbeat_front_addr":"172.21.15.89:6826/50395527","state":["exists","up"]},{"osd":6,"uuid":"81ce0c19-2554-4cae-acbe-66a05e9e9b9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6809","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6812","nonce":3470471752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6815","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6818","nonce":3470471752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6827","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6829","nonce":3470471752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6821","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6824","nonce":3470471752}]},"public_addr":"172.21.15.89:6812/3470471752","cluster_addr":"172.21.15.89:6818/3470471752","heartbeat_back_addr":"172.21.15.89:6829/3470471752","heartbeat_front_addr":"172.21.15.89:6824/3470471752","state":["exists","up"]},{"osd":7,"uuid":"85e4313b-d37d-442e-8042-153bf20a847d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6811","nonce":1796778630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6813","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6816","nonce":1796778630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6825","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6828","nonce":1796778630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6819","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6822","nonce":1796778630}]},"public_addr":"172.21.15.89:6811/1796778630","cluster_addr":"172.21.15.89:6816/1796778630","heartbeat_back_addr":"172.21.15.89:6828/1796778630","heartbeat_front_addr":"172.21.15.89:6822/1796778630","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.439088+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.529970+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.680512+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:21.044161+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.582111+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.536558+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.409522+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.590229+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"17.1","mappings":[{"from":0,"to":5}]},{"pgid":"17.21","mappings":[{"from":0,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:6835/2564952016":"2024-04-01T15:14:39.715615+0000","172.21.15.16:6833/2564952016":"2024-04-01T15:14:39.715615+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-31T15:14:50.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:50.017+0000 7f5bf2400640 1 -- 172.21.15.16:0/1958669833 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f5bc8068380 msgr2=0x7f5bc806a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:50.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:50.017+0000 7f5bf2400640 1 --2- 172.21.15.16:0/1958669833 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f5bc8068380 0x7f5bc806a840 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f5bdc0b1000 tx=0x7f5bdc0a9070 comp rx=0 tx=0).stop 2024-03-31T15:14:50.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:50.017+0000 7f5bf2400640 1 -- 172.21.15.16:0/1958669833 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5bec15eee0 msgr2=0x7f5bec15cd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:50.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:50.017+0000 7f5bf2400640 1 --2- 172.21.15.16:0/1958669833 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5bec15eee0 0x7f5bec15cd50 secure :-1 s=READY pgs=1772 cs=0 l=1 rev1=1 crypto rx=0x7f5be4007e70 tx=0x7f5be400e4b0 comp rx=0 tx=0).stop 2024-03-31T15:14:50.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:50.017+0000 7f5bf2400640 1 -- 172.21.15.16:0/1958669833 shutdown_connections 2024-03-31T15:14:50.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:50.017+0000 7f5bf2400640 1 --2- 172.21.15.16:0/1958669833 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f5bc8068380 0x7f5bc806a840 unknown :-1 s=CLOSED pgs=750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:50.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:50.017+0000 7f5bf2400640 1 --2- 172.21.15.16:0/1958669833 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5bec15eee0 0x7f5bec15cd50 unknown :-1 s=CLOSED pgs=1772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:50.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:50.017+0000 7f5bf2400640 1 --2- 172.21.15.16:0/1958669833 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f5bec15b3d0 0x7f5bec15c810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:50.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:50.017+0000 7f5bf2400640 1 --2- 172.21.15.16:0/1958669833 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5bec0ce2f0 0x7f5bec1367f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:50.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:50.017+0000 7f5bf2400640 1 -- 172.21.15.16:0/1958669833 >> 172.21.15.16:0/1958669833 conn(0x7f5bec06b620 msgr2=0x7f5bec0cf570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:14:50.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:50.017+0000 7f5bf2400640 1 -- 172.21.15.16:0/1958669833 shutdown_connections 2024-03-31T15:14:50.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:50.017+0000 7f5bf2400640 1 -- 172.21.15.16:0/1958669833 wait complete. 2024-03-31T15:14:50.029 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-31T15:03:22.550787+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 16, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-31T15:14:06.417304+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '108', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 17, 'pool_name': 'cephfs_data', 'create_time': '2024-03-31T15:14:07.622562+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '109', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-31T15:14:50.029 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-31T15:14:50.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:50.181+0000 7fa888e00640 1 Processor -- start 2024-03-31T15:14:50.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:50.181+0000 7fa888e00640 1 -- start start 2024-03-31T15:14:50.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:50.181+0000 7fa888e00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fa8840cf150 0x7fa8841670a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:50.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:50.181+0000 7fa888e00640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa88415ca70 con 0x7fa88406f920 2024-03-31T15:14:50.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:50.181+0000 7fa888e00640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa88415cd50 con 0x7fa8840d0a60 2024-03-31T15:14:50.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:50.181+0000 7fa888e00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fa88413f670 con 0x7fa8840cf150 2024-03-31T15:14:50.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:50.181+0000 7fa881e00640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fa88406f920 0x7fa88406fd00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:55444/0 (socket says 172.21.15.16:55444) 2024-03-31T15:14:50.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:50.181+0000 7fa881e00640 1 -- 172.21.15.16:0/330426624 learned_addr learned my addr 172.21.15.16:0/330426624 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:14:50.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:50.181+0000 7fa882800640 1 --2- 172.21.15.16:0/330426624 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fa8840cf150 0x7fa8841670a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:50.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:50.181+0000 7fa881400640 1 -- 172.21.15.16:0/330426624 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2489623920 0 0) 0x7fa88415ca70 con 0x7fa88406f920 2024-03-31T15:14:50.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:50.181+0000 7fa881400640 1 -- 172.21.15.16:0/330426624 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa868003530 con 0x7fa88406f920 2024-03-31T15:14:50.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:50.181+0000 7fa881400640 1 -- 172.21.15.16:0/330426624 <== mon.1 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3219034223 0 0) 0x7fa88415cd50 con 0x7fa8840d0a60 2024-03-31T15:14:50.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:50.181+0000 7fa881400640 1 -- 172.21.15.16:0/330426624 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa88415ca70 con 0x7fa8840d0a60 2024-03-31T15:14:50.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:50.181+0000 7fa882800640 1 -- 172.21.15.16:0/330426624 >> v1:172.21.15.89:6789/0 conn(0x7fa8840d0a60 legacy=0x7fa8840cea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:50.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:50.181+0000 7fa882800640 1 -- 172.21.15.16:0/330426624 >> v1:172.21.15.16:6789/0 conn(0x7fa88406f920 legacy=0x7fa88406fd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:50.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:50.181+0000 7fa882800640 1 -- 172.21.15.16:0/330426624 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa88415d030 con 0x7fa8840cf150 2024-03-31T15:14:50.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:50.185+0000 7fa882800640 1 --2- 172.21.15.16:0/330426624 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fa8840cf150 0x7fa8841670a0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fa87c009dc0 tx=0x7fa87c01c900 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4551b7705a7ea413 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:50.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:50.185+0000 7fa881400640 1 -- 172.21.15.16:0/330426624 <== mon.2 v2:172.21.15.89:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa87c0b2070 con 0x7fa8840cf150 2024-03-31T15:14:50.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:50.185+0000 7fa881400640 1 -- 172.21.15.16:0/330426624 <== mon.2 v2:172.21.15.89:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa87c0a57c0 con 0x7fa8840cf150 2024-03-31T15:14:50.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:50.185+0000 7fa881400640 1 -- 172.21.15.16:0/330426624 <== mon.2 v2:172.21.15.89:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa87c0a5b90 con 0x7fa8840cf150 2024-03-31T15:14:50.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:50.185+0000 7fa888e00640 1 -- 172.21.15.16:0/330426624 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fa8840cf150 msgr2=0x7fa8841670a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:50.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:50.185+0000 7fa888e00640 1 --2- 172.21.15.16:0/330426624 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fa8840cf150 0x7fa8841670a0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fa87c009dc0 tx=0x7fa87c01c900 comp rx=0 tx=0).stop 2024-03-31T15:14:50.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:50.185+0000 7fa888e00640 1 -- 172.21.15.16:0/330426624 shutdown_connections 2024-03-31T15:14:50.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:50.185+0000 7fa888e00640 1 --2- 172.21.15.16:0/330426624 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fa8840cf150 0x7fa8841670a0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:50.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:50.185+0000 7fa888e00640 1 -- 172.21.15.16:0/330426624 >> 172.21.15.16:0/330426624 conn(0x7fa884067930 msgr2=0x7fa8840b3ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:14:50.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:50.185+0000 7fa888e00640 1 -- 172.21.15.16:0/330426624 shutdown_connections 2024-03-31T15:14:50.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:50.185+0000 7fa888e00640 1 -- 172.21.15.16:0/330426624 wait complete. 2024-03-31T15:14:50.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:50.189+0000 7fa888e00640 1 Processor -- start 2024-03-31T15:14:50.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:50.189+0000 7fa888e00640 1 -- start start 2024-03-31T15:14:50.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:50.189+0000 7fa888e00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fa88406f920 0x7fa884166480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:50.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:50.189+0000 7fa888e00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa8840cf150 0x7fa884150fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:50.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:50.189+0000 7fa888e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa8840d0a60 0x7fa88413e520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:50.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:50.189+0000 7fa888e00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fa88416a190 con 0x7fa8840d0a60 2024-03-31T15:14:50.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:50.189+0000 7fa888e00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fa88416a010 con 0x7fa8840cf150 2024-03-31T15:14:50.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:50.189+0000 7fa888e00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fa88416a310 con 0x7fa88406f920 2024-03-31T15:14:50.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:50.189+0000 7fa882800640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa8840d0a60 0x7fa88413e520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:50.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:50.189+0000 7fa882800640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa8840d0a60 0x7fa88413e520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55326/0 (socket says 172.21.15.16:55326) 2024-03-31T15:14:50.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:50.189+0000 7fa882800640 1 -- 172.21.15.16:0/2433708595 learned_addr learned my addr 172.21.15.16:0/2433708595 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:14:50.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:50.189+0000 7fa881e00640 1 --2- 172.21.15.16:0/2433708595 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fa88406f920 0x7fa884166480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:50.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:50.189+0000 7fa87be00640 1 --2- 172.21.15.16:0/2433708595 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa8840cf150 0x7fa884150fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:50.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:50.189+0000 7fa882800640 1 -- 172.21.15.16:0/2433708595 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fa88406f920 msgr2=0x7fa884166480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:50.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:50.189+0000 7fa882800640 1 --2- 172.21.15.16:0/2433708595 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fa88406f920 0x7fa884166480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:50.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:50.189+0000 7fa882800640 1 -- 172.21.15.16:0/2433708595 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa8840cf150 msgr2=0x7fa884150fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:50.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:50.189+0000 7fa882800640 1 --2- 172.21.15.16:0/2433708595 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa8840cf150 0x7fa884150fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:50.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:50.189+0000 7fa882800640 1 -- 172.21.15.16:0/2433708595 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa884154e10 con 0x7fa8840d0a60 2024-03-31T15:14:50.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:50.189+0000 7fa87be00640 1 --2- 172.21.15.16:0/2433708595 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa8840cf150 0x7fa884150fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:14:50.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:50.189+0000 7fa882800640 1 --2- 172.21.15.16:0/2433708595 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa8840d0a60 0x7fa88413e520 secure :-1 s=READY pgs=1774 cs=0 l=1 rev1=1 crypto rx=0x7fa87c01a530 tx=0x7fa87c0196a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:50.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:50.189+0000 7fa87a000640 1 -- 172.21.15.16:0/2433708595 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa87c0b2050 con 0x7fa8840d0a60 2024-03-31T15:14:50.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:50.189+0000 7fa87a000640 1 -- 172.21.15.16:0/2433708595 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa87c0be070 con 0x7fa8840d0a60 2024-03-31T15:14:50.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:50.189+0000 7fa888e00640 1 -- 172.21.15.16:0/2433708595 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa8841550f0 con 0x7fa8840d0a60 2024-03-31T15:14:50.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:50.189+0000 7fa87a000640 1 -- 172.21.15.16:0/2433708595 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa87c0ad850 con 0x7fa8840d0a60 2024-03-31T15:14:50.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:50.193+0000 7fa888e00640 1 -- 172.21.15.16:0/2433708595 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa8841556b0 con 0x7fa8840d0a60 2024-03-31T15:14:50.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:50.193+0000 7fa888e00640 1 -- 172.21.15.16:0/2433708595 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa884069950 con 0x7fa8840d0a60 2024-03-31T15:14:50.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:50.193+0000 7fa87a000640 1 -- 172.21.15.16:0/2433708595 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fa87c0ad9f0 con 0x7fa8840d0a60 2024-03-31T15:14:50.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:50.193+0000 7fa87a000640 1 --2- 172.21.15.16:0/2433708595 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fa860068380 0x7fa86006a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:50.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:50.193+0000 7fa87a000640 1 -- 172.21.15.16:0/2433708595 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(112..112 src has 1..112) ==== 5853+0+0 (secure 0 0 0) 0x7fa87c121a40 con 0x7fa8840d0a60 2024-03-31T15:14:50.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:50.193+0000 7fa881e00640 1 --2- 172.21.15.16:0/2433708595 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fa860068380 0x7fa86006a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:50.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:50.197+0000 7fa881e00640 1 --2- 172.21.15.16:0/2433708595 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fa860068380 0x7fa86006a840 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7fa874000f70 tx=0x7fa874009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:50.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:50.197+0000 7fa87a000640 1 -- 172.21.15.16:0/2433708595 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fa87c016580 con 0x7fa8840d0a60 2024-03-31T15:14:50.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:50.429+0000 7fa888e00640 1 -- 172.21.15.16:0/2433708595 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fa8840b50e0 con 0x7fa8840d0a60 2024-03-31T15:14:50.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:50.429+0000 7fa87a000640 1 -- 172.21.15.16:0/2433708595 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v112) ==== 93+0+10 (secure 0 0 0) 0x7fa87c0f1f80 con 0x7fa8840d0a60 2024-03-31T15:14:50.422 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2024-03-31T15:14:50.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:50.429+0000 7fa888e00640 1 -- 172.21.15.16:0/2433708595 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fa860068380 msgr2=0x7fa86006a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:50.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:50.429+0000 7fa888e00640 1 --2- 172.21.15.16:0/2433708595 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fa860068380 0x7fa86006a840 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7fa874000f70 tx=0x7fa874009040 comp rx=0 tx=0).stop 2024-03-31T15:14:50.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:50.429+0000 7fa888e00640 1 -- 172.21.15.16:0/2433708595 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa8840d0a60 msgr2=0x7fa88413e520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:50.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:50.429+0000 7fa888e00640 1 --2- 172.21.15.16:0/2433708595 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa8840d0a60 0x7fa88413e520 secure :-1 s=READY pgs=1774 cs=0 l=1 rev1=1 crypto rx=0x7fa87c01a530 tx=0x7fa87c0196a0 comp rx=0 tx=0).stop 2024-03-31T15:14:50.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:50.429+0000 7fa888e00640 1 -- 172.21.15.16:0/2433708595 shutdown_connections 2024-03-31T15:14:50.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:50.429+0000 7fa888e00640 1 --2- 172.21.15.16:0/2433708595 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fa860068380 0x7fa86006a840 unknown :-1 s=CLOSED pgs=751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:50.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:50.429+0000 7fa888e00640 1 --2- 172.21.15.16:0/2433708595 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa8840d0a60 0x7fa88413e520 unknown :-1 s=CLOSED pgs=1774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:50.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:50.429+0000 7fa888e00640 1 --2- 172.21.15.16:0/2433708595 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa8840cf150 0x7fa884150fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:50.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:50.429+0000 7fa888e00640 1 --2- 172.21.15.16:0/2433708595 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fa88406f920 0x7fa884166480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:50.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:50.429+0000 7fa888e00640 1 -- 172.21.15.16:0/2433708595 >> 172.21.15.16:0/2433708595 conn(0x7fa884067930 msgr2=0x7fa884157f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:14:50.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:50.429+0000 7fa888e00640 1 -- 172.21.15.16:0/2433708595 shutdown_connections 2024-03-31T15:14:50.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:50.429+0000 7fa888e00640 1 -- 172.21.15.16:0/2433708595 wait complete. 2024-03-31T15:14:50.443 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-31T15:14:50.585 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:50.593+0000 7fb428600640 1 Processor -- start 2024-03-31T15:14:50.586 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:50.593+0000 7fb428600640 1 -- start start 2024-03-31T15:14:50.587 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:50.593+0000 7fb428600640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb42006f920 0x7fb42006fd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:50.587 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:50.593+0000 7fb428600640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fb4200d0a60 0x7fb4200cea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:50.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:50.593+0000 7fb428600640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb4200cef80 0x7fb420070e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:50.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:50.593+0000 7fb428600640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7fb42013f670 con 0x7fb42006f920 2024-03-31T15:14:50.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:50.593+0000 7fb428600640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7fb420070240 con 0x7fb4200d0a60 2024-03-31T15:14:50.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:50.593+0000 7fb428600640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fb42015ca70 con 0x7fb4200cef80 2024-03-31T15:14:50.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:50.593+0000 7fb425c00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb42006f920 0x7fb42006fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:50.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:50.593+0000 7fb425c00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb42006f920 0x7fb42006fd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55330/0 (socket says 172.21.15.16:55330) 2024-03-31T15:14:50.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:50.593+0000 7fb425c00640 1 -- 172.21.15.16:0/3779952053 learned_addr learned my addr 172.21.15.16:0/3779952053 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:14:50.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:50.593+0000 7fb425200640 1 --2- 172.21.15.16:0/3779952053 >> v2:172.21.15.89:3300/0 conn(0x7fb4200d0a60 0x7fb4200cea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:50.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:50.593+0000 7fb426600640 1 --2- 172.21.15.16:0/3779952053 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb4200cef80 0x7fb420070e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:50.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:50.593+0000 7fb425c00640 1 -- 172.21.15.16:0/3779952053 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb4200cef80 msgr2=0x7fb420070e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:14:50.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:50.593+0000 7fb425c00640 1 --2- 172.21.15.16:0/3779952053 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb4200cef80 0x7fb420070e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:50.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:50.593+0000 7fb425c00640 1 -- 172.21.15.16:0/3779952053 >> v2:172.21.15.89:3300/0 conn(0x7fb4200d0a60 msgr2=0x7fb4200cea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:14:50.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:50.593+0000 7fb425c00640 1 --2- 172.21.15.16:0/3779952053 >> v2:172.21.15.89:3300/0 conn(0x7fb4200d0a60 0x7fb4200cea40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:50.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:50.593+0000 7fb425c00640 1 -- 172.21.15.16:0/3779952053 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb42015cd50 con 0x7fb42006f920 2024-03-31T15:14:50.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:50.597+0000 7fb425c00640 1 --2- 172.21.15.16:0/3779952053 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb42006f920 0x7fb42006fd00 secure :-1 s=READY pgs=1775 cs=0 l=1 rev1=1 crypto rx=0x7fb4200736b0 tx=0x7fb41401c260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40074f24bf9dd653 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:50.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:50.597+0000 7fb41fe00640 1 -- 172.21.15.16:0/3779952053 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4140b0070 con 0x7fb42006f920 2024-03-31T15:14:50.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:50.597+0000 7fb41fe00640 1 -- 172.21.15.16:0/3779952053 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb4140ab600 con 0x7fb42006f920 2024-03-31T15:14:50.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:50.597+0000 7fb41fe00640 1 -- 172.21.15.16:0/3779952053 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4140ab9d0 con 0x7fb42006f920 2024-03-31T15:14:50.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:50.597+0000 7fb428600640 1 -- 172.21.15.16:0/3779952053 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb42006f920 msgr2=0x7fb42006fd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:50.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:50.597+0000 7fb428600640 1 --2- 172.21.15.16:0/3779952053 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb42006f920 0x7fb42006fd00 secure :-1 s=READY pgs=1775 cs=0 l=1 rev1=1 crypto rx=0x7fb4200736b0 tx=0x7fb41401c260 comp rx=0 tx=0).stop 2024-03-31T15:14:50.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:50.597+0000 7fb428600640 1 -- 172.21.15.16:0/3779952053 shutdown_connections 2024-03-31T15:14:50.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:50.597+0000 7fb428600640 1 --2- 172.21.15.16:0/3779952053 >> v2:172.21.15.89:3300/0 conn(0x7fb4200d0a60 0x7fb4200cea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:50.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:50.597+0000 7fb428600640 1 --2- 172.21.15.16:0/3779952053 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb4200cef80 0x7fb420070e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:50.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:50.597+0000 7fb428600640 1 --2- 172.21.15.16:0/3779952053 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb42006f920 0x7fb42006fd00 unknown :-1 s=CLOSED pgs=1775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:50.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:50.597+0000 7fb428600640 1 -- 172.21.15.16:0/3779952053 >> 172.21.15.16:0/3779952053 conn(0x7fb420067930 msgr2=0x7fb4200b3ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:14:50.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:50.597+0000 7fb428600640 1 -- 172.21.15.16:0/3779952053 shutdown_connections 2024-03-31T15:14:50.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:50.597+0000 7fb428600640 1 -- 172.21.15.16:0/3779952053 wait complete. 2024-03-31T15:14:50.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:50.601+0000 7fb428600640 1 Processor -- start 2024-03-31T15:14:50.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:50.601+0000 7fb428600640 1 -- start start 2024-03-31T15:14:50.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:50.601+0000 7fb428600640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb42006f920 0x7fb42013ad50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:50.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:50.601+0000 7fb428600640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb4200cef80 0x7fb42013cc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:50.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:50.601+0000 7fb428600640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb4200d0a60 0x7fb420140ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:50.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:50.601+0000 7fb428600640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fb4201419c0 con 0x7fb4200d0a60 2024-03-31T15:14:50.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:50.601+0000 7fb428600640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fb4200cf390 con 0x7fb42006f920 2024-03-31T15:14:50.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:50.601+0000 7fb428600640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fb420141b40 con 0x7fb4200cef80 2024-03-31T15:14:50.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:50.605+0000 7fb426600640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb4200d0a60 0x7fb420140ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:50.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:50.605+0000 7fb426600640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb4200d0a60 0x7fb420140ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55346/0 (socket says 172.21.15.16:55346) 2024-03-31T15:14:50.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:50.605+0000 7fb426600640 1 -- 172.21.15.16:0/1860624996 learned_addr learned my addr 172.21.15.16:0/1860624996 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:14:50.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:50.605+0000 7fb425c00640 1 --2- 172.21.15.16:0/1860624996 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb42006f920 0x7fb42013ad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:50.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:50.605+0000 7fb425200640 1 --2- 172.21.15.16:0/1860624996 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb4200cef80 0x7fb42013cc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:50.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:50.605+0000 7fb426600640 1 -- 172.21.15.16:0/1860624996 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb4200cef80 msgr2=0x7fb42013cc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:50.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:50.605+0000 7fb426600640 1 --2- 172.21.15.16:0/1860624996 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb4200cef80 0x7fb42013cc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:50.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:50.605+0000 7fb426600640 1 -- 172.21.15.16:0/1860624996 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb42006f920 msgr2=0x7fb42013ad50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:50.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:50.605+0000 7fb426600640 1 --2- 172.21.15.16:0/1860624996 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb42006f920 0x7fb42013ad50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:50.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:50.605+0000 7fb426600640 1 -- 172.21.15.16:0/1860624996 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb42006cf50 con 0x7fb4200d0a60 2024-03-31T15:14:50.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:50.605+0000 7fb426600640 1 --2- 172.21.15.16:0/1860624996 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb4200d0a60 0x7fb420140ae0 secure :-1 s=READY pgs=1776 cs=0 l=1 rev1=1 crypto rx=0x7fb418007e40 tx=0x7fb41800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:50.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:50.605+0000 7fb41d600640 1 -- 172.21.15.16:0/1860624996 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb418014620 con 0x7fb4200d0a60 2024-03-31T15:14:50.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:50.605+0000 7fb41d600640 1 -- 172.21.15.16:0/1860624996 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb418019070 con 0x7fb4200d0a60 2024-03-31T15:14:50.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:50.605+0000 7fb41d600640 1 -- 172.21.15.16:0/1860624996 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb418014a20 con 0x7fb4200d0a60 2024-03-31T15:14:50.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:50.605+0000 7fb428600640 1 -- 172.21.15.16:0/1860624996 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb42015ef20 con 0x7fb4200d0a60 2024-03-31T15:14:50.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:50.605+0000 7fb428600640 1 -- 172.21.15.16:0/1860624996 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb42015f200 con 0x7fb4200d0a60 2024-03-31T15:14:50.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:50.605+0000 7fb428600640 1 -- 172.21.15.16:0/1860624996 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb420069950 con 0x7fb4200d0a60 2024-03-31T15:14:50.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:50.609+0000 7fb41d600640 1 -- 172.21.15.16:0/1860624996 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fb418014bc0 con 0x7fb4200d0a60 2024-03-31T15:14:50.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:50.609+0000 7fb41d600640 1 --2- 172.21.15.16:0/1860624996 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fb400068380 0x7fb40006a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:50.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:50.609+0000 7fb41d600640 1 -- 172.21.15.16:0/1860624996 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(112..112 src has 1..112) ==== 5853+0+0 (secure 0 0 0) 0x7fb418087170 con 0x7fb4200d0a60 2024-03-31T15:14:50.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:50.609+0000 7fb425c00640 1 --2- 172.21.15.16:0/1860624996 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fb400068380 0x7fb40006a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:50.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:50.609+0000 7fb425c00640 1 --2- 172.21.15.16:0/1860624996 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fb400068380 0x7fb40006a840 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7fb4140ad000 tx=0x7fb414008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:50.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:50.617+0000 7fb41d600640 1 -- 172.21.15.16:0/1860624996 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fb4180570e0 con 0x7fb4200d0a60 2024-03-31T15:14:50.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:50.785+0000 7fb428600640 1 -- 172.21.15.16:0/1860624996 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fb420166e20 con 0x7fb4200d0a60 2024-03-31T15:14:50.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:50.785+0000 7fb41d600640 1 -- 172.21.15.16:0/1860624996 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v112) ==== 104+0+11 (secure 0 0 0) 0x7fb41805b3e0 con 0x7fb4200d0a60 2024-03-31T15:14:50.781 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2024-03-31T15:14:50.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:50.789+0000 7fb428600640 1 -- 172.21.15.16:0/1860624996 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fb400068380 msgr2=0x7fb40006a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:50.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:50.789+0000 7fb428600640 1 --2- 172.21.15.16:0/1860624996 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fb400068380 0x7fb40006a840 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7fb4140ad000 tx=0x7fb414008040 comp rx=0 tx=0).stop 2024-03-31T15:14:50.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:50.789+0000 7fb428600640 1 -- 172.21.15.16:0/1860624996 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb4200d0a60 msgr2=0x7fb420140ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:50.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:50.789+0000 7fb428600640 1 --2- 172.21.15.16:0/1860624996 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb4200d0a60 0x7fb420140ae0 secure :-1 s=READY pgs=1776 cs=0 l=1 rev1=1 crypto rx=0x7fb418007e40 tx=0x7fb41800e4b0 comp rx=0 tx=0).stop 2024-03-31T15:14:50.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:50.789+0000 7fb428600640 1 -- 172.21.15.16:0/1860624996 shutdown_connections 2024-03-31T15:14:50.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:50.789+0000 7fb428600640 1 --2- 172.21.15.16:0/1860624996 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fb400068380 0x7fb40006a840 unknown :-1 s=CLOSED pgs=752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:50.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:50.789+0000 7fb428600640 1 --2- 172.21.15.16:0/1860624996 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb4200d0a60 0x7fb420140ae0 unknown :-1 s=CLOSED pgs=1776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:50.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:50.789+0000 7fb428600640 1 --2- 172.21.15.16:0/1860624996 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb4200cef80 0x7fb42013cc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:50.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:50.789+0000 7fb428600640 1 --2- 172.21.15.16:0/1860624996 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb42006f920 0x7fb42013ad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:50.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:50.789+0000 7fb428600640 1 -- 172.21.15.16:0/1860624996 >> 172.21.15.16:0/1860624996 conn(0x7fb420067930 msgr2=0x7fb4200d1730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:14:50.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:50.789+0000 7fb428600640 1 -- 172.21.15.16:0/1860624996 shutdown_connections 2024-03-31T15:14:50.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:50.789+0000 7fb428600640 1 -- 172.21.15.16:0/1860624996 wait complete. 2024-03-31T15:14:50.801 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-31T15:14:50.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:50.985+0000 7f9a39800640 1 Processor -- start 2024-03-31T15:14:50.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:50.985+0000 7f9a39800640 1 -- start start 2024-03-31T15:14:50.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:50.989+0000 7f9a39800640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f9a3406dbe0 0x7f9a340b3130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:50.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:50.989+0000 7f9a39800640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f9a34073310 0x7f9a340ada40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:50.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:50.989+0000 7f9a39800640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9a340737a0 con 0x7f9a34147d60 2024-03-31T15:14:50.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:50.989+0000 7f9a39800640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f9a34148140 con 0x7f9a34073310 2024-03-31T15:14:50.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:50.989+0000 7f9a39800640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f9a340b37c0 con 0x7f9a3406dbe0 2024-03-31T15:14:50.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:50.989+0000 7f9a33400640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f9a34073310 0x7f9a340ada40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:50.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:50.989+0000 7f9a2a000640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f9a3406dbe0 0x7f9a340b3130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:50.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:50.989+0000 7f9a32a00640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f9a34147d60 0x7f9a3406d4d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.16:49760/0 (socket says 172.21.15.16:49760) 2024-03-31T15:14:50.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:50.989+0000 7f9a32a00640 1 -- 172.21.15.16:0/783598593 learned_addr learned my addr 172.21.15.16:0/783598593 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:14:50.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:50.989+0000 7f9a32000640 1 -- 172.21.15.16:0/783598593 <== mon.1 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1480866805 0 0) 0x7f9a340737a0 con 0x7f9a34147d60 2024-03-31T15:14:50.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:50.989+0000 7f9a32000640 1 -- 172.21.15.16:0/783598593 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9a18003530 con 0x7f9a34147d60 2024-03-31T15:14:50.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:50.989+0000 7f9a33400640 1 -- 172.21.15.16:0/783598593 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f9a3406dbe0 msgr2=0x7f9a340b3130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:14:50.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:50.989+0000 7f9a33400640 1 --2- 172.21.15.16:0/783598593 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f9a3406dbe0 0x7f9a340b3130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:50.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:50.989+0000 7f9a33400640 1 -- 172.21.15.16:0/783598593 >> v1:172.21.15.89:6789/0 conn(0x7f9a34147d60 legacy=0x7f9a3406d4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:50.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:50.989+0000 7f9a33400640 1 -- 172.21.15.16:0/783598593 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a340b3aa0 con 0x7f9a34073310 2024-03-31T15:14:50.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:50.989+0000 7f9a2a000640 1 --2- 172.21.15.16:0/783598593 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f9a3406dbe0 0x7f9a340b3130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:14:50.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:50.993+0000 7f9a33400640 1 --2- 172.21.15.16:0/783598593 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9a34073310 0x7f9a340ada40 secure :-1 s=READY pgs=1777 cs=0 l=1 rev1=1 crypto rx=0x7f9a2c0097c0 tx=0x7f9a2c01c470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=569494c90f173c82 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:50.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:50.993+0000 7f9a32000640 1 -- 172.21.15.16:0/783598593 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a2c0a7070 con 0x7f9a34073310 2024-03-31T15:14:50.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:50.993+0000 7f9a32000640 1 -- 172.21.15.16:0/783598593 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9a2c0b0070 con 0x7f9a34073310 2024-03-31T15:14:50.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:50.993+0000 7f9a32000640 1 -- 172.21.15.16:0/783598593 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a2c0ab890 con 0x7f9a34073310 2024-03-31T15:14:50.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:50.993+0000 7f9a39800640 1 -- 172.21.15.16:0/783598593 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9a34073310 msgr2=0x7f9a340ada40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:50.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:50.993+0000 7f9a39800640 1 --2- 172.21.15.16:0/783598593 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9a34073310 0x7f9a340ada40 secure :-1 s=READY pgs=1777 cs=0 l=1 rev1=1 crypto rx=0x7f9a2c0097c0 tx=0x7f9a2c01c470 comp rx=0 tx=0).stop 2024-03-31T15:14:50.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:50.993+0000 7f9a39800640 1 -- 172.21.15.16:0/783598593 shutdown_connections 2024-03-31T15:14:50.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:50.993+0000 7f9a39800640 1 --2- 172.21.15.16:0/783598593 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9a34073310 0x7f9a340ada40 unknown :-1 s=CLOSED pgs=1777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:50.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:50.993+0000 7f9a39800640 1 --2- 172.21.15.16:0/783598593 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f9a3406dbe0 0x7f9a340b3130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:50.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:50.993+0000 7f9a39800640 1 -- 172.21.15.16:0/783598593 >> 172.21.15.16:0/783598593 conn(0x7f9a3406ee40 msgr2=0x7f9a3406b620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:14:50.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:50.993+0000 7f9a39800640 1 -- 172.21.15.16:0/783598593 shutdown_connections 2024-03-31T15:14:50.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:50.993+0000 7f9a39800640 1 -- 172.21.15.16:0/783598593 wait complete. 2024-03-31T15:14:50.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:50.997+0000 7f9a39800640 1 Processor -- start 2024-03-31T15:14:50.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:50.997+0000 7f9a39800640 1 -- start start 2024-03-31T15:14:50.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:50.997+0000 7f9a39800640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9a3406dbe0 0x7f9a34141fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:50.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:50.997+0000 7f9a39800640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f9a34073310 0x7f9a3415eca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:50.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:50.997+0000 7f9a39800640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9a34147d60 0x7f9a3415f1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:50.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:50.997+0000 7f9a39800640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f9a340ae910 con 0x7f9a3406dbe0 2024-03-31T15:14:50.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:50.997+0000 7f9a39800640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f9a340ae790 con 0x7f9a34147d60 2024-03-31T15:14:50.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:50.997+0000 7f9a39800640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f9a340aea90 con 0x7f9a34073310 2024-03-31T15:14:50.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:50.997+0000 7f9a32a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9a3406dbe0 0x7f9a34141fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:50.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:50.997+0000 7f9a32a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9a3406dbe0 0x7f9a34141fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55358/0 (socket says 172.21.15.16:55358) 2024-03-31T15:14:50.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:50.997+0000 7f9a32a00640 1 -- 172.21.15.16:0/1148671414 learned_addr learned my addr 172.21.15.16:0/1148671414 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:14:50.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:50.997+0000 7f9a33400640 1 --2- 172.21.15.16:0/1148671414 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9a34147d60 0x7f9a3415f1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:50.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:50.997+0000 7f9a2a000640 1 --2- 172.21.15.16:0/1148671414 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f9a34073310 0x7f9a3415eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:50.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:50.997+0000 7f9a32a00640 1 -- 172.21.15.16:0/1148671414 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f9a34073310 msgr2=0x7f9a3415eca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:50.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:50.997+0000 7f9a32a00640 1 --2- 172.21.15.16:0/1148671414 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f9a34073310 0x7f9a3415eca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:50.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:50.997+0000 7f9a32a00640 1 -- 172.21.15.16:0/1148671414 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9a34147d60 msgr2=0x7f9a3415f1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:50.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:50.997+0000 7f9a32a00640 1 --2- 172.21.15.16:0/1148671414 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9a34147d60 0x7f9a3415f1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:50.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:50.997+0000 7f9a32a00640 1 -- 172.21.15.16:0/1148671414 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a341424f0 con 0x7f9a3406dbe0 2024-03-31T15:14:50.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:50.997+0000 7f9a33400640 1 --2- 172.21.15.16:0/1148671414 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9a34147d60 0x7f9a3415f1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:14:50.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:50.997+0000 7f9a32a00640 1 --2- 172.21.15.16:0/1148671414 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9a3406dbe0 0x7f9a34141fb0 secure :-1 s=READY pgs=1778 cs=0 l=1 rev1=1 crypto rx=0x7f9a2400c3f0 tx=0x7f9a2400c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:50.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:50.997+0000 7f9a2b400640 1 -- 172.21.15.16:0/1148671414 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a24017070 con 0x7f9a3406dbe0 2024-03-31T15:14:50.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:50.997+0000 7f9a2b400640 1 -- 172.21.15.16:0/1148671414 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9a24005cf0 con 0x7f9a3406dbe0 2024-03-31T15:14:50.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:50.997+0000 7f9a39800640 1 -- 172.21.15.16:0/1148671414 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9a34142790 con 0x7f9a3406dbe0 2024-03-31T15:14:50.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:50.997+0000 7f9a2b400640 1 -- 172.21.15.16:0/1148671414 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a240072f0 con 0x7f9a3406dbe0 2024-03-31T15:14:50.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:50.997+0000 7f9a39800640 1 -- 172.21.15.16:0/1148671414 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9a34142d50 con 0x7f9a3406dbe0 2024-03-31T15:14:50.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:50.997+0000 7f9a39800640 1 -- 172.21.15.16:0/1148671414 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9a34069950 con 0x7f9a3406dbe0 2024-03-31T15:14:50.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:51.001+0000 7f9a2b400640 1 -- 172.21.15.16:0/1148671414 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f9a2401a070 con 0x7f9a3406dbe0 2024-03-31T15:14:50.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:51.001+0000 7f9a2b400640 1 --2- 172.21.15.16:0/1148671414 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f9a10068380 0x7f9a1006a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:50.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:51.001+0000 7f9a2b400640 1 -- 172.21.15.16:0/1148671414 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(112..112 src has 1..112) ==== 5853+0+0 (secure 0 0 0) 0x7f9a24085f50 con 0x7f9a3406dbe0 2024-03-31T15:14:50.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:51.001+0000 7f9a2a000640 1 --2- 172.21.15.16:0/1148671414 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f9a10068380 0x7f9a1006a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:50.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:51.001+0000 7f9a2a000640 1 --2- 172.21.15.16:0/1148671414 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f9a10068380 0x7f9a1006a840 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7f9a1c009bd0 tx=0x7f9a1c00b040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:50.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:51.005+0000 7f9a2b400640 1 -- 172.21.15.16:0/1148671414 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f9a240563e0 con 0x7f9a3406dbe0 2024-03-31T15:14:51.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:51.169+0000 7f9a39800640 1 -- 172.21.15.16:0/1148671414 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f9a3406e470 con 0x7f9a3406dbe0 2024-03-31T15:14:51.164 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:51.169+0000 7f9a2b400640 1 -- 172.21.15.16:0/1148671414 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v112) ==== 100+0+11 (secure 0 0 0) 0x7f9a24055ce0 con 0x7f9a3406dbe0 2024-03-31T15:14:51.164 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 64 2024-03-31T15:14:51.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:51.173+0000 7f9a39800640 1 -- 172.21.15.16:0/1148671414 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f9a10068380 msgr2=0x7f9a1006a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:51.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:51.173+0000 7f9a39800640 1 --2- 172.21.15.16:0/1148671414 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f9a10068380 0x7f9a1006a840 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7f9a1c009bd0 tx=0x7f9a1c00b040 comp rx=0 tx=0).stop 2024-03-31T15:14:51.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:51.173+0000 7f9a39800640 1 -- 172.21.15.16:0/1148671414 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9a3406dbe0 msgr2=0x7f9a34141fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:51.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:51.173+0000 7f9a39800640 1 --2- 172.21.15.16:0/1148671414 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9a3406dbe0 0x7f9a34141fb0 secure :-1 s=READY pgs=1778 cs=0 l=1 rev1=1 crypto rx=0x7f9a2400c3f0 tx=0x7f9a2400c8b0 comp rx=0 tx=0).stop 2024-03-31T15:14:51.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:51.173+0000 7f9a39800640 1 -- 172.21.15.16:0/1148671414 shutdown_connections 2024-03-31T15:14:51.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:51.173+0000 7f9a39800640 1 --2- 172.21.15.16:0/1148671414 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f9a10068380 0x7f9a1006a840 unknown :-1 s=CLOSED pgs=753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:51.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:51.173+0000 7f9a39800640 1 --2- 172.21.15.16:0/1148671414 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9a34147d60 0x7f9a3415f1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:51.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:51.173+0000 7f9a39800640 1 --2- 172.21.15.16:0/1148671414 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f9a34073310 0x7f9a3415eca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:51.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:51.173+0000 7f9a39800640 1 --2- 172.21.15.16:0/1148671414 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9a3406dbe0 0x7f9a34141fb0 unknown :-1 s=CLOSED pgs=1778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:51.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:51.173+0000 7f9a39800640 1 -- 172.21.15.16:0/1148671414 >> 172.21.15.16:0/1148671414 conn(0x7f9a3406ee40 msgr2=0x7f9a34162760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:14:51.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:51.173+0000 7f9a39800640 1 -- 172.21.15.16:0/1148671414 shutdown_connections 2024-03-31T15:14:51.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:51.173+0000 7f9a39800640 1 -- 172.21.15.16:0/1148671414 wait complete. 2024-03-31T15:14:51.186 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-31T15:14:51.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:51.333+0000 7fc44a600640 1 Processor -- start 2024-03-31T15:14:51.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:51.333+0000 7fc44a600640 1 -- start start 2024-03-31T15:14:51.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:51.333+0000 7fc44a600640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fc4440d1280 0x7fc4440d1660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:51.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:51.333+0000 7fc44a600640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fc4440b6460 0x7fc444169080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:51.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:51.333+0000 7fc44a600640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc4440d1ba0 con 0x7fc4440b9c60 2024-03-31T15:14:51.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:51.333+0000 7fc44a600640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7fc4440d26a0 con 0x7fc4440d1280 2024-03-31T15:14:51.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:51.333+0000 7fc44a600640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fc4440d3990 con 0x7fc4440b6460 2024-03-31T15:14:51.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:51.333+0000 7fc43aa00640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fc4440b9c60 0x7fc4440b5d50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:55454/0 (socket says 172.21.15.16:55454) 2024-03-31T15:14:51.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:51.333+0000 7fc43aa00640 1 -- 172.21.15.16:0/643975951 learned_addr learned my addr 172.21.15.16:0/643975951 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:14:51.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:51.333+0000 7fc443e00640 1 --2- 172.21.15.16:0/643975951 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fc4440b6460 0x7fc444169080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:51.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:51.333+0000 7fc443400640 1 --2- 172.21.15.16:0/643975951 >> v2:172.21.15.89:3300/0 conn(0x7fc4440d1280 0x7fc4440d1660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:51.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:51.337+0000 7fc442a00640 1 -- 172.21.15.16:0/643975951 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2917822221 0 0) 0x7fc4440d1ba0 con 0x7fc4440b9c60 2024-03-31T15:14:51.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:51.337+0000 7fc442a00640 1 -- 172.21.15.16:0/643975951 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc428003530 con 0x7fc4440b9c60 2024-03-31T15:14:51.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:51.337+0000 7fc442a00640 1 -- 172.21.15.16:0/643975951 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2257642003 0 0) 0x7fc428003530 con 0x7fc4440b9c60 2024-03-31T15:14:51.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:51.337+0000 7fc442a00640 1 -- 172.21.15.16:0/643975951 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fc4440b6460 msgr2=0x7fc444169080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:14:51.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:51.337+0000 7fc442a00640 1 --2- 172.21.15.16:0/643975951 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fc4440b6460 0x7fc444169080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:51.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:51.337+0000 7fc442a00640 1 -- 172.21.15.16:0/643975951 >> v2:172.21.15.89:3300/0 conn(0x7fc4440d1280 msgr2=0x7fc4440d1660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:14:51.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:51.337+0000 7fc442a00640 1 --2- 172.21.15.16:0/643975951 >> v2:172.21.15.89:3300/0 conn(0x7fc4440d1280 0x7fc4440d1660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:51.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:51.337+0000 7fc442a00640 1 -- 172.21.15.16:0/643975951 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4440d28f0 con 0x7fc4440b9c60 2024-03-31T15:14:51.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:51.337+0000 7fc442a00640 1 -- 172.21.15.16:0/643975951 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 214807623 0 0) 0x7fc42c001680 con 0x7fc4440b9c60 2024-03-31T15:14:51.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:51.337+0000 7fc442a00640 1 -- 172.21.15.16:0/643975951 >> v1:172.21.15.16:6789/0 conn(0x7fc4440b9c60 legacy=0x7fc4440b5d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:51.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:51.337+0000 7fc442a00640 1 --2- 172.21.15.16:0/643975951 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fc4440b6460 0x7fc4280039c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:51.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:51.337+0000 7fc442a00640 1 --2- 172.21.15.16:0/643975951 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc428003f90 0x7fc428004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:51.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:51.337+0000 7fc442a00640 1 --2- 172.21.15.16:0/643975951 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc428004d50 0x7fc4280051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:51.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:51.337+0000 7fc442a00640 1 -- 172.21.15.16:0/643975951 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fc4440d26a0 con 0x7fc428003f90 2024-03-31T15:14:51.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:51.337+0000 7fc442a00640 1 -- 172.21.15.16:0/643975951 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fc4440d3990 con 0x7fc428004d50 2024-03-31T15:14:51.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:51.337+0000 7fc442a00640 1 -- 172.21.15.16:0/643975951 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fc42800ba80 con 0x7fc4440b6460 2024-03-31T15:14:51.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:51.337+0000 7fc443400640 1 --2- 172.21.15.16:0/643975951 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fc4440b6460 0x7fc4280039c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:51.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:51.337+0000 7fc43aa00640 1 --2- 172.21.15.16:0/643975951 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc428003f90 0x7fc428004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:51.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:51.337+0000 7fc443e00640 1 --2- 172.21.15.16:0/643975951 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc428004d50 0x7fc4280051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:51.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:51.337+0000 7fc443400640 1 -- 172.21.15.16:0/643975951 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc428004d50 msgr2=0x7fc4280051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:14:51.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:51.337+0000 7fc443400640 1 --2- 172.21.15.16:0/643975951 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc428004d50 0x7fc4280051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:51.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:51.337+0000 7fc443400640 1 -- 172.21.15.16:0/643975951 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc428003f90 msgr2=0x7fc428004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:14:51.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:51.337+0000 7fc443400640 1 --2- 172.21.15.16:0/643975951 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc428003f90 0x7fc428004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:51.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:51.337+0000 7fc443400640 1 -- 172.21.15.16:0/643975951 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc428003530 con 0x7fc4440b6460 2024-03-31T15:14:51.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:51.337+0000 7fc43aa00640 1 --2- 172.21.15.16:0/643975951 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc428003f90 0x7fc428004810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:14:51.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:51.341+0000 7fc443400640 1 --2- 172.21.15.16:0/643975951 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fc4440b6460 0x7fc4280039c0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fc434009ae0 tx=0x7fc43401cab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2de1de0cd7eb891f server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:51.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:51.341+0000 7fc442a00640 1 -- 172.21.15.16:0/643975951 <== mon.2 v2:172.21.15.89:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4340096b0 con 0x7fc4440b6460 2024-03-31T15:14:51.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:51.341+0000 7fc442a00640 1 -- 172.21.15.16:0/643975951 <== mon.2 v2:172.21.15.89:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc4340a6490 con 0x7fc4440b6460 2024-03-31T15:14:51.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:51.341+0000 7fc442a00640 1 -- 172.21.15.16:0/643975951 <== mon.2 v2:172.21.15.89:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4340a68c0 con 0x7fc4440b6460 2024-03-31T15:14:51.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:51.341+0000 7fc44a600640 1 -- 172.21.15.16:0/643975951 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fc4440b6460 msgr2=0x7fc4280039c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:51.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:51.341+0000 7fc44a600640 1 --2- 172.21.15.16:0/643975951 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fc4440b6460 0x7fc4280039c0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fc434009ae0 tx=0x7fc43401cab0 comp rx=0 tx=0).stop 2024-03-31T15:14:51.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:51.341+0000 7fc443e00640 1 -- 172.21.15.16:0/643975951 reap_dead start 2024-03-31T15:14:51.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:51.341+0000 7fc44a600640 1 -- 172.21.15.16:0/643975951 shutdown_connections 2024-03-31T15:14:51.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:51.341+0000 7fc44a600640 1 -- 172.21.15.16:0/643975951 >> 172.21.15.16:0/643975951 conn(0x7fc444067730 msgr2=0x7fc444067b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:14:51.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:51.341+0000 7fc44a600640 1 -- 172.21.15.16:0/643975951 shutdown_connections 2024-03-31T15:14:51.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:51.341+0000 7fc44a600640 1 -- 172.21.15.16:0/643975951 wait complete. 2024-03-31T15:14:51.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:51.345+0000 7fc44a600640 1 Processor -- start 2024-03-31T15:14:51.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:51.345+0000 7fc44a600640 1 -- start start 2024-03-31T15:14:51.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:51.345+0000 7fc44a600640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc4440b6460 0x7fc444159130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:51.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:51.345+0000 7fc44a600640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc4441f7ca0 0x7fc4440c9560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:51.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:51.345+0000 7fc44a600640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fc4440c9aa0 0x7fc44415b150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:51.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:51.345+0000 7fc44a600640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fc4440d3e90 con 0x7fc4440b6460 2024-03-31T15:14:51.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:51.345+0000 7fc44a600640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fc444161d80 con 0x7fc4441f7ca0 2024-03-31T15:14:51.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:51.345+0000 7fc44a600640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fc4440d4010 con 0x7fc4440c9aa0 2024-03-31T15:14:51.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:51.345+0000 7fc443400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc4440b6460 0x7fc444159130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:51.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:51.345+0000 7fc443400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc4440b6460 0x7fc444159130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55378/0 (socket says 172.21.15.16:55378) 2024-03-31T15:14:51.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:51.345+0000 7fc443400640 1 -- 172.21.15.16:0/778964100 learned_addr learned my addr 172.21.15.16:0/778964100 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:14:51.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:51.345+0000 7fc443e00640 1 --2- 172.21.15.16:0/778964100 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fc4440c9aa0 0x7fc44415b150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:51.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:51.345+0000 7fc43aa00640 1 --2- 172.21.15.16:0/778964100 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc4441f7ca0 0x7fc4440c9560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:51.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:51.345+0000 7fc443400640 1 -- 172.21.15.16:0/778964100 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fc4440c9aa0 msgr2=0x7fc44415b150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:51.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:51.345+0000 7fc443400640 1 --2- 172.21.15.16:0/778964100 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fc4440c9aa0 0x7fc44415b150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:51.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:51.345+0000 7fc443400640 1 -- 172.21.15.16:0/778964100 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc4441f7ca0 msgr2=0x7fc4440c9560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:51.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:51.345+0000 7fc443400640 1 --2- 172.21.15.16:0/778964100 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc4441f7ca0 0x7fc4440c9560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:51.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:51.345+0000 7fc443400640 1 -- 172.21.15.16:0/778964100 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4440cb440 con 0x7fc4440b6460 2024-03-31T15:14:51.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:51.345+0000 7fc443400640 1 --2- 172.21.15.16:0/778964100 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc4440b6460 0x7fc444159130 secure :-1 s=READY pgs=1780 cs=0 l=1 rev1=1 crypto rx=0x7fc43401c8e0 tx=0x7fc428005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:51.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:51.345+0000 7fc43be00640 1 -- 172.21.15.16:0/778964100 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4340be070 con 0x7fc4440b6460 2024-03-31T15:14:51.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:51.345+0000 7fc43be00640 1 -- 172.21.15.16:0/778964100 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc43400f000 con 0x7fc4440b6460 2024-03-31T15:14:51.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:51.345+0000 7fc43be00640 1 -- 172.21.15.16:0/778964100 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4340a6de0 con 0x7fc4440b6460 2024-03-31T15:14:51.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:51.345+0000 7fc44a600640 1 -- 172.21.15.16:0/778964100 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc4440cb6e0 con 0x7fc4440b6460 2024-03-31T15:14:51.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:51.345+0000 7fc44a600640 1 -- 172.21.15.16:0/778964100 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc4440cbca0 con 0x7fc4440b6460 2024-03-31T15:14:51.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:51.349+0000 7fc44a600640 1 -- 172.21.15.16:0/778964100 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc444069950 con 0x7fc4440b6460 2024-03-31T15:14:51.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:51.349+0000 7fc43be00640 1 -- 172.21.15.16:0/778964100 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fc4340c1070 con 0x7fc4440b6460 2024-03-31T15:14:51.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:51.349+0000 7fc43be00640 1 --2- 172.21.15.16:0/778964100 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fc4200683b0 0x7fc42006a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:51.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:51.349+0000 7fc43be00640 1 -- 172.21.15.16:0/778964100 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(112..112 src has 1..112) ==== 5853+0+0 (secure 0 0 0) 0x7fc4341203e0 con 0x7fc4440b6460 2024-03-31T15:14:51.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:51.349+0000 7fc43aa00640 1 --2- 172.21.15.16:0/778964100 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fc4200683b0 0x7fc42006a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:51.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:51.349+0000 7fc43aa00640 1 --2- 172.21.15.16:0/778964100 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fc4200683b0 0x7fc42006a870 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7fc42c007fd0 tx=0x7fc42c009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:51.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:51.353+0000 7fc43be00640 1 -- 172.21.15.16:0/778964100 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fc4340ac020 con 0x7fc4440b6460 2024-03-31T15:14:51.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:51.533+0000 7fc44a600640 1 -- 172.21.15.16:0/778964100 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fc4440b7640 con 0x7fc4440b6460 2024-03-31T15:14:51.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:51.533+0000 7fc43be00640 1 -- 172.21.15.16:0/778964100 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 2 entries v112) ==== 80+0+122 (secure 0 0 0) 0x7fc4340167c0 con 0x7fc4440b6460 2024-03-31T15:14:51.527 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:6835/2564952016 2024-04-01T15:14:39.715615+0000 2024-03-31T15:14:51.527 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:6833/2564952016 2024-04-01T15:14:39.715615+0000 2024-03-31T15:14:51.527 INFO:teuthology.orchestra.run.smithi016.stderr:listed 2 entries 2024-03-31T15:14:51.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:51.533+0000 7fc44a600640 1 -- 172.21.15.16:0/778964100 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fc4200683b0 msgr2=0x7fc42006a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:51.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:51.533+0000 7fc44a600640 1 --2- 172.21.15.16:0/778964100 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fc4200683b0 0x7fc42006a870 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7fc42c007fd0 tx=0x7fc42c009040 comp rx=0 tx=0).stop 2024-03-31T15:14:51.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:51.533+0000 7fc44a600640 1 -- 172.21.15.16:0/778964100 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc4440b6460 msgr2=0x7fc444159130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:51.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:51.533+0000 7fc44a600640 1 --2- 172.21.15.16:0/778964100 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc4440b6460 0x7fc444159130 secure :-1 s=READY pgs=1780 cs=0 l=1 rev1=1 crypto rx=0x7fc43401c8e0 tx=0x7fc428005840 comp rx=0 tx=0).stop 2024-03-31T15:14:51.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:51.533+0000 7fc44a600640 1 -- 172.21.15.16:0/778964100 shutdown_connections 2024-03-31T15:14:51.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:51.533+0000 7fc44a600640 1 --2- 172.21.15.16:0/778964100 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fc4200683b0 0x7fc42006a870 unknown :-1 s=CLOSED pgs=754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:51.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:51.533+0000 7fc44a600640 1 --2- 172.21.15.16:0/778964100 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fc4440c9aa0 0x7fc44415b150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:51.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:51.533+0000 7fc44a600640 1 --2- 172.21.15.16:0/778964100 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc4441f7ca0 0x7fc4440c9560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:51.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:51.533+0000 7fc44a600640 1 --2- 172.21.15.16:0/778964100 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc4440b6460 0x7fc444159130 unknown :-1 s=CLOSED pgs=1780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:51.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:51.533+0000 7fc44a600640 1 -- 172.21.15.16:0/778964100 >> 172.21.15.16:0/778964100 conn(0x7fc444067730 msgr2=0x7fc4440d1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:14:51.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:51.533+0000 7fc44a600640 1 -- 172.21.15.16:0/778964100 shutdown_connections 2024-03-31T15:14:51.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:51.533+0000 7fc44a600640 1 -- 172.21.15.16:0/778964100 wait complete. 2024-03-31T15:14:51.549 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi016.front.sepia.ceph.com... 2024-03-31T15:14:51.549 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:14:51.549 DEBUG:teuthology.orchestra.run.smithi016:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-03-31T15:14:51.567 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi016.stderr:ceph-fuse[52262]: fuse finished with error 0 and tester_r 0 2024-03-31T15:14:51.569 INFO:teuthology.orchestra.run:waiting for 300 2024-03-31T15:14:51.640 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi016.front.sepia.ceph.com 2024-03-31T15:14:51.640 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:14:51.640 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-31T15:14:51.649 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-31T15:14:51.835 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:51.841+0000 7f5c8aa00640 1 Processor -- start 2024-03-31T15:14:51.835 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:51.841+0000 7f5c8aa00640 1 -- start start 2024-03-31T15:14:51.836 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:51.841+0000 7f5c8aa00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f5c840af9b0 0x7f5c840d0290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:51.836 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:51.841+0000 7f5c8aa00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f5c840725a0 0x7f5c840ccd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:51.836 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:51.841+0000 7f5c8aa00640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5c84072b50 con 0x7f5c840d07d0 2024-03-31T15:14:51.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:51.841+0000 7f5c8aa00640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f5c8406e6b0 con 0x7f5c840af9b0 2024-03-31T15:14:51.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:51.841+0000 7f5c8aa00640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f5c8415cd00 con 0x7f5c840725a0 2024-03-31T15:14:51.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:51.841+0000 7f5c83e00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f5c840af9b0 0x7f5c840d0290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:51.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:51.841+0000 7f5c83e00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f5c840af9b0 0x7f5c840d0290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55384/0 (socket says 172.21.15.16:55384) 2024-03-31T15:14:51.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:51.841+0000 7f5c83e00640 1 -- 172.21.15.16:0/1391582340 learned_addr learned my addr 172.21.15.16:0/1391582340 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:14:51.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:51.845+0000 7f5c88a00640 1 --2- 172.21.15.16:0/1391582340 >> v2:172.21.15.89:3300/0 conn(0x7f5c840725a0 0x7f5c840ccd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:51.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:51.845+0000 7f5c83e00640 1 -- 172.21.15.16:0/1391582340 >> v2:172.21.15.89:3300/0 conn(0x7f5c840725a0 msgr2=0x7f5c840ccd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:14:51.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:51.845+0000 7f5c83e00640 1 --2- 172.21.15.16:0/1391582340 >> v2:172.21.15.89:3300/0 conn(0x7f5c840725a0 0x7f5c840ccd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:51.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:51.845+0000 7f5c83e00640 1 -- 172.21.15.16:0/1391582340 >> v1:172.21.15.89:6789/0 conn(0x7f5c840d07d0 legacy=0x7f5c84071e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:14:51.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:51.845+0000 7f5c83e00640 1 -- 172.21.15.16:0/1391582340 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c8415cfe0 con 0x7f5c840af9b0 2024-03-31T15:14:51.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:51.845+0000 7f5c88a00640 1 --2- 172.21.15.16:0/1391582340 >> v2:172.21.15.89:3300/0 conn(0x7f5c840725a0 0x7f5c840ccd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:14:51.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:51.845+0000 7f5c83e00640 1 --2- 172.21.15.16:0/1391582340 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5c840af9b0 0x7f5c840d0290 secure :-1 s=READY pgs=1781 cs=0 l=1 rev1=1 crypto rx=0x7f5c84072e60 tx=0x7f5c7401c4e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d862096637299993 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:51.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:51.845+0000 7f5c83400640 1 -- 172.21.15.16:0/1391582340 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c740b0070 con 0x7f5c840af9b0 2024-03-31T15:14:51.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:51.845+0000 7f5c83400640 1 -- 172.21.15.16:0/1391582340 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5c740ab600 con 0x7f5c840af9b0 2024-03-31T15:14:51.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:51.845+0000 7f5c83400640 1 -- 172.21.15.16:0/1391582340 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c740ab9d0 con 0x7f5c840af9b0 2024-03-31T15:14:51.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:51.849+0000 7f5c8aa00640 1 -- 172.21.15.16:0/1391582340 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5c840af9b0 msgr2=0x7f5c840d0290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:51.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:51.849+0000 7f5c8aa00640 1 --2- 172.21.15.16:0/1391582340 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5c840af9b0 0x7f5c840d0290 secure :-1 s=READY pgs=1781 cs=0 l=1 rev1=1 crypto rx=0x7f5c84072e60 tx=0x7f5c7401c4e0 comp rx=0 tx=0).stop 2024-03-31T15:14:51.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:51.849+0000 7f5c8aa00640 1 -- 172.21.15.16:0/1391582340 shutdown_connections 2024-03-31T15:14:51.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:51.849+0000 7f5c8aa00640 1 --2- 172.21.15.16:0/1391582340 >> v2:172.21.15.89:3300/0 conn(0x7f5c840725a0 0x7f5c840ccd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:51.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:51.849+0000 7f5c8aa00640 1 --2- 172.21.15.16:0/1391582340 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5c840af9b0 0x7f5c840d0290 unknown :-1 s=CLOSED pgs=1781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:51.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:51.849+0000 7f5c8aa00640 1 -- 172.21.15.16:0/1391582340 >> 172.21.15.16:0/1391582340 conn(0x7f5c8406b620 msgr2=0x7f5c8406ba30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:14:51.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:51.849+0000 7f5c8aa00640 1 -- 172.21.15.16:0/1391582340 shutdown_connections 2024-03-31T15:14:51.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:51.849+0000 7f5c8aa00640 1 -- 172.21.15.16:0/1391582340 wait complete. 2024-03-31T15:14:51.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:51.849+0000 7f5c8aa00640 1 Processor -- start 2024-03-31T15:14:51.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:51.849+0000 7f5c8aa00640 1 -- start start 2024-03-31T15:14:51.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:51.849+0000 7f5c8aa00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f5c840725a0 0x7f5c8414ee00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:51.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:51.849+0000 7f5c8aa00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5c840af9b0 0x7f5c8415fe70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:51.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:51.849+0000 7f5c8aa00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5c840d07d0 0x7f5c841f70a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:51.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:51.849+0000 7f5c8aa00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f5c8415d960 con 0x7f5c840af9b0 2024-03-31T15:14:51.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:51.849+0000 7f5c8aa00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f5c8415d7e0 con 0x7f5c840d07d0 2024-03-31T15:14:51.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:51.849+0000 7f5c8aa00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f5c84169110 con 0x7f5c840725a0 2024-03-31T15:14:51.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:51.849+0000 7f5c7be00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5c840af9b0 0x7f5c8415fe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:51.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:51.849+0000 7f5c83e00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f5c840725a0 0x7f5c8414ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:51.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:51.849+0000 7f5c7be00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5c840af9b0 0x7f5c8415fe70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55398/0 (socket says 172.21.15.16:55398) 2024-03-31T15:14:51.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:51.849+0000 7f5c7be00640 1 -- 172.21.15.16:0/1762150038 learned_addr learned my addr 172.21.15.16:0/1762150038 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:14:51.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:51.849+0000 7f5c88a00640 1 --2- 172.21.15.16:0/1762150038 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5c840d07d0 0x7f5c841f70a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:51.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:51.849+0000 7f5c7be00640 1 -- 172.21.15.16:0/1762150038 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f5c840725a0 msgr2=0x7f5c8414ee00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:51.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:51.849+0000 7f5c7be00640 1 --2- 172.21.15.16:0/1762150038 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f5c840725a0 0x7f5c8414ee00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:51.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:51.849+0000 7f5c7be00640 1 -- 172.21.15.16:0/1762150038 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5c840d07d0 msgr2=0x7f5c841f70a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:51.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:51.849+0000 7f5c7be00640 1 --2- 172.21.15.16:0/1762150038 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5c840d07d0 0x7f5c841f70a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:51.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:51.853+0000 7f5c7be00640 1 -- 172.21.15.16:0/1762150038 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c84153db0 con 0x7f5c840af9b0 2024-03-31T15:14:51.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:51.853+0000 7f5c7be00640 1 --2- 172.21.15.16:0/1762150038 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5c840af9b0 0x7f5c8415fe70 secure :-1 s=READY pgs=1782 cs=0 l=1 rev1=1 crypto rx=0x7f5c6c00b410 tx=0x7f5c6c00b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:51.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:51.853+0000 7f5c80c00640 1 -- 172.21.15.16:0/1762150038 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c6c015030 con 0x7f5c840af9b0 2024-03-31T15:14:51.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:51.853+0000 7f5c80c00640 1 -- 172.21.15.16:0/1762150038 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5c6c004020 con 0x7f5c840af9b0 2024-03-31T15:14:51.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:51.853+0000 7f5c80c00640 1 -- 172.21.15.16:0/1762150038 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c6c004450 con 0x7f5c840af9b0 2024-03-31T15:14:51.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:51.853+0000 7f5c8aa00640 1 -- 172.21.15.16:0/1762150038 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5c84154090 con 0x7f5c840af9b0 2024-03-31T15:14:51.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:51.853+0000 7f5c8aa00640 1 -- 172.21.15.16:0/1762150038 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5c84154650 con 0x7f5c840af9b0 2024-03-31T15:14:51.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:51.853+0000 7f5c8aa00640 1 -- 172.21.15.16:0/1762150038 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5c84069950 con 0x7f5c840af9b0 2024-03-31T15:14:51.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:51.853+0000 7f5c80c00640 1 -- 172.21.15.16:0/1762150038 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f5c6c018070 con 0x7f5c840af9b0 2024-03-31T15:14:51.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:51.853+0000 7f5c80c00640 1 --2- 172.21.15.16:0/1762150038 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f5c60068380 0x7f5c6006a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:51.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:51.853+0000 7f5c80c00640 1 -- 172.21.15.16:0/1762150038 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(112..112 src has 1..112) ==== 5853+0+0 (secure 0 0 0) 0x7f5c6c086550 con 0x7f5c840af9b0 2024-03-31T15:14:51.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:51.853+0000 7f5c83e00640 1 --2- 172.21.15.16:0/1762150038 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f5c60068380 0x7f5c6006a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:51.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:51.857+0000 7f5c83e00640 1 --2- 172.21.15.16:0/1762150038 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f5c60068380 0x7f5c6006a840 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7f5c740ad000 tx=0x7f5c74008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:51.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:51.865+0000 7f5c80c00640 1 -- 172.21.15.16:0/1762150038 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f5c6c056390 con 0x7f5c840af9b0 2024-03-31T15:14:52.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.061+0000 7f5c8aa00640 1 -- 172.21.15.16:0/1762150038 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5c8406c980 con 0x7f5c840af9b0 2024-03-31T15:14:52.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.061+0000 7f5c80c00640 1 -- 172.21.15.16:0/1762150038 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 105 v105) ==== 95+0+5185 (secure 0 0 0) 0x7f5c6c05a690 con 0x7f5c840af9b0 2024-03-31T15:14:52.055 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:14:52.055 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":105,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7631,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6836/1628897112","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":1628897112},{"type":"v1","addr":"172.21.15.89:6836","nonce":1628897112}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":94},{"gid":8746,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/1036724837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6835","nonce":1036724837},{"type":"v1","addr":"172.21.15.89:6837","nonce":1036724837}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":94},{"gid":9192,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6834/472139147","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":472139147},{"type":"v1","addr":"172.21.15.16:6834","nonce":472139147}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":94}],"filesystems":[{"mdsmap":{"epoch":105,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-31T15:14:08.652180+0000","modified":"2024-03-31T15:14:43.727451+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":112,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":9558},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9558":{"gid":9558,"name":"c","rank":0,"incarnation":102,"state":"up:active","state_seq":4,"addr":"172.21.15.16:6835/4142860904","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6833","nonce":4142860904},{"type":"v1","addr":"172.21.15.16:6835","nonce":4142860904}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9558,"qdb_cluster":[9558]},"id":8}]} 2024-03-31T15:14:52.055 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 105 2024-03-31T15:14:52.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.061+0000 7f5c8aa00640 1 -- 172.21.15.16:0/1762150038 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f5c60068380 msgr2=0x7f5c6006a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:52.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.061+0000 7f5c8aa00640 1 --2- 172.21.15.16:0/1762150038 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f5c60068380 0x7f5c6006a840 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7f5c740ad000 tx=0x7f5c74008040 comp rx=0 tx=0).stop 2024-03-31T15:14:52.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.061+0000 7f5c8aa00640 1 -- 172.21.15.16:0/1762150038 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5c840af9b0 msgr2=0x7f5c8415fe70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:52.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.061+0000 7f5c8aa00640 1 --2- 172.21.15.16:0/1762150038 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5c840af9b0 0x7f5c8415fe70 secure :-1 s=READY pgs=1782 cs=0 l=1 rev1=1 crypto rx=0x7f5c6c00b410 tx=0x7f5c6c00b8d0 comp rx=0 tx=0).stop 2024-03-31T15:14:52.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.061+0000 7f5c8aa00640 1 -- 172.21.15.16:0/1762150038 shutdown_connections 2024-03-31T15:14:52.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.061+0000 7f5c8aa00640 1 --2- 172.21.15.16:0/1762150038 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f5c60068380 0x7f5c6006a840 unknown :-1 s=CLOSED pgs=755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:52.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.061+0000 7f5c8aa00640 1 --2- 172.21.15.16:0/1762150038 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5c840d07d0 0x7f5c841f70a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:52.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.061+0000 7f5c8aa00640 1 --2- 172.21.15.16:0/1762150038 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5c840af9b0 0x7f5c8415fe70 unknown :-1 s=CLOSED pgs=1782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:52.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.061+0000 7f5c8aa00640 1 --2- 172.21.15.16:0/1762150038 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f5c840725a0 0x7f5c8414ee00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:52.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.061+0000 7f5c8aa00640 1 -- 172.21.15.16:0/1762150038 >> 172.21.15.16:0/1762150038 conn(0x7f5c8406b620 msgr2=0x7f5c8406e830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:14:52.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.061+0000 7f5c8aa00640 1 -- 172.21.15.16:0/1762150038 shutdown_connections 2024-03-31T15:14:52.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.061+0000 7f5c8aa00640 1 -- 172.21.15.16:0/1762150038 wait complete. 2024-03-31T15:14:52.077 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json tag path /parent mytag123 2024-03-31T15:14:52.260 INFO:teuthology.orchestra.run.smithi016.stdout:{"return_code":0,"scrub_tag":"mytag123","mode":"asynchronous"} 2024-03-31T15:14:52.260 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mode": "asynchronous", "return_code": 0, "scrub_tag": "mytag123" } 2024-03-31T15:14:52.260 DEBUG:teuthology.orchestra.run.smithi016:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_data getxattr 100000001f7.00000000 scrub_tag 2024-03-31T15:14:52.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.313+0000 7f9d95843080 1 Processor -- start 2024-03-31T15:14:52.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.313+0000 7f9d95843080 1 -- start start 2024-03-31T15:14:52.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.317+0000 7f9d95843080 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9d78002490 0x55a028895f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:52.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.317+0000 7f9d95843080 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a0287f0ec0 0x55a02887d600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:52.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.317+0000 7f9d95843080 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55a0287f12a0 0x55a02887db40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:52.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.317+0000 7f9d95843080 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55a0287674b0 con 0x55a0287f0ec0 2024-03-31T15:14:52.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.317+0000 7f9d95843080 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55a02876b500 con 0x7f9d78002490 2024-03-31T15:14:52.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.317+0000 7f9d95843080 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x55a0287c3740 con 0x55a0287f12a0 2024-03-31T15:14:52.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.317+0000 7f9d92000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a0287f0ec0 0x55a02887d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:52.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.317+0000 7f9d92000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a0287f0ec0 0x55a02887d600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55416/0 (socket says 172.21.15.16:55416) 2024-03-31T15:14:52.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.317+0000 7f9d92000640 1 -- 172.21.15.16:0/1144132178 learned_addr learned my addr 172.21.15.16:0/1144132178 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:14:52.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.317+0000 7f9d92a00640 1 --2- 172.21.15.16:0/1144132178 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9d78002490 0x55a028895f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:52.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.317+0000 7f9d93400640 1 --2- 172.21.15.16:0/1144132178 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55a0287f12a0 0x55a02887db40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:52.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.317+0000 7f9d92000640 1 -- 172.21.15.16:0/1144132178 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55a0287f12a0 msgr2=0x55a02887db40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:14:52.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.317+0000 7f9d92000640 1 --2- 172.21.15.16:0/1144132178 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55a0287f12a0 0x55a02887db40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:52.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.317+0000 7f9d92000640 1 -- 172.21.15.16:0/1144132178 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9d78002490 msgr2=0x55a028895f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:14:52.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.317+0000 7f9d92000640 1 --2- 172.21.15.16:0/1144132178 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9d78002490 0x55a028895f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:52.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.317+0000 7f9d92000640 1 -- 172.21.15.16:0/1144132178 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a0287c1270 con 0x55a0287f0ec0 2024-03-31T15:14:52.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.317+0000 7f9d92000640 1 --2- 172.21.15.16:0/1144132178 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a0287f0ec0 0x55a02887d600 secure :-1 s=READY pgs=1784 cs=0 l=1 rev1=1 crypto rx=0x7f9d80010580 tx=0x7f9d80010a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55e4cca1e8db6b88 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:52.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.317+0000 7f9d90c00640 1 -- 172.21.15.16:0/1144132178 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d8000e940 con 0x55a0287f0ec0 2024-03-31T15:14:52.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.317+0000 7f9d90c00640 1 -- 172.21.15.16:0/1144132178 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9d80005030 con 0x55a0287f0ec0 2024-03-31T15:14:52.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.317+0000 7f9d90c00640 1 -- 172.21.15.16:0/1144132178 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d8000ed40 con 0x55a0287f0ec0 2024-03-31T15:14:52.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.317+0000 7f9d95843080 1 -- 172.21.15.16:0/1144132178 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a0287f0ec0 msgr2=0x55a02887d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:52.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.317+0000 7f9d95843080 1 --2- 172.21.15.16:0/1144132178 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a0287f0ec0 0x55a02887d600 secure :-1 s=READY pgs=1784 cs=0 l=1 rev1=1 crypto rx=0x7f9d80010580 tx=0x7f9d80010a40 comp rx=0 tx=0).stop 2024-03-31T15:14:52.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.317+0000 7f9d95843080 1 -- 172.21.15.16:0/1144132178 shutdown_connections 2024-03-31T15:14:52.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.317+0000 7f9d95843080 1 --2- 172.21.15.16:0/1144132178 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55a0287f12a0 0x55a02887db40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:52.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.317+0000 7f9d95843080 1 --2- 172.21.15.16:0/1144132178 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a0287f0ec0 0x55a02887d600 unknown :-1 s=CLOSED pgs=1784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:52.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.317+0000 7f9d95843080 1 --2- 172.21.15.16:0/1144132178 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9d78002490 0x55a028895f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:52.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.317+0000 7f9d95843080 1 -- 172.21.15.16:0/1144132178 >> 172.21.15.16:0/1144132178 conn(0x55a02880a0b0 msgr2=0x55a0288982a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:14:52.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.317+0000 7f9d95843080 1 -- 172.21.15.16:0/1144132178 shutdown_connections 2024-03-31T15:14:52.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.317+0000 7f9d95843080 1 -- 172.21.15.16:0/1144132178 wait complete. 2024-03-31T15:14:52.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.317+0000 7f9d95843080 1 Processor -- start 2024-03-31T15:14:52.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.317+0000 7f9d95843080 1 -- start start 2024-03-31T15:14:52.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.321+0000 7f9d95843080 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a0287f0ec0 0x55a0288951a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:52.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.321+0000 7f9d95843080 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a0287f12a0 0x55a0287ec110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:52.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.321+0000 7f9d95843080 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f9d78002490 0x55a028807920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:52.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.321+0000 7f9d95843080 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55a0287c17b0 con 0x55a0287f12a0 2024-03-31T15:14:52.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.321+0000 7f9d95843080 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55a0287c1510 con 0x55a0287f0ec0 2024-03-31T15:14:52.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.321+0000 7f9d95843080 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x55a0287c1a50 con 0x7f9d78002490 2024-03-31T15:14:52.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.321+0000 7f9d92000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a0287f12a0 0x55a0287ec110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:52.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.321+0000 7f9d92000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a0287f12a0 0x55a0287ec110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55420/0 (socket says 172.21.15.16:55420) 2024-03-31T15:14:52.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.321+0000 7f9d92000640 1 -- 172.21.15.16:0/3525940057 learned_addr learned my addr 172.21.15.16:0/3525940057 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:14:52.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.321+0000 7f9d92a00640 1 --2- 172.21.15.16:0/3525940057 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a0287f0ec0 0x55a0288951a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:52.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.321+0000 7f9d93400640 1 --2- 172.21.15.16:0/3525940057 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f9d78002490 0x55a028807920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:52.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.321+0000 7f9d92000640 1 -- 172.21.15.16:0/3525940057 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f9d78002490 msgr2=0x55a028807920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:52.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.321+0000 7f9d92000640 1 --2- 172.21.15.16:0/3525940057 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f9d78002490 0x55a028807920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:52.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.321+0000 7f9d92000640 1 -- 172.21.15.16:0/3525940057 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a0287f0ec0 msgr2=0x55a0288951a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:52.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.321+0000 7f9d92000640 1 --2- 172.21.15.16:0/3525940057 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a0287f0ec0 0x55a0288951a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:52.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.321+0000 7f9d92000640 1 -- 172.21.15.16:0/3525940057 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a0287e6e60 con 0x55a0287f12a0 2024-03-31T15:14:52.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.321+0000 7f9d92a00640 1 --2- 172.21.15.16:0/3525940057 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a0287f0ec0 0x55a0288951a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:14:52.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.321+0000 7f9d92000640 1 --2- 172.21.15.16:0/3525940057 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a0287f12a0 0x55a0287ec110 secure :-1 s=READY pgs=1785 cs=0 l=1 rev1=1 crypto rx=0x7f9d80001c10 tx=0x7f9d80001980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:52.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.321+0000 7f9d8b400640 1 -- 172.21.15.16:0/3525940057 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d8001aaa0 con 0x55a0287f12a0 2024-03-31T15:14:52.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.321+0000 7f9d8b400640 1 -- 172.21.15.16:0/3525940057 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9d8001ac60 con 0x55a0287f12a0 2024-03-31T15:14:52.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.321+0000 7f9d95843080 1 -- 172.21.15.16:0/3525940057 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55a0287e7100 con 0x55a0287f12a0 2024-03-31T15:14:52.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.321+0000 7f9d8b400640 1 -- 172.21.15.16:0/3525940057 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d80028660 con 0x55a0287f12a0 2024-03-31T15:14:52.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.321+0000 7f9d95843080 1 -- 172.21.15.16:0/3525940057 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a0287e7640 con 0x55a0287f12a0 2024-03-31T15:14:52.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.321+0000 7f9d8b400640 1 -- 172.21.15.16:0/3525940057 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f9d80025070 con 0x55a0287f12a0 2024-03-31T15:14:52.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.325+0000 7f9d8b400640 1 --2- 172.21.15.16:0/3525940057 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f9d74066d90 0x7f9d74069250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:52.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.325+0000 7f9d8b400640 1 -- 172.21.15.16:0/3525940057 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(112..112 src has 1..112) ==== 5853+0+0 (secure 0 0 0) 0x7f9d8008ad00 con 0x55a0287f12a0 2024-03-31T15:14:52.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.325+0000 7f9d95843080 1 --2- 172.21.15.16:0/3525940057 >> [v2:172.21.15.16:6800/932712244,v1:172.21.15.16:6801/932712244] conn(0x55a0287e7d80 0x55a0287e8160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:52.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.325+0000 7f9d95843080 1 -- 172.21.15.16:0/3525940057 --> [v2:172.21.15.16:6800/932712244,v1:172.21.15.16:6801/932712244] -- osd_op(unknown.0.0:1 17.36 17:6fe38dc2:::100000001f7.00000000:head [getxattr scrub_tag in=9b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e112) -- 0x55a0287e86a0 con 0x55a0287e7d80 2024-03-31T15:14:52.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.325+0000 7f9d92a00640 1 --2- 172.21.15.16:0/3525940057 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f9d74066d90 0x7f9d74069250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:52.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.325+0000 7f9d93400640 1 --2- 172.21.15.16:0/3525940057 >> [v2:172.21.15.16:6800/932712244,v1:172.21.15.16:6801/932712244] conn(0x55a0287e7d80 0x55a0287e8160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:52.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.325+0000 7f9d92a00640 1 --2- 172.21.15.16:0/3525940057 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f9d74066d90 0x7f9d74069250 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f9d840ac000 tx=0x7f9d84003030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:52.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.325+0000 7f9d93400640 1 --2- 172.21.15.16:0/3525940057 >> [v2:172.21.15.16:6800/932712244,v1:172.21.15.16:6801/932712244] conn(0x55a0287e7d80 0x55a0287e8160 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:52.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.325+0000 7f9d93400640 1 -- 172.21.15.16:0/3525940057 <== osd.0 v2:172.21.15.16:6800/932712244 1 ==== osd_op_reply(1 100000001f7.00000000 [getxattr] v0'0 uv0 ondisk = -61 ((61) No data available)) ==== 164+0+0 (crc 0 0 0) 0x7f9d8c0067a0 con 0x55a0287e7d80 2024-03-31T15:14:52.321 INFO:teuthology.orchestra.run.smithi016.stderr:error getting xattr cephfs_data/100000001f7.00000000/scrub_tag: (61) No data available 2024-03-31T15:14:52.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.325+0000 7f9d95843080 1 -- 172.21.15.16:0/3525940057 >> [v2:172.21.15.16:6800/932712244,v1:172.21.15.16:6801/932712244] conn(0x55a0287e7d80 msgr2=0x55a0287e8160 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:52.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.325+0000 7f9d95843080 1 --2- 172.21.15.16:0/3525940057 >> [v2:172.21.15.16:6800/932712244,v1:172.21.15.16:6801/932712244] conn(0x55a0287e7d80 0x55a0287e8160 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:52.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.325+0000 7f9d95843080 1 -- 172.21.15.16:0/3525940057 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f9d74066d90 msgr2=0x7f9d74069250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:52.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.325+0000 7f9d95843080 1 --2- 172.21.15.16:0/3525940057 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f9d74066d90 0x7f9d74069250 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f9d840ac000 tx=0x7f9d84003030 comp rx=0 tx=0).stop 2024-03-31T15:14:52.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.325+0000 7f9d95843080 1 -- 172.21.15.16:0/3525940057 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a0287f12a0 msgr2=0x55a0287ec110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:52.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.325+0000 7f9d95843080 1 --2- 172.21.15.16:0/3525940057 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a0287f12a0 0x55a0287ec110 secure :-1 s=READY pgs=1785 cs=0 l=1 rev1=1 crypto rx=0x7f9d80001c10 tx=0x7f9d80001980 comp rx=0 tx=0).stop 2024-03-31T15:14:52.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.325+0000 7f9d93400640 1 -- 172.21.15.16:0/3525940057 reap_dead start 2024-03-31T15:14:52.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.325+0000 7f9d95843080 1 -- 172.21.15.16:0/3525940057 shutdown_connections 2024-03-31T15:14:52.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.325+0000 7f9d95843080 1 -- 172.21.15.16:0/3525940057 >> 172.21.15.16:0/3525940057 conn(0x55a02880a0b0 msgr2=0x55a0288982a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:14:52.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.325+0000 7f9d95843080 1 -- 172.21.15.16:0/3525940057 shutdown_connections 2024-03-31T15:14:52.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.325+0000 7f9d95843080 1 -- 172.21.15.16:0/3525940057 wait complete. 2024-03-31T15:14:52.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-31T15:14:52.324 DEBUG:teuthology.orchestra.run.smithi016:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_data getxattr 10000000002.00000000 scrub_tag 2024-03-31T15:14:52.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.377+0000 7fe582fc9080 1 Processor -- start 2024-03-31T15:14:52.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.377+0000 7fe582fc9080 1 -- start start 2024-03-31T15:14:52.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.381+0000 7fe582fc9080 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cb19871260 0x55cb1982b1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:52.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.381+0000 7fe582fc9080 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55cb198762e0 0x55cb19905700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:52.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.381+0000 7fe582fc9080 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cb19902ab0 0x55cb1985af10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:52.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.381+0000 7fe582fc9080 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55cb1985d2b0 con 0x55cb19902ab0 2024-03-31T15:14:52.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.381+0000 7fe582fc9080 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55cb197d14b0 con 0x55cb19871260 2024-03-31T15:14:52.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.381+0000 7fe582fc9080 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x55cb1985d430 con 0x55cb198762e0 2024-03-31T15:14:52.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.381+0000 7fe580c00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cb19902ab0 0x55cb1985af10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:52.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.381+0000 7fe580c00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cb19902ab0 0x55cb1985af10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55432/0 (socket says 172.21.15.16:55432) 2024-03-31T15:14:52.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.381+0000 7fe580c00640 1 -- 172.21.15.16:0/2253677110 learned_addr learned my addr 172.21.15.16:0/2253677110 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:14:52.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.381+0000 7fe580200640 1 --2- 172.21.15.16:0/2253677110 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cb19871260 0x55cb1982b1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:52.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.381+0000 7fe57f800640 1 --2- 172.21.15.16:0/2253677110 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55cb198762e0 0x55cb19905700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:52.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.381+0000 7fe580c00640 1 -- 172.21.15.16:0/2253677110 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55cb198762e0 msgr2=0x55cb19905700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:14:52.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.381+0000 7fe580c00640 1 --2- 172.21.15.16:0/2253677110 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55cb198762e0 0x55cb19905700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:52.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.381+0000 7fe580c00640 1 -- 172.21.15.16:0/2253677110 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cb19871260 msgr2=0x55cb1982b1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:14:52.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.381+0000 7fe580c00640 1 --2- 172.21.15.16:0/2253677110 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cb19871260 0x55cb1982b1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:52.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.381+0000 7fe580c00640 1 -- 172.21.15.16:0/2253677110 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cb19904130 con 0x55cb19902ab0 2024-03-31T15:14:52.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.381+0000 7fe580200640 1 --2- 172.21.15.16:0/2253677110 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cb19871260 0x55cb1982b1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:14:52.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.381+0000 7fe580c00640 1 --2- 172.21.15.16:0/2253677110 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cb19902ab0 0x55cb1985af10 secure :-1 s=READY pgs=1788 cs=0 l=1 rev1=1 crypto rx=0x7fe57800c4d0 tx=0x7fe57800c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d4ef0e26e15b772e server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:52.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.381+0000 7fe57e400640 1 -- 172.21.15.16:0/2253677110 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5780116b0 con 0x55cb19902ab0 2024-03-31T15:14:52.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.381+0000 7fe57e400640 1 -- 172.21.15.16:0/2253677110 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe578017070 con 0x55cb19902ab0 2024-03-31T15:14:52.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.381+0000 7fe57e400640 1 -- 172.21.15.16:0/2253677110 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe578011ab0 con 0x55cb19902ab0 2024-03-31T15:14:52.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.381+0000 7fe582fc9080 1 -- 172.21.15.16:0/2253677110 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cb19902ab0 msgr2=0x55cb1985af10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:52.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.381+0000 7fe582fc9080 1 --2- 172.21.15.16:0/2253677110 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cb19902ab0 0x55cb1985af10 secure :-1 s=READY pgs=1788 cs=0 l=1 rev1=1 crypto rx=0x7fe57800c4d0 tx=0x7fe57800c990 comp rx=0 tx=0).stop 2024-03-31T15:14:52.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.381+0000 7fe582fc9080 1 -- 172.21.15.16:0/2253677110 shutdown_connections 2024-03-31T15:14:52.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.381+0000 7fe582fc9080 1 --2- 172.21.15.16:0/2253677110 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cb19902ab0 0x55cb1985af10 unknown :-1 s=CLOSED pgs=1788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:52.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.381+0000 7fe582fc9080 1 --2- 172.21.15.16:0/2253677110 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55cb198762e0 0x55cb19905700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:52.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.381+0000 7fe582fc9080 1 --2- 172.21.15.16:0/2253677110 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cb19871260 0x55cb1982b1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:52.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.381+0000 7fe582fc9080 1 -- 172.21.15.16:0/2253677110 >> 172.21.15.16:0/2253677110 conn(0x55cb198740b0 msgr2=0x55cb19874490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:14:52.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.381+0000 7fe582fc9080 1 -- 172.21.15.16:0/2253677110 shutdown_connections 2024-03-31T15:14:52.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.381+0000 7fe582fc9080 1 -- 172.21.15.16:0/2253677110 wait complete. 2024-03-31T15:14:52.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.385+0000 7fe582fc9080 1 Processor -- start 2024-03-31T15:14:52.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.385+0000 7fe582fc9080 1 -- start start 2024-03-31T15:14:52.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.385+0000 7fe582fc9080 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55cb19871260 0x55cb19905500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:52.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.385+0000 7fe582fc9080 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cb198762e0 0x55cb1999e490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:52.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.385+0000 7fe582fc9080 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cb19902ab0 0x55cb198fd4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:52.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.385+0000 7fe582fc9080 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55cb199043d0 con 0x55cb198762e0 2024-03-31T15:14:52.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.385+0000 7fe582fc9080 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55cb19904670 con 0x55cb19902ab0 2024-03-31T15:14:52.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.385+0000 7fe582fc9080 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x55cb19873a60 con 0x55cb19871260 2024-03-31T15:14:52.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.385+0000 7fe57f800640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cb198762e0 0x55cb1999e490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:52.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.385+0000 7fe57f800640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cb198762e0 0x55cb1999e490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55438/0 (socket says 172.21.15.16:55438) 2024-03-31T15:14:52.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.385+0000 7fe57f800640 1 -- 172.21.15.16:0/3250136606 learned_addr learned my addr 172.21.15.16:0/3250136606 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:14:52.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.385+0000 7fe580200640 1 --2- 172.21.15.16:0/3250136606 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55cb19871260 0x55cb19905500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:52.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.385+0000 7fe580c00640 1 --2- 172.21.15.16:0/3250136606 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cb19902ab0 0x55cb198fd4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:52.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.385+0000 7fe57f800640 1 -- 172.21.15.16:0/3250136606 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55cb19871260 msgr2=0x55cb19905500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:52.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.385+0000 7fe57f800640 1 --2- 172.21.15.16:0/3250136606 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55cb19871260 0x55cb19905500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:52.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.385+0000 7fe57f800640 1 -- 172.21.15.16:0/3250136606 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cb19902ab0 msgr2=0x55cb198fd4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:52.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.385+0000 7fe57f800640 1 --2- 172.21.15.16:0/3250136606 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cb19902ab0 0x55cb198fd4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:52.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.385+0000 7fe57f800640 1 -- 172.21.15.16:0/3250136606 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cb1982e4a0 con 0x55cb198762e0 2024-03-31T15:14:52.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.385+0000 7fe57f800640 1 --2- 172.21.15.16:0/3250136606 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cb198762e0 0x55cb1999e490 secure :-1 s=READY pgs=1789 cs=0 l=1 rev1=1 crypto rx=0x7fe56c0ac040 tx=0x7fe56c0181f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:52.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.385+0000 7fe57d000640 1 -- 172.21.15.16:0/3250136606 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe56c0c3070 con 0x55cb198762e0 2024-03-31T15:14:52.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.385+0000 7fe57d000640 1 -- 172.21.15.16:0/3250136606 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe56c0be3f0 con 0x55cb198762e0 2024-03-31T15:14:52.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.385+0000 7fe582fc9080 1 -- 172.21.15.16:0/3250136606 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55cb1982e740 con 0x55cb198762e0 2024-03-31T15:14:52.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.385+0000 7fe57d000640 1 -- 172.21.15.16:0/3250136606 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe56c0be7e0 con 0x55cb198762e0 2024-03-31T15:14:52.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.385+0000 7fe582fc9080 1 -- 172.21.15.16:0/3250136606 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cb1982ec80 con 0x55cb198762e0 2024-03-31T15:14:52.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.389+0000 7fe57d000640 1 -- 172.21.15.16:0/3250136606 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fe56c0c6070 con 0x55cb198762e0 2024-03-31T15:14:52.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.389+0000 7fe57d000640 1 --2- 172.21.15.16:0/3250136606 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fe560066d90 0x7fe560069250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:52.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.389+0000 7fe57d000640 1 -- 172.21.15.16:0/3250136606 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(112..112 src has 1..112) ==== 5853+0+0 (secure 0 0 0) 0x7fe56c120780 con 0x55cb198762e0 2024-03-31T15:14:52.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.389+0000 7fe582fc9080 1 --2- 172.21.15.16:0/3250136606 >> [v2:172.21.15.16:6809/3360352974,v1:172.21.15.16:6812/3360352974] conn(0x55cb1999b810 0x55cb1999bbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:52.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.389+0000 7fe582fc9080 1 -- 172.21.15.16:0/3250136606 --> [v2:172.21.15.16:6809/3360352974,v1:172.21.15.16:6812/3360352974] -- osd_op(unknown.0.0:1 17.12 17:4bec3886:::10000000002.00000000:head [getxattr scrub_tag in=9b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e112) -- 0x55cb1999c130 con 0x55cb1999b810 2024-03-31T15:14:52.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.389+0000 7fe580200640 1 --2- 172.21.15.16:0/3250136606 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fe560066d90 0x7fe560069250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:52.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.389+0000 7fe580c00640 1 --2- 172.21.15.16:0/3250136606 >> [v2:172.21.15.16:6809/3360352974,v1:172.21.15.16:6812/3360352974] conn(0x55cb1999b810 0x55cb1999bbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:52.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.389+0000 7fe580c00640 1 --2- 172.21.15.16:0/3250136606 >> [v2:172.21.15.16:6809/3360352974,v1:172.21.15.16:6812/3360352974] conn(0x55cb1999b810 0x55cb1999bbf0 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:52.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.389+0000 7fe580200640 1 --2- 172.21.15.16:0/3250136606 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fe560066d90 0x7fe560069250 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7fe570003f00 tx=0x7fe570003840 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:52.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.389+0000 7fe580c00640 1 -- 172.21.15.16:0/3250136606 <== osd.3 v2:172.21.15.16:6809/3360352974 1 ==== osd_op_reply(1 10000000002.00000000 [getxattr (12) out=12b] v0'0 uv22 ondisk = 0) ==== 164+0+12 (crc 0 0 0) 0x7fe578011930 con 0x55cb1999b810 2024-03-31T15:14:52.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.393+0000 7fe582fc9080 1 -- 172.21.15.16:0/3250136606 >> [v2:172.21.15.16:6809/3360352974,v1:172.21.15.16:6812/3360352974] conn(0x55cb1999b810 msgr2=0x55cb1999bbf0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:52.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.393+0000 7fe582fc9080 1 --2- 172.21.15.16:0/3250136606 >> [v2:172.21.15.16:6809/3360352974,v1:172.21.15.16:6812/3360352974] conn(0x55cb1999b810 0x55cb1999bbf0 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:52.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.393+0000 7fe582fc9080 1 -- 172.21.15.16:0/3250136606 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fe560066d90 msgr2=0x7fe560069250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:52.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.393+0000 7fe582fc9080 1 --2- 172.21.15.16:0/3250136606 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fe560066d90 0x7fe560069250 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7fe570003f00 tx=0x7fe570003840 comp rx=0 tx=0).stop 2024-03-31T15:14:52.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.393+0000 7fe582fc9080 1 -- 172.21.15.16:0/3250136606 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cb198762e0 msgr2=0x55cb1999e490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:52.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.393+0000 7fe582fc9080 1 --2- 172.21.15.16:0/3250136606 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cb198762e0 0x55cb1999e490 secure :-1 s=READY pgs=1789 cs=0 l=1 rev1=1 crypto rx=0x7fe56c0ac040 tx=0x7fe56c0181f0 comp rx=0 tx=0).stop 2024-03-31T15:14:52.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.393+0000 7fe580c00640 1 -- 172.21.15.16:0/3250136606 reap_dead start 2024-03-31T15:14:52.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.393+0000 7fe582fc9080 1 -- 172.21.15.16:0/3250136606 shutdown_connections 2024-03-31T15:14:52.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.393+0000 7fe582fc9080 1 -- 172.21.15.16:0/3250136606 >> 172.21.15.16:0/3250136606 conn(0x55cb198740b0 msgr2=0x55cb19874490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:14:52.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.393+0000 7fe582fc9080 1 -- 172.21.15.16:0/3250136606 shutdown_connections 2024-03-31T15:14:52.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.393+0000 7fe582fc9080 1 -- 172.21.15.16:0/3250136606 wait complete. 2024-03-31T15:14:52.389 INFO:teuthology.orchestra.run.smithi016.stdout:mytag123 2024-03-31T15:14:52.390 DEBUG:teuthology.orchestra.run.smithi016:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_data getxattr 100000001f8.00000000 scrub_tag 2024-03-31T15:14:52.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.445+0000 7f6a4239a080 1 Processor -- start 2024-03-31T15:14:52.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.445+0000 7f6a4239a080 1 -- start start 2024-03-31T15:14:52.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.445+0000 7f6a4239a080 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x562ded4f8e70 0x562ded4fa810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:52.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.445+0000 7f6a4239a080 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562ded516c60 0x562ded635a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:52.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.445+0000 7f6a4239a080 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562ded517040 0x562ded635f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:52.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.445+0000 7f6a4239a080 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x562ded5144b0 con 0x562ded517040 2024-03-31T15:14:52.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.445+0000 7f6a4239a080 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x562ded46f4b0 con 0x562ded516c60 2024-03-31T15:14:52.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.445+0000 7f6a4239a080 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x562ded4fbc80 con 0x562ded4f8e70 2024-03-31T15:14:52.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.445+0000 7f6a40000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562ded517040 0x562ded635f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:52.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.445+0000 7f6a40000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562ded517040 0x562ded635f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55448/0 (socket says 172.21.15.16:55448) 2024-03-31T15:14:52.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.445+0000 7f6a40000640 1 -- 172.21.15.16:0/3688005736 learned_addr learned my addr 172.21.15.16:0/3688005736 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:14:52.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.445+0000 7f6a3f600640 1 --2- 172.21.15.16:0/3688005736 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x562ded4f8e70 0x562ded4fa810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:52.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.445+0000 7f6a3ec00640 1 --2- 172.21.15.16:0/3688005736 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562ded516c60 0x562ded635a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:52.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.445+0000 7f6a40000640 1 -- 172.21.15.16:0/3688005736 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x562ded4f8e70 msgr2=0x562ded4fa810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:14:52.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.445+0000 7f6a40000640 1 --2- 172.21.15.16:0/3688005736 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x562ded4f8e70 0x562ded4fa810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:52.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.445+0000 7f6a40000640 1 -- 172.21.15.16:0/3688005736 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562ded516c60 msgr2=0x562ded635a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:14:52.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.445+0000 7f6a40000640 1 --2- 172.21.15.16:0/3688005736 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562ded516c60 0x562ded635a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:52.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.445+0000 7f6a40000640 1 -- 172.21.15.16:0/3688005736 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562ded59c130 con 0x562ded517040 2024-03-31T15:14:52.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.445+0000 7f6a3f600640 1 --2- 172.21.15.16:0/3688005736 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x562ded4f8e70 0x562ded4fa810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:14:52.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.445+0000 7f6a40000640 1 --2- 172.21.15.16:0/3688005736 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562ded517040 0x562ded635f80 secure :-1 s=READY pgs=1791 cs=0 l=1 rev1=1 crypto rx=0x7f6a380ae040 tx=0x7f6a380033b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c6890cd2053086d7 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:52.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.445+0000 7f6a3d800640 1 -- 172.21.15.16:0/3688005736 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a380b9070 con 0x562ded517040 2024-03-31T15:14:52.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.445+0000 7f6a3d800640 1 -- 172.21.15.16:0/3688005736 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6a380ab640 con 0x562ded517040 2024-03-31T15:14:52.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.445+0000 7f6a3d800640 1 -- 172.21.15.16:0/3688005736 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a380b0040 con 0x562ded517040 2024-03-31T15:14:52.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.445+0000 7f6a4239a080 1 -- 172.21.15.16:0/3688005736 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562ded517040 msgr2=0x562ded635f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:52.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.445+0000 7f6a4239a080 1 --2- 172.21.15.16:0/3688005736 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562ded517040 0x562ded635f80 secure :-1 s=READY pgs=1791 cs=0 l=1 rev1=1 crypto rx=0x7f6a380ae040 tx=0x7f6a380033b0 comp rx=0 tx=0).stop 2024-03-31T15:14:52.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.445+0000 7f6a4239a080 1 -- 172.21.15.16:0/3688005736 shutdown_connections 2024-03-31T15:14:52.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.445+0000 7f6a4239a080 1 --2- 172.21.15.16:0/3688005736 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562ded517040 0x562ded635f80 unknown :-1 s=CLOSED pgs=1791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:52.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.445+0000 7f6a4239a080 1 --2- 172.21.15.16:0/3688005736 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562ded516c60 0x562ded635a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:52.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.445+0000 7f6a4239a080 1 --2- 172.21.15.16:0/3688005736 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x562ded4f8e70 0x562ded4fa810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:52.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.445+0000 7f6a4239a080 1 -- 172.21.15.16:0/3688005736 >> 172.21.15.16:0/3688005736 conn(0x562ded4f7420 msgr2=0x562ded59f500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:14:52.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.445+0000 7f6a4239a080 1 -- 172.21.15.16:0/3688005736 shutdown_connections 2024-03-31T15:14:52.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.445+0000 7f6a4239a080 1 -- 172.21.15.16:0/3688005736 wait complete. 2024-03-31T15:14:52.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.449+0000 7f6a4239a080 1 Processor -- start 2024-03-31T15:14:52.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.449+0000 7f6a4239a080 1 -- start start 2024-03-31T15:14:52.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.449+0000 7f6a4239a080 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562ded4f8e70 0x562ded4c9050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:52.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.449+0000 7f6a4239a080 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562ded516c60 0x562ded595290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:52.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.449+0000 7f6a4239a080 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x562ded517040 0x562ded5957d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:52.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.449+0000 7f6a4239a080 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x562ded59c670 con 0x562ded4f8e70 2024-03-31T15:14:52.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.449+0000 7f6a4239a080 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x562ded59c3d0 con 0x562ded516c60 2024-03-31T15:14:52.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.449+0000 7f6a4239a080 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x562ded59c910 con 0x562ded517040 2024-03-31T15:14:52.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.449+0000 7f6a3f600640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562ded4f8e70 0x562ded4c9050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:52.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.449+0000 7f6a3f600640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562ded4f8e70 0x562ded4c9050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55454/0 (socket says 172.21.15.16:55454) 2024-03-31T15:14:52.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.449+0000 7f6a3f600640 1 -- 172.21.15.16:0/253775452 learned_addr learned my addr 172.21.15.16:0/253775452 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:14:52.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.449+0000 7f6a3ec00640 1 --2- 172.21.15.16:0/253775452 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562ded516c60 0x562ded595290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:52.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.449+0000 7f6a40000640 1 --2- 172.21.15.16:0/253775452 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x562ded517040 0x562ded5957d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:52.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.449+0000 7f6a3f600640 1 -- 172.21.15.16:0/253775452 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x562ded517040 msgr2=0x562ded5957d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:52.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.449+0000 7f6a3f600640 1 --2- 172.21.15.16:0/253775452 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x562ded517040 0x562ded5957d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:52.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.449+0000 7f6a3f600640 1 -- 172.21.15.16:0/253775452 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562ded516c60 msgr2=0x562ded595290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:52.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.449+0000 7f6a3f600640 1 --2- 172.21.15.16:0/253775452 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562ded516c60 0x562ded595290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:52.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.449+0000 7f6a3f600640 1 -- 172.21.15.16:0/253775452 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562ded595f60 con 0x562ded4f8e70 2024-03-31T15:14:52.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.449+0000 7f6a3f600640 1 --2- 172.21.15.16:0/253775452 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562ded4f8e70 0x562ded4c9050 secure :-1 s=READY pgs=1792 cs=0 l=1 rev1=1 crypto rx=0x7f6a30008780 tx=0x7f6a30008c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:52.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.449+0000 7f6a37e00640 1 -- 172.21.15.16:0/253775452 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a30006b80 con 0x562ded4f8e70 2024-03-31T15:14:52.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.449+0000 7f6a37e00640 1 -- 172.21.15.16:0/253775452 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6a30006d20 con 0x562ded4f8e70 2024-03-31T15:14:52.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.449+0000 7f6a37e00640 1 -- 172.21.15.16:0/253775452 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a30015660 con 0x562ded4f8e70 2024-03-31T15:14:52.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.449+0000 7f6a4239a080 1 -- 172.21.15.16:0/253775452 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x562ded596200 con 0x562ded4f8e70 2024-03-31T15:14:52.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.449+0000 7f6a4239a080 1 -- 172.21.15.16:0/253775452 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562ded596740 con 0x562ded4f8e70 2024-03-31T15:14:52.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.453+0000 7f6a37e00640 1 -- 172.21.15.16:0/253775452 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f6a3001c020 con 0x562ded4f8e70 2024-03-31T15:14:52.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.453+0000 7f6a37e00640 1 --2- 172.21.15.16:0/253775452 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f6a20066d90 0x7f6a20069250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:52.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.453+0000 7f6a37e00640 1 -- 172.21.15.16:0/253775452 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(112..112 src has 1..112) ==== 5853+0+0 (secure 0 0 0) 0x7f6a30085bc0 con 0x562ded4f8e70 2024-03-31T15:14:52.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.453+0000 7f6a3ec00640 1 --2- 172.21.15.16:0/253775452 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f6a20066d90 0x7f6a20069250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:52.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.453+0000 7f6a4239a080 1 --2- 172.21.15.16:0/253775452 >> [v2:172.21.15.89:6810/50395527,v1:172.21.15.89:6814/50395527] conn(0x562ded4c9590 0x562ded63d8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:52.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.453+0000 7f6a4239a080 1 -- 172.21.15.16:0/253775452 --> [v2:172.21.15.89:6810/50395527,v1:172.21.15.89:6814/50395527] -- osd_op(unknown.0.0:1 17.19 17:997b7129:::100000001f8.00000000:head [getxattr scrub_tag in=9b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e112) -- 0x562ded50dd20 con 0x562ded4c9590 2024-03-31T15:14:52.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.453+0000 7f6a40000640 1 --2- 172.21.15.16:0/253775452 >> [v2:172.21.15.89:6810/50395527,v1:172.21.15.89:6814/50395527] conn(0x562ded4c9590 0x562ded63d8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:52.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.453+0000 7f6a3ec00640 1 --2- 172.21.15.16:0/253775452 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f6a20066d90 0x7f6a20069250 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f6a2c004180 tx=0x7f6a2c000bd0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:52.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.453+0000 7f6a40000640 1 --2- 172.21.15.16:0/253775452 >> [v2:172.21.15.89:6810/50395527,v1:172.21.15.89:6814/50395527] conn(0x562ded4c9590 0x562ded63d8d0 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:52.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.453+0000 7f6a40000640 1 -- 172.21.15.16:0/253775452 <== osd.5 v2:172.21.15.89:6810/50395527 1 ==== osd_op_reply(1 100000001f8.00000000 [getxattr (12) out=12b] v0'0 uv23 ondisk = 0) ==== 164+0+12 (crc 0 0 0) 0x7f6a380bc040 con 0x562ded4c9590 2024-03-31T15:14:52.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.453+0000 7f6a4239a080 1 -- 172.21.15.16:0/253775452 >> [v2:172.21.15.89:6810/50395527,v1:172.21.15.89:6814/50395527] conn(0x562ded4c9590 msgr2=0x562ded63d8d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:52.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.453+0000 7f6a4239a080 1 --2- 172.21.15.16:0/253775452 >> [v2:172.21.15.89:6810/50395527,v1:172.21.15.89:6814/50395527] conn(0x562ded4c9590 0x562ded63d8d0 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:52.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.453+0000 7f6a4239a080 1 -- 172.21.15.16:0/253775452 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f6a20066d90 msgr2=0x7f6a20069250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:52.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.453+0000 7f6a4239a080 1 --2- 172.21.15.16:0/253775452 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f6a20066d90 0x7f6a20069250 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f6a2c004180 tx=0x7f6a2c000bd0 comp rx=0 tx=0).stop 2024-03-31T15:14:52.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.453+0000 7f6a4239a080 1 -- 172.21.15.16:0/253775452 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562ded4f8e70 msgr2=0x562ded4c9050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:52.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.453+0000 7f6a4239a080 1 --2- 172.21.15.16:0/253775452 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562ded4f8e70 0x562ded4c9050 secure :-1 s=READY pgs=1792 cs=0 l=1 rev1=1 crypto rx=0x7f6a30008780 tx=0x7f6a30008c40 comp rx=0 tx=0).stop 2024-03-31T15:14:52.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.453+0000 7f6a40000640 1 -- 172.21.15.16:0/253775452 reap_dead start 2024-03-31T15:14:52.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.453+0000 7f6a4239a080 1 -- 172.21.15.16:0/253775452 shutdown_connections 2024-03-31T15:14:52.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.453+0000 7f6a4239a080 1 -- 172.21.15.16:0/253775452 >> 172.21.15.16:0/253775452 conn(0x562ded4f7420 msgr2=0x562ded59f480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:14:52.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.453+0000 7f6a4239a080 1 -- 172.21.15.16:0/253775452 shutdown_connections 2024-03-31T15:14:52.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.453+0000 7f6a4239a080 1 -- 172.21.15.16:0/253775452 wait complete. 2024-03-31T15:14:52.455 INFO:teuthology.orchestra.run.smithi016.stdout:mytag123 2024-03-31T15:14:52.455 DEBUG:teuthology.orchestra.run.smithi016:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_data getxattr 10000000004.00000000 scrub_tag 2024-03-31T15:14:52.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.509+0000 7f2280db4080 1 Processor -- start 2024-03-31T15:14:52.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.509+0000 7f2280db4080 1 -- start start 2024-03-31T15:14:52.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.509+0000 7f2280db4080 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55dd495fc260 0x55dd4968fe70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:52.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.509+0000 7f2280db4080 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55dd496019f0 0x55dd495b61b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:52.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.509+0000 7f2280db4080 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dd4968dab0 0x55dd49600540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:52.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.509+0000 7f2280db4080 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55dd4955c4b0 con 0x55dd4968dab0 2024-03-31T15:14:52.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.509+0000 7f2280db4080 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55dd49560500 con 0x55dd496019f0 2024-03-31T15:14:52.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.509+0000 7f2280db4080 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x55dd49687880 con 0x55dd495fc260 2024-03-31T15:14:52.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.509+0000 7f227ea00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dd4968dab0 0x55dd49600540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:52.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.509+0000 7f227ea00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dd4968dab0 0x55dd49600540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55468/0 (socket says 172.21.15.16:55468) 2024-03-31T15:14:52.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.509+0000 7f227ea00640 1 -- 172.21.15.16:0/3449811852 learned_addr learned my addr 172.21.15.16:0/3449811852 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:14:52.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.509+0000 7f227e000640 1 --2- 172.21.15.16:0/3449811852 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55dd495fc260 0x55dd4968fe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:52.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.509+0000 7f227d600640 1 --2- 172.21.15.16:0/3449811852 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55dd496019f0 0x55dd495b61b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:52.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.509+0000 7f227ea00640 1 -- 172.21.15.16:0/3449811852 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55dd495fc260 msgr2=0x55dd4968fe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:14:52.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.509+0000 7f227ea00640 1 --2- 172.21.15.16:0/3449811852 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55dd495fc260 0x55dd4968fe70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:52.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.509+0000 7f227ea00640 1 -- 172.21.15.16:0/3449811852 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55dd496019f0 msgr2=0x55dd495b61b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:14:52.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.509+0000 7f227ea00640 1 --2- 172.21.15.16:0/3449811852 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55dd496019f0 0x55dd495b61b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:52.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.509+0000 7f227ea00640 1 -- 172.21.15.16:0/3449811852 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dd497204c0 con 0x55dd4968dab0 2024-03-31T15:14:52.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.509+0000 7f227d600640 1 --2- 172.21.15.16:0/3449811852 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55dd496019f0 0x55dd495b61b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:14:52.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.509+0000 7f227ea00640 1 --2- 172.21.15.16:0/3449811852 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dd4968dab0 0x55dd49600540 secure :-1 s=READY pgs=1794 cs=0 l=1 rev1=1 crypto rx=0x7f2278019710 tx=0x7f22780033b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0df5e3a4dcca390 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:52.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.509+0000 7f2277e00640 1 -- 172.21.15.16:0/3449811852 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22780b9070 con 0x55dd4968dab0 2024-03-31T15:14:52.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.509+0000 7f2277e00640 1 -- 172.21.15.16:0/3449811852 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f22780ab700 con 0x55dd4968dab0 2024-03-31T15:14:52.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.509+0000 7f2277e00640 1 -- 172.21.15.16:0/3449811852 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22780b0040 con 0x55dd4968dab0 2024-03-31T15:14:52.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.509+0000 7f2280db4080 1 -- 172.21.15.16:0/3449811852 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dd4968dab0 msgr2=0x55dd49600540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:52.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.509+0000 7f2280db4080 1 --2- 172.21.15.16:0/3449811852 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dd4968dab0 0x55dd49600540 secure :-1 s=READY pgs=1794 cs=0 l=1 rev1=1 crypto rx=0x7f2278019710 tx=0x7f22780033b0 comp rx=0 tx=0).stop 2024-03-31T15:14:52.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.509+0000 7f2280db4080 1 -- 172.21.15.16:0/3449811852 shutdown_connections 2024-03-31T15:14:52.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.509+0000 7f2280db4080 1 --2- 172.21.15.16:0/3449811852 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dd4968dab0 0x55dd49600540 unknown :-1 s=CLOSED pgs=1794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:52.507 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.509+0000 7f2280db4080 1 --2- 172.21.15.16:0/3449811852 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55dd496019f0 0x55dd495b61b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:52.507 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.509+0000 7f2280db4080 1 --2- 172.21.15.16:0/3449811852 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55dd495fc260 0x55dd4968fe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:52.507 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.509+0000 7f2280db4080 1 -- 172.21.15.16:0/3449811852 >> 172.21.15.16:0/3449811852 conn(0x55dd495ff0b0 msgr2=0x55dd495ff490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:14:52.507 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.509+0000 7f2280db4080 1 -- 172.21.15.16:0/3449811852 shutdown_connections 2024-03-31T15:14:52.507 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.509+0000 7f2280db4080 1 -- 172.21.15.16:0/3449811852 wait complete. 2024-03-31T15:14:52.507 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.513+0000 7f2280db4080 1 Processor -- start 2024-03-31T15:14:52.507 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.513+0000 7f2280db4080 1 -- start start 2024-03-31T15:14:52.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.513+0000 7f2280db4080 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55dd495fc260 0x55dd49670b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:52.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.513+0000 7f2280db4080 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dd496019f0 0x55dd49690860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:52.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.513+0000 7f2280db4080 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55dd4968dab0 0x55dd4966d7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:52.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.513+0000 7f2280db4080 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55dd49720a00 con 0x55dd496019f0 2024-03-31T15:14:52.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.513+0000 7f2280db4080 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55dd49720760 con 0x55dd4968dab0 2024-03-31T15:14:52.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.513+0000 7f2280db4080 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x55dd49720ca0 con 0x55dd495fc260 2024-03-31T15:14:52.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.513+0000 7f227d600640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dd496019f0 0x55dd49690860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:52.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.513+0000 7f227d600640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dd496019f0 0x55dd49690860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55472/0 (socket says 172.21.15.16:55472) 2024-03-31T15:14:52.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.513+0000 7f227d600640 1 -- 172.21.15.16:0/3454494716 learned_addr learned my addr 172.21.15.16:0/3454494716 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:14:52.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.513+0000 7f227e000640 1 --2- 172.21.15.16:0/3454494716 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55dd495fc260 0x55dd49670b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:52.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.513+0000 7f227ea00640 1 --2- 172.21.15.16:0/3454494716 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55dd4968dab0 0x55dd4966d7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:52.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.513+0000 7f227d600640 1 -- 172.21.15.16:0/3454494716 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55dd495fc260 msgr2=0x55dd49670b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:52.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.513+0000 7f227d600640 1 --2- 172.21.15.16:0/3454494716 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55dd495fc260 0x55dd49670b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:52.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.513+0000 7f227d600640 1 -- 172.21.15.16:0/3454494716 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55dd4968dab0 msgr2=0x55dd4966d7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:52.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.513+0000 7f227d600640 1 --2- 172.21.15.16:0/3454494716 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55dd4968dab0 0x55dd4966d7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:52.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.513+0000 7f227d600640 1 -- 172.21.15.16:0/3454494716 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dd495b7dc0 con 0x55dd496019f0 2024-03-31T15:14:52.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.513+0000 7f227d600640 1 --2- 172.21.15.16:0/3454494716 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dd496019f0 0x55dd49690860 secure :-1 s=READY pgs=1795 cs=0 l=1 rev1=1 crypto rx=0x7f226c00df90 tx=0x7f226c00b680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:52.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.513+0000 7f2276a00640 1 -- 172.21.15.16:0/3454494716 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f226c014020 con 0x55dd496019f0 2024-03-31T15:14:52.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.513+0000 7f2276a00640 1 -- 172.21.15.16:0/3454494716 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f226c001660 con 0x55dd496019f0 2024-03-31T15:14:52.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.513+0000 7f2276a00640 1 -- 172.21.15.16:0/3454494716 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f226c00a040 con 0x55dd496019f0 2024-03-31T15:14:52.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.517+0000 7f2280db4080 1 -- 172.21.15.16:0/3454494716 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55dd495b8060 con 0x55dd496019f0 2024-03-31T15:14:52.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.517+0000 7f2280db4080 1 -- 172.21.15.16:0/3454494716 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dd495b85a0 con 0x55dd496019f0 2024-03-31T15:14:52.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.517+0000 7f2276a00640 1 -- 172.21.15.16:0/3454494716 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f226c017020 con 0x55dd496019f0 2024-03-31T15:14:52.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.517+0000 7f2276a00640 1 --2- 172.21.15.16:0/3454494716 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f2260066d60 0x7f2260069220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:52.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.517+0000 7f2276a00640 1 -- 172.21.15.16:0/3454494716 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(112..112 src has 1..112) ==== 5853+0+0 (secure 0 0 0) 0x7f226c085730 con 0x55dd496019f0 2024-03-31T15:14:52.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.517+0000 7f2280db4080 1 --2- 172.21.15.16:0/3454494716 >> [v2:172.21.15.89:6809/3470471752,v1:172.21.15.89:6812/3470471752] conn(0x55dd49690da0 0x55dd495b6c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:52.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.517+0000 7f2280db4080 1 -- 172.21.15.16:0/3454494716 --> [v2:172.21.15.89:6809/3470471752,v1:172.21.15.89:6812/3470471752] -- osd_op(unknown.0.0:1 17.5 17:a367a3e8:::10000000004.00000000:head [getxattr scrub_tag in=9b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e112) -- 0x55dd495b71d0 con 0x55dd49690da0 2024-03-31T15:14:52.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.517+0000 7f227e000640 1 --2- 172.21.15.16:0/3454494716 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f2260066d60 0x7f2260069220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:52.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.517+0000 7f227ea00640 1 --2- 172.21.15.16:0/3454494716 >> [v2:172.21.15.89:6809/3470471752,v1:172.21.15.89:6812/3470471752] conn(0x55dd49690da0 0x55dd495b6c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:52.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.517+0000 7f227e000640 1 --2- 172.21.15.16:0/3454494716 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f2260066d60 0x7f2260069220 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f2270001ee0 tx=0x7f2270001d90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:52.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.521+0000 7f227ea00640 1 --2- 172.21.15.16:0/3454494716 >> [v2:172.21.15.89:6809/3470471752,v1:172.21.15.89:6812/3470471752] conn(0x55dd49690da0 0x55dd495b6c90 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:52.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.521+0000 7f227ea00640 1 -- 172.21.15.16:0/3454494716 <== osd.6 v2:172.21.15.89:6809/3470471752 1 ==== osd_op_reply(1 10000000004.00000000 [getxattr (12) out=12b] v0'0 uv12 ondisk = 0) ==== 164+0+12 (crc 0 0 0) 0x7f22780ab8b0 con 0x55dd49690da0 2024-03-31T15:14:52.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.521+0000 7f2280db4080 1 -- 172.21.15.16:0/3454494716 >> [v2:172.21.15.89:6809/3470471752,v1:172.21.15.89:6812/3470471752] conn(0x55dd49690da0 msgr2=0x55dd495b6c90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:52.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.521+0000 7f2280db4080 1 --2- 172.21.15.16:0/3454494716 >> [v2:172.21.15.89:6809/3470471752,v1:172.21.15.89:6812/3470471752] conn(0x55dd49690da0 0x55dd495b6c90 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:52.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.521+0000 7f2280db4080 1 -- 172.21.15.16:0/3454494716 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f2260066d60 msgr2=0x7f2260069220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:52.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.521+0000 7f2280db4080 1 --2- 172.21.15.16:0/3454494716 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f2260066d60 0x7f2260069220 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f2270001ee0 tx=0x7f2270001d90 comp rx=0 tx=0).stop 2024-03-31T15:14:52.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.521+0000 7f2280db4080 1 -- 172.21.15.16:0/3454494716 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dd496019f0 msgr2=0x55dd49690860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:52.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.521+0000 7f2280db4080 1 --2- 172.21.15.16:0/3454494716 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dd496019f0 0x55dd49690860 secure :-1 s=READY pgs=1795 cs=0 l=1 rev1=1 crypto rx=0x7f226c00df90 tx=0x7f226c00b680 comp rx=0 tx=0).stop 2024-03-31T15:14:52.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.521+0000 7f227ea00640 1 -- 172.21.15.16:0/3454494716 reap_dead start 2024-03-31T15:14:52.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.521+0000 7f2280db4080 1 -- 172.21.15.16:0/3454494716 shutdown_connections 2024-03-31T15:14:52.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.521+0000 7f2280db4080 1 -- 172.21.15.16:0/3454494716 >> 172.21.15.16:0/3454494716 conn(0x55dd495ff0b0 msgr2=0x55dd495ff490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:14:52.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.521+0000 7f2280db4080 1 -- 172.21.15.16:0/3454494716 shutdown_connections 2024-03-31T15:14:52.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.521+0000 7f2280db4080 1 -- 172.21.15.16:0/3454494716 wait complete. 2024-03-31T15:14:52.518 INFO:teuthology.orchestra.run.smithi016.stdout:mytag123 2024-03-31T15:14:52.519 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.8:0 flush journal 2024-03-31T15:14:52.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.669+0000 7f30ec000640 1 Processor -- start 2024-03-31T15:14:52.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.669+0000 7f30ec000640 1 -- start start 2024-03-31T15:14:52.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.669+0000 7f30ec000640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f30e40b60a0 0x7f30e406fc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:52.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.669+0000 7f30ec000640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f30e40d0c00 0x7f30e4070180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:52.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.669+0000 7f30ec000640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f30e406f540 con 0x7f30e406b620 2024-03-31T15:14:52.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.669+0000 7f30ec000640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f30e406f820 con 0x7f30e40b60a0 2024-03-31T15:14:52.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.669+0000 7f30ec000640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f30e40b4a40 con 0x7f30e40d0c00 2024-03-31T15:14:52.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.669+0000 7f30e8c00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f30e40b60a0 0x7f30e406fc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:52.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.669+0000 7f30e9600640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f30e406b620 0x7f30e406ba00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:55502/0 (socket says 172.21.15.16:55502) 2024-03-31T15:14:52.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.669+0000 7f30e9600640 1 -- 172.21.15.16:0/1805606079 learned_addr learned my addr 172.21.15.16:0/1805606079 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:14:52.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.669+0000 7f30ea000640 1 --2- 172.21.15.16:0/1805606079 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f30e40d0c00 0x7f30e4070180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:52.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.669+0000 7f30e8c00640 1 -- 172.21.15.16:0/1805606079 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f30e40d0c00 msgr2=0x7f30e4070180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:14:52.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.669+0000 7f30e8c00640 1 --2- 172.21.15.16:0/1805606079 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f30e40d0c00 0x7f30e4070180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:52.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.669+0000 7f30e8c00640 1 -- 172.21.15.16:0/1805606079 >> v1:172.21.15.16:6789/0 conn(0x7f30e406b620 legacy=0x7f30e406ba00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:52.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.669+0000 7f30e8c00640 1 -- 172.21.15.16:0/1805606079 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f30e40b43f0 con 0x7f30e40b60a0 2024-03-31T15:14:52.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.669+0000 7f30e3e00640 1 -- 172.21.15.16:0/1805606079 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2881104167 0 0) 0x7f30e406f540 con 0x7f30e406b620 2024-03-31T15:14:52.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.673+0000 7f30e8c00640 1 --2- 172.21.15.16:0/1805606079 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f30e40b60a0 0x7f30e406fc40 secure :-1 s=READY pgs=1797 cs=0 l=1 rev1=1 crypto rx=0x7f30d00097c0 tx=0x7f30d001c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca1f1edf59ceaaec server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:52.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.673+0000 7f30e3e00640 1 -- 172.21.15.16:0/1805606079 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30d00ab600 con 0x7f30e40b60a0 2024-03-31T15:14:52.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.673+0000 7f30e3e00640 1 -- 172.21.15.16:0/1805606079 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f30d00b0070 con 0x7f30e40b60a0 2024-03-31T15:14:52.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.673+0000 7f30e3e00640 1 -- 172.21.15.16:0/1805606079 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30d00ab9c0 con 0x7f30e40b60a0 2024-03-31T15:14:52.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.673+0000 7f30ec000640 1 -- 172.21.15.16:0/1805606079 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f30e40b60a0 msgr2=0x7f30e406fc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:52.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.673+0000 7f30ec000640 1 --2- 172.21.15.16:0/1805606079 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f30e40b60a0 0x7f30e406fc40 secure :-1 s=READY pgs=1797 cs=0 l=1 rev1=1 crypto rx=0x7f30d00097c0 tx=0x7f30d001c3e0 comp rx=0 tx=0).stop 2024-03-31T15:14:52.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.673+0000 7f30ec000640 1 -- 172.21.15.16:0/1805606079 shutdown_connections 2024-03-31T15:14:52.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.673+0000 7f30ec000640 1 --2- 172.21.15.16:0/1805606079 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f30e40d0c00 0x7f30e4070180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:52.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.673+0000 7f30ec000640 1 --2- 172.21.15.16:0/1805606079 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f30e40b60a0 0x7f30e406fc40 unknown :-1 s=CLOSED pgs=1797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:52.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.673+0000 7f30ec000640 1 -- 172.21.15.16:0/1805606079 >> 172.21.15.16:0/1805606079 conn(0x7f30e406e670 msgr2=0x7f30e406ea50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:14:52.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.673+0000 7f30ec000640 1 -- 172.21.15.16:0/1805606079 shutdown_connections 2024-03-31T15:14:52.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.673+0000 7f30ec000640 1 -- 172.21.15.16:0/1805606079 wait complete. 2024-03-31T15:14:52.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.677+0000 7f30ec000640 1 Processor -- start 2024-03-31T15:14:52.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.677+0000 7f30ec000640 1 -- start start 2024-03-31T15:14:52.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.677+0000 7f30ec000640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f30e406b620 0x7f30e4153870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:52.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.677+0000 7f30ec000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f30e40b60a0 0x7f30e414c740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:52.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.677+0000 7f30ec000640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f30e40d0c00 0x7f30e41485f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:52.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.677+0000 7f30ec000640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f30e416bd00 con 0x7f30e40b60a0 2024-03-31T15:14:52.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.677+0000 7f30ec000640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f30e416bb80 con 0x7f30e40d0c00 2024-03-31T15:14:52.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.677+0000 7f30ec000640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f30e416be80 con 0x7f30e406b620 2024-03-31T15:14:52.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.677+0000 7f30e8c00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f30e40b60a0 0x7f30e414c740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:52.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.677+0000 7f30e8c00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f30e40b60a0 0x7f30e414c740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55490/0 (socket says 172.21.15.16:55490) 2024-03-31T15:14:52.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.677+0000 7f30e8c00640 1 -- 172.21.15.16:0/1077244602 learned_addr learned my addr 172.21.15.16:0/1077244602 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:14:52.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.677+0000 7f30ea000640 1 --2- 172.21.15.16:0/1077244602 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f30e40d0c00 0x7f30e41485f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:52.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.677+0000 7f30e9600640 1 --2- 172.21.15.16:0/1077244602 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f30e406b620 0x7f30e4153870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:52.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.677+0000 7f30e8c00640 1 -- 172.21.15.16:0/1077244602 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f30e406b620 msgr2=0x7f30e4153870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:52.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.677+0000 7f30e8c00640 1 --2- 172.21.15.16:0/1077244602 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f30e406b620 0x7f30e4153870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:52.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.677+0000 7f30e8c00640 1 -- 172.21.15.16:0/1077244602 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f30e40d0c00 msgr2=0x7f30e41485f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:52.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.677+0000 7f30e8c00640 1 --2- 172.21.15.16:0/1077244602 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f30e40d0c00 0x7f30e41485f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:52.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.677+0000 7f30e8c00640 1 -- 172.21.15.16:0/1077244602 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f30e4153db0 con 0x7f30e40b60a0 2024-03-31T15:14:52.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.677+0000 7f30ea000640 1 --2- 172.21.15.16:0/1077244602 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f30e40d0c00 0x7f30e41485f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:14:52.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.677+0000 7f30e8c00640 1 --2- 172.21.15.16:0/1077244602 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f30e40b60a0 0x7f30e414c740 secure :-1 s=READY pgs=1798 cs=0 l=1 rev1=1 crypto rx=0x7f30d00097c0 tx=0x7f30d001ac20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:52.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.677+0000 7f30e1600640 1 -- 172.21.15.16:0/1077244602 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30d0004220 con 0x7f30e40b60a0 2024-03-31T15:14:52.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.677+0000 7f30e1600640 1 -- 172.21.15.16:0/1077244602 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f30d00043c0 con 0x7f30e40b60a0 2024-03-31T15:14:52.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.677+0000 7f30ec000640 1 -- 172.21.15.16:0/1077244602 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f30e4155800 con 0x7f30e40b60a0 2024-03-31T15:14:52.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.677+0000 7f30e1600640 1 -- 172.21.15.16:0/1077244602 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30d0003820 con 0x7f30e40b60a0 2024-03-31T15:14:52.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.677+0000 7f30ec000640 1 -- 172.21.15.16:0/1077244602 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f30e4153070 con 0x7f30e40b60a0 2024-03-31T15:14:52.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.681+0000 7f30e1600640 1 -- 172.21.15.16:0/1077244602 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f30d00aa070 con 0x7f30e40b60a0 2024-03-31T15:14:52.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.681+0000 7f30e1600640 1 --2- 172.21.15.16:0/1077244602 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f30c4068380 0x7f30c406a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:52.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.681+0000 7f30e1600640 1 -- 172.21.15.16:0/1077244602 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(112..112 src has 1..112) ==== 5853+0+0 (secure 0 0 0) 0x7f30d0120e90 con 0x7f30e40b60a0 2024-03-31T15:14:52.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.681+0000 7f30e9600640 1 --2- 172.21.15.16:0/1077244602 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f30c4068380 0x7f30c406a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:52.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.681+0000 7f30ef590000 1 Processor -- start 2024-03-31T15:14:52.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.681+0000 7f30ef590000 1 -- start start 2024-03-31T15:14:52.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.681+0000 7f30e9600640 1 --2- 172.21.15.16:0/1077244602 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f30c4068380 0x7f30c406a840 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f30d8000f70 tx=0x7f30d8007b80 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:52.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.681+0000 7f30ef590000 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55ee246452d0 0x55ee246456b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:52.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.681+0000 7f30ef590000 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ee24645bf0 0x55ee245f8470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:52.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.681+0000 7f30ef590000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ee245f89b0 0x55ee245f8d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:52.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.681+0000 7f30ef590000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55ee244e1af0 con 0x55ee245f89b0 2024-03-31T15:14:52.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.681+0000 7f30ef590000 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55ee2444ab00 con 0x55ee24645bf0 2024-03-31T15:14:52.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.681+0000 7f30ef590000 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x55ee243e7940 con 0x55ee246452d0 2024-03-31T15:14:52.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.685+0000 7f30ea000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ee245f89b0 0x55ee245f8d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:52.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.685+0000 7f30e8c00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ee24645bf0 0x55ee245f8470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:52.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.685+0000 7f30e9600640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55ee246452d0 0x55ee246456b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:52.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.685+0000 7f30e8c00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ee24645bf0 0x55ee245f8470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.16:53624/0 (socket says 172.21.15.16:53624) 2024-03-31T15:14:52.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.685+0000 7f30e8c00640 1 -- 172.21.15.16:0/1594300682 learned_addr learned my addr 172.21.15.16:0/1594300682 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:14:52.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.685+0000 7f30e8c00640 1 -- 172.21.15.16:0/1594300682 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55ee246452d0 msgr2=0x55ee246456b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:14:52.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.685+0000 7f30e8c00640 1 --2- 172.21.15.16:0/1594300682 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55ee246452d0 0x55ee246456b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:52.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.685+0000 7f30e8c00640 1 -- 172.21.15.16:0/1594300682 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ee245f89b0 msgr2=0x55ee245f8d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:14:52.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.685+0000 7f30e8c00640 1 --2- 172.21.15.16:0/1594300682 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ee245f89b0 0x55ee245f8d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:52.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.685+0000 7f30e8c00640 1 -- 172.21.15.16:0/1594300682 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ee243d0380 con 0x55ee24645bf0 2024-03-31T15:14:52.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.685+0000 7f30ea000640 1 --2- 172.21.15.16:0/1594300682 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ee245f89b0 0x55ee245f8d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:14:52.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.685+0000 7f30e8c00640 1 --2- 172.21.15.16:0/1594300682 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ee24645bf0 0x55ee245f8470 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f30d00f9360 tx=0x7f30d00ed080 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7c0373c21d3dc52d server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:52.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.685+0000 7f30d6000640 1 -- 172.21.15.16:0/1594300682 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30d00c4040 con 0x55ee24645bf0 2024-03-31T15:14:52.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.685+0000 7f30d6000640 1 -- 172.21.15.16:0/1594300682 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f30d00b0050 con 0x55ee24645bf0 2024-03-31T15:14:52.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.685+0000 7f30d6000640 1 -- 172.21.15.16:0/1594300682 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30d00f0cb0 con 0x55ee24645bf0 2024-03-31T15:14:52.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.685+0000 7f30ef590000 1 -- 172.21.15.16:0/1594300682 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ee24645bf0 msgr2=0x55ee245f8470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:52.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.685+0000 7f30ef590000 1 --2- 172.21.15.16:0/1594300682 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ee24645bf0 0x55ee245f8470 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f30d00f9360 tx=0x7f30d00ed080 comp rx=0 tx=0).stop 2024-03-31T15:14:52.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.685+0000 7f30ef590000 1 -- 172.21.15.16:0/1594300682 shutdown_connections 2024-03-31T15:14:52.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.685+0000 7f30ef590000 1 --2- 172.21.15.16:0/1594300682 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ee245f89b0 0x55ee245f8d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:52.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.685+0000 7f30ef590000 1 --2- 172.21.15.16:0/1594300682 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ee24645bf0 0x55ee245f8470 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:52.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.685+0000 7f30ef590000 1 --2- 172.21.15.16:0/1594300682 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55ee246452d0 0x55ee246456b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:52.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.685+0000 7f30ef590000 1 -- 172.21.15.16:0/1594300682 >> 172.21.15.16:0/1594300682 conn(0x55ee24626070 msgr2=0x55ee24628460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:14:52.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.685+0000 7f30ef590000 1 -- 172.21.15.16:0/1594300682 shutdown_connections 2024-03-31T15:14:52.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.685+0000 7f30ef590000 1 -- 172.21.15.16:0/1594300682 wait complete. 2024-03-31T15:14:52.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.685+0000 7f30ef590000 1 -- start start 2024-03-31T15:14:52.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.685+0000 7f30ef590000 1 Processor -- start 2024-03-31T15:14:52.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.685+0000 7f30ef590000 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55ee245f89b0 0x55ee24600a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:52.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.689+0000 7f30ef590000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ee246452d0 0x55ee246461e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:52.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.689+0000 7f30ef590000 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ee24645bf0 0x55ee24646720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:52.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.689+0000 7f30ef590000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55ee244081e0 con 0x55ee246452d0 2024-03-31T15:14:52.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.689+0000 7f30ef590000 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55ee24446ab0 con 0x55ee24645bf0 2024-03-31T15:14:52.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.689+0000 7f30ef590000 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x55ee24604000 con 0x55ee245f89b0 2024-03-31T15:14:52.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.689+0000 7f30ef590000 10 client.0 _ll_register_callbacks cb 0x55ee245e6370 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T15:14:52.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.689+0000 7f30e8c00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ee246452d0 0x55ee246461e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:52.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.689+0000 7f30e9600640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55ee245f89b0 0x55ee24600a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:52.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.689+0000 7f30ea000640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ee24645bf0 0x55ee24646720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:52.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.689+0000 7f30e9600640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55ee245f89b0 0x55ee24600a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3301/0 says I am v2:172.21.15.16:56318/0 (socket says 172.21.15.16:56318) 2024-03-31T15:14:52.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.689+0000 7f30e9600640 1 -- 172.21.15.16:0/2023612474 learned_addr learned my addr 172.21.15.16:0/2023612474 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:14:52.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.689+0000 7f30e9600640 1 -- 172.21.15.16:0/2023612474 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ee24645bf0 msgr2=0x55ee24646720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:14:52.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.689+0000 7f30e9600640 1 --2- 172.21.15.16:0/2023612474 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ee24645bf0 0x55ee24646720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:52.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.689+0000 7f30e9600640 1 -- 172.21.15.16:0/2023612474 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ee246452d0 msgr2=0x55ee246461e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:14:52.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.689+0000 7f30e9600640 1 --2- 172.21.15.16:0/2023612474 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ee246452d0 0x55ee246461e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:52.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.689+0000 7f30e9600640 1 -- 172.21.15.16:0/2023612474 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({osdmap=0}) -- 0x55ee24406b70 con 0x55ee245f89b0 2024-03-31T15:14:52.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.689+0000 7f30e9600640 1 --2- 172.21.15.16:0/2023612474 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55ee245f89b0 0x55ee24600a40 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f30d8001e90 tx=0x7f30d8000ee0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ea23ba5776b26619 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:52.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.689+0000 7f30caa00640 10 client.0 ms_handle_connect on v2:172.21.15.89:3301/0 2024-03-31T15:14:52.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.689+0000 7f30caa00640 1 -- 172.21.15.16:0/2023612474 <== mon.2 v2:172.21.15.89:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30d80213f0 con 0x55ee245f89b0 2024-03-31T15:14:52.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.689+0000 7f30caa00640 1 -- 172.21.15.16:0/2023612474 <== mon.2 v2:172.21.15.89:3301/0 2 ==== osd_map(112..112 src has 1..112) ==== 5853+0+0 (secure 0 0 0) 0x7f30d8010040 con 0x55ee245f89b0 2024-03-31T15:14:52.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.689+0000 7f30ef590000 1 -- 172.21.15.16:0/2023612474 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f30d00b0050 con 0x55ee245f89b0 2024-03-31T15:14:52.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.689+0000 7f30caa00640 1 -- 172.21.15.16:0/2023612474 <== mon.2 v2:172.21.15.89:3301/0 3 ==== mon_get_version_reply(handle=1 version=105) ==== 24+0+0 (secure 0 0 0) 0x7f30d802e050 con 0x55ee245f89b0 2024-03-31T15:14:52.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.689+0000 7f30ef590000 10 client.7997 fetch_fsmap learned FSMap version 105 2024-03-31T15:14:52.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.689+0000 7f30ef590000 1 -- 172.21.15.16:0/2023612474 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({config=0+,fsmap=105,monmap=2+}) -- 0x55ee243e7b90 con 0x55ee245f89b0 2024-03-31T15:14:52.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.689+0000 7f30caa00640 1 -- 172.21.15.16:0/2023612474 <== mon.2 v2:172.21.15.89:3301/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f30d00b0050 con 0x55ee245f89b0 2024-03-31T15:14:52.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.689+0000 7f30caa00640 1 -- 172.21.15.16:0/2023612474 <== mon.2 v2:172.21.15.89:3301/0 5 ==== fsmap(e 105) ==== 3494+0+0 (secure 0 0 0) 0x7f30d8031020 con 0x55ee245f89b0 2024-03-31T15:14:52.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.689+0000 7f30ef590000 10 client.7997 fetch_fsmap finished waiting for FSMap version 105 2024-03-31T15:14:52.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.689+0000 7f30ef590000 10 client.7997 resolve_mds: resolved 8:0 to role '8:0' aka daemon mds.c 2024-03-31T15:14:52.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.689+0000 7f30ef590000 20 client.7997 populate_metadata read hostname 'smithi016' 2024-03-31T15:14:52.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.689+0000 7f30ef590000 1 --2- 172.21.15.16:0/2023612474 >> [v2:172.21.15.16:6833/4142860904,v1:172.21.15.16:6835/4142860904] conn(0x55ee246474c0 0x55ee24656eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:52.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.689+0000 7f30ef590000 4 client.7997 mds_command: new command op to 9558 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-31T15:14:52.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.689+0000 7f30ef590000 1 -- 172.21.15.16:0/2023612474 --> [v2:172.21.15.16:6833/4142860904,v1:172.21.15.16:6835/4142860904] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ee243e5dd0 con 0x55ee246474c0 2024-03-31T15:14:52.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.689+0000 7f30e8c00640 1 --2- 172.21.15.16:0/2023612474 >> [v2:172.21.15.16:6833/4142860904,v1:172.21.15.16:6835/4142860904] conn(0x55ee246474c0 0x55ee24656eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:52.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.689+0000 7f30e8c00640 1 --2- 172.21.15.16:0/2023612474 >> [v2:172.21.15.16:6833/4142860904,v1:172.21.15.16:6835/4142860904] conn(0x55ee246474c0 0x55ee24656eb0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f10d995523689af7 server_cookie=219e9bc4e18f3a15 in_seq=0 out_seq=0 2024-03-31T15:14:52.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.689+0000 7f30caa00640 10 client.7997 ms_handle_connect on v2:172.21.15.16:6833/4142860904 2024-03-31T15:14:52.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.689+0000 7f30caa00640 1 -- 172.21.15.16:0/2023612474 <== mds.0 v2:172.21.15.16:6833/4142860904 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55ee243e5dd0 con 0x55ee246474c0 2024-03-31T15:14:52.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.689+0000 7f30caa00640 10 client.7997 handle_command_reply: tid=0 2024-03-31T15:14:52.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.689+0000 7f30ef590000 1 client.7997 shutdown 2024-03-31T15:14:52.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.689+0000 7f30ef590000 1 -- 172.21.15.16:0/2023612474 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55ee245f89b0 msgr2=0x55ee24600a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:52.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.689+0000 7f30ef590000 1 --2- 172.21.15.16:0/2023612474 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55ee245f89b0 0x55ee24600a40 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f30d8001e90 tx=0x7f30d8000ee0 comp rx=0 tx=0).stop 2024-03-31T15:14:52.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.689+0000 7f30ef590000 1 -- 172.21.15.16:0/2023612474 shutdown_connections 2024-03-31T15:14:52.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.689+0000 7f30ef590000 1 --2- 172.21.15.16:0/2023612474 >> [v2:172.21.15.16:6833/4142860904,v1:172.21.15.16:6835/4142860904] conn(0x55ee246474c0 0x55ee24656eb0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:52.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.689+0000 7f30ef590000 1 --2- 172.21.15.16:0/2023612474 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ee24645bf0 0x55ee24646720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:52.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.689+0000 7f30ef590000 1 --2- 172.21.15.16:0/2023612474 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ee246452d0 0x55ee246461e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:52.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.689+0000 7f30ef590000 1 --2- 172.21.15.16:0/2023612474 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55ee245f89b0 0x55ee24600a40 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:52.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.689+0000 7f30caa00640 0 client.7997 ms_handle_reset on v2:172.21.15.16:6833/4142860904 2024-03-31T15:14:52.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.689+0000 7f30ef590000 1 -- 172.21.15.16:0/2023612474 >> 172.21.15.16:0/2023612474 conn(0x55ee24626070 msgr2=0x55ee24628460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:14:52.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.693+0000 7f30ef590000 1 -- 172.21.15.16:0/2023612474 shutdown_connections 2024-03-31T15:14:52.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.693+0000 7f30ef590000 1 -- 172.21.15.16:0/2023612474 wait complete. 2024-03-31T15:14:52.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.693+0000 7f30ef590000 20 client.7997 trim_cache size 0 max 16384 2024-03-31T15:14:52.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.705+0000 7f30ef590000 1 Processor -- start 2024-03-31T15:14:52.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.705+0000 7f30ef590000 1 -- start start 2024-03-31T15:14:52.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.705+0000 7f30ef590000 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ee245f89b0 0x55ee24643a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:52.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.705+0000 7f30ef590000 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55ee246452d0 0x55ee24643fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:52.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.705+0000 7f30ef590000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ee24645bf0 0x55ee24656eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:52.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.705+0000 7f30ef590000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f30e41512b0 con 0x55ee24645bf0 2024-03-31T15:14:52.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.705+0000 7f30ef590000 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f30e4151900 con 0x55ee245f89b0 2024-03-31T15:14:52.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.705+0000 7f30ef590000 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x55ee24505250 con 0x55ee246452d0 2024-03-31T15:14:52.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.709+0000 7f30ea000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ee24645bf0 0x55ee24656eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:52.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.709+0000 7f30ea000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ee24645bf0 0x55ee24656eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55508/0 (socket says 172.21.15.16:55508) 2024-03-31T15:14:52.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.709+0000 7f30ea000640 1 -- 172.21.15.16:0/1940139079 learned_addr learned my addr 172.21.15.16:0/1940139079 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:14:52.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.709+0000 7f30e9600640 1 --2- 172.21.15.16:0/1940139079 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ee245f89b0 0x55ee24643a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:52.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.709+0000 7f30e8c00640 1 --2- 172.21.15.16:0/1940139079 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55ee246452d0 0x55ee24643fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:52.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.709+0000 7f30ea000640 1 -- 172.21.15.16:0/1940139079 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55ee246452d0 msgr2=0x55ee24643fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:14:52.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.709+0000 7f30ea000640 1 --2- 172.21.15.16:0/1940139079 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55ee246452d0 0x55ee24643fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:52.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.709+0000 7f30ea000640 1 -- 172.21.15.16:0/1940139079 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ee245f89b0 msgr2=0x55ee24643a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:14:52.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.709+0000 7f30ea000640 1 --2- 172.21.15.16:0/1940139079 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ee245f89b0 0x55ee24643a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:52.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.709+0000 7f30ea000640 1 -- 172.21.15.16:0/1940139079 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ee24645a20 con 0x55ee24645bf0 2024-03-31T15:14:52.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.709+0000 7f30ea000640 1 --2- 172.21.15.16:0/1940139079 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ee24645bf0 0x55ee24656eb0 secure :-1 s=READY pgs=1799 cs=0 l=1 rev1=1 crypto rx=0x7f30dc015680 tx=0x7f30dc015b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2e6058e4b58ffec server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:52.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.709+0000 7f30d5600640 1 -- 172.21.15.16:0/1940139079 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30dc007a90 con 0x55ee24645bf0 2024-03-31T15:14:52.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.709+0000 7f30d5600640 1 -- 172.21.15.16:0/1940139079 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f30dc007c50 con 0x55ee24645bf0 2024-03-31T15:14:52.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.709+0000 7f30d5600640 1 -- 172.21.15.16:0/1940139079 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30dc015e80 con 0x55ee24645bf0 2024-03-31T15:14:52.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.709+0000 7f30ef590000 1 -- 172.21.15.16:0/1940139079 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ee24645bf0 msgr2=0x55ee24656eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:52.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.709+0000 7f30ef590000 1 --2- 172.21.15.16:0/1940139079 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ee24645bf0 0x55ee24656eb0 secure :-1 s=READY pgs=1799 cs=0 l=1 rev1=1 crypto rx=0x7f30dc015680 tx=0x7f30dc015b40 comp rx=0 tx=0).stop 2024-03-31T15:14:52.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.709+0000 7f30ef590000 1 -- 172.21.15.16:0/1940139079 shutdown_connections 2024-03-31T15:14:52.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.709+0000 7f30ef590000 1 --2- 172.21.15.16:0/1940139079 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ee24645bf0 0x55ee24656eb0 unknown :-1 s=CLOSED pgs=1799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:52.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.709+0000 7f30ef590000 1 --2- 172.21.15.16:0/1940139079 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55ee246452d0 0x55ee24643fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:52.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.709+0000 7f30ef590000 1 --2- 172.21.15.16:0/1940139079 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ee245f89b0 0x55ee24643a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:52.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.709+0000 7f30ef590000 1 -- 172.21.15.16:0/1940139079 >> 172.21.15.16:0/1940139079 conn(0x55ee24626070 msgr2=0x55ee24643380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:14:52.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.709+0000 7f30ef590000 1 -- 172.21.15.16:0/1940139079 shutdown_connections 2024-03-31T15:14:52.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.709+0000 7f30ef590000 1 -- 172.21.15.16:0/1940139079 wait complete. 2024-03-31T15:14:52.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.709+0000 7f30ef590000 1 -- start start 2024-03-31T15:14:52.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.709+0000 7f30ef590000 1 Processor -- start 2024-03-31T15:14:52.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.713+0000 7f30ef590000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ee245f89b0 0x55ee24659b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:52.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.713+0000 7f30ef590000 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ee246452d0 0x55ee246b5b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:52.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.713+0000 7f30ef590000 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55ee24645bf0 0x55ee246b60d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:52.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.713+0000 7f30ef590000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55ee24416150 con 0x55ee245f89b0 2024-03-31T15:14:52.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.713+0000 7f30ef590000 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55ee24519420 con 0x55ee246452d0 2024-03-31T15:14:52.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.713+0000 7f30ef590000 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x55ee243e5940 con 0x55ee24645bf0 2024-03-31T15:14:52.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.713+0000 7f30ef590000 10 client.0 _ll_register_callbacks cb 0x7f30d80072d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T15:14:52.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.713+0000 7f30e9600640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ee245f89b0 0x55ee24659b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:52.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.713+0000 7f30e9600640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ee245f89b0 0x55ee24659b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55524/0 (socket says 172.21.15.16:55524) 2024-03-31T15:14:52.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.713+0000 7f30e9600640 1 -- 172.21.15.16:0/2818562666 learned_addr learned my addr 172.21.15.16:0/2818562666 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:14:52.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.713+0000 7f30ea000640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55ee24645bf0 0x55ee246b60d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:52.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.713+0000 7f30e8c00640 1 --2- 172.21.15.16:0/2818562666 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ee246452d0 0x55ee246b5b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:52.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.713+0000 7f30e9600640 1 -- 172.21.15.16:0/2818562666 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55ee24645bf0 msgr2=0x55ee246b60d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:14:52.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.713+0000 7f30e9600640 1 --2- 172.21.15.16:0/2818562666 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55ee24645bf0 0x55ee246b60d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:52.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.713+0000 7f30e9600640 1 -- 172.21.15.16:0/2818562666 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ee246452d0 msgr2=0x55ee246b5b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:14:52.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.713+0000 7f30e9600640 1 --2- 172.21.15.16:0/2818562666 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ee246452d0 0x55ee246b5b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:52.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.713+0000 7f30e9600640 1 -- 172.21.15.16:0/2818562666 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ee244cb140 con 0x55ee245f89b0 2024-03-31T15:14:52.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.713+0000 7f30e8c00640 1 --2- 172.21.15.16:0/2818562666 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ee246452d0 0x55ee246b5b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:14:52.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.713+0000 7f30e9600640 1 --2- 172.21.15.16:0/2818562666 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ee245f89b0 0x55ee24659b60 secure :-1 s=READY pgs=1800 cs=0 l=1 rev1=1 crypto rx=0x7f30d8001ec0 tx=0x7f30d800a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=46fd4b12a2a37a43 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:52.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.713+0000 7f30caa00640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-31T15:14:52.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.713+0000 7f30caa00640 1 -- 172.21.15.16:0/2818562666 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30d8036070 con 0x55ee245f89b0 2024-03-31T15:14:52.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.713+0000 7f30caa00640 1 -- 172.21.15.16:0/2818562666 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(112..112 src has 1..112) ==== 5853+0+0 (secure 0 0 0) 0x7f30d80219b0 con 0x55ee245f89b0 2024-03-31T15:14:52.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.713+0000 7f30ef590000 1 -- 172.21.15.16:0/2818562666 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f30dc007c50 con 0x55ee245f89b0 2024-03-31T15:14:52.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.713+0000 7f30caa00640 1 -- 172.21.15.16:0/2818562666 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=105) ==== 24+0+0 (secure 0 0 0) 0x7f30d8040050 con 0x55ee245f89b0 2024-03-31T15:14:52.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.713+0000 7f30ef590000 10 client.9750 fetch_fsmap learned FSMap version 105 2024-03-31T15:14:52.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.713+0000 7f30ef590000 1 -- 172.21.15.16:0/2818562666 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=105,monmap=2+}) -- 0x55ee243d91e0 con 0x55ee245f89b0 2024-03-31T15:14:52.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.713+0000 7f30caa00640 1 -- 172.21.15.16:0/2818562666 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f30dc007c50 con 0x55ee245f89b0 2024-03-31T15:14:52.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.713+0000 7f30caa00640 1 -- 172.21.15.16:0/2818562666 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 105) ==== 3494+0+0 (secure 0 0 0) 0x7f30d8043020 con 0x55ee245f89b0 2024-03-31T15:14:52.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.713+0000 7f30ef590000 10 client.9750 fetch_fsmap finished waiting for FSMap version 105 2024-03-31T15:14:52.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.713+0000 7f30ef590000 10 client.9750 resolve_mds: resolved 8:0 to role '8:0' aka daemon mds.c 2024-03-31T15:14:52.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.713+0000 7f30ef590000 20 client.9750 populate_metadata read hostname 'smithi016' 2024-03-31T15:14:52.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.713+0000 7f30ef590000 1 --2- 172.21.15.16:0/2818562666 >> [v2:172.21.15.16:6833/4142860904,v1:172.21.15.16:6835/4142860904] conn(0x55ee246b6610 0x55ee246b25f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:52.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.713+0000 7f30ef590000 4 client.9750 mds_command: new command op to 9558 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-31T15:14:52.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.713+0000 7f30ef590000 1 -- 172.21.15.16:0/2818562666 --> [v2:172.21.15.16:6833/4142860904,v1:172.21.15.16:6835/4142860904] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55ee24458040 con 0x55ee246b6610 2024-03-31T15:14:52.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.713+0000 7f30e8c00640 1 --2- 172.21.15.16:0/2818562666 >> [v2:172.21.15.16:6833/4142860904,v1:172.21.15.16:6835/4142860904] conn(0x55ee246b6610 0x55ee246b25f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:52.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.713+0000 7f30e8c00640 1 --2- 172.21.15.16:0/2818562666 >> [v2:172.21.15.16:6833/4142860904,v1:172.21.15.16:6835/4142860904] conn(0x55ee246b6610 0x55ee246b25f0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cfd98c1f679a054c server_cookie=d3b06f5ffa01bb61 in_seq=0 out_seq=0 2024-03-31T15:14:52.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.713+0000 7f30caa00640 10 client.9750 ms_handle_connect on v2:172.21.15.16:6833/4142860904 2024-03-31T15:14:52.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.725+0000 7f30caa00640 1 -- 172.21.15.16:0/2818562666 <== mds.0 v2:172.21.15.16:6833/4142860904 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55ee24458040 con 0x55ee246b6610 2024-03-31T15:14:52.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.729+0000 7f30caa00640 10 client.9750 handle_command_reply: tid=0 2024-03-31T15:14:52.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.729+0000 7f30ef590000 1 client.9750 shutdown 2024-03-31T15:14:52.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.729+0000 7f30ef590000 1 -- 172.21.15.16:0/2818562666 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ee245f89b0 msgr2=0x55ee24659b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:52.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.729+0000 7f30ef590000 1 --2- 172.21.15.16:0/2818562666 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ee245f89b0 0x55ee24659b60 secure :-1 s=READY pgs=1800 cs=0 l=1 rev1=1 crypto rx=0x7f30d8001ec0 tx=0x7f30d800a000 comp rx=0 tx=0).stop 2024-03-31T15:14:52.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.729+0000 7f30ef590000 1 -- 172.21.15.16:0/2818562666 shutdown_connections 2024-03-31T15:14:52.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.729+0000 7f30ef590000 1 --2- 172.21.15.16:0/2818562666 >> [v2:172.21.15.16:6833/4142860904,v1:172.21.15.16:6835/4142860904] conn(0x55ee246b6610 0x55ee246b25f0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:52.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.729+0000 7f30ef590000 1 --2- 172.21.15.16:0/2818562666 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55ee24645bf0 0x55ee246b60d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:52.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.729+0000 7f30ef590000 1 --2- 172.21.15.16:0/2818562666 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ee246452d0 0x55ee246b5b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:52.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.729+0000 7f30ef590000 1 --2- 172.21.15.16:0/2818562666 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ee245f89b0 0x55ee24659b60 unknown :-1 s=CLOSED pgs=1800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:52.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.729+0000 7f30caa00640 0 client.9750 ms_handle_reset on v2:172.21.15.16:6833/4142860904 2024-03-31T15:14:52.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.729+0000 7f30ef590000 1 -- 172.21.15.16:0/2818562666 >> 172.21.15.16:0/2818562666 conn(0x55ee24626070 msgr2=0x55ee24643c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:14:52.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.729+0000 7f30ef590000 1 -- 172.21.15.16:0/2818562666 shutdown_connections 2024-03-31T15:14:52.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.729+0000 7f30ef590000 1 -- 172.21.15.16:0/2818562666 wait complete. 2024-03-31T15:14:52.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.729+0000 7f30ef590000 20 client.9750 trim_cache size 0 max 16384 2024-03-31T15:14:52.723 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-03-31T15:14:52.723 INFO:teuthology.orchestra.run.smithi016.stdout: "message": "", 2024-03-31T15:14:52.723 INFO:teuthology.orchestra.run.smithi016.stdout: "return_code": 0 2024-03-31T15:14:52.723 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-03-31T15:14:52.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.729+0000 7f30ec000640 1 -- 172.21.15.16:0/1077244602 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f30c4068380 msgr2=0x7f30c406a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:52.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.729+0000 7f30ec000640 1 --2- 172.21.15.16:0/1077244602 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f30c4068380 0x7f30c406a840 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f30d8000f70 tx=0x7f30d8007b80 comp rx=0 tx=0).stop 2024-03-31T15:14:52.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.729+0000 7f30ec000640 1 -- 172.21.15.16:0/1077244602 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f30e40b60a0 msgr2=0x7f30e414c740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:52.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.729+0000 7f30ec000640 1 --2- 172.21.15.16:0/1077244602 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f30e40b60a0 0x7f30e414c740 secure :-1 s=READY pgs=1798 cs=0 l=1 rev1=1 crypto rx=0x7f30d00097c0 tx=0x7f30d001ac20 comp rx=0 tx=0).stop 2024-03-31T15:14:52.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.729+0000 7f30ec000640 1 -- 172.21.15.16:0/1077244602 shutdown_connections 2024-03-31T15:14:52.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.729+0000 7f30ec000640 1 --2- 172.21.15.16:0/1077244602 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f30c4068380 0x7f30c406a840 unknown :-1 s=CLOSED pgs=760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:52.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.729+0000 7f30ec000640 1 --2- 172.21.15.16:0/1077244602 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f30e40d0c00 0x7f30e41485f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:52.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.729+0000 7f30ec000640 1 --2- 172.21.15.16:0/1077244602 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f30e40b60a0 0x7f30e414c740 unknown :-1 s=CLOSED pgs=1798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:52.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.729+0000 7f30ec000640 1 --2- 172.21.15.16:0/1077244602 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f30e406b620 0x7f30e4153870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:52.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.729+0000 7f30ec000640 1 -- 172.21.15.16:0/1077244602 >> 172.21.15.16:0/1077244602 conn(0x7f30e406e670 msgr2=0x7f30e415b7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:14:52.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.729+0000 7f30ec000640 1 -- 172.21.15.16:0/1077244602 shutdown_connections 2024-03-31T15:14:52.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.729+0000 7f30ec000640 1 -- 172.21.15.16:0/1077244602 wait complete. 2024-03-31T15:14:52.741 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-31T15:14:52.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.893+0000 7f0184600640 1 Processor -- start 2024-03-31T15:14:52.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.893+0000 7f0184600640 1 -- start start 2024-03-31T15:14:52.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.897+0000 7f0184600640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f017c06f920 0x7f017c06fd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:52.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.897+0000 7f0184600640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f017c0d2010 0x7f017c0d24a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:52.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.897+0000 7f0184600640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f017c0d0920 con 0x7f017c145e40 2024-03-31T15:14:52.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.897+0000 7f0184600640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f017c0702c0 con 0x7f017c0d2010 2024-03-31T15:14:52.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.897+0000 7f0184600640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f017c0d06d0 con 0x7f017c06f920 2024-03-31T15:14:52.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.897+0000 7f0182600640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f017c0d2010 0x7f017c0d24a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:52.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.897+0000 7f0182600640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f017c0d2010 0x7f017c0d24a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55530/0 (socket says 172.21.15.16:55530) 2024-03-31T15:14:52.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.897+0000 7f0182600640 1 -- 172.21.15.16:0/3208698429 learned_addr learned my addr 172.21.15.16:0/3208698429 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:14:52.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.897+0000 7f0181200640 1 --2- 172.21.15.16:0/3208698429 >> v2:172.21.15.89:3300/0 conn(0x7f017c06f920 0x7f017c06fd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:52.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.897+0000 7f0182600640 1 -- 172.21.15.16:0/3208698429 >> v2:172.21.15.89:3300/0 conn(0x7f017c06f920 msgr2=0x7f017c06fd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:14:52.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.897+0000 7f0182600640 1 --2- 172.21.15.16:0/3208698429 >> v2:172.21.15.89:3300/0 conn(0x7f017c06f920 0x7f017c06fd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:52.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.897+0000 7f0182600640 1 -- 172.21.15.16:0/3208698429 >> v1:172.21.15.89:6789/0 conn(0x7f017c145e40 legacy=0x7f017c0cc740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:52.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.897+0000 7f0182600640 1 -- 172.21.15.16:0/3208698429 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f017c0d0c00 con 0x7f017c0d2010 2024-03-31T15:14:52.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.897+0000 7f0181200640 1 --2- 172.21.15.16:0/3208698429 >> v2:172.21.15.89:3300/0 conn(0x7f017c06f920 0x7f017c06fd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:14:52.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.897+0000 7f0182600640 1 --2- 172.21.15.16:0/3208698429 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f017c0d2010 0x7f017c0d24a0 secure :-1 s=READY pgs=1801 cs=0 l=1 rev1=1 crypto rx=0x7f017c073cf0 tx=0x7f017401c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=894c67e3c941caa3 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:52.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.901+0000 7f017be00640 1 -- 172.21.15.16:0/3208698429 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01740a57a0 con 0x7f017c0d2010 2024-03-31T15:14:52.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.901+0000 7f017be00640 1 -- 172.21.15.16:0/3208698429 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f01740a5940 con 0x7f017c0d2010 2024-03-31T15:14:52.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.901+0000 7f017be00640 1 -- 172.21.15.16:0/3208698429 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01740a5d10 con 0x7f017c0d2010 2024-03-31T15:14:52.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.901+0000 7f0184600640 1 -- 172.21.15.16:0/3208698429 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f017c0d2010 msgr2=0x7f017c0d24a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:52.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.901+0000 7f0184600640 1 --2- 172.21.15.16:0/3208698429 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f017c0d2010 0x7f017c0d24a0 secure :-1 s=READY pgs=1801 cs=0 l=1 rev1=1 crypto rx=0x7f017c073cf0 tx=0x7f017401c9f0 comp rx=0 tx=0).stop 2024-03-31T15:14:52.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.901+0000 7f0184600640 1 -- 172.21.15.16:0/3208698429 shutdown_connections 2024-03-31T15:14:52.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.901+0000 7f0184600640 1 --2- 172.21.15.16:0/3208698429 >> v2:172.21.15.89:3300/0 conn(0x7f017c06f920 0x7f017c06fd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:52.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.901+0000 7f0184600640 1 --2- 172.21.15.16:0/3208698429 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f017c0d2010 0x7f017c0d24a0 unknown :-1 s=CLOSED pgs=1801 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:52.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.901+0000 7f0184600640 1 -- 172.21.15.16:0/3208698429 >> 172.21.15.16:0/3208698429 conn(0x7f017c067330 msgr2=0x7f017c067740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:14:52.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.901+0000 7f0184600640 1 -- 172.21.15.16:0/3208698429 shutdown_connections 2024-03-31T15:14:52.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.901+0000 7f0184600640 1 -- 172.21.15.16:0/3208698429 wait complete. 2024-03-31T15:14:52.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.901+0000 7f0184600640 1 Processor -- start 2024-03-31T15:14:52.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.901+0000 7f0184600640 1 -- start start 2024-03-31T15:14:52.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.901+0000 7f0184600640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f017c06f920 0x7f017c1370a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:52.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.901+0000 7f0184600640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f017c0d2010 0x7f017c1331b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:52.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.901+0000 7f0184600640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f017c145e40 0x7f017c1336f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:52.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.901+0000 7f0184600640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f017c0d0ee0 con 0x7f017c0d2010 2024-03-31T15:14:52.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.901+0000 7f0184600640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f017c0ce9e0 con 0x7f017c06f920 2024-03-31T15:14:52.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.901+0000 7f0184600640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f017c0d2df0 con 0x7f017c145e40 2024-03-31T15:14:52.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.901+0000 7f0181200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f017c0d2010 0x7f017c1331b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:52.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.901+0000 7f0181c00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f017c06f920 0x7f017c1370a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:52.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.901+0000 7f0181200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f017c0d2010 0x7f017c1331b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55544/0 (socket says 172.21.15.16:55544) 2024-03-31T15:14:52.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.901+0000 7f0181200640 1 -- 172.21.15.16:0/814809060 learned_addr learned my addr 172.21.15.16:0/814809060 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:14:52.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.901+0000 7f0182600640 1 --2- 172.21.15.16:0/814809060 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f017c145e40 0x7f017c1336f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:52.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.905+0000 7f0181200640 1 -- 172.21.15.16:0/814809060 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f017c145e40 msgr2=0x7f017c1336f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:52.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.905+0000 7f0181200640 1 --2- 172.21.15.16:0/814809060 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f017c145e40 0x7f017c1336f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:52.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.905+0000 7f0181200640 1 -- 172.21.15.16:0/814809060 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f017c06f920 msgr2=0x7f017c1370a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:52.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.905+0000 7f0181200640 1 --2- 172.21.15.16:0/814809060 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f017c06f920 0x7f017c1370a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:52.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.905+0000 7f0181200640 1 -- 172.21.15.16:0/814809060 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f017c133d20 con 0x7f017c0d2010 2024-03-31T15:14:52.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.905+0000 7f0181200640 1 --2- 172.21.15.16:0/814809060 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f017c0d2010 0x7f017c1331b0 secure :-1 s=READY pgs=1802 cs=0 l=1 rev1=1 crypto rx=0x7f016800e810 tx=0x7f016800ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:52.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.905+0000 7f0179600640 1 -- 172.21.15.16:0/814809060 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0168017070 con 0x7f017c0d2010 2024-03-31T15:14:52.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.905+0000 7f0179600640 1 -- 172.21.15.16:0/814809060 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f016800c8d0 con 0x7f017c0d2010 2024-03-31T15:14:52.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.905+0000 7f0179600640 1 -- 172.21.15.16:0/814809060 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f016800f040 con 0x7f017c0d2010 2024-03-31T15:14:52.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.905+0000 7f0184600640 1 -- 172.21.15.16:0/814809060 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f017c134000 con 0x7f017c0d2010 2024-03-31T15:14:52.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.905+0000 7f0184600640 1 -- 172.21.15.16:0/814809060 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f017c0b53a0 con 0x7f017c0d2010 2024-03-31T15:14:52.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.905+0000 7f0184600640 1 -- 172.21.15.16:0/814809060 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f017c069950 con 0x7f017c0d2010 2024-03-31T15:14:52.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.905+0000 7f0179600640 1 -- 172.21.15.16:0/814809060 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f016801a070 con 0x7f017c0d2010 2024-03-31T15:14:52.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.905+0000 7f0179600640 1 --2- 172.21.15.16:0/814809060 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f015c068380 0x7f015c06a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:52.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.905+0000 7f0179600640 1 -- 172.21.15.16:0/814809060 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(112..112 src has 1..112) ==== 5853+0+0 (secure 0 0 0) 0x7f0168086720 con 0x7f017c0d2010 2024-03-31T15:14:52.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.905+0000 7f0181c00640 1 --2- 172.21.15.16:0/814809060 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f015c068380 0x7f015c06a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:52.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.909+0000 7f0181c00640 1 --2- 172.21.15.16:0/814809060 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f015c068380 0x7f015c06a840 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f0170007fd0 tx=0x7f0170009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:52.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:52.909+0000 7f0179600640 1 -- 172.21.15.16:0/814809060 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f01680564d0 con 0x7f017c0d2010 2024-03-31T15:14:53.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.109+0000 7f0184600640 1 -- 172.21.15.16:0/814809060 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f017c15b390 con 0x7f017c0d2010 2024-03-31T15:14:53.104 INFO:tasks.ceph.mon.a.smithi016.stderr:2024-03-31T15:14:53.109+0000 7f7dcbc00640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-31T15:14:53.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.113+0000 7f0179600640 1 -- 172.21.15.16:0/814809060 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v106) ==== 161+0+0 (secure 0 0 0) 0x7f016805a7d0 con 0x7f017c0d2010 2024-03-31T15:14:53.106 INFO:teuthology.orchestra.run.smithi016.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-31T15:14:53.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.117+0000 7f0184600640 1 -- 172.21.15.16:0/814809060 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f015c068380 msgr2=0x7f015c06a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:53.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.117+0000 7f0184600640 1 --2- 172.21.15.16:0/814809060 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f015c068380 0x7f015c06a840 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f0170007fd0 tx=0x7f0170009040 comp rx=0 tx=0).stop 2024-03-31T15:14:53.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.117+0000 7f0184600640 1 -- 172.21.15.16:0/814809060 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f017c0d2010 msgr2=0x7f017c1331b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:53.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.117+0000 7f0184600640 1 --2- 172.21.15.16:0/814809060 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f017c0d2010 0x7f017c1331b0 secure :-1 s=READY pgs=1802 cs=0 l=1 rev1=1 crypto rx=0x7f016800e810 tx=0x7f016800ecd0 comp rx=0 tx=0).stop 2024-03-31T15:14:53.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.117+0000 7f0184600640 1 -- 172.21.15.16:0/814809060 shutdown_connections 2024-03-31T15:14:53.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.117+0000 7f0184600640 1 --2- 172.21.15.16:0/814809060 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f015c068380 0x7f015c06a840 unknown :-1 s=CLOSED pgs=761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:53.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.117+0000 7f0184600640 1 --2- 172.21.15.16:0/814809060 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f017c145e40 0x7f017c1336f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:53.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.117+0000 7f0184600640 1 --2- 172.21.15.16:0/814809060 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f017c0d2010 0x7f017c1331b0 unknown :-1 s=CLOSED pgs=1802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:53.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.117+0000 7f0184600640 1 --2- 172.21.15.16:0/814809060 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f017c06f920 0x7f017c1370a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:53.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.117+0000 7f0184600640 1 -- 172.21.15.16:0/814809060 >> 172.21.15.16:0/814809060 conn(0x7f017c067330 msgr2=0x7f017c0b3460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:14:53.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.117+0000 7f0184600640 1 -- 172.21.15.16:0/814809060 shutdown_connections 2024-03-31T15:14:53.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.117+0000 7f0184600640 1 -- 172.21.15.16:0/814809060 wait complete. 2024-03-31T15:14:53.118 INFO:tasks.ceph.mds.c.smithi016.stderr: -2151> 2024-03-31T15:14:45.725+0000 7ff49ee00640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-31T15:14:53.130 DEBUG:teuthology.orchestra.run.smithi016:> cephfs-journal-tool --debug-mds=20 --debug-ms=1 --debug-objecter=1 --rank cephfs:0 journal reset --force 2024-03-31T15:14:53.165 INFO:tasks.ceph.mds.c.smithi016.stdout:starting mds.c at 2024-03-31T15:14:53.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.185+0000 7f0fc794a400 1 -- start start 2024-03-31T15:14:53.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.185+0000 7f0fc794a400 1 Processor -- start 2024-03-31T15:14:53.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.185+0000 7f0fc794a400 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ac61aff800 0x55ac628c0000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:53.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.185+0000 7f0fc794a400 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ac61aff400 0x55ac628c0580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:53.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.185+0000 7f0fc794a400 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55ac61aff000 0x55ac628c0b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:53.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.185+0000 7f0fc794a400 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55ac628d8180 con 0x55ac61aff400 2024-03-31T15:14:53.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.185+0000 7f0fc794a400 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55ac628d8300 con 0x55ac61aff800 2024-03-31T15:14:53.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.185+0000 7f0fc794a400 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x55ac628d8480 con 0x55ac61aff000 2024-03-31T15:14:53.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.185+0000 7f0fc5000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ac61aff400 0x55ac628c0580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:53.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.185+0000 7f0fc5000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ac61aff400 0x55ac628c0580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55594/0 (socket says 172.21.15.16:55594) 2024-03-31T15:14:53.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.185+0000 7f0fc5000640 1 -- 172.21.15.16:0/2104435853 learned_addr learned my addr 172.21.15.16:0/2104435853 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:14:53.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.185+0000 7f0fc6400640 1 --2- 172.21.15.16:0/2104435853 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55ac61aff000 0x55ac628c0b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:53.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.185+0000 7f0fc5000640 1 -- 172.21.15.16:0/2104435853 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55ac61aff000 msgr2=0x55ac628c0b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:14:53.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.185+0000 7f0fc5a00640 1 --2- 172.21.15.16:0/2104435853 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ac61aff800 0x55ac628c0000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:53.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.185+0000 7f0fc5000640 1 --2- 172.21.15.16:0/2104435853 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55ac61aff000 0x55ac628c0b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:53.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.185+0000 7f0fc5000640 1 -- 172.21.15.16:0/2104435853 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ac61aff800 msgr2=0x55ac628c0000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:14:53.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.185+0000 7f0fc5000640 1 --2- 172.21.15.16:0/2104435853 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ac61aff800 0x55ac628c0000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:53.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.185+0000 7f0fc5000640 1 -- 172.21.15.16:0/2104435853 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ac628bab40 con 0x55ac61aff400 2024-03-31T15:14:53.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.185+0000 7f0fc6400640 1 --2- 172.21.15.16:0/2104435853 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55ac61aff000 0x55ac628c0b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:14:53.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.185+0000 7f0fc5a00640 1 --2- 172.21.15.16:0/2104435853 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ac61aff800 0x55ac628c0000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:14:53.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.185+0000 7f0fc5000640 1 --2- 172.21.15.16:0/2104435853 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ac61aff400 0x55ac628c0580 secure :-1 s=READY pgs=1806 cs=0 l=1 rev1=1 crypto rx=0x55ac6296e9c0 tx=0x55ac628cc000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c6e2497f490aa57 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:53.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.185+0000 7f0fc2800640 1 -- 172.21.15.16:0/2104435853 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x55ac61b349c0 con 0x55ac61aff400 2024-03-31T15:14:53.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.185+0000 7f0fc2800640 1 -- 172.21.15.16:0/2104435853 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x55ac61b4ae00 con 0x55ac61aff400 2024-03-31T15:14:53.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.185+0000 7f0fc2800640 1 -- 172.21.15.16:0/2104435853 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x55ac61b344e0 con 0x55ac61aff400 2024-03-31T15:14:53.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.185+0000 7f0fc794a400 1 -- 172.21.15.16:0/2104435853 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ac628bad20 con 0x55ac61aff400 2024-03-31T15:14:53.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.185+0000 7f0fc794a400 1 -- 172.21.15.16:0/2104435853 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({fsmap=0}) -- 0x55ac628baf00 con 0x55ac61aff400 2024-03-31T15:14:53.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.185+0000 7f0fc794a400 4 waiting for MDS map... 2024-03-31T15:14:53.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.185+0000 7f0fc2800640 1 -- 172.21.15.16:0/2104435853 <== mon.0 v2:172.21.15.16:3300/0 4 ==== osd_map(113..113 src has 1..113) ==== 5939+0+0 (secure 0 0 0) 0x55ac62a0c200 con 0x55ac61aff400 2024-03-31T15:14:53.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.185+0000 7f0fc2800640 1 -- 172.21.15.16:0/2104435853 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 106) ==== 2885+0+0 (secure 0 0 0) 0x55ac61b55180 con 0x55ac61aff400 2024-03-31T15:14:53.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.185+0000 7f0fc794a400 4 Got MDS map 106 2024-03-31T15:14:53.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.185+0000 7f0fc794a400 10 main: JournalTool::main 2024-03-31T15:14:53.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.185+0000 7f0fc794a400 4 main: JournalTool: connecting to RADOS... 2024-03-31T15:14:53.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.189+0000 7f0fc794a400 1 Processor -- start 2024-03-31T15:14:53.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.189+0000 7f0fc794a400 1 -- start start 2024-03-31T15:14:53.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.189+0000 7f0fc794a400 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55ac628e1c00 0x55ac628c1080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:53.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.189+0000 7f0fc794a400 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ac62a78000 0x55ac628c1600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:53.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.189+0000 7f0fc794a400 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ac62a78400 0x55ac628c1b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:53.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.189+0000 7f0fc794a400 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55ac628d8600 con 0x55ac62a78400 2024-03-31T15:14:53.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.189+0000 7f0fc794a400 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55ac628d8780 con 0x55ac62a78000 2024-03-31T15:14:53.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.189+0000 7f0fc794a400 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x55ac628d8900 con 0x55ac628e1c00 2024-03-31T15:14:53.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.189+0000 7f0fc5000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ac62a78400 0x55ac628c1b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:53.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.189+0000 7f0fc5000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ac62a78400 0x55ac628c1b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55604/0 (socket says 172.21.15.16:55604) 2024-03-31T15:14:53.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.189+0000 7f0fc5000640 1 -- 172.21.15.16:0/1026558648 learned_addr learned my addr 172.21.15.16:0/1026558648 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:14:53.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.189+0000 7f0fc5a00640 1 --2- 172.21.15.16:0/1026558648 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ac62a78000 0x55ac628c1600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:53.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.189+0000 7f0fc6400640 1 --2- 172.21.15.16:0/1026558648 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55ac628e1c00 0x55ac628c1080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:53.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.189+0000 7f0fc5000640 1 -- 172.21.15.16:0/1026558648 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55ac628e1c00 msgr2=0x55ac628c1080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:14:53.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.189+0000 7f0fc5000640 1 --2- 172.21.15.16:0/1026558648 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55ac628e1c00 0x55ac628c1080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:53.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.189+0000 7f0fc5000640 1 -- 172.21.15.16:0/1026558648 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ac62a78000 msgr2=0x55ac628c1600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:14:53.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.189+0000 7f0fc5000640 1 --2- 172.21.15.16:0/1026558648 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ac62a78000 0x55ac628c1600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:53.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.189+0000 7f0fc5000640 1 -- 172.21.15.16:0/1026558648 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ac628ba1e0 con 0x55ac62a78400 2024-03-31T15:14:53.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.189+0000 7f0fc5000640 1 --2- 172.21.15.16:0/1026558648 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ac62a78400 0x55ac628c1b80 secure :-1 s=READY pgs=1807 cs=0 l=1 rev1=1 crypto rx=0x55ac62a80150 tx=0x55ac628cc3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d82368b898e7ced server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:53.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.189+0000 7f0fc0000640 1 -- 172.21.15.16:0/1026558648 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x55ac61b34b60 con 0x55ac62a78400 2024-03-31T15:14:53.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.189+0000 7f0fc0000640 1 -- 172.21.15.16:0/1026558648 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x55ac61b4ac40 con 0x55ac62a78400 2024-03-31T15:14:53.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.189+0000 7f0fc0000640 1 -- 172.21.15.16:0/1026558648 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x55ac61b34340 con 0x55ac62a78400 2024-03-31T15:14:53.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.189+0000 7f0fc794a400 1 -- 172.21.15.16:0/1026558648 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ac62a78400 msgr2=0x55ac628c1b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:53.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.189+0000 7f0fc794a400 1 --2- 172.21.15.16:0/1026558648 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ac62a78400 0x55ac628c1b80 secure :-1 s=READY pgs=1807 cs=0 l=1 rev1=1 crypto rx=0x55ac62a80150 tx=0x55ac628cc3c0 comp rx=0 tx=0).stop 2024-03-31T15:14:53.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.189+0000 7f0fc794a400 1 -- 172.21.15.16:0/1026558648 shutdown_connections 2024-03-31T15:14:53.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.189+0000 7f0fc794a400 1 --2- 172.21.15.16:0/1026558648 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ac62a78400 0x55ac628c1b80 unknown :-1 s=CLOSED pgs=1807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:53.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.189+0000 7f0fc794a400 1 --2- 172.21.15.16:0/1026558648 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ac62a78000 0x55ac628c1600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:53.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.189+0000 7f0fc794a400 1 --2- 172.21.15.16:0/1026558648 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55ac628e1c00 0x55ac628c1080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:53.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.189+0000 7f0fc794a400 1 -- 172.21.15.16:0/1026558648 >> 172.21.15.16:0/1026558648 conn(0x55ac61afec00 msgr2=0x55ac61b2d800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:14:53.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.189+0000 7f0fc794a400 1 -- 172.21.15.16:0/1026558648 shutdown_connections 2024-03-31T15:14:53.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.189+0000 7f0fc794a400 1 -- 172.21.15.16:0/1026558648 wait complete. 2024-03-31T15:14:53.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.189+0000 7f0fc794a400 1 Processor -- start 2024-03-31T15:14:53.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.189+0000 7f0fc794a400 1 -- start start 2024-03-31T15:14:53.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.193+0000 7f0fc794a400 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ac628e1c00 0x55ac628c1080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:53.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.193+0000 7f0fc794a400 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ac62a78000 0x55ac628c1600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:53.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.193+0000 7f0fc794a400 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55ac62a78800 0x55ac628c2100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:53.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.193+0000 7f0fc794a400 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55ac628d9500 con 0x55ac628e1c00 2024-03-31T15:14:53.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.193+0000 7f0fc794a400 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55ac628d8a80 con 0x55ac62a78000 2024-03-31T15:14:53.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.193+0000 7f0fc794a400 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x55ac628d9680 con 0x55ac62a78800 2024-03-31T15:14:53.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.193+0000 7f0fc6400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ac628e1c00 0x55ac628c1080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:53.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.193+0000 7f0fc6400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ac628e1c00 0x55ac628c1080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55608/0 (socket says 172.21.15.16:55608) 2024-03-31T15:14:53.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.193+0000 7f0fc6400640 1 -- 172.21.15.16:0/849680242 learned_addr learned my addr 172.21.15.16:0/849680242 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:14:53.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.193+0000 7f0fc5a00640 1 --2- 172.21.15.16:0/849680242 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ac62a78000 0x55ac628c1600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:53.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.193+0000 7f0fc5000640 1 --2- 172.21.15.16:0/849680242 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55ac62a78800 0x55ac628c2100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:53.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.193+0000 7f0fc6400640 1 -- 172.21.15.16:0/849680242 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55ac62a78800 msgr2=0x55ac628c2100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:53.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.193+0000 7f0fc6400640 1 --2- 172.21.15.16:0/849680242 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55ac62a78800 0x55ac628c2100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:53.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.193+0000 7f0fc6400640 1 -- 172.21.15.16:0/849680242 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ac62a78000 msgr2=0x55ac628c1600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:53.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.193+0000 7f0fc6400640 1 --2- 172.21.15.16:0/849680242 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ac62a78000 0x55ac628c1600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:53.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.193+0000 7f0fc6400640 1 -- 172.21.15.16:0/849680242 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ac628bba40 con 0x55ac628e1c00 2024-03-31T15:14:53.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.193+0000 7f0fc5a00640 1 --2- 172.21.15.16:0/849680242 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ac62a78000 0x55ac628c1600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:14:53.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.193+0000 7f0fc6400640 1 --2- 172.21.15.16:0/849680242 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ac628e1c00 0x55ac628c1080 secure :-1 s=READY pgs=1808 cs=0 l=1 rev1=1 crypto rx=0x55ac62a81470 tx=0x55ac62a860c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:53.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.193+0000 7f0fbe200640 1 -- 172.21.15.16:0/849680242 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x55ac61b34b60 con 0x55ac628e1c00 2024-03-31T15:14:53.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.193+0000 7f0fbe200640 1 -- 172.21.15.16:0/849680242 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x55ac61b4aa80 con 0x55ac628e1c00 2024-03-31T15:14:53.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.193+0000 7f0fbe200640 1 -- 172.21.15.16:0/849680242 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x55ac61b34340 con 0x55ac628e1c00 2024-03-31T15:14:53.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.193+0000 7f0fc794a400 1 -- 172.21.15.16:0/849680242 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55ac628bbc20 con 0x55ac628e1c00 2024-03-31T15:14:53.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.193+0000 7f0fc794a400 1 -- 172.21.15.16:0/849680242 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ac628bba40 con 0x55ac628e1c00 2024-03-31T15:14:53.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.193+0000 7f0fc794a400 4 main: JournalTool: resolving pool 16 2024-03-31T15:14:53.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.193+0000 7f0fbe200640 1 -- 172.21.15.16:0/849680242 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x55ac62a76c00 con 0x55ac628e1c00 2024-03-31T15:14:53.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.193+0000 7f0fbe200640 1 --2- 172.21.15.16:0/849680242 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x55ac61df2c00 0x55ac628c3700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:53.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.197+0000 7f0fbe200640 1 -- 172.21.15.16:0/849680242 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 5939+0+0 (secure 0 0 0) 0x55ac61df0800 con 0x55ac628e1c00 2024-03-31T15:14:53.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.197+0000 7f0fc794a400 4 main: JournalTool: creating IoCtx.. 2024-03-31T15:14:53.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.197+0000 7f0fc794a400 4 main: Executing for rank 0 2024-03-31T15:14:53.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.197+0000 7f0fc5a00640 1 --2- 172.21.15.16:0/849680242 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x55ac61df2c00 0x55ac628c3700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:53.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.197+0000 7f0fc794a400 1 -- start start 2024-03-31T15:14:53.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.197+0000 7f0fc794a400 1 Processor -- start 2024-03-31T15:14:53.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.197+0000 7f0fc5a00640 1 --2- 172.21.15.16:0/849680242 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x55ac61df2c00 0x55ac628c3700 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x55ac61bec810 tx=0x55ac61ee0c00 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:53.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.197+0000 7f0fc794a400 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55ac62a79000 0x55ac628c2680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:53.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.197+0000 7f0fc794a400 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ac62a79400 0x55ac628c2c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:53.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.197+0000 7f0fc794a400 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ac62a79800 0x55ac628c3180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:53.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.197+0000 7f0fc794a400 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55ac628d9980 con 0x55ac62a79800 2024-03-31T15:14:53.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.197+0000 7f0fc794a400 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55ac628d9b00 con 0x55ac62a79400 2024-03-31T15:14:53.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.197+0000 7f0fc794a400 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x55ac628d9c80 con 0x55ac62a79000 2024-03-31T15:14:53.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.197+0000 7f0fc5000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ac62a79800 0x55ac628c3180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:53.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.197+0000 7f0fc5000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ac62a79800 0x55ac628c3180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55612/0 (socket says 172.21.15.16:55612) 2024-03-31T15:14:53.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.197+0000 7f0fc5000640 1 -- 172.21.15.16:0/469688837 learned_addr learned my addr 172.21.15.16:0/469688837 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:14:53.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.197+0000 7f0fc6400640 1 --2- 172.21.15.16:0/469688837 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55ac62a79000 0x55ac628c2680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:53.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.197+0000 7f0fc5a00640 1 --2- 172.21.15.16:0/469688837 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ac62a79400 0x55ac628c2c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:53.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.197+0000 7f0fc5000640 1 -- 172.21.15.16:0/469688837 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55ac62a79000 msgr2=0x55ac628c2680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:14:53.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.197+0000 7f0fc5000640 1 --2- 172.21.15.16:0/469688837 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55ac62a79000 0x55ac628c2680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:53.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.197+0000 7f0fc5000640 1 -- 172.21.15.16:0/469688837 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ac62a79400 msgr2=0x55ac628c2c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:14:53.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.197+0000 7f0fc5000640 1 --2- 172.21.15.16:0/469688837 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ac62a79400 0x55ac628c2c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:53.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.197+0000 7f0fc5000640 1 -- 172.21.15.16:0/469688837 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ac61f34780 con 0x55ac62a79800 2024-03-31T15:14:53.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.197+0000 7f0fc5a00640 1 --2- 172.21.15.16:0/469688837 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ac62a79400 0x55ac628c2c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:14:53.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.197+0000 7f0fc5000640 1 --2- 172.21.15.16:0/469688837 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ac62a79800 0x55ac628c3180 secure :-1 s=READY pgs=1809 cs=0 l=1 rev1=1 crypto rx=0x55ac62a803f0 tx=0x55ac628cc360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54ec78ed8e3e1c9 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:53.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.197+0000 7f0fba600640 1 -- 172.21.15.16:0/469688837 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x55ac61e2c4e0 con 0x55ac62a79800 2024-03-31T15:14:53.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.197+0000 7f0fba600640 1 -- 172.21.15.16:0/469688837 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x55ac61b4b340 con 0x55ac62a79800 2024-03-31T15:14:53.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.197+0000 7f0fba600640 1 -- 172.21.15.16:0/469688837 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x55ac61e2c340 con 0x55ac62a79800 2024-03-31T15:14:53.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.197+0000 7f0fc794a400 1 -- 172.21.15.16:0/469688837 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ac61f34960 con 0x55ac62a79800 2024-03-31T15:14:53.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.197+0000 7f0fc794a400 1 -- 172.21.15.16:0/469688837 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({fsmap=0}) -- 0x55ac61f34b40 con 0x55ac62a79800 2024-03-31T15:14:53.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.197+0000 7f0fc794a400 4 waiting for MDS map... 2024-03-31T15:14:53.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.197+0000 7f0fba600640 1 -- 172.21.15.16:0/469688837 <== mon.0 v2:172.21.15.16:3300/0 4 ==== osd_map(113..113 src has 1..113) ==== 5939+0+0 (secure 0 0 0) 0x55ac61f3c200 con 0x55ac62a79800 2024-03-31T15:14:53.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.197+0000 7f0fba600640 1 -- 172.21.15.16:0/469688837 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 106) ==== 2885+0+0 (secure 0 0 0) 0x55ac61b55500 con 0x55ac62a79800 2024-03-31T15:14:53.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.197+0000 7f0fc794a400 4 Got MDS map 106 2024-03-31T15:14:53.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.197+0000 7f0fc794a400 1 --2- 172.21.15.16:0/469688837 >> [v2:172.21.15.16:6800/932712244,v1:172.21.15.16:6801/932712244] conn(0x55ac61f42800 0x55ac61f5a000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:53.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.197+0000 7f0fc794a400 1 -- 172.21.15.16:0/469688837 --> [v2:172.21.15.16:6800/932712244,v1:172.21.15.16:6801/932712244] -- osd_op(unknown.0.0:1 16.f 16:f1f69726:::400.00000000:head [writefull 0~22 in=22b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55ac61f42c00 con 0x55ac61f42800 2024-03-31T15:14:53.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.201+0000 7f0fc6400640 1 --2- 172.21.15.16:0/469688837 >> [v2:172.21.15.16:6800/932712244,v1:172.21.15.16:6801/932712244] conn(0x55ac61f42800 0x55ac61f5a000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:53.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.201+0000 7f0fc6400640 1 --2- 172.21.15.16:0/469688837 >> [v2:172.21.15.16:6800/932712244,v1:172.21.15.16:6801/932712244] conn(0x55ac61f42800 0x55ac61f5a000 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=74ed1df3a2e00311 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:53.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.201+0000 7f0fc6400640 1 -- 172.21.15.16:0/469688837 <== osd.0 v2:172.21.15.16:6800/932712244 1 ==== osd_op_reply(1 400.00000000 [writefull 0~22] v113'2 uv2 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x55ac6287a000 con 0x55ac61f42800 2024-03-31T15:14:53.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.201+0000 7f0fc794a400 1 -- 172.21.15.16:0/469688837 --> [v2:172.21.15.16:6800/932712244,v1:172.21.15.16:6801/932712244] -- osd_op(unknown.0.0:2 16.14 16:292cf221:::200.00000000:head [writefull 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55ac61f43000 con 0x55ac61f42800 2024-03-31T15:14:53.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.205+0000 7f0fc6400640 1 -- 172.21.15.16:0/469688837 <== osd.0 v2:172.21.15.16:6800/932712244 2 ==== osd_op_reply(2 200.00000000 [writefull 0~90] v113'41 uv41 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x55ac6287a000 con 0x55ac61f42800 2024-03-31T15:14:53.198 INFO:teuthology.orchestra.run.smithi016.stdout:writing EResetJournal entry 2024-03-31T15:14:53.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.205+0000 7f0fc794a400 1 -- 172.21.15.16:0/469688837 --> [v2:172.21.15.16:6800/932712244,v1:172.21.15.16:6801/932712244] -- osd_op(unknown.0.0:3 16.14 16:2e2fa760:::200.00000001:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55ac61f43400 con 0x55ac61f42800 2024-03-31T15:14:53.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.205+0000 7f0fc794a400 1 --2- 172.21.15.16:0/469688837 >> [v2:172.21.15.89:6810/50395527,v1:172.21.15.89:6814/50395527] conn(0x55ac61f43800 0x55ac61f5a580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:53.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.205+0000 7f0fc794a400 1 -- 172.21.15.16:0/469688837 --> [v2:172.21.15.89:6810/50395527,v1:172.21.15.89:6814/50395527] -- osd_op(unknown.0.0:4 16.1b 16:ddb4e4d7:::200.00000002:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55ac61f43c00 con 0x55ac61f43800 2024-03-31T15:14:53.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.205+0000 7f0fc794a400 1 --2- 172.21.15.16:0/469688837 >> [v2:172.21.15.16:6808/1398737533,v1:172.21.15.16:6810/1398737533] conn(0x55ac61f42c00 0x55ac61f5ab00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:53.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.205+0000 7f0fc794a400 1 -- 172.21.15.16:0/469688837 --> [v2:172.21.15.16:6808/1398737533,v1:172.21.15.16:6810/1398737533] -- osd_op(unknown.0.0:5 16.b 16:d5c7a900:::200.00000003:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55ac61f43000 con 0x55ac61f42c00 2024-03-31T15:14:53.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.205+0000 7f0fc794a400 1 -- 172.21.15.16:0/469688837 --> [v2:172.21.15.89:6810/50395527,v1:172.21.15.89:6814/50395527] -- osd_op(unknown.0.0:6 16.1f 16:f83d65d8:::200.00000004:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55ac61f43400 con 0x55ac61f43800 2024-03-31T15:14:53.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.205+0000 7f0fc794a400 1 --2- 172.21.15.16:0/469688837 >> [v2:172.21.15.16:6809/3360352974,v1:172.21.15.16:6812/3360352974] conn(0x55ac61df2800 0x55ac61f5b080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:53.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.205+0000 7f0fc794a400 1 -- 172.21.15.16:0/469688837 --> [v2:172.21.15.16:6809/3360352974,v1:172.21.15.16:6812/3360352974] -- osd_op(unknown.0.0:7 16.1a 16:5e41ab69:::200.00000005:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55ac61df2400 con 0x55ac61df2800 2024-03-31T15:14:53.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.205+0000 7f0fc794a400 1 --2- 172.21.15.16:0/469688837 >> [v2:172.21.15.89:6808/1796778630,v1:172.21.15.89:6811/1796778630] conn(0x55ac61f6e000 0x55ac61f5b600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:53.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.205+0000 7f0fc794a400 1 -- 172.21.15.16:0/469688837 --> [v2:172.21.15.89:6808/1796778630,v1:172.21.15.89:6811/1796778630] -- osd_op(unknown.0.0:8 16.1e 16:7bf7e7b5:::200.00000006:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55ac61f6e400 con 0x55ac61f6e000 2024-03-31T15:14:53.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.205+0000 7f0fc6400640 1 --2- 172.21.15.16:0/469688837 >> [v2:172.21.15.16:6808/1398737533,v1:172.21.15.16:6810/1398737533] conn(0x55ac61f42c00 0x55ac61f5ab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:53.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.205+0000 7f0fc5a00640 1 --2- 172.21.15.16:0/469688837 >> [v2:172.21.15.89:6810/50395527,v1:172.21.15.89:6814/50395527] conn(0x55ac61f43800 0x55ac61f5a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:53.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.205+0000 7f0fc5a00640 1 --2- 172.21.15.16:0/469688837 >> [v2:172.21.15.16:6809/3360352974,v1:172.21.15.16:6812/3360352974] conn(0x55ac61df2800 0x55ac61f5b080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:53.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.205+0000 7f0fc5000640 1 --2- 172.21.15.16:0/469688837 >> [v2:172.21.15.89:6808/1796778630,v1:172.21.15.89:6811/1796778630] conn(0x55ac61f6e000 0x55ac61f5b600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:53.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.205+0000 7f0fc6400640 1 --2- 172.21.15.16:0/469688837 >> [v2:172.21.15.16:6808/1398737533,v1:172.21.15.16:6810/1398737533] conn(0x55ac61f42c00 0x55ac61f5ab00 crc :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=30534ce3986d5921 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:53.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.205+0000 7f0fc5a00640 1 --2- 172.21.15.16:0/469688837 >> [v2:172.21.15.16:6809/3360352974,v1:172.21.15.16:6812/3360352974] conn(0x55ac61df2800 0x55ac61f5b080 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=f9af32aa5395b709 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:53.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.205+0000 7f0fc5000640 1 --2- 172.21.15.16:0/469688837 >> [v2:172.21.15.89:6808/1796778630,v1:172.21.15.89:6811/1796778630] conn(0x55ac61f6e000 0x55ac61f5b600 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=b916efa8bd478ce6 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:53.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.205+0000 7f0fc5a00640 1 --2- 172.21.15.16:0/469688837 >> [v2:172.21.15.89:6810/50395527,v1:172.21.15.89:6814/50395527] conn(0x55ac61f43800 0x55ac61f5a580 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=40e99b53ce03e9d1 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:53.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.205+0000 7f0fc6400640 1 -- 172.21.15.16:0/469688837 <== osd.0 v2:172.21.15.16:6800/932712244 3 ==== osd_op_reply(3 200.00000001 [delete] v113'42 uv42 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x55ac6287a000 con 0x55ac61f42800 2024-03-31T15:14:53.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.205+0000 7f0fc6400640 1 -- 172.21.15.16:0/469688837 <== osd.1 v2:172.21.15.16:6808/1398737533 1 ==== osd_op_reply(5 200.00000003 [delete] v113'6 uv5 ondisk = -2 ((2) No such file or directory)) ==== 156+0+0 (crc 0 0 0) 0x55ac6287a000 con 0x55ac61f42c00 2024-03-31T15:14:53.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.205+0000 7f0fc5a00640 1 -- 172.21.15.16:0/469688837 <== osd.5 v2:172.21.15.89:6810/50395527 1 ==== osd_op_reply(4 200.00000002 [delete] v113'3 uv0 ondisk = -2 ((2) No such file or directory)) ==== 156+0+0 (crc 0 0 0) 0x55ac6287a500 con 0x55ac61f43800 2024-03-31T15:14:53.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.205+0000 7f0fc5a00640 1 -- 172.21.15.16:0/469688837 <== osd.5 v2:172.21.15.89:6810/50395527 2 ==== osd_op_reply(6 200.00000004 [delete] v113'9 uv8 ondisk = -2 ((2) No such file or directory)) ==== 156+0+0 (crc 0 0 0) 0x55ac6287a500 con 0x55ac61f43800 2024-03-31T15:14:53.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.205+0000 7f0fb8800640 1 -- 172.21.15.16:0/469688837 --> [v2:172.21.15.16:6800/932712244,v1:172.21.15.16:6801/932712244] -- osd_op(unknown.0.0:9 16.14 16:2e2fa760:::200.00000001:head [write 0~48 in=48b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55ac61df3800 con 0x55ac61f42800 2024-03-31T15:14:53.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.205+0000 7f0fc5a00640 1 -- 172.21.15.16:0/469688837 <== osd.3 v2:172.21.15.16:6809/3360352974 1 ==== osd_op_reply(7 200.00000005 [delete] v113'3 uv0 ondisk = -2 ((2) No such file or directory)) ==== 156+0+0 (crc 0 0 0) 0x55ac6287a500 con 0x55ac61df2800 2024-03-31T15:14:53.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.205+0000 7f0fc5000640 1 -- 172.21.15.16:0/469688837 <== osd.7 v2:172.21.15.89:6808/1796778630 1 ==== osd_op_reply(8 200.00000006 [delete] v113'3 uv0 ondisk = -2 ((2) No such file or directory)) ==== 156+0+0 (crc 0 0 0) 0x55ac6287a780 con 0x55ac61f6e000 2024-03-31T15:14:53.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.209+0000 7f0fc6400640 1 -- 172.21.15.16:0/469688837 <== osd.0 v2:172.21.15.16:6800/932712244 4 ==== osd_op_reply(9 200.00000001 [write 0~48] v113'43 uv43 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x55ac6287a000 con 0x55ac61f42800 2024-03-31T15:14:53.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.209+0000 7f0fc794a400 4 Successfully wrote new journal pointer and header for rank 8:0 2024-03-31T15:14:53.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.209+0000 7f0fc794a400 1 -- 172.21.15.16:0/469688837 >> [v2:172.21.15.16:6800/932712244,v1:172.21.15.16:6801/932712244] conn(0x55ac61f42800 msgr2=0x55ac61f5a000 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:53.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.209+0000 7f0fc794a400 1 --2- 172.21.15.16:0/469688837 >> [v2:172.21.15.16:6800/932712244,v1:172.21.15.16:6801/932712244] conn(0x55ac61f42800 0x55ac61f5a000 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:53.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.209+0000 7f0fc794a400 1 -- 172.21.15.16:0/469688837 >> [v2:172.21.15.16:6808/1398737533,v1:172.21.15.16:6810/1398737533] conn(0x55ac61f42c00 msgr2=0x55ac61f5ab00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:53.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.209+0000 7f0fc794a400 1 --2- 172.21.15.16:0/469688837 >> [v2:172.21.15.16:6808/1398737533,v1:172.21.15.16:6810/1398737533] conn(0x55ac61f42c00 0x55ac61f5ab00 crc :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:53.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.209+0000 7f0fc794a400 1 -- 172.21.15.16:0/469688837 >> [v2:172.21.15.16:6809/3360352974,v1:172.21.15.16:6812/3360352974] conn(0x55ac61df2800 msgr2=0x55ac61f5b080 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:53.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.209+0000 7f0fc794a400 1 --2- 172.21.15.16:0/469688837 >> [v2:172.21.15.16:6809/3360352974,v1:172.21.15.16:6812/3360352974] conn(0x55ac61df2800 0x55ac61f5b080 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:53.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.209+0000 7f0fc794a400 1 -- 172.21.15.16:0/469688837 >> [v2:172.21.15.89:6810/50395527,v1:172.21.15.89:6814/50395527] conn(0x55ac61f43800 msgr2=0x55ac61f5a580 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:53.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.209+0000 7f0fc794a400 1 --2- 172.21.15.16:0/469688837 >> [v2:172.21.15.89:6810/50395527,v1:172.21.15.89:6814/50395527] conn(0x55ac61f43800 0x55ac61f5a580 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:53.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.209+0000 7f0fc794a400 1 -- 172.21.15.16:0/469688837 >> [v2:172.21.15.89:6808/1796778630,v1:172.21.15.89:6811/1796778630] conn(0x55ac61f6e000 msgr2=0x55ac61f5b600 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:53.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.209+0000 7f0fc794a400 1 --2- 172.21.15.16:0/469688837 >> [v2:172.21.15.89:6808/1796778630,v1:172.21.15.89:6811/1796778630] conn(0x55ac61f6e000 0x55ac61f5b600 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:53.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.209+0000 7f0fc5000640 1 -- 172.21.15.16:0/469688837 reap_dead start 2024-03-31T15:14:53.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.209+0000 7f0fc5000640 1 -- 172.21.15.16:0/469688837 reap_dead start 2024-03-31T15:14:53.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.209+0000 7f0fc794a400 1 -- 172.21.15.16:0/469688837 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ac62a79800 msgr2=0x55ac628c3180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:53.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.209+0000 7f0fc794a400 1 --2- 172.21.15.16:0/469688837 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ac62a79800 0x55ac628c3180 secure :-1 s=READY pgs=1809 cs=0 l=1 rev1=1 crypto rx=0x55ac62a803f0 tx=0x55ac628cc360 comp rx=0 tx=0).stop 2024-03-31T15:14:53.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.209+0000 7f0fc794a400 1 -- 172.21.15.16:0/469688837 shutdown_connections 2024-03-31T15:14:53.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.209+0000 7f0fc794a400 1 --2- 172.21.15.16:0/469688837 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ac62a79800 0x55ac628c3180 unknown :-1 s=CLOSED pgs=1809 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:53.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.209+0000 7f0fc794a400 1 -- 172.21.15.16:0/469688837 >> 172.21.15.16:0/469688837 conn(0x55ac62a78c00 msgr2=0x55ac61f2a000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:14:53.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.209+0000 7f0fc794a400 1 -- 172.21.15.16:0/469688837 shutdown_connections 2024-03-31T15:14:53.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.209+0000 7f0fc794a400 1 -- 172.21.15.16:0/469688837 wait complete. 2024-03-31T15:14:53.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.209+0000 7f0fc794a400 1 -- 172.21.15.16:0/849680242 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x55ac61df2c00 msgr2=0x55ac628c3700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:53.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.209+0000 7f0fc794a400 1 --2- 172.21.15.16:0/849680242 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x55ac61df2c00 0x55ac628c3700 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x55ac61bec810 tx=0x55ac61ee0c00 comp rx=0 tx=0).stop 2024-03-31T15:14:53.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.209+0000 7f0fc794a400 1 -- 172.21.15.16:0/849680242 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ac628e1c00 msgr2=0x55ac628c1080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:53.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.209+0000 7f0fc794a400 1 --2- 172.21.15.16:0/849680242 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ac628e1c00 0x55ac628c1080 secure :-1 s=READY pgs=1808 cs=0 l=1 rev1=1 crypto rx=0x55ac62a81470 tx=0x55ac62a860c0 comp rx=0 tx=0).stop 2024-03-31T15:14:53.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.209+0000 7f0fc794a400 1 -- 172.21.15.16:0/849680242 shutdown_connections 2024-03-31T15:14:53.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.209+0000 7f0fc794a400 1 --2- 172.21.15.16:0/849680242 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x55ac61df2c00 0x55ac628c3700 unknown :-1 s=CLOSED pgs=762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:53.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.209+0000 7f0fc794a400 1 --2- 172.21.15.16:0/849680242 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55ac62a78800 0x55ac628c2100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:53.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.209+0000 7f0fc794a400 1 --2- 172.21.15.16:0/849680242 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ac62a78000 0x55ac628c1600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:53.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.209+0000 7f0fc794a400 1 --2- 172.21.15.16:0/849680242 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ac628e1c00 0x55ac628c1080 unknown :-1 s=CLOSED pgs=1808 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:53.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.209+0000 7f0fc794a400 1 -- 172.21.15.16:0/849680242 >> 172.21.15.16:0/849680242 conn(0x55ac61afec00 msgr2=0x55ac61b2d800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:14:53.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.209+0000 7f0fc794a400 1 -- 172.21.15.16:0/849680242 shutdown_connections 2024-03-31T15:14:53.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.209+0000 7f0fc794a400 1 -- 172.21.15.16:0/849680242 wait complete. 2024-03-31T15:14:53.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.209+0000 7f0fc794a400 1 -- 172.21.15.16:0/2104435853 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ac61aff400 msgr2=0x55ac628c0580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:53.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.209+0000 7f0fc794a400 1 --2- 172.21.15.16:0/2104435853 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ac61aff400 0x55ac628c0580 secure :-1 s=READY pgs=1806 cs=0 l=1 rev1=1 crypto rx=0x55ac6296e9c0 tx=0x55ac628cc000 comp rx=0 tx=0).stop 2024-03-31T15:14:53.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.209+0000 7f0fc794a400 1 -- 172.21.15.16:0/2104435853 shutdown_connections 2024-03-31T15:14:53.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.209+0000 7f0fc794a400 1 --2- 172.21.15.16:0/2104435853 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x55ac61aff000 0x55ac628c0b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:53.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.209+0000 7f0fc794a400 1 --2- 172.21.15.16:0/2104435853 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ac61aff400 0x55ac628c0580 unknown :-1 s=CLOSED pgs=1806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:53.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.209+0000 7f0fc794a400 1 --2- 172.21.15.16:0/2104435853 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ac61aff800 0x55ac628c0000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:53.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.209+0000 7f0fc794a400 1 -- 172.21.15.16:0/2104435853 >> 172.21.15.16:0/2104435853 conn(0x55ac61affc00 msgr2=0x55ac61b2d000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:14:53.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.209+0000 7f0fc794a400 1 -- 172.21.15.16:0/2104435853 shutdown_connections 2024-03-31T15:14:53.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.209+0000 7f0fc794a400 1 -- 172.21.15.16:0/2104435853 wait complete. 2024-03-31T15:14:53.209 DEBUG:tasks.cephfs.filesystem:Ran ['cephfs-journal-tool', '--debug-mds=20', '--debug-ms=1', '--debug-objecter=1', '--rank', 'cephfs:0', 'journal', 'reset', '--force'] in time 0:00:00.078702, result: writing EResetJournal entry 2024-03-31T15:14:53.209 DEBUG:teuthology.orchestra.run.smithi016:> cephfs-data-scan --debug-mds=20 --debug-ms=1 --debug-objecter=1 scan_extents cephfs_data 2024-03-31T15:14:53.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.305+0000 7f3a69c498c0 1 -- start start 2024-03-31T15:14:53.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.305+0000 7f3a69c498c0 1 Processor -- start 2024-03-31T15:14:53.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.305+0000 7f3a69c498c0 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561bdb895000 0x561bdc65e000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:53.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.305+0000 7f3a69c498c0 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x561bdb895400 0x561bdc65e580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:53.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.305+0000 7f3a69c498c0 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561bdb895800 0x561bdc65eb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:53.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.305+0000 7f3a69c498c0 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x561bdc67a180 con 0x561bdb895000 2024-03-31T15:14:53.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.305+0000 7f3a69c498c0 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x561bdc67a300 con 0x561bdb895800 2024-03-31T15:14:53.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.305+0000 7f3a69c498c0 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x561bdc67a480 con 0x561bdb895400 2024-03-31T15:14:53.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.305+0000 7f3a67000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561bdb895000 0x561bdc65e000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:53.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.305+0000 7f3a67000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561bdb895000 0x561bdc65e000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55640/0 (socket says 172.21.15.16:55640) 2024-03-31T15:14:53.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.305+0000 7f3a67000640 1 -- 172.21.15.16:0/2760865108 learned_addr learned my addr 172.21.15.16:0/2760865108 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:14:53.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.305+0000 7f3a67a00640 1 --2- 172.21.15.16:0/2760865108 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561bdb895800 0x561bdc65eb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:53.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.305+0000 7f3a66600640 1 --2- 172.21.15.16:0/2760865108 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x561bdb895400 0x561bdc65e580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:53.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.305+0000 7f3a67000640 1 -- 172.21.15.16:0/2760865108 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x561bdb895400 msgr2=0x561bdc65e580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:14:53.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.305+0000 7f3a67000640 1 --2- 172.21.15.16:0/2760865108 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x561bdb895400 0x561bdc65e580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:53.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.305+0000 7f3a67000640 1 -- 172.21.15.16:0/2760865108 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561bdb895800 msgr2=0x561bdc65eb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:14:53.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.305+0000 7f3a67000640 1 --2- 172.21.15.16:0/2760865108 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561bdb895800 0x561bdc65eb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:53.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.305+0000 7f3a67000640 1 -- 172.21.15.16:0/2760865108 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561bdb8ba1e0 con 0x561bdb895000 2024-03-31T15:14:53.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.305+0000 7f3a67000640 1 --2- 172.21.15.16:0/2760865108 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561bdb895000 0x561bdc65e000 secure :-1 s=READY pgs=1811 cs=0 l=1 rev1=1 crypto rx=0x561bdc6fda70 tx=0x561bdc674600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c572bf714b2584af server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:53.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.305+0000 7f3a63e00640 1 -- 172.21.15.16:0/2760865108 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x561bdb8ca4e0 con 0x561bdb895000 2024-03-31T15:14:53.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.305+0000 7f3a63e00640 1 -- 172.21.15.16:0/2760865108 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x561bdb8e0e00 con 0x561bdb895000 2024-03-31T15:14:53.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.305+0000 7f3a63e00640 1 -- 172.21.15.16:0/2760865108 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x561bdb8cab60 con 0x561bdb895000 2024-03-31T15:14:53.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.305+0000 7f3a69c498c0 1 -- 172.21.15.16:0/2760865108 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561bdb8bb2c0 con 0x561bdb895000 2024-03-31T15:14:53.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.305+0000 7f3a69c498c0 1 -- 172.21.15.16:0/2760865108 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({fsmap=0}) -- 0x561bdb8bba40 con 0x561bdb895000 2024-03-31T15:14:53.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.305+0000 7f3a69c498c0 4 waiting for MDS map... 2024-03-31T15:14:53.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.305+0000 7f3a63e00640 1 -- 172.21.15.16:0/2760865108 <== mon.0 v2:172.21.15.16:3300/0 4 ==== osd_map(113..113 src has 1..113) ==== 5939+0+0 (secure 0 0 0) 0x561bdc663000 con 0x561bdb895000 2024-03-31T15:14:53.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.305+0000 7f3a63e00640 1 -- 172.21.15.16:0/2760865108 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 106) ==== 2885+0+0 (secure 0 0 0) 0x561bdb8eb180 con 0x561bdb895000 2024-03-31T15:14:53.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.305+0000 7f3a69c498c0 4 Got MDS map 106 2024-03-31T15:14:53.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.305+0000 7f3a69c498c0 4 datascan.main: Using metadata pool output 2024-03-31T15:14:53.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.305+0000 7f3a69c498c0 4 datascan.main: connecting to RADOS... 2024-03-31T15:14:53.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.305+0000 7f3a69c498c0 1 Processor -- start 2024-03-31T15:14:53.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.305+0000 7f3a69c498c0 1 -- start start 2024-03-31T15:14:53.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.309+0000 7f3a69c498c0 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561bdc7d1400 0x561bdc65f080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:53.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.309+0000 7f3a69c498c0 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561bdc7d1800 0x561bdc65f600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:53.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.309+0000 7f3a69c498c0 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x561bdc7d1c00 0x561bdc65fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:53.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.309+0000 7f3a69c498c0 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x561bdc67a600 con 0x561bdc7d1400 2024-03-31T15:14:53.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.309+0000 7f3a69c498c0 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x561bdc67a780 con 0x561bdc7d1800 2024-03-31T15:14:53.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.309+0000 7f3a69c498c0 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x561bdc67a900 con 0x561bdc7d1c00 2024-03-31T15:14:53.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.309+0000 7f3a67a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561bdc7d1400 0x561bdc65f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:53.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.309+0000 7f3a67a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561bdc7d1400 0x561bdc65f080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55650/0 (socket says 172.21.15.16:55650) 2024-03-31T15:14:53.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.309+0000 7f3a67a00640 1 -- 172.21.15.16:0/955509804 learned_addr learned my addr 172.21.15.16:0/955509804 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:14:53.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.309+0000 7f3a67000640 1 --2- 172.21.15.16:0/955509804 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561bdc7d1800 0x561bdc65f600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:53.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.309+0000 7f3a66600640 1 --2- 172.21.15.16:0/955509804 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x561bdc7d1c00 0x561bdc65fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:53.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.309+0000 7f3a67a00640 1 -- 172.21.15.16:0/955509804 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x561bdc7d1c00 msgr2=0x561bdc65fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:14:53.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.309+0000 7f3a67a00640 1 --2- 172.21.15.16:0/955509804 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x561bdc7d1c00 0x561bdc65fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:53.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.309+0000 7f3a67a00640 1 -- 172.21.15.16:0/955509804 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561bdc7d1800 msgr2=0x561bdc65f600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:14:53.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.309+0000 7f3a67a00640 1 --2- 172.21.15.16:0/955509804 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561bdc7d1800 0x561bdc65f600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:53.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.309+0000 7f3a67a00640 1 -- 172.21.15.16:0/955509804 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561bdb8bbc20 con 0x561bdc7d1400 2024-03-31T15:14:53.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.309+0000 7f3a67a00640 1 --2- 172.21.15.16:0/955509804 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561bdc7d1400 0x561bdc65f080 secure :-1 s=READY pgs=1812 cs=0 l=1 rev1=1 crypto rx=0x561bdc6fcff0 tx=0x561bdc674000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b29851679206a2 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:53.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.309+0000 7f3a61600640 1 -- 172.21.15.16:0/955509804 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x561bdb8ca9c0 con 0x561bdc7d1400 2024-03-31T15:14:53.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.309+0000 7f3a61600640 1 -- 172.21.15.16:0/955509804 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x561bdb8e0c40 con 0x561bdc7d1400 2024-03-31T15:14:53.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.309+0000 7f3a61600640 1 -- 172.21.15.16:0/955509804 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x561bdb8ca4e0 con 0x561bdc7d1400 2024-03-31T15:14:53.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.309+0000 7f3a69c498c0 1 -- 172.21.15.16:0/955509804 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561bdc7d1400 msgr2=0x561bdc65f080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:53.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.309+0000 7f3a69c498c0 1 --2- 172.21.15.16:0/955509804 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561bdc7d1400 0x561bdc65f080 secure :-1 s=READY pgs=1812 cs=0 l=1 rev1=1 crypto rx=0x561bdc6fcff0 tx=0x561bdc674000 comp rx=0 tx=0).stop 2024-03-31T15:14:53.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.309+0000 7f3a69c498c0 1 -- 172.21.15.16:0/955509804 shutdown_connections 2024-03-31T15:14:53.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.309+0000 7f3a69c498c0 1 --2- 172.21.15.16:0/955509804 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x561bdc7d1c00 0x561bdc65fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:53.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.309+0000 7f3a69c498c0 1 --2- 172.21.15.16:0/955509804 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561bdc7d1800 0x561bdc65f600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:53.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.309+0000 7f3a69c498c0 1 --2- 172.21.15.16:0/955509804 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561bdc7d1400 0x561bdc65f080 unknown :-1 s=CLOSED pgs=1812 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:53.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.309+0000 7f3a69c498c0 1 -- 172.21.15.16:0/955509804 >> 172.21.15.16:0/955509804 conn(0x561bdc7d1000 msgr2=0x561bdb8c2000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:14:53.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.309+0000 7f3a69c498c0 1 -- 172.21.15.16:0/955509804 shutdown_connections 2024-03-31T15:14:53.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.309+0000 7f3a69c498c0 1 -- 172.21.15.16:0/955509804 wait complete. 2024-03-31T15:14:53.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.309+0000 7f3a69c498c0 1 Processor -- start 2024-03-31T15:14:53.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.309+0000 7f3a69c498c0 1 -- start start 2024-03-31T15:14:53.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.309+0000 7f3a69c498c0 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561bdc7d1400 0x561bdc65f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:53.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.309+0000 7f3a69c498c0 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561bdc7d1800 0x561bdc65f600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:53.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.309+0000 7f3a69c498c0 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x561bdc7d1c00 0x561bdc65fb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:53.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.309+0000 7f3a69c498c0 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x561bdc67bb00 con 0x561bdc7d1400 2024-03-31T15:14:53.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.309+0000 7f3a69c498c0 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x561bdc67b980 con 0x561bdc7d1800 2024-03-31T15:14:53.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.309+0000 7f3a69c498c0 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x561bdc67b800 con 0x561bdc7d1c00 2024-03-31T15:14:53.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.309+0000 7f3a67a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561bdc7d1400 0x561bdc65f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:53.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.309+0000 7f3a67a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561bdc7d1400 0x561bdc65f080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55652/0 (socket says 172.21.15.16:55652) 2024-03-31T15:14:53.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.313+0000 7f3a67a00640 1 -- 172.21.15.16:0/2337957025 learned_addr learned my addr 172.21.15.16:0/2337957025 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:14:53.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.313+0000 7f3a67000640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561bdc7d1800 0x561bdc65f600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:53.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.313+0000 7f3a66600640 1 --2- 172.21.15.16:0/2337957025 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x561bdc7d1c00 0x561bdc65fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:53.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.313+0000 7f3a67a00640 1 -- 172.21.15.16:0/2337957025 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x561bdc7d1c00 msgr2=0x561bdc65fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:53.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.313+0000 7f3a67a00640 1 --2- 172.21.15.16:0/2337957025 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x561bdc7d1c00 0x561bdc65fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:53.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.313+0000 7f3a67a00640 1 -- 172.21.15.16:0/2337957025 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561bdc7d1800 msgr2=0x561bdc65f600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:53.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.313+0000 7f3a67a00640 1 --2- 172.21.15.16:0/2337957025 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561bdc7d1800 0x561bdc65f600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:53.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.313+0000 7f3a67a00640 1 -- 172.21.15.16:0/2337957025 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561bdb8bb4a0 con 0x561bdc7d1400 2024-03-31T15:14:53.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.313+0000 7f3a67000640 1 --2- 172.21.15.16:0/2337957025 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561bdc7d1800 0x561bdc65f600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:14:53.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.313+0000 7f3a67a00640 1 --2- 172.21.15.16:0/2337957025 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561bdc7d1400 0x561bdc65f080 secure :-1 s=READY pgs=1813 cs=0 l=1 rev1=1 crypto rx=0x561bdc7d21b0 tx=0x561bdc674420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:53.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.313+0000 7f3a5f800640 1 -- 172.21.15.16:0/2337957025 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x561bdb8cab60 con 0x561bdc7d1400 2024-03-31T15:14:53.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.313+0000 7f3a5f800640 1 -- 172.21.15.16:0/2337957025 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x561bdb8e0a80 con 0x561bdc7d1400 2024-03-31T15:14:53.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.313+0000 7f3a69c498c0 1 -- 172.21.15.16:0/2337957025 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x561bdb8ba3c0 con 0x561bdc7d1400 2024-03-31T15:14:53.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.313+0000 7f3a5f800640 1 -- 172.21.15.16:0/2337957025 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x561bdb8ca000 con 0x561bdc7d1400 2024-03-31T15:14:53.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.313+0000 7f3a69c498c0 1 -- 172.21.15.16:0/2337957025 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561bdb8bbc20 con 0x561bdc7d1400 2024-03-31T15:14:53.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.313+0000 7f3a69c498c0 4 datascan.init: resolving metadata pool 16 2024-03-31T15:14:53.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.313+0000 7f3a5f800640 1 -- 172.21.15.16:0/2337957025 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x561bdbb80c00 con 0x561bdc7d1400 2024-03-31T15:14:53.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.313+0000 7f3a5f800640 1 --2- 172.21.15.16:0/2337957025 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x561bdbc82000 0x561bdc660100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:53.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.313+0000 7f3a5f800640 1 -- 172.21.15.16:0/2337957025 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 5939+0+0 (secure 0 0 0) 0x561bdbb98400 con 0x561bdc7d1400 2024-03-31T15:14:53.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.313+0000 7f3a69c498c0 4 datascan.init: found metadata pool 'cephfs_metadata' 2024-03-31T15:14:53.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.313+0000 7f3a69c498c0 4 datascan.main: data pool 'cephfs_data' has ID 17 2024-03-31T15:14:53.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.313+0000 7f3a69c498c0 4 datascan.main: opening data pool 'cephfs_data' 2024-03-31T15:14:53.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.313+0000 7f3a69c498c0 1 --2- 172.21.15.16:0/2337957025 >> [v2:172.21.15.89:6808/1796778630,v1:172.21.15.89:6811/1796778630] conn(0x561bdbc82400 0x561bdc660680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:53.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.313+0000 7f3a69c498c0 1 -- 172.21.15.16:0/2337957025 --> [v2:172.21.15.89:6808/1796778630,v1:172.21.15.89:6811/1796778630] -- osd_op(unknown.0.0:1 17.0 17:00000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x561bdbc82800 con 0x561bdbc82400 2024-03-31T15:14:53.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.313+0000 7f3a67000640 1 --2- 172.21.15.16:0/2337957025 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x561bdbc82000 0x561bdc660100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:53.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.313+0000 7f3a66600640 1 --2- 172.21.15.16:0/2337957025 >> [v2:172.21.15.89:6808/1796778630,v1:172.21.15.89:6811/1796778630] conn(0x561bdbc82400 0x561bdc660680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:53.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.317+0000 7f3a67000640 1 --2- 172.21.15.16:0/2337957025 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x561bdbc82000 0x561bdc660100 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x561bdc7d22d0 tx=0x561bdc674180 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:53.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.317+0000 7f3a66600640 1 --2- 172.21.15.16:0/2337957025 >> [v2:172.21.15.89:6808/1796778630,v1:172.21.15.89:6811/1796778630] conn(0x561bdbc82400 0x561bdc660680 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:53.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.317+0000 7f3a66600640 1 -- 172.21.15.16:0/2337957025 <== osd.7 v2:172.21.15.89:6808/1796778630 1 ==== osd_op_reply(1 [pgnls start_epoch 113 out=49b] v111'16 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x561bdba08000 con 0x561bdbc82400 2024-03-31T15:14:53.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.317+0000 7f3a60c00640 1 --2- 172.21.15.16:0/2337957025 >> [v2:172.21.15.16:6808/1398737533,v1:172.21.15.16:6810/1398737533] conn(0x561bdbc8a000 0x561bdbc90000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:53.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.317+0000 7f3a60c00640 1 -- 172.21.15.16:0/2337957025 --> [v2:172.21.15.16:6808/1398737533,v1:172.21.15.16:6810/1398737533] -- osd_op(unknown.0.0:2 17.20 17:04000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x561bdbc8a400 con 0x561bdbc8a000 2024-03-31T15:14:53.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.317+0000 7f3a67a00640 1 --2- 172.21.15.16:0/2337957025 >> [v2:172.21.15.16:6808/1398737533,v1:172.21.15.16:6810/1398737533] conn(0x561bdbc8a000 0x561bdbc90000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:53.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.317+0000 7f3a67a00640 1 --2- 172.21.15.16:0/2337957025 >> [v2:172.21.15.16:6808/1398737533,v1:172.21.15.16:6810/1398737533] conn(0x561bdbc8a000 0x561bdbc90000 crc :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:53.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.317+0000 7f3a67a00640 1 -- 172.21.15.16:0/2337957025 <== osd.1 v2:172.21.15.16:6808/1398737533 1 ==== osd_op_reply(2 [pgnls start_epoch 113 out=49b] v111'18 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x561bdba08500 con 0x561bdbc8a000 2024-03-31T15:14:53.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.317+0000 7f3a61600640 1 --2- 172.21.15.16:0/2337957025 >> [v2:172.21.15.16:6809/3360352974,v1:172.21.15.16:6812/3360352974] conn(0x561bdbc8ac00 0x561bdbc90580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:53.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.317+0000 7f3a61600640 1 -- 172.21.15.16:0/2337957025 --> [v2:172.21.15.16:6809/3360352974,v1:172.21.15.16:6812/3360352974] -- osd_op(unknown.0.0:3 17.10 17:08000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x561bdbc8b000 con 0x561bdbc8ac00 2024-03-31T15:14:53.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.317+0000 7f3a67000640 1 --2- 172.21.15.16:0/2337957025 >> [v2:172.21.15.16:6809/3360352974,v1:172.21.15.16:6812/3360352974] conn(0x561bdbc8ac00 0x561bdbc90580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:53.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.317+0000 7f3a67000640 1 --2- 172.21.15.16:0/2337957025 >> [v2:172.21.15.16:6809/3360352974,v1:172.21.15.16:6812/3360352974] conn(0x561bdbc8ac00 0x561bdbc90580 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:53.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.317+0000 7f3a67000640 1 -- 172.21.15.16:0/2337957025 <== osd.3 v2:172.21.15.16:6809/3360352974 1 ==== osd_op_reply(3 [pgnls start_epoch 113 out=49b] v111'12 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x561bdba08780 con 0x561bdbc8ac00 2024-03-31T15:14:53.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.317+0000 7f3a60c00640 1 --2- 172.21.15.16:0/2337957025 >> [v2:172.21.15.16:6800/932712244,v1:172.21.15.16:6801/932712244] conn(0x561bdbc8a800 0x561bdbc90b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:53.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.317+0000 7f3a60c00640 1 -- 172.21.15.16:0/2337957025 --> [v2:172.21.15.16:6800/932712244,v1:172.21.15.16:6801/932712244] -- osd_op(unknown.0.0:4 17.30 17:0c000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x561bdbc8b800 con 0x561bdbc8a800 2024-03-31T15:14:53.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.317+0000 7f3a66600640 1 --2- 172.21.15.16:0/2337957025 >> [v2:172.21.15.16:6800/932712244,v1:172.21.15.16:6801/932712244] conn(0x561bdbc8a800 0x561bdbc90b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:53.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.317+0000 7f3a66600640 1 --2- 172.21.15.16:0/2337957025 >> [v2:172.21.15.16:6800/932712244,v1:172.21.15.16:6801/932712244] conn(0x561bdbc8a800 0x561bdbc90b00 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:53.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.317+0000 7f3a66600640 1 -- 172.21.15.16:0/2337957025 <== osd.0 v2:172.21.15.16:6800/932712244 1 ==== osd_op_reply(4 [pgnls start_epoch 113 out=49b] v111'18 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x561bdba08000 con 0x561bdbc8a800 2024-03-31T15:14:53.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.317+0000 7f3a61600640 1 -- 172.21.15.16:0/2337957025 --> [v2:172.21.15.89:6808/1796778630,v1:172.21.15.89:6811/1796778630] -- osd_op(unknown.0.0:5 17.8 17:10000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x561bdbc8b400 con 0x561bdbc82400 2024-03-31T15:14:53.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.321+0000 7f3a66600640 1 -- 172.21.15.16:0/2337957025 <== osd.7 v2:172.21.15.89:6808/1796778630 2 ==== osd_op_reply(5 [pgnls start_epoch 113 out=49b] v111'18 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x561bdba08000 con 0x561bdbc82400 2024-03-31T15:14:53.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.321+0000 7f3a60c00640 1 -- 172.21.15.16:0/2337957025 --> [v2:172.21.15.16:6809/3360352974,v1:172.21.15.16:6812/3360352974] -- osd_op(unknown.0.0:6 17.28 17:14000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x561bdbc8bc00 con 0x561bdbc8ac00 2024-03-31T15:14:53.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.321+0000 7f3a67000640 1 -- 172.21.15.16:0/2337957025 <== osd.3 v2:172.21.15.16:6809/3360352974 2 ==== osd_op_reply(6 [pgnls start_epoch 113 out=49b] v111'12 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x561bdba08780 con 0x561bdbc8ac00 2024-03-31T15:14:53.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.321+0000 7f3a61600640 1 -- 172.21.15.16:0/2337957025 --> [v2:172.21.15.16:6800/932712244,v1:172.21.15.16:6801/932712244] -- osd_op(unknown.0.0:7 17.18 17:18000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x561bdbc8b800 con 0x561bdbc8a800 2024-03-31T15:14:53.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.321+0000 7f3a66600640 1 -- 172.21.15.16:0/2337957025 <== osd.0 v2:172.21.15.16:6800/932712244 2 ==== osd_op_reply(7 [pgnls start_epoch 113 out=49b] v111'16 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x561bdba08000 con 0x561bdbc8a800 2024-03-31T15:14:53.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.321+0000 7f3a60c00640 1 -- 172.21.15.16:0/2337957025 --> [v2:172.21.15.89:6808/1796778630,v1:172.21.15.89:6811/1796778630] -- osd_op(unknown.0.0:8 17.38 17:1c000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x561bdbca0000 con 0x561bdbc82400 2024-03-31T15:14:53.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.321+0000 7f3a66600640 1 -- 172.21.15.16:0/2337957025 <== osd.7 v2:172.21.15.89:6808/1796778630 3 ==== osd_op_reply(8 [pgnls start_epoch 113 out=49b] v111'18 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x561bdba08000 con 0x561bdbc82400 2024-03-31T15:14:53.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.321+0000 7f3a61600640 1 -- 172.21.15.16:0/2337957025 --> [v2:172.21.15.16:6800/932712244,v1:172.21.15.16:6801/932712244] -- osd_op(unknown.0.0:9 17.4 17:20000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x561bdbc82800 con 0x561bdbc8a800 2024-03-31T15:14:53.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.321+0000 7f3a66600640 1 -- 172.21.15.16:0/2337957025 <== osd.0 v2:172.21.15.16:6800/932712244 3 ==== osd_op_reply(9 [pgnls start_epoch 113 out=49b] v111'18 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x561bdba08000 con 0x561bdbc8a800 2024-03-31T15:14:53.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.321+0000 7f3a60c00640 1 -- 172.21.15.16:0/2337957025 --> [v2:172.21.15.16:6809/3360352974,v1:172.21.15.16:6812/3360352974] -- osd_op(unknown.0.0:10 17.24 17:24000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x561bdbca0400 con 0x561bdbc8ac00 2024-03-31T15:14:53.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.321+0000 7f3a67000640 1 -- 172.21.15.16:0/2337957025 <== osd.3 v2:172.21.15.16:6809/3360352974 3 ==== osd_op_reply(10 [pgnls start_epoch 113 out=49b] v111'8 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x561bdba08780 con 0x561bdbc8ac00 2024-03-31T15:14:53.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.321+0000 7f3a61600640 1 -- 172.21.15.16:0/2337957025 --> [v2:172.21.15.16:6809/3360352974,v1:172.21.15.16:6812/3360352974] -- osd_op(unknown.0.0:11 17.14 17:28000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x561bdbca0000 con 0x561bdbc8ac00 2024-03-31T15:14:53.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.321+0000 7f3a67000640 1 -- 172.21.15.16:0/2337957025 <== osd.3 v2:172.21.15.16:6809/3360352974 4 ==== osd_op_reply(11 [pgnls start_epoch 113 out=49b] v111'22 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x561bdba08780 con 0x561bdbc8ac00 2024-03-31T15:14:53.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.321+0000 7f3a60c00640 1 -- 172.21.15.16:0/2337957025 --> [v2:172.21.15.16:6800/932712244,v1:172.21.15.16:6801/932712244] -- osd_op(unknown.0.0:12 17.34 17:2c000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x561bdbca0800 con 0x561bdbc8a800 2024-03-31T15:14:53.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.321+0000 7f3a66600640 1 -- 172.21.15.16:0/2337957025 <== osd.0 v2:172.21.15.16:6800/932712244 4 ==== osd_op_reply(12 [pgnls start_epoch 113 out=49b] v111'14 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x561bdba08000 con 0x561bdbc8a800 2024-03-31T15:14:53.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.321+0000 7f3a61600640 1 -- 172.21.15.16:0/2337957025 --> [v2:172.21.15.89:6808/1796778630,v1:172.21.15.89:6811/1796778630] -- osd_op(unknown.0.0:13 17.c 17:30000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x561bdbca1000 con 0x561bdbc82400 2024-03-31T15:14:53.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.321+0000 7f3a66600640 1 -- 172.21.15.16:0/2337957025 <== osd.7 v2:172.21.15.89:6808/1796778630 4 ==== osd_op_reply(13 [pgnls start_epoch 113 out=49b] v111'12 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x561bdba08000 con 0x561bdbc82400 2024-03-31T15:14:53.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.321+0000 7f3a60c00640 1 --2- 172.21.15.16:0/2337957025 >> [v2:172.21.15.89:6800/1856844856,v1:172.21.15.89:6801/1856844856] conn(0x561bdbca0c00 0x561bdbc91080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:53.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.321+0000 7f3a60c00640 1 -- 172.21.15.16:0/2337957025 --> [v2:172.21.15.89:6800/1856844856,v1:172.21.15.89:6801/1856844856] -- osd_op(unknown.0.0:14 17.2c 17:34000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x561bdbca1400 con 0x561bdbca0c00 2024-03-31T15:14:53.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.321+0000 7f3a67a00640 1 --2- 172.21.15.16:0/2337957025 >> [v2:172.21.15.89:6800/1856844856,v1:172.21.15.89:6801/1856844856] conn(0x561bdbca0c00 0x561bdbc91080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:53.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.321+0000 7f3a67a00640 1 --2- 172.21.15.16:0/2337957025 >> [v2:172.21.15.89:6800/1856844856,v1:172.21.15.89:6801/1856844856] conn(0x561bdbca0c00 0x561bdbc91080 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:53.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.325+0000 7f3a67a00640 1 -- 172.21.15.16:0/2337957025 <== osd.4 v2:172.21.15.89:6800/1856844856 1 ==== osd_op_reply(14 [pgnls start_epoch 113 out=49b] v111'16 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x561bdba08500 con 0x561bdbca0c00 2024-03-31T15:14:53.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.325+0000 7f3a61600640 1 -- 172.21.15.16:0/2337957025 --> [v2:172.21.15.16:6808/1398737533,v1:172.21.15.16:6810/1398737533] -- osd_op(unknown.0.0:15 17.1c 17:38000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x561bdbc8b400 con 0x561bdbc8a000 2024-03-31T15:14:53.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.325+0000 7f3a67a00640 1 -- 172.21.15.16:0/2337957025 <== osd.1 v2:172.21.15.16:6808/1398737533 2 ==== osd_op_reply(15 [pgnls start_epoch 113 out=49b] v111'18 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x561bdba08500 con 0x561bdbc8a000 2024-03-31T15:14:53.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.325+0000 7f3a60c00640 1 -- 172.21.15.16:0/2337957025 --> [v2:172.21.15.16:6800/932712244,v1:172.21.15.16:6801/932712244] -- osd_op(unknown.0.0:16 17.3c 17:3c000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x561bdbca1800 con 0x561bdbc8a800 2024-03-31T15:14:53.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.325+0000 7f3a66600640 1 -- 172.21.15.16:0/2337957025 <== osd.0 v2:172.21.15.16:6800/932712244 5 ==== osd_op_reply(16 [pgnls start_epoch 113 out=49b] v111'16 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x561bdba08000 con 0x561bdbc8a800 2024-03-31T15:14:53.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.325+0000 7f3a61600640 1 -- 172.21.15.16:0/2337957025 --> [v2:172.21.15.89:6800/1856844856,v1:172.21.15.89:6801/1856844856] -- osd_op(unknown.0.0:17 17.2 17:40000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x561bdbc8b800 con 0x561bdbca0c00 2024-03-31T15:14:53.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.325+0000 7f3a67a00640 1 -- 172.21.15.16:0/2337957025 <== osd.4 v2:172.21.15.89:6800/1856844856 2 ==== osd_op_reply(17 [pgnls start_epoch 113 out=49b] v111'16 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x561bdba08500 con 0x561bdbca0c00 2024-03-31T15:14:53.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.325+0000 7f3a60c00640 1 --2- 172.21.15.16:0/2337957025 >> [v2:172.21.15.89:6810/50395527,v1:172.21.15.89:6814/50395527] conn(0x561bdbca1c00 0x561bdbc91600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:53.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.325+0000 7f3a60c00640 1 -- 172.21.15.16:0/2337957025 --> [v2:172.21.15.89:6810/50395527,v1:172.21.15.89:6814/50395527] -- osd_op(unknown.0.0:18 17.22 17:44000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x561bdbca1400 con 0x561bdbca1c00 2024-03-31T15:14:53.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.325+0000 7f3a67000640 1 --2- 172.21.15.16:0/2337957025 >> [v2:172.21.15.89:6810/50395527,v1:172.21.15.89:6814/50395527] conn(0x561bdbca1c00 0x561bdbc91600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:53.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.325+0000 7f3a67000640 1 --2- 172.21.15.16:0/2337957025 >> [v2:172.21.15.89:6810/50395527,v1:172.21.15.89:6814/50395527] conn(0x561bdbca1c00 0x561bdbc91600 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:53.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.325+0000 7f3a67000640 1 -- 172.21.15.16:0/2337957025 <== osd.5 v2:172.21.15.89:6810/50395527 1 ==== osd_op_reply(18 [pgnls start_epoch 113 out=49b] v111'16 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x561bdba08780 con 0x561bdbca1c00 2024-03-31T15:14:53.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.325+0000 7f3a61600640 1 -- 172.21.15.16:0/2337957025 --> [v2:172.21.15.16:6809/3360352974,v1:172.21.15.16:6812/3360352974] -- osd_op(unknown.0.0:19 17.12 17:48000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x561bdbca0400 con 0x561bdbc8ac00 2024-03-31T15:14:53.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.325+0000 7f3a67000640 1 -- 172.21.15.16:0/2337957025 <== osd.3 v2:172.21.15.16:6809/3360352974 5 ==== osd_op_reply(19 [pgnls start_epoch 113 out=81b] v112'22 uv22 ondisk = 1) ==== 144+0+81 (crc 0 0 0) 0x561bdba08780 con 0x561bdbc8ac00 2024-03-31T15:14:53.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.325+0000 7f3a69c498c0 1 -- 172.21.15.16:0/2337957025 --> [v2:172.21.15.16:6809/3360352974,v1:172.21.15.16:6812/3360352974] -- osd_op(unknown.0.0:20 17.12 17:4bec3886:::10000000002.00000000:head [stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e113) -- 0x561bdbc82c00 con 0x561bdbc8ac00 2024-03-31T15:14:53.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.325+0000 7f3a67000640 1 -- 172.21.15.16:0/2337957025 <== osd.3 v2:172.21.15.16:6809/3360352974 6 ==== osd_op_reply(20 10000000002.00000000 [stat out=16b] v0'0 uv22 ondisk = 0) ==== 164+0+16 (crc 0 0 0) 0x561bdba08780 con 0x561bdbc8ac00 2024-03-31T15:14:53.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.325+0000 7f3a69c498c0 10 datascan.operator(): handling object 1099511627778.0 2024-03-31T15:14:53.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.325+0000 7f3a69c498c0 1 -- 172.21.15.16:0/2337957025 --> [v2:172.21.15.16:6809/3360352974,v1:172.21.15.16:6812/3360352974] -- osd_op(unknown.0.0:21 17.12 17:4bec3886:::10000000002.00000000:head [call cephfs.accumulate_inode_metadata in=112b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x561bdbc83000 con 0x561bdbc8ac00 2024-03-31T15:14:53.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.329+0000 7f3a67000640 1 -- 172.21.15.16:0/2337957025 <== osd.3 v2:172.21.15.16:6809/3360352974 7 ==== osd_op_reply(21 10000000002.00000000 [call] v113'23 uv23 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x561bdba08780 con 0x561bdbc8ac00 2024-03-31T15:14:53.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.329+0000 7f3a69c498c0 1 -- 172.21.15.16:0/2337957025 --> [v2:172.21.15.89:6808/1796778630,v1:172.21.15.89:6811/1796778630] -- osd_op(unknown.0.0:22 17.32 17:4c000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x561bdbc83400 con 0x561bdbc82400 2024-03-31T15:14:53.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.329+0000 7f3a66600640 1 -- 172.21.15.16:0/2337957025 <== osd.7 v2:172.21.15.89:6808/1796778630 5 ==== osd_op_reply(22 [pgnls start_epoch 113 out=49b] v111'15 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x561bdba08000 con 0x561bdbc82400 2024-03-31T15:14:53.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.329+0000 7f3a61600640 1 -- 172.21.15.16:0/2337957025 --> [v2:172.21.15.89:6808/1796778630,v1:172.21.15.89:6811/1796778630] -- osd_op(unknown.0.0:23 17.a 17:50000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x561bdbca0000 con 0x561bdbc82400 2024-03-31T15:14:53.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.329+0000 7f3a66600640 1 -- 172.21.15.16:0/2337957025 <== osd.7 v2:172.21.15.89:6808/1796778630 6 ==== osd_op_reply(23 [pgnls start_epoch 113 out=49b] v111'10 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x561bdba08000 con 0x561bdbc82400 2024-03-31T15:14:53.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.329+0000 7f3a60c00640 1 -- 172.21.15.16:0/2337957025 --> [v2:172.21.15.16:6809/3360352974,v1:172.21.15.16:6812/3360352974] -- osd_op(unknown.0.0:24 17.2a 17:54000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x561bdbc82800 con 0x561bdbc8ac00 2024-03-31T15:14:53.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.329+0000 7f3a67000640 1 -- 172.21.15.16:0/2337957025 <== osd.3 v2:172.21.15.16:6809/3360352974 8 ==== osd_op_reply(24 [pgnls start_epoch 113 out=49b] v111'20 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x561bdba08780 con 0x561bdbc8ac00 2024-03-31T15:14:53.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.329+0000 7f3a61600640 1 -- 172.21.15.16:0/2337957025 --> [v2:172.21.15.16:6800/932712244,v1:172.21.15.16:6801/932712244] -- osd_op(unknown.0.0:25 17.1a 17:58000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x561bdbc8b000 con 0x561bdbc8a800 2024-03-31T15:14:53.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.329+0000 7f3a66600640 1 -- 172.21.15.16:0/2337957025 <== osd.0 v2:172.21.15.16:6800/932712244 6 ==== osd_op_reply(25 [pgnls start_epoch 113 out=49b] v111'14 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x561bdba08000 con 0x561bdbc8a800 2024-03-31T15:14:53.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.329+0000 7f3a60c00640 1 -- 172.21.15.16:0/2337957025 --> [v2:172.21.15.16:6808/1398737533,v1:172.21.15.16:6810/1398737533] -- osd_op(unknown.0.0:26 17.3a 17:5c000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x561bdbc8a400 con 0x561bdbc8a000 2024-03-31T15:14:53.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.329+0000 7f3a67a00640 1 -- 172.21.15.16:0/2337957025 <== osd.1 v2:172.21.15.16:6808/1398737533 3 ==== osd_op_reply(26 [pgnls start_epoch 113 out=49b] v111'12 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x561bdba08500 con 0x561bdbc8a000 2024-03-31T15:14:53.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.329+0000 7f3a61600640 1 --2- 172.21.15.16:0/2337957025 >> [v2:172.21.15.89:6809/3470471752,v1:172.21.15.89:6812/3470471752] conn(0x561bdbc8bc00 0x561bdbc92680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:53.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.329+0000 7f3a61600640 1 -- 172.21.15.16:0/2337957025 --> [v2:172.21.15.89:6809/3470471752,v1:172.21.15.89:6812/3470471752] -- osd_op(unknown.0.0:27 17.6 17:60000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x561bdbc83c00 con 0x561bdbc8bc00 2024-03-31T15:14:53.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.329+0000 7f3a66600640 1 --2- 172.21.15.16:0/2337957025 >> [v2:172.21.15.89:6809/3470471752,v1:172.21.15.89:6812/3470471752] conn(0x561bdbc8bc00 0x561bdbc92680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:53.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.329+0000 7f3a66600640 1 --2- 172.21.15.16:0/2337957025 >> [v2:172.21.15.89:6809/3470471752,v1:172.21.15.89:6812/3470471752] conn(0x561bdbc8bc00 0x561bdbc92680 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:53.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.333+0000 7f3a66600640 1 -- 172.21.15.16:0/2337957025 <== osd.6 v2:172.21.15.89:6809/3470471752 1 ==== osd_op_reply(27 [pgnls start_epoch 113 out=49b] v111'14 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x561bdba08000 con 0x561bdbc8bc00 2024-03-31T15:14:53.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.333+0000 7f3a60c00640 1 -- 172.21.15.16:0/2337957025 --> [v2:172.21.15.16:6800/932712244,v1:172.21.15.16:6801/932712244] -- osd_op(unknown.0.0:28 17.26 17:64000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x561bdbc8b400 con 0x561bdbc8a800 2024-03-31T15:14:53.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.333+0000 7f3a66600640 1 -- 172.21.15.16:0/2337957025 <== osd.0 v2:172.21.15.16:6800/932712244 7 ==== osd_op_reply(28 [pgnls start_epoch 113 out=49b] v111'30 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x561bdba08000 con 0x561bdbc8a800 2024-03-31T15:14:53.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.333+0000 7f3a61600640 1 -- 172.21.15.16:0/2337957025 --> [v2:172.21.15.89:6810/50395527,v1:172.21.15.89:6814/50395527] -- osd_op(unknown.0.0:29 17.16 17:68000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x561bdbca0000 con 0x561bdbca1c00 2024-03-31T15:14:53.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.333+0000 7f3a67000640 1 -- 172.21.15.16:0/2337957025 <== osd.5 v2:172.21.15.89:6810/50395527 2 ==== osd_op_reply(29 [pgnls start_epoch 113 out=49b] v111'20 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x561bdba08780 con 0x561bdbca1c00 2024-03-31T15:14:53.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.333+0000 7f3a60c00640 1 -- 172.21.15.16:0/2337957025 --> [v2:172.21.15.16:6800/932712244,v1:172.21.15.16:6801/932712244] -- osd_op(unknown.0.0:30 17.36 17:6c000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x561bdbc8b800 con 0x561bdbc8a800 2024-03-31T15:14:53.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.333+0000 7f3a66600640 1 -- 172.21.15.16:0/2337957025 <== osd.0 v2:172.21.15.16:6800/932712244 8 ==== osd_op_reply(30 [pgnls start_epoch 113 out=81b] v111'33 uv17 ondisk = 1) ==== 144+0+81 (crc 0 0 0) 0x561bdba08000 con 0x561bdbc8a800 2024-03-31T15:14:53.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.333+0000 7f3a69c498c0 1 -- 172.21.15.16:0/2337957025 --> [v2:172.21.15.16:6800/932712244,v1:172.21.15.16:6801/932712244] -- osd_op(unknown.0.0:31 17.36 17:6fe38dc2:::100000001f7.00000000:head [stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e113) -- 0x561bdbca0800 con 0x561bdbc8a800 2024-03-31T15:14:53.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.333+0000 7f3a66600640 1 -- 172.21.15.16:0/2337957025 <== osd.0 v2:172.21.15.16:6800/932712244 9 ==== osd_op_reply(31 100000001f7.00000000 [stat out=16b] v0'0 uv17 ondisk = 0) ==== 164+0+16 (crc 0 0 0) 0x561bdba08000 con 0x561bdbc8a800 2024-03-31T15:14:53.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.333+0000 7f3a69c498c0 10 datascan.operator(): handling object 1099511628279.0 2024-03-31T15:14:53.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.333+0000 7f3a69c498c0 1 -- 172.21.15.16:0/2337957025 --> [v2:172.21.15.16:6800/932712244,v1:172.21.15.16:6801/932712244] -- osd_op(unknown.0.0:32 17.36 17:6fe38dc2:::100000001f7.00000000:head [call cephfs.accumulate_inode_metadata in=112b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x561bdbca1000 con 0x561bdbc8a800 2024-03-31T15:14:53.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.333+0000 7f3a66600640 1 -- 172.21.15.16:0/2337957025 <== osd.0 v2:172.21.15.16:6800/932712244 10 ==== osd_op_reply(32 100000001f7.00000000 [call] v113'34 uv34 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x561bdba08000 con 0x561bdbc8a800 2024-03-31T15:14:53.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.333+0000 7f3a69c498c0 1 --2- 172.21.15.16:0/2337957025 >> [v2:172.21.15.16:6819/3018526646,v1:172.21.15.16:6822/3018526646] conn(0x561bdbca1800 0x561bdc660c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:53.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.333+0000 7f3a69c498c0 1 -- 172.21.15.16:0/2337957025 --> [v2:172.21.15.16:6819/3018526646,v1:172.21.15.16:6822/3018526646] -- osd_op(unknown.0.0:33 17.e 17:70000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x561bdbcbc000 con 0x561bdbca1800 2024-03-31T15:14:53.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.337+0000 7f3a67a00640 1 --2- 172.21.15.16:0/2337957025 >> [v2:172.21.15.16:6819/3018526646,v1:172.21.15.16:6822/3018526646] conn(0x561bdbca1800 0x561bdc660c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:53.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.337+0000 7f3a67a00640 1 --2- 172.21.15.16:0/2337957025 >> [v2:172.21.15.16:6819/3018526646,v1:172.21.15.16:6822/3018526646] conn(0x561bdbca1800 0x561bdc660c00 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:53.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.337+0000 7f3a67a00640 1 -- 172.21.15.16:0/2337957025 <== osd.2 v2:172.21.15.16:6819/3018526646 1 ==== osd_op_reply(33 [pgnls start_epoch 113 out=49b] v111'10 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x561bdba08500 con 0x561bdbca1800 2024-03-31T15:14:53.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.337+0000 7f3a60c00640 1 -- 172.21.15.16:0/2337957025 --> [v2:172.21.15.89:6809/3470471752,v1:172.21.15.89:6812/3470471752] -- osd_op(unknown.0.0:34 17.2e 17:74000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x561bdc687000 con 0x561bdbc8bc00 2024-03-31T15:14:53.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.337+0000 7f3a66600640 1 -- 172.21.15.16:0/2337957025 <== osd.6 v2:172.21.15.89:6809/3470471752 2 ==== osd_op_reply(34 [pgnls start_epoch 113 out=49b] v111'20 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x561bdba08000 con 0x561bdbc8bc00 2024-03-31T15:14:53.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.337+0000 7f3a61600640 1 -- 172.21.15.16:0/2337957025 --> [v2:172.21.15.16:6808/1398737533,v1:172.21.15.16:6810/1398737533] -- osd_op(unknown.0.0:35 17.1e 17:78000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x561bdbca1400 con 0x561bdbc8a000 2024-03-31T15:14:53.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.337+0000 7f3a67a00640 1 -- 172.21.15.16:0/2337957025 <== osd.1 v2:172.21.15.16:6808/1398737533 4 ==== osd_op_reply(35 [pgnls start_epoch 113 out=49b] v111'10 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x561bdba08500 con 0x561bdbc8a000 2024-03-31T15:14:53.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.337+0000 7f3a60c00640 1 -- 172.21.15.16:0/2337957025 --> [v2:172.21.15.89:6800/1856844856,v1:172.21.15.89:6801/1856844856] -- osd_op(unknown.0.0:36 17.3e 17:7c000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x561bdbcbdc00 con 0x561bdbca0c00 2024-03-31T15:14:53.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.337+0000 7f3a67a00640 1 -- 172.21.15.16:0/2337957025 <== osd.4 v2:172.21.15.89:6800/1856844856 3 ==== osd_op_reply(36 [pgnls start_epoch 113 out=49b] v111'8 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x561bdba08500 con 0x561bdbca0c00 2024-03-31T15:14:53.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.337+0000 7f3a61600640 1 -- 172.21.15.16:0/2337957025 --> [v2:172.21.15.89:6810/50395527,v1:172.21.15.89:6814/50395527] -- osd_op(unknown.0.0:37 17.1 17:80000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x561bdbc8b000 con 0x561bdbca1c00 2024-03-31T15:14:53.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.337+0000 7f3a67000640 1 -- 172.21.15.16:0/2337957025 <== osd.5 v2:172.21.15.89:6810/50395527 3 ==== osd_op_reply(37 [pgnls start_epoch 113 out=49b] v111'10 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x561bdba08780 con 0x561bdbca1c00 2024-03-31T15:14:53.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.337+0000 7f3a60c00640 1 -- 172.21.15.16:0/2337957025 --> [v2:172.21.15.16:6819/3018526646,v1:172.21.15.16:6822/3018526646] -- osd_op(unknown.0.0:38 17.21 17:84000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x561bdbcbe000 con 0x561bdbca1800 2024-03-31T15:14:53.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.337+0000 7f3a67a00640 1 -- 172.21.15.16:0/2337957025 <== osd.2 v2:172.21.15.16:6819/3018526646 2 ==== osd_op_reply(38 [pgnls start_epoch 113 out=49b] v111'24 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x561bdba08500 con 0x561bdbca1800 2024-03-31T15:14:53.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.337+0000 7f3a61600640 1 -- 172.21.15.16:0/2337957025 --> [v2:172.21.15.16:6809/3360352974,v1:172.21.15.16:6812/3360352974] -- osd_op(unknown.0.0:39 17.11 17:88000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x561bdbc8b400 con 0x561bdbc8ac00 2024-03-31T15:14:53.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.337+0000 7f3a67000640 1 -- 172.21.15.16:0/2337957025 <== osd.3 v2:172.21.15.16:6809/3360352974 9 ==== osd_op_reply(39 [pgnls start_epoch 113 out=49b] v111'12 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x561bdba08780 con 0x561bdbc8ac00 2024-03-31T15:14:53.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.337+0000 7f3a60c00640 1 -- 172.21.15.16:0/2337957025 --> [v2:172.21.15.16:6808/1398737533,v1:172.21.15.16:6810/1398737533] -- osd_op(unknown.0.0:40 17.31 17:8c000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x561bdbcbe400 con 0x561bdbc8a000 2024-03-31T15:14:53.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.337+0000 7f3a67a00640 1 -- 172.21.15.16:0/2337957025 <== osd.1 v2:172.21.15.16:6808/1398737533 5 ==== osd_op_reply(40 [pgnls start_epoch 113 out=81b] v111'15 uv8 ondisk = 1) ==== 144+0+81 (crc 0 0 0) 0x561bdba08500 con 0x561bdbc8a000 2024-03-31T15:14:53.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.337+0000 7f3a69c498c0 1 -- 172.21.15.16:0/2337957025 --> [v2:172.21.15.16:6808/1398737533,v1:172.21.15.16:6810/1398737533] -- osd_op(unknown.0.0:41 17.31 17:8de29a03:::10000000004.00000001:head [stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e113) -- 0x561bdbcbc400 con 0x561bdbc8a000 2024-03-31T15:14:53.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.337+0000 7f3a67a00640 1 -- 172.21.15.16:0/2337957025 <== osd.1 v2:172.21.15.16:6808/1398737533 6 ==== osd_op_reply(41 10000000004.00000001 [stat out=16b] v0'0 uv8 ondisk = 0) ==== 164+0+16 (crc 0 0 0) 0x561bdba08500 con 0x561bdbc8a000 2024-03-31T15:14:53.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.337+0000 7f3a69c498c0 10 datascan.operator(): handling object 1099511627780.1 2024-03-31T15:14:53.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.337+0000 7f3a69c498c0 1 -- 172.21.15.16:0/2337957025 --> [v2:172.21.15.89:6809/3470471752,v1:172.21.15.89:6812/3470471752] -- osd_op(unknown.0.0:42 17.5 17:a367a3e8:::10000000004.00000000:head [call cephfs.accumulate_inode_metadata in=112b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x561bdbcbc800 con 0x561bdbc8bc00 2024-03-31T15:14:53.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.341+0000 7f3a66600640 1 -- 172.21.15.16:0/2337957025 <== osd.6 v2:172.21.15.89:6809/3470471752 3 ==== osd_op_reply(42 10000000004.00000000 [call] v113'14 uv14 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x561bdba08000 con 0x561bdbc8bc00 2024-03-31T15:14:53.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.341+0000 7f3a69c498c0 1 -- 172.21.15.16:0/2337957025 --> [v2:172.21.15.16:6809/3360352974,v1:172.21.15.16:6812/3360352974] -- osd_op(unknown.0.0:43 17.9 17:90000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x561bdbcbcc00 con 0x561bdbc8ac00 2024-03-31T15:14:53.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.341+0000 7f3a67000640 1 -- 172.21.15.16:0/2337957025 <== osd.3 v2:172.21.15.16:6809/3360352974 10 ==== osd_op_reply(43 [pgnls start_epoch 113 out=49b] v111'4 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x561bdba08780 con 0x561bdbc8ac00 2024-03-31T15:14:53.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.341+0000 7f3a60c00640 1 -- 172.21.15.16:0/2337957025 --> [v2:172.21.15.89:6800/1856844856,v1:172.21.15.89:6801/1856844856] -- osd_op(unknown.0.0:44 17.29 17:94000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x561bdbcbe800 con 0x561bdbca0c00 2024-03-31T15:14:53.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.341+0000 7f3a67a00640 1 -- 172.21.15.16:0/2337957025 <== osd.4 v2:172.21.15.89:6800/1856844856 4 ==== osd_op_reply(44 [pgnls start_epoch 113 out=49b] v111'16 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x561bdba08500 con 0x561bdbca0c00 2024-03-31T15:14:53.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.341+0000 7f3a61600640 1 -- 172.21.15.16:0/2337957025 --> [v2:172.21.15.89:6810/50395527,v1:172.21.15.89:6814/50395527] -- osd_op(unknown.0.0:45 17.19 17:98000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x561bdbc83400 con 0x561bdbca1c00 2024-03-31T15:14:53.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.341+0000 7f3a67000640 1 -- 172.21.15.16:0/2337957025 <== osd.5 v2:172.21.15.89:6810/50395527 4 ==== osd_op_reply(45 [pgnls start_epoch 113 out=81b] v112'23 uv23 ondisk = 1) ==== 144+0+81 (crc 0 0 0) 0x561bdba08780 con 0x561bdbca1c00 2024-03-31T15:14:53.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.341+0000 7f3a69c498c0 1 -- 172.21.15.16:0/2337957025 --> [v2:172.21.15.89:6810/50395527,v1:172.21.15.89:6814/50395527] -- osd_op(unknown.0.0:46 17.19 17:997b7129:::100000001f8.00000000:head [stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e113) -- 0x561bdbcbd000 con 0x561bdbca1c00 2024-03-31T15:14:53.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.341+0000 7f3a67000640 1 -- 172.21.15.16:0/2337957025 <== osd.5 v2:172.21.15.89:6810/50395527 5 ==== osd_op_reply(46 100000001f8.00000000 [stat out=16b] v0'0 uv23 ondisk = 0) ==== 164+0+16 (crc 0 0 0) 0x561bdba08780 con 0x561bdbca1c00 2024-03-31T15:14:53.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.341+0000 7f3a69c498c0 10 datascan.operator(): handling object 1099511628280.0 2024-03-31T15:14:53.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.341+0000 7f3a69c498c0 1 -- 172.21.15.16:0/2337957025 --> [v2:172.21.15.89:6810/50395527,v1:172.21.15.89:6814/50395527] -- osd_op(unknown.0.0:47 17.19 17:997b7129:::100000001f8.00000000:head [call cephfs.accumulate_inode_metadata in=112b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x561bdbcbd400 con 0x561bdbca1c00 2024-03-31T15:14:53.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.345+0000 7f3a67000640 1 -- 172.21.15.16:0/2337957025 <== osd.5 v2:172.21.15.89:6810/50395527 6 ==== osd_op_reply(47 100000001f8.00000000 [call] v113'24 uv24 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x561bdba08780 con 0x561bdbca1c00 2024-03-31T15:14:53.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.345+0000 7f3a69c498c0 1 -- 172.21.15.16:0/2337957025 --> [v2:172.21.15.89:6810/50395527,v1:172.21.15.89:6814/50395527] -- osd_op(unknown.0.0:48 17.39 17:9c000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x561bdbcbd800 con 0x561bdbca1c00 2024-03-31T15:14:53.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.345+0000 7f3a67000640 1 -- 172.21.15.16:0/2337957025 <== osd.5 v2:172.21.15.89:6810/50395527 7 ==== osd_op_reply(48 [pgnls start_epoch 113 out=49b] v111'10 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x561bdba08780 con 0x561bdbca1c00 2024-03-31T15:14:53.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.345+0000 7f3a61600640 1 -- 172.21.15.16:0/2337957025 --> [v2:172.21.15.89:6809/3470471752,v1:172.21.15.89:6812/3470471752] -- osd_op(unknown.0.0:49 17.5 17:a0000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x561bdbc83c00 con 0x561bdbc8bc00 2024-03-31T15:14:53.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.345+0000 7f3a66600640 1 -- 172.21.15.16:0/2337957025 <== osd.6 v2:172.21.15.89:6809/3470471752 4 ==== osd_op_reply(49 [pgnls start_epoch 113 out=81b] v113'14 uv14 ondisk = 1) ==== 144+0+81 (crc 0 0 0) 0x561bdba08000 con 0x561bdbc8bc00 2024-03-31T15:14:53.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.345+0000 7f3a69c498c0 1 -- 172.21.15.16:0/2337957025 --> [v2:172.21.15.89:6809/3470471752,v1:172.21.15.89:6812/3470471752] -- osd_op(unknown.0.0:50 17.5 17:a367a3e8:::10000000004.00000000:head [stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e113) -- 0x561bdbc83c00 con 0x561bdbc8bc00 2024-03-31T15:14:53.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.345+0000 7f3a66600640 1 -- 172.21.15.16:0/2337957025 <== osd.6 v2:172.21.15.89:6809/3470471752 5 ==== osd_op_reply(50 10000000004.00000000 [stat out=16b] v0'0 uv14 ondisk = 0) ==== 164+0+16 (crc 0 0 0) 0x561bdba08000 con 0x561bdbc8bc00 2024-03-31T15:14:53.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.345+0000 7f3a69c498c0 10 datascan.operator(): handling object 1099511627780.0 2024-03-31T15:14:53.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.345+0000 7f3a69c498c0 1 -- 172.21.15.16:0/2337957025 --> [v2:172.21.15.89:6809/3470471752,v1:172.21.15.89:6812/3470471752] -- osd_op(unknown.0.0:51 17.5 17:a367a3e8:::10000000004.00000000:head [call cephfs.accumulate_inode_metadata in=112b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x561bdbc82c00 con 0x561bdbc8bc00 2024-03-31T15:14:53.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.345+0000 7f3a66600640 1 -- 172.21.15.16:0/2337957025 <== osd.6 v2:172.21.15.89:6809/3470471752 6 ==== osd_op_reply(51 10000000004.00000000 [call] v113'15 uv15 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x561bdba08000 con 0x561bdbc8bc00 2024-03-31T15:14:53.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.345+0000 7f3a69c498c0 1 -- 172.21.15.16:0/2337957025 --> [v2:172.21.15.16:6808/1398737533,v1:172.21.15.16:6810/1398737533] -- osd_op(unknown.0.0:52 17.25 17:a4000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x561bdbcbe000 con 0x561bdbc8a000 2024-03-31T15:14:53.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.345+0000 7f3a67a00640 1 -- 172.21.15.16:0/2337957025 <== osd.1 v2:172.21.15.16:6808/1398737533 7 ==== osd_op_reply(52 [pgnls start_epoch 113 out=49b] v111'12 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x561bdba08500 con 0x561bdbc8a000 2024-03-31T15:14:53.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.345+0000 7f3a61600640 1 -- 172.21.15.16:0/2337957025 --> [v2:172.21.15.89:6810/50395527,v1:172.21.15.89:6814/50395527] -- osd_op(unknown.0.0:53 17.15 17:a8000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x561bdbca0800 con 0x561bdbca1c00 2024-03-31T15:14:53.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.345+0000 7f3a67000640 1 -- 172.21.15.16:0/2337957025 <== osd.5 v2:172.21.15.89:6810/50395527 8 ==== osd_op_reply(53 [pgnls start_epoch 113 out=49b] v111'12 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x561bdba08780 con 0x561bdbca1c00 2024-03-31T15:14:53.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.345+0000 7f3a60c00640 1 -- 172.21.15.16:0/2337957025 --> [v2:172.21.15.89:6800/1856844856,v1:172.21.15.89:6801/1856844856] -- osd_op(unknown.0.0:54 17.35 17:ac000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x561bdbcbec00 con 0x561bdbca0c00 2024-03-31T15:14:53.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.349+0000 7f3a67a00640 1 -- 172.21.15.16:0/2337957025 <== osd.4 v2:172.21.15.89:6800/1856844856 5 ==== osd_op_reply(54 [pgnls start_epoch 113 out=49b] v111'12 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x561bdba08500 con 0x561bdbca0c00 2024-03-31T15:14:53.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.349+0000 7f3a61600640 1 -- 172.21.15.16:0/2337957025 --> [v2:172.21.15.89:6808/1796778630,v1:172.21.15.89:6811/1796778630] -- osd_op(unknown.0.0:55 17.d 17:b0000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x561bdbca1000 con 0x561bdbc82400 2024-03-31T15:14:53.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.349+0000 7f3a66600640 1 -- 172.21.15.16:0/2337957025 <== osd.7 v2:172.21.15.89:6808/1796778630 7 ==== osd_op_reply(55 [pgnls start_epoch 113 out=49b] v111'14 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x561bdba08000 con 0x561bdbc82400 2024-03-31T15:14:53.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.349+0000 7f3a60c00640 1 -- 172.21.15.16:0/2337957025 --> [v2:172.21.15.89:6808/1796778630,v1:172.21.15.89:6811/1796778630] -- osd_op(unknown.0.0:56 17.2d 17:b4000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x561bdbcbf000 con 0x561bdbc82400 2024-03-31T15:14:53.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.349+0000 7f3a66600640 1 -- 172.21.15.16:0/2337957025 <== osd.7 v2:172.21.15.89:6808/1796778630 8 ==== osd_op_reply(56 [pgnls start_epoch 113 out=49b] v111'14 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x561bdba08000 con 0x561bdbc82400 2024-03-31T15:14:53.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.349+0000 7f3a61600640 1 -- 172.21.15.16:0/2337957025 --> [v2:172.21.15.89:6808/1796778630,v1:172.21.15.89:6811/1796778630] -- osd_op(unknown.0.0:57 17.1d 17:b8000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x561bdbca1400 con 0x561bdbc82400 2024-03-31T15:14:53.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.349+0000 7f3a66600640 1 -- 172.21.15.16:0/2337957025 <== osd.7 v2:172.21.15.89:6808/1796778630 9 ==== osd_op_reply(57 [pgnls start_epoch 113 out=49b] v111'30 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x561bdba08000 con 0x561bdbc82400 2024-03-31T15:14:53.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.349+0000 7f3a60c00640 1 -- 172.21.15.16:0/2337957025 --> [v2:172.21.15.89:6808/1796778630,v1:172.21.15.89:6811/1796778630] -- osd_op(unknown.0.0:58 17.3d 17:bc000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x561bdbcbf400 con 0x561bdbc82400 2024-03-31T15:14:53.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.349+0000 7f3a66600640 1 -- 172.21.15.16:0/2337957025 <== osd.7 v2:172.21.15.89:6808/1796778630 10 ==== osd_op_reply(58 [pgnls start_epoch 113 out=49b] v111'14 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x561bdba08000 con 0x561bdbc82400 2024-03-31T15:14:53.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.349+0000 7f3a61600640 1 -- 172.21.15.16:0/2337957025 --> [v2:172.21.15.89:6800/1856844856,v1:172.21.15.89:6801/1856844856] -- osd_op(unknown.0.0:59 17.3 17:c0000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x561bdbca0400 con 0x561bdbca0c00 2024-03-31T15:14:53.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.349+0000 7f3a67a00640 1 -- 172.21.15.16:0/2337957025 <== osd.4 v2:172.21.15.89:6800/1856844856 6 ==== osd_op_reply(59 [pgnls start_epoch 113 out=49b] v111'14 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x561bdba08500 con 0x561bdbca0c00 2024-03-31T15:14:53.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.349+0000 7f3a60c00640 1 -- 172.21.15.16:0/2337957025 --> [v2:172.21.15.89:6800/1856844856,v1:172.21.15.89:6801/1856844856] -- osd_op(unknown.0.0:60 17.23 17:c4000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x561bdbcbf800 con 0x561bdbca0c00 2024-03-31T15:14:53.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.349+0000 7f3a67a00640 1 -- 172.21.15.16:0/2337957025 <== osd.4 v2:172.21.15.89:6800/1856844856 7 ==== osd_op_reply(60 [pgnls start_epoch 113 out=49b] v111'14 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x561bdba08500 con 0x561bdbca0c00 2024-03-31T15:14:53.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.349+0000 7f3a61600640 1 -- 172.21.15.16:0/2337957025 --> [v2:172.21.15.16:6809/3360352974,v1:172.21.15.16:6812/3360352974] -- osd_op(unknown.0.0:61 17.13 17:c8000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x561bdbca0000 con 0x561bdbc8ac00 2024-03-31T15:14:53.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.349+0000 7f3a67000640 1 -- 172.21.15.16:0/2337957025 <== osd.3 v2:172.21.15.16:6809/3360352974 11 ==== osd_op_reply(61 [pgnls start_epoch 113 out=49b] v111'30 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x561bdba08780 con 0x561bdbc8ac00 2024-03-31T15:14:53.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.349+0000 7f3a60c00640 1 -- 172.21.15.16:0/2337957025 --> [v2:172.21.15.89:6808/1796778630,v1:172.21.15.89:6811/1796778630] -- osd_op(unknown.0.0:62 17.33 17:cc000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x561bdbcbfc00 con 0x561bdbc82400 2024-03-31T15:14:53.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.349+0000 7f3a66600640 1 -- 172.21.15.16:0/2337957025 <== osd.7 v2:172.21.15.89:6808/1796778630 11 ==== osd_op_reply(62 [pgnls start_epoch 113 out=49b] v111'20 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x561bdba08000 con 0x561bdbc82400 2024-03-31T15:14:53.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.349+0000 7f3a61600640 1 -- 172.21.15.16:0/2337957025 --> [v2:172.21.15.16:6800/932712244,v1:172.21.15.16:6801/932712244] -- osd_op(unknown.0.0:63 17.b 17:d0000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x561bdbc83000 con 0x561bdbc8a800 2024-03-31T15:14:53.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.349+0000 7f3a66600640 1 -- 172.21.15.16:0/2337957025 <== osd.0 v2:172.21.15.16:6800/932712244 11 ==== osd_op_reply(63 [pgnls start_epoch 113 out=49b] v111'16 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x561bdba08000 con 0x561bdbc8a800 2024-03-31T15:14:53.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.349+0000 7f3a60c00640 1 -- 172.21.15.16:0/2337957025 --> [v2:172.21.15.89:6800/1856844856,v1:172.21.15.89:6801/1856844856] -- osd_op(unknown.0.0:64 17.2b 17:d4000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x561bdbc8b800 con 0x561bdbca0c00 2024-03-31T15:14:53.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.349+0000 7f3a67a00640 1 -- 172.21.15.16:0/2337957025 <== osd.4 v2:172.21.15.89:6800/1856844856 8 ==== osd_op_reply(64 [pgnls start_epoch 113 out=49b] v111'22 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x561bdba08500 con 0x561bdbca0c00 2024-03-31T15:14:53.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.353+0000 7f3a61600640 1 -- 172.21.15.16:0/2337957025 --> [v2:172.21.15.89:6800/1856844856,v1:172.21.15.89:6801/1856844856] -- osd_op(unknown.0.0:65 17.1b 17:d8000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x561bdbc82800 con 0x561bdbca0c00 2024-03-31T15:14:53.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.353+0000 7f3a67a00640 1 -- 172.21.15.16:0/2337957025 <== osd.4 v2:172.21.15.89:6800/1856844856 9 ==== osd_op_reply(65 [pgnls start_epoch 113 out=49b] v111'16 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x561bdba08500 con 0x561bdbca0c00 2024-03-31T15:14:53.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.353+0000 7f3a60c00640 1 -- 172.21.15.16:0/2337957025 --> [v2:172.21.15.16:6819/3018526646,v1:172.21.15.16:6822/3018526646] -- osd_op(unknown.0.0:66 17.3b 17:dc000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x561bdbc8a400 con 0x561bdbca1800 2024-03-31T15:14:53.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.353+0000 7f3a67a00640 1 -- 172.21.15.16:0/2337957025 <== osd.2 v2:172.21.15.16:6819/3018526646 3 ==== osd_op_reply(66 [pgnls start_epoch 113 out=49b] v111'8 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x561bdba08500 con 0x561bdbca1800 2024-03-31T15:14:53.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.353+0000 7f3a61600640 1 -- 172.21.15.16:0/2337957025 --> [v2:172.21.15.16:6800/932712244,v1:172.21.15.16:6801/932712244] -- osd_op(unknown.0.0:67 17.7 17:e0000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x561bdbc8b800 con 0x561bdbc8a800 2024-03-31T15:14:53.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.353+0000 7f3a66600640 1 -- 172.21.15.16:0/2337957025 <== osd.0 v2:172.21.15.16:6800/932712244 12 ==== osd_op_reply(67 [pgnls start_epoch 113 out=49b] v111'10 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x561bdba08000 con 0x561bdbc8a800 2024-03-31T15:14:53.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.353+0000 7f3a60c00640 1 -- 172.21.15.16:0/2337957025 --> [v2:172.21.15.89:6800/1856844856,v1:172.21.15.89:6801/1856844856] -- osd_op(unknown.0.0:68 17.27 17:e4000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x561bdbcbf000 con 0x561bdbca0c00 2024-03-31T15:14:53.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.353+0000 7f3a67a00640 1 -- 172.21.15.16:0/2337957025 <== osd.4 v2:172.21.15.89:6800/1856844856 10 ==== osd_op_reply(68 [pgnls start_epoch 113 out=49b] v111'4 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x561bdba08500 con 0x561bdbca0c00 2024-03-31T15:14:53.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.353+0000 7f3a61600640 1 -- 172.21.15.16:0/2337957025 --> [v2:172.21.15.89:6809/3470471752,v1:172.21.15.89:6812/3470471752] -- osd_op(unknown.0.0:69 17.17 17:e8000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x561bdbc8b000 con 0x561bdbc8bc00 2024-03-31T15:14:53.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.353+0000 7f3a66600640 1 -- 172.21.15.16:0/2337957025 <== osd.6 v2:172.21.15.89:6809/3470471752 7 ==== osd_op_reply(69 [pgnls start_epoch 113 out=49b] v111'18 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x561bdba08000 con 0x561bdbc8bc00 2024-03-31T15:14:53.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.353+0000 7f3a60c00640 1 -- 172.21.15.16:0/2337957025 --> [v2:172.21.15.16:6800/932712244,v1:172.21.15.16:6801/932712244] -- osd_op(unknown.0.0:70 17.37 17:ec000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x561bdbcbf400 con 0x561bdbc8a800 2024-03-31T15:14:53.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.353+0000 7f3a66600640 1 -- 172.21.15.16:0/2337957025 <== osd.0 v2:172.21.15.16:6800/932712244 13 ==== osd_op_reply(70 [pgnls start_epoch 113 out=49b] v111'22 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x561bdba08000 con 0x561bdbc8a800 2024-03-31T15:14:53.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.353+0000 7f3a61600640 1 -- 172.21.15.16:0/2337957025 --> [v2:172.21.15.16:6808/1398737533,v1:172.21.15.16:6810/1398737533] -- osd_op(unknown.0.0:71 17.f 17:f0000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x561bdbcbe800 con 0x561bdbc8a000 2024-03-31T15:14:53.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.353+0000 7f3a67a00640 1 -- 172.21.15.16:0/2337957025 <== osd.1 v2:172.21.15.16:6808/1398737533 8 ==== osd_op_reply(71 [pgnls start_epoch 113 out=49b] v111'18 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x561bdba08500 con 0x561bdbc8a000 2024-03-31T15:14:53.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.353+0000 7f3a60c00640 1 -- 172.21.15.16:0/2337957025 --> [v2:172.21.15.89:6800/1856844856,v1:172.21.15.89:6801/1856844856] -- osd_op(unknown.0.0:72 17.2f 17:f4000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x561bdbcbfc00 con 0x561bdbca0c00 2024-03-31T15:14:53.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.353+0000 7f3a67a00640 1 -- 172.21.15.16:0/2337957025 <== osd.4 v2:172.21.15.89:6800/1856844856 11 ==== osd_op_reply(72 [pgnls start_epoch 113 out=49b] v111'16 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x561bdba08500 con 0x561bdbca0c00 2024-03-31T15:14:53.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.353+0000 7f3a61600640 1 -- 172.21.15.16:0/2337957025 --> [v2:172.21.15.89:6809/3470471752,v1:172.21.15.89:6812/3470471752] -- osd_op(unknown.0.0:73 17.1f 17:f8000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x561bdbcbe000 con 0x561bdbc8bc00 2024-03-31T15:14:53.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.353+0000 7f3a66600640 1 -- 172.21.15.16:0/2337957025 <== osd.6 v2:172.21.15.89:6809/3470471752 8 ==== osd_op_reply(73 [pgnls start_epoch 113 out=49b] v111'12 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x561bdba08000 con 0x561bdbc8bc00 2024-03-31T15:14:53.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.353+0000 7f3a60c00640 1 -- 172.21.15.16:0/2337957025 --> [v2:172.21.15.16:6800/932712244,v1:172.21.15.16:6801/932712244] -- osd_op(unknown.0.0:74 17.3f 17:fc000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x561bdbca1000 con 0x561bdbc8a800 2024-03-31T15:14:53.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.353+0000 7f3a66600640 1 -- 172.21.15.16:0/2337957025 <== osd.0 v2:172.21.15.16:6800/932712244 14 ==== osd_op_reply(74 [pgnls start_epoch 113 out=49b] v111'17 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x561bdba08000 con 0x561bdbc8a800 2024-03-31T15:14:53.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.353+0000 7f3a69c498c0 1 -- 172.21.15.16:0/2337957025 >> [v2:172.21.15.16:6800/932712244,v1:172.21.15.16:6801/932712244] conn(0x561bdbc8a800 msgr2=0x561bdbc90b00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:53.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.353+0000 7f3a69c498c0 1 --2- 172.21.15.16:0/2337957025 >> [v2:172.21.15.16:6800/932712244,v1:172.21.15.16:6801/932712244] conn(0x561bdbc8a800 0x561bdbc90b00 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:53.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.353+0000 7f3a69c498c0 1 -- 172.21.15.16:0/2337957025 >> [v2:172.21.15.16:6808/1398737533,v1:172.21.15.16:6810/1398737533] conn(0x561bdbc8a000 msgr2=0x561bdbc90000 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:53.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.353+0000 7f3a69c498c0 1 --2- 172.21.15.16:0/2337957025 >> [v2:172.21.15.16:6808/1398737533,v1:172.21.15.16:6810/1398737533] conn(0x561bdbc8a000 0x561bdbc90000 crc :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:53.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.353+0000 7f3a69c498c0 1 -- 172.21.15.16:0/2337957025 >> [v2:172.21.15.16:6819/3018526646,v1:172.21.15.16:6822/3018526646] conn(0x561bdbca1800 msgr2=0x561bdc660c00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:53.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.353+0000 7f3a69c498c0 1 --2- 172.21.15.16:0/2337957025 >> [v2:172.21.15.16:6819/3018526646,v1:172.21.15.16:6822/3018526646] conn(0x561bdbca1800 0x561bdc660c00 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:53.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.353+0000 7f3a69c498c0 1 -- 172.21.15.16:0/2337957025 >> [v2:172.21.15.16:6809/3360352974,v1:172.21.15.16:6812/3360352974] conn(0x561bdbc8ac00 msgr2=0x561bdbc90580 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:53.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.353+0000 7f3a69c498c0 1 --2- 172.21.15.16:0/2337957025 >> [v2:172.21.15.16:6809/3360352974,v1:172.21.15.16:6812/3360352974] conn(0x561bdbc8ac00 0x561bdbc90580 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:53.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.353+0000 7f3a69c498c0 1 -- 172.21.15.16:0/2337957025 >> [v2:172.21.15.89:6800/1856844856,v1:172.21.15.89:6801/1856844856] conn(0x561bdbca0c00 msgr2=0x561bdbc91080 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:53.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.353+0000 7f3a66600640 1 -- 172.21.15.16:0/2337957025 reap_dead start 2024-03-31T15:14:53.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.353+0000 7f3a69c498c0 1 --2- 172.21.15.16:0/2337957025 >> [v2:172.21.15.89:6800/1856844856,v1:172.21.15.89:6801/1856844856] conn(0x561bdbca0c00 0x561bdbc91080 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:53.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.353+0000 7f3a69c498c0 1 -- 172.21.15.16:0/2337957025 >> [v2:172.21.15.89:6810/50395527,v1:172.21.15.89:6814/50395527] conn(0x561bdbca1c00 msgr2=0x561bdbc91600 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:53.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.353+0000 7f3a69c498c0 1 --2- 172.21.15.16:0/2337957025 >> [v2:172.21.15.89:6810/50395527,v1:172.21.15.89:6814/50395527] conn(0x561bdbca1c00 0x561bdbc91600 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:53.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.353+0000 7f3a69c498c0 1 -- 172.21.15.16:0/2337957025 >> [v2:172.21.15.89:6809/3470471752,v1:172.21.15.89:6812/3470471752] conn(0x561bdbc8bc00 msgr2=0x561bdbc92680 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:53.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.353+0000 7f3a69c498c0 1 --2- 172.21.15.16:0/2337957025 >> [v2:172.21.15.89:6809/3470471752,v1:172.21.15.89:6812/3470471752] conn(0x561bdbc8bc00 0x561bdbc92680 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:53.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.353+0000 7f3a69c498c0 1 -- 172.21.15.16:0/2337957025 >> [v2:172.21.15.89:6808/1796778630,v1:172.21.15.89:6811/1796778630] conn(0x561bdbc82400 msgr2=0x561bdc660680 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:53.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.353+0000 7f3a69c498c0 1 --2- 172.21.15.16:0/2337957025 >> [v2:172.21.15.89:6808/1796778630,v1:172.21.15.89:6811/1796778630] conn(0x561bdbc82400 0x561bdc660680 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:53.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.353+0000 7f3a69c498c0 1 -- 172.21.15.16:0/2337957025 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x561bdbc82000 msgr2=0x561bdc660100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:53.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.353+0000 7f3a69c498c0 1 --2- 172.21.15.16:0/2337957025 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x561bdbc82000 0x561bdc660100 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x561bdc7d22d0 tx=0x561bdc674180 comp rx=0 tx=0).stop 2024-03-31T15:14:53.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.357+0000 7f3a69c498c0 1 -- 172.21.15.16:0/2337957025 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561bdc7d1400 msgr2=0x561bdc65f080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:53.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.357+0000 7f3a66600640 1 -- 172.21.15.16:0/2337957025 reap_dead start 2024-03-31T15:14:53.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.357+0000 7f3a69c498c0 1 --2- 172.21.15.16:0/2337957025 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561bdc7d1400 0x561bdc65f080 secure :-1 s=READY pgs=1813 cs=0 l=1 rev1=1 crypto rx=0x561bdc7d21b0 tx=0x561bdc674420 comp rx=0 tx=0).stop 2024-03-31T15:14:53.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.357+0000 7f3a69c498c0 1 -- 172.21.15.16:0/2337957025 shutdown_connections 2024-03-31T15:14:53.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.357+0000 7f3a69c498c0 1 --2- 172.21.15.16:0/2337957025 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561bdc7d1400 0x561bdc65f080 unknown :-1 s=CLOSED pgs=1813 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:53.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.357+0000 7f3a69c498c0 1 -- 172.21.15.16:0/2337957025 >> 172.21.15.16:0/2337957025 conn(0x561bdc7d1000 msgr2=0x561bdb8c2000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:14:53.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.357+0000 7f3a69c498c0 1 -- 172.21.15.16:0/2337957025 shutdown_connections 2024-03-31T15:14:53.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.357+0000 7f3a69c498c0 1 -- 172.21.15.16:0/2337957025 wait complete. 2024-03-31T15:14:53.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.357+0000 7f3a69c498c0 1 -- 172.21.15.16:0/2760865108 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561bdb895000 msgr2=0x561bdc65e000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:53.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.357+0000 7f3a69c498c0 1 --2- 172.21.15.16:0/2760865108 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561bdb895000 0x561bdc65e000 secure :-1 s=READY pgs=1811 cs=0 l=1 rev1=1 crypto rx=0x561bdc6fda70 tx=0x561bdc674600 comp rx=0 tx=0).stop 2024-03-31T15:14:53.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.357+0000 7f3a69c498c0 1 -- 172.21.15.16:0/2760865108 shutdown_connections 2024-03-31T15:14:53.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.357+0000 7f3a69c498c0 1 --2- 172.21.15.16:0/2760865108 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561bdb895800 0x561bdc65eb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:53.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.357+0000 7f3a69c498c0 1 --2- 172.21.15.16:0/2760865108 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x561bdb895400 0x561bdc65e580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:53.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.357+0000 7f3a69c498c0 1 --2- 172.21.15.16:0/2760865108 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561bdb895000 0x561bdc65e000 unknown :-1 s=CLOSED pgs=1811 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:53.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.357+0000 7f3a69c498c0 1 -- 172.21.15.16:0/2760865108 >> 172.21.15.16:0/2760865108 conn(0x561bdb894c00 msgr2=0x561bdb8c1800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:14:53.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.357+0000 7f3a69c498c0 1 -- 172.21.15.16:0/2760865108 shutdown_connections 2024-03-31T15:14:53.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.357+0000 7f3a69c498c0 1 -- 172.21.15.16:0/2760865108 wait complete. 2024-03-31T15:14:53.354 DEBUG:tasks.cephfs.filesystem:Ran ['cephfs-data-scan', '--debug-mds=20', '--debug-ms=1', '--debug-objecter=1', 'scan_extents', 'cephfs_data'] in time 0:00:00.145354, result: 2024-03-31T15:14:53.355 DEBUG:teuthology.orchestra.run.smithi016:> cephfs-data-scan --debug-mds=20 --debug-ms=1 --debug-objecter=1 scan_inodes --filter-tag mytag123 cephfs_data 2024-03-31T15:14:53.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.405+0000 7f4cf1db98c0 1 -- start start 2024-03-31T15:14:53.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.405+0000 7f4cf1db98c0 1 Processor -- start 2024-03-31T15:14:53.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.409+0000 7f4cf1db98c0 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5589eb503800 0x5589ec2d0000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:53.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.409+0000 7f4cf1db98c0 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5589eb503400 0x5589ec2d0580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:53.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.409+0000 7f4cf1db98c0 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5589eb503000 0x5589ec2d0b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:53.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.409+0000 7f4cf1db98c0 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5589eb562900 con 0x5589eb503800 2024-03-31T15:14:53.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.409+0000 7f4cf1db98c0 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x5589ec2da000 con 0x5589eb503000 2024-03-31T15:14:53.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.409+0000 7f4cf1db98c0 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x5589ec2da180 con 0x5589eb503400 2024-03-31T15:14:53.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.409+0000 7f4cefe00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5589eb503800 0x5589ec2d0000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:53.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.409+0000 7f4cefe00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5589eb503800 0x5589ec2d0000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55666/0 (socket says 172.21.15.16:55666) 2024-03-31T15:14:53.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.409+0000 7f4cefe00640 1 -- 172.21.15.16:0/2832673357 learned_addr learned my addr 172.21.15.16:0/2832673357 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:14:53.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.409+0000 7f4cf0800640 1 --2- 172.21.15.16:0/2832673357 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5589eb503000 0x5589ec2d0b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:53.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.409+0000 7f4cef400640 1 --2- 172.21.15.16:0/2832673357 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5589eb503400 0x5589ec2d0580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:53.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.409+0000 7f4cefe00640 1 -- 172.21.15.16:0/2832673357 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5589eb503400 msgr2=0x5589ec2d0580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:14:53.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.409+0000 7f4cefe00640 1 --2- 172.21.15.16:0/2832673357 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5589eb503400 0x5589ec2d0580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:53.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.409+0000 7f4cefe00640 1 -- 172.21.15.16:0/2832673357 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5589eb503000 msgr2=0x5589ec2d0b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:14:53.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.409+0000 7f4cefe00640 1 --2- 172.21.15.16:0/2832673357 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5589eb503000 0x5589ec2d0b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:53.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.409+0000 7f4cefe00640 1 -- 172.21.15.16:0/2832673357 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5589eb5281e0 con 0x5589eb503800 2024-03-31T15:14:53.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.409+0000 7f4cf0800640 1 --2- 172.21.15.16:0/2832673357 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5589eb503000 0x5589ec2d0b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:14:53.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.409+0000 7f4cef400640 1 --2- 172.21.15.16:0/2832673357 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5589eb503400 0x5589ec2d0580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:14:53.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.409+0000 7f4cefe00640 1 --2- 172.21.15.16:0/2832673357 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5589eb503800 0x5589ec2d0000 secure :-1 s=READY pgs=1815 cs=0 l=1 rev1=1 crypto rx=0x5589ec36f8c0 tx=0x5589eb5c8660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ebda35c8d6b4813 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:53.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.409+0000 7f4cecc00640 1 -- 172.21.15.16:0/2832673357 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x5589eb5384e0 con 0x5589eb503800 2024-03-31T15:14:53.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.409+0000 7f4cecc00640 1 -- 172.21.15.16:0/2832673357 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x5589eb54e1c0 con 0x5589eb503800 2024-03-31T15:14:53.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.409+0000 7f4cecc00640 1 -- 172.21.15.16:0/2832673357 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x5589eb538b60 con 0x5589eb503800 2024-03-31T15:14:53.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.409+0000 7f4cf1db98c0 1 -- 172.21.15.16:0/2832673357 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5589eb5292c0 con 0x5589eb503800 2024-03-31T15:14:53.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.409+0000 7f4cf1db98c0 1 -- 172.21.15.16:0/2832673357 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({fsmap=0}) -- 0x5589eb529a40 con 0x5589eb503800 2024-03-31T15:14:53.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.409+0000 7f4cf1db98c0 4 waiting for MDS map... 2024-03-31T15:14:53.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.409+0000 7f4cecc00640 1 -- 172.21.15.16:0/2832673357 <== mon.0 v2:172.21.15.16:3300/0 4 ==== osd_map(113..113 src has 1..113) ==== 5939+0+0 (secure 0 0 0) 0x5589ec422600 con 0x5589eb503800 2024-03-31T15:14:53.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.409+0000 7f4cecc00640 1 -- 172.21.15.16:0/2832673357 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 106) ==== 2885+0+0 (secure 0 0 0) 0x5589eb559180 con 0x5589eb503800 2024-03-31T15:14:53.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.409+0000 7f4cf1db98c0 4 Got MDS map 106 2024-03-31T15:14:53.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.409+0000 7f4cf1db98c0 10 datascan.parse_kwarg: Applying tag filter: 'mytag123' 2024-03-31T15:14:53.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.409+0000 7f4cf1db98c0 4 datascan.main: Using metadata pool output 2024-03-31T15:14:53.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.409+0000 7f4cf1db98c0 4 datascan.main: connecting to RADOS... 2024-03-31T15:14:53.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.409+0000 7f4cf1db98c0 1 Processor -- start 2024-03-31T15:14:53.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.409+0000 7f4cf1db98c0 1 -- start start 2024-03-31T15:14:53.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.409+0000 7f4cf1db98c0 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5589ec425800 0x5589ec2d1080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:53.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.409+0000 7f4cf1db98c0 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5589ec425c00 0x5589ec2d1600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:53.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.409+0000 7f4cf1db98c0 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5589ec476000 0x5589ec2d1b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:53.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.409+0000 7f4cf1db98c0 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5589ec2da300 con 0x5589ec476000 2024-03-31T15:14:53.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.409+0000 7f4cf1db98c0 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x5589ec2da480 con 0x5589ec425c00 2024-03-31T15:14:53.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.409+0000 7f4cf1db98c0 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x5589ec2da600 con 0x5589ec425800 2024-03-31T15:14:53.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.413+0000 7f4cef400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5589ec476000 0x5589ec2d1b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:53.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.413+0000 7f4cef400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5589ec476000 0x5589ec2d1b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55674/0 (socket says 172.21.15.16:55674) 2024-03-31T15:14:53.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.413+0000 7f4cef400640 1 -- 172.21.15.16:0/119484809 learned_addr learned my addr 172.21.15.16:0/119484809 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:14:53.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.413+0000 7f4cf0800640 1 --2- 172.21.15.16:0/119484809 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5589ec425800 0x5589ec2d1080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:53.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.413+0000 7f4cefe00640 1 --2- 172.21.15.16:0/119484809 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5589ec425c00 0x5589ec2d1600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:53.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.413+0000 7f4cef400640 1 -- 172.21.15.16:0/119484809 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5589ec425800 msgr2=0x5589ec2d1080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:14:53.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.413+0000 7f4cef400640 1 --2- 172.21.15.16:0/119484809 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5589ec425800 0x5589ec2d1080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:53.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.413+0000 7f4cef400640 1 -- 172.21.15.16:0/119484809 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5589ec425c00 msgr2=0x5589ec2d1600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:14:53.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.413+0000 7f4cef400640 1 --2- 172.21.15.16:0/119484809 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5589ec425c00 0x5589ec2d1600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:53.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.413+0000 7f4cef400640 1 -- 172.21.15.16:0/119484809 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5589eb529c20 con 0x5589ec476000 2024-03-31T15:14:53.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.413+0000 7f4cef400640 1 --2- 172.21.15.16:0/119484809 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5589ec476000 0x5589ec2d1b80 secure :-1 s=READY pgs=1816 cs=0 l=1 rev1=1 crypto rx=0x5589ec36f020 tx=0x5589ec270000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4e47d5b9634c547 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:53.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.413+0000 7f4cea400640 1 -- 172.21.15.16:0/119484809 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x5589eb5389c0 con 0x5589ec476000 2024-03-31T15:14:53.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.413+0000 7f4cea400640 1 -- 172.21.15.16:0/119484809 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x5589eb54ee00 con 0x5589ec476000 2024-03-31T15:14:53.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.413+0000 7f4cea400640 1 -- 172.21.15.16:0/119484809 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x5589eb5384e0 con 0x5589ec476000 2024-03-31T15:14:53.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.413+0000 7f4cf1db98c0 1 -- 172.21.15.16:0/119484809 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5589ec476000 msgr2=0x5589ec2d1b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:53.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.413+0000 7f4cf1db98c0 1 --2- 172.21.15.16:0/119484809 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5589ec476000 0x5589ec2d1b80 secure :-1 s=READY pgs=1816 cs=0 l=1 rev1=1 crypto rx=0x5589ec36f020 tx=0x5589ec270000 comp rx=0 tx=0).stop 2024-03-31T15:14:53.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.413+0000 7f4cf1db98c0 1 -- 172.21.15.16:0/119484809 shutdown_connections 2024-03-31T15:14:53.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.413+0000 7f4cf1db98c0 1 --2- 172.21.15.16:0/119484809 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5589ec476000 0x5589ec2d1b80 unknown :-1 s=CLOSED pgs=1816 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:53.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.413+0000 7f4cf1db98c0 1 --2- 172.21.15.16:0/119484809 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5589ec425c00 0x5589ec2d1600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:53.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.413+0000 7f4cf1db98c0 1 --2- 172.21.15.16:0/119484809 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5589ec425800 0x5589ec2d1080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:53.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.413+0000 7f4cf1db98c0 1 -- 172.21.15.16:0/119484809 >> 172.21.15.16:0/119484809 conn(0x5589eb502c00 msgr2=0x5589eb531800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:14:53.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.413+0000 7f4cf1db98c0 1 -- 172.21.15.16:0/119484809 shutdown_connections 2024-03-31T15:14:53.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.413+0000 7f4cf1db98c0 1 -- 172.21.15.16:0/119484809 wait complete. 2024-03-31T15:14:53.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.413+0000 7f4cf1db98c0 1 Processor -- start 2024-03-31T15:14:53.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.413+0000 7f4cf1db98c0 1 -- start start 2024-03-31T15:14:53.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.413+0000 7f4cf1db98c0 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5589ec425c00 0x5589ec2d1080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:53.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.413+0000 7f4cf1db98c0 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5589ec425800 0x5589ec2d1600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:53.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.413+0000 7f4cf1db98c0 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5589eb7f4000 0x5589ec2d1b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:53.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.413+0000 7f4cf1db98c0 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5589ec2da900 con 0x5589ec425800 2024-03-31T15:14:53.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.413+0000 7f4cf1db98c0 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x5589ec2da780 con 0x5589eb7f4000 2024-03-31T15:14:53.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.413+0000 7f4cf1db98c0 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x5589ec2daa80 con 0x5589ec425c00 2024-03-31T15:14:53.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.413+0000 7f4cefe00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5589ec425800 0x5589ec2d1600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:53.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.413+0000 7f4cefe00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5589ec425800 0x5589ec2d1600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55676/0 (socket says 172.21.15.16:55676) 2024-03-31T15:14:53.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.413+0000 7f4cefe00640 1 -- 172.21.15.16:0/146512241 learned_addr learned my addr 172.21.15.16:0/146512241 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:14:53.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.413+0000 7f4cf0800640 1 --2- 172.21.15.16:0/146512241 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5589ec425c00 0x5589ec2d1080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:53.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.413+0000 7f4cef400640 1 --2- 172.21.15.16:0/146512241 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5589eb7f4000 0x5589ec2d1b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:53.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.413+0000 7f4cefe00640 1 -- 172.21.15.16:0/146512241 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5589ec425c00 msgr2=0x5589ec2d1080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:53.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.413+0000 7f4cefe00640 1 --2- 172.21.15.16:0/146512241 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5589ec425c00 0x5589ec2d1080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:53.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.413+0000 7f4cefe00640 1 -- 172.21.15.16:0/146512241 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5589eb7f4000 msgr2=0x5589ec2d1b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:53.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.413+0000 7f4cefe00640 1 --2- 172.21.15.16:0/146512241 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5589eb7f4000 0x5589ec2d1b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:53.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.413+0000 7f4cefe00640 1 -- 172.21.15.16:0/146512241 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5589eb5294a0 con 0x5589ec425800 2024-03-31T15:14:53.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.413+0000 7f4cefe00640 1 --2- 172.21.15.16:0/146512241 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5589ec425800 0x5589ec2d1600 secure :-1 s=READY pgs=1817 cs=0 l=1 rev1=1 crypto rx=0x5589ec2f1ad0 tx=0x5589ec28b740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:53.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.417+0000 7f4ce8600640 1 -- 172.21.15.16:0/146512241 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x5589eb538340 con 0x5589ec425800 2024-03-31T15:14:53.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.417+0000 7f4ce8600640 1 -- 172.21.15.16:0/146512241 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x5589eb54e540 con 0x5589ec425800 2024-03-31T15:14:53.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.417+0000 7f4ce8600640 1 -- 172.21.15.16:0/146512241 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x5589eb5389c0 con 0x5589ec425800 2024-03-31T15:14:53.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.417+0000 7f4cf1db98c0 1 -- 172.21.15.16:0/146512241 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x5589eb5283c0 con 0x5589ec425800 2024-03-31T15:14:53.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.417+0000 7f4cf1db98c0 1 -- 172.21.15.16:0/146512241 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5589eb529c20 con 0x5589ec425800 2024-03-31T15:14:53.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.417+0000 7f4cf1db98c0 4 datascan.init: resolving metadata pool 16 2024-03-31T15:14:53.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.417+0000 7f4ce8600640 1 -- 172.21.15.16:0/146512241 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x5589ec48ac00 con 0x5589ec425800 2024-03-31T15:14:53.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.417+0000 7f4ce8600640 1 --2- 172.21.15.16:0/146512241 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x5589eb7f9000 0x5589ec2d2100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:53.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.417+0000 7f4ce8600640 1 -- 172.21.15.16:0/146512241 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 5939+0+0 (secure 0 0 0) 0x5589eb800200 con 0x5589ec425800 2024-03-31T15:14:53.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.417+0000 7f4cf1db98c0 4 datascan.init: found metadata pool 'cephfs_metadata' 2024-03-31T15:14:53.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.417+0000 7f4cf1db98c0 4 datascan.main: data pool 'cephfs_data' has ID 17 2024-03-31T15:14:53.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.417+0000 7f4cf1db98c0 4 datascan.main: opening data pool 'cephfs_data' 2024-03-31T15:14:53.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.417+0000 7f4cf1db98c0 1 --2- 172.21.15.16:0/146512241 >> [v2:172.21.15.16:6800/932712244,v1:172.21.15.16:6801/932712244] conn(0x5589eb7f4400 0x5589ec2d2680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:53.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.417+0000 7f4cf1db98c0 1 -- 172.21.15.16:0/146512241 --> [v2:172.21.15.16:6800/932712244,v1:172.21.15.16:6801/932712244] -- osd_op(unknown.0.0:1 16.14 16:287034c4:::1.00000000.inode:head [stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e113) -- 0x5589eb7f4800 con 0x5589eb7f4400 2024-03-31T15:14:53.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.417+0000 7f4cf0800640 1 --2- 172.21.15.16:0/146512241 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x5589eb7f9000 0x5589ec2d2100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:53.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.417+0000 7f4cef400640 1 --2- 172.21.15.16:0/146512241 >> [v2:172.21.15.16:6800/932712244,v1:172.21.15.16:6801/932712244] conn(0x5589eb7f4400 0x5589ec2d2680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:53.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.417+0000 7f4cf0800640 1 --2- 172.21.15.16:0/146512241 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x5589eb7f9000 0x5589ec2d2100 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x5589ec284cf0 tx=0x5589eb7fe180 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:53.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.417+0000 7f4cef400640 1 --2- 172.21.15.16:0/146512241 >> [v2:172.21.15.16:6800/932712244,v1:172.21.15.16:6801/932712244] conn(0x5589eb7f4400 0x5589ec2d2680 crc :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:53.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.417+0000 7f4cef400640 1 -- 172.21.15.16:0/146512241 <== osd.0 v2:172.21.15.16:6800/932712244 1 ==== osd_op_reply(1 1.00000000.inode [stat out=16b] v0'0 uv36 ondisk = 0) ==== 160+0+16 (crc 0 0 0) 0x5589eb5fa000 con 0x5589eb7f4400 2024-03-31T15:14:53.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.417+0000 7f4cf1db98c0 1 --2- 172.21.15.16:0/146512241 >> [v2:172.21.15.89:6810/50395527,v1:172.21.15.89:6814/50395527] conn(0x5589eb7f4c00 0x5589ec2d2c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:53.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.417+0000 7f4cf1db98c0 1 -- 172.21.15.16:0/146512241 --> [v2:172.21.15.89:6810/50395527,v1:172.21.15.89:6814/50395527] -- osd_op(unknown.0.0:2 16.1f 16:fe07bba1:::100.00000000.inode:head [stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e113) -- 0x5589eb7f5000 con 0x5589eb7f4c00 2024-03-31T15:14:53.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.421+0000 7f4cf0800640 1 --2- 172.21.15.16:0/146512241 >> [v2:172.21.15.89:6810/50395527,v1:172.21.15.89:6814/50395527] conn(0x5589eb7f4c00 0x5589ec2d2c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:53.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.421+0000 7f4cf0800640 1 --2- 172.21.15.16:0/146512241 >> [v2:172.21.15.89:6810/50395527,v1:172.21.15.89:6814/50395527] conn(0x5589eb7f4c00 0x5589ec2d2c00 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:53.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.421+0000 7f4cf0800640 1 -- 172.21.15.16:0/146512241 <== osd.5 v2:172.21.15.89:6810/50395527 1 ==== osd_op_reply(2 100.00000000.inode [stat out=16b] v0'0 uv3 ondisk = 0) ==== 162+0+16 (crc 0 0 0) 0x5589eb5fa500 con 0x5589eb7f4c00 2024-03-31T15:14:53.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.421+0000 7f4cf1db98c0 1 --2- 172.21.15.16:0/146512241 >> [v2:172.21.15.89:6808/1796778630,v1:172.21.15.89:6811/1796778630] conn(0x5589eb7f5400 0x5589ec2d3180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:53.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.421+0000 7f4cf1db98c0 1 -- 172.21.15.16:0/146512241 --> [v2:172.21.15.89:6808/1796778630,v1:172.21.15.89:6811/1796778630] -- osd_op(unknown.0.0:3 17.0 17:00000000::::head [pgnls-filter start_epoch 113 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x5589eb7f5800 con 0x5589eb7f5400 2024-03-31T15:14:53.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.421+0000 7f4cefe00640 1 --2- 172.21.15.16:0/146512241 >> [v2:172.21.15.89:6808/1796778630,v1:172.21.15.89:6811/1796778630] conn(0x5589eb7f5400 0x5589ec2d3180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:53.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.421+0000 7f4cefe00640 1 --2- 172.21.15.16:0/146512241 >> [v2:172.21.15.89:6808/1796778630,v1:172.21.15.89:6811/1796778630] conn(0x5589eb7f5400 0x5589ec2d3180 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:53.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.421+0000 7f4cefe00640 1 -- 172.21.15.16:0/146512241 <== osd.7 v2:172.21.15.89:6808/1796778630 1 ==== osd_op_reply(3 [pgnls-filter start_epoch 113 out=49b] v111'16 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x5589eb5fa780 con 0x5589eb7f5400 2024-03-31T15:14:53.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.421+0000 7f4ce9a00640 1 --2- 172.21.15.16:0/146512241 >> [v2:172.21.15.16:6808/1398737533,v1:172.21.15.16:6810/1398737533] conn(0x5589eb7f9c00 0x5589eb8f6000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:53.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.421+0000 7f4ce9a00640 1 -- 172.21.15.16:0/146512241 --> [v2:172.21.15.16:6808/1398737533,v1:172.21.15.16:6810/1398737533] -- osd_op(unknown.0.0:4 17.20 17:04000000::::head [pgnls-filter start_epoch 113 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x5589eb900000 con 0x5589eb7f9c00 2024-03-31T15:14:53.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.421+0000 7f4cef400640 1 --2- 172.21.15.16:0/146512241 >> [v2:172.21.15.16:6808/1398737533,v1:172.21.15.16:6810/1398737533] conn(0x5589eb7f9c00 0x5589eb8f6000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:53.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.421+0000 7f4cef400640 1 --2- 172.21.15.16:0/146512241 >> [v2:172.21.15.16:6808/1398737533,v1:172.21.15.16:6810/1398737533] conn(0x5589eb7f9c00 0x5589eb8f6000 crc :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:53.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.421+0000 7f4cef400640 1 -- 172.21.15.16:0/146512241 <== osd.1 v2:172.21.15.16:6808/1398737533 1 ==== osd_op_reply(4 [pgnls-filter start_epoch 113 out=49b] v111'18 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x5589eb5fa000 con 0x5589eb7f9c00 2024-03-31T15:14:53.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.421+0000 7f4cea400640 1 --2- 172.21.15.16:0/146512241 >> [v2:172.21.15.16:6809/3360352974,v1:172.21.15.16:6812/3360352974] conn(0x5589ec2fb400 0x5589eb8f6580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:53.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.421+0000 7f4cea400640 1 -- 172.21.15.16:0/146512241 --> [v2:172.21.15.16:6809/3360352974,v1:172.21.15.16:6812/3360352974] -- osd_op(unknown.0.0:5 17.10 17:08000000::::head [pgnls-filter start_epoch 113 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x5589ec2fa800 con 0x5589ec2fb400 2024-03-31T15:14:53.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.421+0000 7f4cf0800640 1 --2- 172.21.15.16:0/146512241 >> [v2:172.21.15.16:6809/3360352974,v1:172.21.15.16:6812/3360352974] conn(0x5589ec2fb400 0x5589eb8f6580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:53.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.425+0000 7f4cf0800640 1 --2- 172.21.15.16:0/146512241 >> [v2:172.21.15.16:6809/3360352974,v1:172.21.15.16:6812/3360352974] conn(0x5589ec2fb400 0x5589eb8f6580 crc :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:53.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.425+0000 7f4cf0800640 1 -- 172.21.15.16:0/146512241 <== osd.3 v2:172.21.15.16:6809/3360352974 1 ==== osd_op_reply(5 [pgnls-filter start_epoch 113 out=49b] v111'12 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x5589eb5fa500 con 0x5589ec2fb400 2024-03-31T15:14:53.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.425+0000 7f4ce9a00640 1 -- 172.21.15.16:0/146512241 --> [v2:172.21.15.16:6800/932712244,v1:172.21.15.16:6801/932712244] -- osd_op(unknown.0.0:6 17.30 17:0c000000::::head [pgnls-filter start_epoch 113 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x5589eb900400 con 0x5589eb7f4400 2024-03-31T15:14:53.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.425+0000 7f4cef400640 1 -- 172.21.15.16:0/146512241 <== osd.0 v2:172.21.15.16:6800/932712244 2 ==== osd_op_reply(6 [pgnls-filter start_epoch 113 out=49b] v111'18 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x5589eb5fa000 con 0x5589eb7f4400 2024-03-31T15:14:53.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.425+0000 7f4cea400640 1 -- 172.21.15.16:0/146512241 --> [v2:172.21.15.89:6808/1796778630,v1:172.21.15.89:6811/1796778630] -- osd_op(unknown.0.0:7 17.8 17:10000000::::head [pgnls-filter start_epoch 113 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x5589ec2fb000 con 0x5589eb7f5400 2024-03-31T15:14:53.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.425+0000 7f4cefe00640 1 -- 172.21.15.16:0/146512241 <== osd.7 v2:172.21.15.89:6808/1796778630 2 ==== osd_op_reply(7 [pgnls-filter start_epoch 113 out=49b] v111'18 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x5589eb5fa780 con 0x5589eb7f5400 2024-03-31T15:14:53.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.425+0000 7f4ce9a00640 1 -- 172.21.15.16:0/146512241 --> [v2:172.21.15.16:6809/3360352974,v1:172.21.15.16:6812/3360352974] -- osd_op(unknown.0.0:8 17.28 17:14000000::::head [pgnls-filter start_epoch 113 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x5589eb900000 con 0x5589ec2fb400 2024-03-31T15:14:53.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.425+0000 7f4cf0800640 1 -- 172.21.15.16:0/146512241 <== osd.3 v2:172.21.15.16:6809/3360352974 2 ==== osd_op_reply(8 [pgnls-filter start_epoch 113 out=49b] v111'12 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x5589eb5fa500 con 0x5589ec2fb400 2024-03-31T15:14:53.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.425+0000 7f4cea400640 1 -- 172.21.15.16:0/146512241 --> [v2:172.21.15.16:6800/932712244,v1:172.21.15.16:6801/932712244] -- osd_op(unknown.0.0:9 17.18 17:18000000::::head [pgnls-filter start_epoch 113 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x5589eb900c00 con 0x5589eb7f4400 2024-03-31T15:14:53.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.425+0000 7f4cef400640 1 -- 172.21.15.16:0/146512241 <== osd.0 v2:172.21.15.16:6800/932712244 3 ==== osd_op_reply(9 [pgnls-filter start_epoch 113 out=49b] v111'16 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x5589eb5fa000 con 0x5589eb7f4400 2024-03-31T15:14:53.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.425+0000 7f4ce9a00640 1 -- 172.21.15.16:0/146512241 --> [v2:172.21.15.89:6808/1796778630,v1:172.21.15.89:6811/1796778630] -- osd_op(unknown.0.0:10 17.38 17:1c000000::::head [pgnls-filter start_epoch 113 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x5589eb900800 con 0x5589eb7f5400 2024-03-31T15:14:53.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.425+0000 7f4cefe00640 1 -- 172.21.15.16:0/146512241 <== osd.7 v2:172.21.15.89:6808/1796778630 3 ==== osd_op_reply(10 [pgnls-filter start_epoch 113 out=49b] v111'18 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x5589eb5fa780 con 0x5589eb7f5400 2024-03-31T15:14:53.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.425+0000 7f4cea400640 1 -- 172.21.15.16:0/146512241 --> [v2:172.21.15.16:6800/932712244,v1:172.21.15.16:6801/932712244] -- osd_op(unknown.0.0:11 17.4 17:20000000::::head [pgnls-filter start_epoch 113 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x5589eb901000 con 0x5589eb7f4400 2024-03-31T15:14:53.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.425+0000 7f4cef400640 1 -- 172.21.15.16:0/146512241 <== osd.0 v2:172.21.15.16:6800/932712244 4 ==== osd_op_reply(11 [pgnls-filter start_epoch 113 out=49b] v111'18 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x5589eb5fa000 con 0x5589eb7f4400 2024-03-31T15:14:53.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.425+0000 7f4ce9a00640 1 -- 172.21.15.16:0/146512241 --> [v2:172.21.15.16:6809/3360352974,v1:172.21.15.16:6812/3360352974] -- osd_op(unknown.0.0:12 17.24 17:24000000::::head [pgnls-filter start_epoch 113 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x5589eb7f4800 con 0x5589ec2fb400 2024-03-31T15:14:53.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.425+0000 7f4cf0800640 1 -- 172.21.15.16:0/146512241 <== osd.3 v2:172.21.15.16:6809/3360352974 3 ==== osd_op_reply(12 [pgnls-filter start_epoch 113 out=49b] v111'8 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x5589eb5fa500 con 0x5589ec2fb400 2024-03-31T15:14:53.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.425+0000 7f4cea400640 1 -- 172.21.15.16:0/146512241 --> [v2:172.21.15.16:6809/3360352974,v1:172.21.15.16:6812/3360352974] -- osd_op(unknown.0.0:13 17.14 17:28000000::::head [pgnls-filter start_epoch 113 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x5589eb901400 con 0x5589ec2fb400 2024-03-31T15:14:53.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.425+0000 7f4cf0800640 1 -- 172.21.15.16:0/146512241 <== osd.3 v2:172.21.15.16:6809/3360352974 4 ==== osd_op_reply(13 [pgnls-filter start_epoch 113 out=49b] v111'22 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x5589eb5fa500 con 0x5589ec2fb400 2024-03-31T15:14:53.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.425+0000 7f4ce9a00640 1 -- 172.21.15.16:0/146512241 --> [v2:172.21.15.16:6800/932712244,v1:172.21.15.16:6801/932712244] -- osd_op(unknown.0.0:14 17.34 17:2c000000::::head [pgnls-filter start_epoch 113 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x5589eb7f5000 con 0x5589eb7f4400 2024-03-31T15:14:53.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.425+0000 7f4cef400640 1 -- 172.21.15.16:0/146512241 <== osd.0 v2:172.21.15.16:6800/932712244 5 ==== osd_op_reply(14 [pgnls-filter start_epoch 113 out=49b] v111'14 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x5589eb5fa000 con 0x5589eb7f4400 2024-03-31T15:14:53.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.425+0000 7f4cea400640 1 -- 172.21.15.16:0/146512241 --> [v2:172.21.15.89:6808/1796778630,v1:172.21.15.89:6811/1796778630] -- osd_op(unknown.0.0:15 17.c 17:30000000::::head [pgnls-filter start_epoch 113 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x5589ec2fa800 con 0x5589eb7f5400 2024-03-31T15:14:53.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.429+0000 7f4cefe00640 1 -- 172.21.15.16:0/146512241 <== osd.7 v2:172.21.15.89:6808/1796778630 4 ==== osd_op_reply(15 [pgnls-filter start_epoch 113 out=49b] v111'12 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x5589eb5fa780 con 0x5589eb7f5400 2024-03-31T15:14:53.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.429+0000 7f4ce9a00640 1 --2- 172.21.15.16:0/146512241 >> [v2:172.21.15.89:6800/1856844856,v1:172.21.15.89:6801/1856844856] conn(0x5589eb7f8c00 0x5589eb8f6b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:53.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.429+0000 7f4ce9a00640 1 -- 172.21.15.16:0/146512241 --> [v2:172.21.15.89:6800/1856844856,v1:172.21.15.89:6801/1856844856] -- osd_op(unknown.0.0:16 17.2c 17:34000000::::head [pgnls-filter start_epoch 113 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x5589eb7f8800 con 0x5589eb7f8c00 2024-03-31T15:14:53.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.429+0000 7f4cefe00640 1 --2- 172.21.15.16:0/146512241 >> [v2:172.21.15.89:6800/1856844856,v1:172.21.15.89:6801/1856844856] conn(0x5589eb7f8c00 0x5589eb8f6b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:53.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.429+0000 7f4cefe00640 1 --2- 172.21.15.16:0/146512241 >> [v2:172.21.15.89:6800/1856844856,v1:172.21.15.89:6801/1856844856] conn(0x5589eb7f8c00 0x5589eb8f6b00 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:53.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.429+0000 7f4cefe00640 1 -- 172.21.15.16:0/146512241 <== osd.4 v2:172.21.15.89:6800/1856844856 1 ==== osd_op_reply(16 [pgnls-filter start_epoch 113 out=49b] v111'16 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x5589eb5fa780 con 0x5589eb7f8c00 2024-03-31T15:14:53.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.429+0000 7f4cea400640 1 -- 172.21.15.16:0/146512241 --> [v2:172.21.15.16:6808/1398737533,v1:172.21.15.16:6810/1398737533] -- osd_op(unknown.0.0:17 17.1c 17:38000000::::head [pgnls-filter start_epoch 113 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x5589ec2fb000 con 0x5589eb7f9c00 2024-03-31T15:14:53.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.429+0000 7f4cef400640 1 -- 172.21.15.16:0/146512241 <== osd.1 v2:172.21.15.16:6808/1398737533 2 ==== osd_op_reply(17 [pgnls-filter start_epoch 113 out=49b] v111'18 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x5589eb5fa000 con 0x5589eb7f9c00 2024-03-31T15:14:53.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.429+0000 7f4ce9a00640 1 -- 172.21.15.16:0/146512241 --> [v2:172.21.15.16:6800/932712244,v1:172.21.15.16:6801/932712244] -- osd_op(unknown.0.0:18 17.3c 17:3c000000::::head [pgnls-filter start_epoch 113 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x5589eb7f8400 con 0x5589eb7f4400 2024-03-31T15:14:53.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.429+0000 7f4cef400640 1 -- 172.21.15.16:0/146512241 <== osd.0 v2:172.21.15.16:6800/932712244 6 ==== osd_op_reply(18 [pgnls-filter start_epoch 113 out=49b] v111'16 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x5589eb5fa000 con 0x5589eb7f4400 2024-03-31T15:14:53.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.429+0000 7f4cea400640 1 -- 172.21.15.16:0/146512241 --> [v2:172.21.15.89:6800/1856844856,v1:172.21.15.89:6801/1856844856] -- osd_op(unknown.0.0:19 17.2 17:40000000::::head [pgnls-filter start_epoch 113 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x5589eb7f4800 con 0x5589eb7f8c00 2024-03-31T15:14:53.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.429+0000 7f4cefe00640 1 -- 172.21.15.16:0/146512241 <== osd.4 v2:172.21.15.89:6800/1856844856 2 ==== osd_op_reply(19 [pgnls-filter start_epoch 113 out=49b] v111'16 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x5589eb5fa780 con 0x5589eb7f8c00 2024-03-31T15:14:53.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.429+0000 7f4ce9a00640 1 -- 172.21.15.16:0/146512241 --> [v2:172.21.15.89:6810/50395527,v1:172.21.15.89:6814/50395527] -- osd_op(unknown.0.0:20 17.22 17:44000000::::head [pgnls-filter start_epoch 113 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x5589eb900400 con 0x5589eb7f4c00 2024-03-31T15:14:53.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.429+0000 7f4cf0800640 1 -- 172.21.15.16:0/146512241 <== osd.5 v2:172.21.15.89:6810/50395527 2 ==== osd_op_reply(20 [pgnls-filter start_epoch 113 out=49b] v111'16 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x5589eb5fa500 con 0x5589eb7f4c00 2024-03-31T15:14:53.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.429+0000 7f4cea400640 1 -- 172.21.15.16:0/146512241 --> [v2:172.21.15.16:6809/3360352974,v1:172.21.15.16:6812/3360352974] -- osd_op(unknown.0.0:21 17.12 17:48000000::::head [pgnls-filter start_epoch 113 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x5589eb7f5800 con 0x5589ec2fb400 2024-03-31T15:14:53.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.429+0000 7f4cf0800640 1 -- 172.21.15.16:0/146512241 <== osd.3 v2:172.21.15.16:6809/3360352974 5 ==== osd_op_reply(21 [pgnls-filter start_epoch 113 out=81b] v113'23 uv23 ondisk = 1) ==== 144+0+81 (crc 0 0 0) 0x5589eb5fa500 con 0x5589ec2fb400 2024-03-31T15:14:53.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.429+0000 7f4cf1db98c0 1 -- 172.21.15.16:0/146512241 --> [v2:172.21.15.89:6808/1796778630,v1:172.21.15.89:6811/1796778630] -- osd_op(unknown.0.0:22 17.0 17:00000000::::head [pgnls-filter start_epoch 113 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x5589eb7f5c00 con 0x5589eb7f5400 2024-03-31T15:14:53.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.429+0000 7f4cefe00640 1 -- 172.21.15.16:0/146512241 <== osd.7 v2:172.21.15.89:6808/1796778630 5 ==== osd_op_reply(22 [pgnls-filter start_epoch 113 out=49b] v111'16 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x5589eb5fa780 con 0x5589eb7f5400 2024-03-31T15:14:53.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.429+0000 7f4cea400640 1 -- 172.21.15.16:0/146512241 --> [v2:172.21.15.16:6808/1398737533,v1:172.21.15.16:6810/1398737533] -- osd_op(unknown.0.0:23 17.20 17:04000000::::head [pgnls-filter start_epoch 113 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x5589eb901000 con 0x5589eb7f9c00 2024-03-31T15:14:53.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.429+0000 7f4cef400640 1 -- 172.21.15.16:0/146512241 <== osd.1 v2:172.21.15.16:6808/1398737533 3 ==== osd_op_reply(23 [pgnls-filter start_epoch 113 out=49b] v111'18 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x5589eb5fa000 con 0x5589eb7f9c00 2024-03-31T15:14:53.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.429+0000 7f4ce9a00640 1 -- 172.21.15.16:0/146512241 --> [v2:172.21.15.16:6809/3360352974,v1:172.21.15.16:6812/3360352974] -- osd_op(unknown.0.0:24 17.10 17:08000000::::head [pgnls-filter start_epoch 113 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x5589eb900c00 con 0x5589ec2fb400 2024-03-31T15:14:53.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.433+0000 7f4cf0800640 1 -- 172.21.15.16:0/146512241 <== osd.3 v2:172.21.15.16:6809/3360352974 6 ==== osd_op_reply(24 [pgnls-filter start_epoch 113 out=49b] v111'12 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x5589eb5fa500 con 0x5589ec2fb400 2024-03-31T15:14:53.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.433+0000 7f4cea400640 1 -- 172.21.15.16:0/146512241 --> [v2:172.21.15.16:6800/932712244,v1:172.21.15.16:6801/932712244] -- osd_op(unknown.0.0:25 17.30 17:0c000000::::head [pgnls-filter start_epoch 113 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x5589eb900000 con 0x5589eb7f4400 2024-03-31T15:14:53.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.433+0000 7f4cef400640 1 -- 172.21.15.16:0/146512241 <== osd.0 v2:172.21.15.16:6800/932712244 7 ==== osd_op_reply(25 [pgnls-filter start_epoch 113 out=49b] v111'18 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x5589eb5fa000 con 0x5589eb7f4400 2024-03-31T15:14:53.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.433+0000 7f4ce9a00640 1 -- 172.21.15.16:0/146512241 --> [v2:172.21.15.89:6808/1796778630,v1:172.21.15.89:6811/1796778630] -- osd_op(unknown.0.0:26 17.8 17:10000000::::head [pgnls-filter start_epoch 113 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x5589eb7f8400 con 0x5589eb7f5400 2024-03-31T15:14:53.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.433+0000 7f4cefe00640 1 -- 172.21.15.16:0/146512241 <== osd.7 v2:172.21.15.89:6808/1796778630 6 ==== osd_op_reply(26 [pgnls-filter start_epoch 113 out=49b] v111'18 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x5589eb5fa780 con 0x5589eb7f5400 2024-03-31T15:14:53.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.433+0000 7f4cea400640 1 -- 172.21.15.16:0/146512241 --> [v2:172.21.15.16:6809/3360352974,v1:172.21.15.16:6812/3360352974] -- osd_op(unknown.0.0:27 17.28 17:14000000::::head [pgnls-filter start_epoch 113 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x5589eb900800 con 0x5589ec2fb400 2024-03-31T15:14:53.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.433+0000 7f4cf0800640 1 -- 172.21.15.16:0/146512241 <== osd.3 v2:172.21.15.16:6809/3360352974 7 ==== osd_op_reply(27 [pgnls-filter start_epoch 113 out=49b] v111'12 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x5589eb5fa500 con 0x5589ec2fb400 2024-03-31T15:14:53.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.433+0000 7f4ce9a00640 1 -- 172.21.15.16:0/146512241 --> [v2:172.21.15.16:6800/932712244,v1:172.21.15.16:6801/932712244] -- osd_op(unknown.0.0:28 17.18 17:18000000::::head [pgnls-filter start_epoch 113 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x5589ec2fb000 con 0x5589eb7f4400 2024-03-31T15:14:53.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.433+0000 7f4cef400640 1 -- 172.21.15.16:0/146512241 <== osd.0 v2:172.21.15.16:6800/932712244 8 ==== osd_op_reply(28 [pgnls-filter start_epoch 113 out=49b] v111'16 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x5589eb5fa000 con 0x5589eb7f4400 2024-03-31T15:14:53.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.433+0000 7f4cea400640 1 -- 172.21.15.16:0/146512241 --> [v2:172.21.15.89:6808/1796778630,v1:172.21.15.89:6811/1796778630] -- osd_op(unknown.0.0:29 17.38 17:1c000000::::head [pgnls-filter start_epoch 113 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x5589eb901800 con 0x5589eb7f5400 2024-03-31T15:14:53.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.433+0000 7f4cefe00640 1 -- 172.21.15.16:0/146512241 <== osd.7 v2:172.21.15.89:6808/1796778630 7 ==== osd_op_reply(29 [pgnls-filter start_epoch 113 out=49b] v111'18 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x5589eb5fa780 con 0x5589eb7f5400 2024-03-31T15:14:53.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.433+0000 7f4ce9a00640 1 -- 172.21.15.16:0/146512241 --> [v2:172.21.15.16:6800/932712244,v1:172.21.15.16:6801/932712244] -- osd_op(unknown.0.0:30 17.4 17:20000000::::head [pgnls-filter start_epoch 113 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x5589eb7f5000 con 0x5589eb7f4400 2024-03-31T15:14:53.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.433+0000 7f4cef400640 1 -- 172.21.15.16:0/146512241 <== osd.0 v2:172.21.15.16:6800/932712244 9 ==== osd_op_reply(30 [pgnls-filter start_epoch 113 out=49b] v111'18 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x5589eb5fa000 con 0x5589eb7f4400 2024-03-31T15:14:53.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.433+0000 7f4cea400640 1 -- 172.21.15.16:0/146512241 --> [v2:172.21.15.16:6809/3360352974,v1:172.21.15.16:6812/3360352974] -- osd_op(unknown.0.0:31 17.24 17:24000000::::head [pgnls-filter start_epoch 113 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x5589eb901c00 con 0x5589ec2fb400 2024-03-31T15:14:53.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.433+0000 7f4cf0800640 1 -- 172.21.15.16:0/146512241 <== osd.3 v2:172.21.15.16:6809/3360352974 8 ==== osd_op_reply(31 [pgnls-filter start_epoch 113 out=49b] v111'8 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x5589eb5fa500 con 0x5589ec2fb400 2024-03-31T15:14:53.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.433+0000 7f4ce9a00640 1 -- 172.21.15.16:0/146512241 --> [v2:172.21.15.16:6809/3360352974,v1:172.21.15.16:6812/3360352974] -- osd_op(unknown.0.0:32 17.14 17:28000000::::head [pgnls-filter start_epoch 113 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x5589eb901400 con 0x5589ec2fb400 2024-03-31T15:14:53.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.433+0000 7f4cf0800640 1 -- 172.21.15.16:0/146512241 <== osd.3 v2:172.21.15.16:6809/3360352974 9 ==== osd_op_reply(32 [pgnls-filter start_epoch 113 out=49b] v111'22 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x5589eb5fa500 con 0x5589ec2fb400 2024-03-31T15:14:53.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.433+0000 7f4cea400640 1 -- 172.21.15.16:0/146512241 --> [v2:172.21.15.16:6800/932712244,v1:172.21.15.16:6801/932712244] -- osd_op(unknown.0.0:33 17.34 17:2c000000::::head [pgnls-filter start_epoch 113 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x5589eb916000 con 0x5589eb7f4400 2024-03-31T15:14:53.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.433+0000 7f4cef400640 1 -- 172.21.15.16:0/146512241 <== osd.0 v2:172.21.15.16:6800/932712244 10 ==== osd_op_reply(33 [pgnls-filter start_epoch 113 out=49b] v111'14 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x5589eb5fa000 con 0x5589eb7f4400 2024-03-31T15:14:53.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.433+0000 7f4ce9a00640 1 -- 172.21.15.16:0/146512241 --> [v2:172.21.15.89:6808/1796778630,v1:172.21.15.89:6811/1796778630] -- osd_op(unknown.0.0:34 17.c 17:30000000::::head [pgnls-filter start_epoch 113 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x5589eb900400 con 0x5589eb7f5400 2024-03-31T15:14:53.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.433+0000 7f4cefe00640 1 -- 172.21.15.16:0/146512241 <== osd.7 v2:172.21.15.89:6808/1796778630 8 ==== osd_op_reply(34 [pgnls-filter start_epoch 113 out=49b] v111'12 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x5589eb5fa780 con 0x5589eb7f5400 2024-03-31T15:14:53.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.433+0000 7f4cea400640 1 -- 172.21.15.16:0/146512241 --> [v2:172.21.15.89:6800/1856844856,v1:172.21.15.89:6801/1856844856] -- osd_op(unknown.0.0:35 17.2c 17:34000000::::head [pgnls-filter start_epoch 113 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x5589eb916400 con 0x5589eb7f8c00 2024-03-31T15:14:53.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.433+0000 7f4cefe00640 1 -- 172.21.15.16:0/146512241 <== osd.4 v2:172.21.15.89:6800/1856844856 3 ==== osd_op_reply(35 [pgnls-filter start_epoch 113 out=49b] v111'16 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x5589eb5fa780 con 0x5589eb7f8c00 2024-03-31T15:14:53.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.433+0000 7f4ce9a00640 1 -- 172.21.15.16:0/146512241 --> [v2:172.21.15.16:6808/1398737533,v1:172.21.15.16:6810/1398737533] -- osd_op(unknown.0.0:36 17.1c 17:38000000::::head [pgnls-filter start_epoch 113 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x5589eb900c00 con 0x5589eb7f9c00 2024-03-31T15:14:53.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.437+0000 7f4cef400640 1 -- 172.21.15.16:0/146512241 <== osd.1 v2:172.21.15.16:6808/1398737533 4 ==== osd_op_reply(36 [pgnls-filter start_epoch 113 out=49b] v111'18 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x5589eb5fa000 con 0x5589eb7f9c00 2024-03-31T15:14:53.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.437+0000 7f4cea400640 1 -- 172.21.15.16:0/146512241 --> [v2:172.21.15.16:6800/932712244,v1:172.21.15.16:6801/932712244] -- osd_op(unknown.0.0:37 17.3c 17:3c000000::::head [pgnls-filter start_epoch 113 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x5589eb916800 con 0x5589eb7f4400 2024-03-31T15:14:53.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.437+0000 7f4cef400640 1 -- 172.21.15.16:0/146512241 <== osd.0 v2:172.21.15.16:6800/932712244 11 ==== osd_op_reply(37 [pgnls-filter start_epoch 113 out=49b] v111'16 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x5589eb5fa000 con 0x5589eb7f4400 2024-03-31T15:14:53.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.437+0000 7f4ce9a00640 1 -- 172.21.15.16:0/146512241 --> [v2:172.21.15.89:6800/1856844856,v1:172.21.15.89:6801/1856844856] -- osd_op(unknown.0.0:38 17.2 17:40000000::::head [pgnls-filter start_epoch 113 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x5589eb900800 con 0x5589eb7f8c00 2024-03-31T15:14:53.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.437+0000 7f4cefe00640 1 -- 172.21.15.16:0/146512241 <== osd.4 v2:172.21.15.89:6800/1856844856 4 ==== osd_op_reply(38 [pgnls-filter start_epoch 113 out=49b] v111'16 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x5589eb5fa780 con 0x5589eb7f8c00 2024-03-31T15:14:53.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.437+0000 7f4cea400640 1 -- 172.21.15.16:0/146512241 --> [v2:172.21.15.89:6810/50395527,v1:172.21.15.89:6814/50395527] -- osd_op(unknown.0.0:39 17.22 17:44000000::::head [pgnls-filter start_epoch 113 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x5589eb916c00 con 0x5589eb7f4c00 2024-03-31T15:14:53.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.437+0000 7f4cf0800640 1 -- 172.21.15.16:0/146512241 <== osd.5 v2:172.21.15.89:6810/50395527 3 ==== osd_op_reply(39 [pgnls-filter start_epoch 113 out=49b] v111'16 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x5589eb5fa500 con 0x5589eb7f4c00 2024-03-31T15:14:53.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.437+0000 7f4ce9a00640 1 -- 172.21.15.16:0/146512241 --> [v2:172.21.15.16:6809/3360352974,v1:172.21.15.16:6812/3360352974] -- osd_op(unknown.0.0:40 17.12 17:48000000::::head [pgnls-filter start_epoch 113 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x5589eb7f5800 con 0x5589ec2fb400 2024-03-31T15:14:53.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.437+0000 7f4cf0800640 1 -- 172.21.15.16:0/146512241 <== osd.3 v2:172.21.15.16:6809/3360352974 10 ==== osd_op_reply(40 [pgnls-filter start_epoch 113 out=49b] v113'23 uv23 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x5589eb5fa500 con 0x5589ec2fb400 2024-03-31T15:14:53.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.437+0000 7f4cea400640 1 -- 172.21.15.16:0/146512241 --> [v2:172.21.15.89:6808/1796778630,v1:172.21.15.89:6811/1796778630] -- osd_op(unknown.0.0:41 17.32 17:4c000000::::head [pgnls-filter start_epoch 113 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x5589eb917000 con 0x5589eb7f5400 2024-03-31T15:14:53.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.437+0000 7f4cefe00640 1 -- 172.21.15.16:0/146512241 <== osd.7 v2:172.21.15.89:6808/1796778630 9 ==== osd_op_reply(41 [pgnls-filter start_epoch 113 out=49b] v111'15 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x5589eb5fa780 con 0x5589eb7f5400 2024-03-31T15:14:53.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.437+0000 7f4ce9a00640 1 -- 172.21.15.16:0/146512241 --> [v2:172.21.15.89:6808/1796778630,v1:172.21.15.89:6811/1796778630] -- osd_op(unknown.0.0:42 17.a 17:50000000::::head [pgnls-filter start_epoch 113 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x5589eb7f8800 con 0x5589eb7f5400 2024-03-31T15:14:53.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.437+0000 7f4cefe00640 1 -- 172.21.15.16:0/146512241 <== osd.7 v2:172.21.15.89:6808/1796778630 10 ==== osd_op_reply(42 [pgnls-filter start_epoch 113 out=49b] v111'10 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x5589eb5fa780 con 0x5589eb7f5400 2024-03-31T15:14:53.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.437+0000 7f4cea400640 1 -- 172.21.15.16:0/146512241 --> [v2:172.21.15.16:6809/3360352974,v1:172.21.15.16:6812/3360352974] -- osd_op(unknown.0.0:43 17.2a 17:54000000::::head [pgnls-filter start_epoch 113 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x5589eb917400 con 0x5589ec2fb400 2024-03-31T15:14:53.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.437+0000 7f4cf0800640 1 -- 172.21.15.16:0/146512241 <== osd.3 v2:172.21.15.16:6809/3360352974 11 ==== osd_op_reply(43 [pgnls-filter start_epoch 113 out=49b] v111'20 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x5589eb5fa500 con 0x5589ec2fb400 2024-03-31T15:14:53.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.437+0000 7f4ce9a00640 1 -- 172.21.15.16:0/146512241 --> [v2:172.21.15.16:6800/932712244,v1:172.21.15.16:6801/932712244] -- osd_op(unknown.0.0:44 17.1a 17:58000000::::head [pgnls-filter start_epoch 113 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x5589eb7f8400 con 0x5589eb7f4400 2024-03-31T15:14:53.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.437+0000 7f4cef400640 1 -- 172.21.15.16:0/146512241 <== osd.0 v2:172.21.15.16:6800/932712244 12 ==== osd_op_reply(44 [pgnls-filter start_epoch 113 out=49b] v111'14 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x5589eb5fa000 con 0x5589eb7f4400 2024-03-31T15:14:53.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.437+0000 7f4cea400640 1 -- 172.21.15.16:0/146512241 --> [v2:172.21.15.16:6808/1398737533,v1:172.21.15.16:6810/1398737533] -- osd_op(unknown.0.0:45 17.3a 17:5c000000::::head [pgnls-filter start_epoch 113 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x5589eb916000 con 0x5589eb7f9c00 2024-03-31T15:14:53.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.437+0000 7f4cef400640 1 -- 172.21.15.16:0/146512241 <== osd.1 v2:172.21.15.16:6808/1398737533 5 ==== osd_op_reply(45 [pgnls-filter start_epoch 113 out=49b] v111'12 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x5589eb5fa000 con 0x5589eb7f9c00 2024-03-31T15:14:53.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.437+0000 7f4ce9a00640 1 --2- 172.21.15.16:0/146512241 >> [v2:172.21.15.89:6809/3470471752,v1:172.21.15.89:6812/3470471752] conn(0x5589eb7f5000 0x5589eb8f7080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:53.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.437+0000 7f4ce9a00640 1 -- 172.21.15.16:0/146512241 --> [v2:172.21.15.89:6809/3470471752,v1:172.21.15.89:6812/3470471752] -- osd_op(unknown.0.0:46 17.6 17:60000000::::head [pgnls-filter start_epoch 113 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x5589eb7f4800 con 0x5589eb7f5000 2024-03-31T15:14:53.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.437+0000 7f4cef400640 1 --2- 172.21.15.16:0/146512241 >> [v2:172.21.15.89:6809/3470471752,v1:172.21.15.89:6812/3470471752] conn(0x5589eb7f5000 0x5589eb8f7080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:53.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.441+0000 7f4cef400640 1 --2- 172.21.15.16:0/146512241 >> [v2:172.21.15.89:6809/3470471752,v1:172.21.15.89:6812/3470471752] conn(0x5589eb7f5000 0x5589eb8f7080 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:53.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.441+0000 7f4cef400640 1 -- 172.21.15.16:0/146512241 <== osd.6 v2:172.21.15.89:6809/3470471752 1 ==== osd_op_reply(46 [pgnls-filter start_epoch 113 out=49b] v111'14 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x5589eb5fa000 con 0x5589eb7f5000 2024-03-31T15:14:53.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.441+0000 7f4cea400640 1 -- 172.21.15.16:0/146512241 --> [v2:172.21.15.16:6800/932712244,v1:172.21.15.16:6801/932712244] -- osd_op(unknown.0.0:47 17.26 17:64000000::::head [pgnls-filter start_epoch 113 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x5589eb916800 con 0x5589eb7f4400 2024-03-31T15:14:53.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.441+0000 7f4cef400640 1 -- 172.21.15.16:0/146512241 <== osd.0 v2:172.21.15.16:6800/932712244 13 ==== osd_op_reply(47 [pgnls-filter start_epoch 113 out=49b] v111'30 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x5589eb5fa000 con 0x5589eb7f4400 2024-03-31T15:14:53.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.441+0000 7f4ce9a00640 1 -- 172.21.15.16:0/146512241 --> [v2:172.21.15.89:6810/50395527,v1:172.21.15.89:6814/50395527] -- osd_op(unknown.0.0:48 17.16 17:68000000::::head [pgnls-filter start_epoch 113 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x5589eb7f5c00 con 0x5589eb7f4c00 2024-03-31T15:14:53.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.441+0000 7f4cf0800640 1 -- 172.21.15.16:0/146512241 <== osd.5 v2:172.21.15.89:6810/50395527 4 ==== osd_op_reply(48 [pgnls-filter start_epoch 113 out=49b] v111'20 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x5589eb5fa500 con 0x5589eb7f4c00 2024-03-31T15:14:53.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.441+0000 7f4cea400640 1 -- 172.21.15.16:0/146512241 --> [v2:172.21.15.16:6800/932712244,v1:172.21.15.16:6801/932712244] -- osd_op(unknown.0.0:49 17.36 17:6c000000::::head [pgnls-filter start_epoch 113 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x5589eb901000 con 0x5589eb7f4400 2024-03-31T15:14:53.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.441+0000 7f4cef400640 1 -- 172.21.15.16:0/146512241 <== osd.0 v2:172.21.15.16:6800/932712244 14 ==== osd_op_reply(49 [pgnls-filter start_epoch 113 out=81b] v113'34 uv34 ondisk = 1) ==== 144+0+81 (crc 0 0 0) 0x5589eb5fa000 con 0x5589eb7f4400 2024-03-31T15:14:53.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.441+0000 7f4cf1db98c0 20 datascan.forall_objects: OSD matched oid 100000001f7.00000000 2024-03-31T15:14:53.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.441+0000 7f4cf1db98c0 10 datascan.operator(): handling object 100000001f7.0 2024-03-31T15:14:53.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.441+0000 7f4cf1db98c0 1 -- 172.21.15.16:0/146512241 --> [v2:172.21.15.16:6800/932712244,v1:172.21.15.16:6801/932712244] -- osd_op(unknown.0.0:50 17.36 17:6fe38dc2:::100000001f7.00000000:head [getxattr scan_ceiling in=12b,getxattr scan_max_size in=13b,getxattr scan_max_mtime in=14b,getxattr scan_pool_id in=12b,getxattr parent in=6b,getxattr layout in=6b,getxattr symlink in=7b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e113) -- 0x5589eb917400 con 0x5589eb7f4400 2024-03-31T15:14:53.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.441+0000 7f4cef400640 1 -- 172.21.15.16:0/146512241 <== osd.0 v2:172.21.15.16:6800/932712244 15 ==== osd_op_reply(50 100000001f7.00000000 [getxattr (22) out=22b,getxattr (8) out=8b,getxattr (8) out=8b,getxattr,getxattr (126) out=126b,getxattr (30) out=30b,getxattr] v0'0 uv34 ondisk = 0) ==== 416+0+194 (crc 0 0 0) 0x5589eb5fa000 con 0x5589eb7f4400 2024-03-31T15:14:53.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.441+0000 7f4cf1db98c0 10 datascan.inject_with_backtrace: inode: 0x0x100000001f7 2024-03-31T15:14:53.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.441+0000 7f4cf1db98c0 10 datascan.inject_with_backtrace: backptr: 0x0x10000000001/bravo 2024-03-31T15:14:53.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.441+0000 7f4cf1db98c0 20 datascan.get_frag_of: dirino=0x10000000001 target_dname=bravo 2024-03-31T15:14:53.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.441+0000 7f4cf1db98c0 1 -- 172.21.15.16:0/146512241 --> [v2:172.21.15.16:6809/3360352974,v1:172.21.15.16:6812/3360352974] -- osd_op(unknown.0.0:51 16.7 16:e2e07930:::10000000001.00000000:head [getxattr parent in=6b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e113) -- 0x5589eb917800 con 0x5589ec2fb400 2024-03-31T15:14:53.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.441+0000 7f4cf0800640 1 -- 172.21.15.16:0/146512241 <== osd.3 v2:172.21.15.16:6809/3360352974 12 ==== osd_op_reply(51 10000000001.00000000 [getxattr (95) out=95b] v0'0 uv8 ondisk = 0) ==== 164+0+95 (crc 0 0 0) 0x5589eb5fa500 con 0x5589ec2fb400 2024-03-31T15:14:53.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.441+0000 7f4cf1db98c0 20 datascan.get_frag_of: got backtrace parent 0x10000000000/flushed 2024-03-31T15:14:53.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.441+0000 7f4cf1db98c0 1 --2- 172.21.15.16:0/146512241 >> [v2:172.21.15.16:6819/3018526646,v1:172.21.15.16:6822/3018526646] conn(0x5589eb917c00 0x5589ec2d3700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:53.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.441+0000 7f4cf1db98c0 1 -- 172.21.15.16:0/146512241 --> [v2:172.21.15.16:6819/3018526646,v1:172.21.15.16:6822/3018526646] -- osd_op(unknown.0.0:52 16.10 16:0cf6ad0f:::10000000000.00000000:head [omap-get-vals-by-keys in=20b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e113) -- 0x5589eb7f5800 con 0x5589eb917c00 2024-03-31T15:14:53.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.441+0000 7f4cf0800640 1 --2- 172.21.15.16:0/146512241 >> [v2:172.21.15.16:6819/3018526646,v1:172.21.15.16:6822/3018526646] conn(0x5589eb917c00 0x5589ec2d3700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:53.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.441+0000 7f4cf0800640 1 --2- 172.21.15.16:0/146512241 >> [v2:172.21.15.16:6819/3018526646,v1:172.21.15.16:6822/3018526646] conn(0x5589eb917c00 0x5589ec2d3700 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:53.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.441+0000 7f4cf0800640 1 -- 172.21.15.16:0/146512241 <== osd.2 v2:172.21.15.16:6819/3018526646 1 ==== osd_op_reply(52 10000000000.00000000 [omap-get-vals-by-keys out=524b] v0'0 uv12 ondisk = 0) ==== 164+0+524 (crc 0 0 0) 0x5589eb5fa500 con 0x5589eb917c00 2024-03-31T15:14:53.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.441+0000 7f4cf1db98c0 20 datascan.read_dentry: oid=10000000000.00000000 dname=flushed frag=*, r=0 2024-03-31T15:14:53.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.441+0000 7f4cf1db98c0 20 datascan.get_frag_of: fast path, fragtree is fragtree_t(*) 2024-03-31T15:14:53.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.441+0000 7f4cf1db98c0 20 datascan.get_frag_of: frag is * 2024-03-31T15:14:53.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.441+0000 7f4cf1db98c0 1 -- 172.21.15.16:0/146512241 --> [v2:172.21.15.16:6809/3360352974,v1:172.21.15.16:6812/3360352974] -- osd_op(unknown.0.0:53 16.7 16:e2e07930:::10000000001.00000000:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e113) -- 0x5589eb7f5c00 con 0x5589ec2fb400 2024-03-31T15:14:53.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.441+0000 7f4cf0800640 1 -- 172.21.15.16:0/146512241 <== osd.3 v2:172.21.15.16:6809/3360352974 13 ==== osd_op_reply(53 10000000001.00000000 [omap-get-header out=274b] v0'0 uv8 ondisk = 0) ==== 164+0+274 (crc 0 0 0) 0x5589eb5fa500 con 0x5589ec2fb400 2024-03-31T15:14:53.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.441+0000 7f4cf1db98c0 10 datascan.find_or_create_dirfrag: read_version = 8 2024-03-31T15:14:53.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.441+0000 7f4cf1db98c0 20 datascan.find_or_create_dirfrag: Dirfrag already exists: 0x0x10000000001 * 2024-03-31T15:14:53.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.441+0000 7f4cf1db98c0 1 -- 172.21.15.16:0/146512241 --> [v2:172.21.15.16:6809/3360352974,v1:172.21.15.16:6812/3360352974] -- osd_op(unknown.0.0:54 16.7 16:e2e07930:::10000000001.00000000:head [omap-get-vals-by-keys in=18b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e113) -- 0x5589eb901c00 con 0x5589ec2fb400 2024-03-31T15:14:53.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.445+0000 7f4cf0800640 1 -- 172.21.15.16:0/146512241 <== osd.3 v2:172.21.15.16:6809/3360352974 14 ==== osd_op_reply(54 10000000001.00000000 [omap-get-vals-by-keys out=4b] v0'0 uv8 ondisk = 0) ==== 164+0+4 (crc 0 0 0) 0x5589eb5fa500 con 0x5589ec2fb400 2024-03-31T15:14:53.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.445+0000 7f4cf1db98c0 20 datascan.read_dentry: oid=10000000001.00000000 dname=bravo frag=*, r=0 2024-03-31T15:14:53.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.445+0000 7f4cf1db98c0 20 datascan.read_dentry: bravo_head not found in result 2024-03-31T15:14:53.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.445+0000 7f4cf1db98c0 10 datascan.inject_with_backtrace: Linking inode 0x0x100000001f7 at 0x0x10000000001/bravo with size=5 bytes 2024-03-31T15:14:53.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.445+0000 7f4cf1db98c0 1 -- 172.21.15.16:0/146512241 --> [v2:172.21.15.16:6809/3360352974,v1:172.21.15.16:6812/3360352974] -- osd_op(unknown.0.0:55 16.7 16:e2e07930:::10000000001.00000000:head [omap-set-vals in=506b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x5589eb901400 con 0x5589ec2fb400 2024-03-31T15:14:53.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.445+0000 7f4cf0800640 1 -- 172.21.15.16:0/146512241 <== osd.3 v2:172.21.15.16:6809/3360352974 15 ==== osd_op_reply(55 10000000001.00000000 [omap-set-vals] v113'9 uv9 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x5589eb5fa500 con 0x5589ec2fb400 2024-03-31T15:14:53.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.445+0000 7f4cf1db98c0 20 datascan.inject_linkage: Injected dentry 0x0x10000000001/bravo pointing to 0x0x100000001f7 2024-03-31T15:14:53.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.445+0000 7f4cf1db98c0 1 -- 172.21.15.16:0/146512241 --> [v2:172.21.15.16:6819/3018526646,v1:172.21.15.16:6822/3018526646] -- osd_op(unknown.0.0:56 17.e 17:70000000::::head [pgnls-filter start_epoch 113 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x5589eb924000 con 0x5589eb917c00 2024-03-31T15:14:53.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.445+0000 7f4cf0800640 1 -- 172.21.15.16:0/146512241 <== osd.2 v2:172.21.15.16:6819/3018526646 2 ==== osd_op_reply(56 [pgnls-filter start_epoch 113 out=49b] v111'10 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x5589eb5fa500 con 0x5589eb917c00 2024-03-31T15:14:53.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.445+0000 7f4cea400640 1 -- 172.21.15.16:0/146512241 --> [v2:172.21.15.89:6809/3470471752,v1:172.21.15.89:6812/3470471752] -- osd_op(unknown.0.0:57 17.2e 17:74000000::::head [pgnls-filter start_epoch 113 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x5589eb900000 con 0x5589eb7f5000 2024-03-31T15:14:53.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.445+0000 7f4cef400640 1 -- 172.21.15.16:0/146512241 <== osd.6 v2:172.21.15.89:6809/3470471752 2 ==== osd_op_reply(57 [pgnls-filter start_epoch 113 out=49b] v111'20 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x5589eb5fa000 con 0x5589eb7f5000 2024-03-31T15:14:53.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.445+0000 7f4ce9a00640 1 -- 172.21.15.16:0/146512241 --> [v2:172.21.15.16:6808/1398737533,v1:172.21.15.16:6810/1398737533] -- osd_op(unknown.0.0:58 17.1e 17:78000000::::head [pgnls-filter start_epoch 113 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x5589ec2fb000 con 0x5589eb7f9c00 2024-03-31T15:14:53.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.445+0000 7f4cef400640 1 -- 172.21.15.16:0/146512241 <== osd.1 v2:172.21.15.16:6808/1398737533 6 ==== osd_op_reply(58 [pgnls-filter start_epoch 113 out=49b] v111'10 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x5589eb5fa000 con 0x5589eb7f9c00 2024-03-31T15:14:53.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.445+0000 7f4cea400640 1 -- 172.21.15.16:0/146512241 --> [v2:172.21.15.89:6800/1856844856,v1:172.21.15.89:6801/1856844856] -- osd_op(unknown.0.0:59 17.3e 17:7c000000::::head [pgnls-filter start_epoch 113 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x5589eb900c00 con 0x5589eb7f8c00 2024-03-31T15:14:53.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.445+0000 7f4cefe00640 1 -- 172.21.15.16:0/146512241 <== osd.4 v2:172.21.15.89:6800/1856844856 5 ==== osd_op_reply(59 [pgnls-filter start_epoch 113 out=49b] v111'8 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x5589eb5fa780 con 0x5589eb7f8c00 2024-03-31T15:14:53.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.445+0000 7f4ce9a00640 1 -- 172.21.15.16:0/146512241 --> [v2:172.21.15.89:6810/50395527,v1:172.21.15.89:6814/50395527] -- osd_op(unknown.0.0:60 17.1 17:80000000::::head [pgnls-filter start_epoch 113 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x5589ec2fa800 con 0x5589eb7f4c00 2024-03-31T15:14:53.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.449+0000 7f4cf0800640 1 -- 172.21.15.16:0/146512241 <== osd.5 v2:172.21.15.89:6810/50395527 5 ==== osd_op_reply(60 [pgnls-filter start_epoch 113 out=49b] v111'10 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x5589eb5fa500 con 0x5589eb7f4c00 2024-03-31T15:14:53.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.449+0000 7f4cea400640 1 -- 172.21.15.16:0/146512241 --> [v2:172.21.15.16:6819/3018526646,v1:172.21.15.16:6822/3018526646] -- osd_op(unknown.0.0:61 17.21 17:84000000::::head [pgnls-filter start_epoch 113 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x5589eb901800 con 0x5589eb917c00 2024-03-31T15:14:53.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.449+0000 7f4cf0800640 1 -- 172.21.15.16:0/146512241 <== osd.2 v2:172.21.15.16:6819/3018526646 3 ==== osd_op_reply(61 [pgnls-filter start_epoch 113 out=49b] v111'24 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x5589eb5fa500 con 0x5589eb917c00 2024-03-31T15:14:53.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.449+0000 7f4ce9a00640 1 -- 172.21.15.16:0/146512241 --> [v2:172.21.15.16:6809/3360352974,v1:172.21.15.16:6812/3360352974] -- osd_op(unknown.0.0:62 17.11 17:88000000::::head [pgnls-filter start_epoch 113 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x5589eb917800 con 0x5589ec2fb400 2024-03-31T15:14:53.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.449+0000 7f4cf0800640 1 -- 172.21.15.16:0/146512241 <== osd.3 v2:172.21.15.16:6809/3360352974 16 ==== osd_op_reply(62 [pgnls-filter start_epoch 113 out=49b] v111'12 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x5589eb5fa500 con 0x5589ec2fb400 2024-03-31T15:14:53.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.449+0000 7f4cea400640 1 -- 172.21.15.16:0/146512241 --> [v2:172.21.15.16:6808/1398737533,v1:172.21.15.16:6810/1398737533] -- osd_op(unknown.0.0:63 17.31 17:8c000000::::head [pgnls-filter start_epoch 113 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x5589eb900400 con 0x5589eb7f9c00 2024-03-31T15:14:53.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.449+0000 7f4cef400640 1 -- 172.21.15.16:0/146512241 <== osd.1 v2:172.21.15.16:6808/1398737533 7 ==== osd_op_reply(63 [pgnls-filter start_epoch 113 out=49b] v111'15 uv8 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x5589eb5fa000 con 0x5589eb7f9c00 2024-03-31T15:14:53.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.449+0000 7f4ce9a00640 1 -- 172.21.15.16:0/146512241 --> [v2:172.21.15.16:6809/3360352974,v1:172.21.15.16:6812/3360352974] -- osd_op(unknown.0.0:64 17.9 17:90000000::::head [pgnls-filter start_epoch 113 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x5589eb916000 con 0x5589ec2fb400 2024-03-31T15:14:53.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.449+0000 7f4cf0800640 1 -- 172.21.15.16:0/146512241 <== osd.3 v2:172.21.15.16:6809/3360352974 17 ==== osd_op_reply(64 [pgnls-filter start_epoch 113 out=49b] v111'4 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x5589eb5fa500 con 0x5589ec2fb400 2024-03-31T15:14:53.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.449+0000 7f4cea400640 1 -- 172.21.15.16:0/146512241 --> [v2:172.21.15.89:6800/1856844856,v1:172.21.15.89:6801/1856844856] -- osd_op(unknown.0.0:65 17.29 17:94000000::::head [pgnls-filter start_epoch 113 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x5589eb901c00 con 0x5589eb7f8c00 2024-03-31T15:14:53.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.449+0000 7f4cefe00640 1 -- 172.21.15.16:0/146512241 <== osd.4 v2:172.21.15.89:6800/1856844856 6 ==== osd_op_reply(65 [pgnls-filter start_epoch 113 out=49b] v111'16 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x5589eb5fa780 con 0x5589eb7f8c00 2024-03-31T15:14:53.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.449+0000 7f4ce9a00640 1 -- 172.21.15.16:0/146512241 --> [v2:172.21.15.89:6810/50395527,v1:172.21.15.89:6814/50395527] -- osd_op(unknown.0.0:66 17.19 17:98000000::::head [pgnls-filter start_epoch 113 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x5589eb7f5800 con 0x5589eb7f4c00 2024-03-31T15:14:53.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.449+0000 7f4cf0800640 1 -- 172.21.15.16:0/146512241 <== osd.5 v2:172.21.15.89:6810/50395527 6 ==== osd_op_reply(66 [pgnls-filter start_epoch 113 out=49b] v113'24 uv24 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x5589eb5fa500 con 0x5589eb7f4c00 2024-03-31T15:14:53.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.449+0000 7f4cea400640 1 -- 172.21.15.16:0/146512241 --> [v2:172.21.15.89:6810/50395527,v1:172.21.15.89:6814/50395527] -- osd_op(unknown.0.0:67 17.39 17:9c000000::::head [pgnls-filter start_epoch 113 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x5589eb901400 con 0x5589eb7f4c00 2024-03-31T15:14:53.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.449+0000 7f4cf0800640 1 -- 172.21.15.16:0/146512241 <== osd.5 v2:172.21.15.89:6810/50395527 7 ==== osd_op_reply(67 [pgnls-filter start_epoch 113 out=49b] v111'10 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x5589eb5fa500 con 0x5589eb7f4c00 2024-03-31T15:14:53.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.449+0000 7f4ce9a00640 1 -- 172.21.15.16:0/146512241 --> [v2:172.21.15.89:6809/3470471752,v1:172.21.15.89:6812/3470471752] -- osd_op(unknown.0.0:68 17.5 17:a0000000::::head [pgnls-filter start_epoch 113 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x5589eb7f5c00 con 0x5589eb7f5000 2024-03-31T15:14:53.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.449+0000 7f4cef400640 1 -- 172.21.15.16:0/146512241 <== osd.6 v2:172.21.15.89:6809/3470471752 3 ==== osd_op_reply(68 [pgnls-filter start_epoch 113 out=49b] v113'15 uv15 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x5589eb5fa000 con 0x5589eb7f5000 2024-03-31T15:14:53.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.449+0000 7f4cea400640 1 -- 172.21.15.16:0/146512241 --> [v2:172.21.15.16:6808/1398737533,v1:172.21.15.16:6810/1398737533] -- osd_op(unknown.0.0:69 17.25 17:a4000000::::head [pgnls-filter start_epoch 113 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x5589eb901000 con 0x5589eb7f9c00 2024-03-31T15:14:53.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.449+0000 7f4cef400640 1 -- 172.21.15.16:0/146512241 <== osd.1 v2:172.21.15.16:6808/1398737533 8 ==== osd_op_reply(69 [pgnls-filter start_epoch 113 out=49b] v111'12 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x5589eb5fa000 con 0x5589eb7f9c00 2024-03-31T15:14:53.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.449+0000 7f4ce9a00640 1 -- 172.21.15.16:0/146512241 --> [v2:172.21.15.89:6810/50395527,v1:172.21.15.89:6814/50395527] -- osd_op(unknown.0.0:70 17.15 17:a8000000::::head [pgnls-filter start_epoch 113 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x5589eb7f4800 con 0x5589eb7f4c00 2024-03-31T15:14:53.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.449+0000 7f4cf0800640 1 -- 172.21.15.16:0/146512241 <== osd.5 v2:172.21.15.89:6810/50395527 8 ==== osd_op_reply(70 [pgnls-filter start_epoch 113 out=49b] v111'12 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x5589eb5fa500 con 0x5589eb7f4c00 2024-03-31T15:14:53.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.449+0000 7f4cea400640 1 -- 172.21.15.16:0/146512241 --> [v2:172.21.15.89:6800/1856844856,v1:172.21.15.89:6801/1856844856] -- osd_op(unknown.0.0:71 17.35 17:ac000000::::head [pgnls-filter start_epoch 113 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x5589eb900000 con 0x5589eb7f8c00 2024-03-31T15:14:53.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.453+0000 7f4cefe00640 1 -- 172.21.15.16:0/146512241 <== osd.4 v2:172.21.15.89:6800/1856844856 7 ==== osd_op_reply(71 [pgnls-filter start_epoch 113 out=49b] v111'12 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x5589eb5fa780 con 0x5589eb7f8c00 2024-03-31T15:14:53.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.453+0000 7f4ce9a00640 1 -- 172.21.15.16:0/146512241 --> [v2:172.21.15.89:6808/1796778630,v1:172.21.15.89:6811/1796778630] -- osd_op(unknown.0.0:72 17.d 17:b0000000::::head [pgnls-filter start_epoch 113 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x5589eb7f8400 con 0x5589eb7f5400 2024-03-31T15:14:53.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.453+0000 7f4cefe00640 1 -- 172.21.15.16:0/146512241 <== osd.7 v2:172.21.15.89:6808/1796778630 11 ==== osd_op_reply(72 [pgnls-filter start_epoch 113 out=49b] v111'14 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x5589eb5fa780 con 0x5589eb7f5400 2024-03-31T15:14:53.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.453+0000 7f4cea400640 1 -- 172.21.15.16:0/146512241 --> [v2:172.21.15.89:6808/1796778630,v1:172.21.15.89:6811/1796778630] -- osd_op(unknown.0.0:73 17.2d 17:b4000000::::head [pgnls-filter start_epoch 113 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x5589eb916800 con 0x5589eb7f5400 2024-03-31T15:14:53.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.453+0000 7f4cefe00640 1 -- 172.21.15.16:0/146512241 <== osd.7 v2:172.21.15.89:6808/1796778630 12 ==== osd_op_reply(73 [pgnls-filter start_epoch 113 out=49b] v111'14 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x5589eb5fa780 con 0x5589eb7f5400 2024-03-31T15:14:53.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.453+0000 7f4ce9a00640 1 -- 172.21.15.16:0/146512241 --> [v2:172.21.15.89:6808/1796778630,v1:172.21.15.89:6811/1796778630] -- osd_op(unknown.0.0:74 17.1d 17:b8000000::::head [pgnls-filter start_epoch 113 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x5589ec2fa800 con 0x5589eb7f5400 2024-03-31T15:14:53.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.453+0000 7f4cefe00640 1 -- 172.21.15.16:0/146512241 <== osd.7 v2:172.21.15.89:6808/1796778630 13 ==== osd_op_reply(74 [pgnls-filter start_epoch 113 out=49b] v111'30 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x5589eb5fa780 con 0x5589eb7f5400 2024-03-31T15:14:53.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.453+0000 7f4cea400640 1 -- 172.21.15.16:0/146512241 --> [v2:172.21.15.89:6808/1796778630,v1:172.21.15.89:6811/1796778630] -- osd_op(unknown.0.0:75 17.3d 17:bc000000::::head [pgnls-filter start_epoch 113 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x5589eb917400 con 0x5589eb7f5400 2024-03-31T15:14:53.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.453+0000 7f4cefe00640 1 -- 172.21.15.16:0/146512241 <== osd.7 v2:172.21.15.89:6808/1796778630 14 ==== osd_op_reply(75 [pgnls-filter start_epoch 113 out=49b] v111'14 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x5589eb5fa780 con 0x5589eb7f5400 2024-03-31T15:14:53.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.453+0000 7f4ce9a00640 1 -- 172.21.15.16:0/146512241 --> [v2:172.21.15.89:6800/1856844856,v1:172.21.15.89:6801/1856844856] -- osd_op(unknown.0.0:76 17.3 17:c0000000::::head [pgnls-filter start_epoch 113 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x5589eb924000 con 0x5589eb7f8c00 2024-03-31T15:14:53.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.453+0000 7f4cefe00640 1 -- 172.21.15.16:0/146512241 <== osd.4 v2:172.21.15.89:6800/1856844856 8 ==== osd_op_reply(76 [pgnls-filter start_epoch 113 out=49b] v111'14 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x5589eb5fa780 con 0x5589eb7f8c00 2024-03-31T15:14:53.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.453+0000 7f4cea400640 1 -- 172.21.15.16:0/146512241 --> [v2:172.21.15.89:6800/1856844856,v1:172.21.15.89:6801/1856844856] -- osd_op(unknown.0.0:77 17.23 17:c4000000::::head [pgnls-filter start_epoch 113 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x5589eb916c00 con 0x5589eb7f8c00 2024-03-31T15:14:53.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.453+0000 7f4cefe00640 1 -- 172.21.15.16:0/146512241 <== osd.4 v2:172.21.15.89:6800/1856844856 9 ==== osd_op_reply(77 [pgnls-filter start_epoch 113 out=49b] v111'14 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x5589eb5fa780 con 0x5589eb7f8c00 2024-03-31T15:14:53.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.453+0000 7f4ce9a00640 1 -- 172.21.15.16:0/146512241 --> [v2:172.21.15.16:6809/3360352974,v1:172.21.15.16:6812/3360352974] -- osd_op(unknown.0.0:78 17.13 17:c8000000::::head [pgnls-filter start_epoch 113 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x5589eb7f8800 con 0x5589ec2fb400 2024-03-31T15:14:53.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.453+0000 7f4cf0800640 1 -- 172.21.15.16:0/146512241 <== osd.3 v2:172.21.15.16:6809/3360352974 18 ==== osd_op_reply(78 [pgnls-filter start_epoch 113 out=49b] v111'30 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x5589eb5fa500 con 0x5589ec2fb400 2024-03-31T15:14:53.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.453+0000 7f4cea400640 1 -- 172.21.15.16:0/146512241 --> [v2:172.21.15.89:6808/1796778630,v1:172.21.15.89:6811/1796778630] -- osd_op(unknown.0.0:79 17.33 17:cc000000::::head [pgnls-filter start_epoch 113 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x5589eb92c000 con 0x5589eb7f5400 2024-03-31T15:14:53.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.453+0000 7f4cefe00640 1 -- 172.21.15.16:0/146512241 <== osd.7 v2:172.21.15.89:6808/1796778630 15 ==== osd_op_reply(79 [pgnls-filter start_epoch 113 out=49b] v111'20 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x5589eb5fa780 con 0x5589eb7f5400 2024-03-31T15:14:53.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.453+0000 7f4ce9a00640 1 -- 172.21.15.16:0/146512241 --> [v2:172.21.15.16:6800/932712244,v1:172.21.15.16:6801/932712244] -- osd_op(unknown.0.0:80 17.b 17:d0000000::::head [pgnls-filter start_epoch 113 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x5589eb7f8400 con 0x5589eb7f4400 2024-03-31T15:14:53.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.453+0000 7f4cef400640 1 -- 172.21.15.16:0/146512241 <== osd.0 v2:172.21.15.16:6800/932712244 16 ==== osd_op_reply(80 [pgnls-filter start_epoch 113 out=49b] v111'16 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x5589eb5fa000 con 0x5589eb7f4400 2024-03-31T15:14:53.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.453+0000 7f4cea400640 1 -- 172.21.15.16:0/146512241 --> [v2:172.21.15.89:6800/1856844856,v1:172.21.15.89:6801/1856844856] -- osd_op(unknown.0.0:81 17.2b 17:d4000000::::head [pgnls-filter start_epoch 113 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x5589eb92c400 con 0x5589eb7f8c00 2024-03-31T15:14:53.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.453+0000 7f4cefe00640 1 -- 172.21.15.16:0/146512241 <== osd.4 v2:172.21.15.89:6800/1856844856 10 ==== osd_op_reply(81 [pgnls-filter start_epoch 113 out=49b] v111'22 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x5589eb5fa780 con 0x5589eb7f8c00 2024-03-31T15:14:53.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.453+0000 7f4ce9a00640 1 -- 172.21.15.16:0/146512241 --> [v2:172.21.15.89:6800/1856844856,v1:172.21.15.89:6801/1856844856] -- osd_op(unknown.0.0:82 17.1b 17:d8000000::::head [pgnls-filter start_epoch 113 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x5589eb900800 con 0x5589eb7f8c00 2024-03-31T15:14:53.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.453+0000 7f4cefe00640 1 -- 172.21.15.16:0/146512241 <== osd.4 v2:172.21.15.89:6800/1856844856 11 ==== osd_op_reply(82 [pgnls-filter start_epoch 113 out=49b] v111'16 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x5589eb5fa780 con 0x5589eb7f8c00 2024-03-31T15:14:53.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.453+0000 7f4cea400640 1 -- 172.21.15.16:0/146512241 --> [v2:172.21.15.16:6819/3018526646,v1:172.21.15.16:6822/3018526646] -- osd_op(unknown.0.0:83 17.3b 17:dc000000::::head [pgnls-filter start_epoch 113 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x5589eb92c800 con 0x5589eb917c00 2024-03-31T15:14:53.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.453+0000 7f4cf0800640 1 -- 172.21.15.16:0/146512241 <== osd.2 v2:172.21.15.16:6819/3018526646 4 ==== osd_op_reply(83 [pgnls-filter start_epoch 113 out=49b] v111'8 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x5589eb5fa500 con 0x5589eb917c00 2024-03-31T15:14:53.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.453+0000 7f4ce9a00640 1 -- 172.21.15.16:0/146512241 --> [v2:172.21.15.16:6800/932712244,v1:172.21.15.16:6801/932712244] -- osd_op(unknown.0.0:84 17.7 17:e0000000::::head [pgnls-filter start_epoch 113 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x5589eb900c00 con 0x5589eb7f4400 2024-03-31T15:14:53.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.457+0000 7f4cef400640 1 -- 172.21.15.16:0/146512241 <== osd.0 v2:172.21.15.16:6800/932712244 17 ==== osd_op_reply(84 [pgnls-filter start_epoch 113 out=49b] v111'10 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x5589eb5fa000 con 0x5589eb7f4400 2024-03-31T15:14:53.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.457+0000 7f4cea400640 1 -- 172.21.15.16:0/146512241 --> [v2:172.21.15.89:6800/1856844856,v1:172.21.15.89:6801/1856844856] -- osd_op(unknown.0.0:85 17.27 17:e4000000::::head [pgnls-filter start_epoch 113 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x5589eb92cc00 con 0x5589eb7f8c00 2024-03-31T15:14:53.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.457+0000 7f4cefe00640 1 -- 172.21.15.16:0/146512241 <== osd.4 v2:172.21.15.89:6800/1856844856 12 ==== osd_op_reply(85 [pgnls-filter start_epoch 113 out=49b] v111'4 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x5589eb5fa780 con 0x5589eb7f8c00 2024-03-31T15:14:53.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.457+0000 7f4ce9a00640 1 -- 172.21.15.16:0/146512241 --> [v2:172.21.15.89:6809/3470471752,v1:172.21.15.89:6812/3470471752] -- osd_op(unknown.0.0:86 17.17 17:e8000000::::head [pgnls-filter start_epoch 113 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x5589eb901c00 con 0x5589eb7f5000 2024-03-31T15:14:53.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.457+0000 7f4cef400640 1 -- 172.21.15.16:0/146512241 <== osd.6 v2:172.21.15.89:6809/3470471752 4 ==== osd_op_reply(86 [pgnls-filter start_epoch 113 out=49b] v111'18 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x5589eb5fa000 con 0x5589eb7f5000 2024-03-31T15:14:53.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.457+0000 7f4cea400640 1 -- 172.21.15.16:0/146512241 --> [v2:172.21.15.16:6800/932712244,v1:172.21.15.16:6801/932712244] -- osd_op(unknown.0.0:87 17.37 17:ec000000::::head [pgnls-filter start_epoch 113 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x5589eb92d000 con 0x5589eb7f4400 2024-03-31T15:14:53.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.457+0000 7f4cef400640 1 -- 172.21.15.16:0/146512241 <== osd.0 v2:172.21.15.16:6800/932712244 18 ==== osd_op_reply(87 [pgnls-filter start_epoch 113 out=49b] v111'22 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x5589eb5fa000 con 0x5589eb7f4400 2024-03-31T15:14:53.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.457+0000 7f4ce9a00640 1 -- 172.21.15.16:0/146512241 --> [v2:172.21.15.16:6808/1398737533,v1:172.21.15.16:6810/1398737533] -- osd_op(unknown.0.0:88 17.f 17:f0000000::::head [pgnls-filter start_epoch 113 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x5589eb900000 con 0x5589eb7f9c00 2024-03-31T15:14:53.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.457+0000 7f4cef400640 1 -- 172.21.15.16:0/146512241 <== osd.1 v2:172.21.15.16:6808/1398737533 9 ==== osd_op_reply(88 [pgnls-filter start_epoch 113 out=49b] v111'18 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x5589eb5fa000 con 0x5589eb7f9c00 2024-03-31T15:14:53.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.457+0000 7f4cea400640 1 -- 172.21.15.16:0/146512241 --> [v2:172.21.15.89:6800/1856844856,v1:172.21.15.89:6801/1856844856] -- osd_op(unknown.0.0:89 17.2f 17:f4000000::::head [pgnls-filter start_epoch 113 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x5589eb92d400 con 0x5589eb7f8c00 2024-03-31T15:14:53.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.457+0000 7f4cefe00640 1 -- 172.21.15.16:0/146512241 <== osd.4 v2:172.21.15.89:6800/1856844856 13 ==== osd_op_reply(89 [pgnls-filter start_epoch 113 out=49b] v111'16 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x5589eb5fa780 con 0x5589eb7f8c00 2024-03-31T15:14:53.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.457+0000 7f4ce9a00640 1 -- 172.21.15.16:0/146512241 --> [v2:172.21.15.89:6809/3470471752,v1:172.21.15.89:6812/3470471752] -- osd_op(unknown.0.0:90 17.1f 17:f8000000::::head [pgnls-filter start_epoch 113 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x5589eb916400 con 0x5589eb7f5000 2024-03-31T15:14:53.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.457+0000 7f4cef400640 1 -- 172.21.15.16:0/146512241 <== osd.6 v2:172.21.15.89:6809/3470471752 5 ==== osd_op_reply(90 [pgnls-filter start_epoch 113 out=49b] v111'12 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x5589eb5fa000 con 0x5589eb7f5000 2024-03-31T15:14:53.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.457+0000 7f4cea400640 1 -- 172.21.15.16:0/146512241 --> [v2:172.21.15.16:6800/932712244,v1:172.21.15.16:6801/932712244] -- osd_op(unknown.0.0:91 17.3f 17:fc000000::::head [pgnls-filter start_epoch 113 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x5589eb92d800 con 0x5589eb7f4400 2024-03-31T15:14:53.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.457+0000 7f4cef400640 1 -- 172.21.15.16:0/146512241 <== osd.0 v2:172.21.15.16:6800/932712244 19 ==== osd_op_reply(91 [pgnls-filter start_epoch 113 out=49b] v111'17 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x5589eb5fa000 con 0x5589eb7f4400 2024-03-31T15:14:53.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.457+0000 7f4cf1db98c0 1 -- 172.21.15.16:0/146512241 >> [v2:172.21.15.16:6800/932712244,v1:172.21.15.16:6801/932712244] conn(0x5589eb7f4400 msgr2=0x5589ec2d2680 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:53.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.457+0000 7f4cf1db98c0 1 --2- 172.21.15.16:0/146512241 >> [v2:172.21.15.16:6800/932712244,v1:172.21.15.16:6801/932712244] conn(0x5589eb7f4400 0x5589ec2d2680 crc :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:53.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.457+0000 7f4cf1db98c0 1 -- 172.21.15.16:0/146512241 >> [v2:172.21.15.16:6808/1398737533,v1:172.21.15.16:6810/1398737533] conn(0x5589eb7f9c00 msgr2=0x5589eb8f6000 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:53.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.457+0000 7f4cf1db98c0 1 --2- 172.21.15.16:0/146512241 >> [v2:172.21.15.16:6808/1398737533,v1:172.21.15.16:6810/1398737533] conn(0x5589eb7f9c00 0x5589eb8f6000 crc :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:53.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.457+0000 7f4cf1db98c0 1 -- 172.21.15.16:0/146512241 >> [v2:172.21.15.16:6819/3018526646,v1:172.21.15.16:6822/3018526646] conn(0x5589eb917c00 msgr2=0x5589ec2d3700 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:53.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.457+0000 7f4cf1db98c0 1 --2- 172.21.15.16:0/146512241 >> [v2:172.21.15.16:6819/3018526646,v1:172.21.15.16:6822/3018526646] conn(0x5589eb917c00 0x5589ec2d3700 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:53.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.457+0000 7f4cf1db98c0 1 -- 172.21.15.16:0/146512241 >> [v2:172.21.15.16:6809/3360352974,v1:172.21.15.16:6812/3360352974] conn(0x5589ec2fb400 msgr2=0x5589eb8f6580 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:53.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.457+0000 7f4cf1db98c0 1 --2- 172.21.15.16:0/146512241 >> [v2:172.21.15.16:6809/3360352974,v1:172.21.15.16:6812/3360352974] conn(0x5589ec2fb400 0x5589eb8f6580 crc :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:53.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.457+0000 7f4cf1db98c0 1 -- 172.21.15.16:0/146512241 >> [v2:172.21.15.89:6800/1856844856,v1:172.21.15.89:6801/1856844856] conn(0x5589eb7f8c00 msgr2=0x5589eb8f6b00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:53.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.457+0000 7f4cf1db98c0 1 --2- 172.21.15.16:0/146512241 >> [v2:172.21.15.89:6800/1856844856,v1:172.21.15.89:6801/1856844856] conn(0x5589eb7f8c00 0x5589eb8f6b00 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:53.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.457+0000 7f4cf1db98c0 1 -- 172.21.15.16:0/146512241 >> [v2:172.21.15.89:6810/50395527,v1:172.21.15.89:6814/50395527] conn(0x5589eb7f4c00 msgr2=0x5589ec2d2c00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:53.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.457+0000 7f4cf1db98c0 1 --2- 172.21.15.16:0/146512241 >> [v2:172.21.15.89:6810/50395527,v1:172.21.15.89:6814/50395527] conn(0x5589eb7f4c00 0x5589ec2d2c00 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:53.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.457+0000 7f4cf1db98c0 1 -- 172.21.15.16:0/146512241 >> [v2:172.21.15.89:6809/3470471752,v1:172.21.15.89:6812/3470471752] conn(0x5589eb7f5000 msgr2=0x5589eb8f7080 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:53.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.457+0000 7f4cf1db98c0 1 --2- 172.21.15.16:0/146512241 >> [v2:172.21.15.89:6809/3470471752,v1:172.21.15.89:6812/3470471752] conn(0x5589eb7f5000 0x5589eb8f7080 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:53.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.457+0000 7f4cf1db98c0 1 -- 172.21.15.16:0/146512241 >> [v2:172.21.15.89:6808/1796778630,v1:172.21.15.89:6811/1796778630] conn(0x5589eb7f5400 msgr2=0x5589ec2d3180 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:53.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.457+0000 7f4cf1db98c0 1 --2- 172.21.15.16:0/146512241 >> [v2:172.21.15.89:6808/1796778630,v1:172.21.15.89:6811/1796778630] conn(0x5589eb7f5400 0x5589ec2d3180 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:53.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.457+0000 7f4cef400640 1 -- 172.21.15.16:0/146512241 reap_dead start 2024-03-31T15:14:53.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.457+0000 7f4cef400640 1 -- 172.21.15.16:0/146512241 reap_dead start 2024-03-31T15:14:53.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.457+0000 7f4cef400640 1 -- 172.21.15.16:0/146512241 reap_dead start 2024-03-31T15:14:53.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.457+0000 7f4cef400640 1 -- 172.21.15.16:0/146512241 reap_dead start 2024-03-31T15:14:53.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.457+0000 7f4cf1db98c0 1 -- 172.21.15.16:0/146512241 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x5589eb7f9000 msgr2=0x5589ec2d2100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:53.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.457+0000 7f4cf1db98c0 1 --2- 172.21.15.16:0/146512241 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x5589eb7f9000 0x5589ec2d2100 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x5589ec284cf0 tx=0x5589eb7fe180 comp rx=0 tx=0).stop 2024-03-31T15:14:53.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.457+0000 7f4cf1db98c0 1 -- 172.21.15.16:0/146512241 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5589ec425800 msgr2=0x5589ec2d1600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:53.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.457+0000 7f4cf1db98c0 1 --2- 172.21.15.16:0/146512241 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5589ec425800 0x5589ec2d1600 secure :-1 s=READY pgs=1817 cs=0 l=1 rev1=1 crypto rx=0x5589ec2f1ad0 tx=0x5589ec28b740 comp rx=0 tx=0).stop 2024-03-31T15:14:53.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.457+0000 7f4cf1db98c0 1 -- 172.21.15.16:0/146512241 shutdown_connections 2024-03-31T15:14:53.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.457+0000 7f4cf1db98c0 1 --2- 172.21.15.16:0/146512241 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x5589eb7f9000 0x5589ec2d2100 unknown :-1 s=CLOSED pgs=764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:53.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.457+0000 7f4cf1db98c0 1 --2- 172.21.15.16:0/146512241 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5589ec425800 0x5589ec2d1600 unknown :-1 s=CLOSED pgs=1817 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:53.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.457+0000 7f4cf1db98c0 1 -- 172.21.15.16:0/146512241 >> 172.21.15.16:0/146512241 conn(0x5589eb502c00 msgr2=0x5589eb531800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:14:53.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.457+0000 7f4cf1db98c0 1 -- 172.21.15.16:0/146512241 shutdown_connections 2024-03-31T15:14:53.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.457+0000 7f4cf1db98c0 1 -- 172.21.15.16:0/146512241 wait complete. 2024-03-31T15:14:53.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.461+0000 7f4cf1db98c0 1 -- 172.21.15.16:0/2832673357 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5589eb503800 msgr2=0x5589ec2d0000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:53.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.461+0000 7f4cf1db98c0 1 --2- 172.21.15.16:0/2832673357 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5589eb503800 0x5589ec2d0000 secure :-1 s=READY pgs=1815 cs=0 l=1 rev1=1 crypto rx=0x5589ec36f8c0 tx=0x5589eb5c8660 comp rx=0 tx=0).stop 2024-03-31T15:14:53.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.461+0000 7f4cf1db98c0 1 -- 172.21.15.16:0/2832673357 shutdown_connections 2024-03-31T15:14:53.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.461+0000 7f4cf1db98c0 1 --2- 172.21.15.16:0/2832673357 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5589eb503000 0x5589ec2d0b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:53.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.461+0000 7f4cf1db98c0 1 --2- 172.21.15.16:0/2832673357 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x5589eb503400 0x5589ec2d0580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:53.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.461+0000 7f4cf1db98c0 1 --2- 172.21.15.16:0/2832673357 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5589eb503800 0x5589ec2d0000 unknown :-1 s=CLOSED pgs=1815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:53.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.461+0000 7f4cf1db98c0 1 -- 172.21.15.16:0/2832673357 >> 172.21.15.16:0/2832673357 conn(0x5589eb503c00 msgr2=0x5589eb531000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:14:53.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.461+0000 7f4cf1db98c0 1 -- 172.21.15.16:0/2832673357 shutdown_connections 2024-03-31T15:14:53.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.461+0000 7f4cf1db98c0 1 -- 172.21.15.16:0/2832673357 wait complete. 2024-03-31T15:14:53.459 DEBUG:tasks.cephfs.filesystem:Ran ['cephfs-data-scan', '--debug-mds=20', '--debug-ms=1', '--debug-objecter=1', 'scan_inodes', '--filter-tag', 'mytag123', 'cephfs_data'] in time 0:00:00.103971, result: 2024-03-31T15:14:53.459 DEBUG:teuthology.orchestra.run.smithi016:> cephfs-data-scan --debug-mds=20 --debug-ms=1 --debug-objecter=1 scan_links 2024-03-31T15:14:53.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.549+0000 7fbce9ae18c0 1 -- start start 2024-03-31T15:14:53.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.549+0000 7fbce9ae18c0 1 Processor -- start 2024-03-31T15:14:53.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.549+0000 7fbce9ae18c0 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x56290df09800 0x56290eccc000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:53.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.549+0000 7fbce9ae18c0 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56290df09400 0x56290eccc580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:53.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.549+0000 7fbce9ae18c0 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56290df09000 0x56290ecccb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:53.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.549+0000 7fbce9ae18c0 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x56290ece0180 con 0x56290df09400 2024-03-31T15:14:53.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.549+0000 7fbce9ae18c0 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x56290ece0300 con 0x56290df09000 2024-03-31T15:14:53.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.549+0000 7fbce9ae18c0 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x56290ece0480 con 0x56290df09800 2024-03-31T15:14:53.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.549+0000 7fbce7200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56290df09400 0x56290eccc580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:53.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.549+0000 7fbce7200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56290df09400 0x56290eccc580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55684/0 (socket says 172.21.15.16:55684) 2024-03-31T15:14:53.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.549+0000 7fbce7200640 1 -- 172.21.15.16:0/4079968052 learned_addr learned my addr 172.21.15.16:0/4079968052 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:14:53.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.549+0000 7fbce8600640 1 --2- 172.21.15.16:0/4079968052 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56290df09000 0x56290ecccb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:53.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.549+0000 7fbce7c00640 1 --2- 172.21.15.16:0/4079968052 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x56290df09800 0x56290eccc000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:53.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.549+0000 7fbce7200640 1 -- 172.21.15.16:0/4079968052 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x56290df09800 msgr2=0x56290eccc000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:14:53.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.549+0000 7fbce7200640 1 --2- 172.21.15.16:0/4079968052 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x56290df09800 0x56290eccc000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:53.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.549+0000 7fbce7200640 1 -- 172.21.15.16:0/4079968052 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56290df09000 msgr2=0x56290ecccb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:14:53.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.549+0000 7fbce7200640 1 --2- 172.21.15.16:0/4079968052 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56290df09000 0x56290ecccb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:53.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.549+0000 7fbce7200640 1 -- 172.21.15.16:0/4079968052 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56290df2e1e0 con 0x56290df09400 2024-03-31T15:14:53.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.549+0000 7fbce7200640 1 --2- 172.21.15.16:0/4079968052 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56290df09400 0x56290eccc580 secure :-1 s=READY pgs=1819 cs=0 l=1 rev1=1 crypto rx=0x56290ed638c0 tx=0x56290ec910e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e661134845144b2 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:53.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.553+0000 7fbce4a00640 1 -- 172.21.15.16:0/4079968052 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x56290df3e4e0 con 0x56290df09400 2024-03-31T15:14:53.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.553+0000 7fbce4a00640 1 -- 172.21.15.16:0/4079968052 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x56290df54e00 con 0x56290df09400 2024-03-31T15:14:53.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.553+0000 7fbce4a00640 1 -- 172.21.15.16:0/4079968052 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x56290df3eb60 con 0x56290df09400 2024-03-31T15:14:53.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.553+0000 7fbce9ae18c0 1 -- 172.21.15.16:0/4079968052 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56290df2f2c0 con 0x56290df09400 2024-03-31T15:14:53.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.553+0000 7fbce9ae18c0 1 -- 172.21.15.16:0/4079968052 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({fsmap=0}) -- 0x56290df2fa40 con 0x56290df09400 2024-03-31T15:14:53.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.553+0000 7fbce9ae18c0 4 waiting for MDS map... 2024-03-31T15:14:53.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.553+0000 7fbce4a00640 1 -- 172.21.15.16:0/4079968052 <== mon.0 v2:172.21.15.16:3300/0 4 ==== osd_map(113..113 src has 1..113) ==== 5939+0+0 (secure 0 0 0) 0x56290ee2a600 con 0x56290df09400 2024-03-31T15:14:53.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.553+0000 7fbce4a00640 1 -- 172.21.15.16:0/4079968052 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 106) ==== 2885+0+0 (secure 0 0 0) 0x56290df5f180 con 0x56290df09400 2024-03-31T15:14:53.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.553+0000 7fbce9ae18c0 4 Got MDS map 106 2024-03-31T15:14:53.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.553+0000 7fbce9ae18c0 4 datascan.main: Using metadata pool output 2024-03-31T15:14:53.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.553+0000 7fbce9ae18c0 4 datascan.main: connecting to RADOS... 2024-03-31T15:14:53.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.553+0000 7fbce9ae18c0 1 Processor -- start 2024-03-31T15:14:53.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.553+0000 7fbce9ae18c0 1 -- start start 2024-03-31T15:14:53.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.553+0000 7fbce9ae18c0 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56290ee2d000 0x56290eccd080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:53.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.553+0000 7fbce9ae18c0 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56290ee2d400 0x56290eccd600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:53.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.553+0000 7fbce9ae18c0 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x56290ee2d800 0x56290eccdb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:53.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.553+0000 7fbce9ae18c0 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x56290ece0600 con 0x56290ee2d400 2024-03-31T15:14:53.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.553+0000 7fbce9ae18c0 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x56290ece0780 con 0x56290ee2d000 2024-03-31T15:14:53.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.553+0000 7fbce9ae18c0 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x56290ece0900 con 0x56290ee2d800 2024-03-31T15:14:53.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.553+0000 7fbce7c00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56290ee2d400 0x56290eccd600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:53.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.553+0000 7fbce7c00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56290ee2d400 0x56290eccd600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55700/0 (socket says 172.21.15.16:55700) 2024-03-31T15:14:53.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.553+0000 7fbce7c00640 1 -- 172.21.15.16:0/879691368 learned_addr learned my addr 172.21.15.16:0/879691368 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:14:53.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.553+0000 7fbce7200640 1 --2- 172.21.15.16:0/879691368 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x56290ee2d800 0x56290eccdb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:53.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.553+0000 7fbce8600640 1 --2- 172.21.15.16:0/879691368 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56290ee2d000 0x56290eccd080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:53.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.553+0000 7fbce7c00640 1 -- 172.21.15.16:0/879691368 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x56290ee2d800 msgr2=0x56290eccdb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:14:53.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.553+0000 7fbce7c00640 1 --2- 172.21.15.16:0/879691368 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x56290ee2d800 0x56290eccdb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:53.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.553+0000 7fbce7c00640 1 -- 172.21.15.16:0/879691368 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56290ee2d000 msgr2=0x56290eccd080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:14:53.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.553+0000 7fbce7c00640 1 --2- 172.21.15.16:0/879691368 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56290ee2d000 0x56290eccd080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:53.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.553+0000 7fbce7c00640 1 -- 172.21.15.16:0/879691368 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56290df2fc20 con 0x56290ee2d400 2024-03-31T15:14:53.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.553+0000 7fbce7c00640 1 --2- 172.21.15.16:0/879691368 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56290ee2d400 0x56290eccd600 secure :-1 s=READY pgs=1820 cs=0 l=1 rev1=1 crypto rx=0x56290ee3cc90 tx=0x56290e1ee000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=68b1f990d628c8e7 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:53.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.553+0000 7fbce2200640 1 -- 172.21.15.16:0/879691368 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x56290df3e4e0 con 0x56290ee2d400 2024-03-31T15:14:53.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.553+0000 7fbce2200640 1 -- 172.21.15.16:0/879691368 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x56290df54540 con 0x56290ee2d400 2024-03-31T15:14:53.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.553+0000 7fbce2200640 1 -- 172.21.15.16:0/879691368 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x56290df3eb60 con 0x56290ee2d400 2024-03-31T15:14:53.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.553+0000 7fbce9ae18c0 1 -- 172.21.15.16:0/879691368 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56290ee2d400 msgr2=0x56290eccd600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:53.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.553+0000 7fbce9ae18c0 1 --2- 172.21.15.16:0/879691368 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56290ee2d400 0x56290eccd600 secure :-1 s=READY pgs=1820 cs=0 l=1 rev1=1 crypto rx=0x56290ee3cc90 tx=0x56290e1ee000 comp rx=0 tx=0).stop 2024-03-31T15:14:53.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.553+0000 7fbce9ae18c0 1 -- 172.21.15.16:0/879691368 shutdown_connections 2024-03-31T15:14:53.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.553+0000 7fbce9ae18c0 1 --2- 172.21.15.16:0/879691368 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x56290ee2d800 0x56290eccdb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:53.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.553+0000 7fbce9ae18c0 1 --2- 172.21.15.16:0/879691368 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56290ee2d400 0x56290eccd600 unknown :-1 s=CLOSED pgs=1820 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:53.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.553+0000 7fbce9ae18c0 1 --2- 172.21.15.16:0/879691368 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56290ee2d000 0x56290eccd080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:53.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.553+0000 7fbce9ae18c0 1 -- 172.21.15.16:0/879691368 >> 172.21.15.16:0/879691368 conn(0x56290df08c00 msgr2=0x56290df37800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:14:53.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.553+0000 7fbce9ae18c0 1 -- 172.21.15.16:0/879691368 shutdown_connections 2024-03-31T15:14:53.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.553+0000 7fbce9ae18c0 1 -- 172.21.15.16:0/879691368 wait complete. 2024-03-31T15:14:53.554 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.557+0000 7fbce9ae18c0 1 Processor -- start 2024-03-31T15:14:53.554 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.557+0000 7fbce9ae18c0 1 -- start start 2024-03-31T15:14:53.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.557+0000 7fbce9ae18c0 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56290ee2d000 0x56290eccd080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:53.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.557+0000 7fbce9ae18c0 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x56290ee2d400 0x56290eccd600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:53.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.557+0000 7fbce9ae18c0 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56290ee2d800 0x56290eccdb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:53.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.557+0000 7fbce9ae18c0 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x56290ece1c80 con 0x56290ee2d000 2024-03-31T15:14:53.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.557+0000 7fbce9ae18c0 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x56290ece1b00 con 0x56290ee2d800 2024-03-31T15:14:53.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.557+0000 7fbce9ae18c0 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x56290ece1980 con 0x56290ee2d400 2024-03-31T15:14:53.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.557+0000 7fbce8600640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56290ee2d000 0x56290eccd080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:53.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.557+0000 7fbce8600640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56290ee2d000 0x56290eccd080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55706/0 (socket says 172.21.15.16:55706) 2024-03-31T15:14:53.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.557+0000 7fbce8600640 1 -- 172.21.15.16:0/2805451625 learned_addr learned my addr 172.21.15.16:0/2805451625 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:14:53.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.557+0000 7fbce7c00640 1 --2- 172.21.15.16:0/2805451625 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x56290ee2d400 0x56290eccd600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:53.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.557+0000 7fbce7200640 1 --2- 172.21.15.16:0/2805451625 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56290ee2d800 0x56290eccdb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:53.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.557+0000 7fbce8600640 1 -- 172.21.15.16:0/2805451625 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x56290ee2d400 msgr2=0x56290eccd600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:53.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.557+0000 7fbce8600640 1 --2- 172.21.15.16:0/2805451625 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x56290ee2d400 0x56290eccd600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:53.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.557+0000 7fbce8600640 1 -- 172.21.15.16:0/2805451625 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56290ee2d800 msgr2=0x56290eccdb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:53.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.557+0000 7fbce8600640 1 --2- 172.21.15.16:0/2805451625 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56290ee2d800 0x56290eccdb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:53.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.557+0000 7fbce8600640 1 -- 172.21.15.16:0/2805451625 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56290df2e1e0 con 0x56290ee2d000 2024-03-31T15:14:53.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.557+0000 7fbce8600640 1 --2- 172.21.15.16:0/2805451625 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56290ee2d000 0x56290eccd080 secure :-1 s=READY pgs=1821 cs=0 l=1 rev1=1 crypto rx=0x56290ed62ff0 tx=0x56290ecd8000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:53.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.557+0000 7fbce0400640 1 -- 172.21.15.16:0/2805451625 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x56290df3e9c0 con 0x56290ee2d000 2024-03-31T15:14:53.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.557+0000 7fbce0400640 1 -- 172.21.15.16:0/2805451625 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x56290df54c40 con 0x56290ee2d000 2024-03-31T15:14:53.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.557+0000 7fbce9ae18c0 1 -- 172.21.15.16:0/2805451625 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x56290df2f2c0 con 0x56290ee2d000 2024-03-31T15:14:53.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.557+0000 7fbce0400640 1 -- 172.21.15.16:0/2805451625 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x56290df3e4e0 con 0x56290ee2d000 2024-03-31T15:14:53.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.557+0000 7fbce9ae18c0 1 -- 172.21.15.16:0/2805451625 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56290df2fe00 con 0x56290ee2d000 2024-03-31T15:14:53.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.557+0000 7fbce9ae18c0 4 datascan.init: resolving metadata pool 16 2024-03-31T15:14:53.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.561+0000 7fbce0400640 1 -- 172.21.15.16:0/2805451625 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x56290e1f8c00 con 0x56290ee2d000 2024-03-31T15:14:53.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.561+0000 7fbce0400640 1 --2- 172.21.15.16:0/2805451625 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x56290e1ecc00 0x56290ecce100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:53.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.561+0000 7fbce0400640 1 -- 172.21.15.16:0/2805451625 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 5939+0+0 (secure 0 0 0) 0x56290e1f0a00 con 0x56290ee2d000 2024-03-31T15:14:53.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.561+0000 7fbce9ae18c0 4 datascan.init: found metadata pool 'cephfs_metadata' 2024-03-31T15:14:53.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.561+0000 7fbce9ae18c0 4 datascan.main: resolving metadata pool 16 2024-03-31T15:14:53.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.561+0000 7fbce9ae18c0 1 --2- 172.21.15.16:0/2805451625 >> [v2:172.21.15.16:6800/932712244,v1:172.21.15.16:6801/932712244] conn(0x56290ee2dc00 0x56290ecce680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:53.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.561+0000 7fbce9ae18c0 1 -- 172.21.15.16:0/2805451625 --> [v2:172.21.15.16:6800/932712244,v1:172.21.15.16:6801/932712244] -- osd_op(unknown.0.0:1 16.0 16:00000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x56290e1ed000 con 0x56290ee2dc00 2024-03-31T15:14:53.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.561+0000 7fbce7c00640 1 --2- 172.21.15.16:0/2805451625 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x56290e1ecc00 0x56290ecce100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:53.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.561+0000 7fbce7200640 1 --2- 172.21.15.16:0/2805451625 >> [v2:172.21.15.16:6800/932712244,v1:172.21.15.16:6801/932712244] conn(0x56290ee2dc00 0x56290ecce680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:53.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.561+0000 7fbce7c00640 1 --2- 172.21.15.16:0/2805451625 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x56290e1ecc00 0x56290ecce100 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x56290ee3cc90 tx=0x56290dfcee40 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:53.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.561+0000 7fbce7200640 1 --2- 172.21.15.16:0/2805451625 >> [v2:172.21.15.16:6800/932712244,v1:172.21.15.16:6801/932712244] conn(0x56290ee2dc00 0x56290ecce680 crc :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:53.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.561+0000 7fbce7200640 1 -- 172.21.15.16:0/2805451625 <== osd.0 v2:172.21.15.16:6800/932712244 1 ==== osd_op_reply(1 [pgnls start_epoch 113 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x56290e07c000 con 0x56290ee2dc00 2024-03-31T15:14:53.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.561+0000 7fbce7200640 1 --2- 172.21.15.16:0/2805451625 >> [v2:172.21.15.16:6819/3018526646,v1:172.21.15.16:6822/3018526646] conn(0x56290e1ed000 0x56290e2f8000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:53.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.561+0000 7fbce7200640 1 -- 172.21.15.16:0/2805451625 --> [v2:172.21.15.16:6819/3018526646,v1:172.21.15.16:6822/3018526646] -- osd_op(unknown.0.0:2 16.10 16:08000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x56290ee2c800 con 0x56290e1ed000 2024-03-31T15:14:53.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.561+0000 7fbce8600640 1 --2- 172.21.15.16:0/2805451625 >> [v2:172.21.15.16:6819/3018526646,v1:172.21.15.16:6822/3018526646] conn(0x56290e1ed000 0x56290e2f8000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:53.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.561+0000 7fbce8600640 1 --2- 172.21.15.16:0/2805451625 >> [v2:172.21.15.16:6819/3018526646,v1:172.21.15.16:6822/3018526646] conn(0x56290e1ed000 0x56290e2f8000 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:53.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.561+0000 7fbce8600640 1 -- 172.21.15.16:0/2805451625 <== osd.2 v2:172.21.15.16:6819/3018526646 1 ==== osd_op_reply(2 [pgnls start_epoch 113 out=177b] v112'16 uv16 ondisk = 1) ==== 144+0+177 (crc 0 0 0) 0x56290e07c500 con 0x56290e1ed000 2024-03-31T15:14:53.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.561+0000 7fbce8600640 1 -- 172.21.15.16:0/2805451625 --> [v2:172.21.15.16:6800/932712244,v1:172.21.15.16:6801/932712244] -- osd_op(unknown.0.0:3 16.8 16:10000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x56290ee2c800 con 0x56290ee2dc00 2024-03-31T15:14:53.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.561+0000 7fbce7200640 1 -- 172.21.15.16:0/2805451625 <== osd.0 v2:172.21.15.16:6800/932712244 2 ==== osd_op_reply(3 [pgnls start_epoch 113 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x56290e07c000 con 0x56290ee2dc00 2024-03-31T15:14:53.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.561+0000 7fbce7200640 1 -- 172.21.15.16:0/2805451625 --> [v2:172.21.15.16:6800/932712244,v1:172.21.15.16:6801/932712244] -- osd_op(unknown.0.0:4 16.18 16:18000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x56290ee2c800 con 0x56290ee2dc00 2024-03-31T15:14:53.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.561+0000 7fbce7200640 1 -- 172.21.15.16:0/2805451625 <== osd.0 v2:172.21.15.16:6800/932712244 3 ==== osd_op_reply(4 [pgnls start_epoch 113 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x56290e07c000 con 0x56290ee2dc00 2024-03-31T15:14:53.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.561+0000 7fbce7200640 1 --2- 172.21.15.16:0/2805451625 >> [v2:172.21.15.16:6809/3360352974,v1:172.21.15.16:6812/3360352974] conn(0x56290ee2c800 0x56290e2f8580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:53.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.561+0000 7fbce7200640 1 -- 172.21.15.16:0/2805451625 --> [v2:172.21.15.16:6809/3360352974,v1:172.21.15.16:6812/3360352974] -- osd_op(unknown.0.0:5 16.4 16:20000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x56290e300000 con 0x56290ee2c800 2024-03-31T15:14:53.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.565+0000 7fbce7c00640 1 --2- 172.21.15.16:0/2805451625 >> [v2:172.21.15.16:6809/3360352974,v1:172.21.15.16:6812/3360352974] conn(0x56290ee2c800 0x56290e2f8580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:53.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.565+0000 7fbce7c00640 1 --2- 172.21.15.16:0/2805451625 >> [v2:172.21.15.16:6809/3360352974,v1:172.21.15.16:6812/3360352974] conn(0x56290ee2c800 0x56290e2f8580 crc :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:53.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.565+0000 7fbce7c00640 1 -- 172.21.15.16:0/2805451625 <== osd.3 v2:172.21.15.16:6809/3360352974 1 ==== osd_op_reply(5 [pgnls start_epoch 113 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x56290e07c780 con 0x56290ee2c800 2024-03-31T15:14:53.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.565+0000 7fbce7c00640 1 -- 172.21.15.16:0/2805451625 --> [v2:172.21.15.16:6800/932712244,v1:172.21.15.16:6801/932712244] -- osd_op(unknown.0.0:6 16.14 16:28000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x56290e300000 con 0x56290ee2dc00 2024-03-31T15:14:53.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.565+0000 7fbce7200640 1 -- 172.21.15.16:0/2805451625 <== osd.0 v2:172.21.15.16:6800/932712244 4 ==== osd_op_reply(6 [pgnls start_epoch 113 out=149b] v113'43 uv43 ondisk = 1) ==== 144+0+149 (crc 0 0 0) 0x56290e07c000 con 0x56290ee2dc00 2024-03-31T15:14:53.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.565+0000 7fbce7200640 1 --2- 172.21.15.16:0/2805451625 >> [v2:172.21.15.89:6810/50395527,v1:172.21.15.89:6814/50395527] conn(0x56290e300000 0x56290e2f8b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:53.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.565+0000 7fbce7200640 1 -- 172.21.15.16:0/2805451625 --> [v2:172.21.15.89:6810/50395527,v1:172.21.15.89:6814/50395527] -- osd_op(unknown.0.0:7 16.c 16:30000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x56290e300400 con 0x56290e300000 2024-03-31T15:14:53.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.565+0000 7fbce7200640 1 --2- 172.21.15.16:0/2805451625 >> [v2:172.21.15.89:6810/50395527,v1:172.21.15.89:6814/50395527] conn(0x56290e300000 0x56290e2f8b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:53.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.565+0000 7fbce7200640 1 --2- 172.21.15.16:0/2805451625 >> [v2:172.21.15.89:6810/50395527,v1:172.21.15.89:6814/50395527] conn(0x56290e300000 0x56290e2f8b00 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:53.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.565+0000 7fbce7200640 1 -- 172.21.15.16:0/2805451625 <== osd.5 v2:172.21.15.89:6810/50395527 1 ==== osd_op_reply(7 [pgnls start_epoch 113 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x56290e07c000 con 0x56290e300000 2024-03-31T15:14:53.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.565+0000 7fbce7200640 1 -- 172.21.15.16:0/2805451625 --> [v2:172.21.15.16:6819/3018526646,v1:172.21.15.16:6822/3018526646] -- osd_op(unknown.0.0:8 16.1c 16:38000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x56290e300400 con 0x56290e1ed000 2024-03-31T15:14:53.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.565+0000 7fbce8600640 1 -- 172.21.15.16:0/2805451625 <== osd.2 v2:172.21.15.16:6819/3018526646 2 ==== osd_op_reply(8 [pgnls start_epoch 113 out=73b] v112'3 uv3 ondisk = 1) ==== 144+0+73 (crc 0 0 0) 0x56290e07c500 con 0x56290e1ed000 2024-03-31T15:14:53.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.565+0000 7fbce8600640 1 --2- 172.21.15.16:0/2805451625 >> [v2:172.21.15.89:6800/1856844856,v1:172.21.15.89:6801/1856844856] conn(0x56290e300400 0x56290e2f9080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:53.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.565+0000 7fbce8600640 1 -- 172.21.15.16:0/2805451625 --> [v2:172.21.15.89:6800/1856844856,v1:172.21.15.89:6801/1856844856] -- osd_op(unknown.0.0:9 16.2 16:40000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x56290eced400 con 0x56290e300400 2024-03-31T15:14:53.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.565+0000 7fbce8600640 1 --2- 172.21.15.16:0/2805451625 >> [v2:172.21.15.89:6800/1856844856,v1:172.21.15.89:6801/1856844856] conn(0x56290e300400 0x56290e2f9080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:53.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.569+0000 7fbce8600640 1 --2- 172.21.15.16:0/2805451625 >> [v2:172.21.15.89:6800/1856844856,v1:172.21.15.89:6801/1856844856] conn(0x56290e300400 0x56290e2f9080 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:53.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.569+0000 7fbce8600640 1 -- 172.21.15.16:0/2805451625 <== osd.4 v2:172.21.15.89:6800/1856844856 1 ==== osd_op_reply(9 [pgnls start_epoch 113 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x56290e07c500 con 0x56290e300400 2024-03-31T15:14:53.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.569+0000 7fbce8600640 1 --2- 172.21.15.16:0/2805451625 >> [v2:172.21.15.89:6809/3470471752,v1:172.21.15.89:6812/3470471752] conn(0x56290eced400 0x56290e2f9600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:53.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.569+0000 7fbce8600640 1 -- 172.21.15.16:0/2805451625 --> [v2:172.21.15.89:6809/3470471752,v1:172.21.15.89:6812/3470471752] -- osd_op(unknown.0.0:10 16.12 16:48000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x56290e301000 con 0x56290eced400 2024-03-31T15:14:53.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.569+0000 7fbce7c00640 1 --2- 172.21.15.16:0/2805451625 >> [v2:172.21.15.89:6809/3470471752,v1:172.21.15.89:6812/3470471752] conn(0x56290eced400 0x56290e2f9600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:53.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.569+0000 7fbce7c00640 1 --2- 172.21.15.16:0/2805451625 >> [v2:172.21.15.89:6809/3470471752,v1:172.21.15.89:6812/3470471752] conn(0x56290eced400 0x56290e2f9600 crc :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:53.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.569+0000 7fbce7c00640 1 -- 172.21.15.16:0/2805451625 <== osd.6 v2:172.21.15.89:6809/3470471752 1 ==== osd_op_reply(10 [pgnls start_epoch 113 out=81b] v112'3 uv3 ondisk = 1) ==== 144+0+81 (crc 0 0 0) 0x56290e07c780 con 0x56290eced400 2024-03-31T15:14:53.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.569+0000 7fbce7c00640 1 -- 172.21.15.16:0/2805451625 --> [v2:172.21.15.16:6819/3018526646,v1:172.21.15.16:6822/3018526646] -- osd_op(unknown.0.0:11 16.a 16:50000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x56290e301000 con 0x56290e1ed000 2024-03-31T15:14:53.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.569+0000 7fbce8600640 1 -- 172.21.15.16:0/2805451625 <== osd.2 v2:172.21.15.16:6819/3018526646 3 ==== osd_op_reply(11 [pgnls start_epoch 113 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x56290e07c500 con 0x56290e1ed000 2024-03-31T15:14:53.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.569+0000 7fbce8600640 1 -- 172.21.15.16:0/2805451625 --> [v2:172.21.15.16:6809/3360352974,v1:172.21.15.16:6812/3360352974] -- osd_op(unknown.0.0:12 16.1a 16:58000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x56290e301000 con 0x56290ee2c800 2024-03-31T15:14:53.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.569+0000 7fbce7c00640 1 -- 172.21.15.16:0/2805451625 <== osd.3 v2:172.21.15.16:6809/3360352974 2 ==== osd_op_reply(12 [pgnls start_epoch 113 out=49b] v113'3 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x56290e07c780 con 0x56290ee2c800 2024-03-31T15:14:53.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.569+0000 7fbce7c00640 1 -- 172.21.15.16:0/2805451625 --> [v2:172.21.15.89:6809/3470471752,v1:172.21.15.89:6812/3470471752] -- osd_op(unknown.0.0:13 16.6 16:60000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x56290e301000 con 0x56290eced400 2024-03-31T15:14:53.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.569+0000 7fbce7c00640 1 -- 172.21.15.16:0/2805451625 <== osd.6 v2:172.21.15.89:6809/3470471752 2 ==== osd_op_reply(13 [pgnls start_epoch 113 out=77b] v112'10 uv10 ondisk = 1) ==== 144+0+77 (crc 0 0 0) 0x56290e07c780 con 0x56290eced400 2024-03-31T15:14:53.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.569+0000 7fbce7c00640 1 -- 172.21.15.16:0/2805451625 --> [v2:172.21.15.16:6809/3360352974,v1:172.21.15.16:6812/3360352974] -- osd_op(unknown.0.0:14 16.16 16:68000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x56290e301000 con 0x56290ee2c800 2024-03-31T15:14:53.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.569+0000 7fbce7c00640 1 -- 172.21.15.16:0/2805451625 <== osd.3 v2:172.21.15.16:6809/3360352974 3 ==== osd_op_reply(14 [pgnls start_epoch 113 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x56290e07c780 con 0x56290ee2c800 2024-03-31T15:14:53.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.569+0000 7fbce7c00640 1 -- 172.21.15.16:0/2805451625 --> [v2:172.21.15.16:6819/3018526646,v1:172.21.15.16:6822/3018526646] -- osd_op(unknown.0.0:15 16.e 16:70000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x56290e301000 con 0x56290e1ed000 2024-03-31T15:14:53.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.569+0000 7fbce8600640 1 -- 172.21.15.16:0/2805451625 <== osd.2 v2:172.21.15.16:6819/3018526646 4 ==== osd_op_reply(15 [pgnls start_epoch 113 out=73b] v112'3 uv3 ondisk = 1) ==== 144+0+73 (crc 0 0 0) 0x56290e07c500 con 0x56290e1ed000 2024-03-31T15:14:53.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.569+0000 7fbce8600640 1 --2- 172.21.15.16:0/2805451625 >> [v2:172.21.15.89:6808/1796778630,v1:172.21.15.89:6811/1796778630] conn(0x56290e301000 0x56290e2f9b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:53.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.569+0000 7fbce8600640 1 -- 172.21.15.16:0/2805451625 --> [v2:172.21.15.89:6808/1796778630,v1:172.21.15.89:6811/1796778630] -- osd_op(unknown.0.0:16 16.1e 16:78000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x56290e301400 con 0x56290e301000 2024-03-31T15:14:53.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.569+0000 7fbce7200640 1 --2- 172.21.15.16:0/2805451625 >> [v2:172.21.15.89:6808/1796778630,v1:172.21.15.89:6811/1796778630] conn(0x56290e301000 0x56290e2f9b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:53.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.573+0000 7fbce7200640 1 --2- 172.21.15.16:0/2805451625 >> [v2:172.21.15.89:6808/1796778630,v1:172.21.15.89:6811/1796778630] conn(0x56290e301000 0x56290e2f9b80 crc :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:53.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.573+0000 7fbce7200640 1 -- 172.21.15.16:0/2805451625 <== osd.7 v2:172.21.15.89:6808/1796778630 1 ==== osd_op_reply(16 [pgnls start_epoch 113 out=49b] v113'3 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x56290e07c000 con 0x56290e301000 2024-03-31T15:14:53.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.573+0000 7fbce7200640 1 -- 172.21.15.16:0/2805451625 --> [v2:172.21.15.16:6819/3018526646,v1:172.21.15.16:6822/3018526646] -- osd_op(unknown.0.0:17 16.1 16:80000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x56290e301400 con 0x56290e1ed000 2024-03-31T15:14:53.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.573+0000 7fbce8600640 1 -- 172.21.15.16:0/2805451625 <== osd.2 v2:172.21.15.16:6819/3018526646 5 ==== osd_op_reply(17 [pgnls start_epoch 113 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x56290e07c500 con 0x56290e1ed000 2024-03-31T15:14:53.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.573+0000 7fbce8600640 1 -- 172.21.15.16:0/2805451625 --> [v2:172.21.15.89:6810/50395527,v1:172.21.15.89:6814/50395527] -- osd_op(unknown.0.0:18 16.11 16:88000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x56290e301400 con 0x56290e300000 2024-03-31T15:14:53.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.573+0000 7fbce7200640 1 -- 172.21.15.16:0/2805451625 <== osd.5 v2:172.21.15.89:6810/50395527 2 ==== osd_op_reply(18 [pgnls start_epoch 113 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x56290e07c000 con 0x56290e300000 2024-03-31T15:14:53.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.573+0000 7fbce7200640 1 --2- 172.21.15.16:0/2805451625 >> [v2:172.21.15.16:6808/1398737533,v1:172.21.15.16:6810/1398737533] conn(0x56290e301400 0x56290e2fa100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:53.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.573+0000 7fbce7200640 1 -- 172.21.15.16:0/2805451625 --> [v2:172.21.15.16:6808/1398737533,v1:172.21.15.16:6810/1398737533] -- osd_op(unknown.0.0:19 16.9 16:90000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x56290e300800 con 0x56290e301400 2024-03-31T15:14:53.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.573+0000 7fbce8600640 1 --2- 172.21.15.16:0/2805451625 >> [v2:172.21.15.16:6808/1398737533,v1:172.21.15.16:6810/1398737533] conn(0x56290e301400 0x56290e2fa100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:53.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.573+0000 7fbce8600640 1 --2- 172.21.15.16:0/2805451625 >> [v2:172.21.15.16:6808/1398737533,v1:172.21.15.16:6810/1398737533] conn(0x56290e301400 0x56290e2fa100 crc :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:53.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.573+0000 7fbce8600640 1 -- 172.21.15.16:0/2805451625 <== osd.1 v2:172.21.15.16:6808/1398737533 1 ==== osd_op_reply(19 [pgnls start_epoch 113 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x56290e07c500 con 0x56290e301400 2024-03-31T15:14:53.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.573+0000 7fbce8600640 1 -- 172.21.15.16:0/2805451625 --> [v2:172.21.15.89:6800/1856844856,v1:172.21.15.89:6801/1856844856] -- osd_op(unknown.0.0:20 16.19 16:98000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x56290e300800 con 0x56290e300400 2024-03-31T15:14:53.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.573+0000 7fbce8600640 1 -- 172.21.15.16:0/2805451625 <== osd.4 v2:172.21.15.89:6800/1856844856 2 ==== osd_op_reply(20 [pgnls start_epoch 113 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x56290e07c500 con 0x56290e300400 2024-03-31T15:14:53.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.573+0000 7fbce8600640 1 -- 172.21.15.16:0/2805451625 --> [v2:172.21.15.16:6800/932712244,v1:172.21.15.16:6801/932712244] -- osd_op(unknown.0.0:21 16.5 16:a0000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x56290e300800 con 0x56290ee2dc00 2024-03-31T15:14:53.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.573+0000 7fbce7200640 1 -- 172.21.15.16:0/2805451625 <== osd.0 v2:172.21.15.16:6800/932712244 5 ==== osd_op_reply(21 [pgnls start_epoch 113 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x56290e07c000 con 0x56290ee2dc00 2024-03-31T15:14:53.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.573+0000 7fbce7200640 1 -- 172.21.15.16:0/2805451625 --> [v2:172.21.15.16:6800/932712244,v1:172.21.15.16:6801/932712244] -- osd_op(unknown.0.0:22 16.15 16:a8000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x56290e300800 con 0x56290ee2dc00 2024-03-31T15:14:53.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.573+0000 7fbce7200640 1 -- 172.21.15.16:0/2805451625 <== osd.0 v2:172.21.15.16:6800/932712244 6 ==== osd_op_reply(22 [pgnls start_epoch 113 out=97b] v112'4 uv4 ondisk = 1) ==== 144+0+97 (crc 0 0 0) 0x56290e07c000 con 0x56290ee2dc00 2024-03-31T15:14:53.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.573+0000 7fbce7200640 1 -- 172.21.15.16:0/2805451625 --> [v2:172.21.15.89:6809/3470471752,v1:172.21.15.89:6812/3470471752] -- osd_op(unknown.0.0:23 16.d 16:b0000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x56290e300800 con 0x56290eced400 2024-03-31T15:14:53.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.573+0000 7fbce7c00640 1 -- 172.21.15.16:0/2805451625 <== osd.6 v2:172.21.15.89:6809/3470471752 3 ==== osd_op_reply(23 [pgnls start_epoch 113 out=74b] v108'1 uv1 ondisk = 1) ==== 144+0+74 (crc 0 0 0) 0x56290e07c780 con 0x56290eced400 2024-03-31T15:14:53.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.573+0000 7fbce7c00640 1 -- 172.21.15.16:0/2805451625 --> [v2:172.21.15.16:6819/3018526646,v1:172.21.15.16:6822/3018526646] -- osd_op(unknown.0.0:24 16.1d 16:b8000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x56290e300800 con 0x56290e1ed000 2024-03-31T15:14:53.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.573+0000 7fbce8600640 1 -- 172.21.15.16:0/2805451625 <== osd.2 v2:172.21.15.16:6819/3018526646 6 ==== osd_op_reply(24 [pgnls start_epoch 113 out=73b] v112'3 uv3 ondisk = 1) ==== 144+0+73 (crc 0 0 0) 0x56290e07c500 con 0x56290e1ed000 2024-03-31T15:14:53.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.573+0000 7fbce8600640 1 -- 172.21.15.16:0/2805451625 --> [v2:172.21.15.89:6810/50395527,v1:172.21.15.89:6814/50395527] -- osd_op(unknown.0.0:25 16.3 16:c0000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x56290e300800 con 0x56290e300000 2024-03-31T15:14:53.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.577+0000 7fbce7200640 1 -- 172.21.15.16:0/2805451625 <== osd.5 v2:172.21.15.89:6810/50395527 3 ==== osd_op_reply(25 [pgnls start_epoch 113 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x56290e07c000 con 0x56290e300000 2024-03-31T15:14:53.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.577+0000 7fbce7200640 1 -- 172.21.15.16:0/2805451625 --> [v2:172.21.15.16:6808/1398737533,v1:172.21.15.16:6810/1398737533] -- osd_op(unknown.0.0:26 16.13 16:c8000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x56290e300800 con 0x56290e301400 2024-03-31T15:14:53.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.577+0000 7fbce8600640 1 -- 172.21.15.16:0/2805451625 <== osd.1 v2:172.21.15.16:6808/1398737533 2 ==== osd_op_reply(26 [pgnls start_epoch 113 out=98b] v112'5 uv5 ondisk = 1) ==== 144+0+98 (crc 0 0 0) 0x56290e07c500 con 0x56290e301400 2024-03-31T15:14:53.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.577+0000 7fbce8600640 1 -- 172.21.15.16:0/2805451625 --> [v2:172.21.15.16:6808/1398737533,v1:172.21.15.16:6810/1398737533] -- osd_op(unknown.0.0:27 16.b 16:d0000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x56290e300800 con 0x56290e301400 2024-03-31T15:14:53.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.577+0000 7fbce8600640 1 -- 172.21.15.16:0/2805451625 <== osd.1 v2:172.21.15.16:6808/1398737533 3 ==== osd_op_reply(27 [pgnls start_epoch 113 out=76b] v113'6 uv5 ondisk = 1) ==== 144+0+76 (crc 0 0 0) 0x56290e07c500 con 0x56290e301400 2024-03-31T15:14:53.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.577+0000 7fbce8600640 1 -- 172.21.15.16:0/2805451625 --> [v2:172.21.15.89:6810/50395527,v1:172.21.15.89:6814/50395527] -- osd_op(unknown.0.0:28 16.1b 16:d8000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x56290e300800 con 0x56290e300000 2024-03-31T15:14:53.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.577+0000 7fbce7200640 1 -- 172.21.15.16:0/2805451625 <== osd.5 v2:172.21.15.89:6810/50395527 4 ==== osd_op_reply(28 [pgnls start_epoch 113 out=49b] v113'3 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x56290e07c000 con 0x56290e300000 2024-03-31T15:14:53.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.577+0000 7fbce7200640 1 -- 172.21.15.16:0/2805451625 --> [v2:172.21.15.16:6809/3360352974,v1:172.21.15.16:6812/3360352974] -- osd_op(unknown.0.0:29 16.7 16:e0000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x56290e300800 con 0x56290ee2c800 2024-03-31T15:14:53.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.577+0000 7fbce7c00640 1 -- 172.21.15.16:0/2805451625 <== osd.3 v2:172.21.15.16:6809/3360352974 4 ==== osd_op_reply(29 [pgnls start_epoch 113 out=105b] v113'9 uv9 ondisk = 1) ==== 144+0+105 (crc 0 0 0) 0x56290e07c780 con 0x56290ee2c800 2024-03-31T15:14:53.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.577+0000 7fbce7c00640 1 -- 172.21.15.16:0/2805451625 --> [v2:172.21.15.16:6819/3018526646,v1:172.21.15.16:6822/3018526646] -- osd_op(unknown.0.0:30 16.17 16:e8000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x56290e300800 con 0x56290e1ed000 2024-03-31T15:14:53.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.577+0000 7fbce8600640 1 -- 172.21.15.16:0/2805451625 <== osd.2 v2:172.21.15.16:6819/3018526646 7 ==== osd_op_reply(30 [pgnls start_epoch 113 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x56290e07c500 con 0x56290e1ed000 2024-03-31T15:14:53.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.577+0000 7fbce8600640 1 -- 172.21.15.16:0/2805451625 --> [v2:172.21.15.16:6800/932712244,v1:172.21.15.16:6801/932712244] -- osd_op(unknown.0.0:31 16.f 16:f0000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x56290e300800 con 0x56290ee2dc00 2024-03-31T15:14:53.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.577+0000 7fbce7200640 1 -- 172.21.15.16:0/2805451625 <== osd.0 v2:172.21.15.16:6800/932712244 7 ==== osd_op_reply(31 [pgnls start_epoch 113 out=73b] v113'2 uv2 ondisk = 1) ==== 144+0+73 (crc 0 0 0) 0x56290e07c000 con 0x56290ee2dc00 2024-03-31T15:14:53.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.577+0000 7fbce7200640 1 -- 172.21.15.16:0/2805451625 --> [v2:172.21.15.89:6810/50395527,v1:172.21.15.89:6814/50395527] -- osd_op(unknown.0.0:32 16.1f 16:f8000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x56290e300800 con 0x56290e300000 2024-03-31T15:14:53.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.577+0000 7fbce7200640 1 -- 172.21.15.16:0/2805451625 <== osd.5 v2:172.21.15.89:6810/50395527 5 ==== osd_op_reply(32 [pgnls start_epoch 113 out=101b] v113'9 uv8 ondisk = 1) ==== 144+0+101 (crc 0 0 0) 0x56290e07c000 con 0x56290e300000 2024-03-31T15:14:53.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.577+0000 7fbce9ae18c0 10 datascan.scan_links: step 1: handling object 601.00000000 2024-03-31T15:14:53.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.577+0000 7fbce9ae18c0 1 -- 172.21.15.16:0/2805451625 --> [v2:172.21.15.16:6819/3018526646,v1:172.21.15.16:6822/3018526646] -- osd_op(unknown.0.0:33 16.10 16:09401a81:::601.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e113) -- 0x56290e1ed400 con 0x56290e1ed000 2024-03-31T15:14:53.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.577+0000 7fbce8600640 1 -- 172.21.15.16:0/2805451625 <== osd.2 v2:172.21.15.16:6819/3018526646 8 ==== osd_op_reply(33 601.00000000 [omap-get-vals out=5b] v0'0 uv13 ondisk = 0) ==== 156+0+5 (crc 0 0 0) 0x56290e07c500 con 0x56290e1ed000 2024-03-31T15:14:53.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.577+0000 7fbce9ae18c0 10 datascan.scan_links: step 1: handling object 602.00000000 2024-03-31T15:14:53.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.577+0000 7fbce9ae18c0 1 -- 172.21.15.16:0/2805451625 --> [v2:172.21.15.16:6819/3018526646,v1:172.21.15.16:6822/3018526646] -- osd_op(unknown.0.0:34 16.10 16:0bd6d154:::602.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e113) -- 0x56290e1ed800 con 0x56290e1ed000 2024-03-31T15:14:53.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.577+0000 7fbce8600640 1 -- 172.21.15.16:0/2805451625 <== osd.2 v2:172.21.15.16:6819/3018526646 9 ==== osd_op_reply(34 602.00000000 [omap-get-vals out=5b] v0'0 uv15 ondisk = 0) ==== 156+0+5 (crc 0 0 0) 0x56290e07c500 con 0x56290e1ed000 2024-03-31T15:14:53.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.577+0000 7fbce9ae18c0 10 datascan.scan_links: step 1: handling object 10000000000.00000000 2024-03-31T15:14:53.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.577+0000 7fbce9ae18c0 1 -- 172.21.15.16:0/2805451625 --> [v2:172.21.15.16:6819/3018526646,v1:172.21.15.16:6822/3018526646] -- osd_op(unknown.0.0:35 16.10 16:0cf6ad0f:::10000000000.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e113) -- 0x56290e1edc00 con 0x56290e1ed000 2024-03-31T15:14:53.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.577+0000 7fbce8600640 1 -- 172.21.15.16:0/2805451625 <== osd.2 v2:172.21.15.16:6819/3018526646 10 ==== osd_op_reply(35 10000000000.00000000 [omap-get-vals out=1047b] v0'0 uv12 ondisk = 0) ==== 164+0+1047 (crc 0 0 0) 0x56290e07c500 con 0x56290e1ed000 2024-03-31T15:14:53.577 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.577+0000 7fbce9ae18c0 10 datascan.scan_links: step 1: handling object 600.00000000 2024-03-31T15:14:53.577 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.577+0000 7fbce9ae18c0 1 -- 172.21.15.16:0/2805451625 --> [v2:172.21.15.16:6819/3018526646,v1:172.21.15.16:6822/3018526646] -- osd_op(unknown.0.0:36 16.10 16:0d82a743:::600.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e113) -- 0x56290e31e000 con 0x56290e1ed000 2024-03-31T15:14:53.577 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.581+0000 7fbce8600640 1 -- 172.21.15.16:0/2805451625 <== osd.2 v2:172.21.15.16:6819/3018526646 11 ==== osd_op_reply(36 600.00000000 [omap-get-vals out=5b] v0'0 uv14 ondisk = 0) ==== 156+0+5 (crc 0 0 0) 0x56290e07c500 con 0x56290e1ed000 2024-03-31T15:14:53.577 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.581+0000 7fbce9ae18c0 10 datascan.scan_links: step 1: handling object 603.00000000 2024-03-31T15:14:53.577 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.581+0000 7fbce9ae18c0 1 -- 172.21.15.16:0/2805451625 --> [v2:172.21.15.16:6819/3018526646,v1:172.21.15.16:6822/3018526646] -- osd_op(unknown.0.0:37 16.10 16:0d89b25e:::603.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e113) -- 0x56290e31e400 con 0x56290e1ed000 2024-03-31T15:14:53.577 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.581+0000 7fbce8600640 1 -- 172.21.15.16:0/2805451625 <== osd.2 v2:172.21.15.16:6819/3018526646 12 ==== osd_op_reply(37 603.00000000 [omap-get-vals out=5b] v0'0 uv16 ondisk = 0) ==== 156+0+5 (crc 0 0 0) 0x56290e07c500 con 0x56290e1ed000 2024-03-31T15:14:53.577 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.581+0000 7fbce9ae18c0 10 datascan.scan_links: step 1: handling object 1.00000000.inode 2024-03-31T15:14:53.577 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.581+0000 7fbce9ae18c0 10 datascan.scan_links: Not a dirfrag: '1.00000000.inode' 2024-03-31T15:14:53.577 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.581+0000 7fbce9ae18c0 10 datascan.scan_links: step 1: handling object 200.00000000 2024-03-31T15:14:53.577 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.581+0000 7fbce9ae18c0 10 datascan.scan_links: Not a dirfrag (invalid ino): '200.00000000' 2024-03-31T15:14:53.577 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.581+0000 7fbce9ae18c0 10 datascan.scan_links: step 1: handling object 200.00000001 2024-03-31T15:14:53.577 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.581+0000 7fbce9ae18c0 10 datascan.scan_links: Not a dirfrag (invalid ino): '200.00000001' 2024-03-31T15:14:53.577 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.581+0000 7fbce9ae18c0 10 datascan.scan_links: step 1: handling object 606.00000000 2024-03-31T15:14:53.577 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.581+0000 7fbce9ae18c0 1 -- 172.21.15.16:0/2805451625 --> [v2:172.21.15.16:6800/932712244,v1:172.21.15.16:6801/932712244] -- osd_op(unknown.0.0:38 16.14 16:2f55791f:::606.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e113) -- 0x56290e31e800 con 0x56290ee2dc00 2024-03-31T15:14:53.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.581+0000 7fbce7200640 1 -- 172.21.15.16:0/2805451625 <== osd.0 v2:172.21.15.16:6800/932712244 8 ==== osd_op_reply(38 606.00000000 [omap-get-vals out=5b] v0'0 uv38 ondisk = 0) ==== 156+0+5 (crc 0 0 0) 0x56290e07c000 con 0x56290ee2dc00 2024-03-31T15:14:53.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.581+0000 7fbce9ae18c0 10 datascan.scan_links: step 1: handling object 607.00000000 2024-03-31T15:14:53.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.581+0000 7fbce9ae18c0 1 -- 172.21.15.16:0/2805451625 --> [v2:172.21.15.16:6819/3018526646,v1:172.21.15.16:6822/3018526646] -- osd_op(unknown.0.0:39 16.1c 16:3ed09add:::607.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e113) -- 0x56290e31ec00 con 0x56290e1ed000 2024-03-31T15:14:53.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.581+0000 7fbce8600640 1 -- 172.21.15.16:0/2805451625 <== osd.2 v2:172.21.15.16:6819/3018526646 13 ==== osd_op_reply(39 607.00000000 [omap-get-vals out=5b] v0'0 uv3 ondisk = 0) ==== 156+0+5 (crc 0 0 0) 0x56290e07c500 con 0x56290e1ed000 2024-03-31T15:14:53.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.581+0000 7fbce9ae18c0 10 datascan.scan_links: step 1: handling object 10000000003.00000000 2024-03-31T15:14:53.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.581+0000 7fbce9ae18c0 1 -- 172.21.15.16:0/2805451625 --> [v2:172.21.15.89:6809/3470471752,v1:172.21.15.89:6812/3470471752] -- osd_op(unknown.0.0:40 16.12 16:4def72d4:::10000000003.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e113) -- 0x56290e31f000 con 0x56290eced400 2024-03-31T15:14:53.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.581+0000 7fbce7c00640 1 -- 172.21.15.16:0/2805451625 <== osd.6 v2:172.21.15.89:6809/3470471752 4 ==== osd_op_reply(40 10000000003.00000000 [omap-get-vals out=523b] v0'0 uv3 ondisk = 0) ==== 164+0+523 (crc 0 0 0) 0x56290e07c780 con 0x56290eced400 2024-03-31T15:14:53.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.581+0000 7fbce9ae18c0 10 datascan.scan_links: step 1: handling object mds0_openfiles.0 2024-03-31T15:14:53.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.581+0000 7fbce9ae18c0 10 datascan.scan_links: Not a dirfrag: 'mds0_openfiles.0' 2024-03-31T15:14:53.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.581+0000 7fbce9ae18c0 10 datascan.scan_links: step 1: handling object 608.00000000 2024-03-31T15:14:53.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.581+0000 7fbce9ae18c0 1 -- 172.21.15.16:0/2805451625 --> [v2:172.21.15.16:6819/3018526646,v1:172.21.15.16:6822/3018526646] -- osd_op(unknown.0.0:41 16.e 16:717a0223:::608.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e113) -- 0x56290e31f400 con 0x56290e1ed000 2024-03-31T15:14:53.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.581+0000 7fbce8600640 1 -- 172.21.15.16:0/2805451625 <== osd.2 v2:172.21.15.16:6819/3018526646 14 ==== osd_op_reply(41 608.00000000 [omap-get-vals out=5b] v0'0 uv3 ondisk = 0) ==== 156+0+5 (crc 0 0 0) 0x56290e07c500 con 0x56290e1ed000 2024-03-31T15:14:53.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.581+0000 7fbce9ae18c0 10 datascan.scan_links: step 1: handling object 604.00000000 2024-03-31T15:14:53.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.581+0000 7fbce9ae18c0 1 -- 172.21.15.16:0/2805451625 --> [v2:172.21.15.16:6800/932712244,v1:172.21.15.16:6801/932712244] -- osd_op(unknown.0.0:42 16.15 16:a93a17c2:::604.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e113) -- 0x56290e31f800 con 0x56290ee2dc00 2024-03-31T15:14:53.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.581+0000 7fbce7200640 1 -- 172.21.15.16:0/2805451625 <== osd.0 v2:172.21.15.16:6800/932712244 9 ==== osd_op_reply(42 604.00000000 [omap-get-vals out=5b] v0'0 uv4 ondisk = 0) ==== 156+0+5 (crc 0 0 0) 0x56290e07c000 con 0x56290ee2dc00 2024-03-31T15:14:53.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.581+0000 7fbce9ae18c0 10 datascan.scan_links: step 1: handling object 500.00000000 2024-03-31T15:14:53.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.581+0000 7fbce9ae18c0 10 datascan.scan_links: Not a dirfrag (invalid ino): '500.00000000' 2024-03-31T15:14:53.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.581+0000 7fbce9ae18c0 10 datascan.scan_links: step 1: handling object mds_snaptable 2024-03-31T15:14:53.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.581+0000 7fbce9ae18c0 10 datascan.scan_links: Not a dirfrag: 'mds_snaptable' 2024-03-31T15:14:53.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.581+0000 7fbce9ae18c0 10 datascan.scan_links: step 1: handling object 605.00000000 2024-03-31T15:14:53.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.581+0000 7fbce9ae18c0 1 -- 172.21.15.16:0/2805451625 --> [v2:172.21.15.16:6819/3018526646,v1:172.21.15.16:6822/3018526646] -- osd_op(unknown.0.0:43 16.1d 16:b871830b:::605.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e113) -- 0x56290e31fc00 con 0x56290e1ed000 2024-03-31T15:14:53.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.581+0000 7fbce8600640 1 -- 172.21.15.16:0/2805451625 <== osd.2 v2:172.21.15.16:6819/3018526646 15 ==== osd_op_reply(43 605.00000000 [omap-get-vals out=5b] v0'0 uv3 ondisk = 0) ==== 156+0+5 (crc 0 0 0) 0x56290e07c500 con 0x56290e1ed000 2024-03-31T15:14:53.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.581+0000 7fbce9ae18c0 10 datascan.scan_links: step 1: handling object mds0_inotable 2024-03-31T15:14:53.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.581+0000 7fbce9ae18c0 10 datascan.scan_links: Not a dirfrag: 'mds0_inotable' 2024-03-31T15:14:53.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.581+0000 7fbce9ae18c0 10 datascan.scan_links: step 1: handling object 100.00000000 2024-03-31T15:14:53.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.581+0000 7fbce9ae18c0 1 -- 172.21.15.16:0/2805451625 --> [v2:172.21.15.16:6808/1398737533,v1:172.21.15.16:6810/1398737533] -- osd_op(unknown.0.0:44 16.13 16:cd5a64a3:::100.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e113) -- 0x56290e31e000 con 0x56290e301400 2024-03-31T15:14:53.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.581+0000 7fbce8600640 1 -- 172.21.15.16:0/2805451625 <== osd.1 v2:172.21.15.16:6808/1398737533 4 ==== osd_op_reply(44 100.00000000 [omap-get-vals out=5195b] v0'0 uv5 ondisk = 0) ==== 156+0+5195 (crc 0 0 0) 0x56290e07c500 con 0x56290e301400 2024-03-31T15:14:53.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.581+0000 7fbce9ae18c0 10 datascan.scan_links: step 1: handling object mds0_sessionmap 2024-03-31T15:14:53.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.581+0000 7fbce9ae18c0 10 datascan.scan_links: Not a dirfrag: 'mds0_sessionmap' 2024-03-31T15:14:53.580 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.581+0000 7fbce9ae18c0 10 datascan.scan_links: step 1: handling object 609.00000000 2024-03-31T15:14:53.580 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.581+0000 7fbce9ae18c0 1 -- 172.21.15.16:0/2805451625 --> [v2:172.21.15.16:6809/3360352974,v1:172.21.15.16:6812/3360352974] -- osd_op(unknown.0.0:45 16.7 16:e0b41d06:::609.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e113) -- 0x56290e1ed400 con 0x56290ee2c800 2024-03-31T15:14:53.580 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.581+0000 7fbce7c00640 1 -- 172.21.15.16:0/2805451625 <== osd.3 v2:172.21.15.16:6809/3360352974 5 ==== osd_op_reply(45 609.00000000 [omap-get-vals out=5b] v0'0 uv7 ondisk = 0) ==== 156+0+5 (crc 0 0 0) 0x56290e07c780 con 0x56290ee2c800 2024-03-31T15:14:53.580 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.581+0000 7fbce9ae18c0 10 datascan.scan_links: step 1: handling object 10000000001.00000000 2024-03-31T15:14:53.580 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.581+0000 7fbce9ae18c0 1 -- 172.21.15.16:0/2805451625 --> [v2:172.21.15.16:6809/3360352974,v1:172.21.15.16:6812/3360352974] -- osd_op(unknown.0.0:46 16.7 16:e2e07930:::10000000001.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e113) -- 0x56290e1ed800 con 0x56290ee2c800 2024-03-31T15:14:53.580 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.581+0000 7fbce7c00640 1 -- 172.21.15.16:0/2805451625 <== osd.3 v2:172.21.15.16:6809/3360352974 6 ==== osd_op_reply(46 10000000001.00000000 [omap-get-vals out=1545b] v0'0 uv9 ondisk = 0) ==== 164+0+1545 (crc 0 0 0) 0x56290e07c780 con 0x56290ee2c800 2024-03-31T15:14:53.580 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.581+0000 7fbce9ae18c0 20 datascan.scan_links: injected ino detected 2024-03-31T15:14:53.580 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.581+0000 7fbce9ae18c0 10 datascan.scan_links: step 1: handling object 400.00000000 2024-03-31T15:14:53.580 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.581+0000 7fbce9ae18c0 10 datascan.scan_links: Not a dirfrag (invalid ino): '400.00000000' 2024-03-31T15:14:53.580 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.581+0000 7fbce9ae18c0 10 datascan.scan_links: step 1: handling object 100.00000000.inode 2024-03-31T15:14:53.580 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.581+0000 7fbce9ae18c0 10 datascan.scan_links: Not a dirfrag: '100.00000000.inode' 2024-03-31T15:14:53.580 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.581+0000 7fbce9ae18c0 10 datascan.scan_links: step 1: handling object 1.00000000 2024-03-31T15:14:53.580 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.581+0000 7fbce9ae18c0 1 -- 172.21.15.16:0/2805451625 --> [v2:172.21.15.89:6810/50395527,v1:172.21.15.89:6814/50395527] -- osd_op(unknown.0.0:47 16.1f 16:ff5b34d6:::1.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e113) -- 0x56290e1edc00 con 0x56290e300000 2024-03-31T15:14:53.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.585+0000 7fbce7200640 1 -- 172.21.15.16:0/2805451625 <== osd.5 v2:172.21.15.89:6810/50395527 6 ==== osd_op_reply(47 1.00000000 [omap-get-vals out=524b] v0'0 uv8 ondisk = 0) ==== 154+0+524 (crc 0 0 0) 0x56290e07c000 con 0x56290e300000 2024-03-31T15:14:53.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.585+0000 7fbce9ae18c0 1 -- 172.21.15.16:0/2805451625 --> [v2:172.21.15.16:6800/932712244,v1:172.21.15.16:6801/932712244] -- osd_op(unknown.0.0:48 16.0 16:00000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x56290e301c00 con 0x56290ee2dc00 2024-03-31T15:14:53.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.585+0000 7fbce7200640 1 -- 172.21.15.16:0/2805451625 <== osd.0 v2:172.21.15.16:6800/932712244 10 ==== osd_op_reply(48 [pgnls start_epoch 113 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x56290e07c000 con 0x56290ee2dc00 2024-03-31T15:14:53.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.585+0000 7fbce7200640 1 -- 172.21.15.16:0/2805451625 --> [v2:172.21.15.16:6819/3018526646,v1:172.21.15.16:6822/3018526646] -- osd_op(unknown.0.0:49 16.10 16:08000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x56290e31e800 con 0x56290e1ed000 2024-03-31T15:14:53.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.585+0000 7fbce8600640 1 -- 172.21.15.16:0/2805451625 <== osd.2 v2:172.21.15.16:6819/3018526646 16 ==== osd_op_reply(49 [pgnls start_epoch 113 out=177b] v112'16 uv16 ondisk = 1) ==== 144+0+177 (crc 0 0 0) 0x56290e07c500 con 0x56290e1ed000 2024-03-31T15:14:53.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.585+0000 7fbce8600640 1 -- 172.21.15.16:0/2805451625 --> [v2:172.21.15.16:6800/932712244,v1:172.21.15.16:6801/932712244] -- osd_op(unknown.0.0:50 16.8 16:10000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x56290e31e800 con 0x56290ee2dc00 2024-03-31T15:14:53.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.585+0000 7fbce7200640 1 -- 172.21.15.16:0/2805451625 <== osd.0 v2:172.21.15.16:6800/932712244 11 ==== osd_op_reply(50 [pgnls start_epoch 113 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x56290e07c000 con 0x56290ee2dc00 2024-03-31T15:14:53.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.585+0000 7fbce7200640 1 -- 172.21.15.16:0/2805451625 --> [v2:172.21.15.16:6800/932712244,v1:172.21.15.16:6801/932712244] -- osd_op(unknown.0.0:51 16.18 16:18000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x56290e31e800 con 0x56290ee2dc00 2024-03-31T15:14:53.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.585+0000 7fbce7200640 1 -- 172.21.15.16:0/2805451625 <== osd.0 v2:172.21.15.16:6800/932712244 12 ==== osd_op_reply(51 [pgnls start_epoch 113 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x56290e07c000 con 0x56290ee2dc00 2024-03-31T15:14:53.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.585+0000 7fbce7200640 1 -- 172.21.15.16:0/2805451625 --> [v2:172.21.15.16:6809/3360352974,v1:172.21.15.16:6812/3360352974] -- osd_op(unknown.0.0:52 16.4 16:20000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x56290e31e800 con 0x56290ee2c800 2024-03-31T15:14:53.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.585+0000 7fbce7c00640 1 -- 172.21.15.16:0/2805451625 <== osd.3 v2:172.21.15.16:6809/3360352974 7 ==== osd_op_reply(52 [pgnls start_epoch 113 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x56290e07c780 con 0x56290ee2c800 2024-03-31T15:14:53.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.585+0000 7fbce7c00640 1 -- 172.21.15.16:0/2805451625 --> [v2:172.21.15.16:6800/932712244,v1:172.21.15.16:6801/932712244] -- osd_op(unknown.0.0:53 16.14 16:28000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x56290e31e800 con 0x56290ee2dc00 2024-03-31T15:14:53.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.585+0000 7fbce7200640 1 -- 172.21.15.16:0/2805451625 <== osd.0 v2:172.21.15.16:6800/932712244 13 ==== osd_op_reply(53 [pgnls start_epoch 113 out=149b] v113'43 uv43 ondisk = 1) ==== 144+0+149 (crc 0 0 0) 0x56290e07c000 con 0x56290ee2dc00 2024-03-31T15:14:53.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.585+0000 7fbce7200640 1 -- 172.21.15.16:0/2805451625 --> [v2:172.21.15.89:6810/50395527,v1:172.21.15.89:6814/50395527] -- osd_op(unknown.0.0:54 16.c 16:30000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x56290e31e800 con 0x56290e300000 2024-03-31T15:14:53.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.585+0000 7fbce7200640 1 -- 172.21.15.16:0/2805451625 <== osd.5 v2:172.21.15.89:6810/50395527 7 ==== osd_op_reply(54 [pgnls start_epoch 113 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x56290e07c000 con 0x56290e300000 2024-03-31T15:14:53.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.585+0000 7fbce7200640 1 -- 172.21.15.16:0/2805451625 --> [v2:172.21.15.16:6819/3018526646,v1:172.21.15.16:6822/3018526646] -- osd_op(unknown.0.0:55 16.1c 16:38000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x56290e31e800 con 0x56290e1ed000 2024-03-31T15:14:53.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.585+0000 7fbce8600640 1 -- 172.21.15.16:0/2805451625 <== osd.2 v2:172.21.15.16:6819/3018526646 17 ==== osd_op_reply(55 [pgnls start_epoch 113 out=73b] v112'3 uv3 ondisk = 1) ==== 144+0+73 (crc 0 0 0) 0x56290e07c500 con 0x56290e1ed000 2024-03-31T15:14:53.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.585+0000 7fbce8600640 1 -- 172.21.15.16:0/2805451625 --> [v2:172.21.15.89:6800/1856844856,v1:172.21.15.89:6801/1856844856] -- osd_op(unknown.0.0:56 16.2 16:40000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x56290e31e800 con 0x56290e300400 2024-03-31T15:14:53.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.585+0000 7fbce8600640 1 -- 172.21.15.16:0/2805451625 <== osd.4 v2:172.21.15.89:6800/1856844856 3 ==== osd_op_reply(56 [pgnls start_epoch 113 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x56290e07c500 con 0x56290e300400 2024-03-31T15:14:53.585 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.585+0000 7fbce8600640 1 -- 172.21.15.16:0/2805451625 --> [v2:172.21.15.89:6809/3470471752,v1:172.21.15.89:6812/3470471752] -- osd_op(unknown.0.0:57 16.12 16:48000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x56290e31e800 con 0x56290eced400 2024-03-31T15:14:53.585 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.585+0000 7fbce7c00640 1 -- 172.21.15.16:0/2805451625 <== osd.6 v2:172.21.15.89:6809/3470471752 5 ==== osd_op_reply(57 [pgnls start_epoch 113 out=81b] v112'3 uv3 ondisk = 1) ==== 144+0+81 (crc 0 0 0) 0x56290e07c780 con 0x56290eced400 2024-03-31T15:14:53.585 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.585+0000 7fbce7c00640 1 -- 172.21.15.16:0/2805451625 --> [v2:172.21.15.16:6819/3018526646,v1:172.21.15.16:6822/3018526646] -- osd_op(unknown.0.0:58 16.a 16:50000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x56290e31e800 con 0x56290e1ed000 2024-03-31T15:14:53.585 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.589+0000 7fbce8600640 1 -- 172.21.15.16:0/2805451625 <== osd.2 v2:172.21.15.16:6819/3018526646 18 ==== osd_op_reply(58 [pgnls start_epoch 113 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x56290e07c500 con 0x56290e1ed000 2024-03-31T15:14:53.585 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.589+0000 7fbce8600640 1 -- 172.21.15.16:0/2805451625 --> [v2:172.21.15.16:6809/3360352974,v1:172.21.15.16:6812/3360352974] -- osd_op(unknown.0.0:59 16.1a 16:58000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x56290e31e800 con 0x56290ee2c800 2024-03-31T15:14:53.585 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.589+0000 7fbce7c00640 1 -- 172.21.15.16:0/2805451625 <== osd.3 v2:172.21.15.16:6809/3360352974 8 ==== osd_op_reply(59 [pgnls start_epoch 113 out=49b] v113'3 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x56290e07c780 con 0x56290ee2c800 2024-03-31T15:14:53.585 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.589+0000 7fbce7c00640 1 -- 172.21.15.16:0/2805451625 --> [v2:172.21.15.89:6809/3470471752,v1:172.21.15.89:6812/3470471752] -- osd_op(unknown.0.0:60 16.6 16:60000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x56290e31e800 con 0x56290eced400 2024-03-31T15:14:53.585 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.589+0000 7fbce7c00640 1 -- 172.21.15.16:0/2805451625 <== osd.6 v2:172.21.15.89:6809/3470471752 6 ==== osd_op_reply(60 [pgnls start_epoch 113 out=77b] v112'10 uv10 ondisk = 1) ==== 144+0+77 (crc 0 0 0) 0x56290e07c780 con 0x56290eced400 2024-03-31T15:14:53.585 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.589+0000 7fbce7c00640 1 -- 172.21.15.16:0/2805451625 --> [v2:172.21.15.16:6809/3360352974,v1:172.21.15.16:6812/3360352974] -- osd_op(unknown.0.0:61 16.16 16:68000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x56290e31e800 con 0x56290ee2c800 2024-03-31T15:14:53.585 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.589+0000 7fbce7c00640 1 -- 172.21.15.16:0/2805451625 <== osd.3 v2:172.21.15.16:6809/3360352974 9 ==== osd_op_reply(61 [pgnls start_epoch 113 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x56290e07c780 con 0x56290ee2c800 2024-03-31T15:14:53.585 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.589+0000 7fbce7c00640 1 -- 172.21.15.16:0/2805451625 --> [v2:172.21.15.16:6819/3018526646,v1:172.21.15.16:6822/3018526646] -- osd_op(unknown.0.0:62 16.e 16:70000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x56290e31e800 con 0x56290e1ed000 2024-03-31T15:14:53.585 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.589+0000 7fbce8600640 1 -- 172.21.15.16:0/2805451625 <== osd.2 v2:172.21.15.16:6819/3018526646 19 ==== osd_op_reply(62 [pgnls start_epoch 113 out=73b] v112'3 uv3 ondisk = 1) ==== 144+0+73 (crc 0 0 0) 0x56290e07c500 con 0x56290e1ed000 2024-03-31T15:14:53.585 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.589+0000 7fbce8600640 1 -- 172.21.15.16:0/2805451625 --> [v2:172.21.15.89:6808/1796778630,v1:172.21.15.89:6811/1796778630] -- osd_op(unknown.0.0:63 16.1e 16:78000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x56290e31e800 con 0x56290e301000 2024-03-31T15:14:53.585 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.589+0000 7fbce7200640 1 -- 172.21.15.16:0/2805451625 <== osd.7 v2:172.21.15.89:6808/1796778630 2 ==== osd_op_reply(63 [pgnls start_epoch 113 out=49b] v113'3 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x56290e07c000 con 0x56290e301000 2024-03-31T15:14:53.586 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.589+0000 7fbce7200640 1 -- 172.21.15.16:0/2805451625 --> [v2:172.21.15.16:6819/3018526646,v1:172.21.15.16:6822/3018526646] -- osd_op(unknown.0.0:64 16.1 16:80000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x56290e31e800 con 0x56290e1ed000 2024-03-31T15:14:53.586 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.589+0000 7fbce8600640 1 -- 172.21.15.16:0/2805451625 <== osd.2 v2:172.21.15.16:6819/3018526646 20 ==== osd_op_reply(64 [pgnls start_epoch 113 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x56290e07c500 con 0x56290e1ed000 2024-03-31T15:14:53.586 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.589+0000 7fbce8600640 1 -- 172.21.15.16:0/2805451625 --> [v2:172.21.15.89:6810/50395527,v1:172.21.15.89:6814/50395527] -- osd_op(unknown.0.0:65 16.11 16:88000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x56290e31e800 con 0x56290e300000 2024-03-31T15:14:53.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.589+0000 7fbce7200640 1 -- 172.21.15.16:0/2805451625 <== osd.5 v2:172.21.15.89:6810/50395527 8 ==== osd_op_reply(65 [pgnls start_epoch 113 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x56290e07c000 con 0x56290e300000 2024-03-31T15:14:53.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.589+0000 7fbce7200640 1 -- 172.21.15.16:0/2805451625 --> [v2:172.21.15.16:6808/1398737533,v1:172.21.15.16:6810/1398737533] -- osd_op(unknown.0.0:66 16.9 16:90000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x56290e31e800 con 0x56290e301400 2024-03-31T15:14:53.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.589+0000 7fbce8600640 1 -- 172.21.15.16:0/2805451625 <== osd.1 v2:172.21.15.16:6808/1398737533 5 ==== osd_op_reply(66 [pgnls start_epoch 113 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x56290e07c500 con 0x56290e301400 2024-03-31T15:14:53.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.589+0000 7fbce8600640 1 -- 172.21.15.16:0/2805451625 --> [v2:172.21.15.89:6800/1856844856,v1:172.21.15.89:6801/1856844856] -- osd_op(unknown.0.0:67 16.19 16:98000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x56290e31e800 con 0x56290e300400 2024-03-31T15:14:53.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.589+0000 7fbce8600640 1 -- 172.21.15.16:0/2805451625 <== osd.4 v2:172.21.15.89:6800/1856844856 4 ==== osd_op_reply(67 [pgnls start_epoch 113 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x56290e07c500 con 0x56290e300400 2024-03-31T15:14:53.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.589+0000 7fbce8600640 1 -- 172.21.15.16:0/2805451625 --> [v2:172.21.15.16:6800/932712244,v1:172.21.15.16:6801/932712244] -- osd_op(unknown.0.0:68 16.5 16:a0000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x56290e31e800 con 0x56290ee2dc00 2024-03-31T15:14:53.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.593+0000 7fbce7200640 1 -- 172.21.15.16:0/2805451625 <== osd.0 v2:172.21.15.16:6800/932712244 14 ==== osd_op_reply(68 [pgnls start_epoch 113 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x56290e07c000 con 0x56290ee2dc00 2024-03-31T15:14:53.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.593+0000 7fbce7200640 1 -- 172.21.15.16:0/2805451625 --> [v2:172.21.15.16:6800/932712244,v1:172.21.15.16:6801/932712244] -- osd_op(unknown.0.0:69 16.15 16:a8000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x56290e31e800 con 0x56290ee2dc00 2024-03-31T15:14:53.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.593+0000 7fbce7200640 1 -- 172.21.15.16:0/2805451625 <== osd.0 v2:172.21.15.16:6800/932712244 15 ==== osd_op_reply(69 [pgnls start_epoch 113 out=97b] v112'4 uv4 ondisk = 1) ==== 144+0+97 (crc 0 0 0) 0x56290e07c000 con 0x56290ee2dc00 2024-03-31T15:14:53.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.593+0000 7fbce7200640 1 -- 172.21.15.16:0/2805451625 --> [v2:172.21.15.89:6809/3470471752,v1:172.21.15.89:6812/3470471752] -- osd_op(unknown.0.0:70 16.d 16:b0000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x56290e31e800 con 0x56290eced400 2024-03-31T15:14:53.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.593+0000 7fbce7c00640 1 -- 172.21.15.16:0/2805451625 <== osd.6 v2:172.21.15.89:6809/3470471752 7 ==== osd_op_reply(70 [pgnls start_epoch 113 out=74b] v108'1 uv1 ondisk = 1) ==== 144+0+74 (crc 0 0 0) 0x56290e07c780 con 0x56290eced400 2024-03-31T15:14:53.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.593+0000 7fbce7c00640 1 -- 172.21.15.16:0/2805451625 --> [v2:172.21.15.16:6819/3018526646,v1:172.21.15.16:6822/3018526646] -- osd_op(unknown.0.0:71 16.1d 16:b8000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x56290e31e800 con 0x56290e1ed000 2024-03-31T15:14:53.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.593+0000 7fbce8600640 1 -- 172.21.15.16:0/2805451625 <== osd.2 v2:172.21.15.16:6819/3018526646 21 ==== osd_op_reply(71 [pgnls start_epoch 113 out=73b] v112'3 uv3 ondisk = 1) ==== 144+0+73 (crc 0 0 0) 0x56290e07c500 con 0x56290e1ed000 2024-03-31T15:14:53.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.593+0000 7fbce8600640 1 -- 172.21.15.16:0/2805451625 --> [v2:172.21.15.89:6810/50395527,v1:172.21.15.89:6814/50395527] -- osd_op(unknown.0.0:72 16.3 16:c0000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x56290e31e800 con 0x56290e300000 2024-03-31T15:14:53.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.593+0000 7fbce7200640 1 -- 172.21.15.16:0/2805451625 <== osd.5 v2:172.21.15.89:6810/50395527 9 ==== osd_op_reply(72 [pgnls start_epoch 113 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x56290e07c000 con 0x56290e300000 2024-03-31T15:14:53.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.593+0000 7fbce7200640 1 -- 172.21.15.16:0/2805451625 --> [v2:172.21.15.16:6808/1398737533,v1:172.21.15.16:6810/1398737533] -- osd_op(unknown.0.0:73 16.13 16:c8000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x56290e31e800 con 0x56290e301400 2024-03-31T15:14:53.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.593+0000 7fbce8600640 1 -- 172.21.15.16:0/2805451625 <== osd.1 v2:172.21.15.16:6808/1398737533 6 ==== osd_op_reply(73 [pgnls start_epoch 113 out=98b] v112'5 uv5 ondisk = 1) ==== 144+0+98 (crc 0 0 0) 0x56290e07c500 con 0x56290e301400 2024-03-31T15:14:53.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.593+0000 7fbce8600640 1 -- 172.21.15.16:0/2805451625 --> [v2:172.21.15.16:6808/1398737533,v1:172.21.15.16:6810/1398737533] -- osd_op(unknown.0.0:74 16.b 16:d0000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x56290e31e800 con 0x56290e301400 2024-03-31T15:14:53.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.593+0000 7fbce8600640 1 -- 172.21.15.16:0/2805451625 <== osd.1 v2:172.21.15.16:6808/1398737533 7 ==== osd_op_reply(74 [pgnls start_epoch 113 out=76b] v113'6 uv5 ondisk = 1) ==== 144+0+76 (crc 0 0 0) 0x56290e07c500 con 0x56290e301400 2024-03-31T15:14:53.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.593+0000 7fbce8600640 1 -- 172.21.15.16:0/2805451625 --> [v2:172.21.15.89:6810/50395527,v1:172.21.15.89:6814/50395527] -- osd_op(unknown.0.0:75 16.1b 16:d8000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x56290e31e800 con 0x56290e300000 2024-03-31T15:14:53.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.593+0000 7fbce7200640 1 -- 172.21.15.16:0/2805451625 <== osd.5 v2:172.21.15.89:6810/50395527 10 ==== osd_op_reply(75 [pgnls start_epoch 113 out=49b] v113'3 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x56290e07c000 con 0x56290e300000 2024-03-31T15:14:53.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.593+0000 7fbce7200640 1 -- 172.21.15.16:0/2805451625 --> [v2:172.21.15.16:6809/3360352974,v1:172.21.15.16:6812/3360352974] -- osd_op(unknown.0.0:76 16.7 16:e0000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x56290e31e800 con 0x56290ee2c800 2024-03-31T15:14:53.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.593+0000 7fbce7c00640 1 -- 172.21.15.16:0/2805451625 <== osd.3 v2:172.21.15.16:6809/3360352974 10 ==== osd_op_reply(76 [pgnls start_epoch 113 out=105b] v113'9 uv9 ondisk = 1) ==== 144+0+105 (crc 0 0 0) 0x56290e07c780 con 0x56290ee2c800 2024-03-31T15:14:53.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.593+0000 7fbce7c00640 1 -- 172.21.15.16:0/2805451625 --> [v2:172.21.15.16:6819/3018526646,v1:172.21.15.16:6822/3018526646] -- osd_op(unknown.0.0:77 16.17 16:e8000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x56290e31e800 con 0x56290e1ed000 2024-03-31T15:14:53.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.593+0000 7fbce8600640 1 -- 172.21.15.16:0/2805451625 <== osd.2 v2:172.21.15.16:6819/3018526646 22 ==== osd_op_reply(77 [pgnls start_epoch 113 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x56290e07c500 con 0x56290e1ed000 2024-03-31T15:14:53.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.593+0000 7fbce8600640 1 -- 172.21.15.16:0/2805451625 --> [v2:172.21.15.16:6800/932712244,v1:172.21.15.16:6801/932712244] -- osd_op(unknown.0.0:78 16.f 16:f0000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x56290e31e800 con 0x56290ee2dc00 2024-03-31T15:14:53.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.593+0000 7fbce7200640 1 -- 172.21.15.16:0/2805451625 <== osd.0 v2:172.21.15.16:6800/932712244 16 ==== osd_op_reply(78 [pgnls start_epoch 113 out=73b] v113'2 uv2 ondisk = 1) ==== 144+0+73 (crc 0 0 0) 0x56290e07c000 con 0x56290ee2dc00 2024-03-31T15:14:53.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.593+0000 7fbce7200640 1 -- 172.21.15.16:0/2805451625 --> [v2:172.21.15.89:6810/50395527,v1:172.21.15.89:6814/50395527] -- osd_op(unknown.0.0:79 16.1f 16:f8000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x56290e31e800 con 0x56290e300000 2024-03-31T15:14:53.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.593+0000 7fbce7200640 1 -- 172.21.15.16:0/2805451625 <== osd.5 v2:172.21.15.89:6810/50395527 11 ==== osd_op_reply(79 [pgnls start_epoch 113 out=101b] v113'9 uv8 ondisk = 1) ==== 144+0+101 (crc 0 0 0) 0x56290e07c000 con 0x56290e300000 2024-03-31T15:14:53.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.593+0000 7fbce9ae18c0 10 datascan.scan_links: step 2: handling object 601.00000000 2024-03-31T15:14:53.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.593+0000 7fbce9ae18c0 1 -- 172.21.15.16:0/2805451625 --> [v2:172.21.15.16:6819/3018526646,v1:172.21.15.16:6822/3018526646] -- osd_op(unknown.0.0:80 16.10 16:09401a81:::601.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e113) -- 0x56290e301800 con 0x56290e1ed000 2024-03-31T15:14:53.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.593+0000 7fbce8600640 1 -- 172.21.15.16:0/2805451625 <== osd.2 v2:172.21.15.16:6819/3018526646 23 ==== osd_op_reply(80 601.00000000 [omap-get-vals out=5b] v0'0 uv13 ondisk = 0) ==== 156+0+5 (crc 0 0 0) 0x56290e07c500 con 0x56290e1ed000 2024-03-31T15:14:53.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.597+0000 7fbce9ae18c0 10 datascan.scan_links: step 2: handling object 602.00000000 2024-03-31T15:14:53.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.597+0000 7fbce9ae18c0 1 -- 172.21.15.16:0/2805451625 --> [v2:172.21.15.16:6819/3018526646,v1:172.21.15.16:6822/3018526646] -- osd_op(unknown.0.0:81 16.10 16:0bd6d154:::602.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e113) -- 0x56290e1edc00 con 0x56290e1ed000 2024-03-31T15:14:53.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.597+0000 7fbce8600640 1 -- 172.21.15.16:0/2805451625 <== osd.2 v2:172.21.15.16:6819/3018526646 24 ==== osd_op_reply(81 602.00000000 [omap-get-vals out=5b] v0'0 uv15 ondisk = 0) ==== 156+0+5 (crc 0 0 0) 0x56290e07c500 con 0x56290e1ed000 2024-03-31T15:14:53.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.597+0000 7fbce9ae18c0 10 datascan.scan_links: step 2: handling object 10000000000.00000000 2024-03-31T15:14:53.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.597+0000 7fbce9ae18c0 1 -- 172.21.15.16:0/2805451625 --> [v2:172.21.15.16:6819/3018526646,v1:172.21.15.16:6822/3018526646] -- osd_op(unknown.0.0:82 16.10 16:0cf6ad0f:::10000000000.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e113) -- 0x56290e1ec800 con 0x56290e1ed000 2024-03-31T15:14:53.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.597+0000 7fbce8600640 1 -- 172.21.15.16:0/2805451625 <== osd.2 v2:172.21.15.16:6819/3018526646 25 ==== osd_op_reply(82 10000000000.00000000 [omap-get-vals out=1047b] v0'0 uv12 ondisk = 0) ==== 164+0+1047 (crc 0 0 0) 0x56290e07c500 con 0x56290e1ed000 2024-03-31T15:14:53.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.597+0000 7fbce9ae18c0 10 datascan.scan_links: step 2: handling object 600.00000000 2024-03-31T15:14:53.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.597+0000 7fbce9ae18c0 1 -- 172.21.15.16:0/2805451625 --> [v2:172.21.15.16:6819/3018526646,v1:172.21.15.16:6822/3018526646] -- osd_op(unknown.0.0:83 16.10 16:0d82a743:::600.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e113) -- 0x56290e1ed400 con 0x56290e1ed000 2024-03-31T15:14:53.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.597+0000 7fbce8600640 1 -- 172.21.15.16:0/2805451625 <== osd.2 v2:172.21.15.16:6819/3018526646 26 ==== osd_op_reply(83 600.00000000 [omap-get-vals out=5b] v0'0 uv14 ondisk = 0) ==== 156+0+5 (crc 0 0 0) 0x56290e07c500 con 0x56290e1ed000 2024-03-31T15:14:53.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.597+0000 7fbce9ae18c0 10 datascan.scan_links: step 2: handling object 603.00000000 2024-03-31T15:14:53.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.597+0000 7fbce9ae18c0 1 -- 172.21.15.16:0/2805451625 --> [v2:172.21.15.16:6819/3018526646,v1:172.21.15.16:6822/3018526646] -- osd_op(unknown.0.0:84 16.10 16:0d89b25e:::603.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e113) -- 0x56290e1ed800 con 0x56290e1ed000 2024-03-31T15:14:53.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.597+0000 7fbce8600640 1 -- 172.21.15.16:0/2805451625 <== osd.2 v2:172.21.15.16:6819/3018526646 27 ==== osd_op_reply(84 603.00000000 [omap-get-vals out=5b] v0'0 uv16 ondisk = 0) ==== 156+0+5 (crc 0 0 0) 0x56290e07c500 con 0x56290e1ed000 2024-03-31T15:14:53.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.597+0000 7fbce9ae18c0 10 datascan.scan_links: step 2: handling object 1.00000000.inode 2024-03-31T15:14:53.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.597+0000 7fbce9ae18c0 10 datascan.scan_links: Not a dirfrag: '1.00000000.inode' 2024-03-31T15:14:53.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.597+0000 7fbce9ae18c0 10 datascan.scan_links: step 2: handling object 200.00000000 2024-03-31T15:14:53.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.597+0000 7fbce9ae18c0 10 datascan.scan_links: Not a dirfrag (invalid ino): '200.00000000' 2024-03-31T15:14:53.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.597+0000 7fbce9ae18c0 10 datascan.scan_links: step 2: handling object 200.00000001 2024-03-31T15:14:53.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.597+0000 7fbce9ae18c0 10 datascan.scan_links: Not a dirfrag (invalid ino): '200.00000001' 2024-03-31T15:14:53.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.597+0000 7fbce9ae18c0 10 datascan.scan_links: step 2: handling object 606.00000000 2024-03-31T15:14:53.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.597+0000 7fbce9ae18c0 1 -- 172.21.15.16:0/2805451625 --> [v2:172.21.15.16:6800/932712244,v1:172.21.15.16:6801/932712244] -- osd_op(unknown.0.0:85 16.14 16:2f55791f:::606.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e113) -- 0x56290e1edc00 con 0x56290ee2dc00 2024-03-31T15:14:53.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.597+0000 7fbce7200640 1 -- 172.21.15.16:0/2805451625 <== osd.0 v2:172.21.15.16:6800/932712244 17 ==== osd_op_reply(85 606.00000000 [omap-get-vals out=5b] v0'0 uv38 ondisk = 0) ==== 156+0+5 (crc 0 0 0) 0x56290e07c000 con 0x56290ee2dc00 2024-03-31T15:14:53.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.597+0000 7fbce9ae18c0 10 datascan.scan_links: step 2: handling object 607.00000000 2024-03-31T15:14:53.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.597+0000 7fbce9ae18c0 1 -- 172.21.15.16:0/2805451625 --> [v2:172.21.15.16:6819/3018526646,v1:172.21.15.16:6822/3018526646] -- osd_op(unknown.0.0:86 16.1c 16:3ed09add:::607.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e113) -- 0x56290e301800 con 0x56290e1ed000 2024-03-31T15:14:53.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.597+0000 7fbce8600640 1 -- 172.21.15.16:0/2805451625 <== osd.2 v2:172.21.15.16:6819/3018526646 28 ==== osd_op_reply(86 607.00000000 [omap-get-vals out=5b] v0'0 uv3 ondisk = 0) ==== 156+0+5 (crc 0 0 0) 0x56290e07c500 con 0x56290e1ed000 2024-03-31T15:14:53.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.597+0000 7fbce9ae18c0 10 datascan.scan_links: step 2: handling object 10000000003.00000000 2024-03-31T15:14:53.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.597+0000 7fbce9ae18c0 1 -- 172.21.15.16:0/2805451625 --> [v2:172.21.15.89:6809/3470471752,v1:172.21.15.89:6812/3470471752] -- osd_op(unknown.0.0:87 16.12 16:4def72d4:::10000000003.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e113) -- 0x56290e31e400 con 0x56290eced400 2024-03-31T15:14:53.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.597+0000 7fbce7c00640 1 -- 172.21.15.16:0/2805451625 <== osd.6 v2:172.21.15.89:6809/3470471752 8 ==== osd_op_reply(87 10000000003.00000000 [omap-get-vals out=523b] v0'0 uv3 ondisk = 0) ==== 164+0+523 (crc 0 0 0) 0x56290e07c780 con 0x56290eced400 2024-03-31T15:14:53.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.597+0000 7fbce9ae18c0 10 datascan.scan_links: step 2: handling object mds0_openfiles.0 2024-03-31T15:14:53.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.597+0000 7fbce9ae18c0 10 datascan.scan_links: Not a dirfrag: 'mds0_openfiles.0' 2024-03-31T15:14:53.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.597+0000 7fbce9ae18c0 10 datascan.scan_links: step 2: handling object 608.00000000 2024-03-31T15:14:53.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.597+0000 7fbce9ae18c0 1 -- 172.21.15.16:0/2805451625 --> [v2:172.21.15.16:6819/3018526646,v1:172.21.15.16:6822/3018526646] -- osd_op(unknown.0.0:88 16.e 16:717a0223:::608.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e113) -- 0x56290e31ec00 con 0x56290e1ed000 2024-03-31T15:14:53.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.597+0000 7fbce8600640 1 -- 172.21.15.16:0/2805451625 <== osd.2 v2:172.21.15.16:6819/3018526646 29 ==== osd_op_reply(88 608.00000000 [omap-get-vals out=5b] v0'0 uv3 ondisk = 0) ==== 156+0+5 (crc 0 0 0) 0x56290e07c500 con 0x56290e1ed000 2024-03-31T15:14:53.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.597+0000 7fbce9ae18c0 10 datascan.scan_links: step 2: handling object 604.00000000 2024-03-31T15:14:53.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.597+0000 7fbce9ae18c0 1 -- 172.21.15.16:0/2805451625 --> [v2:172.21.15.16:6800/932712244,v1:172.21.15.16:6801/932712244] -- osd_op(unknown.0.0:89 16.15 16:a93a17c2:::604.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e113) -- 0x56290e31f400 con 0x56290ee2dc00 2024-03-31T15:14:53.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.597+0000 7fbce7200640 1 -- 172.21.15.16:0/2805451625 <== osd.0 v2:172.21.15.16:6800/932712244 18 ==== osd_op_reply(89 604.00000000 [omap-get-vals out=5b] v0'0 uv4 ondisk = 0) ==== 156+0+5 (crc 0 0 0) 0x56290e07c000 con 0x56290ee2dc00 2024-03-31T15:14:53.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.597+0000 7fbce9ae18c0 10 datascan.scan_links: step 2: handling object 500.00000000 2024-03-31T15:14:53.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.597+0000 7fbce9ae18c0 10 datascan.scan_links: Not a dirfrag (invalid ino): '500.00000000' 2024-03-31T15:14:53.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.597+0000 7fbce9ae18c0 10 datascan.scan_links: step 2: handling object mds_snaptable 2024-03-31T15:14:53.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.597+0000 7fbce9ae18c0 10 datascan.scan_links: Not a dirfrag: 'mds_snaptable' 2024-03-31T15:14:53.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.597+0000 7fbce9ae18c0 10 datascan.scan_links: step 2: handling object 605.00000000 2024-03-31T15:14:53.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.597+0000 7fbce9ae18c0 1 -- 172.21.15.16:0/2805451625 --> [v2:172.21.15.16:6819/3018526646,v1:172.21.15.16:6822/3018526646] -- osd_op(unknown.0.0:90 16.1d 16:b871830b:::605.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e113) -- 0x56290e31fc00 con 0x56290e1ed000 2024-03-31T15:14:53.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.597+0000 7fbce8600640 1 -- 172.21.15.16:0/2805451625 <== osd.2 v2:172.21.15.16:6819/3018526646 30 ==== osd_op_reply(90 605.00000000 [omap-get-vals out=5b] v0'0 uv3 ondisk = 0) ==== 156+0+5 (crc 0 0 0) 0x56290e07c500 con 0x56290e1ed000 2024-03-31T15:14:53.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.597+0000 7fbce9ae18c0 10 datascan.scan_links: step 2: handling object mds0_inotable 2024-03-31T15:14:53.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.597+0000 7fbce9ae18c0 10 datascan.scan_links: Not a dirfrag: 'mds0_inotable' 2024-03-31T15:14:53.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.597+0000 7fbce9ae18c0 10 datascan.scan_links: step 2: handling object 100.00000000 2024-03-31T15:14:53.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.597+0000 7fbce9ae18c0 1 -- 172.21.15.16:0/2805451625 --> [v2:172.21.15.16:6808/1398737533,v1:172.21.15.16:6810/1398737533] -- osd_op(unknown.0.0:91 16.13 16:cd5a64a3:::100.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e113) -- 0x56290e31e000 con 0x56290e301400 2024-03-31T15:14:53.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.597+0000 7fbce8600640 1 -- 172.21.15.16:0/2805451625 <== osd.1 v2:172.21.15.16:6808/1398737533 8 ==== osd_op_reply(91 100.00000000 [omap-get-vals out=5195b] v0'0 uv5 ondisk = 0) ==== 156+0+5195 (crc 0 0 0) 0x56290e07c500 con 0x56290e301400 2024-03-31T15:14:53.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.597+0000 7fbce9ae18c0 10 datascan.scan_links: step 2: handling object mds0_sessionmap 2024-03-31T15:14:53.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.597+0000 7fbce9ae18c0 10 datascan.scan_links: Not a dirfrag: 'mds0_sessionmap' 2024-03-31T15:14:53.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.597+0000 7fbce9ae18c0 10 datascan.scan_links: step 2: handling object 609.00000000 2024-03-31T15:14:53.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.597+0000 7fbce9ae18c0 1 -- 172.21.15.16:0/2805451625 --> [v2:172.21.15.16:6809/3360352974,v1:172.21.15.16:6812/3360352974] -- osd_op(unknown.0.0:92 16.7 16:e0b41d06:::609.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e113) -- 0x56290e31f000 con 0x56290ee2c800 2024-03-31T15:14:53.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.601+0000 7fbce7c00640 1 -- 172.21.15.16:0/2805451625 <== osd.3 v2:172.21.15.16:6809/3360352974 11 ==== osd_op_reply(92 609.00000000 [omap-get-vals out=5b] v0'0 uv7 ondisk = 0) ==== 156+0+5 (crc 0 0 0) 0x56290e07c780 con 0x56290ee2c800 2024-03-31T15:14:53.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.601+0000 7fbce9ae18c0 10 datascan.scan_links: step 2: handling object 10000000001.00000000 2024-03-31T15:14:53.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.601+0000 7fbce9ae18c0 1 -- 172.21.15.16:0/2805451625 --> [v2:172.21.15.16:6809/3360352974,v1:172.21.15.16:6812/3360352974] -- osd_op(unknown.0.0:93 16.7 16:e2e07930:::10000000001.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e113) -- 0x56290e31f800 con 0x56290ee2c800 2024-03-31T15:14:53.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.601+0000 7fbce7c00640 1 -- 172.21.15.16:0/2805451625 <== osd.3 v2:172.21.15.16:6809/3360352974 12 ==== osd_op_reply(93 10000000001.00000000 [omap-get-vals out=1545b] v0'0 uv9 ondisk = 0) ==== 164+0+1545 (crc 0 0 0) 0x56290e07c780 con 0x56290ee2c800 2024-03-31T15:14:53.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.601+0000 7fbce9ae18c0 20 datascan.scan_links: injected ino detected 2024-03-31T15:14:53.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.601+0000 7fbce9ae18c0 20 datascan.scan_links: adding 0x100000001f7 for future processing to fix dnfirst 2024-03-31T15:14:53.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.601+0000 7fbce9ae18c0 10 datascan.scan_links: step 2: handling object 400.00000000 2024-03-31T15:14:53.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.601+0000 7fbce9ae18c0 10 datascan.scan_links: Not a dirfrag (invalid ino): '400.00000000' 2024-03-31T15:14:53.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.601+0000 7fbce9ae18c0 10 datascan.scan_links: step 2: handling object 100.00000000.inode 2024-03-31T15:14:53.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.601+0000 7fbce9ae18c0 10 datascan.scan_links: Not a dirfrag: '100.00000000.inode' 2024-03-31T15:14:53.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.601+0000 7fbce9ae18c0 10 datascan.scan_links: step 2: handling object 1.00000000 2024-03-31T15:14:53.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.601+0000 7fbce9ae18c0 1 -- 172.21.15.16:0/2805451625 --> [v2:172.21.15.89:6810/50395527,v1:172.21.15.89:6814/50395527] -- osd_op(unknown.0.0:94 16.1f 16:ff5b34d6:::1.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e113) -- 0x56290e31e800 con 0x56290e300000 2024-03-31T15:14:53.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.601+0000 7fbce7200640 1 -- 172.21.15.16:0/2805451625 <== osd.5 v2:172.21.15.89:6810/50395527 12 ==== osd_op_reply(94 1.00000000 [omap-get-vals out=524b] v0'0 uv8 ondisk = 0) ==== 154+0+524 (crc 0 0 0) 0x56290e07c000 con 0x56290e300000 2024-03-31T15:14:53.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.601+0000 7fbce9ae18c0 10 datascan.scan_links: processing 0 dup_primaries, 0 remote_links 2024-03-31T15:14:53.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.601+0000 7fbce9ae18c0 10 datascan.scan_links: removing dup dentries from 0 objects 2024-03-31T15:14:53.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.601+0000 7fbce9ae18c0 10 datascan.scan_links: processing 0 bad_nlink_inos 2024-03-31T15:14:53.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.601+0000 7fbce9ae18c0 10 datascan.scan_links: processing 1 injected_inos 2024-03-31T15:14:53.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.601+0000 7fbce9ae18c0 10 datascan.scan_links: handling injected_ino 0x100000001f7 2024-03-31T15:14:53.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.601+0000 7fbce9ae18c0 20 datascan.scan_links: fixing linkage (dnfirst) of 0x10000000001:bravo 2024-03-31T15:14:53.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.601+0000 7fbce9ae18c0 1 -- 172.21.15.16:0/2805451625 --> [v2:172.21.15.16:6809/3360352974,v1:172.21.15.16:6812/3360352974] -- osd_op(unknown.0.0:95 16.7 16:e2e07930:::10000000001.00000000:head [omap-get-vals-by-keys in=18b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e113) -- 0x56290e31f400 con 0x56290ee2c800 2024-03-31T15:14:53.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.601+0000 7fbce7c00640 1 -- 172.21.15.16:0/2805451625 <== osd.3 v2:172.21.15.16:6809/3360352974 13 ==== osd_op_reply(95 10000000001.00000000 [omap-get-vals-by-keys out=506b] v0'0 uv9 ondisk = 0) ==== 164+0+506 (crc 0 0 0) 0x56290e07c780 con 0x56290ee2c800 2024-03-31T15:14:53.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.601+0000 7fbce9ae18c0 20 datascan.read_dentry: oid=10000000001.00000000 dname=bravo frag=*, r=0 2024-03-31T15:14:53.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.601+0000 7fbce9ae18c0 20 datascan.scan_links: first is now 2 2024-03-31T15:14:53.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.601+0000 7fbce9ae18c0 1 -- 172.21.15.16:0/2805451625 --> [v2:172.21.15.16:6809/3360352974,v1:172.21.15.16:6812/3360352974] -- osd_op(unknown.0.0:96 16.7 16:e2e07930:::10000000001.00000000:head [omap-set-vals in=506b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x56290e1edc00 con 0x56290ee2c800 2024-03-31T15:14:53.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.601+0000 7fbce7c00640 1 -- 172.21.15.16:0/2805451625 <== osd.3 v2:172.21.15.16:6809/3360352974 14 ==== osd_op_reply(96 10000000001.00000000 [omap-set-vals] v113'10 uv10 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x56290e07c780 con 0x56290ee2c800 2024-03-31T15:14:53.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.601+0000 7fbce9ae18c0 20 datascan.inject_linkage: Injected dentry 0x0x10000000001/bravo pointing to 0x0x100000001f7 2024-03-31T15:14:53.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.601+0000 7fbce9ae18c0 10 datascan.scan_links: updating inotable 2024-03-31T15:14:53.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.601+0000 7fbce9ae18c0 1 -- 172.21.15.16:0/2805451625 --> [v2:172.21.15.16:6808/1398737533,v1:172.21.15.16:6810/1398737533] -- osd_op(unknown.0.0:97 16.13 16:c91d4a1d:::mds0_inotable:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e113) -- 0x56290e301c00 con 0x56290e301400 2024-03-31T15:14:53.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.601+0000 7fbce8600640 1 -- 172.21.15.16:0/2805451625 <== osd.1 v2:172.21.15.16:6808/1398737533 9 ==== osd_op_reply(97 mds0_inotable [read 0~50 out=50b] v0'0 uv4 ondisk = 0) ==== 157+0+50 (crc 0 0 0) 0x56290e07c500 con 0x56290e301400 2024-03-31T15:14:53.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.601+0000 7fbce9ae18c0 10 mds.0.inotable: skip_inos was [0x10000000005~0x1f2,0x100000001f9~0xfffffffe07] 2024-03-31T15:14:53.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.601+0000 7fbce9ae18c0 10 mds.0.inotable: skip_inos now [0x100000001f9~0xfffffffe07] 2024-03-31T15:14:53.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.601+0000 7fbce9ae18c0 1 -- 172.21.15.16:0/2805451625 --> [v2:172.21.15.16:6808/1398737533,v1:172.21.15.16:6810/1398737533] -- osd_op(unknown.0.0:98 16.13 16:c91d4a1d:::mds0_inotable:head [writefull 0~34 in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x56290e300800 con 0x56290e301400 2024-03-31T15:14:53.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.605+0000 7fbce8600640 1 -- 172.21.15.16:0/2805451625 <== osd.1 v2:172.21.15.16:6808/1398737533 10 ==== osd_op_reply(98 mds0_inotable [writefull 0~34] v113'6 uv6 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x56290e07c500 con 0x56290e301400 2024-03-31T15:14:53.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.605+0000 7fbce9ae18c0 10 datascan.scan_links: updating snaptable 2024-03-31T15:14:53.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.605+0000 7fbce9ae18c0 1 -- 172.21.15.16:0/2805451625 --> [v2:172.21.15.89:6809/3470471752,v1:172.21.15.89:6812/3470471752] -- osd_op(unknown.0.0:99 16.d 16:b50e409b:::mds_snaptable:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e113) -- 0x56290e300c00 con 0x56290eced400 2024-03-31T15:14:53.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.605+0000 7fbce7c00640 1 -- 172.21.15.16:0/2805451625 <== osd.6 v2:172.21.15.89:6809/3470471752 9 ==== osd_op_reply(99 mds_snaptable [read 0~70 out=70b] v0'0 uv1 ondisk = 0) ==== 157+0+70 (crc 0 0 0) 0x56290e07c780 con 0x56290eced400 2024-03-31T15:14:53.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.605+0000 7fbce9ae18c0 1 -- 172.21.15.16:0/2805451625 >> [v2:172.21.15.16:6800/932712244,v1:172.21.15.16:6801/932712244] conn(0x56290ee2dc00 msgr2=0x56290ecce680 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:53.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.605+0000 7fbce9ae18c0 1 --2- 172.21.15.16:0/2805451625 >> [v2:172.21.15.16:6800/932712244,v1:172.21.15.16:6801/932712244] conn(0x56290ee2dc00 0x56290ecce680 crc :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:53.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.605+0000 7fbce9ae18c0 1 -- 172.21.15.16:0/2805451625 >> [v2:172.21.15.16:6808/1398737533,v1:172.21.15.16:6810/1398737533] conn(0x56290e301400 msgr2=0x56290e2fa100 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:53.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.605+0000 7fbce9ae18c0 1 --2- 172.21.15.16:0/2805451625 >> [v2:172.21.15.16:6808/1398737533,v1:172.21.15.16:6810/1398737533] conn(0x56290e301400 0x56290e2fa100 crc :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:53.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.605+0000 7fbce9ae18c0 1 -- 172.21.15.16:0/2805451625 >> [v2:172.21.15.16:6819/3018526646,v1:172.21.15.16:6822/3018526646] conn(0x56290e1ed000 msgr2=0x56290e2f8000 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:53.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.605+0000 7fbce9ae18c0 1 --2- 172.21.15.16:0/2805451625 >> [v2:172.21.15.16:6819/3018526646,v1:172.21.15.16:6822/3018526646] conn(0x56290e1ed000 0x56290e2f8000 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:53.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.605+0000 7fbce9ae18c0 1 -- 172.21.15.16:0/2805451625 >> [v2:172.21.15.16:6809/3360352974,v1:172.21.15.16:6812/3360352974] conn(0x56290ee2c800 msgr2=0x56290e2f8580 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:53.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.605+0000 7fbce9ae18c0 1 --2- 172.21.15.16:0/2805451625 >> [v2:172.21.15.16:6809/3360352974,v1:172.21.15.16:6812/3360352974] conn(0x56290ee2c800 0x56290e2f8580 crc :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:53.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.605+0000 7fbce9ae18c0 1 -- 172.21.15.16:0/2805451625 >> [v2:172.21.15.89:6800/1856844856,v1:172.21.15.89:6801/1856844856] conn(0x56290e300400 msgr2=0x56290e2f9080 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:53.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.605+0000 7fbce9ae18c0 1 --2- 172.21.15.16:0/2805451625 >> [v2:172.21.15.89:6800/1856844856,v1:172.21.15.89:6801/1856844856] conn(0x56290e300400 0x56290e2f9080 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:53.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.605+0000 7fbce7c00640 1 -- 172.21.15.16:0/2805451625 reap_dead start 2024-03-31T15:14:53.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.605+0000 7fbce9ae18c0 1 -- 172.21.15.16:0/2805451625 >> [v2:172.21.15.89:6810/50395527,v1:172.21.15.89:6814/50395527] conn(0x56290e300000 msgr2=0x56290e2f8b00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:53.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.605+0000 7fbce9ae18c0 1 --2- 172.21.15.16:0/2805451625 >> [v2:172.21.15.89:6810/50395527,v1:172.21.15.89:6814/50395527] conn(0x56290e300000 0x56290e2f8b00 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:53.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.605+0000 7fbce7c00640 1 -- 172.21.15.16:0/2805451625 reap_dead start 2024-03-31T15:14:53.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.605+0000 7fbce9ae18c0 1 -- 172.21.15.16:0/2805451625 >> [v2:172.21.15.89:6809/3470471752,v1:172.21.15.89:6812/3470471752] conn(0x56290eced400 msgr2=0x56290e2f9600 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:53.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.605+0000 7fbce9ae18c0 1 --2- 172.21.15.16:0/2805451625 >> [v2:172.21.15.89:6809/3470471752,v1:172.21.15.89:6812/3470471752] conn(0x56290eced400 0x56290e2f9600 crc :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:53.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.605+0000 7fbce7c00640 1 -- 172.21.15.16:0/2805451625 reap_dead start 2024-03-31T15:14:53.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.605+0000 7fbce9ae18c0 1 -- 172.21.15.16:0/2805451625 >> [v2:172.21.15.89:6808/1796778630,v1:172.21.15.89:6811/1796778630] conn(0x56290e301000 msgr2=0x56290e2f9b80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:53.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.605+0000 7fbce9ae18c0 1 --2- 172.21.15.16:0/2805451625 >> [v2:172.21.15.89:6808/1796778630,v1:172.21.15.89:6811/1796778630] conn(0x56290e301000 0x56290e2f9b80 crc :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:53.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.605+0000 7fbce9ae18c0 1 -- 172.21.15.16:0/2805451625 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x56290e1ecc00 msgr2=0x56290ecce100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:53.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.605+0000 7fbce9ae18c0 1 --2- 172.21.15.16:0/2805451625 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x56290e1ecc00 0x56290ecce100 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x56290ee3cc90 tx=0x56290dfcee40 comp rx=0 tx=0).stop 2024-03-31T15:14:53.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.605+0000 7fbce9ae18c0 1 -- 172.21.15.16:0/2805451625 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56290ee2d000 msgr2=0x56290eccd080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:53.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.605+0000 7fbce9ae18c0 1 --2- 172.21.15.16:0/2805451625 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56290ee2d000 0x56290eccd080 secure :-1 s=READY pgs=1821 cs=0 l=1 rev1=1 crypto rx=0x56290ed62ff0 tx=0x56290ecd8000 comp rx=0 tx=0).stop 2024-03-31T15:14:53.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.605+0000 7fbce9ae18c0 1 -- 172.21.15.16:0/2805451625 shutdown_connections 2024-03-31T15:14:53.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.605+0000 7fbce9ae18c0 1 --2- 172.21.15.16:0/2805451625 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x56290e1ecc00 0x56290ecce100 unknown :-1 s=CLOSED pgs=765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:53.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.605+0000 7fbce9ae18c0 1 --2- 172.21.15.16:0/2805451625 >> [v2:172.21.15.89:6808/1796778630,v1:172.21.15.89:6811/1796778630] conn(0x56290e301000 0x56290e2f9b80 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:53.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.605+0000 7fbce9ae18c0 1 --2- 172.21.15.16:0/2805451625 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56290ee2d000 0x56290eccd080 unknown :-1 s=CLOSED pgs=1821 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:53.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.605+0000 7fbce9ae18c0 1 -- 172.21.15.16:0/2805451625 >> 172.21.15.16:0/2805451625 conn(0x56290df08c00 msgr2=0x56290df37800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:14:53.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.605+0000 7fbce9ae18c0 1 -- 172.21.15.16:0/2805451625 shutdown_connections 2024-03-31T15:14:53.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.605+0000 7fbce9ae18c0 1 -- 172.21.15.16:0/2805451625 wait complete. 2024-03-31T15:14:53.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.605+0000 7fbce9ae18c0 1 -- 172.21.15.16:0/4079968052 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56290df09400 msgr2=0x56290eccc580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:53.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.605+0000 7fbce9ae18c0 1 --2- 172.21.15.16:0/4079968052 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56290df09400 0x56290eccc580 secure :-1 s=READY pgs=1819 cs=0 l=1 rev1=1 crypto rx=0x56290ed638c0 tx=0x56290ec910e0 comp rx=0 tx=0).stop 2024-03-31T15:14:53.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.605+0000 7fbce9ae18c0 1 -- 172.21.15.16:0/4079968052 shutdown_connections 2024-03-31T15:14:53.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.605+0000 7fbce9ae18c0 1 --2- 172.21.15.16:0/4079968052 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56290df09000 0x56290ecccb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:53.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.605+0000 7fbce9ae18c0 1 --2- 172.21.15.16:0/4079968052 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56290df09400 0x56290eccc580 unknown :-1 s=CLOSED pgs=1819 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:53.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.605+0000 7fbce9ae18c0 1 --2- 172.21.15.16:0/4079968052 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x56290df09800 0x56290eccc000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:53.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.605+0000 7fbce9ae18c0 1 -- 172.21.15.16:0/4079968052 >> 172.21.15.16:0/4079968052 conn(0x56290df09c00 msgr2=0x56290df37000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:14:53.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.605+0000 7fbce9ae18c0 1 -- 172.21.15.16:0/4079968052 shutdown_connections 2024-03-31T15:14:53.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:53.605+0000 7fbce9ae18c0 1 -- 172.21.15.16:0/4079968052 wait complete. 2024-03-31T15:14:53.607 DEBUG:tasks.cephfs.filesystem:Ran ['cephfs-data-scan', '--debug-mds=20', '--debug-ms=1', '--debug-objecter=1', 'scan_links'] in time 0:00:00.148393, result: 2024-03-31T15:14:53.608 DEBUG:teuthology.orchestra.run.smithi016:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-31T15:14:53.651 DEBUG:teuthology.orchestra.run.smithi089:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-31T15:14:53.691 INFO:teuthology.orchestra.run.smithi016.stdout:[global] 2024-03-31T15:14:53.691 INFO:teuthology.orchestra.run.smithi016.stdout: chdir = "" 2024-03-31T15:14:53.691 INFO:teuthology.orchestra.run.smithi016.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-31T15:14:53.691 INFO:teuthology.orchestra.run.smithi016.stdout: auth supported = cephx 2024-03-31T15:14:53.692 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:14:53.692 INFO:teuthology.orchestra.run.smithi016.stdout: filestore xattr use omap = true 2024-03-31T15:14:53.692 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:14:53.692 INFO:teuthology.orchestra.run.smithi016.stdout: mon clock drift allowed = 1.000 2024-03-31T15:14:53.692 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:14:53.692 INFO:teuthology.orchestra.run.smithi016.stdout: osd crush chooseleaf type = 0 2024-03-31T15:14:53.692 INFO:teuthology.orchestra.run.smithi016.stdout: auth debug = true 2024-03-31T15:14:53.692 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:14:53.692 INFO:teuthology.orchestra.run.smithi016.stdout: ms die on old message = true 2024-03-31T15:14:53.692 INFO:teuthology.orchestra.run.smithi016.stdout: ms die on bug = true 2024-03-31T15:14:53.692 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:14:53.692 INFO:teuthology.orchestra.run.smithi016.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-31T15:14:53.692 INFO:teuthology.orchestra.run.smithi016.stdout: mon pg warn max object skew = 0 2024-03-31T15:14:53.693 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:14:53.693 INFO:teuthology.orchestra.run.smithi016.stdout: # disable pg_autoscaler by default for new pools 2024-03-31T15:14:53.693 INFO:teuthology.orchestra.run.smithi016.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-31T15:14:53.693 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:14:53.693 INFO:teuthology.orchestra.run.smithi016.stdout: osd pool default size = 2 2024-03-31T15:14:53.693 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:14:53.693 INFO:teuthology.orchestra.run.smithi016.stdout: mon osd allow primary affinity = true 2024-03-31T15:14:53.693 INFO:teuthology.orchestra.run.smithi016.stdout: mon osd allow pg remap = true 2024-03-31T15:14:53.693 INFO:teuthology.orchestra.run.smithi016.stdout: mon warn on legacy crush tunables = false 2024-03-31T15:14:53.693 INFO:teuthology.orchestra.run.smithi016.stdout: mon warn on crush straw calc version zero = false 2024-03-31T15:14:53.693 INFO:teuthology.orchestra.run.smithi016.stdout: mon warn on no sortbitwise = false 2024-03-31T15:14:53.693 INFO:teuthology.orchestra.run.smithi016.stdout: mon warn on osd down out interval zero = false 2024-03-31T15:14:53.693 INFO:teuthology.orchestra.run.smithi016.stdout: mon warn on too few osds = false 2024-03-31T15:14:53.693 INFO:teuthology.orchestra.run.smithi016.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-31T15:14:53.694 INFO:teuthology.orchestra.run.smithi016.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-31T15:14:53.694 INFO:teuthology.orchestra.run.smithi016.stdout: mon_allow_pool_size_one = true 2024-03-31T15:14:53.694 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:14:53.694 INFO:teuthology.orchestra.run.smithi016.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-31T15:14:53.694 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:14:53.694 INFO:teuthology.orchestra.run.smithi016.stdout: osd default data pool replay window = 5 2024-03-31T15:14:53.694 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:14:53.694 INFO:teuthology.orchestra.run.smithi016.stdout: mon allow pool delete = true 2024-03-31T15:14:53.694 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:14:53.694 INFO:teuthology.orchestra.run.smithi016.stdout: mon cluster log file level = debug 2024-03-31T15:14:53.694 INFO:teuthology.orchestra.run.smithi016.stdout: debug asserts on shutdown = true 2024-03-31T15:14:53.694 INFO:teuthology.orchestra.run.smithi016.stdout: mon health detail to clog = false 2024-03-31T15:14:53.694 INFO:teuthology.orchestra.run.smithi016.stdout: mon host = "172.21.15.16,172.21.15.89,[v2:172.21.15.89:3301,v1:172.21.15.89:6790]" 2024-03-31T15:14:53.694 INFO:teuthology.orchestra.run.smithi016.stdout: fsid = 286eb5da-fb47-4550-b290-d9519c13b9a5 2024-03-31T15:14:53.695 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:14:53.695 INFO:teuthology.orchestra.run.smithi016.stdout:[osd] 2024-03-31T15:14:53.695 INFO:teuthology.orchestra.run.smithi016.stdout: osd journal size = 100 2024-03-31T15:14:53.695 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:14:53.695 INFO:teuthology.orchestra.run.smithi016.stdout: osd scrub load threshold = 5.0 2024-03-31T15:14:53.695 INFO:teuthology.orchestra.run.smithi016.stdout: osd scrub max interval = 600 2024-03-31T15:14:53.695 INFO:teuthology.orchestra.run.smithi016.stdout: osd mclock profile = high_recovery_ops 2024-03-31T15:14:53.695 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:14:53.695 INFO:teuthology.orchestra.run.smithi016.stdout: osd recover clone overlap = true 2024-03-31T15:14:53.695 INFO:teuthology.orchestra.run.smithi016.stdout: osd recovery max chunk = 1048576 2024-03-31T15:14:53.695 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:14:53.695 INFO:teuthology.orchestra.run.smithi016.stdout: osd debug shutdown = true 2024-03-31T15:14:53.695 INFO:teuthology.orchestra.run.smithi016.stdout: osd debug op order = true 2024-03-31T15:14:53.695 INFO:teuthology.orchestra.run.smithi016.stdout: osd debug verify stray on activate = true 2024-03-31T15:14:53.696 INFO:teuthology.orchestra.run.smithi016.stdout: osd debug trim objects = true 2024-03-31T15:14:53.696 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:14:53.696 INFO:teuthology.orchestra.run.smithi016.stdout: osd open classes on start = true 2024-03-31T15:14:53.696 INFO:teuthology.orchestra.run.smithi016.stdout: osd debug pg log writeout = true 2024-03-31T15:14:53.696 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:14:53.696 INFO:teuthology.orchestra.run.smithi016.stdout: osd deep scrub update digest min age = 30 2024-03-31T15:14:53.696 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:14:53.696 INFO:teuthology.orchestra.run.smithi016.stdout: osd map max advance = 10 2024-03-31T15:14:53.696 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:14:53.696 INFO:teuthology.orchestra.run.smithi016.stdout: journal zero on create = true 2024-03-31T15:14:53.696 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:14:53.696 INFO:teuthology.orchestra.run.smithi016.stdout: filestore ondisk finisher threads = 3 2024-03-31T15:14:53.696 INFO:teuthology.orchestra.run.smithi016.stdout: filestore apply finisher threads = 3 2024-03-31T15:14:53.696 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:14:53.697 INFO:teuthology.orchestra.run.smithi016.stdout: bdev debug aio = true 2024-03-31T15:14:53.697 INFO:teuthology.orchestra.run.smithi016.stdout: osd debug misdirected ops = true 2024-03-31T15:14:53.697 INFO:teuthology.orchestra.run.smithi016.stdout: bdev async discard = True 2024-03-31T15:14:53.697 INFO:teuthology.orchestra.run.smithi016.stdout: bdev enable discard = True 2024-03-31T15:14:53.697 INFO:teuthology.orchestra.run.smithi016.stdout: bluestore allocator = bitmap 2024-03-31T15:14:53.697 INFO:teuthology.orchestra.run.smithi016.stdout: bluestore block size = 96636764160 2024-03-31T15:14:53.697 INFO:teuthology.orchestra.run.smithi016.stdout: bluestore fsck on mount = True 2024-03-31T15:14:53.697 INFO:teuthology.orchestra.run.smithi016.stdout: debug bluefs = 1/20 2024-03-31T15:14:53.697 INFO:teuthology.orchestra.run.smithi016.stdout: debug bluestore = 1/20 2024-03-31T15:14:53.697 INFO:teuthology.orchestra.run.smithi016.stdout: debug ms = 1 2024-03-31T15:14:53.697 INFO:teuthology.orchestra.run.smithi016.stdout: debug osd = 20 2024-03-31T15:14:53.697 INFO:teuthology.orchestra.run.smithi016.stdout: debug rocksdb = 4/10 2024-03-31T15:14:53.697 INFO:teuthology.orchestra.run.smithi016.stdout: mon osd backfillfull_ratio = 0.85 2024-03-31T15:14:53.697 INFO:teuthology.orchestra.run.smithi016.stdout: mon osd full ratio = 0.9 2024-03-31T15:14:53.698 INFO:teuthology.orchestra.run.smithi016.stdout: mon osd nearfull ratio = 0.8 2024-03-31T15:14:53.698 INFO:teuthology.orchestra.run.smithi016.stdout: osd failsafe full ratio = 0.95 2024-03-31T15:14:53.698 INFO:teuthology.orchestra.run.smithi016.stdout: osd objectstore = bluestore 2024-03-31T15:14:53.698 INFO:teuthology.orchestra.run.smithi016.stdout: osd op complaint time = 180 2024-03-31T15:14:53.698 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:14:53.698 INFO:teuthology.orchestra.run.smithi016.stdout:[mgr] 2024-03-31T15:14:53.698 INFO:teuthology.orchestra.run.smithi016.stdout: debug ms = 1 2024-03-31T15:14:53.698 INFO:teuthology.orchestra.run.smithi016.stdout: debug mgr = 20 2024-03-31T15:14:53.698 INFO:teuthology.orchestra.run.smithi016.stdout: debug mon = 20 2024-03-31T15:14:53.698 INFO:teuthology.orchestra.run.smithi016.stdout: debug auth = 20 2024-03-31T15:14:53.698 INFO:teuthology.orchestra.run.smithi016.stdout: mon reweight min pgs per osd = 4 2024-03-31T15:14:53.698 INFO:teuthology.orchestra.run.smithi016.stdout: mon reweight min bytes per osd = 10 2024-03-31T15:14:53.698 INFO:teuthology.orchestra.run.smithi016.stdout: mgr/telemetry/nag = false 2024-03-31T15:14:53.699 INFO:teuthology.orchestra.run.smithi016.stdout: debug client = 20 2024-03-31T15:14:53.699 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:14:53.699 INFO:teuthology.orchestra.run.smithi016.stdout:[mon] 2024-03-31T15:14:53.699 INFO:teuthology.orchestra.run.smithi016.stdout: debug ms = 1 2024-03-31T15:14:53.699 INFO:teuthology.orchestra.run.smithi016.stdout: debug mon = 20 2024-03-31T15:14:53.699 INFO:teuthology.orchestra.run.smithi016.stdout: debug paxos = 20 2024-03-31T15:14:53.699 INFO:teuthology.orchestra.run.smithi016.stdout: debug auth = 20 2024-03-31T15:14:53.699 INFO:teuthology.orchestra.run.smithi016.stdout: mon data avail warn = 5 2024-03-31T15:14:53.699 INFO:teuthology.orchestra.run.smithi016.stdout: mon mgr mkfs grace = 240 2024-03-31T15:14:53.699 INFO:teuthology.orchestra.run.smithi016.stdout: mon reweight min pgs per osd = 4 2024-03-31T15:14:53.699 INFO:teuthology.orchestra.run.smithi016.stdout: mon osd reporter subtree level = osd 2024-03-31T15:14:53.699 INFO:teuthology.orchestra.run.smithi016.stdout: mon osd prime pg temp = true 2024-03-31T15:14:53.699 INFO:teuthology.orchestra.run.smithi016.stdout: mon reweight min bytes per osd = 10 2024-03-31T15:14:53.699 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:14:53.700 INFO:teuthology.orchestra.run.smithi016.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-31T15:14:53.700 INFO:teuthology.orchestra.run.smithi016.stdout: auth mon ticket ttl = 660 # 11m 2024-03-31T15:14:53.700 INFO:teuthology.orchestra.run.smithi016.stdout: auth service ticket ttl = 240 # 4m 2024-03-31T15:14:53.700 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:14:53.700 INFO:teuthology.orchestra.run.smithi016.stdout: # don't complain about insecure global_id in the test suite 2024-03-31T15:14:53.700 INFO:teuthology.orchestra.run.smithi016.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-31T15:14:53.700 INFO:teuthology.orchestra.run.smithi016.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-31T15:14:53.700 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:14:53.700 INFO:teuthology.orchestra.run.smithi016.stdout: # 1m isn't quite enough 2024-03-31T15:14:53.700 INFO:teuthology.orchestra.run.smithi016.stdout: mon_down_mkfs_grace = 2m 2024-03-31T15:14:53.701 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:14:53.701 INFO:teuthology.orchestra.run.smithi016.stdout: mon_warn_on_filestore_osds = false 2024-03-31T15:14:53.701 INFO:teuthology.orchestra.run.smithi016.stdout: mon down mkfs grace = 300 2024-03-31T15:14:53.701 INFO:teuthology.orchestra.run.smithi016.stdout: mon op complaint time = 120 2024-03-31T15:14:53.701 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:14:53.701 INFO:teuthology.orchestra.run.smithi016.stdout:[client] 2024-03-31T15:14:53.701 INFO:teuthology.orchestra.run.smithi016.stdout: rgw cache enabled = true 2024-03-31T15:14:53.701 INFO:teuthology.orchestra.run.smithi016.stdout: rgw enable ops log = true 2024-03-31T15:14:53.701 INFO:teuthology.orchestra.run.smithi016.stdout: rgw enable usage log = true 2024-03-31T15:14:53.701 INFO:teuthology.orchestra.run.smithi016.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-31T15:14:53.701 INFO:teuthology.orchestra.run.smithi016.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-31T15:14:53.701 INFO:teuthology.orchestra.run.smithi016.stdout: client mount timeout = 600 2024-03-31T15:14:53.701 INFO:teuthology.orchestra.run.smithi016.stdout: debug client = 20 2024-03-31T15:14:53.701 INFO:teuthology.orchestra.run.smithi016.stdout: debug ms = 1 2024-03-31T15:14:53.702 INFO:teuthology.orchestra.run.smithi016.stdout: pid file = "" 2024-03-31T15:14:53.702 INFO:teuthology.orchestra.run.smithi016.stdout: rados mon op timeout = 900 2024-03-31T15:14:53.702 INFO:teuthology.orchestra.run.smithi016.stdout: rados osd op timeout = 900 2024-03-31T15:14:53.702 INFO:teuthology.orchestra.run.smithi016.stdout:[mon.a] 2024-03-31T15:14:53.702 INFO:teuthology.orchestra.run.smithi016.stdout:[mon.b] 2024-03-31T15:14:53.702 INFO:teuthology.orchestra.run.smithi016.stdout:[mon.c] 2024-03-31T15:14:53.702 INFO:teuthology.orchestra.run.smithi016.stdout:[mds.a] 2024-03-31T15:14:53.702 INFO:teuthology.orchestra.run.smithi016.stdout:[mds.c] 2024-03-31T15:14:53.702 INFO:teuthology.orchestra.run.smithi016.stdout:[mds.b] 2024-03-31T15:14:53.702 INFO:teuthology.orchestra.run.smithi016.stdout:[mds.d] 2024-03-31T15:14:53.702 INFO:teuthology.orchestra.run.smithi016.stdout:[mds] 2024-03-31T15:14:53.702 INFO:teuthology.orchestra.run.smithi016.stdout: debug mds = 20 2024-03-31T15:14:53.702 INFO:teuthology.orchestra.run.smithi016.stdout: debug mds balancer = 20 2024-03-31T15:14:53.702 INFO:teuthology.orchestra.run.smithi016.stdout: debug ms = 1 2024-03-31T15:14:53.703 INFO:teuthology.orchestra.run.smithi016.stdout: mds debug frag = True 2024-03-31T15:14:53.703 INFO:teuthology.orchestra.run.smithi016.stdout: mds debug scatterstat = False 2024-03-31T15:14:53.703 INFO:teuthology.orchestra.run.smithi016.stdout: mds op complaint time = 180 2024-03-31T15:14:53.703 INFO:teuthology.orchestra.run.smithi016.stdout: mds verify scatter = True 2024-03-31T15:14:53.703 INFO:teuthology.orchestra.run.smithi016.stdout: osd op complaint time = 180 2024-03-31T15:14:53.703 INFO:teuthology.orchestra.run.smithi016.stdout: rados mon op timeout = 900 2024-03-31T15:14:53.703 INFO:teuthology.orchestra.run.smithi016.stdout: rados osd op timeout = 900 2024-03-31T15:14:53.703 INFO:teuthology.orchestra.run.smithi089.stdout:[global] 2024-03-31T15:14:53.703 INFO:teuthology.orchestra.run.smithi089.stdout: chdir = "" 2024-03-31T15:14:53.704 INFO:teuthology.orchestra.run.smithi089.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-31T15:14:53.704 INFO:teuthology.orchestra.run.smithi089.stdout: auth supported = cephx 2024-03-31T15:14:53.704 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-31T15:14:53.704 INFO:teuthology.orchestra.run.smithi089.stdout: filestore xattr use omap = true 2024-03-31T15:14:53.704 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-31T15:14:53.704 INFO:teuthology.orchestra.run.smithi089.stdout: mon clock drift allowed = 1.000 2024-03-31T15:14:53.704 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-31T15:14:53.704 INFO:teuthology.orchestra.run.smithi089.stdout: osd crush chooseleaf type = 0 2024-03-31T15:14:53.704 INFO:teuthology.orchestra.run.smithi089.stdout: auth debug = true 2024-03-31T15:14:53.704 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-31T15:14:53.704 INFO:teuthology.orchestra.run.smithi089.stdout: ms die on old message = true 2024-03-31T15:14:53.704 INFO:teuthology.orchestra.run.smithi089.stdout: ms die on bug = true 2024-03-31T15:14:53.704 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-31T15:14:53.704 INFO:teuthology.orchestra.run.smithi089.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-31T15:14:53.705 INFO:teuthology.orchestra.run.smithi089.stdout: mon pg warn max object skew = 0 2024-03-31T15:14:53.705 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-31T15:14:53.705 INFO:teuthology.orchestra.run.smithi089.stdout: # disable pg_autoscaler by default for new pools 2024-03-31T15:14:53.705 INFO:teuthology.orchestra.run.smithi089.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-31T15:14:53.705 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-31T15:14:53.705 INFO:teuthology.orchestra.run.smithi089.stdout: osd pool default size = 2 2024-03-31T15:14:53.705 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-31T15:14:53.705 INFO:teuthology.orchestra.run.smithi089.stdout: mon osd allow primary affinity = true 2024-03-31T15:14:53.705 INFO:teuthology.orchestra.run.smithi089.stdout: mon osd allow pg remap = true 2024-03-31T15:14:53.705 INFO:teuthology.orchestra.run.smithi089.stdout: mon warn on legacy crush tunables = false 2024-03-31T15:14:53.705 INFO:teuthology.orchestra.run.smithi089.stdout: mon warn on crush straw calc version zero = false 2024-03-31T15:14:53.705 INFO:teuthology.orchestra.run.smithi089.stdout: mon warn on no sortbitwise = false 2024-03-31T15:14:53.705 INFO:teuthology.orchestra.run.smithi089.stdout: mon warn on osd down out interval zero = false 2024-03-31T15:14:53.705 INFO:teuthology.orchestra.run.smithi089.stdout: mon warn on too few osds = false 2024-03-31T15:14:53.705 INFO:teuthology.orchestra.run.smithi089.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-31T15:14:53.706 INFO:teuthology.orchestra.run.smithi089.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-31T15:14:53.706 INFO:teuthology.orchestra.run.smithi089.stdout: mon_allow_pool_size_one = true 2024-03-31T15:14:53.706 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-31T15:14:53.706 INFO:teuthology.orchestra.run.smithi089.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-31T15:14:53.706 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-31T15:14:53.706 INFO:teuthology.orchestra.run.smithi089.stdout: osd default data pool replay window = 5 2024-03-31T15:14:53.706 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-31T15:14:53.706 INFO:teuthology.orchestra.run.smithi089.stdout: mon allow pool delete = true 2024-03-31T15:14:53.706 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-31T15:14:53.706 INFO:teuthology.orchestra.run.smithi089.stdout: mon cluster log file level = debug 2024-03-31T15:14:53.706 INFO:teuthology.orchestra.run.smithi089.stdout: debug asserts on shutdown = true 2024-03-31T15:14:53.706 INFO:teuthology.orchestra.run.smithi089.stdout: mon health detail to clog = false 2024-03-31T15:14:53.706 INFO:teuthology.orchestra.run.smithi089.stdout: mon host = "172.21.15.16,172.21.15.89,[v2:172.21.15.89:3301,v1:172.21.15.89:6790]" 2024-03-31T15:14:53.706 INFO:teuthology.orchestra.run.smithi089.stdout: fsid = 286eb5da-fb47-4550-b290-d9519c13b9a5 2024-03-31T15:14:53.707 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-31T15:14:53.707 INFO:teuthology.orchestra.run.smithi089.stdout:[osd] 2024-03-31T15:14:53.707 INFO:teuthology.orchestra.run.smithi089.stdout: osd journal size = 100 2024-03-31T15:14:53.707 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-31T15:14:53.707 INFO:teuthology.orchestra.run.smithi089.stdout: osd scrub load threshold = 5.0 2024-03-31T15:14:53.707 INFO:teuthology.orchestra.run.smithi089.stdout: osd scrub max interval = 600 2024-03-31T15:14:53.707 INFO:teuthology.orchestra.run.smithi089.stdout: osd mclock profile = high_recovery_ops 2024-03-31T15:14:53.707 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-31T15:14:53.707 INFO:teuthology.orchestra.run.smithi089.stdout: osd recover clone overlap = true 2024-03-31T15:14:53.707 INFO:teuthology.orchestra.run.smithi089.stdout: osd recovery max chunk = 1048576 2024-03-31T15:14:53.707 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-31T15:14:53.707 INFO:teuthology.orchestra.run.smithi089.stdout: osd debug shutdown = true 2024-03-31T15:14:53.707 INFO:teuthology.orchestra.run.smithi089.stdout: osd debug op order = true 2024-03-31T15:14:53.707 INFO:teuthology.orchestra.run.smithi089.stdout: osd debug verify stray on activate = true 2024-03-31T15:14:53.707 INFO:teuthology.orchestra.run.smithi089.stdout: osd debug trim objects = true 2024-03-31T15:14:53.708 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-31T15:14:53.708 INFO:teuthology.orchestra.run.smithi089.stdout: osd open classes on start = true 2024-03-31T15:14:53.708 INFO:teuthology.orchestra.run.smithi089.stdout: osd debug pg log writeout = true 2024-03-31T15:14:53.708 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-31T15:14:53.708 INFO:teuthology.orchestra.run.smithi089.stdout: osd deep scrub update digest min age = 30 2024-03-31T15:14:53.708 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-31T15:14:53.708 INFO:teuthology.orchestra.run.smithi089.stdout: osd map max advance = 10 2024-03-31T15:14:53.708 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-31T15:14:53.708 INFO:teuthology.orchestra.run.smithi089.stdout: journal zero on create = true 2024-03-31T15:14:53.708 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-31T15:14:53.708 INFO:teuthology.orchestra.run.smithi089.stdout: filestore ondisk finisher threads = 3 2024-03-31T15:14:53.708 INFO:teuthology.orchestra.run.smithi089.stdout: filestore apply finisher threads = 3 2024-03-31T15:14:53.708 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-31T15:14:53.708 INFO:teuthology.orchestra.run.smithi089.stdout: bdev debug aio = true 2024-03-31T15:14:53.709 INFO:teuthology.orchestra.run.smithi089.stdout: osd debug misdirected ops = true 2024-03-31T15:14:53.709 INFO:teuthology.orchestra.run.smithi089.stdout: bdev async discard = True 2024-03-31T15:14:53.709 INFO:teuthology.orchestra.run.smithi089.stdout: bdev enable discard = True 2024-03-31T15:14:53.709 INFO:teuthology.orchestra.run.smithi089.stdout: bluestore allocator = bitmap 2024-03-31T15:14:53.709 INFO:teuthology.orchestra.run.smithi089.stdout: bluestore block size = 96636764160 2024-03-31T15:14:53.709 INFO:teuthology.orchestra.run.smithi089.stdout: bluestore fsck on mount = True 2024-03-31T15:14:53.709 INFO:teuthology.orchestra.run.smithi089.stdout: debug bluefs = 1/20 2024-03-31T15:14:53.709 INFO:teuthology.orchestra.run.smithi089.stdout: debug bluestore = 1/20 2024-03-31T15:14:53.709 INFO:teuthology.orchestra.run.smithi089.stdout: debug ms = 1 2024-03-31T15:14:53.709 INFO:teuthology.orchestra.run.smithi089.stdout: debug osd = 20 2024-03-31T15:14:53.709 INFO:teuthology.orchestra.run.smithi089.stdout: debug rocksdb = 4/10 2024-03-31T15:14:53.709 INFO:teuthology.orchestra.run.smithi089.stdout: mon osd backfillfull_ratio = 0.85 2024-03-31T15:14:53.709 INFO:teuthology.orchestra.run.smithi089.stdout: mon osd full ratio = 0.9 2024-03-31T15:14:53.709 INFO:teuthology.orchestra.run.smithi089.stdout: mon osd nearfull ratio = 0.8 2024-03-31T15:14:53.709 INFO:teuthology.orchestra.run.smithi089.stdout: osd failsafe full ratio = 0.95 2024-03-31T15:14:53.710 INFO:teuthology.orchestra.run.smithi089.stdout: osd objectstore = bluestore 2024-03-31T15:14:53.710 INFO:teuthology.orchestra.run.smithi089.stdout: osd op complaint time = 180 2024-03-31T15:14:53.710 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-31T15:14:53.710 INFO:teuthology.orchestra.run.smithi089.stdout:[mgr] 2024-03-31T15:14:53.710 INFO:teuthology.orchestra.run.smithi089.stdout: debug ms = 1 2024-03-31T15:14:53.710 INFO:teuthology.orchestra.run.smithi089.stdout: debug mgr = 20 2024-03-31T15:14:53.710 INFO:teuthology.orchestra.run.smithi089.stdout: debug mon = 20 2024-03-31T15:14:53.710 INFO:teuthology.orchestra.run.smithi089.stdout: debug auth = 20 2024-03-31T15:14:53.710 INFO:teuthology.orchestra.run.smithi089.stdout: mon reweight min pgs per osd = 4 2024-03-31T15:14:53.710 INFO:teuthology.orchestra.run.smithi089.stdout: mon reweight min bytes per osd = 10 2024-03-31T15:14:53.710 INFO:teuthology.orchestra.run.smithi089.stdout: mgr/telemetry/nag = false 2024-03-31T15:14:53.710 INFO:teuthology.orchestra.run.smithi089.stdout: debug client = 20 2024-03-31T15:14:53.710 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-31T15:14:53.710 INFO:teuthology.orchestra.run.smithi089.stdout:[mon] 2024-03-31T15:14:53.711 INFO:teuthology.orchestra.run.smithi089.stdout: debug ms = 1 2024-03-31T15:14:53.711 INFO:teuthology.orchestra.run.smithi089.stdout: debug mon = 20 2024-03-31T15:14:53.711 INFO:teuthology.orchestra.run.smithi089.stdout: debug paxos = 20 2024-03-31T15:14:53.711 INFO:teuthology.orchestra.run.smithi089.stdout: debug auth = 20 2024-03-31T15:14:53.711 INFO:teuthology.orchestra.run.smithi089.stdout: mon data avail warn = 5 2024-03-31T15:14:53.711 INFO:teuthology.orchestra.run.smithi089.stdout: mon mgr mkfs grace = 240 2024-03-31T15:14:53.711 INFO:teuthology.orchestra.run.smithi089.stdout: mon reweight min pgs per osd = 4 2024-03-31T15:14:53.711 INFO:teuthology.orchestra.run.smithi089.stdout: mon osd reporter subtree level = osd 2024-03-31T15:14:53.711 INFO:teuthology.orchestra.run.smithi089.stdout: mon osd prime pg temp = true 2024-03-31T15:14:53.711 INFO:teuthology.orchestra.run.smithi089.stdout: mon reweight min bytes per osd = 10 2024-03-31T15:14:53.711 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-31T15:14:53.711 INFO:teuthology.orchestra.run.smithi089.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-31T15:14:53.711 INFO:teuthology.orchestra.run.smithi089.stdout: auth mon ticket ttl = 660 # 11m 2024-03-31T15:14:53.711 INFO:teuthology.orchestra.run.smithi089.stdout: auth service ticket ttl = 240 # 4m 2024-03-31T15:14:53.712 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-31T15:14:53.712 INFO:teuthology.orchestra.run.smithi089.stdout: # don't complain about insecure global_id in the test suite 2024-03-31T15:14:53.712 INFO:teuthology.orchestra.run.smithi089.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-31T15:14:53.712 INFO:teuthology.orchestra.run.smithi089.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-31T15:14:53.712 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-31T15:14:53.712 INFO:teuthology.orchestra.run.smithi089.stdout: # 1m isn't quite enough 2024-03-31T15:14:53.712 INFO:teuthology.orchestra.run.smithi089.stdout: mon_down_mkfs_grace = 2m 2024-03-31T15:14:53.712 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-31T15:14:53.712 INFO:teuthology.orchestra.run.smithi089.stdout: mon_warn_on_filestore_osds = false 2024-03-31T15:14:53.712 INFO:teuthology.orchestra.run.smithi089.stdout: mon down mkfs grace = 300 2024-03-31T15:14:53.712 INFO:teuthology.orchestra.run.smithi089.stdout: mon op complaint time = 120 2024-03-31T15:14:53.712 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-31T15:14:53.712 INFO:teuthology.orchestra.run.smithi089.stdout:[client] 2024-03-31T15:14:53.712 INFO:teuthology.orchestra.run.smithi089.stdout: rgw cache enabled = true 2024-03-31T15:14:53.713 INFO:teuthology.orchestra.run.smithi089.stdout: rgw enable ops log = true 2024-03-31T15:14:53.713 INFO:teuthology.orchestra.run.smithi089.stdout: rgw enable usage log = true 2024-03-31T15:14:53.713 INFO:teuthology.orchestra.run.smithi089.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-31T15:14:53.713 INFO:teuthology.orchestra.run.smithi089.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-31T15:14:53.713 INFO:teuthology.orchestra.run.smithi089.stdout: client mount timeout = 600 2024-03-31T15:14:53.713 INFO:teuthology.orchestra.run.smithi089.stdout: debug client = 20 2024-03-31T15:14:53.713 INFO:teuthology.orchestra.run.smithi089.stdout: debug ms = 1 2024-03-31T15:14:53.713 INFO:teuthology.orchestra.run.smithi089.stdout: pid file = "" 2024-03-31T15:14:53.713 INFO:teuthology.orchestra.run.smithi089.stdout: rados mon op timeout = 900 2024-03-31T15:14:53.713 INFO:teuthology.orchestra.run.smithi089.stdout: rados osd op timeout = 900 2024-03-31T15:14:53.713 INFO:teuthology.orchestra.run.smithi089.stdout:[mon.a] 2024-03-31T15:14:53.713 INFO:teuthology.orchestra.run.smithi089.stdout:[mon.b] 2024-03-31T15:14:53.713 INFO:teuthology.orchestra.run.smithi089.stdout:[mon.c] 2024-03-31T15:14:53.713 INFO:teuthology.orchestra.run.smithi089.stdout:[mds.a] 2024-03-31T15:14:53.713 INFO:teuthology.orchestra.run.smithi089.stdout:[mds.c] 2024-03-31T15:14:53.714 INFO:teuthology.orchestra.run.smithi089.stdout:[mds.b] 2024-03-31T15:14:53.714 INFO:teuthology.orchestra.run.smithi089.stdout:[mds.d] 2024-03-31T15:14:53.714 INFO:teuthology.orchestra.run.smithi089.stdout:[mds] 2024-03-31T15:14:53.714 INFO:teuthology.orchestra.run.smithi089.stdout: debug mds = 20 2024-03-31T15:14:53.714 INFO:teuthology.orchestra.run.smithi089.stdout: debug mds balancer = 20 2024-03-31T15:14:53.714 INFO:teuthology.orchestra.run.smithi089.stdout: debug ms = 1 2024-03-31T15:14:53.714 INFO:teuthology.orchestra.run.smithi089.stdout: mds debug frag = True 2024-03-31T15:14:53.714 INFO:teuthology.orchestra.run.smithi089.stdout: mds debug scatterstat = False 2024-03-31T15:14:53.714 INFO:teuthology.orchestra.run.smithi089.stdout: mds op complaint time = 180 2024-03-31T15:14:53.714 INFO:teuthology.orchestra.run.smithi089.stdout: mds verify scatter = True 2024-03-31T15:14:53.714 INFO:teuthology.orchestra.run.smithi089.stdout: osd op complaint time = 180 2024-03-31T15:14:53.714 INFO:teuthology.orchestra.run.smithi089.stdout: rados mon op timeout = 900 2024-03-31T15:14:53.714 INFO:teuthology.orchestra.run.smithi089.stdout: rados osd op timeout = 900 2024-03-31T15:14:53.716 DEBUG:teuthology.orchestra.run.smithi016:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-31T15:14:53.763 DEBUG:teuthology.orchestra.run.smithi089:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-31T15:14:53.803 INFO:teuthology.orchestra.run.smithi016.stdout:[global] 2024-03-31T15:14:53.803 INFO:teuthology.orchestra.run.smithi016.stdout: chdir = "" 2024-03-31T15:14:53.803 INFO:teuthology.orchestra.run.smithi016.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-31T15:14:53.803 INFO:teuthology.orchestra.run.smithi016.stdout: auth supported = cephx 2024-03-31T15:14:53.803 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:14:53.803 INFO:teuthology.orchestra.run.smithi016.stdout: filestore xattr use omap = true 2024-03-31T15:14:53.804 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:14:53.804 INFO:teuthology.orchestra.run.smithi016.stdout: mon clock drift allowed = 1.000 2024-03-31T15:14:53.804 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:14:53.806 INFO:teuthology.orchestra.run.smithi016.stdout: osd crush chooseleaf type = 0 2024-03-31T15:14:53.806 INFO:teuthology.orchestra.run.smithi016.stdout: auth debug = true 2024-03-31T15:14:53.806 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:14:53.806 INFO:teuthology.orchestra.run.smithi016.stdout: ms die on old message = true 2024-03-31T15:14:53.806 INFO:teuthology.orchestra.run.smithi016.stdout: ms die on bug = true 2024-03-31T15:14:53.806 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:14:53.806 INFO:teuthology.orchestra.run.smithi016.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-31T15:14:53.807 INFO:teuthology.orchestra.run.smithi016.stdout: mon pg warn max object skew = 0 2024-03-31T15:14:53.807 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:14:53.807 INFO:teuthology.orchestra.run.smithi016.stdout: # disable pg_autoscaler by default for new pools 2024-03-31T15:14:53.807 INFO:teuthology.orchestra.run.smithi016.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-31T15:14:53.807 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:14:53.807 INFO:teuthology.orchestra.run.smithi016.stdout: osd pool default size = 2 2024-03-31T15:14:53.807 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:14:53.807 INFO:teuthology.orchestra.run.smithi016.stdout: mon osd allow primary affinity = true 2024-03-31T15:14:53.807 INFO:teuthology.orchestra.run.smithi016.stdout: mon osd allow pg remap = true 2024-03-31T15:14:53.807 INFO:teuthology.orchestra.run.smithi016.stdout: mon warn on legacy crush tunables = false 2024-03-31T15:14:53.807 INFO:teuthology.orchestra.run.smithi016.stdout: mon warn on crush straw calc version zero = false 2024-03-31T15:14:53.807 INFO:teuthology.orchestra.run.smithi016.stdout: mon warn on no sortbitwise = false 2024-03-31T15:14:53.807 INFO:teuthology.orchestra.run.smithi016.stdout: mon warn on osd down out interval zero = false 2024-03-31T15:14:53.807 INFO:teuthology.orchestra.run.smithi016.stdout: mon warn on too few osds = false 2024-03-31T15:14:53.808 INFO:teuthology.orchestra.run.smithi016.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-31T15:14:53.808 INFO:teuthology.orchestra.run.smithi016.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-31T15:14:53.808 INFO:teuthology.orchestra.run.smithi016.stdout: mon_allow_pool_size_one = true 2024-03-31T15:14:53.808 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:14:53.808 INFO:teuthology.orchestra.run.smithi016.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-31T15:14:53.808 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:14:53.808 INFO:teuthology.orchestra.run.smithi016.stdout: osd default data pool replay window = 5 2024-03-31T15:14:53.808 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:14:53.808 INFO:teuthology.orchestra.run.smithi016.stdout: mon allow pool delete = true 2024-03-31T15:14:53.808 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:14:53.808 INFO:teuthology.orchestra.run.smithi016.stdout: mon cluster log file level = debug 2024-03-31T15:14:53.808 INFO:teuthology.orchestra.run.smithi016.stdout: debug asserts on shutdown = true 2024-03-31T15:14:53.808 INFO:teuthology.orchestra.run.smithi016.stdout: mon health detail to clog = false 2024-03-31T15:14:53.808 INFO:teuthology.orchestra.run.smithi016.stdout: mon host = "172.21.15.16,172.21.15.89,[v2:172.21.15.89:3301,v1:172.21.15.89:6790]" 2024-03-31T15:14:53.809 INFO:teuthology.orchestra.run.smithi016.stdout: fsid = 286eb5da-fb47-4550-b290-d9519c13b9a5 2024-03-31T15:14:53.809 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:14:53.809 INFO:teuthology.orchestra.run.smithi016.stdout:[osd] 2024-03-31T15:14:53.809 INFO:teuthology.orchestra.run.smithi016.stdout: osd journal size = 100 2024-03-31T15:14:53.809 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:14:53.809 INFO:teuthology.orchestra.run.smithi016.stdout: osd scrub load threshold = 5.0 2024-03-31T15:14:53.809 INFO:teuthology.orchestra.run.smithi016.stdout: osd scrub max interval = 600 2024-03-31T15:14:53.809 INFO:teuthology.orchestra.run.smithi016.stdout: osd mclock profile = high_recovery_ops 2024-03-31T15:14:53.809 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:14:53.809 INFO:teuthology.orchestra.run.smithi016.stdout: osd recover clone overlap = true 2024-03-31T15:14:53.809 INFO:teuthology.orchestra.run.smithi016.stdout: osd recovery max chunk = 1048576 2024-03-31T15:14:53.809 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:14:53.809 INFO:teuthology.orchestra.run.smithi016.stdout: osd debug shutdown = true 2024-03-31T15:14:53.809 INFO:teuthology.orchestra.run.smithi016.stdout: osd debug op order = true 2024-03-31T15:14:53.809 INFO:teuthology.orchestra.run.smithi016.stdout: osd debug verify stray on activate = true 2024-03-31T15:14:53.810 INFO:teuthology.orchestra.run.smithi016.stdout: osd debug trim objects = true 2024-03-31T15:14:53.810 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:14:53.810 INFO:teuthology.orchestra.run.smithi016.stdout: osd open classes on start = true 2024-03-31T15:14:53.810 INFO:teuthology.orchestra.run.smithi016.stdout: osd debug pg log writeout = true 2024-03-31T15:14:53.810 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:14:53.810 INFO:teuthology.orchestra.run.smithi016.stdout: osd deep scrub update digest min age = 30 2024-03-31T15:14:53.810 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:14:53.810 INFO:teuthology.orchestra.run.smithi016.stdout: osd map max advance = 10 2024-03-31T15:14:53.810 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:14:53.810 INFO:teuthology.orchestra.run.smithi016.stdout: journal zero on create = true 2024-03-31T15:14:53.810 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:14:53.810 INFO:teuthology.orchestra.run.smithi016.stdout: filestore ondisk finisher threads = 3 2024-03-31T15:14:53.810 INFO:teuthology.orchestra.run.smithi016.stdout: filestore apply finisher threads = 3 2024-03-31T15:14:53.810 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:14:53.811 INFO:teuthology.orchestra.run.smithi016.stdout: bdev debug aio = true 2024-03-31T15:14:53.811 INFO:teuthology.orchestra.run.smithi016.stdout: osd debug misdirected ops = true 2024-03-31T15:14:53.811 INFO:teuthology.orchestra.run.smithi016.stdout: bdev async discard = True 2024-03-31T15:14:53.811 INFO:teuthology.orchestra.run.smithi016.stdout: bdev enable discard = True 2024-03-31T15:14:53.811 INFO:teuthology.orchestra.run.smithi016.stdout: bluestore allocator = bitmap 2024-03-31T15:14:53.811 INFO:teuthology.orchestra.run.smithi016.stdout: bluestore block size = 96636764160 2024-03-31T15:14:53.811 INFO:teuthology.orchestra.run.smithi016.stdout: bluestore fsck on mount = True 2024-03-31T15:14:53.811 INFO:teuthology.orchestra.run.smithi016.stdout: debug bluefs = 1/20 2024-03-31T15:14:53.811 INFO:teuthology.orchestra.run.smithi016.stdout: debug bluestore = 1/20 2024-03-31T15:14:53.811 INFO:teuthology.orchestra.run.smithi016.stdout: debug ms = 1 2024-03-31T15:14:53.811 INFO:teuthology.orchestra.run.smithi016.stdout: debug osd = 20 2024-03-31T15:14:53.811 INFO:teuthology.orchestra.run.smithi016.stdout: debug rocksdb = 4/10 2024-03-31T15:14:53.811 INFO:teuthology.orchestra.run.smithi016.stdout: mon osd backfillfull_ratio = 0.85 2024-03-31T15:14:53.811 INFO:teuthology.orchestra.run.smithi016.stdout: mon osd full ratio = 0.9 2024-03-31T15:14:53.812 INFO:teuthology.orchestra.run.smithi016.stdout: mon osd nearfull ratio = 0.8 2024-03-31T15:14:53.812 INFO:teuthology.orchestra.run.smithi016.stdout: osd failsafe full ratio = 0.95 2024-03-31T15:14:53.812 INFO:teuthology.orchestra.run.smithi016.stdout: osd objectstore = bluestore 2024-03-31T15:14:53.812 INFO:teuthology.orchestra.run.smithi016.stdout: osd op complaint time = 180 2024-03-31T15:14:53.812 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:14:53.812 INFO:teuthology.orchestra.run.smithi016.stdout:[mgr] 2024-03-31T15:14:53.812 INFO:teuthology.orchestra.run.smithi016.stdout: debug ms = 1 2024-03-31T15:14:53.812 INFO:teuthology.orchestra.run.smithi016.stdout: debug mgr = 20 2024-03-31T15:14:53.812 INFO:teuthology.orchestra.run.smithi016.stdout: debug mon = 20 2024-03-31T15:14:53.812 INFO:teuthology.orchestra.run.smithi016.stdout: debug auth = 20 2024-03-31T15:14:53.812 INFO:teuthology.orchestra.run.smithi016.stdout: mon reweight min pgs per osd = 4 2024-03-31T15:14:53.812 INFO:teuthology.orchestra.run.smithi016.stdout: mon reweight min bytes per osd = 10 2024-03-31T15:14:53.812 INFO:teuthology.orchestra.run.smithi016.stdout: mgr/telemetry/nag = false 2024-03-31T15:14:53.812 INFO:teuthology.orchestra.run.smithi016.stdout: debug client = 20 2024-03-31T15:14:53.812 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:14:53.813 INFO:teuthology.orchestra.run.smithi016.stdout:[mon] 2024-03-31T15:14:53.813 INFO:teuthology.orchestra.run.smithi016.stdout: debug ms = 1 2024-03-31T15:14:53.813 INFO:teuthology.orchestra.run.smithi016.stdout: debug mon = 20 2024-03-31T15:14:53.813 INFO:teuthology.orchestra.run.smithi016.stdout: debug paxos = 20 2024-03-31T15:14:53.813 INFO:teuthology.orchestra.run.smithi016.stdout: debug auth = 20 2024-03-31T15:14:53.813 INFO:teuthology.orchestra.run.smithi016.stdout: mon data avail warn = 5 2024-03-31T15:14:53.813 INFO:teuthology.orchestra.run.smithi016.stdout: mon mgr mkfs grace = 240 2024-03-31T15:14:53.813 INFO:teuthology.orchestra.run.smithi016.stdout: mon reweight min pgs per osd = 4 2024-03-31T15:14:53.813 INFO:teuthology.orchestra.run.smithi016.stdout: mon osd reporter subtree level = osd 2024-03-31T15:14:53.813 INFO:teuthology.orchestra.run.smithi016.stdout: mon osd prime pg temp = true 2024-03-31T15:14:53.813 INFO:teuthology.orchestra.run.smithi016.stdout: mon reweight min bytes per osd = 10 2024-03-31T15:14:53.813 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:14:53.813 INFO:teuthology.orchestra.run.smithi016.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-31T15:14:53.813 INFO:teuthology.orchestra.run.smithi016.stdout: auth mon ticket ttl = 660 # 11m 2024-03-31T15:14:53.813 INFO:teuthology.orchestra.run.smithi016.stdout: auth service ticket ttl = 240 # 4m 2024-03-31T15:14:53.814 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:14:53.814 INFO:teuthology.orchestra.run.smithi016.stdout: # don't complain about insecure global_id in the test suite 2024-03-31T15:14:53.814 INFO:teuthology.orchestra.run.smithi016.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-31T15:14:53.814 INFO:teuthology.orchestra.run.smithi016.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-31T15:14:53.814 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:14:53.814 INFO:teuthology.orchestra.run.smithi016.stdout: # 1m isn't quite enough 2024-03-31T15:14:53.814 INFO:teuthology.orchestra.run.smithi016.stdout: mon_down_mkfs_grace = 2m 2024-03-31T15:14:53.814 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:14:53.814 INFO:teuthology.orchestra.run.smithi016.stdout: mon_warn_on_filestore_osds = false 2024-03-31T15:14:53.814 INFO:teuthology.orchestra.run.smithi016.stdout: mon down mkfs grace = 300 2024-03-31T15:14:53.814 INFO:teuthology.orchestra.run.smithi016.stdout: mon op complaint time = 120 2024-03-31T15:14:53.814 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:14:53.814 INFO:teuthology.orchestra.run.smithi016.stdout:[client] 2024-03-31T15:14:53.814 INFO:teuthology.orchestra.run.smithi016.stdout: rgw cache enabled = true 2024-03-31T15:14:53.814 INFO:teuthology.orchestra.run.smithi016.stdout: rgw enable ops log = true 2024-03-31T15:14:53.815 INFO:teuthology.orchestra.run.smithi016.stdout: rgw enable usage log = true 2024-03-31T15:14:53.815 INFO:teuthology.orchestra.run.smithi016.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-31T15:14:53.815 INFO:teuthology.orchestra.run.smithi016.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-31T15:14:53.815 INFO:teuthology.orchestra.run.smithi016.stdout: client mount timeout = 600 2024-03-31T15:14:53.815 INFO:teuthology.orchestra.run.smithi016.stdout: debug client = 20 2024-03-31T15:14:53.815 INFO:teuthology.orchestra.run.smithi016.stdout: debug ms = 1 2024-03-31T15:14:53.815 INFO:teuthology.orchestra.run.smithi016.stdout: pid file = "" 2024-03-31T15:14:53.815 INFO:teuthology.orchestra.run.smithi016.stdout: rados mon op timeout = 900 2024-03-31T15:14:53.815 INFO:teuthology.orchestra.run.smithi016.stdout: rados osd op timeout = 900 2024-03-31T15:14:53.815 INFO:teuthology.orchestra.run.smithi016.stdout:[mon.a] 2024-03-31T15:14:53.815 INFO:teuthology.orchestra.run.smithi016.stdout:[mon.b] 2024-03-31T15:14:53.815 INFO:teuthology.orchestra.run.smithi016.stdout:[mon.c] 2024-03-31T15:14:53.815 INFO:teuthology.orchestra.run.smithi016.stdout:[mds.a] 2024-03-31T15:14:53.815 INFO:teuthology.orchestra.run.smithi016.stdout:[mds.c] 2024-03-31T15:14:53.816 INFO:teuthology.orchestra.run.smithi016.stdout:[mds.b] 2024-03-31T15:14:53.816 INFO:teuthology.orchestra.run.smithi016.stdout:[mds.d] 2024-03-31T15:14:53.816 INFO:teuthology.orchestra.run.smithi016.stdout:[mds] 2024-03-31T15:14:53.816 INFO:teuthology.orchestra.run.smithi016.stdout: debug mds = 20 2024-03-31T15:14:53.816 INFO:teuthology.orchestra.run.smithi016.stdout: debug mds balancer = 20 2024-03-31T15:14:53.816 INFO:teuthology.orchestra.run.smithi016.stdout: debug ms = 1 2024-03-31T15:14:53.816 INFO:teuthology.orchestra.run.smithi016.stdout: mds debug frag = True 2024-03-31T15:14:53.816 INFO:teuthology.orchestra.run.smithi016.stdout: mds debug scatterstat = True 2024-03-31T15:14:53.816 INFO:teuthology.orchestra.run.smithi016.stdout: mds op complaint time = 180 2024-03-31T15:14:53.816 INFO:teuthology.orchestra.run.smithi016.stdout: mds verify scatter = True 2024-03-31T15:14:53.816 INFO:teuthology.orchestra.run.smithi016.stdout: osd op complaint time = 180 2024-03-31T15:14:53.816 INFO:teuthology.orchestra.run.smithi016.stdout: rados mon op timeout = 900 2024-03-31T15:14:53.816 INFO:teuthology.orchestra.run.smithi016.stdout: rados osd op timeout = 900 2024-03-31T15:14:53.817 INFO:teuthology.orchestra.run.smithi089.stdout:[global] 2024-03-31T15:14:53.817 INFO:teuthology.orchestra.run.smithi089.stdout: chdir = "" 2024-03-31T15:14:53.817 INFO:teuthology.orchestra.run.smithi089.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-31T15:14:53.817 INFO:teuthology.orchestra.run.smithi089.stdout: auth supported = cephx 2024-03-31T15:14:53.817 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-31T15:14:53.817 INFO:teuthology.orchestra.run.smithi089.stdout: filestore xattr use omap = true 2024-03-31T15:14:53.818 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-31T15:14:53.818 INFO:teuthology.orchestra.run.smithi089.stdout: mon clock drift allowed = 1.000 2024-03-31T15:14:53.818 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-31T15:14:53.818 INFO:teuthology.orchestra.run.smithi089.stdout: osd crush chooseleaf type = 0 2024-03-31T15:14:53.818 INFO:teuthology.orchestra.run.smithi089.stdout: auth debug = true 2024-03-31T15:14:53.818 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-31T15:14:53.818 INFO:teuthology.orchestra.run.smithi089.stdout: ms die on old message = true 2024-03-31T15:14:53.818 INFO:teuthology.orchestra.run.smithi089.stdout: ms die on bug = true 2024-03-31T15:14:53.818 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-31T15:14:53.818 INFO:teuthology.orchestra.run.smithi089.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-31T15:14:53.818 INFO:teuthology.orchestra.run.smithi089.stdout: mon pg warn max object skew = 0 2024-03-31T15:14:53.818 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-31T15:14:53.818 INFO:teuthology.orchestra.run.smithi089.stdout: # disable pg_autoscaler by default for new pools 2024-03-31T15:14:53.818 INFO:teuthology.orchestra.run.smithi089.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-31T15:14:53.818 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-31T15:14:53.819 INFO:teuthology.orchestra.run.smithi089.stdout: osd pool default size = 2 2024-03-31T15:14:53.819 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-31T15:14:53.819 INFO:teuthology.orchestra.run.smithi089.stdout: mon osd allow primary affinity = true 2024-03-31T15:14:53.819 INFO:teuthology.orchestra.run.smithi089.stdout: mon osd allow pg remap = true 2024-03-31T15:14:53.819 INFO:teuthology.orchestra.run.smithi089.stdout: mon warn on legacy crush tunables = false 2024-03-31T15:14:53.819 INFO:teuthology.orchestra.run.smithi089.stdout: mon warn on crush straw calc version zero = false 2024-03-31T15:14:53.819 INFO:teuthology.orchestra.run.smithi089.stdout: mon warn on no sortbitwise = false 2024-03-31T15:14:53.819 INFO:teuthology.orchestra.run.smithi089.stdout: mon warn on osd down out interval zero = false 2024-03-31T15:14:53.819 INFO:teuthology.orchestra.run.smithi089.stdout: mon warn on too few osds = false 2024-03-31T15:14:53.819 INFO:teuthology.orchestra.run.smithi089.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-31T15:14:53.819 INFO:teuthology.orchestra.run.smithi089.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-31T15:14:53.819 INFO:teuthology.orchestra.run.smithi089.stdout: mon_allow_pool_size_one = true 2024-03-31T15:14:53.819 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-31T15:14:53.819 INFO:teuthology.orchestra.run.smithi089.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-31T15:14:53.820 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-31T15:14:53.820 INFO:teuthology.orchestra.run.smithi089.stdout: osd default data pool replay window = 5 2024-03-31T15:14:53.820 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-31T15:14:53.820 INFO:teuthology.orchestra.run.smithi089.stdout: mon allow pool delete = true 2024-03-31T15:14:53.820 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-31T15:14:53.820 INFO:teuthology.orchestra.run.smithi089.stdout: mon cluster log file level = debug 2024-03-31T15:14:53.820 INFO:teuthology.orchestra.run.smithi089.stdout: debug asserts on shutdown = true 2024-03-31T15:14:53.820 INFO:teuthology.orchestra.run.smithi089.stdout: mon health detail to clog = false 2024-03-31T15:14:53.820 INFO:teuthology.orchestra.run.smithi089.stdout: mon host = "172.21.15.16,172.21.15.89,[v2:172.21.15.89:3301,v1:172.21.15.89:6790]" 2024-03-31T15:14:53.820 INFO:teuthology.orchestra.run.smithi089.stdout: fsid = 286eb5da-fb47-4550-b290-d9519c13b9a5 2024-03-31T15:14:53.820 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-31T15:14:53.820 INFO:teuthology.orchestra.run.smithi089.stdout:[osd] 2024-03-31T15:14:53.820 INFO:teuthology.orchestra.run.smithi089.stdout: osd journal size = 100 2024-03-31T15:14:53.820 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-31T15:14:53.820 INFO:teuthology.orchestra.run.smithi089.stdout: osd scrub load threshold = 5.0 2024-03-31T15:14:53.821 INFO:teuthology.orchestra.run.smithi089.stdout: osd scrub max interval = 600 2024-03-31T15:14:53.821 INFO:teuthology.orchestra.run.smithi089.stdout: osd mclock profile = high_recovery_ops 2024-03-31T15:14:53.821 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-31T15:14:53.821 INFO:teuthology.orchestra.run.smithi089.stdout: osd recover clone overlap = true 2024-03-31T15:14:53.821 INFO:teuthology.orchestra.run.smithi089.stdout: osd recovery max chunk = 1048576 2024-03-31T15:14:53.821 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-31T15:14:53.821 INFO:teuthology.orchestra.run.smithi089.stdout: osd debug shutdown = true 2024-03-31T15:14:53.821 INFO:teuthology.orchestra.run.smithi089.stdout: osd debug op order = true 2024-03-31T15:14:53.821 INFO:teuthology.orchestra.run.smithi089.stdout: osd debug verify stray on activate = true 2024-03-31T15:14:53.821 INFO:teuthology.orchestra.run.smithi089.stdout: osd debug trim objects = true 2024-03-31T15:14:53.821 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-31T15:14:53.821 INFO:teuthology.orchestra.run.smithi089.stdout: osd open classes on start = true 2024-03-31T15:14:53.821 INFO:teuthology.orchestra.run.smithi089.stdout: osd debug pg log writeout = true 2024-03-31T15:14:53.821 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-31T15:14:53.821 INFO:teuthology.orchestra.run.smithi089.stdout: osd deep scrub update digest min age = 30 2024-03-31T15:14:53.822 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-31T15:14:53.822 INFO:teuthology.orchestra.run.smithi089.stdout: osd map max advance = 10 2024-03-31T15:14:53.822 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-31T15:14:53.822 INFO:teuthology.orchestra.run.smithi089.stdout: journal zero on create = true 2024-03-31T15:14:53.822 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-31T15:14:53.822 INFO:teuthology.orchestra.run.smithi089.stdout: filestore ondisk finisher threads = 3 2024-03-31T15:14:53.822 INFO:teuthology.orchestra.run.smithi089.stdout: filestore apply finisher threads = 3 2024-03-31T15:14:53.822 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-31T15:14:53.822 INFO:teuthology.orchestra.run.smithi089.stdout: bdev debug aio = true 2024-03-31T15:14:53.822 INFO:teuthology.orchestra.run.smithi089.stdout: osd debug misdirected ops = true 2024-03-31T15:14:53.822 INFO:teuthology.orchestra.run.smithi089.stdout: bdev async discard = True 2024-03-31T15:14:53.822 INFO:teuthology.orchestra.run.smithi089.stdout: bdev enable discard = True 2024-03-31T15:14:53.822 INFO:teuthology.orchestra.run.smithi089.stdout: bluestore allocator = bitmap 2024-03-31T15:14:53.822 INFO:teuthology.orchestra.run.smithi089.stdout: bluestore block size = 96636764160 2024-03-31T15:14:53.822 INFO:teuthology.orchestra.run.smithi089.stdout: bluestore fsck on mount = True 2024-03-31T15:14:53.823 INFO:teuthology.orchestra.run.smithi089.stdout: debug bluefs = 1/20 2024-03-31T15:14:53.823 INFO:teuthology.orchestra.run.smithi089.stdout: debug bluestore = 1/20 2024-03-31T15:14:53.823 INFO:teuthology.orchestra.run.smithi089.stdout: debug ms = 1 2024-03-31T15:14:53.823 INFO:teuthology.orchestra.run.smithi089.stdout: debug osd = 20 2024-03-31T15:14:53.823 INFO:teuthology.orchestra.run.smithi089.stdout: debug rocksdb = 4/10 2024-03-31T15:14:53.823 INFO:teuthology.orchestra.run.smithi089.stdout: mon osd backfillfull_ratio = 0.85 2024-03-31T15:14:53.823 INFO:teuthology.orchestra.run.smithi089.stdout: mon osd full ratio = 0.9 2024-03-31T15:14:53.823 INFO:teuthology.orchestra.run.smithi089.stdout: mon osd nearfull ratio = 0.8 2024-03-31T15:14:53.823 INFO:teuthology.orchestra.run.smithi089.stdout: osd failsafe full ratio = 0.95 2024-03-31T15:14:53.823 INFO:teuthology.orchestra.run.smithi089.stdout: osd objectstore = bluestore 2024-03-31T15:14:53.823 INFO:teuthology.orchestra.run.smithi089.stdout: osd op complaint time = 180 2024-03-31T15:14:53.823 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-31T15:14:53.823 INFO:teuthology.orchestra.run.smithi089.stdout:[mgr] 2024-03-31T15:14:53.823 INFO:teuthology.orchestra.run.smithi089.stdout: debug ms = 1 2024-03-31T15:14:53.823 INFO:teuthology.orchestra.run.smithi089.stdout: debug mgr = 20 2024-03-31T15:14:53.824 INFO:teuthology.orchestra.run.smithi089.stdout: debug mon = 20 2024-03-31T15:14:53.824 INFO:teuthology.orchestra.run.smithi089.stdout: debug auth = 20 2024-03-31T15:14:53.824 INFO:teuthology.orchestra.run.smithi089.stdout: mon reweight min pgs per osd = 4 2024-03-31T15:14:53.824 INFO:teuthology.orchestra.run.smithi089.stdout: mon reweight min bytes per osd = 10 2024-03-31T15:14:53.824 INFO:teuthology.orchestra.run.smithi089.stdout: mgr/telemetry/nag = false 2024-03-31T15:14:53.824 INFO:teuthology.orchestra.run.smithi089.stdout: debug client = 20 2024-03-31T15:14:53.824 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-31T15:14:53.824 INFO:teuthology.orchestra.run.smithi089.stdout:[mon] 2024-03-31T15:14:53.824 INFO:teuthology.orchestra.run.smithi089.stdout: debug ms = 1 2024-03-31T15:14:53.824 INFO:teuthology.orchestra.run.smithi089.stdout: debug mon = 20 2024-03-31T15:14:53.824 INFO:teuthology.orchestra.run.smithi089.stdout: debug paxos = 20 2024-03-31T15:14:53.824 INFO:teuthology.orchestra.run.smithi089.stdout: debug auth = 20 2024-03-31T15:14:53.824 INFO:teuthology.orchestra.run.smithi089.stdout: mon data avail warn = 5 2024-03-31T15:14:53.824 INFO:teuthology.orchestra.run.smithi089.stdout: mon mgr mkfs grace = 240 2024-03-31T15:14:53.825 INFO:teuthology.orchestra.run.smithi089.stdout: mon reweight min pgs per osd = 4 2024-03-31T15:14:53.825 INFO:teuthology.orchestra.run.smithi089.stdout: mon osd reporter subtree level = osd 2024-03-31T15:14:53.825 INFO:teuthology.orchestra.run.smithi089.stdout: mon osd prime pg temp = true 2024-03-31T15:14:53.825 INFO:teuthology.orchestra.run.smithi089.stdout: mon reweight min bytes per osd = 10 2024-03-31T15:14:53.825 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-31T15:14:53.825 INFO:teuthology.orchestra.run.smithi089.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-31T15:14:53.825 INFO:teuthology.orchestra.run.smithi089.stdout: auth mon ticket ttl = 660 # 11m 2024-03-31T15:14:53.825 INFO:teuthology.orchestra.run.smithi089.stdout: auth service ticket ttl = 240 # 4m 2024-03-31T15:14:53.825 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-31T15:14:53.825 INFO:teuthology.orchestra.run.smithi089.stdout: # don't complain about insecure global_id in the test suite 2024-03-31T15:14:53.825 INFO:teuthology.orchestra.run.smithi089.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-31T15:14:53.825 INFO:teuthology.orchestra.run.smithi089.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-31T15:14:53.825 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-31T15:14:53.825 INFO:teuthology.orchestra.run.smithi089.stdout: # 1m isn't quite enough 2024-03-31T15:14:53.826 INFO:teuthology.orchestra.run.smithi089.stdout: mon_down_mkfs_grace = 2m 2024-03-31T15:14:53.826 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-31T15:14:53.826 INFO:teuthology.orchestra.run.smithi089.stdout: mon_warn_on_filestore_osds = false 2024-03-31T15:14:53.826 INFO:teuthology.orchestra.run.smithi089.stdout: mon down mkfs grace = 300 2024-03-31T15:14:53.826 INFO:teuthology.orchestra.run.smithi089.stdout: mon op complaint time = 120 2024-03-31T15:14:53.826 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-31T15:14:53.826 INFO:teuthology.orchestra.run.smithi089.stdout:[client] 2024-03-31T15:14:53.826 INFO:teuthology.orchestra.run.smithi089.stdout: rgw cache enabled = true 2024-03-31T15:14:53.826 INFO:teuthology.orchestra.run.smithi089.stdout: rgw enable ops log = true 2024-03-31T15:14:53.826 INFO:teuthology.orchestra.run.smithi089.stdout: rgw enable usage log = true 2024-03-31T15:14:53.826 INFO:teuthology.orchestra.run.smithi089.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-31T15:14:53.826 INFO:teuthology.orchestra.run.smithi089.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-31T15:14:53.826 INFO:teuthology.orchestra.run.smithi089.stdout: client mount timeout = 600 2024-03-31T15:14:53.826 INFO:teuthology.orchestra.run.smithi089.stdout: debug client = 20 2024-03-31T15:14:53.827 INFO:teuthology.orchestra.run.smithi089.stdout: debug ms = 1 2024-03-31T15:14:53.827 INFO:teuthology.orchestra.run.smithi089.stdout: pid file = "" 2024-03-31T15:14:53.827 INFO:teuthology.orchestra.run.smithi089.stdout: rados mon op timeout = 900 2024-03-31T15:14:53.827 INFO:teuthology.orchestra.run.smithi089.stdout: rados osd op timeout = 900 2024-03-31T15:14:53.827 INFO:teuthology.orchestra.run.smithi089.stdout:[mon.a] 2024-03-31T15:14:53.827 INFO:teuthology.orchestra.run.smithi089.stdout:[mon.b] 2024-03-31T15:14:53.827 INFO:teuthology.orchestra.run.smithi089.stdout:[mon.c] 2024-03-31T15:14:53.827 INFO:teuthology.orchestra.run.smithi089.stdout:[mds.a] 2024-03-31T15:14:53.827 INFO:teuthology.orchestra.run.smithi089.stdout:[mds.c] 2024-03-31T15:14:53.827 INFO:teuthology.orchestra.run.smithi089.stdout:[mds.b] 2024-03-31T15:14:53.827 INFO:teuthology.orchestra.run.smithi089.stdout:[mds.d] 2024-03-31T15:14:53.827 INFO:teuthology.orchestra.run.smithi089.stdout:[mds] 2024-03-31T15:14:53.827 INFO:teuthology.orchestra.run.smithi089.stdout: debug mds = 20 2024-03-31T15:14:53.827 INFO:teuthology.orchestra.run.smithi089.stdout: debug mds balancer = 20 2024-03-31T15:14:53.827 INFO:teuthology.orchestra.run.smithi089.stdout: debug ms = 1 2024-03-31T15:14:53.828 INFO:teuthology.orchestra.run.smithi089.stdout: mds debug frag = True 2024-03-31T15:14:53.828 INFO:teuthology.orchestra.run.smithi089.stdout: mds debug scatterstat = True 2024-03-31T15:14:53.828 INFO:teuthology.orchestra.run.smithi089.stdout: mds op complaint time = 180 2024-03-31T15:14:53.828 INFO:teuthology.orchestra.run.smithi089.stdout: mds verify scatter = True 2024-03-31T15:14:53.828 INFO:teuthology.orchestra.run.smithi089.stdout: osd op complaint time = 180 2024-03-31T15:14:53.828 INFO:teuthology.orchestra.run.smithi089.stdout: rados mon op timeout = 900 2024-03-31T15:14:53.828 INFO:teuthology.orchestra.run.smithi089.stdout: rados osd op timeout = 900 2024-03-31T15:14:53.829 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs joinable true 2024-03-31T15:14:54.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:54.021+0000 7f6915c00640 1 Processor -- start 2024-03-31T15:14:54.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:54.021+0000 7f6915c00640 1 -- start start 2024-03-31T15:14:54.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:54.021+0000 7f6915c00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f691006f920 0x7f691006fd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:54.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:54.021+0000 7f6915c00640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6910070240 con 0x7f69100cf150 2024-03-31T15:14:54.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:54.021+0000 7f6915c00640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f691015ca70 con 0x7f69100d0a60 2024-03-31T15:14:54.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:54.021+0000 7f6915c00640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f691013f670 con 0x7f691006f920 2024-03-31T15:14:54.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:54.021+0000 7f690f400640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f69100cf150 0x7f69101419c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:55512/0 (socket says 172.21.15.16:55512) 2024-03-31T15:14:54.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:54.021+0000 7f690f400640 1 -- 172.21.15.16:0/2152668999 learned_addr learned my addr 172.21.15.16:0/2152668999 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:14:54.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:54.021+0000 7f690ea00640 1 --2- 172.21.15.16:0/2152668999 >> v2:172.21.15.89:3300/0 conn(0x7f691006f920 0x7f691006fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:54.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:54.021+0000 7f690e000640 1 -- 172.21.15.16:0/2152668999 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3945194649 0 0) 0x7f6910070240 con 0x7f69100cf150 2024-03-31T15:14:54.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:54.021+0000 7f690e000640 1 -- 172.21.15.16:0/2152668999 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f68f4003530 con 0x7f69100cf150 2024-03-31T15:14:54.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:54.021+0000 7f690e000640 1 -- 172.21.15.16:0/2152668999 <== mon.1 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 589253244 0 0) 0x7f691015ca70 con 0x7f69100d0a60 2024-03-31T15:14:54.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:54.021+0000 7f690e000640 1 -- 172.21.15.16:0/2152668999 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6910070240 con 0x7f69100d0a60 2024-03-31T15:14:54.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:54.025+0000 7f690e000640 1 -- 172.21.15.16:0/2152668999 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2872015036 0 0) 0x7f68f4003530 con 0x7f69100cf150 2024-03-31T15:14:54.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:54.025+0000 7f690ea00640 1 -- 172.21.15.16:0/2152668999 >> v1:172.21.15.89:6789/0 conn(0x7f69100d0a60 legacy=0x7f69100cea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:54.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:54.025+0000 7f690ea00640 1 -- 172.21.15.16:0/2152668999 >> v1:172.21.15.16:6789/0 conn(0x7f69100cf150 legacy=0x7f69101419c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:54.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:54.025+0000 7f690ea00640 1 -- 172.21.15.16:0/2152668999 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f691015cd50 con 0x7f691006f920 2024-03-31T15:14:54.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:54.025+0000 7f690ea00640 1 --2- 172.21.15.16:0/2152668999 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f691006f920 0x7f691006fd00 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f6900009740 tx=0x7f690001c270 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8a57d1486d7cf1e8 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:54.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:54.025+0000 7f690e000640 1 -- 172.21.15.16:0/2152668999 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69000b0070 con 0x7f691006f920 2024-03-31T15:14:54.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:54.025+0000 7f690e000640 1 -- 172.21.15.16:0/2152668999 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f69000ab400 con 0x7f691006f920 2024-03-31T15:14:54.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:54.025+0000 7f690e000640 1 -- 172.21.15.16:0/2152668999 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6908001a00 con 0x7f691006f920 2024-03-31T15:14:54.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:54.025+0000 7f6915c00640 1 -- 172.21.15.16:0/2152668999 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f691006f920 msgr2=0x7f691006fd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:54.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:54.025+0000 7f6915c00640 1 --2- 172.21.15.16:0/2152668999 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f691006f920 0x7f691006fd00 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f6900009740 tx=0x7f690001c270 comp rx=0 tx=0).stop 2024-03-31T15:14:54.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:54.025+0000 7f6915c00640 1 -- 172.21.15.16:0/2152668999 shutdown_connections 2024-03-31T15:14:54.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:54.025+0000 7f6915c00640 1 --2- 172.21.15.16:0/2152668999 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f691006f920 0x7f691006fd00 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:54.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:54.025+0000 7f6915c00640 1 -- 172.21.15.16:0/2152668999 >> 172.21.15.16:0/2152668999 conn(0x7f6910067930 msgr2=0x7f69100b3ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:14:54.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:54.025+0000 7f6915c00640 1 -- 172.21.15.16:0/2152668999 shutdown_connections 2024-03-31T15:14:54.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:54.025+0000 7f6915c00640 1 -- 172.21.15.16:0/2152668999 wait complete. 2024-03-31T15:14:54.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:54.029+0000 7f6915c00640 1 Processor -- start 2024-03-31T15:14:54.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:54.029+0000 7f6915c00640 1 -- start start 2024-03-31T15:14:54.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:54.029+0000 7f6915c00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f691006f920 0x7f691013e430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:54.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:54.029+0000 7f6915c00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f69100cf150 0x7f69101f7790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:54.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:54.029+0000 7f6915c00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f69100d0a60 0x7f691015e260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:54.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:54.029+0000 7f6915c00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f691016a240 con 0x7f69100cf150 2024-03-31T15:14:54.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:54.029+0000 7f6915c00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f691016a0c0 con 0x7f691006f920 2024-03-31T15:14:54.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:54.029+0000 7f6915c00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f691016a3c0 con 0x7f69100d0a60 2024-03-31T15:14:54.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:54.029+0000 7f6907e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f69100cf150 0x7f69101f7790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:54.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:54.029+0000 7f6907e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f69100cf150 0x7f69101f7790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55718/0 (socket says 172.21.15.16:55718) 2024-03-31T15:14:54.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:54.029+0000 7f6907e00640 1 -- 172.21.15.16:0/2222694387 learned_addr learned my addr 172.21.15.16:0/2222694387 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:14:54.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:54.029+0000 7f690ea00640 1 --2- 172.21.15.16:0/2222694387 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f691006f920 0x7f691013e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:54.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:54.029+0000 7f690f400640 1 --2- 172.21.15.16:0/2222694387 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f69100d0a60 0x7f691015e260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:54.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:54.029+0000 7f6907e00640 1 -- 172.21.15.16:0/2222694387 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f69100d0a60 msgr2=0x7f691015e260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:54.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:54.029+0000 7f6907e00640 1 --2- 172.21.15.16:0/2222694387 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f69100d0a60 0x7f691015e260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:54.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:54.029+0000 7f6907e00640 1 -- 172.21.15.16:0/2222694387 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f691006f920 msgr2=0x7f691013e430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:54.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:54.029+0000 7f6907e00640 1 --2- 172.21.15.16:0/2222694387 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f691006f920 0x7f691013e430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:54.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:54.029+0000 7f6907e00640 1 -- 172.21.15.16:0/2222694387 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69101f7cd0 con 0x7f69100cf150 2024-03-31T15:14:54.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:54.029+0000 7f690ea00640 1 --2- 172.21.15.16:0/2222694387 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f691006f920 0x7f691013e430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:14:54.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:54.029+0000 7f6907e00640 1 --2- 172.21.15.16:0/2222694387 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f69100cf150 0x7f69101f7790 secure :-1 s=READY pgs=1823 cs=0 l=1 rev1=1 crypto rx=0x7f68f800c800 tx=0x7f68f800ccc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:54.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:54.029+0000 7f6906a00640 1 -- 172.21.15.16:0/2222694387 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68f8005480 con 0x7f69100cf150 2024-03-31T15:14:54.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:54.033+0000 7f6906a00640 1 -- 172.21.15.16:0/2222694387 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f68f800a640 con 0x7f69100cf150 2024-03-31T15:14:54.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:54.033+0000 7f6906a00640 1 -- 172.21.15.16:0/2222694387 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68f800aa30 con 0x7f69100cf150 2024-03-31T15:14:54.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:54.033+0000 7f6915c00640 1 -- 172.21.15.16:0/2222694387 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f691015eb40 con 0x7f69100cf150 2024-03-31T15:14:54.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:54.033+0000 7f6915c00640 1 -- 172.21.15.16:0/2222694387 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f691015f100 con 0x7f69100cf150 2024-03-31T15:14:54.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:54.033+0000 7f6915c00640 1 -- 172.21.15.16:0/2222694387 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6910069950 con 0x7f69100cf150 2024-03-31T15:14:54.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:54.033+0000 7f6906a00640 1 -- 172.21.15.16:0/2222694387 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f68f800abd0 con 0x7f69100cf150 2024-03-31T15:14:54.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:54.033+0000 7f6906a00640 1 --2- 172.21.15.16:0/2222694387 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f68ec068380 0x7f68ec06a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:54.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:54.033+0000 7f6906a00640 1 -- 172.21.15.16:0/2222694387 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 5939+0+0 (secure 0 0 0) 0x7f68f8086910 con 0x7f69100cf150 2024-03-31T15:14:54.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:54.033+0000 7f690ea00640 1 --2- 172.21.15.16:0/2222694387 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f68ec068380 0x7f68ec06a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:54.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:54.037+0000 7f690ea00640 1 --2- 172.21.15.16:0/2222694387 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f68ec068380 0x7f68ec06a840 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7f69000ad000 tx=0x7f69000a6000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:54.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:54.037+0000 7f6906a00640 1 -- 172.21.15.16:0/2222694387 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f68f8056780 con 0x7f69100cf150 2024-03-31T15:14:54.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:54.241+0000 7f6915c00640 1 -- 172.21.15.16:0/2222694387 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"} v 0) -- 0x7f691006c730 con 0x7f69100cf150 2024-03-31T15:14:55.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:55.121+0000 7f6906a00640 1 -- 172.21.15.16:0/2222694387 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]=0 cephfs marked joinable; MDS may join as newly active. v108) ==== 162+0+0 (secure 0 0 0) 0x7f68f805aa80 con 0x7f69100cf150 2024-03-31T15:14:55.114 INFO:teuthology.orchestra.run.smithi016.stderr:cephfs marked joinable; MDS may join as newly active. 2024-03-31T15:14:55.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:55.121+0000 7f6915c00640 1 -- 172.21.15.16:0/2222694387 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f68ec068380 msgr2=0x7f68ec06a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:55.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:55.121+0000 7f6915c00640 1 --2- 172.21.15.16:0/2222694387 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f68ec068380 0x7f68ec06a840 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7f69000ad000 tx=0x7f69000a6000 comp rx=0 tx=0).stop 2024-03-31T15:14:55.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:55.121+0000 7f6915c00640 1 -- 172.21.15.16:0/2222694387 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f69100cf150 msgr2=0x7f69101f7790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:55.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:55.121+0000 7f6915c00640 1 --2- 172.21.15.16:0/2222694387 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f69100cf150 0x7f69101f7790 secure :-1 s=READY pgs=1823 cs=0 l=1 rev1=1 crypto rx=0x7f68f800c800 tx=0x7f68f800ccc0 comp rx=0 tx=0).stop 2024-03-31T15:14:55.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:55.121+0000 7f6915c00640 1 -- 172.21.15.16:0/2222694387 shutdown_connections 2024-03-31T15:14:55.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:55.121+0000 7f6915c00640 1 --2- 172.21.15.16:0/2222694387 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f68ec068380 0x7f68ec06a840 unknown :-1 s=CLOSED pgs=766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:55.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:55.121+0000 7f6915c00640 1 --2- 172.21.15.16:0/2222694387 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f69100d0a60 0x7f691015e260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:55.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:55.121+0000 7f6915c00640 1 --2- 172.21.15.16:0/2222694387 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f69100cf150 0x7f69101f7790 unknown :-1 s=CLOSED pgs=1823 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:55.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:55.121+0000 7f6915c00640 1 --2- 172.21.15.16:0/2222694387 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f691006f920 0x7f691013e430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:55.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:55.121+0000 7f6915c00640 1 -- 172.21.15.16:0/2222694387 >> 172.21.15.16:0/2222694387 conn(0x7f6910067930 msgr2=0x7f69100d0ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:14:55.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:55.121+0000 7f6915c00640 1 -- 172.21.15.16:0/2222694387 shutdown_connections 2024-03-31T15:14:55.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:55.121+0000 7f6915c00640 1 -- 172.21.15.16:0/2222694387 wait complete. 2024-03-31T15:14:55.145 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-31T15:14:55.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:55.297+0000 7f7e5fc00640 1 Processor -- start 2024-03-31T15:14:55.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:55.297+0000 7f7e5fc00640 1 -- start start 2024-03-31T15:14:55.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:55.297+0000 7f7e5fc00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f7e5806dbc0 0x7f7e5815f650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:55.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:55.297+0000 7f7e5fc00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f7e580cb440 0x7f7e5806f920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:55.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:55.297+0000 7f7e5fc00640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7e580cbe90 con 0x7f7e580cb8c0 2024-03-31T15:14:55.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:55.297+0000 7f7e5fc00640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f7e5815fb90 con 0x7f7e5806dbc0 2024-03-31T15:14:55.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:55.297+0000 7f7e5fc00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f7e5815f1c0 con 0x7f7e580cb440 2024-03-31T15:14:55.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:55.297+0000 7f7e4fe00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f7e580cb440 0x7f7e5806f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:55.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:55.297+0000 7f7e4fe00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f7e580cb440 0x7f7e5806f920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3301/0 says I am v2:172.21.15.16:56538/0 (socket says 172.21.15.16:56538) 2024-03-31T15:14:55.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:55.297+0000 7f7e5d200640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f7e5806dbc0 0x7f7e5815f650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:55.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:55.297+0000 7f7e4fe00640 1 -- 172.21.15.16:0/3773414486 learned_addr learned my addr 172.21.15.16:0/3773414486 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:14:55.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:55.297+0000 7f7e5dc00640 1 --1- 172.21.15.16:0/3773414486 >> v1:172.21.15.89:6789/0 conn(0x7f7e580cb8c0 0x7f7e58162180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.16:49846/0 (socket says 172.21.15.16:49846) 2024-03-31T15:14:55.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:55.297+0000 7f7e4fe00640 1 -- 172.21.15.16:0/3773414486 >> v2:172.21.15.89:3300/0 conn(0x7f7e5806dbc0 msgr2=0x7f7e5815f650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:14:55.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:55.297+0000 7f7e4fe00640 1 --2- 172.21.15.16:0/3773414486 >> v2:172.21.15.89:3300/0 conn(0x7f7e5806dbc0 0x7f7e5815f650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:55.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:55.297+0000 7f7e4fe00640 1 -- 172.21.15.16:0/3773414486 >> v1:172.21.15.89:6789/0 conn(0x7f7e580cb8c0 legacy=0x7f7e58162180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:55.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:55.297+0000 7f7e4fe00640 1 -- 172.21.15.16:0/3773414486 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e580cc170 con 0x7f7e580cb440 2024-03-31T15:14:55.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:55.301+0000 7f7e4fe00640 1 --2- 172.21.15.16:0/3773414486 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f7e580cb440 0x7f7e5806f920 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f7e40009870 tx=0x7f7e4001c4a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8fad907fd91fa3ec server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:55.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:55.301+0000 7f7e57e00640 1 -- 172.21.15.16:0/3773414486 <== mon.2 v2:172.21.15.89:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e400aa620 con 0x7f7e580cb440 2024-03-31T15:14:55.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:55.301+0000 7f7e57e00640 1 -- 172.21.15.16:0/3773414486 <== mon.2 v2:172.21.15.89:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7e400af070 con 0x7f7e580cb440 2024-03-31T15:14:55.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:55.301+0000 7f7e57e00640 1 -- 172.21.15.16:0/3773414486 <== mon.2 v2:172.21.15.89:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e400aa9e0 con 0x7f7e580cb440 2024-03-31T15:14:55.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:55.301+0000 7f7e5fc00640 1 -- 172.21.15.16:0/3773414486 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f7e580cb440 msgr2=0x7f7e5806f920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:55.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:55.301+0000 7f7e5fc00640 1 --2- 172.21.15.16:0/3773414486 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f7e580cb440 0x7f7e5806f920 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f7e40009870 tx=0x7f7e4001c4a0 comp rx=0 tx=0).stop 2024-03-31T15:14:55.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:55.301+0000 7f7e5fc00640 1 -- 172.21.15.16:0/3773414486 shutdown_connections 2024-03-31T15:14:55.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:55.301+0000 7f7e5fc00640 1 --2- 172.21.15.16:0/3773414486 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f7e580cb440 0x7f7e5806f920 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:55.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:55.301+0000 7f7e5fc00640 1 --2- 172.21.15.16:0/3773414486 >> v2:172.21.15.89:3300/0 conn(0x7f7e5806dbc0 0x7f7e5815f650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:55.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:55.301+0000 7f7e5fc00640 1 -- 172.21.15.16:0/3773414486 >> 172.21.15.16:0/3773414486 conn(0x7f7e58067330 msgr2=0x7f7e58067740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:14:55.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:55.301+0000 7f7e5fc00640 1 -- 172.21.15.16:0/3773414486 shutdown_connections 2024-03-31T15:14:55.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:55.301+0000 7f7e5fc00640 1 -- 172.21.15.16:0/3773414486 wait complete. 2024-03-31T15:14:55.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:55.305+0000 7f7e5fc00640 1 Processor -- start 2024-03-31T15:14:55.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:55.305+0000 7f7e5fc00640 1 -- start start 2024-03-31T15:14:55.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:55.305+0000 7f7e5fc00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f7e5806dbc0 0x7f7e58144630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:55.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:55.305+0000 7f7e5fc00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7e580cb440 0x7f7e5815a570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:55.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:55.305+0000 7f7e5fc00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7e580cb8c0 0x7f7e580b6a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:55.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:55.305+0000 7f7e5fc00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f7e58162ca0 con 0x7f7e580cb8c0 2024-03-31T15:14:55.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:55.305+0000 7f7e5fc00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f7e58070270 con 0x7f7e580cb440 2024-03-31T15:14:55.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:55.305+0000 7f7e5fc00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f7e5806c950 con 0x7f7e5806dbc0 2024-03-31T15:14:55.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:55.305+0000 7f7e5dc00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7e580cb8c0 0x7f7e580b6a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:55.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:55.305+0000 7f7e5d200640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f7e5806dbc0 0x7f7e58144630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:55.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:55.305+0000 7f7e4fe00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7e580cb440 0x7f7e5815a570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:55.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:55.305+0000 7f7e5d200640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f7e5806dbc0 0x7f7e58144630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3301/0 says I am v2:172.21.15.16:56552/0 (socket says 172.21.15.16:56552) 2024-03-31T15:14:55.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:55.305+0000 7f7e5d200640 1 -- 172.21.15.16:0/959956146 learned_addr learned my addr 172.21.15.16:0/959956146 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:14:55.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:55.305+0000 7f7e5d200640 1 -- 172.21.15.16:0/959956146 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7e580cb440 msgr2=0x7f7e5815a570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:55.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:55.305+0000 7f7e5d200640 1 --2- 172.21.15.16:0/959956146 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7e580cb440 0x7f7e5815a570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:55.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:55.305+0000 7f7e5d200640 1 -- 172.21.15.16:0/959956146 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7e580cb8c0 msgr2=0x7f7e580b6a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:55.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:55.305+0000 7f7e5d200640 1 --2- 172.21.15.16:0/959956146 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7e580cb8c0 0x7f7e580b6a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:55.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:55.305+0000 7f7e5d200640 1 -- 172.21.15.16:0/959956146 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e5815afa0 con 0x7f7e5806dbc0 2024-03-31T15:14:55.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:55.305+0000 7f7e5d200640 1 --2- 172.21.15.16:0/959956146 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f7e5806dbc0 0x7f7e58144630 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f7e4800e8a0 tx=0x7f7e4800eca0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:55.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:55.305+0000 7f7e55600640 1 -- 172.21.15.16:0/959956146 <== mon.2 v2:172.21.15.89:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e48017070 con 0x7f7e5806dbc0 2024-03-31T15:14:55.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:55.305+0000 7f7e55600640 1 -- 172.21.15.16:0/959956146 <== mon.2 v2:172.21.15.89:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7e4800c940 con 0x7f7e5806dbc0 2024-03-31T15:14:55.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:55.305+0000 7f7e55600640 1 -- 172.21.15.16:0/959956146 <== mon.2 v2:172.21.15.89:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e4800ccf0 con 0x7f7e5806dbc0 2024-03-31T15:14:55.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:55.305+0000 7f7e5fc00640 1 -- 172.21.15.16:0/959956146 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7e5815b280 con 0x7f7e5806dbc0 2024-03-31T15:14:55.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:55.305+0000 7f7e5fc00640 1 -- 172.21.15.16:0/959956146 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f7e5815b840 con 0x7f7e5806dbc0 2024-03-31T15:14:55.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:55.309+0000 7f7e5fc00640 1 -- 172.21.15.16:0/959956146 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7e58069950 con 0x7f7e5806dbc0 2024-03-31T15:14:55.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:55.309+0000 7f7e55600640 1 -- 172.21.15.16:0/959956146 <== mon.2 v2:172.21.15.89:3301/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f7e4801a070 con 0x7f7e5806dbc0 2024-03-31T15:14:55.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:55.309+0000 7f7e55600640 1 --2- 172.21.15.16:0/959956146 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f7e34068380 0x7f7e3406a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:55.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:55.309+0000 7f7e55600640 1 -- 172.21.15.16:0/959956146 <== mon.2 v2:172.21.15.89:3301/0 5 ==== osd_map(113..113 src has 1..113) ==== 5939+0+0 (secure 0 0 0) 0x7f7e48086d10 con 0x7f7e5806dbc0 2024-03-31T15:14:55.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:55.309+0000 7f7e4fe00640 1 --2- 172.21.15.16:0/959956146 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f7e34068380 0x7f7e3406a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:55.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:55.309+0000 7f7e4fe00640 1 --2- 172.21.15.16:0/959956146 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f7e34068380 0x7f7e3406a840 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f7e400ae000 tx=0x7f7e400a7000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:55.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:55.313+0000 7f7e55600640 1 -- 172.21.15.16:0/959956146 <== mon.2 v2:172.21.15.89:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f7e4808c020 con 0x7f7e5806dbc0 2024-03-31T15:14:55.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:55.517+0000 7f7e5fc00640 1 -- 172.21.15.16:0/959956146 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7e580cf380 con 0x7f7e5806dbc0 2024-03-31T15:14:55.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:55.517+0000 7f7e55600640 1 -- 172.21.15.16:0/959956146 <== mon.2 v2:172.21.15.89:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 109 v109) ==== 95+0+5178 (secure 0 0 0) 0x7f7e48056af0 con 0x7f7e5806dbc0 2024-03-31T15:14:55.510 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:14:55.510 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":109,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7631,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6836/1628897112","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":1628897112},{"type":"v1","addr":"172.21.15.89:6836","nonce":1628897112}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":94},{"gid":8746,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/1036724837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6835","nonce":1036724837},{"type":"v1","addr":"172.21.15.89:6837","nonce":1036724837}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":94},{"gid":9192,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6834/472139147","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":472139147},{"type":"v1","addr":"172.21.15.16:6834","nonce":472139147}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":94}],"filesystems":[{"mdsmap":{"epoch":109,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-31T15:14:08.652180+0000","modified":"2024-03-31T15:14:55.129474+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":113,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":9762},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9762":{"gid":9762,"name":"c","rank":0,"incarnation":109,"state":"up:replay","state_seq":1,"addr":"172.21.15.16:6835/2152452999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6833","nonce":2152452999},{"type":"v1","addr":"172.21.15.16:6835","nonce":2152452999}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":8}]} 2024-03-31T15:14:55.510 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 109 2024-03-31T15:14:55.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:55.517+0000 7f7e5fc00640 1 -- 172.21.15.16:0/959956146 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f7e34068380 msgr2=0x7f7e3406a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:55.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:55.517+0000 7f7e5fc00640 1 --2- 172.21.15.16:0/959956146 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f7e34068380 0x7f7e3406a840 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f7e400ae000 tx=0x7f7e400a7000 comp rx=0 tx=0).stop 2024-03-31T15:14:55.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:55.517+0000 7f7e5fc00640 1 -- 172.21.15.16:0/959956146 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f7e5806dbc0 msgr2=0x7f7e58144630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:55.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:55.517+0000 7f7e5fc00640 1 --2- 172.21.15.16:0/959956146 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f7e5806dbc0 0x7f7e58144630 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f7e4800e8a0 tx=0x7f7e4800eca0 comp rx=0 tx=0).stop 2024-03-31T15:14:55.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:55.517+0000 7f7e5fc00640 1 -- 172.21.15.16:0/959956146 shutdown_connections 2024-03-31T15:14:55.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:55.517+0000 7f7e5fc00640 1 --2- 172.21.15.16:0/959956146 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f7e34068380 0x7f7e3406a840 unknown :-1 s=CLOSED pgs=768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:55.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:55.517+0000 7f7e5fc00640 1 --2- 172.21.15.16:0/959956146 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7e580cb8c0 0x7f7e580b6a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:55.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:55.517+0000 7f7e5fc00640 1 --2- 172.21.15.16:0/959956146 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7e580cb440 0x7f7e5815a570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:55.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:55.517+0000 7f7e5fc00640 1 --2- 172.21.15.16:0/959956146 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f7e5806dbc0 0x7f7e58144630 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:55.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:55.517+0000 7f7e5fc00640 1 -- 172.21.15.16:0/959956146 >> 172.21.15.16:0/959956146 conn(0x7f7e58067330 msgr2=0x7f7e580d07c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:14:55.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:55.517+0000 7f7e5fc00640 1 -- 172.21.15.16:0/959956146 shutdown_connections 2024-03-31T15:14:55.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:55.517+0000 7f7e5fc00640 1 -- 172.21.15.16:0/959956146 wait complete. 2024-03-31T15:14:55.934 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 109, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-31T15:14:08.652180+0000', 'modified': '2024-03-31T15:14:55.129474+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 113, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 9762}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9762': {'gid': 9762, 'name': 'c', 'rank': 0, 'incarnation': 109, 'state': 'up:replay', 'state_seq': 1, 'addr': '172.21.15.16:6835/2152452999', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6833', 'nonce': 2152452999}, {'type': 'v1', 'addr': '172.21.15.16:6835', 'nonce': 2152452999}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [17], 'metadata_pool': 16, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-03-31T15:14:55.935 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9762:up:replay 2024-03-31T15:14:56.936 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-31T15:14:57.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:57.093+0000 7fa1d5400640 1 Processor -- start 2024-03-31T15:14:57.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:57.093+0000 7fa1d5400640 1 -- start start 2024-03-31T15:14:57.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:57.097+0000 7fa1d5400640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fa1d015b3d0 0x7fa1d00b1cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:57.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:57.097+0000 7fa1d5400640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fa1d015eee0 0x7fa1d00b2200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:57.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:57.097+0000 7fa1d5400640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fa1d00ce2f0 0x7fa1d00cc9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:57.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:57.097+0000 7fa1d5400640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7fa1d015b7b0 con 0x7fa1d015eee0 2024-03-31T15:14:57.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:57.097+0000 7fa1d5400640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7fa1d015f3c0 con 0x7fa1d00ce2f0 2024-03-31T15:14:57.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:57.097+0000 7fa1d5400640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fa1d015f6a0 con 0x7fa1d015b3d0 2024-03-31T15:14:57.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:57.097+0000 7fa1c5e00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fa1d015eee0 0x7fa1d00b2200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:57.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:57.097+0000 7fa1c5e00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fa1d015eee0 0x7fa1d00b2200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50122/0 (socket says 172.21.15.16:50122) 2024-03-31T15:14:57.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:57.097+0000 7fa1c5e00640 1 -- 172.21.15.16:0/1791268979 learned_addr learned my addr 172.21.15.16:0/1791268979 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:14:57.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:57.097+0000 7fa1cf200640 1 --2- 172.21.15.16:0/1791268979 >> v2:172.21.15.89:3300/0 conn(0x7fa1d00ce2f0 0x7fa1d00cc9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:57.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:57.097+0000 7fa1ce800640 1 --2- 172.21.15.16:0/1791268979 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fa1d015b3d0 0x7fa1d00b1cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:57.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:57.097+0000 7fa1c5e00640 1 -- 172.21.15.16:0/1791268979 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fa1d015b3d0 msgr2=0x7fa1d00b1cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:14:57.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:57.097+0000 7fa1c5e00640 1 --2- 172.21.15.16:0/1791268979 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fa1d015b3d0 0x7fa1d00b1cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:57.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:57.097+0000 7fa1c5e00640 1 -- 172.21.15.16:0/1791268979 >> v2:172.21.15.89:3300/0 conn(0x7fa1d00ce2f0 msgr2=0x7fa1d00cc9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:14:57.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:57.097+0000 7fa1c5e00640 1 --2- 172.21.15.16:0/1791268979 >> v2:172.21.15.89:3300/0 conn(0x7fa1d00ce2f0 0x7fa1d00cc9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:57.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:57.097+0000 7fa1c5e00640 1 -- 172.21.15.16:0/1791268979 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1d015f980 con 0x7fa1d015eee0 2024-03-31T15:14:57.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:57.097+0000 7fa1cf200640 1 --2- 172.21.15.16:0/1791268979 >> v2:172.21.15.89:3300/0 conn(0x7fa1d00ce2f0 0x7fa1d00cc9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:14:57.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:57.101+0000 7fa1c5e00640 1 --2- 172.21.15.16:0/1791268979 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa1d015eee0 0x7fa1d00b2200 secure :-1 s=READY pgs=1824 cs=0 l=1 rev1=1 crypto rx=0x7fa1d00b2df0 tx=0x7fa1b801c4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8cc4e16af04545de server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:57.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:57.101+0000 7fa1cde00640 1 -- 172.21.15.16:0/1791268979 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1b80a7070 con 0x7fa1d015eee0 2024-03-31T15:14:57.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:57.101+0000 7fa1cde00640 1 -- 172.21.15.16:0/1791268979 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa1b80b0070 con 0x7fa1d015eee0 2024-03-31T15:14:57.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:57.101+0000 7fa1cde00640 1 -- 172.21.15.16:0/1791268979 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1b80ab890 con 0x7fa1d015eee0 2024-03-31T15:14:57.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:57.101+0000 7fa1d5400640 1 -- 172.21.15.16:0/1791268979 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa1d015eee0 msgr2=0x7fa1d00b2200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:57.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:57.101+0000 7fa1d5400640 1 --2- 172.21.15.16:0/1791268979 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa1d015eee0 0x7fa1d00b2200 secure :-1 s=READY pgs=1824 cs=0 l=1 rev1=1 crypto rx=0x7fa1d00b2df0 tx=0x7fa1b801c4f0 comp rx=0 tx=0).stop 2024-03-31T15:14:57.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:57.101+0000 7fa1d5400640 1 -- 172.21.15.16:0/1791268979 shutdown_connections 2024-03-31T15:14:57.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:57.101+0000 7fa1d5400640 1 --2- 172.21.15.16:0/1791268979 >> v2:172.21.15.89:3300/0 conn(0x7fa1d00ce2f0 0x7fa1d00cc9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:57.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:57.101+0000 7fa1d5400640 1 --2- 172.21.15.16:0/1791268979 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa1d015eee0 0x7fa1d00b2200 unknown :-1 s=CLOSED pgs=1824 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:57.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:57.101+0000 7fa1d5400640 1 --2- 172.21.15.16:0/1791268979 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fa1d015b3d0 0x7fa1d00b1cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:57.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:57.101+0000 7fa1d5400640 1 -- 172.21.15.16:0/1791268979 >> 172.21.15.16:0/1791268979 conn(0x7fa1d006b620 msgr2=0x7fa1d006ba30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:14:57.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:57.101+0000 7fa1d5400640 1 -- 172.21.15.16:0/1791268979 shutdown_connections 2024-03-31T15:14:57.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:57.101+0000 7fa1d5400640 1 -- 172.21.15.16:0/1791268979 wait complete. 2024-03-31T15:14:57.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:57.101+0000 7fa1d5400640 1 Processor -- start 2024-03-31T15:14:57.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:57.101+0000 7fa1d5400640 1 -- start start 2024-03-31T15:14:57.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:57.101+0000 7fa1d5400640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa1d00ce2f0 0x7fa1d014c4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:57.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:57.101+0000 7fa1d5400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa1d015b3d0 0x7fa1d0154160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:57.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:57.101+0000 7fa1d5400640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fa1d015eee0 0x7fa1d01546a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:57.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:57.101+0000 7fa1d5400640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fa1d0071e70 con 0x7fa1d015b3d0 2024-03-31T15:14:57.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:57.101+0000 7fa1d5400640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fa1d0071cf0 con 0x7fa1d00ce2f0 2024-03-31T15:14:57.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:57.101+0000 7fa1d5400640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fa1d0071ff0 con 0x7fa1d015eee0 2024-03-31T15:14:57.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:57.101+0000 7fa1c5e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa1d015b3d0 0x7fa1d0154160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:57.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:57.105+0000 7fa1c5e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa1d015b3d0 0x7fa1d0154160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50138/0 (socket says 172.21.15.16:50138) 2024-03-31T15:14:57.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:57.105+0000 7fa1c5e00640 1 -- 172.21.15.16:0/726735297 learned_addr learned my addr 172.21.15.16:0/726735297 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:14:57.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:57.105+0000 7fa1cf200640 1 --2- 172.21.15.16:0/726735297 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fa1d015eee0 0x7fa1d01546a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:57.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:57.105+0000 7fa1ce800640 1 --2- 172.21.15.16:0/726735297 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa1d00ce2f0 0x7fa1d014c4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:57.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:57.105+0000 7fa1c5e00640 1 -- 172.21.15.16:0/726735297 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fa1d015eee0 msgr2=0x7fa1d01546a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:57.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:57.105+0000 7fa1c5e00640 1 --2- 172.21.15.16:0/726735297 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fa1d015eee0 0x7fa1d01546a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:57.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:57.105+0000 7fa1c5e00640 1 -- 172.21.15.16:0/726735297 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa1d00ce2f0 msgr2=0x7fa1d014c4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:57.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:57.105+0000 7fa1c5e00640 1 --2- 172.21.15.16:0/726735297 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa1d00ce2f0 0x7fa1d014c4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:57.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:57.105+0000 7fa1c5e00640 1 -- 172.21.15.16:0/726735297 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1d01533a0 con 0x7fa1d015b3d0 2024-03-31T15:14:57.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:57.105+0000 7fa1c5e00640 1 --2- 172.21.15.16:0/726735297 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa1d015b3d0 0x7fa1d0154160 secure :-1 s=READY pgs=1825 cs=0 l=1 rev1=1 crypto rx=0x7fa1b801c9d0 tx=0x7fa1b801ad30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:57.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:57.105+0000 7fa1c7400640 1 -- 172.21.15.16:0/726735297 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1b80be5f0 con 0x7fa1d015b3d0 2024-03-31T15:14:57.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:57.105+0000 7fa1c7400640 1 -- 172.21.15.16:0/726735297 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa1b80be790 con 0x7fa1d015b3d0 2024-03-31T15:14:57.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:57.105+0000 7fa1c7400640 1 -- 172.21.15.16:0/726735297 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1b80beb80 con 0x7fa1d015b3d0 2024-03-31T15:14:57.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:57.105+0000 7fa1d5400640 1 -- 172.21.15.16:0/726735297 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa1d0153680 con 0x7fa1d015b3d0 2024-03-31T15:14:57.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:57.105+0000 7fa1d5400640 1 -- 172.21.15.16:0/726735297 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa1d01485f0 con 0x7fa1d015b3d0 2024-03-31T15:14:57.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:57.105+0000 7fa1d5400640 1 -- 172.21.15.16:0/726735297 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa1d0069950 con 0x7fa1d015b3d0 2024-03-31T15:14:57.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:57.105+0000 7fa1c7400640 1 -- 172.21.15.16:0/726735297 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fa1b80b43f0 con 0x7fa1d015b3d0 2024-03-31T15:14:57.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:57.105+0000 7fa1c7400640 1 --2- 172.21.15.16:0/726735297 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fa1ac068380 0x7fa1ac06a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:57.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:57.109+0000 7fa1c7400640 1 -- 172.21.15.16:0/726735297 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 5939+0+0 (secure 0 0 0) 0x7fa1b81217d0 con 0x7fa1d015b3d0 2024-03-31T15:14:57.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:57.109+0000 7fa1ce800640 1 --2- 172.21.15.16:0/726735297 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fa1ac068380 0x7fa1ac06a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:57.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:57.109+0000 7fa1ce800640 1 --2- 172.21.15.16:0/726735297 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fa1ac068380 0x7fa1ac06a840 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7fa1c0003b90 tx=0x7fa1c00098d0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:57.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:57.113+0000 7fa1c7400640 1 -- 172.21.15.16:0/726735297 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fa1b80c4060 con 0x7fa1d015b3d0 2024-03-31T15:14:57.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:57.317+0000 7fa1d5400640 1 -- 172.21.15.16:0/726735297 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa1d00cdea0 con 0x7fa1d015b3d0 2024-03-31T15:14:57.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:57.317+0000 7fa1c7400640 1 -- 172.21.15.16:0/726735297 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 111 v111) ==== 95+0+5178 (secure 0 0 0) 0x7fa1b8016110 con 0x7fa1d015b3d0 2024-03-31T15:14:57.312 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:14:57.312 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":111,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7631,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6836/1628897112","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":1628897112},{"type":"v1","addr":"172.21.15.89:6836","nonce":1628897112}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":94},{"gid":8746,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/1036724837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6835","nonce":1036724837},{"type":"v1","addr":"172.21.15.89:6837","nonce":1036724837}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":94},{"gid":9192,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6834/472139147","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":472139147},{"type":"v1","addr":"172.21.15.16:6834","nonce":472139147}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":94}],"filesystems":[{"mdsmap":{"epoch":111,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-31T15:14:08.652180+0000","modified":"2024-03-31T15:14:56.135324+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":113,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":9762},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9762":{"gid":9762,"name":"c","rank":0,"incarnation":109,"state":"up:rejoin","state_seq":3,"addr":"172.21.15.16:6835/2152452999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6833","nonce":2152452999},{"type":"v1","addr":"172.21.15.16:6835","nonce":2152452999}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":8}]} 2024-03-31T15:14:57.312 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 111 2024-03-31T15:14:57.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:57.321+0000 7fa1d5400640 1 -- 172.21.15.16:0/726735297 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fa1ac068380 msgr2=0x7fa1ac06a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:57.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:57.321+0000 7fa1d5400640 1 --2- 172.21.15.16:0/726735297 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fa1ac068380 0x7fa1ac06a840 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7fa1c0003b90 tx=0x7fa1c00098d0 comp rx=0 tx=0).stop 2024-03-31T15:14:57.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:57.321+0000 7fa1d5400640 1 -- 172.21.15.16:0/726735297 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa1d015b3d0 msgr2=0x7fa1d0154160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:57.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:57.321+0000 7fa1d5400640 1 --2- 172.21.15.16:0/726735297 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa1d015b3d0 0x7fa1d0154160 secure :-1 s=READY pgs=1825 cs=0 l=1 rev1=1 crypto rx=0x7fa1b801c9d0 tx=0x7fa1b801ad30 comp rx=0 tx=0).stop 2024-03-31T15:14:57.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:57.321+0000 7fa1d5400640 1 -- 172.21.15.16:0/726735297 shutdown_connections 2024-03-31T15:14:57.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:57.321+0000 7fa1d5400640 1 --2- 172.21.15.16:0/726735297 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fa1ac068380 0x7fa1ac06a840 unknown :-1 s=CLOSED pgs=769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:57.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:57.321+0000 7fa1d5400640 1 --2- 172.21.15.16:0/726735297 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fa1d015eee0 0x7fa1d01546a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:57.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:57.321+0000 7fa1d5400640 1 --2- 172.21.15.16:0/726735297 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa1d015b3d0 0x7fa1d0154160 unknown :-1 s=CLOSED pgs=1825 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:57.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:57.321+0000 7fa1d5400640 1 --2- 172.21.15.16:0/726735297 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa1d00ce2f0 0x7fa1d014c4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:57.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:57.321+0000 7fa1d5400640 1 -- 172.21.15.16:0/726735297 >> 172.21.15.16:0/726735297 conn(0x7fa1d006b620 msgr2=0x7fa1d0152480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:14:57.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:57.321+0000 7fa1d5400640 1 -- 172.21.15.16:0/726735297 shutdown_connections 2024-03-31T15:14:57.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:57.321+0000 7fa1d5400640 1 -- 172.21.15.16:0/726735297 wait complete. 2024-03-31T15:14:57.735 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 111, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-31T15:14:08.652180+0000', 'modified': '2024-03-31T15:14:56.135324+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 113, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 9762}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9762': {'gid': 9762, 'name': 'c', 'rank': 0, 'incarnation': 109, 'state': 'up:rejoin', 'state_seq': 3, 'addr': '172.21.15.16:6835/2152452999', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6833', 'nonce': 2152452999}, {'type': 'v1', 'addr': '172.21.15.16:6835', 'nonce': 2152452999}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [17], 'metadata_pool': 16, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-03-31T15:14:57.735 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9762:up:rejoin 2024-03-31T15:14:58.737 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-31T15:14:58.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:58.881+0000 7f687dc00640 1 Processor -- start 2024-03-31T15:14:58.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:58.885+0000 7f687dc00640 1 -- start start 2024-03-31T15:14:58.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:58.885+0000 7f687dc00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f68780728b0 0x7f68780b1cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:58.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:58.885+0000 7f687dc00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f68780b2200 0x7f687815cd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:58.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:58.885+0000 7f687dc00640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6878160e70 con 0x7f6878145d10 2024-03-31T15:14:58.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:58.885+0000 7f687dc00640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f68781602f0 con 0x7f68780b2200 2024-03-31T15:14:58.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:58.885+0000 7f687dc00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f68781605d0 con 0x7f68780728b0 2024-03-31T15:14:58.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:58.885+0000 7f6877400640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f68780b2200 0x7f687815cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:58.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:58.885+0000 7f6877400640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f68780b2200 0x7f687815cd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50140/0 (socket says 172.21.15.16:50140) 2024-03-31T15:14:58.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:58.885+0000 7f6877400640 1 -- 172.21.15.16:0/1854378595 learned_addr learned my addr 172.21.15.16:0/1854378595 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:14:58.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:58.885+0000 7f686e000640 1 --2- 172.21.15.16:0/1854378595 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f68780728b0 0x7f68780b1cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:58.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:58.885+0000 7f6877400640 1 -- 172.21.15.16:0/1854378595 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f68780728b0 msgr2=0x7f68780b1cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:14:58.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:58.885+0000 7f6877400640 1 --2- 172.21.15.16:0/1854378595 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f68780728b0 0x7f68780b1cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:58.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:58.885+0000 7f6877400640 1 -- 172.21.15.16:0/1854378595 >> v1:172.21.15.89:6789/0 conn(0x7f6878145d10 legacy=0x7f68780cf790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:58.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:58.885+0000 7f6877400640 1 -- 172.21.15.16:0/1854378595 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68781608b0 con 0x7f68780b2200 2024-03-31T15:14:58.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:58.889+0000 7f6877400640 1 --2- 172.21.15.16:0/1854378595 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f68780b2200 0x7f687815cd50 secure :-1 s=READY pgs=1826 cs=0 l=1 rev1=1 crypto rx=0x7f6870009e50 tx=0x7f687001ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc16c43484fe62cc server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:58.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:58.889+0000 7f6876000640 1 -- 172.21.15.16:0/1854378595 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68700a5700 con 0x7f68780b2200 2024-03-31T15:14:58.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:58.889+0000 7f6876000640 1 -- 172.21.15.16:0/1854378595 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f68700a58a0 con 0x7f68780b2200 2024-03-31T15:14:58.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:58.889+0000 7f6876000640 1 -- 172.21.15.16:0/1854378595 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68700a5c70 con 0x7f68780b2200 2024-03-31T15:14:58.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:58.889+0000 7f687dc00640 1 -- 172.21.15.16:0/1854378595 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f68780b2200 msgr2=0x7f687815cd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:58.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:58.889+0000 7f687dc00640 1 --2- 172.21.15.16:0/1854378595 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f68780b2200 0x7f687815cd50 secure :-1 s=READY pgs=1826 cs=0 l=1 rev1=1 crypto rx=0x7f6870009e50 tx=0x7f687001ca30 comp rx=0 tx=0).stop 2024-03-31T15:14:58.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:58.889+0000 7f687dc00640 1 -- 172.21.15.16:0/1854378595 shutdown_connections 2024-03-31T15:14:58.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:58.889+0000 7f687dc00640 1 --2- 172.21.15.16:0/1854378595 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f68780b2200 0x7f687815cd50 unknown :-1 s=CLOSED pgs=1826 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:58.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:58.889+0000 7f687dc00640 1 --2- 172.21.15.16:0/1854378595 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f68780728b0 0x7f68780b1cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:58.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:58.889+0000 7f687dc00640 1 -- 172.21.15.16:0/1854378595 >> 172.21.15.16:0/1854378595 conn(0x7f687806b620 msgr2=0x7f687806ba30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:14:58.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:58.889+0000 7f687dc00640 1 -- 172.21.15.16:0/1854378595 shutdown_connections 2024-03-31T15:14:58.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:58.889+0000 7f687dc00640 1 -- 172.21.15.16:0/1854378595 wait complete. 2024-03-31T15:14:58.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:58.889+0000 7f687dc00640 1 Processor -- start 2024-03-31T15:14:58.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:58.889+0000 7f687dc00640 1 -- start start 2024-03-31T15:14:58.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:58.893+0000 7f687dc00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f68780728b0 0x7f687813bd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:58.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:58.893+0000 7f687dc00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f68780b2200 0x7f687813ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:58.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:58.893+0000 7f687dc00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f6878145d10 0x7f68780d01b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:58.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:58.893+0000 7f687dc00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f6878072e10 con 0x7f68780b2200 2024-03-31T15:14:58.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:58.893+0000 7f687dc00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f6878072c90 con 0x7f68780728b0 2024-03-31T15:14:58.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:58.893+0000 7f687dc00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f68781690f0 con 0x7f6878145d10 2024-03-31T15:14:58.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:58.893+0000 7f686e000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f68780b2200 0x7f687813ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:58.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:58.893+0000 7f686e000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f68780b2200 0x7f687813ec80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50148/0 (socket says 172.21.15.16:50148) 2024-03-31T15:14:58.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:58.893+0000 7f686e000640 1 -- 172.21.15.16:0/2821729497 learned_addr learned my addr 172.21.15.16:0/2821729497 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:14:58.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:58.893+0000 7f6877400640 1 --2- 172.21.15.16:0/2821729497 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f6878145d10 0x7f68780d01b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:58.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:58.893+0000 7f6876a00640 1 --2- 172.21.15.16:0/2821729497 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f68780728b0 0x7f687813bd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:58.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:58.893+0000 7f686e000640 1 -- 172.21.15.16:0/2821729497 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f6878145d10 msgr2=0x7f68780d01b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:58.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:58.893+0000 7f686e000640 1 --2- 172.21.15.16:0/2821729497 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f6878145d10 0x7f68780d01b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:58.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:58.893+0000 7f686e000640 1 -- 172.21.15.16:0/2821729497 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f68780728b0 msgr2=0x7f687813bd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:58.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:58.893+0000 7f686e000640 1 --2- 172.21.15.16:0/2821729497 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f68780728b0 0x7f687813bd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:58.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:58.893+0000 7f686e000640 1 -- 172.21.15.16:0/2821729497 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f687813f4d0 con 0x7f68780b2200 2024-03-31T15:14:58.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:58.893+0000 7f6877400640 1 --2- 172.21.15.16:0/2821729497 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f6878145d10 0x7f68780d01b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:14:58.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:58.893+0000 7f686e000640 1 --2- 172.21.15.16:0/2821729497 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f68780b2200 0x7f687813ec80 secure :-1 s=READY pgs=1827 cs=0 l=1 rev1=1 crypto rx=0x7f686000d830 tx=0x7f686000dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:58.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:58.893+0000 7f686f400640 1 -- 172.21.15.16:0/2821729497 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f686000b6c0 con 0x7f68780b2200 2024-03-31T15:14:58.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:58.893+0000 7f686f400640 1 -- 172.21.15.16:0/2821729497 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6860019070 con 0x7f68780b2200 2024-03-31T15:14:58.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:58.893+0000 7f687dc00640 1 -- 172.21.15.16:0/2821729497 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f68781394e0 con 0x7f68780b2200 2024-03-31T15:14:58.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:58.893+0000 7f686f400640 1 -- 172.21.15.16:0/2821729497 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f686000ba60 con 0x7f68780b2200 2024-03-31T15:14:58.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:58.893+0000 7f687dc00640 1 -- 172.21.15.16:0/2821729497 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f687813f1c0 con 0x7f68780b2200 2024-03-31T15:14:58.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:58.893+0000 7f687dc00640 1 -- 172.21.15.16:0/2821729497 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6878069950 con 0x7f68780b2200 2024-03-31T15:14:58.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:58.893+0000 7f686f400640 1 -- 172.21.15.16:0/2821729497 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f686001e460 con 0x7f68780b2200 2024-03-31T15:14:58.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:58.897+0000 7f686f400640 1 --2- 172.21.15.16:0/2821729497 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f68540683b0 0x7f685406a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:58.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:58.897+0000 7f686f400640 1 -- 172.21.15.16:0/2821729497 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 5939+0+0 (secure 0 0 0) 0x7f6860087180 con 0x7f68780b2200 2024-03-31T15:14:58.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:58.897+0000 7f6876a00640 1 --2- 172.21.15.16:0/2821729497 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f68540683b0 0x7f685406a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:58.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:58.897+0000 7f6876a00640 1 --2- 172.21.15.16:0/2821729497 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f68540683b0 0x7f685406a870 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7f68780d0c80 tx=0x7f6868009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:58.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:58.905+0000 7f686f400640 1 -- 172.21.15.16:0/2821729497 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f6860057640 con 0x7f68780b2200 2024-03-31T15:14:59.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.105+0000 7f687dc00640 1 -- 172.21.15.16:0/2821729497 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f687815ce60 con 0x7f68780b2200 2024-03-31T15:14:59.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.105+0000 7f686f400640 1 -- 172.21.15.16:0/2821729497 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 112 v112) ==== 95+0+5185 (secure 0 0 0) 0x7f6860056f40 con 0x7f68780b2200 2024-03-31T15:14:59.098 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:14:59.098 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":112,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7631,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6836/1628897112","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":1628897112},{"type":"v1","addr":"172.21.15.89:6836","nonce":1628897112}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":94},{"gid":8746,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/1036724837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6835","nonce":1036724837},{"type":"v1","addr":"172.21.15.89:6837","nonce":1036724837}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":94},{"gid":9192,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6834/472139147","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":472139147},{"type":"v1","addr":"172.21.15.16:6834","nonce":472139147}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":94}],"filesystems":[{"mdsmap":{"epoch":112,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-31T15:14:08.652180+0000","modified":"2024-03-31T15:14:58.141614+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":113,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":9762},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9762":{"gid":9762,"name":"c","rank":0,"incarnation":109,"state":"up:active","state_seq":4,"addr":"172.21.15.16:6835/2152452999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6833","nonce":2152452999},{"type":"v1","addr":"172.21.15.16:6835","nonce":2152452999}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9762,"qdb_cluster":[9762]},"id":8}]} 2024-03-31T15:14:59.098 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 112 2024-03-31T15:14:59.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.105+0000 7f687dc00640 1 -- 172.21.15.16:0/2821729497 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f68540683b0 msgr2=0x7f685406a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:59.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.105+0000 7f687dc00640 1 --2- 172.21.15.16:0/2821729497 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f68540683b0 0x7f685406a870 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7f68780d0c80 tx=0x7f6868009040 comp rx=0 tx=0).stop 2024-03-31T15:14:59.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.105+0000 7f687dc00640 1 -- 172.21.15.16:0/2821729497 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f68780b2200 msgr2=0x7f687813ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:59.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.105+0000 7f687dc00640 1 --2- 172.21.15.16:0/2821729497 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f68780b2200 0x7f687813ec80 secure :-1 s=READY pgs=1827 cs=0 l=1 rev1=1 crypto rx=0x7f686000d830 tx=0x7f686000dcf0 comp rx=0 tx=0).stop 2024-03-31T15:14:59.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.105+0000 7f687dc00640 1 -- 172.21.15.16:0/2821729497 shutdown_connections 2024-03-31T15:14:59.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.105+0000 7f687dc00640 1 --2- 172.21.15.16:0/2821729497 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f68540683b0 0x7f685406a870 unknown :-1 s=CLOSED pgs=770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:59.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.105+0000 7f687dc00640 1 --2- 172.21.15.16:0/2821729497 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f6878145d10 0x7f68780d01b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:59.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.105+0000 7f687dc00640 1 --2- 172.21.15.16:0/2821729497 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f68780b2200 0x7f687813ec80 unknown :-1 s=CLOSED pgs=1827 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:59.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.105+0000 7f687dc00640 1 --2- 172.21.15.16:0/2821729497 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f68780728b0 0x7f687813bd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:59.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.105+0000 7f687dc00640 1 -- 172.21.15.16:0/2821729497 >> 172.21.15.16:0/2821729497 conn(0x7f687806b620 msgr2=0x7f68781657f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:14:59.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.105+0000 7f687dc00640 1 -- 172.21.15.16:0/2821729497 shutdown_connections 2024-03-31T15:14:59.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.105+0000 7f687dc00640 1 -- 172.21.15.16:0/2821729497 wait complete. 2024-03-31T15:14:59.114 INFO:tasks.ceph.mds.c.smithi016.stderr:2024-03-31T15:14:59.121+0000 7f22f2000640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-31T15:14:59.521 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 112, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-31T15:14:08.652180+0000', 'modified': '2024-03-31T15:14:58.141614+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 113, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 9762}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9762': {'gid': 9762, 'name': 'c', 'rank': 0, 'incarnation': 109, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.16:6835/2152452999', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6833', 'nonce': 2152452999}, {'type': 'v1', 'addr': '172.21.15.16:6835', 'nonce': 2152452999}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [17], 'metadata_pool': 16, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9762, 'qdb_cluster': [9762]} 2024-03-31T15:14:59.521 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-31T15:14:59.522 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-03-31T15:14:59.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.669+0000 7f06e5800640 1 Processor -- start 2024-03-31T15:14:59.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.669+0000 7f06e5800640 1 -- start start 2024-03-31T15:14:59.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.669+0000 7f06e5800640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f06e015e550 0x7f06e00afd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:59.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.669+0000 7f06e5800640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f06e015c080 0x7f06e006c630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:59.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.669+0000 7f06e5800640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f06e00b4950 con 0x7f06e006cb70 2024-03-31T15:14:59.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.669+0000 7f06e5800640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f06e00b4c30 con 0x7f06e015e550 2024-03-31T15:14:59.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.669+0000 7f06e5800640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f06e00b4f10 con 0x7f06e015c080 2024-03-31T15:14:59.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.673+0000 7f06d7e00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f06e015c080 0x7f06e006c630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:59.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.673+0000 7f06d7e00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f06e015c080 0x7f06e006c630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3301/0 says I am v2:172.21.15.16:39400/0 (socket says 172.21.15.16:39400) 2024-03-31T15:14:59.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.673+0000 7f06d7e00640 1 -- 172.21.15.16:0/1380887467 learned_addr learned my addr 172.21.15.16:0/1380887467 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:14:59.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.673+0000 7f06dfe00640 1 --2- 172.21.15.16:0/1380887467 >> v2:172.21.15.89:3300/0 conn(0x7f06e015e550 0x7f06e00afd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:59.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.673+0000 7f06d7e00640 1 -- 172.21.15.16:0/1380887467 >> v2:172.21.15.89:3300/0 conn(0x7f06e015e550 msgr2=0x7f06e00afd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:14:59.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.673+0000 7f06d7e00640 1 --2- 172.21.15.16:0/1380887467 >> v2:172.21.15.89:3300/0 conn(0x7f06e015e550 0x7f06e00afd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:59.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.673+0000 7f06d7e00640 1 -- 172.21.15.16:0/1380887467 >> v1:172.21.15.89:6789/0 conn(0x7f06e006cb70 legacy=0x7f06e015ec40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:59.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.673+0000 7f06d7e00640 1 -- 172.21.15.16:0/1380887467 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06e00b51f0 con 0x7f06e015c080 2024-03-31T15:14:59.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.673+0000 7f06df400640 1 -- 172.21.15.16:0/1380887467 <== mon.1 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3728823767 0 0) 0x7f06e00b4950 con 0x7f06e006cb70 2024-03-31T15:14:59.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.673+0000 7f06d7e00640 1 --2- 172.21.15.16:0/1380887467 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f06e015c080 0x7f06e006c630 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f06c8009870 tx=0x7f06c801c4d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=23b8648eaec2a4a1 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:59.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.673+0000 7f06df400640 1 -- 172.21.15.16:0/1380887467 <== mon.2 v2:172.21.15.89:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06c80b1070 con 0x7f06e015c080 2024-03-31T15:14:59.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.673+0000 7f06df400640 1 -- 172.21.15.16:0/1380887467 <== mon.2 v2:172.21.15.89:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f06c80a8050 con 0x7f06e015c080 2024-03-31T15:14:59.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.673+0000 7f06df400640 1 -- 172.21.15.16:0/1380887467 <== mon.2 v2:172.21.15.89:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06c80ac860 con 0x7f06e015c080 2024-03-31T15:14:59.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.677+0000 7f06e5800640 1 -- 172.21.15.16:0/1380887467 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f06e015c080 msgr2=0x7f06e006c630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:59.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.677+0000 7f06e5800640 1 --2- 172.21.15.16:0/1380887467 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f06e015c080 0x7f06e006c630 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f06c8009870 tx=0x7f06c801c4d0 comp rx=0 tx=0).stop 2024-03-31T15:14:59.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.677+0000 7f06e5800640 1 -- 172.21.15.16:0/1380887467 shutdown_connections 2024-03-31T15:14:59.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.677+0000 7f06e5800640 1 --2- 172.21.15.16:0/1380887467 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f06e015c080 0x7f06e006c630 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:59.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.677+0000 7f06e5800640 1 --2- 172.21.15.16:0/1380887467 >> v2:172.21.15.89:3300/0 conn(0x7f06e015e550 0x7f06e00afd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:59.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.677+0000 7f06e5800640 1 -- 172.21.15.16:0/1380887467 >> 172.21.15.16:0/1380887467 conn(0x7f06e0071bb0 msgr2=0x7f06e0071fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:14:59.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.677+0000 7f06e5800640 1 -- 172.21.15.16:0/1380887467 shutdown_connections 2024-03-31T15:14:59.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.677+0000 7f06e5800640 1 -- 172.21.15.16:0/1380887467 wait complete. 2024-03-31T15:14:59.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.677+0000 7f06e5800640 1 Processor -- start 2024-03-31T15:14:59.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.677+0000 7f06e5800640 1 -- start start 2024-03-31T15:14:59.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.677+0000 7f06e5800640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f06e006cb70 0x7f06e0151950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:59.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.677+0000 7f06e5800640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f06e015c080 0x7f06e006ed00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:59.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.677+0000 7f06e5800640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f06e015e550 0x7f06e01470c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:59.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.677+0000 7f06e5800640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f06e0168370 con 0x7f06e015e550 2024-03-31T15:14:59.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.677+0000 7f06e5800640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f06e01681f0 con 0x7f06e015c080 2024-03-31T15:14:59.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.677+0000 7f06e5800640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f06e01684f0 con 0x7f06e006cb70 2024-03-31T15:14:59.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.677+0000 7f06e4e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f06e015e550 0x7f06e01470c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:59.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.677+0000 7f06e4e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f06e015e550 0x7f06e01470c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50154/0 (socket says 172.21.15.16:50154) 2024-03-31T15:14:59.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.677+0000 7f06e4e00640 1 -- 172.21.15.16:0/2471329231 learned_addr learned my addr 172.21.15.16:0/2471329231 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:14:59.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.677+0000 7f06dfe00640 1 --2- 172.21.15.16:0/2471329231 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f06e006cb70 0x7f06e0151950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:59.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.677+0000 7f06d7e00640 1 --2- 172.21.15.16:0/2471329231 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f06e015c080 0x7f06e006ed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:59.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.681+0000 7f06e4e00640 1 -- 172.21.15.16:0/2471329231 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f06e006cb70 msgr2=0x7f06e0151950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:59.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.681+0000 7f06e4e00640 1 --2- 172.21.15.16:0/2471329231 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f06e006cb70 0x7f06e0151950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:59.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.681+0000 7f06e4e00640 1 -- 172.21.15.16:0/2471329231 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f06e015c080 msgr2=0x7f06e006ed00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:59.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.681+0000 7f06e4e00640 1 --2- 172.21.15.16:0/2471329231 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f06e015c080 0x7f06e006ed00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:59.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.681+0000 7f06e4e00640 1 -- 172.21.15.16:0/2471329231 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06e0164200 con 0x7f06e015e550 2024-03-31T15:14:59.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.681+0000 7f06d7e00640 1 --2- 172.21.15.16:0/2471329231 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f06e015c080 0x7f06e006ed00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:14:59.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.681+0000 7f06e4e00640 1 --2- 172.21.15.16:0/2471329231 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f06e015e550 0x7f06e01470c0 secure :-1 s=READY pgs=1828 cs=0 l=1 rev1=1 crypto rx=0x7f06d800c780 tx=0x7f06d800cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:59.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.681+0000 7f06dcc00640 1 -- 172.21.15.16:0/2471329231 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06d8003db0 con 0x7f06e015e550 2024-03-31T15:14:59.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.681+0000 7f06dcc00640 1 -- 172.21.15.16:0/2471329231 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f06d800ce30 con 0x7f06e015e550 2024-03-31T15:14:59.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.681+0000 7f06dcc00640 1 -- 172.21.15.16:0/2471329231 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06d8005290 con 0x7f06e015e550 2024-03-31T15:14:59.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.681+0000 7f06e5800640 1 -- 172.21.15.16:0/2471329231 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f06e01644e0 con 0x7f06e015e550 2024-03-31T15:14:59.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.681+0000 7f06e5800640 1 -- 172.21.15.16:0/2471329231 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f06e01463c0 con 0x7f06e015e550 2024-03-31T15:14:59.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.681+0000 7f06dcc00640 1 -- 172.21.15.16:0/2471329231 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f06d8005430 con 0x7f06e015e550 2024-03-31T15:14:59.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.681+0000 7f06dcc00640 1 --2- 172.21.15.16:0/2471329231 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f06bc068380 0x7f06bc06a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:59.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.681+0000 7f06dcc00640 1 -- 172.21.15.16:0/2471329231 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 5939+0+0 (secure 0 0 0) 0x7f06d8086940 con 0x7f06e015e550 2024-03-31T15:14:59.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.685+0000 7f06dfe00640 1 --2- 172.21.15.16:0/2471329231 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f06bc068380 0x7f06bc06a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:59.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.685+0000 7f06ea261000 1 Processor -- start 2024-03-31T15:14:59.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.685+0000 7f06ea261000 1 -- start start 2024-03-31T15:14:59.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.685+0000 7f06dfe00640 1 --2- 172.21.15.16:0/2471329231 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f06bc068380 0x7f06bc06a840 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f06d0009b60 tx=0x7f06d0009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:59.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.685+0000 7f06ea261000 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x562fe015b4a0 0x562fe015b880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:59.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.685+0000 7f06ea261000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562fe015bdc0 0x562fe010eb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:59.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.685+0000 7f06ea261000 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562fe010f0b0 0x562fe010f490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:59.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.685+0000 7f06ea261000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x562fdff61950 con 0x562fe015bdc0 2024-03-31T15:14:59.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.685+0000 7f06ea261000 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x562fdfefe990 con 0x562fe010f0b0 2024-03-31T15:14:59.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.685+0000 7f06ea261000 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x562fdff5d900 con 0x562fe015b4a0 2024-03-31T15:14:59.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.685+0000 7f06d7e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562fe015bdc0 0x562fe010eb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:59.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.685+0000 7f06d7e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562fe015bdc0 0x562fe010eb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50156/0 (socket says 172.21.15.16:50156) 2024-03-31T15:14:59.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.685+0000 7f06d7e00640 1 -- 172.21.15.16:0/1862943227 learned_addr learned my addr 172.21.15.16:0/1862943227 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:14:59.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.685+0000 7f06dfe00640 1 --2- 172.21.15.16:0/1862943227 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x562fe015b4a0 0x562fe015b880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:59.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.685+0000 7f06e4e00640 1 --2- 172.21.15.16:0/1862943227 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562fe010f0b0 0x562fe010f490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:59.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.685+0000 7f06d7e00640 1 -- 172.21.15.16:0/1862943227 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x562fe015b4a0 msgr2=0x562fe015b880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:14:59.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.685+0000 7f06d7e00640 1 --2- 172.21.15.16:0/1862943227 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x562fe015b4a0 0x562fe015b880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:59.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.685+0000 7f06d7e00640 1 -- 172.21.15.16:0/1862943227 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562fe010f0b0 msgr2=0x562fe010f490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:14:59.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.685+0000 7f06d7e00640 1 --2- 172.21.15.16:0/1862943227 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562fe010f0b0 0x562fe010f490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:59.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.685+0000 7f06d7e00640 1 -- 172.21.15.16:0/1862943227 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562fdffa6100 con 0x562fe015bdc0 2024-03-31T15:14:59.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.685+0000 7f06d7e00640 1 --2- 172.21.15.16:0/1862943227 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562fe015bdc0 0x562fe010eb70 secure :-1 s=READY pgs=1829 cs=0 l=1 rev1=1 crypto rx=0x7f06c801c9b0 tx=0x7f06c8004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ab8cdf4cad25212 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:59.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.685+0000 7f06d4c00640 1 -- 172.21.15.16:0/1862943227 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06c80b1040 con 0x562fe015bdc0 2024-03-31T15:14:59.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.685+0000 7f06d4c00640 1 -- 172.21.15.16:0/1862943227 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f06c80a8040 con 0x562fe015bdc0 2024-03-31T15:14:59.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.685+0000 7f06d4c00640 1 -- 172.21.15.16:0/1862943227 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06c80b6460 con 0x562fe015bdc0 2024-03-31T15:14:59.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.685+0000 7f06ea261000 1 -- 172.21.15.16:0/1862943227 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562fe015bdc0 msgr2=0x562fe010eb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:59.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.685+0000 7f06ea261000 1 --2- 172.21.15.16:0/1862943227 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562fe015bdc0 0x562fe010eb70 secure :-1 s=READY pgs=1829 cs=0 l=1 rev1=1 crypto rx=0x7f06c801c9b0 tx=0x7f06c8004740 comp rx=0 tx=0).stop 2024-03-31T15:14:59.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.685+0000 7f06ea261000 1 -- 172.21.15.16:0/1862943227 shutdown_connections 2024-03-31T15:14:59.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.685+0000 7f06ea261000 1 --2- 172.21.15.16:0/1862943227 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562fe010f0b0 0x562fe010f490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:59.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.685+0000 7f06ea261000 1 --2- 172.21.15.16:0/1862943227 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562fe015bdc0 0x562fe010eb70 unknown :-1 s=CLOSED pgs=1829 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:59.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.685+0000 7f06ea261000 1 --2- 172.21.15.16:0/1862943227 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x562fe015b4a0 0x562fe015b880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:59.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.685+0000 7f06ea261000 1 -- 172.21.15.16:0/1862943227 >> 172.21.15.16:0/1862943227 conn(0x562fdfff9b80 msgr2=0x562fe013e460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:14:59.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.685+0000 7f06ea261000 1 -- 172.21.15.16:0/1862943227 shutdown_connections 2024-03-31T15:14:59.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.685+0000 7f06ea261000 1 -- 172.21.15.16:0/1862943227 wait complete. 2024-03-31T15:14:59.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.689+0000 7f06ea261000 1 -- start start 2024-03-31T15:14:59.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.689+0000 7f06ea261000 1 Processor -- start 2024-03-31T15:14:59.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.689+0000 7f06ea261000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562fe010f0b0 0x562fe0163600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:59.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.689+0000 7f06ea261000 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x562fe015b4a0 0x562fe0163b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:59.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.689+0000 7f06ea261000 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562fe015bdc0 0x562fe011ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:59.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.689+0000 7f06ea261000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x562fdff60c60 con 0x562fe010f0b0 2024-03-31T15:14:59.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.689+0000 7f06ea261000 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x562fdff1f3e0 con 0x562fe015bdc0 2024-03-31T15:14:59.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.689+0000 7f06ea261000 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x562fdff6e060 con 0x562fe015b4a0 2024-03-31T15:14:59.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.689+0000 7f06ea261000 10 client.0 _ll_register_callbacks cb 0x562fdffc20f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T15:14:59.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.689+0000 7f06dfe00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562fe010f0b0 0x562fe0163600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:59.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.689+0000 7f06dfe00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562fe010f0b0 0x562fe0163600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50174/0 (socket says 172.21.15.16:50174) 2024-03-31T15:14:59.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.689+0000 7f06dfe00640 1 -- 172.21.15.16:0/3629065718 learned_addr learned my addr 172.21.15.16:0/3629065718 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:14:59.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.689+0000 7f06e4e00640 1 --2- 172.21.15.16:0/3629065718 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562fe015bdc0 0x562fe011ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:59.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.689+0000 7f06d7e00640 1 --2- 172.21.15.16:0/3629065718 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x562fe015b4a0 0x562fe0163b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:59.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.689+0000 7f06dfe00640 1 -- 172.21.15.16:0/3629065718 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x562fe015b4a0 msgr2=0x562fe0163b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:14:59.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.689+0000 7f06dfe00640 1 --2- 172.21.15.16:0/3629065718 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x562fe015b4a0 0x562fe0163b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:59.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.689+0000 7f06dfe00640 1 -- 172.21.15.16:0/3629065718 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562fe015bdc0 msgr2=0x562fe011ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:14:59.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.689+0000 7f06dfe00640 1 --2- 172.21.15.16:0/3629065718 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562fe015bdc0 0x562fe011ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:59.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.689+0000 7f06dfe00640 1 -- 172.21.15.16:0/3629065718 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562fdff2b8c0 con 0x562fe010f0b0 2024-03-31T15:14:59.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.689+0000 7f06d7e00640 1 --2- 172.21.15.16:0/3629065718 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x562fe015b4a0 0x562fe0163b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:14:59.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.689+0000 7f06dfe00640 1 --2- 172.21.15.16:0/3629065718 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562fe010f0b0 0x562fe0163600 secure :-1 s=READY pgs=1830 cs=0 l=1 rev1=1 crypto rx=0x7f06d0003650 tx=0x7f06d00046d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=adb872a119fc44ec server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:59.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.689+0000 7f06cd600640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-31T15:14:59.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.689+0000 7f06cd600640 1 -- 172.21.15.16:0/3629065718 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06d0027070 con 0x562fe010f0b0 2024-03-31T15:14:59.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.689+0000 7f06ea261000 1 -- 172.21.15.16:0/3629065718 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f06c80a8040 con 0x562fe010f0b0 2024-03-31T15:14:59.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.689+0000 7f06cd600640 1 -- 172.21.15.16:0/3629065718 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(113..113 src has 1..113) ==== 5939+0+0 (secure 0 0 0) 0x7f06d0017810 con 0x562fe010f0b0 2024-03-31T15:14:59.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.689+0000 7f06cd600640 1 -- 172.21.15.16:0/3629065718 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=112) ==== 24+0+0 (secure 0 0 0) 0x7f06d0031050 con 0x562fe010f0b0 2024-03-31T15:14:59.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.689+0000 7f06ea261000 10 client.9843 fetch_fsmap learned FSMap version 112 2024-03-31T15:14:59.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.689+0000 7f06ea261000 1 -- 172.21.15.16:0/3629065718 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=112,monmap=2+}) -- 0x562fdfee9e80 con 0x562fe010f0b0 2024-03-31T15:14:59.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.689+0000 7f06cd600640 1 -- 172.21.15.16:0/3629065718 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f06c80a8040 con 0x562fe010f0b0 2024-03-31T15:14:59.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.689+0000 7f06cd600640 1 -- 172.21.15.16:0/3629065718 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 112) ==== 3494+0+0 (secure 0 0 0) 0x7f06d0034020 con 0x562fe010f0b0 2024-03-31T15:14:59.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.689+0000 7f06ea261000 10 client.9843 fetch_fsmap finished waiting for FSMap version 112 2024-03-31T15:14:59.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.689+0000 7f06ea261000 10 client.9843 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-31T15:14:59.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.689+0000 7f06ea261000 20 client.9843 populate_metadata read hostname 'smithi016' 2024-03-31T15:14:59.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.689+0000 7f06ea261000 1 --2- 172.21.15.16:0/3629065718 >> [v2:172.21.15.16:6833/2152452999,v1:172.21.15.16:6835/2152452999] conn(0x562fe011c780 0x562fe011cb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:59.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.689+0000 7f06ea261000 4 client.9843 mds_command: new command op to 9762 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-31T15:14:59.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.689+0000 7f06ea261000 1 -- 172.21.15.16:0/3629065718 --> [v2:172.21.15.16:6833/2152452999,v1:172.21.15.16:6835/2152452999] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562fdfefcbc0 con 0x562fe011c780 2024-03-31T15:14:59.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.689+0000 7f06d7e00640 1 --2- 172.21.15.16:0/3629065718 >> [v2:172.21.15.16:6833/2152452999,v1:172.21.15.16:6835/2152452999] conn(0x562fe011c780 0x562fe011cb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:59.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.689+0000 7f06d7e00640 1 --2- 172.21.15.16:0/3629065718 >> [v2:172.21.15.16:6833/2152452999,v1:172.21.15.16:6835/2152452999] conn(0x562fe011c780 0x562fe011cb60 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7295ad4136ee5737 server_cookie=a7e698b808d1069a in_seq=0 out_seq=0 2024-03-31T15:14:59.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.689+0000 7f06cd600640 10 client.9843 ms_handle_connect on v2:172.21.15.16:6833/2152452999 2024-03-31T15:14:59.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.693+0000 7f06cd600640 1 -- 172.21.15.16:0/3629065718 <== mds.0 v2:172.21.15.16:6833/2152452999 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x562fdfefcbc0 con 0x562fe011c780 2024-03-31T15:14:59.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.693+0000 7f06cd600640 10 client.9843 handle_command_reply: tid=0 2024-03-31T15:14:59.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.693+0000 7f06ea261000 1 client.9843 shutdown 2024-03-31T15:14:59.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.693+0000 7f06ea261000 1 -- 172.21.15.16:0/3629065718 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562fe010f0b0 msgr2=0x562fe0163600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:59.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.693+0000 7f06ea261000 1 --2- 172.21.15.16:0/3629065718 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562fe010f0b0 0x562fe0163600 secure :-1 s=READY pgs=1830 cs=0 l=1 rev1=1 crypto rx=0x7f06d0003650 tx=0x7f06d00046d0 comp rx=0 tx=0).stop 2024-03-31T15:14:59.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.693+0000 7f06ea261000 1 -- 172.21.15.16:0/3629065718 shutdown_connections 2024-03-31T15:14:59.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.693+0000 7f06ea261000 1 --2- 172.21.15.16:0/3629065718 >> [v2:172.21.15.16:6833/2152452999,v1:172.21.15.16:6835/2152452999] conn(0x562fe011c780 0x562fe011cb60 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:59.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.693+0000 7f06ea261000 1 --2- 172.21.15.16:0/3629065718 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562fe015bdc0 0x562fe011ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:59.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.693+0000 7f06ea261000 1 --2- 172.21.15.16:0/3629065718 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x562fe015b4a0 0x562fe0163b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:59.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.693+0000 7f06ea261000 1 --2- 172.21.15.16:0/3629065718 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562fe010f0b0 0x562fe0163600 unknown :-1 s=CLOSED pgs=1830 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:59.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.693+0000 7f06cd600640 0 client.9843 ms_handle_reset on v2:172.21.15.16:6833/2152452999 2024-03-31T15:14:59.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.693+0000 7f06ea261000 1 -- 172.21.15.16:0/3629065718 >> 172.21.15.16:0/3629065718 conn(0x562fdfff9b80 msgr2=0x562fe010e270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:14:59.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.693+0000 7f06ea261000 1 -- 172.21.15.16:0/3629065718 shutdown_connections 2024-03-31T15:14:59.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.693+0000 7f06ea261000 1 -- 172.21.15.16:0/3629065718 wait complete. 2024-03-31T15:14:59.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.693+0000 7f06ea261000 20 client.9843 trim_cache size 0 max 16384 2024-03-31T15:14:59.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.709+0000 7f06ea261000 1 Processor -- start 2024-03-31T15:14:59.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.709+0000 7f06ea261000 1 -- start start 2024-03-31T15:14:59.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.713+0000 7f06ea261000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562fe010f0b0 0x562fe015ada0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:59.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.713+0000 7f06ea261000 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x562fe015b4a0 0x562fe01169d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:59.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.713+0000 7f06ea261000 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562fe015bdc0 0x562fe0116f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:59.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.713+0000 7f06ea261000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f06e0203c00 con 0x562fe010f0b0 2024-03-31T15:14:59.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.713+0000 7f06ea261000 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f06e0202c50 con 0x562fe015bdc0 2024-03-31T15:14:59.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.713+0000 7f06ea261000 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f06e01fb560 con 0x562fe015b4a0 2024-03-31T15:14:59.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.713+0000 7f06dfe00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562fe010f0b0 0x562fe015ada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:59.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.713+0000 7f06dfe00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562fe010f0b0 0x562fe015ada0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50182/0 (socket says 172.21.15.16:50182) 2024-03-31T15:14:59.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.713+0000 7f06dfe00640 1 -- 172.21.15.16:0/2349171680 learned_addr learned my addr 172.21.15.16:0/2349171680 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:14:59.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.713+0000 7f06d7e00640 1 --2- 172.21.15.16:0/2349171680 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x562fe015b4a0 0x562fe01169d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:59.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.713+0000 7f06e4e00640 1 --2- 172.21.15.16:0/2349171680 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562fe015bdc0 0x562fe0116f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:59.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.713+0000 7f06dfe00640 1 -- 172.21.15.16:0/2349171680 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x562fe015b4a0 msgr2=0x562fe01169d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:14:59.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.713+0000 7f06dfe00640 1 --2- 172.21.15.16:0/2349171680 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x562fe015b4a0 0x562fe01169d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:59.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.713+0000 7f06dfe00640 1 -- 172.21.15.16:0/2349171680 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562fe015bdc0 msgr2=0x562fe0116f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:14:59.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.713+0000 7f06dfe00640 1 --2- 172.21.15.16:0/2349171680 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562fe015bdc0 0x562fe0116f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:59.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.713+0000 7f06dfe00640 1 -- 172.21.15.16:0/2349171680 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562fdff1c490 con 0x562fe010f0b0 2024-03-31T15:14:59.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.713+0000 7f06dfe00640 1 --2- 172.21.15.16:0/2349171680 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562fe010f0b0 0x562fe015ada0 secure :-1 s=READY pgs=1831 cs=0 l=1 rev1=1 crypto rx=0x7f06d00046a0 tx=0x7f06d0024000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a915349eba13de2 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:59.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.713+0000 7f06cfe00640 1 -- 172.21.15.16:0/2349171680 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06d0027070 con 0x562fe010f0b0 2024-03-31T15:14:59.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.713+0000 7f06cfe00640 1 -- 172.21.15.16:0/2349171680 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f06d00170c0 con 0x562fe010f0b0 2024-03-31T15:14:59.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.713+0000 7f06cfe00640 1 -- 172.21.15.16:0/2349171680 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06d00174b0 con 0x562fe010f0b0 2024-03-31T15:14:59.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.713+0000 7f06ea261000 1 -- 172.21.15.16:0/2349171680 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562fe010f0b0 msgr2=0x562fe015ada0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:59.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.713+0000 7f06ea261000 1 --2- 172.21.15.16:0/2349171680 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562fe010f0b0 0x562fe015ada0 secure :-1 s=READY pgs=1831 cs=0 l=1 rev1=1 crypto rx=0x7f06d00046a0 tx=0x7f06d0024000 comp rx=0 tx=0).stop 2024-03-31T15:14:59.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.713+0000 7f06ea261000 1 -- 172.21.15.16:0/2349171680 shutdown_connections 2024-03-31T15:14:59.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.713+0000 7f06ea261000 1 --2- 172.21.15.16:0/2349171680 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562fe015bdc0 0x562fe0116f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:59.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.713+0000 7f06ea261000 1 --2- 172.21.15.16:0/2349171680 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x562fe015b4a0 0x562fe01169d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:59.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.713+0000 7f06ea261000 1 --2- 172.21.15.16:0/2349171680 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562fe010f0b0 0x562fe015ada0 unknown :-1 s=CLOSED pgs=1831 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:59.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.713+0000 7f06ea261000 1 -- 172.21.15.16:0/2349171680 >> 172.21.15.16:0/2349171680 conn(0x562fdfff9b80 msgr2=0x562fe01162c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:14:59.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.713+0000 7f06ea261000 1 -- 172.21.15.16:0/2349171680 shutdown_connections 2024-03-31T15:14:59.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.713+0000 7f06ea261000 1 -- 172.21.15.16:0/2349171680 wait complete. 2024-03-31T15:14:59.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.713+0000 7f06ea261000 1 -- start start 2024-03-31T15:14:59.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.713+0000 7f06ea261000 1 Processor -- start 2024-03-31T15:14:59.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.717+0000 7f06ea261000 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562fe010f0b0 0x562fe010e710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:59.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.717+0000 7f06ea261000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562fe015b4a0 0x562fe01cbde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:59.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.717+0000 7f06ea261000 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x562fe015bdc0 0x562fe01cc320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:59.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.717+0000 7f06ea261000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x562fdff65f80 con 0x562fe015b4a0 2024-03-31T15:14:59.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.717+0000 7f06ea261000 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f06e0165750 con 0x562fe010f0b0 2024-03-31T15:14:59.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.717+0000 7f06ea261000 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x562fdff5e8d0 con 0x562fe015bdc0 2024-03-31T15:14:59.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.717+0000 7f06ea261000 10 client.0 _ll_register_callbacks cb 0x7f06d0009060 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T15:14:59.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.717+0000 7f06d7e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562fe015b4a0 0x562fe01cbde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:59.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.717+0000 7f06d7e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562fe015b4a0 0x562fe01cbde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50186/0 (socket says 172.21.15.16:50186) 2024-03-31T15:14:59.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.717+0000 7f06d7e00640 1 -- 172.21.15.16:0/925852706 learned_addr learned my addr 172.21.15.16:0/925852706 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:14:59.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.717+0000 7f06dfe00640 1 --2- 172.21.15.16:0/925852706 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562fe010f0b0 0x562fe010e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:59.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.717+0000 7f06e4e00640 1 --2- 172.21.15.16:0/925852706 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x562fe015bdc0 0x562fe01cc320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:59.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.717+0000 7f06d7e00640 1 -- 172.21.15.16:0/925852706 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x562fe015bdc0 msgr2=0x562fe01cc320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:14:59.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.717+0000 7f06d7e00640 1 --2- 172.21.15.16:0/925852706 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x562fe015bdc0 0x562fe01cc320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:59.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.717+0000 7f06d7e00640 1 -- 172.21.15.16:0/925852706 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562fe010f0b0 msgr2=0x562fe010e710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:14:59.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.717+0000 7f06d7e00640 1 --2- 172.21.15.16:0/925852706 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562fe010f0b0 0x562fe010e710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:59.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.717+0000 7f06d7e00640 1 -- 172.21.15.16:0/925852706 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562fdff1ae80 con 0x562fe015b4a0 2024-03-31T15:14:59.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.717+0000 7f06dfe00640 1 --2- 172.21.15.16:0/925852706 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562fe010f0b0 0x562fe010e710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:14:59.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.717+0000 7f06d7e00640 1 --2- 172.21.15.16:0/925852706 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562fe015b4a0 0x562fe01cbde0 secure :-1 s=READY pgs=1832 cs=0 l=1 rev1=1 crypto rx=0x7f06c8008010 tx=0x7f06c80046d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=555e17a9534ff1a9 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:59.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.717+0000 7f06cd600640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-31T15:14:59.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.717+0000 7f06cd600640 1 -- 172.21.15.16:0/925852706 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06c80b1040 con 0x562fe015b4a0 2024-03-31T15:14:59.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.717+0000 7f06cd600640 1 -- 172.21.15.16:0/925852706 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(113..113 src has 1..113) ==== 5939+0+0 (secure 0 0 0) 0x7f06c8004020 con 0x562fe015b4a0 2024-03-31T15:14:59.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.717+0000 7f06ea261000 1 -- 172.21.15.16:0/925852706 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f06d00170c0 con 0x562fe015b4a0 2024-03-31T15:14:59.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.717+0000 7f06cd600640 1 -- 172.21.15.16:0/925852706 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=112) ==== 24+0+0 (secure 0 0 0) 0x7f06c8004220 con 0x562fe015b4a0 2024-03-31T15:14:59.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.717+0000 7f06ea261000 10 client.9849 fetch_fsmap learned FSMap version 112 2024-03-31T15:14:59.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.717+0000 7f06ea261000 1 -- 172.21.15.16:0/925852706 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=112,monmap=2+}) -- 0x562fdff08270 con 0x562fe015b4a0 2024-03-31T15:14:59.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.717+0000 7f06cd600640 1 -- 172.21.15.16:0/925852706 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f06d00170c0 con 0x562fe015b4a0 2024-03-31T15:14:59.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.717+0000 7f06cd600640 1 -- 172.21.15.16:0/925852706 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 112) ==== 3494+0+0 (secure 0 0 0) 0x7f06c80c7440 con 0x562fe015b4a0 2024-03-31T15:14:59.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.717+0000 7f06ea261000 10 client.9849 fetch_fsmap finished waiting for FSMap version 112 2024-03-31T15:14:59.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.717+0000 7f06ea261000 10 client.9849 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-31T15:14:59.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.717+0000 7f06ea261000 20 client.9849 populate_metadata read hostname 'smithi016' 2024-03-31T15:14:59.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.717+0000 7f06ea261000 1 --2- 172.21.15.16:0/925852706 >> [v2:172.21.15.16:6833/2152452999,v1:172.21.15.16:6835/2152452999] conn(0x562fe0117260 0x562fe01cc860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:59.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.717+0000 7f06ea261000 4 client.9849 mds_command: new command op to 9762 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-31T15:14:59.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.717+0000 7f06ea261000 1 -- 172.21.15.16:0/925852706 --> [v2:172.21.15.16:6833/2152452999,v1:172.21.15.16:6835/2152452999] -- command(tid 0: {"prefix": "status"}) -- 0x562fdff6ee90 con 0x562fe0117260 2024-03-31T15:14:59.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.717+0000 7f06dfe00640 1 --2- 172.21.15.16:0/925852706 >> [v2:172.21.15.16:6833/2152452999,v1:172.21.15.16:6835/2152452999] conn(0x562fe0117260 0x562fe01cc860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:59.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.717+0000 7f06cd600640 10 client.9849 ms_handle_connect on v2:172.21.15.16:6833/2152452999 2024-03-31T15:14:59.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.717+0000 7f06dfe00640 1 --2- 172.21.15.16:0/925852706 >> [v2:172.21.15.16:6833/2152452999,v1:172.21.15.16:6835/2152452999] conn(0x562fe0117260 0x562fe01cc860 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=db6350a1bcfba65b server_cookie=6e57a84fc0790ee8 in_seq=0 out_seq=0 2024-03-31T15:14:59.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.717+0000 7f06cd600640 1 -- 172.21.15.16:0/925852706 <== mds.0 v2:172.21.15.16:6833/2152452999 1 ==== command_reply(tid 0: 0 ) ==== 8+0+334 (crc 0 0 0) 0x562fdff6ee90 con 0x562fe0117260 2024-03-31T15:14:59.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.717+0000 7f06cd600640 10 client.9849 handle_command_reply: tid=0 2024-03-31T15:14:59.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.717+0000 7f06ea261000 1 client.9849 shutdown 2024-03-31T15:14:59.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.717+0000 7f06ea261000 1 -- 172.21.15.16:0/925852706 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562fe015b4a0 msgr2=0x562fe01cbde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:59.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.717+0000 7f06ea261000 1 --2- 172.21.15.16:0/925852706 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562fe015b4a0 0x562fe01cbde0 secure :-1 s=READY pgs=1832 cs=0 l=1 rev1=1 crypto rx=0x7f06c8008010 tx=0x7f06c80046d0 comp rx=0 tx=0).stop 2024-03-31T15:14:59.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.717+0000 7f06ea261000 1 -- 172.21.15.16:0/925852706 shutdown_connections 2024-03-31T15:14:59.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.717+0000 7f06ea261000 1 --2- 172.21.15.16:0/925852706 >> [v2:172.21.15.16:6833/2152452999,v1:172.21.15.16:6835/2152452999] conn(0x562fe0117260 0x562fe01cc860 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:59.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.717+0000 7f06ea261000 1 --2- 172.21.15.16:0/925852706 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x562fe015bdc0 0x562fe01cc320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:59.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.717+0000 7f06ea261000 1 --2- 172.21.15.16:0/925852706 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562fe015b4a0 0x562fe01cbde0 unknown :-1 s=CLOSED pgs=1832 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:59.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.717+0000 7f06ea261000 1 --2- 172.21.15.16:0/925852706 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562fe010f0b0 0x562fe010e710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:59.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.717+0000 7f06ea261000 1 -- 172.21.15.16:0/925852706 >> 172.21.15.16:0/925852706 conn(0x562fdfff9b80 msgr2=0x562fe0116b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:14:59.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.717+0000 7f06cd600640 0 client.9849 ms_handle_reset on v2:172.21.15.16:6833/2152452999 2024-03-31T15:14:59.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.717+0000 7f06ea261000 1 -- 172.21.15.16:0/925852706 shutdown_connections 2024-03-31T15:14:59.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.717+0000 7f06ea261000 1 -- 172.21.15.16:0/925852706 wait complete. 2024-03-31T15:14:59.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.717+0000 7f06ea261000 20 client.9849 trim_cache size 0 max 16384 2024-03-31T15:14:59.717 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-03-31T15:14:59.717 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_fsid": "286eb5da-fb47-4550-b290-d9519c13b9a5", 2024-03-31T15:14:59.717 INFO:teuthology.orchestra.run.smithi016.stdout: "whoami": 0, 2024-03-31T15:14:59.717 INFO:teuthology.orchestra.run.smithi016.stdout: "id": 9762, 2024-03-31T15:14:59.717 INFO:teuthology.orchestra.run.smithi016.stdout: "want_state": "up:active", 2024-03-31T15:14:59.717 INFO:teuthology.orchestra.run.smithi016.stdout: "state": "up:active", 2024-03-31T15:14:59.717 INFO:teuthology.orchestra.run.smithi016.stdout: "fs_name": "cephfs", 2024-03-31T15:14:59.717 INFO:teuthology.orchestra.run.smithi016.stdout: "rank_uptime": 4.5924279080000003, 2024-03-31T15:14:59.717 INFO:teuthology.orchestra.run.smithi016.stdout: "mdsmap_epoch": 112, 2024-03-31T15:14:59.718 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch": 113, 2024-03-31T15:14:59.718 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch_barrier": 113, 2024-03-31T15:14:59.718 INFO:teuthology.orchestra.run.smithi016.stdout: "uptime": 6.5434368540000003 2024-03-31T15:14:59.718 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-03-31T15:14:59.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.717+0000 7f06e5800640 1 -- 172.21.15.16:0/2471329231 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f06bc068380 msgr2=0x7f06bc06a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:59.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.717+0000 7f06e5800640 1 --2- 172.21.15.16:0/2471329231 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f06bc068380 0x7f06bc06a840 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f06d0009b60 tx=0x7f06d0009190 comp rx=0 tx=0).stop 2024-03-31T15:14:59.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.721+0000 7f06e5800640 1 -- 172.21.15.16:0/2471329231 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f06e015e550 msgr2=0x7f06e01470c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:59.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.721+0000 7f06e5800640 1 --2- 172.21.15.16:0/2471329231 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f06e015e550 0x7f06e01470c0 secure :-1 s=READY pgs=1828 cs=0 l=1 rev1=1 crypto rx=0x7f06d800c780 tx=0x7f06d800cc40 comp rx=0 tx=0).stop 2024-03-31T15:14:59.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.721+0000 7f06e5800640 1 -- 172.21.15.16:0/2471329231 shutdown_connections 2024-03-31T15:14:59.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.721+0000 7f06e5800640 1 --2- 172.21.15.16:0/2471329231 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f06bc068380 0x7f06bc06a840 unknown :-1 s=CLOSED pgs=771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:59.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.721+0000 7f06e5800640 1 --2- 172.21.15.16:0/2471329231 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f06e015e550 0x7f06e01470c0 unknown :-1 s=CLOSED pgs=1828 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:59.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.721+0000 7f06e5800640 1 --2- 172.21.15.16:0/2471329231 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f06e015c080 0x7f06e006ed00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:59.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.721+0000 7f06e5800640 1 --2- 172.21.15.16:0/2471329231 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f06e006cb70 0x7f06e0151950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:59.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.721+0000 7f06e5800640 1 -- 172.21.15.16:0/2471329231 >> 172.21.15.16:0/2471329231 conn(0x7f06e0071bb0 msgr2=0x7f06e014f7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:14:59.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.721+0000 7f06e5800640 1 -- 172.21.15.16:0/2471329231 shutdown_connections 2024-03-31T15:14:59.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.721+0000 7f06e5800640 1 -- 172.21.15.16:0/2471329231 wait complete. 2024-03-31T15:14:59.732 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-31T15:14:59.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.877+0000 7fcffa400640 1 Processor -- start 2024-03-31T15:14:59.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.877+0000 7fcffa400640 1 -- start start 2024-03-31T15:14:59.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.881+0000 7fcffa400640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fcff40b9c60 0x7fcff40b7470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:59.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.881+0000 7fcffa400640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fcff40b79b0 0x7fcff40b7e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:59.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.881+0000 7fcffa400640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcff40d3990 con 0x7fcff40d1280 2024-03-31T15:14:59.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.881+0000 7fcffa400640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7fcff40d26a0 con 0x7fcff40b79b0 2024-03-31T15:14:59.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.881+0000 7fcffa400640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fcff40d28f0 con 0x7fcff40b9c60 2024-03-31T15:14:59.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.881+0000 7fcff3e00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fcff40b79b0 0x7fcff40b7e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:59.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.881+0000 7fcff3400640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fcff40d1280 0x7fcff40d1660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:36666/0 (socket says 172.21.15.16:36666) 2024-03-31T15:14:59.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.881+0000 7fcfebe00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fcff40b9c60 0x7fcff40b7470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:59.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.881+0000 7fcff3400640 1 -- 172.21.15.16:0/359614005 learned_addr learned my addr 172.21.15.16:0/359614005 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:14:59.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.881+0000 7fcff3e00640 1 -- 172.21.15.16:0/359614005 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fcff40b9c60 msgr2=0x7fcff40b7470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:14:59.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.881+0000 7fcff3e00640 1 --2- 172.21.15.16:0/359614005 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fcff40b9c60 0x7fcff40b7470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:59.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.881+0000 7fcff2a00640 1 -- 172.21.15.16:0/359614005 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3094009424 0 0) 0x7fcff40d3990 con 0x7fcff40d1280 2024-03-31T15:14:59.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.881+0000 7fcff3e00640 1 -- 172.21.15.16:0/359614005 >> v1:172.21.15.16:6789/0 conn(0x7fcff40d1280 legacy=0x7fcff40d1660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:59.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.881+0000 7fcff3e00640 1 -- 172.21.15.16:0/359614005 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcff40d2bd0 con 0x7fcff40b79b0 2024-03-31T15:14:59.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.885+0000 7fcff3e00640 1 --2- 172.21.15.16:0/359614005 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcff40b79b0 0x7fcff40b7e30 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7fcfec009d90 tx=0x7fcfec01c8e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e6a584fddee9e87f server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:59.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.885+0000 7fcff2a00640 1 -- 172.21.15.16:0/359614005 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfec0b2070 con 0x7fcff40b79b0 2024-03-31T15:14:59.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.885+0000 7fcff2a00640 1 -- 172.21.15.16:0/359614005 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcfec0a5890 con 0x7fcff40b79b0 2024-03-31T15:14:59.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.885+0000 7fcff2a00640 1 -- 172.21.15.16:0/359614005 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfec0a5c60 con 0x7fcff40b79b0 2024-03-31T15:14:59.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.885+0000 7fcffa400640 1 -- 172.21.15.16:0/359614005 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcff40b79b0 msgr2=0x7fcff40b7e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:59.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.885+0000 7fcffa400640 1 --2- 172.21.15.16:0/359614005 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcff40b79b0 0x7fcff40b7e30 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7fcfec009d90 tx=0x7fcfec01c8e0 comp rx=0 tx=0).stop 2024-03-31T15:14:59.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.885+0000 7fcffa400640 1 -- 172.21.15.16:0/359614005 shutdown_connections 2024-03-31T15:14:59.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.885+0000 7fcffa400640 1 --2- 172.21.15.16:0/359614005 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcff40b79b0 0x7fcff40b7e30 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:59.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.885+0000 7fcffa400640 1 --2- 172.21.15.16:0/359614005 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fcff40b9c60 0x7fcff40b7470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:59.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.885+0000 7fcffa400640 1 -- 172.21.15.16:0/359614005 >> 172.21.15.16:0/359614005 conn(0x7fcff4067730 msgr2=0x7fcff4067b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:14:59.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.885+0000 7fcffa400640 1 -- 172.21.15.16:0/359614005 shutdown_connections 2024-03-31T15:14:59.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.885+0000 7fcffa400640 1 -- 172.21.15.16:0/359614005 wait complete. 2024-03-31T15:14:59.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.889+0000 7fcffa400640 1 Processor -- start 2024-03-31T15:14:59.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.889+0000 7fcffa400640 1 -- start start 2024-03-31T15:14:59.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.889+0000 7fcffa400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcff40b79b0 0x7fcff415c780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:59.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.889+0000 7fcffa400640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcff40b9c60 0x7fcff4156230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:59.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.889+0000 7fcffa400640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fcff40d1280 0x7fcff4156770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:59.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.889+0000 7fcffa400640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fcff40d4010 con 0x7fcff40b79b0 2024-03-31T15:14:59.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.889+0000 7fcffa400640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fcff40d3e90 con 0x7fcff40b9c60 2024-03-31T15:14:59.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.889+0000 7fcffa400640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fcff416b8a0 con 0x7fcff40d1280 2024-03-31T15:14:59.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.889+0000 7fcff3400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcff40b79b0 0x7fcff415c780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:59.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.889+0000 7fcff3400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcff40b79b0 0x7fcff415c780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50190/0 (socket says 172.21.15.16:50190) 2024-03-31T15:14:59.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.889+0000 7fcff3400640 1 -- 172.21.15.16:0/3345388837 learned_addr learned my addr 172.21.15.16:0/3345388837 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:14:59.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.889+0000 7fcfebe00640 1 --2- 172.21.15.16:0/3345388837 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcff40b9c60 0x7fcff4156230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:59.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.889+0000 7fcff3e00640 1 --2- 172.21.15.16:0/3345388837 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fcff40d1280 0x7fcff4156770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:59.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.889+0000 7fcff3400640 1 -- 172.21.15.16:0/3345388837 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fcff40d1280 msgr2=0x7fcff4156770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:59.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.889+0000 7fcff3400640 1 --2- 172.21.15.16:0/3345388837 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fcff40d1280 0x7fcff4156770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:59.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.889+0000 7fcff3400640 1 -- 172.21.15.16:0/3345388837 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcff40b9c60 msgr2=0x7fcff4156230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:14:59.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.889+0000 7fcff3400640 1 --2- 172.21.15.16:0/3345388837 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcff40b9c60 0x7fcff4156230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:14:59.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.889+0000 7fcff3400640 1 -- 172.21.15.16:0/3345388837 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcff4156e10 con 0x7fcff40b79b0 2024-03-31T15:14:59.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.889+0000 7fcfebe00640 1 --2- 172.21.15.16:0/3345388837 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcff40b9c60 0x7fcff4156230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:14:59.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.889+0000 7fcff3400640 1 --2- 172.21.15.16:0/3345388837 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcff40b79b0 0x7fcff415c780 secure :-1 s=READY pgs=1834 cs=0 l=1 rev1=1 crypto rx=0x7fcfe400c4b0 tx=0x7fcfe400c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:59.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.889+0000 7fcfeb400640 1 -- 172.21.15.16:0/3345388837 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfe4018070 con 0x7fcff40b79b0 2024-03-31T15:14:59.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.889+0000 7fcfeb400640 1 -- 172.21.15.16:0/3345388837 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcfe4007500 con 0x7fcff40b79b0 2024-03-31T15:14:59.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.889+0000 7fcffa400640 1 -- 172.21.15.16:0/3345388837 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcff41570f0 con 0x7fcff40b79b0 2024-03-31T15:14:59.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.889+0000 7fcfeb400640 1 -- 172.21.15.16:0/3345388837 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfe4004a30 con 0x7fcff40b79b0 2024-03-31T15:14:59.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.889+0000 7fcffa400640 1 -- 172.21.15.16:0/3345388837 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcff41601e0 con 0x7fcff40b79b0 2024-03-31T15:14:59.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.893+0000 7fcffa400640 1 -- 172.21.15.16:0/3345388837 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcff4069950 con 0x7fcff40b79b0 2024-03-31T15:14:59.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.893+0000 7fcfeb400640 1 -- 172.21.15.16:0/3345388837 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fcfe401b070 con 0x7fcff40b79b0 2024-03-31T15:14:59.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.893+0000 7fcfeb400640 1 --2- 172.21.15.16:0/3345388837 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fcfd0068380 0x7fcfd006a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:14:59.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.893+0000 7fcfeb400640 1 -- 172.21.15.16:0/3345388837 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 5939+0+0 (secure 0 0 0) 0x7fcfe4085d10 con 0x7fcff40b79b0 2024-03-31T15:14:59.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.893+0000 7fcfebe00640 1 --2- 172.21.15.16:0/3345388837 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fcfd0068380 0x7fcfd006a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:14:59.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.897+0000 7fcfebe00640 1 --2- 172.21.15.16:0/3345388837 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fcfd0068380 0x7fcfd006a840 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7fcff415e4f0 tx=0x7fcfdc009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:14:59.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:14:59.901+0000 7fcfeb400640 1 -- 172.21.15.16:0/3345388837 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fcfe408b020 con 0x7fcff40b79b0 2024-03-31T15:15:00.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:00.109+0000 7fcffa400640 1 -- 172.21.15.16:0/3345388837 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7fcff40b7e80 con 0x7fcff40b79b0 2024-03-31T15:15:00.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:00.109+0000 7fcfeb400640 1 -- 172.21.15.16:0/3345388837 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v112) ==== 53+0+73 (secure 0 0 0) 0x7fcfe4055b80 con 0x7fcff40b79b0 2024-03-31T15:15:00.102 INFO:teuthology.orchestra.run.smithi016.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-31T15:15:00.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:00.109+0000 7fcffa400640 1 -- 172.21.15.16:0/3345388837 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fcfd0068380 msgr2=0x7fcfd006a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:00.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:00.109+0000 7fcffa400640 1 --2- 172.21.15.16:0/3345388837 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fcfd0068380 0x7fcfd006a840 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7fcff415e4f0 tx=0x7fcfdc009190 comp rx=0 tx=0).stop 2024-03-31T15:15:00.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:00.109+0000 7fcffa400640 1 -- 172.21.15.16:0/3345388837 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcff40b79b0 msgr2=0x7fcff415c780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:00.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:00.109+0000 7fcffa400640 1 --2- 172.21.15.16:0/3345388837 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcff40b79b0 0x7fcff415c780 secure :-1 s=READY pgs=1834 cs=0 l=1 rev1=1 crypto rx=0x7fcfe400c4b0 tx=0x7fcfe400c970 comp rx=0 tx=0).stop 2024-03-31T15:15:00.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:00.109+0000 7fcffa400640 1 -- 172.21.15.16:0/3345388837 shutdown_connections 2024-03-31T15:15:00.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:00.109+0000 7fcffa400640 1 --2- 172.21.15.16:0/3345388837 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fcfd0068380 0x7fcfd006a840 unknown :-1 s=CLOSED pgs=772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:00.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:00.109+0000 7fcffa400640 1 --2- 172.21.15.16:0/3345388837 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fcff40d1280 0x7fcff4156770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:00.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:00.109+0000 7fcffa400640 1 --2- 172.21.15.16:0/3345388837 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcff40b9c60 0x7fcff4156230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:00.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:00.109+0000 7fcffa400640 1 --2- 172.21.15.16:0/3345388837 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcff40b79b0 0x7fcff415c780 unknown :-1 s=CLOSED pgs=1834 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:00.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:00.109+0000 7fcffa400640 1 -- 172.21.15.16:0/3345388837 >> 172.21.15.16:0/3345388837 conn(0x7fcff4067730 msgr2=0x7fcff40ca570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:15:00.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:00.109+0000 7fcffa400640 1 -- 172.21.15.16:0/3345388837 shutdown_connections 2024-03-31T15:15:00.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:00.109+0000 7fcffa400640 1 -- 172.21.15.16:0/3345388837 wait complete. 2024-03-31T15:15:00.123 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-31T15:15:00.123 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-31T15:15:00.123 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi016.front.sepia.ceph.com 2024-03-31T15:15:00.124 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-31T15:15:00.124 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-31T15:15:00.124 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-31T15:15:00.124 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-31T15:15:00.124 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-31T15:15:00.124 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-31T15:15:00.124 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:15:00.124 DEBUG:teuthology.orchestra.run.smithi016:> ip addr 2024-03-31T15:15:00.135 INFO:teuthology.orchestra.run.smithi016.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-31T15:15:00.135 INFO:teuthology.orchestra.run.smithi016.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-31T15:15:00.135 INFO:teuthology.orchestra.run.smithi016.stdout: inet 127.0.0.1/8 scope host lo 2024-03-31T15:15:00.135 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2024-03-31T15:15:00.135 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 ::1/128 scope host 2024-03-31T15:15:00.135 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2024-03-31T15:15:00.135 INFO:teuthology.orchestra.run.smithi016.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-03-31T15:15:00.135 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:6b:fd:e0 brd ff:ff:ff:ff:ff:ff 2024-03-31T15:15:00.135 INFO:teuthology.orchestra.run.smithi016.stdout: altname enp7s0f0 2024-03-31T15:15:00.136 INFO:teuthology.orchestra.run.smithi016.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-03-31T15:15:00.136 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:6b:fd:e1 brd ff:ff:ff:ff:ff:ff 2024-03-31T15:15:00.136 INFO:teuthology.orchestra.run.smithi016.stdout: altname enp7s0f1 2024-03-31T15:15:00.136 INFO:teuthology.orchestra.run.smithi016.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-31T15:15:00.136 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:bd:15:64 brd ff:ff:ff:ff:ff:ff 2024-03-31T15:15:00.136 INFO:teuthology.orchestra.run.smithi016.stdout: altname enp1s0f0 2024-03-31T15:15:00.136 INFO:teuthology.orchestra.run.smithi016.stdout: inet 172.21.15.16/20 brd 172.21.15.255 scope global dynamic ens1f0 2024-03-31T15:15:00.136 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft 41781sec preferred_lft 41781sec 2024-03-31T15:15:00.136 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 fe80::ec4:7aff:febd:1564/64 scope link 2024-03-31T15:15:00.137 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2024-03-31T15:15:00.137 INFO:teuthology.orchestra.run.smithi016.stdout:5: ens1f1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-03-31T15:15:00.137 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:bd:15:65 brd ff:ff:ff:ff:ff:ff 2024-03-31T15:15:00.137 INFO:teuthology.orchestra.run.smithi016.stdout: altname enp1s0f1 2024-03-31T15:15:00.137 INFO:teuthology.orchestra.run.smithi016.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2024-03-31T15:15:00.137 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 02:42:d4:1f:7d:73 brd ff:ff:ff:ff:ff:ff 2024-03-31T15:15:00.137 INFO:teuthology.orchestra.run.smithi016.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2024-03-31T15:15:00.137 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2024-03-31T15:15:00.137 INFO:teuthology.orchestra.run.smithi016.stdout:7: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-31T15:15:00.137 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 5e:1f:84:6d:79:ae brd ff:ff:ff:ff:ff:ff 2024-03-31T15:15:00.137 INFO:teuthology.orchestra.run.smithi016.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-31T15:15:00.137 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2024-03-31T15:15:00.137 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 fe80::5c1f:84ff:fe6d:79ae/64 scope link 2024-03-31T15:15:00.137 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2024-03-31T15:15:00.138 INFO:teuthology.orchestra.run.smithi016.stdout:8: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-31T15:15:00.138 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0e:5c:f6:1f:82:2f brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-31T15:15:00.138 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 fe80::c5c:f6ff:fe1f:822f/64 scope link 2024-03-31T15:15:00.138 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2024-03-31T15:15:00.138 INFO:teuthology.orchestra.run.smithi016.stdout:9: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-31T15:15:00.138 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether c6:a6:3d:69:d0:db brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-31T15:15:00.138 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 fe80::c4a6:3dff:fe69:d0db/64 scope link 2024-03-31T15:15:00.138 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2024-03-31T15:15:00.138 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:15:00.139 DEBUG:teuthology.orchestra.run.smithi016:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-31T15:15:00.199 INFO:teuthology.orchestra.run.smithi016.stdout:1 2024-03-31T15:15:00.201 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:15:00.202 DEBUG:teuthology.orchestra.run.smithi016:> route 2024-03-31T15:15:00.258 INFO:teuthology.orchestra.run.smithi016.stdout:Kernel IP routing table 2024-03-31T15:15:00.258 INFO:teuthology.orchestra.run.smithi016.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-31T15:15:00.259 INFO:teuthology.orchestra.run.smithi016.stdout:default _gateway 0.0.0.0 UG 0 0 0 ens1f0 2024-03-31T15:15:00.259 INFO:teuthology.orchestra.run.smithi016.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2024-03-31T15:15:00.259 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 0 0 0 ens1f0 2024-03-31T15:15:00.259 INFO:teuthology.orchestra.run.smithi016.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-31T15:15:00.259 DEBUG:teuthology.orchestra.run.smithi016:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-31T15:15:00.259 DEBUG:teuthology.orchestra.run.smithi016:> set -e 2024-03-31T15:15:00.259 DEBUG:teuthology.orchestra.run.smithi016:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-03-31T15:15:00.259 DEBUG:teuthology.orchestra.run.smithi016:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-03-31T15:15:00.259 DEBUG:teuthology.orchestra.run.smithi016:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-03-31T15:15:00.259 DEBUG:teuthology.orchestra.run.smithi016:> ') 2024-03-31T15:15:00.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:00 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-31T15:15:00.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:00 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-31T15:15:00.397 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:15:00.397 DEBUG:teuthology.orchestra.run.smithi016:> ip netns list 2024-03-31T15:15:00.449 INFO:teuthology.orchestra.run.smithi016.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-31T15:15:00.449 INFO:teuthology.orchestra.run.smithi016.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-31T15:15:00.449 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-31T15:15:00.449 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:15:00.450 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip link set brx.0 up 2024-03-31T15:15:00.512 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-03-31T15:15:00.513 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-31T15:15:00.513 DEBUG:teuthology.orchestra.run.smithi016:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-31T15:15:00.568 INFO:teuthology.orchestra.run.smithi016.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-31T15:15:00.569 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-31T15:15:00.569 DEBUG:teuthology.orchestra.run.smithi016:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-31T15:15:00.624 DEBUG:teuthology.orchestra.run.smithi016:> sudo modprobe fuse 2024-03-31T15:15:00.688 DEBUG:teuthology.orchestra.run.smithi016:> cat /proc/self/mounts | awk '{print $2}' 2024-03-31T15:15:00.737 INFO:teuthology.orchestra.run.smithi016.stdout:/sys 2024-03-31T15:15:00.738 INFO:teuthology.orchestra.run.smithi016.stdout:/proc 2024-03-31T15:15:00.738 INFO:teuthology.orchestra.run.smithi016.stdout:/dev 2024-03-31T15:15:00.738 INFO:teuthology.orchestra.run.smithi016.stdout:/dev/pts 2024-03-31T15:15:00.738 INFO:teuthology.orchestra.run.smithi016.stdout:/run 2024-03-31T15:15:00.738 INFO:teuthology.orchestra.run.smithi016.stdout:/ 2024-03-31T15:15:00.738 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/kernel/security 2024-03-31T15:15:00.738 INFO:teuthology.orchestra.run.smithi016.stdout:/dev/shm 2024-03-31T15:15:00.738 INFO:teuthology.orchestra.run.smithi016.stdout:/run/lock 2024-03-31T15:15:00.738 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/cgroup 2024-03-31T15:15:00.738 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/pstore 2024-03-31T15:15:00.738 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/bpf 2024-03-31T15:15:00.738 INFO:teuthology.orchestra.run.smithi016.stdout:/proc/sys/fs/binfmt_misc 2024-03-31T15:15:00.738 INFO:teuthology.orchestra.run.smithi016.stdout:/dev/hugepages 2024-03-31T15:15:00.739 INFO:teuthology.orchestra.run.smithi016.stdout:/dev/mqueue 2024-03-31T15:15:00.739 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/kernel/debug 2024-03-31T15:15:00.739 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/kernel/tracing 2024-03-31T15:15:00.739 INFO:teuthology.orchestra.run.smithi016.stdout:/proc/fs/nfsd 2024-03-31T15:15:00.739 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/fuse/connections 2024-03-31T15:15:00.739 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/kernel/config 2024-03-31T15:15:00.739 INFO:teuthology.orchestra.run.smithi016.stdout:/run/credentials/systemd-sysusers.service 2024-03-31T15:15:00.739 INFO:teuthology.orchestra.run.smithi016.stdout:/run/rpc_pipefs 2024-03-31T15:15:00.739 INFO:teuthology.orchestra.run.smithi016.stdout:/run/user/1000 2024-03-31T15:15:00.739 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/ceph 2024-03-31T15:15:00.739 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/ceph/osd/ceph-0 2024-03-31T15:15:00.739 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/ceph/osd/ceph-1 2024-03-31T15:15:00.739 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/ceph/osd/ceph-2 2024-03-31T15:15:00.739 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/ceph/osd/ceph-3 2024-03-31T15:15:00.740 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns 2024-03-31T15:15:00.740 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-31T15:15:00.740 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-31T15:15:00.740 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-31T15:15:00.740 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-31T15:15:00.741 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:15:00.741 DEBUG:teuthology.orchestra.run.smithi016:> ls /sys/fs/fuse/connections 2024-03-31T15:15:00.792 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-31T15:15:00.792 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-03-31T15:15:00.835 DEBUG:teuthology.orchestra.run.smithi016:> sudo modprobe fuse 2024-03-31T15:15:00.855 DEBUG:teuthology.orchestra.run.smithi016:> cat /proc/self/mounts | awk '{print $2}' 2024-03-31T15:15:00.910 INFO:teuthology.orchestra.run.smithi016.stdout:/sys 2024-03-31T15:15:00.910 INFO:teuthology.orchestra.run.smithi016.stdout:/proc 2024-03-31T15:15:00.910 INFO:teuthology.orchestra.run.smithi016.stdout:/dev 2024-03-31T15:15:00.910 INFO:teuthology.orchestra.run.smithi016.stdout:/dev/pts 2024-03-31T15:15:00.910 INFO:teuthology.orchestra.run.smithi016.stdout:/run 2024-03-31T15:15:00.910 INFO:teuthology.orchestra.run.smithi016.stdout:/ 2024-03-31T15:15:00.910 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/kernel/security 2024-03-31T15:15:00.911 INFO:teuthology.orchestra.run.smithi016.stdout:/dev/shm 2024-03-31T15:15:00.911 INFO:teuthology.orchestra.run.smithi016.stdout:/run/lock 2024-03-31T15:15:00.911 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/cgroup 2024-03-31T15:15:00.911 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/pstore 2024-03-31T15:15:00.911 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/bpf 2024-03-31T15:15:00.911 INFO:teuthology.orchestra.run.smithi016.stdout:/proc/sys/fs/binfmt_misc 2024-03-31T15:15:00.911 INFO:teuthology.orchestra.run.smithi016.stdout:/dev/hugepages 2024-03-31T15:15:00.911 INFO:teuthology.orchestra.run.smithi016.stdout:/dev/mqueue 2024-03-31T15:15:00.911 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/kernel/debug 2024-03-31T15:15:00.911 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/kernel/tracing 2024-03-31T15:15:00.911 INFO:teuthology.orchestra.run.smithi016.stdout:/proc/fs/nfsd 2024-03-31T15:15:00.911 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/fuse/connections 2024-03-31T15:15:00.911 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/kernel/config 2024-03-31T15:15:00.911 INFO:teuthology.orchestra.run.smithi016.stdout:/run/credentials/systemd-sysusers.service 2024-03-31T15:15:00.912 INFO:teuthology.orchestra.run.smithi016.stdout:/run/rpc_pipefs 2024-03-31T15:15:00.912 INFO:teuthology.orchestra.run.smithi016.stdout:/run/user/1000 2024-03-31T15:15:00.912 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/ceph 2024-03-31T15:15:00.912 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/ceph/osd/ceph-0 2024-03-31T15:15:00.912 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/ceph/osd/ceph-1 2024-03-31T15:15:00.912 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/ceph/osd/ceph-2 2024-03-31T15:15:00.912 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/ceph/osd/ceph-3 2024-03-31T15:15:00.912 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns 2024-03-31T15:15:00.912 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-31T15:15:00.912 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-31T15:15:00.912 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-31T15:15:00.912 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-31T15:15:00.913 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:15:00.913 DEBUG:teuthology.orchestra.run.smithi016:> ls /sys/fs/fuse/connections 2024-03-31T15:15:00.932 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi016.stderr:2024-03-31T15:15:00.937+0000 7f4870446580 -1 init, newargv = 0x5653a20e1530 newargc=15 2024-03-31T15:15:00.932 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi016.stderr:ceph-fuse[53165]: starting ceph client 2024-03-31T15:15:00.940 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi016.stderr:ceph-fuse[53165]: starting fuse 2024-03-31T15:15:00.941 INFO:teuthology.orchestra.run.smithi016.stdout:43 2024-03-31T15:15:00.942 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [43] 2024-03-31T15:15:00.942 DEBUG:teuthology.orchestra.run.smithi016:> sudo stdin-killer -- python3 -c ' 2024-03-31T15:15:00.942 DEBUG:teuthology.orchestra.run.smithi016:> import glob 2024-03-31T15:15:00.942 DEBUG:teuthology.orchestra.run.smithi016:> import re 2024-03-31T15:15:00.942 DEBUG:teuthology.orchestra.run.smithi016:> import os 2024-03-31T15:15:00.942 DEBUG:teuthology.orchestra.run.smithi016:> import subprocess 2024-03-31T15:15:00.942 DEBUG:teuthology.orchestra.run.smithi016:> 2024-03-31T15:15:00.942 DEBUG:teuthology.orchestra.run.smithi016:> def _find_admin_socket(client_name): 2024-03-31T15:15:00.942 DEBUG:teuthology.orchestra.run.smithi016:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-03-31T15:15:00.942 DEBUG:teuthology.orchestra.run.smithi016:> files = glob.glob(asok_path) 2024-03-31T15:15:00.943 DEBUG:teuthology.orchestra.run.smithi016:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-03-31T15:15:00.943 DEBUG:teuthology.orchestra.run.smithi016:> 2024-03-31T15:15:00.943 DEBUG:teuthology.orchestra.run.smithi016:> # Given a non-glob path, it better be there 2024-03-31T15:15:00.943 DEBUG:teuthology.orchestra.run.smithi016:> if "*" not in asok_path: 2024-03-31T15:15:00.943 DEBUG:teuthology.orchestra.run.smithi016:> assert(len(files) == 1) 2024-03-31T15:15:00.943 DEBUG:teuthology.orchestra.run.smithi016:> return files[0] 2024-03-31T15:15:00.943 DEBUG:teuthology.orchestra.run.smithi016:> 2024-03-31T15:15:00.943 DEBUG:teuthology.orchestra.run.smithi016:> for f in files: 2024-03-31T15:15:00.943 DEBUG:teuthology.orchestra.run.smithi016:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-31T15:15:00.943 DEBUG:teuthology.orchestra.run.smithi016:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-31T15:15:00.943 DEBUG:teuthology.orchestra.run.smithi016:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-31T15:15:00.943 DEBUG:teuthology.orchestra.run.smithi016:> contents = proc_f.read() 2024-03-31T15:15:00.943 DEBUG:teuthology.orchestra.run.smithi016:> if mountpoint in contents: 2024-03-31T15:15:00.943 DEBUG:teuthology.orchestra.run.smithi016:> return f 2024-03-31T15:15:00.943 DEBUG:teuthology.orchestra.run.smithi016:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-31T15:15:00.944 DEBUG:teuthology.orchestra.run.smithi016:> 2024-03-31T15:15:00.944 DEBUG:teuthology.orchestra.run.smithi016:> print(_find_admin_socket("client.0")) 2024-03-31T15:15:00.944 DEBUG:teuthology.orchestra.run.smithi016:> ' 2024-03-31T15:15:01.070 INFO:teuthology.orchestra.run.smithi016.stdout:/var/run/ceph/ceph-client.0.53165.asok 2024-03-31T15:15:01.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:01 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-31T15:15:01.082 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.53165.asok 2024-03-31T15:15:01.082 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:15:01.082 DEBUG:teuthology.orchestra.run.smithi016:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.53165.asok status 2024-03-31T15:15:01.256 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-03-31T15:15:01.256 INFO:teuthology.orchestra.run.smithi016.stdout: "metadata": { 2024-03-31T15:15:01.257 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph_sha1": "d2271579d5e735cfd6d436b8349c004f06527cb0", 2024-03-31T15:15:01.257 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph_version": "ceph version 19.0.0-2574-gd2271579 (d2271579d5e735cfd6d436b8349c004f06527cb0) squid (dev)", 2024-03-31T15:15:01.257 INFO:teuthology.orchestra.run.smithi016.stdout: "entity_id": "0", 2024-03-31T15:15:01.257 INFO:teuthology.orchestra.run.smithi016.stdout: "hostname": "smithi016", 2024-03-31T15:15:01.257 INFO:teuthology.orchestra.run.smithi016.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-03-31T15:15:01.257 INFO:teuthology.orchestra.run.smithi016.stdout: "pid": "53165", 2024-03-31T15:15:01.257 INFO:teuthology.orchestra.run.smithi016.stdout: "root": "/" 2024-03-31T15:15:01.257 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-31T15:15:01.257 INFO:teuthology.orchestra.run.smithi016.stdout: "dentry_count": 0, 2024-03-31T15:15:01.257 INFO:teuthology.orchestra.run.smithi016.stdout: "dentry_pinned_count": 0, 2024-03-31T15:15:01.257 INFO:teuthology.orchestra.run.smithi016.stdout: "id": 9379, 2024-03-31T15:15:01.257 INFO:teuthology.orchestra.run.smithi016.stdout: "inst": { 2024-03-31T15:15:01.257 INFO:teuthology.orchestra.run.smithi016.stdout: "name": { 2024-03-31T15:15:01.258 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "client", 2024-03-31T15:15:01.258 INFO:teuthology.orchestra.run.smithi016.stdout: "num": 9379 2024-03-31T15:15:01.258 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-31T15:15:01.258 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": { 2024-03-31T15:15:01.258 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-03-31T15:15:01.258 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "192.168.0.1:0", 2024-03-31T15:15:01.258 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2454691604 2024-03-31T15:15:01.258 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-31T15:15:01.258 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-31T15:15:01.258 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": { 2024-03-31T15:15:01.258 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-03-31T15:15:01.258 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "192.168.0.1:0", 2024-03-31T15:15:01.258 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2454691604 2024-03-31T15:15:01.258 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-31T15:15:01.259 INFO:teuthology.orchestra.run.smithi016.stdout: "inst_str": "client.9379 192.168.0.1:0/2454691604", 2024-03-31T15:15:01.259 INFO:teuthology.orchestra.run.smithi016.stdout: "addr_str": "192.168.0.1:0/2454691604", 2024-03-31T15:15:01.259 INFO:teuthology.orchestra.run.smithi016.stdout: "inode_count": 1, 2024-03-31T15:15:01.259 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_epoch": 112, 2024-03-31T15:15:01.259 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_epoch": 113, 2024-03-31T15:15:01.259 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_epoch_barrier": 0, 2024-03-31T15:15:01.259 INFO:teuthology.orchestra.run.smithi016.stdout: "blocklisted": false, 2024-03-31T15:15:01.259 INFO:teuthology.orchestra.run.smithi016.stdout: "fs_name": "cephfs" 2024-03-31T15:15:01.259 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-03-31T15:15:01.273 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:15:01.273 DEBUG:teuthology.orchestra.run.smithi016:> stat --file-system '--printf=%T 2024-03-31T15:15:01.273 DEBUG:teuthology.orchestra.run.smithi016:> ' -- /home/ubuntu/cephtest/mnt.0 2024-03-31T15:15:01.282 INFO:teuthology.orchestra.run.smithi016.stdout:fuseblk 2024-03-31T15:15:01.282 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-03-31T15:15:01.283 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:15:01.283 DEBUG:teuthology.orchestra.run.smithi016:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-31T15:15:01.345 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:15:01.345 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec find ./) 2024-03-31T15:15:01.412 INFO:teuthology.orchestra.run.smithi016.stdout:./ 2024-03-31T15:15:01.412 INFO:teuthology.orchestra.run.smithi016.stdout:./parent 2024-03-31T15:15:01.412 INFO:teuthology.orchestra.run.smithi016.stdout:./parent/flushed 2024-03-31T15:15:01.412 INFO:teuthology.orchestra.run.smithi016.stdout:./parent/flushed/charlie 2024-03-31T15:15:01.412 INFO:teuthology.orchestra.run.smithi016.stdout:./parent/flushed/alpha 2024-03-31T15:15:01.412 INFO:teuthology.orchestra.run.smithi016.stdout:./parent/flushed/bravo 2024-03-31T15:15:01.413 INFO:teuthology.orchestra.run.smithi016.stdout:./parent/unflushed 2024-03-31T15:15:01.413 INFO:teuthology.orchestra.run.smithi016.stdout:./parent/unflushed/jfile 2024-03-31T15:15:01.414 DEBUG:teuthology.orchestra.run.smithi016:> stdin-killer -- python3 -c ' 2024-03-31T15:15:01.414 DEBUG:teuthology.orchestra.run.smithi016:> import os 2024-03-31T15:15:01.414 DEBUG:teuthology.orchestra.run.smithi016:> import stat 2024-03-31T15:15:01.414 DEBUG:teuthology.orchestra.run.smithi016:> 2024-03-31T15:15:01.414 DEBUG:teuthology.orchestra.run.smithi016:> print(os.stat("/home/ubuntu/cephtest/mnt.0/./").st_ino) 2024-03-31T15:15:01.414 DEBUG:teuthology.orchestra.run.smithi016:> ' 2024-03-31T15:15:01.536 INFO:teuthology.orchestra.run.smithi016.stdout:1 2024-03-31T15:15:01.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:01 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-31T15:15:01.547 DEBUG:teuthology.orchestra.run.smithi016:> stdin-killer -- python3 -c ' 2024-03-31T15:15:01.547 DEBUG:teuthology.orchestra.run.smithi016:> import os 2024-03-31T15:15:01.547 DEBUG:teuthology.orchestra.run.smithi016:> import stat 2024-03-31T15:15:01.547 DEBUG:teuthology.orchestra.run.smithi016:> 2024-03-31T15:15:01.547 DEBUG:teuthology.orchestra.run.smithi016:> print(os.stat("/home/ubuntu/cephtest/mnt.0/./parent").st_ino) 2024-03-31T15:15:01.547 DEBUG:teuthology.orchestra.run.smithi016:> ' 2024-03-31T15:15:01.654 INFO:teuthology.orchestra.run.smithi016.stdout:1099511627776 2024-03-31T15:15:01.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:01 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-31T15:15:01.663 DEBUG:teuthology.orchestra.run.smithi016:> stdin-killer -- python3 -c ' 2024-03-31T15:15:01.663 DEBUG:teuthology.orchestra.run.smithi016:> import os 2024-03-31T15:15:01.663 DEBUG:teuthology.orchestra.run.smithi016:> import stat 2024-03-31T15:15:01.663 DEBUG:teuthology.orchestra.run.smithi016:> 2024-03-31T15:15:01.663 DEBUG:teuthology.orchestra.run.smithi016:> print(os.stat("/home/ubuntu/cephtest/mnt.0/./parent/flushed").st_ino) 2024-03-31T15:15:01.663 DEBUG:teuthology.orchestra.run.smithi016:> ' 2024-03-31T15:15:01.770 INFO:teuthology.orchestra.run.smithi016.stdout:1099511627777 2024-03-31T15:15:01.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:01 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-31T15:15:01.779 DEBUG:teuthology.orchestra.run.smithi016:> stdin-killer -- python3 -c ' 2024-03-31T15:15:01.779 DEBUG:teuthology.orchestra.run.smithi016:> import os 2024-03-31T15:15:01.779 DEBUG:teuthology.orchestra.run.smithi016:> import stat 2024-03-31T15:15:01.780 DEBUG:teuthology.orchestra.run.smithi016:> 2024-03-31T15:15:01.780 DEBUG:teuthology.orchestra.run.smithi016:> print(os.stat("/home/ubuntu/cephtest/mnt.0/./parent/flushed/charlie").st_ino) 2024-03-31T15:15:01.780 DEBUG:teuthology.orchestra.run.smithi016:> ' 2024-03-31T15:15:01.886 INFO:teuthology.orchestra.run.smithi016.stdout:1099511628280 2024-03-31T15:15:01.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:01 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-31T15:15:01.895 DEBUG:teuthology.orchestra.run.smithi016:> stdin-killer -- python3 -c ' 2024-03-31T15:15:01.895 DEBUG:teuthology.orchestra.run.smithi016:> import os 2024-03-31T15:15:01.896 DEBUG:teuthology.orchestra.run.smithi016:> import stat 2024-03-31T15:15:01.896 DEBUG:teuthology.orchestra.run.smithi016:> 2024-03-31T15:15:01.896 DEBUG:teuthology.orchestra.run.smithi016:> print(os.stat("/home/ubuntu/cephtest/mnt.0/./parent/flushed/alpha").st_ino) 2024-03-31T15:15:01.896 DEBUG:teuthology.orchestra.run.smithi016:> ' 2024-03-31T15:15:02.002 INFO:teuthology.orchestra.run.smithi016.stdout:1099511627778 2024-03-31T15:15:02.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:02 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-31T15:15:02.012 DEBUG:teuthology.orchestra.run.smithi016:> stdin-killer -- python3 -c ' 2024-03-31T15:15:02.013 DEBUG:teuthology.orchestra.run.smithi016:> import os 2024-03-31T15:15:02.013 DEBUG:teuthology.orchestra.run.smithi016:> import stat 2024-03-31T15:15:02.013 DEBUG:teuthology.orchestra.run.smithi016:> 2024-03-31T15:15:02.013 DEBUG:teuthology.orchestra.run.smithi016:> print(os.stat("/home/ubuntu/cephtest/mnt.0/./parent/flushed/bravo").st_ino) 2024-03-31T15:15:02.013 DEBUG:teuthology.orchestra.run.smithi016:> ' 2024-03-31T15:15:02.118 INFO:teuthology.orchestra.run.smithi016.stdout:1099511628279 2024-03-31T15:15:02.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:02 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-31T15:15:02.128 DEBUG:teuthology.orchestra.run.smithi016:> stdin-killer -- python3 -c ' 2024-03-31T15:15:02.128 DEBUG:teuthology.orchestra.run.smithi016:> import os 2024-03-31T15:15:02.128 DEBUG:teuthology.orchestra.run.smithi016:> import stat 2024-03-31T15:15:02.128 DEBUG:teuthology.orchestra.run.smithi016:> 2024-03-31T15:15:02.128 DEBUG:teuthology.orchestra.run.smithi016:> print(os.stat("/home/ubuntu/cephtest/mnt.0/./parent/unflushed").st_ino) 2024-03-31T15:15:02.128 DEBUG:teuthology.orchestra.run.smithi016:> ' 2024-03-31T15:15:02.242 INFO:teuthology.orchestra.run.smithi016.stdout:1099511627779 2024-03-31T15:15:02.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:02 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-31T15:15:02.254 DEBUG:teuthology.orchestra.run.smithi016:> stdin-killer -- python3 -c ' 2024-03-31T15:15:02.254 DEBUG:teuthology.orchestra.run.smithi016:> import os 2024-03-31T15:15:02.254 DEBUG:teuthology.orchestra.run.smithi016:> import stat 2024-03-31T15:15:02.254 DEBUG:teuthology.orchestra.run.smithi016:> 2024-03-31T15:15:02.254 DEBUG:teuthology.orchestra.run.smithi016:> print(os.stat("/home/ubuntu/cephtest/mnt.0/./parent/unflushed/jfile").st_ino) 2024-03-31T15:15:02.254 DEBUG:teuthology.orchestra.run.smithi016:> ' 2024-03-31T15:15:02.362 INFO:teuthology.orchestra.run.smithi016.stdout:1099511627780 2024-03-31T15:15:02.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:02 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-31T15:15:02.372 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-31T15:15:02.372 DEBUG:teuthology.orchestra.run.smithi016:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-31T15:15:02.480 DEBUG:teuthology.orchestra.run.smithi089:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-31T15:15:02.571 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-31T15:15:02.571 INFO:tasks.cephfs.mount:Terminating background process 2024-03-31T15:15:02.572 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-03-31T15:15:02.572 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-31T15:15:02.580 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-31T15:15:02.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:02.769+0000 7fd7c2200640 1 Processor -- start 2024-03-31T15:15:02.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:02.769+0000 7fd7c2200640 1 -- start start 2024-03-31T15:15:02.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:02.773+0000 7fd7c2200640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fd7bc145aa0 0x7fd7bc15c8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:02.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:02.773+0000 7fd7c2200640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd7bc0d09f0 con 0x7fd7bc06b620 2024-03-31T15:15:02.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:02.773+0000 7fd7c2200640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd7bc0d0cd0 con 0x7fd7bc06c1c0 2024-03-31T15:15:02.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:02.773+0000 7fd7c2200640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fd7bc06fd50 con 0x7fd7bc145aa0 2024-03-31T15:15:02.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:02.773+0000 7fd7bbe00640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fd7bc06b620 0x7fd7bc0cdab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:36670/0 (socket says 172.21.15.16:36670) 2024-03-31T15:15:02.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:02.773+0000 7fd7bbe00640 1 -- 172.21.15.16:0/1106723613 learned_addr learned my addr 172.21.15.16:0/1106723613 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:15:02.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:02.773+0000 7fd7bb400640 1 --2- 172.21.15.16:0/1106723613 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fd7bc145aa0 0x7fd7bc15c8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:02.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:02.773+0000 7fd7baa00640 1 -- 172.21.15.16:0/1106723613 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 375115603 0 0) 0x7fd7bc0d09f0 con 0x7fd7bc06b620 2024-03-31T15:15:02.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:02.773+0000 7fd7baa00640 1 -- 172.21.15.16:0/1106723613 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd7a0003530 con 0x7fd7bc06b620 2024-03-31T15:15:02.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:02.773+0000 7fd7baa00640 1 -- 172.21.15.16:0/1106723613 <== mon.1 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1366395023 0 0) 0x7fd7bc0d0cd0 con 0x7fd7bc06c1c0 2024-03-31T15:15:02.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:02.773+0000 7fd7baa00640 1 -- 172.21.15.16:0/1106723613 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd7bc0d09f0 con 0x7fd7bc06c1c0 2024-03-31T15:15:02.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:02.773+0000 7fd7bb400640 1 -- 172.21.15.16:0/1106723613 >> v1:172.21.15.89:6789/0 conn(0x7fd7bc06c1c0 legacy=0x7fd7bc072e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:02.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:02.773+0000 7fd7bb400640 1 -- 172.21.15.16:0/1106723613 >> v1:172.21.15.16:6789/0 conn(0x7fd7bc06b620 legacy=0x7fd7bc0cdab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:02.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:02.773+0000 7fd7bb400640 1 -- 172.21.15.16:0/1106723613 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7bc0d06e0 con 0x7fd7bc145aa0 2024-03-31T15:15:02.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:02.777+0000 7fd7bb400640 1 --2- 172.21.15.16:0/1106723613 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fd7bc145aa0 0x7fd7bc15c8e0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fd7ac0097f0 tx=0x7fd7ac01c4e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3bb562cc13fdf529 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:15:02.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:02.777+0000 7fd7baa00640 1 -- 172.21.15.16:0/1106723613 <== mon.2 v2:172.21.15.89:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7ac0b0070 con 0x7fd7bc145aa0 2024-03-31T15:15:02.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:02.777+0000 7fd7baa00640 1 -- 172.21.15.16:0/1106723613 <== mon.2 v2:172.21.15.89:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd7ac0ab600 con 0x7fd7bc145aa0 2024-03-31T15:15:02.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:02.777+0000 7fd7baa00640 1 -- 172.21.15.16:0/1106723613 <== mon.2 v2:172.21.15.89:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7ac0ab9b0 con 0x7fd7bc145aa0 2024-03-31T15:15:02.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:02.777+0000 7fd7c2200640 1 -- 172.21.15.16:0/1106723613 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fd7bc145aa0 msgr2=0x7fd7bc15c8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:02.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:02.777+0000 7fd7c2200640 1 --2- 172.21.15.16:0/1106723613 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fd7bc145aa0 0x7fd7bc15c8e0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fd7ac0097f0 tx=0x7fd7ac01c4e0 comp rx=0 tx=0).stop 2024-03-31T15:15:02.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:02.777+0000 7fd7c2200640 1 -- 172.21.15.16:0/1106723613 shutdown_connections 2024-03-31T15:15:02.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:02.777+0000 7fd7c2200640 1 --2- 172.21.15.16:0/1106723613 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fd7bc145aa0 0x7fd7bc15c8e0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:02.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:02.777+0000 7fd7c2200640 1 -- 172.21.15.16:0/1106723613 >> 172.21.15.16:0/1106723613 conn(0x7fd7bc0d1060 msgr2=0x7fd7bc0d1470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:15:02.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:02.777+0000 7fd7c2200640 1 -- 172.21.15.16:0/1106723613 shutdown_connections 2024-03-31T15:15:02.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:02.777+0000 7fd7c2200640 1 -- 172.21.15.16:0/1106723613 wait complete. 2024-03-31T15:15:02.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:02.777+0000 7fd7c2200640 1 Processor -- start 2024-03-31T15:15:02.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:02.777+0000 7fd7c2200640 1 -- start start 2024-03-31T15:15:02.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:02.777+0000 7fd7c2200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd7bc06b620 0x7fd7bc13c130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:02.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:02.777+0000 7fd7c2200640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fd7bc06c1c0 0x7fd7bc144000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:02.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:02.777+0000 7fd7c2200640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd7bc145aa0 0x7fd7bc0b1e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:02.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:02.785+0000 7fd7c2200640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fd7bc16a290 con 0x7fd7bc06b620 2024-03-31T15:15:02.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:02.785+0000 7fd7c2200640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fd7bc16a110 con 0x7fd7bc145aa0 2024-03-31T15:15:02.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:02.785+0000 7fd7c2200640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fd7bc16a410 con 0x7fd7bc06c1c0 2024-03-31T15:15:02.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:02.785+0000 7fd7bb400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd7bc06b620 0x7fd7bc13c130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:02.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:02.785+0000 7fd7bb400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd7bc06b620 0x7fd7bc13c130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50200/0 (socket says 172.21.15.16:50200) 2024-03-31T15:15:02.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:02.785+0000 7fd7bb400640 1 -- 172.21.15.16:0/3587383507 learned_addr learned my addr 172.21.15.16:0/3587383507 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:15:02.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:02.785+0000 7fd7b2a00640 1 --2- 172.21.15.16:0/3587383507 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fd7bc06c1c0 0x7fd7bc144000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:02.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:02.785+0000 7fd7bbe00640 1 --2- 172.21.15.16:0/3587383507 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd7bc145aa0 0x7fd7bc0b1e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:02.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:02.785+0000 7fd7bb400640 1 -- 172.21.15.16:0/3587383507 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fd7bc06c1c0 msgr2=0x7fd7bc144000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:02.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:02.785+0000 7fd7bb400640 1 --2- 172.21.15.16:0/3587383507 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fd7bc06c1c0 0x7fd7bc144000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:02.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:02.785+0000 7fd7bb400640 1 -- 172.21.15.16:0/3587383507 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd7bc145aa0 msgr2=0x7fd7bc0b1e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:02.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:02.785+0000 7fd7bb400640 1 --2- 172.21.15.16:0/3587383507 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd7bc145aa0 0x7fd7bc0b1e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:02.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:02.785+0000 7fd7bb400640 1 -- 172.21.15.16:0/3587383507 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7bc144aa0 con 0x7fd7bc06b620 2024-03-31T15:15:02.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:02.785+0000 7fd7bb400640 1 --2- 172.21.15.16:0/3587383507 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd7bc06b620 0x7fd7bc13c130 secure :-1 s=READY pgs=1838 cs=0 l=1 rev1=1 crypto rx=0x7fd7ac0190f0 tx=0x7fd7ac019280 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:15:02.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:02.785+0000 7fd7b3e00640 1 -- 172.21.15.16:0/3587383507 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7ac0b9020 con 0x7fd7bc06b620 2024-03-31T15:15:02.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:02.785+0000 7fd7b3e00640 1 -- 172.21.15.16:0/3587383507 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd7ac0ab7b0 con 0x7fd7bc06b620 2024-03-31T15:15:02.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:02.785+0000 7fd7b3e00640 1 -- 172.21.15.16:0/3587383507 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7ac0b0040 con 0x7fd7bc06b620 2024-03-31T15:15:02.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:02.785+0000 7fd7c2200640 1 -- 172.21.15.16:0/3587383507 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd7bc144d80 con 0x7fd7bc06b620 2024-03-31T15:15:02.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:02.785+0000 7fd7c2200640 1 -- 172.21.15.16:0/3587383507 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd7bc13d630 con 0x7fd7bc06b620 2024-03-31T15:15:02.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:02.785+0000 7fd7c2200640 1 -- 172.21.15.16:0/3587383507 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd7bc069950 con 0x7fd7bc06b620 2024-03-31T15:15:02.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:02.785+0000 7fd7b3e00640 1 -- 172.21.15.16:0/3587383507 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fd7ac0bc020 con 0x7fd7bc06b620 2024-03-31T15:15:02.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:02.785+0000 7fd7b3e00640 1 --2- 172.21.15.16:0/3587383507 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fd798068380 0x7fd79806a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:02.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:02.785+0000 7fd7b3e00640 1 -- 172.21.15.16:0/3587383507 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 5939+0+0 (secure 0 0 0) 0x7fd7ac120830 con 0x7fd7bc06b620 2024-03-31T15:15:02.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:02.785+0000 7fd7b2a00640 1 --2- 172.21.15.16:0/3587383507 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fd798068380 0x7fd79806a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:02.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:02.785+0000 7fd7b2a00640 1 --2- 172.21.15.16:0/3587383507 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fd798068380 0x7fd79806a840 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7fd7bc0b17e0 tx=0x7fd7a4009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:15:02.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:02.789+0000 7fd7b3e00640 1 -- 172.21.15.16:0/3587383507 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fd7ac0c3030 con 0x7fd7bc06b620 2024-03-31T15:15:02.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:02.953+0000 7fd7c2200640 1 -- 172.21.15.16:0/3587383507 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fd7bc0b10c0 con 0x7fd7bc06b620 2024-03-31T15:15:02.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:02.953+0000 7fd7b3e00640 1 -- 172.21.15.16:0/3587383507 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v113) ==== 74+0+16907 (secure 0 0 0) 0x7fd7ac0160d0 con 0x7fd7bc06b620 2024-03-31T15:15:02.947 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:15:02.947 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":113,"fsid":"286eb5da-fb47-4550-b290-d9519c13b9a5","created":"2024-03-31T15:03:11.438952+0000","modified":"2024-03-31T15:14:53.117209+0000","last_up_change":"2024-03-31T15:03:22.885535+0000","last_in_change":"2024-03-31T15:03:13.941667+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-31T15:03:22.550787+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-03-31T15:14:06.417304+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"108","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-03-31T15:14:07.622562+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"109","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"92a66450-ef19-40d8-ac3d-d760b23b0bda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6801","nonce":932712244}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6803","nonce":932712244}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6807","nonce":932712244}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6805","nonce":932712244}]},"public_addr":"172.21.15.16:6801/932712244","cluster_addr":"172.21.15.16:6803/932712244","heartbeat_back_addr":"172.21.15.16:6807/932712244","heartbeat_front_addr":"172.21.15.16:6805/932712244","state":["exists","up"]},{"osd":1,"uuid":"da9d1f2b-1968-4959-b843-faa231988e92","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6810","nonce":1398737533}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6811","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6813","nonce":1398737533}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6821","nonce":1398737533}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6815","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6816","nonce":1398737533}]},"public_addr":"172.21.15.16:6810/1398737533","cluster_addr":"172.21.15.16:6813/1398737533","heartbeat_back_addr":"172.21.15.16:6821/1398737533","heartbeat_front_addr":"172.21.15.16:6816/1398737533","state":["exists","up"]},{"osd":2,"uuid":"01a73662-9a21-4b1f-bcbb-42b9636a4638","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6819","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6822","nonce":3018526646}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6825","nonce":3018526646}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6831","nonce":3018526646}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6827","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6829","nonce":3018526646}]},"public_addr":"172.21.15.16:6822/3018526646","cluster_addr":"172.21.15.16:6825/3018526646","heartbeat_back_addr":"172.21.15.16:6831/3018526646","heartbeat_front_addr":"172.21.15.16:6829/3018526646","state":["exists","up"]},{"osd":3,"uuid":"c7771fd3-e18e-4500-ac38-23cf10ebcd75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6809","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6812","nonce":3360352974}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6817","nonce":3360352974}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6828","nonce":3360352974}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6823","nonce":3360352974}]},"public_addr":"172.21.15.16:6812/3360352974","cluster_addr":"172.21.15.16:6817/3360352974","heartbeat_back_addr":"172.21.15.16:6828/3360352974","heartbeat_front_addr":"172.21.15.16:6823/3360352974","state":["exists","up"]},{"osd":4,"uuid":"88260730-1482-4fbd-a40f-cb1124e9df4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6801","nonce":1856844856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6803","nonce":1856844856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6807","nonce":1856844856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6805","nonce":1856844856}]},"public_addr":"172.21.15.89:6801/1856844856","cluster_addr":"172.21.15.89:6803/1856844856","heartbeat_back_addr":"172.21.15.89:6807/1856844856","heartbeat_front_addr":"172.21.15.89:6805/1856844856","state":["exists","up"]},{"osd":5,"uuid":"4cb2c00c-fae0-4dd8-9d27-ec4f0093ccf7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6814","nonce":50395527}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6817","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6820","nonce":50395527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6831","nonce":50395527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6823","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6826","nonce":50395527}]},"public_addr":"172.21.15.89:6814/50395527","cluster_addr":"172.21.15.89:6820/50395527","heartbeat_back_addr":"172.21.15.89:6831/50395527","heartbeat_front_addr":"172.21.15.89:6826/50395527","state":["exists","up"]},{"osd":6,"uuid":"81ce0c19-2554-4cae-acbe-66a05e9e9b9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6809","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6812","nonce":3470471752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6815","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6818","nonce":3470471752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6827","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6829","nonce":3470471752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6821","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6824","nonce":3470471752}]},"public_addr":"172.21.15.89:6812/3470471752","cluster_addr":"172.21.15.89:6818/3470471752","heartbeat_back_addr":"172.21.15.89:6829/3470471752","heartbeat_front_addr":"172.21.15.89:6824/3470471752","state":["exists","up"]},{"osd":7,"uuid":"85e4313b-d37d-442e-8042-153bf20a847d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6811","nonce":1796778630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6813","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6816","nonce":1796778630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6825","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6828","nonce":1796778630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6819","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6822","nonce":1796778630}]},"public_addr":"172.21.15.89:6811/1796778630","cluster_addr":"172.21.15.89:6816/1796778630","heartbeat_back_addr":"172.21.15.89:6828/1796778630","heartbeat_front_addr":"172.21.15.89:6822/1796778630","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.439088+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.529970+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.680512+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:21.044161+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.582111+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.536558+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.409522+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.590229+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"17.1","mappings":[{"from":0,"to":5}]},{"pgid":"17.21","mappings":[{"from":0,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:6833/4142860904":"2024-04-01T15:14:53.117187+0000","172.21.15.16:6835/2564952016":"2024-04-01T15:14:39.715615+0000","172.21.15.16:6835/4142860904":"2024-04-01T15:14:53.117187+0000","172.21.15.16:6833/2564952016":"2024-04-01T15:14:39.715615+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-31T15:15:02.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:02.957+0000 7fd7c2200640 1 -- 172.21.15.16:0/3587383507 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fd798068380 msgr2=0x7fd79806a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:02.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:02.957+0000 7fd7c2200640 1 --2- 172.21.15.16:0/3587383507 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fd798068380 0x7fd79806a840 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7fd7bc0b17e0 tx=0x7fd7a4009040 comp rx=0 tx=0).stop 2024-03-31T15:15:02.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:02.957+0000 7fd7c2200640 1 -- 172.21.15.16:0/3587383507 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd7bc06b620 msgr2=0x7fd7bc13c130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:02.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:02.957+0000 7fd7c2200640 1 --2- 172.21.15.16:0/3587383507 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd7bc06b620 0x7fd7bc13c130 secure :-1 s=READY pgs=1838 cs=0 l=1 rev1=1 crypto rx=0x7fd7ac0190f0 tx=0x7fd7ac019280 comp rx=0 tx=0).stop 2024-03-31T15:15:02.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:02.957+0000 7fd7c2200640 1 -- 172.21.15.16:0/3587383507 shutdown_connections 2024-03-31T15:15:02.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:02.957+0000 7fd7c2200640 1 --2- 172.21.15.16:0/3587383507 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fd798068380 0x7fd79806a840 unknown :-1 s=CLOSED pgs=773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:02.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:02.957+0000 7fd7c2200640 1 --2- 172.21.15.16:0/3587383507 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd7bc145aa0 0x7fd7bc0b1e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:02.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:02.957+0000 7fd7c2200640 1 --2- 172.21.15.16:0/3587383507 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fd7bc06c1c0 0x7fd7bc144000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:02.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:02.957+0000 7fd7c2200640 1 --2- 172.21.15.16:0/3587383507 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd7bc06b620 0x7fd7bc13c130 unknown :-1 s=CLOSED pgs=1838 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:02.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:02.957+0000 7fd7c2200640 1 -- 172.21.15.16:0/3587383507 >> 172.21.15.16:0/3587383507 conn(0x7fd7bc0d1060 msgr2=0x7fd7bc15e9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:15:02.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:02.957+0000 7fd7c2200640 1 -- 172.21.15.16:0/3587383507 shutdown_connections 2024-03-31T15:15:02.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:02.957+0000 7fd7c2200640 1 -- 172.21.15.16:0/3587383507 wait complete. 2024-03-31T15:15:02.969 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-31T15:03:22.550787+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 16, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-31T15:14:06.417304+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '108', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 17, 'pool_name': 'cephfs_data', 'create_time': '2024-03-31T15:14:07.622562+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '109', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-31T15:15:02.969 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-31T15:15:03.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:03.121+0000 7f904d400640 1 Processor -- start 2024-03-31T15:15:03.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:03.121+0000 7f904d400640 1 -- start start 2024-03-31T15:15:03.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:03.125+0000 7f904d400640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f904815e140 0x7f904806b620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:03.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:03.125+0000 7f904d400640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f90480ae9a0 0x7f9048072480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:03.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:03.125+0000 7f904d400640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f904815e920 con 0x7f904806bb60 2024-03-31T15:15:03.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:03.125+0000 7f904d400640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f90480aee10 con 0x7f904815e140 2024-03-31T15:15:03.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:03.125+0000 7f904d400640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f904815ec00 con 0x7f90480ae9a0 2024-03-31T15:15:03.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:03.125+0000 7f9047e00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f904815e140 0x7f904806b620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:03.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:03.125+0000 7f903fe00640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f904806bb60 0x7f90480ae290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:36684/0 (socket says 172.21.15.16:36684) 2024-03-31T15:15:03.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:03.125+0000 7f903fe00640 1 -- 172.21.15.16:0/1924115864 learned_addr learned my addr 172.21.15.16:0/1924115864 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:15:03.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:03.125+0000 7f904ca00640 1 --2- 172.21.15.16:0/1924115864 >> v2:172.21.15.89:3300/0 conn(0x7f90480ae9a0 0x7f9048072480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:03.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:03.125+0000 7f9047e00640 1 -- 172.21.15.16:0/1924115864 >> v2:172.21.15.89:3300/0 conn(0x7f90480ae9a0 msgr2=0x7f9048072480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:15:03.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:03.125+0000 7f9047e00640 1 --2- 172.21.15.16:0/1924115864 >> v2:172.21.15.89:3300/0 conn(0x7f90480ae9a0 0x7f9048072480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:03.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:03.125+0000 7f9047e00640 1 -- 172.21.15.16:0/1924115864 >> v1:172.21.15.16:6789/0 conn(0x7f904806bb60 legacy=0x7f90480ae290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:03.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:03.125+0000 7f9047e00640 1 -- 172.21.15.16:0/1924115864 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f904815eee0 con 0x7f904815e140 2024-03-31T15:15:03.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:03.125+0000 7f9047400640 1 -- 172.21.15.16:0/1924115864 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2565454416 0 0) 0x7f904815e920 con 0x7f904806bb60 2024-03-31T15:15:03.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:03.129+0000 7f9047e00640 1 --2- 172.21.15.16:0/1924115864 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f904815e140 0x7f904806b620 secure :-1 s=READY pgs=1840 cs=0 l=1 rev1=1 crypto rx=0x7f90480b5ca0 tx=0x7f903801c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ce873234e1cd211 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:15:03.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:03.129+0000 7f9047400640 1 -- 172.21.15.16:0/1924115864 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90380b0070 con 0x7f904815e140 2024-03-31T15:15:03.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:03.129+0000 7f9047400640 1 -- 172.21.15.16:0/1924115864 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f90380ab600 con 0x7f904815e140 2024-03-31T15:15:03.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:03.129+0000 7f9047400640 1 -- 172.21.15.16:0/1924115864 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90380ab9d0 con 0x7f904815e140 2024-03-31T15:15:03.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:03.129+0000 7f904d400640 1 -- 172.21.15.16:0/1924115864 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f904815e140 msgr2=0x7f904806b620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:03.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:03.129+0000 7f904d400640 1 --2- 172.21.15.16:0/1924115864 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f904815e140 0x7f904806b620 secure :-1 s=READY pgs=1840 cs=0 l=1 rev1=1 crypto rx=0x7f90480b5ca0 tx=0x7f903801c480 comp rx=0 tx=0).stop 2024-03-31T15:15:03.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:03.129+0000 7f904d400640 1 -- 172.21.15.16:0/1924115864 shutdown_connections 2024-03-31T15:15:03.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:03.129+0000 7f904d400640 1 --2- 172.21.15.16:0/1924115864 >> v2:172.21.15.89:3300/0 conn(0x7f90480ae9a0 0x7f9048072480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:03.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:03.129+0000 7f904d400640 1 --2- 172.21.15.16:0/1924115864 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f904815e140 0x7f904806b620 unknown :-1 s=CLOSED pgs=1840 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:03.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:03.129+0000 7f904d400640 1 -- 172.21.15.16:0/1924115864 >> 172.21.15.16:0/1924115864 conn(0x7f90480b2b30 msgr2=0x7f90480b2f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:15:03.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:03.129+0000 7f904d400640 1 -- 172.21.15.16:0/1924115864 shutdown_connections 2024-03-31T15:15:03.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:03.129+0000 7f904d400640 1 -- 172.21.15.16:0/1924115864 wait complete. 2024-03-31T15:15:03.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:03.133+0000 7f904d400640 1 Processor -- start 2024-03-31T15:15:03.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:03.133+0000 7f904d400640 1 -- start start 2024-03-31T15:15:03.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:03.133+0000 7f904d400640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f904806bb60 0x7f904814ffa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:03.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:03.133+0000 7f904d400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f90480ae9a0 0x7f90480ce3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:03.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:03.133+0000 7f904d400640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f904815e140 0x7f90480b5580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:03.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:03.133+0000 7f904d400640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f9048073350 con 0x7f90480ae9a0 2024-03-31T15:15:03.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:03.133+0000 7f904d400640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f90480731d0 con 0x7f904815e140 2024-03-31T15:15:03.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:03.133+0000 7f904d400640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f904816af90 con 0x7f904806bb60 2024-03-31T15:15:03.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:03.133+0000 7f903fe00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f90480ae9a0 0x7f90480ce3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:03.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:03.137+0000 7f9047e00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f904806bb60 0x7f904814ffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:03.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:03.137+0000 7f903fe00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f90480ae9a0 0x7f90480ce3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50226/0 (socket says 172.21.15.16:50226) 2024-03-31T15:15:03.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:03.137+0000 7f903fe00640 1 -- 172.21.15.16:0/1558874263 learned_addr learned my addr 172.21.15.16:0/1558874263 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:15:03.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:03.137+0000 7f9047e00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f904806bb60 0x7f904814ffa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3301/0 says I am v2:172.21.15.16:39504/0 (socket says 172.21.15.16:39504) 2024-03-31T15:15:03.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:03.137+0000 7f904ca00640 1 --2- 172.21.15.16:0/1558874263 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f904815e140 0x7f90480b5580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:03.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:03.137+0000 7f903fe00640 1 -- 172.21.15.16:0/1558874263 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f904806bb60 msgr2=0x7f904814ffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:03.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:03.137+0000 7f903fe00640 1 --2- 172.21.15.16:0/1558874263 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f904806bb60 0x7f904814ffa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:03.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:03.137+0000 7f903fe00640 1 -- 172.21.15.16:0/1558874263 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f904815e140 msgr2=0x7f90480b5580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:03.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:03.137+0000 7f903fe00640 1 --2- 172.21.15.16:0/1558874263 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f904815e140 0x7f90480b5580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:03.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:03.137+0000 7f903fe00640 1 -- 172.21.15.16:0/1558874263 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f90480725a0 con 0x7f90480ae9a0 2024-03-31T15:15:03.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:03.137+0000 7f9047e00640 1 --2- 172.21.15.16:0/1558874263 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f904806bb60 0x7f904814ffa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:15:03.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:03.137+0000 7f904ca00640 1 --2- 172.21.15.16:0/1558874263 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f904815e140 0x7f90480b5580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:15:03.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:03.137+0000 7f903fe00640 1 --2- 172.21.15.16:0/1558874263 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f90480ae9a0 0x7f90480ce3d0 secure :-1 s=READY pgs=1841 cs=0 l=1 rev1=1 crypto rx=0x7f903000c4b0 tx=0x7f903000c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:15:03.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:03.137+0000 7f9044c00640 1 -- 172.21.15.16:0/1558874263 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9030018070 con 0x7f90480ae9a0 2024-03-31T15:15:03.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:03.137+0000 7f9044c00640 1 -- 172.21.15.16:0/1558874263 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9030007500 con 0x7f90480ae9a0 2024-03-31T15:15:03.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:03.137+0000 7f9044c00640 1 -- 172.21.15.16:0/1558874263 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9030004a30 con 0x7f90480ae9a0 2024-03-31T15:15:03.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:03.137+0000 7f904d400640 1 -- 172.21.15.16:0/1558874263 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f904815c300 con 0x7f90480ae9a0 2024-03-31T15:15:03.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:03.137+0000 7f904d400640 1 -- 172.21.15.16:0/1558874263 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f90480cd8f0 con 0x7f90480ae9a0 2024-03-31T15:15:03.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:03.137+0000 7f904d400640 1 -- 172.21.15.16:0/1558874263 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f904804f8c0 con 0x7f90480ae9a0 2024-03-31T15:15:03.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:03.137+0000 7f9044c00640 1 -- 172.21.15.16:0/1558874263 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f9030004bd0 con 0x7f90480ae9a0 2024-03-31T15:15:03.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:03.141+0000 7f9044c00640 1 --2- 172.21.15.16:0/1558874263 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f9024068380 0x7f902406a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:03.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:03.141+0000 7f9044c00640 1 -- 172.21.15.16:0/1558874263 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 5939+0+0 (secure 0 0 0) 0x7f9030085bf0 con 0x7f90480ae9a0 2024-03-31T15:15:03.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:03.141+0000 7f9047e00640 1 --2- 172.21.15.16:0/1558874263 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f9024068380 0x7f902406a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:03.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:03.141+0000 7f9047e00640 1 --2- 172.21.15.16:0/1558874263 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f9024068380 0x7f902406a840 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7f90380ad000 tx=0x7f90380a6000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:15:03.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:03.145+0000 7f9044c00640 1 -- 172.21.15.16:0/1558874263 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f9030055a60 con 0x7f90480ae9a0 2024-03-31T15:15:03.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:03.321+0000 7f904d400640 1 -- 172.21.15.16:0/1558874263 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f9048070c00 con 0x7f90480ae9a0 2024-03-31T15:15:03.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:03.321+0000 7f9044c00640 1 -- 172.21.15.16:0/1558874263 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v113) ==== 93+0+10 (secure 0 0 0) 0x7f9030059d60 con 0x7f90480ae9a0 2024-03-31T15:15:03.313 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2024-03-31T15:15:03.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:03.325+0000 7f904d400640 1 -- 172.21.15.16:0/1558874263 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f9024068380 msgr2=0x7f902406a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:03.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:03.325+0000 7f904d400640 1 --2- 172.21.15.16:0/1558874263 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f9024068380 0x7f902406a840 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7f90380ad000 tx=0x7f90380a6000 comp rx=0 tx=0).stop 2024-03-31T15:15:03.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:03.325+0000 7f904d400640 1 -- 172.21.15.16:0/1558874263 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f90480ae9a0 msgr2=0x7f90480ce3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:03.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:03.325+0000 7f904d400640 1 --2- 172.21.15.16:0/1558874263 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f90480ae9a0 0x7f90480ce3d0 secure :-1 s=READY pgs=1841 cs=0 l=1 rev1=1 crypto rx=0x7f903000c4b0 tx=0x7f903000c970 comp rx=0 tx=0).stop 2024-03-31T15:15:03.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:03.325+0000 7f904d400640 1 -- 172.21.15.16:0/1558874263 shutdown_connections 2024-03-31T15:15:03.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:03.325+0000 7f904d400640 1 --2- 172.21.15.16:0/1558874263 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f9024068380 0x7f902406a840 unknown :-1 s=CLOSED pgs=774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:03.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:03.325+0000 7f904d400640 1 --2- 172.21.15.16:0/1558874263 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f904815e140 0x7f90480b5580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:03.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:03.325+0000 7f904d400640 1 --2- 172.21.15.16:0/1558874263 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f90480ae9a0 0x7f90480ce3d0 unknown :-1 s=CLOSED pgs=1841 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:03.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:03.325+0000 7f904d400640 1 --2- 172.21.15.16:0/1558874263 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f904806bb60 0x7f904814ffa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:03.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:03.325+0000 7f904d400640 1 -- 172.21.15.16:0/1558874263 >> 172.21.15.16:0/1558874263 conn(0x7f90480b2b30 msgr2=0x7f90480701e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:15:03.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:03.325+0000 7f904d400640 1 -- 172.21.15.16:0/1558874263 shutdown_connections 2024-03-31T15:15:03.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:03.325+0000 7f904d400640 1 -- 172.21.15.16:0/1558874263 wait complete. 2024-03-31T15:15:03.334 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-31T15:15:03.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:03.485+0000 7f155f800640 1 Processor -- start 2024-03-31T15:15:03.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:03.485+0000 7f155f800640 1 -- start start 2024-03-31T15:15:03.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:03.489+0000 7f155f800640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f15580d0a60 0x7f15580cea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:03.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:03.489+0000 7f155f800640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f15580cef80 0x7f15581419c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:03.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:03.489+0000 7f155f800640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f155815ca70 con 0x7f155806f920 2024-03-31T15:15:03.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:03.489+0000 7f155f800640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f155813f670 con 0x7f15580cef80 2024-03-31T15:15:03.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:03.489+0000 7f155f800640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f155815cd50 con 0x7f15580d0a60 2024-03-31T15:15:03.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:03.489+0000 7f155d800640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f15580cef80 0x7f15581419c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:03.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:03.489+0000 7f155d800640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f15580cef80 0x7f15581419c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50234/0 (socket says 172.21.15.16:50234) 2024-03-31T15:15:03.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:03.489+0000 7f155d800640 1 -- 172.21.15.16:0/1725278068 learned_addr learned my addr 172.21.15.16:0/1725278068 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:15:03.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:03.489+0000 7f155ce00640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f155806f920 0x7f155806fd00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:36698/0 (socket says 172.21.15.16:36698) 2024-03-31T15:15:03.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:03.489+0000 7f154fe00640 1 --2- 172.21.15.16:0/1725278068 >> v2:172.21.15.89:3300/0 conn(0x7f15580d0a60 0x7f15580cea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:03.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:03.489+0000 7f155d800640 1 -- 172.21.15.16:0/1725278068 >> v2:172.21.15.89:3300/0 conn(0x7f15580d0a60 msgr2=0x7f15580cea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:15:03.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:03.489+0000 7f155d800640 1 --2- 172.21.15.16:0/1725278068 >> v2:172.21.15.89:3300/0 conn(0x7f15580d0a60 0x7f15580cea40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:03.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:03.489+0000 7f155d800640 1 -- 172.21.15.16:0/1725278068 >> v1:172.21.15.16:6789/0 conn(0x7f155806f920 legacy=0x7f155806fd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:03.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:03.489+0000 7f155d800640 1 -- 172.21.15.16:0/1725278068 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f155815d030 con 0x7f15580cef80 2024-03-31T15:15:03.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:03.493+0000 7f155d800640 1 --2- 172.21.15.16:0/1725278068 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f15580cef80 0x7f15581419c0 secure :-1 s=READY pgs=1843 cs=0 l=1 rev1=1 crypto rx=0x7f15580b5070 tx=0x7f155001c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=563d87c86a463884 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:15:03.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:03.493+0000 7f1557e00640 1 -- 172.21.15.16:0/1725278068 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15500a56c0 con 0x7f15580cef80 2024-03-31T15:15:03.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:03.493+0000 7f1557e00640 1 -- 172.21.15.16:0/1725278068 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f15500a5860 con 0x7f15580cef80 2024-03-31T15:15:03.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:03.493+0000 7f1557e00640 1 -- 172.21.15.16:0/1725278068 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15500a5c30 con 0x7f15580cef80 2024-03-31T15:15:03.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:03.493+0000 7f155f800640 1 -- 172.21.15.16:0/1725278068 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f15580cef80 msgr2=0x7f15581419c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:03.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:03.493+0000 7f155f800640 1 --2- 172.21.15.16:0/1725278068 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f15580cef80 0x7f15581419c0 secure :-1 s=READY pgs=1843 cs=0 l=1 rev1=1 crypto rx=0x7f15580b5070 tx=0x7f155001c9f0 comp rx=0 tx=0).stop 2024-03-31T15:15:03.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:03.493+0000 7f155f800640 1 -- 172.21.15.16:0/1725278068 shutdown_connections 2024-03-31T15:15:03.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:03.493+0000 7f155f800640 1 --2- 172.21.15.16:0/1725278068 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f15580cef80 0x7f15581419c0 unknown :-1 s=CLOSED pgs=1843 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:03.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:03.493+0000 7f155f800640 1 --2- 172.21.15.16:0/1725278068 >> v2:172.21.15.89:3300/0 conn(0x7f15580d0a60 0x7f15580cea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:03.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:03.493+0000 7f155f800640 1 -- 172.21.15.16:0/1725278068 >> 172.21.15.16:0/1725278068 conn(0x7f1558067930 msgr2=0x7f15580b3ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:15:03.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:03.493+0000 7f155f800640 1 -- 172.21.15.16:0/1725278068 shutdown_connections 2024-03-31T15:15:03.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:03.493+0000 7f155f800640 1 -- 172.21.15.16:0/1725278068 wait complete. 2024-03-31T15:15:03.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:03.493+0000 7f155f800640 1 Processor -- start 2024-03-31T15:15:03.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:03.493+0000 7f155f800640 1 -- start start 2024-03-31T15:15:03.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:03.493+0000 7f155f800640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f155806f920 0x7f1558131690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:03.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:03.497+0000 7f155f800640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f15580cef80 0x7f1558166500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:03.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:03.497+0000 7f155f800640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f15580d0a60 0x7f155815e230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:03.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:03.497+0000 7f155f800640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f1558169230 con 0x7f155806f920 2024-03-31T15:15:03.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:03.497+0000 7f155f800640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f15581690b0 con 0x7f15580cef80 2024-03-31T15:15:03.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:03.497+0000 7f155f800640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f15581693b0 con 0x7f15580d0a60 2024-03-31T15:15:03.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:03.497+0000 7f155ce00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f155806f920 0x7f1558131690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:03.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:03.497+0000 7f155ce00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f155806f920 0x7f1558131690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50236/0 (socket says 172.21.15.16:50236) 2024-03-31T15:15:03.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:03.497+0000 7f155ce00640 1 -- 172.21.15.16:0/4061130429 learned_addr learned my addr 172.21.15.16:0/4061130429 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:15:03.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:03.497+0000 7f154fe00640 1 --2- 172.21.15.16:0/4061130429 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f15580cef80 0x7f1558166500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:03.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:03.497+0000 7f155d800640 1 --2- 172.21.15.16:0/4061130429 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f15580d0a60 0x7f155815e230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:03.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:03.497+0000 7f155ce00640 1 -- 172.21.15.16:0/4061130429 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f15580d0a60 msgr2=0x7f155815e230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:03.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:03.497+0000 7f155ce00640 1 --2- 172.21.15.16:0/4061130429 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f15580d0a60 0x7f155815e230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:03.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:03.497+0000 7f155ce00640 1 -- 172.21.15.16:0/4061130429 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f15580cef80 msgr2=0x7f1558166500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:03.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:03.497+0000 7f155ce00640 1 --2- 172.21.15.16:0/4061130429 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f15580cef80 0x7f1558166500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:03.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:03.497+0000 7f155ce00640 1 -- 172.21.15.16:0/4061130429 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15581651c0 con 0x7f155806f920 2024-03-31T15:15:03.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:03.497+0000 7f154fe00640 1 --2- 172.21.15.16:0/4061130429 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f15580cef80 0x7f1558166500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:15:03.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:03.497+0000 7f155d800640 1 --2- 172.21.15.16:0/4061130429 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f15580d0a60 0x7f155815e230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:15:03.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:03.497+0000 7f155ce00640 1 --2- 172.21.15.16:0/4061130429 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f155806f920 0x7f1558131690 secure :-1 s=READY pgs=1844 cs=0 l=1 rev1=1 crypto rx=0x7f154800c420 tx=0x7f154800c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:15:03.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:03.497+0000 7f1555600640 1 -- 172.21.15.16:0/4061130429 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15480074e0 con 0x7f155806f920 2024-03-31T15:15:03.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:03.497+0000 7f1555600640 1 -- 172.21.15.16:0/4061130429 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1548007680 con 0x7f155806f920 2024-03-31T15:15:03.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:03.497+0000 7f155f800640 1 -- 172.21.15.16:0/4061130429 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1558165460 con 0x7f155806f920 2024-03-31T15:15:03.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:03.497+0000 7f1555600640 1 -- 172.21.15.16:0/4061130429 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1548004a30 con 0x7f155806f920 2024-03-31T15:15:03.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:03.497+0000 7f155f800640 1 -- 172.21.15.16:0/4061130429 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1558165a20 con 0x7f155806f920 2024-03-31T15:15:03.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:03.497+0000 7f155f800640 1 -- 172.21.15.16:0/4061130429 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1558069950 con 0x7f155806f920 2024-03-31T15:15:03.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:03.497+0000 7f1555600640 1 -- 172.21.15.16:0/4061130429 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f154801a020 con 0x7f155806f920 2024-03-31T15:15:03.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:03.501+0000 7f1555600640 1 --2- 172.21.15.16:0/4061130429 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f1534068380 0x7f153406a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:03.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:03.501+0000 7f1555600640 1 -- 172.21.15.16:0/4061130429 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 5939+0+0 (secure 0 0 0) 0x7f1548086220 con 0x7f155806f920 2024-03-31T15:15:03.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:03.501+0000 7f154fe00640 1 --2- 172.21.15.16:0/4061130429 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f1534068380 0x7f153406a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:03.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:03.501+0000 7f154fe00640 1 --2- 172.21.15.16:0/4061130429 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f1534068380 0x7f153406a840 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f15580b3fe0 tx=0x7f1540008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:15:03.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:03.505+0000 7f1555600640 1 -- 172.21.15.16:0/4061130429 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f1548056780 con 0x7f155806f920 2024-03-31T15:15:03.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:03.669+0000 7f155f800640 1 -- 172.21.15.16:0/4061130429 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f15580b50e0 con 0x7f155806f920 2024-03-31T15:15:03.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:03.669+0000 7f1555600640 1 -- 172.21.15.16:0/4061130429 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v113) ==== 104+0+11 (secure 0 0 0) 0x7f1548056080 con 0x7f155806f920 2024-03-31T15:15:03.660 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2024-03-31T15:15:03.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:03.673+0000 7f155f800640 1 -- 172.21.15.16:0/4061130429 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f1534068380 msgr2=0x7f153406a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:03.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:03.673+0000 7f155f800640 1 --2- 172.21.15.16:0/4061130429 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f1534068380 0x7f153406a840 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f15580b3fe0 tx=0x7f1540008040 comp rx=0 tx=0).stop 2024-03-31T15:15:03.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:03.673+0000 7f155f800640 1 -- 172.21.15.16:0/4061130429 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f155806f920 msgr2=0x7f1558131690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:03.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:03.673+0000 7f155f800640 1 --2- 172.21.15.16:0/4061130429 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f155806f920 0x7f1558131690 secure :-1 s=READY pgs=1844 cs=0 l=1 rev1=1 crypto rx=0x7f154800c420 tx=0x7f154800c8e0 comp rx=0 tx=0).stop 2024-03-31T15:15:03.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:03.673+0000 7f155f800640 1 -- 172.21.15.16:0/4061130429 shutdown_connections 2024-03-31T15:15:03.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:03.673+0000 7f155f800640 1 --2- 172.21.15.16:0/4061130429 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f1534068380 0x7f153406a840 unknown :-1 s=CLOSED pgs=775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:03.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:03.673+0000 7f155f800640 1 --2- 172.21.15.16:0/4061130429 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f15580d0a60 0x7f155815e230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:03.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:03.673+0000 7f155f800640 1 --2- 172.21.15.16:0/4061130429 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f15580cef80 0x7f1558166500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:03.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:03.673+0000 7f155f800640 1 --2- 172.21.15.16:0/4061130429 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f155806f920 0x7f1558131690 unknown :-1 s=CLOSED pgs=1844 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:03.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:03.673+0000 7f155f800640 1 -- 172.21.15.16:0/4061130429 >> 172.21.15.16:0/4061130429 conn(0x7f1558067930 msgr2=0x7f1558141010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:15:03.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:03.673+0000 7f155f800640 1 -- 172.21.15.16:0/4061130429 shutdown_connections 2024-03-31T15:15:03.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:03.673+0000 7f155f800640 1 -- 172.21.15.16:0/4061130429 wait complete. 2024-03-31T15:15:03.681 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-31T15:15:03.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:03.837+0000 7fb5ff600640 1 Processor -- start 2024-03-31T15:15:03.829 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:03.837+0000 7fb5ff600640 1 -- start start 2024-03-31T15:15:03.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:03.837+0000 7fb5ff600640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb5f80b60a0 0x7fb5f806fc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:03.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:03.837+0000 7fb5ff600640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fb5f80d0c00 0x7fb5f8070180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:03.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:03.837+0000 7fb5ff600640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb5f806f540 con 0x7fb5f806b620 2024-03-31T15:15:03.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:03.837+0000 7fb5ff600640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7fb5f806f820 con 0x7fb5f80d0c00 2024-03-31T15:15:03.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:03.837+0000 7fb5ff600640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fb5f80b4a40 con 0x7fb5f80b60a0 2024-03-31T15:15:03.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:03.837+0000 7fb5fcc00640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fb5f806b620 0x7fb5f806ba00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:36700/0 (socket says 172.21.15.16:36700) 2024-03-31T15:15:03.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:03.837+0000 7fb5fcc00640 1 -- 172.21.15.16:0/3240782145 learned_addr learned my addr 172.21.15.16:0/3240782145 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:15:03.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:03.837+0000 7fb5fd600640 1 --2- 172.21.15.16:0/3240782145 >> v2:172.21.15.89:3300/0 conn(0x7fb5f80d0c00 0x7fb5f8070180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:03.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:03.837+0000 7fb5efe00640 1 --2- 172.21.15.16:0/3240782145 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb5f80b60a0 0x7fb5f806fc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:03.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:03.841+0000 7fb5f7e00640 1 -- 172.21.15.16:0/3240782145 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2651561435 0 0) 0x7fb5f806f540 con 0x7fb5f806b620 2024-03-31T15:15:03.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:03.841+0000 7fb5f7e00640 1 -- 172.21.15.16:0/3240782145 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb5dc003530 con 0x7fb5f806b620 2024-03-31T15:15:03.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:03.841+0000 7fb5fd600640 1 -- 172.21.15.16:0/3240782145 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb5f80b60a0 msgr2=0x7fb5f806fc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:15:03.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:03.841+0000 7fb5fd600640 1 --2- 172.21.15.16:0/3240782145 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb5f80b60a0 0x7fb5f806fc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:03.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:03.841+0000 7fb5fd600640 1 -- 172.21.15.16:0/3240782145 >> v1:172.21.15.16:6789/0 conn(0x7fb5f806b620 legacy=0x7fb5f806ba00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:03.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:03.841+0000 7fb5fd600640 1 -- 172.21.15.16:0/3240782145 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5f80b43f0 con 0x7fb5f80d0c00 2024-03-31T15:15:03.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:03.841+0000 7fb5fd600640 1 --2- 172.21.15.16:0/3240782145 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb5f80d0c00 0x7fb5f8070180 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7fb5f0009d90 tx=0x7fb5f001c8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ed5da2ac0752cf55 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:15:03.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:03.841+0000 7fb5f7e00640 1 -- 172.21.15.16:0/3240782145 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5f00b2070 con 0x7fb5f80d0c00 2024-03-31T15:15:03.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:03.841+0000 7fb5f7e00640 1 -- 172.21.15.16:0/3240782145 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb5f00a5880 con 0x7fb5f80d0c00 2024-03-31T15:15:03.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:03.841+0000 7fb5f7e00640 1 -- 172.21.15.16:0/3240782145 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5f00a5c30 con 0x7fb5f80d0c00 2024-03-31T15:15:03.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:03.845+0000 7fb5ff600640 1 -- 172.21.15.16:0/3240782145 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb5f80d0c00 msgr2=0x7fb5f8070180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:03.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:03.845+0000 7fb5ff600640 1 --2- 172.21.15.16:0/3240782145 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb5f80d0c00 0x7fb5f8070180 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7fb5f0009d90 tx=0x7fb5f001c8b0 comp rx=0 tx=0).stop 2024-03-31T15:15:03.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:03.845+0000 7fb5ff600640 1 -- 172.21.15.16:0/3240782145 shutdown_connections 2024-03-31T15:15:03.835 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:03.845+0000 7fb5ff600640 1 --2- 172.21.15.16:0/3240782145 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb5f80d0c00 0x7fb5f8070180 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:03.835 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:03.845+0000 7fb5ff600640 1 --2- 172.21.15.16:0/3240782145 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb5f80b60a0 0x7fb5f806fc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:03.835 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:03.845+0000 7fb5ff600640 1 -- 172.21.15.16:0/3240782145 >> 172.21.15.16:0/3240782145 conn(0x7fb5f806e670 msgr2=0x7fb5f806ea50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:15:03.835 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:03.845+0000 7fb5ff600640 1 -- 172.21.15.16:0/3240782145 shutdown_connections 2024-03-31T15:15:03.835 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:03.845+0000 7fb5ff600640 1 -- 172.21.15.16:0/3240782145 wait complete. 2024-03-31T15:15:03.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:03.845+0000 7fb5ff600640 1 Processor -- start 2024-03-31T15:15:03.839 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:03.845+0000 7fb5ff600640 1 -- start start 2024-03-31T15:15:03.839 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:03.845+0000 7fb5ff600640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb5f806b620 0x7fb5f8149d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:03.839 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:03.845+0000 7fb5ff600640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb5f80b60a0 0x7fb5f81552a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:03.839 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:03.845+0000 7fb5ff600640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb5f80d0c00 0x7fb5f81557e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:03.839 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:03.845+0000 7fb5ff600640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fb5f816bd00 con 0x7fb5f80d0c00 2024-03-31T15:15:03.839 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:03.845+0000 7fb5ff600640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fb5f816bb80 con 0x7fb5f806b620 2024-03-31T15:15:03.839 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:03.845+0000 7fb5ff600640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fb5f816be80 con 0x7fb5f80b60a0 2024-03-31T15:15:03.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:03.845+0000 7fb5fd600640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb5f80d0c00 0x7fb5f81557e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:03.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:03.845+0000 7fb5fd600640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb5f80d0c00 0x7fb5f81557e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50246/0 (socket says 172.21.15.16:50246) 2024-03-31T15:15:03.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:03.845+0000 7fb5fd600640 1 -- 172.21.15.16:0/698845404 learned_addr learned my addr 172.21.15.16:0/698845404 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:15:03.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:03.845+0000 7fb5fcc00640 1 --2- 172.21.15.16:0/698845404 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb5f806b620 0x7fb5f8149d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:03.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:03.845+0000 7fb5efe00640 1 --2- 172.21.15.16:0/698845404 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb5f80b60a0 0x7fb5f81552a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:03.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:03.845+0000 7fb5fd600640 1 -- 172.21.15.16:0/698845404 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb5f80b60a0 msgr2=0x7fb5f81552a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:03.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:03.845+0000 7fb5fd600640 1 --2- 172.21.15.16:0/698845404 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb5f80b60a0 0x7fb5f81552a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:03.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:03.845+0000 7fb5fd600640 1 -- 172.21.15.16:0/698845404 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb5f806b620 msgr2=0x7fb5f8149d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:03.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:03.845+0000 7fb5fd600640 1 --2- 172.21.15.16:0/698845404 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb5f806b620 0x7fb5f8149d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:03.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:03.845+0000 7fb5fd600640 1 -- 172.21.15.16:0/698845404 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5f8155e20 con 0x7fb5f80d0c00 2024-03-31T15:15:03.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:03.845+0000 7fb5efe00640 1 --2- 172.21.15.16:0/698845404 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb5f80b60a0 0x7fb5f81552a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:15:03.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:03.845+0000 7fb5fd600640 1 --2- 172.21.15.16:0/698845404 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb5f80d0c00 0x7fb5f81557e0 secure :-1 s=READY pgs=1846 cs=0 l=1 rev1=1 crypto rx=0x7fb5f00b0040 tx=0x7fb5f0019650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:15:03.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:03.849+0000 7fb5f5600640 1 -- 172.21.15.16:0/698845404 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5f00b2050 con 0x7fb5f80d0c00 2024-03-31T15:15:03.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:03.849+0000 7fb5f5600640 1 -- 172.21.15.16:0/698845404 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb5f00be070 con 0x7fb5f80d0c00 2024-03-31T15:15:03.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:03.849+0000 7fb5f5600640 1 -- 172.21.15.16:0/698845404 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5f0003dd0 con 0x7fb5f80d0c00 2024-03-31T15:15:03.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:03.849+0000 7fb5ff600640 1 -- 172.21.15.16:0/698845404 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb5f8156100 con 0x7fb5f80d0c00 2024-03-31T15:15:03.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:03.849+0000 7fb5ff600640 1 -- 172.21.15.16:0/698845404 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb5f81566c0 con 0x7fb5f80d0c00 2024-03-31T15:15:03.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:03.849+0000 7fb5ff600640 1 -- 172.21.15.16:0/698845404 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb5f8069950 con 0x7fb5f80d0c00 2024-03-31T15:15:03.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:03.849+0000 7fb5f5600640 1 -- 172.21.15.16:0/698845404 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fb5f0004040 con 0x7fb5f80d0c00 2024-03-31T15:15:03.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:03.849+0000 7fb5f5600640 1 --2- 172.21.15.16:0/698845404 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fb5d4068380 0x7fb5d406a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:03.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:03.849+0000 7fb5f5600640 1 -- 172.21.15.16:0/698845404 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 5939+0+0 (secure 0 0 0) 0x7fb5f0121ad0 con 0x7fb5f80d0c00 2024-03-31T15:15:03.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:03.853+0000 7fb5fcc00640 1 --2- 172.21.15.16:0/698845404 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fb5d4068380 0x7fb5d406a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:03.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:03.853+0000 7fb5fcc00640 1 --2- 172.21.15.16:0/698845404 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fb5d4068380 0x7fb5d406a840 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7fb5e8005890 tx=0x7fb5e8009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:15:03.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:03.857+0000 7fb5f5600640 1 -- 172.21.15.16:0/698845404 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fb5f00c8020 con 0x7fb5f80d0c00 2024-03-31T15:15:04.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:04.021+0000 7fb5ff600640 1 -- 172.21.15.16:0/698845404 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fb5f814df20 con 0x7fb5f80d0c00 2024-03-31T15:15:04.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:04.021+0000 7fb5f5600640 1 -- 172.21.15.16:0/698845404 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v113) ==== 100+0+11 (secure 0 0 0) 0x7fb5f0016560 con 0x7fb5f80d0c00 2024-03-31T15:15:04.010 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 64 2024-03-31T15:15:04.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:04.021+0000 7fb5ff600640 1 -- 172.21.15.16:0/698845404 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fb5d4068380 msgr2=0x7fb5d406a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:04.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:04.021+0000 7fb5ff600640 1 --2- 172.21.15.16:0/698845404 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fb5d4068380 0x7fb5d406a840 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7fb5e8005890 tx=0x7fb5e8009040 comp rx=0 tx=0).stop 2024-03-31T15:15:04.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:04.021+0000 7fb5ff600640 1 -- 172.21.15.16:0/698845404 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb5f80d0c00 msgr2=0x7fb5f81557e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:04.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:04.021+0000 7fb5ff600640 1 --2- 172.21.15.16:0/698845404 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb5f80d0c00 0x7fb5f81557e0 secure :-1 s=READY pgs=1846 cs=0 l=1 rev1=1 crypto rx=0x7fb5f00b0040 tx=0x7fb5f0019650 comp rx=0 tx=0).stop 2024-03-31T15:15:04.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:04.021+0000 7fb5ff600640 1 -- 172.21.15.16:0/698845404 shutdown_connections 2024-03-31T15:15:04.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:04.021+0000 7fb5ff600640 1 --2- 172.21.15.16:0/698845404 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fb5d4068380 0x7fb5d406a840 unknown :-1 s=CLOSED pgs=776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:04.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:04.021+0000 7fb5ff600640 1 --2- 172.21.15.16:0/698845404 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb5f80d0c00 0x7fb5f81557e0 unknown :-1 s=CLOSED pgs=1846 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:04.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:04.021+0000 7fb5ff600640 1 --2- 172.21.15.16:0/698845404 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb5f80b60a0 0x7fb5f81552a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:04.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:04.021+0000 7fb5ff600640 1 --2- 172.21.15.16:0/698845404 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb5f806b620 0x7fb5f8149d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:04.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:04.021+0000 7fb5ff600640 1 -- 172.21.15.16:0/698845404 >> 172.21.15.16:0/698845404 conn(0x7fb5f806e670 msgr2=0x7fb5f815c540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:15:04.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:04.021+0000 7fb5ff600640 1 -- 172.21.15.16:0/698845404 shutdown_connections 2024-03-31T15:15:04.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:04.021+0000 7fb5ff600640 1 -- 172.21.15.16:0/698845404 wait complete. 2024-03-31T15:15:04.034 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-31T15:15:04.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:04.185+0000 7f28f6c00640 1 Processor -- start 2024-03-31T15:15:04.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:04.185+0000 7f28f6c00640 1 -- start start 2024-03-31T15:15:04.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:04.189+0000 7f28f6c00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f28f0145e40 0x7f28f00cc740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:04.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:04.189+0000 7f28f6c00640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f28f00d0920 con 0x7f28f00ccc80 2024-03-31T15:15:04.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:04.189+0000 7f28f6c00640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f28f00d0c00 con 0x7f28f0070030 2024-03-31T15:15:04.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:04.189+0000 7f28f6c00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f28f00d06d0 con 0x7f28f0145e40 2024-03-31T15:15:04.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:04.189+0000 7f28e7400640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f28f00ccc80 0x7f28f006f920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:36708/0 (socket says 172.21.15.16:36708) 2024-03-31T15:15:04.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:04.189+0000 7f28e7400640 1 -- 172.21.15.16:0/2611760766 learned_addr learned my addr 172.21.15.16:0/2611760766 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:15:04.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:04.189+0000 7f28efe00640 1 --2- 172.21.15.16:0/2611760766 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f28f0145e40 0x7f28f00cc740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:04.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:04.189+0000 7f28ef400640 1 -- 172.21.15.16:0/2611760766 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2062268351 0 0) 0x7f28f00d0920 con 0x7f28f00ccc80 2024-03-31T15:15:04.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:04.189+0000 7f28ef400640 1 -- 172.21.15.16:0/2611760766 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f28d4003530 con 0x7f28f00ccc80 2024-03-31T15:15:04.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:04.189+0000 7f28ef400640 1 -- 172.21.15.16:0/2611760766 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2923159839 0 0) 0x7f28d4003530 con 0x7f28f00ccc80 2024-03-31T15:15:04.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:04.189+0000 7f28ef400640 1 -- 172.21.15.16:0/2611760766 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f28f0145e40 msgr2=0x7f28f00cc740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:15:04.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:04.189+0000 7f28ef400640 1 --2- 172.21.15.16:0/2611760766 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f28f0145e40 0x7f28f00cc740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:04.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:04.189+0000 7f28ef400640 1 -- 172.21.15.16:0/2611760766 >> v1:172.21.15.89:6789/0 conn(0x7f28f0070030 legacy=0x7f28f00d2010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:04.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:04.189+0000 7f28ef400640 1 -- 172.21.15.16:0/2611760766 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28f00d1810 con 0x7f28f00ccc80 2024-03-31T15:15:04.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:04.189+0000 7f28ef400640 1 -- 172.21.15.16:0/2611760766 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 214807623 0 0) 0x7f28d8001680 con 0x7f28f00ccc80 2024-03-31T15:15:04.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:04.189+0000 7f28ef400640 1 -- 172.21.15.16:0/2611760766 >> v1:172.21.15.16:6789/0 conn(0x7f28f00ccc80 legacy=0x7f28f006f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:04.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:04.189+0000 7f28ef400640 1 --2- 172.21.15.16:0/2611760766 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f28d40039f0 0x7f28d4003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:04.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:04.189+0000 7f28ef400640 1 --2- 172.21.15.16:0/2611760766 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f28f0145e40 0x7f28d4004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:04.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:04.189+0000 7f28ef400640 1 --2- 172.21.15.16:0/2611760766 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f28d4004d50 0x7f28d40051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:04.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:04.189+0000 7f28ef400640 1 -- 172.21.15.16:0/2611760766 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f28f00d06d0 con 0x7f28d4004d50 2024-03-31T15:15:04.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:04.189+0000 7f28ef400640 1 -- 172.21.15.16:0/2611760766 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f28d400bab0 con 0x7f28d40039f0 2024-03-31T15:15:04.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:04.189+0000 7f28ef400640 1 -- 172.21.15.16:0/2611760766 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f28d400bcb0 con 0x7f28f0145e40 2024-03-31T15:15:04.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:04.189+0000 7f28efe00640 1 --2- 172.21.15.16:0/2611760766 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f28d40039f0 0x7f28d4003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:04.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:04.189+0000 7f28f4c00640 1 --2- 172.21.15.16:0/2611760766 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f28d4004d50 0x7f28d40051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:04.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:04.189+0000 7f28e7400640 1 --2- 172.21.15.16:0/2611760766 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f28f0145e40 0x7f28d4004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:04.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:04.189+0000 7f28f4c00640 1 -- 172.21.15.16:0/2611760766 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f28f0145e40 msgr2=0x7f28d4004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:15:04.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:04.189+0000 7f28f4c00640 1 --2- 172.21.15.16:0/2611760766 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f28f0145e40 0x7f28d4004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:04.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:04.189+0000 7f28f4c00640 1 -- 172.21.15.16:0/2611760766 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f28d40039f0 msgr2=0x7f28d4003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:15:04.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:04.189+0000 7f28f4c00640 1 --2- 172.21.15.16:0/2611760766 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f28d40039f0 0x7f28d4003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:04.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:04.189+0000 7f28f4c00640 1 -- 172.21.15.16:0/2611760766 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28d4003530 con 0x7f28d4004d50 2024-03-31T15:15:04.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:04.189+0000 7f28efe00640 1 --2- 172.21.15.16:0/2611760766 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f28d40039f0 0x7f28d4003e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:15:04.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:04.197+0000 7f28f4c00640 1 --2- 172.21.15.16:0/2611760766 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f28d4004d50 0x7f28d40051d0 secure :-1 s=READY pgs=1848 cs=0 l=1 rev1=1 crypto rx=0x7f28e8001750 tx=0x7f28e801c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ba06c8053e5df1b server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:15:04.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:04.197+0000 7f28ef400640 1 -- 172.21.15.16:0/2611760766 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28e80013a0 con 0x7f28d4004d50 2024-03-31T15:15:04.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:04.197+0000 7f28ef400640 1 -- 172.21.15.16:0/2611760766 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f28e80052f0 con 0x7f28d4004d50 2024-03-31T15:15:04.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:04.197+0000 7f28ef400640 1 -- 172.21.15.16:0/2611760766 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28e8003840 con 0x7f28d4004d50 2024-03-31T15:15:04.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:04.197+0000 7f28f6c00640 1 -- 172.21.15.16:0/2611760766 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f28d4004d50 msgr2=0x7f28d40051d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:04.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:04.197+0000 7f28f6c00640 1 --2- 172.21.15.16:0/2611760766 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f28d4004d50 0x7f28d40051d0 secure :-1 s=READY pgs=1848 cs=0 l=1 rev1=1 crypto rx=0x7f28e8001750 tx=0x7f28e801c3e0 comp rx=0 tx=0).stop 2024-03-31T15:15:04.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:04.197+0000 7f28f4c00640 1 -- 172.21.15.16:0/2611760766 reap_dead start 2024-03-31T15:15:04.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:04.197+0000 7f28f6c00640 1 -- 172.21.15.16:0/2611760766 shutdown_connections 2024-03-31T15:15:04.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:04.197+0000 7f28f6c00640 1 -- 172.21.15.16:0/2611760766 >> 172.21.15.16:0/2611760766 conn(0x7f28f0067330 msgr2=0x7f28f0067740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:15:04.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:04.197+0000 7f28f6c00640 1 -- 172.21.15.16:0/2611760766 shutdown_connections 2024-03-31T15:15:04.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:04.197+0000 7f28f6c00640 1 -- 172.21.15.16:0/2611760766 wait complete. 2024-03-31T15:15:04.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:04.197+0000 7f28f6c00640 1 Processor -- start 2024-03-31T15:15:04.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:04.197+0000 7f28f6c00640 1 -- start start 2024-03-31T15:15:04.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:04.197+0000 7f28f6c00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f28d4004d50 0x7f28f00aad00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:04.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:04.197+0000 7f28f6c00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f28f006d570 0x7f28f00a94c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:04.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:04.197+0000 7f28f6c00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f28f0144550 0x7f28f006dd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:04.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:04.197+0000 7f28f6c00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f28f00d0ee0 con 0x7f28f006d570 2024-03-31T15:15:04.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:04.197+0000 7f28f6c00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f28f00ce9e0 con 0x7f28f0144550 2024-03-31T15:15:04.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:04.197+0000 7f28f6c00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f28f00b50c0 con 0x7f28d4004d50 2024-03-31T15:15:04.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:04.201+0000 7f28e7400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f28f006d570 0x7f28f00a94c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:04.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:04.201+0000 7f28e7400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f28f006d570 0x7f28f00a94c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50258/0 (socket says 172.21.15.16:50258) 2024-03-31T15:15:04.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:04.201+0000 7f28e7400640 1 -- 172.21.15.16:0/3389944980 learned_addr learned my addr 172.21.15.16:0/3389944980 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:15:04.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:04.201+0000 7f28efe00640 1 --2- 172.21.15.16:0/3389944980 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f28d4004d50 0x7f28f00aad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:04.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:04.201+0000 7f28f4c00640 1 --2- 172.21.15.16:0/3389944980 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f28f0144550 0x7f28f006dd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:04.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:04.201+0000 7f28e7400640 1 -- 172.21.15.16:0/3389944980 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f28d4004d50 msgr2=0x7f28f00aad00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:04.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:04.201+0000 7f28e7400640 1 --2- 172.21.15.16:0/3389944980 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f28d4004d50 0x7f28f00aad00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:04.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:04.201+0000 7f28e7400640 1 -- 172.21.15.16:0/3389944980 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f28f0144550 msgr2=0x7f28f006dd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:04.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:04.201+0000 7f28e7400640 1 --2- 172.21.15.16:0/3389944980 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f28f0144550 0x7f28f006dd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:04.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:04.201+0000 7f28e7400640 1 -- 172.21.15.16:0/3389944980 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28f00a9c60 con 0x7f28f006d570 2024-03-31T15:15:04.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:04.201+0000 7f28f4c00640 1 --2- 172.21.15.16:0/3389944980 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f28f0144550 0x7f28f006dd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:15:04.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:04.201+0000 7f28e7400640 1 --2- 172.21.15.16:0/3389944980 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f28f006d570 0x7f28f00a94c0 secure :-1 s=READY pgs=1849 cs=0 l=1 rev1=1 crypto rx=0x7f28d8007290 tx=0x7f28d8007690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:15:04.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:04.201+0000 7f28ecc00640 1 -- 172.21.15.16:0/3389944980 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28d8001640 con 0x7f28f006d570 2024-03-31T15:15:04.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:04.201+0000 7f28ecc00640 1 -- 172.21.15.16:0/3389944980 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f28d8018070 con 0x7f28f006d570 2024-03-31T15:15:04.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:04.201+0000 7f28ecc00640 1 -- 172.21.15.16:0/3389944980 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28d800fc00 con 0x7f28f006d570 2024-03-31T15:15:04.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:04.201+0000 7f28f6c00640 1 -- 172.21.15.16:0/3389944980 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f28f00a9f40 con 0x7f28f006d570 2024-03-31T15:15:04.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:04.201+0000 7f28f6c00640 1 -- 172.21.15.16:0/3389944980 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f28f00aa500 con 0x7f28f006d570 2024-03-31T15:15:04.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:04.201+0000 7f28f6c00640 1 -- 172.21.15.16:0/3389944980 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f28f0069950 con 0x7f28f006d570 2024-03-31T15:15:04.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:04.201+0000 7f28ecc00640 1 -- 172.21.15.16:0/3389944980 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f28d801d460 con 0x7f28f006d570 2024-03-31T15:15:04.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:04.201+0000 7f28ecc00640 1 --2- 172.21.15.16:0/3389944980 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f28cc0683b0 0x7f28cc06a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:04.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:04.201+0000 7f28ecc00640 1 -- 172.21.15.16:0/3389944980 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 5939+0+0 (secure 0 0 0) 0x7f28d8086b30 con 0x7f28f006d570 2024-03-31T15:15:04.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:04.205+0000 7f28efe00640 1 --2- 172.21.15.16:0/3389944980 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f28cc0683b0 0x7f28cc06a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:04.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:04.205+0000 7f28efe00640 1 --2- 172.21.15.16:0/3389944980 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f28cc0683b0 0x7f28cc06a870 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7f28e0009fd0 tx=0x7f28e000b040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:15:04.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:04.209+0000 7f28ecc00640 1 -- 172.21.15.16:0/3389944980 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f28d8056ff0 con 0x7f28f006d570 2024-03-31T15:15:04.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:04.433+0000 7f28f6c00640 1 -- 172.21.15.16:0/3389944980 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f28f00cf3c0 con 0x7f28f006d570 2024-03-31T15:15:04.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:04.433+0000 7f28ecc00640 1 -- 172.21.15.16:0/3389944980 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 4 entries v113) ==== 80+0+244 (secure 0 0 0) 0x7f28d80568f0 con 0x7f28f006d570 2024-03-31T15:15:04.422 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:6833/4142860904 2024-04-01T15:14:53.117187+0000 2024-03-31T15:15:04.422 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:6835/2564952016 2024-04-01T15:14:39.715615+0000 2024-03-31T15:15:04.422 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:6835/4142860904 2024-04-01T15:14:53.117187+0000 2024-03-31T15:15:04.422 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:6833/2564952016 2024-04-01T15:14:39.715615+0000 2024-03-31T15:15:04.423 INFO:teuthology.orchestra.run.smithi016.stderr:listed 4 entries 2024-03-31T15:15:04.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:04.433+0000 7f28f6c00640 1 -- 172.21.15.16:0/3389944980 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f28cc0683b0 msgr2=0x7f28cc06a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:04.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:04.433+0000 7f28f6c00640 1 --2- 172.21.15.16:0/3389944980 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f28cc0683b0 0x7f28cc06a870 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7f28e0009fd0 tx=0x7f28e000b040 comp rx=0 tx=0).stop 2024-03-31T15:15:04.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:04.433+0000 7f28f6c00640 1 -- 172.21.15.16:0/3389944980 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f28f006d570 msgr2=0x7f28f00a94c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:04.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:04.433+0000 7f28f6c00640 1 --2- 172.21.15.16:0/3389944980 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f28f006d570 0x7f28f00a94c0 secure :-1 s=READY pgs=1849 cs=0 l=1 rev1=1 crypto rx=0x7f28d8007290 tx=0x7f28d8007690 comp rx=0 tx=0).stop 2024-03-31T15:15:04.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:04.433+0000 7f28f6c00640 1 -- 172.21.15.16:0/3389944980 shutdown_connections 2024-03-31T15:15:04.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:04.433+0000 7f28f6c00640 1 --2- 172.21.15.16:0/3389944980 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f28cc0683b0 0x7f28cc06a870 unknown :-1 s=CLOSED pgs=777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:04.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:04.433+0000 7f28f6c00640 1 --2- 172.21.15.16:0/3389944980 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f28f0144550 0x7f28f006dd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:04.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:04.433+0000 7f28f6c00640 1 --2- 172.21.15.16:0/3389944980 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f28f006d570 0x7f28f00a94c0 unknown :-1 s=CLOSED pgs=1849 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:04.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:04.433+0000 7f28f6c00640 1 --2- 172.21.15.16:0/3389944980 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f28d4004d50 0x7f28f00aad00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:04.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:04.433+0000 7f28f6c00640 1 -- 172.21.15.16:0/3389944980 >> 172.21.15.16:0/3389944980 conn(0x7f28f0067330 msgr2=0x7f28f00b78e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:15:04.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:04.433+0000 7f28f6c00640 1 -- 172.21.15.16:0/3389944980 shutdown_connections 2024-03-31T15:15:04.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:04.433+0000 7f28f6c00640 1 -- 172.21.15.16:0/3389944980 wait complete. 2024-03-31T15:15:04.443 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi016.front.sepia.ceph.com... 2024-03-31T15:15:04.443 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:15:04.443 DEBUG:teuthology.orchestra.run.smithi016:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-03-31T15:15:04.461 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi016.stderr:ceph-fuse[53165]: fuse finished with error 0 and tester_r 0 2024-03-31T15:15:04.465 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi016.front.sepia.ceph.com 2024-03-31T15:15:04.465 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:15:04.465 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-31T15:15:04.518 INFO:tasks.cephfs.mount:Terminating background process 2024-03-31T15:15:04.518 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-03-31T15:15:04.518 DEBUG:teuthology.orchestra.run.smithi089:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-31T15:15:04.529 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16354452k,nr_inodes=4088613,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275344k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=901 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275340k,nr_inodes=818835,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275344k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-31T15:15:04.529 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi089.front.sepia.ceph.com 2024-03-31T15:15:04.530 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:15:04.530 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-03-31T15:15:04.580 INFO:teuthology.orchestra.run.smithi089.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.1': No such file or directory 2024-03-31T15:15:04.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-31T15:15:04.581 INFO:tasks.cephfs.mount:Terminating background process 2024-03-31T15:15:04.581 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-03-31T15:15:04.581 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-31T15:15:04.590 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16357708k,nr_inodes=4089427,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275996k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=6539 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275992k,nr_inodes=818998,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275996k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-31T15:15:04.590 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi016.front.sepia.ceph.com 2024-03-31T15:15:04.591 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:15:04.591 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-03-31T15:15:04.639 INFO:teuthology.orchestra.run.smithi016.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2024-03-31T15:15:04.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-31T15:15:04.640 INFO:tasks.cephfs.mount:Terminating background process 2024-03-31T15:15:04.641 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-03-31T15:15:04.641 DEBUG:teuthology.orchestra.run.smithi089:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-31T15:15:04.650 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16354452k,nr_inodes=4088613,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275344k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=901 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275340k,nr_inodes=818835,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275344k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-31T15:15:04.650 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi089.front.sepia.ceph.com 2024-03-31T15:15:04.650 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:15:04.650 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-03-31T15:15:04.699 INFO:teuthology.orchestra.run.smithi089.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2024-03-31T15:15:04.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-31T15:15:04.701 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-31T15:15:04.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:04.849+0000 7fd57aa00640 1 Processor -- start 2024-03-31T15:15:04.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:04.853+0000 7fd57aa00640 1 -- start start 2024-03-31T15:15:04.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:04.853+0000 7fd57aa00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fd57409b170 0x7fd5740ccb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:04.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:04.853+0000 7fd57aa00640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd5740d0800 con 0x7fd574070d50 2024-03-31T15:15:04.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:04.853+0000 7fd57aa00640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd5740d0ae0 con 0x7fd57415e3b0 2024-03-31T15:15:04.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:04.853+0000 7fd57aa00640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7fd57415e790 con 0x7fd57409b170 2024-03-31T15:15:04.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:04.853+0000 7fd57a000640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fd574070d50 0x7fd5740cd0d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:36724/0 (socket says 172.21.15.16:36724) 2024-03-31T15:15:04.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:04.853+0000 7fd57a000640 1 -- 172.21.15.16:0/3316621229 learned_addr learned my addr 172.21.15.16:0/3316621229 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:15:04.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:04.853+0000 7fd578c00640 1 --2- 172.21.15.16:0/3316621229 >> v2:172.21.15.89:3300/0 conn(0x7fd57409b170 0x7fd5740ccb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:04.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:04.853+0000 7fd573e00640 1 -- 172.21.15.16:0/3316621229 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 380345527 0 0) 0x7fd5740d0800 con 0x7fd574070d50 2024-03-31T15:15:04.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:04.853+0000 7fd573e00640 1 -- 172.21.15.16:0/3316621229 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd55c003530 con 0x7fd574070d50 2024-03-31T15:15:04.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:04.853+0000 7fd578c00640 1 -- 172.21.15.16:0/3316621229 >> v1:172.21.15.89:6789/0 conn(0x7fd57415e3b0 legacy=0x7fd57409aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:04.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:04.853+0000 7fd578c00640 1 -- 172.21.15.16:0/3316621229 >> v1:172.21.15.16:6789/0 conn(0x7fd574070d50 legacy=0x7fd5740cd0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:04.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:04.853+0000 7fd578c00640 1 -- 172.21.15.16:0/3316621229 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5740d0dc0 con 0x7fd57409b170 2024-03-31T15:15:04.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:04.857+0000 7fd578c00640 1 --2- 172.21.15.16:0/3316621229 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd57409b170 0x7fd5740ccb90 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7fd5600097c0 tx=0x7fd56001c4a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9ebad4f6fcfe62c2 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:15:04.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:04.857+0000 7fd573e00640 1 -- 172.21.15.16:0/3316621229 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5600b1070 con 0x7fd57409b170 2024-03-31T15:15:04.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:04.857+0000 7fd573e00640 1 -- 172.21.15.16:0/3316621229 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd5600ac410 con 0x7fd57409b170 2024-03-31T15:15:04.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:04.857+0000 7fd573e00640 1 -- 172.21.15.16:0/3316621229 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5600ac7c0 con 0x7fd57409b170 2024-03-31T15:15:04.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:04.857+0000 7fd57aa00640 1 -- 172.21.15.16:0/3316621229 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd57409b170 msgr2=0x7fd5740ccb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:04.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:04.857+0000 7fd57aa00640 1 --2- 172.21.15.16:0/3316621229 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd57409b170 0x7fd5740ccb90 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7fd5600097c0 tx=0x7fd56001c4a0 comp rx=0 tx=0).stop 2024-03-31T15:15:04.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:04.857+0000 7fd57aa00640 1 -- 172.21.15.16:0/3316621229 shutdown_connections 2024-03-31T15:15:04.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:04.857+0000 7fd57aa00640 1 --2- 172.21.15.16:0/3316621229 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd57409b170 0x7fd5740ccb90 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:04.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:04.857+0000 7fd57aa00640 1 -- 172.21.15.16:0/3316621229 >> 172.21.15.16:0/3316621229 conn(0x7fd5740b39c0 msgr2=0x7fd5740b3dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:15:04.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:04.857+0000 7fd57aa00640 1 -- 172.21.15.16:0/3316621229 shutdown_connections 2024-03-31T15:15:04.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:04.857+0000 7fd57aa00640 1 -- 172.21.15.16:0/3316621229 wait complete. 2024-03-31T15:15:04.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:04.857+0000 7fd57aa00640 1 Processor -- start 2024-03-31T15:15:04.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:04.857+0000 7fd57aa00640 1 -- start start 2024-03-31T15:15:04.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:04.861+0000 7fd57aa00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fd574070d50 0x7fd5740b7400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:04.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:04.861+0000 7fd57aa00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd57409b170 0x7fd5740b5640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:04.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:04.861+0000 7fd57aa00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd57415e3b0 0x7fd574153140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:04.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:04.861+0000 7fd57aa00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fd57416c230 con 0x7fd57409b170 2024-03-31T15:15:04.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:04.861+0000 7fd57aa00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fd57416c0b0 con 0x7fd57415e3b0 2024-03-31T15:15:04.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:04.861+0000 7fd57aa00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fd57416c3b0 con 0x7fd574070d50 2024-03-31T15:15:04.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:04.861+0000 7fd578c00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd57409b170 0x7fd5740b5640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:04.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:04.861+0000 7fd578c00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd57409b170 0x7fd5740b5640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50266/0 (socket says 172.21.15.16:50266) 2024-03-31T15:15:04.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:04.861+0000 7fd578c00640 1 -- 172.21.15.16:0/1774297028 learned_addr learned my addr 172.21.15.16:0/1774297028 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:15:04.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:04.861+0000 7fd57a000640 1 --2- 172.21.15.16:0/1774297028 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd57415e3b0 0x7fd574153140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:04.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:04.861+0000 7fd579600640 1 --2- 172.21.15.16:0/1774297028 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fd574070d50 0x7fd5740b7400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:04.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:04.861+0000 7fd578c00640 1 -- 172.21.15.16:0/1774297028 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fd574070d50 msgr2=0x7fd5740b7400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:04.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:04.861+0000 7fd578c00640 1 --2- 172.21.15.16:0/1774297028 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fd574070d50 0x7fd5740b7400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:04.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:04.861+0000 7fd578c00640 1 -- 172.21.15.16:0/1774297028 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd57415e3b0 msgr2=0x7fd574153140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:04.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:04.861+0000 7fd578c00640 1 --2- 172.21.15.16:0/1774297028 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd57415e3b0 0x7fd574153140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:04.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:04.861+0000 7fd578c00640 1 -- 172.21.15.16:0/1774297028 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd574158110 con 0x7fd57409b170 2024-03-31T15:15:04.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:04.861+0000 7fd578c00640 1 --2- 172.21.15.16:0/1774297028 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd57409b170 0x7fd5740b5640 secure :-1 s=READY pgs=1851 cs=0 l=1 rev1=1 crypto rx=0x7fd5600097c0 tx=0x7fd560019240 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:15:04.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:04.861+0000 7fd571600640 1 -- 172.21.15.16:0/1774297028 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5600bc070 con 0x7fd57409b170 2024-03-31T15:15:04.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:04.861+0000 7fd571600640 1 -- 172.21.15.16:0/1774297028 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd5600acd70 con 0x7fd57409b170 2024-03-31T15:15:04.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:04.861+0000 7fd57aa00640 1 -- 172.21.15.16:0/1774297028 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd5741583b0 con 0x7fd57409b170 2024-03-31T15:15:04.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:04.861+0000 7fd571600640 1 -- 172.21.15.16:0/1774297028 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5600b1040 con 0x7fd57409b170 2024-03-31T15:15:04.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:04.861+0000 7fd57aa00640 1 -- 172.21.15.16:0/1774297028 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd574158970 con 0x7fd57409b170 2024-03-31T15:15:04.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:04.861+0000 7fd57aa00640 1 -- 172.21.15.16:0/1774297028 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd57404f8c0 con 0x7fd57409b170 2024-03-31T15:15:04.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:04.865+0000 7fd571600640 1 -- 172.21.15.16:0/1774297028 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fd5600bf070 con 0x7fd57409b170 2024-03-31T15:15:04.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:04.865+0000 7fd571600640 1 --2- 172.21.15.16:0/1774297028 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fd554068380 0x7fd55406a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:04.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:04.865+0000 7fd571600640 1 -- 172.21.15.16:0/1774297028 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 5939+0+0 (secure 0 0 0) 0x7fd56011fc50 con 0x7fd57409b170 2024-03-31T15:15:04.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:04.865+0000 7fd579600640 1 --2- 172.21.15.16:0/1774297028 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fd554068380 0x7fd55406a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:04.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:04.865+0000 7fd579600640 1 --2- 172.21.15.16:0/1774297028 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fd554068380 0x7fd55406a840 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7fd5680015b0 tx=0x7fd568009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:15:04.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:04.869+0000 7fd571600640 1 -- 172.21.15.16:0/1774297028 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fd560016090 con 0x7fd57409b170 2024-03-31T15:15:05.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:05.097+0000 7fd57aa00640 1 -- 172.21.15.16:0/1774297028 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd574068ae0 con 0x7fd57409b170 2024-03-31T15:15:05.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:05.097+0000 7fd571600640 1 -- 172.21.15.16:0/1774297028 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 112 v112) ==== 95+0+5185 (secure 0 0 0) 0x7fd5600f0080 con 0x7fd57409b170 2024-03-31T15:15:05.086 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:15:05.086 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":112,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7631,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6836/1628897112","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":1628897112},{"type":"v1","addr":"172.21.15.89:6836","nonce":1628897112}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":94},{"gid":8746,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/1036724837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6835","nonce":1036724837},{"type":"v1","addr":"172.21.15.89:6837","nonce":1036724837}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":94},{"gid":9192,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6834/472139147","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":472139147},{"type":"v1","addr":"172.21.15.16:6834","nonce":472139147}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":94}],"filesystems":[{"mdsmap":{"epoch":112,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-31T15:14:08.652180+0000","modified":"2024-03-31T15:14:58.141614+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":113,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":9762},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9762":{"gid":9762,"name":"c","rank":0,"incarnation":109,"state":"up:active","state_seq":4,"addr":"172.21.15.16:6835/2152452999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6833","nonce":2152452999},{"type":"v1","addr":"172.21.15.16:6835","nonce":2152452999}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9762,"qdb_cluster":[9762]},"id":8}]} 2024-03-31T15:15:05.086 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 112 2024-03-31T15:15:05.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:05.097+0000 7fd57aa00640 1 -- 172.21.15.16:0/1774297028 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fd554068380 msgr2=0x7fd55406a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:05.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:05.097+0000 7fd57aa00640 1 --2- 172.21.15.16:0/1774297028 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fd554068380 0x7fd55406a840 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7fd5680015b0 tx=0x7fd568009040 comp rx=0 tx=0).stop 2024-03-31T15:15:05.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:05.097+0000 7fd57aa00640 1 -- 172.21.15.16:0/1774297028 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd57409b170 msgr2=0x7fd5740b5640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:05.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:05.097+0000 7fd57aa00640 1 --2- 172.21.15.16:0/1774297028 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd57409b170 0x7fd5740b5640 secure :-1 s=READY pgs=1851 cs=0 l=1 rev1=1 crypto rx=0x7fd5600097c0 tx=0x7fd560019240 comp rx=0 tx=0).stop 2024-03-31T15:15:05.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:05.097+0000 7fd57aa00640 1 -- 172.21.15.16:0/1774297028 shutdown_connections 2024-03-31T15:15:05.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:05.097+0000 7fd57aa00640 1 --2- 172.21.15.16:0/1774297028 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fd554068380 0x7fd55406a840 unknown :-1 s=CLOSED pgs=778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:05.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:05.097+0000 7fd57aa00640 1 --2- 172.21.15.16:0/1774297028 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd57415e3b0 0x7fd574153140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:05.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:05.097+0000 7fd57aa00640 1 --2- 172.21.15.16:0/1774297028 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd57409b170 0x7fd5740b5640 unknown :-1 s=CLOSED pgs=1851 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:05.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:05.097+0000 7fd57aa00640 1 --2- 172.21.15.16:0/1774297028 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fd574070d50 0x7fd5740b7400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:05.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:05.097+0000 7fd57aa00640 1 -- 172.21.15.16:0/1774297028 >> 172.21.15.16:0/1774297028 conn(0x7fd5740b39c0 msgr2=0x7fd57406cf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:15:05.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:05.097+0000 7fd57aa00640 1 -- 172.21.15.16:0/1774297028 shutdown_connections 2024-03-31T15:15:05.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:05.097+0000 7fd57aa00640 1 -- 172.21.15.16:0/1774297028 wait complete. 2024-03-31T15:15:05.108 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-31T15:15:05.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:05.261+0000 7fc1ed800640 1 Processor -- start 2024-03-31T15:15:05.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:05.261+0000 7fc1ed800640 1 -- start start 2024-03-31T15:15:05.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:05.261+0000 7fc1ed800640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fc1e8070030 0x7fc1e806c730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:05.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:05.261+0000 7fc1ed800640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fc1e8147dd0 0x7fc1e80cc8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:05.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:05.261+0000 7fc1ed800640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc1e8147660 con 0x7fc1e8147280 2024-03-31T15:15:05.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:05.261+0000 7fc1ed800640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7fc1e81620f0 con 0x7fc1e8070030 2024-03-31T15:15:05.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:05.261+0000 7fc1ed800640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7fc1e8162340 con 0x7fc1e8147dd0 2024-03-31T15:15:05.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:05.261+0000 7fc1de000640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fc1e8070030 0x7fc1e806c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:05.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:05.261+0000 7fc1e6a00640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fc1e8147280 0x7fc1e806f920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:36740/0 (socket says 172.21.15.16:36740) 2024-03-31T15:15:05.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:05.261+0000 7fc1e6a00640 1 -- 172.21.15.16:0/4253567449 learned_addr learned my addr 172.21.15.16:0/4253567449 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:15:05.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:05.261+0000 7fc1e7400640 1 --2- 172.21.15.16:0/4253567449 >> v2:172.21.15.89:3300/0 conn(0x7fc1e8147dd0 0x7fc1e80cc8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:05.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:05.265+0000 7fc1de000640 1 -- 172.21.15.16:0/4253567449 >> v2:172.21.15.89:3300/0 conn(0x7fc1e8147dd0 msgr2=0x7fc1e80cc8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:15:05.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:05.265+0000 7fc1de000640 1 --2- 172.21.15.16:0/4253567449 >> v2:172.21.15.89:3300/0 conn(0x7fc1e8147dd0 0x7fc1e80cc8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:05.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:05.265+0000 7fc1de000640 1 -- 172.21.15.16:0/4253567449 >> v1:172.21.15.16:6789/0 conn(0x7fc1e8147280 legacy=0x7fc1e806f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:05.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:05.265+0000 7fc1de000640 1 -- 172.21.15.16:0/4253567449 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc1e8162620 con 0x7fc1e8070030 2024-03-31T15:15:05.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:05.265+0000 7fc1e7400640 1 --2- 172.21.15.16:0/4253567449 >> v2:172.21.15.89:3300/0 conn(0x7fc1e8147dd0 0x7fc1e80cc8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:15:05.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:05.265+0000 7fc1de000640 1 --2- 172.21.15.16:0/4253567449 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc1e8070030 0x7fc1e806c730 secure :-1 s=READY pgs=1853 cs=0 l=1 rev1=1 crypto rx=0x7fc1e8073be0 tx=0x7fc1d001c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54a6de015bd84e5d server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:15:05.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:05.265+0000 7fc1e6000640 1 -- 172.21.15.16:0/4253567449 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1d00b1070 con 0x7fc1e8070030 2024-03-31T15:15:05.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:05.265+0000 7fc1e6000640 1 -- 172.21.15.16:0/4253567449 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc1d00ac5f0 con 0x7fc1e8070030 2024-03-31T15:15:05.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:05.265+0000 7fc1e6000640 1 -- 172.21.15.16:0/4253567449 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1d00ac9c0 con 0x7fc1e8070030 2024-03-31T15:15:05.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:05.265+0000 7fc1ed800640 1 -- 172.21.15.16:0/4253567449 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc1e8070030 msgr2=0x7fc1e806c730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:05.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:05.265+0000 7fc1ed800640 1 --2- 172.21.15.16:0/4253567449 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc1e8070030 0x7fc1e806c730 secure :-1 s=READY pgs=1853 cs=0 l=1 rev1=1 crypto rx=0x7fc1e8073be0 tx=0x7fc1d001c480 comp rx=0 tx=0).stop 2024-03-31T15:15:05.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:05.265+0000 7fc1ed800640 1 -- 172.21.15.16:0/4253567449 shutdown_connections 2024-03-31T15:15:05.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:05.265+0000 7fc1ed800640 1 --2- 172.21.15.16:0/4253567449 >> v2:172.21.15.89:3300/0 conn(0x7fc1e8147dd0 0x7fc1e80cc8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:05.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:05.265+0000 7fc1ed800640 1 --2- 172.21.15.16:0/4253567449 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc1e8070030 0x7fc1e806c730 unknown :-1 s=CLOSED pgs=1853 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:05.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:05.265+0000 7fc1ed800640 1 -- 172.21.15.16:0/4253567449 >> 172.21.15.16:0/4253567449 conn(0x7fc1e8067330 msgr2=0x7fc1e8067740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:15:05.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:05.265+0000 7fc1ed800640 1 -- 172.21.15.16:0/4253567449 shutdown_connections 2024-03-31T15:15:05.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:05.269+0000 7fc1ed800640 1 -- 172.21.15.16:0/4253567449 wait complete. 2024-03-31T15:15:05.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:05.269+0000 7fc1ed800640 1 Processor -- start 2024-03-31T15:15:05.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:05.269+0000 7fc1ed800640 1 -- start start 2024-03-31T15:15:05.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:05.269+0000 7fc1ed800640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc1e8070030 0x7fc1e8140ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:05.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:05.269+0000 7fc1ed800640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc1e8147280 0x7fc1e815db20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:05.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:05.269+0000 7fc1ed800640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fc1e8147dd0 0x7fc1e8160b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:05.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:05.269+0000 7fc1ed800640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fc1e80cda10 con 0x7fc1e8147280 2024-03-31T15:15:05.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:05.269+0000 7fc1ed800640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fc1e80cd890 con 0x7fc1e8070030 2024-03-31T15:15:05.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:05.269+0000 7fc1ed800640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fc1e80cdb90 con 0x7fc1e8147dd0 2024-03-31T15:15:05.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:05.269+0000 7fc1de000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc1e8147280 0x7fc1e815db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:05.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:05.269+0000 7fc1de000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc1e8147280 0x7fc1e815db20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50276/0 (socket says 172.21.15.16:50276) 2024-03-31T15:15:05.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:05.269+0000 7fc1de000640 1 -- 172.21.15.16:0/3571576713 learned_addr learned my addr 172.21.15.16:0/3571576713 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:15:05.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:05.269+0000 7fc1e6a00640 1 --2- 172.21.15.16:0/3571576713 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc1e8070030 0x7fc1e8140ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:05.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:05.269+0000 7fc1e7400640 1 --2- 172.21.15.16:0/3571576713 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fc1e8147dd0 0x7fc1e8160b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:05.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:05.269+0000 7fc1de000640 1 -- 172.21.15.16:0/3571576713 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fc1e8147dd0 msgr2=0x7fc1e8160b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:05.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:05.269+0000 7fc1de000640 1 --2- 172.21.15.16:0/3571576713 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fc1e8147dd0 0x7fc1e8160b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:05.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:05.269+0000 7fc1de000640 1 -- 172.21.15.16:0/3571576713 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc1e8070030 msgr2=0x7fc1e8140ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:05.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:05.269+0000 7fc1de000640 1 --2- 172.21.15.16:0/3571576713 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc1e8070030 0x7fc1e8140ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:05.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:05.269+0000 7fc1de000640 1 -- 172.21.15.16:0/3571576713 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc1e80b4230 con 0x7fc1e8147280 2024-03-31T15:15:05.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:05.269+0000 7fc1de000640 1 --2- 172.21.15.16:0/3571576713 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc1e8147280 0x7fc1e815db20 secure :-1 s=READY pgs=1854 cs=0 l=1 rev1=1 crypto rx=0x7fc1d001ab30 tx=0x7fc1d0019220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:15:05.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:05.269+0000 7fc1df400640 1 -- 172.21.15.16:0/3571576713 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1d00b9020 con 0x7fc1e8147280 2024-03-31T15:15:05.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:05.269+0000 7fc1df400640 1 -- 172.21.15.16:0/3571576713 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc1d00b5410 con 0x7fc1e8147280 2024-03-31T15:15:05.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:05.269+0000 7fc1ed800640 1 -- 172.21.15.16:0/3571576713 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc1e80b4510 con 0x7fc1e8147280 2024-03-31T15:15:05.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:05.273+0000 7fc1df400640 1 -- 172.21.15.16:0/3571576713 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1d00b1040 con 0x7fc1e8147280 2024-03-31T15:15:05.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:05.273+0000 7fc1ed800640 1 -- 172.21.15.16:0/3571576713 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc1e80b4ad0 con 0x7fc1e8147280 2024-03-31T15:15:05.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:05.273+0000 7fc1ed800640 1 -- 172.21.15.16:0/3571576713 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc1e8069950 con 0x7fc1e8147280 2024-03-31T15:15:05.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:05.273+0000 7fc1df400640 1 -- 172.21.15.16:0/3571576713 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fc1d00bc020 con 0x7fc1e8147280 2024-03-31T15:15:05.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:05.273+0000 7fc1df400640 1 --2- 172.21.15.16:0/3571576713 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fc1c40683d0 0x7fc1c406a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:05.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:05.273+0000 7fc1df400640 1 -- 172.21.15.16:0/3571576713 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 5939+0+0 (secure 0 0 0) 0x7fc1d0120370 con 0x7fc1e8147280 2024-03-31T15:15:05.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:05.273+0000 7fc1e6a00640 1 --2- 172.21.15.16:0/3571576713 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fc1c40683d0 0x7fc1c406a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:05.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:05.277+0000 7fc1e6a00640 1 --2- 172.21.15.16:0/3571576713 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fc1c40683d0 0x7fc1c406a890 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7fc1e815e800 tx=0x7fc1d8007e90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:15:05.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:05.281+0000 7fc1df400640 1 -- 172.21.15.16:0/3571576713 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fc1d00c3030 con 0x7fc1e8147280 2024-03-31T15:15:05.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:05.449+0000 7fc1ed800640 1 -- 172.21.15.16:0/3571576713 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc1e806e260 con 0x7fc1e8147280 2024-03-31T15:15:05.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:05.449+0000 7fc1df400640 1 -- 172.21.15.16:0/3571576713 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v113) ==== 74+0+16907 (secure 0 0 0) 0x7fc1d00a9070 con 0x7fc1e8147280 2024-03-31T15:15:05.439 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:15:05.439 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":113,"fsid":"286eb5da-fb47-4550-b290-d9519c13b9a5","created":"2024-03-31T15:03:11.438952+0000","modified":"2024-03-31T15:14:53.117209+0000","last_up_change":"2024-03-31T15:03:22.885535+0000","last_in_change":"2024-03-31T15:03:13.941667+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-31T15:03:22.550787+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-03-31T15:14:06.417304+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"108","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-03-31T15:14:07.622562+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"109","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"92a66450-ef19-40d8-ac3d-d760b23b0bda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6801","nonce":932712244}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6803","nonce":932712244}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6807","nonce":932712244}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6805","nonce":932712244}]},"public_addr":"172.21.15.16:6801/932712244","cluster_addr":"172.21.15.16:6803/932712244","heartbeat_back_addr":"172.21.15.16:6807/932712244","heartbeat_front_addr":"172.21.15.16:6805/932712244","state":["exists","up"]},{"osd":1,"uuid":"da9d1f2b-1968-4959-b843-faa231988e92","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6810","nonce":1398737533}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6811","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6813","nonce":1398737533}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6821","nonce":1398737533}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6815","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6816","nonce":1398737533}]},"public_addr":"172.21.15.16:6810/1398737533","cluster_addr":"172.21.15.16:6813/1398737533","heartbeat_back_addr":"172.21.15.16:6821/1398737533","heartbeat_front_addr":"172.21.15.16:6816/1398737533","state":["exists","up"]},{"osd":2,"uuid":"01a73662-9a21-4b1f-bcbb-42b9636a4638","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6819","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6822","nonce":3018526646}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6825","nonce":3018526646}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6831","nonce":3018526646}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6827","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6829","nonce":3018526646}]},"public_addr":"172.21.15.16:6822/3018526646","cluster_addr":"172.21.15.16:6825/3018526646","heartbeat_back_addr":"172.21.15.16:6831/3018526646","heartbeat_front_addr":"172.21.15.16:6829/3018526646","state":["exists","up"]},{"osd":3,"uuid":"c7771fd3-e18e-4500-ac38-23cf10ebcd75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6809","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6812","nonce":3360352974}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6817","nonce":3360352974}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6828","nonce":3360352974}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6823","nonce":3360352974}]},"public_addr":"172.21.15.16:6812/3360352974","cluster_addr":"172.21.15.16:6817/3360352974","heartbeat_back_addr":"172.21.15.16:6828/3360352974","heartbeat_front_addr":"172.21.15.16:6823/3360352974","state":["exists","up"]},{"osd":4,"uuid":"88260730-1482-4fbd-a40f-cb1124e9df4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6801","nonce":1856844856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6803","nonce":1856844856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6807","nonce":1856844856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6805","nonce":1856844856}]},"public_addr":"172.21.15.89:6801/1856844856","cluster_addr":"172.21.15.89:6803/1856844856","heartbeat_back_addr":"172.21.15.89:6807/1856844856","heartbeat_front_addr":"172.21.15.89:6805/1856844856","state":["exists","up"]},{"osd":5,"uuid":"4cb2c00c-fae0-4dd8-9d27-ec4f0093ccf7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6814","nonce":50395527}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6817","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6820","nonce":50395527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6831","nonce":50395527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6823","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6826","nonce":50395527}]},"public_addr":"172.21.15.89:6814/50395527","cluster_addr":"172.21.15.89:6820/50395527","heartbeat_back_addr":"172.21.15.89:6831/50395527","heartbeat_front_addr":"172.21.15.89:6826/50395527","state":["exists","up"]},{"osd":6,"uuid":"81ce0c19-2554-4cae-acbe-66a05e9e9b9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6809","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6812","nonce":3470471752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6815","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6818","nonce":3470471752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6827","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6829","nonce":3470471752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6821","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6824","nonce":3470471752}]},"public_addr":"172.21.15.89:6812/3470471752","cluster_addr":"172.21.15.89:6818/3470471752","heartbeat_back_addr":"172.21.15.89:6829/3470471752","heartbeat_front_addr":"172.21.15.89:6824/3470471752","state":["exists","up"]},{"osd":7,"uuid":"85e4313b-d37d-442e-8042-153bf20a847d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6811","nonce":1796778630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6813","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6816","nonce":1796778630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6825","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6828","nonce":1796778630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6819","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6822","nonce":1796778630}]},"public_addr":"172.21.15.89:6811/1796778630","cluster_addr":"172.21.15.89:6816/1796778630","heartbeat_back_addr":"172.21.15.89:6828/1796778630","heartbeat_front_addr":"172.21.15.89:6822/1796778630","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.439088+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.529970+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.680512+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:21.044161+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.582111+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.536558+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.409522+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.590229+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"17.1","mappings":[{"from":0,"to":5}]},{"pgid":"17.21","mappings":[{"from":0,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:6833/4142860904":"2024-04-01T15:14:53.117187+0000","172.21.15.16:6835/2564952016":"2024-04-01T15:14:39.715615+0000","172.21.15.16:6835/4142860904":"2024-04-01T15:14:53.117187+0000","172.21.15.16:6833/2564952016":"2024-04-01T15:14:39.715615+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-31T15:15:05.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:05.449+0000 7fc1ed800640 1 -- 172.21.15.16:0/3571576713 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fc1c40683d0 msgr2=0x7fc1c406a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:05.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:05.449+0000 7fc1ed800640 1 --2- 172.21.15.16:0/3571576713 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fc1c40683d0 0x7fc1c406a890 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7fc1e815e800 tx=0x7fc1d8007e90 comp rx=0 tx=0).stop 2024-03-31T15:15:05.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:05.449+0000 7fc1ed800640 1 -- 172.21.15.16:0/3571576713 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc1e8147280 msgr2=0x7fc1e815db20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:05.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:05.449+0000 7fc1ed800640 1 --2- 172.21.15.16:0/3571576713 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc1e8147280 0x7fc1e815db20 secure :-1 s=READY pgs=1854 cs=0 l=1 rev1=1 crypto rx=0x7fc1d001ab30 tx=0x7fc1d0019220 comp rx=0 tx=0).stop 2024-03-31T15:15:05.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:05.453+0000 7fc1ed800640 1 -- 172.21.15.16:0/3571576713 shutdown_connections 2024-03-31T15:15:05.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:05.453+0000 7fc1ed800640 1 --2- 172.21.15.16:0/3571576713 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fc1c40683d0 0x7fc1c406a890 unknown :-1 s=CLOSED pgs=779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:05.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:05.453+0000 7fc1ed800640 1 --2- 172.21.15.16:0/3571576713 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fc1e8147dd0 0x7fc1e8160b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:05.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:05.453+0000 7fc1ed800640 1 --2- 172.21.15.16:0/3571576713 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc1e8147280 0x7fc1e815db20 unknown :-1 s=CLOSED pgs=1854 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:05.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:05.453+0000 7fc1ed800640 1 --2- 172.21.15.16:0/3571576713 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc1e8070030 0x7fc1e8140ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:05.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:05.453+0000 7fc1ed800640 1 -- 172.21.15.16:0/3571576713 >> 172.21.15.16:0/3571576713 conn(0x7fc1e8067330 msgr2=0x7fc1e8067710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:15:05.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:05.453+0000 7fc1ed800640 1 -- 172.21.15.16:0/3571576713 shutdown_connections 2024-03-31T15:15:05.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:05.453+0000 7fc1ed800640 1 -- 172.21.15.16:0/3571576713 wait complete. 2024-03-31T15:15:05.461 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-31T15:03:22.550787+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 16, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-31T15:14:06.417304+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '108', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 17, 'pool_name': 'cephfs_data', 'create_time': '2024-03-31T15:14:07.622562+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '109', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-31T15:15:05.462 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-31T15:15:05.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:05.617+0000 7efd85a00640 1 Processor -- start 2024-03-31T15:15:05.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:05.617+0000 7efd85a00640 1 -- start start 2024-03-31T15:15:05.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:05.617+0000 7efd85a00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7efd8006f920 0x7efd8006fd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:05.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:05.617+0000 7efd85a00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7efd800cf150 0x7efd801419c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:05.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:05.617+0000 7efd85a00640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efd80070240 con 0x7efd800d0a60 2024-03-31T15:15:05.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:05.617+0000 7efd85a00640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7efd8013f670 con 0x7efd800cf150 2024-03-31T15:15:05.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:05.617+0000 7efd85a00640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7efd8015ca70 con 0x7efd8006f920 2024-03-31T15:15:05.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:05.617+0000 7efd7f400640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7efd800cf150 0x7efd801419c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:05.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:05.617+0000 7efd77e00640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7efd800d0a60 0x7efd800cea40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:36744/0 (socket says 172.21.15.16:36744) 2024-03-31T15:15:05.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:05.617+0000 7efd7f400640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7efd800cf150 0x7efd801419c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50290/0 (socket says 172.21.15.16:50290) 2024-03-31T15:15:05.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:05.617+0000 7efd77e00640 1 -- 172.21.15.16:0/1575658733 learned_addr learned my addr 172.21.15.16:0/1575658733 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:15:05.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:05.617+0000 7efd7ea00640 1 --2- 172.21.15.16:0/1575658733 >> v2:172.21.15.89:3300/0 conn(0x7efd8006f920 0x7efd8006fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:05.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:05.617+0000 7efd7f400640 1 -- 172.21.15.16:0/1575658733 >> v2:172.21.15.89:3300/0 conn(0x7efd8006f920 msgr2=0x7efd8006fd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:15:05.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:05.617+0000 7efd7f400640 1 --2- 172.21.15.16:0/1575658733 >> v2:172.21.15.89:3300/0 conn(0x7efd8006f920 0x7efd8006fd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:05.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:05.617+0000 7efd7f400640 1 -- 172.21.15.16:0/1575658733 >> v1:172.21.15.16:6789/0 conn(0x7efd800d0a60 legacy=0x7efd800cea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:05.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:05.617+0000 7efd7f400640 1 -- 172.21.15.16:0/1575658733 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd8015cd50 con 0x7efd800cf150 2024-03-31T15:15:05.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:05.617+0000 7efd7ea00640 1 --2- 172.21.15.16:0/1575658733 >> v2:172.21.15.89:3300/0 conn(0x7efd8006f920 0x7efd8006fd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:15:05.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:05.621+0000 7efd7f400640 1 --2- 172.21.15.16:0/1575658733 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efd800cf150 0x7efd801419c0 secure :-1 s=READY pgs=1856 cs=0 l=1 rev1=1 crypto rx=0x7efd800708e0 tx=0x7efd7801c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=879cd9e076d7330b server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:15:05.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:05.621+0000 7efd7e000640 1 -- 172.21.15.16:0/1575658733 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd780b2070 con 0x7efd800cf150 2024-03-31T15:15:05.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:05.621+0000 7efd7e000640 1 -- 172.21.15.16:0/1575658733 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efd780a5760 con 0x7efd800cf150 2024-03-31T15:15:05.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:05.621+0000 7efd7e000640 1 -- 172.21.15.16:0/1575658733 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd780a5b30 con 0x7efd800cf150 2024-03-31T15:15:05.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:05.621+0000 7efd85a00640 1 -- 172.21.15.16:0/1575658733 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efd800cf150 msgr2=0x7efd801419c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:05.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:05.621+0000 7efd85a00640 1 --2- 172.21.15.16:0/1575658733 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efd800cf150 0x7efd801419c0 secure :-1 s=READY pgs=1856 cs=0 l=1 rev1=1 crypto rx=0x7efd800708e0 tx=0x7efd7801c8a0 comp rx=0 tx=0).stop 2024-03-31T15:15:05.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:05.621+0000 7efd85a00640 1 -- 172.21.15.16:0/1575658733 shutdown_connections 2024-03-31T15:15:05.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:05.621+0000 7efd85a00640 1 --2- 172.21.15.16:0/1575658733 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efd800cf150 0x7efd801419c0 unknown :-1 s=CLOSED pgs=1856 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:05.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:05.621+0000 7efd85a00640 1 --2- 172.21.15.16:0/1575658733 >> v2:172.21.15.89:3300/0 conn(0x7efd8006f920 0x7efd8006fd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:05.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:05.621+0000 7efd85a00640 1 -- 172.21.15.16:0/1575658733 >> 172.21.15.16:0/1575658733 conn(0x7efd80067930 msgr2=0x7efd800b3ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:15:05.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:05.621+0000 7efd85a00640 1 -- 172.21.15.16:0/1575658733 shutdown_connections 2024-03-31T15:15:05.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:05.621+0000 7efd85a00640 1 -- 172.21.15.16:0/1575658733 wait complete. 2024-03-31T15:15:05.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:05.625+0000 7efd85a00640 1 Processor -- start 2024-03-31T15:15:05.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:05.625+0000 7efd85a00640 1 -- start start 2024-03-31T15:15:05.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:05.625+0000 7efd85a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efd8006f920 0x7efd80160db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:05.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:05.625+0000 7efd85a00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7efd800cf150 0x7efd800b61c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:05.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:05.625+0000 7efd85a00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efd800d0a60 0x7efd800b6700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:05.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:05.625+0000 7efd85a00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7efd80169230 con 0x7efd8006f920 2024-03-31T15:15:05.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:05.625+0000 7efd85a00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7efd801690b0 con 0x7efd800d0a60 2024-03-31T15:15:05.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:05.625+0000 7efd85a00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7efd801693b0 con 0x7efd800cf150 2024-03-31T15:15:05.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:05.625+0000 7efd7ea00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efd8006f920 0x7efd80160db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:05.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:05.625+0000 7efd7ea00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efd8006f920 0x7efd80160db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50298/0 (socket says 172.21.15.16:50298) 2024-03-31T15:15:05.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:05.625+0000 7efd7ea00640 1 -- 172.21.15.16:0/180927432 learned_addr learned my addr 172.21.15.16:0/180927432 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:15:05.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:05.625+0000 7efd77e00640 1 --2- 172.21.15.16:0/180927432 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7efd800cf150 0x7efd800b61c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:05.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:05.625+0000 7efd7f400640 1 --2- 172.21.15.16:0/180927432 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efd800d0a60 0x7efd800b6700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:05.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:05.629+0000 7efd7ea00640 1 -- 172.21.15.16:0/180927432 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7efd800cf150 msgr2=0x7efd800b61c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:05.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:05.629+0000 7efd7ea00640 1 --2- 172.21.15.16:0/180927432 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7efd800cf150 0x7efd800b61c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:05.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:05.629+0000 7efd7ea00640 1 -- 172.21.15.16:0/180927432 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efd800d0a60 msgr2=0x7efd800b6700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:05.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:05.629+0000 7efd7ea00640 1 --2- 172.21.15.16:0/180927432 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efd800d0a60 0x7efd800b6700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:05.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:05.629+0000 7efd7ea00640 1 -- 172.21.15.16:0/180927432 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd80161600 con 0x7efd8006f920 2024-03-31T15:15:05.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:05.629+0000 7efd7ea00640 1 --2- 172.21.15.16:0/180927432 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efd8006f920 0x7efd80160db0 secure :-1 s=READY pgs=1857 cs=0 l=1 rev1=1 crypto rx=0x7efd7000e810 tx=0x7efd7000ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:15:05.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:05.629+0000 7efd76a00640 1 -- 172.21.15.16:0/180927432 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd70017070 con 0x7efd8006f920 2024-03-31T15:15:05.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:05.629+0000 7efd76a00640 1 -- 172.21.15.16:0/180927432 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efd7000c8d0 con 0x7efd8006f920 2024-03-31T15:15:05.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:05.629+0000 7efd76a00640 1 -- 172.21.15.16:0/180927432 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd7000f040 con 0x7efd8006f920 2024-03-31T15:15:05.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:05.629+0000 7efd85a00640 1 -- 172.21.15.16:0/180927432 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efd801618e0 con 0x7efd8006f920 2024-03-31T15:15:05.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:05.629+0000 7efd85a00640 1 -- 172.21.15.16:0/180927432 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efd80161ea0 con 0x7efd8006f920 2024-03-31T15:15:05.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:05.629+0000 7efd85a00640 1 -- 172.21.15.16:0/180927432 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efd80069950 con 0x7efd8006f920 2024-03-31T15:15:05.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:05.629+0000 7efd76a00640 1 -- 172.21.15.16:0/180927432 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7efd7001a070 con 0x7efd8006f920 2024-03-31T15:15:05.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:05.629+0000 7efd76a00640 1 --2- 172.21.15.16:0/180927432 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7efd5c0683d0 0x7efd5c06a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:05.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:05.629+0000 7efd76a00640 1 -- 172.21.15.16:0/180927432 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 5939+0+0 (secure 0 0 0) 0x7efd70086670 con 0x7efd8006f920 2024-03-31T15:15:05.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:05.629+0000 7efd77e00640 1 --2- 172.21.15.16:0/180927432 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7efd5c0683d0 0x7efd5c06a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:05.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:05.633+0000 7efd77e00640 1 --2- 172.21.15.16:0/180927432 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7efd5c0683d0 0x7efd5c06a890 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7efd800b39e0 tx=0x7efd68007e90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:15:05.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:05.633+0000 7efd76a00640 1 -- 172.21.15.16:0/180927432 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7efd700564d0 con 0x7efd8006f920 2024-03-31T15:15:05.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:05.797+0000 7efd85a00640 1 -- 172.21.15.16:0/180927432 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7efd800b4a20 con 0x7efd8006f920 2024-03-31T15:15:05.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:05.801+0000 7efd76a00640 1 -- 172.21.15.16:0/180927432 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v113) ==== 93+0+10 (secure 0 0 0) 0x7efd7005a7d0 con 0x7efd8006f920 2024-03-31T15:15:05.789 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2024-03-31T15:15:05.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:05.801+0000 7efd85a00640 1 -- 172.21.15.16:0/180927432 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7efd5c0683d0 msgr2=0x7efd5c06a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:05.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:05.801+0000 7efd85a00640 1 --2- 172.21.15.16:0/180927432 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7efd5c0683d0 0x7efd5c06a890 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7efd800b39e0 tx=0x7efd68007e90 comp rx=0 tx=0).stop 2024-03-31T15:15:05.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:05.801+0000 7efd85a00640 1 -- 172.21.15.16:0/180927432 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efd8006f920 msgr2=0x7efd80160db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:05.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:05.801+0000 7efd85a00640 1 --2- 172.21.15.16:0/180927432 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efd8006f920 0x7efd80160db0 secure :-1 s=READY pgs=1857 cs=0 l=1 rev1=1 crypto rx=0x7efd7000e810 tx=0x7efd7000ecd0 comp rx=0 tx=0).stop 2024-03-31T15:15:05.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:05.801+0000 7efd85a00640 1 -- 172.21.15.16:0/180927432 shutdown_connections 2024-03-31T15:15:05.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:05.801+0000 7efd85a00640 1 --2- 172.21.15.16:0/180927432 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7efd5c0683d0 0x7efd5c06a890 unknown :-1 s=CLOSED pgs=780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:05.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:05.801+0000 7efd85a00640 1 --2- 172.21.15.16:0/180927432 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efd800d0a60 0x7efd800b6700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:05.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:05.801+0000 7efd85a00640 1 --2- 172.21.15.16:0/180927432 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7efd800cf150 0x7efd800b61c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:05.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:05.801+0000 7efd85a00640 1 --2- 172.21.15.16:0/180927432 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efd8006f920 0x7efd80160db0 unknown :-1 s=CLOSED pgs=1857 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:05.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:05.801+0000 7efd85a00640 1 -- 172.21.15.16:0/180927432 >> 172.21.15.16:0/180927432 conn(0x7efd80067930 msgr2=0x7efd800d2410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:15:05.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:05.801+0000 7efd85a00640 1 -- 172.21.15.16:0/180927432 shutdown_connections 2024-03-31T15:15:05.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:05.801+0000 7efd85a00640 1 -- 172.21.15.16:0/180927432 wait complete. 2024-03-31T15:15:05.818 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-31T15:15:05.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:05.973+0000 7fe550000640 1 Processor -- start 2024-03-31T15:15:05.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:05.973+0000 7fe550000640 1 -- start start 2024-03-31T15:15:05.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:05.973+0000 7fe550000640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fe5480cf150 0x7fe5481670a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:05.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:05.973+0000 7fe550000640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe54815ca70 con 0x7fe54806f920 2024-03-31T15:15:05.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:05.973+0000 7fe550000640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe54815cd50 con 0x7fe5480d0a60 2024-03-31T15:15:05.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:05.973+0000 7fe550000640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7fe54813f670 con 0x7fe5480cf150 2024-03-31T15:15:05.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:05.973+0000 7fe54d600640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fe54806f920 0x7fe54806fd00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:36754/0 (socket says 172.21.15.16:36754) 2024-03-31T15:15:05.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:05.973+0000 7fe54d600640 1 -- 172.21.15.16:0/3632718289 learned_addr learned my addr 172.21.15.16:0/3632718289 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:15:05.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:05.973+0000 7fe547e00640 1 -- 172.21.15.16:0/3632718289 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4153577376 0 0) 0x7fe54815ca70 con 0x7fe54806f920 2024-03-31T15:15:05.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:05.973+0000 7fe547e00640 1 -- 172.21.15.16:0/3632718289 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe530003530 con 0x7fe54806f920 2024-03-31T15:15:05.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:05.973+0000 7fe54e000640 1 --2- 172.21.15.16:0/3632718289 >> v2:172.21.15.89:3300/0 conn(0x7fe5480cf150 0x7fe5481670a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:05.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:05.973+0000 7fe547e00640 1 -- 172.21.15.16:0/3632718289 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2164225597 0 0) 0x7fe530003530 con 0x7fe54806f920 2024-03-31T15:15:05.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:05.973+0000 7fe547e00640 1 -- 172.21.15.16:0/3632718289 >> v2:172.21.15.89:3300/0 conn(0x7fe5480cf150 msgr2=0x7fe5481670a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:15:05.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:05.973+0000 7fe547e00640 1 --2- 172.21.15.16:0/3632718289 >> v2:172.21.15.89:3300/0 conn(0x7fe5480cf150 0x7fe5481670a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:05.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:05.973+0000 7fe547e00640 1 -- 172.21.15.16:0/3632718289 >> v1:172.21.15.89:6789/0 conn(0x7fe5480d0a60 legacy=0x7fe5480cea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:05.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:05.973+0000 7fe547e00640 1 -- 172.21.15.16:0/3632718289 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe54815d030 con 0x7fe54806f920 2024-03-31T15:15:05.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:05.973+0000 7fe547e00640 1 -- 172.21.15.16:0/3632718289 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 214807623 0 0) 0x7fe53c001680 con 0x7fe54806f920 2024-03-31T15:15:05.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:05.973+0000 7fe547e00640 1 -- 172.21.15.16:0/3632718289 >> v1:172.21.15.16:6789/0 conn(0x7fe54806f920 legacy=0x7fe54806fd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:05.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:05.973+0000 7fe547e00640 1 --2- 172.21.15.16:0/3632718289 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe5300039c0 0x7fe530003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:05.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:05.973+0000 7fe547e00640 1 --2- 172.21.15.16:0/3632718289 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fe530004810 0x7fe530004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:05.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:05.973+0000 7fe547e00640 1 --2- 172.21.15.16:0/3632718289 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe5300051d0 0x7fe53000b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:05.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:05.973+0000 7fe547e00640 1 -- 172.21.15.16:0/3632718289 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fe54813f670 con 0x7fe5300051d0 2024-03-31T15:15:05.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:05.973+0000 7fe547e00640 1 -- 172.21.15.16:0/3632718289 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fe53000c070 con 0x7fe5300039c0 2024-03-31T15:15:05.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:05.973+0000 7fe547e00640 1 -- 172.21.15.16:0/3632718289 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fe53000c270 con 0x7fe530004810 2024-03-31T15:15:05.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:05.977+0000 7fe54e000640 1 --2- 172.21.15.16:0/3632718289 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe5300051d0 0x7fe53000b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:05.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:05.977+0000 7fe54d600640 1 --2- 172.21.15.16:0/3632718289 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe5300039c0 0x7fe530003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:05.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:05.977+0000 7fe54cc00640 1 --2- 172.21.15.16:0/3632718289 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fe530004810 0x7fe530004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:05.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:05.977+0000 7fe54d600640 1 -- 172.21.15.16:0/3632718289 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fe530004810 msgr2=0x7fe530004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:15:05.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:05.977+0000 7fe54d600640 1 --2- 172.21.15.16:0/3632718289 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fe530004810 0x7fe530004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:05.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:05.977+0000 7fe54d600640 1 -- 172.21.15.16:0/3632718289 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe5300051d0 msgr2=0x7fe53000b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:15:05.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:05.977+0000 7fe54d600640 1 --2- 172.21.15.16:0/3632718289 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe5300051d0 0x7fe53000b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:05.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:05.977+0000 7fe54d600640 1 -- 172.21.15.16:0/3632718289 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe530003530 con 0x7fe5300039c0 2024-03-31T15:15:05.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:05.977+0000 7fe54e000640 1 --2- 172.21.15.16:0/3632718289 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe5300051d0 0x7fe53000b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:15:05.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:05.977+0000 7fe54e000640 1 -- 172.21.15.16:0/3632718289 reap_dead start 2024-03-31T15:15:05.968 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:05.977+0000 7fe54d600640 1 --2- 172.21.15.16:0/3632718289 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe5300039c0 0x7fe530003de0 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7fe53c00c240 tx=0x7fe53c01cc00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ec0a05e2dbeca0df server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:15:05.968 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:05.977+0000 7fe547e00640 1 -- 172.21.15.16:0/3632718289 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe53c007d20 con 0x7fe5300039c0 2024-03-31T15:15:05.968 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:05.977+0000 7fe547e00640 1 -- 172.21.15.16:0/3632718289 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe53c0a9070 con 0x7fe5300039c0 2024-03-31T15:15:05.968 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:05.977+0000 7fe547e00640 1 -- 172.21.15.16:0/3632718289 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe53c0a5da0 con 0x7fe5300039c0 2024-03-31T15:15:05.969 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:05.981+0000 7fe550000640 1 -- 172.21.15.16:0/3632718289 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe5300039c0 msgr2=0x7fe530003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:05.969 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:05.981+0000 7fe550000640 1 --2- 172.21.15.16:0/3632718289 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe5300039c0 0x7fe530003de0 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7fe53c00c240 tx=0x7fe53c01cc00 comp rx=0 tx=0).stop 2024-03-31T15:15:05.969 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:05.981+0000 7fe550000640 1 -- 172.21.15.16:0/3632718289 shutdown_connections 2024-03-31T15:15:05.969 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:05.981+0000 7fe550000640 1 --2- 172.21.15.16:0/3632718289 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe5300039c0 0x7fe530003de0 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:05.969 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:05.981+0000 7fe550000640 1 -- 172.21.15.16:0/3632718289 >> 172.21.15.16:0/3632718289 conn(0x7fe548067930 msgr2=0x7fe5480b3ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:15:05.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:05.981+0000 7fe550000640 1 -- 172.21.15.16:0/3632718289 shutdown_connections 2024-03-31T15:15:05.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:05.981+0000 7fe550000640 1 -- 172.21.15.16:0/3632718289 wait complete. 2024-03-31T15:15:05.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:05.981+0000 7fe550000640 1 Processor -- start 2024-03-31T15:15:05.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:05.981+0000 7fe550000640 1 -- start start 2024-03-31T15:15:05.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:05.985+0000 7fe550000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe5300039c0 0x7fe548137330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:05.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:05.985+0000 7fe550000640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe5481315e0 0x7fe548152340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:05.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:05.985+0000 7fe550000640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fe548152880 0x7fe548154450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:05.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:05.985+0000 7fe550000640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fe54806fd00 con 0x7fe5300039c0 2024-03-31T15:15:05.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:05.985+0000 7fe550000640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fe54815c720 con 0x7fe5481315e0 2024-03-31T15:15:05.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:05.985+0000 7fe550000640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fe54806fe80 con 0x7fe548152880 2024-03-31T15:15:05.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:05.985+0000 7fe54d600640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe5300039c0 0x7fe548137330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:05.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:05.985+0000 7fe54d600640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe5300039c0 0x7fe548137330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50318/0 (socket says 172.21.15.16:50318) 2024-03-31T15:15:05.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:05.985+0000 7fe54d600640 1 -- 172.21.15.16:0/2392188715 learned_addr learned my addr 172.21.15.16:0/2392188715 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:15:05.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:05.985+0000 7fe54d600640 1 -- 172.21.15.16:0/2392188715 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fe548152880 msgr2=0x7fe548154450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:05.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:05.985+0000 7fe54d600640 1 --2- 172.21.15.16:0/2392188715 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fe548152880 0x7fe548154450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:05.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:05.985+0000 7fe54d600640 1 -- 172.21.15.16:0/2392188715 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe5481315e0 msgr2=0x7fe548152340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:05.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:05.985+0000 7fe54d600640 1 --2- 172.21.15.16:0/2392188715 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe5481315e0 0x7fe548152340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:05.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:05.985+0000 7fe54d600640 1 -- 172.21.15.16:0/2392188715 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe548155280 con 0x7fe5300039c0 2024-03-31T15:15:05.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:05.985+0000 7fe54d600640 1 --2- 172.21.15.16:0/2392188715 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe5300039c0 0x7fe548137330 secure :-1 s=READY pgs=1859 cs=0 l=1 rev1=1 crypto rx=0x7fe53c01ca30 tx=0x7fe530005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:15:05.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:05.985+0000 7fe545600640 1 -- 172.21.15.16:0/2392188715 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe53c0bc070 con 0x7fe5300039c0 2024-03-31T15:15:05.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:05.985+0000 7fe545600640 1 -- 172.21.15.16:0/2392188715 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe53c007140 con 0x7fe5300039c0 2024-03-31T15:15:05.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:05.985+0000 7fe545600640 1 -- 172.21.15.16:0/2392188715 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe53c007d20 con 0x7fe5300039c0 2024-03-31T15:15:05.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:05.985+0000 7fe550000640 1 -- 172.21.15.16:0/2392188715 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe548155560 con 0x7fe5300039c0 2024-03-31T15:15:05.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:05.985+0000 7fe550000640 1 -- 172.21.15.16:0/2392188715 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe548155ba0 con 0x7fe5300039c0 2024-03-31T15:15:05.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:05.985+0000 7fe550000640 1 -- 172.21.15.16:0/2392188715 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe548152ca0 con 0x7fe5300039c0 2024-03-31T15:15:05.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:05.989+0000 7fe545600640 1 -- 172.21.15.16:0/2392188715 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fe53c007510 con 0x7fe5300039c0 2024-03-31T15:15:05.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:05.989+0000 7fe545600640 1 --2- 172.21.15.16:0/2392188715 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fe5280683b0 0x7fe52806a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:05.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:05.989+0000 7fe545600640 1 -- 172.21.15.16:0/2392188715 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 5939+0+0 (secure 0 0 0) 0x7fe53c120200 con 0x7fe5300039c0 2024-03-31T15:15:05.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:05.989+0000 7fe54cc00640 1 --2- 172.21.15.16:0/2392188715 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fe5280683b0 0x7fe52806a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:05.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:05.989+0000 7fe54cc00640 1 --2- 172.21.15.16:0/2392188715 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fe5280683b0 0x7fe52806a870 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7fe534000c10 tx=0x7fe534009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:15:05.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:05.993+0000 7fe545600640 1 -- 172.21.15.16:0/2392188715 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fe53c016910 con 0x7fe5300039c0 2024-03-31T15:15:06.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:06.189+0000 7fe550000640 1 -- 172.21.15.16:0/2392188715 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fe548069950 con 0x7fe5300039c0 2024-03-31T15:15:06.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:06.189+0000 7fe545600640 1 -- 172.21.15.16:0/2392188715 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v113) ==== 104+0+11 (secure 0 0 0) 0x7fe53c0f0630 con 0x7fe5300039c0 2024-03-31T15:15:06.180 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2024-03-31T15:15:06.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:06.193+0000 7fe550000640 1 -- 172.21.15.16:0/2392188715 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fe5280683b0 msgr2=0x7fe52806a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:06.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:06.193+0000 7fe550000640 1 --2- 172.21.15.16:0/2392188715 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fe5280683b0 0x7fe52806a870 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7fe534000c10 tx=0x7fe534009040 comp rx=0 tx=0).stop 2024-03-31T15:15:06.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:06.193+0000 7fe550000640 1 -- 172.21.15.16:0/2392188715 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe5300039c0 msgr2=0x7fe548137330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:06.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:06.193+0000 7fe550000640 1 --2- 172.21.15.16:0/2392188715 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe5300039c0 0x7fe548137330 secure :-1 s=READY pgs=1859 cs=0 l=1 rev1=1 crypto rx=0x7fe53c01ca30 tx=0x7fe530005840 comp rx=0 tx=0).stop 2024-03-31T15:15:06.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:06.193+0000 7fe550000640 1 -- 172.21.15.16:0/2392188715 shutdown_connections 2024-03-31T15:15:06.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:06.193+0000 7fe550000640 1 --2- 172.21.15.16:0/2392188715 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fe5280683b0 0x7fe52806a870 unknown :-1 s=CLOSED pgs=781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:06.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:06.193+0000 7fe550000640 1 --2- 172.21.15.16:0/2392188715 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fe548152880 0x7fe548154450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:06.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:06.193+0000 7fe550000640 1 --2- 172.21.15.16:0/2392188715 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe5481315e0 0x7fe548152340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:06.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:06.193+0000 7fe550000640 1 --2- 172.21.15.16:0/2392188715 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe5300039c0 0x7fe548137330 unknown :-1 s=CLOSED pgs=1859 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:06.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:06.193+0000 7fe550000640 1 -- 172.21.15.16:0/2392188715 >> 172.21.15.16:0/2392188715 conn(0x7fe548067930 msgr2=0x7fe548159390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:15:06.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:06.193+0000 7fe550000640 1 -- 172.21.15.16:0/2392188715 shutdown_connections 2024-03-31T15:15:06.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:06.193+0000 7fe550000640 1 -- 172.21.15.16:0/2392188715 wait complete. 2024-03-31T15:15:06.202 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-31T15:15:06.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:06.357+0000 7f3dd4e00640 1 Processor -- start 2024-03-31T15:15:06.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:06.357+0000 7f3dd4e00640 1 -- start start 2024-03-31T15:15:06.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:06.357+0000 7f3dd4e00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f3dd006f920 0x7f3dd006fd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:06.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:06.357+0000 7f3dd4e00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f3dd00cf150 0x7f3dd01419c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:06.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:06.357+0000 7f3dd4e00640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3dd0070240 con 0x7f3dd00d0a60 2024-03-31T15:15:06.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:06.357+0000 7f3dd4e00640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f3dd013f670 con 0x7f3dd006f920 2024-03-31T15:15:06.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:06.357+0000 7f3dd4e00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f3dd015ca70 con 0x7f3dd00cf150 2024-03-31T15:15:06.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:06.357+0000 7f3dc7e00640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f3dd00d0a60 0x7f3dd00cea40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:36770/0 (socket says 172.21.15.16:36770) 2024-03-31T15:15:06.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:06.357+0000 7f3dc7e00640 1 -- 172.21.15.16:0/3268899546 learned_addr learned my addr 172.21.15.16:0/3268899546 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:15:06.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:06.357+0000 7f3dcde00640 1 --2- 172.21.15.16:0/3268899546 >> v2:172.21.15.89:3300/0 conn(0x7f3dd006f920 0x7f3dd006fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:06.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:06.357+0000 7f3dce800640 1 --2- 172.21.15.16:0/3268899546 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f3dd00cf150 0x7f3dd01419c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:06.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:06.357+0000 7f3dcd400640 1 -- 172.21.15.16:0/3268899546 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 854241745 0 0) 0x7f3dd0070240 con 0x7f3dd00d0a60 2024-03-31T15:15:06.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:06.357+0000 7f3dcd400640 1 -- 172.21.15.16:0/3268899546 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3db4003530 con 0x7f3dd00d0a60 2024-03-31T15:15:06.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:06.357+0000 7f3dcd400640 1 -- 172.21.15.16:0/3268899546 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3386453606 0 0) 0x7f3db4003530 con 0x7f3dd00d0a60 2024-03-31T15:15:06.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:06.357+0000 7f3dcd400640 1 -- 172.21.15.16:0/3268899546 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f3dd00cf150 msgr2=0x7f3dd01419c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:15:06.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:06.357+0000 7f3dcd400640 1 --2- 172.21.15.16:0/3268899546 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f3dd00cf150 0x7f3dd01419c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:06.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:06.357+0000 7f3dcd400640 1 -- 172.21.15.16:0/3268899546 >> v2:172.21.15.89:3300/0 conn(0x7f3dd006f920 msgr2=0x7f3dd006fd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:15:06.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:06.357+0000 7f3dcd400640 1 --2- 172.21.15.16:0/3268899546 >> v2:172.21.15.89:3300/0 conn(0x7f3dd006f920 0x7f3dd006fd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:06.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:06.357+0000 7f3dcd400640 1 -- 172.21.15.16:0/3268899546 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3dd015cd50 con 0x7f3dd00d0a60 2024-03-31T15:15:06.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:06.357+0000 7f3dcd400640 1 -- 172.21.15.16:0/3268899546 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 214807623 0 0) 0x7f3db8001680 con 0x7f3dd00d0a60 2024-03-31T15:15:06.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:06.357+0000 7f3dcd400640 1 -- 172.21.15.16:0/3268899546 >> v1:172.21.15.16:6789/0 conn(0x7f3dd00d0a60 legacy=0x7f3dd00cea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:06.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:06.357+0000 7f3dcd400640 1 --2- 172.21.15.16:0/3268899546 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f3dd00cf150 0x7f3db40039c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:06.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:06.357+0000 7f3dcd400640 1 --2- 172.21.15.16:0/3268899546 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3db4003f90 0x7f3db4004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:06.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:06.357+0000 7f3dcd400640 1 --2- 172.21.15.16:0/3268899546 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3db4004d50 0x7f3db40051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:06.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:06.357+0000 7f3dcd400640 1 -- 172.21.15.16:0/3268899546 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f3dd013f670 con 0x7f3db4003f90 2024-03-31T15:15:06.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:06.357+0000 7f3dcd400640 1 -- 172.21.15.16:0/3268899546 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f3dd015ca70 con 0x7f3db4004d50 2024-03-31T15:15:06.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:06.357+0000 7f3dcd400640 1 -- 172.21.15.16:0/3268899546 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f3db400b9d0 con 0x7f3dd00cf150 2024-03-31T15:15:06.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:06.357+0000 7f3dc7e00640 1 --2- 172.21.15.16:0/3268899546 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3db4003f90 0x7f3db4004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:06.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:06.357+0000 7f3dcde00640 1 --2- 172.21.15.16:0/3268899546 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f3dd00cf150 0x7f3db40039c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:06.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:06.357+0000 7f3dce800640 1 --2- 172.21.15.16:0/3268899546 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3db4004d50 0x7f3db40051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:06.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:06.357+0000 7f3dc7e00640 1 -- 172.21.15.16:0/3268899546 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f3dd00cf150 msgr2=0x7f3db40039c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:15:06.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:06.357+0000 7f3dc7e00640 1 --2- 172.21.15.16:0/3268899546 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f3dd00cf150 0x7f3db40039c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:06.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:06.357+0000 7f3dc7e00640 1 -- 172.21.15.16:0/3268899546 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3db4004d50 msgr2=0x7f3db40051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:15:06.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:06.357+0000 7f3dc7e00640 1 --2- 172.21.15.16:0/3268899546 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3db4004d50 0x7f3db40051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:06.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:06.357+0000 7f3dc7e00640 1 -- 172.21.15.16:0/3268899546 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3db4003530 con 0x7f3db4003f90 2024-03-31T15:15:06.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:06.357+0000 7f3dce800640 1 --2- 172.21.15.16:0/3268899546 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3db4004d50 0x7f3db40051d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:15:06.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:06.357+0000 7f3dcde00640 1 --2- 172.21.15.16:0/3268899546 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f3dd00cf150 0x7f3db40039c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:15:06.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:06.361+0000 7f3dc7e00640 1 --2- 172.21.15.16:0/3268899546 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3db4003f90 0x7f3db4004810 secure :-1 s=READY pgs=1861 cs=0 l=1 rev1=1 crypto rx=0x7f3db800c190 tx=0x7f3db801f310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26774381a60572b0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:15:06.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:06.361+0000 7f3dcd400640 1 -- 172.21.15.16:0/3268899546 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3db8007c90 con 0x7f3db4003f90 2024-03-31T15:15:06.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:06.361+0000 7f3dcd400640 1 -- 172.21.15.16:0/3268899546 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3db8005ae0 con 0x7f3db4003f90 2024-03-31T15:15:06.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:06.361+0000 7f3dcd400640 1 -- 172.21.15.16:0/3268899546 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3db8007070 con 0x7f3db4003f90 2024-03-31T15:15:06.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:06.361+0000 7f3dd4e00640 1 -- 172.21.15.16:0/3268899546 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3db4003f90 msgr2=0x7f3db4004810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:06.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:06.361+0000 7f3dd4e00640 1 --2- 172.21.15.16:0/3268899546 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3db4003f90 0x7f3db4004810 secure :-1 s=READY pgs=1861 cs=0 l=1 rev1=1 crypto rx=0x7f3db800c190 tx=0x7f3db801f310 comp rx=0 tx=0).stop 2024-03-31T15:15:06.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:06.361+0000 7f3dce800640 1 -- 172.21.15.16:0/3268899546 reap_dead start 2024-03-31T15:15:06.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:06.361+0000 7f3dd4e00640 1 -- 172.21.15.16:0/3268899546 shutdown_connections 2024-03-31T15:15:06.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:06.361+0000 7f3dd4e00640 1 -- 172.21.15.16:0/3268899546 >> 172.21.15.16:0/3268899546 conn(0x7f3dd0067930 msgr2=0x7f3dd00b3ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:15:06.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:06.361+0000 7f3dd4e00640 1 -- 172.21.15.16:0/3268899546 shutdown_connections 2024-03-31T15:15:06.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:06.361+0000 7f3dd4e00640 1 -- 172.21.15.16:0/3268899546 wait complete. 2024-03-31T15:15:06.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:06.365+0000 7f3dd4e00640 1 Processor -- start 2024-03-31T15:15:06.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:06.365+0000 7f3dd4e00640 1 -- start start 2024-03-31T15:15:06.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:06.365+0000 7f3dd4e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3db4003f90 0x7f3dd015b7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:06.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:06.365+0000 7f3dd4e00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3dd0153b20 0x7f3dd013ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:06.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:06.365+0000 7f3dd4e00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f3dd006db90 0x7f3dd006e010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:06.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:06.365+0000 7f3dd4e00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f3dd015c720 con 0x7f3db4003f90 2024-03-31T15:15:06.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:06.365+0000 7f3dd4e00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f3dd015d310 con 0x7f3dd0153b20 2024-03-31T15:15:06.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:06.365+0000 7f3dd4e00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f3dd01419c0 con 0x7f3dd006db90 2024-03-31T15:15:06.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:06.365+0000 7f3dcde00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3db4003f90 0x7f3dd015b7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:06.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:06.365+0000 7f3dcde00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3db4003f90 0x7f3dd015b7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50338/0 (socket says 172.21.15.16:50338) 2024-03-31T15:15:06.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:06.365+0000 7f3dcde00640 1 -- 172.21.15.16:0/2940431756 learned_addr learned my addr 172.21.15.16:0/2940431756 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:15:06.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:06.365+0000 7f3dc7e00640 1 --2- 172.21.15.16:0/2940431756 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3dd0153b20 0x7f3dd013ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:06.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:06.365+0000 7f3dce800640 1 --2- 172.21.15.16:0/2940431756 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f3dd006db90 0x7f3dd006e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:06.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:06.365+0000 7f3dcde00640 1 -- 172.21.15.16:0/2940431756 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f3dd006db90 msgr2=0x7f3dd006e010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:06.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:06.365+0000 7f3dcde00640 1 --2- 172.21.15.16:0/2940431756 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f3dd006db90 0x7f3dd006e010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:06.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:06.365+0000 7f3dcde00640 1 -- 172.21.15.16:0/2940431756 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3dd0153b20 msgr2=0x7f3dd013ea20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:06.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:06.365+0000 7f3dcde00640 1 --2- 172.21.15.16:0/2940431756 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3dd0153b20 0x7f3dd013ea20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:06.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:06.365+0000 7f3dcde00640 1 -- 172.21.15.16:0/2940431756 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3dd0154090 con 0x7f3db4003f90 2024-03-31T15:15:06.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:06.365+0000 7f3dc7e00640 1 --2- 172.21.15.16:0/2940431756 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3dd0153b20 0x7f3dd013ea20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:15:06.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:06.365+0000 7f3dcde00640 1 --2- 172.21.15.16:0/2940431756 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3db4003f90 0x7f3dd015b7d0 secure :-1 s=READY pgs=1862 cs=0 l=1 rev1=1 crypto rx=0x7f3dc00116a0 tx=0x7f3dc0011b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:15:06.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:06.365+0000 7f3dc6000640 1 -- 172.21.15.16:0/2940431756 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3dc0004220 con 0x7f3db4003f90 2024-03-31T15:15:06.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:06.365+0000 7f3dc6000640 1 -- 172.21.15.16:0/2940431756 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3dc00043c0 con 0x7f3db4003f90 2024-03-31T15:15:06.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:06.365+0000 7f3dc6000640 1 -- 172.21.15.16:0/2940431756 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3dc000f9e0 con 0x7f3db4003f90 2024-03-31T15:15:06.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:06.365+0000 7f3dd4e00640 1 -- 172.21.15.16:0/2940431756 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3dd0154370 con 0x7f3db4003f90 2024-03-31T15:15:06.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:06.365+0000 7f3dd4e00640 1 -- 172.21.15.16:0/2940431756 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3dd0154930 con 0x7f3db4003f90 2024-03-31T15:15:06.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:06.369+0000 7f3dd4e00640 1 -- 172.21.15.16:0/2940431756 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3dd0069950 con 0x7f3db4003f90 2024-03-31T15:15:06.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:06.369+0000 7f3dc6000640 1 -- 172.21.15.16:0/2940431756 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f3dc000fb80 con 0x7f3db4003f90 2024-03-31T15:15:06.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:06.369+0000 7f3dc6000640 1 --2- 172.21.15.16:0/2940431756 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f3dac068380 0x7f3dac06a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:06.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:06.369+0000 7f3dc6000640 1 -- 172.21.15.16:0/2940431756 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 5939+0+0 (secure 0 0 0) 0x7f3dc0087230 con 0x7f3db4003f90 2024-03-31T15:15:06.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:06.369+0000 7f3dc7e00640 1 --2- 172.21.15.16:0/2940431756 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f3dac068380 0x7f3dac06a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:06.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:06.369+0000 7f3dc7e00640 1 --2- 172.21.15.16:0/2940431756 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f3dac068380 0x7f3dac06a840 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f3db80ae000 tx=0x7f3db80019f0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:15:06.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:06.373+0000 7f3dc6000640 1 -- 172.21.15.16:0/2940431756 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f3dc0019030 con 0x7f3db4003f90 2024-03-31T15:15:06.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:06.537+0000 7f3dd4e00640 1 -- 172.21.15.16:0/2940431756 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f3dd00b4a20 con 0x7f3db4003f90 2024-03-31T15:15:06.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:06.537+0000 7f3dc6000640 1 -- 172.21.15.16:0/2940431756 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v113) ==== 100+0+11 (secure 0 0 0) 0x7f3dc00576f0 con 0x7f3db4003f90 2024-03-31T15:15:06.527 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 64 2024-03-31T15:15:06.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:06.537+0000 7f3dd4e00640 1 -- 172.21.15.16:0/2940431756 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f3dac068380 msgr2=0x7f3dac06a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:06.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:06.537+0000 7f3dd4e00640 1 --2- 172.21.15.16:0/2940431756 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f3dac068380 0x7f3dac06a840 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f3db80ae000 tx=0x7f3db80019f0 comp rx=0 tx=0).stop 2024-03-31T15:15:06.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:06.537+0000 7f3dd4e00640 1 -- 172.21.15.16:0/2940431756 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3db4003f90 msgr2=0x7f3dd015b7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:06.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:06.537+0000 7f3dd4e00640 1 --2- 172.21.15.16:0/2940431756 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3db4003f90 0x7f3dd015b7d0 secure :-1 s=READY pgs=1862 cs=0 l=1 rev1=1 crypto rx=0x7f3dc00116a0 tx=0x7f3dc0011b60 comp rx=0 tx=0).stop 2024-03-31T15:15:06.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:06.537+0000 7f3dd4e00640 1 -- 172.21.15.16:0/2940431756 shutdown_connections 2024-03-31T15:15:06.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:06.537+0000 7f3dd4e00640 1 --2- 172.21.15.16:0/2940431756 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f3dac068380 0x7f3dac06a840 unknown :-1 s=CLOSED pgs=782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:06.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:06.537+0000 7f3dd4e00640 1 --2- 172.21.15.16:0/2940431756 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f3dd006db90 0x7f3dd006e010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:06.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:06.537+0000 7f3dd4e00640 1 --2- 172.21.15.16:0/2940431756 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3dd0153b20 0x7f3dd013ea20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:06.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:06.537+0000 7f3dd4e00640 1 --2- 172.21.15.16:0/2940431756 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3db4003f90 0x7f3dd015b7d0 unknown :-1 s=CLOSED pgs=1862 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:06.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:06.541+0000 7f3dd4e00640 1 -- 172.21.15.16:0/2940431756 >> 172.21.15.16:0/2940431756 conn(0x7f3dd0067930 msgr2=0x7f3dd00d1740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:15:06.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:06.541+0000 7f3dd4e00640 1 -- 172.21.15.16:0/2940431756 shutdown_connections 2024-03-31T15:15:06.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:06.541+0000 7f3dd4e00640 1 -- 172.21.15.16:0/2940431756 wait complete. 2024-03-31T15:15:06.548 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-31T15:15:06.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:06.701+0000 7f9adee00640 1 Processor -- start 2024-03-31T15:15:06.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:06.701+0000 7f9adee00640 1 -- start start 2024-03-31T15:15:06.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:06.701+0000 7f9adee00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f9ad8145d10 0x7f9ad80cf790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:06.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:06.701+0000 7f9adee00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f9ad80728b0 0x7f9ad80b1cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:06.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:06.701+0000 7f9adee00640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9ad8160e70 con 0x7f9ad80b2200 2024-03-31T15:15:06.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:06.701+0000 7f9adee00640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f9ad81602f0 con 0x7f9ad8145d10 2024-03-31T15:15:06.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:06.701+0000 7f9adee00640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f9ad81605d0 con 0x7f9ad80728b0 2024-03-31T15:15:06.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:06.701+0000 7f9ad7e00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f9ad8145d10 0x7f9ad80cf790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:06.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:06.701+0000 7f9acfe00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f9ad80728b0 0x7f9ad80b1cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:06.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:06.701+0000 7f9ad7e00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f9ad8145d10 0x7f9ad80cf790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50352/0 (socket says 172.21.15.16:50352) 2024-03-31T15:15:06.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:06.701+0000 7f9ad7e00640 1 -- 172.21.15.16:0/2925989637 learned_addr learned my addr 172.21.15.16:0/2925989637 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:15:06.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:06.701+0000 7f9adce00640 1 --1- 172.21.15.16:0/2925989637 >> v1:172.21.15.89:6789/0 conn(0x7f9ad80b2200 0x7f9ad815cd50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.16:56400/0 (socket says 172.21.15.16:56400) 2024-03-31T15:15:06.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:06.701+0000 7f9ad7e00640 1 -- 172.21.15.16:0/2925989637 >> v2:172.21.15.89:3300/0 conn(0x7f9ad80728b0 msgr2=0x7f9ad80b1cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:15:06.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:06.701+0000 7f9ad7e00640 1 --2- 172.21.15.16:0/2925989637 >> v2:172.21.15.89:3300/0 conn(0x7f9ad80728b0 0x7f9ad80b1cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:06.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:06.701+0000 7f9ad7e00640 1 -- 172.21.15.16:0/2925989637 >> v1:172.21.15.89:6789/0 conn(0x7f9ad80b2200 legacy=0x7f9ad815cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:06.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:06.701+0000 7f9ad7e00640 1 -- 172.21.15.16:0/2925989637 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ad80cfcd0 con 0x7f9ad8145d10 2024-03-31T15:15:06.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:06.705+0000 7f9ad7e00640 1 --2- 172.21.15.16:0/2925989637 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9ad8145d10 0x7f9ad80cf790 secure :-1 s=READY pgs=1863 cs=0 l=1 rev1=1 crypto rx=0x7f9ad80b2980 tx=0x7f9ac801c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=23039cf1785b9986 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:15:06.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:06.705+0000 7f9ad7400640 1 -- 172.21.15.16:0/2925989637 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ac80b0070 con 0x7f9ad8145d10 2024-03-31T15:15:06.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:06.705+0000 7f9ad7400640 1 -- 172.21.15.16:0/2925989637 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9ac80ab600 con 0x7f9ad8145d10 2024-03-31T15:15:06.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:06.705+0000 7f9ad7400640 1 -- 172.21.15.16:0/2925989637 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ac80ab9d0 con 0x7f9ad8145d10 2024-03-31T15:15:06.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:06.705+0000 7f9adee00640 1 -- 172.21.15.16:0/2925989637 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9ad8145d10 msgr2=0x7f9ad80cf790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:06.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:06.705+0000 7f9adee00640 1 --2- 172.21.15.16:0/2925989637 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9ad8145d10 0x7f9ad80cf790 secure :-1 s=READY pgs=1863 cs=0 l=1 rev1=1 crypto rx=0x7f9ad80b2980 tx=0x7f9ac801c520 comp rx=0 tx=0).stop 2024-03-31T15:15:06.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:06.705+0000 7f9adee00640 1 -- 172.21.15.16:0/2925989637 shutdown_connections 2024-03-31T15:15:06.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:06.705+0000 7f9adee00640 1 --2- 172.21.15.16:0/2925989637 >> v2:172.21.15.89:3300/0 conn(0x7f9ad80728b0 0x7f9ad80b1cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:06.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:06.705+0000 7f9adee00640 1 --2- 172.21.15.16:0/2925989637 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9ad8145d10 0x7f9ad80cf790 unknown :-1 s=CLOSED pgs=1863 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:06.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:06.705+0000 7f9adee00640 1 -- 172.21.15.16:0/2925989637 >> 172.21.15.16:0/2925989637 conn(0x7f9ad806b620 msgr2=0x7f9ad806ba30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:15:06.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:06.705+0000 7f9adee00640 1 -- 172.21.15.16:0/2925989637 shutdown_connections 2024-03-31T15:15:06.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:06.705+0000 7f9adee00640 1 -- 172.21.15.16:0/2925989637 wait complete. 2024-03-31T15:15:06.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:06.709+0000 7f9adee00640 1 Processor -- start 2024-03-31T15:15:06.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:06.709+0000 7f9adee00640 1 -- start start 2024-03-31T15:15:06.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:06.709+0000 7f9adee00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9ad80728b0 0x7f9ad8141280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:06.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:06.709+0000 7f9adee00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f9ad80b2200 0x7f9ad813fae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:06.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:06.709+0000 7f9adee00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9ad8145d10 0x7f9ad813dc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:06.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:06.709+0000 7f9adee00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f9ad815d870 con 0x7f9ad80728b0 2024-03-31T15:15:06.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:06.709+0000 7f9adee00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f9ad80b2610 con 0x7f9ad8145d10 2024-03-31T15:15:06.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:06.709+0000 7f9adee00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f9ad80cf350 con 0x7f9ad80b2200 2024-03-31T15:15:06.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:06.709+0000 7f9ad7e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9ad80728b0 0x7f9ad8141280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:06.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:06.709+0000 7f9ad7e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9ad80728b0 0x7f9ad8141280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50354/0 (socket says 172.21.15.16:50354) 2024-03-31T15:15:06.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:06.709+0000 7f9ad7e00640 1 -- 172.21.15.16:0/4019563645 learned_addr learned my addr 172.21.15.16:0/4019563645 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:15:06.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:06.709+0000 7f9adce00640 1 --2- 172.21.15.16:0/4019563645 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9ad8145d10 0x7f9ad813dc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:06.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:06.709+0000 7f9acfe00640 1 --2- 172.21.15.16:0/4019563645 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f9ad80b2200 0x7f9ad813fae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:06.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:06.709+0000 7f9ad7e00640 1 -- 172.21.15.16:0/4019563645 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f9ad80b2200 msgr2=0x7f9ad813fae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:06.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:06.709+0000 7f9ad7e00640 1 --2- 172.21.15.16:0/4019563645 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f9ad80b2200 0x7f9ad813fae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:06.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:06.709+0000 7f9ad7e00640 1 -- 172.21.15.16:0/4019563645 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9ad8145d10 msgr2=0x7f9ad813dc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:06.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:06.709+0000 7f9ad7e00640 1 --2- 172.21.15.16:0/4019563645 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9ad8145d10 0x7f9ad813dc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:06.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:06.709+0000 7f9ad7e00640 1 -- 172.21.15.16:0/4019563645 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ad81401e0 con 0x7f9ad80728b0 2024-03-31T15:15:06.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:06.709+0000 7f9adce00640 1 --2- 172.21.15.16:0/4019563645 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9ad8145d10 0x7f9ad813dc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:15:06.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:06.709+0000 7f9ad7e00640 1 --2- 172.21.15.16:0/4019563645 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9ad80728b0 0x7f9ad8141280 secure :-1 s=READY pgs=1864 cs=0 l=1 rev1=1 crypto rx=0x7f9ac801abd0 tx=0x7f9ac801ad60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:15:06.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:06.709+0000 7f9ad4c00640 1 -- 172.21.15.16:0/4019563645 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ac80b0040 con 0x7f9ad80728b0 2024-03-31T15:15:06.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:06.709+0000 7f9ad4c00640 1 -- 172.21.15.16:0/4019563645 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9ac80ab400 con 0x7f9ad80728b0 2024-03-31T15:15:06.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:06.709+0000 7f9adee00640 1 -- 172.21.15.16:0/4019563645 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9ad81404c0 con 0x7f9ad80728b0 2024-03-31T15:15:06.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:06.709+0000 7f9ad4c00640 1 -- 172.21.15.16:0/4019563645 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ac80b6660 con 0x7f9ad80728b0 2024-03-31T15:15:06.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:06.709+0000 7f9adee00640 1 -- 172.21.15.16:0/4019563645 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9ad8140a80 con 0x7f9ad80728b0 2024-03-31T15:15:06.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:06.713+0000 7f9adee00640 1 -- 172.21.15.16:0/4019563645 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9ad8069950 con 0x7f9ad80728b0 2024-03-31T15:15:06.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:06.713+0000 7f9ad4c00640 1 -- 172.21.15.16:0/4019563645 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f9ac80be030 con 0x7f9ad80728b0 2024-03-31T15:15:06.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:06.713+0000 7f9ad4c00640 1 --2- 172.21.15.16:0/4019563645 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f9ab4068380 0x7f9ab406a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:06.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:06.713+0000 7f9ad4c00640 1 -- 172.21.15.16:0/4019563645 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 5939+0+0 (secure 0 0 0) 0x7f9ac81201f0 con 0x7f9ad80728b0 2024-03-31T15:15:06.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:06.713+0000 7f9acfe00640 1 --2- 172.21.15.16:0/4019563645 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f9ab4068380 0x7f9ab406a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:06.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:06.713+0000 7f9acfe00640 1 --2- 172.21.15.16:0/4019563645 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f9ab4068380 0x7f9ab406a840 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7f9ad8145390 tx=0x7f9ac0009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:15:06.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:06.717+0000 7f9ad4c00640 1 -- 172.21.15.16:0/4019563645 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f9ac80c5060 con 0x7f9ad80728b0 2024-03-31T15:15:06.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:06.917+0000 7f9adee00640 1 -- 172.21.15.16:0/4019563645 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9ad815ee70 con 0x7f9ad80728b0 2024-03-31T15:15:06.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:06.917+0000 7f9ad4c00640 1 -- 172.21.15.16:0/4019563645 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 112 v112) ==== 95+0+5185 (secure 0 0 0) 0x7f9ac8016110 con 0x7f9ad80728b0 2024-03-31T15:15:06.909 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:15:06.909 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":112,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7631,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6836/1628897112","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":1628897112},{"type":"v1","addr":"172.21.15.89:6836","nonce":1628897112}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":94},{"gid":8746,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/1036724837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6835","nonce":1036724837},{"type":"v1","addr":"172.21.15.89:6837","nonce":1036724837}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":94},{"gid":9192,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6834/472139147","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":472139147},{"type":"v1","addr":"172.21.15.16:6834","nonce":472139147}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":94}],"filesystems":[{"mdsmap":{"epoch":112,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-31T15:14:08.652180+0000","modified":"2024-03-31T15:14:58.141614+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":113,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":9762},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9762":{"gid":9762,"name":"c","rank":0,"incarnation":109,"state":"up:active","state_seq":4,"addr":"172.21.15.16:6835/2152452999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6833","nonce":2152452999},{"type":"v1","addr":"172.21.15.16:6835","nonce":2152452999}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9762,"qdb_cluster":[9762]},"id":8}]} 2024-03-31T15:15:06.910 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 112 2024-03-31T15:15:06.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:06.921+0000 7f9adee00640 1 -- 172.21.15.16:0/4019563645 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f9ab4068380 msgr2=0x7f9ab406a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:06.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:06.921+0000 7f9adee00640 1 --2- 172.21.15.16:0/4019563645 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f9ab4068380 0x7f9ab406a840 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7f9ad8145390 tx=0x7f9ac0009190 comp rx=0 tx=0).stop 2024-03-31T15:15:06.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:06.921+0000 7f9adee00640 1 -- 172.21.15.16:0/4019563645 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9ad80728b0 msgr2=0x7f9ad8141280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:06.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:06.921+0000 7f9adee00640 1 --2- 172.21.15.16:0/4019563645 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9ad80728b0 0x7f9ad8141280 secure :-1 s=READY pgs=1864 cs=0 l=1 rev1=1 crypto rx=0x7f9ac801abd0 tx=0x7f9ac801ad60 comp rx=0 tx=0).stop 2024-03-31T15:15:06.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:06.921+0000 7f9adee00640 1 -- 172.21.15.16:0/4019563645 shutdown_connections 2024-03-31T15:15:06.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:06.921+0000 7f9adee00640 1 --2- 172.21.15.16:0/4019563645 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f9ab4068380 0x7f9ab406a840 unknown :-1 s=CLOSED pgs=783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:06.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:06.921+0000 7f9adee00640 1 --2- 172.21.15.16:0/4019563645 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9ad8145d10 0x7f9ad813dc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:06.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:06.921+0000 7f9adee00640 1 --2- 172.21.15.16:0/4019563645 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f9ad80b2200 0x7f9ad813fae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:06.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:06.921+0000 7f9adee00640 1 --2- 172.21.15.16:0/4019563645 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9ad80728b0 0x7f9ad8141280 unknown :-1 s=CLOSED pgs=1864 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:06.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:06.921+0000 7f9adee00640 1 -- 172.21.15.16:0/4019563645 >> 172.21.15.16:0/4019563645 conn(0x7f9ad806b620 msgr2=0x7f9ad815e3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:15:06.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:06.921+0000 7f9adee00640 1 -- 172.21.15.16:0/4019563645 shutdown_connections 2024-03-31T15:15:06.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:06.921+0000 7f9adee00640 1 -- 172.21.15.16:0/4019563645 wait complete. 2024-03-31T15:15:06.930 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-31T15:15:07.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:07.081+0000 7f4a66a00640 1 Processor -- start 2024-03-31T15:15:07.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:07.081+0000 7f4a66a00640 1 -- start start 2024-03-31T15:15:07.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:07.081+0000 7f4a66a00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f4a60145d10 0x7f4a600cf790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:07.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:07.081+0000 7f4a66a00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f4a600728b0 0x7f4a600b1cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:07.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:07.081+0000 7f4a66a00640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4a60160e70 con 0x7f4a600b2200 2024-03-31T15:15:07.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:07.081+0000 7f4a66a00640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f4a601602f0 con 0x7f4a60145d10 2024-03-31T15:15:07.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:07.081+0000 7f4a66a00640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f4a601605d0 con 0x7f4a600728b0 2024-03-31T15:15:07.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:07.081+0000 7f4a5fe00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f4a60145d10 0x7f4a600cf790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:07.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:07.081+0000 7f4a5fe00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f4a60145d10 0x7f4a600cf790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46052/0 (socket says 172.21.15.16:46052) 2024-03-31T15:15:07.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:07.081+0000 7f4a5fe00640 1 -- 172.21.15.16:0/2014510675 learned_addr learned my addr 172.21.15.16:0/2014510675 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:15:07.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:07.081+0000 7f4a5f400640 1 --2- 172.21.15.16:0/2014510675 >> v2:172.21.15.89:3300/0 conn(0x7f4a600728b0 0x7f4a600b1cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:07.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:07.081+0000 7f4a5fe00640 1 -- 172.21.15.16:0/2014510675 >> v2:172.21.15.89:3300/0 conn(0x7f4a600728b0 msgr2=0x7f4a600b1cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:15:07.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:07.081+0000 7f4a5fe00640 1 --2- 172.21.15.16:0/2014510675 >> v2:172.21.15.89:3300/0 conn(0x7f4a600728b0 0x7f4a600b1cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:07.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:07.081+0000 7f4a5fe00640 1 -- 172.21.15.16:0/2014510675 >> v1:172.21.15.89:6789/0 conn(0x7f4a600b2200 legacy=0x7f4a6015cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:07.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:07.081+0000 7f4a5fe00640 1 -- 172.21.15.16:0/2014510675 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a600cfcd0 con 0x7f4a60145d10 2024-03-31T15:15:07.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:07.081+0000 7f4a5f400640 1 --2- 172.21.15.16:0/2014510675 >> v2:172.21.15.89:3300/0 conn(0x7f4a600728b0 0x7f4a600b1cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:15:07.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:07.085+0000 7f4a5fe00640 1 --2- 172.21.15.16:0/2014510675 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4a60145d10 0x7f4a600cf790 secure :-1 s=READY pgs=1865 cs=0 l=1 rev1=1 crypto rx=0x7f4a600b2980 tx=0x7f4a5001c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce0351d2bfd7c227 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:15:07.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:07.085+0000 7f4a5ea00640 1 -- 172.21.15.16:0/2014510675 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a500b0070 con 0x7f4a60145d10 2024-03-31T15:15:07.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:07.085+0000 7f4a5ea00640 1 -- 172.21.15.16:0/2014510675 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4a500ab600 con 0x7f4a60145d10 2024-03-31T15:15:07.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:07.085+0000 7f4a5ea00640 1 -- 172.21.15.16:0/2014510675 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a500ab9d0 con 0x7f4a60145d10 2024-03-31T15:15:07.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:07.085+0000 7f4a66a00640 1 -- 172.21.15.16:0/2014510675 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4a60145d10 msgr2=0x7f4a600cf790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:07.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:07.085+0000 7f4a66a00640 1 --2- 172.21.15.16:0/2014510675 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4a60145d10 0x7f4a600cf790 secure :-1 s=READY pgs=1865 cs=0 l=1 rev1=1 crypto rx=0x7f4a600b2980 tx=0x7f4a5001c520 comp rx=0 tx=0).stop 2024-03-31T15:15:07.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:07.085+0000 7f4a66a00640 1 -- 172.21.15.16:0/2014510675 shutdown_connections 2024-03-31T15:15:07.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:07.085+0000 7f4a66a00640 1 --2- 172.21.15.16:0/2014510675 >> v2:172.21.15.89:3300/0 conn(0x7f4a600728b0 0x7f4a600b1cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:07.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:07.085+0000 7f4a66a00640 1 --2- 172.21.15.16:0/2014510675 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4a60145d10 0x7f4a600cf790 unknown :-1 s=CLOSED pgs=1865 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:07.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:07.085+0000 7f4a66a00640 1 -- 172.21.15.16:0/2014510675 >> 172.21.15.16:0/2014510675 conn(0x7f4a6006b620 msgr2=0x7f4a6006ba30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:15:07.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:07.085+0000 7f4a66a00640 1 -- 172.21.15.16:0/2014510675 shutdown_connections 2024-03-31T15:15:07.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:07.085+0000 7f4a66a00640 1 -- 172.21.15.16:0/2014510675 wait complete. 2024-03-31T15:15:07.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:07.089+0000 7f4a66a00640 1 Processor -- start 2024-03-31T15:15:07.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:07.089+0000 7f4a66a00640 1 -- start start 2024-03-31T15:15:07.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:07.089+0000 7f4a66a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4a600728b0 0x7f4a60166160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:07.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:07.089+0000 7f4a66a00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f4a600b2200 0x7f4a60140e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:07.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:07.089+0000 7f4a66a00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4a60145d10 0x7f4a6013baa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:07.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:07.089+0000 7f4a66a00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f4a6015d870 con 0x7f4a600728b0 2024-03-31T15:15:07.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:07.089+0000 7f4a66a00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f4a600b2610 con 0x7f4a60145d10 2024-03-31T15:15:07.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:07.089+0000 7f4a66a00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f4a600cf350 con 0x7f4a600b2200 2024-03-31T15:15:07.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:07.089+0000 7f4a5fe00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4a600728b0 0x7f4a60166160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:07.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:07.089+0000 7f4a5fe00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4a600728b0 0x7f4a60166160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46064/0 (socket says 172.21.15.16:46064) 2024-03-31T15:15:07.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:07.089+0000 7f4a5fe00640 1 -- 172.21.15.16:0/247125841 learned_addr learned my addr 172.21.15.16:0/247125841 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:15:07.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:07.089+0000 7f4a64a00640 1 --2- 172.21.15.16:0/247125841 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4a60145d10 0x7f4a6013baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:07.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:07.089+0000 7f4a5f400640 1 --2- 172.21.15.16:0/247125841 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f4a600b2200 0x7f4a60140e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:07.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:07.089+0000 7f4a5fe00640 1 -- 172.21.15.16:0/247125841 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f4a600b2200 msgr2=0x7f4a60140e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:07.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:07.089+0000 7f4a5fe00640 1 --2- 172.21.15.16:0/247125841 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f4a600b2200 0x7f4a60140e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:07.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:07.089+0000 7f4a5fe00640 1 -- 172.21.15.16:0/247125841 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4a60145d10 msgr2=0x7f4a6013baa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:07.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:07.089+0000 7f4a5fe00640 1 --2- 172.21.15.16:0/247125841 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4a60145d10 0x7f4a6013baa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:07.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:07.089+0000 7f4a5fe00640 1 -- 172.21.15.16:0/247125841 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a600b3f10 con 0x7f4a600728b0 2024-03-31T15:15:07.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:07.089+0000 7f4a5f400640 1 --2- 172.21.15.16:0/247125841 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f4a600b2200 0x7f4a60140e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:15:07.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:07.089+0000 7f4a5fe00640 1 --2- 172.21.15.16:0/247125841 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4a600728b0 0x7f4a60166160 secure :-1 s=READY pgs=1866 cs=0 l=1 rev1=1 crypto rx=0x7f4a5001abd0 tx=0x7f4a5001ad60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:15:07.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:07.089+0000 7f4a57e00640 1 -- 172.21.15.16:0/247125841 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a500b0040 con 0x7f4a600728b0 2024-03-31T15:15:07.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:07.089+0000 7f4a57e00640 1 -- 172.21.15.16:0/247125841 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4a500ab400 con 0x7f4a600728b0 2024-03-31T15:15:07.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:07.089+0000 7f4a57e00640 1 -- 172.21.15.16:0/247125841 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a500b6660 con 0x7f4a600728b0 2024-03-31T15:15:07.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:07.089+0000 7f4a66a00640 1 -- 172.21.15.16:0/247125841 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4a601653a0 con 0x7f4a600728b0 2024-03-31T15:15:07.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:07.089+0000 7f4a66a00640 1 -- 172.21.15.16:0/247125841 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4a60165960 con 0x7f4a600728b0 2024-03-31T15:15:07.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:07.093+0000 7f4a66a00640 1 -- 172.21.15.16:0/247125841 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4a60069950 con 0x7f4a600728b0 2024-03-31T15:15:07.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:07.093+0000 7f4a57e00640 1 -- 172.21.15.16:0/247125841 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f4a500be030 con 0x7f4a600728b0 2024-03-31T15:15:07.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:07.093+0000 7f4a57e00640 1 --2- 172.21.15.16:0/247125841 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f4a40068380 0x7f4a4006a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:07.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:07.093+0000 7f4a57e00640 1 -- 172.21.15.16:0/247125841 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 5939+0+0 (secure 0 0 0) 0x7f4a50120320 con 0x7f4a600728b0 2024-03-31T15:15:07.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:07.093+0000 7f4a5f400640 1 --2- 172.21.15.16:0/247125841 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f4a40068380 0x7f4a4006a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:07.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:07.093+0000 7f4a5f400640 1 --2- 172.21.15.16:0/247125841 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f4a40068380 0x7f4a4006a840 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7f4a600b52d0 tx=0x7f4a4c008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:15:07.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:07.097+0000 7f4a57e00640 1 -- 172.21.15.16:0/247125841 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f4a500c5060 con 0x7f4a600728b0 2024-03-31T15:15:07.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:07.269+0000 7f4a66a00640 1 -- 172.21.15.16:0/247125841 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f4a6015ee70 con 0x7f4a600728b0 2024-03-31T15:15:07.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:07.269+0000 7f4a57e00640 1 -- 172.21.15.16:0/247125841 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v113) ==== 74+0+16907 (secure 0 0 0) 0x7f4a50016110 con 0x7f4a600728b0 2024-03-31T15:15:07.259 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:15:07.259 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":113,"fsid":"286eb5da-fb47-4550-b290-d9519c13b9a5","created":"2024-03-31T15:03:11.438952+0000","modified":"2024-03-31T15:14:53.117209+0000","last_up_change":"2024-03-31T15:03:22.885535+0000","last_in_change":"2024-03-31T15:03:13.941667+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-31T15:03:22.550787+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-03-31T15:14:06.417304+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"108","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-03-31T15:14:07.622562+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"109","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"92a66450-ef19-40d8-ac3d-d760b23b0bda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6801","nonce":932712244}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6803","nonce":932712244}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6807","nonce":932712244}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6805","nonce":932712244}]},"public_addr":"172.21.15.16:6801/932712244","cluster_addr":"172.21.15.16:6803/932712244","heartbeat_back_addr":"172.21.15.16:6807/932712244","heartbeat_front_addr":"172.21.15.16:6805/932712244","state":["exists","up"]},{"osd":1,"uuid":"da9d1f2b-1968-4959-b843-faa231988e92","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6810","nonce":1398737533}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6811","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6813","nonce":1398737533}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6821","nonce":1398737533}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6815","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6816","nonce":1398737533}]},"public_addr":"172.21.15.16:6810/1398737533","cluster_addr":"172.21.15.16:6813/1398737533","heartbeat_back_addr":"172.21.15.16:6821/1398737533","heartbeat_front_addr":"172.21.15.16:6816/1398737533","state":["exists","up"]},{"osd":2,"uuid":"01a73662-9a21-4b1f-bcbb-42b9636a4638","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6819","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6822","nonce":3018526646}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6825","nonce":3018526646}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6831","nonce":3018526646}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6827","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6829","nonce":3018526646}]},"public_addr":"172.21.15.16:6822/3018526646","cluster_addr":"172.21.15.16:6825/3018526646","heartbeat_back_addr":"172.21.15.16:6831/3018526646","heartbeat_front_addr":"172.21.15.16:6829/3018526646","state":["exists","up"]},{"osd":3,"uuid":"c7771fd3-e18e-4500-ac38-23cf10ebcd75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6809","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6812","nonce":3360352974}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6817","nonce":3360352974}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6828","nonce":3360352974}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6823","nonce":3360352974}]},"public_addr":"172.21.15.16:6812/3360352974","cluster_addr":"172.21.15.16:6817/3360352974","heartbeat_back_addr":"172.21.15.16:6828/3360352974","heartbeat_front_addr":"172.21.15.16:6823/3360352974","state":["exists","up"]},{"osd":4,"uuid":"88260730-1482-4fbd-a40f-cb1124e9df4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6801","nonce":1856844856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6803","nonce":1856844856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6807","nonce":1856844856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6805","nonce":1856844856}]},"public_addr":"172.21.15.89:6801/1856844856","cluster_addr":"172.21.15.89:6803/1856844856","heartbeat_back_addr":"172.21.15.89:6807/1856844856","heartbeat_front_addr":"172.21.15.89:6805/1856844856","state":["exists","up"]},{"osd":5,"uuid":"4cb2c00c-fae0-4dd8-9d27-ec4f0093ccf7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6814","nonce":50395527}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6817","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6820","nonce":50395527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6831","nonce":50395527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6823","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6826","nonce":50395527}]},"public_addr":"172.21.15.89:6814/50395527","cluster_addr":"172.21.15.89:6820/50395527","heartbeat_back_addr":"172.21.15.89:6831/50395527","heartbeat_front_addr":"172.21.15.89:6826/50395527","state":["exists","up"]},{"osd":6,"uuid":"81ce0c19-2554-4cae-acbe-66a05e9e9b9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6809","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6812","nonce":3470471752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6815","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6818","nonce":3470471752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6827","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6829","nonce":3470471752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6821","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6824","nonce":3470471752}]},"public_addr":"172.21.15.89:6812/3470471752","cluster_addr":"172.21.15.89:6818/3470471752","heartbeat_back_addr":"172.21.15.89:6829/3470471752","heartbeat_front_addr":"172.21.15.89:6824/3470471752","state":["exists","up"]},{"osd":7,"uuid":"85e4313b-d37d-442e-8042-153bf20a847d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6811","nonce":1796778630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6813","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6816","nonce":1796778630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6825","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6828","nonce":1796778630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6819","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6822","nonce":1796778630}]},"public_addr":"172.21.15.89:6811/1796778630","cluster_addr":"172.21.15.89:6816/1796778630","heartbeat_back_addr":"172.21.15.89:6828/1796778630","heartbeat_front_addr":"172.21.15.89:6822/1796778630","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.439088+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.529970+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.680512+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:21.044161+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.582111+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.536558+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.409522+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.590229+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"17.1","mappings":[{"from":0,"to":5}]},{"pgid":"17.21","mappings":[{"from":0,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:6833/4142860904":"2024-04-01T15:14:53.117187+0000","172.21.15.16:6835/2564952016":"2024-04-01T15:14:39.715615+0000","172.21.15.16:6835/4142860904":"2024-04-01T15:14:53.117187+0000","172.21.15.16:6833/2564952016":"2024-04-01T15:14:39.715615+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-31T15:15:07.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:07.269+0000 7f4a66a00640 1 -- 172.21.15.16:0/247125841 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f4a40068380 msgr2=0x7f4a4006a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:07.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:07.269+0000 7f4a66a00640 1 --2- 172.21.15.16:0/247125841 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f4a40068380 0x7f4a4006a840 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7f4a600b52d0 tx=0x7f4a4c008040 comp rx=0 tx=0).stop 2024-03-31T15:15:07.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:07.269+0000 7f4a66a00640 1 -- 172.21.15.16:0/247125841 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4a600728b0 msgr2=0x7f4a60166160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:07.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:07.269+0000 7f4a66a00640 1 --2- 172.21.15.16:0/247125841 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4a600728b0 0x7f4a60166160 secure :-1 s=READY pgs=1866 cs=0 l=1 rev1=1 crypto rx=0x7f4a5001abd0 tx=0x7f4a5001ad60 comp rx=0 tx=0).stop 2024-03-31T15:15:07.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:07.269+0000 7f4a66a00640 1 -- 172.21.15.16:0/247125841 shutdown_connections 2024-03-31T15:15:07.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:07.269+0000 7f4a66a00640 1 --2- 172.21.15.16:0/247125841 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f4a40068380 0x7f4a4006a840 unknown :-1 s=CLOSED pgs=784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:07.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:07.273+0000 7f4a66a00640 1 --2- 172.21.15.16:0/247125841 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4a60145d10 0x7f4a6013baa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:07.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:07.273+0000 7f4a66a00640 1 --2- 172.21.15.16:0/247125841 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f4a600b2200 0x7f4a60140e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:07.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:07.273+0000 7f4a66a00640 1 --2- 172.21.15.16:0/247125841 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4a600728b0 0x7f4a60166160 unknown :-1 s=CLOSED pgs=1866 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:07.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:07.273+0000 7f4a66a00640 1 -- 172.21.15.16:0/247125841 >> 172.21.15.16:0/247125841 conn(0x7f4a6006b620 msgr2=0x7f4a6015e3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:15:07.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:07.273+0000 7f4a66a00640 1 -- 172.21.15.16:0/247125841 shutdown_connections 2024-03-31T15:15:07.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:07.273+0000 7f4a66a00640 1 -- 172.21.15.16:0/247125841 wait complete. 2024-03-31T15:15:07.281 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-31T15:15:07.281 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-31T15:15:07.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:07.433+0000 7f4f36600640 1 Processor -- start 2024-03-31T15:15:07.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:07.433+0000 7f4f36600640 1 -- start start 2024-03-31T15:15:07.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:07.433+0000 7f4f36600640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f4f30145e40 0x7f4f300cc740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:07.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:07.433+0000 7f4f36600640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4f300d0920 con 0x7f4f300ccc80 2024-03-31T15:15:07.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:07.433+0000 7f4f36600640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4f300d0c00 con 0x7f4f30070030 2024-03-31T15:15:07.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:07.433+0000 7f4f36600640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f4f300d06d0 con 0x7f4f30145e40 2024-03-31T15:15:07.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:07.433+0000 7f4f2f400640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f4f30145e40 0x7f4f300cc740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:07.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:07.433+0000 7f4f2ea00640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f4f300ccc80 0x7f4f3006f920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:46620/0 (socket says 172.21.15.16:46620) 2024-03-31T15:15:07.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:07.433+0000 7f4f2ea00640 1 -- 172.21.15.16:0/846064926 learned_addr learned my addr 172.21.15.16:0/846064926 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:15:07.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:07.433+0000 7f4f2f400640 1 -- 172.21.15.16:0/846064926 >> v1:172.21.15.89:6789/0 conn(0x7f4f30070030 legacy=0x7f4f300d2010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:07.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:07.433+0000 7f4f2f400640 1 -- 172.21.15.16:0/846064926 >> v1:172.21.15.16:6789/0 conn(0x7f4f300ccc80 legacy=0x7f4f3006f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:07.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:07.433+0000 7f4f2f400640 1 -- 172.21.15.16:0/846064926 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f300d1810 con 0x7f4f30145e40 2024-03-31T15:15:07.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:07.433+0000 7f4f2e000640 1 -- 172.21.15.16:0/846064926 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1288946691 0 0) 0x7f4f300d0920 con 0x7f4f300ccc80 2024-03-31T15:15:07.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:07.437+0000 7f4f2f400640 1 --2- 172.21.15.16:0/846064926 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4f30145e40 0x7f4f300cc740 secure :-1 s=READY pgs=1868 cs=0 l=1 rev1=1 crypto rx=0x7f4f200097c0 tx=0x7f4f2001c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b1d4d91f52bd852 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:15:07.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:07.437+0000 7f4f2e000640 1 -- 172.21.15.16:0/846064926 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f200b1070 con 0x7f4f30145e40 2024-03-31T15:15:07.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:07.437+0000 7f4f2e000640 1 -- 172.21.15.16:0/846064926 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4f200ac410 con 0x7f4f30145e40 2024-03-31T15:15:07.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:07.437+0000 7f4f2e000640 1 -- 172.21.15.16:0/846064926 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f200ac7e0 con 0x7f4f30145e40 2024-03-31T15:15:07.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:07.437+0000 7f4f36600640 1 -- 172.21.15.16:0/846064926 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4f30145e40 msgr2=0x7f4f300cc740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:07.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:07.437+0000 7f4f36600640 1 --2- 172.21.15.16:0/846064926 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4f30145e40 0x7f4f300cc740 secure :-1 s=READY pgs=1868 cs=0 l=1 rev1=1 crypto rx=0x7f4f200097c0 tx=0x7f4f2001c4a0 comp rx=0 tx=0).stop 2024-03-31T15:15:07.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:07.437+0000 7f4f36600640 1 -- 172.21.15.16:0/846064926 shutdown_connections 2024-03-31T15:15:07.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:07.437+0000 7f4f36600640 1 --2- 172.21.15.16:0/846064926 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4f30145e40 0x7f4f300cc740 unknown :-1 s=CLOSED pgs=1868 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:07.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:07.437+0000 7f4f36600640 1 -- 172.21.15.16:0/846064926 >> 172.21.15.16:0/846064926 conn(0x7f4f30067330 msgr2=0x7f4f30067740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:15:07.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:07.437+0000 7f4f36600640 1 -- 172.21.15.16:0/846064926 shutdown_connections 2024-03-31T15:15:07.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:07.437+0000 7f4f36600640 1 -- 172.21.15.16:0/846064926 wait complete. 2024-03-31T15:15:07.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:07.441+0000 7f4f36600640 1 Processor -- start 2024-03-31T15:15:07.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:07.441+0000 7f4f36600640 1 -- start start 2024-03-31T15:15:07.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:07.441+0000 7f4f36600640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f4f30070030 0x7f4f3006cac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:07.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:07.441+0000 7f4f36600640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4f300ccc80 0x7f4f3006e0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:07.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:07.441+0000 7f4f36600640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4f30145e40 0x7f4f30142280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:07.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:07.441+0000 7f4f36600640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f4f300b5240 con 0x7f4f30145e40 2024-03-31T15:15:07.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:07.441+0000 7f4f36600640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f4f300b50c0 con 0x7f4f300ccc80 2024-03-31T15:15:07.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:07.441+0000 7f4f36600640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f4f3016a1a0 con 0x7f4f30070030 2024-03-31T15:15:07.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:07.441+0000 7f4f2fe00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4f30145e40 0x7f4f30142280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:07.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:07.441+0000 7f4f2f400640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f4f30070030 0x7f4f3006cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:07.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:07.441+0000 7f4f2f400640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f4f30070030 0x7f4f3006cac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3301/0 says I am v2:172.21.15.16:56028/0 (socket says 172.21.15.16:56028) 2024-03-31T15:15:07.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:07.441+0000 7f4f2fe00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4f30145e40 0x7f4f30142280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46074/0 (socket says 172.21.15.16:46074) 2024-03-31T15:15:07.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:07.441+0000 7f4f2f400640 1 -- 172.21.15.16:0/970863887 learned_addr learned my addr 172.21.15.16:0/970863887 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:15:07.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:07.441+0000 7f4f2ea00640 1 --2- 172.21.15.16:0/970863887 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4f300ccc80 0x7f4f3006e0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:07.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:07.441+0000 7f4f2fe00640 1 -- 172.21.15.16:0/970863887 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f4f30070030 msgr2=0x7f4f3006cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:07.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:07.441+0000 7f4f2fe00640 1 --2- 172.21.15.16:0/970863887 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f4f30070030 0x7f4f3006cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:07.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:07.441+0000 7f4f2fe00640 1 -- 172.21.15.16:0/970863887 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4f300ccc80 msgr2=0x7f4f3006e0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:07.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:07.441+0000 7f4f2fe00640 1 --2- 172.21.15.16:0/970863887 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4f300ccc80 0x7f4f3006e0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:07.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:07.441+0000 7f4f2fe00640 1 -- 172.21.15.16:0/970863887 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f3006d000 con 0x7f4f30145e40 2024-03-31T15:15:07.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:07.441+0000 7f4f2f400640 1 --2- 172.21.15.16:0/970863887 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f4f30070030 0x7f4f3006cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:15:07.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:07.441+0000 7f4f2fe00640 1 --2- 172.21.15.16:0/970863887 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4f30145e40 0x7f4f30142280 secure :-1 s=READY pgs=1869 cs=0 l=1 rev1=1 crypto rx=0x7f4f300735d0 tx=0x7f4f2800cb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:15:07.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:07.441+0000 7f4f27400640 1 -- 172.21.15.16:0/970863887 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f28003b80 con 0x7f4f30145e40 2024-03-31T15:15:07.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:07.441+0000 7f4f27400640 1 -- 172.21.15.16:0/970863887 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4f28003d20 con 0x7f4f30145e40 2024-03-31T15:15:07.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:07.441+0000 7f4f27400640 1 -- 172.21.15.16:0/970863887 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f28005310 con 0x7f4f30145e40 2024-03-31T15:15:07.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:07.441+0000 7f4f36600640 1 -- 172.21.15.16:0/970863887 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4f3006d300 con 0x7f4f30145e40 2024-03-31T15:15:07.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:07.441+0000 7f4f36600640 1 -- 172.21.15.16:0/970863887 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4f3006d5c0 con 0x7f4f30145e40 2024-03-31T15:15:07.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:07.445+0000 7f4f36600640 1 -- 172.21.15.16:0/970863887 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4f30069950 con 0x7f4f30145e40 2024-03-31T15:15:07.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:07.445+0000 7f4f27400640 1 -- 172.21.15.16:0/970863887 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f4f280056c0 con 0x7f4f30145e40 2024-03-31T15:15:07.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:07.445+0000 7f4f27400640 1 --2- 172.21.15.16:0/970863887 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f4f100683b0 0x7f4f1006a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:07.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:07.445+0000 7f4f27400640 1 -- 172.21.15.16:0/970863887 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 5939+0+0 (secure 0 0 0) 0x7f4f280869d0 con 0x7f4f30145e40 2024-03-31T15:15:07.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:07.445+0000 7f4f2f400640 1 --2- 172.21.15.16:0/970863887 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f4f100683b0 0x7f4f1006a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:07.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:07.445+0000 7f4f2f400640 1 --2- 172.21.15.16:0/970863887 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f4f100683b0 0x7f4f1006a870 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f4f2001c4e0 tx=0x7f4f20008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:15:07.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:07.449+0000 7f4f27400640 1 -- 172.21.15.16:0/970863887 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f4f280567b0 con 0x7f4f30145e40 2024-03-31T15:15:07.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:07.649+0000 7f4f36600640 1 -- 172.21.15.16:0/970863887 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4f300cf3c0 con 0x7f4f30145e40 2024-03-31T15:15:07.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:07.649+0000 7f4f27400640 1 -- 172.21.15.16:0/970863887 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 112 v112) ==== 95+0+5185 (secure 0 0 0) 0x7f4f2805aab0 con 0x7f4f30145e40 2024-03-31T15:15:07.639 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:15:07.639 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":112,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7631,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6836/1628897112","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":1628897112},{"type":"v1","addr":"172.21.15.89:6836","nonce":1628897112}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":94},{"gid":8746,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/1036724837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6835","nonce":1036724837},{"type":"v1","addr":"172.21.15.89:6837","nonce":1036724837}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":94},{"gid":9192,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6834/472139147","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":472139147},{"type":"v1","addr":"172.21.15.16:6834","nonce":472139147}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":94}],"filesystems":[{"mdsmap":{"epoch":112,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-31T15:14:08.652180+0000","modified":"2024-03-31T15:14:58.141614+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":113,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":9762},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9762":{"gid":9762,"name":"c","rank":0,"incarnation":109,"state":"up:active","state_seq":4,"addr":"172.21.15.16:6835/2152452999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6833","nonce":2152452999},{"type":"v1","addr":"172.21.15.16:6835","nonce":2152452999}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9762,"qdb_cluster":[9762]},"id":8}]} 2024-03-31T15:15:07.639 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 112 2024-03-31T15:15:07.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:07.653+0000 7f4f36600640 1 -- 172.21.15.16:0/970863887 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f4f100683b0 msgr2=0x7f4f1006a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:07.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:07.653+0000 7f4f36600640 1 --2- 172.21.15.16:0/970863887 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f4f100683b0 0x7f4f1006a870 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f4f2001c4e0 tx=0x7f4f20008040 comp rx=0 tx=0).stop 2024-03-31T15:15:07.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:07.653+0000 7f4f36600640 1 -- 172.21.15.16:0/970863887 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4f30145e40 msgr2=0x7f4f30142280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:07.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:07.653+0000 7f4f36600640 1 --2- 172.21.15.16:0/970863887 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4f30145e40 0x7f4f30142280 secure :-1 s=READY pgs=1869 cs=0 l=1 rev1=1 crypto rx=0x7f4f300735d0 tx=0x7f4f2800cb90 comp rx=0 tx=0).stop 2024-03-31T15:15:07.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:07.653+0000 7f4f36600640 1 -- 172.21.15.16:0/970863887 shutdown_connections 2024-03-31T15:15:07.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:07.653+0000 7f4f36600640 1 --2- 172.21.15.16:0/970863887 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f4f100683b0 0x7f4f1006a870 unknown :-1 s=CLOSED pgs=785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:07.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:07.653+0000 7f4f36600640 1 --2- 172.21.15.16:0/970863887 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4f30145e40 0x7f4f30142280 unknown :-1 s=CLOSED pgs=1869 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:07.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:07.653+0000 7f4f36600640 1 --2- 172.21.15.16:0/970863887 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4f300ccc80 0x7f4f3006e0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:07.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:07.653+0000 7f4f36600640 1 --2- 172.21.15.16:0/970863887 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f4f30070030 0x7f4f3006cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:07.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:07.653+0000 7f4f36600640 1 -- 172.21.15.16:0/970863887 >> 172.21.15.16:0/970863887 conn(0x7f4f30067330 msgr2=0x7f4f30067710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:15:07.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:07.653+0000 7f4f36600640 1 -- 172.21.15.16:0/970863887 shutdown_connections 2024-03-31T15:15:07.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:07.653+0000 7f4f36600640 1 -- 172.21.15.16:0/970863887 wait complete. 2024-03-31T15:15:07.661 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-31T15:15:07.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:07.813+0000 7fc750c00640 1 Processor -- start 2024-03-31T15:15:07.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:07.813+0000 7fc750c00640 1 -- start start 2024-03-31T15:15:07.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:07.813+0000 7fc750c00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fc74c147dd0 0x7fc74c06c730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:07.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:07.813+0000 7fc750c00640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc74c147660 con 0x7fc74c070030 2024-03-31T15:15:07.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:07.813+0000 7fc750c00640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc74c162340 con 0x7fc74c147280 2024-03-31T15:15:07.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:07.813+0000 7fc750c00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fc74c1620f0 con 0x7fc74c147dd0 2024-03-31T15:15:07.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:07.813+0000 7fc743e00640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fc74c070030 0x7fc74c0cc8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:46622/0 (socket says 172.21.15.16:46622) 2024-03-31T15:15:07.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:07.813+0000 7fc743e00640 1 -- 172.21.15.16:0/3223048669 learned_addr learned my addr 172.21.15.16:0/3223048669 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:15:07.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:07.813+0000 7fc74a800640 1 --2- 172.21.15.16:0/3223048669 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fc74c147dd0 0x7fc74c06c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:07.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:07.813+0000 7fc749400640 1 -- 172.21.15.16:0/3223048669 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2646585818 0 0) 0x7fc74c147660 con 0x7fc74c070030 2024-03-31T15:15:07.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:07.813+0000 7fc749400640 1 -- 172.21.15.16:0/3223048669 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc730003530 con 0x7fc74c070030 2024-03-31T15:15:07.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:07.813+0000 7fc749400640 1 -- 172.21.15.16:0/3223048669 <== mon.1 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3456104201 0 0) 0x7fc74c162340 con 0x7fc74c147280 2024-03-31T15:15:07.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:07.813+0000 7fc749400640 1 -- 172.21.15.16:0/3223048669 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc74c147660 con 0x7fc74c147280 2024-03-31T15:15:07.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:07.813+0000 7fc749400640 1 -- 172.21.15.16:0/3223048669 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2842168738 0 0) 0x7fc730003530 con 0x7fc74c070030 2024-03-31T15:15:07.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:07.813+0000 7fc749400640 1 -- 172.21.15.16:0/3223048669 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fc74c147dd0 msgr2=0x7fc74c06c730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:15:07.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:07.813+0000 7fc749400640 1 --2- 172.21.15.16:0/3223048669 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fc74c147dd0 0x7fc74c06c730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:07.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:07.813+0000 7fc749400640 1 -- 172.21.15.16:0/3223048669 >> v1:172.21.15.89:6789/0 conn(0x7fc74c147280 legacy=0x7fc74c06f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:07.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:07.813+0000 7fc749400640 1 -- 172.21.15.16:0/3223048669 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc74c162620 con 0x7fc74c070030 2024-03-31T15:15:07.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:07.813+0000 7fc749400640 1 -- 172.21.15.16:0/3223048669 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 214807623 0 0) 0x7fc734001680 con 0x7fc74c070030 2024-03-31T15:15:07.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:07.813+0000 7fc74a800640 1 --2- 172.21.15.16:0/3223048669 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fc74c147dd0 0x7fc74c06c730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:15:07.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:07.813+0000 7fc749400640 1 -- 172.21.15.16:0/3223048669 >> v1:172.21.15.16:6789/0 conn(0x7fc74c070030 legacy=0x7fc74c0cc8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:07.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:07.817+0000 7fc749400640 1 --2- 172.21.15.16:0/3223048669 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc730003b80 0x7fc730003f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:07.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:07.817+0000 7fc749400640 1 --2- 172.21.15.16:0/3223048669 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc730004810 0x7fc730004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:07.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:07.817+0000 7fc749400640 1 --2- 172.21.15.16:0/3223048669 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fc74c147dd0 0x7fc7300051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:07.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:07.817+0000 7fc749400640 1 -- 172.21.15.16:0/3223048669 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fc74c1620f0 con 0x7fc730004810 2024-03-31T15:15:07.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:07.817+0000 7fc749400640 1 -- 172.21.15.16:0/3223048669 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fc73000ba00 con 0x7fc730003b80 2024-03-31T15:15:07.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:07.817+0000 7fc749400640 1 -- 172.21.15.16:0/3223048669 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fc73000bc00 con 0x7fc74c147dd0 2024-03-31T15:15:07.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:07.817+0000 7fc743e00640 1 --2- 172.21.15.16:0/3223048669 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc730004810 0x7fc730004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:07.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:07.817+0000 7fc749e00640 1 --2- 172.21.15.16:0/3223048669 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc730003b80 0x7fc730003f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:07.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:07.817+0000 7fc74a800640 1 --2- 172.21.15.16:0/3223048669 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fc74c147dd0 0x7fc7300051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:07.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:07.817+0000 7fc743e00640 1 -- 172.21.15.16:0/3223048669 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fc74c147dd0 msgr2=0x7fc7300051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:15:07.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:07.817+0000 7fc743e00640 1 --2- 172.21.15.16:0/3223048669 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fc74c147dd0 0x7fc7300051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:07.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:07.817+0000 7fc743e00640 1 -- 172.21.15.16:0/3223048669 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc730003b80 msgr2=0x7fc730003f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:15:07.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:07.817+0000 7fc743e00640 1 --2- 172.21.15.16:0/3223048669 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc730003b80 0x7fc730003f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:07.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:07.817+0000 7fc743e00640 1 -- 172.21.15.16:0/3223048669 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc730003530 con 0x7fc730004810 2024-03-31T15:15:07.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:07.817+0000 7fc74a800640 1 --2- 172.21.15.16:0/3223048669 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fc74c147dd0 0x7fc7300051d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:15:07.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:07.817+0000 7fc749e00640 1 --2- 172.21.15.16:0/3223048669 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc730003b80 0x7fc730003f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:15:07.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:07.817+0000 7fc743e00640 1 --2- 172.21.15.16:0/3223048669 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc730004810 0x7fc730004c90 secure :-1 s=READY pgs=1871 cs=0 l=1 rev1=1 crypto rx=0x7fc73400c190 tx=0x7fc73401cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb48d92cca344fd7 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:15:07.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:07.817+0000 7fc749400640 1 -- 172.21.15.16:0/3223048669 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc734007cf0 con 0x7fc730004810 2024-03-31T15:15:07.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:07.817+0000 7fc749400640 1 -- 172.21.15.16:0/3223048669 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc7340a9020 con 0x7fc730004810 2024-03-31T15:15:07.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:07.817+0000 7fc749400640 1 -- 172.21.15.16:0/3223048669 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc734005b00 con 0x7fc730004810 2024-03-31T15:15:07.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:07.821+0000 7fc750c00640 1 -- 172.21.15.16:0/3223048669 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc730004810 msgr2=0x7fc730004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:07.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:07.821+0000 7fc750c00640 1 --2- 172.21.15.16:0/3223048669 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc730004810 0x7fc730004c90 secure :-1 s=READY pgs=1871 cs=0 l=1 rev1=1 crypto rx=0x7fc73400c190 tx=0x7fc73401cbc0 comp rx=0 tx=0).stop 2024-03-31T15:15:07.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:07.821+0000 7fc74a800640 1 -- 172.21.15.16:0/3223048669 reap_dead start 2024-03-31T15:15:07.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:07.821+0000 7fc750c00640 1 -- 172.21.15.16:0/3223048669 shutdown_connections 2024-03-31T15:15:07.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:07.821+0000 7fc750c00640 1 -- 172.21.15.16:0/3223048669 >> 172.21.15.16:0/3223048669 conn(0x7fc74c067330 msgr2=0x7fc74c067740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:15:07.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:07.821+0000 7fc750c00640 1 -- 172.21.15.16:0/3223048669 shutdown_connections 2024-03-31T15:15:07.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:07.821+0000 7fc750c00640 1 -- 172.21.15.16:0/3223048669 wait complete. 2024-03-31T15:15:07.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:07.821+0000 7fc750c00640 1 Processor -- start 2024-03-31T15:15:07.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:07.821+0000 7fc750c00640 1 -- start start 2024-03-31T15:15:07.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:07.821+0000 7fc750c00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc730004810 0x7fc74c137840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:07.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:07.821+0000 7fc750c00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc74c1436d0 0x7fc74c143b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:07.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:07.821+0000 7fc750c00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fc74c0a95a0 0x7fc74c0a9a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:07.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:07.821+0000 7fc750c00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fc74c06c730 con 0x7fc74c1436d0 2024-03-31T15:15:07.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:07.821+0000 7fc750c00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fc74c162be0 con 0x7fc730004810 2024-03-31T15:15:07.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:07.821+0000 7fc750c00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fc74c06c8b0 con 0x7fc74c0a95a0 2024-03-31T15:15:07.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:07.821+0000 7fc743e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc74c1436d0 0x7fc74c143b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:07.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:07.821+0000 7fc743e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc74c1436d0 0x7fc74c143b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46090/0 (socket says 172.21.15.16:46090) 2024-03-31T15:15:07.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:07.821+0000 7fc743e00640 1 -- 172.21.15.16:0/931932272 learned_addr learned my addr 172.21.15.16:0/931932272 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:15:07.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:07.821+0000 7fc749e00640 1 --2- 172.21.15.16:0/931932272 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc730004810 0x7fc74c137840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:07.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:07.821+0000 7fc74a800640 1 --2- 172.21.15.16:0/931932272 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fc74c0a95a0 0x7fc74c0a9a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:07.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:07.821+0000 7fc743e00640 1 -- 172.21.15.16:0/931932272 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fc74c0a95a0 msgr2=0x7fc74c0a9a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:07.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:07.821+0000 7fc743e00640 1 --2- 172.21.15.16:0/931932272 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fc74c0a95a0 0x7fc74c0a9a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:07.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:07.821+0000 7fc743e00640 1 -- 172.21.15.16:0/931932272 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc730004810 msgr2=0x7fc74c137840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:07.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:07.821+0000 7fc743e00640 1 --2- 172.21.15.16:0/931932272 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc730004810 0x7fc74c137840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:07.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:07.821+0000 7fc743e00640 1 -- 172.21.15.16:0/931932272 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc74c0aa2f0 con 0x7fc74c1436d0 2024-03-31T15:15:07.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:07.821+0000 7fc74a800640 1 --2- 172.21.15.16:0/931932272 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fc74c0a95a0 0x7fc74c0a9a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:15:07.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:07.825+0000 7fc743e00640 1 --2- 172.21.15.16:0/931932272 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc74c1436d0 0x7fc74c143b30 secure :-1 s=READY pgs=1872 cs=0 l=1 rev1=1 crypto rx=0x7fc7340ab040 tx=0x7fc730005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:15:07.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:07.825+0000 7fc742000640 1 -- 172.21.15.16:0/931932272 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc734007cf0 con 0x7fc74c1436d0 2024-03-31T15:15:07.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:07.825+0000 7fc742000640 1 -- 172.21.15.16:0/931932272 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc7340049f0 con 0x7fc74c1436d0 2024-03-31T15:15:07.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:07.825+0000 7fc742000640 1 -- 172.21.15.16:0/931932272 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc734001c20 con 0x7fc74c1436d0 2024-03-31T15:15:07.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:07.825+0000 7fc750c00640 1 -- 172.21.15.16:0/931932272 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc74c0aad50 con 0x7fc74c1436d0 2024-03-31T15:15:07.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:07.825+0000 7fc750c00640 1 -- 172.21.15.16:0/931932272 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc74c0ab310 con 0x7fc74c1436d0 2024-03-31T15:15:07.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:07.825+0000 7fc750c00640 1 -- 172.21.15.16:0/931932272 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc74c144070 con 0x7fc74c1436d0 2024-03-31T15:15:07.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:07.825+0000 7fc742000640 1 -- 172.21.15.16:0/931932272 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fc734007110 con 0x7fc74c1436d0 2024-03-31T15:15:07.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:07.825+0000 7fc742000640 1 --2- 172.21.15.16:0/931932272 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fc728068380 0x7fc72806a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:07.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:07.825+0000 7fc742000640 1 -- 172.21.15.16:0/931932272 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 5939+0+0 (secure 0 0 0) 0x7fc734120b80 con 0x7fc74c1436d0 2024-03-31T15:15:07.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:07.829+0000 7fc749e00640 1 --2- 172.21.15.16:0/931932272 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fc728068380 0x7fc72806a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:07.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:07.829+0000 7fc749e00640 1 --2- 172.21.15.16:0/931932272 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fc728068380 0x7fc72806a840 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7fc73c007bf0 tx=0x7fc73c007120 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:15:07.820 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:07.829+0000 7fc742000640 1 -- 172.21.15.16:0/931932272 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fc7340168d0 con 0x7fc74c1436d0 2024-03-31T15:15:07.977 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-31T15:15:07.980 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-31T15:15:08.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:08.029+0000 7fc750c00640 1 -- 172.21.15.16:0/931932272 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc74c069950 con 0x7fc74c1436d0 2024-03-31T15:15:08.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:08.029+0000 7fc742000640 1 -- 172.21.15.16:0/931932272 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 112 v112) ==== 95+0+5185 (secure 0 0 0) 0x7fc7340f1040 con 0x7fc74c1436d0 2024-03-31T15:15:08.021 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:15:08.021 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":112,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7631,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6836/1628897112","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":1628897112},{"type":"v1","addr":"172.21.15.89:6836","nonce":1628897112}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":94},{"gid":8746,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/1036724837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6835","nonce":1036724837},{"type":"v1","addr":"172.21.15.89:6837","nonce":1036724837}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":94},{"gid":9192,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6834/472139147","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":472139147},{"type":"v1","addr":"172.21.15.16:6834","nonce":472139147}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":94}],"filesystems":[{"mdsmap":{"epoch":112,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-31T15:14:08.652180+0000","modified":"2024-03-31T15:14:58.141614+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":113,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":9762},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9762":{"gid":9762,"name":"c","rank":0,"incarnation":109,"state":"up:active","state_seq":4,"addr":"172.21.15.16:6835/2152452999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6833","nonce":2152452999},{"type":"v1","addr":"172.21.15.16:6835","nonce":2152452999}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9762,"qdb_cluster":[9762]},"id":8}]} 2024-03-31T15:15:08.021 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 112 2024-03-31T15:15:08.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:08.033+0000 7fc750c00640 1 -- 172.21.15.16:0/931932272 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fc728068380 msgr2=0x7fc72806a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:08.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:08.033+0000 7fc750c00640 1 --2- 172.21.15.16:0/931932272 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fc728068380 0x7fc72806a840 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7fc73c007bf0 tx=0x7fc73c007120 comp rx=0 tx=0).stop 2024-03-31T15:15:08.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:08.033+0000 7fc750c00640 1 -- 172.21.15.16:0/931932272 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc74c1436d0 msgr2=0x7fc74c143b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:08.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:08.033+0000 7fc750c00640 1 --2- 172.21.15.16:0/931932272 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc74c1436d0 0x7fc74c143b30 secure :-1 s=READY pgs=1872 cs=0 l=1 rev1=1 crypto rx=0x7fc7340ab040 tx=0x7fc730005840 comp rx=0 tx=0).stop 2024-03-31T15:15:08.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:08.033+0000 7fc750c00640 1 -- 172.21.15.16:0/931932272 shutdown_connections 2024-03-31T15:15:08.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:08.033+0000 7fc750c00640 1 --2- 172.21.15.16:0/931932272 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fc728068380 0x7fc72806a840 unknown :-1 s=CLOSED pgs=786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:08.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:08.033+0000 7fc750c00640 1 --2- 172.21.15.16:0/931932272 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fc74c0a95a0 0x7fc74c0a9a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:08.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:08.033+0000 7fc750c00640 1 --2- 172.21.15.16:0/931932272 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc74c1436d0 0x7fc74c143b30 unknown :-1 s=CLOSED pgs=1872 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:08.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:08.033+0000 7fc750c00640 1 --2- 172.21.15.16:0/931932272 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc730004810 0x7fc74c137840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:08.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:08.033+0000 7fc750c00640 1 -- 172.21.15.16:0/931932272 >> 172.21.15.16:0/931932272 conn(0x7fc74c067330 msgr2=0x7fc74c161340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:15:08.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:08.033+0000 7fc750c00640 1 -- 172.21.15.16:0/931932272 shutdown_connections 2024-03-31T15:15:08.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:08.033+0000 7fc750c00640 1 -- 172.21.15.16:0/931932272 wait complete. 2024-03-31T15:15:08.042 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-31T15:15:08.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:08.233+0000 7ff131600640 1 Processor -- start 2024-03-31T15:15:08.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:08.233+0000 7ff131600640 1 -- start start 2024-03-31T15:15:08.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:08.237+0000 7ff131600640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7ff12c071150 0x7ff12c09aa60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:08.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:08.237+0000 7ff131600640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff12c0d0c40 0x7ff12c0cca90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:08.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:08.237+0000 7ff131600640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff12c15c790 con 0x7ff12c15c3b0 2024-03-31T15:15:08.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:08.237+0000 7ff131600640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7ff12c09b170 con 0x7ff12c0d0c40 2024-03-31T15:15:08.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:08.237+0000 7ff131600640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7ff12c06f660 con 0x7ff12c071150 2024-03-31T15:15:08.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:08.237+0000 7ff130c00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff12c0d0c40 0x7ff12c0cca90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:08.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:08.237+0000 7ff130c00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff12c0d0c40 0x7ff12c0cca90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46104/0 (socket says 172.21.15.16:46104) 2024-03-31T15:15:08.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:08.237+0000 7ff130c00640 1 -- 172.21.15.16:0/3613831102 learned_addr learned my addr 172.21.15.16:0/3613831102 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:15:08.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:08.237+0000 7ff123400640 1 --2- 172.21.15.16:0/3613831102 >> v2:172.21.15.89:3300/0 conn(0x7ff12c071150 0x7ff12c09aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:08.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:08.237+0000 7ff130c00640 1 -- 172.21.15.16:0/3613831102 >> v2:172.21.15.89:3300/0 conn(0x7ff12c071150 msgr2=0x7ff12c09aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:15:08.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:08.237+0000 7ff130c00640 1 --2- 172.21.15.16:0/3613831102 >> v2:172.21.15.89:3300/0 conn(0x7ff12c071150 0x7ff12c09aa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:08.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:08.237+0000 7ff130c00640 1 -- 172.21.15.16:0/3613831102 >> v1:172.21.15.89:6789/0 conn(0x7ff12c15c3b0 legacy=0x7ff12c0d0530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:08.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:08.237+0000 7ff130c00640 1 -- 172.21.15.16:0/3613831102 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff12c06f940 con 0x7ff12c0d0c40 2024-03-31T15:15:08.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:08.241+0000 7ff130c00640 1 --2- 172.21.15.16:0/3613831102 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff12c0d0c40 0x7ff12c0cca90 secure :-1 s=READY pgs=1873 cs=0 l=1 rev1=1 crypto rx=0x7ff12c072f50 tx=0x7ff12401c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fab3b78c6ee3c768 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:15:08.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:08.241+0000 7ff12b400640 1 -- 172.21.15.16:0/3613831102 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1240b0070 con 0x7ff12c0d0c40 2024-03-31T15:15:08.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:08.241+0000 7ff12b400640 1 -- 172.21.15.16:0/3613831102 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff1240ab600 con 0x7ff12c0d0c40 2024-03-31T15:15:08.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:08.241+0000 7ff12b400640 1 -- 172.21.15.16:0/3613831102 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1240ab9d0 con 0x7ff12c0d0c40 2024-03-31T15:15:08.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:08.241+0000 7ff131600640 1 -- 172.21.15.16:0/3613831102 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff12c0d0c40 msgr2=0x7ff12c0cca90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:08.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:08.241+0000 7ff131600640 1 --2- 172.21.15.16:0/3613831102 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff12c0d0c40 0x7ff12c0cca90 secure :-1 s=READY pgs=1873 cs=0 l=1 rev1=1 crypto rx=0x7ff12c072f50 tx=0x7ff12401c520 comp rx=0 tx=0).stop 2024-03-31T15:15:08.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:08.241+0000 7ff131600640 1 -- 172.21.15.16:0/3613831102 shutdown_connections 2024-03-31T15:15:08.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:08.241+0000 7ff131600640 1 --2- 172.21.15.16:0/3613831102 >> v2:172.21.15.89:3300/0 conn(0x7ff12c071150 0x7ff12c09aa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:08.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:08.241+0000 7ff131600640 1 --2- 172.21.15.16:0/3613831102 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff12c0d0c40 0x7ff12c0cca90 unknown :-1 s=CLOSED pgs=1873 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:08.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:08.241+0000 7ff131600640 1 -- 172.21.15.16:0/3613831102 >> 172.21.15.16:0/3613831102 conn(0x7ff12c0b3a70 msgr2=0x7ff12c0b3e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:15:08.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:08.241+0000 7ff131600640 1 -- 172.21.15.16:0/3613831102 shutdown_connections 2024-03-31T15:15:08.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:08.241+0000 7ff131600640 1 -- 172.21.15.16:0/3613831102 wait complete. 2024-03-31T15:15:08.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:08.241+0000 7ff131600640 1 Processor -- start 2024-03-31T15:15:08.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:08.241+0000 7ff131600640 1 -- start start 2024-03-31T15:15:08.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:08.241+0000 7ff131600640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff12c071150 0x7ff12c159490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:08.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:08.241+0000 7ff131600640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff12c0d0c40 0x7ff12c1f4d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:08.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:08.241+0000 7ff131600640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff12c15c3b0 0x7ff12c154290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:08.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:08.241+0000 7ff131600640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7ff12c0cdb70 con 0x7ff12c0d0c40 2024-03-31T15:15:08.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:08.241+0000 7ff131600640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7ff12c0cd9f0 con 0x7ff12c071150 2024-03-31T15:15:08.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:08.241+0000 7ff131600640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7ff12c0cdcf0 con 0x7ff12c15c3b0 2024-03-31T15:15:08.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:08.245+0000 7ff123400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff12c0d0c40 0x7ff12c1f4d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:08.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:08.245+0000 7ff123400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff12c0d0c40 0x7ff12c1f4d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46108/0 (socket says 172.21.15.16:46108) 2024-03-31T15:15:08.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:08.245+0000 7ff123400640 1 -- 172.21.15.16:0/1302808485 learned_addr learned my addr 172.21.15.16:0/1302808485 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:15:08.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:08.245+0000 7ff12be00640 1 --2- 172.21.15.16:0/1302808485 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff12c071150 0x7ff12c159490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:08.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:08.245+0000 7ff130c00640 1 --2- 172.21.15.16:0/1302808485 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff12c15c3b0 0x7ff12c154290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:08.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:08.245+0000 7ff123400640 1 -- 172.21.15.16:0/1302808485 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff12c15c3b0 msgr2=0x7ff12c154290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:08.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:08.245+0000 7ff123400640 1 --2- 172.21.15.16:0/1302808485 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff12c15c3b0 0x7ff12c154290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:08.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:08.245+0000 7ff123400640 1 -- 172.21.15.16:0/1302808485 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff12c071150 msgr2=0x7ff12c159490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:08.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:08.245+0000 7ff123400640 1 --2- 172.21.15.16:0/1302808485 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff12c071150 0x7ff12c159490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:08.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:08.245+0000 7ff123400640 1 -- 172.21.15.16:0/1302808485 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff12c1547d0 con 0x7ff12c0d0c40 2024-03-31T15:15:08.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:08.245+0000 7ff123400640 1 --2- 172.21.15.16:0/1302808485 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff12c0d0c40 0x7ff12c1f4d30 secure :-1 s=READY pgs=1874 cs=0 l=1 rev1=1 crypto rx=0x7ff11400efc0 tx=0x7ff11400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:15:08.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:08.245+0000 7ff128c00640 1 -- 172.21.15.16:0/1302808485 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff114018020 con 0x7ff12c0d0c40 2024-03-31T15:15:08.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:08.245+0000 7ff128c00640 1 -- 172.21.15.16:0/1302808485 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff114014480 con 0x7ff12c0d0c40 2024-03-31T15:15:08.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:08.245+0000 7ff131600640 1 -- 172.21.15.16:0/1302808485 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff12c1f5380 con 0x7ff12c0d0c40 2024-03-31T15:15:08.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:08.245+0000 7ff128c00640 1 -- 172.21.15.16:0/1302808485 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff114014850 con 0x7ff12c0d0c40 2024-03-31T15:15:08.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:08.245+0000 7ff131600640 1 -- 172.21.15.16:0/1302808485 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff12c1f5940 con 0x7ff12c0d0c40 2024-03-31T15:15:08.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:08.245+0000 7ff131600640 1 -- 172.21.15.16:0/1302808485 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff12c04f8c0 con 0x7ff12c0d0c40 2024-03-31T15:15:08.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:08.245+0000 7ff128c00640 1 -- 172.21.15.16:0/1302808485 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7ff11401b020 con 0x7ff12c0d0c40 2024-03-31T15:15:08.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:08.249+0000 7ff128c00640 1 --2- 172.21.15.16:0/1302808485 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7ff108068380 0x7ff10806a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:08.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:08.249+0000 7ff128c00640 1 -- 172.21.15.16:0/1302808485 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 5939+0+0 (secure 0 0 0) 0x7ff114087aa0 con 0x7ff12c0d0c40 2024-03-31T15:15:08.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:08.249+0000 7ff12be00640 1 --2- 172.21.15.16:0/1302808485 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7ff108068380 0x7ff10806a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:08.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:08.249+0000 7ff12be00640 1 --2- 172.21.15.16:0/1302808485 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7ff108068380 0x7ff10806a840 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7ff11c001810 tx=0x7ff11c009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:15:08.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:08.253+0000 7ff128c00640 1 -- 172.21.15.16:0/1302808485 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7ff114022070 con 0x7ff12c0d0c40 2024-03-31T15:15:08.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:08.537+0000 7ff131600640 1 -- 172.21.15.16:0/1302808485 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ff12c0d1d60 con 0x7ff12c0d0c40 2024-03-31T15:15:08.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:08.537+0000 7ff128c00640 1 -- 172.21.15.16:0/1302808485 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v113) ==== 74+0+16907 (secure 0 0 0) 0x7ff12c1547d0 con 0x7ff12c0d0c40 2024-03-31T15:15:08.528 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:15:08.528 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":113,"fsid":"286eb5da-fb47-4550-b290-d9519c13b9a5","created":"2024-03-31T15:03:11.438952+0000","modified":"2024-03-31T15:14:53.117209+0000","last_up_change":"2024-03-31T15:03:22.885535+0000","last_in_change":"2024-03-31T15:03:13.941667+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-31T15:03:22.550787+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-03-31T15:14:06.417304+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"108","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-03-31T15:14:07.622562+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"109","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"92a66450-ef19-40d8-ac3d-d760b23b0bda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6801","nonce":932712244}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6803","nonce":932712244}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6807","nonce":932712244}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6805","nonce":932712244}]},"public_addr":"172.21.15.16:6801/932712244","cluster_addr":"172.21.15.16:6803/932712244","heartbeat_back_addr":"172.21.15.16:6807/932712244","heartbeat_front_addr":"172.21.15.16:6805/932712244","state":["exists","up"]},{"osd":1,"uuid":"da9d1f2b-1968-4959-b843-faa231988e92","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6810","nonce":1398737533}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6811","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6813","nonce":1398737533}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6821","nonce":1398737533}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6815","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6816","nonce":1398737533}]},"public_addr":"172.21.15.16:6810/1398737533","cluster_addr":"172.21.15.16:6813/1398737533","heartbeat_back_addr":"172.21.15.16:6821/1398737533","heartbeat_front_addr":"172.21.15.16:6816/1398737533","state":["exists","up"]},{"osd":2,"uuid":"01a73662-9a21-4b1f-bcbb-42b9636a4638","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6819","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6822","nonce":3018526646}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6825","nonce":3018526646}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6831","nonce":3018526646}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6827","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6829","nonce":3018526646}]},"public_addr":"172.21.15.16:6822/3018526646","cluster_addr":"172.21.15.16:6825/3018526646","heartbeat_back_addr":"172.21.15.16:6831/3018526646","heartbeat_front_addr":"172.21.15.16:6829/3018526646","state":["exists","up"]},{"osd":3,"uuid":"c7771fd3-e18e-4500-ac38-23cf10ebcd75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6809","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6812","nonce":3360352974}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6817","nonce":3360352974}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6828","nonce":3360352974}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6823","nonce":3360352974}]},"public_addr":"172.21.15.16:6812/3360352974","cluster_addr":"172.21.15.16:6817/3360352974","heartbeat_back_addr":"172.21.15.16:6828/3360352974","heartbeat_front_addr":"172.21.15.16:6823/3360352974","state":["exists","up"]},{"osd":4,"uuid":"88260730-1482-4fbd-a40f-cb1124e9df4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6801","nonce":1856844856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6803","nonce":1856844856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6807","nonce":1856844856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6805","nonce":1856844856}]},"public_addr":"172.21.15.89:6801/1856844856","cluster_addr":"172.21.15.89:6803/1856844856","heartbeat_back_addr":"172.21.15.89:6807/1856844856","heartbeat_front_addr":"172.21.15.89:6805/1856844856","state":["exists","up"]},{"osd":5,"uuid":"4cb2c00c-fae0-4dd8-9d27-ec4f0093ccf7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6814","nonce":50395527}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6817","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6820","nonce":50395527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6831","nonce":50395527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6823","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6826","nonce":50395527}]},"public_addr":"172.21.15.89:6814/50395527","cluster_addr":"172.21.15.89:6820/50395527","heartbeat_back_addr":"172.21.15.89:6831/50395527","heartbeat_front_addr":"172.21.15.89:6826/50395527","state":["exists","up"]},{"osd":6,"uuid":"81ce0c19-2554-4cae-acbe-66a05e9e9b9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6809","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6812","nonce":3470471752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6815","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6818","nonce":3470471752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6827","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6829","nonce":3470471752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6821","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6824","nonce":3470471752}]},"public_addr":"172.21.15.89:6812/3470471752","cluster_addr":"172.21.15.89:6818/3470471752","heartbeat_back_addr":"172.21.15.89:6829/3470471752","heartbeat_front_addr":"172.21.15.89:6824/3470471752","state":["exists","up"]},{"osd":7,"uuid":"85e4313b-d37d-442e-8042-153bf20a847d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6811","nonce":1796778630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6813","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6816","nonce":1796778630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6825","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6828","nonce":1796778630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6819","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6822","nonce":1796778630}]},"public_addr":"172.21.15.89:6811/1796778630","cluster_addr":"172.21.15.89:6816/1796778630","heartbeat_back_addr":"172.21.15.89:6828/1796778630","heartbeat_front_addr":"172.21.15.89:6822/1796778630","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.439088+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.529970+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.680512+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:21.044161+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.582111+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.536558+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.409522+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.590229+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"17.1","mappings":[{"from":0,"to":5}]},{"pgid":"17.21","mappings":[{"from":0,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:6833/4142860904":"2024-04-01T15:14:53.117187+0000","172.21.15.16:6835/2564952016":"2024-04-01T15:14:39.715615+0000","172.21.15.16:6835/4142860904":"2024-04-01T15:14:53.117187+0000","172.21.15.16:6833/2564952016":"2024-04-01T15:14:39.715615+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-31T15:15:08.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:08.541+0000 7ff131600640 1 -- 172.21.15.16:0/1302808485 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7ff108068380 msgr2=0x7ff10806a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:08.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:08.541+0000 7ff131600640 1 --2- 172.21.15.16:0/1302808485 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7ff108068380 0x7ff10806a840 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7ff11c001810 tx=0x7ff11c009040 comp rx=0 tx=0).stop 2024-03-31T15:15:08.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:08.541+0000 7ff131600640 1 -- 172.21.15.16:0/1302808485 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff12c0d0c40 msgr2=0x7ff12c1f4d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:08.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:08.541+0000 7ff131600640 1 --2- 172.21.15.16:0/1302808485 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff12c0d0c40 0x7ff12c1f4d30 secure :-1 s=READY pgs=1874 cs=0 l=1 rev1=1 crypto rx=0x7ff11400efc0 tx=0x7ff11400c490 comp rx=0 tx=0).stop 2024-03-31T15:15:08.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:08.541+0000 7ff131600640 1 -- 172.21.15.16:0/1302808485 shutdown_connections 2024-03-31T15:15:08.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:08.541+0000 7ff131600640 1 --2- 172.21.15.16:0/1302808485 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7ff108068380 0x7ff10806a840 unknown :-1 s=CLOSED pgs=787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:08.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:08.541+0000 7ff131600640 1 --2- 172.21.15.16:0/1302808485 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff12c15c3b0 0x7ff12c154290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:08.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:08.541+0000 7ff131600640 1 --2- 172.21.15.16:0/1302808485 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff12c0d0c40 0x7ff12c1f4d30 unknown :-1 s=CLOSED pgs=1874 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:08.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:08.541+0000 7ff131600640 1 --2- 172.21.15.16:0/1302808485 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff12c071150 0x7ff12c159490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:08.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:08.541+0000 7ff131600640 1 -- 172.21.15.16:0/1302808485 >> 172.21.15.16:0/1302808485 conn(0x7ff12c0b3a70 msgr2=0x7ff12c0af6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:15:08.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:08.541+0000 7ff131600640 1 -- 172.21.15.16:0/1302808485 shutdown_connections 2024-03-31T15:15:08.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:08.541+0000 7ff131600640 1 -- 172.21.15.16:0/1302808485 wait complete. 2024-03-31T15:15:08.551 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-31T15:15:08.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:08.701+0000 7f37cc400640 1 Processor -- start 2024-03-31T15:15:08.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:08.701+0000 7f37cc400640 1 -- start start 2024-03-31T15:15:08.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:08.705+0000 7f37cc400640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f37c4145e40 0x7f37c40cc740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:08.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:08.705+0000 7f37cc400640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f37c40ccc80 0x7f37c406f920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:08.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:08.705+0000 7f37cc400640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f37c40d0920 con 0x7f37c406fe60 2024-03-31T15:15:08.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:08.705+0000 7f37cc400640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f37c40d06d0 con 0x7f37c4145e40 2024-03-31T15:15:08.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:08.705+0000 7f37cc400640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f37c40d0c00 con 0x7f37c40ccc80 2024-03-31T15:15:08.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:08.705+0000 7f37c9a00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f37c4145e40 0x7f37c40cc740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:08.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:08.705+0000 7f37c9000640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f37c40ccc80 0x7f37c406f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:08.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:08.705+0000 7f37ca400640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f37c406fe60 0x7f37c415fdf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.16:57320/0 (socket says 172.21.15.16:57320) 2024-03-31T15:15:08.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:08.705+0000 7f37ca400640 1 -- 172.21.15.16:0/3933016211 learned_addr learned my addr 172.21.15.16:0/3933016211 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:15:08.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:08.705+0000 7f37c3e00640 1 -- 172.21.15.16:0/3933016211 <== mon.1 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2788250814 0 0) 0x7f37c40d0920 con 0x7f37c406fe60 2024-03-31T15:15:08.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:08.705+0000 7f37c3e00640 1 -- 172.21.15.16:0/3933016211 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f37ac003530 con 0x7f37c406fe60 2024-03-31T15:15:08.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:08.705+0000 7f37c9a00640 1 -- 172.21.15.16:0/3933016211 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f37c40ccc80 msgr2=0x7f37c406f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:15:08.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:08.705+0000 7f37c9a00640 1 --2- 172.21.15.16:0/3933016211 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f37c40ccc80 0x7f37c406f920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:08.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:08.705+0000 7f37c9a00640 1 -- 172.21.15.16:0/3933016211 >> v1:172.21.15.89:6789/0 conn(0x7f37c406fe60 legacy=0x7f37c415fdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:08.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:08.705+0000 7f37c9a00640 1 -- 172.21.15.16:0/3933016211 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37c40d1810 con 0x7f37c4145e40 2024-03-31T15:15:08.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:08.705+0000 7f37c9000640 1 --2- 172.21.15.16:0/3933016211 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f37c40ccc80 0x7f37c406f920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:15:08.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:08.709+0000 7f37c9a00640 1 --2- 172.21.15.16:0/3933016211 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f37c4145e40 0x7f37c40cc740 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f37b80097c0 tx=0x7f37b801c4a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7e2b2a1d8f095056 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:15:08.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:08.709+0000 7f37c3e00640 1 -- 172.21.15.16:0/3933016211 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37b80b0070 con 0x7f37c4145e40 2024-03-31T15:15:08.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:08.709+0000 7f37c3e00640 1 -- 172.21.15.16:0/3933016211 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f37b80a4e50 con 0x7f37c4145e40 2024-03-31T15:15:08.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:08.709+0000 7f37c3e00640 1 -- 172.21.15.16:0/3933016211 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37b80ab820 con 0x7f37c4145e40 2024-03-31T15:15:08.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:08.709+0000 7f37cc400640 1 -- 172.21.15.16:0/3933016211 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f37c4145e40 msgr2=0x7f37c40cc740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:08.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:08.709+0000 7f37cc400640 1 --2- 172.21.15.16:0/3933016211 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f37c4145e40 0x7f37c40cc740 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f37b80097c0 tx=0x7f37b801c4a0 comp rx=0 tx=0).stop 2024-03-31T15:15:08.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:08.709+0000 7f37cc400640 1 -- 172.21.15.16:0/3933016211 shutdown_connections 2024-03-31T15:15:08.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:08.709+0000 7f37cc400640 1 --2- 172.21.15.16:0/3933016211 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f37c40ccc80 0x7f37c406f920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:08.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:08.709+0000 7f37cc400640 1 --2- 172.21.15.16:0/3933016211 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f37c4145e40 0x7f37c40cc740 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:08.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:08.709+0000 7f37cc400640 1 -- 172.21.15.16:0/3933016211 >> 172.21.15.16:0/3933016211 conn(0x7f37c4067330 msgr2=0x7f37c4067740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:15:08.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:08.709+0000 7f37cc400640 1 -- 172.21.15.16:0/3933016211 shutdown_connections 2024-03-31T15:15:08.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:08.709+0000 7f37cc400640 1 -- 172.21.15.16:0/3933016211 wait complete. 2024-03-31T15:15:08.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:08.709+0000 7f37cc400640 1 Processor -- start 2024-03-31T15:15:08.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:08.709+0000 7f37cc400640 1 -- start start 2024-03-31T15:15:08.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:08.713+0000 7f37cc400640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f37c406fe60 0x7f37c415e310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:08.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:08.713+0000 7f37cc400640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f37c40ccc80 0x7f37c415e850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:08.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:08.713+0000 7f37cc400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f37c4145e40 0x7f37c41f6e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:08.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:08.713+0000 7f37cc400640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f37c40b5240 con 0x7f37c4145e40 2024-03-31T15:15:08.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:08.713+0000 7f37cc400640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f37c40b50c0 con 0x7f37c40ccc80 2024-03-31T15:15:08.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:08.713+0000 7f37cc400640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f37c4169190 con 0x7f37c406fe60 2024-03-31T15:15:08.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:08.713+0000 7f37ca400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f37c4145e40 0x7f37c41f6e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:08.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:08.713+0000 7f37ca400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f37c4145e40 0x7f37c41f6e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46120/0 (socket says 172.21.15.16:46120) 2024-03-31T15:15:08.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:08.713+0000 7f37ca400640 1 -- 172.21.15.16:0/2767080998 learned_addr learned my addr 172.21.15.16:0/2767080998 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:15:08.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:08.713+0000 7f37c9a00640 1 --2- 172.21.15.16:0/2767080998 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f37c406fe60 0x7f37c415e310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:08.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:08.713+0000 7f37c9000640 1 --2- 172.21.15.16:0/2767080998 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f37c40ccc80 0x7f37c415e850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:08.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:08.713+0000 7f37ca400640 1 -- 172.21.15.16:0/2767080998 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f37c406fe60 msgr2=0x7f37c415e310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:08.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:08.713+0000 7f37ca400640 1 --2- 172.21.15.16:0/2767080998 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f37c406fe60 0x7f37c415e310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:08.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:08.713+0000 7f37ca400640 1 -- 172.21.15.16:0/2767080998 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f37c40ccc80 msgr2=0x7f37c415e850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:08.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:08.713+0000 7f37ca400640 1 --2- 172.21.15.16:0/2767080998 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f37c40ccc80 0x7f37c415e850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:08.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:08.713+0000 7f37ca400640 1 -- 172.21.15.16:0/2767080998 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37c41f80f0 con 0x7f37c4145e40 2024-03-31T15:15:08.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:08.713+0000 7f37c9000640 1 --2- 172.21.15.16:0/2767080998 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f37c40ccc80 0x7f37c415e850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:15:08.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:08.713+0000 7f37ca400640 1 --2- 172.21.15.16:0/2767080998 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f37c4145e40 0x7f37c41f6e60 secure :-1 s=READY pgs=1875 cs=0 l=1 rev1=1 crypto rx=0x7f37bc00cb70 tx=0x7f37bc00a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:15:08.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:08.713+0000 7f37c1600640 1 -- 172.21.15.16:0/2767080998 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37bc017070 con 0x7f37c4145e40 2024-03-31T15:15:08.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:08.713+0000 7f37c1600640 1 -- 172.21.15.16:0/2767080998 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f37bc00ad40 con 0x7f37c4145e40 2024-03-31T15:15:08.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:08.713+0000 7f37c1600640 1 -- 172.21.15.16:0/2767080998 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37bc012620 con 0x7f37c4145e40 2024-03-31T15:15:08.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:08.713+0000 7f37cc400640 1 -- 172.21.15.16:0/2767080998 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f37c41f4ad0 con 0x7f37c4145e40 2024-03-31T15:15:08.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:08.713+0000 7f37cc400640 1 -- 172.21.15.16:0/2767080998 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f37c41f8640 con 0x7f37c4145e40 2024-03-31T15:15:08.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:08.713+0000 7f37cc400640 1 -- 172.21.15.16:0/2767080998 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f37c4069950 con 0x7f37c4145e40 2024-03-31T15:15:08.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:08.717+0000 7f37c1600640 1 -- 172.21.15.16:0/2767080998 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f37bc01a070 con 0x7f37c4145e40 2024-03-31T15:15:08.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:08.717+0000 7f37c1600640 1 --2- 172.21.15.16:0/2767080998 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f37a4068380 0x7f37a406a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:08.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:08.717+0000 7f37c1600640 1 -- 172.21.15.16:0/2767080998 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 5939+0+0 (secure 0 0 0) 0x7f37bc0868e0 con 0x7f37c4145e40 2024-03-31T15:15:08.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:08.717+0000 7f37c9a00640 1 --2- 172.21.15.16:0/2767080998 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f37a4068380 0x7f37a406a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:08.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:08.717+0000 7f37c9a00640 1 --2- 172.21.15.16:0/2767080998 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f37a4068380 0x7f37a406a840 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7f37b80ad000 tx=0x7f37b80a6000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:15:08.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:08.721+0000 7f37c1600640 1 -- 172.21.15.16:0/2767080998 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f37c41f4ad0 con 0x7f37c4145e40 2024-03-31T15:15:09.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:09.049+0000 7f37cc400640 1 -- 172.21.15.16:0/2767080998 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f37c40cf3c0 con 0x7f37c4145e40 2024-03-31T15:15:09.040 INFO:tasks.ceph.mon.a.smithi016.stderr:2024-03-31T15:15:09.049+0000 7f7dcbc00640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-31T15:15:09.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:09.053+0000 7f37c1600640 1 -- 172.21.15.16:0/2767080998 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v113) ==== 161+0+0 (secure 0 0 0) 0x7f37bc0566c0 con 0x7f37c4145e40 2024-03-31T15:15:09.042 INFO:teuthology.orchestra.run.smithi016.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-31T15:15:09.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:09.057+0000 7f37cc400640 1 -- 172.21.15.16:0/2767080998 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f37a4068380 msgr2=0x7f37a406a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:09.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:09.057+0000 7f37cc400640 1 --2- 172.21.15.16:0/2767080998 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f37a4068380 0x7f37a406a840 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7f37b80ad000 tx=0x7f37b80a6000 comp rx=0 tx=0).stop 2024-03-31T15:15:09.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:09.057+0000 7f37cc400640 1 -- 172.21.15.16:0/2767080998 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f37c4145e40 msgr2=0x7f37c41f6e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:09.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:09.057+0000 7f37cc400640 1 --2- 172.21.15.16:0/2767080998 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f37c4145e40 0x7f37c41f6e60 secure :-1 s=READY pgs=1875 cs=0 l=1 rev1=1 crypto rx=0x7f37bc00cb70 tx=0x7f37bc00a430 comp rx=0 tx=0).stop 2024-03-31T15:15:09.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:09.057+0000 7f37cc400640 1 -- 172.21.15.16:0/2767080998 shutdown_connections 2024-03-31T15:15:09.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:09.057+0000 7f37cc400640 1 --2- 172.21.15.16:0/2767080998 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f37a4068380 0x7f37a406a840 unknown :-1 s=CLOSED pgs=788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:09.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:09.057+0000 7f37cc400640 1 --2- 172.21.15.16:0/2767080998 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f37c4145e40 0x7f37c41f6e60 unknown :-1 s=CLOSED pgs=1875 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:09.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:09.057+0000 7f37cc400640 1 --2- 172.21.15.16:0/2767080998 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f37c40ccc80 0x7f37c415e850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:09.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:09.057+0000 7f37cc400640 1 --2- 172.21.15.16:0/2767080998 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f37c406fe60 0x7f37c415e310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:09.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:09.057+0000 7f37cc400640 1 -- 172.21.15.16:0/2767080998 >> 172.21.15.16:0/2767080998 conn(0x7f37c4067330 msgr2=0x7f37c40b5440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:15:09.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:09.057+0000 7f37cc400640 1 -- 172.21.15.16:0/2767080998 shutdown_connections 2024-03-31T15:15:09.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:09.057+0000 7f37cc400640 1 -- 172.21.15.16:0/2767080998 wait complete. 2024-03-31T15:15:09.050 INFO:tasks.ceph.mds.c.smithi016.stderr: -1487> 2024-03-31T15:14:59.121+0000 7f22f2000640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-31T15:15:09.067 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-31T15:15:09.095 INFO:tasks.ceph.mds.c.smithi016.stdout:starting mds.c at 2024-03-31T15:15:09.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:09.221+0000 7f6b2c600640 1 Processor -- start 2024-03-31T15:15:09.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:09.225+0000 7f6b2c600640 1 -- start start 2024-03-31T15:15:09.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:09.225+0000 7f6b2c600640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f6b241567e0 0x7f6b24072c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:09.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:09.225+0000 7f6b2c600640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f6b240731c0 0x7f6b2406d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:09.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:09.225+0000 7f6b2c600640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f6b2406da10 0x7f6b2415fdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:09.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:09.225+0000 7f6b2c600640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f6b2406de80 con 0x7f6b2406da10 2024-03-31T15:15:09.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:09.225+0000 7f6b2c600640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f6b2415c7e0 con 0x7f6b241567e0 2024-03-31T15:15:09.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:09.225+0000 7f6b2c600640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f6b240736b0 con 0x7f6b240731c0 2024-03-31T15:15:09.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:09.225+0000 7f6b2a600640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f6b2406da10 0x7f6b2415fdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:09.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:09.225+0000 7f6b2a600640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f6b2406da10 0x7f6b2415fdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46138/0 (socket says 172.21.15.16:46138) 2024-03-31T15:15:09.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:09.225+0000 7f6b2a600640 1 -- 172.21.15.16:0/1299831 learned_addr learned my addr 172.21.15.16:0/1299831 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:15:09.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:09.225+0000 7f6b29200640 1 --2- 172.21.15.16:0/1299831 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f6b240731c0 0x7f6b2406d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:09.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:09.225+0000 7f6b29c00640 1 --2- 172.21.15.16:0/1299831 >> v2:172.21.15.89:3300/0 conn(0x7f6b241567e0 0x7f6b24072c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:09.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:09.225+0000 7f6b2a600640 1 -- 172.21.15.16:0/1299831 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f6b240731c0 msgr2=0x7f6b2406d4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:15:09.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:09.225+0000 7f6b2a600640 1 --2- 172.21.15.16:0/1299831 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f6b240731c0 0x7f6b2406d4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:09.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:09.225+0000 7f6b2a600640 1 -- 172.21.15.16:0/1299831 >> v2:172.21.15.89:3300/0 conn(0x7f6b241567e0 msgr2=0x7f6b24072c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:15:09.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:09.225+0000 7f6b2a600640 1 --2- 172.21.15.16:0/1299831 >> v2:172.21.15.89:3300/0 conn(0x7f6b241567e0 0x7f6b24072c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:09.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:09.225+0000 7f6b2a600640 1 -- 172.21.15.16:0/1299831 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b2415caa0 con 0x7f6b2406da10 2024-03-31T15:15:09.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:09.229+0000 7f6b2a600640 1 --2- 172.21.15.16:0/1299831 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6b2406da10 0x7f6b2415fdd0 secure :-1 s=READY pgs=1878 cs=0 l=1 rev1=1 crypto rx=0x7f6b240cffe0 tx=0x7f6b1c01c730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b0258f4e23362c4 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:15:09.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:09.229+0000 7f6b23e00640 1 -- 172.21.15.16:0/1299831 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b1c0a64d0 con 0x7f6b2406da10 2024-03-31T15:15:09.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:09.229+0000 7f6b23e00640 1 -- 172.21.15.16:0/1299831 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6b1c0a6670 con 0x7f6b2406da10 2024-03-31T15:15:09.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:09.229+0000 7f6b23e00640 1 -- 172.21.15.16:0/1299831 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b1c0a6a40 con 0x7f6b2406da10 2024-03-31T15:15:09.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:09.229+0000 7f6b2c600640 1 -- 172.21.15.16:0/1299831 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6b2406da10 msgr2=0x7f6b2415fdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:09.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:09.229+0000 7f6b2c600640 1 --2- 172.21.15.16:0/1299831 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6b2406da10 0x7f6b2415fdd0 secure :-1 s=READY pgs=1878 cs=0 l=1 rev1=1 crypto rx=0x7f6b240cffe0 tx=0x7f6b1c01c730 comp rx=0 tx=0).stop 2024-03-31T15:15:09.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:09.229+0000 7f6b2c600640 1 -- 172.21.15.16:0/1299831 shutdown_connections 2024-03-31T15:15:09.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:09.229+0000 7f6b2c600640 1 --2- 172.21.15.16:0/1299831 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6b2406da10 0x7f6b2415fdd0 unknown :-1 s=CLOSED pgs=1878 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:09.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:09.229+0000 7f6b2c600640 1 --2- 172.21.15.16:0/1299831 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f6b240731c0 0x7f6b2406d4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:09.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:09.229+0000 7f6b2c600640 1 --2- 172.21.15.16:0/1299831 >> v2:172.21.15.89:3300/0 conn(0x7f6b241567e0 0x7f6b24072c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:09.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:09.229+0000 7f6b2c600640 1 -- 172.21.15.16:0/1299831 >> 172.21.15.16:0/1299831 conn(0x7f6b2406ee40 msgr2=0x7f6b2406b620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:15:09.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:09.229+0000 7f6b2c600640 1 -- 172.21.15.16:0/1299831 shutdown_connections 2024-03-31T15:15:09.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:09.229+0000 7f6b2c600640 1 -- 172.21.15.16:0/1299831 wait complete. 2024-03-31T15:15:09.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:09.229+0000 7f6b2c600640 1 Processor -- start 2024-03-31T15:15:09.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:09.229+0000 7f6b2c600640 1 -- start start 2024-03-31T15:15:09.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:09.233+0000 7f6b2c600640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f6b2406da10 0x7f6b24143310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:09.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:09.233+0000 7f6b2c600640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6b240731c0 0x7f6b2415b870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:09.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:09.233+0000 7f6b2c600640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6b241567e0 0x7f6b2415bdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:09.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:09.233+0000 7f6b2c600640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f6b2415d6e0 con 0x7f6b241567e0 2024-03-31T15:15:09.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:09.233+0000 7f6b2c600640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f6b2415d560 con 0x7f6b240731c0 2024-03-31T15:15:09.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:09.233+0000 7f6b2c600640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f6b24071550 con 0x7f6b2406da10 2024-03-31T15:15:09.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:09.233+0000 7f6b2a600640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6b241567e0 0x7f6b2415bdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:09.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:09.233+0000 7f6b29c00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f6b2406da10 0x7f6b24143310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:09.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:09.233+0000 7f6b29c00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f6b2406da10 0x7f6b24143310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3301/0 says I am v2:172.21.15.16:56098/0 (socket says 172.21.15.16:56098) 2024-03-31T15:15:09.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:09.233+0000 7f6b29c00640 1 -- 172.21.15.16:0/3051802321 learned_addr learned my addr 172.21.15.16:0/3051802321 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:15:09.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:09.233+0000 7f6b29200640 1 --2- 172.21.15.16:0/3051802321 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6b240731c0 0x7f6b2415b870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:09.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:09.233+0000 7f6b2a600640 1 -- 172.21.15.16:0/3051802321 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f6b2406da10 msgr2=0x7f6b24143310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:09.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:09.233+0000 7f6b2a600640 1 --2- 172.21.15.16:0/3051802321 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f6b2406da10 0x7f6b24143310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:09.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:09.233+0000 7f6b2a600640 1 -- 172.21.15.16:0/3051802321 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6b240731c0 msgr2=0x7f6b2415b870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:09.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:09.233+0000 7f6b2a600640 1 --2- 172.21.15.16:0/3051802321 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6b240731c0 0x7f6b2415b870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:09.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:09.233+0000 7f6b2a600640 1 -- 172.21.15.16:0/3051802321 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b2415b3c0 con 0x7f6b241567e0 2024-03-31T15:15:09.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:09.233+0000 7f6b29c00640 1 --2- 172.21.15.16:0/3051802321 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f6b2406da10 0x7f6b24143310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:15:09.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:09.233+0000 7f6b2a600640 1 --2- 172.21.15.16:0/3051802321 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6b241567e0 0x7f6b2415bdb0 secure :-1 s=READY pgs=1879 cs=0 l=1 rev1=1 crypto rx=0x7f6b2406f600 tx=0x7f6b1c0194d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:15:09.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:09.233+0000 7f6b21600640 1 -- 172.21.15.16:0/3051802321 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b1c0bf070 con 0x7f6b241567e0 2024-03-31T15:15:09.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:09.233+0000 7f6b21600640 1 -- 172.21.15.16:0/3051802321 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6b1c0091d0 con 0x7f6b241567e0 2024-03-31T15:15:09.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:09.233+0000 7f6b21600640 1 -- 172.21.15.16:0/3051802321 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b1c0a6a60 con 0x7f6b241567e0 2024-03-31T15:15:09.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:09.233+0000 7f6b2c600640 1 -- 172.21.15.16:0/3051802321 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6b24151820 con 0x7f6b241567e0 2024-03-31T15:15:09.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:09.233+0000 7f6b2c600640 1 -- 172.21.15.16:0/3051802321 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6b24151dc0 con 0x7f6b241567e0 2024-03-31T15:15:09.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:09.233+0000 7f6b2c600640 1 -- 172.21.15.16:0/3051802321 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6b24069950 con 0x7f6b241567e0 2024-03-31T15:15:09.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:09.233+0000 7f6b21600640 1 -- 172.21.15.16:0/3051802321 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f6b1c0c2070 con 0x7f6b241567e0 2024-03-31T15:15:09.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:09.237+0000 7f6b21600640 1 --2- 172.21.15.16:0/3051802321 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f6b04068380 0x7f6b0406a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:09.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:09.237+0000 7f6b21600640 1 -- 172.21.15.16:0/3051802321 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(114..114 src has 1..114) ==== 6025+0+0 (secure 0 0 0) 0x7f6b1c120960 con 0x7f6b241567e0 2024-03-31T15:15:09.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:09.237+0000 7f6b29c00640 1 --2- 172.21.15.16:0/3051802321 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f6b04068380 0x7f6b0406a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:09.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:09.237+0000 7f6b29c00640 1 --2- 172.21.15.16:0/3051802321 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f6b04068380 0x7f6b0406a840 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7f6b18007e60 tx=0x7f6b18008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:15:09.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:09.241+0000 7f6b21600640 1 -- 172.21.15.16:0/3051802321 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f6b1c0165e0 con 0x7f6b241567e0 2024-03-31T15:15:09.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:09.441+0000 7f6b2c600640 1 -- 172.21.15.16:0/3051802321 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7f6b24156010 con 0x7f6b241567e0 2024-03-31T15:15:10.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:10.053+0000 7f6b21600640 1 -- 172.21.15.16:0/3051802321 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v114) ==== 104+0+0 (secure 0 0 0) 0x7f6b1c0a6640 con 0x7f6b241567e0 2024-03-31T15:15:10.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:10.057+0000 7f6b2c600640 1 -- 172.21.15.16:0/3051802321 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f6b04068380 msgr2=0x7f6b0406a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:10.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:10.057+0000 7f6b2c600640 1 --2- 172.21.15.16:0/3051802321 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f6b04068380 0x7f6b0406a840 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7f6b18007e60 tx=0x7f6b18008040 comp rx=0 tx=0).stop 2024-03-31T15:15:10.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:10.057+0000 7f6b2c600640 1 -- 172.21.15.16:0/3051802321 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6b241567e0 msgr2=0x7f6b2415bdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:10.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:10.057+0000 7f6b2c600640 1 --2- 172.21.15.16:0/3051802321 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6b241567e0 0x7f6b2415bdb0 secure :-1 s=READY pgs=1879 cs=0 l=1 rev1=1 crypto rx=0x7f6b2406f600 tx=0x7f6b1c0194d0 comp rx=0 tx=0).stop 2024-03-31T15:15:10.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:10.057+0000 7f6b2c600640 1 -- 172.21.15.16:0/3051802321 shutdown_connections 2024-03-31T15:15:10.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:10.057+0000 7f6b2c600640 1 --2- 172.21.15.16:0/3051802321 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f6b04068380 0x7f6b0406a840 unknown :-1 s=CLOSED pgs=789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:10.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:10.057+0000 7f6b2c600640 1 --2- 172.21.15.16:0/3051802321 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6b241567e0 0x7f6b2415bdb0 unknown :-1 s=CLOSED pgs=1879 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:10.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:10.057+0000 7f6b2c600640 1 --2- 172.21.15.16:0/3051802321 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6b240731c0 0x7f6b2415b870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:10.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:10.057+0000 7f6b2c600640 1 --2- 172.21.15.16:0/3051802321 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f6b2406da10 0x7f6b24143310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:10.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:10.057+0000 7f6b2c600640 1 -- 172.21.15.16:0/3051802321 >> 172.21.15.16:0/3051802321 conn(0x7f6b2406ee40 msgr2=0x7f6b2415a450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:15:10.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:10.057+0000 7f6b2c600640 1 -- 172.21.15.16:0/3051802321 shutdown_connections 2024-03-31T15:15:10.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:10.057+0000 7f6b2c600640 1 -- 172.21.15.16:0/3051802321 wait complete. 2024-03-31T15:15:10.070 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-31T15:15:10.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:10.269+0000 7f597a800640 1 Processor -- start 2024-03-31T15:15:10.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:10.269+0000 7f597a800640 1 -- start start 2024-03-31T15:15:10.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:10.269+0000 7f597a800640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f597413efd0 0x7f59740b1be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:10.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:10.269+0000 7f597a800640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f59740b2120 0x7f597415fde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:10.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:10.269+0000 7f597a800640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f597413f7b0 con 0x7f5974160320 2024-03-31T15:15:10.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:10.269+0000 7f597a800640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f59740b2590 con 0x7f597413efd0 2024-03-31T15:15:10.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:10.269+0000 7f597a800640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f5974160b80 con 0x7f59740b2120 2024-03-31T15:15:10.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:10.269+0000 7f5973400640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f597413efd0 0x7f59740b1be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:10.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:10.269+0000 7f5973400640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f597413efd0 0x7f59740b1be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46166/0 (socket says 172.21.15.16:46166) 2024-03-31T15:15:10.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:10.269+0000 7f5973400640 1 -- 172.21.15.16:0/3556484794 learned_addr learned my addr 172.21.15.16:0/3556484794 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:15:10.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:10.269+0000 7f596aa00640 1 --2- 172.21.15.16:0/3556484794 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f59740b2120 0x7f597415fde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:10.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:10.269+0000 7f5973400640 1 -- 172.21.15.16:0/3556484794 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f59740b2120 msgr2=0x7f597415fde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:15:10.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:10.269+0000 7f5973400640 1 --2- 172.21.15.16:0/3556484794 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f59740b2120 0x7f597415fde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:10.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:10.269+0000 7f5973400640 1 -- 172.21.15.16:0/3556484794 >> v1:172.21.15.89:6789/0 conn(0x7f5974160320 legacy=0x7f5974141840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:10.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:10.269+0000 7f5973400640 1 -- 172.21.15.16:0/3556484794 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f597413fa70 con 0x7f597413efd0 2024-03-31T15:15:10.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:10.273+0000 7f5973400640 1 --2- 172.21.15.16:0/3556484794 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f597413efd0 0x7f59740b1be0 secure :-1 s=READY pgs=1880 cs=0 l=1 rev1=1 crypto rx=0x7f5964009870 tx=0x7f596401c530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc2779a7d6ad94e0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:15:10.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:10.273+0000 7f5972a00640 1 -- 172.21.15.16:0/3556484794 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59640ab650 con 0x7f597413efd0 2024-03-31T15:15:10.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:10.273+0000 7f5972a00640 1 -- 172.21.15.16:0/3556484794 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f59640b0070 con 0x7f597413efd0 2024-03-31T15:15:10.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:10.273+0000 7f5972a00640 1 -- 172.21.15.16:0/3556484794 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59640aba10 con 0x7f597413efd0 2024-03-31T15:15:10.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:10.273+0000 7f597a800640 1 -- 172.21.15.16:0/3556484794 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f597413efd0 msgr2=0x7f59740b1be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:10.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:10.273+0000 7f597a800640 1 --2- 172.21.15.16:0/3556484794 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f597413efd0 0x7f59740b1be0 secure :-1 s=READY pgs=1880 cs=0 l=1 rev1=1 crypto rx=0x7f5964009870 tx=0x7f596401c530 comp rx=0 tx=0).stop 2024-03-31T15:15:10.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:10.273+0000 7f597a800640 1 -- 172.21.15.16:0/3556484794 shutdown_connections 2024-03-31T15:15:10.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:10.273+0000 7f597a800640 1 --2- 172.21.15.16:0/3556484794 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f59740b2120 0x7f597415fde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:10.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:10.273+0000 7f597a800640 1 --2- 172.21.15.16:0/3556484794 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f597413efd0 0x7f59740b1be0 unknown :-1 s=CLOSED pgs=1880 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:10.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:10.273+0000 7f597a800640 1 -- 172.21.15.16:0/3556484794 >> 172.21.15.16:0/3556484794 conn(0x7f597406b620 msgr2=0x7f597406ba30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:15:10.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:10.273+0000 7f597a800640 1 -- 172.21.15.16:0/3556484794 shutdown_connections 2024-03-31T15:15:10.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:10.273+0000 7f597a800640 1 -- 172.21.15.16:0/3556484794 wait complete. 2024-03-31T15:15:10.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:10.277+0000 7f597a800640 1 Processor -- start 2024-03-31T15:15:10.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:10.277+0000 7f597a800640 1 -- start start 2024-03-31T15:15:10.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:10.277+0000 7f597a800640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f59740b2120 0x7f5974152390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:10.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:10.277+0000 7f597a800640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f597413efd0 0x7f597413c780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:10.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:10.277+0000 7f597a800640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5974160320 0x7f597413ccc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:10.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:10.277+0000 7f597a800640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f597413f3b0 con 0x7f5974160320 2024-03-31T15:15:10.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:10.277+0000 7f597a800640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f5974141560 con 0x7f59740b2120 2024-03-31T15:15:10.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:10.277+0000 7f597a800640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f597413f530 con 0x7f597413efd0 2024-03-31T15:15:10.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:10.277+0000 7f5973e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5974160320 0x7f597413ccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:10.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:10.277+0000 7f5973400640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f59740b2120 0x7f5974152390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:10.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:10.277+0000 7f5973400640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f59740b2120 0x7f5974152390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.16:55466/0 (socket says 172.21.15.16:55466) 2024-03-31T15:15:10.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:10.277+0000 7f5973400640 1 -- 172.21.15.16:0/682956800 learned_addr learned my addr 172.21.15.16:0/682956800 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:15:10.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:10.277+0000 7f596aa00640 1 --2- 172.21.15.16:0/682956800 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f597413efd0 0x7f597413c780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:10.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:10.277+0000 7f5973e00640 1 -- 172.21.15.16:0/682956800 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f597413efd0 msgr2=0x7f597413c780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:10.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:10.277+0000 7f5973e00640 1 --2- 172.21.15.16:0/682956800 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f597413efd0 0x7f597413c780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:10.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:10.277+0000 7f5973e00640 1 -- 172.21.15.16:0/682956800 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f59740b2120 msgr2=0x7f5974152390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:10.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:10.277+0000 7f5973e00640 1 --2- 172.21.15.16:0/682956800 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f59740b2120 0x7f5974152390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:10.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:10.277+0000 7f5973e00640 1 -- 172.21.15.16:0/682956800 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f597413d2f0 con 0x7f5974160320 2024-03-31T15:15:10.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:10.277+0000 7f5973400640 1 --2- 172.21.15.16:0/682956800 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f59740b2120 0x7f5974152390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:15:10.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:10.277+0000 7f5973e00640 1 --2- 172.21.15.16:0/682956800 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5974160320 0x7f597413ccc0 secure :-1 s=READY pgs=1881 cs=0 l=1 rev1=1 crypto rx=0x7f596c00c700 tx=0x7f596c00cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:15:10.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:10.277+0000 7f596be00640 1 -- 172.21.15.16:0/682956800 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f596c003b80 con 0x7f5974160320 2024-03-31T15:15:10.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:10.277+0000 7f596be00640 1 -- 172.21.15.16:0/682956800 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f596c003d20 con 0x7f5974160320 2024-03-31T15:15:10.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:10.277+0000 7f596be00640 1 -- 172.21.15.16:0/682956800 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f596c00aa30 con 0x7f5974160320 2024-03-31T15:15:10.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:10.277+0000 7f597a800640 1 -- 172.21.15.16:0/682956800 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f597413d5d0 con 0x7f5974160320 2024-03-31T15:15:10.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:10.277+0000 7f597a800640 1 -- 172.21.15.16:0/682956800 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f597413db90 con 0x7f5974160320 2024-03-31T15:15:10.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:10.281+0000 7f597a800640 1 -- 172.21.15.16:0/682956800 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5974069950 con 0x7f5974160320 2024-03-31T15:15:10.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:10.281+0000 7f596be00640 1 -- 172.21.15.16:0/682956800 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f596c00abd0 con 0x7f5974160320 2024-03-31T15:15:10.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:10.281+0000 7f596be00640 1 --2- 172.21.15.16:0/682956800 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f5950068380 0x7f595006a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:10.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:10.281+0000 7f596be00640 1 -- 172.21.15.16:0/682956800 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(114..114 src has 1..114) ==== 6025+0+0 (secure 0 0 0) 0x7f596c0868a0 con 0x7f5974160320 2024-03-31T15:15:10.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:10.281+0000 7f5973400640 1 --2- 172.21.15.16:0/682956800 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f5950068380 0x7f595006a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:10.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:10.281+0000 7f5973400640 1 --2- 172.21.15.16:0/682956800 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f5950068380 0x7f595006a840 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f5964009870 tx=0x7f59640a7000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:15:10.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:10.285+0000 7f596be00640 1 -- 172.21.15.16:0/682956800 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f596c056630 con 0x7f5974160320 2024-03-31T15:15:10.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:10.453+0000 7f597a800640 1 -- 172.21.15.16:0/682956800 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7f59740b29e0 con 0x7f5974160320 2024-03-31T15:15:11.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:11.197+0000 7f596be00640 1 -- 172.21.15.16:0/682956800 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v115) ==== 181+0+0 (secure 0 0 0) 0x7f596c05a930 con 0x7f5974160320 2024-03-31T15:15:11.187 INFO:teuthology.orchestra.run.smithi016.stderr:pool 'cephfs_metadata' removed 2024-03-31T15:15:11.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:11.201+0000 7f597a800640 1 -- 172.21.15.16:0/682956800 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f5950068380 msgr2=0x7f595006a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:11.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:11.201+0000 7f597a800640 1 --2- 172.21.15.16:0/682956800 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f5950068380 0x7f595006a840 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f5964009870 tx=0x7f59640a7000 comp rx=0 tx=0).stop 2024-03-31T15:15:11.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:11.201+0000 7f597a800640 1 -- 172.21.15.16:0/682956800 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5974160320 msgr2=0x7f597413ccc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:11.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:11.201+0000 7f597a800640 1 --2- 172.21.15.16:0/682956800 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5974160320 0x7f597413ccc0 secure :-1 s=READY pgs=1881 cs=0 l=1 rev1=1 crypto rx=0x7f596c00c700 tx=0x7f596c00cbc0 comp rx=0 tx=0).stop 2024-03-31T15:15:11.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:11.201+0000 7f597a800640 1 -- 172.21.15.16:0/682956800 shutdown_connections 2024-03-31T15:15:11.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:11.201+0000 7f597a800640 1 --2- 172.21.15.16:0/682956800 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f5950068380 0x7f595006a840 unknown :-1 s=CLOSED pgs=791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:11.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:11.201+0000 7f597a800640 1 --2- 172.21.15.16:0/682956800 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5974160320 0x7f597413ccc0 unknown :-1 s=CLOSED pgs=1881 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:11.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:11.201+0000 7f597a800640 1 --2- 172.21.15.16:0/682956800 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f597413efd0 0x7f597413c780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:11.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:11.201+0000 7f597a800640 1 --2- 172.21.15.16:0/682956800 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f59740b2120 0x7f5974152390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:11.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:11.201+0000 7f597a800640 1 -- 172.21.15.16:0/682956800 >> 172.21.15.16:0/682956800 conn(0x7f597406b620 msgr2=0x7f597406ba00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:15:11.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:11.201+0000 7f597a800640 1 -- 172.21.15.16:0/682956800 shutdown_connections 2024-03-31T15:15:11.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:11.201+0000 7f597a800640 1 -- 172.21.15.16:0/682956800 wait complete. 2024-03-31T15:15:11.217 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-31T15:15:11.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:11.377+0000 7fb414000640 1 Processor -- start 2024-03-31T15:15:11.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:11.377+0000 7fb414000640 1 -- start start 2024-03-31T15:15:11.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:11.377+0000 7fb414000640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb40c0730c0 0x7fb40c0b45d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:11.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:11.377+0000 7fb414000640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb40c0734f0 con 0x7fb40c143f40 2024-03-31T15:15:11.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:11.377+0000 7fb414000640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb40c144320 con 0x7fb40c0b22f0 2024-03-31T15:15:11.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:11.377+0000 7fb414000640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7fb40c15ef80 con 0x7fb40c0730c0 2024-03-31T15:15:11.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:11.377+0000 7fb411600640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fb40c143f40 0x7fb40c0b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:46632/0 (socket says 172.21.15.16:46632) 2024-03-31T15:15:11.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:11.377+0000 7fb411600640 1 -- 172.21.15.16:0/3287716982 learned_addr learned my addr 172.21.15.16:0/3287716982 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:15:11.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:11.377+0000 7fb412000640 1 --2- 172.21.15.16:0/3287716982 >> v2:172.21.15.16:3300/0 conn(0x7fb40c0730c0 0x7fb40c0b45d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:11.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:11.377+0000 7fb412000640 1 -- 172.21.15.16:0/3287716982 >> v1:172.21.15.89:6789/0 conn(0x7fb40c0b22f0 legacy=0x7fb40c0b2f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:11.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:11.377+0000 7fb412000640 1 -- 172.21.15.16:0/3287716982 >> v1:172.21.15.16:6789/0 conn(0x7fb40c143f40 legacy=0x7fb40c0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:11.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:11.377+0000 7fb412000640 1 -- 172.21.15.16:0/3287716982 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb40c15f260 con 0x7fb40c0730c0 2024-03-31T15:15:11.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:11.381+0000 7fb412000640 1 --2- 172.21.15.16:0/3287716982 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb40c0730c0 0x7fb40c0b45d0 secure :-1 s=READY pgs=1883 cs=0 l=1 rev1=1 crypto rx=0x7fb404009d90 tx=0x7fb40401c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3af2c4ec44f2b1a1 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:15:11.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:11.381+0000 7fb40be00640 1 -- 172.21.15.16:0/3287716982 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4040b2070 con 0x7fb40c0730c0 2024-03-31T15:15:11.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:11.381+0000 7fb40be00640 1 -- 172.21.15.16:0/3287716982 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb4040a5830 con 0x7fb40c0730c0 2024-03-31T15:15:11.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:11.381+0000 7fb40be00640 1 -- 172.21.15.16:0/3287716982 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4040a5c00 con 0x7fb40c0730c0 2024-03-31T15:15:11.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:11.381+0000 7fb414000640 1 -- 172.21.15.16:0/3287716982 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb40c0730c0 msgr2=0x7fb40c0b45d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:11.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:11.381+0000 7fb414000640 1 --2- 172.21.15.16:0/3287716982 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb40c0730c0 0x7fb40c0b45d0 secure :-1 s=READY pgs=1883 cs=0 l=1 rev1=1 crypto rx=0x7fb404009d90 tx=0x7fb40401c970 comp rx=0 tx=0).stop 2024-03-31T15:15:11.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:11.381+0000 7fb414000640 1 -- 172.21.15.16:0/3287716982 shutdown_connections 2024-03-31T15:15:11.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:11.381+0000 7fb414000640 1 --2- 172.21.15.16:0/3287716982 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb40c0730c0 0x7fb40c0b45d0 unknown :-1 s=CLOSED pgs=1883 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:11.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:11.381+0000 7fb414000640 1 -- 172.21.15.16:0/3287716982 >> 172.21.15.16:0/3287716982 conn(0x7fb40c06b620 msgr2=0x7fb40c06ba30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:15:11.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:11.381+0000 7fb414000640 1 -- 172.21.15.16:0/3287716982 shutdown_connections 2024-03-31T15:15:11.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:11.381+0000 7fb414000640 1 -- 172.21.15.16:0/3287716982 wait complete. 2024-03-31T15:15:11.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:11.385+0000 7fb414000640 1 Processor -- start 2024-03-31T15:15:11.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:11.385+0000 7fb414000640 1 -- start start 2024-03-31T15:15:11.375 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:11.385+0000 7fb414000640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb40c0730c0 0x7fb40c15bdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:11.375 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:11.385+0000 7fb414000640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb40c0b22f0 0x7fb40c071f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:11.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:11.385+0000 7fb414000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb40c143f40 0x7fb40c157870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:11.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:11.385+0000 7fb414000640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fb40c169240 con 0x7fb40c143f40 2024-03-31T15:15:11.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:11.385+0000 7fb414000640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fb40c1690c0 con 0x7fb40c0730c0 2024-03-31T15:15:11.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:11.385+0000 7fb414000640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fb40c1693c0 con 0x7fb40c0b22f0 2024-03-31T15:15:11.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:11.385+0000 7fb412000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb40c143f40 0x7fb40c157870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:11.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:11.385+0000 7fb412000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb40c143f40 0x7fb40c157870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46196/0 (socket says 172.21.15.16:46196) 2024-03-31T15:15:11.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:11.385+0000 7fb412000640 1 -- 172.21.15.16:0/2059397334 learned_addr learned my addr 172.21.15.16:0/2059397334 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:15:11.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:11.385+0000 7fb411600640 1 --2- 172.21.15.16:0/2059397334 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb40c0730c0 0x7fb40c15bdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:11.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:11.385+0000 7fb410c00640 1 --2- 172.21.15.16:0/2059397334 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb40c0b22f0 0x7fb40c071f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:11.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:11.385+0000 7fb412000640 1 -- 172.21.15.16:0/2059397334 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb40c0b22f0 msgr2=0x7fb40c071f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:11.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:11.385+0000 7fb412000640 1 --2- 172.21.15.16:0/2059397334 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb40c0b22f0 0x7fb40c071f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:11.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:11.385+0000 7fb412000640 1 -- 172.21.15.16:0/2059397334 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb40c0730c0 msgr2=0x7fb40c15bdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:11.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:11.385+0000 7fb412000640 1 --2- 172.21.15.16:0/2059397334 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb40c0730c0 0x7fb40c15bdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:11.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:11.385+0000 7fb412000640 1 -- 172.21.15.16:0/2059397334 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb40c1409e0 con 0x7fb40c143f40 2024-03-31T15:15:11.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:11.385+0000 7fb410c00640 1 --2- 172.21.15.16:0/2059397334 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb40c0b22f0 0x7fb40c071f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:15:11.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:11.385+0000 7fb412000640 1 --2- 172.21.15.16:0/2059397334 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb40c143f40 0x7fb40c157870 secure :-1 s=READY pgs=1884 cs=0 l=1 rev1=1 crypto rx=0x7fb40401a5a0 tx=0x7fb404019710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:15:11.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:11.385+0000 7fb409600640 1 -- 172.21.15.16:0/2059397334 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4040b2050 con 0x7fb40c143f40 2024-03-31T15:15:11.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:11.385+0000 7fb409600640 1 -- 172.21.15.16:0/2059397334 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb4040be070 con 0x7fb40c143f40 2024-03-31T15:15:11.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:11.385+0000 7fb409600640 1 -- 172.21.15.16:0/2059397334 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4040ada50 con 0x7fb40c143f40 2024-03-31T15:15:11.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:11.385+0000 7fb414000640 1 -- 172.21.15.16:0/2059397334 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb40c15b000 con 0x7fb40c143f40 2024-03-31T15:15:11.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:11.385+0000 7fb414000640 1 -- 172.21.15.16:0/2059397334 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb40c15b5c0 con 0x7fb40c143f40 2024-03-31T15:15:11.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:11.389+0000 7fb414000640 1 -- 172.21.15.16:0/2059397334 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb40c069950 con 0x7fb40c143f40 2024-03-31T15:15:11.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:11.389+0000 7fb409600640 1 -- 172.21.15.16:0/2059397334 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fb4040adbf0 con 0x7fb40c143f40 2024-03-31T15:15:11.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:11.389+0000 7fb409600640 1 --2- 172.21.15.16:0/2059397334 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fb3ec068380 0x7fb3ec06a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:11.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:11.389+0000 7fb409600640 1 -- 172.21.15.16:0/2059397334 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(115..115 src has 1..115) ==== 5587+0+0 (secure 0 0 0) 0x7fb404121d50 con 0x7fb40c143f40 2024-03-31T15:15:11.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:11.389+0000 7fb411600640 1 --2- 172.21.15.16:0/2059397334 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fb3ec068380 0x7fb3ec06a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:11.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:11.389+0000 7fb411600640 1 --2- 172.21.15.16:0/2059397334 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fb3ec068380 0x7fb3ec06a840 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7fb400005890 tx=0x7fb400009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:15:11.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:11.393+0000 7fb409600640 1 -- 172.21.15.16:0/2059397334 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fb404016560 con 0x7fb40c143f40 2024-03-31T15:15:11.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:11.561+0000 7fb414000640 1 -- 172.21.15.16:0/2059397334 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7fb40c0b4060 con 0x7fb40c143f40 2024-03-31T15:15:12.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:12.197+0000 7fb409600640 1 -- 172.21.15.16:0/2059397334 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v116) ==== 169+0+0 (secure 0 0 0) 0x7fb4040f2410 con 0x7fb40c143f40 2024-03-31T15:15:12.189 INFO:teuthology.orchestra.run.smithi016.stderr:pool 'cephfs_data' removed 2024-03-31T15:15:12.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:12.201+0000 7fb414000640 1 -- 172.21.15.16:0/2059397334 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fb3ec068380 msgr2=0x7fb3ec06a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:12.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:12.201+0000 7fb414000640 1 --2- 172.21.15.16:0/2059397334 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fb3ec068380 0x7fb3ec06a840 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7fb400005890 tx=0x7fb400009040 comp rx=0 tx=0).stop 2024-03-31T15:15:12.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:12.201+0000 7fb414000640 1 -- 172.21.15.16:0/2059397334 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb40c143f40 msgr2=0x7fb40c157870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:12.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:12.201+0000 7fb414000640 1 --2- 172.21.15.16:0/2059397334 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb40c143f40 0x7fb40c157870 secure :-1 s=READY pgs=1884 cs=0 l=1 rev1=1 crypto rx=0x7fb40401a5a0 tx=0x7fb404019710 comp rx=0 tx=0).stop 2024-03-31T15:15:12.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:12.201+0000 7fb414000640 1 -- 172.21.15.16:0/2059397334 shutdown_connections 2024-03-31T15:15:12.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:12.201+0000 7fb414000640 1 --2- 172.21.15.16:0/2059397334 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fb3ec068380 0x7fb3ec06a840 unknown :-1 s=CLOSED pgs=792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:12.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:12.201+0000 7fb414000640 1 --2- 172.21.15.16:0/2059397334 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb40c143f40 0x7fb40c157870 unknown :-1 s=CLOSED pgs=1884 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:12.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:12.201+0000 7fb414000640 1 --2- 172.21.15.16:0/2059397334 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fb40c0b22f0 0x7fb40c071f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:12.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:12.201+0000 7fb414000640 1 --2- 172.21.15.16:0/2059397334 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb40c0730c0 0x7fb40c15bdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:12.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:12.201+0000 7fb414000640 1 -- 172.21.15.16:0/2059397334 >> 172.21.15.16:0/2059397334 conn(0x7fb40c06b620 msgr2=0x7fb40c0adc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:15:12.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:12.201+0000 7fb414000640 1 -- 172.21.15.16:0/2059397334 shutdown_connections 2024-03-31T15:15:12.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:12.201+0000 7fb414000640 1 -- 172.21.15.16:0/2059397334 wait complete. 2024-03-31T15:15:12.215 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-03-31T15:15:12.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:12.373+0000 7fcae9400640 1 Processor -- start 2024-03-31T15:15:12.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:12.373+0000 7fcae9400640 1 -- start start 2024-03-31T15:15:12.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:12.377+0000 7fcae9400640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fcae41411b0 0x7fcae406f920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:12.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:12.377+0000 7fcae9400640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fcae4141590 0x7fcae406fe60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:12.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:12.377+0000 7fcae9400640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fcae4143c30 0x7fcae41440a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:12.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:12.377+0000 7fcae9400640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7fcae4162910 con 0x7fcae4141590 2024-03-31T15:15:12.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:12.377+0000 7fcae9400640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7fcae4144980 con 0x7fcae41411b0 2024-03-31T15:15:12.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:12.377+0000 7fcae9400640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fcae4161fe0 con 0x7fcae4143c30 2024-03-31T15:15:12.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:12.377+0000 7fcadbe00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fcae4141590 0x7fcae406fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:12.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:12.377+0000 7fcadbe00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fcae4141590 0x7fcae406fe60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46210/0 (socket says 172.21.15.16:46210) 2024-03-31T15:15:12.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:12.377+0000 7fcadbe00640 1 -- 172.21.15.16:0/2796983084 learned_addr learned my addr 172.21.15.16:0/2796983084 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:15:12.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:12.377+0000 7fcae2800640 1 --2- 172.21.15.16:0/2796983084 >> v2:172.21.15.89:3300/0 conn(0x7fcae41411b0 0x7fcae406f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:12.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:12.377+0000 7fcae3200640 1 --2- 172.21.15.16:0/2796983084 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fcae4143c30 0x7fcae41440a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:12.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:12.377+0000 7fcadbe00640 1 -- 172.21.15.16:0/2796983084 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fcae4143c30 msgr2=0x7fcae41440a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:15:12.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:12.377+0000 7fcadbe00640 1 --2- 172.21.15.16:0/2796983084 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fcae4143c30 0x7fcae41440a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:12.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:12.377+0000 7fcadbe00640 1 -- 172.21.15.16:0/2796983084 >> v2:172.21.15.89:3300/0 conn(0x7fcae41411b0 msgr2=0x7fcae406f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:15:12.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:12.377+0000 7fcadbe00640 1 --2- 172.21.15.16:0/2796983084 >> v2:172.21.15.89:3300/0 conn(0x7fcae41411b0 0x7fcae406f920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:12.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:12.377+0000 7fcadbe00640 1 -- 172.21.15.16:0/2796983084 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcae41622c0 con 0x7fcae4141590 2024-03-31T15:15:12.369 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:12.381+0000 7fcadbe00640 1 --2- 172.21.15.16:0/2796983084 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcae4141590 0x7fcae406fe60 secure :-1 s=READY pgs=1885 cs=0 l=1 rev1=1 crypto rx=0x7fcae40d3120 tx=0x7fcacc01c4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cec781ed5d2b7bbf server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:15:12.369 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:12.381+0000 7fcae1e00640 1 -- 172.21.15.16:0/2796983084 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcacc0aa430 con 0x7fcae4141590 2024-03-31T15:15:12.369 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:12.381+0000 7fcae1e00640 1 -- 172.21.15.16:0/2796983084 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcacc0af070 con 0x7fcae4141590 2024-03-31T15:15:12.369 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:12.381+0000 7fcae1e00640 1 -- 172.21.15.16:0/2796983084 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcacc0aa7f0 con 0x7fcae4141590 2024-03-31T15:15:12.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:12.381+0000 7fcae9400640 1 -- 172.21.15.16:0/2796983084 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcae4141590 msgr2=0x7fcae406fe60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:12.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:12.381+0000 7fcae9400640 1 --2- 172.21.15.16:0/2796983084 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcae4141590 0x7fcae406fe60 secure :-1 s=READY pgs=1885 cs=0 l=1 rev1=1 crypto rx=0x7fcae40d3120 tx=0x7fcacc01c4f0 comp rx=0 tx=0).stop 2024-03-31T15:15:12.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:12.381+0000 7fcae9400640 1 -- 172.21.15.16:0/2796983084 shutdown_connections 2024-03-31T15:15:12.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:12.381+0000 7fcae9400640 1 --2- 172.21.15.16:0/2796983084 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fcae4143c30 0x7fcae41440a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:12.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:12.381+0000 7fcae9400640 1 --2- 172.21.15.16:0/2796983084 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcae4141590 0x7fcae406fe60 unknown :-1 s=CLOSED pgs=1885 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:12.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:12.381+0000 7fcae9400640 1 --2- 172.21.15.16:0/2796983084 >> v2:172.21.15.89:3300/0 conn(0x7fcae41411b0 0x7fcae406f920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:12.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:12.381+0000 7fcae9400640 1 -- 172.21.15.16:0/2796983084 >> 172.21.15.16:0/2796983084 conn(0x7fcae4067930 msgr2=0x7fcae40b3ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:15:12.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:12.381+0000 7fcae9400640 1 -- 172.21.15.16:0/2796983084 shutdown_connections 2024-03-31T15:15:12.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:12.381+0000 7fcae9400640 1 -- 172.21.15.16:0/2796983084 wait complete. 2024-03-31T15:15:12.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:12.381+0000 7fcae9400640 1 Processor -- start 2024-03-31T15:15:12.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:12.381+0000 7fcae9400640 1 -- start start 2024-03-31T15:15:12.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:12.381+0000 7fcae9400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcae41411b0 0x7fcae413d800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:12.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:12.381+0000 7fcae9400640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcae4141590 0x7fcae4156380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:12.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:12.381+0000 7fcae9400640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fcae4143c30 0x7fcae415fdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:12.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:12.381+0000 7fcae9400640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fcae40717f0 con 0x7fcae41411b0 2024-03-31T15:15:12.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:12.381+0000 7fcae9400640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fcae4071670 con 0x7fcae4141590 2024-03-31T15:15:12.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:12.381+0000 7fcae9400640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fcae4071970 con 0x7fcae4143c30 2024-03-31T15:15:12.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:12.381+0000 7fcae2800640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcae41411b0 0x7fcae413d800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:12.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:12.381+0000 7fcae2800640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcae41411b0 0x7fcae413d800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46226/0 (socket says 172.21.15.16:46226) 2024-03-31T15:15:12.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:12.381+0000 7fcae2800640 1 -- 172.21.15.16:0/2347332180 learned_addr learned my addr 172.21.15.16:0/2347332180 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:15:12.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:12.385+0000 7fcadbe00640 1 --2- 172.21.15.16:0/2347332180 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcae4141590 0x7fcae4156380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:12.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:12.385+0000 7fcae3200640 1 --2- 172.21.15.16:0/2347332180 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fcae4143c30 0x7fcae415fdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:12.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:12.385+0000 7fcae2800640 1 -- 172.21.15.16:0/2347332180 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fcae4143c30 msgr2=0x7fcae415fdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:12.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:12.385+0000 7fcae2800640 1 --2- 172.21.15.16:0/2347332180 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fcae4143c30 0x7fcae415fdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:12.375 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:12.385+0000 7fcae2800640 1 -- 172.21.15.16:0/2347332180 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcae4141590 msgr2=0x7fcae4156380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:12.375 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:12.385+0000 7fcae2800640 1 --2- 172.21.15.16:0/2347332180 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcae4141590 0x7fcae4156380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:12.375 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:12.385+0000 7fcae2800640 1 -- 172.21.15.16:0/2347332180 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcae413dd40 con 0x7fcae41411b0 2024-03-31T15:15:12.375 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:12.385+0000 7fcae2800640 1 --2- 172.21.15.16:0/2347332180 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcae41411b0 0x7fcae413d800 secure :-1 s=READY pgs=1886 cs=0 l=1 rev1=1 crypto rx=0x7fcad400d800 tx=0x7fcad400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:15:12.375 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:12.385+0000 7fcadaa00640 1 -- 172.21.15.16:0/2347332180 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcad400b6c0 con 0x7fcae41411b0 2024-03-31T15:15:12.375 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:12.385+0000 7fcadaa00640 1 -- 172.21.15.16:0/2347332180 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcad4019070 con 0x7fcae41411b0 2024-03-31T15:15:12.375 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:12.385+0000 7fcadaa00640 1 -- 172.21.15.16:0/2347332180 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcad400ba60 con 0x7fcae41411b0 2024-03-31T15:15:12.375 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:12.385+0000 7fcae9400640 1 -- 172.21.15.16:0/2347332180 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcae4156c80 con 0x7fcae41411b0 2024-03-31T15:15:12.375 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:12.385+0000 7fcae9400640 1 -- 172.21.15.16:0/2347332180 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcae413e120 con 0x7fcae41411b0 2024-03-31T15:15:12.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:12.385+0000 7fcae9400640 1 -- 172.21.15.16:0/2347332180 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcae4069950 con 0x7fcae41411b0 2024-03-31T15:15:12.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:12.385+0000 7fcadaa00640 1 -- 172.21.15.16:0/2347332180 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fcad400bc00 con 0x7fcae41411b0 2024-03-31T15:15:12.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:12.385+0000 7fcadaa00640 1 --2- 172.21.15.16:0/2347332180 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fcac0068380 0x7fcac006a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:12.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:12.385+0000 7fcadaa00640 1 -- 172.21.15.16:0/2347332180 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(116..116 src has 1..116) ==== 5115+0+0 (secure 0 0 0) 0x7fcad4086bf0 con 0x7fcae41411b0 2024-03-31T15:15:12.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:12.389+0000 7fcadbe00640 1 --2- 172.21.15.16:0/2347332180 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fcac0068380 0x7fcac006a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:12.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:12.389+0000 7fcadbe00640 1 --2- 172.21.15.16:0/2347332180 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fcac0068380 0x7fcac006a840 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7fcacc0ae000 tx=0x7fcacc0a7000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:15:12.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:12.389+0000 7fcadaa00640 1 -- 172.21.15.16:0/2347332180 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fcad4057360 con 0x7fcae41411b0 2024-03-31T15:15:12.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:12.581+0000 7fcae9400640 1 -- 172.21.15.16:0/2347332180 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) -- 0x7fcae4154fc0 con 0x7fcae41411b0 2024-03-31T15:15:12.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:12.585+0000 7fcadaa00640 1 -- 172.21.15.16:0/2347332180 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v23) ==== 104+0+0 (secure 0 0 0) 0x7fcad4056c60 con 0x7fcae41411b0 2024-03-31T15:15:12.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:12.585+0000 7fcae9400640 1 -- 172.21.15.16:0/2347332180 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fcac0068380 msgr2=0x7fcac006a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:12.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:12.585+0000 7fcae9400640 1 --2- 172.21.15.16:0/2347332180 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fcac0068380 0x7fcac006a840 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7fcacc0ae000 tx=0x7fcacc0a7000 comp rx=0 tx=0).stop 2024-03-31T15:15:12.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:12.585+0000 7fcae9400640 1 -- 172.21.15.16:0/2347332180 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcae41411b0 msgr2=0x7fcae413d800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:12.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:12.585+0000 7fcae9400640 1 --2- 172.21.15.16:0/2347332180 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcae41411b0 0x7fcae413d800 secure :-1 s=READY pgs=1886 cs=0 l=1 rev1=1 crypto rx=0x7fcad400d800 tx=0x7fcad400dcc0 comp rx=0 tx=0).stop 2024-03-31T15:15:12.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:12.585+0000 7fcae9400640 1 -- 172.21.15.16:0/2347332180 shutdown_connections 2024-03-31T15:15:12.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:12.585+0000 7fcae9400640 1 --2- 172.21.15.16:0/2347332180 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fcac0068380 0x7fcac006a840 unknown :-1 s=CLOSED pgs=793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:12.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:12.585+0000 7fcae9400640 1 --2- 172.21.15.16:0/2347332180 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fcae4143c30 0x7fcae415fdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:12.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:12.585+0000 7fcae9400640 1 --2- 172.21.15.16:0/2347332180 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcae4141590 0x7fcae4156380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:12.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:12.585+0000 7fcae9400640 1 --2- 172.21.15.16:0/2347332180 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcae41411b0 0x7fcae413d800 unknown :-1 s=CLOSED pgs=1886 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:12.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:12.585+0000 7fcae9400640 1 -- 172.21.15.16:0/2347332180 >> 172.21.15.16:0/2347332180 conn(0x7fcae4067930 msgr2=0x7fcae40b3c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:15:12.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:12.585+0000 7fcae9400640 1 -- 172.21.15.16:0/2347332180 shutdown_connections 2024-03-31T15:15:12.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:12.585+0000 7fcae9400640 1 -- 172.21.15.16:0/2347332180 wait complete. 2024-03-31T15:15:12.595 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_orphan_scan' 2024-03-31T15:15:12.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:12.745+0000 7f223ae00640 1 Processor -- start 2024-03-31T15:15:12.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:12.745+0000 7f223ae00640 1 -- start start 2024-03-31T15:15:12.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:12.749+0000 7f223ae00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f223415f650 0x7f223406f920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:12.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:12.749+0000 7f223ae00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f223406fe60 0x7f2234162180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:12.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:12.749+0000 7f223ae00640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f22340cbe90 con 0x7f223406dbc0 2024-03-31T15:15:12.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:12.749+0000 7f223ae00640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f223415f1c0 con 0x7f223406fe60 2024-03-31T15:15:12.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:12.749+0000 7f223ae00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f22340cc170 con 0x7f223415f650 2024-03-31T15:15:12.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:12.749+0000 7f2233e00640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f223406dbc0 0x7f22340cb440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:46646/0 (socket says 172.21.15.16:46646) 2024-03-31T15:15:12.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:12.749+0000 7f2233e00640 1 -- 172.21.15.16:0/960992904 learned_addr learned my addr 172.21.15.16:0/960992904 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:15:12.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:12.749+0000 7f2233400640 1 -- 172.21.15.16:0/960992904 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1998317891 0 0) 0x7f22340cbe90 con 0x7f223406dbc0 2024-03-31T15:15:12.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:12.749+0000 7f2233400640 1 -- 172.21.15.16:0/960992904 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2218003530 con 0x7f223406dbc0 2024-03-31T15:15:12.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:12.749+0000 7f222be00640 1 --2- 172.21.15.16:0/960992904 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f223415f650 0x7f223406f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:12.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:12.749+0000 7f2238e00640 1 --2- 172.21.15.16:0/960992904 >> v2:172.21.15.89:3300/0 conn(0x7f223406fe60 0x7f2234162180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:12.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:12.749+0000 7f2233400640 1 -- 172.21.15.16:0/960992904 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3639661178 0 0) 0x7f2218003530 con 0x7f223406dbc0 2024-03-31T15:15:12.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:12.749+0000 7f2233400640 1 -- 172.21.15.16:0/960992904 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f223415f650 msgr2=0x7f223406f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:15:12.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:12.749+0000 7f2233400640 1 --2- 172.21.15.16:0/960992904 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f223415f650 0x7f223406f920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:12.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:12.749+0000 7f2233400640 1 -- 172.21.15.16:0/960992904 >> v2:172.21.15.89:3300/0 conn(0x7f223406fe60 msgr2=0x7f2234162180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:15:12.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:12.749+0000 7f2233400640 1 --2- 172.21.15.16:0/960992904 >> v2:172.21.15.89:3300/0 conn(0x7f223406fe60 0x7f2234162180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:12.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:12.749+0000 7f2233400640 1 -- 172.21.15.16:0/960992904 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f223415ee50 con 0x7f223406dbc0 2024-03-31T15:15:12.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:12.749+0000 7f2233400640 1 -- 172.21.15.16:0/960992904 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 214807623 0 0) 0x7f2224001680 con 0x7f223406dbc0 2024-03-31T15:15:12.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:12.749+0000 7f2233400640 1 -- 172.21.15.16:0/960992904 >> v1:172.21.15.16:6789/0 conn(0x7f223406dbc0 legacy=0x7f22340cb440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:12.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:12.749+0000 7f2233400640 1 --2- 172.21.15.16:0/960992904 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f223415f650 0x7f22180039c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:12.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:12.749+0000 7f2233400640 1 --2- 172.21.15.16:0/960992904 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2218003ff0 0x7f2218004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:12.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:12.749+0000 7f2233400640 1 --2- 172.21.15.16:0/960992904 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2218004d50 0x7f22180051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:12.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:12.749+0000 7f2233400640 1 -- 172.21.15.16:0/960992904 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f223415f1c0 con 0x7f2218003ff0 2024-03-31T15:15:12.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:12.749+0000 7f2233400640 1 -- 172.21.15.16:0/960992904 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f22340cc170 con 0x7f2218004d50 2024-03-31T15:15:12.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:12.749+0000 7f2233400640 1 -- 172.21.15.16:0/960992904 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f221800bb90 con 0x7f223415f650 2024-03-31T15:15:12.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:12.749+0000 7f222be00640 1 --2- 172.21.15.16:0/960992904 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2218003ff0 0x7f2218004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:12.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:12.749+0000 7f2233e00640 1 --2- 172.21.15.16:0/960992904 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f223415f650 0x7f22180039c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:12.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:12.749+0000 7f2238e00640 1 --2- 172.21.15.16:0/960992904 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2218004d50 0x7f22180051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:12.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:12.749+0000 7f222be00640 1 -- 172.21.15.16:0/960992904 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f223415f650 msgr2=0x7f22180039c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:15:12.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:12.749+0000 7f222be00640 1 --2- 172.21.15.16:0/960992904 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f223415f650 0x7f22180039c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:12.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:12.749+0000 7f222be00640 1 -- 172.21.15.16:0/960992904 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2218004d50 msgr2=0x7f22180051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:15:12.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:12.749+0000 7f222be00640 1 --2- 172.21.15.16:0/960992904 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2218004d50 0x7f22180051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:12.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:12.749+0000 7f222be00640 1 -- 172.21.15.16:0/960992904 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2218003530 con 0x7f2218003ff0 2024-03-31T15:15:12.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:12.753+0000 7f222be00640 1 --2- 172.21.15.16:0/960992904 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2218003ff0 0x7f2218004810 secure :-1 s=READY pgs=1888 cs=0 l=1 rev1=1 crypto rx=0x7f221c00ba30 tx=0x7f221c01c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f63fe0035e1c5a3d server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:15:12.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:12.753+0000 7f2233400640 1 -- 172.21.15.16:0/960992904 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f221c0b3070 con 0x7f2218003ff0 2024-03-31T15:15:12.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:12.753+0000 7f2233400640 1 -- 172.21.15.16:0/960992904 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f221c0ae630 con 0x7f2218003ff0 2024-03-31T15:15:12.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:12.753+0000 7f2233400640 1 -- 172.21.15.16:0/960992904 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f221c0aa050 con 0x7f2218003ff0 2024-03-31T15:15:12.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:12.753+0000 7f223ae00640 1 -- 172.21.15.16:0/960992904 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2218003ff0 msgr2=0x7f2218004810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:12.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:12.753+0000 7f223ae00640 1 --2- 172.21.15.16:0/960992904 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2218003ff0 0x7f2218004810 secure :-1 s=READY pgs=1888 cs=0 l=1 rev1=1 crypto rx=0x7f221c00ba30 tx=0x7f221c01c6a0 comp rx=0 tx=0).stop 2024-03-31T15:15:12.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:12.753+0000 7f2238e00640 1 -- 172.21.15.16:0/960992904 reap_dead start 2024-03-31T15:15:12.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:12.753+0000 7f223ae00640 1 -- 172.21.15.16:0/960992904 shutdown_connections 2024-03-31T15:15:12.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:12.753+0000 7f223ae00640 1 -- 172.21.15.16:0/960992904 >> 172.21.15.16:0/960992904 conn(0x7f2234067330 msgr2=0x7f2234067740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:15:12.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:12.753+0000 7f223ae00640 1 -- 172.21.15.16:0/960992904 shutdown_connections 2024-03-31T15:15:12.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:12.753+0000 7f223ae00640 1 -- 172.21.15.16:0/960992904 wait complete. 2024-03-31T15:15:12.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:12.753+0000 7f223ae00640 1 Processor -- start 2024-03-31T15:15:12.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:12.753+0000 7f223ae00640 1 -- start start 2024-03-31T15:15:12.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:12.757+0000 7f223ae00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f2218003ff0 0x7f22341501a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:12.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:12.757+0000 7f223ae00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2234150710 0x7f22341566e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:12.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:12.757+0000 7f223ae00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f22341642e0 0x7f2234164760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:12.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:12.757+0000 7f223ae00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f223406c950 con 0x7f22341642e0 2024-03-31T15:15:12.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:12.757+0000 7f223ae00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f223406dfa0 con 0x7f2234150710 2024-03-31T15:15:12.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:12.757+0000 7f223ae00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f223406cad0 con 0x7f2218003ff0 2024-03-31T15:15:12.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:12.757+0000 7f2238e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f22341642e0 0x7f2234164760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:12.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:12.757+0000 7f2238e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f22341642e0 0x7f2234164760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46244/0 (socket says 172.21.15.16:46244) 2024-03-31T15:15:12.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:12.757+0000 7f2238e00640 1 -- 172.21.15.16:0/1116650276 learned_addr learned my addr 172.21.15.16:0/1116650276 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:15:12.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:12.757+0000 7f2233e00640 1 --2- 172.21.15.16:0/1116650276 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f2218003ff0 0x7f22341501a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:12.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:12.757+0000 7f222be00640 1 --2- 172.21.15.16:0/1116650276 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2234150710 0x7f22341566e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:12.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:12.757+0000 7f2238e00640 1 -- 172.21.15.16:0/1116650276 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f2218003ff0 msgr2=0x7f22341501a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:12.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:12.757+0000 7f2238e00640 1 --2- 172.21.15.16:0/1116650276 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f2218003ff0 0x7f22341501a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:12.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:12.757+0000 7f2238e00640 1 -- 172.21.15.16:0/1116650276 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2234150710 msgr2=0x7f22341566e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:12.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:12.757+0000 7f2238e00640 1 --2- 172.21.15.16:0/1116650276 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2234150710 0x7f22341566e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:12.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:12.757+0000 7f2238e00640 1 -- 172.21.15.16:0/1116650276 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22341530c0 con 0x7f22341642e0 2024-03-31T15:15:12.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:12.757+0000 7f2238e00640 1 --2- 172.21.15.16:0/1116650276 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f22341642e0 0x7f2234164760 secure :-1 s=READY pgs=1889 cs=0 l=1 rev1=1 crypto rx=0x7f222c009bb0 tx=0x7f222c007700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:15:12.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:12.757+0000 7f2230c00640 1 -- 172.21.15.16:0/1116650276 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f222c014410 con 0x7f22341642e0 2024-03-31T15:15:12.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:12.757+0000 7f2230c00640 1 -- 172.21.15.16:0/1116650276 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f222c019070 con 0x7f22341642e0 2024-03-31T15:15:12.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:12.757+0000 7f223ae00640 1 -- 172.21.15.16:0/1116650276 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f22341533a0 con 0x7f22341642e0 2024-03-31T15:15:12.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:12.757+0000 7f2230c00640 1 -- 172.21.15.16:0/1116650276 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f222c014810 con 0x7f22341642e0 2024-03-31T15:15:12.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:12.757+0000 7f223ae00640 1 -- 172.21.15.16:0/1116650276 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2234153960 con 0x7f22341642e0 2024-03-31T15:15:12.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:12.757+0000 7f223ae00640 1 -- 172.21.15.16:0/1116650276 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2234069950 con 0x7f22341642e0 2024-03-31T15:15:12.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:12.757+0000 7f2230c00640 1 -- 172.21.15.16:0/1116650276 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f222c0149b0 con 0x7f22341642e0 2024-03-31T15:15:12.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:12.761+0000 7f2230c00640 1 --2- 172.21.15.16:0/1116650276 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f22100683b0 0x7f221006a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:12.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:12.761+0000 7f2230c00640 1 -- 172.21.15.16:0/1116650276 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(116..116 src has 1..116) ==== 5115+0+0 (secure 0 0 0) 0x7f222c086b00 con 0x7f22341642e0 2024-03-31T15:15:12.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:12.761+0000 7f2233e00640 1 --2- 172.21.15.16:0/1116650276 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f22100683b0 0x7f221006a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:12.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:12.761+0000 7f2233e00640 1 --2- 172.21.15.16:0/1116650276 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f22100683b0 0x7f221006a870 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7f2224000c10 tx=0x7f2224007900 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:15:12.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:12.765+0000 7f2230c00640 1 -- 172.21.15.16:0/1116650276 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f222c056cb0 con 0x7f22341642e0 2024-03-31T15:15:12.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:12.985+0000 7f223ae00640 1 -- 172.21.15.16:0/1116650276 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_orphan_scan"]} v 0) -- 0x7f22340cf380 con 0x7f22341642e0 2024-03-31T15:15:13.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:13.209+0000 7f2230c00640 1 -- 172.21.15.16:0/1116650276 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_orphan_scan"]}]=0 v546) ==== 144+0+0 (secure 0 0 0) 0x7f222c05afb0 con 0x7f22341642e0 2024-03-31T15:15:13.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:13.209+0000 7f223ae00640 1 -- 172.21.15.16:0/1116650276 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f22100683b0 msgr2=0x7f221006a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:13.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:13.209+0000 7f223ae00640 1 --2- 172.21.15.16:0/1116650276 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f22100683b0 0x7f221006a870 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7f2224000c10 tx=0x7f2224007900 comp rx=0 tx=0).stop 2024-03-31T15:15:13.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:13.209+0000 7f223ae00640 1 -- 172.21.15.16:0/1116650276 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f22341642e0 msgr2=0x7f2234164760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:13.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:13.209+0000 7f223ae00640 1 --2- 172.21.15.16:0/1116650276 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f22341642e0 0x7f2234164760 secure :-1 s=READY pgs=1889 cs=0 l=1 rev1=1 crypto rx=0x7f222c009bb0 tx=0x7f222c007700 comp rx=0 tx=0).stop 2024-03-31T15:15:13.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:13.209+0000 7f223ae00640 1 -- 172.21.15.16:0/1116650276 shutdown_connections 2024-03-31T15:15:13.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:13.209+0000 7f223ae00640 1 --2- 172.21.15.16:0/1116650276 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f22100683b0 0x7f221006a870 unknown :-1 s=CLOSED pgs=794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:13.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:13.209+0000 7f223ae00640 1 --2- 172.21.15.16:0/1116650276 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f22341642e0 0x7f2234164760 unknown :-1 s=CLOSED pgs=1889 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:13.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:13.209+0000 7f223ae00640 1 --2- 172.21.15.16:0/1116650276 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2234150710 0x7f22341566e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:13.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:13.209+0000 7f223ae00640 1 --2- 172.21.15.16:0/1116650276 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f2218003ff0 0x7f22341501a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:13.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:13.209+0000 7f223ae00640 1 -- 172.21.15.16:0/1116650276 >> 172.21.15.16:0/1116650276 conn(0x7f2234067330 msgr2=0x7f22340d07c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:15:13.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:13.209+0000 7f223ae00640 1 -- 172.21.15.16:0/1116650276 shutdown_connections 2024-03-31T15:15:13.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:13.209+0000 7f223ae00640 1 -- 172.21.15.16:0/1116650276 wait complete. 2024-03-31T15:15:13.220 INFO:tasks.cephfs_test_runner:test_orphan_scan (tasks.cephfs.test_forward_scrub.TestForwardScrub) ... ok 2024-03-31T15:15:13.220 INFO:tasks.cephfs_test_runner: 2024-03-31T15:15:13.220 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2024-03-31T15:15:13.220 INFO:tasks.cephfs_test_runner:Ran 7 tests in 665.193s 2024-03-31T15:15:13.220 INFO:tasks.cephfs_test_runner: 2024-03-31T15:15:13.220 INFO:tasks.cephfs_test_runner:OK 2024-03-31T15:15:13.221 DEBUG:teuthology.run_tasks:Unwinding manager cephfs_test_runner 2024-03-31T15:15:13.230 DEBUG:teuthology.run_tasks:Unwinding manager ceph-fuse 2024-03-31T15:15:13.252 INFO:tasks.ceph_fuse:Unmounting ceph-fuse clients... 2024-03-31T15:15:13.252 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-03-31T15:15:13.252 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-31T15:15:13.261 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16357708k,nr_inodes=4089427,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275996k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=6539 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275992k,nr_inodes=818998,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275996k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-31T15:15:13.261 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-03-31T15:15:13.261 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-31T15:15:13.312 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16357708k,nr_inodes=4089427,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275996k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=6539 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275992k,nr_inodes=818998,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275996k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-31T15:15:13.312 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-03-31T15:15:13.312 DEBUG:teuthology.orchestra.run.smithi089:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-31T15:15:13.322 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16354452k,nr_inodes=4088613,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275344k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=901 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275340k,nr_inodes=818835,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275344k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-31T15:15:13.322 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-03-31T15:15:13.322 DEBUG:teuthology.orchestra.run.smithi089:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-31T15:15:13.372 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16354452k,nr_inodes=4088613,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275344k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=901 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275340k,nr_inodes=818835,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275344k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-31T15:15:13.373 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:15:13.373 DEBUG:teuthology.orchestra.run.smithi016:> ip netns list 2024-03-31T15:15:13.382 INFO:teuthology.orchestra.run.smithi016.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-31T15:15:13.382 INFO:teuthology.orchestra.run.smithi016.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-31T15:15:13.382 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:15:13.382 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-31T15:15:13.445 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:15:13.445 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-31T15:15:13.510 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:15:13.510 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip link delete ceph-brx 2024-03-31T15:15:13.683 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:15:13.683 DEBUG:teuthology.orchestra.run.smithi089:> ip netns list 2024-03-31T15:15:13.693 INFO:teuthology.orchestra.run.smithi089.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-03-31T15:15:13.693 INFO:teuthology.orchestra.run.smithi089.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2024-03-31T15:15:13.694 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:15:13.694 DEBUG:teuthology.orchestra.run.smithi089:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-31T15:15:13.759 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:15:13.759 DEBUG:teuthology.orchestra.run.smithi089:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.3 2024-03-31T15:15:13.829 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:15:13.884 DEBUG:teuthology.orchestra.run.smithi089:> sudo ip link delete ceph-brx 2024-03-31T15:15:14.019 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-03-31T15:15:14.028 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-03-31T15:15:14.028 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-31T15:15:14.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:14.181+0000 7ff039c00640 1 Processor -- start 2024-03-31T15:15:14.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:14.181+0000 7ff039c00640 1 -- start start 2024-03-31T15:15:14.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:14.181+0000 7ff039c00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7ff03415e3b0 0x7ff03409aa60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:14.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:14.181+0000 7ff039c00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff0340b1360 0x7ff03409afa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:14.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:14.181+0000 7ff039c00640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff0340d0800 con 0x7ff03406fd40 2024-03-31T15:15:14.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:14.181+0000 7ff039c00640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7ff03415e790 con 0x7ff03415e3b0 2024-03-31T15:15:14.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:14.181+0000 7ff039c00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7ff03406f5d0 con 0x7ff0340b1360 2024-03-31T15:15:14.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:14.181+0000 7ff039200640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7ff03406fd40 0x7ff0340701d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:46660/0 (socket says 172.21.15.16:46660) 2024-03-31T15:15:14.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:14.181+0000 7ff039200640 1 -- 172.21.15.16:0/4166807610 learned_addr learned my addr 172.21.15.16:0/4166807610 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:15:14.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:14.181+0000 7ff033e00640 1 --2- 172.21.15.16:0/4166807610 >> v2:172.21.15.89:3300/0 conn(0x7ff03415e3b0 0x7ff03409aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:14.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:14.181+0000 7ff033400640 1 --2- 172.21.15.16:0/4166807610 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff0340b1360 0x7ff03409afa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:14.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:14.181+0000 7ff032a00640 1 -- 172.21.15.16:0/4166807610 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1040249480 0 0) 0x7ff0340d0800 con 0x7ff03406fd40 2024-03-31T15:15:14.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:14.181+0000 7ff032a00640 1 -- 172.21.15.16:0/4166807610 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff018003530 con 0x7ff03406fd40 2024-03-31T15:15:14.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:14.181+0000 7ff032a00640 1 -- 172.21.15.16:0/4166807610 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 636279913 0 0) 0x7ff018003530 con 0x7ff03406fd40 2024-03-31T15:15:14.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:14.181+0000 7ff032a00640 1 -- 172.21.15.16:0/4166807610 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff0340b1360 msgr2=0x7ff03409afa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:15:14.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:14.181+0000 7ff032a00640 1 --2- 172.21.15.16:0/4166807610 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff0340b1360 0x7ff03409afa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:14.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:14.181+0000 7ff032a00640 1 -- 172.21.15.16:0/4166807610 >> v2:172.21.15.89:3300/0 conn(0x7ff03415e3b0 msgr2=0x7ff03409aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:15:14.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:14.181+0000 7ff032a00640 1 --2- 172.21.15.16:0/4166807610 >> v2:172.21.15.89:3300/0 conn(0x7ff03415e3b0 0x7ff03409aa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:14.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:14.181+0000 7ff032a00640 1 -- 172.21.15.16:0/4166807610 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0340d0ae0 con 0x7ff03406fd40 2024-03-31T15:15:14.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:14.181+0000 7ff032a00640 1 -- 172.21.15.16:0/4166807610 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 214807623 0 0) 0x7ff02c001a00 con 0x7ff03406fd40 2024-03-31T15:15:14.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:14.181+0000 7ff032a00640 1 -- 172.21.15.16:0/4166807610 >> v1:172.21.15.16:6789/0 conn(0x7ff03406fd40 legacy=0x7ff0340701d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:14.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:14.181+0000 7ff032a00640 1 --2- 172.21.15.16:0/4166807610 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff0180039c0 0x7ff018003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:14.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:14.181+0000 7ff032a00640 1 --2- 172.21.15.16:0/4166807610 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff0340b1360 0x7ff018004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:14.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:14.181+0000 7ff032a00640 1 --2- 172.21.15.16:0/4166807610 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff018004d50 0x7ff0180051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:14.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:14.181+0000 7ff032a00640 1 -- 172.21.15.16:0/4166807610 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7ff03415e790 con 0x7ff018004d50 2024-03-31T15:15:14.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:14.181+0000 7ff032a00640 1 -- 172.21.15.16:0/4166807610 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7ff03406f5d0 con 0x7ff0180039c0 2024-03-31T15:15:14.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:14.181+0000 7ff032a00640 1 -- 172.21.15.16:0/4166807610 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7ff01800b9d0 con 0x7ff0340b1360 2024-03-31T15:15:14.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:14.181+0000 7ff039200640 1 --2- 172.21.15.16:0/4166807610 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff018004d50 0x7ff0180051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:14.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:14.181+0000 7ff033e00640 1 --2- 172.21.15.16:0/4166807610 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff0180039c0 0x7ff018003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:14.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:14.181+0000 7ff033400640 1 --2- 172.21.15.16:0/4166807610 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff0340b1360 0x7ff018004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:14.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:14.181+0000 7ff039200640 1 -- 172.21.15.16:0/4166807610 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff0340b1360 msgr2=0x7ff018004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:15:14.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:14.181+0000 7ff039200640 1 --2- 172.21.15.16:0/4166807610 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff0340b1360 0x7ff018004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:14.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:14.181+0000 7ff039200640 1 -- 172.21.15.16:0/4166807610 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff0180039c0 msgr2=0x7ff018003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:15:14.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:14.181+0000 7ff039200640 1 --2- 172.21.15.16:0/4166807610 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff0180039c0 0x7ff018003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:14.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:14.181+0000 7ff039200640 1 -- 172.21.15.16:0/4166807610 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff018003530 con 0x7ff018004d50 2024-03-31T15:15:14.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:14.185+0000 7ff039200640 1 --2- 172.21.15.16:0/4166807610 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff018004d50 0x7ff0180051d0 secure :-1 s=READY pgs=1891 cs=0 l=1 rev1=1 crypto rx=0x7ff02c00a870 tx=0x7ff02c01d3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c9520c7f2e42d8f server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:15:14.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:14.185+0000 7ff032a00640 1 -- 172.21.15.16:0/4166807610 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff02c00a4c0 con 0x7ff018004d50 2024-03-31T15:15:14.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:14.185+0000 7ff032a00640 1 -- 172.21.15.16:0/4166807610 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff02c0072d0 con 0x7ff018004d50 2024-03-31T15:15:14.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:14.185+0000 7ff032a00640 1 -- 172.21.15.16:0/4166807610 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff02c0076c0 con 0x7ff018004d50 2024-03-31T15:15:14.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:14.185+0000 7ff039c00640 1 -- 172.21.15.16:0/4166807610 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff018004d50 msgr2=0x7ff0180051d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:14.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:14.185+0000 7ff039c00640 1 --2- 172.21.15.16:0/4166807610 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff018004d50 0x7ff0180051d0 secure :-1 s=READY pgs=1891 cs=0 l=1 rev1=1 crypto rx=0x7ff02c00a870 tx=0x7ff02c01d3a0 comp rx=0 tx=0).stop 2024-03-31T15:15:14.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:14.185+0000 7ff039200640 1 -- 172.21.15.16:0/4166807610 reap_dead start 2024-03-31T15:15:14.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:14.185+0000 7ff039c00640 1 -- 172.21.15.16:0/4166807610 shutdown_connections 2024-03-31T15:15:14.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:14.185+0000 7ff039c00640 1 -- 172.21.15.16:0/4166807610 >> 172.21.15.16:0/4166807610 conn(0x7ff0340b39c0 msgr2=0x7ff0340b3dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:15:14.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:14.185+0000 7ff039c00640 1 -- 172.21.15.16:0/4166807610 shutdown_connections 2024-03-31T15:15:14.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:14.185+0000 7ff039c00640 1 -- 172.21.15.16:0/4166807610 wait complete. 2024-03-31T15:15:14.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:14.189+0000 7ff039c00640 1 Processor -- start 2024-03-31T15:15:14.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:14.189+0000 7ff039c00640 1 -- start start 2024-03-31T15:15:14.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:14.189+0000 7ff039c00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff034156f60 0x7ff03415fde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:14.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:14.189+0000 7ff039c00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff0341574e0 0x7ff034160c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:14.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:14.189+0000 7ff039c00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff034157940 0x7ff0341611d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:14.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:14.189+0000 7ff039c00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7ff0340d04b0 con 0x7ff0341574e0 2024-03-31T15:15:14.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:14.189+0000 7ff039c00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7ff03406f820 con 0x7ff034156f60 2024-03-31T15:15:14.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:14.189+0000 7ff039c00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7ff0340cdba0 con 0x7ff034157940 2024-03-31T15:15:14.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:14.189+0000 7ff033400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff0341574e0 0x7ff034160c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:14.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:14.189+0000 7ff033400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff0341574e0 0x7ff034160c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46270/0 (socket says 172.21.15.16:46270) 2024-03-31T15:15:14.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:14.189+0000 7ff033400640 1 -- 172.21.15.16:0/3556973955 learned_addr learned my addr 172.21.15.16:0/3556973955 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:15:14.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:14.189+0000 7ff033e00640 1 --2- 172.21.15.16:0/3556973955 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff034156f60 0x7ff03415fde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:14.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:14.189+0000 7ff039200640 1 --2- 172.21.15.16:0/3556973955 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff034157940 0x7ff0341611d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:14.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:14.189+0000 7ff033400640 1 -- 172.21.15.16:0/3556973955 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff034157940 msgr2=0x7ff0341611d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:14.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:14.189+0000 7ff033400640 1 --2- 172.21.15.16:0/3556973955 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff034157940 0x7ff0341611d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:14.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:14.189+0000 7ff033400640 1 -- 172.21.15.16:0/3556973955 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff034156f60 msgr2=0x7ff03415fde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:14.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:14.189+0000 7ff033400640 1 --2- 172.21.15.16:0/3556973955 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff034156f60 0x7ff03415fde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:14.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:14.189+0000 7ff033400640 1 -- 172.21.15.16:0/3556973955 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff03415ed40 con 0x7ff0341574e0 2024-03-31T15:15:14.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:14.189+0000 7ff033400640 1 --2- 172.21.15.16:0/3556973955 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff0341574e0 0x7ff034160c90 secure :-1 s=READY pgs=1892 cs=0 l=1 rev1=1 crypto rx=0x7ff0200119d0 tx=0x7ff020011dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:15:14.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:14.189+0000 7ff02be00640 1 -- 172.21.15.16:0/3556973955 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff02000fa90 con 0x7ff0341574e0 2024-03-31T15:15:14.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:14.189+0000 7ff02be00640 1 -- 172.21.15.16:0/3556973955 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff02000fc30 con 0x7ff0341574e0 2024-03-31T15:15:14.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:14.189+0000 7ff039c00640 1 -- 172.21.15.16:0/3556973955 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff03415f020 con 0x7ff0341574e0 2024-03-31T15:15:14.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:14.189+0000 7ff02be00640 1 -- 172.21.15.16:0/3556973955 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff020014440 con 0x7ff0341574e0 2024-03-31T15:15:14.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:14.189+0000 7ff039c00640 1 -- 172.21.15.16:0/3556973955 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff03415f5e0 con 0x7ff0341574e0 2024-03-31T15:15:14.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:14.193+0000 7ff039c00640 1 -- 172.21.15.16:0/3556973955 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff03404f8c0 con 0x7ff0341574e0 2024-03-31T15:15:14.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:14.193+0000 7ff02be00640 1 -- 172.21.15.16:0/3556973955 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7ff020012040 con 0x7ff0341574e0 2024-03-31T15:15:14.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:14.193+0000 7ff02be00640 1 --2- 172.21.15.16:0/3556973955 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7ff0140683b0 0x7ff01406a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:14.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:14.193+0000 7ff02be00640 1 -- 172.21.15.16:0/3556973955 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(116..116 src has 1..116) ==== 5115+0+0 (secure 0 0 0) 0x7ff020086d00 con 0x7ff0341574e0 2024-03-31T15:15:14.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:14.193+0000 7ff033e00640 1 --2- 172.21.15.16:0/3556973955 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7ff0140683b0 0x7ff01406a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:14.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:14.193+0000 7ff033e00640 1 --2- 172.21.15.16:0/3556973955 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7ff0140683b0 0x7ff01406a870 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7ff024009be0 tx=0x7ff024009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:15:14.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:14.197+0000 7ff02be00640 1 -- 172.21.15.16:0/3556973955 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7ff020019030 con 0x7ff0341574e0 2024-03-31T15:15:14.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:14.357+0000 7ff039c00640 1 -- 172.21.15.16:0/3556973955 --> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7ff034068ae0 con 0x7ff0140683b0 2024-03-31T15:15:14.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:14.361+0000 7ff02be00640 1 -- 172.21.15.16:0/3556973955 <== mgr.4100 v2:172.21.15.89:6832/1549407273 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+13475 (secure 0 0 0) 0x7ff034068ae0 con 0x7ff0140683b0 2024-03-31T15:15:14.350 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:15:14.350 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-03-31T15:15:14.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:14.361+0000 7ff039c00640 1 -- 172.21.15.16:0/3556973955 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7ff0140683b0 msgr2=0x7ff01406a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:14.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:14.361+0000 7ff039c00640 1 --2- 172.21.15.16:0/3556973955 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7ff0140683b0 0x7ff01406a870 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7ff024009be0 tx=0x7ff024009190 comp rx=0 tx=0).stop 2024-03-31T15:15:14.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:14.361+0000 7ff039c00640 1 -- 172.21.15.16:0/3556973955 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff0341574e0 msgr2=0x7ff034160c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:14.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:14.361+0000 7ff039c00640 1 --2- 172.21.15.16:0/3556973955 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff0341574e0 0x7ff034160c90 secure :-1 s=READY pgs=1892 cs=0 l=1 rev1=1 crypto rx=0x7ff0200119d0 tx=0x7ff020011dd0 comp rx=0 tx=0).stop 2024-03-31T15:15:14.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:14.361+0000 7ff039c00640 1 -- 172.21.15.16:0/3556973955 shutdown_connections 2024-03-31T15:15:14.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:14.361+0000 7ff039c00640 1 --2- 172.21.15.16:0/3556973955 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7ff0140683b0 0x7ff01406a870 unknown :-1 s=CLOSED pgs=795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:14.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:14.361+0000 7ff039c00640 1 --2- 172.21.15.16:0/3556973955 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff034157940 0x7ff0341611d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:14.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:14.361+0000 7ff039c00640 1 --2- 172.21.15.16:0/3556973955 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff0341574e0 0x7ff034160c90 unknown :-1 s=CLOSED pgs=1892 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:14.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:14.361+0000 7ff039c00640 1 --2- 172.21.15.16:0/3556973955 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff034156f60 0x7ff03415fde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:14.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:14.361+0000 7ff039c00640 1 -- 172.21.15.16:0/3556973955 >> 172.21.15.16:0/3556973955 conn(0x7ff0340b39c0 msgr2=0x7ff0340cdf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:15:14.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:14.361+0000 7ff039c00640 1 -- 172.21.15.16:0/3556973955 shutdown_connections 2024-03-31T15:15:14.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:14.361+0000 7ff039c00640 1 -- 172.21.15.16:0/3556973955 wait complete. 2024-03-31T15:15:14.372 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":464,"stamp":"2024-03-31T15:15:12.709405+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":5,"kb":754974720,"kb_used":282196,"kb_used_data":28604,"kb_used_omap":12,"kb_used_meta":253491,"kb_avail":754692524,"statfs":{"total":773094113280,"available":772805144576,"internally_reserved":0,"allocated":29290496,"data_stored":22864455,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":259575344},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":18,"apply_latency_ms":18,"commit_latency_ns":18000000,"apply_latency_ns":18000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-8390584,"num_objects":-30,"num_object_clones":0,"num_object_copies":-60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-30,"num_whiteouts":0,"num_read":-258,"num_read_kb":-262,"num_write":-175,"num_write_kb":-8293,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-17,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001517"},"pg_stats":[{"pgid":"1.0","version":"16'117","reported_seq":343,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-03-31T15:14:53.178690+0000","last_change":"2024-03-31T15:03:24.329890+0000","last_active":"2024-03-31T15:14:53.178690+0000","last_peered":"2024-03-31T15:14:53.178690+0000","last_clean":"2024-03-31T15:14:53.178690+0000","last_became_active":"2024-03-31T15:03:24.328882+0000","last_became_peered":"2024-03-31T15:03:24.328882+0000","last_unstale":"2024-03-31T15:14:53.178690+0000","last_undegraded":"2024-03-31T15:14:53.178690+0000","last_fullsized":"2024-03-31T15:14:53.178690+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:22.885535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:22.885535+0000","last_clean_scrub_stamp":"2024-03-31T15:03:22.885535+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:18:29.004296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509586,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33128,"kb_used_data":1952,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338712,"statfs":{"total":96636764160,"available":96602841088,"internally_reserved":0,"allocated":1998848,"data_stored":1199429,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509585,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32556,"kb_used_data":1380,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339284,"statfs":{"total":96636764160,"available":96603426816,"internally_reserved":0,"allocated":1413120,"data_stored":609061,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":15,"seq":64424509584,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32544,"kb_used_data":1364,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339296,"statfs":{"total":96636764160,"available":96603439104,"internally_reserved":0,"allocated":1396736,"data_stored":608445,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509583,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36708,"kb_used_data":1368,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335132,"statfs":{"total":96636764160,"available":96599175168,"internally_reserved":0,"allocated":1400832,"data_stored":608450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542290,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":36672,"kb_used_data":5492,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94335168,"statfs":{"total":96636764160,"available":96599212032,"internally_reserved":0,"allocated":5623808,"data_stored":4815300,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542290,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":36672,"kb_used_data":5492,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94335168,"statfs":{"total":96636764160,"available":96599212032,"internally_reserved":0,"allocated":5623808,"data_stored":4812088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542290,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":36660,"kb_used_data":5480,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94335180,"statfs":{"total":96636764160,"available":96599224320,"internally_reserved":0,"allocated":5611520,"data_stored":4809184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542290,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37256,"kb_used_data":6076,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94334584,"statfs":{"total":96636764160,"available":96598614016,"internally_reserved":0,"allocated":6221824,"data_stored":5402498,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-31T15:15:14.373 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-31T15:15:14.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:14.525+0000 7f7722e00640 1 Processor -- start 2024-03-31T15:15:14.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:14.525+0000 7f7722e00640 1 -- start start 2024-03-31T15:15:14.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:14.525+0000 7f7722e00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f771c0d1280 0x7f771c0d1660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:14.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:14.525+0000 7f7722e00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f771c0b9c60 0x7f771c0b5d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:14.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:14.525+0000 7f7722e00640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f771c0d1ba0 con 0x7f771c0b6290 2024-03-31T15:15:14.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:14.525+0000 7f7722e00640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f771c0d26a0 con 0x7f771c0d1280 2024-03-31T15:15:14.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:14.525+0000 7f7722e00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f771c0d3990 con 0x7f771c0b9c60 2024-03-31T15:15:14.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:14.525+0000 7f7713400640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f771c0b9c60 0x7f771c0b5d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:14.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:14.525+0000 7f7713400640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f771c0b9c60 0x7f771c0b5d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3301/0 says I am v2:172.21.15.16:56186/0 (socket says 172.21.15.16:56186) 2024-03-31T15:15:14.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:14.525+0000 7f7713400640 1 -- 172.21.15.16:0/409541477 learned_addr learned my addr 172.21.15.16:0/409541477 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:15:14.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:14.525+0000 7f771be00640 1 --2- 172.21.15.16:0/409541477 >> v2:172.21.15.89:3300/0 conn(0x7f771c0d1280 0x7f771c0d1660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:14.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:14.525+0000 7f7713400640 1 -- 172.21.15.16:0/409541477 >> v2:172.21.15.89:3300/0 conn(0x7f771c0d1280 msgr2=0x7f771c0d1660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:15:14.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:14.525+0000 7f7713400640 1 --2- 172.21.15.16:0/409541477 >> v2:172.21.15.89:3300/0 conn(0x7f771c0d1280 0x7f771c0d1660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:14.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:14.525+0000 7f771b400640 1 -- 172.21.15.16:0/409541477 <== mon.1 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1170462998 0 0) 0x7f771c0d1ba0 con 0x7f771c0b6290 2024-03-31T15:15:14.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:14.525+0000 7f7713400640 1 -- 172.21.15.16:0/409541477 >> v1:172.21.15.89:6789/0 conn(0x7f771c0b6290 legacy=0x7f771c0b7470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:14.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:14.525+0000 7f7713400640 1 -- 172.21.15.16:0/409541477 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f771c0d28f0 con 0x7f771c0b9c60 2024-03-31T15:15:14.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:14.529+0000 7f7713400640 1 --2- 172.21.15.16:0/409541477 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f771c0b9c60 0x7f771c0b5d50 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f7704009870 tx=0x7f770401c4d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=44bc03b2943eb633 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:15:14.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:14.529+0000 7f771b400640 1 -- 172.21.15.16:0/409541477 <== mon.2 v2:172.21.15.89:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77040b1070 con 0x7f771c0b9c60 2024-03-31T15:15:14.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:14.529+0000 7f771b400640 1 -- 172.21.15.16:0/409541477 <== mon.2 v2:172.21.15.89:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f77040ac620 con 0x7f771c0b9c60 2024-03-31T15:15:14.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:14.529+0000 7f771b400640 1 -- 172.21.15.16:0/409541477 <== mon.2 v2:172.21.15.89:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77040ac9f0 con 0x7f771c0b9c60 2024-03-31T15:15:14.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:14.529+0000 7f7722e00640 1 -- 172.21.15.16:0/409541477 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f771c0b9c60 msgr2=0x7f771c0b5d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:14.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:14.529+0000 7f7722e00640 1 --2- 172.21.15.16:0/409541477 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f771c0b9c60 0x7f771c0b5d50 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f7704009870 tx=0x7f770401c4d0 comp rx=0 tx=0).stop 2024-03-31T15:15:14.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:14.529+0000 7f7722e00640 1 -- 172.21.15.16:0/409541477 shutdown_connections 2024-03-31T15:15:14.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:14.529+0000 7f7722e00640 1 --2- 172.21.15.16:0/409541477 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f771c0b9c60 0x7f771c0b5d50 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:14.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:14.529+0000 7f7722e00640 1 --2- 172.21.15.16:0/409541477 >> v2:172.21.15.89:3300/0 conn(0x7f771c0d1280 0x7f771c0d1660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:14.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:14.529+0000 7f7722e00640 1 -- 172.21.15.16:0/409541477 >> 172.21.15.16:0/409541477 conn(0x7f771c067730 msgr2=0x7f771c067b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:15:14.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:14.529+0000 7f7722e00640 1 -- 172.21.15.16:0/409541477 shutdown_connections 2024-03-31T15:15:14.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:14.529+0000 7f7722e00640 1 -- 172.21.15.16:0/409541477 wait complete. 2024-03-31T15:15:14.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:14.533+0000 7f7722e00640 1 Processor -- start 2024-03-31T15:15:14.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:14.533+0000 7f7722e00640 1 -- start start 2024-03-31T15:15:14.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:14.533+0000 7f7722e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f771c0b6290 0x7f771c156a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:14.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:14.533+0000 7f7722e00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f771c0b9c60 0x7f771c0cb3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:14.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:14.533+0000 7f7722e00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f771c0d1280 0x7f771c158e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:14.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:14.533+0000 7f7722e00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f771c0b6ae0 con 0x7f771c0b6290 2024-03-31T15:15:14.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:14.533+0000 7f7722e00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f771c161d80 con 0x7f771c0b9c60 2024-03-31T15:15:14.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:14.533+0000 7f7722e00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f771c0d3e90 con 0x7f771c0d1280 2024-03-31T15:15:14.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:14.533+0000 7f771be00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f771c0b6290 0x7f771c156a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:14.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:14.533+0000 7f771be00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f771c0b6290 0x7f771c156a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46282/0 (socket says 172.21.15.16:46282) 2024-03-31T15:15:14.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:14.533+0000 7f771be00640 1 -- 172.21.15.16:0/219774090 learned_addr learned my addr 172.21.15.16:0/219774090 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:15:14.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:14.533+0000 7f7720e00640 1 --2- 172.21.15.16:0/219774090 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f771c0d1280 0x7f771c158e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:14.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:14.533+0000 7f7713400640 1 --2- 172.21.15.16:0/219774090 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f771c0b9c60 0x7f771c0cb3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:14.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:14.533+0000 7f771be00640 1 -- 172.21.15.16:0/219774090 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f771c0d1280 msgr2=0x7f771c158e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:14.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:14.533+0000 7f771be00640 1 --2- 172.21.15.16:0/219774090 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f771c0d1280 0x7f771c158e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:14.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:14.533+0000 7f771be00640 1 -- 172.21.15.16:0/219774090 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f771c0b9c60 msgr2=0x7f771c0cb3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:14.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:14.533+0000 7f771be00640 1 --2- 172.21.15.16:0/219774090 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f771c0b9c60 0x7f771c0cb3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:14.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:14.533+0000 7f771be00640 1 -- 172.21.15.16:0/219774090 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f771c0cbdf0 con 0x7f771c0b6290 2024-03-31T15:15:14.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:14.533+0000 7f7720e00640 1 --2- 172.21.15.16:0/219774090 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f771c0d1280 0x7f771c158e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:15:14.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:14.533+0000 7f771be00640 1 --2- 172.21.15.16:0/219774090 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f771c0b6290 0x7f771c156a50 secure :-1 s=READY pgs=1893 cs=0 l=1 rev1=1 crypto rx=0x7f770c00e870 tx=0x7f770c00ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:15:14.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:14.533+0000 7f7718c00640 1 -- 172.21.15.16:0/219774090 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f770c017070 con 0x7f771c0b6290 2024-03-31T15:15:14.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:14.533+0000 7f7718c00640 1 -- 172.21.15.16:0/219774090 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f770c00c940 con 0x7f771c0b6290 2024-03-31T15:15:14.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:14.533+0000 7f7718c00640 1 -- 172.21.15.16:0/219774090 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f770c00ccf0 con 0x7f771c0b6290 2024-03-31T15:15:14.525 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:14.533+0000 7f7722e00640 1 -- 172.21.15.16:0/219774090 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f771c0cc0d0 con 0x7f771c0b6290 2024-03-31T15:15:14.525 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:14.533+0000 7f7722e00640 1 -- 172.21.15.16:0/219774090 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f771c0cc690 con 0x7f771c0b6290 2024-03-31T15:15:14.525 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:14.533+0000 7f7722e00640 1 -- 172.21.15.16:0/219774090 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f771c069950 con 0x7f771c0b6290 2024-03-31T15:15:14.525 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:14.537+0000 7f7718c00640 1 -- 172.21.15.16:0/219774090 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f770c01a070 con 0x7f771c0b6290 2024-03-31T15:15:14.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:14.537+0000 7f7718c00640 1 --2- 172.21.15.16:0/219774090 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f76f8068380 0x7f76f806a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:14.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:14.537+0000 7f7718c00640 1 -- 172.21.15.16:0/219774090 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(116..116 src has 1..116) ==== 5115+0+0 (secure 0 0 0) 0x7f770c086950 con 0x7f771c0b6290 2024-03-31T15:15:14.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:14.537+0000 7f7713400640 1 --2- 172.21.15.16:0/219774090 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f76f8068380 0x7f76f806a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:14.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:14.537+0000 7f7713400640 1 --2- 172.21.15.16:0/219774090 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f76f8068380 0x7f76f806a840 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7f77040ae000 tx=0x7f77040a7000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:15:14.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:14.541+0000 7f7718c00640 1 -- 172.21.15.16:0/219774090 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f770c056b00 con 0x7f771c0b6290 2024-03-31T15:15:14.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:14.705+0000 7f7722e00640 1 -- 172.21.15.16:0/219774090 --> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f771c159ca0 con 0x7f76f8068380 2024-03-31T15:15:14.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:14.705+0000 7f7718c00640 1 -- 172.21.15.16:0/219774090 <== mgr.4100 v2:172.21.15.89:6832/1549407273 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+13522 (secure 0 0 0) 0x7f771c159ca0 con 0x7f76f8068380 2024-03-31T15:15:14.694 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:15:14.695 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-03-31T15:15:14.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:14.705+0000 7f7722e00640 1 -- 172.21.15.16:0/219774090 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f76f8068380 msgr2=0x7f76f806a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:14.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:14.705+0000 7f7722e00640 1 --2- 172.21.15.16:0/219774090 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f76f8068380 0x7f76f806a840 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7f77040ae000 tx=0x7f77040a7000 comp rx=0 tx=0).stop 2024-03-31T15:15:14.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:14.705+0000 7f7722e00640 1 -- 172.21.15.16:0/219774090 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f771c0b6290 msgr2=0x7f771c156a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:14.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:14.705+0000 7f7722e00640 1 --2- 172.21.15.16:0/219774090 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f771c0b6290 0x7f771c156a50 secure :-1 s=READY pgs=1893 cs=0 l=1 rev1=1 crypto rx=0x7f770c00e870 tx=0x7f770c00ec70 comp rx=0 tx=0).stop 2024-03-31T15:15:14.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:14.705+0000 7f7722e00640 1 -- 172.21.15.16:0/219774090 shutdown_connections 2024-03-31T15:15:14.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:14.705+0000 7f7722e00640 1 --2- 172.21.15.16:0/219774090 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f76f8068380 0x7f76f806a840 unknown :-1 s=CLOSED pgs=796 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:14.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:14.705+0000 7f7722e00640 1 --2- 172.21.15.16:0/219774090 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f771c0d1280 0x7f771c158e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:14.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:14.705+0000 7f7722e00640 1 --2- 172.21.15.16:0/219774090 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f771c0b9c60 0x7f771c0cb3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:14.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:14.705+0000 7f7722e00640 1 --2- 172.21.15.16:0/219774090 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f771c0b6290 0x7f771c156a50 unknown :-1 s=CLOSED pgs=1893 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:14.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:14.705+0000 7f7722e00640 1 -- 172.21.15.16:0/219774090 >> 172.21.15.16:0/219774090 conn(0x7f771c067730 msgr2=0x7f771c15a4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:15:14.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:14.705+0000 7f7722e00640 1 -- 172.21.15.16:0/219774090 shutdown_connections 2024-03-31T15:15:14.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:14.705+0000 7f7722e00640 1 -- 172.21.15.16:0/219774090 wait complete. 2024-03-31T15:15:14.715 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":465,"stamp":"2024-03-31T15:15:14.709847+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":172,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":282356,"kb_used_data":28752,"kb_used_omap":12,"kb_used_meta":253491,"kb_avail":754692364,"statfs":{"total":773094113280,"available":772804980736,"internally_reserved":0,"allocated":29442048,"data_stored":22936478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":259575344},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":31,"apply_latency_ms":31,"commit_latency_ns":31000000,"apply_latency_ns":31000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-8392376,"num_objects":-30,"num_object_clones":0,"num_object_copies":-60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-30,"num_whiteouts":0,"num_read":-258,"num_read_kb":-262,"num_write":-179,"num_write_kb":-8296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-17,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001495"},"pg_stats":[{"pgid":"1.0","version":"16'117","reported_seq":348,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-31T15:15:12.483895+0000","last_change":"2024-03-31T15:03:24.329890+0000","last_active":"2024-03-31T15:15:12.483895+0000","last_peered":"2024-03-31T15:15:12.483895+0000","last_clean":"2024-03-31T15:15:12.483895+0000","last_became_active":"2024-03-31T15:03:24.328882+0000","last_became_peered":"2024-03-31T15:03:24.328882+0000","last_unstale":"2024-03-31T15:15:12.483895+0000","last_undegraded":"2024-03-31T15:15:12.483895+0000","last_fullsized":"2024-03-31T15:15:12.483895+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:22.885535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:22.885535+0000","last_clean_scrub_stamp":"2024-03-31T15:03:22.885535+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:18:29.004296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509587,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33168,"kb_used_data":1988,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338672,"statfs":{"total":96636764160,"available":96602800128,"internally_reserved":0,"allocated":2035712,"data_stored":1217128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509586,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32588,"kb_used_data":1408,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339252,"statfs":{"total":96636764160,"available":96603394048,"internally_reserved":0,"allocated":1441792,"data_stored":626760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":15,"seq":64424509585,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32588,"kb_used_data":1408,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339252,"statfs":{"total":96636764160,"available":96603394048,"internally_reserved":0,"allocated":1441792,"data_stored":626760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509584,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":36752,"kb_used_data":1408,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335088,"statfs":{"total":96636764160,"available":96599130112,"internally_reserved":0,"allocated":1441792,"data_stored":626760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542290,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":36672,"kb_used_data":5492,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94335168,"statfs":{"total":96636764160,"available":96599212032,"internally_reserved":0,"allocated":5623808,"data_stored":4815300,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542290,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":36672,"kb_used_data":5492,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94335168,"statfs":{"total":96636764160,"available":96599212032,"internally_reserved":0,"allocated":5623808,"data_stored":4812088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542290,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":36660,"kb_used_data":5480,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94335180,"statfs":{"total":96636764160,"available":96599224320,"internally_reserved":0,"allocated":5611520,"data_stored":4809184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542290,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37256,"kb_used_data":6076,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94334584,"statfs":{"total":96636764160,"available":96598614016,"internally_reserved":0,"allocated":6221824,"data_stored":5402498,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-31T15:15:14.715 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-03-31T15:15:14.716 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-31T15:15:14.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:14.865+0000 7fa367a00640 1 Processor -- start 2024-03-31T15:15:14.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:14.865+0000 7fa367a00640 1 -- start start 2024-03-31T15:15:14.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:14.869+0000 7fa367a00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fa36006dbc0 0x7fa36015f650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:14.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:14.869+0000 7fa367a00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fa3600cb440 0x7fa36006f920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:14.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:14.869+0000 7fa367a00640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa3600cbe90 con 0x7fa3600cb8c0 2024-03-31T15:15:14.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:14.869+0000 7fa367a00640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7fa36015fb90 con 0x7fa36006dbc0 2024-03-31T15:15:14.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:14.869+0000 7fa367a00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fa36015f1c0 con 0x7fa3600cb440 2024-03-31T15:15:14.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:14.869+0000 7fa365a00640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fa3600cb8c0 0x7fa360162180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:46672/0 (socket says 172.21.15.16:46672) 2024-03-31T15:15:14.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:14.869+0000 7fa365a00640 1 -- 172.21.15.16:0/4008565908 learned_addr learned my addr 172.21.15.16:0/4008565908 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:15:14.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:14.869+0000 7fa365000640 1 --2- 172.21.15.16:0/4008565908 >> v2:172.21.15.89:3300/0 conn(0x7fa36006dbc0 0x7fa36015f650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:14.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:14.869+0000 7fa357e00640 1 --2- 172.21.15.16:0/4008565908 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fa3600cb440 0x7fa36006f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:14.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:14.869+0000 7fa35fe00640 1 -- 172.21.15.16:0/4008565908 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 413177128 0 0) 0x7fa3600cbe90 con 0x7fa3600cb8c0 2024-03-31T15:15:14.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:14.869+0000 7fa35fe00640 1 -- 172.21.15.16:0/4008565908 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa344003530 con 0x7fa3600cb8c0 2024-03-31T15:15:14.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:14.869+0000 7fa35fe00640 1 -- 172.21.15.16:0/4008565908 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2525991703 0 0) 0x7fa344003530 con 0x7fa3600cb8c0 2024-03-31T15:15:14.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:14.869+0000 7fa35fe00640 1 -- 172.21.15.16:0/4008565908 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fa3600cb440 msgr2=0x7fa36006f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:15:14.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:14.869+0000 7fa35fe00640 1 --2- 172.21.15.16:0/4008565908 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fa3600cb440 0x7fa36006f920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:14.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:14.869+0000 7fa35fe00640 1 -- 172.21.15.16:0/4008565908 >> v2:172.21.15.89:3300/0 conn(0x7fa36006dbc0 msgr2=0x7fa36015f650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:15:14.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:14.869+0000 7fa35fe00640 1 --2- 172.21.15.16:0/4008565908 >> v2:172.21.15.89:3300/0 conn(0x7fa36006dbc0 0x7fa36015f650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:14.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:14.869+0000 7fa35fe00640 1 -- 172.21.15.16:0/4008565908 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa3600cc170 con 0x7fa3600cb8c0 2024-03-31T15:15:14.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:14.869+0000 7fa35fe00640 1 -- 172.21.15.16:0/4008565908 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 214807623 0 0) 0x7fa358001a00 con 0x7fa3600cb8c0 2024-03-31T15:15:14.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:14.869+0000 7fa35fe00640 1 -- 172.21.15.16:0/4008565908 >> v1:172.21.15.16:6789/0 conn(0x7fa3600cb8c0 legacy=0x7fa360162180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:14.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:14.869+0000 7fa35fe00640 1 --2- 172.21.15.16:0/4008565908 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fa3600cb440 0x7fa3440039c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:14.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:14.869+0000 7fa35fe00640 1 --2- 172.21.15.16:0/4008565908 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa344003f40 0x7fa344004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:14.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:14.869+0000 7fa35fe00640 1 --2- 172.21.15.16:0/4008565908 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa344004d50 0x7fa3440051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:14.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:14.869+0000 7fa35fe00640 1 -- 172.21.15.16:0/4008565908 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fa36015fb90 con 0x7fa344003f40 2024-03-31T15:15:14.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:14.869+0000 7fa35fe00640 1 -- 172.21.15.16:0/4008565908 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fa36015f1c0 con 0x7fa344004d50 2024-03-31T15:15:14.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:14.869+0000 7fa35fe00640 1 -- 172.21.15.16:0/4008565908 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fa34400ba20 con 0x7fa3600cb440 2024-03-31T15:15:14.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:14.869+0000 7fa357e00640 1 --2- 172.21.15.16:0/4008565908 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa344003f40 0x7fa344004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:14.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:14.869+0000 7fa365000640 1 --2- 172.21.15.16:0/4008565908 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fa3600cb440 0x7fa3440039c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:14.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:14.869+0000 7fa365a00640 1 --2- 172.21.15.16:0/4008565908 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa344004d50 0x7fa3440051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:14.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:14.869+0000 7fa357e00640 1 -- 172.21.15.16:0/4008565908 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fa3600cb440 msgr2=0x7fa3440039c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:15:14.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:14.869+0000 7fa357e00640 1 --2- 172.21.15.16:0/4008565908 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fa3600cb440 0x7fa3440039c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:14.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:14.869+0000 7fa357e00640 1 -- 172.21.15.16:0/4008565908 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa344004d50 msgr2=0x7fa3440051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:15:14.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:14.869+0000 7fa357e00640 1 --2- 172.21.15.16:0/4008565908 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa344004d50 0x7fa3440051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:14.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:14.869+0000 7fa357e00640 1 -- 172.21.15.16:0/4008565908 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa344003530 con 0x7fa344003f40 2024-03-31T15:15:14.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:14.873+0000 7fa357e00640 1 --2- 172.21.15.16:0/4008565908 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa344003f40 0x7fa344004810 secure :-1 s=READY pgs=1895 cs=0 l=1 rev1=1 crypto rx=0x7fa348009b10 tx=0x7fa34801ca40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9de6fa8856fab278 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:15:14.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:14.873+0000 7fa35fe00640 1 -- 172.21.15.16:0/4008565908 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3480b2070 con 0x7fa344003f40 2024-03-31T15:15:14.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:14.873+0000 7fa35fe00640 1 -- 172.21.15.16:0/4008565908 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa3480a5630 con 0x7fa344003f40 2024-03-31T15:15:14.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:14.873+0000 7fa35fe00640 1 -- 172.21.15.16:0/4008565908 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3480096b0 con 0x7fa344003f40 2024-03-31T15:15:14.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:14.873+0000 7fa367a00640 1 -- 172.21.15.16:0/4008565908 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa344003f40 msgr2=0x7fa344004810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:14.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:14.873+0000 7fa367a00640 1 --2- 172.21.15.16:0/4008565908 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa344003f40 0x7fa344004810 secure :-1 s=READY pgs=1895 cs=0 l=1 rev1=1 crypto rx=0x7fa348009b10 tx=0x7fa34801ca40 comp rx=0 tx=0).stop 2024-03-31T15:15:14.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:14.873+0000 7fa365a00640 1 -- 172.21.15.16:0/4008565908 reap_dead start 2024-03-31T15:15:14.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:14.873+0000 7fa367a00640 1 -- 172.21.15.16:0/4008565908 shutdown_connections 2024-03-31T15:15:14.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:14.873+0000 7fa367a00640 1 -- 172.21.15.16:0/4008565908 >> 172.21.15.16:0/4008565908 conn(0x7fa360067330 msgr2=0x7fa360067740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:15:14.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:14.873+0000 7fa367a00640 1 -- 172.21.15.16:0/4008565908 shutdown_connections 2024-03-31T15:15:14.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:14.873+0000 7fa367a00640 1 -- 172.21.15.16:0/4008565908 wait complete. 2024-03-31T15:15:14.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:14.877+0000 7fa367a00640 1 Processor -- start 2024-03-31T15:15:14.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:14.877+0000 7fa367a00640 1 -- start start 2024-03-31T15:15:14.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:14.877+0000 7fa367a00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa344003f40 0x7fa360156990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:14.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:14.877+0000 7fa367a00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fa360145ab0 0x7fa3601643c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:14.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:14.877+0000 7fa367a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa360164900 0x7fa360154bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:14.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:14.877+0000 7fa367a00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fa36006c950 con 0x7fa360164900 2024-03-31T15:15:14.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:14.877+0000 7fa367a00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fa360162ca0 con 0x7fa344003f40 2024-03-31T15:15:14.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:14.877+0000 7fa367a00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fa36006cad0 con 0x7fa360145ab0 2024-03-31T15:15:14.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:14.877+0000 7fa365a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa360164900 0x7fa360154bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:14.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:14.877+0000 7fa365a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa360164900 0x7fa360154bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46312/0 (socket says 172.21.15.16:46312) 2024-03-31T15:15:14.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:14.877+0000 7fa365a00640 1 -- 172.21.15.16:0/402760471 learned_addr learned my addr 172.21.15.16:0/402760471 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:15:14.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:14.877+0000 7fa365a00640 1 -- 172.21.15.16:0/402760471 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fa360145ab0 msgr2=0x7fa3601643c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:14.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:14.877+0000 7fa365000640 1 --2- 172.21.15.16:0/402760471 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa344003f40 0x7fa360156990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:14.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:14.877+0000 7fa357e00640 1 --2- 172.21.15.16:0/402760471 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fa360145ab0 0x7fa3601643c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:14.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:14.877+0000 7fa365a00640 1 --2- 172.21.15.16:0/402760471 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fa360145ab0 0x7fa3601643c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:14.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:14.877+0000 7fa365a00640 1 -- 172.21.15.16:0/402760471 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa344003f40 msgr2=0x7fa360156990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:14.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:14.877+0000 7fa365a00640 1 --2- 172.21.15.16:0/402760471 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa344003f40 0x7fa360156990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:14.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:14.877+0000 7fa365a00640 1 -- 172.21.15.16:0/402760471 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa360159b70 con 0x7fa360164900 2024-03-31T15:15:14.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:14.877+0000 7fa365000640 1 --2- 172.21.15.16:0/402760471 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa344003f40 0x7fa360156990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:15:14.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:14.877+0000 7fa357e00640 1 --2- 172.21.15.16:0/402760471 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fa360145ab0 0x7fa3601643c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:15:14.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:14.877+0000 7fa365a00640 1 --2- 172.21.15.16:0/402760471 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa360164900 0x7fa360154bc0 secure :-1 s=READY pgs=1896 cs=0 l=1 rev1=1 crypto rx=0x7fa358000f90 tx=0x7fa35800e530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:15:14.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:14.877+0000 7fa35d600640 1 -- 172.21.15.16:0/402760471 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa35800ec10 con 0x7fa360164900 2024-03-31T15:15:14.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:14.877+0000 7fa35d600640 1 -- 172.21.15.16:0/402760471 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa358018070 con 0x7fa360164900 2024-03-31T15:15:14.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:14.877+0000 7fa35d600640 1 -- 172.21.15.16:0/402760471 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa358013420 con 0x7fa360164900 2024-03-31T15:15:14.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:14.877+0000 7fa367a00640 1 -- 172.21.15.16:0/402760471 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa360159e50 con 0x7fa360164900 2024-03-31T15:15:14.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:14.877+0000 7fa367a00640 1 -- 172.21.15.16:0/402760471 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa36015a410 con 0x7fa360164900 2024-03-31T15:15:14.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:14.877+0000 7fa367a00640 1 -- 172.21.15.16:0/402760471 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa360069950 con 0x7fa360164900 2024-03-31T15:15:14.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:14.881+0000 7fa35d600640 1 -- 172.21.15.16:0/402760471 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fa3580135c0 con 0x7fa360164900 2024-03-31T15:15:14.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:14.881+0000 7fa35d600640 1 --2- 172.21.15.16:0/402760471 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fa33c068380 0x7fa33c06a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:14.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:14.881+0000 7fa35d600640 1 -- 172.21.15.16:0/402760471 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(116..116 src has 1..116) ==== 5115+0+0 (secure 0 0 0) 0x7fa358086ba0 con 0x7fa360164900 2024-03-31T15:15:14.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:14.881+0000 7fa365000640 1 --2- 172.21.15.16:0/402760471 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fa33c068380 0x7fa33c06a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:14.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:14.881+0000 7fa365000640 1 --2- 172.21.15.16:0/402760471 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fa33c068380 0x7fa33c06a840 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7fa350009b50 tx=0x7fa350008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:15:14.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:14.885+0000 7fa35d600640 1 -- 172.21.15.16:0/402760471 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fa358057310 con 0x7fa360164900 2024-03-31T15:15:15.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:15.049+0000 7fa367a00640 1 -- 172.21.15.16:0/402760471 --> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fa3600b1a90 con 0x7fa33c068380 2024-03-31T15:15:15.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:15.049+0000 7fa35d600640 1 -- 172.21.15.16:0/402760471 <== mgr.4100 v2:172.21.15.89:6832/1549407273 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+13522 (secure 0 0 0) 0x7fa3600b1a90 con 0x7fa33c068380 2024-03-31T15:15:15.039 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:15:15.039 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-03-31T15:15:15.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:15.053+0000 7fa367a00640 1 -- 172.21.15.16:0/402760471 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fa33c068380 msgr2=0x7fa33c06a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:15.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:15.053+0000 7fa367a00640 1 --2- 172.21.15.16:0/402760471 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fa33c068380 0x7fa33c06a840 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7fa350009b50 tx=0x7fa350008040 comp rx=0 tx=0).stop 2024-03-31T15:15:15.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:15.053+0000 7fa367a00640 1 -- 172.21.15.16:0/402760471 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa360164900 msgr2=0x7fa360154bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:15.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:15.053+0000 7fa367a00640 1 --2- 172.21.15.16:0/402760471 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa360164900 0x7fa360154bc0 secure :-1 s=READY pgs=1896 cs=0 l=1 rev1=1 crypto rx=0x7fa358000f90 tx=0x7fa35800e530 comp rx=0 tx=0).stop 2024-03-31T15:15:15.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:15.053+0000 7fa367a00640 1 -- 172.21.15.16:0/402760471 shutdown_connections 2024-03-31T15:15:15.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:15.053+0000 7fa367a00640 1 --2- 172.21.15.16:0/402760471 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fa33c068380 0x7fa33c06a840 unknown :-1 s=CLOSED pgs=797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:15.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:15.053+0000 7fa367a00640 1 --2- 172.21.15.16:0/402760471 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa360164900 0x7fa360154bc0 unknown :-1 s=CLOSED pgs=1896 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:15.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:15.053+0000 7fa367a00640 1 --2- 172.21.15.16:0/402760471 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fa360145ab0 0x7fa3601643c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:15.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:15.053+0000 7fa367a00640 1 --2- 172.21.15.16:0/402760471 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa344003f40 0x7fa360156990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:15.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:15.053+0000 7fa367a00640 1 -- 172.21.15.16:0/402760471 >> 172.21.15.16:0/402760471 conn(0x7fa360067330 msgr2=0x7fa360161510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:15:15.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:15.053+0000 7fa367a00640 1 -- 172.21.15.16:0/402760471 shutdown_connections 2024-03-31T15:15:15.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:15.053+0000 7fa367a00640 1 -- 172.21.15.16:0/402760471 wait complete. 2024-03-31T15:15:15.060 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":465,"stamp":"2024-03-31T15:15:14.709847+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":172,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":282356,"kb_used_data":28752,"kb_used_omap":12,"kb_used_meta":253491,"kb_avail":754692364,"statfs":{"total":773094113280,"available":772804980736,"internally_reserved":0,"allocated":29442048,"data_stored":22936478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":259575344},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":31,"apply_latency_ms":31,"commit_latency_ns":31000000,"apply_latency_ns":31000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-8392376,"num_objects":-30,"num_object_clones":0,"num_object_copies":-60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-30,"num_whiteouts":0,"num_read":-258,"num_read_kb":-262,"num_write":-179,"num_write_kb":-8296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-17,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001495"},"pg_stats":[{"pgid":"1.0","version":"16'117","reported_seq":348,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-31T15:15:12.483895+0000","last_change":"2024-03-31T15:03:24.329890+0000","last_active":"2024-03-31T15:15:12.483895+0000","last_peered":"2024-03-31T15:15:12.483895+0000","last_clean":"2024-03-31T15:15:12.483895+0000","last_became_active":"2024-03-31T15:03:24.328882+0000","last_became_peered":"2024-03-31T15:03:24.328882+0000","last_unstale":"2024-03-31T15:15:12.483895+0000","last_undegraded":"2024-03-31T15:15:12.483895+0000","last_fullsized":"2024-03-31T15:15:12.483895+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:22.885535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:22.885535+0000","last_clean_scrub_stamp":"2024-03-31T15:03:22.885535+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:18:29.004296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509587,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33168,"kb_used_data":1988,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338672,"statfs":{"total":96636764160,"available":96602800128,"internally_reserved":0,"allocated":2035712,"data_stored":1217128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509586,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32588,"kb_used_data":1408,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339252,"statfs":{"total":96636764160,"available":96603394048,"internally_reserved":0,"allocated":1441792,"data_stored":626760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":15,"seq":64424509585,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32588,"kb_used_data":1408,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339252,"statfs":{"total":96636764160,"available":96603394048,"internally_reserved":0,"allocated":1441792,"data_stored":626760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509584,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":36752,"kb_used_data":1408,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335088,"statfs":{"total":96636764160,"available":96599130112,"internally_reserved":0,"allocated":1441792,"data_stored":626760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542290,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":36672,"kb_used_data":5492,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94335168,"statfs":{"total":96636764160,"available":96599212032,"internally_reserved":0,"allocated":5623808,"data_stored":4815300,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542290,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":36672,"kb_used_data":5492,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94335168,"statfs":{"total":96636764160,"available":96599212032,"internally_reserved":0,"allocated":5623808,"data_stored":4812088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542290,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":36660,"kb_used_data":5480,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94335180,"statfs":{"total":96636764160,"available":96599224320,"internally_reserved":0,"allocated":5611520,"data_stored":4809184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542290,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37256,"kb_used_data":6076,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94334584,"statfs":{"total":96636764160,"available":96598614016,"internally_reserved":0,"allocated":6221824,"data_stored":5402498,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-31T15:15:15.061 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-31T15:15:15.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:15.213+0000 7ff36e200640 1 Processor -- start 2024-03-31T15:15:15.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:15.213+0000 7ff36e200640 1 -- start start 2024-03-31T15:15:15.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:15.213+0000 7ff36e200640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7ff368145d10 0x7ff3680cf790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:15.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:15.213+0000 7ff36e200640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff368160e70 con 0x7ff3680b23d0 2024-03-31T15:15:15.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:15.213+0000 7ff36e200640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff3680cfcd0 con 0x7ff3680728b0 2024-03-31T15:15:15.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:15.213+0000 7ff36e200640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7ff3681602f0 con 0x7ff368145d10 2024-03-31T15:15:15.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:15.213+0000 7ff367e00640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7ff3680b23d0 0x7ff36815cd50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:46680/0 (socket says 172.21.15.16:46680) 2024-03-31T15:15:15.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:15.213+0000 7ff367e00640 1 -- 172.21.15.16:0/3204534076 learned_addr learned my addr 172.21.15.16:0/3204534076 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:15:15.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:15.213+0000 7ff367400640 1 --2- 172.21.15.16:0/3204534076 >> v2:172.21.15.89:3300/0 conn(0x7ff368145d10 0x7ff3680cf790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:15.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:15.217+0000 7ff367400640 1 -- 172.21.15.16:0/3204534076 >> v1:172.21.15.89:6789/0 conn(0x7ff3680728b0 legacy=0x7ff3680b1cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:15.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:15.217+0000 7ff367400640 1 -- 172.21.15.16:0/3204534076 >> v1:172.21.15.16:6789/0 conn(0x7ff3680b23d0 legacy=0x7ff36815cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:15.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:15.217+0000 7ff367400640 1 -- 172.21.15.16:0/3204534076 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff3681605d0 con 0x7ff368145d10 2024-03-31T15:15:15.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:15.217+0000 7ff367400640 1 --2- 172.21.15.16:0/3204534076 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff368145d10 0x7ff3680cf790 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7ff3580097c0 tx=0x7ff35801c4a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=121393ca3e49ac3b server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:15:15.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:15.221+0000 7ff366a00640 1 -- 172.21.15.16:0/3204534076 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3580b0070 con 0x7ff368145d10 2024-03-31T15:15:15.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:15.221+0000 7ff366a00640 1 -- 172.21.15.16:0/3204534076 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff3580ab600 con 0x7ff368145d10 2024-03-31T15:15:15.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:15.221+0000 7ff366a00640 1 -- 172.21.15.16:0/3204534076 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3580ab9d0 con 0x7ff368145d10 2024-03-31T15:15:15.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:15.221+0000 7ff36e200640 1 -- 172.21.15.16:0/3204534076 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff368145d10 msgr2=0x7ff3680cf790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:15.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:15.221+0000 7ff36e200640 1 --2- 172.21.15.16:0/3204534076 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff368145d10 0x7ff3680cf790 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7ff3580097c0 tx=0x7ff35801c4a0 comp rx=0 tx=0).stop 2024-03-31T15:15:15.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:15.221+0000 7ff36e200640 1 -- 172.21.15.16:0/3204534076 shutdown_connections 2024-03-31T15:15:15.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:15.221+0000 7ff36e200640 1 --2- 172.21.15.16:0/3204534076 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff368145d10 0x7ff3680cf790 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:15.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:15.221+0000 7ff36e200640 1 -- 172.21.15.16:0/3204534076 >> 172.21.15.16:0/3204534076 conn(0x7ff36806b620 msgr2=0x7ff36806ba30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:15:15.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:15.221+0000 7ff36e200640 1 -- 172.21.15.16:0/3204534076 shutdown_connections 2024-03-31T15:15:15.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:15.221+0000 7ff36e200640 1 -- 172.21.15.16:0/3204534076 wait complete. 2024-03-31T15:15:15.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:15.225+0000 7ff36e200640 1 Processor -- start 2024-03-31T15:15:15.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:15.225+0000 7ff36e200640 1 -- start start 2024-03-31T15:15:15.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:15.225+0000 7ff36e200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff3680728b0 0x7ff36813b8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:15.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:15.225+0000 7ff36e200640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff3680b23d0 0x7ff368143e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:15.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:15.225+0000 7ff36e200640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff368145d10 0x7ff368162b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:15.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:15.225+0000 7ff36e200640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7ff3680cf350 con 0x7ff3680728b0 2024-03-31T15:15:15.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:15.225+0000 7ff36e200640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7ff36815d870 con 0x7ff3680b23d0 2024-03-31T15:15:15.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:15.225+0000 7ff36e200640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7ff3680cf4d0 con 0x7ff368145d10 2024-03-31T15:15:15.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:15.225+0000 7ff367400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff3680728b0 0x7ff36813b8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:15.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:15.225+0000 7ff367400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff3680728b0 0x7ff36813b8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46322/0 (socket says 172.21.15.16:46322) 2024-03-31T15:15:15.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:15.225+0000 7ff367400640 1 -- 172.21.15.16:0/1437277636 learned_addr learned my addr 172.21.15.16:0/1437277636 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:15:15.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:15.225+0000 7ff367e00640 1 --2- 172.21.15.16:0/1437277636 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff368145d10 0x7ff368162b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:15.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:15.225+0000 7ff35fe00640 1 --2- 172.21.15.16:0/1437277636 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff3680b23d0 0x7ff368143e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:15.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:15.225+0000 7ff367400640 1 -- 172.21.15.16:0/1437277636 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff368145d10 msgr2=0x7ff368162b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:15.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:15.225+0000 7ff367400640 1 --2- 172.21.15.16:0/1437277636 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff368145d10 0x7ff368162b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:15.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:15.225+0000 7ff367400640 1 -- 172.21.15.16:0/1437277636 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff3680b23d0 msgr2=0x7ff368143e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:15.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:15.225+0000 7ff367400640 1 --2- 172.21.15.16:0/1437277636 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff3680b23d0 0x7ff368143e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:15.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:15.225+0000 7ff367400640 1 -- 172.21.15.16:0/1437277636 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff368142790 con 0x7ff3680728b0 2024-03-31T15:15:15.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:15.225+0000 7ff35fe00640 1 --2- 172.21.15.16:0/1437277636 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff3680b23d0 0x7ff368143e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:15:15.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:15.225+0000 7ff367400640 1 --2- 172.21.15.16:0/1437277636 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff3680728b0 0x7ff36813b8e0 secure :-1 s=READY pgs=1898 cs=0 l=1 rev1=1 crypto rx=0x7ff3580190b0 tx=0x7ff358019240 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:15:15.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:15.225+0000 7ff35f400640 1 -- 172.21.15.16:0/1437277636 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3580b0040 con 0x7ff3680728b0 2024-03-31T15:15:15.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:15.225+0000 7ff35f400640 1 -- 172.21.15.16:0/1437277636 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff3580ba070 con 0x7ff3680728b0 2024-03-31T15:15:15.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:15.225+0000 7ff35f400640 1 -- 172.21.15.16:0/1437277636 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3580b4830 con 0x7ff3680728b0 2024-03-31T15:15:15.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:15.225+0000 7ff36e200640 1 -- 172.21.15.16:0/1437277636 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff368142a70 con 0x7ff3680728b0 2024-03-31T15:15:15.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:15.229+0000 7ff36e200640 1 -- 172.21.15.16:0/1437277636 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff368143030 con 0x7ff3680728b0 2024-03-31T15:15:15.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:15.229+0000 7ff36e200640 1 -- 172.21.15.16:0/1437277636 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff368069950 con 0x7ff3680728b0 2024-03-31T15:15:15.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:15.229+0000 7ff35f400640 1 -- 172.21.15.16:0/1437277636 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7ff3580ab4c0 con 0x7ff3680728b0 2024-03-31T15:15:15.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:15.229+0000 7ff35f400640 1 --2- 172.21.15.16:0/1437277636 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7ff344068380 0x7ff34406a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:15.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:15.229+0000 7ff35f400640 1 -- 172.21.15.16:0/1437277636 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(116..116 src has 1..116) ==== 5115+0+0 (secure 0 0 0) 0x7ff358120430 con 0x7ff3680728b0 2024-03-31T15:15:15.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:15.229+0000 7ff35fe00640 1 --2- 172.21.15.16:0/1437277636 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7ff344068380 0x7ff34406a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:15.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:15.229+0000 7ff35fe00640 1 --2- 172.21.15.16:0/1437277636 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7ff344068380 0x7ff34406a840 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7ff368143bb0 tx=0x7ff350009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:15:15.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:15.233+0000 7ff35f400640 1 -- 172.21.15.16:0/1437277636 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7ff358016090 con 0x7ff3680728b0 2024-03-31T15:15:15.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:15.401+0000 7ff36e200640 1 -- 172.21.15.16:0/1437277636 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ff36806c270 con 0x7ff3680728b0 2024-03-31T15:15:15.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:15.401+0000 7ff35f400640 1 -- 172.21.15.16:0/1437277636 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v116) ==== 74+0+13218 (secure 0 0 0) 0x7ff3580f0c40 con 0x7ff3680728b0 2024-03-31T15:15:15.392 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:15:15.392 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":116,"fsid":"286eb5da-fb47-4550-b290-d9519c13b9a5","created":"2024-03-31T15:03:11.438952+0000","modified":"2024-03-31T15:15:12.199310+0000","last_up_change":"2024-03-31T15:03:22.885535+0000","last_in_change":"2024-03-31T15:03:13.941667+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-31T15:03:22.550787+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"92a66450-ef19-40d8-ac3d-d760b23b0bda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6801","nonce":932712244}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6803","nonce":932712244}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6807","nonce":932712244}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":932712244},{"type":"v1","addr":"172.21.15.16:6805","nonce":932712244}]},"public_addr":"172.21.15.16:6801/932712244","cluster_addr":"172.21.15.16:6803/932712244","heartbeat_back_addr":"172.21.15.16:6807/932712244","heartbeat_front_addr":"172.21.15.16:6805/932712244","state":["exists","up"]},{"osd":1,"uuid":"da9d1f2b-1968-4959-b843-faa231988e92","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6810","nonce":1398737533}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6811","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6813","nonce":1398737533}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6821","nonce":1398737533}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6815","nonce":1398737533},{"type":"v1","addr":"172.21.15.16:6816","nonce":1398737533}]},"public_addr":"172.21.15.16:6810/1398737533","cluster_addr":"172.21.15.16:6813/1398737533","heartbeat_back_addr":"172.21.15.16:6821/1398737533","heartbeat_front_addr":"172.21.15.16:6816/1398737533","state":["exists","up"]},{"osd":2,"uuid":"01a73662-9a21-4b1f-bcbb-42b9636a4638","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6819","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6822","nonce":3018526646}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6825","nonce":3018526646}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6831","nonce":3018526646}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6827","nonce":3018526646},{"type":"v1","addr":"172.21.15.16:6829","nonce":3018526646}]},"public_addr":"172.21.15.16:6822/3018526646","cluster_addr":"172.21.15.16:6825/3018526646","heartbeat_back_addr":"172.21.15.16:6831/3018526646","heartbeat_front_addr":"172.21.15.16:6829/3018526646","state":["exists","up"]},{"osd":3,"uuid":"c7771fd3-e18e-4500-ac38-23cf10ebcd75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6809","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6812","nonce":3360352974}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6817","nonce":3360352974}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6828","nonce":3360352974}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":3360352974},{"type":"v1","addr":"172.21.15.16:6823","nonce":3360352974}]},"public_addr":"172.21.15.16:6812/3360352974","cluster_addr":"172.21.15.16:6817/3360352974","heartbeat_back_addr":"172.21.15.16:6828/3360352974","heartbeat_front_addr":"172.21.15.16:6823/3360352974","state":["exists","up"]},{"osd":4,"uuid":"88260730-1482-4fbd-a40f-cb1124e9df4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6801","nonce":1856844856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6803","nonce":1856844856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6807","nonce":1856844856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":1856844856},{"type":"v1","addr":"172.21.15.89:6805","nonce":1856844856}]},"public_addr":"172.21.15.89:6801/1856844856","cluster_addr":"172.21.15.89:6803/1856844856","heartbeat_back_addr":"172.21.15.89:6807/1856844856","heartbeat_front_addr":"172.21.15.89:6805/1856844856","state":["exists","up"]},{"osd":5,"uuid":"4cb2c00c-fae0-4dd8-9d27-ec4f0093ccf7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6814","nonce":50395527}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6817","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6820","nonce":50395527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6831","nonce":50395527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6823","nonce":50395527},{"type":"v1","addr":"172.21.15.89:6826","nonce":50395527}]},"public_addr":"172.21.15.89:6814/50395527","cluster_addr":"172.21.15.89:6820/50395527","heartbeat_back_addr":"172.21.15.89:6831/50395527","heartbeat_front_addr":"172.21.15.89:6826/50395527","state":["exists","up"]},{"osd":6,"uuid":"81ce0c19-2554-4cae-acbe-66a05e9e9b9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6809","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6812","nonce":3470471752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6815","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6818","nonce":3470471752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6827","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6829","nonce":3470471752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6821","nonce":3470471752},{"type":"v1","addr":"172.21.15.89:6824","nonce":3470471752}]},"public_addr":"172.21.15.89:6812/3470471752","cluster_addr":"172.21.15.89:6818/3470471752","heartbeat_back_addr":"172.21.15.89:6829/3470471752","heartbeat_front_addr":"172.21.15.89:6824/3470471752","state":["exists","up"]},{"osd":7,"uuid":"85e4313b-d37d-442e-8042-153bf20a847d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6811","nonce":1796778630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6813","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6816","nonce":1796778630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6825","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6828","nonce":1796778630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6819","nonce":1796778630},{"type":"v1","addr":"172.21.15.89:6822","nonce":1796778630}]},"public_addr":"172.21.15.89:6811/1796778630","cluster_addr":"172.21.15.89:6816/1796778630","heartbeat_back_addr":"172.21.15.89:6828/1796778630","heartbeat_front_addr":"172.21.15.89:6822/1796778630","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.439088+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.529970+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.680512+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:21.044161+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.582111+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.536558+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.409522+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:03:19.590229+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:6835/2152452999":"2024-04-01T15:15:09.053203+0000","172.21.15.16:6833/2152452999":"2024-04-01T15:15:09.053203+0000","172.21.15.16:6833/4142860904":"2024-04-01T15:14:53.117187+0000","172.21.15.16:6835/2564952016":"2024-04-01T15:14:39.715615+0000","172.21.15.16:6835/4142860904":"2024-04-01T15:14:53.117187+0000","172.21.15.16:6833/2564952016":"2024-04-01T15:14:39.715615+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-31T15:15:15.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:15.405+0000 7ff36e200640 1 -- 172.21.15.16:0/1437277636 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7ff344068380 msgr2=0x7ff34406a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:15.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:15.405+0000 7ff36e200640 1 --2- 172.21.15.16:0/1437277636 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7ff344068380 0x7ff34406a840 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7ff368143bb0 tx=0x7ff350009040 comp rx=0 tx=0).stop 2024-03-31T15:15:15.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:15.405+0000 7ff36e200640 1 -- 172.21.15.16:0/1437277636 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff3680728b0 msgr2=0x7ff36813b8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:15.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:15.405+0000 7ff36e200640 1 --2- 172.21.15.16:0/1437277636 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff3680728b0 0x7ff36813b8e0 secure :-1 s=READY pgs=1898 cs=0 l=1 rev1=1 crypto rx=0x7ff3580190b0 tx=0x7ff358019240 comp rx=0 tx=0).stop 2024-03-31T15:15:15.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:15.405+0000 7ff36e200640 1 -- 172.21.15.16:0/1437277636 shutdown_connections 2024-03-31T15:15:15.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:15.405+0000 7ff36e200640 1 --2- 172.21.15.16:0/1437277636 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7ff344068380 0x7ff34406a840 unknown :-1 s=CLOSED pgs=798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:15.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:15.405+0000 7ff36e200640 1 --2- 172.21.15.16:0/1437277636 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7ff368145d10 0x7ff368162b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:15.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:15.405+0000 7ff36e200640 1 --2- 172.21.15.16:0/1437277636 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff3680b23d0 0x7ff368143e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:15.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:15.405+0000 7ff36e200640 1 --2- 172.21.15.16:0/1437277636 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff3680728b0 0x7ff36813b8e0 unknown :-1 s=CLOSED pgs=1898 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:15.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:15.405+0000 7ff36e200640 1 -- 172.21.15.16:0/1437277636 >> 172.21.15.16:0/1437277636 conn(0x7ff36806b620 msgr2=0x7ff36815da80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:15:15.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:15.405+0000 7ff36e200640 1 -- 172.21.15.16:0/1437277636 shutdown_connections 2024-03-31T15:15:15.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:15.405+0000 7ff36e200640 1 -- 172.21.15.16:0/1437277636 wait complete. 2024-03-31T15:15:16.414 INFO:tasks.ceph:Scrubbing osd.0 2024-03-31T15:15:16.414 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-03-31T15:15:16.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:16.581+0000 7f8bf6000640 1 Processor -- start 2024-03-31T15:15:16.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:16.581+0000 7f8bf6000640 1 -- start start 2024-03-31T15:15:16.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:16.585+0000 7f8bf6000640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f8bf00b1400 0x7f8bf006c730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:16.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:16.585+0000 7f8bf6000640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f8bf00d0ce0 0x7f8bf00d1100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:16.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:16.585+0000 7f8bf6000640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8bf00b17e0 con 0x7f8bf006fea0 2024-03-31T15:15:16.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:16.585+0000 7f8bf6000640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f8bf00cc5e0 con 0x7f8bf00b1400 2024-03-31T15:15:16.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:16.585+0000 7f8bf6000640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f8bf00cc8a0 con 0x7f8bf00d0ce0 2024-03-31T15:15:16.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:16.585+0000 7f8bf4c00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f8bf00b1400 0x7f8bf006c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:16.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:16.585+0000 7f8bf5600640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f8bf006fea0 0x7f8bf015dc00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:46692/0 (socket says 172.21.15.16:46692) 2024-03-31T15:15:16.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:16.585+0000 7f8bf5600640 1 -- 172.21.15.16:0/3966027798 learned_addr learned my addr 172.21.15.16:0/3966027798 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:15:16.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:16.585+0000 7f8be7e00640 1 --2- 172.21.15.16:0/3966027798 >> v2:172.21.15.89:3300/0 conn(0x7f8bf00d0ce0 0x7f8bf00d1100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:16.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:16.585+0000 7f8befe00640 1 -- 172.21.15.16:0/3966027798 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1572945643 0 0) 0x7f8bf00b17e0 con 0x7f8bf006fea0 2024-03-31T15:15:16.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:16.585+0000 7f8befe00640 1 -- 172.21.15.16:0/3966027798 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8bd4003530 con 0x7f8bf006fea0 2024-03-31T15:15:16.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:16.585+0000 7f8bf4c00640 1 -- 172.21.15.16:0/3966027798 >> v2:172.21.15.89:3300/0 conn(0x7f8bf00d0ce0 msgr2=0x7f8bf00d1100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:15:16.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:16.585+0000 7f8bf4c00640 1 --2- 172.21.15.16:0/3966027798 >> v2:172.21.15.89:3300/0 conn(0x7f8bf00d0ce0 0x7f8bf00d1100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:16.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:16.585+0000 7f8bf4c00640 1 -- 172.21.15.16:0/3966027798 >> v1:172.21.15.16:6789/0 conn(0x7f8bf006fea0 legacy=0x7f8bf015dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:16.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:16.585+0000 7f8bf4c00640 1 -- 172.21.15.16:0/3966027798 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8bf00ccb80 con 0x7f8bf00b1400 2024-03-31T15:15:16.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:16.585+0000 7f8bf4c00640 1 --2- 172.21.15.16:0/3966027798 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8bf00b1400 0x7f8bf006c730 secure :-1 s=READY pgs=1900 cs=0 l=1 rev1=1 crypto rx=0x7f8bf00720e0 tx=0x7f8be001c450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f1ce7d5d0fda9e0d server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:15:16.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:16.589+0000 7f8befe00640 1 -- 172.21.15.16:0/3966027798 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8be00ab620 con 0x7f8bf00b1400 2024-03-31T15:15:16.577 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:16.589+0000 7f8befe00640 1 -- 172.21.15.16:0/3966027798 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8be00b0070 con 0x7f8bf00b1400 2024-03-31T15:15:16.577 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:16.589+0000 7f8befe00640 1 -- 172.21.15.16:0/3966027798 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8be00ab9e0 con 0x7f8bf00b1400 2024-03-31T15:15:16.577 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:16.589+0000 7f8bf6000640 1 -- 172.21.15.16:0/3966027798 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8bf00b1400 msgr2=0x7f8bf006c730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:16.577 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:16.589+0000 7f8bf6000640 1 --2- 172.21.15.16:0/3966027798 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8bf00b1400 0x7f8bf006c730 secure :-1 s=READY pgs=1900 cs=0 l=1 rev1=1 crypto rx=0x7f8bf00720e0 tx=0x7f8be001c450 comp rx=0 tx=0).stop 2024-03-31T15:15:16.577 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:16.589+0000 7f8bf6000640 1 -- 172.21.15.16:0/3966027798 shutdown_connections 2024-03-31T15:15:16.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:16.589+0000 7f8bf6000640 1 --2- 172.21.15.16:0/3966027798 >> v2:172.21.15.89:3300/0 conn(0x7f8bf00d0ce0 0x7f8bf00d1100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:16.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:16.589+0000 7f8bf6000640 1 --2- 172.21.15.16:0/3966027798 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8bf00b1400 0x7f8bf006c730 unknown :-1 s=CLOSED pgs=1900 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:16.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:16.589+0000 7f8bf6000640 1 -- 172.21.15.16:0/3966027798 >> 172.21.15.16:0/3966027798 conn(0x7f8bf00d1f90 msgr2=0x7f8bf00d23a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:15:16.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:16.589+0000 7f8bf6000640 1 -- 172.21.15.16:0/3966027798 shutdown_connections 2024-03-31T15:15:16.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:16.589+0000 7f8bf6000640 1 -- 172.21.15.16:0/3966027798 wait complete. 2024-03-31T15:15:16.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:16.589+0000 7f8bf6000640 1 Processor -- start 2024-03-31T15:15:16.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:16.589+0000 7f8bf6000640 1 -- start start 2024-03-31T15:15:16.580 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:16.589+0000 7f8bf6000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8bf006fea0 0x7f8bf00c9060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:16.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:16.589+0000 7f8bf6000640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8bf00b1400 0x7f8bf00c6d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:16.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:16.589+0000 7f8bf6000640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f8bf00d0ce0 0x7f8bf01fa610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:16.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:16.589+0000 7f8bf6000640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f8bf0070840 con 0x7f8bf006fea0 2024-03-31T15:15:16.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:16.589+0000 7f8bf6000640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f8bf00706c0 con 0x7f8bf00b1400 2024-03-31T15:15:16.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:16.589+0000 7f8bf6000640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f8bf00cd660 con 0x7f8bf00d0ce0 2024-03-31T15:15:16.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:16.589+0000 7f8bf4c00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8bf006fea0 0x7f8bf00c9060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:16.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:16.589+0000 7f8bf4c00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8bf006fea0 0x7f8bf00c9060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46354/0 (socket says 172.21.15.16:46354) 2024-03-31T15:15:16.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:16.589+0000 7f8bf4c00640 1 -- 172.21.15.16:0/2404809888 learned_addr learned my addr 172.21.15.16:0/2404809888 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:15:16.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:16.589+0000 7f8bf4c00640 1 -- 172.21.15.16:0/2404809888 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f8bf00d0ce0 msgr2=0x7f8bf01fa610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:16.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:16.593+0000 7f8bf4c00640 1 --2- 172.21.15.16:0/2404809888 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f8bf00d0ce0 0x7f8bf01fa610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:16.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:16.593+0000 7f8bf4c00640 1 -- 172.21.15.16:0/2404809888 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8bf00b1400 msgr2=0x7f8bf00c6d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:16.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:16.593+0000 7f8bf4c00640 1 --2- 172.21.15.16:0/2404809888 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8bf00b1400 0x7f8bf00c6d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:16.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:16.593+0000 7f8bf4c00640 1 -- 172.21.15.16:0/2404809888 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8bf00c98b0 con 0x7f8bf006fea0 2024-03-31T15:15:16.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:16.593+0000 7f8bf4c00640 1 --2- 172.21.15.16:0/2404809888 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8bf006fea0 0x7f8bf00c9060 secure :-1 s=READY pgs=1901 cs=0 l=1 rev1=1 crypto rx=0x7f8be00ad040 tx=0x7f8be001ac90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:15:16.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:16.593+0000 7f8bed600640 1 -- 172.21.15.16:0/2404809888 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8be00abcd0 con 0x7f8bf006fea0 2024-03-31T15:15:16.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:16.593+0000 7f8bed600640 1 -- 172.21.15.16:0/2404809888 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8be00bf410 con 0x7f8bf006fea0 2024-03-31T15:15:16.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:16.593+0000 7f8bed600640 1 -- 172.21.15.16:0/2404809888 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8be00bf800 con 0x7f8bf006fea0 2024-03-31T15:15:16.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:16.593+0000 7f8bf6000640 1 -- 172.21.15.16:0/2404809888 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8bf00c9b90 con 0x7f8bf006fea0 2024-03-31T15:15:16.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:16.593+0000 7f8bf6000640 1 -- 172.21.15.16:0/2404809888 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8bf015ab40 con 0x7f8bf006fea0 2024-03-31T15:15:16.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:16.593+0000 7f8bf6000640 1 -- 172.21.15.16:0/2404809888 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f8bf004f8c0 con 0x7f8bf006fea0 2024-03-31T15:15:16.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:16.593+0000 7f8bed600640 1 -- 172.21.15.16:0/2404809888 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f8be00bd020 con 0x7f8bf006fea0 2024-03-31T15:15:16.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:16.593+0000 7f8bed600640 1 --2- 172.21.15.16:0/2404809888 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f8bcc068380 0x7f8bcc06a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:16.585 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:16.593+0000 7f8bed600640 1 -- 172.21.15.16:0/2404809888 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(116..116 src has 1..116) ==== 5115+0+0 (secure 0 0 0) 0x7f8be01211f0 con 0x7f8bf006fea0 2024-03-31T15:15:16.585 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:16.593+0000 7f8bed600640 1 --2- 172.21.15.16:0/2404809888 >> [v2:172.21.15.16:6800/932712244,v1:172.21.15.16:6801/932712244] conn(0x7f8bcc06e380 0x7f8bcc0707a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:16.585 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:16.593+0000 7f8bed600640 1 -- 172.21.15.16:0/2404809888 --> [v2:172.21.15.16:6800/932712244,v1:172.21.15.16:6801/932712244] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f8be00ace90 con 0x7f8bcc06e380 2024-03-31T15:15:16.585 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:16.593+0000 7f8bed600640 1 -- 172.21.15.16:0/2404809888 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_get_version_reply(handle=1 version=116) ==== 24+0+0 (secure 0 0 0) 0x7f8be01215a0 con 0x7f8bf006fea0 2024-03-31T15:15:16.585 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:16.593+0000 7f8be7e00640 1 --2- 172.21.15.16:0/2404809888 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f8bcc068380 0x7f8bcc06a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:16.585 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:16.597+0000 7f8bf5600640 1 --2- 172.21.15.16:0/2404809888 >> [v2:172.21.15.16:6800/932712244,v1:172.21.15.16:6801/932712244] conn(0x7f8bcc06e380 0x7f8bcc0707a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:16.586 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:16.597+0000 7f8be7e00640 1 --2- 172.21.15.16:0/2404809888 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f8bcc068380 0x7f8bcc06a840 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7f8bf00caad0 tx=0x7f8bd8008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:15:16.586 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:16.597+0000 7f8bf5600640 1 --2- 172.21.15.16:0/2404809888 >> [v2:172.21.15.16:6800/932712244,v1:172.21.15.16:6801/932712244] conn(0x7f8bcc06e380 0x7f8bcc0707a0 crc :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:15:16.586 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:16.597+0000 7f8bed600640 1 -- 172.21.15.16:0/2404809888 <== osd.0 v2:172.21.15.16:6800/932712244 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f8be00ace90 con 0x7f8bcc06e380 2024-03-31T15:15:16.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:16.609+0000 7f8bf6000640 1 -- 172.21.15.16:0/2404809888 --> [v2:172.21.15.16:6800/932712244,v1:172.21.15.16:6801/932712244] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f8bf00cc050 con 0x7f8bcc06e380 2024-03-31T15:15:16.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:16.609+0000 7f8bed600640 1 -- 172.21.15.16:0/2404809888 <== osd.0 v2:172.21.15.16:6800/932712244 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f8bf00cc050 con 0x7f8bcc06e380 2024-03-31T15:15:16.600 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-03-31T15:15:16.600 INFO:teuthology.orchestra.run.smithi016.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-31T15:15:16.600 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-03-31T15:15:16.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:16.609+0000 7f8bf6000640 1 -- 172.21.15.16:0/2404809888 >> [v2:172.21.15.16:6800/932712244,v1:172.21.15.16:6801/932712244] conn(0x7f8bcc06e380 msgr2=0x7f8bcc0707a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:16.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:16.609+0000 7f8bf6000640 1 --2- 172.21.15.16:0/2404809888 >> [v2:172.21.15.16:6800/932712244,v1:172.21.15.16:6801/932712244] conn(0x7f8bcc06e380 0x7f8bcc0707a0 crc :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:16.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:16.609+0000 7f8bf6000640 1 -- 172.21.15.16:0/2404809888 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f8bcc068380 msgr2=0x7f8bcc06a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:16.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:16.609+0000 7f8bf6000640 1 --2- 172.21.15.16:0/2404809888 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f8bcc068380 0x7f8bcc06a840 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7f8bf00caad0 tx=0x7f8bd8008040 comp rx=0 tx=0).stop 2024-03-31T15:15:16.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:16.609+0000 7f8bf6000640 1 -- 172.21.15.16:0/2404809888 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8bf006fea0 msgr2=0x7f8bf00c9060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:16.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:16.609+0000 7f8bf6000640 1 --2- 172.21.15.16:0/2404809888 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8bf006fea0 0x7f8bf00c9060 secure :-1 s=READY pgs=1901 cs=0 l=1 rev1=1 crypto rx=0x7f8be00ad040 tx=0x7f8be001ac90 comp rx=0 tx=0).stop 2024-03-31T15:15:16.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:16.609+0000 7f8bf5600640 1 -- 172.21.15.16:0/2404809888 reap_dead start 2024-03-31T15:15:16.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:16.609+0000 7f8bf6000640 1 -- 172.21.15.16:0/2404809888 shutdown_connections 2024-03-31T15:15:16.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:16.609+0000 7f8bf6000640 1 -- 172.21.15.16:0/2404809888 >> 172.21.15.16:0/2404809888 conn(0x7f8bf00d1f90 msgr2=0x7f8bf00cd9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:15:16.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:16.609+0000 7f8bf6000640 1 -- 172.21.15.16:0/2404809888 shutdown_connections 2024-03-31T15:15:16.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:16.609+0000 7f8bf6000640 1 -- 172.21.15.16:0/2404809888 wait complete. 2024-03-31T15:15:16.618 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2024-03-31T15:15:16.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:16.769+0000 7f0020a00640 1 Processor -- start 2024-03-31T15:15:16.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:16.769+0000 7f0020a00640 1 -- start start 2024-03-31T15:15:16.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:16.769+0000 7f0020a00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f001c145e40 0x7f001c0cc740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:16.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:16.769+0000 7f0020a00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f001c0ccc80 0x7f001c06f920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:16.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:16.769+0000 7f0020a00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f001c06fe60 0x7f001c15fdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:16.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:16.769+0000 7f0020a00640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f001c0d06d0 con 0x7f001c145e40 2024-03-31T15:15:16.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:16.769+0000 7f0020a00640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f001c0d0920 con 0x7f001c0ccc80 2024-03-31T15:15:16.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:16.769+0000 7f0020a00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f001c0d0c00 con 0x7f001c06fe60 2024-03-31T15:15:16.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:16.769+0000 7f0019e00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f001c145e40 0x7f001c0cc740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:16.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:16.769+0000 7f0019e00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f001c145e40 0x7f001c0cc740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46366/0 (socket says 172.21.15.16:46366) 2024-03-31T15:15:16.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:16.769+0000 7f0019e00640 1 -- 172.21.15.16:0/2737391998 learned_addr learned my addr 172.21.15.16:0/2737391998 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:15:16.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:16.769+0000 7f001a800640 1 --2- 172.21.15.16:0/2737391998 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f001c06fe60 0x7f001c15fdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:16.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:16.769+0000 7f0013e00640 1 --2- 172.21.15.16:0/2737391998 >> v2:172.21.15.89:3300/0 conn(0x7f001c0ccc80 0x7f001c06f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:16.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:16.769+0000 7f0019e00640 1 -- 172.21.15.16:0/2737391998 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f001c06fe60 msgr2=0x7f001c15fdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:15:16.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:16.769+0000 7f0019e00640 1 --2- 172.21.15.16:0/2737391998 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f001c06fe60 0x7f001c15fdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:16.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:16.769+0000 7f0019e00640 1 -- 172.21.15.16:0/2737391998 >> v2:172.21.15.89:3300/0 conn(0x7f001c0ccc80 msgr2=0x7f001c06f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:15:16.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:16.769+0000 7f0019e00640 1 --2- 172.21.15.16:0/2737391998 >> v2:172.21.15.89:3300/0 conn(0x7f001c0ccc80 0x7f001c06f920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:16.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:16.769+0000 7f0019e00640 1 -- 172.21.15.16:0/2737391998 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f001c0d1810 con 0x7f001c145e40 2024-03-31T15:15:16.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:16.773+0000 7f0019e00640 1 --2- 172.21.15.16:0/2737391998 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f001c145e40 0x7f001c0cc740 secure :-1 s=READY pgs=1902 cs=0 l=1 rev1=1 crypto rx=0x7f001c073cf0 tx=0x7f000c01c260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de7fe3abc27bc53 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:15:16.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:16.773+0000 7f0019400640 1 -- 172.21.15.16:0/2737391998 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f000c0b0070 con 0x7f001c145e40 2024-03-31T15:15:16.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:16.773+0000 7f0019400640 1 -- 172.21.15.16:0/2737391998 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f000c0ab600 con 0x7f001c145e40 2024-03-31T15:15:16.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:16.773+0000 7f0019400640 1 -- 172.21.15.16:0/2737391998 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f000c0ab9d0 con 0x7f001c145e40 2024-03-31T15:15:16.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:16.773+0000 7f0020a00640 1 -- 172.21.15.16:0/2737391998 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f001c145e40 msgr2=0x7f001c0cc740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:16.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:16.773+0000 7f0020a00640 1 --2- 172.21.15.16:0/2737391998 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f001c145e40 0x7f001c0cc740 secure :-1 s=READY pgs=1902 cs=0 l=1 rev1=1 crypto rx=0x7f001c073cf0 tx=0x7f000c01c260 comp rx=0 tx=0).stop 2024-03-31T15:15:16.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:16.773+0000 7f0020a00640 1 -- 172.21.15.16:0/2737391998 shutdown_connections 2024-03-31T15:15:16.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:16.773+0000 7f0020a00640 1 --2- 172.21.15.16:0/2737391998 >> v2:172.21.15.89:3300/0 conn(0x7f001c0ccc80 0x7f001c06f920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:16.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:16.773+0000 7f0020a00640 1 --2- 172.21.15.16:0/2737391998 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f001c06fe60 0x7f001c15fdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:16.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:16.773+0000 7f0020a00640 1 --2- 172.21.15.16:0/2737391998 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f001c145e40 0x7f001c0cc740 unknown :-1 s=CLOSED pgs=1902 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:16.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:16.773+0000 7f0020a00640 1 -- 172.21.15.16:0/2737391998 >> 172.21.15.16:0/2737391998 conn(0x7f001c067330 msgr2=0x7f001c067740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:15:16.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:16.773+0000 7f0020a00640 1 -- 172.21.15.16:0/2737391998 shutdown_connections 2024-03-31T15:15:16.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:16.773+0000 7f0020a00640 1 -- 172.21.15.16:0/2737391998 wait complete. 2024-03-31T15:15:16.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:16.777+0000 7f0020a00640 1 Processor -- start 2024-03-31T15:15:16.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:16.777+0000 7f0020a00640 1 -- start start 2024-03-31T15:15:16.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:16.777+0000 7f0020a00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f001c06fe60 0x7f001c06e3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:16.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:16.777+0000 7f0020a00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f001c0ccc80 0x7f001c145160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:16.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:16.777+0000 7f0020a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f001c145e40 0x7f001c14f580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:16.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:16.777+0000 7f0020a00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f001c0b5240 con 0x7f001c145e40 2024-03-31T15:15:16.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:16.777+0000 7f0020a00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f001c0b50c0 con 0x7f001c0ccc80 2024-03-31T15:15:16.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:16.777+0000 7f0020a00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f001c0d2010 con 0x7f001c06fe60 2024-03-31T15:15:16.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:16.777+0000 7f001a800640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f001c145e40 0x7f001c14f580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:16.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:16.777+0000 7f001a800640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f001c145e40 0x7f001c14f580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46372/0 (socket says 172.21.15.16:46372) 2024-03-31T15:15:16.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:16.777+0000 7f001a800640 1 -- 172.21.15.16:0/764675529 learned_addr learned my addr 172.21.15.16:0/764675529 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:15:16.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:16.777+0000 7f0019e00640 1 --2- 172.21.15.16:0/764675529 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f001c06fe60 0x7f001c06e3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:16.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:16.777+0000 7f0013e00640 1 --2- 172.21.15.16:0/764675529 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f001c0ccc80 0x7f001c145160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:16.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:16.777+0000 7f001a800640 1 -- 172.21.15.16:0/764675529 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f001c06fe60 msgr2=0x7f001c06e3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:16.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:16.777+0000 7f001a800640 1 --2- 172.21.15.16:0/764675529 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f001c06fe60 0x7f001c06e3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:16.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:16.777+0000 7f001a800640 1 -- 172.21.15.16:0/764675529 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f001c0ccc80 msgr2=0x7f001c145160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:16.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:16.777+0000 7f001a800640 1 --2- 172.21.15.16:0/764675529 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f001c0ccc80 0x7f001c145160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:16.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:16.777+0000 7f001a800640 1 -- 172.21.15.16:0/764675529 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f001c06d910 con 0x7f001c145e40 2024-03-31T15:15:16.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:16.777+0000 7f001a800640 1 --2- 172.21.15.16:0/764675529 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f001c145e40 0x7f001c14f580 secure :-1 s=READY pgs=1903 cs=0 l=1 rev1=1 crypto rx=0x7f0014007e40 tx=0x7f001400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:15:16.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:16.777+0000 7f0012000640 1 -- 172.21.15.16:0/764675529 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0014014620 con 0x7f001c145e40 2024-03-31T15:15:16.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:16.777+0000 7f0012000640 1 -- 172.21.15.16:0/764675529 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0014019070 con 0x7f001c145e40 2024-03-31T15:15:16.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:16.777+0000 7f0012000640 1 -- 172.21.15.16:0/764675529 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0014014a20 con 0x7f001c145e40 2024-03-31T15:15:16.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:16.777+0000 7f0020a00640 1 -- 172.21.15.16:0/764675529 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f001c157970 con 0x7f001c145e40 2024-03-31T15:15:16.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:16.777+0000 7f0020a00640 1 -- 172.21.15.16:0/764675529 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f001c06dbb0 con 0x7f001c145e40 2024-03-31T15:15:16.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:16.781+0000 7f0020a00640 1 -- 172.21.15.16:0/764675529 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f001c069950 con 0x7f001c145e40 2024-03-31T15:15:16.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:16.781+0000 7f0012000640 1 -- 172.21.15.16:0/764675529 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f0014014bc0 con 0x7f001c145e40 2024-03-31T15:15:16.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:16.781+0000 7f0012000640 1 --2- 172.21.15.16:0/764675529 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7efff80683b0 0x7efff806a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:16.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:16.781+0000 7f0012000640 1 -- 172.21.15.16:0/764675529 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(116..116 src has 1..116) ==== 5115+0+0 (secure 0 0 0) 0x7f00140870b0 con 0x7f001c145e40 2024-03-31T15:15:16.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:16.781+0000 7f0019e00640 1 --2- 172.21.15.16:0/764675529 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7efff80683b0 0x7efff806a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:16.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:16.785+0000 7f0019e00640 1 --2- 172.21.15.16:0/764675529 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7efff80683b0 0x7efff806a870 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7f000c0ad000 tx=0x7f000c0a6000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:15:16.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:16.785+0000 7f0012000640 1 -- 172.21.15.16:0/764675529 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f0014057820 con 0x7f001c145e40 2024-03-31T15:15:16.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:16.949+0000 7f0020a00640 1 -- 172.21.15.16:0/764675529 --> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "0", "target": ["mon-mgr", ""]}) -- 0x7f001c0cf3c0 con 0x7efff80683b0 2024-03-31T15:15:16.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:16.949+0000 7f0012000640 1 -- 172.21.15.16:0/764675529 <== mgr.4100 v2:172.21.15.89:6832/1549407273 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 0 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f001c0cf3c0 con 0x7efff80683b0 2024-03-31T15:15:16.940 INFO:teuthology.orchestra.run.smithi016.stderr:instructed osd(s) 0 to deep-scrub 2024-03-31T15:15:16.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:16.953+0000 7f0020a00640 1 -- 172.21.15.16:0/764675529 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7efff80683b0 msgr2=0x7efff806a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:16.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:16.953+0000 7f0020a00640 1 --2- 172.21.15.16:0/764675529 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7efff80683b0 0x7efff806a870 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7f000c0ad000 tx=0x7f000c0a6000 comp rx=0 tx=0).stop 2024-03-31T15:15:16.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:16.953+0000 7f0020a00640 1 -- 172.21.15.16:0/764675529 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f001c145e40 msgr2=0x7f001c14f580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:16.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:16.953+0000 7f0020a00640 1 --2- 172.21.15.16:0/764675529 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f001c145e40 0x7f001c14f580 secure :-1 s=READY pgs=1903 cs=0 l=1 rev1=1 crypto rx=0x7f0014007e40 tx=0x7f001400e4b0 comp rx=0 tx=0).stop 2024-03-31T15:15:16.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:16.953+0000 7f0020a00640 1 -- 172.21.15.16:0/764675529 shutdown_connections 2024-03-31T15:15:16.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:16.953+0000 7f0020a00640 1 --2- 172.21.15.16:0/764675529 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7efff80683b0 0x7efff806a870 unknown :-1 s=CLOSED pgs=800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:16.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:16.953+0000 7f0020a00640 1 --2- 172.21.15.16:0/764675529 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f001c145e40 0x7f001c14f580 unknown :-1 s=CLOSED pgs=1903 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:16.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:16.953+0000 7f0020a00640 1 --2- 172.21.15.16:0/764675529 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f001c0ccc80 0x7f001c145160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:16.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:16.953+0000 7f0020a00640 1 --2- 172.21.15.16:0/764675529 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f001c06fe60 0x7f001c06e3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:16.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:16.953+0000 7f0020a00640 1 -- 172.21.15.16:0/764675529 >> 172.21.15.16:0/764675529 conn(0x7f001c067330 msgr2=0x7f001c15ccd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:15:16.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:16.953+0000 7f0020a00640 1 -- 172.21.15.16:0/764675529 shutdown_connections 2024-03-31T15:15:16.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:16.953+0000 7f0020a00640 1 -- 172.21.15.16:0/764675529 wait complete. 2024-03-31T15:15:16.961 INFO:tasks.ceph:Scrubbing osd.1 2024-03-31T15:15:16.962 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-03-31T15:15:17.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.117+0000 7fd574a00640 1 Processor -- start 2024-03-31T15:15:17.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.117+0000 7fd574a00640 1 -- start start 2024-03-31T15:15:17.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.117+0000 7fd574a00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fd570143f40 0x7fd5700730c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:17.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.117+0000 7fd574a00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fd5700b45d0 0x7fd5700b1be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:17.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.117+0000 7fd574a00640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd570144320 con 0x7fd5700b2120 2024-03-31T15:15:17.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.117+0000 7fd574a00640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7fd5700b2570 con 0x7fd570143f40 2024-03-31T15:15:17.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.117+0000 7fd574a00640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7fd5700b4a30 con 0x7fd5700b45d0 2024-03-31T15:15:17.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.117+0000 7fd56de00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fd570143f40 0x7fd5700730c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:17.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.117+0000 7fd56de00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fd570143f40 0x7fd5700730c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37754/0 (socket says 172.21.15.16:37754) 2024-03-31T15:15:17.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.117+0000 7fd56de00640 1 -- 172.21.15.16:0/800676834 learned_addr learned my addr 172.21.15.16:0/800676834 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:15:17.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.117+0000 7fd565400640 1 --2- 172.21.15.16:0/800676834 >> v2:172.21.15.89:3300/0 conn(0x7fd5700b45d0 0x7fd5700b1be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:17.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.117+0000 7fd56de00640 1 -- 172.21.15.16:0/800676834 >> v2:172.21.15.89:3300/0 conn(0x7fd5700b45d0 msgr2=0x7fd5700b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:15:17.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.117+0000 7fd56de00640 1 --2- 172.21.15.16:0/800676834 >> v2:172.21.15.89:3300/0 conn(0x7fd5700b45d0 0x7fd5700b1be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:17.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.117+0000 7fd56de00640 1 -- 172.21.15.16:0/800676834 >> v1:172.21.15.89:6789/0 conn(0x7fd5700b2120 legacy=0x7fd570071490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:15:17.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.117+0000 7fd56de00640 1 -- 172.21.15.16:0/800676834 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd57015ef80 con 0x7fd570143f40 2024-03-31T15:15:17.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.117+0000 7fd565400640 1 --2- 172.21.15.16:0/800676834 >> v2:172.21.15.89:3300/0 conn(0x7fd5700b45d0 0x7fd5700b1be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:15:17.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.121+0000 7fd56de00640 1 --2- 172.21.15.16:0/800676834 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd570143f40 0x7fd5700730c0 secure :-1 s=READY pgs=1904 cs=0 l=1 rev1=1 crypto rx=0x7fd5701450e0 tx=0x7fd56001c4e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df6592518d403ab server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:15:17.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.121+0000 7fd56d400640 1 -- 172.21.15.16:0/800676834 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5600b0070 con 0x7fd570143f40 2024-03-31T15:15:17.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.121+0000 7fd56d400640 1 -- 172.21.15.16:0/800676834 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd5600ab600 con 0x7fd570143f40 2024-03-31T15:15:17.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.121+0000 7fd56d400640 1 -- 172.21.15.16:0/800676834 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5600ab9d0 con 0x7fd570143f40 2024-03-31T15:15:17.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.121+0000 7fd574a00640 1 -- 172.21.15.16:0/800676834 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd570143f40 msgr2=0x7fd5700730c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:17.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.121+0000 7fd574a00640 1 --2- 172.21.15.16:0/800676834 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd570143f40 0x7fd5700730c0 secure :-1 s=READY pgs=1904 cs=0 l=1 rev1=1 crypto rx=0x7fd5701450e0 tx=0x7fd56001c4e0 comp rx=0 tx=0).stop 2024-03-31T15:15:17.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.121+0000 7fd574a00640 1 -- 172.21.15.16:0/800676834 shutdown_connections 2024-03-31T15:15:17.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.121+0000 7fd574a00640 1 --2- 172.21.15.16:0/800676834 >> v2:172.21.15.89:3300/0 conn(0x7fd5700b45d0 0x7fd5700b1be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:17.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.121+0000 7fd574a00640 1 --2- 172.21.15.16:0/800676834 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd570143f40 0x7fd5700730c0 unknown :-1 s=CLOSED pgs=1904 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:17.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.121+0000 7fd574a00640 1 -- 172.21.15.16:0/800676834 >> 172.21.15.16:0/800676834 conn(0x7fd57006b620 msgr2=0x7fd57006ba30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:15:17.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.121+0000 7fd574a00640 1 -- 172.21.15.16:0/800676834 shutdown_connections 2024-03-31T15:15:17.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.121+0000 7fd574a00640 1 -- 172.21.15.16:0/800676834 wait complete. 2024-03-31T15:15:17.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.125+0000 7fd574a00640 1 Processor -- start 2024-03-31T15:15:17.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.125+0000 7fd574a00640 1 -- start start 2024-03-31T15:15:17.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.125+0000 7fd574a00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd5700b2120 0x7fd57015e3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:17.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.125+0000 7fd574a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd5700b45d0 0x7fd570140e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:17.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.125+0000 7fd574a00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fd570143f40 0x7fd57015aa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:17.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.125+0000 7fd574a00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fd5700cd910 con 0x7fd5700b45d0 2024-03-31T15:15:17.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.125+0000 7fd574a00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fd5700cd790 con 0x7fd5700b2120 2024-03-31T15:15:17.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.125+0000 7fd574a00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fd5701660a0 con 0x7fd570143f40 2024-03-31T15:15:17.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.125+0000 7fd565400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd5700b45d0 0x7fd570140e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:17.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.125+0000 7fd565400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd5700b45d0 0x7fd570140e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37766/0 (socket says 172.21.15.16:37766) 2024-03-31T15:15:17.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.125+0000 7fd565400640 1 -- 172.21.15.16:0/3116773331 learned_addr learned my addr 172.21.15.16:0/3116773331 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:15:17.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.125+0000 7fd56de00640 1 --2- 172.21.15.16:0/3116773331 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd5700b2120 0x7fd57015e3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:17.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.125+0000 7fd56e800640 1 --2- 172.21.15.16:0/3116773331 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fd570143f40 0x7fd57015aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:17.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.125+0000 7fd565400640 1 -- 172.21.15.16:0/3116773331 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fd570143f40 msgr2=0x7fd57015aa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:17.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.125+0000 7fd565400640 1 --2- 172.21.15.16:0/3116773331 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fd570143f40 0x7fd57015aa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:17.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.125+0000 7fd565400640 1 -- 172.21.15.16:0/3116773331 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd5700b2120 msgr2=0x7fd57015e3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:17.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.125+0000 7fd565400640 1 --2- 172.21.15.16:0/3116773331 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd5700b2120 0x7fd57015e3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:17.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.125+0000 7fd565400640 1 -- 172.21.15.16:0/3116773331 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd57015af50 con 0x7fd5700b45d0 2024-03-31T15:15:17.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.125+0000 7fd565400640 1 --2- 172.21.15.16:0/3116773331 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd5700b45d0 0x7fd570140e00 secure :-1 s=READY pgs=1905 cs=0 l=1 rev1=1 crypto rx=0x7fd55800cf60 tx=0x7fd55800cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:15:17.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.125+0000 7fd566a00640 1 -- 172.21.15.16:0/3116773331 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd55801a070 con 0x7fd5700b45d0 2024-03-31T15:15:17.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.125+0000 7fd566a00640 1 -- 172.21.15.16:0/3116773331 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd55800d840 con 0x7fd5700b45d0 2024-03-31T15:15:17.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.125+0000 7fd566a00640 1 -- 172.21.15.16:0/3116773331 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd55800dc10 con 0x7fd5700b45d0 2024-03-31T15:15:17.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.125+0000 7fd574a00640 1 -- 172.21.15.16:0/3116773331 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd57015ea30 con 0x7fd5700b45d0 2024-03-31T15:15:17.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.125+0000 7fd574a00640 1 -- 172.21.15.16:0/3116773331 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd57013e5a0 con 0x7fd5700b45d0 2024-03-31T15:15:17.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.125+0000 7fd574a00640 1 -- 172.21.15.16:0/3116773331 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fd570069950 con 0x7fd5700b45d0 2024-03-31T15:15:17.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.129+0000 7fd566a00640 1 -- 172.21.15.16:0/3116773331 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fd55801d070 con 0x7fd5700b45d0 2024-03-31T15:15:17.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.129+0000 7fd566a00640 1 --2- 172.21.15.16:0/3116773331 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fd54c068380 0x7fd54c06a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:17.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.129+0000 7fd566a00640 1 -- 172.21.15.16:0/3116773331 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(116..116 src has 1..116) ==== 5115+0+0 (secure 0 0 0) 0x7fd558086f70 con 0x7fd5700b45d0 2024-03-31T15:15:17.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.129+0000 7fd566a00640 1 --2- 172.21.15.16:0/3116773331 >> [v2:172.21.15.16:6808/1398737533,v1:172.21.15.16:6810/1398737533] conn(0x7fd54c06e380 0x7fd54c0707a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:17.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.129+0000 7fd566a00640 1 -- 172.21.15.16:0/3116773331 --> [v2:172.21.15.16:6808/1398737533,v1:172.21.15.16:6810/1398737533] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd54c070e50 con 0x7fd54c06e380 2024-03-31T15:15:17.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.129+0000 7fd566a00640 1 -- 172.21.15.16:0/3116773331 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_get_version_reply(handle=1 version=116) ==== 24+0+0 (secure 0 0 0) 0x7fd558089590 con 0x7fd5700b45d0 2024-03-31T15:15:17.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.129+0000 7fd56de00640 1 --2- 172.21.15.16:0/3116773331 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fd54c068380 0x7fd54c06a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:17.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.129+0000 7fd56e800640 1 --2- 172.21.15.16:0/3116773331 >> [v2:172.21.15.16:6808/1398737533,v1:172.21.15.16:6810/1398737533] conn(0x7fd54c06e380 0x7fd54c0707a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:17.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.129+0000 7fd56e800640 1 --2- 172.21.15.16:0/3116773331 >> [v2:172.21.15.16:6808/1398737533,v1:172.21.15.16:6810/1398737533] conn(0x7fd54c06e380 0x7fd54c0707a0 crc :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:15:17.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.129+0000 7fd56de00640 1 --2- 172.21.15.16:0/3116773331 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fd54c068380 0x7fd54c06a840 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7fd5600ad000 tx=0x7fd5600a6000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:15:17.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.129+0000 7fd566a00640 1 -- 172.21.15.16:0/3116773331 <== osd.1 v2:172.21.15.16:6808/1398737533 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fd54c070e50 con 0x7fd54c06e380 2024-03-31T15:15:17.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.145+0000 7fd574a00640 1 -- 172.21.15.16:0/3116773331 --> [v2:172.21.15.16:6808/1398737533,v1:172.21.15.16:6810/1398737533] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7fd57013e9f0 con 0x7fd54c06e380 2024-03-31T15:15:17.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.145+0000 7fd566a00640 1 -- 172.21.15.16:0/3116773331 <== osd.1 v2:172.21.15.16:6808/1398737533 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7fd57013e9f0 con 0x7fd54c06e380 2024-03-31T15:15:17.134 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-03-31T15:15:17.134 INFO:teuthology.orchestra.run.smithi016.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-31T15:15:17.134 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-03-31T15:15:17.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.145+0000 7fd574a00640 1 -- 172.21.15.16:0/3116773331 >> [v2:172.21.15.16:6808/1398737533,v1:172.21.15.16:6810/1398737533] conn(0x7fd54c06e380 msgr2=0x7fd54c0707a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:17.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.145+0000 7fd574a00640 1 --2- 172.21.15.16:0/3116773331 >> [v2:172.21.15.16:6808/1398737533,v1:172.21.15.16:6810/1398737533] conn(0x7fd54c06e380 0x7fd54c0707a0 crc :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:17.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.145+0000 7fd574a00640 1 -- 172.21.15.16:0/3116773331 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fd54c068380 msgr2=0x7fd54c06a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:17.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.145+0000 7fd574a00640 1 --2- 172.21.15.16:0/3116773331 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fd54c068380 0x7fd54c06a840 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7fd5600ad000 tx=0x7fd5600a6000 comp rx=0 tx=0).stop 2024-03-31T15:15:17.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.145+0000 7fd574a00640 1 -- 172.21.15.16:0/3116773331 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd5700b45d0 msgr2=0x7fd570140e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:17.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.145+0000 7fd574a00640 1 --2- 172.21.15.16:0/3116773331 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd5700b45d0 0x7fd570140e00 secure :-1 s=READY pgs=1905 cs=0 l=1 rev1=1 crypto rx=0x7fd55800cf60 tx=0x7fd55800cf90 comp rx=0 tx=0).stop 2024-03-31T15:15:17.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.145+0000 7fd56e800640 1 -- 172.21.15.16:0/3116773331 reap_dead start 2024-03-31T15:15:17.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.145+0000 7fd574a00640 1 -- 172.21.15.16:0/3116773331 shutdown_connections 2024-03-31T15:15:17.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.145+0000 7fd574a00640 1 -- 172.21.15.16:0/3116773331 >> 172.21.15.16:0/3116773331 conn(0x7fd57006b620 msgr2=0x7fd57015b350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:15:17.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.145+0000 7fd574a00640 1 -- 172.21.15.16:0/3116773331 shutdown_connections 2024-03-31T15:15:17.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.145+0000 7fd574a00640 1 -- 172.21.15.16:0/3116773331 wait complete. 2024-03-31T15:15:17.153 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2024-03-31T15:15:17.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.305+0000 7fc567000640 1 Processor -- start 2024-03-31T15:15:17.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.305+0000 7fc567000640 1 -- start start 2024-03-31T15:15:17.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.305+0000 7fc567000640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fc56015f650 0x7fc56006f920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:17.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.305+0000 7fc567000640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fc56006fe60 0x7fc560162180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:17.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.305+0000 7fc567000640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc5600cbe90 con 0x7fc56006dbc0 2024-03-31T15:15:17.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.305+0000 7fc567000640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7fc56015f1c0 con 0x7fc56015f650 2024-03-31T15:15:17.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.305+0000 7fc567000640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fc5600cc170 con 0x7fc56006fe60 2024-03-31T15:15:17.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.305+0000 7fc557400640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fc56015f650 0x7fc56006f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:17.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.305+0000 7fc55fe00640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fc56006dbc0 0x7fc5600cb440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:42866/0 (socket says 172.21.15.16:42866) 2024-03-31T15:15:17.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.305+0000 7fc55fe00640 1 -- 172.21.15.16:0/1767761977 learned_addr learned my addr 172.21.15.16:0/1767761977 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:15:17.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.305+0000 7fc565000640 1 --2- 172.21.15.16:0/1767761977 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fc56006fe60 0x7fc560162180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:17.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.305+0000 7fc557400640 1 -- 172.21.15.16:0/1767761977 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fc56006fe60 msgr2=0x7fc560162180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:15:17.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.305+0000 7fc557400640 1 --2- 172.21.15.16:0/1767761977 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fc56006fe60 0x7fc560162180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:17.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.305+0000 7fc557400640 1 -- 172.21.15.16:0/1767761977 >> v1:172.21.15.16:6789/0 conn(0x7fc56006dbc0 legacy=0x7fc5600cb440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:17.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.305+0000 7fc557400640 1 -- 172.21.15.16:0/1767761977 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc56015ee50 con 0x7fc56015f650 2024-03-31T15:15:17.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.305+0000 7fc565000640 1 --2- 172.21.15.16:0/1767761977 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fc56006fe60 0x7fc560162180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:15:17.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.309+0000 7fc557400640 1 --2- 172.21.15.16:0/1767761977 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc56015f650 0x7fc56006f920 secure :-1 s=READY pgs=1907 cs=0 l=1 rev1=1 crypto rx=0x7fc5480097c0 tx=0x7fc54801c410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e77968dc89c4e4cf server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:15:17.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.309+0000 7fc55f400640 1 -- 172.21.15.16:0/1767761977 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5480b0070 con 0x7fc56015f650 2024-03-31T15:15:17.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.309+0000 7fc55f400640 1 -- 172.21.15.16:0/1767761977 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc5480ab600 con 0x7fc56015f650 2024-03-31T15:15:17.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.309+0000 7fc55f400640 1 -- 172.21.15.16:0/1767761977 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5480ab9d0 con 0x7fc56015f650 2024-03-31T15:15:17.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.309+0000 7fc567000640 1 -- 172.21.15.16:0/1767761977 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc56015f650 msgr2=0x7fc56006f920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:17.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.309+0000 7fc567000640 1 --2- 172.21.15.16:0/1767761977 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc56015f650 0x7fc56006f920 secure :-1 s=READY pgs=1907 cs=0 l=1 rev1=1 crypto rx=0x7fc5480097c0 tx=0x7fc54801c410 comp rx=0 tx=0).stop 2024-03-31T15:15:17.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.309+0000 7fc567000640 1 -- 172.21.15.16:0/1767761977 shutdown_connections 2024-03-31T15:15:17.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.309+0000 7fc567000640 1 --2- 172.21.15.16:0/1767761977 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fc56006fe60 0x7fc560162180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:17.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.309+0000 7fc567000640 1 --2- 172.21.15.16:0/1767761977 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc56015f650 0x7fc56006f920 unknown :-1 s=CLOSED pgs=1907 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:17.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.309+0000 7fc567000640 1 -- 172.21.15.16:0/1767761977 >> 172.21.15.16:0/1767761977 conn(0x7fc560067330 msgr2=0x7fc560067740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:15:17.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.309+0000 7fc567000640 1 -- 172.21.15.16:0/1767761977 shutdown_connections 2024-03-31T15:15:17.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.309+0000 7fc567000640 1 -- 172.21.15.16:0/1767761977 wait complete. 2024-03-31T15:15:17.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.313+0000 7fc567000640 1 Processor -- start 2024-03-31T15:15:17.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.313+0000 7fc567000640 1 -- start start 2024-03-31T15:15:17.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.313+0000 7fc567000640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fc56006dbc0 0x7fc56015bd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:17.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.313+0000 7fc567000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc56006fe60 0x7fc56013f560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:17.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.313+0000 7fc567000640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc56015f650 0x7fc56015a520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:17.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.313+0000 7fc567000640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fc56006c950 con 0x7fc56006fe60 2024-03-31T15:15:17.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.313+0000 7fc567000640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fc56006dfa0 con 0x7fc56015f650 2024-03-31T15:15:17.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.313+0000 7fc567000640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fc56006cad0 con 0x7fc56006dbc0 2024-03-31T15:15:17.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.313+0000 7fc557400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc56006fe60 0x7fc56013f560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:17.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.313+0000 7fc557400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc56006fe60 0x7fc56013f560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37790/0 (socket says 172.21.15.16:37790) 2024-03-31T15:15:17.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.313+0000 7fc557400640 1 -- 172.21.15.16:0/764069079 learned_addr learned my addr 172.21.15.16:0/764069079 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:15:17.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.313+0000 7fc55fe00640 1 --2- 172.21.15.16:0/764069079 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fc56006dbc0 0x7fc56015bd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:17.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.313+0000 7fc565000640 1 --2- 172.21.15.16:0/764069079 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc56015f650 0x7fc56015a520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:17.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.313+0000 7fc557400640 1 -- 172.21.15.16:0/764069079 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fc56006dbc0 msgr2=0x7fc56015bd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:17.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.313+0000 7fc557400640 1 --2- 172.21.15.16:0/764069079 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fc56006dbc0 0x7fc56015bd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:17.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.313+0000 7fc557400640 1 -- 172.21.15.16:0/764069079 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc56015f650 msgr2=0x7fc56015a520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:17.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.313+0000 7fc557400640 1 --2- 172.21.15.16:0/764069079 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc56015f650 0x7fc56015a520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:17.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.313+0000 7fc557400640 1 -- 172.21.15.16:0/764069079 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc56015b360 con 0x7fc56006fe60 2024-03-31T15:15:17.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.313+0000 7fc565000640 1 --2- 172.21.15.16:0/764069079 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc56015f650 0x7fc56015a520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:15:17.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.313+0000 7fc557400640 1 --2- 172.21.15.16:0/764069079 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc56006fe60 0x7fc56013f560 secure :-1 s=READY pgs=1908 cs=0 l=1 rev1=1 crypto rx=0x7fc54801aac0 tx=0x7fc5480191b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:15:17.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.313+0000 7fc55cc00640 1 -- 172.21.15.16:0/764069079 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5480b0040 con 0x7fc56006fe60 2024-03-31T15:15:17.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.313+0000 7fc55cc00640 1 -- 172.21.15.16:0/764069079 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc5480bb070 con 0x7fc56006fe60 2024-03-31T15:15:17.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.313+0000 7fc55cc00640 1 -- 172.21.15.16:0/764069079 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5480b6670 con 0x7fc56006fe60 2024-03-31T15:15:17.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.313+0000 7fc567000640 1 -- 172.21.15.16:0/764069079 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc560145130 con 0x7fc56006fe60 2024-03-31T15:15:17.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.313+0000 7fc567000640 1 -- 172.21.15.16:0/764069079 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc56015b8e0 con 0x7fc56006fe60 2024-03-31T15:15:17.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.313+0000 7fc567000640 1 -- 172.21.15.16:0/764069079 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc560069950 con 0x7fc56006fe60 2024-03-31T15:15:17.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.317+0000 7fc55cc00640 1 -- 172.21.15.16:0/764069079 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fc5480b3070 con 0x7fc56006fe60 2024-03-31T15:15:17.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.317+0000 7fc55cc00640 1 --2- 172.21.15.16:0/764069079 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fc53c068450 0x7fc53c06a910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:17.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.317+0000 7fc55cc00640 1 -- 172.21.15.16:0/764069079 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(116..116 src has 1..116) ==== 5115+0+0 (secure 0 0 0) 0x7fc548120560 con 0x7fc56006fe60 2024-03-31T15:15:17.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.317+0000 7fc55fe00640 1 --2- 172.21.15.16:0/764069079 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fc53c068450 0x7fc53c06a910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:17.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.317+0000 7fc55fe00640 1 --2- 172.21.15.16:0/764069079 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fc53c068450 0x7fc53c06a910 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7fc550000f70 tx=0x7fc550009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:15:17.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.321+0000 7fc55cc00640 1 -- 172.21.15.16:0/764069079 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fc548016090 con 0x7fc56006fe60 2024-03-31T15:15:17.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.485+0000 7fc567000640 1 -- 172.21.15.16:0/764069079 --> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "1", "target": ["mon-mgr", ""]}) -- 0x7fc5600cf380 con 0x7fc53c068450 2024-03-31T15:15:17.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.485+0000 7fc55cc00640 1 -- 172.21.15.16:0/764069079 <== mgr.4100 v2:172.21.15.89:6832/1549407273 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 1 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7fc5600cf380 con 0x7fc53c068450 2024-03-31T15:15:17.476 INFO:teuthology.orchestra.run.smithi016.stderr:instructed osd(s) 1 to deep-scrub 2024-03-31T15:15:17.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.489+0000 7fc567000640 1 -- 172.21.15.16:0/764069079 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fc53c068450 msgr2=0x7fc53c06a910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:17.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.489+0000 7fc567000640 1 --2- 172.21.15.16:0/764069079 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fc53c068450 0x7fc53c06a910 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7fc550000f70 tx=0x7fc550009040 comp rx=0 tx=0).stop 2024-03-31T15:15:17.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.489+0000 7fc567000640 1 -- 172.21.15.16:0/764069079 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc56006fe60 msgr2=0x7fc56013f560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:17.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.489+0000 7fc567000640 1 --2- 172.21.15.16:0/764069079 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc56006fe60 0x7fc56013f560 secure :-1 s=READY pgs=1908 cs=0 l=1 rev1=1 crypto rx=0x7fc54801aac0 tx=0x7fc5480191b0 comp rx=0 tx=0).stop 2024-03-31T15:15:17.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.489+0000 7fc567000640 1 -- 172.21.15.16:0/764069079 shutdown_connections 2024-03-31T15:15:17.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.489+0000 7fc567000640 1 --2- 172.21.15.16:0/764069079 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fc53c068450 0x7fc53c06a910 unknown :-1 s=CLOSED pgs=802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:17.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.489+0000 7fc567000640 1 --2- 172.21.15.16:0/764069079 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc56015f650 0x7fc56015a520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:17.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.489+0000 7fc567000640 1 --2- 172.21.15.16:0/764069079 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc56006fe60 0x7fc56013f560 unknown :-1 s=CLOSED pgs=1908 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:17.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.489+0000 7fc567000640 1 --2- 172.21.15.16:0/764069079 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fc56006dbc0 0x7fc56015bd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:17.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.489+0000 7fc567000640 1 -- 172.21.15.16:0/764069079 >> 172.21.15.16:0/764069079 conn(0x7fc560067330 msgr2=0x7fc56015c920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:15:17.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.489+0000 7fc567000640 1 -- 172.21.15.16:0/764069079 shutdown_connections 2024-03-31T15:15:17.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.489+0000 7fc567000640 1 -- 172.21.15.16:0/764069079 wait complete. 2024-03-31T15:15:17.498 INFO:tasks.ceph:Scrubbing osd.2 2024-03-31T15:15:17.498 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-03-31T15:15:17.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.653+0000 7fbdf2400640 1 Processor -- start 2024-03-31T15:15:17.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.653+0000 7fbdf2400640 1 -- start start 2024-03-31T15:15:17.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.653+0000 7fbdf2400640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fbdec0cc7b0 0x7fbdec0b3dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:17.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.653+0000 7fbdf2400640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fbdec0b4300 0x7fbdec0d1a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:17.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.653+0000 7fbdf2400640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbdec0ce2d0 con 0x7fbdec0b7900 2024-03-31T15:15:17.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.653+0000 7fbdf2400640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7fbdec0b4770 con 0x7fbdec0b4300 2024-03-31T15:15:17.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.653+0000 7fbdf2400640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fbdec0ceb10 con 0x7fbdec0cc7b0 2024-03-31T15:15:17.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.653+0000 7fbdeb400640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fbdec0b7900 0x7fbdec160740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:42870/0 (socket says 172.21.15.16:42870) 2024-03-31T15:15:17.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.653+0000 7fbdeb400640 1 -- 172.21.15.16:0/3619486092 learned_addr learned my addr 172.21.15.16:0/3619486092 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:15:17.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.657+0000 7fbde2a00640 1 --2- 172.21.15.16:0/3619486092 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fbdec0cc7b0 0x7fbdec0b3dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:17.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.657+0000 7fbdeaa00640 1 -- 172.21.15.16:0/3619486092 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 35398878 0 0) 0x7fbddc001120 con 0x7fbdec0b7900 2024-03-31T15:15:17.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.657+0000 7fbdebe00640 1 --2- 172.21.15.16:0/3619486092 >> v2:172.21.15.89:3300/0 conn(0x7fbdec0b4300 0x7fbdec0d1a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:17.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.657+0000 7fbdeaa00640 1 -- 172.21.15.16:0/3619486092 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbdd0003530 con 0x7fbdec0b7900 2024-03-31T15:15:17.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.657+0000 7fbdeaa00640 1 -- 172.21.15.16:0/3619486092 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1604895513 0 0) 0x7fbdd0003530 con 0x7fbdec0b7900 2024-03-31T15:15:17.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.657+0000 7fbdeaa00640 1 -- 172.21.15.16:0/3619486092 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fbdec0cc7b0 msgr2=0x7fbdec0b3dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:15:17.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.657+0000 7fbdeaa00640 1 --2- 172.21.15.16:0/3619486092 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fbdec0cc7b0 0x7fbdec0b3dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:17.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.657+0000 7fbdeaa00640 1 -- 172.21.15.16:0/3619486092 >> v2:172.21.15.89:3300/0 conn(0x7fbdec0b4300 msgr2=0x7fbdec0d1a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:15:17.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.657+0000 7fbdeaa00640 1 --2- 172.21.15.16:0/3619486092 >> v2:172.21.15.89:3300/0 conn(0x7fbdec0b4300 0x7fbdec0d1a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:17.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.657+0000 7fbdeaa00640 1 -- 172.21.15.16:0/3619486092 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbdec0ce550 con 0x7fbdec0b7900 2024-03-31T15:15:17.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.657+0000 7fbdeaa00640 1 -- 172.21.15.16:0/3619486092 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 214807623 0 0) 0x7fbddc001850 con 0x7fbdec0b7900 2024-03-31T15:15:17.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.657+0000 7fbdeaa00640 1 -- 172.21.15.16:0/3619486092 >> v1:172.21.15.16:6789/0 conn(0x7fbdec0b7900 legacy=0x7fbdec160740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:17.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.657+0000 7fbdeaa00640 1 --2- 172.21.15.16:0/3619486092 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fbdec0cc7b0 0x7fbdd00039c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:17.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.657+0000 7fbdeaa00640 1 --2- 172.21.15.16:0/3619486092 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbdd0003f40 0x7fbdd0004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:17.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.657+0000 7fbdeaa00640 1 --2- 172.21.15.16:0/3619486092 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbdd0004d50 0x7fbdd00051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:17.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.657+0000 7fbdeaa00640 1 -- 172.21.15.16:0/3619486092 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fbdec0b4770 con 0x7fbdd0003f40 2024-03-31T15:15:17.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.657+0000 7fbdeaa00640 1 -- 172.21.15.16:0/3619486092 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fbdec0ceb10 con 0x7fbdd0004d50 2024-03-31T15:15:17.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.657+0000 7fbdeaa00640 1 -- 172.21.15.16:0/3619486092 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fbdd000ba20 con 0x7fbdec0cc7b0 2024-03-31T15:15:17.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.657+0000 7fbde2a00640 1 --2- 172.21.15.16:0/3619486092 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbdd0003f40 0x7fbdd0004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:17.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.657+0000 7fbdebe00640 1 --2- 172.21.15.16:0/3619486092 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbdd0004d50 0x7fbdd00051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:17.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.657+0000 7fbdeb400640 1 --2- 172.21.15.16:0/3619486092 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fbdec0cc7b0 0x7fbdd00039c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:17.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.657+0000 7fbde2a00640 1 -- 172.21.15.16:0/3619486092 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fbdec0cc7b0 msgr2=0x7fbdd00039c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:15:17.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.657+0000 7fbde2a00640 1 --2- 172.21.15.16:0/3619486092 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fbdec0cc7b0 0x7fbdd00039c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:17.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.657+0000 7fbde2a00640 1 -- 172.21.15.16:0/3619486092 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbdd0004d50 msgr2=0x7fbdd00051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:15:17.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.657+0000 7fbde2a00640 1 --2- 172.21.15.16:0/3619486092 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbdd0004d50 0x7fbdd00051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:17.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.657+0000 7fbde2a00640 1 -- 172.21.15.16:0/3619486092 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbdd0003530 con 0x7fbdd0003f40 2024-03-31T15:15:17.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.661+0000 7fbde2a00640 1 --2- 172.21.15.16:0/3619486092 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbdd0003f40 0x7fbdd0004810 secure :-1 s=READY pgs=1910 cs=0 l=1 rev1=1 crypto rx=0x7fbdd4009b10 tx=0x7fbdec0d27d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5bcb04ca6c2466ca server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:15:17.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.661+0000 7fbdeaa00640 1 -- 172.21.15.16:0/3619486092 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbdd40b2070 con 0x7fbdd0003f40 2024-03-31T15:15:17.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.661+0000 7fbdeaa00640 1 -- 172.21.15.16:0/3619486092 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbdd40a5520 con 0x7fbdd0003f40 2024-03-31T15:15:17.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.661+0000 7fbdeaa00640 1 -- 172.21.15.16:0/3619486092 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbdd40096b0 con 0x7fbdd0003f40 2024-03-31T15:15:17.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.661+0000 7fbdf2400640 1 -- 172.21.15.16:0/3619486092 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbdd0003f40 msgr2=0x7fbdd0004810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:17.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.661+0000 7fbdf2400640 1 --2- 172.21.15.16:0/3619486092 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbdd0003f40 0x7fbdd0004810 secure :-1 s=READY pgs=1910 cs=0 l=1 rev1=1 crypto rx=0x7fbdd4009b10 tx=0x7fbdec0d27d0 comp rx=0 tx=0).stop 2024-03-31T15:15:17.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.661+0000 7fbdebe00640 1 -- 172.21.15.16:0/3619486092 reap_dead start 2024-03-31T15:15:17.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.661+0000 7fbdf2400640 1 -- 172.21.15.16:0/3619486092 shutdown_connections 2024-03-31T15:15:17.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.661+0000 7fbdf2400640 1 -- 172.21.15.16:0/3619486092 >> 172.21.15.16:0/3619486092 conn(0x7fbdec06b7f0 msgr2=0x7fbdec067330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:15:17.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.661+0000 7fbdf2400640 1 -- 172.21.15.16:0/3619486092 shutdown_connections 2024-03-31T15:15:17.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.661+0000 7fbdf2400640 1 -- 172.21.15.16:0/3619486092 wait complete. 2024-03-31T15:15:17.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.661+0000 7fbdf2400640 1 Processor -- start 2024-03-31T15:15:17.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.661+0000 7fbdf2400640 1 -- start start 2024-03-31T15:15:17.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.661+0000 7fbdf2400640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbdd0003f40 0x7fbdec140bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:17.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.661+0000 7fbdf2400640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fbdec160f20 0x7fbdec0b6850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:17.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.661+0000 7fbdf2400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbdec158080 0x7fbdec158500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:17.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.661+0000 7fbdf2400640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fbdec15d540 con 0x7fbdec158080 2024-03-31T15:15:17.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.661+0000 7fbdf2400640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fbdec0b7ce0 con 0x7fbdd0003f40 2024-03-31T15:15:17.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.661+0000 7fbdf2400640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fbdec15d6c0 con 0x7fbdec160f20 2024-03-31T15:15:17.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.665+0000 7fbdebe00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbdec158080 0x7fbdec158500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:17.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.665+0000 7fbdebe00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbdec158080 0x7fbdec158500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37820/0 (socket says 172.21.15.16:37820) 2024-03-31T15:15:17.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.665+0000 7fbdebe00640 1 -- 172.21.15.16:0/3514177725 learned_addr learned my addr 172.21.15.16:0/3514177725 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:15:17.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.665+0000 7fbdeb400640 1 --2- 172.21.15.16:0/3514177725 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbdd0003f40 0x7fbdec140bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:17.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.665+0000 7fbde2a00640 1 --2- 172.21.15.16:0/3514177725 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fbdec160f20 0x7fbdec0b6850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:17.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.665+0000 7fbdebe00640 1 -- 172.21.15.16:0/3514177725 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fbdec160f20 msgr2=0x7fbdec0b6850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:17.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.665+0000 7fbdebe00640 1 --2- 172.21.15.16:0/3514177725 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fbdec160f20 0x7fbdec0b6850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:17.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.665+0000 7fbdebe00640 1 -- 172.21.15.16:0/3514177725 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbdd0003f40 msgr2=0x7fbdec140bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:17.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.665+0000 7fbdebe00640 1 --2- 172.21.15.16:0/3514177725 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbdd0003f40 0x7fbdec140bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:17.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.665+0000 7fbdebe00640 1 -- 172.21.15.16:0/3514177725 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbdec1614c0 con 0x7fbdec158080 2024-03-31T15:15:17.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.665+0000 7fbdebe00640 1 --2- 172.21.15.16:0/3514177725 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbdec158080 0x7fbdec158500 secure :-1 s=READY pgs=1911 cs=0 l=1 rev1=1 crypto rx=0x7fbdec067c80 tx=0x7fbde400f1b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:15:17.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.665+0000 7fbde3e00640 1 -- 172.21.15.16:0/3514177725 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbde400fd10 con 0x7fbdec158080 2024-03-31T15:15:17.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.665+0000 7fbde3e00640 1 -- 172.21.15.16:0/3514177725 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbde401d070 con 0x7fbdec158080 2024-03-31T15:15:17.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.665+0000 7fbde3e00640 1 -- 172.21.15.16:0/3514177725 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbde4015670 con 0x7fbdec158080 2024-03-31T15:15:17.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.665+0000 7fbdf2400640 1 -- 172.21.15.16:0/3514177725 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbdec1617a0 con 0x7fbdec158080 2024-03-31T15:15:17.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.665+0000 7fbdf2400640 1 -- 172.21.15.16:0/3514177725 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbdec161d60 con 0x7fbdec158080 2024-03-31T15:15:17.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.665+0000 7fbdf2400640 1 -- 172.21.15.16:0/3514177725 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fbdec069950 con 0x7fbdec158080 2024-03-31T15:15:17.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.665+0000 7fbde3e00640 1 -- 172.21.15.16:0/3514177725 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fbde4015810 con 0x7fbdec158080 2024-03-31T15:15:17.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.665+0000 7fbde3e00640 1 --2- 172.21.15.16:0/3514177725 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fbdc8068380 0x7fbdc806a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:17.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.669+0000 7fbde3e00640 1 -- 172.21.15.16:0/3514177725 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(116..116 src has 1..116) ==== 5115+0+0 (secure 0 0 0) 0x7fbde4086f50 con 0x7fbdec158080 2024-03-31T15:15:17.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.669+0000 7fbde3e00640 1 --2- 172.21.15.16:0/3514177725 >> [v2:172.21.15.16:6819/3018526646,v1:172.21.15.16:6822/3018526646] conn(0x7fbdc806e380 0x7fbdc80707a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:17.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.669+0000 7fbde3e00640 1 -- 172.21.15.16:0/3514177725 --> [v2:172.21.15.16:6819/3018526646,v1:172.21.15.16:6822/3018526646] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fbde400ce90 con 0x7fbdc806e380 2024-03-31T15:15:17.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.669+0000 7fbde3e00640 1 -- 172.21.15.16:0/3514177725 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_get_version_reply(handle=1 version=116) ==== 24+0+0 (secure 0 0 0) 0x7fbde40895f0 con 0x7fbdec158080 2024-03-31T15:15:17.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.669+0000 7fbdeb400640 1 --2- 172.21.15.16:0/3514177725 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fbdc8068380 0x7fbdc806a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:17.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.669+0000 7fbde2a00640 1 --2- 172.21.15.16:0/3514177725 >> [v2:172.21.15.16:6819/3018526646,v1:172.21.15.16:6822/3018526646] conn(0x7fbdc806e380 0x7fbdc80707a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:17.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.669+0000 7fbde2a00640 1 --2- 172.21.15.16:0/3514177725 >> [v2:172.21.15.16:6819/3018526646,v1:172.21.15.16:6822/3018526646] conn(0x7fbdc806e380 0x7fbdc80707a0 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:15:17.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.669+0000 7fbdeb400640 1 --2- 172.21.15.16:0/3514177725 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fbdc8068380 0x7fbdc806a840 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7fbddc001300 tx=0x7fbddc007ad0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:15:17.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.669+0000 7fbde3e00640 1 -- 172.21.15.16:0/3514177725 <== osd.2 v2:172.21.15.16:6819/3018526646 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fbde400ce90 con 0x7fbdc806e380 2024-03-31T15:15:17.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.681+0000 7fbdf2400640 1 -- 172.21.15.16:0/3514177725 --> [v2:172.21.15.16:6819/3018526646,v1:172.21.15.16:6822/3018526646] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7fbdec15f740 con 0x7fbdc806e380 2024-03-31T15:15:17.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.681+0000 7fbde3e00640 1 -- 172.21.15.16:0/3514177725 <== osd.2 v2:172.21.15.16:6819/3018526646 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7fbdec15f740 con 0x7fbdc806e380 2024-03-31T15:15:17.672 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-03-31T15:15:17.672 INFO:teuthology.orchestra.run.smithi016.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-31T15:15:17.672 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-03-31T15:15:17.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.681+0000 7fbdf2400640 1 -- 172.21.15.16:0/3514177725 >> [v2:172.21.15.16:6819/3018526646,v1:172.21.15.16:6822/3018526646] conn(0x7fbdc806e380 msgr2=0x7fbdc80707a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:17.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.681+0000 7fbdf2400640 1 --2- 172.21.15.16:0/3514177725 >> [v2:172.21.15.16:6819/3018526646,v1:172.21.15.16:6822/3018526646] conn(0x7fbdc806e380 0x7fbdc80707a0 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:17.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.681+0000 7fbdf2400640 1 -- 172.21.15.16:0/3514177725 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fbdc8068380 msgr2=0x7fbdc806a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:17.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.681+0000 7fbdf2400640 1 --2- 172.21.15.16:0/3514177725 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fbdc8068380 0x7fbdc806a840 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7fbddc001300 tx=0x7fbddc007ad0 comp rx=0 tx=0).stop 2024-03-31T15:15:17.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.681+0000 7fbdf2400640 1 -- 172.21.15.16:0/3514177725 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbdec158080 msgr2=0x7fbdec158500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:17.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.681+0000 7fbdf2400640 1 --2- 172.21.15.16:0/3514177725 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbdec158080 0x7fbdec158500 secure :-1 s=READY pgs=1911 cs=0 l=1 rev1=1 crypto rx=0x7fbdec067c80 tx=0x7fbde400f1b0 comp rx=0 tx=0).stop 2024-03-31T15:15:17.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.681+0000 7fbdebe00640 1 -- 172.21.15.16:0/3514177725 reap_dead start 2024-03-31T15:15:17.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.681+0000 7fbdf2400640 1 -- 172.21.15.16:0/3514177725 shutdown_connections 2024-03-31T15:15:17.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.681+0000 7fbdf2400640 1 -- 172.21.15.16:0/3514177725 >> 172.21.15.16:0/3514177725 conn(0x7fbdec06b7f0 msgr2=0x7fbdec067330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:15:17.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.685+0000 7fbdf2400640 1 -- 172.21.15.16:0/3514177725 shutdown_connections 2024-03-31T15:15:17.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.685+0000 7fbdf2400640 1 -- 172.21.15.16:0/3514177725 wait complete. 2024-03-31T15:15:17.690 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2024-03-31T15:15:17.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.841+0000 7f46ef800640 1 Processor -- start 2024-03-31T15:15:17.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.841+0000 7f46ef800640 1 -- start start 2024-03-31T15:15:17.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.845+0000 7f46ef800640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f46e815b3d0 0x7f46e80b1cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:17.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.845+0000 7f46ef800640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f46e815eee0 0x7f46e80b2200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:17.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.845+0000 7f46ef800640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f46e815f3c0 con 0x7f46e80ce2f0 2024-03-31T15:15:17.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.845+0000 7f46ef800640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f46e815b7b0 con 0x7f46e815b3d0 2024-03-31T15:15:17.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.845+0000 7f46ef800640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f46e815f6a0 con 0x7f46e815eee0 2024-03-31T15:15:17.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.845+0000 7f46ece00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f46e815b3d0 0x7f46e80b1cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:17.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.845+0000 7f46e7e00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f46e815eee0 0x7f46e80b2200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:17.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.845+0000 7f46ed800640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f46e80ce2f0 0x7f46e80cc9c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.16:48996/0 (socket says 172.21.15.16:48996) 2024-03-31T15:15:17.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.845+0000 7f46ed800640 1 -- 172.21.15.16:0/934695705 learned_addr learned my addr 172.21.15.16:0/934695705 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:15:17.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.845+0000 7f46ece00640 1 -- 172.21.15.16:0/934695705 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f46e815eee0 msgr2=0x7f46e80b2200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:15:17.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.845+0000 7f46ece00640 1 --2- 172.21.15.16:0/934695705 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f46e815eee0 0x7f46e80b2200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:17.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.845+0000 7f46ece00640 1 -- 172.21.15.16:0/934695705 >> v1:172.21.15.89:6789/0 conn(0x7f46e80ce2f0 legacy=0x7f46e80cc9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:17.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.845+0000 7f46ece00640 1 -- 172.21.15.16:0/934695705 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46e815f980 con 0x7f46e815b3d0 2024-03-31T15:15:17.836 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.849+0000 7f46ece00640 1 --2- 172.21.15.16:0/934695705 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f46e815b3d0 0x7f46e80b1cc0 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f46dc0097c0 tx=0x7f46dc01c470 comp rx=0 tx=0).ready entity=mon.1 client_cookie=80787b2849db60e2 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:15:17.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.849+0000 7f46e7400640 1 -- 172.21.15.16:0/934695705 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46dc0ab430 con 0x7f46e815b3d0 2024-03-31T15:15:17.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.849+0000 7f46e7400640 1 -- 172.21.15.16:0/934695705 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f46dc0b0070 con 0x7f46e815b3d0 2024-03-31T15:15:17.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.849+0000 7f46e7400640 1 -- 172.21.15.16:0/934695705 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46dc0ab7f0 con 0x7f46e815b3d0 2024-03-31T15:15:17.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.849+0000 7f46ef800640 1 -- 172.21.15.16:0/934695705 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f46e815b3d0 msgr2=0x7f46e80b1cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:17.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.849+0000 7f46ef800640 1 --2- 172.21.15.16:0/934695705 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f46e815b3d0 0x7f46e80b1cc0 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f46dc0097c0 tx=0x7f46dc01c470 comp rx=0 tx=0).stop 2024-03-31T15:15:17.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.849+0000 7f46ef800640 1 -- 172.21.15.16:0/934695705 shutdown_connections 2024-03-31T15:15:17.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.849+0000 7f46ef800640 1 --2- 172.21.15.16:0/934695705 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f46e815eee0 0x7f46e80b2200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:17.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.849+0000 7f46ef800640 1 --2- 172.21.15.16:0/934695705 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f46e815b3d0 0x7f46e80b1cc0 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:17.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.849+0000 7f46ef800640 1 -- 172.21.15.16:0/934695705 >> 172.21.15.16:0/934695705 conn(0x7f46e806b620 msgr2=0x7f46e806ba30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:15:17.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.849+0000 7f46ef800640 1 -- 172.21.15.16:0/934695705 shutdown_connections 2024-03-31T15:15:17.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.849+0000 7f46ef800640 1 -- 172.21.15.16:0/934695705 wait complete. 2024-03-31T15:15:17.839 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.849+0000 7f46ef800640 1 Processor -- start 2024-03-31T15:15:17.839 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.849+0000 7f46ef800640 1 -- start start 2024-03-31T15:15:17.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.849+0000 7f46ef800640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f46e80ce2f0 0x7f46e80ceff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:17.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.849+0000 7f46ef800640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f46e815b3d0 0x7f46e80b0d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:17.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.849+0000 7f46ef800640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f46e815eee0 0x7f46e8148830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:17.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.849+0000 7f46ef800640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f46e8166600 con 0x7f46e815eee0 2024-03-31T15:15:17.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.849+0000 7f46ef800640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f46e8166480 con 0x7f46e80ce2f0 2024-03-31T15:15:17.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.849+0000 7f46ef800640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f46e8166780 con 0x7f46e815b3d0 2024-03-31T15:15:17.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.849+0000 7f46ed800640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f46e815eee0 0x7f46e8148830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:17.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.849+0000 7f46ed800640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f46e815eee0 0x7f46e8148830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37828/0 (socket says 172.21.15.16:37828) 2024-03-31T15:15:17.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.849+0000 7f46e7e00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f46e815b3d0 0x7f46e80b0d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:17.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.849+0000 7f46ed800640 1 -- 172.21.15.16:0/1090856289 learned_addr learned my addr 172.21.15.16:0/1090856289 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:15:17.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.849+0000 7f46ece00640 1 --2- 172.21.15.16:0/1090856289 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f46e80ce2f0 0x7f46e80ceff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:17.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.853+0000 7f46ece00640 1 -- 172.21.15.16:0/1090856289 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f46e815b3d0 msgr2=0x7f46e80b0d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:17.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.853+0000 7f46ece00640 1 --2- 172.21.15.16:0/1090856289 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f46e815b3d0 0x7f46e80b0d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:17.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.853+0000 7f46ece00640 1 -- 172.21.15.16:0/1090856289 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f46e815eee0 msgr2=0x7f46e8148830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:17.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.853+0000 7f46ece00640 1 --2- 172.21.15.16:0/1090856289 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f46e815eee0 0x7f46e8148830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:17.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.853+0000 7f46ece00640 1 -- 172.21.15.16:0/1090856289 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46e80cf940 con 0x7f46e80ce2f0 2024-03-31T15:15:17.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.853+0000 7f46ed800640 1 --2- 172.21.15.16:0/1090856289 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f46e815eee0 0x7f46e8148830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:15:17.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.853+0000 7f46ece00640 1 --2- 172.21.15.16:0/1090856289 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f46e80ce2f0 0x7f46e80ceff0 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f46dc0ac000 tx=0x7f46dc019210 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:15:17.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.853+0000 7f46e4c00640 1 -- 172.21.15.16:0/1090856289 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46dc0bc420 con 0x7f46e80ce2f0 2024-03-31T15:15:17.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.853+0000 7f46e4c00640 1 -- 172.21.15.16:0/1090856289 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f46dc0bc600 con 0x7f46e80ce2f0 2024-03-31T15:15:17.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.853+0000 7f46ef800640 1 -- 172.21.15.16:0/1090856289 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f46e80cfc20 con 0x7f46e80ce2f0 2024-03-31T15:15:17.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.853+0000 7f46e4c00640 1 -- 172.21.15.16:0/1090856289 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46dc0bc9f0 con 0x7f46e80ce2f0 2024-03-31T15:15:17.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.853+0000 7f46ef800640 1 -- 172.21.15.16:0/1090856289 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f46e80d01e0 con 0x7f46e80ce2f0 2024-03-31T15:15:17.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.853+0000 7f46ef800640 1 -- 172.21.15.16:0/1090856289 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f46e8069950 con 0x7f46e80ce2f0 2024-03-31T15:15:17.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.853+0000 7f46e4c00640 1 -- 172.21.15.16:0/1090856289 <== mon.1 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f46dc0c3060 con 0x7f46e80ce2f0 2024-03-31T15:15:17.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.857+0000 7f46e4c00640 1 --2- 172.21.15.16:0/1090856289 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f46c8068380 0x7f46c806a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:17.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.857+0000 7f46e4c00640 1 -- 172.21.15.16:0/1090856289 <== mon.1 v2:172.21.15.89:3300/0 5 ==== osd_map(116..116 src has 1..116) ==== 5115+0+0 (secure 0 0 0) 0x7f46dc120530 con 0x7f46e80ce2f0 2024-03-31T15:15:17.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.857+0000 7f46e7e00640 1 --2- 172.21.15.16:0/1090856289 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f46c8068380 0x7f46c806a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:17.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.857+0000 7f46e7e00640 1 --2- 172.21.15.16:0/1090856289 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f46c8068380 0x7f46c806a840 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7f46e80b1b60 tx=0x7f46d4009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:15:17.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:17.857+0000 7f46e4c00640 1 -- 172.21.15.16:0/1090856289 <== mon.1 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f46dc016090 con 0x7f46e80ce2f0 2024-03-31T15:15:18.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.025+0000 7f46ef800640 1 -- 172.21.15.16:0/1090856289 --> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "2", "target": ["mon-mgr", ""]}) -- 0x7f46e81520c0 con 0x7f46c8068380 2024-03-31T15:15:18.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.025+0000 7f46e4c00640 1 -- 172.21.15.16:0/1090856289 <== mgr.4100 v2:172.21.15.89:6832/1549407273 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 2 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f46e81520c0 con 0x7f46c8068380 2024-03-31T15:15:18.016 INFO:teuthology.orchestra.run.smithi016.stderr:instructed osd(s) 2 to deep-scrub 2024-03-31T15:15:18.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.029+0000 7f46ef800640 1 -- 172.21.15.16:0/1090856289 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f46c8068380 msgr2=0x7f46c806a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:18.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.029+0000 7f46ef800640 1 --2- 172.21.15.16:0/1090856289 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f46c8068380 0x7f46c806a840 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7f46e80b1b60 tx=0x7f46d4009190 comp rx=0 tx=0).stop 2024-03-31T15:15:18.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.029+0000 7f46ef800640 1 -- 172.21.15.16:0/1090856289 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f46e80ce2f0 msgr2=0x7f46e80ceff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:18.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.029+0000 7f46ef800640 1 --2- 172.21.15.16:0/1090856289 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f46e80ce2f0 0x7f46e80ceff0 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f46dc0ac000 tx=0x7f46dc019210 comp rx=0 tx=0).stop 2024-03-31T15:15:18.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.029+0000 7f46ef800640 1 -- 172.21.15.16:0/1090856289 shutdown_connections 2024-03-31T15:15:18.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.029+0000 7f46ef800640 1 --2- 172.21.15.16:0/1090856289 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f46c8068380 0x7f46c806a840 unknown :-1 s=CLOSED pgs=804 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:18.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.029+0000 7f46ef800640 1 --2- 172.21.15.16:0/1090856289 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f46e815eee0 0x7f46e8148830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:18.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.029+0000 7f46ef800640 1 --2- 172.21.15.16:0/1090856289 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f46e815b3d0 0x7f46e80b0d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:18.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.029+0000 7f46ef800640 1 --2- 172.21.15.16:0/1090856289 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f46e80ce2f0 0x7f46e80ceff0 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:18.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.029+0000 7f46ef800640 1 -- 172.21.15.16:0/1090856289 >> 172.21.15.16:0/1090856289 conn(0x7f46e806b620 msgr2=0x7f46e80b1ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:15:18.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.029+0000 7f46ef800640 1 -- 172.21.15.16:0/1090856289 shutdown_connections 2024-03-31T15:15:18.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.029+0000 7f46ef800640 1 -- 172.21.15.16:0/1090856289 wait complete. 2024-03-31T15:15:18.037 INFO:tasks.ceph:Scrubbing osd.3 2024-03-31T15:15:18.038 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2024-03-31T15:15:18.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.193+0000 7f0ca4a00640 1 Processor -- start 2024-03-31T15:15:18.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.193+0000 7f0ca4a00640 1 -- start start 2024-03-31T15:15:18.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.193+0000 7f0ca4a00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f0ca015c390 0x7f0ca00cfb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:18.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.193+0000 7f0ca4a00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f0ca006fbc0 0x7f0ca0070070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:18.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.193+0000 7f0ca4a00640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0ca00cd750 con 0x7f0ca00d2b40 2024-03-31T15:15:18.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.193+0000 7f0ca4a00640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f0ca015c770 con 0x7f0ca015c390 2024-03-31T15:15:18.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.193+0000 7f0ca4a00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f0ca00cda30 con 0x7f0ca006fbc0 2024-03-31T15:15:18.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.193+0000 7f0c9e800640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f0ca015c390 0x7f0ca00cfb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:18.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.193+0000 7f0c95e00640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f0ca00d2b40 0x7f0ca00d00a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.16:48998/0 (socket says 172.21.15.16:48998) 2024-03-31T15:15:18.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.193+0000 7f0c9f200640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f0ca006fbc0 0x7f0ca0070070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:18.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.193+0000 7f0c95e00640 1 -- 172.21.15.16:0/558047149 learned_addr learned my addr 172.21.15.16:0/558047149 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:15:18.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.197+0000 7f0c9f200640 1 -- 172.21.15.16:0/558047149 >> v2:172.21.15.89:3300/0 conn(0x7f0ca015c390 msgr2=0x7f0ca00cfb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:15:18.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.197+0000 7f0c9f200640 1 --2- 172.21.15.16:0/558047149 >> v2:172.21.15.89:3300/0 conn(0x7f0ca015c390 0x7f0ca00cfb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:18.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.197+0000 7f0c9f200640 1 -- 172.21.15.16:0/558047149 >> v1:172.21.15.89:6789/0 conn(0x7f0ca00d2b40 legacy=0x7f0ca00d00a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:18.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.197+0000 7f0c9f200640 1 -- 172.21.15.16:0/558047149 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ca00cdd10 con 0x7f0ca006fbc0 2024-03-31T15:15:18.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.197+0000 7f0c9de00640 1 -- 172.21.15.16:0/558047149 <== mon.1 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2433871639 0 0) 0x7f0ca00cd750 con 0x7f0ca00d2b40 2024-03-31T15:15:18.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.197+0000 7f0c9f200640 1 --2- 172.21.15.16:0/558047149 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f0ca006fbc0 0x7f0ca0070070 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f0c980097c0 tx=0x7f0c9801c4a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f030426a9ff37acb server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:15:18.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.197+0000 7f0c9de00640 1 -- 172.21.15.16:0/558047149 <== mon.2 v2:172.21.15.89:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c980b0070 con 0x7f0ca006fbc0 2024-03-31T15:15:18.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.197+0000 7f0c9de00640 1 -- 172.21.15.16:0/558047149 <== mon.2 v2:172.21.15.89:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0c980ab600 con 0x7f0ca006fbc0 2024-03-31T15:15:18.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.197+0000 7f0c9de00640 1 -- 172.21.15.16:0/558047149 <== mon.2 v2:172.21.15.89:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c980ab9d0 con 0x7f0ca006fbc0 2024-03-31T15:15:18.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.201+0000 7f0ca4a00640 1 -- 172.21.15.16:0/558047149 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f0ca006fbc0 msgr2=0x7f0ca0070070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:18.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.201+0000 7f0ca4a00640 1 --2- 172.21.15.16:0/558047149 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f0ca006fbc0 0x7f0ca0070070 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f0c980097c0 tx=0x7f0c9801c4a0 comp rx=0 tx=0).stop 2024-03-31T15:15:18.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.201+0000 7f0ca4a00640 1 -- 172.21.15.16:0/558047149 shutdown_connections 2024-03-31T15:15:18.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.201+0000 7f0ca4a00640 1 --2- 172.21.15.16:0/558047149 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f0ca006fbc0 0x7f0ca0070070 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:18.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.201+0000 7f0ca4a00640 1 --2- 172.21.15.16:0/558047149 >> v2:172.21.15.89:3300/0 conn(0x7f0ca015c390 0x7f0ca00cfb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:18.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.201+0000 7f0ca4a00640 1 -- 172.21.15.16:0/558047149 >> 172.21.15.16:0/558047149 conn(0x7f0ca00d1f90 msgr2=0x7f0ca00d23a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:15:18.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.201+0000 7f0ca4a00640 1 -- 172.21.15.16:0/558047149 shutdown_connections 2024-03-31T15:15:18.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.201+0000 7f0ca4a00640 1 -- 172.21.15.16:0/558047149 wait complete. 2024-03-31T15:15:18.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.201+0000 7f0ca4a00640 1 Processor -- start 2024-03-31T15:15:18.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.201+0000 7f0ca4a00640 1 -- start start 2024-03-31T15:15:18.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.201+0000 7f0ca4a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0ca006fbc0 0x7f0ca014c880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:18.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.201+0000 7f0ca4a00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f0ca00d2b40 0x7f0ca00c7d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:18.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.201+0000 7f0ca4a00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0ca015c390 0x7f0ca006ca30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:18.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.201+0000 7f0ca4a00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f0ca0168fd0 con 0x7f0ca006fbc0 2024-03-31T15:15:18.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.201+0000 7f0ca4a00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f0ca0168e50 con 0x7f0ca015c390 2024-03-31T15:15:18.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.201+0000 7f0ca4a00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f0ca0169150 con 0x7f0ca00d2b40 2024-03-31T15:15:18.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.201+0000 7f0c9e800640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0ca006fbc0 0x7f0ca014c880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:18.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.201+0000 7f0c9e800640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0ca006fbc0 0x7f0ca014c880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37832/0 (socket says 172.21.15.16:37832) 2024-03-31T15:15:18.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.201+0000 7f0c9e800640 1 -- 172.21.15.16:0/2666509418 learned_addr learned my addr 172.21.15.16:0/2666509418 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:15:18.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.201+0000 7f0c95e00640 1 --2- 172.21.15.16:0/2666509418 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f0ca00d2b40 0x7f0ca00c7d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:18.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.201+0000 7f0c9f200640 1 --2- 172.21.15.16:0/2666509418 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0ca015c390 0x7f0ca006ca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:18.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.201+0000 7f0c9e800640 1 -- 172.21.15.16:0/2666509418 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f0ca00d2b40 msgr2=0x7f0ca00c7d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:18.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.201+0000 7f0c9e800640 1 --2- 172.21.15.16:0/2666509418 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f0ca00d2b40 0x7f0ca00c7d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:18.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.201+0000 7f0c9e800640 1 -- 172.21.15.16:0/2666509418 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0ca015c390 msgr2=0x7f0ca006ca30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:18.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.201+0000 7f0c9e800640 1 --2- 172.21.15.16:0/2666509418 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0ca015c390 0x7f0ca006ca30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:18.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.201+0000 7f0c9e800640 1 -- 172.21.15.16:0/2666509418 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ca00c8800 con 0x7f0ca006fbc0 2024-03-31T15:15:18.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.201+0000 7f0c9f200640 1 --2- 172.21.15.16:0/2666509418 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0ca015c390 0x7f0ca006ca30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:15:18.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.205+0000 7f0c9e800640 1 --2- 172.21.15.16:0/2666509418 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0ca006fbc0 0x7f0ca014c880 secure :-1 s=READY pgs=1912 cs=0 l=1 rev1=1 crypto rx=0x7f0c90009d10 tx=0x7f0c9000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:15:18.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.205+0000 7f0c97400640 1 -- 172.21.15.16:0/2666509418 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c90018020 con 0x7f0ca006fbc0 2024-03-31T15:15:18.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.205+0000 7f0c97400640 1 -- 172.21.15.16:0/2666509418 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0c90014480 con 0x7f0ca006fbc0 2024-03-31T15:15:18.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.205+0000 7f0ca4a00640 1 -- 172.21.15.16:0/2666509418 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0ca00c9060 con 0x7f0ca006fbc0 2024-03-31T15:15:18.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.205+0000 7f0c97400640 1 -- 172.21.15.16:0/2666509418 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c90014830 con 0x7f0ca006fbc0 2024-03-31T15:15:18.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.205+0000 7f0ca4a00640 1 -- 172.21.15.16:0/2666509418 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0ca00c9620 con 0x7f0ca006fbc0 2024-03-31T15:15:18.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.205+0000 7f0ca4a00640 1 -- 172.21.15.16:0/2666509418 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f0ca00c9a20 con 0x7f0ca006fbc0 2024-03-31T15:15:18.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.205+0000 7f0c97400640 1 -- 172.21.15.16:0/2666509418 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f0c9001b020 con 0x7f0ca006fbc0 2024-03-31T15:15:18.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.205+0000 7f0c97400640 1 --2- 172.21.15.16:0/2666509418 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f0c7c068380 0x7f0c7c06a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:18.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.205+0000 7f0c97400640 1 -- 172.21.15.16:0/2666509418 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(116..116 src has 1..116) ==== 5115+0+0 (secure 0 0 0) 0x7f0c90087190 con 0x7f0ca006fbc0 2024-03-31T15:15:18.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.205+0000 7f0c97400640 1 --2- 172.21.15.16:0/2666509418 >> [v2:172.21.15.16:6809/3360352974,v1:172.21.15.16:6812/3360352974] conn(0x7f0c7c06e380 0x7f0c7c0707a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:18.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.205+0000 7f0c97400640 1 -- 172.21.15.16:0/2666509418 --> [v2:172.21.15.16:6809/3360352974,v1:172.21.15.16:6812/3360352974] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f0c7c070e50 con 0x7f0c7c06e380 2024-03-31T15:15:18.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.205+0000 7f0c97400640 1 -- 172.21.15.16:0/2666509418 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_get_version_reply(handle=1 version=116) ==== 24+0+0 (secure 0 0 0) 0x7f0c9008e050 con 0x7f0ca006fbc0 2024-03-31T15:15:18.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.205+0000 7f0c95e00640 1 --2- 172.21.15.16:0/2666509418 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f0c7c068380 0x7f0c7c06a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:18.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.205+0000 7f0c9f200640 1 --2- 172.21.15.16:0/2666509418 >> [v2:172.21.15.16:6809/3360352974,v1:172.21.15.16:6812/3360352974] conn(0x7f0c7c06e380 0x7f0c7c0707a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:18.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.209+0000 7f0c95e00640 1 --2- 172.21.15.16:0/2666509418 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f0c7c068380 0x7f0c7c06a840 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7f0ca00c72c0 tx=0x7f0c88007b60 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:15:18.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.209+0000 7f0c9f200640 1 --2- 172.21.15.16:0/2666509418 >> [v2:172.21.15.16:6809/3360352974,v1:172.21.15.16:6812/3360352974] conn(0x7f0c7c06e380 0x7f0c7c0707a0 crc :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:15:18.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.209+0000 7f0c97400640 1 -- 172.21.15.16:0/2666509418 <== osd.3 v2:172.21.15.16:6809/3360352974 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f0c7c070e50 con 0x7f0c7c06e380 2024-03-31T15:15:18.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.221+0000 7f0ca4a00640 1 -- 172.21.15.16:0/2666509418 --> [v2:172.21.15.16:6809/3360352974,v1:172.21.15.16:6812/3360352974] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f0ca004f8c0 con 0x7f0c7c06e380 2024-03-31T15:15:18.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.221+0000 7f0c97400640 1 -- 172.21.15.16:0/2666509418 <== osd.3 v2:172.21.15.16:6809/3360352974 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f0ca004f8c0 con 0x7f0c7c06e380 2024-03-31T15:15:18.211 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-03-31T15:15:18.211 INFO:teuthology.orchestra.run.smithi016.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-31T15:15:18.211 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-03-31T15:15:18.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.221+0000 7f0ca4a00640 1 -- 172.21.15.16:0/2666509418 >> [v2:172.21.15.16:6809/3360352974,v1:172.21.15.16:6812/3360352974] conn(0x7f0c7c06e380 msgr2=0x7f0c7c0707a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:18.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.221+0000 7f0ca4a00640 1 --2- 172.21.15.16:0/2666509418 >> [v2:172.21.15.16:6809/3360352974,v1:172.21.15.16:6812/3360352974] conn(0x7f0c7c06e380 0x7f0c7c0707a0 crc :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:18.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.221+0000 7f0ca4a00640 1 -- 172.21.15.16:0/2666509418 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f0c7c068380 msgr2=0x7f0c7c06a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:18.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.221+0000 7f0ca4a00640 1 --2- 172.21.15.16:0/2666509418 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f0c7c068380 0x7f0c7c06a840 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7f0ca00c72c0 tx=0x7f0c88007b60 comp rx=0 tx=0).stop 2024-03-31T15:15:18.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.221+0000 7f0ca4a00640 1 -- 172.21.15.16:0/2666509418 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0ca006fbc0 msgr2=0x7f0ca014c880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:18.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.221+0000 7f0ca4a00640 1 --2- 172.21.15.16:0/2666509418 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0ca006fbc0 0x7f0ca014c880 secure :-1 s=READY pgs=1912 cs=0 l=1 rev1=1 crypto rx=0x7f0c90009d10 tx=0x7f0c9000c490 comp rx=0 tx=0).stop 2024-03-31T15:15:18.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.221+0000 7f0c9f200640 1 -- 172.21.15.16:0/2666509418 reap_dead start 2024-03-31T15:15:18.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.221+0000 7f0ca4a00640 1 -- 172.21.15.16:0/2666509418 shutdown_connections 2024-03-31T15:15:18.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.221+0000 7f0ca4a00640 1 -- 172.21.15.16:0/2666509418 >> 172.21.15.16:0/2666509418 conn(0x7f0ca00d1f90 msgr2=0x7f0ca00cfb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:15:18.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.221+0000 7f0ca4a00640 1 -- 172.21.15.16:0/2666509418 shutdown_connections 2024-03-31T15:15:18.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.221+0000 7f0ca4a00640 1 -- 172.21.15.16:0/2666509418 wait complete. 2024-03-31T15:15:18.229 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2024-03-31T15:15:18.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.381+0000 7f0775600640 1 Processor -- start 2024-03-31T15:15:18.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.381+0000 7f0775600640 1 -- start start 2024-03-31T15:15:18.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.381+0000 7f0775600640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f077015b4d0 0x7f07700b5b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:18.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.381+0000 7f0775600640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f07700b61e0 0x7f07700b42e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:18.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.381+0000 7f0775600640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f07700ce8a0 con 0x7f07700cef40 2024-03-31T15:15:18.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.381+0000 7f0775600640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f07700709a0 con 0x7f077015b4d0 2024-03-31T15:15:18.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.381+0000 7f0775600640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f077006fbe0 con 0x7f07700b61e0 2024-03-31T15:15:18.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.381+0000 7f0766000640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f077015b4d0 0x7f07700b5b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:18.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.381+0000 7f0766000640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f077015b4d0 0x7f07700b5b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37834/0 (socket says 172.21.15.16:37834) 2024-03-31T15:15:18.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.381+0000 7f0766000640 1 -- 172.21.15.16:0/3131772609 learned_addr learned my addr 172.21.15.16:0/3131772609 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:15:18.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.381+0000 7f076f400640 1 --2- 172.21.15.16:0/3131772609 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f07700b61e0 0x7f07700b42e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:18.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.381+0000 7f0766000640 1 -- 172.21.15.16:0/3131772609 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f07700b61e0 msgr2=0x7f07700b42e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:15:18.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.381+0000 7f0766000640 1 --2- 172.21.15.16:0/3131772609 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f07700b61e0 0x7f07700b42e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:18.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.381+0000 7f0766000640 1 -- 172.21.15.16:0/3131772609 >> v1:172.21.15.89:6789/0 conn(0x7f07700cef40 legacy=0x7f07700cf320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:18.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.381+0000 7f0766000640 1 -- 172.21.15.16:0/3131772609 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f077006fec0 con 0x7f077015b4d0 2024-03-31T15:15:18.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.381+0000 7f076e000640 1 -- 172.21.15.16:0/3131772609 <== mon.1 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3780213302 0 0) 0x7f07700ce8a0 con 0x7f07700cef40 2024-03-31T15:15:18.375 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.385+0000 7f0766000640 1 --2- 172.21.15.16:0/3131772609 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f077015b4d0 0x7f07700b5b50 secure :-1 s=READY pgs=1913 cs=0 l=1 rev1=1 crypto rx=0x7f0758009870 tx=0x7f075801c530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=90203d5a28ba71b server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:15:18.375 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.385+0000 7f076e000640 1 -- 172.21.15.16:0/3131772609 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07580a95f0 con 0x7f077015b4d0 2024-03-31T15:15:18.375 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.385+0000 7f076e000640 1 -- 172.21.15.16:0/3131772609 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f07580a9790 con 0x7f077015b4d0 2024-03-31T15:15:18.375 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.385+0000 7f076e000640 1 -- 172.21.15.16:0/3131772609 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07580a9b60 con 0x7f077015b4d0 2024-03-31T15:15:18.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.385+0000 7f0775600640 1 -- 172.21.15.16:0/3131772609 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f077015b4d0 msgr2=0x7f07700b5b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:18.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.385+0000 7f0775600640 1 --2- 172.21.15.16:0/3131772609 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f077015b4d0 0x7f07700b5b50 secure :-1 s=READY pgs=1913 cs=0 l=1 rev1=1 crypto rx=0x7f0758009870 tx=0x7f075801c530 comp rx=0 tx=0).stop 2024-03-31T15:15:18.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.385+0000 7f0775600640 1 -- 172.21.15.16:0/3131772609 shutdown_connections 2024-03-31T15:15:18.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.385+0000 7f0775600640 1 --2- 172.21.15.16:0/3131772609 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f07700b61e0 0x7f07700b42e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:18.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.385+0000 7f0775600640 1 --2- 172.21.15.16:0/3131772609 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f077015b4d0 0x7f07700b5b50 unknown :-1 s=CLOSED pgs=1913 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:18.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.385+0000 7f0775600640 1 -- 172.21.15.16:0/3131772609 >> 172.21.15.16:0/3131772609 conn(0x7f07700d1060 msgr2=0x7f07700d1470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:15:18.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.385+0000 7f0775600640 1 -- 172.21.15.16:0/3131772609 shutdown_connections 2024-03-31T15:15:18.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.385+0000 7f0775600640 1 -- 172.21.15.16:0/3131772609 wait complete. 2024-03-31T15:15:18.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.389+0000 7f0775600640 1 Processor -- start 2024-03-31T15:15:18.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.389+0000 7f0775600640 1 -- start start 2024-03-31T15:15:18.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.389+0000 7f0775600640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f07700b61e0 0x7f077015a720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:18.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.389+0000 7f0775600640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f07700cef40 0x7f0770160800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:18.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.389+0000 7f0775600640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f077015b4d0 0x7f0770156da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:18.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.389+0000 7f0775600640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f07700b51b0 con 0x7f07700cef40 2024-03-31T15:15:18.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.389+0000 7f0775600640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f07700b5030 con 0x7f077015b4d0 2024-03-31T15:15:18.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.389+0000 7f0775600640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f07700b5330 con 0x7f07700b61e0 2024-03-31T15:15:18.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.389+0000 7f0766000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f07700cef40 0x7f0770160800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:18.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.389+0000 7f0766000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f07700cef40 0x7f0770160800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37850/0 (socket says 172.21.15.16:37850) 2024-03-31T15:15:18.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.389+0000 7f0766000640 1 -- 172.21.15.16:0/1474355562 learned_addr learned my addr 172.21.15.16:0/1474355562 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:15:18.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.389+0000 7f076ea00640 1 --2- 172.21.15.16:0/1474355562 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f07700b61e0 0x7f077015a720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:18.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.389+0000 7f076f400640 1 --2- 172.21.15.16:0/1474355562 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f077015b4d0 0x7f0770156da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:18.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.389+0000 7f0766000640 1 -- 172.21.15.16:0/1474355562 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f07700b61e0 msgr2=0x7f077015a720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:18.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.389+0000 7f0766000640 1 --2- 172.21.15.16:0/1474355562 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f07700b61e0 0x7f077015a720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:18.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.389+0000 7f0766000640 1 -- 172.21.15.16:0/1474355562 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f077015b4d0 msgr2=0x7f0770156da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:18.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.389+0000 7f0766000640 1 --2- 172.21.15.16:0/1474355562 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f077015b4d0 0x7f0770156da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:18.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.389+0000 7f0766000640 1 -- 172.21.15.16:0/1474355562 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0770160f70 con 0x7f07700cef40 2024-03-31T15:15:18.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.389+0000 7f076f400640 1 --2- 172.21.15.16:0/1474355562 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f077015b4d0 0x7f0770156da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:15:18.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.389+0000 7f076ea00640 1 --2- 172.21.15.16:0/1474355562 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f07700b61e0 0x7f077015a720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:15:18.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.389+0000 7f0766000640 1 --2- 172.21.15.16:0/1474355562 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f07700cef40 0x7f0770160800 secure :-1 s=READY pgs=1914 cs=0 l=1 rev1=1 crypto rx=0x7f075801abe0 tx=0x7f07580192d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:15:18.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.389+0000 7f0767400640 1 -- 172.21.15.16:0/1474355562 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0758004220 con 0x7f07700cef40 2024-03-31T15:15:18.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.389+0000 7f0767400640 1 -- 172.21.15.16:0/1474355562 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f07580043c0 con 0x7f07700cef40 2024-03-31T15:15:18.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.389+0000 7f0775600640 1 -- 172.21.15.16:0/1474355562 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f07701f8120 con 0x7f07700cef40 2024-03-31T15:15:18.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.389+0000 7f0767400640 1 -- 172.21.15.16:0/1474355562 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0758003820 con 0x7f07700cef40 2024-03-31T15:15:18.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.389+0000 7f0775600640 1 -- 172.21.15.16:0/1474355562 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f07701f86e0 con 0x7f07700cef40 2024-03-31T15:15:18.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.389+0000 7f0775600640 1 -- 172.21.15.16:0/1474355562 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f07701f88e0 con 0x7f07700cef40 2024-03-31T15:15:18.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.393+0000 7f0767400640 1 -- 172.21.15.16:0/1474355562 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f07580a9600 con 0x7f07700cef40 2024-03-31T15:15:18.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.393+0000 7f0767400640 1 --2- 172.21.15.16:0/1474355562 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f074c068380 0x7f074c06a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:18.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.393+0000 7f0767400640 1 -- 172.21.15.16:0/1474355562 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(116..116 src has 1..116) ==== 5115+0+0 (secure 0 0 0) 0x7f0758120a80 con 0x7f07700cef40 2024-03-31T15:15:18.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.393+0000 7f076ea00640 1 --2- 172.21.15.16:0/1474355562 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f074c068380 0x7f074c06a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:18.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.393+0000 7f076ea00640 1 --2- 172.21.15.16:0/1474355562 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f074c068380 0x7f074c06a840 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7f0760000f70 tx=0x7f0760009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:15:18.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.397+0000 7f0767400640 1 -- 172.21.15.16:0/1474355562 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f07701f88e0 con 0x7f07700cef40 2024-03-31T15:15:18.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.565+0000 7f0775600640 1 -- 172.21.15.16:0/1474355562 --> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "3", "target": ["mon-mgr", ""]}) -- 0x7f0770069950 con 0x7f074c068380 2024-03-31T15:15:18.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.565+0000 7f0767400640 1 -- 172.21.15.16:0/1474355562 <== mgr.4100 v2:172.21.15.89:6832/1549407273 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 3 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f0770069950 con 0x7f074c068380 2024-03-31T15:15:18.556 INFO:teuthology.orchestra.run.smithi016.stderr:instructed osd(s) 3 to deep-scrub 2024-03-31T15:15:18.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.569+0000 7f0775600640 1 -- 172.21.15.16:0/1474355562 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f074c068380 msgr2=0x7f074c06a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:18.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.569+0000 7f0775600640 1 --2- 172.21.15.16:0/1474355562 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f074c068380 0x7f074c06a840 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7f0760000f70 tx=0x7f0760009040 comp rx=0 tx=0).stop 2024-03-31T15:15:18.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.569+0000 7f0775600640 1 -- 172.21.15.16:0/1474355562 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f07700cef40 msgr2=0x7f0770160800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:18.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.569+0000 7f0775600640 1 --2- 172.21.15.16:0/1474355562 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f07700cef40 0x7f0770160800 secure :-1 s=READY pgs=1914 cs=0 l=1 rev1=1 crypto rx=0x7f075801abe0 tx=0x7f07580192d0 comp rx=0 tx=0).stop 2024-03-31T15:15:18.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.569+0000 7f0775600640 1 -- 172.21.15.16:0/1474355562 shutdown_connections 2024-03-31T15:15:18.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.569+0000 7f0775600640 1 --2- 172.21.15.16:0/1474355562 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f074c068380 0x7f074c06a840 unknown :-1 s=CLOSED pgs=806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:18.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.569+0000 7f0775600640 1 --2- 172.21.15.16:0/1474355562 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f077015b4d0 0x7f0770156da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:18.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.569+0000 7f0775600640 1 --2- 172.21.15.16:0/1474355562 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f07700cef40 0x7f0770160800 unknown :-1 s=CLOSED pgs=1914 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:18.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.569+0000 7f0775600640 1 --2- 172.21.15.16:0/1474355562 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f07700b61e0 0x7f077015a720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:18.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.569+0000 7f0775600640 1 -- 172.21.15.16:0/1474355562 >> 172.21.15.16:0/1474355562 conn(0x7f07700d1060 msgr2=0x7f07700b56e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:15:18.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.569+0000 7f0775600640 1 -- 172.21.15.16:0/1474355562 shutdown_connections 2024-03-31T15:15:18.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.569+0000 7f0775600640 1 -- 172.21.15.16:0/1474355562 wait complete. 2024-03-31T15:15:18.577 INFO:tasks.ceph:Scrubbing osd.4 2024-03-31T15:15:18.578 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2024-03-31T15:15:18.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.733+0000 7f4c60a00640 1 Processor -- start 2024-03-31T15:15:18.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.733+0000 7f4c60a00640 1 -- start start 2024-03-31T15:15:18.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.733+0000 7f4c60a00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f4c5c147230 0x7f4c5c0b3dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:18.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.733+0000 7f4c60a00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f4c5c0b4300 0x7f4c5c0d0930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:18.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.733+0000 7f4c60a00640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4c5c0d2020 con 0x7f4c5c0d0e70 2024-03-31T15:15:18.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.733+0000 7f4c60a00640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f4c5c0b4760 con 0x7f4c5c0b4300 2024-03-31T15:15:18.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.733+0000 7f4c60a00640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f4c5c0d2c30 con 0x7f4c5c147230 2024-03-31T15:15:18.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.733+0000 7f4c53e00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f4c5c0b4300 0x7f4c5c0d0930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:18.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.733+0000 7f4c53e00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f4c5c0b4300 0x7f4c5c0d0930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37858/0 (socket says 172.21.15.16:37858) 2024-03-31T15:15:18.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.733+0000 7f4c53e00640 1 -- 172.21.15.16:0/2509379711 learned_addr learned my addr 172.21.15.16:0/2509379711 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:15:18.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.733+0000 7f4c59e00640 1 --2- 172.21.15.16:0/2509379711 >> v2:172.21.15.89:3300/0 conn(0x7f4c5c147230 0x7f4c5c0b3dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:18.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.737+0000 7f4c53e00640 1 -- 172.21.15.16:0/2509379711 >> v2:172.21.15.89:3300/0 conn(0x7f4c5c147230 msgr2=0x7f4c5c0b3dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:15:18.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.737+0000 7f4c53e00640 1 --2- 172.21.15.16:0/2509379711 >> v2:172.21.15.89:3300/0 conn(0x7f4c5c147230 0x7f4c5c0b3dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:18.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.737+0000 7f4c53e00640 1 -- 172.21.15.16:0/2509379711 >> v1:172.21.15.89:6789/0 conn(0x7f4c5c0d0e70 legacy=0x7f4c5c162110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:15:18.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.737+0000 7f4c53e00640 1 -- 172.21.15.16:0/2509379711 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c5c0d2300 con 0x7f4c5c0b4300 2024-03-31T15:15:18.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.737+0000 7f4c53e00640 1 --2- 172.21.15.16:0/2509379711 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4c5c0b4300 0x7f4c5c0d0930 secure :-1 s=READY pgs=1915 cs=0 l=1 rev1=1 crypto rx=0x7f4c5c067e00 tx=0x7f4c4401c4e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47df8c70ab7d7a57 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:15:18.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.737+0000 7f4c59400640 1 -- 172.21.15.16:0/2509379711 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c440b0070 con 0x7f4c5c0b4300 2024-03-31T15:15:18.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.737+0000 7f4c59400640 1 -- 172.21.15.16:0/2509379711 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4c440ab600 con 0x7f4c5c0b4300 2024-03-31T15:15:18.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.737+0000 7f4c59400640 1 -- 172.21.15.16:0/2509379711 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c440ab9d0 con 0x7f4c5c0b4300 2024-03-31T15:15:18.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.737+0000 7f4c60a00640 1 -- 172.21.15.16:0/2509379711 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4c5c0b4300 msgr2=0x7f4c5c0d0930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:18.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.737+0000 7f4c60a00640 1 --2- 172.21.15.16:0/2509379711 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4c5c0b4300 0x7f4c5c0d0930 secure :-1 s=READY pgs=1915 cs=0 l=1 rev1=1 crypto rx=0x7f4c5c067e00 tx=0x7f4c4401c4e0 comp rx=0 tx=0).stop 2024-03-31T15:15:18.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.737+0000 7f4c60a00640 1 -- 172.21.15.16:0/2509379711 shutdown_connections 2024-03-31T15:15:18.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.737+0000 7f4c60a00640 1 --2- 172.21.15.16:0/2509379711 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4c5c0b4300 0x7f4c5c0d0930 unknown :-1 s=CLOSED pgs=1915 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:18.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.737+0000 7f4c60a00640 1 --2- 172.21.15.16:0/2509379711 >> v2:172.21.15.89:3300/0 conn(0x7f4c5c147230 0x7f4c5c0b3dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:18.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.737+0000 7f4c60a00640 1 -- 172.21.15.16:0/2509379711 >> 172.21.15.16:0/2509379711 conn(0x7f4c5c06b7f0 msgr2=0x7f4c5c067330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:15:18.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.737+0000 7f4c60a00640 1 -- 172.21.15.16:0/2509379711 shutdown_connections 2024-03-31T15:15:18.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.741+0000 7f4c60a00640 1 -- 172.21.15.16:0/2509379711 wait complete. 2024-03-31T15:15:18.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.741+0000 7f4c60a00640 1 Processor -- start 2024-03-31T15:15:18.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.741+0000 7f4c60a00640 1 -- start start 2024-03-31T15:15:18.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.741+0000 7f4c60a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4c5c0b4300 0x7f4c5c13eda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:18.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.741+0000 7f4c60a00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f4c5c0d0e70 0x7f4c5c13b630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:18.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.741+0000 7f4c60a00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4c5c147230 0x7f4c5c13bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:18.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.741+0000 7f4c60a00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f4c5c0d12e0 con 0x7f4c5c0b4300 2024-03-31T15:15:18.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.741+0000 7f4c60a00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f4c5c0cc4c0 con 0x7f4c5c147230 2024-03-31T15:15:18.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.741+0000 7f4c60a00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f4c5c0d1460 con 0x7f4c5c0d0e70 2024-03-31T15:15:18.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.741+0000 7f4c59e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4c5c0b4300 0x7f4c5c13eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:18.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.741+0000 7f4c59e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4c5c0b4300 0x7f4c5c13eda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37872/0 (socket says 172.21.15.16:37872) 2024-03-31T15:15:18.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.741+0000 7f4c59e00640 1 -- 172.21.15.16:0/2184877954 learned_addr learned my addr 172.21.15.16:0/2184877954 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:15:18.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.741+0000 7f4c5a800640 1 --2- 172.21.15.16:0/2184877954 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4c5c147230 0x7f4c5c13bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:18.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.741+0000 7f4c53e00640 1 --2- 172.21.15.16:0/2184877954 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f4c5c0d0e70 0x7f4c5c13b630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:18.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.741+0000 7f4c59e00640 1 -- 172.21.15.16:0/2184877954 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f4c5c0d0e70 msgr2=0x7f4c5c13b630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:18.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.741+0000 7f4c59e00640 1 --2- 172.21.15.16:0/2184877954 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f4c5c0d0e70 0x7f4c5c13b630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:18.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.741+0000 7f4c59e00640 1 -- 172.21.15.16:0/2184877954 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4c5c147230 msgr2=0x7f4c5c13bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:18.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.741+0000 7f4c59e00640 1 --2- 172.21.15.16:0/2184877954 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4c5c147230 0x7f4c5c13bb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:18.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.741+0000 7f4c59e00640 1 -- 172.21.15.16:0/2184877954 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c5c13f430 con 0x7f4c5c0b4300 2024-03-31T15:15:18.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.741+0000 7f4c59e00640 1 --2- 172.21.15.16:0/2184877954 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4c5c0b4300 0x7f4c5c13eda0 secure :-1 s=READY pgs=1916 cs=0 l=1 rev1=1 crypto rx=0x7f4c4c00d800 tx=0x7f4c4c00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:15:18.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.741+0000 7f4c52000640 1 -- 172.21.15.16:0/2184877954 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c4c00b6c0 con 0x7f4c5c0b4300 2024-03-31T15:15:18.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.741+0000 7f4c52000640 1 -- 172.21.15.16:0/2184877954 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4c4c019070 con 0x7f4c5c0b4300 2024-03-31T15:15:18.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.741+0000 7f4c60a00640 1 -- 172.21.15.16:0/2184877954 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4c5c13f710 con 0x7f4c5c0b4300 2024-03-31T15:15:18.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.741+0000 7f4c52000640 1 -- 172.21.15.16:0/2184877954 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c4c00ba60 con 0x7f4c5c0b4300 2024-03-31T15:15:18.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.741+0000 7f4c60a00640 1 -- 172.21.15.16:0/2184877954 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4c5c13fcd0 con 0x7f4c5c0b4300 2024-03-31T15:15:18.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.745+0000 7f4c60a00640 1 -- 172.21.15.16:0/2184877954 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f4c5c1400b0 con 0x7f4c5c0b4300 2024-03-31T15:15:18.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.745+0000 7f4c52000640 1 -- 172.21.15.16:0/2184877954 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f4c4c00bc00 con 0x7f4c5c0b4300 2024-03-31T15:15:18.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.745+0000 7f4c52000640 1 --2- 172.21.15.16:0/2184877954 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f4c38068380 0x7f4c3806a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:18.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.745+0000 7f4c52000640 1 -- 172.21.15.16:0/2184877954 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(116..116 src has 1..116) ==== 5115+0+0 (secure 0 0 0) 0x7f4c4c086c90 con 0x7f4c5c0b4300 2024-03-31T15:15:18.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.745+0000 7f4c52000640 1 --2- 172.21.15.16:0/2184877954 >> [v2:172.21.15.89:6800/1856844856,v1:172.21.15.89:6801/1856844856] conn(0x7f4c3806e380 0x7f4c380707a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:18.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.745+0000 7f4c52000640 1 -- 172.21.15.16:0/2184877954 --> [v2:172.21.15.89:6800/1856844856,v1:172.21.15.89:6801/1856844856] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f4c38070e50 con 0x7f4c3806e380 2024-03-31T15:15:18.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.745+0000 7f4c52000640 1 -- 172.21.15.16:0/2184877954 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_get_version_reply(handle=1 version=116) ==== 24+0+0 (secure 0 0 0) 0x7f4c4c08c2a0 con 0x7f4c5c0b4300 2024-03-31T15:15:18.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.745+0000 7f4c53e00640 1 --2- 172.21.15.16:0/2184877954 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f4c38068380 0x7f4c3806a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:18.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.745+0000 7f4c5a800640 1 --2- 172.21.15.16:0/2184877954 >> [v2:172.21.15.89:6800/1856844856,v1:172.21.15.89:6801/1856844856] conn(0x7f4c3806e380 0x7f4c380707a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:18.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.745+0000 7f4c53e00640 1 --2- 172.21.15.16:0/2184877954 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f4c38068380 0x7f4c3806a840 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7f4c440ad000 tx=0x7f4c44008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:15:18.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.749+0000 7f4c5a800640 1 --2- 172.21.15.16:0/2184877954 >> [v2:172.21.15.89:6800/1856844856,v1:172.21.15.89:6801/1856844856] conn(0x7f4c3806e380 0x7f4c380707a0 crc :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:15:18.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.749+0000 7f4c52000640 1 -- 172.21.15.16:0/2184877954 <== osd.4 v2:172.21.15.89:6800/1856844856 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f4c38070e50 con 0x7f4c3806e380 2024-03-31T15:15:18.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.761+0000 7f4c60a00640 1 -- 172.21.15.16:0/2184877954 --> [v2:172.21.15.89:6800/1856844856,v1:172.21.15.89:6801/1856844856] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f4c5c1464f0 con 0x7f4c3806e380 2024-03-31T15:15:18.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.761+0000 7f4c52000640 1 -- 172.21.15.16:0/2184877954 <== osd.4 v2:172.21.15.89:6800/1856844856 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f4c5c1464f0 con 0x7f4c3806e380 2024-03-31T15:15:18.751 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-03-31T15:15:18.752 INFO:teuthology.orchestra.run.smithi016.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-31T15:15:18.752 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-03-31T15:15:18.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.761+0000 7f4c60a00640 1 -- 172.21.15.16:0/2184877954 >> [v2:172.21.15.89:6800/1856844856,v1:172.21.15.89:6801/1856844856] conn(0x7f4c3806e380 msgr2=0x7f4c380707a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:18.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.761+0000 7f4c60a00640 1 --2- 172.21.15.16:0/2184877954 >> [v2:172.21.15.89:6800/1856844856,v1:172.21.15.89:6801/1856844856] conn(0x7f4c3806e380 0x7f4c380707a0 crc :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:18.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.761+0000 7f4c60a00640 1 -- 172.21.15.16:0/2184877954 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f4c38068380 msgr2=0x7f4c3806a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:18.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.761+0000 7f4c60a00640 1 --2- 172.21.15.16:0/2184877954 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f4c38068380 0x7f4c3806a840 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7f4c440ad000 tx=0x7f4c44008040 comp rx=0 tx=0).stop 2024-03-31T15:15:18.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.761+0000 7f4c60a00640 1 -- 172.21.15.16:0/2184877954 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4c5c0b4300 msgr2=0x7f4c5c13eda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:18.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.761+0000 7f4c60a00640 1 --2- 172.21.15.16:0/2184877954 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4c5c0b4300 0x7f4c5c13eda0 secure :-1 s=READY pgs=1916 cs=0 l=1 rev1=1 crypto rx=0x7f4c4c00d800 tx=0x7f4c4c00dcc0 comp rx=0 tx=0).stop 2024-03-31T15:15:18.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.761+0000 7f4c5a800640 1 -- 172.21.15.16:0/2184877954 reap_dead start 2024-03-31T15:15:18.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.761+0000 7f4c60a00640 1 -- 172.21.15.16:0/2184877954 shutdown_connections 2024-03-31T15:15:18.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.761+0000 7f4c60a00640 1 -- 172.21.15.16:0/2184877954 >> 172.21.15.16:0/2184877954 conn(0x7f4c5c06b7f0 msgr2=0x7f4c5c0cf500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:15:18.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.761+0000 7f4c60a00640 1 -- 172.21.15.16:0/2184877954 shutdown_connections 2024-03-31T15:15:18.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.761+0000 7f4c60a00640 1 -- 172.21.15.16:0/2184877954 wait complete. 2024-03-31T15:15:18.770 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2024-03-31T15:15:18.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.921+0000 7f147ba00640 1 Processor -- start 2024-03-31T15:15:18.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.921+0000 7f147ba00640 1 -- start start 2024-03-31T15:15:18.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.925+0000 7f147ba00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f147415b3d0 0x7f14740b1cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:18.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.925+0000 7f147ba00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f14740ce2f0 0x7f14740b2200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:18.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.925+0000 7f147ba00640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f147415f3c0 con 0x7f147415eee0 2024-03-31T15:15:18.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.925+0000 7f147ba00640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f147415b7b0 con 0x7f147415b3d0 2024-03-31T15:15:18.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.925+0000 7f147ba00640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f147415f6a0 con 0x7f14740ce2f0 2024-03-31T15:15:18.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.925+0000 7f1479000640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f147415b3d0 0x7f14740b1cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:18.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.925+0000 7f1473e00640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f147415eee0 0x7f14740cc9c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:42886/0 (socket says 172.21.15.16:42886) 2024-03-31T15:15:18.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.925+0000 7f1473e00640 1 -- 172.21.15.16:0/3806852721 learned_addr learned my addr 172.21.15.16:0/3806852721 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:15:18.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.925+0000 7f1479a00640 1 --2- 172.21.15.16:0/3806852721 >> v2:172.21.15.89:3300/0 conn(0x7f14740ce2f0 0x7f14740b2200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:18.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.925+0000 7f1479000640 1 -- 172.21.15.16:0/3806852721 >> v2:172.21.15.89:3300/0 conn(0x7f14740ce2f0 msgr2=0x7f14740b2200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:15:18.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.925+0000 7f1479000640 1 --2- 172.21.15.16:0/3806852721 >> v2:172.21.15.89:3300/0 conn(0x7f14740ce2f0 0x7f14740b2200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:18.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.925+0000 7f1479000640 1 -- 172.21.15.16:0/3806852721 >> v1:172.21.15.16:6789/0 conn(0x7f147415eee0 legacy=0x7f14740cc9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:18.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.925+0000 7f1479000640 1 -- 172.21.15.16:0/3806852721 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f147415f980 con 0x7f147415b3d0 2024-03-31T15:15:18.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.925+0000 7f1473400640 1 -- 172.21.15.16:0/3806852721 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3034980012 0 0) 0x7f147415f3c0 con 0x7f147415eee0 2024-03-31T15:15:18.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.925+0000 7f1479000640 1 --2- 172.21.15.16:0/3806852721 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f147415b3d0 0x7f14740b1cc0 secure :-1 s=READY pgs=1918 cs=0 l=1 rev1=1 crypto rx=0x7f147406f8a0 tx=0x7f146801c450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a52cf50b40662ccc server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:15:18.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.929+0000 7f1473400640 1 -- 172.21.15.16:0/3806852721 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14680ab610 con 0x7f147415b3d0 2024-03-31T15:15:18.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.929+0000 7f1473400640 1 -- 172.21.15.16:0/3806852721 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f14680b0070 con 0x7f147415b3d0 2024-03-31T15:15:18.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.929+0000 7f1473400640 1 -- 172.21.15.16:0/3806852721 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14680ab9d0 con 0x7f147415b3d0 2024-03-31T15:15:18.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.929+0000 7f147ba00640 1 -- 172.21.15.16:0/3806852721 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f147415b3d0 msgr2=0x7f14740b1cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:18.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.929+0000 7f147ba00640 1 --2- 172.21.15.16:0/3806852721 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f147415b3d0 0x7f14740b1cc0 secure :-1 s=READY pgs=1918 cs=0 l=1 rev1=1 crypto rx=0x7f147406f8a0 tx=0x7f146801c450 comp rx=0 tx=0).stop 2024-03-31T15:15:18.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.929+0000 7f147ba00640 1 -- 172.21.15.16:0/3806852721 shutdown_connections 2024-03-31T15:15:18.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.929+0000 7f147ba00640 1 --2- 172.21.15.16:0/3806852721 >> v2:172.21.15.89:3300/0 conn(0x7f14740ce2f0 0x7f14740b2200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:18.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.929+0000 7f147ba00640 1 --2- 172.21.15.16:0/3806852721 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f147415b3d0 0x7f14740b1cc0 unknown :-1 s=CLOSED pgs=1918 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:18.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.929+0000 7f147ba00640 1 -- 172.21.15.16:0/3806852721 >> 172.21.15.16:0/3806852721 conn(0x7f147406b620 msgr2=0x7f147406ba30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:15:18.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.929+0000 7f147ba00640 1 -- 172.21.15.16:0/3806852721 shutdown_connections 2024-03-31T15:15:18.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.929+0000 7f147ba00640 1 -- 172.21.15.16:0/3806852721 wait complete. 2024-03-31T15:15:18.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.929+0000 7f147ba00640 1 Processor -- start 2024-03-31T15:15:18.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.929+0000 7f147ba00640 1 -- start start 2024-03-31T15:15:18.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.929+0000 7f147ba00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f14740ce2f0 0x7f14740af970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:18.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.929+0000 7f147ba00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f147415b3d0 0x7f147413a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:18.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.929+0000 7f147ba00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f147415eee0 0x7f147415b930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:18.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.929+0000 7f147ba00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f14740d0140 con 0x7f14740ce2f0 2024-03-31T15:15:18.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.929+0000 7f147ba00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f14740cffc0 con 0x7f147415b3d0 2024-03-31T15:15:18.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.929+0000 7f147ba00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f14740d02c0 con 0x7f147415eee0 2024-03-31T15:15:18.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.929+0000 7f1479000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f14740ce2f0 0x7f14740af970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:18.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.929+0000 7f1479000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f14740ce2f0 0x7f14740af970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37888/0 (socket says 172.21.15.16:37888) 2024-03-31T15:15:18.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.929+0000 7f1479000640 1 -- 172.21.15.16:0/1097378498 learned_addr learned my addr 172.21.15.16:0/1097378498 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:15:18.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.929+0000 7f1479a00640 1 --2- 172.21.15.16:0/1097378498 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f147415eee0 0x7f147415b930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:18.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.929+0000 7f1473e00640 1 --2- 172.21.15.16:0/1097378498 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f147415b3d0 0x7f147413a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:18.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.933+0000 7f1479000640 1 -- 172.21.15.16:0/1097378498 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f147415eee0 msgr2=0x7f147415b930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:18.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.933+0000 7f1479000640 1 --2- 172.21.15.16:0/1097378498 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f147415eee0 0x7f147415b930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:18.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.933+0000 7f1479000640 1 -- 172.21.15.16:0/1097378498 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f147415b3d0 msgr2=0x7f147413a8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:18.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.933+0000 7f1479000640 1 --2- 172.21.15.16:0/1097378498 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f147415b3d0 0x7f147413a8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:18.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.933+0000 7f1479000640 1 -- 172.21.15.16:0/1097378498 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14740b1840 con 0x7f14740ce2f0 2024-03-31T15:15:18.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.933+0000 7f1479000640 1 --2- 172.21.15.16:0/1097378498 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f14740ce2f0 0x7f14740af970 secure :-1 s=READY pgs=1919 cs=0 l=1 rev1=1 crypto rx=0x7f14680ad040 tx=0x7f146801ac90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:15:18.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.933+0000 7f1470c00640 1 -- 172.21.15.16:0/1097378498 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14680bf610 con 0x7f14740ce2f0 2024-03-31T15:15:18.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.933+0000 7f1470c00640 1 -- 172.21.15.16:0/1097378498 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f14680bf7d0 con 0x7f14740ce2f0 2024-03-31T15:15:18.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.933+0000 7f147ba00640 1 -- 172.21.15.16:0/1097378498 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f147414d780 con 0x7f14740ce2f0 2024-03-31T15:15:18.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.933+0000 7f1470c00640 1 -- 172.21.15.16:0/1097378498 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14680bfbc0 con 0x7f14740ce2f0 2024-03-31T15:15:18.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.933+0000 7f147ba00640 1 -- 172.21.15.16:0/1097378498 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f14740afeb0 con 0x7f14740ce2f0 2024-03-31T15:15:18.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.933+0000 7f147ba00640 1 -- 172.21.15.16:0/1097378498 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1474069950 con 0x7f14740ce2f0 2024-03-31T15:15:18.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.933+0000 7f1470c00640 1 -- 172.21.15.16:0/1097378498 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f14680bd020 con 0x7f14740ce2f0 2024-03-31T15:15:18.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.933+0000 7f1470c00640 1 --2- 172.21.15.16:0/1097378498 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f1454068380 0x7f145406a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:18.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.933+0000 7f1470c00640 1 -- 172.21.15.16:0/1097378498 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(116..116 src has 1..116) ==== 5115+0+0 (secure 0 0 0) 0x7f1468120fe0 con 0x7f14740ce2f0 2024-03-31T15:15:18.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.933+0000 7f1473e00640 1 --2- 172.21.15.16:0/1097378498 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f1454068380 0x7f145406a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:18.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.937+0000 7f1473e00640 1 --2- 172.21.15.16:0/1097378498 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f1454068380 0x7f145406a840 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7f147414f660 tx=0x7f1460007c10 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:15:18.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:18.937+0000 7f1470c00640 1 -- 172.21.15.16:0/1097378498 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f14680a8050 con 0x7f14740ce2f0 2024-03-31T15:15:19.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.101+0000 7f147ba00640 1 -- 172.21.15.16:0/1097378498 --> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "4", "target": ["mon-mgr", ""]}) -- 0x7f14740cdea0 con 0x7f1454068380 2024-03-31T15:15:19.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.101+0000 7f1470c00640 1 -- 172.21.15.16:0/1097378498 <== mgr.4100 v2:172.21.15.89:6832/1549407273 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 4 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f14740cdea0 con 0x7f1454068380 2024-03-31T15:15:19.092 INFO:teuthology.orchestra.run.smithi016.stderr:instructed osd(s) 4 to deep-scrub 2024-03-31T15:15:19.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.105+0000 7f147ba00640 1 -- 172.21.15.16:0/1097378498 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f1454068380 msgr2=0x7f145406a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:19.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.105+0000 7f147ba00640 1 --2- 172.21.15.16:0/1097378498 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f1454068380 0x7f145406a840 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7f147414f660 tx=0x7f1460007c10 comp rx=0 tx=0).stop 2024-03-31T15:15:19.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.105+0000 7f147ba00640 1 -- 172.21.15.16:0/1097378498 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f14740ce2f0 msgr2=0x7f14740af970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:19.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.105+0000 7f147ba00640 1 --2- 172.21.15.16:0/1097378498 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f14740ce2f0 0x7f14740af970 secure :-1 s=READY pgs=1919 cs=0 l=1 rev1=1 crypto rx=0x7f14680ad040 tx=0x7f146801ac90 comp rx=0 tx=0).stop 2024-03-31T15:15:19.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.105+0000 7f147ba00640 1 -- 172.21.15.16:0/1097378498 shutdown_connections 2024-03-31T15:15:19.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.105+0000 7f147ba00640 1 --2- 172.21.15.16:0/1097378498 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f1454068380 0x7f145406a840 unknown :-1 s=CLOSED pgs=808 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:19.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.105+0000 7f147ba00640 1 --2- 172.21.15.16:0/1097378498 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f147415eee0 0x7f147415b930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:19.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.105+0000 7f147ba00640 1 --2- 172.21.15.16:0/1097378498 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f147415b3d0 0x7f147413a8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:19.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.105+0000 7f147ba00640 1 --2- 172.21.15.16:0/1097378498 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f14740ce2f0 0x7f14740af970 unknown :-1 s=CLOSED pgs=1919 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:19.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.105+0000 7f147ba00640 1 -- 172.21.15.16:0/1097378498 >> 172.21.15.16:0/1097378498 conn(0x7f147406b620 msgr2=0x7f14740d0710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:15:19.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.105+0000 7f147ba00640 1 -- 172.21.15.16:0/1097378498 shutdown_connections 2024-03-31T15:15:19.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.105+0000 7f147ba00640 1 -- 172.21.15.16:0/1097378498 wait complete. 2024-03-31T15:15:19.114 INFO:tasks.ceph:Scrubbing osd.5 2024-03-31T15:15:19.114 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2024-03-31T15:15:19.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.269+0000 7f91f8e00640 1 Processor -- start 2024-03-31T15:15:19.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.269+0000 7f91f8e00640 1 -- start start 2024-03-31T15:15:19.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.269+0000 7f91f8e00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f91f40730c0 0x7f91f40b45d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:19.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.269+0000 7f91f8e00640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f91f40734f0 con 0x7f91f40b22f0 2024-03-31T15:15:19.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.269+0000 7f91f8e00640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f91f4144320 con 0x7f91f4143f40 2024-03-31T15:15:19.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.269+0000 7f91f8e00640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f91f415ef80 con 0x7f91f40730c0 2024-03-31T15:15:19.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.269+0000 7f91f2800640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f91f40730c0 0x7f91f40b45d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:19.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.269+0000 7f91f2800640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f91f40730c0 0x7f91f40b45d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37896/0 (socket says 172.21.15.16:37896) 2024-03-31T15:15:19.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.269+0000 7f91f2800640 1 -- 172.21.15.16:0/3386432519 learned_addr learned my addr 172.21.15.16:0/3386432519 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:15:19.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.269+0000 7f91e9400640 1 --1- 172.21.15.16:0/3386432519 >> v1:172.21.15.16:6789/0 conn(0x7f91f40b22f0 0x7f91f40b2f50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:42896/0 (socket says 172.21.15.16:42896) 2024-03-31T15:15:19.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.269+0000 7f91f2800640 1 -- 172.21.15.16:0/3386432519 >> v1:172.21.15.89:6789/0 conn(0x7f91f4143f40 legacy=0x7f91f40b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:19.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.269+0000 7f91f2800640 1 -- 172.21.15.16:0/3386432519 >> v1:172.21.15.16:6789/0 conn(0x7f91f40b22f0 legacy=0x7f91f40b2f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:19.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.269+0000 7f91f2800640 1 -- 172.21.15.16:0/3386432519 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f91f415f260 con 0x7f91f40730c0 2024-03-31T15:15:19.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.273+0000 7f91f2800640 1 --2- 172.21.15.16:0/3386432519 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f91f40730c0 0x7f91f40b45d0 secure :-1 s=READY pgs=1921 cs=0 l=1 rev1=1 crypto rx=0x7f91ec009e50 tx=0x7f91ec01ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=442a76e8ef20a539 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:15:19.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.273+0000 7f91f1400640 1 -- 172.21.15.16:0/3386432519 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91ec0a5700 con 0x7f91f40730c0 2024-03-31T15:15:19.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.273+0000 7f91f1400640 1 -- 172.21.15.16:0/3386432519 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f91ec0a58a0 con 0x7f91f40730c0 2024-03-31T15:15:19.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.273+0000 7f91f1400640 1 -- 172.21.15.16:0/3386432519 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91ec0a5c70 con 0x7f91f40730c0 2024-03-31T15:15:19.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.273+0000 7f91f8e00640 1 -- 172.21.15.16:0/3386432519 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f91f40730c0 msgr2=0x7f91f40b45d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:19.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.273+0000 7f91f8e00640 1 --2- 172.21.15.16:0/3386432519 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f91f40730c0 0x7f91f40b45d0 secure :-1 s=READY pgs=1921 cs=0 l=1 rev1=1 crypto rx=0x7f91ec009e50 tx=0x7f91ec01ca30 comp rx=0 tx=0).stop 2024-03-31T15:15:19.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.273+0000 7f91f8e00640 1 -- 172.21.15.16:0/3386432519 shutdown_connections 2024-03-31T15:15:19.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.273+0000 7f91f8e00640 1 --2- 172.21.15.16:0/3386432519 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f91f40730c0 0x7f91f40b45d0 unknown :-1 s=CLOSED pgs=1921 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:19.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.273+0000 7f91f8e00640 1 -- 172.21.15.16:0/3386432519 >> 172.21.15.16:0/3386432519 conn(0x7f91f406b620 msgr2=0x7f91f406ba30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:15:19.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.273+0000 7f91f8e00640 1 -- 172.21.15.16:0/3386432519 shutdown_connections 2024-03-31T15:15:19.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.273+0000 7f91f8e00640 1 -- 172.21.15.16:0/3386432519 wait complete. 2024-03-31T15:15:19.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.277+0000 7f91f8e00640 1 Processor -- start 2024-03-31T15:15:19.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.277+0000 7f91f8e00640 1 -- start start 2024-03-31T15:15:19.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.277+0000 7f91f8e00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f91f40730c0 0x7f91f4160220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:19.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.277+0000 7f91f8e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f91f40b22f0 0x7f91f41fb590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:19.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.277+0000 7f91f8e00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f91f4143f40 0x7f91f413efd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:19.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.277+0000 7f91f8e00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f91f40d0020 con 0x7f91f40b22f0 2024-03-31T15:15:19.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.277+0000 7f91f8e00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f91f40cfea0 con 0x7f91f4143f40 2024-03-31T15:15:19.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.277+0000 7f91f8e00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f91f40d01a0 con 0x7f91f40730c0 2024-03-31T15:15:19.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.277+0000 7f91e9400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f91f40b22f0 0x7f91f41fb590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:19.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.277+0000 7f91e9400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f91f40b22f0 0x7f91f41fb590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37906/0 (socket says 172.21.15.16:37906) 2024-03-31T15:15:19.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.277+0000 7f91e9400640 1 -- 172.21.15.16:0/2313056892 learned_addr learned my addr 172.21.15.16:0/2313056892 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:15:19.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.277+0000 7f91f1e00640 1 --2- 172.21.15.16:0/2313056892 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f91f40730c0 0x7f91f4160220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:19.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.277+0000 7f91f2800640 1 --2- 172.21.15.16:0/2313056892 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f91f4143f40 0x7f91f413efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:19.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.277+0000 7f91e9400640 1 -- 172.21.15.16:0/2313056892 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f91f40730c0 msgr2=0x7f91f4160220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:19.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.277+0000 7f91e9400640 1 --2- 172.21.15.16:0/2313056892 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f91f40730c0 0x7f91f4160220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:19.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.277+0000 7f91e9400640 1 -- 172.21.15.16:0/2313056892 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f91f4143f40 msgr2=0x7f91f413efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:19.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.277+0000 7f91e9400640 1 --2- 172.21.15.16:0/2313056892 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f91f4143f40 0x7f91f413efd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:19.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.277+0000 7f91e9400640 1 -- 172.21.15.16:0/2313056892 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f91f41608b0 con 0x7f91f40b22f0 2024-03-31T15:15:19.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.277+0000 7f91e9400640 1 --2- 172.21.15.16:0/2313056892 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f91f40b22f0 0x7f91f41fb590 secure :-1 s=READY pgs=1922 cs=0 l=1 rev1=1 crypto rx=0x7f91dc00c420 tx=0x7f91dc00c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:15:19.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.277+0000 7f91eaa00640 1 -- 172.21.15.16:0/2313056892 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91dc017070 con 0x7f91f40b22f0 2024-03-31T15:15:19.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.277+0000 7f91eaa00640 1 -- 172.21.15.16:0/2313056892 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f91dc0064e0 con 0x7f91f40b22f0 2024-03-31T15:15:19.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.277+0000 7f91eaa00640 1 -- 172.21.15.16:0/2313056892 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91dc0049f0 con 0x7f91f40b22f0 2024-03-31T15:15:19.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.277+0000 7f91f8e00640 1 -- 172.21.15.16:0/2313056892 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f91f4160b90 con 0x7f91f40b22f0 2024-03-31T15:15:19.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.277+0000 7f91f8e00640 1 -- 172.21.15.16:0/2313056892 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f91f4161150 con 0x7f91f40b22f0 2024-03-31T15:15:19.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.277+0000 7f91f8e00640 1 -- 172.21.15.16:0/2313056892 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f91f4069950 con 0x7f91f40b22f0 2024-03-31T15:15:19.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.281+0000 7f91eaa00640 1 -- 172.21.15.16:0/2313056892 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f91dc01a070 con 0x7f91f40b22f0 2024-03-31T15:15:19.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.281+0000 7f91eaa00640 1 --2- 172.21.15.16:0/2313056892 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f91d0068380 0x7f91d006a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:19.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.281+0000 7f91eaa00640 1 -- 172.21.15.16:0/2313056892 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(116..116 src has 1..116) ==== 5115+0+0 (secure 0 0 0) 0x7f91dc086120 con 0x7f91f40b22f0 2024-03-31T15:15:19.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.281+0000 7f91eaa00640 1 --2- 172.21.15.16:0/2313056892 >> [v2:172.21.15.89:6810/50395527,v1:172.21.15.89:6814/50395527] conn(0x7f91d006e380 0x7f91d00707a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:19.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.281+0000 7f91eaa00640 1 -- 172.21.15.16:0/2313056892 --> [v2:172.21.15.89:6810/50395527,v1:172.21.15.89:6814/50395527] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f91d0070e50 con 0x7f91d006e380 2024-03-31T15:15:19.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.281+0000 7f91eaa00640 1 -- 172.21.15.16:0/2313056892 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_get_version_reply(handle=1 version=116) ==== 24+0+0 (secure 0 0 0) 0x7f91dc0865b0 con 0x7f91f40b22f0 2024-03-31T15:15:19.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.281+0000 7f91f1e00640 1 --2- 172.21.15.16:0/2313056892 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f91d0068380 0x7f91d006a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:19.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.281+0000 7f91f2800640 1 --2- 172.21.15.16:0/2313056892 >> [v2:172.21.15.89:6810/50395527,v1:172.21.15.89:6814/50395527] conn(0x7f91d006e380 0x7f91d00707a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:19.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.281+0000 7f91f1e00640 1 --2- 172.21.15.16:0/2313056892 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f91d0068380 0x7f91d006a840 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7f91e4007fd0 tx=0x7f91e4009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:15:19.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.281+0000 7f91f2800640 1 --2- 172.21.15.16:0/2313056892 >> [v2:172.21.15.89:6810/50395527,v1:172.21.15.89:6814/50395527] conn(0x7f91d006e380 0x7f91d00707a0 crc :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:15:19.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.281+0000 7f91eaa00640 1 -- 172.21.15.16:0/2313056892 <== osd.5 v2:172.21.15.89:6810/50395527 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f91d0070e50 con 0x7f91d006e380 2024-03-31T15:15:19.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.297+0000 7f91f8e00640 1 -- 172.21.15.16:0/2313056892 --> [v2:172.21.15.89:6810/50395527,v1:172.21.15.89:6814/50395527] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f91f40b4060 con 0x7f91d006e380 2024-03-31T15:15:19.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.297+0000 7f91eaa00640 1 -- 172.21.15.16:0/2313056892 <== osd.5 v2:172.21.15.89:6810/50395527 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f91f40b4060 con 0x7f91d006e380 2024-03-31T15:15:19.287 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-03-31T15:15:19.287 INFO:teuthology.orchestra.run.smithi016.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-31T15:15:19.287 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-03-31T15:15:19.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.297+0000 7f91f8e00640 1 -- 172.21.15.16:0/2313056892 >> [v2:172.21.15.89:6810/50395527,v1:172.21.15.89:6814/50395527] conn(0x7f91d006e380 msgr2=0x7f91d00707a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:19.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.297+0000 7f91f8e00640 1 --2- 172.21.15.16:0/2313056892 >> [v2:172.21.15.89:6810/50395527,v1:172.21.15.89:6814/50395527] conn(0x7f91d006e380 0x7f91d00707a0 crc :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:19.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.297+0000 7f91f8e00640 1 -- 172.21.15.16:0/2313056892 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f91d0068380 msgr2=0x7f91d006a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:19.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.297+0000 7f91f8e00640 1 --2- 172.21.15.16:0/2313056892 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f91d0068380 0x7f91d006a840 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7f91e4007fd0 tx=0x7f91e4009040 comp rx=0 tx=0).stop 2024-03-31T15:15:19.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.297+0000 7f91f8e00640 1 -- 172.21.15.16:0/2313056892 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f91f40b22f0 msgr2=0x7f91f41fb590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:19.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.297+0000 7f91f8e00640 1 --2- 172.21.15.16:0/2313056892 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f91f40b22f0 0x7f91f41fb590 secure :-1 s=READY pgs=1922 cs=0 l=1 rev1=1 crypto rx=0x7f91dc00c420 tx=0x7f91dc00c8e0 comp rx=0 tx=0).stop 2024-03-31T15:15:19.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.297+0000 7f91f2800640 1 -- 172.21.15.16:0/2313056892 reap_dead start 2024-03-31T15:15:19.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.297+0000 7f91f8e00640 1 -- 172.21.15.16:0/2313056892 shutdown_connections 2024-03-31T15:15:19.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.297+0000 7f91f8e00640 1 -- 172.21.15.16:0/2313056892 >> 172.21.15.16:0/2313056892 conn(0x7f91f406b620 msgr2=0x7f91f415e5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:15:19.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.297+0000 7f91f8e00640 1 -- 172.21.15.16:0/2313056892 shutdown_connections 2024-03-31T15:15:19.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.297+0000 7f91f8e00640 1 -- 172.21.15.16:0/2313056892 wait complete. 2024-03-31T15:15:19.305 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2024-03-31T15:15:19.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.457+0000 7f24c0400640 1 Processor -- start 2024-03-31T15:15:19.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.457+0000 7f24c0400640 1 -- start start 2024-03-31T15:15:19.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.457+0000 7f24c0400640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f24b806dbc0 0x7f24b815f650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:19.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.457+0000 7f24c0400640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f24b80cb440 0x7f24b806f920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:19.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.457+0000 7f24c0400640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f24b80cbe90 con 0x7f24b80cb8c0 2024-03-31T15:15:19.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.457+0000 7f24c0400640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f24b815fb90 con 0x7f24b80cb440 2024-03-31T15:15:19.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.457+0000 7f24c0400640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f24b815f1c0 con 0x7f24b806dbc0 2024-03-31T15:15:19.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.457+0000 7f24be400640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f24b80cb8c0 0x7f24b8162180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:42904/0 (socket says 172.21.15.16:42904) 2024-03-31T15:15:19.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.457+0000 7f24be400640 1 -- 172.21.15.16:0/3282449693 learned_addr learned my addr 172.21.15.16:0/3282449693 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:15:19.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.457+0000 7f24bda00640 1 --2- 172.21.15.16:0/3282449693 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f24b806dbc0 0x7f24b815f650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:19.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.457+0000 7f24bd000640 1 --2- 172.21.15.16:0/3282449693 >> v2:172.21.15.89:3300/0 conn(0x7f24b80cb440 0x7f24b806f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:19.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.457+0000 7f24b7e00640 1 -- 172.21.15.16:0/3282449693 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3293729652 0 0) 0x7f24b80cbe90 con 0x7f24b80cb8c0 2024-03-31T15:15:19.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.457+0000 7f24b7e00640 1 -- 172.21.15.16:0/3282449693 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f24a0003530 con 0x7f24b80cb8c0 2024-03-31T15:15:19.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.457+0000 7f24bda00640 1 -- 172.21.15.16:0/3282449693 >> v2:172.21.15.89:3300/0 conn(0x7f24b80cb440 msgr2=0x7f24b806f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:15:19.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.457+0000 7f24bda00640 1 --2- 172.21.15.16:0/3282449693 >> v2:172.21.15.89:3300/0 conn(0x7f24b80cb440 0x7f24b806f920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:19.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.457+0000 7f24bda00640 1 -- 172.21.15.16:0/3282449693 >> v1:172.21.15.16:6789/0 conn(0x7f24b80cb8c0 legacy=0x7f24b8162180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:19.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.457+0000 7f24bda00640 1 -- 172.21.15.16:0/3282449693 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24b80cc170 con 0x7f24b806dbc0 2024-03-31T15:15:19.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.457+0000 7f24bd000640 1 --2- 172.21.15.16:0/3282449693 >> v2:172.21.15.89:3300/0 conn(0x7f24b80cb440 0x7f24b806f920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:15:19.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.461+0000 7f24bda00640 1 --2- 172.21.15.16:0/3282449693 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f24b806dbc0 0x7f24b815f650 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f24ac0097c0 tx=0x7f24ac01c3e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3c37b20be8365c89 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:15:19.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.461+0000 7f24b7e00640 1 -- 172.21.15.16:0/3282449693 <== mon.2 v2:172.21.15.89:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24ac0ab600 con 0x7f24b806dbc0 2024-03-31T15:15:19.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.461+0000 7f24b7e00640 1 -- 172.21.15.16:0/3282449693 <== mon.2 v2:172.21.15.89:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f24ac0b0070 con 0x7f24b806dbc0 2024-03-31T15:15:19.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.461+0000 7f24b7e00640 1 -- 172.21.15.16:0/3282449693 <== mon.2 v2:172.21.15.89:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24ac0ab9c0 con 0x7f24b806dbc0 2024-03-31T15:15:19.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.461+0000 7f24c0400640 1 -- 172.21.15.16:0/3282449693 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f24b806dbc0 msgr2=0x7f24b815f650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:19.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.461+0000 7f24c0400640 1 --2- 172.21.15.16:0/3282449693 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f24b806dbc0 0x7f24b815f650 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f24ac0097c0 tx=0x7f24ac01c3e0 comp rx=0 tx=0).stop 2024-03-31T15:15:19.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.461+0000 7f24c0400640 1 -- 172.21.15.16:0/3282449693 shutdown_connections 2024-03-31T15:15:19.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.461+0000 7f24c0400640 1 --2- 172.21.15.16:0/3282449693 >> v2:172.21.15.89:3300/0 conn(0x7f24b80cb440 0x7f24b806f920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:19.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.461+0000 7f24c0400640 1 --2- 172.21.15.16:0/3282449693 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f24b806dbc0 0x7f24b815f650 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:19.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.461+0000 7f24c0400640 1 -- 172.21.15.16:0/3282449693 >> 172.21.15.16:0/3282449693 conn(0x7f24b8067330 msgr2=0x7f24b8067740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:15:19.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.461+0000 7f24c0400640 1 -- 172.21.15.16:0/3282449693 shutdown_connections 2024-03-31T15:15:19.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.461+0000 7f24c0400640 1 -- 172.21.15.16:0/3282449693 wait complete. 2024-03-31T15:15:19.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.465+0000 7f24c0400640 1 Processor -- start 2024-03-31T15:15:19.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.465+0000 7f24c0400640 1 -- start start 2024-03-31T15:15:19.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.465+0000 7f24c0400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f24b806dbc0 0x7f24b813d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:19.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.465+0000 7f24c0400640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f24b80cb440 0x7f24b8140170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:19.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.465+0000 7f24c0400640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f24b80cb8c0 0x7f24b81406b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:19.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.465+0000 7f24c0400640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f24b8162ca0 con 0x7f24b806dbc0 2024-03-31T15:15:19.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.465+0000 7f24c0400640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f24b8070270 con 0x7f24b80cb8c0 2024-03-31T15:15:19.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.465+0000 7f24c0400640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f24b806c950 con 0x7f24b80cb440 2024-03-31T15:15:19.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.465+0000 7f24bda00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f24b806dbc0 0x7f24b813d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:19.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.465+0000 7f24bda00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f24b806dbc0 0x7f24b813d570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37914/0 (socket says 172.21.15.16:37914) 2024-03-31T15:15:19.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.465+0000 7f24bda00640 1 -- 172.21.15.16:0/1147127947 learned_addr learned my addr 172.21.15.16:0/1147127947 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:15:19.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.465+0000 7f24bd000640 1 --2- 172.21.15.16:0/1147127947 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f24b80cb440 0x7f24b8140170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:19.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.465+0000 7f24be400640 1 --2- 172.21.15.16:0/1147127947 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f24b80cb8c0 0x7f24b81406b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:19.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.465+0000 7f24bda00640 1 -- 172.21.15.16:0/1147127947 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f24b80cb440 msgr2=0x7f24b8140170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:19.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.465+0000 7f24bda00640 1 --2- 172.21.15.16:0/1147127947 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f24b80cb440 0x7f24b8140170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:19.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.465+0000 7f24bda00640 1 -- 172.21.15.16:0/1147127947 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f24b80cb8c0 msgr2=0x7f24b81406b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:19.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.465+0000 7f24bda00640 1 --2- 172.21.15.16:0/1147127947 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f24b80cb8c0 0x7f24b81406b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:19.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.465+0000 7f24bda00640 1 -- 172.21.15.16:0/1147127947 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24b81413d0 con 0x7f24b806dbc0 2024-03-31T15:15:19.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.465+0000 7f24bda00640 1 --2- 172.21.15.16:0/1147127947 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f24b806dbc0 0x7f24b813d570 secure :-1 s=READY pgs=1924 cs=0 l=1 rev1=1 crypto rx=0x7f24ac0ad040 tx=0x7f24ac01ac20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:15:19.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.465+0000 7f24b5600640 1 -- 172.21.15.16:0/1147127947 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24ac0bf410 con 0x7f24b806dbc0 2024-03-31T15:15:19.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.465+0000 7f24b5600640 1 -- 172.21.15.16:0/1147127947 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f24ac0bf5f0 con 0x7f24b806dbc0 2024-03-31T15:15:19.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.465+0000 7f24c0400640 1 -- 172.21.15.16:0/1147127947 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f24b81416b0 con 0x7f24b806dbc0 2024-03-31T15:15:19.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.465+0000 7f24b5600640 1 -- 172.21.15.16:0/1147127947 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24ac0bf9e0 con 0x7f24b806dbc0 2024-03-31T15:15:19.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.465+0000 7f24c0400640 1 -- 172.21.15.16:0/1147127947 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f24b813dab0 con 0x7f24b806dbc0 2024-03-31T15:15:19.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.469+0000 7f24c0400640 1 -- 172.21.15.16:0/1147127947 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f24b8069950 con 0x7f24b806dbc0 2024-03-31T15:15:19.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.469+0000 7f24b5600640 1 -- 172.21.15.16:0/1147127947 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f24ac0bd020 con 0x7f24b806dbc0 2024-03-31T15:15:19.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.469+0000 7f24b5600640 1 --2- 172.21.15.16:0/1147127947 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f2498068380 0x7f249806a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:19.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.469+0000 7f24b5600640 1 -- 172.21.15.16:0/1147127947 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(116..116 src has 1..116) ==== 5115+0+0 (secure 0 0 0) 0x7f24ac120bb0 con 0x7f24b806dbc0 2024-03-31T15:15:19.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.469+0000 7f24bd000640 1 --2- 172.21.15.16:0/1147127947 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f2498068380 0x7f249806a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:19.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.469+0000 7f24bd000640 1 --2- 172.21.15.16:0/1147127947 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f2498068380 0x7f249806a840 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7f24a4009580 tx=0x7f24a4009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:15:19.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.473+0000 7f24b5600640 1 -- 172.21.15.16:0/1147127947 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f24ac016090 con 0x7f24b806dbc0 2024-03-31T15:15:19.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.637+0000 7f24c0400640 1 -- 172.21.15.16:0/1147127947 --> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "5", "target": ["mon-mgr", ""]}) -- 0x7f24b813dda0 con 0x7f2498068380 2024-03-31T15:15:19.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.637+0000 7f24b5600640 1 -- 172.21.15.16:0/1147127947 <== mgr.4100 v2:172.21.15.89:6832/1549407273 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 5 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f24b813dda0 con 0x7f2498068380 2024-03-31T15:15:19.627 INFO:teuthology.orchestra.run.smithi016.stderr:instructed osd(s) 5 to deep-scrub 2024-03-31T15:15:19.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.641+0000 7f24c0400640 1 -- 172.21.15.16:0/1147127947 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f2498068380 msgr2=0x7f249806a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:19.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.641+0000 7f24c0400640 1 --2- 172.21.15.16:0/1147127947 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f2498068380 0x7f249806a840 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7f24a4009580 tx=0x7f24a4009190 comp rx=0 tx=0).stop 2024-03-31T15:15:19.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.641+0000 7f24c0400640 1 -- 172.21.15.16:0/1147127947 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f24b806dbc0 msgr2=0x7f24b813d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:19.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.641+0000 7f24c0400640 1 --2- 172.21.15.16:0/1147127947 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f24b806dbc0 0x7f24b813d570 secure :-1 s=READY pgs=1924 cs=0 l=1 rev1=1 crypto rx=0x7f24ac0ad040 tx=0x7f24ac01ac20 comp rx=0 tx=0).stop 2024-03-31T15:15:19.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.641+0000 7f24c0400640 1 -- 172.21.15.16:0/1147127947 shutdown_connections 2024-03-31T15:15:19.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.641+0000 7f24c0400640 1 --2- 172.21.15.16:0/1147127947 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f2498068380 0x7f249806a840 unknown :-1 s=CLOSED pgs=810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:19.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.641+0000 7f24c0400640 1 --2- 172.21.15.16:0/1147127947 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f24b80cb8c0 0x7f24b81406b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:19.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.641+0000 7f24c0400640 1 --2- 172.21.15.16:0/1147127947 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f24b80cb440 0x7f24b8140170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:19.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.641+0000 7f24c0400640 1 --2- 172.21.15.16:0/1147127947 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f24b806dbc0 0x7f24b813d570 unknown :-1 s=CLOSED pgs=1924 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:19.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.641+0000 7f24c0400640 1 -- 172.21.15.16:0/1147127947 >> 172.21.15.16:0/1147127947 conn(0x7f24b8067330 msgr2=0x7f24b815d050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:15:19.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.641+0000 7f24c0400640 1 -- 172.21.15.16:0/1147127947 shutdown_connections 2024-03-31T15:15:19.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.641+0000 7f24c0400640 1 -- 172.21.15.16:0/1147127947 wait complete. 2024-03-31T15:15:19.649 INFO:tasks.ceph:Scrubbing osd.6 2024-03-31T15:15:19.649 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 config set osd_debug_deep_scrub_sleep 0 2024-03-31T15:15:19.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.805+0000 7f144be00640 1 Processor -- start 2024-03-31T15:15:19.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.805+0000 7f144be00640 1 -- start start 2024-03-31T15:15:19.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.805+0000 7f144be00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f144415e400 0x7f14440b1be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:19.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.805+0000 7f144be00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f1444073090 0x7f14440b2120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:19.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.805+0000 7f144be00640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f14440b39f0 con 0x7f14440d05f0 2024-03-31T15:15:19.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.805+0000 7f144be00640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f14440b4250 con 0x7f144415e400 2024-03-31T15:15:19.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.805+0000 7f144be00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f1444072ca0 con 0x7f1444073090 2024-03-31T15:15:19.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.805+0000 7f1449400640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f144415e400 0x7f14440b1be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:19.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.805+0000 7f1449e00640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f14440d05f0 0x7f14441620d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:42918/0 (socket says 172.21.15.16:42918) 2024-03-31T15:15:19.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.805+0000 7f1449e00640 1 -- 172.21.15.16:0/3210411943 learned_addr learned my addr 172.21.15.16:0/3210411943 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:15:19.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.805+0000 7f1448a00640 1 --2- 172.21.15.16:0/3210411943 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f1444073090 0x7f14440b2120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:19.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.805+0000 7f1449400640 1 -- 172.21.15.16:0/3210411943 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f1444073090 msgr2=0x7f14440b2120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:15:19.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.805+0000 7f1449400640 1 --2- 172.21.15.16:0/3210411943 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f1444073090 0x7f14440b2120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:19.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.805+0000 7f1449400640 1 -- 172.21.15.16:0/3210411943 >> v1:172.21.15.16:6789/0 conn(0x7f14440d05f0 legacy=0x7f14441620d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:19.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.805+0000 7f1449400640 1 -- 172.21.15.16:0/3210411943 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14440d0b90 con 0x7f144415e400 2024-03-31T15:15:19.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.805+0000 7f1443e00640 1 -- 172.21.15.16:0/3210411943 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 299637565 0 0) 0x7f14440b39f0 con 0x7f14440d05f0 2024-03-31T15:15:19.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.809+0000 7f1449400640 1 --2- 172.21.15.16:0/3210411943 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f144415e400 0x7f14440b1be0 secure :-1 s=READY pgs=1926 cs=0 l=1 rev1=1 crypto rx=0x7f14380097c0 tx=0x7f143801c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c4e52e1a30eae8d server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:15:19.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.809+0000 7f1443e00640 1 -- 172.21.15.16:0/3210411943 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14380b0070 con 0x7f144415e400 2024-03-31T15:15:19.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.809+0000 7f1443e00640 1 -- 172.21.15.16:0/3210411943 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f14380ab600 con 0x7f144415e400 2024-03-31T15:15:19.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.809+0000 7f1443e00640 1 -- 172.21.15.16:0/3210411943 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14380ab9d0 con 0x7f144415e400 2024-03-31T15:15:19.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.809+0000 7f144be00640 1 -- 172.21.15.16:0/3210411943 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f144415e400 msgr2=0x7f14440b1be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:19.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.809+0000 7f144be00640 1 --2- 172.21.15.16:0/3210411943 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f144415e400 0x7f14440b1be0 secure :-1 s=READY pgs=1926 cs=0 l=1 rev1=1 crypto rx=0x7f14380097c0 tx=0x7f143801c3e0 comp rx=0 tx=0).stop 2024-03-31T15:15:19.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.809+0000 7f144be00640 1 -- 172.21.15.16:0/3210411943 shutdown_connections 2024-03-31T15:15:19.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.809+0000 7f144be00640 1 --2- 172.21.15.16:0/3210411943 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f1444073090 0x7f14440b2120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:19.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.809+0000 7f144be00640 1 --2- 172.21.15.16:0/3210411943 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f144415e400 0x7f14440b1be0 unknown :-1 s=CLOSED pgs=1926 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:19.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.809+0000 7f144be00640 1 -- 172.21.15.16:0/3210411943 >> 172.21.15.16:0/3210411943 conn(0x7f144406b620 msgr2=0x7f144406ba30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:15:19.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.809+0000 7f144be00640 1 -- 172.21.15.16:0/3210411943 shutdown_connections 2024-03-31T15:15:19.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.809+0000 7f144be00640 1 -- 172.21.15.16:0/3210411943 wait complete. 2024-03-31T15:15:19.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.813+0000 7f144be00640 1 Processor -- start 2024-03-31T15:15:19.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.813+0000 7f144be00640 1 -- start start 2024-03-31T15:15:19.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.813+0000 7f144be00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1444073090 0x7f1444149950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:19.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.813+0000 7f144be00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f14440d05f0 0x7f14441631a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:19.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.813+0000 7f144be00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f144415e400 0x7f1444157be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:19.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.813+0000 7f144be00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f14440ae0f0 con 0x7f1444073090 2024-03-31T15:15:19.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.813+0000 7f144be00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f14440adf70 con 0x7f144415e400 2024-03-31T15:15:19.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.813+0000 7f144be00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f14440b3050 con 0x7f14440d05f0 2024-03-31T15:15:19.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.813+0000 7f1449400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1444073090 0x7f1444149950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:19.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.813+0000 7f1449400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1444073090 0x7f1444149950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37930/0 (socket says 172.21.15.16:37930) 2024-03-31T15:15:19.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.813+0000 7f1449400640 1 -- 172.21.15.16:0/3553545300 learned_addr learned my addr 172.21.15.16:0/3553545300 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:15:19.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.813+0000 7f1449e00640 1 --2- 172.21.15.16:0/3553545300 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f144415e400 0x7f1444157be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:19.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.813+0000 7f1448a00640 1 --2- 172.21.15.16:0/3553545300 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f14440d05f0 0x7f14441631a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:19.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.813+0000 7f1449400640 1 -- 172.21.15.16:0/3553545300 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f14440d05f0 msgr2=0x7f14441631a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:19.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.813+0000 7f1449400640 1 --2- 172.21.15.16:0/3553545300 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f14440d05f0 0x7f14441631a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:19.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.813+0000 7f1449400640 1 -- 172.21.15.16:0/3553545300 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f144415e400 msgr2=0x7f1444157be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:19.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.813+0000 7f1449400640 1 --2- 172.21.15.16:0/3553545300 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f144415e400 0x7f1444157be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:19.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.813+0000 7f1449400640 1 -- 172.21.15.16:0/3553545300 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1444149e90 con 0x7f1444073090 2024-03-31T15:15:19.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.813+0000 7f1449400640 1 --2- 172.21.15.16:0/3553545300 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1444073090 0x7f1444149950 secure :-1 s=READY pgs=1927 cs=0 l=1 rev1=1 crypto rx=0x7f143801a010 tx=0x7f143801ac20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:15:19.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.813+0000 7f1441600640 1 -- 172.21.15.16:0/3553545300 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14380b0050 con 0x7f1444073090 2024-03-31T15:15:19.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.813+0000 7f1441600640 1 -- 172.21.15.16:0/3553545300 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f14380bc070 con 0x7f1444073090 2024-03-31T15:15:19.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.813+0000 7f1441600640 1 -- 172.21.15.16:0/3553545300 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14380b5830 con 0x7f1444073090 2024-03-31T15:15:19.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.813+0000 7f144be00640 1 -- 172.21.15.16:0/3553545300 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f14440b2000 con 0x7f1444073090 2024-03-31T15:15:19.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.813+0000 7f144be00640 1 -- 172.21.15.16:0/3553545300 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f14440cd0c0 con 0x7f1444073090 2024-03-31T15:15:19.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.813+0000 7f144be00640 1 -- 172.21.15.16:0/3553545300 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f1444069950 con 0x7f1444073090 2024-03-31T15:15:19.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.817+0000 7f1441600640 1 -- 172.21.15.16:0/3553545300 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f14380b59d0 con 0x7f1444073090 2024-03-31T15:15:19.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.817+0000 7f1441600640 1 --2- 172.21.15.16:0/3553545300 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f1424068380 0x7f142406a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:19.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.817+0000 7f1441600640 1 -- 172.21.15.16:0/3553545300 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(116..116 src has 1..116) ==== 5115+0+0 (secure 0 0 0) 0x7f1438121a50 con 0x7f1444073090 2024-03-31T15:15:19.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.817+0000 7f1441600640 1 --2- 172.21.15.16:0/3553545300 >> [v2:172.21.15.89:6809/3470471752,v1:172.21.15.89:6812/3470471752] conn(0x7f142406e380 0x7f14240707a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:19.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.817+0000 7f1441600640 1 -- 172.21.15.16:0/3553545300 --> [v2:172.21.15.89:6809/3470471752,v1:172.21.15.89:6812/3470471752] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f14380c3e70 con 0x7f142406e380 2024-03-31T15:15:19.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.817+0000 7f1441600640 1 -- 172.21.15.16:0/3553545300 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_get_version_reply(handle=1 version=116) ==== 24+0+0 (secure 0 0 0) 0x7f1438121ee0 con 0x7f1444073090 2024-03-31T15:15:19.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.817+0000 7f1448a00640 1 --2- 172.21.15.16:0/3553545300 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f1424068380 0x7f142406a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:19.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.817+0000 7f1449e00640 1 --2- 172.21.15.16:0/3553545300 >> [v2:172.21.15.89:6809/3470471752,v1:172.21.15.89:6812/3470471752] conn(0x7f142406e380 0x7f14240707a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:19.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.817+0000 7f1448a00640 1 --2- 172.21.15.16:0/3553545300 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f1424068380 0x7f142406a840 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7f14440cf550 tx=0x7f1430008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:15:19.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.817+0000 7f1449e00640 1 --2- 172.21.15.16:0/3553545300 >> [v2:172.21.15.89:6809/3470471752,v1:172.21.15.89:6812/3470471752] conn(0x7f142406e380 0x7f14240707a0 crc :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:15:19.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.817+0000 7f1441600640 1 -- 172.21.15.16:0/3553545300 <== osd.6 v2:172.21.15.89:6809/3470471752 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f14380c3e70 con 0x7f142406e380 2024-03-31T15:15:19.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.833+0000 7f144be00640 1 -- 172.21.15.16:0/3553545300 --> [v2:172.21.15.89:6809/3470471752,v1:172.21.15.89:6812/3470471752] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f144415cf40 con 0x7f142406e380 2024-03-31T15:15:19.822 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.833+0000 7f1441600640 1 -- 172.21.15.16:0/3553545300 <== osd.6 v2:172.21.15.89:6809/3470471752 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f144415cf40 con 0x7f142406e380 2024-03-31T15:15:19.822 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-03-31T15:15:19.822 INFO:teuthology.orchestra.run.smithi016.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-31T15:15:19.822 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-03-31T15:15:19.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.833+0000 7f144be00640 1 -- 172.21.15.16:0/3553545300 >> [v2:172.21.15.89:6809/3470471752,v1:172.21.15.89:6812/3470471752] conn(0x7f142406e380 msgr2=0x7f14240707a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:19.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.833+0000 7f144be00640 1 --2- 172.21.15.16:0/3553545300 >> [v2:172.21.15.89:6809/3470471752,v1:172.21.15.89:6812/3470471752] conn(0x7f142406e380 0x7f14240707a0 crc :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:19.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.833+0000 7f144be00640 1 -- 172.21.15.16:0/3553545300 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f1424068380 msgr2=0x7f142406a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:19.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.833+0000 7f144be00640 1 --2- 172.21.15.16:0/3553545300 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f1424068380 0x7f142406a840 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7f14440cf550 tx=0x7f1430008040 comp rx=0 tx=0).stop 2024-03-31T15:15:19.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.833+0000 7f144be00640 1 -- 172.21.15.16:0/3553545300 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1444073090 msgr2=0x7f1444149950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:19.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.833+0000 7f144be00640 1 --2- 172.21.15.16:0/3553545300 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1444073090 0x7f1444149950 secure :-1 s=READY pgs=1927 cs=0 l=1 rev1=1 crypto rx=0x7f143801a010 tx=0x7f143801ac20 comp rx=0 tx=0).stop 2024-03-31T15:15:19.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.833+0000 7f1449e00640 1 -- 172.21.15.16:0/3553545300 reap_dead start 2024-03-31T15:15:19.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.833+0000 7f144be00640 1 -- 172.21.15.16:0/3553545300 shutdown_connections 2024-03-31T15:15:19.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.833+0000 7f144be00640 1 -- 172.21.15.16:0/3553545300 >> 172.21.15.16:0/3553545300 conn(0x7f144406b620 msgr2=0x7f14440afac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:15:19.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.833+0000 7f144be00640 1 -- 172.21.15.16:0/3553545300 shutdown_connections 2024-03-31T15:15:19.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.833+0000 7f144be00640 1 -- 172.21.15.16:0/3553545300 wait complete. 2024-03-31T15:15:19.840 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 6 2024-03-31T15:15:19.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.989+0000 7fe87ba00640 1 Processor -- start 2024-03-31T15:15:19.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.993+0000 7fe87ba00640 1 -- start start 2024-03-31T15:15:19.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.993+0000 7fe87ba00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fe87406dbc0 0x7fe87415f650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:19.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.993+0000 7fe87ba00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fe874070030 0x7fe874162180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:19.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.993+0000 7fe87ba00640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe8740cbe90 con 0x7fe8740cb440 2024-03-31T15:15:19.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.993+0000 7fe87ba00640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7fe87415fb90 con 0x7fe874070030 2024-03-31T15:15:19.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.993+0000 7fe87ba00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fe87415f1c0 con 0x7fe87406dbc0 2024-03-31T15:15:19.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.993+0000 7fe879000640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fe87406dbc0 0x7fe87415f650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:19.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.993+0000 7fe86be00640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fe8740cb440 0x7fe87406f920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.16:49038/0 (socket says 172.21.15.16:49038) 2024-03-31T15:15:19.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.993+0000 7fe86be00640 1 -- 172.21.15.16:0/769614162 learned_addr learned my addr 172.21.15.16:0/769614162 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:15:19.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.993+0000 7fe879a00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fe874070030 0x7fe874162180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:19.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.993+0000 7fe873e00640 1 -- 172.21.15.16:0/769614162 <== mon.1 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 258595990 0 0) 0x7fe8740cbe90 con 0x7fe8740cb440 2024-03-31T15:15:19.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.993+0000 7fe873e00640 1 -- 172.21.15.16:0/769614162 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe858003530 con 0x7fe8740cb440 2024-03-31T15:15:19.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.993+0000 7fe879a00640 1 -- 172.21.15.16:0/769614162 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fe87406dbc0 msgr2=0x7fe87415f650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:15:19.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.993+0000 7fe879a00640 1 --2- 172.21.15.16:0/769614162 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fe87406dbc0 0x7fe87415f650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:19.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.993+0000 7fe879a00640 1 -- 172.21.15.16:0/769614162 >> v1:172.21.15.89:6789/0 conn(0x7fe8740cb440 legacy=0x7fe87406f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:19.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.993+0000 7fe879a00640 1 -- 172.21.15.16:0/769614162 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8740cc170 con 0x7fe874070030 2024-03-31T15:15:19.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.993+0000 7fe879000640 1 --2- 172.21.15.16:0/769614162 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fe87406dbc0 0x7fe87415f650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:15:19.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.997+0000 7fe879a00640 1 --2- 172.21.15.16:0/769614162 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe874070030 0x7fe874162180 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7fe86c009d90 tx=0x7fe86c01c8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=996eeb029c54e0cc server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:15:19.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.997+0000 7fe873e00640 1 -- 172.21.15.16:0/769614162 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe86c0b2070 con 0x7fe874070030 2024-03-31T15:15:19.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.997+0000 7fe873e00640 1 -- 172.21.15.16:0/769614162 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe86c0a5880 con 0x7fe874070030 2024-03-31T15:15:19.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.997+0000 7fe873e00640 1 -- 172.21.15.16:0/769614162 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe86c0a5c50 con 0x7fe874070030 2024-03-31T15:15:19.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.997+0000 7fe87ba00640 1 -- 172.21.15.16:0/769614162 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe874070030 msgr2=0x7fe874162180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:19.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.997+0000 7fe87ba00640 1 --2- 172.21.15.16:0/769614162 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe874070030 0x7fe874162180 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7fe86c009d90 tx=0x7fe86c01c8b0 comp rx=0 tx=0).stop 2024-03-31T15:15:19.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.997+0000 7fe87ba00640 1 -- 172.21.15.16:0/769614162 shutdown_connections 2024-03-31T15:15:19.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.997+0000 7fe87ba00640 1 --2- 172.21.15.16:0/769614162 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe874070030 0x7fe874162180 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:19.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.997+0000 7fe87ba00640 1 --2- 172.21.15.16:0/769614162 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fe87406dbc0 0x7fe87415f650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:19.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.997+0000 7fe87ba00640 1 -- 172.21.15.16:0/769614162 >> 172.21.15.16:0/769614162 conn(0x7fe874067330 msgr2=0x7fe874067740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:15:19.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.997+0000 7fe87ba00640 1 -- 172.21.15.16:0/769614162 shutdown_connections 2024-03-31T15:15:19.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:19.997+0000 7fe87ba00640 1 -- 172.21.15.16:0/769614162 wait complete. 2024-03-31T15:15:19.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.001+0000 7fe87ba00640 1 Processor -- start 2024-03-31T15:15:19.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.001+0000 7fe87ba00640 1 -- start start 2024-03-31T15:15:19.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.001+0000 7fe87ba00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fe87406dbc0 0x7fe8740b6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:19.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.001+0000 7fe87ba00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe874070030 0x7fe874143570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:19.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.001+0000 7fe87ba00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe8740cb440 0x7fe874141700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:19.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.001+0000 7fe87ba00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fe87406cad0 con 0x7fe8740cb440 2024-03-31T15:15:19.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.001+0000 7fe87ba00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fe87406c950 con 0x7fe874070030 2024-03-31T15:15:19.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.001+0000 7fe87ba00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fe87416b520 con 0x7fe87406dbc0 2024-03-31T15:15:19.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.001+0000 7fe879a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe8740cb440 0x7fe874141700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:19.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.001+0000 7fe879a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe8740cb440 0x7fe874141700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37936/0 (socket says 172.21.15.16:37936) 2024-03-31T15:15:19.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.001+0000 7fe879a00640 1 -- 172.21.15.16:0/1643723166 learned_addr learned my addr 172.21.15.16:0/1643723166 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:15:19.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.001+0000 7fe86be00640 1 --2- 172.21.15.16:0/1643723166 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe874070030 0x7fe874143570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:19.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.001+0000 7fe879000640 1 --2- 172.21.15.16:0/1643723166 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fe87406dbc0 0x7fe8740b6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:19.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.001+0000 7fe879a00640 1 -- 172.21.15.16:0/1643723166 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fe87406dbc0 msgr2=0x7fe8740b6190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:19.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.001+0000 7fe879a00640 1 --2- 172.21.15.16:0/1643723166 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fe87406dbc0 0x7fe8740b6190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:19.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.001+0000 7fe879a00640 1 -- 172.21.15.16:0/1643723166 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe874070030 msgr2=0x7fe874143570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:19.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.001+0000 7fe879a00640 1 --2- 172.21.15.16:0/1643723166 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe874070030 0x7fe874143570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:19.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.001+0000 7fe879a00640 1 -- 172.21.15.16:0/1643723166 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8740b6900 con 0x7fe8740cb440 2024-03-31T15:15:19.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.001+0000 7fe879a00640 1 --2- 172.21.15.16:0/1643723166 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe8740cb440 0x7fe874141700 secure :-1 s=READY pgs=1928 cs=0 l=1 rev1=1 crypto rx=0x7fe86c0b0040 tx=0x7fe86c019650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:15:19.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.001+0000 7fe871600640 1 -- 172.21.15.16:0/1643723166 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe86c0b2050 con 0x7fe8740cb440 2024-03-31T15:15:19.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.001+0000 7fe871600640 1 -- 172.21.15.16:0/1643723166 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe86c0be070 con 0x7fe8740cb440 2024-03-31T15:15:19.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.001+0000 7fe871600640 1 -- 172.21.15.16:0/1643723166 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe86c0ad890 con 0x7fe8740cb440 2024-03-31T15:15:19.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.001+0000 7fe87ba00640 1 -- 172.21.15.16:0/1643723166 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe8740b6be0 con 0x7fe8740cb440 2024-03-31T15:15:19.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.001+0000 7fe87ba00640 1 -- 172.21.15.16:0/1643723166 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe8740b71a0 con 0x7fe8740cb440 2024-03-31T15:15:19.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.001+0000 7fe87ba00640 1 -- 172.21.15.16:0/1643723166 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe874069950 con 0x7fe8740cb440 2024-03-31T15:15:19.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.005+0000 7fe871600640 1 -- 172.21.15.16:0/1643723166 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fe86c0ada30 con 0x7fe8740cb440 2024-03-31T15:15:19.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.005+0000 7fe871600640 1 --2- 172.21.15.16:0/1643723166 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fe850068380 0x7fe85006a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:19.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.005+0000 7fe871600640 1 -- 172.21.15.16:0/1643723166 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(116..116 src has 1..116) ==== 5115+0+0 (secure 0 0 0) 0x7fe86c121760 con 0x7fe8740cb440 2024-03-31T15:15:19.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.005+0000 7fe879000640 1 --2- 172.21.15.16:0/1643723166 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fe850068380 0x7fe85006a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:19.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.005+0000 7fe879000640 1 --2- 172.21.15.16:0/1643723166 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fe850068380 0x7fe85006a840 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7fe874144220 tx=0x7fe864009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:15:19.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.009+0000 7fe871600640 1 -- 172.21.15.16:0/1643723166 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fe86c016560 con 0x7fe8740cb440 2024-03-31T15:15:20.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.173+0000 7fe87ba00640 1 -- 172.21.15.16:0/1643723166 --> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "6", "target": ["mon-mgr", ""]}) -- 0x7fe8740cf380 con 0x7fe850068380 2024-03-31T15:15:20.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.173+0000 7fe871600640 1 -- 172.21.15.16:0/1643723166 <== mgr.4100 v2:172.21.15.89:6832/1549407273 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 6 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7fe8740cf380 con 0x7fe850068380 2024-03-31T15:15:20.163 INFO:teuthology.orchestra.run.smithi016.stderr:instructed osd(s) 6 to deep-scrub 2024-03-31T15:15:20.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.177+0000 7fe87ba00640 1 -- 172.21.15.16:0/1643723166 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fe850068380 msgr2=0x7fe85006a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:20.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.177+0000 7fe87ba00640 1 --2- 172.21.15.16:0/1643723166 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fe850068380 0x7fe85006a840 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7fe874144220 tx=0x7fe864009190 comp rx=0 tx=0).stop 2024-03-31T15:15:20.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.177+0000 7fe87ba00640 1 -- 172.21.15.16:0/1643723166 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe8740cb440 msgr2=0x7fe874141700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:20.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.177+0000 7fe87ba00640 1 --2- 172.21.15.16:0/1643723166 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe8740cb440 0x7fe874141700 secure :-1 s=READY pgs=1928 cs=0 l=1 rev1=1 crypto rx=0x7fe86c0b0040 tx=0x7fe86c019650 comp rx=0 tx=0).stop 2024-03-31T15:15:20.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.177+0000 7fe87ba00640 1 -- 172.21.15.16:0/1643723166 shutdown_connections 2024-03-31T15:15:20.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.177+0000 7fe87ba00640 1 --2- 172.21.15.16:0/1643723166 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fe850068380 0x7fe85006a840 unknown :-1 s=CLOSED pgs=812 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:20.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.177+0000 7fe87ba00640 1 --2- 172.21.15.16:0/1643723166 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe8740cb440 0x7fe874141700 unknown :-1 s=CLOSED pgs=1928 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:20.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.177+0000 7fe87ba00640 1 --2- 172.21.15.16:0/1643723166 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe874070030 0x7fe874143570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:20.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.177+0000 7fe87ba00640 1 --2- 172.21.15.16:0/1643723166 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fe87406dbc0 0x7fe8740b6190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:20.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.177+0000 7fe87ba00640 1 -- 172.21.15.16:0/1643723166 >> 172.21.15.16:0/1643723166 conn(0x7fe874067330 msgr2=0x7fe8740d15c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:15:20.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.177+0000 7fe87ba00640 1 -- 172.21.15.16:0/1643723166 shutdown_connections 2024-03-31T15:15:20.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.177+0000 7fe87ba00640 1 -- 172.21.15.16:0/1643723166 wait complete. 2024-03-31T15:15:20.184 INFO:tasks.ceph:Scrubbing osd.7 2024-03-31T15:15:20.184 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 config set osd_debug_deep_scrub_sleep 0 2024-03-31T15:15:20.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.341+0000 7fea03a00640 1 Processor -- start 2024-03-31T15:15:20.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.341+0000 7fea03a00640 1 -- start start 2024-03-31T15:15:20.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.341+0000 7fea03a00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fe9fc0cc7b0 0x7fe9fc0b3dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:20.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.341+0000 7fea03a00640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe9fc0ce2d0 con 0x7fe9fc0b7900 2024-03-31T15:15:20.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.341+0000 7fea03a00640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe9fc0b4700 con 0x7fe9fc0b4300 2024-03-31T15:15:20.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.341+0000 7fea03a00640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7fe9fc0ceb10 con 0x7fe9fc0cc7b0 2024-03-31T15:15:20.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.341+0000 7fea01000640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fe9fc0b7900 0x7fe9fc160740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:42920/0 (socket says 172.21.15.16:42920) 2024-03-31T15:15:20.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.341+0000 7fea01000640 1 -- 172.21.15.16:0/2873106273 learned_addr learned my addr 172.21.15.16:0/2873106273 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:15:20.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.341+0000 7fe9f3e00640 1 --2- 172.21.15.16:0/2873106273 >> v2:172.21.15.89:3300/0 conn(0x7fe9fc0cc7b0 0x7fe9fc0b3dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:20.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.341+0000 7fe9fbe00640 1 -- 172.21.15.16:0/2873106273 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2550274734 0 0) 0x7fe9ec001120 con 0x7fe9fc0b7900 2024-03-31T15:15:20.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.341+0000 7fe9fbe00640 1 -- 172.21.15.16:0/2873106273 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe9e0003530 con 0x7fe9fc0b7900 2024-03-31T15:15:20.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.341+0000 7fe9fbe00640 1 -- 172.21.15.16:0/2873106273 <== mon.1 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 889892453 0 0) 0x7fe9fc0b4700 con 0x7fe9fc0b4300 2024-03-31T15:15:20.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.341+0000 7fe9fbe00640 1 -- 172.21.15.16:0/2873106273 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe9ec001120 con 0x7fe9fc0b4300 2024-03-31T15:15:20.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.341+0000 7fe9fbe00640 1 -- 172.21.15.16:0/2873106273 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3183981591 0 0) 0x7fe9e0003530 con 0x7fe9fc0b7900 2024-03-31T15:15:20.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.341+0000 7fe9fbe00640 1 -- 172.21.15.16:0/2873106273 >> v2:172.21.15.89:3300/0 conn(0x7fe9fc0cc7b0 msgr2=0x7fe9fc0b3dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:15:20.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.341+0000 7fe9fbe00640 1 --2- 172.21.15.16:0/2873106273 >> v2:172.21.15.89:3300/0 conn(0x7fe9fc0cc7b0 0x7fe9fc0b3dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:20.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.341+0000 7fe9fbe00640 1 -- 172.21.15.16:0/2873106273 >> v1:172.21.15.89:6789/0 conn(0x7fe9fc0b4300 legacy=0x7fe9fc0d1a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:20.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.341+0000 7fe9fbe00640 1 -- 172.21.15.16:0/2873106273 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9fc0ce550 con 0x7fe9fc0b7900 2024-03-31T15:15:20.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.341+0000 7fe9fbe00640 1 -- 172.21.15.16:0/2873106273 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 214807623 0 0) 0x7fe9ec001850 con 0x7fe9fc0b7900 2024-03-31T15:15:20.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.341+0000 7fe9fbe00640 1 -- 172.21.15.16:0/2873106273 >> v1:172.21.15.16:6789/0 conn(0x7fe9fc0b7900 legacy=0x7fe9fc160740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:20.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.341+0000 7fe9fbe00640 1 --2- 172.21.15.16:0/2873106273 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fe9e0003b50 0x7fe9e0003f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:20.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.341+0000 7fe9fbe00640 1 --2- 172.21.15.16:0/2873106273 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe9e0004810 0x7fe9e0004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:20.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.341+0000 7fe9fbe00640 1 --2- 172.21.15.16:0/2873106273 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe9e00051d0 0x7fe9e000b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:20.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.341+0000 7fe9fbe00640 1 -- 172.21.15.16:0/2873106273 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fe9fc0ceb10 con 0x7fe9e00051d0 2024-03-31T15:15:20.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.341+0000 7fe9fbe00640 1 -- 172.21.15.16:0/2873106273 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fe9e000c100 con 0x7fe9e0004810 2024-03-31T15:15:20.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.341+0000 7fe9fbe00640 1 -- 172.21.15.16:0/2873106273 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fe9e000c300 con 0x7fe9e0003b50 2024-03-31T15:15:20.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.345+0000 7fea01000640 1 --2- 172.21.15.16:0/2873106273 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fe9e0003b50 0x7fe9e0003f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:20.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.345+0000 7fe9f3e00640 1 --2- 172.21.15.16:0/2873106273 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe9e0004810 0x7fe9e0004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:20.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.345+0000 7fea01a00640 1 --2- 172.21.15.16:0/2873106273 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe9e00051d0 0x7fe9e000b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:20.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.345+0000 7fea01000640 1 -- 172.21.15.16:0/2873106273 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe9e0004810 msgr2=0x7fe9e0004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:15:20.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.345+0000 7fea01000640 1 --2- 172.21.15.16:0/2873106273 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe9e0004810 0x7fe9e0004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:20.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.345+0000 7fea01000640 1 -- 172.21.15.16:0/2873106273 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe9e00051d0 msgr2=0x7fe9e000b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:15:20.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.345+0000 7fea01000640 1 --2- 172.21.15.16:0/2873106273 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe9e00051d0 0x7fe9e000b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:20.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.345+0000 7fea01000640 1 -- 172.21.15.16:0/2873106273 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9e0003530 con 0x7fe9e0003b50 2024-03-31T15:15:20.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.345+0000 7fea01a00640 1 --2- 172.21.15.16:0/2873106273 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe9e00051d0 0x7fe9e000b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:15:20.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.345+0000 7fea01a00640 1 -- 172.21.15.16:0/2873106273 reap_dead start 2024-03-31T15:15:20.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.345+0000 7fe9f3e00640 1 --2- 172.21.15.16:0/2873106273 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe9e0004810 0x7fe9e0004c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:15:20.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.345+0000 7fea01000640 1 --2- 172.21.15.16:0/2873106273 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fe9e0003b50 0x7fe9e0003f50 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fe9ec00c4c0 tx=0x7fe9ec01cc00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=779773ef121c9300 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:15:20.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.345+0000 7fe9fbe00640 1 -- 172.21.15.16:0/2873106273 <== mon.2 v2:172.21.15.89:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9ec00c0e0 con 0x7fe9e0003b50 2024-03-31T15:15:20.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.345+0000 7fe9fbe00640 1 -- 172.21.15.16:0/2873106273 <== mon.2 v2:172.21.15.89:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe9ec0a9020 con 0x7fe9e0003b50 2024-03-31T15:15:20.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.345+0000 7fe9fbe00640 1 -- 172.21.15.16:0/2873106273 <== mon.2 v2:172.21.15.89:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9ec005ae0 con 0x7fe9e0003b50 2024-03-31T15:15:20.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.349+0000 7fea03a00640 1 -- 172.21.15.16:0/2873106273 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fe9e0003b50 msgr2=0x7fe9e0003f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:20.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.349+0000 7fea03a00640 1 --2- 172.21.15.16:0/2873106273 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fe9e0003b50 0x7fe9e0003f50 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fe9ec00c4c0 tx=0x7fe9ec01cc00 comp rx=0 tx=0).stop 2024-03-31T15:15:20.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.349+0000 7fea03a00640 1 -- 172.21.15.16:0/2873106273 shutdown_connections 2024-03-31T15:15:20.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.349+0000 7fea03a00640 1 --2- 172.21.15.16:0/2873106273 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fe9e0003b50 0x7fe9e0003f50 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:20.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.349+0000 7fea03a00640 1 -- 172.21.15.16:0/2873106273 >> 172.21.15.16:0/2873106273 conn(0x7fe9fc06b7f0 msgr2=0x7fe9fc067330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:15:20.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.349+0000 7fea03a00640 1 -- 172.21.15.16:0/2873106273 shutdown_connections 2024-03-31T15:15:20.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.349+0000 7fea03a00640 1 -- 172.21.15.16:0/2873106273 wait complete. 2024-03-31T15:15:20.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.349+0000 7fea03a00640 1 Processor -- start 2024-03-31T15:15:20.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.349+0000 7fea03a00640 1 -- start start 2024-03-31T15:15:20.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.349+0000 7fea03a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe9e0003b50 0x7fe9fc153690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:20.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.349+0000 7fea03a00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fe9fc156610 0x7fe9fc1549d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:20.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.349+0000 7fea03a00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe9fc156a90 0x7fe9fc154f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:20.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.349+0000 7fea03a00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fe9fc160220 con 0x7fe9e0003b50 2024-03-31T15:15:20.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.349+0000 7fea03a00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fe9fc15ff40 con 0x7fe9fc156a90 2024-03-31T15:15:20.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.349+0000 7fea03a00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fe9fc0b7ce0 con 0x7fe9fc156610 2024-03-31T15:15:20.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.349+0000 7fea01000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe9e0003b50 0x7fe9fc153690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:20.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.349+0000 7fea01000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe9e0003b50 0x7fe9fc153690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37948/0 (socket says 172.21.15.16:37948) 2024-03-31T15:15:20.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.349+0000 7fea01000640 1 -- 172.21.15.16:0/2192690202 learned_addr learned my addr 172.21.15.16:0/2192690202 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:15:20.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.349+0000 7fe9f3e00640 1 --2- 172.21.15.16:0/2192690202 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fe9fc156610 0x7fe9fc1549d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:20.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.349+0000 7fea01a00640 1 --2- 172.21.15.16:0/2192690202 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe9fc156a90 0x7fe9fc154f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:20.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.349+0000 7fea01000640 1 -- 172.21.15.16:0/2192690202 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fe9fc156610 msgr2=0x7fe9fc1549d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:20.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.349+0000 7fea01000640 1 --2- 172.21.15.16:0/2192690202 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fe9fc156610 0x7fe9fc1549d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:20.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.349+0000 7fea01000640 1 -- 172.21.15.16:0/2192690202 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe9fc156a90 msgr2=0x7fe9fc154f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:20.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.349+0000 7fea01000640 1 --2- 172.21.15.16:0/2192690202 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe9fc156a90 0x7fe9fc154f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:20.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.349+0000 7fea01000640 1 -- 172.21.15.16:0/2192690202 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9fc155570 con 0x7fe9e0003b50 2024-03-31T15:15:20.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.349+0000 7fea01a00640 1 --2- 172.21.15.16:0/2192690202 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe9fc156a90 0x7fe9fc154f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:15:20.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.349+0000 7fe9f3e00640 1 --2- 172.21.15.16:0/2192690202 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fe9fc156610 0x7fe9fc1549d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:15:20.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.353+0000 7fea01000640 1 --2- 172.21.15.16:0/2192690202 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe9e0003b50 0x7fe9fc153690 secure :-1 s=READY pgs=1930 cs=0 l=1 rev1=1 crypto rx=0x7fe9ec0ab040 tx=0x7fe9e0005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:15:20.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.353+0000 7fe9f9600640 1 -- 172.21.15.16:0/2192690202 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9ec004cc0 con 0x7fe9e0003b50 2024-03-31T15:15:20.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.353+0000 7fe9f9600640 1 -- 172.21.15.16:0/2192690202 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe9ec0b2070 con 0x7fe9e0003b50 2024-03-31T15:15:20.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.353+0000 7fe9f9600640 1 -- 172.21.15.16:0/2192690202 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9ec004e60 con 0x7fe9e0003b50 2024-03-31T15:15:20.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.353+0000 7fea03a00640 1 -- 172.21.15.16:0/2192690202 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe9fc155850 con 0x7fe9e0003b50 2024-03-31T15:15:20.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.353+0000 7fea03a00640 1 -- 172.21.15.16:0/2192690202 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe9fc155e10 con 0x7fe9e0003b50 2024-03-31T15:15:20.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.353+0000 7fea03a00640 1 -- 172.21.15.16:0/2192690202 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fe9fc069950 con 0x7fe9e0003b50 2024-03-31T15:15:20.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.353+0000 7fe9f9600640 1 -- 172.21.15.16:0/2192690202 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fe9ec0a59f0 con 0x7fe9e0003b50 2024-03-31T15:15:20.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.353+0000 7fe9f9600640 1 --2- 172.21.15.16:0/2192690202 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fe9d80683b0 0x7fe9d806a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:20.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.353+0000 7fe9f9600640 1 -- 172.21.15.16:0/2192690202 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(116..116 src has 1..116) ==== 5115+0+0 (secure 0 0 0) 0x7fe9ec120120 con 0x7fe9e0003b50 2024-03-31T15:15:20.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.353+0000 7fe9f9600640 1 --2- 172.21.15.16:0/2192690202 >> [v2:172.21.15.89:6808/1796778630,v1:172.21.15.89:6811/1796778630] conn(0x7fe9d806e3b0 0x7fe9d80707d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:20.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.353+0000 7fe9f9600640 1 -- 172.21.15.16:0/2192690202 --> [v2:172.21.15.89:6808/1796778630,v1:172.21.15.89:6811/1796778630] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fe9ec0aae90 con 0x7fe9d806e3b0 2024-03-31T15:15:20.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.353+0000 7fe9f9600640 1 -- 172.21.15.16:0/2192690202 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_get_version_reply(handle=1 version=116) ==== 24+0+0 (secure 0 0 0) 0x7fe9ec1205b0 con 0x7fe9e0003b50 2024-03-31T15:15:20.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.353+0000 7fe9f3e00640 1 --2- 172.21.15.16:0/2192690202 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fe9d80683b0 0x7fe9d806a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:20.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.353+0000 7fea01a00640 1 --2- 172.21.15.16:0/2192690202 >> [v2:172.21.15.89:6808/1796778630,v1:172.21.15.89:6811/1796778630] conn(0x7fe9d806e3b0 0x7fe9d80707d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:20.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.357+0000 7fe9f3e00640 1 --2- 172.21.15.16:0/2192690202 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fe9d80683b0 0x7fe9d806a870 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7fe9fc1547c0 tx=0x7fe9e4008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:15:20.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.357+0000 7fea01a00640 1 --2- 172.21.15.16:0/2192690202 >> [v2:172.21.15.89:6808/1796778630,v1:172.21.15.89:6811/1796778630] conn(0x7fe9d806e3b0 0x7fe9d80707d0 crc :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:15:20.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.357+0000 7fe9f9600640 1 -- 172.21.15.16:0/2192690202 <== osd.7 v2:172.21.15.89:6808/1796778630 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fe9ec0aae90 con 0x7fe9d806e3b0 2024-03-31T15:15:20.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.369+0000 7fea03a00640 1 -- 172.21.15.16:0/2192690202 --> [v2:172.21.15.89:6808/1796778630,v1:172.21.15.89:6811/1796778630] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7fe9fc0b6f80 con 0x7fe9d806e3b0 2024-03-31T15:15:20.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.369+0000 7fe9f9600640 1 -- 172.21.15.16:0/2192690202 <== osd.7 v2:172.21.15.89:6808/1796778630 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7fe9fc0b6f80 con 0x7fe9d806e3b0 2024-03-31T15:15:20.360 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-03-31T15:15:20.360 INFO:teuthology.orchestra.run.smithi016.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-31T15:15:20.360 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-03-31T15:15:20.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.369+0000 7fea03a00640 1 -- 172.21.15.16:0/2192690202 >> [v2:172.21.15.89:6808/1796778630,v1:172.21.15.89:6811/1796778630] conn(0x7fe9d806e3b0 msgr2=0x7fe9d80707d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:20.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.369+0000 7fea03a00640 1 --2- 172.21.15.16:0/2192690202 >> [v2:172.21.15.89:6808/1796778630,v1:172.21.15.89:6811/1796778630] conn(0x7fe9d806e3b0 0x7fe9d80707d0 crc :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:20.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.373+0000 7fea03a00640 1 -- 172.21.15.16:0/2192690202 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fe9d80683b0 msgr2=0x7fe9d806a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:20.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.373+0000 7fea03a00640 1 --2- 172.21.15.16:0/2192690202 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fe9d80683b0 0x7fe9d806a870 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7fe9fc1547c0 tx=0x7fe9e4008040 comp rx=0 tx=0).stop 2024-03-31T15:15:20.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.373+0000 7fea03a00640 1 -- 172.21.15.16:0/2192690202 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe9e0003b50 msgr2=0x7fe9fc153690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:20.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.373+0000 7fea03a00640 1 --2- 172.21.15.16:0/2192690202 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe9e0003b50 0x7fe9fc153690 secure :-1 s=READY pgs=1930 cs=0 l=1 rev1=1 crypto rx=0x7fe9ec0ab040 tx=0x7fe9e0005840 comp rx=0 tx=0).stop 2024-03-31T15:15:20.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.373+0000 7fea01a00640 1 -- 172.21.15.16:0/2192690202 reap_dead start 2024-03-31T15:15:20.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.373+0000 7fea03a00640 1 -- 172.21.15.16:0/2192690202 shutdown_connections 2024-03-31T15:15:20.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.373+0000 7fea03a00640 1 -- 172.21.15.16:0/2192690202 >> 172.21.15.16:0/2192690202 conn(0x7fe9fc06b7f0 msgr2=0x7fe9fc15c890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:15:20.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.373+0000 7fea03a00640 1 -- 172.21.15.16:0/2192690202 shutdown_connections 2024-03-31T15:15:20.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.373+0000 7fea03a00640 1 -- 172.21.15.16:0/2192690202 wait complete. 2024-03-31T15:15:20.378 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 7 2024-03-31T15:15:20.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.529+0000 7fd4bca00640 1 Processor -- start 2024-03-31T15:15:20.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.529+0000 7fd4bca00640 1 -- start start 2024-03-31T15:15:20.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.533+0000 7fd4bca00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fd4b8070030 0x7fd4b8162180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:20.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.533+0000 7fd4bca00640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd4b80cbe90 con 0x7fd4b815f650 2024-03-31T15:15:20.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.533+0000 7fd4bca00640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd4b80cc170 con 0x7fd4b806dbc0 2024-03-31T15:15:20.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.533+0000 7fd4bca00640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7fd4b815f1c0 con 0x7fd4b8070030 2024-03-31T15:15:20.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.533+0000 7fd4ad400640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fd4b815f650 0x7fd4b806f920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:42922/0 (socket says 172.21.15.16:42922) 2024-03-31T15:15:20.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.533+0000 7fd4ad400640 1 -- 172.21.15.16:0/3272062445 learned_addr learned my addr 172.21.15.16:0/3272062445 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:15:20.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.533+0000 7fd4b6800640 1 --2- 172.21.15.16:0/3272062445 >> v2:172.21.15.89:3300/0 conn(0x7fd4b8070030 0x7fd4b8162180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:20.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.533+0000 7fd4b5400640 1 -- 172.21.15.16:0/3272062445 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3413939284 0 0) 0x7fd4b80cbe90 con 0x7fd4b815f650 2024-03-31T15:15:20.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.533+0000 7fd4b5400640 1 -- 172.21.15.16:0/3272062445 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd49c003530 con 0x7fd4b815f650 2024-03-31T15:15:20.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.533+0000 7fd4b5400640 1 -- 172.21.15.16:0/3272062445 <== mon.1 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1681191759 0 0) 0x7fd4b80cc170 con 0x7fd4b806dbc0 2024-03-31T15:15:20.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.533+0000 7fd4b5400640 1 -- 172.21.15.16:0/3272062445 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd4b80cbe90 con 0x7fd4b806dbc0 2024-03-31T15:15:20.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.533+0000 7fd4b5400640 1 -- 172.21.15.16:0/3272062445 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4218244606 0 0) 0x7fd49c003530 con 0x7fd4b815f650 2024-03-31T15:15:20.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.533+0000 7fd4b6800640 1 -- 172.21.15.16:0/3272062445 >> v1:172.21.15.89:6789/0 conn(0x7fd4b806dbc0 legacy=0x7fd4b80cb440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:20.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.533+0000 7fd4b6800640 1 -- 172.21.15.16:0/3272062445 >> v1:172.21.15.16:6789/0 conn(0x7fd4b815f650 legacy=0x7fd4b806f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:20.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.533+0000 7fd4b6800640 1 -- 172.21.15.16:0/3272062445 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd4b815ee50 con 0x7fd4b8070030 2024-03-31T15:15:20.525 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.537+0000 7fd4b6800640 1 --2- 172.21.15.16:0/3272062445 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd4b8070030 0x7fd4b8162180 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7fd4b0009d90 tx=0x7fd4b001c800 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e748e273a6f64f43 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:15:20.525 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.537+0000 7fd4b5400640 1 -- 172.21.15.16:0/3272062445 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4b00b2070 con 0x7fd4b8070030 2024-03-31T15:15:20.525 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.537+0000 7fd4b5400640 1 -- 172.21.15.16:0/3272062445 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd4b00a5600 con 0x7fd4b8070030 2024-03-31T15:15:20.525 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.537+0000 7fd4b5400640 1 -- 172.21.15.16:0/3272062445 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4a0001680 con 0x7fd4b8070030 2024-03-31T15:15:20.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.537+0000 7fd4bca00640 1 -- 172.21.15.16:0/3272062445 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd4b8070030 msgr2=0x7fd4b8162180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:20.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.537+0000 7fd4bca00640 1 --2- 172.21.15.16:0/3272062445 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd4b8070030 0x7fd4b8162180 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7fd4b0009d90 tx=0x7fd4b001c800 comp rx=0 tx=0).stop 2024-03-31T15:15:20.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.537+0000 7fd4bca00640 1 -- 172.21.15.16:0/3272062445 shutdown_connections 2024-03-31T15:15:20.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.537+0000 7fd4bca00640 1 --2- 172.21.15.16:0/3272062445 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd4b8070030 0x7fd4b8162180 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:20.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.537+0000 7fd4bca00640 1 -- 172.21.15.16:0/3272062445 >> 172.21.15.16:0/3272062445 conn(0x7fd4b8067330 msgr2=0x7fd4b8067740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:15:20.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.537+0000 7fd4bca00640 1 -- 172.21.15.16:0/3272062445 shutdown_connections 2024-03-31T15:15:20.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.537+0000 7fd4bca00640 1 -- 172.21.15.16:0/3272062445 wait complete. 2024-03-31T15:15:20.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.537+0000 7fd4bca00640 1 Processor -- start 2024-03-31T15:15:20.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.537+0000 7fd4bca00640 1 -- start start 2024-03-31T15:15:20.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.537+0000 7fd4bca00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd4b806dbc0 0x7fd4b80b58a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:20.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.537+0000 7fd4bca00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd4b8070030 0x7fd4b813f570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:20.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.537+0000 7fd4bca00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fd4b815f650 0x7fd4b81595e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:20.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.537+0000 7fd4bca00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fd4b806c950 con 0x7fd4b8070030 2024-03-31T15:15:20.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.537+0000 7fd4bca00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fd4b806dfa0 con 0x7fd4b806dbc0 2024-03-31T15:15:20.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.537+0000 7fd4bca00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fd4b806cad0 con 0x7fd4b815f650 2024-03-31T15:15:20.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.537+0000 7fd4ad400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd4b8070030 0x7fd4b813f570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:20.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.541+0000 7fd4ad400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd4b8070030 0x7fd4b813f570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37956/0 (socket says 172.21.15.16:37956) 2024-03-31T15:15:20.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.541+0000 7fd4ad400640 1 -- 172.21.15.16:0/4142988431 learned_addr learned my addr 172.21.15.16:0/4142988431 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:15:20.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.541+0000 7fd4b5e00640 1 --2- 172.21.15.16:0/4142988431 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd4b806dbc0 0x7fd4b80b58a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:20.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.541+0000 7fd4b6800640 1 --2- 172.21.15.16:0/4142988431 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fd4b815f650 0x7fd4b81595e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:20.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.541+0000 7fd4ad400640 1 -- 172.21.15.16:0/4142988431 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fd4b815f650 msgr2=0x7fd4b81595e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:20.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.541+0000 7fd4ad400640 1 --2- 172.21.15.16:0/4142988431 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fd4b815f650 0x7fd4b81595e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:20.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.541+0000 7fd4ad400640 1 -- 172.21.15.16:0/4142988431 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd4b806dbc0 msgr2=0x7fd4b80b58a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:20.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.541+0000 7fd4ad400640 1 --2- 172.21.15.16:0/4142988431 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd4b806dbc0 0x7fd4b80b58a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:20.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.541+0000 7fd4ad400640 1 -- 172.21.15.16:0/4142988431 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd4b80b61a0 con 0x7fd4b8070030 2024-03-31T15:15:20.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.541+0000 7fd4ad400640 1 --2- 172.21.15.16:0/4142988431 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd4b8070030 0x7fd4b813f570 secure :-1 s=READY pgs=1932 cs=0 l=1 rev1=1 crypto rx=0x7fd4a000c600 tx=0x7fd4a000cac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:15:20.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.541+0000 7fd4aea00640 1 -- 172.21.15.16:0/4142988431 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4a0017070 con 0x7fd4b8070030 2024-03-31T15:15:20.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.541+0000 7fd4aea00640 1 -- 172.21.15.16:0/4142988431 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd4a0005d10 con 0x7fd4b8070030 2024-03-31T15:15:20.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.541+0000 7fd4aea00640 1 -- 172.21.15.16:0/4142988431 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4a00072b0 con 0x7fd4b8070030 2024-03-31T15:15:20.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.541+0000 7fd4bca00640 1 -- 172.21.15.16:0/4142988431 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd4b80b6480 con 0x7fd4b8070030 2024-03-31T15:15:20.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.541+0000 7fd4bca00640 1 -- 172.21.15.16:0/4142988431 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd4b80b6a40 con 0x7fd4b8070030 2024-03-31T15:15:20.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.541+0000 7fd4bca00640 1 -- 172.21.15.16:0/4142988431 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd4b8069950 con 0x7fd4b8070030 2024-03-31T15:15:20.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.541+0000 7fd4aea00640 1 -- 172.21.15.16:0/4142988431 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fd4a001a070 con 0x7fd4b8070030 2024-03-31T15:15:20.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.545+0000 7fd4aea00640 1 --2- 172.21.15.16:0/4142988431 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fd494068380 0x7fd49406a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:20.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.545+0000 7fd4aea00640 1 -- 172.21.15.16:0/4142988431 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(116..116 src has 1..116) ==== 5115+0+0 (secure 0 0 0) 0x7fd4a0085ce0 con 0x7fd4b8070030 2024-03-31T15:15:20.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.545+0000 7fd4b5e00640 1 --2- 172.21.15.16:0/4142988431 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fd494068380 0x7fd49406a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:20.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.545+0000 7fd4b5e00640 1 --2- 172.21.15.16:0/4142988431 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fd494068380 0x7fd49406a840 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7fd4a80011f0 tx=0x7fd4a8009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:15:20.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.549+0000 7fd4aea00640 1 -- 172.21.15.16:0/4142988431 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fd4a0055e00 con 0x7fd4b8070030 2024-03-31T15:15:20.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.713+0000 7fd4bca00640 1 -- 172.21.15.16:0/4142988431 --> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "7", "target": ["mon-mgr", ""]}) -- 0x7fd4b815e4b0 con 0x7fd494068380 2024-03-31T15:15:20.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.713+0000 7fd4aea00640 1 -- 172.21.15.16:0/4142988431 <== mgr.4100 v2:172.21.15.89:6832/1549407273 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 7 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7fd4b815e4b0 con 0x7fd494068380 2024-03-31T15:15:20.703 INFO:teuthology.orchestra.run.smithi016.stderr:instructed osd(s) 7 to deep-scrub 2024-03-31T15:15:20.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.713+0000 7fd4bca00640 1 -- 172.21.15.16:0/4142988431 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fd494068380 msgr2=0x7fd49406a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:20.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.713+0000 7fd4bca00640 1 --2- 172.21.15.16:0/4142988431 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fd494068380 0x7fd49406a840 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7fd4a80011f0 tx=0x7fd4a8009040 comp rx=0 tx=0).stop 2024-03-31T15:15:20.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.713+0000 7fd4bca00640 1 -- 172.21.15.16:0/4142988431 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd4b8070030 msgr2=0x7fd4b813f570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:20.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.713+0000 7fd4bca00640 1 --2- 172.21.15.16:0/4142988431 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd4b8070030 0x7fd4b813f570 secure :-1 s=READY pgs=1932 cs=0 l=1 rev1=1 crypto rx=0x7fd4a000c600 tx=0x7fd4a000cac0 comp rx=0 tx=0).stop 2024-03-31T15:15:20.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.713+0000 7fd4bca00640 1 -- 172.21.15.16:0/4142988431 shutdown_connections 2024-03-31T15:15:20.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.713+0000 7fd4bca00640 1 --2- 172.21.15.16:0/4142988431 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fd494068380 0x7fd49406a840 unknown :-1 s=CLOSED pgs=814 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:20.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.713+0000 7fd4bca00640 1 --2- 172.21.15.16:0/4142988431 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fd4b815f650 0x7fd4b81595e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:20.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.713+0000 7fd4bca00640 1 --2- 172.21.15.16:0/4142988431 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd4b8070030 0x7fd4b813f570 unknown :-1 s=CLOSED pgs=1932 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:20.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.713+0000 7fd4bca00640 1 --2- 172.21.15.16:0/4142988431 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd4b806dbc0 0x7fd4b80b58a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:20.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.713+0000 7fd4bca00640 1 -- 172.21.15.16:0/4142988431 >> 172.21.15.16:0/4142988431 conn(0x7fd4b8067330 msgr2=0x7fd4b80ce300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:15:20.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.717+0000 7fd4bca00640 1 -- 172.21.15.16:0/4142988431 shutdown_connections 2024-03-31T15:15:20.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.717+0000 7fd4bca00640 1 -- 172.21.15.16:0/4142988431 wait complete. 2024-03-31T15:15:20.724 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-31T15:15:20.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.873+0000 7f62f9400640 1 Processor -- start 2024-03-31T15:15:20.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.873+0000 7f62f9400640 1 -- start start 2024-03-31T15:15:20.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.877+0000 7f62f9400640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f62f4165dc0 0x7f62f40ccfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:20.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.877+0000 7f62f9400640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f62f40b48d0 0x7f62f40cc130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:20.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.877+0000 7f62f9400640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f62f4072f40 con 0x7f62f40b4d30 2024-03-31T15:15:20.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.877+0000 7f62f9400640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f62f4071500 con 0x7f62f40b48d0 2024-03-31T15:15:20.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.877+0000 7f62f9400640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f62f4073570 con 0x7f62f4165dc0 2024-03-31T15:15:20.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.877+0000 7f62ebe00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f62f40b48d0 0x7f62f40cc130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:20.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.877+0000 7f62ebe00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f62f40b48d0 0x7f62f40cc130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37972/0 (socket says 172.21.15.16:37972) 2024-03-31T15:15:20.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.877+0000 7f62ebe00640 1 -- 172.21.15.16:0/1571346589 learned_addr learned my addr 172.21.15.16:0/1571346589 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:15:20.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.877+0000 7f62f2800640 1 --2- 172.21.15.16:0/1571346589 >> v2:172.21.15.89:3300/0 conn(0x7f62f4165dc0 0x7f62f40ccfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:20.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.877+0000 7f62ebe00640 1 -- 172.21.15.16:0/1571346589 >> v2:172.21.15.89:3300/0 conn(0x7f62f4165dc0 msgr2=0x7f62f40ccfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:15:20.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.877+0000 7f62ebe00640 1 --2- 172.21.15.16:0/1571346589 >> v2:172.21.15.89:3300/0 conn(0x7f62f4165dc0 0x7f62f40ccfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:20.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.877+0000 7f62ebe00640 1 -- 172.21.15.16:0/1571346589 >> v1:172.21.15.89:6789/0 conn(0x7f62f40b4d30 legacy=0x7f62f40cdea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:15:20.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.877+0000 7f62ebe00640 1 -- 172.21.15.16:0/1571346589 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62f40b51e0 con 0x7f62f40b48d0 2024-03-31T15:15:20.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.877+0000 7f62f2800640 1 --2- 172.21.15.16:0/1571346589 >> v2:172.21.15.89:3300/0 conn(0x7f62f4165dc0 0x7f62f40ccfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:15:20.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.881+0000 7f62ebe00640 1 --2- 172.21.15.16:0/1571346589 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f62f40b48d0 0x7f62f40cc130 secure :-1 s=READY pgs=1933 cs=0 l=1 rev1=1 crypto rx=0x7f62f40736f0 tx=0x7f62dc01c420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5982e0265889e10b server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:15:20.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.881+0000 7f62f1e00640 1 -- 172.21.15.16:0/1571346589 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62dc0b0070 con 0x7f62f40b48d0 2024-03-31T15:15:20.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.881+0000 7f62f1e00640 1 -- 172.21.15.16:0/1571346589 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f62dc0ab600 con 0x7f62f40b48d0 2024-03-31T15:15:20.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.881+0000 7f62f1e00640 1 -- 172.21.15.16:0/1571346589 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62dc0ab9d0 con 0x7f62f40b48d0 2024-03-31T15:15:20.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.881+0000 7f62f9400640 1 -- 172.21.15.16:0/1571346589 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f62f40b48d0 msgr2=0x7f62f40cc130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:20.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.881+0000 7f62f9400640 1 --2- 172.21.15.16:0/1571346589 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f62f40b48d0 0x7f62f40cc130 secure :-1 s=READY pgs=1933 cs=0 l=1 rev1=1 crypto rx=0x7f62f40736f0 tx=0x7f62dc01c420 comp rx=0 tx=0).stop 2024-03-31T15:15:20.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.881+0000 7f62f9400640 1 -- 172.21.15.16:0/1571346589 shutdown_connections 2024-03-31T15:15:20.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.881+0000 7f62f9400640 1 --2- 172.21.15.16:0/1571346589 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f62f40b48d0 0x7f62f40cc130 unknown :-1 s=CLOSED pgs=1933 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:20.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.881+0000 7f62f9400640 1 --2- 172.21.15.16:0/1571346589 >> v2:172.21.15.89:3300/0 conn(0x7f62f4165dc0 0x7f62f40ccfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:20.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.881+0000 7f62f9400640 1 -- 172.21.15.16:0/1571346589 >> 172.21.15.16:0/1571346589 conn(0x7f62f406b620 msgr2=0x7f62f406ba30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:15:20.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.881+0000 7f62f9400640 1 -- 172.21.15.16:0/1571346589 shutdown_connections 2024-03-31T15:15:20.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.881+0000 7f62f9400640 1 -- 172.21.15.16:0/1571346589 wait complete. 2024-03-31T15:15:20.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.881+0000 7f62f9400640 1 Processor -- start 2024-03-31T15:15:20.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.881+0000 7f62f9400640 1 -- start start 2024-03-31T15:15:20.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.881+0000 7f62f9400640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f62f40b48d0 0x7f62f41ff640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:20.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.881+0000 7f62f9400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f62f40b4d30 0x7f62f415c1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:20.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.881+0000 7f62f9400640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f62f4165dc0 0x7f62f41551b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:20.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.881+0000 7f62f9400640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f62f4166320 con 0x7f62f40b4d30 2024-03-31T15:15:20.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.881+0000 7f62f9400640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f62f41661a0 con 0x7f62f4165dc0 2024-03-31T15:15:20.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.881+0000 7f62f9400640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f62f416f980 con 0x7f62f40b48d0 2024-03-31T15:15:20.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.881+0000 7f62ebe00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f62f40b4d30 0x7f62f415c1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:20.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.881+0000 7f62ebe00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f62f40b4d30 0x7f62f415c1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37980/0 (socket says 172.21.15.16:37980) 2024-03-31T15:15:20.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.881+0000 7f62ebe00640 1 -- 172.21.15.16:0/2850422955 learned_addr learned my addr 172.21.15.16:0/2850422955 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:15:20.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.885+0000 7f62f2800640 1 --2- 172.21.15.16:0/2850422955 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f62f40b48d0 0x7f62f41ff640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:20.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.885+0000 7f62f3200640 1 --2- 172.21.15.16:0/2850422955 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f62f4165dc0 0x7f62f41551b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:20.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.885+0000 7f62ebe00640 1 -- 172.21.15.16:0/2850422955 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f62f40b48d0 msgr2=0x7f62f41ff640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:20.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.885+0000 7f62ebe00640 1 --2- 172.21.15.16:0/2850422955 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f62f40b48d0 0x7f62f41ff640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:20.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.885+0000 7f62ebe00640 1 -- 172.21.15.16:0/2850422955 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f62f4165dc0 msgr2=0x7f62f41551b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:20.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.885+0000 7f62ebe00640 1 --2- 172.21.15.16:0/2850422955 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f62f4165dc0 0x7f62f41551b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:20.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.885+0000 7f62ebe00640 1 -- 172.21.15.16:0/2850422955 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62f42000c0 con 0x7f62f40b4d30 2024-03-31T15:15:20.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.885+0000 7f62ebe00640 1 --2- 172.21.15.16:0/2850422955 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f62f40b4d30 0x7f62f415c1a0 secure :-1 s=READY pgs=1934 cs=0 l=1 rev1=1 crypto rx=0x7f62dc0ae040 tx=0x7f62dc01ac60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:15:20.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.885+0000 7f62eaa00640 1 -- 172.21.15.16:0/2850422955 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62dc0ba020 con 0x7f62f40b4d30 2024-03-31T15:15:20.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.885+0000 7f62eaa00640 1 -- 172.21.15.16:0/2850422955 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f62dc0b6640 con 0x7f62f40b4d30 2024-03-31T15:15:20.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.885+0000 7f62eaa00640 1 -- 172.21.15.16:0/2850422955 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62dc0b0040 con 0x7f62f40b4d30 2024-03-31T15:15:20.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.885+0000 7f62f9400640 1 -- 172.21.15.16:0/2850422955 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f62f42003a0 con 0x7f62f40b4d30 2024-03-31T15:15:20.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.885+0000 7f62f9400640 1 -- 172.21.15.16:0/2850422955 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f62f4164f10 con 0x7f62f40b4d30 2024-03-31T15:15:20.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.885+0000 7f62f9400640 1 -- 172.21.15.16:0/2850422955 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f62f4200570 con 0x7f62f40b4d30 2024-03-31T15:15:20.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.885+0000 7f62eaa00640 1 -- 172.21.15.16:0/2850422955 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f62dc0bd020 con 0x7f62f40b4d30 2024-03-31T15:15:20.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.885+0000 7f62eaa00640 1 --2- 172.21.15.16:0/2850422955 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f62d00683b0 0x7f62d006a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:20.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.885+0000 7f62eaa00640 1 -- 172.21.15.16:0/2850422955 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(116..116 src has 1..116) ==== 5115+0+0 (secure 0 0 0) 0x7f62dc120b40 con 0x7f62f40b4d30 2024-03-31T15:15:20.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.889+0000 7f62f2800640 1 --2- 172.21.15.16:0/2850422955 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f62d00683b0 0x7f62d006a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:20.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.889+0000 7f62f2800640 1 --2- 172.21.15.16:0/2850422955 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f62d00683b0 0x7f62d006a870 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7f62e4006fd0 tx=0x7f62e4008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:15:20.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:20.889+0000 7f62eaa00640 1 -- 172.21.15.16:0/2850422955 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f62dc0c4030 con 0x7f62f40b4d30 2024-03-31T15:15:21.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:21.053+0000 7f62f9400640 1 -- 172.21.15.16:0/2850422955 --> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f62f41653d0 con 0x7f62d00683b0 2024-03-31T15:15:21.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:21.053+0000 7f62eaa00640 1 -- 172.21.15.16:0/2850422955 <== mgr.4100 v2:172.21.15.89:6832/1549407273 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+13390 (secure 0 0 0) 0x7f62f41653d0 con 0x7f62d00683b0 2024-03-31T15:15:21.044 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:15:21.044 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-03-31T15:15:21.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:21.057+0000 7f62f9400640 1 -- 172.21.15.16:0/2850422955 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f62d00683b0 msgr2=0x7f62d006a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:21.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:21.057+0000 7f62f9400640 1 --2- 172.21.15.16:0/2850422955 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f62d00683b0 0x7f62d006a870 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7f62e4006fd0 tx=0x7f62e4008040 comp rx=0 tx=0).stop 2024-03-31T15:15:21.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:21.057+0000 7f62f9400640 1 -- 172.21.15.16:0/2850422955 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f62f40b4d30 msgr2=0x7f62f415c1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:21.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:21.057+0000 7f62f9400640 1 --2- 172.21.15.16:0/2850422955 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f62f40b4d30 0x7f62f415c1a0 secure :-1 s=READY pgs=1934 cs=0 l=1 rev1=1 crypto rx=0x7f62dc0ae040 tx=0x7f62dc01ac60 comp rx=0 tx=0).stop 2024-03-31T15:15:21.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:21.057+0000 7f62f9400640 1 -- 172.21.15.16:0/2850422955 shutdown_connections 2024-03-31T15:15:21.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:21.057+0000 7f62f9400640 1 --2- 172.21.15.16:0/2850422955 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f62d00683b0 0x7f62d006a870 unknown :-1 s=CLOSED pgs=815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:21.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:21.057+0000 7f62f9400640 1 --2- 172.21.15.16:0/2850422955 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f62f4165dc0 0x7f62f41551b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:21.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:21.057+0000 7f62f9400640 1 --2- 172.21.15.16:0/2850422955 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f62f40b4d30 0x7f62f415c1a0 unknown :-1 s=CLOSED pgs=1934 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:21.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:21.057+0000 7f62f9400640 1 --2- 172.21.15.16:0/2850422955 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f62f40b48d0 0x7f62f41ff640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:21.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:21.057+0000 7f62f9400640 1 -- 172.21.15.16:0/2850422955 >> 172.21.15.16:0/2850422955 conn(0x7f62f406b620 msgr2=0x7f62f406df40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:15:21.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:21.057+0000 7f62f9400640 1 -- 172.21.15.16:0/2850422955 shutdown_connections 2024-03-31T15:15:21.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:21.057+0000 7f62f9400640 1 -- 172.21.15.16:0/2850422955 wait complete. 2024-03-31T15:15:21.064 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":468,"stamp":"2024-03-31T15:15:20.711046+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":266032,"kb_used_data":12424,"kb_used_omap":12,"kb_used_meta":253491,"kb_avail":754708688,"statfs":{"total":773094113280,"available":772821696512,"internally_reserved":0,"allocated":12722176,"data_stored":6194816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":259575344},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-8388625,"num_objects":-5,"num_object_clones":0,"num_object_copies":-10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-5,"num_whiteouts":0,"num_read":-32,"num_read_kb":-13,"num_write":-29,"num_write_kb":-8195,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.506571"},"pg_stats":[{"pgid":"1.0","version":"16'117","reported_seq":348,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-31T15:15:12.483895+0000","last_change":"2024-03-31T15:03:24.329890+0000","last_active":"2024-03-31T15:15:12.483895+0000","last_peered":"2024-03-31T15:15:12.483895+0000","last_clean":"2024-03-31T15:15:12.483895+0000","last_became_active":"2024-03-31T15:03:24.328882+0000","last_became_peered":"2024-03-31T15:03:24.328882+0000","last_unstale":"2024-03-31T15:15:12.483895+0000","last_undegraded":"2024-03-31T15:15:12.483895+0000","last_fullsized":"2024-03-31T15:15:12.483895+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:03:22.885535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:03:22.885535+0000","last_clean_scrub_stamp":"2024-03-31T15:03:22.885535+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:18:29.004296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509588,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33168,"kb_used_data":1988,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338672,"statfs":{"total":96636764160,"available":96602800128,"internally_reserved":0,"allocated":2035712,"data_stored":1217128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509587,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32588,"kb_used_data":1408,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339252,"statfs":{"total":96636764160,"available":96603394048,"internally_reserved":0,"allocated":1441792,"data_stored":626760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":15,"seq":64424509586,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32588,"kb_used_data":1408,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339252,"statfs":{"total":96636764160,"available":96603394048,"internally_reserved":0,"allocated":1441792,"data_stored":626760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509585,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":36752,"kb_used_data":1408,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335088,"statfs":{"total":96636764160,"available":96599130112,"internally_reserved":0,"allocated":1441792,"data_stored":626760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542291,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32592,"kb_used_data":1408,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339248,"statfs":{"total":96636764160,"available":96603389952,"internally_reserved":0,"allocated":1441792,"data_stored":626760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542291,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32588,"kb_used_data":1408,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339252,"statfs":{"total":96636764160,"available":96603394048,"internally_reserved":0,"allocated":1441792,"data_stored":626760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542291,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32588,"kb_used_data":1408,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339252,"statfs":{"total":96636764160,"available":96603394048,"internally_reserved":0,"allocated":1441792,"data_stored":626760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542291,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33168,"kb_used_data":1988,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338672,"statfs":{"total":96636764160,"available":96602800128,"internally_reserved":0,"allocated":2035712,"data_stored":1217128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-31T15:15:21.067 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-03-31T15:03:22.885535+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=31, tm_hour=15, tm_min=3, tm_sec=22, tm_wday=6, tm_yday=91, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=31, tm_hour=15, tm_min=15, tm_sec=15, tm_wday=6, tm_yday=91, tm_isdst=0) 2024-03-31T15:15:21.068 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-03-31T15:15:38.043 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-31T15:15:38.047 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-31T15:15:41.069 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-31T15:15:41.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:41.233+0000 7f9bb5e00640 1 Processor -- start 2024-03-31T15:15:41.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:41.233+0000 7f9bb5e00640 1 -- start start 2024-03-31T15:15:41.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:41.233+0000 7f9bb5e00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f9bb006dbc0 0x7f9bb015f650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:41.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:41.233+0000 7f9bb5e00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f9bb00cb440 0x7f9bb006f920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:41.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:41.233+0000 7f9bb5e00640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9bb00cbe90 con 0x7f9bb00cb8c0 2024-03-31T15:15:41.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:41.233+0000 7f9bb5e00640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f9bb015fb90 con 0x7f9bb00cb440 2024-03-31T15:15:41.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:41.233+0000 7f9bb5e00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f9bb015f1c0 con 0x7f9bb006dbc0 2024-03-31T15:15:41.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:41.233+0000 7f9ba6000640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f9bb00cb440 0x7f9bb006f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:41.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:41.233+0000 7f9ba6000640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f9bb00cb440 0x7f9bb006f920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59060/0 (socket says 172.21.15.16:59060) 2024-03-31T15:15:41.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:41.233+0000 7f9ba6000640 1 -- 172.21.15.16:0/1604575445 learned_addr learned my addr 172.21.15.16:0/1604575445 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:15:41.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:41.233+0000 7f9ba6000640 1 -- 172.21.15.16:0/1604575445 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f9bb006dbc0 msgr2=0x7f9bb015f650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:15:41.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:41.233+0000 7f9baea00640 1 --2- 172.21.15.16:0/1604575445 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f9bb006dbc0 0x7f9bb015f650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:41.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:41.233+0000 7f9ba6000640 1 --2- 172.21.15.16:0/1604575445 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f9bb006dbc0 0x7f9bb015f650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:41.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:41.233+0000 7f9ba6000640 1 -- 172.21.15.16:0/1604575445 >> v1:172.21.15.89:6789/0 conn(0x7f9bb00cb8c0 legacy=0x7f9bb0162180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:15:41.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:41.233+0000 7f9ba6000640 1 -- 172.21.15.16:0/1604575445 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9bb00cc170 con 0x7f9bb00cb440 2024-03-31T15:15:41.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:41.233+0000 7f9baea00640 1 --2- 172.21.15.16:0/1604575445 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f9bb006dbc0 0x7f9bb015f650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:15:41.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:41.237+0000 7f9ba6000640 1 --2- 172.21.15.16:0/1604575445 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9bb00cb440 0x7f9bb006f920 secure :-1 s=READY pgs=1935 cs=0 l=1 rev1=1 crypto rx=0x7f9b98009870 tx=0x7f9b9801c460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d5f93a39e2d9eebd server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:15:41.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:41.237+0000 7f9bae000640 1 -- 172.21.15.16:0/1604575445 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b980b0070 con 0x7f9bb00cb440 2024-03-31T15:15:41.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:41.237+0000 7f9bae000640 1 -- 172.21.15.16:0/1604575445 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9b980ab600 con 0x7f9bb00cb440 2024-03-31T15:15:41.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:41.237+0000 7f9bae000640 1 -- 172.21.15.16:0/1604575445 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b980ab9d0 con 0x7f9bb00cb440 2024-03-31T15:15:41.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:41.237+0000 7f9bb5e00640 1 -- 172.21.15.16:0/1604575445 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9bb00cb440 msgr2=0x7f9bb006f920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:41.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:41.237+0000 7f9bb5e00640 1 --2- 172.21.15.16:0/1604575445 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9bb00cb440 0x7f9bb006f920 secure :-1 s=READY pgs=1935 cs=0 l=1 rev1=1 crypto rx=0x7f9b98009870 tx=0x7f9b9801c460 comp rx=0 tx=0).stop 2024-03-31T15:15:41.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:41.237+0000 7f9bb5e00640 1 -- 172.21.15.16:0/1604575445 shutdown_connections 2024-03-31T15:15:41.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:41.237+0000 7f9bb5e00640 1 --2- 172.21.15.16:0/1604575445 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9bb00cb440 0x7f9bb006f920 unknown :-1 s=CLOSED pgs=1935 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:41.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:41.237+0000 7f9bb5e00640 1 --2- 172.21.15.16:0/1604575445 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f9bb006dbc0 0x7f9bb015f650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:41.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:41.237+0000 7f9bb5e00640 1 -- 172.21.15.16:0/1604575445 >> 172.21.15.16:0/1604575445 conn(0x7f9bb0067330 msgr2=0x7f9bb0067740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:15:41.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:41.237+0000 7f9bb5e00640 1 -- 172.21.15.16:0/1604575445 shutdown_connections 2024-03-31T15:15:41.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:41.237+0000 7f9bb5e00640 1 -- 172.21.15.16:0/1604575445 wait complete. 2024-03-31T15:15:41.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:41.241+0000 7f9bb5e00640 1 Processor -- start 2024-03-31T15:15:41.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:41.241+0000 7f9bb5e00640 1 -- start start 2024-03-31T15:15:41.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:41.241+0000 7f9bb5e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9bb006dbc0 0x7f9bb0140720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:41.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:41.241+0000 7f9bb5e00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f9bb00cb440 0x7f9bb00b72f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:41.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:41.241+0000 7f9bb5e00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9bb00cb8c0 0x7f9bb00cc6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:41.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:41.241+0000 7f9bb5e00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f9bb006c950 con 0x7f9bb006dbc0 2024-03-31T15:15:41.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:41.241+0000 7f9bb5e00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f9bb0162ca0 con 0x7f9bb00cb8c0 2024-03-31T15:15:41.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:41.241+0000 7f9bb5e00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f9bb006cad0 con 0x7f9bb00cb440 2024-03-31T15:15:41.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:41.241+0000 7f9baea00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9bb006dbc0 0x7f9bb0140720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:41.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:41.241+0000 7f9baea00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9bb006dbc0 0x7f9bb0140720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59062/0 (socket says 172.21.15.16:59062) 2024-03-31T15:15:41.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:41.241+0000 7f9baea00640 1 -- 172.21.15.16:0/411484638 learned_addr learned my addr 172.21.15.16:0/411484638 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:15:41.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:41.241+0000 7f9ba6000640 1 --2- 172.21.15.16:0/411484638 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f9bb00cb440 0x7f9bb00b72f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:41.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:41.241+0000 7f9baf400640 1 --2- 172.21.15.16:0/411484638 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9bb00cb8c0 0x7f9bb00cc6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:41.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:41.241+0000 7f9baea00640 1 -- 172.21.15.16:0/411484638 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f9bb00cb440 msgr2=0x7f9bb00b72f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:41.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:41.241+0000 7f9baea00640 1 --2- 172.21.15.16:0/411484638 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f9bb00cb440 0x7f9bb00b72f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:41.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:41.241+0000 7f9baea00640 1 -- 172.21.15.16:0/411484638 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9bb00cb8c0 msgr2=0x7f9bb00cc6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:41.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:41.241+0000 7f9baea00640 1 --2- 172.21.15.16:0/411484638 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9bb00cb8c0 0x7f9bb00cc6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:41.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:41.241+0000 7f9baea00640 1 -- 172.21.15.16:0/411484638 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9bb0140d40 con 0x7f9bb006dbc0 2024-03-31T15:15:41.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:41.241+0000 7f9baf400640 1 --2- 172.21.15.16:0/411484638 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9bb00cb8c0 0x7f9bb00cc6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:15:41.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:41.241+0000 7f9baea00640 1 --2- 172.21.15.16:0/411484638 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9bb006dbc0 0x7f9bb0140720 secure :-1 s=READY pgs=1936 cs=0 l=1 rev1=1 crypto rx=0x7f9ba000ce00 tx=0x7f9ba000a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:15:41.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:41.241+0000 7f9ba7e00640 1 -- 172.21.15.16:0/411484638 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ba000f680 con 0x7f9bb006dbc0 2024-03-31T15:15:41.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:41.241+0000 7f9ba7e00640 1 -- 172.21.15.16:0/411484638 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9ba000f820 con 0x7f9bb006dbc0 2024-03-31T15:15:41.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:41.241+0000 7f9ba7e00640 1 -- 172.21.15.16:0/411484638 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ba000fc30 con 0x7f9bb006dbc0 2024-03-31T15:15:41.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:41.241+0000 7f9bb5e00640 1 -- 172.21.15.16:0/411484638 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9bb00b58a0 con 0x7f9bb006dbc0 2024-03-31T15:15:41.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:41.241+0000 7f9bb5e00640 1 -- 172.21.15.16:0/411484638 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9bb00b5e60 con 0x7f9bb006dbc0 2024-03-31T15:15:41.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:41.245+0000 7f9bb5e00640 1 -- 172.21.15.16:0/411484638 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9bb0069950 con 0x7f9bb006dbc0 2024-03-31T15:15:41.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:41.245+0000 7f9ba7e00640 1 -- 172.21.15.16:0/411484638 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f9ba0018400 con 0x7f9bb006dbc0 2024-03-31T15:15:41.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:41.245+0000 7f9ba7e00640 1 --2- 172.21.15.16:0/411484638 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f9b8c068380 0x7f9b8c06a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:41.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:41.245+0000 7f9ba7e00640 1 -- 172.21.15.16:0/411484638 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(116..116 src has 1..116) ==== 5115+0+0 (secure 0 0 0) 0x7f9ba0087450 con 0x7f9bb006dbc0 2024-03-31T15:15:41.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:41.245+0000 7f9ba6000640 1 --2- 172.21.15.16:0/411484638 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f9b8c068380 0x7f9b8c06a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:41.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:41.245+0000 7f9ba6000640 1 --2- 172.21.15.16:0/411484638 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f9b8c068380 0x7f9b8c06a840 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7f9b980ad000 tx=0x7f9b980a6000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:15:41.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:41.249+0000 7f9ba7e00640 1 -- 172.21.15.16:0/411484638 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f9ba0014030 con 0x7f9bb006dbc0 2024-03-31T15:15:41.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:41.409+0000 7f9bb5e00640 1 -- 172.21.15.16:0/411484638 --> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f9bb00cf380 con 0x7f9b8c068380 2024-03-31T15:15:41.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:41.413+0000 7f9ba7e00640 1 -- 172.21.15.16:0/411484638 <== mgr.4100 v2:172.21.15.89:6832/1549407273 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+13379 (secure 0 0 0) 0x7f9bb00cf380 con 0x7f9b8c068380 2024-03-31T15:15:41.401 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:15:41.402 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-03-31T15:15:41.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:41.413+0000 7f9bb5e00640 1 -- 172.21.15.16:0/411484638 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f9b8c068380 msgr2=0x7f9b8c06a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:41.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:41.413+0000 7f9bb5e00640 1 --2- 172.21.15.16:0/411484638 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f9b8c068380 0x7f9b8c06a840 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7f9b980ad000 tx=0x7f9b980a6000 comp rx=0 tx=0).stop 2024-03-31T15:15:41.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:41.413+0000 7f9bb5e00640 1 -- 172.21.15.16:0/411484638 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9bb006dbc0 msgr2=0x7f9bb0140720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:41.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:41.413+0000 7f9bb5e00640 1 --2- 172.21.15.16:0/411484638 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9bb006dbc0 0x7f9bb0140720 secure :-1 s=READY pgs=1936 cs=0 l=1 rev1=1 crypto rx=0x7f9ba000ce00 tx=0x7f9ba000a650 comp rx=0 tx=0).stop 2024-03-31T15:15:41.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:41.413+0000 7f9bb5e00640 1 -- 172.21.15.16:0/411484638 shutdown_connections 2024-03-31T15:15:41.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:41.413+0000 7f9bb5e00640 1 --2- 172.21.15.16:0/411484638 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f9b8c068380 0x7f9b8c06a840 unknown :-1 s=CLOSED pgs=816 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:41.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:41.413+0000 7f9bb5e00640 1 --2- 172.21.15.16:0/411484638 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9bb00cb8c0 0x7f9bb00cc6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:41.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:41.413+0000 7f9bb5e00640 1 --2- 172.21.15.16:0/411484638 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f9bb00cb440 0x7f9bb00b72f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:41.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:41.413+0000 7f9bb5e00640 1 --2- 172.21.15.16:0/411484638 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9bb006dbc0 0x7f9bb0140720 unknown :-1 s=CLOSED pgs=1936 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:41.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:41.413+0000 7f9bb5e00640 1 -- 172.21.15.16:0/411484638 >> 172.21.15.16:0/411484638 conn(0x7f9bb0067330 msgr2=0x7f9bb00d0890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:15:41.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:41.413+0000 7f9bb5e00640 1 -- 172.21.15.16:0/411484638 shutdown_connections 2024-03-31T15:15:41.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:41.413+0000 7f9bb5e00640 1 -- 172.21.15.16:0/411484638 wait complete. 2024-03-31T15:15:41.422 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":478,"stamp":"2024-03-31T15:15:40.716321+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":266024,"kb_used_data":12424,"kb_used_omap":12,"kb_used_meta":253491,"kb_avail":754708696,"statfs":{"total":773094113280,"available":772821704704,"internally_reserved":0,"allocated":12722176,"data_stored":6194816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":259575344},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003392"},"pg_stats":[{"pgid":"1.0","version":"116'119","reported_seq":360,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-31T15:15:21.040541+0000","last_change":"2024-03-31T15:15:21.040541+0000","last_active":"2024-03-31T15:15:21.040541+0000","last_peered":"2024-03-31T15:15:21.040541+0000","last_clean":"2024-03-31T15:15:21.040541+0000","last_became_active":"2024-03-31T15:03:24.328882+0000","last_became_peered":"2024-03-31T15:03:24.328882+0000","last_unstale":"2024-03-31T15:15:21.040541+0000","last_undegraded":"2024-03-31T15:15:21.040541+0000","last_fullsized":"2024-03-31T15:15:21.040541+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'119","last_scrub_stamp":"2024-03-31T15:15:21.040423+0000","last_deep_scrub":"116'119","last_deep_scrub_stamp":"2024-03-31T15:15:21.040423+0000","last_clean_scrub_stamp":"2024-03-31T15:15:21.040423+0000","objects_scrubbed":2,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:38:28.490306+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509592,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33164,"kb_used_data":1988,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338676,"statfs":{"total":96636764160,"available":96602804224,"internally_reserved":0,"allocated":2035712,"data_stored":1217128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509591,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32588,"kb_used_data":1408,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339252,"statfs":{"total":96636764160,"available":96603394048,"internally_reserved":0,"allocated":1441792,"data_stored":626760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":15,"seq":64424509590,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32588,"kb_used_data":1408,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339252,"statfs":{"total":96636764160,"available":96603394048,"internally_reserved":0,"allocated":1441792,"data_stored":626760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509589,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":36752,"kb_used_data":1408,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335088,"statfs":{"total":96636764160,"available":96599130112,"internally_reserved":0,"allocated":1441792,"data_stored":626760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542295,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32592,"kb_used_data":1408,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339248,"statfs":{"total":96636764160,"available":96603389952,"internally_reserved":0,"allocated":1441792,"data_stored":626760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542295,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32588,"kb_used_data":1408,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339252,"statfs":{"total":96636764160,"available":96603394048,"internally_reserved":0,"allocated":1441792,"data_stored":626760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542295,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32588,"kb_used_data":1408,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339252,"statfs":{"total":96636764160,"available":96603394048,"internally_reserved":0,"allocated":1441792,"data_stored":626760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542295,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33164,"kb_used_data":1988,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338676,"statfs":{"total":96636764160,"available":96602804224,"internally_reserved":0,"allocated":2035712,"data_stored":1217128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-31T15:15:41.423 DEBUG:teuthology.orchestra.run.smithi016:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-03-31T15:15:41.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:41.573+0000 7fbffb400640 1 Processor -- start 2024-03-31T15:15:41.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:41.573+0000 7fbffb400640 1 -- start start 2024-03-31T15:15:41.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:41.573+0000 7fbffb400640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fbff41411b0 0x7fbff406f920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:41.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:41.573+0000 7fbffb400640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbff4144980 con 0x7fbff4141590 2024-03-31T15:15:41.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:41.573+0000 7fbffb400640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbff4161fe0 con 0x7fbff4144340 2024-03-31T15:15:41.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:41.573+0000 7fbffb400640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fbff4162910 con 0x7fbff41411b0 2024-03-31T15:15:41.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:41.573+0000 7fbfebe00640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fbff4141590 0x7fbff4143c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:54770/0 (socket says 172.21.15.16:54770) 2024-03-31T15:15:41.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:41.573+0000 7fbfebe00640 1 -- 172.21.15.16:0/2667942386 learned_addr learned my addr 172.21.15.16:0/2667942386 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:15:41.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:41.573+0000 7fbff8a00640 1 --2- 172.21.15.16:0/2667942386 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fbff41411b0 0x7fbff406f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:41.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:41.573+0000 7fbff3e00640 1 -- 172.21.15.16:0/2667942386 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 407299560 0 0) 0x7fbff4144980 con 0x7fbff4141590 2024-03-31T15:15:41.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:41.573+0000 7fbff3e00640 1 -- 172.21.15.16:0/2667942386 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbfd8003530 con 0x7fbff4141590 2024-03-31T15:15:41.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:41.577+0000 7fbff3e00640 1 -- 172.21.15.16:0/2667942386 <== mon.1 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1378788356 0 0) 0x7fbff4161fe0 con 0x7fbff4144340 2024-03-31T15:15:41.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:41.577+0000 7fbff3e00640 1 -- 172.21.15.16:0/2667942386 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbff4144980 con 0x7fbff4144340 2024-03-31T15:15:41.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:41.577+0000 7fbff3e00640 1 -- 172.21.15.16:0/2667942386 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3506942043 0 0) 0x7fbfd8003530 con 0x7fbff4141590 2024-03-31T15:15:41.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:41.577+0000 7fbff3e00640 1 -- 172.21.15.16:0/2667942386 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fbff41411b0 msgr2=0x7fbff406f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:15:41.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:41.577+0000 7fbff3e00640 1 --2- 172.21.15.16:0/2667942386 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fbff41411b0 0x7fbff406f920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:41.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:41.577+0000 7fbff3e00640 1 -- 172.21.15.16:0/2667942386 >> v1:172.21.15.89:6789/0 conn(0x7fbff4144340 legacy=0x7fbff415e900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:41.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:41.577+0000 7fbff3e00640 1 -- 172.21.15.16:0/2667942386 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbff41622c0 con 0x7fbff4141590 2024-03-31T15:15:41.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:41.577+0000 7fbff3e00640 1 -- 172.21.15.16:0/2667942386 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 214807623 0 0) 0x7fbfdc001600 con 0x7fbff4141590 2024-03-31T15:15:41.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:41.577+0000 7fbff3e00640 1 -- 172.21.15.16:0/2667942386 >> v1:172.21.15.16:6789/0 conn(0x7fbff4141590 legacy=0x7fbff4143c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:41.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:41.577+0000 7fbff3e00640 1 --2- 172.21.15.16:0/2667942386 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbfd8003b80 0x7fbfd8003fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:41.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:41.577+0000 7fbff3e00640 1 --2- 172.21.15.16:0/2667942386 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbfd8004810 0x7fbfd8004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:41.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:41.577+0000 7fbff3e00640 1 --2- 172.21.15.16:0/2667942386 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fbff41411b0 0x7fbfd80051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:41.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:41.577+0000 7fbff3e00640 1 -- 172.21.15.16:0/2667942386 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fbff4162910 con 0x7fbfd8003b80 2024-03-31T15:15:41.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:41.577+0000 7fbff3e00640 1 -- 172.21.15.16:0/2667942386 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fbfd800bab0 con 0x7fbfd8004810 2024-03-31T15:15:41.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:41.577+0000 7fbff3e00640 1 -- 172.21.15.16:0/2667942386 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fbfd800bcb0 con 0x7fbff41411b0 2024-03-31T15:15:41.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:41.577+0000 7fbff8a00640 1 --2- 172.21.15.16:0/2667942386 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbfd8003b80 0x7fbfd8003fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:41.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:41.577+0000 7fbfebe00640 1 --2- 172.21.15.16:0/2667942386 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbfd8004810 0x7fbfd8004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:41.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:41.577+0000 7fbff9400640 1 --2- 172.21.15.16:0/2667942386 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fbff41411b0 0x7fbfd80051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:41.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:41.577+0000 7fbff8a00640 1 -- 172.21.15.16:0/2667942386 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fbff41411b0 msgr2=0x7fbfd80051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:15:41.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:41.577+0000 7fbff8a00640 1 --2- 172.21.15.16:0/2667942386 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fbff41411b0 0x7fbfd80051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:41.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:41.577+0000 7fbff8a00640 1 -- 172.21.15.16:0/2667942386 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbfd8004810 msgr2=0x7fbfd8004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:15:41.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:41.577+0000 7fbff8a00640 1 --2- 172.21.15.16:0/2667942386 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbfd8004810 0x7fbfd8004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:41.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:41.577+0000 7fbff8a00640 1 -- 172.21.15.16:0/2667942386 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbfd8003530 con 0x7fbfd8003b80 2024-03-31T15:15:41.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:41.577+0000 7fbff8a00640 1 --2- 172.21.15.16:0/2667942386 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbfd8003b80 0x7fbfd8003fa0 secure :-1 s=READY pgs=1938 cs=0 l=1 rev1=1 crypto rx=0x7fbfe4009ab0 tx=0x7fbfe401ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7efab0187c953b13 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:15:41.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:41.581+0000 7fbff3e00640 1 -- 172.21.15.16:0/2667942386 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfe40b2070 con 0x7fbfd8003b80 2024-03-31T15:15:41.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:41.581+0000 7fbff3e00640 1 -- 172.21.15.16:0/2667942386 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbfe40a5510 con 0x7fbfd8003b80 2024-03-31T15:15:41.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:41.581+0000 7fbff3e00640 1 -- 172.21.15.16:0/2667942386 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfe40096b0 con 0x7fbfd8003b80 2024-03-31T15:15:41.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:41.581+0000 7fbffb400640 1 -- 172.21.15.16:0/2667942386 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbfd8003b80 msgr2=0x7fbfd8003fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:41.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:41.581+0000 7fbffb400640 1 --2- 172.21.15.16:0/2667942386 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbfd8003b80 0x7fbfd8003fa0 secure :-1 s=READY pgs=1938 cs=0 l=1 rev1=1 crypto rx=0x7fbfe4009ab0 tx=0x7fbfe401ca30 comp rx=0 tx=0).stop 2024-03-31T15:15:41.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:41.581+0000 7fbff9400640 1 -- 172.21.15.16:0/2667942386 reap_dead start 2024-03-31T15:15:41.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:41.581+0000 7fbffb400640 1 -- 172.21.15.16:0/2667942386 shutdown_connections 2024-03-31T15:15:41.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:41.581+0000 7fbffb400640 1 -- 172.21.15.16:0/2667942386 >> 172.21.15.16:0/2667942386 conn(0x7fbff4067930 msgr2=0x7fbff40b3ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:15:41.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:41.581+0000 7fbffb400640 1 -- 172.21.15.16:0/2667942386 shutdown_connections 2024-03-31T15:15:41.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:41.581+0000 7fbffb400640 1 -- 172.21.15.16:0/2667942386 wait complete. 2024-03-31T15:15:41.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:41.581+0000 7fbffb400640 1 Processor -- start 2024-03-31T15:15:41.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:41.581+0000 7fbffb400640 1 -- start start 2024-03-31T15:15:41.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:41.581+0000 7fbffb400640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbfd8003b80 0x7fbff41f85a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:41.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:41.581+0000 7fbffb400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbff413fc40 0x7fbff4160af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:41.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:41.581+0000 7fbffb400640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fbff4161030 0x7fbff40cba30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:41.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:41.581+0000 7fbffb400640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fbff406fd30 con 0x7fbff413fc40 2024-03-31T15:15:41.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:41.581+0000 7fbffb400640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fbff406fbb0 con 0x7fbfd8003b80 2024-03-31T15:15:41.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:41.581+0000 7fbffb400640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7fbff406feb0 con 0x7fbff4161030 2024-03-31T15:15:41.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:41.581+0000 7fbfebe00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbff413fc40 0x7fbff4160af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:41.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:41.581+0000 7fbfebe00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbff413fc40 0x7fbff4160af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59090/0 (socket says 172.21.15.16:59090) 2024-03-31T15:15:41.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:41.581+0000 7fbfebe00640 1 -- 172.21.15.16:0/1792462842 learned_addr learned my addr 172.21.15.16:0/1792462842 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:15:41.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:41.581+0000 7fbff8a00640 1 --2- 172.21.15.16:0/1792462842 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbfd8003b80 0x7fbff41f85a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:41.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:41.581+0000 7fbff9400640 1 --2- 172.21.15.16:0/1792462842 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fbff4161030 0x7fbff40cba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:41.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:41.585+0000 7fbfebe00640 1 -- 172.21.15.16:0/1792462842 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fbff4161030 msgr2=0x7fbff40cba30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:41.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:41.585+0000 7fbfebe00640 1 --2- 172.21.15.16:0/1792462842 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fbff4161030 0x7fbff40cba30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:41.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:41.585+0000 7fbfebe00640 1 -- 172.21.15.16:0/1792462842 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbfd8003b80 msgr2=0x7fbff41f85a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:41.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:41.585+0000 7fbfebe00640 1 --2- 172.21.15.16:0/1792462842 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbfd8003b80 0x7fbff41f85a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:41.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:41.585+0000 7fbfebe00640 1 -- 172.21.15.16:0/1792462842 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbff415a670 con 0x7fbff413fc40 2024-03-31T15:15:41.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:41.585+0000 7fbfebe00640 1 --2- 172.21.15.16:0/1792462842 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbff413fc40 0x7fbff4160af0 secure :-1 s=READY pgs=1939 cs=0 l=1 rev1=1 crypto rx=0x7fbfdc0072f0 tx=0x7fbfdc0077b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:15:41.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:41.585+0000 7fbff1600640 1 -- 172.21.15.16:0/1792462842 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfdc00f8c0 con 0x7fbff413fc40 2024-03-31T15:15:41.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:41.585+0000 7fbff1600640 1 -- 172.21.15.16:0/1792462842 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbfdc018070 con 0x7fbff413fc40 2024-03-31T15:15:41.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:41.585+0000 7fbff1600640 1 -- 172.21.15.16:0/1792462842 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfdc00fcc0 con 0x7fbff413fc40 2024-03-31T15:15:41.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:41.585+0000 7fbffb400640 1 -- 172.21.15.16:0/1792462842 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbff415a950 con 0x7fbff413fc40 2024-03-31T15:15:41.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:41.585+0000 7fbffb400640 1 -- 172.21.15.16:0/1792462842 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbff415af10 con 0x7fbff413fc40 2024-03-31T15:15:41.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:41.585+0000 7fbffb400640 1 -- 172.21.15.16:0/1792462842 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbff415b110 con 0x7fbff413fc40 2024-03-31T15:15:41.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:41.585+0000 7fbff1600640 1 -- 172.21.15.16:0/1792462842 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fbfdc01d460 con 0x7fbff413fc40 2024-03-31T15:15:41.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:41.585+0000 7fbff1600640 1 --2- 172.21.15.16:0/1792462842 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fbfd00683b0 0x7fbfd006a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:41.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:41.585+0000 7fbff1600640 1 -- 172.21.15.16:0/1792462842 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(116..116 src has 1..116) ==== 5115+0+0 (secure 0 0 0) 0x7fbfdc0866b0 con 0x7fbff413fc40 2024-03-31T15:15:41.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:41.585+0000 7fbff8a00640 1 --2- 172.21.15.16:0/1792462842 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fbfd00683b0 0x7fbfd006a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:41.577 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:41.589+0000 7fbff8a00640 1 --2- 172.21.15.16:0/1792462842 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fbfd00683b0 0x7fbfd006a870 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7fbfe40af000 tx=0x7fbfe40046d0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:15:41.580 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:41.593+0000 7fbff1600640 1 -- 172.21.15.16:0/1792462842 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fbff415b110 con 0x7fbff413fc40 2024-03-31T15:15:41.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:41.757+0000 7fbffb400640 1 -- 172.21.15.16:0/1792462842 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command([{prefix=config set, name=mon_health_to_clog}] v 0) -- 0x7fbff4069950 con 0x7fbff413fc40 2024-03-31T15:15:41.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:41.761+0000 7fbff1600640 1 -- 172.21.15.16:0/1792462842 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_health_to_clog}]=0 v24)=0 v24) ==== 123+0+0 (secure 0 0 0) 0x7fbfdc086030 con 0x7fbff413fc40 2024-03-31T15:15:41.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:41.761+0000 7fbff1600640 1 -- 172.21.15.16:0/1792462842 <== mon.0 v2:172.21.15.16:3300/0 8 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7fbfdc00fe60 con 0x7fbff413fc40 2024-03-31T15:15:41.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:41.761+0000 7fbffb400640 1 -- 172.21.15.16:0/1792462842 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fbfd00683b0 msgr2=0x7fbfd006a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:41.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:41.761+0000 7fbffb400640 1 --2- 172.21.15.16:0/1792462842 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fbfd00683b0 0x7fbfd006a870 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7fbfe40af000 tx=0x7fbfe40046d0 comp rx=0 tx=0).stop 2024-03-31T15:15:41.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:41.761+0000 7fbffb400640 1 -- 172.21.15.16:0/1792462842 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbff413fc40 msgr2=0x7fbff4160af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:41.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:41.761+0000 7fbffb400640 1 --2- 172.21.15.16:0/1792462842 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbff413fc40 0x7fbff4160af0 secure :-1 s=READY pgs=1939 cs=0 l=1 rev1=1 crypto rx=0x7fbfdc0072f0 tx=0x7fbfdc0077b0 comp rx=0 tx=0).stop 2024-03-31T15:15:41.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:41.761+0000 7fbffb400640 1 -- 172.21.15.16:0/1792462842 shutdown_connections 2024-03-31T15:15:41.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:41.761+0000 7fbffb400640 1 --2- 172.21.15.16:0/1792462842 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7fbfd00683b0 0x7fbfd006a870 unknown :-1 s=CLOSED pgs=817 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:41.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:41.761+0000 7fbffb400640 1 --2- 172.21.15.16:0/1792462842 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7fbff4161030 0x7fbff40cba30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:41.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:41.761+0000 7fbffb400640 1 --2- 172.21.15.16:0/1792462842 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbff413fc40 0x7fbff4160af0 unknown :-1 s=CLOSED pgs=1939 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:41.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:41.761+0000 7fbffb400640 1 --2- 172.21.15.16:0/1792462842 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbfd8003b80 0x7fbff41f85a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:41.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:41.761+0000 7fbffb400640 1 -- 172.21.15.16:0/1792462842 >> 172.21.15.16:0/1792462842 conn(0x7fbff4067930 msgr2=0x7fbff41587a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:15:41.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:41.761+0000 7fbffb400640 1 -- 172.21.15.16:0/1792462842 shutdown_connections 2024-03-31T15:15:41.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:41.761+0000 7fbffb400640 1 -- 172.21.15.16:0/1792462842 wait complete. 2024-03-31T15:15:41.771 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-31T15:15:41.771 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-31T15:15:41.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:41.925+0000 7f491ec00640 1 Processor -- start 2024-03-31T15:15:41.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:41.925+0000 7f491ec00640 1 -- start start 2024-03-31T15:15:41.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:41.925+0000 7f491ec00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f49180d0c50 0x7f4918072fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:41.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:41.925+0000 7f491ec00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f4918073510 0x7f491815eb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:41.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:41.925+0000 7f491ec00640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f491806bd30 con 0x7f4918147e70 2024-03-31T15:15:41.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:41.925+0000 7f491ec00640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 -- 0x7f49180739a0 con 0x7f4918073510 2024-03-31T15:15:41.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:41.925+0000 7f491ec00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f49181621b0 con 0x7f49180d0c50 2024-03-31T15:15:41.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:41.925+0000 7f4917e00640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f4918147e70 0x7f491806b620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:54782/0 (socket says 172.21.15.16:54782) 2024-03-31T15:15:41.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:41.925+0000 7f4917e00640 1 -- 172.21.15.16:0/1363715171 learned_addr learned my addr 172.21.15.16:0/1363715171 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-31T15:15:41.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:41.925+0000 7f490f400640 1 --2- 172.21.15.16:0/1363715171 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f49180d0c50 0x7f4918072fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:41.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:41.925+0000 7f491cc00640 1 --2- 172.21.15.16:0/1363715171 >> v2:172.21.15.89:3300/0 conn(0x7f4918073510 0x7f491815eb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:41.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:41.925+0000 7f4917400640 1 -- 172.21.15.16:0/1363715171 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3781680067 0 0) 0x7f491806bd30 con 0x7f4918147e70 2024-03-31T15:15:41.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:41.925+0000 7f4917400640 1 -- 172.21.15.16:0/1363715171 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f48fc003530 con 0x7f4918147e70 2024-03-31T15:15:41.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:41.925+0000 7f4917400640 1 -- 172.21.15.16:0/1363715171 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2618304872 0 0) 0x7f48fc003530 con 0x7f4918147e70 2024-03-31T15:15:41.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:41.925+0000 7f4917400640 1 -- 172.21.15.16:0/1363715171 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f49180d0c50 msgr2=0x7f4918072fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:15:41.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:41.925+0000 7f4917400640 1 --2- 172.21.15.16:0/1363715171 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f49180d0c50 0x7f4918072fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:41.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:41.925+0000 7f4917400640 1 -- 172.21.15.16:0/1363715171 >> v2:172.21.15.89:3300/0 conn(0x7f4918073510 msgr2=0x7f491815eb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:15:41.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:41.925+0000 7f4917400640 1 --2- 172.21.15.16:0/1363715171 >> v2:172.21.15.89:3300/0 conn(0x7f4918073510 0x7f491815eb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:41.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:41.925+0000 7f4917400640 1 -- 172.21.15.16:0/1363715171 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4918162490 con 0x7f4918147e70 2024-03-31T15:15:41.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:41.925+0000 7f4917400640 1 -- 172.21.15.16:0/1363715171 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 214807623 0 0) 0x7f4908001680 con 0x7f4918147e70 2024-03-31T15:15:41.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:41.925+0000 7f4917400640 1 -- 172.21.15.16:0/1363715171 >> v1:172.21.15.16:6789/0 conn(0x7f4918147e70 legacy=0x7f491806b620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:41.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:41.925+0000 7f4917400640 1 --2- 172.21.15.16:0/1363715171 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f48fc0039c0 0x7f48fc003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:41.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:41.929+0000 7f4917400640 1 --2- 172.21.15.16:0/1363715171 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f48fc004810 0x7f48fc004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:41.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:41.929+0000 7f4917400640 1 --2- 172.21.15.16:0/1363715171 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f49180d0c50 0x7f48fc0051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:41.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:41.929+0000 7f4917400640 1 -- 172.21.15.16:0/1363715171 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f49180739a0 con 0x7f48fc004810 2024-03-31T15:15:41.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:41.929+0000 7f4917400640 1 -- 172.21.15.16:0/1363715171 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f49181621b0 con 0x7f48fc0039c0 2024-03-31T15:15:41.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:41.929+0000 7f4917400640 1 -- 172.21.15.16:0/1363715171 --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f48fc00b9d0 con 0x7f49180d0c50 2024-03-31T15:15:41.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:41.929+0000 7f490f400640 1 --2- 172.21.15.16:0/1363715171 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f48fc004810 0x7f48fc004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:41.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:41.929+0000 7f491cc00640 1 --2- 172.21.15.16:0/1363715171 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f49180d0c50 0x7f48fc0051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:41.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:41.929+0000 7f4917e00640 1 --2- 172.21.15.16:0/1363715171 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f48fc0039c0 0x7f48fc003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:41.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:41.929+0000 7f490f400640 1 -- 172.21.15.16:0/1363715171 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f49180d0c50 msgr2=0x7f48fc0051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:15:41.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:41.929+0000 7f490f400640 1 --2- 172.21.15.16:0/1363715171 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f49180d0c50 0x7f48fc0051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:41.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:41.929+0000 7f490f400640 1 -- 172.21.15.16:0/1363715171 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f48fc0039c0 msgr2=0x7f48fc003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:15:41.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:41.929+0000 7f490f400640 1 --2- 172.21.15.16:0/1363715171 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f48fc0039c0 0x7f48fc003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:41.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:41.929+0000 7f490f400640 1 -- 172.21.15.16:0/1363715171 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48fc003530 con 0x7f48fc004810 2024-03-31T15:15:41.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:41.929+0000 7f491cc00640 1 --2- 172.21.15.16:0/1363715171 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f49180d0c50 0x7f48fc0051d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:15:41.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:41.929+0000 7f4917e00640 1 --2- 172.21.15.16:0/1363715171 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f48fc0039c0 0x7f48fc003de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:15:41.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:41.929+0000 7f490f400640 1 --2- 172.21.15.16:0/1363715171 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f48fc004810 0x7f48fc004c90 secure :-1 s=READY pgs=1941 cs=0 l=1 rev1=1 crypto rx=0x7f4900009ab0 tx=0x7f490001ca50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5bc7ce3cc64c3960 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:15:41.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:41.929+0000 7f4917400640 1 -- 172.21.15.16:0/1363715171 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49000b2070 con 0x7f48fc004810 2024-03-31T15:15:41.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:41.929+0000 7f4917400640 1 -- 172.21.15.16:0/1363715171 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7f49000a5560 con 0x7f48fc004810 2024-03-31T15:15:41.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:41.929+0000 7f4917400640 1 -- 172.21.15.16:0/1363715171 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49000096b0 con 0x7f48fc004810 2024-03-31T15:15:41.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:41.933+0000 7f491ec00640 1 -- 172.21.15.16:0/1363715171 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f48fc004810 msgr2=0x7f48fc004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:41.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:41.933+0000 7f491ec00640 1 --2- 172.21.15.16:0/1363715171 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f48fc004810 0x7f48fc004c90 secure :-1 s=READY pgs=1941 cs=0 l=1 rev1=1 crypto rx=0x7f4900009ab0 tx=0x7f490001ca50 comp rx=0 tx=0).stop 2024-03-31T15:15:41.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:41.933+0000 7f491cc00640 1 -- 172.21.15.16:0/1363715171 reap_dead start 2024-03-31T15:15:41.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:41.933+0000 7f491ec00640 1 -- 172.21.15.16:0/1363715171 shutdown_connections 2024-03-31T15:15:41.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:41.933+0000 7f491ec00640 1 -- 172.21.15.16:0/1363715171 >> 172.21.15.16:0/1363715171 conn(0x7f49180d1060 msgr2=0x7f49180d1470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:15:41.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:41.933+0000 7f491ec00640 1 -- 172.21.15.16:0/1363715171 shutdown_connections 2024-03-31T15:15:41.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:41.933+0000 7f491ec00640 1 -- 172.21.15.16:0/1363715171 wait complete. 2024-03-31T15:15:41.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:41.933+0000 7f491ec00640 1 Processor -- start 2024-03-31T15:15:41.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:41.933+0000 7f491ec00640 1 -- start start 2024-03-31T15:15:41.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:41.933+0000 7f491ec00640 1 --2- >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f48fc004810 0x7f49180af300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:41.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:41.933+0000 7f491ec00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f491813f9b0 0x7f4918160710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:41.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:41.933+0000 7f491ec00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f49180cdab0 0x7f49180cdf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:41.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:41.933+0000 7f491ec00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f491806bfa0 con 0x7f491813f9b0 2024-03-31T15:15:41.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:41.933+0000 7f491ec00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f4918162a50 con 0x7f49180cdab0 2024-03-31T15:15:41.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:41.933+0000 7f491ec00640 1 -- --> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] -- mon_getmap magic: 0 -- 0x7f4918162f70 con 0x7f48fc004810 2024-03-31T15:15:41.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:41.933+0000 7f490f400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f491813f9b0 0x7f4918160710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:41.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:41.933+0000 7f490f400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f491813f9b0 0x7f4918160710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59106/0 (socket says 172.21.15.16:59106) 2024-03-31T15:15:41.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:41.933+0000 7f490f400640 1 -- 172.21.15.16:0/1930359892 learned_addr learned my addr 172.21.15.16:0/1930359892 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-31T15:15:41.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:41.933+0000 7f491cc00640 1 --2- 172.21.15.16:0/1930359892 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f49180cdab0 0x7f49180cdf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:41.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:41.933+0000 7f4917e00640 1 --2- 172.21.15.16:0/1930359892 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f48fc004810 0x7f49180af300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:41.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:41.933+0000 7f490f400640 1 -- 172.21.15.16:0/1930359892 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f48fc004810 msgr2=0x7f49180af300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:41.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:41.933+0000 7f490f400640 1 --2- 172.21.15.16:0/1930359892 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f48fc004810 0x7f49180af300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:41.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:41.933+0000 7f490f400640 1 -- 172.21.15.16:0/1930359892 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f49180cdab0 msgr2=0x7f49180cdf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:41.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:41.933+0000 7f490f400640 1 --2- 172.21.15.16:0/1930359892 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f49180cdab0 0x7f49180cdf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:41.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:41.933+0000 7f490f400640 1 -- 172.21.15.16:0/1930359892 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49180af8f0 con 0x7f491813f9b0 2024-03-31T15:15:41.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:41.933+0000 7f4917e00640 1 --2- 172.21.15.16:0/1930359892 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f48fc004810 0x7f49180af300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:15:41.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:41.937+0000 7f490f400640 1 --2- 172.21.15.16:0/1930359892 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f491813f9b0 0x7f4918160710 secure :-1 s=READY pgs=1942 cs=0 l=1 rev1=1 crypto rx=0x7f490001c880 tx=0x7f48fc005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:15:41.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:41.937+0000 7f4914c00640 1 -- 172.21.15.16:0/1930359892 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49000bc030 con 0x7f491813f9b0 2024-03-31T15:15:41.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:41.937+0000 7f4914c00640 1 -- 172.21.15.16:0/1930359892 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7f490000f000 con 0x7f491813f9b0 2024-03-31T15:15:41.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:41.937+0000 7f491ec00640 1 -- 172.21.15.16:0/1930359892 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f49180afbd0 con 0x7f491813f9b0 2024-03-31T15:15:41.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:41.937+0000 7f4914c00640 1 -- 172.21.15.16:0/1930359892 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49000b2040 con 0x7f491813f9b0 2024-03-31T15:15:41.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:41.937+0000 7f491ec00640 1 -- 172.21.15.16:0/1930359892 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f49180b0190 con 0x7f491813f9b0 2024-03-31T15:15:41.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:41.937+0000 7f491ec00640 1 -- 172.21.15.16:0/1930359892 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4918069950 con 0x7f491813f9b0 2024-03-31T15:15:41.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:41.937+0000 7f4914c00640 1 -- 172.21.15.16:0/1930359892 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f49000bf070 con 0x7f491813f9b0 2024-03-31T15:15:41.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:41.937+0000 7f4914c00640 1 --2- 172.21.15.16:0/1930359892 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f48f4068380 0x7f48f406a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:15:41.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:41.937+0000 7f4914c00640 1 -- 172.21.15.16:0/1930359892 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(116..116 src has 1..116) ==== 5115+0+0 (secure 0 0 0) 0x7f4900120a10 con 0x7f491813f9b0 2024-03-31T15:15:41.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:41.937+0000 7f4917e00640 1 --2- 172.21.15.16:0/1930359892 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f48f4068380 0x7f48f406a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:15:41.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:41.941+0000 7f4917e00640 1 --2- 172.21.15.16:0/1930359892 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f48f4068380 0x7f48f406a840 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7f4908001ec0 tx=0x7f49080047f0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:15:41.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:41.941+0000 7f4914c00640 1 -- 172.21.15.16:0/1930359892 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f4900016760 con 0x7f491813f9b0 2024-03-31T15:15:42.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:42.153+0000 7f491ec00640 1 -- 172.21.15.16:0/1930359892 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f49180d1bf0 con 0x7f491813f9b0 2024-03-31T15:15:42.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:42.153+0000 7f4914c00640 1 -- 172.21.15.16:0/1930359892 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 114 v114) ==== 95+0+3865 (secure 0 0 0) 0x7f49000f10f0 con 0x7f491813f9b0 2024-03-31T15:15:42.143 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-31T15:15:42.143 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":114,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7631,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6836/1628897112","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":1628897112},{"type":"v1","addr":"172.21.15.89:6836","nonce":1628897112}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":94},{"gid":8746,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/1036724837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6835","nonce":1036724837},{"type":"v1","addr":"172.21.15.89:6837","nonce":1036724837}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":94},{"gid":9192,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6834/472139147","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":472139147},{"type":"v1","addr":"172.21.15.16:6834","nonce":472139147}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":94},{"gid":9978,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/683439677","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6833","nonce":683439677},{"type":"v1","addr":"172.21.15.16:6835","nonce":683439677}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":114}],"filesystems":[]} 2024-03-31T15:15:42.143 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 114 2024-03-31T15:15:42.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:42.157+0000 7f491ec00640 1 -- 172.21.15.16:0/1930359892 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f48f4068380 msgr2=0x7f48f406a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:42.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:42.157+0000 7f491ec00640 1 --2- 172.21.15.16:0/1930359892 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f48f4068380 0x7f48f406a840 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7f4908001ec0 tx=0x7f49080047f0 comp rx=0 tx=0).stop 2024-03-31T15:15:42.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:42.157+0000 7f491ec00640 1 -- 172.21.15.16:0/1930359892 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f491813f9b0 msgr2=0x7f4918160710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:15:42.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:42.157+0000 7f491ec00640 1 --2- 172.21.15.16:0/1930359892 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f491813f9b0 0x7f4918160710 secure :-1 s=READY pgs=1942 cs=0 l=1 rev1=1 crypto rx=0x7f490001c880 tx=0x7f48fc005840 comp rx=0 tx=0).stop 2024-03-31T15:15:42.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:42.157+0000 7f491ec00640 1 -- 172.21.15.16:0/1930359892 shutdown_connections 2024-03-31T15:15:42.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:42.157+0000 7f491ec00640 1 --2- 172.21.15.16:0/1930359892 >> [v2:172.21.15.89:6832/1549407273,v1:172.21.15.89:6833/1549407273] conn(0x7f48f4068380 0x7f48f406a840 unknown :-1 s=CLOSED pgs=818 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:42.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:42.157+0000 7f491ec00640 1 --2- 172.21.15.16:0/1930359892 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f49180cdab0 0x7f49180cdf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:42.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:42.157+0000 7f491ec00640 1 --2- 172.21.15.16:0/1930359892 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f491813f9b0 0x7f4918160710 unknown :-1 s=CLOSED pgs=1942 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:42.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:42.157+0000 7f491ec00640 1 --2- 172.21.15.16:0/1930359892 >> [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] conn(0x7f48fc004810 0x7f49180af300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:15:42.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:42.157+0000 7f491ec00640 1 -- 172.21.15.16:0/1930359892 >> 172.21.15.16:0/1930359892 conn(0x7f49180d1060 msgr2=0x7f49180b5f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:15:42.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:42.157+0000 7f491ec00640 1 -- 172.21.15.16:0/1930359892 shutdown_connections 2024-03-31T15:15:42.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-31T15:15:42.157+0000 7f491ec00640 1 -- 172.21.15.16:0/1930359892 wait complete. 2024-03-31T15:15:42.165 DEBUG:tasks.cephfs.filesystem:already dead... 2024-03-31T15:15:42.165 INFO:teuthology.misc:Shutting down mds daemons... 2024-03-31T15:15:42.165 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-31T15:15:42.165 INFO:teuthology.orchestra.run:waiting for 300 2024-03-31T15:15:42.177 INFO:tasks.ceph.mds.a:Stopped 2024-03-31T15:15:42.177 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-31T15:15:42.177 INFO:teuthology.orchestra.run:waiting for 300 2024-03-31T15:15:42.233 INFO:tasks.ceph.mds.c:Stopped 2024-03-31T15:15:42.233 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-31T15:15:42.234 INFO:teuthology.orchestra.run:waiting for 300 2024-03-31T15:15:42.247 INFO:tasks.ceph.mds.b:Stopped 2024-03-31T15:15:42.248 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-31T15:15:42.248 INFO:teuthology.orchestra.run:waiting for 300 2024-03-31T15:15:42.309 INFO:tasks.ceph.mds.d:Stopped 2024-03-31T15:15:42.309 INFO:teuthology.misc:Shutting down osd daemons... 2024-03-31T15:15:42.309 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-03-31T15:15:42.309 INFO:teuthology.orchestra.run:waiting for 300 2024-03-31T15:15:42.396 INFO:tasks.ceph.osd.0:Stopped 2024-03-31T15:15:42.396 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-03-31T15:15:42.396 INFO:teuthology.orchestra.run:waiting for 300 2024-03-31T15:15:42.484 INFO:tasks.ceph.osd.1:Stopped 2024-03-31T15:15:42.485 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-03-31T15:15:42.485 INFO:teuthology.orchestra.run:waiting for 300 2024-03-31T15:15:42.584 INFO:tasks.ceph.osd.2:Stopped 2024-03-31T15:15:42.585 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-03-31T15:15:42.585 INFO:teuthology.orchestra.run:waiting for 300 2024-03-31T15:15:42.681 INFO:tasks.ceph.osd.3:Stopped 2024-03-31T15:15:42.681 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-03-31T15:15:42.681 INFO:teuthology.orchestra.run:waiting for 300 2024-03-31T15:15:42.751 INFO:tasks.ceph.osd.4:Stopped 2024-03-31T15:15:42.751 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-03-31T15:15:42.751 INFO:teuthology.orchestra.run:waiting for 300 2024-03-31T15:15:42.838 INFO:tasks.ceph.osd.5:Stopped 2024-03-31T15:15:42.838 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-03-31T15:15:42.838 INFO:teuthology.orchestra.run:waiting for 300 2024-03-31T15:15:42.920 INFO:tasks.ceph.osd.6:Stopped 2024-03-31T15:15:42.921 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-03-31T15:15:42.921 INFO:teuthology.orchestra.run:waiting for 300 2024-03-31T15:15:43.014 INFO:tasks.ceph.osd.7:Stopped 2024-03-31T15:15:43.014 INFO:teuthology.misc:Shutting down mgr daemons... 2024-03-31T15:15:43.014 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-03-31T15:15:43.014 INFO:teuthology.orchestra.run:waiting for 300 2024-03-31T15:15:43.068 INFO:tasks.ceph.mgr.y:Stopped 2024-03-31T15:15:43.068 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-03-31T15:15:43.068 INFO:teuthology.orchestra.run:waiting for 300 2024-03-31T15:15:43.125 INFO:tasks.ceph.mgr.x:Stopped 2024-03-31T15:15:43.125 INFO:teuthology.misc:Shutting down mon daemons... 2024-03-31T15:15:43.125 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-03-31T15:15:43.126 INFO:teuthology.orchestra.run:waiting for 300 2024-03-31T15:15:43.144 INFO:tasks.ceph.mon.a:Stopped 2024-03-31T15:15:43.144 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-03-31T15:15:43.144 INFO:teuthology.orchestra.run:waiting for 300 2024-03-31T15:15:43.160 INFO:tasks.ceph.mon.b:Stopped 2024-03-31T15:15:43.161 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-03-31T15:15:43.161 INFO:teuthology.orchestra.run:waiting for 300 2024-03-31T15:15:43.225 INFO:tasks.ceph.mon.c:Stopped 2024-03-31T15:15:43.225 INFO:tasks.ceph:Checking cluster log for badness... 2024-03-31T15:15:43.225 DEBUG:teuthology.orchestra.run.smithi016:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v 'filesystem is online with fewer MDS than max_mds' | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v PG_AVAILABILITY | egrep -v PG_DEGRADED | egrep -v 'Reduced data availability' | egrep -v 'inode wrongly marked free' | egrep -v 'bad backtrace on inode' | egrep -v 'inode table repaired for inode' | egrep -v 'Scrub error on inode' | egrep -v 'Scrub error on dir' | egrep -v 'Metadata damage detected' | head -n 1 2024-03-31T15:15:43.261 INFO:teuthology.orchestra.run.smithi016.stdout:2024-03-31T15:08:17.426356+0000 mds.a (mds.0) 1 : cluster [WRN] Corrupt dentry 'file_to_be_damaged' in dir frag 0x10000000000: Invalid tag char 'd' pos 0: Malformed input [buffer:3](/subdir) 2024-03-31T15:15:43.262 WARNING:tasks.ceph:Found errors (ERR|WRN|SEC) in cluster log 2024-03-31T15:15:43.262 DEBUG:teuthology.orchestra.run.smithi016:> sudo egrep '\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v 'filesystem is online with fewer MDS than max_mds' | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v PG_AVAILABILITY | egrep -v PG_DEGRADED | egrep -v 'Reduced data availability' | egrep -v 'inode wrongly marked free' | egrep -v 'bad backtrace on inode' | egrep -v 'inode table repaired for inode' | egrep -v 'Scrub error on inode' | egrep -v 'Scrub error on dir' | egrep -v 'Metadata damage detected' | head -n 1 2024-03-31T15:15:43.337 DEBUG:teuthology.orchestra.run.smithi016:> sudo egrep '\[ERR\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v 'filesystem is online with fewer MDS than max_mds' | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v PG_AVAILABILITY | egrep -v PG_DEGRADED | egrep -v 'Reduced data availability' | egrep -v 'inode wrongly marked free' | egrep -v 'bad backtrace on inode' | egrep -v 'inode table repaired for inode' | egrep -v 'Scrub error on inode' | egrep -v 'Scrub error on dir' | egrep -v 'Metadata damage detected' | head -n 1 2024-03-31T15:15:43.413 INFO:teuthology.orchestra.run.smithi016.stdout:2024-03-31T15:10:11.682944+0000 mds.c (mds.0) 1 : cluster [ERR] dir 0x10000000000 object missing on disk; some files may be lost (/dir) 2024-03-31T15:15:43.414 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi016.front.sepia.ceph.com 2024-03-31T15:15:43.414 DEBUG:teuthology.orchestra.run.smithi016:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-03-31T15:15:43.726 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi016.front.sepia.ceph.com 2024-03-31T15:15:43.726 DEBUG:teuthology.orchestra.run.smithi016:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-03-31T15:15:43.964 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi016.front.sepia.ceph.com 2024-03-31T15:15:43.964 DEBUG:teuthology.orchestra.run.smithi016:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-03-31T15:15:44.130 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi016.front.sepia.ceph.com 2024-03-31T15:15:44.130 DEBUG:teuthology.orchestra.run.smithi016:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-03-31T15:15:44.297 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi089.front.sepia.ceph.com 2024-03-31T15:15:44.297 DEBUG:teuthology.orchestra.run.smithi089:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-03-31T15:15:44.588 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi089.front.sepia.ceph.com 2024-03-31T15:15:44.588 DEBUG:teuthology.orchestra.run.smithi089:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-03-31T15:15:44.802 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi089.front.sepia.ceph.com 2024-03-31T15:15:44.802 DEBUG:teuthology.orchestra.run.smithi089:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-03-31T15:15:45.010 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi089.front.sepia.ceph.com 2024-03-31T15:15:45.010 DEBUG:teuthology.orchestra.run.smithi089:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-03-31T15:15:45.191 INFO:tasks.ceph:Archiving mon data... 2024-03-31T15:15:45.192 DEBUG:teuthology.misc:Transferring archived files from smithi016:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/vshankar-2024-03-30_23:12:08-fs-wip-vshankar-testing-20240330.170739-testing-default-smithi/7631609/data/mon.a.tgz 2024-03-31T15:15:45.192 DEBUG:teuthology.orchestra.run.smithi016:> mktemp 2024-03-31T15:15:45.201 INFO:teuthology.orchestra.run.smithi016.stdout:/tmp/tmp.F6OKywqpVO 2024-03-31T15:15:45.201 DEBUG:teuthology.orchestra.run.smithi016:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.F6OKywqpVO 2024-03-31T15:15:45.351 DEBUG:teuthology.orchestra.run.smithi016:> sudo chmod 0666 /tmp/tmp.F6OKywqpVO 2024-03-31T15:15:45.425 DEBUG:teuthology.orchestra.remote:smithi016:/tmp/tmp.F6OKywqpVO is 426KB 2024-03-31T15:15:45.491 DEBUG:teuthology.orchestra.run.smithi016:> rm -fr /tmp/tmp.F6OKywqpVO 2024-03-31T15:15:45.499 DEBUG:teuthology.misc:Transferring archived files from smithi089:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/vshankar-2024-03-30_23:12:08-fs-wip-vshankar-testing-20240330.170739-testing-default-smithi/7631609/data/mon.b.tgz 2024-03-31T15:15:45.499 DEBUG:teuthology.orchestra.run.smithi089:> mktemp 2024-03-31T15:15:45.508 INFO:teuthology.orchestra.run.smithi089.stdout:/tmp/tmp.xC3k50gsWs 2024-03-31T15:15:45.509 DEBUG:teuthology.orchestra.run.smithi089:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.xC3k50gsWs 2024-03-31T15:15:45.656 DEBUG:teuthology.orchestra.run.smithi089:> sudo chmod 0666 /tmp/tmp.xC3k50gsWs 2024-03-31T15:15:45.732 DEBUG:teuthology.orchestra.remote:smithi089:/tmp/tmp.xC3k50gsWs is 435KB 2024-03-31T15:15:45.798 DEBUG:teuthology.orchestra.run.smithi089:> rm -fr /tmp/tmp.xC3k50gsWs 2024-03-31T15:15:45.806 DEBUG:teuthology.misc:Transferring archived files from smithi089:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/vshankar-2024-03-30_23:12:08-fs-wip-vshankar-testing-20240330.170739-testing-default-smithi/7631609/data/mon.c.tgz 2024-03-31T15:15:45.807 DEBUG:teuthology.orchestra.run.smithi089:> mktemp 2024-03-31T15:15:45.855 INFO:teuthology.orchestra.run.smithi089.stdout:/tmp/tmp.Bn9l2F6Nub 2024-03-31T15:15:45.855 DEBUG:teuthology.orchestra.run.smithi089:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.Bn9l2F6Nub 2024-03-31T15:15:45.998 DEBUG:teuthology.orchestra.run.smithi089:> sudo chmod 0666 /tmp/tmp.Bn9l2F6Nub 2024-03-31T15:15:46.072 DEBUG:teuthology.orchestra.remote:smithi089:/tmp/tmp.Bn9l2F6Nub is 435KB 2024-03-31T15:15:46.134 DEBUG:teuthology.orchestra.run.smithi089:> rm -fr /tmp/tmp.Bn9l2F6Nub 2024-03-31T15:15:46.142 INFO:tasks.ceph:Cleaning ceph cluster... 2024-03-31T15:15:46.142 DEBUG:teuthology.orchestra.run.smithi016:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-31T15:15:46.145 DEBUG:teuthology.orchestra.run.smithi089:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-31T15:15:46.216 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-03-31T15:15:46.251 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-03-31T15:15:46.251 INFO:tasks.ceph:Archiving crash dumps... 2024-03-31T15:15:46.253 DEBUG:teuthology.misc:Transferring archived files from smithi016:/var/lib/ceph/crash to /home/teuthworker/archive/vshankar-2024-03-30_23:12:08-fs-wip-vshankar-testing-20240330.170739-testing-default-smithi/7631609/remote/smithi016/crash 2024-03-31T15:15:46.254 DEBUG:teuthology.orchestra.run.smithi016:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-31T15:15:46.281 DEBUG:teuthology.misc:Transferring archived files from smithi089:/var/lib/ceph/crash to /home/teuthworker/archive/vshankar-2024-03-30_23:12:08-fs-wip-vshankar-testing-20240330.170739-testing-default-smithi/7631609/remote/smithi089/crash 2024-03-31T15:15:46.282 DEBUG:teuthology.orchestra.run.smithi089:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-31T15:15:46.309 INFO:tasks.ceph:Shutting down logrotate 2024-03-31T15:15:46.309 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-31T15:15:46.327 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-31T15:15:46.390 DEBUG:teuthology.orchestra.run.smithi016:> sudo rm /etc/logrotate.d/ceph-test.conf 2024-03-31T15:15:46.411 DEBUG:teuthology.orchestra.run.smithi089:> sudo rm /etc/logrotate.d/ceph-test.conf 2024-03-31T15:15:46.450 INFO:tasks.ceph:Compressing logs... 2024-03-31T15:15:46.450 DEBUG:teuthology.orchestra.run.smithi016:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-31T15:15:46.459 DEBUG:teuthology.orchestra.run.smithi089:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-31T15:15:46.487 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47737.log 2024-03-31T15:15:46.487 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48900.log 2024-03-31T15:15:46.488 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.47737.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45243.log 2024-03-31T15:15:46.489 INFO:teuthology.orchestra.run.smithi016.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.47737.log.gz 2024-03-31T15:15:46.489 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.48900.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40358.log 2024-03-31T15:15:46.490 INFO:teuthology.orchestra.run.smithi016.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.48900.log.gz 2024-03-31T15:15:46.490 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.45243.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42859.log 2024-03-31T15:15:46.491 INFO:teuthology.orchestra.run.smithi016.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.45243.log.gz 2024-03-31T15:15:46.491 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.40358.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32128.log 2024-03-31T15:15:46.491 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.42859.log: 91.5% -- replaced with /var/log/ceph/ceph-client.admin.40358.log.gz 2024-03-31T15:15:46.491 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52977.log 2024-03-31T15:15:46.493 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.32128.log: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.42859.log.gz 2024-03-31T15:15:46.493 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39971.log 2024-03-31T15:15:46.493 INFO:teuthology.orchestra.run.smithi016.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.32128.log.gz 2024-03-31T15:15:46.494 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.52977.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35441.log 2024-03-31T15:15:46.494 INFO:teuthology.orchestra.run.smithi016.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.52977.log.gz 2024-03-31T15:15:46.507 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.39971.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37379.log 2024-03-31T15:15:46.507 INFO:teuthology.orchestra.run.smithi016.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.39971.log.gz 2024-03-31T15:15:46.508 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.35441.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47029.log 2024-03-31T15:15:46.508 INFO:teuthology.orchestra.run.smithi016.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.35441.log.gz 2024-03-31T15:15:46.508 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.37379.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48383.log 2024-03-31T15:15:46.508 INFO:teuthology.orchestra.run.smithi016.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.37379.log.gz 2024-03-31T15:15:46.508 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.47029.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41417.log 2024-03-31T15:15:46.509 INFO:teuthology.orchestra.run.smithi016.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.47029.log.gz 2024-03-31T15:15:46.509 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.48383.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32751.log 2024-03-31T15:15:46.509 INFO:teuthology.orchestra.run.smithi016.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.48383.log.gz 2024-03-31T15:15:46.509 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.41417.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46925.log 2024-03-31T15:15:46.509 INFO:teuthology.orchestra.run.smithi016.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.41417.log.gz 2024-03-31T15:15:46.509 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.32751.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32314.log 2024-03-31T15:15:46.509 INFO:teuthology.orchestra.run.smithi016.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.32751.log.gz 2024-03-31T15:15:46.510 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.46925.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38633.log 2024-03-31T15:15:46.510 INFO:teuthology.orchestra.run.smithi016.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.46925.log.gz 2024-03-31T15:15:46.510 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.32314.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40466.log 2024-03-31T15:15:46.510 INFO:teuthology.orchestra.run.smithi016.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.32314.log.gz 2024-03-31T15:15:46.510 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.38633.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32114.log 2024-03-31T15:15:46.510 INFO:teuthology.orchestra.run.smithi016.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.38633.log.gz 2024-03-31T15:15:46.510 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.40466.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53863.log 2024-03-31T15:15:46.510 INFO:teuthology.orchestra.run.smithi016.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.40466.log.gz 2024-03-31T15:15:46.510 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.32114.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38990.log 2024-03-31T15:15:46.510 INFO:teuthology.orchestra.run.smithi016.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.32114.log.gz 2024-03-31T15:15:46.510 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.53863.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49805.log 2024-03-31T15:15:46.511 INFO:teuthology.orchestra.run.smithi016.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.53863.log.gz 2024-03-31T15:15:46.511 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.38990.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54279.log 2024-03-31T15:15:46.511 INFO:teuthology.orchestra.run.smithi016.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.38990.log.gz 2024-03-31T15:15:46.511 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.49805.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36996.log 2024-03-31T15:15:46.511 INFO:teuthology.orchestra.run.smithi016.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.49805.log.gz 2024-03-31T15:15:46.511 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.54279.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43479.log 2024-03-31T15:15:46.511 INFO:teuthology.orchestra.run.smithi016.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.54279.log.gz 2024-03-31T15:15:46.511 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.36996.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35641.log 2024-03-31T15:15:46.511 INFO:teuthology.orchestra.run.smithi016.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.36996.log.gz 2024-03-31T15:15:46.511 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.43479.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2024-03-31T15:15:46.514 INFO:teuthology.orchestra.run.smithi016.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.43479.log.gz 2024-03-31T15:15:46.515 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.35641.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53941.log 2024-03-31T15:15:46.515 INFO:teuthology.orchestra.run.smithi016.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.35641.log.gz 2024-03-31T15:15:46.515 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53837.log 2024-03-31T15:15:46.515 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-mgr.y.log: /var/log/ceph/ceph-client.admin.53941.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40229.log 2024-03-31T15:15:46.515 INFO:teuthology.orchestra.run.smithi016.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.53941.log.gz 2024-03-31T15:15:46.515 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.53837.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53458.log 2024-03-31T15:15:46.515 INFO:teuthology.orchestra.run.smithi016.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.53837.log.gz 2024-03-31T15:15:46.515 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.40229.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51785.log 2024-03-31T15:15:46.515 INFO:teuthology.orchestra.run.smithi016.stderr: 91.4% -- replaced with /var/log/ceph/ceph-client.admin.40229.log.gz 2024-03-31T15:15:46.515 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.53458.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43401.log 2024-03-31T15:15:46.515 INFO:teuthology.orchestra.run.smithi016.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.53458.log.gz 2024-03-31T15:15:46.515 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.51785.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35615.log 2024-03-31T15:15:46.515 INFO:teuthology.orchestra.run.smithi016.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.51785.log.gz 2024-03-31T15:15:46.515 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.43401.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43748.log 2024-03-31T15:15:46.516 INFO:teuthology.orchestra.run.smithi016.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.43401.log.gz 2024-03-31T15:15:46.520 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.35615.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38410.log 2024-03-31T15:15:46.520 INFO:teuthology.orchestra.run.smithi016.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.35615.log.gz 2024-03-31T15:15:46.520 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.43748.log: gzip -5 94.6% --verbose -- -- replaced with /var/log/ceph/ceph-mgr.y.log.gz /var/log/ceph/ceph-client.admin.51568.log 2024-03-31T15:15:46.520 INFO:teuthology.orchestra.run.smithi016.stderr: 2024-03-31T15:15:46.520 INFO:teuthology.orchestra.run.smithi016.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.43748.log.gz 2024-03-31T15:15:46.521 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.38410.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53668.log 2024-03-31T15:15:46.521 INFO:teuthology.orchestra.run.smithi016.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.38410.log.gz 2024-03-31T15:15:46.521 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.51568.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44087.log 2024-03-31T15:15:46.521 INFO:teuthology.orchestra.run.smithi016.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.51568.log.gz 2024-03-31T15:15:46.521 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.53668.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49630.log 2024-03-31T15:15:46.521 INFO:teuthology.orchestra.run.smithi016.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.53668.log.gz 2024-03-31T15:15:46.521 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.44087.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48330.log 2024-03-31T15:15:46.521 INFO:teuthology.orchestra.run.smithi016.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.44087.log.gz 2024-03-31T15:15:46.521 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.49630.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.42435.log 2024-03-31T15:15:46.523 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27807.log 2024-03-31T15:15:46.523 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27783.log 2024-03-31T15:15:46.523 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.27807.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2024-03-31T15:15:46.524 INFO:teuthology.orchestra.run.smithi089.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.27807.log.gz 2024-03-31T15:15:46.524 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.27783.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2024-03-31T15:15:46.524 INFO:teuthology.orchestra.run.smithi089.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.27783.log.gz 2024-03-31T15:15:46.524 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-03-31T15:15:46.524 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27711.log 2024-03-31T15:15:46.524 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27639.log 2024-03-31T15:15:46.524 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.27711.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.27711.log.gz 2024-03-31T15:15:46.524 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27759.log 2024-03-31T15:15:46.524 INFO:teuthology.orchestra.run.smithi089.stderr:gzip/var/log/ceph/ceph-client.admin.27639.log: -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-03-31T15:15:46.524 INFO:teuthology.orchestra.run.smithi089.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.27639.log.gz 2024-03-31T15:15:46.524 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.27759.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-03-31T15:15:46.524 INFO:teuthology.orchestra.run.smithi089.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.27759.log.gz 2024-03-31T15:15:46.524 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27663.log 2024-03-31T15:15:46.524 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.1.32929.log 2024-03-31T15:15:46.527 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.48330.log: 91.3% -- replaced with /var/log/ceph/ceph-client.admin.49630.log.gz 2024-03-31T15:15:46.527 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47063.log 2024-03-31T15:15:46.527 INFO:teuthology.orchestra.run.smithi016.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.48330.log.gz 2024-03-31T15:15:46.528 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.0.42435.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32537.log 2024-03-31T15:15:46.528 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.47063.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49828.log 2024-03-31T15:15:46.528 INFO:teuthology.orchestra.run.smithi016.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.47063.log.gz 2024-03-31T15:15:46.528 INFO:teuthology.orchestra.run.smithi016.stderr: 91.9% -- replaced with /var/log/ceph/ceph-client.0.42435.log.gz 2024-03-31T15:15:46.528 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.32537.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44488.log 2024-03-31T15:15:46.528 INFO:teuthology.orchestra.run.smithi016.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.32537.log.gz 2024-03-31T15:15:46.528 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.49828.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54071.log 2024-03-31T15:15:46.528 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.44488.log: 91.4% -- replaced with /var/log/ceph/ceph-client.admin.49828.log.gz 2024-03-31T15:15:46.528 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47271.log 2024-03-31T15:15:46.528 INFO:teuthology.orchestra.run.smithi016.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.44488.log.gz 2024-03-31T15:15:46.528 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.54071.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34496.log 2024-03-31T15:15:46.528 INFO:teuthology.orchestra.run.smithi016.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.54071.log.gz 2024-03-31T15:15:46.528 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.47271.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44940.log 2024-03-31T15:15:46.529 INFO:teuthology.orchestra.run.smithi016.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.47271.log.gz 2024-03-31T15:15:46.529 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.34496.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54175.log 2024-03-31T15:15:46.534 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.27663.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27687.log 2024-03-31T15:15:46.534 INFO:teuthology.orchestra.run.smithi089.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.27663.log.gz 2024-03-31T15:15:46.534 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.1.32929.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-03-31T15:15:46.534 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.27687.log: 90.1% -- replaced with /var/log/ceph/ceph-client.1.32929.log.gz 2024-03-31T15:15:46.534 INFO:teuthology.orchestra.run.smithi089.stderr: 92.5% -- replaced with /var/log/ceph/ceph.log.gz 2024-03-31T15:15:46.534 INFO:teuthology.orchestra.run.smithi089.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.27687.log.gz 2024-03-31T15:15:46.534 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.3.33047.log 2024-03-31T15:15:46.535 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-31T15:15:46.535 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.3.33047.log: gzip -5 --verbose -- /var/log/ceph/ceph-mds.d.log 2024-03-31T15:15:46.535 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph.audit.log: 90.8% -- replaced with /var/log/ceph/ceph-client.3.33047.log.gz 2024-03-31T15:15:46.535 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27735.log 2024-03-31T15:15:46.535 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-mds.d.log: /var/log/ceph/ceph-client.admin.27735.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.27735.log.gz 2024-03-31T15:15:46.535 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.b.log 2024-03-31T15:15:46.536 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-mds.b.log: 93.8% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-31T15:15:46.536 INFO:teuthology.orchestra.run.smithi089.stderr: 92.6% -- replaced with /var/log/ceph/ceph-mds.d.log.gz 2024-03-31T15:15:46.537 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-03-31T15:15:46.538 INFO:teuthology.orchestra.run.smithi016.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.34496.log.gz 2024-03-31T15:15:46.538 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.44940.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40864.log 2024-03-31T15:15:46.538 INFO:teuthology.orchestra.run.smithi016.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.44940.log.gz 2024-03-31T15:15:46.538 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.54175.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50252.log 2024-03-31T15:15:46.538 INFO:teuthology.orchestra.run.smithi016.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.54175.log.gz 2024-03-31T15:15:46.538 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.40864.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49341.log 2024-03-31T15:15:46.539 INFO:teuthology.orchestra.run.smithi016.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.40864.log.gz 2024-03-31T15:15:46.539 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.50252.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35181.log 2024-03-31T15:15:46.539 INFO:teuthology.orchestra.run.smithi016.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.50252.log.gz 2024-03-31T15:15:46.539 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.49341.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52595.log 2024-03-31T15:15:46.539 INFO:teuthology.orchestra.run.smithi016.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.49341.log.gz 2024-03-31T15:15:46.539 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.35181.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54201.log 2024-03-31T15:15:46.539 INFO:teuthology.orchestra.run.smithi016.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.35181.log.gz 2024-03-31T15:15:46.539 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.52595.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54331.log 2024-03-31T15:15:46.539 INFO:teuthology.orchestra.run.smithi016.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.52595.log.gz 2024-03-31T15:15:46.539 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.54201.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31627.log 2024-03-31T15:15:46.539 INFO:teuthology.orchestra.run.smithi016.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.54201.log.gz 2024-03-31T15:15:46.539 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.54331.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41655.log 2024-03-31T15:15:46.539 INFO:teuthology.orchestra.run.smithi016.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.54331.log.gz 2024-03-31T15:15:46.539 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.31627.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36234.log 2024-03-31T15:15:46.539 INFO:teuthology.orchestra.run.smithi016.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.31627.log.gz 2024-03-31T15:15:46.540 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.41655.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52891.log 2024-03-31T15:15:46.540 INFO:teuthology.orchestra.run.smithi016.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.41655.log.gz 2024-03-31T15:15:46.540 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.36234.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53242.log 2024-03-31T15:15:46.540 INFO:teuthology.orchestra.run.smithi016.stderr: 91.5% -- replaced with /var/log/ceph/ceph-client.admin.36234.log.gz 2024-03-31T15:15:46.540 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.52891.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49368.log 2024-03-31T15:15:46.541 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-03-31T15:15:46.543 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.53242.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33831.log 2024-03-31T15:15:46.543 INFO:teuthology.orchestra.run.smithi016.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.53242.log.gz 2024-03-31T15:15:46.543 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.49368.log: 92.5% -- replaced with /var/log/ceph/ceph-client.admin.52891.log.gz 2024-03-31T15:15:46.543 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41896.log 2024-03-31T15:15:46.543 INFO:teuthology.orchestra.run.smithi016.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.49368.log.gz 2024-03-31T15:15:46.543 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.33831.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-03-31T15:15:46.543 INFO:teuthology.orchestra.run.smithi016.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.33831.log.gz 2024-03-31T15:15:46.543 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.41896.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41499.log 2024-03-31T15:15:46.544 INFO:teuthology.orchestra.run.smithi016.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.41896.log.gz 2024-03-31T15:15:46.544 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50148.log 2024-03-31T15:15:46.546 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.41499.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42909.log 2024-03-31T15:15:46.546 INFO:teuthology.orchestra.run.smithi016.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.41499.log.gz 2024-03-31T15:15:46.546 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.50148.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34785.log 2024-03-31T15:15:46.546 INFO:teuthology.orchestra.run.smithi016.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.50148.log.gz 2024-03-31T15:15:46.546 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.42909.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32959.log 2024-03-31T15:15:46.546 INFO:teuthology.orchestra.run.smithi016.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.42909.log.gz 2024-03-31T15:15:46.546 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.34785.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35103.log 2024-03-31T15:15:46.547 INFO:teuthology.orchestra.run.smithi016.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.34785.log.gz 2024-03-31T15:15:46.547 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.32959.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49913.log 2024-03-31T15:15:46.548 INFO:teuthology.orchestra.run.smithi016.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.32959.log.gz 2024-03-31T15:15:46.548 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.35103.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32907.log 2024-03-31T15:15:46.548 INFO:teuthology.orchestra.run.smithi016.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.35103.log.gz 2024-03-31T15:15:46.548 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.49913.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31600.log 2024-03-31T15:15:46.548 INFO:teuthology.orchestra.run.smithi016.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.49913.log.gz 2024-03-31T15:15:46.549 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.32907.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35551.log 2024-03-31T15:15:46.549 INFO:teuthology.orchestra.run.smithi016.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.32907.log.gz 2024-03-31T15:15:46.550 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.31600.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47245.log 2024-03-31T15:15:46.550 INFO:teuthology.orchestra.run.smithi016.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.31600.log.gz 2024-03-31T15:15:46.552 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.35551.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35389.log 2024-03-31T15:15:46.552 INFO:teuthology.orchestra.run.smithi016.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.35551.log.gz 2024-03-31T15:15:46.552 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.47245.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36742.log 2024-03-31T15:15:46.552 INFO:teuthology.orchestra.run.smithi016.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.47245.log.gz 2024-03-31T15:15:46.553 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.35389.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44263.log 2024-03-31T15:15:46.553 INFO:teuthology.orchestra.run.smithi016.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.35389.log.gz 2024-03-31T15:15:46.554 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.36742.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44992.log 2024-03-31T15:15:46.554 INFO:teuthology.orchestra.run.smithi016.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.36742.log.gz 2024-03-31T15:15:46.554 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.44263.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35493.log 2024-03-31T15:15:46.555 INFO:teuthology.orchestra.run.smithi016.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.44263.log.gz 2024-03-31T15:15:46.555 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.44992.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43505.log 2024-03-31T15:15:46.556 INFO:teuthology.orchestra.run.smithi016.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.44992.log.gz 2024-03-31T15:15:46.556 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.35493.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37510.log 2024-03-31T15:15:46.557 INFO:teuthology.orchestra.run.smithi016.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.35493.log.gz 2024-03-31T15:15:46.557 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.43505.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37706.log 2024-03-31T15:15:46.557 INFO:teuthology.orchestra.run.smithi016.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.43505.log.gz 2024-03-31T15:15:46.558 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.37510.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38964.log 2024-03-31T15:15:46.558 INFO:teuthology.orchestra.run.smithi016.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.37510.log.gz 2024-03-31T15:15:46.559 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.37706.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33115.log 2024-03-31T15:15:46.559 INFO:teuthology.orchestra.run.smithi016.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.37706.log.gz 2024-03-31T15:15:46.560 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.38964.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52864.log 2024-03-31T15:15:46.560 INFO:teuthology.orchestra.run.smithi016.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.38964.log.gz 2024-03-31T15:15:46.561 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.33115.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50513.log 2024-03-31T15:15:46.562 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.52864.log: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.33115.log.gz 2024-03-31T15:15:46.562 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43564.log 2024-03-31T15:15:46.563 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.50513.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41273.log 2024-03-31T15:15:46.563 INFO:teuthology.orchestra.run.smithi016.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.50513.log.gz 2024-03-31T15:15:46.563 INFO:teuthology.orchestra.run.smithi016.stderr: 92.3% -- replaced with /var/log/ceph/ceph-client.admin.52864.log.gz 2024-03-31T15:15:46.564 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.43564.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33513.log 2024-03-31T15:15:46.564 INFO:teuthology.orchestra.run.smithi016.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.43564.log.gz 2024-03-31T15:15:46.565 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-mon.b.log: /var/log/ceph/ceph-osd.4.log: 91.5% -- replaced with /var/log/ceph/ceph-mds.b.log.gz 2024-03-31T15:15:46.566 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.41273.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44027.log 2024-03-31T15:15:46.566 INFO:teuthology.orchestra.run.smithi016.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.41273.log.gz 2024-03-31T15:15:46.566 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.33513.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49966.log 2024-03-31T15:15:46.566 INFO:teuthology.orchestra.run.smithi016.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.33513.log.gz 2024-03-31T15:15:46.566 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.44027.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36623.log 2024-03-31T15:15:46.566 INFO:teuthology.orchestra.run.smithi016.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.44027.log.gz 2024-03-31T15:15:46.567 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.49966.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41870.log 2024-03-31T15:15:46.567 INFO:teuthology.orchestra.run.smithi016.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.49966.log.gz 2024-03-31T15:15:46.568 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.36623.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35519.log 2024-03-31T15:15:46.568 INFO:teuthology.orchestra.run.smithi016.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.36623.log.gz 2024-03-31T15:15:46.569 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.41870.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48979.log 2024-03-31T15:15:46.569 INFO:teuthology.orchestra.run.smithi016.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.41870.log.gz 2024-03-31T15:15:46.569 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.35519.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41354.log 2024-03-31T15:15:46.570 INFO:teuthology.orchestra.run.smithi016.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.35519.log.gz 2024-03-31T15:15:46.570 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.48979.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.53165.log 2024-03-31T15:15:46.571 INFO:teuthology.orchestra.run.smithi016.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.48979.log.gz 2024-03-31T15:15:46.571 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.41354.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.51184.log 2024-03-31T15:15:46.571 INFO:teuthology.orchestra.run.smithi016.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.41354.log.gz 2024-03-31T15:15:46.572 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.0.53165.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48514.log 2024-03-31T15:15:46.573 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.0.51184.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53059.log 2024-03-31T15:15:46.574 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.48514.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37445.log 2024-03-31T15:15:46.574 INFO:teuthology.orchestra.run.smithi016.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.48514.log.gz 2024-03-31T15:15:46.575 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.53059.log: 93.3% -- replaced with /var/log/ceph/ceph-client.0.53165.log.gz 2024-03-31T15:15:46.575 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46286.log 2024-03-31T15:15:46.576 INFO:teuthology.orchestra.run.smithi016.stderr: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.53059.log.gz 2024-03-31T15:15:46.576 INFO:teuthology.orchestra.run.smithi016.stderr:gzip/var/log/ceph/ceph-client.admin.37445.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.27409.log 2024-03-31T15:15:46.577 INFO:teuthology.orchestra.run.smithi016.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.37445.log.gz 2024-03-31T15:15:46.577 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.46286.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41300.log 2024-03-31T15:15:46.578 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.27409.log: 92.2% -- replaced with /var/log/ceph/ceph-client.admin.46286.log.gz 93.0% 2024-03-31T15:15:46.578 INFO:teuthology.orchestra.run.smithi016.stderr: -- replaced with /var/log/ceph/ceph-client.0.51184.log.gz 2024-03-31T15:15:46.579 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42490.log 2024-03-31T15:15:46.579 INFO:teuthology.orchestra.run.smithi016.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.27409.log.gz 2024-03-31T15:15:46.579 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46233.log 2024-03-31T15:15:46.579 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.41300.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.41300.log.gz 2024-03-31T15:15:46.580 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.42490.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40935.log 2024-03-31T15:15:46.580 INFO:teuthology.orchestra.run.smithi016.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.42490.log.gz 2024-03-31T15:15:46.581 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.46233.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54436.log 2024-03-31T15:15:46.581 INFO:teuthology.orchestra.run.smithi016.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.46233.log.gz 2024-03-31T15:15:46.582 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.40935.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36571.log 2024-03-31T15:15:46.582 INFO:teuthology.orchestra.run.smithi016.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.40935.log.gz 2024-03-31T15:15:46.582 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.54436.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39846.log 2024-03-31T15:15:46.583 INFO:teuthology.orchestra.run.smithi016.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.54436.log.gz 2024-03-31T15:15:46.583 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.36571.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35747.log 2024-03-31T15:15:46.584 INFO:teuthology.orchestra.run.smithi016.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.36571.log.gz 2024-03-31T15:15:46.584 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.39846.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.39846.log.gz 2024-03-31T15:15:46.585 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54227.log 2024-03-31T15:15:46.585 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.35747.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33989.log 2024-03-31T15:15:46.585 INFO:teuthology.orchestra.run.smithi016.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.35747.log.gz 2024-03-31T15:15:46.586 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.54227.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43590.log 2024-03-31T15:15:46.586 INFO:teuthology.orchestra.run.smithi016.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.54227.log.gz 2024-03-31T15:15:46.589 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.33989.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50200.log 2024-03-31T15:15:46.591 INFO:teuthology.orchestra.run.smithi016.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.33989.log.gz 2024-03-31T15:15:46.591 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.43590.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38494.log 2024-03-31T15:15:46.591 INFO:teuthology.orchestra.run.smithi016.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.43590.log.gz 2024-03-31T15:15:46.591 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.50200.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39633.log 2024-03-31T15:15:46.591 INFO:teuthology.orchestra.run.smithi016.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.50200.log.gz 2024-03-31T15:15:46.592 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.38494.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49755.log 2024-03-31T15:15:46.592 INFO:teuthology.orchestra.run.smithi016.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.38494.log.gz 2024-03-31T15:15:46.592 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.39633.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38207.log 2024-03-31T15:15:46.592 INFO:teuthology.orchestra.run.smithi016.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.39633.log.gz 2024-03-31T15:15:46.592 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.49755.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33727.log 2024-03-31T15:15:46.593 INFO:teuthology.orchestra.run.smithi016.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.49755.log.gz 2024-03-31T15:15:46.593 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.38207.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45586.log 2024-03-31T15:15:46.593 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.33727.log: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.38207.log.gz 2024-03-31T15:15:46.593 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 87.8% --verbose -- replaced with /var/log/ceph/ceph-client.admin.33727.log.gz 2024-03-31T15:15:46.593 INFO:teuthology.orchestra.run.smithi016.stderr: -- /var/log/ceph/ceph-client.admin.34522.log 2024-03-31T15:15:46.594 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.45586.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.38076.log 2024-03-31T15:15:46.594 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.34522.log: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.45586.log.gz 2024-03-31T15:15:46.595 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33805.log 2024-03-31T15:15:46.595 INFO:teuthology.orchestra.run.smithi016.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.34522.log.gz 2024-03-31T15:15:46.595 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.0.38076.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35285.log 2024-03-31T15:15:46.596 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.33805.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44644.log 2024-03-31T15:15:46.596 INFO:teuthology.orchestra.run.smithi016.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.33805.log.gz 2024-03-31T15:15:46.597 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.35285.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35337.log 2024-03-31T15:15:46.598 INFO:teuthology.orchestra.run.smithi016.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.35285.log.gz 2024-03-31T15:15:46.598 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.44644.log: 93.2% -- replaced with /var/log/ceph/ceph-client.0.38076.log.gz 2024-03-31T15:15:46.598 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.2.33443.log 2024-03-31T15:15:46.598 INFO:teuthology.orchestra.run.smithi016.stderr: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.44644.log.gz 2024-03-31T15:15:46.599 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.35337.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42699.log 2024-03-31T15:15:46.599 INFO:teuthology.orchestra.run.smithi016.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.35337.log.gz 2024-03-31T15:15:46.600 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.2.33443.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47872.log 2024-03-31T15:15:46.601 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.42699.log: 91.2% -- replaced with /var/log/ceph/ceph-client.2.33443.log.gz 2024-03-31T15:15:46.601 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51918.log 2024-03-31T15:15:46.601 INFO:teuthology.orchestra.run.smithi016.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.42699.log.gz 2024-03-31T15:15:46.602 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.47872.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35415.log 2024-03-31T15:15:46.602 INFO:teuthology.orchestra.run.smithi016.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.47872.log.gz 2024-03-31T15:15:46.603 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.51918.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44184.log 2024-03-31T15:15:46.603 INFO:teuthology.orchestra.run.smithi016.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.51918.log.gz 2024-03-31T15:15:46.603 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.35415.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35825.log 2024-03-31T15:15:46.603 INFO:teuthology.orchestra.run.smithi016.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.35415.log.gz 2024-03-31T15:15:46.604 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.44184.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37249.log 2024-03-31T15:15:46.605 INFO:teuthology.orchestra.run.smithi016.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.44184.log.gz 2024-03-31T15:15:46.605 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.35825.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48488.log 2024-03-31T15:15:46.605 INFO:teuthology.orchestra.run.smithi016.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.35825.log.gz 2024-03-31T15:15:46.606 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.37249.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53268.log 2024-03-31T15:15:46.606 INFO:teuthology.orchestra.run.smithi016.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.37249.log.gz 2024-03-31T15:15:46.606 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.48488.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31893.log 2024-03-31T15:15:46.607 INFO:teuthology.orchestra.run.smithi016.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.48488.log.gz 2024-03-31T15:15:46.607 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.53268.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36764.log 2024-03-31T15:15:46.608 INFO:teuthology.orchestra.run.smithi016.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.53268.log.gz 2024-03-31T15:15:46.608 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.31893.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32545.log 2024-03-31T15:15:46.608 INFO:teuthology.orchestra.run.smithi016.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.31893.log.gz 2024-03-31T15:15:46.609 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.36764.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34178.log 2024-03-31T15:15:46.610 INFO:teuthology.orchestra.run.smithi016.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.36764.log.gz 2024-03-31T15:15:46.610 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.32545.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36918.log 2024-03-31T15:15:46.610 INFO:teuthology.orchestra.run.smithi016.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.32545.log.gz 2024-03-31T15:15:46.611 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.34178.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37022.log 2024-03-31T15:15:46.611 INFO:teuthology.orchestra.run.smithi016.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.34178.log.gz 2024-03-31T15:15:46.612 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.36918.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44001.log 2024-03-31T15:15:46.612 INFO:teuthology.orchestra.run.smithi016.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.36918.log.gz 2024-03-31T15:15:46.613 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.37022.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35233.log 2024-03-31T15:15:46.613 INFO:teuthology.orchestra.run.smithi016.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.37022.log.gz 2024-03-31T15:15:46.614 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.44001.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34392.log 2024-03-31T15:15:46.614 INFO:teuthology.orchestra.run.smithi016.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.44001.log.gz 2024-03-31T15:15:46.614 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.35233.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52622.log 2024-03-31T15:15:46.615 INFO:teuthology.orchestra.run.smithi016.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.35233.log.gz 2024-03-31T15:15:46.615 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.34392.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46510.log 2024-03-31T15:15:46.616 INFO:teuthology.orchestra.run.smithi016.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.34392.log.gz 2024-03-31T15:15:46.616 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.52622.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-03-31T15:15:46.616 INFO:teuthology.orchestra.run.smithi016.stderr: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.52622.log.gz 2024-03-31T15:15:46.617 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.46510.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38685.log 2024-03-31T15:15:46.618 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47396.log 2024-03-31T15:15:46.618 INFO:teuthology.orchestra.run.smithi016.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.46510.log.gz 2024-03-31T15:15:46.619 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.38685.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32803.log 2024-03-31T15:15:46.619 INFO:teuthology.orchestra.run.smithi016.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.38685.log.gz 2024-03-31T15:15:46.620 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.47396.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33089.log 2024-03-31T15:15:46.620 INFO:teuthology.orchestra.run.smithi016.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.47396.log.gz 2024-03-31T15:15:46.621 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.32803.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51952.log 2024-03-31T15:15:46.621 INFO:teuthology.orchestra.run.smithi016.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.32803.log.gz 2024-03-31T15:15:46.622 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.33089.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37223.log 2024-03-31T15:15:46.622 INFO:teuthology.orchestra.run.smithi016.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.33089.log.gz 2024-03-31T15:15:46.622 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.51952.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49223.log 2024-03-31T15:15:46.623 INFO:teuthology.orchestra.run.smithi016.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.51952.log.gz 2024-03-31T15:15:46.623 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.37223.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52837.log 2024-03-31T15:15:46.623 INFO:teuthology.orchestra.run.smithi016.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.37223.log.gz 2024-03-31T15:15:46.624 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.49223.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33545.log 2024-03-31T15:15:46.624 INFO:teuthology.orchestra.run.smithi016.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.49223.log.gz 2024-03-31T15:15:46.625 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.52837.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42303.log 2024-03-31T15:15:46.626 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.33545.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.33545.log.gz 2024-03-31T15:15:46.626 INFO:teuthology.orchestra.run.smithi016.stderr: 92.4% -- replaced with /var/log/ceph/ceph-client.admin.52837.log.gz 2024-03-31T15:15:46.627 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52130.log 2024-03-31T15:15:46.627 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.42303.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41469.log 2024-03-31T15:15:46.627 INFO:teuthology.orchestra.run.smithi016.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.42303.log.gz 2024-03-31T15:15:46.628 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.52130.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49171.log 2024-03-31T15:15:46.628 INFO:teuthology.orchestra.run.smithi016.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.52130.log.gz 2024-03-31T15:15:46.629 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.41469.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33348.log 2024-03-31T15:15:46.629 INFO:teuthology.orchestra.run.smithi016.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.41469.log.gz 2024-03-31T15:15:46.630 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.49171.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49702.log 2024-03-31T15:15:46.630 INFO:teuthology.orchestra.run.smithi016.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.49171.log.gz 2024-03-31T15:15:46.631 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.33348.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49552.log 2024-03-31T15:15:46.631 INFO:teuthology.orchestra.run.smithi016.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.33348.log.gz 2024-03-31T15:15:46.632 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.49702.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39358.log 2024-03-31T15:15:46.632 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.49552.log: 91.3% -- replaced with /var/log/ceph/ceph-client.admin.49702.log.gz 2024-03-31T15:15:46.633 INFO:teuthology.orchestra.run.smithi016.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.49552.log.gz 2024-03-31T15:15:46.633 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34230.log 2024-03-31T15:15:46.634 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.39358.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32329.log 2024-03-31T15:15:46.634 INFO:teuthology.orchestra.run.smithi016.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.39358.log.gz 2024-03-31T15:15:46.634 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.34230.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39473.log 2024-03-31T15:15:46.635 INFO:teuthology.orchestra.run.smithi016.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.34230.log.gz 2024-03-31T15:15:46.635 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.32329.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39529.log 2024-03-31T15:15:46.635 INFO:teuthology.orchestra.run.smithi016.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.32329.log.gz 2024-03-31T15:15:46.636 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.39473.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34837.log 2024-03-31T15:15:46.636 INFO:teuthology.orchestra.run.smithi016.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.39473.log.gz 2024-03-31T15:15:46.637 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.39529.log: gzip -5 --verbose 89.1% -- -- replaced with /var/log/ceph/ceph-client.admin.39529.log.gz 2024-03-31T15:15:46.637 INFO:teuthology.orchestra.run.smithi016.stderr: /var/log/ceph/ceph-client.admin.27520.log 2024-03-31T15:15:46.638 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.34837.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32109.log 2024-03-31T15:15:46.639 INFO:teuthology.orchestra.run.smithi016.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.34837.log.gz 2024-03-31T15:15:46.639 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.27520.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34889.log 2024-03-31T15:15:46.639 INFO:teuthology.orchestra.run.smithi016.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.27520.log.gz 2024-03-31T15:15:46.640 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.32109.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35051.log 2024-03-31T15:15:46.640 INFO:teuthology.orchestra.run.smithi016.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.32109.log.gz 2024-03-31T15:15:46.641 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.34889.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35935.log 2024-03-31T15:15:46.641 INFO:teuthology.orchestra.run.smithi016.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.34889.log.gz 2024-03-31T15:15:46.642 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.35051.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47711.log 2024-03-31T15:15:46.642 INFO:teuthology.orchestra.run.smithi016.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.35051.log.gz 2024-03-31T15:15:46.643 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.35935.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52460.log 2024-03-31T15:15:46.643 INFO:teuthology.orchestra.run.smithi016.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.35935.log.gz 2024-03-31T15:15:46.644 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.47711.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53616.log 2024-03-31T15:15:46.644 INFO:teuthology.orchestra.run.smithi016.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.47711.log.gz 2024-03-31T15:15:46.645 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.52460.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53993.log 2024-03-31T15:15:46.645 INFO:teuthology.orchestra.run.smithi016.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.52460.log.gz 2024-03-31T15:15:46.646 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.53616.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38180.log 2024-03-31T15:15:46.646 INFO:teuthology.orchestra.run.smithi016.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.53616.log.gz 2024-03-31T15:15:46.647 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.53993.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46427.log 2024-03-31T15:15:46.647 INFO:teuthology.orchestra.run.smithi016.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.53993.log.gz 2024-03-31T15:15:46.648 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.38180.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41731.log 2024-03-31T15:15:46.648 INFO:teuthology.orchestra.run.smithi016.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.38180.log.gz 2024-03-31T15:15:46.649 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.46427.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39068.log 2024-03-31T15:15:46.649 INFO:teuthology.orchestra.run.smithi016.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.46427.log.gz 2024-03-31T15:15:46.650 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.41731.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-03-31T15:15:46.650 INFO:teuthology.orchestra.run.smithi016.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.41731.log.gz 2024-03-31T15:15:46.651 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.39068.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45916.log 2024-03-31T15:15:46.651 INFO:teuthology.orchestra.run.smithi016.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.39068.log.gz 2024-03-31T15:15:46.652 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34204.log 2024-03-31T15:15:46.653 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.45916.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35907.log 2024-03-31T15:15:46.653 INFO:teuthology.orchestra.run.smithi016.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.45916.log.gz 2024-03-31T15:15:46.654 INFO:teuthology.orchestra.run.smithi016.stderr:gzip/var/log/ceph/ceph-client.admin.34204.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.41013.log 2024-03-31T15:15:46.654 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.35907.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.34204.log.gz 2024-03-31T15:15:46.655 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50356.log 2024-03-31T15:15:46.655 INFO:teuthology.orchestra.run.smithi016.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.35907.log.gz 2024-03-31T15:15:46.656 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.41013.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32125.log 2024-03-31T15:15:46.656 INFO:teuthology.orchestra.run.smithi016.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.41013.log.gz 2024-03-31T15:15:46.657 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31653.log 2024-03-31T15:15:46.658 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.50356.log: /var/log/ceph/ceph-client.admin.32125.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.50356.log.gz 2024-03-31T15:15:46.658 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37197.log 2024-03-31T15:15:46.658 INFO:teuthology.orchestra.run.smithi016.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.32125.log.gz 2024-03-31T15:15:46.659 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.31653.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34574.log 2024-03-31T15:15:46.659 INFO:teuthology.orchestra.run.smithi016.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.31653.log.gz 2024-03-31T15:15:46.660 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.51624.log 2024-03-31T15:15:46.661 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.37197.log: /var/log/ceph/ceph-client.admin.34574.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.37197.log.gz 2024-03-31T15:15:46.661 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47349.log 2024-03-31T15:15:46.661 INFO:teuthology.orchestra.run.smithi016.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.34574.log.gz 2024-03-31T15:15:46.662 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.0.51624.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45662.log 2024-03-31T15:15:46.663 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.47349.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38157.log 2024-03-31T15:15:46.664 INFO:teuthology.orchestra.run.smithi016.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.47349.log.gz 2024-03-31T15:15:46.664 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.45662.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42329.log 2024-03-31T15:15:46.665 INFO:teuthology.orchestra.run.smithi016.stderr: 91.5% -- replaced with /var/log/ceph/ceph-client.admin.45662.log.gz 2024-03-31T15:15:46.665 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.38157.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39581.log 2024-03-31T15:15:46.666 INFO:teuthology.orchestra.run.smithi016.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.38157.log.gz 2024-03-31T15:15:46.667 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.42329.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35695.log 2024-03-31T15:15:46.667 INFO:teuthology.orchestra.run.smithi016.stderr: 94.5% -- replaced with /var/log/ceph/ceph-client.0.51624.log.gz 2024-03-31T15:15:46.667 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.39581.log: 91.5% -- replaced with /var/log/ceph/ceph-client.admin.42329.log.gz 2024-03-31T15:15:46.668 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40786.log 2024-03-31T15:15:46.668 INFO:teuthology.orchestra.run.smithi016.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.39581.log.gz 2024-03-31T15:15:46.669 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.35695.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36052.log 2024-03-31T15:15:46.669 INFO:teuthology.orchestra.run.smithi016.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.35695.log.gz 2024-03-31T15:15:46.670 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41551.log 2024-03-31T15:15:46.670 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.40786.log: /var/log/ceph/ceph-client.admin.36052.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.40786.log.gz 2024-03-31T15:15:46.671 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39159.log 2024-03-31T15:15:46.671 INFO:teuthology.orchestra.run.smithi016.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.36052.log.gz 2024-03-31T15:15:46.672 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.41551.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38659.log 2024-03-31T15:15:46.672 INFO:teuthology.orchestra.run.smithi016.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.41551.log.gz 2024-03-31T15:15:46.673 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.39159.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34973.log 2024-03-31T15:15:46.673 INFO:teuthology.orchestra.run.smithi016.stderr: 88.7%/var/log/ceph/ceph-client.admin.38659.log: -- replaced with /var/log/ceph/ceph-client.admin.39159.log.gz 2024-03-31T15:15:46.674 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52206.log 2024-03-31T15:15:46.674 INFO:teuthology.orchestra.run.smithi016.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.38659.log.gz 2024-03-31T15:15:46.675 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.34973.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38711.log 2024-03-31T15:15:46.675 INFO:teuthology.orchestra.run.smithi016.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.34973.log.gz 2024-03-31T15:15:46.676 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.43322.log 2024-03-31T15:15:46.676 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.52206.log: 88.2%/var/log/ceph/ceph-client.admin.38711.log: -- replaced with /var/log/ceph/ceph-client.admin.52206.log.gz 2024-03-31T15:15:46.677 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47528.log 2024-03-31T15:15:46.677 INFO:teuthology.orchestra.run.smithi016.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.38711.log.gz 2024-03-31T15:15:46.678 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.0.43322.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35799.log 2024-03-31T15:15:46.679 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.47528.log: 90.5% -- replaced with /var/log/ceph/ceph-client.0.43322.log.gz 2024-03-31T15:15:46.680 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44380.log 2024-03-31T15:15:46.680 INFO:teuthology.orchestra.run.smithi016.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.47528.log.gz 2024-03-31T15:15:46.681 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.35799.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34152.log 2024-03-31T15:15:46.681 INFO:teuthology.orchestra.run.smithi016.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.35799.log.gz 2024-03-31T15:15:46.682 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.44380.log: gzip -5 --verbose -- 88.2% /var/log/ceph/ceph-client.admin.47115.log -- replaced with /var/log/ceph/ceph-client.admin.44380.log.gz 2024-03-31T15:15:46.682 INFO:teuthology.orchestra.run.smithi016.stderr: 2024-03-31T15:15:46.683 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.34152.log: /var/log/ceph/ceph-client.admin.48356.log 2024-03-31T15:15:46.683 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.47115.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.34152.log.gz 2024-03-31T15:15:46.684 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39735.log 2024-03-31T15:15:46.684 INFO:teuthology.orchestra.run.smithi016.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.47115.log.gz 2024-03-31T15:15:46.685 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.48356.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47933.log 2024-03-31T15:15:46.685 INFO:teuthology.orchestra.run.smithi016.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.48356.log.gz 2024-03-31T15:15:46.686 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.39735.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41065.log 2024-03-31T15:15:46.686 INFO:teuthology.orchestra.run.smithi016.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.39735.log.gz 2024-03-31T15:15:46.687 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.47933.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.41790.log 2024-03-31T15:15:46.688 INFO:teuthology.orchestra.run.smithi016.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.47933.log.gz 2024-03-31T15:15:46.688 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.41065.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41977.log 2024-03-31T15:15:46.689 INFO:teuthology.orchestra.run.smithi016.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.41065.log.gz 2024-03-31T15:15:46.689 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41681.log 2024-03-31T15:15:46.690 INFO:teuthology.orchestra.run.smithi089.stderr: 91.7% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-03-31T15:15:46.690 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.0.41790.log: /var/log/ceph/ceph-client.admin.41977.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.41977.log.gz 2024-03-31T15:15:46.690 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45967.log 2024-03-31T15:15:46.691 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36078.log 2024-03-31T15:15:46.693 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.41681.log: /var/log/ceph/ceph-client.admin.45967.log: 91.3%gzip -- replaced with /var/log/ceph/ceph-client.admin.41681.log.gz -5 2024-03-31T15:15:46.693 INFO:teuthology.orchestra.run.smithi016.stderr: --verbose -- /var/log/ceph/ceph-client.admin.36208.log 2024-03-31T15:15:46.693 INFO:teuthology.orchestra.run.smithi016.stderr: 92.8% -- replaced with /var/log/ceph/ceph-client.0.41790.log.gz 2024-03-31T15:15:46.693 INFO:teuthology.orchestra.run.smithi016.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.45967.log.gz 2024-03-31T15:15:46.694 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.36078.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34947.log 2024-03-31T15:15:46.694 INFO:teuthology.orchestra.run.smithi016.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.36078.log.gz 2024-03-31T15:15:46.694 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.36208.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33675.log 2024-03-31T15:15:46.695 INFO:teuthology.orchestra.run.smithi016.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.36208.log.gz 2024-03-31T15:15:46.695 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.34947.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.34947.log.gz 2024-03-31T15:15:46.696 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48927.log 2024-03-31T15:15:46.697 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.33675.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.39791.log 2024-03-31T15:15:46.697 INFO:teuthology.orchestra.run.smithi016.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.33675.log.gz 2024-03-31T15:15:46.698 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.48927.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43453.log 2024-03-31T15:15:46.698 INFO:teuthology.orchestra.run.smithi016.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.48927.log.gz 2024-03-31T15:15:46.699 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.0.39791.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43164.log 2024-03-31T15:15:46.700 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.43453.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32075.log 2024-03-31T15:15:46.700 INFO:teuthology.orchestra.run.smithi016.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.43453.log.gz 2024-03-31T15:15:46.701 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.43164.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31971.log 2024-03-31T15:15:46.701 INFO:teuthology.orchestra.run.smithi016.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.43164.log.gz 2024-03-31T15:15:46.702 INFO:teuthology.orchestra.run.smithi016.stderr: 93.9% -- replaced with /var/log/ceph/ceph-client.0.39791.log.gz 2024-03-31T15:15:46.702 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.32075.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43871.log 2024-03-31T15:15:46.703 INFO:teuthology.orchestra.run.smithi016.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.32075.log.gz 2024-03-31T15:15:46.703 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37048.log 2024-03-31T15:15:46.704 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.31971.log: /var/log/ceph/ceph-client.admin.43871.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.31971.log.gz 2024-03-31T15:15:46.704 INFO:teuthology.orchestra.run.smithi016.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.43871.log.gz 2024-03-31T15:15:46.704 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33063.log 2024-03-31T15:15:46.705 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32117.log 2024-03-31T15:15:46.706 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.37048.log: /var/log/ceph/ceph-client.admin.33063.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.37048.log.gz 2024-03-31T15:15:46.706 INFO:teuthology.orchestra.run.smithi016.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.33063.log.gz 2024-03-31T15:15:46.706 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46454.log 2024-03-31T15:15:46.707 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.32117.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43616.log 2024-03-31T15:15:46.707 INFO:teuthology.orchestra.run.smithi016.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.32117.log.gz 2024-03-31T15:15:46.708 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.46454.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53713.log 2024-03-31T15:15:46.708 INFO:teuthology.orchestra.run.smithi016.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.46454.log.gz 2024-03-31T15:15:46.709 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.43616.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51866.log 2024-03-31T15:15:46.709 INFO:teuthology.orchestra.run.smithi016.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.43616.log.gz 2024-03-31T15:15:46.710 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.53713.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36284.log 2024-03-31T15:15:46.711 INFO:teuthology.orchestra.run.smithi016.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.53713.log.gz 2024-03-31T15:15:46.711 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.51866.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34015.log 2024-03-31T15:15:46.712 INFO:teuthology.orchestra.run.smithi016.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.51866.log.gz 2024-03-31T15:15:46.713 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.36284.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54019.log 2024-03-31T15:15:46.713 INFO:teuthology.orchestra.run.smithi016.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.36284.log.gz 2024-03-31T15:15:46.713 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.34015.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36970.log 2024-03-31T15:15:46.714 INFO:teuthology.orchestra.run.smithi016.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.34015.log.gz 2024-03-31T15:15:46.714 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.54019.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34041.log 2024-03-31T15:15:46.715 INFO:teuthology.orchestra.run.smithi016.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.54019.log.gz 2024-03-31T15:15:46.715 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.36970.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35025.log 2024-03-31T15:15:46.716 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.34041.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.36970.log.gz 2024-03-31T15:15:46.717 INFO:teuthology.orchestra.run.smithi016.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.34041.log.gz 2024-03-31T15:15:46.717 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39873.log 2024-03-31T15:15:46.717 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39304.log 2024-03-31T15:15:46.718 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.35025.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.35025.log.gz 2024-03-31T15:15:46.719 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.39873.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50710.log 2024-03-31T15:15:46.719 INFO:teuthology.orchestra.run.smithi016.stderr: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.39873.log.gz 2024-03-31T15:15:46.720 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.39304.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42246.log 2024-03-31T15:15:46.720 INFO:teuthology.orchestra.run.smithi016.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.39304.log.gz 2024-03-31T15:15:46.721 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.50710.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52100.log 2024-03-31T15:15:46.721 INFO:teuthology.orchestra.run.smithi016.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.50710.log.gz 2024-03-31T15:15:46.722 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34706.log 2024-03-31T15:15:46.722 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.42246.log: /var/log/ceph/ceph-client.admin.52100.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.42246.log.gz 2024-03-31T15:15:46.723 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37152.log 2024-03-31T15:15:46.723 INFO:teuthology.orchestra.run.smithi016.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.52100.log.gz 2024-03-31T15:15:46.724 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.34706.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51812.log 2024-03-31T15:15:46.724 INFO:teuthology.orchestra.run.smithi016.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.34706.log.gz 2024-03-31T15:15:46.725 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.37152.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37589.log 2024-03-31T15:15:46.725 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.51812.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.37152.log.gz 2024-03-31T15:15:46.726 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50565.log 2024-03-31T15:15:46.726 INFO:teuthology.orchestra.run.smithi016.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.51812.log.gz 2024-03-31T15:15:46.727 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.37589.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42725.log 2024-03-31T15:15:46.727 INFO:teuthology.orchestra.run.smithi016.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.37589.log.gz 2024-03-31T15:15:46.728 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.50565.log: -- /var/log/ceph/ceph-client.admin.35773.log 2024-03-31T15:15:46.728 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.42725.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.50565.log.gz 2024-03-31T15:15:46.729 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53765.log 2024-03-31T15:15:46.729 INFO:teuthology.orchestra.run.smithi016.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.42725.log.gz 2024-03-31T15:15:46.730 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.35773.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31476.log 2024-03-31T15:15:46.730 INFO:teuthology.orchestra.run.smithi016.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.35773.log.gz 2024-03-31T15:15:46.731 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39921.log 2024-03-31T15:15:46.731 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.53765.log: /var/log/ceph/ceph-client.admin.31476.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.53765.log.gz 2024-03-31T15:15:46.732 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42757.log 2024-03-31T15:15:46.732 INFO:teuthology.orchestra.run.smithi016.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.31476.log.gz 2024-03-31T15:15:46.733 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.39921.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52786.log 2024-03-31T15:15:46.733 INFO:teuthology.orchestra.run.smithi016.stderr: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.39921.log.gz 2024-03-31T15:15:46.734 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.42757.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40151.log 2024-03-31T15:15:46.735 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.52786.log: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.42757.log.gz 2024-03-31T15:15:46.735 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32049.log 2024-03-31T15:15:46.736 INFO:teuthology.orchestra.run.smithi016.stderr: 91.6% -- replaced with /var/log/ceph/ceph-client.admin.52786.log.gz 2024-03-31T15:15:46.737 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.40151.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34361.log 2024-03-31T15:15:46.737 INFO:teuthology.orchestra.run.smithi016.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.40151.log.gz 2024-03-31T15:15:46.737 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.32049.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42542.log 2024-03-31T15:15:46.738 INFO:teuthology.orchestra.run.smithi016.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.32049.log.gz 2024-03-31T15:15:46.739 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.34361.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35853.log 2024-03-31T15:15:46.739 INFO:teuthology.orchestra.run.smithi016.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.34361.log.gz 2024-03-31T15:15:46.740 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.42542.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33857.log 2024-03-31T15:15:46.740 INFO:teuthology.orchestra.run.smithi016.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.42542.log.gz 2024-03-31T15:15:46.740 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36492.log 2024-03-31T15:15:46.741 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.35853.log: /var/log/ceph/ceph-client.admin.33857.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.35853.log.gz 2024-03-31T15:15:46.742 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49397.log 2024-03-31T15:15:46.742 INFO:teuthology.orchestra.run.smithi016.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.33857.log.gz 2024-03-31T15:15:46.743 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.36492.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35667.log 2024-03-31T15:15:46.743 INFO:teuthology.orchestra.run.smithi016.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.36492.log.gz 2024-03-31T15:15:46.744 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45788.log 2024-03-31T15:15:46.744 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.49397.log: /var/log/ceph/ceph-client.admin.35667.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.49397.log.gz 2024-03-31T15:15:46.744 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34811.log 2024-03-31T15:15:46.745 INFO:teuthology.orchestra.run.smithi016.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.35667.log.gz 2024-03-31T15:15:46.746 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.45788.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47323.log 2024-03-31T15:15:46.746 INFO:teuthology.orchestra.run.smithi016.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.45788.log.gz 2024-03-31T15:15:46.747 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.34811.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53432.log 2024-03-31T15:15:46.747 INFO:teuthology.orchestra.run.smithi016.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.34811.log.gz 2024-03-31T15:15:46.748 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.47323.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32328.log 2024-03-31T15:15:46.748 INFO:teuthology.orchestra.run.smithi016.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.47323.log.gz 2024-03-31T15:15:46.749 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.53432.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41577.log 2024-03-31T15:15:46.749 INFO:teuthology.orchestra.run.smithi016.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.53432.log.gz 2024-03-31T15:15:46.750 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5/var/log/ceph/ceph-client.admin.32328.log: --verbose -- /var/log/ceph/ceph-client.admin.54149.log 2024-03-31T15:15:46.750 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.41577.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.32328.log.gz 2024-03-31T15:15:46.751 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35129.log 2024-03-31T15:15:46.751 INFO:teuthology.orchestra.run.smithi016.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.41577.log.gz 2024-03-31T15:15:46.752 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.54149.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38867.log 2024-03-31T15:15:46.752 INFO:teuthology.orchestra.run.smithi016.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.54149.log.gz 2024-03-31T15:15:46.753 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45712.log 2024-03-31T15:15:46.754 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.35129.log: /var/log/ceph/ceph-client.admin.38867.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.35129.log.gz 2024-03-31T15:15:46.754 INFO:teuthology.orchestra.run.smithi016.stderr:gzip 87.7% -- replaced with /var/log/ceph/ceph-client.admin.38867.log.gz -5 2024-03-31T15:15:46.755 INFO:teuthology.orchestra.run.smithi016.stderr: --verbose -- /var/log/ceph/ceph-client.admin.50018.log 2024-03-31T15:15:46.755 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.45712.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35996.log 2024-03-31T15:15:46.756 INFO:teuthology.orchestra.run.smithi016.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.45712.log.gz 2024-03-31T15:15:46.757 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.50018.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49887.log 2024-03-31T15:15:46.757 INFO:teuthology.orchestra.run.smithi016.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.50018.log.gz 2024-03-31T15:15:46.758 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.35996.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48627.log 2024-03-31T15:15:46.758 INFO:teuthology.orchestra.run.smithi016.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.35996.log.gz 2024-03-31T15:15:46.759 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.49887.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37892.log 2024-03-31T15:15:46.759 INFO:teuthology.orchestra.run.smithi016.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.49887.log.gz 2024-03-31T15:15:46.760 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.48627.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50122.log 2024-03-31T15:15:46.760 INFO:teuthology.orchestra.run.smithi016.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.48627.log.gz 2024-03-31T15:15:46.761 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39896.log 2024-03-31T15:15:46.761 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.37892.log: /var/log/ceph/ceph-client.admin.50122.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.37892.log.gz 2024-03-31T15:15:46.762 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45044.log 2024-03-31T15:15:46.762 INFO:teuthology.orchestra.run.smithi016.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.50122.log.gz 2024-03-31T15:15:46.763 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.39896.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50974.log 2024-03-31T15:15:46.763 INFO:teuthology.orchestra.run.smithi016.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.39896.log.gz 2024-03-31T15:15:46.764 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.45044.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50792.log 2024-03-31T15:15:46.764 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.50974.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.45044.log.gz 2024-03-31T15:15:46.765 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45840.log 2024-03-31T15:15:46.765 INFO:teuthology.orchestra.run.smithi016.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.50974.log.gz 2024-03-31T15:15:46.766 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.50792.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49031.log 2024-03-31T15:15:46.766 INFO:teuthology.orchestra.run.smithi016.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.50792.log.gz 2024-03-31T15:15:46.767 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47614.log 2024-03-31T15:15:46.768 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.45840.log: /var/log/ceph/ceph-client.admin.49031.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47763.log 2024-03-31T15:15:46.768 INFO:teuthology.orchestra.run.smithi016.stderr: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.45840.log.gz 2024-03-31T15:15:46.768 INFO:teuthology.orchestra.run.smithi016.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.49031.log.gz 2024-03-31T15:15:46.769 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.47614.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51487.log 2024-03-31T15:15:46.769 INFO:teuthology.orchestra.run.smithi016.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.47614.log.gz 2024-03-31T15:15:46.770 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40909.log/var/log/ceph/ceph-client.admin.47763.log: 2024-03-31T15:15:46.771 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.51487.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.47763.log.gz 2024-03-31T15:15:46.771 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47580.log 2024-03-31T15:15:46.771 INFO:teuthology.orchestra.run.smithi016.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.51487.log.gz 2024-03-31T15:15:46.773 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.40909.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53346.log 2024-03-31T15:15:46.773 INFO:teuthology.orchestra.run.smithi016.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.40909.log.gz 2024-03-31T15:15:46.773 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.47580.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44289.log 2024-03-31T15:15:46.774 INFO:teuthology.orchestra.run.smithi016.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.47580.log.gz 2024-03-31T15:15:46.774 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.53346.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38607.log 2024-03-31T15:15:46.775 INFO:teuthology.orchestra.run.smithi016.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.53346.log.gz 2024-03-31T15:15:46.776 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.44289.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34309.log 2024-03-31T15:15:46.776 INFO:teuthology.orchestra.run.smithi016.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.44289.log.gz 2024-03-31T15:15:46.776 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.38607.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-03-31T15:15:46.777 INFO:teuthology.orchestra.run.smithi016.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.38607.log.gz 2024-03-31T15:15:46.778 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.34309.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51461.log 2024-03-31T15:15:46.778 INFO:teuthology.orchestra.run.smithi016.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.34309.log.gz 2024-03-31T15:15:46.778 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31945.log 2024-03-31T15:15:46.779 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.51461.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50818.log 2024-03-31T15:15:46.780 INFO:teuthology.orchestra.run.smithi016.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.51461.log.gz 2024-03-31T15:15:46.781 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.31945.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39503.log 2024-03-31T15:15:46.781 INFO:teuthology.orchestra.run.smithi016.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.31945.log.gz 2024-03-31T15:15:46.781 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.50818.log: -- /var/log/ceph/ceph-client.admin.43949.log 2024-03-31T15:15:46.782 INFO:teuthology.orchestra.run.smithi016.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.50818.log.gz 2024-03-31T15:15:46.783 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.39503.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34999.log 2024-03-31T15:15:46.783 INFO:teuthology.orchestra.run.smithi016.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.39503.log.gz 2024-03-31T15:15:46.784 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.43949.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41039.log 2024-03-31T15:15:46.784 INFO:teuthology.orchestra.run.smithi016.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.43949.log.gz 2024-03-31T15:15:46.785 INFO:teuthology.orchestra.run.smithi016.stderr:gzip/var/log/ceph/ceph-client.admin.34999.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.35077.log 2024-03-31T15:15:46.786 INFO:teuthology.orchestra.run.smithi016.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.34999.log.gz 2024-03-31T15:15:46.786 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.41039.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45327.log 2024-03-31T15:15:46.786 INFO:teuthology.orchestra.run.smithi016.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.41039.log.gz 2024-03-31T15:15:46.787 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.35077.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36156.log 2024-03-31T15:15:46.787 INFO:teuthology.orchestra.run.smithi016.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.35077.log.gz 2024-03-31T15:15:46.788 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.45327.log: /var/log/ceph/ceph-client.admin.42273.log 2024-03-31T15:15:46.789 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.36156.log: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.45327.log.gz 2024-03-31T15:15:46.790 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39330.log 2024-03-31T15:15:46.790 INFO:teuthology.orchestra.run.smithi016.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.36156.log.gz 2024-03-31T15:15:46.795 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.42273.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44317.log 2024-03-31T15:15:46.795 INFO:teuthology.orchestra.run.smithi016.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.42273.log.gz 2024-03-31T15:15:46.795 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.39330.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37758.log 2024-03-31T15:15:46.795 INFO:teuthology.orchestra.run.smithi016.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.39330.log.gz 2024-03-31T15:15:46.795 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.44317.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53588.log 2024-03-31T15:15:46.795 INFO:teuthology.orchestra.run.smithi016.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.44317.log.gz 2024-03-31T15:15:46.795 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.37758.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44694.log 2024-03-31T15:15:46.795 INFO:teuthology.orchestra.run.smithi016.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.37758.log.gz 2024-03-31T15:15:46.795 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.53588.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48462.log 2024-03-31T15:15:46.796 INFO:teuthology.orchestra.run.smithi016.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.53588.log.gz 2024-03-31T15:15:46.796 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.44694.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38381.log 2024-03-31T15:15:46.796 INFO:teuthology.orchestra.run.smithi016.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.44694.log.gz 2024-03-31T15:15:46.797 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.48462.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.48462.log.gz 2024-03-31T15:15:46.797 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34283.log 2024-03-31T15:15:46.798 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.38381.log: /var/log/ceph/ceph-client.admin.45205.log 2024-03-31T15:15:46.798 INFO:teuthology.orchestra.run.smithi016.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.38381.log.gz 2024-03-31T15:15:46.799 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.34283.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31705.log 2024-03-31T15:15:46.799 INFO:teuthology.orchestra.run.smithi016.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.34283.log.gz 2024-03-31T15:15:46.800 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.45205.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51078.log 2024-03-31T15:15:46.800 INFO:teuthology.orchestra.run.smithi016.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.45205.log.gz 2024-03-31T15:15:46.801 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.31705.log: /var/log/ceph/ceph-client.admin.41948.log 2024-03-31T15:15:46.801 INFO:teuthology.orchestra.run.smithi016.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.31705.log.gz 2024-03-31T15:15:46.802 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.51078.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41247.log 2024-03-31T15:15:46.803 INFO:teuthology.orchestra.run.smithi016.stderr: 91.3% -- replaced with /var/log/ceph/ceph-client.admin.51078.log.gz 2024-03-31T15:15:46.803 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.41948.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36022.log 2024-03-31T15:15:46.804 INFO:teuthology.orchestra.run.smithi016.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.41948.log.gz 2024-03-31T15:15:46.805 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.41247.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40604.log 2024-03-31T15:15:46.805 INFO:teuthology.orchestra.run.smithi016.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.41247.log.gz 2024-03-31T15:15:46.806 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.36022.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47141.log 2024-03-31T15:15:46.806 INFO:teuthology.orchestra.run.smithi016.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.36022.log.gz 2024-03-31T15:15:46.807 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.40604.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47907.log 2024-03-31T15:15:46.807 INFO:teuthology.orchestra.run.smithi016.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.40604.log.gz 2024-03-31T15:15:46.807 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48822.log 2024-03-31T15:15:46.808 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.47141.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.47141.log.gz 2024-03-31T15:15:46.809 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.47907.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38279.log 2024-03-31T15:15:46.809 INFO:teuthology.orchestra.run.smithi016.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.47907.log.gz 2024-03-31T15:15:46.810 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.48822.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48540.log 2024-03-31T15:15:46.810 INFO:teuthology.orchestra.run.smithi016.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.48822.log.gz 2024-03-31T15:15:46.810 INFO:teuthology.orchestra.run.smithi016.stderr:gzip/var/log/ceph/ceph-client.admin.38279.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.42165.log 2024-03-31T15:15:46.812 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.48540.log: 91.4% -- replaced with /var/log/ceph/ceph-client.admin.38279.log.gz 2024-03-31T15:15:46.812 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35311.log 2024-03-31T15:15:46.812 INFO:teuthology.orchestra.run.smithi016.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.48540.log.gz 2024-03-31T15:15:46.813 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.42165.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31997.log 2024-03-31T15:15:46.813 INFO:teuthology.orchestra.run.smithi016.stderr: 90.1% -- replaced with /var/log/ceph/ceph-client.admin.42165.log.gz 2024-03-31T15:15:46.814 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.35311.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50044.log 2024-03-31T15:15:46.814 INFO:teuthology.orchestra.run.smithi016.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.35311.log.gz 2024-03-31T15:15:46.815 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.31997.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.52262.log 2024-03-31T15:15:46.815 INFO:teuthology.orchestra.run.smithi016.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.31997.log.gz 2024-03-31T15:15:46.816 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.50044.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47502.log 2024-03-31T15:15:46.816 INFO:teuthology.orchestra.run.smithi016.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.50044.log.gz 2024-03-31T15:15:46.817 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.0.52262.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48796.log 2024-03-31T15:15:46.819 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.47502.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35207.log 2024-03-31T15:15:46.819 INFO:teuthology.orchestra.run.smithi016.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.47502.log.gz 2024-03-31T15:15:46.820 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.48796.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.48796.log.gz 2024-03-31T15:15:46.820 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.35207.log: 93.6% -- replaced with /var/log/ceph/ceph-client.0.52262.log.gz 2024-03-31T15:15:46.821 INFO:teuthology.orchestra.run.smithi016.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.35207.log.gz 2024-03-31T15:15:46.821 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34654.log 2024-03-31T15:15:46.822 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53536.log 2024-03-31T15:15:46.823 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.34654.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39016.log 2024-03-31T15:15:46.823 INFO:teuthology.orchestra.run.smithi016.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.34654.log.gz 2024-03-31T15:15:46.824 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.53536.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31574.log 2024-03-31T15:15:46.825 INFO:teuthology.orchestra.run.smithi016.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.53536.log.gz 2024-03-31T15:15:46.825 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.39016.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40440.log 2024-03-31T15:15:46.826 INFO:teuthology.orchestra.run.smithi016.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.39016.log.gz 2024-03-31T15:15:46.827 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.31574.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52036.log 2024-03-31T15:15:46.827 INFO:teuthology.orchestra.run.smithi016.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.31574.log.gz 2024-03-31T15:15:46.828 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.40440.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50539.log 2024-03-31T15:15:46.828 INFO:teuthology.orchestra.run.smithi016.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.40440.log.gz 2024-03-31T15:15:46.828 INFO:teuthology.orchestra.run.smithi016.stderr:gzip/var/log/ceph/ceph-client.admin.52036.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.34444.log 2024-03-31T15:15:46.829 INFO:teuthology.orchestra.run.smithi016.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.52036.log.gz 2024-03-31T15:15:46.830 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.50539.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35721.log 2024-03-31T15:15:46.830 INFO:teuthology.orchestra.run.smithi016.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.50539.log.gz 2024-03-31T15:15:46.831 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.34444.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32985.log 2024-03-31T15:15:46.831 INFO:teuthology.orchestra.run.smithi016.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.34444.log.gz 2024-03-31T15:15:46.832 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.35721.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51518.log 2024-03-31T15:15:46.833 INFO:teuthology.orchestra.run.smithi016.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.35721.log.gz 2024-03-31T15:15:46.833 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.32985.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39250.log 2024-03-31T15:15:46.834 INFO:teuthology.orchestra.run.smithi016.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.32985.log.gz 2024-03-31T15:15:46.835 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.51518.log: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.51518.log.gz 2024-03-31T15:15:46.836 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.39250.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.39250.log.gz 2024-03-31T15:15:46.836 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44887.log 2024-03-31T15:15:46.837 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44540.log 2024-03-31T15:15:46.839 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.44887.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51252.log 2024-03-31T15:15:46.839 INFO:teuthology.orchestra.run.smithi016.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.44887.log.gz 2024-03-31T15:15:46.840 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.44540.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-03-31T15:15:46.840 INFO:teuthology.orchestra.run.smithi016.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.44540.log.gz 2024-03-31T15:15:46.840 INFO:teuthology.orchestra.run.smithi016.stderr:gzip/var/log/ceph/ceph-client.admin.51252.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.48409.log 2024-03-31T15:15:46.841 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-osd.1.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.51252.log.gz 2024-03-31T15:15:46.841 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53484.log 2024-03-31T15:15:46.843 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.48409.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45533.log 2024-03-31T15:15:46.843 INFO:teuthology.orchestra.run.smithi016.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.48409.log.gz 2024-03-31T15:15:46.844 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.53484.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.53484.log.gz 2024-03-31T15:15:46.846 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.45533.log: 91.3% -- replaced with /var/log/ceph/ceph-client.admin.45533.log.gz 2024-03-31T15:15:46.846 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48874.log 2024-03-31T15:15:46.847 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.46616.log 2024-03-31T15:15:46.849 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.48874.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41221.log 2024-03-31T15:15:46.849 INFO:teuthology.orchestra.run.smithi016.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.48874.log.gz 2024-03-31T15:15:46.851 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.0.46616.log: /var/log/ceph/ceph-client.admin.41221.log: 92.3% -- replaced with /var/log/ceph/ceph-client.0.46616.log.gz 2024-03-31T15:15:46.851 INFO:teuthology.orchestra.run.smithi016.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.41221.log.gz 2024-03-31T15:15:46.851 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46263.log 2024-03-31T15:15:46.852 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43642.log 2024-03-31T15:15:46.853 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.46263.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39659.log 2024-03-31T15:15:46.853 INFO:teuthology.orchestra.run.smithi016.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.46263.log.gz 2024-03-31T15:15:46.854 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.43642.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.43642.log.gz 2024-03-31T15:15:46.856 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.39659.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.39659.log.gz 2024-03-31T15:15:46.856 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50617.log 2024-03-31T15:15:46.857 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44618.log 2024-03-31T15:15:46.859 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.50617.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32535.log 2024-03-31T15:15:46.859 INFO:teuthology.orchestra.run.smithi016.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.50617.log.gz 2024-03-31T15:15:46.860 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.44618.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.44618.log.gz 2024-03-31T15:15:46.861 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.32535.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.32535.log.gz 2024-03-31T15:15:46.862 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37074.log 2024-03-31T15:15:46.863 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49992.log 2024-03-31T15:15:46.864 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.37074.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42142.log 2024-03-31T15:15:46.864 INFO:teuthology.orchestra.run.smithi016.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.37074.log.gz 2024-03-31T15:15:46.865 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.49992.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.49992.log.gz 2024-03-31T15:15:46.866 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.42142.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.42142.log.gz 2024-03-31T15:15:46.867 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44835.log 2024-03-31T15:15:46.868 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45169.log 2024-03-31T15:15:46.869 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.44835.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31759.log 2024-03-31T15:15:46.869 INFO:teuthology.orchestra.run.smithi016.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.44835.log.gz 2024-03-31T15:15:46.871 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.45169.log: /var/log/ceph/ceph-client.admin.31759.log: 91.5% -- replaced with /var/log/ceph/ceph-client.admin.45169.log.gz 2024-03-31T15:15:46.871 INFO:teuthology.orchestra.run.smithi016.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.31759.log.gz 2024-03-31T15:15:46.872 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49496.log 2024-03-31T15:15:46.873 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32777.log 2024-03-31T15:15:46.875 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.49496.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34863.log 2024-03-31T15:15:46.875 INFO:teuthology.orchestra.run.smithi016.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.49496.log.gz 2024-03-31T15:15:46.875 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.32777.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.32777.log.gz 2024-03-31T15:15:46.876 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.34863.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52666.log 2024-03-31T15:15:46.877 INFO:teuthology.orchestra.run.smithi016.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.34863.log.gz 2024-03-31T15:15:46.878 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41124.log 2024-03-31T15:15:46.879 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.52666.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50762.log 2024-03-31T15:15:46.880 INFO:teuthology.orchestra.run.smithi016.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.52666.log.gz 2024-03-31T15:15:46.880 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.41124.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.41124.log.gz 2024-03-31T15:15:46.880 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52486.log 2024-03-31T15:15:46.883 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.50762.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.50762.log.gz 2024-03-31T15:15:46.883 INFO:teuthology.orchestra.run.smithi016.stderr:gzip/var/log/ceph/ceph-client.admin.52486.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.45377.log 2024-03-31T15:15:46.883 INFO:teuthology.orchestra.run.smithi016.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.52486.log.gz 2024-03-31T15:15:46.884 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51304.log 2024-03-31T15:15:46.885 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.45377.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41382.log 2024-03-31T15:15:46.886 INFO:teuthology.orchestra.run.smithi016.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.45377.log.gz 2024-03-31T15:15:46.886 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.51304.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.51304.log.gz 2024-03-31T15:15:46.887 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.41382.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.41382.log.gz 2024-03-31T15:15:46.888 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39447.log 2024-03-31T15:15:46.889 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45129.log 2024-03-31T15:15:46.890 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.39447.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37100.log 2024-03-31T15:15:46.890 INFO:teuthology.orchestra.run.smithi016.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.39447.log.gz 2024-03-31T15:15:46.891 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.45129.log: 90.5% -- replaced with /var/log/ceph/ceph-client.admin.45129.log.gz 2024-03-31T15:15:46.893 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.37100.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.37100.log.gz 2024-03-31T15:15:46.893 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38789.log 2024-03-31T15:15:46.894 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40961.log 2024-03-31T15:15:46.896 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.38789.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33011.log 2024-03-31T15:15:46.896 INFO:teuthology.orchestra.run.smithi016.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.38789.log.gz 2024-03-31T15:15:46.897 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.40961.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.40961.log.gz 2024-03-31T15:15:46.898 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.33011.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.33011.log.gz 2024-03-31T15:15:46.898 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52512.log 2024-03-31T15:15:46.900 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52355.log 2024-03-31T15:15:46.901 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.52512.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32527.log 2024-03-31T15:15:46.901 INFO:teuthology.orchestra.run.smithi016.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.52512.log.gz 2024-03-31T15:15:46.902 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.52355.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.52355.log.gz 2024-03-31T15:15:46.903 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.32527.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.32527.log.gz 2024-03-31T15:15:46.903 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43011.log 2024-03-31T15:15:46.905 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37918.log 2024-03-31T15:15:46.906 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.43011.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48172.log 2024-03-31T15:15:46.906 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.37918.log: 91.3% -- replaced with /var/log/ceph/ceph-client.admin.43011.log.gz 2024-03-31T15:15:46.907 INFO:teuthology.orchestra.run.smithi016.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.37918.log.gz 2024-03-31T15:15:46.909 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.48172.log: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.48172.log.gz 2024-03-31T15:15:46.909 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40025.log 2024-03-31T15:15:46.910 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48566.log 2024-03-31T15:15:46.912 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.40025.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54045.log 2024-03-31T15:15:46.912 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.48566.log: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.40025.log.gz 2024-03-31T15:15:46.913 INFO:teuthology.orchestra.run.smithi016.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.48566.log.gz 2024-03-31T15:15:46.914 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.54045.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.54045.log.gz 2024-03-31T15:15:46.914 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44592.log 2024-03-31T15:15:46.915 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34600.log 2024-03-31T15:15:46.917 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.44592.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51409.log 2024-03-31T15:15:46.917 INFO:teuthology.orchestra.run.smithi016.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.44592.log.gz 2024-03-31T15:15:46.918 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.34600.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.34600.log.gz 2024-03-31T15:15:46.919 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.51409.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.51409.log.gz 2024-03-31T15:15:46.919 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53562.log 2024-03-31T15:15:46.920 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34067.log 2024-03-31T15:15:46.922 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.53562.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33885.log 2024-03-31T15:15:46.922 INFO:teuthology.orchestra.run.smithi016.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.53562.log.gz 2024-03-31T15:15:46.923 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.34067.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.34067.log.gz 2024-03-31T15:15:46.924 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.33885.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.33885.log.gz 2024-03-31T15:15:46.924 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35970.log 2024-03-31T15:15:46.925 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31731.log 2024-03-31T15:15:46.927 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.35970.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48435.log 2024-03-31T15:15:46.927 INFO:teuthology.orchestra.run.smithi016.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.35970.log.gz 2024-03-31T15:15:46.928 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.31731.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.31731.log.gz 2024-03-31T15:15:46.929 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.48435.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.48435.log.gz 2024-03-31T15:15:46.930 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50675.log 2024-03-31T15:15:46.931 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51278.log 2024-03-31T15:15:46.932 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.50675.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53406.log 2024-03-31T15:15:46.932 INFO:teuthology.orchestra.run.smithi016.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.50675.log.gz 2024-03-31T15:15:46.933 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.51278.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.51278.log.gz 2024-03-31T15:15:46.933 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52329.log 2024-03-31T15:15:46.935 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.53406.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46071.log 2024-03-31T15:15:46.935 INFO:teuthology.orchestra.run.smithi016.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.53406.log.gz 2024-03-31T15:15:46.936 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.52329.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.52329.log.gz 2024-03-31T15:15:46.936 INFO:teuthology.orchestra.run.smithi089.stderr: 91.6% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-03-31T15:15:46.937 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.46071.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.46071.log.gz 2024-03-31T15:15:46.937 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33597.log 2024-03-31T15:15:46.939 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43061.log 2024-03-31T15:15:46.940 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.33597.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38841.log 2024-03-31T15:15:46.940 INFO:teuthology.orchestra.run.smithi016.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.33597.log.gz 2024-03-31T15:15:46.942 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.43061.log: /var/log/ceph/ceph-client.admin.38841.log: 91.4% -- replaced with /var/log/ceph/ceph-client.admin.43061.log.gz 2024-03-31T15:15:46.942 INFO:teuthology.orchestra.run.smithi016.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.38841.log.gz 2024-03-31T15:15:46.943 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43774.log 2024-03-31T15:15:46.944 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40518.log 2024-03-31T15:15:46.945 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.43774.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34626.log 2024-03-31T15:15:46.946 INFO:teuthology.orchestra.run.smithi016.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.43774.log.gz 2024-03-31T15:15:46.946 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.40518.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.40518.log.gz 2024-03-31T15:15:46.947 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.34626.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.34626.log.gz 2024-03-31T15:15:46.948 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36182.log 2024-03-31T15:15:46.949 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46842.log 2024-03-31T15:15:46.950 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.36182.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38520.log 2024-03-31T15:15:46.950 INFO:teuthology.orchestra.run.smithi016.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.36182.log.gz 2024-03-31T15:15:46.952 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.46842.log: /var/log/ceph/ceph-client.admin.38520.log: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.46842.log.gz 2024-03-31T15:15:46.952 INFO:teuthology.orchestra.run.smithi016.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.38520.log.gz 2024-03-31T15:15:46.953 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36466.log 2024-03-31T15:15:46.954 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52713.log 2024-03-31T15:15:46.956 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.36466.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44119.log 2024-03-31T15:15:46.956 INFO:teuthology.orchestra.run.smithi016.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.36466.log.gz 2024-03-31T15:15:46.957 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.52713.log: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.52713.log.gz 2024-03-31T15:15:46.958 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.44119.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.44119.log.gz 2024-03-31T15:15:46.958 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40630.log 2024-03-31T15:15:46.959 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40656.log 2024-03-31T15:15:46.961 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.40630.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35467.log 2024-03-31T15:15:46.961 INFO:teuthology.orchestra.run.smithi016.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.40630.log.gz 2024-03-31T15:15:46.962 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.40656.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.40656.log.gz 2024-03-31T15:15:46.963 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.35467.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.35467.log.gz 2024-03-31T15:15:46.963 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-03-31T15:15:46.965 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36649.log 2024-03-31T15:15:46.966 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37536.log 2024-03-31T15:15:46.968 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.36649.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.36649.log.gz 2024-03-31T15:15:46.968 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.37536.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.37536.log.gz 2024-03-31T15:15:46.969 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36545.log 2024-03-31T15:15:46.969 INFO:teuthology.orchestra.run.smithi016.stderr: 86.9% -- replaced with /var/log/ceph/ceph.log.gz 2024-03-31T15:15:46.970 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52688.log 2024-03-31T15:15:46.971 INFO:teuthology.orchestra.run.smithi089.stderr: 91.5% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-03-31T15:15:46.971 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.36545.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44406.log 2024-03-31T15:15:46.972 INFO:teuthology.orchestra.run.smithi016.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.36545.log.gz 2024-03-31T15:15:46.972 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.52688.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.52688.log.gz 2024-03-31T15:15:46.973 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.44406.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.44406.log.gz 2024-03-31T15:15:46.974 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48145.log 2024-03-31T15:15:46.975 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31919.log 2024-03-31T15:15:46.976 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.48145.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40682.log 2024-03-31T15:15:46.977 INFO:teuthology.orchestra.run.smithi016.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.48145.log.gz 2024-03-31T15:15:46.977 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.31919.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.31919.log.gz 2024-03-31T15:15:46.979 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.40682.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.40682.log.gz 2024-03-31T15:15:46.979 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45993.log 2024-03-31T15:15:46.981 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34732.log 2024-03-31T15:15:46.982 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.45993.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46332.log 2024-03-31T15:15:46.982 INFO:teuthology.orchestra.run.smithi016.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.45993.log.gz 2024-03-31T15:15:46.984 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.34732.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.34732.log.gz 2024-03-31T15:15:46.984 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.46332.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44514.log 2024-03-31T15:15:46.985 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52538.log 2024-03-31T15:15:46.986 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.44514.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43826.log 2024-03-31T15:15:46.987 INFO:teuthology.orchestra.run.smithi016.stderr: 89.7% 92.5% -- replaced with /var/log/ceph/ceph-client.admin.46332.log.gz 2024-03-31T15:15:46.987 INFO:teuthology.orchestra.run.smithi016.stderr: -- replaced with /var/log/ceph/ceph-client.admin.44514.log.gz 2024-03-31T15:15:46.988 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.52538.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.52538.log.gz 2024-03-31T15:15:46.989 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.43826.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.43826.log.gz 2024-03-31T15:15:46.990 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32121.log 2024-03-31T15:15:46.991 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37788.log 2024-03-31T15:15:46.993 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.32121.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54097.log 2024-03-31T15:15:46.993 INFO:teuthology.orchestra.run.smithi016.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.32121.log.gz 2024-03-31T15:15:46.994 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.37788.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.37788.log.gz 2024-03-31T15:15:46.995 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.54097.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.54097.log.gz 2024-03-31T15:15:46.995 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52011.log 2024-03-31T15:15:46.996 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47790.log 2024-03-31T15:15:46.998 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.52011.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47554.log 2024-03-31T15:15:46.998 INFO:teuthology.orchestra.run.smithi016.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.52011.log.gz 2024-03-31T15:15:46.999 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.47790.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.47790.log.gz 2024-03-31T15:15:47.000 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.47554.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.47554.log.gz 2024-03-31T15:15:47.000 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37970.log 2024-03-31T15:15:47.002 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32336.log 2024-03-31T15:15:47.003 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.37970.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45075.log 2024-03-31T15:15:47.004 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.32336.log: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.37970.log.gz 2024-03-31T15:15:47.004 INFO:teuthology.orchestra.run.smithi016.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.32336.log.gz 2024-03-31T15:15:47.004 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51838.log 2024-03-31T15:15:47.007 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.45075.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.45075.log.gz 2024-03-31T15:15:47.007 INFO:teuthology.orchestra.run.smithi016.stderr:gzip/var/log/ceph/ceph-client.admin.51838.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.46951.log 2024-03-31T15:15:47.007 INFO:teuthology.orchestra.run.smithi016.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.51838.log.gz 2024-03-31T15:15:47.008 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34758.log 2024-03-31T15:15:47.009 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.46951.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32517.log 2024-03-31T15:15:47.009 INFO:teuthology.orchestra.run.smithi016.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.46951.log.gz 2024-03-31T15:15:47.010 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.34758.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.34758.log.gz 2024-03-31T15:15:47.011 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.32517.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.32517.log.gz 2024-03-31T15:15:47.012 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51383.log 2024-03-31T15:15:47.013 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40708.log 2024-03-31T15:15:47.014 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.51383.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37562.log 2024-03-31T15:15:47.014 INFO:teuthology.orchestra.run.smithi016.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.51383.log.gz 2024-03-31T15:15:47.016 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.40708.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.40708.log.gz 2024-03-31T15:15:47.016 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.37562.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40544.log 2024-03-31T15:15:47.017 INFO:teuthology.orchestra.run.smithi016.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.37562.log.gz 2024-03-31T15:15:47.018 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40760.log 2024-03-31T15:15:47.020 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.40544.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32129.log 2024-03-31T15:15:47.020 INFO:teuthology.orchestra.run.smithi016.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.40544.log.gz 2024-03-31T15:15:47.020 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.40760.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.40760.log.gz 2024-03-31T15:15:47.020 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46816.log 2024-03-31T15:15:47.021 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.32129.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44966.log 2024-03-31T15:15:47.022 INFO:teuthology.orchestra.run.smithi016.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.32129.log.gz 2024-03-31T15:15:47.023 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.46816.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41443.log 2024-03-31T15:15:47.023 INFO:teuthology.orchestra.run.smithi016.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.46816.log.gz 2024-03-31T15:15:47.024 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.44966.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.44966.log.gz 2024-03-31T15:15:47.026 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.41443.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.41443.log.gz 2024-03-31T15:15:47.026 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46359.log 2024-03-31T15:15:47.027 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32725.log 2024-03-31T15:15:47.030 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.46359.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41156.log 2024-03-31T15:15:47.030 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.32725.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.32725.log.gz 2024-03-31T15:15:47.030 INFO:teuthology.orchestra.run.smithi016.stderr: 92.4% -- replaced with /var/log/ceph/ceph-client.admin.46359.log.gz 2024-03-31T15:15:47.030 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.41156.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.41156.log.gz 2024-03-31T15:15:47.031 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40408.log 2024-03-31T15:15:47.032 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54386.log 2024-03-31T15:15:47.033 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.40408.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.40408.log.gz 2024-03-31T15:15:47.035 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.54386.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.54386.log.gz 2024-03-31T15:15:47.035 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32124.log 2024-03-31T15:15:47.036 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42621.log 2024-03-31T15:15:47.037 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.32124.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51026.log 2024-03-31T15:15:47.038 INFO:teuthology.orchestra.run.smithi016.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.32124.log.gz 2024-03-31T15:15:47.039 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.42621.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36808.log 2024-03-31T15:15:47.039 INFO:teuthology.orchestra.run.smithi016.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.42621.log.gz 2024-03-31T15:15:47.040 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.51026.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52074.log 2024-03-31T15:15:47.042 INFO:teuthology.orchestra.run.smithi016.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.51026.log.gz 2024-03-31T15:15:47.042 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.36808.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46098.log 2024-03-31T15:15:47.042 INFO:teuthology.orchestra.run.smithi016.stderr: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.36808.log.gz 2024-03-31T15:15:47.042 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.52074.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.52074.log.gz 2024-03-31T15:15:47.043 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.46098.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.46098.log.gz 2024-03-31T15:15:47.044 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50948.log 2024-03-31T15:15:47.045 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38938.log 2024-03-31T15:15:47.047 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.50948.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52381.log 2024-03-31T15:15:47.047 INFO:teuthology.orchestra.run.smithi016.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.50948.log.gz 2024-03-31T15:15:47.048 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.38938.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.38938.log.gz 2024-03-31T15:15:47.048 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47646.log 2024-03-31T15:15:47.050 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.52381.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45106.log 2024-03-31T15:15:47.050 INFO:teuthology.orchestra.run.smithi016.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.52381.log.gz 2024-03-31T15:15:47.051 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.47646.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45636.log 2024-03-31T15:15:47.052 INFO:teuthology.orchestra.run.smithi016.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.47646.log.gz 2024-03-31T15:15:47.052 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.45106.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.45106.log.gz 2024-03-31T15:15:47.053 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.45636.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.45636.log.gz 2024-03-31T15:15:47.054 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38020.log 2024-03-31T15:15:47.055 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49064.log 2024-03-31T15:15:47.056 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.38020.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50330.log 2024-03-31T15:15:47.057 INFO:teuthology.orchestra.run.smithi016.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.38020.log.gz 2024-03-31T15:15:47.057 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.49064.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.49064.log.gz 2024-03-31T15:15:47.058 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.50330.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.50330.log.gz 2024-03-31T15:15:47.059 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48093.log 2024-03-31T15:15:47.060 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33649.log 2024-03-31T15:15:47.061 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.48093.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.48093.log.gz 2024-03-31T15:15:47.062 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.33649.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.33649.log.gz 2024-03-31T15:15:47.063 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43114.log 2024-03-31T15:15:47.065 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50096.log 2024-03-31T15:15:47.066 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.43114.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50645.log 2024-03-31T15:15:47.066 INFO:teuthology.orchestra.run.smithi089.stderr: 93.6% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-03-31T15:15:47.067 INFO:teuthology.orchestra.run.smithi016.stderr: 91.4% -- replaced with /var/log/ceph/ceph-client.admin.43114.log.gz 2024-03-31T15:15:47.067 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.50096.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.50096.log.gz 2024-03-31T15:15:47.068 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.50645.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.50645.log.gz 2024-03-31T15:15:47.068 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33623.log 2024-03-31T15:15:47.069 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35879.log 2024-03-31T15:15:47.071 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.33623.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42807.log 2024-03-31T15:15:47.071 INFO:teuthology.orchestra.run.smithi016.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.33623.log.gz 2024-03-31T15:15:47.072 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.35879.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.35879.log.gz 2024-03-31T15:15:47.073 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.42807.log: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.42807.log.gz 2024-03-31T15:15:47.074 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48675.log 2024-03-31T15:15:47.075 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36440.log 2024-03-31T15:15:47.076 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.48675.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54410.log 2024-03-31T15:15:47.077 INFO:teuthology.orchestra.run.smithi016.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.48675.log.gz 2024-03-31T15:15:47.078 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.36440.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.36440.log.gz 2024-03-31T15:15:47.079 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.54410.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.54410.log.gz 2024-03-31T15:15:47.079 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46389.log 2024-03-31T15:15:47.080 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45301.log 2024-03-31T15:15:47.082 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.46389.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47450.log 2024-03-31T15:15:47.082 INFO:teuthology.orchestra.run.smithi016.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.46389.log.gz 2024-03-31T15:15:47.082 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.45301.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.45301.log.gz 2024-03-31T15:15:47.084 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.47450.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.47450.log.gz 2024-03-31T15:15:47.084 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53791.log 2024-03-31T15:15:47.085 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40492.log 2024-03-31T15:15:47.087 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.53791.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53967.log 2024-03-31T15:15:47.087 INFO:teuthology.orchestra.run.smithi016.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.53791.log.gz 2024-03-31T15:15:47.088 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.40492.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.40492.log.gz 2024-03-31T15:15:47.089 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.53967.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.53967.log.gz 2024-03-31T15:15:47.089 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51707.log 2024-03-31T15:15:47.090 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47219.log 2024-03-31T15:15:47.092 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.51707.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48597.log 2024-03-31T15:15:47.092 INFO:teuthology.orchestra.run.smithi016.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.51707.log.gz 2024-03-31T15:15:47.093 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.47219.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.47219.log.gz 2024-03-31T15:15:47.094 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.48597.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.48597.log.gz 2024-03-31T15:15:47.094 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41629.log 2024-03-31T15:15:47.096 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44210.log 2024-03-31T15:15:47.097 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.41629.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37615.log 2024-03-31T15:15:47.098 INFO:teuthology.orchestra.run.smithi016.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.41629.log.gz 2024-03-31T15:15:47.098 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.44210.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.44210.log.gz 2024-03-31T15:15:47.099 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.37615.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.37615.log.gz 2024-03-31T15:15:47.100 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.48731.log 2024-03-31T15:15:47.101 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49940.log 2024-03-31T15:15:47.104 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.0.48731.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50844.log 2024-03-31T15:15:47.104 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.49940.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.49940.log.gz 2024-03-31T15:15:47.104 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.50844.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.50844.log.gz 2024-03-31T15:15:47.105 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38763.log 2024-03-31T15:15:47.109 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51681.log 2024-03-31T15:15:47.109 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.38763.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.38763.log.gz 2024-03-31T15:15:47.109 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.51681.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.51681.log.gz 2024-03-31T15:15:47.109 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40282.log 2024-03-31T15:15:47.110 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54123.log 2024-03-31T15:15:47.111 INFO:teuthology.orchestra.run.smithi016.stderr: 94.5% -- replaced with /var/log/ceph/ceph-client.0.48731.log.gz 2024-03-31T15:15:47.112 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.40282.log: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.40282.log.gz 2024-03-31T15:15:47.112 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.54123.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40812.log 2024-03-31T15:15:47.113 INFO:teuthology.orchestra.run.smithi016.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.54123.log.gz 2024-03-31T15:15:47.114 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41922.log 2024-03-31T15:15:47.115 INFO:teuthology.orchestra.run.smithi089.stderr: 93.8% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-03-31T15:15:47.115 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.40812.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42055.log 2024-03-31T15:15:47.116 INFO:teuthology.orchestra.run.smithi016.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.40812.log.gz 2024-03-31T15:15:47.116 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.41922.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.41922.log.gz 2024-03-31T15:15:47.117 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.42055.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.42055.log.gz 2024-03-31T15:15:47.118 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49273.log 2024-03-31T15:15:47.119 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45269.log 2024-03-31T15:15:47.121 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.49273.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38468.log 2024-03-31T15:15:47.121 INFO:teuthology.orchestra.run.smithi016.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.49273.log.gz 2024-03-31T15:15:47.121 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.45269.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.45269.log.gz 2024-03-31T15:15:47.123 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.38468.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.38468.log.gz 2024-03-31T15:15:47.123 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44053.log 2024-03-31T15:15:47.124 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43190.log 2024-03-31T15:15:47.126 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.44053.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32319.log 2024-03-31T15:15:47.126 INFO:teuthology.orchestra.run.smithi016.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.44053.log.gz 2024-03-31T15:15:47.127 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.43190.log: /var/log/ceph/ceph-client.admin.32319.log: 91.3% -- replaced with /var/log/ceph/ceph-client.admin.43190.log.gz 2024-03-31T15:15:47.128 INFO:teuthology.orchestra.run.smithi016.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.32319.log.gz 2024-03-31T15:15:47.128 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33911.log 2024-03-31T15:15:47.129 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39042.log 2024-03-31T15:15:47.131 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.33911.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33701.log 2024-03-31T15:15:47.131 INFO:teuthology.orchestra.run.smithi016.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.33911.log.gz 2024-03-31T15:15:47.132 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.39042.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.39042.log.gz 2024-03-31T15:15:47.133 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.33701.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.33701.log.gz 2024-03-31T15:15:47.133 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35155.log 2024-03-31T15:15:47.134 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32933.log 2024-03-31T15:15:47.136 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.35155.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47003.log 2024-03-31T15:15:47.136 INFO:teuthology.orchestra.run.smithi016.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.35155.log.gz 2024-03-31T15:15:47.137 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.32933.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.32933.log.gz 2024-03-31T15:15:47.138 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.47003.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.47003.log.gz 2024-03-31T15:15:47.139 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39421.log 2024-03-31T15:15:47.139 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43266.log 2024-03-31T15:15:47.141 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.39421.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34418.log 2024-03-31T15:15:47.142 INFO:teuthology.orchestra.run.smithi016.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.39421.log.gz 2024-03-31T15:15:47.142 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.43266.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.43266.log.gz 2024-03-31T15:15:47.143 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.34418.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.34418.log.gz 2024-03-31T15:15:47.144 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49005.log 2024-03-31T15:15:47.145 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34335.log 2024-03-31T15:15:47.146 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.49005.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48953.log 2024-03-31T15:15:47.147 INFO:teuthology.orchestra.run.smithi016.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.49005.log.gz 2024-03-31T15:15:47.147 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.34335.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.34335.log.gz 2024-03-31T15:15:47.148 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.48953.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.48953.log.gz 2024-03-31T15:15:47.149 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48650.log 2024-03-31T15:15:47.150 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47844.log 2024-03-31T15:15:47.152 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.48650.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47989.log 2024-03-31T15:15:47.152 INFO:teuthology.orchestra.run.smithi016.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.48650.log.gz 2024-03-31T15:15:47.153 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.47844.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33170.log 2024-03-31T15:15:47.153 INFO:teuthology.orchestra.run.smithi016.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.47844.log.gz 2024-03-31T15:15:47.154 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.47989.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.47989.log.gz 2024-03-31T15:15:47.155 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.33170.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36130.log 2024-03-31T15:15:47.155 INFO:teuthology.orchestra.run.smithi016.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.33170.log.gz 2024-03-31T15:15:47.156 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38546.log 2024-03-31T15:15:47.157 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.36130.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32544.log 2024-03-31T15:15:47.158 INFO:teuthology.orchestra.run.smithi016.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.36130.log.gz 2024-03-31T15:15:47.158 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.38546.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.38546.log.gz 2024-03-31T15:15:47.160 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.32544.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.32544.log.gz 2024-03-31T15:15:47.160 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31867.log 2024-03-31T15:15:47.161 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32529.log 2024-03-31T15:15:47.163 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.31867.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32326.log 2024-03-31T15:15:47.163 INFO:teuthology.orchestra.run.smithi016.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.31867.log.gz 2024-03-31T15:15:47.163 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.32529.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.32529.log.gz 2024-03-31T15:15:47.164 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.32326.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.32326.log.gz 2024-03-31T15:15:47.165 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36892.log 2024-03-31T15:15:47.166 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47816.log 2024-03-31T15:15:47.168 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.36892.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41091.log 2024-03-31T15:15:47.168 INFO:teuthology.orchestra.run.smithi016.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.36892.log.gz 2024-03-31T15:15:47.169 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.47816.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.36341.log 2024-03-31T15:15:47.169 INFO:teuthology.orchestra.run.smithi016.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.47816.log.gz 2024-03-31T15:15:47.170 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.41091.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.41091.log.gz 2024-03-31T15:15:47.171 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.0.36341.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35363.log 2024-03-31T15:15:47.172 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47422.log 2024-03-31T15:15:47.174 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.35363.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.35363.log.gz 2024-03-31T15:15:47.174 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.47422.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.47422.log.gz 2024-03-31T15:15:47.175 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36786.log 2024-03-31T15:15:47.176 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37412.log 2024-03-31T15:15:47.178 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.36786.log: 93.2% -- replaced with /var/log/ceph/ceph-client.0.36341.log.gz 2024-03-31T15:15:47.178 INFO:teuthology.orchestra.run.smithi016.stderr: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.36786.log.gz 2024-03-31T15:15:47.178 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.37412.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.37412.log.gz 2024-03-31T15:15:47.179 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49099.log 2024-03-31T15:15:47.180 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48848.log 2024-03-31T15:15:47.181 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.49099.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-31T15:15:47.182 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.48848.log: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.49099.log.gz 2024-03-31T15:15:47.183 INFO:teuthology.orchestra.run.smithi016.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.48848.log.gz 2024-03-31T15:15:47.185 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43975.log 2024-03-31T15:15:47.186 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46766.log 2024-03-31T15:15:47.187 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.43975.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43538.log 2024-03-31T15:15:47.187 INFO:teuthology.orchestra.run.smithi016.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.43975.log.gz 2024-03-31T15:15:47.189 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.46766.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47297.log 2024-03-31T15:15:47.189 INFO:teuthology.orchestra.run.smithi016.stderr: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.46766.log.gz 2024-03-31T15:15:47.189 INFO:teuthology.orchestra.run.smithi016.stderr: 89.6% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-31T15:15:47.190 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.47297.log: /var/log/ceph/ceph-client.admin.43538.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.47297.log.gz 2024-03-31T15:15:47.191 INFO:teuthology.orchestra.run.smithi016.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.43538.log.gzgzip 2024-03-31T15:15:47.191 INFO:teuthology.orchestra.run.smithi016.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.37327.log 2024-03-31T15:15:47.192 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.c.log 2024-03-31T15:15:47.194 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.37327.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38130.log 2024-03-31T15:15:47.194 INFO:teuthology.orchestra.run.smithi016.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.37327.log.gz 2024-03-31T15:15:47.196 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-mds.c.log: /var/log/ceph/ceph-client.admin.38130.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.38130.log.gz 2024-03-31T15:15:47.196 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37814.log 2024-03-31T15:15:47.199 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.37814.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.37814.log.gz 2024-03-31T15:15:47.199 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43722.log 2024-03-31T15:15:47.202 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45890.log 2024-03-31T15:15:47.202 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.43722.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.43722.log.gz 2024-03-31T15:15:47.203 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.45890.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.45890.log.gz 2024-03-31T15:15:47.203 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44913.log 2024-03-31T15:15:47.204 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34093.log 2024-03-31T15:15:47.206 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.44913.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.44913.log.gz 2024-03-31T15:15:47.207 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.34093.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.34093.log.gz 2024-03-31T15:15:47.207 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52644.log 2024-03-31T15:15:47.208 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39386.log 2024-03-31T15:15:47.210 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.52644.log: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.52644.log.gz 2024-03-31T15:15:47.211 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.39386.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.39386.log.gz 2024-03-31T15:15:47.211 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49578.log 2024-03-31T15:15:47.212 INFO:teuthology.orchestra.run.smithi089.stderr: 93.8% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2024-03-31T15:15:47.213 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35259.log 2024-03-31T15:15:47.216 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.49578.log: /var/log/ceph/ceph-client.admin.35259.log: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.49578.log.gz 2024-03-31T15:15:47.216 INFO:teuthology.orchestra.run.smithi016.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.35259.log.gz 2024-03-31T15:15:47.216 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.a.log 2024-03-31T15:15:47.217 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53380.log 2024-03-31T15:15:47.220 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-mds.a.log: /var/log/ceph/ceph-client.admin.53380.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.53380.log.gz 2024-03-31T15:15:47.220 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31509.log 2024-03-31T15:15:47.222 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.31509.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.31509.log.gz 2024-03-31T15:15:47.223 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43427.log 2024-03-31T15:15:47.224 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46150.log 2024-03-31T15:15:47.227 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.43427.log: gzip/var/log/ceph/ceph-client.admin.46150.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.43427.log.gz 2024-03-31T15:15:47.227 INFO:teuthology.orchestra.run.smithi016.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.46150.log.gz 2024-03-31T15:15:47.227 INFO:teuthology.orchestra.run.smithi016.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.32881.log 2024-03-31T15:15:47.229 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.32881.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.32881.log.gz 2024-03-31T15:15:47.230 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50070.log 2024-03-31T15:15:47.234 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42003.log 2024-03-31T15:15:47.234 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.50070.log: 87.7%/var/log/ceph/ceph-client.admin.42003.log: -- replaced with /var/log/ceph/ceph-client.admin.50070.log.gz 2024-03-31T15:15:47.234 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44566.log 88.1% -- replaced with /var/log/ceph/ceph-client.admin.42003.log.gz 2024-03-31T15:15:47.234 INFO:teuthology.orchestra.run.smithi016.stderr: 2024-03-31T15:15:47.236 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.44566.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.44566.log.gz 2024-03-31T15:15:47.236 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38581.log 2024-03-31T15:15:47.239 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.38581.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.38581.log.gz 2024-03-31T15:15:47.239 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48041.log 2024-03-31T15:15:47.240 INFO:teuthology.orchestra.run.smithi089.stderr: 93.9% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2024-03-31T15:15:47.241 INFO:teuthology.orchestra.run.smithi089.stderr: 2024-03-31T15:15:47.242 INFO:teuthology.orchestra.run.smithi089.stderr:real 0m0.744s 2024-03-31T15:15:47.242 INFO:teuthology.orchestra.run.smithi089.stderr:user 0m3.343s 2024-03-31T15:15:47.242 INFO:teuthology.orchestra.run.smithi089.stderr:sys 0m0.332s 2024-03-31T15:15:47.243 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.48041.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.48041.log.gz 2024-03-31T15:15:47.243 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39224.log 2024-03-31T15:15:47.245 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.44750.log 2024-03-31T15:15:47.245 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33037.log 2024-03-31T15:15:47.247 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.39224.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.39224.log.gz 2024-03-31T15:15:47.251 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.0.44750.log: /var/log/ceph/ceph-client.admin.33037.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.33037.log.gz 2024-03-31T15:15:47.251 INFO:teuthology.orchestra.run.smithi016.stderr: 92.7% -- replaced with /var/log/ceph/ceph-client.0.44750.log.gz 2024-03-31T15:15:47.253 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38815.log 2024-03-31T15:15:47.255 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.38815.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.38815.log.gz 2024-03-31T15:15:47.256 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50174.log 2024-03-31T15:15:47.258 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.50174.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.50174.log.gz 2024-03-31T15:15:47.258 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53915.log 2024-03-31T15:15:47.261 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.53915.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50447.log 2024-03-31T15:15:47.261 INFO:teuthology.orchestra.run.smithi016.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.53915.log.gz 2024-03-31T15:15:47.262 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38442.log 2024-03-31T15:15:47.264 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.50447.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.50447.log.gz 2024-03-31T15:15:47.267 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.38442.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.38442.log.gz 2024-03-31T15:15:47.267 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37840.log 2024-03-31T15:15:47.270 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.37840.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.37840.log.gz 2024-03-31T15:15:47.270 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43923.log 2024-03-31T15:15:47.273 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.43923.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.43923.log.gz 2024-03-31T15:15:47.273 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44861.log 2024-03-31T15:15:47.276 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.44861.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.44861.log.gz 2024-03-31T15:15:47.276 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31679.log 2024-03-31T15:15:47.279 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43240.log 2024-03-31T15:15:47.279 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.31679.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.31679.log.gz 2024-03-31T15:15:47.282 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.43240.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.43240.log.gz 2024-03-31T15:15:47.282 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33779.log 2024-03-31T15:15:47.285 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.33779.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.33779.log.gz 2024-03-31T15:15:47.285 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47089.log 2024-03-31T15:15:47.288 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.47089.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.47089.log.gz 2024-03-31T15:15:47.288 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34126.log 2024-03-31T15:15:47.289 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36104.log 2024-03-31T15:15:47.291 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.34126.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.34126.log.gz 2024-03-31T15:15:47.294 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.36104.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.36104.log.gz 2024-03-31T15:15:47.294 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42647.log 2024-03-31T15:15:47.297 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.42647.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.42647.log.gz 2024-03-31T15:15:47.297 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33487.log 2024-03-31T15:15:47.300 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48015.log 2024-03-31T15:15:47.300 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.33487.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.33487.log.gz 2024-03-31T15:15:47.303 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.48015.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.48015.log.gz 2024-03-31T15:15:47.303 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49100.log 2024-03-31T15:15:47.306 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.49100.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.49100.log.gz 2024-03-31T15:15:47.306 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50896.log 2024-03-31T15:15:47.309 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.50896.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.50896.log.gz 2024-03-31T15:15:47.309 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32324.log 2024-03-31T15:15:47.312 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37643.log 2024-03-31T15:15:47.312 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.32324.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.32324.log.gz 2024-03-31T15:15:47.315 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.37643.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.37643.log.gz 2024-03-31T15:15:47.315 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47959.log 2024-03-31T15:15:47.321 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.47959.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.47959.log.gz 2024-03-31T15:15:47.321 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40838.log 2024-03-31T15:15:47.321 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.40838.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.40838.log.gz 2024-03-31T15:15:47.321 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37732.log 2024-03-31T15:15:47.324 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.37732.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.37732.log.gz 2024-03-31T15:15:47.324 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38208.log 2024-03-31T15:15:47.328 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51733.log 2024-03-31T15:15:47.328 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.38208.log: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.38208.log.gz 2024-03-31T15:15:47.329 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.51733.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51052.log 2024-03-31T15:15:47.329 INFO:teuthology.orchestra.run.smithi016.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.51733.log.gz 2024-03-31T15:15:47.332 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.51052.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.51052.log.gz 2024-03-31T15:15:47.332 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52433.log 2024-03-31T15:15:47.335 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.52433.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42112.log 2024-03-31T15:15:47.336 INFO:teuthology.orchestra.run.smithi016.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.52433.log.gz 2024-03-31T15:15:47.337 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.45433.log 2024-03-31T15:15:47.339 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.42112.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.42112.log.gz 2024-03-31T15:15:47.342 INFO:teuthology.orchestra.run.smithi016.stderr:gzip/var/log/ceph/ceph-client.0.45433.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.31815.log 2024-03-31T15:15:47.345 INFO:teuthology.orchestra.run.smithi016.stderr: 93.9% -- replaced with /var/log/ceph/ceph-client.0.45433.log.gz 2024-03-31T15:15:47.345 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39127.log 2024-03-31T15:15:47.347 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.31815.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.31815.log.gz 2024-03-31T15:15:47.349 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.39127.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.39127.log.gz 2024-03-31T15:15:47.350 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33753.log 2024-03-31T15:15:47.353 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.33753.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.33753.log.gz 2024-03-31T15:15:47.353 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52939.log 2024-03-31T15:15:47.356 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.52939.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.52939.log.gz 2024-03-31T15:15:47.356 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43369.log 2024-03-31T15:15:47.359 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.43369.log: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.43369.log.gz 2024-03-31T15:15:47.359 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.33298.log 2024-03-31T15:15:47.363 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38737.log 2024-03-31T15:15:47.363 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.0.33298.log: 90.8% -- replaced with /var/log/ceph/ceph-client.0.33298.log.gz 2024-03-31T15:15:47.366 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.38737.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.38737.log.gz 2024-03-31T15:15:47.366 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27628.log 2024-03-31T15:15:47.369 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.27628.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.27628.log.gz 2024-03-31T15:15:47.369 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44432.log 2024-03-31T15:15:47.372 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.44432.log: 93.8% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-03-31T15:15:47.372 INFO:teuthology.orchestra.run.smithi016.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.44432.log.gz 2024-03-31T15:15:47.372 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44236.log 2024-03-31T15:15:47.373 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40075.log 2024-03-31T15:15:47.375 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.44236.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.44236.log.gz 2024-03-31T15:15:47.376 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.40075.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48067.log 2024-03-31T15:15:47.376 INFO:teuthology.orchestra.run.smithi016.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.40075.log.gz 2024-03-31T15:15:47.377 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36680.log 2024-03-31T15:15:47.378 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.48067.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.48067.log.gz 2024-03-31T15:15:47.379 INFO:teuthology.orchestra.run.smithi016.stderr:gzip/var/log/ceph/ceph-client.admin.36680.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.33571.log 2024-03-31T15:15:47.380 INFO:teuthology.orchestra.run.smithi016.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.36680.log.gz 2024-03-31T15:15:47.381 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.33571.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.33571.log.gz 2024-03-31T15:15:47.382 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42985.log 2024-03-31T15:15:47.383 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41844.log 2024-03-31T15:15:47.385 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.42985.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.42985.log.gz 2024-03-31T15:15:47.385 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.41844.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.41844.log.gz 2024-03-31T15:15:47.386 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31420.log 2024-03-31T15:15:47.387 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39094.log 2024-03-31T15:15:47.389 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.31420.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.31420.log.gz 2024-03-31T15:15:47.390 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.39094.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.39094.log.gz 2024-03-31T15:15:47.390 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50591.log 2024-03-31T15:15:47.391 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46045.log 2024-03-31T15:15:47.393 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.50591.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.50591.log.gz 2024-03-31T15:15:47.394 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.46045.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.46045.log.gz 2024-03-31T15:15:47.394 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54253.log 2024-03-31T15:15:47.396 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47167.log 2024-03-31T15:15:47.397 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.54253.log: 93.8% 88.1% -- replaced with /var/log/ceph/ceph-client.admin.54253.log.gz 2024-03-31T15:15:47.397 INFO:teuthology.orchestra.run.smithi016.stderr: -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-03-31T15:15:47.398 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.47167.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.47167.log.gz 2024-03-31T15:15:47.398 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34680.log 2024-03-31T15:15:47.399 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50226.log 2024-03-31T15:15:47.401 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.34680.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53642.log 2024-03-31T15:15:47.401 INFO:teuthology.orchestra.run.smithi016.stderr: 91.7% -- replaced with /var/log/ceph/ceph-mds.a.log.gz 2024-03-31T15:15:47.402 INFO:teuthology.orchestra.run.smithi016.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.34680.log.gz 2024-03-31T15:15:47.402 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.50226.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47476.log 2024-03-31T15:15:47.402 INFO:teuthology.orchestra.run.smithi016.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.50226.log.gz 2024-03-31T15:15:47.403 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.53642.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33937.log 2024-03-31T15:15:47.403 INFO:teuthology.orchestra.run.smithi016.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.53642.log.gz 2024-03-31T15:15:47.404 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.47476.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51759.log 2024-03-31T15:15:47.405 INFO:teuthology.orchestra.run.smithi016.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.47476.log.gz 2024-03-31T15:15:47.405 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.33937.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50870.log 2024-03-31T15:15:47.406 INFO:teuthology.orchestra.run.smithi016.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.33937.log.gz 2024-03-31T15:15:47.407 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.51759.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50415.log 2024-03-31T15:15:47.407 INFO:teuthology.orchestra.run.smithi016.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.51759.log.gz 2024-03-31T15:15:47.407 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.50870.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42379.log 2024-03-31T15:15:47.408 INFO:teuthology.orchestra.run.smithi016.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.50870.log.gz 2024-03-31T15:15:47.409 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.50415.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31841.log 2024-03-31T15:15:47.409 INFO:teuthology.orchestra.run.smithi016.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.50415.log.gz 2024-03-31T15:15:47.409 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.42379.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40177.log 2024-03-31T15:15:47.410 INFO:teuthology.orchestra.run.smithi016.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.42379.log.gz 2024-03-31T15:15:47.411 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.31841.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42935.log 2024-03-31T15:15:47.411 INFO:teuthology.orchestra.run.smithi016.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.31841.log.gz 2024-03-31T15:15:47.412 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.40177.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37671.log 2024-03-31T15:15:47.413 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.42935.log: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.40177.log.gz 2024-03-31T15:15:47.413 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39276.log 2024-03-31T15:15:47.414 INFO:teuthology.orchestra.run.smithi016.stderr: 91.5% -- replaced with /var/log/ceph/ceph-client.admin.42935.log.gz 2024-03-31T15:15:47.414 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.37671.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51000.log 2024-03-31T15:15:47.414 INFO:teuthology.orchestra.run.smithi016.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.37671.log.gz 2024-03-31T15:15:47.415 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42594.log 2024-03-31T15:15:47.415 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.39276.log: 88.1%/var/log/ceph/ceph-client.admin.51000.log: -- replaced with /var/log/ceph/ceph-client.admin.39276.log.gz 2024-03-31T15:15:47.416 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 88.4% --verbose -- replaced with /var/log/ceph/ceph-client.admin.51000.log.gz -- 2024-03-31T15:15:47.417 INFO:teuthology.orchestra.run.smithi016.stderr: /var/log/ceph/ceph-client.admin.46714.log 2024-03-31T15:15:47.417 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.42594.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38331.log 2024-03-31T15:15:47.418 INFO:teuthology.orchestra.run.smithi016.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.42594.log.gz 2024-03-31T15:15:47.418 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.46714.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36866.log 2024-03-31T15:15:47.419 INFO:teuthology.orchestra.run.smithi016.stderr: 91.3% -- replaced with /var/log/ceph/ceph-client.admin.46714.log.gz 2024-03-31T15:15:47.419 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.38331.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50304.log 2024-03-31T15:15:47.420 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36518.log 2024-03-31T15:15:47.420 INFO:teuthology.orchestra.run.smithi016.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.38331.log.gz 2024-03-31T15:15:47.421 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.36866.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.36866.log.gz 2024-03-31T15:15:47.422 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.50304.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.50304.log.gz 2024-03-31T15:15:47.422 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42029.log 2024-03-31T15:15:47.423 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.36518.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.36518.log.gz 2024-03-31T15:15:47.423 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53109.log 2024-03-31T15:15:47.424 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.42029.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53320.log 2024-03-31T15:15:47.424 INFO:teuthology.orchestra.run.smithi016.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.42029.log.gz 2024-03-31T15:15:47.426 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.53109.log: gzip 88.1% -5 -- replaced with /var/log/ceph/ceph-client.admin.53109.log.gz 2024-03-31T15:15:47.426 INFO:teuthology.orchestra.run.smithi016.stderr: --verbose -- /var/log/ceph/ceph-client.admin.37866.log 2024-03-31T15:15:47.426 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32023.log 2024-03-31T15:15:47.427 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.53320.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.53320.log.gz 2024-03-31T15:15:47.427 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.37866.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45483.log 2024-03-31T15:15:47.428 INFO:teuthology.orchestra.run.smithi016.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.37866.log.gz 2024-03-31T15:15:47.429 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.32023.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.32023.log.gz 2024-03-31T15:15:47.429 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53033.log 2024-03-31T15:15:47.429 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.45483.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46202.log 2024-03-31T15:15:47.431 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.53033.log: gzip -5 90.8% --verbose -- replaced with /var/log/ceph/ceph-client.admin.45483.log.gz -- 2024-03-31T15:15:47.431 INFO:teuthology.orchestra.run.smithi016.stderr: /var/log/ceph/ceph-client.admin.46664.log 2024-03-31T15:15:47.431 INFO:teuthology.orchestra.run.smithi016.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.53033.log.gz 2024-03-31T15:15:47.432 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.46202.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43694.log 2024-03-31T15:15:47.432 INFO:teuthology.orchestra.run.smithi016.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.46202.log.gz 2024-03-31T15:15:47.433 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.46664.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42516.log 2024-03-31T15:15:47.434 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.43694.log: 91.3% -- replaced with /var/log/ceph/ceph-client.admin.46664.log.gz 2024-03-31T15:15:47.434 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36830.log 2024-03-31T15:15:47.434 INFO:teuthology.orchestra.run.smithi016.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.43694.log.gz 2024-03-31T15:15:47.435 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.42516.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49522.log 2024-03-31T15:15:47.435 INFO:teuthology.orchestra.run.smithi016.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.42516.log.gz 2024-03-31T15:15:47.436 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.36830.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40578.log 2024-03-31T15:15:47.437 INFO:teuthology.orchestra.run.smithi016.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.36830.log.gz 2024-03-31T15:15:47.438 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.49522.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32536.log 2024-03-31T15:15:47.438 INFO:teuthology.orchestra.run.smithi016.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.49522.log.gz 2024-03-31T15:15:47.439 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.40578.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34470.log 2024-03-31T15:15:47.439 INFO:teuthology.orchestra.run.smithi016.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.40578.log.gz 2024-03-31T15:15:47.439 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40332.log 2024-03-31T15:15:47.440 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.32536.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.32536.log.gz 2024-03-31T15:15:47.441 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.34470.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47193.log 2024-03-31T15:15:47.441 INFO:teuthology.orchestra.run.smithi016.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.34470.log.gz 2024-03-31T15:15:47.441 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.40332.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48222.log 2024-03-31T15:15:47.442 INFO:teuthology.orchestra.run.smithi016.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.40332.log.gz 2024-03-31T15:15:47.442 INFO:teuthology.orchestra.run.smithi016.stderr:gzip/var/log/ceph/ceph-client.admin.47193.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.37126.log 2024-03-31T15:15:47.443 INFO:teuthology.orchestra.run.smithi016.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.47193.log.gz 2024-03-31T15:15:47.444 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.48222.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50382.log 2024-03-31T15:15:47.444 INFO:teuthology.orchestra.run.smithi016.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.48222.log.gz 2024-03-31T15:15:47.445 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.37126.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.37126.log.gz 2024-03-31T15:15:47.445 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53003.log 2024-03-31T15:15:47.446 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.48278.log 2024-03-31T15:15:47.447 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.50382.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.50382.log.gz 2024-03-31T15:15:47.447 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.53003.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39607.log 2024-03-31T15:15:47.447 INFO:teuthology.orchestra.run.smithi016.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.53003.log.gz 2024-03-31T15:15:47.448 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.0.48278.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43668.log 2024-03-31T15:15:47.450 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.39607.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53510.log 2024-03-31T15:15:47.450 INFO:teuthology.orchestra.run.smithi016.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.39607.log.gz 2024-03-31T15:15:47.450 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.43668.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.43668.log.gz 2024-03-31T15:15:47.452 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.53510.log: 94.2% -- replaced with /var/log/ceph/ceph-client.0.48278.log.gz 2024-03-31T15:15:47.452 INFO:teuthology.orchestra.run.smithi016.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.53510.log.gz 2024-03-31T15:15:47.452 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34257.log 2024-03-31T15:15:47.453 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41525.log 2024-03-31T15:15:47.454 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.34257.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45018.log 2024-03-31T15:15:47.455 INFO:teuthology.orchestra.run.smithi016.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.34257.log.gz 2024-03-31T15:15:47.455 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.41525.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51892.log 2024-03-31T15:15:47.456 INFO:teuthology.orchestra.run.smithi016.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.41525.log.gz 2024-03-31T15:15:47.456 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.45018.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38912.log 2024-03-31T15:15:47.457 INFO:teuthology.orchestra.run.smithi016.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.45018.log.gz 2024-03-31T15:15:47.458 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.51892.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.22601.log 2024-03-31T15:15:47.458 INFO:teuthology.orchestra.run.smithi016.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.51892.log.gz 2024-03-31T15:15:47.459 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.38912.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36597.log 2024-03-31T15:15:47.459 INFO:teuthology.orchestra.run.smithi016.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.38912.log.gz 2024-03-31T15:15:47.460 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41603.log 2024-03-31T15:15:47.460 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph.tmp-client.admin.22601.log: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.22601.log.gz 2024-03-31T15:15:47.461 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.36597.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51356.log 2024-03-31T15:15:47.461 INFO:teuthology.orchestra.run.smithi016.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.36597.log.gz 2024-03-31T15:15:47.462 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.41603.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32855.log 2024-03-31T15:15:47.462 INFO:teuthology.orchestra.run.smithi016.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.41603.log.gz 2024-03-31T15:15:47.463 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50278.log 2024-03-31T15:15:47.464 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.51356.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.51356.log.gz 2024-03-31T15:15:47.464 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.32855.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31787.log 2024-03-31T15:15:47.464 INFO:teuthology.orchestra.run.smithi016.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.32855.log.gz 2024-03-31T15:15:47.465 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.50278.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.50278.log.gz 2024-03-31T15:15:47.467 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.31787.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.31787.log.gz 2024-03-31T15:15:47.467 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50736.log 2024-03-31T15:15:47.468 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40987.log 2024-03-31T15:15:47.469 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.50736.log: gzip 87.9% -- replaced with /var/log/ceph/ceph-client.admin.50736.log.gz -5 2024-03-31T15:15:47.469 INFO:teuthology.orchestra.run.smithi016.stderr: --verbose -- /var/log/ceph/ceph-client.admin.46484.log 2024-03-31T15:15:47.470 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.40987.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49296.log 2024-03-31T15:15:47.471 INFO:teuthology.orchestra.run.smithi016.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.40987.log.gz 2024-03-31T15:15:47.471 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.46484.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49631.log 2024-03-31T15:15:47.472 INFO:teuthology.orchestra.run.smithi016.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.46484.log.gz 2024-03-31T15:15:47.473 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.49296.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48119.log 2024-03-31T15:15:47.473 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.49631.log: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.49296.log.gz 2024-03-31T15:15:47.474 INFO:teuthology.orchestra.run.smithi016.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.49631.log.gz 2024-03-31T15:15:47.475 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.48119.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.48119.log.gz 2024-03-31T15:15:47.475 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37301.log 2024-03-31T15:15:47.476 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52564.log 2024-03-31T15:15:47.477 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.37301.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46019.log 2024-03-31T15:15:47.477 INFO:teuthology.orchestra.run.smithi016.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.37301.log.gz 2024-03-31T15:15:47.479 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.52564.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.52564.log.gz 2024-03-31T15:15:47.480 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.46019.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.46019.log.gz 2024-03-31T15:15:47.480 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39685.log 2024-03-31T15:15:47.481 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36414.log 2024-03-31T15:15:47.482 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.39685.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44462.log 2024-03-31T15:15:47.483 INFO:teuthology.orchestra.run.smithi016.stderr: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.39685.log.gz 2024-03-31T15:15:47.483 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.36414.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51128.log 2024-03-31T15:15:47.483 INFO:teuthology.orchestra.run.smithi016.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.36414.log.gz 2024-03-31T15:15:47.484 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.44462.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36710.log 2024-03-31T15:15:47.485 INFO:teuthology.orchestra.run.smithi016.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.44462.log.gz 2024-03-31T15:15:47.486 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.51128.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34921.log 2024-03-31T15:15:47.486 INFO:teuthology.orchestra.run.smithi016.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.51128.log.gz 2024-03-31T15:15:47.487 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.36710.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32829.log 2024-03-31T15:15:47.487 INFO:teuthology.orchestra.run.smithi016.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.36710.log.gz 2024-03-31T15:15:47.488 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41326.log 2024-03-31T15:15:47.488 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.34921.log: /var/log/ceph/ceph-client.admin.32829.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.34921.log.gz 2024-03-31T15:15:47.489 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37944.log 2024-03-31T15:15:47.489 INFO:teuthology.orchestra.run.smithi016.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.32829.log.gz 2024-03-31T15:15:47.490 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.41326.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39555.log 2024-03-31T15:15:47.490 INFO:teuthology.orchestra.run.smithi016.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.41326.log.gz 2024-03-31T15:15:47.491 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43800.log 2024-03-31T15:15:47.491 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.37944.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.37944.log.gz 2024-03-31T15:15:47.492 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.39555.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46977.log 2024-03-31T15:15:47.492 INFO:teuthology.orchestra.run.smithi016.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.39555.log.gz 2024-03-31T15:15:47.493 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.43800.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51435.log 2024-03-31T15:15:47.493 INFO:teuthology.orchestra.run.smithi016.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.43800.log.gz 2024-03-31T15:15:47.494 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50922.log 2024-03-31T15:15:47.495 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.46977.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.46977.log.gz 2024-03-31T15:15:47.495 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.51435.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53739.log 2024-03-31T15:15:47.495 INFO:teuthology.orchestra.run.smithi016.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.51435.log.gz 2024-03-31T15:15:47.496 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.50922.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46124.log 2024-03-31T15:15:47.496 INFO:teuthology.orchestra.run.smithi016.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.50922.log.gz 2024-03-31T15:15:47.497 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42568.log 2024-03-31T15:15:47.498 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.53739.log: /var/log/ceph/ceph-client.admin.46124.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.53739.log.gz 2024-03-31T15:15:47.499 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44809.log 2024-03-31T15:15:47.499 INFO:teuthology.orchestra.run.smithi016.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.46124.log.gz 2024-03-31T15:15:47.499 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.42568.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33963.log 2024-03-31T15:15:47.499 INFO:teuthology.orchestra.run.smithi016.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.42568.log.gz 2024-03-31T15:15:47.500 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44345.log 2024-03-31T15:15:47.501 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.44809.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.44809.log.gz 2024-03-31T15:15:47.502 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.33963.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37353.log 2024-03-31T15:15:47.502 INFO:teuthology.orchestra.run.smithi016.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.33963.log.gz 2024-03-31T15:15:47.503 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.44345.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.44345.log.gz 2024-03-31T15:15:47.503 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32337.log 2024-03-31T15:15:47.504 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40101.log 2024-03-31T15:15:47.504 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.37353.log: /var/log/ceph/ceph-client.admin.32337.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.37353.log.gz 2024-03-31T15:15:47.505 INFO:teuthology.orchestra.run.smithi016.stderr:gzip 88.8% -- replaced with /var/log/ceph/ceph-client.admin.32337.log.gz -5 2024-03-31T15:15:47.505 INFO:teuthology.orchestra.run.smithi016.stderr: --verbose -- /var/log/ceph/ceph-client.admin.54305.log 2024-03-31T15:15:47.506 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.40101.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54357.log 2024-03-31T15:15:47.506 INFO:teuthology.orchestra.run.smithi016.stderr: 91.8% -- replaced with /var/log/ceph/ceph-mds.c.log.gz 2024-03-31T15:15:47.507 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.54305.log: 91.4% -- replaced with /var/log/ceph/ceph-client.admin.40101.log.gz 2024-03-31T15:15:47.507 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53889.log 2024-03-31T15:15:47.507 INFO:teuthology.orchestra.run.smithi016.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.54305.log.gz 2024-03-31T15:15:47.508 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.54357.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46560.log 2024-03-31T15:15:47.509 INFO:teuthology.orchestra.run.smithi016.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.54357.log.gz 2024-03-31T15:15:47.509 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.53889.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43897.log 2024-03-31T15:15:47.510 INFO:teuthology.orchestra.run.smithi016.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.53889.log.gz 2024-03-31T15:15:47.510 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.46560.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46892.log 2024-03-31T15:15:47.511 INFO:teuthology.orchestra.run.smithi016.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.46560.log.gz 2024-03-31T15:15:47.512 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.43897.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34548.log 2024-03-31T15:15:47.512 INFO:teuthology.orchestra.run.smithi016.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.43897.log.gz 2024-03-31T15:15:47.512 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.46892.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52763.log 2024-03-31T15:15:47.513 INFO:teuthology.orchestra.run.smithi016.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.46892.log.gz 2024-03-31T15:15:47.514 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.34548.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46176.log 2024-03-31T15:15:47.514 INFO:teuthology.orchestra.run.smithi016.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.34548.log.gz 2024-03-31T15:15:47.515 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.52763.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52407.log 2024-03-31T15:15:47.515 INFO:teuthology.orchestra.run.smithi016.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.52763.log.gz 2024-03-31T15:15:47.516 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.46176.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40734.log 2024-03-31T15:15:47.516 INFO:teuthology.orchestra.run.smithi016.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.46176.log.gz 2024-03-31T15:15:47.517 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.52407.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36944.log 2024-03-31T15:15:47.517 INFO:teuthology.orchestra.run.smithi016.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.52407.log.gz 2024-03-31T15:15:47.518 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.40734.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45738.log 2024-03-31T15:15:47.518 INFO:teuthology.orchestra.run.smithi016.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.40734.log.gz 2024-03-31T15:15:47.519 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.36944.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42673.log 2024-03-31T15:15:47.520 INFO:teuthology.orchestra.run.smithi016.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.36944.log.gz 2024-03-31T15:15:47.520 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.45738.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53294.log 2024-03-31T15:15:47.520 INFO:teuthology.orchestra.run.smithi016.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.45738.log.gz 2024-03-31T15:15:47.521 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.42673.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37275.log 2024-03-31T15:15:47.522 INFO:teuthology.orchestra.run.smithi016.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.42673.log.gz 2024-03-31T15:15:47.523 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.53294.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42081.log 2024-03-31T15:15:47.523 INFO:teuthology.orchestra.run.smithi016.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.53294.log.gz 2024-03-31T15:15:47.523 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.37275.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51330.log 2024-03-31T15:15:47.525 INFO:teuthology.orchestra.run.smithi016.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.37275.log.gz 2024-03-31T15:15:47.527 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52156.log 2024-03-31T15:15:47.527 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.42081.log: /var/log/ceph/ceph-client.admin.51330.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.42081.log.gz 2024-03-31T15:15:47.527 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42208.log 2024-03-31T15:15:47.527 INFO:teuthology.orchestra.run.smithi016.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.51330.log.gz 2024-03-31T15:15:47.527 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.52156.log: 91.4% -- replaced with /var/log/ceph/ceph-client.admin.52156.log.gz 2024-03-31T15:15:47.528 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.42208.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.42208.log.gz 2024-03-31T15:15:47.542 INFO:teuthology.orchestra.run.smithi016.stderr: 90.7% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-03-31T15:15:47.587 INFO:teuthology.orchestra.run.smithi016.stderr: 93.7% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-03-31T15:15:47.604 INFO:teuthology.orchestra.run.smithi016.stderr: 93.8% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-03-31T15:15:47.606 INFO:teuthology.orchestra.run.smithi016.stderr: 2024-03-31T15:15:47.606 INFO:teuthology.orchestra.run.smithi016.stderr:real 0m1.144s 2024-03-31T15:15:47.606 INFO:teuthology.orchestra.run.smithi016.stderr:user 0m5.132s 2024-03-31T15:15:47.606 INFO:teuthology.orchestra.run.smithi016.stderr:sys 0m1.609s 2024-03-31T15:15:47.607 INFO:tasks.ceph:Archiving logs... 2024-03-31T15:15:47.608 DEBUG:teuthology.misc:Transferring archived files from smithi016:/var/log/ceph to /home/teuthworker/archive/vshankar-2024-03-30_23:12:08-fs-wip-vshankar-testing-20240330.170739-testing-default-smithi/7631609/remote/smithi016/log 2024-03-31T15:15:47.609 DEBUG:teuthology.orchestra.run.smithi016:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-31T15:15:48.704 DEBUG:teuthology.misc:Transferring archived files from smithi089:/var/log/ceph to /home/teuthworker/archive/vshankar-2024-03-30_23:12:08-fs-wip-vshankar-testing-20240330.170739-testing-default-smithi/7631609/remote/smithi089/log 2024-03-31T15:15:48.705 DEBUG:teuthology.orchestra.run.smithi089:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-31T15:15:49.073 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-03-31T15:15:49.082 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-03-31T15:15:49.082 DEBUG:teuthology.orchestra.run.smithi016:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-31T15:15:49.085 DEBUG:teuthology.orchestra.run.smithi089:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-31T15:15:49.164 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-31T15:15:49.172 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-31T15:15:49.173 DEBUG:teuthology.orchestra.run.smithi016:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-31T15:15:49.176 DEBUG:teuthology.orchestra.run.smithi089:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-31T15:15:49.189 INFO:teuthology.orchestra.run.smithi016.stdout: remote refid st t when poll reach delay offset jitter 2024-03-31T15:15:49.189 INFO:teuthology.orchestra.run.smithi016.stdout:============================================================================== 2024-03-31T15:15:49.189 INFO:teuthology.orchestra.run.smithi016.stdout:*hv01.front.sepi 44.4.53.2 2 u 43 64 377 0.091 -1.780 1.431 2024-03-31T15:15:49.189 INFO:teuthology.orchestra.run.smithi016.stdout:+hv02.front.sepi 63.231.80.2 3 u 36 64 377 0.087 -0.234 1.402 2024-03-31T15:15:49.189 INFO:teuthology.orchestra.run.smithi016.stdout:+hv03.front.sepi 74.6.168.72 3 u 44 64 377 0.073 -0.984 1.098 2024-03-31T15:15:49.189 INFO:teuthology.orchestra.run.smithi016.stdout: hv04.front.sepi .INIT. 16 u - 256 0 0.000 +0.000 0.000 2024-03-31T15:15:49.191 INFO:teuthology.orchestra.run.smithi089.stdout: remote refid st t when poll reach delay offset jitter 2024-03-31T15:15:49.192 INFO:teuthology.orchestra.run.smithi089.stdout:============================================================================== 2024-03-31T15:15:49.192 INFO:teuthology.orchestra.run.smithi089.stdout:*hv01.front.sepi 44.4.53.2 2 u 36 64 377 0.107 -6.862 3.210 2024-03-31T15:15:49.192 INFO:teuthology.orchestra.run.smithi089.stdout:+hv02.front.sepi 63.231.80.2 3 u 45 64 377 0.105 -2.404 2.180 2024-03-31T15:15:49.192 INFO:teuthology.orchestra.run.smithi089.stdout:+hv03.front.sepi 74.6.168.72 3 u 44 64 377 0.088 -0.921 3.352 2024-03-31T15:15:49.192 INFO:teuthology.orchestra.run.smithi089.stdout: hv04.front.sepi .INIT. 16 u - 256 0 0.000 +0.000 0.000 2024-03-31T15:15:49.193 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-31T15:15:49.220 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-31T15:15:49.221 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-31T15:15:49.245 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-31T15:15:49.266 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-31T15:15:49.286 INFO:teuthology.task.internal:Duration was 1430.458390 seconds 2024-03-31T15:15:49.287 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-31T15:15:49.295 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-31T15:15:49.295 DEBUG:teuthology.orchestra.run.smithi016:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-31T15:15:49.298 DEBUG:teuthology.orchestra.run.smithi089:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-31T15:15:49.402 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-31T15:15:49.402 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi016.front.sepia.ceph.com 2024-03-31T15:15:49.402 DEBUG:teuthology.orchestra.run.smithi016:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-31T15:15:49.424 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi089.front.sepia.ceph.com 2024-03-31T15:15:49.424 DEBUG:teuthology.orchestra.run.smithi089:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-31T15:15:49.444 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-31T15:15:49.444 DEBUG:teuthology.orchestra.run.smithi016:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-31T15:15:49.475 DEBUG:teuthology.orchestra.run.smithi089:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-31T15:15:49.529 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-31T15:15:49.529 DEBUG:teuthology.orchestra.run.smithi016:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-31T15:15:49.558 DEBUG:teuthology.orchestra.run.smithi089:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-31T15:15:49.647 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-31T15:15:49.657 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-31T15:15:49.658 DEBUG:teuthology.orchestra.run.smithi016:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-31T15:15:49.678 DEBUG:teuthology.orchestra.run.smithi089:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-31T15:15:49.697 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-31T15:15:49.707 DEBUG:teuthology.orchestra.run.smithi016:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-31T15:15:49.722 DEBUG:teuthology.orchestra.run.smithi089:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-31T15:15:49.737 INFO:teuthology.orchestra.run.smithi016.stdout:kernel.core_pattern = core 2024-03-31T15:15:49.756 INFO:teuthology.orchestra.run.smithi089.stdout:kernel.core_pattern = core 2024-03-31T15:15:49.775 DEBUG:teuthology.orchestra.run.smithi016:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-31T15:15:49.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-31T15:15:49.803 DEBUG:teuthology.orchestra.run.smithi089:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-31T15:15:49.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-31T15:15:49.826 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-31T15:15:49.836 INFO:teuthology.task.internal:Transferring archived files... 2024-03-31T15:15:49.837 DEBUG:teuthology.misc:Transferring archived files from smithi016:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/vshankar-2024-03-30_23:12:08-fs-wip-vshankar-testing-20240330.170739-testing-default-smithi/7631609/remote/smithi016 2024-03-31T15:15:49.837 DEBUG:teuthology.orchestra.run.smithi016:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-31T15:15:49.871 DEBUG:teuthology.misc:Transferring archived files from smithi089:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/vshankar-2024-03-30_23:12:08-fs-wip-vshankar-testing-20240330.170739-testing-default-smithi/7631609/remote/smithi089 2024-03-31T15:15:49.871 DEBUG:teuthology.orchestra.run.smithi089:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-31T15:15:49.896 INFO:teuthology.task.internal:Removing archive directory... 2024-03-31T15:15:49.897 DEBUG:teuthology.orchestra.run.smithi016:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-31T15:15:49.911 DEBUG:teuthology.orchestra.run.smithi089:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-31T15:15:49.943 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-31T15:15:49.954 INFO:teuthology.task.internal:Not uploading archives. 2024-03-31T15:15:49.954 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-31T15:15:49.973 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-31T15:15:49.974 DEBUG:teuthology.orchestra.run.smithi016:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-31T15:15:49.976 DEBUG:teuthology.orchestra.run.smithi089:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-31T15:15:49.982 INFO:teuthology.orchestra.run.smithi016.stdout: 655431 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 31 15:15 /home/ubuntu/cephtest 2024-03-31T15:15:49.992 INFO:teuthology.orchestra.run.smithi089.stdout: 655431 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 31 15:15 /home/ubuntu/cephtest 2024-03-31T15:15:49.994 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-31T15:15:50.005 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-31T15:15:50.025 INFO:teuthology.nuke:Checking targets against current locks 2024-03-31T15:15:50.051 DEBUG:teuthology.nuke:shortname: smithi016 2024-03-31T15:15:50.052 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-31T15:15:50.079 DEBUG:teuthology.nuke:shortname: smithi089 2024-03-31T15:15:50.079 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-31T15:15:50.082 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi016.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/vshankar-2024-03-30_23:12:08-fs-wip-vshankar-testing-20240330.170739-testing-default-smithi/7631609', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-31 14:42:50.434724', 'locked_by': 'scheduled_vshankar@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGrNGRw477WUQu17A6o65D4kDQ6WSLg/SUYnTZXG9vNI9kj8M4+jdSsqhTA1AVsIdEZoduIq19dWV/GhMsuzXqM='} 2024-03-31T15:15:50.099 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi089.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/vshankar-2024-03-30_23:12:08-fs-wip-vshankar-testing-20240330.170739-testing-default-smithi/7631609', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-31 14:42:50.436123', 'locked_by': 'scheduled_vshankar@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBANsxlXyxrksTMZg1EXGMkYGT/ASySvp6CS9Cas0w9gzWiSepEnTbAeyQC0DgQosrPfrSbAwG8nwNKSj4IuxKLE='} 2024-03-31T15:15:50.131 INFO:teuthology.orchestra.console.smithi016:Power off 2024-03-31T15:15:50.131 DEBUG:teuthology.orchestra.console.smithi016:pexpect command: ipmitool -H smithi016.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-31T15:15:50.145 INFO:teuthology.orchestra.console.smithi089:Power off 2024-03-31T15:15:50.146 DEBUG:teuthology.orchestra.console.smithi089:pexpect command: ipmitool -H smithi089.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-31T15:15:50.165 DEBUG:teuthology.orchestra.console.smithi016:power off output: Chassis Power Control: Down/Off 2024-03-31T15:15:50.165 DEBUG:teuthology.orchestra.console.smithi016:pexpect command: ipmitool -H smithi016.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-31T15:15:50.175 DEBUG:teuthology.orchestra.console.smithi089:power off output: Chassis Power Control: Down/Off 2024-03-31T15:15:50.175 DEBUG:teuthology.orchestra.console.smithi089:pexpect command: ipmitool -H smithi089.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-31T15:15:50.188 DEBUG:teuthology.orchestra.console.smithi016:check power output: Chassis Power is on 2024-03-31T15:15:50.195 DEBUG:teuthology.orchestra.console.smithi089:check power output: Chassis Power is on 2024-03-31T15:15:54.189 DEBUG:teuthology.orchestra.console.smithi016:pexpect command: ipmitool -H smithi016.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-31T15:15:54.198 DEBUG:teuthology.orchestra.console.smithi089:pexpect command: ipmitool -H smithi089.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-31T15:15:54.300 DEBUG:teuthology.orchestra.console.smithi016:check power output: Chassis Power is on 2024-03-31T15:15:54.307 DEBUG:teuthology.orchestra.console.smithi089:check power output: Chassis Power is on 2024-03-31T15:15:58.302 DEBUG:teuthology.orchestra.console.smithi016:pexpect command: ipmitool -H smithi016.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-31T15:15:58.309 DEBUG:teuthology.orchestra.console.smithi089:pexpect command: ipmitool -H smithi089.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-31T15:15:58.412 DEBUG:teuthology.orchestra.console.smithi016:check power output: Chassis Power is off 2024-03-31T15:15:58.419 DEBUG:teuthology.orchestra.console.smithi089:check power output: Chassis Power is off 2024-03-31T15:15:58.513 INFO:teuthology.orchestra.console.smithi016:Power off completed 2024-03-31T15:15:58.520 INFO:teuthology.orchestra.console.smithi089:Power off completed 2024-03-31T15:15:58.625 INFO:teuthology.run:Summary data: description: fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v1 tasks/forward-scrub} duration: 1430.4583902359009 failure_reason: '"2024-03-31T15:10:11.682944+0000 mds.c (mds.0) 1 : cluster [ERR] dir 0x10000000000 object missing on disk; some files may be lost (/dir)" in cluster log' flavor: default mon.a-kernel-sha1: 1df0c7e8a2354c42d738e5454ba83ea7332deac2 mon.b-kernel-sha1: 1df0c7e8a2354c42d738e5454ba83ea7332deac2 owner: scheduled_vshankar@teuthology success: false 2024-03-31T15:15:58.625 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-31T15:15:58.709 INFO:teuthology.run:FAIL